quilltap 4.7.0-dev.117 → 4.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -0
- package/bin/quilltap.js +15 -2
- package/lib/__tests__/completion-coverage.test.js +61 -0
- package/lib/completion/bash.template +15 -2
- package/lib/completion/fish.template +11 -2
- package/lib/completion/zsh.template +29 -0
- package/lib/file-verify-commands.js +229 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -89,6 +89,20 @@ Quilltap stores its database, files, and logs in a platform-specific directory:
|
|
|
89
89
|
|
|
90
90
|
Override with `--data-dir` or the `QUILLTAP_DATA_DIR` environment variable.
|
|
91
91
|
|
|
92
|
+
## Named Instances
|
|
93
|
+
|
|
94
|
+
Register a data directory once and point the CLI at it by name, instead of repeating `--data-dir`:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
quilltap instances add Friday ~/iCloud/Quilltap/Friday # Register (prompts for the passphrase if encrypted)
|
|
98
|
+
quilltap instances list # Registered instances (* marks the default)
|
|
99
|
+
quilltap instances default Friday # Make it the fall-through for flag-free runs
|
|
100
|
+
quilltap instances rename Friday Weekday # Rename, preserving the stored passphrase
|
|
101
|
+
quilltap instances remove Friday # Unregister
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Every subcommand then accepts `--instance <name>` in place of `--data-dir`. The registry lives at `<app-support>/Quilltap/instances.json` (mode 0600; e.g. `~/Library/Application Support/Quilltap/instances.json` on macOS). **Resolution precedence:** `--data-dir` > `--instance` > registered default > `QUILLTAP_DATA_DIR` > the OS platform default. Pass the **instance root** (e.g. `~/iCloud/Quilltap/Friday`), not its `data/` subdirectory.
|
|
105
|
+
|
|
92
106
|
## Database Tool
|
|
93
107
|
|
|
94
108
|
The encrypted SQLite databases (main, LLM logs, mount index) can be queried directly via `quilltap db`. There are two modes: high-level subcommands that auto-pick the right database and resolve characters/chats/projects by name, and a low-level path for arbitrary SQL.
|
|
@@ -216,6 +230,17 @@ quilltap memories status [--character <name|id>] # Per-hol
|
|
|
216
230
|
|
|
217
231
|
Shared filter flags apply to `ls`, `find`, `grep`, and `status` where they make sense: `--character`, `--about` (with `self` / `none` shortcuts), `--source`, `--chat` (with `none` for manual entries), `--project`, `--since`, `--until`, `--min-importance`, `--min-reinforced`, `--has-embedding` / `--no-embedding`. Sort flags (`--sort reinforced|importance|created|accessed|reinforcement-count|links`, plus `-r` to reverse) apply to `ls`, `find`, and `grep`. Names accept fuzzy substrings; ambiguous names print candidates and exit 2. `--json` is supported by every verb. The legacy `quilltap db memories --character <name>` verb remains undisturbed.
|
|
218
232
|
|
|
233
|
+
## Memory Extraction Dry-Run
|
|
234
|
+
|
|
235
|
+
`quilltap memory-diff <chatId>` is a diagnostic that dumps a chat's existing memories and dry-runs re-extraction against it **without writing anything** — useful for seeing how an extractor change would land on a real conversation.
|
|
236
|
+
|
|
237
|
+
```bash
|
|
238
|
+
quilltap memory-diff <chatId> --instance Friday # Report to the current directory
|
|
239
|
+
quilltap memory-diff <chatId> --out /tmp/diff --concurrency 8
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
Needs a running server (`--port`, default 3000) to reach the extraction pipeline. `--out <dir>` sets the report destination (default: cwd); `--concurrency N` bounds parallel turns (default 4, max 32).
|
|
243
|
+
|
|
219
244
|
## Maintenance & Cleanup
|
|
220
245
|
|
|
221
246
|
`quilltap maintenance` is the manual trigger for the retention sweeps that otherwise run on the server's daily maintenance tick. It reaps data with no bearing on characters, stories, or memories.
|
|
@@ -228,6 +253,31 @@ quilltap maintenance run --instance Friday # Run the sweeps once (lock-gated;
|
|
|
228
253
|
|
|
229
254
|
`maintenance run` is a DB writer: it claims `<dataDir>/quilltap.lock` and **refuses while a running Quilltap server holds it** — stop the server first. Because it can only run with the server down, it performs the sweeps expressible as direct SQL/filesystem work: reaping finished background jobs (COMPLETED after 7 days, DEAD after 30, keyed off `completedAt`), closed terminal sessions older than 30 days plus their transcript files, and orphaned mount-index files. The **stale-chat asset collapse** (superseded story-backgrounds and wardrobe avatars) needs the server's file-storage machinery and runs only on the server's daily tick — `status` reports a stale-chat count so you can see the backlog. Retention windows mirror `lib/background-jobs/maintenance/retention-constants.ts`.
|
|
230
255
|
|
|
256
|
+
## Logs
|
|
257
|
+
|
|
258
|
+
`quilltap logs` tails or prints an instance's log files without hunting down the logs directory. Prefer it over `tail -f` — it follows across log rotation and can merge streams.
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
quilltap logs # Last 100 lines of combined.log
|
|
262
|
+
quilltap logs --tail 0 --stream error # Whole error log
|
|
263
|
+
quilltap logs -f --grep "MEMORY_EXTRACTION" # Follow, filtered by regex
|
|
264
|
+
quilltap logs --stream combined,error # Merge streams with [stream] prefixes
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Flags: `--stream combined|error|stdout|stderr|startup` (comma-separated for multiple), `--tail N` (default 100; `0` = full file), `--follow` / `-f`, `--grep <pattern>` (JS regex). Resolves the logs directory via the same `--instance` / `--data-dir` plumbing as the rest of the CLI.
|
|
268
|
+
|
|
269
|
+
## Migrations
|
|
270
|
+
|
|
271
|
+
`quilltap migrations` inspects migration state read-only. The actual runner stays at server startup (where the loading screen reports progress), so the CLI deliberately won't apply anything.
|
|
272
|
+
|
|
273
|
+
```bash
|
|
274
|
+
quilltap migrations status # Counts: in-source vs recorded-applied vs not-yet-recorded
|
|
275
|
+
quilltap migrations pending # Just the not-yet-recorded list
|
|
276
|
+
quilltap migrations run --dry-run # List what would run (refuses without --dry-run)
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
`--json` works on all three. "Not yet recorded" includes migrations whose `shouldRun()` is `false` on this instance — the CLI doesn't evaluate the predicate, so it can't distinguish "would skip" from "would run."
|
|
280
|
+
|
|
231
281
|
## Theme Management
|
|
232
282
|
|
|
233
283
|
The CLI includes theme management commands:
|
package/bin/quilltap.js
CHANGED
|
@@ -94,7 +94,12 @@ Subcommands:
|
|
|
94
94
|
docs Inspect, read, and export document mounts
|
|
95
95
|
memories Search, browse, and graph memories
|
|
96
96
|
instances Register / inspect named Quilltap instances
|
|
97
|
+
logs Tail or print an instance log file
|
|
98
|
+
migrations Inspect migration status (status / pending / run)
|
|
99
|
+
maintenance Run retention / cleanup sweeps (status / run)
|
|
100
|
+
file-verify Force-download cloud-evicted data files (iCloud, etc.)
|
|
97
101
|
memory-diff <chatId> Dump existing memories and dry-run re-extraction for a chat
|
|
102
|
+
completion <shell> Generate a shell completion script (bash / zsh / fish)
|
|
98
103
|
|
|
99
104
|
Options:
|
|
100
105
|
-p, --port <number> Port to listen on (default: 3000)
|
|
@@ -1103,7 +1108,7 @@ async function dbCommand(args) {
|
|
|
1103
1108
|
// to the subcommand. Each subcommand parses these flags position-independently,
|
|
1104
1109
|
// so they behave the same before or after the verb.
|
|
1105
1110
|
const SUBCOMMANDS = new Set([
|
|
1106
|
-
'db', 'themes', 'docs', 'memories', 'instances', 'memory-diff', 'completion', 'logs', 'migrations', 'maintenance',
|
|
1111
|
+
'db', 'themes', 'docs', 'memories', 'instances', 'memory-diff', 'completion', 'logs', 'migrations', 'maintenance', 'file-verify',
|
|
1107
1112
|
]);
|
|
1108
1113
|
// Global flags that consume the following token as their value.
|
|
1109
1114
|
const GLOBAL_VALUE_FLAGS = new Set(['-p', '--port', '-d', '--data-dir', '-i', '--instance', '--passphrase']);
|
|
@@ -1128,7 +1133,9 @@ const subArgs = subIdx >= 0 ? [...cliArgs.slice(0, subIdx), ...cliArgs.slice(sub
|
|
|
1128
1133
|
// native-module heal, so self-heal the database ABI here first. Cheap no-op when
|
|
1129
1134
|
// healthy; rebuilds (with a friendly notice, not an error) only on a real
|
|
1130
1135
|
// Node-ABI mismatch — e.g. after the user upgrades Node under a cached install.
|
|
1131
|
-
|
|
1136
|
+
// `file-verify` is excluded: it's a pure-fs recovery tool that never opens the
|
|
1137
|
+
// database, so it must work even when the native binding is broken.
|
|
1138
|
+
if (subName && subName !== 'file-verify') {
|
|
1132
1139
|
ensureDatabaseNativeModule();
|
|
1133
1140
|
}
|
|
1134
1141
|
|
|
@@ -1185,6 +1192,12 @@ if (subName === 'db') {
|
|
|
1185
1192
|
console.error(`Error: ${err.message}`);
|
|
1186
1193
|
process.exit(1);
|
|
1187
1194
|
});
|
|
1195
|
+
} else if (subName === 'file-verify') {
|
|
1196
|
+
const { fileVerifyCommand } = require('../lib/file-verify-commands');
|
|
1197
|
+
fileVerifyCommand(subArgs).catch(err => {
|
|
1198
|
+
console.error(`Error: ${err.message}`);
|
|
1199
|
+
process.exit(1);
|
|
1200
|
+
});
|
|
1188
1201
|
} else {
|
|
1189
1202
|
main();
|
|
1190
1203
|
}
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Guard test: the shell-completion templates and the top-level `--help` must
|
|
3
|
+
* stay in sync with the real subcommand dispatch table in bin/quilltap.js.
|
|
4
|
+
*
|
|
5
|
+
* This is the check that was missing when the docs/completions drifted behind
|
|
6
|
+
* the CLI (e.g. the `maintenance` subcommand shipped without ever being added
|
|
7
|
+
* to any completion script or to `quilltap --help`). If you add a top-level
|
|
8
|
+
* subcommand to SUBCOMMANDS, this test fails until you also teach the three
|
|
9
|
+
* completion templates and printHelp() about it.
|
|
10
|
+
*
|
|
11
|
+
* @jest-environment node
|
|
12
|
+
*/
|
|
13
|
+
|
|
14
|
+
'use strict';
|
|
15
|
+
|
|
16
|
+
const fs = require('fs');
|
|
17
|
+
const path = require('path');
|
|
18
|
+
|
|
19
|
+
const BIN = path.join(__dirname, '..', '..', 'bin', 'quilltap.js');
|
|
20
|
+
const COMPLETION_DIR = path.join(__dirname, '..', 'completion');
|
|
21
|
+
|
|
22
|
+
function readBin() {
|
|
23
|
+
return fs.readFileSync(BIN, 'utf8');
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
/** Parse the authoritative `const SUBCOMMANDS = new Set([...])` literal. */
|
|
27
|
+
function readSubcommands(src) {
|
|
28
|
+
const m = src.match(/const SUBCOMMANDS = new Set\(\[([\s\S]*?)\]\)/);
|
|
29
|
+
if (!m) throw new Error('Could not locate SUBCOMMANDS set in bin/quilltap.js');
|
|
30
|
+
return m[1]
|
|
31
|
+
.split(',')
|
|
32
|
+
.map((s) => s.trim().replace(/^['"]|['"]$/g, ''))
|
|
33
|
+
.filter(Boolean);
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
const SRC = readBin();
|
|
37
|
+
const SUBCOMMANDS = readSubcommands(SRC);
|
|
38
|
+
|
|
39
|
+
describe('CLI subcommand surface stays documented', () => {
|
|
40
|
+
it('parses a non-trivial subcommand set from bin/quilltap.js', () => {
|
|
41
|
+
expect(SUBCOMMANDS).toContain('db');
|
|
42
|
+
expect(SUBCOMMANDS).toContain('maintenance');
|
|
43
|
+
expect(SUBCOMMANDS.length).toBeGreaterThanOrEqual(10);
|
|
44
|
+
});
|
|
45
|
+
|
|
46
|
+
it.each(['bash', 'zsh', 'fish'])(
|
|
47
|
+
'%s completion template lists every top-level subcommand',
|
|
48
|
+
(shell) => {
|
|
49
|
+
const tpl = fs.readFileSync(path.join(COMPLETION_DIR, `${shell}.template`), 'utf8');
|
|
50
|
+
const missing = SUBCOMMANDS.filter((sub) => !tpl.includes(sub));
|
|
51
|
+
expect(missing).toEqual([]);
|
|
52
|
+
}
|
|
53
|
+
);
|
|
54
|
+
|
|
55
|
+
it('top-level --help lists every subcommand', () => {
|
|
56
|
+
const help = SRC.match(/function printHelp\(\) \{([\s\S]*?)\n\}/);
|
|
57
|
+
expect(help).toBeTruthy();
|
|
58
|
+
const missing = SUBCOMMANDS.filter((sub) => !help[1].includes(sub));
|
|
59
|
+
expect(missing).toEqual([]);
|
|
60
|
+
});
|
|
61
|
+
});
|
|
@@ -15,7 +15,7 @@ _quilltap_complete() {
|
|
|
15
15
|
local global_opts="-d --data-dir -i --instance -p --port -o --open -v --version -h --help --update --passphrase"
|
|
16
16
|
|
|
17
17
|
# Top-level subcommands
|
|
18
|
-
local top_cmds="db docs themes instances memories memory-diff logs migrations completion"
|
|
18
|
+
local top_cmds="db docs themes instances memories memory-diff logs migrations maintenance file-verify completion"
|
|
19
19
|
|
|
20
20
|
# Get the subcommand (first non-option word after quilltap)
|
|
21
21
|
local subcommand=""
|
|
@@ -126,7 +126,7 @@ _quilltap_complete() {
|
|
|
126
126
|
fi
|
|
127
127
|
;;
|
|
128
128
|
docs)
|
|
129
|
-
local docs_verbs="list show files ls dir tree read export scan write delete mkdir move copy status find grep reindex embed"
|
|
129
|
+
local docs_verbs="list show files ls dir tree read export scan write delete mkdir move copy link rmdir mvdir status find grep reindex embed"
|
|
130
130
|
local docs_flags="--mount --instance --data-dir --passphrase --port --json --help \
|
|
131
131
|
--force --rendered --links --folder --type --ext --limit --max --context --top --threshold \
|
|
132
132
|
--ignore-case -l --wait -R --recursive --sort -r --reverse --depth --max-nodes --long --semantic"
|
|
@@ -227,6 +227,19 @@ _quilltap_complete() {
|
|
|
227
227
|
COMPREPLY=($(compgen -W "$mig_flags" -- "$cur"))
|
|
228
228
|
fi
|
|
229
229
|
;;
|
|
230
|
+
maintenance)
|
|
231
|
+
local maint_verbs="status run"
|
|
232
|
+
local maint_flags="--instance --data-dir --passphrase --json --help"
|
|
233
|
+
if [[ -z "$subverb" ]]; then
|
|
234
|
+
if [[ "$cur" == -* ]]; then
|
|
235
|
+
COMPREPLY=($(compgen -W "$maint_flags" -- "$cur"))
|
|
236
|
+
else
|
|
237
|
+
COMPREPLY=($(compgen -W "$maint_verbs" -- "$cur"))
|
|
238
|
+
fi
|
|
239
|
+
else
|
|
240
|
+
COMPREPLY=($(compgen -W "$maint_flags" -- "$cur"))
|
|
241
|
+
fi
|
|
242
|
+
;;
|
|
230
243
|
completion)
|
|
231
244
|
if [[ "$cur" == -* ]]; then
|
|
232
245
|
COMPREPLY=($(compgen -W "--help" -- "$cur"))
|
|
@@ -15,7 +15,7 @@ function __quilltap_no_subcommand
|
|
|
15
15
|
end
|
|
16
16
|
for i in (seq 2 (count $cmd))
|
|
17
17
|
switch $cmd[$i]
|
|
18
|
-
case db docs themes instances memories memory-diff logs migrations completion
|
|
18
|
+
case db docs themes instances memories memory-diff logs migrations maintenance file-verify completion
|
|
19
19
|
return 1
|
|
20
20
|
end
|
|
21
21
|
end
|
|
@@ -27,7 +27,7 @@ function __quilltap_using_subcommand
|
|
|
27
27
|
set -l cmd (commandline -opc)
|
|
28
28
|
for i in (seq 2 (count $cmd))
|
|
29
29
|
switch $cmd[$i]
|
|
30
|
-
case db docs themes instances memories memory-diff logs migrations completion
|
|
30
|
+
case db docs themes instances memories memory-diff logs migrations maintenance file-verify completion
|
|
31
31
|
test "$cmd[$i]" = "$target"
|
|
32
32
|
return $status
|
|
33
33
|
end
|
|
@@ -57,6 +57,8 @@ complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'memories' -d 'Search a
|
|
|
57
57
|
complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'memory-diff' -d 'Memory extraction dry-run'
|
|
58
58
|
complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'logs' -d 'Tail or print log files'
|
|
59
59
|
complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'migrations' -d 'Inspect migration status'
|
|
60
|
+
complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'maintenance' -d 'Run retention/cleanup sweeps'
|
|
61
|
+
complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'file-verify' -d 'Force-download cloud-evicted data files'
|
|
60
62
|
complete -c quilltap -n '__quilltap_no_subcommand' -f -a 'completion' -d 'Generate shell completion'
|
|
61
63
|
|
|
62
64
|
# Global options
|
|
@@ -140,6 +142,9 @@ complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'delete' -d 'De
|
|
|
140
142
|
complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'mkdir' -d 'Create folder'
|
|
141
143
|
complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'move' -d 'Move file'
|
|
142
144
|
complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'copy' -d 'Copy file'
|
|
145
|
+
complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'link' -d 'Hard-link a file into another mount'
|
|
146
|
+
complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'rmdir' -d 'Remove a folder'
|
|
147
|
+
complete -c quilltap -n '__quilltap_using_subcommand docs' -f -a 'mvdir' -d 'Rename or move a folder'
|
|
143
148
|
|
|
144
149
|
# docs flags
|
|
145
150
|
complete -c quilltap -n '__quilltap_using_subcommand docs' -l 'mount' -d 'Mount name or id' -x
|
|
@@ -268,6 +273,10 @@ complete -c quilltap -n '__quilltap_using_subcommand migrations' -f -a 'pending'
|
|
|
268
273
|
complete -c quilltap -n '__quilltap_using_subcommand migrations' -f -a 'run' -d 'Run pending migrations'
|
|
269
274
|
complete -c quilltap -n '__quilltap_using_subcommand migrations' -l 'dry-run' -d 'Dry run (required for run)'
|
|
270
275
|
|
|
276
|
+
# ---------- maintenance verbs ----------
|
|
277
|
+
complete -c quilltap -n '__quilltap_using_subcommand maintenance' -f -a 'status' -d 'Show dry-run retention counts'
|
|
278
|
+
complete -c quilltap -n '__quilltap_using_subcommand maintenance' -f -a 'run' -d 'Run retention/cleanup sweeps (lock-gated)'
|
|
279
|
+
|
|
271
280
|
# ---------- completion sub-args ----------
|
|
272
281
|
complete -c quilltap -n '__quilltap_using_subcommand completion' -f -a 'bash' -d 'Bash completion'
|
|
273
282
|
complete -c quilltap -n '__quilltap_using_subcommand completion' -f -a 'zsh' -d 'Zsh completion'
|
|
@@ -28,6 +28,8 @@ _quilltap() {
|
|
|
28
28
|
'memory-diff:Dump existing memories and dry-run re-extraction'
|
|
29
29
|
'logs:Tail or print an instance log file'
|
|
30
30
|
'migrations:Inspect migration status'
|
|
31
|
+
'maintenance:Run retention/cleanup sweeps'
|
|
32
|
+
'file-verify:Force-download cloud-evicted data files'
|
|
31
33
|
'completion:Generate shell completion scripts'
|
|
32
34
|
)
|
|
33
35
|
|
|
@@ -76,6 +78,9 @@ _quilltap_subcommand() {
|
|
|
76
78
|
migrations)
|
|
77
79
|
_quilltap_migrations
|
|
78
80
|
;;
|
|
81
|
+
maintenance)
|
|
82
|
+
_quilltap_maintenance
|
|
83
|
+
;;
|
|
79
84
|
completion)
|
|
80
85
|
_quilltap_completion
|
|
81
86
|
;;
|
|
@@ -163,6 +168,9 @@ _quilltap_docs() {
|
|
|
163
168
|
'mkdir:Create folder'
|
|
164
169
|
'move:Move file'
|
|
165
170
|
'copy:Copy file'
|
|
171
|
+
'link:Hard-link a file into another mount'
|
|
172
|
+
'rmdir:Remove a folder'
|
|
173
|
+
'mvdir:Rename or move a folder'
|
|
166
174
|
)
|
|
167
175
|
|
|
168
176
|
docs_opts=(
|
|
@@ -395,6 +403,27 @@ _quilltap_migrations() {
|
|
|
395
403
|
_arguments $mig_opts
|
|
396
404
|
}
|
|
397
405
|
|
|
406
|
+
_quilltap_maintenance() {
|
|
407
|
+
local -a subverbs maint_opts
|
|
408
|
+
subverbs=(
|
|
409
|
+
'status:Show dry-run retention counts'
|
|
410
|
+
'run:Run retention/cleanup sweeps (lock-gated)'
|
|
411
|
+
)
|
|
412
|
+
|
|
413
|
+
maint_opts=(
|
|
414
|
+
'(-i --instance)'{-i,--instance}'[Registered instance name]:instance:_quilltap_instance_names'
|
|
415
|
+
'(-d --data-dir)'{-d,--data-dir}'[Data directory]:directory:_directories'
|
|
416
|
+
'--passphrase[Database passphrase]:passphrase:'
|
|
417
|
+
'--json[JSON output]'
|
|
418
|
+
'(-h --help)'{-h,--help}'[Show help]'
|
|
419
|
+
)
|
|
420
|
+
|
|
421
|
+
if (( CURRENT == 2 )); then
|
|
422
|
+
_describe 'maintenance subcommand' subverbs
|
|
423
|
+
fi
|
|
424
|
+
_arguments $maint_opts
|
|
425
|
+
}
|
|
426
|
+
|
|
398
427
|
_quilltap_completion() {
|
|
399
428
|
_values 'shell' bash zsh fish
|
|
400
429
|
}
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* `quilltap file-verify` — force-download an instance's cloud-evicted data files.
|
|
5
|
+
*
|
|
6
|
+
* An instance can live in a cloud-synced folder (iCloud Drive on macOS today;
|
|
7
|
+
* OneDrive / Google Drive File Stream on Windows later). Those providers EVICT
|
|
8
|
+
* idle files to dataless placeholders. If a database file is still dataless when
|
|
9
|
+
* SQLite/SQLCipher opens it, the read fails with "file is not a database" or
|
|
10
|
+
* returns partially-materialized garbage — which has wedged whole startups.
|
|
11
|
+
*
|
|
12
|
+
* The server does this automatically at boot (Phase -1, before the .dbkey is
|
|
13
|
+
* read). This command is the manual/diagnostic twin for the CLI, which opens the
|
|
14
|
+
* encrypted databases directly and benefits from the same guarantee. It needs no
|
|
15
|
+
* passphrase and never decrypts anything — it just reads the bytes to nowhere,
|
|
16
|
+
* which is what faults a dataless file in.
|
|
17
|
+
*
|
|
18
|
+
* Logic MIRRORS the TypeScript source of truth at
|
|
19
|
+
* `lib/startup/materialize-cloud-files.ts`. Keep them in sync.
|
|
20
|
+
*/
|
|
21
|
+
|
|
22
|
+
const fs = require('fs');
|
|
23
|
+
const path = require('path');
|
|
24
|
+
const { resolveDataDirAndPassphrase, printDefaultInstanceHint } = require('./db-helpers');
|
|
25
|
+
|
|
26
|
+
const DEFAULT_STALL_MS = 30_000;
|
|
27
|
+
const READ_CHUNK_BYTES = 8 * 1024 * 1024;
|
|
28
|
+
|
|
29
|
+
// ---------- detection (platform seam) ----------
|
|
30
|
+
|
|
31
|
+
const STAT_BLOCK_BYTES = 512;
|
|
32
|
+
|
|
33
|
+
// macOS not-fully-materialized heuristic: a real file (size > 0) whose locally
|
|
34
|
+
// allocated blocks hold fewer bytes than its size has not finished downloading.
|
|
35
|
+
// Covers both the fully-evicted placeholder (blocks === 0, mirrors SF_DATALESS)
|
|
36
|
+
// and the partially-materialized file (blocks > 0 but blocks*512 < size) — the
|
|
37
|
+
// latter still fails SQLite open with "file is not a database". A fully-resident
|
|
38
|
+
// file always reports blocks*512 >= size (allocation rounds up), so this never
|
|
39
|
+
// flags a healthy file. Zero-byte files are never flagged. Other platforms fall
|
|
40
|
+
// through as a no-op for now.
|
|
41
|
+
function isDatalessStat(stat) {
|
|
42
|
+
return stat.size > 0 && stat.blocks * STAT_BLOCK_BYTES < stat.size;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
function listTopLevelFiles(dataDir) {
|
|
46
|
+
let entries;
|
|
47
|
+
try {
|
|
48
|
+
entries = fs.readdirSync(dataDir, { withFileTypes: true });
|
|
49
|
+
} catch {
|
|
50
|
+
return [];
|
|
51
|
+
}
|
|
52
|
+
const out = [];
|
|
53
|
+
for (const entry of entries) {
|
|
54
|
+
if (!entry.isFile()) continue; // skip directories (backups/) and symlinks
|
|
55
|
+
const full = path.join(dataDir, entry.name);
|
|
56
|
+
let stat;
|
|
57
|
+
try {
|
|
58
|
+
stat = fs.statSync(full);
|
|
59
|
+
} catch {
|
|
60
|
+
continue;
|
|
61
|
+
}
|
|
62
|
+
out.push({ name: entry.name, full, size: stat.size, dataless: isDatalessStat(stat) });
|
|
63
|
+
}
|
|
64
|
+
return out;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
// ---------- streaming materialize with per-chunk stall guard ----------
|
|
68
|
+
|
|
69
|
+
function streamMaterialize(filePath, stallMs) {
|
|
70
|
+
return new Promise((resolve, reject) => {
|
|
71
|
+
const stream = fs.createReadStream(filePath, { highWaterMark: READ_CHUNK_BYTES });
|
|
72
|
+
let timer;
|
|
73
|
+
const arm = () => {
|
|
74
|
+
clearTimeout(timer);
|
|
75
|
+
timer = setTimeout(() => {
|
|
76
|
+
stream.destroy(new Error(`stalled — no data for ${Math.round(stallMs / 1000)}s`));
|
|
77
|
+
}, stallMs);
|
|
78
|
+
};
|
|
79
|
+
arm();
|
|
80
|
+
stream.on('data', () => arm()); // discard chunk; the read faults the bytes in
|
|
81
|
+
stream.on('end', () => {
|
|
82
|
+
clearTimeout(timer);
|
|
83
|
+
resolve();
|
|
84
|
+
});
|
|
85
|
+
stream.on('error', (err) => {
|
|
86
|
+
clearTimeout(timer);
|
|
87
|
+
reject(err);
|
|
88
|
+
});
|
|
89
|
+
});
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// ---------- argument parsing ----------
|
|
93
|
+
|
|
94
|
+
function parseFlags(args) {
|
|
95
|
+
const flags = {
|
|
96
|
+
dataDir: '',
|
|
97
|
+
instance: '',
|
|
98
|
+
all: false,
|
|
99
|
+
json: false,
|
|
100
|
+
help: false,
|
|
101
|
+
stallMs: DEFAULT_STALL_MS,
|
|
102
|
+
};
|
|
103
|
+
let i = 0;
|
|
104
|
+
while (i < args.length) {
|
|
105
|
+
const a = args[i];
|
|
106
|
+
switch (a) {
|
|
107
|
+
case '-d': case '--data-dir': flags.dataDir = args[++i]; break;
|
|
108
|
+
case '-i': case '--instance': flags.instance = args[++i]; break;
|
|
109
|
+
case '--all': flags.all = true; break;
|
|
110
|
+
case '--json': flags.json = true; break;
|
|
111
|
+
case '--stall-ms': flags.stallMs = Number(args[++i]) || DEFAULT_STALL_MS; break;
|
|
112
|
+
case '-h': case '--help': flags.help = true; break;
|
|
113
|
+
default:
|
|
114
|
+
if (a.startsWith('-')) console.error(`unknown flag: ${a}`);
|
|
115
|
+
break;
|
|
116
|
+
}
|
|
117
|
+
i++;
|
|
118
|
+
}
|
|
119
|
+
return flags;
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
function printHelp() {
|
|
123
|
+
console.log(`Usage: quilltap file-verify [options]
|
|
124
|
+
|
|
125
|
+
Force-download the instance's cloud-evicted (dataless) data files so the
|
|
126
|
+
databases are fully local before anything opens them. Reads each placeholder to
|
|
127
|
+
nowhere, which faults it in through the cloud provider (iCloud Drive, etc.).
|
|
128
|
+
Safe to run repeatedly; a no-op when nothing is evicted. macOS only for now.
|
|
129
|
+
|
|
130
|
+
Only the TOP-LEVEL files of the data directory are considered (the backups/
|
|
131
|
+
subdirectory is left alone).
|
|
132
|
+
|
|
133
|
+
Options:
|
|
134
|
+
-d, --data-dir <path> Use a specific data directory (instance root)
|
|
135
|
+
-i, --instance <name> Use a named instance
|
|
136
|
+
--all Read every top-level file, not just dataless ones
|
|
137
|
+
--stall-ms <ms> Treat a download as stalled after this many ms with
|
|
138
|
+
no bytes (per-chunk, not per-file; default ${DEFAULT_STALL_MS})
|
|
139
|
+
--json Output as JSON
|
|
140
|
+
-h, --help Show this help message
|
|
141
|
+
|
|
142
|
+
Examples:
|
|
143
|
+
quilltap file-verify --instance Ignite
|
|
144
|
+
quilltap file-verify --instance Friday --json
|
|
145
|
+
quilltap file-verify -d ~/iCloud/Quilltap/Ignite
|
|
146
|
+
`);
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
// ---------- main entry point ----------
|
|
150
|
+
|
|
151
|
+
async function fileVerifyCommand(args) {
|
|
152
|
+
const flags = parseFlags(args);
|
|
153
|
+
if (flags.help) {
|
|
154
|
+
printHelp();
|
|
155
|
+
return;
|
|
156
|
+
}
|
|
157
|
+
|
|
158
|
+
let resolved;
|
|
159
|
+
try {
|
|
160
|
+
resolved = resolveDataDirAndPassphrase({ dataDir: flags.dataDir, instance: flags.instance });
|
|
161
|
+
} catch (err) {
|
|
162
|
+
console.error(`Error: ${err.message}`);
|
|
163
|
+
process.exit(1);
|
|
164
|
+
}
|
|
165
|
+
printDefaultInstanceHint(resolved);
|
|
166
|
+
// resolved.dataDir already points at the instance's data/ directory.
|
|
167
|
+
const targetDir = resolved.dataDir;
|
|
168
|
+
|
|
169
|
+
const files = listTopLevelFiles(targetDir);
|
|
170
|
+
const targets = flags.all ? files : files.filter((f) => f.dataless);
|
|
171
|
+
|
|
172
|
+
const summary = {
|
|
173
|
+
dataDir: targetDir,
|
|
174
|
+
checked: files.length,
|
|
175
|
+
targeted: targets.length,
|
|
176
|
+
downloaded: 0,
|
|
177
|
+
failed: 0,
|
|
178
|
+
failedNames: [],
|
|
179
|
+
};
|
|
180
|
+
|
|
181
|
+
if (targets.length === 0) {
|
|
182
|
+
if (flags.json) {
|
|
183
|
+
console.log(JSON.stringify(summary, null, 2));
|
|
184
|
+
} else {
|
|
185
|
+
console.log(
|
|
186
|
+
`Nothing to fetch — all ${files.length} top-level file(s) in ${targetDir} are already ashore.`,
|
|
187
|
+
);
|
|
188
|
+
}
|
|
189
|
+
return;
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
if (!flags.json) {
|
|
193
|
+
console.log(
|
|
194
|
+
`Fetching ${targets.length} file(s) down from the cloud (of ${files.length} top-level):`,
|
|
195
|
+
);
|
|
196
|
+
}
|
|
197
|
+
|
|
198
|
+
for (let i = 0; i < targets.length; i++) {
|
|
199
|
+
const file = targets[i];
|
|
200
|
+
const mb = (file.size / (1024 * 1024)).toFixed(1);
|
|
201
|
+
if (!flags.json) {
|
|
202
|
+
process.stdout.write(` Coaxing «${file.name}» (${mb} MB) down — ${i + 1}/${targets.length}… `);
|
|
203
|
+
}
|
|
204
|
+
try {
|
|
205
|
+
await streamMaterialize(file.full, flags.stallMs);
|
|
206
|
+
summary.downloaded++;
|
|
207
|
+
if (!flags.json) console.log('secured.');
|
|
208
|
+
} catch (err) {
|
|
209
|
+
summary.failed++;
|
|
210
|
+
summary.failedNames.push(file.name);
|
|
211
|
+
if (!flags.json) console.log(`failed (${err.message}).`);
|
|
212
|
+
}
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
if (flags.json) {
|
|
216
|
+
console.log(JSON.stringify(summary, null, 2));
|
|
217
|
+
} else {
|
|
218
|
+
console.log(
|
|
219
|
+
`Done — ${summary.downloaded} secured, ${summary.failed} failed.` +
|
|
220
|
+
(summary.failed ? ` Failed: ${summary.failedNames.join(', ')}` : ''),
|
|
221
|
+
);
|
|
222
|
+
}
|
|
223
|
+
|
|
224
|
+
if (summary.failed > 0) process.exit(1);
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
module.exports = {
|
|
228
|
+
fileVerifyCommand,
|
|
229
|
+
};
|