@awebai/oats 0.61.0 → 0.63.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/bin/oats.mjs +27 -12
- package/docs/capabilities.md +1 -1
- package/docs/desktop-cli-api.md +30 -2
- package/docs/execution-targets.md +89 -4
- package/docs/implementation.md +2 -1
- package/docs/named-test-runs.md +382 -0
- package/docs/official-catalog.md +1 -1
- package/docs/packages.md +5 -5
- package/docs/release-lane.md +13 -0
- package/docs/release-notes/v0.62.0.md +223 -0
- package/docs/release-notes/v0.63.0.md +135 -0
- package/docs/souls-and-instances.md +6 -1
- package/docs/upgrading.md +135 -11
- package/docs/workspaces.md +1 -1
- package/lib/core.mjs +73 -23
- package/lib/doctor-capture.mjs +35 -0
- package/lib/session-viewer.mjs +86 -7
- package/lib/tree-copy.mjs +188 -7
- package/package-catalog.json +1 -1
- package/package.json +1 -1
- package/packages/client/cli-locator.mjs +2 -2
- package/packages/record/README.md +87 -13
- package/packages/record/bin/capture-hook.mjs +8 -0
- package/packages/record/bin/capture.mjs +200 -139
- package/packages/record/bin/recall.mjs +8 -3
- package/packages/record/bin/setup.mjs +116 -225
- package/packages/record/bin/turn-record.mjs +2 -2
- package/packages/record/docs/capture-service.md +1527 -28
- package/packages/record/docs/capture-status.schema.json +429 -12
- package/packages/record/lib/capture-admission.mjs +82 -0
- package/packages/record/lib/capture-aw-unit.mjs +137 -0
- package/packages/record/lib/capture-aw.mjs +31 -12
- package/packages/record/lib/capture-budget.mjs +31 -0
- package/packages/record/lib/capture-cc.mjs +39 -6
- package/packages/record/lib/capture-cleanup-queue.mjs +135 -0
- package/packages/record/lib/capture-cleanup-work.mjs +77 -0
- package/packages/record/lib/capture-command-operation.mjs +37 -0
- package/packages/record/lib/capture-command-output.mjs +62 -0
- package/packages/record/lib/capture-command-worker.mjs +6 -0
- package/packages/record/lib/capture-command.mjs +65 -0
- package/packages/record/lib/capture-completion.mjs +152 -0
- package/packages/record/lib/capture-context.mjs +50 -0
- package/packages/record/lib/capture-default-roots.mjs +30 -0
- package/packages/record/lib/capture-discovery.mjs +114 -0
- package/packages/record/lib/capture-failures.mjs +45 -0
- package/packages/record/lib/capture-health-doctor.mjs +13 -0
- package/packages/record/lib/capture-health-installation.mjs +117 -0
- package/packages/record/lib/capture-health.mjs +116 -0
- package/packages/record/lib/capture-hook-ownership.mjs +58 -0
- package/packages/record/lib/capture-hook-runner.mjs +17 -0
- package/packages/record/lib/capture-hook.mjs +26 -0
- package/packages/record/lib/capture-index-inventory.mjs +205 -0
- package/packages/record/lib/capture-index-outcomes.mjs +113 -0
- package/packages/record/lib/capture-index-reset.mjs +110 -0
- package/packages/record/lib/capture-index-work.mjs +77 -0
- package/packages/record/lib/capture-issue-preparation.mjs +73 -0
- package/packages/record/lib/capture-issues.mjs +15 -2
- package/packages/record/lib/capture-lock.mjs +137 -17
- package/packages/record/lib/capture-native-unit.mjs +322 -0
- package/packages/record/lib/capture-pass.mjs +116 -0
- package/packages/record/lib/capture-recovery-snapshot.mjs +182 -0
- package/packages/record/lib/capture-retained-scratch.mjs +213 -0
- package/packages/record/lib/capture-retry.mjs +102 -0
- package/packages/record/lib/capture-run.mjs +125 -0
- package/packages/record/lib/capture-scratch.mjs +12 -3
- package/packages/record/lib/capture-service-definition.mjs +104 -0
- package/packages/record/lib/capture-service-install.mjs +362 -0
- package/packages/record/lib/capture-service-ownership.mjs +137 -0
- package/packages/record/lib/capture-service-runner.mjs +18 -0
- package/packages/record/lib/capture-small-snapshot.mjs +85 -0
- package/packages/record/lib/capture-source-discovery.mjs +319 -0
- package/packages/record/lib/capture-source-queue.mjs +199 -0
- package/packages/record/lib/capture-source-work.mjs +85 -0
- package/packages/record/lib/capture-state.mjs +16 -71
- package/packages/record/lib/capture-status.mjs +171 -0
- package/packages/record/lib/capture-supervisor.mjs +136 -0
- package/packages/record/lib/capture-watch.mjs +32 -0
- package/packages/record/lib/capture-work-session.mjs +241 -0
- package/packages/record/lib/capture-worker.mjs +74 -0
- package/packages/record/lib/derived-state.mjs +58 -17
- package/packages/record/lib/file-lines.mjs +22 -6
- package/packages/record/lib/formats.mjs +4 -3
- package/packages/record/lib/index-capture.mjs +97 -0
- package/packages/record/lib/index-db.mjs +121 -66
- package/packages/record/lib/index-reset.mjs +19 -0
- package/packages/record/lib/store.mjs +46 -14
- package/packages/record/lib/transcript-prefix.mjs +125 -0
- package/skills/oats-getting-started/SKILL.md +1 -1
- package/docs/macos-test-runs.md +0 -246
|
@@ -1,8 +1,20 @@
|
|
|
1
1
|
# Capture algorithms and recovery
|
|
2
2
|
|
|
3
3
|
The record package owns source cursors, journal validation, indexing and capture
|
|
4
|
-
status.
|
|
5
|
-
|
|
4
|
+
status. Ordinary/source-selected capture and explicit retry use the fixed
|
|
5
|
+
supervised worker. Explicit hooks launch a detached supervisor. Setup migrates
|
|
6
|
+
supported service definitions through the preflight/apply gate described below;
|
|
7
|
+
the package also owns offline installation/receipt/issue diagnosis. Root
|
|
8
|
+
registration and real-manager qualification remain separate. Deployment doctor
|
|
9
|
+
delegates to the package diagnosis through the bounded command boundary below.
|
|
10
|
+
|
|
11
|
+
The integration described below is service A, which ships in the next release
|
|
12
|
+
after 0.61.0; source and fake manager fixtures do not establish real manager
|
|
13
|
+
qualification. OATS 0.56.0 through 0.61.0 shipped without the bounded periodic
|
|
14
|
+
service. See
|
|
15
|
+
[capture service release order](../../../docs/upgrading.md#capture-service-release-order)
|
|
16
|
+
for the distinction between upgrading those releases and a future setup
|
|
17
|
+
conversion after installing a release that ships service A.
|
|
6
18
|
|
|
7
19
|
## Input contract
|
|
8
20
|
|
|
@@ -39,6 +51,15 @@ Detected changes have two outcomes:
|
|
|
39
51
|
from the physical last transcript turn, so appending a revised earlier line
|
|
40
52
|
would make those writers duplicate the remaining history.
|
|
41
53
|
|
|
54
|
+
Protected command prefix-conflict diagnostics name the refused input, conflicting stream
|
|
55
|
+
and native line/source byte position. They say "establishing source path
|
|
56
|
+
unavailable": the journal records native session and line, not that path.
|
|
57
|
+
Current discovery and derived cursors cannot supply missing provenance, and a
|
|
58
|
+
prefix mismatch does not itself prove that a second source exists. Native stream
|
|
59
|
+
identity and traversal precedence remain unchanged.
|
|
60
|
+
Ordinary supervised passes report confirmed appended counts and the worker
|
|
61
|
+
outcome; `capture --health` supplies active set-aside locations and remedies.
|
|
62
|
+
|
|
42
63
|
## Operator recovery
|
|
43
64
|
|
|
44
65
|
`turn-record capture --reset-cursors [--source aw|cc|pi|codex]` uses the selected
|
|
@@ -60,10 +81,50 @@ For a conflicting native prefix, either:
|
|
|
60
81
|
input from active set-aside entries. Removing the ignore rule later does not
|
|
61
82
|
authorize the conflict.
|
|
62
83
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
84
|
+
**Oversized records.** The 8 MiB raw-record limit applies to capture beyond
|
|
85
|
+
the captured prefix. The supervised worker never allocates, captures or
|
|
86
|
+
modifies a new raw source record larger than 8 MiB. It appends and checkpoints
|
|
87
|
+
the complete records before that record through the normal unit commit, then
|
|
88
|
+
sets the input aside (`record-too-large`) at the oversized record's start
|
|
89
|
+
offset. Records after it stay pending until the operator edits the source or
|
|
90
|
+
adds the whole input to `<root>/ignore`, which, as above, also stops later
|
|
91
|
+
turns from that session. `--retry-set-aside` revalidates the input, appending
|
|
92
|
+
no captured record twice, but the oversized record still blocks it.
|
|
93
|
+
|
|
94
|
+
Records already captured (by a release without this limit) are not refused.
|
|
95
|
+
Staged recovery (after cache loss, a reset, a retry or a rewritten source)
|
|
96
|
+
copies the source in chunks without joining lines, and proves the captured
|
|
97
|
+
prefix by a SHA-256 digest of each staged line streamed over those chunks,
|
|
98
|
+
compared with the journal's line; a captured line of any size is compared
|
|
99
|
+
without being held whole, and a changed byte refuses the prefix as usual. The
|
|
100
|
+
journal side of that proof, like exact-version journal validation, still reads
|
|
101
|
+
each journal record whole: the largest captured journal record bounds recovery
|
|
102
|
+
memory, which the large-frame qualification measures in this recovery shape.
|
|
103
|
+
An AW comm log has no prefix proof: when its cursor stops qualifying (a new
|
|
104
|
+
journal epoch, a reset, a retry, or an older release's cursor), the rescan keeps
|
|
105
|
+
the cursor's byte offset as a captured frontier, provided the cursor names the
|
|
106
|
+
same file and its boundary witness still matches. Records before that frontier
|
|
107
|
+
are read without the limit, journal-id dedupe keeps them from being appended
|
|
108
|
+
twice, and the limit applies at and beyond it. While a rescan is below its
|
|
109
|
+
frontier, the cursor it writes carries the frontier and that offset's own
|
|
110
|
+
witness, so a second invalidation between the rescan's units keeps the larger
|
|
111
|
+
of the two proofs. A replaced or rewritten log has no frontier. That is the
|
|
112
|
+
price of never refusing a captured record: an already-captured AW record too
|
|
113
|
+
large for the worker's 256 MiB heap fails the worker on every rescan
|
|
114
|
+
(`capture-pass-failed`), and that stream does not advance.
|
|
115
|
+
One native case still captures nothing before an oversized record: an
|
|
116
|
+
oversized line before the session's first timestamped line, which the
|
|
117
|
+
unstamped lines before it need.
|
|
118
|
+
`--health` gives each listed `record-too-large` entry this remedy (edit the
|
|
119
|
+
source, then retry; or ignore the whole input, discarding it whole), never the
|
|
120
|
+
prefix-restoration remedy above.
|
|
121
|
+
|
|
122
|
+
Retry takes the same nonblocking capture lock, prepares current entries for
|
|
123
|
+
revalidation in durable bounded pages, then performs supervised reconciliation
|
|
124
|
+
while retaining the lock across replacement workers. An interruption after
|
|
125
|
+
marking leaves the warning active. The run-bound preparation cursor prevents
|
|
126
|
+
repeating committed retry generations. Resetting cursors does not replace retry
|
|
127
|
+
or clear a refusal. Public synchronous retry/reset APIs keep their contracts.
|
|
67
128
|
|
|
68
129
|
`capture --status` keeps its existing store summary. `capture --health [--json]`
|
|
69
130
|
reads current active set-aside rows from `index/capture.db`, without capturing,
|
|
@@ -74,15 +135,254 @@ or acquire an implicit root capture lock. Health gets historical run fields from
|
|
|
74
135
|
the snapshot and never advances them from current issues. A database with no
|
|
75
136
|
snapshot still exposes its active issues.
|
|
76
137
|
|
|
138
|
+
A supervised pass that finds the snapshot unreadable, malformed or future-dated
|
|
139
|
+
treats it as absent: under the root lock and before any worker, it moves the
|
|
140
|
+
file aside, never overwriting it, to `capture-status.json.invalid-<UTC stamp>`
|
|
141
|
+
(the root directory synced after the rename). A foreground, setup, watch or
|
|
142
|
+
service pass names that path on stderr; a hook pass, whose output is detached,
|
|
143
|
+
leaves only the moved-aside file. A system error reading the snapshot
|
|
144
|
+
(permission, I/O) fails the pass and leaves the file in place.
|
|
145
|
+
The pass then publishes a new snapshot with no history: no success is carried
|
|
146
|
+
over or invented. One moved-aside copy is kept; a newer one replaces it. An
|
|
147
|
+
older moved-aside copy that is a directory is not removed: each later
|
|
148
|
+
move-aside reports it, and an operator removes it. A move that fails leaves the
|
|
149
|
+
file in place and fails the pass.
|
|
150
|
+
|
|
151
|
+
### Stopping capture and the lock owner record
|
|
152
|
+
|
|
153
|
+
SIGTERM and SIGINT stop a supervised pass (manual, watch, hook, setup, service
|
|
154
|
+
and retry) cooperatively. They are observed from before the root lock is taken
|
|
155
|
+
until after it is released. The first one ends admission: no further worker,
|
|
156
|
+
continuation or finalization starts. A worker not yet started is killed at
|
|
157
|
+
once. A started worker is asked to stop through its own lock owner record,
|
|
158
|
+
which it already rereads between bounded units, so it stops at its next unit
|
|
159
|
+
boundary and writes its own `failed`/`interrupted` receipt with the counts it
|
|
160
|
+
committed; prior `lastSuccess` is kept. A worker that has already sent its
|
|
161
|
+
result keeps it. The worker catches TERM/INT itself as well, so a signal sent
|
|
162
|
+
to the caller's whole process group (a terminal's Ctrl-C, `timeout`, a service
|
|
163
|
+
manager's control group) does not kill it mid-unit; it too ends the worker as
|
|
164
|
+
interrupted at its next check. Either way the worker gets one grace of 5 seconds, capped by
|
|
165
|
+
the pass's absolute deadline, and a worker still running then is killed and
|
|
166
|
+
reaped; later signals neither shorten nor extend the grace. A killed worker
|
|
167
|
+
leaves its unfinished `running` receipt, never a finished one, and the next
|
|
168
|
+
pass records it as `lastFailure` `interrupted` and continues from the durable
|
|
169
|
+
checkpoints. The command exits 1. An earlier failure cause observed in the
|
|
170
|
+
same pass is reported instead of the interruption. SIGKILL, or any death that
|
|
171
|
+
runs no cleanup, leaves the lock to the next pass's reclaim of a dead owner
|
|
172
|
+
(see below); there is no cleanup guarantee for it.
|
|
173
|
+
|
|
174
|
+
The exact commands, `--home`, `--file` and `--reset-cursors`, run in the
|
|
175
|
+
calling process, and their work is synchronous. They observe SIGTERM and SIGINT
|
|
176
|
+
only at safe points: before taking the lock, after taking it, before each
|
|
177
|
+
format's batch of exact files, before the index update, and once the work
|
|
178
|
+
under the lock (and the receipt summaries) is done, before any answer. Discovery, a whole
|
|
179
|
+
format's batch, the index update and the receipt summaries cannot be
|
|
180
|
+
preempted, so a signal waits for the segment it arrived in. No response time is
|
|
181
|
+
promised, and these commands have no deadline. An interruption at a safe point
|
|
182
|
+
releases the lock and answers the command's existing receipt, `failed` with the
|
|
183
|
+
error `capture interrupted by SIGTERM` (or SIGINT) and exit 1. `appended`
|
|
184
|
+
counts only the batches that completed, so it is confirmed; an interruption
|
|
185
|
+
observed after all of them says `after every batch completed`, and one after a
|
|
186
|
+
cursor reset says `after the cursor reset was committed`. These commands
|
|
187
|
+
write no run receipt, so a killed exact command leaves no `running` run behind:
|
|
188
|
+
the next pass reclaims its lock and records only `reclaimedLock`.
|
|
189
|
+
|
|
190
|
+
The lock's `owner.json` holds the owner's `pid`, a per-acquisition `nonce`,
|
|
191
|
+
`startedAt` and `host`, and, while a supervised pass runs, its `worker`'s pid,
|
|
192
|
+
nonce, start and host. A lock taken by this release also says who took it:
|
|
193
|
+
`command` (`pass`, `retry`, `home`, `file` or `reset-cursors`), `caller`
|
|
194
|
+
(`entry`, the program: `oats`, `turn-record`, `hook-runner` or
|
|
195
|
+
`service-runner`, left out when the command runs in another program; `trigger`,
|
|
196
|
+
from `manual`, `watch`, `hook`, `setup` and `service`) and `ppid`, the owner's immediate parent process. A cooperative stop
|
|
197
|
+
adds `stop: true`. These are fixed labels and process facts. No argv,
|
|
198
|
+
environment or shell text is recorded, and nothing names an external caller or
|
|
199
|
+
why it stopped the command. The fields are diagnostic only: liveness, release
|
|
200
|
+
and reclaim read only the pid, nonce, start, host and worker, so records from
|
|
201
|
+
earlier releases, without them, are read and reclaimed exactly as before.
|
|
202
|
+
|
|
203
|
+
### Offline installation and run diagnosis
|
|
204
|
+
|
|
205
|
+
The package owns `captureHealth` and `turn-record capture --health [--json]`.
|
|
206
|
+
Its existing `schemaVersion: 1` health envelope remains separate from the
|
|
207
|
+
[status file schema](capture-status.schema.json): `rows` contains `{code,message}`,
|
|
208
|
+
`lastRun`/`lastSuccess`/`lastFailure` retain historical receipt values, and
|
|
209
|
+
`backlog` is the current issue projection. A warning diagnosis exits zero;
|
|
210
|
+
an installation/issue reader failure exits nonzero, without an empty successful
|
|
211
|
+
projection. No store is constructed on the health CLI branch.
|
|
212
|
+
|
|
213
|
+
Capture CLI terminal branches return from a single command dispatch and retain
|
|
214
|
+
their exit code while Node drains stdout and stderr naturally. Health cannot
|
|
215
|
+
fall through into capture, including on reader failure. The same boundary
|
|
216
|
+
applies to usage/help, recovery, hook launch reporting, explicit-file receipts,
|
|
217
|
+
install hints, status and ordinary errors; exact-home receipts already finish
|
|
218
|
+
naturally. It adds no stream closure or timed delay. Privacy failures print once
|
|
219
|
+
and stop dispatch before admission; a later watch privacy failure closes future
|
|
220
|
+
watch scheduling. Lock cleanup and admitted-child ownership remain with their
|
|
221
|
+
existing owners; hook launch still unrefs its detached child, and foreground
|
|
222
|
+
watch retains its signal lifecycle. Output delivery is distinct from diagnosis
|
|
223
|
+
and durably confirmed processing counts.
|
|
224
|
+
|
|
225
|
+
Installation evidence means service definitions, hooks or setup's pending marker, any of:
|
|
226
|
+
|
|
227
|
+
- On macOS, `~/Library/LaunchAgents/ai.aweb.turn-record-capture.plist` parses as
|
|
228
|
+
a supported owned periodic or legacy definition. On Linux, the candidates
|
|
229
|
+
are `~/.config/systemd/user/turn-record-capture.service` and
|
|
230
|
+
`turn-record-capture.timer`. Health shares setup's strict template/argv
|
|
231
|
+
ownership reader; a basename, marker or mention alone is insufficient.
|
|
232
|
+
- A `settings.json` in a top-level `~/.claude*` directory or directory symlink
|
|
233
|
+
has a recognized command in a Stop or SessionEnd hook group. Health shares
|
|
234
|
+
setup's unchanged tokenizer: script in the second word ending in
|
|
235
|
+
`/capture-hook.mjs` (this release's hook entry) or `/capture.mjs` (earlier
|
|
236
|
+
releases' forms), whole `--owner` and `--quiet` words, supported plain/single
|
|
237
|
+
quoted words, and no shell expansion, operator or second command.
|
|
238
|
+
- Setup's pending-activation marker, `capture-service-pending.json`, is in the
|
|
239
|
+
selected root. Only setup writes it, so a setup that failed after writing it
|
|
240
|
+
is diagnosed (the marker's row) even with no definition or hook.
|
|
241
|
+
|
|
242
|
+
A record root alone is not an installation: one `oats recall` or one manual
|
|
243
|
+
capture creates it (with `index/`), and such a host has never set capture up.
|
|
244
|
+
The selected root (`--root`, then `TURN_RECORD_ROOT`, then `~/.turn-record`)
|
|
245
|
+
still has to be a directory when it exists. Readable foreign service files alone do not establish installation. No evidence
|
|
246
|
+
means `ok`, no warnings and no root creation. That includes a host that runs
|
|
247
|
+
capture by other means, or was set up with `--no-hooks --no-service`: health
|
|
248
|
+
reads `ok` and withholds its history and backlog, even when a status file or
|
|
249
|
+
set-aside inputs exist. Unreadable candidates, dangling
|
|
250
|
+
links, malformed hook settings, observed changes or exceeded reader limits are
|
|
251
|
+
unknown and fail diagnosis, even if another candidate establishes installation.
|
|
252
|
+
Only named files and the top level of the caller home are inspected; source
|
|
253
|
+
roots, journal content, network and managers are never visited. Definition bytes
|
|
254
|
+
show installation evidence, **not** registered/enabled scheduling. Even an `ok`
|
|
255
|
+
receipt diagnosis does not certify a manager schedule or a permanent service PID.
|
|
256
|
+
|
|
257
|
+
Installation reads accept regular files up to 1 MiB each, with 16 MiB total,
|
|
258
|
+
64 hook candidates, 4,096 home entries and 256 path components. These are private
|
|
259
|
+
fail-closed reader guards, not capture policy or truncation-based absence proof.
|
|
260
|
+
The reader uses a 15-second cooperative monotonic deadline between operations;
|
|
261
|
+
the isolated issue reader retains its hard 10-second child deadline. Synchronous
|
|
262
|
+
filesystem calls still depend on the host filesystem returning. Deployment
|
|
263
|
+
doctor supplies the separate finite command timeout described below; it does
|
|
264
|
+
not turn the synchronous library's cooperative deadline into a hard one.
|
|
265
|
+
|
|
266
|
+
Installed but absent status produces `capture-status-missing` with a setup
|
|
267
|
+
remedy; malformed, unsupported, unreadable or future-dated status produces
|
|
268
|
+
`capture-status-invalid` until the next supervised pass moves it aside and
|
|
269
|
+
writes a new status, as the setup remedy's pass does. Valid older nullable
|
|
270
|
+
receipts remain valid absence of run evidence. `capture-success-stale` means no recorded success or one **older
|
|
271
|
+
than** 3,600 seconds, not exactly that age. It names the last success and committed
|
|
272
|
+
progress, with an asleep/off-host caveat. Recent progress is not full success.
|
|
273
|
+
A failure newer than success and current unresolved stream failures produce
|
|
274
|
+
`capture-pass-failed` independently of current set-asides. An unfinished run
|
|
275
|
+
beyond its recorded deadline also warns, without inventing its termination cause.
|
|
276
|
+
Before that deadline, death evidence requires an unchanged local lock registration
|
|
277
|
+
whose worker PID matches the receipt and whose acquisition predates the run,
|
|
278
|
+
with both supervisor and worker positively dead. Missing/off-host/hostless,
|
|
279
|
+
changed or uncertain registration and live/reused PIDs supply no death proof.
|
|
280
|
+
The receipt is reread before accepting that observation; health never repairs,
|
|
281
|
+
reclaims, signals termination, writes counters or invents a finished receipt.
|
|
282
|
+
Owned legacy watcher definitions additionally produce `capture-setup-required`,
|
|
283
|
+
as does an owned capture hook whose entry is not this release's
|
|
284
|
+
`bin/capture-hook.mjs` (the `capture.mjs` forms earlier releases' setup wrote),
|
|
285
|
+
hooks-only installations included; the row names the settings files.
|
|
286
|
+
Until setup converts it, the hook line from before this release,
|
|
287
|
+
`<node> <package>/bin/capture.mjs --owner <owner> --quiet` run asynchronously
|
|
288
|
+
with a 120-second timeout, still works against this release: it is one
|
|
289
|
+
ordinary quiet foreground pass under the root lock, attributed `manual`, up to
|
|
290
|
+
its 300-second deadline. A harness that stops it at 120 seconds with SIGTERM or
|
|
291
|
+
SIGINT stops it cooperatively: the worker ends at its next unit boundary, or is
|
|
292
|
+
killed after a 5-second grace, the pass ends as interrupted and the lock is
|
|
293
|
+
released ([stopping capture](#stopping-capture-and-the-lock-owner-record)).
|
|
294
|
+
With SIGKILL the supervisor dies holding the lock, which the next pass reclaims
|
|
295
|
+
as a dead owner. Either way the next pass continues from the durable
|
|
296
|
+
checkpoints.
|
|
297
|
+
Setup's pending-activation marker (`<root>/capture-service-pending.json`, see
|
|
298
|
+
[setup service migration](#setup-service-migration)) produces
|
|
299
|
+
`capture-service-pending` from file evidence only: beside setup's own marked
|
|
300
|
+
definitions it says the periodic schedule's activation was not verified (hooks
|
|
301
|
+
may be installed, the schedule is not confirmed), and with no such definition
|
|
302
|
+
it names the marker as stale. Both say "not verified", never "not running": a
|
|
303
|
+
crash after a verified activation can leave the marker until the next setup
|
|
304
|
+
clears it. The remedy is a setup rerun; health never removes the marker. It is
|
|
305
|
+
a warning and hides no other row.
|
|
306
|
+
Standalone remedies name `turn-record`; the private doctor entry, and
|
|
307
|
+
`capture --health` when the kernel runs it (`oats capture --health`) or a node
|
|
308
|
+
was handed in (`OATS_NODE_BIN`, as for setup), select `oats` when constructing
|
|
309
|
+
authored remedy literals, before interpolating paths/errors or receipt data. It never replaces substrings in completed messages. All diagnosis
|
|
310
|
+
is local observed evidence, not full service qualification.
|
|
311
|
+
|
|
312
|
+
### Kernel doctor delegation
|
|
313
|
+
|
|
314
|
+
Deployment `oats doctor` (text or JSON) starts exactly one fixed package health
|
|
315
|
+
entry with `process.execPath`. `--soul` stays composition-only and starts none.
|
|
316
|
+
The kernel maps validated package `{code,message}` rows into its existing
|
|
317
|
+
`warnings` list; it does not read status, open the database, probe managers or
|
|
318
|
+
reimplement installation/freshness/issue rules. The package validates its
|
|
319
|
+
existing `schemaVersion: 1` envelope and reuses receipt/backlog validation.
|
|
320
|
+
Nullable historical values remain absence of evidence. No doctor schema field
|
|
321
|
+
or public capture option is added.
|
|
322
|
+
|
|
323
|
+
The child keeps the caller cwd and HOME selection; `TURN_RECORD_ROOT` selects
|
|
324
|
+
the root, otherwise `~/.turn-record` applies. A relative root is relative to the
|
|
325
|
+
calling process cwd, not doctor's deployment positional argument. Only the
|
|
326
|
+
reader's home/temp/cache/root environment and an explicit experimental SQLite
|
|
327
|
+
Node flag are forwarded; no inherited Node options, worker/executable override
|
|
328
|
+
or shell dispatch. The private entry constructs no store and exits naturally,
|
|
329
|
+
so queued JSON output can drain.
|
|
330
|
+
|
|
331
|
+
Transport uses a 20-second command timeout, SIGKILL on timeout/overflow and a
|
|
332
|
+
1 MiB output cap. These are private fail-closed guards, not a response-time
|
|
333
|
+
measurement or public tuning policy. The synchronous child transport waits for
|
|
334
|
+
its direct child to terminate. The package's isolated SQLite reader retains its
|
|
335
|
+
own 10-second timeout; if the outer reader is killed while that child is active,
|
|
336
|
+
its existing finite lifetime still applies. OS scheduling/filesystem response
|
|
337
|
+
and process termination remain host dependencies; no real-manager or storage
|
|
338
|
+
qualification follows from these source guards.
|
|
339
|
+
|
|
340
|
+
Timeout, launch/transport/reader failure, excessive output or invalid JSON/envelope
|
|
341
|
+
produces `capture-health-unavailable`, never an empty successful diagnosis or
|
|
342
|
+
rows reconstructed from partial output. Doctor retains its warning exit/output
|
|
343
|
+
contract; standalone package reader failures remain nonzero. Valid package
|
|
344
|
+
warnings are forwarded as diagnosis, not command failure. Text output applies
|
|
345
|
+
the kernel's terminal escaping; JSON retains literal values. Even no warnings
|
|
346
|
+
does not establish enabled/registered scheduling or a permanent service PID.
|
|
347
|
+
|
|
77
348
|
Both command snapshots and health use the same transactional count/sample
|
|
78
|
-
projection
|
|
79
|
-
|
|
349
|
+
projection. Source set-asides have `setAsideCount`, at most 100 deterministic
|
|
350
|
+
`setAside` entries and `truncated`. Operational failures independently have
|
|
351
|
+
`failedCount`, at most 100 `failed` entries and `failedTruncated`; neither category
|
|
352
|
+
borrows the other's sample allowance. Text output has the same bounds (at most
|
|
353
|
+
200 combined entries) and explains both source remedies. Absent state on a
|
|
80
354
|
never-installed root is not a warning or a reason to create files. A missing
|
|
81
355
|
current database cannot erase a known nonempty command snapshot. Malformed or
|
|
82
356
|
unreadable status is reported as `capture-status-invalid`; an unavailable current
|
|
83
357
|
issue reader fails the command rather than returning successful stale/empty
|
|
84
358
|
health JSON.
|
|
85
359
|
|
|
360
|
+
An operational failure entry contains `stream`, `offset`, `reason`, `attempts`
|
|
361
|
+
and `nextRetryAt`. The offset is the journal byte offset where the durably
|
|
362
|
+
admitted failed **single-stream unit starts**, never an exact inner-frame claim.
|
|
363
|
+
Text says "unit starting at byte X" and "failing frame unknown". An interrupted
|
|
364
|
+
multi-stream group supplies run failure evidence, not guessed rows for every
|
|
365
|
+
member. Private journal generation and admission evidence stay out of this
|
|
366
|
+
projection. Active entries produce `capture-pass-failed`, including during
|
|
367
|
+
backoff; they do not become source set-asides or advance a processing cursor.
|
|
368
|
+
|
|
369
|
+
The fixed retry schedule is 2, 4, 8, then 16 capture intervals (30, 60, 120, then
|
|
370
|
+
240 minutes, capped there). Only an actual failed attempt increments `attempts`
|
|
371
|
+
and sets a new date. Skipping a deferred unit does neither. Time passing, source
|
|
372
|
+
growth, cursor reset and `--retry-set-aside` do not resolve these failures or
|
|
373
|
+
override their schedule. Resolution requires committed progress past the failed
|
|
374
|
+
unit or validated reconciliation that it no longer needs work. A cooperative
|
|
375
|
+
stop that makes durable progress is ordinary partial work, not automatically a
|
|
376
|
+
failure. Old snapshots without these fields contain no recorded operational
|
|
377
|
+
failure evidence; their absence does not establish a successful pass.
|
|
378
|
+
|
|
379
|
+
An interrupted multi-stream group defers its members without assigning each a
|
|
380
|
+
failure. On the next due attempt, its members enter the normal queue singly,
|
|
381
|
+
after healthy work. A member's own failed admission then supplies its failure
|
|
382
|
+
row; successful members advance. There is no in-pass group retry or diagnostic
|
|
383
|
+
execution mode. Unknown attribution remains a run failure, with deferred work
|
|
384
|
+
keeping the backlog pending even when no member has an attributable failure row.
|
|
385
|
+
|
|
86
386
|
Reading an existing database requires Node's read-only SQLite support: 22.12+
|
|
87
387
|
on the 22.x line (22.13+ avoids needing `--experimental-sqlite`), 23.2+ on 23.x,
|
|
88
388
|
or a newer major. Normal capture retains its declared runtime floor. Missing
|
|
@@ -101,19 +401,65 @@ from releasing POSIX locks belonging to another SQLite connection in the caller.
|
|
|
101
401
|
Command snapshots reuse their owned connection for the shared projection.
|
|
102
402
|
|
|
103
403
|
The shared schema is [capture-status.schema.json](capture-status.schema.json).
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
404
|
+
Existing algorithm-phase snapshots retain null `policy`, `lastRun`,
|
|
405
|
+
`lastSuccess`, `lastFailure` and `lastProgressAt`: evidence has not been supplied.
|
|
406
|
+
They never mean a completed pass or prove service activation. The same version-1
|
|
407
|
+
schema accepts the fixed service policy and evidenced worker run/history fields;
|
|
408
|
+
no second status file is used. Run identities name their trigger, worker PID and
|
|
409
|
+
start; a running receipt has no finish/reason. Complete, partial and failed
|
|
410
|
+
receipts have a finish, and failure has an explicit reason. Success/failure
|
|
411
|
+
summaries retain run identity, finish, per-run counts and sampled peak RSS;
|
|
412
|
+
failure additionally has a safe message and optional path/offset. Future-dated,
|
|
413
|
+
malformed or unsupported evidence is invalid, not a freshness conclusion.
|
|
414
|
+
|
|
415
|
+
Failed/interrupted-run counters report only durably confirmed unit-accounting
|
|
416
|
+
results, not an exhaustive count of physical journal writes during that run.
|
|
417
|
+
A zero `turnsAppended` does not prove that no append reached disk: journal fsync
|
|
418
|
+
can precede a lost derived checkpoint/accounting commit. The failure message
|
|
419
|
+
states this qualification. Confirmed healthy-unit counts within the same failed
|
|
420
|
+
pass remain visible, without erasing the earlier interruption. Recovery dedupes
|
|
421
|
+
against journal truth; it neither attributes later journal growth retroactively
|
|
422
|
+
to the failed run nor counts recovered existing turns as newly appended.
|
|
423
|
+
Explicit `--file`/`--home` receipt semantics are unchanged.
|
|
424
|
+
|
|
425
|
+
`capture-status.mjs` validates this shared format and atomically replaces a
|
|
426
|
+
private same-directory temporary file after its fsync. Publication failure is
|
|
427
|
+
reported; the prior snapshot is not replaced by an invented success. Exact
|
|
428
|
+
receipt commands refresh backlog while preserving all prior service history,
|
|
429
|
+
without claiming their own unbounded operation was a bounded pass. Snapshot
|
|
430
|
+
format support alone does not produce run evidence: the admitted command worker
|
|
431
|
+
must supply committed facts.
|
|
432
|
+
|
|
433
|
+
The private `CaptureRun` helper publishes a running identity before initializing
|
|
434
|
+
its derived counter row. Counter changes join the processing transaction after
|
|
435
|
+
journal fsync or index commit; rolled-back work cannot publish progress. A
|
|
436
|
+
continuing worker combines those committed counters with the published sampled
|
|
437
|
+
RSS peak. Sampling alone never advances `lastProgressAt`. An unfinished run
|
|
438
|
+
whose counter cache is lost retains interruption evidence rather than becoming
|
|
439
|
+
a successful empty pass. A finished file receipt takes precedence over a stale
|
|
440
|
+
running counter row, and finalization needs one atomic file publication, not a
|
|
441
|
+
second derived-state commit. Supervisor failure evidence is bound explicitly to
|
|
442
|
+
the current run ID, including a child lost before any running receipt exists.
|
|
443
|
+
It is distinct from failure of an older abandoned run: only failure in the
|
|
444
|
+
current pass prevents that pass from succeeding after healthy work. Partial
|
|
445
|
+
work preserves previous success and failure history. These are private
|
|
446
|
+
accounting mechanics, not evidence that command integration has run a pass.
|
|
447
|
+
|
|
448
|
+
Source-stage fixtures use the private retained recovery cleanup unit described
|
|
449
|
+
below. A due retry may need it even for a tiny source after an interrupted
|
|
450
|
+
append. Cleanup failure is a failure in the current pass and preserves
|
|
451
|
+
confirmed work; it cannot be ignored to make a later receipt successful.
|
|
452
|
+
The private worker dispatcher also routes these units; fixture coverage is
|
|
453
|
+
separate from ordinary-command and installed-service qualification.
|
|
111
454
|
|
|
112
455
|
## Resource model
|
|
113
456
|
|
|
114
457
|
Source and journal reads use 64 KiB chunks. Batches commit at 2,000 records or
|
|
115
|
-
8 MiB of accumulated raw bytes
|
|
116
|
-
|
|
458
|
+
8 MiB of accumulated raw bytes. Admission flushes the prior batch before adding
|
|
459
|
+
a record that would exceed that byte target. Its checkpoint excludes the next
|
|
460
|
+
record's bytes and timestamp, even when the parser has already scanned that
|
|
461
|
+
line. A single larger record remains supported by public library calls and can
|
|
462
|
+
exceed the batch target. SQLite connections use a 2 MiB page-cache target
|
|
117
463
|
and disk-backed temporary tables. Metadata query pages hold at most 128 rows;
|
|
118
464
|
keyset queries support the declared Node 22.5 SQLite API without rescanning
|
|
119
465
|
prior pages. Content-processing memory therefore depends on fixed buffers,
|
|
@@ -123,7 +469,14 @@ are algorithm settings, **not** a memory/deadline policy or record-size limit.
|
|
|
123
469
|
|
|
124
470
|
Native root and path discovery retain in-memory metadata inventories, sorted by
|
|
125
471
|
the existing per-directory `localeCompare` order with stable ties and depth-first
|
|
126
|
-
traversal. Aweb account
|
|
472
|
+
traversal. Aweb account discovery retains code-unit sorted basenames. Public synchronous
|
|
473
|
+
`captureAwLogs` keeps sorted whole-file traversal. Cross-account append order
|
|
474
|
+
in `owner~aw` under the service is unspecified; per-account order is preserved.
|
|
475
|
+
Bounded command work may rotate between account files, serially under the same
|
|
476
|
+
stream lock. IDs, projection, dedupe and journal-generation checks are unchanged.
|
|
477
|
+
[`recall --thread --json`](../README.md) exposes the resulting durable sequence,
|
|
478
|
+
including mixed-account ID windows; it does not regroup or sort journal history.
|
|
479
|
+
Discovery metadata memory grows with file/profile counts, separately from streamed
|
|
127
480
|
content processing; metadata enumeration and sorting still run on unchanged
|
|
128
481
|
passes. Public list APIs retain their full sorted-array output. Per-input
|
|
129
482
|
cursors in `index/capture.db` replace whole-map JSON loads and rewrites. The
|
|
@@ -154,9 +507,57 @@ bounded transactions. Ordinary passes seek new journal bytes without parsing
|
|
|
154
507
|
consumed prefixes. Inconsistent identities or cursors invalidate derived state;
|
|
155
508
|
recovery can cost a historical scan. No journal format or turn ID changes.
|
|
156
509
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
510
|
+
The synchronous index update (`RecordIndex.update`, run by `capture --home`,
|
|
511
|
+
`--file` and `recall --reindex`) uses one `capture.db` connection for the pass.
|
|
512
|
+
It first takes a census without locks. A stream is current, and is skipped
|
|
513
|
+
without a stream lock or validation, only when all of the following hold:
|
|
514
|
+
|
|
515
|
+
- its complete validation checkpoint names the journal's exact current version;
|
|
516
|
+
- the search index's byte cursor and epoch match that checkpoint;
|
|
517
|
+
- both count tables agree with it.
|
|
518
|
+
|
|
519
|
+
Every other stream is a candidate. That includes a stream whose evidence is
|
|
520
|
+
missing, partial or disagreeing, backlog left by `--no-index` passes, and
|
|
521
|
+
streams that other writers changed. Each candidate is validated again under its
|
|
522
|
+
stream lock before any write.
|
|
523
|
+
|
|
524
|
+
A candidate that proves the union stale resets it atomically, and selection
|
|
525
|
+
restarts against the empty index. Stale means the journal is missing or
|
|
526
|
+
replaced, a new epoch, an invalid cursor, or a legacy count disagreement. A
|
|
527
|
+
journal that changes between that check and catch-up refuses the pass with the
|
|
528
|
+
existing retry error. A skipped stream's cursor is read again when its turn
|
|
529
|
+
comes, so a reset by another index connection is caught up, not skipped. The
|
|
530
|
+
pass refuses if `capture.db` was replaced or removed while it held the
|
|
531
|
+
connection.
|
|
532
|
+
|
|
533
|
+
The census is not an atomic snapshot of concurrent writers. A change made after
|
|
534
|
+
a stream was observed is indexed by a later pass. A missing checkpoint still
|
|
535
|
+
costs one historical validation of the stream, and the reset it implies then
|
|
536
|
+
reindexes the union.
|
|
537
|
+
|
|
538
|
+
Search-index reset remains one atomic transaction. Ordinary tables use SQLite's
|
|
539
|
+
truncate clear; the content-storing FTS table is dropped and recreated using the
|
|
540
|
+
owning schema instead of deleting text rows individually. Readers retain the
|
|
541
|
+
existing old-union/empty-union semantics, including already-open connections.
|
|
542
|
+
An interrupted transaction rolls back the entire reset. Successful timing alone
|
|
543
|
+
does not establish remaining deadline budget; the operator-only
|
|
544
|
+
[reset benchmark](../../../scripts/benchmark-record-reset.md) measures the full
|
|
545
|
+
commit separately from sampled reader blocking on a disposable SQLite backup.
|
|
546
|
+
|
|
547
|
+
The [operator's real-index qualification](https://github.com/awebai/oats/issues/1108#issuecomment-6165193608)
|
|
548
|
+
used benchmark commit `16c62e39aaa43c2bb510cc446c7528359a9a4bd2` on two fresh
|
|
549
|
+
online backups with identical counts: 4.82 GB, 2,286,985 turns and 1,313 streams.
|
|
550
|
+
On macOS ARM/APFS internal SSD, Node 26.11.1/SQLite 3.54.0 with DELETE/FULL,
|
|
551
|
+
the original reset took 45,291 ms and atomic drop/recreate took 824 ms, including
|
|
552
|
+
COMMIT. Longest sampled reader-busy spans were 44,453 ms and 463 ms; 10 ms
|
|
553
|
+
sampling with gaps up to 56 ms does not measure exact reader-exclusion time.
|
|
554
|
+
Sampled rollback-journal peaks (2.39 GB and 4.9 MB) are lower bounds. These
|
|
555
|
+
results qualify the isolated atomic reset on that storage, not the subsequent
|
|
556
|
+
2.3-million-turn reconstruction, full-pass deadline or Linux service performance.
|
|
557
|
+
|
|
558
|
+
Ordinary scratch directories are private per synchronous caller (0700 directories,
|
|
559
|
+
0600 owner/payload files), with a same-host PID and nonce marker. Those per-call
|
|
560
|
+
allocations never transfer ownership or open descriptors to a subprocess. A command holding
|
|
160
561
|
the capture lock removes recognized allocations only after positively proving
|
|
161
562
|
the local allocator dead and rechecking directory and marker identity. Public
|
|
162
563
|
synchronous callers still do not acquire an implicit capture lock; their live
|
|
@@ -167,16 +568,434 @@ For such leftovers, stop all capture/library users, establish that no caller is
|
|
|
167
568
|
using the specific allocation, and remove only that confirmed orphan; do not
|
|
168
569
|
blanket-delete the scratch area while callers may be active. No age-only cleanup.
|
|
169
570
|
|
|
571
|
+
Command recovery uses separately recognized `recovery-*` allocations. Ordinary
|
|
572
|
+
dead-owner scavenging preserves them: allocator death alone cannot erase a job
|
|
573
|
+
that must resume. The registered root-lock worker requires a durable job
|
|
574
|
+
reference, stable allocation nonce and exact directory/payload identities before
|
|
575
|
+
opening one. Adoption requires positive death proof of the previous local owner
|
|
576
|
+
and rechecks the job, marker and registered worker before publication. A
|
|
577
|
+
self-identified owner publication names the prior committed marker stamp, so a
|
|
578
|
+
crash after publication but before its DB checkpoint can be recovered without
|
|
579
|
+
accepting a copied/replaced owner file. No descriptor transfers between workers.
|
|
580
|
+
Lost references, unknown owners and unpublished allocations cannot become proof
|
|
581
|
+
or cleanup authority by pathname. They retain the conservative orphan procedure
|
|
582
|
+
above. A committed deletion mark precedes allocation-specific removal; reference
|
|
583
|
+
finalization can be batched after removal, and interruption is repeatable.
|
|
584
|
+
|
|
585
|
+
`capture-cleanup-work.mjs` supplies that lifecycle as a private work unit.
|
|
586
|
+
`captureCleanupPage` reads at most 128 allocation references by key, including
|
|
587
|
+
active jobs, and returns only deleting jobs. It never scans indefinitely to
|
|
588
|
+
find a match. The caller saves its returned key and resumes another page;
|
|
589
|
+
reaching the end of a page sequence is enumeration evidence, not a completed
|
|
590
|
+
capture pass. Selection does not depend on a source remaining ready, so a
|
|
591
|
+
finished or refused source's deleting reference is still discoverable after
|
|
592
|
+
a worker dies. Active allocations are not selected for deletion.
|
|
593
|
+
|
|
594
|
+
`cleanupCaptureRecovery` binds a selected job to its allocation nonce and
|
|
595
|
+
stream, commits verified adoption, removes the allocation outside a database
|
|
596
|
+
transaction, then divides one bounded hash-row allowance between recovery
|
|
597
|
+
chunks and transcript prefix rows under the stream lock. Both tables must be
|
|
598
|
+
finished before it removes the allocation and staging references. A partial
|
|
599
|
+
unit, cooperative stop after removal, or failed final metadata commit retains
|
|
600
|
+
the deleting reference; the next registered worker revalidates it and resumes.
|
|
601
|
+
The source's committed cursor and journal bytes are not rewritten or recounted.
|
|
602
|
+
The caller still owns fair selection, operational failure handling, budgeting
|
|
603
|
+
and final receipt publication; this unit adds no separate retry policy. Its
|
|
604
|
+
optional commit callback joins admission retirement and confirmed progress to
|
|
605
|
+
the final stream-locked metadata transaction. Callback failure rolls back those
|
|
606
|
+
facts and metadata deletion together, even if the directory is already absent.
|
|
607
|
+
|
|
608
|
+
`CaptureCleanupQueue` supplies private worker scheduling for deleting claims.
|
|
609
|
+
Healthy and retry enumeration have independent durable keyset cursors; each
|
|
610
|
+
turn inspects at most one page of 128 references past its cursor and processes
|
|
611
|
+
at most one allocation under the shared hash-row allowance. When that page ends
|
|
612
|
+
the list without a job, the turn inspects one more page from the start, so an
|
|
613
|
+
unfinished allocation the cursor has passed continues in the same pass; with
|
|
614
|
+
no job there either, the cursor wraps for later work. A cleanup unit that
|
|
615
|
+
commits no metadata progress is not selected again in its pass; the next pass
|
|
616
|
+
tries it again. A scan endpoint describes enumeration only: partial jobs or
|
|
617
|
+
newly added earlier keys can still need work on the next rotation.
|
|
618
|
+
|
|
619
|
+
Before filesystem work, the queue commits an allocation-bound admission and
|
|
620
|
+
the next cursor. The cleanup unit must still acquire the stream lock, prove
|
|
621
|
+
ownership and commit adoption; queued identities never grant pathname authority.
|
|
622
|
+
Its final metadata transaction retires admission and records actual metadata
|
|
623
|
+
progress without inventing source/index counts. Cooperative deadline/RSS stop
|
|
624
|
+
retires only admission, preserving the deleting claim for continuation. Other
|
|
625
|
+
throws or worker death retain admission until the registered replacement
|
|
626
|
+
reconciles it after predecessor death.
|
|
627
|
+
|
|
628
|
+
Reconciliation records a pass-bound failure and allocation-specific backoff
|
|
629
|
+
using the fixed 30/60/120/240-minute schedule. It does not diagnose a journal
|
|
630
|
+
frame, create a public failed-stream row, or change source/index retry evidence.
|
|
631
|
+
Skipped work does not increment attempts or move the retry date. A failed
|
|
632
|
+
allocation cannot retry in the same pass; a due partial unit retains its retry
|
|
633
|
+
until complete and admits at most once in a pass. Successful deletion clears
|
|
634
|
+
only that allocation's retry. Historical run failure remains history; current
|
|
635
|
+
pass failure survives independent healthy processing. Final completion still
|
|
636
|
+
requires the owning runner's all-stage audit, not an empty cleanup page.
|
|
637
|
+
|
|
638
|
+
`capture-recovery-snapshot.mjs` copies the fixed source end in 64 KiB chunks under
|
|
639
|
+
the stream group. Payload fsync precedes committed copy offsets and chunk hashes.
|
|
640
|
+
An uncommitted suffix is truncated only inside the adopted allocation, followed
|
|
641
|
+
by bounded verification of all committed chunk hashes before continuing. Lost
|
|
642
|
+
chunk metadata or mismatched bytes revoke the staged proof. Copying checks the
|
|
643
|
+
raw-record limit incrementally, before allocating a whole oversized record.
|
|
644
|
+
After copying, consumers read only that staged descriptor and verify its version;
|
|
645
|
+
the mutable source remains an observation/custody check, not a replacement data
|
|
646
|
+
source. Append growth uses the shared append-only assumption and bounded source
|
|
647
|
+
boundary witness; replacement, shrink, same-size edits or witness mismatch revoke
|
|
648
|
+
the job. The scheduler's durable reset/retry generation also invalidates a staged
|
|
649
|
+
snapshot; it stays fixed across partial workers so retry cannot restart forever.
|
|
650
|
+
This source qualification is separate from exact journal-generation
|
|
651
|
+
validation. A journal change between workers retires the old fixed-end snapshot
|
|
652
|
+
before comparing a possibly longer captured prefix, including after public or
|
|
653
|
+
outgoing-release writes. Later source growth alone is work for a later snapshot.
|
|
654
|
+
The native command unit may carry a completed proof into its own next journal
|
|
655
|
+
generation only after its fsynced append under the same stream lock, committing
|
|
656
|
+
the source cursor and staged job's new journal version together. An interrupted
|
|
657
|
+
commit instead leaves the old generation and forces revalidation.
|
|
658
|
+
|
|
659
|
+
Small command recovery snapshots have a private optimization when their **total
|
|
660
|
+
fixed end** is at most 8 MiB. Each worker streams a new private copy and full
|
|
661
|
+
SHA-256 in 64 KiB chunks before trusting saved comparison progress. Complete
|
|
662
|
+
digest plus size identifies the proven bytes; source identity/observation,
|
|
663
|
+
reset/retry generation and exact journal generation must also qualify. The
|
|
664
|
+
comparison consumes that worker's verified snapshot, never a later source-path
|
|
665
|
+
reopen. Same-content replacement takes the normal validation path. Matching
|
|
666
|
+
snapshots preserve comparison progress; a changed digest or other qualification
|
|
667
|
+
restarts proof before append. Ordinary unchanged/growth capture does not use
|
|
668
|
+
this recovery optimization.
|
|
669
|
+
|
|
670
|
+
These small copies are disposable per-call allocations, with no payload or
|
|
671
|
+
ownership-handoff fsync. Their loss never loses a proven source checkpoint:
|
|
672
|
+
the next worker must copy/hash again. Repeated source/stage bytes are recovery
|
|
673
|
+
work and are counted separately from probes and ordinary incremental work.
|
|
674
|
+
If restaging exhausts the unit before required work can advance, the processing
|
|
675
|
+
commit selects retained resumable copying for subsequent admissions. A killed
|
|
676
|
+
unit cannot publish an in-memory fallback decision; its durable failed-admission
|
|
677
|
+
retry selects the retained path. The fixed source end is preserved by committed
|
|
678
|
+
observation metadata, which alone never authorizes prefix proof. SQLite commits,
|
|
679
|
+
journal syncs, root-lock publication and retained fallback syncs remain separate
|
|
680
|
+
costs; avoiding disposable-copy fsync is not a claim that first-upgrade storage
|
|
681
|
+
cost or the complete service has been qualified.
|
|
682
|
+
|
|
683
|
+
Staging consumes source-sized private space plus bounded-per-chunk hash metadata.
|
|
684
|
+
Revalidating staged bytes after interruption is recovery work, reported separately
|
|
685
|
+
from mutable-source reads and boundary probes. Completed or invalidated jobs
|
|
686
|
+
reclaim hashes in bounded units; a hash-cache entry never authorizes deletion of
|
|
687
|
+
a filesystem path. These are private worker primitives; their tests do not by
|
|
688
|
+
themselves qualify the complete command scheduler, run receipts or service limits.
|
|
689
|
+
|
|
690
|
+
The private `NativeCaptureDiscovery` walker checkpoints only its active
|
|
691
|
+
per-directory stack, directory identities and exact last basenames. A resumed
|
|
692
|
+
walker loads that stack directly rather than revisiting completed sibling
|
|
693
|
+
subtrees. Within a worker it retains only the sorted arrays for active
|
|
694
|
+
ancestors; completion, close and errors release those arrays. Selection and
|
|
695
|
+
stable `localeCompare` ordering match the public cc/pi/codex inventories,
|
|
696
|
+
including Claude's restricted subagent layout and existing link distinctions.
|
|
697
|
+
A replaced active directory invalidates its position. An absent fresh default
|
|
698
|
+
can be empty; disappearance of an already selected directory is an error.
|
|
699
|
+
|
|
700
|
+
Each discovery unit accounts for visited metadata entries, including irrelevant
|
|
701
|
+
ones, and returns a bounded file page plus the next checkpoint. The scheduling
|
|
702
|
+
caller must commit that page and checkpoint together. Directory reads/sorts
|
|
703
|
+
can recur on restart and their arrays still grow with directory size; this is
|
|
704
|
+
not a constant-memory metadata claim or new-byte-only work. Newly inserted
|
|
705
|
+
names before the saved position belong to a later sweep. A missing anchor
|
|
706
|
+
conservatively replays its collation equivalence class, requiring idempotent
|
|
707
|
+
inventory insertion. This walker does not itself select roots, schedule
|
|
708
|
+
capture/index work or certify that a complete command sweep occurred.
|
|
709
|
+
|
|
710
|
+
## Internal transaction groups
|
|
711
|
+
|
|
712
|
+
The package-private `capture-context.mjs` helper can group synchronous work over
|
|
713
|
+
at most 128 streams. It acquires every stream lock in canonical order before
|
|
714
|
+
starting a shared `capture.db` transaction. Helpers reuse that connection and
|
|
715
|
+
use savepoints for nested failure isolation; journal appends still fsync before
|
|
716
|
+
the group commits derived offsets/IDs. No additional stream may be acquired
|
|
717
|
+
inside the transaction. This preserves public writers' stream-before-database
|
|
718
|
+
lock order. Group acquisition accepts an internal stop check and unwinds partial
|
|
719
|
+
acquisitions on failure. Callers must select bounded work before entering a group;
|
|
720
|
+
the helper alone does not turn a full reconciliation into a bounded pass.
|
|
721
|
+
|
|
722
|
+
Command groups can supply a bounded admission manifest. After acquiring all
|
|
723
|
+
member locks, the helper rechecks each exact journal generation and commits the
|
|
724
|
+
manifest plus the next fairness position in a separate transaction, before any
|
|
725
|
+
processing. That position is queue progress, not a processing cursor. The same
|
|
726
|
+
intent transaction also retains each stage's last position independently;
|
|
727
|
+
intervening index work cannot reset source rotation. This adds one state-row
|
|
728
|
+
write per admission, not another transaction or a per-record sync. Completed
|
|
729
|
+
checkpoints and removal of the active manifest then share the processing commit.
|
|
730
|
+
A crash or thrown error leaves the manifest and fairness position intact, while
|
|
731
|
+
only journal bytes already fsynced survive a rolled-back processing transaction.
|
|
732
|
+
The scheduler must reconcile an unfinished manifest before admitting another;
|
|
733
|
+
the helper refuses to overwrite it or guess per-member failure rows. A normal
|
|
734
|
+
changed group therefore adds one intent commit, regardless of its record count.
|
|
735
|
+
No-op groups without admission add no intent transaction. These counts describe
|
|
736
|
+
explicit SQLite transactions, not filesystem sync calls or storage latency.
|
|
737
|
+
|
|
738
|
+
`capture-retry.mjs` reconciles an interrupted manifest after the previous worker
|
|
739
|
+
has exited. One transaction stores retry membership, run failure evidence and
|
|
740
|
+
any attributable single-stream failure, then removes the manifest. A failed
|
|
741
|
+
commit leaves the manifest available for retry; completed reconciliation is
|
|
742
|
+
idempotent. Neither reconciliation nor admission advances processing cursors.
|
|
743
|
+
Deferred membership is separate from public failure rows. Admission itself
|
|
744
|
+
requires a due, single-stream retry and rejects a second attempt in the same
|
|
745
|
+
run. A separate durable retry position rotates incomplete members across runs;
|
|
746
|
+
bounded keyset enumeration avoids retaining all deferred members in memory.
|
|
747
|
+
The scheduler supplies a finite, fair healthy sweep before deferred selection.
|
|
748
|
+
Enumeration still costs metadata work and is not a new-content-only operation.
|
|
749
|
+
|
|
750
|
+
Resolution joins the processing checkpoint transaction. An unchanged journal
|
|
751
|
+
must be processed through the failed snapshot's end, or the stage must validate
|
|
752
|
+
that the unit no longer needs work. A checkpoint inside that snapshot, a changed
|
|
753
|
+
generation, expiration of the retry date or a process restart cannot by itself
|
|
754
|
+
clear the warning. The stage remains responsible for proving reconciliation;
|
|
755
|
+
these private queue helpers do not implement the full source/index scheduler.
|
|
756
|
+
|
|
757
|
+
Journal recovery exposes an internal work-unit interface: obsolete-ID deletion
|
|
758
|
+
has a persisted phase, and validation commits its byte/count cursor after each
|
|
759
|
+
bounded batch. A changed exact journal version invalidates that progress. A
|
|
760
|
+
partial checkpoint never authorizes append. Source framing accepts an internal
|
|
761
|
+
record limit and checks it before joining an oversized line; public callers keep
|
|
762
|
+
their unlimited framing default. These primitives require a scheduling caller to
|
|
763
|
+
provide budgets and continuation; they do not bound public library operations.
|
|
764
|
+
|
|
765
|
+
`advanceTranscriptPrefix` is the internal resumable comparison for a verified
|
|
766
|
+
immutable source snapshot. Its durable phases clear obsolete hashes, consume
|
|
767
|
+
the journal from a byte checkpoint, then compare source lines from a byte/line
|
|
768
|
+
checkpoint. Hashes and phase progress share the surrounding stream group's
|
|
769
|
+
transaction. A snapshot identity or exact journal generation change revokes
|
|
770
|
+
the proof; losing the hash table or its epoch cannot qualify old checkpoints.
|
|
771
|
+
Only the ready phase authorizes the caller to append. A late disagreement
|
|
772
|
+
reports the conflicting source line/offset without appending any suffix.
|
|
773
|
+
Cleanup revokes the proof before deleting bounded batches of hashes. Journal
|
|
774
|
+
processing retains the historical-single-record qualification; the raw source
|
|
775
|
+
record policy is not a serialized-journal size limit. This comparison helper
|
|
776
|
+
does not own snapshot staging, transfer its ownership or read the mutable source.
|
|
777
|
+
|
|
778
|
+
`capture-native-unit.mjs` combines these private primitives for one native input.
|
|
779
|
+
It performs bounded journal validation before any append, resumes verified
|
|
780
|
+
snapshot/prefix work when a source cursor is unqualified, and admits one bounded
|
|
781
|
+
batch from the verified bytes. Ordinary known growth uses the source cursor and
|
|
782
|
+
bounded witness without staging historical content. The fixed end survives
|
|
783
|
+
partial units; later growth is a later sweep. A separately checkpointed search
|
|
784
|
+
frontier finds the first timestamp across units, then carries it backward while
|
|
785
|
+
capturing the unstamped prefix. An unchanged unstamped or torn tail stays pending
|
|
786
|
+
without repeatedly scanning it. A record over the raw-record limit commits the
|
|
787
|
+
unit's complete records before it, then sets the input aside at its start
|
|
788
|
+
offset ([operator recovery](#operator-recovery)). Explicit retry generations
|
|
789
|
+
persist separately from active issue rows, so clearing a resolved issue cannot
|
|
790
|
+
restart its proof.
|
|
791
|
+
Ignore is applied before source/custody access; retry still compares the complete
|
|
792
|
+
captured prefix. The scheduler must account for pending/failed work and perform
|
|
793
|
+
allocation cleanup after its deletion mark commits. These are unit interfaces,
|
|
794
|
+
not a resource promise for public synchronous capture or exact receipts.
|
|
795
|
+
|
|
796
|
+
`capture-aw-unit.mjs` similarly processes one fixed-end log unit. Every scanned
|
|
797
|
+
raw line consumes its work allowance, including blank lines, malformed complete
|
|
798
|
+
lines and projection refusals. Cursor publication follows the pending batch's
|
|
799
|
+
durable append. A valid final object without its newline retains its starting
|
|
800
|
+
offset and keeps the command pending; an unchanged snapshot does not repeatedly
|
|
801
|
+
project that fragment. Replacement or explicit source reset reprojects through
|
|
802
|
+
persistent journal-ID deduplication. Raw oversized input uses the same set-aside
|
|
803
|
+
store, prefix capture and retry/ignore rules as native inputs
|
|
804
|
+
([operator recovery](#operator-recovery)); a projection refusal itself is
|
|
805
|
+
consumed evidence and does not create a second issue mechanism.
|
|
806
|
+
|
|
807
|
+
Public synchronous library calls do not implicitly enter this context or acquire
|
|
808
|
+
a root capture lock. Healthy derived-state opens do not run a schema write
|
|
809
|
+
transaction, and identical state values do not produce redundant row updates.
|
|
810
|
+
|
|
811
|
+
The private supervisor transport admits one child at a time under the root lock. The child
|
|
812
|
+
announces its nonce and stays idle until the supervisor durably publishes its
|
|
813
|
+
identity in that lock and sends permission to start. A live, reused or unknown
|
|
814
|
+
worker PID prevents reclamation after supervisor death. Release occurs only
|
|
815
|
+
after the owned child exits; signals target the spawned child handle, never a
|
|
816
|
+
PID inferred from an abandoned lock or a process group. Continuation retains the
|
|
817
|
+
same root lock: it replaces only the exact previously registered worker identity
|
|
818
|
+
after positive death proof. Reused or uncertain PIDs prevent replacement.
|
|
819
|
+
Workers check their parent and lock identity between work units; the same read
|
|
820
|
+
carries the supervisor's cooperative stop request, which the supervisor
|
|
821
|
+
publishes into that owner record exactly as it publishes a worker. The supervisor never writes status
|
|
822
|
+
on behalf of a crashed worker. A shared monotonic deadline includes process
|
|
823
|
+
startup and lock publication, with a separate cooperative drain boundary.
|
|
824
|
+
Launch arguments fix the heap policy and exclude inherited Node flags and
|
|
825
|
+
preloads. This transport requires a reconciliation entry that supplies bounded
|
|
826
|
+
units and persists run facts; it does not itself implement a complete pass.
|
|
827
|
+
|
|
828
|
+
`capture-pass.mjs` supplies serial child continuation under that same absolute
|
|
829
|
+
deadline. Startup, publication, checkpoint finalization and reaping consume the
|
|
830
|
+
pass budget; child replacement cannot renew it. Normal work has internal drain,
|
|
831
|
+
finalization and forced-stop boundaries, and is admitted only with its full
|
|
832
|
+
allowance. A reserved start and time allowance let a final worker reconcile an
|
|
833
|
+
interrupted predecessor and persist run facts. All starts, including spawn
|
|
834
|
+
failures and finalization, count against the fixed internal ceiling. The parent
|
|
835
|
+
can report its observed process failure, but only a worker can reconcile durable
|
|
836
|
+
admission and attribute it. There is no in-pass diagnostic retry mode. The
|
|
837
|
+
reconciliation entry must select independent ready work before deferred failed
|
|
838
|
+
work; transport tests alone do not establish that scheduling or recovery rule.
|
|
839
|
+
|
|
840
|
+
Bounded index groups reuse the already acquired stream group and capture-state
|
|
841
|
+
connection. Validation precedes the index write transaction; completed index
|
|
842
|
+
rows and byte/count cursors commit together once per group. A cooperative stop
|
|
843
|
+
commits completed work, while an exception rolls back that index group. Each
|
|
844
|
+
selected stream receives a share of the record/byte budget, so a large journal
|
|
845
|
+
does not consume every selected stream's allowance. The scheduler must visit
|
|
846
|
+
both live streams and indexed streams whose journals disappeared, and restart
|
|
847
|
+
its index sweep after an atomic union reset. The group helper never declares
|
|
848
|
+
the entire root complete. It defaults to returning `resetRequired` without
|
|
849
|
+
mutating index rows when reset is needed. The scheduler must reserve a fresh
|
|
850
|
+
full worker allowance for that atomic operation; the detection unit and the
|
|
851
|
+
finalization reserve cannot implicitly start it. Explicit reset permission is
|
|
852
|
+
an internal seam, not a public policy override. A no-op group performs no index write transaction;
|
|
853
|
+
its capture-state transaction remains separate from the index commit. A failure
|
|
854
|
+
between the two commits is recovered through journal version/epoch validation,
|
|
855
|
+
not by assuming a cross-database atomic transaction.
|
|
856
|
+
|
|
857
|
+
`capture-index-work.mjs` joins a selected index group to durable admission,
|
|
858
|
+
per-stage scheduling position and run accounting. Its caller supplies a finite,
|
|
859
|
+
fairly ordered union of live and indexed streams; this bridge neither builds
|
|
860
|
+
that inventory nor certifies sweep completion. Failure attribution starts at
|
|
861
|
+
the saved validation boundary when recovery is unfinished, otherwise at the
|
|
862
|
+
qualified index cursor. Changed journal generations start validation at zero.
|
|
863
|
+
Only the locked group validates and consumes those cursors.
|
|
864
|
+
|
|
865
|
+
Index rows commit before their confirmed `turnsIndexed` count joins the
|
|
866
|
+
capture-state commit. If that later transaction fails, the durable admission
|
|
867
|
+
remains for reconciliation while the index rows can survive. Resuming an
|
|
868
|
+
already indexed unit does not reconstruct its lost run count. Validation or
|
|
869
|
+
indexed-byte advancement can update progress; a no-op visit does not. Due
|
|
870
|
+
index retries are single-stream admissions; only completion of the current
|
|
871
|
+
validated journal resolves them. Partial validation/indexing and reset requests
|
|
872
|
+
retain failure evidence, and other-stage retries return to the scheduler.
|
|
873
|
+
Reset detection always defers: the owning scheduler must reserve a fresh full
|
|
874
|
+
worker allowance for reset and restart the union sweep afterward. Source,
|
|
875
|
+
discovery, cleanup and final receipt routing remain separate responsibilities.
|
|
876
|
+
|
|
877
|
+
The private `CaptureIndexInventory` builds the union of indexed stream names
|
|
878
|
+
and live journal stream names in `capture.db`. Indexed names are enumerated
|
|
879
|
+
first, including names whose journal disappeared; they remain in the inventory
|
|
880
|
+
so locked index validation can detect changes to cross-stream tombstone
|
|
881
|
+
authority. Each discovery call inspects at most 128 index rows or directory
|
|
882
|
+
entries, including irrelevant names. Inserting that page and advancing its
|
|
883
|
+
checkpoint share one transaction. Names shared by both sources occur once.
|
|
884
|
+
The inventory never reads journal content, changes index rows or grants reset
|
|
885
|
+
permission.
|
|
886
|
+
|
|
887
|
+
A scheduler-owned sweep identity resumes discovery across connection/worker
|
|
888
|
+
restarts. Live discovery retains a sorted directory-name array only within the
|
|
889
|
+
worker and resumes by a binary name key when reopened, even if that anchor was
|
|
890
|
+
removed. The selected streams directory's identity must remain unchanged.
|
|
891
|
+
Directory listing/sorting is still metadata work proportional to directory
|
|
892
|
+
size, and can recur after restart; it is not a bounded-memory metadata claim.
|
|
893
|
+
New names before the saved position belong to a later sweep. A fresh absent
|
|
894
|
+
directory can be empty; disappearance/replacement after selection is an error.
|
|
895
|
+
|
|
896
|
+
Ordered reads use bounded database pages and rotate after the durable index
|
|
897
|
+
admission position, visiting the discovered union once per iterator. Inventory
|
|
898
|
+
rows are not removed on an apparent index completion: membership is not cursor
|
|
899
|
+
or generation proof. The `complete` flag only means enumeration finished, never
|
|
900
|
+
that indexing or a pass succeeded. The scheduler still owns index outcomes,
|
|
901
|
+
new streams created by interleaved capture, reset admission and a fresh union
|
|
902
|
+
sweep after reset. Replacing a sweep retires its exact scope; indexed cleanup
|
|
903
|
+
deletes at most 128 obsolete rows per unit and refuses the active scope. Losing
|
|
904
|
+
this private table restarts enumeration without clearing retries, journal
|
|
905
|
+
checkpoints or the index. These primitives alone are not the full command runner.
|
|
906
|
+
|
|
907
|
+
`CaptureIndexOutcomes` records processing phases separately from membership in
|
|
908
|
+
that private inventory. The index-work bridge publishes outcomes inside the
|
|
909
|
+
admitted stream group's capture-state transaction, after the index commit and
|
|
910
|
+
run accounting. A failure in outcome publication rolls back its marks and run
|
|
911
|
+
counters together, leaving the admission for reconciliation even when index
|
|
912
|
+
rows survived. A later no-op can qualify those existing rows without guessing
|
|
913
|
+
the lost earlier run count.
|
|
914
|
+
|
|
915
|
+
A stored complete phase is only cached evidence. Before skipping that stream,
|
|
916
|
+
the outcome reader compares the current journal file version, completed
|
|
917
|
+
validation epoch/byte/count checkpoint and index count/byte/epoch cursors with
|
|
918
|
+
the recorded evidence. Growth, replacement, validation loss, outgoing count
|
|
919
|
+
writes or an index reset revoke mismatching evidence. This is metadata
|
|
920
|
+
requalification, not a historical content scan or new integrity mechanism;
|
|
921
|
+
locked validation/index work still owns processing authority. Outstanding
|
|
922
|
+
retries keep the stream pending. Ordinary inventory enumeration retains all
|
|
923
|
+
names, while outcome-aware selection skips only still-qualified completions.
|
|
924
|
+
|
|
925
|
+
The scheduler requalifies each stream once per worker pass. A stream found
|
|
926
|
+
qualified, or recorded complete by an index group, is not compared again by
|
|
927
|
+
later steps of that pass, and a walk that finds every stream qualified leaves
|
|
928
|
+
the index stage with no candidates. Each write the pass makes to a stream
|
|
929
|
+
revokes that: a capture unit other than an unchanged one, or an index outcome
|
|
930
|
+
other than complete. So does a new inventory member. A new worker, and so
|
|
931
|
+
every pass, requalifies every stream once; due retries, the completion audit
|
|
932
|
+
and other outcome reads always compare the evidence.
|
|
933
|
+
|
|
934
|
+
The source-work bridge can join an active index inventory to its processing
|
|
935
|
+
commit. It inserts every processed stream, including replay with no new append,
|
|
936
|
+
so streams created after directory discovery are not omitted. Duplicate
|
|
937
|
+
membership does not alter a prior outcome; subsequent outcome reads requalify
|
|
938
|
+
that evidence against the changed journal. A failed source transaction rolls
|
|
939
|
+
back membership with its source checkpoint and counters, while the durable
|
|
940
|
+
admission remains available to reconcile surviving journal bytes.
|
|
941
|
+
|
|
942
|
+
Reset-required outcomes remain durable and block further outcome-aware index
|
|
943
|
+
work in that sweep. Neither detection nor the outcome scheduler executes reset.
|
|
944
|
+
The owning scheduler must admit the separate atomic operation and replace the
|
|
945
|
+
inventory sweep afterward. Outcome audit pages inspect at most 128 names and
|
|
946
|
+
report how many still need work; their end-of-page `complete` flag is only audit
|
|
947
|
+
enumeration completion. The caller must also account for unfinished discovery,
|
|
948
|
+
new captures, retries, reset and other stages, and recheck after writes before
|
|
949
|
+
finalizing a pass. These outcome/source connections supply evidence to the
|
|
950
|
+
separate mixed-stage audit and owning command entry described below.
|
|
951
|
+
|
|
952
|
+
The private `CaptureIndexReset` bridge persists a union-reset request only
|
|
953
|
+
after an inventory outcome requires reset. The detecting worker publishes the
|
|
954
|
+
request and yields; a transient `resetRequired` continuation asks the pass
|
|
955
|
+
transport for a fresh normal worker allowance. Finalization never receives
|
|
956
|
+
that allowance. The reset bridge refuses the detecting worker, a normal worker
|
|
957
|
+
without reset admission, or an unreconciled stream admission. The owning entry
|
|
958
|
+
must dispatch reset before ordinary processing in that fresh worker. A restart
|
|
959
|
+
that lost the transient continuation can request another fresh worker from the
|
|
960
|
+
durable request; it cannot silently reset during ordinary processing.
|
|
961
|
+
|
|
962
|
+
Union-reset admission is separate from stream admissions. The reset transaction
|
|
963
|
+
uses the same atomic table mutations as public reset and commits a private token
|
|
964
|
+
in the existing index metadata table. It does not change schema version, owner,
|
|
965
|
+
journals, stream retry/failure rows or reader visibility. Only after that index
|
|
966
|
+
commit does a capture-state transaction publish a new, empty, pending inventory
|
|
967
|
+
scope, retire the old scope for bounded cleanup, record progress and remove the
|
|
968
|
+
request. These two commits are not a cross-database atomic transaction.
|
|
969
|
+
|
|
970
|
+
If sweep publication fails after the reset commits, a replacement worker uses
|
|
971
|
+
the index token to finish the bounded metadata publication without resetting
|
|
972
|
+
again. Index writes performed after the completed reset survive that recovery;
|
|
973
|
+
their counts are not reconstructed into the run receipt. If the reset itself
|
|
974
|
+
rolls back, the old index and request remain. The next registered worker, after
|
|
975
|
+
the predecessor has ended, reconciles the private admission into a pass-bound
|
|
976
|
+
failure and the fixed 30/60/120/240-minute retry delay. An attempted reset cannot
|
|
977
|
+
repeat in the same pass. A union failure does not identify a failing stream or
|
|
978
|
+
frame, so it creates no guessed stream-failure row and erases no stream retry.
|
|
979
|
+
The run writer must carry that returned failure into its actual pass receipt.
|
|
980
|
+
Completed-token recovery may finish metadata in a finalization worker; that
|
|
981
|
+
does not grant permission to execute another index reset there.
|
|
982
|
+
|
|
983
|
+
Reset completion starts index discovery/processing again; it is never indexing
|
|
984
|
+
completion or pass success. These reset and transport helpers still require
|
|
985
|
+
the owning mixed-stage entry to route reconciliation, healthy independent work,
|
|
986
|
+
deferred resets and final receipts. Transaction/reopen fixtures model this
|
|
987
|
+
handoff separately from actual worker-death and hosted manager qualification.
|
|
988
|
+
|
|
170
989
|
## First-upgrade cost
|
|
171
990
|
|
|
172
991
|
The first capture after upgrade must reconcile existing journals and rebuild the
|
|
173
|
-
search index.
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
992
|
+
search index. Capture and index groups amortize connections and transactions
|
|
993
|
+
over their member streams ([internal transaction groups](#internal-transaction-groups)):
|
|
994
|
+
a changed index group of several streams commits once per database, and an
|
|
995
|
+
unchanged one begins no index transaction. The bounded service must separately
|
|
996
|
+
qualify first-upgrade and unchanged passes beside actual v0.53.0 on a Linux
|
|
997
|
+
runner, including filesystem/runtime details, connection/transaction/sync
|
|
998
|
+
counts, resumable prefix proof and deadline progress.
|
|
180
999
|
|
|
181
1000
|
## Downgrade and re-upgrade
|
|
182
1001
|
|
|
@@ -191,3 +1010,683 @@ versions and mismatched index counts on re-upgrade, requiring derived rebuilds.
|
|
|
191
1010
|
Deleting the derived caches has the same recovery cost. No journal-byte migration
|
|
192
1011
|
or rollback is required. Native-prefix set-aside is essential to that guarantee;
|
|
193
1012
|
it does not rewrite history or append a sentinel or duplicate terminal turn.
|
|
1013
|
+
|
|
1014
|
+
|
|
1015
|
+
## Private source inventory
|
|
1016
|
+
|
|
1017
|
+
`capture-source-queue.mjs` stores command discovery pages in the derived
|
|
1018
|
+
`capture_sources` table. At most 128 input descriptors and the walk checkpoint
|
|
1019
|
+
join one transaction. Replayed discovery preserves original ordinal positions
|
|
1020
|
+
and completed outcomes. A new selection or a finished visit sweep creates a
|
|
1021
|
+
new private scope; superseded scope rows are deleted by indexed 128-row units,
|
|
1022
|
+
without scanning or resetting another owner's current inventory. Losing the
|
|
1023
|
+
inventory table invalidates its discovery checkpoints and re-enumerates; it
|
|
1024
|
+
never changes journals, source cursors, IDs or prefix-proof authority.
|
|
1025
|
+
|
|
1026
|
+
Ready-input reads use bounded database pages. Admission advances the persisted
|
|
1027
|
+
source position before processing; source outcome changes commit alongside the
|
|
1028
|
+
owning capture helper's source checkpoint. Rollback therefore retains the old
|
|
1029
|
+
processing state while allowing another independent ready input its turn.
|
|
1030
|
+
Native paths sharing a stream retain discovery precedence until the earlier
|
|
1031
|
+
input reaches its fixed end, pending state or refusal. AW files may rotate
|
|
1032
|
+
between bounded units in their shared stream; the existing retry selector still
|
|
1033
|
+
applies stream-scoped backoff and admits no concurrent writer. The queue does
|
|
1034
|
+
not clear failure rows or treat a finished inventory containing pending,
|
|
1035
|
+
set-aside or failed inputs as a successful pass. Directory metadata work and
|
|
1036
|
+
queue metadata work remain separate from source/journal content work.
|
|
1037
|
+
|
|
1038
|
+
`capture-source-discovery.mjs` connects an explicit ordered list of ordinary
|
|
1039
|
+
`{ source, path }` roots to that queue. A persisted plan identity rejects a
|
|
1040
|
+
changed root list within an unfinished sweep. Discovery rotates among cc, pi,
|
|
1041
|
+
codex and AW, retaining caller root order within each format. Native traversal
|
|
1042
|
+
uses the existing format layouts and stable `localeCompare` order; AW uses
|
|
1043
|
+
code-unit sorted basenames. Each unit visits at most 128 metadata entries,
|
|
1044
|
+
including irrelevant names, and inserts its selected files with the updated
|
|
1045
|
+
walk position in one capture-state transaction. Its `added` count describes
|
|
1046
|
+
the selected page, including any idempotent replay, not newly captured files
|
|
1047
|
+
or processing progress. A failed commit discards volatile walkers so the next
|
|
1048
|
+
unit reloads the durable checkpoint.
|
|
1049
|
+
|
|
1050
|
+
Directory-name arrays remain proportional to the selected directories and may
|
|
1051
|
+
be rebuilt after worker replacement. Native walkers cache only active directory
|
|
1052
|
+
stacks; AW caches one directory per current root. These are metadata costs, not
|
|
1053
|
+
a bounded metadata-memory guarantee. Selected directory identities are checked
|
|
1054
|
+
on reopening and before each page, including cached native ancestors. A fresh
|
|
1055
|
+
missing optional root can enumerate as empty; a selected root cannot disappear
|
|
1056
|
+
or be replaced silently. AW resumes by basename even when its anchor vanishes;
|
|
1057
|
+
native traversal retains its conservative replay rule. Earlier arrivals can
|
|
1058
|
+
belong to a later sweep. Closing or failing the bridge releases its caches.
|
|
1059
|
+
|
|
1060
|
+
The owning caller still supplies the queue's selection identity and either
|
|
1061
|
+
explicit roots or a stable default-root plan. `capture-default-roots.mjs`
|
|
1062
|
+
constructs the latter without filesystem work: explicit caller home/cwd,
|
|
1063
|
+
selected formats, and only the native location environment keys. It does not
|
|
1064
|
+
forward unrelated environment, credentials or Node runtime options. The
|
|
1065
|
+
existing format selectors retain their configured-root refusal, relative-path,
|
|
1066
|
+
Pi tilde/override precedence and Claude profile order. AW retains its home-based
|
|
1067
|
+
log directory rather than inheriting XDG selection. The public synchronous
|
|
1068
|
+
selectors and capture library are unchanged.
|
|
1069
|
+
|
|
1070
|
+
Default-root resolution is a separate owned discovery unit before file pages.
|
|
1071
|
+
Its durable attempt binds the same sweep/plan/run/worker and unresolved format
|
|
1072
|
+
group before the selector performs stat/readdir/sort. The private attempt's
|
|
1073
|
+
`roots` phase identifies selection, and its path is the caller home context,
|
|
1074
|
+
not an attributed journal or a claimed selected source path. Success commits
|
|
1075
|
+
the format's ordered root list and directory identities with attempt retirement;
|
|
1076
|
+
replacement reuses that list rather than rerunning selection. A new profile
|
|
1077
|
+
appearing afterward belongs to a later sweep. Changed caller configuration
|
|
1078
|
+
within an unfinished plan is refused. Excluded formats require no selector
|
|
1079
|
+
I/O. A default-root plan cannot resolve without owning-run/worker context;
|
|
1080
|
+
direct explicit-root readers retain their existing defaults.
|
|
1081
|
+
|
|
1082
|
+
Caught resolution errors and killed attempts use the existing format rotation,
|
|
1083
|
+
pending failure/backoff and replacement reconciliation. An unresolved format
|
|
1084
|
+
keeps its original selection inputs, not an invented partial root list. A due
|
|
1085
|
+
resolution retry reruns that selector; successfully publishing the selection
|
|
1086
|
+
resolves that operation's private retry, not the later file enumeration or
|
|
1087
|
+
processing. The failed pass's failure evidence is retained. Publication rollback
|
|
1088
|
+
keeps the unresolved attempt recoverable. Shared storage errors still propagate.
|
|
1089
|
+
Once roots are published, their directory witnesses are carried into the first
|
|
1090
|
+
reader and checked on later pages: deletion or replacement cannot become an
|
|
1091
|
+
empty default merely because enumeration had not started. A fresh absent AW
|
|
1092
|
+
default remains optional. No selection result itself grants completion.
|
|
1093
|
+
|
|
1094
|
+
Default-root selection may materialize and sort the home directory and retain
|
|
1095
|
+
a list of matching profiles; this is proportional metadata cost, not a bounded
|
|
1096
|
+
memory claim or transcript parsing. Each selected format adds one admission and
|
|
1097
|
+
one publication transaction before its file pages; retries/reconciliation and
|
|
1098
|
+
directory identity stats add work as well. All belong in hosted allowance and
|
|
1099
|
+
storage/synchronization qualification. The ordinary CLI uses this plan, including
|
|
1100
|
+
append-only `--no-index`. Protected exact-home/file operations retain their
|
|
1101
|
+
separate synchronous path. Foreground watch may also enumerate roots in the
|
|
1102
|
+
parent for advisory notification hints; those hints cannot change this plan.
|
|
1103
|
+
|
|
1104
|
+
Descriptors do not authorize access to protected captured history: existing
|
|
1105
|
+
path guards apply, and this ordinary-root bridge accepts no custody-proof
|
|
1106
|
+
fields. Protected exact-home/file frontdoors retain their own proof contracts.
|
|
1107
|
+
Discovery completion means only enumeration ended; queued source processing,
|
|
1108
|
+
newly created streams, retries, cleanup and final index validation remain
|
|
1109
|
+
separate obligations.
|
|
1110
|
+
|
|
1111
|
+
When the owning session supplies its run and registered worker identity, each
|
|
1112
|
+
selected discovery unit first commits an attempt bound to that worker, run,
|
|
1113
|
+
sweep, plan, format/root and original checkpoint. This intent is not enumeration
|
|
1114
|
+
or processing progress. Reader work runs outside that transaction. Successful
|
|
1115
|
+
page/checkpoint publication retires the exact attempt in the same transaction;
|
|
1116
|
+
publication rollback preserves both the original checkpoint and the attempt.
|
|
1117
|
+
Before scheduling or opening run accounting, the registered replacement
|
|
1118
|
+
reconciles an ended predecessor's attempt into private pending root failure and
|
|
1119
|
+
backoff. The original run owns that failure; an observed current-pass process
|
|
1120
|
+
failure cannot be assigned to an older admission. Same-worker reconciliation
|
|
1121
|
+
and changed scope/plan/checkpoint are refused. This relies on the owning
|
|
1122
|
+
bootstrap ending the predecessor before registering its replacement.
|
|
1123
|
+
|
|
1124
|
+
A caught reader refusal likewise commits private root-failure evidence and
|
|
1125
|
+
format rotation while retiring its attempt before propagating the error.
|
|
1126
|
+
The refused root keeps its exact checkpoint and stays incomplete. Its format
|
|
1127
|
+
cannot advance to later roots, but a replacement worker can discover other
|
|
1128
|
+
formats and process independent queued capture, index and cleanup work. The
|
|
1129
|
+
failure remains bound to the failed pass; later healthy work cannot clear it
|
|
1130
|
+
or invent a failed journal stream. Direct readers without owning-run context
|
|
1131
|
+
continue to throw without publishing failure state.
|
|
1132
|
+
|
|
1133
|
+
Failed roots use the fixed 30/60/120/240-minute schedule and exclude the pass
|
|
1134
|
+
that attempted them. Due discovery follows independent healthy work and
|
|
1135
|
+
eligible stream/cleanup retries. One partial retry page per pass preserves its
|
|
1136
|
+
attempt/date and marks that pass attempted; only completing the refused root
|
|
1137
|
+
retires its pending failure. Restoring a selected directory permits recovery;
|
|
1138
|
+
a different directory never inherits its checkpoint. A deferred root keeps
|
|
1139
|
+
enumeration pending without blocking unrelated due work. These guarantees
|
|
1140
|
+
require functioning shared capture state: failure-evidence publication and
|
|
1141
|
+
page/checkpoint storage errors propagate, and cannot promise independent
|
|
1142
|
+
progress. An unfinished attempt can be reconciled only once the shared state
|
|
1143
|
+
is usable again. Owned discovery adds one admission transaction per selected
|
|
1144
|
+
page; page publication and retirement share the existing page transaction
|
|
1145
|
+
(with its nested queue savepoint). Final storage accounting must include these
|
|
1146
|
+
commits, retry/reconciliation transactions and actual synchronization cost;
|
|
1147
|
+
no lower fsync-cost claim follows from connection reuse.
|
|
1148
|
+
|
|
1149
|
+
`capture-source-work.mjs` joins one selected source group to those queue
|
|
1150
|
+
checkpoints and `CaptureRun` accounting. Locks precede the admission commit and
|
|
1151
|
+
processing transaction. Each input/category contributes to its run's counters
|
|
1152
|
+
once even when recovery spans workers; the visit flags live in derived database
|
|
1153
|
+
rows, not a process-wide file map. No-op visits do not count as processing
|
|
1154
|
+
progress. One AW account per shared-stream admission preserves serialized
|
|
1155
|
+
appends while the queue position allows other accounts their turn. A due retry
|
|
1156
|
+
is a single-stream admission, and an incomplete proof cannot clear its failure.
|
|
1157
|
+
Other-stage retry work returns to the scheduler rather than being mistaken for
|
|
1158
|
+
capture work. Allocation cleanup remains a separate committed-lifecycle step.
|
|
1159
|
+
|
|
1160
|
+
These private primitives do not themselves hold a root lock or implement the
|
|
1161
|
+
complete command runner. Their caller must own that lock, enforce the common
|
|
1162
|
+
budget, reconcile unfinished admissions, and publish the actual run outcome.
|
|
1163
|
+
|
|
1164
|
+
## Private worker dispatch
|
|
1165
|
+
|
|
1166
|
+
`CaptureWorkSession` joins source/index processing, reset handoff and run
|
|
1167
|
+
accounting inside an already registered supervisor worker. Its bootstrap caller
|
|
1168
|
+
must own the root lock and have ended the predecessor before construction. The
|
|
1169
|
+
index-enabled session shares one capture-state connection and one search-index connection,
|
|
1170
|
+
reconciles durable stream, cleanup and union-reset admissions, then opens current run
|
|
1171
|
+
accounting. A recorded failure is bound to its run identity: an observed failure
|
|
1172
|
+
in the new pass cannot diagnose an older abandoned admission. Prior history
|
|
1173
|
+
remains distinct from failure in the current pass, including a child that died
|
|
1174
|
+
before publishing any receipt. Persisted reconciliation facts survive a second
|
|
1175
|
+
interruption before run publication.
|
|
1176
|
+
|
|
1177
|
+
With an explicit root plan, each dispatch step advances at most one bounded
|
|
1178
|
+
source-discovery page, one index-discovery page, one source or index group and
|
|
1179
|
+
one healthy cleanup turn after a selected group. Without a root plan, the
|
|
1180
|
+
caller supplies source pages as before. The last durable admission's stage
|
|
1181
|
+
determines which processing stage goes first next, including after a rolled-back
|
|
1182
|
+
group; no extra per-record or per-unit routing transaction is added. Source groups register new
|
|
1183
|
+
streams in the current index inventory as part of their processing commit.
|
|
1184
|
+
Cleanup uses its own allocation admission and cursor transactions; it does not
|
|
1185
|
+
replace the durable source/index stage position or claim lower sync cost.
|
|
1186
|
+
|
|
1187
|
+
Healthy selection in both processing stages suppresses retries. Only after both
|
|
1188
|
+
healthy stages and eligible discovery are exhausted does the session route a
|
|
1189
|
+
due retry through its own stage as a single-stream admission. Retry filtering retains
|
|
1190
|
+
unavailable capture inputs and other-stage work without changing their attempt
|
|
1191
|
+
or retry date, so they do not hide eligible source/index retries. Selection
|
|
1192
|
+
checks cooperative stop between bounded metadata pages/rows. Stage helpers keep
|
|
1193
|
+
their earlier default retry behavior when used without this dispatcher.
|
|
1194
|
+
|
|
1195
|
+
When no healthy group is selected, the session advances healthy cleanup pages
|
|
1196
|
+
before due work. Due cleanup follows healthy source/index work, eligible source
|
|
1197
|
+
and index discovery, and source/index retries. Cleanup can progress without a
|
|
1198
|
+
ready source. A cleanup exception leaves the already-committed group counts
|
|
1199
|
+
intact; replacement reconciliation applies backoff while independent healthy
|
|
1200
|
+
work remains eligible. A completed page or idle selection does not prove that
|
|
1201
|
+
every deleting reference has finished, especially across cursor wraparound.
|
|
1202
|
+
|
|
1203
|
+
A native input with its exact `stream:path` recovery claim in deleting phase
|
|
1204
|
+
depends on that allocation's cleanup. Selection rechecks the durable claim and
|
|
1205
|
+
excludes that input from both healthy capture and due capture retry until the
|
|
1206
|
+
claim retires, even if its directory is already absent. The source stays ready;
|
|
1207
|
+
neither its completion nor retry resolution is invented. Independent source and
|
|
1208
|
+
index work can drain before due allocation cleanup, after which the next
|
|
1209
|
+
selection can resume the native input. This avoids repeated cleanup-only source
|
|
1210
|
+
admissions starving the allocation's own backoff queue. Unrelated paths/streams
|
|
1211
|
+
and existing stream retry evidence retain their normal scheduling rules.
|
|
1212
|
+
|
|
1213
|
+
A fresh reset worker dispatches the union operation before ordinary work. A
|
|
1214
|
+
committed reset reopens the published pending inventory; an interrupted one is
|
|
1215
|
+
reconciled before processing. Detection requests a new worker rather than
|
|
1216
|
+
resetting in place. Deferred reset leaves independent source work eligible while
|
|
1217
|
+
index processing is blocked. Finalization can recover a committed reset's
|
|
1218
|
+
bounded metadata publication, but performs no discovery, source/index group,
|
|
1219
|
+
cleanup deletion or new reset.
|
|
1220
|
+
|
|
1221
|
+
Yielding publishes pending run facts. `finish()` is a conservative stop receipt,
|
|
1222
|
+
including for a finalization-only worker; it does not infer success from idle
|
|
1223
|
+
selection. The owning normal worker calls `complete()` for the separate final
|
|
1224
|
+
decision. `capture-completion.mjs` verifies source enumeration and outcomes,
|
|
1225
|
+
discovery/root failures, stream/cleanup/reset/discovery admissions, retries,
|
|
1226
|
+
retained allocations/stages, issue rows and obsolete inventory cleanup. Only
|
|
1227
|
+
complete or ignored source outcomes can contribute to success. Every current
|
|
1228
|
+
index outcome is checked against its journal generation, validation and index
|
|
1229
|
+
cursor evidence again. A fresh index-name/live-name union check adds missing
|
|
1230
|
+
members as ready work, including journals arriving before an earlier discovery
|
|
1231
|
+
cursor; they require processing before a later audit can succeed. Neither idle,
|
|
1232
|
+
reset publication nor a completed enumeration is a completion certificate.
|
|
1233
|
+
|
|
1234
|
+
The audit uses SQL pages of at most 128 rows and cooperative checks between
|
|
1235
|
+
metadata entries. Sorted live directory names retain the existing metadata-array
|
|
1236
|
+
cost qualification. Capture-state then index write reservations prevent supported
|
|
1237
|
+
database writers from changing the checked state during the audit. It acquires
|
|
1238
|
+
no stream locks and reads no journal content while holding those reservations.
|
|
1239
|
+
Journal evidence retains its per-stream metadata qualification; this is not a
|
|
1240
|
+
root-wide instantaneous filesystem snapshot or a new historical integrity proof
|
|
1241
|
+
against arbitrary raw file edits. Directory identity replacement is refused;
|
|
1242
|
+
observed directory changes or newly discovered members leave pending work.
|
|
1243
|
+
|
|
1244
|
+
Audit progress is worker-local: an interruption or deadline cannot turn a saved
|
|
1245
|
+
partial scan into later completion authority. A replacement verifies afresh.
|
|
1246
|
+
Source/content/index recovery and discovery retain their resumable checkpoints.
|
|
1247
|
+
The final metadata audit must fit one admitted normal worker; its lock duration,
|
|
1248
|
+
directory metadata memory and 1,300/thousands-stream completion time remain
|
|
1249
|
+
explicit hosted Linux/storage qualification gates. No measured bound or final
|
|
1250
|
+
service qualification follows from this private implementation. Each audit adds
|
|
1251
|
+
one capture-state transaction and an index reservation released without index
|
|
1252
|
+
mutation. Missing memberships or a retirement intent can add capture-state
|
|
1253
|
+
writes; include their real synchronization cost in storage accounting.
|
|
1254
|
+
|
|
1255
|
+
Terminal source visits can request retirement even when a torn, set-aside or
|
|
1256
|
+
failed input makes the receipt partial. The intent is bound to the source scope
|
|
1257
|
+
and run. It takes effect only on a later normal construction that sees that run's
|
|
1258
|
+
published final receipt; death before publication preserves the same source
|
|
1259
|
+
scope and confirmed counters. The next source sweep also renews index discovery.
|
|
1260
|
+
Old source/index inventories retire in bounded pages after healthy work, without
|
|
1261
|
+
manufacturing capture/index progress or counts. Pending inputs are thus revisited
|
|
1262
|
+
on a later sweep without being treated as successful in the earlier one.
|
|
1263
|
+
|
|
1264
|
+
An empty healthy audit may publish complete. Current-pass failure still overrides
|
|
1265
|
+
healthy work and complete audit evidence; an older failure remains history and
|
|
1266
|
+
does not poison a later success. Confirmed counts come only from their existing
|
|
1267
|
+
processing commits, never audit scans or reconstruction. Ordinary CLI dispatch
|
|
1268
|
+
uses this session. Protected exact-home/file operations retain their existing
|
|
1269
|
+
path; explicit hooks use a detached caller of the same session. Installed
|
|
1270
|
+
periodic service migration uses the setup gate below.
|
|
1271
|
+
|
|
1272
|
+
### Append-only worker mode
|
|
1273
|
+
|
|
1274
|
+
The private session's `indexEnabled: false` option implements the supervised
|
|
1275
|
+
ordinary-command path for `--no-index`. The default remains enabled.
|
|
1276
|
+
A disabled session never constructs or opens `RecordIndex`, including during
|
|
1277
|
+
replacement or finalization. It does not create `turns.db`, register new stream
|
|
1278
|
+
membership, instantiate/rebuild index inventory, dispatch/reconcile union reset,
|
|
1279
|
+
retire index inventory or modify its source binding. Existing index/reset
|
|
1280
|
+
requests, retries and outcome evidence remain for a later enabled pass. Common
|
|
1281
|
+
stream-admission reconciliation still records an abandoned index group in
|
|
1282
|
+
capture state; its deferred index retry is not processed or erased by capture.
|
|
1283
|
+
|
|
1284
|
+
Source/default-root discovery, source capture, retained-allocation cleanup and
|
|
1285
|
+
their eligible retries keep their admission, dependency, backoff and fairness
|
|
1286
|
+
rules. Index/reset work cannot block independent healthy capture. Housekeeping
|
|
1287
|
+
collects only obsolete source scopes, leaving index garbage and inventory intact.
|
|
1288
|
+
Source-sweep renewal does not invalidate that retained index evidence; an
|
|
1289
|
+
enabled session subsequently reconciles its source binding and discovers all
|
|
1290
|
+
live/indexed names before claiming completion.
|
|
1291
|
+
|
|
1292
|
+
Append-only `complete()` performs a fresh source-only retirement audit, never
|
|
1293
|
+
the all-stage/index audit. It checks terminal visits and source discovery,
|
|
1294
|
+
refuses retirement while source/stream, cleanup or discovery admission remains,
|
|
1295
|
+
and publishes an intent bound to the source scope and run with no index scope.
|
|
1296
|
+
An unfinished or failed discovery group prevents this intent. As with full
|
|
1297
|
+
audits, terminal unsuccessful input visits may be renewed only after a final
|
|
1298
|
+
receipt, and interruption before that receipt retains the same scope/counters.
|
|
1299
|
+
This audit adds one capture-state transaction, paged source-row inspection and
|
|
1300
|
+
an optional retirement write, with no index reservation. A stopped scan provides
|
|
1301
|
+
no reusable proof. Its normal-worker allowance, reservation duration and actual
|
|
1302
|
+
synchronization costs remain hosted storage-qualification obligations.
|
|
1303
|
+
|
|
1304
|
+
Even an empty or fully visited append-only sweep remains pending/partial, or
|
|
1305
|
+
failed for a current-pass failure: it supplies no fresh index/all-stage proof.
|
|
1306
|
+
Prior success/failure history and confirmed counts remain distinct from that
|
|
1307
|
+
decision. The ordinary CLI uses this mode without changing public synchronous
|
|
1308
|
+
APIs or protected exact-home/file receipts. Explicit hooks can select this mode;
|
|
1309
|
+
installed service integration and hosted qualification remain separate obligations.
|
|
1310
|
+
|
|
1311
|
+
### Fixed private command entry and issue preparation
|
|
1312
|
+
|
|
1313
|
+
`runCaptureCommand(root, options)` fixes the package worker entry and existing
|
|
1314
|
+
supervisor policy. Its private options require an owner, caller-bound default
|
|
1315
|
+
root plan, index-enabled boolean, explicit trigger and explicit retry boolean.
|
|
1316
|
+
Unknown options cannot supply a different worker path, executable, Node flags
|
|
1317
|
+
or timing/resource policy. Relative record roots resolve against the supplied
|
|
1318
|
+
caller cwd; source selectors still run only after worker admission. Triggers
|
|
1319
|
+
are manual, watch, hook, setup or service, never inferred from quiet output.
|
|
1320
|
+
The fixed worker requires the existing IPC/registered-lock bootstrap; direct
|
|
1321
|
+
invocation does not authorize capture.
|
|
1322
|
+
|
|
1323
|
+
The operation constructs the owning session, dispatches its bounded units,
|
|
1324
|
+
yields for fresh reset/continuation allowance and requests the actual final
|
|
1325
|
+
audit on idle. Only the worker publishes run facts. A finalized matching
|
|
1326
|
+
receipt supplies the private IPC appended/pending projection while the
|
|
1327
|
+
supervisor still holds the root lock; the caller neither reconstructs missing
|
|
1328
|
+
counts nor reads another pass's later status as its own. Operational failures
|
|
1329
|
+
leave admissions and unfinished receipts for replacement, while cooperative
|
|
1330
|
+
stops retain the existing pending continuation behavior. Transport failure may
|
|
1331
|
+
return without that receipt projection and must not become a zero-count success.
|
|
1332
|
+
|
|
1333
|
+
Before source enumeration/processing, the command session prepares issue rows
|
|
1334
|
+
in keyset pages of at most 128. Each page commits issue changes, any existing
|
|
1335
|
+
queued set-aside rows made ready, and its resume key together. The checkpoint
|
|
1336
|
+
binds owner, source sweep, run, selected formats and retry request. Replacement
|
|
1337
|
+
resumes committed pages; rollback keeps both the cursor and issue generations
|
|
1338
|
+
unchanged. Preparation supplies no capture/index counts or progress.
|
|
1339
|
+
|
|
1340
|
+
Ordinary preparation retains the synchronous native reconciliation rule:
|
|
1341
|
+
selected-owner, selected-format ignored issues can be removed without opening
|
|
1342
|
+
their source, even if the default directory disappeared. Other owners/formats
|
|
1343
|
+
remain untouched by that exclusion. Explicit all-input retry requires the full
|
|
1344
|
+
format selection and marks the root's unresolved issues as the existing public
|
|
1345
|
+
retry operation does, including other owners' retry intent without capturing
|
|
1346
|
+
under another owner. The run identity is the stable private retry generation;
|
|
1347
|
+
once prepared, a refusal in the same run is not marked for retry again.
|
|
1348
|
+
Existing source/index/cleanup failure backoff is separate and never cleared.
|
|
1349
|
+
Public synchronous `retrySetAside` and `resetSourceCursors` are unchanged.
|
|
1350
|
+
|
|
1351
|
+
Preparation finishes before content dispatch so newly refused inputs cannot
|
|
1352
|
+
re-enter its initial scan. Stop/finalization cannot start a page. Both completion
|
|
1353
|
+
audits refuse success/retirement while preparation is unfinished. This adds one
|
|
1354
|
+
capture-state transaction per page, including its checkpoint and per-issue
|
|
1355
|
+
generation updates; ignore loading/matching and total metadata scan/lock/sync
|
|
1356
|
+
costs remain hosted qualification obligations. Pages resume across workers,
|
|
1357
|
+
but preparation is not source enumeration or all-stage completion.
|
|
1358
|
+
|
|
1359
|
+
### Ordinary CLI and foreground watch
|
|
1360
|
+
|
|
1361
|
+
Ordinary capture, `--sessions-only`, `--aw-only`, `--no-index` and
|
|
1362
|
+
`--retry-set-aside` await the fixed private entry. The CLI copies home, cwd and
|
|
1363
|
+
the selected native location environment into its plan. Quiet controls output,
|
|
1364
|
+
never trigger identity: ordinary and explicit retry use `manual`; the first
|
|
1365
|
+
foreground watch pass and later ticks use `watch`. Legacy quiet hook argv
|
|
1366
|
+
remains manual; the setup first pass explicitly uses `setup`. Quiet
|
|
1367
|
+
never invents hook attribution or a timeout cause. `--reset-cursors` remains reset-only;
|
|
1368
|
+
exact-home/file proofs and public synchronous libraries keep their paths.
|
|
1369
|
+
|
|
1370
|
+
Ordinary `--json` remains a usage error. Retry JSON retains exactly
|
|
1371
|
+
`{appended, pending}` from the matching worker projection. No caller reads a
|
|
1372
|
+
later status snapshot to fill a missing projection. Operational failure exits
|
|
1373
|
+
nonzero; a missing projection explicitly says the appended count is unknown.
|
|
1374
|
+
An `io` failure line (`capture worker failed (io): <message>`) carries the first
|
|
1375
|
+
failing worker's one-line diagnostic, bounded to 512 characters with control
|
|
1376
|
+
characters removed and parser errors reduced to their class; it reaches stderr
|
|
1377
|
+
only, never the published status, whose failure message stays fixed.
|
|
1378
|
+
A dead lock owner whose recorded start is missing, malformed or in the future
|
|
1379
|
+
is still reclaimed and reported on stderr, but published as `reclaimedLock: null`.
|
|
1380
|
+
Partial work exits zero without claiming completion. Ordinary lock overlap
|
|
1381
|
+
exits zero and preserves the existing receipt; recovery overlap instead reports
|
|
1382
|
+
that recovery was not performed and exits one. Lock reclamation, uncertain
|
|
1383
|
+
holders, initialization and release failures retain operator diagnostics.
|
|
1384
|
+
Nonquiet progress reports confirmed new turns and outcome; `index: updated`
|
|
1385
|
+
is emitted only for an enabled, fully completed pass, never a partial one.
|
|
1386
|
+
|
|
1387
|
+
Foreground watch retains its two-second debounce and fifteen-minute fallback.
|
|
1388
|
+
Its parent may enumerate selected default directories to install best-effort
|
|
1389
|
+
filesystem notification hints. This metadata work is proportional to profile
|
|
1390
|
+
inventory and is not a worker memory/allowance claim. Neither a hint nor its
|
|
1391
|
+
absence selects capture inputs or grants custody; only the admitted saved plan
|
|
1392
|
+
does that. Enumeration/installation errors leave periodic reconciliation active.
|
|
1393
|
+
Notifications coalesce while a pass runs; at most one pass is active, with one
|
|
1394
|
+
pending rerun. Stop cancels timers and future work, closes watchers, and leaves
|
|
1395
|
+
an active supervisor responsible for child termination/reaping and lock release.
|
|
1396
|
+
A failed later tick reports the error and retains a nonzero eventual exit status
|
|
1397
|
+
while periodic scheduling continues. It cannot strand the queue as busy.
|
|
1398
|
+
|
|
1399
|
+
The small-heap algorithm fixture invokes synchronous capture/index libraries
|
|
1400
|
+
directly: lowering the CLI parent's heap cannot lower its fixed-policy worker
|
|
1401
|
+
heap. Actual CLI fixtures separately cover worker routing, selected capture,
|
|
1402
|
+
append-only catch-up, recovery receipts, overlap and foreground scheduling.
|
|
1403
|
+
Package health and [kernel doctor delegation](#kernel-doctor-delegation) are
|
|
1404
|
+
implemented; health/doctor runtime qualification, fresh-audit/parent
|
|
1405
|
+
metadata/storage/synchronization and full hosted manager and rollback
|
|
1406
|
+
qualification are still required before service acceptance.
|
|
1407
|
+
|
|
1408
|
+
### Detached hook launcher
|
|
1409
|
+
|
|
1410
|
+
`capture --hook` copies owner, home/cwd, selected native location environment,
|
|
1411
|
+
source selection and index mode into the fixed package supervisor's argument.
|
|
1412
|
+
It performs argument validation and OS spawn only: no capture lock, status
|
|
1413
|
+
read/write, network, default-directory resolution or source scan. Root selection
|
|
1414
|
+
still belongs to the admitted worker. Caller options cannot replace the Node
|
|
1415
|
+
executable, package entry, worker or resource policy. The supervisor starts in
|
|
1416
|
+
its own session/process group with all three standard streams ignored and a
|
|
1417
|
+
filtered environment; it does not retain harness pipes or Node preload options.
|
|
1418
|
+
The launcher unrefs it after the spawn event and exits. Killing the launcher
|
|
1419
|
+
after spawn does not terminate that detached child.
|
|
1420
|
+
|
|
1421
|
+
Exit zero means **capture launched**, not supervisor initialization, admission
|
|
1422
|
+
or capture completion. Nonquiet acknowledgement says so; quiet suppresses it.
|
|
1423
|
+
Spawn or argument failure prints one stderr line and exits one. Hook mode never
|
|
1424
|
+
uses exit two, including usage errors, because Claude interprets that status as
|
|
1425
|
+
Stop-blocking/model feedback. Ordinary usage errors outside hook mode retain
|
|
1426
|
+
their existing status. Proof, recovery, JSON, health, watch and reporting modes
|
|
1427
|
+
cannot be combined with `--hook`. Sessions-only/AW-only and append-only flags
|
|
1428
|
+
are copied explicitly, without resolving inputs in the launcher.
|
|
1429
|
+
|
|
1430
|
+
The detached supervisor owns the usual root lock, registered workers, failure
|
|
1431
|
+
handling, child reaping and lock release. Admitted worker receipts use trigger
|
|
1432
|
+
`hook`; overlapping launches do not create another pass or overwrite history.
|
|
1433
|
+
The launcher writes no receipt. Failure after OS spawn but before a worker can
|
|
1434
|
+
publish status may leave absent or older status, and ignored stderr cannot
|
|
1435
|
+
report that failure back to the completed harness call. Launch acknowledgement
|
|
1436
|
+
is therefore not a health check: use `capture --health` or deployment doctor.
|
|
1437
|
+
Hosted lifecycle qualification remains a separate gate. No numeric launch
|
|
1438
|
+
latency is established by this implementation.
|
|
1439
|
+
|
|
1440
|
+
Setup refreshes recognized owned Stop/SessionEnd entries to the hook entry,
|
|
1441
|
+
`<node> <package>/bin/capture-hook.mjs --owner <owner> --quiet --root <root>`
|
|
1442
|
+
(the same launch as `capture --quiet --hook`), with `async:false` and
|
|
1443
|
+
`timeout:10`, even when their command already matches. The hook runs its own
|
|
1444
|
+
entry because 0.61.0 and earlier releases do not have that file: after a
|
|
1445
|
+
rollback the persisted line fails to load with exit 1, which does not block
|
|
1446
|
+
Stop or SessionEnd, where an older `capture.mjs` would reject `--hook` with
|
|
1447
|
+
exit 2. Setup recognizes the `capture.mjs` forms earlier releases wrote and
|
|
1448
|
+
replaces them in place. The
|
|
1449
|
+
harness waits for the short launcher so its spawn result is ordered before
|
|
1450
|
+
harness shutdown; capture itself remains detached. The timeout applies to that
|
|
1451
|
+
launcher, not to the detached supervisor or its fixed worker allowance. The
|
|
1452
|
+
existing ownership recognition, duplicate groups, foreign commands and other
|
|
1453
|
+
fields are preserved; dry-run does not write settings. The same user
|
|
1454
|
+
`settings.json` (the person's `~/.claude`, or the Claude config a launch uses)
|
|
1455
|
+
has a second writer, oats.setup's `/oats-agent-permissions`, which merges only
|
|
1456
|
+
`permissions` and the `autoMode` block after keeping a `.before-oats` copy;
|
|
1457
|
+
setup touches only the capture `hooks` entries, so the two do not conflict. The install hint uses
|
|
1458
|
+
the same scheduling fields while retaining `--sessions-only --no-index`.
|
|
1459
|
+
This changes persisted settings from `async:true`/`timeout:120`; installing a
|
|
1460
|
+
CLI alone does not rewrite those settings. No release from 0.56.0 through 0.61.0
|
|
1461
|
+
shipped this hook migration. Periodic service conversion uses
|
|
1462
|
+
the separate setup gate below.
|
|
1463
|
+
|
|
1464
|
+
### Setup first pass
|
|
1465
|
+
|
|
1466
|
+
After its requested installation steps, non-dry setup awaits the same fixed
|
|
1467
|
+
private supervisor as ordinary capture, with explicit `setup` trigger,
|
|
1468
|
+
all sources and indexing enabled. It copies caller home/cwd and native location
|
|
1469
|
+
environment; the admitted worker resolves source directories. The first pass
|
|
1470
|
+
uses the running `process.execPath`, not the stable Node pathname being written
|
|
1471
|
+
into future hooks or manager definitions. `--no-service` skips only manager
|
|
1472
|
+
files and manager inspection/actions; it does not skip this pass. `--dry-run` starts no worker and
|
|
1473
|
+
creates no capture store, lock or status.
|
|
1474
|
+
|
|
1475
|
+
Setup retains the foreground privacy-file preflight before lock acquisition.
|
|
1476
|
+
Shared foreground presentation uses only the matching worker projection for
|
|
1477
|
+
counts and pending state. Complete and partial return zero with their actual
|
|
1478
|
+
outcome; overlap returns zero with the holder/recovery and skip message without
|
|
1479
|
+
rewriting the holder's receipt. Operational failures and missing projections
|
|
1480
|
+
return nonzero; absent counts are unknown, never fabricated zero. Initialization,
|
|
1481
|
+
reclamation and release diagnostics retain their existing recovery guidance.
|
|
1482
|
+
The supervisor owns signal handling, child reaping and lock release; setup is
|
|
1483
|
+
not another status writer. Ordinary/retry text and JSON, protected exact-home/file
|
|
1484
|
+
receipts and public synchronous APIs retain their contracts.
|
|
1485
|
+
|
|
1486
|
+
First-pass routing follows the service migration gate when service installation
|
|
1487
|
+
is requested. It does not provide health/doctor acceptance or hosted
|
|
1488
|
+
Linux/manager/rollback qualification. The shared presentation
|
|
1489
|
+
adds no status transaction or counter reconstruction; existing source/audit
|
|
1490
|
+
allowance, reservation and actual storage/synchronization costs remain gates.
|
|
1491
|
+
|
|
1492
|
+
### Finite service entry and periodic definitions
|
|
1493
|
+
|
|
1494
|
+
The private `capture-service-runner.mjs` entry takes one serialized request with
|
|
1495
|
+
explicit record root, owner and default-root plan. Relative record roots resolve
|
|
1496
|
+
against the copied caller cwd. Home, cwd and the four native location values
|
|
1497
|
+
come from that plan rather than the service process's ambient source selection.
|
|
1498
|
+
It requires all formats and fixes indexing enabled, retry preparation off and
|
|
1499
|
+
trigger `service`; the request cannot select an executable, worker or resource
|
|
1500
|
+
policy. After validation it reads the privacy policy before acquiring a lock,
|
|
1501
|
+
then uses the fixed supervisor and shared foreground presentation. Complete,
|
|
1502
|
+
partial, overlap, operational failure and absent receipt retain the setup/manual
|
|
1503
|
+
meanings above. There is no second status writer, repeated pass loop or manager
|
|
1504
|
+
operation in this entry.
|
|
1505
|
+
|
|
1506
|
+
`capture-service-definition.mjs` constructs definitions without filesystem or
|
|
1507
|
+
manager access. Both formats carry `oats-turn-record-service:1`. The recorded
|
|
1508
|
+
absolute Node runs the fixed package entry with the explicit request as one
|
|
1509
|
+
argument, with XML/systemd-specific escaping and refusal of values the selected
|
|
1510
|
+
manager cannot represent safely. No shell evaluates the request. XML preserves
|
|
1511
|
+
literal carriage returns through character references; systemd arguments escape
|
|
1512
|
+
both specifier and environment expansion syntax. Unsupported executable quote
|
|
1513
|
+
or backslash spellings are refused for systemd rather than silently rewritten.
|
|
1514
|
+
|
|
1515
|
+
The macOS definition names `ai.aweb.turn-record-capture`, uses `RunAtLoad` and
|
|
1516
|
+
`StartInterval` 900, and has no `KeepAlive` or watch argument. Linux pairs a finite
|
|
1517
|
+
`turn-record-capture.service` (`Type=exec`, `MemoryMax=2G`, `RuntimeMaxSec=300`, no
|
|
1518
|
+
restart loop) with `turn-record-capture.timer` (`OnBootSec=1min`,
|
|
1519
|
+
`OnUnitActiveSec=15min`). A timer/schedule is not a permanent worker PID.
|
|
1520
|
+
|
|
1521
|
+
Setup uses these definitions through the migration gate below. Constructing a
|
|
1522
|
+
definition alone proves neither installation nor enabled/registered scheduling.
|
|
1523
|
+
Source-only argument-decoding fixtures do not replace hosted manager tests;
|
|
1524
|
+
this entry introduces no additional capture-state transactions beyond the
|
|
1525
|
+
existing fixed command path.
|
|
1526
|
+
|
|
1527
|
+
### Setup service migration
|
|
1528
|
+
|
|
1529
|
+
`setup --root <dir> --owner <name>` persists the resolved root and owner in
|
|
1530
|
+
hooks and the service request, alongside explicit caller home/cwd/native
|
|
1531
|
+
location selection. The recorded stable Node is used for future launches;
|
|
1532
|
+
the first pass still uses the running Node. These are user-manager definitions,
|
|
1533
|
+
not deployment automation timers. No external wrapper is removed.
|
|
1534
|
+
|
|
1535
|
+
The setup coordinator prepares hook edits in memory, then
|
|
1536
|
+
`capture-service-install.mjs` snapshots every named service target and inspects
|
|
1537
|
+
applicable manager state before any service/file/hook/capture mutation.
|
|
1538
|
+
Snapshots retain regular-file bytes and identity or the symlink's own identity
|
|
1539
|
+
and target, plus ancestor identities and explicit missing-ancestor records. Reads are limited to 1 MiB per target;
|
|
1540
|
+
unknown reads, nonregular targets and unsupported manager output refuse.
|
|
1541
|
+
`capture-service-ownership.mjs` parses only supported marked templates and
|
|
1542
|
+
the actual unmarked direct-Node `capture.mjs --watch --owner` templates with
|
|
1543
|
+
their known structure. Paths may name a moved or missing install; recognition
|
|
1544
|
+
does not execute them. A basename, shell mention, wrapper, unsupported marker,
|
|
1545
|
+
malformed file or extra directive is not ownership.
|
|
1546
|
+
|
|
1547
|
+
Foreign files or registered identities produce `E_CAPTURE_SERVICE_FOREIGN`;
|
|
1548
|
+
installed disabled/masked state produces `E_CAPTURE_SERVICE_DISABLED`, whose
|
|
1549
|
+
message states the manager state observed. It does not claim the operator
|
|
1550
|
+
disabled the unit: a timer that setup wrote but failed to enable reads as
|
|
1551
|
+
disabled too. Both refuse before mutation and print quoted recovery commands
|
|
1552
|
+
in the user-facing form preserving the selected owner/root: `oats setup …`
|
|
1553
|
+
when a node was handed in through the kernel environment (`OATS_NODE_BIN`,
|
|
1554
|
+
which `oats setup` sets and instance shells also carry), otherwise
|
|
1555
|
+
`turn-record setup …`; both commands work and record the same node: explicitly choose
|
|
1556
|
+
`--replace-service`, or `--no-service` to preserve the manager configuration.
|
|
1557
|
+
Other setup failures print the same form of retry command; none names the
|
|
1558
|
+
running Node or the package-internal setup script. New absent definitions
|
|
1559
|
+
are not disabled. A macOS `launchctl print-disabled` entry for the label is the
|
|
1560
|
+
exception: launchd keeps that override after the plist is gone and setup never
|
|
1561
|
+
writes one, so it is operator state and refuses even when the named plist is
|
|
1562
|
+
absent. Setup reads only the disabled-services block of that output, accepting
|
|
1563
|
+
both `true`/`false` and `disabled`/`enabled` values; any other entry refuses. Replacement intent never waives unknown state, read errors,
|
|
1564
|
+
unsupported drop-ins, backup failure or identity changes. On macOS, a
|
|
1565
|
+
registration of `ai.aweb.turn-record-capture` loaded from any plist other than
|
|
1566
|
+
the named one is foreign and is refused naming that path, even with
|
|
1567
|
+
`--replace-service`; setup never unlinks or boots out that file. `--no-service`
|
|
1568
|
+
bypasses service target reads and manager calls entirely. `--dry-run` reads
|
|
1569
|
+
target files but writes nothing and calls no manager; unloaded/disabled state
|
|
1570
|
+
is explicitly unexamined; a dry run also names a present pending-activation
|
|
1571
|
+
marker and whether a run would retry it or remove it as stale.
|
|
1572
|
+
|
|
1573
|
+
Apply rechecks file/ancestor identities and manager ownership/enablement.
|
|
1574
|
+
Replaced bytes or symlinks are saved nonoverwriting in a reported datable
|
|
1575
|
+
`<root>/service-backups/<date>-<unique-id>/` directory. Symlink backups
|
|
1576
|
+
preserve the link instead of reading or removing an external wrapper target.
|
|
1577
|
+
An ancestor absent at preflight must remain absent until this apply creates it.
|
|
1578
|
+
Creation proceeds one directory at a time without recursive adoption of an
|
|
1579
|
+
existing path; each created non-link directory's identity is retained in a
|
|
1580
|
+
per-apply map shared by subsequent service and hook writes. Newly introduced
|
|
1581
|
+
directories or links are not accepted as extra ancestry. Existing ancestor
|
|
1582
|
+
identities and observed symlink targets remain binding. The extra metadata
|
|
1583
|
+
checks and directory creation do not establish an atomic filesystem operation.
|
|
1584
|
+
Each write uses a temporary file and another identity check before rename;
|
|
1585
|
+
cleanup also refuses changed ancestry instead of unlinking through a redirected
|
|
1586
|
+
path. An interrupted/refused write can leave its original temporary file.
|
|
1587
|
+
An expected manager unmask can remove only the saved named `/dev/null` link;
|
|
1588
|
+
other observed replacements refuse. These checks detect observed races, not
|
|
1589
|
+
an OS-wide atomic transaction over files and managers.
|
|
1590
|
+
|
|
1591
|
+
On macOS, changed/legacy registered jobs are booted out before replacement;
|
|
1592
|
+
missing registration is bootstrapped. An unchanged enabled registered periodic
|
|
1593
|
+
job receives no restart. On Linux, conversion stops the timer before the
|
|
1594
|
+
service, disables the legacy service's boot activation, writes the finite
|
|
1595
|
+
service and timer, reloads and enables/starts the timer. Explicit replacement
|
|
1596
|
+
can unmask only the named units. An unchanged enabled active timer preserves
|
|
1597
|
+
an in-flight pass; an inactive finite service between ticks is normal.
|
|
1598
|
+
Setup verifies registration/enablement and expected definitions before saying
|
|
1599
|
+
“periodic capture scheduled (every 15 minutes)”. It does not promise a permanent
|
|
1600
|
+
PID or infer worker completion. Manager and setup first passes can overlap;
|
|
1601
|
+
the capture root lock admits one.
|
|
1602
|
+
|
|
1603
|
+
Manager commands have finite time/output limits; completed mutation commands
|
|
1604
|
+
and definition writes are reported before subsequent steps. Failure reports the action,
|
|
1605
|
+
reason and recovery without inventing scheduling or capture results. Apply
|
|
1606
|
+
can leave backups, stopped jobs or written definitions before a later failure;
|
|
1607
|
+
there is no global atomic rollback. Inspect the reported state and rerun.
|
|
1608
|
+
|
|
1609
|
+
**Pending activation.** Before its first replacement backup, manager change or
|
|
1610
|
+
definition write, setup writes a private (0600) pending-activation marker,
|
|
1611
|
+
`<root>/capture-service-pending.json`, the way it writes every setup target:
|
|
1612
|
+
against what preflight saw, refusing a marker or record root that was replaced,
|
|
1613
|
+
linked or created since (`setup target changed since preflight` / `setup target
|
|
1614
|
+
parent changed since preflight`; re-run setup). It removes the marker only once
|
|
1615
|
+
activation is verified (`service: activation verified; pending-activation marker removed`).
|
|
1616
|
+
A failure in between leaves it. On a rerun, a loaded, unmasked disabled unit
|
|
1617
|
+
beside setup's own marked definitions and a live marker is setup's own
|
|
1618
|
+
unverified activation (on Linux, the timer a failed `enable --now` left): it is
|
|
1619
|
+
retried with no flag. Without the marker, the same state is an operator's
|
|
1620
|
+
disable and is refused with `E_CAPTURE_SERVICE_DISABLED` and both remedies. A
|
|
1621
|
+
marker with no unit, or beside a unit without setup's mark, is stale: setup
|
|
1622
|
+
removes it and names it, and it authorizes nothing. A verified activation,
|
|
1623
|
+
including an explicit `--replace-service`, removes the marker. The package has
|
|
1624
|
+
no service uninstall command. On macOS an unfinished activation leaves the
|
|
1625
|
+
plist unregistered, which a plain rerun already bootstraps; a launchd disable
|
|
1626
|
+
entry is always an operator's. `capture --health` reports a present marker as
|
|
1627
|
+
`capture-service-pending` ([offline diagnosis](#offline-installation-and-run-diagnosis)).
|
|
1628
|
+
|
|
1629
|
+
**Service failure.** Preflight refusals (foreign, operator-disabled,
|
|
1630
|
+
indeterminate state) change nothing. When apply fails after preflight passed,
|
|
1631
|
+
setup still writes the hooks and runs the bounded first pass, then exits 1:
|
|
1632
|
+
capture is never skipped for a service failure, and the failure is never
|
|
1633
|
+
hidden in the exit status. Capture receipts retain their own failure and
|
|
1634
|
+
partial meanings. A rerun after the remedy changes nothing that is already
|
|
1635
|
+
right; it only completes the activation.
|
|
1636
|
+
|
|
1637
|
+
**Setup summary.** Once setup starts changing things (every run but
|
|
1638
|
+
`--dry-run` and preflight refusals), it ends with one setup summary on stdout:
|
|
1639
|
+
fixed labels, one item per line, in this order on every run. With a nonzero
|
|
1640
|
+
exit it is the setup failure summary.
|
|
1641
|
+
|
|
1642
|
+
```
|
|
1643
|
+
setup summary:
|
|
1644
|
+
status: complete | incomplete (exit 1)
|
|
1645
|
+
hooks written: <settings path> (one line per file written, or "none")
|
|
1646
|
+
first capture pass: appended <n>, pending yes|no | skipped: another pass holds the record root | failed: <reason> | not run
|
|
1647
|
+
service: scheduled | failed: <reason> | skipped (--no-service) | no service template for <platform> | not changed
|
|
1648
|
+
service changed: <change> (after a service failure: one line per change it had already made)
|
|
1649
|
+
remedy: none | rerun <oats|turn-record> setup --owner <owner> --root <root> [flags] | inspect <oats|turn-record> capture --health --json, then rerun …
|
|
1650
|
+
```
|
|
1651
|
+
|
|
1652
|
+
A caller reads a nonzero exit with this block as "these changes were made",
|
|
1653
|
+
never as "nothing happened". After a service failure, `service changed` lines
|
|
1654
|
+
name, in order, what the failed apply had already changed: the replacement
|
|
1655
|
+
backup directory, each manager action completed (units stopped, disabled or
|
|
1656
|
+
unmasked, a job booted out), each definition written, and the pending-activation
|
|
1657
|
+
marker written or a stale one removed. Setup has no `--json` mode.
|
|
1658
|
+
|
|
1659
|
+
Fake-manager and isolated CLI fixtures exercise this ordering and refusal
|
|
1660
|
+
contract. Real-manager cases live in `test/capture-service-real-darwin.test.mjs`
|
|
1661
|
+
(launchd) and `test/capture-service-real-linux.test.mjs` (systemd --user). They
|
|
1662
|
+
run only when `GITHUB_EVENT_NAME` is `workflow_dispatch`, `GITHUB_WORKFLOW`
|
|
1663
|
+
names their own workflow ("macOS tests" or "Linux tests") and the platform
|
|
1664
|
+
matches, and skip with that reason everywhere else, pull-request CI and any
|
|
1665
|
+
developer machine included; a skip is not evidence. They cover the real
|
|
1666
|
+
`print-disabled` output against setup's parser (first), first install,
|
|
1667
|
+
conversion from an owned legacy watcher with no surviving watcher, an
|
|
1668
|
+
unchanged rerun preserving an active pass, a changed definition taking effect,
|
|
1669
|
+
the #1099 failed-enable rerun, unloaded recovery, the next idle trigger, the
|
|
1670
|
+
effective `MemoryMax`/`RuntimeMaxSec`, and missing, failing or signalled
|
|
1671
|
+
managers. The idle-trigger case uses the definition's internal interval seam
|
|
1672
|
+
(default and production value 900 seconds; no CLI or setting reaches it).
|
|
1673
|
+
The qualification harness (`test/capture-qualify-*-linux.test.mjs`, helpers in
|
|
1674
|
+
`test/fixtures/qualify*.mjs`) runs under the same guard on the dispatched
|
|
1675
|
+
"Linux tests" runner only. It measures, and reports as JSON lines and tables,
|
|
1676
|
+
first upgrade and no-op at 1,300 and 3,000 synthetic streams (scaled against
|
|
1677
|
+
a population of about 1,300 streams and 8.4 GB of journals by a stated factor), an
|
|
1678
|
+
operator-shaped first upgrade at both counts with most bytes in sessions over
|
|
1679
|
+
8 MiB on the retained path, caps-off
|
|
1680
|
+
memory scaling, the outgoing 0.61.0 and 0.53.0 releases side by side,
|
|
1681
|
+
8/16/22.3 MB journal frames and near-limit records under the capped worker,
|
|
1682
|
+
Codex in-prefix recovery, old-new-old-new rollback with released 0.61.0, the
|
|
1683
|
+
rollback of the hooks and the periodic service to 0.61.0 (the hook entry exits
|
|
1684
|
+
1, a service start fails, and 0.61.0's setup restores its watcher and hooks),
|
|
1685
|
+
and hook launcher latency. It runs the production operation under the real
|
|
1686
|
+
supervisor through the existing worker seam with the test counting patches;
|
|
1687
|
+
its assertions are structural, and its numbers are evidence only from an
|
|
1688
|
+
executed run.
|
|
1689
|
+
Quoting qualification remains a hosted obligation.
|
|
1690
|
+
Health/doctor runtime qualification, final schema delivery and full service
|
|
1691
|
+
acceptance remain separate; audit allowance/reservation/storage/synchronization
|
|
1692
|
+
gates are unchanged.
|