@evident-ai/runner-synchroniser 0.1.1-dev.d8865c7 → 3.1.1-dev.297b927

package/README.md

@@ -23,7 +23,7 @@ bundle directly (`node cli.js <command>`).
 ## Install
 
 The package is published to npm as `@evident-ai/runner-synchroniser`, exposing one
-`runner-synchroniser` binary and requiring Node >= 22:
+`runner-synchroniser` binary and requiring Node >= 22.13.0:
 
 ```bash
 npx @evident-ai/runner-synchroniser@dev env
@@ -62,7 +62,8 @@ version at all.
 | `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
-| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, skip replicate), `30` = fatal; other = usage/tool broken |
+| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
+| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
 (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -115,8 +116,11 @@ Implementation-facing reference for the 7th command: it decides what a just-run
 | `recovered`                                     | `32` | attempt 2, a successful `prune` (1 key) or any `clear` (including a partial one)  |
 | `unusableReplica(strategy)`                     | `31` | attempt 2, `leave`                                                                |
 | `unusableReplica(attemptsExhausted)`            | `31` | attempt ≥ 3, before the probe                                                     |
-| `unusableReplica(targetHealthy\|layoutMismatch)` | `31` | attempt 2, `prune` declined to delete anything                                   |
-| `unusableReplica(recoveryFailed)`               | `31` | attempt 2, `prune`'s delete itself threw                                         |
+| `unusableReplica(targetHealthy)`                | `31` | attempt 2, `prune`; the newest L0 isn't provably corrupt AND the escalation declined (unreadable, or nothing parses) |
+| `unusableReplica(noL0Present)`                  | `31` | attempt 2, `prune`; the prefix is **empty**, so there is nothing to prune and nothing to quarantine |
+| `unusableReplica(layoutMismatch)`               | `31` | attempt 2, `prune`; objects exist under the prefix but **none** parse as an LTX key at all — **a bug**, see below |
+| `unusableReplica(recoveryFailed)`               | `31` | attempt 2, `prune`'s delete itself threw, or the quarantine escalation moved nothing |
+| `recovered(quarantine)`                         | `32` | attempt 2, `prune` found nothing at L0 but the damage is deeper: the replica was moved aside |
 | `fatal(misconfig)`                              | `30` | no object store while persistence is enabled, or the probe failed on attempt 2   |
 | `fatal(deliberate)`                             | `30` | attempt 2, `crash`                                                                |
 
@@ -124,16 +128,171 @@ Implementation-facing reference for the 7th command: it decides what a just-run
 
 The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
 
+#### Boot size signals
+
+Two markers report the numbers a boot already computes but used to throw away (#1113):
+
+| Marker | Emitted from | When it fires | Fields |
+| --- | --- | --- | --- |
+| `SESSION-DB-SIZE` | `describeSessionDbClassification` (`diagnostics.ts`) | Once per `session-db-classify` invocation that reaches `restored` or `noReplica` — i.e. every boot where the restore exit code was `0`. `disabled` and every non-zero-exit outcome (`retryTransient`, `recovered`, `unusableReplica`, `fatal`) never `stat` the local DB, so they emit no size. | `state` (`restored`\|`empty`\|`absent`), `bytes` |
+| `SESSION-DB-REPLICA-SIZE` | `probeReplica` (`replica-recovery.ts`), its `ok` branch | Only on attempt 2's probe, i.e. only after a **failed** first restore — see the caveat below. | `objects`, `bytes` |
+
+**Format convention:** `<MARKER>: <prose> <k>=<v> … bytes=<n>`, with the machine-readable
+fields appended at the end of the line and `bytes` always last:
+
+```
+SESSION-DB-SIZE: <prose> state=<restored|empty|absent> bytes=<n>
+SESSION-DB-REPLICA-SIZE: <prose> objects=<m> bytes=<n>
+```
+
+Numbers go last because a CloudWatch Logs Insights `parse` capture needs a **literal
+delimiter after every capture except the final one**, which runs to end of line — free
+prose after a capture would make the pattern ambiguous. `diagnostics.test.ts` and
+`replica-recovery.test.ts` each pin a line matching `/ (state=\w+ )?bytes=\d+$/`, so an
+edit that appends prose after the number breaks a test rather than silently breaking an
+operator's saved query.
+
+**Logs Insights queries** (copy-pasteable, and the reason the format above is fixed):
+
+Session-DB size distribution across the fleet:
+
+```
+fields @timestamp, @message
+| parse @message "SESSION-DB-SIZE: * state=* bytes=*" as dbProse, dbState, dbBytes
+| filter ispresent(dbBytes)
+| stats count(*) as boots,
+        avg(dbBytes) as avgBytes,
+        pct(dbBytes, 50) as p50Bytes,
+        pct(dbBytes, 95) as p95Bytes,
+        max(dbBytes) as maxBytes
+  by bin(1d)
+| sort @timestamp desc
+```
+
+Growth curve, and how many boots restored nothing:
+
+```
+fields @timestamp, @message
+| parse @message "SESSION-DB-SIZE: * state=* bytes=*" as dbProse, dbState, dbBytes
+| filter ispresent(dbState)
+| stats count(*) as boots, max(dbBytes) as maxBytes by dbState, bin(1d)
+```
+
+Replica size + object count (ECS, failed-first-restore boots only — see the caveat below):
+
+```
+fields @timestamp, @message
+| parse @message "SESSION-DB-REPLICA-SIZE: * objects=* bytes=*" as replicaProse, replicaObjects, replicaBytes
+| filter ispresent(replicaBytes)
+| stats max(replicaBytes) as maxReplicaBytes, max(replicaObjects) as maxObjects by bin(1d)
+```
+
+**Caveat: the two markers have very different fleet coverage.** `SESSION-DB-SIZE` fires
+on **every healthy boot on both runtimes** and is the fleet-wide growth-curve signal.
+`SESSION-DB-REPLICA-SIZE` fires **only** on ECS's attempt 2 — i.e. only when a first
+restore already failed — and essentially never on the MicroVM, whose hook hard-codes the
+classify attempt to `1` (`common.sh:352`), which never reaches the probe. Chart
+`SESSION-DB-SIZE` for growth; treat `SESSION-DB-REPLICA-SIZE` as a diagnostic for
+failed-restore boots only, or an empty series will look like a broken signal instead of
+what it is.
+
 #### The four `--on-unusable-replica` strategies
 
-- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; declined (`targetHealthy`/`layoutMismatch`) or a failed delete (`recoveryFailed`) → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone — the real trade-off, and **not** "can destroy a healthy L0 object when an older one is corrupt" (false of merged `main`: the delete is unreachable unless the object's own bytes fail the check). `targetHealthy` also covers an unreadable object (absent evidence must not unlock a delete). Never escalates to `clear`, never prunes twice.
-- **`leave`** — zero S3 mutation → `31`. Cost is not just "prior history lost": `entrypoint.sh` also skips `litestream replicate` for the **whole boot**, so this boot's history is ephemeral too. Upside: replica left byte-intact for forensics, no crash loop.
+- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
+- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". Since #1106 it is **not** also ephemeral: the boot still starts a fresh backup chain, because the classifier guarantees the local DB is fresh rather than half-restored. Upside: replica left byte-intact for forensics, no crash loop.
 - **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
 - **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
 
-`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+#### The measured real-world key layout (litestream 0.5.13)
+
+`parseLtxKey` (`src/replica-keys.ts`) expects
+`<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
+despite litestream.io's generic docs prose suggesting otherwise (that mismatch was
+#1081: every real key failed to parse, silently disabling `prune` in production).
+Verbatim key sampled live from the production replica:
+`agents/e5d866ac-b699-4940-b369-633b25f8b601/opencode.db/0000/000000000001a978-000000000001a978.ltx`.
+Levels observed on that replica: `0000`/`0001`/`0002`/`0003`/`0009`. The zero-padding is
+an observation, not a contract — `parseLtxKey` accepts an unpadded level directory too.
+
+**L0 objects are short-lived.** litestream's own startup log measured a level-0
+retention of **5 minutes** (`starting L0 retention monitor interval=15s retention=5m0s`;
+compaction runs L1=30s, L2=5m, L3=1h, L9=24h). litestream expires L0 objects itself, so
+at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written and sound
+(`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
+and it can never repair corruption at a higher compaction level.
+
+`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+
+See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
+the behavioural anchor (31 comes online, 30 does not).
+
+### `session-db-verify`
+
+The 8th command: it runs *after* the restore loop above has settled and *before*
+`opencode` opens the DB. The restore loop only proves litestream could write a file;
+this command is what proves that file is actually intact (#1345 — a leaf-page-corrupt
+but openable `opencode.db` was re-restored unchanged every boot, forever, because
+nothing had ever run a real `PRAGMA integrity_check`).
+
+#### Positional
+
+| Positional | Meaning |
+| --- | --- |
+| `<litestream-config-path>` | Path to the `litestream.yml` this CLI itself generated (`litestream-config`); used to enumerate and restore from retained points. |
+
+#### What it does
+
+1. `PRAGMA integrity_check` the local `opencode.db`. Healthy → done, `0`.
+2. On failure, enumerate litestream's retained restore points (`litestream ltx`) and
+   walk them back from newest to oldest (excluding the one that just failed), restoring
+   each into a scratch path and integrity-checking it there. The first one that passes
+   is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
+3. If nothing retained passes (or there's nothing to try, or the search runs out of
+   time/points), the corrupt local DB is discarded and the boot proceeds with a
+   genuinely fresh one — exactly `session-db-classify`'s `31` guarantee, so
+   `litestream replicate` starting against it afterwards is just as safe.
+
+A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
+prefix. litestream re-bases to the replica's high-water mark and continues the txid
+chain from there; the pre-walkback lineage stays independently restorable at its old
+txids. (Measured end-to-end against a `file` replica with a tiny DB; not measured
+against S3 at the ~1 GB scale a production replica can reach.)
+
+#### Outcome → exit code
 
-See `specs/local-runner.feature:49-70` for the behavioural anchor (31 comes online, 30 does not).
+| Outcome | Code | When |
+| --- | --- | --- |
+| `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
+| `healthy` | `0` | the local DB passed its integrity check as-is |
+| `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
+| `exhausted` (`noCandidates`) | `33` | the local DB failed and litestream retains no older restore point |
+| `exhausted` (`allFailed`) | `33` | every retained restore point was tried and every one failed |
+| `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
+| `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
+
+`healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
+afterwards — start `litestream replicate` — is identical either way, so a further split
+would be a distinction with no consumer. `noCandidates`/`allFailed` are kept distinct
+from `budgetExhausted` for the same reason `unusableReplica`'s reasons are kept distinct
+above: conflating "we looked and nothing was good" with "we ran out of time/points
+before looking at everything" would report the wrong conclusion to the operator.
+
+#### Boot markers
+
+| Marker | Emitted from | When it fires | Fields |
+| --- | --- | --- | --- |
+| `WARNING: SESSION-DB-INTEGRITY-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB fails its check, before the walkback starts | — (prose carries the PRAGMA's own detail) |
+| `WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED` | the walkback loop (`session-db-verify.ts`) | `litestream ltx` exited non-zero (prose carries its stderr — missing binary, bad `-config`, S3 error) or answered with a listing that could not be parsed | — (prose carries litestream's own stderr) |
+| `WARNING: SESSION-DB-WALKBACK-CANDIDATE-FAILED` | the walkback loop (`session-db-verify.ts`) | A candidate restore point fails to restore, or restores but also fails its own check | `txid` |
+| `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
+| `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
+| `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
+| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed (any `exhausted` reason); the boot proceeds with a fresh DB | `candidatesTried` (in prose, not `k=v`) |
+
+The first four fire only while the walkback loop runs; the last three are always the
+one message this command logs as its own final word on the outcome — `entrypoint.sh`
+reuses `SESSION-DB-INTEGRITY-EXHAUSTED` verbatim when it logs its own `33` handling, the
+same way it already reuses `SESSION-DB-REPLICA-UNUSABLE` for `session-db-classify`'s `31`.
 
 ### Why the contract is asymmetric
 
@@ -186,6 +345,8 @@ key, so they never appear in `env`'s output or `litestream.yml`.
 | `LITESTREAM_BUCKET` / `LITESTREAM_PREFIX` | Object-store bucket and key prefix.                                       | Persistence off (needs **both**): `restore`/`sync-once` no-op, `env` warns.       |
 | `AWS_REGION`                              | Region for the AWS clients (S3, and the ECS/STS calls `self-stop` makes). | Left to the SDK's own resolution.                                                 |
 | `CREDS_SYNC_INTERVAL`                     | Seconds between sync ticks, reported by `env` for the caller's loop.     | Falls back to `60`; also on unparseable or non-positive values.                   |
+| `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`  | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values.                   |
+| `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop.           | Falls back to `180`; also on unparseable or non-positive values.                  |
 | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`    | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
 | `CLUSTER` † / `SERVICE` †                 | ECS cluster/service `self-stop` scales to `desiredCount=0`.              | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
 | `EVIDENT_SELFSTOP_ROLE_ARN` †             | Role `self-stop` assumes for its ECS calls.                              | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
@@ -248,10 +409,11 @@ running that version — the runner installs this package from a floating npm ta
 
 `restore`/`sync` and `session-db-classify` talk to a four-method interface: `get(key)`
 returning `null` when the object is absent, `put(key, body)`, `list(prefix)` and
-`delete(key)` for session-DB recovery. `list` throws on any error and never maps one to
-`[]` (an empty array means "reached the store, nothing there"); the adapter owns
-pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`, the typed
-`NoSuchKey`/`NotFound` errors) is confined to the single adapter in
+`delete(key)` for session-DB recovery. `list(prefix)` returns `{ key, size }` entries
+(`ObjectSummary[]`) — every object under `prefix`, size in bytes. It throws on any error
+and never maps one to `[]` (an empty array means "reached the store, nothing there"); the
+adapter owns pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`,
+the typed `NoSuchKey`/`NotFound` errors) is confined to the single adapter in
 `src/s3-object-store.ts`, so the storage backend can be swapped without touching the
 restore/sync/classify logic.