@evident-ai/runner-synchroniser 3.1.1-dev.997b038 → 3.2.0

package/README.md

@@ -23,7 +23,7 @@ bundle directly (`node cli.js <command>`).
 ## Install
 
 The package is published to npm as `@evident-ai/runner-synchroniser`, exposing one
-`runner-synchroniser` binary and requiring Node >= 22:
+`runner-synchroniser` binary and requiring Node >= 22.13.0:
 
 ```bash
 npx @evident-ai/runner-synchroniser@dev env
@@ -63,6 +63,7 @@ version at all.
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
 | `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
+| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
 (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -220,11 +221,79 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
 (`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
 and it can never repair corruption at a higher compaction level.
 
-`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
 
 See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
 the behavioural anchor (31 comes online, 30 does not).
 
+### `session-db-verify`
+
+The 8th command: it runs *after* the restore loop above has settled and *before*
+`opencode` opens the DB. The restore loop only proves litestream could write a file;
+this command is what proves that file is actually intact (#1345 — a leaf-page-corrupt
+but openable `opencode.db` was re-restored unchanged every boot, forever, because
+nothing had ever run a real `PRAGMA integrity_check`).
+
+#### Positional
+
+| Positional | Meaning |
+| --- | --- |
+| `<litestream-config-path>` | Path to the `litestream.yml` this CLI itself generated (`litestream-config`); used to enumerate and restore from retained points. |
+
+#### What it does
+
+1. `PRAGMA integrity_check` the local `opencode.db`. Healthy → done, `0`.
+2. On failure, enumerate litestream's retained restore points (`litestream ltx`) and
+   walk them back from newest to oldest (excluding the one that just failed), restoring
+   each into a scratch path and integrity-checking it there. The first one that passes
+   is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
+3. If nothing retained passes (or there's nothing to try, or the search runs out of
+   time/points), the corrupt local DB is discarded and the boot proceeds with a
+   genuinely fresh one — exactly `session-db-classify`'s `31` guarantee, so
+   `litestream replicate` starting against it afterwards is just as safe.
+
+A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
+prefix. litestream re-bases to the replica's high-water mark and continues the txid
+chain from there; the pre-walkback lineage stays independently restorable at its old
+txids. (Measured end-to-end against a `file` replica with a tiny DB; not measured
+against S3 at the ~1 GB scale a production replica can reach.)
+
+#### Outcome → exit code
+
+| Outcome | Code | When |
+| --- | --- | --- |
+| `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
+| `healthy` | `0` | the local DB passed its integrity check as-is |
+| `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
+| `exhausted` (`noCandidates`) | `33` | the local DB failed and litestream retains no older restore point |
+| `exhausted` (`allFailed`) | `33` | every retained restore point was tried and every one failed |
+| `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
+| `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
+
+`healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
+afterwards — start `litestream replicate` — is identical either way, so a further split
+would be a distinction with no consumer. `noCandidates`/`allFailed` are kept distinct
+from `budgetExhausted` for the same reason `unusableReplica`'s reasons are kept distinct
+above: conflating "we looked and nothing was good" with "we ran out of time/points
+before looking at everything" would report the wrong conclusion to the operator.
+
+#### Boot markers
+
+| Marker | Emitted from | When it fires | Fields |
+| --- | --- | --- | --- |
+| `WARNING: SESSION-DB-INTEGRITY-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB fails its check, before the walkback starts | — (prose carries the PRAGMA's own detail) |
+| `WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED` | the walkback loop (`session-db-verify.ts`) | `litestream ltx` exited non-zero (prose carries its stderr — missing binary, bad `-config`, S3 error) or answered with a listing that could not be parsed | — (prose carries litestream's own stderr) |
+| `WARNING: SESSION-DB-WALKBACK-CANDIDATE-FAILED` | the walkback loop (`session-db-verify.ts`) | A candidate restore point fails to restore, or restores but also fails its own check | `txid` |
+| `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
+| `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
+| `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
+| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed (any `exhausted` reason); the boot proceeds with a fresh DB | `candidatesTried` (in prose, not `k=v`) |
+
+The first four fire only while the walkback loop runs; the last three are always the
+one message this command logs as its own final word on the outcome — `entrypoint.sh`
+reuses `SESSION-DB-INTEGRITY-EXHAUSTED` verbatim when it logs its own `33` handling, the
+same way it already reuses `SESSION-DB-REPLICA-UNUSABLE` for `session-db-classify`'s `31`.
+
 ### Why the contract is asymmetric
 
 `restore` and `sync-once` **never** report a domain outcome through the exit code. A
@@ -276,6 +345,8 @@ key, so they never appear in `env`'s output or `litestream.yml`.
 | `LITESTREAM_BUCKET` / `LITESTREAM_PREFIX` | Object-store bucket and key prefix.                                       | Persistence off (needs **both**): `restore`/`sync-once` no-op, `env` warns.       |
 | `AWS_REGION`                              | Region for the AWS clients (S3, and the ECS/STS calls `self-stop` makes). | Left to the SDK's own resolution.                                                 |
 | `CREDS_SYNC_INTERVAL`                     | Seconds between sync ticks, reported by `env` for the caller's loop.     | Falls back to `60`; also on unparseable or non-positive values.                   |
+| `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`  | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values.                   |
+| `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop.           | Falls back to `180`; also on unparseable or non-positive values.                  |
 | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`    | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
 | `CLUSTER` † / `SERVICE` †                 | ECS cluster/service `self-stop` scales to `desiredCount=0`.              | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
 | `EVIDENT_SELFSTOP_ROLE_ARN` †             | Role `self-stop` assumes for its ECS calls.                              | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |