@evident-ai/runner-synchroniser 3.4.1-dev.aa71e9a → 3.4.1-dev.b9f71fa

package/README.md

@@ -56,11 +56,11 @@ build time.
 | `env`                          | resolved config as shell-eval'able vars | `0` = ran; non-zero = tool broken                        |
 | `litestream-config`            | the generated `litestream.yml`          | `0` = ran; non-zero = tool broken                        |
 | `restore <claude\|opencode>`   | —                                       | `0` = ran; non-zero = tool broken                        |
-| `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
+| `sync-once <claude\|opencode>` | —                                       | `0` = uploaded/unchanged/absent/disabled; `40` = not persisted; other = tool broken |
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
-| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
-| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
+| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, replicates into the existing prefix), `30` = fatal; other = usage/tool broken |
+| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = exhausted after verified separation and local disposal, `34` = separation or disposal could not be proven; other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
 (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -75,7 +75,7 @@ saved query.
 
 ### `session-db-classify`
 
-Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
+Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `runner/docker-images/fargate/README.md`'s [Strategy/What/Cost table](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
 
 #### Activity recovery report
 
@@ -86,6 +86,13 @@ a warning `restore_retried` record before the retry's result is known. Writing i
 and never changes this command's exit code. The CLI reader owns this record contract: add a new
 `v` rather than repurposing a field.
 
+MicroVM boot-shell give-ups use that same JSONL contract and path. The hook truncates the
+report at the start of each `/run`, so a later startup decision supersedes an earlier one;
+`/resume` preserves the current report. Their `replication_suspended` field is `true`, meaning
+that start is not backing up its new session history. The normal mapper defaults the field to
+`false`; the MicroVM `--fresh-db-fallback` outcome is also `true` because it cannot safely
+replicate that boot.
+
 #### Positionals
 
 | Positional                       | Meaning                                                                                                                                |
@@ -119,6 +126,9 @@ marks a runtime that intentionally has no retry budget and will therefore boot f
 otherwise-transient failure. Each flag may appear once, does not change the command's exit code,
 and is rejected when repeated.
 
+On the MicroVM's one-attempt path, `--fresh-db-fallback` also means the fresh database does not
+replicate during that boot.
+
 #### Outcome → exit code
 
 | Outcome                                        | Code | When                                                                        |
@@ -138,6 +148,9 @@ and is rejected when repeated.
 | `fatal(misconfig)`                              | `30` | no object store while persistence is enabled, or the probe failed on attempt 2   |
 | `fatal(deliberate)`                             | `30` | attempt 2, `crash`                                                                |
 
+Fatal exit codes 30 and 34 are reported directly to `POST /v1/runners/self/startup-failure`
+before the CLI starts. Reporting is best-effort and never changes the exit code.
+
 `31` also guarantees local debris (`opencode.db`, `-wal`, `-shm`) is discarded, one `try` per path, at the classifier's single return point. Note `32` isn't purely "transient" — `recovered` shares it with `retryTransient` even though a `recovered` outcome already mutated S3.
 
 The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
@@ -213,11 +226,11 @@ what it is.
 #### The four `--on-unusable-replica` strategies
 
 - **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
-- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". Since #1106 it is **not** also ephemeral: the boot still starts a fresh backup chain, because the classifier guarantees the local DB is fresh rather than half-restored. Upside: replica left byte-intact for forensics, no crash loop.
+- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". The fresh local DB replicates into the existing prefix; no new chain is started. Upside: the runner still comes online without a crash loop.
 - **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
 - **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
 
-#### The measured real-world key layout (litestream 0.5.13)
+#### The measured real-world key layout (litestream 0.5.13 historical sample)
 
 `parseLtxKey` (`src/replica-keys.ts`) expects
 `<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
@@ -235,7 +248,7 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
 (`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
 and it can never repair corruption at a higher compaction level.
 
-`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+`EVIDENT_ON_UNUSABLE_REPLICA` is a `runner/docker-images/fargate` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [the Fargate image README](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
 
 See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
 the behavioural anchor (31 comes online, 30 does not).
@@ -262,9 +275,12 @@ nothing had ever run a real `PRAGMA integrity_check`).
    each into a scratch path and integrity-checking it there. The first one that passes
    is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
 3. If nothing retained passes (or there's nothing to try, or the search runs out of
-   time/points), the corrupt local DB is discarded and the boot proceeds with a
-   genuinely fresh one — exactly `session-db-classify`'s `31` guarantee, so
-   `litestream replicate` starting against it afterwards is just as safe.
+   time/points), the active replica prefix is separated first: its history is moved aside,
+   then the prefix is re-listed as empty before the corrupt local DB and sidecars are
+   discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
+   OpenCode or Litestream starts; the backup history remains readable at its quarantine
+    destination or original key, and surviving local files remain in place without a process
+    opening or writing them.
 
 A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
 prefix. litestream re-bases to the replica's high-water mark and continues the txid
@@ -279,10 +295,8 @@ against S3 at the ~1 GB scale a production replica can reach.)
 | `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
 | `healthy` | `0` | the local DB passed its integrity check as-is |
 | `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
-| `exhausted` (`noCandidates`) | `33` | the local DB failed and litestream retains no older restore point |
-| `exhausted` (`allFailed`) | `33` | every retained restore point was tried and every one failed |
-| `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
-| `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
+| `exhausted` (`noCandidates`, `allFailed`, `budgetExhausted`, or `enumerationFailed`) | `33` | the active prefix was proven separated (or persistence is disabled) **and** the corrupt local DB and sidecars were proven discarded |
+| `exhausted` | `34` | either the replica separation or local discard could not be proven; the boot stops before OpenCode and Litestream start |
 
 `healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
 afterwards — start `litestream replicate` — is identical either way, so a further split
@@ -301,7 +315,10 @@ before looking at everything" would report the wrong conclusion to the operator.
 | `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
 | `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
 | `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
-| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed (any `exhausted` reason); the boot proceeds with a fresh DB | `candidatesTried` (in prose, not `k=v`) |
+| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed; a fresh boot follows only when both separation and local disposal were proven | `candidatesTried` (in prose, not `k=v`) |
+| `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
+| `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
+| `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
 
 The first four fire only while the walkback loop runs; the last three are always the
 one message this command logs as its own final word on the outcome — `entrypoint.sh`
@@ -310,15 +327,16 @@ same way it already reuses `SESSION-DB-REPLICA-UNUSABLE` for `session-db-classif
 
 ### Why the contract is asymmetric
 
-`restore` and `sync-once` **never** report a domain outcome through the exit code. A
-missing remote object, a corrupt local file, a failed upload, an IAM denial — all of
-those are logged and still exit `0`. None of them should abort a boot: a runner with no
-credentials yet is a runner a human can still log into.
+`restore` logs domain outcomes and still exits `0`: a missing remote object or a
+restore failure should not abort a boot, because a runner with no credentials yet
+is a runner a human can still log into. `sync-once` returns `40` when the current
+credential file is not persisted (`failed`, `hashFailed`, or `localInvalid`), while
+`absent` and `disabled` remain legitimate `0` outcomes.
 
-The consequence is the point: **any non-zero status from `restore`/`sync-once` means the
-tool itself broke** — bad arguments (`2`), an uncaught throw (`1`), or a bundle that
-would not run at all. The shell needs no case analysis to know something is wrong, so
-its "credential persistence is DEGRADED" error lives in exactly one helper.
+The consequence is the point: **any non-zero status from `restore` means the tool itself
+broke** — and `sync-once` uses `40` for its typed persistence answer; other non-zero
+statuses still mean bad arguments (`2`), an uncaught throw (`1`), or a bundle that would
+not run at all. The shell keeps that distinction in one helper.
 
 The **two predicates** — `model-auth-ready` and `self-stop` — each need to distinguish
 "the answer is no" from "the tool is broken", so each answers no with its own dedicated
@@ -334,13 +352,14 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
 `session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
 
 `session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
-history. It is drained once by `evident run` into runner activity and does not alter its exit
-code.
+history. A refused boot records `session_db_boot_refused`, but `evident run` has not started to
+drain it, so operators see that refusal in boot logs and the restart loop. The report does not
+alter the command's exit code.
 
 `src/shell-contract.json` is the machine-checked source of truth for the command list, and
 `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
-`packages/runner-image/entrypoint.sh` and the MicroVM's
-`packages/runner-cdk/microvm-image/hooks` (#608), discovered by grep so a third one
+`runner/docker-images/fargate/entrypoint.sh` and the MicroVM's
+`runner/docker-images/microvm/hooks` (#608), discovered by grep so a third one
 cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
 `entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
 a broken tool; and take each answer code the commands it calls can return **silently and by
@@ -370,7 +389,7 @@ key, so they never appear in `env`'s output or `litestream.yml`.
 | `CLUSTER` † / `SERVICE` †                 | ECS cluster/service `self-stop` scales to `desiredCount=0`.              | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
 | `EVIDENT_SELFSTOP_ROLE_ARN` †             | Role `self-stop` assumes for its ECS calls.                              | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
 
-`packages/runner-image/README.md` documents these same three from the deployment side —
+`runner/docker-images/fargate/README.md` documents these same three from the deployment side —
 keep them in sync.
 
 Requiring **both** `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` (never just one) means