@evident-ai/runner-synchroniser 0.1.1-dev.9639f77 → 0.1.1-dev.96bc2ed

package/README.md

@@ -62,6 +62,7 @@ version at all.
 | `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
+| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, skip replicate), `30` = fatal; other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
 (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -74,6 +75,66 @@ broader: `src/logger.ts` marks it verbatim because operator greps and CloudWatch
 match on that exact string, so widening what it labels is cheaper than breaking every
 saved query.
 
+### `session-db-classify`
+
+Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
+
+#### Positionals
+
+| Positional                       | Meaning                                                                                                                                |
+| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
+| `<litestream-restore-exit-code>` | Status of the `litestream restore` the shell just ran. `0` → stat the local DB; any non-zero → the recovery ladder below. The value itself is never inspected (a Go panic `2` is treated exactly like a `1`). |
+| `<attempt>`                      | 1-based (`/^\d+$/`, `>= 1`) — the sole mutation gate.                                                                                    |
+
+#### The attempt ladder
+
+| Attempt | Behaviour                                                     |
+| ------- | ---------------------------------------------------------------- |
+| 1       | Plain retry — **zero S3 calls, not even the probe**.              |
+| 2       | The **only** attempt that may probe or mutate S3.                 |
+| ≥ 3     | Gives up **before** the probe.                                    |
+
+`entrypoint.sh` caps its retry loop at a hard-coded `3` as a real second bound: the classifier is stateless per invocation, so nothing except that loop stops attempt 2 from being asked twice.
+
+#### `--on-unusable-replica` flag syntax
+
+- Only the `=`-joined form (`--on-unusable-replica=leave`); the space-separated form is rejected.
+- Position-free.
+- Repeating the flag is rejected, even with the same value both times.
+- Any other `--` token is rejected.
+- Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
+
+#### Outcome → exit code
+
+| Outcome                                        | Code | When                                                                        |
+| ----------------------------------------------- | ---- | -------------------------------------------------------------------------------- |
+| `disabled`                                      | `0`  | bucket or prefix unset — short-circuits **before** the exit code is examined     |
+| `restored`                                      | `0`  | restore exit `0` and the local DB is non-empty                                   |
+| `noReplica`                                     | `0`  | restore exit `0` but the DB is absent or zero-byte                               |
+| `retryTransient`                                | `32` | restore failed, attempt 1 — no probe, no S3 call at all                          |
+| `recovered`                                     | `32` | attempt 2, a successful `prune` (1 key) or any `clear` (including a partial one)  |
+| `unusableReplica(strategy)`                     | `31` | attempt 2, `leave`                                                                |
+| `unusableReplica(attemptsExhausted)`            | `31` | attempt ≥ 3, before the probe                                                     |
+| `unusableReplica(targetHealthy\|layoutMismatch)` | `31` | attempt 2, `prune` declined to delete anything                                   |
+| `unusableReplica(recoveryFailed)`               | `31` | attempt 2, `prune`'s delete itself threw                                         |
+| `fatal(misconfig)`                              | `30` | no object store while persistence is enabled, or the probe failed on attempt 2   |
+| `fatal(deliberate)`                             | `30` | attempt 2, `crash`                                                                |
+
+`31` also guarantees local debris (`opencode.db`, `-wal`, `-shm`) is discarded, one `try` per path, at the classifier's single return point. Note `32` isn't purely "transient" — `recovered` shares it with `retryTransient` even though a `recovered` outcome already mutated S3.
+
+The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
+
+#### The four `--on-unusable-replica` strategies
+
+- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; declined (`targetHealthy`/`layoutMismatch`) or a failed delete (`recoveryFailed`) → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone — the real trade-off, and **not** "can destroy a healthy L0 object when an older one is corrupt" (false of merged `main`: the delete is unreachable unless the object's own bytes fail the check). `targetHealthy` also covers an unreadable object (absent evidence must not unlock a delete). Never escalates to `clear`, never prunes twice.
+- **`leave`** — zero S3 mutation → `31`. Cost is not just "prior history lost": `entrypoint.sh` also skips `litestream replicate` for the **whole boot**, so this boot's history is ephemeral too. Upside: replica left byte-intact for forensics, no crash loop.
+- **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
+- **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
+
+`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+
+See `specs/local-runner.feature:49-70` for the behavioural anchor (31 comes online, 30 does not).
+
 ### Why the contract is asymmetric
 
 `restore` and `sync-once` **never** report a domain outcome through the exit code. A
@@ -97,6 +158,8 @@ that it has no auth. `self-stop` decides whether a task may go away, so a broken
 must never kill one: `entrypoint.sh` reads any non-zero exactly like `20` and keeps the
 task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 — lets it stop.
 
+`session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
+
 `src/shell-contract.json` is the machine-checked source of truth for the command list, and
 `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
 `packages/runner-image/entrypoint.sh` and the MicroVM's
@@ -142,10 +205,23 @@ The paths and keys that follow from the bucket/prefix:
 | OpenCode DB | `$HOME/.local/share/opencode/opencode.db` | `<prefix>/opencode.db` (litestream) |
 
 The session database is replicated by litestream itself; this package only generates its
-config. `env` emits `CREDS_BUCKET`, `CLAUDE_CREDS`, `OPENCODE_DB_PATH` and
+config. `env` emits `PERSISTENCE_BUCKET`, `CLAUDE_CREDS`, `OPENCODE_DB_PATH` and
 `CREDS_SYNC_INTERVAL`, each single-quoted so a value containing shell metacharacters is
 inert when `eval`'d.
 
+`entrypoint.sh` asserts, right after it `eval`s this output, that each key is *defined*
+(not merely non-empty) and splits the four into two groups: a key is **required** — the
+shell dies naming it — if it has no safe degradation; it is **tolerated** — the shell logs
+a WARNING naming it and carries on — if it does. `OPENCODE_DB_PATH` and
+`CREDS_SYNC_INTERVAL` are required: both are dereferenced for real work (deleting/creating
+the session DB, sizing the sync-loop sleep), so a guessed value would be actively wrong,
+not merely absent. `PERSISTENCE_BUCKET` and `CLAUDE_CREDS` are tolerated: an empty
+`PERSISTENCE_BUCKET` is already the designed "persistence disabled" state, and
+`CLAUDE_CREDS` is dereferenced only in operator-facing message text. If you change
+`renderEnv` to make a required key conditional, the entrypoint will brick every container
+running that version — the runner installs this package from a floating npm tag (`dev` /
+`latest`), so the shell and the CLI can genuinely be different versions (#779, #821).
+
 ## Behaviour worth knowing
 
 - **A non-empty local file that is not valid JSON is protected**, never overwritten and
@@ -170,11 +246,14 @@ inert when `eval`'d.
 
 ## The `ObjectStore` port
 
-`restore` and `sync` talk to a two-method interface — `get(key)` returning `null` when
-the object is absent, and `put(key, body)`. S3 vocabulary (`@aws-sdk/client-s3`, the
-typed `NoSuchKey`/`NotFound` errors) is confined to the single adapter in
+`restore`/`sync` and `session-db-classify` talk to a four-method interface: `get(key)`
+returning `null` when the object is absent, `put(key, body)`, `list(prefix)` and
+`delete(key)` for session-DB recovery. `list` throws on any error and never maps one to
+`[]` (an empty array means "reached the store, nothing there"); the adapter owns
+pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`, the typed
+`NoSuchKey`/`NotFound` errors) is confined to the single adapter in
 `src/s3-object-store.ts`, so the storage backend can be swapped without touching the
-restore/sync logic.
+restore/sync/classify logic.
 
 The S3 adapter is the only one that exists. It uses no static credentials — auth is
 whatever the ambient AWS credential chain resolves (on the Evident runner, the ECS task

package/dist/cli.js

@@ -50104,9 +50104,13 @@ function parseSessionDbClassifyArgs(args) {
 }
 function renderEnv(config) {
   return [
-    // Empty when persistence is disabled — entrypoint.sh gates the sync loop
-    // on this being non-empty, exactly as it did when it computed it itself.
-    `CREDS_BUCKET=${shellQuote(config.bucket ?? "")}`,
+    // The bucket NAME, kept (not reduced to a boolean) so an operator can
+    // read it; empty exactly when persistence is disabled (both
+    // LITESTREAM_BUCKET and LITESTREAM_PREFIX must be set — see config.ts's
+    // persistenceEnabled). entrypoint.sh gates BOTH the credential-sync
+    // loop and whether it starts litestream replicate on this being
+    // non-empty.
+    `PERSISTENCE_BUCKET=${shellQuote(config.bucket ?? "")}`,
     `CLAUDE_CREDS=${shellQuote(config.claude.path)}`,
     `OPENCODE_DB_PATH=${shellQuote(config.opencodeDbPath)}`,
     `CREDS_SYNC_INTERVAL=${shellQuote(String(config.syncIntervalSeconds))}`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evident-ai/runner-synchroniser",
-  "version": "0.1.1-dev.9639f77",
+  "version": "0.1.1-dev.96bc2ed",
   "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
   "type": "module",
   "main": "./dist/cli.js",