@evident-ai/runner-synchroniser 0.1.1-dev.0d89dd3 → 0.1.1-dev.1e72251

package/README.md

@@ -32,11 +32,25 @@ npx @evident-ai/runner-synchroniser@dev env
 `dist/cli.js` is a single self-contained ESM bundle with **no runtime dependencies** —
 the AWS SDK is bundled at build time, so installing it pulls nothing else in.
 
-> Publishing is being rolled out (#612). The `dev` tag is published on every merge to
-> `main` that touches this package; the first publish is a human bootstrap (#694), and
-> the `latest`-on-release path is not wired up yet. The Fargate runner image installs
-> the `dev`-tagged package (`ARG RUNNER_SYNCHRONISER_VERSION=dev`); the MicroVM still
-> builds the bundle from source and is not expected to change.
+### Versioning
+
+The package's version **tracks the product release version**, exactly like
+`@evident-ai/cli`. `latest` is published on a product release, at the release
+version. `dev` is published on every merge to `main` that touches this package or
+the root `package.json`, at `<root major>.<minor>.<patch+1>-dev.<sha>`, so `dev`
+stays strictly ahead of `latest`.
+
+The version committed in this package's `package.json` is **vestigial** — CI
+overwrites it before every publish. This is why `latest` jumped straight from
+`0.1.0` to `3.x.y`: the `0.1.x` line predates the policy, not a missing
+major-version story, and because npm publishes are immutable that jump is
+permanent.
+
+The two consumers still differ in how they get the binary: the Fargate runner
+image installs the published package
+(`ARG RUNNER_SYNCHRONISER_VERSION=dev`, resolved to a concrete version by the
+deploy workflow); the MicroVM still builds it from source and reads no npm
+version at all.
 
 ## Commands
 
@@ -48,6 +62,7 @@ the AWS SDK is bundled at build time, so installing it pulls nothing else in.
 | `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
+| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, skip replicate), `30` = fatal; other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
 (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -60,6 +75,87 @@ broader: `src/logger.ts` marks it verbatim because operator greps and CloudWatch
 match on that exact string, so widening what it labels is cheaper than breaking every
 saved query.
 
+### `session-db-classify`
+
+Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
+
+#### Positionals
+
+| Positional                       | Meaning                                                                                                                                |
+| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
+| `<litestream-restore-exit-code>` | Status of the `litestream restore` the shell just ran. `0` → stat the local DB; any non-zero → the recovery ladder below. The value itself is never inspected (a Go panic `2` is treated exactly like a `1`). |
+| `<attempt>`                      | 1-based (`/^\d+$/`, `>= 1`) — the sole mutation gate.                                                                                    |
+
+#### The attempt ladder
+
+| Attempt | Behaviour                                                     |
+| ------- | ---------------------------------------------------------------- |
+| 1       | Plain retry — **zero S3 calls, not even the probe**.              |
+| 2       | The **only** attempt that may probe or mutate S3.                 |
+| ≥ 3     | Gives up **before** the probe.                                    |
+
+`entrypoint.sh` caps its retry loop at a hard-coded `3` as a real second bound: the classifier is stateless per invocation, so nothing except that loop stops attempt 2 from being asked twice.
+
+#### `--on-unusable-replica` flag syntax
+
+- Only the `=`-joined form (`--on-unusable-replica=leave`); the space-separated form is rejected.
+- Position-free.
+- Repeating the flag is rejected, even with the same value both times.
+- Any other `--` token is rejected.
+- Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
+
+#### Outcome → exit code
+
+| Outcome                                        | Code | When                                                                        |
+| ----------------------------------------------- | ---- | -------------------------------------------------------------------------------- |
+| `disabled`                                      | `0`  | bucket or prefix unset — short-circuits **before** the exit code is examined     |
+| `restored`                                      | `0`  | restore exit `0` and the local DB is non-empty                                   |
+| `noReplica`                                     | `0`  | restore exit `0` but the DB is absent or zero-byte                               |
+| `retryTransient`                                | `32` | restore failed, attempt 1 — no probe, no S3 call at all                          |
+| `recovered`                                     | `32` | attempt 2, a successful `prune` (1 key) or any `clear` (including a partial one)  |
+| `unusableReplica(strategy)`                     | `31` | attempt 2, `leave`                                                                |
+| `unusableReplica(attemptsExhausted)`            | `31` | attempt ≥ 3, before the probe                                                     |
+| `unusableReplica(targetHealthy)`                | `31` | attempt 2, `prune`; the newest L0 isn't provably corrupt, so it declined to delete it |
+| `unusableReplica(noL0Present)`                  | `31` | attempt 2, `prune`; no level-0 object was present to prune — **normal**, see below |
+| `unusableReplica(layoutMismatch)`               | `31` | attempt 2, `prune`; objects exist under the prefix but **none** parse as an LTX key at all — **a bug**, see below |
+| `unusableReplica(recoveryFailed)`               | `31` | attempt 2, `prune`'s delete itself threw                                         |
+| `fatal(misconfig)`                              | `30` | no object store while persistence is enabled, or the probe failed on attempt 2   |
+| `fatal(deliberate)`                             | `30` | attempt 2, `crash`                                                                |
+
+`31` also guarantees local debris (`opencode.db`, `-wal`, `-shm`) is discarded, one `try` per path, at the classifier's single return point. Note `32` isn't purely "transient" — `recovered` shares it with `retryTransient` even though a `recovered` outcome already mutated S3.
+
+The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
+
+#### The four `--on-unusable-replica` strategies
+
+- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; declined (`targetHealthy`/`noL0Present`/`layoutMismatch`) or a failed delete (`recoveryFailed`) → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone — the real trade-off, and **not** "can destroy a healthy L0 object when an older one is corrupt" (false of merged `main`: the delete is unreachable unless the object's own bytes fail the check). `targetHealthy` also covers an unreadable object (absent evidence must not unlock a delete). Never escalates to `clear`, never prunes twice.
+- **`leave`** — zero S3 mutation → `31`. Cost is not just "prior history lost": `entrypoint.sh` also skips `litestream replicate` for the **whole boot**, so this boot's history is ephemeral too. Upside: replica left byte-intact for forensics, no crash loop.
+- **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
+- **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
+
+#### The measured real-world key layout (litestream 0.5.13)
+
+`parseLtxKey` (`src/replica-keys.ts`) expects
+`<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
+despite litestream.io's generic docs prose suggesting otherwise (that mismatch was
+#1081: every real key failed to parse, silently disabling `prune` in production).
+Verbatim key sampled live from the production replica:
+`agents/e5d866ac-b699-4940-b369-633b25f8b601/opencode.db/0000/000000000001a978-000000000001a978.ltx`.
+Levels observed on that replica: `0000`/`0001`/`0002`/`0003`/`0009`. The zero-padding is
+an observation, not a contract — `parseLtxKey` accepts an unpadded level directory too.
+
+**L0 objects are short-lived.** litestream's own startup log measured a level-0
+retention of **5 minutes** (`starting L0 retention monitor interval=15s retention=5m0s`;
+compaction runs L1=30s, L2=5m, L3=1h, L9=24h). litestream expires L0 objects itself, so
+at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written and sound
+(`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
+and it can never repair corruption at a higher compaction level.
+
+`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+
+See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
+the behavioural anchor (31 comes online, 30 does not).
+
 ### Why the contract is asymmetric
 
 `restore` and `sync-once` **never** report a domain outcome through the exit code. A
@@ -83,6 +179,8 @@ that it has no auth. `self-stop` decides whether a task may go away, so a broken
 must never kill one: `entrypoint.sh` reads any non-zero exactly like `20` and keeps the
 task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 — lets it stop.
 
+`session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
+
 `src/shell-contract.json` is the machine-checked source of truth for the command list, and
 `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
 `packages/runner-image/entrypoint.sh` and the MicroVM's
@@ -128,10 +226,23 @@ The paths and keys that follow from the bucket/prefix:
 | OpenCode DB | `$HOME/.local/share/opencode/opencode.db` | `<prefix>/opencode.db` (litestream) |
 
 The session database is replicated by litestream itself; this package only generates its
-config. `env` emits `CREDS_BUCKET`, `CLAUDE_CREDS`, `OPENCODE_DB_PATH` and
+config. `env` emits `PERSISTENCE_BUCKET`, `CLAUDE_CREDS`, `OPENCODE_DB_PATH` and
 `CREDS_SYNC_INTERVAL`, each single-quoted so a value containing shell metacharacters is
 inert when `eval`'d.
 
+`entrypoint.sh` asserts, right after it `eval`s this output, that each key is *defined*
+(not merely non-empty) and splits the four into two groups: a key is **required** — the
+shell dies naming it — if it has no safe degradation; it is **tolerated** — the shell logs
+a WARNING naming it and carries on — if it does. `OPENCODE_DB_PATH` and
+`CREDS_SYNC_INTERVAL` are required: both are dereferenced for real work (deleting/creating
+the session DB, sizing the sync-loop sleep), so a guessed value would be actively wrong,
+not merely absent. `PERSISTENCE_BUCKET` and `CLAUDE_CREDS` are tolerated: an empty
+`PERSISTENCE_BUCKET` is already the designed "persistence disabled" state, and
+`CLAUDE_CREDS` is dereferenced only in operator-facing message text. If you change
+`renderEnv` to make a required key conditional, the entrypoint will brick every container
+running that version — the runner installs this package from a floating npm tag (`dev` /
+`latest`), so the shell and the CLI can genuinely be different versions (#779, #821).
+
 ## Behaviour worth knowing
 
 - **A non-empty local file that is not valid JSON is protected**, never overwritten and
@@ -156,11 +267,14 @@ inert when `eval`'d.
 
 ## The `ObjectStore` port
 
-`restore` and `sync` talk to a two-method interface — `get(key)` returning `null` when
-the object is absent, and `put(key, body)`. S3 vocabulary (`@aws-sdk/client-s3`, the
-typed `NoSuchKey`/`NotFound` errors) is confined to the single adapter in
+`restore`/`sync` and `session-db-classify` talk to a four-method interface: `get(key)`
+returning `null` when the object is absent, `put(key, body)`, `list(prefix)` and
+`delete(key)` for session-DB recovery. `list` throws on any error and never maps one to
+`[]` (an empty array means "reached the store, nothing there"); the adapter owns
+pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`, the typed
+`NoSuchKey`/`NotFound` errors) is confined to the single adapter in
 `src/s3-object-store.ts`, so the storage backend can be swapped without touching the
-restore/sync logic.
+restore/sync/classify logic.
 
 The S3 adapter is the only one that exists. It uses no static credentials — auth is
 whatever the ambient AWS credential chain resolves (on the Evident runner, the ECS task

package/dist/cli.js

@@ -39606,6 +39606,89 @@ function describeRestoreOutcome(label, path2, outcome) {
       return assertNever(outcome);
   }
 }
+function assertNeverSessionDb(value) {
+  throw new Error(`Unhandled session-DB restore outcome: ${JSON.stringify(value)}`);
+}
+function assertNeverUnusableReplicaReason(value) {
+  throw new Error(`Unhandled unusable-replica reason: ${JSON.stringify(value)}`);
+}
+function assertNeverFatalCause(value) {
+  throw new Error(`Unhandled fatal cause: ${JSON.stringify(value)}`);
+}
+function sessionDbExitCode(outcome) {
+  switch (outcome.kind) {
+    case "disabled":
+    case "restored":
+    case "noReplica":
+      return 0;
+    case "retryTransient":
+    case "recovered":
+      return 32;
+    // re-run `litestream restore` and ask again
+    case "unusableReplica":
+      return 31;
+    // booted with a fresh DB; the shell must skip `litestream replicate`
+    case "fatal":
+      return 30;
+    // shared by a genuine misconfig and a deliberate `crash` choice
+    default:
+      return assertNeverSessionDb(outcome);
+  }
+}
+function describeUnusableReplica(reason) {
+  const base = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
+  switch (reason) {
+    case "strategy":
+      return `${base} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
+    case "attemptsExhausted":
+      return `${base} (recovery did not make the replica usable after retrying; giving up.)`;
+    case "layoutMismatch":
+      return `${base} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
+    case "noL0Present":
+      return `${base} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
+    case "recoveryFailed":
+      return `${base} (the recovery delete itself failed; see the warning above for the S3 error.)`;
+    case "targetHealthy":
+      return `${base} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
+    default:
+      return assertNeverUnusableReplicaReason(reason);
+  }
+}
+function describeSessionDbFatal(outcome, config) {
+  switch (outcome.cause) {
+    // Names the bucket/region and the probe error — never the crash flag.
+    case "misconfig":
+      return `FATAL: could not verify the opencode.db replica in S3 is reachable/authorised (bucket=${config.bucket ?? "<unset>"}, region=${config.region ?? "<unset>"}): ${outcome.detail}`;
+    // Names the flag and states nothing was touched — never bucket/region, so an
+    // operator reading this is never sent chasing IAM over their own flag.
+    case "deliberate":
+      return `FATAL: ${outcome.detail}`;
+    default:
+      return assertNeverFatalCause(outcome.cause);
+  }
+}
+function describeSessionDbClassification(outcome, config) {
+  switch (outcome.kind) {
+    case "disabled":
+      return "No opencode.db restore attempted; credential persistence is disabled (LITESTREAM_BUCKET/LITESTREAM_PREFIX unset).";
+    case "restored":
+      return "opencode.db restored from the object store.";
+    case "noReplica":
+      return "No opencode.db replica found in the object store; opencode will create a fresh session DB.";
+    case "retryTransient":
+      return "opencode.db replica restore failed (transient); retrying.";
+    case "recovered": {
+      const objects = outcome.deleted.length === 0 ? "no objects (see the warnings above for what failed)" : `${outcome.deleted.length} object${outcome.deleted.length === 1 ? "" : "s"} (${outcome.deleted.join(", ")})`;
+      return `Attempted opencode.db replica recovery via --on-unusable-replica=${outcome.strategy}, deleting ${objects}; retrying the restore.` + (outcome.strategy === "clear" ? " Prior session history is lost." : "");
+    }
+    case "unusableReplica":
+      return describeUnusableReplica(outcome.reason);
+    case "fatal":
+      return describeSessionDbFatal(outcome, config);
+    default:
+      return assertNeverSessionDb(outcome);
+  }
+}
 
 // src/file-ops.ts
 init_esm_shims();
@@ -39708,6 +39791,51 @@ async function modelAuthReady(config, fileOps) {
   return config.hasModelApiKey;
 }
 
+// src/replica-keys.ts
+init_esm_shims();
+var DEFAULT_UNUSABLE_REPLICA_STRATEGY = "prune";
+var STRATEGIES = ["prune", "leave", "clear", "crash"];
+function parseUnusableReplicaStrategy(value) {
+  if (value === void 0) return DEFAULT_UNUSABLE_REPLICA_STRATEGY;
+  return STRATEGIES.includes(value) ? value : null;
+}
+function replicaDbPrefix(prefix) {
+  return `${prefix}/opencode.db/`;
+}
+var LTX_FILENAME = /^([0-9a-fA-F]+)-([0-9a-fA-F]+)\.ltx$/;
+function parseLtxKey(prefix, key) {
+  if (key.split("/").includes("..")) return null;
+  const root12 = replicaDbPrefix(prefix);
+  if (!key.startsWith(root12)) return null;
+  const [levelPart, filename, ...rest] = key.slice(root12.length).split("/");
+  if (rest.length > 0 || filename === void 0 || !/^\d+$/.test(levelPart)) return null;
+  const match = LTX_FILENAME.exec(filename);
+  if (!match) return null;
+  return {
+    level: Number.parseInt(levelPart, 10),
+    minTxid: BigInt(`0x${match[1]}`),
+    maxTxid: BigInt(`0x${match[2]}`)
+  };
+}
+function newestL0Key(prefix, keys) {
+  let newest = null;
+  for (const key of keys) {
+    const parsed = parseLtxKey(prefix, key);
+    if (parsed === null || parsed.level !== 0) continue;
+    if (newest === null || parsed.maxTxid > newest.maxTxid) {
+      newest = { key, maxTxid: parsed.maxTxid };
+    }
+  }
+  return newest?.key ?? null;
+}
+function hasParsableLtxKey(prefix, keys) {
+  return keys.some((key) => parseLtxKey(prefix, key) !== null);
+}
+function isDeletableReplicaKey(prefix, key) {
+  if (!key.startsWith(replicaDbPrefix(prefix))) return false;
+  return !key.split("/").includes("..");
+}
+
 // src/restore.ts
 init_esm_shims();
 import { dirname as dirname3 } from "path";
@@ -43273,6 +43401,10 @@ var _ep4 = {
   DisableS3ExpressSessionAuth: { type: "staticContextParams", value: true },
   Bucket: { type: "contextParams", name: "Bucket" }
 };
+var _ep8 = {
+  Bucket: { type: "contextParams", name: "Bucket" },
+  Prefix: { type: "contextParams", name: "Prefix" }
+};
 var _mw011 = (Command3, cs, config, o7) => [
   getThrow200ExceptionsPlugin(config)
 ];
@@ -49549,11 +49681,21 @@ var S3Client = class extends Client {
   }
 };
 
+// ../../node_modules/.pnpm/@aws-sdk+client-s3@3.1095.0/node_modules/@aws-sdk/client-s3/dist-es/commands/DeleteObjectCommand.js
+init_esm_shims();
+var DeleteObjectCommand = class extends command11(_ep011, _mw011, "DeleteObject", DeleteObject$) {
+};
+
 // ../../node_modules/.pnpm/@aws-sdk+client-s3@3.1095.0/node_modules/@aws-sdk/client-s3/dist-es/commands/GetObjectCommand.js
 init_esm_shims();
 var GetObjectCommand = class extends command11(_ep011, _mw7, "GetObject", GetObject$) {
 };
 
+// ../../node_modules/.pnpm/@aws-sdk+client-s3@3.1095.0/node_modules/@aws-sdk/client-s3/dist-es/commands/ListObjectsV2Command.js
+init_esm_shims();
+var ListObjectsV2Command = class extends command11(_ep8, _mw011, "ListObjectsV2", ListObjectsV2$) {
+};
+
 // ../../node_modules/.pnpm/@aws-sdk+client-s3@3.1095.0/node_modules/@aws-sdk/client-s3/dist-es/commands/PutObjectCommand.js
 init_esm_shims();
 var PutObjectCommand = class extends command11(_ep011, _mw11, "PutObject", PutObject$) {
@@ -49588,8 +49730,200 @@ var S3ObjectStore = class {
   async put(key, body) {
     await this.client.send(new PutObjectCommand({ Bucket: this.bucket, Key: key, Body: body }));
   }
+  async list(prefix) {
+    const keys = [];
+    let continuationToken;
+    do {
+      const response = await this.client.send(
+        new ListObjectsV2Command({
+          Bucket: this.bucket,
+          Prefix: prefix,
+          ContinuationToken: continuationToken
+        })
+      );
+      for (const object of response.Contents ?? []) {
+        if (object.Key) keys.push(object.Key);
+      }
+      continuationToken = response.NextContinuationToken;
+    } while (continuationToken);
+    return keys;
+  }
+  async delete(key) {
+    try {
+      await this.client.send(new DeleteObjectCommand({ Bucket: this.bucket, Key: key }));
+    } catch (error) {
+      if (isNotFound(error)) return;
+      throw error;
+    }
+  }
 };
 
+// src/session-db.ts
+init_esm_shims();
+
+// src/replica-recovery.ts
+init_esm_shims();
+
+// src/ltx-format.ts
+init_esm_shims();
+var LTX_HEADER_SIZE = 100;
+var LTX_MAGIC = "LTX1";
+function isStructurallySoundLtxObject(bytes) {
+  if (bytes.length < LTX_HEADER_SIZE) return false;
+  return bytes.subarray(0, LTX_MAGIC.length).toString("ascii") === LTX_MAGIC;
+}
+
+// src/replica-recovery.ts
+async function probeReplica(store, prefix, log) {
+  const root12 = replicaDbPrefix(prefix);
+  try {
+    const keys = await store.list(root12);
+    return { ok: true, keys };
+  } catch (error) {
+    const detail = describeError(error);
+    log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
+    return { ok: false, detail };
+  }
+}
+function logLayoutMismatch(prefix, keys, log) {
+  log(
+    `WARNING: SESSION-DB-REPLICA-LAYOUT-MISMATCH: listed ${keys.length} object(s) under the replica prefix ${replicaDbPrefix(prefix)} but none of them match the expected LTX key layout (<prefix>/opencode.db/<level>/<minTxid>-<maxTxid>.ltx). Sample key: ${keys[0]}. --on-unusable-replica=prune cannot select anything to delete until this is fixed.`
+  );
+}
+async function pruneNewestL0(store, prefix, keys, log) {
+  const target = newestL0Key(prefix, keys);
+  if (target === null) {
+    if (keys.length > 0 && !hasParsableLtxKey(prefix, keys)) {
+      logLayoutMismatch(prefix, keys, log);
+      return { deleted: null, reason: "layoutMismatch" };
+    }
+    return { deleted: null, reason: "noL0Present" };
+  }
+  if (!isDeletableReplicaKey(prefix, target)) {
+    log(`WARNING: refusing to prune ${target}: failed the replica-key guard.`);
+    return { deleted: null, reason: "layoutMismatch" };
+  }
+  let bytes;
+  try {
+    bytes = await store.get(target);
+  } catch (error) {
+    log(
+      `WARNING: could not re-read ${target} to check whether it is actually corrupt before pruning it: ${describeError(error)}. Declining to prune without positive evidence.`
+    );
+    return { deleted: null, reason: "targetHealthy" };
+  }
+  if (bytes === null || isStructurallySoundLtxObject(bytes)) {
+    log(
+      `INFO: the newest L0 replica object ${target} ` + (bytes === null ? "is already gone" : "looks structurally sound") + "; declining to prune it without positive evidence it is the corrupt one."
+    );
+    return { deleted: null, reason: "targetHealthy" };
+  }
+  log(`INFO: pruning the newest L0 replica object ${target} (fails the LTX structural check).`);
+  try {
+    await store.delete(target);
+  } catch (error) {
+    log(`WARNING: failed to prune ${target}: ${describeError(error)}`);
+    return { deleted: null, reason: "deleteFailed" };
+  }
+  return { deleted: target };
+}
+async function clearReplica(store, prefix, keys, log) {
+  const deleted = [];
+  const failed = [];
+  for (const key of keys) {
+    if (!isDeletableReplicaKey(prefix, key)) {
+      log(`WARNING: refusing to delete ${key}: outside the replica prefix.`);
+      continue;
+    }
+    try {
+      await store.delete(key);
+      deleted.push(key);
+    } catch (error) {
+      log(`WARNING: failed to delete ${key}: ${describeError(error)}`);
+      failed.push(key);
+    }
+  }
+  return { deleted, failed };
+}
+
+// src/session-db.ts
+function assertNeverStrategy(value) {
+  throw new Error(`Unhandled unusable-replica strategy: ${JSON.stringify(value)}`);
+}
+async function classifySessionDbRestore(params) {
+  const outcome = await decideSessionDbRestore(params);
+  if (outcome.kind === "unusableReplica") {
+    await discardLocalDebris(params.fileOps, params.config.opencodeDbPath, params.log);
+  }
+  return outcome;
+}
+async function decideSessionDbRestore(params) {
+  const { exitCode, attempt, strategy, config, fileOps, store, log } = params;
+  if (config.bucket === null || config.prefix === null) {
+    return { kind: "disabled" };
+  }
+  const prefix = config.prefix;
+  if (exitCode === 0) {
+    const stat2 = await fileOps.stat(config.opencodeDbPath);
+    return stat2 !== null && stat2.size > 0 ? { kind: "restored" } : { kind: "noReplica" };
+  }
+  if (store === null) {
+    return {
+      kind: "fatal",
+      cause: "misconfig",
+      detail: "persistence is enabled (LITESTREAM_BUCKET/LITESTREAM_PREFIX are set) but no object store is available to verify the replica is reachable"
+    };
+  }
+  if (attempt === 1) {
+    return { kind: "retryTransient" };
+  }
+  if (attempt >= 3) {
+    return { kind: "unusableReplica", reason: "attemptsExhausted" };
+  }
+  const probe = await probeReplica(store, prefix, log);
+  if (!probe.ok) {
+    return {
+      kind: "fatal",
+      cause: "misconfig",
+      detail: `could not verify the session-DB replica is reachable: ${probe.detail}`
+    };
+  }
+  const keys = probe.keys;
+  switch (strategy) {
+    case "crash":
+      return {
+        kind: "fatal",
+        cause: "deliberate",
+        detail: "the replica is unusable and --on-unusable-replica=crash was selected; no S3 object was touched"
+      };
+    case "leave":
+      return { kind: "unusableReplica", reason: "strategy" };
+    case "prune": {
+      const result = await pruneNewestL0(store, prefix, keys, log);
+      if (result.deleted === null) {
+        const reason = result.reason === "deleteFailed" ? "recoveryFailed" : result.reason;
+        return { kind: "unusableReplica", reason };
+      }
+      return { kind: "recovered", strategy: "prune", deleted: [result.deleted] };
+    }
+    case "clear": {
+      const result = await clearReplica(store, prefix, keys, log);
+      return { kind: "recovered", strategy: "clear", deleted: result.deleted };
+    }
+    default:
+      return assertNeverStrategy(strategy);
+  }
+}
+async function discardLocalDebris(fileOps, dbPath, log) {
+  for (const path2 of [dbPath, `${dbPath}-wal`, `${dbPath}-shm`]) {
+    try {
+      await fileOps.remove(path2);
+    } catch (error) {
+      log(`WARNING: could not remove local session-DB debris at ${path2}: ${describeError(error)}`);
+    }
+  }
+}
+
 // src/self-stop.ts
 init_esm_shims();
 function isComplete(credentials) {
@@ -49666,7 +50000,15 @@ async function selfStop({
 
 // src/shell-contract.json
 var shell_contract_default = {
-  commands: ["env", "litestream-config", "restore", "sync-once", "model-auth-ready", "self-stop"]
+  commands: [
+    "env",
+    "litestream-config",
+    "restore",
+    "sync-once",
+    "model-auth-ready",
+    "self-stop",
+    "session-db-classify"
+  ]
 };
 
 // src/sync.ts
@@ -49718,6 +50060,9 @@ var EXIT_NOT_READY = 10;
 var EXIT_KEEP_TASK = 20;
 var EXIT_USAGE = 2;
 var EXIT_BROKEN = 1;
+var EXIT_SESSION_DB_FATAL = 30;
+var EXIT_SESSION_DB_UNUSABLE = 31;
+var EXIT_SESSION_DB_RETRY = 32;
 var STATE_SUFFIX = ".synchash";
 var COMMANDS = shell_contract_default.commands;
 var USAGE = `Usage: runner-synchroniser <command> [args]
@@ -49729,6 +50074,12 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
   model-auth-ready             exit 0 when some model auth is configured, ${EXIT_NOT_READY} when not
   self-stop                    scale this agent's own ECS service to 0; exit 0 only
                                when desiredCount is confirmed 0, ${EXIT_KEEP_TASK} to keep the task
+  session-db-classify <litestream-restore-exit-code> <attempt>
+                               [--on-unusable-replica=<prune|leave|clear|crash>]
+                               classify a just-run \`litestream restore\` of opencode.db;
+                               exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
+                               and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
+                               skip replicate), ${EXIT_SESSION_DB_FATAL} fatal
 `;
 function shellQuote(value) {
   return `'${value.replaceAll("'", `'\\''`)}'`;
@@ -49742,11 +50093,39 @@ function awsSelfStopAccessFor(config) {
 function parseStoreName(value) {
   return value === "claude" || value === "opencode" ? value : null;
 }
+var ON_UNUSABLE_REPLICA_FLAG = "--on-unusable-replica=";
+function parseNonNegativeInt(value) {
+  return /^\d+$/.test(value) ? Number.parseInt(value, 10) : null;
+}
+function parseSessionDbClassifyArgs(args) {
+  let strategyRaw;
+  const positional = [];
+  for (const arg of args) {
+    if (arg.startsWith(ON_UNUSABLE_REPLICA_FLAG)) {
+      if (strategyRaw !== void 0) return null;
+      strategyRaw = arg.slice(ON_UNUSABLE_REPLICA_FLAG.length);
+    } else if (arg.startsWith("--")) {
+      return null;
+    } else {
+      positional.push(arg);
+    }
+  }
+  if (positional.length !== 2) return null;
+  const exitCode = parseNonNegativeInt(positional[0]);
+  const attempt = parseNonNegativeInt(positional[1]);
+  const strategy = parseUnusableReplicaStrategy(strategyRaw);
+  if (exitCode === null || attempt === null || attempt < 1 || strategy === null) return null;
+  return { exitCode, attempt, strategy };
+}
 function renderEnv(config) {
   return [
-    // Empty when persistence is disabled — entrypoint.sh gates the sync loop
-    // on this being non-empty, exactly as it did when it computed it itself.
-    `CREDS_BUCKET=${shellQuote(config.bucket ?? "")}`,
+    // The bucket NAME, kept (not reduced to a boolean) so an operator can
+    // read it; empty exactly when persistence is disabled (both
+    // LITESTREAM_BUCKET and LITESTREAM_PREFIX must be set — see config.ts's
+    // persistenceEnabled). entrypoint.sh gates BOTH the credential-sync
+    // loop and whether it starts litestream replicate on this being
+    // non-empty.
+    `PERSISTENCE_BUCKET=${shellQuote(config.bucket ?? "")}`,
     `CLAUDE_CREDS=${shellQuote(config.claude.path)}`,
     `OPENCODE_DB_PATH=${shellQuote(config.opencodeDbPath)}`,
     `CREDS_SYNC_INTERVAL=${shellQuote(String(config.syncIntervalSeconds))}`
@@ -49812,6 +50191,20 @@ async function commandSyncOnce(name, config, deps) {
     await writeLastHash(fileOps, statePath, result.hash, log);
   }
 }
+async function commandSessionDbClassify(args, config, deps) {
+  const { fileOps, log } = deps;
+  const outcome = await classifySessionDbRestore({
+    exitCode: args.exitCode,
+    attempt: args.attempt,
+    strategy: args.strategy,
+    config: { bucket: config.bucket, prefix: config.prefix, opencodeDbPath: config.opencodeDbPath },
+    fileOps,
+    store: (deps.objectStoreFor ?? s3ObjectStoreFor)(config),
+    log
+  });
+  log(describeSessionDbClassification(outcome, { bucket: config.bucket, region: config.region }));
+  return sessionDbExitCode(outcome);
+}
 async function main(argv, deps) {
   const { env: env3, fileOps, log, out } = deps;
   const [command12, ...rest] = argv;
@@ -49847,6 +50240,17 @@ ${USAGE}`);
     });
     return stopped ? 0 : EXIT_KEEP_TASK;
   }
+  if (command12 === "session-db-classify") {
+    const args = parseSessionDbClassifyArgs(rest);
+    if (args === null) {
+      log(
+        `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>.
+${USAGE}`
+      );
+      return EXIT_USAGE;
+    }
+    return commandSessionDbClassify(args, config, deps);
+  }
   const name = parseStoreName(rest[0]);
   if (name === null) {
     log(`'${command12}' needs a store name: claude or opencode.
@@ -49886,6 +50290,9 @@ export {
   EXIT_BROKEN,
   EXIT_KEEP_TASK,
   EXIT_NOT_READY,
+  EXIT_SESSION_DB_FATAL,
+  EXIT_SESSION_DB_RETRY,
+  EXIT_SESSION_DB_UNUSABLE,
   EXIT_USAGE,
   main
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evident-ai/runner-synchroniser",
-  "version": "0.1.1-dev.0d89dd3",
+  "version": "0.1.1-dev.1e72251",
   "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
   "type": "module",
   "main": "./dist/cli.js",