@evident-ai/runner-synchroniser 3.4.1-dev.7802262 → 3.4.1-dev.b740141

package/README.md

@@ -59,8 +59,8 @@ build time.
 | `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
-| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
-| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
+| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, replicates into the existing prefix), `30` = fatal; other = usage/tool broken |
+| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = exhausted after verified separation and local disposal, `34` = separation or disposal could not be proven; other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
 (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -75,7 +75,7 @@ saved query.
 
 ### `session-db-classify`
 
-Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
+Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `runner/docker-images/fargate/README.md`'s [Strategy/What/Cost table](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
 
 #### Activity recovery report
 
@@ -213,7 +213,7 @@ what it is.
 #### The four `--on-unusable-replica` strategies
 
 - **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
-- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". Since #1106 it is **not** also ephemeral: the boot still starts a fresh backup chain, because the classifier guarantees the local DB is fresh rather than half-restored. Upside: replica left byte-intact for forensics, no crash loop.
+- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". The fresh local DB replicates into the existing prefix; no new chain is started. Upside: the runner still comes online without a crash loop.
 - **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
 - **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
 
@@ -235,7 +235,7 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
 (`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
 and it can never repair corruption at a higher compaction level.
 
-`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
+`EVIDENT_ON_UNUSABLE_REPLICA` is a `runner/docker-images/fargate` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [the Fargate image README](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
 
 See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
 the behavioural anchor (31 comes online, 30 does not).
@@ -262,9 +262,12 @@ nothing had ever run a real `PRAGMA integrity_check`).
    each into a scratch path and integrity-checking it there. The first one that passes
    is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
 3. If nothing retained passes (or there's nothing to try, or the search runs out of
-   time/points), the corrupt local DB is discarded and the boot proceeds with a
-   genuinely fresh one — exactly `session-db-classify`'s `31` guarantee, so
-   `litestream replicate` starting against it afterwards is just as safe.
+   time/points), the active replica prefix is separated first: its history is moved aside,
+   then the prefix is re-listed as empty before the corrupt local DB and sidecars are
+   discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
+   OpenCode or Litestream starts; the backup history remains readable at its quarantine
+    destination or original key, and surviving local files remain in place without a process
+    opening or writing them.
 
 A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
 prefix. litestream re-bases to the replica's high-water mark and continues the txid
@@ -279,10 +282,8 @@ against S3 at the ~1 GB scale a production replica can reach.)
 | `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
 | `healthy` | `0` | the local DB passed its integrity check as-is |
 | `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
-| `exhausted` (`noCandidates`) | `33` | the local DB failed and litestream retains no older restore point |
-| `exhausted` (`allFailed`) | `33` | every retained restore point was tried and every one failed |
-| `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
-| `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
+| `exhausted` (`noCandidates`, `allFailed`, `budgetExhausted`, or `enumerationFailed`) | `33` | the active prefix was proven separated (or persistence is disabled) **and** the corrupt local DB and sidecars were proven discarded |
+| `exhausted` | `34` | either the replica separation or local discard could not be proven; the boot stops before OpenCode and Litestream start |
 
 `healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
 afterwards — start `litestream replicate` — is identical either way, so a further split
@@ -301,7 +302,10 @@ before looking at everything" would report the wrong conclusion to the operator.
 | `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
 | `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
 | `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
-| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed (any `exhausted` reason); the boot proceeds with a fresh DB | `candidatesTried` (in prose, not `k=v`) |
+| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed; a fresh boot follows only when both separation and local disposal were proven | `candidatesTried` (in prose, not `k=v`) |
+| `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
+| `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
+| `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
 
 The first four fire only while the walkback loop runs; the last three are always the
 one message this command logs as its own final word on the outcome — `entrypoint.sh`
@@ -334,13 +338,14 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
 `session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
 
 `session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
-history. It is drained once by `evident run` into runner activity and does not alter its exit
-code.
+history. A refused boot records `session_db_boot_refused`, but `evident run` has not started to
+drain it, so operators see that refusal in boot logs and the restart loop. The report does not
+alter the command's exit code.
 
 `src/shell-contract.json` is the machine-checked source of truth for the command list, and
 `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
-`packages/runner-image/entrypoint.sh` and the MicroVM's
-`packages/runner-cdk/microvm-image/hooks` (#608), discovered by grep so a third one
+`runner/docker-images/fargate/entrypoint.sh` and the MicroVM's
+`runner/docker-images/microvm/hooks` (#608), discovered by grep so a third one
 cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
 `entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
 a broken tool; and take each answer code the commands it calls can return **silently and by
@@ -370,7 +375,7 @@ key, so they never appear in `env`'s output or `litestream.yml`.
 | `CLUSTER` † / `SERVICE` †                 | ECS cluster/service `self-stop` scales to `desiredCount=0`.              | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
 | `EVIDENT_SELFSTOP_ROLE_ARN` †             | Role `self-stop` assumes for its ECS calls.                              | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
 
-`packages/runner-image/README.md` documents these same three from the deployment side —
+`runner/docker-images/fargate/README.md` documents these same three from the deployment side —
 keep them in sync.
 
 Requiring **both** `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` (never just one) means

package/dist/cli.js

@@ -52000,7 +52000,7 @@ function sessionDbExitCode(outcome) {
     // re-run `litestream restore` and ask again
     case "unusableReplica":
       return 31;
-    // booted with a fresh DB; the shell still starts a fresh backup chain (#1106)
+    // booted with a fresh DB; the shell still replicates into the existing prefix
     case "fatal":
       return 30;
     // shared by a genuine misconfig and a deliberate `crash` choice
@@ -52063,7 +52063,7 @@ function describeSessionDbClassification(outcome, config) {
     case "recovered": {
       const objects = outcome.deleted.length === 0 ? "no objects (see the warnings above for what failed)" : `${outcome.deleted.length} object${outcome.deleted.length === 1 ? "" : "s"} (${outcome.deleted.join(", ")})`;
       if (outcome.strategy === "quarantine") {
-        return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside, NOT deleted \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
+        return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside and remains readable at the quarantine destination \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
       }
       return `Attempted opencode.db replica recovery via --on-unusable-replica=${outcome.strategy}, deleting ${objects}; retrying the restore.` + (outcome.strategy === "clear" ? " Prior session history is lost." : "");
     }
@@ -52084,6 +52084,12 @@ function assertNeverSessionDbVerifySkippedLocalDb(value) {
 function assertNeverSessionDbVerifyExhaustedReason(value) {
   throw new Error(`Unhandled session-DB verify exhausted reason: ${JSON.stringify(value)}`);
 }
+function assertNeverSessionDbVerifySeparation(value) {
+  throw new Error(`Unhandled session-DB verify separation: ${JSON.stringify(value)}`);
+}
+function assertNeverSessionDbVerifyLocalDb(value) {
+  throw new Error(`Unhandled session-DB verify local-DB disposal: ${JSON.stringify(value)}`);
+}
 function sessionDbVerifyExitCode(outcome) {
   switch (outcome.kind) {
     case "skipped":
@@ -52091,8 +52097,26 @@ function sessionDbVerifyExitCode(outcome) {
     case "walkedBack":
       return 0;
     case "exhausted":
-      return 33;
-    // booted with a fresh DB; the shell still starts a fresh backup chain
+      switch (outcome.separation.kind) {
+        case "notConfigured":
+        case "alreadyEmpty":
+        case "quarantined":
+          break;
+        case "incomplete":
+        case "unreachable":
+          return 34;
+        default:
+          return assertNeverSessionDbVerifySeparation(outcome.separation);
+      }
+      switch (outcome.localDb.kind) {
+        case "discarded":
+          return 33;
+        case "retained":
+        case "discardFailed":
+          return 34;
+        default:
+          return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
+      }
     default:
       return assertNeverSessionDbVerify(outcome);
   }
@@ -52109,18 +52133,56 @@ function describeSessionDbVerifySkipped(localDb) {
 }
 function describeSessionDbVerifyExhausted(outcome) {
   const tried = `tried ${outcome.candidatesTried} restore point${outcome.candidatesTried === 1 ? "" : "s"}`;
+  let reason;
   switch (outcome.reason) {
     case "enumerationFailed":
-      return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried}); booting with a FRESH opencode.db.`;
+      reason = `opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried})`;
+      break;
     case "noCandidates":
-      return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and no older restore point is retained (${tried}); booting with a FRESH opencode.db.`;
+      reason = `opencode.db failed its integrity check and no older restore point is retained (${tried})`;
+      break;
     case "allFailed":
-      return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and every retained restore point also failed (${tried}); booting with a FRESH opencode.db.`;
+      reason = `opencode.db failed its integrity check and every retained restore point also failed (${tried})`;
+      break;
     case "budgetExhausted":
-      return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try; booting with a FRESH opencode.db.`;
+      reason = `opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try`;
+      break;
     default:
       return assertNeverSessionDbVerifyExhaustedReason(outcome.reason);
   }
+  let separation;
+  switch (outcome.separation.kind) {
+    case "quarantined":
+      separation = ` The active replica prefix was verified empty; moved history remains readable at ${outcome.separation.destination}.`;
+      break;
+    case "alreadyEmpty":
+      separation = " The active replica prefix was already empty.";
+      break;
+    case "notConfigured":
+      separation = " Persistence is disabled, so there is no replica to separate.";
+      break;
+    case "incomplete":
+    case "unreachable":
+      separation = ` Boot is stopping because replica separation could not be proven (${outcome.separation.detail}); backup history remains readable at its original key or quarantine destination.`;
+      break;
+    default:
+      return assertNeverSessionDbVerifySeparation(outcome.separation);
+  }
+  let localDb;
+  switch (outcome.localDb.kind) {
+    case "discarded":
+      localDb = " booting with a FRESH opencode.db.";
+      break;
+    case "retained":
+      localDb = " The corrupt local database and sidecars were left untouched for investigation.";
+      break;
+    case "discardFailed":
+      localDb = ` Boot is stopping because local disposal could not be proven; removed=${outcome.localDb.removed.join(", ") || "(none)"} survived=${outcome.localDb.survived.join(", ") || "(none)"}. See SESSION-DB-LOCAL-DISCARD-FAILED.`;
+      break;
+    default:
+      return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
+  }
+  return `SESSION-DB-INTEGRITY-EXHAUSTED: ${reason}.${separation}${localDb}`;
 }
 function describeSessionDbVerification(outcome) {
   switch (outcome.kind) {
@@ -52162,8 +52224,9 @@ var nodeFileOps = {
     try {
       const stats = await stat(path);
       return { size: stats.size };
-    } catch {
-      return null;
+    } catch (error) {
+      if (error instanceof Error && "code" in error && error.code === "ENOENT") return null;
+      throw error;
     }
   },
   readFile: readFile5,
@@ -62435,8 +62498,40 @@ async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PUR
 }
 function logQuarantined(destination, moved, failed, log) {
   log(
-    `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved) so this boot can start a fresh backup chain. The bytes are NOT deleted \u2014 copy them back from there to investigate. See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
+    `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (failed.length > 0 ? " The active replica prefix is not clear." : "") + `See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
+  );
+}
+async function separateCorruptReplica(store, prefix, log, stamp) {
+  const probe = await probeReplica(store, prefix, log);
+  if (!probe.ok) return { kind: "unreachable", detail: probe.detail };
+  const root12 = replicaDbPrefix(prefix);
+  if (probe.objects.length === 0) {
+    log(
+      `INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was already empty; this boot starts a fresh backup chain.`
+    );
+    return { kind: "alreadyEmpty" };
+  }
+  const result = await quarantineReplica(store, prefix, probe.objects, log, stamp);
+  try {
+    const remaining = await store.list(root12);
+    if (remaining.length > 0) {
+      const detail = `${remaining.length} object(s) remain`;
+      log(
+        `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: ${root12} still has ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
+      );
+      return { kind: "incomplete", detail };
+    }
+  } catch (error) {
+    const detail = describeError(error);
+    log(
+      `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: could not re-list ${root12}: ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
+    );
+    return { kind: "incomplete", detail };
+  }
+  log(
+    `INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was verified empty by re-listing after moving ${result.moved.length} object(s) to ${result.destination}; this boot starts a fresh backup chain.`
   );
+  return { kind: "quarantined", destination: result.destination, moved: result.moved };
 }
 async function clearReplica(store, prefix, keys, log) {
   const deleted = [];
@@ -62648,6 +62743,15 @@ function recordVerifyOutcome(outcome, at2) {
       verified_restore_point: outcome.txid,
       restore_points_tried: outcome.candidatesTried
     };
+  if (sessionDbVerifyExitCode(outcome) === 34)
+    return {
+      ...record,
+      outcome: "session_db_boot_refused",
+      severity: "error",
+      reason: outcome.localDb.kind === "discardFailed" ? "local_discard_failed" : "replica_separation_unproven",
+      restore_points_tried: outcome.candidatesTried,
+      quarantine_destination: outcome.separation.kind === "quarantined" ? outcome.separation.destination : null
+    };
   return {
     ...record,
     outcome: "fresh_session_db",
@@ -62730,7 +62834,11 @@ async function verifySessionDb(params) {
   log(
     `WARNING: SESSION-DB-INTEGRITY-FAILED: opencode.db at ${dbPath} failed its integrity check: ${result.detail}`
   );
-  return walkBack(params);
+  const decision = await walkBack(params);
+  if (decision.kind !== "exhausted") return decision;
+  const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(params.store, config.prefix, log);
+  const localDb = separation.kind === "notConfigured" || separation.kind === "alreadyEmpty" || separation.kind === "quarantined" ? await discardCorruptDb(fileOps, dbPath, log) : { kind: "retained", reason: "separationUnproven" };
+  return { ...decision, separation, localDb };
 }
 async function clearScratch(fileOps, scratch) {
   await fileOps.remove(scratch);
@@ -62738,15 +62846,32 @@ async function clearScratch(fileOps, scratch) {
   await fileOps.remove(`${scratch}-shm`);
 }
 async function discardCorruptDb(fileOps, dbPath, log) {
-  for (const path of [dbPath, `${dbPath}-wal`, `${dbPath}-shm`]) {
+  const paths = [dbPath, `${dbPath}-wal`, `${dbPath}-shm`];
+  const errors = [];
+  for (const path of paths) {
     try {
       await fileOps.remove(path);
     } catch (error) {
-      log(
-        `WARNING: could not remove corrupt local session-DB debris at ${path}: ${describeError(error)}`
-      );
+      errors.push(`${path}: ${describeError(error)}`);
     }
   }
+  const survived = [];
+  const removed = [];
+  for (const path of paths) {
+    try {
+      if (await fileOps.stat(path) === null) removed.push(path);
+      else survived.push(path);
+    } catch (error) {
+      survived.push(path);
+      errors.push(`${path}: could not prove removal: ${describeError(error)}`);
+    }
+  }
+  if (errors.length === 0 && survived.length === 0) return { kind: "discarded" };
+  const detail = errors.length === 0 ? "paths remained after removal" : errors.join("; ");
+  log(
+    `WARNING: SESSION-DB-LOCAL-DISCARD-FAILED: removed=${removed.join(", ") || "(none)"} survived=${survived.join(", ") || "(none)"} errors=${detail}; boot is stopping so no process opens or writes what remains.`
+  );
+  return { kind: "discardFailed", removed, survived, detail };
 }
 async function adopt(fileOps, scratch, dbPath, log, txid, candidatesTried, bytes) {
   await fileOps.remove(`${dbPath}-wal`);
@@ -62768,12 +62893,10 @@ async function walkBack(params) {
     log(
       `WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED: ${listing.stdout === null ? `litestream ltx exited non-zero: ${listing.detail}` : "litestream ltx succeeded but its listing could not be parsed"}`
     );
-    await discardCorruptDb(fileOps, dbPath, log);
     return { kind: "exhausted", reason: "enumerationFailed", candidatesTried: 0 };
   }
   const candidates = walkbackCandidates(points);
   if (candidates.length === 0) {
-    await discardCorruptDb(fileOps, dbPath, log);
     return { kind: "exhausted", reason: "noCandidates", candidatesTried: 0 };
   }
   const deadline = now() + budgetSeconds * 1e3;
@@ -62804,7 +62927,6 @@ async function walkBack(params) {
     return { kind: "walkedBack", txid, bytes, candidatesTried };
   }
   await clearScratch(fileOps, scratch);
-  await discardCorruptDb(fileOps, dbPath, log);
   return {
     kind: "exhausted",
     reason: index < candidates.length ? "budgetExhausted" : "allFailed",
@@ -62989,6 +63111,7 @@ var EXIT_SESSION_DB_FATAL = 30;
 var EXIT_SESSION_DB_UNUSABLE = 31;
 var EXIT_SESSION_DB_RETRY = 32;
 var EXIT_SESSION_DB_UNVERIFIABLE = 33;
+var EXIT_SESSION_DB_UNSEPARATED = 34;
 var STATE_SUFFIX = ".synchash";
 var COMMANDS = shell_contract_default.commands;
 var USAGE = `Usage: runner-synchroniser <command> [args]
@@ -63005,13 +63128,14 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
                                 [--fresh-db-fallback]
                                classify a just-run \`litestream restore\` of opencode.db;
                                exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
-                               and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
-                               fresh backup chain), ${EXIT_SESSION_DB_FATAL} fatal
+                                and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
+                                replicates into the existing prefix), ${EXIT_SESSION_DB_FATAL} fatal
   session-db-verify <litestream-config-path>
                                verify opencode.db's integrity at boot, walking back through
                                litestream's retained restore points if it's corrupt; exit 0
                                healthy/skipped/walked-back, ${EXIT_SESSION_DB_UNVERIFIABLE} every
-                               candidate was exhausted (booted fresh)
+                                candidate was exhausted (booted fresh), ${EXIT_SESSION_DB_UNSEPARATED}
+                                separation/local disposal could not be proven
 `;
 function shellQuote(value) {
   return `'${value.replaceAll("'", `'\\''`)}'`;
@@ -63170,7 +63294,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
     config.opencodeDbPath
   );
   const outcome = await verifySessionDb({
-    config: { opencodeDbPath: config.opencodeDbPath },
+    config: { opencodeDbPath: config.opencodeDbPath, prefix: config.prefix },
+    store: (deps.objectStoreFor ?? s3ObjectStoreFor)(config),
     fileOps,
     sqlite: (deps.sqliteIntegrityFor ?? nodeSqliteIntegrityFor)(),
     litestream,
@@ -63279,6 +63404,7 @@ export {
   EXIT_NOT_READY,
   EXIT_SESSION_DB_FATAL,
   EXIT_SESSION_DB_RETRY,
+  EXIT_SESSION_DB_UNSEPARATED,
   EXIT_SESSION_DB_UNUSABLE,
   EXIT_SESSION_DB_UNVERIFIABLE,
   EXIT_USAGE,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evident-ai/runner-synchroniser",
-  "version": "3.4.1-dev.7802262",
+  "version": "3.4.1-dev.b740141",
   "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
   "type": "module",
   "main": "./dist/cli.js",
@@ -26,7 +26,7 @@
     "tsup": "^8.3.5",
     "typescript": "^5.7.2",
     "vitest": "^3.1.1",
-    "runner-image": "workspace:*"
+    "runner-fargate-image": "workspace:*"
   },
   "engines": {
     "node": ">=22.13.0"
@@ -42,7 +42,7 @@
   "repository": {
     "type": "git",
     "url": "https://github.com/sroze/evident.git",
-    "directory": "packages/runner-synchroniser"
+    "directory": "runner/synchroniser"
   },
   "homepage": "https://evident.run",
   "author": "Evident <hello@evident.run>",