@evident-ai/runner-synchroniser 3.5.2-dev.06654ec → 3.5.2-dev.31c0c2e

package/README.md

@@ -278,8 +278,13 @@ nothing had ever run a real `PRAGMA integrity_check`).
    then the prefix is re-listed as empty before the corrupt local DB and sidecars are
    discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
    OpenCode or Litestream starts; the backup history remains readable at its quarantine
-    destination or original key, and surviving local files remain in place without a process
-    opening or writing them.
+   destination or original key, and surviving local files remain in place without a process
+   opening or writing them.
+
+Each restore-point listing and each individual candidate restore has its own hard timeout.
+Replica separation also has a wall-clock budget. When that budget expires, the synchroniser
+reports the number moved and remaining, leaves the active prefix unverified, and exits `34`;
+the next refused start can continue moving the remaining prefix.
 
 A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
 prefix. litestream re-bases to the replica's high-water mark and continues the txid
@@ -311,6 +316,7 @@ before looking at everything" would report the wrong conclusion to the operator.
 | `WARNING: SESSION-DB-INTEGRITY-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB fails its check, before the walkback starts | — (prose carries the PRAGMA's own detail) |
 | `WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED` | the walkback loop (`session-db-verify.ts`) | `litestream ltx` exited non-zero (prose carries its stderr — missing binary, bad `-config`, S3 error) or answered with a listing that could not be parsed | — (prose carries litestream's own stderr) |
 | `WARNING: SESSION-DB-WALKBACK-CANDIDATE-FAILED` | the walkback loop (`session-db-verify.ts`) | A candidate restore point fails to restore, or restores but also fails its own check | `txid` |
+| `WARNING: SESSION-DB-REPLICA-QUARANTINED` | `quarantineReplica` (`replica-recovery.ts`) | A corrupt replica is moved aside, whether the move finishes or its disposal budget stops it early | `moved`, `remaining`, quarantine destination |
 | `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
 | `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
 | `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
@@ -318,11 +324,36 @@ before looking at everything" would report the wrong conclusion to the operator.
 | `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
 | `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
 | `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
-
-The first four fire only while the walkback loop runs; the last three are always the
-one message this command logs as its own final word on the outcome — `entrypoint.sh`
-reuses `SESSION-DB-INTEGRITY-EXHAUSTED` verbatim when it logs its own `33` handling, the
-same way it already reuses `SESSION-DB-REPLICA-UNUSABLE` for `session-db-classify`'s `31`.
+| `SESSION-DB-VERIFY-TIMEOUT` | the boot wrapper around `session-db-verify` | Verification did not finish before its outer deadline; the message says whether corruption was already proven and whether separation had started | timeout and refusal reason |
+| `SESSION-DB-VERIFY-SKIPPED` | `evident run`'s session-DB step | The start serves a session database without verifying it: the restore flag was absent, OpenCode was already serving it, or the health check returned an authentication challenge. `flag-absent` is expected on a `/resume` start by design. | `reason` (`flag-absent`\|`opencode-serving`\|`opencode-auth-challenge`) |
+
+The walkback markers describe each attempt and its progress. The final outcome markers
+describe whether the active prefix and local files were proven safe; `entrypoint.sh` reuses
+`SESSION-DB-INTEGRITY-EXHAUSTED` verbatim when it logs its own `33` handling, the same way it
+already reuses `SESSION-DB-REPLICA-UNUSABLE` for `session-db-classify`'s `31`. The CLI wrapper
+emits `SESSION-DB-VERIFY-TIMEOUT` when the outer verification deadline is reached.
+
+#### Refused verification outcomes
+
+The recovery report records `replica_separation_unproven` when separation did not prove the
+active prefix empty, `local_discard_failed` when local session-database files could not be
+proven removed, `verification_timeout_before_separation` when verification timed out after
+integrity failure but before replica separation began, and
+`verification_timeout_during_separation` when the timeout occurred after separation had
+started. The timeout reasons point to `SESSION-DB-VERIFY-TIMEOUT` and do not claim that
+objects were removed.
+
+On a refused boot, wait while the reported `remaining` count falls: each start moves another
+bounded slice and the runner comes online after the active prefix drains. If it stops falling,
+move or delete the objects under `<prefix>/opencode.db/` manually; the next start then uses a
+fresh session database. Set-aside history remains at
+`<prefix>/quarantine/opencode.db/<stamp>/`, with one stamp for each start that moved anything.
+Do not use `EVIDENT_ON_UNUSABLE_REPLICA` for this case; that variable controls the earlier
+restore-failure classification path, not post-restore verification.
+
+A store too large for one candidate restore to finish inside
+`EVIDENT_SESSION_DB_RESTORE_CANDIDATE_TIMEOUT_SECONDS` is refused and drained rather than
+adopting an older restore point.
 
 ### Why the contract is asymmetric
 
@@ -386,6 +417,9 @@ key, so they never appear in `env`'s output or `litestream.yml`.
 | `CREDS_SYNC_INTERVAL`                     | Seconds between sync ticks, reported by `env` for the caller's loop.     | Falls back to `60`; also on unparseable or non-positive values.                   |
 | `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`  | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values.                   |
 | `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop.           | Falls back to `180`; also on unparseable or non-positive values.                  |
+| `EVIDENT_SESSION_DB_RESTORE_POINTS_TIMEOUT_SECONDS` | Hard timeout, in seconds, for one `litestream ltx` restore-point listing. | Falls back to `120`; also on unparseable or non-positive values.                  |
+| `EVIDENT_SESSION_DB_RESTORE_CANDIDATE_TIMEOUT_SECONDS` | Hard timeout, in seconds, for each individual `litestream restore` walkback candidate. | Falls back to `300`; also on unparseable or non-positive values.                  |
+| `EVIDENT_SESSION_DB_DISPOSAL_BUDGET_SECONDS` | Wall-clock budget, in seconds, for moving a corrupt replica aside. | Falls back to `900`; also on unparseable or non-positive values.                  |
 | `EVIDENT_SESSION_DB_RECOVERY_REPORT` | JSONL report drained once into runner activity after authentication. | `$HOME/.local/state/evident/session-db-recovery.jsonl`; blank values use the default. |
 | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`    | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
 | `CLUSTER` † / `SERVICE` †                 | ECS cluster/service `self-stop` scales to `desiredCount=0`.              | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |

package/dist/cli.js

@@ -50887,6 +50887,9 @@ init_esm_shims();
 var DEFAULT_SYNC_INTERVAL_SECONDS = 60;
 var DEFAULT_WALKBACK_MAX_POINTS = 10;
 var DEFAULT_WALKBACK_BUDGET_SECONDS = 180;
+var DEFAULT_RESTORE_POINTS_TIMEOUT_SECONDS = 120;
+var DEFAULT_RESTORE_CANDIDATE_TIMEOUT_SECONDS = 300;
+var DEFAULT_DISPOSAL_BUDGET_SECONDS = 900;
 function positiveIntOr(value, defaultValue) {
   const parsed = Number.parseInt(value ?? "", 10);
   return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue;
@@ -50931,6 +50934,18 @@ function resolveConfig(env4) {
       env4.EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS,
       DEFAULT_WALKBACK_BUDGET_SECONDS
     ),
+    restorePointsTimeoutSeconds: positiveIntOr(
+      env4.EVIDENT_SESSION_DB_RESTORE_POINTS_TIMEOUT_SECONDS,
+      DEFAULT_RESTORE_POINTS_TIMEOUT_SECONDS
+    ),
+    restoreCandidateTimeoutSeconds: positiveIntOr(
+      env4.EVIDENT_SESSION_DB_RESTORE_CANDIDATE_TIMEOUT_SECONDS,
+      DEFAULT_RESTORE_CANDIDATE_TIMEOUT_SECONDS
+    ),
+    disposalBudgetSeconds: positiveIntOr(
+      env4.EVIDENT_SESSION_DB_DISPOSAL_BUDGET_SECONDS,
+      DEFAULT_DISPOSAL_BUDGET_SECONDS
+    ),
     hasModelApiKey: nonEmpty(env4.ANTHROPIC_API_KEY) !== null || nonEmpty(env4.OPENAI_API_KEY) !== null,
     runnerKey: nonEmpty(env4.EVIDENT_RUNNER_KEY) ?? nonEmpty(env4.EVIDENT_AGENT_KEY),
     apiUrl: nonEmpty(env4.EVIDENT_API_URL)
@@ -51241,49 +51256,88 @@ function stderrOf(error) {
   }
   return describeError(error);
 }
+function commandDetail(error, operation4, timeoutSeconds) {
+  if (typeof error === "object" && error !== null) {
+    const processError = error;
+    const signal = typeof processError.signal === "string" ? processError.signal : null;
+    if (processError.killed === true || signal === "SIGKILL") {
+      return `${operation4} timed out after ${timeoutSeconds}s (process killed with ${signal ?? "SIGKILL"})`;
+    }
+  }
+  return stderrOf(error);
+}
 var LitestreamCli = class {
-  constructor(configPath, dbPath) {
+  constructor(configPath, dbPath, timeouts) {
     this.configPath = configPath;
     this.dbPath = dbPath;
+    this.timeouts = timeouts;
   }
   async listRestorePoints() {
     try {
-      const { stdout, stderr } = await execFileAsync("litestream", [
-        "ltx",
-        "-config",
-        this.configPath,
-        // MEASURED: `-level` defaults to 0, which only shows L0 — `all` is
-        // required to see the L1/L2/L9 compaction levels too (plan §2.3).
-        "-level",
-        "all",
-        "-json",
-        this.dbPath
-      ]);
+      const { stdout, stderr } = await execFileAsync(
+        "litestream",
+        [
+          "ltx",
+          "-config",
+          this.configPath,
+          // MEASURED: `-level` defaults to 0, which only shows L0 — `all` is
+          // required to see the L1/L2/L9 compaction levels too (plan §2.3).
+          "-level",
+          "all",
+          "-json",
+          this.dbPath
+        ],
+        {
+          timeout: this.timeouts.restorePointsTimeoutSeconds * 1e3,
+          killSignal: "SIGKILL"
+        }
+      );
       return { stdout, detail: stderr };
     } catch (error) {
-      return { stdout: null, detail: stderrOf(error) };
+      return {
+        stdout: null,
+        detail: commandDetail(
+          error,
+          "litestream restore-point enumeration",
+          this.timeouts.restorePointsTimeoutSeconds
+        )
+      };
     }
   }
   async restoreAt(txid, outputPath) {
     try {
-      const { stderr } = await execFileAsync("litestream", [
-        "restore",
-        "-config",
-        this.configPath,
-        "-txid",
-        txid,
-        // MEASURED: without `-force`, restoring into a path that already
-        // exists exits 1 with "output path already exists and is not
-        // empty" (plan §2.4) — the caller always restores into a scratch
-        // path it just cleared, so overwriting it is always intended.
-        "-force",
-        "-o",
-        outputPath,
-        this.dbPath
-      ]);
+      const { stderr } = await execFileAsync(
+        "litestream",
+        [
+          "restore",
+          "-config",
+          this.configPath,
+          "-txid",
+          txid,
+          // MEASURED: without `-force`, restoring into a path that already
+          // exists exits 1 with "output path already exists and is not
+          // empty" (plan §2.4) — the caller always restores into a scratch
+          // path it just cleared, so overwriting it is always intended.
+          "-force",
+          "-o",
+          outputPath,
+          this.dbPath
+        ],
+        {
+          timeout: this.timeouts.restoreCandidateTimeoutSeconds * 1e3,
+          killSignal: "SIGKILL"
+        }
+      );
       return { ok: true, detail: stderr };
     } catch (error) {
-      return { ok: false, detail: stderrOf(error) };
+      return {
+        ok: false,
+        detail: commandDetail(
+          error,
+          `litestream restore of txid=${txid}`,
+          this.timeouts.restoreCandidateTimeoutSeconds
+        )
+      };
     }
   }
 };
@@ -61444,7 +61498,9 @@ async function probeReplica(store, prefix, log) {
     return { ok: true, objects };
   } catch (error) {
     const detail = describeError(error);
-    log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
+    log(
+      `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: could not list the replica prefix ${root12}: ${detail}`
+    );
     return { ok: false, detail };
   }
 }
@@ -61499,13 +61555,15 @@ async function pruneNewestL0(store, prefix, keys, log) {
 function quarantineRoot(prefix, stamp) {
   return `${prefix}/quarantine/opencode.db/${stamp}/`;
 }
-async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
+async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-"), budgetSeconds = DEFAULT_DISPOSAL_BUDGET_SECONDS, now = Date.now) {
   const destination = quarantineRoot(prefix, stamp);
   const root12 = replicaDbPrefix(prefix);
   const moved = [];
   const failed = [];
   let movedBytes = 0;
+  const deadline = now() + budgetSeconds * 1e3;
   for (const object of objects) {
+    if (now() >= deadline) break;
     const { key } = object;
     if (!isDeletableReplicaKey(prefix, key)) {
       log(`WARNING: refusing to quarantine ${key}: outside the replica prefix.`);
@@ -61533,15 +61591,15 @@ async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PUR
     moved.push(key);
     movedBytes += object.size;
   }
-  logQuarantined(destination, moved, failed, log);
+  logQuarantined(destination, moved, failed, objects.length - moved.length, log);
   return { destination, moved, failed, movedBytes };
 }
-function logQuarantined(destination, moved, failed, log) {
+function logQuarantined(destination, moved, failed, remaining, log) {
   log(
-    `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (failed.length > 0 ? " The active replica prefix is not clear." : "") + `See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
+    `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (remaining > 0 ? ` The active replica prefix still has ${remaining} object(s).` : "") + ` Progress: moved=${moved.length} remaining=${remaining}. See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
   );
 }
-async function separateCorruptReplica(store, prefix, log, stamp) {
+async function separateCorruptReplica(store, prefix, log, stamp, disposalBudgetSeconds = DEFAULT_DISPOSAL_BUDGET_SECONDS, now = Date.now) {
   const probe = await probeReplica(store, prefix, log);
   if (!probe.ok) return { kind: "unreachable", detail: probe.detail };
   const root12 = replicaDbPrefix(prefix);
@@ -61551,7 +61609,15 @@ async function separateCorruptReplica(store, prefix, log, stamp) {
     );
     return { kind: "alreadyEmpty" };
   }
-  const result = await quarantineReplica(store, prefix, probe.objects, log, stamp);
+  const result = await quarantineReplica(
+    store,
+    prefix,
+    probe.objects,
+    log,
+    stamp,
+    disposalBudgetSeconds,
+    now
+  );
   try {
     const remaining = await store.list(root12);
     if (remaining.length > 0) {
@@ -61882,7 +61948,14 @@ async function verifySessionDb(params) {
   );
   const decision = await walkBack(params);
   if (decision.kind !== "exhausted") return decision;
-  const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(params.store, config.prefix, log);
+  const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(
+    params.store,
+    config.prefix,
+    log,
+    void 0,
+    params.disposalBudgetSeconds,
+    params.now
+  );
   const localDb = separation.kind === "notConfigured" || separation.kind === "alreadyEmpty" || separation.kind === "quarantined" ? await discardCorruptDb(fileOps, dbPath, log) : { kind: "retained", reason: "separationUnproven" };
   return { ...decision, separation, localDb };
 }
@@ -62527,8 +62600,11 @@ function awsSelfStopAccessFor(config) {
 function nodeSqliteIntegrityFor() {
   return nodeSqliteIntegrity;
 }
-function litestreamCliFor(configPath, dbPath) {
-  return new LitestreamCli(configPath, dbPath);
+function litestreamCliFor(configPath, dbPath, config) {
+  return new LitestreamCli(configPath, dbPath, {
+    restorePointsTimeoutSeconds: config.restorePointsTimeoutSeconds,
+    restoreCandidateTimeoutSeconds: config.restoreCandidateTimeoutSeconds
+  });
 }
 function parseStoreName(value) {
   return value === "claude" || value === "opencode" ? value : null;
@@ -62688,7 +62764,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
   const { fileOps, log } = deps;
   const litestream = (deps.litestreamFor ?? litestreamCliFor)(
     litestreamConfigPath,
-    config.opencodeDbPath
+    config.opencodeDbPath,
+    config
   );
   const outcome = await verifySessionDb({
     config: { opencodeDbPath: config.opencodeDbPath, prefix: config.prefix },
@@ -62698,7 +62775,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
     litestream,
     log,
     maxPoints: config.walkbackMaxPoints,
-    budgetSeconds: config.walkbackBudgetSeconds
+    budgetSeconds: config.walkbackBudgetSeconds,
+    disposalBudgetSeconds: config.disposalBudgetSeconds
   });
   log(describeSessionDbVerification(outcome));
   if (outcome.kind === "exhausted" && sessionDbVerifyExitCode(outcome) === 34) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evident-ai/runner-synchroniser",
-  "version": "3.5.2-dev.06654ec",
+  "version": "3.5.2-dev.31c0c2e",
   "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
   "type": "module",
   "main": "./dist/cli.js",