@evident-ai/runner-synchroniser 3.4.0 → 3.4.1-dev.7802262

package/README.md

@@ -59,7 +59,7 @@ build time.
 | `sync-once <claude\|opencode>` | —                                       | `0` = ran; non-zero = tool broken                        |
 | `model-auth-ready`             | —                                       | `0` = ready, `10` = not ready; other = tool broken       |
 | `self-stop`                    | —                                       | `0` = stopped, `20` = keep the task; other = tool broken |
-| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
+| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
 | `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
 
 `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
@@ -77,6 +77,15 @@ saved query.
 
 Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
 
+#### Activity recovery report
+
+Reportable recovery outcomes append one versioned JSON line to the session-DB recovery
+report. `evident run` drains that file once after authentication and records the outcome in
+the runner activity log. Healthy boots do not write a report. An initial restore failure writes
+a warning `restore_retried` record before the retry's result is known. Writing is best-effort
+and never changes this command's exit code. The CLI reader owns this record contract: add a new
+`v` rather than repurposing a field.
+
 #### Positionals
 
 | Positional                       | Meaning                                                                                                                                |
@@ -102,6 +111,14 @@ Implementation-facing reference for the 7th command: it decides what a just-run
 - Any other `--` token is rejected.
 - Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
 
+#### Recovery-report flags
+
+`--recovery-occurred` marks a later successful restore after this boot already recovered a
+replica, so an empty result is reported as a fresh session database. `--fresh-db-fallback`
+marks a runtime that intentionally has no retry budget and will therefore boot fresh after this
+otherwise-transient failure. Each flag may appear once, does not change the command's exit code,
+and is rejected when repeated.
+
 #### Outcome → exit code
 
 | Outcome                                        | Code | When                                                                        |
@@ -316,6 +333,10 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
 
 `session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
 
+`session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
+history. It is drained once by `evident run` into runner activity and does not alter its exit
+code.
+
 `src/shell-contract.json` is the machine-checked source of truth for the command list, and
 `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
 `packages/runner-image/entrypoint.sh` and the MicroVM's
@@ -344,6 +365,7 @@ key, so they never appear in `env`'s output or `litestream.yml`.
 | `CREDS_SYNC_INTERVAL`                     | Seconds between sync ticks, reported by `env` for the caller's loop.     | Falls back to `60`; also on unparseable or non-positive values.                   |
 | `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`  | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values.                   |
 | `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop.           | Falls back to `180`; also on unparseable or non-positive values.                  |
+| `EVIDENT_SESSION_DB_RECOVERY_REPORT` | JSONL report drained once into runner activity after authentication. | `$HOME/.local/state/evident/session-db-recovery.jsonl`; blank values use the default. |
 | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`    | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
 | `CLUSTER` † / `SERVICE` †                 | ECS cluster/service `self-stop` scales to `desiredCount=0`.              | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
 | `EVIDENT_SELFSTOP_ROLE_ARN` †             | Role `self-stop` assumes for its ECS calls.                              | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |

package/dist/cli.js

@@ -51907,7 +51907,8 @@ function positiveIntOr(value, defaultValue) {
   return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue;
 }
 function nonEmpty(value) {
-  return value !== void 0 && value !== "" ? value : null;
+  const trimmed = value?.trim();
+  return trimmed ? trimmed : null;
 }
 function resolveConfig(env4) {
   const homeDir = nonEmpty(env4.HOME);
@@ -51930,6 +51931,7 @@ function resolveConfig(env4) {
       key: persistenceEnabled ? `${prefix}/opencode/auth.json` : null
     },
     opencodeDbPath: `${homeDir}/.local/share/opencode/opencode.db`,
+    sessionDbRecoveryReportPath: nonEmpty(env4.EVIDENT_SESSION_DB_RECOVERY_REPORT) ?? `${homeDir}/.local/state/evident/session-db-recovery.jsonl`,
     bucket: persistenceEnabled ? bucket : null,
     prefix: persistenceEnabled ? prefix : null,
     region: nonEmpty(env4.AWS_REGION),
@@ -52007,20 +52009,20 @@ function sessionDbExitCode(outcome) {
   }
 }
 function describeUnusableReplica(reason) {
-  const base = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
+  const base2 = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
   switch (reason) {
     case "strategy":
-      return `${base} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
+      return `${base2} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
     case "attemptsExhausted":
-      return `${base} (recovery did not make the replica usable after retrying; giving up.)`;
+      return `${base2} (recovery did not make the replica usable after retrying; giving up.)`;
     case "layoutMismatch":
-      return `${base} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
+      return `${base2} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
     case "noL0Present":
-      return `${base} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
+      return `${base2} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
     case "recoveryFailed":
-      return `${base} (the recovery delete itself failed; see the warning above for the S3 error.)`;
+      return `${base2} (the recovery delete itself failed; see the warning above for the S3 error.)`;
     case "targetHealthy":
-      return `${base} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
+      return `${base2} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
     default:
       return assertNeverUnusableReplicaReason(reason);
   }
@@ -52138,7 +52140,17 @@ function describeSessionDbVerification(outcome) {
 // src/file-ops.ts
 init_esm_shims();
 import { createHash as createHash8 } from "node:crypto";
-import { chmod, mkdir, open, readFile as readFile5, rename, rm, stat, writeFile as writeFile4 } from "node:fs/promises";
+import {
+  appendFile,
+  chmod,
+  mkdir,
+  open,
+  readFile as readFile5,
+  rename,
+  rm,
+  stat,
+  writeFile as writeFile4
+} from "node:fs/promises";
 var nodeFileOps = {
   async mkdirp(dir) {
     await mkdir(dir, { recursive: true });
@@ -52158,6 +52170,9 @@ var nodeFileOps = {
   async writeFile(path, data) {
     await writeFile4(path, data);
   },
+  async appendFile(path, data) {
+    await appendFile(path, data);
+  },
   async createExclusive(path) {
     const handle = await open(path, "wx", 384);
     await handle.close();
@@ -62324,7 +62339,7 @@ async function probeReplica(store, prefix, log) {
     const objectCount = objects.length;
     const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
     logReplicaSize(root12, objectCount, totalBytes, log);
-    return { ok: true, keys: objects.map((object) => object.key) };
+    return { ok: true, objects };
   } catch (error) {
     const detail = describeError(error);
     log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
@@ -62381,12 +62396,14 @@ async function pruneNewestL0(store, prefix, keys, log) {
 function quarantineRoot(prefix, stamp) {
   return `${prefix}/quarantine/opencode.db/${stamp}/`;
 }
-async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
+async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
   const destination = quarantineRoot(prefix, stamp);
   const root12 = replicaDbPrefix(prefix);
   const moved = [];
   const failed = [];
-  for (const key of keys) {
+  let movedBytes = 0;
+  for (const object of objects) {
+    const { key } = object;
     if (!isDeletableReplicaKey(prefix, key)) {
       log(`WARNING: refusing to quarantine ${key}: outside the replica prefix.`);
       continue;
@@ -62411,9 +62428,10 @@ async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__
       continue;
     }
     moved.push(key);
+    movedBytes += object.size;
   }
   logQuarantined(destination, moved, failed, log);
-  return { destination, moved, failed };
+  return { destination, moved, failed, movedBytes };
 }
 function logQuarantined(destination, moved, failed, log) {
   log(
@@ -62471,7 +62489,7 @@ async function decideSessionDbRestore(params) {
     return { kind: "retryTransient" };
   }
   if (attempt >= 3) {
-    return { kind: "unusableReplica", reason: "attemptsExhausted" };
+    return { kind: "unusableReplica", reason: "attemptsExhausted", recovery: null };
   }
   const probe = await probeReplica(store, prefix, log);
   if (!probe.ok) {
@@ -62481,7 +62499,12 @@ async function decideSessionDbRestore(params) {
       detail: `could not verify the session-DB replica is reachable: ${probe.detail}`
     };
   }
-  const keys = probe.keys;
+  const keys = probe.objects.map((object) => object.key);
+  const recovery = {
+    replicaObjects: probe.objects.length,
+    replicaBytes: probe.objects.reduce((total, object) => total + object.size, 0),
+    quarantine: null
+  };
   switch (strategy) {
     case "crash":
       return {
@@ -62490,30 +62513,43 @@ async function decideSessionDbRestore(params) {
         detail: "the replica is unusable and --on-unusable-replica=crash was selected; no S3 object was touched"
       };
     case "leave":
-      return { kind: "unusableReplica", reason: "strategy" };
+      return { kind: "unusableReplica", reason: "strategy", recovery };
     case "prune": {
       const result = await pruneNewestL0(store, prefix, keys, log);
       if (result.deleted === null) {
         if (shouldEscalateToQuarantine(result.reason, prefix, keys)) {
-          const quarantined = await quarantineReplica(store, prefix, keys, log);
+          const quarantined = await quarantineReplica(store, prefix, probe.objects, log);
           if (quarantined.moved.length === 0) {
-            return { kind: "unusableReplica", reason: "recoveryFailed" };
+            return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
           }
-          return { kind: "recovered", strategy: "quarantine", deleted: quarantined.moved };
+          return {
+            kind: "recovered",
+            strategy: "quarantine",
+            deleted: quarantined.moved,
+            recovery: {
+              ...recovery,
+              quarantine: {
+                destination: quarantined.destination,
+                movedObjects: quarantined.moved.length,
+                failedObjects: quarantined.failed.length,
+                movedBytes: quarantined.movedBytes
+              }
+            }
+          };
         }
         if (result.reason === "deleteFailed") {
-          return { kind: "unusableReplica", reason: "recoveryFailed" };
+          return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
         }
         if (result.reason === "unreadable") {
-          return { kind: "unusableReplica", reason: "targetHealthy" };
+          return { kind: "unusableReplica", reason: "targetHealthy", recovery };
         }
-        return { kind: "unusableReplica", reason: result.reason };
+        return { kind: "unusableReplica", reason: result.reason, recovery };
       }
-      return { kind: "recovered", strategy: "prune", deleted: [result.deleted] };
+      return { kind: "recovered", strategy: "prune", deleted: [result.deleted], recovery };
     }
     case "clear": {
       const result = await clearReplica(store, prefix, keys, log);
-      return { kind: "recovered", strategy: "clear", deleted: result.deleted };
+      return { kind: "recovered", strategy: "clear", deleted: result.deleted, recovery };
     }
     default:
       return assertNeverStrategy(strategy);
@@ -62533,6 +62569,114 @@ async function discardLocalDebris(fileOps, dbPath, log) {
   }
 }
 
+// src/session-db-recovery-report.ts
+init_esm_shims();
+import { dirname as dirname4 } from "node:path";
+function base(at2, stage) {
+  return {
+    v: 1,
+    event: "session_db_recovery",
+    at: at2,
+    stage,
+    litestream_exit_code: null,
+    attempt: null,
+    replica_objects: null,
+    replica_bytes: null,
+    quarantine_destination: null,
+    quarantined_objects: null,
+    quarantine_failed_objects: null,
+    quarantined_bytes: null,
+    verified_restore_point: null,
+    restore_points_tried: null
+  };
+}
+function recordRestoreOutcome(outcome, at2, exitCode, attempt, recoveryOccurred, freshDbFallback = false) {
+  const record = { ...base(at2, "restore"), litestream_exit_code: exitCode, attempt };
+  if (outcome.kind === "disabled" || outcome.kind === "restored") return null;
+  if (outcome.kind === "retryTransient")
+    return freshDbFallback ? {
+      ...record,
+      outcome: "fresh_session_db",
+      severity: "error",
+      reason: "retry_budget_exhausted"
+    } : {
+      ...record,
+      outcome: "restore_retried",
+      severity: "warning",
+      reason: "initial_restore_failed"
+    };
+  if (outcome.kind === "noReplica") {
+    return recoveryOccurred ? {
+      ...record,
+      outcome: "fresh_session_db",
+      severity: "error",
+      reason: "no_replica_after_recovery"
+    } : null;
+  }
+  if (outcome.kind === "fatal") return null;
+  if (outcome.kind === "unusableReplica")
+    return {
+      ...record,
+      outcome: "fresh_session_db",
+      severity: "error",
+      reason: outcome.reason,
+      replica_objects: outcome.recovery?.replicaObjects ?? null,
+      replica_bytes: outcome.recovery?.replicaBytes ?? null
+    };
+  return {
+    ...record,
+    outcome: "replica_recovered",
+    severity: outcome.strategy === "clear" ? "error" : "warning",
+    reason: outcome.strategy,
+    replica_objects: outcome.recovery.replicaObjects,
+    replica_bytes: outcome.recovery.replicaBytes,
+    quarantine_destination: outcome.recovery.quarantine?.destination ?? null,
+    quarantined_objects: outcome.recovery.quarantine?.movedObjects ?? null,
+    quarantine_failed_objects: outcome.recovery.quarantine?.failedObjects ?? null,
+    quarantined_bytes: outcome.recovery.quarantine?.movedBytes ?? null
+  };
+}
+function recordVerifyOutcome(outcome, at2) {
+  const record = base(at2, "verify");
+  if (outcome.kind === "skipped" || outcome.kind === "healthy") return null;
+  if (outcome.kind === "walkedBack")
+    return {
+      ...record,
+      outcome: "history_rolled_back",
+      severity: "warning",
+      reason: "walkback",
+      verified_restore_point: outcome.txid,
+      restore_points_tried: outcome.candidatesTried
+    };
+  return {
+    ...record,
+    outcome: "fresh_session_db",
+    severity: "error",
+    reason: outcome.reason,
+    restore_points_tried: outcome.candidatesTried
+  };
+}
+var MAX_REPORT_BYTES = 64 * 1024;
+async function appendSessionDbRecoveryRecord(record, path, fileOps, log) {
+  try {
+    const line = Buffer.from(`${JSON.stringify(record)}
+`);
+    const existing = await fileOps.stat(path);
+    if ((existing?.size ?? 0) + line.length > MAX_REPORT_BYTES) {
+      log(
+        `WARNING: session-DB recovery report at ${path} reached its ${MAX_REPORT_BYTES}-byte limit`
+      );
+      return;
+    }
+    await fileOps.mkdirp(dirname4(path));
+    await fileOps.appendFile(path, line);
+  } catch (error) {
+    log(
+      `WARNING: could not record the session-DB recovery outcome for the runner's activity log at ${path}: ${describeError(error)}`
+    );
+  }
+}
+
 // src/session-db-verify.ts
 init_esm_shims();
 
@@ -62857,7 +63001,8 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
   self-stop                    scale this agent's own ECS service to 0; exit 0 only
                                when desiredCount is confirmed 0, ${EXIT_KEEP_TASK} to keep the task
   session-db-classify <litestream-restore-exit-code> <attempt>
-                               [--on-unusable-replica=<prune|leave|clear|crash>]
+                                [--on-unusable-replica=<prune|leave|clear|crash>] [--recovery-occurred]
+                                [--fresh-db-fallback]
                                classify a just-run \`litestream restore\` of opencode.db;
                                exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
                                and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
@@ -62887,16 +63032,26 @@ function parseStoreName(value) {
   return value === "claude" || value === "opencode" ? value : null;
 }
 var ON_UNUSABLE_REPLICA_FLAG = "--on-unusable-replica=";
+var RECOVERY_OCCURRED_FLAG = "--recovery-occurred";
+var FRESH_DB_FALLBACK_FLAG = "--fresh-db-fallback";
 function parseNonNegativeInt(value) {
   return /^\d+$/.test(value) ? Number.parseInt(value, 10) : null;
 }
 function parseSessionDbClassifyArgs(args) {
   let strategyRaw;
+  let recoveryOccurred = false;
+  let freshDbFallback = false;
   const positional = [];
   for (const arg of args) {
     if (arg.startsWith(ON_UNUSABLE_REPLICA_FLAG)) {
       if (strategyRaw !== void 0) return null;
       strategyRaw = arg.slice(ON_UNUSABLE_REPLICA_FLAG.length);
+    } else if (arg === RECOVERY_OCCURRED_FLAG) {
+      if (recoveryOccurred) return null;
+      recoveryOccurred = true;
+    } else if (arg === FRESH_DB_FALLBACK_FLAG) {
+      if (freshDbFallback) return null;
+      freshDbFallback = true;
     } else if (arg.startsWith("--")) {
       return null;
     } else {
@@ -62908,7 +63063,7 @@ function parseSessionDbClassifyArgs(args) {
   const attempt = parseNonNegativeInt(positional[1]);
   const strategy = parseUnusableReplicaStrategy(strategyRaw);
   if (exitCode === null || attempt === null || attempt < 1 || strategy === null) return null;
-  return { exitCode, attempt, strategy };
+  return { exitCode, attempt, strategy, recoveryOccurred, freshDbFallback };
 }
 function renderEnv(config) {
   return [
@@ -62996,6 +63151,16 @@ async function commandSessionDbClassify(args, config, deps) {
     log
   });
   log(describeSessionDbClassification(outcome, { bucket: config.bucket, region: config.region }));
+  const record = recordRestoreOutcome(
+    outcome,
+    (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString(),
+    args.exitCode,
+    args.attempt,
+    args.recoveryOccurred,
+    args.freshDbFallback
+  );
+  if (record !== null)
+    await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
   return sessionDbExitCode(outcome);
 }
 async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
@@ -63014,6 +63179,9 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
     budgetSeconds: config.walkbackBudgetSeconds
   });
   log(describeSessionDbVerification(outcome));
+  const record = recordVerifyOutcome(outcome, (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString());
+  if (record !== null)
+    await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
   return sessionDbVerifyExitCode(outcome);
 }
 async function main(argv, deps) {
@@ -63055,7 +63223,7 @@ ${USAGE}`);
     const args = parseSessionDbClassifyArgs(rest);
     if (args === null) {
       log(
-        `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>.
+        `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>, --recovery-occurred, and --fresh-db-fallback.
 ${USAGE}`
       );
       return EXIT_USAGE;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evident-ai/runner-synchroniser",
-  "version": "3.4.0",
+  "version": "3.4.1-dev.7802262",
   "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
   "type": "module",
   "main": "./dist/cli.js",