@evident-ai/runner-synchroniser 3.4.0 → 3.4.1-dev.3adba63

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +44 -17
  2. package/dist/cli.js +345 -51
  3. package/package.json +3 -3
package/README.md CHANGED
@@ -59,8 +59,8 @@ build time.
59
59
  | `sync-once <claude\|opencode>` | — | `0` = ran; non-zero = tool broken |
60
60
  | `model-auth-ready` | — | `0` = ready, `10` = not ready; other = tool broken |
61
61
  | `self-stop` | — | `0` = stopped, `20` = keep the task; other = tool broken |
62
- | `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
63
- | `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
62
+ | `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, replicates into the existing prefix), `30` = fatal; other = usage/tool broken |
63
+ | `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = exhausted after verified separation and local disposal, `34` = separation or disposal could not be proven; other = usage/tool broken |
64
64
 
65
65
  `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
66
66
  (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -75,7 +75,16 @@ saved query.
75
75
 
76
76
  ### `session-db-classify`
77
77
 
78
- Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
78
+ Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `runner/docker-images/fargate/README.md`'s [Strategy/What/Cost table](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
79
+
80
+ #### Activity recovery report
81
+
82
+ Reportable recovery outcomes append one versioned JSON line to the session-DB recovery
83
+ report. `evident run` drains that file once after authentication and records the outcome in
84
+ the runner activity log. Healthy boots do not write a report. An initial restore failure writes
85
+ a warning `restore_retried` record before the retry's result is known. Writing is best-effort
86
+ and never changes this command's exit code. The CLI reader owns this record contract: add a new
87
+ `v` rather than repurposing a field.
79
88
 
80
89
  #### Positionals
81
90
 
@@ -102,6 +111,14 @@ Implementation-facing reference for the 7th command: it decides what a just-run
102
111
  - Any other `--` token is rejected.
103
112
  - Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
104
113
 
114
+ #### Recovery-report flags
115
+
116
+ `--recovery-occurred` marks a later successful restore after this boot already recovered a
117
+ replica, so an empty result is reported as a fresh session database. `--fresh-db-fallback`
118
+ marks a runtime that intentionally has no retry budget and will therefore boot fresh after this
119
+ otherwise-transient failure. Each flag may appear once, does not change the command's exit code,
120
+ and is rejected when repeated.
121
+
105
122
  #### Outcome → exit code
106
123
 
107
124
  | Outcome | Code | When |
@@ -196,11 +213,11 @@ what it is.
196
213
  #### The four `--on-unusable-replica` strategies
197
214
 
198
215
  - **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
199
- - **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". Since #1106 it is **not** also ephemeral: the boot still starts a fresh backup chain, because the classifier guarantees the local DB is fresh rather than half-restored. Upside: replica left byte-intact for forensics, no crash loop.
216
+ - **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". The fresh local DB replicates into the existing prefix; no new chain is started. Upside: the runner still comes online without a crash loop.
200
217
  - **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
201
218
  - **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
202
219
 
203
- #### The measured real-world key layout (litestream 0.5.13)
220
+ #### The measured real-world key layout (litestream 0.5.13 historical sample)
204
221
 
205
222
  `parseLtxKey` (`src/replica-keys.ts`) expects
206
223
  `<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
@@ -218,7 +235,7 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
218
235
  (`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
219
236
  and it can never repair corruption at a higher compaction level.
220
237
 
221
- `EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
238
+ `EVIDENT_ON_UNUSABLE_REPLICA` is a `runner/docker-images/fargate` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [the Fargate image README](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
222
239
 
223
240
  See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
224
241
  the behavioural anchor (31 comes online, 30 does not).
@@ -245,9 +262,12 @@ nothing had ever run a real `PRAGMA integrity_check`).
245
262
  each into a scratch path and integrity-checking it there. The first one that passes
246
263
  is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
247
264
  3. If nothing retained passes (or there's nothing to try, or the search runs out of
248
- time/points), the corrupt local DB is discarded and the boot proceeds with a
249
- genuinely fresh one exactly `session-db-classify`'s `31` guarantee, so
250
- `litestream replicate` starting against it afterwards is just as safe.
265
+ time/points), the active replica prefix is separated first: its history is moved aside,
266
+ then the prefix is re-listed as empty before the corrupt local DB and sidecars are
267
+ discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
268
+ OpenCode or Litestream starts; the backup history remains readable at its quarantine
269
+ destination or original key, and surviving local files remain in place without a process
270
+ opening or writing them.
251
271
 
252
272
  A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
253
273
  prefix. litestream re-bases to the replica's high-water mark and continues the txid
@@ -262,10 +282,8 @@ against S3 at the ~1 GB scale a production replica can reach.)
262
282
  | `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
263
283
  | `healthy` | `0` | the local DB passed its integrity check as-is |
264
284
  | `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
265
- | `exhausted` (`noCandidates`) | `33` | the local DB failed and litestream retains no older restore point |
266
- | `exhausted` (`allFailed`) | `33` | every retained restore point was tried and every one failed |
267
- | `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
268
- | `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
285
+ | `exhausted` (`noCandidates`, `allFailed`, `budgetExhausted`, or `enumerationFailed`) | `33` | the active prefix was proven separated (or persistence is disabled) **and** the corrupt local DB and sidecars were proven discarded |
286
+ | `exhausted` | `34` | either the replica separation or local discard could not be proven; the boot stops before OpenCode and Litestream start |
269
287
 
270
288
  `healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
271
289
  afterwards — start `litestream replicate` — is identical either way, so a further split
@@ -284,7 +302,10 @@ before looking at everything" would report the wrong conclusion to the operator.
284
302
  | `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
285
303
  | `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
286
304
  | `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
287
- | `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed (any `exhausted` reason); the boot proceeds with a fresh DB | `candidatesTried` (in prose, not `k=v`) |
305
+ | `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed; a fresh boot follows only when both separation and local disposal were proven | `candidatesTried` (in prose, not `k=v`) |
306
+ | `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
307
+ | `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
308
+ | `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
288
309
 
289
310
  The first four fire only while the walkback loop runs; the last three are always the
290
311
  one message this command logs as its own final word on the outcome — `entrypoint.sh`
@@ -316,10 +337,15 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
316
337
 
317
338
  `session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
318
339
 
340
+ `session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
341
+ history. A refused boot records `session_db_boot_refused`, but `evident run` has not started to
342
+ drain it, so operators see that refusal in boot logs and the restart loop. The report does not
343
+ alter the command's exit code.
344
+
319
345
  `src/shell-contract.json` is the machine-checked source of truth for the command list, and
320
346
  `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
321
- `packages/runner-image/entrypoint.sh` and the MicroVM's
322
- `packages/runner-cdk/microvm-image/hooks` (#608), discovered by grep so a third one
347
+ `runner/docker-images/fargate/entrypoint.sh` and the MicroVM's
348
+ `runner/docker-images/microvm/hooks` (#608), discovered by grep so a third one
323
349
  cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
324
350
  `entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
325
351
  a broken tool; and take each answer code the commands it calls can return **silently and by
@@ -344,11 +370,12 @@ key, so they never appear in `env`'s output or `litestream.yml`.
344
370
  | `CREDS_SYNC_INTERVAL` | Seconds between sync ticks, reported by `env` for the caller's loop. | Falls back to `60`; also on unparseable or non-positive values. |
345
371
  | `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS` | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values. |
346
372
  | `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop. | Falls back to `180`; also on unparseable or non-positive values. |
373
+ | `EVIDENT_SESSION_DB_RECOVERY_REPORT` | JSONL report drained once into runner activity after authentication. | `$HOME/.local/state/evident/session-db-recovery.jsonl`; blank values use the default. |
347
374
  | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
348
375
  | `CLUSTER` † / `SERVICE` † | ECS cluster/service `self-stop` scales to `desiredCount=0`. | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
349
376
  | `EVIDENT_SELFSTOP_ROLE_ARN` † | Role `self-stop` assumes for its ECS calls. | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
350
377
 
351
- `packages/runner-image/README.md` documents these same three from the deployment side —
378
+ `runner/docker-images/fargate/README.md` documents these same three from the deployment side —
352
379
  keep them in sync.
353
380
 
354
381
  Requiring **both** `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` (never just one) means
package/dist/cli.js CHANGED
@@ -51907,7 +51907,8 @@ function positiveIntOr(value, defaultValue) {
51907
51907
  return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue;
51908
51908
  }
51909
51909
  function nonEmpty(value) {
51910
- return value !== void 0 && value !== "" ? value : null;
51910
+ const trimmed = value?.trim();
51911
+ return trimmed ? trimmed : null;
51911
51912
  }
51912
51913
  function resolveConfig(env4) {
51913
51914
  const homeDir = nonEmpty(env4.HOME);
@@ -51930,6 +51931,7 @@ function resolveConfig(env4) {
51930
51931
  key: persistenceEnabled ? `${prefix}/opencode/auth.json` : null
51931
51932
  },
51932
51933
  opencodeDbPath: `${homeDir}/.local/share/opencode/opencode.db`,
51934
+ sessionDbRecoveryReportPath: nonEmpty(env4.EVIDENT_SESSION_DB_RECOVERY_REPORT) ?? `${homeDir}/.local/state/evident/session-db-recovery.jsonl`,
51933
51935
  bucket: persistenceEnabled ? bucket : null,
51934
51936
  prefix: persistenceEnabled ? prefix : null,
51935
51937
  region: nonEmpty(env4.AWS_REGION),
@@ -51998,7 +52000,7 @@ function sessionDbExitCode(outcome) {
51998
52000
  // re-run `litestream restore` and ask again
51999
52001
  case "unusableReplica":
52000
52002
  return 31;
52001
- // booted with a fresh DB; the shell still starts a fresh backup chain (#1106)
52003
+ // booted with a fresh DB; the shell still replicates into the existing prefix
52002
52004
  case "fatal":
52003
52005
  return 30;
52004
52006
  // shared by a genuine misconfig and a deliberate `crash` choice
@@ -52007,20 +52009,20 @@ function sessionDbExitCode(outcome) {
52007
52009
  }
52008
52010
  }
52009
52011
  function describeUnusableReplica(reason) {
52010
- const base = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
52012
+ const base2 = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
52011
52013
  switch (reason) {
52012
52014
  case "strategy":
52013
- return `${base} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
52015
+ return `${base2} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
52014
52016
  case "attemptsExhausted":
52015
- return `${base} (recovery did not make the replica usable after retrying; giving up.)`;
52017
+ return `${base2} (recovery did not make the replica usable after retrying; giving up.)`;
52016
52018
  case "layoutMismatch":
52017
- return `${base} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
52019
+ return `${base2} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
52018
52020
  case "noL0Present":
52019
- return `${base} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
52021
+ return `${base2} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
52020
52022
  case "recoveryFailed":
52021
- return `${base} (the recovery delete itself failed; see the warning above for the S3 error.)`;
52023
+ return `${base2} (the recovery delete itself failed; see the warning above for the S3 error.)`;
52022
52024
  case "targetHealthy":
52023
- return `${base} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
52025
+ return `${base2} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
52024
52026
  default:
52025
52027
  return assertNeverUnusableReplicaReason(reason);
52026
52028
  }
@@ -52061,7 +52063,7 @@ function describeSessionDbClassification(outcome, config) {
52061
52063
  case "recovered": {
52062
52064
  const objects = outcome.deleted.length === 0 ? "no objects (see the warnings above for what failed)" : `${outcome.deleted.length} object${outcome.deleted.length === 1 ? "" : "s"} (${outcome.deleted.join(", ")})`;
52063
52065
  if (outcome.strategy === "quarantine") {
52064
- return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside, NOT deleted \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
52066
+ return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside and remains readable at the quarantine destination \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
52065
52067
  }
52066
52068
  return `Attempted opencode.db replica recovery via --on-unusable-replica=${outcome.strategy}, deleting ${objects}; retrying the restore.` + (outcome.strategy === "clear" ? " Prior session history is lost." : "");
52067
52069
  }
@@ -52082,6 +52084,12 @@ function assertNeverSessionDbVerifySkippedLocalDb(value) {
52082
52084
  function assertNeverSessionDbVerifyExhaustedReason(value) {
52083
52085
  throw new Error(`Unhandled session-DB verify exhausted reason: ${JSON.stringify(value)}`);
52084
52086
  }
52087
+ function assertNeverSessionDbVerifySeparation(value) {
52088
+ throw new Error(`Unhandled session-DB verify separation: ${JSON.stringify(value)}`);
52089
+ }
52090
+ function assertNeverSessionDbVerifyLocalDb(value) {
52091
+ throw new Error(`Unhandled session-DB verify local-DB disposal: ${JSON.stringify(value)}`);
52092
+ }
52085
52093
  function sessionDbVerifyExitCode(outcome) {
52086
52094
  switch (outcome.kind) {
52087
52095
  case "skipped":
@@ -52089,8 +52097,26 @@ function sessionDbVerifyExitCode(outcome) {
52089
52097
  case "walkedBack":
52090
52098
  return 0;
52091
52099
  case "exhausted":
52092
- return 33;
52093
- // booted with a fresh DB; the shell still starts a fresh backup chain
52100
+ switch (outcome.separation.kind) {
52101
+ case "notConfigured":
52102
+ case "alreadyEmpty":
52103
+ case "quarantined":
52104
+ break;
52105
+ case "incomplete":
52106
+ case "unreachable":
52107
+ return 34;
52108
+ default:
52109
+ return assertNeverSessionDbVerifySeparation(outcome.separation);
52110
+ }
52111
+ switch (outcome.localDb.kind) {
52112
+ case "discarded":
52113
+ return 33;
52114
+ case "retained":
52115
+ case "discardFailed":
52116
+ return 34;
52117
+ default:
52118
+ return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
52119
+ }
52094
52120
  default:
52095
52121
  return assertNeverSessionDbVerify(outcome);
52096
52122
  }
@@ -52107,18 +52133,56 @@ function describeSessionDbVerifySkipped(localDb) {
52107
52133
  }
52108
52134
  function describeSessionDbVerifyExhausted(outcome) {
52109
52135
  const tried = `tried ${outcome.candidatesTried} restore point${outcome.candidatesTried === 1 ? "" : "s"}`;
52136
+ let reason;
52110
52137
  switch (outcome.reason) {
52111
52138
  case "enumerationFailed":
52112
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried}); booting with a FRESH opencode.db.`;
52139
+ reason = `opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried})`;
52140
+ break;
52113
52141
  case "noCandidates":
52114
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and no older restore point is retained (${tried}); booting with a FRESH opencode.db.`;
52142
+ reason = `opencode.db failed its integrity check and no older restore point is retained (${tried})`;
52143
+ break;
52115
52144
  case "allFailed":
52116
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and every retained restore point also failed (${tried}); booting with a FRESH opencode.db.`;
52145
+ reason = `opencode.db failed its integrity check and every retained restore point also failed (${tried})`;
52146
+ break;
52117
52147
  case "budgetExhausted":
52118
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try; booting with a FRESH opencode.db.`;
52148
+ reason = `opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try`;
52149
+ break;
52119
52150
  default:
52120
52151
  return assertNeverSessionDbVerifyExhaustedReason(outcome.reason);
52121
52152
  }
52153
+ let separation;
52154
+ switch (outcome.separation.kind) {
52155
+ case "quarantined":
52156
+ separation = ` The active replica prefix was verified empty; moved history remains readable at ${outcome.separation.destination}.`;
52157
+ break;
52158
+ case "alreadyEmpty":
52159
+ separation = " The active replica prefix was already empty.";
52160
+ break;
52161
+ case "notConfigured":
52162
+ separation = " Persistence is disabled, so there is no replica to separate.";
52163
+ break;
52164
+ case "incomplete":
52165
+ case "unreachable":
52166
+ separation = ` Boot is stopping because replica separation could not be proven (${outcome.separation.detail}); backup history remains readable at its original key or quarantine destination.`;
52167
+ break;
52168
+ default:
52169
+ return assertNeverSessionDbVerifySeparation(outcome.separation);
52170
+ }
52171
+ let localDb;
52172
+ switch (outcome.localDb.kind) {
52173
+ case "discarded":
52174
+ localDb = " booting with a FRESH opencode.db.";
52175
+ break;
52176
+ case "retained":
52177
+ localDb = " The corrupt local database and sidecars were left untouched for investigation.";
52178
+ break;
52179
+ case "discardFailed":
52180
+ localDb = ` Boot is stopping because local disposal could not be proven; removed=${outcome.localDb.removed.join(", ") || "(none)"} survived=${outcome.localDb.survived.join(", ") || "(none)"}. See SESSION-DB-LOCAL-DISCARD-FAILED.`;
52181
+ break;
52182
+ default:
52183
+ return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
52184
+ }
52185
+ return `SESSION-DB-INTEGRITY-EXHAUSTED: ${reason}.${separation}${localDb}`;
52122
52186
  }
52123
52187
  function describeSessionDbVerification(outcome) {
52124
52188
  switch (outcome.kind) {
@@ -52138,7 +52202,17 @@ function describeSessionDbVerification(outcome) {
52138
52202
  // src/file-ops.ts
52139
52203
  init_esm_shims();
52140
52204
  import { createHash as createHash8 } from "node:crypto";
52141
- import { chmod, mkdir, open, readFile as readFile5, rename, rm, stat, writeFile as writeFile4 } from "node:fs/promises";
52205
+ import {
52206
+ appendFile,
52207
+ chmod,
52208
+ mkdir,
52209
+ open,
52210
+ readFile as readFile5,
52211
+ rename,
52212
+ rm,
52213
+ stat,
52214
+ writeFile as writeFile4
52215
+ } from "node:fs/promises";
52142
52216
  var nodeFileOps = {
52143
52217
  async mkdirp(dir) {
52144
52218
  await mkdir(dir, { recursive: true });
@@ -52150,14 +52224,18 @@ var nodeFileOps = {
52150
52224
  try {
52151
52225
  const stats = await stat(path);
52152
52226
  return { size: stats.size };
52153
- } catch {
52154
- return null;
52227
+ } catch (error) {
52228
+ if (error instanceof Error && "code" in error && error.code === "ENOENT") return null;
52229
+ throw error;
52155
52230
  }
52156
52231
  },
52157
52232
  readFile: readFile5,
52158
52233
  async writeFile(path, data) {
52159
52234
  await writeFile4(path, data);
52160
52235
  },
52236
+ async appendFile(path, data) {
52237
+ await appendFile(path, data);
52238
+ },
52161
52239
  async createExclusive(path) {
52162
52240
  const handle = await open(path, "wx", 384);
52163
52241
  await handle.close();
@@ -52252,7 +52330,7 @@ function renderLitestreamConfig(config) {
52252
52330
  # (no static keys here).
52253
52331
  #
52254
52332
  # INVARIANT (see README "Single-writer invariant"): exactly ONE writer per S3
52255
- # prefix \u2014 v0.5.13 does NOT enforce a server-side lease, so never run two
52333
+ # prefix \u2014 we do not enable Litestream's server-side lease, so never run two
52256
52334
  # replicators on one prefix. Only opencode.db is replicated.
52257
52335
 
52258
52336
  dbs:
@@ -62324,7 +62402,7 @@ async function probeReplica(store, prefix, log) {
62324
62402
  const objectCount = objects.length;
62325
62403
  const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
62326
62404
  logReplicaSize(root12, objectCount, totalBytes, log);
62327
- return { ok: true, keys: objects.map((object) => object.key) };
62405
+ return { ok: true, objects };
62328
62406
  } catch (error) {
62329
62407
  const detail = describeError(error);
62330
62408
  log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
@@ -62381,12 +62459,14 @@ async function pruneNewestL0(store, prefix, keys, log) {
62381
62459
  function quarantineRoot(prefix, stamp) {
62382
62460
  return `${prefix}/quarantine/opencode.db/${stamp}/`;
62383
62461
  }
62384
- async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
62462
+ async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
62385
62463
  const destination = quarantineRoot(prefix, stamp);
62386
62464
  const root12 = replicaDbPrefix(prefix);
62387
62465
  const moved = [];
62388
62466
  const failed = [];
62389
- for (const key of keys) {
62467
+ let movedBytes = 0;
62468
+ for (const object of objects) {
62469
+ const { key } = object;
62390
62470
  if (!isDeletableReplicaKey(prefix, key)) {
62391
62471
  log(`WARNING: refusing to quarantine ${key}: outside the replica prefix.`);
62392
62472
  continue;
@@ -62411,14 +62491,47 @@ async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__
62411
62491
  continue;
62412
62492
  }
62413
62493
  moved.push(key);
62494
+ movedBytes += object.size;
62414
62495
  }
62415
62496
  logQuarantined(destination, moved, failed, log);
62416
- return { destination, moved, failed };
62497
+ return { destination, moved, failed, movedBytes };
62417
62498
  }
62418
62499
  function logQuarantined(destination, moved, failed, log) {
62419
62500
  log(
62420
- `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved) so this boot can start a fresh backup chain. The bytes are NOT deleted \u2014 copy them back from there to investigate. See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
62501
+ `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (failed.length > 0 ? " The active replica prefix is not clear." : "") + `See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
62502
+ );
62503
+ }
62504
+ async function separateCorruptReplica(store, prefix, log, stamp) {
62505
+ const probe = await probeReplica(store, prefix, log);
62506
+ if (!probe.ok) return { kind: "unreachable", detail: probe.detail };
62507
+ const root12 = replicaDbPrefix(prefix);
62508
+ if (probe.objects.length === 0) {
62509
+ log(
62510
+ `INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was already empty; this boot starts a fresh backup chain.`
62511
+ );
62512
+ return { kind: "alreadyEmpty" };
62513
+ }
62514
+ const result = await quarantineReplica(store, prefix, probe.objects, log, stamp);
62515
+ try {
62516
+ const remaining = await store.list(root12);
62517
+ if (remaining.length > 0) {
62518
+ const detail = `${remaining.length} object(s) remain`;
62519
+ log(
62520
+ `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: ${root12} still has ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
62521
+ );
62522
+ return { kind: "incomplete", detail };
62523
+ }
62524
+ } catch (error) {
62525
+ const detail = describeError(error);
62526
+ log(
62527
+ `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: could not re-list ${root12}: ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
62528
+ );
62529
+ return { kind: "incomplete", detail };
62530
+ }
62531
+ log(
62532
+ `INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was verified empty by re-listing after moving ${result.moved.length} object(s) to ${result.destination}; this boot starts a fresh backup chain.`
62421
62533
  );
62534
+ return { kind: "quarantined", destination: result.destination, moved: result.moved };
62422
62535
  }
62423
62536
  async function clearReplica(store, prefix, keys, log) {
62424
62537
  const deleted = [];
@@ -62471,7 +62584,7 @@ async function decideSessionDbRestore(params) {
62471
62584
  return { kind: "retryTransient" };
62472
62585
  }
62473
62586
  if (attempt >= 3) {
62474
- return { kind: "unusableReplica", reason: "attemptsExhausted" };
62587
+ return { kind: "unusableReplica", reason: "attemptsExhausted", recovery: null };
62475
62588
  }
62476
62589
  const probe = await probeReplica(store, prefix, log);
62477
62590
  if (!probe.ok) {
@@ -62481,7 +62594,12 @@ async function decideSessionDbRestore(params) {
62481
62594
  detail: `could not verify the session-DB replica is reachable: ${probe.detail}`
62482
62595
  };
62483
62596
  }
62484
- const keys = probe.keys;
62597
+ const keys = probe.objects.map((object) => object.key);
62598
+ const recovery = {
62599
+ replicaObjects: probe.objects.length,
62600
+ replicaBytes: probe.objects.reduce((total, object) => total + object.size, 0),
62601
+ quarantine: null
62602
+ };
62485
62603
  switch (strategy) {
62486
62604
  case "crash":
62487
62605
  return {
@@ -62490,30 +62608,43 @@ async function decideSessionDbRestore(params) {
62490
62608
  detail: "the replica is unusable and --on-unusable-replica=crash was selected; no S3 object was touched"
62491
62609
  };
62492
62610
  case "leave":
62493
- return { kind: "unusableReplica", reason: "strategy" };
62611
+ return { kind: "unusableReplica", reason: "strategy", recovery };
62494
62612
  case "prune": {
62495
62613
  const result = await pruneNewestL0(store, prefix, keys, log);
62496
62614
  if (result.deleted === null) {
62497
62615
  if (shouldEscalateToQuarantine(result.reason, prefix, keys)) {
62498
- const quarantined = await quarantineReplica(store, prefix, keys, log);
62616
+ const quarantined = await quarantineReplica(store, prefix, probe.objects, log);
62499
62617
  if (quarantined.moved.length === 0) {
62500
- return { kind: "unusableReplica", reason: "recoveryFailed" };
62618
+ return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
62501
62619
  }
62502
- return { kind: "recovered", strategy: "quarantine", deleted: quarantined.moved };
62620
+ return {
62621
+ kind: "recovered",
62622
+ strategy: "quarantine",
62623
+ deleted: quarantined.moved,
62624
+ recovery: {
62625
+ ...recovery,
62626
+ quarantine: {
62627
+ destination: quarantined.destination,
62628
+ movedObjects: quarantined.moved.length,
62629
+ failedObjects: quarantined.failed.length,
62630
+ movedBytes: quarantined.movedBytes
62631
+ }
62632
+ }
62633
+ };
62503
62634
  }
62504
62635
  if (result.reason === "deleteFailed") {
62505
- return { kind: "unusableReplica", reason: "recoveryFailed" };
62636
+ return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
62506
62637
  }
62507
62638
  if (result.reason === "unreadable") {
62508
- return { kind: "unusableReplica", reason: "targetHealthy" };
62639
+ return { kind: "unusableReplica", reason: "targetHealthy", recovery };
62509
62640
  }
62510
- return { kind: "unusableReplica", reason: result.reason };
62641
+ return { kind: "unusableReplica", reason: result.reason, recovery };
62511
62642
  }
62512
- return { kind: "recovered", strategy: "prune", deleted: [result.deleted] };
62643
+ return { kind: "recovered", strategy: "prune", deleted: [result.deleted], recovery };
62513
62644
  }
62514
62645
  case "clear": {
62515
62646
  const result = await clearReplica(store, prefix, keys, log);
62516
- return { kind: "recovered", strategy: "clear", deleted: result.deleted };
62647
+ return { kind: "recovered", strategy: "clear", deleted: result.deleted, recovery };
62517
62648
  }
62518
62649
  default:
62519
62650
  return assertNeverStrategy(strategy);
@@ -62533,6 +62664,123 @@ async function discardLocalDebris(fileOps, dbPath, log) {
62533
62664
  }
62534
62665
  }
62535
62666
 
62667
+ // src/session-db-recovery-report.ts
62668
+ init_esm_shims();
62669
+ import { dirname as dirname4 } from "node:path";
62670
+ function base(at2, stage) {
62671
+ return {
62672
+ v: 1,
62673
+ event: "session_db_recovery",
62674
+ at: at2,
62675
+ stage,
62676
+ litestream_exit_code: null,
62677
+ attempt: null,
62678
+ replica_objects: null,
62679
+ replica_bytes: null,
62680
+ quarantine_destination: null,
62681
+ quarantined_objects: null,
62682
+ quarantine_failed_objects: null,
62683
+ quarantined_bytes: null,
62684
+ verified_restore_point: null,
62685
+ restore_points_tried: null
62686
+ };
62687
+ }
62688
+ function recordRestoreOutcome(outcome, at2, exitCode, attempt, recoveryOccurred, freshDbFallback = false) {
62689
+ const record = { ...base(at2, "restore"), litestream_exit_code: exitCode, attempt };
62690
+ if (outcome.kind === "disabled" || outcome.kind === "restored") return null;
62691
+ if (outcome.kind === "retryTransient")
62692
+ return freshDbFallback ? {
62693
+ ...record,
62694
+ outcome: "fresh_session_db",
62695
+ severity: "error",
62696
+ reason: "retry_budget_exhausted"
62697
+ } : {
62698
+ ...record,
62699
+ outcome: "restore_retried",
62700
+ severity: "warning",
62701
+ reason: "initial_restore_failed"
62702
+ };
62703
+ if (outcome.kind === "noReplica") {
62704
+ return recoveryOccurred ? {
62705
+ ...record,
62706
+ outcome: "fresh_session_db",
62707
+ severity: "error",
62708
+ reason: "no_replica_after_recovery"
62709
+ } : null;
62710
+ }
62711
+ if (outcome.kind === "fatal") return null;
62712
+ if (outcome.kind === "unusableReplica")
62713
+ return {
62714
+ ...record,
62715
+ outcome: "fresh_session_db",
62716
+ severity: "error",
62717
+ reason: outcome.reason,
62718
+ replica_objects: outcome.recovery?.replicaObjects ?? null,
62719
+ replica_bytes: outcome.recovery?.replicaBytes ?? null
62720
+ };
62721
+ return {
62722
+ ...record,
62723
+ outcome: "replica_recovered",
62724
+ severity: outcome.strategy === "clear" ? "error" : "warning",
62725
+ reason: outcome.strategy,
62726
+ replica_objects: outcome.recovery.replicaObjects,
62727
+ replica_bytes: outcome.recovery.replicaBytes,
62728
+ quarantine_destination: outcome.recovery.quarantine?.destination ?? null,
62729
+ quarantined_objects: outcome.recovery.quarantine?.movedObjects ?? null,
62730
+ quarantine_failed_objects: outcome.recovery.quarantine?.failedObjects ?? null,
62731
+ quarantined_bytes: outcome.recovery.quarantine?.movedBytes ?? null
62732
+ };
62733
+ }
62734
+ function recordVerifyOutcome(outcome, at2) {
62735
+ const record = base(at2, "verify");
62736
+ if (outcome.kind === "skipped" || outcome.kind === "healthy") return null;
62737
+ if (outcome.kind === "walkedBack")
62738
+ return {
62739
+ ...record,
62740
+ outcome: "history_rolled_back",
62741
+ severity: "warning",
62742
+ reason: "walkback",
62743
+ verified_restore_point: outcome.txid,
62744
+ restore_points_tried: outcome.candidatesTried
62745
+ };
62746
+ if (sessionDbVerifyExitCode(outcome) === 34)
62747
+ return {
62748
+ ...record,
62749
+ outcome: "session_db_boot_refused",
62750
+ severity: "error",
62751
+ reason: outcome.localDb.kind === "discardFailed" ? "local_discard_failed" : "replica_separation_unproven",
62752
+ restore_points_tried: outcome.candidatesTried,
62753
+ quarantine_destination: outcome.separation.kind === "quarantined" ? outcome.separation.destination : null
62754
+ };
62755
+ return {
62756
+ ...record,
62757
+ outcome: "fresh_session_db",
62758
+ severity: "error",
62759
+ reason: outcome.reason,
62760
+ restore_points_tried: outcome.candidatesTried
62761
+ };
62762
+ }
62763
+ var MAX_REPORT_BYTES = 64 * 1024;
62764
+ async function appendSessionDbRecoveryRecord(record, path, fileOps, log) {
62765
+ try {
62766
+ const line = Buffer.from(`${JSON.stringify(record)}
62767
+ `);
62768
+ const existing = await fileOps.stat(path);
62769
+ if ((existing?.size ?? 0) + line.length > MAX_REPORT_BYTES) {
62770
+ log(
62771
+ `WARNING: session-DB recovery report at ${path} reached its ${MAX_REPORT_BYTES}-byte limit`
62772
+ );
62773
+ return;
62774
+ }
62775
+ await fileOps.mkdirp(dirname4(path));
62776
+ await fileOps.appendFile(path, line);
62777
+ } catch (error) {
62778
+ log(
62779
+ `WARNING: could not record the session-DB recovery outcome for the runner's activity log at ${path}: ${describeError(error)}`
62780
+ );
62781
+ }
62782
+ }
62783
+
62536
62784
  // src/session-db-verify.ts
62537
62785
  init_esm_shims();
62538
62786
 
@@ -62586,7 +62834,11 @@ async function verifySessionDb(params) {
62586
62834
  log(
62587
62835
  `WARNING: SESSION-DB-INTEGRITY-FAILED: opencode.db at ${dbPath} failed its integrity check: ${result.detail}`
62588
62836
  );
62589
- return walkBack(params);
62837
+ const decision = await walkBack(params);
62838
+ if (decision.kind !== "exhausted") return decision;
62839
+ const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(params.store, config.prefix, log);
62840
+ const localDb = separation.kind === "notConfigured" || separation.kind === "alreadyEmpty" || separation.kind === "quarantined" ? await discardCorruptDb(fileOps, dbPath, log) : { kind: "retained", reason: "separationUnproven" };
62841
+ return { ...decision, separation, localDb };
62590
62842
  }
62591
62843
  async function clearScratch(fileOps, scratch) {
62592
62844
  await fileOps.remove(scratch);
@@ -62594,15 +62846,32 @@ async function clearScratch(fileOps, scratch) {
62594
62846
  await fileOps.remove(`${scratch}-shm`);
62595
62847
  }
62596
62848
  async function discardCorruptDb(fileOps, dbPath, log) {
62597
- for (const path of [dbPath, `${dbPath}-wal`, `${dbPath}-shm`]) {
62849
+ const paths = [dbPath, `${dbPath}-wal`, `${dbPath}-shm`];
62850
+ const errors = [];
62851
+ for (const path of paths) {
62598
62852
  try {
62599
62853
  await fileOps.remove(path);
62600
62854
  } catch (error) {
62601
- log(
62602
- `WARNING: could not remove corrupt local session-DB debris at ${path}: ${describeError(error)}`
62603
- );
62855
+ errors.push(`${path}: ${describeError(error)}`);
62604
62856
  }
62605
62857
  }
62858
+ const survived = [];
62859
+ const removed = [];
62860
+ for (const path of paths) {
62861
+ try {
62862
+ if (await fileOps.stat(path) === null) removed.push(path);
62863
+ else survived.push(path);
62864
+ } catch (error) {
62865
+ survived.push(path);
62866
+ errors.push(`${path}: could not prove removal: ${describeError(error)}`);
62867
+ }
62868
+ }
62869
+ if (errors.length === 0 && survived.length === 0) return { kind: "discarded" };
62870
+ const detail = errors.length === 0 ? "paths remained after removal" : errors.join("; ");
62871
+ log(
62872
+ `WARNING: SESSION-DB-LOCAL-DISCARD-FAILED: removed=${removed.join(", ") || "(none)"} survived=${survived.join(", ") || "(none)"} errors=${detail}; boot is stopping so no process opens or writes what remains.`
62873
+ );
62874
+ return { kind: "discardFailed", removed, survived, detail };
62606
62875
  }
62607
62876
  async function adopt(fileOps, scratch, dbPath, log, txid, candidatesTried, bytes) {
62608
62877
  await fileOps.remove(`${dbPath}-wal`);
@@ -62624,12 +62893,10 @@ async function walkBack(params) {
62624
62893
  log(
62625
62894
  `WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED: ${listing.stdout === null ? `litestream ltx exited non-zero: ${listing.detail}` : "litestream ltx succeeded but its listing could not be parsed"}`
62626
62895
  );
62627
- await discardCorruptDb(fileOps, dbPath, log);
62628
62896
  return { kind: "exhausted", reason: "enumerationFailed", candidatesTried: 0 };
62629
62897
  }
62630
62898
  const candidates = walkbackCandidates(points);
62631
62899
  if (candidates.length === 0) {
62632
- await discardCorruptDb(fileOps, dbPath, log);
62633
62900
  return { kind: "exhausted", reason: "noCandidates", candidatesTried: 0 };
62634
62901
  }
62635
62902
  const deadline = now() + budgetSeconds * 1e3;
@@ -62660,7 +62927,6 @@ async function walkBack(params) {
62660
62927
  return { kind: "walkedBack", txid, bytes, candidatesTried };
62661
62928
  }
62662
62929
  await clearScratch(fileOps, scratch);
62663
- await discardCorruptDb(fileOps, dbPath, log);
62664
62930
  return {
62665
62931
  kind: "exhausted",
62666
62932
  reason: index < candidates.length ? "budgetExhausted" : "allFailed",
@@ -62845,6 +63111,7 @@ var EXIT_SESSION_DB_FATAL = 30;
62845
63111
  var EXIT_SESSION_DB_UNUSABLE = 31;
62846
63112
  var EXIT_SESSION_DB_RETRY = 32;
62847
63113
  var EXIT_SESSION_DB_UNVERIFIABLE = 33;
63114
+ var EXIT_SESSION_DB_UNSEPARATED = 34;
62848
63115
  var STATE_SUFFIX = ".synchash";
62849
63116
  var COMMANDS = shell_contract_default.commands;
62850
63117
  var USAGE = `Usage: runner-synchroniser <command> [args]
@@ -62857,16 +63124,18 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
62857
63124
  self-stop scale this agent's own ECS service to 0; exit 0 only
62858
63125
  when desiredCount is confirmed 0, ${EXIT_KEEP_TASK} to keep the task
62859
63126
  session-db-classify <litestream-restore-exit-code> <attempt>
62860
- [--on-unusable-replica=<prune|leave|clear|crash>]
63127
+ [--on-unusable-replica=<prune|leave|clear|crash>] [--recovery-occurred]
63128
+ [--fresh-db-fallback]
62861
63129
  classify a just-run \`litestream restore\` of opencode.db;
62862
63130
  exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
62863
- and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
62864
- fresh backup chain), ${EXIT_SESSION_DB_FATAL} fatal
63131
+ and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
63132
+ replicates into the existing prefix), ${EXIT_SESSION_DB_FATAL} fatal
62865
63133
  session-db-verify <litestream-config-path>
62866
63134
  verify opencode.db's integrity at boot, walking back through
62867
63135
  litestream's retained restore points if it's corrupt; exit 0
62868
63136
  healthy/skipped/walked-back, ${EXIT_SESSION_DB_UNVERIFIABLE} every
62869
- candidate was exhausted (booted fresh)
63137
+ candidate was exhausted (booted fresh), ${EXIT_SESSION_DB_UNSEPARATED}
63138
+ separation/local disposal could not be proven
62870
63139
  `;
62871
63140
  function shellQuote(value) {
62872
63141
  return `'${value.replaceAll("'", `'\\''`)}'`;
@@ -62887,16 +63156,26 @@ function parseStoreName(value) {
62887
63156
  return value === "claude" || value === "opencode" ? value : null;
62888
63157
  }
62889
63158
  var ON_UNUSABLE_REPLICA_FLAG = "--on-unusable-replica=";
63159
+ var RECOVERY_OCCURRED_FLAG = "--recovery-occurred";
63160
+ var FRESH_DB_FALLBACK_FLAG = "--fresh-db-fallback";
62890
63161
  function parseNonNegativeInt(value) {
62891
63162
  return /^\d+$/.test(value) ? Number.parseInt(value, 10) : null;
62892
63163
  }
62893
63164
  function parseSessionDbClassifyArgs(args) {
62894
63165
  let strategyRaw;
63166
+ let recoveryOccurred = false;
63167
+ let freshDbFallback = false;
62895
63168
  const positional = [];
62896
63169
  for (const arg of args) {
62897
63170
  if (arg.startsWith(ON_UNUSABLE_REPLICA_FLAG)) {
62898
63171
  if (strategyRaw !== void 0) return null;
62899
63172
  strategyRaw = arg.slice(ON_UNUSABLE_REPLICA_FLAG.length);
63173
+ } else if (arg === RECOVERY_OCCURRED_FLAG) {
63174
+ if (recoveryOccurred) return null;
63175
+ recoveryOccurred = true;
63176
+ } else if (arg === FRESH_DB_FALLBACK_FLAG) {
63177
+ if (freshDbFallback) return null;
63178
+ freshDbFallback = true;
62900
63179
  } else if (arg.startsWith("--")) {
62901
63180
  return null;
62902
63181
  } else {
@@ -62908,7 +63187,7 @@ function parseSessionDbClassifyArgs(args) {
62908
63187
  const attempt = parseNonNegativeInt(positional[1]);
62909
63188
  const strategy = parseUnusableReplicaStrategy(strategyRaw);
62910
63189
  if (exitCode === null || attempt === null || attempt < 1 || strategy === null) return null;
62911
- return { exitCode, attempt, strategy };
63190
+ return { exitCode, attempt, strategy, recoveryOccurred, freshDbFallback };
62912
63191
  }
62913
63192
  function renderEnv(config) {
62914
63193
  return [
@@ -62996,6 +63275,16 @@ async function commandSessionDbClassify(args, config, deps) {
62996
63275
  log
62997
63276
  });
62998
63277
  log(describeSessionDbClassification(outcome, { bucket: config.bucket, region: config.region }));
63278
+ const record = recordRestoreOutcome(
63279
+ outcome,
63280
+ (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString(),
63281
+ args.exitCode,
63282
+ args.attempt,
63283
+ args.recoveryOccurred,
63284
+ args.freshDbFallback
63285
+ );
63286
+ if (record !== null)
63287
+ await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
62999
63288
  return sessionDbExitCode(outcome);
63000
63289
  }
63001
63290
  async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
@@ -63005,7 +63294,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
63005
63294
  config.opencodeDbPath
63006
63295
  );
63007
63296
  const outcome = await verifySessionDb({
63008
- config: { opencodeDbPath: config.opencodeDbPath },
63297
+ config: { opencodeDbPath: config.opencodeDbPath, prefix: config.prefix },
63298
+ store: (deps.objectStoreFor ?? s3ObjectStoreFor)(config),
63009
63299
  fileOps,
63010
63300
  sqlite: (deps.sqliteIntegrityFor ?? nodeSqliteIntegrityFor)(),
63011
63301
  litestream,
@@ -63014,6 +63304,9 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
63014
63304
  budgetSeconds: config.walkbackBudgetSeconds
63015
63305
  });
63016
63306
  log(describeSessionDbVerification(outcome));
63307
+ const record = recordVerifyOutcome(outcome, (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString());
63308
+ if (record !== null)
63309
+ await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
63017
63310
  return sessionDbVerifyExitCode(outcome);
63018
63311
  }
63019
63312
  async function main(argv, deps) {
@@ -63055,7 +63348,7 @@ ${USAGE}`);
63055
63348
  const args = parseSessionDbClassifyArgs(rest);
63056
63349
  if (args === null) {
63057
63350
  log(
63058
- `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>.
63351
+ `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>, --recovery-occurred, and --fresh-db-fallback.
63059
63352
  ${USAGE}`
63060
63353
  );
63061
63354
  return EXIT_USAGE;
@@ -63111,6 +63404,7 @@ export {
63111
63404
  EXIT_NOT_READY,
63112
63405
  EXIT_SESSION_DB_FATAL,
63113
63406
  EXIT_SESSION_DB_RETRY,
63407
+ EXIT_SESSION_DB_UNSEPARATED,
63114
63408
  EXIT_SESSION_DB_UNUSABLE,
63115
63409
  EXIT_SESSION_DB_UNVERIFIABLE,
63116
63410
  EXIT_USAGE,
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@evident-ai/runner-synchroniser",
3
- "version": "3.4.0",
3
+ "version": "3.4.1-dev.3adba63",
4
4
  "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
5
5
  "type": "module",
6
6
  "main": "./dist/cli.js",
@@ -26,7 +26,7 @@
26
26
  "tsup": "^8.3.5",
27
27
  "typescript": "^5.7.2",
28
28
  "vitest": "^3.1.1",
29
- "runner-image": "workspace:*"
29
+ "runner-fargate-image": "workspace:*"
30
30
  },
31
31
  "engines": {
32
32
  "node": ">=22.13.0"
@@ -42,7 +42,7 @@
42
42
  "repository": {
43
43
  "type": "git",
44
44
  "url": "https://github.com/sroze/evident.git",
45
- "directory": "packages/runner-synchroniser"
45
+ "directory": "runner/synchroniser"
46
46
  },
47
47
  "homepage": "https://evident.run",
48
48
  "author": "Evident <hello@evident.run>",