@evident-ai/runner-synchroniser 3.4.0 → 3.4.1-dev.1549524

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +47 -17
  2. package/dist/cli.js +419 -52
  3. package/package.json +3 -3
package/README.md CHANGED
@@ -59,8 +59,8 @@ build time.
59
59
  | `sync-once <claude\|opencode>` | — | `0` = ran; non-zero = tool broken |
60
60
  | `model-auth-ready` | — | `0` = ready, `10` = not ready; other = tool broken |
61
61
  | `self-stop` | — | `0` = stopped, `20` = keep the task; other = tool broken |
62
- | `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, starts a fresh backup chain), `30` = fatal; other = usage/tool broken |
63
- | `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = every retained restore point was exhausted (booted fresh, starts a fresh backup chain); other = usage/tool broken |
62
+ | `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, replicates into the existing prefix), `30` = fatal; other = usage/tool broken |
63
+ | `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = exhausted after verified separation and local disposal, `34` = separation or disposal could not be proven; other = usage/tool broken |
64
64
 
65
65
  `self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
66
66
  (see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
@@ -75,7 +75,16 @@ saved query.
75
75
 
76
76
  ### `session-db-classify`
77
77
 
78
- Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
78
+ Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `runner/docker-images/fargate/README.md`'s [Strategy/What/Cost table](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
79
+
80
+ #### Activity recovery report
81
+
82
+ Reportable recovery outcomes append one versioned JSON line to the session-DB recovery
83
+ report. `evident run` drains that file once after authentication and records the outcome in
84
+ the runner activity log. Healthy boots do not write a report. An initial restore failure writes
85
+ a warning `restore_retried` record before the retry's result is known. Writing is best-effort
86
+ and never changes this command's exit code. The CLI reader owns this record contract: add a new
87
+ `v` rather than repurposing a field.
79
88
 
80
89
  #### Positionals
81
90
 
@@ -102,6 +111,14 @@ Implementation-facing reference for the 7th command: it decides what a just-run
102
111
  - Any other `--` token is rejected.
103
112
  - Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
104
113
 
114
+ #### Recovery-report flags
115
+
116
+ `--recovery-occurred` marks a later successful restore after this boot already recovered a
117
+ replica, so an empty result is reported as a fresh session database. `--fresh-db-fallback`
118
+ marks a runtime that intentionally has no retry budget and will therefore boot fresh after this
119
+ otherwise-transient failure. Each flag may appear once, does not change the command's exit code,
120
+ and is rejected when repeated.
121
+
105
122
  #### Outcome → exit code
106
123
 
107
124
  | Outcome | Code | When |
@@ -121,6 +138,9 @@ Implementation-facing reference for the 7th command: it decides what a just-run
121
138
  | `fatal(misconfig)` | `30` | no object store while persistence is enabled, or the probe failed on attempt 2 |
122
139
  | `fatal(deliberate)` | `30` | attempt 2, `crash` |
123
140
 
141
+ Fatal exit codes 30 and 34 are reported directly to `POST /v1/runners/self/startup-failure`
142
+ before the CLI starts. Reporting is best-effort and never changes the exit code.
143
+
124
144
  `31` also guarantees local debris (`opencode.db`, `-wal`, `-shm`) is discarded, one `try` per path, at the classifier's single return point. Note `32` isn't purely "transient" — `recovered` shares it with `retryTransient` even though a `recovered` outcome already mutated S3.
125
145
 
126
146
  The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
@@ -196,11 +216,11 @@ what it is.
196
216
  #### The four `--on-unusable-replica` strategies
197
217
 
198
218
  - **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
199
- - **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". Since #1106 it is **not** also ephemeral: the boot still starts a fresh backup chain, because the classifier guarantees the local DB is fresh rather than half-restored. Upside: replica left byte-intact for forensics, no crash loop.
219
+ - **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". The fresh local DB replicates into the existing prefix; no new chain is started. Upside: the runner still comes online without a crash loop.
200
220
  - **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
201
221
  - **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
202
222
 
203
- #### The measured real-world key layout (litestream 0.5.13)
223
+ #### The measured real-world key layout (litestream 0.5.13 historical sample)
204
224
 
205
225
  `parseLtxKey` (`src/replica-keys.ts`) expects
206
226
  `<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
@@ -218,7 +238,7 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
218
238
  (`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
219
239
  and it can never repair corruption at a higher compaction level.
220
240
 
221
- `EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
241
+ `EVIDENT_ON_UNUSABLE_REPLICA` is a `runner/docker-images/fargate` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [the Fargate image README](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
222
242
 
223
243
  See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
224
244
  the behavioural anchor (31 comes online, 30 does not).
@@ -245,9 +265,12 @@ nothing had ever run a real `PRAGMA integrity_check`).
245
265
  each into a scratch path and integrity-checking it there. The first one that passes
246
266
  is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
247
267
  3. If nothing retained passes (or there's nothing to try, or the search runs out of
248
- time/points), the corrupt local DB is discarded and the boot proceeds with a
249
- genuinely fresh one exactly `session-db-classify`'s `31` guarantee, so
250
- `litestream replicate` starting against it afterwards is just as safe.
268
+ time/points), the active replica prefix is separated first: its history is moved aside,
269
+ then the prefix is re-listed as empty before the corrupt local DB and sidecars are
270
+ discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
271
+ OpenCode or Litestream starts; the backup history remains readable at its quarantine
272
+ destination or original key, and surviving local files remain in place without a process
273
+ opening or writing them.
251
274
 
252
275
  A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
253
276
  prefix. litestream re-bases to the replica's high-water mark and continues the txid
@@ -262,10 +285,8 @@ against S3 at the ~1 GB scale a production replica can reach.)
262
285
  | `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
263
286
  | `healthy` | `0` | the local DB passed its integrity check as-is |
264
287
  | `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
265
- | `exhausted` (`noCandidates`) | `33` | the local DB failed and litestream retains no older restore point |
266
- | `exhausted` (`allFailed`) | `33` | every retained restore point was tried and every one failed |
267
- | `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
268
- | `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
288
+ | `exhausted` (`noCandidates`, `allFailed`, `budgetExhausted`, or `enumerationFailed`) | `33` | the active prefix was proven separated (or persistence is disabled) **and** the corrupt local DB and sidecars were proven discarded |
289
+ | `exhausted` | `34` | either the replica separation or local discard could not be proven; the boot stops before OpenCode and Litestream start |
269
290
 
270
291
  `healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
271
292
  afterwards — start `litestream replicate` — is identical either way, so a further split
@@ -284,7 +305,10 @@ before looking at everything" would report the wrong conclusion to the operator.
284
305
  | `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
285
306
  | `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
286
307
  | `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
287
- | `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed (any `exhausted` reason); the boot proceeds with a fresh DB | `candidatesTried` (in prose, not `k=v`) |
308
+ | `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed; a fresh boot follows only when both separation and local disposal were proven | `candidatesTried` (in prose, not `k=v`) |
309
+ | `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
310
+ | `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
311
+ | `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
288
312
 
289
313
  The first four fire only while the walkback loop runs; the last three are always the
290
314
  one message this command logs as its own final word on the outcome — `entrypoint.sh`
@@ -316,10 +340,15 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
316
340
 
317
341
  `session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
318
342
 
343
+ `session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
344
+ history. A refused boot records `session_db_boot_refused`, but `evident run` has not started to
345
+ drain it, so operators see that refusal in boot logs and the restart loop. The report does not
346
+ alter the command's exit code.
347
+
319
348
  `src/shell-contract.json` is the machine-checked source of truth for the command list, and
320
349
  `shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
321
- `packages/runner-image/entrypoint.sh` and the MicroVM's
322
- `packages/runner-cdk/microvm-image/hooks` (#608), discovered by grep so a third one
350
+ `runner/docker-images/fargate/entrypoint.sh` and the MicroVM's
351
+ `runner/docker-images/microvm/hooks` (#608), discovered by grep so a third one
323
352
  cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
324
353
  `entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
325
354
  a broken tool; and take each answer code the commands it calls can return **silently and by
@@ -344,11 +373,12 @@ key, so they never appear in `env`'s output or `litestream.yml`.
344
373
  | `CREDS_SYNC_INTERVAL` | Seconds between sync ticks, reported by `env` for the caller's loop. | Falls back to `60`; also on unparseable or non-positive values. |
345
374
  | `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS` | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values. |
346
375
  | `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop. | Falls back to `180`; also on unparseable or non-positive values. |
376
+ | `EVIDENT_SESSION_DB_RECOVERY_REPORT` | JSONL report drained once into runner activity after authentication. | `$HOME/.local/state/evident/session-db-recovery.jsonl`; blank values use the default. |
347
377
  | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
348
378
  | `CLUSTER` † / `SERVICE` † | ECS cluster/service `self-stop` scales to `desiredCount=0`. | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
349
379
  | `EVIDENT_SELFSTOP_ROLE_ARN` † | Role `self-stop` assumes for its ECS calls. | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
350
380
 
351
- `packages/runner-image/README.md` documents these same three from the deployment side —
381
+ `runner/docker-images/fargate/README.md` documents these same three from the deployment side —
352
382
  keep them in sync.
353
383
 
354
384
  Requiring **both** `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` (never just one) means
package/dist/cli.js CHANGED
@@ -51907,7 +51907,8 @@ function positiveIntOr(value, defaultValue) {
51907
51907
  return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue;
51908
51908
  }
51909
51909
  function nonEmpty(value) {
51910
- return value !== void 0 && value !== "" ? value : null;
51910
+ const trimmed = value?.trim();
51911
+ return trimmed ? trimmed : null;
51911
51912
  }
51912
51913
  function resolveConfig(env4) {
51913
51914
  const homeDir = nonEmpty(env4.HOME);
@@ -51930,6 +51931,7 @@ function resolveConfig(env4) {
51930
51931
  key: persistenceEnabled ? `${prefix}/opencode/auth.json` : null
51931
51932
  },
51932
51933
  opencodeDbPath: `${homeDir}/.local/share/opencode/opencode.db`,
51934
+ sessionDbRecoveryReportPath: nonEmpty(env4.EVIDENT_SESSION_DB_RECOVERY_REPORT) ?? `${homeDir}/.local/state/evident/session-db-recovery.jsonl`,
51933
51935
  bucket: persistenceEnabled ? bucket : null,
51934
51936
  prefix: persistenceEnabled ? prefix : null,
51935
51937
  region: nonEmpty(env4.AWS_REGION),
@@ -51942,7 +51944,9 @@ function resolveConfig(env4) {
51942
51944
  env4.EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS,
51943
51945
  DEFAULT_WALKBACK_BUDGET_SECONDS
51944
51946
  ),
51945
- hasModelApiKey: nonEmpty(env4.ANTHROPIC_API_KEY) !== null || nonEmpty(env4.OPENAI_API_KEY) !== null
51947
+ hasModelApiKey: nonEmpty(env4.ANTHROPIC_API_KEY) !== null || nonEmpty(env4.OPENAI_API_KEY) !== null,
51948
+ runnerKey: nonEmpty(env4.EVIDENT_RUNNER_KEY) ?? nonEmpty(env4.EVIDENT_AGENT_KEY),
51949
+ apiUrl: nonEmpty(env4.EVIDENT_API_URL)
51946
51950
  };
51947
51951
  }
51948
51952
  function storeOf(config, name) {
@@ -51998,7 +52002,7 @@ function sessionDbExitCode(outcome) {
51998
52002
  // re-run `litestream restore` and ask again
51999
52003
  case "unusableReplica":
52000
52004
  return 31;
52001
- // booted with a fresh DB; the shell still starts a fresh backup chain (#1106)
52005
+ // booted with a fresh DB; the shell still replicates into the existing prefix
52002
52006
  case "fatal":
52003
52007
  return 30;
52004
52008
  // shared by a genuine misconfig and a deliberate `crash` choice
@@ -52007,20 +52011,20 @@ function sessionDbExitCode(outcome) {
52007
52011
  }
52008
52012
  }
52009
52013
  function describeUnusableReplica(reason) {
52010
- const base = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
52014
+ const base2 = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
52011
52015
  switch (reason) {
52012
52016
  case "strategy":
52013
- return `${base} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
52017
+ return `${base2} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
52014
52018
  case "attemptsExhausted":
52015
- return `${base} (recovery did not make the replica usable after retrying; giving up.)`;
52019
+ return `${base2} (recovery did not make the replica usable after retrying; giving up.)`;
52016
52020
  case "layoutMismatch":
52017
- return `${base} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
52021
+ return `${base2} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
52018
52022
  case "noL0Present":
52019
- return `${base} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
52023
+ return `${base2} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
52020
52024
  case "recoveryFailed":
52021
- return `${base} (the recovery delete itself failed; see the warning above for the S3 error.)`;
52025
+ return `${base2} (the recovery delete itself failed; see the warning above for the S3 error.)`;
52022
52026
  case "targetHealthy":
52023
- return `${base} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
52027
+ return `${base2} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
52024
52028
  default:
52025
52029
  return assertNeverUnusableReplicaReason(reason);
52026
52030
  }
@@ -52061,7 +52065,7 @@ function describeSessionDbClassification(outcome, config) {
52061
52065
  case "recovered": {
52062
52066
  const objects = outcome.deleted.length === 0 ? "no objects (see the warnings above for what failed)" : `${outcome.deleted.length} object${outcome.deleted.length === 1 ? "" : "s"} (${outcome.deleted.join(", ")})`;
52063
52067
  if (outcome.strategy === "quarantine") {
52064
- return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside, NOT deleted \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
52068
+ return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside and remains readable at the quarantine destination \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
52065
52069
  }
52066
52070
  return `Attempted opencode.db replica recovery via --on-unusable-replica=${outcome.strategy}, deleting ${objects}; retrying the restore.` + (outcome.strategy === "clear" ? " Prior session history is lost." : "");
52067
52071
  }
@@ -52082,6 +52086,12 @@ function assertNeverSessionDbVerifySkippedLocalDb(value) {
52082
52086
  function assertNeverSessionDbVerifyExhaustedReason(value) {
52083
52087
  throw new Error(`Unhandled session-DB verify exhausted reason: ${JSON.stringify(value)}`);
52084
52088
  }
52089
+ function assertNeverSessionDbVerifySeparation(value) {
52090
+ throw new Error(`Unhandled session-DB verify separation: ${JSON.stringify(value)}`);
52091
+ }
52092
+ function assertNeverSessionDbVerifyLocalDb(value) {
52093
+ throw new Error(`Unhandled session-DB verify local-DB disposal: ${JSON.stringify(value)}`);
52094
+ }
52085
52095
  function sessionDbVerifyExitCode(outcome) {
52086
52096
  switch (outcome.kind) {
52087
52097
  case "skipped":
@@ -52089,8 +52099,26 @@ function sessionDbVerifyExitCode(outcome) {
52089
52099
  case "walkedBack":
52090
52100
  return 0;
52091
52101
  case "exhausted":
52092
- return 33;
52093
- // booted with a fresh DB; the shell still starts a fresh backup chain
52102
+ switch (outcome.separation.kind) {
52103
+ case "notConfigured":
52104
+ case "alreadyEmpty":
52105
+ case "quarantined":
52106
+ break;
52107
+ case "incomplete":
52108
+ case "unreachable":
52109
+ return 34;
52110
+ default:
52111
+ return assertNeverSessionDbVerifySeparation(outcome.separation);
52112
+ }
52113
+ switch (outcome.localDb.kind) {
52114
+ case "discarded":
52115
+ return 33;
52116
+ case "retained":
52117
+ case "discardFailed":
52118
+ return 34;
52119
+ default:
52120
+ return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
52121
+ }
52094
52122
  default:
52095
52123
  return assertNeverSessionDbVerify(outcome);
52096
52124
  }
@@ -52107,18 +52135,56 @@ function describeSessionDbVerifySkipped(localDb) {
52107
52135
  }
52108
52136
  function describeSessionDbVerifyExhausted(outcome) {
52109
52137
  const tried = `tried ${outcome.candidatesTried} restore point${outcome.candidatesTried === 1 ? "" : "s"}`;
52138
+ let reason;
52110
52139
  switch (outcome.reason) {
52111
52140
  case "enumerationFailed":
52112
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried}); booting with a FRESH opencode.db.`;
52141
+ reason = `opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried})`;
52142
+ break;
52113
52143
  case "noCandidates":
52114
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and no older restore point is retained (${tried}); booting with a FRESH opencode.db.`;
52144
+ reason = `opencode.db failed its integrity check and no older restore point is retained (${tried})`;
52145
+ break;
52115
52146
  case "allFailed":
52116
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check and every retained restore point also failed (${tried}); booting with a FRESH opencode.db.`;
52147
+ reason = `opencode.db failed its integrity check and every retained restore point also failed (${tried})`;
52148
+ break;
52117
52149
  case "budgetExhausted":
52118
- return `SESSION-DB-INTEGRITY-EXHAUSTED: opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try; booting with a FRESH opencode.db.`;
52150
+ reason = `opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try`;
52151
+ break;
52119
52152
  default:
52120
52153
  return assertNeverSessionDbVerifyExhaustedReason(outcome.reason);
52121
52154
  }
52155
+ let separation;
52156
+ switch (outcome.separation.kind) {
52157
+ case "quarantined":
52158
+ separation = ` The active replica prefix was verified empty; moved history remains readable at ${outcome.separation.destination}.`;
52159
+ break;
52160
+ case "alreadyEmpty":
52161
+ separation = " The active replica prefix was already empty.";
52162
+ break;
52163
+ case "notConfigured":
52164
+ separation = " Persistence is disabled, so there is no replica to separate.";
52165
+ break;
52166
+ case "incomplete":
52167
+ case "unreachable":
52168
+ separation = ` Boot is stopping because replica separation could not be proven (${outcome.separation.detail}); backup history remains readable at its original key or quarantine destination.`;
52169
+ break;
52170
+ default:
52171
+ return assertNeverSessionDbVerifySeparation(outcome.separation);
52172
+ }
52173
+ let localDb;
52174
+ switch (outcome.localDb.kind) {
52175
+ case "discarded":
52176
+ localDb = " booting with a FRESH opencode.db.";
52177
+ break;
52178
+ case "retained":
52179
+ localDb = " The corrupt local database and sidecars were left untouched for investigation.";
52180
+ break;
52181
+ case "discardFailed":
52182
+ localDb = ` Boot is stopping because local disposal could not be proven; removed=${outcome.localDb.removed.join(", ") || "(none)"} survived=${outcome.localDb.survived.join(", ") || "(none)"}. See SESSION-DB-LOCAL-DISCARD-FAILED.`;
52183
+ break;
52184
+ default:
52185
+ return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
52186
+ }
52187
+ return `SESSION-DB-INTEGRITY-EXHAUSTED: ${reason}.${separation}${localDb}`;
52122
52188
  }
52123
52189
  function describeSessionDbVerification(outcome) {
52124
52190
  switch (outcome.kind) {
@@ -52138,7 +52204,17 @@ function describeSessionDbVerification(outcome) {
52138
52204
  // src/file-ops.ts
52139
52205
  init_esm_shims();
52140
52206
  import { createHash as createHash8 } from "node:crypto";
52141
- import { chmod, mkdir, open, readFile as readFile5, rename, rm, stat, writeFile as writeFile4 } from "node:fs/promises";
52207
+ import {
52208
+ appendFile,
52209
+ chmod,
52210
+ mkdir,
52211
+ open,
52212
+ readFile as readFile5,
52213
+ rename,
52214
+ rm,
52215
+ stat,
52216
+ writeFile as writeFile4
52217
+ } from "node:fs/promises";
52142
52218
  var nodeFileOps = {
52143
52219
  async mkdirp(dir) {
52144
52220
  await mkdir(dir, { recursive: true });
@@ -52150,14 +52226,18 @@ var nodeFileOps = {
52150
52226
  try {
52151
52227
  const stats = await stat(path);
52152
52228
  return { size: stats.size };
52153
- } catch {
52154
- return null;
52229
+ } catch (error) {
52230
+ if (error instanceof Error && "code" in error && error.code === "ENOENT") return null;
52231
+ throw error;
52155
52232
  }
52156
52233
  },
52157
52234
  readFile: readFile5,
52158
52235
  async writeFile(path, data) {
52159
52236
  await writeFile4(path, data);
52160
52237
  },
52238
+ async appendFile(path, data) {
52239
+ await appendFile(path, data);
52240
+ },
52161
52241
  async createExclusive(path) {
52162
52242
  const handle = await open(path, "wx", 384);
52163
52243
  await handle.close();
@@ -52252,7 +52332,7 @@ function renderLitestreamConfig(config) {
52252
52332
  # (no static keys here).
52253
52333
  #
52254
52334
  # INVARIANT (see README "Single-writer invariant"): exactly ONE writer per S3
52255
- # prefix \u2014 v0.5.13 does NOT enforce a server-side lease, so never run two
52335
+ # prefix \u2014 we do not enable Litestream's server-side lease, so never run two
52256
52336
  # replicators on one prefix. Only opencode.db is replicated.
52257
52337
 
52258
52338
  dbs:
@@ -62324,7 +62404,7 @@ async function probeReplica(store, prefix, log) {
62324
62404
  const objectCount = objects.length;
62325
62405
  const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
62326
62406
  logReplicaSize(root12, objectCount, totalBytes, log);
62327
- return { ok: true, keys: objects.map((object) => object.key) };
62407
+ return { ok: true, objects };
62328
62408
  } catch (error) {
62329
62409
  const detail = describeError(error);
62330
62410
  log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
@@ -62381,12 +62461,14 @@ async function pruneNewestL0(store, prefix, keys, log) {
62381
62461
  function quarantineRoot(prefix, stamp) {
62382
62462
  return `${prefix}/quarantine/opencode.db/${stamp}/`;
62383
62463
  }
62384
- async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
62464
+ async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
62385
62465
  const destination = quarantineRoot(prefix, stamp);
62386
62466
  const root12 = replicaDbPrefix(prefix);
62387
62467
  const moved = [];
62388
62468
  const failed = [];
62389
- for (const key of keys) {
62469
+ let movedBytes = 0;
62470
+ for (const object of objects) {
62471
+ const { key } = object;
62390
62472
  if (!isDeletableReplicaKey(prefix, key)) {
62391
62473
  log(`WARNING: refusing to quarantine ${key}: outside the replica prefix.`);
62392
62474
  continue;
@@ -62411,15 +62493,48 @@ async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__
62411
62493
  continue;
62412
62494
  }
62413
62495
  moved.push(key);
62496
+ movedBytes += object.size;
62414
62497
  }
62415
62498
  logQuarantined(destination, moved, failed, log);
62416
- return { destination, moved, failed };
62499
+ return { destination, moved, failed, movedBytes };
62417
62500
  }
62418
62501
  function logQuarantined(destination, moved, failed, log) {
62419
62502
  log(
62420
- `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved) so this boot can start a fresh backup chain. The bytes are NOT deleted \u2014 copy them back from there to investigate. See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
62503
+ `WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (failed.length > 0 ? " The active replica prefix is not clear." : "") + `See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
62421
62504
  );
62422
62505
  }
62506
+ async function separateCorruptReplica(store, prefix, log, stamp) {
62507
+ const probe = await probeReplica(store, prefix, log);
62508
+ if (!probe.ok) return { kind: "unreachable", detail: probe.detail };
62509
+ const root12 = replicaDbPrefix(prefix);
62510
+ if (probe.objects.length === 0) {
62511
+ log(
62512
+ `INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was already empty; this boot starts a fresh backup chain.`
62513
+ );
62514
+ return { kind: "alreadyEmpty" };
62515
+ }
62516
+ const result = await quarantineReplica(store, prefix, probe.objects, log, stamp);
62517
+ try {
62518
+ const remaining = await store.list(root12);
62519
+ if (remaining.length > 0) {
62520
+ const detail = `${remaining.length} object(s) remain`;
62521
+ log(
62522
+ `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: ${root12} still has ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
62523
+ );
62524
+ return { kind: "incomplete", detail };
62525
+ }
62526
+ } catch (error) {
62527
+ const detail = describeError(error);
62528
+ log(
62529
+ `WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: could not re-list ${root12}: ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
62530
+ );
62531
+ return { kind: "incomplete", detail };
62532
+ }
62533
+ log(
62534
+ `INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was verified empty by re-listing after moving ${result.moved.length} object(s) to ${result.destination}; this boot starts a fresh backup chain.`
62535
+ );
62536
+ return { kind: "quarantined", destination: result.destination, moved: result.moved };
62537
+ }
62423
62538
  async function clearReplica(store, prefix, keys, log) {
62424
62539
  const deleted = [];
62425
62540
  const failed = [];
@@ -62471,7 +62586,7 @@ async function decideSessionDbRestore(params) {
62471
62586
  return { kind: "retryTransient" };
62472
62587
  }
62473
62588
  if (attempt >= 3) {
62474
- return { kind: "unusableReplica", reason: "attemptsExhausted" };
62589
+ return { kind: "unusableReplica", reason: "attemptsExhausted", recovery: null };
62475
62590
  }
62476
62591
  const probe = await probeReplica(store, prefix, log);
62477
62592
  if (!probe.ok) {
@@ -62481,7 +62596,12 @@ async function decideSessionDbRestore(params) {
62481
62596
  detail: `could not verify the session-DB replica is reachable: ${probe.detail}`
62482
62597
  };
62483
62598
  }
62484
- const keys = probe.keys;
62599
+ const keys = probe.objects.map((object) => object.key);
62600
+ const recovery = {
62601
+ replicaObjects: probe.objects.length,
62602
+ replicaBytes: probe.objects.reduce((total, object) => total + object.size, 0),
62603
+ quarantine: null
62604
+ };
62485
62605
  switch (strategy) {
62486
62606
  case "crash":
62487
62607
  return {
@@ -62490,30 +62610,43 @@ async function decideSessionDbRestore(params) {
62490
62610
  detail: "the replica is unusable and --on-unusable-replica=crash was selected; no S3 object was touched"
62491
62611
  };
62492
62612
  case "leave":
62493
- return { kind: "unusableReplica", reason: "strategy" };
62613
+ return { kind: "unusableReplica", reason: "strategy", recovery };
62494
62614
  case "prune": {
62495
62615
  const result = await pruneNewestL0(store, prefix, keys, log);
62496
62616
  if (result.deleted === null) {
62497
62617
  if (shouldEscalateToQuarantine(result.reason, prefix, keys)) {
62498
- const quarantined = await quarantineReplica(store, prefix, keys, log);
62618
+ const quarantined = await quarantineReplica(store, prefix, probe.objects, log);
62499
62619
  if (quarantined.moved.length === 0) {
62500
- return { kind: "unusableReplica", reason: "recoveryFailed" };
62620
+ return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
62501
62621
  }
62502
- return { kind: "recovered", strategy: "quarantine", deleted: quarantined.moved };
62622
+ return {
62623
+ kind: "recovered",
62624
+ strategy: "quarantine",
62625
+ deleted: quarantined.moved,
62626
+ recovery: {
62627
+ ...recovery,
62628
+ quarantine: {
62629
+ destination: quarantined.destination,
62630
+ movedObjects: quarantined.moved.length,
62631
+ failedObjects: quarantined.failed.length,
62632
+ movedBytes: quarantined.movedBytes
62633
+ }
62634
+ }
62635
+ };
62503
62636
  }
62504
62637
  if (result.reason === "deleteFailed") {
62505
- return { kind: "unusableReplica", reason: "recoveryFailed" };
62638
+ return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
62506
62639
  }
62507
62640
  if (result.reason === "unreadable") {
62508
- return { kind: "unusableReplica", reason: "targetHealthy" };
62641
+ return { kind: "unusableReplica", reason: "targetHealthy", recovery };
62509
62642
  }
62510
- return { kind: "unusableReplica", reason: result.reason };
62643
+ return { kind: "unusableReplica", reason: result.reason, recovery };
62511
62644
  }
62512
- return { kind: "recovered", strategy: "prune", deleted: [result.deleted] };
62645
+ return { kind: "recovered", strategy: "prune", deleted: [result.deleted], recovery };
62513
62646
  }
62514
62647
  case "clear": {
62515
62648
  const result = await clearReplica(store, prefix, keys, log);
62516
- return { kind: "recovered", strategy: "clear", deleted: result.deleted };
62649
+ return { kind: "recovered", strategy: "clear", deleted: result.deleted, recovery };
62517
62650
  }
62518
62651
  default:
62519
62652
  return assertNeverStrategy(strategy);
@@ -62533,6 +62666,123 @@ async function discardLocalDebris(fileOps, dbPath, log) {
62533
62666
  }
62534
62667
  }
62535
62668
 
62669
+ // src/session-db-recovery-report.ts
62670
+ init_esm_shims();
62671
+ import { dirname as dirname4 } from "node:path";
62672
+ function base(at2, stage) {
62673
+ return {
62674
+ v: 1,
62675
+ event: "session_db_recovery",
62676
+ at: at2,
62677
+ stage,
62678
+ litestream_exit_code: null,
62679
+ attempt: null,
62680
+ replica_objects: null,
62681
+ replica_bytes: null,
62682
+ quarantine_destination: null,
62683
+ quarantined_objects: null,
62684
+ quarantine_failed_objects: null,
62685
+ quarantined_bytes: null,
62686
+ verified_restore_point: null,
62687
+ restore_points_tried: null
62688
+ };
62689
+ }
62690
+ function recordRestoreOutcome(outcome, at2, exitCode, attempt, recoveryOccurred, freshDbFallback = false) {
62691
+ const record = { ...base(at2, "restore"), litestream_exit_code: exitCode, attempt };
62692
+ if (outcome.kind === "disabled" || outcome.kind === "restored") return null;
62693
+ if (outcome.kind === "retryTransient")
62694
+ return freshDbFallback ? {
62695
+ ...record,
62696
+ outcome: "fresh_session_db",
62697
+ severity: "error",
62698
+ reason: "retry_budget_exhausted"
62699
+ } : {
62700
+ ...record,
62701
+ outcome: "restore_retried",
62702
+ severity: "warning",
62703
+ reason: "initial_restore_failed"
62704
+ };
62705
+ if (outcome.kind === "noReplica") {
62706
+ return recoveryOccurred ? {
62707
+ ...record,
62708
+ outcome: "fresh_session_db",
62709
+ severity: "error",
62710
+ reason: "no_replica_after_recovery"
62711
+ } : null;
62712
+ }
62713
+ if (outcome.kind === "fatal") return null;
62714
+ if (outcome.kind === "unusableReplica")
62715
+ return {
62716
+ ...record,
62717
+ outcome: "fresh_session_db",
62718
+ severity: "error",
62719
+ reason: outcome.reason,
62720
+ replica_objects: outcome.recovery?.replicaObjects ?? null,
62721
+ replica_bytes: outcome.recovery?.replicaBytes ?? null
62722
+ };
62723
+ return {
62724
+ ...record,
62725
+ outcome: "replica_recovered",
62726
+ severity: outcome.strategy === "clear" ? "error" : "warning",
62727
+ reason: outcome.strategy,
62728
+ replica_objects: outcome.recovery.replicaObjects,
62729
+ replica_bytes: outcome.recovery.replicaBytes,
62730
+ quarantine_destination: outcome.recovery.quarantine?.destination ?? null,
62731
+ quarantined_objects: outcome.recovery.quarantine?.movedObjects ?? null,
62732
+ quarantine_failed_objects: outcome.recovery.quarantine?.failedObjects ?? null,
62733
+ quarantined_bytes: outcome.recovery.quarantine?.movedBytes ?? null
62734
+ };
62735
+ }
62736
+ function recordVerifyOutcome(outcome, at2) {
62737
+ const record = base(at2, "verify");
62738
+ if (outcome.kind === "skipped" || outcome.kind === "healthy") return null;
62739
+ if (outcome.kind === "walkedBack")
62740
+ return {
62741
+ ...record,
62742
+ outcome: "history_rolled_back",
62743
+ severity: "warning",
62744
+ reason: "walkback",
62745
+ verified_restore_point: outcome.txid,
62746
+ restore_points_tried: outcome.candidatesTried
62747
+ };
62748
+ if (sessionDbVerifyExitCode(outcome) === 34)
62749
+ return {
62750
+ ...record,
62751
+ outcome: "session_db_boot_refused",
62752
+ severity: "error",
62753
+ reason: outcome.localDb.kind === "discardFailed" ? "local_discard_failed" : "replica_separation_unproven",
62754
+ restore_points_tried: outcome.candidatesTried,
62755
+ quarantine_destination: outcome.separation.kind === "quarantined" ? outcome.separation.destination : null
62756
+ };
62757
+ return {
62758
+ ...record,
62759
+ outcome: "fresh_session_db",
62760
+ severity: "error",
62761
+ reason: outcome.reason,
62762
+ restore_points_tried: outcome.candidatesTried
62763
+ };
62764
+ }
62765
+ var MAX_REPORT_BYTES = 64 * 1024;
62766
+ async function appendSessionDbRecoveryRecord(record, path, fileOps, log) {
62767
+ try {
62768
+ const line = Buffer.from(`${JSON.stringify(record)}
62769
+ `);
62770
+ const existing = await fileOps.stat(path);
62771
+ if ((existing?.size ?? 0) + line.length > MAX_REPORT_BYTES) {
62772
+ log(
62773
+ `WARNING: session-DB recovery report at ${path} reached its ${MAX_REPORT_BYTES}-byte limit`
62774
+ );
62775
+ return;
62776
+ }
62777
+ await fileOps.mkdirp(dirname4(path));
62778
+ await fileOps.appendFile(path, line);
62779
+ } catch (error) {
62780
+ log(
62781
+ `WARNING: could not record the session-DB recovery outcome for the runner's activity log at ${path}: ${describeError(error)}`
62782
+ );
62783
+ }
62784
+ }
62785
+
62536
62786
  // src/session-db-verify.ts
62537
62787
  init_esm_shims();
62538
62788
 
@@ -62586,7 +62836,11 @@ async function verifySessionDb(params) {
62586
62836
  log(
62587
62837
  `WARNING: SESSION-DB-INTEGRITY-FAILED: opencode.db at ${dbPath} failed its integrity check: ${result.detail}`
62588
62838
  );
62589
- return walkBack(params);
62839
+ const decision = await walkBack(params);
62840
+ if (decision.kind !== "exhausted") return decision;
62841
+ const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(params.store, config.prefix, log);
62842
+ const localDb = separation.kind === "notConfigured" || separation.kind === "alreadyEmpty" || separation.kind === "quarantined" ? await discardCorruptDb(fileOps, dbPath, log) : { kind: "retained", reason: "separationUnproven" };
62843
+ return { ...decision, separation, localDb };
62590
62844
  }
62591
62845
  async function clearScratch(fileOps, scratch) {
62592
62846
  await fileOps.remove(scratch);
@@ -62594,15 +62848,32 @@ async function clearScratch(fileOps, scratch) {
62594
62848
  await fileOps.remove(`${scratch}-shm`);
62595
62849
  }
62596
62850
  async function discardCorruptDb(fileOps, dbPath, log) {
62597
- for (const path of [dbPath, `${dbPath}-wal`, `${dbPath}-shm`]) {
62851
+ const paths = [dbPath, `${dbPath}-wal`, `${dbPath}-shm`];
62852
+ const errors = [];
62853
+ for (const path of paths) {
62598
62854
  try {
62599
62855
  await fileOps.remove(path);
62600
62856
  } catch (error) {
62601
- log(
62602
- `WARNING: could not remove corrupt local session-DB debris at ${path}: ${describeError(error)}`
62603
- );
62857
+ errors.push(`${path}: ${describeError(error)}`);
62604
62858
  }
62605
62859
  }
62860
+ const survived = [];
62861
+ const removed = [];
62862
+ for (const path of paths) {
62863
+ try {
62864
+ if (await fileOps.stat(path) === null) removed.push(path);
62865
+ else survived.push(path);
62866
+ } catch (error) {
62867
+ survived.push(path);
62868
+ errors.push(`${path}: could not prove removal: ${describeError(error)}`);
62869
+ }
62870
+ }
62871
+ if (errors.length === 0 && survived.length === 0) return { kind: "discarded" };
62872
+ const detail = errors.length === 0 ? "paths remained after removal" : errors.join("; ");
62873
+ log(
62874
+ `WARNING: SESSION-DB-LOCAL-DISCARD-FAILED: removed=${removed.join(", ") || "(none)"} survived=${survived.join(", ") || "(none)"} errors=${detail}; boot is stopping so no process opens or writes what remains.`
62875
+ );
62876
+ return { kind: "discardFailed", removed, survived, detail };
62606
62877
  }
62607
62878
  async function adopt(fileOps, scratch, dbPath, log, txid, candidatesTried, bytes) {
62608
62879
  await fileOps.remove(`${dbPath}-wal`);
@@ -62624,12 +62895,10 @@ async function walkBack(params) {
62624
62895
  log(
62625
62896
  `WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED: ${listing.stdout === null ? `litestream ltx exited non-zero: ${listing.detail}` : "litestream ltx succeeded but its listing could not be parsed"}`
62626
62897
  );
62627
- await discardCorruptDb(fileOps, dbPath, log);
62628
62898
  return { kind: "exhausted", reason: "enumerationFailed", candidatesTried: 0 };
62629
62899
  }
62630
62900
  const candidates = walkbackCandidates(points);
62631
62901
  if (candidates.length === 0) {
62632
- await discardCorruptDb(fileOps, dbPath, log);
62633
62902
  return { kind: "exhausted", reason: "noCandidates", candidatesTried: 0 };
62634
62903
  }
62635
62904
  const deadline = now() + budgetSeconds * 1e3;
@@ -62660,7 +62929,6 @@ async function walkBack(params) {
62660
62929
  return { kind: "walkedBack", txid, bytes, candidatesTried };
62661
62930
  }
62662
62931
  await clearScratch(fileOps, scratch);
62663
- await discardCorruptDb(fileOps, dbPath, log);
62664
62932
  return {
62665
62933
  kind: "exhausted",
62666
62934
  reason: index < candidates.length ? "budgetExhausted" : "allFailed",
@@ -62668,6 +62936,49 @@ async function walkBack(params) {
62668
62936
  };
62669
62937
  }
62670
62938
 
62939
+ // src/startup-failure-report.ts
62940
+ init_esm_shims();
62941
+ var STARTUP_FAILURE_REPORT_TIMEOUT_MS = 5e3;
62942
+ var MAX_RESPONSE_BODY_LENGTH = 1e3;
62943
+ async function reportStartupFailure(report, config, deps) {
62944
+ if (config.apiUrl === null || config.runnerKey === null) {
62945
+ deps.log(
62946
+ `Startup failure was not reported to Evident because ${config.apiUrl === null ? "EVIDENT_API_URL" : "a runner key"} is unavailable.`
62947
+ );
62948
+ return;
62949
+ }
62950
+ const endpoint = `${config.apiUrl.replace(/\/+$/, "").replace(/\/v1$/, "")}/v1/runners/self/startup-failure`;
62951
+ try {
62952
+ const response = await deps.fetchImpl(endpoint, {
62953
+ method: "POST",
62954
+ headers: {
62955
+ Authorization: `SandboxKey ${config.runnerKey}`,
62956
+ "Content-Type": "application/json"
62957
+ },
62958
+ body: JSON.stringify({
62959
+ classification: report.classification,
62960
+ stage: report.stage,
62961
+ litestream_exit_code: report.litestreamExitCode,
62962
+ attempt: report.attempt,
62963
+ restore_points_tried: report.restorePointsTried,
62964
+ report_id: deps.randomUUID()
62965
+ }),
62966
+ // The container is about to exit, so a long wait only delays ECS replacement.
62967
+ signal: AbortSignal.timeout(STARTUP_FAILURE_REPORT_TIMEOUT_MS)
62968
+ });
62969
+ if (!response.ok) {
62970
+ const body = (await response.text()).slice(0, MAX_RESPONSE_BODY_LENGTH);
62971
+ deps.log(
62972
+ `WARNING: fatal session-DB startup failure report was rejected (${response.status}): ${body}`
62973
+ );
62974
+ }
62975
+ } catch (error) {
62976
+ deps.log(
62977
+ `WARNING: could not report the fatal session-DB startup failure to Evident (${describeError(error)}); the failure is still shown above and in this machine's logs.`
62978
+ );
62979
+ }
62980
+ }
62981
+
62671
62982
  // src/self-stop.ts
62672
62983
  init_esm_shims();
62673
62984
  function isComplete(credentials) {
@@ -62845,6 +63156,7 @@ var EXIT_SESSION_DB_FATAL = 30;
62845
63156
  var EXIT_SESSION_DB_UNUSABLE = 31;
62846
63157
  var EXIT_SESSION_DB_RETRY = 32;
62847
63158
  var EXIT_SESSION_DB_UNVERIFIABLE = 33;
63159
+ var EXIT_SESSION_DB_UNSEPARATED = 34;
62848
63160
  var STATE_SUFFIX = ".synchash";
62849
63161
  var COMMANDS = shell_contract_default.commands;
62850
63162
  var USAGE = `Usage: runner-synchroniser <command> [args]
@@ -62857,16 +63169,18 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
62857
63169
  self-stop scale this agent's own ECS service to 0; exit 0 only
62858
63170
  when desiredCount is confirmed 0, ${EXIT_KEEP_TASK} to keep the task
62859
63171
  session-db-classify <litestream-restore-exit-code> <attempt>
62860
- [--on-unusable-replica=<prune|leave|clear|crash>]
63172
+ [--on-unusable-replica=<prune|leave|clear|crash>] [--recovery-occurred]
63173
+ [--fresh-db-fallback]
62861
63174
  classify a just-run \`litestream restore\` of opencode.db;
62862
63175
  exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
62863
- and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
62864
- fresh backup chain), ${EXIT_SESSION_DB_FATAL} fatal
63176
+ and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
63177
+ replicates into the existing prefix), ${EXIT_SESSION_DB_FATAL} fatal
62865
63178
  session-db-verify <litestream-config-path>
62866
63179
  verify opencode.db's integrity at boot, walking back through
62867
63180
  litestream's retained restore points if it's corrupt; exit 0
62868
63181
  healthy/skipped/walked-back, ${EXIT_SESSION_DB_UNVERIFIABLE} every
62869
- candidate was exhausted (booted fresh)
63182
+ candidate was exhausted (booted fresh), ${EXIT_SESSION_DB_UNSEPARATED}
63183
+ separation/local disposal could not be proven
62870
63184
  `;
62871
63185
  function shellQuote(value) {
62872
63186
  return `'${value.replaceAll("'", `'\\''`)}'`;
@@ -62887,16 +63201,26 @@ function parseStoreName(value) {
62887
63201
  return value === "claude" || value === "opencode" ? value : null;
62888
63202
  }
62889
63203
  var ON_UNUSABLE_REPLICA_FLAG = "--on-unusable-replica=";
63204
+ var RECOVERY_OCCURRED_FLAG = "--recovery-occurred";
63205
+ var FRESH_DB_FALLBACK_FLAG = "--fresh-db-fallback";
62890
63206
  function parseNonNegativeInt(value) {
62891
63207
  return /^\d+$/.test(value) ? Number.parseInt(value, 10) : null;
62892
63208
  }
62893
63209
  function parseSessionDbClassifyArgs(args) {
62894
63210
  let strategyRaw;
63211
+ let recoveryOccurred = false;
63212
+ let freshDbFallback = false;
62895
63213
  const positional = [];
62896
63214
  for (const arg of args) {
62897
63215
  if (arg.startsWith(ON_UNUSABLE_REPLICA_FLAG)) {
62898
63216
  if (strategyRaw !== void 0) return null;
62899
63217
  strategyRaw = arg.slice(ON_UNUSABLE_REPLICA_FLAG.length);
63218
+ } else if (arg === RECOVERY_OCCURRED_FLAG) {
63219
+ if (recoveryOccurred) return null;
63220
+ recoveryOccurred = true;
63221
+ } else if (arg === FRESH_DB_FALLBACK_FLAG) {
63222
+ if (freshDbFallback) return null;
63223
+ freshDbFallback = true;
62900
63224
  } else if (arg.startsWith("--")) {
62901
63225
  return null;
62902
63226
  } else {
@@ -62908,7 +63232,7 @@ function parseSessionDbClassifyArgs(args) {
62908
63232
  const attempt = parseNonNegativeInt(positional[1]);
62909
63233
  const strategy = parseUnusableReplicaStrategy(strategyRaw);
62910
63234
  if (exitCode === null || attempt === null || attempt < 1 || strategy === null) return null;
62911
- return { exitCode, attempt, strategy };
63235
+ return { exitCode, attempt, strategy, recoveryOccurred, freshDbFallback };
62912
63236
  }
62913
63237
  function renderEnv(config) {
62914
63238
  return [
@@ -62996,6 +63320,30 @@ async function commandSessionDbClassify(args, config, deps) {
62996
63320
  log
62997
63321
  });
62998
63322
  log(describeSessionDbClassification(outcome, { bucket: config.bucket, region: config.region }));
63323
+ if (outcome.kind === "fatal") {
63324
+ await reportStartupFailure(
63325
+ {
63326
+ classification: outcome.cause === "misconfig" ? "session_db_restore_misconfigured" : "session_db_restore_crash_chosen",
63327
+ stage: "restore",
63328
+ litestreamExitCode: args.exitCode,
63329
+ attempt: args.attempt,
63330
+ restorePointsTried: null
63331
+ },
63332
+ config,
63333
+ { fetchImpl: deps.fetchImpl ?? fetch, randomUUID: deps.randomUUID ?? crypto.randomUUID, log }
63334
+ );
63335
+ return sessionDbExitCode(outcome);
63336
+ }
63337
+ const record = recordRestoreOutcome(
63338
+ outcome,
63339
+ (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString(),
63340
+ args.exitCode,
63341
+ args.attempt,
63342
+ args.recoveryOccurred,
63343
+ args.freshDbFallback
63344
+ );
63345
+ if (record !== null)
63346
+ await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
62999
63347
  return sessionDbExitCode(outcome);
63000
63348
  }
63001
63349
  async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
@@ -63005,7 +63353,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
63005
63353
  config.opencodeDbPath
63006
63354
  );
63007
63355
  const outcome = await verifySessionDb({
63008
- config: { opencodeDbPath: config.opencodeDbPath },
63356
+ config: { opencodeDbPath: config.opencodeDbPath, prefix: config.prefix },
63357
+ store: (deps.objectStoreFor ?? s3ObjectStoreFor)(config),
63009
63358
  fileOps,
63010
63359
  sqlite: (deps.sqliteIntegrityFor ?? nodeSqliteIntegrityFor)(),
63011
63360
  litestream,
@@ -63014,6 +63363,23 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
63014
63363
  budgetSeconds: config.walkbackBudgetSeconds
63015
63364
  });
63016
63365
  log(describeSessionDbVerification(outcome));
63366
+ if (outcome.kind === "exhausted" && sessionDbVerifyExitCode(outcome) === 34) {
63367
+ await reportStartupFailure(
63368
+ {
63369
+ classification: outcome.localDb.kind === "discardFailed" ? "session_db_local_discard_failed" : "session_db_separation_unproven",
63370
+ stage: "verify",
63371
+ litestreamExitCode: null,
63372
+ attempt: null,
63373
+ restorePointsTried: outcome.candidatesTried
63374
+ },
63375
+ config,
63376
+ { fetchImpl: deps.fetchImpl ?? fetch, randomUUID: deps.randomUUID ?? crypto.randomUUID, log }
63377
+ );
63378
+ return sessionDbVerifyExitCode(outcome);
63379
+ }
63380
+ const record = recordVerifyOutcome(outcome, (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString());
63381
+ if (record !== null)
63382
+ await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
63017
63383
  return sessionDbVerifyExitCode(outcome);
63018
63384
  }
63019
63385
  async function main(argv, deps) {
@@ -63055,7 +63421,7 @@ ${USAGE}`);
63055
63421
  const args = parseSessionDbClassifyArgs(rest);
63056
63422
  if (args === null) {
63057
63423
  log(
63058
- `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>.
63424
+ `'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>, --recovery-occurred, and --fresh-db-fallback.
63059
63425
  ${USAGE}`
63060
63426
  );
63061
63427
  return EXIT_USAGE;
@@ -63111,6 +63477,7 @@ export {
63111
63477
  EXIT_NOT_READY,
63112
63478
  EXIT_SESSION_DB_FATAL,
63113
63479
  EXIT_SESSION_DB_RETRY,
63480
+ EXIT_SESSION_DB_UNSEPARATED,
63114
63481
  EXIT_SESSION_DB_UNUSABLE,
63115
63482
  EXIT_SESSION_DB_UNVERIFIABLE,
63116
63483
  EXIT_USAGE,
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@evident-ai/runner-synchroniser",
3
- "version": "3.4.0",
3
+ "version": "3.4.1-dev.1549524",
4
4
  "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
5
5
  "type": "module",
6
6
  "main": "./dist/cli.js",
@@ -26,7 +26,7 @@
26
26
  "tsup": "^8.3.5",
27
27
  "typescript": "^5.7.2",
28
28
  "vitest": "^3.1.1",
29
- "runner-image": "workspace:*"
29
+ "runner-fargate-image": "workspace:*"
30
30
  },
31
31
  "engines": {
32
32
  "node": ">=22.13.0"
@@ -42,7 +42,7 @@
42
42
  "repository": {
43
43
  "type": "git",
44
44
  "url": "https://github.com/sroze/evident.git",
45
- "directory": "packages/runner-synchroniser"
45
+ "directory": "runner/synchroniser"
46
46
  },
47
47
  "homepage": "https://evident.run",
48
48
  "author": "Evident <hello@evident.run>",