@evident-ai/runner-synchroniser 3.4.1-dev.74a16b2 → 3.4.1-dev.77d5cb6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +44 -17
- package/dist/cli.js +345 -51
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -59,8 +59,8 @@ build time.
|
|
|
59
59
|
| `sync-once <claude\|opencode>` | — | `0` = ran; non-zero = tool broken |
|
|
60
60
|
| `model-auth-ready` | — | `0` = ready, `10` = not ready; other = tool broken |
|
|
61
61
|
| `self-stop` | — | `0` = stopped, `20` = keep the task; other = tool broken |
|
|
62
|
-
| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh,
|
|
63
|
-
| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` =
|
|
62
|
+
| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, replicates into the existing prefix), `30` = fatal; other = usage/tool broken |
|
|
63
|
+
| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = exhausted after verified separation and local disposal, `34` = separation or disposal could not be proven; other = usage/tool broken |
|
|
64
64
|
|
|
65
65
|
`self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
|
|
66
66
|
(see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
|
|
@@ -75,7 +75,16 @@ saved query.
|
|
|
75
75
|
|
|
76
76
|
### `session-db-classify`
|
|
77
77
|
|
|
78
|
-
Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `
|
|
78
|
+
Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `runner/docker-images/fargate/README.md`'s [Strategy/What/Cost table](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
|
|
79
|
+
|
|
80
|
+
#### Activity recovery report
|
|
81
|
+
|
|
82
|
+
Reportable recovery outcomes append one versioned JSON line to the session-DB recovery
|
|
83
|
+
report. `evident run` drains that file once after authentication and records the outcome in
|
|
84
|
+
the runner activity log. Healthy boots do not write a report. An initial restore failure writes
|
|
85
|
+
a warning `restore_retried` record before the retry's result is known. Writing is best-effort
|
|
86
|
+
and never changes this command's exit code. The CLI reader owns this record contract: add a new
|
|
87
|
+
`v` rather than repurposing a field.
|
|
79
88
|
|
|
80
89
|
#### Positionals
|
|
81
90
|
|
|
@@ -102,6 +111,14 @@ Implementation-facing reference for the 7th command: it decides what a just-run
|
|
|
102
111
|
- Any other `--` token is rejected.
|
|
103
112
|
- Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
|
|
104
113
|
|
|
114
|
+
#### Recovery-report flags
|
|
115
|
+
|
|
116
|
+
`--recovery-occurred` marks a later successful restore after this boot already recovered a
|
|
117
|
+
replica, so an empty result is reported as a fresh session database. `--fresh-db-fallback`
|
|
118
|
+
marks a runtime that intentionally has no retry budget and will therefore boot fresh after this
|
|
119
|
+
otherwise-transient failure. Each flag may appear once, does not change the command's exit code,
|
|
120
|
+
and is rejected when repeated.
|
|
121
|
+
|
|
105
122
|
#### Outcome → exit code
|
|
106
123
|
|
|
107
124
|
| Outcome | Code | When |
|
|
@@ -196,11 +213,11 @@ what it is.
|
|
|
196
213
|
#### The four `--on-unusable-replica` strategies
|
|
197
214
|
|
|
198
215
|
- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
|
|
199
|
-
- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot".
|
|
216
|
+
- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". The fresh local DB replicates into the existing prefix; no new chain is started. Upside: the runner still comes online without a crash loop.
|
|
200
217
|
- **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
|
|
201
218
|
- **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
|
|
202
219
|
|
|
203
|
-
#### The measured real-world key layout (litestream 0.5.13)
|
|
220
|
+
#### The measured real-world key layout (litestream 0.5.13 historical sample)
|
|
204
221
|
|
|
205
222
|
`parseLtxKey` (`src/replica-keys.ts`) expects
|
|
206
223
|
`<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
|
|
@@ -218,7 +235,7 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
|
|
|
218
235
|
(`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
|
|
219
236
|
and it can never repair corruption at a higher compaction level.
|
|
220
237
|
|
|
221
|
-
`EVIDENT_ON_UNUSABLE_REPLICA` is a `
|
|
238
|
+
`EVIDENT_ON_UNUSABLE_REPLICA` is a `runner/docker-images/fargate` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [the Fargate image README](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
|
|
222
239
|
|
|
223
240
|
See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
|
|
224
241
|
the behavioural anchor (31 comes online, 30 does not).
|
|
@@ -245,9 +262,12 @@ nothing had ever run a real `PRAGMA integrity_check`).
|
|
|
245
262
|
each into a scratch path and integrity-checking it there. The first one that passes
|
|
246
263
|
is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
|
|
247
264
|
3. If nothing retained passes (or there's nothing to try, or the search runs out of
|
|
248
|
-
time/points), the
|
|
249
|
-
|
|
250
|
-
|
|
265
|
+
time/points), the active replica prefix is separated first: its history is moved aside,
|
|
266
|
+
then the prefix is re-listed as empty before the corrupt local DB and sidecars are
|
|
267
|
+
discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
|
|
268
|
+
OpenCode or Litestream starts; the backup history remains readable at its quarantine
|
|
269
|
+
destination or original key, and surviving local files remain in place without a process
|
|
270
|
+
opening or writing them.
|
|
251
271
|
|
|
252
272
|
A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
|
|
253
273
|
prefix. litestream re-bases to the replica's high-water mark and continues the txid
|
|
@@ -262,10 +282,8 @@ against S3 at the ~1 GB scale a production replica can reach.)
|
|
|
262
282
|
| `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
|
|
263
283
|
| `healthy` | `0` | the local DB passed its integrity check as-is |
|
|
264
284
|
| `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
|
|
265
|
-
| `exhausted` (`noCandidates`) | `33` | the
|
|
266
|
-
| `exhausted`
|
|
267
|
-
| `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
|
|
268
|
-
| `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
|
|
285
|
+
| `exhausted` (`noCandidates`, `allFailed`, `budgetExhausted`, or `enumerationFailed`) | `33` | the active prefix was proven separated (or persistence is disabled) **and** the corrupt local DB and sidecars were proven discarded |
|
|
286
|
+
| `exhausted` | `34` | either the replica separation or local discard could not be proven; the boot stops before OpenCode and Litestream start |
|
|
269
287
|
|
|
270
288
|
`healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
|
|
271
289
|
afterwards — start `litestream replicate` — is identical either way, so a further split
|
|
@@ -284,7 +302,10 @@ before looking at everything" would report the wrong conclusion to the operator.
|
|
|
284
302
|
| `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
|
|
285
303
|
| `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
|
|
286
304
|
| `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
|
|
287
|
-
| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed
|
|
305
|
+
| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed; a fresh boot follows only when both separation and local disposal were proven | `candidatesTried` (in prose, not `k=v`) |
|
|
306
|
+
| `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
|
|
307
|
+
| `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
|
|
308
|
+
| `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
|
|
288
309
|
|
|
289
310
|
The first four fire only while the walkback loop runs; the last three are always the
|
|
290
311
|
one message this command logs as its own final word on the outcome — `entrypoint.sh`
|
|
@@ -316,10 +337,15 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
|
|
|
316
337
|
|
|
317
338
|
`session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
|
|
318
339
|
|
|
340
|
+
`session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
|
|
341
|
+
history. A refused boot records `session_db_boot_refused`, but `evident run` has not started to
|
|
342
|
+
drain it, so operators see that refusal in boot logs and the restart loop. The report does not
|
|
343
|
+
alter the command's exit code.
|
|
344
|
+
|
|
319
345
|
`src/shell-contract.json` is the machine-checked source of truth for the command list, and
|
|
320
346
|
`shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
|
|
321
|
-
`
|
|
322
|
-
`
|
|
347
|
+
`runner/docker-images/fargate/entrypoint.sh` and the MicroVM's
|
|
348
|
+
`runner/docker-images/microvm/hooks` (#608), discovered by grep so a third one
|
|
323
349
|
cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
|
|
324
350
|
`entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
|
|
325
351
|
a broken tool; and take each answer code the commands it calls can return **silently and by
|
|
@@ -344,11 +370,12 @@ key, so they never appear in `env`'s output or `litestream.yml`.
|
|
|
344
370
|
| `CREDS_SYNC_INTERVAL` | Seconds between sync ticks, reported by `env` for the caller's loop. | Falls back to `60`; also on unparseable or non-positive values. |
|
|
345
371
|
| `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS` | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values. |
|
|
346
372
|
| `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop. | Falls back to `180`; also on unparseable or non-positive values. |
|
|
373
|
+
| `EVIDENT_SESSION_DB_RECOVERY_REPORT` | JSONL report drained once into runner activity after authentication. | `$HOME/.local/state/evident/session-db-recovery.jsonl`; blank values use the default. |
|
|
347
374
|
| `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
|
|
348
375
|
| `CLUSTER` † / `SERVICE` † | ECS cluster/service `self-stop` scales to `desiredCount=0`. | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
|
|
349
376
|
| `EVIDENT_SELFSTOP_ROLE_ARN` † | Role `self-stop` assumes for its ECS calls. | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
|
|
350
377
|
|
|
351
|
-
`
|
|
378
|
+
`runner/docker-images/fargate/README.md` documents these same three from the deployment side —
|
|
352
379
|
keep them in sync.
|
|
353
380
|
|
|
354
381
|
Requiring **both** `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` (never just one) means
|
package/dist/cli.js
CHANGED
|
@@ -51907,7 +51907,8 @@ function positiveIntOr(value, defaultValue) {
|
|
|
51907
51907
|
return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue;
|
|
51908
51908
|
}
|
|
51909
51909
|
function nonEmpty(value) {
|
|
51910
|
-
|
|
51910
|
+
const trimmed = value?.trim();
|
|
51911
|
+
return trimmed ? trimmed : null;
|
|
51911
51912
|
}
|
|
51912
51913
|
function resolveConfig(env4) {
|
|
51913
51914
|
const homeDir = nonEmpty(env4.HOME);
|
|
@@ -51930,6 +51931,7 @@ function resolveConfig(env4) {
|
|
|
51930
51931
|
key: persistenceEnabled ? `${prefix}/opencode/auth.json` : null
|
|
51931
51932
|
},
|
|
51932
51933
|
opencodeDbPath: `${homeDir}/.local/share/opencode/opencode.db`,
|
|
51934
|
+
sessionDbRecoveryReportPath: nonEmpty(env4.EVIDENT_SESSION_DB_RECOVERY_REPORT) ?? `${homeDir}/.local/state/evident/session-db-recovery.jsonl`,
|
|
51933
51935
|
bucket: persistenceEnabled ? bucket : null,
|
|
51934
51936
|
prefix: persistenceEnabled ? prefix : null,
|
|
51935
51937
|
region: nonEmpty(env4.AWS_REGION),
|
|
@@ -51998,7 +52000,7 @@ function sessionDbExitCode(outcome) {
|
|
|
51998
52000
|
// re-run `litestream restore` and ask again
|
|
51999
52001
|
case "unusableReplica":
|
|
52000
52002
|
return 31;
|
|
52001
|
-
// booted with a fresh DB; the shell still
|
|
52003
|
+
// booted with a fresh DB; the shell still replicates into the existing prefix
|
|
52002
52004
|
case "fatal":
|
|
52003
52005
|
return 30;
|
|
52004
52006
|
// shared by a genuine misconfig and a deliberate `crash` choice
|
|
@@ -52007,20 +52009,20 @@ function sessionDbExitCode(outcome) {
|
|
|
52007
52009
|
}
|
|
52008
52010
|
}
|
|
52009
52011
|
function describeUnusableReplica(reason) {
|
|
52010
|
-
const
|
|
52012
|
+
const base2 = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
|
|
52011
52013
|
switch (reason) {
|
|
52012
52014
|
case "strategy":
|
|
52013
|
-
return `${
|
|
52015
|
+
return `${base2} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
|
|
52014
52016
|
case "attemptsExhausted":
|
|
52015
|
-
return `${
|
|
52017
|
+
return `${base2} (recovery did not make the replica usable after retrying; giving up.)`;
|
|
52016
52018
|
case "layoutMismatch":
|
|
52017
|
-
return `${
|
|
52019
|
+
return `${base2} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
|
|
52018
52020
|
case "noL0Present":
|
|
52019
|
-
return `${
|
|
52021
|
+
return `${base2} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
|
|
52020
52022
|
case "recoveryFailed":
|
|
52021
|
-
return `${
|
|
52023
|
+
return `${base2} (the recovery delete itself failed; see the warning above for the S3 error.)`;
|
|
52022
52024
|
case "targetHealthy":
|
|
52023
|
-
return `${
|
|
52025
|
+
return `${base2} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
|
|
52024
52026
|
default:
|
|
52025
52027
|
return assertNeverUnusableReplicaReason(reason);
|
|
52026
52028
|
}
|
|
@@ -52061,7 +52063,7 @@ function describeSessionDbClassification(outcome, config) {
|
|
|
52061
52063
|
case "recovered": {
|
|
52062
52064
|
const objects = outcome.deleted.length === 0 ? "no objects (see the warnings above for what failed)" : `${outcome.deleted.length} object${outcome.deleted.length === 1 ? "" : "s"} (${outcome.deleted.join(", ")})`;
|
|
52063
52065
|
if (outcome.strategy === "quarantine") {
|
|
52064
|
-
return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside
|
|
52066
|
+
return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside and remains readable at the quarantine destination \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
|
|
52065
52067
|
}
|
|
52066
52068
|
return `Attempted opencode.db replica recovery via --on-unusable-replica=${outcome.strategy}, deleting ${objects}; retrying the restore.` + (outcome.strategy === "clear" ? " Prior session history is lost." : "");
|
|
52067
52069
|
}
|
|
@@ -52082,6 +52084,12 @@ function assertNeverSessionDbVerifySkippedLocalDb(value) {
|
|
|
52082
52084
|
function assertNeverSessionDbVerifyExhaustedReason(value) {
|
|
52083
52085
|
throw new Error(`Unhandled session-DB verify exhausted reason: ${JSON.stringify(value)}`);
|
|
52084
52086
|
}
|
|
52087
|
+
function assertNeverSessionDbVerifySeparation(value) {
|
|
52088
|
+
throw new Error(`Unhandled session-DB verify separation: ${JSON.stringify(value)}`);
|
|
52089
|
+
}
|
|
52090
|
+
function assertNeverSessionDbVerifyLocalDb(value) {
|
|
52091
|
+
throw new Error(`Unhandled session-DB verify local-DB disposal: ${JSON.stringify(value)}`);
|
|
52092
|
+
}
|
|
52085
52093
|
function sessionDbVerifyExitCode(outcome) {
|
|
52086
52094
|
switch (outcome.kind) {
|
|
52087
52095
|
case "skipped":
|
|
@@ -52089,8 +52097,26 @@ function sessionDbVerifyExitCode(outcome) {
|
|
|
52089
52097
|
case "walkedBack":
|
|
52090
52098
|
return 0;
|
|
52091
52099
|
case "exhausted":
|
|
52092
|
-
|
|
52093
|
-
|
|
52100
|
+
switch (outcome.separation.kind) {
|
|
52101
|
+
case "notConfigured":
|
|
52102
|
+
case "alreadyEmpty":
|
|
52103
|
+
case "quarantined":
|
|
52104
|
+
break;
|
|
52105
|
+
case "incomplete":
|
|
52106
|
+
case "unreachable":
|
|
52107
|
+
return 34;
|
|
52108
|
+
default:
|
|
52109
|
+
return assertNeverSessionDbVerifySeparation(outcome.separation);
|
|
52110
|
+
}
|
|
52111
|
+
switch (outcome.localDb.kind) {
|
|
52112
|
+
case "discarded":
|
|
52113
|
+
return 33;
|
|
52114
|
+
case "retained":
|
|
52115
|
+
case "discardFailed":
|
|
52116
|
+
return 34;
|
|
52117
|
+
default:
|
|
52118
|
+
return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
|
|
52119
|
+
}
|
|
52094
52120
|
default:
|
|
52095
52121
|
return assertNeverSessionDbVerify(outcome);
|
|
52096
52122
|
}
|
|
@@ -52107,18 +52133,56 @@ function describeSessionDbVerifySkipped(localDb) {
|
|
|
52107
52133
|
}
|
|
52108
52134
|
function describeSessionDbVerifyExhausted(outcome) {
|
|
52109
52135
|
const tried = `tried ${outcome.candidatesTried} restore point${outcome.candidatesTried === 1 ? "" : "s"}`;
|
|
52136
|
+
let reason;
|
|
52110
52137
|
switch (outcome.reason) {
|
|
52111
52138
|
case "enumerationFailed":
|
|
52112
|
-
|
|
52139
|
+
reason = `opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried})`;
|
|
52140
|
+
break;
|
|
52113
52141
|
case "noCandidates":
|
|
52114
|
-
|
|
52142
|
+
reason = `opencode.db failed its integrity check and no older restore point is retained (${tried})`;
|
|
52143
|
+
break;
|
|
52115
52144
|
case "allFailed":
|
|
52116
|
-
|
|
52145
|
+
reason = `opencode.db failed its integrity check and every retained restore point also failed (${tried})`;
|
|
52146
|
+
break;
|
|
52117
52147
|
case "budgetExhausted":
|
|
52118
|
-
|
|
52148
|
+
reason = `opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try`;
|
|
52149
|
+
break;
|
|
52119
52150
|
default:
|
|
52120
52151
|
return assertNeverSessionDbVerifyExhaustedReason(outcome.reason);
|
|
52121
52152
|
}
|
|
52153
|
+
let separation;
|
|
52154
|
+
switch (outcome.separation.kind) {
|
|
52155
|
+
case "quarantined":
|
|
52156
|
+
separation = ` The active replica prefix was verified empty; moved history remains readable at ${outcome.separation.destination}.`;
|
|
52157
|
+
break;
|
|
52158
|
+
case "alreadyEmpty":
|
|
52159
|
+
separation = " The active replica prefix was already empty.";
|
|
52160
|
+
break;
|
|
52161
|
+
case "notConfigured":
|
|
52162
|
+
separation = " Persistence is disabled, so there is no replica to separate.";
|
|
52163
|
+
break;
|
|
52164
|
+
case "incomplete":
|
|
52165
|
+
case "unreachable":
|
|
52166
|
+
separation = ` Boot is stopping because replica separation could not be proven (${outcome.separation.detail}); backup history remains readable at its original key or quarantine destination.`;
|
|
52167
|
+
break;
|
|
52168
|
+
default:
|
|
52169
|
+
return assertNeverSessionDbVerifySeparation(outcome.separation);
|
|
52170
|
+
}
|
|
52171
|
+
let localDb;
|
|
52172
|
+
switch (outcome.localDb.kind) {
|
|
52173
|
+
case "discarded":
|
|
52174
|
+
localDb = " booting with a FRESH opencode.db.";
|
|
52175
|
+
break;
|
|
52176
|
+
case "retained":
|
|
52177
|
+
localDb = " The corrupt local database and sidecars were left untouched for investigation.";
|
|
52178
|
+
break;
|
|
52179
|
+
case "discardFailed":
|
|
52180
|
+
localDb = ` Boot is stopping because local disposal could not be proven; removed=${outcome.localDb.removed.join(", ") || "(none)"} survived=${outcome.localDb.survived.join(", ") || "(none)"}. See SESSION-DB-LOCAL-DISCARD-FAILED.`;
|
|
52181
|
+
break;
|
|
52182
|
+
default:
|
|
52183
|
+
return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
|
|
52184
|
+
}
|
|
52185
|
+
return `SESSION-DB-INTEGRITY-EXHAUSTED: ${reason}.${separation}${localDb}`;
|
|
52122
52186
|
}
|
|
52123
52187
|
function describeSessionDbVerification(outcome) {
|
|
52124
52188
|
switch (outcome.kind) {
|
|
@@ -52138,7 +52202,17 @@ function describeSessionDbVerification(outcome) {
|
|
|
52138
52202
|
// src/file-ops.ts
|
|
52139
52203
|
init_esm_shims();
|
|
52140
52204
|
import { createHash as createHash8 } from "node:crypto";
|
|
52141
|
-
import {
|
|
52205
|
+
import {
|
|
52206
|
+
appendFile,
|
|
52207
|
+
chmod,
|
|
52208
|
+
mkdir,
|
|
52209
|
+
open,
|
|
52210
|
+
readFile as readFile5,
|
|
52211
|
+
rename,
|
|
52212
|
+
rm,
|
|
52213
|
+
stat,
|
|
52214
|
+
writeFile as writeFile4
|
|
52215
|
+
} from "node:fs/promises";
|
|
52142
52216
|
var nodeFileOps = {
|
|
52143
52217
|
async mkdirp(dir) {
|
|
52144
52218
|
await mkdir(dir, { recursive: true });
|
|
@@ -52150,14 +52224,18 @@ var nodeFileOps = {
|
|
|
52150
52224
|
try {
|
|
52151
52225
|
const stats = await stat(path);
|
|
52152
52226
|
return { size: stats.size };
|
|
52153
|
-
} catch {
|
|
52154
|
-
return null;
|
|
52227
|
+
} catch (error) {
|
|
52228
|
+
if (error instanceof Error && "code" in error && error.code === "ENOENT") return null;
|
|
52229
|
+
throw error;
|
|
52155
52230
|
}
|
|
52156
52231
|
},
|
|
52157
52232
|
readFile: readFile5,
|
|
52158
52233
|
async writeFile(path, data) {
|
|
52159
52234
|
await writeFile4(path, data);
|
|
52160
52235
|
},
|
|
52236
|
+
async appendFile(path, data) {
|
|
52237
|
+
await appendFile(path, data);
|
|
52238
|
+
},
|
|
52161
52239
|
async createExclusive(path) {
|
|
52162
52240
|
const handle = await open(path, "wx", 384);
|
|
52163
52241
|
await handle.close();
|
|
@@ -52252,7 +52330,7 @@ function renderLitestreamConfig(config) {
|
|
|
52252
52330
|
# (no static keys here).
|
|
52253
52331
|
#
|
|
52254
52332
|
# INVARIANT (see README "Single-writer invariant"): exactly ONE writer per S3
|
|
52255
|
-
# prefix \u2014
|
|
52333
|
+
# prefix \u2014 we do not enable Litestream's server-side lease, so never run two
|
|
52256
52334
|
# replicators on one prefix. Only opencode.db is replicated.
|
|
52257
52335
|
|
|
52258
52336
|
dbs:
|
|
@@ -62324,7 +62402,7 @@ async function probeReplica(store, prefix, log) {
|
|
|
62324
62402
|
const objectCount = objects.length;
|
|
62325
62403
|
const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
|
|
62326
62404
|
logReplicaSize(root12, objectCount, totalBytes, log);
|
|
62327
|
-
return { ok: true,
|
|
62405
|
+
return { ok: true, objects };
|
|
62328
62406
|
} catch (error) {
|
|
62329
62407
|
const detail = describeError(error);
|
|
62330
62408
|
log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
|
|
@@ -62381,12 +62459,14 @@ async function pruneNewestL0(store, prefix, keys, log) {
|
|
|
62381
62459
|
function quarantineRoot(prefix, stamp) {
|
|
62382
62460
|
return `${prefix}/quarantine/opencode.db/${stamp}/`;
|
|
62383
62461
|
}
|
|
62384
|
-
async function quarantineReplica(store, prefix,
|
|
62462
|
+
async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
|
|
62385
62463
|
const destination = quarantineRoot(prefix, stamp);
|
|
62386
62464
|
const root12 = replicaDbPrefix(prefix);
|
|
62387
62465
|
const moved = [];
|
|
62388
62466
|
const failed = [];
|
|
62389
|
-
|
|
62467
|
+
let movedBytes = 0;
|
|
62468
|
+
for (const object of objects) {
|
|
62469
|
+
const { key } = object;
|
|
62390
62470
|
if (!isDeletableReplicaKey(prefix, key)) {
|
|
62391
62471
|
log(`WARNING: refusing to quarantine ${key}: outside the replica prefix.`);
|
|
62392
62472
|
continue;
|
|
@@ -62411,14 +62491,47 @@ async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__
|
|
|
62411
62491
|
continue;
|
|
62412
62492
|
}
|
|
62413
62493
|
moved.push(key);
|
|
62494
|
+
movedBytes += object.size;
|
|
62414
62495
|
}
|
|
62415
62496
|
logQuarantined(destination, moved, failed, log);
|
|
62416
|
-
return { destination, moved, failed };
|
|
62497
|
+
return { destination, moved, failed, movedBytes };
|
|
62417
62498
|
}
|
|
62418
62499
|
function logQuarantined(destination, moved, failed, log) {
|
|
62419
62500
|
log(
|
|
62420
|
-
`WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved)
|
|
62501
|
+
`WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (failed.length > 0 ? " The active replica prefix is not clear." : "") + `See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
|
|
62502
|
+
);
|
|
62503
|
+
}
|
|
62504
|
+
async function separateCorruptReplica(store, prefix, log, stamp) {
|
|
62505
|
+
const probe = await probeReplica(store, prefix, log);
|
|
62506
|
+
if (!probe.ok) return { kind: "unreachable", detail: probe.detail };
|
|
62507
|
+
const root12 = replicaDbPrefix(prefix);
|
|
62508
|
+
if (probe.objects.length === 0) {
|
|
62509
|
+
log(
|
|
62510
|
+
`INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was already empty; this boot starts a fresh backup chain.`
|
|
62511
|
+
);
|
|
62512
|
+
return { kind: "alreadyEmpty" };
|
|
62513
|
+
}
|
|
62514
|
+
const result = await quarantineReplica(store, prefix, probe.objects, log, stamp);
|
|
62515
|
+
try {
|
|
62516
|
+
const remaining = await store.list(root12);
|
|
62517
|
+
if (remaining.length > 0) {
|
|
62518
|
+
const detail = `${remaining.length} object(s) remain`;
|
|
62519
|
+
log(
|
|
62520
|
+
`WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: ${root12} still has ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
|
|
62521
|
+
);
|
|
62522
|
+
return { kind: "incomplete", detail };
|
|
62523
|
+
}
|
|
62524
|
+
} catch (error) {
|
|
62525
|
+
const detail = describeError(error);
|
|
62526
|
+
log(
|
|
62527
|
+
`WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: could not re-list ${root12}: ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
|
|
62528
|
+
);
|
|
62529
|
+
return { kind: "incomplete", detail };
|
|
62530
|
+
}
|
|
62531
|
+
log(
|
|
62532
|
+
`INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was verified empty by re-listing after moving ${result.moved.length} object(s) to ${result.destination}; this boot starts a fresh backup chain.`
|
|
62421
62533
|
);
|
|
62534
|
+
return { kind: "quarantined", destination: result.destination, moved: result.moved };
|
|
62422
62535
|
}
|
|
62423
62536
|
async function clearReplica(store, prefix, keys, log) {
|
|
62424
62537
|
const deleted = [];
|
|
@@ -62471,7 +62584,7 @@ async function decideSessionDbRestore(params) {
|
|
|
62471
62584
|
return { kind: "retryTransient" };
|
|
62472
62585
|
}
|
|
62473
62586
|
if (attempt >= 3) {
|
|
62474
|
-
return { kind: "unusableReplica", reason: "attemptsExhausted" };
|
|
62587
|
+
return { kind: "unusableReplica", reason: "attemptsExhausted", recovery: null };
|
|
62475
62588
|
}
|
|
62476
62589
|
const probe = await probeReplica(store, prefix, log);
|
|
62477
62590
|
if (!probe.ok) {
|
|
@@ -62481,7 +62594,12 @@ async function decideSessionDbRestore(params) {
|
|
|
62481
62594
|
detail: `could not verify the session-DB replica is reachable: ${probe.detail}`
|
|
62482
62595
|
};
|
|
62483
62596
|
}
|
|
62484
|
-
const keys = probe.
|
|
62597
|
+
const keys = probe.objects.map((object) => object.key);
|
|
62598
|
+
const recovery = {
|
|
62599
|
+
replicaObjects: probe.objects.length,
|
|
62600
|
+
replicaBytes: probe.objects.reduce((total, object) => total + object.size, 0),
|
|
62601
|
+
quarantine: null
|
|
62602
|
+
};
|
|
62485
62603
|
switch (strategy) {
|
|
62486
62604
|
case "crash":
|
|
62487
62605
|
return {
|
|
@@ -62490,30 +62608,43 @@ async function decideSessionDbRestore(params) {
|
|
|
62490
62608
|
detail: "the replica is unusable and --on-unusable-replica=crash was selected; no S3 object was touched"
|
|
62491
62609
|
};
|
|
62492
62610
|
case "leave":
|
|
62493
|
-
return { kind: "unusableReplica", reason: "strategy" };
|
|
62611
|
+
return { kind: "unusableReplica", reason: "strategy", recovery };
|
|
62494
62612
|
case "prune": {
|
|
62495
62613
|
const result = await pruneNewestL0(store, prefix, keys, log);
|
|
62496
62614
|
if (result.deleted === null) {
|
|
62497
62615
|
if (shouldEscalateToQuarantine(result.reason, prefix, keys)) {
|
|
62498
|
-
const quarantined = await quarantineReplica(store, prefix,
|
|
62616
|
+
const quarantined = await quarantineReplica(store, prefix, probe.objects, log);
|
|
62499
62617
|
if (quarantined.moved.length === 0) {
|
|
62500
|
-
return { kind: "unusableReplica", reason: "recoveryFailed" };
|
|
62618
|
+
return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
|
|
62501
62619
|
}
|
|
62502
|
-
return {
|
|
62620
|
+
return {
|
|
62621
|
+
kind: "recovered",
|
|
62622
|
+
strategy: "quarantine",
|
|
62623
|
+
deleted: quarantined.moved,
|
|
62624
|
+
recovery: {
|
|
62625
|
+
...recovery,
|
|
62626
|
+
quarantine: {
|
|
62627
|
+
destination: quarantined.destination,
|
|
62628
|
+
movedObjects: quarantined.moved.length,
|
|
62629
|
+
failedObjects: quarantined.failed.length,
|
|
62630
|
+
movedBytes: quarantined.movedBytes
|
|
62631
|
+
}
|
|
62632
|
+
}
|
|
62633
|
+
};
|
|
62503
62634
|
}
|
|
62504
62635
|
if (result.reason === "deleteFailed") {
|
|
62505
|
-
return { kind: "unusableReplica", reason: "recoveryFailed" };
|
|
62636
|
+
return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
|
|
62506
62637
|
}
|
|
62507
62638
|
if (result.reason === "unreadable") {
|
|
62508
|
-
return { kind: "unusableReplica", reason: "targetHealthy" };
|
|
62639
|
+
return { kind: "unusableReplica", reason: "targetHealthy", recovery };
|
|
62509
62640
|
}
|
|
62510
|
-
return { kind: "unusableReplica", reason: result.reason };
|
|
62641
|
+
return { kind: "unusableReplica", reason: result.reason, recovery };
|
|
62511
62642
|
}
|
|
62512
|
-
return { kind: "recovered", strategy: "prune", deleted: [result.deleted] };
|
|
62643
|
+
return { kind: "recovered", strategy: "prune", deleted: [result.deleted], recovery };
|
|
62513
62644
|
}
|
|
62514
62645
|
case "clear": {
|
|
62515
62646
|
const result = await clearReplica(store, prefix, keys, log);
|
|
62516
|
-
return { kind: "recovered", strategy: "clear", deleted: result.deleted };
|
|
62647
|
+
return { kind: "recovered", strategy: "clear", deleted: result.deleted, recovery };
|
|
62517
62648
|
}
|
|
62518
62649
|
default:
|
|
62519
62650
|
return assertNeverStrategy(strategy);
|
|
@@ -62533,6 +62664,123 @@ async function discardLocalDebris(fileOps, dbPath, log) {
|
|
|
62533
62664
|
}
|
|
62534
62665
|
}
|
|
62535
62666
|
|
|
62667
|
+
// src/session-db-recovery-report.ts
|
|
62668
|
+
init_esm_shims();
|
|
62669
|
+
import { dirname as dirname4 } from "node:path";
|
|
62670
|
+
function base(at2, stage) {
|
|
62671
|
+
return {
|
|
62672
|
+
v: 1,
|
|
62673
|
+
event: "session_db_recovery",
|
|
62674
|
+
at: at2,
|
|
62675
|
+
stage,
|
|
62676
|
+
litestream_exit_code: null,
|
|
62677
|
+
attempt: null,
|
|
62678
|
+
replica_objects: null,
|
|
62679
|
+
replica_bytes: null,
|
|
62680
|
+
quarantine_destination: null,
|
|
62681
|
+
quarantined_objects: null,
|
|
62682
|
+
quarantine_failed_objects: null,
|
|
62683
|
+
quarantined_bytes: null,
|
|
62684
|
+
verified_restore_point: null,
|
|
62685
|
+
restore_points_tried: null
|
|
62686
|
+
};
|
|
62687
|
+
}
|
|
62688
|
+
function recordRestoreOutcome(outcome, at2, exitCode, attempt, recoveryOccurred, freshDbFallback = false) {
|
|
62689
|
+
const record = { ...base(at2, "restore"), litestream_exit_code: exitCode, attempt };
|
|
62690
|
+
if (outcome.kind === "disabled" || outcome.kind === "restored") return null;
|
|
62691
|
+
if (outcome.kind === "retryTransient")
|
|
62692
|
+
return freshDbFallback ? {
|
|
62693
|
+
...record,
|
|
62694
|
+
outcome: "fresh_session_db",
|
|
62695
|
+
severity: "error",
|
|
62696
|
+
reason: "retry_budget_exhausted"
|
|
62697
|
+
} : {
|
|
62698
|
+
...record,
|
|
62699
|
+
outcome: "restore_retried",
|
|
62700
|
+
severity: "warning",
|
|
62701
|
+
reason: "initial_restore_failed"
|
|
62702
|
+
};
|
|
62703
|
+
if (outcome.kind === "noReplica") {
|
|
62704
|
+
return recoveryOccurred ? {
|
|
62705
|
+
...record,
|
|
62706
|
+
outcome: "fresh_session_db",
|
|
62707
|
+
severity: "error",
|
|
62708
|
+
reason: "no_replica_after_recovery"
|
|
62709
|
+
} : null;
|
|
62710
|
+
}
|
|
62711
|
+
if (outcome.kind === "fatal") return null;
|
|
62712
|
+
if (outcome.kind === "unusableReplica")
|
|
62713
|
+
return {
|
|
62714
|
+
...record,
|
|
62715
|
+
outcome: "fresh_session_db",
|
|
62716
|
+
severity: "error",
|
|
62717
|
+
reason: outcome.reason,
|
|
62718
|
+
replica_objects: outcome.recovery?.replicaObjects ?? null,
|
|
62719
|
+
replica_bytes: outcome.recovery?.replicaBytes ?? null
|
|
62720
|
+
};
|
|
62721
|
+
return {
|
|
62722
|
+
...record,
|
|
62723
|
+
outcome: "replica_recovered",
|
|
62724
|
+
severity: outcome.strategy === "clear" ? "error" : "warning",
|
|
62725
|
+
reason: outcome.strategy,
|
|
62726
|
+
replica_objects: outcome.recovery.replicaObjects,
|
|
62727
|
+
replica_bytes: outcome.recovery.replicaBytes,
|
|
62728
|
+
quarantine_destination: outcome.recovery.quarantine?.destination ?? null,
|
|
62729
|
+
quarantined_objects: outcome.recovery.quarantine?.movedObjects ?? null,
|
|
62730
|
+
quarantine_failed_objects: outcome.recovery.quarantine?.failedObjects ?? null,
|
|
62731
|
+
quarantined_bytes: outcome.recovery.quarantine?.movedBytes ?? null
|
|
62732
|
+
};
|
|
62733
|
+
}
|
|
62734
|
+
function recordVerifyOutcome(outcome, at2) {
|
|
62735
|
+
const record = base(at2, "verify");
|
|
62736
|
+
if (outcome.kind === "skipped" || outcome.kind === "healthy") return null;
|
|
62737
|
+
if (outcome.kind === "walkedBack")
|
|
62738
|
+
return {
|
|
62739
|
+
...record,
|
|
62740
|
+
outcome: "history_rolled_back",
|
|
62741
|
+
severity: "warning",
|
|
62742
|
+
reason: "walkback",
|
|
62743
|
+
verified_restore_point: outcome.txid,
|
|
62744
|
+
restore_points_tried: outcome.candidatesTried
|
|
62745
|
+
};
|
|
62746
|
+
if (sessionDbVerifyExitCode(outcome) === 34)
|
|
62747
|
+
return {
|
|
62748
|
+
...record,
|
|
62749
|
+
outcome: "session_db_boot_refused",
|
|
62750
|
+
severity: "error",
|
|
62751
|
+
reason: outcome.localDb.kind === "discardFailed" ? "local_discard_failed" : "replica_separation_unproven",
|
|
62752
|
+
restore_points_tried: outcome.candidatesTried,
|
|
62753
|
+
quarantine_destination: outcome.separation.kind === "quarantined" ? outcome.separation.destination : null
|
|
62754
|
+
};
|
|
62755
|
+
return {
|
|
62756
|
+
...record,
|
|
62757
|
+
outcome: "fresh_session_db",
|
|
62758
|
+
severity: "error",
|
|
62759
|
+
reason: outcome.reason,
|
|
62760
|
+
restore_points_tried: outcome.candidatesTried
|
|
62761
|
+
};
|
|
62762
|
+
}
|
|
62763
|
+
var MAX_REPORT_BYTES = 64 * 1024;
|
|
62764
|
+
async function appendSessionDbRecoveryRecord(record, path, fileOps, log) {
|
|
62765
|
+
try {
|
|
62766
|
+
const line = Buffer.from(`${JSON.stringify(record)}
|
|
62767
|
+
`);
|
|
62768
|
+
const existing = await fileOps.stat(path);
|
|
62769
|
+
if ((existing?.size ?? 0) + line.length > MAX_REPORT_BYTES) {
|
|
62770
|
+
log(
|
|
62771
|
+
`WARNING: session-DB recovery report at ${path} reached its ${MAX_REPORT_BYTES}-byte limit`
|
|
62772
|
+
);
|
|
62773
|
+
return;
|
|
62774
|
+
}
|
|
62775
|
+
await fileOps.mkdirp(dirname4(path));
|
|
62776
|
+
await fileOps.appendFile(path, line);
|
|
62777
|
+
} catch (error) {
|
|
62778
|
+
log(
|
|
62779
|
+
`WARNING: could not record the session-DB recovery outcome for the runner's activity log at ${path}: ${describeError(error)}`
|
|
62780
|
+
);
|
|
62781
|
+
}
|
|
62782
|
+
}
|
|
62783
|
+
|
|
62536
62784
|
// src/session-db-verify.ts
|
|
62537
62785
|
init_esm_shims();
|
|
62538
62786
|
|
|
@@ -62586,7 +62834,11 @@ async function verifySessionDb(params) {
|
|
|
62586
62834
|
log(
|
|
62587
62835
|
`WARNING: SESSION-DB-INTEGRITY-FAILED: opencode.db at ${dbPath} failed its integrity check: ${result.detail}`
|
|
62588
62836
|
);
|
|
62589
|
-
|
|
62837
|
+
const decision = await walkBack(params);
|
|
62838
|
+
if (decision.kind !== "exhausted") return decision;
|
|
62839
|
+
const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(params.store, config.prefix, log);
|
|
62840
|
+
const localDb = separation.kind === "notConfigured" || separation.kind === "alreadyEmpty" || separation.kind === "quarantined" ? await discardCorruptDb(fileOps, dbPath, log) : { kind: "retained", reason: "separationUnproven" };
|
|
62841
|
+
return { ...decision, separation, localDb };
|
|
62590
62842
|
}
|
|
62591
62843
|
async function clearScratch(fileOps, scratch) {
|
|
62592
62844
|
await fileOps.remove(scratch);
|
|
@@ -62594,15 +62846,32 @@ async function clearScratch(fileOps, scratch) {
|
|
|
62594
62846
|
await fileOps.remove(`${scratch}-shm`);
|
|
62595
62847
|
}
|
|
62596
62848
|
async function discardCorruptDb(fileOps, dbPath, log) {
|
|
62597
|
-
|
|
62849
|
+
const paths = [dbPath, `${dbPath}-wal`, `${dbPath}-shm`];
|
|
62850
|
+
const errors = [];
|
|
62851
|
+
for (const path of paths) {
|
|
62598
62852
|
try {
|
|
62599
62853
|
await fileOps.remove(path);
|
|
62600
62854
|
} catch (error) {
|
|
62601
|
-
|
|
62602
|
-
`WARNING: could not remove corrupt local session-DB debris at ${path}: ${describeError(error)}`
|
|
62603
|
-
);
|
|
62855
|
+
errors.push(`${path}: ${describeError(error)}`);
|
|
62604
62856
|
}
|
|
62605
62857
|
}
|
|
62858
|
+
const survived = [];
|
|
62859
|
+
const removed = [];
|
|
62860
|
+
for (const path of paths) {
|
|
62861
|
+
try {
|
|
62862
|
+
if (await fileOps.stat(path) === null) removed.push(path);
|
|
62863
|
+
else survived.push(path);
|
|
62864
|
+
} catch (error) {
|
|
62865
|
+
survived.push(path);
|
|
62866
|
+
errors.push(`${path}: could not prove removal: ${describeError(error)}`);
|
|
62867
|
+
}
|
|
62868
|
+
}
|
|
62869
|
+
if (errors.length === 0 && survived.length === 0) return { kind: "discarded" };
|
|
62870
|
+
const detail = errors.length === 0 ? "paths remained after removal" : errors.join("; ");
|
|
62871
|
+
log(
|
|
62872
|
+
`WARNING: SESSION-DB-LOCAL-DISCARD-FAILED: removed=${removed.join(", ") || "(none)"} survived=${survived.join(", ") || "(none)"} errors=${detail}; boot is stopping so no process opens or writes what remains.`
|
|
62873
|
+
);
|
|
62874
|
+
return { kind: "discardFailed", removed, survived, detail };
|
|
62606
62875
|
}
|
|
62607
62876
|
async function adopt(fileOps, scratch, dbPath, log, txid, candidatesTried, bytes) {
|
|
62608
62877
|
await fileOps.remove(`${dbPath}-wal`);
|
|
@@ -62624,12 +62893,10 @@ async function walkBack(params) {
|
|
|
62624
62893
|
log(
|
|
62625
62894
|
`WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED: ${listing.stdout === null ? `litestream ltx exited non-zero: ${listing.detail}` : "litestream ltx succeeded but its listing could not be parsed"}`
|
|
62626
62895
|
);
|
|
62627
|
-
await discardCorruptDb(fileOps, dbPath, log);
|
|
62628
62896
|
return { kind: "exhausted", reason: "enumerationFailed", candidatesTried: 0 };
|
|
62629
62897
|
}
|
|
62630
62898
|
const candidates = walkbackCandidates(points);
|
|
62631
62899
|
if (candidates.length === 0) {
|
|
62632
|
-
await discardCorruptDb(fileOps, dbPath, log);
|
|
62633
62900
|
return { kind: "exhausted", reason: "noCandidates", candidatesTried: 0 };
|
|
62634
62901
|
}
|
|
62635
62902
|
const deadline = now() + budgetSeconds * 1e3;
|
|
@@ -62660,7 +62927,6 @@ async function walkBack(params) {
|
|
|
62660
62927
|
return { kind: "walkedBack", txid, bytes, candidatesTried };
|
|
62661
62928
|
}
|
|
62662
62929
|
await clearScratch(fileOps, scratch);
|
|
62663
|
-
await discardCorruptDb(fileOps, dbPath, log);
|
|
62664
62930
|
return {
|
|
62665
62931
|
kind: "exhausted",
|
|
62666
62932
|
reason: index < candidates.length ? "budgetExhausted" : "allFailed",
|
|
@@ -62845,6 +63111,7 @@ var EXIT_SESSION_DB_FATAL = 30;
|
|
|
62845
63111
|
var EXIT_SESSION_DB_UNUSABLE = 31;
|
|
62846
63112
|
var EXIT_SESSION_DB_RETRY = 32;
|
|
62847
63113
|
var EXIT_SESSION_DB_UNVERIFIABLE = 33;
|
|
63114
|
+
var EXIT_SESSION_DB_UNSEPARATED = 34;
|
|
62848
63115
|
var STATE_SUFFIX = ".synchash";
|
|
62849
63116
|
var COMMANDS = shell_contract_default.commands;
|
|
62850
63117
|
var USAGE = `Usage: runner-synchroniser <command> [args]
|
|
@@ -62857,16 +63124,18 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
|
|
|
62857
63124
|
self-stop scale this agent's own ECS service to 0; exit 0 only
|
|
62858
63125
|
when desiredCount is confirmed 0, ${EXIT_KEEP_TASK} to keep the task
|
|
62859
63126
|
session-db-classify <litestream-restore-exit-code> <attempt>
|
|
62860
|
-
|
|
63127
|
+
[--on-unusable-replica=<prune|leave|clear|crash>] [--recovery-occurred]
|
|
63128
|
+
[--fresh-db-fallback]
|
|
62861
63129
|
classify a just-run \`litestream restore\` of opencode.db;
|
|
62862
63130
|
exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
|
|
62863
|
-
|
|
62864
|
-
|
|
63131
|
+
and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
|
|
63132
|
+
replicates into the existing prefix), ${EXIT_SESSION_DB_FATAL} fatal
|
|
62865
63133
|
session-db-verify <litestream-config-path>
|
|
62866
63134
|
verify opencode.db's integrity at boot, walking back through
|
|
62867
63135
|
litestream's retained restore points if it's corrupt; exit 0
|
|
62868
63136
|
healthy/skipped/walked-back, ${EXIT_SESSION_DB_UNVERIFIABLE} every
|
|
62869
|
-
|
|
63137
|
+
candidate was exhausted (booted fresh), ${EXIT_SESSION_DB_UNSEPARATED}
|
|
63138
|
+
separation/local disposal could not be proven
|
|
62870
63139
|
`;
|
|
62871
63140
|
function shellQuote(value) {
|
|
62872
63141
|
return `'${value.replaceAll("'", `'\\''`)}'`;
|
|
@@ -62887,16 +63156,26 @@ function parseStoreName(value) {
|
|
|
62887
63156
|
return value === "claude" || value === "opencode" ? value : null;
|
|
62888
63157
|
}
|
|
62889
63158
|
var ON_UNUSABLE_REPLICA_FLAG = "--on-unusable-replica=";
|
|
63159
|
+
var RECOVERY_OCCURRED_FLAG = "--recovery-occurred";
|
|
63160
|
+
var FRESH_DB_FALLBACK_FLAG = "--fresh-db-fallback";
|
|
62890
63161
|
function parseNonNegativeInt(value) {
|
|
62891
63162
|
return /^\d+$/.test(value) ? Number.parseInt(value, 10) : null;
|
|
62892
63163
|
}
|
|
62893
63164
|
function parseSessionDbClassifyArgs(args) {
|
|
62894
63165
|
let strategyRaw;
|
|
63166
|
+
let recoveryOccurred = false;
|
|
63167
|
+
let freshDbFallback = false;
|
|
62895
63168
|
const positional = [];
|
|
62896
63169
|
for (const arg of args) {
|
|
62897
63170
|
if (arg.startsWith(ON_UNUSABLE_REPLICA_FLAG)) {
|
|
62898
63171
|
if (strategyRaw !== void 0) return null;
|
|
62899
63172
|
strategyRaw = arg.slice(ON_UNUSABLE_REPLICA_FLAG.length);
|
|
63173
|
+
} else if (arg === RECOVERY_OCCURRED_FLAG) {
|
|
63174
|
+
if (recoveryOccurred) return null;
|
|
63175
|
+
recoveryOccurred = true;
|
|
63176
|
+
} else if (arg === FRESH_DB_FALLBACK_FLAG) {
|
|
63177
|
+
if (freshDbFallback) return null;
|
|
63178
|
+
freshDbFallback = true;
|
|
62900
63179
|
} else if (arg.startsWith("--")) {
|
|
62901
63180
|
return null;
|
|
62902
63181
|
} else {
|
|
@@ -62908,7 +63187,7 @@ function parseSessionDbClassifyArgs(args) {
|
|
|
62908
63187
|
const attempt = parseNonNegativeInt(positional[1]);
|
|
62909
63188
|
const strategy = parseUnusableReplicaStrategy(strategyRaw);
|
|
62910
63189
|
if (exitCode === null || attempt === null || attempt < 1 || strategy === null) return null;
|
|
62911
|
-
return { exitCode, attempt, strategy };
|
|
63190
|
+
return { exitCode, attempt, strategy, recoveryOccurred, freshDbFallback };
|
|
62912
63191
|
}
|
|
62913
63192
|
function renderEnv(config) {
|
|
62914
63193
|
return [
|
|
@@ -62996,6 +63275,16 @@ async function commandSessionDbClassify(args, config, deps) {
|
|
|
62996
63275
|
log
|
|
62997
63276
|
});
|
|
62998
63277
|
log(describeSessionDbClassification(outcome, { bucket: config.bucket, region: config.region }));
|
|
63278
|
+
const record = recordRestoreOutcome(
|
|
63279
|
+
outcome,
|
|
63280
|
+
(deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString(),
|
|
63281
|
+
args.exitCode,
|
|
63282
|
+
args.attempt,
|
|
63283
|
+
args.recoveryOccurred,
|
|
63284
|
+
args.freshDbFallback
|
|
63285
|
+
);
|
|
63286
|
+
if (record !== null)
|
|
63287
|
+
await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
|
|
62999
63288
|
return sessionDbExitCode(outcome);
|
|
63000
63289
|
}
|
|
63001
63290
|
async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
|
|
@@ -63005,7 +63294,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
|
|
|
63005
63294
|
config.opencodeDbPath
|
|
63006
63295
|
);
|
|
63007
63296
|
const outcome = await verifySessionDb({
|
|
63008
|
-
config: { opencodeDbPath: config.opencodeDbPath },
|
|
63297
|
+
config: { opencodeDbPath: config.opencodeDbPath, prefix: config.prefix },
|
|
63298
|
+
store: (deps.objectStoreFor ?? s3ObjectStoreFor)(config),
|
|
63009
63299
|
fileOps,
|
|
63010
63300
|
sqlite: (deps.sqliteIntegrityFor ?? nodeSqliteIntegrityFor)(),
|
|
63011
63301
|
litestream,
|
|
@@ -63014,6 +63304,9 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
|
|
|
63014
63304
|
budgetSeconds: config.walkbackBudgetSeconds
|
|
63015
63305
|
});
|
|
63016
63306
|
log(describeSessionDbVerification(outcome));
|
|
63307
|
+
const record = recordVerifyOutcome(outcome, (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString());
|
|
63308
|
+
if (record !== null)
|
|
63309
|
+
await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
|
|
63017
63310
|
return sessionDbVerifyExitCode(outcome);
|
|
63018
63311
|
}
|
|
63019
63312
|
async function main(argv, deps) {
|
|
@@ -63055,7 +63348,7 @@ ${USAGE}`);
|
|
|
63055
63348
|
const args = parseSessionDbClassifyArgs(rest);
|
|
63056
63349
|
if (args === null) {
|
|
63057
63350
|
log(
|
|
63058
|
-
`'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash
|
|
63351
|
+
`'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>, --recovery-occurred, and --fresh-db-fallback.
|
|
63059
63352
|
${USAGE}`
|
|
63060
63353
|
);
|
|
63061
63354
|
return EXIT_USAGE;
|
|
@@ -63111,6 +63404,7 @@ export {
|
|
|
63111
63404
|
EXIT_NOT_READY,
|
|
63112
63405
|
EXIT_SESSION_DB_FATAL,
|
|
63113
63406
|
EXIT_SESSION_DB_RETRY,
|
|
63407
|
+
EXIT_SESSION_DB_UNSEPARATED,
|
|
63114
63408
|
EXIT_SESSION_DB_UNUSABLE,
|
|
63115
63409
|
EXIT_SESSION_DB_UNVERIFIABLE,
|
|
63116
63410
|
EXIT_USAGE,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@evident-ai/runner-synchroniser",
|
|
3
|
-
"version": "3.4.1-dev.
|
|
3
|
+
"version": "3.4.1-dev.77d5cb6",
|
|
4
4
|
"description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/cli.js",
|
|
@@ -26,7 +26,7 @@
|
|
|
26
26
|
"tsup": "^8.3.5",
|
|
27
27
|
"typescript": "^5.7.2",
|
|
28
28
|
"vitest": "^3.1.1",
|
|
29
|
-
"runner-image": "workspace:*"
|
|
29
|
+
"runner-fargate-image": "workspace:*"
|
|
30
30
|
},
|
|
31
31
|
"engines": {
|
|
32
32
|
"node": ">=22.13.0"
|
|
@@ -42,7 +42,7 @@
|
|
|
42
42
|
"repository": {
|
|
43
43
|
"type": "git",
|
|
44
44
|
"url": "https://github.com/sroze/evident.git",
|
|
45
|
-
"directory": "
|
|
45
|
+
"directory": "runner/synchroniser"
|
|
46
46
|
},
|
|
47
47
|
"homepage": "https://evident.run",
|
|
48
48
|
"author": "Evident <hello@evident.run>",
|