@evident-ai/runner-synchroniser 3.4.0 → 3.4.1-dev.1549524
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -17
- package/dist/cli.js +419 -52
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -59,8 +59,8 @@ build time.
|
|
|
59
59
|
| `sync-once <claude\|opencode>` | — | `0` = ran; non-zero = tool broken |
|
|
60
60
|
| `model-auth-ready` | — | `0` = ready, `10` = not ready; other = tool broken |
|
|
61
61
|
| `self-stop` | — | `0` = stopped, `20` = keep the task; other = tool broken |
|
|
62
|
-
| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh,
|
|
63
|
-
| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` =
|
|
62
|
+
| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>] [--recovery-occurred] [--fresh-db-fallback]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, replicates into the existing prefix), `30` = fatal; other = usage/tool broken |
|
|
63
|
+
| `session-db-verify <litestream-config-path>` | — | `0` = healthy/skipped/walked-back, `33` = exhausted after verified separation and local disposal, `34` = separation or disposal could not be proven; other = usage/tool broken |
|
|
64
64
|
|
|
65
65
|
`self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
|
|
66
66
|
(see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
|
|
@@ -75,7 +75,16 @@ saved query.
|
|
|
75
75
|
|
|
76
76
|
### `session-db-classify`
|
|
77
77
|
|
|
78
|
-
Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `
|
|
78
|
+
Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `runner/docker-images/fargate/README.md`'s [Strategy/What/Cost table](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
|
|
79
|
+
|
|
80
|
+
#### Activity recovery report
|
|
81
|
+
|
|
82
|
+
Reportable recovery outcomes append one versioned JSON line to the session-DB recovery
|
|
83
|
+
report. `evident run` drains that file once after authentication and records the outcome in
|
|
84
|
+
the runner activity log. Healthy boots do not write a report. An initial restore failure writes
|
|
85
|
+
a warning `restore_retried` record before the retry's result is known. Writing is best-effort
|
|
86
|
+
and never changes this command's exit code. The CLI reader owns this record contract: add a new
|
|
87
|
+
`v` rather than repurposing a field.
|
|
79
88
|
|
|
80
89
|
#### Positionals
|
|
81
90
|
|
|
@@ -102,6 +111,14 @@ Implementation-facing reference for the 7th command: it decides what a just-run
|
|
|
102
111
|
- Any other `--` token is rejected.
|
|
103
112
|
- Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
|
|
104
113
|
|
|
114
|
+
#### Recovery-report flags
|
|
115
|
+
|
|
116
|
+
`--recovery-occurred` marks a later successful restore after this boot already recovered a
|
|
117
|
+
replica, so an empty result is reported as a fresh session database. `--fresh-db-fallback`
|
|
118
|
+
marks a runtime that intentionally has no retry budget and will therefore boot fresh after this
|
|
119
|
+
otherwise-transient failure. Each flag may appear once, does not change the command's exit code,
|
|
120
|
+
and is rejected when repeated.
|
|
121
|
+
|
|
105
122
|
#### Outcome → exit code
|
|
106
123
|
|
|
107
124
|
| Outcome | Code | When |
|
|
@@ -121,6 +138,9 @@ Implementation-facing reference for the 7th command: it decides what a just-run
|
|
|
121
138
|
| `fatal(misconfig)` | `30` | no object store while persistence is enabled, or the probe failed on attempt 2 |
|
|
122
139
|
| `fatal(deliberate)` | `30` | attempt 2, `crash` |
|
|
123
140
|
|
|
141
|
+
Fatal exit codes 30 and 34 are reported directly to `POST /v1/runners/self/startup-failure`
|
|
142
|
+
before the CLI starts. Reporting is best-effort and never changes the exit code.
|
|
143
|
+
|
|
124
144
|
`31` also guarantees local debris (`opencode.db`, `-wal`, `-shm`) is discarded, one `try` per path, at the classifier's single return point. Note `32` isn't purely "transient" — `recovered` shares it with `retryTransient` even though a `recovered` outcome already mutated S3.
|
|
125
145
|
|
|
126
146
|
The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
|
|
@@ -196,11 +216,11 @@ what it is.
|
|
|
196
216
|
#### The four `--on-unusable-replica` strategies
|
|
197
217
|
|
|
198
218
|
- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; a failed delete (`recoveryFailed`) or a declined escalation → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone by the *delete* — and since #1106 that is no longer the end of the road. When prune finds nothing to delete (`targetHealthy`/`noL0Present`) **and** at least one key parses as an LTX object, it **escalates to quarantine**: every object under `<prefix>/opencode.db/` is moved to `<prefix>/quarantine/opencode.db/<timestamp>/` → `recovered(quarantine)`/`32`. It does **not** escalate on `layoutMismatch` (we don't understand the layout), on an unreadable object (absent evidence), or on an empty prefix. Never prunes twice.
|
|
199
|
-
- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot".
|
|
219
|
+
- **`leave`** — zero S3 mutation → `31`. Cost is "prior history lost this boot". The fresh local DB replicates into the existing prefix; no new chain is started. Upside: the runner still comes online without a crash loop.
|
|
200
220
|
- **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
|
|
201
221
|
- **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
|
|
202
222
|
|
|
203
|
-
#### The measured real-world key layout (litestream 0.5.13)
|
|
223
|
+
#### The measured real-world key layout (litestream 0.5.13 historical sample)
|
|
204
224
|
|
|
205
225
|
`parseLtxKey` (`src/replica-keys.ts`) expects
|
|
206
226
|
`<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
|
|
@@ -218,7 +238,7 @@ at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written an
|
|
|
218
238
|
(`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
|
|
219
239
|
and it can never repair corruption at a higher compaction level.
|
|
220
240
|
|
|
221
|
-
`EVIDENT_ON_UNUSABLE_REPLICA` is a `
|
|
241
|
+
`EVIDENT_ON_UNUSABLE_REPLICA` is a `runner/docker-images/fargate` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [the Fargate image README](../docker-images/fargate/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32|33` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
|
|
222
242
|
|
|
223
243
|
See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
|
|
224
244
|
the behavioural anchor (31 comes online, 30 does not).
|
|
@@ -245,9 +265,12 @@ nothing had ever run a real `PRAGMA integrity_check`).
|
|
|
245
265
|
each into a scratch path and integrity-checking it there. The first one that passes
|
|
246
266
|
is adopted — renamed over `opencode.db`, replacing it and its stale sidecars.
|
|
247
267
|
3. If nothing retained passes (or there's nothing to try, or the search runs out of
|
|
248
|
-
time/points), the
|
|
249
|
-
|
|
250
|
-
|
|
268
|
+
time/points), the active replica prefix is separated first: its history is moved aside,
|
|
269
|
+
then the prefix is re-listed as empty before the corrupt local DB and sidecars are
|
|
270
|
+
discarded. Only both proofs permit a fresh boot. Otherwise exit `34` stops the boot before
|
|
271
|
+
OpenCode or Litestream starts; the backup history remains readable at its quarantine
|
|
272
|
+
destination or original key, and surviving local files remain in place without a process
|
|
273
|
+
opening or writing them.
|
|
251
274
|
|
|
252
275
|
A walked-back boot keeps replicating into the **same** replica — no S3 mutation, no new
|
|
253
276
|
prefix. litestream re-bases to the replica's high-water mark and continues the txid
|
|
@@ -262,10 +285,8 @@ against S3 at the ~1 GB scale a production replica can reach.)
|
|
|
262
285
|
| `skipped` | `0` | local DB absent or zero bytes — nothing to verify |
|
|
263
286
|
| `healthy` | `0` | the local DB passed its integrity check as-is |
|
|
264
287
|
| `walkedBack` | `0` | the local DB failed, but an older retained restore point passed and was adopted |
|
|
265
|
-
| `exhausted` (`noCandidates`) | `33` | the
|
|
266
|
-
| `exhausted`
|
|
267
|
-
| `exhausted` (`budgetExhausted`) | `33` | the search stopped on `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS`/`_BUDGET_SECONDS` before trying every retained point |
|
|
268
|
-
| `exhausted` (`enumerationFailed`) | `33` | `litestream ltx` could not be read (bad exit, non-JSON, malformed listing) |
|
|
288
|
+
| `exhausted` (`noCandidates`, `allFailed`, `budgetExhausted`, or `enumerationFailed`) | `33` | the active prefix was proven separated (or persistence is disabled) **and** the corrupt local DB and sidecars were proven discarded |
|
|
289
|
+
| `exhausted` | `34` | either the replica separation or local discard could not be proven; the boot stops before OpenCode and Litestream start |
|
|
269
290
|
|
|
270
291
|
`healthy`/`walkedBack`/`skipped` all share exit `0` deliberately: the shell's behaviour
|
|
271
292
|
afterwards — start `litestream replicate` — is identical either way, so a further split
|
|
@@ -284,7 +305,10 @@ before looking at everything" would report the wrong conclusion to the operator.
|
|
|
284
305
|
| `INFO: SESSION-DB-WALKBACK-ADOPTED` | `adopt` (`session-db-verify.ts`) | A candidate passes and is adopted over `opencode.db` | `txid`, `candidates`, `bytes` |
|
|
285
306
|
| `SESSION-DB-INTEGRITY` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | The local DB passed its check as-is | `bytes` |
|
|
286
307
|
| `SESSION-DB-INTEGRITY-WALKBACK` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Same event as `SESSION-DB-WALKBACK-ADOPTED` above, restated as the command's outcome | `txid`, `candidates`, `bytes` |
|
|
287
|
-
| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed
|
|
308
|
+
| `SESSION-DB-INTEGRITY-EXHAUSTED` | `describeSessionDbVerification` (`diagnostics.ts`), logged once per invocation as the final outcome | Nothing retained passed; a fresh boot follows only when both separation and local disposal were proven | `candidatesTried` (in prose, not `k=v`) |
|
|
309
|
+
| `SESSION-DB-REPLICA-SEPARATED` | `separateCorruptReplica` (`replica-recovery.ts`) | The active prefix was re-listed empty after separation | replica root and quarantine destination |
|
|
310
|
+
| `SESSION-DB-REPLICA-SEPARATION-UNVERIFIED` | `separateCorruptReplica` (`replica-recovery.ts`) | A separation could not be proven | remaining objects or list error |
|
|
311
|
+
| `SESSION-DB-LOCAL-DISCARD-FAILED` | `verifySessionDb` (`session-db-verify.ts`) | The local DB or a sidecar could not be proven gone | removed and surviving paths |
|
|
288
312
|
|
|
289
313
|
The first four fire only while the walkback loop runs; the last three are always the
|
|
290
314
|
one message this command logs as its own final word on the outcome — `entrypoint.sh`
|
|
@@ -316,10 +340,15 @@ task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 —
|
|
|
316
340
|
|
|
317
341
|
`session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
|
|
318
342
|
|
|
343
|
+
`session-db-verify` appends the same best-effort versioned report for a walkback or exhausted
|
|
344
|
+
history. A refused boot records `session_db_boot_refused`, but `evident run` has not started to
|
|
345
|
+
drain it, so operators see that refusal in boot logs and the restart loop. The report does not
|
|
346
|
+
alter the command's exit code.
|
|
347
|
+
|
|
319
348
|
`src/shell-contract.json` is the machine-checked source of truth for the command list, and
|
|
320
349
|
`shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
|
|
321
|
-
`
|
|
322
|
-
`
|
|
350
|
+
`runner/docker-images/fargate/entrypoint.sh` and the MicroVM's
|
|
351
|
+
`runner/docker-images/microvm/hooks` (#608), discovered by grep so a third one
|
|
323
352
|
cannot go unchecked. Each shell must: call only subcommands the CLI implements (and, for
|
|
324
353
|
`entrypoint.sh`, call all of them); route every call through one `run_synchroniser`; report
|
|
325
354
|
a broken tool; and take each answer code the commands it calls can return **silently and by
|
|
@@ -344,11 +373,12 @@ key, so they never appear in `env`'s output or `litestream.yml`.
|
|
|
344
373
|
| `CREDS_SYNC_INTERVAL` | Seconds between sync ticks, reported by `env` for the caller's loop. | Falls back to `60`; also on unparseable or non-positive values. |
|
|
345
374
|
| `EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS` | Distinct restore points `session-db-verify`'s walkback will try before giving up. | Falls back to `10`; also on unparseable or non-positive values. |
|
|
346
375
|
| `EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS` | Wall-clock budget, in seconds, for the whole walkback loop. | Falls back to `180`; also on unparseable or non-positive values. |
|
|
376
|
+
| `EVIDENT_SESSION_DB_RECOVERY_REPORT` | JSONL report drained once into runner activity after authentication. | `$HOME/.local/state/evident/session-db-recovery.jsonl`; blank values use the default. |
|
|
347
377
|
| `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Presence alone counts as configured model auth (see `model-auth-ready`). | No API-key fallback; `model-auth-ready` then depends solely on the credential files. |
|
|
348
378
|
| `CLUSTER` † / `SERVICE` † | ECS cluster/service `self-stop` scales to `desiredCount=0`. | Warns "cannot self-stop" and exits `20` (keep the task). Ignored by every other command. |
|
|
349
379
|
| `EVIDENT_SELFSTOP_ROLE_ARN` † | Role `self-stop` assumes for its ECS calls. | Optional: falls back to the task role's own credentials. A failed/incomplete assume-role → `20`. |
|
|
350
380
|
|
|
351
|
-
`
|
|
381
|
+
`runner/docker-images/fargate/README.md` documents these same three from the deployment side —
|
|
352
382
|
keep them in sync.
|
|
353
383
|
|
|
354
384
|
Requiring **both** `LITESTREAM_BUCKET` and `LITESTREAM_PREFIX` (never just one) means
|
package/dist/cli.js
CHANGED
|
@@ -51907,7 +51907,8 @@ function positiveIntOr(value, defaultValue) {
|
|
|
51907
51907
|
return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue;
|
|
51908
51908
|
}
|
|
51909
51909
|
function nonEmpty(value) {
|
|
51910
|
-
|
|
51910
|
+
const trimmed = value?.trim();
|
|
51911
|
+
return trimmed ? trimmed : null;
|
|
51911
51912
|
}
|
|
51912
51913
|
function resolveConfig(env4) {
|
|
51913
51914
|
const homeDir = nonEmpty(env4.HOME);
|
|
@@ -51930,6 +51931,7 @@ function resolveConfig(env4) {
|
|
|
51930
51931
|
key: persistenceEnabled ? `${prefix}/opencode/auth.json` : null
|
|
51931
51932
|
},
|
|
51932
51933
|
opencodeDbPath: `${homeDir}/.local/share/opencode/opencode.db`,
|
|
51934
|
+
sessionDbRecoveryReportPath: nonEmpty(env4.EVIDENT_SESSION_DB_RECOVERY_REPORT) ?? `${homeDir}/.local/state/evident/session-db-recovery.jsonl`,
|
|
51933
51935
|
bucket: persistenceEnabled ? bucket : null,
|
|
51934
51936
|
prefix: persistenceEnabled ? prefix : null,
|
|
51935
51937
|
region: nonEmpty(env4.AWS_REGION),
|
|
@@ -51942,7 +51944,9 @@ function resolveConfig(env4) {
|
|
|
51942
51944
|
env4.EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS,
|
|
51943
51945
|
DEFAULT_WALKBACK_BUDGET_SECONDS
|
|
51944
51946
|
),
|
|
51945
|
-
hasModelApiKey: nonEmpty(env4.ANTHROPIC_API_KEY) !== null || nonEmpty(env4.OPENAI_API_KEY) !== null
|
|
51947
|
+
hasModelApiKey: nonEmpty(env4.ANTHROPIC_API_KEY) !== null || nonEmpty(env4.OPENAI_API_KEY) !== null,
|
|
51948
|
+
runnerKey: nonEmpty(env4.EVIDENT_RUNNER_KEY) ?? nonEmpty(env4.EVIDENT_AGENT_KEY),
|
|
51949
|
+
apiUrl: nonEmpty(env4.EVIDENT_API_URL)
|
|
51946
51950
|
};
|
|
51947
51951
|
}
|
|
51948
51952
|
function storeOf(config, name) {
|
|
@@ -51998,7 +52002,7 @@ function sessionDbExitCode(outcome) {
|
|
|
51998
52002
|
// re-run `litestream restore` and ask again
|
|
51999
52003
|
case "unusableReplica":
|
|
52000
52004
|
return 31;
|
|
52001
|
-
// booted with a fresh DB; the shell still
|
|
52005
|
+
// booted with a fresh DB; the shell still replicates into the existing prefix
|
|
52002
52006
|
case "fatal":
|
|
52003
52007
|
return 30;
|
|
52004
52008
|
// shared by a genuine misconfig and a deliberate `crash` choice
|
|
@@ -52007,20 +52011,20 @@ function sessionDbExitCode(outcome) {
|
|
|
52007
52011
|
}
|
|
52008
52012
|
}
|
|
52009
52013
|
function describeUnusableReplica(reason) {
|
|
52010
|
-
const
|
|
52014
|
+
const base2 = "SESSION-DB-REPLICA-UNUSABLE: booting with a FRESH opencode.db; prior session history is lost until the replica is fixed.";
|
|
52011
52015
|
switch (reason) {
|
|
52012
52016
|
case "strategy":
|
|
52013
|
-
return `${
|
|
52017
|
+
return `${base2} (--on-unusable-replica=leave was selected; nothing in S3 was touched.)`;
|
|
52014
52018
|
case "attemptsExhausted":
|
|
52015
|
-
return `${
|
|
52019
|
+
return `${base2} (recovery did not make the replica usable after retrying; giving up.)`;
|
|
52016
52020
|
case "layoutMismatch":
|
|
52017
|
-
return `${
|
|
52021
|
+
return `${base2} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
|
|
52018
52022
|
case "noL0Present":
|
|
52019
|
-
return `${
|
|
52023
|
+
return `${base2} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
|
|
52020
52024
|
case "recoveryFailed":
|
|
52021
|
-
return `${
|
|
52025
|
+
return `${base2} (the recovery delete itself failed; see the warning above for the S3 error.)`;
|
|
52022
52026
|
case "targetHealthy":
|
|
52023
|
-
return `${
|
|
52027
|
+
return `${base2} (--on-unusable-replica=prune found a newest L0 object but could not confirm it is corrupt \u2014 it either passed the LTX structural check or could not be re-read \u2014 so it declined to delete it; nothing in S3 was touched.)`;
|
|
52024
52028
|
default:
|
|
52025
52029
|
return assertNeverUnusableReplicaReason(reason);
|
|
52026
52030
|
}
|
|
@@ -52061,7 +52065,7 @@ function describeSessionDbClassification(outcome, config) {
|
|
|
52061
52065
|
case "recovered": {
|
|
52062
52066
|
const objects = outcome.deleted.length === 0 ? "no objects (see the warnings above for what failed)" : `${outcome.deleted.length} object${outcome.deleted.length === 1 ? "" : "s"} (${outcome.deleted.join(", ")})`;
|
|
52063
52067
|
if (outcome.strategy === "quarantine") {
|
|
52064
|
-
return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside
|
|
52068
|
+
return `Escalated opencode.db replica recovery to quarantine: --on-unusable-replica=prune found nothing at level 0 to delete, so the corruption is deeper than prune can reach. Moved ${objects} aside; retrying the restore. Prior session history is set aside and remains readable at the quarantine destination \u2014 see the SESSION-DB-REPLICA-QUARANTINED warning above for where.`;
|
|
52065
52069
|
}
|
|
52066
52070
|
return `Attempted opencode.db replica recovery via --on-unusable-replica=${outcome.strategy}, deleting ${objects}; retrying the restore.` + (outcome.strategy === "clear" ? " Prior session history is lost." : "");
|
|
52067
52071
|
}
|
|
@@ -52082,6 +52086,12 @@ function assertNeverSessionDbVerifySkippedLocalDb(value) {
|
|
|
52082
52086
|
function assertNeverSessionDbVerifyExhaustedReason(value) {
|
|
52083
52087
|
throw new Error(`Unhandled session-DB verify exhausted reason: ${JSON.stringify(value)}`);
|
|
52084
52088
|
}
|
|
52089
|
+
function assertNeverSessionDbVerifySeparation(value) {
|
|
52090
|
+
throw new Error(`Unhandled session-DB verify separation: ${JSON.stringify(value)}`);
|
|
52091
|
+
}
|
|
52092
|
+
function assertNeverSessionDbVerifyLocalDb(value) {
|
|
52093
|
+
throw new Error(`Unhandled session-DB verify local-DB disposal: ${JSON.stringify(value)}`);
|
|
52094
|
+
}
|
|
52085
52095
|
function sessionDbVerifyExitCode(outcome) {
|
|
52086
52096
|
switch (outcome.kind) {
|
|
52087
52097
|
case "skipped":
|
|
@@ -52089,8 +52099,26 @@ function sessionDbVerifyExitCode(outcome) {
|
|
|
52089
52099
|
case "walkedBack":
|
|
52090
52100
|
return 0;
|
|
52091
52101
|
case "exhausted":
|
|
52092
|
-
|
|
52093
|
-
|
|
52102
|
+
switch (outcome.separation.kind) {
|
|
52103
|
+
case "notConfigured":
|
|
52104
|
+
case "alreadyEmpty":
|
|
52105
|
+
case "quarantined":
|
|
52106
|
+
break;
|
|
52107
|
+
case "incomplete":
|
|
52108
|
+
case "unreachable":
|
|
52109
|
+
return 34;
|
|
52110
|
+
default:
|
|
52111
|
+
return assertNeverSessionDbVerifySeparation(outcome.separation);
|
|
52112
|
+
}
|
|
52113
|
+
switch (outcome.localDb.kind) {
|
|
52114
|
+
case "discarded":
|
|
52115
|
+
return 33;
|
|
52116
|
+
case "retained":
|
|
52117
|
+
case "discardFailed":
|
|
52118
|
+
return 34;
|
|
52119
|
+
default:
|
|
52120
|
+
return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
|
|
52121
|
+
}
|
|
52094
52122
|
default:
|
|
52095
52123
|
return assertNeverSessionDbVerify(outcome);
|
|
52096
52124
|
}
|
|
@@ -52107,18 +52135,56 @@ function describeSessionDbVerifySkipped(localDb) {
|
|
|
52107
52135
|
}
|
|
52108
52136
|
function describeSessionDbVerifyExhausted(outcome) {
|
|
52109
52137
|
const tried = `tried ${outcome.candidatesTried} restore point${outcome.candidatesTried === 1 ? "" : "s"}`;
|
|
52138
|
+
let reason;
|
|
52110
52139
|
switch (outcome.reason) {
|
|
52111
52140
|
case "enumerationFailed":
|
|
52112
|
-
|
|
52141
|
+
reason = `opencode.db failed its integrity check and litestream's restore-point listing could not be read (${tried})`;
|
|
52142
|
+
break;
|
|
52113
52143
|
case "noCandidates":
|
|
52114
|
-
|
|
52144
|
+
reason = `opencode.db failed its integrity check and no older restore point is retained (${tried})`;
|
|
52145
|
+
break;
|
|
52115
52146
|
case "allFailed":
|
|
52116
|
-
|
|
52147
|
+
reason = `opencode.db failed its integrity check and every retained restore point also failed (${tried})`;
|
|
52148
|
+
break;
|
|
52117
52149
|
case "budgetExhausted":
|
|
52118
|
-
|
|
52150
|
+
reason = `opencode.db failed its integrity check; the walkback search budget ran out before every retained restore point was tried (${tried}) \u2014 widen it with EVIDENT_SESSION_DB_WALKBACK_MAX_POINTS/EVIDENT_SESSION_DB_WALKBACK_BUDGET_SECONDS if there was more to try`;
|
|
52151
|
+
break;
|
|
52119
52152
|
default:
|
|
52120
52153
|
return assertNeverSessionDbVerifyExhaustedReason(outcome.reason);
|
|
52121
52154
|
}
|
|
52155
|
+
let separation;
|
|
52156
|
+
switch (outcome.separation.kind) {
|
|
52157
|
+
case "quarantined":
|
|
52158
|
+
separation = ` The active replica prefix was verified empty; moved history remains readable at ${outcome.separation.destination}.`;
|
|
52159
|
+
break;
|
|
52160
|
+
case "alreadyEmpty":
|
|
52161
|
+
separation = " The active replica prefix was already empty.";
|
|
52162
|
+
break;
|
|
52163
|
+
case "notConfigured":
|
|
52164
|
+
separation = " Persistence is disabled, so there is no replica to separate.";
|
|
52165
|
+
break;
|
|
52166
|
+
case "incomplete":
|
|
52167
|
+
case "unreachable":
|
|
52168
|
+
separation = ` Boot is stopping because replica separation could not be proven (${outcome.separation.detail}); backup history remains readable at its original key or quarantine destination.`;
|
|
52169
|
+
break;
|
|
52170
|
+
default:
|
|
52171
|
+
return assertNeverSessionDbVerifySeparation(outcome.separation);
|
|
52172
|
+
}
|
|
52173
|
+
let localDb;
|
|
52174
|
+
switch (outcome.localDb.kind) {
|
|
52175
|
+
case "discarded":
|
|
52176
|
+
localDb = " booting with a FRESH opencode.db.";
|
|
52177
|
+
break;
|
|
52178
|
+
case "retained":
|
|
52179
|
+
localDb = " The corrupt local database and sidecars were left untouched for investigation.";
|
|
52180
|
+
break;
|
|
52181
|
+
case "discardFailed":
|
|
52182
|
+
localDb = ` Boot is stopping because local disposal could not be proven; removed=${outcome.localDb.removed.join(", ") || "(none)"} survived=${outcome.localDb.survived.join(", ") || "(none)"}. See SESSION-DB-LOCAL-DISCARD-FAILED.`;
|
|
52183
|
+
break;
|
|
52184
|
+
default:
|
|
52185
|
+
return assertNeverSessionDbVerifyLocalDb(outcome.localDb);
|
|
52186
|
+
}
|
|
52187
|
+
return `SESSION-DB-INTEGRITY-EXHAUSTED: ${reason}.${separation}${localDb}`;
|
|
52122
52188
|
}
|
|
52123
52189
|
function describeSessionDbVerification(outcome) {
|
|
52124
52190
|
switch (outcome.kind) {
|
|
@@ -52138,7 +52204,17 @@ function describeSessionDbVerification(outcome) {
|
|
|
52138
52204
|
// src/file-ops.ts
|
|
52139
52205
|
init_esm_shims();
|
|
52140
52206
|
import { createHash as createHash8 } from "node:crypto";
|
|
52141
|
-
import {
|
|
52207
|
+
import {
|
|
52208
|
+
appendFile,
|
|
52209
|
+
chmod,
|
|
52210
|
+
mkdir,
|
|
52211
|
+
open,
|
|
52212
|
+
readFile as readFile5,
|
|
52213
|
+
rename,
|
|
52214
|
+
rm,
|
|
52215
|
+
stat,
|
|
52216
|
+
writeFile as writeFile4
|
|
52217
|
+
} from "node:fs/promises";
|
|
52142
52218
|
var nodeFileOps = {
|
|
52143
52219
|
async mkdirp(dir) {
|
|
52144
52220
|
await mkdir(dir, { recursive: true });
|
|
@@ -52150,14 +52226,18 @@ var nodeFileOps = {
|
|
|
52150
52226
|
try {
|
|
52151
52227
|
const stats = await stat(path);
|
|
52152
52228
|
return { size: stats.size };
|
|
52153
|
-
} catch {
|
|
52154
|
-
return null;
|
|
52229
|
+
} catch (error) {
|
|
52230
|
+
if (error instanceof Error && "code" in error && error.code === "ENOENT") return null;
|
|
52231
|
+
throw error;
|
|
52155
52232
|
}
|
|
52156
52233
|
},
|
|
52157
52234
|
readFile: readFile5,
|
|
52158
52235
|
async writeFile(path, data) {
|
|
52159
52236
|
await writeFile4(path, data);
|
|
52160
52237
|
},
|
|
52238
|
+
async appendFile(path, data) {
|
|
52239
|
+
await appendFile(path, data);
|
|
52240
|
+
},
|
|
52161
52241
|
async createExclusive(path) {
|
|
52162
52242
|
const handle = await open(path, "wx", 384);
|
|
52163
52243
|
await handle.close();
|
|
@@ -52252,7 +52332,7 @@ function renderLitestreamConfig(config) {
|
|
|
52252
52332
|
# (no static keys here).
|
|
52253
52333
|
#
|
|
52254
52334
|
# INVARIANT (see README "Single-writer invariant"): exactly ONE writer per S3
|
|
52255
|
-
# prefix \u2014
|
|
52335
|
+
# prefix \u2014 we do not enable Litestream's server-side lease, so never run two
|
|
52256
52336
|
# replicators on one prefix. Only opencode.db is replicated.
|
|
52257
52337
|
|
|
52258
52338
|
dbs:
|
|
@@ -62324,7 +62404,7 @@ async function probeReplica(store, prefix, log) {
|
|
|
62324
62404
|
const objectCount = objects.length;
|
|
62325
62405
|
const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
|
|
62326
62406
|
logReplicaSize(root12, objectCount, totalBytes, log);
|
|
62327
|
-
return { ok: true,
|
|
62407
|
+
return { ok: true, objects };
|
|
62328
62408
|
} catch (error) {
|
|
62329
62409
|
const detail = describeError(error);
|
|
62330
62410
|
log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
|
|
@@ -62381,12 +62461,14 @@ async function pruneNewestL0(store, prefix, keys, log) {
|
|
|
62381
62461
|
function quarantineRoot(prefix, stamp) {
|
|
62382
62462
|
return `${prefix}/quarantine/opencode.db/${stamp}/`;
|
|
62383
62463
|
}
|
|
62384
|
-
async function quarantineReplica(store, prefix,
|
|
62464
|
+
async function quarantineReplica(store, prefix, objects, log, stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-")) {
|
|
62385
62465
|
const destination = quarantineRoot(prefix, stamp);
|
|
62386
62466
|
const root12 = replicaDbPrefix(prefix);
|
|
62387
62467
|
const moved = [];
|
|
62388
62468
|
const failed = [];
|
|
62389
|
-
|
|
62469
|
+
let movedBytes = 0;
|
|
62470
|
+
for (const object of objects) {
|
|
62471
|
+
const { key } = object;
|
|
62390
62472
|
if (!isDeletableReplicaKey(prefix, key)) {
|
|
62391
62473
|
log(`WARNING: refusing to quarantine ${key}: outside the replica prefix.`);
|
|
62392
62474
|
continue;
|
|
@@ -62411,15 +62493,48 @@ async function quarantineReplica(store, prefix, keys, log, stamp = (/* @__PURE__
|
|
|
62411
62493
|
continue;
|
|
62412
62494
|
}
|
|
62413
62495
|
moved.push(key);
|
|
62496
|
+
movedBytes += object.size;
|
|
62414
62497
|
}
|
|
62415
62498
|
logQuarantined(destination, moved, failed, log);
|
|
62416
|
-
return { destination, moved, failed };
|
|
62499
|
+
return { destination, moved, failed, movedBytes };
|
|
62417
62500
|
}
|
|
62418
62501
|
function logQuarantined(destination, moved, failed, log) {
|
|
62419
62502
|
log(
|
|
62420
|
-
`WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved)
|
|
62503
|
+
`WARNING: SESSION-DB-REPLICA-QUARANTINED: moved ${moved.length} unusable replica object(s) aside to ${destination} (${failed.length} could not be moved). Moved history remains readable at that destination; failed moves remain at their original keys.` + (failed.length > 0 ? " The active replica prefix is not clear." : "") + `See SESSION-DB-REPLICA-SIZE above for how much was set aside.`
|
|
62421
62504
|
);
|
|
62422
62505
|
}
|
|
62506
|
+
async function separateCorruptReplica(store, prefix, log, stamp) {
|
|
62507
|
+
const probe = await probeReplica(store, prefix, log);
|
|
62508
|
+
if (!probe.ok) return { kind: "unreachable", detail: probe.detail };
|
|
62509
|
+
const root12 = replicaDbPrefix(prefix);
|
|
62510
|
+
if (probe.objects.length === 0) {
|
|
62511
|
+
log(
|
|
62512
|
+
`INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was already empty; this boot starts a fresh backup chain.`
|
|
62513
|
+
);
|
|
62514
|
+
return { kind: "alreadyEmpty" };
|
|
62515
|
+
}
|
|
62516
|
+
const result = await quarantineReplica(store, prefix, probe.objects, log, stamp);
|
|
62517
|
+
try {
|
|
62518
|
+
const remaining = await store.list(root12);
|
|
62519
|
+
if (remaining.length > 0) {
|
|
62520
|
+
const detail = `${remaining.length} object(s) remain`;
|
|
62521
|
+
log(
|
|
62522
|
+
`WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: ${root12} still has ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
|
|
62523
|
+
);
|
|
62524
|
+
return { kind: "incomplete", detail };
|
|
62525
|
+
}
|
|
62526
|
+
} catch (error) {
|
|
62527
|
+
const detail = describeError(error);
|
|
62528
|
+
log(
|
|
62529
|
+
`WARNING: SESSION-DB-REPLICA-SEPARATION-UNVERIFIED: could not re-list ${root12}: ${detail}; moved=${result.moved.length} failed=${result.failed.length}.`
|
|
62530
|
+
);
|
|
62531
|
+
return { kind: "incomplete", detail };
|
|
62532
|
+
}
|
|
62533
|
+
log(
|
|
62534
|
+
`INFO: SESSION-DB-REPLICA-SEPARATED: ${root12} was verified empty by re-listing after moving ${result.moved.length} object(s) to ${result.destination}; this boot starts a fresh backup chain.`
|
|
62535
|
+
);
|
|
62536
|
+
return { kind: "quarantined", destination: result.destination, moved: result.moved };
|
|
62537
|
+
}
|
|
62423
62538
|
async function clearReplica(store, prefix, keys, log) {
|
|
62424
62539
|
const deleted = [];
|
|
62425
62540
|
const failed = [];
|
|
@@ -62471,7 +62586,7 @@ async function decideSessionDbRestore(params) {
|
|
|
62471
62586
|
return { kind: "retryTransient" };
|
|
62472
62587
|
}
|
|
62473
62588
|
if (attempt >= 3) {
|
|
62474
|
-
return { kind: "unusableReplica", reason: "attemptsExhausted" };
|
|
62589
|
+
return { kind: "unusableReplica", reason: "attemptsExhausted", recovery: null };
|
|
62475
62590
|
}
|
|
62476
62591
|
const probe = await probeReplica(store, prefix, log);
|
|
62477
62592
|
if (!probe.ok) {
|
|
@@ -62481,7 +62596,12 @@ async function decideSessionDbRestore(params) {
|
|
|
62481
62596
|
detail: `could not verify the session-DB replica is reachable: ${probe.detail}`
|
|
62482
62597
|
};
|
|
62483
62598
|
}
|
|
62484
|
-
const keys = probe.
|
|
62599
|
+
const keys = probe.objects.map((object) => object.key);
|
|
62600
|
+
const recovery = {
|
|
62601
|
+
replicaObjects: probe.objects.length,
|
|
62602
|
+
replicaBytes: probe.objects.reduce((total, object) => total + object.size, 0),
|
|
62603
|
+
quarantine: null
|
|
62604
|
+
};
|
|
62485
62605
|
switch (strategy) {
|
|
62486
62606
|
case "crash":
|
|
62487
62607
|
return {
|
|
@@ -62490,30 +62610,43 @@ async function decideSessionDbRestore(params) {
|
|
|
62490
62610
|
detail: "the replica is unusable and --on-unusable-replica=crash was selected; no S3 object was touched"
|
|
62491
62611
|
};
|
|
62492
62612
|
case "leave":
|
|
62493
|
-
return { kind: "unusableReplica", reason: "strategy" };
|
|
62613
|
+
return { kind: "unusableReplica", reason: "strategy", recovery };
|
|
62494
62614
|
case "prune": {
|
|
62495
62615
|
const result = await pruneNewestL0(store, prefix, keys, log);
|
|
62496
62616
|
if (result.deleted === null) {
|
|
62497
62617
|
if (shouldEscalateToQuarantine(result.reason, prefix, keys)) {
|
|
62498
|
-
const quarantined = await quarantineReplica(store, prefix,
|
|
62618
|
+
const quarantined = await quarantineReplica(store, prefix, probe.objects, log);
|
|
62499
62619
|
if (quarantined.moved.length === 0) {
|
|
62500
|
-
return { kind: "unusableReplica", reason: "recoveryFailed" };
|
|
62620
|
+
return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
|
|
62501
62621
|
}
|
|
62502
|
-
return {
|
|
62622
|
+
return {
|
|
62623
|
+
kind: "recovered",
|
|
62624
|
+
strategy: "quarantine",
|
|
62625
|
+
deleted: quarantined.moved,
|
|
62626
|
+
recovery: {
|
|
62627
|
+
...recovery,
|
|
62628
|
+
quarantine: {
|
|
62629
|
+
destination: quarantined.destination,
|
|
62630
|
+
movedObjects: quarantined.moved.length,
|
|
62631
|
+
failedObjects: quarantined.failed.length,
|
|
62632
|
+
movedBytes: quarantined.movedBytes
|
|
62633
|
+
}
|
|
62634
|
+
}
|
|
62635
|
+
};
|
|
62503
62636
|
}
|
|
62504
62637
|
if (result.reason === "deleteFailed") {
|
|
62505
|
-
return { kind: "unusableReplica", reason: "recoveryFailed" };
|
|
62638
|
+
return { kind: "unusableReplica", reason: "recoveryFailed", recovery };
|
|
62506
62639
|
}
|
|
62507
62640
|
if (result.reason === "unreadable") {
|
|
62508
|
-
return { kind: "unusableReplica", reason: "targetHealthy" };
|
|
62641
|
+
return { kind: "unusableReplica", reason: "targetHealthy", recovery };
|
|
62509
62642
|
}
|
|
62510
|
-
return { kind: "unusableReplica", reason: result.reason };
|
|
62643
|
+
return { kind: "unusableReplica", reason: result.reason, recovery };
|
|
62511
62644
|
}
|
|
62512
|
-
return { kind: "recovered", strategy: "prune", deleted: [result.deleted] };
|
|
62645
|
+
return { kind: "recovered", strategy: "prune", deleted: [result.deleted], recovery };
|
|
62513
62646
|
}
|
|
62514
62647
|
case "clear": {
|
|
62515
62648
|
const result = await clearReplica(store, prefix, keys, log);
|
|
62516
|
-
return { kind: "recovered", strategy: "clear", deleted: result.deleted };
|
|
62649
|
+
return { kind: "recovered", strategy: "clear", deleted: result.deleted, recovery };
|
|
62517
62650
|
}
|
|
62518
62651
|
default:
|
|
62519
62652
|
return assertNeverStrategy(strategy);
|
|
@@ -62533,6 +62666,123 @@ async function discardLocalDebris(fileOps, dbPath, log) {
|
|
|
62533
62666
|
}
|
|
62534
62667
|
}
|
|
62535
62668
|
|
|
62669
|
+
// src/session-db-recovery-report.ts
|
|
62670
|
+
init_esm_shims();
|
|
62671
|
+
import { dirname as dirname4 } from "node:path";
|
|
62672
|
+
function base(at2, stage) {
|
|
62673
|
+
return {
|
|
62674
|
+
v: 1,
|
|
62675
|
+
event: "session_db_recovery",
|
|
62676
|
+
at: at2,
|
|
62677
|
+
stage,
|
|
62678
|
+
litestream_exit_code: null,
|
|
62679
|
+
attempt: null,
|
|
62680
|
+
replica_objects: null,
|
|
62681
|
+
replica_bytes: null,
|
|
62682
|
+
quarantine_destination: null,
|
|
62683
|
+
quarantined_objects: null,
|
|
62684
|
+
quarantine_failed_objects: null,
|
|
62685
|
+
quarantined_bytes: null,
|
|
62686
|
+
verified_restore_point: null,
|
|
62687
|
+
restore_points_tried: null
|
|
62688
|
+
};
|
|
62689
|
+
}
|
|
62690
|
+
function recordRestoreOutcome(outcome, at2, exitCode, attempt, recoveryOccurred, freshDbFallback = false) {
|
|
62691
|
+
const record = { ...base(at2, "restore"), litestream_exit_code: exitCode, attempt };
|
|
62692
|
+
if (outcome.kind === "disabled" || outcome.kind === "restored") return null;
|
|
62693
|
+
if (outcome.kind === "retryTransient")
|
|
62694
|
+
return freshDbFallback ? {
|
|
62695
|
+
...record,
|
|
62696
|
+
outcome: "fresh_session_db",
|
|
62697
|
+
severity: "error",
|
|
62698
|
+
reason: "retry_budget_exhausted"
|
|
62699
|
+
} : {
|
|
62700
|
+
...record,
|
|
62701
|
+
outcome: "restore_retried",
|
|
62702
|
+
severity: "warning",
|
|
62703
|
+
reason: "initial_restore_failed"
|
|
62704
|
+
};
|
|
62705
|
+
if (outcome.kind === "noReplica") {
|
|
62706
|
+
return recoveryOccurred ? {
|
|
62707
|
+
...record,
|
|
62708
|
+
outcome: "fresh_session_db",
|
|
62709
|
+
severity: "error",
|
|
62710
|
+
reason: "no_replica_after_recovery"
|
|
62711
|
+
} : null;
|
|
62712
|
+
}
|
|
62713
|
+
if (outcome.kind === "fatal") return null;
|
|
62714
|
+
if (outcome.kind === "unusableReplica")
|
|
62715
|
+
return {
|
|
62716
|
+
...record,
|
|
62717
|
+
outcome: "fresh_session_db",
|
|
62718
|
+
severity: "error",
|
|
62719
|
+
reason: outcome.reason,
|
|
62720
|
+
replica_objects: outcome.recovery?.replicaObjects ?? null,
|
|
62721
|
+
replica_bytes: outcome.recovery?.replicaBytes ?? null
|
|
62722
|
+
};
|
|
62723
|
+
return {
|
|
62724
|
+
...record,
|
|
62725
|
+
outcome: "replica_recovered",
|
|
62726
|
+
severity: outcome.strategy === "clear" ? "error" : "warning",
|
|
62727
|
+
reason: outcome.strategy,
|
|
62728
|
+
replica_objects: outcome.recovery.replicaObjects,
|
|
62729
|
+
replica_bytes: outcome.recovery.replicaBytes,
|
|
62730
|
+
quarantine_destination: outcome.recovery.quarantine?.destination ?? null,
|
|
62731
|
+
quarantined_objects: outcome.recovery.quarantine?.movedObjects ?? null,
|
|
62732
|
+
quarantine_failed_objects: outcome.recovery.quarantine?.failedObjects ?? null,
|
|
62733
|
+
quarantined_bytes: outcome.recovery.quarantine?.movedBytes ?? null
|
|
62734
|
+
};
|
|
62735
|
+
}
|
|
62736
|
+
function recordVerifyOutcome(outcome, at2) {
|
|
62737
|
+
const record = base(at2, "verify");
|
|
62738
|
+
if (outcome.kind === "skipped" || outcome.kind === "healthy") return null;
|
|
62739
|
+
if (outcome.kind === "walkedBack")
|
|
62740
|
+
return {
|
|
62741
|
+
...record,
|
|
62742
|
+
outcome: "history_rolled_back",
|
|
62743
|
+
severity: "warning",
|
|
62744
|
+
reason: "walkback",
|
|
62745
|
+
verified_restore_point: outcome.txid,
|
|
62746
|
+
restore_points_tried: outcome.candidatesTried
|
|
62747
|
+
};
|
|
62748
|
+
if (sessionDbVerifyExitCode(outcome) === 34)
|
|
62749
|
+
return {
|
|
62750
|
+
...record,
|
|
62751
|
+
outcome: "session_db_boot_refused",
|
|
62752
|
+
severity: "error",
|
|
62753
|
+
reason: outcome.localDb.kind === "discardFailed" ? "local_discard_failed" : "replica_separation_unproven",
|
|
62754
|
+
restore_points_tried: outcome.candidatesTried,
|
|
62755
|
+
quarantine_destination: outcome.separation.kind === "quarantined" ? outcome.separation.destination : null
|
|
62756
|
+
};
|
|
62757
|
+
return {
|
|
62758
|
+
...record,
|
|
62759
|
+
outcome: "fresh_session_db",
|
|
62760
|
+
severity: "error",
|
|
62761
|
+
reason: outcome.reason,
|
|
62762
|
+
restore_points_tried: outcome.candidatesTried
|
|
62763
|
+
};
|
|
62764
|
+
}
|
|
62765
|
+
var MAX_REPORT_BYTES = 64 * 1024;
|
|
62766
|
+
async function appendSessionDbRecoveryRecord(record, path, fileOps, log) {
|
|
62767
|
+
try {
|
|
62768
|
+
const line = Buffer.from(`${JSON.stringify(record)}
|
|
62769
|
+
`);
|
|
62770
|
+
const existing = await fileOps.stat(path);
|
|
62771
|
+
if ((existing?.size ?? 0) + line.length > MAX_REPORT_BYTES) {
|
|
62772
|
+
log(
|
|
62773
|
+
`WARNING: session-DB recovery report at ${path} reached its ${MAX_REPORT_BYTES}-byte limit`
|
|
62774
|
+
);
|
|
62775
|
+
return;
|
|
62776
|
+
}
|
|
62777
|
+
await fileOps.mkdirp(dirname4(path));
|
|
62778
|
+
await fileOps.appendFile(path, line);
|
|
62779
|
+
} catch (error) {
|
|
62780
|
+
log(
|
|
62781
|
+
`WARNING: could not record the session-DB recovery outcome for the runner's activity log at ${path}: ${describeError(error)}`
|
|
62782
|
+
);
|
|
62783
|
+
}
|
|
62784
|
+
}
|
|
62785
|
+
|
|
62536
62786
|
// src/session-db-verify.ts
|
|
62537
62787
|
init_esm_shims();
|
|
62538
62788
|
|
|
@@ -62586,7 +62836,11 @@ async function verifySessionDb(params) {
|
|
|
62586
62836
|
log(
|
|
62587
62837
|
`WARNING: SESSION-DB-INTEGRITY-FAILED: opencode.db at ${dbPath} failed its integrity check: ${result.detail}`
|
|
62588
62838
|
);
|
|
62589
|
-
|
|
62839
|
+
const decision = await walkBack(params);
|
|
62840
|
+
if (decision.kind !== "exhausted") return decision;
|
|
62841
|
+
const separation = params.store === null || config.prefix === null ? { kind: "notConfigured" } : await separateCorruptReplica(params.store, config.prefix, log);
|
|
62842
|
+
const localDb = separation.kind === "notConfigured" || separation.kind === "alreadyEmpty" || separation.kind === "quarantined" ? await discardCorruptDb(fileOps, dbPath, log) : { kind: "retained", reason: "separationUnproven" };
|
|
62843
|
+
return { ...decision, separation, localDb };
|
|
62590
62844
|
}
|
|
62591
62845
|
async function clearScratch(fileOps, scratch) {
|
|
62592
62846
|
await fileOps.remove(scratch);
|
|
@@ -62594,15 +62848,32 @@ async function clearScratch(fileOps, scratch) {
|
|
|
62594
62848
|
await fileOps.remove(`${scratch}-shm`);
|
|
62595
62849
|
}
|
|
62596
62850
|
async function discardCorruptDb(fileOps, dbPath, log) {
|
|
62597
|
-
|
|
62851
|
+
const paths = [dbPath, `${dbPath}-wal`, `${dbPath}-shm`];
|
|
62852
|
+
const errors = [];
|
|
62853
|
+
for (const path of paths) {
|
|
62598
62854
|
try {
|
|
62599
62855
|
await fileOps.remove(path);
|
|
62600
62856
|
} catch (error) {
|
|
62601
|
-
|
|
62602
|
-
`WARNING: could not remove corrupt local session-DB debris at ${path}: ${describeError(error)}`
|
|
62603
|
-
);
|
|
62857
|
+
errors.push(`${path}: ${describeError(error)}`);
|
|
62604
62858
|
}
|
|
62605
62859
|
}
|
|
62860
|
+
const survived = [];
|
|
62861
|
+
const removed = [];
|
|
62862
|
+
for (const path of paths) {
|
|
62863
|
+
try {
|
|
62864
|
+
if (await fileOps.stat(path) === null) removed.push(path);
|
|
62865
|
+
else survived.push(path);
|
|
62866
|
+
} catch (error) {
|
|
62867
|
+
survived.push(path);
|
|
62868
|
+
errors.push(`${path}: could not prove removal: ${describeError(error)}`);
|
|
62869
|
+
}
|
|
62870
|
+
}
|
|
62871
|
+
if (errors.length === 0 && survived.length === 0) return { kind: "discarded" };
|
|
62872
|
+
const detail = errors.length === 0 ? "paths remained after removal" : errors.join("; ");
|
|
62873
|
+
log(
|
|
62874
|
+
`WARNING: SESSION-DB-LOCAL-DISCARD-FAILED: removed=${removed.join(", ") || "(none)"} survived=${survived.join(", ") || "(none)"} errors=${detail}; boot is stopping so no process opens or writes what remains.`
|
|
62875
|
+
);
|
|
62876
|
+
return { kind: "discardFailed", removed, survived, detail };
|
|
62606
62877
|
}
|
|
62607
62878
|
async function adopt(fileOps, scratch, dbPath, log, txid, candidatesTried, bytes) {
|
|
62608
62879
|
await fileOps.remove(`${dbPath}-wal`);
|
|
@@ -62624,12 +62895,10 @@ async function walkBack(params) {
|
|
|
62624
62895
|
log(
|
|
62625
62896
|
`WARNING: SESSION-DB-WALKBACK-ENUMERATION-FAILED: ${listing.stdout === null ? `litestream ltx exited non-zero: ${listing.detail}` : "litestream ltx succeeded but its listing could not be parsed"}`
|
|
62626
62897
|
);
|
|
62627
|
-
await discardCorruptDb(fileOps, dbPath, log);
|
|
62628
62898
|
return { kind: "exhausted", reason: "enumerationFailed", candidatesTried: 0 };
|
|
62629
62899
|
}
|
|
62630
62900
|
const candidates = walkbackCandidates(points);
|
|
62631
62901
|
if (candidates.length === 0) {
|
|
62632
|
-
await discardCorruptDb(fileOps, dbPath, log);
|
|
62633
62902
|
return { kind: "exhausted", reason: "noCandidates", candidatesTried: 0 };
|
|
62634
62903
|
}
|
|
62635
62904
|
const deadline = now() + budgetSeconds * 1e3;
|
|
@@ -62660,7 +62929,6 @@ async function walkBack(params) {
|
|
|
62660
62929
|
return { kind: "walkedBack", txid, bytes, candidatesTried };
|
|
62661
62930
|
}
|
|
62662
62931
|
await clearScratch(fileOps, scratch);
|
|
62663
|
-
await discardCorruptDb(fileOps, dbPath, log);
|
|
62664
62932
|
return {
|
|
62665
62933
|
kind: "exhausted",
|
|
62666
62934
|
reason: index < candidates.length ? "budgetExhausted" : "allFailed",
|
|
@@ -62668,6 +62936,49 @@ async function walkBack(params) {
|
|
|
62668
62936
|
};
|
|
62669
62937
|
}
|
|
62670
62938
|
|
|
62939
|
+
// src/startup-failure-report.ts
|
|
62940
|
+
init_esm_shims();
|
|
62941
|
+
var STARTUP_FAILURE_REPORT_TIMEOUT_MS = 5e3;
|
|
62942
|
+
var MAX_RESPONSE_BODY_LENGTH = 1e3;
|
|
62943
|
+
async function reportStartupFailure(report, config, deps) {
|
|
62944
|
+
if (config.apiUrl === null || config.runnerKey === null) {
|
|
62945
|
+
deps.log(
|
|
62946
|
+
`Startup failure was not reported to Evident because ${config.apiUrl === null ? "EVIDENT_API_URL" : "a runner key"} is unavailable.`
|
|
62947
|
+
);
|
|
62948
|
+
return;
|
|
62949
|
+
}
|
|
62950
|
+
const endpoint = `${config.apiUrl.replace(/\/+$/, "").replace(/\/v1$/, "")}/v1/runners/self/startup-failure`;
|
|
62951
|
+
try {
|
|
62952
|
+
const response = await deps.fetchImpl(endpoint, {
|
|
62953
|
+
method: "POST",
|
|
62954
|
+
headers: {
|
|
62955
|
+
Authorization: `SandboxKey ${config.runnerKey}`,
|
|
62956
|
+
"Content-Type": "application/json"
|
|
62957
|
+
},
|
|
62958
|
+
body: JSON.stringify({
|
|
62959
|
+
classification: report.classification,
|
|
62960
|
+
stage: report.stage,
|
|
62961
|
+
litestream_exit_code: report.litestreamExitCode,
|
|
62962
|
+
attempt: report.attempt,
|
|
62963
|
+
restore_points_tried: report.restorePointsTried,
|
|
62964
|
+
report_id: deps.randomUUID()
|
|
62965
|
+
}),
|
|
62966
|
+
// The container is about to exit, so a long wait only delays ECS replacement.
|
|
62967
|
+
signal: AbortSignal.timeout(STARTUP_FAILURE_REPORT_TIMEOUT_MS)
|
|
62968
|
+
});
|
|
62969
|
+
if (!response.ok) {
|
|
62970
|
+
const body = (await response.text()).slice(0, MAX_RESPONSE_BODY_LENGTH);
|
|
62971
|
+
deps.log(
|
|
62972
|
+
`WARNING: fatal session-DB startup failure report was rejected (${response.status}): ${body}`
|
|
62973
|
+
);
|
|
62974
|
+
}
|
|
62975
|
+
} catch (error) {
|
|
62976
|
+
deps.log(
|
|
62977
|
+
`WARNING: could not report the fatal session-DB startup failure to Evident (${describeError(error)}); the failure is still shown above and in this machine's logs.`
|
|
62978
|
+
);
|
|
62979
|
+
}
|
|
62980
|
+
}
|
|
62981
|
+
|
|
62671
62982
|
// src/self-stop.ts
|
|
62672
62983
|
init_esm_shims();
|
|
62673
62984
|
function isComplete(credentials) {
|
|
@@ -62845,6 +63156,7 @@ var EXIT_SESSION_DB_FATAL = 30;
|
|
|
62845
63156
|
var EXIT_SESSION_DB_UNUSABLE = 31;
|
|
62846
63157
|
var EXIT_SESSION_DB_RETRY = 32;
|
|
62847
63158
|
var EXIT_SESSION_DB_UNVERIFIABLE = 33;
|
|
63159
|
+
var EXIT_SESSION_DB_UNSEPARATED = 34;
|
|
62848
63160
|
var STATE_SUFFIX = ".synchash";
|
|
62849
63161
|
var COMMANDS = shell_contract_default.commands;
|
|
62850
63162
|
var USAGE = `Usage: runner-synchroniser <command> [args]
|
|
@@ -62857,16 +63169,18 @@ var USAGE = `Usage: runner-synchroniser <command> [args]
|
|
|
62857
63169
|
self-stop scale this agent's own ECS service to 0; exit 0 only
|
|
62858
63170
|
when desiredCount is confirmed 0, ${EXIT_KEEP_TASK} to keep the task
|
|
62859
63171
|
session-db-classify <litestream-restore-exit-code> <attempt>
|
|
62860
|
-
|
|
63172
|
+
[--on-unusable-replica=<prune|leave|clear|crash>] [--recovery-occurred]
|
|
63173
|
+
[--fresh-db-fallback]
|
|
62861
63174
|
classify a just-run \`litestream restore\` of opencode.db;
|
|
62862
63175
|
exit 0 restored/no-replica/disabled, ${EXIT_SESSION_DB_RETRY} re-run
|
|
62863
|
-
|
|
62864
|
-
|
|
63176
|
+
and ask again, ${EXIT_SESSION_DB_UNUSABLE} unusable (booted fresh,
|
|
63177
|
+
replicates into the existing prefix), ${EXIT_SESSION_DB_FATAL} fatal
|
|
62865
63178
|
session-db-verify <litestream-config-path>
|
|
62866
63179
|
verify opencode.db's integrity at boot, walking back through
|
|
62867
63180
|
litestream's retained restore points if it's corrupt; exit 0
|
|
62868
63181
|
healthy/skipped/walked-back, ${EXIT_SESSION_DB_UNVERIFIABLE} every
|
|
62869
|
-
|
|
63182
|
+
candidate was exhausted (booted fresh), ${EXIT_SESSION_DB_UNSEPARATED}
|
|
63183
|
+
separation/local disposal could not be proven
|
|
62870
63184
|
`;
|
|
62871
63185
|
function shellQuote(value) {
|
|
62872
63186
|
return `'${value.replaceAll("'", `'\\''`)}'`;
|
|
@@ -62887,16 +63201,26 @@ function parseStoreName(value) {
|
|
|
62887
63201
|
return value === "claude" || value === "opencode" ? value : null;
|
|
62888
63202
|
}
|
|
62889
63203
|
var ON_UNUSABLE_REPLICA_FLAG = "--on-unusable-replica=";
|
|
63204
|
+
var RECOVERY_OCCURRED_FLAG = "--recovery-occurred";
|
|
63205
|
+
var FRESH_DB_FALLBACK_FLAG = "--fresh-db-fallback";
|
|
62890
63206
|
function parseNonNegativeInt(value) {
|
|
62891
63207
|
return /^\d+$/.test(value) ? Number.parseInt(value, 10) : null;
|
|
62892
63208
|
}
|
|
62893
63209
|
function parseSessionDbClassifyArgs(args) {
|
|
62894
63210
|
let strategyRaw;
|
|
63211
|
+
let recoveryOccurred = false;
|
|
63212
|
+
let freshDbFallback = false;
|
|
62895
63213
|
const positional = [];
|
|
62896
63214
|
for (const arg of args) {
|
|
62897
63215
|
if (arg.startsWith(ON_UNUSABLE_REPLICA_FLAG)) {
|
|
62898
63216
|
if (strategyRaw !== void 0) return null;
|
|
62899
63217
|
strategyRaw = arg.slice(ON_UNUSABLE_REPLICA_FLAG.length);
|
|
63218
|
+
} else if (arg === RECOVERY_OCCURRED_FLAG) {
|
|
63219
|
+
if (recoveryOccurred) return null;
|
|
63220
|
+
recoveryOccurred = true;
|
|
63221
|
+
} else if (arg === FRESH_DB_FALLBACK_FLAG) {
|
|
63222
|
+
if (freshDbFallback) return null;
|
|
63223
|
+
freshDbFallback = true;
|
|
62900
63224
|
} else if (arg.startsWith("--")) {
|
|
62901
63225
|
return null;
|
|
62902
63226
|
} else {
|
|
@@ -62908,7 +63232,7 @@ function parseSessionDbClassifyArgs(args) {
|
|
|
62908
63232
|
const attempt = parseNonNegativeInt(positional[1]);
|
|
62909
63233
|
const strategy = parseUnusableReplicaStrategy(strategyRaw);
|
|
62910
63234
|
if (exitCode === null || attempt === null || attempt < 1 || strategy === null) return null;
|
|
62911
|
-
return { exitCode, attempt, strategy };
|
|
63235
|
+
return { exitCode, attempt, strategy, recoveryOccurred, freshDbFallback };
|
|
62912
63236
|
}
|
|
62913
63237
|
function renderEnv(config) {
|
|
62914
63238
|
return [
|
|
@@ -62996,6 +63320,30 @@ async function commandSessionDbClassify(args, config, deps) {
|
|
|
62996
63320
|
log
|
|
62997
63321
|
});
|
|
62998
63322
|
log(describeSessionDbClassification(outcome, { bucket: config.bucket, region: config.region }));
|
|
63323
|
+
if (outcome.kind === "fatal") {
|
|
63324
|
+
await reportStartupFailure(
|
|
63325
|
+
{
|
|
63326
|
+
classification: outcome.cause === "misconfig" ? "session_db_restore_misconfigured" : "session_db_restore_crash_chosen",
|
|
63327
|
+
stage: "restore",
|
|
63328
|
+
litestreamExitCode: args.exitCode,
|
|
63329
|
+
attempt: args.attempt,
|
|
63330
|
+
restorePointsTried: null
|
|
63331
|
+
},
|
|
63332
|
+
config,
|
|
63333
|
+
{ fetchImpl: deps.fetchImpl ?? fetch, randomUUID: deps.randomUUID ?? crypto.randomUUID, log }
|
|
63334
|
+
);
|
|
63335
|
+
return sessionDbExitCode(outcome);
|
|
63336
|
+
}
|
|
63337
|
+
const record = recordRestoreOutcome(
|
|
63338
|
+
outcome,
|
|
63339
|
+
(deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString(),
|
|
63340
|
+
args.exitCode,
|
|
63341
|
+
args.attempt,
|
|
63342
|
+
args.recoveryOccurred,
|
|
63343
|
+
args.freshDbFallback
|
|
63344
|
+
);
|
|
63345
|
+
if (record !== null)
|
|
63346
|
+
await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
|
|
62999
63347
|
return sessionDbExitCode(outcome);
|
|
63000
63348
|
}
|
|
63001
63349
|
async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
|
|
@@ -63005,7 +63353,8 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
|
|
|
63005
63353
|
config.opencodeDbPath
|
|
63006
63354
|
);
|
|
63007
63355
|
const outcome = await verifySessionDb({
|
|
63008
|
-
config: { opencodeDbPath: config.opencodeDbPath },
|
|
63356
|
+
config: { opencodeDbPath: config.opencodeDbPath, prefix: config.prefix },
|
|
63357
|
+
store: (deps.objectStoreFor ?? s3ObjectStoreFor)(config),
|
|
63009
63358
|
fileOps,
|
|
63010
63359
|
sqlite: (deps.sqliteIntegrityFor ?? nodeSqliteIntegrityFor)(),
|
|
63011
63360
|
litestream,
|
|
@@ -63014,6 +63363,23 @@ async function commandSessionDbVerify(litestreamConfigPath, config, deps) {
|
|
|
63014
63363
|
budgetSeconds: config.walkbackBudgetSeconds
|
|
63015
63364
|
});
|
|
63016
63365
|
log(describeSessionDbVerification(outcome));
|
|
63366
|
+
if (outcome.kind === "exhausted" && sessionDbVerifyExitCode(outcome) === 34) {
|
|
63367
|
+
await reportStartupFailure(
|
|
63368
|
+
{
|
|
63369
|
+
classification: outcome.localDb.kind === "discardFailed" ? "session_db_local_discard_failed" : "session_db_separation_unproven",
|
|
63370
|
+
stage: "verify",
|
|
63371
|
+
litestreamExitCode: null,
|
|
63372
|
+
attempt: null,
|
|
63373
|
+
restorePointsTried: outcome.candidatesTried
|
|
63374
|
+
},
|
|
63375
|
+
config,
|
|
63376
|
+
{ fetchImpl: deps.fetchImpl ?? fetch, randomUUID: deps.randomUUID ?? crypto.randomUUID, log }
|
|
63377
|
+
);
|
|
63378
|
+
return sessionDbVerifyExitCode(outcome);
|
|
63379
|
+
}
|
|
63380
|
+
const record = recordVerifyOutcome(outcome, (deps.now ?? (() => /* @__PURE__ */ new Date()))().toISOString());
|
|
63381
|
+
if (record !== null)
|
|
63382
|
+
await appendSessionDbRecoveryRecord(record, config.sessionDbRecoveryReportPath, fileOps, log);
|
|
63017
63383
|
return sessionDbVerifyExitCode(outcome);
|
|
63018
63384
|
}
|
|
63019
63385
|
async function main(argv, deps) {
|
|
@@ -63055,7 +63421,7 @@ ${USAGE}`);
|
|
|
63055
63421
|
const args = parseSessionDbClassifyArgs(rest);
|
|
63056
63422
|
if (args === null) {
|
|
63057
63423
|
log(
|
|
63058
|
-
`'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash
|
|
63424
|
+
`'session-db-classify' needs <litestream-restore-exit-code> <attempt> and an optional --on-unusable-replica=<prune|leave|clear|crash>, --recovery-occurred, and --fresh-db-fallback.
|
|
63059
63425
|
${USAGE}`
|
|
63060
63426
|
);
|
|
63061
63427
|
return EXIT_USAGE;
|
|
@@ -63111,6 +63477,7 @@ export {
|
|
|
63111
63477
|
EXIT_NOT_READY,
|
|
63112
63478
|
EXIT_SESSION_DB_FATAL,
|
|
63113
63479
|
EXIT_SESSION_DB_RETRY,
|
|
63480
|
+
EXIT_SESSION_DB_UNSEPARATED,
|
|
63114
63481
|
EXIT_SESSION_DB_UNUSABLE,
|
|
63115
63482
|
EXIT_SESSION_DB_UNVERIFIABLE,
|
|
63116
63483
|
EXIT_USAGE,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@evident-ai/runner-synchroniser",
|
|
3
|
-
"version": "3.4.
|
|
3
|
+
"version": "3.4.1-dev.1549524",
|
|
4
4
|
"description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/cli.js",
|
|
@@ -26,7 +26,7 @@
|
|
|
26
26
|
"tsup": "^8.3.5",
|
|
27
27
|
"typescript": "^5.7.2",
|
|
28
28
|
"vitest": "^3.1.1",
|
|
29
|
-
"runner-image": "workspace:*"
|
|
29
|
+
"runner-fargate-image": "workspace:*"
|
|
30
30
|
},
|
|
31
31
|
"engines": {
|
|
32
32
|
"node": ">=22.13.0"
|
|
@@ -42,7 +42,7 @@
|
|
|
42
42
|
"repository": {
|
|
43
43
|
"type": "git",
|
|
44
44
|
"url": "https://github.com/sroze/evident.git",
|
|
45
|
-
"directory": "
|
|
45
|
+
"directory": "runner/synchroniser"
|
|
46
46
|
},
|
|
47
47
|
"homepage": "https://evident.run",
|
|
48
48
|
"author": "Evident <hello@evident.run>",
|