@evident-ai/runner-synchroniser 0.1.1-dev.9639f77 → 0.1.1-dev.9dbd844
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +174 -5
- package/dist/cli.js +56 -16
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -62,6 +62,7 @@ version at all.
|
|
|
62
62
|
| `sync-once <claude\|opencode>` | — | `0` = ran; non-zero = tool broken |
|
|
63
63
|
| `model-auth-ready` | — | `0` = ready, `10` = not ready; other = tool broken |
|
|
64
64
|
| `self-stop` | — | `0` = stopped, `20` = keep the task; other = tool broken |
|
|
65
|
+
| `session-db-classify <litestream-restore-exit-code> <attempt> [--on-unusable-replica=<prune\|leave\|clear\|crash>]` | — | `0` = restored/no-replica/disabled, `32` = re-run and ask again, `31` = unusable (booted fresh, skip replicate), `30` = fatal; other = usage/tool broken |
|
|
65
66
|
|
|
66
67
|
`self-stop` scales this agent's own ECS service to `desiredCount=0` on a clean idle exit
|
|
67
68
|
(see Configuration for `CLUSTER`/`SERVICE`/`EVIDENT_SELFSTOP_ROLE_ARN`). It exits `0`
|
|
@@ -74,6 +75,155 @@ broader: `src/logger.ts` marks it verbatim because operator greps and CloudWatch
|
|
|
74
75
|
match on that exact string, so widening what it labels is cheaper than breaking every
|
|
75
76
|
saved query.
|
|
76
77
|
|
|
78
|
+
### `session-db-classify`
|
|
79
|
+
|
|
80
|
+
Implementation-facing reference for the 7th command: it decides what a just-run `litestream restore` of `opencode.db` means and, on attempt 2 only, may run a recovery strategy against S3. `packages/runner-image/README.md`'s [Strategy/What/Cost table](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica) is the operator-facing view of the same command — this section doesn't restate it.
|
|
81
|
+
|
|
82
|
+
#### Positionals
|
|
83
|
+
|
|
84
|
+
| Positional | Meaning |
|
|
85
|
+
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
86
|
+
| `<litestream-restore-exit-code>` | Status of the `litestream restore` the shell just ran. `0` → stat the local DB; any non-zero → the recovery ladder below. The value itself is never inspected (a Go panic `2` is treated exactly like a `1`). |
|
|
87
|
+
| `<attempt>` | 1-based (`/^\d+$/`, `>= 1`) — the sole mutation gate. |
|
|
88
|
+
|
|
89
|
+
#### The attempt ladder
|
|
90
|
+
|
|
91
|
+
| Attempt | Behaviour |
|
|
92
|
+
| ------- | ---------------------------------------------------------------- |
|
|
93
|
+
| 1 | Plain retry — **zero S3 calls, not even the probe**. |
|
|
94
|
+
| 2 | The **only** attempt that may probe or mutate S3. |
|
|
95
|
+
| ≥ 3 | Gives up **before** the probe. |
|
|
96
|
+
|
|
97
|
+
`entrypoint.sh` caps its retry loop at a hard-coded `3` as a real second bound: the classifier is stateless per invocation, so nothing except that loop stops attempt 2 from being asked twice.
|
|
98
|
+
|
|
99
|
+
#### `--on-unusable-replica` flag syntax
|
|
100
|
+
|
|
101
|
+
- Only the `=`-joined form (`--on-unusable-replica=leave`); the space-separated form is rejected.
|
|
102
|
+
- Position-free.
|
|
103
|
+
- Repeating the flag is rejected, even with the same value both times.
|
|
104
|
+
- Any other `--` token is rejected.
|
|
105
|
+
- Every rejection exits `2` (`EXIT_USAGE`) with the usage message — **never** a silent fall-back to the destructive default.
|
|
106
|
+
|
|
107
|
+
#### Outcome → exit code
|
|
108
|
+
|
|
109
|
+
| Outcome | Code | When |
|
|
110
|
+
| ----------------------------------------------- | ---- | -------------------------------------------------------------------------------- |
|
|
111
|
+
| `disabled` | `0` | bucket or prefix unset — short-circuits **before** the exit code is examined |
|
|
112
|
+
| `restored` | `0` | restore exit `0` and the local DB is non-empty |
|
|
113
|
+
| `noReplica` | `0` | restore exit `0` but the DB is absent or zero-byte |
|
|
114
|
+
| `retryTransient` | `32` | restore failed, attempt 1 — no probe, no S3 call at all |
|
|
115
|
+
| `recovered` | `32` | attempt 2, a successful `prune` (1 key) or any `clear` (including a partial one) |
|
|
116
|
+
| `unusableReplica(strategy)` | `31` | attempt 2, `leave` |
|
|
117
|
+
| `unusableReplica(attemptsExhausted)` | `31` | attempt ≥ 3, before the probe |
|
|
118
|
+
| `unusableReplica(targetHealthy)` | `31` | attempt 2, `prune`; the newest L0 isn't provably corrupt, so it declined to delete it |
|
|
119
|
+
| `unusableReplica(noL0Present)` | `31` | attempt 2, `prune`; no level-0 object was present to prune — **normal**, see below |
|
|
120
|
+
| `unusableReplica(layoutMismatch)` | `31` | attempt 2, `prune`; objects exist under the prefix but **none** parse as an LTX key at all — **a bug**, see below |
|
|
121
|
+
| `unusableReplica(recoveryFailed)` | `31` | attempt 2, `prune`'s delete itself threw |
|
|
122
|
+
| `fatal(misconfig)` | `30` | no object store while persistence is enabled, or the probe failed on attempt 2 |
|
|
123
|
+
| `fatal(deliberate)` | `30` | attempt 2, `crash` |
|
|
124
|
+
|
|
125
|
+
`31` also guarantees local debris (`opencode.db`, `-wal`, `-shm`) is discarded, one `try` per path, at the classifier's single return point. Note `32` isn't purely "transient" — `recovered` shares it with `retryTransient` even though a `recovered` outcome already mutated S3.
|
|
126
|
+
|
|
127
|
+
The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
|
|
128
|
+
|
|
129
|
+
#### Boot size signals
|
|
130
|
+
|
|
131
|
+
Two markers report the numbers a boot already computes but used to throw away (#1113):
|
|
132
|
+
|
|
133
|
+
| Marker | Emitted from | When it fires | Fields |
|
|
134
|
+
| --- | --- | --- | --- |
|
|
135
|
+
| `SESSION-DB-SIZE` | `describeSessionDbClassification` (`diagnostics.ts`) | Once per `session-db-classify` invocation that reaches `restored` or `noReplica` — i.e. every boot where the restore exit code was `0`. `disabled` and every non-zero-exit outcome (`retryTransient`, `recovered`, `unusableReplica`, `fatal`) never `stat` the local DB, so they emit no size. | `state` (`restored`\|`empty`\|`absent`), `bytes` |
|
|
136
|
+
| `SESSION-DB-REPLICA-SIZE` | `probeReplica` (`replica-recovery.ts`), its `ok` branch | Only on attempt 2's probe, i.e. only after a **failed** first restore — see the caveat below. | `objects`, `bytes` |
|
|
137
|
+
|
|
138
|
+
**Format convention:** `<MARKER>: <prose> <k>=<v> … bytes=<n>`, with the machine-readable
|
|
139
|
+
fields appended at the end of the line and `bytes` always last:
|
|
140
|
+
|
|
141
|
+
```
|
|
142
|
+
SESSION-DB-SIZE: <prose> state=<restored|empty|absent> bytes=<n>
|
|
143
|
+
SESSION-DB-REPLICA-SIZE: <prose> objects=<m> bytes=<n>
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Numbers go last because a CloudWatch Logs Insights `parse` capture needs a **literal
|
|
147
|
+
delimiter after every capture except the final one**, which runs to end of line — free
|
|
148
|
+
prose after a capture would make the pattern ambiguous. `diagnostics.test.ts` and
|
|
149
|
+
`replica-recovery.test.ts` each pin a line matching `/ (state=\w+ )?bytes=\d+$/`, so an
|
|
150
|
+
edit that appends prose after the number breaks a test rather than silently breaking an
|
|
151
|
+
operator's saved query.
|
|
152
|
+
|
|
153
|
+
**Logs Insights queries** (copy-pasteable, and the reason the format above is fixed):
|
|
154
|
+
|
|
155
|
+
Session-DB size distribution across the fleet:
|
|
156
|
+
|
|
157
|
+
```
|
|
158
|
+
fields @timestamp, @message
|
|
159
|
+
| parse @message "SESSION-DB-SIZE: * state=* bytes=*" as dbProse, dbState, dbBytes
|
|
160
|
+
| filter ispresent(dbBytes)
|
|
161
|
+
| stats count(*) as boots,
|
|
162
|
+
avg(dbBytes) as avgBytes,
|
|
163
|
+
pct(dbBytes, 50) as p50Bytes,
|
|
164
|
+
pct(dbBytes, 95) as p95Bytes,
|
|
165
|
+
max(dbBytes) as maxBytes
|
|
166
|
+
by bin(1d)
|
|
167
|
+
| sort @timestamp desc
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Growth curve, and how many boots restored nothing:
|
|
171
|
+
|
|
172
|
+
```
|
|
173
|
+
fields @timestamp, @message
|
|
174
|
+
| parse @message "SESSION-DB-SIZE: * state=* bytes=*" as dbProse, dbState, dbBytes
|
|
175
|
+
| filter ispresent(dbState)
|
|
176
|
+
| stats count(*) as boots, max(dbBytes) as maxBytes by dbState, bin(1d)
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
Replica size + object count (ECS, failed-first-restore boots only — see the caveat below):
|
|
180
|
+
|
|
181
|
+
```
|
|
182
|
+
fields @timestamp, @message
|
|
183
|
+
| parse @message "SESSION-DB-REPLICA-SIZE: * objects=* bytes=*" as replicaProse, replicaObjects, replicaBytes
|
|
184
|
+
| filter ispresent(replicaBytes)
|
|
185
|
+
| stats max(replicaBytes) as maxReplicaBytes, max(replicaObjects) as maxObjects by bin(1d)
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
**Caveat: the two markers have very different fleet coverage.** `SESSION-DB-SIZE` fires
|
|
189
|
+
on **every healthy boot on both runtimes** and is the fleet-wide growth-curve signal.
|
|
190
|
+
`SESSION-DB-REPLICA-SIZE` fires **only** on ECS's attempt 2 — i.e. only when a first
|
|
191
|
+
restore already failed — and essentially never on the MicroVM, whose hook hard-codes the
|
|
192
|
+
classify attempt to `1` (`common.sh:352`), which never reaches the probe. Chart
|
|
193
|
+
`SESSION-DB-SIZE` for growth; treat `SESSION-DB-REPLICA-SIZE` as a diagnostic for
|
|
194
|
+
failed-restore boots only, or an empty series will look like a broken signal instead of
|
|
195
|
+
what it is.
|
|
196
|
+
|
|
197
|
+
#### The four `--on-unusable-replica` strategies
|
|
198
|
+
|
|
199
|
+
- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; declined (`targetHealthy`/`noL0Present`/`layoutMismatch`) or a failed delete (`recoveryFailed`) → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone — the real trade-off, and **not** "can destroy a healthy L0 object when an older one is corrupt" (false of merged `main`: the delete is unreachable unless the object's own bytes fail the check). `targetHealthy` also covers an unreadable object (absent evidence must not unlock a delete). Never escalates to `clear`, never prunes twice.
|
|
200
|
+
- **`leave`** — zero S3 mutation → `31`. Cost is not just "prior history lost": `entrypoint.sh` also skips `litestream replicate` for the **whole boot**, so this boot's history is ephemeral too. Upside: replica left byte-intact for forensics, no crash loop.
|
|
201
|
+
- **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
|
|
202
|
+
- **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
|
|
203
|
+
|
|
204
|
+
#### The measured real-world key layout (litestream 0.5.13)
|
|
205
|
+
|
|
206
|
+
`parseLtxKey` (`src/replica-keys.ts`) expects
|
|
207
|
+
`<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
|
|
208
|
+
despite litestream.io's generic docs prose suggesting otherwise (that mismatch was
|
|
209
|
+
#1081: every real key failed to parse, silently disabling `prune` in production).
|
|
210
|
+
Verbatim key sampled live from the production replica:
|
|
211
|
+
`agents/e5d866ac-b699-4940-b369-633b25f8b601/opencode.db/0000/000000000001a978-000000000001a978.ltx`.
|
|
212
|
+
Levels observed on that replica: `0000`/`0001`/`0002`/`0003`/`0009`. The zero-padding is
|
|
213
|
+
an observation, not a contract — `parseLtxKey` accepts an unpadded level directory too.
|
|
214
|
+
|
|
215
|
+
**L0 objects are short-lived.** litestream's own startup log measured a level-0
|
|
216
|
+
retention of **5 minutes** (`starting L0 retention monitor interval=15s retention=5m0s`;
|
|
217
|
+
compaction runs L1=30s, L2=5m, L3=1h, L9=24h). litestream expires L0 objects itself, so
|
|
218
|
+
at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written and sound
|
|
219
|
+
(`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
|
|
220
|
+
and it can never repair corruption at a higher compaction level.
|
|
221
|
+
|
|
222
|
+
`EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
|
|
223
|
+
|
|
224
|
+
See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
|
|
225
|
+
the behavioural anchor (31 comes online, 30 does not).
|
|
226
|
+
|
|
77
227
|
### Why the contract is asymmetric
|
|
78
228
|
|
|
79
229
|
`restore` and `sync-once` **never** report a domain outcome through the exit code. A
|
|
@@ -97,6 +247,8 @@ that it has no auth. `self-stop` decides whether a task may go away, so a broken
|
|
|
97
247
|
must never kill one: `entrypoint.sh` reads any non-zero exactly like `20` and keeps the
|
|
98
248
|
task, and only a `0` — returned solely on a confirmed `desiredCount` of 0 — lets it stop.
|
|
99
249
|
|
|
250
|
+
`session-db-classify` is neither a domain outcome nor a predicate — it's a **third category**: a typed classification with four actionable answers (`0` ran, `32` retry, `31` unusable, `30` fatal), numbered above the two predicates' codes so they can't collide with a future one (`cli.ts:42-47`; see `diagnostics.ts`'s `sessionDbExitCode` for the single outcome → code mapping). The fail-safe direction inverts here: the two predicates above treat an unexpected code as the *safe* answer, but `entrypoint.sh` treats an unexpected code from `session-db-classify` as **fatal** (it `die`s) — correctly, because a boot that can't classify its own replica must not guess about deleting S3 objects.
|
|
251
|
+
|
|
100
252
|
`src/shell-contract.json` is the machine-checked source of truth for the command list, and
|
|
101
253
|
`shell-contract.test.ts` holds **every** shell that speaks it to it — Fargate's
|
|
102
254
|
`packages/runner-image/entrypoint.sh` and the MicroVM's
|
|
@@ -142,10 +294,23 @@ The paths and keys that follow from the bucket/prefix:
|
|
|
142
294
|
| OpenCode DB | `$HOME/.local/share/opencode/opencode.db` | `<prefix>/opencode.db` (litestream) |
|
|
143
295
|
|
|
144
296
|
The session database is replicated by litestream itself; this package only generates its
|
|
145
|
-
config. `env` emits `
|
|
297
|
+
config. `env` emits `PERSISTENCE_BUCKET`, `CLAUDE_CREDS`, `OPENCODE_DB_PATH` and
|
|
146
298
|
`CREDS_SYNC_INTERVAL`, each single-quoted so a value containing shell metacharacters is
|
|
147
299
|
inert when `eval`'d.
|
|
148
300
|
|
|
301
|
+
`entrypoint.sh` asserts, right after it `eval`s this output, that each key is *defined*
|
|
302
|
+
(not merely non-empty) and splits the four into two groups: a key is **required** — the
|
|
303
|
+
shell dies naming it — if it has no safe degradation; it is **tolerated** — the shell logs
|
|
304
|
+
a WARNING naming it and carries on — if it does. `OPENCODE_DB_PATH` and
|
|
305
|
+
`CREDS_SYNC_INTERVAL` are required: both are dereferenced for real work (deleting/creating
|
|
306
|
+
the session DB, sizing the sync-loop sleep), so a guessed value would be actively wrong,
|
|
307
|
+
not merely absent. `PERSISTENCE_BUCKET` and `CLAUDE_CREDS` are tolerated: an empty
|
|
308
|
+
`PERSISTENCE_BUCKET` is already the designed "persistence disabled" state, and
|
|
309
|
+
`CLAUDE_CREDS` is dereferenced only in operator-facing message text. If you change
|
|
310
|
+
`renderEnv` to make a required key conditional, the entrypoint will brick every container
|
|
311
|
+
running that version — the runner installs this package from a floating npm tag (`dev` /
|
|
312
|
+
`latest`), so the shell and the CLI can genuinely be different versions (#779, #821).
|
|
313
|
+
|
|
149
314
|
## Behaviour worth knowing
|
|
150
315
|
|
|
151
316
|
- **A non-empty local file that is not valid JSON is protected**, never overwritten and
|
|
@@ -170,11 +335,15 @@ inert when `eval`'d.
|
|
|
170
335
|
|
|
171
336
|
## The `ObjectStore` port
|
|
172
337
|
|
|
173
|
-
`restore` and `
|
|
174
|
-
the object is absent,
|
|
175
|
-
|
|
338
|
+
`restore`/`sync` and `session-db-classify` talk to a four-method interface: `get(key)`
|
|
339
|
+
returning `null` when the object is absent, `put(key, body)`, `list(prefix)` and
|
|
340
|
+
`delete(key)` for session-DB recovery. `list(prefix)` returns `{ key, size }` entries
|
|
341
|
+
(`ObjectSummary[]`) — every object under `prefix`, size in bytes. It throws on any error
|
|
342
|
+
and never maps one to `[]` (an empty array means "reached the store, nothing there"); the
|
|
343
|
+
adapter owns pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`,
|
|
344
|
+
the typed `NoSuchKey`/`NotFound` errors) is confined to the single adapter in
|
|
176
345
|
`src/s3-object-store.ts`, so the storage backend can be swapped without touching the
|
|
177
|
-
restore/sync logic.
|
|
346
|
+
restore/sync/classify logic.
|
|
178
347
|
|
|
179
348
|
The S3 adapter is the only one that exists. It uses no static credentials — auth is
|
|
180
349
|
whatever the ambient AWS credential chain resolves (on the Evident runner, the ECS task
|
package/dist/cli.js
CHANGED
|
@@ -39615,6 +39615,9 @@ function assertNeverUnusableReplicaReason(value) {
|
|
|
39615
39615
|
function assertNeverFatalCause(value) {
|
|
39616
39616
|
throw new Error(`Unhandled fatal cause: ${JSON.stringify(value)}`);
|
|
39617
39617
|
}
|
|
39618
|
+
function assertNeverNoReplicaLocalDb(value) {
|
|
39619
|
+
throw new Error(`Unhandled noReplica local-DB state: ${JSON.stringify(value)}`);
|
|
39620
|
+
}
|
|
39618
39621
|
function sessionDbExitCode(outcome) {
|
|
39619
39622
|
switch (outcome.kind) {
|
|
39620
39623
|
case "disabled":
|
|
@@ -39643,7 +39646,9 @@ function describeUnusableReplica(reason) {
|
|
|
39643
39646
|
case "attemptsExhausted":
|
|
39644
39647
|
return `${base} (recovery did not make the replica usable after retrying; giving up.)`;
|
|
39645
39648
|
case "layoutMismatch":
|
|
39646
|
-
return `${base} (
|
|
39649
|
+
return `${base} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
|
|
39650
|
+
case "noL0Present":
|
|
39651
|
+
return `${base} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
|
|
39647
39652
|
case "recoveryFailed":
|
|
39648
39653
|
return `${base} (the recovery delete itself failed; see the warning above for the S3 error.)`;
|
|
39649
39654
|
case "targetHealthy":
|
|
@@ -39665,14 +39670,24 @@ function describeSessionDbFatal(outcome, config) {
|
|
|
39665
39670
|
return assertNeverFatalCause(outcome.cause);
|
|
39666
39671
|
}
|
|
39667
39672
|
}
|
|
39673
|
+
function describeNoReplica(localDb) {
|
|
39674
|
+
switch (localDb) {
|
|
39675
|
+
case "absent":
|
|
39676
|
+
return "No opencode.db replica found in the object store; opencode will create a fresh session DB.";
|
|
39677
|
+
case "empty":
|
|
39678
|
+
return "litestream restore succeeded but left a zero-byte opencode.db; opencode will create a fresh session DB.";
|
|
39679
|
+
default:
|
|
39680
|
+
return assertNeverNoReplicaLocalDb(localDb);
|
|
39681
|
+
}
|
|
39682
|
+
}
|
|
39668
39683
|
function describeSessionDbClassification(outcome, config) {
|
|
39669
39684
|
switch (outcome.kind) {
|
|
39670
39685
|
case "disabled":
|
|
39671
39686
|
return "No opencode.db restore attempted; credential persistence is disabled (LITESTREAM_BUCKET/LITESTREAM_PREFIX unset).";
|
|
39672
39687
|
case "restored":
|
|
39673
|
-
return
|
|
39688
|
+
return `SESSION-DB-SIZE: opencode.db restored from the object store. state=restored bytes=${outcome.bytes}`;
|
|
39674
39689
|
case "noReplica":
|
|
39675
|
-
return
|
|
39690
|
+
return `SESSION-DB-SIZE: ${describeNoReplica(outcome.localDb)} state=${outcome.localDb} bytes=0`;
|
|
39676
39691
|
case "retryTransient":
|
|
39677
39692
|
return "opencode.db replica restore failed (transient); retrying.";
|
|
39678
39693
|
case "recovered": {
|
|
@@ -39803,7 +39818,7 @@ function replicaDbPrefix(prefix) {
|
|
|
39803
39818
|
var LTX_FILENAME = /^([0-9a-fA-F]+)-([0-9a-fA-F]+)\.ltx$/;
|
|
39804
39819
|
function parseLtxKey(prefix, key) {
|
|
39805
39820
|
if (key.split("/").includes("..")) return null;
|
|
39806
|
-
const root12 =
|
|
39821
|
+
const root12 = replicaDbPrefix(prefix);
|
|
39807
39822
|
if (!key.startsWith(root12)) return null;
|
|
39808
39823
|
const [levelPart, filename, ...rest] = key.slice(root12.length).split("/");
|
|
39809
39824
|
if (rest.length > 0 || filename === void 0 || !/^\d+$/.test(levelPart)) return null;
|
|
@@ -39826,6 +39841,9 @@ function newestL0Key(prefix, keys) {
|
|
|
39826
39841
|
}
|
|
39827
39842
|
return newest?.key ?? null;
|
|
39828
39843
|
}
|
|
39844
|
+
function hasParsableLtxKey(prefix, keys) {
|
|
39845
|
+
return keys.some((key) => parseLtxKey(prefix, key) !== null);
|
|
39846
|
+
}
|
|
39829
39847
|
function isDeletableReplicaKey(prefix, key) {
|
|
39830
39848
|
if (!key.startsWith(replicaDbPrefix(prefix))) return false;
|
|
39831
39849
|
return !key.split("/").includes("..");
|
|
@@ -49726,7 +49744,7 @@ var S3ObjectStore = class {
|
|
|
49726
49744
|
await this.client.send(new PutObjectCommand({ Bucket: this.bucket, Key: key, Body: body }));
|
|
49727
49745
|
}
|
|
49728
49746
|
async list(prefix) {
|
|
49729
|
-
const
|
|
49747
|
+
const objects = [];
|
|
49730
49748
|
let continuationToken;
|
|
49731
49749
|
do {
|
|
49732
49750
|
const response = await this.client.send(
|
|
@@ -49737,11 +49755,11 @@ var S3ObjectStore = class {
|
|
|
49737
49755
|
})
|
|
49738
49756
|
);
|
|
49739
49757
|
for (const object of response.Contents ?? []) {
|
|
49740
|
-
if (object.Key)
|
|
49758
|
+
if (object.Key) objects.push({ key: object.Key, size: object.Size ?? 0 });
|
|
49741
49759
|
}
|
|
49742
49760
|
continuationToken = response.NextContinuationToken;
|
|
49743
49761
|
} while (continuationToken);
|
|
49744
|
-
return
|
|
49762
|
+
return objects;
|
|
49745
49763
|
}
|
|
49746
49764
|
async delete(key) {
|
|
49747
49765
|
try {
|
|
@@ -49772,20 +49790,38 @@ function isStructurallySoundLtxObject(bytes) {
|
|
|
49772
49790
|
async function probeReplica(store, prefix, log) {
|
|
49773
49791
|
const root12 = replicaDbPrefix(prefix);
|
|
49774
49792
|
try {
|
|
49775
|
-
const
|
|
49776
|
-
|
|
49793
|
+
const objects = await store.list(root12);
|
|
49794
|
+
const objectCount = objects.length;
|
|
49795
|
+
const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
|
|
49796
|
+
logReplicaSize(root12, objectCount, totalBytes, log);
|
|
49797
|
+
return { ok: true, keys: objects.map((object) => object.key) };
|
|
49777
49798
|
} catch (error) {
|
|
49778
49799
|
const detail = describeError(error);
|
|
49779
49800
|
log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
|
|
49780
49801
|
return { ok: false, detail };
|
|
49781
49802
|
}
|
|
49782
49803
|
}
|
|
49804
|
+
function logReplicaSize(root12, objectCount, totalBytes, log) {
|
|
49805
|
+
log(
|
|
49806
|
+
`INFO: SESSION-DB-REPLICA-SIZE: the opencode.db replica under ${root12} objects=${objectCount} bytes=${totalBytes}`
|
|
49807
|
+
);
|
|
49808
|
+
}
|
|
49809
|
+
function logLayoutMismatch(prefix, keys, log) {
|
|
49810
|
+
log(
|
|
49811
|
+
`WARNING: SESSION-DB-REPLICA-LAYOUT-MISMATCH: listed ${keys.length} object(s) under the replica prefix ${replicaDbPrefix(prefix)} but none of them match the expected LTX key layout (<prefix>/opencode.db/<level>/<minTxid>-<maxTxid>.ltx). Sample key: ${keys[0]}. --on-unusable-replica=prune cannot select anything to delete until this is fixed.`
|
|
49812
|
+
);
|
|
49813
|
+
}
|
|
49783
49814
|
async function pruneNewestL0(store, prefix, keys, log) {
|
|
49784
49815
|
const target = newestL0Key(prefix, keys);
|
|
49785
|
-
if (target === null
|
|
49786
|
-
if (
|
|
49787
|
-
|
|
49816
|
+
if (target === null) {
|
|
49817
|
+
if (keys.length > 0 && !hasParsableLtxKey(prefix, keys)) {
|
|
49818
|
+
logLayoutMismatch(prefix, keys, log);
|
|
49819
|
+
return { deleted: null, reason: "layoutMismatch" };
|
|
49788
49820
|
}
|
|
49821
|
+
return { deleted: null, reason: "noL0Present" };
|
|
49822
|
+
}
|
|
49823
|
+
if (!isDeletableReplicaKey(prefix, target)) {
|
|
49824
|
+
log(`WARNING: refusing to prune ${target}: failed the replica-key guard.`);
|
|
49789
49825
|
return { deleted: null, reason: "layoutMismatch" };
|
|
49790
49826
|
}
|
|
49791
49827
|
let bytes;
|
|
@@ -49850,7 +49886,7 @@ async function decideSessionDbRestore(params) {
|
|
|
49850
49886
|
const prefix = config.prefix;
|
|
49851
49887
|
if (exitCode === 0) {
|
|
49852
49888
|
const stat2 = await fileOps.stat(config.opencodeDbPath);
|
|
49853
|
-
return stat2 !== null && stat2.size > 0 ? { kind: "restored" } : { kind: "noReplica" };
|
|
49889
|
+
return stat2 !== null && stat2.size > 0 ? { kind: "restored", bytes: stat2.size } : { kind: "noReplica", localDb: stat2 === null ? "absent" : "empty" };
|
|
49854
49890
|
}
|
|
49855
49891
|
if (store === null) {
|
|
49856
49892
|
return {
|
|
@@ -50104,9 +50140,13 @@ function parseSessionDbClassifyArgs(args) {
|
|
|
50104
50140
|
}
|
|
50105
50141
|
function renderEnv(config) {
|
|
50106
50142
|
return [
|
|
50107
|
-
//
|
|
50108
|
-
//
|
|
50109
|
-
|
|
50143
|
+
// The bucket NAME, kept (not reduced to a boolean) so an operator can
|
|
50144
|
+
// read it; empty exactly when persistence is disabled (both
|
|
50145
|
+
// LITESTREAM_BUCKET and LITESTREAM_PREFIX must be set — see config.ts's
|
|
50146
|
+
// persistenceEnabled). entrypoint.sh gates BOTH the credential-sync
|
|
50147
|
+
// loop and whether it starts litestream replicate on this being
|
|
50148
|
+
// non-empty.
|
|
50149
|
+
`PERSISTENCE_BUCKET=${shellQuote(config.bucket ?? "")}`,
|
|
50110
50150
|
`CLAUDE_CREDS=${shellQuote(config.claude.path)}`,
|
|
50111
50151
|
`OPENCODE_DB_PATH=${shellQuote(config.opencodeDbPath)}`,
|
|
50112
50152
|
`CREDS_SYNC_INTERVAL=${shellQuote(String(config.syncIntervalSeconds))}`
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@evident-ai/runner-synchroniser",
|
|
3
|
-
"version": "0.1.1-dev.
|
|
3
|
+
"version": "0.1.1-dev.9dbd844",
|
|
4
4
|
"description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/cli.js",
|