@evident-ai/runner-synchroniser 0.1.1-dev.96bc2ed → 0.1.1-dev.9dbd844

package/README.md

@@ -115,7 +115,9 @@ Implementation-facing reference for the 7th command: it decides what a just-run
 | `recovered`                                     | `32` | attempt 2, a successful `prune` (1 key) or any `clear` (including a partial one)  |
 | `unusableReplica(strategy)`                     | `31` | attempt 2, `leave`                                                                |
 | `unusableReplica(attemptsExhausted)`            | `31` | attempt ≥ 3, before the probe                                                     |
-| `unusableReplica(targetHealthy\|layoutMismatch)` | `31` | attempt 2, `prune` declined to delete anything                                   |
+| `unusableReplica(targetHealthy)`                | `31` | attempt 2, `prune`; the newest L0 isn't provably corrupt, so it declined to delete it |
+| `unusableReplica(noL0Present)`                  | `31` | attempt 2, `prune`; no level-0 object was present to prune — **normal**, see below |
+| `unusableReplica(layoutMismatch)`               | `31` | attempt 2, `prune`; objects exist under the prefix but **none** parse as an LTX key at all — **a bug**, see below |
 | `unusableReplica(recoveryFailed)`               | `31` | attempt 2, `prune`'s delete itself threw                                         |
 | `fatal(misconfig)`                              | `30` | no object store while persistence is enabled, or the probe failed on attempt 2   |
 | `fatal(deliberate)`                             | `30` | attempt 2, `crash`                                                                |
@@ -124,16 +126,103 @@ Implementation-facing reference for the 7th command: it decides what a just-run
 
 The probe is a `list`, not a `get`: S3 answers a wrong bucket name with `NoSuchBucket`, also a 404, so a `get`-based probe would read a misconfigured bucket as healthy and unlock recovery against it.
 
+#### Boot size signals
+
+Two markers report the numbers a boot already computes but used to throw away (#1113):
+
+| Marker | Emitted from | When it fires | Fields |
+| --- | --- | --- | --- |
+| `SESSION-DB-SIZE` | `describeSessionDbClassification` (`diagnostics.ts`) | Once per `session-db-classify` invocation that reaches `restored` or `noReplica` — i.e. every boot where the restore exit code was `0`. `disabled` and every non-zero-exit outcome (`retryTransient`, `recovered`, `unusableReplica`, `fatal`) never `stat` the local DB, so they emit no size. | `state` (`restored`\|`empty`\|`absent`), `bytes` |
+| `SESSION-DB-REPLICA-SIZE` | `probeReplica` (`replica-recovery.ts`), its `ok` branch | Only on attempt 2's probe, i.e. only after a **failed** first restore — see the caveat below. | `objects`, `bytes` |
+
+**Format convention:** `<MARKER>: <prose> <k>=<v> … bytes=<n>`, with the machine-readable
+fields appended at the end of the line and `bytes` always last:
+
+```
+SESSION-DB-SIZE: <prose> state=<restored|empty|absent> bytes=<n>
+SESSION-DB-REPLICA-SIZE: <prose> objects=<m> bytes=<n>
+```
+
+Numbers go last because a CloudWatch Logs Insights `parse` capture needs a **literal
+delimiter after every capture except the final one**, which runs to end of line — free
+prose after a capture would make the pattern ambiguous. `diagnostics.test.ts` and
+`replica-recovery.test.ts` each pin a line matching `/ (state=\w+ )?bytes=\d+$/`, so an
+edit that appends prose after the number breaks a test rather than silently breaking an
+operator's saved query.
+
+**Logs Insights queries** (copy-pasteable, and the reason the format above is fixed):
+
+Session-DB size distribution across the fleet:
+
+```
+fields @timestamp, @message
+| parse @message "SESSION-DB-SIZE: * state=* bytes=*" as dbProse, dbState, dbBytes
+| filter ispresent(dbBytes)
+| stats count(*) as boots,
+        avg(dbBytes) as avgBytes,
+        pct(dbBytes, 50) as p50Bytes,
+        pct(dbBytes, 95) as p95Bytes,
+        max(dbBytes) as maxBytes
+  by bin(1d)
+| sort @timestamp desc
+```
+
+Growth curve, and how many boots restored nothing:
+
+```
+fields @timestamp, @message
+| parse @message "SESSION-DB-SIZE: * state=* bytes=*" as dbProse, dbState, dbBytes
+| filter ispresent(dbState)
+| stats count(*) as boots, max(dbBytes) as maxBytes by dbState, bin(1d)
+```
+
+Replica size + object count (ECS, failed-first-restore boots only — see the caveat below):
+
+```
+fields @timestamp, @message
+| parse @message "SESSION-DB-REPLICA-SIZE: * objects=* bytes=*" as replicaProse, replicaObjects, replicaBytes
+| filter ispresent(replicaBytes)
+| stats max(replicaBytes) as maxReplicaBytes, max(replicaObjects) as maxObjects by bin(1d)
+```
+
+**Caveat: the two markers have very different fleet coverage.** `SESSION-DB-SIZE` fires
+on **every healthy boot on both runtimes** and is the fleet-wide growth-curve signal.
+`SESSION-DB-REPLICA-SIZE` fires **only** on ECS's attempt 2 — i.e. only when a first
+restore already failed — and essentially never on the MicroVM, whose hook hard-codes the
+classify attempt to `1` (`common.sh:352`), which never reaches the probe. Chart
+`SESSION-DB-SIZE` for growth; treat `SESSION-DB-REPLICA-SIZE` as a diagnostic for
+failed-restore boots only, or an empty series will look like a broken signal instead of
+what it is.
+
 #### The four `--on-unusable-replica` strategies
 
-- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; declined (`targetHealthy`/`layoutMismatch`) or a failed delete (`recoveryFailed`) → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone — the real trade-off, and **not** "can destroy a healthy L0 object when an older one is corrupt" (false of merged `main`: the delete is unreachable unless the object's own bytes fail the check). `targetHealthy` also covers an unreadable object (absent evidence must not unlock a delete). Never escalates to `clear`, never prunes twice.
+- **`prune` (default)** — ≤1 guarded delete iff the newest L0's own bytes fail the LTX header check (`length >= 100` and magic `"LTX1"`) → `recovered`/`32`; declined (`targetHealthy`/`noL0Present`/`layoutMismatch`) or a failed delete (`recoveryFailed`) → `31`. **Header-only** check: a valid header with deeper corruption is judged sound and left alone — the real trade-off, and **not** "can destroy a healthy L0 object when an older one is corrupt" (false of merged `main`: the delete is unreachable unless the object's own bytes fail the check). `targetHealthy` also covers an unreadable object (absent evidence must not unlock a delete). Never escalates to `clear`, never prunes twice.
 - **`leave`** — zero S3 mutation → `31`. Cost is not just "prior history lost": `entrypoint.sh` also skips `litestream replicate` for the **whole boot**, so this boot's history is ephemeral too. Upside: replica left byte-intact for forensics, no crash loop.
 - **`clear`** — deletes every key under `<prefix>/opencode.db/` passing `isDeletableReplicaKey`; that guard, not the `list()` prefix, is the boundary (IAM grants `s3:DeleteObject*` bucket-wide). One `try` per key. A **partial** clear still reports `recovered`/`32`. Cost: all saved history, unconditionally.
 - **`crash`** — first in the switch, no S3 mutation even considered → `fatal(deliberate)`/`30` → `entrypoint.sh` `die`s → task replaced → **crash loop** until an operator intervenes.
 
+#### The measured real-world key layout (litestream 0.5.13)
+
+`parseLtxKey` (`src/replica-keys.ts`) expects
+`<prefix>/opencode.db/<level:04d>/<minTxid>-<maxTxid>.ltx` — **no `ltx/` path segment**,
+despite litestream.io's generic docs prose suggesting otherwise (that mismatch was
+#1081: every real key failed to parse, silently disabling `prune` in production).
+Verbatim key sampled live from the production replica:
+`agents/e5d866ac-b699-4940-b369-633b25f8b601/opencode.db/0000/000000000001a978-000000000001a978.ltx`.
+Levels observed on that replica: `0000`/`0001`/`0002`/`0003`/`0009`. The zero-padding is
+an observation, not a contract — `parseLtxKey` accepts an unpadded level directory too.
+
+**L0 objects are short-lived.** litestream's own startup log measured a level-0
+retention of **5 minutes** (`starting L0 retention monitor interval=15s retention=5m0s`;
+compaction runs L1=30s, L2=5m, L3=1h, L9=24h). litestream expires L0 objects itself, so
+at boot the newest L0 is normally *absent* (`noL0Present`) or freshly written and sound
+(`targetHealthy`) — `prune`'s real-world reach is narrower than the code alone suggests,
+and it can never repair corruption at a higher compaction level.
+
 `EVIDENT_ON_UNUSABLE_REPLICA` is a `packages/runner-image` (entrypoint) variable translated into the flag above at boot — this CLI never reads it, so it has no row in this README's Configuration table below; see [runner-image's README](../runner-image/README.md#when-the-replica-is-unusable-evident_on_unusable_replica). An unrecognised value (including wrong casing, e.g. `Prune`) exits `2`, outside `entrypoint.sh`'s `0|10|20|30|31|32` allow-list — a typo **crash-loops the task on attempt 1** rather than falling back to `prune`, and also logs the misleading "credential persistence is DEGRADED" ERROR.
 
-See `specs/local-runner.feature:49-70` for the behavioural anchor (31 comes online, 30 does not).
+See `specs/local-runner.feature`'s "Recovering session history at startup" scenarios for
+the behavioural anchor (31 comes online, 30 does not).
 
 ### Why the contract is asymmetric
 
@@ -248,10 +337,11 @@ running that version — the runner installs this package from a floating npm ta
 
 `restore`/`sync` and `session-db-classify` talk to a four-method interface: `get(key)`
 returning `null` when the object is absent, `put(key, body)`, `list(prefix)` and
-`delete(key)` for session-DB recovery. `list` throws on any error and never maps one to
-`[]` (an empty array means "reached the store, nothing there"); the adapter owns
-pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`, the typed
-`NoSuchKey`/`NotFound` errors) is confined to the single adapter in
+`delete(key)` for session-DB recovery. `list(prefix)` returns `{ key, size }` entries
+(`ObjectSummary[]`) — every object under `prefix`, size in bytes. It throws on any error
+and never maps one to `[]` (an empty array means "reached the store, nothing there"); the
+adapter owns pagination, and `delete` is idempotent. S3 vocabulary (`@aws-sdk/client-s3`,
+the typed `NoSuchKey`/`NotFound` errors) is confined to the single adapter in
 `src/s3-object-store.ts`, so the storage backend can be swapped without touching the
 restore/sync/classify logic.
 

package/dist/cli.js

@@ -39615,6 +39615,9 @@ function assertNeverUnusableReplicaReason(value) {
 function assertNeverFatalCause(value) {
   throw new Error(`Unhandled fatal cause: ${JSON.stringify(value)}`);
 }
+function assertNeverNoReplicaLocalDb(value) {
+  throw new Error(`Unhandled noReplica local-DB state: ${JSON.stringify(value)}`);
+}
 function sessionDbExitCode(outcome) {
   switch (outcome.kind) {
     case "disabled":
@@ -39643,7 +39646,9 @@ function describeUnusableReplica(reason) {
     case "attemptsExhausted":
       return `${base} (recovery did not make the replica usable after retrying; giving up.)`;
     case "layoutMismatch":
-      return `${base} (no level-0 LTX object was found under the replica prefix, so --on-unusable-replica=prune had nothing matching its expected layout to delete.)`;
+      return `${base} (objects exist under the replica prefix but NONE of them match the LTX key layout --on-unusable-replica=prune expects \u2014 see the SESSION-DB-REPLICA-LAYOUT-MISMATCH warning above for a sample key. This means our key parsing is wrong, or litestream's on-disk layout changed.)`;
+    case "noL0Present":
+      return `${base} (--on-unusable-replica=prune found no level-0 object to prune. This is expected: litestream expires level-0 objects itself after a few minutes, so at boot the newest one is normally absent. prune only ever targets level 0, so it cannot repair corruption at a higher compaction level.)`;
     case "recoveryFailed":
       return `${base} (the recovery delete itself failed; see the warning above for the S3 error.)`;
     case "targetHealthy":
@@ -39665,14 +39670,24 @@ function describeSessionDbFatal(outcome, config) {
       return assertNeverFatalCause(outcome.cause);
   }
 }
+function describeNoReplica(localDb) {
+  switch (localDb) {
+    case "absent":
+      return "No opencode.db replica found in the object store; opencode will create a fresh session DB.";
+    case "empty":
+      return "litestream restore succeeded but left a zero-byte opencode.db; opencode will create a fresh session DB.";
+    default:
+      return assertNeverNoReplicaLocalDb(localDb);
+  }
+}
 function describeSessionDbClassification(outcome, config) {
   switch (outcome.kind) {
     case "disabled":
       return "No opencode.db restore attempted; credential persistence is disabled (LITESTREAM_BUCKET/LITESTREAM_PREFIX unset).";
     case "restored":
-      return "opencode.db restored from the object store.";
+      return `SESSION-DB-SIZE: opencode.db restored from the object store. state=restored bytes=${outcome.bytes}`;
     case "noReplica":
-      return "No opencode.db replica found in the object store; opencode will create a fresh session DB.";
+      return `SESSION-DB-SIZE: ${describeNoReplica(outcome.localDb)} state=${outcome.localDb} bytes=0`;
     case "retryTransient":
       return "opencode.db replica restore failed (transient); retrying.";
     case "recovered": {
@@ -39803,7 +39818,7 @@ function replicaDbPrefix(prefix) {
 var LTX_FILENAME = /^([0-9a-fA-F]+)-([0-9a-fA-F]+)\.ltx$/;
 function parseLtxKey(prefix, key) {
   if (key.split("/").includes("..")) return null;
-  const root12 = `${replicaDbPrefix(prefix)}ltx/`;
+  const root12 = replicaDbPrefix(prefix);
   if (!key.startsWith(root12)) return null;
   const [levelPart, filename, ...rest] = key.slice(root12.length).split("/");
   if (rest.length > 0 || filename === void 0 || !/^\d+$/.test(levelPart)) return null;
@@ -39826,6 +39841,9 @@ function newestL0Key(prefix, keys) {
   }
   return newest?.key ?? null;
 }
+function hasParsableLtxKey(prefix, keys) {
+  return keys.some((key) => parseLtxKey(prefix, key) !== null);
+}
 function isDeletableReplicaKey(prefix, key) {
   if (!key.startsWith(replicaDbPrefix(prefix))) return false;
   return !key.split("/").includes("..");
@@ -49726,7 +49744,7 @@ var S3ObjectStore = class {
     await this.client.send(new PutObjectCommand({ Bucket: this.bucket, Key: key, Body: body }));
   }
   async list(prefix) {
-    const keys = [];
+    const objects = [];
     let continuationToken;
     do {
       const response = await this.client.send(
@@ -49737,11 +49755,11 @@ var S3ObjectStore = class {
         })
       );
       for (const object of response.Contents ?? []) {
-        if (object.Key) keys.push(object.Key);
+        if (object.Key) objects.push({ key: object.Key, size: object.Size ?? 0 });
       }
       continuationToken = response.NextContinuationToken;
     } while (continuationToken);
-    return keys;
+    return objects;
   }
   async delete(key) {
     try {
@@ -49772,20 +49790,38 @@ function isStructurallySoundLtxObject(bytes) {
 async function probeReplica(store, prefix, log) {
   const root12 = replicaDbPrefix(prefix);
   try {
-    const keys = await store.list(root12);
-    return { ok: true, keys };
+    const objects = await store.list(root12);
+    const objectCount = objects.length;
+    const totalBytes = objects.reduce((sum, object) => sum + object.size, 0);
+    logReplicaSize(root12, objectCount, totalBytes, log);
+    return { ok: true, keys: objects.map((object) => object.key) };
   } catch (error) {
     const detail = describeError(error);
     log(`WARNING: could not list the replica prefix ${root12}: ${detail}`);
     return { ok: false, detail };
   }
 }
+function logReplicaSize(root12, objectCount, totalBytes, log) {
+  log(
+    `INFO: SESSION-DB-REPLICA-SIZE: the opencode.db replica under ${root12} objects=${objectCount} bytes=${totalBytes}`
+  );
+}
+function logLayoutMismatch(prefix, keys, log) {
+  log(
+    `WARNING: SESSION-DB-REPLICA-LAYOUT-MISMATCH: listed ${keys.length} object(s) under the replica prefix ${replicaDbPrefix(prefix)} but none of them match the expected LTX key layout (<prefix>/opencode.db/<level>/<minTxid>-<maxTxid>.ltx). Sample key: ${keys[0]}. --on-unusable-replica=prune cannot select anything to delete until this is fixed.`
+  );
+}
 async function pruneNewestL0(store, prefix, keys, log) {
   const target = newestL0Key(prefix, keys);
-  if (target === null || !isDeletableReplicaKey(prefix, target)) {
-    if (target !== null) {
-      log(`WARNING: refusing to prune ${target}: failed the replica-key guard.`);
+  if (target === null) {
+    if (keys.length > 0 && !hasParsableLtxKey(prefix, keys)) {
+      logLayoutMismatch(prefix, keys, log);
+      return { deleted: null, reason: "layoutMismatch" };
     }
+    return { deleted: null, reason: "noL0Present" };
+  }
+  if (!isDeletableReplicaKey(prefix, target)) {
+    log(`WARNING: refusing to prune ${target}: failed the replica-key guard.`);
     return { deleted: null, reason: "layoutMismatch" };
   }
   let bytes;
@@ -49850,7 +49886,7 @@ async function decideSessionDbRestore(params) {
   const prefix = config.prefix;
   if (exitCode === 0) {
     const stat2 = await fileOps.stat(config.opencodeDbPath);
-    return stat2 !== null && stat2.size > 0 ? { kind: "restored" } : { kind: "noReplica" };
+    return stat2 !== null && stat2.size > 0 ? { kind: "restored", bytes: stat2.size } : { kind: "noReplica", localDb: stat2 === null ? "absent" : "empty" };
   }
   if (store === null) {
     return {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evident-ai/runner-synchroniser",
-  "version": "0.1.1-dev.96bc2ed",
+  "version": "0.1.1-dev.9dbd844",
   "description": "Restores and syncs the Evident runner's OpenCode credential stores (and litestream config) to an object store, so a runner survives task replacement with almost no state loss.",
   "type": "module",
   "main": "./dist/cli.js",