instar 1.3.42 → 1.3.44

package/upgrades/side-effects/cross-machine-seamlessness-noloss-recovery.md

@@ -0,0 +1,61 @@
+# Side-Effects Review — Cross-Machine Seamlessness: no-loss stuck-message recovery (D-noloss)
+
+**Spec:** docs/specs/CROSS-MACHINE-SEAMLESSNESS-SPEC.md §8 G3a ("Stuck-`processing` recovery")
+
+The complement to the inbound dedup gate. With dedup alone, a crash AFTER claiming an
+inbound event but BEFORE the reply commits would lose the reply (Telegram won't redeliver
+once polled). This adds the spec's explicit "re-run from the stored input by the current
+lease holder" mechanism: a `processing` entry stuck past `maxProcessingMs` is re-routed from
+its `inputSnapshot`, producing the lost reply. Together with the dedup gate this makes the
+flag's guarantee genuinely *exactly-once* (no-loss AND no-duplicate). Still flag-gated dark
+(`multiMachine.exactlyOnceIngress`, default false).
+
+## What changed
+- `src/messaging/stuckMessageRecovery.ts` — NEW. `recoverStuckMessages(deps)`: lease-gated
+  (a standby never injects); `reclaimStuck(maxProcessingMs)` → for each entry with a stored
+  input and `attempts < maxReplayAttempts` (default 3), re-claim under the current epoch and
+  call the injected `reinject(topic, dedupeKey, text)`. Pure orchestration over injected deps.
+- `src/commands/server.ts` — after `server.start()`, ONLY when the ledger is wired (flag on)
+  AND telegram is present: builds `reinject` (sets the per-topic current-inbound key, then
+  routes the stored text via the same `telegram.onTopicMessage` path a fresh forward uses),
+  runs recovery at boot, and on a cadence (`max(30s, maxProcessingMs)`), lease-gated.
+
+## Over-block / under-block
+- OVER-re-run (the risk = re-answering a message the agent legitimately left unanswered, or a
+  persistently-failing turn looping): bounded by (1) only entries stuck past `maxProcessingMs`
+  (a genuinely timed-out turn, not one in flight), (2) `attempts < maxReplayAttempts` cap —
+  after the cap the entry is left alone, never re-run again, (3) lease-gate (only the awake
+  machine), (4) flag-off default. Proven both sides in stuck-message-recovery.test.ts.
+- UNDER-recover: an entry with no `inputSnapshot` can't be replayed (nothing captured) → skipped
+  honestly rather than guessed. Inputs ARE captured by the dedup gate's `decideIngress(input:text)`.
+- The known single-duplicate residual (spec §3 / §8 G3a impossibility floor): a crash AFTER the
+  reply physically sent but BEFORE `reply_committed` persisted → re-run produces one duplicate.
+  This is the documented Two-Generals floor, bounded to one, not a defect.
+
+## Signal vs authority
+- recoverStuckMessages decides which entries to re-run; the re-injection authority is the
+  existing onTopicMessage routing. The module holds no Telegram/server coupling (injected deps).
+
+## Interactions
+- Re-injection sets `currentInboundByTopic[topic]` BEFORE routing so the eventual reply commits
+  the correct entry (same coupling the inbound gate uses) — closing the loop: re-run → reply →
+  commit → not re-run again.
+- Telegram-only for v1 (the `--no-telegram` sessionManager path is a tracked refinement; the
+  block is guarded on `telegram` present).
+- Cadenced re-scan is `unref()`'d (never holds the process open) and lease-gated (a standby's
+  timer no-ops).
+
+## Rollback cost
+- Low. Flag default-off ⇒ the whole block is skipped (messageLedger undefined). Revert = drop the
+  post-start block + the module. No persisted schema beyond the opt-in ledger SQLite.
+
+## Tests
+- `tests/unit/stuck-message-recovery.test.ts` (7): re-runs a stuck entry from stored input;
+  standby no-op; already-replied not re-run; in-flight (within window) not re-run; attempts-cap
+  gives up (no storm); no-input skipped; + boot wiring-integrity (server.ts calls it after
+  server.start(), gated on the ledger + coordinator lease).
+
+## NOT in this increment (tracked, topic-13481)
+- Behavioral boot-recovery e2e (needs a live Telegram adapter) — exercised by the live flag-flip
+  test-as-self, which is the gate before defaulting the flag on. Cross-machine marker (D-xmachine)
+  + CONTINUATION resume (D3) remain.

package/upgrades/side-effects/cross-machine-seamlessness-reply-marker.md

@@ -0,0 +1,65 @@
+# Side-Effects Review — Cross-Machine Seamlessness: reply-marker propagation (D-xmachine)
+
+**Spec:** docs/specs/CROSS-MACHINE-SEAMLESSNESS-SPEC.md §8 G3a ("dual-medium reply_committed marker")
+
+The cross-machine half of exactly-once. The same-machine dedup gate + no-loss recovery handle
+one machine; this closes the window where, AFTER a handoff/failover, a provider redelivery of a
+message the OLD holder already answered arrives at the NEW holder — whose ledger wouldn't know,
+so it would re-answer. The holder now propagates a `reply_committed` marker to standby peers on
+each commit; the standby applies it via `applyRemoteReplyMarker`, so the new holder's dedup gate
+drops the redelivery. Still flag-gated dark (`multiMachine.exactlyOnceIngress`, default false).
+
+## What changed
+- `src/core/ReplyMarkerTransport.ts` — NEW. `broadcast(marker)` signs + POSTs
+  {dedupeKey, platform, replyIdempotencyKey, epoch, topic} to each standby peer's
+  `/api/message-marker` over the authenticated machine channel. NO encryption (the marker
+  carries no conversation content — authentication only). No peers → reachable no-op. Never
+  throws (best-effort; the dedup gate + provider redelivery are the backstop).
+- `src/server/machineRoutes.ts` — `POST /api/message-marker` (machineAuth) + `onReplyMarker`
+  ctx callback. Validates the marker shape; 503 when no receiver (exactly-once off).
+- `src/server/AgentServer.ts` — `replyMarkerTransport?` + `onReplyMarker?` options → routeCtx /
+  machineRoutes.
+- `src/server/routes.ts` — RouteContext.replyMarkerTransport; the outbound commit at
+  `/telegram/reply/:topicId` reads the just-committed `replyIdempotencyKey` from the ledger and
+  fires `replyMarkerTransport.broadcast(...)` (fire-and-forget, after the local commit).
+- `src/commands/server.ts` — constructs `ReplyMarkerTransport` (peer resolver = registry
+  machines with a lastKnownUrl) ONLY when `exactlyOnceIngress` + coordinator.enabled; wires
+  `onReplyMarker` → `messageLedger.applyRemoteReplyMarker`.
+
+## Over-block / under-block
+- The marker NEVER over-marks a fresh event: `applyRemoteReplyMarker` only sets reply_committed
+  on an entry not already acted-on, and the dedupeKey is the same provider-stable id both
+  machines compute — so a marker for event X can only dedup event X. The deterministic
+  `replyIdempotencyKey` (hash(dedupeKey+index)) is identical on both machines.
+- UNDER-propagate (a lost marker): best-effort — if the marker doesn't reach the standby, the
+  provider's at-least-once redelivery + the standby's own ledger (once it catches up) still
+  dedup; the residual is the documented single-duplicate Two-Generals floor, not a new failure.
+- Signed channel: `/api/message-marker` requires machineAuth (unsigned → 401, proven in the e2e);
+  the marker carries no secrets so no encryption is needed.
+
+## Signal vs authority
+- The transport propagates; the receiving ledger's `applyRemoteReplyMarker` is the authority on
+  its own state (only promotes a not-yet-acted entry). The dedup gate remains the single drop authority.
+
+## Interactions
+- Fires AFTER the local `commitInboundReply` (the local guarantee never depends on propagation).
+- Skipped for proxy/system sends (same guard as the commit) — a proxy message isn't a user reply.
+- Peer resolver mirrors the live-tail/handoff resolvers (registry machines with lastKnownUrl,
+  excludes self/revoked). Solo agent → no peers → no-op.
+
+## Rollback cost
+- Low. Flag default-off ⇒ transport not constructed, route 503s, no propagation. Revert = drop
+  the transport + route + the broadcast call. No persisted schema.
+
+## Tests
+- `tests/unit/reply-marker-transport.test.ts` (3): no-peers no-op; signs + POSTs to each peer;
+  resolves false (never throws) when all peers reject.
+- `tests/e2e/reply-marker-cross-machine-e2e.test.ts` (2): a PEER-signed marker over real HTTP +
+  machine-auth applies to the receiver's ledger, then a redelivery of that message is DEDUPED
+  (already-replied) — the cross-machine guarantee end-to-end; + unsigned marker rejected (401).
+
+## Completion status
+- With D-dedup + D-noloss + D-xmachine, the exactly-once guarantee is COMPLETE in all dimensions
+  (no-dup same+cross machine, no-loss on crash), all DARK behind the flag. Remaining before the
+  flag flips: D3 (CONTINUATION resume verify) + the over-Telegram two-machine test-as-self (the
+  live acceptance bar, needs the operator's machines).

package/upgrades/side-effects/source-tree-guard-readsync-bypass.md

@@ -0,0 +1,34 @@
+# Side-Effects Review — SourceTreeGuard read-tier bypass (extend to readSync)
+
+**Driver:** dogfood follow-up to PR #450 (source-tree-guard-data-pull). The narrow `sourceTreeReadOk` opt was added to `execSync` + `spawn` to let `git fetch` work on the agent's own instar checkout. After deploying v1.3.38 to Echo, the next layer of the same gap surfaced: `readSync` also runs source-tree-checks ("defense-in-depth"), and the canonical-ref scan path uses readonly verbs (`rev-parse`, `ls-tree`, `show`, `log`, `merge-base`, `remote -v`) that all go through `readSync` → all blocked on the source tree.
+
+## What changed
+
+- **`src/core/SafeGitExecutor.ts`** —
+  - `readSync` now honors `opts.sourceTreeReadOk` the same way `execSync` / `spawn` do: when the opt is true AND the verb is in `SOURCE_TREE_READ_TIER_VERBS`, skip `runSourceTreeChecks`; audit the bypass with the `sourceTreeReadOk-bypass` reason.
+  - `SOURCE_TREE_READ_TIER_VERBS` extended from `['fetch']` to `['fetch', 'rev-parse', 'ls-tree', 'show', 'log', 'cat-file', 'merge-base', 'remote']`. Still a small closed enumeration; still requires a spec edit to grow. Every verb is read-tier: none mutate the working tree or committed refs.
+- **`src/monitoring/releaseReadinessWiring.ts`** — every `SafeGitExecutor.run` callsite passes `sourceTreeReadOk: true`: `resolveCanonicalRemote` (`remote -v`), `fetchCanonical` (`rev-parse FETCH_HEAD` — fetch already had it), `oldestUnreleasedCommit` (`log`), `isAncestor` (`merge-base --is-ancestor`).
+- **`src/core/featureRolloutScan.ts`** — every `SafeGitExecutor.run` callsite in the canonical scan path passes `sourceTreeReadOk: true`: `rev-parse FETCH_HEAD`, `ls-tree` (spec + trace enumeration), `show` (spec + trace blob reads). `fetch` already had it.
+
+## Side-effects analysis
+
+**Why this is safe.** Every verb in the (still-small, still-closed) `SOURCE_TREE_READ_TIER_VERBS` set is read-tier: it does NOT modify the working tree, any committed ref, or any source file. `fetch` writes only to `FETCH_HEAD` + objects (transient). `rev-parse`, `ls-tree`, `show`, `log`, `cat-file`, `merge-base`, `remote -v/-get-url` are pure-read. The bypass is opt-in per-call, audit-logged, and only effective inside the closed verb allowlist.
+
+**Why the previous fix wasn't enough.** PR-450 only added the bypass to `execSync` + `spawn`, on the assumption that readonly verbs went through `readSync` which "didn't run source-tree checks." That was wrong: `readSync` runs them as defense-in-depth (catches repo-local aliases that could rebind a "read-only" verb). Every readonly verb on the agent's source tree was therefore still being blocked. This PR closes the gap at the right layer.
+
+**Reach.** Two real-code callsites touched (the two new ones added by the spec: Layer B wiring + Layer C scanner). Every existing `SafeGitExecutor.readSync` caller that does NOT pass `sourceTreeReadOk: true` is byte-identically guarded by the prior behavior.
+
+**Rollback.** Reverting this PR re-blocks the canonical-ref readSync calls on source trees. Layer B's fail-loud path keeps signaling on the failure; Layer C's local-scan fallback keeps working. No broken state.
+
+## Testing
+
+`tests/unit/SafeGitExecutor-sourceTreeReadOk.test.ts` (7 tests, all green):
+- default: fetch on non-source-tree works
+- default: fetch on source-tree blocked
+- `sourceTreeReadOk: true` + fetch on source-tree passes
+- `sourceTreeReadOk: true` does NOT bypass non-allowlist verbs (e.g. `add`)
+- `SOURCE_TREE_READ_TIER_VERBS` is a closed read-tier set — no destructive write verbs (commit/push/reset/checkout/rebase/merge/clean/rm/branch/tag); size ≤ 10
+- **readSync path also honors `sourceTreeReadOk` — `rev-parse FETCH_HEAD` on a source tree passes**
+- **readSync path WITHOUT `sourceTreeReadOk` — `rev-parse FETCH_HEAD` on a source tree STILL blocked**
+
+Lint clean.