instar 1.3.628 → 1.3.630

package/upgrades/side-effects/multi-machine-lease-self-heal.md

@@ -0,0 +1,45 @@
+# Side-Effects Review — Multi-Machine Lease Self-Heal & Preferred-Awake
+
+Spec: `docs/specs/multi-machine-lease-self-heal.md` (review-convergence 2026-06-20, approved: true). Driven by a LIVE incident on the Echo Mini+Laptop pair (CMT-699).
+
+## Phase 1 — Principle check (signal vs authority)
+
+**Does this change involve a decision point that gates information flow / blocks actions / constrains agent behavior?** YES — it changes the lease coordinator, which decides which machine is "awake" (the authority for ingress/scheduler/state-writes). So `docs/signal-vs-authority.md` applies directly. How each feature lands on the principle:
+
+- **F1 (tick self-heal) — RECOVERY, not a new authority.** The bounded await (`withTickTimeout`) only *bounds* an already-failure-tolerant call. The watchdog only *re-arms a stalled timer* and (ceiling-gated) resets a stuck reentrancy guard; it NEVER decides who is awake, never touches epoch/sign/suspend state, can't crash (try/catch), and self-disarms (one DegradationReporter signal) if it fires too often. It is authority-neutral self-heal — safe to ship enabled.
+- **F2 (stale-holder takeover) — widens *eligibility*, never bypasses fencing.** It only changes *when* a standby may *attempt* a CAS-fenced acquire; the Ed25519 signature + epoch-CAS + git-floor remain the load-bearing authority. The signal it consumes is a LOCALLY-CLOCKED monotonic observation (skew-immune, fail-closed). Ships DARK.
+- **F3 (silent-standby relinquish) — self-only.** A machine relinquishes ONLY its own lease and broadcasts a SIGNED tombstone; no remote-relinquish surface. Ships DARK.
+- **F4 (preferred-awake) — abstention, not a brittle override.** Implemented as a deferential standby (a non-preferred machine simply *abstains* from acquiring while it observes a healthy preferred holder). It adds NO blocking authority and NO tie-break override (see "F4 design refinement" below). Opt-in (null = off).
+
+No feature adds brittle blocking authority. The one always-on piece (F1) is pure recovery.
+
+## F4 design refinement (deviation from the spec, documented)
+
+The approved spec described F4 as TWO effects — (1) a tie-break override in `resolveContestedSplitBrain` and (2) a deferential standby — with an agreement-gossip wire-format to keep the two machines from a divergent-config flap. **This implementation ships ONLY effect (2): the deferential standby + a shared `isHolderHealthy` health-gate.** Rationale: a deferential machine creates NO contention for effect (1) to resolve, so the tie-break override is unnecessary; and because nothing asserts a non-lower-machineId winner, the divergent-config flap the gossip was designed to prevent CANNOT occur (a misconfigured pair degrades to the existing lower-machineId baseline, never worse). This is a strictly SAFER subset of the approved design — fewer moving parts, no new wire format, same operator outcome ("preferred machine wins, traveling machine defers, failover when preferred is down"). It is flagged here and in the PR for reviewer visibility.
+
+## The 8 questions
+
+1. **Over-block (rejects legitimate inputs it shouldn't?)** — F1: the watchdog's ceiling gate means it never preempts a legitimately-slow live tick (tested both-sides). F2: fail-closed — an absent/NaN/never-observed freshness never grants takeover, so a just-booted machine can't wrongly take over. F4: a non-preferred machine never strands coverage — `isHolderHealthy` flips false the moment the preferred is expired/released/absent, and the machine acquires normally. No over-block identified.
+2. **Under-block (failure modes still missed?)** — F1 recovers a hung-await / stuck-guard / lost-timer stall but, by construction, a TRUE event-loop stall freezes the watchdog timer too; that residual is explicitly delegated to the out-of-process fleet/launchd watchdog (documented in the spec). This is a stated scope boundary, not a silent gap.
+3. **Level-of-abstraction fit** — F1/F2/F3/F4 all live in the lease coordinator (`MultiMachineCoordinator` / `LeaseCoordinator` / `FencedLease`), the correct layer that owns "who is awake." F2's takeover rides the EXISTING `canAcquire` + CAS rather than a parallel path; F3's tombstone rides the EXISTING signed-lease broadcast. No higher/lower layer should own these.
+4. **Signal vs authority compliance** — see Phase 1. Compliant: F1 is authority-neutral recovery; F2/F3/F4 widen eligibility / self-relinquish / abstain, all behind the existing fencing authority. No brittle blocking authority added.
+5. **Interactions** — F2's dwell (= `ttl × nonRenewalMissedObservations`, ~6 TTLs) is `>> ESCALATE_AFTER_CYCLES × leasePullIntervalMs` so it cannot race the existing contested-split-brain resolver. F3's tombstone is a higher-nonce `released` record that a higher-epoch takeover always dominates (no re-zombie). F1's guard-reset is ceiling-gated so it cannot double-fire a live tick into the CAS path. F4 only abstains; it adds no contention to shadow/double-fire. `released` is OMITTED-when-false in `canonicalize` so legacy signed leases verify byte-for-byte (no signature-version split with un-upgraded peers).
+6. **External surfaces** — `/health → multiMachine.syncStatus` gains additive read-only fields (`leaseTickWatchdog`, `preferredAwakeMachineId`); the CLAUDE.md template gains an awareness bullet. No outbound message/dispatch surface changes. F2/F3 are dark; F4 is opt-in. The `released` lease field changes the cross-machine wire format additively (back-compat by omit-when-false).
+7. **Multi-machine posture (Cross-Machine Coherence)** — this feature IS the multi-machine lease layer. Posture per piece: F1 watchdog state (`lastTickRunMonoMs`, re-arm counters) is **machine-local BY DESIGN** (about this process's timer health). F2 observation-freshness is machine-local observer state derived from the EXISTING replicated lease pull (**proxied-on-read**). F3 relinquish is a machine-local trigger with a **replicated** signed-tombstone consequence. F4 `preferredAwakeMachineId` is machine-local config (operator sets the same value on all machines); the deferential decision reads the EXISTING observed lease (proxied-on-read). No silent single-machine assumption; a single-machine agent is a strict no-op for every feature.
+8. **Rollback cost** — F1: live off-switch (`leaseSelfHeal.tickWatchdog.enabled:false`, read each fire — no restart). F2/F3: dark by default (just don't enable). F4: opt-in (null = off). The `released` field is additive + back-compat. Worst-case back-out is a config flip or a revert; no data migration, no agent-state repair. ConfigDefaults adds the block via `applyDefaults` (add-missing), so a revert simply stops seeding it.
+
+## Migration parity
+`leaseSelfHeal` is added to the `multiMachine` block in `src/config/ConfigDefaults.ts`; `applyDefaults` deep-merges it onto existing agents on update WITHOUT clobbering operator-set values — so existing agents get F1-enabled + the dark flags on the next update. F2/F3 `enabled:false` are classified in `DARK_GATE_EXCLUSIONS` (action-bearing); the lint golden-path map is updated by hand.
+
+## Tests
+~50 new unit tests across `FencedLease.test.ts` (canonicalize back-compat + tombstone tamper-fail + F2 both-sides + fail-closed), `LeaseCoordinator-selfHeal.test.ts` (F2 takeover wiring, F3 tombstone, F4 `isHolderHealthy`), `MultiMachineCoordinator-tickSelfHeal.test.ts` (F1a bounded-await, F1b watchdog both-sides + self-disarm, F4 deferral). Existing lease/coordinator/transport/store suites regression-clean (129 green). Dark-gate lint green (24). Pre-existing unrelated failure: `autonomous-stop-hook-realcheck.test.ts` UTF-8 case (not in this diff; env/locale-specific; CI authoritative).
+
+## Phase 5 — Second-pass review (independent reviewer)
+
+**Concur with the review.** The reviewer independently verified all 7 risk areas against the actual code (not just the artifact's claims): (1) the F1 watchdog mutates only the two reentrancy booleans + timer/counters — never `acquireIfEligible`/`renew`/`signLease`, never epoch/suspended/selfIssued; (2) the guard-reset ceiling gate (`now - leaseTickStartMonoMs > staleMs`) provably never preempts a live in-flight tick (tested both sides); (3) F2 `canAcquire` is byte-for-byte legacy when the flag is off (`staleHolderTakeover: () => null` ⇒ `undefined` opts ⇒ the block is skipped); (4) the `released` tombstone is Ed25519-signature-covered AND back-compat via omit-when-false (legacy leases verify byte-identically; strip/inject fails verification); (5) F4 never strands coverage (`isHolderHealthy` false on expired/released/absent → the non-preferred machine acquires within ~one TTL); (6) dropping the tie-break override is a strictly safer subset (the abstention branch creates zero contention, so a divergent config degrades to the lower-machineId baseline); (7) single-machine agents are a genuine no-op.
+
+## Known boundary (tracked, captured before any enablement)
+
+The reviewer noted that F3's same-epoch tombstone clears the zombie on a **git-less** LocalLeaseStore pair (the Echo Mini+Laptop target — verified by the unit test) but, on a **git-BACKED** mesh, a tombstone at the shared committed epoch N is not folded over the equal git epoch. F3 ships **hard-dark on every agent** and its live-verification is scoped to the git-less pair, so this is a latent boundary, not a shipped defect. It must be closed (tombstone fold at `>=` epoch with a released-vs-held tie-break, or a dedicated git-floor relinquish) before F3 is ever enabled on a git-backed agent. <!-- tracked: CMT-699 (mesh stability; F3 git-backed enablement gated on closing this boundary) -->
+
+Config validation note: implemented as use-site clamping (`Math.max` floors), not startup-rejection — the safer direction (a bad value floors to a safe minimum on the always-on F1 path) and defense-in-depth at the point of use.

package/upgrades/side-effects/multi-transport-mesh-comms.md

@@ -0,0 +1,70 @@
+# Side-Effects Review — Robust Multi-Transport Mesh Communication
+
+Spec: `docs/specs/multi-transport-mesh-comms.md` (review-convergence 2026-06-20, 5 rounds, approved: true by Justin). Driven by the operator directive (2026-06-20, topic 27515) to make cross-machine comms robust with multiple connections (Cloudflare + Tailscale + LAN). The follow-up to CMT-699 (the lease-self-heal that stopped the freeze; this stops the *flap* by fixing the single-rope fragility underneath).
+
+## Phase 1 — Principle check (signal vs authority)
+
+**Does this change a decision point that gates information flow / blocks actions / constrains agent behavior?** YES for ONE piece (Layer 3 — who holds the awake lease), NO for the rest (transport details).
+
+- **Layers 0-2 (advertise endpoints / resolve+order ropes / hedged failover + accept-ack)** — NOT authority. They change *which rope* carries the same signed lease payload and *whether a rope counts as reached*. They add no blocking authority: a peer is "reachable" exactly when a rope genuinely confirms, and "unreachable" only when ALL ropes fail. The accept-ack is a *correctness* tightening (a rope is reached only on a verified peer response), not a new gate.
+- **Layer 3 (solo-captain hold)** — touches authority (who is awake under partition). It is a **signal-consumer**: it consumes three EXISTING signals (the F4 preferred-awake config, the existing `presumedDeadHolders()` liveness timeout, the existing effective-view epoch) and only ever *withholds a self-suspend* when the peer is provably gone. It adds NO new brittle blocking authority — it never advances the epoch, never takes over a peer's lease, and leaves epoch-CAS + signature fencing untouched. Ships DARK + opt-in.
+
+No feature adds brittle blocking authority. The always-on pieces (Layers 0-2) are pure transport robustness.
+
+## Documented deviations from the converged spec (reviewer visibility)
+
+Three implementation choices deviate from the literal spec text; each is a SAFER or equivalent subset, flagged here:
+
+1. **`meshAckCapable` keys on endpoint-set presence, not a protocol-version bump.** The spec (Decision 9 rolling-deploy gate) implied a version gate. Implementation: a peer that advertises an `endpoints[]` set is running THIS build (the field + the accept-ack receiver ship atomically in this PR), so `meshAckCapable = endpoints.length > 0 && resolvablePubkey`. Rationale: this avoids bumping `SEAMLESSNESS_PROTOCOL_VERSION`, which gates lease *eligibility during a handoff window* — a broader, unrelated effect. Endpoint-presence is an accurate, narrower capability signal. A peer with no endpoints (older build) stays on the 2xx back-compat path (fail-closed-safe).
+
+2. **`endpoints[]` ride the EXISTING authenticated registry-sync path, not a signed heartbeat body.** The spec (Decision 10) called for signing endpoints in the heartbeat body. Implementation: endpoints live in the registry entry (same as `lastKnownUrl`), replicated over the EXISTING registry sync (per-author `syncSequence` + `authoredUnderEpoch` replay guards). The load-bearing spoof defense is moved to the **accept-ack's responder-identity verification**: even if a peer advertised a bogus/spoofed endpoint, dialing it fails the accept-ack (wrong responder identity / no valid sig from the expected machine) → a FAILED rope, never a trusted one. This is a cleaner, equally-strong defense (a spoofed endpoint is inert) consistent with how `lastKnownUrl` already works, and avoids touching the heartbeat-signing path.
+
+3. **Config validation is use-site clamping, not startup-rejection.** The spec (Decision 14) mentioned startup range-validation. Implementation: the resolver/transport read config with `?? default` clamps at the point of use (a bad/missing value floors to a safe default on the always-on path). This matches the F1-F4 precedent ("use-site clamping, not startup-rejection — the safer direction") and means a nonsensical value degrades safely rather than refusing boot.
+
+## The 8 questions
+
+1. **Over-block (rejects legitimate inputs it shouldn't?)** — The resolver drops endpoints by per-kind URL shape (loopback/link-local/metadata/wrong-subnet/wrong-CGNAT). A legitimate rope is never dropped: tailscale=100.64/10, lan=RFC-1918-same-subnet, cloudflare=public-https are exactly the real shapes (verified on the live pair: Mini tailscale 100.64.165.27, LAN 192.168.87.67, cloudflare echo-mini.dawn-tunnel.dev all validate). The hedge confirms over the FIRST working rope, so a slow rope never blocks a fast one. No over-block identified.
+2. **Under-block (failure modes still missed?)** — A peer that is genuinely unreachable on every rope is correctly declared unreachable (feeds the existing renewal-confirmation path). A targeted jammer that silences a *live* peer can drive it into `presumedDeadHolders()` — the spec's safety note covers this: the Layer-3 hold never advances the epoch, so even a wrongly-presumed-gone live peer cannot double-write (epoch-CAS remains the authority). A true event-loop stall still freezes everything (delegated to the F1 watchdog / out-of-process fleet watchdog — out of scope, stated).
+3. **Level-of-abstraction fit** — Resolver (pure ordering), advertiser (discovery), transport (the wire), coordinator (the hold) each live at the right layer. The accept-ack reuses the EXISTING `machineAuth` sign/verify primitives. The bind change lives in `AgentServer` (it owns `listen()`). No higher/lower layer should own these.
+4. **Signal vs authority compliance** — See Phase 1. Layers 0-2 authority-neutral; Layer 3 is a signal-consumer behind the existing fence. No brittle blocking authority added. (`docs/signal-vs-authority.md` referenced.)
+5. **Interactions** — The hedge's per-attempt timeout is the EXISTING undivided `min(ttl/2,30s)` (Decision 6 — NOT divided, so a slow-but-alive peer isn't converted to "no medium", preserving the receiver-stall-envelope rationale). The accept-ack receiver change is back-compat (an old caller ignores the extra fields; a new caller against an old receiver fails closed to "rope not confirmed", never a false confirm). The `recordObserved` return value (new) is additive — the existing void callers are unaffected (TS void-compatible). The bind change is gated on `multiMachine.enabled` so a single-machine agent is byte-for-byte unchanged. No shadowing/double-fire/race identified.
+6. **External surfaces** — `/health → multiMachine.syncStatus.meshEndpoints` gains an additive KIND-ONLY field (raw IPs stay off it). The server, when multi-machine, newly LISTENS on 0.0.0.0 (Tailscale/LAN) — but it is ALREADY publicly reachable via the always-on Cloudflare tunnel, so this is *less* exposure, and every route keeps its existing auth (machine-auth on /api/*, PIN on dashboard, Bearer elsewhere). The CLAUDE.md template gains an awareness bullet. No outbound message/dispatch surface changes.
+7. **Multi-machine posture (Cross-Machine Coherence)** — This feature IS the multi-machine transport. **Endpoint set** = replicated (each machine advertises its own; peers read it — the existing `lastKnownUrl` registry path, extended). **Per-endpoint health** = machine-local BY DESIGN (this machine's view of which rope works; replicating a peer's rope-health would mislead). **Solo-captain-hold decision** = machine-local trigger reading the existing replicated liveness/effective-view (proxied-on-read); its consequence is the already-replicated lease. **Single-machine agent** = strict no-op end to end (no peers ⇒ resolver/transport no-op; no preferred-peer ⇒ Layer 3 no-op; bind stays 127.0.0.1). The `meshEndpoints` /health field is per-machine (own ropes).
+8. **Rollback cost** — Layers 0-2: `multiMachine.meshTransport.enabled:false` → peers resolve to the single `lastKnownUrl` + bind reverts to 127.0.0.1 (one restart; the bind host is read at listen-time). Layer 3: `soloCaptainHold.enabled:false` (default) → `renew()` byte-for-byte today's self-fence. The `endpoints[]` registry field is additive (omit ⇒ single-rope). No data migration, no agent-state repair. Worst-case back-out is a config flip or a revert.
+
+## Migration parity
+
+- `multiMachine.meshTransport` (FLAT knobs) + `leaseSelfHeal.soloCaptainHold` added to `ConfigDefaults`; `applyDefaults` (add-missing) backfills them onto existing agents on update without clobbering operator values — same path the F1-F4 block used.
+- `soloCaptainHold.enabled:false` classified in `DARK_GATE_EXCLUSIONS` (action-bearing, full reason); the lint golden-path map updated by hand (+791 + the +24 downstream shift). Lint green (24/24).
+- CLAUDE.md: `generateClaudeMd()` (new installs) + a content-sniffed `migrateClaudeMd()` append (existing agents) — Agent Awareness Standard.
+- `/health` field is additive.
+
+## Tests (all three tiers)
+
+- **Unit** (79 new): `PeerEndpointResolver.test.ts` (21 — validation/cap/ordering/EWMA stickiness/backoff/eviction), `MeshEndpointAdvertiser.test.ts` (17 — tailscale detect + app-bundle path, LAN pick, compute/advertise), `leaseAckAuth.test.ts` (10 — accept-ack confirm/higher-epoch/replay/wrong-responder/tamper/domain-sep), `HttpLeaseTransport-mesh.test.ts` (10 — hedge/failover/verified-accept/stranger-reject/legacy-backcompat), `LeaseCoordinator-selfHeal.test.ts` (+6 — Layer-3 hold/conservative-suspend/not-preferred/flag-off/higher-epoch).
+- **Integration**: `mesh-accept-ack.test.ts` (5 — real HTTP receiver: confirmed/higher-epoch/bare-200-backcompat/pull-identity/replay-reject), `mesh-failover-2server.test.ts` (2 — TWO real express servers: dead-tailscale→live-LAN hedge confirm + stranger rejection, skips honestly without a private IPv4).
+- **Regression**: all existing lease/transport/fenced/routes suites green (FencedLease 28, LeaseCoordinator 11, tickSelfHeal 10, HttpLeaseTransport 16, StandbyObs 3, sibling-brakes 6, machine-routes 26). dark-gate lint 24. Total 189 across the affected area; tsc clean.
+- **Live-verify (the non-negotiable, runs at deploy)**: on the real Mini+Laptop pair over Tailscale — confirm `meshEndpoints` includes tailscale+lan+cloudflare, force Cloudflare down, assert the lease epoch stays stable ≥15min over the Tailscale/LAN rope. (Layer 3's sever-the-peer live-verify is a SEPARATE gate before its dark flag is ever enabled.)
+
+## Post-CI follow-up — no-silent-fallbacks ratchet
+
+CI flagged 6 new catch blocks via the `no-silent-fallbacks` ratchet. All are genuinely best-effort / fail-closed, not authority swallows: the two accept-ack signature-verify catches (a verify throw ⇒ FAILED rope, fail-closed — never a false confirmation), the two endpoint-URL-parse catches (an unparseable rope is dropped, never dialed), the best-effort endpoint-advertisement catch (the next heartbeat retries; the mesh degrades to the remaining ropes), and the `/health` meshEndpoints read catch (an unreadable registry yields an empty kinds list, never errors the health path). Each was tagged with an in-brace `@silent-fallback-ok` + justification, so the ratchet count drops UNDER the baseline (476) rather than bumping it — keeping the gate honest (it only ever decreases). Comment-only, zero behavior change.
+
+## Post-CI follow-up — F1 watchdog test flake (pre-existing, fixed)
+
+CI surfaced an intermittent failure in `MultiMachineCoordinator-tickSelfHeal.test.ts` (the F1 watchdog tests from CMT-699, which this change builds on): green on one runner, red on another. Root cause: the tests set `lastTickRunMonoMs = 1` (absolute) and rely on `monoNowMs()` (= `process.hrtime.bigint()/1e6`, offset from an ARBITRARY monotonic epoch) being far above the stale threshold — but on a freshly-booted runner the raw monotonic value can be *below* the threshold, so `now - 1` doesn't register as stale and the watchdog no-ops. Fixed test-only by pinning `c.monoNowMs` to a fixed large value in the affected tests (the production code is correct — it always compares two `monoNowMs()` readings, so the epoch offset cancels). Verified 3× locally; tsc clean. No production behavior change.
+
+## Post-CI follow-up 2 — CLAUDE.md section-parity tracking
+The `feature-delivery-completeness` parity test requires every CLAUDE.md section added by `migrateClaudeMd` to be registered in its tracking list. Registered "Multi-transport mesh comms" in `legacyMigratorSections` (it is emitted by BOTH the template and the migrator — a fresh init self-matches, never double-patched — same classification as the lease-self-heal / honest-progress sections). Test-only; 254 meta/parity tests green.
+
+## Phase 5 — Second-pass review (independent reviewer)
+
+**Concur with the review.** The reviewer independently verified all five load-bearing safety claims against the actual worktree code (not just the artifact's claims), with file:line citations:
+
+1. **Layer 3 hold is safe-by-construction.** `renew()` (LeaseCoordinator.ts:636-641) re-signs at `view.epoch` (same epoch — line 602), sets `selfIssued`, returns true — never calls `casWrite`/`buildAcquisition`, never touches a peer's lease, so it cannot advance the epoch or take over. `soloCaptainHoldEligible` (318-327) fails closed on a null dep and requires F4-agreed-preferred + `allPeersPresumedGone()` (aged-out via the existing `failoverThresholdMs`) + no-higher-epoch. A merely-unreachable peer falls through to the monotonic self-fence ⇒ self-suspends. Flag-off ⇒ inert ⇒ byte-for-byte today.
+2. **Accept-ack is fail-closed with real domain separation.** `verifyLeaseAck` (machineAuth.ts:249-273) rejects missing ack/sig (bare-200), wrong machineId (stranger), wrong reqNonce (replay), bad sig, and lower-epoch; 'higher-epoch' is a distinct stand-down and `interpretResponse` only treats `=== 'confirmed'` as confirmed. The ack message is prefixed `mesh-ack-v1|` while a request is `machineId|ts|nonce|seq|bodyHash` — disjoint prefixes prevent a request sig being replayed as an ack. A missing pubkey degrades to back-compat, not a crash.
+3. **0.0.0.0 bind is gated** on `multiMachine.enabled` (AgentServer.ts:3437); single-machine ⇒ 127.0.0.1. Override wins; reversible.
+4. **Legacy single-rope is byte-for-byte** (`legacyDial` + the receiver's bare `{ok:true}`/`{lease}` when no reqNonce).
+5. **The three deviations are equivalent-or-safer** — endpoint-presence is an accurate atomic-ship capability signal; a spoofed endpoint is inert because dialing it fails the responder-identity ack (matching the existing `lastKnownUrl` trust model); use-site clamping floors bad config safely (F1-F4 precedent).
+
+Reinforcing detail recorded: the resolver's host validation (PeerEndpointResolver.ts) dropping loopback/link-local/metadata/0.0.0.0 + enforcing per-kind IP shape is what keeps the new 0.0.0.0 bind from being reachable via a maliciously-advertised internal-target rope — the safety case for claim 3 depends on that filter, which is present and correct.