instar 1.3.628 → 1.3.629

package/upgrades/1.3.629.md

@@ -0,0 +1,31 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+The multi-machine "who's awake?" election (the fenced lease coordinator) is now **self-healing**, fixing a live failure class where the mesh could get stuck with NO awake machine and never recover on its own.
+
+- **F1 — lease-tick self-heal (ENABLED by default, safe-by-construction).** Every network call in the lease tick is now bounded by a timeout (`withTickTimeout`), so a single never-settling call can no longer wedge the election loop (the proven 2026-06-19 freeze: a hung call left a reentrancy guard stuck `true`, silencing the tick for 91 minutes while `/health` stayed 200). A monotonic-clocked watchdog re-arms a stalled tick and resets a stuck guard — ceiling-gated so it never preempts a legitimately-slow live tick, never touches authority/epoch state, can't crash the process, and self-disarms (raising one degradation signal) if it fires too often. A true event-loop stall is explicitly delegated to the existing out-of-process fleet watchdog.
+- **F2 — stale-holder takeover (⚗️ EXPERIMENTAL, dark by default).** When enabled, a standby may take over the lease from a holder that has silently stopped renewing — detected on the standby's OWN monotonic clock (the holder's signed nonce watermark stalls), so clock drift between machines can never cause a wrongful takeover. CAS-fenced; off ⇒ byte-for-byte the legacy behavior.
+- **F3 — silent-standby relinquish (⚗️ EXPERIMENTAL, dark by default).** When enabled, a machine muted to silent-standby that still holds the lease relinquishes it and broadcasts a cryptographically SIGNED tombstone, so peers stop deferring to a zombie holder.
+- **F4 — preferred-awake (preview, opt-in; `null` = off).** Name a stationary machine as preferred and a traveling machine defers to it while it's healthy, failing over only when it's genuinely down. Implemented as a deferential standby (the non-preferred machine simply abstains while the preferred is a healthy holder), so a divergent config degrades to the existing baseline rather than flapping.
+
+Observability: `GET /health → multiMachine.syncStatus` gains `leaseTickWatchdog` (`lastTickAgeMs`, `reArmCount`, `disarmed`) and `preferredAwakeMachineId`. Config lives under `multiMachine.leaseSelfHeal`.
+
+## What to Tell Your User
+
+If you run one agent across more than one machine, the part that decides which machine is "in charge" can no longer get permanently stuck — it now times out hung operations and restarts itself. This is on automatically and is a no-op on single-machine setups. Three further robustness behaviors (faster takeover of a dead machine, cleaner hand-off of the "in-charge" badge, and naming a preferred machine) ship turned OFF and are opt-in per machine after verification on a real two-machine setup.
+
+## Summary of New Capabilities
+
+- Self-healing multi-machine awake-election (bounded timeouts + monotonic watchdog) — ON by default, single-machine no-op.
+- `multiMachine.leaseSelfHeal` config: `tickWatchdog` (on), `staleHolderTakeover` (dark), `silentStandbyRelinquish` (dark), `preferredAwakeMachineId` (opt-in), `churnDetector`.
+- `/health` lease-watchdog + preferred-awake visibility.
+
+## Evidence
+
+- ~50 new unit tests green: `tests/unit/FencedLease.test.ts` (signed-tombstone tamper-fail, canonicalize back-compat, F2 gate both-sides + fail-closed), `tests/unit/LeaseCoordinator-selfHeal.test.ts` (F2 takeover, F3 tombstone, F4 health-gate), `tests/unit/MultiMachineCoordinator-tickSelfHeal.test.ts` (F1 bounded-await + watchdog both-sides + self-disarm, F4 deferral).
+- Existing lease/coordinator/transport/store suites regression-clean (129 green); dark-gate lint green (24); tsc clean.
+- Spec: `docs/specs/multi-machine-lease-self-heal.md` (3-round spec-converge, approved). Side-effects: `upgrades/side-effects/multi-machine-lease-self-heal.md`.

package/upgrades/side-effects/multi-machine-lease-self-heal.md

@@ -0,0 +1,45 @@
+# Side-Effects Review — Multi-Machine Lease Self-Heal & Preferred-Awake
+
+Spec: `docs/specs/multi-machine-lease-self-heal.md` (review-convergence 2026-06-20, approved: true). Driven by a LIVE incident on the Echo Mini+Laptop pair (CMT-699).
+
+## Phase 1 — Principle check (signal vs authority)
+
+**Does this change involve a decision point that gates information flow / blocks actions / constrains agent behavior?** YES — it changes the lease coordinator, which decides which machine is "awake" (the authority for ingress/scheduler/state-writes). So `docs/signal-vs-authority.md` applies directly. How each feature lands on the principle:
+
+- **F1 (tick self-heal) — RECOVERY, not a new authority.** The bounded await (`withTickTimeout`) only *bounds* an already-failure-tolerant call. The watchdog only *re-arms a stalled timer* and (ceiling-gated) resets a stuck reentrancy guard; it NEVER decides who is awake, never touches epoch/sign/suspend state, can't crash (try/catch), and self-disarms (one DegradationReporter signal) if it fires too often. It is authority-neutral self-heal — safe to ship enabled.
+- **F2 (stale-holder takeover) — widens *eligibility*, never bypasses fencing.** It only changes *when* a standby may *attempt* a CAS-fenced acquire; the Ed25519 signature + epoch-CAS + git-floor remain the load-bearing authority. The signal it consumes is a LOCALLY-CLOCKED monotonic observation (skew-immune, fail-closed). Ships DARK.
+- **F3 (silent-standby relinquish) — self-only.** A machine relinquishes ONLY its own lease and broadcasts a SIGNED tombstone; no remote-relinquish surface. Ships DARK.
+- **F4 (preferred-awake) — abstention, not a brittle override.** Implemented as a deferential standby (a non-preferred machine simply *abstains* from acquiring while it observes a healthy preferred holder). It adds NO blocking authority and NO tie-break override (see "F4 design refinement" below). Opt-in (null = off).
+
+No feature adds brittle blocking authority. The one always-on piece (F1) is pure recovery.
+
+## F4 design refinement (deviation from the spec, documented)
+
+The approved spec described F4 as TWO effects — (1) a tie-break override in `resolveContestedSplitBrain` and (2) a deferential standby — with an agreement-gossip wire-format to keep the two machines from a divergent-config flap. **This implementation ships ONLY effect (2): the deferential standby + a shared `isHolderHealthy` health-gate.** Rationale: a deferential machine creates NO contention for effect (1) to resolve, so the tie-break override is unnecessary; and because nothing asserts a non-lower-machineId winner, the divergent-config flap the gossip was designed to prevent CANNOT occur (a misconfigured pair degrades to the existing lower-machineId baseline, never worse). This is a strictly SAFER subset of the approved design — fewer moving parts, no new wire format, same operator outcome ("preferred machine wins, traveling machine defers, failover when preferred is down"). It is flagged here and in the PR for reviewer visibility.
+
+## The 8 questions
+
+1. **Over-block (rejects legitimate inputs it shouldn't?)** — F1: the watchdog's ceiling gate means it never preempts a legitimately-slow live tick (tested both-sides). F2: fail-closed — an absent/NaN/never-observed freshness never grants takeover, so a just-booted machine can't wrongly take over. F4: a non-preferred machine never strands coverage — `isHolderHealthy` flips false the moment the preferred is expired/released/absent, and the machine acquires normally. No over-block identified.
+2. **Under-block (failure modes still missed?)** — F1 recovers a hung-await / stuck-guard / lost-timer stall but, by construction, a TRUE event-loop stall freezes the watchdog timer too; that residual is explicitly delegated to the out-of-process fleet/launchd watchdog (documented in the spec). This is a stated scope boundary, not a silent gap.
+3. **Level-of-abstraction fit** — F1/F2/F3/F4 all live in the lease coordinator (`MultiMachineCoordinator` / `LeaseCoordinator` / `FencedLease`), the correct layer that owns "who is awake." F2's takeover rides the EXISTING `canAcquire` + CAS rather than a parallel path; F3's tombstone rides the EXISTING signed-lease broadcast. No higher/lower layer should own these.
+4. **Signal vs authority compliance** — see Phase 1. Compliant: F1 is authority-neutral recovery; F2/F3/F4 widen eligibility / self-relinquish / abstain, all behind the existing fencing authority. No brittle blocking authority added.
+5. **Interactions** — F2's dwell (= `ttl × nonRenewalMissedObservations`, ~6 TTLs) is `>> ESCALATE_AFTER_CYCLES × leasePullIntervalMs` so it cannot race the existing contested-split-brain resolver. F3's tombstone is a higher-nonce `released` record that a higher-epoch takeover always dominates (no re-zombie). F1's guard-reset is ceiling-gated so it cannot double-fire a live tick into the CAS path. F4 only abstains; it adds no contention to shadow/double-fire. `released` is OMITTED-when-false in `canonicalize` so legacy signed leases verify byte-for-byte (no signature-version split with un-upgraded peers).
+6. **External surfaces** — `/health → multiMachine.syncStatus` gains additive read-only fields (`leaseTickWatchdog`, `preferredAwakeMachineId`); the CLAUDE.md template gains an awareness bullet. No outbound message/dispatch surface changes. F2/F3 are dark; F4 is opt-in. The `released` lease field changes the cross-machine wire format additively (back-compat by omit-when-false).
+7. **Multi-machine posture (Cross-Machine Coherence)** — this feature IS the multi-machine lease layer. Posture per piece: F1 watchdog state (`lastTickRunMonoMs`, re-arm counters) is **machine-local BY DESIGN** (about this process's timer health). F2 observation-freshness is machine-local observer state derived from the EXISTING replicated lease pull (**proxied-on-read**). F3 relinquish is a machine-local trigger with a **replicated** signed-tombstone consequence. F4 `preferredAwakeMachineId` is machine-local config (operator sets the same value on all machines); the deferential decision reads the EXISTING observed lease (proxied-on-read). No silent single-machine assumption; a single-machine agent is a strict no-op for every feature.
+8. **Rollback cost** — F1: live off-switch (`leaseSelfHeal.tickWatchdog.enabled:false`, read each fire — no restart). F2/F3: dark by default (just don't enable). F4: opt-in (null = off). The `released` field is additive + back-compat. Worst-case back-out is a config flip or a revert; no data migration, no agent-state repair. ConfigDefaults adds the block via `applyDefaults` (add-missing), so a revert simply stops seeding it.
+
+## Migration parity
+`leaseSelfHeal` is added to the `multiMachine` block in `src/config/ConfigDefaults.ts`; `applyDefaults` deep-merges it onto existing agents on update WITHOUT clobbering operator-set values — so existing agents get F1-enabled + the dark flags on the next update. F2/F3 `enabled:false` are classified in `DARK_GATE_EXCLUSIONS` (action-bearing); the lint golden-path map is updated by hand.
+
+## Tests
+~50 new unit tests across `FencedLease.test.ts` (canonicalize back-compat + tombstone tamper-fail + F2 both-sides + fail-closed), `LeaseCoordinator-selfHeal.test.ts` (F2 takeover wiring, F3 tombstone, F4 `isHolderHealthy`), `MultiMachineCoordinator-tickSelfHeal.test.ts` (F1a bounded-await, F1b watchdog both-sides + self-disarm, F4 deferral). Existing lease/coordinator/transport/store suites regression-clean (129 green). Dark-gate lint green (24). Pre-existing unrelated failure: `autonomous-stop-hook-realcheck.test.ts` UTF-8 case (not in this diff; env/locale-specific; CI authoritative).
+
+## Phase 5 — Second-pass review (independent reviewer)
+
+**Concur with the review.** The reviewer independently verified all 7 risk areas against the actual code (not just the artifact's claims): (1) the F1 watchdog mutates only the two reentrancy booleans + timer/counters — never `acquireIfEligible`/`renew`/`signLease`, never epoch/suspended/selfIssued; (2) the guard-reset ceiling gate (`now - leaseTickStartMonoMs > staleMs`) provably never preempts a live in-flight tick (tested both sides); (3) F2 `canAcquire` is byte-for-byte legacy when the flag is off (`staleHolderTakeover: () => null` ⇒ `undefined` opts ⇒ the block is skipped); (4) the `released` tombstone is Ed25519-signature-covered AND back-compat via omit-when-false (legacy leases verify byte-identically; strip/inject fails verification); (5) F4 never strands coverage (`isHolderHealthy` false on expired/released/absent → the non-preferred machine acquires within ~one TTL); (6) dropping the tie-break override is a strictly safer subset (the abstention branch creates zero contention, so a divergent config degrades to the lower-machineId baseline); (7) single-machine agents are a genuine no-op.
+
+## Known boundary (tracked, captured before any enablement)
+
+The reviewer noted that F3's same-epoch tombstone clears the zombie on a **git-less** LocalLeaseStore pair (the Echo Mini+Laptop target — verified by the unit test) but, on a **git-BACKED** mesh, a tombstone at the shared committed epoch N is not folded over the equal git epoch. F3 ships **hard-dark on every agent** and its live-verification is scoped to the git-less pair, so this is a latent boundary, not a shipped defect. It must be closed (tombstone fold at `>=` epoch with a released-vs-held tie-break, or a dedicated git-floor relinquish) before F3 is ever enabled on a git-backed agent. <!-- tracked: CMT-699 (mesh stability; F3 git-backed enablement gated on closing this boundary) -->
+
+Config validation note: implemented as use-site clamping (`Math.max` floors), not startup-rejection — the safer direction (a bad value floors to a safe minimum on the always-on F1 path) and defense-in-depth at the point of use.