instar 1.3.658 → 1.3.659

package/upgrades/1.3.659.md

@@ -0,0 +1,29 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+A new pure-signal monitoring sentinel, `StrandedTopicSentinel`, ships behind a DARK dev-gate (`monitoring.strandedTopicSentinel.enabled`, omitted from ConfigDefaults ⇒ live on a development agent, dark on the fleet). It closes a real, proven gap: a Telegram/Slack topic's durable ownership record can name a machine that is **online-by-heartbeat but cannot serve** — quota-walled (`quotaState.blocked`) or adapter-disconnected (`servesChannels` omits the topic's channel) — while a healthy machine holds the lease. Inbound for that topic routes to the owner that can't answer, so it is **silently dead** (outbound still flows from the healthy machine — the "my replies send but the user's messages never arrive" split). The existing `OwnershipReconciler` only force-claims a *provably-DEAD* owner (offline ≥180s) and only iterates *pinned* topics, so a walled-but-online owner defers forever and an unpinned strand is never even considered.
+
+The sentinel scans the in-memory ownership cache against the replicated machine-pool view each tick (default 60s), applies a fail-closed predicate with a persistence gate (the unable-to-serve condition must hold ≥2 rich beats over ≥`dwellMs`, default 30s, so a transient quota/adapter blip can't trip it), and raises ONE aggregated `agent-health` attention item per (owner-machine, stranding-window) — never one-per-topic — plus a separate LOW "can't-assess" item if a heartbeat/schema regression blinds it. It is **lease-holder-only** (so peers don't double-report), a **strict no-op on a single-machine agent**, registers in the GuardRegistry (`GET /guards`), and **MUTATES NOTHING** — no ownership CAS, no pin write, no session kill.
+
+This is the deliberately-safe HALF of the fix. Spec-convergence review (3 rounds) established that the *instinctive* auto-failover is unsafe with today's primitives — there is no per-topic remote-liveness signal, the reachability signal is self-reported and up to `failoverThresholdMs` stale, and there's no hysteresis — so a wrong failover would yank a live conversation mid-reply, which is strictly worse than the bug. The auto-failover is therefore deferred to a tracked v2 (`CMT-1786`) whose prerequisites are each named in the spec.
+
+## What to Tell Your User
+
+Nothing visible day-to-day — it's off everywhere except the development machine until it's soaked. The eventual benefit: a conversation whose messages are silently going to a machine that can't answer them gets caught within about a minute, instead of staying invisible until you notice you're being ignored.
+
+## Summary of New Capabilities
+
+- `monitoring.strandedTopicSentinel.enabled` (dev-gated dark flag): a pure-signal sentinel that detects an online-but-unable-to-serve owner stranding a topic's inbound and raises one aggregated `agent-health` attention item; registered on `GET /guards`.
+- `evaluateStrandedTopics(...)` (pure helper): the unit-testable, fail-closed strand decision (quota arm + best-effort adapter arm + dwell persistence + reconciliation).
+- Deferred (CMT-1786): the auto-failover that actually re-points a stranded topic to a healthy server, plus its named prerequisites (per-topic remote liveness signal, hysteresis/cooldown, claim-time re-assertion, atomic CAS+pin, reason-stamped nonce, OwnershipReconciler unification) and a live userbot harness for real-Telegram UX regression testing.
+
+## Evidence
+
+- **Reproduction (the live incident, 2026-06-24):** 17 of 25 topics were owned by an offline/quota-walled Mac Mini; every one had silently-dead Telegram inbound; every code review and CI gate was green the whole time; the wedge surfaced only when the operator reported missing messages. `GET /pool/placement?topic=N` showed `owner=Mini, pendingReplacement:true` — the signal was present and unmonitored. The only recovery was a human hand-editing `.instar/ownership/local/<topicId>.json`.
+- **Before → after:** before, this class of breakage was invisible to all automated checks until a user got bitten. After (dev-gated), the moment a topic's owner is persistently online-but-unable-to-serve, ONE attention item fires within ~`dwellMs + tickMs` (≈ a minute or two on a healthy heartbeat cadence) naming the stranded topics, the walled owner + reason, whether a servable peer exists, and the signal's staleness. Flag off (the fleet) is byte-identical to today (no sentinel).
+- **Safety verified:** the sentinel is pure signal — it raises an attention item and writes nothing. The fail-closed predicate (missing field / stale beat / underivable scope / pool view unavailable ⇒ SKIP) means an uncertainty can never manufacture a strand; the lease-holder-sole-actor + single-machine no-op prevent duplicate items; the aggregated item rides the existing `AttentionTopicGuard` flood ceiling.
+- **Tests:** 29 unit tests on the pure decision core (both sides of every boundary — quota arm, three-valued adapter arm, persistence/dwell, fail-closed skips, `strandedSince` reconciliation, can't-assess counting, servable-peer, single-machine/non-lease-holder no-op, sync/LLM-free invariant) + GuardRegistry registration + a `/guards`-posture integration check. `tsc --noEmit` clean. Multi-angle spec review (6 internal reviewers + GPT-5.5 cross-model, 3 rounds) drove the design from an unsafe auto-failover to this safe detector and caught a critical predicate bug (the three-valued `machineServesChannel` would have made a `!fn(...)` detector never fire).

package/upgrades/side-effects/stranded-inbound-self-heal.md

@@ -0,0 +1,80 @@
+# Side-Effects Review — StrandedTopicSentinel (online-but-unable-to-serve inbound detector)
+
+**Version / slug:** `stranded-inbound-self-heal`
+**Date:** `2026-06-24`
+**Author:** Echo (autonomous)
+**Second-pass reviewer:** not-required (Tier 2; dev-gated, pure-signal monitoring sentinel — no mutation, no fleet runtime path, no operator surface)
+
+## Summary of the change
+
+Adds a new dark-gated, pure-signal monitoring sentinel, `StrandedTopicSentinel`, that detects a Telegram/Slack topic whose durable ownership record names a machine that is **online-by-heartbeat but cannot serve** (quota-walled `quotaState.blocked`, or adapter-disconnected `servesChannels` omits the topic's channel) while a healthy machine holds the lease — so inbound for that topic is silently dead. It raises ONE aggregated `agent-health` attention item per (owner-machine, stranding-window) and a separate LOW "can't-assess" item if a heartbeat/schema regression blinds it. It **MUTATES NOTHING** (no ownership CAS, no pin write, no session kill); its sole output is an advisory attention item. The instinctive auto-failover was deferred to a tracked v2 (CMT-1786) because spec-convergence review proved it unsafe with today's primitives (no per-topic remote liveness, self-reported stale reachability, no hysteresis) — a wrong failover drops a live conversation, strictly worse than the bug.
+
+Files added:
+- `src/monitoring/strandedTopicDecision.ts` — the PURE, unit-testable decision core (`evaluateStrandedTopics`): fail-closed predicate (quota arm + best-effort adapter arm), dwell persistence, `strandedSince` reconciliation, `servablePeerExists`.
+- `src/monitoring/StrandedTopicSentinel.ts` — the sentinel (tick loop, per-owner stranding-window discipline, attention emission, `guardStatus`).
+- `tests/unit/stranded-topic-sentinel.test.ts` (29), `tests/integration/stranded-topic-guard-posture.test.ts` (4), `tests/e2e/stranded-topic-guards-lifecycle.test.ts` (2).
+- `docs/specs/stranded-inbound-self-heal.md` (+ `.eli16.md` + convergence report), `upgrades/next/stranded-inbound-self-heal.md`.
+
+Files modified:
+- `src/core/types.ts` — `MonitoringConfig.strandedTopicSentinel?` config type.
+- `src/config/ConfigDefaults.ts` — the `strandedTopicSentinel` defaults block (tickMs/dwellMs/freshnessBoundMs/clearAfterTicks); `enabled` OMITTED (dev-gate decides).
+- `src/core/devGatedFeatures.ts` — `DEV_GATED_FEATURES` entry (pure-signal justification).
+- `src/monitoring/guardManifest.ts` — `GUARD_MANIFEST` entry (`expectRuntime`, `expectedTickMs`).
+- `src/core/SessionOwnershipRegistry.ts` — added `all()` (delegates to `store.all?.()`; reads empty for a store lacking it) so the registry the server holds can be scanned.
+- `src/commands/server.ts` — wires the sentinel (late, after pool boot; `selfMachineId` via a lazy getter to avoid the #1190 boot-ordering null-capture; `raiseAttention` → `telegram.createAttentionItem`; GuardRegistry registration).
+- `tests/unit/lint-dev-agent-dark-gate.test.ts` — hand-shifted the golden line-map keys +14 (my ConfigDefaults block added 14 lines, NO new `enabled:` literal).
+
+## Decision-point inventory
+
+- **Added**: the strand predicate (`evaluateStrandedTopics`) — a READ-ONLY verdict, not a gate. It decides whether to RAISE an attention item; it authorizes no mutation. Fail-closed on every uncertainty.
+- **Added**: the lease-holder-sole-actor + single-machine early-no-op gates — decide WHETHER THIS MACHINE evaluates at all (so peers don't double-report). Observe-only.
+- No agent-to-user or ownership mutation decision point is added. This is pure-signal monitoring.
+
+## 1. Over-block
+
+None — the change blocks nothing. It is observe-only. The only "false positive" risk is a spurious attention item, mitigated by: ≥2-rich-beat dwell persistence (kills transient blips), missing-field/stale-beat/underivable-scope ⇒ SKIP (fail-closed — an uncertainty can never manufacture a strand), and lease-holder-only emission (no duplicate items across machines). A spurious item is cheap and rides the existing `AttentionTopicGuard` flood ceiling.
+
+## 2. Under-block
+
+By design, v1 only DETECTS — it does not remediate; the operator/agent acts on the alert (manual remediation documented in the spec; auto-failover is the tracked v2). The adapter arm is best-effort: for Telegram the channel scope is shared adapter config so the adapter arm rarely fires (the quota arm carries the Telegram case — the actual incident); the adapter arm's real value is Slack per-workspace. A dead (offline) owner is intentionally NOT covered (that is the existing OwnershipReconciler Case C's job) — this sentinel covers only the online-but-unable gap.
+
+## 3. Level-of-abstraction fit
+
+Right layer: a `src/monitoring/` sentinel, mirroring the established pattern (ContextWedgeSentinel/ResumeQueue) — deps-injected, `lastTickAt` liveness, GuardRegistry registration, dark-gate via DEV_GATED_FEATURES. The pure decision is extracted to a separate module so it is unit-testable without tmux/HTTP.
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md). FULLY compliant — this is the textbook SIGNAL side. The sentinel holds NO authority: it raises an advisory attention item and writes nothing else (no CAS, no pin, no kill, no direct user message). The deliberate review-driven retreat from a mutating design to a pure detector is exactly the standard's prescription (the dangerous mutation is deferred until the primitives that make it safe exist).
+
+## 5. Interactions
+
+- **OwnershipReconciler (Case C):** disjoint by trigger — Case C covers offline-DEAD+pinned owners; this covers online-but-UNABLE owners (which Case C explicitly defers as `deferredNoEvidence`). No shared writer (this sentinel writes nothing).
+- **AttentionTopicGuard:** the aggregated item rides the existing flood ceiling; NORMAL/LOW priority never bypasses it.
+- **GuardRegistry / GET /guards:** registers so a silently-disabled instance is visible (the exact failure class this feature exists to surface).
+- **Host spawn cap / event loop:** the tick is synchronous, LLM-free, acquires NO spawn-cap slot (asserted by test), and does NO synchronous peer probe — it reads the in-memory ownership cache + replicated heartbeat view only.
+
+## 6. External surfaces
+
+The only external-visible effect is an `agent-health` attention item (and a LOW can't-assess item) when a strand is detected — calm, aggregated, deduped, flood-ceiling-bounded, with signal-staleness disclosed in the text. No new HTTP route, no dashboard change, no message to a user channel.
+
+## 6b. Operator-surface quality (Operator-Surface Quality standard)
+
+No operator surface — not applicable. This change touches no dashboard renderer/markup, approval page, or grant/revoke/secret-drop form. Its only operator-facing output is a plain-language attention item whose text leads with the situation ("inbound for topics X is going to <machine>, which can't serve them; <machine> can"), de-emphasizes identifiers, and discloses signal staleness.
+
+## 7. Multi-machine posture (Cross-Machine Coherence)
+
+**machine-local BY DESIGN.** The detection is per-machine observability over the REPLICATED heartbeat machine-pool view + the local in-memory ownership cache; it writes no durable state, so there is nothing to replicate or strand on a topic transfer. Exactly ONE machine reports per strand: the lease-holder is the sole actor (`syncStatus.holdsLease`), so peers stay observe-only and the user hears one voice. A single-machine agent is a strict no-op (`machines().length < 2` early return — nothing can be stranded on a peer that doesn't exist).
+
+## 8. Rollback cost
+
+Trivial and instant. The feature ships dark on the fleet (`monitoring.strandedTopicSentinel.enabled` omitted ⇒ resolves dark off a dev agent); flag-off is byte-identical to today (no sentinel constructed). On a dev agent, set `monitoring.strandedTopicSentinel.enabled: false` to force-dark. No durable state is written, so there is nothing to clean up on back-out.
+
+## Conclusion
+
+A safe, dark-gated, pure-signal detector that makes a real, proven, previously-invisible cross-machine inbound wedge loud within a bounded window — the deliberately-safe half of the fix, with the dangerous auto-failover deferred behind named, tracked prerequisites (CMT-1786). All three test tiers green; the design passed 3 spec-convergence rounds (6 internal reviewers + GPT-5.5 cross-model).
+
+## Evidence pointers
+
+- Spec + convergence report: `docs/specs/stranded-inbound-self-heal.md`, `docs/specs/reports/stranded-inbound-self-heal-convergence.md`.
+- Live incident (the reproduction): 2026-06-24, 17 of 25 topics stranded on a quota-walled Mac Mini; inbound silently dead; surfaced only when the operator reported missing messages.
+- Tests: unit (29) `tests/unit/stranded-topic-sentinel.test.ts`; integration (4) `tests/integration/stranded-topic-guard-posture.test.ts`; e2e (2) `tests/e2e/stranded-topic-guards-lifecycle.test.ts`.