instar 1.3.710 → 1.3.711

package/src/scaffold/templates/jobs/instar/rope-health-digest.md

@@ -0,0 +1,42 @@
+---
+name: Rope-Health Digest
+description: "Daily one-line mesh rope-health digest (docs/specs/u4-5-rope-health-alerts.md §2). Reads the RopeHealthMonitor's GET /mesh/rope-health surface and, ONLY when something is non-ok, emits at most ONE consolidated section (≤3 sentences, machine-named, content-scrubbed: rope kind + machine nickname + relative times only). Ships enabled:true with a 503-silent body — a DELIBERATE, argued divergence from the feedback-factory-process enabled:false precedent (R-r2-7/R-r3-2): the digest's whole gating already lives in the monitoring.ropeHealth dev-agent gate (dark fleet → the route 503s → this job exits silently at zero cost; live dev agent → the digest flows from day one). Delivery honors monitoring.ropeHealth.digestTopicId (R-r2-8): UNSET → log only, never a send. Tier-1 supervised (this haiku job wraps the deterministic endpoint; the server composes the digest text). Operator note: an agent-home G1 coherence script that audits mesh health should CONSUME GET /mesh/rope-health rather than re-deriving rope state — this monitor is the mechanism; any daily script is a reader."
+schedule: "0 9 * * *"
+priority: low
+expectedDurationMinutes: 2
+model: haiku
+supervision: tier1
+enabled: true
+tags:
+  - cat:monitoring
+  - role:worker
+  - exec:prompt
+gate: curl -sf http://localhost:${INSTAR_PORT:-4042}/health >/dev/null 2>&1
+toolAllowlist: "*"
+unrestrictedTools: true
+mcpAccess: none
+---
+Emit the daily mesh rope-health digest. This is a mechanical, near-silent reporter — when everything is healthy you say NOTHING anywhere.
+
+AUTH="${INSTAR_AUTH_TOKEN:-$(python3 -c "import json; v=json.load(open('.instar/config.json')).get('authToken',''); print(v if isinstance(v, str) else '')" 2>/dev/null)}"
+AGENT_ID="${INSTAR_AGENT_ID:-$(python3 -c "import json; print(json.load(open('.instar/config.json')).get('projectName',''))" 2>/dev/null)}"
+PORT="${INSTAR_PORT:-4042}"
+
+1. Read the monitor (the ?digest=1 read records the emission metric):
+   `curl -s -H "Authorization: Bearer $AUTH" -H "X-Instar-AgentId: $AGENT_ID" "http://localhost:$PORT/mesh/rope-health?digest=1"`
+   - A **503** means the rope-health monitor is dark for this agent (`monitoring.ropeHealth` dev-gated off) — exit silently, there is nothing to do. This is the designed fleet posture, not an error.
+   - On 200 the body is `{ lastEvaluatedAt, peers: [...], keyExpiry, digest, counters }`. The `digest` field is the server-composed, content-scrubbed, ≤3-sentence consolidated section — you never compose rope text yourself.
+
+2. If `digest` is `null`: everything is ok. Exit silently — no message, no log noise.
+
+3. If `digest` is non-null, resolve the delivery topic:
+   `python3 -c "import json; c=json.load(open('.instar/config.json')); print(c.get('monitoring',{}).get('ropeHealth',{}).get('digestTopicId',''))"`
+   - **Empty/unset** (the default): LOG the digest line to stdout (it lands in the job transcript) and exit. Do NOT send it anywhere — the operator has not named a hub topic yet.
+   - **Set to a topic id N**: send EXACTLY the digest text (one consolidated section, nothing appended) to that topic:
+     `cat <<'EOF' | .instar/scripts/telegram-reply.sh N`
+     `<the digest text>`
+     `EOF`
+
+4. **Tier-1 supervision (your job).** Sanity-check before sending: the digest must be ≤3 sentences and must NOT contain raw IPs, URLs, tunnel hostnames, tailnet names, or emails (the server scrubs by construction — if you ever see one, do NOT send; note it once in the transcript and exit; the next day's run re-checks). Never retry-flood a failed send; the failure is recorded server-side.
+
+5. Exit silently. Do not summarize, do not message the user beyond the single digest delivery above.

package/src/scaffold/templates.ts

@@ -511,6 +511,15 @@ Every guard (monitoring sentinels, reapers, the scheduler, …) is graded by wha
 
 - **Multi-transport mesh comms** (\`multiMachine.meshTransport\`): when I run on more than one machine, my machines talk to each other over MULTIPLE ropes — Tailscale, the local network (LAN), and the Cloudflare tunnel — and automatically use whichever is healthy, so a single flaky tunnel no longer makes a machine look unreachable (the root cause of the lease flap). Each machine auto-advertises its reachable addresses; the lease layer hedges across them and verifies the answering machine really is the peer (a replay-proof signed handshake). \`GET /health → multiMachine.syncStatus.meshEndpoints\` lists the rope KINDS this machine advertises. Ships ENABLED (Layers 0-2 are strictly additive; a single-machine agent is a no-op and keeps its localhost bind). When multi-machine, the server also listens on the Tailscale/LAN interfaces so peers can reach it — strictly less exposure than the always-on public tunnel, all routes keep their existing auth. **Proactive trigger:** operator asks "why is my machine unreachable / why does the lease keep flapping?" → the single Cloudflare rope was flapping; multi-transport fixes it (recommend installing Tailscale on both machines for the strongest rope). Kill-switch: \`meshTransport.enabled:false\` (back to single-rope, one restart to apply). A preferred stationary captain can also HOLD the lease alone when its peer is provably gone — that piece (\`leaseSelfHeal.soloCaptainHold\`) ships dark/opt-in. Spec: \`docs/specs/multi-transport-mesh-comms.md\`.
 
+
+### Mesh Rope Health (recovery probe + partition alerts)
+
+Two layers keep my machine-to-machine \"ropes\" (Tailscale / LAN / Cloudflare) honest. **Recovery probe (U4.3):** a rope marked dead no longer stays presumed-dead for a week — an in-server prober rides the ~5s lease tick and re-dials dead ropes with a pinned, signed canary (typed-refusal contract; any-2xx never counts), feeding the ONE health authority so a healed rope closes in minutes. Episode-scoped with a 15-min P19 floor and ONE deduped escalation per episode. **Rope-health alerts (U4.5):** a monitor classifies each peer every 30s — \`ok\` (silence), \`degraded\` (a rope down, another carrying traffic — digest only), \`peer-offline\` (all ropes down AND its heartbeat stopped — a lid-close is NEVER an alarm), \`urgent\` (all ropes down while its git-synced heartbeat still ADVANCES = alive but partitioned → ONE HIGH attention item per episode; honest latency: a genuine partition is confirmed in ~30-90 min, bounded by the heartbeat+sync cadence). A Tailscale key expiring within 14 days warns in the digest.
+- Rope state per (peer, kind): \`curl -H \"Authorization: Bearer $AUTH\" http://localhost:${port}/health\` → \`multiMachine.syncStatus.ropeHealth\` (authed only).
+- The classification + digest: \`curl -H \"Authorization: Bearer $AUTH\" http://localhost:${port}/mesh/rope-health\` (503 = the monitor is dark on this agent). The daily \`rope-health-digest\` job logs the digest; set \`monitoring.ropeHealth.digestTopicId\` to have it delivered.
+- **When to use** (PROACTIVE): \"why did a dead rope come back by itself?\" → the recovery probe (read \`ropeHealth\`); \"is the mesh healthy? / why did I get a partition alert?\" → \`GET /mesh/rope-health\`. Alert text carries rope KIND + machine NICKNAME only — never IPs/tailnet names/emails.
+- Both ship dev-gated (\`multiMachine.meshTransport.recoveryProbeEnabled\`, \`monitoring.ropeHealth.enabled\` — omitted ⇒ live on a development agent, dark on the fleet; probe dry-run first via \`recoveryProbeDryRun\`). Specs: \`docs/specs/u4-3-breaker-recovery-probe.md\`, \`docs/specs/u4-5-rope-health-alerts.md\`.
+
 **Links that survive machine boundaries (WS4.4 — pool-stable private-view links)** — A private-view link (\`/view/:id\`) keeps working no matter WHICH of my machines is fronting the tunnel, even when the content lives on a DIFFERENT machine. The fronting machine resolves the actual HOLDER of the view (view-id ownership ≠ topic ownership — by probing peers, since each view lives on the disk of the machine that made it) and proxies to it. Ships DARK behind \`multiMachine.seamlessness.ws44PoolLinks\` (dev-agent gated); a single-machine agent is a no-op (no peers to proxy to). Security model (what to tell the user if asked "is a shared link safe across my machines?"):
 - The END-USER credential is enforced end-to-end and the HOLDER makes the authorization decision — the fronting machine is a dumb relay. It NEVER substitutes a machine/mesh credential for the user's, NEVER logs the token, and NEVER caches private content at the edge (responses are \`Cache-Control: no-store\`).
 - The user's PIN/token is validated at the fronting edge, then the proxied request carries a SHORT-LIVED, audience-bound (target holder + the exact view id + HTTP method), SINGLE-USE, mesh-signed ASSERTION of that authentication — NOT the raw PIN. Each machine's PIN secret never crosses the boundary. A captured assertion cannot be replayed against another resource, another holder, or reused within its window.

package/upgrades/1.3.711.md

@@ -0,0 +1,79 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+Two mesh-transport reliability features land together (shared health-snapshot seam;
+specs `docs/specs/u4-3-breaker-recovery-probe.md` + `docs/specs/u4-5-rope-health-alerts.md`).
+
+**U4.3 — traffic-independent rope recovery probe.** The hedged mesh dialer starves dead
+ropes: the healthiest rope wins inside the hedge window and the losers are cancelled, so a
+rope marked dead was never re-dialed — and (the second arm) a cancelled recovering dial's
+AbortError was recorded as a real failure, letting the winner perpetually reset the loser's
+recovery streak. That is how a healed Tailscale rope stayed presumed-dead for a week. The
+fix: (1) hedge-abort neutrality in `HttpLeaseTransport` (an abort-after-winner records
+nothing; a real dial failure still records), and (2) a `RopeRecoveryProber` riding the
+existing ~5s lease-pull tick that re-dials dead ropes with a pinned, signed, bogus-uid
+canary (typed-refusal contract — an untyped 2xx never closes a rope) and feeds
+`PeerEndpointResolver.recordResult`, the ONE health authority. Episode-scoped probing (no
+permanent limbo after a partial recovery, no hot-loop on a slow-but-alive rope), P19
+Eternal-Sentinel floor (15 min cap, escalate-once per episode), dry-run-first. Per-(peer,
+kind) state serves on the authed `GET /health` → `multiMachine.syncStatus.ropeHealth`.
+
+**U4.5 — rope-health alerts.** A productized in-server `RopeHealthMonitor` (own bounded 30s
+loop) classifies each peer from the U4.3 snapshot: `ok` / `degraded` (digest-only) /
+`peer-offline` (heartbeat stopped — a lid-close is NEVER urgent) / `urgent` (all ropes down
+while the peer's git-synced heartbeat still ADVANCES — advancement-since-onset semantics,
+honest 30-90 min confirmation latency, ONE HIGH attention item per episode, split-brain item
+wins). Tailscale key expiry warns at 14 days via a bounded hourly `tailscale status --json`
+exec (fixture-registered parser; identifying fields never leave it). New `GET
+/mesh/rope-health` + a daily `rope-health-digest` built-in job (enabled with a 503-silent
+body; delivery only where `monitoring.ropeHealth.digestTopicId` is set). The self-wake
+urgent suppression is HARD-BOUNDED (5 min) because SleepWakeDetector emits false wake
+events under event-loop stalls (audit finding P1-A7) — a spurious sleep signal can delay a
+partition alert, never veto it.
+
+Both ship dev-gated (`multiMachine.meshTransport.recoveryProbeEnabled`,
+`monitoring.ropeHealth.enabled` omitted → live on development agents, dark on the fleet);
+G3 loadBearing guard-manifest entries with 30-day soak windows; single-machine installs are
+strict no-ops.
+
+## What to Tell Your User
+
+When you run me on more than one machine, my machines talk over several network "ropes"
+(Tailscale, local network, Cloudflare). Two quiet failure modes are now fixed: a rope that
+came back to life used to stay marked dead for days — now I re-check dead ropes myself and
+put them back in service within minutes. And when something genuinely goes wrong I tell
+you honestly: a broken-but-recoverable rope shows up in a daily one-line digest, a machine
+that is simply asleep never triggers an alarm, and only a machine that is provably alive
+yet unreachable (a real partition) raises one urgent notice — once, not a flood. These are
+still maturing: they run on development agents first and stay off elsewhere.
+
+## Summary of New Capabilities
+
+- `GET /mesh/rope-health` — per-peer rope classification + episode state + daily digest
+  text (503 while dark).
+- `GET /health` → `multiMachine.syncStatus.ropeHealth` (authed only) — per-(peer, kind)
+  rope health + probe scheduling state.
+- `rope-health-digest` built-in daily job (log-only until
+  `monitoring.ropeHealth.digestTopicId` is set).
+- Config: `multiMachine.meshTransport.recoveryProbe*` knobs; `monitoring.ropeHealth.*`.
+- CLAUDE.md template section "Mesh Rope Health (recovery probe + partition alerts)"
+  (new agents via init, existing agents via migration, Codex/Gemini via shadow markers).
+
+## Evidence
+
+- Unit: 26 U4.3 tests (episode scoping incl. the limbo case, hedge-abort neutrality both
+  sides, typed-contract classifier over captured byte-for-byte /mesh/rpc fixtures, P19
+  simulated-day bound in both all-fail and all-succeed arms, dry-run scheduling honesty) +
+  29 U4.5 tests (classifier both sides of every boundary incl. the lid-close false-alarm
+  arm and the between-heartbeats late upgrade, bounded wake grace with the P1-A7
+  spurious-sleep test, episode-key determinism + adjacent grouping, transition-only writes,
+  state round-trip, content scrub, captured tailscale fixtures).
+- Integration: authed `/health` carries `ropeHealth` while an unauthed caller sees no mesh
+  topology; `GET /mesh/rope-health` 503-dark/200-live with episode-deduped counters.
+- E2E: production-path gate resolution → prober rides the REAL coordinator lease-pull tick
+  (lastProbeAt advances, recordResult reaches the same resolver instance); monitor's own
+  30s loop ticks and tears down; dark → zero probes, no timer, route 503.

package/upgrades/side-effects/u4-rope-probe-alerts.md

@@ -0,0 +1,140 @@
+# Side-Effects Review — U4.3 rope recovery probe + U4.5 rope-health alerts
+
+**Version / slug:** `u4-rope-probe-alerts`
+**Date:** `2026-07-02`
+**Author:** Echo (autonomous)
+**Second-pass reviewer:** not-required (Tier 2; specs converged + approved — codex-cli:gpt-5.5 cross-model review, 4 + 5 iterations; both features dev-gated dark on the fleet)
+
+## Summary of the change
+
+One shared PR for the two converged U4 specs (`docs/specs/u4-3-breaker-recovery-probe.md`,
+`docs/specs/u4-5-rope-health-alerts.md`) — they share the `PeerEndpointResolver.snapshot()`
+seam (U4.3 builds it, U4.5 consumes it).
+
+**U4.3** fixes hedge-winner-abort starvation of dead mesh ropes:
+
+- `src/core/HttpLeaseTransport.ts` — hedge-abort neutrality (R-r2-1): an
+  AbortError caused by the hedge winner's `finish()` records NOTHING (never
+  `recordResult(false)`); a real dial failure still records. New exported
+  `isAbortShapedError`.
+- `src/core/PeerEndpointResolver.ts` — the `snapshot()` read seam (rows are
+  copies; kind+counters only, no URLs).
+- `src/core/RopeRecoveryProber.ts` (new) — episode-scoped prober riding the
+  lease-pull tick; feeds `recordResult` on the ONE health authority; P19
+  Eternal-Sentinel floor + escalate-once; dry-run keeps a shadow streak and
+  never mutates health.
+- `src/core/ropeProbeContract.ts` (new) — the typed G4 canary payload contract +
+  the REGISTERED `parseProbeResponse` classifier (captured byte-for-byte
+  fixtures).
+- `src/core/MultiMachineCoordinator.ts` — `attachLeasePullTickListener` (the
+  carrier; error-isolated) + `attachRopeHealthProvider` → `getSyncStatus()`
+  serves `ropeHealth` on the authed /health branch only.
+- `src/commands/server.ts` — dev-gated wiring (probe client, registry-validated
+  target list, feature metrics `rope-recovery-probe`, attention sink).
+
+**U4.5** productizes rope-health alerting:
+
+- `src/monitoring/RopeHealthMonitor.ts` (new) — own bounded 30s loop;
+  deterministic classifier (ok / degraded / peer-offline / urgent) with the
+  R-r3-1 advancement-since-onset heartbeat discriminator; episode-deduped ONE
+  HIGH item; split-brain-item suppression; sustained-clear; transition-only
+  durable state (`state/rope-health.json`, registered + retention-declared);
+  BOUNDED self-wake grace (P1-A7 hazard documented in-code); detected-not-
+  notified retry; content scrub by construction.
+- `src/core/tailscaleStatusParser.ts` (new) — REGISTERED parser for the hourly
+  bounded `tailscale status --json` exec; only role + KeyExpiry leave it.
+- `src/server/routes.ts` — `GET /mesh/rope-health` (503 when dark; `?digest=1`
+  records the digest-emission metric).
+- `src/scaffold/templates/jobs/instar/rope-health-digest.md` (new) — daily
+  tier-1 job, enabled with a 503-silent body (argued divergence from the
+  feedback-factory precedent, R-r2-7/R-r3-2); log-only until
+  `monitoring.ropeHealth.digestTopicId` is set.
+- Config/registry parity: `types.ts` + `ConfigDefaults.ts` (both features'
+  knobs; `enabled` OMITTED — dev gate), `devGatedFeatures.ts` (ropeRecoveryProbe
+  + ropeHealthAlerts with written justifications), `guardManifest.ts` (both
+  loadBearing G3 entries, 30-day soak), `templates.ts` + `PostUpdateMigrator.ts`
+  (the "Mesh Rope Health" CLAUDE.md section, migrator + shadow markers),
+  `CapabilityIndex.ts` (mesh prefix note), state-coherence registry,
+  `upgrades/next/u4-rope-probe-health-alerts.md`, site docs page.
+
+## Decision-point inventory
+
+- **Modified — `HttpLeaseTransport` attempt catch**: the abort-after-winner
+  discriminator (`signal.aborted && isAbortShapedError`). Fail direction: an
+  ambiguous rejection (aborted signal + abort-shaped error while a winner
+  confirmed) records NEUTRALLY — health truth is fed only by REAL dial
+  evidence. A real failure (timeout fires the merged signal only) still records.
+- **Added — probe eligibility (RopeRecoveryProber episodes)**: opens on dead,
+  survives fail-after-partial-recovery (the limbo fix), closes only on
+  lastKnownGood reclaim; cadence owned by the probe layer in BOTH modes (never
+  the resolver's trivially-true 5s). Bounded by the P19 floor in dry-run too.
+- **Added — probe success classifier (`parseProbeResponse`)**: success is the
+  EXACT typed contract (403 not-router / 200 sender-rejected ack); an untyped
+  2xx, malformed body, auth rejection, or unexpected ACCEPTANCE records as
+  failure — any-2xx can never close a rope.
+- **Added — urgent classifier (RopeHealthMonitor)**: urgent requires all-down +
+  post-onset heartbeat + time-pinned debounce + ≥2 observations; every ambiguous
+  arm (absent rows, unreadable heartbeat, registry-offline, pre-onset beat)
+  fails toward NOT-urgent (peer-offline/unknown). The wake-grace suppressor is
+  HARD-BOUNDED (wakeGraceMaxMs, default 5 min) so the known-false SleepWakeDetector
+  wake events (P1-A7) can delay but never veto a partition alert.
+- **Added — split-brain suppression**: an active split-brain state suppresses the
+  monitor's item (one episode, one ask). Fail direction of an unreadable check:
+  raise (a possible duplicate ask, never a silent partition).
+
+## Roll-up across the seven review dimensions
+
+1. **Over-block**: none. Nothing gates user messages. The tone/coherence gates
+   are untouched. The only "block"-shaped behavior is the probe refusing to
+   count an untyped 2xx as recovery — the conservative direction for a healer.
+2. **Under-block**: the hedge-abort neutrality REMOVES failure records for
+   winner-cancelled dials. Bounded: the neutrality branch requires the hedge
+   controller's signal aborted AND an abort-shaped error; a rope that is
+   genuinely failing still accumulates failures from its own real dial errors,
+   and the prober now guarantees dead ropes are re-verified (strictly more
+   evidence flows than before).
+3. **Level-of-abstraction fit**: health truth stays in the ONE authority
+   (PeerEndpointResolver.HealthRecord); the prober holds SCHEDULING state only;
+   the monitor holds CLASSIFICATION state only; the parser owns the scrub
+   boundary. No second state machine (spec Decision 1).
+4. **Signal-vs-authority compliance**: both features are SIGNALS. The probe
+   feeds evidence into the existing health machinery; the monitor only raises
+   attention items (episode-deduped, split-brain-suppressed) and serves reads.
+   Every error path fails toward silence/no-emit; no silent try/catch (each
+   carries `@silent-fallback-ok` with the fail direction named).
+5. **Interactions**: the probe rides the lease-pull tick via an error-isolated
+   listener (a throwing scan never breaks the pull — e2e-tested); the monitor
+   subscribes to SleepWakeDetector 'wake' (read-only); both read the machine
+   registry/heartbeat read-only. The digest job reads one route. No shared
+   mutable state beyond the resolver's own recordResult funnel.
+6. **External surfaces**: the probe dials peers' OWN registry-validated mesh
+   URLs (resolver.resolve output — URL-shape + LAN-gate validated) with a
+   signed, deliberately-unresolvable canary the peer answers with a typed
+   refusal; it never rides the router-forward funnel, so no user-facing
+   sender-rejection notice can fire. The tailscale exec is local, bounded (5s
+   timeout), absent-CLI-tolerant. Alerts are content-scrubbed (kind + nickname
+   only). Both features dark on the fleet (dev gate); probe additionally
+   dry-run (sends probes, never mutates health).
+7. **Rollback cost**: low. `recoveryProbeEnabled:false` → zero probes, no orphan
+   state (the HealthRecord is the same store traffic feeds);
+   `monitoring.ropeHealth.enabled:false` → monitor inert, route 503s, the state
+   file is inert data. The transport neutrality fix is a one-hunk revert.
+
+## Evidence pointers
+
+- `npx tsc --noEmit` — clean.
+- Unit: `RopeRecoveryProber` 14/14, `hedge-abort-neutrality` 3/3,
+  `ropeProbeContract` 5/5 (captured fixtures), `peer-endpoint-resolver-snapshot`
+  4/4, `RopeHealthMonitor` 22/22, `tailscaleStatusParser` 7/7 (captured
+  fixtures).
+- Integration: `mesh-rope-health-route` 7/7 (authed /health ropeHealth +
+  unauthed exclusion; route dark/live; digest metric; episode-dedup counters).
+- E2E: `rope-recovery-probe-alive` 3/3 (real coordinator lease-pull carrier;
+  recordResult reaches the SAME resolver; dark = zero probes),
+  `rope-health-alerts-lifecycle` 5/5 (production gate path; own 30s loop ticks
+  + tears down; dark = 503 + no timer; real heartbeat file drives the
+  discriminator).
+- Ratchets green: full `npm run lint` battery (incl. dark-gate golden map
+  hand-updated, guard-manifest, scrape-fixture realness, state registry +
+  retention), `feature-delivery-completeness` 111/111, docs-coverage ≥
+  thresholds.