instar 1.3.709 → 1.3.711
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +228 -1
- package/dist/commands/server.js.map +1 -1
- package/dist/config/ConfigDefaults.d.ts.map +1 -1
- package/dist/config/ConfigDefaults.js +25 -0
- package/dist/config/ConfigDefaults.js.map +1 -1
- package/dist/core/HttpLeaseTransport.d.ts +9 -0
- package/dist/core/HttpLeaseTransport.d.ts.map +1 -1
- package/dist/core/HttpLeaseTransport.js +28 -2
- package/dist/core/HttpLeaseTransport.js.map +1 -1
- package/dist/core/MultiMachineCoordinator.d.ts +44 -0
- package/dist/core/MultiMachineCoordinator.d.ts.map +1 -1
- package/dist/core/MultiMachineCoordinator.js +52 -0
- package/dist/core/MultiMachineCoordinator.js.map +1 -1
- package/dist/core/PeerEndpointResolver.d.ts +0 -0
- package/dist/core/PeerEndpointResolver.d.ts.map +1 -1
- package/dist/core/PeerEndpointResolver.js +0 -0
- package/dist/core/PeerEndpointResolver.js.map +1 -1
- package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
- package/dist/core/PostUpdateMigrator.js +19 -0
- package/dist/core/PostUpdateMigrator.js.map +1 -1
- package/dist/core/RopeRecoveryProber.d.ts +126 -0
- package/dist/core/RopeRecoveryProber.d.ts.map +1 -0
- package/dist/core/RopeRecoveryProber.js +266 -0
- package/dist/core/RopeRecoveryProber.js.map +1 -0
- package/dist/core/devGatedFeatures.d.ts.map +1 -1
- package/dist/core/devGatedFeatures.js +12 -0
- package/dist/core/devGatedFeatures.js.map +1 -1
- package/dist/core/ropeProbeContract.d.ts +56 -0
- package/dist/core/ropeProbeContract.d.ts.map +1 -0
- package/dist/core/ropeProbeContract.js +92 -0
- package/dist/core/ropeProbeContract.js.map +1 -0
- package/dist/core/tailscaleStatusParser.d.ts +46 -0
- package/dist/core/tailscaleStatusParser.d.ts.map +1 -0
- package/dist/core/tailscaleStatusParser.js +80 -0
- package/dist/core/tailscaleStatusParser.js.map +1 -0
- package/dist/core/types.d.ts +56 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/monitoring/RopeHealthMonitor.d.ts +243 -0
- package/dist/monitoring/RopeHealthMonitor.d.ts.map +1 -0
- package/dist/monitoring/RopeHealthMonitor.js +659 -0
- package/dist/monitoring/RopeHealthMonitor.js.map +1 -0
- package/dist/monitoring/guardManifest.d.ts.map +1 -1
- package/dist/monitoring/guardManifest.js +45 -0
- package/dist/monitoring/guardManifest.js.map +1 -1
- package/dist/scaffold/templates.d.ts.map +1 -1
- package/dist/scaffold/templates.js +9 -0
- package/dist/scaffold/templates.js.map +1 -1
- package/dist/server/AgentServer.d.ts +2 -0
- package/dist/server/AgentServer.d.ts.map +1 -1
- package/dist/server/AgentServer.js +1 -0
- package/dist/server/AgentServer.js.map +1 -1
- package/dist/server/CapabilityIndex.js +1 -1
- package/dist/server/CapabilityIndex.js.map +1 -1
- package/dist/server/routes.d.ts +3 -0
- package/dist/server/routes.d.ts.map +1 -1
- package/dist/server/routes.js +19 -0
- package/dist/server/routes.js.map +1 -1
- package/package.json +1 -1
- package/scripts/lint-scrape-fixture-realness.js +20 -0
- package/src/data/builtin-manifest.json +65 -65
- package/src/data/state-coherence-registry.json +17 -0
- package/src/scaffold/templates/jobs/instar/rope-health-digest.md +42 -0
- package/src/scaffold/templates.ts +9 -0
- package/upgrades/1.3.710.md +22 -0
- package/upgrades/1.3.711.md +79 -0
- package/upgrades/side-effects/tpo-breaker-flake-fix.md +16 -0
- package/upgrades/side-effects/u4-rope-probe-alerts.md +140 -0
- package/upgrades/tpo-breaker-flake-fix.eli16.md +11 -0
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Rope-Health Digest
|
|
3
|
+
description: "Daily one-line mesh rope-health digest (docs/specs/u4-5-rope-health-alerts.md §2). Reads the RopeHealthMonitor's GET /mesh/rope-health surface and, ONLY when something is non-ok, emits at most ONE consolidated section (≤3 sentences, machine-named, content-scrubbed: rope kind + machine nickname + relative times only). Ships enabled:true with a 503-silent body — a DELIBERATE, argued divergence from the feedback-factory-process enabled:false precedent (R-r2-7/R-r3-2): the digest's whole gating already lives in the monitoring.ropeHealth dev-agent gate (dark fleet → the route 503s → this job exits silently at zero cost; live dev agent → the digest flows from day one). Delivery honors monitoring.ropeHealth.digestTopicId (R-r2-8): UNSET → log only, never a send. Tier-1 supervised (this haiku job wraps the deterministic endpoint; the server composes the digest text). Operator note: an agent-home G1 coherence script that audits mesh health should CONSUME GET /mesh/rope-health rather than re-deriving rope state — this monitor is the mechanism; any daily script is a reader."
|
|
4
|
+
schedule: "0 9 * * *"
|
|
5
|
+
priority: low
|
|
6
|
+
expectedDurationMinutes: 2
|
|
7
|
+
model: haiku
|
|
8
|
+
supervision: tier1
|
|
9
|
+
enabled: true
|
|
10
|
+
tags:
|
|
11
|
+
- cat:monitoring
|
|
12
|
+
- role:worker
|
|
13
|
+
- exec:prompt
|
|
14
|
+
gate: curl -sf http://localhost:${INSTAR_PORT:-4042}/health >/dev/null 2>&1
|
|
15
|
+
toolAllowlist: "*"
|
|
16
|
+
unrestrictedTools: true
|
|
17
|
+
mcpAccess: none
|
|
18
|
+
---
|
|
19
|
+
Emit the daily mesh rope-health digest. This is a mechanical, near-silent reporter — when everything is healthy you say NOTHING anywhere.
|
|
20
|
+
|
|
21
|
+
AUTH="${INSTAR_AUTH_TOKEN:-$(python3 -c "import json; v=json.load(open('.instar/config.json')).get('authToken',''); print(v if isinstance(v, str) else '')" 2>/dev/null)}"
|
|
22
|
+
AGENT_ID="${INSTAR_AGENT_ID:-$(python3 -c "import json; print(json.load(open('.instar/config.json')).get('projectName',''))" 2>/dev/null)}"
|
|
23
|
+
PORT="${INSTAR_PORT:-4042}"
|
|
24
|
+
|
|
25
|
+
1. Read the monitor (the ?digest=1 read records the emission metric):
|
|
26
|
+
`curl -s -H "Authorization: Bearer $AUTH" -H "X-Instar-AgentId: $AGENT_ID" "http://localhost:$PORT/mesh/rope-health?digest=1"`
|
|
27
|
+
- A **503** means the rope-health monitor is dark for this agent (`monitoring.ropeHealth` dev-gated off) — exit silently, there is nothing to do. This is the designed fleet posture, not an error.
|
|
28
|
+
- On 200 the body is `{ lastEvaluatedAt, peers: [...], keyExpiry, digest, counters }`. The `digest` field is the server-composed, content-scrubbed, ≤3-sentence consolidated section — you never compose rope text yourself.
|
|
29
|
+
|
|
30
|
+
2. If `digest` is `null`: everything is ok. Exit silently — no message, no log noise.
|
|
31
|
+
|
|
32
|
+
3. If `digest` is non-null, resolve the delivery topic:
|
|
33
|
+
`python3 -c "import json; c=json.load(open('.instar/config.json')); print(c.get('monitoring',{}).get('ropeHealth',{}).get('digestTopicId',''))"`
|
|
34
|
+
- **Empty/unset** (the default): LOG the digest line to stdout (it lands in the job transcript) and exit. Do NOT send it anywhere — the operator has not named a hub topic yet.
|
|
35
|
+
- **Set to a topic id N**: send EXACTLY the digest text (one consolidated section, nothing appended) to that topic:
|
|
36
|
+
`cat <<'EOF' | .instar/scripts/telegram-reply.sh N`
|
|
37
|
+
`<the digest text>`
|
|
38
|
+
`EOF`
|
|
39
|
+
|
|
40
|
+
4. **Tier-1 supervision (your job).** Sanity-check before sending: the digest must be ≤3 sentences and must NOT contain raw IPs, URLs, tunnel hostnames, tailnet names, or emails (the server scrubs by construction — if you ever see one, do NOT send; note it once in the transcript and exit; the next day's run re-checks). Never retry-flood a failed send; the failure is recorded server-side.
|
|
41
|
+
|
|
42
|
+
5. Exit silently. Do not summarize, do not message the user beyond the single digest delivery above.
|
|
@@ -511,6 +511,15 @@ Every guard (monitoring sentinels, reapers, the scheduler, …) is graded by wha
|
|
|
511
511
|
|
|
512
512
|
- **Multi-transport mesh comms** (\`multiMachine.meshTransport\`): when I run on more than one machine, my machines talk to each other over MULTIPLE ropes — Tailscale, the local network (LAN), and the Cloudflare tunnel — and automatically use whichever is healthy, so a single flaky tunnel no longer makes a machine look unreachable (the root cause of the lease flap). Each machine auto-advertises its reachable addresses; the lease layer hedges across them and verifies the answering machine really is the peer (a replay-proof signed handshake). \`GET /health → multiMachine.syncStatus.meshEndpoints\` lists the rope KINDS this machine advertises. Ships ENABLED (Layers 0-2 are strictly additive; a single-machine agent is a no-op and keeps its localhost bind). When multi-machine, the server also listens on the Tailscale/LAN interfaces so peers can reach it — strictly less exposure than the always-on public tunnel, all routes keep their existing auth. **Proactive trigger:** operator asks "why is my machine unreachable / why does the lease keep flapping?" → the single Cloudflare rope was flapping; multi-transport fixes it (recommend installing Tailscale on both machines for the strongest rope). Kill-switch: \`meshTransport.enabled:false\` (back to single-rope, one restart to apply). A preferred stationary captain can also HOLD the lease alone when its peer is provably gone — that piece (\`leaseSelfHeal.soloCaptainHold\`) ships dark/opt-in. Spec: \`docs/specs/multi-transport-mesh-comms.md\`.
|
|
513
513
|
|
|
514
|
+
|
|
515
|
+
### Mesh Rope Health (recovery probe + partition alerts)
|
|
516
|
+
|
|
517
|
+
Two layers keep my machine-to-machine \"ropes\" (Tailscale / LAN / Cloudflare) honest. **Recovery probe (U4.3):** a rope marked dead no longer stays presumed-dead for a week — an in-server prober rides the ~5s lease tick and re-dials dead ropes with a pinned, signed canary (typed-refusal contract; any-2xx never counts), feeding the ONE health authority so a healed rope closes in minutes. Episode-scoped with a 15-min P19 floor and ONE deduped escalation per episode. **Rope-health alerts (U4.5):** a monitor classifies each peer every 30s — \`ok\` (silence), \`degraded\` (a rope down, another carrying traffic — digest only), \`peer-offline\` (all ropes down AND its heartbeat stopped — a lid-close is NEVER an alarm), \`urgent\` (all ropes down while its git-synced heartbeat still ADVANCES = alive but partitioned → ONE HIGH attention item per episode; honest latency: a genuine partition is confirmed in ~30-90 min, bounded by the heartbeat+sync cadence). A Tailscale key expiring within 14 days warns in the digest.
|
|
518
|
+
- Rope state per (peer, kind): \`curl -H \"Authorization: Bearer $AUTH\" http://localhost:${port}/health\` → \`multiMachine.syncStatus.ropeHealth\` (authed only).
|
|
519
|
+
- The classification + digest: \`curl -H \"Authorization: Bearer $AUTH\" http://localhost:${port}/mesh/rope-health\` (503 = the monitor is dark on this agent). The daily \`rope-health-digest\` job logs the digest; set \`monitoring.ropeHealth.digestTopicId\` to have it delivered.
|
|
520
|
+
- **When to use** (PROACTIVE): \"why did a dead rope come back by itself?\" → the recovery probe (read \`ropeHealth\`); \"is the mesh healthy? / why did I get a partition alert?\" → \`GET /mesh/rope-health\`. Alert text carries rope KIND + machine NICKNAME only — never IPs/tailnet names/emails.
|
|
521
|
+
- Both ship dev-gated (\`multiMachine.meshTransport.recoveryProbeEnabled\`, \`monitoring.ropeHealth.enabled\` — omitted ⇒ live on a development agent, dark on the fleet; probe dry-run first via \`recoveryProbeDryRun\`). Specs: \`docs/specs/u4-3-breaker-recovery-probe.md\`, \`docs/specs/u4-5-rope-health-alerts.md\`.
|
|
522
|
+
|
|
514
523
|
**Links that survive machine boundaries (WS4.4 — pool-stable private-view links)** — A private-view link (\`/view/:id\`) keeps working no matter WHICH of my machines is fronting the tunnel, even when the content lives on a DIFFERENT machine. The fronting machine resolves the actual HOLDER of the view (view-id ownership ≠ topic ownership — by probing peers, since each view lives on the disk of the machine that made it) and proxies to it. Ships DARK behind \`multiMachine.seamlessness.ws44PoolLinks\` (dev-agent gated); a single-machine agent is a no-op (no peers to proxy to). Security model (what to tell the user if asked "is a shared link safe across my machines?"):
|
|
515
524
|
- The END-USER credential is enforced end-to-end and the HOLDER makes the authorization decision — the fronting machine is a dumb relay. It NEVER substitutes a machine/mesh credential for the user's, NEVER logs the token, and NEVER caches private content at the edge (responses are \`Cache-Control: no-store\`).
|
|
516
525
|
- The user's PIN/token is validated at the fronting edge, then the proxied request carries a SHORT-LIVED, audience-bound (target holder + the exact view id + HTTP method), SINGLE-USE, mesh-signed ASSERTION of that authentication — NOT the raw PIN. Each machine's PIN secret never crosses the boundary. A captured assertion cannot be replayed against another resource, another holder, or reused within its window.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: patch -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
The five circuit-breaker tests in `tests/unit/TopicProfileOrchestrator.test.ts` synchronized on `parkedFor(...) !== null` — an intermediate state that `parkAndRevert` makes visible before its internal `flushDurably` await yields control. Under CI load, the 10ms poll could observe the park and assert on trip effects (un-park, `breaker-revert` audit, disclosure) that had not executed yet, failing the shard intermittently (twice on PR #1320's CI tonight; reproduced locally on iteration 1 of a 15-run loop). The tests now synchronize on the `breaker-revert` audit — emitted after every other synchronous trip side-effect — via a shared `waitForBreakerTrip(h)` helper. Test-only; no runtime source touched.
|
|
9
|
+
|
|
10
|
+
## What to Tell Your User
|
|
11
|
+
|
|
12
|
+
None — internal change (no user-facing surface).
|
|
13
|
+
|
|
14
|
+
## Summary of New Capabilities
|
|
15
|
+
|
|
16
|
+
None — internal change (no user-facing surface).
|
|
17
|
+
|
|
18
|
+
## Evidence
|
|
19
|
+
|
|
20
|
+
- Pre-fix: `tests/unit/TopicProfileOrchestrator.test.ts` failed iteration 1/15 of a local repetition loop with the exact CI assertion (`expected [] to include '7'` at the `unparks` check), matching both CI shard 2/4 failures on PR #1320 (runs 28573712457 and its predecessor).
|
|
21
|
+
- Post-fix: 25 consecutive local runs green under concurrent benchmark load on the same machine.
|
|
22
|
+
- Full unit suite green at push (husky pre-push).
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: minor -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
Two mesh-transport reliability features land together (shared health-snapshot seam;
|
|
9
|
+
specs `docs/specs/u4-3-breaker-recovery-probe.md` + `docs/specs/u4-5-rope-health-alerts.md`).
|
|
10
|
+
|
|
11
|
+
**U4.3 — traffic-independent rope recovery probe.** The hedged mesh dialer starves dead
|
|
12
|
+
ropes: the healthiest rope wins inside the hedge window and the losers are cancelled, so a
|
|
13
|
+
rope marked dead was never re-dialed — and (the second arm) a cancelled recovering dial's
|
|
14
|
+
AbortError was recorded as a real failure, letting the winner perpetually reset the loser's
|
|
15
|
+
recovery streak. That is how a healed Tailscale rope stayed presumed-dead for a week. The
|
|
16
|
+
fix: (1) hedge-abort neutrality in `HttpLeaseTransport` (an abort-after-winner records
|
|
17
|
+
nothing; a real dial failure still records), and (2) a `RopeRecoveryProber` riding the
|
|
18
|
+
existing ~5s lease-pull tick that re-dials dead ropes with a pinned, signed, bogus-uid
|
|
19
|
+
canary (typed-refusal contract — an untyped 2xx never closes a rope) and feeds
|
|
20
|
+
`PeerEndpointResolver.recordResult`, the ONE health authority. Episode-scoped probing (no
|
|
21
|
+
permanent limbo after a partial recovery, no hot-loop on a slow-but-alive rope), P19
|
|
22
|
+
Eternal-Sentinel floor (15 min cap, escalate-once per episode), dry-run-first. Per-(peer,
|
|
23
|
+
kind) state serves on the authed `GET /health` → `multiMachine.syncStatus.ropeHealth`.
|
|
24
|
+
|
|
25
|
+
**U4.5 — rope-health alerts.** A productized in-server `RopeHealthMonitor` (own bounded 30s
|
|
26
|
+
loop) classifies each peer from the U4.3 snapshot: `ok` / `degraded` (digest-only) /
|
|
27
|
+
`peer-offline` (heartbeat stopped — a lid-close is NEVER urgent) / `urgent` (all ropes down
|
|
28
|
+
while the peer's git-synced heartbeat still ADVANCES — advancement-since-onset semantics,
|
|
29
|
+
honest 30-90 min confirmation latency, ONE HIGH attention item per episode, split-brain item
|
|
30
|
+
wins). Tailscale key expiry warns at 14 days via a bounded hourly `tailscale status --json`
|
|
31
|
+
exec (fixture-registered parser; identifying fields never leave it). New `GET
|
|
32
|
+
/mesh/rope-health` + a daily `rope-health-digest` built-in job (enabled with a 503-silent
|
|
33
|
+
body; delivery only where `monitoring.ropeHealth.digestTopicId` is set). The self-wake
|
|
34
|
+
urgent suppression is HARD-BOUNDED (5 min) because SleepWakeDetector emits false wake
|
|
35
|
+
events under event-loop stalls (audit finding P1-A7) — a spurious sleep signal can delay a
|
|
36
|
+
partition alert, never veto it.
|
|
37
|
+
|
|
38
|
+
Both ship dev-gated (`multiMachine.meshTransport.recoveryProbeEnabled`,
|
|
39
|
+
`monitoring.ropeHealth.enabled` omitted → live on development agents, dark on the fleet);
|
|
40
|
+
G3 loadBearing guard-manifest entries with 30-day soak windows; single-machine installs are
|
|
41
|
+
strict no-ops.
|
|
42
|
+
|
|
43
|
+
## What to Tell Your User
|
|
44
|
+
|
|
45
|
+
When you run me on more than one machine, my machines talk over several network "ropes"
|
|
46
|
+
(Tailscale, local network, Cloudflare). Two quiet failure modes are now fixed: a rope that
|
|
47
|
+
came back to life used to stay marked dead for days — now I re-check dead ropes myself and
|
|
48
|
+
put them back in service within minutes. And when something genuinely goes wrong I tell
|
|
49
|
+
you honestly: a broken-but-recoverable rope shows up in a daily one-line digest, a machine
|
|
50
|
+
that is simply asleep never triggers an alarm, and only a machine that is provably alive
|
|
51
|
+
yet unreachable (a real partition) raises one urgent notice — once, not a flood. These are
|
|
52
|
+
still maturing: they run on development agents first and stay off elsewhere.
|
|
53
|
+
|
|
54
|
+
## Summary of New Capabilities
|
|
55
|
+
|
|
56
|
+
- `GET /mesh/rope-health` — per-peer rope classification + episode state + daily digest
|
|
57
|
+
text (503 while dark).
|
|
58
|
+
- `GET /health` → `multiMachine.syncStatus.ropeHealth` (authed only) — per-(peer, kind)
|
|
59
|
+
rope health + probe scheduling state.
|
|
60
|
+
- `rope-health-digest` built-in daily job (log-only until
|
|
61
|
+
`monitoring.ropeHealth.digestTopicId` is set).
|
|
62
|
+
- Config: `multiMachine.meshTransport.recoveryProbe*` knobs; `monitoring.ropeHealth.*`.
|
|
63
|
+
- CLAUDE.md template section "Mesh Rope Health (recovery probe + partition alerts)"
|
|
64
|
+
(new agents via init, existing agents via migration, Codex/Gemini via shadow markers).
|
|
65
|
+
|
|
66
|
+
## Evidence
|
|
67
|
+
|
|
68
|
+
- Unit: 26 U4.3 tests (episode scoping incl. the limbo case, hedge-abort neutrality both
|
|
69
|
+
sides, typed-contract classifier over captured byte-for-byte /mesh/rpc fixtures, P19
|
|
70
|
+
simulated-day bound in both all-fail and all-succeed arms, dry-run scheduling honesty) +
|
|
71
|
+
29 U4.5 tests (classifier both sides of every boundary incl. the lid-close false-alarm
|
|
72
|
+
arm and the between-heartbeats late upgrade, bounded wake grace with the P1-A7
|
|
73
|
+
spurious-sleep test, episode-key determinism + adjacent grouping, transition-only writes,
|
|
74
|
+
state round-trip, content scrub, captured tailscale fixtures).
|
|
75
|
+
- Integration: authed `/health` carries `ropeHealth` while an unauthed caller sees no mesh
|
|
76
|
+
topology; `GET /mesh/rope-health` 503-dark/200-live with episode-deduped counters.
|
|
77
|
+
- E2E: production-path gate resolution → prober rides the REAL coordinator lease-pull tick
|
|
78
|
+
(lastProbeAt advances, recordResult reaches the same resolver instance); monitor's own
|
|
79
|
+
30s loop ticks and tears down; dark → zero probes, no timer, route 503.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Side-Effects Review — tpo-breaker-flake-fix
|
|
2
|
+
|
|
3
|
+
**Change:** test-only. `tests/unit/TopicProfileOrchestrator.test.ts` — the five §10.4 circuit-breaker tests waited on `parkedFor('7') !== null`, an intermediate state `parkAndRevert` makes visible before its internal `flushDurably` await yields; under CI load the 10ms poll observed the park and asserted on trip effects (unpark / audit / disclosure) that had not run yet. They now wait on the `breaker-revert` audit — the trip's terminal signal, emitted after every other synchronous trip side-effect — via a shared `waitForBreakerTrip(h)` helper. No `src/` file touched.
|
|
4
|
+
|
|
5
|
+
1. **Over-block** — none. No runtime decision surface; the tests assert the same behavior with a later, race-free synchronization point.
|
|
6
|
+
2. **Under-block** — none. Coverage is unchanged: every assertion that existed still runs (including `parkedFor` non-null, asserted explicitly after the wait). The wait is stricter, not looser — it requires the full trip, not just the park.
|
|
7
|
+
3. **Level-of-abstraction fit** — considered fixing the production ordering instead (assign `entry.parked` after `flushDurably`). Rejected: the mid-trip visibility is harmless in production (no production consumer polls `parkedFor` during a trip; the store lock serializes real readers), and reordering durable-write vs in-memory state has its own failure-mode tradeoffs (a crash between write and assign). The test was the thing making a timing assumption; the test is the right layer to fix.
|
|
8
|
+
4. **Signal vs authority compliance** — n/a; no decision point, no gate, no blocking authority. Test-only.
|
|
9
|
+
5. **Interactions** — the helper waits on `h.audits`, which each test's fresh harness resets, so no cross-test bleed. The two tests that subsequently call `requestRecoveryWrite` also benefit (they previously raced `durable.breakerTrips` being set mid-trip).
|
|
10
|
+
6. **External surfaces** — none. Nothing visible to agents, users, or other systems changes.
|
|
11
|
+
7. **Multi-machine posture** — n/a (test-only; no state, no replication surface).
|
|
12
|
+
8. **Rollback cost** — `git revert` of one commit restores the old waits; zero data or fleet impact.
|
|
13
|
+
|
|
14
|
+
**Evidence:** old code failed iteration 1 of a 15-run local loop (same assertion as CI shard 2/4 failures on PR #1320 tonight, twice); fixed code passed 25 consecutive runs under benchmark load on the same machine.
|
|
15
|
+
|
|
16
|
+
**Second-pass review:** not required — no sentinel/gate/lifecycle runtime code touched (test file only).
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# Side-Effects Review — U4.3 rope recovery probe + U4.5 rope-health alerts
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `u4-rope-probe-alerts`
|
|
4
|
+
**Date:** `2026-07-02`
|
|
5
|
+
**Author:** Echo (autonomous)
|
|
6
|
+
**Second-pass reviewer:** not-required (Tier 2; specs converged + approved — codex-cli:gpt-5.5 cross-model review, 4 + 5 iterations; both features dev-gated dark on the fleet)
|
|
7
|
+
|
|
8
|
+
## Summary of the change
|
|
9
|
+
|
|
10
|
+
One shared PR for the two converged U4 specs (`docs/specs/u4-3-breaker-recovery-probe.md`,
|
|
11
|
+
`docs/specs/u4-5-rope-health-alerts.md`) — they share the `PeerEndpointResolver.snapshot()`
|
|
12
|
+
seam (U4.3 builds it, U4.5 consumes it).
|
|
13
|
+
|
|
14
|
+
**U4.3** fixes hedge-winner-abort starvation of dead mesh ropes:
|
|
15
|
+
|
|
16
|
+
- `src/core/HttpLeaseTransport.ts` — hedge-abort neutrality (R-r2-1): an
|
|
17
|
+
AbortError caused by the hedge winner's `finish()` records NOTHING (never
|
|
18
|
+
`recordResult(false)`); a real dial failure still records. New exported
|
|
19
|
+
`isAbortShapedError`.
|
|
20
|
+
- `src/core/PeerEndpointResolver.ts` — the `snapshot()` read seam (rows are
|
|
21
|
+
copies; kind+counters only, no URLs).
|
|
22
|
+
- `src/core/RopeRecoveryProber.ts` (new) — episode-scoped prober riding the
|
|
23
|
+
lease-pull tick; feeds `recordResult` on the ONE health authority; P19
|
|
24
|
+
Eternal-Sentinel floor + escalate-once; dry-run keeps a shadow streak and
|
|
25
|
+
never mutates health.
|
|
26
|
+
- `src/core/ropeProbeContract.ts` (new) — the typed G4 canary payload contract +
|
|
27
|
+
the REGISTERED `parseProbeResponse` classifier (captured byte-for-byte
|
|
28
|
+
fixtures).
|
|
29
|
+
- `src/core/MultiMachineCoordinator.ts` — `attachLeasePullTickListener` (the
|
|
30
|
+
carrier; error-isolated) + `attachRopeHealthProvider` → `getSyncStatus()`
|
|
31
|
+
serves `ropeHealth` on the authed /health branch only.
|
|
32
|
+
- `src/commands/server.ts` — dev-gated wiring (probe client, registry-validated
|
|
33
|
+
target list, feature metrics `rope-recovery-probe`, attention sink).
|
|
34
|
+
|
|
35
|
+
**U4.5** productizes rope-health alerting:
|
|
36
|
+
|
|
37
|
+
- `src/monitoring/RopeHealthMonitor.ts` (new) — own bounded 30s loop;
|
|
38
|
+
deterministic classifier (ok / degraded / peer-offline / urgent) with the
|
|
39
|
+
R-r3-1 advancement-since-onset heartbeat discriminator; episode-deduped ONE
|
|
40
|
+
HIGH item; split-brain-item suppression; sustained-clear; transition-only
|
|
41
|
+
durable state (`state/rope-health.json`, registered + retention-declared);
|
|
42
|
+
BOUNDED self-wake grace (P1-A7 hazard documented in-code); detected-not-
|
|
43
|
+
notified retry; content scrub by construction.
|
|
44
|
+
- `src/core/tailscaleStatusParser.ts` (new) — REGISTERED parser for the hourly
|
|
45
|
+
bounded `tailscale status --json` exec; only role + KeyExpiry leave it.
|
|
46
|
+
- `src/server/routes.ts` — `GET /mesh/rope-health` (503 when dark; `?digest=1`
|
|
47
|
+
records the digest-emission metric).
|
|
48
|
+
- `src/scaffold/templates/jobs/instar/rope-health-digest.md` (new) — daily
|
|
49
|
+
tier-1 job, enabled with a 503-silent body (argued divergence from the
|
|
50
|
+
feedback-factory precedent, R-r2-7/R-r3-2); log-only until
|
|
51
|
+
`monitoring.ropeHealth.digestTopicId` is set.
|
|
52
|
+
- Config/registry parity: `types.ts` + `ConfigDefaults.ts` (both features'
|
|
53
|
+
knobs; `enabled` OMITTED — dev gate), `devGatedFeatures.ts` (ropeRecoveryProbe
|
|
54
|
+
+ ropeHealthAlerts with written justifications), `guardManifest.ts` (both
|
|
55
|
+
loadBearing G3 entries, 30-day soak), `templates.ts` + `PostUpdateMigrator.ts`
|
|
56
|
+
(the "Mesh Rope Health" CLAUDE.md section, migrator + shadow markers),
|
|
57
|
+
`CapabilityIndex.ts` (mesh prefix note), state-coherence registry,
|
|
58
|
+
`upgrades/next/u4-rope-probe-health-alerts.md`, site docs page.
|
|
59
|
+
|
|
60
|
+
## Decision-point inventory
|
|
61
|
+
|
|
62
|
+
- **Modified — `HttpLeaseTransport` attempt catch**: the abort-after-winner
|
|
63
|
+
discriminator (`signal.aborted && isAbortShapedError`). Fail direction: an
|
|
64
|
+
ambiguous rejection (aborted signal + abort-shaped error while a winner
|
|
65
|
+
confirmed) records NEUTRALLY — health truth is fed only by REAL dial
|
|
66
|
+
evidence. A real failure (timeout fires the merged signal only) still records.
|
|
67
|
+
- **Added — probe eligibility (RopeRecoveryProber episodes)**: opens on dead,
|
|
68
|
+
survives fail-after-partial-recovery (the limbo fix), closes only on
|
|
69
|
+
lastKnownGood reclaim; cadence owned by the probe layer in BOTH modes (never
|
|
70
|
+
the resolver's trivially-true 5s). Bounded by the P19 floor in dry-run too.
|
|
71
|
+
- **Added — probe success classifier (`parseProbeResponse`)**: success is the
|
|
72
|
+
EXACT typed contract (403 not-router / 200 sender-rejected ack); an untyped
|
|
73
|
+
2xx, malformed body, auth rejection, or unexpected ACCEPTANCE records as
|
|
74
|
+
failure — any-2xx can never close a rope.
|
|
75
|
+
- **Added — urgent classifier (RopeHealthMonitor)**: urgent requires all-down +
|
|
76
|
+
post-onset heartbeat + time-pinned debounce + ≥2 observations; every ambiguous
|
|
77
|
+
arm (absent rows, unreadable heartbeat, registry-offline, pre-onset beat)
|
|
78
|
+
fails toward NOT-urgent (peer-offline/unknown). The wake-grace suppressor is
|
|
79
|
+
HARD-BOUNDED (wakeGraceMaxMs, default 5 min) so the known-false SleepWakeDetector
|
|
80
|
+
wake events (P1-A7) can delay but never veto a partition alert.
|
|
81
|
+
- **Added — split-brain suppression**: an active split-brain state suppresses the
|
|
82
|
+
monitor's item (one episode, one ask). Fail direction of an unreadable check:
|
|
83
|
+
raise (a possible duplicate ask, never a silent partition).
|
|
84
|
+
|
|
85
|
+
## Roll-up across the seven review dimensions
|
|
86
|
+
|
|
87
|
+
1. **Over-block**: none. Nothing gates user messages. The tone/coherence gates
|
|
88
|
+
are untouched. The only "block"-shaped behavior is the probe refusing to
|
|
89
|
+
count an untyped 2xx as recovery — the conservative direction for a healer.
|
|
90
|
+
2. **Under-block**: the hedge-abort neutrality REMOVES failure records for
|
|
91
|
+
winner-cancelled dials. Bounded: the neutrality branch requires the hedge
|
|
92
|
+
controller's signal aborted AND an abort-shaped error; a rope that is
|
|
93
|
+
genuinely failing still accumulates failures from its own real dial errors,
|
|
94
|
+
and the prober now guarantees dead ropes are re-verified (strictly more
|
|
95
|
+
evidence flows than before).
|
|
96
|
+
3. **Level-of-abstraction fit**: health truth stays in the ONE authority
|
|
97
|
+
(PeerEndpointResolver.HealthRecord); the prober holds SCHEDULING state only;
|
|
98
|
+
the monitor holds CLASSIFICATION state only; the parser owns the scrub
|
|
99
|
+
boundary. No second state machine (spec Decision 1).
|
|
100
|
+
4. **Signal-vs-authority compliance**: both features are SIGNALS. The probe
|
|
101
|
+
feeds evidence into the existing health machinery; the monitor only raises
|
|
102
|
+
attention items (episode-deduped, split-brain-suppressed) and serves reads.
|
|
103
|
+
Every error path fails toward silence/no-emit; no silent try/catch (each
|
|
104
|
+
carries `@silent-fallback-ok` with the fail direction named).
|
|
105
|
+
5. **Interactions**: the probe rides the lease-pull tick via an error-isolated
|
|
106
|
+
listener (a throwing scan never breaks the pull — e2e-tested); the monitor
|
|
107
|
+
subscribes to SleepWakeDetector 'wake' (read-only); both read the machine
|
|
108
|
+
registry/heartbeat read-only. The digest job reads one route. No shared
|
|
109
|
+
mutable state beyond the resolver's own recordResult funnel.
|
|
110
|
+
6. **External surfaces**: the probe dials peers' OWN registry-validated mesh
|
|
111
|
+
URLs (resolver.resolve output — URL-shape + LAN-gate validated) with a
|
|
112
|
+
signed, deliberately-unresolvable canary the peer answers with a typed
|
|
113
|
+
refusal; it never rides the router-forward funnel, so no user-facing
|
|
114
|
+
sender-rejection notice can fire. The tailscale exec is local, bounded (5s
|
|
115
|
+
timeout), absent-CLI-tolerant. Alerts are content-scrubbed (kind + nickname
|
|
116
|
+
only). Both features dark on the fleet (dev gate); probe additionally
|
|
117
|
+
dry-run (sends probes, never mutates health).
|
|
118
|
+
7. **Rollback cost**: low. `recoveryProbeEnabled:false` → zero probes, no orphan
|
|
119
|
+
state (the HealthRecord is the same store traffic feeds);
|
|
120
|
+
`monitoring.ropeHealth.enabled:false` → monitor inert, route 503s, the state
|
|
121
|
+
file is inert data. The transport neutrality fix is a one-hunk revert.
|
|
122
|
+
|
|
123
|
+
## Evidence pointers
|
|
124
|
+
|
|
125
|
+
- `npx tsc --noEmit` — clean.
|
|
126
|
+
- Unit: `RopeRecoveryProber` 14/14, `hedge-abort-neutrality` 3/3,
|
|
127
|
+
`ropeProbeContract` 5/5 (captured fixtures), `peer-endpoint-resolver-snapshot`
|
|
128
|
+
4/4, `RopeHealthMonitor` 22/22, `tailscaleStatusParser` 7/7 (captured
|
|
129
|
+
fixtures).
|
|
130
|
+
- Integration: `mesh-rope-health-route` 7/7 (authed /health ropeHealth +
|
|
131
|
+
unauthed exclusion; route dark/live; digest metric; episode-dedup counters).
|
|
132
|
+
- E2E: `rope-recovery-probe-alive` 3/3 (real coordinator lease-pull carrier;
|
|
133
|
+
recordResult reaches the SAME resolver; dark = zero probes),
|
|
134
|
+
`rope-health-alerts-lifecycle` 5/5 (production gate path; own 30s loop ticks
|
|
135
|
+
+ tears down; dark = 503 + no timer; real heartbeat file drives the
|
|
136
|
+
discriminator).
|
|
137
|
+
- Ratchets green: full `npm run lint` battery (incl. dark-gate golden map
|
|
138
|
+
hand-updated, guard-manifest, scrape-fixture realness, state registry +
|
|
139
|
+
retention), `feature-delivery-completeness` 111/111, docs-coverage ≥
|
|
140
|
+
thresholds.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# A flaky safety-net test now waits for the finish line, not the halfway mark
|
|
2
|
+
|
|
3
|
+
Instar has a "circuit breaker" for conversation settings: if an operator pins a topic to a model that keeps failing to launch, the breaker parks the bad pin, reverts to the last working settings, and restarts the session — so a typo'd model name can never brick a conversation.
|
|
4
|
+
|
|
5
|
+
The breaker itself works fine. Its TESTS, though, were flaky — they'd pass most of the time and occasionally fail on the build servers (twice tonight alone, blocking an unrelated PR from merging). Flaky tests are poison: every red build that ISN'T real teaches people to ignore red builds.
|
|
6
|
+
|
|
7
|
+
The root cause is a classic race. When the breaker trips, it does several things in a row: park the bad pin → revert → un-park the resume entry → write an audit record → tell the user → restart the session. The tests waited for the FIRST visible step ("is the pin parked yet?") and then immediately checked ALL the later steps. There's a tiny window where the park is visible but the rest hasn't happened yet — normally sub-millisecond, but on a loaded build machine the test's polling could land exactly inside it and fail on assertions about steps that were still in flight.
|
|
8
|
+
|
|
9
|
+
The fix: the tests now wait for the audit record — which the breaker writes only AFTER everything else in the trip is done — before checking any of the trip's effects. Same coverage, zero timing dependence. Verified by running the test file 25 times in a row under load (the old version failed on the very first try of a 15-run loop).
|
|
10
|
+
|
|
11
|
+
No production code changed at all — this is a test-only fix. The breaker behaves exactly as before; its report card just stopped lying about it.
|