instar 1.3.647 → 1.3.649
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dashboard/index.html +14 -14
- package/dist/commands/reflect.d.ts.map +1 -1
- package/dist/commands/reflect.js +7 -3
- package/dist/commands/reflect.js.map +1 -1
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +35 -0
- package/dist/commands/server.js.map +1 -1
- package/dist/commands/setup.d.ts.map +1 -1
- package/dist/commands/setup.js +17 -0
- package/dist/commands/setup.js.map +1 -1
- package/dist/config/ConfigDefaults.d.ts.map +1 -1
- package/dist/config/ConfigDefaults.js +14 -0
- package/dist/config/ConfigDefaults.js.map +1 -1
- package/dist/core/AutoUpdater.d.ts.map +1 -1
- package/dist/core/AutoUpdater.js +23 -3
- package/dist/core/AutoUpdater.js.map +1 -1
- package/dist/core/CoherenceGate.d.ts.map +1 -1
- package/dist/core/CoherenceGate.js +24 -0
- package/dist/core/CoherenceGate.js.map +1 -1
- package/dist/core/CoherenceReviewer.d.ts +9 -0
- package/dist/core/CoherenceReviewer.d.ts.map +1 -1
- package/dist/core/CoherenceReviewer.js +18 -2
- package/dist/core/CoherenceReviewer.js.map +1 -1
- package/dist/core/InputGuard.d.ts.map +1 -1
- package/dist/core/InputGuard.js +17 -3
- package/dist/core/InputGuard.js.map +1 -1
- package/dist/core/MessageSentinel.d.ts.map +1 -1
- package/dist/core/MessageSentinel.js +19 -2
- package/dist/core/MessageSentinel.js.map +1 -1
- package/dist/core/MessagingToneGate.d.ts +8 -0
- package/dist/core/MessagingToneGate.d.ts.map +1 -1
- package/dist/core/MessagingToneGate.js +16 -1
- package/dist/core/MessagingToneGate.js.map +1 -1
- package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
- package/dist/core/PostUpdateMigrator.js +10 -0
- package/dist/core/PostUpdateMigrator.js.map +1 -1
- package/dist/core/SingleInstanceLock.d.ts +103 -0
- package/dist/core/SingleInstanceLock.d.ts.map +1 -0
- package/dist/core/SingleInstanceLock.js +300 -0
- package/dist/core/SingleInstanceLock.js.map +1 -0
- package/dist/core/SpawnCapIntelligenceProvider.d.ts +89 -0
- package/dist/core/SpawnCapIntelligenceProvider.d.ts.map +1 -0
- package/dist/core/SpawnCapIntelligenceProvider.js +164 -0
- package/dist/core/SpawnCapIntelligenceProvider.js.map +1 -0
- package/dist/core/UpdateChecker.d.ts +12 -0
- package/dist/core/UpdateChecker.d.ts.map +1 -1
- package/dist/core/UpdateChecker.js +22 -0
- package/dist/core/UpdateChecker.js.map +1 -1
- package/dist/core/hostSpawnSemaphore.d.ts +188 -0
- package/dist/core/hostSpawnSemaphore.d.ts.map +1 -0
- package/dist/core/hostSpawnSemaphore.js +471 -0
- package/dist/core/hostSpawnSemaphore.js.map +1 -0
- package/dist/core/intelligenceProviderFactory.d.ts +0 -12
- package/dist/core/intelligenceProviderFactory.d.ts.map +1 -1
- package/dist/core/intelligenceProviderFactory.js +27 -9
- package/dist/core/intelligenceProviderFactory.js.map +1 -1
- package/dist/core/types.d.ts +18 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/monitoring/DegradationReporter.d.ts +18 -0
- package/dist/monitoring/DegradationReporter.d.ts.map +1 -1
- package/dist/monitoring/DegradationReporter.js +24 -0
- package/dist/monitoring/DegradationReporter.js.map +1 -1
- package/dist/monitoring/WorktreeMonitor.d.ts +13 -6
- package/dist/monitoring/WorktreeMonitor.d.ts.map +1 -1
- package/dist/monitoring/WorktreeMonitor.js +50 -33
- package/dist/monitoring/WorktreeMonitor.js.map +1 -1
- package/dist/scaffold/templates.d.ts.map +1 -1
- package/dist/scaffold/templates.js +5 -0
- package/dist/scaffold/templates.js.map +1 -1
- package/dist/server/CapabilityIndex.d.ts.map +1 -1
- package/dist/server/CapabilityIndex.js +9 -0
- package/dist/server/CapabilityIndex.js.map +1 -1
- package/dist/server/routes.d.ts.map +1 -1
- package/dist/server/routes.js +36 -3
- package/dist/server/routes.js.map +1 -1
- package/package.json +4 -2
- package/scripts/lint-no-direct-destructive.js +4 -0
- package/scripts/lint-no-unbounded-llm-spawn.js +137 -0
- package/src/data/builtin-manifest.json +65 -65
- package/src/scaffold/templates.ts +5 -0
- package/upgrades/1.3.648.md +99 -0
- package/upgrades/1.3.649.md +30 -0
- package/upgrades/eli16/autoupdate-strand-detector.md +22 -0
- package/upgrades/eli16/dashboard-health-signal-polish.md +26 -0
- package/upgrades/eli16/worktree-monitor-async-git.md +24 -0
- package/upgrades/side-effects/autoupdate-strand-detector.md +36 -0
- package/upgrades/side-effects/dashboard-health-signal-polish.md +59 -0
- package/upgrades/side-effects/forkbomb-prevention-simple.md +138 -0
- package/upgrades/side-effects/worktree-monitor-async-git.md +39 -0
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: minor -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
Fork-bomb prevention — the SIMPLE design (`docs/specs/forkbomb-prevention-simple.md`).
|
|
9
|
+
On 2026-06-20 the Instar server fork-bombed its macOS host into OOM **twice**
|
|
10
|
+
(~230-289 concurrent `claude -p` ≈ 90-115GB): `evaluate()` spawned one LLM
|
|
11
|
+
subprocess per call with **zero** concurrency control, `CoherenceGate` fanned ~10
|
|
12
|
+
reviewers in parallel per message, and up to three server instances each
|
|
13
|
+
re-flooded. This release adds the three structural primitives that make that
|
|
14
|
+
physically impossible:
|
|
15
|
+
|
|
16
|
+
- **P1 — Host-wide concurrent-spawn cap (the PRIMARY control).** A host-local
|
|
17
|
+
counting semaphore (`src/core/hostSpawnSemaphore.ts`) bounds how many LLM
|
|
18
|
+
subprocesses run **at once** across every compliant Instar process on the host
|
|
19
|
+
(default **8**). It uses a holder-SET model (count live holder records under an
|
|
20
|
+
exclusive flock, not decrement/increment counter math) so it is crash-safe by
|
|
21
|
+
construction — a double-release is a no-op, a pid-reuse can't steal a slot, a
|
|
22
|
+
partial write is discarded, a crashed holder is reclaimed (pid-dead AND
|
|
23
|
+
heartbeat-stale, **this host only**; a foreign-hostname holder is NEVER
|
|
24
|
+
reclaimed; a `df -P` host-local-disk check gates reclaim, fail-closed). The cap
|
|
25
|
+
is enforced by a per-`evaluate()` wrapper provider
|
|
26
|
+
(`src/core/SpawnCapIntelligenceProvider.ts`) installed at **every** return arm
|
|
27
|
+
of `buildIntelligenceProvider` (the factory funnel) — so CoherenceGate's
|
|
28
|
+
shared-instance fan-out is bound per call. A lint
|
|
29
|
+
(`scripts/lint-no-unbounded-llm-spawn.js`) forward-guards any new provider
|
|
30
|
+
constructed outside the funnel.
|
|
31
|
+
- **P2 — Single-instance lock (`src/core/SingleInstanceLock.ts`).** Refuses a
|
|
32
|
+
duplicate server instance of the same agent on a host (the 3× launchd/fleet/tmux
|
|
33
|
+
multiplier), with a bounded deploy-handoff grace so a normal restart hands off
|
|
34
|
+
cleanly. Foreign-host locks are never reclaimed; `INSTAR_ALLOW_SECOND_INSTANCE=1`
|
|
35
|
+
overrides for a deliberate second instance.
|
|
36
|
+
- **P3 — Bounded ingress.** A saturated cap makes a spawn request poll-retry up to
|
|
37
|
+
`acquireMs` (default 5000ms, ~100ms interval, no in-memory waiter queue so no
|
|
38
|
+
heap growth), bounded by a concurrent-poller ceiling (`waitersMax`, default 64).
|
|
39
|
+
On genuine exhaustion the call throws a typed `LlmCapacityUnavailableError`. The
|
|
40
|
+
**four safety-gating seams** fail CLOSED on that shed instead of auto-passing:
|
|
41
|
+
`MessageSentinel.classify` (held, not `normal`), `InputGuard.reviewTopicCoherence`
|
|
42
|
+
(flagged `suspicious`, not `coherent`), `MessagingToneGate.review` (held
|
|
43
|
+
`pass:false`), and the outbound `CoherenceGate` reviewer path (block-the-turn
|
|
44
|
+
`pass:false`). The deterministic emergency-stop keyword pre-check runs **before**
|
|
45
|
+
the LLM classifier and is **exempt** from the cap — a "stop everything" halt
|
|
46
|
+
always halts.
|
|
47
|
+
|
|
48
|
+
The cap is a **SAFETY FLOOR**: it ships **ON by default** for every agent (read
|
|
49
|
+
with a plain `?? default`, never a dev gate), tunable via `intelligence.spawnCap`
|
|
50
|
+
(`maxConcurrent`/`acquireMs`/`waitersMax`) or env (`INSTAR_HOST_SPAWN_MAX`,
|
|
51
|
+
`INSTAR_SPAWN_ACQUIRE_MS`, `INSTAR_SPAWN_WAITERS_MAX`). A new `GET /spawn-limiter`
|
|
52
|
+
route reports live holder count, cap, available slots, saturation, and waiters. A
|
|
53
|
+
conservative OS-level `NumberOfProcesses` ceiling (512) is added to the launchd
|
|
54
|
+
plist as a host-global belt under the semaphore. A new constitutional standard —
|
|
55
|
+
**"Bounded Blast Radius"** — lands in `docs/STANDARDS-REGISTRY.md` with its lint +
|
|
56
|
+
burst-invariant ratchet in the same PR.
|
|
57
|
+
|
|
58
|
+
## What to Tell Your User
|
|
59
|
+
|
|
60
|
+
- **Fork-bomb protection is on**: "I now have a hard ceiling on how much work I can
|
|
61
|
+
run at the same time, so I can never flood your machine into running out of
|
|
62
|
+
memory again — there's a safe limit (8 by default) that's always on, plus a guard
|
|
63
|
+
that stops a duplicate of me from starting up. You can ask me 'are we protected
|
|
64
|
+
against a fork-bomb?' or 'how many tasks am I running right now?' any time."
|
|
65
|
+
|
|
66
|
+
## Summary of New Capabilities
|
|
67
|
+
|
|
68
|
+
| Capability | How to Use |
|
|
69
|
+
|-----------|-----------|
|
|
70
|
+
| Host-wide concurrent-LLM-spawn cap (SEV-1 OOM fix) | Automatic — on by default (limit 8); tune with `intelligence.spawnCap` or env |
|
|
71
|
+
| Spawn-cap status | `GET /spawn-limiter` |
|
|
72
|
+
| Single-instance lock (no duplicate server) | Automatic at server boot; override with `INSTAR_ALLOW_SECOND_INSTANCE=1` |
|
|
73
|
+
| Fail-closed gating under capacity pressure | Automatic — the four inbound/outbound safety gates hold instead of auto-passing |
|
|
74
|
+
|
|
75
|
+
## Evidence
|
|
76
|
+
|
|
77
|
+
This release fixes a SEV-1 (full host OOM ×2 on 2026-06-20). Evidence:
|
|
78
|
+
|
|
79
|
+
- **Reproduction → fix mechanism.** The incident driver — one `claude -p` per
|
|
80
|
+
`evaluate()` with no concurrency control, fanned ~10-wide by CoherenceGate — is
|
|
81
|
+
now bound at the factory funnel: every provider `buildIntelligenceProvider`
|
|
82
|
+
returns is wrapped per-call by `SpawnCapIntelligenceProvider`, which acquires a
|
|
83
|
+
host-wide slot before the inner spawn and releases it in `finally`.
|
|
84
|
+
- **Burst-invariant ratchet** (`tests/unit/host-spawn-semaphore-burst-invariant.test.ts`):
|
|
85
|
+
10,000 churned acquire/release attempts NEVER let live holders exceed the cap;
|
|
86
|
+
a TRUE multi-process test forks 12 OS processes racing for 3 slots and observes
|
|
87
|
+
≤3 successes (the cross-process flock holds the cap). This is the standing
|
|
88
|
+
Bounded-Accumulation proof that the OOM vector cannot recur.
|
|
89
|
+
- **Four fail-closed seams** (`tests/unit/spawn-cap-fail-closed-gates.test.ts`): a
|
|
90
|
+
capacity shed at each gating seam is asserted to HOLD (not produce
|
|
91
|
+
`normal`/`coherent`/`pass:true`), while a generic LLM error still fails open —
|
|
92
|
+
proving capacity is the only new hold. Emergency-stop is asserted exempt
|
|
93
|
+
(classified by the deterministic fast-path before any LLM call).
|
|
94
|
+
- **E2E lifecycle** (`tests/e2e/spawn-cap-forkbomb-lifecycle.test.ts`):
|
|
95
|
+
`GET /spawn-limiter` is alive (200, cap 8, on by default) on a production-init
|
|
96
|
+
server, and a second single-instance-lock acquisition while a live holder exists
|
|
97
|
+
is refused (the duplicate-flood guard), while a clean release hands off.
|
|
98
|
+
- All three test tiers green; `tsc --noEmit` clean; the new lint + the
|
|
99
|
+
`no-silent-llm-fallback` + `no-silent-fallbacks` ratchets green.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: patch -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
Three small, pre-existing dashboard issues, found while verifying the dashboard-freeze fix (#1253). None affect the dashboard connection.
|
|
9
|
+
|
|
10
|
+
1. The dashboard polled the WhatsApp QR endpoint every 3s and never stopped when WhatsApp wasn't configured — the route answers 503, but the poll kept firing, piling ~20 console 503 errors/min into the browser for the whole session. It now stops the poll on the first 503.
|
|
11
|
+
2. A parse-time script block attached a WebSocket "message" listener to `ws` before `connectWebSocket()` had assigned it, so `ws` was null and `ws.onmessage` threw "Cannot read properties of null (reading 'onmessage')" on every page load (and the paste-panel refresh it enabled never actually worked). Paste handling moved into the main `handleMessage` switch — no more error, and it now survives reconnects.
|
|
12
|
+
3. The `/health` "Degraded" badge turned orange whenever any degradation event had ever been recorded, and those events never self-clear — so one transient/benign fallback pinned the badge red for the whole process lifetime even after recovery. `/health` now counts only degradations from the last 30 minutes (`DegradationReporter.getRecentEvents`); a persistent problem keeps re-reporting so it stays visible, a one-off ages out. An unparseable timestamp is kept (fail-safe). The full degradation log and the mark-reported path are unchanged, and on a Telegram agent the `status` field never gated restarts.
|
|
13
|
+
|
|
14
|
+
## What to Tell Your User
|
|
15
|
+
|
|
16
|
+
If your dashboard's health badge was stuck on "Degraded" even though everything looked fine, that's fixed — it now reflects current health and clears once a transient blip has passed (a real, ongoing problem still shows). You won't notice the other two fixes directly: they quietly remove browser-console noise and an on-load error. (A separate, deeper item — a "system review" panel showing days-old data because it never re-runs — is tracked separately.)
|
|
17
|
+
|
|
18
|
+
## Summary of New Capabilities
|
|
19
|
+
|
|
20
|
+
- `DegradationReporter.getRecentEvents(windowMs = 30min, now)` — a time-windowed view of degradation events. `/health` uses it for `status` + the reported count/summary, so the dashboard "Degraded" badge reflects CURRENT health rather than any-event-ever-recorded. The full append-only log (`getEvents`) and the mark-reported path are untouched.
|
|
21
|
+
- Dashboard: the WhatsApp QR poll stops on a 503 (adapter absent) — removes ~20 console 503s/min of noise.
|
|
22
|
+
- Dashboard: paste WebSocket events (`paste_delivered` / `paste_acknowledged`) are handled in the main message router — removes the on-load null-deref TypeError and makes the paste-panel refresh survive reconnects.
|
|
23
|
+
|
|
24
|
+
## Evidence
|
|
25
|
+
|
|
26
|
+
Observed live on this agent (Echo) while verifying #1253:
|
|
27
|
+
|
|
28
|
+
1. **WhatsApp poll spam** — Before: with no WhatsApp adapter configured, the browser console accumulated repeated `GET /whatsapp/qr 503` errors at the 3s poll cadence (~20/min) for the whole session. After: the poll stops on the first 503 (`stopWaPolling()`), so the console stays clean.
|
|
29
|
+
2. **On-load null deref** — Before: `Uncaught TypeError: Cannot read properties of null (reading 'onmessage')` fired at `dashboard:~9289` on every page load (the block ran at parse time, before `connectWebSocket()` assigned `ws`), and the paste-panel refresh it was meant to wire never ran. After: paste events are handled in `handleMessage`; the parse-time block is gone, so the load-time TypeError no longer occurs.
|
|
30
|
+
3. **Stuck "Degraded" badge** — Before: `curl /health` on a healthy box returned `{"status":"degraded","degradations":2,...}` where both events were a benign `"Stop event allowed without authority ruling … fail-open"` recorded earlier in the boot — never cleared, pinning the badge red. After (unit-proven, `tests/unit/degradation-reporter.test.ts`): `getRecentEvents()` returns only events within the 30-min window, so once those benign events age out `status` returns to `ok`; a re-reported (persistent) degradation stays inside the window and keeps the badge red.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# ELI16 — Notice when an update silently fell off the truck
|
|
2
|
+
|
|
3
|
+
## The problem in plain words
|
|
4
|
+
|
|
5
|
+
Instar updates itself by installing the new version into a private folder (the "shadow install") and then restarting onto it. To avoid loops, it keeps a note: "I already applied version X." On each check, if that note matches the latest version, it shrugs and says "downloaded, just waiting for a restart."
|
|
6
|
+
|
|
7
|
+
But that note is just a *memory* — it's not the same as *checking the folder*. And the version number the program reports for itself is read once when it starts up and then cached, so it never notices if the folder changes underneath it.
|
|
8
|
+
|
|
9
|
+
So here's the failure: an update installs successfully (the folder now has the new version, the note says "applied X"). Then something knocks the folder back to the OLD version — for example, a crash-loop where two copies of the program fight, or a half-finished re-install. Now the note still says "I applied X," but X isn't actually in the folder anymore. The updater believes it's up to date, refuses to re-apply, and every time it checks it cheerfully reports "downloaded, waiting for a restart" — which is a lie. The agent is quietly stuck on the old version forever, and nobody can tell from the message.
|
|
10
|
+
|
|
11
|
+
This really happened: an agent got stranded on the old version while its note insisted it had the new one. It only got unstuck when a human manually re-installed.
|
|
12
|
+
|
|
13
|
+
## The fix
|
|
14
|
+
|
|
15
|
+
Add a way to read the *live* version straight from the folder on disk (uncached), and use it. When the updater is about to print "downloaded, waiting for a restart," it now first checks the folder. If the folder doesn't actually have the version the note claims, it knows it's stranded — so instead of the misleading "waiting for a restart" it prints a loud warning and sends one honest heads-up: "an update was recorded as installed but isn't actually on disk — I'm stuck on the old version and won't auto-update until this is re-applied. Nothing's broken, but I won't pick up new versions until it's fixed."
|
|
16
|
+
|
|
17
|
+
## What it does and doesn't do
|
|
18
|
+
|
|
19
|
+
- It does: catch the stranded state immediately and tell a human (and the logs) the truth, with the exact versions involved and how to re-apply.
|
|
20
|
+
- It doesn't (yet): automatically fix the strand by re-installing. That auto-repair is a deliberate follow-up, because re-installing on the critical update path can loop if the folder keeps getting knocked back, so it needs a careful retry limit and proper review. This change is the safe, observe-only first step: stop hiding the problem.
|
|
21
|
+
|
|
22
|
+
Low risk: it's a new read plus a corrected message; it only triggers on the genuine strand (a normal "applied, waiting for restart" never reaches it), and it's covered by tests.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# ELI16 — Three small dashboard annoyances, cleaned up
|
|
2
|
+
|
|
3
|
+
## The problem in plain words
|
|
4
|
+
|
|
5
|
+
While fixing the big "dashboard keeps saying Disconnected" bug, I noticed three smaller, pre-existing issues. None of them break the connection — they're noise and one misleading light — but they're the kind of papercut that makes a healthy system look unwell.
|
|
6
|
+
|
|
7
|
+
**1. A button that never stopped knocking.** The dashboard checks "is WhatsApp set up?" every 3 seconds by pinging a WhatsApp address. If WhatsApp *isn't* set up (the normal case here), that address politely answers "not available" — but the dashboard kept asking anyway, forever. The result was about twenty error lines a minute piling up in the browser's hidden console. Harmless, but messy, and it buries real errors in the noise.
|
|
8
|
+
|
|
9
|
+
**2. An error thrown on every page load.** Tucked at the very end of the dashboard's code was a line that tried to hook into the live connection to refresh a small "paste" panel. But it ran *too early* — before the connection actually existed — so it tried to use something that wasn't there yet and threw a "can't read a property of nothing" error each time the page loaded. Worse, because it ran at the wrong moment, the paste-refresh it was supposed to enable never actually worked.
|
|
10
|
+
|
|
11
|
+
**3. A warning light stuck on.** At the top of the dashboard there's a little health badge: green "Healthy" or orange "Degraded". It turns orange if *anything* ever reported a hiccup. The catch: those hiccup reports are never erased. So one tiny, long-recovered blip would keep the badge orange for the entire time the server was running — telling you something's wrong when everything is actually fine.
|
|
12
|
+
|
|
13
|
+
## The fix
|
|
14
|
+
|
|
15
|
+
**1.** When the WhatsApp check gets the "not available" answer, stop checking. (If WhatsApp gets set up later, a page refresh starts it again.) The console noise is gone.
|
|
16
|
+
|
|
17
|
+
**2.** Move the paste-refresh handling into the dashboard's normal message router — the place that's always listening once the connection is live. That removes the early-run error *and* makes the paste-refresh actually work, even after the connection drops and comes back.
|
|
18
|
+
|
|
19
|
+
**3.** Make the health badge only count hiccups from the last 30 minutes. A real, ongoing problem keeps reporting itself, so it stays visible — but a one-off blip that recovered quietly ages out, and the badge goes back to green like it should. To be safe, a hiccup with a missing or garbled timestamp is always kept (better to show a maybe-stale warning than to hide a real one).
|
|
20
|
+
|
|
21
|
+
## What it does and doesn't do
|
|
22
|
+
|
|
23
|
+
- It does: silence the console spam, remove the on-load error (and revive the paste-refresh), and make the "Degraded" badge tell the truth about *current* health.
|
|
24
|
+
- It doesn't: change anything about restarts (on this agent the badge doesn't control restarts — that's decided separately by whether the server process is actually alive), and it doesn't touch the full record of past hiccups — only the *summary badge* is time-windowed.
|
|
25
|
+
|
|
26
|
+
Low risk: two of the three are small front-end guards in the dashboard page; the third is a contained, well-tested change to which hiccups the health summary counts. A separate, deeper issue — a different "system review" panel showing data from days ago because it never re-runs — is left for a follow-up, because re-running its checks on a schedule is a bigger, cost-bearing change.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# ELI16 — Make the worktree scan stop freezing the server
|
|
2
|
+
|
|
3
|
+
## The problem in plain words
|
|
4
|
+
|
|
5
|
+
Instar's server runs everything on a single thread — one line at a time, like a single cashier serving a queue. If any one task takes a long time, *everything else waits*: the dashboard's live connection, the message handlers, all of it.
|
|
6
|
+
|
|
7
|
+
One of the background helpers, the **WorktreeMonitor**, periodically looks at all the git "worktrees" (separate working copies of the code) to spot abandoned work. To do that it shells out to `git worktree list` and a few other git commands. The way it called git was **synchronous** — meaning the single thread sat there and *waited* for git to finish before doing anything else.
|
|
8
|
+
|
|
9
|
+
That's fine when there are a handful of worktrees. But on a machine where worktrees pile up (one agent had ~282 of them!), `git worktree list` takes a few *seconds*. And this scan ran on a 5-minute timer **and** every time a background job finished. So every few minutes the whole server froze for a second or two.
|
|
10
|
+
|
|
11
|
+
## Why it showed up as "dashboard disconnected"
|
|
12
|
+
|
|
13
|
+
On the local address you'd never notice — the dashboard's live connection just reconnects instantly. But when you reach the dashboard through the internet tunnel, a one- or two-second server freeze is enough to **drop the live connection and time out the data**. The result is the dashboard showing "Disconnected, 0 sessions, 0% memory" — even though the server is actually fine, it was just frozen for a moment.
|
|
14
|
+
|
|
15
|
+
## The fix
|
|
16
|
+
|
|
17
|
+
Make the git calls **asynchronous**. Instead of the single thread sitting and waiting for git, it kicks off the git command and goes back to serving everyone else; when git finishes, it picks the result back up. Same scans, same schedule, same results — but the server stays responsive the whole time, so the dashboard's live connection never drops.
|
|
18
|
+
|
|
19
|
+
## What changed and what didn't
|
|
20
|
+
|
|
21
|
+
- Changed: the worktree scan's git calls no longer block the server thread.
|
|
22
|
+
- Unchanged: it still scans on the same 5-minute timer and after each job, still reports the same things, and the `/hooks/worktrees` endpoint returns the exact same data. The git calls keep their old behavior of returning whatever output they produced even if git exits with an error.
|
|
23
|
+
|
|
24
|
+
It's a small, low-risk change covered by the existing tests (all green), and it's the permanent replacement for a quick hotpatch that had temporarily disabled the scans to stop the freezing.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Side-Effects Review — Auto-update version-strand detector
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `autoupdate-strand-detector`
|
|
4
|
+
**Date:** `2026-06-23`
|
|
5
|
+
**Author:** Echo (autonomous)
|
|
6
|
+
**Tier:** 1 (observe-only diagnostic — no change to update behavior, no new config, no decision points)
|
|
7
|
+
**Second-pass reviewer:** not-required (Tier 1; the new method + a corrected log/notify branch, covered by unit tests)
|
|
8
|
+
|
|
9
|
+
## Summary of the change
|
|
10
|
+
|
|
11
|
+
The AutoUpdater has a loop-breaker (`if lastAppliedVersion === latestVersion`) that, on every tick, assumes the recorded-applied version is genuinely installed in the shadow and prints a benign "downloaded, waiting for a restart" message. But `getInstalledVersion()` caches the version the running process booted with — it does NOT reflect the LIVE shadow on disk.
|
|
12
|
+
|
|
13
|
+
If the shadow REVERTS after a successful apply (e.g., crash-loop collateral, or a partial re-install) while `lastAppliedVersion` still records the new version, the agent is **stranded**: the updater believes it is current, the loop-breaker never re-applies, and the misleading "waiting for a restart" message hides a permanent stuck state. This actually happened (an agent stranded on the old version while `lastAppliedVersion` claimed the new one).
|
|
14
|
+
|
|
15
|
+
## The change
|
|
16
|
+
|
|
17
|
+
- `src/core/UpdateChecker.ts` — new `getShadowInstalledVersion(): string | null`, an **uncached** read of `{stateDir}/shadow-install/node_modules/instar/package.json`. Distinct from `getInstalledVersion()` (which caches the running-process version).
|
|
18
|
+
- `src/core/AutoUpdater.ts` — in the loop-breaker, read the live shadow-disk version; if it differs from `lastAppliedVersion` (the shadow doesn't actually have what the record claims), emit a distinct loud STRAND warning + a one-time honest notification ("an update was recorded as installed but isn't on disk — I'm stuck until it's re-applied") instead of the benign "awaiting restart" message.
|
|
19
|
+
- `tests/unit/UpdateChecker.test.ts` — 4 tests for `getShadowInstalledVersion` (reads the disk version, reflects a reverted version uncached, returns null when absent / version-less).
|
|
20
|
+
|
|
21
|
+
## Side effects & risk
|
|
22
|
+
|
|
23
|
+
- **Observe-only.** Update behavior is unchanged: the strand path still `return`s (no auto re-apply — a bounded auto-heal is a deliberate follow-up). The change only corrects the diagnosis and the one-time notification text.
|
|
24
|
+
- **No false positives on the benign case.** A genuine "applied, awaiting restart" has shadow-disk === lastAppliedVersion → `updateAvailable` is false → the tick returns before the loop-breaker. Reaching the loop-breaker with shadow-disk ≠ lastAppliedVersion is the strand.
|
|
25
|
+
- **Fail-safe reads.** `getShadowInstalledVersion()` returns null on any read error; a null shadow version never trips the strand branch (it falls through to the existing benign messages).
|
|
26
|
+
- **Risk:** low. New diagnostic method + a corrected branch, both covered by unit tests; AutoUpdater's existing 29 tests stay green.
|
|
27
|
+
|
|
28
|
+
## Verification
|
|
29
|
+
|
|
30
|
+
- `tsc --noEmit`: 0 errors.
|
|
31
|
+
- `tests/unit/UpdateChecker.test.ts`: 17/17 (incl. 4 new).
|
|
32
|
+
- `tests/unit/AutoUpdater.test.ts` + `tests/unit/auto-updater-failures.test.ts`: 29/29.
|
|
33
|
+
|
|
34
|
+
## Rollout
|
|
35
|
+
|
|
36
|
+
No flag, no migration. Strictly additive diagnostic — surfaces a previously-silent stuck state. The bounded auto-re-apply self-heal (which would also FIX the strand, not just surface it) is a tracked Tier-2 follow-up (it mutates the critical update path and warrants spec-converge review).
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Side-Effects Review — Dashboard health-signal polish (3 pre-existing fixes)
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `dashboard-health-signal-polish`
|
|
4
|
+
**Date:** `2026-06-23`
|
|
5
|
+
**Author:** Echo (autonomous)
|
|
6
|
+
**Tier:** 1 (three contained, low-risk fixes — two client-side dashboard guards + one health-status windowing on an endpoint whose `status` field does not gate restarts on this agent)
|
|
7
|
+
**Second-pass reviewer:** not-required (Tier 1; small contained fixes, the server-side change covered by new unit tests)
|
|
8
|
+
|
|
9
|
+
## Summary of the change
|
|
10
|
+
|
|
11
|
+
Three minor, pre-existing dashboard issues found while verifying the dashboard-freeze fix (#1253). None affect the dashboard connection; all are noise/cosmetic.
|
|
12
|
+
|
|
13
|
+
1. **WhatsApp QR poll spam.** `pollWaQr()` polls `/whatsapp/qr` every 3s. When WhatsApp isn't configured the route returns 503, but the poll loop was never stopped — so the browser console accumulated ~20 503s/min for the whole session. Fix: call `stopWaPolling()` on the 503 (adapter absent).
|
|
14
|
+
2. **Null-deref TypeError on load.** A parse-time block attached a `'message'` listener to `ws`, guarded by `if (typeof ws !== 'undefined')`. `ws` is a declared `let` that is **null** at parse time (`connectWebSocket()` runs later in a `.then()`), and `typeof null === 'object' !== 'undefined'`, so the guard passed and `ws.onmessage` threw `Cannot read properties of null (reading 'onmessage')` on every load. The block was the only place paste WebSocket events refreshed the drop-zone — so the feature was also effectively dead. Fix: fold `paste_delivered` / `paste_acknowledged` into the main `handleMessage` switch (where it survives reconnects) and remove the broken parse-time block.
|
|
15
|
+
3. **Stale "Degraded" badge.** The vital-signs badge renders `/health`'s `status`, which is `degraded` whenever `DegradationReporter.getEvents().length > 0`. Degradation events are append-only and never self-clear, so a single transient/benign fallback pins the badge red for the entire process lifetime even after recovery. Fix: `/health` now uses a new `DegradationReporter.getRecentEvents()` (30-min window) for status + the reported count/summary — a persistent problem keeps re-reporting so it stays visible; a one-off ages out.
|
|
16
|
+
|
|
17
|
+
## The change
|
|
18
|
+
|
|
19
|
+
- `dashboard/index.html` — (1) `stopWaPolling()` on 503; (2) paste cases added to `handleMessage`, broken parse-time block removed.
|
|
20
|
+
- `src/monitoring/DegradationReporter.ts` — new `getRecentEvents(windowMs = HEALTH_WINDOW_MS, now = Date.now())` + `static HEALTH_WINDOW_MS = 30min`. Filters by event `timestamp`; an unparseable timestamp is KEPT (fail-safe: surface, never hide). `getEvents()` (full log) is unchanged.
|
|
21
|
+
- `src/server/routes.ts` — `/health` calls `getRecentEvents()` instead of `getEvents()` for the `degradations` list that feeds `status` + `degradations` count + `degradationSummary`.
|
|
22
|
+
- `tests/unit/degradation-reporter.test.ts` — 3 new tests (recent kept / old aged out / unparseable kept).
|
|
23
|
+
|
|
24
|
+
## Side effects & risk
|
|
25
|
+
|
|
26
|
+
- **`/health` status semantics.** `status` flips back to `ok` once degradations age past 30 min without recurring. Reviewed consumers: the **Telegram lifeline** derives `serverHealthy` from the supervisor's process status, NOT from `/health.status`, so this never affects restart behavior on a Telegram agent (Echo). `SlackLifeline` keys on `status === 'ok'` only to detect the unhealthy→healthy *recovery* transition (a log/notify), not to trigger restarts. The change makes status MORE truthful (reflects current health), and is the safe direction (a real, recurring problem keeps re-reporting and stays visible).
|
|
27
|
+
- **The full degradation log is untouched.** `getEvents()`, `getUnreportedEvents()`, mark-reported, and persistence all still see every event — only the /health *status view* is windowed.
|
|
28
|
+
- **Client-side fixes are isolated** to the dashboard static asset; no server behavior changes; the dashboard isn't in the typecheck/test suite, so both were syntax-checked with `node --check`.
|
|
29
|
+
- **Risk:** low. No new config, no new route, no migration (the dashboard asset + core source ship with the package on update). Reversible (revert the three edits).
|
|
30
|
+
|
|
31
|
+
## 6b. Operator-surface quality (Operator-Surface Quality standard)
|
|
32
|
+
|
|
33
|
+
This change touches `dashboard/index.html`. It alters **background behavior**, not layout or primary actions — so it adds no new operator-facing controls or content. Against the four criteria:
|
|
34
|
+
|
|
35
|
+
1. **Leads with the primary action?** Unchanged — the dashboard still leads with the session list + vital-signs strip. No new primary action is introduced; nothing is moved, collapsed, or pushed below the fold.
|
|
36
|
+
2. **Zero raw internals as primary content?** Yes — the change adds NO displayed content. It REMOVES noise (the WhatsApp console 503 spam) and an on-load TypeError, and makes the existing plain-language health badge ("Healthy" / uptime vs "Degraded") accurate. No JSON/UUID/fingerprint/hash is shown.
|
|
37
|
+
3. **Destructive actions de-emphasized?** N/A — no destructive control is added or touched.
|
|
38
|
+
4. **Plain language + phone width?** Yes — the only user-visible behavioral change is the top-of-dashboard badge now correctly reads "Healthy" when the box is healthy (it was stuck on "Degraded"). Plain wording, same layout, same width — no new elements, so no phone-width regression.
|
|
39
|
+
|
|
40
|
+
(The dashboard's pre-existing raw-input field — the PIN/token unlock — already carries the co-located `operator-surface-power-user` marker and is untouched here.)
|
|
41
|
+
|
|
42
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
43
|
+
|
|
44
|
+
**machine-local BY DESIGN.** The vital-signs badge reads THIS machine's `/health`; each machine's `DegradationReporter` and health state are local, per-machine observability (a degradation on machine A is machine A's concern). The dashboard already exposes pool-wide state through separate `?scope=pool` surfaces (sessions, guards, attention, …), which this change does not touch. The WhatsApp poll and the paste WS handler are pure client-side, per-browser-tab.
|
|
45
|
+
|
|
46
|
+
- **User-facing notices?** None emitted — no one-voice gating concern.
|
|
47
|
+
- **Durable state strand on topic transfer?** No durable state added; nothing strands.
|
|
48
|
+
- **Generated URLs?** None.
|
|
49
|
+
|
|
50
|
+
## Verification
|
|
51
|
+
|
|
52
|
+
- `tsc --noEmit`: 0 errors.
|
|
53
|
+
- `tests/unit/degradation-reporter.test.ts`: 20/20 (incl. 3 new windowing tests).
|
|
54
|
+
- `tests/unit/{routes-degradations-mark-reported,degradation-never-silent,degradation-reporter-reentrancy-wedge}.test.ts`: 14/14 (no regression on the mark-reported / never-silent paths).
|
|
55
|
+
- `dashboard/index.html` embedded script: `node --check` passes.
|
|
56
|
+
|
|
57
|
+
## Rollout
|
|
58
|
+
|
|
59
|
+
No flag, no migration. Three additive fixes that surface previously-hidden console errors and make the health badge truthful. The genuinely-stale `systemReview` *panel* (last ran 2026-06-20, never re-runs) is a SEPARATE, more-involved concern (re-running 16 probes on a cadence is cost-bearing) — explicitly NOT bundled here; tracked for follow-up.
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Side-Effects Review — fork-bomb prevention (SIMPLE): host-wide spawn cap + single-instance lock + bounded ingress
|
|
2
|
+
|
|
3
|
+
**Slug:** `forkbomb-prevention-simple` · **Tier:** 2 (converged + approved spec —
|
|
4
|
+
`docs/specs/forkbomb-prevention-simple.md`, cross-model + lessons-aware convergence at
|
|
5
|
+
`docs/specs/reports/forkbomb-prevention-simple-convergence.md`). Parent principles:
|
|
6
|
+
**Structure beats Willpower** (a hard ceiling enforced in code, not a "don't spawn too much"
|
|
7
|
+
note) and **Bounded Blast Radius** (the new constitutional standard this PR lands — safety
|
|
8
|
+
must be physical/capacity-bounded, not only semantic). This is the SEV-1 fix for the
|
|
9
|
+
2026-06-20 host-OOM incident: the Instar server fork-bombed its macOS host into OOM **twice**
|
|
10
|
+
(~230–289 concurrent `claude -p`, ≈ 90–115 GB) because `IntelligenceProvider.evaluate()`
|
|
11
|
+
spawned one LLM subprocess per call with **zero** concurrency control, `CoherenceGate` fanned
|
|
12
|
+
~10 reviewers in parallel per message, and up to three server instances each re-flooded.
|
|
13
|
+
|
|
14
|
+
This is the deliberately SIMPLE redesign — three primitives, ON by default. It SUPERSEDES the
|
|
15
|
+
earlier elaborate predecessor (`docs/specs/forkbomb-prevention.md`): the lanes / reserve-math /
|
|
16
|
+
hysteresis / per-gate-typed-shed of that design are explicitly OUT of scope. The cap caps the
|
|
17
|
+
unbounded LLM-subprocess spawns that directly produced the real crash-loop.
|
|
18
|
+
|
|
19
|
+
## Summary of the change — the 3 primitives + the 4 fail-closed seams
|
|
20
|
+
|
|
21
|
+
**P1 — Host-wide concurrent-spawn cap (the PRIMARY control).** A host-local counting semaphore
|
|
22
|
+
(`src/core/hostSpawnSemaphore.ts`) bounds how many LLM subprocesses run **at once** across every
|
|
23
|
+
compliant Instar process on the host (default **8**). Holder-SET model: it counts live holder
|
|
24
|
+
records under an exclusive `O_CREAT|O_EXCL` flock (reusing the `ProjectRoundLock` primitive), not
|
|
25
|
+
decrement/increment counter math, so it is crash-safe by construction — a double-release is a
|
|
26
|
+
no-op, pid-reuse can't steal a slot, a partial write is discarded. A crashed holder is reclaimed
|
|
27
|
+
only when pid-dead AND heartbeat-stale **on this host**; a foreign-hostname holder is NEVER
|
|
28
|
+
reclaimed (refuse-loud), and a `df -P` host-local-disk check gates reclaim (fail-closed). The cap
|
|
29
|
+
is enforced by a per-`evaluate()` **wrapper provider** (`src/core/SpawnCapIntelligenceProvider.ts`)
|
|
30
|
+
installed at **every** return arm of `buildIntelligenceProvider`, INSIDE the circuit-breaker wrap —
|
|
31
|
+
so `CoherenceGate`'s shared-instance fan-out (one `intelligence` field fanned ~10-wide via
|
|
32
|
+
`Promise.allSettled`) is bound per call. `reflect.ts`'s former raw-fallback construction is routed
|
|
33
|
+
through the same factory funnel.
|
|
34
|
+
|
|
35
|
+
**P2 — Single-instance lock (`src/core/SingleInstanceLock.ts`).** Refuses a duplicate server
|
|
36
|
+
instance of the same agent on a host (the 3× launchd/fleet/tmux multiplier that made the flood
|
|
37
|
+
worse), via the same O_CREAT|O_EXCL pid+hostname+heartbeat holder record. Foreign-host locks are
|
|
38
|
+
never reclaimed; release runs via `finally` + SIGTERM/SIGINT/exit handlers; a bounded deploy-handoff
|
|
39
|
+
grace lets a normal restart hand off cleanly; `INSTAR_ALLOW_SECOND_INSTANCE=1` overrides for a
|
|
40
|
+
deliberate second instance. A genuine duplicate refuses-loud rather than starting a second flooder.
|
|
41
|
+
|
|
42
|
+
**P3 — Bounded ingress + 4 fail-CLOSED gate seams.** A saturated cap makes a spawn request
|
|
43
|
+
poll-retry every ~100 ms up to `acquireMs` (default 5000 ms) — poll-retry, NOT an in-memory waiter
|
|
44
|
+
queue, so no per-waiter heap growth — bounded by a concurrent-poller ceiling (`waitersMax`, default
|
|
45
|
+
64). On genuine exhaustion the call throws a typed `LlmCapacityUnavailableError`
|
|
46
|
+
(`isCapacityUnavailable`). The **four safety-gating seams** fail CLOSED on that shed instead of
|
|
47
|
+
auto-passing:
|
|
48
|
+
1. `MessageSentinel.classify` → held, NOT `category:'normal'`.
|
|
49
|
+
2. `InputGuard.reviewTopicCoherence` → flagged `suspicious`, NOT `verdict:'coherent'`.
|
|
50
|
+
3. `MessagingToneGate.review` → records unavailable, holds `pass:false`.
|
|
51
|
+
4. the OUTBOUND `CoherenceGate` reviewer path → block-the-turn `pass:false`, NOT `pass:true`.
|
|
52
|
+
|
|
53
|
+
A non-gating (background) call sheds to its existing heuristic path — loud + counted, **never
|
|
54
|
+
silent**. The deterministic emergency-stop keyword pre-check runs BEFORE the LLM classifier and is
|
|
55
|
+
EXEMPT from the cap — a "stop everything" halt is never gated on capacity.
|
|
56
|
+
|
|
57
|
+
**Safety-floor posture + observability.** The cap is read with a plain `?? default` (NOT a dev gate)
|
|
58
|
+
— it ships **ON by default for every agent**, tunable via `intelligence.spawnCap`
|
|
59
|
+
(`maxConcurrent`/`acquireMs`/`waitersMax`) or env (`INSTAR_HOST_SPAWN_MAX`, `INSTAR_SPAWN_ACQUIRE_MS`,
|
|
60
|
+
`INSTAR_SPAWN_WAITERS_MAX`), migrated existence-checked. A new `GET /spawn-limiter` route reports
|
|
61
|
+
live holder count, cap, available slots, saturation, and waiters. A conservative OS-level
|
|
62
|
+
`NumberOfProcesses` ceiling (512) is added to the launchd plist as a host-global belt under the
|
|
63
|
+
semaphore. The **"Bounded Blast Radius"** constitutional standard lands in
|
|
64
|
+
`docs/STANDARDS-REGISTRY.md` with its lint + burst-invariant ratchet in the same PR.
|
|
65
|
+
|
|
66
|
+
## 1. Correctness / behavioral equivalence
|
|
67
|
+
|
|
68
|
+
The wrapper is a pure pass-through except for the acquire/release bracket: `evaluate()` acquires a
|
|
69
|
+
host slot, calls `inner.evaluate()`, releases in `finally`. When a slot is available (the normal,
|
|
70
|
+
unsaturated case — ≤8 concurrent) behavior is byte-identical to before; the cap only changes
|
|
71
|
+
behavior under genuine saturation, where the four gating seams hold (the safe direction) and
|
|
72
|
+
background calls degrade to their existing heuristics. The breaker-INSIDE ordering (breaker →
|
|
73
|
+
spawn-cap → provider) ensures a breaker-open shed never holds a slot. Emergency-stop is unchanged:
|
|
74
|
+
it is classified by the deterministic fast-path before any LLM call. Verified: the four fail-closed
|
|
75
|
+
seams are asserted (a capacity shed does NOT produce `normal`/`coherent`/`pass:true`) while a generic
|
|
76
|
+
LLM error still fails open — proving capacity is the ONLY new hold.
|
|
77
|
+
|
|
78
|
+
## 2. Concurrency / multi-process safety
|
|
79
|
+
|
|
80
|
+
The semaphore is the load-bearing concurrency primitive. The burst-invariant ratchet
|
|
81
|
+
(`tests/unit/host-spawn-semaphore-burst-invariant.test.ts`) churns 10,000 acquire/release attempts
|
|
82
|
+
and asserts live holders NEVER exceed the cap (the standing Bounded-Accumulation proof that the OOM
|
|
83
|
+
vector cannot recur), and a TRUE multi-process test forks 12 OS processes racing for 3 slots and
|
|
84
|
+
observes ≤3 successes — the cross-process flock holds the cap. Every mutation is under the exclusive
|
|
85
|
+
flock; the holder-SET model makes double-release / pid-reuse / partial-write all safe by
|
|
86
|
+
construction. The single-instance lock uses the same flock primitive with the same foreign-host /
|
|
87
|
+
host-local-disk guards.
|
|
88
|
+
|
|
89
|
+
## 3. Fail-safe / fail-closed
|
|
90
|
+
|
|
91
|
+
Two deliberately opposite directions, both safe:
|
|
92
|
+
- **Gating calls fail CLOSED** under capacity shed (hold the inbound/outbound turn) — a security/
|
|
93
|
+
coherence gate must never auto-pass when it could not actually evaluate.
|
|
94
|
+
- **Background (non-gating) calls fail OPEN to their heuristic** — loud + counted, never silent
|
|
95
|
+
(each is tagged `@silent-fallback-ok` only where genuinely fail-safe; the `no-silent-llm-fallback`
|
|
96
|
+
+ `no-silent-fallbacks` ratchets stay green).
|
|
97
|
+
- Reclaim is fail-closed: a foreign-hostname holder is never reclaimed; a `df -P` non-host-local-disk
|
|
98
|
+
result refuses reclaim. Uncertainty never frees a slot it shouldn't.
|
|
99
|
+
|
|
100
|
+
## 4. Blast radius
|
|
101
|
+
|
|
102
|
+
New: 3 source modules (`hostSpawnSemaphore`, `SpawnCapIntelligenceProvider`, `SingleInstanceLock`),
|
|
103
|
+
1 lint, 1 route (`GET /spawn-limiter`). Touched: the factory funnel + `reflect.ts` (route the raw
|
|
104
|
+
fallback), `server.ts` boot (configure semaphore + acquire/release the instance lock), the four gate
|
|
105
|
+
seams, `types.ts` + `ConfigDefaults.ts` (the `spawnCap` config), `routes.ts` + `CapabilityIndex.ts`
|
|
106
|
+
(the route), `setup.ts` (launchd belt), `STANDARDS-REGISTRY.md` (the standard),
|
|
107
|
+
`templates.ts` + `PostUpdateMigrator.ts` (CLAUDE.md awareness), `package.json` + the destructive-lint
|
|
108
|
+
allowlist (wire the new lint). No schema change, no write-path semantics change beyond the
|
|
109
|
+
acquire/release bracket. The cap is ON by default — this is intentional (a SAFETY FLOOR), and the
|
|
110
|
+
unsaturated path is behavior-preserving.
|
|
111
|
+
|
|
112
|
+
## 5. On-by-default justification
|
|
113
|
+
|
|
114
|
+
This is a safety floor, not a feature behind a dev gate. The 2026-06-20 incident was a full host OOM
|
|
115
|
+
×2; the failure mode the cap prevents is catastrophic and physical. Reading the cap with `?? default`
|
|
116
|
+
(default 8) means every agent — new and updated — gets the protection without opting in. The default
|
|
117
|
+
of 8 is conservative (well below the ~230+ that OOM'd a 128 GB box) yet ample for normal CoherenceGate
|
|
118
|
+
fan-out; it is fully tunable for a larger host. Migration is existence-checked so an operator's
|
|
119
|
+
explicit override is never clobbered.
|
|
120
|
+
|
|
121
|
+
## 6. Residual (tracked, NOT fixed here — out of scope)
|
|
122
|
+
|
|
123
|
+
- The elaborate predecessor's lanes / reserve-math / hysteresis / per-gate-typed-shed are
|
|
124
|
+
deliberately deferred — the simple design was chosen precisely because it is implementable
|
|
125
|
+
CORRECTLY in one pass. If real load data later shows the single global cap is too coarse, the lane
|
|
126
|
+
model is the documented follow-up.
|
|
127
|
+
- The launchd `NumberOfProcesses` belt is macOS-only; a Linux/systemd equivalent is a follow-up for
|
|
128
|
+
non-macOS hosts (the in-process semaphore is OS-agnostic and is the primary control regardless).
|
|
129
|
+
- No baseline JSON for the new lint — it ships clean (zero violations), so a baseline would be an
|
|
130
|
+
empty no-op (the sibling `lint-no-unfunneled-headless-launch.js` has none either).
|
|
131
|
+
|
|
132
|
+
## 7. Rollback
|
|
133
|
+
|
|
134
|
+
Revert the source files. The semaphore/lock/wrapper are additive; reverting the factory-funnel wrap +
|
|
135
|
+
the `server.ts` boot lines restores the prior unbounded-spawn behavior, and reverting the four gate
|
|
136
|
+
seams restores their prior fail-open. No data migration and no on-disk format to undo — the
|
|
137
|
+
host-spawn-holders / single-instance-lock files are host-local runtime state that is simply abandoned
|
|
138
|
+
on revert. The config defaults are additive and harmless if left in place.
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Side-Effects Review — WorktreeMonitor async git scans
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `worktree-monitor-async-git`
|
|
4
|
+
**Date:** `2026-06-23`
|
|
5
|
+
**Author:** Echo (autonomous)
|
|
6
|
+
**Tier:** 1 (behavior-preserving async refactor — no new feature, no decision points, no dark-gate, no migration)
|
|
7
|
+
**Second-pass reviewer:** not-required (Tier 1; small, low-risk, fully covered by existing unit tests)
|
|
8
|
+
|
|
9
|
+
## Summary of the change
|
|
10
|
+
|
|
11
|
+
`WorktreeMonitor.scanWorktrees()` issues several git commands — `git worktree list --porcelain` plus per-worktree `rev-list`/`diff` — through a private `gitCommand()` that used **`spawnSync('/bin/sh', ['-c', 'git …'])`**. On a repository with many worktrees (Echo accumulated ~282) `git worktree list` alone takes seconds, and the scan ran on BOTH a 5-minute periodic timer AND on every `sessionComplete` event. Each scan therefore blocked the **server event loop** synchronously for seconds.
|
|
12
|
+
|
|
13
|
+
That freeze was invisible on `localhost` (the dashboard websocket reconnects instantly), but over the Cloudflare tunnel (`echo.dawn-tunnel.dev`) a 1–2s freeze times out in-flight requests, dropping the dashboard websocket and failing its polls → the user-visible "Disconnected / 0 sessions / Mem 0%". This is the residual event-loop-block cause behind the dashboard "disconnected" reports, after the sync-READ fixes in #1247–#1251.
|
|
14
|
+
|
|
15
|
+
## The change
|
|
16
|
+
|
|
17
|
+
Convert `gitCommand` from `spawnSync` to async `execFile` (promisified), and propagate `await` through the scan call graph so the event loop stays responsive while git runs in the background.
|
|
18
|
+
|
|
19
|
+
Files modified:
|
|
20
|
+
- `src/monitoring/WorktreeMonitor.ts` — `gitCommand` → `async` (execFile, stdout-on-nonzero-exit/timeout semantics preserved to exactly match the old `spawnSync result.stdout ?? ''`); `listWorktrees`, `checkUnmergedWork`, `findOrphanBranches`, `getDefaultBranch`, `getWorktreeAge`, `scanWorktrees` → `async`; `periodicScan` resolves worktree ages with `Promise.all` before filtering; `formatPeriodicAlert` → `async`.
|
|
21
|
+
- `src/server/routes.ts` — `GET /hooks/worktrees` handler → `async`, awaits `scanWorktrees()`.
|
|
22
|
+
- `tests/unit/worktree-monitor.test.ts` — the 13 tests that called the now-async methods updated to `await` them.
|
|
23
|
+
|
|
24
|
+
## Side effects & risk
|
|
25
|
+
|
|
26
|
+
- **Behavior preserved.** The scans still run on the same triggers and the same 5-minute interval; only the blocking changed to non-blocking. `gitCommand`'s return contract (stdout even on a non-zero exit / timeout) is kept identically, so callers see the same strings.
|
|
27
|
+
- **No external surface change.** `GET /hooks/worktrees` returns the same JSON; the periodic + post-session alert paths are unchanged.
|
|
28
|
+
- **Concurrency.** Scans are serialized per the existing event flow (no new parallelism introduced beyond `Promise.all` over `getWorktreeAge`, which is read-only `git show` per worktree).
|
|
29
|
+
- **Risk:** low. A behavior-preserving async conversion, fully covered by the existing 22 unit tests (updated to await) + the serendipity-routes integration tests, both green.
|
|
30
|
+
|
|
31
|
+
## Verification
|
|
32
|
+
|
|
33
|
+
- `tsc --noEmit`: 0 errors.
|
|
34
|
+
- `tests/unit/worktree-monitor.test.ts`: 22/22 pass.
|
|
35
|
+
- `tests/integration/serendipity-routes.test.ts`: 22/22 pass.
|
|
36
|
+
|
|
37
|
+
## Rollout
|
|
38
|
+
|
|
39
|
+
No flag, no migration — the fix is strictly better behavior on the existing code path and ships on the next release. (A shadow-dist hotpatch — disabling both scan triggers — relieved the freeze immediately on the affected agent; this PR is the durable replacement that keeps the feature working without blocking.)
|