instar 1.3.646 → 1.3.648
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/reflect.d.ts.map +1 -1
- package/dist/commands/reflect.js +7 -3
- package/dist/commands/reflect.js.map +1 -1
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +41 -2
- package/dist/commands/server.js.map +1 -1
- package/dist/commands/setup.d.ts.map +1 -1
- package/dist/commands/setup.js +17 -0
- package/dist/commands/setup.js.map +1 -1
- package/dist/config/ConfigDefaults.d.ts.map +1 -1
- package/dist/config/ConfigDefaults.js +14 -0
- package/dist/config/ConfigDefaults.js.map +1 -1
- package/dist/core/CoherenceGate.d.ts.map +1 -1
- package/dist/core/CoherenceGate.js +24 -0
- package/dist/core/CoherenceGate.js.map +1 -1
- package/dist/core/CoherenceReviewer.d.ts +9 -0
- package/dist/core/CoherenceReviewer.d.ts.map +1 -1
- package/dist/core/CoherenceReviewer.js +18 -2
- package/dist/core/CoherenceReviewer.js.map +1 -1
- package/dist/core/InputGuard.d.ts.map +1 -1
- package/dist/core/InputGuard.js +17 -3
- package/dist/core/InputGuard.js.map +1 -1
- package/dist/core/MessageSentinel.d.ts.map +1 -1
- package/dist/core/MessageSentinel.js +19 -2
- package/dist/core/MessageSentinel.js.map +1 -1
- package/dist/core/MessagingToneGate.d.ts +8 -0
- package/dist/core/MessagingToneGate.d.ts.map +1 -1
- package/dist/core/MessagingToneGate.js +16 -1
- package/dist/core/MessagingToneGate.js.map +1 -1
- package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
- package/dist/core/PostUpdateMigrator.js +10 -0
- package/dist/core/PostUpdateMigrator.js.map +1 -1
- package/dist/core/SingleInstanceLock.d.ts +103 -0
- package/dist/core/SingleInstanceLock.d.ts.map +1 -0
- package/dist/core/SingleInstanceLock.js +300 -0
- package/dist/core/SingleInstanceLock.js.map +1 -0
- package/dist/core/SpawnCapIntelligenceProvider.d.ts +89 -0
- package/dist/core/SpawnCapIntelligenceProvider.d.ts.map +1 -0
- package/dist/core/SpawnCapIntelligenceProvider.js +164 -0
- package/dist/core/SpawnCapIntelligenceProvider.js.map +1 -0
- package/dist/core/hostSpawnSemaphore.d.ts +188 -0
- package/dist/core/hostSpawnSemaphore.d.ts.map +1 -0
- package/dist/core/hostSpawnSemaphore.js +471 -0
- package/dist/core/hostSpawnSemaphore.js.map +1 -0
- package/dist/core/intelligenceProviderFactory.d.ts +0 -12
- package/dist/core/intelligenceProviderFactory.d.ts.map +1 -1
- package/dist/core/intelligenceProviderFactory.js +27 -9
- package/dist/core/intelligenceProviderFactory.js.map +1 -1
- package/dist/core/types.d.ts +18 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/feedback-factory/store/JsonlFeedbackStore.d.ts +2 -0
- package/dist/feedback-factory/store/JsonlFeedbackStore.d.ts.map +1 -1
- package/dist/feedback-factory/store/JsonlFeedbackStore.js +36 -2
- package/dist/feedback-factory/store/JsonlFeedbackStore.js.map +1 -1
- package/dist/messaging/TelegramAdapter.d.ts.map +1 -1
- package/dist/messaging/TelegramAdapter.js +6 -2
- package/dist/messaging/TelegramAdapter.js.map +1 -1
- package/dist/monitoring/CoherenceMonitor.d.ts.map +1 -1
- package/dist/monitoring/CoherenceMonitor.js +6 -1
- package/dist/monitoring/CoherenceMonitor.js.map +1 -1
- package/dist/monitoring/CommitmentSentinel.d.ts.map +1 -1
- package/dist/monitoring/CommitmentSentinel.js +7 -2
- package/dist/monitoring/CommitmentSentinel.js.map +1 -1
- package/dist/scaffold/templates.d.ts.map +1 -1
- package/dist/scaffold/templates.js +5 -0
- package/dist/scaffold/templates.js.map +1 -1
- package/dist/server/CapabilityIndex.d.ts.map +1 -1
- package/dist/server/CapabilityIndex.js +9 -0
- package/dist/server/CapabilityIndex.js.map +1 -1
- package/dist/server/routes.d.ts.map +1 -1
- package/dist/server/routes.js +28 -0
- package/dist/server/routes.js.map +1 -1
- package/dist/utils/jsonl-tail.d.ts +50 -0
- package/dist/utils/jsonl-tail.d.ts.map +1 -0
- package/dist/utils/jsonl-tail.js +96 -0
- package/dist/utils/jsonl-tail.js.map +1 -0
- package/package.json +4 -2
- package/scripts/lint-no-direct-destructive.js +4 -0
- package/scripts/lint-no-unbounded-llm-spawn.js +137 -0
- package/src/data/builtin-manifest.json +63 -63
- package/src/scaffold/templates.ts +5 -0
- package/upgrades/1.3.647.md +56 -0
- package/upgrades/1.3.648.md +99 -0
- package/upgrades/eli16/eventloop-bounded-jsonl-reads.md +47 -0
- package/upgrades/side-effects/eventloop-bounded-jsonl-reads.md +96 -0
- package/upgrades/side-effects/forkbomb-prevention-simple.md +138 -0
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: minor -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
Fork-bomb prevention — the SIMPLE design (`docs/specs/forkbomb-prevention-simple.md`).
|
|
9
|
+
On 2026-06-20 the Instar server fork-bombed its macOS host into OOM **twice**
|
|
10
|
+
(~230-289 concurrent `claude -p` ≈ 90-115GB): `evaluate()` spawned one LLM
|
|
11
|
+
subprocess per call with **zero** concurrency control, `CoherenceGate` fanned ~10
|
|
12
|
+
reviewers in parallel per message, and up to three server instances each
|
|
13
|
+
re-flooded. This release adds the three structural primitives that make that
|
|
14
|
+
physically impossible:
|
|
15
|
+
|
|
16
|
+
- **P1 — Host-wide concurrent-spawn cap (the PRIMARY control).** A host-local
|
|
17
|
+
counting semaphore (`src/core/hostSpawnSemaphore.ts`) bounds how many LLM
|
|
18
|
+
subprocesses run **at once** across every compliant Instar process on the host
|
|
19
|
+
(default **8**). It uses a holder-SET model (count live holder records under an
|
|
20
|
+
exclusive flock, not decrement/increment counter math) so it is crash-safe by
|
|
21
|
+
construction — a double-release is a no-op, a pid-reuse can't steal a slot, a
|
|
22
|
+
partial write is discarded, a crashed holder is reclaimed (pid-dead AND
|
|
23
|
+
heartbeat-stale, **this host only**; a foreign-hostname holder is NEVER
|
|
24
|
+
reclaimed; a `df -P` host-local-disk check gates reclaim, fail-closed). The cap
|
|
25
|
+
is enforced by a per-`evaluate()` wrapper provider
|
|
26
|
+
(`src/core/SpawnCapIntelligenceProvider.ts`) installed at **every** return arm
|
|
27
|
+
of `buildIntelligenceProvider` (the factory funnel) — so CoherenceGate's
|
|
28
|
+
shared-instance fan-out is bound per call. A lint
|
|
29
|
+
(`scripts/lint-no-unbounded-llm-spawn.js`) forward-guards any new provider
|
|
30
|
+
constructed outside the funnel.
|
|
31
|
+
- **P2 — Single-instance lock (`src/core/SingleInstanceLock.ts`).** Refuses a
|
|
32
|
+
duplicate server instance of the same agent on a host (the 3× launchd/fleet/tmux
|
|
33
|
+
multiplier), with a bounded deploy-handoff grace so a normal restart hands off
|
|
34
|
+
cleanly. Foreign-host locks are never reclaimed; `INSTAR_ALLOW_SECOND_INSTANCE=1`
|
|
35
|
+
overrides for a deliberate second instance.
|
|
36
|
+
- **P3 — Bounded ingress.** A saturated cap makes a spawn request poll-retry up to
|
|
37
|
+
`acquireMs` (default 5000ms, ~100ms interval, no in-memory waiter queue so no
|
|
38
|
+
heap growth), bounded by a concurrent-poller ceiling (`waitersMax`, default 64).
|
|
39
|
+
On genuine exhaustion the call throws a typed `LlmCapacityUnavailableError`. The
|
|
40
|
+
**four safety-gating seams** fail CLOSED on that shed instead of auto-passing:
|
|
41
|
+
`MessageSentinel.classify` (held, not `normal`), `InputGuard.reviewTopicCoherence`
|
|
42
|
+
(flagged `suspicious`, not `coherent`), `MessagingToneGate.review` (held
|
|
43
|
+
`pass:false`), and the outbound `CoherenceGate` reviewer path (block-the-turn
|
|
44
|
+
`pass:false`). The deterministic emergency-stop keyword pre-check runs **before**
|
|
45
|
+
the LLM classifier and is **exempt** from the cap — a "stop everything" halt
|
|
46
|
+
always halts.
|
|
47
|
+
|
|
48
|
+
The cap is a **SAFETY FLOOR**: it ships **ON by default** for every agent (read
|
|
49
|
+
with a plain `?? default`, never a dev gate), tunable via `intelligence.spawnCap`
|
|
50
|
+
(`maxConcurrent`/`acquireMs`/`waitersMax`) or env (`INSTAR_HOST_SPAWN_MAX`,
|
|
51
|
+
`INSTAR_SPAWN_ACQUIRE_MS`, `INSTAR_SPAWN_WAITERS_MAX`). A new `GET /spawn-limiter`
|
|
52
|
+
route reports live holder count, cap, available slots, saturation, and waiters. A
|
|
53
|
+
conservative OS-level `NumberOfProcesses` ceiling (512) is added to the launchd
|
|
54
|
+
plist as a host-global belt under the semaphore. A new constitutional standard —
|
|
55
|
+
**"Bounded Blast Radius"** — lands in `docs/STANDARDS-REGISTRY.md` with its lint +
|
|
56
|
+
burst-invariant ratchet in the same PR.
|
|
57
|
+
|
|
58
|
+
## What to Tell Your User
|
|
59
|
+
|
|
60
|
+
- **Fork-bomb protection is on**: "I now have a hard ceiling on how much work I can
|
|
61
|
+
run at the same time, so I can never flood your machine into running out of
|
|
62
|
+
memory again — there's a safe limit (8 by default) that's always on, plus a guard
|
|
63
|
+
that stops a duplicate of me from starting up. You can ask me 'are we protected
|
|
64
|
+
against a fork-bomb?' or 'how many tasks am I running right now?' any time."
|
|
65
|
+
|
|
66
|
+
## Summary of New Capabilities
|
|
67
|
+
|
|
68
|
+
| Capability | How to Use |
|
|
69
|
+
|-----------|-----------|
|
|
70
|
+
| Host-wide concurrent-LLM-spawn cap (SEV-1 OOM fix) | Automatic — on by default (limit 8); tune with `intelligence.spawnCap` or env |
|
|
71
|
+
| Spawn-cap status | `GET /spawn-limiter` |
|
|
72
|
+
| Single-instance lock (no duplicate server) | Automatic at server boot; override with `INSTAR_ALLOW_SECOND_INSTANCE=1` |
|
|
73
|
+
| Fail-closed gating under capacity pressure | Automatic — the four inbound/outbound safety gates hold instead of auto-passing |
|
|
74
|
+
|
|
75
|
+
## Evidence
|
|
76
|
+
|
|
77
|
+
This release fixes a SEV-1 (full host OOM ×2 on 2026-06-20). Evidence:
|
|
78
|
+
|
|
79
|
+
- **Reproduction → fix mechanism.** The incident driver — one `claude -p` per
|
|
80
|
+
`evaluate()` with no concurrency control, fanned ~10-wide by CoherenceGate — is
|
|
81
|
+
now bound at the factory funnel: every provider `buildIntelligenceProvider`
|
|
82
|
+
returns is wrapped per-call by `SpawnCapIntelligenceProvider`, which acquires a
|
|
83
|
+
host-wide slot before the inner spawn and releases it in `finally`.
|
|
84
|
+
- **Burst-invariant ratchet** (`tests/unit/host-spawn-semaphore-burst-invariant.test.ts`):
|
|
85
|
+
10,000 churned acquire/release attempts NEVER let live holders exceed the cap;
|
|
86
|
+
a TRUE multi-process test forks 12 OS processes racing for 3 slots and observes
|
|
87
|
+
≤3 successes (the cross-process flock holds the cap). This is the standing
|
|
88
|
+
Bounded-Accumulation proof that the OOM vector cannot recur.
|
|
89
|
+
- **Four fail-closed seams** (`tests/unit/spawn-cap-fail-closed-gates.test.ts`): a
|
|
90
|
+
capacity shed at each gating seam is asserted to HOLD (not produce
|
|
91
|
+
`normal`/`coherent`/`pass:true`), while a generic LLM error still fails open —
|
|
92
|
+
proving capacity is the only new hold. Emergency-stop is asserted exempt
|
|
93
|
+
(classified by the deterministic fast-path before any LLM call).
|
|
94
|
+
- **E2E lifecycle** (`tests/e2e/spawn-cap-forkbomb-lifecycle.test.ts`):
|
|
95
|
+
`GET /spawn-limiter` is alive (200, cap 8, on by default) on a production-init
|
|
96
|
+
server, and a second single-instance-lock acquisition while a live holder exists
|
|
97
|
+
is refused (the duplicate-flood guard), while a clean release hands off.
|
|
98
|
+
- All three test tiers green; `tsc --noEmit` clean; the new lint + the
|
|
99
|
+
`no-silent-llm-fallback` + `no-silent-fallbacks` ratchets green.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Bounded tail-reads for the 12MB telegram-log + 14MB feedback-store — ELI16
|
|
2
|
+
|
|
3
|
+
## What this is
|
|
4
|
+
|
|
5
|
+
Your agent's server runs everything on one main thread. If any single operation makes that thread
|
|
6
|
+
wait, *everything* waits — the dashboard, your messages, every background check. When that wait is
|
|
7
|
+
long enough, the dashboard's live connection drops (the "Disconnected" flapping you may have seen).
|
|
8
|
+
|
|
9
|
+
This fixes the THIRD and FOURTH freezes behind that flapping. (The first was a slow shared terminal
|
|
10
|
+
manager; the second was reading the macOS keychain the blocking way; the third was a 13MB job-history
|
|
11
|
+
file re-read whole every minute — all already fixed.) This one is about two more big files.
|
|
12
|
+
|
|
13
|
+
## What was wrong
|
|
14
|
+
|
|
15
|
+
Your agent keeps a running log of every Telegram message it has ever sent or received, in a file
|
|
16
|
+
called `telegram-messages.jsonl`. That file has grown to about **12 megabytes**. Several background
|
|
17
|
+
checks only ever need to glance at the LAST few messages in that log — the last 50, or the last 100.
|
|
18
|
+
But the way they were written, they read the **entire 12MB file** from disk and chopped it up, just
|
|
19
|
+
to look at the tail. And some of these checks run on a **5-minute timer**, every few minutes, forever.
|
|
20
|
+
Each one froze the server for up to **20 seconds**. (A second large file, a 14MB feedback log, had
|
|
21
|
+
the same problem: it was re-read and re-parsed from scratch every processing pass even when nothing
|
|
22
|
+
had changed.)
|
|
23
|
+
|
|
24
|
+
A frozen main thread uses almost no CPU, so the freeze even looked to the agent like the laptop had
|
|
25
|
+
briefly gone to sleep — a misleading signal on top of a real problem.
|
|
26
|
+
|
|
27
|
+
## What's new
|
|
28
|
+
|
|
29
|
+
Two simple changes, both behavior-preserving:
|
|
30
|
+
|
|
31
|
+
- The background checks that only want the tail of the message log now read just the **last chunk**
|
|
32
|
+
of the file (a 512-kilobyte window — far more than enough to hold the last few hundred messages),
|
|
33
|
+
not the whole thing. So they're instant no matter how big the file gets.
|
|
34
|
+
- The 14MB feedback log now remembers the file's size and last-modified time. If nothing changed
|
|
35
|
+
since it last read it, it returns the copy it already has, reading nothing from disk.
|
|
36
|
+
|
|
37
|
+
So those per-call whole-file reads are gone from every one of these hot paths, and the freezes they
|
|
38
|
+
caused are eliminated. Everything these checks actually do — what they look for, how they parse,
|
|
39
|
+
how they recover from a corrupt file — is unchanged, and proven by tests. One of the new tests even
|
|
40
|
+
plants a problem message at the very END of a multi-megabyte log to prove the tail-read still catches
|
|
41
|
+
recent entries.
|
|
42
|
+
|
|
43
|
+
## What you need to decide
|
|
44
|
+
|
|
45
|
+
Nothing. It's self-contained and low-risk, fully covered by tests. One honest note: the log files
|
|
46
|
+
themselves still grow large over time — putting a size cap on them is a separate cleanup I've flagged
|
|
47
|
+
to track, not fixed here. Reading them is no longer a freeze; capping their growth comes next.
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# Side-Effects Review — bounded tail-reads for the 12MB telegram-log + 14MB feedback-store (event-loop freeze fix)
|
|
2
|
+
|
|
3
|
+
**Slug:** `eventloop-bounded-jsonl-reads` · **Tier:** 1 (focused low-risk bounded-read class fix, no
|
|
4
|
+
spec; live-profiler root-cause). Parent principles: **Structure beats Willpower** — never block the
|
|
5
|
+
event loop (the THIRD + FOURTH distinct event-loop-blocker class behind the dashboard "disconnected"
|
|
6
|
+
flapping, after sync tmux + sync keychain + the JobRunHistory 13MB re-read); and **Bounded
|
|
7
|
+
Accumulation** (the size of an append-only log must never determine the cost of reading it).
|
|
8
|
+
|
|
9
|
+
## Summary of the change
|
|
10
|
+
|
|
11
|
+
FOUR callers each did `fs.readFileSync(messageLogPath, 'utf-8')` on the ~12MB
|
|
12
|
+
`.instar/telegram-messages.jsonl` just to inspect the last 50–200 lines:
|
|
13
|
+
- `CommitmentSentinel.parseRecentMessages` (`src/monitoring/CommitmentSentinel.ts`) — 5-minute timer,
|
|
14
|
+
only ever looks at the last `maxMessagesPerScan*10` lines.
|
|
15
|
+
- `CoherenceMonitor` output-sanity check (`src/monitoring/CoherenceMonitor.ts`) — 5-minute timer,
|
|
16
|
+
only ever looks at the last 50 agent messages.
|
|
17
|
+
- `checkLogForAgentResponse` (PresenceProxy ack path, `src/commands/server.ts`) — event-driven on
|
|
18
|
+
every PresenceProxy tier verdict, only ever looks at the last 50 lines.
|
|
19
|
+
- `TelegramAdapter.getMessageLog` (`src/messaging/TelegramAdapter.ts`) — analysis route, only ever
|
|
20
|
+
returns the last `limit` (default 100) entries.
|
|
21
|
+
|
|
22
|
+
A 12MB synchronous read + `.split('\n')` on a 5-minute timer froze the event loop for up to ~20s per
|
|
23
|
+
pass — the same ~0-CPU false-sleep signature the JobRunHistory fix removed. PLUS `JsonlFeedbackStore`
|
|
24
|
+
(`src/feedback-factory/store/JsonlFeedbackStore.ts`) re-read + re-`JSON.parse`d the entire ~14MB
|
|
25
|
+
`feedback.jsonl` at the start of every processing pass even when nothing had changed.
|
|
26
|
+
|
|
27
|
+
The fix is two behavior-preserving mechanisms:
|
|
28
|
+
|
|
29
|
+
**(A) A shared bounded tail-read utility** — `src/utils/jsonl-tail.ts` (`readJsonlTailLines` /
|
|
30
|
+
`readJsonlTailLastLines`). It `statSync`s the file (O(1)), `openSync` + `readSync`s only the trailing
|
|
31
|
+
`maxBytes` (default 512KB ≈ ~2,600 recent lines, far more than any caller needs) into a fixed buffer,
|
|
32
|
+
drops the first partial line when the window started mid-file, and returns the trailing lines in file
|
|
33
|
+
order. It NEVER loads the whole file and NEVER throws (a read failure returns an empty result,
|
|
34
|
+
matching the `@silent-fallback-ok` behavior of every former full-file caller). It mirrors
|
|
35
|
+
`CoherenceJournal.readTailTolerant`, generalized off that journal's typed-entry shape. The four
|
|
36
|
+
telegram-log readers above now call it instead of `readFileSync`.
|
|
37
|
+
|
|
38
|
+
**(B) A `(size, mtimeMs)` load cache in `JsonlFeedbackStore.loadJsonl`** — when the on-disk file is
|
|
39
|
+
byte-for-byte the one last folded, it serves a clone of the prior parse with ZERO IO/parse; any
|
|
40
|
+
change (append, shrink, delete) re-reads from disk. Mirrors the JobRunHistory cache pattern. The
|
|
41
|
+
clone-on-serve guarantees a caller mutating the returned Map can never poison the cache.
|
|
42
|
+
|
|
43
|
+
## 1. Correctness / behavioral equivalence
|
|
44
|
+
|
|
45
|
+
The telegram-log readers each consumed only a fixed tail (last 50 / last 200 / last `limit`) of the
|
|
46
|
+
log, so the 512KB window (≈ 2,600 lines) is strictly a superset of every caller's need — the parsed +
|
|
47
|
+
filtered result is identical to the old full-split-then-slice. The partial-first-line drop guarantees
|
|
48
|
+
a truncated record at the window boundary is never mis-parsed. The feedback store's load semantics
|
|
49
|
+
(per-line parse, dedup-last-wins-by-id, torn-line skip, corrupt-file recovery) are preserved exactly;
|
|
50
|
+
the cache is only consulted when the fingerprint matches and is invalidated on any change. Verified:
|
|
51
|
+
48/48 across the four touched test files (`jsonl-tail` 9, `CoherenceMonitor-bounded-read` 2,
|
|
52
|
+
`CommitmentSentinel` 26, `jsonl-feedback-store` 11) plus the cross-impact suites; tsc exit 0; lints
|
|
53
|
+
green. The two new regression tests spy on `fs.readFileSync` to FAIL if the whole telegram log is
|
|
54
|
+
read, AND prove a bad pattern in a RECENT agent message at the END of a multi-MB log is still
|
|
55
|
+
detected via the tail.
|
|
56
|
+
|
|
57
|
+
## 2. Cache-coherence / multi-writer safety
|
|
58
|
+
|
|
59
|
+
The telegram-log tail-read holds no state — every call re-reads the live tail, so a concurrent
|
|
60
|
+
appender is always reflected. The feedback-store `(size, mtimeMs)` key invalidates on any byte/mtime
|
|
61
|
+
change (append, compaction rewrite, delete-then-recreate), so an unchanged file is the only cache-hit
|
|
62
|
+
case and a stale fold can never be served. The cache is per-process in-memory (not shared) — no
|
|
63
|
+
cross-process coherence concern (each process reads its own view, same as before). `loadCache` is
|
|
64
|
+
keyed per absolute path and dropped on `existsSync` failure.
|
|
65
|
+
|
|
66
|
+
## 3. Fail-safe
|
|
67
|
+
|
|
68
|
+
`jsonl-tail.ts` never throws: missing file, stat failure, open/read failure each return an empty
|
|
69
|
+
result — exactly the prior `@silent-fallback-ok` degradation of the full-file callers (an
|
|
70
|
+
observability/housekeeping read must never endanger the observed operation). The feedback store's
|
|
71
|
+
stat-failure path falls through to a full read (`/* fall through to a full read — never let stat
|
|
72
|
+
failure skip the load */`), so a stat error degrades to the correct (slow) result, never wrong data.
|
|
73
|
+
|
|
74
|
+
## 4. Blast radius
|
|
75
|
+
|
|
76
|
+
One new utility module + its test; four read-site one-liner swaps (CommitmentSentinel, CoherenceMonitor,
|
|
77
|
+
server.ts PresenceProxy ack, TelegramAdapter.getMessageLog); one read-path cache in JsonlFeedbackStore
|
|
78
|
+
+ a test-only cache-clear export. No API/route change, no schema change, no write-path semantics
|
|
79
|
+
change, no new external surface.
|
|
80
|
+
|
|
81
|
+
## 5. Residual (tracked, NOT fixed here — out of scope)
|
|
82
|
+
|
|
83
|
+
The existing `lint-no-wholefile-sync-read` ratchet only catches LITERAL path basenames in the read
|
|
84
|
+
call, but these four sites read VARIABLE paths (`this.messageLogPath`, a `logPath` param), so the
|
|
85
|
+
lint could never have flagged them — which is why they survived the earlier sweeps. The durable
|
|
86
|
+
structural prevention is an **accessor funnel** (route all append-log reads through a single bounded
|
|
87
|
+
accessor that the lint can enforce), which is **Bounded Accumulation Increment 2** — a tracked
|
|
88
|
+
follow-up named here so it is not silently dropped. Separately, the telegram-messages.jsonl and
|
|
89
|
+
feedback.jsonl files themselves still grow UNBOUNDED — a size/retention cap is the same Bounded
|
|
90
|
+
Accumulation track. This change makes READS cheap regardless of size; it does not cap the files.
|
|
91
|
+
|
|
92
|
+
## 6. Rollback
|
|
93
|
+
|
|
94
|
+
Revert the source files. The change is purely read-path (a bounded tail-read + a fingerprint cache);
|
|
95
|
+
reverting restores the (slow but correct) full-read behavior. No data migration, no on-disk format
|
|
96
|
+
change to undo.
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Side-Effects Review — fork-bomb prevention (SIMPLE): host-wide spawn cap + single-instance lock + bounded ingress
|
|
2
|
+
|
|
3
|
+
**Slug:** `forkbomb-prevention-simple` · **Tier:** 2 (converged + approved spec —
|
|
4
|
+
`docs/specs/forkbomb-prevention-simple.md`, cross-model + lessons-aware convergence at
|
|
5
|
+
`docs/specs/reports/forkbomb-prevention-simple-convergence.md`). Parent principles:
|
|
6
|
+
**Structure beats Willpower** (a hard ceiling enforced in code, not a "don't spawn too much"
|
|
7
|
+
note) and **Bounded Blast Radius** (the new constitutional standard this PR lands — safety
|
|
8
|
+
must be physical/capacity-bounded, not only semantic). This is the SEV-1 fix for the
|
|
9
|
+
2026-06-20 host-OOM incident: the Instar server fork-bombed its macOS host into OOM **twice**
|
|
10
|
+
(~230–289 concurrent `claude -p`, ≈ 90–115 GB) because `IntelligenceProvider.evaluate()`
|
|
11
|
+
spawned one LLM subprocess per call with **zero** concurrency control, `CoherenceGate` fanned
|
|
12
|
+
~10 reviewers in parallel per message, and up to three server instances each re-flooded.
|
|
13
|
+
|
|
14
|
+
This is the deliberately SIMPLE redesign — three primitives, ON by default. It SUPERSEDES the
|
|
15
|
+
earlier elaborate predecessor (`docs/specs/forkbomb-prevention.md`): the lanes / reserve-math /
|
|
16
|
+
hysteresis / per-gate-typed-shed of that design are explicitly OUT of scope. The cap caps the
|
|
17
|
+
unbounded LLM-subprocess spawns that directly produced the real crash-loop.
|
|
18
|
+
|
|
19
|
+
## Summary of the change — the 3 primitives + the 4 fail-closed seams
|
|
20
|
+
|
|
21
|
+
**P1 — Host-wide concurrent-spawn cap (the PRIMARY control).** A host-local counting semaphore
|
|
22
|
+
(`src/core/hostSpawnSemaphore.ts`) bounds how many LLM subprocesses run **at once** across every
|
|
23
|
+
compliant Instar process on the host (default **8**). Holder-SET model: it counts live holder
|
|
24
|
+
records under an exclusive `O_CREAT|O_EXCL` flock (reusing the `ProjectRoundLock` primitive), not
|
|
25
|
+
decrement/increment counter math, so it is crash-safe by construction — a double-release is a
|
|
26
|
+
no-op, pid-reuse can't steal a slot, a partial write is discarded. A crashed holder is reclaimed
|
|
27
|
+
only when pid-dead AND heartbeat-stale **on this host**; a foreign-hostname holder is NEVER
|
|
28
|
+
reclaimed (refuse-loud), and a `df -P` host-local-disk check gates reclaim (fail-closed). The cap
|
|
29
|
+
is enforced by a per-`evaluate()` **wrapper provider** (`src/core/SpawnCapIntelligenceProvider.ts`)
|
|
30
|
+
installed at **every** return arm of `buildIntelligenceProvider`, INSIDE the circuit-breaker wrap —
|
|
31
|
+
so `CoherenceGate`'s shared-instance fan-out (one `intelligence` field fanned ~10-wide via
|
|
32
|
+
`Promise.allSettled`) is bound per call. `reflect.ts`'s former raw-fallback construction is routed
|
|
33
|
+
through the same factory funnel.
|
|
34
|
+
|
|
35
|
+
**P2 — Single-instance lock (`src/core/SingleInstanceLock.ts`).** Refuses a duplicate server
|
|
36
|
+
instance of the same agent on a host (the 3× launchd/fleet/tmux multiplier that made the flood
|
|
37
|
+
worse), via the same O_CREAT|O_EXCL pid+hostname+heartbeat holder record. Foreign-host locks are
|
|
38
|
+
never reclaimed; release runs via `finally` + SIGTERM/SIGINT/exit handlers; a bounded deploy-handoff
|
|
39
|
+
grace lets a normal restart hand off cleanly; `INSTAR_ALLOW_SECOND_INSTANCE=1` overrides for a
|
|
40
|
+
deliberate second instance. A genuine duplicate refuses-loud rather than starting a second flooder.
|
|
41
|
+
|
|
42
|
+
**P3 — Bounded ingress + 4 fail-CLOSED gate seams.** A saturated cap makes a spawn request
|
|
43
|
+
poll-retry every ~100 ms up to `acquireMs` (default 5000 ms) — poll-retry, NOT an in-memory waiter
|
|
44
|
+
queue, so no per-waiter heap growth — bounded by a concurrent-poller ceiling (`waitersMax`, default
|
|
45
|
+
64). On genuine exhaustion the call throws a typed `LlmCapacityUnavailableError`
|
|
46
|
+
(`isCapacityUnavailable`). The **four safety-gating seams** fail CLOSED on that shed instead of
|
|
47
|
+
auto-passing:
|
|
48
|
+
1. `MessageSentinel.classify` → held, NOT `category:'normal'`.
|
|
49
|
+
2. `InputGuard.reviewTopicCoherence` → flagged `suspicious`, NOT `verdict:'coherent'`.
|
|
50
|
+
3. `MessagingToneGate.review` → records unavailable, holds `pass:false`.
|
|
51
|
+
4. the OUTBOUND `CoherenceGate` reviewer path → block-the-turn `pass:false`, NOT `pass:true`.
|
|
52
|
+
|
|
53
|
+
A non-gating (background) call sheds to its existing heuristic path — loud + counted, **never
|
|
54
|
+
silent**. The deterministic emergency-stop keyword pre-check runs BEFORE the LLM classifier and is
|
|
55
|
+
EXEMPT from the cap — a "stop everything" halt is never gated on capacity.
|
|
56
|
+
|
|
57
|
+
**Safety-floor posture + observability.** The cap is read with a plain `?? default` (NOT a dev gate)
|
|
58
|
+
— it ships **ON by default for every agent**, tunable via `intelligence.spawnCap`
|
|
59
|
+
(`maxConcurrent`/`acquireMs`/`waitersMax`) or env (`INSTAR_HOST_SPAWN_MAX`, `INSTAR_SPAWN_ACQUIRE_MS`,
|
|
60
|
+
`INSTAR_SPAWN_WAITERS_MAX`), migrated existence-checked. A new `GET /spawn-limiter` route reports
|
|
61
|
+
live holder count, cap, available slots, saturation, and waiters. A conservative OS-level
|
|
62
|
+
`NumberOfProcesses` ceiling (512) is added to the launchd plist as a host-global belt under the
|
|
63
|
+
semaphore. The **"Bounded Blast Radius"** constitutional standard lands in
|
|
64
|
+
`docs/STANDARDS-REGISTRY.md` with its lint + burst-invariant ratchet in the same PR.
|
|
65
|
+
|
|
66
|
+
## 1. Correctness / behavioral equivalence
|
|
67
|
+
|
|
68
|
+
The wrapper is a pure pass-through except for the acquire/release bracket: `evaluate()` acquires a
|
|
69
|
+
host slot, calls `inner.evaluate()`, releases in `finally`. When a slot is available (the normal,
|
|
70
|
+
unsaturated case — ≤8 concurrent) behavior is byte-identical to before; the cap only changes
|
|
71
|
+
behavior under genuine saturation, where the four gating seams hold (the safe direction) and
|
|
72
|
+
background calls degrade to their existing heuristics. The breaker-INSIDE ordering (breaker →
|
|
73
|
+
spawn-cap → provider) ensures a breaker-open shed never holds a slot. Emergency-stop is unchanged:
|
|
74
|
+
it is classified by the deterministic fast-path before any LLM call. Verified: the four fail-closed
|
|
75
|
+
seams are asserted (a capacity shed does NOT produce `normal`/`coherent`/`pass:true`) while a generic
|
|
76
|
+
LLM error still fails open — proving capacity is the ONLY new hold.
|
|
77
|
+
|
|
78
|
+
## 2. Concurrency / multi-process safety
|
|
79
|
+
|
|
80
|
+
The semaphore is the load-bearing concurrency primitive. The burst-invariant ratchet
|
|
81
|
+
(`tests/unit/host-spawn-semaphore-burst-invariant.test.ts`) churns 10,000 acquire/release attempts
|
|
82
|
+
and asserts live holders NEVER exceed the cap (the standing Bounded-Accumulation proof that the OOM
|
|
83
|
+
vector cannot recur), and a TRUE multi-process test forks 12 OS processes racing for 3 slots and
|
|
84
|
+
observes ≤3 successes — the cross-process flock holds the cap. Every mutation is under the exclusive
|
|
85
|
+
flock; the holder-SET model makes double-release / pid-reuse / partial-write all safe by
|
|
86
|
+
construction. The single-instance lock uses the same flock primitive with the same foreign-host /
|
|
87
|
+
host-local-disk guards.
|
|
88
|
+
|
|
89
|
+
## 3. Fail-safe / fail-closed
|
|
90
|
+
|
|
91
|
+
Two deliberately opposite directions, both safe:
|
|
92
|
+
- **Gating calls fail CLOSED** under capacity shed (hold the inbound/outbound turn) — a security/
|
|
93
|
+
coherence gate must never auto-pass when it could not actually evaluate.
|
|
94
|
+
- **Background (non-gating) calls fail OPEN to their heuristic** — loud + counted, never silent
|
|
95
|
+
(each is tagged `@silent-fallback-ok` only where genuinely fail-safe; the `no-silent-llm-fallback`
|
|
96
|
+
+ `no-silent-fallbacks` ratchets stay green).
|
|
97
|
+
- Reclaim is fail-closed: a foreign-hostname holder is never reclaimed; a `df -P` non-host-local-disk
|
|
98
|
+
result refuses reclaim. Uncertainty never frees a slot it shouldn't.
|
|
99
|
+
|
|
100
|
+
## 4. Blast radius
|
|
101
|
+
|
|
102
|
+
New: 3 source modules (`hostSpawnSemaphore`, `SpawnCapIntelligenceProvider`, `SingleInstanceLock`),
|
|
103
|
+
1 lint, 1 route (`GET /spawn-limiter`). Touched: the factory funnel + `reflect.ts` (route the raw
|
|
104
|
+
fallback), `server.ts` boot (configure semaphore + acquire/release the instance lock), the four gate
|
|
105
|
+
seams, `types.ts` + `ConfigDefaults.ts` (the `spawnCap` config), `routes.ts` + `CapabilityIndex.ts`
|
|
106
|
+
(the route), `setup.ts` (launchd belt), `STANDARDS-REGISTRY.md` (the standard),
|
|
107
|
+
`templates.ts` + `PostUpdateMigrator.ts` (CLAUDE.md awareness), `package.json` + the destructive-lint
|
|
108
|
+
allowlist (wire the new lint). No schema change, no write-path semantics change beyond the
|
|
109
|
+
acquire/release bracket. The cap is ON by default — this is intentional (a SAFETY FLOOR), and the
|
|
110
|
+
unsaturated path is behavior-preserving.
|
|
111
|
+
|
|
112
|
+
## 5. On-by-default justification
|
|
113
|
+
|
|
114
|
+
This is a safety floor, not a feature behind a dev gate. The 2026-06-20 incident was a full host OOM
|
|
115
|
+
×2; the failure mode the cap prevents is catastrophic and physical. Reading the cap with `?? default`
|
|
116
|
+
(default 8) means every agent — new and updated — gets the protection without opting in. The default
|
|
117
|
+
of 8 is conservative (well below the ~230+ that OOM'd a 128 GB box) yet ample for normal CoherenceGate
|
|
118
|
+
fan-out; it is fully tunable for a larger host. Migration is existence-checked so an operator's
|
|
119
|
+
explicit override is never clobbered.
|
|
120
|
+
|
|
121
|
+
## 6. Residual (tracked, NOT fixed here — out of scope)
|
|
122
|
+
|
|
123
|
+
- The elaborate predecessor's lanes / reserve-math / hysteresis / per-gate-typed-shed are
|
|
124
|
+
deliberately deferred — the simple design was chosen precisely because it is implementable
|
|
125
|
+
CORRECTLY in one pass. If real load data later shows the single global cap is too coarse, the lane
|
|
126
|
+
model is the documented follow-up.
|
|
127
|
+
- The launchd `NumberOfProcesses` belt is macOS-only; a Linux/systemd equivalent is a follow-up for
|
|
128
|
+
non-macOS hosts (the in-process semaphore is OS-agnostic and is the primary control regardless).
|
|
129
|
+
- No baseline JSON for the new lint — it ships clean (zero violations), so a baseline would be an
|
|
130
|
+
empty no-op (the sibling `lint-no-unfunneled-headless-launch.js` has none either).
|
|
131
|
+
|
|
132
|
+
## 7. Rollback
|
|
133
|
+
|
|
134
|
+
Revert the source files. The semaphore/lock/wrapper are additive; reverting the factory-funnel wrap +
|
|
135
|
+
the `server.ts` boot lines restores the prior unbounded-spawn behavior, and reverting the four gate
|
|
136
|
+
seams restores their prior fail-open. No data migration and no on-disk format to undo — the
|
|
137
|
+
host-spawn-holders / single-instance-lock files are host-local runtime state that is simply abandoned
|
|
138
|
+
on revert. The config defaults are additive and harmless if left in place.
|