instar 1.3.1067 → 1.3.1069
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/core/benchmarkDivergenceCore.d.ts +8 -4
- package/dist/core/benchmarkDivergenceCore.d.ts.map +1 -1
- package/dist/core/benchmarkDivergenceCore.js +21 -7
- package/dist/core/benchmarkDivergenceCore.js.map +1 -1
- package/dist/data/standards-registry.meta.json +1 -1
- package/dist/messaging/SpawnRequestManager.d.ts.map +1 -1
- package/dist/messaging/SpawnRequestManager.js +43 -2
- package/dist/messaging/SpawnRequestManager.js.map +1 -1
- package/dist/monitoring/BenchmarkDivergenceAnalyzer.d.ts.map +1 -1
- package/dist/monitoring/BenchmarkDivergenceAnalyzer.js +5 -1
- package/dist/monitoring/BenchmarkDivergenceAnalyzer.js.map +1 -1
- package/dist/monitoring/ClaimClauseArbiter.d.ts +49 -1
- package/dist/monitoring/ClaimClauseArbiter.d.ts.map +1 -1
- package/dist/monitoring/ClaimClauseArbiter.js +52 -6
- package/dist/monitoring/ClaimClauseArbiter.js.map +1 -1
- package/dist/monitoring/CompletionClaimVerifier.d.ts +28 -0
- package/dist/monitoring/CompletionClaimVerifier.d.ts.map +1 -1
- package/dist/monitoring/CompletionClaimVerifier.js +38 -2
- package/dist/monitoring/CompletionClaimVerifier.js.map +1 -1
- package/dist/monitoring/FeatureMetricsLedger.d.ts +3 -0
- package/dist/monitoring/FeatureMetricsLedger.d.ts.map +1 -1
- package/dist/monitoring/FeatureMetricsLedger.js +23 -3
- package/dist/monitoring/FeatureMetricsLedger.js.map +1 -1
- package/dist/server/AgentServer.d.ts.map +1 -1
- package/dist/server/AgentServer.js +5 -1
- package/dist/server/AgentServer.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +3 -3
- package/src/data/standards-registry.meta.json +1 -1
- package/upgrades/1.3.1068.md +82 -0
- package/upgrades/1.3.1069.md +91 -0
- package/upgrades/side-effects/benchmark-orphan-rate-unknown.md +85 -0
- package/upgrades/side-effects/claim-arbiter-general-gating.md +206 -0
- package/upgrades/side-effects/spawn-denial-preserves-payload.md +421 -0
|
@@ -0,0 +1,421 @@
|
|
|
1
|
+
# Side-Effects Review — a transient-pressure spawn refusal must not destroy the payload it refused
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `spawn-denial-preserves-payload`
|
|
4
|
+
**Date:** `2026-07-29`
|
|
5
|
+
**Author:** `echo`
|
|
6
|
+
**Second-pass reviewer:** `required (session-lifecycle / spawn surface)`
|
|
7
|
+
|
|
8
|
+
## Summary of the change
|
|
9
|
+
|
|
10
|
+
`SpawnRequestManager.evaluate()` is the admission funnel every inbound agent-to-agent dispatch passes
|
|
11
|
+
through before a session is spawned to handle it. It has six exits that do not spawn. Three of them
|
|
12
|
+
destroyed the inbound payload; three did not. This change makes the whole set consistent, so a refusal
|
|
13
|
+
never means data loss.
|
|
14
|
+
|
|
15
|
+
Before → after, all six exits:
|
|
16
|
+
|
|
17
|
+
| Exit | Transient? | Before | After |
|
|
18
|
+
|---|---|---|---|
|
|
19
|
+
| `envelope-too-large` | no — permanent rejection | drops | **drops (correct, unchanged)** |
|
|
20
|
+
| cooldown | yes | queues | queues |
|
|
21
|
+
| session limit | yes (`retryAfterMs: 60_000`) | **drops** | **queues** |
|
|
22
|
+
| memory pressure | yes (`retryAfterMs: 120_000`) | **drops** | **queues** |
|
|
23
|
+
| subscription quota | yes | queues | queues |
|
|
24
|
+
| spawn threw | yes (`retryAfterMs: 30_000`) | drains queue, then destroys all of it | **UNCHANGED — restoring is unsafe, see §1a. Tracked: CMT-1114** |
|
|
25
|
+
|
|
26
|
+
Files touched:
|
|
27
|
+
|
|
28
|
+
- `src/messaging/SpawnRequestManager.ts` — the two missing
|
|
29
|
+
`if (request.context) { #queueMessage(...) }` guards, plus one line making a GLOBAL-cap refusal set
|
|
30
|
+
the truncation marker (§1a finding 2). **Every denial verdict — predicate, `reason` string, and
|
|
31
|
+
`retryAfterMs` — is byte-for-byte unchanged.**
|
|
32
|
+
- `tests/unit/spawn-request-manager.test.ts` — 5 tests added (3 positive, 2 pure controls) plus one
|
|
33
|
+
renamed exactly-once control.
|
|
34
|
+
|
|
35
|
+
**Scope was CUT after second-pass review.** An earlier revision of this change also restored the drained
|
|
36
|
+
queue when `spawnSession` threw. Review proved that unsafe and it was removed — §1a below.
|
|
37
|
+
|
|
38
|
+
Live evidence (this machine, 2026-07-29):
|
|
39
|
+
|
|
40
|
+
- Codey's `logs/server.log` carries **40** `Spawn denied: Memory pressure too high for new session`
|
|
41
|
+
entries in one day.
|
|
42
|
+
- Three of my own dispatches to him were destroyed this way, at `19:45:23.541Z`, `23:18:33.406Z`,
|
|
43
|
+
`23:20:35.503Z`. Each logged `async handleInboundMessage complete: {"handled":true,...,"error":"Spawn
|
|
44
|
+
denied: Memory pressure too high for new session"}` and each returned
|
|
45
|
+
`{"delivered":true,"outcome":"accepted"}` to me.
|
|
46
|
+
- Consequence: Codey merged 25 PRs up to `15:01Z` and then produced nothing for eight hours. He was not
|
|
47
|
+
idle — he was **unreachable, and the transport reported success to both ends.**
|
|
48
|
+
- His `MemoryPressureMonitor` oscillates 70–78% against the default `elevated: 75` threshold, flipping
|
|
49
|
+
every 30–80 seconds, so roughly half of all inbound A2A dispatches were coin-flipped into deletion.
|
|
50
|
+
|
|
51
|
+
I explicitly considered and **rejected** raising the pressure threshold, which would have been the
|
|
52
|
+
cheap answer. The pressure is genuine, not miscalibrated: swap 16.4 GB used of 17.4 GB (94%), total RSS
|
|
53
|
+
7.9 GB plus 7.3 GB held by the compressor on a 16 GB machine, and only ~67 MB safely reclaimable. The
|
|
54
|
+
guard is right to fire. The defect is that firing destroyed data.
|
|
55
|
+
|
|
56
|
+
**Scope grew, then was cut back.** I opened this to fix memory pressure. Auditing the function's other
|
|
57
|
+
exits for this artifact found the same defect twice more, so I fixed all three — the third being the worst,
|
|
58
|
+
since `#drainQueue` empties the queue *before* the spawn and a throw destroyed everything it had taken,
|
|
59
|
+
including payloads the cooldown and quota branches had just rescued. Second-pass review then proved the
|
|
60
|
+
third fix UNSAFE (§1a) and it was removed. The remaining two are the ones with the live evidence behind
|
|
61
|
+
them: every one of the 40 observed losses was a memory-pressure denial, which happens *before* any spawn
|
|
62
|
+
is attempted, so nothing could have been delivered.
|
|
63
|
+
|
|
64
|
+
## Decision-point inventory
|
|
65
|
+
|
|
66
|
+
- `SpawnRequestManager.evaluate()` — session-limit gate — **pass-through** — verdict identical in every
|
|
67
|
+
input case; payload retained on the already-denied path.
|
|
68
|
+
- `SpawnRequestManager.evaluate()` — memory-pressure gate — **pass-through** — same.
|
|
69
|
+
- `SpawnRequestManager.evaluate()` — spawn-failure catch — **untouched** (an earlier revision changed it;
|
|
70
|
+
reverted after review, see §1a).
|
|
71
|
+
- `#queueMessage()` global-cap refusal — **modified, non-decision** — still refuses exactly the same
|
|
72
|
+
payloads; now sets the existing `#truncated` marker so the refusal leaves a trace.
|
|
73
|
+
|
|
74
|
+
No decision point is added, modified, or removed. No threshold, predicate, or priority rule is touched.
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## 1. Over-block
|
|
79
|
+
|
|
80
|
+
**No block/allow surface change — over-block not applicable.** Nothing new can be refused. Every
|
|
81
|
+
predicate and reason string is unchanged, and pinned tests assert the unchanged verdict (`reason`
|
|
82
|
+
contains, `retryAfterMs` exact) *alongside* each new queue assertion, specifically so a later edit
|
|
83
|
+
cannot quietly turn one of these into a threshold change.
|
|
84
|
+
|
|
85
|
+
The nearest real concern is queue *capacity*, in §2 and §5.
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
## 1a. What second-pass review changed (the scope cut)
|
|
90
|
+
|
|
91
|
+
Review ran BEFORE this shipped and produced one blocking finding and one accuracy finding. Both are
|
|
92
|
+
recorded here rather than quietly absorbed, because the first one means an earlier revision of this
|
|
93
|
+
change would have made things worse.
|
|
94
|
+
|
|
95
|
+
**Finding 1 — BLOCKING. Restoring the drained queue on a failed spawn can DUPLICATE delivery.**
|
|
96
|
+
The removed code assumed a `spawnSession` rejection proves nothing was delivered. I verified that
|
|
97
|
+
assumption against the real implementation and it is false. `SessionManager.spawnSession`:
|
|
98
|
+
|
|
99
|
+
1. calls `execFileSync(tmux, ['new-session', '-d', …headlessSpec.argv])` — this creates a LIVE process
|
|
100
|
+
already running the framework CLI **with the full prompt**, which is the moment of delivery;
|
|
101
|
+
2. that call sits in a `try/catch` that rethrows only tmux-creation failures;
|
|
102
|
+
3. **after** that block, it builds the `Session` object and calls `this.state.saveSession(session)`
|
|
103
|
+
— outside any `try/catch` — and only then returns.
|
|
104
|
+
|
|
105
|
+
So a throw from `saveSession` (disk I/O, a `guardWrite` refusal, a serialization edge) means the payload
|
|
106
|
+
was **already delivered** while `spawnSession` reports failure. The restore would have put that payload
|
|
107
|
+
back, and the drain loop — 5 seconds later — would have delivered the same instruction to a second live
|
|
108
|
+
session. Two real agent processes acting on one instruction.
|
|
109
|
+
|
|
110
|
+
That is strictly worse than the bug being fixed: a lost message is recoverable by re-sending, a duplicated
|
|
111
|
+
instruction can act twice. And the failure modes are *correlated* — `saveSession` I/O failures are more
|
|
112
|
+
likely under exactly the memory pressure this change targets, not less.
|
|
113
|
+
|
|
114
|
+
I considered three ways to keep the restore safely and rejected all three:
|
|
115
|
+
- **Gate on `#classifyFailure`'s cause.** The real `spawnSession` never throws `SpawnFailureError`, so
|
|
116
|
+
every production failure classifies as `ambiguous` — the restore would either never fire (useless) or
|
|
117
|
+
fire on the unsafe case (the bug).
|
|
118
|
+
- **String-match the tmux-creation error message.** Brittle, and prohibited by the project's own rule
|
|
119
|
+
against matching on text where structure is required.
|
|
120
|
+
- **Ask whether the session exists before restoring.** `SpawnRequestManager` has no such accessor;
|
|
121
|
+
`getActiveSessions()` reads persisted state, which is precisely what failed to write.
|
|
122
|
+
|
|
123
|
+
So the restore was **removed**, not weakened. The prerequisite is upstream: `spawnSession` must stop
|
|
124
|
+
reporting failure once the session is live and holding the prompt — a bookkeeping failure after delivery
|
|
125
|
+
should be reported without rejecting. That is the same defect class this whole change is about, wearing
|
|
126
|
+
the opposite mask: **an operation that succeeded and reported failure.** Tracked: CMT-1114.
|
|
127
|
+
|
|
128
|
+
What ships is the part with the live evidence: all 40 observed losses were memory-pressure denials, which
|
|
129
|
+
occur *before* any spawn is attempted, so non-delivery there is structural rather than assumed.
|
|
130
|
+
|
|
131
|
+
**Finding 2 — accuracy. My own §2.1 claim was false for the global cap.**
|
|
132
|
+
I had written that queue loss now "goes through an accounted, marked, capped path instead of an unrecorded
|
|
133
|
+
`return`." True for the per-agent cap. False for the global cap: `#queueMessage` returns `false` as its
|
|
134
|
+
first statement, before any marker logic, and **every callsite discards the boolean**. So a global-cap
|
|
135
|
+
refusal was the one queue loss with no trace whatsoever. I fixed the code to make the sentence true
|
|
136
|
+
(the global-cap path now sets `#truncated`) rather than softening the sentence to match the code.
|
|
137
|
+
|
|
138
|
+
**Findings the review cleared:** TTL preservation was correct; no ordering bug from moving the drain; and
|
|
139
|
+
the three specific claims it checked (verdicts byte-for-byte unchanged, the drain request carries no
|
|
140
|
+
`context`, a successful spawn does not restore) were all accurate as stated.
|
|
141
|
+
|
|
142
|
+
**Reviewer's non-blocking note, now moot:** it observed that restore-at-the-front combined with
|
|
143
|
+
front-first truncation could re-discard the very entries being rescued. Removing the restore removes that
|
|
144
|
+
question entirely.
|
|
145
|
+
|
|
146
|
+
---
|
|
147
|
+
|
|
148
|
+
## 2. Under-block
|
|
149
|
+
|
|
150
|
+
Framed as under-*fix*, since there is no block surface:
|
|
151
|
+
|
|
152
|
+
1. **Sustained pressure can still lose content, but no longer silently — and this claim is now true for
|
|
153
|
+
BOTH caps.** `#queueMessage` enforces a per-agent cap (drops oldest, sets `#truncated`) and a global
|
|
154
|
+
cap. Review caught that my first draft of this sentence was FALSE for the global cap: it returns
|
|
155
|
+
`false` as its very first statement, before any marker logic, and **every callsite discards that
|
|
156
|
+
boolean** (`if (request.context) { #queueMessage(...) }`), so a global-cap refusal was the one queue
|
|
157
|
+
loss with no trace at all — exactly the silent-loss shape this change exists to remove. Rather than
|
|
158
|
+
soften the sentence I made the code match it: the global-cap path now sets `#truncated` too. Content
|
|
159
|
+
can still be lost under sustained pressure; it can no longer be lost invisibly.
|
|
160
|
+
2. **The 10-minute TTL still applies.** `QUEUE_MAX_AGE_MS` is 10 minutes, and `#drainQueue` filters
|
|
161
|
+
expired entries. A machine wedged above the threshold for longer than that will still drop. On the
|
|
162
|
+
machine where the loss was observed, windows open every 30–80 seconds against a 5-second drain tick,
|
|
163
|
+
so this is not the operating case.
|
|
164
|
+
3. **The sender is still told `delivered: true`.** `ThreadlineRouter`'s `handled: true` on denial is
|
|
165
|
+
deliberately untouched here. After this change that report is *approximately* true — the payload is
|
|
166
|
+
accepted for imminent retry rather than deleted — but a cap-drop or TTL expiry remains invisible to
|
|
167
|
+
the remote peer. Fixing it means changing the router's return contract and its callers, a wider blast
|
|
168
|
+
radius than this change should carry. <!-- tracked: CMT-1111 -->
|
|
169
|
+
4. **The queue does not survive a restart.** It is in-process memory, so a server restart between
|
|
170
|
+
enqueue and drain loses queued entries. This is the pre-existing behavior of the two branches that
|
|
171
|
+
already queued; this change matches them rather than diverging. <!-- tracked: CMT-1112 -->
|
|
172
|
+
5. **`envelope-too-large` still drops, and should.** It is not transient — the payload is over a hard
|
|
173
|
+
byte cap, so queueing it would store the oversized content and re-refuse it every drain tick. Its
|
|
174
|
+
sender deserves an honest rejection, which is a different fix (see §2.3), not a queue slot.
|
|
175
|
+
|
|
176
|
+
---
|
|
177
|
+
|
|
178
|
+
## 3. Level-of-abstraction fit
|
|
179
|
+
|
|
180
|
+
Right layer, and deliberately the smallest one that closes the class.
|
|
181
|
+
|
|
182
|
+
The loss happened inside `evaluate()`, so `evaluate()` is where it is fixed. Two alternatives were
|
|
183
|
+
considered and rejected:
|
|
184
|
+
|
|
185
|
+
- **Fix it in `ThreadlineRouter`** (have the router queue on denial): wrong layer. The router is one of
|
|
186
|
+
several `evaluate()` callers, so the loss would remain for all the others. Fixing the property at the
|
|
187
|
+
admission funnel covers every caller at once — which is the same argument the constitution's
|
|
188
|
+
fix-the-property-not-the-instance rule makes.
|
|
189
|
+
- **Add a retry driver**: unnecessary, and would have been a duplicate. One already exists and is
|
|
190
|
+
already running: `runTick()` plus the `onDrainReady` callback wired in `src/commands/server.ts`
|
|
191
|
+
re-attempt a spawn for any agent holding queued messages, on a **5-second** tick, enabled by default
|
|
192
|
+
(`spawnManager.start()` unless `threadline.spawn.drainEnabled === false`). Codey's log proves it live
|
|
193
|
+
— `09:32:28.013Z` and `09:32:33.021Z` both read `[spawn-manager] drain re-attempt for echo not
|
|
194
|
+
approved: Memory pressure too high for new session`, five seconds apart, for this exact agent.
|
|
195
|
+
|
|
196
|
+
So this change builds no new machinery. It connects a dropped payload to the retry driver that was
|
|
197
|
+
already there and already trying, using the `#queueMessage` primitive that was already being called
|
|
198
|
+
twice in the same function.
|
|
199
|
+
|
|
200
|
+
---
|
|
201
|
+
|
|
202
|
+
## 4. Signal vs authority compliance
|
|
203
|
+
|
|
204
|
+
**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
|
|
205
|
+
|
|
206
|
+
**Does this change hold blocking authority with brittle logic?**
|
|
207
|
+
|
|
208
|
+
- [x] No — this change has no block/allow surface.
|
|
209
|
+
|
|
210
|
+
Each gate remains exactly the authority it already was, with the same threshold, predicate, and verdict.
|
|
211
|
+
No detector is added and no blocking logic is introduced. The change makes existing authorities'
|
|
212
|
+
denials **non-destructive**, which strengthens the principle rather than bending it: today a guard's
|
|
213
|
+
action is externally indistinguishable from a message-eating bug, so an operator cannot tell a working
|
|
214
|
+
guard from a broken one by observing it. After this change, a denial means "not now" instead of
|
|
215
|
+
"deleted" — the guard becomes legible.
|
|
216
|
+
|
|
217
|
+
---
|
|
218
|
+
|
|
219
|
+
## 4b. Judgment-point check (Judgment Within Floors standard)
|
|
220
|
+
|
|
221
|
+
**No new static heuristic at a competing-signals decision point.** No heuristic is added at all. The
|
|
222
|
+
two conditionals introduced are `if (request.context)` — a structural presence check on a field, copied
|
|
223
|
+
verbatim from the two sibling branches precisely so the four cases cannot drift apart — and the
|
|
224
|
+
global-cap marker, which fires on an already-existing comparison rather than introducing a threshold. No
|
|
225
|
+
signal is weighed against another.
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
## 5. Interactions
|
|
230
|
+
|
|
231
|
+
- **Shadowing:** branch order in `evaluate()` is unchanged. Each new `#queueMessage` call runs
|
|
232
|
+
immediately *before* an existing `return`, so nothing that previously ran is now skipped.
|
|
233
|
+
- **Double-fire:** no, and this was the sharpest thing to verify. Three independent reasons, in order of
|
|
234
|
+
how load-bearing they are:
|
|
235
|
+
1. **`#drainQueue` is fully synchronous and deletes the map key before returning** (`this.#pendingMessages.delete(agent)`),
|
|
236
|
+
and there is no `await` between the drain and that delete. So two concurrent `evaluate()` calls for
|
|
237
|
+
the same agent — say an inline inbound and the drain loop's own re-attempt interleaving — cannot both
|
|
238
|
+
receive the same entries: the first takes them all, the second sees an empty queue. This is the
|
|
239
|
+
property that makes the restore safe to add at all.
|
|
240
|
+
2. **The cooldown stamp serializes the common case.** `#lastSpawnByAgent.set` happens *before* the spawn
|
|
241
|
+
and is never rolled back, so a second request for the same agent while one is in flight hits the
|
|
242
|
+
cooldown branch and queues rather than racing.
|
|
243
|
+
3. **Restore only runs when the spawn threw**, meaning nothing in that prompt reached a session, so
|
|
244
|
+
putting the payloads back cannot re-deliver work that already arrived.
|
|
245
|
+
`#drainQueue(agent)` is also called on the spawn path only, after every denial branch — so a denial can
|
|
246
|
+
never drain the queue it just wrote to.
|
|
247
|
+
The drain loop's re-attempt is a synthetic request carrying **no** `context` (verified by reading
|
|
248
|
+
`onDrainReady` in `src/commands/server.ts`), so a re-attempt that is itself refused cannot enqueue a
|
|
249
|
+
duplicate of the payload it is trying to deliver — the two no-context control tests pin exactly this.
|
|
250
|
+
And the restore path only runs when `spawnSession` **threw**, meaning nothing in that prompt reached a
|
|
251
|
+
session, so restoring cannot cause a double delivery. A dedicated control test asserts a *successful*
|
|
252
|
+
spawn drains and does **not** restore.
|
|
253
|
+
- **Races:** `#pendingMessages` is a plain `Map` mutated from the single-threaded event loop; `runTick`
|
|
254
|
+
guards re-entry with `#tickInflight`. No new shared state and no new mutation ordering: the queueing
|
|
255
|
+
calls happen on paths that already returned without touching anything else.
|
|
256
|
+
- **Feedback loops:** none. Queueing cannot affect the pressure reading, the pressure monitor, the
|
|
257
|
+
session count, or the cooldown/penalty state, so no new call can influence its own gate.
|
|
258
|
+
- **Failure-attribution and escalation:** `#applyFailureAttribution` still runs first in the catch, and
|
|
259
|
+
`handleDenial`'s escalation is untouched. The restore is additive to the existing failure path.
|
|
260
|
+
- **Global cap: bound preserved, and its refusal is no longer silent.** With the restore removed, nothing
|
|
261
|
+
in this change can exceed `DEFAULT_MAX_GLOBAL_QUEUED` (1000) — `#queueMessage` is the only enqueue path
|
|
262
|
+
and it checks the global total first. The change here is that the refusal now sets `#truncated` (see
|
|
263
|
+
§2.1). Per-agent growth stays bounded by `MAX_QUEUED_PER_AGENT` (10), or
|
|
264
|
+
`DEGRADED_MAX_QUEUED_PER_AGENT_DEFAULT` (1) once `isInfraDegraded()` trips after
|
|
265
|
+
`INFRA_FAILURE_THRESHOLD` (5) infra failures in 10 minutes — so a peer that repeatedly fails spawns has
|
|
266
|
+
its queue footprint reduced, not expanded.
|
|
267
|
+
- **One consequence of the marker worth naming:** `isTruncated(agent)` will now report true in a case it
|
|
268
|
+
did not before (global-cap refusal). Its only consumer treats it as an advisory "this agent's queue lost
|
|
269
|
+
something" signal, and `#drainQueue` still clears it on the next successful drain, so the new true is
|
|
270
|
+
more accurate rather than newly load-bearing.
|
|
271
|
+
|
|
272
|
+
---
|
|
273
|
+
|
|
274
|
+
## 6. External surfaces
|
|
275
|
+
|
|
276
|
+
- **Other agents on the same machine:** yes, positively and by design — this is the transport that
|
|
277
|
+
carries work between agents. No interface change: `SpawnResult` fields are identical.
|
|
278
|
+
- **Install base:** every agent receives it on update. The behavioral delta is strictly "a payload that
|
|
279
|
+
was previously destroyed is now retried," which is why it ships without a flag — there is no
|
|
280
|
+
configuration under which silently deleting an accepted message is the desired behavior.
|
|
281
|
+
- **External systems:** none. No Telegram, Slack, GitHub, or network surface.
|
|
282
|
+
- **Persistent state:** none. The queue is in-process memory; nothing is written to disk.
|
|
283
|
+
- **Timing / runtime conditions:** delivery timing depends on when pressure clears and on the 5-second
|
|
284
|
+
drain tick — outside our control, honestly bounded by the 10-minute TTL (§2.2). No *new* timing
|
|
285
|
+
dependency: the change joins a path that was already scheduled.
|
|
286
|
+
- **Operator surface (Mobile-Complete Operator Actions):** **no operator-facing action** is added or
|
|
287
|
+
touched. Queue depth is already observable via `getStatus().queuedMessages` and `getQueuedCount()`,
|
|
288
|
+
and the existing `threadline.spawn.drainEnabled` kill switch is unchanged.
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## 6b. Operator-surface quality
|
|
293
|
+
|
|
294
|
+
**No operator surface — not applicable.** No dashboard renderer, approval page, or grant/secret-drop
|
|
295
|
+
form is staged in this change.
|
|
296
|
+
|
|
297
|
+
---
|
|
298
|
+
|
|
299
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
300
|
+
|
|
301
|
+
**Machine-local BY DESIGN**, with the reason: this queue holds transient inbound payloads for sessions
|
|
302
|
+
that will run **in this machine's own process**, gated by **this machine's own** memory reading and
|
|
303
|
+
session count. A payload queued on machine A must be delivered by A, because the session that consumes
|
|
304
|
+
it is A's. Replicating it would be actively wrong — it would let machine B spawn a session for a message
|
|
305
|
+
A accepted, which is exactly the duplicate-session class the pool's ownership rules exist to prevent.
|
|
306
|
+
|
|
307
|
+
- **User-facing notices:** none — this path emits no user-facing message, so no one-voice gating is
|
|
308
|
+
needed. (The absence of *any* notice on loss is the honest gap recorded in §2.3, tracked as CMT-1111.)
|
|
309
|
+
- **Durable state / topic transfer:** none, so nothing can strand on a transfer. A queued payload dies
|
|
310
|
+
with the process — pre-existing behavior, tracked as CMT-1112.
|
|
311
|
+
- **Generated URLs:** none.
|
|
312
|
+
|
|
313
|
+
The pool-wide question ("did a peer's dispatch reach me?") is already answered by a separate
|
|
314
|
+
proxied-on-read surface: `GET /threadline/peers/health`, whose `pendingCount` / `stale` fields are how I
|
|
315
|
+
first noticed 126 unacked messages to this peer.
|
|
316
|
+
|
|
317
|
+
---
|
|
318
|
+
|
|
319
|
+
## 8. Rollback cost
|
|
320
|
+
|
|
321
|
+
- **Hot-fix release:** revert this commit and ship as the next patch. Behavior returns exactly to
|
|
322
|
+
dropping payloads on those three exits.
|
|
323
|
+
- **Data migration:** none. No schema, file, ledger, or config key is introduced.
|
|
324
|
+
- **Agent state repair:** none. Nothing to notify or reset; queued entries are in-memory and expire on
|
|
325
|
+
their own within 10 minutes.
|
|
326
|
+
- **User visibility:** none during the rollback window. Rolling back restores the previous silent-loss
|
|
327
|
+
behavior; it cannot corrupt state or leave a half-applied condition.
|
|
328
|
+
- **Pre-existing kill switch:** an operator who wants the drain machinery off entirely already has
|
|
329
|
+
`threadline.spawn.drainEnabled: false`, unchanged by this work.
|
|
330
|
+
|
|
331
|
+
Close to the cheapest possible rollback: one revert, no state.
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
## Conclusion
|
|
336
|
+
|
|
337
|
+
The review did real work three times, and the most valuable pass was the one that took work away.
|
|
338
|
+
|
|
339
|
+
First it grew the scope correctly: auditing the function's other exits for the decision-point inventory
|
|
340
|
+
found the same defect in two more branches.
|
|
341
|
+
|
|
342
|
+
Then it corrected a claim I would otherwise have shipped. I had planned to describe the fix as "the
|
|
343
|
+
payload survives until the peer sends again," because I assumed no retry driver existed. Reading for §3
|
|
344
|
+
found `runTick` / `onDrainReady` already wired and started by default, and Codey's own log showed it
|
|
345
|
+
re-attempting for this exact agent every five seconds.
|
|
346
|
+
|
|
347
|
+
Then second-pass review cut the scope back, and this is the part that matters most. My third fix would
|
|
348
|
+
have turned a rare lost message into a rare duplicated instruction, because I asserted "the spawn threw
|
|
349
|
+
so nothing was delivered" from the shape of `evaluate()` without reading what the injected `spawnSession`
|
|
350
|
+
actually does. It creates the live session with the prompt and *then* does work that can throw. I could
|
|
351
|
+
not have found that by re-reading my own diff — I had read it several times and thought it was sound —
|
|
352
|
+
which is the whole argument for the gate existing.
|
|
353
|
+
|
|
354
|
+
What ships is smaller than what I built and better evidenced: the two branches that refuse *before* any
|
|
355
|
+
spawn is attempted, where non-delivery is structural rather than assumed, plus one line making the last
|
|
356
|
+
silent queue-loss path leave a trace. Four honest limits remain, each with a real tracked commitment
|
|
357
|
+
(CMT-1111, CMT-1112, CMT-1113, CMT-1114) rather than a good intention.
|
|
358
|
+
|
|
359
|
+
Clear to ship.
|
|
360
|
+
|
|
361
|
+
---
|
|
362
|
+
|
|
363
|
+
## Second-pass review
|
|
364
|
+
|
|
365
|
+
**Reviewer:** independent reviewer (fresh context, no prior involvement in the change)
|
|
366
|
+
**Independent read of the artifact: CONCERN RAISED — resolved by cutting scope.**
|
|
367
|
+
|
|
368
|
+
- **Duplicate delivery via partial-success-then-throw** — CONFIRMED by my own reading of
|
|
369
|
+
`SessionManager.spawnSession` (tmux `new-session` delivers the prompt; `state.saveSession()` runs
|
|
370
|
+
afterwards outside any `try/catch`). The offending code was **removed**, not softened. §1a finding 1;
|
|
371
|
+
tracked CMT-1114.
|
|
372
|
+
- **Global-cap refusal is silent, so §2.1's "accounted, marked, capped" was inaccurate** — CONFIRMED
|
|
373
|
+
(`return false` precedes all marker logic; all callsites discard the boolean). **Code changed so the
|
|
374
|
+
claim is true**, rather than the claim changed to match the code. §1a finding 2.
|
|
375
|
+
- **Restored-entries-truncated-first** — non-blocking, now moot with the restore removed.
|
|
376
|
+
- Categories the reviewer cleared: TTL preservation, drain-move ordering, and three specific artifact
|
|
377
|
+
claims it checked independently.
|
|
378
|
+
|
|
379
|
+
I verified both findings against the source myself before accepting them rather than taking the review on
|
|
380
|
+
trust; the blocking one reproduces in the code as described.
|
|
381
|
+
|
|
382
|
+
## Evidence pointers
|
|
383
|
+
|
|
384
|
+
- **Live repro, three destroyed dispatches:** `instar-codey/logs/server.log` at `19:45:23.541Z`,
|
|
385
|
+
`23:18:33.406Z`, `23:20:35.503Z` — each `handled:true` carrying
|
|
386
|
+
`error: "Spawn denied: Memory pressure too high for new session"`, each returning
|
|
387
|
+
`{"delivered":true,"outcome":"accepted"}` to the sender.
|
|
388
|
+
- **Denial volume:** 40 matches for `Spawn denied: Memory pressure` in that log for 2026-07-29.
|
|
389
|
+
- **Retry driver proven live:** same log, `09:32:28.013Z` and `09:32:33.021Z` —
|
|
390
|
+
`[spawn-manager] drain re-attempt for echo not approved: …`, five seconds apart. Tick confirmed
|
|
391
|
+
`tick=5000ms` at `[spawn-manager] drain loop started`.
|
|
392
|
+
- **Pressure genuinely high (threshold not miscalibrated):** `vm.swapusage` 16385 MB used of 17408 MB
|
|
393
|
+
(94%); `Pages occupied by compressor` 467843 × 16 KB ≈ 7.3 GB; total RSS across 427 processes 7.9 GB
|
|
394
|
+
on 16 GB of RAM; ~67 MB safely reclaimable.
|
|
395
|
+
- **Negative control run BEFORE trusting the tests, and re-run after the scope cut:** with
|
|
396
|
+
`src/messaging/SpawnRequestManager.ts` stashed to original, **5 of the new tests FAIL (5 failed / 78
|
|
397
|
+
passed)**; the two pure no-context controls PASS on both old and new code, proving they discriminate
|
|
398
|
+
rather than merely agreeing with whatever is in front of them. With the fix restored: **83/83 pass**.
|
|
399
|
+
- **Wider runs, all green:** `spawn-request-manager` + `subscription-quota-gates` +
|
|
400
|
+
`threadline/ThreadlineRouter` + `SpawnAdmission`; `threadline-fixes` + `NovelFailureReviewer`.
|
|
401
|
+
- `npx tsc --noEmit` exits 0; `npm run lint` (33-lint chain) passes.
|
|
402
|
+
- **The running copy provably has the defect:** both agents' installed
|
|
403
|
+
`dist/messaging/SpawnRequestManager.js` shows the memory-pressure branch returning with no enqueue —
|
|
404
|
+
so this is a live defect in deployed code, not a source-only observation.
|
|
405
|
+
|
|
406
|
+
---
|
|
407
|
+
|
|
408
|
+
## Class-Closure Declaration (display-only mirror)
|
|
409
|
+
|
|
410
|
+
**No agent-authored-artifact defect — not applicable.** This is a defect in hand-written TypeScript, not
|
|
411
|
+
in an LLM prompt, hook, config, skill, or standards text. It also adds no self-triggered controller: the
|
|
412
|
+
retry driver it feeds (`runTick` / `onDrainReady`) already exists, is already bounded by DRR quantum,
|
|
413
|
+
`maxDrainsPerTick`, per-agent and global queue caps, and a 10-minute entry TTL, and is not modified here.
|
|
414
|
+
No new firing edge is added: the two queueing calls sit on paths that already returned a denial, and
|
|
415
|
+
neither can extend an entry's lifetime.
|
|
416
|
+
|
|
417
|
+
For the record, the defect's *shape* is the one this project has spent the week cataloguing — an
|
|
418
|
+
operation that fails and reports success. The guard that would generalize it does not exist: a rule that
|
|
419
|
+
a transient-pressure refusal must never destroy the payload it refused. Three of six branches in one
|
|
420
|
+
function had it wrong, and nothing in the repo would have caught any of them. That standards gap is
|
|
421
|
+
tracked, not closed here. <!-- tracked: CMT-1113 -->
|