instar 1.3.1067 → 1.3.1069
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/core/benchmarkDivergenceCore.d.ts +8 -4
- package/dist/core/benchmarkDivergenceCore.d.ts.map +1 -1
- package/dist/core/benchmarkDivergenceCore.js +21 -7
- package/dist/core/benchmarkDivergenceCore.js.map +1 -1
- package/dist/data/standards-registry.meta.json +1 -1
- package/dist/messaging/SpawnRequestManager.d.ts.map +1 -1
- package/dist/messaging/SpawnRequestManager.js +43 -2
- package/dist/messaging/SpawnRequestManager.js.map +1 -1
- package/dist/monitoring/BenchmarkDivergenceAnalyzer.d.ts.map +1 -1
- package/dist/monitoring/BenchmarkDivergenceAnalyzer.js +5 -1
- package/dist/monitoring/BenchmarkDivergenceAnalyzer.js.map +1 -1
- package/dist/monitoring/ClaimClauseArbiter.d.ts +49 -1
- package/dist/monitoring/ClaimClauseArbiter.d.ts.map +1 -1
- package/dist/monitoring/ClaimClauseArbiter.js +52 -6
- package/dist/monitoring/ClaimClauseArbiter.js.map +1 -1
- package/dist/monitoring/CompletionClaimVerifier.d.ts +28 -0
- package/dist/monitoring/CompletionClaimVerifier.d.ts.map +1 -1
- package/dist/monitoring/CompletionClaimVerifier.js +38 -2
- package/dist/monitoring/CompletionClaimVerifier.js.map +1 -1
- package/dist/monitoring/FeatureMetricsLedger.d.ts +3 -0
- package/dist/monitoring/FeatureMetricsLedger.d.ts.map +1 -1
- package/dist/monitoring/FeatureMetricsLedger.js +23 -3
- package/dist/monitoring/FeatureMetricsLedger.js.map +1 -1
- package/dist/server/AgentServer.d.ts.map +1 -1
- package/dist/server/AgentServer.js +5 -1
- package/dist/server/AgentServer.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +3 -3
- package/src/data/standards-registry.meta.json +1 -1
- package/upgrades/1.3.1068.md +82 -0
- package/upgrades/1.3.1069.md +91 -0
- package/upgrades/side-effects/benchmark-orphan-rate-unknown.md +85 -0
- package/upgrades/side-effects/claim-arbiter-general-gating.md +206 -0
- package/upgrades/side-effects/spawn-denial-preserves-payload.md +421 -0
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "./builtin-manifest.schema.json",
|
|
3
3
|
"schemaVersion": 1,
|
|
4
|
-
"generatedAt": "2026-07-
|
|
5
|
-
"instarVersion": "1.3.
|
|
4
|
+
"generatedAt": "2026-07-30T02:09:04.451Z",
|
|
5
|
+
"instarVersion": "1.3.1069",
|
|
6
6
|
"entryCount": 202,
|
|
7
7
|
"entries": {
|
|
8
8
|
"hook:session-start": {
|
|
@@ -1530,7 +1530,7 @@
|
|
|
1530
1530
|
"type": "subsystem",
|
|
1531
1531
|
"domain": "server",
|
|
1532
1532
|
"sourcePath": "src/server/AgentServer.ts",
|
|
1533
|
-
"contentHash": "
|
|
1533
|
+
"contentHash": "894e0598284292440c21230f87a3b9cf2a71b27ccc5f0349fb5aecb1c2f2edb0",
|
|
1534
1534
|
"since": "2025-01-01"
|
|
1535
1535
|
},
|
|
1536
1536
|
"subsystem:session-manager": {
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: patch -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
`SpawnRequestManager.evaluate()` — the admission funnel every inbound agent-to-agent dispatch passes
|
|
9
|
+
through before a session is spawned for it — had six exits that do not spawn. Three preserved the
|
|
10
|
+
inbound payload by queueing it; three destroyed it. All transient-pressure exits now preserve it:
|
|
11
|
+
|
|
12
|
+
- **Memory pressure** — now queues the payload (was: dropped it).
|
|
13
|
+
- **Session limit** — now queues the payload (was: dropped it).
|
|
14
|
+
- **Spawn threw** — deliberately UNCHANGED. This branch also destroys payloads, and an earlier revision
|
|
15
|
+
of this change restored them — but second-pass review proved that unsafe and it was removed.
|
|
16
|
+
`SessionManager.spawnSession` creates the live tmux session *with the prompt* and only afterwards calls
|
|
17
|
+
`state.saveSession()` outside any `try/catch`, so a rejection does NOT prove non-delivery. Restoring
|
|
18
|
+
would risk delivering the same instruction to a second live session — trading a rare lost message for a
|
|
19
|
+
rare duplicated one. The prerequisite is upstream: `spawnSession` must stop reporting failure once the
|
|
20
|
+
session is live and holding the prompt. <!-- tracked: CMT-1114 -->
|
|
21
|
+
- **`envelope-too-large`** — still drops, deliberately: it is a permanent size rejection, not transient,
|
|
22
|
+
so queueing it would re-refuse the same oversized payload on every drain tick.
|
|
23
|
+
|
|
24
|
+
Every denial verdict is byte-for-byte unchanged — same predicate, same `reason`, same `retryAfterMs`. No
|
|
25
|
+
threshold, priority rule, or gate was touched. Queued payloads join the drain loop
|
|
26
|
+
(`runTick` / `onDrainReady`) that was already running on a 5-second tick, so delivery lands at the next
|
|
27
|
+
window instead of never.
|
|
28
|
+
|
|
29
|
+
One further line: a GLOBAL-cap refusal in `#queueMessage` now sets the existing `#truncated` marker. It
|
|
30
|
+
returns `false` before any marker logic and every callsite discards that boolean, so this was the one
|
|
31
|
+
queue loss that left no trace at all — the same silent-loss shape being fixed here.
|
|
32
|
+
|
|
33
|
+
## What to Tell Your User
|
|
34
|
+
|
|
35
|
+
If two of your agents hand each other work, a message could previously be **deleted** when the receiving
|
|
36
|
+
machine was briefly short on memory or at its session cap — and both sides reported success, so neither
|
|
37
|
+
you nor the agents had any way to know. That is fixed: a refused message now waits and is retried
|
|
38
|
+
automatically, usually within a minute.
|
|
39
|
+
|
|
40
|
+
The symptom this explains: an agent that looks idle for hours despite being handed work, with no error
|
|
41
|
+
anywhere. It was not idle — it was unreachable behind a transport that reported delivery.
|
|
42
|
+
|
|
43
|
+
Two honest limits. If a machine stays under pressure for more than ten minutes, messages can still
|
|
44
|
+
expire; and if the agent's server restarts while a message is waiting, that message is lost. Neither is
|
|
45
|
+
left as a good intention — both are tracked. <!-- tracked: CMT-1111 --> <!-- tracked: CMT-1112 -->
|
|
46
|
+
|
|
47
|
+
## Summary of New Capabilities
|
|
48
|
+
|
|
49
|
+
No new capability, endpoint, config key, or flag — this is a defect fix in existing behavior. Rollback is
|
|
50
|
+
a single revert with no state to clean up. The pre-existing kill switch for the retry machinery
|
|
51
|
+
(`threadline.spawn.drainEnabled: false`) is unchanged.
|
|
52
|
+
|
|
53
|
+
## Evidence
|
|
54
|
+
|
|
55
|
+
Found live on a two-agent machine, 2026-07-29:
|
|
56
|
+
|
|
57
|
+
- **40** `Spawn denied: Memory pressure too high for new session` entries in one agent's server log in a
|
|
58
|
+
single day. Each one destroyed its payload.
|
|
59
|
+
- Three dispatches confirmed lost at `19:45:23.541Z`, `23:18:33.406Z`, `23:20:35.503Z` — each logged
|
|
60
|
+
`handled: true` alongside the spawn-denied error, while the sending agent's `threadline_send` returned
|
|
61
|
+
`{"delivered":true,"outcome":"accepted"}`. Both ends recorded success.
|
|
62
|
+
- Impact: the receiving agent merged 25 PRs up to `15:01Z` then produced nothing for 8 hours. Not idle —
|
|
63
|
+
unreachable, with the transport reporting success.
|
|
64
|
+
- Pressure was genuine, not a miscalibrated threshold (checked, because loosening it would have been the
|
|
65
|
+
cheap answer): swap 16361 MB used of 17408 MB (94%), compressor holding ~7.3 GB, total RSS 7.9 GB on
|
|
66
|
+
16 GB of RAM.
|
|
67
|
+
- The retry driver was already live and already trying: `[spawn-manager] drain re-attempt for echo not
|
|
68
|
+
approved: …` at `09:32:28.013Z` and `09:32:33.021Z`, 5 seconds apart, `tick=5000ms`.
|
|
69
|
+
|
|
70
|
+
Tests: 5 added to `tests/unit/spawn-request-manager.test.ts` (3 positive, 2 pure controls) plus one
|
|
71
|
+
renamed exactly-once control. **Every new test was run against the unfixed source first** — 5 failed / 78
|
|
72
|
+
passed on original code; the two no-context controls passed on both old and new, proving they
|
|
73
|
+
discriminate. With the fix: 83/83. Wider run across `spawn-request-manager`, `subscription-quota-gates`,
|
|
74
|
+
`threadline/ThreadlineRouter`, `SpawnAdmission` and `threadline-fixes`: 197 passed. `tsc --noEmit` exits
|
|
75
|
+
0 and the 33-lint chain passes.
|
|
76
|
+
|
|
77
|
+
The running copy provably carries the defect: both agents' installed
|
|
78
|
+
`dist/messaging/SpawnRequestManager.js` shows the memory-pressure branch returning with no enqueue.
|
|
79
|
+
|
|
80
|
+
Known limits, tracked rather than papered over: the sender is still told `delivered: true` (CMT-1111),
|
|
81
|
+
the queue does not survive a server restart (CMT-1112), no class-level guard exists yet (CMT-1113), and
|
|
82
|
+
the spawn-failure branch still loses payloads pending the upstream truthfulness fix (CMT-1114).
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: patch -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
`ClaimClauseArbiter` asked every call for a two-part envelope — `legacy` (the clause labelling the
|
|
9
|
+
system consumes) and `general` (an experimental extraction of up to 4 factual claims, ~20 fields
|
|
10
|
+
each, with byte offsets and 8–11-option enums).
|
|
11
|
+
|
|
12
|
+
But the call site admits `general` only on one framework:
|
|
13
|
+
|
|
14
|
+
```
|
|
15
|
+
const general = resolvedModel?.framework === 'claude-code' ? parsedGeneral : null;
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The prompt was never told about that rule. On an install whose internal components route off
|
|
19
|
+
Claude — which is now the shipped default (`provider-fallback-default-policy`) — the model was
|
|
20
|
+
asked for `general`, spent real latency and tokens producing it, and the result was discarded
|
|
21
|
+
100% of the time.
|
|
22
|
+
|
|
23
|
+
The knock-on effect is what makes it a fault rather than only waste: generating the discarded
|
|
24
|
+
half regularly pushed the call past its own `timeoutMs: 60_000`, which killed the `legacy` half
|
|
25
|
+
the system actually uses.
|
|
26
|
+
|
|
27
|
+
Now the framework is resolved **before** the prompt is built, via the router's existing
|
|
28
|
+
`for(component)` resolver, and `general` is requested only when that resolves to `claude-code`.
|
|
29
|
+
|
|
30
|
+
- **Claude-routed installs are byte-identical.** `includeGeneral` defaults to `true`; only a
|
|
31
|
+
positively-resolved, non-empty, non-`claude-code` string suppresses it.
|
|
32
|
+
- **Every uncertainty fails toward the old behaviour** — resolver absent, returning `undefined`,
|
|
33
|
+
returning a non-string, or throwing all send the full prompt. Suppressing a shipped (if dark)
|
|
34
|
+
extractor on a guess is the one outcome worth avoiding, so nothing guesses.
|
|
35
|
+
- **The two prompt shapes get distinct ids.** `CLAIM_ARBITER_PROMPT_ID_LEGACY_ONLY` keeps
|
|
36
|
+
per-prompt latency/quality attribution from blending a fast small prompt with a slow large one.
|
|
37
|
+
- **`CLAIM_ARBITER_COMPONENT` is shared** between the resolver and the call's attribution, so the
|
|
38
|
+
gate can never consult a different routing entry than the call bills to.
|
|
39
|
+
|
|
40
|
+
The admission rule itself is unchanged, byte for byte. This changes what is *asked for*, never
|
|
41
|
+
what is *accepted*.
|
|
42
|
+
|
|
43
|
+
## What to Tell Your User
|
|
44
|
+
|
|
45
|
+
If your agent's internal checks run on a non-Claude model — the shipped default — one background
|
|
46
|
+
check was quietly asking for a large answer it then threw away, and the cost of producing it was
|
|
47
|
+
making that check time out and fail about 83% of the time. It now asks only for the part it uses.
|
|
48
|
+
Nothing changes for agents routed to Claude.
|
|
49
|
+
|
|
50
|
+
## Summary of New Capabilities
|
|
51
|
+
|
|
52
|
+
None. This removes waste from an existing check; no new surface, route, config key, or capability.
|
|
53
|
+
|
|
54
|
+
## Evidence
|
|
55
|
+
|
|
56
|
+
Measured on a live two-agent machine, 2026-07-29, before writing the change.
|
|
57
|
+
|
|
58
|
+
**Production symptom** (`/metrics/features`, 24h, `completion-claim-verify`): 1,207 calls,
|
|
59
|
+
**956 errors (83.5%)**, 62 shed, 189 succeeded. `p50 49,315 ms`, `p95 60,082 ms`, `max 64,966 ms`
|
|
60
|
+
against a 60,000 ms wall. Input 2,430,258 tokens; output 462,638. 964 of the 1,207 calls ran on
|
|
61
|
+
`codex-cli` — the path where `general` is discarded.
|
|
62
|
+
|
|
63
|
+
**Controlled A/B**, interleaved A,B,A,B,A,B so a slow patch on the door could not be attributed
|
|
64
|
+
to one variant. Same model (`gpt-5.4-mini`), same door, same message, clauses and evidence.
|
|
65
|
+
|
|
66
|
+
| Variant | n | Median | Range | Avg output tokens | Inside the 60s wall |
|
|
67
|
+
|---|---|---|---|---|---|
|
|
68
|
+
| Full (`legacy`+`general`) | 3 | **129,204 ms** | 91,520 – 148,719 | 8,338 | **0 / 3** |
|
|
69
|
+
| Legacy-only | 3 | **28,153 ms** | 24,965 – 33,708 | 1,386 | **3 / 3** |
|
|
70
|
+
|
|
71
|
+
**4.6× median latency, 6.0× output tokens, and the ranges do not overlap.**
|
|
72
|
+
|
|
73
|
+
**Correctness, not only speed.** The legacy-only reply was confirmed valid JSON matching the
|
|
74
|
+
legacy schema and materially correct: it labelled *"Restart whenever suits you"* as `neither`
|
|
75
|
+
(*"imperative to the user, not a commitment or completion assertion"*) and flagged
|
|
76
|
+
*"Everything green"* as `corroborated: false` (*"no independent test evidence is provided here"*).
|
|
77
|
+
So this is a cost removal, not a capability trade.
|
|
78
|
+
|
|
79
|
+
**Test discrimination.** The new tests were run against the unfixed source with only the test
|
|
80
|
+
staged: **6 failed / 4 passed**. The 4 passes are labelled `CONTROL` and pass on both revisions by
|
|
81
|
+
design — two of them only revealed themselves as non-discriminating *because* that unfixed run was
|
|
82
|
+
performed. Only the 6 failures are counted as evidence.
|
|
83
|
+
|
|
84
|
+
## Honest limits
|
|
85
|
+
|
|
86
|
+
- n=3 per arm. The conclusion rests on the non-overlap and the size of the gap, not on sample count.
|
|
87
|
+
- Timings came from a direct `codex exec` invocation rather than through instar's provider
|
|
88
|
+
(different sandbox/env/out-dir), so **absolute** numbers may differ in production. The **ratio**
|
|
89
|
+
is the load-bearing result and it is controlled.
|
|
90
|
+
- The expected drop in that 83.5% error rate is a **prediction until the shipped number moves**,
|
|
91
|
+
and is deliberately not claimed as proven. <!-- tracked: CMT-1118 -->
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Side-Effects Review — Benchmark orphan-rate unknown verdict
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `benchmark-orphan-rate-unknown`
|
|
4
|
+
**Date:** `2026-07-29`
|
|
5
|
+
**Author:** `Instar-codey`
|
|
6
|
+
**Second-pass reviewer:** `not required`
|
|
7
|
+
|
|
8
|
+
## Summary of the change
|
|
9
|
+
|
|
10
|
+
The benchmark divergence detector now distinguishes a measured zero orphan rate from an unmeasurable orphan rate. `VerdictInput.orphanShare` becomes `number | null`; `null` returns `partial` with `partialReason: orphan-rate-unknown` before aligned or divergent verdicts can be emitted. The analyzer computes `null` when the pool-merged decided denominator is unavailable, and the finding ledger/API surface carries a nullable `partialReason` field. Touched runtime files: `src/core/benchmarkDivergenceCore.ts`, `src/monitoring/BenchmarkDivergenceAnalyzer.ts`, and `src/monitoring/FeatureMetricsLedger.ts`.
|
|
11
|
+
|
|
12
|
+
## Decision-point inventory
|
|
13
|
+
|
|
14
|
+
- `computeVerdict` benchmark divergence ladder — modified — Step 7 now treats unknown orphan rate as a non-actionable partial verdict.
|
|
15
|
+
- `BenchmarkDivergenceAnalyzer.runPass` orphan-share calculation — modified — empty decided denominator now emits `null` rather than a fabricated `0`.
|
|
16
|
+
- `FeatureMetricsLedger` benchmark findings persistence — modified — stores and reads the explanatory `partialReason`.
|
|
17
|
+
|
|
18
|
+
## 1. Over-block
|
|
19
|
+
|
|
20
|
+
No block/allow surface. This can suppress an actionable benchmark finding into `partial` when the orphan denominator is missing but enough graded rows exist. That is intentional: the rate is not measurable, so aligned/divergent would be dishonest.
|
|
21
|
+
|
|
22
|
+
## 2. Under-block
|
|
23
|
+
|
|
24
|
+
If a denominator is present but wrong, this change does not prove it is correct; it only prevents the specific empty-denominator collapse. Existing aggregate clamps still handle implausible peer rows, and the regression test covers the join-miss shape where `decided_total = 0` while graded rows meet `minSample`.
|
|
25
|
+
|
|
26
|
+
## 3. Level-of-abstraction fit
|
|
27
|
+
|
|
28
|
+
This belongs in the pure verdict core because it is part of the detector's evidence ladder, not a display-only concern. The analyzer owns measurement (`number` vs `null`); the core owns the consequence; the ledger owns durable explanation.
|
|
29
|
+
|
|
30
|
+
## 4. Signal vs authority compliance
|
|
31
|
+
|
|
32
|
+
**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
|
|
33
|
+
|
|
34
|
+
- [x] No — this change produces a signal consumed by an existing smart gate.
|
|
35
|
+
|
|
36
|
+
The benchmark divergence detector remains observe-only and every finding remains `advisory: true`. This change reduces false actionable signals by making an unknown orphan rate non-actionable.
|
|
37
|
+
|
|
38
|
+
## 4b. Judgment-point check (Judgment Within Floors standard)
|
|
39
|
+
|
|
40
|
+
No new static heuristic at a competing-signals decision point. This is an enumerable data-validity invariant: an orphan share with no denominator is unknown, not zero.
|
|
41
|
+
|
|
42
|
+
## 5. Interactions
|
|
43
|
+
|
|
44
|
+
- **Shadowing:** The new partial gate runs before evidence floors and divergence comparison, so it intentionally shadows aligned/divergent when orphan rate is unknown.
|
|
45
|
+
- **Double-fire:** No duplicate actor; `partialReason` is a nullable annotation on the same finding row.
|
|
46
|
+
- **Races:** No new concurrency path. Analyzer remains lease-gated and writes through the existing idempotent upsert.
|
|
47
|
+
- **Feedback loops:** Chronic streak behavior remains unchanged because `partial` was already non-actionable.
|
|
48
|
+
|
|
49
|
+
## 6. External surfaces
|
|
50
|
+
|
|
51
|
+
The `GET /benchmark-divergence` finding envelope gains optional `partialReason`. Existing consumers reading `verdict` and `orphanTainted` continue to work; readers can now tell `orphan-rate-unknown` from `orphan-share-exceeded`. Persistent state gains nullable `benchmark_divergence_findings.partial_reason`, added idempotently for existing databases.
|
|
52
|
+
|
|
53
|
+
No operator-facing actions. No Telegram/Slack/GitHub/Cloudflare behavior. No generated URLs.
|
|
54
|
+
|
|
55
|
+
## 6b. Operator-surface quality (Operator-Surface Quality standard)
|
|
56
|
+
|
|
57
|
+
No operator surface — not applicable.
|
|
58
|
+
|
|
59
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
60
|
+
|
|
61
|
+
Proxied-on-read: benchmark divergence already supports local and pool findings through `GET /benchmark-divergence?scope=pool`, and peer findings are clamped through `clampPeerFinding`. The new `partialReason` field is allowlisted and enum-clamped for peer merge. No user-facing notices, no topic-transfer state, no generated URLs.
|
|
62
|
+
|
|
63
|
+
## 8. Rollback cost
|
|
64
|
+
|
|
65
|
+
Hot-fix release: revert the code and tests. The nullable `partial_reason` column can remain harmlessly unused; no data cleanup is required. During rollback, unknown orphan rates would again risk collapsing to fabricated zero until the revert is replaced.
|
|
66
|
+
|
|
67
|
+
## Conclusion
|
|
68
|
+
|
|
69
|
+
The review found the main side effect: fewer actionable benchmark findings when orphan-rate measurement is missing. That is the intended honesty improvement. The change is clear to ship with the targeted regression test and typecheck passing.
|
|
70
|
+
|
|
71
|
+
## Second-pass review (if required)
|
|
72
|
+
|
|
73
|
+
**Reviewer:** `not required`
|
|
74
|
+
**Independent read of the artifact:** `not required`
|
|
75
|
+
|
|
76
|
+
This change does not touch outbound messaging, session lifecycle, dispatch, context exhaustion, coherence gates, or a guard/watchdog/sentinel path.
|
|
77
|
+
|
|
78
|
+
## Evidence pointers
|
|
79
|
+
|
|
80
|
+
- `npx vitest run tests/unit/benchmarkDivergenceCore.test.ts tests/unit/BenchmarkDivergenceAnalyzer.test.ts tests/integration/benchmark-divergence-routes.test.ts`
|
|
81
|
+
- `npx tsc --noEmit`
|
|
82
|
+
|
|
83
|
+
## Class-Closure Declaration (display-only mirror)
|
|
84
|
+
|
|
85
|
+
No agent-authored-artifact defect — not applicable.
|
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
# Side-effects review — gate the `general` envelope on the pre-call resolved framework
|
|
2
|
+
|
|
3
|
+
**Change:** `ClaimClauseArbiter` asks for the `general` claim-extraction envelope only when the
|
|
4
|
+
component's resolved framework is `claude-code` — the sole framework whose `general` result the
|
|
5
|
+
call site can admit.
|
|
6
|
+
**Files:** `src/monitoring/ClaimClauseArbiter.ts`, `src/monitoring/CompletionClaimVerifier.ts`,
|
|
7
|
+
`tests/unit/claim-arbiter-general-gating.test.ts`
|
|
8
|
+
**Tracked:** CMT-1118. Escalates ACT-1466 (filed 2026-07-28 as a coverage gap) with its cost consequence.
|
|
9
|
+
**Tier declared:** 1 — small (2 source files, ~45 net LOC), no new capability, no migration, no
|
|
10
|
+
irreversibility, no safety-invariant change. Risk floor unchanged.
|
|
11
|
+
|
|
12
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
13
|
+
|
|
14
|
+
**Does this change touch a decision point that gates information flow, blocks actions, filters
|
|
15
|
+
messages, or constrains agent behaviour?** No — and the distinction matters.
|
|
16
|
+
|
|
17
|
+
The admission rule is untouched. `const general = resolvedModel?.framework === 'claude-code' ?
|
|
18
|
+
parsedGeneral : null` is byte-identical before and after. What changes is only what we **ask the
|
|
19
|
+
model to generate**, not what the system **accepts**. No authority is added, moved, or widened;
|
|
20
|
+
if anything the change removes generation work whose product was already unreachable.
|
|
21
|
+
|
|
22
|
+
`CompletionClaimVerifier` is observe-only by construction (per the constitution: claim
|
|
23
|
+
verification "never blocks, rewrites, delays, sends, corrects, or authorizes an action"), so
|
|
24
|
+
there is no blocking authority anywhere on this path for brittle logic to attach to.
|
|
25
|
+
|
|
26
|
+
**Verdict: compliant.** This is a cost reduction inside a signal producer.
|
|
27
|
+
|
|
28
|
+
## Phase 4 — The eight questions
|
|
29
|
+
|
|
30
|
+
### 1. Over-block — what legitimate inputs does this reject that it shouldn't?
|
|
31
|
+
|
|
32
|
+
The failure mode is suppressing `general` on an install where it **would** have been admitted.
|
|
33
|
+
Every uncertain path is biased against that. `includeGeneral` starts `true` and only a
|
|
34
|
+
positively-resolved, non-empty, non-`claude-code` string flips it:
|
|
35
|
+
|
|
36
|
+
- no `resolveFramework` injected → full prompt (this is also the default for every existing
|
|
37
|
+
caller that constructs the arbiter directly, e.g. tests and `opts.arbiter` overrides)
|
|
38
|
+
- resolver returns `undefined` / `''` / a non-string → full prompt
|
|
39
|
+
- resolver throws → caught, full prompt
|
|
40
|
+
- provider is not a router (no `for()` method) → resolver is `undefined` → full prompt
|
|
41
|
+
|
|
42
|
+
So over-blocking requires the router to positively and correctly report a non-Claude framework —
|
|
43
|
+
which is precisely the case where the result could not have been admitted anyway. **No issue
|
|
44
|
+
identified.**
|
|
45
|
+
|
|
46
|
+
### 2. Under-block — what failure modes does this still miss?
|
|
47
|
+
|
|
48
|
+
Two, both known and both leaving today's behaviour intact rather than degrading it:
|
|
49
|
+
|
|
50
|
+
- **Mid-call failure-swap.** The framework is resolved before the call; a runtime swap can land
|
|
51
|
+
the request on a different framework than predicted. Claude-predicted → codex-actual keeps
|
|
52
|
+
today's waste (no regression). Codex-predicted → Claude-actual sends the lean prompt to Claude,
|
|
53
|
+
so `general` is simply absent — and the call site already tolerates absence:
|
|
54
|
+
`envelopeIncludesGeneral = Object.hasOwn(modelRoot, 'general')` only fails when `general` is
|
|
55
|
+
*present but unparseable*. Verified by reading that branch, not assumed.
|
|
56
|
+
- **The 83%→lower error-rate improvement is a prediction, not a measurement.** The latency and
|
|
57
|
+
token ratios are measured; the production error rate moving is not yet observed. Recorded here
|
|
58
|
+
rather than claimed. Closing evidence is owed on CMT-1118 <!-- tracked: CMT-1118 -->.
|
|
59
|
+
|
|
60
|
+
### 3. Level-of-abstraction fit
|
|
61
|
+
|
|
62
|
+
The gate belongs at the prompt-construction site because that is where the cost is incurred and
|
|
63
|
+
where the admission rule's twin already lives — the two are now adjacent and reference each other,
|
|
64
|
+
so they cannot drift apart unnoticed. Pushing it lower (into the provider) would make the provider
|
|
65
|
+
aware of one component's schema; pushing it higher (into routing config) would make an operator
|
|
66
|
+
maintain a duplicate of a rule the code already enforces. **Right layer.**
|
|
67
|
+
|
|
68
|
+
A smarter gate does not already exist for this: nothing else inspects what this component asks for.
|
|
69
|
+
|
|
70
|
+
### 4. Signal vs authority compliance
|
|
71
|
+
|
|
72
|
+
Compliant — see Phase 1. No brittle logic acquires blocking authority; the one existing authority
|
|
73
|
+
(the `claude-code` admission check) is unchanged.
|
|
74
|
+
|
|
75
|
+
### 5. Interactions — shadowing, double-fire, races
|
|
76
|
+
|
|
77
|
+
- **Prompt-id attribution.** The two prompt shapes have materially different latency and output
|
|
78
|
+
size. Recording both under one id would blend them in per-prompt quality/latency attribution and
|
|
79
|
+
make the decision-quality meter quietly wrong about whichever install it looked at. A distinct
|
|
80
|
+
`CLAIM_ARBITER_PROMPT_ID_LEGACY_ONLY` prevents that. This is the interaction most likely to have
|
|
81
|
+
been missed.
|
|
82
|
+
- **Component-name drift.** The resolver must ask about the *same* component the call attributes
|
|
83
|
+
to. If they diverged, the gate would consult the wrong routing entry and mis-gate silently, with
|
|
84
|
+
nothing visibly wrong. Both now read the shared `CLAIM_ARBITER_COMPONENT` constant, making the
|
|
85
|
+
drift structurally impossible rather than merely unlikely.
|
|
86
|
+
- **No shadowing or double-fire:** nothing else builds this prompt; the arbiter has one call site.
|
|
87
|
+
|
|
88
|
+
### 6. External surfaces
|
|
89
|
+
|
|
90
|
+
No API, route, config key, or user-visible surface changes. The only externally-observable deltas
|
|
91
|
+
are (a) a smaller prompt on non-Claude installs and (b) a second `promptId` value appearing in
|
|
92
|
+
provenance records. Nothing another agent, user, or system consumes by name is removed or renamed.
|
|
93
|
+
|
|
94
|
+
### 7. Multi-machine posture (Cross-Machine Coherence)
|
|
95
|
+
|
|
96
|
+
**Machine-local BY DESIGN, and correctly so.** The decision is derived per-call from that machine's
|
|
97
|
+
own resolved routing (`sessions.componentFrameworks`), which is legitimately per-machine — a pool
|
|
98
|
+
can genuinely run Claude on one machine and codex on another, and each machine should ask for
|
|
99
|
+
exactly what its own door can deliver. Nothing durable is written, so there is no state to
|
|
100
|
+
replicate, strand on topic transfer, or merge on read. No user-facing notice, so no one-voice
|
|
101
|
+
gating needed. No generated URLs.
|
|
102
|
+
|
|
103
|
+
### 8. Rollback cost
|
|
104
|
+
|
|
105
|
+
Near-zero, and available three ways without a release: stop injecting `resolveFramework` at the
|
|
106
|
+
construction site (one line), or route the component to `claude-code` via
|
|
107
|
+
`sessions.componentFrameworks`, or revert the commit. No data migration, no agent-state repair,
|
|
108
|
+
no schema change. Nothing persists that a rollback would leave inconsistent, except historical
|
|
109
|
+
provenance rows carrying the legacy-only prompt id — which remain accurate for the calls they
|
|
110
|
+
describe.
|
|
111
|
+
|
|
112
|
+
## Phase 4.5 — No-deferrals
|
|
113
|
+
|
|
114
|
+
No orphan deferrals. The one outstanding item (confirming the production error rate actually
|
|
115
|
+
falls) is tracked on CMT-1118 <!-- tracked: CMT-1118 --> and is evidence-gathering on shipped
|
|
116
|
+
behaviour, not deferred work.
|
|
117
|
+
|
|
118
|
+
## Phase 5 — Second-pass review: NOT REQUIRED, with reasoning
|
|
119
|
+
|
|
120
|
+
Checked against the skill's own high-risk list. This change touches **none** of: block/allow
|
|
121
|
+
decisions on inbound/outbound messaging or dispatch (the admission rule is unchanged and the
|
|
122
|
+
component cannot block); session lifecycle; context exhaustion/compaction/respawn; coherence
|
|
123
|
+
gates, idempotency, or trust levels; nor anything named sentinel/guard/gate/watchdog.
|
|
124
|
+
|
|
125
|
+
Declared `--second-pass not-required`. Stated plainly because the analogous judgement was made on
|
|
126
|
+
PR #1749 earlier tonight and an independent reviewer there found a genuine, ship-blocking defect —
|
|
127
|
+
so "it looked small" is not by itself reassuring. The difference is concrete rather than a feeling:
|
|
128
|
+
#1749 altered delivery semantics on a path that could double-deliver a live instruction; this
|
|
129
|
+
alters only what is asked of a model, on a path whose result is discarded, with every failure mode
|
|
130
|
+
resolving to the pre-existing behaviour.
|
|
131
|
+
|
|
132
|
+
## Evidence
|
|
133
|
+
|
|
134
|
+
- **Measurement:** interleaved A/B (A,B,A,B,A,B so door drift cannot be attributed to a variant),
|
|
135
|
+
`gpt-5.4-mini` via codex, identical message/clauses/evidence. n=3 per arm.
|
|
136
|
+
Full prompt median **129,204 ms** / ~8,338 output tokens, range 91,520–148,719.
|
|
137
|
+
Legacy-only median **28,153 ms** / ~1,386 output tokens, range 24,965–33,708. **Ranges do not
|
|
138
|
+
overlap.** Against this call's own `timeoutMs: 60_000`: full exceeded the wall 3/3, legacy-only
|
|
139
|
+
fit 3/3 — consistent with the observed production `errorRate 0.835`, `p50 49,315`, `p95 60,082`
|
|
140
|
+
over 1,207 calls/24h.
|
|
141
|
+
- **Correctness, not just speed:** the legacy-only response was confirmed valid JSON matching the
|
|
142
|
+
legacy schema, and materially correct — it labelled an imperative to the reader `neither`, and
|
|
143
|
+
flagged an unsupported "everything green" claim `corroborated: false`.
|
|
144
|
+
- **Test discrimination measured, not assumed:** the new tests were run against the unfixed source
|
|
145
|
+
with only the test staged → **6 failed / 4 passed**. The 4 passes are marked `CONTROL` and pass
|
|
146
|
+
on both revisions by design; two of them only revealed themselves as non-discriminating *because*
|
|
147
|
+
that unfixed run was done. Only the 6 failures are counted as evidence for the change.
|
|
148
|
+
- tsc clean (exit 0). 30/30 on this file plus `claim-observation-v1` and `action-claim`.
|
|
149
|
+
|
|
150
|
+
## Two defects found AFTER this artifact was first written, both in my own change
|
|
151
|
+
|
|
152
|
+
Recorded rather than silently folded in, because both are the kind of thing this review exists to
|
|
153
|
+
catch and both were missed by the review's first pass.
|
|
154
|
+
|
|
155
|
+
### 1. The change would have been INERT in production (wiring integrity)
|
|
156
|
+
|
|
157
|
+
`AgentServer` does not hand `CompletionClaimVerifier` the router. It hands it an anonymous
|
|
158
|
+
`{ evaluate }` wrapper so the call rides the metered LLM queue, and that wrapper exposes no
|
|
159
|
+
routing surface. The duck-typed resolver therefore returned `undefined` on every real install,
|
|
160
|
+
`includeGeneral` stayed `true`, and nothing changed — **while all ten unit tests passed**, because
|
|
161
|
+
they inject a router directly.
|
|
162
|
+
|
|
163
|
+
This is exactly "the logic is proven and the feature is not wired." Found by tracing the real
|
|
164
|
+
construction site and asking *what object is actually passed here?* — not by any test.
|
|
165
|
+
|
|
166
|
+
Fixed by adding an explicit `resolveFramework?` to `CompletionClaimVerifierOptions`, passed from
|
|
167
|
+
`AgentServer` where the real router is in scope; the duck-typed fallback remains for callers that
|
|
168
|
+
pass a router straight through. Two WIRING tests added — one pins the defect (production-shaped
|
|
169
|
+
wrapper yields NO resolver), one asserts an explicit resolver reaches the arbiter.
|
|
170
|
+
|
|
171
|
+
### 2. I broke `lint-llm-attribution` by hoisting the attribution to a constant
|
|
172
|
+
|
|
173
|
+
Replacing the inlined `component: 'completion-claim-verify'` literal with `CLAIM_ARBITER_COMPONENT`
|
|
174
|
+
made the callsite look unattributed: that lint reads the callsite **statically** and cannot resolve
|
|
175
|
+
a constant. It failed the build, which surfaced as two e2e preflight failures in the full suite.
|
|
176
|
+
|
|
177
|
+
Reverted to the inlined literal with a comment explaining why it must stay inlined, and added a test
|
|
178
|
+
pinning the constant equal to the literal so the two cannot drift while the callsite stays lintable.
|
|
179
|
+
|
|
180
|
+
## Suite status, measured
|
|
181
|
+
|
|
182
|
+
Full unit suite: **2 failed / 3003 passed** (46,869 tests). Both failures are the same
|
|
183
|
+
`tests/e2e/dev-preflight-cli.test.ts` case.
|
|
184
|
+
|
|
185
|
+
**Discriminated rather than assumed:** re-ran that test with my changes stashed — **it fails
|
|
186
|
+
identically without them.** Cause is `ERR_PNPM_IGNORED_BUILDS` on a fresh clone under pnpm 11.5.1,
|
|
187
|
+
which now demands explicit build-script approval; the repo declares no `pnpm.onlyBuiltDependencies`
|
|
188
|
+
and existing worktrees only pass because their `node_modules` predates the enforcement. So it is a
|
|
189
|
+
fresh-checkout packaging condition affecting anyone cloning today, independent of this change.
|
|
190
|
+
|
|
191
|
+
Filed separately rather than bundled here — mixing an unrelated packaging fix into this change is
|
|
192
|
+
the "batched release" anti-pattern, and it would make it impossible to attribute a later regression
|
|
193
|
+
to the right half. <!-- tracked: CMT-1120 -->
|
|
194
|
+
|
|
195
|
+
After the lint fix: `npm run lint` exit 0, `tsc --noEmit` exit 0, 13/13 on this file.
|
|
196
|
+
|
|
197
|
+
## Honest limits
|
|
198
|
+
|
|
199
|
+
- n=3 per arm is small; the conclusion rests on the size of the separation and the non-overlap,
|
|
200
|
+
not on sample count.
|
|
201
|
+
- Timings came from a direct `codex exec` invocation, not through instar's provider (different
|
|
202
|
+
sandbox/env/out-dir), so **absolute** numbers may differ in production. The **ratio** is the
|
|
203
|
+
load-bearing result and it is controlled.
|
|
204
|
+
- A first attempt to confirm a *different* fix tonight compared two installed builds that were both
|
|
205
|
+
`1.3.1068` — a false control that would have produced a confident wrong answer. Recorded here as
|
|
206
|
+
the reason the A/B above was interleaved and the unfixed test run was not skipped.
|