instar 1.3.770 → 1.3.772
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +36 -0
- package/dist/commands/server.js.map +1 -1
- package/dist/core/IntelligenceRouter.d.ts +167 -0
- package/dist/core/IntelligenceRouter.d.ts.map +1 -1
- package/dist/core/IntelligenceRouter.js +281 -1
- package/dist/core/IntelligenceRouter.js.map +1 -1
- package/dist/core/PostUpdateMigrator.d.ts +14 -0
- package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
- package/dist/core/PostUpdateMigrator.js +37 -0
- package/dist/core/PostUpdateMigrator.js.map +1 -1
- package/dist/core/types.d.ts +51 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/data/llmBenchCoverage.d.ts +63 -0
- package/dist/data/llmBenchCoverage.d.ts.map +1 -1
- package/dist/data/llmBenchCoverage.js +92 -0
- package/dist/data/llmBenchCoverage.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +19 -19
- package/src/data/llmBenchCoverage.ts +145 -0
- package/upgrades/1.3.771.md +53 -0
- package/upgrades/1.3.772.md +63 -0
- package/upgrades/la4-degrade-clamp-a1.eli16.md +42 -0
- package/upgrades/nature-routing-a2.eli16.md +50 -0
- package/upgrades/side-effects/la4-degrade-clamp-a1.md +77 -0
- package/upgrades/side-effects/nature-routing-a2.md +89 -0
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# ELI16 — Nature-Axis Routing, Increment A2.1 (the dark/dryRun mechanism)
|
|
2
|
+
|
|
3
|
+
## The one-sentence version
|
|
4
|
+
|
|
5
|
+
We taught the internal LLM router how to pick *which model, on which access door* a
|
|
6
|
+
background check should run on — based on the **kind** of thinking the check does — but
|
|
7
|
+
we shipped it **turned off** everywhere except as a silent "here's what I *would* pick"
|
|
8
|
+
observer on a development agent.
|
|
9
|
+
|
|
10
|
+
## Why
|
|
11
|
+
|
|
12
|
+
Different internal jobs need different brains. A quick "is this a STOP command?" sort is a
|
|
13
|
+
different task than a careful "should this message be blocked?" judgment. The bench (INSTAR
|
|
14
|
+
v3) measured which model+door combo is best for each kind — and found one combo that's
|
|
15
|
+
actively *dangerous* for careful judgments: Opus running through the Claude Code CLI (the
|
|
16
|
+
CLI wraps every prompt in ~20k tokens of "helpful assistant" framing, which makes a skeptical
|
|
17
|
+
judge go soft — it missed real STOP commands 27% of the time). So routing has to be able to
|
|
18
|
+
say "for a judgment call, NEVER land on that door."
|
|
19
|
+
|
|
20
|
+
## What this increment actually adds (and what it deliberately doesn't)
|
|
21
|
+
|
|
22
|
+
**Adds (all pure, all dark):**
|
|
23
|
+
- A **resolver** — a small, side-effect-free function: given a component, it works out the
|
|
24
|
+
task *nature*, walks an ordered list of `(door, model)` candidates, skips the ones that
|
|
25
|
+
aren't reachable, and returns the winner + the fallback tail. It has exactly four honest
|
|
26
|
+
outcomes: a route, "fall through to today's routing" (for a component we haven't mapped),
|
|
27
|
+
"no route — use your own backup" (for a low-stakes sorter when everything's down), or
|
|
28
|
+
**throw a distinct fail-closed error** (for a safety gate when everything's down — a gate
|
|
29
|
+
must fail shut, never open).
|
|
30
|
+
- **FD4.1 — the concrete pin.** A2 finishes the job A1 left: the sanctioned Claude reserve
|
|
31
|
+
now pins to a *concrete* model id (`claude-sonnet-4-6`), not a tier nickname that could
|
|
32
|
+
drift. And a **deny-by-default allowlist** clamp: on the Claude door, a judgment/sort call
|
|
33
|
+
may ONLY use that one reserve id — anything else gets clamped down to it. (Open-ended
|
|
34
|
+
*writing* is exempt — that's where Opus-via-CLI is legitimately the best tool.)
|
|
35
|
+
- The `sessions.natureRouting` **config knob** (seeded dark on update), the wiring, and the
|
|
36
|
+
full test suite.
|
|
37
|
+
|
|
38
|
+
**Doesn't (tracked, not dropped):** the actual *re-routing* (enforcing mode), the injection
|
|
39
|
+
map, the build-time lint, the live-config validator, the durable audit log + dashboard read
|
|
40
|
+
surface, the critical-gate drift notice, and the Fable→Opus migration — those are the ordered
|
|
41
|
+
A2.2 remainder. And **nothing** about the paid metered doors or the money/PIN go-live — that
|
|
42
|
+
is Increment B, deferred and PIN-gated.
|
|
43
|
+
|
|
44
|
+
## The safety promise
|
|
45
|
+
|
|
46
|
+
When the feature is **off** (which is everywhere on the fleet), the router behaves **bit-for-bit
|
|
47
|
+
like today** — same door, same model, same everything. That's the whole safety case, and there's
|
|
48
|
+
a named test that asserts it (`natureRouting UNSET ⇒ selection unchanged`). On a dev agent it runs
|
|
49
|
+
in **dryRun**: it computes and logs what it *would* pick, then still does exactly today's thing.
|
|
50
|
+
Flipping it to actually re-route is a later, operator-driven step.
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Side-Effects Review — LA4 unconditional degrade-path safety clamp (S4 Increment A1)
|
|
2
|
+
|
|
3
|
+
**Spec:** docs/specs/nature-axis-routing.md (status: **converged — pending operator approval** — NOT yet
|
|
4
|
+
approved; this ships as a **Tier-1** standalone safety narrowing, the smallest independently-landable
|
|
5
|
+
unit the spec's FD9 defines as **A1**). **Parent standards:** "Structure > Willpower", "No Silent
|
|
6
|
+
Degradation to Brittle Fallback", benchmark-cited routing (INSTAR-Bench v3, rules R1/R2).
|
|
7
|
+
**Not dark-gated:** A1 is UNCONDITIONAL by design — it fires regardless of `sessions.natureRouting`.
|
|
8
|
+
**Files:** src/core/IntelligenceRouter.ts, tests/unit/la4-degrade-path-clamp.test.ts,
|
|
9
|
+
docs/specs/nature-axis-routing.md (+ .eli16.md + reports/nature-axis-routing-convergence.md — the
|
|
10
|
+
converged spec docs riding to main), upgrades/la4-degrade-clamp-a1.eli16.md,
|
|
11
|
+
upgrades/side-effects/la4-degrade-clamp-a1.md, upgrades/next/la4-degrade-clamp-a1.md.
|
|
12
|
+
|
|
13
|
+
## What changed
|
|
14
|
+
|
|
15
|
+
1. **IntelligenceRouter.ts — the degrade-to-default path (`evaluate()`, the `if (!primary)` block).**
|
|
16
|
+
The shipped router degrades a binary-missing routed framework to
|
|
17
|
+
`defaultProvider.evaluate(prompt, options)` UNCLAMPED. When `defaultFramework === 'claude-code'` and
|
|
18
|
+
`options.model === 'capable'`, that landing is Opus-via-Claude-CLI — the measured-banned route for a
|
|
19
|
+
bounded/gating verdict (99.1% clean-API vs 81.7% via the CLI; emergency-stop 73%). The merged S2 clamp
|
|
20
|
+
(`clampClaudeCliSwapModel`) only guarded the failure-swap LOOP, leaving this exit open (the spec's LA4
|
|
21
|
+
fail-open). A1 clamps this exit: for a bounded/gating call, `options.model` is passed through
|
|
22
|
+
`clampClaudeCliSwapModel(defaultFramework, options.model)` (the SAME `capable→balanced` reserve S2
|
|
23
|
+
uses), and a distinct `degrade-path-model-clamp (LA4)` degrade note is emitted.
|
|
24
|
+
2. **Two new exported pure helpers** in the same module: `routingNatureFor(component)` (a read-only lookup
|
|
25
|
+
of the merged S1 `LLM_ROUTING_NATURE` map with the same per-operation "/segment" + `server:` key
|
|
26
|
+
handling `categoryForComponent` uses) and `isBoundedGatingDegrade(component, options)` — the FD4/CR6-3
|
|
27
|
+
predicate: TRUE iff `attribution.gating === true` OR the component maps to a non-`WRITE` chain. A
|
|
28
|
+
`WRITE`-chain component (its legitimate Opus-CLI quality lane) or an unmapped, non-gating call ⇒ FALSE
|
|
29
|
+
(left unchanged).
|
|
30
|
+
3. **Spec docs ride to main** (same pattern as reviewer-door inc1): the converged nature-axis-routing spec
|
|
31
|
+
+ its ELI16 + convergence report, with `status: "converged — pending operator approval"` and the
|
|
32
|
+
`review-convergence` tag preserved AS-IS (no `approved: true` — that is the operator's step).
|
|
33
|
+
|
|
34
|
+
## Blast radius
|
|
35
|
+
|
|
36
|
+
- **The clamp only ever NARROWS.** `clampClaudeCliSwapModel` returns `capable→balanced` only for
|
|
37
|
+
`claude-code`; every other door and every non-`capable` tier passes through unchanged. It never upgrades
|
|
38
|
+
a tier, never blocks a call, never touches WRITE.
|
|
39
|
+
- **Scope is exactly R1/R2.** `isBoundedGatingDegrade` fires only for a mapped non-WRITE nature or an
|
|
40
|
+
explicit `attribution.gating` flag. An unmapped, non-gating degrade (the common "other" category) is
|
|
41
|
+
left byte-identical. A non-`claude-code` default door is never clamped (Opus-via-API is fine).
|
|
42
|
+
- **No new config, no new route, no state.** Two pure functions + a few lines on an existing branch. The
|
|
43
|
+
static map read is O(1) and always available, so the predicate never depends on `natureRouting` being
|
|
44
|
+
set — which is what makes the clamp genuinely unconditional.
|
|
45
|
+
- **Honest non-byte-identical deviation.** This is the ONE deliberate behavior change on the degrade path:
|
|
46
|
+
a binary-missing bounded/gating `capable` degrade with a `claude-code` default now yields Sonnet-CLI
|
|
47
|
+
instead of Opus-CLI. Strictly the safe direction (measured-worse route → sanctioned reserve). Every
|
|
48
|
+
other path is untouched.
|
|
49
|
+
|
|
50
|
+
## Risk + mitigation
|
|
51
|
+
|
|
52
|
+
- **Risk:** the clamp over-reaches and downgrades a legitimate Opus-CLI WRITE call. **Mitigation:**
|
|
53
|
+
`isBoundedGatingDegrade` keys on `chain !== 'WRITE'`; a WRITE component is exempt. Proven by a property
|
|
54
|
+
test over the whole `LLM_ROUTING_NATURE` map (predicate === `chain !== 'WRITE'` for every mapped row).
|
|
55
|
+
- **Risk:** the clamp silently fails to fire when the feature is off (the exact fleet-default state where
|
|
56
|
+
the fail-open lives). **Mitigation:** the predicate reads only the static map + `attribution`, never
|
|
57
|
+
`natureRouting`; a test asserts the clamp fires with `natureRouting` UNSET, and separately that a
|
|
58
|
+
non-`claude-code` default is left alone (both-default-framework arms).
|
|
59
|
+
- **Risk:** collateral change to unrelated calls. **Mitigation:** unmapped, non-gating degrade test
|
|
60
|
+
asserts `model` is passed through untouched (`capable` stays `capable`).
|
|
61
|
+
|
|
62
|
+
## Framework generality
|
|
63
|
+
|
|
64
|
+
Not applicable to the session-launch/inject abstraction (this touches the internal LLM router only). The
|
|
65
|
+
clamp is keyed on the `claude-code` DOOR specifically because that door carries the measured harness
|
|
66
|
+
penalty; codex-cli / gemini-cli / pi-cli defaults are correctly never clamped (Opus-via-those-doors is
|
|
67
|
+
not the banned route). This is framework-correct, not a Claude-only assumption.
|
|
68
|
+
|
|
69
|
+
## Tests
|
|
70
|
+
|
|
71
|
+
`tests/unit/la4-degrade-path-clamp.test.ts` (13 cases): `routingNatureFor` key handling; the
|
|
72
|
+
`isBoundedGatingDegrade` predicate incl. the whole-map property test; and the load-bearing `evaluate()`
|
|
73
|
+
integration — the clamp fires on a bounded/gating degrade with `natureRouting` unset, fires on a
|
|
74
|
+
gating-flagged unmapped call, does NOT over-clamp an unmapped non-gating call, does NOT clamp a
|
|
75
|
+
non-`claude-code` default, and leaves a non-`capable` tier alone. Existing
|
|
76
|
+
`opus-claude-cli-gating-guardrail.test.ts` (14) + `intelligence-router.test.ts` (17) +
|
|
77
|
+
`provider-fallback-swap-timeout.test.ts` (11) stay green; `tsc` clean.
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# Side-Effects Review — Nature-Axis Routing, Increment A2.1 (dark/dryRun mechanism)
|
|
2
|
+
|
|
3
|
+
**Spec:** docs/specs/nature-axis-routing.md (status: **converged — pending operator approval**;
|
|
4
|
+
`review-convergence` tag, NOT `approved:true` — the operator's step). This ships as a **Tier-1**
|
|
5
|
+
change: the smallest independently-landable **dark** unit of FD9's Increment A2. **Parent standards:**
|
|
6
|
+
"Structure > Willpower", "No Silent Degradation to Brittle Fallback", benchmark-cited routing
|
|
7
|
+
(INSTAR-Bench v3, rules R1/R2/R8), the Maturation Path (dev-gated dark), Migration Parity.
|
|
8
|
+
|
|
9
|
+
**Files:** src/data/llmBenchCoverage.ts, src/core/IntelligenceRouter.ts, src/core/types.ts,
|
|
10
|
+
src/commands/server.ts, src/core/PostUpdateMigrator.ts, tests/unit/nature-routing-resolver.test.ts,
|
|
11
|
+
tests/unit/migrate-nature-routing-dark.test.ts, upgrades/nature-routing-a2.eli16.md,
|
|
12
|
+
upgrades/side-effects/nature-routing-a2.md, upgrades/next/nature-routing-a2.md.
|
|
13
|
+
|
|
14
|
+
## What changed
|
|
15
|
+
|
|
16
|
+
1. **src/data/llmBenchCoverage.ts — the DATA half.** New pure, read-only constants: `RoutingDoor`
|
|
17
|
+
(CLI + metered-API door taxonomy, FD1), `CLI_ROUTING_DOORS` / `METERED_ROUTING_DOORS`,
|
|
18
|
+
`ChainPosition` + `NATURE_ROUTING_DEFAULT_CHAINS` (the four v3 CLI-only chains, FD2 — metered
|
|
19
|
+
positions are present but always skipped in Increment A), `ROUTING_LABEL_TO_MODEL_ID` (FD-LABEL/
|
|
20
|
+
FD4.1 — benchmark-label → concrete model id), `CLAUDE_CODE_RESERVE_MODEL_ID` (the single
|
|
21
|
+
sanctioned `claude-sonnet-4-6` reserve, FD4 place 1), and `NATURE_ROUTING_CRITICAL_GATES` (FD6).
|
|
22
|
+
Importing this changes NO behavior — it is data actuated only when the feature is enabled.
|
|
23
|
+
2. **src/core/IntelligenceRouter.ts — the resolver.** New pure exports: `resolveNatureAndChain`
|
|
24
|
+
(FD3 tighten rule `E,B ≥ D ≥ A`), `resolvePositionModelId` (FD4.1), `clampToReserveOnCleanDoor`
|
|
25
|
+
(FD4 place-3 allowlist clamp), `resolveRoute` (the stateless fold with the four outcomes),
|
|
26
|
+
`mergeNatureRoutingChains`, and the `RouterFailClosedError` typed error. Two new
|
|
27
|
+
`IntelligenceRouterOptions` fields (`resolveNatureRouting`, `onNatureRoutePlan`) and a new,
|
|
28
|
+
tightly-scoped block in `evaluate()` that OBSERVES when the feature is enabled.
|
|
29
|
+
3. **src/core/types.ts** — the `attribution.nature` opt-in tightening field (FD3) and the
|
|
30
|
+
`sessions.natureRouting` config schema (type-only, NOT in ConfigDefaults, per the
|
|
31
|
+
absent-equals-unchanged rule that `componentFrameworks`/`dynamicMcp` follow).
|
|
32
|
+
4. **src/commands/server.ts** — the construction wiring: `resolveNatureRouting` reads config LIVE
|
|
33
|
+
per call and resolves `enabled` through `resolveDevAgentGate` (live-in-dryRun on a dev agent,
|
|
34
|
+
dark on the fleet); `onNatureRoutePlan` is an env-gated observe-only breadcrumb.
|
|
35
|
+
5. **src/core/PostUpdateMigrator.ts** — `migrateConfigNatureRoutingDark` seeds `sessions.natureRouting`
|
|
36
|
+
dark on existing agents (`schemaVersion:3`, `dryRun:true`, `metered.goLive:false`; `enabled`
|
|
37
|
+
OMITTED for the dev-gate — the #1001 enable-path-integrity pattern), existence-checked + idempotent.
|
|
38
|
+
|
|
39
|
+
## Explicitly DEFERRED (tracked A2.2 remainder — NOT dropped)
|
|
40
|
+
|
|
41
|
+
Read this as intentional scope, not omission. This increment ships the pure mechanism + dark/dryRun
|
|
42
|
+
observation. The following are the ordered remainder, each a separate landable change:
|
|
43
|
+
|
|
44
|
+
- **Enforcing SELECTION** (dryRun:false actually re-routes) — A2.1 wires OBSERVATION only; flipping
|
|
45
|
+
`dryRun:false` logs a one-time "enforcing not yet wired" warning and stays byte-identical (an honest
|
|
46
|
+
no-op, never a silent dead switch and never a mis-route).
|
|
47
|
+
- **FD4 places 1 & 2** — the build-time lint (`scripts/lint-nature-chains.mjs`) and the resolve-time
|
|
48
|
+
live-config validator. **Place 3 (the runtime allowlist clamp) DOES ship here** and is the actual
|
|
49
|
+
runtime guarantee: it clamps every resolved claude-code FAST/SORT/JUDGE position to the sanctioned
|
|
50
|
+
reserve id at selection time — so even a hand-edited chain cannot open the banned Opus-via-CLI route.
|
|
51
|
+
Places 1-2 are defense-in-depth for the build + config-edit surfaces.
|
|
52
|
+
- **FD5b injection-exposure map** + R-rule lints (FD5c) — inert in A2.1 anyway: the only injection-
|
|
53
|
+
restricted door (`groq-api`) and the R8 Flash-Lite pin are METERED doors, always skipped in Increment
|
|
54
|
+
A; no CLI door in any chain is injection-restricted.
|
|
55
|
+
- **The durable audit** (`logs/nature-routing.jsonl`) + `GET /intelligence/routing` dryRun plan/diff/
|
|
56
|
+
`?trace` read surface, the **FD6 aggregated critical-gate drift notice** + baseline, the **FD8
|
|
57
|
+
Fable→Opus** migration, and the **CLAUDE.md** capability blurb.
|
|
58
|
+
- **Increment B** (metered-door live routing + FD12 money/PIN go-live) — DEFERRED + PIN-gated, an
|
|
59
|
+
operator step, never autonomous. No metered door routes and no spend ledger is touched here.
|
|
60
|
+
|
|
61
|
+
## Blast radius
|
|
62
|
+
|
|
63
|
+
- **Byte-identical when off — THE safety case.** When `sessions.natureRouting` is absent OR
|
|
64
|
+
`enabled:false` (the fleet default), the new block in `evaluate()` is skipped entirely and selection
|
|
65
|
+
is bit-for-bit today's. Asserted by name: `natureRouting UNSET ⇒ selection unchanged, onNatureRoutePlan
|
|
66
|
+
NEVER called` (the same options object is passed through untouched).
|
|
67
|
+
- **A1 is untouched.** `clampClaudeCliSwapModel` (A1's always-on degrade/swap clamp, returning the
|
|
68
|
+
`balanced` TIER token) is NOT modified — the new `clampToReserveOnCleanDoor` is a SEPARATE, nature-
|
|
69
|
+
routing-scoped, concrete-id clamp used only inside `resolveRoute`. Touching A1's fn would have changed
|
|
70
|
+
its shipped byte-identical behavior; a test asserts A1 still returns `{model:'balanced'}`.
|
|
71
|
+
- **dryRun cannot mis-route.** In dryRun the resolver only computes + logs; a resolver throw
|
|
72
|
+
(critical-gate fail-closed) is swallowed and recorded, never surfaced to the call path.
|
|
73
|
+
- **Hot path.** The observation block runs on every internal LLM call ONLY when the feature is enabled
|
|
74
|
+
(dev agents). It is a stateless fold over static maps + O(1) door-reachability reads (cached by the
|
|
75
|
+
provider cache); the breadcrumb log is env-gated so a dev agent's hot path stays quiet.
|
|
76
|
+
- **Config.** The seed is dark, existence-checked, idempotent, `enabled` omitted — it can never
|
|
77
|
+
force-dark a dev agent nor clobber an operator's config.
|
|
78
|
+
|
|
79
|
+
## Second-pass review (Phase 5 — required: touches a "gate" / routing-of-safety-gates)
|
|
80
|
+
|
|
81
|
+
The change routes safety GATES, so it triggers the high-risk second pass. Independent audit focus:
|
|
82
|
+
(a) can the banned Opus-via-CLI route open? — No: the place-3 allowlist clamp is deny-by-default and
|
|
83
|
+
fires on every resolved claude-code bounded/gating position, including a hand-edited chain. (b) can a
|
|
84
|
+
critical gate fail OPEN? — No: the empty-set branch throws a distinct `RouterFailClosedError` for a
|
|
85
|
+
critical gate (never `no-route`, never legacy routing); a unit test asserts both a nature-B gate
|
|
86
|
+
(`MessagingToneGate`) and the nature-A R2-critical gate (`MessageSentinel`) throw. (c) is "off" truly
|
|
87
|
+
inert? — Yes, the byte-identical-when-off test proves it. **Reviewer verdict: Concur with the review.**
|
|
88
|
+
The one residual is that enforcing SELECTION is deferred — but since it is not wired, it cannot
|
|
89
|
+
mis-route; the runtime clamp already makes the future enforcing path safe on its own.
|