omp-llm-role 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -8
- package/docs/dev/architecture.md +25 -4
- package/docs/dev/ci.md +12 -1
- package/docs/dev/current-state.md +63 -26
- package/docs/dev/data-sources.md +33 -4
- package/docs/dev/explorer.md +22 -4
- package/docs/dev/quirks.md +32 -0
- package/docs/dev/releasing.md +31 -30
- package/docs/dev/scoring.md +98 -35
- package/docs/dev/testing.md +70 -23
- package/docs/dev/writes-and-state.md +56 -0
- package/docs/llm-role-rankings.md +127 -127
- package/package.json +82 -22
- package/src/availability.ts +87 -9
- package/src/cli/create-role.ts +1 -1
- package/src/engine.ts +157 -76
- package/src/explorer/explain.ts +24 -6
- package/src/explorer/server.ts +15 -2
- package/src/extension.ts +58 -15
- package/src/features.ts +9 -7
- package/src/provider-display.ts +67 -0
- package/src/settings.ts +8 -2
- package/src/updater.ts +91 -13
- package/web/app.js +24 -1
package/docs/dev/scoring.md
CHANGED
|
@@ -137,12 +137,60 @@ route instead of the `1/price²` blend:
|
|
|
137
137
|
- a model with **no matching route** is ineligible for that role
|
|
138
138
|
(`providerPinDrops` → `Decision.pinBlocked`, rendered `; pin-blocked: …`);
|
|
139
139
|
`explainModel` reports "no route matches the role's provider pin".
|
|
140
|
-
- an **unpinned** role ignores `routes` entirely — the blend path is unchanged
|
|
140
|
+
- an **unpinned** role ignores `routes` entirely — the blend path is unchanged —
|
|
141
|
+
**unless** it sets the soft `preferOwnProvider` policy (the `providerPinning`
|
|
142
|
+
capability), which prices it on its **best route** (the max-value-for-money
|
|
143
|
+
candidate route; see *Route ordering* and *Capability presets*).
|
|
141
144
|
|
|
142
145
|
The pin is a cost-and-throughput posture, not a quality edit: it moves the price
|
|
143
146
|
axis to the route's billed price and the throughput axis to the route's p50,
|
|
144
147
|
while every other weighted metric stays the model's own.
|
|
145
148
|
|
|
149
|
+
### Route ordering (the value-for-money score)
|
|
150
|
+
|
|
151
|
+
`rankedRoutes(model, def)` (`src/engine.ts`) is the one function that orders a
|
|
152
|
+
model's candidate routes; `bestRoute` is `rankedRoutes[0]`, and `roleRoute` (the
|
|
153
|
+
route `rankRole` prices the model on) and the updater's auto-pin both read it, so
|
|
154
|
+
the priced route, the written pin and the fallback chain move together.
|
|
155
|
+
|
|
156
|
+
- **Candidates.** A route is a candidate when it is non-degraded (`status === 0`),
|
|
157
|
+
passes `routePassesEndpointFilters(route, def.filters)`, carries a usable billed
|
|
158
|
+
price, and (when `filters.maxPriceUsdPerM > 0`) a thinking-adjusted price at or
|
|
159
|
+
below the cap. The allowed-providers whitelist is applied upstream by pruning
|
|
160
|
+
`m.routes` (see *Whitelist-aware route selection*).
|
|
161
|
+
- **The score.** Each candidate is scored
|
|
162
|
+
`vfm = wT·tNorm − wP·pNorm`, where `tNorm`/`pNorm` are the route's p50
|
|
163
|
+
throughput and cache-adjusted billed price **min/max-normalized over the
|
|
164
|
+
candidate set** (`tNorm = (tput − tMin)/(tMax − tMin)`, `pNorm = (priceEff −
|
|
165
|
+
pMin)/(pMax − pMin)`; a zero span normalizes to 0), and `wT =
|
|
166
|
+
wTput/(wPrice+wTput)`, `wP = wPrice/(wPrice+wTput)`. Sort desc; ties break by
|
|
167
|
+
`providerSlug` then `id`, so a re-run is deterministic. A route with no p50 uses
|
|
168
|
+
the model's blended throughput (an absent p50 is unknown, not slow).
|
|
169
|
+
- **The soft floor.** `filters.preferredMinThroughput` (tok/s; `0`/absent = off)
|
|
170
|
+
is a **preference, not a gate**: routes at or above the floor sort before
|
|
171
|
+
routes below it, and the score orders within each band. A below-floor route is
|
|
172
|
+
**never dropped** — it stays a last-resort candidate.
|
|
173
|
+
- **`routeValue` is deleted.** The old `q(route p50) − λ·route price` route sort
|
|
174
|
+
key is gone; the score above is the only route basis. (The model-level
|
|
175
|
+
`value = q − λ·priceEff` is unchanged — see *`λ` and `value`*.)
|
|
176
|
+
|
|
177
|
+
### Whitelist-aware route selection
|
|
178
|
+
|
|
179
|
+
The updater narrows every model's route pool to the account's
|
|
180
|
+
**allowed-providers privacy whitelist** before ranking (data-sources.md,
|
|
181
|
+
*OpenRouter — allowed-providers whitelist*; architecture.md, *Data flow*). The
|
|
182
|
+
prune is the single change point — `roleRoute`, `bestRoute`, `rankedRoutes` and
|
|
183
|
+
`endpointFilteredModel` all read `m.routes` — so every route consumer above is
|
|
184
|
+
whitelist-aware: a hard `providerPin` on a blocked provider matches no route and
|
|
185
|
+
drops the model (`providerPinDrops`), the soft `preferOwnProvider` basis prices
|
|
186
|
+
each model on its best **allowed** route, the endpoint-filtered blend is computed
|
|
187
|
+
over the allowed pool, and the fallback chains carry only allowed routes. A
|
|
188
|
+
`null` whitelist (no whitelist set, harvest failed) leaves every pool untouched,
|
|
189
|
+
so the ranking is byte-identical to today. The whitelist is a route-selection
|
|
190
|
+
filter, not a re-pricing: the role-independent `1/price²` blend (`m.price`/
|
|
191
|
+
`m.throughput`) is left as the account-agnostic expected value, and a model with
|
|
192
|
+
no allowed route keeps that blend (the soft basis never drops a model).
|
|
193
|
+
|
|
146
194
|
## Capability presets (feature flags)
|
|
147
195
|
|
|
148
196
|
One named flag per opt-in capability (`features.<id>` global,
|
|
@@ -164,44 +212,55 @@ no flags the resolved roles are byte-identical to `DEFAULT_ROLES`. The effective
|
|
|
164
212
|
def is what `rankRole`/`explainModel` see; the authored def (flags + explicit
|
|
165
213
|
keys) is what the explorer edits and exports.
|
|
166
214
|
|
|
167
|
-
What each recommendation does to the ranking math (grounded on the 2026-10-
|
|
168
|
-
cache, `default` role,
|
|
215
|
+
What each recommendation does to the ranking math (grounded on the 2026-10-09
|
|
216
|
+
cache, `default` role, 146 eligible; the `providerPinning` bullet is re-measured
|
|
217
|
+
on the 2026-10-11 cache):
|
|
169
218
|
|
|
170
219
|
- **`endpointCeilings`** (`filters.tools: true`, `filters.minOutputTokens:
|
|
171
220
|
16384`) drops tool-incapable routes and routes whose output ceiling is below
|
|
172
|
-
16 384 tokens from the priced pool (see *Endpoint filters*):
|
|
173
|
-
eligible, leader unchanged (`deepseek-v4.1-flash`, 0.
|
|
221
|
+
16 384 tokens from the priced pool (see *Endpoint filters*): 146 → 135
|
|
222
|
+
eligible, leader unchanged (`deepseek-v4.1-flash`, 0.37 $/M). 16 384 is the
|
|
174
223
|
video's own threshold (19 endpoints cap output at or below it).
|
|
175
224
|
- **`cachePricing`** (`cacheHitRate: 0.5`) reprices the role on a cache-heavy
|
|
176
|
-
loop's invoice (see *Cache-read pricing*):
|
|
177
|
-
`priceEff` 0.
|
|
225
|
+
loop's invoice (see *Cache-read pricing*): 146 eligible, leader unchanged, its
|
|
226
|
+
`priceEff` 0.3732 → 0.3522 (−5.6 %). 0.5 is the low end of the measured
|
|
178
227
|
third-party range (51–69 %), so the preset never over-credits a cache.
|
|
179
|
-
- **`providerPinning`** (`preferOwnProvider: true
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
(unlike the hard `providerPin`, which drops a
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
228
|
+
- **`providerPinning`** (`{ preferOwnProvider: true, filters: {
|
|
229
|
+
preferredMinThroughput: 50 } }`) prices a model on its **best route** — the
|
|
230
|
+
max-value-for-money candidate route (see *Route ordering*: the `vfm` score
|
|
231
|
+
desc, with the soft 50 tok/s floor sorting below-floor routes last) among the
|
|
232
|
+
routes that pass the role's endpoint filters and the account's
|
|
233
|
+
allowed-providers whitelist (see *Whitelist-aware route selection*) — falling
|
|
234
|
+
back to
|
|
235
|
+
the default `1/price²` blend only when the model has no candidate route. It is
|
|
236
|
+
a **per-model policy, not a scalar** — the sensible pin differs per model, so
|
|
237
|
+
the bundle carries a policy field evaluated against the model's own routes at
|
|
238
|
+
rank time, not a slug baked into the lock file. `bestRoute` (`src/engine.ts`)
|
|
239
|
+
is `rankedRoutes(model, def)[0]` (see *Route ordering* for the candidate
|
|
240
|
+
criteria and the score). The basis is **soft**: a model with no
|
|
241
|
+
candidate route keeps the blend and is never dropped, so the capability
|
|
242
|
+
**never changes eligibility** (unlike the hard `providerPin`, which drops a
|
|
243
|
+
model with no matching route). The `maxPriceUsdPerM` gate is **basis-independent**:
|
|
244
|
+
with no hard pin it caps the model's own blend price, not the soft route's, so
|
|
245
|
+
the flag can neither add nor drop a model; the candidate-route cap filter only
|
|
246
|
+
keeps the written pin at or below the cap. Measured coverage on the 2026-10-11 cache: all
|
|
247
|
+
144 routed models have a candidate route (the old own-lab slug-prefix rule
|
|
248
|
+
matched only 51 of 149), so the capability covers the whole routed field. The
|
|
249
|
+
resolved route is the same basis a hard pin uses, so the preference moves the
|
|
250
|
+
**throughput** axis as well as the billed price — a role that routes to the
|
|
251
|
+
best endpoint gets that endpoint's p50 tok/s, not the blend's. Measured on the
|
|
252
|
+
2026-10-11 cache (`rankRole` over the full model set), the `default` leader is
|
|
253
|
+
`deepseek-v4.1-flash` either way: the blend prices it at `priceEff` 0.2317 /
|
|
254
|
+
`value` 0.8056 / `q` 0.8069, while the soft best-route basis prices it at
|
|
255
|
+
`priceEff` 0.1187 / `value` 0.8092 / `q` 0.8099 on its best route `decart/fp4`
|
|
256
|
+
($0.0639 billed, 72 tok/s). The soft route is cheaper than the blend, so
|
|
257
|
+
`priceEff` falls and `value` rises; `q` moves too because the route's p50
|
|
258
|
+
throughput replaces the blend's. That is deliberate (the pin path has always
|
|
259
|
+
priced and measured on the pinned route); a role that wants the blend's
|
|
260
|
+
throughput should leave the flag off.
|
|
202
261
|
- **`costCap`** (`filters.maxPriceUsdPerM: 10`) drops the priciest tail of the
|
|
203
|
-
eligible pool:
|
|
204
|
-
7.8 and p95
|
|
262
|
+
eligible pool: 146 → 137 (the top ≈6 %), leader unchanged. The pool's p90 is
|
|
263
|
+
7.8 and p95 11.1 $/M, and every role leader sits at 0.20–3.71 $/M, so the
|
|
205
264
|
preset never excludes today's leader.
|
|
206
265
|
|
|
207
266
|
**Interaction with the switch hysteresis.** `switchMargin`/`priceSwitchFraction`
|
|
@@ -242,7 +301,10 @@ A model ranks for a role only when:
|
|
|
242
301
|
- `minContextTokens` filters the **endpoint** `context_length`, not the
|
|
243
302
|
model-level `context`;
|
|
244
303
|
- `maxPriceUsdPerM` caps the thinking-adjusted `priceEff` (the price the role
|
|
245
|
-
actually pays), not the bare billed blend
|
|
304
|
+
actually pays), not the bare billed blend. For a **pinned** role that is the
|
|
305
|
+
pin route's price; for a soft `preferOwnProvider` role it is the model's own
|
|
306
|
+
blend price (basis-independent, so the flag cannot change eligibility), while
|
|
307
|
+
the candidate-route filter keeps the auto-pin at or below the cap;
|
|
246
308
|
- a role that declares `tools`/`minContextTokens`/`minOutputTokens` is **priced**
|
|
247
309
|
on the surviving pool: `endpointFilteredModel` recomputes `price`/`throughput`
|
|
248
310
|
as the `1/price²` blend over the routes that clear the filters
|
|
@@ -252,8 +314,9 @@ A model ranks for a role only when:
|
|
|
252
314
|
|
|
253
315
|
`endpointFilterDrops(def, models)` lists the ranking ids the gate removed (they
|
|
254
316
|
pass every other eligibility gate but have no capable route) for the decision
|
|
255
|
-
log. `resolveSettings` validates
|
|
256
|
-
|
|
317
|
+
log. `resolveSettings` validates the four endpoint-filter keys plus the soft
|
|
318
|
+
`filters.preferredMinThroughput`; no shipped role sets an endpoint filter.
|
|
319
|
+
Filters apply before ranking eligibility so percentile norms stay on the
|
|
257
320
|
unfiltered pool.
|
|
258
321
|
|
|
259
322
|
## Pareto frontier (★)
|
package/docs/dev/testing.md
CHANGED
|
@@ -52,28 +52,45 @@ passing.
|
|
|
52
52
|
equal sets → no-filter; empty → unavailable; a keyed id absent from public →
|
|
53
53
|
no-filter; aliases collapse; two parallel GETs, one authenticated; never
|
|
54
54
|
throws.
|
|
55
|
-
11. **
|
|
55
|
+
11. **Allowed-providers whitelist** (`allowed-providers.test.ts`,
|
|
56
|
+
`whitelist.test.ts`) — `fetchAllowedProviders` parses the 404 body's
|
|
57
|
+
"permits only:" list into a set and returns `null` for a 200, a 404 without
|
|
58
|
+
the clause, or a transport throw, with the canary request shape (POST,
|
|
59
|
+
bearer, `provider.only`, `max_tokens: 1`); the updater prunes every model's
|
|
60
|
+
route pool to allowed providers before ranking, so a blocked best route
|
|
61
|
+
falls to the next-best allowed route as the auto-pin, the chain carries only
|
|
62
|
+
allowed routes, a bare entry matches a tiered slug by prefix, the notify line
|
|
63
|
+
reports the active whitelist and the pruned count (or "no whitelist"), and a
|
|
64
|
+
`null` whitelist reproduces today's output byte-for-byte.
|
|
65
|
+
12. **Endpoint ceilings** (`endpoint-ceilings.test.ts`) — the narrower reads
|
|
56
66
|
context/output/tools, a missing field is null; a failing route drops, a null
|
|
57
67
|
field is kept; a model with no capable route is ineligible; a filtered role
|
|
58
68
|
is priced on the surviving pool; `maxPriceUsdPerM` caps the thinking-adjusted
|
|
59
69
|
price.
|
|
60
|
-
|
|
70
|
+
13. **Route pricing (provider pin)** (`route-pricing.test.ts`) — a pinned role
|
|
61
71
|
prices the matching route, not the blend; a tiered pin matches verbatim;
|
|
62
72
|
`maxPriceUsdPerM` caps the route price; throughput is the route p50 with a
|
|
63
73
|
blend fallback; no matching route → ineligible; the `@<slug>` selector and
|
|
64
|
-
chain carry the pin; a pin matching no route leaves the role unchanged.
|
|
65
|
-
|
|
74
|
+
chain carry the pin; a pin matching no route leaves the role unchanged. The
|
|
75
|
+
route-ordering score (spec #71): `rankedRoutes` orders by the value-for-money
|
|
76
|
+
score `vfm = wT·tNorm − wP·pNorm` (a throughput-heavy role picks a faster
|
|
77
|
+
route than a price-heavy role); the soft `filters.preferredMinThroughput`
|
|
78
|
+
floor sorts a below-floor route last **without dropping it**; a route with no
|
|
79
|
+
p50 uses the model's blended throughput (for both the score and the floor
|
|
80
|
+
check); equal scores break by `providerSlug` then `id`; `explainModel`'s
|
|
81
|
+
`autoPin` is the score-max route.
|
|
82
|
+
14. **Thinking price** (`thinking-price.test.ts`) — bare/off unadjusted; a level
|
|
66
83
|
scales only thinking-capable models; the frontier uses the effective price;
|
|
67
84
|
`resolveSettings` validates thinking and rejects the legacy `suffixes` map;
|
|
68
85
|
catalog `thinking[]` wins over the OR flag; meta levels never gate.
|
|
69
|
-
|
|
86
|
+
15. **OpenRouter blend** (`openrouter-blend.test.ts`) — price/throughput are the
|
|
70
87
|
`1/price²`-weighted means over the page routes; flex/priority/degraded/free/
|
|
71
88
|
`:batch` stay out; throughput renormalizes; the find-row fallback; the weight
|
|
72
89
|
follows the input price.
|
|
73
|
-
|
|
90
|
+
16. **Writing metric** (`writing.test.ts`) — `parseWritingEvidence` keys finite
|
|
74
91
|
scores and rejects junk; `applyWritingScores` fills uncovered models; a
|
|
75
92
|
covered model ranks above an uncovered one.
|
|
76
|
-
|
|
93
|
+
17. **Benchmark sources** (`benchmark-sources.test.ts`) — `resolveBenchmarkSource`
|
|
77
94
|
maps an llm-stats benchmark page, a bare id, the writing leaderboard and a
|
|
78
95
|
Design Arena link, `null` for an unknown host; `normalizeMetricKey` is
|
|
79
96
|
dot-free; `parseBenchmarkPayload` reads the llm-stats `entries[]` and writing
|
|
@@ -91,7 +108,7 @@ passing.
|
|
|
91
108
|
fields, `scoreMax`, join) and a metric colliding with a shipped key is
|
|
92
109
|
rejected; `dryRunDeclaration` reports coverage/leader and rejects a zero-join;
|
|
93
110
|
`extractBenchmarkLinks` pulls URLs from prose.
|
|
94
|
-
|
|
111
|
+
18. **Agent creation** (`create-agent.test.ts`) — a purpose fits the expected
|
|
95
112
|
archetype; a `--weights` set violating Σ(non-price) = 1 − w_price is refused;
|
|
96
113
|
an existing agent file without `--force` is refused and writes no role;
|
|
97
114
|
`--dry-run` writes neither file; an architect `spec` replaces the
|
|
@@ -99,46 +116,51 @@ passing.
|
|
|
99
116
|
share, is a no-op for the archetype's own specialist set, and keeps both
|
|
100
117
|
invariants; the nine-axis gate (the maintenance, provenance and cross-source
|
|
101
118
|
axes are annotations, never a drop).
|
|
102
|
-
|
|
119
|
+
19. **Agent removal** (`remove-agent.test.ts`) — the role's lock-file keys and the
|
|
103
120
|
agent file are deleted (backup written); a shipped default role, a reserved
|
|
104
121
|
name and an invalid name are refused; nothing-to-remove errors; `--dry-run`
|
|
105
122
|
writes nothing; the updater drops `modelRoles.<n>` and the plugin-managed
|
|
106
123
|
`task.disabledAgents` entry.
|
|
107
|
-
|
|
124
|
+
20. **Agent pins / disable** (`agent-pins.test.ts`, `agent-disable.test.ts`) —
|
|
108
125
|
`parseAgentPin` takes the first `@role`; `discoverAgentPins` scans the three
|
|
109
126
|
scopes, most specific wins; the sync is not day-gated; an unrelated entry
|
|
110
127
|
keeps its position; an unknown role is never added.
|
|
111
|
-
|
|
128
|
+
21. **Role enable / lock** (`role-enable.test.ts`, `role-lock.test.ts`) —
|
|
112
129
|
disabling drops the role from the resolved set and its `modelRoles` line; a
|
|
113
130
|
locked role is not switched and its chain survives the prune; unlocking
|
|
114
131
|
restores the switch.
|
|
115
|
-
|
|
132
|
+
22. **Settings** (`settings.test.ts`, `settings-schema.test.ts`) — opt-in roles;
|
|
116
133
|
`priceSwitchFraction` range; flat/nested merge; `required` as string or
|
|
117
134
|
array; an external metric key is weightable, a dotted one rejected;
|
|
118
|
-
`package.json` `omp.settings` deep-equals `deriveSettingsSchema()`.
|
|
119
|
-
|
|
135
|
+
`package.json` `omp.settings` deep-equals `deriveSettingsSchema()`. The
|
|
136
|
+
`providerPinning` preset sets `preferOwnProvider: true` +
|
|
137
|
+
`filters.preferredMinThroughput: 50` (the registry row and the expanded def
|
|
138
|
+
agree), an explicit `filters.preferredMinThroughput` beats the preset, and
|
|
139
|
+
the schema carries the `roles.<role>.filters.preferredMinThroughput` row
|
|
140
|
+
(number, min 0, default 0).
|
|
141
|
+
23. **Explorer** (`explorer.test.ts`, `explorer-scopes.test.ts`) — rank deltas;
|
|
120
142
|
contributions sum to `q`; `inverseCardinal` round-trips; Export preserves
|
|
121
143
|
siblings and normalizes nested roles; the scope switcher; the availability
|
|
122
144
|
overlay.
|
|
123
|
-
|
|
145
|
+
24. **Project scope** (`project-updater.test.ts`, `project-setup.test.ts`,
|
|
124
146
|
`project-roles-command.test.ts`) — project mode scopes config/state/history/
|
|
125
147
|
lock to `<cwd>/.omp`; the day gate is per scope; the registry write;
|
|
126
148
|
`setupProject` add/drop/keep, `--dry-run`, `--force`; the command is
|
|
127
149
|
registered.
|
|
128
|
-
|
|
150
|
+
25. **Session model** (`session-model.test.ts`) — the hook fires only after a real
|
|
129
151
|
write on a real `default` change; a dry run and a kept default skip it; a
|
|
130
152
|
throwing hook is contained.
|
|
131
|
-
|
|
153
|
+
26. **Docs** (`docs.test.ts`) — every relative markdown link resolves; no
|
|
132
154
|
reference to a moved doc; the CHANGELOG shape; `package.json` `files` entries
|
|
133
155
|
exist; the README has no dev-only headings.
|
|
134
|
-
|
|
156
|
+
27. **Architect provenance** (`architect-provenance.test.ts`) — the vendored
|
|
135
157
|
prompt header matches `ARCHITECT_PROMPT_VERSION` and the version named in
|
|
136
158
|
[`agent-authoring.md`](agent-authoring.md).
|
|
137
|
-
|
|
159
|
+
28. **Architect retry** (`agent-architect.test.ts`) — a malformed architect
|
|
138
160
|
response is retried once and the parsed spec returned; an always-malformed
|
|
139
161
|
run throws carrying the raw output; a `run()` failure propagates with no
|
|
140
162
|
retry.
|
|
141
|
-
|
|
163
|
+
29. **Error reporting** (`error-report.test.ts`) — `reportUnexpectedError`
|
|
142
164
|
through its one entry point with fully injected deps: `off` → no fetch, no
|
|
143
165
|
ledger write, `action: "off"`; `auto` + token → exactly one create whose body
|
|
144
166
|
carries the fingerprint marker, plugin version, runtime, platform, label and
|
|
@@ -166,17 +188,42 @@ passing.
|
|
|
166
188
|
aborts under one label differ; the in-process guard; and a ledger that
|
|
167
189
|
tolerates junk and truncates a pathological stack — including a multi-byte
|
|
168
190
|
one — to the 64 KB **byte** budget.
|
|
169
|
-
|
|
191
|
+
30. **CI/CD** (`ci.test.ts`) — the workflow parses and declares both jobs with
|
|
170
192
|
`release` needing `test`; the `test` job's command is identical to the
|
|
171
193
|
pre-commit hook's; the tag trigger covers every CHANGELOG tag link and the
|
|
172
194
|
package version; `.node-version` is the file the workflow reads; the release
|
|
173
|
-
job declares `id-token: write` and references no npm token; the
|
|
174
|
-
|
|
195
|
+
job declares `id-token: write` and references no npm token; the release job
|
|
196
|
+
runs the `notes` guard before `npm publish` and both steps name the same
|
|
197
|
+
notes file; the four lockstep version fields agree; no tracked file lives
|
|
198
|
+
under `cache/`; and
|
|
175
199
|
`scripts/release.ts` against fixture repos — `due`'s exit codes, `notes`'s
|
|
176
200
|
five refusals and its exact section body, `cut --dry-run` writing nothing,
|
|
177
201
|
and `cut` bumping the four fields, cutting the changelog, committing and
|
|
178
202
|
tagging while refusing a dirty tree, an existing tag, an empty
|
|
179
203
|
`## [Unreleased]` and a HEAD that is not `origin/main`.
|
|
204
|
+
31. **Auto-pin** (`auto-pin.test.ts`) — the selector gains the best route's
|
|
205
|
+
`@<slug>`; the chain carries every gate-passing route of the primary (all but
|
|
206
|
+
the pinned best) then of each fallback model, in score order (the `vfm` route
|
|
207
|
+
score desc, spec #71), deduped by
|
|
208
|
+
provider slug; a degraded / over-`maxPriceUsdPerM` / endpoint-filter-failing /
|
|
209
|
+
whitelist-blocked route is excluded; the manual pin wins and the no-pin path
|
|
210
|
+
is bare; a re-run is idempotent and a pin change prunes the old key.
|
|
211
|
+
32. **Provider display** (`provider-display.test.ts`,
|
|
212
|
+
`provider-display-wiring.test.ts`) — `providerPinOf` reads the single
|
|
213
|
+
`openRouterRouting.only[0]` (else `vercelGatewayRouting`), preferring
|
|
214
|
+
OpenRouter, and returns null for a missing/empty/multi-entry block or a
|
|
215
|
+
non-string entry; `providerPinOfSelector` parses the trailing `@<slug>` with
|
|
216
|
+
and without a thinking suffix and returns null without one;
|
|
217
|
+
`withProviderSuffix` appends, strips a previous suffix, is idempotent, and
|
|
218
|
+
returns the base name for a null pin; `applyProviderDisplay` mutates a
|
|
219
|
+
record's `name` and no-ops on a non-record or a non-string name. The wiring
|
|
220
|
+
(a fake `pi`/`ctx` recording handlers) suffixes the live model on
|
|
221
|
+
`session_start`, applies the pin parsed from `event.to` on
|
|
222
|
+
`retry_fallback_applied` (falling back to the live model's routing block),
|
|
223
|
+
applies the pin from `event.model` on `retry_fallback_succeeded`, re-derives
|
|
224
|
+
from the live model on `turn_start` (the silent-restore case), strips when
|
|
225
|
+
neither source carries a pin, never calls `pi.setModel`, and contains a
|
|
226
|
+
throwing live-model read.
|
|
180
227
|
|
|
181
228
|
## Live checks
|
|
182
229
|
|
|
@@ -134,6 +134,62 @@ gating — that this plugin never reads (zero occurrences in `src/`, `web/`,
|
|
|
134
134
|
`tests/`). The plugin's `features.<id>` keys are plugin settings, not omp
|
|
135
135
|
feature flags; do not merge the two.
|
|
136
136
|
|
|
137
|
+
## The provider-pin auto-configuration (spec #58)
|
|
138
|
+
|
|
139
|
+
When a role's **expanded** def sets `preferOwnProvider: true` (the
|
|
140
|
+
`providerPinning` capability's recommended bundle) and carries **no** manual
|
|
141
|
+
`providerPin`, the updater binds the role's requests to the chosen model's best
|
|
142
|
+
route by writing that route's provider slug as the selector's `@<slug>` suffix:
|
|
143
|
+
|
|
144
|
+
```
|
|
145
|
+
openrouter/<id>@<slug>:<level>
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
omp's `splitUpstreamRouting` parses the trailing `@<slug>` and applies
|
|
149
|
+
`compat.openRouterRouting = { only: [slug] }`, so the price the ranking assumed
|
|
150
|
+
is the price the request pays. The slug is `bestRoute(chosenModel, def)`'s
|
|
151
|
+
`providerSlug` — the max-value-for-money candidate route (see scoring.md,
|
|
152
|
+
*Route ordering*). The pin is re-derived on every run, so it follows the ranking when the
|
|
153
|
+
chosen model changes. A manual `roles.<role>.providerPin` **wins** over the
|
|
154
|
+
automatic choice and stays a **hard** gate (a model with no matching route is
|
|
155
|
+
ineligible); clearing it resumes the automatic choice.
|
|
156
|
+
|
|
157
|
+
**Chain shape.** The primary's chain key is the selector minus the trailing
|
|
158
|
+
`:level` — so it **includes** the pin (`openrouter/<id>@<slug>`). Its values, in
|
|
159
|
+
order:
|
|
160
|
+
|
|
161
|
+
1. the same model on **every** gate-passing route except the pinned best, in
|
|
162
|
+
score order (the `vfm` route score desc — see scoring.md, *Route ordering*),
|
|
163
|
+
2. then, for each of the `fallbackChainDepth` fallback models, that model on
|
|
164
|
+
**every** gate-passing route, in score order.
|
|
165
|
+
|
|
166
|
+
Length = `(R_primary − 1) + Σ R_fallback`, where `R` is the number of distinct
|
|
167
|
+
gate-passing provider slugs for that model — data-driven, not a fixed cap.
|
|
168
|
+
Entries dedup by chain value, so two routes sharing a provider slug collapse to
|
|
169
|
+
one. Every entry carries the same `@<slug>` suffix (and the role's thinking level
|
|
170
|
+
where the target advertises it). A model with no gate-passing route contributes
|
|
171
|
+
nothing. The gates are the engine's `rankedRoutes` set: non-degraded
|
|
172
|
+
(`status === 0`), the endpoint capability filters (`tools` / `minContextTokens` /
|
|
173
|
+
`minOutputTokens`), a usable billed price, and the `maxPriceUsdPerM` cap
|
|
174
|
+
(thinking-adjusted). The soft `filters.preferredMinThroughput` floor is a
|
|
175
|
+
**preference, not a gate** — it sorts below-floor routes last but never drops
|
|
176
|
+
them, so a below-floor route still appears in the chain. The allowed-providers
|
|
177
|
+
whitelist is not a `rankedRoutes` filter: the updater prunes each model's route
|
|
178
|
+
pool to allowed providers before ranking, which is what makes `rankedRoutes`
|
|
179
|
+
whitelist-aware. The probe walk target is unchanged (`1 + fallbackChainDepth` —
|
|
180
|
+
it counts models, not routes).
|
|
181
|
+
|
|
182
|
+
**Prune on a pin change.** Because the chain key includes the pin, a pin change
|
|
183
|
+
changes the key; the updater's chain prune (`state.pluginWrittenChainKeys`) drops
|
|
184
|
+
the old key so a stale pinned chain does not linger.
|
|
185
|
+
|
|
186
|
+
**No-ops.** The auto-pin writes nothing when: the capability is off
|
|
187
|
+
(`preferOwnProvider` not `true`); a manual `providerPin` is set; the role is
|
|
188
|
+
locked or disabled; the selector is not an OpenRouter selector; the model has no
|
|
189
|
+
route data; no route passes the role's endpoint filters; or the best route has no
|
|
190
|
+
usable billed price. With the capability off the written config is byte-identical
|
|
191
|
+
to before.
|
|
192
|
+
|
|
137
193
|
## State, history, lock
|
|
138
194
|
|
|
139
195
|
All under the agent dir (`agentDir()` in `src/state.ts`), next to the config
|