omp-llm-role 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +75 -13
- package/docs/dev/architecture.md +35 -11
- package/docs/dev/ci.md +21 -1
- package/docs/dev/current-state.md +99 -27
- package/docs/dev/data-sources.md +33 -4
- package/docs/dev/explorer.md +126 -21
- package/docs/dev/quirks.md +32 -0
- package/docs/dev/releasing.md +63 -32
- package/docs/dev/scoring.md +153 -39
- package/docs/dev/testing.md +112 -24
- package/docs/dev/writes-and-state.md +86 -0
- package/docs/llm-role-rankings.md +126 -126
- package/package.json +92 -22
- package/src/agent-create.ts +4 -2
- package/src/availability.ts +87 -9
- package/src/cli/create-role.ts +1 -1
- package/src/engine.ts +190 -78
- package/src/explorer/boot.ts +43 -1
- package/src/explorer/explain.ts +113 -10
- package/src/explorer/levers.ts +100 -0
- package/src/explorer/pool.ts +46 -0
- package/src/explorer/server.ts +107 -23
- package/src/extension.ts +189 -16
- package/src/features.ts +9 -7
- package/src/provider-display.ts +67 -0
- package/src/settings.ts +39 -2
- package/src/updater.ts +97 -14
- package/src/usage-ledger.ts +127 -0
- package/web/app.js +156 -1
- package/web/style.css +3 -0
package/README.md
CHANGED
|
@@ -48,11 +48,28 @@ omp plugin install omp-llm-role@gott50-plugins
|
|
|
48
48
|
session actually booted on the previous `default` selector (an explicit
|
|
49
49
|
`omp --model X` is never clobbered), and only on a real change. Disable with
|
|
50
50
|
`activateDefaultOnEmptySession=false`.
|
|
51
|
+
- **Provider in the model display.** When the live session model carries a
|
|
52
|
+
provider pin — a hand-set `providerPin`, or the automatic pin the
|
|
53
|
+
`providerPinning` capability writes — the status line's model segment shows
|
|
54
|
+
the route next to the model name, e.g. `DeepSeek V4.1 Flash @coreweave/fp8`.
|
|
55
|
+
The suffix is applied on session start, refreshed when a fallback switches the
|
|
56
|
+
model, and re-derived at the start of every turn, so it self-heals after omp
|
|
57
|
+
silently restores the primary model following a fallback cooldown. An unpinned
|
|
58
|
+
model's display is unchanged, and the suffix is display-only: it rides the
|
|
59
|
+
model object's `name`, so attribution, usage recording and model comparison
|
|
60
|
+
are unaffected.
|
|
51
61
|
- **Surgical writes.** Only managed role lines, managed chain keys, and the
|
|
52
62
|
plugin-managed `task.disabledAgents` names change; comments, blank lines and
|
|
53
63
|
unknown keys stay byte-identical, and a no-change run writes nothing.
|
|
54
64
|
`~/.omp/agent/llm-role-state.json` snapshots the previous `modelRoles` block
|
|
55
65
|
on every write as a rollback aid.
|
|
66
|
+
- **Usage ledger.** The plugin appends one row per assistant turn (the model,
|
|
67
|
+
the role, the provider, the token counts and the cost) to
|
|
68
|
+
`llm-role-usage.jsonl` beside its other state — `<project>/.omp` in project
|
|
69
|
+
mode, `~/.omp/agent` otherwise. It is append-only, capped at the newest 2000
|
|
70
|
+
rows, and best-effort (a recording failure never breaks a session). The
|
|
71
|
+
explorer's `$/task` column reads it to learn your own token mix; nothing else
|
|
72
|
+
uses it.
|
|
56
73
|
|
|
57
74
|
## Roles
|
|
58
75
|
|
|
@@ -110,6 +127,7 @@ Global knobs:
|
|
|
110
127
|
| `writeFallbackChains` | `true` | Write per-role fallback chains into `config.yml` |
|
|
111
128
|
| `fallbackChainDepth` | `2` | Fallback models per role chain |
|
|
112
129
|
| `activateDefaultOnEmptySession` | `true` | Reapply the new `default` selector to an empty session's active model |
|
|
130
|
+
| `tokenProfile` | unset | `{ inputTokens, outputTokens }` per task for the explorer's `$/task` column, used when a role has no recorded usage. Display-only; the recorded usage takes precedence. Unset = the column reads `—` |
|
|
113
131
|
| `errorReporting` | `off` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role`; one of `off`, `ask`, `auto` (see below) |
|
|
114
132
|
|
|
115
133
|
Per-role knobs (`roles.<name>.*`):
|
|
@@ -126,7 +144,7 @@ Per-role knobs (`roles.<name>.*`):
|
|
|
126
144
|
| `filters.minOutputTokens` | `0` (off) | Drop routes whose endpoint output ceiling is below this (tokens) |
|
|
127
145
|
| `filters.maxPriceUsdPerM` | `0` (off) | Drop a model whose thinking-adjusted blend exceeds this ($/M) |
|
|
128
146
|
| `thinking` | shipped level | Thinking level appended to the selector (`off`…`max`, `auto`) |
|
|
129
|
-
| `providerPin` | `""` (unset) | Pin the role's requests to one OpenRouter provider route (`@<slug>`, may be tiered like `deepinfra/fp8`). The role is then priced and gated by that route — its billed price and p50 throughput, not the `1/price²` blend — and a model with no matching route is ineligible |
|
|
147
|
+
| `providerPin` | `""` (unset) | Pin the role's requests to one OpenRouter provider route (`@<slug>`, may be tiered like `deepinfra/fp8`). The role is then priced and gated by that route — its billed price and p50 throughput, not the `1/price²` blend — and a model with no matching route is ineligible. A manual pin **overrides** the automatic pin the `providerPinning` capability would write |
|
|
130
148
|
| `cacheHitRate` | `0` (off) | Assumed cache-hit rate (0–1) for the role's input tokens; each route's effective input price becomes `h·cacheRead + (1−h)·input` before the 3:1 billed blend and the thinking factor. The `1/price²` weight basis stays the **listed** input price (the router's sort key), so a cheap cache on a lightly-weighted route cannot dominate; a route with no cache-read price keeps its full input price. `0`/absent = off (byte-identical ranking) |
|
|
131
149
|
| `lambda` | derived | Explicit λ ($ per quality point) override |
|
|
132
150
|
| `locked` | `false` | Leave the role alone: rank it but never rewrite its selector or chain (explorer toggle) |
|
|
@@ -142,7 +160,7 @@ defaults.
|
|
|
142
160
|
|---|---|---|
|
|
143
161
|
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | Routes that cannot call tools and routes whose output ceiling is below 16 384 tokens leave the pool — a reasoning model is not cut off mid-thought |
|
|
144
162
|
| `cachePricing` | `cacheHitRate: 0.5` | The role is priced on a cache-heavy agent loop's invoice (the endpoint's `cacheReadPrice` blended in) rather than the sticker price |
|
|
145
|
-
| `providerPinning` | `preferOwnProvider: true` |
|
|
163
|
+
| `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | Each request is bound to the model's **best provider route** — the max-value-for-money route (the `vfm` score, with a soft 50 tok/s floor that deprioritizes a pathologically slow route without dropping it) — the selector gains `@<slug>` — and the fallback chain carries the primary model on every gate-passing route except the pinned best, then each fallback model on every gate-passing route, so a down provider fails over to another provider of the same model before the chain switches models. The model is priced **and measured** on that route (its billed price and p50 throughput, so `q` moves too), falling back to the default blend only when it has no candidate route |
|
|
146
164
|
| `costCap` | `filters.maxPriceUsdPerM: 10` | The priciest tail of the eligible pool leaves it, so the pick cannot be an expensive outlier |
|
|
147
165
|
|
|
148
166
|
Precedence is one rule:
|
|
@@ -156,10 +174,13 @@ and otherwise when the global `features.<id>` is `true` — so a per-role `false
|
|
|
156
174
|
beats a global `true`. The preset fills only knobs the role does not already
|
|
157
175
|
set, and an explicit key is merged last, so it always wins. `providerPinning`'s
|
|
158
176
|
recommended setting is a **soft per-model policy** (`preferOwnProvider`), not a
|
|
159
|
-
hard `providerPin`: it
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
177
|
+
hard `providerPin`: it prices a model on its **best route** (the max-`vfm` route
|
|
178
|
+
— the value-for-money score, with the soft 50 tok/s floor — among the
|
|
179
|
+
filter-passing, non-degraded, priced routes the account's
|
|
180
|
+
allowed-providers whitelist permits) — the route's billed price
|
|
181
|
+
**and** its p50 throughput, so `q` moves as well as `priceEff` — and never makes
|
|
182
|
+
a model ineligible (the hard `providerPin` stays available by hand, and overrides
|
|
183
|
+
the automatic choice). `minContextTokens` is deliberately **not** part of `endpointCeilings` —
|
|
163
184
|
the complaint it would answer is an endpoint advertising a *fraction of the
|
|
164
185
|
model's own window*, which an absolute token floor cannot express, so it stays a
|
|
165
186
|
hand-set knob.
|
|
@@ -408,21 +429,31 @@ editing source and re-running the CLI.
|
|
|
408
429
|
that scope's lock file — a project scope resolves as the project lock merged
|
|
409
430
|
over the user-level lock, exactly what the updater ranks. A project whose lock
|
|
410
431
|
file is gone is shown disabled. Switching with unsaved edits asks first.
|
|
432
|
+
- **Pool toggle** — the header switches the ranked pool between **raw** (the
|
|
433
|
+
account-agnostic fetch) and **as-written** (the pool the updater actually
|
|
434
|
+
writes from: the catalog/tier gate plus your account's allowed-providers
|
|
435
|
+
whitelist). The as-written pool is what you will get, so its leader is the
|
|
436
|
+
model the next `/refresh-roles` will pick; the raw pool shows the
|
|
437
|
+
account-agnostic ranking. The toggle is best-effort (a failed whitelist fetch
|
|
438
|
+
degrades to the raw pool) and per-scope, and it never affects **Export** —
|
|
439
|
+
Export always writes the role you authored, from either pool.
|
|
411
440
|
- **Role tabs** — every role the plugin knows: the nine omp built-in roles, the
|
|
412
441
|
shipped opt-in `designer`, and any role present only in the lock file. Each
|
|
413
442
|
tab is tinted by provenance — `default` (omp built-in), `plugin` (shipped but
|
|
414
443
|
opt-in), `user` (lock-file only) — and a disabled role is muted and dashed.
|
|
415
444
|
Selecting a disabled role still ranks its models.
|
|
416
445
|
- **Rank table** — every eligible model for the selected role, with `value`,
|
|
417
|
-
`q`, `$/M` (effective, thinking-adjusted),
|
|
446
|
+
`q`, `$/M` (effective, thinking-adjusted), **`$/task`** (what you actually pay
|
|
447
|
+
per task under your own token mix — see below), `tok/s`, `ctx`, a `★` Pareto
|
|
418
448
|
marker, and a `Δ` column showing the rank delta against the role's effective
|
|
419
449
|
def (what the plugin does today).
|
|
420
450
|
- **Explain panel** — click a row for the full decomposition: each weighted
|
|
421
451
|
metric's raw value → cardinal transform → renormalized weight → contribution,
|
|
422
452
|
the cost block (`λ`, effective vs billed price, `penalty = λ·price`,
|
|
423
|
-
`value = q − penalty`), "why not higher"
|
|
424
|
-
plus the per-metric target that would close
|
|
425
|
-
it on (price, q)
|
|
453
|
+
`value = q − penalty`), the **per-task price** breakdown, "why not higher"
|
|
454
|
+
(the value gap to the model above plus the per-metric target that would close
|
|
455
|
+
it), the models that dominate it on (price, q), and the **Price levers**
|
|
456
|
+
section (below).
|
|
426
457
|
- **Weight editor** — edit the role's description, weights, `required` set,
|
|
427
458
|
`filters.image`, `thinking` level, and `λ` override live; the table re-ranks
|
|
428
459
|
on every change. Weights are edited freely (no implicit rescaling): the `Σ`
|
|
@@ -444,8 +475,30 @@ editing source and re-running the CLI.
|
|
|
444
475
|
- **Hover explanations** — every column header, metric name, role tab, and
|
|
445
476
|
control carries a tooltip explaining its semantics.
|
|
446
477
|
|
|
478
|
+
**The per-task price (`$/task`).** The `$/M` column is the reported price — the
|
|
479
|
+
billed 3:1 input:output blend OpenRouter charges, scaled by the role's thinking
|
|
480
|
+
factor. The `$/task` column is what *you* pay per task: your own input:output
|
|
481
|
+
token mix applied to the role's priced route. The mix comes from the plugin's
|
|
482
|
+
own recorded usage (the role's median per-turn counts, from the
|
|
483
|
+
`llm-role-usage.jsonl` ledger the plugin appends to as you work), falling back
|
|
484
|
+
to the `tokenProfile` setting when a role has no recorded usage. It is
|
|
485
|
+
**display-only** — the ranking still sorts on `$/M` — and it reads `—` when no
|
|
486
|
+
profile is available (no recorded usage and no `tokenProfile` set) or the model
|
|
487
|
+
has no usable route price. A model that is cheaper per token but
|
|
488
|
+
more verbose can cost more per task; this column is where that shows up.
|
|
489
|
+
|
|
490
|
+
**The price levers.** The explain panel's **Price levers** section makes the
|
|
491
|
+
price posture visible. The **leader flip** is the price weight `w*` at which a
|
|
492
|
+
cheaper challenger overtakes the current leader, with the challenger named and
|
|
493
|
+
the distance from your current weight — so you can see whether your price
|
|
494
|
+
posture binds (a small bump flips the pick) or is slack. It is absent when no
|
|
495
|
+
cheaper challenger exists. The **thinking level** table shows which model wins
|
|
496
|
+
at each level, with the level's price factor — the factor multiplies the price
|
|
497
|
+
axis, so it penalizes expensive models more, which is why `auto` favours the
|
|
498
|
+
cheap model. Both readouts are computed on the pool you are viewing.
|
|
499
|
+
|
|
447
500
|
The UI never reimplements the ranking math: the numbers on screen are exactly
|
|
448
|
-
the numbers the plugin would use. The server binds `
|
|
501
|
+
the numbers the plugin would use. The server binds `[IP_ADDRESS]` only (no auth)
|
|
449
502
|
and serves the SPA with no build step and no external requests.
|
|
450
503
|
|
|
451
504
|
## Availability
|
|
@@ -465,6 +518,13 @@ The plugin keeps only models the OpenRouter key can actually run:
|
|
|
465
518
|
whitelist** is invisible to the key endpoint and the unauthenticated catalog, and a
|
|
466
519
|
real request's 404 is the only reliable signal. Blocked candidates are excluded from
|
|
467
520
|
selection and chains; a role with no clean candidate is left untouched.
|
|
521
|
+
- **Allowed-providers whitelist** — the account's OpenRouter **allowed-providers
|
|
522
|
+
privacy whitelist** (openrouter.ai/settings/privacy) is learned once per run (a
|
|
523
|
+
canary request that bills no tokens) and every model's route pool is narrowed to
|
|
524
|
+
allowed providers before ranking, so a pin or a fallback hop is never written on a
|
|
525
|
+
provider the account cannot use. The run notes `OpenRouter allowed-providers
|
|
526
|
+
whitelist active — N providers; M routes pruned` (or `… whitelist: no whitelist`).
|
|
527
|
+
With no whitelist set — or when the harvest fails — nothing changes.
|
|
468
528
|
- **Explorer key marking** — the explorer overlays every ranked model with a
|
|
469
529
|
**usable** / **key-blocked** / **unknown** badge derived from the
|
|
470
530
|
key-authenticated `GET /api/v1/models` (the account setting **"Filter the model
|
|
@@ -485,8 +545,10 @@ The plugin keeps only models the OpenRouter key can actually run:
|
|
|
485
545
|
[docs/dev/quirks.md](docs/dev/quirks.md)).
|
|
486
546
|
- **Provider whitelist.** The account's allowed-providers whitelist blocks many
|
|
487
547
|
models, so the report's #1 and the written selector can differ — judge weights
|
|
488
|
-
on the reachable pool.
|
|
489
|
-
|
|
548
|
+
on the reachable pool. The updater prunes *routes* to the whitelist (see
|
|
549
|
+
Availability), but the report has no token and is not whitelist-aware.
|
|
550
|
+
- **Provider pinning opts out of auto-Exacto.** A `providerPin` — or the
|
|
551
|
+
automatic pin the `providerPinning` capability writes — puts a trailing
|
|
490
552
|
`@<slug>` on the selector, which omp applies as OpenRouter
|
|
491
553
|
`provider: { only: [slug] }` routing. `only` is exclusive, so a pinned role's
|
|
492
554
|
requests bypass OpenRouter's automatic Exacto tool routing (which routes tool
|
package/docs/dev/architecture.md
CHANGED
|
@@ -25,24 +25,26 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
25
25
|
|
|
26
26
|
| File | Role / key exports |
|
|
27
27
|
|---|---|
|
|
28
|
-
| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
|
|
28
|
+
| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, the route-price accessors (`routeInputPrice`, `routeEffectivePrice`, `routeOutputPrice`, `blendOutputPrice`), `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
|
|
29
29
|
| `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `parseBenchmarkPayloadMeta`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `cachedSourceInfo`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`BenchmarkPayloadMeta`/`FocusSourceInfo`/`BenchmarkCatalogEntry`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
|
|
30
|
-
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
30
|
+
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `ResolvedSettings` carries `tokenProfile` (`{ inputTokens, outputTokens } | null`, default `null`) — the user-set per-task token profile the explorer's per-task price falls back to when a role has no recorded usage; validated to finite numbers ≥ 0, schema rows `tokenProfile.inputTokens`/`tokenProfile.outputTokens`. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
31
31
|
| `src/agent-pins.ts` | Agent → pinned-role derivation. Exports `parseAgentPin` (first `@<role>` in the `model:` frontmatter) and `discoverAgentPins` (scans the shipped, user and project agent dirs; project > user > plugin). Drives the `task.disabledAgents` sync. |
|
|
32
|
-
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe,
|
|
32
|
+
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe, the keyed-catalog availability primitive, and the allowed-providers whitelist harvest. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `fetchAllowedProviders`, `providerAllowed`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
|
|
33
33
|
| `src/config-edit.ts` | Surgical line-oriented YAML patch for `modelRoles` + `retry.fallbackChains` + `task.disabledAgents`, plus the atomic writer. Exports `parseConfig`, `patchConfig`, `writeConfigAtomic`, `ConfigEditError`, and the `ConfigPatch` type. No runtime YAML dependency. |
|
|
34
34
|
| `src/state.ts` | State/history/lock files under the agent dir, and agent-dir resolution. Exports `agentDir`, `loadState`, `saveState`, `appendHistory`, `acquireLock`, `releaseLock`, `freshState`, and the `PluginState` type (which carries `managedDisabledAgents` — the `task.disabledAgents` names the plugin added). |
|
|
35
|
+
| `src/usage-ledger.ts` | The per-role usage ledger (spec #81): an append-only jsonl file (`llm-role-usage.jsonl`, cap `USAGE_LEDGER_CAP = 2000` rows) beside the plugin's other state, one row per assistant turn. Exports `UsageRow`, `TokenProfile`, `USAGE_LEDGER_FILE`, `USAGE_LEDGER_CAP`, `appendUsageRow` (best-effort, never throws), `readUsageRows` (tolerant: missing/malformed → `[]`), `roleTokenProfile` (the role's median per-turn input/output counts via `agent-create.ts`'s `quantile` — the repo's one quantile convention — or `null`), and `resolveTokenProfile` (recorded usage → the `tokenProfile` setting → `null`). The caller owns the dir; the extension records into `resolveProjectDir() ?? agentDir()`, the explorer reads the active scope's dir. |
|
|
35
36
|
| `src/error-report.ts` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role` (the `errorReporting` setting). The plugin's only GitHub client and the only home for a fingerprint, a redaction rule, the agent-dir ledger (`ERROR_REPORT_LEDGER_FILE`) and the issue body. **Entry point:** `reportUnexpectedError(input, deps)` — the feature's one test seam (per-call input + fully injected `deps`; tests supply a fake `fetch`, clock, env, dir and token resolver). Also exports `defaultErrorReportDeps()` (production deps: resolved settings, global `fetch`, real clock, `agentDir`, the shipped manifest, `gh auth token`, the OS opener) and `resetErrorReportGuard()` (the in-process dedupe guard's test hook). Never throws, never rejects; a no-op when `errorReporting` is `off`. |
|
|
36
|
-
| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
37
|
-
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
37
|
+
| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, `resolveProjectDir` (the project scope's `<cwd>/.omp` when it carries a non-empty `omp-llm-role` settings entry, else `null` — the same test the explorer's scope resolution and the extension's usage-ledger dir use), the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
38
|
+
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Records the per-role usage ledger (spec #81) on `turn_end` and `session_shutdown` — one row per assistant turn read from `sessionManager.getBranch()`, into `resolveProjectDir() ?? agentDir()`, deduped by `ts|model` so the two hooks cannot double-record. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
38
39
|
| `src/guards.ts` | The package's one type guard: `isRecord`. |
|
|
40
|
+
| `src/provider-display.ts` | The provider display (spec #77): keep the live session model's `name` suffixed with its upstream provider pin, so omp's model segment renders e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. Pure and dual-runtime (no SDK import). Exports `providerPinOf` (the single `compat.openRouterRouting.only[0]`, else `compat.vercelGatewayRouting.only[0]`; null unless exactly one non-empty string), `providerPinOfSelector` (the trailing `@<slug>` of a selector, after stripping a thinking suffix), `withProviderSuffix` (strip any previous ` @<slug>` then append, idempotent), and `applyProviderDisplay` (mutate a record's `name`; no-op otherwise). |
|
|
39
41
|
| `src/role-settings.ts` | The one validated role write path (validate → merge → backup → atomic write) and its removal half. Exports `validateRole`, `mergeExport`, `writeRoleSettings`, `mergeRemove`, `removeRoleSettings`, `timestamp`. Shared by the explorer's Export, `src/cli/create-role.ts`, `/create-agent` and `/remove-agent`. |
|
|
40
42
|
| `src/role-archetypes.ts` | Purpose → weight archetype table (10 sets) + keyword fitting. Exports `ARCHETYPES`, `FALLBACK_ARCHETYPE`, `fitArchetype`, `archetypeById`, and the `Archetype`/`ArchetypeMatch` types. |
|
|
41
43
|
| `src/agent-file.ts` | Agent `.md` rendering/placement. Exports `renderAgentFile`, `writeAgentFile`, `removeAgentFile`, `isReadOnlyTools`, `userAgentsDir`, `projectAgentsDir`, `READ_ONLY_TOOLS`, `RESERVED_AGENT_NAMES`, `AGENT_NAME_RE`, and the `AgentFileSpec`/`AgentWriteResult`/`AgentRemoveResult` types. |
|
|
42
44
|
| `src/agent-architect.ts` | omp's in-process architects: the agent-creation architect (`/create-agent`, `/agents` hub) and the project-profile architect (`/project-roles`). Exports `generateAgentSpec`, `generateProjectProfile`, `parseAgentSpec`, `parseProjectProfile`, `extractJsonObject`, `extractAssistantText`, and the `ArchitectOptions`/`ProjectArchitectOptions` types. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the runner), so the module is importable under plain Node. |
|
|
43
45
|
| `src/benchmark-author.ts` | Benchmark-source authoring for an unknown link: fetch the link, run an in-process architect (`src/prompts/benchmark-source-architect.md`) to propose a declarative source spec, and validate it. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the function). Exports `authorBenchmarkSource`, `parseSourceDeclarationJson`, and the `BenchmarkAuthorOptions` type. |
|
|
44
46
|
| `src/benchmark-discovery.ts` | Judge-backed benchmark relevance for `/create-agent` discovery: one `noul` (yes/no probability) question per catalog candidate, batched into a single judgment through omp's configured `judge` role (TypeSafe jev or its fallback chain). Extension-only (imports `@oh-my-pi/pi-coding-agent` and its `./judgment` subpath — the deeper `./judgment/standalone` path does not resolve under omp's SDK injection). The SDK is imported dynamically so a resolution failure degrades to "no discovery" instead of failing the whole extension load. Exports `judgeBenchmarkRelevance`. |
|
|
45
|
-
| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
|
|
47
|
+
| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `quantile` (the repo's one quantile convention — the focus-axis dispersion stats and the usage ledger's per-turn median), `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
|
|
46
48
|
| `src/agent-remove.ts` | `/remove-agent` core: delete an agent `.md` and its role, plus the shared flag parser and report formatter. Exports `removeAgent`, `parseRemoveAgentArgs`, `formatRemoveAgentReport`, `REMOVE_AGENT_USAGE`, and the request/result types. Refuses shipped default roles. |
|
|
47
49
|
| `src/project-setup.ts` | `/project-roles` core: a discovered `ProjectProfile` → a project-scoped role set (kept shipped roles, dropped ones, new fitted roles) written to the project plugin settings lock file, plus the project agents for the new roles. All-or-nothing (every check — names, metrics, weight math, the `--force` gate, agent-file collisions — before any write), `--dry-run`. Exports `setupProject`, `parseProjectRolesArgs`, `parseRolesSpec`, `applyProfileOverrides`, `formatProjectRolesReport`, `PROJECT_ROLES_USAGE`, and the `ProjectProfile`/`ProposedRole`/`SetupProjectOpts`/`SetupProjectResult` types (the result carries an `archetypes` map). The plan report prints the capability flags each new role will author (`features: <id>=<on>`). Dual-runtime safe (no SDK import); the extension command does the LLM work and calls it. |
|
|
48
50
|
|
|
@@ -50,9 +52,11 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
50
52
|
|
|
51
53
|
| File | Role / key exports |
|
|
52
54
|
|---|---|
|
|
53
|
-
| `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. |
|
|
54
|
-
| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`), `POST /api/
|
|
55
|
-
| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets,
|
|
55
|
+
| `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. `ExplorerBootOpts` carries the as-written pool's inputs (`tier`, `allowedProviders`, `reloadAllowedProviders`), the `tokenProfile` fallback, and `usageDir` (the user-level ledger dir; a project scope reads its own `<root>/.omp` ledger). |
|
|
56
|
+
| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`, `poolMode`, `tokenProfile`), `POST /api/scope`, `POST /api/pool` (switch the ranked pool, raw vs as-written), `POST /api/rank` (rows carry the display-only `perTask`), `POST /api/explain` (carries `leaderFlip`/`thinkingLevels`), `POST /api/export`, `POST /api/refresh`. `ExplorerOpts` gains `getPoolInputs()` and the optional `getUsageRows(scope)`/`tokenProfile` seams. |
|
|
57
|
+
| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, the nine-axis focus assessment, and the display-only per-task price. Exports `rankRows`, `explainModel`, `perTaskPrice`, `inverseCardinal`, `focusMetricsOf`, `focusAssessments`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta`/`PerTaskPrice` types. `rankRows`/`explainModel` take a trailing optional `profile?: TokenProfile \| null`; `RankRow.perTask` and `Explanation.perTask` are `null` when no profile resolves or the model has no usable route price. |
|
|
58
|
+
| `src/explorer/levers.ts` | Pure price-lever readouts (spec #81): the leader-flip threshold and the thinking level's effect on the leader, both computed on the pool they are handed. Exports `leaderFlip`, `thinkingLevelEffect`, and the `LeaderFlip`/`ThinkingLevelRow` types. Reuses the engine's `roleLambda`/`rankRole`/`SUFFIX_LEVELS`/`thinkingPriceFactor` — never re-derives the math. |
|
|
59
|
+
| `src/explorer/pool.ts` | The as-written pool (spec #81): the explorer's dataset filtered exactly as the updater filters it before ranking — the catalog/tier gate (`filterCatalog` + `resolveVariant`) and the allowed-providers whitelist prune (`providerAllowed`). Pure (never mutates its input). Exports `asWrittenPool`, and the `PoolMode`/`PoolInputs` types. |
|
|
56
60
|
|
|
57
61
|
### Assets and non-code
|
|
58
62
|
|
|
@@ -116,7 +120,22 @@ The omp extension injects the production `Deps`; tests inject fakes.
|
|
|
116
120
|
fetchKeyMeta(token)` → `tierGate` → `"billed" | "free" | "none"`; `"none"`
|
|
117
121
|
aborts.
|
|
118
122
|
5. **Catalog + probe.** `deps.getCatalog()` → `filterCatalog(catalog, tier)`;
|
|
119
|
-
`enrichThinkingLevels(rank.models, catalog)`.
|
|
123
|
+
`enrichThinkingLevels(rank.models, catalog)`.
|
|
124
|
+
**Whitelist prune.** The updater then harvests the account's
|
|
125
|
+
allowed-providers privacy whitelist once per run
|
|
126
|
+
(`deps.getAllowedProviders ? deps.getAllowedProviders(token, canaryModelId)
|
|
127
|
+
: fetchAllowedProviders(token, canaryModelId)`, a canary 404 that bills no
|
|
128
|
+
tokens) and, when it is non-null, replaces every model's route pool with the
|
|
129
|
+
routes whose `providerSlug` matches a whitelist entry (`providerAllowed`,
|
|
130
|
+
prefix match `slug === entry || slug.startsWith(entry + "/")`,
|
|
131
|
+
non-mutating). The prune is
|
|
132
|
+
the single change point — `roleRoute`, `bestRoute`, `rankedRoutes` and
|
|
133
|
+
`endpointFilteredModel` all read `m.routes` — so the soft basis, the auto-pin
|
|
134
|
+
and the fallback chains all become whitelist-aware. A `null` whitelist (no
|
|
135
|
+
whitelist set, harvest failed) leaves every pool untouched. The run notes
|
|
136
|
+
`OpenRouter allowed-providers whitelist active — N providers; M routes pruned`
|
|
137
|
+
(or `… whitelist: no whitelist`).
|
|
138
|
+
Per role, candidates are
|
|
120
139
|
`resolveVariant`-resolved and then verified by a bounded `probeModel` walk
|
|
121
140
|
(current selector first, then rank order; budget `PROBE_BUDGET = 12`; verdicts
|
|
122
141
|
cached per run). Only the narrow no-allowed-providers 404 disqualifies; every
|
|
@@ -202,6 +221,7 @@ type Deps = {
|
|
|
202
221
|
getSettings?(): Promise<Record<string, unknown>>;
|
|
203
222
|
getKeyMeta?(token: string): Promise<KeyMeta>;
|
|
204
223
|
probeModel?(token: string, catalogId: string): Promise<ProbeVerdict>;
|
|
224
|
+
getAllowedProviders?(token: string, modelId: string): Promise<Set<string> | null>;
|
|
205
225
|
// Host coupling (extension only): apply a concrete selector to the live session model.
|
|
206
226
|
applySessionModel?(selector: string, previous: string | null): Promise<void>;
|
|
207
227
|
};
|
|
@@ -221,7 +241,11 @@ before `catalogFromOmpModelsJson`.
|
|
|
221
241
|
`startExplorer`), `/create-agent` (architect → `createAgent` → in-process
|
|
222
242
|
`runUpdater`), `/remove-agent` (`removeAgent` → in-process `runUpdater`),
|
|
223
243
|
`/project-roles` (profile architect → `setupProject` → in-process
|
|
224
|
-
`runUpdater`), `session_shutdown` (closes the explorer handle).
|
|
244
|
+
`runUpdater`), `session_shutdown` (closes the explorer handle). The provider
|
|
245
|
+
display adds four event subscriptions — `session_start`,
|
|
246
|
+
`retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` — each
|
|
247
|
+
applying the live model's provider pin to its `name` (see
|
|
248
|
+
[`quirks.md`](quirks.md) for why `turn_start` is the self-healing hook).
|
|
225
249
|
- **CLI**: `src/cli/llm-role-rank.ts` (report), `src/cli/create-role.ts`
|
|
226
250
|
(authoring). The updater and explorer have no CLI shim — they run through
|
|
227
251
|
`/refresh-roles` and `/explore-roles`.
|
package/docs/dev/ci.md
CHANGED
|
@@ -21,6 +21,15 @@ ref must be green first. `test` runs **exactly** the command the local
|
|
|
21
21
|
pre-commit hook runs (`.githooks/pre-commit`), and `tests/ci.test.ts` fails when
|
|
22
22
|
the two drift apart — "green locally" and "green in CI" cannot diverge.
|
|
23
23
|
|
|
24
|
+
The hook scrubs `GIT_DIR`, `GIT_WORK_TREE`, `GIT_INDEX_FILE` and
|
|
25
|
+
`GIT_OBJECT_DIRECTORY` before running the suite, and `tests/ci.test.ts` scrubs
|
|
26
|
+
the same variables from every child process it spawns. Git exports `GIT_DIR` to
|
|
27
|
+
every hook, and `GIT_DIR` overrides `cwd` — so without the scrub the test file's
|
|
28
|
+
temp-repo `git init`/`config`/`add`/`commit` calls target the **real**
|
|
29
|
+
repository, rewriting its config and moving a branch (issue #89). The test file
|
|
30
|
+
carries the scrub because a test that shells out to git must not depend on its
|
|
31
|
+
caller's environment; the hook carries it as defence in depth.
|
|
32
|
+
|
|
24
33
|
The Node version comes from `.node-version` at the repo root (the exact local
|
|
25
34
|
toolchain; bumping it is a normal commit). `engines` is deliberately not in
|
|
26
35
|
`package.json`: the plugin executes inside omp's runtime, so an `engines` field
|
|
@@ -141,10 +150,21 @@ exit 1, so the command is safe to run unattended.
|
|
|
141
150
|
is that version's CHANGELOG section. This is also the recovery path for a
|
|
142
151
|
failed publish: the tag exists, so no new version is needed.
|
|
143
152
|
|
|
153
|
+
A successful `npm publish` is **not** immediately visible: npm prints "Your
|
|
154
|
+
package is being processed and may take a few minutes to become available",
|
|
155
|
+
and registry reads can serve the pre-publish packument for several minutes
|
|
156
|
+
(third-party mirrors such as jsDelivr lag longer). The step's log is the
|
|
157
|
+
discriminator — a direct publish prints `publishing to
|
|
158
|
+
https://registry.npmjs.org/ with tag latest`, `Signed provenance statement`
|
|
159
|
+
and `+ <pkg>@<version>`; a staged publish is a separate `npm stage publish`
|
|
160
|
+
and prints none of those.
|
|
161
|
+
|
|
144
162
|
The dispatch resolves the workflow **from the ref**, so the tag's commit must
|
|
145
163
|
contain `.github/workflows/ci.yml`. A tag cut before this workflow existed
|
|
146
164
|
cannot be dispatched (`HTTP 422: Workflow does not have 'workflow_dispatch'
|
|
147
|
-
trigger`) — the recovery path only works for tags cut after it landed.
|
|
165
|
+
trigger`) — the recovery path only works for tags cut after it landed. The
|
|
166
|
+
workaround is to repoint the tag at a commit that contains the workflow (done
|
|
167
|
+
once, for `v1.1.0` → `4f78ce2`).
|
|
148
168
|
|
|
149
169
|
`tests/ci.test.ts` asserts the offline half: the workflow's structure (jobs,
|
|
150
170
|
triggers, permissions, the command it runs), the hook/workflow command parity,
|
|
@@ -1,15 +1,80 @@
|
|
|
1
|
-
# Current state (2026-10-
|
|
1
|
+
# Current state (2026-10-18)
|
|
2
2
|
|
|
3
3
|
Dated snapshot of the ranking and the landed work as of the date in the heading —
|
|
4
4
|
matched/priced/eligible counts, per-role leaders, the open defects, and the
|
|
5
5
|
per-issue work log below. It is refreshed by
|
|
6
6
|
regenerating the report (`node src/cli/llm-role-rank.ts --out docs/llm-role-rankings.md`) and
|
|
7
7
|
copying the new numbers here; release prose belongs in
|
|
8
|
-
[`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The
|
|
9
|
-
below are the 2026-10-
|
|
10
|
-
#17/#18/#19/#20 wave
|
|
11
|
-
|
|
8
|
+
[`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The model-level
|
|
9
|
+
ranking numbers below are the 2026-10-18 measurement (the input-price blend basis
|
|
10
|
+
#21, the #17/#18/#19/#20 wave, the #34–#40 focus-axis/capability-flag work, and
|
|
11
|
+
spec #81's explorer surfaces — the ranking CLI itself is unchanged); the
|
|
12
|
+
provider-pin numbers are the 2026-10-11 measurement (spec #71; see the top
|
|
13
|
+
bullet). Neither is re-derived on read.
|
|
12
14
|
|
|
15
|
+
- Explorer as-written pool, per-task price, and price levers (2026-10-18, spec
|
|
16
|
+
#81): the explorer gains a **pool toggle** (raw vs as-written), a **per-task
|
|
17
|
+
price column**, and two **price-lever readouts**. The as-written pool applies
|
|
18
|
+
the same two filters the updater applies before ranking — the catalog/tier gate
|
|
19
|
+
and the allowed-providers whitelist prune — so the leader the user sees is the
|
|
20
|
+
model the updater will write (the raw pool's leader differed from the config's
|
|
21
|
+
for 6 of 10 roles). The per-task price is `inputTokens·p_in +
|
|
22
|
+
outputTokens·p_out·thinkingFactor` over the role's priced route, using the
|
|
23
|
+
plugin's own recorded per-role usage (the median per-turn counts) or the
|
|
24
|
+
`tokenProfile` setting; it is **display-only** — the price axis stays `$/M`
|
|
25
|
+
(issue #32's verdict stands) and the reported `$/M` keeps its billed 3:1 blend.
|
|
26
|
+
The readouts are the leader-flip threshold (`λ* = min over cheaper challengers
|
|
27
|
+
X of (q_L − q_X)/(pEff_L − pEff_X)`, `w* = 20λ*/(1+20λ*)`) and the thinking
|
|
28
|
+
level's effect on the leader. A new append-only, capped, best-effort
|
|
29
|
+
**usage ledger** (`llm-role-usage.jsonl`, one row per assistant turn) follows
|
|
30
|
+
the plugin's state dir (`<project>/.omp` in project mode, `agentDir()`
|
|
31
|
+
otherwise). The ranking numbers are unchanged — the ranking CLI does not apply
|
|
32
|
+
the pool toggle, and the per-task price never feeds the sort key.
|
|
33
|
+
- Provider display (2026-10-12, spec #77): when the live session model carries a
|
|
34
|
+
provider pin (a hand-set `providerPin`, or the automatic pin the
|
|
35
|
+
`providerPinning` capability writes), the plugin keeps the model object's
|
|
36
|
+
`name` suffixed with the route, so omp's status-line model segment renders
|
|
37
|
+
e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. The suffix is applied on session
|
|
38
|
+
start, refreshed on the two fallback events, and re-derived at every
|
|
39
|
+
`turn_start` — the re-derive is what heals the display after omp silently
|
|
40
|
+
restores the primary model following a fallback cooldown (no session event is
|
|
41
|
+
emitted), and it also covers a manual `/model` change with a one-turn lag. An
|
|
42
|
+
event's selector is trusted only when it names the live session model, so an
|
|
43
|
+
advisor fallback (the same events, the advisor's model) re-derives from the
|
|
44
|
+
live model instead of stamping the advisor's pin onto the session.
|
|
45
|
+
Display-only: the suffix rides the model object's `name`, so attribution,
|
|
46
|
+
usage recording and model comparison are unaffected. The ranking numbers are
|
|
47
|
+
unchanged.
|
|
48
|
+
|
|
49
|
+
- Provider ordering: value-for-money route score (2026-10-11, spec #71): the
|
|
50
|
+
auto-pin route basis changed from the route's own value (`q_route −
|
|
51
|
+
λ·priceEff_route`) to a min/max-normalized **value-for-money score** `vfm =
|
|
52
|
+
wT·tNorm − wP·pNorm` (`wT = wTput/(wPrice+wTput)`, `wP =
|
|
53
|
+
wPrice/(wPrice+wTput)`; `tNorm`/`pNorm` min/max over the candidate set), with a
|
|
54
|
+
new **soft** `filters.preferredMinThroughput` floor (routes at or above sort
|
|
55
|
+
first; below-floor routes are kept and sorted last) that the `providerPinning`
|
|
56
|
+
preset sets to 50. `routeValue` is deleted. The model-level ranking is
|
|
57
|
+
unchanged — the report is not regenerated (the ranking CLI does not apply
|
|
58
|
+
capability features and no shipped role sets `preferOwnProvider`). Measured
|
|
59
|
+
2026-10-11 on the current endpoints cache with the live lock's expanded defs
|
|
60
|
+
(all four capabilities on, `cacheHitRate 0.9`; `deepseek/deepseek-v4.1-flash`,
|
|
61
|
+
11 allowed providers, 12 allowed routes):
|
|
62
|
+
`morph/fp8` ($0.0840,
|
|
63
|
+
103 tok/s) wins `default`, `smol`, `commit`, `task`, `advisor`; `coreweave/fp8`
|
|
64
|
+
($0.1978, 185.5 tok/s) wins `slow`, `tiny`, `designer`; `decart/fp4` ($0.0639,
|
|
65
|
+
72 tok/s) wins `vision`, `plan`.
|
|
66
|
+
|
|
67
|
+
- Auto-pin chain expansion (2026-10-10, issue #69): with `providerPinning` on,
|
|
68
|
+
the fallback chain no longer stops at three providers per model — the primary
|
|
69
|
+
model contributes every gate-passing route except the pinned best, then each
|
|
70
|
+
`fallbackChainDepth` fallback model contributes every gate-passing route, in
|
|
71
|
+
score order (the `vfm` route score — spec #71 replaced the original
|
|
72
|
+
route-value order) and deduped by provider slug, so a down provider fails over
|
|
73
|
+
to another provider of the same model before the chain switches models. Chain
|
|
74
|
+
length is data-driven (`(R_primary − 1) + Σ R_fallback`) instead of the fixed
|
|
75
|
+
`2 + 3·fallbackChainDepth`; the manual-`providerPin` and no-pin paths are
|
|
76
|
+
unchanged. The ranking numbers are unchanged — the chain is a write-side
|
|
77
|
+
shape, not a ranking input.
|
|
13
78
|
- Opt-in error reporting (2026-10-08, issue #48): a new `errorReporting` setting
|
|
14
79
|
(`off` — the default — `ask`, `auto`) turns a genuinely unexpected failure — a
|
|
15
80
|
command handler throwing, or a `runUpdater` abort whose `RunResult.defect` is
|
|
@@ -73,26 +138,33 @@ bullets) and are not re-derived on read.
|
|
|
73
138
|
`src/features.ts`) is the single source of truth and `expandFeatures` the
|
|
74
139
|
single precedence implementation (`effective def = shipped default ⊕
|
|
75
140
|
preset(enabled flags) ⊕ explicit user keys`; a per-role `false` beats a global
|
|
76
|
-
`true`; an explicit knob always wins). Grounding on the 2026-10-
|
|
77
|
-
(`default` role,
|
|
141
|
+
`true`; an explicit knob always wins). Grounding on the 2026-10-09 cache
|
|
142
|
+
(`default` role, 146 eligible; the `providerPinning` row is re-measured on the
|
|
143
|
+
2026-10-11 cache):
|
|
78
144
|
|
|
79
145
|
| Flag | Recommended settings | Measured effect |
|
|
80
146
|
|---|---|---|
|
|
81
|
-
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` |
|
|
82
|
-
| `cachePricing` | `cacheHitRate: 0.5` |
|
|
83
|
-
| `providerPinning` | `preferOwnProvider: true` | per-model policy, never changes eligibility;
|
|
84
|
-
| `costCap` | `filters.maxPriceUsdPerM: 10` |
|
|
147
|
+
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | 146 → 135 eligible, leader unchanged (`deepseek-v4.1-flash`, 0.37 $/M) |
|
|
148
|
+
| `cachePricing` | `cacheHitRate: 0.5` | 146 eligible, leader unchanged, its `priceEff` 0.3732 → 0.3522 (−5.6 %) |
|
|
149
|
+
| `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | per-model policy, never changes eligibility; prices each model on its **best route** (max `vfm` among the filter-passing, non-degraded, priced routes, cap-respecting; the soft 50 tok/s floor sorts below-floor routes last; ties by slug then id) — 144 of 144 routed models have a candidate route (2026-10-11), so the soft basis covers the whole routed field |
|
|
150
|
+
| `costCap` | `filters.maxPriceUsdPerM: 10` | 146 → 137 eligible (the top ≈6 %), leader unchanged; pool p90 7.8, p95 11.1 $/M, every leader 0.20–3.71 $/M |
|
|
85
151
|
|
|
86
152
|
`providerPinning`'s recommended setting is a **soft per-model policy**
|
|
87
153
|
(`preferOwnProvider`), not a hard `providerPin`: a hard pin would make a model
|
|
88
|
-
with no matching route ineligible and empty
|
|
89
|
-
policy
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
`
|
|
94
|
-
|
|
95
|
-
|
|
154
|
+
with no matching route ineligible and empty most of a role's pool, so the
|
|
155
|
+
policy prices a model on its **best route** — the max-`vfm` route (the
|
|
156
|
+
value-for-money score, with the soft 50 tok/s floor sorting below-floor routes
|
|
157
|
+
last) among the filter-passing, non-degraded, priced routes (thinking-adjusted
|
|
158
|
+
price ≤ `maxPriceUsdPerM`; ties by slug then id) — and falls back to the
|
|
159
|
+
`1/price²` blend only when the model has no candidate route. The basis is soft:
|
|
160
|
+
it never changes eligibility, and on the 2026-10-11 cache every one of the 144
|
|
161
|
+
routed models has a candidate route, so the capability covers the whole routed
|
|
162
|
+
field (the old own-lab slug-prefix rule matched only 51 of 149). `minContextTokens`
|
|
163
|
+
is deliberately not in the `endpointCeilings` bundle (an absolute floor cannot
|
|
164
|
+
express the video's fraction-of-the-model's-own-window complaint). The flags
|
|
165
|
+
are settable from `create-role.ts`, `/create-agent`, `/project-roles`
|
|
166
|
+
(`--feature`/`--list-features`) and the explorer's Features panel; the ranking
|
|
167
|
+
numbers are unchanged until a flag is turned on.
|
|
96
168
|
|
|
97
169
|
- Cross-source axis (2026-10-07, issue #38): the focus-metric gate gained the
|
|
98
170
|
cross-source axis — the zeroeval per-benchmark payload's **second, independent**
|
|
@@ -616,18 +688,18 @@ bullets) and are not re-derived on read.
|
|
|
616
688
|
0.663, margin 0.039 > `switchMargin`), because the incumbent has no Design
|
|
617
689
|
Arena data and takes the neutral fill while MiMo carries a measured 0.97.
|
|
618
690
|
The other nine roles are untouched (only `designer` weights `website`).
|
|
619
|
-
-
|
|
620
|
-
2026-10-
|
|
691
|
+
- 408 llm-stats models; OpenRouter matched 149/408 (throughput), 148 priced (the
|
|
692
|
+
2026-10-09 snapshot read 155/408 — the daily cache moved, not the code; the
|
|
621
693
|
#34–#40 axes and flags leave every ranking byte-identical until a flag is
|
|
622
694
|
turned on).
|
|
623
|
-
- Eligible per role:
|
|
695
|
+
- Eligible per role: 140 (vision 73, designer 87, image-input filter).
|
|
624
696
|
- Value-ranking leaders (this report, thinking-adjusted prices; the *ranking*
|
|
625
697
|
leaders, before the account's provider whitelist drops the blocked ones):
|
|
626
|
-
`default` DeepSeek-V4.1-Flash (0.
|
|
627
|
-
`slow`
|
|
628
|
-
Muse Spark 1.3 (0.
|
|
629
|
-
Mercury 2 (0.797), `task` DeepSeek-V4.1-Flash (0.
|
|
630
|
-
Muse Spark 1.3 (0.
|
|
698
|
+
`default` DeepSeek-V4.1-Flash (0.807), `smol` DeepSeek-V4.1-Flash (0.706),
|
|
699
|
+
`slow` Gemini 4 Argon (0.834), `vision` Gemini 4 Argon (0.732),
|
|
700
|
+
`plan` Muse Spark 1.3 (0.789), `commit` DeepSeek-V4.1-Flash (0.736), `tiny`
|
|
701
|
+
Mercury 2 (0.797), `task` DeepSeek-V4.1-Flash (0.750), `advisor`
|
|
702
|
+
Muse Spark 1.3 (0.785), `designer` DeepSeek-V4.1-Flash (0.772).
|
|
631
703
|
- Thinking-adjusted pricing landed (2026-09-27): the suffix table moved into
|
|
632
704
|
`DEFAULT_ROLES` as a per-role `thinking` field, and the price axis scales by
|
|
633
705
|
the level's factor for thinking-capable models — `slow` (`:max`, ×7.86)
|
package/docs/dev/data-sources.md
CHANGED
|
@@ -159,8 +159,11 @@ probability proportional to `1/price²`). The weight basis is the **input
|
|
|
159
159
|
|
|
160
160
|
A role with a `providerPin` does **not** use this blend: it prices the
|
|
161
161
|
model's matching route directly — the route's billed 3:1 price and its p50
|
|
162
|
-
throughput (scoring.md, *Route-aware pricing*).
|
|
163
|
-
|
|
162
|
+
throughput (scoring.md, *Route-aware pricing*). A role with the soft
|
|
163
|
+
`preferOwnProvider` policy (the `providerPinning` capability) likewise prices
|
|
164
|
+
each model on its **best route** rather than the blend, falling back to the blend
|
|
165
|
+
only when the model has no candidate route (scoring.md, *Capability presets*).
|
|
166
|
+
The blend is the default for every other model.
|
|
164
167
|
|
|
165
168
|
## OpenRouter — key metadata (tier gate)
|
|
166
169
|
|
|
@@ -180,8 +183,8 @@ freeUsable = free_model_daily_requests.remaining > 0
|
|
|
180
183
|
billed → billed variants only (exclude `:free`, `:batch`); else free → `:free`
|
|
181
184
|
variants only (exclude `:batch`); else `"none"` aborts the run. The key endpoint
|
|
182
185
|
carries **no** model/provider restriction field — the account's allowed-providers
|
|
183
|
-
whitelist is invisible to it (see the
|
|
184
|
-
[`architecture.md`](architecture.md)).
|
|
186
|
+
whitelist is invisible to it (see the allowed-providers whitelist below and the
|
|
187
|
+
probe in [`architecture.md`](architecture.md)).
|
|
185
188
|
|
|
186
189
|
## OpenRouter — keyed catalog (availability)
|
|
187
190
|
|
|
@@ -202,6 +205,32 @@ makes the Bearer-authenticated catalog a per-key allowlist.
|
|
|
202
205
|
every consumer re-fetches, and `fetchKeyAvailability` **never throws** (a failure
|
|
203
206
|
degrades to `unavailable` with empty sets, so the probe walk / `unknown` marks stand).
|
|
204
207
|
|
|
208
|
+
## OpenRouter — allowed-providers whitelist (availability)
|
|
209
|
+
|
|
210
|
+
The account's **allowed-providers privacy whitelist**
|
|
211
|
+
(openrouter.ai/settings/privacy) restricts which provider routes a request may
|
|
212
|
+
use. It is invisible to `/api/v1/key` and to the catalog endpoints; the only
|
|
213
|
+
place it appears is the 404 body of a request that fails because of it.
|
|
214
|
+
|
|
215
|
+
- **URL**: `POST https://openrouter.ai/api/v1/chat/completions` with
|
|
216
|
+
`provider.only: ["__omp-llm-role-canary__"]` (a nonexistent provider slug) on a
|
|
217
|
+
valid catalog model. `fetchAllowedProviders(token, modelId, fetchImpl?)`
|
|
218
|
+
(`src/availability.ts`) issues it.
|
|
219
|
+
- **Signal**: the 404 body names the whitelist after "permits only:" — the
|
|
220
|
+
message is parsed into a `Set<string>` of bare lab names (`coreweave`,
|
|
221
|
+
`baseten`, …). A non-404 response, a body without the clause (no whitelist
|
|
222
|
+
set), or any transport/shape failure returns `null`. The canary 404s, so it
|
|
223
|
+
bills no tokens.
|
|
224
|
+
- **Match rule**: a route's `providerSlug` is allowed when it equals a whitelist
|
|
225
|
+
entry or starts with `<entry>/` — the whitelist carries bare lab names while
|
|
226
|
+
route slugs are tiered (`coreweave/fp8`, `baseten/fp8`).
|
|
227
|
+
- **Consumer**: the updater harvests it **once per run** (with a valid catalog
|
|
228
|
+
model id) and prunes every model's route pool to allowed providers before
|
|
229
|
+
ranking (architecture.md, *Data flow*). It is **not cached to disk** — every
|
|
230
|
+
run re-learns it, so a privacy-setting change takes effect on the next refresh
|
|
231
|
+
— and `fetchAllowedProviders` **never throws** (a failure degrades to `null`,
|
|
232
|
+
which leaves every route pool untouched).
|
|
233
|
+
|
|
205
234
|
## Design Arena — design quality
|
|
206
235
|
|
|
207
236
|
Two routes, never averaged (same Elo family; mean diff −1.0 website … −5.1 svg,
|