omp-llm-role 1.1.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -48,11 +48,28 @@ omp plugin install omp-llm-role@gott50-plugins
48
48
  session actually booted on the previous `default` selector (an explicit
49
49
  `omp --model X` is never clobbered), and only on a real change. Disable with
50
50
  `activateDefaultOnEmptySession=false`.
51
+ - **Provider in the model display.** When the live session model carries a
52
+ provider pin — a hand-set `providerPin`, or the automatic pin the
53
+ `providerPinning` capability writes — the status line's model segment shows
54
+ the route next to the model name, e.g. `DeepSeek V4.1 Flash @coreweave/fp8`.
55
+ The suffix is applied on session start, refreshed when a fallback switches the
56
+ model, and re-derived at the start of every turn, so it self-heals after omp
57
+ silently restores the primary model following a fallback cooldown. An unpinned
58
+ model's display is unchanged, and the suffix is display-only: it rides the
59
+ model object's `name`, so attribution, usage recording and model comparison
60
+ are unaffected.
51
61
  - **Surgical writes.** Only managed role lines, managed chain keys, and the
52
62
  plugin-managed `task.disabledAgents` names change; comments, blank lines and
53
63
  unknown keys stay byte-identical, and a no-change run writes nothing.
54
64
  `~/.omp/agent/llm-role-state.json` snapshots the previous `modelRoles` block
55
65
  on every write as a rollback aid.
66
+ - **Usage ledger.** The plugin appends one row per assistant turn (the model,
67
+ the role, the provider, the token counts and the cost) to
68
+ `llm-role-usage.jsonl` beside its other state — `<project>/.omp` in project
69
+ mode, `~/.omp/agent` otherwise. It is append-only, capped at the newest 2000
70
+ rows, and best-effort (a recording failure never breaks a session). The
71
+ explorer's `$/task` column reads it to learn your own token mix; nothing else
72
+ uses it.
56
73
 
57
74
  ## Roles
58
75
 
@@ -110,6 +127,7 @@ Global knobs:
110
127
  | `writeFallbackChains` | `true` | Write per-role fallback chains into `config.yml` |
111
128
  | `fallbackChainDepth` | `2` | Fallback models per role chain |
112
129
  | `activateDefaultOnEmptySession` | `true` | Reapply the new `default` selector to an empty session's active model |
130
+ | `tokenProfile` | unset | `{ inputTokens, outputTokens }` per task for the explorer's `$/task` column, used when a role has no recorded usage. Display-only; the recorded usage takes precedence. Unset = the column reads `—` |
113
131
  | `errorReporting` | `off` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role`; one of `off`, `ask`, `auto` (see below) |
114
132
 
115
133
  Per-role knobs (`roles.<name>.*`):
@@ -126,7 +144,7 @@ Per-role knobs (`roles.<name>.*`):
126
144
  | `filters.minOutputTokens` | `0` (off) | Drop routes whose endpoint output ceiling is below this (tokens) |
127
145
  | `filters.maxPriceUsdPerM` | `0` (off) | Drop a model whose thinking-adjusted blend exceeds this ($/M) |
128
146
  | `thinking` | shipped level | Thinking level appended to the selector (`off`…`max`, `auto`) |
129
- | `providerPin` | `""` (unset) | Pin the role's requests to one OpenRouter provider route (`@<slug>`, may be tiered like `deepinfra/fp8`). The role is then priced and gated by that route — its billed price and p50 throughput, not the `1/price²` blend — and a model with no matching route is ineligible |
147
+ | `providerPin` | `""` (unset) | Pin the role's requests to one OpenRouter provider route (`@<slug>`, may be tiered like `deepinfra/fp8`). The role is then priced and gated by that route — its billed price and p50 throughput, not the `1/price²` blend — and a model with no matching route is ineligible. A manual pin **overrides** the automatic pin the `providerPinning` capability would write |
130
148
  | `cacheHitRate` | `0` (off) | Assumed cache-hit rate (0–1) for the role's input tokens; each route's effective input price becomes `h·cacheRead + (1−h)·input` before the 3:1 billed blend and the thinking factor. The `1/price²` weight basis stays the **listed** input price (the router's sort key), so a cheap cache on a lightly-weighted route cannot dominate; a route with no cache-read price keeps its full input price. `0`/absent = off (byte-identical ranking) |
131
149
  | `lambda` | derived | Explicit λ ($ per quality point) override |
132
150
  | `locked` | `false` | Leave the role alone: rank it but never rewrite its selector or chain (explorer toggle) |
@@ -142,7 +160,7 @@ defaults.
142
160
  |---|---|---|
143
161
  | `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | Routes that cannot call tools and routes whose output ceiling is below 16 384 tokens leave the pool — a reasoning model is not cut off mid-thought |
144
162
  | `cachePricing` | `cacheHitRate: 0.5` | The role is priced on a cache-heavy agent loop's invoice (the endpoint's `cacheReadPrice` blended in) rather than the sticker price |
145
- | `providerPinning` | `preferOwnProvider: true` | A model is priced **and measured** on its own lab's route where one exists (DeepSeek's own endpoint, 94.5 % cache hit) — the route's billed price and its p50 throughput, so `q` moves too — falling back to the default blend otherwise |
163
+ | `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | Each request is bound to the model's **best provider route** — the max-value-for-money route (the `vfm` score, with a soft 50 tok/s floor that deprioritizes a pathologically slow route without dropping it) — the selector gains `@<slug>` — and the fallback chain carries the primary model on every gate-passing route except the pinned best, then each fallback model on every gate-passing route, so a down provider fails over to another provider of the same model before the chain switches models. The model is priced **and measured** on that route (its billed price and p50 throughput, so `q` moves too), falling back to the default blend only when it has no candidate route |
146
164
  | `costCap` | `filters.maxPriceUsdPerM: 10` | The priciest tail of the eligible pool leaves it, so the pick cannot be an expensive outlier |
147
165
 
148
166
  Precedence is one rule:
@@ -156,10 +174,13 @@ and otherwise when the global `features.<id>` is `true` — so a per-role `false
156
174
  beats a global `true`. The preset fills only knobs the role does not already
157
175
  set, and an explicit key is merged last, so it always wins. `providerPinning`'s
158
176
  recommended setting is a **soft per-model policy** (`preferOwnProvider`), not a
159
- hard `providerPin`: it re-prices a model on its own lab's route where one exists
160
- (the route's billed price **and** its p50 throughput, so `q` moves as well as
161
- `priceEff`) and never makes a model ineligible (the hard `providerPin` stays
162
- available by hand). `minContextTokens` is deliberately **not** part of `endpointCeilings` —
177
+ hard `providerPin`: it prices a model on its **best route** (the max-`vfm` route
178
+ — the value-for-money score, with the soft 50 tok/s floor — among the
179
+ filter-passing, non-degraded, priced routes the account's
180
+ allowed-providers whitelist permits) — the route's billed price
181
+ **and** its p50 throughput, so `q` moves as well as `priceEff` — and never makes
182
+ a model ineligible (the hard `providerPin` stays available by hand, and overrides
183
+ the automatic choice). `minContextTokens` is deliberately **not** part of `endpointCeilings` —
163
184
  the complaint it would answer is an endpoint advertising a *fraction of the
164
185
  model's own window*, which an absolute token floor cannot express, so it stays a
165
186
  hand-set knob.
@@ -408,21 +429,31 @@ editing source and re-running the CLI.
408
429
  that scope's lock file — a project scope resolves as the project lock merged
409
430
  over the user-level lock, exactly what the updater ranks. A project whose lock
410
431
  file is gone is shown disabled. Switching with unsaved edits asks first.
432
+ - **Pool toggle** — the header switches the ranked pool between **raw** (the
433
+ account-agnostic fetch) and **as-written** (the pool the updater actually
434
+ writes from: the catalog/tier gate plus your account's allowed-providers
435
+ whitelist). The as-written pool is what you will get, so its leader is the
436
+ model the next `/refresh-roles` will pick; the raw pool shows the
437
+ account-agnostic ranking. The toggle is best-effort (a failed whitelist fetch
438
+ degrades to the raw pool) and per-scope, and it never affects **Export** —
439
+ Export always writes the role you authored, from either pool.
411
440
  - **Role tabs** — every role the plugin knows: the nine omp built-in roles, the
412
441
  shipped opt-in `designer`, and any role present only in the lock file. Each
413
442
  tab is tinted by provenance — `default` (omp built-in), `plugin` (shipped but
414
443
  opt-in), `user` (lock-file only) — and a disabled role is muted and dashed.
415
444
  Selecting a disabled role still ranks its models.
416
445
  - **Rank table** — every eligible model for the selected role, with `value`,
417
- `q`, `$/M` (effective, thinking-adjusted), `tok/s`, `ctx`, a `★` Pareto
446
+ `q`, `$/M` (effective, thinking-adjusted), **`$/task`** (what you actually pay
447
+ per task under your own token mix — see below), `tok/s`, `ctx`, a `★` Pareto
418
448
  marker, and a `Δ` column showing the rank delta against the role's effective
419
449
  def (what the plugin does today).
420
450
  - **Explain panel** — click a row for the full decomposition: each weighted
421
451
  metric's raw value → cardinal transform → renormalized weight → contribution,
422
452
  the cost block (`λ`, effective vs billed price, `penalty = λ·price`,
423
- `value = q − penalty`), "why not higher" (the value gap to the model above
424
- plus the per-metric target that would close it), and the models that dominate
425
- it on (price, q).
453
+ `value = q − penalty`), the **per-task price** breakdown, "why not higher"
454
+ (the value gap to the model above plus the per-metric target that would close
455
+ it), the models that dominate it on (price, q), and the **Price levers**
456
+ section (below).
426
457
  - **Weight editor** — edit the role's description, weights, `required` set,
427
458
  `filters.image`, `thinking` level, and `λ` override live; the table re-ranks
428
459
  on every change. Weights are edited freely (no implicit rescaling): the `Σ`
@@ -444,8 +475,30 @@ editing source and re-running the CLI.
444
475
  - **Hover explanations** — every column header, metric name, role tab, and
445
476
  control carries a tooltip explaining its semantics.
446
477
 
478
+ **The per-task price (`$/task`).** The `$/M` column is the reported price — the
479
+ billed 3:1 input:output blend OpenRouter charges, scaled by the role's thinking
480
+ factor. The `$/task` column is what *you* pay per task: your own input:output
481
+ token mix applied to the role's priced route. The mix comes from the plugin's
482
+ own recorded usage (the role's median per-turn counts, from the
483
+ `llm-role-usage.jsonl` ledger the plugin appends to as you work), falling back
484
+ to the `tokenProfile` setting when a role has no recorded usage. It is
485
+ **display-only** — the ranking still sorts on `$/M` — and it reads `—` when no
486
+ profile is available (no recorded usage and no `tokenProfile` set) or the model
487
+ has no usable route price. A model that is cheaper per token but
488
+ more verbose can cost more per task; this column is where that shows up.
489
+
490
+ **The price levers.** The explain panel's **Price levers** section makes the
491
+ price posture visible. The **leader flip** is the price weight `w*` at which a
492
+ cheaper challenger overtakes the current leader, with the challenger named and
493
+ the distance from your current weight — so you can see whether your price
494
+ posture binds (a small bump flips the pick) or is slack. It is absent when no
495
+ cheaper challenger exists. The **thinking level** table shows which model wins
496
+ at each level, with the level's price factor — the factor multiplies the price
497
+ axis, so it penalizes expensive models more, which is why `auto` favours the
498
+ cheap model. Both readouts are computed on the pool you are viewing.
499
+
447
500
  The UI never reimplements the ranking math: the numbers on screen are exactly
448
- the numbers the plugin would use. The server binds `127.0.0.1` only (no auth)
501
+ the numbers the plugin would use. The server binds `[IP_ADDRESS]` only (no auth)
449
502
  and serves the SPA with no build step and no external requests.
450
503
 
451
504
  ## Availability
@@ -465,6 +518,13 @@ The plugin keeps only models the OpenRouter key can actually run:
465
518
  whitelist** is invisible to the key endpoint and the unauthenticated catalog, and a
466
519
  real request's 404 is the only reliable signal. Blocked candidates are excluded from
467
520
  selection and chains; a role with no clean candidate is left untouched.
521
+ - **Allowed-providers whitelist** — the account's OpenRouter **allowed-providers
522
+ privacy whitelist** (openrouter.ai/settings/privacy) is learned once per run (a
523
+ canary request that bills no tokens) and every model's route pool is narrowed to
524
+ allowed providers before ranking, so a pin or a fallback hop is never written on a
525
+ provider the account cannot use. The run notes `OpenRouter allowed-providers
526
+ whitelist active — N providers; M routes pruned` (or `… whitelist: no whitelist`).
527
+ With no whitelist set — or when the harvest fails — nothing changes.
468
528
  - **Explorer key marking** — the explorer overlays every ranked model with a
469
529
  **usable** / **key-blocked** / **unknown** badge derived from the
470
530
  key-authenticated `GET /api/v1/models` (the account setting **"Filter the model
@@ -485,8 +545,10 @@ The plugin keeps only models the OpenRouter key can actually run:
485
545
  [docs/dev/quirks.md](docs/dev/quirks.md)).
486
546
  - **Provider whitelist.** The account's allowed-providers whitelist blocks many
487
547
  models, so the report's #1 and the written selector can differ — judge weights
488
- on the reachable pool.
489
- - **Provider pinning opts out of auto-Exacto.** A `providerPin` writes a trailing
548
+ on the reachable pool. The updater prunes *routes* to the whitelist (see
549
+ Availability), but the report has no token and is not whitelist-aware.
550
+ - **Provider pinning opts out of auto-Exacto.** A `providerPin` — or the
551
+ automatic pin the `providerPinning` capability writes — puts a trailing
490
552
  `@<slug>` on the selector, which omp applies as OpenRouter
491
553
  `provider: { only: [slug] }` routing. `only` is exclusive, so a pinned role's
492
554
  requests bypass OpenRouter's automatic Exacto tool routing (which routes tool
@@ -25,24 +25,26 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
25
25
 
26
26
  | File | Role / key exports |
27
27
  |---|---|
28
- | `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
28
+ | `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, the route-price accessors (`routeInputPrice`, `routeEffectivePrice`, `routeOutputPrice`, `blendOutputPrice`), `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
29
29
  | `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `parseBenchmarkPayloadMeta`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `cachedSourceInfo`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`BenchmarkPayloadMeta`/`FocusSourceInfo`/`BenchmarkCatalogEntry`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
30
- | `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
30
+ | `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `ResolvedSettings` carries `tokenProfile` (`{ inputTokens, outputTokens } | null`, default `null`) — the user-set per-task token profile the explorer's per-task price falls back to when a role has no recorded usage; validated to finite numbers ≥ 0, schema rows `tokenProfile.inputTokens`/`tokenProfile.outputTokens`. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
31
31
  | `src/agent-pins.ts` | Agent → pinned-role derivation. Exports `parseAgentPin` (first `@<role>` in the `model:` frontmatter) and `discoverAgentPins` (scans the shipped, user and project agent dirs; project > user > plugin). Drives the `task.disabledAgents` sync. |
32
- | `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe, and the keyed-catalog availability primitive. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
32
+ | `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe, the keyed-catalog availability primitive, and the allowed-providers whitelist harvest. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `fetchAllowedProviders`, `providerAllowed`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
33
33
  | `src/config-edit.ts` | Surgical line-oriented YAML patch for `modelRoles` + `retry.fallbackChains` + `task.disabledAgents`, plus the atomic writer. Exports `parseConfig`, `patchConfig`, `writeConfigAtomic`, `ConfigEditError`, and the `ConfigPatch` type. No runtime YAML dependency. |
34
34
  | `src/state.ts` | State/history/lock files under the agent dir, and agent-dir resolution. Exports `agentDir`, `loadState`, `saveState`, `appendHistory`, `acquireLock`, `releaseLock`, `freshState`, and the `PluginState` type (which carries `managedDisabledAgents` — the `task.disabledAgents` names the plugin added). |
35
+ | `src/usage-ledger.ts` | The per-role usage ledger (spec #81): an append-only jsonl file (`llm-role-usage.jsonl`, cap `USAGE_LEDGER_CAP = 2000` rows) beside the plugin's other state, one row per assistant turn. Exports `UsageRow`, `TokenProfile`, `USAGE_LEDGER_FILE`, `USAGE_LEDGER_CAP`, `appendUsageRow` (best-effort, never throws), `readUsageRows` (tolerant: missing/malformed → `[]`), `roleTokenProfile` (the role's median per-turn input/output counts via `agent-create.ts`'s `quantile` — the repo's one quantile convention — or `null`), and `resolveTokenProfile` (recorded usage → the `tokenProfile` setting → `null`). The caller owns the dir; the extension records into `resolveProjectDir() ?? agentDir()`, the explorer reads the active scope's dir. |
35
36
  | `src/error-report.ts` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role` (the `errorReporting` setting). The plugin's only GitHub client and the only home for a fingerprint, a redaction rule, the agent-dir ledger (`ERROR_REPORT_LEDGER_FILE`) and the issue body. **Entry point:** `reportUnexpectedError(input, deps)` — the feature's one test seam (per-call input + fully injected `deps`; tests supply a fake `fetch`, clock, env, dir and token resolver). Also exports `defaultErrorReportDeps()` (production deps: resolved settings, global `fetch`, real clock, `agentDir`, the shipped manifest, `gh auth token`, the OS opener) and `resetErrorReportGuard()` (the in-process dedupe guard's test hook). Never throws, never rejects; a no-op when `errorReporting` is `off`. |
36
- | `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
37
- | `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
37
+ | `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, `resolveProjectDir` (the project scope's `<cwd>/.omp` when it carries a non-empty `omp-llm-role` settings entry, else `null` — the same test the explorer's scope resolution and the extension's usage-ledger dir use), the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
38
+ | `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Records the per-role usage ledger (spec #81) on `turn_end` and `session_shutdown` — one row per assistant turn read from `sessionManager.getBranch()`, into `resolveProjectDir() ?? agentDir()`, deduped by `ts|model` so the two hooks cannot double-record. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
38
39
  | `src/guards.ts` | The package's one type guard: `isRecord`. |
40
+ | `src/provider-display.ts` | The provider display (spec #77): keep the live session model's `name` suffixed with its upstream provider pin, so omp's model segment renders e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. Pure and dual-runtime (no SDK import). Exports `providerPinOf` (the single `compat.openRouterRouting.only[0]`, else `compat.vercelGatewayRouting.only[0]`; null unless exactly one non-empty string), `providerPinOfSelector` (the trailing `@<slug>` of a selector, after stripping a thinking suffix), `withProviderSuffix` (strip any previous ` @<slug>` then append, idempotent), and `applyProviderDisplay` (mutate a record's `name`; no-op otherwise). |
39
41
  | `src/role-settings.ts` | The one validated role write path (validate → merge → backup → atomic write) and its removal half. Exports `validateRole`, `mergeExport`, `writeRoleSettings`, `mergeRemove`, `removeRoleSettings`, `timestamp`. Shared by the explorer's Export, `src/cli/create-role.ts`, `/create-agent` and `/remove-agent`. |
40
42
  | `src/role-archetypes.ts` | Purpose → weight archetype table (10 sets) + keyword fitting. Exports `ARCHETYPES`, `FALLBACK_ARCHETYPE`, `fitArchetype`, `archetypeById`, and the `Archetype`/`ArchetypeMatch` types. |
41
43
  | `src/agent-file.ts` | Agent `.md` rendering/placement. Exports `renderAgentFile`, `writeAgentFile`, `removeAgentFile`, `isReadOnlyTools`, `userAgentsDir`, `projectAgentsDir`, `READ_ONLY_TOOLS`, `RESERVED_AGENT_NAMES`, `AGENT_NAME_RE`, and the `AgentFileSpec`/`AgentWriteResult`/`AgentRemoveResult` types. |
42
44
  | `src/agent-architect.ts` | omp's in-process architects: the agent-creation architect (`/create-agent`, `/agents` hub) and the project-profile architect (`/project-roles`). Exports `generateAgentSpec`, `generateProjectProfile`, `parseAgentSpec`, `parseProjectProfile`, `extractJsonObject`, `extractAssistantText`, and the `ArchitectOptions`/`ProjectArchitectOptions` types. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the runner), so the module is importable under plain Node. |
43
45
  | `src/benchmark-author.ts` | Benchmark-source authoring for an unknown link: fetch the link, run an in-process architect (`src/prompts/benchmark-source-architect.md`) to propose a declarative source spec, and validate it. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the function). Exports `authorBenchmarkSource`, `parseSourceDeclarationJson`, and the `BenchmarkAuthorOptions` type. |
44
46
  | `src/benchmark-discovery.ts` | Judge-backed benchmark relevance for `/create-agent` discovery: one `noul` (yes/no probability) question per catalog candidate, batched into a single judgment through omp's configured `judge` role (TypeSafe jev or its fallback chain). Extension-only (imports `@oh-my-pi/pi-coding-agent` and its `./judgment` subpath — the deeper `./judgment/standalone` path does not resolve under omp's SDK injection). The SDK is imported dynamically so a resolution failure degrades to "no discovery" instead of failing the whole extension load. Exports `judgeBenchmarkRelevance`. |
45
- | `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
47
+ | `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `quantile` (the repo's one quantile convention — the focus-axis dispersion stats and the usage ledger's per-turn median), `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
46
48
  | `src/agent-remove.ts` | `/remove-agent` core: delete an agent `.md` and its role, plus the shared flag parser and report formatter. Exports `removeAgent`, `parseRemoveAgentArgs`, `formatRemoveAgentReport`, `REMOVE_AGENT_USAGE`, and the request/result types. Refuses shipped default roles. |
47
49
  | `src/project-setup.ts` | `/project-roles` core: a discovered `ProjectProfile` → a project-scoped role set (kept shipped roles, dropped ones, new fitted roles) written to the project plugin settings lock file, plus the project agents for the new roles. All-or-nothing (every check — names, metrics, weight math, the `--force` gate, agent-file collisions — before any write), `--dry-run`. Exports `setupProject`, `parseProjectRolesArgs`, `parseRolesSpec`, `applyProfileOverrides`, `formatProjectRolesReport`, `PROJECT_ROLES_USAGE`, and the `ProjectProfile`/`ProposedRole`/`SetupProjectOpts`/`SetupProjectResult` types (the result carries an `archetypes` map). The plan report prints the capability flags each new role will author (`features: <id>=<on>`). Dual-runtime safe (no SDK import); the extension command does the LLM work and calls it. |
48
50
 
@@ -50,9 +52,11 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
50
52
 
51
53
  | File | Role / key exports |
52
54
  |---|---|
53
- | `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. |
54
- | `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`), `POST /api/rank`, `POST /api/explain`, `POST /api/export`, `POST /api/refresh`. |
55
- | `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, and the nine-axis focus assessment. Exports `rankRows`, `explainModel`, `inverseCardinal`, `focusMetricsOf`, `focusAssessments`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta` types. |
55
+ | `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. `ExplorerBootOpts` carries the as-written pool's inputs (`tier`, `allowedProviders`, `reloadAllowedProviders`), the `tokenProfile` fallback, and `usageDir` (the user-level ledger dir; a project scope reads its own `<root>/.omp` ledger). |
56
+ | `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`, `poolMode`, `tokenProfile`), `POST /api/scope`, `POST /api/pool` (switch the ranked pool, raw vs as-written), `POST /api/rank` (rows carry the display-only `perTask`), `POST /api/explain` (carries `leaderFlip`/`thinkingLevels`), `POST /api/export`, `POST /api/refresh`. `ExplorerOpts` gains `getPoolInputs()` and the optional `getUsageRows(scope)`/`tokenProfile` seams. |
57
+ | `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, the nine-axis focus assessment, and the display-only per-task price. Exports `rankRows`, `explainModel`, `perTaskPrice`, `inverseCardinal`, `focusMetricsOf`, `focusAssessments`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta`/`PerTaskPrice` types. `rankRows`/`explainModel` take a trailing optional `profile?: TokenProfile \| null`; `RankRow.perTask` and `Explanation.perTask` are `null` when no profile resolves or the model has no usable route price. |
58
+ | `src/explorer/levers.ts` | Pure price-lever readouts (spec #81): the leader-flip threshold and the thinking level's effect on the leader, both computed on the pool they are handed. Exports `leaderFlip`, `thinkingLevelEffect`, and the `LeaderFlip`/`ThinkingLevelRow` types. Reuses the engine's `roleLambda`/`rankRole`/`SUFFIX_LEVELS`/`thinkingPriceFactor` — never re-derives the math. |
59
+ | `src/explorer/pool.ts` | The as-written pool (spec #81): the explorer's dataset filtered exactly as the updater filters it before ranking — the catalog/tier gate (`filterCatalog` + `resolveVariant`) and the allowed-providers whitelist prune (`providerAllowed`). Pure (never mutates its input). Exports `asWrittenPool`, and the `PoolMode`/`PoolInputs` types. |
56
60
 
57
61
  ### Assets and non-code
58
62
 
@@ -116,7 +120,22 @@ The omp extension injects the production `Deps`; tests inject fakes.
116
120
  fetchKeyMeta(token)` → `tierGate` → `"billed" | "free" | "none"`; `"none"`
117
121
  aborts.
118
122
  5. **Catalog + probe.** `deps.getCatalog()` → `filterCatalog(catalog, tier)`;
119
- `enrichThinkingLevels(rank.models, catalog)`. Per role, candidates are
123
+ `enrichThinkingLevels(rank.models, catalog)`.
124
+ **Whitelist prune.** The updater then harvests the account's
125
+ allowed-providers privacy whitelist once per run
126
+ (`deps.getAllowedProviders ? deps.getAllowedProviders(token, canaryModelId)
127
+ : fetchAllowedProviders(token, canaryModelId)`, a canary 404 that bills no
128
+ tokens) and, when it is non-null, replaces every model's route pool with the
129
+ routes whose `providerSlug` matches a whitelist entry (`providerAllowed`,
130
+ prefix match `slug === entry || slug.startsWith(entry + "/")`,
131
+ non-mutating). The prune is
132
+ the single change point — `roleRoute`, `bestRoute`, `rankedRoutes` and
133
+ `endpointFilteredModel` all read `m.routes` — so the soft basis, the auto-pin
134
+ and the fallback chains all become whitelist-aware. A `null` whitelist (no
135
+ whitelist set, harvest failed) leaves every pool untouched. The run notes
136
+ `OpenRouter allowed-providers whitelist active — N providers; M routes pruned`
137
+ (or `… whitelist: no whitelist`).
138
+ Per role, candidates are
120
139
  `resolveVariant`-resolved and then verified by a bounded `probeModel` walk
121
140
  (current selector first, then rank order; budget `PROBE_BUDGET = 12`; verdicts
122
141
  cached per run). Only the narrow no-allowed-providers 404 disqualifies; every
@@ -202,6 +221,7 @@ type Deps = {
202
221
  getSettings?(): Promise<Record<string, unknown>>;
203
222
  getKeyMeta?(token: string): Promise<KeyMeta>;
204
223
  probeModel?(token: string, catalogId: string): Promise<ProbeVerdict>;
224
+ getAllowedProviders?(token: string, modelId: string): Promise<Set<string> | null>;
205
225
  // Host coupling (extension only): apply a concrete selector to the live session model.
206
226
  applySessionModel?(selector: string, previous: string | null): Promise<void>;
207
227
  };
@@ -221,7 +241,11 @@ before `catalogFromOmpModelsJson`.
221
241
  `startExplorer`), `/create-agent` (architect → `createAgent` → in-process
222
242
  `runUpdater`), `/remove-agent` (`removeAgent` → in-process `runUpdater`),
223
243
  `/project-roles` (profile architect → `setupProject` → in-process
224
- `runUpdater`), `session_shutdown` (closes the explorer handle).
244
+ `runUpdater`), `session_shutdown` (closes the explorer handle). The provider
245
+ display adds four event subscriptions — `session_start`,
246
+ `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` — each
247
+ applying the live model's provider pin to its `name` (see
248
+ [`quirks.md`](quirks.md) for why `turn_start` is the self-healing hook).
225
249
  - **CLI**: `src/cli/llm-role-rank.ts` (report), `src/cli/create-role.ts`
226
250
  (authoring). The updater and explorer have no CLI shim — they run through
227
251
  `/refresh-roles` and `/explore-roles`.
package/docs/dev/ci.md CHANGED
@@ -21,6 +21,15 @@ ref must be green first. `test` runs **exactly** the command the local
21
21
  pre-commit hook runs (`.githooks/pre-commit`), and `tests/ci.test.ts` fails when
22
22
  the two drift apart — "green locally" and "green in CI" cannot diverge.
23
23
 
24
+ The hook scrubs `GIT_DIR`, `GIT_WORK_TREE`, `GIT_INDEX_FILE` and
25
+ `GIT_OBJECT_DIRECTORY` before running the suite, and `tests/ci.test.ts` scrubs
26
+ the same variables from every child process it spawns. Git exports `GIT_DIR` to
27
+ every hook, and `GIT_DIR` overrides `cwd` — so without the scrub the test file's
28
+ temp-repo `git init`/`config`/`add`/`commit` calls target the **real**
29
+ repository, rewriting its config and moving a branch (issue #89). The test file
30
+ carries the scrub because a test that shells out to git must not depend on its
31
+ caller's environment; the hook carries it as defence in depth.
32
+
24
33
  The Node version comes from `.node-version` at the repo root (the exact local
25
34
  toolchain; bumping it is a normal commit). `engines` is deliberately not in
26
35
  `package.json`: the plugin executes inside omp's runtime, so an `engines` field
@@ -141,10 +150,21 @@ exit 1, so the command is safe to run unattended.
141
150
  is that version's CHANGELOG section. This is also the recovery path for a
142
151
  failed publish: the tag exists, so no new version is needed.
143
152
 
153
+ A successful `npm publish` is **not** immediately visible: npm prints "Your
154
+ package is being processed and may take a few minutes to become available",
155
+ and registry reads can serve the pre-publish packument for several minutes
156
+ (third-party mirrors such as jsDelivr lag longer). The step's log is the
157
+ discriminator — a direct publish prints `publishing to
158
+ https://registry.npmjs.org/ with tag latest`, `Signed provenance statement`
159
+ and `+ <pkg>@<version>`; a staged publish is a separate `npm stage publish`
160
+ and prints none of those.
161
+
144
162
  The dispatch resolves the workflow **from the ref**, so the tag's commit must
145
163
  contain `.github/workflows/ci.yml`. A tag cut before this workflow existed
146
164
  cannot be dispatched (`HTTP 422: Workflow does not have 'workflow_dispatch'
147
- trigger`) — the recovery path only works for tags cut after it landed.
165
+ trigger`) — the recovery path only works for tags cut after it landed. The
166
+ workaround is to repoint the tag at a commit that contains the workflow (done
167
+ once, for `v1.1.0` → `4f78ce2`).
148
168
 
149
169
  `tests/ci.test.ts` asserts the offline half: the workflow's structure (jobs,
150
170
  triggers, permissions, the command it runs), the hook/workflow command parity,
@@ -1,15 +1,80 @@
1
- # Current state (2026-10-08)
1
+ # Current state (2026-10-18)
2
2
 
3
3
  Dated snapshot of the ranking and the landed work as of the date in the heading —
4
4
  matched/priced/eligible counts, per-role leaders, the open defects, and the
5
5
  per-issue work log below. It is refreshed by
6
6
  regenerating the report (`node src/cli/llm-role-rank.ts --out docs/llm-role-rankings.md`) and
7
7
  copying the new numbers here; release prose belongs in
8
- [`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The ranking numbers
9
- below are the 2026-10-08 measurement (the input-price blend basis #21, the
10
- #17/#18/#19/#20 wave and the #34–#40 focus-axis/capability-flag work; see the top
11
- bullets) and are not re-derived on read.
8
+ [`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The model-level
9
+ ranking numbers below are the 2026-10-18 measurement (the input-price blend basis
10
+ #21, the #17/#18/#19/#20 wave, the #34–#40 focus-axis/capability-flag work, and
11
+ spec #81's explorer surfaces — the ranking CLI itself is unchanged); the
12
+ provider-pin numbers are the 2026-10-11 measurement (spec #71; see the top
13
+ bullet). Neither is re-derived on read.
12
14
 
15
+ - Explorer as-written pool, per-task price, and price levers (2026-10-18, spec
16
+ #81): the explorer gains a **pool toggle** (raw vs as-written), a **per-task
17
+ price column**, and two **price-lever readouts**. The as-written pool applies
18
+ the same two filters the updater applies before ranking — the catalog/tier gate
19
+ and the allowed-providers whitelist prune — so the leader the user sees is the
20
+ model the updater will write (the raw pool's leader differed from the config's
21
+ for 6 of 10 roles). The per-task price is `inputTokens·p_in +
22
+ outputTokens·p_out·thinkingFactor` over the role's priced route, using the
23
+ plugin's own recorded per-role usage (the median per-turn counts) or the
24
+ `tokenProfile` setting; it is **display-only** — the price axis stays `$/M`
25
+ (issue #32's verdict stands) and the reported `$/M` keeps its billed 3:1 blend.
26
+ The readouts are the leader-flip threshold (`λ* = min over cheaper challengers
27
+ X of (q_L − q_X)/(pEff_L − pEff_X)`, `w* = 20λ*/(1+20λ*)`) and the thinking
28
+ level's effect on the leader. A new append-only, capped, best-effort
29
+ **usage ledger** (`llm-role-usage.jsonl`, one row per assistant turn) follows
30
+ the plugin's state dir (`<project>/.omp` in project mode, `agentDir()`
31
+ otherwise). The ranking numbers are unchanged — the ranking CLI does not apply
32
+ the pool toggle, and the per-task price never feeds the sort key.
33
+ - Provider display (2026-10-12, spec #77): when the live session model carries a
34
+ provider pin (a hand-set `providerPin`, or the automatic pin the
35
+ `providerPinning` capability writes), the plugin keeps the model object's
36
+ `name` suffixed with the route, so omp's status-line model segment renders
37
+ e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. The suffix is applied on session
38
+ start, refreshed on the two fallback events, and re-derived at every
39
+ `turn_start` — the re-derive is what heals the display after omp silently
40
+ restores the primary model following a fallback cooldown (no session event is
41
+ emitted), and it also covers a manual `/model` change with a one-turn lag. An
42
+ event's selector is trusted only when it names the live session model, so an
43
+ advisor fallback (the same events, the advisor's model) re-derives from the
44
+ live model instead of stamping the advisor's pin onto the session.
45
+ Display-only: the suffix rides the model object's `name`, so attribution,
46
+ usage recording and model comparison are unaffected. The ranking numbers are
47
+ unchanged.
48
+
49
+ - Provider ordering: value-for-money route score (2026-10-11, spec #71): the
50
+ auto-pin route basis changed from the route's own value (`q_route −
51
+ λ·priceEff_route`) to a min/max-normalized **value-for-money score** `vfm =
52
+ wT·tNorm − wP·pNorm` (`wT = wTput/(wPrice+wTput)`, `wP =
53
+ wPrice/(wPrice+wTput)`; `tNorm`/`pNorm` min/max over the candidate set), with a
54
+ new **soft** `filters.preferredMinThroughput` floor (routes at or above sort
55
+ first; below-floor routes are kept and sorted last) that the `providerPinning`
56
+ preset sets to 50. `routeValue` is deleted. The model-level ranking is
57
+ unchanged — the report is not regenerated (the ranking CLI does not apply
58
+ capability features and no shipped role sets `preferOwnProvider`). Measured
59
+ 2026-10-11 on the current endpoints cache with the live lock's expanded defs
60
+ (all four capabilities on, `cacheHitRate 0.9`; `deepseek/deepseek-v4.1-flash`,
61
+ 11 allowed providers, 12 allowed routes):
62
+ `morph/fp8` ($0.0840,
63
+ 103 tok/s) wins `default`, `smol`, `commit`, `task`, `advisor`; `coreweave/fp8`
64
+ ($0.1978, 185.5 tok/s) wins `slow`, `tiny`, `designer`; `decart/fp4` ($0.0639,
65
+ 72 tok/s) wins `vision`, `plan`.
66
+
67
+ - Auto-pin chain expansion (2026-10-10, issue #69): with `providerPinning` on,
68
+ the fallback chain no longer stops at three providers per model — the primary
69
+ model contributes every gate-passing route except the pinned best, then each
70
+ `fallbackChainDepth` fallback model contributes every gate-passing route, in
71
+ score order (the `vfm` route score — spec #71 replaced the original
72
+ route-value order) and deduped by provider slug, so a down provider fails over
73
+ to another provider of the same model before the chain switches models. Chain
74
+ length is data-driven (`(R_primary − 1) + Σ R_fallback`) instead of the fixed
75
+ `2 + 3·fallbackChainDepth`; the manual-`providerPin` and no-pin paths are
76
+ unchanged. The ranking numbers are unchanged — the chain is a write-side
77
+ shape, not a ranking input.
13
78
  - Opt-in error reporting (2026-10-08, issue #48): a new `errorReporting` setting
14
79
  (`off` — the default — `ask`, `auto`) turns a genuinely unexpected failure — a
15
80
  command handler throwing, or a `runUpdater` abort whose `RunResult.defect` is
@@ -73,26 +138,33 @@ bullets) and are not re-derived on read.
73
138
  `src/features.ts`) is the single source of truth and `expandFeatures` the
74
139
  single precedence implementation (`effective def = shipped default ⊕
75
140
  preset(enabled flags) ⊕ explicit user keys`; a per-role `false` beats a global
76
- `true`; an explicit knob always wins). Grounding on the 2026-10-07 cache
77
- (`default` role, 142 eligible):
141
+ `true`; an explicit knob always wins). Grounding on the 2026-10-09 cache
142
+ (`default` role, 146 eligible; the `providerPinning` row is re-measured on the
143
+ 2026-10-11 cache):
78
144
 
79
145
  | Flag | Recommended settings | Measured effect |
80
146
  |---|---|---|
81
- | `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | 142 → 132 eligible, leader unchanged (`deepseek-v4.1-flash`, 0.579 $/M) |
82
- | `cachePricing` | `cacheHitRate: 0.5` | 142 eligible, leader unchanged, its `priceEff` 0.579 → 0.508 (−12 %) |
83
- | `providerPinning` | `preferOwnProvider: true` | per-model policy, never changes eligibility; 51 of 149 routed models match under a slug-prefix rule (33 exact, 17 prefix, 1 display-org), 10 of 25 orgs match for no model |
84
- | `costCap` | `filters.maxPriceUsdPerM: 10` | 142 → 135 eligible (the top ≈5 %), leader unchanged; pool p90 7.8, p95 8.9 $/M, every leader 0.58–0.85 $/M |
147
+ | `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | 146 → 135 eligible, leader unchanged (`deepseek-v4.1-flash`, 0.37 $/M) |
148
+ | `cachePricing` | `cacheHitRate: 0.5` | 146 eligible, leader unchanged, its `priceEff` 0.3732 → 0.3522 (−5.6 %) |
149
+ | `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | per-model policy, never changes eligibility; prices each model on its **best route** (max `vfm` among the filter-passing, non-degraded, priced routes, cap-respecting; the soft 50 tok/s floor sorts below-floor routes last; ties by slug then id) — 144 of 144 routed models have a candidate route (2026-10-11), so the soft basis covers the whole routed field |
150
+ | `costCap` | `filters.maxPriceUsdPerM: 10` | 146 → 137 eligible (the top ≈6 %), leader unchanged; pool p90 7.8, p95 11.1 $/M, every leader 0.20–3.71 $/M |
85
151
 
86
152
  `providerPinning`'s recommended setting is a **soft per-model policy**
87
153
  (`preferOwnProvider`), not a hard `providerPin`: a hard pin would make a model
88
- with no matching route ineligible and empty ~2/3 of a role's pool, so the
89
- policy re-prices a model on its own lab's route where one exists and falls back
90
- to the `1/price²` blend otherwise. `minContextTokens` is deliberately not in
91
- the `endpointCeilings` bundle (an absolute floor cannot express the video's
92
- fraction-of-the-model's-own-window complaint). The flags are settable from
93
- `create-role.ts`, `/create-agent`, `/project-roles` (`--feature`/
94
- `--list-features`) and the explorer's Features panel; the ranking numbers are
95
- unchanged until a flag is turned on.
154
+ with no matching route ineligible and empty most of a role's pool, so the
155
+ policy prices a model on its **best route** — the max-`vfm` route (the
156
+ value-for-money score, with the soft 50 tok/s floor sorting below-floor routes
157
+ last) among the filter-passing, non-degraded, priced routes (thinking-adjusted
158
+ price ≤ `maxPriceUsdPerM`; ties by slug then id) — and falls back to the
159
+ `1/price²` blend only when the model has no candidate route. The basis is soft:
160
+ it never changes eligibility, and on the 2026-10-11 cache every one of the 144
161
+ routed models has a candidate route, so the capability covers the whole routed
162
+ field (the old own-lab slug-prefix rule matched only 51 of 149). `minContextTokens`
163
+ is deliberately not in the `endpointCeilings` bundle (an absolute floor cannot
164
+ express the video's fraction-of-the-model's-own-window complaint). The flags
165
+ are settable from `create-role.ts`, `/create-agent`, `/project-roles`
166
+ (`--feature`/`--list-features`) and the explorer's Features panel; the ranking
167
+ numbers are unchanged until a flag is turned on.
96
168
 
97
169
  - Cross-source axis (2026-10-07, issue #38): the focus-metric gate gained the
98
170
  cross-source axis — the zeroeval per-benchmark payload's **second, independent**
@@ -616,18 +688,18 @@ bullets) and are not re-derived on read.
616
688
  0.663, margin 0.039 > `switchMargin`), because the incumbent has no Design
617
689
  Arena data and takes the neutral fill while MiMo carries a measured 0.97.
618
690
  The other nine roles are untouched (only `designer` weights `website`).
619
- - 407 llm-stats models; OpenRouter matched 155/407 (throughput), 153 priced (the
620
- 2026-10-07 snapshot read 150/400 — the daily cache moved, not the code; the
691
+ - 408 llm-stats models; OpenRouter matched 149/408 (throughput), 148 priced (the
692
+ 2026-10-09 snapshot read 155/408 — the daily cache moved, not the code; the
621
693
  #34–#40 axes and flags leave every ranking byte-identical until a flag is
622
694
  turned on).
623
- - Eligible per role: 146 (vision 75, designer 89, image-input filter).
695
+ - Eligible per role: 140 (vision 73, designer 87, image-input filter).
624
696
  - Value-ranking leaders (this report, thinking-adjusted prices; the *ranking*
625
697
  leaders, before the account's provider whitelist drops the blocked ones):
626
- `default` DeepSeek-V4.1-Flash (0.808), `smol` Muse Spark 1.1 (0.687),
627
- `slow` Muse Spark 1.3 (0.832), `vision` Qwen3.8 Flash (0.731), `plan`
628
- Muse Spark 1.3 (0.795), `commit` Muse Spark 1.1 (0.725), `tiny`
629
- Mercury 2 (0.797), `task` DeepSeek-V4.1-Flash (0.749), `advisor`
630
- Muse Spark 1.3 (0.786), `designer` DeepSeek-V4.1-Flash (0.761).
698
+ `default` DeepSeek-V4.1-Flash (0.807), `smol` DeepSeek-V4.1-Flash (0.706),
699
+ `slow` Gemini 4 Argon (0.834), `vision` Gemini 4 Argon (0.732),
700
+ `plan` Muse Spark 1.3 (0.789), `commit` DeepSeek-V4.1-Flash (0.736), `tiny`
701
+ Mercury 2 (0.797), `task` DeepSeek-V4.1-Flash (0.750), `advisor`
702
+ Muse Spark 1.3 (0.785), `designer` DeepSeek-V4.1-Flash (0.772).
631
703
  - Thinking-adjusted pricing landed (2026-09-27): the suffix table moved into
632
704
  `DEFAULT_ROLES` as a per-role `thinking` field, and the price axis scales by
633
705
  the level's factor for thinking-capable models — `slow` (`:max`, ×7.86)
@@ -159,8 +159,11 @@ probability proportional to `1/price²`). The weight basis is the **input
159
159
 
160
160
  A role with a `providerPin` does **not** use this blend: it prices the
161
161
  model's matching route directly — the route's billed 3:1 price and its p50
162
- throughput (scoring.md, *Route-aware pricing*). The blend is the unpinned
163
- default.
162
+ throughput (scoring.md, *Route-aware pricing*). A role with the soft
163
+ `preferOwnProvider` policy (the `providerPinning` capability) likewise prices
164
+ each model on its **best route** rather than the blend, falling back to the blend
165
+ only when the model has no candidate route (scoring.md, *Capability presets*).
166
+ The blend is the default for every other model.
164
167
 
165
168
  ## OpenRouter — key metadata (tier gate)
166
169
 
@@ -180,8 +183,8 @@ freeUsable = free_model_daily_requests.remaining > 0
180
183
  billed → billed variants only (exclude `:free`, `:batch`); else free → `:free`
181
184
  variants only (exclude `:batch`); else `"none"` aborts the run. The key endpoint
182
185
  carries **no** model/provider restriction field — the account's allowed-providers
183
- whitelist is invisible to it (see the keyed catalog below and the probe in
184
- [`architecture.md`](architecture.md)).
186
+ whitelist is invisible to it (see the allowed-providers whitelist below and the
187
+ probe in [`architecture.md`](architecture.md)).
185
188
 
186
189
  ## OpenRouter — keyed catalog (availability)
187
190
 
@@ -202,6 +205,32 @@ makes the Bearer-authenticated catalog a per-key allowlist.
202
205
  every consumer re-fetches, and `fetchKeyAvailability` **never throws** (a failure
203
206
  degrades to `unavailable` with empty sets, so the probe walk / `unknown` marks stand).
204
207
 
208
+ ## OpenRouter — allowed-providers whitelist (availability)
209
+
210
+ The account's **allowed-providers privacy whitelist**
211
+ (openrouter.ai/settings/privacy) restricts which provider routes a request may
212
+ use. It is invisible to `/api/v1/key` and to the catalog endpoints; the only
213
+ place it appears is the 404 body of a request that fails because of it.
214
+
215
+ - **URL**: `POST https://openrouter.ai/api/v1/chat/completions` with
216
+ `provider.only: ["__omp-llm-role-canary__"]` (a nonexistent provider slug) on a
217
+ valid catalog model. `fetchAllowedProviders(token, modelId, fetchImpl?)`
218
+ (`src/availability.ts`) issues it.
219
+ - **Signal**: the 404 body names the whitelist after "permits only:" — the
220
+ message is parsed into a `Set<string>` of bare lab names (`coreweave`,
221
+ `baseten`, …). A non-404 response, a body without the clause (no whitelist
222
+ set), or any transport/shape failure returns `null`. The canary 404s, so it
223
+ bills no tokens.
224
+ - **Match rule**: a route's `providerSlug` is allowed when it equals a whitelist
225
+ entry or starts with `<entry>/` — the whitelist carries bare lab names while
226
+ route slugs are tiered (`coreweave/fp8`, `baseten/fp8`).
227
+ - **Consumer**: the updater harvests it **once per run** (with a valid catalog
228
+ model id) and prunes every model's route pool to allowed providers before
229
+ ranking (architecture.md, *Data flow*). It is **not cached to disk** — every
230
+ run re-learns it, so a privacy-setting change takes effect on the next refresh
231
+ — and `fetchAllowedProviders` **never throws** (a failure degrades to `null`,
232
+ which leaves every route pool untouched).
233
+
205
234
  ## Design Arena — design quality
206
235
 
207
236
  Two routes, never averaged (same Elo family; mean diff −1.0 website … −5.1 svg,