omp-llm-role 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -6
- package/docs/dev/architecture.md +42 -19
- package/docs/dev/ci.md +9 -0
- package/docs/dev/current-state.md +126 -20
- package/docs/dev/explorer.md +126 -23
- package/docs/dev/releasing.md +66 -3
- package/docs/dev/scoring.md +76 -8
- package/docs/dev/testing.md +105 -14
- package/docs/dev/writes-and-state.md +161 -26
- package/docs/llm-role-rankings.md +115 -115
- package/package.json +19 -1
- package/src/agent-create.ts +4 -2
- package/src/availability.ts +40 -7
- package/src/engine.ts +33 -2
- package/src/explorer/boot.ts +43 -1
- package/src/explorer/explain.ts +101 -11
- package/src/explorer/levers.ts +100 -0
- package/src/explorer/pool.ts +46 -0
- package/src/explorer/server.ts +92 -21
- package/src/extension.ts +132 -2
- package/src/settings.ts +50 -0
- package/src/updater.ts +132 -33
- package/src/usage-ledger.ts +127 -0
- package/web/app.js +139 -1
- package/web/style.css +3 -0
package/README.md
CHANGED
|
@@ -63,6 +63,13 @@ omp plugin install omp-llm-role@gott50-plugins
|
|
|
63
63
|
unknown keys stay byte-identical, and a no-change run writes nothing.
|
|
64
64
|
`~/.omp/agent/llm-role-state.json` snapshots the previous `modelRoles` block
|
|
65
65
|
on every write as a rollback aid.
|
|
66
|
+
- **Usage ledger.** The plugin appends one row per assistant turn (the model,
|
|
67
|
+
the role, the provider, the token counts and the cost) to
|
|
68
|
+
`llm-role-usage.jsonl` beside its other state — `<project>/.omp` in project
|
|
69
|
+
mode, `~/.omp/agent` otherwise. It is append-only, capped at the newest 2000
|
|
70
|
+
rows, and best-effort (a recording failure never breaks a session). The
|
|
71
|
+
explorer's `$/task` column reads it to learn your own token mix; nothing else
|
|
72
|
+
uses it.
|
|
66
73
|
|
|
67
74
|
## Roles
|
|
68
75
|
|
|
@@ -119,7 +126,9 @@ Global knobs:
|
|
|
119
126
|
| `priceSwitchFraction` | `0.5` | Inside the margin, adopt a challenger at least this fraction cheaper (0.5 = half the $/M); `0` disables |
|
|
120
127
|
| `writeFallbackChains` | `true` | Write per-role fallback chains into `config.yml` |
|
|
121
128
|
| `fallbackChainDepth` | `2` | Fallback models per role chain |
|
|
129
|
+
| `fallbackQualityFloor` | `0.9` | A cheaper-or-equal fallback must retain at least this fraction of the chosen model's quality; `0` disables |
|
|
122
130
|
| `activateDefaultOnEmptySession` | `true` | Reapply the new `default` selector to an empty session's active model |
|
|
131
|
+
| `tokenProfile` | unset | `{ inputTokens, outputTokens }` per task for the explorer's `$/task` column, used when a role has no recorded usage. Display-only; the recorded usage takes precedence. Unset = the column reads `—` |
|
|
123
132
|
| `errorReporting` | `off` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role`; one of `off`, `ask`, `auto` (see below) |
|
|
124
133
|
|
|
125
134
|
Per-role knobs (`roles.<name>.*`):
|
|
@@ -152,7 +161,7 @@ defaults.
|
|
|
152
161
|
|---|---|---|
|
|
153
162
|
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | Routes that cannot call tools and routes whose output ceiling is below 16 384 tokens leave the pool — a reasoning model is not cut off mid-thought |
|
|
154
163
|
| `cachePricing` | `cacheHitRate: 0.5` | The role is priced on a cache-heavy agent loop's invoice (the endpoint's `cacheReadPrice` blended in) rather than the sticker price |
|
|
155
|
-
| `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | Each request is bound to
|
|
164
|
+
| `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | Each request is bound to a provider route **that OpenRouter actually serves**: the updater walks the model's value-for-money route order (the `vfm` score, with a soft 50 tok/s floor that deprioritizes a pathologically slow route without dropping it) and **probes each candidate with a one-token request bound to it** (`provider: { only: [slug] }`), pinning the first route the request does not reject and **dropping the pin** (bare selector, OpenRouter's own routing) when no candidate is routable — so a role never 404s on a route the endpoints data still called healthy. The selector gains `@<slug>`, and the fallback chain carries the primary model on every gate-passing route except the pinned one, then each of the next `fallbackChainDepth` fallback models on every gate-passing route — each drawn from the role's own ranking below the primary, the cheaper-or-equal ones that keep at least `fallbackQualityFloor` of its quality first, then the pricier — so a down provider fails over to another provider of the same model before the chain switches models, and a failover never silently multiplies the spend or drops the role onto a much weaker model. The model is priced **and measured** on its score-max route (its billed price and p50 throughput, so `q` moves too), falling back to the default blend only when it has no candidate route |
|
|
156
165
|
| `costCap` | `filters.maxPriceUsdPerM: 10` | The priciest tail of the eligible pool leaves it, so the pick cannot be an expensive outlier |
|
|
157
166
|
|
|
158
167
|
Precedence is one rule:
|
|
@@ -421,21 +430,31 @@ editing source and re-running the CLI.
|
|
|
421
430
|
that scope's lock file — a project scope resolves as the project lock merged
|
|
422
431
|
over the user-level lock, exactly what the updater ranks. A project whose lock
|
|
423
432
|
file is gone is shown disabled. Switching with unsaved edits asks first.
|
|
433
|
+
- **Pool toggle** — the header switches the ranked pool between **raw** (the
|
|
434
|
+
account-agnostic fetch) and **as-written** (the pool the updater actually
|
|
435
|
+
writes from: the catalog/tier gate plus your account's allowed-providers
|
|
436
|
+
whitelist). The as-written pool is what you will get, so its leader is the
|
|
437
|
+
model the next `/refresh-roles` will pick; the raw pool shows the
|
|
438
|
+
account-agnostic ranking. The toggle is best-effort (a failed whitelist fetch
|
|
439
|
+
degrades to the raw pool) and per-scope, and it never affects **Export** —
|
|
440
|
+
Export always writes the role you authored, from either pool.
|
|
424
441
|
- **Role tabs** — every role the plugin knows: the nine omp built-in roles, the
|
|
425
442
|
shipped opt-in `designer`, and any role present only in the lock file. Each
|
|
426
443
|
tab is tinted by provenance — `default` (omp built-in), `plugin` (shipped but
|
|
427
444
|
opt-in), `user` (lock-file only) — and a disabled role is muted and dashed.
|
|
428
445
|
Selecting a disabled role still ranks its models.
|
|
429
446
|
- **Rank table** — every eligible model for the selected role, with `value`,
|
|
430
|
-
`q`, `$/M` (effective, thinking-adjusted),
|
|
447
|
+
`q`, `$/M` (effective, thinking-adjusted), **`$/task`** (what you actually pay
|
|
448
|
+
per task under your own token mix — see below), `tok/s`, `ctx`, a `★` Pareto
|
|
431
449
|
marker, and a `Δ` column showing the rank delta against the role's effective
|
|
432
450
|
def (what the plugin does today).
|
|
433
451
|
- **Explain panel** — click a row for the full decomposition: each weighted
|
|
434
452
|
metric's raw value → cardinal transform → renormalized weight → contribution,
|
|
435
453
|
the cost block (`λ`, effective vs billed price, `penalty = λ·price`,
|
|
436
|
-
`value = q − penalty`), "why not higher"
|
|
437
|
-
plus the per-metric target that would close
|
|
438
|
-
it on (price, q)
|
|
454
|
+
`value = q − penalty`), the **per-task price** breakdown, "why not higher"
|
|
455
|
+
(the value gap to the model above plus the per-metric target that would close
|
|
456
|
+
it), the models that dominate it on (price, q), and the **Price levers**
|
|
457
|
+
section (below).
|
|
439
458
|
- **Weight editor** — edit the role's description, weights, `required` set,
|
|
440
459
|
`filters.image`, `thinking` level, and `λ` override live; the table re-ranks
|
|
441
460
|
on every change. Weights are edited freely (no implicit rescaling): the `Σ`
|
|
@@ -457,8 +476,30 @@ editing source and re-running the CLI.
|
|
|
457
476
|
- **Hover explanations** — every column header, metric name, role tab, and
|
|
458
477
|
control carries a tooltip explaining its semantics.
|
|
459
478
|
|
|
479
|
+
**The per-task price (`$/task`).** The `$/M` column is the reported price — the
|
|
480
|
+
billed 3:1 input:output blend OpenRouter charges, scaled by the role's thinking
|
|
481
|
+
factor. The `$/task` column is what *you* pay per task: your own input:output
|
|
482
|
+
token mix applied to the role's priced route. The mix comes from the plugin's
|
|
483
|
+
own recorded usage (the role's median per-turn counts, from the
|
|
484
|
+
`llm-role-usage.jsonl` ledger the plugin appends to as you work), falling back
|
|
485
|
+
to the `tokenProfile` setting when a role has no recorded usage. It is
|
|
486
|
+
**display-only** — the ranking still sorts on `$/M` — and it reads `—` when no
|
|
487
|
+
profile is available (no recorded usage and no `tokenProfile` set) or the model
|
|
488
|
+
has no usable route price. A model that is cheaper per token but
|
|
489
|
+
more verbose can cost more per task; this column is where that shows up.
|
|
490
|
+
|
|
491
|
+
**The price levers.** The explain panel's **Price levers** section makes the
|
|
492
|
+
price posture visible. The **leader flip** is the price weight `w*` at which a
|
|
493
|
+
cheaper challenger overtakes the current leader, with the challenger named and
|
|
494
|
+
the distance from your current weight — so you can see whether your price
|
|
495
|
+
posture binds (a small bump flips the pick) or is slack. It is absent when no
|
|
496
|
+
cheaper challenger exists. The **thinking level** table shows which model wins
|
|
497
|
+
at each level, with the level's price factor — the factor multiplies the price
|
|
498
|
+
axis, so it penalizes expensive models more, which is why `auto` favours the
|
|
499
|
+
cheap model. Both readouts are computed on the pool you are viewing.
|
|
500
|
+
|
|
460
501
|
The UI never reimplements the ranking math: the numbers on screen are exactly
|
|
461
|
-
the numbers the plugin would use. The server binds `
|
|
502
|
+
the numbers the plugin would use. The server binds `[IP_ADDRESS]` only (no auth)
|
|
462
503
|
and serves the SPA with no build step and no external requests.
|
|
463
504
|
|
|
464
505
|
## Availability
|
package/docs/dev/architecture.md
CHANGED
|
@@ -25,16 +25,17 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
25
25
|
|
|
26
26
|
| File | Role / key exports |
|
|
27
27
|
|---|---|
|
|
28
|
-
| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
|
|
28
|
+
| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, the route-price accessors (`routeInputPrice`, `routeEffectivePrice`, `routeOutputPrice`, `blendOutputPrice`), `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
|
|
29
29
|
| `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `parseBenchmarkPayloadMeta`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `cachedSourceInfo`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`BenchmarkPayloadMeta`/`FocusSourceInfo`/`BenchmarkCatalogEntry`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
|
|
30
|
-
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
30
|
+
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `ResolvedSettings` carries `tokenProfile` (`{ inputTokens, outputTokens } | null`, default `null`) — the user-set per-task token profile the explorer's per-task price falls back to when a role has no recorded usage; validated to finite numbers ≥ 0, schema rows `tokenProfile.inputTokens`/`tokenProfile.outputTokens`. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
31
31
|
| `src/agent-pins.ts` | Agent → pinned-role derivation. Exports `parseAgentPin` (first `@<role>` in the `model:` frontmatter) and `discoverAgentPins` (scans the shipped, user and project agent dirs; project > user > plugin). Drives the `task.disabledAgents` sync. |
|
|
32
|
-
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution,
|
|
32
|
+
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, the model probe, the auto-pin's route probe, the keyed-catalog availability primitive, and the allowed-providers whitelist harvest. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `probeRoute` (the same one-token completion plus `provider: { only: [slug] }` — the auto-pin's routability gate; spec #90), `fetchAllowedProviders`, `providerAllowed`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
|
|
33
33
|
| `src/config-edit.ts` | Surgical line-oriented YAML patch for `modelRoles` + `retry.fallbackChains` + `task.disabledAgents`, plus the atomic writer. Exports `parseConfig`, `patchConfig`, `writeConfigAtomic`, `ConfigEditError`, and the `ConfigPatch` type. No runtime YAML dependency. |
|
|
34
34
|
| `src/state.ts` | State/history/lock files under the agent dir, and agent-dir resolution. Exports `agentDir`, `loadState`, `saveState`, `appendHistory`, `acquireLock`, `releaseLock`, `freshState`, and the `PluginState` type (which carries `managedDisabledAgents` — the `task.disabledAgents` names the plugin added). |
|
|
35
|
+
| `src/usage-ledger.ts` | The per-role usage ledger (spec #81): an append-only jsonl file (`llm-role-usage.jsonl`, cap `USAGE_LEDGER_CAP = 2000` rows) beside the plugin's other state, one row per assistant turn. Exports `UsageRow`, `TokenProfile`, `USAGE_LEDGER_FILE`, `USAGE_LEDGER_CAP`, `appendUsageRow` (best-effort, never throws), `readUsageRows` (tolerant: missing/malformed → `[]`), `roleTokenProfile` (the role's median per-turn input/output counts via `agent-create.ts`'s `quantile` — the repo's one quantile convention — or `null`), and `resolveTokenProfile` (recorded usage → the `tokenProfile` setting → `null`). The caller owns the dir; the extension records into `resolveProjectDir() ?? agentDir()`, the explorer reads the active scope's dir. |
|
|
35
36
|
| `src/error-report.ts` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role` (the `errorReporting` setting). The plugin's only GitHub client and the only home for a fingerprint, a redaction rule, the agent-dir ledger (`ERROR_REPORT_LEDGER_FILE`) and the issue body. **Entry point:** `reportUnexpectedError(input, deps)` — the feature's one test seam (per-call input + fully injected `deps`; tests supply a fake `fetch`, clock, env, dir and token resolver). Also exports `defaultErrorReportDeps()` (production deps: resolved settings, global `fetch`, real clock, `agentDir`, the shipped manifest, `gh auth token`, the OS opener) and `resetErrorReportGuard()` (the in-process dedupe guard's test hook). Never throws, never rejects; a no-op when `errorReporting` is `off`. |
|
|
36
|
-
| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
37
|
-
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
37
|
+
| `src/updater.ts` | Orchestration: rank → tier gate → model probe → hysteresis → auto-pin route probe → chains → agent-disable sync → config write. Exports `runUpdater`, `resolveProjectDir` (the project scope's `<cwd>/.omp` when it carries a non-empty `omp-llm-role` settings entry, else `null` — the same test the explorer's scope resolution and the extension's usage-ledger dir use), the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
38
|
+
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Records the per-role usage ledger (spec #81) on `turn_end` and `session_shutdown` — one row per assistant turn read from `sessionManager.getBranch()`, into `resolveProjectDir() ?? agentDir()`, deduped by `ts|model` so the two hooks cannot double-record. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
38
39
|
| `src/guards.ts` | The package's one type guard: `isRecord`. |
|
|
39
40
|
| `src/provider-display.ts` | The provider display (spec #77): keep the live session model's `name` suffixed with its upstream provider pin, so omp's model segment renders e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. Pure and dual-runtime (no SDK import). Exports `providerPinOf` (the single `compat.openRouterRouting.only[0]`, else `compat.vercelGatewayRouting.only[0]`; null unless exactly one non-empty string), `providerPinOfSelector` (the trailing `@<slug>` of a selector, after stripping a thinking suffix), `withProviderSuffix` (strip any previous ` @<slug>` then append, idempotent), and `applyProviderDisplay` (mutate a record's `name`; no-op otherwise). |
|
|
40
41
|
| `src/role-settings.ts` | The one validated role write path (validate → merge → backup → atomic write) and its removal half. Exports `validateRole`, `mergeExport`, `writeRoleSettings`, `mergeRemove`, `removeRoleSettings`, `timestamp`. Shared by the explorer's Export, `src/cli/create-role.ts`, `/create-agent` and `/remove-agent`. |
|
|
@@ -43,7 +44,7 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
43
44
|
| `src/agent-architect.ts` | omp's in-process architects: the agent-creation architect (`/create-agent`, `/agents` hub) and the project-profile architect (`/project-roles`). Exports `generateAgentSpec`, `generateProjectProfile`, `parseAgentSpec`, `parseProjectProfile`, `extractJsonObject`, `extractAssistantText`, and the `ArchitectOptions`/`ProjectArchitectOptions` types. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the runner), so the module is importable under plain Node. |
|
|
44
45
|
| `src/benchmark-author.ts` | Benchmark-source authoring for an unknown link: fetch the link, run an in-process architect (`src/prompts/benchmark-source-architect.md`) to propose a declarative source spec, and validate it. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the function). Exports `authorBenchmarkSource`, `parseSourceDeclarationJson`, and the `BenchmarkAuthorOptions` type. |
|
|
45
46
|
| `src/benchmark-discovery.ts` | Judge-backed benchmark relevance for `/create-agent` discovery: one `noul` (yes/no probability) question per catalog candidate, batched into a single judgment through omp's configured `judge` role (TypeSafe jev or its fallback chain). Extension-only (imports `@oh-my-pi/pi-coding-agent` and its `./judgment` subpath — the deeper `./judgment/standalone` path does not resolve under omp's SDK injection). The SDK is imported dynamically so a resolution failure degrades to "no discovery" instead of failing the whole extension load. Exports `judgeBenchmarkRelevance`. |
|
|
46
|
-
| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
|
|
47
|
+
| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `quantile` (the repo's one quantile convention — the focus-axis dispersion stats and the usage ledger's per-turn median), `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
|
|
47
48
|
| `src/agent-remove.ts` | `/remove-agent` core: delete an agent `.md` and its role, plus the shared flag parser and report formatter. Exports `removeAgent`, `parseRemoveAgentArgs`, `formatRemoveAgentReport`, `REMOVE_AGENT_USAGE`, and the request/result types. Refuses shipped default roles. |
|
|
48
49
|
| `src/project-setup.ts` | `/project-roles` core: a discovered `ProjectProfile` → a project-scoped role set (kept shipped roles, dropped ones, new fitted roles) written to the project plugin settings lock file, plus the project agents for the new roles. All-or-nothing (every check — names, metrics, weight math, the `--force` gate, agent-file collisions — before any write), `--dry-run`. Exports `setupProject`, `parseProjectRolesArgs`, `parseRolesSpec`, `applyProfileOverrides`, `formatProjectRolesReport`, `PROJECT_ROLES_USAGE`, and the `ProjectProfile`/`ProposedRole`/`SetupProjectOpts`/`SetupProjectResult` types (the result carries an `archetypes` map). The plan report prints the capability flags each new role will author (`features: <id>=<on>`). Dual-runtime safe (no SDK import); the extension command does the LLM work and calls it. |
|
|
49
50
|
|
|
@@ -51,9 +52,11 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
51
52
|
|
|
52
53
|
| File | Role / key exports |
|
|
53
54
|
|---|---|
|
|
54
|
-
| `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. |
|
|
55
|
-
| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`), `POST /api/
|
|
56
|
-
| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets,
|
|
55
|
+
| `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. `ExplorerBootOpts` carries the as-written pool's inputs (`tier`, `allowedProviders`, `reloadAllowedProviders`), the `tokenProfile` fallback, and `usageDir` (the user-level ledger dir; a project scope reads its own `<root>/.omp` ledger). |
|
|
56
|
+
| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`, `poolMode`, `tokenProfile`), `POST /api/scope`, `POST /api/pool` (switch the ranked pool, raw vs as-written), `POST /api/rank` (rows carry the display-only `perTask`), `POST /api/explain` (carries `leaderFlip`/`thinkingLevels`), `POST /api/export`, `POST /api/refresh`. `ExplorerOpts` gains `getPoolInputs()` and the optional `getUsageRows(scope)`/`tokenProfile` seams. |
|
|
57
|
+
| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, the nine-axis focus assessment, and the display-only per-task price. Exports `rankRows`, `explainModel`, `perTaskPrice`, `inverseCardinal`, `focusMetricsOf`, `focusAssessments`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta`/`PerTaskPrice` types. `rankRows`/`explainModel` take a trailing optional `profile?: TokenProfile \| null`; `RankRow.perTask` and `Explanation.perTask` are `null` when no profile resolves or the model has no usable route price. |
|
|
58
|
+
| `src/explorer/levers.ts` | Pure price-lever readouts (spec #81): the leader-flip threshold and the thinking level's effect on the leader, both computed on the pool they are handed. Exports `leaderFlip`, `thinkingLevelEffect`, and the `LeaderFlip`/`ThinkingLevelRow` types. Reuses the engine's `roleLambda`/`rankRole`/`SUFFIX_LEVELS`/`thinkingPriceFactor` — never re-derives the math. |
|
|
59
|
+
| `src/explorer/pool.ts` | The as-written pool (spec #81): the explorer's dataset filtered exactly as the updater filters it before ranking — the catalog/tier gate (`filterCatalog` + `resolveVariant`) and the allowed-providers whitelist prune (`providerAllowed`). Pure (never mutates its input). Exports `asWrittenPool`, and the `PoolMode`/`PoolInputs` types. |
|
|
57
60
|
|
|
58
61
|
### Assets and non-code
|
|
59
62
|
|
|
@@ -142,20 +145,39 @@ The omp extension injects the production `Deps`; tests inject fakes.
|
|
|
142
145
|
6. **Hysteresis.** `no-current` → adopt; ineligible current → `adopted`; best
|
|
143
146
|
beats current by `switchMargin` → `switched`; inside the margin but
|
|
144
147
|
`cheaperInsideMargin` → `switched-cost`; else `kept-margin` / `kept-eligible`.
|
|
145
|
-
7. **
|
|
148
|
+
7. **Route pin (auto-pin).** With the `providerPinning` capability on and no
|
|
149
|
+
manual `providerPin`, the chosen model's `rankedRoutes` order is walked and
|
|
150
|
+
each candidate is probed with `probeRoute` (a one-token completion bound to
|
|
151
|
+
the slug; optional seam `deps.probeRoute?`, DI like `probeModel`); the first
|
|
152
|
+
route that is not `blocked` becomes the `@<slug>` pin. `unknown` counts as
|
|
153
|
+
usable; a model whose every candidate is rejected (or whose probe budget ran
|
|
154
|
+
out) drops the pin — the selector stays bare and `Decision.pinDropped` is set.
|
|
155
|
+
Verdicts are cached per run by `catalogId@slug`; the walk spends at most
|
|
156
|
+
`ROUTE_PROBE_BUDGET = 12` new probes per role. It never runs for a manual pin,
|
|
157
|
+
a locked or disabled role, a non-OpenRouter selector or a model with no
|
|
158
|
+
candidate route. Contract:
|
|
159
|
+
[`writes-and-state.md`](writes-and-state.md), *The provider-pin
|
|
160
|
+
auto-configuration*.
|
|
161
|
+
8. **Suffix.** Append `roles.<role>.thinking` only when the chosen catalog row's
|
|
146
162
|
`thinking[]` includes the level (meta levels `off`/`auto` need only a
|
|
147
163
|
non-empty list).
|
|
148
|
-
|
|
149
|
-
`fallbackChainDepth`
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
`
|
|
154
|
-
|
|
155
|
-
|
|
164
|
+
9. **Chains.** For every managed role, `chainUpserts` (bare key → the next
|
|
165
|
+
`fallbackChainDepth` fallback models drawn from the role's ranked candidate
|
|
166
|
+
list below the chosen model — the cheaper-or-equal candidates that retain at
|
|
167
|
+
least `fallbackQualityFloor` of its `q` first, then the pricier ones, each
|
|
168
|
+
bucket in value order, truncated to the depth — suffixed per target) and
|
|
169
|
+
`chainPrunes` (plugin-written keys no longer referenced). The list is the
|
|
170
|
+
ranked candidate list minus the key-blocked and probe-`blocked` candidates; an
|
|
171
|
+
unprobed candidate stays eligible, so the chain can reach a cheaper-or-equal
|
|
172
|
+
model below the probe window (writes-and-state.md, *Which models are
|
|
173
|
+
chained*). A key claimed by more than one role stays level-free.
|
|
174
|
+
10. **Config patch.** `writePatch` acquires the refresh lock, reads `config.yml`,
|
|
175
|
+
`patchConfig` (surgical, self-checked), `writeConfigAtomic` (mtime-guarded
|
|
176
|
+
tmp+rename, 3 conflict retries). Zero changes → no write, no mtime change.
|
|
177
|
+
11. **State + history.** `saveState` (day gate, managed roles, last selectors,
|
|
156
178
|
plugin-written chain keys, managed `task.disabledAgents` names, previous
|
|
157
179
|
`modelRoles` snapshot) and `appendHistory` (one row per completed run).
|
|
158
|
-
|
|
180
|
+
12. **Live-session coupling.** When `default` actually changed and
|
|
159
181
|
`activateDefaultOnEmptySession` is set, `deps.applySessionModel?.(selector,
|
|
160
182
|
previous)` hands the new selector to the host (extension only).
|
|
161
183
|
|
|
@@ -218,6 +240,7 @@ type Deps = {
|
|
|
218
240
|
getSettings?(): Promise<Record<string, unknown>>;
|
|
219
241
|
getKeyMeta?(token: string): Promise<KeyMeta>;
|
|
220
242
|
probeModel?(token: string, catalogId: string): Promise<ProbeVerdict>;
|
|
243
|
+
probeRoute?(token: string, catalogId: string, providerSlug: string): Promise<ProbeVerdict>;
|
|
221
244
|
getAllowedProviders?(token: string, modelId: string): Promise<Set<string> | null>;
|
|
222
245
|
// Host coupling (extension only): apply a concrete selector to the live session model.
|
|
223
246
|
applySessionModel?(selector: string, previous: string | null): Promise<void>;
|
package/docs/dev/ci.md
CHANGED
|
@@ -21,6 +21,15 @@ ref must be green first. `test` runs **exactly** the command the local
|
|
|
21
21
|
pre-commit hook runs (`.githooks/pre-commit`), and `tests/ci.test.ts` fails when
|
|
22
22
|
the two drift apart — "green locally" and "green in CI" cannot diverge.
|
|
23
23
|
|
|
24
|
+
The hook scrubs `GIT_DIR`, `GIT_WORK_TREE`, `GIT_INDEX_FILE` and
|
|
25
|
+
`GIT_OBJECT_DIRECTORY` before running the suite, and `tests/ci.test.ts` scrubs
|
|
26
|
+
the same variables from every child process it spawns. Git exports `GIT_DIR` to
|
|
27
|
+
every hook, and `GIT_DIR` overrides `cwd` — so without the scrub the test file's
|
|
28
|
+
temp-repo `git init`/`config`/`add`/`commit` calls target the **real**
|
|
29
|
+
repository, rewriting its config and moving a branch (issue #89). The test file
|
|
30
|
+
carries the scrub because a test that shells out to git must not depend on its
|
|
31
|
+
caller's environment; the hook carries it as defence in depth.
|
|
32
|
+
|
|
24
33
|
The Node version comes from `.node-version` at the repo root (the exact local
|
|
25
34
|
toolchain; bumping it is a normal commit). `engines` is deliberately not in
|
|
26
35
|
`package.json`: the plugin executes inside omp's runtime, so an `engines` field
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Current state (2026-10-
|
|
1
|
+
# Current state (2026-10-19)
|
|
2
2
|
|
|
3
3
|
Dated snapshot of the ranking and the landed work as of the date in the heading —
|
|
4
4
|
matched/priced/eligible counts, per-role leaders, the open defects, and the
|
|
@@ -6,11 +6,111 @@ per-issue work log below. It is refreshed by
|
|
|
6
6
|
regenerating the report (`node src/cli/llm-role-rank.ts --out docs/llm-role-rankings.md`) and
|
|
7
7
|
copying the new numbers here; release prose belongs in
|
|
8
8
|
[`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The model-level
|
|
9
|
-
ranking numbers below are the 2026-10-
|
|
10
|
-
|
|
11
|
-
the
|
|
9
|
+
ranking numbers below are the 2026-10-19 measurement (regenerated at the 1.3.0
|
|
10
|
+
cut; byte-identical to the 2026-10-18 one apart from the report's date line —
|
|
11
|
+
the input-price blend basis #21, the #17/#18/#19/#20 wave, the #34–#40
|
|
12
|
+
focus-axis/capability-flag work, and
|
|
13
|
+
spec #81's explorer surfaces — the ranking CLI itself is unchanged); the
|
|
14
|
+
provider-pin numbers are the 2026-10-11 measurement (spec #71; see the top
|
|
12
15
|
bullet). Neither is re-derived on read.
|
|
13
16
|
|
|
17
|
+
- Fallback chain quality floor (2026-10-19, spec #95): a role's fallback chain no
|
|
18
|
+
longer only reorders the probe window — that window is `1 + fallbackChainDepth`
|
|
19
|
+
models, the same size as the chain, so the reorder could never reach a cheaper
|
|
20
|
+
model at rank 4 or deeper. The chain's fallback **models** now come from the
|
|
21
|
+
role's whole ranked candidate list below the chosen model: the
|
|
22
|
+
**cheaper-or-equal** ones first — a candidate qualifies only if it retains at
|
|
23
|
+
least `fallbackQualityFloor` (a new setting, default `0.9`; `0` disables the
|
|
24
|
+
floor, `1` requires an exact quality match) of the chosen model's own `q`, so a
|
|
25
|
+
failover never silently drops the role onto a much weaker model — then the
|
|
26
|
+
pricier ones, each bucket in value order, truncated to `fallbackChainDepth`.
|
|
27
|
+
Key-blocked and probe-`blocked` candidates are excluded; an **unprobed**
|
|
28
|
+
candidate stays eligible, so the chain can reach a cheaper-or-equal model ranked
|
|
29
|
+
below the probe window (chain entries are never probe-verified — spec #90). A
|
|
30
|
+
candidate priced exactly at the chosen model's price whose `q` is below the
|
|
31
|
+
floor lands in neither bucket and is dropped. The floor gates **chain
|
|
32
|
+
membership only**: the primary choice, the hysteresis decision, the ranking, the
|
|
33
|
+
probe walk's stop condition and budget, the chain key, the route expansion, the
|
|
34
|
+
thinking levels and the prune-on-pin-change are all unchanged, as are the
|
|
35
|
+
manual-`providerPin` and no-pin paths. Measured on the 2026-10-19 caches with
|
|
36
|
+
the live resolved defs (`fallbackQualityFloor` 0.9, depth 2): `slow`, `vision`
|
|
37
|
+
and `task` gain cheaper fallbacks, the other seven roles are byte-identical, and
|
|
38
|
+
no role is degraded. That is the **per-role** effect: in the written config only
|
|
39
|
+
the `vision` key changes, because `deepseek-v4.1-flash@coreweave/fp8` is claimed
|
|
40
|
+
by seven roles and carries the last claimer's chain (the shared-key behaviour
|
|
41
|
+
spec #95 leaves out of scope). The ranking numbers are unchanged — the chain is a
|
|
42
|
+
write-side shape, not a ranking input — so `docs/llm-role-rankings.md` is not
|
|
43
|
+
regenerated and the dated numbers below stand as written; the suite grew
|
|
44
|
+
652 → 666.
|
|
45
|
+
|
|
46
|
+
- Routability-verified auto-pin (2026-10-19, spec #90): the `providerPinning`
|
|
47
|
+
auto-pin no longer writes the score-max route blind. The daily update walks the
|
|
48
|
+
chosen model's `vfm` route order and probes each candidate with a one-token
|
|
49
|
+
completion bound to it (`provider: { only: [slug] }`), because OpenRouter's
|
|
50
|
+
endpoints API can list a route as healthy — non-degraded, a full
|
|
51
|
+
`supported_parameters` list, 100 % uptime — while its routing table excludes it:
|
|
52
|
+
the live config's `deepseek/deepseek-v4.1-flash@relace` passed every gate and
|
|
53
|
+
then 404'd on every request, including every fallback revert. Only the narrow
|
|
54
|
+
no-allowed-providers 404 skips a route; a timeout, a 5xx, a 429 and a 404 with
|
|
55
|
+
any other body are `unknown` and count as usable, so a transient blip never
|
|
56
|
+
unpins a role. Verdicts are cached per run by `catalogId@slug`, and the walk is
|
|
57
|
+
capped at `ROUTE_PROBE_BUDGET = 12` probes per role. When no candidate is
|
|
58
|
+
routable the pin is **dropped** — the selector stays bare and OpenRouter routes
|
|
59
|
+
the request itself — and the run notes name the rejected slugs
|
|
60
|
+
(`; route-blocked: …`), plus the dropped pin (`; pin dropped (no routable
|
|
61
|
+
route)`) when none survived. The fallback chain keeps its shape
|
|
62
|
+
(the primary contributes every gate-passing route except the pinned **slug** —
|
|
63
|
+
see the chain bullet below), and **chain entries are not probed**: one
|
|
64
|
+
unroutable entry is survivable, omp fails over. A manual
|
|
65
|
+
`roles.<role>.providerPin` stays a hard gate: never probed, never replaced. The
|
|
66
|
+
explorer's auto-pin line now reads as a **candidate** — the first route the walk
|
|
67
|
+
tries, not a promise of the written pin. The ranking numbers are unchanged — the
|
|
68
|
+
probe is a write-side gate, not a ranking input, and neither the ranking CLI nor
|
|
69
|
+
`docs/llm-role-rankings.md` changed; the suite grew 631 → 652.
|
|
70
|
+
|
|
71
|
+
- Release 1.3.0 (2026-10-19, cut `55416d4`, tag `v1.3.0`): the spec #81 explorer
|
|
72
|
+
surfaces ship. The report was regenerated at the cut and is **unchanged** from
|
|
73
|
+
the 2026-10-18 measurement — matched 149/408, priced 148, eligible 140
|
|
74
|
+
(vision 73, designer 87), the same ten leaders — so the numbers below stand as
|
|
75
|
+
written. The cut also carried the #89 fix: the pre-commit hook and
|
|
76
|
+
`tests/ci.test.ts` now scrub `GIT_*` from every child process, so the suite's
|
|
77
|
+
temp-repo git calls can no longer target the real repository (suite 630 →
|
|
78
|
+
631).
|
|
79
|
+
|
|
80
|
+
- Explorer as-written pool, per-task price, and price levers (2026-10-18, spec
|
|
81
|
+
#81): the explorer gains a **pool toggle** (raw vs as-written), a **per-task
|
|
82
|
+
price column**, and two **price-lever readouts**. The as-written pool applies
|
|
83
|
+
the same two filters the updater applies before ranking — the catalog/tier gate
|
|
84
|
+
and the allowed-providers whitelist prune — so the leader the user sees is the
|
|
85
|
+
model the updater will write (the raw pool's leader differed from the config's
|
|
86
|
+
for 6 of 10 roles). The per-task price is `inputTokens·p_in +
|
|
87
|
+
outputTokens·p_out·thinkingFactor` over the role's priced route, using the
|
|
88
|
+
plugin's own recorded per-role usage (the median per-turn counts) or the
|
|
89
|
+
`tokenProfile` setting; it is **display-only** — the price axis stays `$/M`
|
|
90
|
+
(issue #32's verdict stands) and the reported `$/M` keeps its billed 3:1 blend.
|
|
91
|
+
The readouts are the leader-flip threshold (`λ* = min over cheaper challengers
|
|
92
|
+
X of (q_L − q_X)/(pEff_L − pEff_X)`, `w* = 20λ*/(1+20λ*)`) and the thinking
|
|
93
|
+
level's effect on the leader. A new append-only, capped, best-effort
|
|
94
|
+
**usage ledger** (`llm-role-usage.jsonl`, one row per assistant turn) follows
|
|
95
|
+
the plugin's state dir (`<project>/.omp` in project mode, `agentDir()`
|
|
96
|
+
otherwise). The ranking numbers are unchanged — the ranking CLI does not apply
|
|
97
|
+
the pool toggle, and the per-task price never feeds the sort key.
|
|
98
|
+
- Provider display (2026-10-12, spec #77): when the live session model carries a
|
|
99
|
+
provider pin (a hand-set `providerPin`, or the automatic pin the
|
|
100
|
+
`providerPinning` capability writes), the plugin keeps the model object's
|
|
101
|
+
`name` suffixed with the route, so omp's status-line model segment renders
|
|
102
|
+
e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. The suffix is applied on session
|
|
103
|
+
start, refreshed on the two fallback events, and re-derived at every
|
|
104
|
+
`turn_start` — the re-derive is what heals the display after omp silently
|
|
105
|
+
restores the primary model following a fallback cooldown (no session event is
|
|
106
|
+
emitted), and it also covers a manual `/model` change with a one-turn lag. An
|
|
107
|
+
event's selector is trusted only when it names the live session model, so an
|
|
108
|
+
advisor fallback (the same events, the advisor's model) re-derives from the
|
|
109
|
+
live model instead of stamping the advisor's pin onto the session.
|
|
110
|
+
Display-only: the suffix rides the model object's `name`, so attribution,
|
|
111
|
+
usage recording and model comparison are unaffected. The ranking numbers are
|
|
112
|
+
unchanged.
|
|
113
|
+
|
|
14
114
|
- Provider ordering: value-for-money route score (2026-10-11, spec #71): the
|
|
15
115
|
auto-pin route basis changed from the route's own value (`q_route −
|
|
16
116
|
λ·priceEff_route`) to a min/max-normalized **value-for-money score** `vfm =
|
|
@@ -31,15 +131,21 @@ bullet). Neither is re-derived on read.
|
|
|
31
131
|
|
|
32
132
|
- Auto-pin chain expansion (2026-10-10, issue #69): with `providerPinning` on,
|
|
33
133
|
the fallback chain no longer stops at three providers per model — the primary
|
|
34
|
-
model contributes every gate-passing route except the pinned
|
|
35
|
-
|
|
36
|
-
score
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
134
|
+
model contributes every gate-passing route **except the pinned route** (the
|
|
135
|
+
exclusion is by pinned slug, not by position: spec #90's probe can pin a route
|
|
136
|
+
other than the score leader, so the leader rides as a chain entry, and a dropped
|
|
137
|
+
pin excludes nothing), then each `fallbackChainDepth` fallback model contributes
|
|
138
|
+
every gate-passing route, in score order (the `vfm` route score — spec #71
|
|
139
|
+
replaced the original route-value order) and deduped by provider slug, so a down
|
|
140
|
+
provider fails over to another provider of the same model before the chain
|
|
141
|
+
switches models. Chain length is data-driven (`(R_primary − 1) + Σ R_fallback`,
|
|
142
|
+
or `R_primary + Σ R_fallback` with a dropped pin) instead of the fixed
|
|
40
143
|
`2 + 3·fallbackChainDepth`; the manual-`providerPin` and no-pin paths are
|
|
41
|
-
unchanged.
|
|
42
|
-
|
|
144
|
+
unchanged. Which `fallbackChainDepth` **models** each chain falls back to was
|
|
145
|
+
the probe window until spec #95, which draws them from the role's ranked
|
|
146
|
+
candidate list below the chosen model with the quality floor (see the top
|
|
147
|
+
bullet) — the route expansion above is unchanged. The ranking numbers are
|
|
148
|
+
unchanged — the chain is a write-side shape, not a ranking input.
|
|
43
149
|
- Opt-in error reporting (2026-10-08, issue #48): a new `errorReporting` setting
|
|
44
150
|
(`off` — the default — `ask`, `auto`) turns a genuinely unexpected failure — a
|
|
45
151
|
command handler throwing, or a `runUpdater` abort whose `RunResult.defect` is
|
|
@@ -653,18 +759,18 @@ bullet). Neither is re-derived on read.
|
|
|
653
759
|
0.663, margin 0.039 > `switchMargin`), because the incumbent has no Design
|
|
654
760
|
Arena data and takes the neutral fill while MiMo carries a measured 0.97.
|
|
655
761
|
The other nine roles are untouched (only `designer` weights `website`).
|
|
656
|
-
- 408 llm-stats models; OpenRouter matched
|
|
657
|
-
2026-10-
|
|
762
|
+
- 408 llm-stats models; OpenRouter matched 149/408 (throughput), 148 priced (the
|
|
763
|
+
2026-10-09 snapshot read 155/408 — the daily cache moved, not the code; the
|
|
658
764
|
#34–#40 axes and flags leave every ranking byte-identical until a flag is
|
|
659
765
|
turned on).
|
|
660
|
-
- Eligible per role:
|
|
766
|
+
- Eligible per role: 140 (vision 73, designer 87, image-input filter).
|
|
661
767
|
- Value-ranking leaders (this report, thinking-adjusted prices; the *ranking*
|
|
662
768
|
leaders, before the account's provider whitelist drops the blocked ones):
|
|
663
|
-
`default` DeepSeek-V4.1-Flash (0.
|
|
664
|
-
`slow`
|
|
665
|
-
`plan` Muse Spark 1.3 (0.
|
|
666
|
-
|
|
667
|
-
Muse Spark 1.3 (0.
|
|
769
|
+
`default` DeepSeek-V4.1-Flash (0.807), `smol` DeepSeek-V4.1-Flash (0.706),
|
|
770
|
+
`slow` Gemini 4 Argon (0.834), `vision` Gemini 4 Argon (0.732),
|
|
771
|
+
`plan` Muse Spark 1.3 (0.789), `commit` DeepSeek-V4.1-Flash (0.736), `tiny`
|
|
772
|
+
Mercury 2 (0.797), `task` DeepSeek-V4.1-Flash (0.750), `advisor`
|
|
773
|
+
Muse Spark 1.3 (0.785), `designer` DeepSeek-V4.1-Flash (0.772).
|
|
668
774
|
- Thinking-adjusted pricing landed (2026-09-27): the suffix table moved into
|
|
669
775
|
`DEFAULT_ROLES` as a per-role `thinking` field, and the price axis scales by
|
|
670
776
|
the level's factor for thinking-capable models — `slow` (`:max`, ×7.86)
|