omp-llm-role 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -5
- package/docs/dev/architecture.md +11 -8
- package/docs/dev/ci.md +9 -0
- package/docs/dev/current-state.md +47 -12
- package/docs/dev/explorer.md +106 -19
- package/docs/dev/releasing.md +33 -3
- package/docs/dev/scoring.md +55 -4
- package/docs/dev/testing.md +43 -2
- package/docs/dev/writes-and-state.md +30 -0
- package/docs/llm-role-rankings.md +115 -115
- package/package.json +11 -1
- package/src/agent-create.ts +4 -2
- package/src/engine.ts +33 -2
- package/src/explorer/boot.ts +43 -1
- package/src/explorer/explain.ts +90 -5
- package/src/explorer/levers.ts +100 -0
- package/src/explorer/pool.ts +46 -0
- package/src/explorer/server.ts +92 -21
- package/src/extension.ts +132 -2
- package/src/settings.ts +31 -0
- package/src/updater.ts +6 -1
- package/src/usage-ledger.ts +127 -0
- package/web/app.js +132 -0
- package/web/style.css +3 -0
package/README.md
CHANGED
|
@@ -63,6 +63,13 @@ omp plugin install omp-llm-role@gott50-plugins
|
|
|
63
63
|
unknown keys stay byte-identical, and a no-change run writes nothing.
|
|
64
64
|
`~/.omp/agent/llm-role-state.json` snapshots the previous `modelRoles` block
|
|
65
65
|
on every write as a rollback aid.
|
|
66
|
+
- **Usage ledger.** The plugin appends one row per assistant turn (the model,
|
|
67
|
+
the role, the provider, the token counts and the cost) to
|
|
68
|
+
`llm-role-usage.jsonl` beside its other state — `<project>/.omp` in project
|
|
69
|
+
mode, `~/.omp/agent` otherwise. It is append-only, capped at the newest 2000
|
|
70
|
+
rows, and best-effort (a recording failure never breaks a session). The
|
|
71
|
+
explorer's `$/task` column reads it to learn your own token mix; nothing else
|
|
72
|
+
uses it.
|
|
66
73
|
|
|
67
74
|
## Roles
|
|
68
75
|
|
|
@@ -120,6 +127,7 @@ Global knobs:
|
|
|
120
127
|
| `writeFallbackChains` | `true` | Write per-role fallback chains into `config.yml` |
|
|
121
128
|
| `fallbackChainDepth` | `2` | Fallback models per role chain |
|
|
122
129
|
| `activateDefaultOnEmptySession` | `true` | Reapply the new `default` selector to an empty session's active model |
|
|
130
|
+
| `tokenProfile` | unset | `{ inputTokens, outputTokens }` per task for the explorer's `$/task` column, used when a role has no recorded usage. Display-only; the recorded usage takes precedence. Unset = the column reads `—` |
|
|
123
131
|
| `errorReporting` | `off` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role`; one of `off`, `ask`, `auto` (see below) |
|
|
124
132
|
|
|
125
133
|
Per-role knobs (`roles.<name>.*`):
|
|
@@ -421,21 +429,31 @@ editing source and re-running the CLI.
|
|
|
421
429
|
that scope's lock file — a project scope resolves as the project lock merged
|
|
422
430
|
over the user-level lock, exactly what the updater ranks. A project whose lock
|
|
423
431
|
file is gone is shown disabled. Switching with unsaved edits asks first.
|
|
432
|
+
- **Pool toggle** — the header switches the ranked pool between **raw** (the
|
|
433
|
+
account-agnostic fetch) and **as-written** (the pool the updater actually
|
|
434
|
+
writes from: the catalog/tier gate plus your account's allowed-providers
|
|
435
|
+
whitelist). The as-written pool is what you will get, so its leader is the
|
|
436
|
+
model the next `/refresh-roles` will pick; the raw pool shows the
|
|
437
|
+
account-agnostic ranking. The toggle is best-effort (a failed whitelist fetch
|
|
438
|
+
degrades to the raw pool) and per-scope, and it never affects **Export** —
|
|
439
|
+
Export always writes the role you authored, from either pool.
|
|
424
440
|
- **Role tabs** — every role the plugin knows: the nine omp built-in roles, the
|
|
425
441
|
shipped opt-in `designer`, and any role present only in the lock file. Each
|
|
426
442
|
tab is tinted by provenance — `default` (omp built-in), `plugin` (shipped but
|
|
427
443
|
opt-in), `user` (lock-file only) — and a disabled role is muted and dashed.
|
|
428
444
|
Selecting a disabled role still ranks its models.
|
|
429
445
|
- **Rank table** — every eligible model for the selected role, with `value`,
|
|
430
|
-
`q`, `$/M` (effective, thinking-adjusted),
|
|
446
|
+
`q`, `$/M` (effective, thinking-adjusted), **`$/task`** (what you actually pay
|
|
447
|
+
per task under your own token mix — see below), `tok/s`, `ctx`, a `★` Pareto
|
|
431
448
|
marker, and a `Δ` column showing the rank delta against the role's effective
|
|
432
449
|
def (what the plugin does today).
|
|
433
450
|
- **Explain panel** — click a row for the full decomposition: each weighted
|
|
434
451
|
metric's raw value → cardinal transform → renormalized weight → contribution,
|
|
435
452
|
the cost block (`λ`, effective vs billed price, `penalty = λ·price`,
|
|
436
|
-
`value = q − penalty`), "why not higher"
|
|
437
|
-
plus the per-metric target that would close
|
|
438
|
-
it on (price, q)
|
|
453
|
+
`value = q − penalty`), the **per-task price** breakdown, "why not higher"
|
|
454
|
+
(the value gap to the model above plus the per-metric target that would close
|
|
455
|
+
it), the models that dominate it on (price, q), and the **Price levers**
|
|
456
|
+
section (below).
|
|
439
457
|
- **Weight editor** — edit the role's description, weights, `required` set,
|
|
440
458
|
`filters.image`, `thinking` level, and `λ` override live; the table re-ranks
|
|
441
459
|
on every change. Weights are edited freely (no implicit rescaling): the `Σ`
|
|
@@ -457,8 +475,30 @@ editing source and re-running the CLI.
|
|
|
457
475
|
- **Hover explanations** — every column header, metric name, role tab, and
|
|
458
476
|
control carries a tooltip explaining its semantics.
|
|
459
477
|
|
|
478
|
+
**The per-task price (`$/task`).** The `$/M` column is the reported price — the
|
|
479
|
+
billed 3:1 input:output blend OpenRouter charges, scaled by the role's thinking
|
|
480
|
+
factor. The `$/task` column is what *you* pay per task: your own input:output
|
|
481
|
+
token mix applied to the role's priced route. The mix comes from the plugin's
|
|
482
|
+
own recorded usage (the role's median per-turn counts, from the
|
|
483
|
+
`llm-role-usage.jsonl` ledger the plugin appends to as you work), falling back
|
|
484
|
+
to the `tokenProfile` setting when a role has no recorded usage. It is
|
|
485
|
+
**display-only** — the ranking still sorts on `$/M` — and it reads `—` when no
|
|
486
|
+
profile is available (no recorded usage and no `tokenProfile` set) or the model
|
|
487
|
+
has no usable route price. A model that is cheaper per token but
|
|
488
|
+
more verbose can cost more per task; this column is where that shows up.
|
|
489
|
+
|
|
490
|
+
**The price levers.** The explain panel's **Price levers** section makes the
|
|
491
|
+
price posture visible. The **leader flip** is the price weight `w*` at which a
|
|
492
|
+
cheaper challenger overtakes the current leader, with the challenger named and
|
|
493
|
+
the distance from your current weight — so you can see whether your price
|
|
494
|
+
posture binds (a small bump flips the pick) or is slack. It is absent when no
|
|
495
|
+
cheaper challenger exists. The **thinking level** table shows which model wins
|
|
496
|
+
at each level, with the level's price factor — the factor multiplies the price
|
|
497
|
+
axis, so it penalizes expensive models more, which is why `auto` favours the
|
|
498
|
+
cheap model. Both readouts are computed on the pool you are viewing.
|
|
499
|
+
|
|
460
500
|
The UI never reimplements the ranking math: the numbers on screen are exactly
|
|
461
|
-
the numbers the plugin would use. The server binds `
|
|
501
|
+
the numbers the plugin would use. The server binds `[IP_ADDRESS]` only (no auth)
|
|
462
502
|
and serves the SPA with no build step and no external requests.
|
|
463
503
|
|
|
464
504
|
## Availability
|
package/docs/dev/architecture.md
CHANGED
|
@@ -25,16 +25,17 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
25
25
|
|
|
26
26
|
| File | Role / key exports |
|
|
27
27
|
|---|---|
|
|
28
|
-
| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
|
|
28
|
+
| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, the route-price accessors (`routeInputPrice`, `routeEffectivePrice`, `routeOutputPrice`, `blendOutputPrice`), `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
|
|
29
29
|
| `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `parseBenchmarkPayloadMeta`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `cachedSourceInfo`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`BenchmarkPayloadMeta`/`FocusSourceInfo`/`BenchmarkCatalogEntry`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
|
|
30
|
-
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
30
|
+
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `ResolvedSettings` carries `tokenProfile` (`{ inputTokens, outputTokens } | null`, default `null`) — the user-set per-task token profile the explorer's per-task price falls back to when a role has no recorded usage; validated to finite numbers ≥ 0, schema rows `tokenProfile.inputTokens`/`tokenProfile.outputTokens`. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
31
31
|
| `src/agent-pins.ts` | Agent → pinned-role derivation. Exports `parseAgentPin` (first `@<role>` in the `model:` frontmatter) and `discoverAgentPins` (scans the shipped, user and project agent dirs; project > user > plugin). Drives the `task.disabledAgents` sync. |
|
|
32
32
|
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe, the keyed-catalog availability primitive, and the allowed-providers whitelist harvest. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `fetchAllowedProviders`, `providerAllowed`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
|
|
33
33
|
| `src/config-edit.ts` | Surgical line-oriented YAML patch for `modelRoles` + `retry.fallbackChains` + `task.disabledAgents`, plus the atomic writer. Exports `parseConfig`, `patchConfig`, `writeConfigAtomic`, `ConfigEditError`, and the `ConfigPatch` type. No runtime YAML dependency. |
|
|
34
34
|
| `src/state.ts` | State/history/lock files under the agent dir, and agent-dir resolution. Exports `agentDir`, `loadState`, `saveState`, `appendHistory`, `acquireLock`, `releaseLock`, `freshState`, and the `PluginState` type (which carries `managedDisabledAgents` — the `task.disabledAgents` names the plugin added). |
|
|
35
|
+
| `src/usage-ledger.ts` | The per-role usage ledger (spec #81): an append-only jsonl file (`llm-role-usage.jsonl`, cap `USAGE_LEDGER_CAP = 2000` rows) beside the plugin's other state, one row per assistant turn. Exports `UsageRow`, `TokenProfile`, `USAGE_LEDGER_FILE`, `USAGE_LEDGER_CAP`, `appendUsageRow` (best-effort, never throws), `readUsageRows` (tolerant: missing/malformed → `[]`), `roleTokenProfile` (the role's median per-turn input/output counts via `agent-create.ts`'s `quantile` — the repo's one quantile convention — or `null`), and `resolveTokenProfile` (recorded usage → the `tokenProfile` setting → `null`). The caller owns the dir; the extension records into `resolveProjectDir() ?? agentDir()`, the explorer reads the active scope's dir. |
|
|
35
36
|
| `src/error-report.ts` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role` (the `errorReporting` setting). The plugin's only GitHub client and the only home for a fingerprint, a redaction rule, the agent-dir ledger (`ERROR_REPORT_LEDGER_FILE`) and the issue body. **Entry point:** `reportUnexpectedError(input, deps)` — the feature's one test seam (per-call input + fully injected `deps`; tests supply a fake `fetch`, clock, env, dir and token resolver). Also exports `defaultErrorReportDeps()` (production deps: resolved settings, global `fetch`, real clock, `agentDir`, the shipped manifest, `gh auth token`, the OS opener) and `resetErrorReportGuard()` (the in-process dedupe guard's test hook). Never throws, never rejects; a no-op when `errorReporting` is `off`. |
|
|
36
|
-
| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
37
|
-
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
37
|
+
| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, `resolveProjectDir` (the project scope's `<cwd>/.omp` when it carries a non-empty `omp-llm-role` settings entry, else `null` — the same test the explorer's scope resolution and the extension's usage-ledger dir use), the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
38
|
+
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Records the per-role usage ledger (spec #81) on `turn_end` and `session_shutdown` — one row per assistant turn read from `sessionManager.getBranch()`, into `resolveProjectDir() ?? agentDir()`, deduped by `ts|model` so the two hooks cannot double-record. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
38
39
|
| `src/guards.ts` | The package's one type guard: `isRecord`. |
|
|
39
40
|
| `src/provider-display.ts` | The provider display (spec #77): keep the live session model's `name` suffixed with its upstream provider pin, so omp's model segment renders e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. Pure and dual-runtime (no SDK import). Exports `providerPinOf` (the single `compat.openRouterRouting.only[0]`, else `compat.vercelGatewayRouting.only[0]`; null unless exactly one non-empty string), `providerPinOfSelector` (the trailing `@<slug>` of a selector, after stripping a thinking suffix), `withProviderSuffix` (strip any previous ` @<slug>` then append, idempotent), and `applyProviderDisplay` (mutate a record's `name`; no-op otherwise). |
|
|
40
41
|
| `src/role-settings.ts` | The one validated role write path (validate → merge → backup → atomic write) and its removal half. Exports `validateRole`, `mergeExport`, `writeRoleSettings`, `mergeRemove`, `removeRoleSettings`, `timestamp`. Shared by the explorer's Export, `src/cli/create-role.ts`, `/create-agent` and `/remove-agent`. |
|
|
@@ -43,7 +44,7 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
43
44
|
| `src/agent-architect.ts` | omp's in-process architects: the agent-creation architect (`/create-agent`, `/agents` hub) and the project-profile architect (`/project-roles`). Exports `generateAgentSpec`, `generateProjectProfile`, `parseAgentSpec`, `parseProjectProfile`, `extractJsonObject`, `extractAssistantText`, and the `ArchitectOptions`/`ProjectArchitectOptions` types. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the runner), so the module is importable under plain Node. |
|
|
44
45
|
| `src/benchmark-author.ts` | Benchmark-source authoring for an unknown link: fetch the link, run an in-process architect (`src/prompts/benchmark-source-architect.md`) to propose a declarative source spec, and validate it. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the function). Exports `authorBenchmarkSource`, `parseSourceDeclarationJson`, and the `BenchmarkAuthorOptions` type. |
|
|
45
46
|
| `src/benchmark-discovery.ts` | Judge-backed benchmark relevance for `/create-agent` discovery: one `noul` (yes/no probability) question per catalog candidate, batched into a single judgment through omp's configured `judge` role (TypeSafe jev or its fallback chain). Extension-only (imports `@oh-my-pi/pi-coding-agent` and its `./judgment` subpath — the deeper `./judgment/standalone` path does not resolve under omp's SDK injection). The SDK is imported dynamically so a resolution failure degrades to "no discovery" instead of failing the whole extension load. Exports `judgeBenchmarkRelevance`. |
|
|
46
|
-
| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
|
|
47
|
+
| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `quantile` (the repo's one quantile convention — the focus-axis dispersion stats and the usage ledger's per-turn median), `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
|
|
47
48
|
| `src/agent-remove.ts` | `/remove-agent` core: delete an agent `.md` and its role, plus the shared flag parser and report formatter. Exports `removeAgent`, `parseRemoveAgentArgs`, `formatRemoveAgentReport`, `REMOVE_AGENT_USAGE`, and the request/result types. Refuses shipped default roles. |
|
|
48
49
|
| `src/project-setup.ts` | `/project-roles` core: a discovered `ProjectProfile` → a project-scoped role set (kept shipped roles, dropped ones, new fitted roles) written to the project plugin settings lock file, plus the project agents for the new roles. All-or-nothing (every check — names, metrics, weight math, the `--force` gate, agent-file collisions — before any write), `--dry-run`. Exports `setupProject`, `parseProjectRolesArgs`, `parseRolesSpec`, `applyProfileOverrides`, `formatProjectRolesReport`, `PROJECT_ROLES_USAGE`, and the `ProjectProfile`/`ProposedRole`/`SetupProjectOpts`/`SetupProjectResult` types (the result carries an `archetypes` map). The plan report prints the capability flags each new role will author (`features: <id>=<on>`). Dual-runtime safe (no SDK import); the extension command does the LLM work and calls it. |
|
|
49
50
|
|
|
@@ -51,9 +52,11 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
51
52
|
|
|
52
53
|
| File | Role / key exports |
|
|
53
54
|
|---|---|
|
|
54
|
-
| `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. |
|
|
55
|
-
| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`), `POST /api/
|
|
56
|
-
| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets,
|
|
55
|
+
| `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. `ExplorerBootOpts` carries the as-written pool's inputs (`tier`, `allowedProviders`, `reloadAllowedProviders`), the `tokenProfile` fallback, and `usageDir` (the user-level ledger dir; a project scope reads its own `<root>/.omp` ledger). |
|
|
56
|
+
| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`, `poolMode`, `tokenProfile`), `POST /api/scope`, `POST /api/pool` (switch the ranked pool, raw vs as-written), `POST /api/rank` (rows carry the display-only `perTask`), `POST /api/explain` (carries `leaderFlip`/`thinkingLevels`), `POST /api/export`, `POST /api/refresh`. `ExplorerOpts` gains `getPoolInputs()` and the optional `getUsageRows(scope)`/`tokenProfile` seams. |
|
|
57
|
+
| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, the nine-axis focus assessment, and the display-only per-task price. Exports `rankRows`, `explainModel`, `perTaskPrice`, `inverseCardinal`, `focusMetricsOf`, `focusAssessments`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta`/`PerTaskPrice` types. `rankRows`/`explainModel` take a trailing optional `profile?: TokenProfile \| null`; `RankRow.perTask` and `Explanation.perTask` are `null` when no profile resolves or the model has no usable route price. |
|
|
58
|
+
| `src/explorer/levers.ts` | Pure price-lever readouts (spec #81): the leader-flip threshold and the thinking level's effect on the leader, both computed on the pool they are handed. Exports `leaderFlip`, `thinkingLevelEffect`, and the `LeaderFlip`/`ThinkingLevelRow` types. Reuses the engine's `roleLambda`/`rankRole`/`SUFFIX_LEVELS`/`thinkingPriceFactor` — never re-derives the math. |
|
|
59
|
+
| `src/explorer/pool.ts` | The as-written pool (spec #81): the explorer's dataset filtered exactly as the updater filters it before ranking — the catalog/tier gate (`filterCatalog` + `resolveVariant`) and the allowed-providers whitelist prune (`providerAllowed`). Pure (never mutates its input). Exports `asWrittenPool`, and the `PoolMode`/`PoolInputs` types. |
|
|
57
60
|
|
|
58
61
|
### Assets and non-code
|
|
59
62
|
|
package/docs/dev/ci.md
CHANGED
|
@@ -21,6 +21,15 @@ ref must be green first. `test` runs **exactly** the command the local
|
|
|
21
21
|
pre-commit hook runs (`.githooks/pre-commit`), and `tests/ci.test.ts` fails when
|
|
22
22
|
the two drift apart — "green locally" and "green in CI" cannot diverge.
|
|
23
23
|
|
|
24
|
+
The hook scrubs `GIT_DIR`, `GIT_WORK_TREE`, `GIT_INDEX_FILE` and
|
|
25
|
+
`GIT_OBJECT_DIRECTORY` before running the suite, and `tests/ci.test.ts` scrubs
|
|
26
|
+
the same variables from every child process it spawns. Git exports `GIT_DIR` to
|
|
27
|
+
every hook, and `GIT_DIR` overrides `cwd` — so without the scrub the test file's
|
|
28
|
+
temp-repo `git init`/`config`/`add`/`commit` calls target the **real**
|
|
29
|
+
repository, rewriting its config and moving a branch (issue #89). The test file
|
|
30
|
+
carries the scrub because a test that shells out to git must not depend on its
|
|
31
|
+
caller's environment; the hook carries it as defence in depth.
|
|
32
|
+
|
|
24
33
|
The Node version comes from `.node-version` at the repo root (the exact local
|
|
25
34
|
toolchain; bumping it is a normal commit). `engines` is deliberately not in
|
|
26
35
|
`package.json`: the plugin executes inside omp's runtime, so an `engines` field
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Current state (2026-10-
|
|
1
|
+
# Current state (2026-10-18)
|
|
2
2
|
|
|
3
3
|
Dated snapshot of the ranking and the landed work as of the date in the heading —
|
|
4
4
|
matched/priced/eligible counts, per-role leaders, the open defects, and the
|
|
@@ -6,11 +6,46 @@ per-issue work log below. It is refreshed by
|
|
|
6
6
|
regenerating the report (`node src/cli/llm-role-rank.ts --out docs/llm-role-rankings.md`) and
|
|
7
7
|
copying the new numbers here; release prose belongs in
|
|
8
8
|
[`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The model-level
|
|
9
|
-
ranking numbers below are the 2026-10-
|
|
10
|
-
#21, the #17/#18/#19/#20 wave
|
|
11
|
-
|
|
9
|
+
ranking numbers below are the 2026-10-18 measurement (the input-price blend basis
|
|
10
|
+
#21, the #17/#18/#19/#20 wave, the #34–#40 focus-axis/capability-flag work, and
|
|
11
|
+
spec #81's explorer surfaces — the ranking CLI itself is unchanged); the
|
|
12
|
+
provider-pin numbers are the 2026-10-11 measurement (spec #71; see the top
|
|
12
13
|
bullet). Neither is re-derived on read.
|
|
13
14
|
|
|
15
|
+
- Explorer as-written pool, per-task price, and price levers (2026-10-18, spec
|
|
16
|
+
#81): the explorer gains a **pool toggle** (raw vs as-written), a **per-task
|
|
17
|
+
price column**, and two **price-lever readouts**. The as-written pool applies
|
|
18
|
+
the same two filters the updater applies before ranking — the catalog/tier gate
|
|
19
|
+
and the allowed-providers whitelist prune — so the leader the user sees is the
|
|
20
|
+
model the updater will write (the raw pool's leader differed from the config's
|
|
21
|
+
for 6 of 10 roles). The per-task price is `inputTokens·p_in +
|
|
22
|
+
outputTokens·p_out·thinkingFactor` over the role's priced route, using the
|
|
23
|
+
plugin's own recorded per-role usage (the median per-turn counts) or the
|
|
24
|
+
`tokenProfile` setting; it is **display-only** — the price axis stays `$/M`
|
|
25
|
+
(issue #32's verdict stands) and the reported `$/M` keeps its billed 3:1 blend.
|
|
26
|
+
The readouts are the leader-flip threshold (`λ* = min over cheaper challengers
|
|
27
|
+
X of (q_L − q_X)/(pEff_L − pEff_X)`, `w* = 20λ*/(1+20λ*)`) and the thinking
|
|
28
|
+
level's effect on the leader. A new append-only, capped, best-effort
|
|
29
|
+
**usage ledger** (`llm-role-usage.jsonl`, one row per assistant turn) follows
|
|
30
|
+
the plugin's state dir (`<project>/.omp` in project mode, `agentDir()`
|
|
31
|
+
otherwise). The ranking numbers are unchanged — the ranking CLI does not apply
|
|
32
|
+
the pool toggle, and the per-task price never feeds the sort key.
|
|
33
|
+
- Provider display (2026-10-12, spec #77): when the live session model carries a
|
|
34
|
+
provider pin (a hand-set `providerPin`, or the automatic pin the
|
|
35
|
+
`providerPinning` capability writes), the plugin keeps the model object's
|
|
36
|
+
`name` suffixed with the route, so omp's status-line model segment renders
|
|
37
|
+
e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. The suffix is applied on session
|
|
38
|
+
start, refreshed on the two fallback events, and re-derived at every
|
|
39
|
+
`turn_start` — the re-derive is what heals the display after omp silently
|
|
40
|
+
restores the primary model following a fallback cooldown (no session event is
|
|
41
|
+
emitted), and it also covers a manual `/model` change with a one-turn lag. An
|
|
42
|
+
event's selector is trusted only when it names the live session model, so an
|
|
43
|
+
advisor fallback (the same events, the advisor's model) re-derives from the
|
|
44
|
+
live model instead of stamping the advisor's pin onto the session.
|
|
45
|
+
Display-only: the suffix rides the model object's `name`, so attribution,
|
|
46
|
+
usage recording and model comparison are unaffected. The ranking numbers are
|
|
47
|
+
unchanged.
|
|
48
|
+
|
|
14
49
|
- Provider ordering: value-for-money route score (2026-10-11, spec #71): the
|
|
15
50
|
auto-pin route basis changed from the route's own value (`q_route −
|
|
16
51
|
λ·priceEff_route`) to a min/max-normalized **value-for-money score** `vfm =
|
|
@@ -653,18 +688,18 @@ bullet). Neither is re-derived on read.
|
|
|
653
688
|
0.663, margin 0.039 > `switchMargin`), because the incumbent has no Design
|
|
654
689
|
Arena data and takes the neutral fill while MiMo carries a measured 0.97.
|
|
655
690
|
The other nine roles are untouched (only `designer` weights `website`).
|
|
656
|
-
- 408 llm-stats models; OpenRouter matched
|
|
657
|
-
2026-10-
|
|
691
|
+
- 408 llm-stats models; OpenRouter matched 149/408 (throughput), 148 priced (the
|
|
692
|
+
2026-10-09 snapshot read 155/408 — the daily cache moved, not the code; the
|
|
658
693
|
#34–#40 axes and flags leave every ranking byte-identical until a flag is
|
|
659
694
|
turned on).
|
|
660
|
-
- Eligible per role:
|
|
695
|
+
- Eligible per role: 140 (vision 73, designer 87, image-input filter).
|
|
661
696
|
- Value-ranking leaders (this report, thinking-adjusted prices; the *ranking*
|
|
662
697
|
leaders, before the account's provider whitelist drops the blocked ones):
|
|
663
|
-
`default` DeepSeek-V4.1-Flash (0.
|
|
664
|
-
`slow`
|
|
665
|
-
`plan` Muse Spark 1.3 (0.
|
|
666
|
-
|
|
667
|
-
Muse Spark 1.3 (0.
|
|
698
|
+
`default` DeepSeek-V4.1-Flash (0.807), `smol` DeepSeek-V4.1-Flash (0.706),
|
|
699
|
+
`slow` Gemini 4 Argon (0.834), `vision` Gemini 4 Argon (0.732),
|
|
700
|
+
`plan` Muse Spark 1.3 (0.789), `commit` DeepSeek-V4.1-Flash (0.736), `tiny`
|
|
701
|
+
Mercury 2 (0.797), `task` DeepSeek-V4.1-Flash (0.750), `advisor`
|
|
702
|
+
Muse Spark 1.3 (0.785), `designer` DeepSeek-V4.1-Flash (0.772).
|
|
668
703
|
- Thinking-adjusted pricing landed (2026-09-27): the suffix table moved into
|
|
669
704
|
`DEFAULT_ROLES` as a per-role `thinking` field, and the price axis scales by
|
|
670
705
|
the level's factor for thinking-capable models — `slow` (`:max`, ×7.86)
|
package/docs/dev/explorer.md
CHANGED
|
@@ -12,7 +12,9 @@ explorer never reimplements `value = q − λ·$/M`.
|
|
|
12
12
|
| File | Role |
|
|
13
13
|
|---|---|
|
|
14
14
|
| `src/explorer/server.ts` | Zero-dependency `node:http` surface: static SPA + JSON API. Exports `createExplorerServer`, `ExplorerOpts`. |
|
|
15
|
-
| `src/explorer/explain.ts` | Pure explanation layer (no I/O): rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets,
|
|
15
|
+
| `src/explorer/explain.ts` | Pure explanation layer (no I/O): rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, the keyed-catalog availability overlay, and the display-only per-task price. Exports `rankRows`, `explainModel`, `perTaskPrice`, `inverseCardinal`, `METRIC_META`, `WEIGHTABLE_METRICS`. |
|
|
16
|
+
| `src/explorer/levers.ts` | Pure price-lever readouts (no I/O): the leader-flip threshold and the thinking level's effect on the leader, computed on the pool they are handed. Exports `leaderFlip`, `thinkingLevelEffect`, `LeaderFlip`, `ThinkingLevelRow`. |
|
|
17
|
+
| `src/explorer/pool.ts` | The as-written pool (no I/O): the dataset filtered exactly as the updater filters it before ranking (catalog/tier gate + allowed-providers prune). Exports `asWrittenPool`, `PoolMode`, `PoolInputs`. |
|
|
16
18
|
| `src/explorer/scopes.ts` | Scope resolution (no HTTP): the user-level scope, the registry's projects, and the session's project; the per-scope read; the dataset union. Exports `Scope`, `resolveScopes`, `defaultScopeId`, `resolveScope`, `readScopeRoles`, `unionRoles`. |
|
|
17
19
|
| `src/explorer/boot.ts` | Shared launcher: bind/port fallback, scope resolution, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, `ExplorerHandle`, `ExplorerBootOpts`. |
|
|
18
20
|
| `src/project-registry.ts` | The global project registry (`llm-role-projects.json` in the agent dir): read/register/prune, atomic write. Exports `PROJECT_REGISTRY_FILE`, `readProjectRegistry`, `registerProject`. |
|
|
@@ -47,12 +49,29 @@ type ExplorerOpts = {
|
|
|
47
49
|
availability: KeyAvailability; // keyed-catalog overlay (re-derived on refresh)
|
|
48
50
|
};
|
|
49
51
|
refresh(): Promise<void>; // re-runs loadRankData({ refresh: true, roles, extraMetrics }) and swaps rank
|
|
52
|
+
// The as-written pool's inputs: the omp catalog, the key's tier, and the
|
|
53
|
+
// account's allowed-providers whitelist (null = no whitelist / harvest failed,
|
|
54
|
+
// which leaves every route in place). Re-derived per call, so a project's
|
|
55
|
+
// whitelist cannot leak into another scope's view.
|
|
56
|
+
getPoolInputs(): PoolInputs;
|
|
57
|
+
// The recorded usage rows the per-task price resolves its token profile from,
|
|
58
|
+
// for the request's active scope (a project scope reads its own `<root>/.omp`
|
|
59
|
+
// ledger). Optional: a host without a ledger supplies none.
|
|
60
|
+
getUsageRows?(scope: Scope): readonly UsageRow[];
|
|
61
|
+
// The user-set token profile the per-task column falls back to (the
|
|
62
|
+
// `tokenProfile` setting); null (the default) = unset.
|
|
63
|
+
tokenProfile?: TokenProfile | null;
|
|
50
64
|
};
|
|
51
65
|
```
|
|
52
66
|
|
|
53
|
-
The server holds the active scope id
|
|
54
|
-
`rank`/`availability`); `activeScope` resolves
|
|
55
|
-
fallback when the active scope became unavailable.
|
|
67
|
+
The server holds the active scope id **and the pool mode** as mutable state
|
|
68
|
+
(alongside the mutable `rank`/`availability`); `activeScope` resolves the scope
|
|
69
|
+
per request, persisting the fallback when the active scope became unavailable.
|
|
70
|
+
`poolRank` applies the session's pool mode to the dataset before ranking — the
|
|
71
|
+
`as-written` mode runs `asWrittenPool(rank, opts.getPoolInputs())`, the `raw`
|
|
72
|
+
mode (the default, the historical behaviour) passes the fetch through. The pool
|
|
73
|
+
mode is **never** applied to the export path, which validates a pin against the
|
|
74
|
+
raw dataset.
|
|
56
75
|
|
|
57
76
|
Endpoints:
|
|
58
77
|
|
|
@@ -62,8 +81,9 @@ Endpoints:
|
|
|
62
81
|
| `GET /app.js`, `/style.css` | — | static asset (traversal-guarded) |
|
|
63
82
|
| `GET /api/bootstrap` | — | `bootstrapPayload(opts, activeScope)` |
|
|
64
83
|
| `POST /api/scope` | `{scope}` | switch the active scope, then the bootstrap payload (404 unknown id, 409 non-present) |
|
|
65
|
-
| `POST /api/
|
|
66
|
-
| `POST /api/
|
|
84
|
+
| `POST /api/pool` | `{mode}` | switch the ranked pool (`"raw"` \| `"as-written"`), then the bootstrap payload (400 on any other mode) |
|
|
85
|
+
| `POST /api/rank` | `{role, def}` | `{eligible, lambda, derivedLambda, rows, errors}` — each row carries the display-only `perTask` |
|
|
86
|
+
| `POST /api/explain` | `{role, def, modelId}` | `explainModel(...)` spread + `leaderFlip` + `thinkingLevels` + `errors` |
|
|
67
87
|
| `POST /api/export` | `{roles}` | `{ok:true, backupPath, roles, lockPath}` or `{ok:false, errors}` |
|
|
68
88
|
| `POST /api/refresh` | — | `await opts.refresh()` (re-ranks and re-derives the availability overlay) then the bootstrap payload |
|
|
69
89
|
|
|
@@ -75,7 +95,9 @@ Endpoints:
|
|
|
75
95
|
focus metrics over the resolved dataset), `levels` (`Object.keys(SUFFIX_LEVELS)`), `thinkingFactors`
|
|
76
96
|
(per-level billed-blend multiplier from the engine's own `thinkingPriceFactor`,
|
|
77
97
|
so the UI readout cannot drift), `fetchedAt`, `modelCount`, `orMatched`,
|
|
78
|
-
`orPriced`, `
|
|
98
|
+
`orPriced`, `poolMode` (the ranked pool the counts describe — `"raw"` or
|
|
99
|
+
`"as-written"`), `tokenProfile` (the user-set per-task token profile, or
|
|
100
|
+
`null`), `lockPath` (the **active scope's** lock file), `scopes` (id, label,
|
|
79
101
|
kind, present per known scope), `activeScope` (the active scope id),
|
|
80
102
|
the active scope's global capability flags and the feature registry (the
|
|
81
103
|
`FEATURES` rows: id, label, description, recommended bundle, buys), so the SPA
|
|
@@ -124,28 +146,48 @@ Pure, no I/O; all ranking math is delegated to `src/engine.ts`.
|
|
|
124
146
|
declared?)`** merges the shipped table with a derived entry per external metric.
|
|
125
147
|
- **`weightableMetrics(extra?)`** — the shipped keys plus any external metric in
|
|
126
148
|
`extra`; exactly what the validator accepts.
|
|
127
|
-
- **`rankRows(def, models, baseline, availability?)`** — `rankRole(def, models)` +
|
|
149
|
+
- **`rankRows(def, models, baseline, availability?, profile?)`** — `rankRole(def, models)` +
|
|
128
150
|
`paretoFrontier`, annotating each row with `baselineRank`/`delta`
|
|
129
151
|
(`baselineRank − rank`; positive = moved up) against the effective role's
|
|
130
152
|
ranking. Rows carry `priceEff` (the thinking-adjusted price the ranking
|
|
131
153
|
penalized), `parts`, `missing` (weighted non-price metrics whose raw value is
|
|
132
|
-
null), `frontier`,
|
|
154
|
+
null), `frontier`, `key` — the availability overlay verdict
|
|
133
155
|
(`usable`/`blocked`/`unknown`), orthogonal to `missing` and to the eligibility
|
|
134
|
-
gates
|
|
135
|
-
|
|
156
|
+
gates — and `perTask` (the display-only dollars-per-task, or `null`).
|
|
157
|
+
`availability` and `profile` are optional and pure (no I/O); omitted or
|
|
158
|
+
inactive ⇒ every row reads `unknown` / `perTask: null`. Neither affects the
|
|
159
|
+
ranking order or the deltas.
|
|
136
160
|
- **`inverseCardinal(metric, t)`** — inverse of `cardinalMetric` for the "what
|
|
137
161
|
would it take" targets: `index` → `t*80 − 20`; `throughput` → `10 * 30**t`
|
|
138
162
|
(null outside `[0,1]`, because the forward transform clamps); everything else
|
|
139
163
|
(benchmark, percentile, price) → identity. See *Open defect* below for the
|
|
140
164
|
`gpqa` exception.
|
|
141
|
-
- **`
|
|
165
|
+
- **`perTaskPrice(model, def, profile)`** — the display-only dollars-per-task:
|
|
166
|
+
`p_in` is the role's priced route's input price (cache-adjusted by the role's
|
|
167
|
+
`cacheHitRate`, `routeInputPrice`) and `p_out` that same route's output price —
|
|
168
|
+
the inverse 3:1 blend of it (`routeOutputPrice`), never re-derived here — and
|
|
169
|
+
the output term is scaled by the role's thinking factor
|
|
170
|
+
(`thinkingAdjustedPrice(model, def, 1)`, so the per-task price and the row's
|
|
171
|
+
`priceEff` cannot disagree). When the role prices on the `1/price²` blend
|
|
172
|
+
(`roleRoute` returns `null`) the model's billed blend and the input blend over
|
|
173
|
+
the routes that survive the role's endpoint filters stand in — the same pool the
|
|
174
|
+
ranking prices on, so `p_in` and the billed blend never mix two route pools.
|
|
175
|
+
`null` when there is no usable route price — **absent, never a silent
|
|
176
|
+
zero**. The SPA renders `—` in both cases: no token profile resolves (no
|
|
177
|
+
recorded usage and no `tokenProfile` set), **or** the model has no usable
|
|
178
|
+
route price (e.g. every route pruned by the as-written whitelist — the model
|
|
179
|
+
still ranks on its blend price, but the per-task column has no route to
|
|
180
|
+
price). It never feeds the ranking's sort key, `value`, or the explanation's
|
|
181
|
+
`cost` block.
|
|
182
|
+
- **`explainModel(def, models, modelId, roleName, availability?, profile?)`** — the full
|
|
142
183
|
decomposition, or the eligibility gates it failed (same gates `rankRole`
|
|
143
184
|
applies, same order: missing required, image filter, no billed price). On
|
|
144
185
|
success: `contributions`
|
|
145
186
|
(raw → cardinal `t` → renormalized weight → contribution → share of `q`, with
|
|
146
187
|
a `fillNote` when the value is a capability fill), `cost` (`priceEff`, the
|
|
147
188
|
cache-adjusted billed price, λ, penalty, `q`, `value`), `role.cacheHitRate`,
|
|
148
|
-
`
|
|
189
|
+
`perTask` (the `PerTaskPrice` breakdown, or `null` when no profile was
|
|
190
|
+
supplied or the model has no usable route price), `gapAbove`/`gapToTop`/`above`, `closing` (the
|
|
149
191
|
per-metric raw target that would close the gap, via `inverseCardinal`, with
|
|
150
192
|
unreachable/extrapolated notes), `dominators` (models both cheaper and at
|
|
151
193
|
least as good on `q`, top 3), `ownProvider` (true when the model was priced on
|
|
@@ -156,6 +198,15 @@ Pure, no I/O; all ranking math is delegated to `src/engine.ts`.
|
|
|
156
198
|
availability overlay `key`/`keyReason`
|
|
157
199
|
(the same verdict as `rankRows`, plus the reason it reads that way). It mirrors
|
|
158
200
|
`rankRole`'s capability fill so the contributions sum exactly to `q`.
|
|
201
|
+
- **`leaderFlip(def, models)`** / **`thinkingLevelEffect(def, models)`**
|
|
202
|
+
(`src/explorer/levers.ts`) — the two price-lever readouts, computed on the pool
|
|
203
|
+
they are handed (so the readout and the ranking above it agree). `leaderFlip`
|
|
204
|
+
returns the price weight at which a cheaper challenger overtakes the value
|
|
205
|
+
leader (`{ wPrice, flipW, distance, challenger }`), or `null` when no cheaper
|
|
206
|
+
challenger exists. `thinkingLevelEffect` returns one `{ level, factor, leader }`
|
|
207
|
+
row per thinking level, or `null` for a bare role. Both reuse the engine's own
|
|
208
|
+
`roleLambda`/`rankRole`/`SUFFIX_LEVELS`/`thinkingPriceFactor`; the formulas are
|
|
209
|
+
in [`scoring.md`](scoring.md).
|
|
159
210
|
|
|
160
211
|
## Shared boot path (`src/explorer/boot.ts`)
|
|
161
212
|
|
|
@@ -202,6 +253,22 @@ launcher `/explore-roles` uses.
|
|
|
202
253
|
factor on the omp catalog; `refresh()` re-runs `opts.reload(true)`,
|
|
203
254
|
re-enriches, and — when `opts.reloadAvailability` is supplied — re-derives the
|
|
204
255
|
availability overlay.
|
|
256
|
+
- **As-written pool inputs.** `ExplorerBootOpts` carries `tier` (default
|
|
257
|
+
`"billed"`), `allowedProviders` (`null` = no pruning), and
|
|
258
|
+
`reloadAllowedProviders` (re-derived on `refresh()`). `getPoolInputs()` returns
|
|
259
|
+
`{ catalog: opts.catalog, tier, allowed }` per call, so a project's whitelist
|
|
260
|
+
cannot leak into another scope's view. Both are **best-effort**: an absent tier
|
|
261
|
+
reads as `billed`, and a failed re-harvest (`reloadAllowedProviders` throwing
|
|
262
|
+
or returning `null`) degrades the as-written pool to no route pruning rather
|
|
263
|
+
than failing the refresh. `/explore-roles` builds them in-process from the
|
|
264
|
+
session's OpenRouter key (the canary must be a **catalog** slug, not a ranking
|
|
265
|
+
id).
|
|
266
|
+
- **Usage ledger.** `getUsageRows(scope)` reads the active scope's ledger: a
|
|
267
|
+
project scope reads its own `<root>/.omp` ledger, the user scope reads
|
|
268
|
+
`usageDir` (default `agentDir()`), so a project's usage never mixes with the
|
|
269
|
+
user-level profile. Re-read per call, so a turn recorded while the explorer is
|
|
270
|
+
open shows up without a restart. `tokenProfile` is the user-set fallback the
|
|
271
|
+
per-task column uses when a role has no recorded usage.
|
|
205
272
|
- `opts.availability` is the boot-time keyed-catalog overlay; `getState(scope)`
|
|
206
273
|
returns it on every request, so `/api/rank` and `/api/explain` annotate rows
|
|
207
274
|
without a second fetch. It is scope-independent and carried across switches
|
|
@@ -246,12 +313,23 @@ a scraped third-party site, so the DOM is built with
|
|
|
246
313
|
`textContent`/`createElement` only (the `el()` helper).
|
|
247
314
|
|
|
248
315
|
- **Boot**: `boot()` → `GET /api/bootstrap` → `applyBootstrap(data)`, which fills
|
|
249
|
-
`state` (`scopes`, `activeScope`, `
|
|
316
|
+
`state` (`scopes`, `activeScope`, `poolMode`, `tokenProfile`, `universe`,
|
|
317
|
+
`effective` = each entry's `def`,
|
|
250
318
|
`defs` = a structuredClone of it, `defaults`, `metrics`, `metricMeta`,
|
|
251
319
|
`focusAssessments`, `levels`, `thinkingFactors`, `lockPath`, `availability`),
|
|
252
320
|
renders the header/roles/table, and selects the previously selected role when
|
|
253
321
|
the new scope has it, else the first role. `boot()` then wires the
|
|
254
322
|
filter/topn/hide-key-blocked/export/copy/download controls.
|
|
323
|
+
- **Pool toggle**: the header renders a `#pool` `<select>` (`raw` / `as-written`)
|
|
324
|
+
from `state.poolMode`. Changing it confirms first when there are unsaved edits
|
|
325
|
+
(mirroring the scope switcher), then `POST /api/pool {mode}` and
|
|
326
|
+
`applyBootstrap` on the returned payload — so the header's model count, the
|
|
327
|
+
table, the focus assessments and the two readouts all describe the pool the
|
|
328
|
+
user is now looking at. The toggle is **session-level** state — the server
|
|
329
|
+
holds one pool mode, kept across scope switches (not persisted) — while the
|
|
330
|
+
pool *inputs* are re-derived per call, so a project's whitelist never leaks
|
|
331
|
+
into another scope's view. Export is unaffected: it always
|
|
332
|
+
writes the authored def, never the pool.
|
|
255
333
|
- **Scope switcher**: the header renders a `#scope` `<select>` from
|
|
256
334
|
`state.scopes` (the user-level scope plus every known project); a scope whose
|
|
257
335
|
lock file is gone is `disabled` with a reason `title`. Changing the selection
|
|
@@ -282,15 +360,24 @@ a scraped third-party site, so the DOM is built with
|
|
|
282
360
|
- **Recompute**: `selectRole` → `renderRoles`/`renderEditor`/`renderExplain` +
|
|
283
361
|
`recompute()`. `recompute()` → `POST /api/rank {role, def}` → `state.rows`,
|
|
284
362
|
`lambda`, `derivedLambda`, `errors`; re-renders the table and readouts, and
|
|
285
|
-
reloads the explain panel if the selected model is still present.
|
|
286
|
-
|
|
363
|
+
reloads the explain panel if the selected model is still present. The table
|
|
364
|
+
gains a **`$/task`** column (`r.perTask`, rendered `—` when `null`), beside the
|
|
365
|
+
`$/M` column. Editor changes call `scheduleRecompute()` (120 ms debounce).
|
|
287
366
|
- **Explain**: `selectModel(id)` → `POST /api/explain {role, def, modelId}` →
|
|
288
|
-
`renderExplain()` (composition table, cost line,
|
|
289
|
-
dominators,
|
|
367
|
+
`renderExplain()` (composition table, cost line, **per-task price** line, "why
|
|
368
|
+
not higher", dominators, the **Price levers** section, and the availability
|
|
369
|
+
line). The cost line appends `· best route`
|
|
290
370
|
when the model was priced on the soft best route (`ownProvider`), and a
|
|
291
371
|
**pin line** shows the pin in force: `pin: <slug> (manual)` when the edited def
|
|
292
372
|
carries a `providerPin`, else `auto pin: <slug>` from the explanation's
|
|
293
|
-
`autoPin` (read-only — the manual field is the only way to set one).
|
|
373
|
+
`autoPin` (read-only — the manual field is the only way to set one). The
|
|
374
|
+
per-task line shows the breakdown (`<in> in × $<p_in>/M + <out> out ×
|
|
375
|
+
$<p_out>/M × <factor> = $<total>`), or `— (no token profile — record usage or
|
|
376
|
+
set tokenProfile)` when `perTask` is `null`. The **Price levers** section
|
|
377
|
+
renders the leader-flip line (`w* = <flipW> · now <wPrice> · distance
|
|
378
|
+
<±distance> (<ratio>)`) and a per-level table of the thinking level's effect on
|
|
379
|
+
the leader; each is omitted when the server sends `null` (no cheaper
|
|
380
|
+
challenger, or a bare role).
|
|
294
381
|
- **Provider pin field**: the editor renders a text input for
|
|
295
382
|
`roles.<role>.providerPin` (a tiered slug like `deepinfra/fp8` is kept
|
|
296
383
|
verbatim; empty clears the key, resuming the automatic choice). It is the
|
package/docs/dev/releasing.md
CHANGED
|
@@ -122,6 +122,35 @@ path, plus `marketplace update` + `upgrade` to a bumped catalog version), and
|
|
|
122
122
|
push publishes to npm over OIDC and creates the GitHub Release. The guard was
|
|
123
123
|
verified with a negative control — tag `v0.0.1` against version `1.1.0` failed
|
|
124
124
|
in `notes` before any publish.
|
|
125
|
+
- **1.2.0 published by CD 2026-10-12** (run 38299141569, tag push): the guard
|
|
126
|
+
and the npm floor passed, `npm publish` completed over OIDC trusted publishing
|
|
127
|
+
with a provenance attestation (`npm audit signatures`: 1 package has a
|
|
128
|
+
verified attestation), and the GitHub Release was created with the 1.2.0
|
|
129
|
+
CHANGELOG section as its body
|
|
130
|
+
(`https://github.com/Gott50/omp-llm-role/releases/tag/v1.2.0`). Verified:
|
|
131
|
+
`dist-tags.latest` = `1.2.0`, npm's `gitHead` = `50fce66` (the tag's commit),
|
|
132
|
+
and the publish step's log carries the direct-publish signature (`Publishing
|
|
133
|
+
to https://registry.npmjs.org/ with tag latest`, `Signed provenance
|
|
134
|
+
statement`, `+ omp-llm-role@1.2.0`). The cut (`50fce66`, tagged `v1.2.0`)
|
|
135
|
+
bumped the four lockstep fields, renamed `## [Unreleased]` to
|
|
136
|
+
`## [1.2.0] - 2026-10-12`, and regenerated the report (408 models, matched
|
|
137
|
+
150/408, priced 148; `docs/dev/current-state.md` refreshed to the same
|
|
138
|
+
measurement).
|
|
139
|
+
- **1.2.0 install routes smoke-tested** (2026-10-12, omp 18.10.3, throwaway
|
|
140
|
+
`HOME`s, cwd outside the real home tree): npm
|
|
141
|
+
(`omp plugin install omp-llm-role`), git
|
|
142
|
+
(`omp plugin install github:Gott50/omp-llm-role`) and marketplace
|
|
143
|
+
(`omp plugin marketplace add Gott50/omp-llm-role` +
|
|
144
|
+
`omp plugin install omp-llm-role@gott50-plugins`) each installed 1.2.0
|
|
145
|
+
(`omp plugin doctor`: 4 ok / 0 warnings / 0 errors for npm and git; the
|
|
146
|
+
marketplace install's one warning is the expected
|
|
147
|
+
`package_manifest: Not created yet`). Each post-install session run
|
|
148
|
+
(`omp -p "say ok"` with `OPENROUTER_API_KEY` from `omp token openrouter`)
|
|
149
|
+
wrote `modelRoles` (9 roles), `retry.fallbackChains` and
|
|
150
|
+
`task.disabledAgents: [designer]` into that HOME's `config.yml`, landed
|
|
151
|
+
`llm-role-state.json` + `llm-role-history.jsonl`, and answered. The npm-route
|
|
152
|
+
run also exercised the spec #65 whitelist harvest live (`OpenRouter
|
|
153
|
+
allowed-providers whitelist active — 11 providers; 542 routes pruned`).
|
|
125
154
|
- **1.1.0 published by CD 2026-10-09** (run 37897647817,
|
|
126
155
|
`gh workflow run ci.yml --ref v1.1.0`): the guard and the npm floor passed,
|
|
127
156
|
`npm publish` completed over OIDC trusted publishing with a provenance
|
|
@@ -171,6 +200,7 @@ path, plus `marketplace update` + `upgrade` to a bumped catalog version), and
|
|
|
171
200
|
- **Stray registry placeholder.** The registry also carries a `0.0.0-stage`
|
|
172
201
|
version ("Temporary package placeholder for staged publishing", 334 B,
|
|
173
202
|
published 2026-10-03). It is not from this tree and does not affect installs
|
|
174
|
-
(`latest` is `1.
|
|
175
|
-
- **Next release:
|
|
176
|
-
|
|
203
|
+
(`latest` is `1.2.0`).
|
|
204
|
+
- **Next release: none owed.** `## [Unreleased]` is empty — the 1.2.0 cut moved
|
|
205
|
+
its entries under `## [1.2.0] - 2026-10-12`; the number comes from the
|
|
206
|
+
Unreleased scope, not from the open issues.
|
package/docs/dev/scoring.md
CHANGED
|
@@ -101,10 +101,13 @@ cache reads; the knob prices that, opt-in, without touching any shipped role.
|
|
|
101
101
|
- `endpointFilteredModel(model, filters, cacheHitRate)` recomputes the blend when
|
|
102
102
|
the role declares endpoint filters **or** a non-zero hit rate; the hit rate
|
|
103
103
|
composes with the filters (filter the pool first, then cache-price it);
|
|
104
|
-
- `
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
104
|
+
- the route-price accessors — `routeInputPrice` (the cache-adjusted input term),
|
|
105
|
+
`routeEffectivePrice` (the billed 3:1 blend, which reuses the input accessor),
|
|
106
|
+
`routeOutputPrice`/`blendOutputPrice` (the inverse blend) — are what the
|
|
107
|
+
blend, the pinned-route path, `roleMetricValue` and the explorer's per-task
|
|
108
|
+
price share; a pinned role prices its route's cache-adjusted billed price, and
|
|
109
|
+
`maxPriceUsdPerM` caps the thinking-adjusted `priceEff` (the same cap as the
|
|
110
|
+
blend path);
|
|
108
111
|
- `explainModel` mirrors all of it, carries `role.cacheHitRate`, and reports the
|
|
109
112
|
cache-adjusted `cost.billedPrice`; the SPA cost line appends `· cache <rate>`
|
|
110
113
|
when the rate is set;
|
|
@@ -191,6 +194,54 @@ filter, not a re-pricing: the role-independent `1/price²` blend (`m.price`/
|
|
|
191
194
|
`m.throughput`) is left as the account-agnostic expected value, and a model with
|
|
192
195
|
no allowed route keeps that blend (the soft basis never drops a model).
|
|
193
196
|
|
|
197
|
+
## Price levers (the explorer's readouts)
|
|
198
|
+
|
|
199
|
+
Two readouts make the price posture visible (spec #81). Both are pure
|
|
200
|
+
(`src/explorer/levers.ts`), computed on the pool the explorer is viewing, and
|
|
201
|
+
reuse the engine's own `roleLambda`/`rankRole`/`SUFFIX_LEVELS`/
|
|
202
|
+
`thinkingPriceFactor` — never a re-derivation.
|
|
203
|
+
|
|
204
|
+
### The leader-flip threshold
|
|
205
|
+
|
|
206
|
+
The price weight at which a **cheaper** challenger overtakes the value leader.
|
|
207
|
+
Let `L = rankRole(def, models)[0]` (the value leader). For every challenger `X`
|
|
208
|
+
with `pEff_X < pEff_L` (strictly cheaper), the flip λ is
|
|
209
|
+
`(q_L − q_X)/(pEff_L − pEff_X)`; the threshold is the minimum over those
|
|
210
|
+
challengers:
|
|
211
|
+
|
|
212
|
+
```
|
|
213
|
+
λ* = min over cheaper challengers X of (q_L − q_X)/(pEff_L − pEff_X)
|
|
214
|
+
w* = 20·λ* / (1 + 20·λ*)
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
`w*` is the inverse of `roleLambda`'s `λ = w/(1−w)/20` (`P_REF_USD = 20`). The
|
|
218
|
+
readout reports the role's current `wPrice`, the flip `w*`, the distance
|
|
219
|
+
`w* − wPrice`, and the challenger that would take over. **No cheaper challenger
|
|
220
|
+
⇒ `null`** — the readout is absent, never invented. The degenerate case (the
|
|
221
|
+
value leader is also the quality leader, where the naive
|
|
222
|
+
`(q_QL − q_VL)/(p_QL − p_VL)` is `0/0`) is handled by this same min formula: it
|
|
223
|
+
is *undefined*, not zero.
|
|
224
|
+
|
|
225
|
+
**Exactness.** The threshold is exact when the role's price basis is
|
|
226
|
+
weight-independent — the `1/price²` blend, or a hard `providerPin`. Under the
|
|
227
|
+
soft `preferOwnProvider` basis the route choice itself moves with the price
|
|
228
|
+
weight, so the threshold is a **first-order estimate**: measured on `smol`,
|
|
229
|
+
moving `w` from 0.45 to 0.9586 moves the leader's route `coreweave/fp8 →
|
|
230
|
+
decart/fp4`, so the flip is approximate there.
|
|
231
|
+
|
|
232
|
+
### The thinking level's effect
|
|
233
|
+
|
|
234
|
+
`thinkingLevelEffect(def, models)` re-ranks the role at each thinking level and
|
|
235
|
+
reports the leader per level, with the level's price factor
|
|
236
|
+
`thinkingPriceFactor(level) = (1 + 3·0.25 + THINKING_TOKEN_OVERHEAD[level]) /
|
|
237
|
+
(1 + 3·0.25)`. It is **inert for a bare role** (no thinking level) — the readout
|
|
238
|
+
is `null`. The factor is a **price-sensitivity amplifier, not a hidden cost**: it
|
|
239
|
+
is multiplicative on `priceEff`, so it penalizes expensive models more in
|
|
240
|
+
absolute λ-units — which is why `auto` favours the cheap model. On the user's
|
|
241
|
+
live `default` def: `off` → Gemini 4 Argon $2.99, `auto` → Gemini 4 Argon $5.55,
|
|
242
|
+
`high` → DeepSeek-V4.1-Flash $0.17. The readout shows this effect; it never
|
|
243
|
+
frames the level as a cost to remove.
|
|
244
|
+
|
|
194
245
|
## Capability presets (feature flags)
|
|
195
246
|
|
|
196
247
|
One named flag per opt-in capability (`features.<id>` global,
|