model-compare-opencode-plugin 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -8
- package/dist/tui.js +18 -6
- package/package.json +1 -1
- package/src/core/catalog.ts +98 -16
- package/src/core/reasoning.ts +96 -0
- package/src/core/steering.ts +36 -0
- package/src/core/types.ts +34 -1
- package/src/index.ts +41 -14
- package/src/opencode/aliases.ts +19 -5
- package/src/opencode/commands.ts +41 -3
- package/src/opencode/guard.ts +4 -2
- package/src/opencode/usebest.ts +13 -3
- package/src/rpc.ts +30 -2
- package/src/tui.tsx +24 -4
- package/vendor/UPSTREAM.json +3 -3
- package/vendor/model_compare.py +2140 -430
package/README.md
CHANGED
|
@@ -10,7 +10,7 @@ Each priority is a live alias inside the existing OpenRouter provider — no new
|
|
|
10
10
|
| `openrouter/model-compare/price` | best price pick |
|
|
11
11
|
| `openrouter/model-compare/quality` | best quality pick |
|
|
12
12
|
|
|
13
|
-
The alias re-points at the current pick while OpenCode runs (refresh every 30 minutes, data cache 6 hours). OpenCode's model picker shows it as `Model Compare: <priority> → <pick
|
|
13
|
+
The alias re-points at the current pick while OpenCode runs (refresh every 30 minutes, data cache 6 hours). It also pins the catalog's scored reasoning effort on each request, so the model runs at the depth its `quality` score was measured at. OpenCode's model picker shows it as `Model Compare: <priority> → <pick> · <effort> [alt <effort> <index>]`, with `(fallback)` while the fallback is serving, `(unavailable)` when the alias is disabled, and the `alt` part naming the best measured effort above the current one that OpenRouter can actually select.
|
|
14
14
|
|
|
15
15
|
|
|
16
16
|
## Install
|
|
@@ -69,7 +69,7 @@ opencode run --model "openrouter/model-compare/price" "msg"
|
|
|
69
69
|
Commands:
|
|
70
70
|
|
|
71
71
|
- `/use-best [priority]` — switch the session to `openrouter/model-compare/<p>` (defaults to `defaultPriority`). Refuses a disabled alias.
|
|
72
|
-
- `/model-compare` — top 5 per priority with scores and data age;
|
|
72
|
+
- `/model-compare` — top 5 per priority with scores and data age; each line also shows the reasoning effort behind `quality` (`· high`) and the best selectable measured alternative (`· alt max 56.0`). Priorities whose `weights` override actually takes effect are labelled `(custom weights)` with recomputed scores, and each alias's current pick is marked `← current pick`. The output is a synthetic message (`resume: false`): it costs no model turn and, in a headless session, surfaces in the transcript at the next turn. In the TUI it opens the report dialog instead, with `effort` and `alt` columns.
|
|
73
73
|
|
|
74
74
|
|
|
75
75
|
### Keyboard shortcuts
|
|
@@ -85,7 +85,7 @@ Commands:
|
|
|
85
85
|
|
|
86
86
|
Both dialogs open anywhere — including the opening screen, where no session exists yet. There, picking a row (a priority in the picks dialog, a model in the report) creates a new session already bound to that model and opens it; merely opening a dialog never creates a session. With a session open, a pick switches that session as before. The TUI has no `/use-best` keybinding: the slash command already covers switching.
|
|
87
87
|
|
|
88
|
-
Status: the footer shows `MC: balanced → <pick>` (balanced only — `disabled`/`unavailable`/`(fallback)` as applicable; all three priorities live in the picks dialog). Toasts announce `/use-best` outcomes and pick changes (never for the first observation after load), the `alt+e` served endpoint, and the cannot-steer notice — see [Steering and served endpoints](#steering-and-served-endpoints). Headless, command feedback is synthetic transcript messages at the next turn; the steering toasts have no headless surface.
|
|
88
|
+
Status: the footer shows `MC: balanced → <pick>` (balanced only — `disabled`/`unavailable`/`(fallback)` as applicable; all three priorities live in the picks dialog). The picker's alias name carries the effort and alternative; the footer stays a compact served-model line. Toasts announce `/use-best` outcomes and pick changes (never for the first observation after load), the `alt+e` served endpoint, and the cannot-steer notice — see [Steering and served endpoints](#steering-and-served-endpoints). Headless, command feedback is synthetic transcript messages at the next turn; the steering toasts have no headless surface.
|
|
89
89
|
|
|
90
90
|
|
|
91
91
|
## Configuration
|
|
@@ -155,18 +155,18 @@ The alt+e capture is best-effort by design, and alias-scoped: only a session who
|
|
|
155
155
|
|
|
156
156
|
The last validated catalog is persisted, so ranking and `/model-compare` work offline after a restart. The cache is keyed by source (URL or script path), so changing the source never serves the previous source's data. Script exit code 2 means "no candidates" *or* an argparse usage error — never read it as "no candidates" alone.
|
|
157
157
|
|
|
158
|
-
**Vendored pin:** `vendor/model_compare.py` is byte-identical to upstream `v0.
|
|
158
|
+
**Vendored pin:** `vendor/model_compare.py` is byte-identical to upstream `v0.6.0` (`vendor/UPSTREAM.json` records repo, tag, commit, sha256). The catalog is schema v5 (since upstream 0.6.0): `quality` is the AA intelligence index at the model's effective OpenRouter default reasoning effort, and every entry carries `quality_effort` plus a `reasoning` block (`default_effort`, the expanded `supported_efforts`, the raw flags, and the measured `depths` with AA `$/task`). The plugin accepts schema v5 ONLY — `validateCatalog` rejects anything else — and releases are gated on the live remote serving v5 (the release-gate check pipes `https://model-compare.dev/catalog.json` through `scripts/canary-check.mjs`, which fails closed when the served document is not the expected schema). Each alias request pins exactly the pick's `quality_effort` in its body (`reasoning: { effort }`), so the alias runs at the depth its score represents; an explicit `#variant` selection, or a body that already sets `reasoning`/`reasoning_effort`, wins and is never overwritten. The catalog's `reasoning` sentinel is display-only and never sent to the gateway. Effective-tier pricing semantics still apply: `pricing` scalars hold effective-tier values — the tier in effect at 1M-token prompts — so models with OpenRouter price overrides are ranked at what a long session actually pays.
|
|
159
159
|
|
|
160
|
-
**Vendored 0.
|
|
160
|
+
**Vendored 0.6.0 cold-cache network profile:** since 0.4.0 the vendored script performs per-model endpoint-pricing fetches when building a fresh catalog (76 candidates in the recorded run). Each fetch fails closed per model — after the script's internal retries, a failed endpoint fetch drops that model and the catalog shrinks silently (429-class responses raise without retry) — so a degraded network costs pool coverage, not pricing correctness. Unkeyed runs hydrate quality from the published catalog (one fetch of `https://model-compare.dev/catalog.json`, cached under `published-catalog`; keyed runs consult the AA API — the primary quality source — and never fetch the published catalog); every quality hydration fails closed to "no quality data" per model, and a missing record at the selected effort shows `-` rather than borrowing another effort's score. A cold cache under a slow network can exceed the plugin's 60-second local budget, in which case the plugin warns and falls back (`auto` → remote, `local` → its cached document).
|
|
161
161
|
|
|
162
162
|
**Upgrading the pin** — reviewed PRs only; nothing merges automatically:
|
|
163
163
|
|
|
164
164
|
1. `node scripts/sync-vendor.mjs <tag>` — fetches the script at that commit, reports the sha256 change, re-pins `UPSTREAM.json`, prints the matching CHANGELOG section (`--check` verifies offline).
|
|
165
|
-
2. Conformance tests check `--version`, schema
|
|
165
|
+
2. Conformance tests check `--version`, schema v5, required fields, recomputed `overall` scores, and the standard-library-only import allowlist.
|
|
166
166
|
3. The upgrade PR body lists script lines touching imports, URLs, environment variables, subprocess calls and file writes, for reviewer attention.
|
|
167
167
|
4. Release with a changelog line naming the upstream tag; users run `opencode plugin update` (OpenCode checks but doesn't apply).
|
|
168
168
|
|
|
169
|
-
A scheduled workflow opens a draft upgrade PR when upstream publishes a newer release (never a downgrade); a nightly canary exercises the vendored script against live upstream data and asserts the generated catalog is schema
|
|
169
|
+
A scheduled workflow opens a draft upgrade PR when upstream publishes a newer release (never a downgrade); a nightly canary exercises the vendored script against live upstream data and asserts the generated catalog is schema v5 (`scripts/canary-check.mjs` pins `EXPECTED_SCHEMA_VERSION = 5` and exits nonzero on any other version); releases additionally gate on the live remote itself (see the publishing gates below); a weekly drift check compares the vendored sha256 against upstream's pinned commit and the published rankings against the published scores.
|
|
170
170
|
|
|
171
171
|
|
|
172
172
|
## Troubleshooting
|
|
@@ -210,7 +210,7 @@ MC_INTEGRATION=1 npx vitest run tests/integration/ # live-server suite (opt-i
|
|
|
210
210
|
- The integration files skip unless `MC_INTEGRATION=1`, so plain `npm test` stays hermetic. The suite drives a real `opencode serve` on 127.0.0.1:14721 and needs the `opencode` CLI and `python3` (skips cleanly when either is missing); one case needs network access and the `sqlite3` CLI.
|
|
211
211
|
- `node scripts/pack-smoke.mjs` (also in CI, after `node scripts/build-tui.mjs`) packs the plugin and checks the tarball against the packaging list (files, `exports`, relative imports, and that the TUI artifact is precompiled — no react import). It does **not** load the package in OpenCode — install the tarball into a scratch project and confirm OpenCode loads it before publishing.
|
|
212
212
|
- Publishing: cut a GitHub release tagged `vX.Y.Z` and `.github/workflows/release.yaml` stages the package on npm through trusted publishing (tokenless OIDC — no token secret; the npmjs.com trusted-publisher entry must name `release.yaml`, the `canonical/model-compare-opencode-plugin` repository and the `release` environment, with "Allow npm publish" and "Allow npm dist-tag" unchecked — stage-only, npm's recommended posture). Nothing goes live until a maintainer approves it with 2FA on npmjs.com (package page → Staged Packages) or via `npm stage approve`. The workflow fails closed when the tag does not match `package.json`'s version, and the gates (build, typecheck, tests, pack-smoke, live-remote canary) run before staging.
|
|
213
|
-
- `fixtures/catalog-v0.
|
|
213
|
+
- `fixtures/catalog-v0.6.0.json` is the sole contract fixture — a recorded schema-v5 catalog; `fixtures/catalog-no-quality.json` is derived from it by `scripts/make-no-quality-fixture.py` (the vendored script's own quality-blind weights, so its scores are internally consistent). No pre-v5 fixture exists; the v5-only gate rejects one. `node scripts/sync-vendor.mjs --check` verifies the pin's sha256 offline.
|
|
214
214
|
- Repo layout: `src/core/` (pure ranking/pinning), `src/sources/` (local + remote acquisition), `src/opencode/` (aliases, guard, commands, storage, refresh), `dist/` (generated `tui.js` artifact, gitignored), `vendor/` (pinned upstream script), `tests/{unit,conformance,integration}`.
|
|
215
215
|
|
|
216
216
|
|
package/dist/tui.js
CHANGED
|
@@ -157,6 +157,8 @@ function toReportEntries(raw) {
|
|
|
157
157
|
value: entry.value,
|
|
158
158
|
score: typeof entry.score === "string" ? entry.score : "-",
|
|
159
159
|
quality: typeof entry.quality === "string" ? entry.quality : "-",
|
|
160
|
+
effort: typeof entry.effort === "string" ? entry.effort : "",
|
|
161
|
+
alternative: typeof entry.alternative === "string" ? entry.alternative : "",
|
|
160
162
|
price: typeof entry.price === "string" ? entry.price : "-",
|
|
161
163
|
current: entry.current === true,
|
|
162
164
|
fromFallback: entry.fromFallback === true
|
|
@@ -708,8 +710,15 @@ export default Plugin.define({
|
|
|
708
710
|
const labels = data.entries.map(entry => `${entry.rank === null ? "·" : entry.rank}. ${entry.id}${entry.fromFallback ? " (fallback)" : ""}`);
|
|
709
711
|
const labelOf = new Map(data.entries.map((entry, index) => [entry, labels[index]]));
|
|
710
712
|
const colW = (values, floor, cap) => Math.min(cap, Math.max(floor, ...values.map(value => value.length)));
|
|
711
|
-
|
|
712
|
-
|
|
713
|
+
// Quality is a one-decimal AA index (max "100.0"); the caps leave the
|
|
714
|
+
// 52-char row budget intact (see `fixed` below) while fitting real
|
|
715
|
+
// values: "minimal" (effort), "minimal 100.0" (alt), "10.00/100.00".
|
|
716
|
+
const qualW = colW(data.entries.map(entry => entry.quality), "qual".length, 5);
|
|
717
|
+
// Catalog v5: the effort the quality was measured at and the best
|
|
718
|
+
// selectable measured alternative ("max 56.0"). A missing value is "-".
|
|
719
|
+
const effortW = colW(data.entries.map(entry => entry.effort || "-"), "effort".length, 7);
|
|
720
|
+
const altW = colW(data.entries.map(entry => entry.alternative || "-"), "alt".length, 13);
|
|
721
|
+
const priceW = colW(data.entries.map(entry => entry.price), "$ in/out".length, 13);
|
|
713
722
|
// A cell wider than its clamped column is truncated to fit.
|
|
714
723
|
const cell = (value, width) => value.length > width ? value.slice(0, width - 1) + "…" : value;
|
|
715
724
|
// Prices use padEnd (not padStart): padded to a common width the
|
|
@@ -717,10 +726,13 @@ export default Plugin.define({
|
|
|
717
726
|
// stays exactly 2 (padStart pushed each value right by its slack).
|
|
718
727
|
// The default dialog is 60 wide: 2 list insets + 6 row insets leave 52
|
|
719
728
|
// for row content; the header shares that budget (plus its 3-indent).
|
|
720
|
-
|
|
721
|
-
|
|
729
|
+
// Column caps keep the budget even for hostile strings: 5+7+13+13 plus
|
|
730
|
+
// the four 2-char paddings and the 2-char row indent = 46, leaving the
|
|
731
|
+
// name floor of 6 (the name column shrinks first for real content).
|
|
732
|
+
const fixed = 2 + qualW + 2 + effortW + 2 + altW + 2 + priceW;
|
|
733
|
+
const nameW = Math.max(6, Math.min(52 - fixed, Math.max(...labels.map(label => label.length))));
|
|
722
734
|
const trim = label => label.length > nameW ? label.slice(0, nameW - 1) + "…" : label;
|
|
723
|
-
const headerLine = " " + " ".repeat(nameW) + " " + "qual".padStart(qualW) + " " + "$ in/out".padEnd(priceW);
|
|
735
|
+
const headerLine = " " + " ".repeat(nameW) + " " + "qual".padStart(qualW) + " " + "effort".padStart(effortW) + " " + "alt".padEnd(altW) + " " + "$ in/out".padEnd(priceW);
|
|
724
736
|
const rows = [{
|
|
725
737
|
label: "",
|
|
726
738
|
headerLine
|
|
@@ -735,7 +747,7 @@ export default Plugin.define({
|
|
|
735
747
|
for (const entry of section) {
|
|
736
748
|
rows.push({
|
|
737
749
|
label: trim(labelOf.get(entry) ?? "").padEnd(nameW),
|
|
738
|
-
columns: " " + cell(entry.quality, qualW).padStart(qualW) + " " + cell(entry.price, priceW).padEnd(priceW),
|
|
750
|
+
columns: " " + cell(entry.quality, qualW).padStart(qualW) + " " + cell(entry.effort || "-", effortW).padStart(effortW) + " " + cell(entry.alternative || "-", altW).padEnd(altW) + " " + cell(entry.price, priceW).padEnd(priceW),
|
|
739
751
|
// F-20 id split: the row DISPLAYS the sanitized id (in `label`)
|
|
740
752
|
// but picks `value`, the raw catalog id switchToModel needs.
|
|
741
753
|
value: entry.value,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "model-compare-opencode-plugin",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "OpenCode plugin that switches OpenRouter model aliases to the best value-for-money pick from canonical/model-compare's ranking (balanced, price, quality).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"opencode",
|
package/src/core/catalog.ts
CHANGED
|
@@ -1,9 +1,16 @@
|
|
|
1
1
|
// src/core/catalog.ts
|
|
2
|
-
// Catalog document validation (design §5). Accepts schema
|
|
2
|
+
// Catalog document validation (design §5). Accepts schema v5 documents (only) from
|
|
3
3
|
// either source (vendored script or remote catalog) and projects them onto
|
|
4
4
|
// the plugin's Catalog type. Validation only — ranking lives elsewhere.
|
|
5
5
|
|
|
6
|
-
import type {
|
|
6
|
+
import type {
|
|
7
|
+
Catalog,
|
|
8
|
+
ModelEntry,
|
|
9
|
+
Priority,
|
|
10
|
+
Reasoning,
|
|
11
|
+
ReasoningDepth,
|
|
12
|
+
Weights,
|
|
13
|
+
} from "./types"
|
|
7
14
|
|
|
8
15
|
const PRIORITIES: readonly Priority[] = ["balanced", "price", "quality"]
|
|
9
16
|
const SCORE_KEYS = ["quality", "price", "context", "age"] as const
|
|
@@ -44,18 +51,67 @@ function steer(raw: unknown): ModelEntry["steering"] {
|
|
|
44
51
|
return steering
|
|
45
52
|
}
|
|
46
53
|
|
|
54
|
+
function finiteOrNull(value: unknown): number | null {
|
|
55
|
+
return typeof value === "number" && Number.isFinite(value) ? value : null
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
function stringOrNull(value: unknown): string | null {
|
|
59
|
+
return typeof value === "string" ? value : null
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
function boolOrNull(value: unknown): boolean | null {
|
|
63
|
+
return typeof value === "boolean" ? value : null
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
// The v5 reasoning block (display + request injection). Field-by-field
|
|
67
|
+
// projection, never a spread of the raw object: a hostile catalogURL serves
|
|
68
|
+
// arbitrary JSON here, and only the fields the plugin reads survive. A
|
|
69
|
+
// non-object block reads as absent; a malformed field reads as its neutral
|
|
70
|
+
// value (empty list / null) rather than rejecting the whole catalog — the
|
|
71
|
+
// block is advisory metadata, not ranking input. Depth records without a
|
|
72
|
+
// non-empty string effort are dropped; every other depth field is nullable.
|
|
73
|
+
function projectReasoning(raw: unknown): Reasoning | undefined {
|
|
74
|
+
if (!isObject(raw)) return undefined
|
|
75
|
+
const supported =
|
|
76
|
+
Array.isArray(raw.supported_efforts) && raw.supported_efforts.every((effort) => typeof effort === "string")
|
|
77
|
+
? [...(raw.supported_efforts as string[])]
|
|
78
|
+
: []
|
|
79
|
+
const depths: ReasoningDepth[] = []
|
|
80
|
+
if (Array.isArray(raw.depths)) {
|
|
81
|
+
for (const record of raw.depths) {
|
|
82
|
+
if (!isObject(record)) continue
|
|
83
|
+
if (typeof record.effort !== "string" || record.effort === "") continue
|
|
84
|
+
depths.push({
|
|
85
|
+
effort: record.effort,
|
|
86
|
+
index: finiteOrNull(record.index),
|
|
87
|
+
cost_per_task_usd: finiteOrNull(record.cost_per_task_usd),
|
|
88
|
+
source: stringOrNull(record.source),
|
|
89
|
+
as_of: stringOrNull(record.as_of),
|
|
90
|
+
})
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
return {
|
|
94
|
+
default_effort: stringOrNull(raw.default_effort),
|
|
95
|
+
supported_efforts: supported,
|
|
96
|
+
default_enabled: boolOrNull(raw.default_enabled),
|
|
97
|
+
mandatory: boolOrNull(raw.mandatory),
|
|
98
|
+
supports_max_tokens: boolOrNull(raw.supports_max_tokens),
|
|
99
|
+
depths,
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
|
|
47
103
|
export function validateCatalog(
|
|
48
104
|
raw: unknown,
|
|
49
105
|
): { ok: true; catalog: Catalog } | { ok: false; reason: string } {
|
|
50
106
|
if (!isObject(raw)) return { ok: false, reason: "catalog must be a JSON object" }
|
|
51
107
|
|
|
52
|
-
// Supported schema:
|
|
53
|
-
// schema_version
|
|
54
|
-
//
|
|
55
|
-
//
|
|
56
|
-
// reason string the canary and the stored-document path rely on.
|
|
57
|
-
if (raw.schema_version !==
|
|
58
|
-
return { ok: false, reason: `schema_version must be
|
|
108
|
+
// Supported schema: v5 only — upstream's current contract (0.6.0's
|
|
109
|
+
// schema_version 5). v5 changes the meaning of `quality` (the AA score at
|
|
110
|
+
// the selected/default reasoning effort) and adds per-model
|
|
111
|
+
// `quality_effort` and `reasoning`; every other version is rejected with
|
|
112
|
+
// the reason string the canary and the stored-document path rely on.
|
|
113
|
+
if (raw.schema_version !== 5) {
|
|
114
|
+
return { ok: false, reason: `schema_version must be 5, got ${JSON.stringify(raw.schema_version)}` }
|
|
59
115
|
}
|
|
60
116
|
if (typeof raw.generated_at !== "string") {
|
|
61
117
|
return { ok: false, reason: "generated_at must be a string" }
|
|
@@ -160,12 +216,17 @@ export function validateCatalog(
|
|
|
160
216
|
const input = pricing.input_per_1m
|
|
161
217
|
const output = pricing.output_per_1m
|
|
162
218
|
// The steering block projects (or reads as absent) per the steer() rules.
|
|
163
|
-
// The
|
|
164
|
-
//
|
|
165
|
-
//
|
|
219
|
+
// The v5 quality_effort/reasoning fields project the same way: strip the
|
|
220
|
+
// raw keys before the spread so the camelCase projection is the only
|
|
221
|
+
// representation on the entry (a hostile block never survives verbatim).
|
|
222
|
+
const { steering: _rawSteering, quality_effort: _rawQualityEffort, reasoning: _rawReasoning, ...passthrough } =
|
|
223
|
+
entry
|
|
166
224
|
const steering = steer(entry.steering)
|
|
225
|
+
const qualityEffort =
|
|
226
|
+
typeof entry.quality_effort === "string" && entry.quality_effort !== "" ? entry.quality_effort : null
|
|
227
|
+
const reasoning = projectReasoning(entry.reasoning)
|
|
167
228
|
models.push({
|
|
168
|
-
...
|
|
229
|
+
...passthrough,
|
|
169
230
|
id: entry.id,
|
|
170
231
|
name: entry.name,
|
|
171
232
|
pricing: {
|
|
@@ -177,6 +238,8 @@ export function validateCatalog(
|
|
|
177
238
|
context: entry.context,
|
|
178
239
|
scores: { ...components, overall },
|
|
179
240
|
quality,
|
|
241
|
+
qualityEffort,
|
|
242
|
+
reasoning,
|
|
180
243
|
steering,
|
|
181
244
|
})
|
|
182
245
|
}
|
|
@@ -220,14 +283,33 @@ export function validateStoredCatalog(
|
|
|
220
283
|
stored: unknown,
|
|
221
284
|
): { ok: true; catalog: Catalog } | { ok: false; reason: string } {
|
|
222
285
|
if (!isObject(stored)) return { ok: false, reason: "stored catalog must be an object" }
|
|
286
|
+
// v5 projections always carry qualityEffort (null or a string). A record
|
|
287
|
+
// written by a pre-v5 plugin lacks it: accepting it would mix v4's max-row
|
|
288
|
+
// quality with v5's default-effort semantics, so it is a cache miss.
|
|
289
|
+
if (Array.isArray(stored.models)) {
|
|
290
|
+
for (const [i, entry] of stored.models.entries()) {
|
|
291
|
+
if (isObject(entry) && !Object.hasOwn(entry, "qualityEffort")) {
|
|
292
|
+
return { ok: false, reason: `models[${i}].qualityEffort is missing (not a v5 catalog)` }
|
|
293
|
+
}
|
|
294
|
+
}
|
|
295
|
+
}
|
|
223
296
|
const models = Array.isArray(stored.models)
|
|
224
297
|
? stored.models.map((entry) => {
|
|
225
|
-
if (!isObject(entry)
|
|
226
|
-
|
|
298
|
+
if (!isObject(entry)) return entry
|
|
299
|
+
// The projection renamed quality_effort to qualityEffort and unwrapped
|
|
300
|
+
// steering; rebuild the raw v5 shape. A hand-edited record re-projects
|
|
301
|
+
// (or reads as absent) exactly like a raw one would.
|
|
302
|
+
const raw: Record<string, unknown> = {
|
|
303
|
+
...entry,
|
|
304
|
+
quality_effort: entry.qualityEffort ?? null,
|
|
305
|
+
}
|
|
306
|
+
if (entry.reasoning !== undefined) raw.reasoning = entry.reasoning
|
|
307
|
+
if (entry.steering !== undefined) raw.steering = { model: entry.id, provider: entry.steering }
|
|
308
|
+
return raw
|
|
227
309
|
})
|
|
228
310
|
: stored.models
|
|
229
311
|
return validateCatalog({
|
|
230
|
-
schema_version:
|
|
312
|
+
schema_version: 5,
|
|
231
313
|
generated_at: stored.generatedAt,
|
|
232
314
|
parameters: { weights: stored.weights },
|
|
233
315
|
models,
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
// src/core/reasoning.ts
|
|
2
|
+
// Reasoning-depth helpers (catalog v5): gateway-effort validation, the
|
|
3
|
+
// selectable-depth rule, and the best measured alternative to the effort a
|
|
4
|
+
// model was scored at. Pure functions over a projected ModelEntry — no I/O,
|
|
5
|
+
// no OpenCode imports.
|
|
6
|
+
//
|
|
7
|
+
// The selectable rule is upstream's site rule (web/site/app.js depthRows):
|
|
8
|
+
// OpenRouter can select an effort only when the model's expanded
|
|
9
|
+
// `supported_efforts` lists it (or the model is a single-state toggle that
|
|
10
|
+
// exposes `#none`). An AA row for an unlisted effort is display-only and is
|
|
11
|
+
// never advertised as an alternative. The catalog's `reasoning` sentinel is a
|
|
12
|
+
// producer label for a single-state reasoning model, not a gateway value.
|
|
13
|
+
|
|
14
|
+
import type { ModelEntry, Reasoning } from "./types"
|
|
15
|
+
|
|
16
|
+
// What OpenRouter accepts as a discrete effort. Mirrors upstream's
|
|
17
|
+
// REASONING_GATEWAY_EFFORTS. Deliberately excludes the catalog sentinels
|
|
18
|
+
// "reasoning" and "default".
|
|
19
|
+
export const GATEWAY_EFFORTS = ["max", "xhigh", "high", "medium", "low", "minimal", "none"] as const
|
|
20
|
+
|
|
21
|
+
// Deepest-first display order, mirroring upstream's AA_EFFORT_ORDER.
|
|
22
|
+
const EFFORT_ORDER: readonly string[] = [...GATEWAY_EFFORTS, "reasoning"]
|
|
23
|
+
|
|
24
|
+
export function gatewayEffort(value: string | null | undefined): string | null {
|
|
25
|
+
return value !== null && value !== undefined && (GATEWAY_EFFORTS as readonly string[]).includes(value)
|
|
26
|
+
? value
|
|
27
|
+
: null
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
function effortOrder(effort: string): number {
|
|
31
|
+
const index = EFFORT_ORDER.indexOf(effort)
|
|
32
|
+
return index === -1 ? EFFORT_ORDER.length : index
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
// Upstream's v5 non-reasoning rule (the producer's is_reasoning flag is not
|
|
36
|
+
// published): no reasoning metadata at all, so the bare id is the only form.
|
|
37
|
+
function isNonReasoningBlock(reasoning: Reasoning): boolean {
|
|
38
|
+
return (
|
|
39
|
+
reasoning.default_effort === "none" &&
|
|
40
|
+
reasoning.default_enabled !== false &&
|
|
41
|
+
reasoning.mandatory !== true &&
|
|
42
|
+
reasoning.supports_max_tokens !== true
|
|
43
|
+
)
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
// A single-state reasoning model with reasoning off by default exposes only
|
|
47
|
+
// the "#none" toggle.
|
|
48
|
+
function isSingleStateToggle(reasoning: Reasoning): boolean {
|
|
49
|
+
return reasoning.mandatory !== true && reasoning.default_enabled === false
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
export function isSelectableDepth(model: ModelEntry, effort: string): boolean {
|
|
53
|
+
if (effort === "reasoning") return false
|
|
54
|
+
const reasoning = model.reasoning
|
|
55
|
+
if (reasoning === undefined) return false
|
|
56
|
+
if (reasoning.supported_efforts.includes(effort)) return true
|
|
57
|
+
if (effort !== "none" || reasoning.supported_efforts.length > 0) return false
|
|
58
|
+
if (isNonReasoningBlock(reasoning)) return false
|
|
59
|
+
return isSingleStateToggle(reasoning)
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
export type DepthAlternative = {
|
|
63
|
+
effort: string
|
|
64
|
+
index: number
|
|
65
|
+
cost: number | null
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
// The best measured, selectable effort that scores strictly higher than the
|
|
69
|
+
// effort `quality` already represents. Unselectable rows never qualify; ties
|
|
70
|
+
// resolve to the deeper effort (the display order).
|
|
71
|
+
export function bestDepthAlternative(model: ModelEntry): DepthAlternative | null {
|
|
72
|
+
const base = model.quality
|
|
73
|
+
const selected = model.qualityEffort
|
|
74
|
+
const reasoning = model.reasoning
|
|
75
|
+
if (base == null || selected == null || reasoning === undefined) return null
|
|
76
|
+
|
|
77
|
+
const depths = [...reasoning.depths].sort((a, b) => effortOrder(a.effort) - effortOrder(b.effort))
|
|
78
|
+
let best: DepthAlternative | null = null
|
|
79
|
+
for (const depth of depths) {
|
|
80
|
+
if (depth.index === null || depth.effort === selected) continue
|
|
81
|
+
if (!isSelectableDepth(model, depth.effort)) continue
|
|
82
|
+
if (depth.index <= base) continue
|
|
83
|
+
if (best === null || depth.index > best.index) {
|
|
84
|
+
best = { effort: depth.effort, index: depth.index, cost: depth.cost_per_task_usd }
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
return best
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
// The report's compact alternative label: "<effort> <index>" ("max 56.0"),
|
|
91
|
+
// the same shape upstream's site shows under QUAL. Empty when there is none.
|
|
92
|
+
export function formatDepthAlternative(model: ModelEntry | undefined): string {
|
|
93
|
+
if (model === undefined) return ""
|
|
94
|
+
const best = bestDepthAlternative(model)
|
|
95
|
+
return best === null ? "" : `${best.effort} ${best.index.toFixed(1)}`
|
|
96
|
+
}
|
package/src/core/steering.ts
CHANGED
|
@@ -12,6 +12,7 @@
|
|
|
12
12
|
// and the steeringFor cases here); a semantic change to one must be mirrored
|
|
13
13
|
// in the other.
|
|
14
14
|
|
|
15
|
+
import { gatewayEffort } from "./reasoning"
|
|
15
16
|
import type { ModelEntry, Priority } from "./types"
|
|
16
17
|
|
|
17
18
|
// A model reference as the hook event carries it (`event.model`), mirroring
|
|
@@ -75,6 +76,30 @@ export function steeringFor(
|
|
|
75
76
|
return models.get(pick.id)?.steering ?? null
|
|
76
77
|
}
|
|
77
78
|
|
|
79
|
+
// The reasoning effort an alias request must pin, or null when it must not:
|
|
80
|
+
// the same alias/body/pick/model gates as steeringFor, plus the explicit
|
|
81
|
+
// variant rule. `#variant` is OpenCode's per-request selection (the event
|
|
82
|
+
// carries it on `model.variant`); "default" is the no-selection sentinel
|
|
83
|
+
// OpenCode stamps on `--model` refs (spike fact 1), so it does NOT count as an
|
|
84
|
+
// explicit variant. The catalog's `quality_effort` is the effort the score was
|
|
85
|
+
// measured at; the "reasoning" sentinel is a producer label and is never sent
|
|
86
|
+
// to the gateway.
|
|
87
|
+
export function reasoningEffortFor(
|
|
88
|
+
eventModel: ModelRef,
|
|
89
|
+
variant: string | undefined,
|
|
90
|
+
bodyModel: string | undefined,
|
|
91
|
+
picks: Partial<Record<Priority, AliasTarget>>,
|
|
92
|
+
models: Map<string, ModelEntry>,
|
|
93
|
+
): string | null {
|
|
94
|
+
const p = aliasPriority(eventModel)
|
|
95
|
+
if (p === null) return null
|
|
96
|
+
if (variant !== undefined && variant !== "default") return null
|
|
97
|
+
const pick = picks[p]
|
|
98
|
+
if (pick === undefined || pick.id === null) return null
|
|
99
|
+
if (bodyModel !== undefined && bodyModel !== pick.id) return null
|
|
100
|
+
return gatewayEffort(models.get(pick.id)?.qualityEffort)
|
|
101
|
+
}
|
|
102
|
+
|
|
78
103
|
function isPlainObject(value: unknown): value is Record<string, unknown> {
|
|
79
104
|
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
80
105
|
}
|
|
@@ -103,6 +128,17 @@ export function injectSteering(body: unknown, steering: Steering): unknown {
|
|
|
103
128
|
}
|
|
104
129
|
}
|
|
105
130
|
|
|
131
|
+
// The body with `reasoning: { effort }` added — or the input untouched when
|
|
132
|
+
// it already carries `reasoning` or the legacy `reasoning_effort` field (an
|
|
133
|
+
// explicit #variant, user config or another plugin wins; the plugin never
|
|
134
|
+
// clobbers a reasoning choice it did not make) or is not a JSON object.
|
|
135
|
+
// Reference inequality is the wiring's "re-create the Request" signal.
|
|
136
|
+
export function injectReasoning(body: unknown, effort: string): unknown {
|
|
137
|
+
if (!isPlainObject(body)) return body
|
|
138
|
+
if (Object.hasOwn(body, "reasoning") || Object.hasOwn(body, "reasoning_effort")) return body
|
|
139
|
+
return { ...body, reasoning: { effort } }
|
|
140
|
+
}
|
|
141
|
+
|
|
106
142
|
// normalize_provider (vendor/model_compare.py): casefold the name and strip
|
|
107
143
|
// ALL non-alphanumeric characters — "Z.AI", "z-ai" and "z ai" all normalize
|
|
108
144
|
// to "zai". JS toLowerCase stands in for Python's casefold (provider display
|
package/src/core/types.ts
CHANGED
|
@@ -6,6 +6,31 @@ export type Priority = "balanced" | "price" | "quality"
|
|
|
6
6
|
|
|
7
7
|
export type Weights = { quality: number; price: number; context: number; age: number }
|
|
8
8
|
|
|
9
|
+
// One measured reasoning effort (catalog v5): the AA intelligence index at
|
|
10
|
+
// that effort, AA's reference-price benchmark cost per task (display-only,
|
|
11
|
+
// never scored), and the record's provenance. `null` marks a missing value;
|
|
12
|
+
// the record itself survives so the effort can still be offered as a variant.
|
|
13
|
+
export type ReasoningDepth = {
|
|
14
|
+
effort: string
|
|
15
|
+
index: number | null
|
|
16
|
+
cost_per_task_usd: number | null
|
|
17
|
+
source: string | null
|
|
18
|
+
as_of: string | null
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
// The v5 per-model reasoning block. `supported_efforts` is what OpenRouter
|
|
22
|
+
// accepts as a discrete effort (expanded upstream, never null); `depths` is
|
|
23
|
+
// what some source measured (empty when nothing did). The raw flags are kept
|
|
24
|
+
// for provenance and for the single-state/non-reasoning display rules.
|
|
25
|
+
export type Reasoning = {
|
|
26
|
+
default_effort: string | null
|
|
27
|
+
supported_efforts: string[]
|
|
28
|
+
default_enabled: boolean | null
|
|
29
|
+
mandatory: boolean | null
|
|
30
|
+
supports_max_tokens: boolean | null
|
|
31
|
+
depths: ReasoningDepth[]
|
|
32
|
+
}
|
|
33
|
+
|
|
9
34
|
export type ModelEntry = {
|
|
10
35
|
id: string
|
|
11
36
|
name: string
|
|
@@ -20,6 +45,14 @@ export type ModelEntry = {
|
|
|
20
45
|
}
|
|
21
46
|
context: number
|
|
22
47
|
quality: number | null
|
|
48
|
+
// The effort `quality` was measured at (catalog v5 `quality_effort`). The
|
|
49
|
+
// alias request pins exactly this effort; the sentinel "reasoning" is a
|
|
50
|
+
// catalog label, never a gateway value. Optional only for hand-built /
|
|
51
|
+
// legacy projected entries: validateCatalog always writes it (null or a
|
|
52
|
+
// string), and validateStoredCatalog requires it on cached records.
|
|
53
|
+
qualityEffort?: string | null
|
|
54
|
+
// Present when the raw block projects cleanly (absent stays absent).
|
|
55
|
+
reasoning?: Reasoning
|
|
23
56
|
// The steering block (F-11/F-12): upstream records the provider's
|
|
24
57
|
// endpoint-routing hints as { model, provider: { order, ignore,
|
|
25
58
|
// quantizations?, allow_fallbacks } }; the projection unwraps the provider
|
|
@@ -47,7 +80,7 @@ export type Catalog = {
|
|
|
47
80
|
models: ModelEntry[]
|
|
48
81
|
}
|
|
49
82
|
|
|
50
|
-
// Upstream
|
|
83
|
+
// Upstream's PRIORITY_WEIGHTS at the pinned tag (v0.6.0; unchanged since v0.3.0). Used when a
|
|
51
84
|
// document carries no `parameters.weights` at all (design §5); individual
|
|
52
85
|
// missing keys fall back to weight 0 instead.
|
|
53
86
|
export const DEFAULT_WEIGHTS: Record<Priority, Weights> = {
|
package/src/index.ts
CHANGED
|
@@ -19,7 +19,13 @@ import { parseConfig } from "./config"
|
|
|
19
19
|
import { pinKey, type Pin } from "./core/pins"
|
|
20
20
|
import { effectiveWeights, rankModels } from "./core/rank"
|
|
21
21
|
import type { Catalog, EndpointCapture, ModelEntry, Priority, Sourced } from "./core/types"
|
|
22
|
-
import {
|
|
22
|
+
import {
|
|
23
|
+
captureEndpoint,
|
|
24
|
+
injectReasoning,
|
|
25
|
+
injectSteering,
|
|
26
|
+
reasoningEffortFor,
|
|
27
|
+
steeringFor,
|
|
28
|
+
} from "./core/steering"
|
|
23
29
|
import { formatReport, humanizeAge, reportEntries, steeringNote } from "./opencode/commands"
|
|
24
30
|
import { ModelCompare } from "./rpc"
|
|
25
31
|
import { planAlias, type AliasTarget } from "./opencode/aliases"
|
|
@@ -329,7 +335,11 @@ function applyAliases(state: WireState, editor: AliasEditor): void {
|
|
|
329
335
|
// plan: planAlias only checks for undefined (it never reads model
|
|
330
336
|
// internals — the applier does, below).
|
|
331
337
|
const source = target.id === null ? undefined : editor.get(ALIAS_PROVIDER, target.id)
|
|
332
|
-
|
|
338
|
+
// The concrete editor model proves presence; the catalog entry supplies
|
|
339
|
+
// the v5 effort/alternative labels (the editor model has no ranking data).
|
|
340
|
+
const catalogModel =
|
|
341
|
+
target.id === null ? undefined : state.sourced?.doc.models.find((entry) => entry.id === target.id)
|
|
342
|
+
const plan = planAlias(p, target, source as unknown as ModelEntry | undefined, catalogModel)
|
|
333
343
|
|
|
334
344
|
if (plan.enabled && plan.modelID !== null && source !== undefined) {
|
|
335
345
|
editor.update(ALIAS_PROVIDER, aliasId(p), (model) => {
|
|
@@ -558,21 +568,36 @@ export async function wire(ctx: PluginCtx, deps: WireDeps = {}): Promise<() => v
|
|
|
558
568
|
state.appliedSteering[event.sessionID] = null
|
|
559
569
|
return
|
|
560
570
|
}
|
|
561
|
-
const
|
|
571
|
+
const byId = new Map(
|
|
572
|
+
state.sourced.doc.models.map((entry): [string, ModelEntry] => [entry.id, entry]),
|
|
573
|
+
)
|
|
574
|
+
const bodyModel = bodyModelOf(body)
|
|
575
|
+
const steering = steeringFor(event.model, bodyModel, state.picks, byId)
|
|
576
|
+
// Two independent injections share one re-created Request: the
|
|
577
|
+
// provider-routing block and the catalog's scored reasoning effort.
|
|
578
|
+
// Each helper preserves a value the body already sets, so an explicit
|
|
579
|
+
// #variant (either signalled by the event's variant or by the body's
|
|
580
|
+
// own reasoning field) wins without withholding the other block.
|
|
581
|
+
let modified: unknown = body
|
|
582
|
+
let providerApplied = false
|
|
583
|
+
if (steering !== null) {
|
|
584
|
+
const next = injectSteering(modified, steering)
|
|
585
|
+
if (next !== modified) {
|
|
586
|
+
modified = next
|
|
587
|
+
providerApplied = true
|
|
588
|
+
}
|
|
589
|
+
}
|
|
590
|
+
const effort = reasoningEffortFor(
|
|
562
591
|
event.model,
|
|
563
|
-
|
|
592
|
+
event.model.variant,
|
|
593
|
+
bodyModel,
|
|
564
594
|
state.picks,
|
|
565
|
-
|
|
595
|
+
byId,
|
|
566
596
|
)
|
|
567
|
-
if (
|
|
568
|
-
// No steering data for the pick: untouched, fail open.
|
|
569
|
-
state.appliedSteering[event.sessionID] = null
|
|
570
|
-
return
|
|
571
|
-
}
|
|
572
|
-
const modified = injectSteering(body, steering)
|
|
597
|
+
if (effort !== null) modified = injectReasoning(modified, effort)
|
|
573
598
|
if (modified === body) {
|
|
574
|
-
//
|
|
575
|
-
//
|
|
599
|
+
// Neither block had anything to add: no steering data, no scored
|
|
600
|
+
// effort, or the body already carries the corresponding field.
|
|
576
601
|
state.appliedSteering[event.sessionID] = null
|
|
577
602
|
return
|
|
578
603
|
}
|
|
@@ -592,7 +617,9 @@ export async function wire(ctx: PluginCtx, deps: WireDeps = {}): Promise<() => v
|
|
|
592
617
|
})
|
|
593
618
|
// Record the order actually applied — the response hook reads it as
|
|
594
619
|
// captureEndpoint's order (the `!` flag fires only on a non-empty one).
|
|
595
|
-
|
|
620
|
+
// A reasoning-only injection (or a body that already carried routing)
|
|
621
|
+
// applies no routing, so the record is null.
|
|
622
|
+
state.appliedSteering[event.sessionID] = providerApplied ? steering!.order : null
|
|
596
623
|
} catch {
|
|
597
624
|
// Fail open: any error leaves the original request proceeding
|
|
598
625
|
// untouched, and records that nothing was injected.
|