model-compare-opencode-plugin 0.1.1 → 0.3.0

package/README.md

@@ -77,14 +77,15 @@ Commands:
 | Key | Action |
 |---|---|
 | `alt+m` | picks dialog — all three priorities with their current picks |
+| `alt+e` | toast the last response's served endpoint, alias responses only (`!` prefix flags a downgrade — not in the steering order) |
 | `alt+shift+m` | report dialog — the top-5 table; `return` switches the session to the highlighted model |
 | `up` / `down` | move the dialog cursor |
 | `return` | select the highlighted row (picks dialog: switch to that priority) |
 | `esc` | close the dialog |
 
-Both dialogs need an open session — with none, a toast explains and nothing else happens. The TUI has no `/use-best` keybinding: the slash command already covers switching.
+Both dialogs open anywhere — including the opening screen, where no session exists yet. There, picking a row (a priority in the picks dialog, a model in the report) creates a new session already bound to that model and opens it; merely opening a dialog never creates a session. With a session open, a pick switches that session as before. The TUI has no `/use-best` keybinding: the slash command already covers switching.
 
-Status: the footer shows `MC: balanced → <pick>` (balanced only — `disabled`/`unavailable`/`(fallback)` as applicable; all three priorities live in the picks dialog). Toasts announce `/use-best` outcomes and pick changes — never for the first observation after load. Headless, all feedback is synthetic transcript messages at the next turn.
+Status: the footer shows `MC: balanced → <pick>` (balanced only — `disabled`/`unavailable`/`(fallback)` as applicable; all three priorities live in the picks dialog). Toasts announce `/use-best` outcomes and pick changes (never for the first observation after load), the `alt+e` served endpoint, and the cannot-steer notice — see [Steering and served endpoints](#steering-and-served-endpoints). Headless, command feedback is synthetic transcript messages at the next turn; the steering toasts have no headless surface.
 
 
 ## Configuration
@@ -99,7 +100,7 @@ Options live under `plugins[].options`:
 | `weights` | upstream | Partial overrides per named priority (`quality`, `price`, `context`, `age`), merged with upstream's weights and renormalized to sum 1 |
 | `pinDays` | `14` | Age in days after which a pick is re-evaluated |
 | `minLead` | `0.03` | Score lead needed to replace a pinned pick |
-| `catalogURL` | published catalog | Remote source URL (`https://canonical.github.io/model-compare/catalog.json`). Only point it at a source you trust: it decides which models can be picked |
+| `catalogURL` | published catalog | Remote source URL. The catalog is published at `https://model-compare.dev/catalog.json` — the plugin's built-in default, fetched directly over https. (The old `https://canonical.github.io/model-compare/catalog.json` home redirects to a plaintext `http://` target; the plugin fails closed on any redirect that leaves https.) Only point it at a source you trust: it decides which models can be picked |
 | `scriptPath` | vendored copy | Development override for a local `model_compare.py` |
 
 A present-but-invalid value logs a warning and applies the default (`weights` drop invalid entries individually; the valid remainder applies). Absent options apply defaults silently. Setup never throws; the aliases stay functional.
@@ -117,6 +118,35 @@ Per priority the plugin keeps one pin/last-known-good record in plugin storage,
 Resolution order: pin (if runnable) → best runnable candidate → `fallback` (if runnable) → the alias is disabled. The fallback is never stored as a pin. Pins are advisory: concurrent OpenCode processes refresh independently, last writer wins.
 
 
+## Steering and served endpoints
+
+Requests made through one of the three aliases carry the resolved pick's routing: the catalog's per-model `steering` block is injected into the request body under OpenRouter's `provider` key, so the call is routed onto the endpoints whose pricing the ranking actually ranked — not onto whatever OpenRouter's default routing picks, which can be a sub-default-quantization endpoint at a price the ranking never saw.
+
+The block comes from the catalog, whichever source produced it: the vendored script derives it from the same per-model endpoint data its pricing uses, and the published catalog carries it ready-made. Only the block's `provider` sub-object is injected — its top-level `model` field never reaches the body — and the routing object is built field by field:
+
+- `order` — the endpoints the ranking priced, cheapest blended price first; eligible endpoints whose price can't be computed are appended last rather than dropped.
+- `ignore` — the live endpoints the ranking excluded, so OpenRouter cannot route onto pricing it never ranked (sub-default quantizations, schedule-gate failures, variants).
+- `quantizations` — the precision floor: the model's default precision and anything more precise among its live endpoints. The key is omitted entirely when the model has no default precision — never sent as an empty list, which would exclude every endpoint.
+- `allow_fallbacks` — always `true`: when nothing in `order` can serve, OpenRouter may still route outside the list.
+
+The rules, and the exceptions:
+
+- **Aliases only.** Steering fires only when the session's model is one of the three aliases and the request's model is that alias's current pick. What decides is the request's model reference: alias references steer, concrete references never do — a model you picked directly is untouched even when it is the exact model an alias currently resolves to. Primary turns and compaction alike are steered; auxiliary calls that run a different model (title generation) are not.
+- **A pre-existing `provider` key wins.** If the request body already carries a `provider` key — routing you or a provider integration configured — the plugin does not inject; it never clobbers routing it did not choose.
+- **No steering data, no injection.** When the plugin has no catalog source, or the current pick's catalog entry carries no `steering` block, the request proceeds untouched: OpenRouter's default routing applies and the run completes normally — unsteered, not failed. With no catalog source at all, the TUI says so once per session, on the first alias request that has a pick to steer: `cannot steer this session; OpenRouter default routing applies` (skipped when that request already carries its own `provider` — then default routing is not what applies). Headless runs render no toasts; the routing behavior is identical. A disabled alias steers nothing and toasts nothing; what serves then is the binding's own fallback behavior (see [Troubleshooting](#troubleshooting)).
+- **Always-on.** No option turns steering off. It is what makes the plugin's value promise — the ranked price is what you pay — true.
+
+`order` is a preference list, not a pinned destination: OpenRouter works down the list and serves from whichever ranked endpoint can take the call — which one serves is OpenRouter's decision, and with fallbacks allowed it can even land outside the list.
+
+**Which endpoint served a call.** Three ways to check:
+
+- The `alt+e` toast (see [Keyboard shortcuts](#keyboard-shortcuts)) — the current session's last primary alias response's served endpoint: `last response served by <endpoint>`, or `! served by <endpoint> — not in steering order` when the served provider matched no entry of the steering order actually applied. The report dialog's footer carries the same capture as `last response: <model> @ <endpoint>`, under the note `routing steered to <quant> endpoints (aliases only)`.
+- The response body — every OpenRouter response names its serving endpoint in the `provider` field.
+- OpenRouter's dashboard — per-request endpoint records.
+
+The alt+e capture is best-effort by design, and alias-scoped: only a session whose model is one of the three aliases has responses peeked and recorded — a direct pick's or another agent's response is never read, let alone modified. The plugin peeks at only the first 4 KB of the response body — a pass-through read; the body itself flows byte-for-byte. Streaming responses carry `provider` in the first chunk, so the peek normally catches them; a non-streaming response whose `provider` field sits beyond the peek window is forwarded uncaptured, and `alt+e` then toasts nothing. The response body's `provider` field and OpenRouter's dashboard remain the authoritative verification paths.
+
+
 ## Data sources
 
 - `auto` — vendored `vendor/model_compare.py` run with `python3` (no shell, 60 s timeout, minimal environment); degrades to remote when python3 is missing or the run fails.
@@ -125,16 +155,18 @@ Resolution order: pin (if runnable) → best runnable candidate → `fallback` (
 
 The last validated catalog is persisted, so ranking and `/model-compare` work offline after a restart. The cache is keyed by source (URL or script path), so changing the source never serves the previous source's data. Script exit code 2 means "no candidates" *or* an argparse usage error — never read it as "no candidates" alone.
 
-**Vendored pin:** `vendor/model_compare.py` is byte-identical to upstream `v0.3.1` (`vendor/UPSTREAM.json` records repo, tag, commit, sha256). Since upstream 0.3.0 the catalog is schema v2: `pricing` scalars hold effective-tier values — the tier in effect at 1M-token prompts — so models with OpenRouter price overrides are ranked at what a long session actually pays.
+**Vendored pin:** `vendor/model_compare.py` is byte-identical to upstream `v0.4.0` (`vendor/UPSTREAM.json` records repo, tag, commit, sha256). Since upstream 0.4.0 the catalog is schema v3: `pricing` carries tier and schedule structure alongside the flat scalars, and `parameters` carries the published ranking weights. The plugin accepts schema v3 ONLY — `validateCatalog` rejects anything else — and releases are gated on the live remote serving v3 (the release-gate check pipes `https://model-compare.dev/catalog.json` through `scripts/canary-check.mjs`, which fails closed when the served document is not the expected schema). Effective-tier pricing semantics still apply: `pricing` scalars hold effective-tier values — the tier in effect at 1M-token prompts — so models with OpenRouter price overrides are ranked at what a long session actually pays.
+
+**Vendored 0.4.0 cold-cache network profile:** since 0.4.0 the vendored script performs per-model endpoint-pricing fetches when building a fresh catalog (71 in the recorded run). Each fetch fails closed per model — after the script's internal retries, a failed endpoint fetch drops that model and the catalog shrinks silently (429-class responses raise without retry) — so a degraded network costs pool coverage, not pricing correctness. A cold cache under a slow network can exceed the plugin's 60-second local budget, in which case the plugin warns and falls back (`auto` → remote, `local` → its cached document).
 
 **Upgrading the pin** — reviewed PRs only; nothing merges automatically:
 
 1. `node scripts/sync-vendor.mjs <tag>` — fetches the script at that commit, reports the sha256 change, re-pins `UPSTREAM.json`, prints the matching CHANGELOG section (`--check` verifies offline).
-2. Conformance tests check `--version`, schema v2, required fields, recomputed `overall` scores, and the standard-library-only import allowlist.
+2. Conformance tests check `--version`, schema v3, required fields, recomputed `overall` scores, and the standard-library-only import allowlist.
 3. The upgrade PR body lists script lines touching imports, URLs, environment variables, subprocess calls and file writes, for reviewer attention.
 4. Release with a changelog line naming the upstream tag; users run `opencode plugin update` (OpenCode checks but doesn't apply).
 
-A scheduled workflow opens a draft upgrade PR when upstream publishes a newer release (never a downgrade); a nightly canary exercises the script live; a weekly drift check compares the vendored sha256 against upstream's pinned commit and the published rankings against the published scores.
+A scheduled workflow opens a draft upgrade PR when upstream publishes a newer release (never a downgrade); a nightly canary exercises the vendored script against live upstream data and asserts the generated catalog is schema v3 (`scripts/canary-check.mjs` pins `EXPECTED_SCHEMA_VERSION = 3` and exits nonzero on any other version); releases additionally gate on the live remote itself (see the publishing gates below); a weekly drift check compares the vendored sha256 against upstream's pinned commit and the published rankings against the published scores.
 
 
 ## Troubleshooting
@@ -177,8 +209,8 @@ MC_INTEGRATION=1 npx vitest run tests/integration/    # live-server suite (opt-i
 - Local development loads `src/` directly and needs no build; the *published* package does — OpenCode's Solid JSX transform skips `node_modules` paths, so the npm-installed raw TSX cannot load (see [Install](#install) and the troubleshooting entry). `npm run build` precompiles `src/tui.tsx` through the same Babel pipeline as the host's loader (taken from the pinned `@opentui/solid`) into `dist/tui.js`, rebasing relative imports from `src/`-relative to `dist/`-relative; `prepublishOnly` runs it, and CI builds before the packaging check. `dist/` is regenerated, never committed.
 - The integration files skip unless `MC_INTEGRATION=1`, so plain `npm test` stays hermetic. The suite drives a real `opencode serve` on 127.0.0.1:14721 and needs the `opencode` CLI and `python3` (skips cleanly when either is missing); one case needs network access and the `sqlite3` CLI.
 - `node scripts/pack-smoke.mjs` (also in CI, after `node scripts/build-tui.mjs`) packs the plugin and checks the tarball against the packaging list (files, `exports`, relative imports, and that the TUI artifact is precompiled — no react import). It does **not** load the package in OpenCode — install the tarball into a scratch project and confirm OpenCode loads it before publishing.
-- Publishing: cut a GitHub release tagged `vX.Y.Z` and `.github/workflows/release.yaml` stages the package on npm through trusted publishing (tokenless OIDC — no token secret; the npmjs.com trusted-publisher entry must name `release.yaml`, the `canonical/model-compare-opencode-plugin` repository and the `release` environment, with "Allow npm publish" and "Allow npm dist-tag" unchecked — stage-only, npm's recommended posture). Nothing goes live until a maintainer approves it with 2FA on npmjs.com (package page → Staged Packages) or via `npm stage approve`. The workflow fails closed when the tag does not match `package.json`'s version, and the gates (build, typecheck, tests, pack-smoke) run before staging.
-- `fixtures/catalog-v0.3.0.json` is a recorded schema-v2 catalog; `fixtures/catalog-no-quality.json` is generated from it by `scripts/make-no-quality-fixture.py` (the vendored script's own quality-blind weights, so its scores are internally consistent). `node scripts/sync-vendor.mjs --check` verifies the pin's sha256 offline.
+- Publishing: cut a GitHub release tagged `vX.Y.Z` and `.github/workflows/release.yaml` stages the package on npm through trusted publishing (tokenless OIDC — no token secret; the npmjs.com trusted-publisher entry must name `release.yaml`, the `canonical/model-compare-opencode-plugin` repository and the `release` environment, with "Allow npm publish" and "Allow npm dist-tag" unchecked — stage-only, npm's recommended posture). Nothing goes live until a maintainer approves it with 2FA on npmjs.com (package page → Staged Packages) or via `npm stage approve`. The workflow fails closed when the tag does not match `package.json`'s version, and the gates (build, typecheck, tests, pack-smoke, live-remote canary) run before staging.
+- `fixtures/catalog-v0.4.0.json` is the sole contract fixture — a recorded schema-v3 catalog; `fixtures/catalog-no-quality.json` is derived from it by `scripts/make-no-quality-fixture.py` (the vendored script's own quality-blind weights, so its scores are internally consistent). No v2 fixture exists; the v3-only gate rejects one. `node scripts/sync-vendor.mjs --check` verifies the pin's sha256 offline.
 - Repo layout: `src/core/` (pure ranking/pinning), `src/sources/` (local + remote acquisition), `src/opencode/` (aliases, guard, commands, storage, refresh), `dist/` (generated `tui.js` artifact, gitignored), `vendor/` (pinned upstream script), `tests/{unit,conformance,integration}`.