model-compare-opencode-plugin 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -5
- package/package.json +7 -6
- package/src/core/catalog.ts +9 -7
- package/src/core/types.ts +1 -1
- package/src/tui.tsx +2 -2
- package/vendor/UPSTREAM.json +3 -3
- package/vendor/model_compare.py +705 -102
package/README.md
CHANGED
|
@@ -155,18 +155,18 @@ The alt+e capture is best-effort by design, and alias-scoped: only a session who
|
|
|
155
155
|
|
|
156
156
|
The last validated catalog is persisted, so ranking and `/model-compare` work offline after a restart. The cache is keyed by source (URL or script path), so changing the source never serves the previous source's data. Script exit code 2 means "no candidates" *or* an argparse usage error — never read it as "no candidates" alone.
|
|
157
157
|
|
|
158
|
-
**Vendored pin:** `vendor/model_compare.py` is byte-identical to upstream `v0.
|
|
158
|
+
**Vendored pin:** `vendor/model_compare.py` is byte-identical to upstream `v0.5.0` (`vendor/UPSTREAM.json` records repo, tag, commit, sha256). The catalog is schema v4 (since upstream 0.5.0): v3's `pricing` tier/schedule structure and published ranking `parameters.weights` are unchanged, and v4 adds per-model quality provenance (`quality_as_of`, `quality_drift`) plus a reshaped `sources.aa` block — nothing `validateCatalog` reads changed shape. The plugin accepts schema v4 ONLY — `validateCatalog` rejects anything else — and releases are gated on the live remote serving v4 (the release-gate check pipes `https://model-compare.dev/catalog.json` through `scripts/canary-check.mjs`, which fails closed when the served document is not the expected schema). Effective-tier pricing semantics still apply: `pricing` scalars hold effective-tier values — the tier in effect at 1M-token prompts — so models with OpenRouter price overrides are ranked at what a long session actually pays.
|
|
159
159
|
|
|
160
|
-
**Vendored 0.
|
|
160
|
+
**Vendored 0.5.0 cold-cache network profile:** since 0.4.0 the vendored script performs per-model endpoint-pricing fetches when building a fresh catalog (74 candidates in the recorded run). Each fetch fails closed per model — after the script's internal retries, a failed endpoint fetch drops that model and the catalog shrinks silently (429-class responses raise without retry) — so a degraded network costs pool coverage, not pricing correctness. Since 0.5.0, unkeyed runs also hydrate quality from the published catalog (one fetch of `https://model-compare.dev/catalog.json`, cached under `published-catalog`; keyed runs consult the AA API — now the primary quality source — and never fetch the published catalog); every quality hydration fails closed to "no quality data" per model. A cold cache under a slow network can exceed the plugin's 60-second local budget, in which case the plugin warns and falls back (`auto` → remote, `local` → its cached document).
|
|
161
161
|
|
|
162
162
|
**Upgrading the pin** — reviewed PRs only; nothing merges automatically:
|
|
163
163
|
|
|
164
164
|
1. `node scripts/sync-vendor.mjs <tag>` — fetches the script at that commit, reports the sha256 change, re-pins `UPSTREAM.json`, prints the matching CHANGELOG section (`--check` verifies offline).
|
|
165
|
-
2. Conformance tests check `--version`, schema
|
|
165
|
+
2. Conformance tests check `--version`, schema v4, required fields, recomputed `overall` scores, and the standard-library-only import allowlist.
|
|
166
166
|
3. The upgrade PR body lists script lines touching imports, URLs, environment variables, subprocess calls and file writes, for reviewer attention.
|
|
167
167
|
4. Release with a changelog line naming the upstream tag; users run `opencode plugin update` (OpenCode checks but doesn't apply).
|
|
168
168
|
|
|
169
|
-
A scheduled workflow opens a draft upgrade PR when upstream publishes a newer release (never a downgrade); a nightly canary exercises the vendored script against live upstream data and asserts the generated catalog is schema
|
|
169
|
+
A scheduled workflow opens a draft upgrade PR when upstream publishes a newer release (never a downgrade); a nightly canary exercises the vendored script against live upstream data and asserts the generated catalog is schema v4 (`scripts/canary-check.mjs` pins `EXPECTED_SCHEMA_VERSION = 4` and exits nonzero on any other version); releases additionally gate on the live remote itself (see the publishing gates below); a weekly drift check compares the vendored sha256 against upstream's pinned commit and the published rankings against the published scores.
|
|
170
170
|
|
|
171
171
|
|
|
172
172
|
## Troubleshooting
|
|
@@ -210,7 +210,7 @@ MC_INTEGRATION=1 npx vitest run tests/integration/ # live-server suite (opt-i
|
|
|
210
210
|
- The integration files skip unless `MC_INTEGRATION=1`, so plain `npm test` stays hermetic. The suite drives a real `opencode serve` on 127.0.0.1:14721 and needs the `opencode` CLI and `python3` (skips cleanly when either is missing); one case needs network access and the `sqlite3` CLI.
|
|
211
211
|
- `node scripts/pack-smoke.mjs` (also in CI, after `node scripts/build-tui.mjs`) packs the plugin and checks the tarball against the packaging list (files, `exports`, relative imports, and that the TUI artifact is precompiled — no react import). It does **not** load the package in OpenCode — install the tarball into a scratch project and confirm OpenCode loads it before publishing.
|
|
212
212
|
- Publishing: cut a GitHub release tagged `vX.Y.Z` and `.github/workflows/release.yaml` stages the package on npm through trusted publishing (tokenless OIDC — no token secret; the npmjs.com trusted-publisher entry must name `release.yaml`, the `canonical/model-compare-opencode-plugin` repository and the `release` environment, with "Allow npm publish" and "Allow npm dist-tag" unchecked — stage-only, npm's recommended posture). Nothing goes live until a maintainer approves it with 2FA on npmjs.com (package page → Staged Packages) or via `npm stage approve`. The workflow fails closed when the tag does not match `package.json`'s version, and the gates (build, typecheck, tests, pack-smoke, live-remote canary) run before staging.
|
|
213
|
-
- `fixtures/catalog-v0.
|
|
213
|
+
- `fixtures/catalog-v0.5.0.json` is the sole contract fixture — a recorded schema-v4 catalog; `fixtures/catalog-no-quality.json` is derived from it by `scripts/make-no-quality-fixture.py` (the vendored script's own quality-blind weights, so its scores are internally consistent). No pre-v4 fixture exists; the v4-only gate rejects one. `node scripts/sync-vendor.mjs --check` verifies the pin's sha256 offline.
|
|
214
214
|
- Repo layout: `src/core/` (pure ranking/pinning), `src/sources/` (local + remote acquisition), `src/opencode/` (aliases, guard, commands, storage, refresh), `dist/` (generated `tui.js` artifact, gitignored), `vendor/` (pinned upstream script), `tests/{unit,conformance,integration}`.
|
|
215
215
|
|
|
216
216
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "model-compare-opencode-plugin",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"description": "OpenCode plugin that switches OpenRouter model aliases to the best value-for-money pick from canonical/model-compare's ranking (balanced, price, quality).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"opencode",
|
|
@@ -44,16 +44,17 @@
|
|
|
44
44
|
"prepublishOnly": "npm run build && npm run typecheck && npm test && node scripts/pack-smoke.mjs"
|
|
45
45
|
},
|
|
46
46
|
"dependencies": {
|
|
47
|
-
"@opencode/plugin": "2.
|
|
47
|
+
"@opencode/plugin": "2.1.4"
|
|
48
48
|
},
|
|
49
49
|
"peerDependencies": {
|
|
50
|
-
"@opentui/core": "^0.
|
|
51
|
-
"@opentui/solid": "^0.
|
|
50
|
+
"@opentui/core": "^0.6.8",
|
|
51
|
+
"@opentui/solid": "^0.6.8",
|
|
52
52
|
"solid-js": "^1.9.0"
|
|
53
53
|
},
|
|
54
54
|
"devDependencies": {
|
|
55
|
-
"@
|
|
56
|
-
"@opentui/
|
|
55
|
+
"@opencode/theme": "2.1.4",
|
|
56
|
+
"@opentui/core": "0.6.8",
|
|
57
|
+
"@opentui/solid": "0.6.8",
|
|
57
58
|
"@types/node": "^22.18.13",
|
|
58
59
|
"solid-js": "1.9.12",
|
|
59
60
|
"typescript": "^7.0.2",
|
package/src/core/catalog.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// src/core/catalog.ts
|
|
2
|
-
// Catalog document validation (design §5). Accepts schema
|
|
2
|
+
// Catalog document validation (design §5). Accepts schema v4 documents (only) from
|
|
3
3
|
// either source (vendored script or remote catalog) and projects them onto
|
|
4
4
|
// the plugin's Catalog type. Validation only — ranking lives elsewhere.
|
|
5
5
|
|
|
@@ -49,11 +49,13 @@ export function validateCatalog(
|
|
|
49
49
|
): { ok: true; catalog: Catalog } | { ok: false; reason: string } {
|
|
50
50
|
if (!isObject(raw)) return { ok: false, reason: "catalog must be a JSON object" }
|
|
51
51
|
|
|
52
|
-
// Supported schema:
|
|
53
|
-
// schema_version
|
|
54
|
-
//
|
|
55
|
-
|
|
56
|
-
|
|
52
|
+
// Supported schema: v4 only — upstream's current contract (0.5.0's
|
|
53
|
+
// schema_version 4). The v4 changes are additive provenance (per-model
|
|
54
|
+
// quality_as_of/quality_drift, a reshaped sources.aa block) and change
|
|
55
|
+
// nothing this validator reads; every other version is rejected with the
|
|
56
|
+
// reason string the canary and the stored-document path rely on.
|
|
57
|
+
if (raw.schema_version !== 4) {
|
|
58
|
+
return { ok: false, reason: `schema_version must be 4, got ${JSON.stringify(raw.schema_version)}` }
|
|
57
59
|
}
|
|
58
60
|
if (typeof raw.generated_at !== "string") {
|
|
59
61
|
return { ok: false, reason: "generated_at must be a string" }
|
|
@@ -225,7 +227,7 @@ export function validateStoredCatalog(
|
|
|
225
227
|
})
|
|
226
228
|
: stored.models
|
|
227
229
|
return validateCatalog({
|
|
228
|
-
schema_version:
|
|
230
|
+
schema_version: 4,
|
|
229
231
|
generated_at: stored.generatedAt,
|
|
230
232
|
parameters: { weights: stored.weights },
|
|
231
233
|
models,
|
package/src/core/types.ts
CHANGED
|
@@ -47,7 +47,7 @@ export type Catalog = {
|
|
|
47
47
|
models: ModelEntry[]
|
|
48
48
|
}
|
|
49
49
|
|
|
50
|
-
// Upstream.s PRIORITY_WEIGHTS at the pinned tag (v0.
|
|
50
|
+
// Upstream.s PRIORITY_WEIGHTS at the pinned tag (v0.5.0; unchanged since v0.3.0). Used when a
|
|
51
51
|
// document carries no `parameters.weights` at all (design §5); individual
|
|
52
52
|
// missing keys fall back to weight 0 instead.
|
|
53
53
|
export const DEFAULT_WEIGHTS: Record<Priority, Weights> = {
|
package/src/tui.tsx
CHANGED
|
@@ -39,7 +39,7 @@
|
|
|
39
39
|
|
|
40
40
|
import { Plugin } from "@opencode/plugin/tui"
|
|
41
41
|
import type { KeymapCommand } from "@opencode/plugin/tui/context"
|
|
42
|
-
import { TextAttributes, RGBA } from "@opentui/core"
|
|
42
|
+
import { TextAttributes, RGBA, type ColorInput } from "@opentui/core"
|
|
43
43
|
// The OpenTUI JSX element types (<text fg>, <box>): a type-only import loads
|
|
44
44
|
// the package's solid-js JSX augmentation without a runtime import — the TUI
|
|
45
45
|
// host already provides the runtime side.
|
|
@@ -101,7 +101,7 @@ let armedClose: (() => void) | null = null
|
|
|
101
101
|
// when a dialog is armed, cleared by its close() alongside armedClose.
|
|
102
102
|
let armedReveal: ((center: boolean) => void) | null = null
|
|
103
103
|
|
|
104
|
-
function StatusLine(props: { line: () => string; fg?:
|
|
104
|
+
function StatusLine(props: { line: () => string; fg?: ColorInput }) {
|
|
105
105
|
return <text fg={props.fg}>{props.line()}</text>
|
|
106
106
|
}
|
|
107
107
|
|
package/vendor/UPSTREAM.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"repo": "canonical/model-compare",
|
|
3
|
-
"tag": "v0.
|
|
4
|
-
"commit": "
|
|
5
|
-
"sha256": "
|
|
3
|
+
"tag": "v0.5.0",
|
|
4
|
+
"commit": "2e0e0cbb0ebf1d1b3f03b2c1c949eb930730d8a4",
|
|
5
|
+
"sha256": "cebf2d100b020183c0236f1d127768933bca0e10ed615dd00ee549ebb86323b4"
|
|
6
6
|
}
|
package/vendor/model_compare.py
CHANGED
|
@@ -68,7 +68,23 @@ OPENROUTER_ZDR_URL = "https://openrouter.ai/api/v1/endpoints/zdr"
|
|
|
68
68
|
# (legacy endpoints 410 after 2026-11-04; see
|
|
69
69
|
# https://artificialanalysis.ai/data-api/migrate-v2-data).
|
|
70
70
|
AA_API_URL = "https://artificialanalysis.ai/api/v2/language/models/free"
|
|
71
|
-
|
|
71
|
+
# Cache key for the AA API entries envelope. v3 adds the per-fetch `as_of`
|
|
72
|
+
# timestamp on every entry and the envelope's intelligence_index_version;
|
|
73
|
+
# payloads under the pre-v3 keys predate those fields and must never be read.
|
|
74
|
+
AA_ENTRIES_CACHE = "aa-intelligence-v3"
|
|
75
|
+
# The repo's own published catalog.json -- built by keyed CI runs, hydrated
|
|
76
|
+
# by unkeyed local runs (design: quality source reconciliation, source 2).
|
|
77
|
+
# The github.io host 301s to plain http://model-compare.dev/catalog.json, an
|
|
78
|
+
# unauthenticated downgrade on ranking-critical data; the https direct URL is
|
|
79
|
+
# verified 200 application/json. Do not point this back at github.io.
|
|
80
|
+
QUALITY_CATALOG_URL = "https://model-compare.dev/catalog.json"
|
|
81
|
+
# Cache key for the hydration envelope {"by_id", "generated_at",
|
|
82
|
+
# "schema_version"}. Only usable (non-empty by_id) results are ever written.
|
|
83
|
+
QUALITY_CATALOG_CACHE = "published-catalog"
|
|
84
|
+
# A published catalog generated more than this many days ago is stale:
|
|
85
|
+
# warn but use -- the records' as_of makes the staleness visible downstream.
|
|
86
|
+
QUALITY_CATALOG_MAX_AGE_DAYS = 14
|
|
87
|
+
VERSION = "0.5.0"
|
|
72
88
|
USER_AGENT = f"model-compare/{VERSION} (https://github.com/canonical/model-compare)"
|
|
73
89
|
|
|
74
90
|
# opencode expects provider-qualified model ids: openrouter/<provider/model>.
|
|
@@ -87,7 +103,7 @@ PRIORITY_WEIGHTS = {
|
|
|
87
103
|
"quality": {"quality": 0.60, "price": 0.20, "context": 0.10, "age": 0.10},
|
|
88
104
|
}
|
|
89
105
|
|
|
90
|
-
CATALOG_SCHEMA_VERSION =
|
|
106
|
+
CATALOG_SCHEMA_VERSION = 4
|
|
91
107
|
|
|
92
108
|
# The drop-reason keys build_candidates counts, verbatim -- keep in sync with
|
|
93
109
|
# its drop() call sites. pool.dropped lists all of them zero-filled so the
|
|
@@ -1196,6 +1212,183 @@ def fetch_session_cost(args) -> dict | None:
|
|
|
1196
1212
|
return result
|
|
1197
1213
|
|
|
1198
1214
|
|
|
1215
|
+
# ---------------------------------------------------------------------------
|
|
1216
|
+
# Published catalog quality (hydration for unkeyed runs)
|
|
1217
|
+
# ---------------------------------------------------------------------------
|
|
1218
|
+
|
|
1219
|
+
|
|
1220
|
+
# Pre-reconciliation published catalogs (schema v3, written before the AA
|
|
1221
|
+
# tier was relabeled) record the AA tier as "api"; hydration normalizes it
|
|
1222
|
+
# to the final "aa" label. Every other recorded label passes through as-is.
|
|
1223
|
+
_QUALITY_SOURCE_NORMALIZATION = {"api": "aa"}
|
|
1224
|
+
|
|
1225
|
+
|
|
1226
|
+
def catalog_quality_by_id(doc) -> dict:
|
|
1227
|
+
"""Pure parser: a published catalog document -> {id: quality record}.
|
|
1228
|
+
|
|
1229
|
+
No fetch, no cache, no args object -- the fetcher owns transport and
|
|
1230
|
+
freshness; this owns shape. Records are {"index", "source", "as_of"}
|
|
1231
|
+
and feed resolve_quality's catalog_quality parameter on unkeyed runs;
|
|
1232
|
+
the values were produced by a keyed CI run, so hydration is an exact
|
|
1233
|
+
catalog-id join with no re-matching.
|
|
1234
|
+
|
|
1235
|
+
Gates (design: failure modes):
|
|
1236
|
+
- schema_version must be an int >= 3; anything older warns
|
|
1237
|
+
"published catalog schema vX, expected >= 3; skipping hydration"
|
|
1238
|
+
and yields {}. Accepting v3 is the deploy-day skew case: the live
|
|
1239
|
+
artifact predates the v4 per-row quality_as_of.
|
|
1240
|
+
- Any shape surprise -- a non-dict document, a non-int schema_version,
|
|
1241
|
+
a non-list models, a row without a usable id, or a matched row whose
|
|
1242
|
+
quality is not a finite number or whose quality_match is not a
|
|
1243
|
+
non-empty string -- warns and yields {} for the WHOLE artifact.
|
|
1244
|
+
Never a partial hydration, never a raise: the catalog is
|
|
1245
|
+
machine-generated, so one malformed row means this is not the
|
|
1246
|
+
artifact we think it is.
|
|
1247
|
+
- Rows with quality None and quality_match None are legitimate
|
|
1248
|
+
unmatched models; they are skipped silently (nothing to hydrate).
|
|
1249
|
+
|
|
1250
|
+
Normalization: v3 rows map quality_match "api" -> "aa" and date every
|
|
1251
|
+
row with the catalog's generated_at -- the live v3 artifact carries
|
|
1252
|
+
quality_as_of: null, which must be ignored, not passed through.
|
|
1253
|
+
v4+ rows keep their recorded quality_as_of (None when absent) and
|
|
1254
|
+
source label as-is: that label is the row-level truth inside the
|
|
1255
|
+
published artifact (a keyed CI run emits "aa"/"openrouter", never
|
|
1256
|
+
"catalog"). resolve_quality's catalog tier owns the run-level label:
|
|
1257
|
+
it relabels every hydrated record to the chain label "catalog".
|
|
1258
|
+
"""
|
|
1259
|
+
if not isinstance(doc, dict):
|
|
1260
|
+
warn("published catalog returned an unexpected response shape")
|
|
1261
|
+
return {}
|
|
1262
|
+
schema_version = doc.get("schema_version")
|
|
1263
|
+
if type(schema_version) is not int:
|
|
1264
|
+
warn("published catalog returned an unexpected response shape")
|
|
1265
|
+
return {}
|
|
1266
|
+
if schema_version < 3:
|
|
1267
|
+
warn(
|
|
1268
|
+
f"published catalog schema v{schema_version}, expected >= 3; "
|
|
1269
|
+
"skipping hydration"
|
|
1270
|
+
)
|
|
1271
|
+
return {}
|
|
1272
|
+
models = doc.get("models")
|
|
1273
|
+
if not isinstance(models, list):
|
|
1274
|
+
warn("published catalog returned an unexpected response shape")
|
|
1275
|
+
return {}
|
|
1276
|
+
generated_at = doc.get("generated_at")
|
|
1277
|
+
v3_as_of = generated_at if isinstance(generated_at, str) and generated_at else None
|
|
1278
|
+
by_id: dict = {}
|
|
1279
|
+
for entry in models:
|
|
1280
|
+
if not isinstance(entry, dict):
|
|
1281
|
+
warn("published catalog returned an unexpected response shape")
|
|
1282
|
+
return {}
|
|
1283
|
+
model_id = entry.get("id")
|
|
1284
|
+
if not (isinstance(model_id, str) and model_id):
|
|
1285
|
+
warn("published catalog returned an unexpected response shape")
|
|
1286
|
+
return {}
|
|
1287
|
+
quality = entry.get("quality")
|
|
1288
|
+
match = entry.get("quality_match")
|
|
1289
|
+
if match is None and quality is None:
|
|
1290
|
+
continue # legitimate unmatched row: nothing to hydrate
|
|
1291
|
+
if not (isinstance(match, str) and match):
|
|
1292
|
+
warn("published catalog returned an unexpected response shape")
|
|
1293
|
+
return {}
|
|
1294
|
+
if not isinstance(quality, (int, float)) or isinstance(quality, bool):
|
|
1295
|
+
warn("published catalog returned an unexpected response shape")
|
|
1296
|
+
return {}
|
|
1297
|
+
try:
|
|
1298
|
+
index = float(quality) # huge ints raise OverflowError
|
|
1299
|
+
except OverflowError:
|
|
1300
|
+
warn("published catalog returned an unexpected response shape")
|
|
1301
|
+
return {}
|
|
1302
|
+
# Non-finite indexes would serialize as NaN and poison quality.
|
|
1303
|
+
if not math.isfinite(index):
|
|
1304
|
+
warn("published catalog returned an unexpected response shape")
|
|
1305
|
+
return {}
|
|
1306
|
+
# The AA indexes run 0..100 (the range the catalog validator
|
|
1307
|
+
# enforces on quality); an out-of-range value is poisoned data and
|
|
1308
|
+
# degrades the whole artifact. The surface is unkeyed runs: only
|
|
1309
|
+
# they hydrate from the published catalog.
|
|
1310
|
+
if not 0.0 <= index <= 100.0:
|
|
1311
|
+
warn("published catalog returned an unexpected response shape")
|
|
1312
|
+
return {}
|
|
1313
|
+
if schema_version == 3:
|
|
1314
|
+
as_of = v3_as_of
|
|
1315
|
+
else:
|
|
1316
|
+
recorded = entry.get("quality_as_of")
|
|
1317
|
+
as_of = recorded if isinstance(recorded, str) and recorded else None
|
|
1318
|
+
by_id[model_id] = {
|
|
1319
|
+
"index": index,
|
|
1320
|
+
"source": _QUALITY_SOURCE_NORMALIZATION.get(match, match),
|
|
1321
|
+
"as_of": as_of,
|
|
1322
|
+
}
|
|
1323
|
+
return by_id
|
|
1324
|
+
|
|
1325
|
+
|
|
1326
|
+
def fetch_quality_catalog(args) -> dict | None:
|
|
1327
|
+
"""Quality records from the published catalog.json, or None.
|
|
1328
|
+
|
|
1329
|
+
The third quality source (design: source 2): the repo's own published
|
|
1330
|
+
catalog artifact, built by keyed CI runs and hydrated by unkeyed local
|
|
1331
|
+
runs. Never called on keyed runs -- a CI/site build hydrating from the
|
|
1332
|
+
published catalog would consume its own output (self-reference loop);
|
|
1333
|
+
Task 4 owns that wiring.
|
|
1334
|
+
|
|
1335
|
+
Returns None only on TRANSPORT failure (after fetch_json's retries);
|
|
1336
|
+
otherwise the envelope {"by_id": {id: record}, "generated_at":
|
|
1337
|
+
str|None, "schema_version": int}. An empty by_id means
|
|
1338
|
+
fetched-but-unusable (schema gate fail, malformed shape) -- Task 4's
|
|
1339
|
+
footer treats None and empty by_id identically ("catalog unavailable").
|
|
1340
|
+
|
|
1341
|
+
The envelope is cached under QUALITY_CATALOG_CACHE with the standard
|
|
1342
|
+
TTL, and only a usable (non-empty by_id) result is ever written: a
|
|
1343
|
+
degraded payload must self-heal on the next run. --no-cache neither
|
|
1344
|
+
reads nor writes. Staleness: generated_at older than
|
|
1345
|
+
QUALITY_CATALOG_MAX_AGE_DAYS warns but uses -- as_of makes the
|
|
1346
|
+
staleness visible downstream (the check runs only when rows actually
|
|
1347
|
+
hydrate -- warning about the age of a catalog we are not using is
|
|
1348
|
+
noise); an unusable generated_at warns that
|
|
1349
|
+
staleness cannot be checked and still uses the rows (their as_of then
|
|
1350
|
+
shows None rather than a wrong date).
|
|
1351
|
+
"""
|
|
1352
|
+
if not args.no_cache:
|
|
1353
|
+
cached = load_cache(QUALITY_CATALOG_CACHE, args.cache_ttl)
|
|
1354
|
+
if isinstance(cached, dict) and isinstance(cached.get("by_id"), dict):
|
|
1355
|
+
return cached
|
|
1356
|
+
try:
|
|
1357
|
+
doc = fetch_json(QUALITY_CATALOG_URL, timeout=30, retries=3)
|
|
1358
|
+
except Exception as exc:
|
|
1359
|
+
warn(f"published catalog request failed ({exc})")
|
|
1360
|
+
return None
|
|
1361
|
+
by_id = catalog_quality_by_id(doc)
|
|
1362
|
+
generated_at = doc.get("generated_at") if isinstance(doc, dict) else None
|
|
1363
|
+
if not isinstance(generated_at, str):
|
|
1364
|
+
generated_at = None
|
|
1365
|
+
schema_version = doc.get("schema_version") if isinstance(doc, dict) else 0
|
|
1366
|
+
if type(schema_version) is not int:
|
|
1367
|
+
schema_version = 0
|
|
1368
|
+
result = {
|
|
1369
|
+
"by_id": by_id,
|
|
1370
|
+
"generated_at": generated_at,
|
|
1371
|
+
"schema_version": schema_version,
|
|
1372
|
+
}
|
|
1373
|
+
if by_id:
|
|
1374
|
+
generated = parse_iso_datetime(generated_at)
|
|
1375
|
+
if generated is None:
|
|
1376
|
+
warn(
|
|
1377
|
+
"published catalog generated_at is missing or unparseable; "
|
|
1378
|
+
"cannot check staleness"
|
|
1379
|
+
)
|
|
1380
|
+
else:
|
|
1381
|
+
age_days = (datetime.now(timezone.utc) - generated).total_seconds() / 86400
|
|
1382
|
+
if age_days > QUALITY_CATALOG_MAX_AGE_DAYS:
|
|
1383
|
+
warn(
|
|
1384
|
+
f"published catalog is stale (generated {generated_at}, "
|
|
1385
|
+
f"{age_days:.0f} days old); using it anyway"
|
|
1386
|
+
)
|
|
1387
|
+
if not args.no_cache:
|
|
1388
|
+
save_cache(QUALITY_CATALOG_CACHE, result)
|
|
1389
|
+
return result
|
|
1390
|
+
|
|
1391
|
+
|
|
1199
1392
|
# ---------------------------------------------------------------------------
|
|
1200
1393
|
# OpenRouter catalog
|
|
1201
1394
|
# ---------------------------------------------------------------------------
|
|
@@ -1215,7 +1408,9 @@ def fetch_openrouter_models(args):
|
|
|
1215
1408
|
|
|
1216
1409
|
|
|
1217
1410
|
FRONTEND_DISCOUNTS_CACHE = "openrouter-frontend-discounts"
|
|
1218
|
-
|
|
1411
|
+
# v2: AA records are keyed by exact catalog slug and carry as_of (#17);
|
|
1412
|
+
# v1 cached the conflated date-stripped shape, so the old key is never read.
|
|
1413
|
+
FRONTEND_AA_CACHE = "openrouter-frontend-aa-v2"
|
|
1219
1414
|
|
|
1220
1415
|
|
|
1221
1416
|
def fetch_openrouter_frontend(args):
|
|
@@ -1273,7 +1468,8 @@ def fetch_openrouter_frontend(args):
|
|
|
1273
1468
|
key = slug if variant in ("", "standard") else f"{slug}:{variant}"
|
|
1274
1469
|
fresh_discounts.setdefault(key, float(raw))
|
|
1275
1470
|
fresh_aa = build_aa_benchmarks(
|
|
1276
|
-
data.get("benchmarks") if isinstance(data, dict) else None
|
|
1471
|
+
data.get("benchmarks") if isinstance(data, dict) else None,
|
|
1472
|
+
entries,
|
|
1277
1473
|
)
|
|
1278
1474
|
if fresh_discounts:
|
|
1279
1475
|
discounts = fresh_discounts
|
|
@@ -1764,7 +1960,7 @@ def parse_iso_datetime(value):
|
|
|
1764
1960
|
# ---------------------------------------------------------------------------
|
|
1765
1961
|
|
|
1766
1962
|
|
|
1767
|
-
def aa_api_entries(api_key: str) -> list:
|
|
1963
|
+
def aa_api_entries(api_key: str) -> tuple[list, float | int | None]:
|
|
1768
1964
|
"""Intelligence entries from AA's V2 language models endpoint.
|
|
1769
1965
|
|
|
1770
1966
|
Documented envelope: {tier, intelligence_index_version, pagination,
|
|
@@ -1775,8 +1971,16 @@ def aa_api_entries(api_key: str) -> list:
|
|
|
1775
1971
|
has_more/total_pages contract with a 25-page hard cap; a failure past
|
|
1776
1972
|
page 1 is best-effort (warn, keep earlier pages), a page-1 failure
|
|
1777
1973
|
propagates.
|
|
1974
|
+
|
|
1975
|
+
Returns (entries, version): version is the envelope's
|
|
1976
|
+
intelligence_index_version (number-or-null) -- provenance for the whole
|
|
1977
|
+
fetch. Every surviving entry carries `as_of`, one ISO fetch timestamp
|
|
1978
|
+
shared by the whole fetch, so the cache and any downstream record can
|
|
1979
|
+
date the data even after the envelope's version is lost.
|
|
1778
1980
|
"""
|
|
1779
1981
|
entries = {}
|
|
1982
|
+
version = None
|
|
1983
|
+
as_of = datetime.now(timezone.utc).isoformat()
|
|
1780
1984
|
page = 1
|
|
1781
1985
|
while page <= 25: # hard cap: halt after processing page 25
|
|
1782
1986
|
try:
|
|
@@ -1791,6 +1995,15 @@ def aa_api_entries(api_key: str) -> list:
|
|
|
1791
1995
|
warn(f"AA API page {page} failed ({exc}); using entries collected so far")
|
|
1792
1996
|
break
|
|
1793
1997
|
data = payload.get("data") if isinstance(payload, dict) else None
|
|
1998
|
+
# The envelope's version is one provenance fact for the whole
|
|
1999
|
+
# fetch; the last page's value wins (they are identical in practice).
|
|
2000
|
+
raw_version = (
|
|
2001
|
+
payload.get("intelligence_index_version")
|
|
2002
|
+
if isinstance(payload, dict)
|
|
2003
|
+
else None
|
|
2004
|
+
)
|
|
2005
|
+
if isinstance(raw_version, (int, float)) and not isinstance(raw_version, bool):
|
|
2006
|
+
version = raw_version
|
|
1794
2007
|
for item in data if isinstance(data, list) else []:
|
|
1795
2008
|
if not isinstance(item, dict):
|
|
1796
2009
|
continue
|
|
@@ -1823,6 +2036,10 @@ def aa_api_entries(api_key: str) -> list:
|
|
|
1823
2036
|
"key": key,
|
|
1824
2037
|
"name": name if isinstance(name, str) and name else key,
|
|
1825
2038
|
"index": index,
|
|
2039
|
+
# One fetch timestamp for the whole fetch: every entry
|
|
2040
|
+
# from this run shares it, so the cache payload and any
|
|
2041
|
+
# downstream record can date the data exactly.
|
|
2042
|
+
"as_of": as_of,
|
|
1826
2043
|
},
|
|
1827
2044
|
)
|
|
1828
2045
|
pag = payload.get("pagination") if isinstance(payload, dict) else None
|
|
@@ -1834,37 +2051,53 @@ def aa_api_entries(api_key: str) -> list:
|
|
|
1834
2051
|
if isinstance(total_pages, int) and page >= total_pages:
|
|
1835
2052
|
break
|
|
1836
2053
|
page += 1
|
|
1837
|
-
return list(entries.values())
|
|
2054
|
+
return list(entries.values()), version
|
|
1838
2055
|
|
|
1839
2056
|
|
|
1840
2057
|
def fetch_aa_entries(args):
|
|
1841
2058
|
"""AA intelligence entries from the keyed AA API v2, or nothing.
|
|
1842
2059
|
|
|
1843
|
-
Returns (entries, source, cached). With a key, an API
|
|
1844
|
-
empty result warns and yields ([], None, False).
|
|
1845
|
-
silent and yields ([], None, False):
|
|
1846
|
-
|
|
2060
|
+
Returns (entries, source, cached, index_version). With a key, an API
|
|
2061
|
+
failure or an empty result warns and yields ([], None, False, None).
|
|
2062
|
+
Without a key this is silent and yields ([], None, False, None):
|
|
2063
|
+
OpenRouter's benchmarks cover what they cover and the catalog records
|
|
2064
|
+
the absence (sources.aa).
|
|
2065
|
+
|
|
2066
|
+
`index_version` is the envelope's intelligence_index_version (number
|
|
2067
|
+
or null) -- provenance for the whole fetch. Entries carry `as_of`, one
|
|
2068
|
+
ISO fetch timestamp for the whole fetch; a cache hit reuses the
|
|
2069
|
+
stamped payload verbatim so cached rows stay dated exactly like fresh
|
|
2070
|
+
ones.
|
|
1847
2071
|
"""
|
|
1848
2072
|
if not args.no_cache:
|
|
1849
|
-
#
|
|
1850
|
-
#
|
|
1851
|
-
# build_catalog rejects any other
|
|
1852
|
-
|
|
2073
|
+
# v3: payloads cached under earlier keys predate the as_of stamp and
|
|
2074
|
+
# the intelligence_index_version capture and must never be read.
|
|
2075
|
+
# Only AA API entries are read; build_catalog rejects any other
|
|
2076
|
+
# source as unknown.
|
|
2077
|
+
cached = load_cache(AA_ENTRIES_CACHE, args.cache_ttl)
|
|
1853
2078
|
if isinstance(cached, dict) and cached.get("source") == "AA API v2":
|
|
1854
|
-
return
|
|
2079
|
+
return (
|
|
2080
|
+
cached.get("entries", []),
|
|
2081
|
+
cached.get("source"),
|
|
2082
|
+
True,
|
|
2083
|
+
cached.get("version"),
|
|
2084
|
+
)
|
|
1855
2085
|
api_key = args.aa_api_key or os.environ.get("AA_API_KEY")
|
|
1856
2086
|
if not api_key:
|
|
1857
|
-
return [], None, False
|
|
2087
|
+
return [], None, False, None
|
|
1858
2088
|
try:
|
|
1859
|
-
entries = aa_api_entries(api_key)
|
|
2089
|
+
entries, version = aa_api_entries(api_key)
|
|
1860
2090
|
except Exception as exc:
|
|
1861
2091
|
warn(f"AA API request failed ({exc})")
|
|
1862
|
-
return [], None, False
|
|
2092
|
+
return [], None, False, None
|
|
1863
2093
|
if not entries:
|
|
1864
2094
|
warn("AA API returned no intelligence scores")
|
|
1865
|
-
return [], None, False
|
|
1866
|
-
save_cache(
|
|
1867
|
-
|
|
2095
|
+
return [], None, False, None
|
|
2096
|
+
save_cache(
|
|
2097
|
+
AA_ENTRIES_CACHE,
|
|
2098
|
+
{"entries": entries, "source": "AA API v2", "version": version},
|
|
2099
|
+
)
|
|
2100
|
+
return entries, "AA API v2", False, version
|
|
1868
2101
|
|
|
1869
2102
|
|
|
1870
2103
|
# ---------------------------------------------------------------------------
|
|
@@ -1873,6 +2106,13 @@ def fetch_aa_entries(args):
|
|
|
1873
2106
|
|
|
1874
2107
|
|
|
1875
2108
|
def build_aa_lookup(entries):
|
|
2109
|
+
"""Index the AA entries into exact and fuzzy lookup maps.
|
|
2110
|
+
|
|
2111
|
+
Values are (index, as_of) pairs, not bare floats: the as_of travels
|
|
2112
|
+
with the match so resolve_quality can date every AA-sourced record
|
|
2113
|
+
without re-scanning the raw entry list. Entries without a usable key
|
|
2114
|
+
or a finite numeric index are skipped.
|
|
2115
|
+
"""
|
|
1876
2116
|
exact = {}
|
|
1877
2117
|
fuzzy = []
|
|
1878
2118
|
for entry in entries:
|
|
@@ -1885,11 +2125,12 @@ def build_aa_lookup(entries):
|
|
|
1885
2125
|
# Non-finite indexes would serialize as NaN and poison quality.
|
|
1886
2126
|
if not math.isfinite(index):
|
|
1887
2127
|
continue
|
|
2128
|
+
as_of = entry.get("as_of") if isinstance(entry.get("as_of"), str) else None
|
|
1888
2129
|
|
|
1889
2130
|
def put(tier, raw):
|
|
1890
2131
|
normalized = norm_key(raw)
|
|
1891
2132
|
if normalized:
|
|
1892
|
-
exact.setdefault((tier, normalized), index)
|
|
2133
|
+
exact.setdefault((tier, normalized), (index, as_of))
|
|
1893
2134
|
|
|
1894
2135
|
if "/" in key:
|
|
1895
2136
|
put(0, key)
|
|
@@ -1900,11 +2141,17 @@ def build_aa_lookup(entries):
|
|
|
1900
2141
|
put(3, PAREN_RE.sub(" ", name))
|
|
1901
2142
|
tokens = set(norm_key(key).split()) | set(norm_key(name).split())
|
|
1902
2143
|
if tokens:
|
|
1903
|
-
fuzzy.append((tokens, index))
|
|
2144
|
+
fuzzy.append((tokens, (index, as_of)))
|
|
1904
2145
|
return exact, fuzzy
|
|
1905
2146
|
|
|
1906
2147
|
|
|
1907
2148
|
def match_quality(model, exact, fuzzy, allow_fuzzy=False):
|
|
2149
|
+
"""Match a candidate to an AA entry; exact tiers only unless fuzzy is on.
|
|
2150
|
+
|
|
2151
|
+
Returns the winning entry's (index, as_of) pair, or None when no tier
|
|
2152
|
+
matches. Values come straight out of build_aa_lookup's maps, so the
|
|
2153
|
+
as_of is the entry's own fetch stamp, never a re-lookup.
|
|
2154
|
+
"""
|
|
1908
2155
|
model_id = model["id"]
|
|
1909
2156
|
name = model.get("name") or ""
|
|
1910
2157
|
display = PROVIDER_PREFIX_RE.sub("", name).strip()
|
|
@@ -1927,14 +2174,14 @@ def match_quality(model, exact, fuzzy, allow_fuzzy=False):
|
|
|
1927
2174
|
own = set(norm_key(base).split()) | set(norm_key(display).split())
|
|
1928
2175
|
if own:
|
|
1929
2176
|
best = 0.0
|
|
1930
|
-
|
|
1931
|
-
for tokens,
|
|
2177
|
+
best_match = None
|
|
2178
|
+
for tokens, match in fuzzy:
|
|
1932
2179
|
overlap = len(own & tokens) / len(own | tokens)
|
|
1933
2180
|
if overlap > best:
|
|
1934
2181
|
best = overlap
|
|
1935
|
-
|
|
1936
|
-
if best >= 0.5 and
|
|
1937
|
-
return
|
|
2182
|
+
best_match = match
|
|
2183
|
+
if best >= 0.5 and best_match is not None:
|
|
2184
|
+
return best_match
|
|
1938
2185
|
return None
|
|
1939
2186
|
|
|
1940
2187
|
|
|
@@ -1946,24 +2193,37 @@ def base_model_id(model_id: str) -> str:
|
|
|
1946
2193
|
AA_BENCHMARK_FIELDS = ("intelligence_index", "coding_index", "agentic_index")
|
|
1947
2194
|
|
|
1948
2195
|
|
|
1949
|
-
def build_aa_benchmarks(benchmarks) -> dict:
|
|
1950
|
-
"""Map
|
|
1951
|
-
|
|
1952
|
-
|
|
1953
|
-
|
|
1954
|
-
|
|
1955
|
-
|
|
1956
|
-
|
|
2196
|
+
def build_aa_benchmarks(benchmarks, models=None) -> dict:
|
|
2197
|
+
"""Map exact catalog slug -> OR-published AA trio from data.benchmarks.
|
|
2198
|
+
|
|
2199
|
+
Benchmark keys are dated permaslugs (deepseek/deepseek-v4.1-flash-
|
|
2200
|
+
20261017) whose date-stripped form can collide across listings of one
|
|
2201
|
+
model family: the snapshot listing deepseek/deepseek-v4.1-flash-1017
|
|
2202
|
+
and the base listing deepseek/deepseek-v4.1-flash both strip to
|
|
2203
|
+
deepseek/deepseek-v4.1-flash, so blind stripping with latest-date-wins
|
|
2204
|
+
made the base listing borrow the snapshot's score (#17). Keys resolve
|
|
2205
|
+
through three explicit passes over the surviving entries, and every
|
|
2206
|
+
accepted id is claimed by the pass that resolved it, so later passes
|
|
2207
|
+
never overwrite an earlier one:
|
|
2208
|
+
|
|
2209
|
+
1. Primary: a key equal to a models entry's permaslug maps to that
|
|
2210
|
+
entry's slug -- the catalog's own pairing, no guessing. Several
|
|
2211
|
+
dated permaslugs may legitimately pair with one slug (snapshot
|
|
2212
|
+
variants of one listing); among those the latest date wins.
|
|
2213
|
+
2. Undated: a key without a -YYYYMMDD suffix maps to itself, unless
|
|
2214
|
+
pass 1 claimed the id.
|
|
2215
|
+
3. Dated fallback: a dated key the catalog does not pair strips via
|
|
2216
|
+
strip_permaslug_date, accepted only when the stripped id is
|
|
2217
|
+
unclaimed and no other pass-3 key strips to the same id -- an
|
|
2218
|
+
ambiguous strip is the #17 conflation shape, so every candidate
|
|
2219
|
+
for that id drops rather than guess.
|
|
2220
|
+
|
|
2221
|
+
Entries whose values fail the finite-number guard are treated as
|
|
2222
|
+
absent before the passes run; entries with at least one valid field
|
|
2223
|
+
are kept. Records carry "as_of": the YYYYMMDD of the permaslug the
|
|
2224
|
+
trio came from, or None for undated keys.
|
|
1957
2225
|
"""
|
|
1958
|
-
|
|
1959
|
-
seen_date = {}
|
|
1960
|
-
|
|
1961
|
-
def accept(bare, date, trio):
|
|
1962
|
-
if bare in seen_date and seen_date[bare] >= date:
|
|
1963
|
-
return
|
|
1964
|
-
seen_date[bare] = date
|
|
1965
|
-
aa_by_id[bare] = trio
|
|
1966
|
-
|
|
2226
|
+
valid = {} # benchmark key -> (trio, permaslug date or "")
|
|
1967
2227
|
for key, node in (benchmarks or {}).items():
|
|
1968
2228
|
aa = node.get("aa") if isinstance(node, dict) else None
|
|
1969
2229
|
if not isinstance(aa, dict):
|
|
@@ -1979,36 +2239,152 @@ def build_aa_benchmarks(benchmarks) -> dict:
|
|
|
1979
2239
|
trio[field] = float(value)
|
|
1980
2240
|
if not trio:
|
|
1981
2241
|
continue
|
|
1982
|
-
|
|
1983
|
-
|
|
1984
|
-
|
|
1985
|
-
else:
|
|
1986
|
-
|
|
1987
|
-
|
|
2242
|
+
bare = strip_permaslug_date(key)
|
|
2243
|
+
# strip_permaslug_date removed a trailing -YYYYMMDD exactly when
|
|
2244
|
+
# the key was dated; the date is then the final 8 characters.
|
|
2245
|
+
valid[key] = (trio, "" if bare == key else key[-8:])
|
|
2246
|
+
|
|
2247
|
+
permaslug_to_slug = {} # the catalog's own slug/permaslug pairing
|
|
2248
|
+
for entry in models or []:
|
|
2249
|
+
if not isinstance(entry, dict):
|
|
2250
|
+
continue
|
|
2251
|
+
slug = entry.get("slug") or ""
|
|
2252
|
+
permaslug = entry.get("permaslug") or ""
|
|
2253
|
+
if slug and permaslug:
|
|
2254
|
+
# standard and batch variants repeat an identical pair; the
|
|
2255
|
+
# overwrite is the dedup
|
|
2256
|
+
permaslug_to_slug[permaslug] = slug
|
|
2257
|
+
|
|
2258
|
+
aa_by_id = {}
|
|
2259
|
+
claimed = set()
|
|
2260
|
+
|
|
2261
|
+
def record(trio, date):
|
|
2262
|
+
return {**trio, "as_of": date or None}
|
|
1988
2263
|
|
|
2264
|
+
# Pass 1: permaslug keys pair with their own slug; same-slug
|
|
2265
|
+
# snapshots resolve to the latest date (first wins on an equal date).
|
|
2266
|
+
seen_date = {}
|
|
2267
|
+
for key, (trio, date) in valid.items():
|
|
2268
|
+
slug = permaslug_to_slug.get(key)
|
|
2269
|
+
if slug is None or (slug in seen_date and seen_date[slug] >= date):
|
|
2270
|
+
continue
|
|
2271
|
+
seen_date[slug] = date
|
|
2272
|
+
aa_by_id[slug] = record(trio, date)
|
|
2273
|
+
claimed.add(slug)
|
|
1989
2274
|
|
|
1990
|
-
|
|
1991
|
-
|
|
2275
|
+
# Pass 2: undated keys self-map unless pass 1 claimed the id.
|
|
2276
|
+
for key, (trio, date) in valid.items():
|
|
2277
|
+
if date or key in permaslug_to_slug or key in claimed:
|
|
2278
|
+
continue
|
|
2279
|
+
aa_by_id[key] = record(trio, date)
|
|
2280
|
+
claimed.add(key)
|
|
2281
|
+
|
|
2282
|
+
# Pass 3: unpaired dated keys strip, but only onto an unclaimed id
|
|
2283
|
+
# and only when the strip is unambiguous.
|
|
2284
|
+
strip_candidates = {}
|
|
2285
|
+
for key, (trio, date) in valid.items():
|
|
2286
|
+
if not date or key in permaslug_to_slug:
|
|
2287
|
+
continue
|
|
2288
|
+
stripped = strip_permaslug_date(key)
|
|
2289
|
+
strip_candidates.setdefault(stripped, []).append((date, trio))
|
|
2290
|
+
for bare, candidates in strip_candidates.items():
|
|
2291
|
+
if bare in claimed or len(candidates) > 1:
|
|
2292
|
+
continue
|
|
2293
|
+
date, trio = candidates[0]
|
|
2294
|
+
aa_by_id[bare] = record(trio, date)
|
|
2295
|
+
claimed.add(bare)
|
|
2296
|
+
|
|
2297
|
+
return aa_by_id
|
|
1992
2298
|
|
|
1993
|
-
|
|
1994
|
-
|
|
1995
|
-
|
|
1996
|
-
|
|
2299
|
+
|
|
2300
|
+
def resolve_quality(
|
|
2301
|
+
candidates, aa_by_id, exact, fuzzy, aa_source, catalog_quality=None
|
|
2302
|
+
):
|
|
2303
|
+
"""Per-candidate quality records: AA API primary, catalog next, OR last.
|
|
2304
|
+
|
|
2305
|
+
Resolution order per candidate (inverted from the pre-reconciliation
|
|
2306
|
+
behavior where OpenRouter's benchmarks won):
|
|
2307
|
+
1. AA API entries -- match_quality's exact tiers (fuzzy stays off in
|
|
2308
|
+
production: it is the proven variant-conflation bug).
|
|
2309
|
+
2. catalog_quality -- records hydrated from the published catalog on
|
|
2310
|
+
unkeyed runs. Every winner is labeled "catalog": the run-level
|
|
2311
|
+
provenance is the source chain (read from our published catalog),
|
|
2312
|
+
overriding the row's own "aa"/"openrouter" label -- that label is
|
|
2313
|
+
the keyed CI run's row-level truth inside the artifact, and no
|
|
2314
|
+
real catalog row ever carries "catalog". as_of stays the row's
|
|
2315
|
+
own stamp.
|
|
2316
|
+
3. OpenRouter's benchmark trio -- per-slug, matched via
|
|
2317
|
+
base_model_id, so a :variant suffix inherits its base's trio.
|
|
2318
|
+
4. No match -- the candidate stays quality-blind downstream.
|
|
2319
|
+
|
|
2320
|
+
Every winner is a record {"index", "source", "as_of", "drift",
|
|
2321
|
+
"or_index", "or_as_of"}; compute_scores and build_schedules read
|
|
2322
|
+
record["index"], so cand["quality"] stays the float index they already
|
|
2323
|
+
expect. Drift is computed only when the winner is NOT OpenRouter: the
|
|
2324
|
+
OR trio for the same base model acts as the check value, and a
|
|
2325
|
+
disagreement of >= 3.0 points flags the record. When the winner IS
|
|
2326
|
+
OpenRouter (or no OR value exists) the drift pair is None -- there is
|
|
2327
|
+
no second source to disagree with.
|
|
1997
2328
|
"""
|
|
1998
|
-
fallback_source = {"AA API v2": "api"}.get(aa_source)
|
|
1999
2329
|
quality_by_id = {}
|
|
2000
2330
|
source_by_id = {}
|
|
2001
2331
|
for cand in candidates:
|
|
2002
|
-
|
|
2003
|
-
|
|
2004
|
-
|
|
2005
|
-
|
|
2006
|
-
|
|
2332
|
+
record = None
|
|
2333
|
+
# Tier 1: AA API exact match. match_quality returns the winning
|
|
2334
|
+
# entry's (index, as_of) pair.
|
|
2335
|
+
match = match_quality({"id": cand["id"], "name": cand["name"]}, exact, fuzzy)
|
|
2336
|
+
if match is not None:
|
|
2337
|
+
index, as_of = match
|
|
2338
|
+
record = {"index": index, "source": "aa", "as_of": as_of}
|
|
2339
|
+
# Tier 2: catalog record (unkeyed runs hydrate from the published
|
|
2340
|
+
# catalog). The run-level provenance label is the source chain:
|
|
2341
|
+
# every hydrated record is tagged "catalog" -- read from our
|
|
2342
|
+
# published catalog -- overriding the row's own label, which is
|
|
2343
|
+
# the keyed CI run's row-level truth inside the artifact ("aa"/
|
|
2344
|
+
# "openrouter"; no real catalog row ever carries "catalog").
|
|
2345
|
+
# as_of stays the row's own stamp (v4 rows carry the CI run's
|
|
2346
|
+
# recorded quality_as_of; v3 rows the catalog's generated_at --
|
|
2347
|
+
# catalog_quality_by_id normalizes that).
|
|
2348
|
+
if record is None and catalog_quality is not None:
|
|
2349
|
+
catalog_record = catalog_quality.get(cand["id"])
|
|
2350
|
+
if catalog_record is not None:
|
|
2351
|
+
record = {
|
|
2352
|
+
"index": catalog_record["index"],
|
|
2353
|
+
"source": "catalog",
|
|
2354
|
+
"as_of": catalog_record["as_of"],
|
|
2355
|
+
}
|
|
2356
|
+
# Tier 3: OpenRouter's per-slug benchmark trio (last resort).
|
|
2357
|
+
if record is None:
|
|
2358
|
+
bench = aa_by_id.get(base_model_id(cand["id"]))
|
|
2359
|
+
intelligence = bench.get("intelligence_index") if bench else None
|
|
2360
|
+
if intelligence is not None:
|
|
2361
|
+
record = {
|
|
2362
|
+
"index": intelligence,
|
|
2363
|
+
"source": "openrouter",
|
|
2364
|
+
"as_of": bench.get("as_of"),
|
|
2365
|
+
}
|
|
2366
|
+
if record is None:
|
|
2007
2367
|
continue
|
|
2008
|
-
|
|
2009
|
-
|
|
2010
|
-
|
|
2011
|
-
|
|
2368
|
+
|
|
2369
|
+
# Drift: only meaningful when the winner is not OpenRouter itself.
|
|
2370
|
+
bench = aa_by_id.get(base_model_id(cand["id"])) or {}
|
|
2371
|
+
or_intelligence = bench.get("intelligence_index")
|
|
2372
|
+
if (
|
|
2373
|
+
record["source"] in ("aa", "catalog")
|
|
2374
|
+
and isinstance(or_intelligence, (int, float))
|
|
2375
|
+
and not isinstance(or_intelligence, bool)
|
|
2376
|
+
and math.isfinite(or_intelligence)
|
|
2377
|
+
):
|
|
2378
|
+
record["or_index"] = or_intelligence
|
|
2379
|
+
record["or_as_of"] = bench.get("as_of")
|
|
2380
|
+
record["drift"] = abs(record["index"] - or_intelligence) >= 3.0
|
|
2381
|
+
else:
|
|
2382
|
+
record["or_index"] = None
|
|
2383
|
+
record["or_as_of"] = None
|
|
2384
|
+
record["drift"] = None
|
|
2385
|
+
|
|
2386
|
+
quality_by_id[cand["id"]] = record
|
|
2387
|
+
source_by_id[cand["id"]] = record["source"]
|
|
2012
2388
|
return quality_by_id, source_by_id
|
|
2013
2389
|
|
|
2014
2390
|
|
|
@@ -2033,7 +2409,9 @@ def _per_1m(price_per_token: float) -> float:
|
|
|
2033
2409
|
return round(price_per_token * 1_000_000.0, 6)
|
|
2034
2410
|
|
|
2035
2411
|
|
|
2036
|
-
def build_schedules(
|
|
2412
|
+
def build_schedules(
|
|
2413
|
+
models, args, quality_by_id, aa_by_id, balanced_by_id, catalog_quality=None
|
|
2414
|
+
):
|
|
2037
2415
|
"""Catalog-wide off-peak index built from the RAW model list (spec 5.4).
|
|
2038
2416
|
|
|
2039
2417
|
Deliberately unfiltered by the candidate pool: sub-floor-context and
|
|
@@ -2041,6 +2419,12 @@ def build_schedules(models, args, quality_by_id, aa_by_id, balanced_by_id):
|
|
|
2041
2419
|
schedule models mostly sit below the 1M context floor). Router aliases
|
|
2042
2420
|
(~ids) and malformed ids are excluded; :batch variants are included as
|
|
2043
2421
|
distinct deals. Sorted by id.
|
|
2422
|
+
|
|
2423
|
+
Quality falls back through the same chain resolve_quality walks, minus
|
|
2424
|
+
its AA tier (the AA lookup only ever matched candidates): candidate
|
|
2425
|
+
records, then the hydrated catalog (unkeyed runs), then OpenRouter's
|
|
2426
|
+
benchmark trio. Every schedule row carries the winner's provenance
|
|
2427
|
+
(quality_source/quality_as_of) so the site's QUAL tooltip can render.
|
|
2044
2428
|
"""
|
|
2045
2429
|
entries = []
|
|
2046
2430
|
for model in models:
|
|
@@ -2065,10 +2449,38 @@ def build_schedules(models, args, quality_by_id, aa_by_id, balanced_by_id):
|
|
|
2065
2449
|
continue
|
|
2066
2450
|
peak = max(windows, key=lambda window: window["blended"])
|
|
2067
2451
|
offpeak = min(windows, key=lambda window: window["blended"])
|
|
2068
|
-
|
|
2452
|
+
# quality_by_id values are records {"index", "source", ...}; the
|
|
2453
|
+
# schedule row keeps the bare float index verbatim (never a dict)
|
|
2454
|
+
# and echoes the winning record's provenance for the QUAL tooltip.
|
|
2455
|
+
# Fallback chain mirrors resolve_quality's minus its AA tier:
|
|
2456
|
+
# candidate record, hydrated catalog record (unkeyed runs; every
|
|
2457
|
+
# hydrated record is relabeled with the run-level chain label
|
|
2458
|
+
# "catalog", exactly like resolve_quality's catalog tier), then
|
|
2459
|
+
# OpenRouter's benchmark trio (row label "openrouter").
|
|
2460
|
+
record = quality_by_id.get(model_id)
|
|
2461
|
+
if isinstance(record, dict):
|
|
2462
|
+
quality = record.get("index")
|
|
2463
|
+
quality_source = record.get("source")
|
|
2464
|
+
quality_as_of = record.get("as_of")
|
|
2465
|
+
else:
|
|
2466
|
+
quality = record
|
|
2467
|
+
quality_source = None
|
|
2468
|
+
quality_as_of = None
|
|
2469
|
+
if quality is None and catalog_quality is not None:
|
|
2470
|
+
catalog_record = catalog_quality.get(model_id)
|
|
2471
|
+
if catalog_record is not None:
|
|
2472
|
+
quality = catalog_record["index"]
|
|
2473
|
+
quality_source = "catalog"
|
|
2474
|
+
quality_as_of = catalog_record["as_of"]
|
|
2069
2475
|
if quality is None:
|
|
2070
2476
|
benchmark = aa_by_id.get(base_model_id(model_id)) or {}
|
|
2071
2477
|
quality = benchmark.get("intelligence_index")
|
|
2478
|
+
if quality is not None:
|
|
2479
|
+
quality_source = "openrouter"
|
|
2480
|
+
quality_as_of = benchmark.get("as_of")
|
|
2481
|
+
else:
|
|
2482
|
+
quality_source = None
|
|
2483
|
+
quality_as_of = None
|
|
2072
2484
|
name = model.get("name") or model_id
|
|
2073
2485
|
entries.append(
|
|
2074
2486
|
{
|
|
@@ -2076,6 +2488,10 @@ def build_schedules(models, args, quality_by_id, aa_by_id, balanced_by_id):
|
|
|
2076
2488
|
"name": PROVIDER_PREFIX_RE.sub("", name).strip() or name,
|
|
2077
2489
|
"context": coerce_int(model.get("context_length"), 0),
|
|
2078
2490
|
"quality": quality,
|
|
2491
|
+
# Provenance for the site's QUAL tooltip (null when the
|
|
2492
|
+
# quality came without any -- e.g. a bare-float test map).
|
|
2493
|
+
"quality_source": quality_source,
|
|
2494
|
+
"quality_as_of": quality_as_of,
|
|
2079
2495
|
"score": balanced_by_id.get(model_id),
|
|
2080
2496
|
"max_discount": eff["max_discount"],
|
|
2081
2497
|
"sched_note": fmt_sched_note(eff["max_discount"]),
|
|
@@ -2285,11 +2701,19 @@ def compute_scores(candidates, args, quality_by_id):
|
|
|
2285
2701
|
0.0, min(1.0, 1.0 - (math.log10(price + 0.01) - low) / price_span)
|
|
2286
2702
|
)
|
|
2287
2703
|
quality = quality_by_id.get(cand["id"])
|
|
2288
|
-
|
|
2289
|
-
|
|
2704
|
+
# quality_by_id values are records {"index", "source", ...}; the
|
|
2705
|
+
# candidate keeps the bare float index (that is what ranking_key,
|
|
2706
|
+
# build_catalog's quality field and print_json's quality_index all
|
|
2707
|
+
# expect) and the full record travels alongside for provenance.
|
|
2708
|
+
record = quality if isinstance(quality, dict) else None
|
|
2709
|
+
cand["quality"] = record["index"] if record else quality
|
|
2710
|
+
cand["quality_record"] = record
|
|
2711
|
+
if cand["quality"] is None:
|
|
2290
2712
|
cand["quality_score"] = 0.0
|
|
2291
2713
|
else:
|
|
2292
|
-
cand["quality_score"] = max(
|
|
2714
|
+
cand["quality_score"] = max(
|
|
2715
|
+
0.0, min(1.0, cand["quality"] / args.quality_ref)
|
|
2716
|
+
)
|
|
2293
2717
|
context = cand["context"]
|
|
2294
2718
|
cand["context_score"] = (
|
|
2295
2719
|
max(0.0, min(1.0, math.log(context / floor_ctx) / ctx_span))
|
|
@@ -2427,6 +2851,109 @@ def fmt_quant_note(annotations: dict) -> str | None:
|
|
|
2427
2851
|
return note
|
|
2428
2852
|
|
|
2429
2853
|
|
|
2854
|
+
def _footer_as_of(source, value):
|
|
2855
|
+
"""Format a record's as_of for its footer block (None when unusable).
|
|
2856
|
+
|
|
2857
|
+
Two stamp shapes circulate. A YYYYMMDD date (OpenRouter's benchmark
|
|
2858
|
+
date, and catalog rows hydrated from OR rows) renders as a plain date
|
|
2859
|
+
("as of 2026-10-17") for ANY source -- a date-only stamp never
|
|
2860
|
+
fabricates a time of day. ISO stamps: live AA records carry one fetch
|
|
2861
|
+
timestamp for the whole fetch (checked today), so their block shows
|
|
2862
|
+
the time of day only; other rows carry the full stamp ("as of
|
|
2863
|
+
2026-10-23 13:15Z").
|
|
2864
|
+
"""
|
|
2865
|
+
if not value:
|
|
2866
|
+
return None
|
|
2867
|
+
text = str(value)
|
|
2868
|
+
if len(text) == 8 and text.isdigit():
|
|
2869
|
+
return f"{text[:4]}-{text[4:6]}-{text[6:]}"
|
|
2870
|
+
parsed = parse_iso_datetime(value)
|
|
2871
|
+
if parsed is None:
|
|
2872
|
+
return None
|
|
2873
|
+
if source == "aa":
|
|
2874
|
+
return parsed.strftime("%H:%MZ")
|
|
2875
|
+
return parsed.strftime("%Y-%m-%d %H:%MZ")
|
|
2876
|
+
|
|
2877
|
+
|
|
2878
|
+
def _as_of_instant(value):
|
|
2879
|
+
"""Parse an as_of stamp to a comparable UTC instant (None when unusable).
|
|
2880
|
+
|
|
2881
|
+
The pipeline carries ISO stamps and the YYYYMMDD benchmark date; a
|
|
2882
|
+
date-only stamp means "that day" and compares as midnight UTC.
|
|
2883
|
+
"""
|
|
2884
|
+
text = str(value)
|
|
2885
|
+
if len(text) == 8 and text.isdigit():
|
|
2886
|
+
return parse_iso_datetime(f"{text[:4]}-{text[4:6]}-{text[6:]}")
|
|
2887
|
+
return parse_iso_datetime(value)
|
|
2888
|
+
|
|
2889
|
+
|
|
2890
|
+
def _later_as_of(value, current) -> bool:
|
|
2891
|
+
"""True when the value stamp names a later instant than current.
|
|
2892
|
+
|
|
2893
|
+
Parsed instants, never raw strings: a YYYYMMDD stamp must not
|
|
2894
|
+
lexicographically beat a fresher ISO stamp ("20261017" sorts above
|
|
2895
|
+
every "2026-..." string). Ties keep the earlier pick; an unparseable
|
|
2896
|
+
stamp never displaces anything.
|
|
2897
|
+
"""
|
|
2898
|
+
new, old = _as_of_instant(value), _as_of_instant(current)
|
|
2899
|
+
if new is None or old is None:
|
|
2900
|
+
return new is not None
|
|
2901
|
+
return new > old
|
|
2902
|
+
|
|
2903
|
+
|
|
2904
|
+
def quality_footer_note(quality_by_id, aa_index_version, unmatched):
|
|
2905
|
+
"""Compose the footer's quality note from the resolved records.
|
|
2906
|
+
|
|
2907
|
+
Lists per-source counts, each source's own as_of and the AA index
|
|
2908
|
+
version when known, e.g. "AA index v4.3 - live AA (58, checked 13:15Z)
|
|
2909
|
+
| our catalog (12, as of 2026-10-23 13:15Z) | OpenRouter snapshot (2,
|
|
2910
|
+
as of 2026-10-17)". A trailing "unmatched score 0" names candidates
|
|
2911
|
+
with no quality record, and a drift footnote appears when any resolved
|
|
2912
|
+
record disagrees with the OpenRouter check value. Returns None when no
|
|
2913
|
+
record resolved -- run() falls back to the quality-blind note.
|
|
2914
|
+
"""
|
|
2915
|
+
labels = {
|
|
2916
|
+
"aa": "live AA",
|
|
2917
|
+
"catalog": "our catalog",
|
|
2918
|
+
"openrouter": "OpenRouter snapshot",
|
|
2919
|
+
}
|
|
2920
|
+
stamp_words = {"aa": "checked", "catalog": "as of", "openrouter": "as of"}
|
|
2921
|
+
counts = {}
|
|
2922
|
+
latest = {}
|
|
2923
|
+
for record in quality_by_id.values():
|
|
2924
|
+
source = record.get("source")
|
|
2925
|
+
if source not in labels:
|
|
2926
|
+
continue
|
|
2927
|
+
counts[source] = counts.get(source, 0) + 1
|
|
2928
|
+
value = record.get("as_of")
|
|
2929
|
+
if value:
|
|
2930
|
+
current = latest.get(source)
|
|
2931
|
+
if current is None or _later_as_of(value, current):
|
|
2932
|
+
latest[source] = value
|
|
2933
|
+
blocks = []
|
|
2934
|
+
for source in ("aa", "catalog", "openrouter"):
|
|
2935
|
+
count = counts.get(source, 0)
|
|
2936
|
+
if not count:
|
|
2937
|
+
continue
|
|
2938
|
+
stamp = _footer_as_of(source, latest.get(source))
|
|
2939
|
+
if stamp:
|
|
2940
|
+
blocks.append(f"{labels[source]} ({count}, {stamp_words[source]} {stamp})")
|
|
2941
|
+
else:
|
|
2942
|
+
blocks.append(f"{labels[source]} ({count})")
|
|
2943
|
+
if not blocks:
|
|
2944
|
+
return None
|
|
2945
|
+
note = "quality: "
|
|
2946
|
+
if aa_index_version is not None:
|
|
2947
|
+
note += f"AA index v{aa_index_version} - "
|
|
2948
|
+
note += " | ".join(blocks)
|
|
2949
|
+
if unmatched:
|
|
2950
|
+
note += "; unmatched score 0"
|
|
2951
|
+
drift = sum(1 for record in quality_by_id.values() if record.get("drift"))
|
|
2952
|
+
if drift:
|
|
2953
|
+
note += f"; drift vs OpenRouter snapshot ({drift})"
|
|
2954
|
+
return note
|
|
2955
|
+
|
|
2956
|
+
|
|
2430
2957
|
def print_table(top, total_candidates, weights, quality_note):
|
|
2431
2958
|
headers = [
|
|
2432
2959
|
"RANK",
|
|
@@ -2514,6 +3041,15 @@ def print_json(top):
|
|
|
2514
3041
|
"name": cand["name"],
|
|
2515
3042
|
"score": round(cand["score"], 4),
|
|
2516
3043
|
"quality_index": cand["quality"],
|
|
3044
|
+
# Provenance block, verbatim from the quality record resolved
|
|
3045
|
+
# in compute_scores: the winning tier's source and as_of plus
|
|
3046
|
+
# the drift check against OpenRouter's benchmarks. Rows without
|
|
3047
|
+
# a record (pre-endpoint test paths) degrade to None.
|
|
3048
|
+
"quality_source": (cand.get("quality_record") or {}).get("source"),
|
|
3049
|
+
"quality_as_of": (cand.get("quality_record") or {}).get("as_of"),
|
|
3050
|
+
"quality_drift": (cand.get("quality_record") or {}).get("drift"),
|
|
3051
|
+
"quality_or_index": (cand.get("quality_record") or {}).get("or_index"),
|
|
3052
|
+
"quality_or_as_of": (cand.get("quality_record") or {}).get("or_as_of"),
|
|
2517
3053
|
"input_usd_per_m": round(cand["price_in"], 6),
|
|
2518
3054
|
"output_usd_per_m": round(cand["price_out"], 6),
|
|
2519
3055
|
"blended_usd_per_m": round(cand["blended"], 6),
|
|
@@ -2584,6 +3120,9 @@ def build_catalog(
|
|
|
2584
3120
|
quality_source_by_id,
|
|
2585
3121
|
session_cost=None,
|
|
2586
3122
|
session_cost_mode="skipped",
|
|
3123
|
+
index_version=None,
|
|
3124
|
+
catalog_generated_at=None,
|
|
3125
|
+
catalog_quality=None,
|
|
2587
3126
|
):
|
|
2588
3127
|
"""Assemble the full-catalog document (see README "Catalog output").
|
|
2589
3128
|
|
|
@@ -2595,22 +3134,52 @@ def build_catalog(
|
|
|
2595
3134
|
session_cost is the Task 7 fetch envelope (verbatim, medians included)
|
|
2596
3135
|
or None; session_cost_mode is run()'s resolution of the key-presence
|
|
2597
3136
|
fact ("ok" / "unavailable" / "skipped") -- build_catalog stays pure and
|
|
2598
|
-
never fetches or inspects the environment itself.
|
|
3137
|
+
never fetches or inspects the environment itself. index_version is the
|
|
3138
|
+
AA envelope's intelligence_index_version (number or null) and
|
|
3139
|
+
catalog_generated_at the published catalog's generated_at when this
|
|
3140
|
+
run hydrated from it (else None); both pass verbatim into sources.aa.
|
|
3141
|
+
catalog_quality is the hydrated record map (or None): candidates read
|
|
3142
|
+
it through resolve_quality, and build_schedules falls back to it for
|
|
3143
|
+
non-candidates.
|
|
2599
3144
|
"""
|
|
2600
3145
|
now = datetime.now(timezone.utc)
|
|
2601
3146
|
weights = catalog_weights(candidates, quality_by_id)
|
|
2602
|
-
|
|
2603
|
-
|
|
2604
|
-
#
|
|
2605
|
-
# would contradict itself (mode none with matched quality scores).
|
|
3147
|
+
if aa_source is not None and aa_source != "AA API v2":
|
|
3148
|
+
# Never publish a document whose AA source we cannot name: the
|
|
3149
|
+
# consumers' mode enum has no slot for it.
|
|
2606
3150
|
raise ValueError(f"unknown AA source: {aa_source!r}")
|
|
3151
|
+
# The three counts are disjoint label buckets: resolve_quality's strict
|
|
3152
|
+
# rank gives every candidate at most one record, so their sum is exactly
|
|
3153
|
+
# the number of entries with a non-null quality_match -- the invariant
|
|
3154
|
+
# the site validator enforces. A label outside the buckets would break
|
|
3155
|
+
# it silently, so it fails loudly here.
|
|
3156
|
+
matched_aa = sum(1 for s in quality_source_by_id.values() if s == "aa")
|
|
3157
|
+
matched_catalog = sum(1 for s in quality_source_by_id.values() if s == "catalog")
|
|
2607
3158
|
matched_openrouter = sum(
|
|
2608
3159
|
1 for s in quality_source_by_id.values() if s == "openrouter"
|
|
2609
3160
|
)
|
|
2610
|
-
|
|
2611
|
-
|
|
2612
|
-
|
|
2613
|
-
|
|
3161
|
+
unknown = sorted(
|
|
3162
|
+
{
|
|
3163
|
+
s
|
|
3164
|
+
for s in quality_source_by_id.values()
|
|
3165
|
+
if s not in ("aa", "catalog", "openrouter")
|
|
3166
|
+
}
|
|
3167
|
+
)
|
|
3168
|
+
if unknown:
|
|
3169
|
+
raise ValueError(f"unknown quality source: {unknown[0]!r}")
|
|
3170
|
+
# mode is the run's quality chain, not a match outcome: "aa" when the
|
|
3171
|
+
# AA API returned usable entries, "catalog" when unkeyed runs hydrated
|
|
3172
|
+
# the published catalog, "none" when neither tier produced data.
|
|
3173
|
+
# OpenRouter's benchmarks are the chain's shared last resort, never a
|
|
3174
|
+
# mode of their own -- matched_openrouter carries their count. The
|
|
3175
|
+
# catalog branch also fires on the counts alone: hydrated records are
|
|
3176
|
+
# proof of a usable catalog even when its generated_at was unusable.
|
|
3177
|
+
if aa_source is not None:
|
|
3178
|
+
aa_mode = "aa"
|
|
3179
|
+
elif catalog_generated_at is not None or matched_catalog or matched_aa:
|
|
3180
|
+
aa_mode = "catalog"
|
|
3181
|
+
else:
|
|
3182
|
+
aa_mode = "none"
|
|
2614
3183
|
|
|
2615
3184
|
entries = []
|
|
2616
3185
|
for cand in candidates:
|
|
@@ -2641,6 +3210,12 @@ def build_catalog(
|
|
|
2641
3210
|
4,
|
|
2642
3211
|
)
|
|
2643
3212
|
bench = aa_by_id.get(base_model_id(cand["id"])) or {}
|
|
3213
|
+
# The quality record map is already a parameter; the winning
|
|
3214
|
+
# record's as_of and drift verdict become entry fields verbatim.
|
|
3215
|
+
# Unmatched candidates degrade to null -- same shape as the v3
|
|
3216
|
+
# additive blocks below.
|
|
3217
|
+
record = quality_by_id.get(cand["id"])
|
|
3218
|
+
record = record if isinstance(record, dict) else None
|
|
2644
3219
|
entries.append(
|
|
2645
3220
|
{
|
|
2646
3221
|
"id": cand["id"],
|
|
@@ -2708,6 +3283,11 @@ def build_catalog(
|
|
|
2708
3283
|
},
|
|
2709
3284
|
"quality": cand["quality"],
|
|
2710
3285
|
"quality_match": quality_source_by_id.get(cand["id"]),
|
|
3286
|
+
# Provenance (schema v4), verbatim from the record: the
|
|
3287
|
+
# winning source's own as_of and the drift verdict against
|
|
3288
|
+
# OpenRouter's check value.
|
|
3289
|
+
"quality_as_of": record["as_of"] if record else None,
|
|
3290
|
+
"quality_drift": record["drift"] if record else None,
|
|
2711
3291
|
"scores": {**scores, "overall": overall},
|
|
2712
3292
|
# Additive provider-aware blocks (schema v3), verbatim from
|
|
2713
3293
|
# the candidate. Candidates that never went through the
|
|
@@ -2732,7 +3312,7 @@ def build_catalog(
|
|
|
2732
3312
|
|
|
2733
3313
|
return {
|
|
2734
3314
|
"schedules": build_schedules(
|
|
2735
|
-
models, args, quality_by_id, aa_by_id, balanced_by_id
|
|
3315
|
+
models, args, quality_by_id, aa_by_id, balanced_by_id, catalog_quality
|
|
2736
3316
|
),
|
|
2737
3317
|
"schema_version": CATALOG_SCHEMA_VERSION,
|
|
2738
3318
|
"tool": "model-compare",
|
|
@@ -2753,9 +3333,15 @@ def build_catalog(
|
|
|
2753
3333
|
"openrouter": "ok",
|
|
2754
3334
|
"aa": {
|
|
2755
3335
|
"mode": aa_mode,
|
|
2756
|
-
"
|
|
2757
|
-
"
|
|
3336
|
+
"matched_aa": matched_aa,
|
|
3337
|
+
"matched_catalog": matched_catalog,
|
|
2758
3338
|
"matched_openrouter": matched_openrouter,
|
|
3339
|
+
# Provenance meta (schema v4): the published catalog's stamp
|
|
3340
|
+
# when this run hydrated from it, and the AA envelope's
|
|
3341
|
+
# intelligence index version when the AA API ran. Both null
|
|
3342
|
+
# when their source was not consulted.
|
|
3343
|
+
"catalog_generated_at": catalog_generated_at,
|
|
3344
|
+
"intelligence_index_version": index_version,
|
|
2759
3345
|
},
|
|
2760
3346
|
"zdr": "skipped" if args.no_zdr else "ok",
|
|
2761
3347
|
"discounts": "ok" if discounts else "unavailable",
|
|
@@ -2941,7 +3527,30 @@ def run(args, models, or_cached) -> int:
|
|
|
2941
3527
|
"(--no-zdr to override)"
|
|
2942
3528
|
)
|
|
2943
3529
|
return 1
|
|
2944
|
-
|
|
3530
|
+
# Mode-scoped quality chains: a keyed run consults the AA API only; an
|
|
3531
|
+
# unkeyed run hydrates from the published catalog only. The catalog
|
|
3532
|
+
# fetcher must NEVER run on keyed builds -- a CI/site build hydrating
|
|
3533
|
+
# from the published catalog would consume its own output (a
|
|
3534
|
+
# self-reference loop). Both chains end in OpenRouter's benchmarks as
|
|
3535
|
+
# the last resort, then nothing.
|
|
3536
|
+
if args.aa_api_key or os.environ.get("AA_API_KEY"):
|
|
3537
|
+
aa_entries, aa_source, aa_cached, aa_index_version = fetch_aa_entries(args)
|
|
3538
|
+
catalog_quality = None
|
|
3539
|
+
catalog_generated_at = None
|
|
3540
|
+
else:
|
|
3541
|
+
aa_entries, aa_source, aa_cached, aa_index_version = [], None, False, None
|
|
3542
|
+
# None (transport failure) and an empty by_id (fetched-but-unusable
|
|
3543
|
+
# artifact) both degrade to no catalog tier -- resolve_quality and
|
|
3544
|
+
# the footer treat them identically as "catalog unavailable".
|
|
3545
|
+
catalog_doc = fetch_quality_catalog(args)
|
|
3546
|
+
catalog_quality = (catalog_doc or {}).get("by_id") or None
|
|
3547
|
+
# The catalog meta only means something when hydration actually
|
|
3548
|
+
# produced rows: an unusable artifact contributes no meta, so the
|
|
3549
|
+
# sources block and the mode never credit a catalog that yielded
|
|
3550
|
+
# nothing.
|
|
3551
|
+
catalog_generated_at = (
|
|
3552
|
+
(catalog_doc or {}).get("generated_at") if catalog_quality else None
|
|
3553
|
+
)
|
|
2945
3554
|
exact, fuzzy = build_aa_lookup(aa_entries)
|
|
2946
3555
|
|
|
2947
3556
|
filtered = []
|
|
@@ -2967,7 +3576,7 @@ def run(args, models, or_cached) -> int:
|
|
|
2967
3576
|
return 2
|
|
2968
3577
|
|
|
2969
3578
|
quality_by_id, quality_source_by_id = resolve_quality(
|
|
2970
|
-
candidates, aa_by_id, exact, fuzzy, aa_source
|
|
3579
|
+
candidates, aa_by_id, exact, fuzzy, aa_source, catalog_quality
|
|
2971
3580
|
)
|
|
2972
3581
|
weights = compute_scores(candidates, args, quality_by_id)
|
|
2973
3582
|
|
|
@@ -3009,6 +3618,9 @@ def run(args, models, or_cached) -> int:
|
|
|
3009
3618
|
quality_source_by_id,
|
|
3010
3619
|
session_cost,
|
|
3011
3620
|
session_cost_mode,
|
|
3621
|
+
aa_index_version,
|
|
3622
|
+
catalog_generated_at,
|
|
3623
|
+
catalog_quality,
|
|
3012
3624
|
)
|
|
3013
3625
|
)
|
|
3014
3626
|
return 0
|
|
@@ -3027,23 +3639,14 @@ def run(args, models, or_cached) -> int:
|
|
|
3027
3639
|
f"{v} {k}" for k, v in sorted(dropped.items(), key=lambda kv: -kv[1])
|
|
3028
3640
|
)
|
|
3029
3641
|
drop_note = f" (dropped: {bits})"
|
|
3030
|
-
|
|
3031
|
-
|
|
3642
|
+
# The footer lists per-source counts, each source's own as_of and
|
|
3643
|
+
# the AA index version when known; unmatched candidates and drift
|
|
3644
|
+
# get trailing footnotes. No record resolved -> the quality-blind
|
|
3645
|
+
# fallback note.
|
|
3032
3646
|
unmatched = len(candidates) - len(quality_by_id)
|
|
3033
|
-
|
|
3034
|
-
|
|
3035
|
-
|
|
3036
|
-
if matched_aa:
|
|
3037
|
-
bits.append(f"{aa_source} exact ({matched_aa})")
|
|
3038
|
-
if bits:
|
|
3039
|
-
quality_note = (
|
|
3040
|
-
f"quality via {' + '.join(bits)}: matched "
|
|
3041
|
-
f"{len(quality_by_id)}/{len(candidates)} candidates"
|
|
3042
|
-
)
|
|
3043
|
-
if unmatched:
|
|
3044
|
-
quality_note += "; unmatched candidates score 0 on quality"
|
|
3045
|
-
else:
|
|
3046
|
-
quality_note = "quality data unavailable, ranked on price/context/age"
|
|
3647
|
+
quality_note = quality_footer_note(
|
|
3648
|
+
quality_by_id, aa_index_version, unmatched
|
|
3649
|
+
) or ("quality data unavailable, ranked on price/context/age")
|
|
3047
3650
|
source_note = []
|
|
3048
3651
|
if or_cached:
|
|
3049
3652
|
source_note.append("catalog cached")
|