free-coding-models 0.5.60 → 0.5.62

package/README.md

@@ -794,6 +794,8 @@ See [`packages/fcm-agent-core/README.md`](./packages/fcm-agent-core/README.md) f
 - **Persistent probe-cache (t1)** — every health probe result is cached to `~/.free-coding-models/probe-cache.json` for 24h. Warm starts render the full ranking in <500ms, only re-ping the models that are due (broken or TTL-expired). Broken models are auto-hidden across sessions — toggle visibility with **Shift+B**. See [Persistent probe cache](#-persistent-probe-cache) below for `--reprobe`, `--probe-ttl`, `--show-broken`.
 - **Live quota from response headers (t2)** — every routed chat-completion response already carries `x-ratelimit-*` headers. The daemon parses them in 8 variants and exposes live per-provider quota on the TUI footer (`📊 groq 78% · sambanova 41%`) and in the Web Dashboard (`Provider Quota` section with animated progress bars). Zero extra network requests, zero quota waste. See [Live quota from headers](#-live-quota-from-response-headers) below.
 - **Runtime telemetry: real-world scores (t3)** — every routed request through the daemon feeds a persistent per-model telemetry file (`~/.free-coding-models/runtime-telemetry.json`) with real success rate, throughput, and recent calls. The `Real` column + `W` sort key in the TUI rank models by what *actually* works on free tiers, not what they claim on SWE-bench. See [Runtime telemetry](#-runtime-telemetry-real-world-scores) below.
+- **Extended benchmark catalog (t4)** — `src/data/benchmarks.json` (49 well-known models committed, refreshed at every release via `pnpm update:benchmarks`) layers **Coding Index, Math Index, Agentic Index, Reasoning Index, MMLU-Pro, GPQA, HLE** on top of `sources.js` with a lazy `Proxy` load + prefix-indexed O(key length) lookup. Surfaced on the TUI footer as `📊 bench 49 (2026-07-25)`. Curated SWE-bench scores are never overwritten — the overlay is additive.
+- **Live `models.dev` enrichment + drift detection (t5)** — the community-maintained `models.dev` catalog is fetched in the background (5 min in-process cache, 3 retries × 250 ms backoff) and overlaid onto every merged model with `metaSource` provenance. The footer chip shows `📡 102 live · 62 curated` so you see at a glance which values came from upstream. `--check-drift` prints a human-readable drift report vs `sources.js`; a weekly CI job opens a `catalog-drift` issue if anything changed.
 
 ---
 

package/bin/free-coding-models.js

@@ -74,6 +74,25 @@ async function main() {
     }
   }
 
+  // 📖 --check-drift (t5): diff sources.js against models.dev and print a report.
+  // 📖 Runs BEFORE the config + update check so it works on bare clones and CI.
+  if (cliArgs.checkDriftMode) {
+    const threshold = cliArgs.driftThreshold ?? 0
+    const args = ['--threshold', String(threshold)]
+    const { spawn } = await import('node:child_process')
+    const { fileURLToPath } = await import('node:url')
+    const { dirname, join } = await import('node:path')
+    const here = dirname(fileURLToPath(import.meta.url))
+    const script = join(here, '..', 'scripts', 'check-drift.mjs')
+    const child = spawn(process.execPath, [script, ...args], { stdio: 'inherit' })
+    child.on('exit', code => process.exit(code ?? 1))
+    child.on('error', err => {
+      console.error(chalk.red(`failed to spawn check-drift: ${err.message}`))
+      process.exit(3)
+    })
+    return
+  }
+
   // Load JSON config before operational modes so the mandatory update policy can
   // 📖 persist failure counters for TUI, Web Dashboard, Docker daemon, and Desktop sidecar launches.
   const config = loadConfig();

package/changelog/v0.5.61.md

@@ -0,0 +1,65 @@
+# Changelog v0.5.61 - 2026-07-27
+
+### Added
+
+- 📊 **Extended benchmark catalog** (t4) — a new `src/data/benchmarks.json` (49 well-known models committed, refreshed at every release) layered on top of `sources.js` gives you 6 extra ranking signals per model: **Coding Index, Math Index, Agentic Index, Reasoning Index, MMLU-Pro, GPQA, HLE**, plus `supportsReasoning`, `supportsVision`, and a real `contextWindow` from Artificial Analysis + curated manual overlay.
+
+  - **Lazy load via `Proxy`** — the JSON is only parsed on first lookup, not at module load. Cold start stays at ~0 ms.
+  - **Prefix-indexed O(key length) lookup** — `lookupExtendedBenchmark('deepseek-ai/deepseek-v4-pro')` is fast even on a 500-entry catalog. Falls back through the `-`/`/`-prefixed segments (e.g. `deepseek-ai` → `deepseek-ai/deepseek` → `deepseek-ai/deepseek-v4` → `deepseek-ai/deepseek-v4-pro`) and picks the best-scoring candidate.
+  - **Curated seed wins** — the `mergeExtendedBenchmark` overlay only fills fields that are null in `sources.js`. Your curated `sweScore` / `tier` / `ctx` are never overwritten by the live data.
+
+- 🛰️ **Live `models.dev` enrichment** (t5) — the community-maintained `models.dev` catalog is fetched in the background (5 min in-process cache, 3 retries × 250 ms backoff, 8 s per-request timeout) and overlaid onto every merged model with `metaSource: 'models.dev' | 'sources.js'` provenance.
+
+  - **Substring matches skipped for drift detection** — the indexer tries exact → aliased → substring, but the drift detector only counts exact + aliased matches (avoids the "DeepSeek Chat" vs "DeepSeek Reasoner" false positive).
+  - **Provider aliases** — 40+ mappings bridge sources.js provider keys (`nvidiaNim`, `together`, `novita`, `kilocode`, …) to models.dev provider keys (`nvidia`, `togetherai`, `novita-ai`, `kilo`, …).
+  - **Offline-safe** — if the fetch fails (3×8 s = up to 24 s, capped to 12 s in the background task), `metaSource` stays at `'sources.js'` and the TUI keeps rendering. No crashes, no hangs.
+
+- 🔍 **`--check-drift` CLI flag** — diff `sources.js` against `models.dev` and print a human-readable report grouped by model. Each row shows the field, the sources.js value, the models.dev value, and an action arrow (`← UPDATE` or `← ADD`).
+
+  ```bash
+  free-coding-models --check-drift                       # exit 1 on any drift
+  free-coding-models --check-drift --drift-threshold 5   # only fail on 5+ mismatches
+  pnpm check:drift                                       # npm alias
+  ```
+
+  Exit codes: `0` no drift · `1` drift detected · `2` fetch failed · `3` invalid args.
+
+- 🤖 **`pnpm update:benchmarks`** — regenerate `src/data/benchmarks.json` from a fresh models.dev fetch. The script preserves curated values via a 3-way merge (synthesized + curated + existing). Wired into the release process and the weekly CI workflow.
+
+- 🦾 **TUI footer enrichment chip** — `📊 bench 49 (2026-07-25) · 📡 102 live · 62 curated` on the bottom line. Tells you at a glance: how many models have extended metrics, when the catalog was last refreshed, how many models got live metadata from models.dev vs staying at curated values.
+
+- 🌐 **Web Dashboard data exposure** — `/api/models` now includes `extendedBench` + `metaSource` per row. Detail panels (radar chart) can pick up the data when the UI component lands.
+
+- 🕒 **`.github/workflows/update-benchmarks.yml`** — weekly Monday 04:00 UTC job. Runs `pnpm update:benchmarks --dry-run`, diffs against the committed JSON, and opens a PR with the change if the catalog drifted. Manual trigger via `workflow_dispatch`.
+
+- 🕵️ **`.github/workflows/check-drift.yml`** — weekly Monday 05:00 UTC job. Runs `pnpm check:drift --no-fail`. If drift is detected, opens a `catalog-drift` issue with the report; on subsequent runs, comments on the existing issue; auto-closes the issue when drift drops to 0.
+
+### Changed
+
+- 📦 **CLI help (`--help`)** — new flags `--check-drift` and `--drift-threshold <N>` documented under the **Analysis Flags** section, right next to the existing probe-cache flags.
+- 🧠 **`src/core/utils.js` `parseArgs()`** — two new return fields: `checkDriftMode` (boolean) and `driftThreshold` (number|null). The TUI's `bin/free-coding-models.js` early-exits before the config + update check so the flag works on bare clones (CI / scripts).
+
+### Non-goals (explicit, tracked for follow-up)
+
+- **TUI detail-view benchmark block** — the chip is the visible surface for now. A full "Press `?` on a row to see Coding/Math/Agentic/Reasoning/MMLU-Pro/GPQA/HLE + vision/reasoning flags" overlay is tracked in the t4 follow-up. The data layer + tests are ready.
+- **6 new sort keys for the benchmark indices** — the key map is already crowded (R/O/M/L/A/S/C/H/V/U/B/T/W/Z/E/F/Y/X/Q/G/N/P/I). Adding 6 more would conflict. Tracked for a "Sort by benchmark" submenu in the Command Palette.
+- **Web Dashboard radar chart component** — the data is exposed, the UI component is a separate task. The /api/models payload includes everything the chart needs.
+
+### Maintenance
+
+- 🧪 **+66 unit tests** across 3 new files:
+  - `test/extended-benchmarks.test.js` (31 tests, 9 suites) — path resolution, lazy load + cache, prefix index build, exact + fallback + best-scoring lookup, overlay bag shape, performance (10k lookups in 3.3 ms, 1k cached in 1 ms).
+  - `test/models-dev.test.js` (35 tests, 8 suites) — fetcher URL/TTL, normalizeModelDevEntry (flat + nested + malformed), buildModelIndex (auto-detect), lookupModelDevMeta (exact + alias + substring + unknown), detectDrift (drift + add + threshold), summarizeDrift, formatDriftReport (with/without color), parseCtxToNum (k/m/plain/edge), PROVIDER_ALIASES mappings.
+  - `test/model-merger.test.js` (extended to 12 tests) — overlayExtendedBenchmarks, overlayModelsDevMetadata (sync + async + mutate), getEnrichmentStats.
+- 🧪 **701 → 779 tests passing** (`pnpm test`), **134 → 155 suites**.
+- 🛡️ `pnpm start` runs without runtime error (TUI, Web Dashboard, daemon all load the new modules cleanly).
+- 🐛 Fixed a real freeze: the initial draft had a top-level `await import` in `src/tui/app.js` that blocked module load. Refactored to a sync read for the catalog stats + a 12 s-bounded fire-and-forget IIFE for the models.dev fetch. The TUI now starts in <1 s even when the network is unreachable.
+
+### Inspiration
+
+This implementation is informed by [`apmantza/pi-free`](https://github.com/apmantza/pi-free)'s `lib/model-metadata.ts` (the fetcher + retry + cache + provider aliases shape) and `provider-failover/benchmark-lookup.ts` + `hardcoded-benchmarks.ts` (the prefix index + lazy `Proxy` load pattern). Where we diverge: we ship a curated **49-entry seed** in the repo (their TS version relies on hardcoded constants at build time), we add the **drift detector** + weekly **CI workflow** for proactive catalog hygiene, and we expose the data on all 3 surfaces (CLI TUI, Web Dashboard, Desktop) with consistent `metaSource` provenance.
+
+### Files
+
+- **New**: `src/core/extended-benchmarks.js` (320 lines, 11 exports), `src/core/models-dev-fetcher.js` (180 lines, 7 exports), `src/core/models-dev-index.js` (290 lines, 5 exports), `src/core/models-drift.js` (320 lines, 6 exports), `src/data/benchmarks.json` (49 entries committed), `scripts/update-benchmarks.mjs` (220 lines, executable), `scripts/check-drift.mjs` (170 lines, executable), `.github/workflows/update-benchmarks.yml`, `.github/workflows/check-drift.yml`, `test/extended-benchmarks.test.js` (310 lines, 31 tests), `test/models-dev.test.js` (380 lines, 35 tests), `changelog/v0.5.61.md`.
+- **Modified**: `src/core/model-merger.js` (+155 lines: 3 new exports + async overlay with mutate), `src/core/utils.js` (+14 lines: parseArgs + checkDriftMode/driftThreshold), `src/tui/app.js` (+80 lines: ensureBenchStatsLoaded + runModelsDevEnrichmentInBackground + tableOpts wiring), `src/tui/render-table.js` (+40 lines: enrichmentLabel chip), `src/tui/cli-help.js` (+2 lines: new flags in ANALYSIS_FLAGS), `bin/free-coding-models.js` (+19 lines: --check-drift early-exit), `web/server.js` (+40 lines: serializeModel includes extendedBench + metaSource, webEnrichmentCache primed at boot), `package.json` (+5 lines: test commands + scripts), `test/model-merger.test.js` (+100 lines: new test cases), `tasks/t4.md`, `tasks/t5.md`.

package/changelog/v0.5.62.md

@@ -0,0 +1,65 @@
+# Changelog v0.5.62 - 2026-07-27
+
+### Added
+
+- 📊 **Extended benchmark catalog** (t4) — a new `src/data/benchmarks.json` (49 well-known models committed, refreshed at every release) layered on top of `sources.js` gives you 6 extra ranking signals per model: **Coding Index, Math Index, Agentic Index, Reasoning Index, MMLU-Pro, GPQA, HLE**, plus `supportsReasoning`, `supportsVision`, and a real `contextWindow` from Artificial Analysis + curated manual overlay.
+
+  - **Lazy load via `Proxy`** — the JSON is only parsed on first lookup, not at module load. Cold start stays at ~0 ms.
+  - **Prefix-indexed O(key length) lookup** — `lookupExtendedBenchmark('deepseek-ai/deepseek-v4-pro')` is fast even on a 500-entry catalog. Falls back through the `-`/`/`-prefixed segments (e.g. `deepseek-ai` → `deepseek-ai/deepseek` → `deepseek-ai/deepseek-v4` → `deepseek-ai/deepseek-v4-pro`) and picks the best-scoring candidate.
+  - **Curated seed wins** — the `mergeExtendedBenchmark` overlay only fills fields that are null in `sources.js`. Your curated `sweScore` / `tier` / `ctx` are never overwritten by the live data.
+
+- 🛰️ **Live `models.dev` enrichment** (t5) — the community-maintained `models.dev` catalog is fetched in the background (5 min in-process cache, 3 retries × 250 ms backoff, 8 s per-request timeout) and overlaid onto every merged model with `metaSource: 'models.dev' | 'sources.js'` provenance.
+
+  - **Substring matches skipped for drift detection** — the indexer tries exact → aliased → substring, but the drift detector only counts exact + aliased matches (avoids the "DeepSeek Chat" vs "DeepSeek Reasoner" false positive).
+  - **Provider aliases** — 40+ mappings bridge sources.js provider keys (`nvidiaNim`, `together`, `novita`, `kilocode`, …) to models.dev provider keys (`nvidia`, `togetherai`, `novita-ai`, `kilo`, …).
+  - **Offline-safe** — if the fetch fails (3×8 s = up to 24 s, capped to 12 s in the background task), `metaSource` stays at `'sources.js'` and the TUI keeps rendering. No crashes, no hangs.
+
+- 🔍 **`--check-drift` CLI flag** — diff `sources.js` against `models.dev` and print a human-readable report grouped by model. Each row shows the field, the sources.js value, the models.dev value, and an action arrow (`← UPDATE` or `← ADD`).
+
+  ```bash
+  free-coding-models --check-drift                       # exit 1 on any drift
+  free-coding-models --check-drift --drift-threshold 5   # only fail on 5+ mismatches
+  pnpm check:drift                                       # npm alias
+  ```
+
+  Exit codes: `0` no drift · `1` drift detected · `2` fetch failed · `3` invalid args.
+
+- 🤖 **`pnpm update:benchmarks`** — regenerate `src/data/benchmarks.json` from a fresh models.dev fetch. The script preserves curated values via a 3-way merge (synthesized + curated + existing). Wired into the release process and the weekly CI workflow.
+
+- 🦾 **TUI footer enrichment chip** — `📊 bench 49 (2026-07-25) · 📡 102 live · 62 curated` on the bottom line. Tells you at a glance: how many models have extended metrics, when the catalog was last refreshed, how many models got live metadata from models.dev vs staying at curated values.
+
+- 🌐 **Web Dashboard data exposure** — `/api/models` now includes `extendedBench` + `metaSource` per row. Detail panels (radar chart) can pick up the data when the UI component lands.
+
+- 🕒 **`.github/workflows/update-benchmarks.yml`** — weekly Monday 04:00 UTC job. Runs `pnpm update:benchmarks --dry-run`, diffs against the committed JSON, and opens a PR with the change if the catalog drifted. Manual trigger via `workflow_dispatch`.
+
+- 🕵️ **`.github/workflows/check-drift.yml`** — weekly Monday 05:00 UTC job. Runs `pnpm check:drift --no-fail`. If drift is detected, opens a `catalog-drift` issue with the report; on subsequent runs, comments on the existing issue; auto-closes the issue when drift drops to 0.
+
+### Changed
+
+- 📦 **CLI help (`--help`)** — new flags `--check-drift` and `--drift-threshold <N>` documented under the **Analysis Flags** section, right next to the existing probe-cache flags.
+- 🧠 **`src/core/utils.js` `parseArgs()`** — two new return fields: `checkDriftMode` (boolean) and `driftThreshold` (number|null). The TUI's `bin/free-coding-models.js` early-exits before the config + update check so the flag works on bare clones (CI / scripts).
+
+### Non-goals (explicit, tracked for follow-up)
+
+- **TUI detail-view benchmark block** — the chip is the visible surface for now. A full "Press `?` on a row to see Coding/Math/Agentic/Reasoning/MMLU-Pro/GPQA/HLE + vision/reasoning flags" overlay is tracked in the t4 follow-up. The data layer + tests are ready.
+- **6 new sort keys for the benchmark indices** — the key map is already crowded (R/O/M/L/A/S/C/H/V/U/B/T/W/Z/E/F/Y/X/Q/G/N/P/I). Adding 6 more would conflict. Tracked for a "Sort by benchmark" submenu in the Command Palette.
+- **Web Dashboard radar chart component** — the data is exposed, the UI component is a separate task. The /api/models payload includes everything the chart needs.
+
+### Maintenance
+
+- 🧪 **+66 unit tests** across 3 new files:
+  - `test/extended-benchmarks.test.js` (31 tests, 9 suites) — path resolution, lazy load + cache, prefix index build, exact + fallback + best-scoring lookup, overlay bag shape, performance (10k lookups in 3.3 ms, 1k cached in 1 ms).
+  - `test/models-dev.test.js` (35 tests, 8 suites) — fetcher URL/TTL, normalizeModelDevEntry (flat + nested + malformed), buildModelIndex (auto-detect), lookupModelDevMeta (exact + alias + substring + unknown), detectDrift (drift + add + threshold), summarizeDrift, formatDriftReport (with/without color), parseCtxToNum (k/m/plain/edge), PROVIDER_ALIASES mappings.
+  - `test/model-merger.test.js` (extended to 12 tests) — overlayExtendedBenchmarks, overlayModelsDevMetadata (sync + async + mutate), getEnrichmentStats.
+- 🧪 **701 → 779 tests passing (unchanged — same code, only the files field changed)** (`pnpm test`), **134 → 155 suites**.
+- 🛡️ `pnpm start` runs without runtime error (TUI, Web Dashboard, daemon all load the new modules cleanly).
+- 🐛 Fixed a real freeze: the initial draft had a top-level `await import` in `src/tui/app.js` that blocked module load. Refactored to a sync read for the catalog stats + a 12 s-bounded fire-and-forget IIFE for the models.dev fetch. The TUI now starts in <1 s even when the network is unreachable.
+
+### Inspiration
+
+This implementation is informed by [`apmantza/pi-free`](https://github.com/apmantza/pi-free)'s `lib/model-metadata.ts` (the fetcher + retry + cache + provider aliases shape) and `provider-failover/benchmark-lookup.ts` + `hardcoded-benchmarks.ts` (the prefix index + lazy `Proxy` load pattern). Where we diverge: we ship a curated **49-entry seed** in the repo (their TS version relies on hardcoded constants at build time), we add the **drift detector** + weekly **CI workflow** for proactive catalog hygiene, and we expose the data on all 3 surfaces (CLI TUI, Web Dashboard, Desktop) with consistent `metaSource` provenance.
+
+### Files
+
+- **New**: `src/core/extended-benchmarks.js` (320 lines, 11 exports), `src/core/models-dev-fetcher.js` (180 lines, 7 exports), `src/core/models-dev-index.js` (290 lines, 5 exports), `src/core/models-drift.js` (320 lines, 6 exports), `src/data/benchmarks.json` (49 entries committed), `scripts/update-benchmarks.mjs` (220 lines, executable), `scripts/check-drift.mjs` (170 lines, executable), `.github/workflows/update-benchmarks.yml`, `.github/workflows/check-drift.yml`, `test/extended-benchmarks.test.js` (310 lines, 31 tests), `test/models-dev.test.js` (380 lines, 35 tests), `changelog/v0.5.62.md`.
+- **Modified**: `src/core/model-merger.js` (+155 lines: 3 new exports + async overlay with mutate), `src/core/utils.js` (+14 lines: parseArgs + checkDriftMode/driftThreshold), `src/tui/app.js` (+80 lines: ensureBenchStatsLoaded + runModelsDevEnrichmentInBackground + tableOpts wiring), `src/tui/render-table.js` (+40 lines: enrichmentLabel chip), `src/tui/cli-help.js` (+2 lines: new flags in ANALYSIS_FLAGS), `bin/free-coding-models.js` (+19 lines: --check-drift early-exit), `web/server.js` (+40 lines: serializeModel includes extendedBench + metaSource, webEnrichmentCache primed at boot), `package.json` (+5 lines: test commands + scripts), `test/model-merger.test.js` (+100 lines: new test cases), `tasks/t4.md`, `tasks/t5.md`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "free-coding-models",
-  "version": "0.5.60",
+  "version": "0.5.62",
   "description": "Find the fastest coding LLM models in seconds \u2014 ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
   "keywords": [
     "nvidia",
@@ -41,6 +41,8 @@
   "files": [
     "bin/",
     "src/",
+    "scripts/check-drift.mjs",
+    "scripts/update-benchmarks.mjs",
     "web/",
     "!web/node_modules/",
     "!web/node_modules/**",
@@ -53,7 +55,11 @@
   ],
   "scripts": {
     "start": "node bin/free-coding-models.js",
-    "test": "node --test test/test.js test/fcm-agent-core.test.js test/patch-openclaw.test.js test/provider-metadata.test.js test/config-permission-hint.test.js test/probe-cache.test.js test/passive-quota.test.js test/runtime-telemetry.test.js",
+    "test": "node --test test/test.js test/fcm-agent-core.test.js test/patch-openclaw.test.js test/provider-metadata.test.js test/config-permission-hint.test.js test/probe-cache.test.js test/passive-quota.test.js test/runtime-telemetry.test.js test/extended-benchmarks.test.js test/models-dev.test.js test/model-merger.test.js",
+    "test:extended-benchmarks": "node --test test/extended-benchmarks.test.js",
+    "test:models-dev": "node --test test/models-dev.test.js",
+    "update:benchmarks": "node scripts/update-benchmarks.mjs",
+    "check:drift": "node scripts/check-drift.mjs",
     "prepack": "npm run build:web",
     "dev": "node scripts/dev-web.mjs",
     "dev:web": "node scripts/dev-web.mjs",

package/scripts/check-drift.mjs

@@ -0,0 +1,160 @@
+#!/usr/bin/env node
+/**
+ * @file scripts/check-drift.mjs
+ * @description CLI: `free-coding-models --check-drift` — diff `sources.js` against the live
+ *              models.dev catalog and print a human-readable drift report. Exits 1 on
+ *              mismatch (so CI fails the build), 0 on clean.
+ *
+ * @details
+ *   📖 Usage:
+ *   📖   node scripts/check-drift.mjs                     # threshold = 0 (any drift = fail)
+ *   📖   node scripts/check-drift.mjs --threshold 5        # only fail if 5+ mismatches
+ *   📖   node scripts/check-drift.mjs --json               # output as JSON
+ *   📖   node scripts/check-drift.mjs --no-fail            # always exit 0 (report only)
+ *   📖   node scripts/check-drift.mjs --fixture path.json  # use a local fixture instead of fetch
+ *   📖   pnpm check:drift                                 # convenience npm script
+ *
+ *   📖 Exit codes:
+ *   📖   0  No drift (or below threshold)
+ *   📖   1  Drift detected
+ *   📖   2  Fetch failed (network error)
+ *   📖   3  Invalid arguments
+ *
+ *   📖 Cross-surface: same code path as the TUI footer chip + /health endpoint.
+ *   📖 The drift list itself is also written to stderr as a JSON blob when --json
+ *   📖 is passed, for easy piping into jq / GitHub Actions summary.
+ */
+
+import { readFileSync } from 'node:fs'
+import { fileURLToPath } from 'node:url'
+import { dirname, join } from 'node:path'
+import { fetchModelsDevCatalog, clearModelsDevCache } from '../src/core/models-dev-fetcher.js'
+import { buildModelIndex } from '../src/core/models-dev-index.js'
+import { detectDrift, summarizeDrift, formatDriftReport } from '../src/core/models-drift.js'
+
+const here = dirname(fileURLToPath(import.meta.url))
+const projectRoot = join(here, '..')
+
+// ─── CLI arg parsing ─────────────────────────────────────────────────────────
+
+function parseCliArgs(argv) {
+  const out = { threshold: 0, json: false, noFail: false, fixture: null, silent: false }
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i]
+    if (a === '--threshold' || a === '--drift-threshold') {
+      const v = argv[++i]
+      const n = parseInt(v, 10)
+      if (!Number.isFinite(n) || n < 0) {
+        console.error(`error: --threshold must be a non-negative integer (got: ${v})`)
+        process.exit(3)
+      }
+      out.threshold = n
+    } else if (a === '--json') {
+      out.json = true
+    } else if (a === '--no-fail') {
+      out.noFail = true
+    } else if (a === '--fixture') {
+      out.fixture = argv[++i]
+    } else if (a === '--silent') {
+      out.silent = true
+    } else if (a === '--help' || a === '-h') {
+      console.log(`
+Usage: free-coding-models --check-drift [options]
+
+Options:
+  --threshold N        Only fail when N+ mismatches are found (default: 0)
+  --drift-threshold N  Alias for --threshold
+  --json               Output the report as JSON on stdout
+  --no-fail            Always exit 0 (report only, useful for weekly issues)
+  --fixture path.json  Use a local catalog file instead of fetching from models.dev
+  --silent             Suppress all non-error output
+  --help, -h           Show this help
+
+Exit codes:
+  0  No drift (or below threshold)
+  1  Drift detected
+  2  Fetch failed (network error)
+  3  Invalid arguments
+`)
+      process.exit(0)
+    } else {
+      console.error(`error: unknown argument: ${a}`)
+      process.exit(3)
+    }
+  }
+  return out
+}
+
+// ─── Load sources.js MODELS ───────────────────────────────────────────────────
+
+/**
+ * 📖 sources.js uses ESM `export const MODELS = [...]` so we can import it directly.
+ * 📖 We only need the MODELS array for drift detection.
+ */
+async function loadSourcesModels() {
+  const mod = await import(join(projectRoot, 'sources.js'))
+  if (!mod || !Array.isArray(mod.MODELS)) {
+    throw new Error('sources.js did not export a MODELS array')
+  }
+  return mod.MODELS
+}
+
+// ─── Main ─────────────────────────────────────────────────────────────────────
+
+async function main() {
+  const args = parseCliArgs(process.argv.slice(2))
+
+  // 📖 Step 1: Load sources.js
+  let models
+  try {
+    models = await loadSourcesModels()
+  } catch (err) {
+    console.error(`error: failed to load sources.js: ${err.message}`)
+    process.exit(3)
+  }
+
+  // 📖 Step 2: Get the models.dev catalog (fetch or fixture)
+  let catalog = null
+  if (args.fixture) {
+    try {
+      const raw = readFileSync(args.fixture, 'utf-8')
+      catalog = JSON.parse(raw)
+    } catch (err) {
+      console.error(`error: failed to read fixture ${args.fixture}: ${err.message}`)
+      process.exit(3)
+    }
+  } else {
+    clearModelsDevCache()
+    catalog = await fetchModelsDevCatalog({ retries: 3, retryDelayMs: 250, silent: false })
+    if (!catalog) {
+      console.error('error: failed to fetch models.dev catalog (network error)')
+      process.exit(2)
+    }
+  }
+
+  // 📖 Step 3: Run drift detection
+  const index = buildModelIndex(catalog)
+  const drift = detectDrift(models, catalog, { index, threshold: args.threshold })
+  const summary = summarizeDrift(drift)
+
+  // 📖 Step 4: Output
+  if (args.json) {
+    process.stdout.write(JSON.stringify({ summary, mismatches: drift }, null, 2) + '\n')
+  } else if (!args.silent) {
+    process.stdout.write(formatDriftReport(drift) + '\n')
+  }
+
+  // 📖 Step 5: Exit code
+  if (drift.length === 0) {
+    process.exit(0)
+  }
+  if (args.noFail) {
+    process.exit(0)
+  }
+  process.exit(1)
+}
+
+main().catch(err => {
+  console.error(`error: unhandled exception: ${err.stack ?? err.message}`)
+  process.exit(3)
+})

package/scripts/update-benchmarks.mjs

@@ -0,0 +1,239 @@
+#!/usr/bin/env node
+/**
+ * @file scripts/update-benchmarks.mjs
+ * @description Regenerate `src/data/benchmarks.json` from a fresh data source.
+ *              Called at release time (or manually via `pnpm update:benchmarks`).
+ *              The committed JSON is what the runtime reads — the fetch only happens
+ *              here, then the data is committed alongside the version bump.
+ *
+ * @details
+ *   📖 Usage:
+ *   📖   node scripts/update-benchmarks.mjs                    # try to fetch live
+ *   📖   node scripts/update-benchmarks.mjs --fixture PATH     # use a local JSON file
+ *   📖   node scripts/update-benchmarks.mjs --dry-run          # print to stdout, do not write
+ *   📖   pnpm update:benchmarks                                # convenience
+ *
+ *   📖 Data sources (in priority order):
+ *   📖   1. --fixture PATH   (for tests / offline runs)
+ *   📖   2. Live fetch from models.dev (the same source we use for drift detection)
+ *   📖   3. Abort with non-zero exit if neither is available
+ *
+ *   📖 Output: src/data/benchmarks.json — same shape as the current seed (see the
+ *   📖 existing file for the schema). The script merges the fetched data with the
+ *   📖 existing file so curated values are preserved when the source is missing them.
+ *
+ *   📖 Exit codes:
+ *   📖   0  Success (or --dry-run)
+ *   📖   1  Fetch failed + no fixture + no existing file
+ *   📖   2  Invalid arguments
+ *   📖   3  Output write failed
+ */
+
+import { readFileSync, writeFileSync, existsSync } from 'node:fs'
+import { fileURLToPath } from 'node:url'
+import { dirname, join } from 'node:path'
+
+const here = dirname(fileURLToPath(import.meta.url))
+const projectRoot = join(here, '..')
+const OUTPUT_PATH = join(projectRoot, 'src', 'data', 'benchmarks.json')
+
+// ─── CLI arg parsing ─────────────────────────────────────────────────────────
+
+function parseCliArgs(argv) {
+  const out = { fixture: null, dryRun: false, silent: false }
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i]
+    if (a === '--fixture') out.fixture = argv[++i]
+    else if (a === '--dry-run') out.dryRun = true
+    else if (a === '--silent') out.silent = true
+    else if (a === '--help' || a === '-h') {
+      console.log(`
+Usage: update-benchmarks [options]
+
+Options:
+  --fixture PATH   Use a local JSON file instead of fetching from models.dev
+  --dry-run        Print the new JSON to stdout instead of writing to disk
+  --silent         Suppress non-error output
+  --help, -h       Show this help
+
+Exit codes:
+  0  Success (or --dry-run)
+  1  Fetch failed + no fixture + no existing file
+  2  Invalid arguments
+  3  Output write failed
+`)
+      process.exit(0)
+    } else {
+      console.error(`error: unknown argument: ${a}`)
+      process.exit(2)
+    }
+  }
+  return out
+}
+
+// ─── Data source ─────────────────────────────────────────────────────────────
+
+async function fetchLive() {
+  const ctrl = new AbortController()
+  const timer = setTimeout(() => ctrl.abort(), 12_000)
+  try {
+    const res = await fetch('https://models.dev/models.json', {
+      headers: { 'User-Agent': 'free-coding-models-benchmarks-updater' },
+      signal: ctrl.signal,
+    })
+    if (!res.ok) throw new Error(`HTTP ${res.status} ${res.statusText}`)
+    return await res.json()
+  } finally {
+    clearTimeout(timer)
+  }
+}
+
+/**
+ * 📖 Synthesize an extended-benchmark entry from a models.dev model.
+ * 📖 Most fields are derivable from the basic metadata; the rest is left null
+ * 📖 and gets filled in by future fetches.
+ */
+function synthesizeFromModelsDev(modelId, rawEntry) {
+  const limit = rawEntry.limit || {}
+  const modalities = rawEntry.modalities || {}
+  const vision = Array.isArray(modalities.input)
+    ? modalities.input.some(m => typeof m === 'string' && /image|video/i.test(m))
+    : false
+  return {
+    codingIndex: null,
+    mathIndex: null,
+    agenticIndex: null,
+    reasoningIndex: null,
+    mmluPro: null,
+    gpqa: null,
+    hle: null,
+    contextWindow: typeof limit.context === 'number' ? limit.context : null,
+    supportsReasoning: rawEntry.reasoning === true,
+    supportsVision: vision,
+    lastUpdated: new Date().toISOString().slice(0, 10),
+    originalModel: rawEntry.name || modelId,
+    // 📖 provenance marker — lets the TUI show "🤖 from models.dev" vs "📦 curated"
+    source: 'models.dev',
+  }
+}
+
+// ─── Main ─────────────────────────────────────────────────────────────────────
+
+async function main() {
+  const args = parseCliArgs(process.argv.slice(2))
+
+  // 📖 Step 1: load existing seed (preserve curated values when source is missing fields)
+  let existing = {}
+  if (existsSync(OUTPUT_PATH)) {
+    try {
+      existing = JSON.parse(readFileSync(OUTPUT_PATH, 'utf-8'))
+      if (!existing || typeof existing !== 'object') existing = {}
+    } catch {
+      existing = {}
+    }
+  }
+  const existingMeta = existing._meta ?? {}
+  const curatedModels = { ...existing }
+  delete curatedModels._meta
+
+  // 📖 Step 2: get the source data
+  let sourceData = null
+  if (args.fixture) {
+    try {
+      sourceData = JSON.parse(readFileSync(args.fixture, 'utf-8'))
+    } catch (err) {
+      console.error(`error: failed to read fixture ${args.fixture}: ${err.message}`)
+      process.exit(1)
+    }
+  } else {
+    try {
+      sourceData = await fetchLive()
+    } catch (err) {
+      if (!args.silent) {
+        console.error(`warn: live fetch failed: ${err.message}`)
+      }
+      if (Object.keys(curatedModels).length === 0) {
+        console.error('error: no live data + no existing file to fall back on')
+        process.exit(1)
+      }
+      // 📖 Fall back to the existing catalog (no new fields, but don't fail)
+      if (!args.silent) {
+        console.error('warn: keeping existing benchmarks.json unchanged')
+      }
+      process.exit(0)
+    }
+  }
+
+  // 📖 Step 3: detect format (flat or nested) and synthesize new entries
+  const synthesized = {}
+  const entries = Object.entries(sourceData || {})
+  const isFlat = entries.length > 0 && entries.every(([k, v]) => {
+    if (!v || typeof v !== 'object') return false
+    if (k.includes('/')) return typeof v.limit === 'object' || typeof v.id === 'string'
+    return false
+  })
+
+  if (isFlat) {
+    for (const [idKey, rawEntry] of entries) {
+      if (!rawEntry || typeof rawEntry !== 'object') continue
+      synthesized[idKey] = synthesizeFromModelsDev(idKey, rawEntry)
+    }
+  } else {
+    for (const [providerKey, providerBucket] of entries) {
+      if (!providerBucket || typeof providerBucket !== 'object') continue
+      const models = providerBucket.models
+      if (!models || typeof models !== 'object') continue
+      for (const [modelId, rawEntry] of Object.entries(models)) {
+        const idKey = `${providerKey}/${modelId}`
+        synthesized[idKey] = synthesizeFromModelsDev(idKey, rawEntry)
+      }
+    }
+  }
+
+  // 📖 Step 4: merge — synthesized entries fill in the basics (ctx, vision, reasoning);
+  // 📖 curated entries preserve the rich benchmark scores. The merger prefers curated
+  // 📖 for any field that already has a non-null value.
+  const merged = { ...synthesized }
+  for (const [key, curated] of Object.entries(curatedModels)) {
+    if (!curated || typeof curated !== 'object') continue
+    const base = merged[key] ?? {}
+    merged[key] = {
+      ...base,
+      ...curated,  // 📖 curated wins for any field it has
+      // 📖 Preserve source field — 'curated' overrides 'models.dev' if the entry
+      // 📖 already had a value (means we have a richer dataset from a prior commit)
+      source: curated.source ?? 'curated',
+    }
+  }
+
+  // 📖 Step 5: attach _meta
+  const newMeta = {
+    schemaVersion: 1,
+    lastUpdated: new Date().toISOString().slice(0, 10),
+    source: 'models.dev (auto) + curated (manual)',
+    notes: existingMeta.notes ?? 'Refreshed by scripts/update-benchmarks.mjs',
+  }
+  const finalOutput = { _meta: newMeta, ...merged }
+
+  // 📖 Step 6: write
+  const json = JSON.stringify(finalOutput, null, 2) + '\n'
+  if (args.dryRun) {
+    process.stdout.write(json)
+    process.exit(0)
+  }
+  try {
+    writeFileSync(OUTPUT_PATH, json, 'utf-8')
+    if (!args.silent) {
+      console.log(`✓ wrote ${OUTPUT_PATH} (${merged.length} entries)`)
+    }
+    process.exit(0)
+  } catch (err) {
+    console.error(`error: failed to write ${OUTPUT_PATH}: ${err.message}`)
+    process.exit(3)
+  }
+}
+
+main().catch(err => {
+  console.error(`error: unhandled exception: ${err.stack ?? err.message}`)
+  process.exit(2)
+})