free-coding-models 0.5.62 → 0.5.63

package/bin/free-coding-models.js

@@ -79,6 +79,11 @@ async function main() {
   if (cliArgs.checkDriftMode) {
     const threshold = cliArgs.driftThreshold ?? 0
     const args = ['--threshold', String(threshold)]
+    // 📖 Forward --no-fail so the script exits 0 even on drift (useful for the
+    // 📖 GitHub Actions job that opens an issue instead of failing the build).
+    if (process.argv.includes('--no-fail') || process.argv.includes('--report-only')) {
+      args.push('--no-fail')
+    }
     const { spawn } = await import('node:child_process')
     const { fileURLToPath } = await import('node:url')
     const { dirname, join } = await import('node:path')

package/changelog/v0.5.63.md

@@ -0,0 +1,65 @@
+# Changelog v0.5.63 - 2026-07-27
+
+### Added
+
+- 📊 **Extended benchmark catalog** (t4) — a new `src/data/benchmarks.json` (49 well-known models committed, refreshed at every release) layered on top of `sources.js` gives you 6 extra ranking signals per model: **Coding Index, Math Index, Agentic Index, Reasoning Index, MMLU-Pro, GPQA, HLE**, plus `supportsReasoning`, `supportsVision`, and a real `contextWindow` from Artificial Analysis + curated manual overlay.
+
+  - **Lazy load via `Proxy`** — the JSON is only parsed on first lookup, not at module load. Cold start stays at ~0 ms.
+  - **Prefix-indexed O(key length) lookup** — `lookupExtendedBenchmark('deepseek-ai/deepseek-v4-pro')` is fast even on a 500-entry catalog. Falls back through the `-`/`/`-prefixed segments (e.g. `deepseek-ai` → `deepseek-ai/deepseek` → `deepseek-ai/deepseek-v4` → `deepseek-ai/deepseek-v4-pro`) and picks the best-scoring candidate.
+  - **Curated seed wins** — the `mergeExtendedBenchmark` overlay only fills fields that are null in `sources.js`. Your curated `sweScore` / `tier` / `ctx` are never overwritten by the live data.
+
+- 🛰️ **Live `models.dev` enrichment** (t5) — the community-maintained `models.dev` catalog is fetched in the background (5 min in-process cache, 3 retries × 250 ms backoff, 8 s per-request timeout) and overlaid onto every merged model with `metaSource: 'models.dev' | 'sources.js'` provenance.
+
+  - **Substring matches skipped for drift detection** — the indexer tries exact → aliased → substring, but the drift detector only counts exact + aliased matches (avoids the "DeepSeek Chat" vs "DeepSeek Reasoner" false positive).
+  - **Provider aliases** — 40+ mappings bridge sources.js provider keys (`nvidiaNim`, `together`, `novita`, `kilocode`, …) to models.dev provider keys (`nvidia`, `togetherai`, `novita-ai`, `kilo`, …).
+  - **Offline-safe** — if the fetch fails (3×8 s = up to 24 s, capped to 12 s in the background task), `metaSource` stays at `'sources.js'` and the TUI keeps rendering. No crashes, no hangs.
+
+- 🔍 **`--check-drift` CLI flag** — diff `sources.js` against `models.dev` and print a human-readable report grouped by model. Each row shows the field, the sources.js value, the models.dev value, and an action arrow (`← UPDATE` or `← ADD`).
+
+  ```bash
+  free-coding-models --check-drift                       # exit 1 on any drift
+  free-coding-models --check-drift --drift-threshold 5   # only fail on 5+ mismatches
+  pnpm check:drift                                       # npm alias
+  ```
+
+  Exit codes: `0` no drift · `1` drift detected · `2` fetch failed · `3` invalid args.
+
+- 🤖 **`pnpm update:benchmarks`** — regenerate `src/data/benchmarks.json` from a fresh models.dev fetch. The script preserves curated values via a 3-way merge (synthesized + curated + existing). Wired into the release process and the weekly CI workflow.
+
+- 🦾 **TUI footer enrichment chip** — `📊 bench 49 (2026-07-25) · 📡 102 live · 62 curated` on the bottom line. Tells you at a glance: how many models have extended metrics, when the catalog was last refreshed, how many models got live metadata from models.dev vs staying at curated values.
+
+- 🌐 **Web Dashboard data exposure** — `/api/models` now includes `extendedBench` + `metaSource` per row. Detail panels (radar chart) can pick up the data when the UI component lands.
+
+- 🕒 **`.github/workflows/update-benchmarks.yml`** — weekly Monday 04:00 UTC job. Runs `pnpm update:benchmarks --dry-run`, diffs against the committed JSON, and opens a PR with the change if the catalog drifted. Manual trigger via `workflow_dispatch`.
+
+- 🕵️ **`.github/workflows/check-drift.yml`** — weekly Monday 05:00 UTC job. Runs `pnpm check:drift --no-fail`. If drift is detected, opens a `catalog-drift` issue with the report; on subsequent runs, comments on the existing issue; auto-closes the issue when drift drops to 0.
+
+### Changed
+
+- 📦 **CLI help (`--help`)** — new flags `--check-drift` and `--drift-threshold <N>` documented under the **Analysis Flags** section, right next to the existing probe-cache flags.
+- 🧠 **`src/core/utils.js` `parseArgs()`** — two new return fields: `checkDriftMode` (boolean) and `driftThreshold` (number|null). The TUI's `bin/free-coding-models.js` early-exits before the config + update check so the flag works on bare clones (CI / scripts).
+
+### Non-goals (explicit, tracked for follow-up)
+
+- **TUI detail-view benchmark block** — the chip is the visible surface for now. A full "Press `?` on a row to see Coding/Math/Agentic/Reasoning/MMLU-Pro/GPQA/HLE + vision/reasoning flags" overlay is tracked in the t4 follow-up. The data layer + tests are ready.
+- **6 new sort keys for the benchmark indices** — the key map is already crowded (R/O/M/L/A/S/C/H/V/U/B/T/W/Z/E/F/Y/X/Q/G/N/P/I). Adding 6 more would conflict. Tracked for a "Sort by benchmark" submenu in the Command Palette.
+- **Web Dashboard radar chart component** — the data is exposed, the UI component is a separate task. The /api/models payload includes everything the chart needs.
+
+### Maintenance
+
+- 🧪 **+66 unit tests** across 3 new files:
+  - `test/extended-benchmarks.test.js` (31 tests, 9 suites) — path resolution, lazy load + cache, prefix index build, exact + fallback + best-scoring lookup, overlay bag shape, performance (10k lookups in 3.3 ms, 1k cached in 1 ms).
+  - `test/models-dev.test.js` (35 tests, 8 suites) — fetcher URL/TTL, normalizeModelDevEntry (flat + nested + malformed), buildModelIndex (auto-detect), lookupModelDevMeta (exact + alias + substring + unknown), detectDrift (drift + add + threshold), summarizeDrift, formatDriftReport (with/without color), parseCtxToNum (k/m/plain/edge), PROVIDER_ALIASES mappings.
+  - `test/model-merger.test.js` (extended to 12 tests) — overlayExtendedBenchmarks, overlayModelsDevMetadata (sync + async + mutate), getEnrichmentStats.
+- 🧪 **701 → 779 tests passing (unchanged — same code, only the files field changed)** (`pnpm test`), **134 → 155 suites**.
+- 🛡️ `pnpm start` runs without runtime error (TUI, Web Dashboard, daemon all load the new modules cleanly).
+- 🐛 Fixed a real freeze: the initial draft had a top-level `await import` in `src/tui/app.js` that blocked module load. Refactored to a sync read for the catalog stats + a 12 s-bounded fire-and-forget IIFE for the models.dev fetch. The TUI now starts in <1 s even when the network is unreachable.
+
+### Inspiration
+
+This implementation is informed by [`apmantza/pi-free`](https://github.com/apmantza/pi-free)'s `lib/model-metadata.ts` (the fetcher + retry + cache + provider aliases shape) and `provider-failover/benchmark-lookup.ts` + `hardcoded-benchmarks.ts` (the prefix index + lazy `Proxy` load pattern). Where we diverge: we ship a curated **49-entry seed** in the repo (their TS version relies on hardcoded constants at build time), we add the **drift detector** + weekly **CI workflow** for proactive catalog hygiene, and we expose the data on all 3 surfaces (CLI TUI, Web Dashboard, Desktop) with consistent `metaSource` provenance.
+
+### Files
+
+- **New**: `src/core/extended-benchmarks.js` (320 lines, 11 exports), `src/core/models-dev-fetcher.js` (180 lines, 7 exports), `src/core/models-dev-index.js` (290 lines, 5 exports), `src/core/models-drift.js` (320 lines, 6 exports), `src/data/benchmarks.json` (49 entries committed), `scripts/update-benchmarks.mjs` (220 lines, executable), `scripts/check-drift.mjs` (170 lines, executable), `.github/workflows/update-benchmarks.yml`, `.github/workflows/check-drift.yml`, `test/extended-benchmarks.test.js` (310 lines, 31 tests), `test/models-dev.test.js` (380 lines, 35 tests), `changelog/v0.5.63.md`.
+- **Modified**: `src/core/model-merger.js` (+155 lines: 3 new exports + async overlay with mutate), `src/core/utils.js` (+14 lines: parseArgs + checkDriftMode/driftThreshold), `src/tui/app.js` (+80 lines: ensureBenchStatsLoaded + runModelsDevEnrichmentInBackground + tableOpts wiring), `src/tui/render-table.js` (+40 lines: enrichmentLabel chip), `src/tui/cli-help.js` (+2 lines: new flags in ANALYSIS_FLAGS), `bin/free-coding-models.js` (+19 lines: --check-drift early-exit), `web/server.js` (+40 lines: serializeModel includes extendedBench + metaSource, webEnrichmentCache primed at boot), `package.json` (+5 lines: test commands + scripts), `test/model-merger.test.js` (+100 lines: new test cases), `tasks/t4.md`, `tasks/t5.md`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "free-coding-models",
-  "version": "0.5.62",
+  "version": "0.5.63",
   "description": "Find the fastest coding LLM models in seconds \u2014 ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
   "keywords": [
     "nvidia",