free-coding-models 0.5.59 → 0.5.60

package/README.md

@@ -550,6 +550,35 @@ The `source` field is `"header"` when the value came from a passive response hea
 
 ---
 
+## 📈 Runtime telemetry: real-world scores
+
+Every routed request through the daemon feeds a persistent per-model telemetry file (`~/.free-coding-models/runtime-telemetry.json`) — the **honesty layer** that complements SWE-bench with what models actually do on free tiers.
+
+- **Real success rate** — `(successCalls / totalCalls)` updated on every routed request (success and failure paths alike).
+- **Real throughput** — `avgTokensPerSecond` derived from completion tokens / total latency, so you see what your code actually streams at.
+- **Recent calls (last 50)** — for debugging "why did this just 500?" without rebuilding state.
+- **Composite `Real` score** — `successRate × 0.6 + sigmoid01(tok/s) × 0.25 + recency × 0.15`. Null when below `MIN_CALLS_FOR_SCORE = 5` (no penalty for new models).
+
+### Where it shows up
+
+| Surface | What you'll see |
+|---------|-----------------|
+| TUI `Real` column | Inline composite score per row, `–` when insufficient data |
+| TUI `W` sort key | Sort by real-world score descending |
+| TUI `Shift+W` | Runtime Report overlay — per-model breakdown + recent calls |
+| Web Dashboard | "Runtime Telemetry" cards with animated success-rate bars |
+| `/api/router/stats.runtime` | `{ stats: { modelsTracked, totalCalls, modelsWithSignal }, models: { ... } }` |
+
+### Privacy
+
+The telemetry file lives **locally only** (`~/.free-coding-models/runtime-telemetry.json`, `0600` perms). Nothing is sent upstream unless you opt in to a future aggregate leaderboard. The file holds **metadata only** — success, latency, tokens, error reason. No prompts, no responses, no content.
+
+### CLI flag
+
+- `--clear-runtime` — wipe the file before launching (reset the baseline).
+
+---
+
 ## π Pi Extension — FCM-Pi ⚠️ BETA
 
 **FCM-Pi** is a native [Pi coding agent](https://pi.dev) extension that integrates `free-coding-models` directly into your Pi session. It stays silent by default, scans only when you run `/fcm`, and lets you explicitly hot-swap models mid-session.
@@ -764,6 +793,7 @@ See [`packages/fcm-agent-core/README.md`](./packages/fcm-agent-core/README.md) f
 - **Last release timestamp** — light pink footer shows `Last release: Mar 27, 2026, 09:42 PM` from npm so users know how fresh the data is
 - **Persistent probe-cache (t1)** — every health probe result is cached to `~/.free-coding-models/probe-cache.json` for 24h. Warm starts render the full ranking in <500ms, only re-ping the models that are due (broken or TTL-expired). Broken models are auto-hidden across sessions — toggle visibility with **Shift+B**. See [Persistent probe cache](#-persistent-probe-cache) below for `--reprobe`, `--probe-ttl`, `--show-broken`.
 - **Live quota from response headers (t2)** — every routed chat-completion response already carries `x-ratelimit-*` headers. The daemon parses them in 8 variants and exposes live per-provider quota on the TUI footer (`📊 groq 78% · sambanova 41%`) and in the Web Dashboard (`Provider Quota` section with animated progress bars). Zero extra network requests, zero quota waste. See [Live quota from headers](#-live-quota-from-response-headers) below.
+- **Runtime telemetry: real-world scores (t3)** — every routed request through the daemon feeds a persistent per-model telemetry file (`~/.free-coding-models/runtime-telemetry.json`) with real success rate, throughput, and recent calls. The `Real` column + `W` sort key in the TUI rank models by what *actually* works on free tiers, not what they claim on SWE-bench. See [Runtime telemetry](#-runtime-telemetry-real-world-scores) below.
 
 ---
 

package/bin/free-coding-models.js

@@ -62,6 +62,18 @@ async function main() {
     process.exit(0);
   }
 
+  // 📖 --clear-runtime (t3): wipe ~/.free-coding-models/runtime-telemetry.json
+  // 📖 before launching any surface. Keeps the TUI / daemon / web flows consistent.
+  if (cliArgs.clearRuntimeMode) {
+    try {
+      const { clearRuntimeTelemetry } = await import('../src/core/runtime-telemetry.js')
+      const ok = clearRuntimeTelemetry()
+      console.log(chalk.dim(`  ${ok ? '✓' : '✗'} runtime-telemetry.json ${ok ? 'cleared' : 'clear failed'}`))
+    } catch (err) {
+      console.log(chalk.dim(`  runtime-telemetry.json clear failed: ${err?.message || err}`))
+    }
+  }
+
   // Load JSON config before operational modes so the mandatory update policy can
   // 📖 persist failure counters for TUI, Web Dashboard, Docker daemon, and Desktop sidecar launches.
   const config = loadConfig();

package/changelog/v0.5.60.md

@@ -0,0 +1,54 @@
+# Changelog v0.5.60 - 2026-07-26
+
+### Added
+
+- 📈 **Runtime telemetry: real-world scores** (t3) — every routed request through the daemon now feeds a persistent per-model telemetry file with real success rate, throughput, and recent calls. FCM stops showing you only what models *claim* on SWE-bench and starts showing you what they **actually do** on free tiers.
+
+  - **Composite `Real` score** (0..100) — `successRate × 0.6 + sigmoid01(tok/s) × 0.25 + recency × 0.15`. Null when below `MIN_CALLS_FOR_SCORE = 5` (no penalty for new models).
+  - **Real success rate** — `(successCalls / totalCalls)`. Updated on every routed request, success AND failure paths alike (auth_error / 5xx / 429 all count).
+  - **Real throughput** — `avgTokensPerSecond` from completion tokens / total latency. What your code actually streams at.
+  - **Recent calls (last 50)** — FIFO trim per model. Useful for debugging "why did this just 500?" without rebuilding state.
+  - **Local-only by default** — the file lives at `~/.free-coding-models/runtime-telemetry.json` (`0600` perms). Nothing leaves your machine unless you opt in to a future aggregate leaderboard. Metadata only — no prompts, no responses.
+
+- 🆕 **New sort key `W`** in the TUI: sorts by `Real` score descending. Models with insufficient data (null score) sink to the bottom in both directions.
+
+- 📊 **New TUI `Real` column**: shows the composite score inline per row, or `–` when there's not enough signal yet.
+
+- 🔥 **`Shift+W` Runtime Report overlay** — per-model breakdown (success rate, avg latency, avg tok/s, recent calls list). Same pattern as `Shift+R` Router Dashboard / `Shift+T` Token Usage. Scroll with j/k/up/down/pageup/pagedown/home/end, close with Escape.
+
+- 🌐 **Web Dashboard `Runtime Telemetry` section** — per-model cards with animated success-rate bars (red ≤ 50% / amber 50-80% / green ≥ 80%). Updated every `/api/router/stats` poll (5 s). Models with < 5 calls are excluded so the section only shows real signal.
+
+- 🛰️ **New daemon endpoint `/stats/runtime`** — returns `{ ok, stats: { modelsTracked, totalCalls, modelsWithSignal }, models: { [key]: ModelTelemetry } }`. Same shape as the in-TUI dashboard digest.
+
+- 🧪 **`recordModelCall(providerKey, modelId, callResult, opts?)`** — drop-in hook for any future surface (Tauri Desktop, scripts) that wants to feed runtime telemetry.
+
+- 🛠️ **`--clear-runtime` CLI flag** — wipe the telemetry file before launching any surface. Useful when the user wants to reset the baseline.
+
+### Changed
+
+- 🩺 **Daemon reverse proxy** records every successful + failed routed outcome via `recordRuntimeCall()`. A debounced 5 s flush keeps the file in sync without thrashing the disk. The flush timer is cleared in `shutdown()` so no orphan writes after exit.
+
+- 📋 **Daemon `statsPayload`** now exposes `runtimeTelemetry: { stats, models }` so the existing `/stats` consumers get the digest for free.
+
+- 🔧 **Internal** — added `sigmoid01(x)` (logistic curve so 50 tok/s → 0.5) and `recencyDecay(lastUpdatedMs)` (1.0 today → 0.0 at 30 d) to the real-world score formula. Both are pure functions with no dependencies.
+
+### Non-goals (explicit)
+
+- **Streaming token extraction** — the known gap from the PRD stays. The runtime telemetry only counts requests where the upstream returned `usage` (i.e. non-streaming, or streaming providers that emit usage on the final chunk).
+- **Sharing / leaderboard** — no aggregation server yet. When there is one, opt-in.
+
+### Maintenance
+
+- 🧪 **+30 unit tests** (`test/runtime-telemetry.test.js`, new file): path resolution (XDG-aware), constants sanity, `recordModelCall` write/accumulate/drop garbage/FIFO cap, `tokensPerSecond` computation, derived fields, `getRealWorldScore` null below threshold / healthy → high / error-heavy → mid / age decay, `getCacheStats` aggregates + `modelsWithSignal` threshold, `pruneStaleEntries` drops old, persistence round-trip + corrupt recovery + missing file + clear + read-merge-write concurrency safety.
+- 🧪 **671 → 701 tests passing** (`pnpm test`), **125 → 134 suites**.
+- 🧹 `vite build` succeeds.
+- 📖 README updated with a new **📈 Runtime telemetry: real-world scores** section (where it shows up, privacy, `--clear-runtime`).
+
+### Inspiration
+
+This implementation is informed by [`apmantza/pi-free`](https://github.com/apmantza/pi-free)'s `lib/telemetry.ts` (which credits us in their other direction too). pi-free's `turn_end` hook is the daemon-side equivalent of our `recordRuntimeCall`; the schema, the 50-call FIFO, and the local-only file mirror their approach. Where we diverge: we expose the composite `Real` score as a first-class ranking signal (not just a debugging surface), ship it in 3 user-facing surfaces (CLI TUI / Web Dashboard / daemon `/stats`), and add the success×0.6 / speed×0.25 / recency×0.15 weights so the user sees a single number per model.
+
+### Files
+
+- **New**: `src/core/runtime-telemetry.js` (377 lines, 11 exports), `test/runtime-telemetry.test.js` (390 lines, 30 tests), `changelog/v0.5.60.md`.
+- **Modified**: `src/core/router-daemon.js` (record on every routed outcome + `/stats/runtime` + runtimeTelemetry in statsPayload), `src/core/utils.js` (`'realworld'` sort case), `src/tui/app.js` (boot-time score computation per result), `src/tui/tui-state.js` (5 new state fields for the overlay), `src/tui/key-handler.js` (Shift+W + overlay keyboard handling + command palette action), `bin/free-coding-models.js` (`--clear-runtime` flag plumbing), `web/src/components/router/RouterView.jsx` (Runtime Telemetry section), `README.md`, `tasks/t3.md`, `package.json` (test script).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "free-coding-models",
-  "version": "0.5.59",
+  "version": "0.5.60",
   "description": "Find the fastest coding LLM models in seconds \u2014 ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
   "keywords": [
     "nvidia",
@@ -53,7 +53,7 @@
   ],
   "scripts": {
     "start": "node bin/free-coding-models.js",
-    "test": "node --test test/test.js test/fcm-agent-core.test.js test/patch-openclaw.test.js test/provider-metadata.test.js test/config-permission-hint.test.js test/probe-cache.test.js test/passive-quota.test.js",
+    "test": "node --test test/test.js test/fcm-agent-core.test.js test/patch-openclaw.test.js test/provider-metadata.test.js test/config-permission-hint.test.js test/probe-cache.test.js test/passive-quota.test.js test/runtime-telemetry.test.js",
     "prepack": "npm run build:web",
     "dev": "node scripts/dev-web.mjs",
     "dev:web": "node scripts/dev-web.mjs",

package/src/core/router-daemon.js

@@ -68,6 +68,17 @@ import {
   getAllQuotas as getAllPassiveQuotas,
   STALENESS_MS as PASSIVE_QUOTA_STALENESS_MS,
 } from './provider-quota-fetchers.js'
+import {
+  recordModelCall as recordRuntimeModelCall,
+  getAllModelTelemetry as getAllRuntimeTelemetry,
+  getRealWorldScore as getRuntimeRealWorldScore,
+  getCacheStats as getRuntimeCacheStats,
+  loadRuntimeTelemetry,
+  flushRuntimeTelemetry as flushRuntimeTelemetryStore,
+  clearRuntimeTelemetry as clearRuntimeTelemetryStore,
+  pruneStaleEntries as pruneRuntimeTelemetry,
+  DEFAULT_MIN_CALLS_FOR_SCORE as RUNTIME_MIN_CALLS,
+} from './runtime-telemetry.js'
 
 export const ROUTER_DEFAULT_PORT = 19280
 export const ROUTER_MAX_PORT = 19289
@@ -895,9 +906,50 @@ class RouterRuntime {
     this.probeCache = loadProbeCache()
     this.probeCacheDirty = false
     this.probeCacheFlushTimer = null
+    // 📖 Runtime telemetry (t3): load the persistent per-model metrics file on boot.
+    // 📖 Prune models that haven't been seen in 30 days to keep file size bounded.
+    loadRuntimeTelemetry()
+    pruneRuntimeTelemetry(30 * 24 * 60 * 60 * 1000)
+    this.runtimeTelemetryDirty = false
+    this.runtimeTelemetryFlushTimer = null
     this.refreshRouteState()
   }
 
+  /**
+   * 📖 scheduleRuntimeTelemetryFlush — debounced write so we don't thrash the
+   * 📖 filesystem when many requests succeed in quick succession.
+   */
+  scheduleRuntimeTelemetryFlush() {
+    if (this.runtimeTelemetryFlushTimer) return
+    this.runtimeTelemetryFlushTimer = setTimeout(() => {
+      this.runtimeTelemetryFlushTimer = null
+      if (this.runtimeTelemetryDirty) {
+        flushRuntimeTelemetryStore()
+        this.runtimeTelemetryDirty = false
+      }
+    }, 5000)
+    if (typeof this.runtimeTelemetryFlushTimer.unref === 'function') this.runtimeTelemetryFlushTimer.unref()
+  }
+
+  /**
+   * 📖 recordRuntimeCall — central helper invoked from both success and failure
+   * 📖 paths in the reverse proxy. Wraps recordModelCall with the call-shape
+   * 📖 translation so every routed outcome (success or fail) feeds the telemetry.
+   */
+  recordRuntimeCall({ providerKey, modelId, success, latencyMs, usage, error }) {
+    if (!providerKey || !modelId) return
+    recordRuntimeModelCall(providerKey, modelId, {
+      success: !!success,
+      latencyMs: typeof latencyMs === 'number' && Number.isFinite(latencyMs) ? latencyMs : 0,
+      promptTokens: usage?.prompt_tokens,
+      completionTokens: usage?.completion_tokens,
+      stopReason: success ? 'stop' : null,
+      error: success ? null : (error || 'unknown'),
+    })
+    this.runtimeTelemetryDirty = true
+    this.scheduleRuntimeTelemetryFlush()
+  }
+
   buildModelCatalog() {
     const catalog = new Map()
     for (const [providerKey, source] of Object.entries(sources)) {
@@ -1503,6 +1555,15 @@ class RouterRuntime {
       // 📖 fetcher fallback). Stale entries (older than PASSIVE_QUOTA_STALENESS_MS)
       // 📖 are excluded so the consumer only sees fresh data.
       quota: Object.fromEntries(getAllPassiveQuotas()),
+      // 📖 Runtime telemetry (t3): aggregate counters + every model's derived
+      // 📖 snapshot (successRate, avgLatencyMs, avgTokensPerSecond, recentCalls).
+      // 📖 Surfaced via /stats/runtime (separate endpoint) and here as a digest
+      // 📖 so dashboards can show '1,420 calls tracked across 12 models' without
+      // 📖 an extra round-trip.
+      runtimeTelemetry: {
+        stats: getRuntimeCacheStats(),
+        models: getAllRuntimeTelemetry(),
+      },
     }
   }
 
@@ -2087,6 +2148,15 @@ class RouterRuntime {
         this.markSuccess(key, latencyMs)
         const usage = extractUsage(parsed.value)
         this.tokenTracker.record(candidate.provider, candidate.model, usage)
+        // 📖 Runtime telemetry (t3): track every successful routed request so the
+        // 📖 real-world score can rank models by what *actually* works.
+        this.recordRuntimeCall({
+          providerKey: candidate.provider,
+          modelId: candidate.model,
+          success: true,
+          latencyMs,
+          usage,
+        })
         this.totalRequestsRouted += 1
         // 📖 Fire app_router_use telemetry once per 10 routed requests
         if (this.totalRequestsRouted % 10 === 0) {
@@ -2122,12 +2192,20 @@ class RouterRuntime {
       if (AUTH_STATUS_CODES.has(response.status)) {
         this.markAuthError(key, `HTTP ${response.status}`)
         this.addRequestLog({ request_id: requestId, model: key, status: response.status, latency_ms: latencyMs, tokens: 0, failover: attemptIndex > 0, error: 'auth_error' })
+        this.recordRuntimeCall({
+          providerKey: candidate.provider, modelId: candidate.model,
+          success: false, latencyMs, error: `auth_${response.status}`,
+        })
         return { done: false, failoverToNext: true, reason: `auth_${response.status}`, authFailure: true }
       }
 
       if (RETRYABLE_STATUS_CODES.has(response.status)) {
         this.markFailure(key, `HTTP ${response.status}`, response.status, upstreamMeta)
         this.addRequestLog({ request_id: requestId, model: key, status: response.status, latency_ms: latencyMs, tokens: 0, failover: attemptIndex > 0, error: `http_${response.status}` })
+        this.recordRuntimeCall({
+          providerKey: candidate.provider, modelId: candidate.model,
+          success: false, latencyMs, error: `http_${response.status}`,
+        })
         return { done: false, failoverToNext: true, reason: `http_${response.status}` }
       }
 
@@ -2137,6 +2215,10 @@ class RouterRuntime {
         this.recordRouterError(`http_${response.status}`, requestId, { model: key, status: response.status, body: text })
         this.markFailure(key, `HTTP ${response.status}`)
         this.addRequestLog({ request_id: requestId, model: key, status: response.status, latency_ms: latencyMs, tokens: 0, failover: attemptIndex > 0, error: `http_${response.status}` })
+        this.recordRuntimeCall({
+          providerKey: candidate.provider, modelId: candidate.model,
+          success: false, latencyMs, error: `http_${response.status}`,
+        })
         return { done: false, failoverToNext: true, reason: `http_${response.status}` }
       }
 
@@ -2696,6 +2778,18 @@ class RouterRuntime {
         sendJson(res, 200, this.statsPayload(), { 'x-request-id': requestId })
         return
       }
+      // 📖 Runtime telemetry (t3): /stats/runtime returns every model's derived
+      // 📖 snapshot (successRate, avgLatencyMs, avgTokensPerSecond, recentCalls).
+      // 📖 Useful for the Web Dashboard's runtime cards + the upcoming Shift+W
+      // 📖 'Runtime Report' screen in the TUI.
+      if (req.method === 'GET' && url.pathname === '/stats/runtime') {
+        sendJson(res, 200, {
+          ok: true,
+          stats: getRuntimeCacheStats(),
+          models: getAllRuntimeTelemetry(),
+        }, { 'x-request-id': requestId })
+        return
+      }
       if (req.method === 'GET' && url.pathname === '/stats/tokens') {
         sendJson(res, 200, this.tokenTracker.summary(), { 'x-request-id': requestId })
         return
@@ -3177,6 +3271,7 @@ class RouterRuntime {
     if (this.configReloadTimer) clearInterval(this.configReloadTimer)
     if (this.tokenFlushTimer) clearInterval(this.tokenFlushTimer)
     if (this.probeCacheFlushTimer) clearInterval(this.probeCacheFlushTimer)
+    if (this.runtimeTelemetryFlushTimer) clearTimeout(this.runtimeTelemetryFlushTimer)
     for (const timeout of this.probeTimeouts) clearTimeout(timeout)
     const started = Date.now()
     while (this.inFlight > 0 && Date.now() - started < 30000) {
@@ -3184,6 +3279,7 @@ class RouterRuntime {
     }
     this.tokenTracker.flush({ force: true })
     flushProbeCache()  // 📖 t1: persist any pending probe-cache deltas before exit
+    if (this.runtimeTelemetryDirty) flushRuntimeTelemetryStore()  // 📖 t3
     try { this.server?.close() } catch {}
     try { unlinkSync(ROUTER_PID_PATH) } catch {}
     try { unlinkSync(ROUTER_PORT_PATH) } catch {}