free-coding-models 0.3.5 → 0.3.9

package/CHANGELOG.md

@@ -2,6 +2,30 @@
 
 ---
 
+## 0.3.9
+
+### Improved
+- **Enhanced `--premium` flag**: Now applies strict elite-only constraints. Shows only **S/S+** tier models with perfect health (**UP**) and a good verdict (**Perfect**, **Normal**, or **Slow**). Models with 429 errors, auth failures, or poor performance are automatically hidden.
+- **Accurate Token Usage Tracking**: The "Used" column now uses the persistent `token-stats.json` file as the source of truth, providing accurate historical totals instead of only the most recent logs.
+- **Enhanced Log Transparency**: The request log page now always shows the requested model and the actual upstream model (e.g., `llama-3.1-405b → meta/llama-3.1-405b-instruct`) whenever they differ.
+- **Pretty Provider Labels**: The request log page now uses human-readable provider labels (e.g., "NVIDIA NIM", "SambaNova") instead of raw internal keys.
+- **Fixed Tier Filtering Family Logic**: Updated `--tier S` behavior to correctly include both **S** and **S+** models (matching documentation).
+
+---
+
+## 0.3.6
+
+### Added
+- **AI `/testfcm` workflow**: Added a repo-local PTY runner, workflow doc, slash-command prompts, and artifact/report directories so an agent can drive the real TUI, launch a tool, send `hi`, and write a Markdown bug report with evidence.
+- **Mock tool verification path**: Added a tiny fake `crush` binary plus `test:fcm:mock` so maintainers can validate the TUI → launcher → prompt plumbing even when a real coding tool is not installed locally.
+
+### Fixed
+- **`--json` startup crash**: JSON mode now reuses the same provider-aware ping function as the TUI without crashing on `pingModel is not a function`.
+- **Managed endpoint installs no longer resurrect stale disk entries**: install/refresh saves now replace the tracked `endpointInstalls` snapshot so old provider-tool records from another config state do not leak back into the current catalog set.
+- **Favorites persistence is now much harder to break**: favorite toggles now reload the latest disk config before saving, keep the active profile snapshot in sync, and use atomic config writes so pinned rows no longer disappear after unrelated saves or updates.
+- **API key saves no longer clobber the rest of the config**: editing one provider now persists only that provider against the latest on-disk snapshot, preserves rotated extra keys, and stops stale config writes from wiping other saved keys.
+- **Configured Only no longer hides favorites**: starred rows now stay visible and pinned at the top even when the provider has no currently configured key.
+
 ## 0.3.5
 
 ### Fixed

package/README.md

@@ -84,6 +84,7 @@ By Vanessa Depraute
 - **📊 Token usage tracking** — The proxy logs prompt+completion token usage per exact provider/model pair, and the TUI surfaces that history in the `Used` column and the request log overlay.
 - **📜 Request Log Overlay** — Press `X` to inspect recent proxied requests and token usage for exact provider/model pairs.
 - **📋 Changelog Overlay** — Press `N` to browse all versions in an index, then `Enter` to view details for any version with full scroll support
+- **🧪 AI end-to-end workflow** — Run the repo-local `/testfcm` flow to drive the TUI in a PTY, launch one tool, send `hi`, and generate a Markdown bug report plus raw artifacts under `task/`
 - **🛠 MODEL_NOT_FOUND Rotation** — If a specific provider returns a 404 for a model, the TUI intelligently rotates through other available providers for the same model.
 - **🔄 Auto-retry** — Timeout models keep getting retried, nothing is ever "given up on"
 - **🎮 Interactive selection** — Navigate with arrow keys directly in the table, press Enter to act
@@ -199,6 +200,30 @@ free-coding-models
 # Explicitly target OpenCode CLI (TUI + Enter launches OpenCode CLI)
 free-coding-models --opencode
 
+## 📋 CLI Flags (expanded)
+
+The tool now supports a comprehensive set of flags to fine‑tune its behavior. All flags can be combined in any order.
+
+| Flag | Type | Description |
+|------|------|-------------|
+| `--best` | boolean | Show only top‑tier models (A+, S, S+). |
+| `--fiable` | boolean | Run a 10 s reliability analysis and output the most reliable model. |
+| `--json` | boolean | Output results as JSON for scripting/automation. |
+| `--tier <S|A|B|C>` | value | Filter models by tier family (e.g. `S` shows S+ and S). |
+| `--recommend` | boolean | Open Smart Recommend mode immediately on startup. |
+| `--sort <column>` | value | Sort by a specific column (`rank`, `tier`, `origin`, `model`, `ping`, `avg`, `swe`, `ctx`, `condition`, `verdict`, `uptime`, `stability`, `usage`). |
+| `--desc` / `--asc` | boolean | Set sort direction explicitly (descending or ascending). |
+| `--origin <provider>` | value | Filter models by provider origin (e.g. `nvidia`, `groq`). |
+| `--ping-interval <ms>` | value | Override the ping interval in milliseconds (affects live monitoring speed). |
+| `--hide-unconfigured` | boolean | Hide models whose providers have no configured API key. |
+| `--show-unconfigured` | boolean | Show all models regardless of API key configuration. |
+| `--disable-widths-warning` | boolean | Disable the terminal width warning banner. |
+| `--profile <name>` | value | Load a saved configuration profile before startup. |
+| `--no-telemetry` | boolean | Disable anonymous telemetry for this run. |
+| `--clean-proxy`, `--proxy-clean` | boolean | Remove persisted FCM proxy configuration from OpenCode. |
+| `--help`, `-h` | boolean | Print the complete help text and exit. |
+
+These flags are also reflected in the built‑in help (`free-coding-models --help`).
 # Explicitly target OpenCode Desktop (TUI + Enter sets model & opens Desktop app)
 free-coding-models --opencode-desktop
 
@@ -235,6 +260,26 @@ free-coding-models --opencode --best
 free-coding-models --tier S --json
 ```
 
+### AI E2E workflow (`/testfcm`)
+
+For repo-level validation, this project now ships a repeatable AI-driven manual test flow:
+
+- Preferred: `pnpm test:fcm -- --tool crush`
+- Fallback when `pnpm` is unavailable: `npm run test:fcm -- --tool crush`
+- Mock plumbing check: `pnpm test:fcm:mock`
+
+What it does:
+
+1. Copies your current `~/.free-coding-models.json` into an isolated HOME
+2. Runs a `--json` preflight to catch obvious startup regressions
+3. Starts the real TUI in a PTY via the system `expect` command
+4. Presses `Enter` like a user to launch the chosen tool
+5. Sends `hi`
+6. Captures the response, `request-log.jsonl`, daemon logs, and generated tool config
+7. Writes a Markdown report to `task/reports/` and raw artifacts to `task/artifacts/`
+
+The command workflow is documented in [task/TESTFCM-WORKFLOW.md](task/TESTFCM-WORKFLOW.md). Project-local slash commands are also included at [.claude/commands/testfcm.md](.claude/commands/testfcm.md) and [.crush/commands/testfcm.md](.crush/commands/testfcm.md).
+
 ### Choosing the target tool
 
 Running `free-coding-models` with no launcher flag starts in **OpenCode CLI** mode.
@@ -318,7 +363,8 @@ Press **`P`** to open the Settings screen at any time:
 
  Manual update is in the same Settings screen (`P`) under **Maintenance** (Enter to check, Enter again to install when an update is available).
  When a newer npm release is known, the main footer also adds a full-width red warning line with the manual recovery command `npm install -g free-coding-models@latest`.
- Favorites are also persisted in the same config file and survive restarts.
+ Favorites are also persisted in the same config file and survive restarts, app relaunches, and package updates.
+ Favorite rows stay pinned at the top and remain visible even when `Configured Only` mode is enabled.
  The main table now starts in `Configured Only` mode, so if nothing is set up yet you can press `P` and add your first API key immediately.
 
 ### Environment variable overrides
@@ -898,6 +944,25 @@ This script:
 
 ## 📋 API Reference
 
+### 🎁 Premium Flag
+
+The `--premium` flag provides a quick view of only the elite **S/S+ tier** models with perfect health (**UP**) and a good verdict (**Perfect**, **Normal**, or **Slow**). This is useful when you want to focus exclusively on the highest‑quality, most reliable models that are currently available.
+
+```bash
+free-coding-models --premium
+```
+
+What it does under the hood:
+- Sets `tierFilter` to `S` (showing only S+ and S tier models).
+- Filters out any model that is not currently **UP** (hides 429, 410, auth fail, timeouts, etc.).
+- Filters out models with poor verdicts (hides **Spiky**, **Very Slow**, **Overloaded**, **Unstable**, etc.).
+- Forces the sort column to `verdict` with ascending order, so the best‑rated models appear at the top.
+- Leaves other settings untouched, so you can still combine it with flags like `--json` for scripting.
+
+You can combine `--premium` with other flags (e.g., `--json --hide-unconfigured`) to further tailor the output.
+
+---
+
 **Environment variables (override config file):**
 
 | Variable | Description |
@@ -1073,6 +1138,7 @@ Profiles let you save and restore different TUI configurations — useful if you
 **Managing profiles:**
 - Open Settings (**P** key) — scroll down to the **Profiles** section
 - **Enter** on a profile row to load it
+- While a profile is active, edits to favorites and API keys update that active profile immediately
 - **Backspace** on a profile row to delete it
 
 Profiles are stored inside `~/.free-coding-models.json` under the `profiles` key.

package/bin/free-coding-models.js

@@ -99,7 +99,7 @@ import { homedir } from 'os'
 import { join, dirname } from 'path'
 import { MODELS, sources } from '../sources.js'
 import { getAvg, getVerdict, getUptime, getP95, getJitter, getStabilityScore, sortResults, filterByTier, findBestModel, parseArgs, TIER_ORDER, VERDICT_ORDER, TIER_LETTER_MAP, scoreModelForTask, getTopRecommendations, TASK_TYPES, PRIORITY_TYPES, CONTEXT_BUDGETS, formatCtxWindow, labelFromId, getProxyStatusInfo, formatResultsAsJSON } from '../src/utils.js'
-import { loadConfig, saveConfig, getApiKey, getProxySettings, resolveApiKeys, addApiKey, removeApiKey, isProviderEnabled, saveAsProfile, loadProfile, listProfiles, deleteProfile, getActiveProfileName, setActiveProfile, _emptyProfileSettings } from '../src/config.js'
+import { loadConfig, saveConfig, getApiKey, getProxySettings, resolveApiKeys, addApiKey, removeApiKey, isProviderEnabled, saveAsProfile, loadProfile, listProfiles, deleteProfile, getActiveProfileName, setActiveProfile, _emptyProfileSettings, persistApiKeysForProvider } from '../src/config.js'
 import { buildMergedModels } from '../src/model-merger.js'
 import { ProxyServer } from '../src/proxy-server.js'
 import { loadOpenCodeConfig, saveOpenCodeConfig, syncToOpenCode, restoreOpenCodeBackup, cleanupOpenCodeProxyConfig } from '../src/opencode-sync.js'
@@ -203,6 +203,22 @@ async function main() {
     // 📖 User declined auto-fix or it failed — continue anyway, just warned
   }
 
+  // 📖 Apply CLI overrides for settings
+  if (cliArgs.sortColumn) config.settings.sortColumn = cliArgs.sortColumn
+  if (cliArgs.sortDirection) config.settings.sortAsc = cliArgs.sortDirection === 'asc'
+  if (cliArgs.originFilter) config.settings.originFilter = cliArgs.originFilter
+  if (cliArgs.pingInterval) config.settings.pingInterval = cliArgs.pingInterval
+  if (cliArgs.hideUnconfigured) config.settings.hideUnconfiguredModels = true
+  if (cliArgs.showUnconfigured) config.settings.hideUnconfiguredModels = false
+  if (cliArgs.disableWidthsWarning) config.settings.disableWidthsWarning = true
+
+  // 📖 Apply premium mode: show only S‑tier models sorted by verdict
+  if (cliArgs.premiumMode) {
+    config.settings.tierFilter = 'S'
+    config.settings.sortColumn = 'verdict'
+    config.settings.sortAsc = true
+  }
+
   if (cliArgs.cleanProxyMode) {
     const cleaned = cleanupOpenCodeProxyConfig()
     console.log()
@@ -307,7 +323,10 @@ async function main() {
       console.error(chalk.red(`  Unknown profile "${cliArgs.profileName}". Available: ${listProfiles(config).join(', ') || '(none)'}`))
       process.exit(1)
     }
-    saveConfig(config)
+    saveConfig(config, {
+      replaceApiKeys: true,
+      replaceFavorites: true,
+    })
   }
 
   // 📖 Check if any provider has a key — if not, run the first-time setup wizard
@@ -496,8 +515,9 @@ async function main() {
     mode,                         // 📖 'opencode' or 'openclaw' — controls Enter action
     tierFilterMode: 0,            // 📖 Index into TIER_CYCLE (0=All, 1=S+, 2=S, ...)
     originFilterMode: 0,          // 📖 Index into ORIGIN_CYCLE (0=All, then providers)
-hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true || config.settings?.hideUnconfiguredModels === true, // 📖 Hide providers with no configured API key when true.
-      disableWidthsWarning: config.settings?.disableWidthsWarning ?? false, // 📖 Disable widths warning toggle (default off)
+    premiumMode: cliArgs.premiumMode, // 📖 Special elite-only mode: S/S+ only, Health UP only, Perfect/Normal/Slow verdict only.
+    hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true || config.settings?.hideUnconfiguredModels === true, // 📖 Hide providers with no configured API key when true.
+    disableWidthsWarning: config.settings?.disableWidthsWarning ?? false, // 📖 Disable widths warning toggle (default off)
       scrollOffset: 0,              // 📖 First visible model index in viewport
       terminalRows: process.stdout.rows || 24,  // 📖 Current terminal height
       terminalCols: process.stdout.columns || 80, // 📖 Current terminal width
@@ -674,6 +694,52 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
     }
   }
 
+  // 📖 Define pingModel before JSON mode so `--json` can reuse the same provider-aware
+  // 📖 ping path as the interactive TUI without waiting for the PTY/render loop setup.
+  pingModel = async (r) => {
+    state.pendingPings += 1
+    r.isPinging = true
+
+    try {
+      const providerApiKey = getApiKey(state.config, r.providerKey) ?? null
+      const providerUrl = sources[r.providerKey]?.url ?? sources.nvidia.url
+      let { code, ms, quotaPercent } = await ping(providerApiKey, r.modelId, r.providerKey, providerUrl)
+
+      if ((quotaPercent === null || quotaPercent === undefined) && providerApiKey) {
+        const providerQuota = await getProviderQuotaPercentCached(r.providerKey, providerApiKey)
+        if (typeof providerQuota === 'number' && Number.isFinite(providerQuota)) {
+          quotaPercent = providerQuota
+        }
+      }
+
+      r.pings.push({ ms, code })
+
+      if (code === '200') {
+        r.status = 'up'
+      } else if (code === '000') {
+        r.status = 'timeout'
+      } else if (code === '401' || code === '403') {
+        r.status = providerApiKey ? 'auth_error' : 'noauth'
+        r.httpCode = code
+      } else {
+        r.status = 'down'
+        r.httpCode = code
+      }
+
+      if (typeof quotaPercent === 'number' && Number.isFinite(quotaPercent)) {
+        r.usagePercent = quotaPercent
+        for (const sibling of state.results) {
+          if (sibling.providerKey === r.providerKey && (sibling.usagePercent === undefined || sibling.usagePercent === null)) {
+            sibling.usagePercent = quotaPercent
+          }
+        }
+      }
+    } finally {
+      r.isPinging = false
+      state.pendingPings = Math.max(0, state.pendingPings - 1)
+    }
+  }
+
   // 📖 JSON output mode: skip TUI, output results as JSON after initial pings
   if (cliArgs.jsonMode) {
     console.log(chalk.cyan('  ⚡ Pinging models for JSON output...'))
@@ -707,6 +773,17 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
       outputResults = outputResults.filter(r => ['S+', 'S', 'A+'].includes(r.tier))
     }
 
+    // 📖 Apply premium mode filter if specified: elite-only (S/S+, UP, Good Verdict)
+    if (cliArgs.premiumMode) {
+      outputResults = outputResults.filter(r => {
+        const isEliteTier = r.tier === 'S' || r.tier === 'S+'
+        const isHealthUp = r.status === 'up'
+        const verdict = getVerdict(r)
+        const isGoodVerdict = ['Perfect', 'Normal', 'Slow'].includes(verdict)
+        return isEliteTier && isHealthUp && isGoodVerdict
+      })
+    }
+
     // 📖 Sort by avg ping (ascending)
     outputResults = sortResults(outputResults, 'avg', 'asc')
 
@@ -745,20 +822,34 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
     const activeTier = TIER_CYCLE[state.tierFilterMode]
     const activeOrigin = ORIGIN_CYCLE[state.originFilterMode]
     state.results.forEach(r => {
+      // 📖 Favorites stay visible and pinned regardless of configured-only, tier, or provider filters.
+      if (r.isFavorite) {
+        r.hidden = false
+        return
+      }
       const unconfiguredHide = state.hideUnconfiguredModels && !getApiKey(state.config, r.providerKey)
       if (unconfiguredHide) {
         r.hidden = true
         return
       }
-      // 📖 Favorites stay visible regardless of tier/origin filters.
-      if (r.isFavorite) {
-        r.hidden = false
-        return
-      }
       // 📖 Apply both tier and origin filters — model is hidden if it fails either
-      const tierHide = activeTier !== null && r.tier !== activeTier
+      // 📖 TIER_LETTER_MAP is used so --tier S also includes S+ models (tier family behavior).
+      const allowedTiers = (activeTier && TIER_LETTER_MAP[activeTier]) ? TIER_LETTER_MAP[activeTier] : [activeTier]
+      const tierHide = activeTier !== null && !allowedTiers.includes(r.tier)
       const originHide = activeOrigin !== null && r.providerKey !== activeOrigin
       r.hidden = tierHide || originHide
+
+      // 📖 Premium Mode: elite-only constraints (Health UP, Good Verdict, S/S+ only)
+      if (state.premiumMode && !r.hidden) {
+        const isEliteTier = r.tier === 'S' || r.tier === 'S+'
+        const isHealthUp = r.status === 'up'
+        const verdict = getVerdict(r)
+        const isGoodVerdict = ['Perfect', 'Normal', 'Slow'].includes(verdict)
+
+        if (!isEliteTier || !isHealthUp || !isGoodVerdict) {
+          r.hidden = true
+        }
+      }
     })
     return state.results
   }
@@ -826,6 +917,7 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
     resolveApiKeys,
     addApiKey,
     removeApiKey,
+    persistApiKeysForProvider,
     isProviderEnabled,
     listProfiles,
     loadProfile,
@@ -954,60 +1046,6 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
 
   // ── Continuous ping loop — ping all models every N seconds forever ──────────
 
-  // 📖 Single ping function that updates result
-  // 📖 Uses per-provider API key and URL from sources.js
-  // 📖 If no API key is configured, pings without auth — a 401 still tells us latency + server is up
-  pingModel = async (r) => {
-    state.pendingPings += 1
-    r.isPinging = true
-
-    try {
-      const providerApiKey = getApiKey(state.config, r.providerKey) ?? null
-      const providerUrl = sources[r.providerKey]?.url ?? sources.nvidia.url
-      let { code, ms, quotaPercent } = await ping(providerApiKey, r.modelId, r.providerKey, providerUrl)
-
-      if ((quotaPercent === null || quotaPercent === undefined) && providerApiKey) {
-        const providerQuota = await getProviderQuotaPercentCached(r.providerKey, providerApiKey)
-        if (typeof providerQuota === 'number' && Number.isFinite(providerQuota)) {
-          quotaPercent = providerQuota
-        }
-      }
-
-      // 📖 Store ping result as object with ms and code
-      // 📖 ms = actual response time (even for errors like 429)
-      // 📖 code = HTTP status code ('200', '429', '500', '000' for timeout)
-      r.pings.push({ ms, code })
-
-      // 📖 Update status based on latest ping
-      if (code === '200') {
-        r.status = 'up'
-      } else if (code === '000') {
-        r.status = 'timeout'
-      } else if (code === '401' || code === '403') {
-        // 📖 Distinguish "no key configured" from "configured key rejected" so the
-        // 📖 Health column stays honest when Configured Only mode is enabled.
-        r.status = providerApiKey ? 'auth_error' : 'noauth'
-        r.httpCode = code
-      } else {
-        r.status = 'down'
-        r.httpCode = code
-      }
-
-      if (typeof quotaPercent === 'number' && Number.isFinite(quotaPercent)) {
-        r.usagePercent = quotaPercent
-        // Provider-level fallback: apply latest known quota to sibling rows on same provider.
-        for (const sibling of state.results) {
-          if (sibling.providerKey === r.providerKey && (sibling.usagePercent === undefined || sibling.usagePercent === null)) {
-            sibling.usagePercent = quotaPercent
-          }
-        }
-      }
-    } finally {
-      r.isPinging = false
-      state.pendingPings = Math.max(0, state.pendingPings - 1)
-    }
-  }
-
   // 📖 Initial ping of all models
   const initialPing = Promise.all(state.results.map(r => pingModel(r)))
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "free-coding-models",
-  "version": "0.3.5",
+  "version": "0.3.9",
   "description": "Find the fastest coding LLM models in seconds — ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
   "keywords": [
     "nvidia",
@@ -51,7 +51,9 @@
   ],
   "scripts": {
     "start": "node bin/free-coding-models.js",
-    "test": "node --test test/test.js"
+    "test": "node --test test/test.js",
+    "test:fcm": "node scripts/testfcm-runner.mjs",
+    "test:fcm:mock": "node scripts/testfcm-runner.mjs --tool crush --tool-bin-dir test/fixtures/mock-bin"
   },
   "dependencies": {
     "chalk": "^5.4.1"

package/src/cli-help.js

@@ -26,6 +26,14 @@ const ANALYSIS_FLAGS = [
   { flag: '--json', description: 'Output results as JSON for scripts/automation' },
   { flag: '--tier <S|A|B|C>', description: 'Filter models by tier family' },
   { flag: '--recommend', description: 'Open Smart Recommend immediately on startup' },
+  { flag: '--premium', description: 'Show only S/S+ models with perfect health and good verdict' },
+  { flag: '--sort <column>', description: 'Sort by column (rank, tier, origin, model, ping, avg, swe, ctx, condition, verdict, uptime, stability, usage)' },
+  { flag: '--desc | --asc', description: 'Set sort direction (descending or ascending)' },
+  { flag: '--origin <provider>', description: 'Filter models by provider origin' },
+  { flag: '--ping-interval <ms>', description: 'Override ping interval in milliseconds' },
+  { flag: '--hide-unconfigured', description: 'Hide models without configured API keys' },
+  { flag: '--show-unconfigured', description: 'Show all models regardless of API key config' },
+  { flag: '--disable-widths-warning', description: 'Disable terminal width warning' },
 ]
 
 const CONFIG_FLAGS = [