free-coding-models 0.3.5 → 0.3.6

package/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 ---
 
+## 0.3.6
+
+### Added
+- **AI `/testfcm` workflow**: Added a repo-local PTY runner, workflow doc, slash-command prompts, and artifact/report directories so an agent can drive the real TUI, launch a tool, send `hi`, and write a Markdown bug report with evidence.
+- **Mock tool verification path**: Added a tiny fake `crush` binary plus `test:fcm:mock` so maintainers can validate the TUI → launcher → prompt plumbing even when a real coding tool is not installed locally.
+
+### Fixed
+- **`--json` startup crash**: JSON mode now reuses the same provider-aware ping function as the TUI without crashing on `pingModel is not a function`.
+- **Managed endpoint installs no longer resurrect stale disk entries**: install/refresh saves now replace the tracked `endpointInstalls` snapshot so old provider-tool records from another config state do not leak back into the current catalog set.
+- **Favorites persistence is now much harder to break**: favorite toggles now reload the latest disk config before saving, keep the active profile snapshot in sync, and use atomic config writes so pinned rows no longer disappear after unrelated saves or updates.
+- **API key saves no longer clobber the rest of the config**: editing one provider now persists only that provider against the latest on-disk snapshot, preserves rotated extra keys, and stops stale config writes from wiping other saved keys.
+- **Configured Only no longer hides favorites**: starred rows now stay visible and pinned at the top even when the provider has no currently configured key.
+
 ## 0.3.5
 
 ### Fixed

package/README.md

@@ -84,6 +84,7 @@ By Vanessa Depraute
 - **📊 Token usage tracking** — The proxy logs prompt+completion token usage per exact provider/model pair, and the TUI surfaces that history in the `Used` column and the request log overlay.
 - **📜 Request Log Overlay** — Press `X` to inspect recent proxied requests and token usage for exact provider/model pairs.
 - **📋 Changelog Overlay** — Press `N` to browse all versions in an index, then `Enter` to view details for any version with full scroll support
+- **🧪 AI end-to-end workflow** — Run the repo-local `/testfcm` flow to drive the TUI in a PTY, launch one tool, send `hi`, and generate a Markdown bug report plus raw artifacts under `task/`
 - **🛠 MODEL_NOT_FOUND Rotation** — If a specific provider returns a 404 for a model, the TUI intelligently rotates through other available providers for the same model.
 - **🔄 Auto-retry** — Timeout models keep getting retried, nothing is ever "given up on"
 - **🎮 Interactive selection** — Navigate with arrow keys directly in the table, press Enter to act
@@ -235,6 +236,26 @@ free-coding-models --opencode --best
 free-coding-models --tier S --json
 ```
 
+### AI E2E workflow (`/testfcm`)
+
+For repo-level validation, this project now ships a repeatable AI-driven manual test flow:
+
+- Preferred: `pnpm test:fcm -- --tool crush`
+- Fallback when `pnpm` is unavailable: `npm run test:fcm -- --tool crush`
+- Mock plumbing check: `pnpm test:fcm:mock`
+
+What it does:
+
+1. Copies your current `~/.free-coding-models.json` into an isolated HOME
+2. Runs a `--json` preflight to catch obvious startup regressions
+3. Starts the real TUI in a PTY via the system `expect` command
+4. Presses `Enter` like a user to launch the chosen tool
+5. Sends `hi`
+6. Captures the response, `request-log.jsonl`, daemon logs, and generated tool config
+7. Writes a Markdown report to `task/reports/` and raw artifacts to `task/artifacts/`
+
+The command workflow is documented in [task/TESTFCM-WORKFLOW.md](task/TESTFCM-WORKFLOW.md). Project-local slash commands are also included at [.claude/commands/testfcm.md](.claude/commands/testfcm.md) and [.crush/commands/testfcm.md](.crush/commands/testfcm.md).
+
 ### Choosing the target tool
 
 Running `free-coding-models` with no launcher flag starts in **OpenCode CLI** mode.
@@ -318,7 +339,8 @@ Press **`P`** to open the Settings screen at any time:
 
  Manual update is in the same Settings screen (`P`) under **Maintenance** (Enter to check, Enter again to install when an update is available).
  When a newer npm release is known, the main footer also adds a full-width red warning line with the manual recovery command `npm install -g free-coding-models@latest`.
- Favorites are also persisted in the same config file and survive restarts.
+ Favorites are also persisted in the same config file and survive restarts, app relaunches, and package updates.
+ Favorite rows stay pinned at the top and remain visible even when `Configured Only` mode is enabled.
  The main table now starts in `Configured Only` mode, so if nothing is set up yet you can press `P` and add your first API key immediately.
 
 ### Environment variable overrides
@@ -1073,6 +1095,7 @@ Profiles let you save and restore different TUI configurations — useful if you
 **Managing profiles:**
 - Open Settings (**P** key) — scroll down to the **Profiles** section
 - **Enter** on a profile row to load it
+- While a profile is active, edits to favorites and API keys update that active profile immediately
 - **Backspace** on a profile row to delete it
 
 Profiles are stored inside `~/.free-coding-models.json` under the `profiles` key.

package/bin/free-coding-models.js

@@ -99,7 +99,7 @@ import { homedir } from 'os'
 import { join, dirname } from 'path'
 import { MODELS, sources } from '../sources.js'
 import { getAvg, getVerdict, getUptime, getP95, getJitter, getStabilityScore, sortResults, filterByTier, findBestModel, parseArgs, TIER_ORDER, VERDICT_ORDER, TIER_LETTER_MAP, scoreModelForTask, getTopRecommendations, TASK_TYPES, PRIORITY_TYPES, CONTEXT_BUDGETS, formatCtxWindow, labelFromId, getProxyStatusInfo, formatResultsAsJSON } from '../src/utils.js'
-import { loadConfig, saveConfig, getApiKey, getProxySettings, resolveApiKeys, addApiKey, removeApiKey, isProviderEnabled, saveAsProfile, loadProfile, listProfiles, deleteProfile, getActiveProfileName, setActiveProfile, _emptyProfileSettings } from '../src/config.js'
+import { loadConfig, saveConfig, getApiKey, getProxySettings, resolveApiKeys, addApiKey, removeApiKey, isProviderEnabled, saveAsProfile, loadProfile, listProfiles, deleteProfile, getActiveProfileName, setActiveProfile, _emptyProfileSettings, persistApiKeysForProvider } from '../src/config.js'
 import { buildMergedModels } from '../src/model-merger.js'
 import { ProxyServer } from '../src/proxy-server.js'
 import { loadOpenCodeConfig, saveOpenCodeConfig, syncToOpenCode, restoreOpenCodeBackup, cleanupOpenCodeProxyConfig } from '../src/opencode-sync.js'
@@ -307,7 +307,10 @@ async function main() {
       console.error(chalk.red(`  Unknown profile "${cliArgs.profileName}". Available: ${listProfiles(config).join(', ') || '(none)'}`))
       process.exit(1)
     }
-    saveConfig(config)
+    saveConfig(config, {
+      replaceApiKeys: true,
+      replaceFavorites: true,
+    })
   }
 
   // 📖 Check if any provider has a key — if not, run the first-time setup wizard
@@ -674,6 +677,52 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
     }
   }
 
+  // 📖 Define pingModel before JSON mode so `--json` can reuse the same provider-aware
+  // 📖 ping path as the interactive TUI without waiting for the PTY/render loop setup.
+  pingModel = async (r) => {
+    state.pendingPings += 1
+    r.isPinging = true
+
+    try {
+      const providerApiKey = getApiKey(state.config, r.providerKey) ?? null
+      const providerUrl = sources[r.providerKey]?.url ?? sources.nvidia.url
+      let { code, ms, quotaPercent } = await ping(providerApiKey, r.modelId, r.providerKey, providerUrl)
+
+      if ((quotaPercent === null || quotaPercent === undefined) && providerApiKey) {
+        const providerQuota = await getProviderQuotaPercentCached(r.providerKey, providerApiKey)
+        if (typeof providerQuota === 'number' && Number.isFinite(providerQuota)) {
+          quotaPercent = providerQuota
+        }
+      }
+
+      r.pings.push({ ms, code })
+
+      if (code === '200') {
+        r.status = 'up'
+      } else if (code === '000') {
+        r.status = 'timeout'
+      } else if (code === '401' || code === '403') {
+        r.status = providerApiKey ? 'auth_error' : 'noauth'
+        r.httpCode = code
+      } else {
+        r.status = 'down'
+        r.httpCode = code
+      }
+
+      if (typeof quotaPercent === 'number' && Number.isFinite(quotaPercent)) {
+        r.usagePercent = quotaPercent
+        for (const sibling of state.results) {
+          if (sibling.providerKey === r.providerKey && (sibling.usagePercent === undefined || sibling.usagePercent === null)) {
+            sibling.usagePercent = quotaPercent
+          }
+        }
+      }
+    } finally {
+      r.isPinging = false
+      state.pendingPings = Math.max(0, state.pendingPings - 1)
+    }
+  }
+
   // 📖 JSON output mode: skip TUI, output results as JSON after initial pings
   if (cliArgs.jsonMode) {
     console.log(chalk.cyan('  ⚡ Pinging models for JSON output...'))
@@ -745,16 +794,16 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
     const activeTier = TIER_CYCLE[state.tierFilterMode]
     const activeOrigin = ORIGIN_CYCLE[state.originFilterMode]
     state.results.forEach(r => {
+      // 📖 Favorites stay visible and pinned regardless of configured-only, tier, or provider filters.
+      if (r.isFavorite) {
+        r.hidden = false
+        return
+      }
       const unconfiguredHide = state.hideUnconfiguredModels && !getApiKey(state.config, r.providerKey)
       if (unconfiguredHide) {
         r.hidden = true
         return
       }
-      // 📖 Favorites stay visible regardless of tier/origin filters.
-      if (r.isFavorite) {
-        r.hidden = false
-        return
-      }
       // 📖 Apply both tier and origin filters — model is hidden if it fails either
       const tierHide = activeTier !== null && r.tier !== activeTier
       const originHide = activeOrigin !== null && r.providerKey !== activeOrigin
@@ -826,6 +875,7 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
     resolveApiKeys,
     addApiKey,
     removeApiKey,
+    persistApiKeysForProvider,
     isProviderEnabled,
     listProfiles,
     loadProfile,
@@ -954,60 +1004,6 @@ hideUnconfiguredModels: startupProfileSettings?.hideUnconfiguredModels === true
 
   // ── Continuous ping loop — ping all models every N seconds forever ──────────
 
-  // 📖 Single ping function that updates result
-  // 📖 Uses per-provider API key and URL from sources.js
-  // 📖 If no API key is configured, pings without auth — a 401 still tells us latency + server is up
-  pingModel = async (r) => {
-    state.pendingPings += 1
-    r.isPinging = true
-
-    try {
-      const providerApiKey = getApiKey(state.config, r.providerKey) ?? null
-      const providerUrl = sources[r.providerKey]?.url ?? sources.nvidia.url
-      let { code, ms, quotaPercent } = await ping(providerApiKey, r.modelId, r.providerKey, providerUrl)
-
-      if ((quotaPercent === null || quotaPercent === undefined) && providerApiKey) {
-        const providerQuota = await getProviderQuotaPercentCached(r.providerKey, providerApiKey)
-        if (typeof providerQuota === 'number' && Number.isFinite(providerQuota)) {
-          quotaPercent = providerQuota
-        }
-      }
-
-      // 📖 Store ping result as object with ms and code
-      // 📖 ms = actual response time (even for errors like 429)
-      // 📖 code = HTTP status code ('200', '429', '500', '000' for timeout)
-      r.pings.push({ ms, code })
-
-      // 📖 Update status based on latest ping
-      if (code === '200') {
-        r.status = 'up'
-      } else if (code === '000') {
-        r.status = 'timeout'
-      } else if (code === '401' || code === '403') {
-        // 📖 Distinguish "no key configured" from "configured key rejected" so the
-        // 📖 Health column stays honest when Configured Only mode is enabled.
-        r.status = providerApiKey ? 'auth_error' : 'noauth'
-        r.httpCode = code
-      } else {
-        r.status = 'down'
-        r.httpCode = code
-      }
-
-      if (typeof quotaPercent === 'number' && Number.isFinite(quotaPercent)) {
-        r.usagePercent = quotaPercent
-        // Provider-level fallback: apply latest known quota to sibling rows on same provider.
-        for (const sibling of state.results) {
-          if (sibling.providerKey === r.providerKey && (sibling.usagePercent === undefined || sibling.usagePercent === null)) {
-            sibling.usagePercent = quotaPercent
-          }
-        }
-      }
-    } finally {
-      r.isPinging = false
-      state.pendingPings = Math.max(0, state.pendingPings - 1)
-    }
-  }
-
   // 📖 Initial ping of all models
   const initialPing = Promise.all(state.results.map(r => pingModel(r)))
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "free-coding-models",
-  "version": "0.3.5",
+  "version": "0.3.6",
   "description": "Find the fastest coding LLM models in seconds — ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
   "keywords": [
     "nvidia",
@@ -51,7 +51,9 @@
   ],
   "scripts": {
     "start": "node bin/free-coding-models.js",
-    "test": "node --test test/test.js"
+    "test": "node --test test/test.js",
+    "test:fcm": "node scripts/testfcm-runner.mjs",
+    "test:fcm:mock": "node scripts/testfcm-runner.mjs --tool crush --tool-bin-dir test/fixtures/mock-bin"
   },
   "dependencies": {
     "chalk": "^5.4.1"