fm-bench 0.6.3 → 0.7.1

package/docs/report-format.md

@@ -1,6 +1,6 @@
 # Report format (schema v1)
 
-Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines.
+Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines. The schema version stayed at `1` in 0.7.0: the new `capabilities`, `metrics`, and per-result `attempts` fields are additive, so older readers and older reports both keep working.
 
 ## Top-level fields
 
@@ -13,12 +13,88 @@ Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** includ
 | `startedAt` / `finishedAt` | ISO-8601 timestamps |
 | `options` | Public run configuration (profile, runs, concurrency, SLOs, tags, note) |
 | `environment` | Host fingerprint: platform, arch, Node, hardware model, CPU, memory, macOS version/build, `fm` help digest, thermal/power snapshot |
+| `capabilities` | What the installed `fm` build exposes (see below) |
+| `metrics` | Per-metric availability and provenance for this run (see below) |
 | `suite` | Derived suite key + fingerprint for apples-to-apples comparison |
 | `prompts` | Prompt ids, text, and token counts |
 | `models` | Discovered models and availability |
 | `summary` | Per-model / per-concurrency roll-up statistics |
 | `results` | Per-run measurements (optional `output` when `--capture-output`) |
 
+## `capabilities`
+
+Detected once per run from `fm --help` and `fm respond --help`.
+
+```json
+{
+  "capabilities": {
+    "bin": "fm",
+    "digest": "921a7839714e3707",
+    "commands": ["available", "chat", "count-tokens", "license", "respond", "schema", "serve"],
+    "models": [{ "name": "system", "description": "On-device Apple Foundation Model" }],
+    "features": {
+      "tokenCounting": true,
+      "tokenCountCommand": "count-tokens",
+      "quota": false,
+      "streaming": true,
+      "modelSelection": true,
+      "instructions": true,
+      "greedy": true,
+      "useCase": true,
+      "guardrails": true,
+      "images": true,
+      "tools": true,
+      "structuredOutput": true,
+      "server": true
+    },
+    "warnings": ["this fm build exposes no quota command, so quota is not reported"]
+  }
+}
+```
+
+`digest` is a short hash of the normalized `fm --help` output, so a report records which CLI surface produced it. Reports from two different `fm` builds are not directly comparable even on the same machine.
+
+## `metrics`
+
+Each entry states how a metric was obtained for this run and whether it was available. `kind` is one of `measured`, `proxy`, `derived`, or `controlled`.
+
+```json
+{
+  "metrics": {
+    "ttft": {
+      "label": "TTFT",
+      "kind": "proxy",
+      "source": "arrival time of the first streamed stdout chunk",
+      "available": true,
+      "unavailableReason": ""
+    },
+    "quota": {
+      "label": "quota",
+      "kind": "measured",
+      "source": "fm quota-usage",
+      "available": false,
+      "unavailableReason": "this fm build exposes no quota command"
+    }
+  }
+}
+```
+
+Consumers should read `available` before trusting a metric. When a metric is unavailable, the corresponding fields are `null` (JSON) or blank (CSV) — never `0`.
+
+Per-run rows in `results` carry:
+
+| Field | Notes |
+|-------|-------|
+| `attempts` | Total `fm` invocations for this measured run, including retries. `1` when no retry was needed. |
+| `ok` | Whether the call produced a response and exited cleanly. |
+| `firstTokenMs`, `generationMs`, `tpotMs` | `null` when the run cannot supply them (no streaming, single-chunk answer, failed run). |
+| `promptTokens`, `outputTokens`, `tokensPerSecond`, `decodeTokensPerSecond`, `prefillTokensPerSecond` | `null` when the `fm` build cannot count tokens. |
+| `error` | Actionable failure text (`timed out after 30000ms`, `fm exited with code 3`, the first actionable line of `fm` stderr). |
+
+## CSV
+
+`--format csv` and `--out runs.csv` write per-run rows. Columns are stable; `attempts` was added in 0.7.0. Text cells that begin with `=`, `+`, `@`, or a non-numeric `-` are prefixed with a single quote so spreadsheets do not execute prompt or model output as a formula.
+
 ## Sharing results
 
 1. **JSON** — best for automation and `fm-bench compare`. Save with `--out bench.json` or `--output-dir reports/`.
@@ -38,6 +114,7 @@ For fair comparison, match:
 - Same `--profile` (or same `--prompt-file`)
 - Same `--runs` and `--warmup`
 - Same concurrency operating points (`--concurrency` or `--sweep-concurrency`)
+- Same `fm` build (compare `capabilities.digest` / `environment.fmHelpDigest`)
 - Same or intentionally changed macOS build (especially for beta-to-beta comparisons)
 - Similar power/thermal state (see `environment.power` and `environment.thermal`)
 
@@ -50,4 +127,4 @@ fm-bench compare before.json after.json --strict   # exit 2 if suites differ
 
 ## Legacy reports
 
-Reports from fm-bench before 0.6.0 remain valid JSON. They lack `schemaVersion`, `reportId`, `suite`, and enriched `environment`. `validate` still checks required fields; `compare` works on `summary` as before.
+Reports from fm-bench before 0.6.0 remain valid JSON. They lack `schemaVersion`, `reportId`, `suite`, `capabilities`, and the enriched `environment`. `validate` still checks required fields, and `compare` works on `summary` as before.

package/docs/supported-platforms.md

@@ -8,8 +8,6 @@
 - **Node.js 20 or newer**.
 - **Apple Intelligence enabled** on the device.
 
-`pcc` (Private Cloud Compute) availability additionally depends on Apple's current eligibility. `fm-bench` reports it as skipped when `fm available --model pcc` reports unavailable.
-
 ## Version enforcement
 
 Commands that launch `fm` benchmarks — the default `run` command and `models` — check the macOS version first and refuse to start on anything older than macOS 27:
@@ -21,6 +19,25 @@ Latest supported: macOS 27.0 or newer (fm is not available on older macOS releas
 
 The process exits with code `2`. This is deliberate: running on an unsupported macOS could never produce a valid benchmark, so the CLI fails fast with the exact version it found and the latest supported macOS version.
 
+The gate guards the **default** `fm` discovery path, because Apple only ships the CLI from macOS 27. If you explicitly provide a binary with `--fm-bin <path>` or `FM_BIN`, the host version is not a constraint and fm-bench proceeds on any platform — the capability probe still exits `2` if that binary is unusable.
+
 Commands that only read local report files — `compare`, `history`, `validate`, `export`, and `legend` — are not gated and work anywhere Node.js runs.
 
 `fm-bench doctor` still runs on unsupported hosts so you can diagnose the environment: it prints the detected macOS version, an explicit `macOS support` line, and the latest supported version.
+
+## Which `fm` builds work
+
+Support is capability-based rather than version-pinned. The CLI probes the installed `fm` and reports what it can measure; a build that exposes different subcommand names or fewer flags still works, with the unsupported metrics reported as unavailable. See [compatibility.md](./compatibility.md) for the detection policy and the verified build table.
+
+## Models
+
+`fm-bench` benchmarks exactly the models the installed `fm` reports — it does not assume that any particular cloud or adapter model exists. On the verified macOS 27.0 build that is the on-device `system` model only. If a build adds models (for example a Private Cloud Compute model), they are discovered automatically and reported with their own availability.
+
+Requesting only models the build cannot run exits with code `2` before any benchmark starts, and `fm-bench models` shows the same reasons without failing:
+
+```text
+fm-bench: No benchmark was run: none of the requested models are usable right now.
+  requested: pcc
+  pcc: not supported by this fm build (supported: system)
+  run "fm-bench models" to see availability and reasons
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fm-bench",
-  "version": "0.6.3",
+  "version": "0.7.1",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {
@@ -14,13 +14,14 @@
     "LICENSE"
   ],
   "scripts": {
-    "test": "node --test",
-    "lint": "node --check bin/fm-bench.js && find src test -name '*.js' -print0 | xargs -0 -n1 node --check",
-    "prepack": "npm test && npm run lint && npm run check:pack",
+    "test": "node --test test/*.test.js",
+    "lint": "node --check bin/fm-bench.js && find src test scripts \\( -name '*.js' -o -name '*.mjs' \\) -print0 | xargs -0 -n1 node --check",
+    "check": "npm run lint && npm test && npm run check:pack",
+    "prepack": "npm run check >&2",
     "release:patch": "npm version patch && git push --follow-tags",
     "release:minor": "npm version minor && git push --follow-tags",
     "release:major": "npm version major && git push --follow-tags",
-    "publish:dry-run": "npm publish --dry-run --access public",
+    "publish:dry-run": "node scripts/publish-dry-run.mjs",
     "check:pack": "node scripts/check-package.mjs"
   },
   "keywords": [

package/src/bench.js

@@ -1,51 +1,85 @@
 import crypto from 'node:crypto';
-import { checkModelAvailability, collectEnvironment, countTokens, discoverModels, getQuotaUsage, respond } from './fm.js';
+import { detectFmCapabilities } from './capabilities.js';
+import { checkModelAvailability, collectEnvironment, countTokens, fmBinaryFromOptions, getQuotaUsage, respond } from './fm.js';
+import { metricAvailability } from './metrics.js';
 import { loadPrompts } from './prompts.js';
 import { finalizeReportPayload } from './schema.js';
 import { summarizeByModel } from './stats.js';
 
 export async function inspectModels(options = {}) {
-  const discovered = await discoverModels(options);
+  const fmBin = fmBinaryFromOptions(options);
+  const capabilities = options.capabilities ?? await detectFmCapabilities(fmBin, options);
+  const discovered = {
+    fmBin,
+    models: capabilities.models,
+    help: capabilities.help,
+    capabilities
+  };
   const requested = normalizeModelSelection(options.models);
   const models = requested.length > 0
-    ? discovered.models.filter((model) => requested.includes(model.name))
+    ? requested.map((name) => discovered.models.find((model) => model.name === name)
+      ?? { name, description: 'Requested model not reported by this fm build' })
     : discovered.models;
 
-  const missing = requested.filter((name) => !models.some((model) => model.name === name));
-  for (const name of missing) {
-    models.push({ name, description: 'Requested model not reported by fm --help' });
-  }
-
   const inspected = [];
   for (const model of models) {
-    const availability = await checkModelAvailability(discovered.fmBin, model.name, options);
-    const quota = await getQuotaUsage(discovered.fmBin, model.name, options);
+    const availability = await checkModelAvailability(discovered.fmBin, model.name, {
+      ...options,
+      capabilities
+    });
+    const quota = await getQuotaUsage(discovered.fmBin, model.name, {
+      ...options,
+      capabilities
+    });
     inspected.push({
       ...model,
       available: availability.available,
-      reason: availability.reason || availability.raw,
-      quota: quota.raw
+      unsupported: Boolean(availability.unsupported),
+      reason: availability.available ? '' : (availability.reason || availability.raw || 'unavailable'),
+      quota: quota.supported ? (quota.raw || quota.reason) : '',
+      quotaSupported: quota.supported,
+      quotaReason: quota.reason
     });
   }
 
   return {
     fmBin: discovered.fmBin,
     models: inspected,
-    help: discovered.help
+    help: discovered.help,
+    capabilities
   };
 }
 
 export async function runBenchmark(options = {}) {
   const startedAt = new Date().toISOString();
+  const fmBin = fmBinaryFromOptions(options);
+
+  notify(options, { type: 'phase', phase: 'capabilities', message: 'probing fm capabilities' });
+  const capabilities = options.capabilities ?? await detectFmCapabilities(fmBin, options);
+  if (!capabilities.ok) {
+    const error = new Error(capabilities.error
+      ? `${capabilities.error}\nInstall Apple's fm CLI (macOS 27+) or point --fm-bin / FM_BIN at a compatible binary.`
+      : `No usable fm commands were found in ${fmBin} --help.`);
+    error.exitCode = 2;
+    throw error;
+  }
+
   notify(options, { type: 'phase', phase: 'prompts', message: 'loading prompts' });
   const prompts = await loadPrompts(options);
   notify(options, { type: 'phase', phase: 'models', message: 'discovering models' });
-  const inspection = await inspectModels(options);
+  const inspection = await inspectModels({ ...options, capabilities });
   const modelStatuses = options.availableOnly
     ? inspection.models.filter((model) => model.available)
     : inspection.models;
   const runnableModels = modelStatuses.filter((model) => model.available);
-  const environment = await collectEnvironment(inspection.fmBin);
+  if (runnableModels.length === 0) {
+    throw noRunnableModelsError(inspection.models, options);
+  }
+  const environment = await collectEnvironment(inspection.fmBin, { ...options, capabilities });
+  const metrics = metricAvailability(capabilities, {
+    stream: options.stream,
+    slo: Boolean(options.sloTtftMs || options.sloE2eMs || options.sloTpotMs)
+  });
   const promptTokenCounts = new Map();
   const concurrencies = normalizeConcurrencySweep(options);
   const totalRuns = concurrencies.length * runnableModels.length * prompts.length * options.runs;
@@ -53,10 +87,13 @@ export async function runBenchmark(options = {}) {
   notify(options, {
     type: 'tokens:start',
     total: prompts.length,
-    message: 'counting prompt tokens'
+    supported: metrics.promptTokens.available,
+    message: metrics.promptTokens.available ? 'counting prompt tokens' : 'token counting unavailable'
   });
   for (const prompt of prompts) {
-    const counted = await countTokens(inspection.fmBin, prompt.prompt, options);
+    const counted = metrics.promptTokens.available
+      ? await countTokens(inspection.fmBin, prompt.prompt, { ...options, capabilities })
+      : { ok: false, count: null };
     promptTokenCounts.set(prompt.id, counted.ok ? counted.count : null);
     notify(options, {
       type: 'tokens:progress',
@@ -80,10 +117,12 @@ export async function runBenchmark(options = {}) {
   for (const [scenarioIndex, concurrency] of concurrencies.entries()) {
     const scenario = await runScenario({
       fmBin: inspection.fmBin,
+      capabilities,
       prompts,
       runnableModels,
       modelStatuses,
       promptTokenCounts,
+      tokenCounting: metrics.outputTokens.available,
       options,
       concurrency,
       scenarioIndex: scenarioIndex + 1,
@@ -123,6 +162,15 @@ export async function runBenchmark(options = {}) {
     finishedAt: new Date().toISOString(),
     options: publicOptions(options),
     environment,
+    capabilities: {
+      bin: capabilities.bin,
+      digest: capabilities.digest,
+      commands: capabilities.commands,
+      models: capabilities.models,
+      features: capabilities.features,
+      warnings: capabilities.warnings
+    },
+    metrics,
     prompts: prompts.map((prompt) => ({
       id: prompt.id,
       prompt: prompt.prompt,
@@ -145,10 +193,12 @@ export async function runBenchmark(options = {}) {
 async function runScenario(context) {
   const {
     fmBin,
+    capabilities,
     prompts,
     runnableModels,
     modelStatuses,
     promptTokenCounts,
+    tokenCounting,
     options,
     concurrency,
     scenarioIndex,
@@ -172,6 +222,7 @@ async function runScenario(context) {
     for (const model of runnableModels) {
       await respond(fmBin, model.name, prompts[0].prompt, {
         ...options,
+        capabilities,
         stream: false
       });
       warmupCompleted += 1;
@@ -206,7 +257,15 @@ async function runScenario(context) {
     total: jobs.length
   });
   await runLimited(jobs, concurrency, async (job) => {
-    const result = await runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt);
+    const result = await runSingleBenchmark({
+      fmBin,
+      capabilities,
+      job,
+      promptTokenCounts,
+      tokenCounting,
+      options,
+      benchmarkStartedAt
+    });
     results.push(result);
     if (onMeasuredResult) onMeasuredResult(result);
     if (!result.ok && options.failFast) {
@@ -229,13 +288,17 @@ async function runScenario(context) {
   };
 }
 
-async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt) {
+async function runSingleBenchmark(context) {
+  const { fmBin, capabilities, job, promptTokenCounts, tokenCounting, options, benchmarkStartedAt } = context;
   const maxAttempts = 1 + Math.max(0, options.retry ?? 0);
   const startOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
   let response;
+  let attempts = 0;
   for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
+    attempts = attempt;
     response = await respond(fmBin, job.model.name, job.prompt.prompt, {
       ...options,
+      capabilities,
       stream: options.stream
     });
     if (response.ok || attempt >= maxAttempts) break;
@@ -243,28 +306,39 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     await new Promise((resolve) => setTimeout(resolve, backoffMs));
   }
   const endOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
-  const outputTokens = response.ok
-    ? await countTokens(fmBin, response.output, options)
-    : { ok: false, count: null };
+
+  const ok = response.ok;
   const seconds = response.durationMs / 1000;
-  const firstTokenMs = response.firstOutputMs;
-  const generationMs = response.ok && firstTokenMs != null
+  const chunks = response.stdoutChunks ?? 0;
+
+  // A single stdout chunk carries the whole answer, so the streamed portion is
+  // not separable: report generation time and TPOT as unavailable rather than
+  // as a near-zero decode phase.
+  const firstTokenMs = ok ? response.firstOutputMs : null;
+  const generationMs = ok && firstTokenMs != null && chunks > 1
     ? Math.max(0, response.durationMs - firstTokenMs)
     : null;
+
+  const outputTokens = ok && tokenCounting
+    ? await countTokens(fmBin, response.output, { ...options, capabilities })
+    : { ok: false, count: null };
   const countedOutputTokens = outputTokens.ok ? outputTokens.count : null;
-  const decodeTokenCount = countedOutputTokens != null ? Math.max(0, countedOutputTokens - 1) : null;
-  const hasDecodeCadence = response.stdoutChunks > 2 && generationMs != null && generationMs > 0 && decodeTokenCount > 0;
-  const tpotMs = hasDecodeCadence ? generationMs / decodeTokenCount : null;
-  const decodeTokensPerSecond = hasDecodeCadence
-    ? decodeTokenCount / (generationMs / 1000)
+  // Two decode tokens is the minimum for an inter-token interval that is not
+  // simply the inverse of a single chunk gap.
+  const decodeTokenCount = countedOutputTokens != null && countedOutputTokens > 2
+    ? countedOutputTokens - 1
     : null;
+  const hasDecodeCadence = generationMs != null && generationMs > 0 && decodeTokenCount != null;
+  const tpotMs = hasDecodeCadence ? generationMs / decodeTokenCount : null;
+  const decodeTokensPerSecond = hasDecodeCadence ? decodeTokenCount / (generationMs / 1000) : null;
+
   const chars = response.output.length;
   const words = response.output.trim() ? response.output.trim().split(/\s+/).length : 0;
-  const promptTokens = promptTokenCounts.get(job.prompt.id);
-  const prefillTokensPerSecond = promptTokens != null && firstTokenMs > 0
+  const promptTokens = promptTokenCounts.get(job.prompt.id) ?? null;
+  const prefillTokensPerSecond = promptTokens != null && firstTokenMs != null && firstTokenMs > 0
     ? promptTokens / (firstTokenMs / 1000)
     : null;
-  const chunkGapsMs = chunkGaps(response.stdoutChunkTimesMs);
+  const chunkGapsMs = ok ? chunkGaps(response.stdoutChunkTimesMs) : [];
   const secondChunkMs = chunkGapsMs.length > 0 ? chunkGapsMs[0] : null;
 
   return {
@@ -272,7 +346,8 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     concurrency: job.concurrency,
     promptId: job.prompt.id,
     run: job.run,
-    ok: response.ok,
+    attempts,
+    ok,
     durationMs: response.durationMs,
     firstTokenMs,
     generationMs,
@@ -284,7 +359,7 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     tokensPerSecond: countedOutputTokens != null && seconds > 0 ? countedOutputTokens / seconds : null,
     decodeTokensPerSecond,
     prefillTokensPerSecond,
-    charsPerSecond: seconds > 0 ? chars / seconds : 0,
+    charsPerSecond: seconds > 0 ? chars / seconds : null,
     startOffsetMs,
     endOffsetMs,
     streamed: response.streamed,
@@ -293,14 +368,14 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     chunkGapsMs,
     chunkGapAvgMs: average(chunkGapsMs),
     chunkGapMaxMs: chunkGapsMs.length > 0 ? Math.max(...chunkGapsMs) : null,
-    outputHash: response.ok ? hashOutput(response.output) : null,
-    good: response.ok ? evaluateSlo({
+    outputHash: ok ? hashOutput(response.output) : null,
+    good: ok ? evaluateSlo({
       firstTokenMs,
       durationMs: response.durationMs,
       tpotMs
     }, options) : false,
     output: options.captureOutput ? response.output : undefined,
-    error: response.ok ? '' : response.stderr || `fm exited with code ${response.code ?? response.signal}`
+    error: ok ? '' : (response.error || `fm exited with code ${response.code ?? response.signal}`)
   };
 }
 
@@ -318,6 +393,23 @@ async function runLimited(items, concurrency, worker, options = {}) {
   await Promise.all(workers);
 }
 
+// A benchmark with nothing to run is a configuration error, not an empty
+// report: say which models were asked for and which ones the build supports.
+function noRunnableModelsError(models, options) {
+  const requested = normalizeModelSelection(options.models);
+  const supported = models.filter((model) => !model.unsupported).map((model) => model.name);
+  const lines = ['No benchmark was run: none of the requested models are usable right now.'];
+  if (requested.length > 0) lines.push(`  requested: ${requested.join(', ')}`);
+  if (supported.length > 0) lines.push(`  models reported by this fm build: ${supported.join(', ')}`);
+  for (const model of models) {
+    if (model.reason) lines.push(`  ${model.name}: ${model.reason}`);
+  }
+  lines.push('  run "fm-bench models" to see availability and reasons');
+  const error = new Error(lines.join('\n'));
+  error.exitCode = 2;
+  return error;
+}
+
 function normalizeModelSelection(models) {
   if (!models) return [];
   const values = Array.isArray(models) ? models : [models];

package/src/capabilities.js

@@ -0,0 +1,188 @@
+// Capability detection for the installed `fm` CLI.
+//
+// fm-bench talks to whatever `fm` build is on the machine, and Apple has
+// changed subcommand names and flags between releases (for example
+// `count-tokens` replaces the older `token-count`). Everything downstream
+// reads this detection result instead of hardcoding subcommand names, so a
+// compatible `fm` build is supported without a code change and an
+// incompatible one is reported instead of producing wrong numbers.
+
+import crypto from 'node:crypto';
+import { stripAnsi } from './ansi.js';
+import { parseAvailabilityList, parseModelsFromHelp } from './fm-help.js';
+import { runProcess } from './process.js';
+
+const SECTION_HEADER = /^\s*[A-Z][A-Z0-9 /-]+\s*$/;
+
+/**
+ * Parse the COMMANDS section of `fm --help`.
+ * @param {string} helpText
+ * @returns {string[]}
+ */
+export function parseCommandsFromHelp(helpText = '') {
+  const lines = stripAnsi(helpText).split(/\r?\n/);
+  const commands = [];
+  let inCommands = false;
+
+  for (const line of lines) {
+    if (/^\s*COMMANDS\s*$/.test(line)) {
+      inCommands = true;
+      continue;
+    }
+    if (inCommands && SECTION_HEADER.test(line)) {
+      inCommands = false;
+    }
+    if (!inCommands) continue;
+
+    const match = line.match(/^\s{2,}([a-z][a-z0-9-]*)\s{2,}\S/);
+    if (match) commands.push(match[1]);
+  }
+
+  return commands;
+}
+
+/**
+ * True when a long flag appears in `fm` help text. Apple prints boolean flags
+ * with a negatable form (`--[no-]stream`), so accept that shape too.
+ * @param {string} helpText
+ * @param {string} flag e.g. "--use-case"
+ */
+export function hasFlagInHelp(helpText, flag) {
+  const clean = stripAnsi(helpText);
+  if (new RegExp(`${escapeRegExp(flag)}\\b`).test(clean)) return true;
+  if (flag.startsWith('--no-')) {
+    return clean.includes(`[no-]${flag.slice('--no-'.length)}`);
+  }
+  return false;
+}
+
+export function helpDigest(helpText = '') {
+  const text = stripAnsi(helpText).trim();
+  if (!text) return null;
+  return crypto.createHash('sha256').update(text).digest('hex').slice(0, 16);
+}
+
+/**
+ * Pick the subcommand this `fm` build exposes for token counting.
+ * @returns {string|null}
+ */
+export function resolveTokenCountCommand(commands = []) {
+  if (commands.includes('count-tokens')) return 'count-tokens';
+  if (commands.includes('token-count')) return 'token-count';
+  return null;
+}
+
+function buildFeatures(helpText, respondHelpText, commands) {
+  const tokenCountCommand = resolveTokenCountCommand(commands);
+  const respondHelp = respondHelpText || '';
+  return {
+    tokenCounting: tokenCountCommand != null,
+    tokenCountCommand,
+    quota: commands.includes('quota-usage'),
+    streaming: hasFlagInHelp(respondHelp, '--no-stream') || hasFlagInHelp(respondHelp, '--stream'),
+    modelSelection: hasFlagInHelp(respondHelp, '--model'),
+    instructions: hasFlagInHelp(respondHelp, '--instructions'),
+    greedy: hasFlagInHelp(respondHelp, '--greedy'),
+    useCase: hasFlagInHelp(respondHelp, '--use-case'),
+    guardrails: hasFlagInHelp(respondHelp, '--guardrails'),
+    images: hasFlagInHelp(respondHelp, '--image'),
+    tools: hasFlagInHelp(respondHelp, '--tool'),
+    structuredOutput: hasFlagInHelp(respondHelp, '--schema'),
+    server: commands.includes('serve')
+  };
+}
+
+function capabilityWarnings(commands, features, models) {
+  const warnings = [];
+  if (!features.tokenCounting) {
+    warnings.push('this fm build exposes no token-counting command, so token counts and token throughput are unavailable');
+  }
+  if (!features.quota) {
+    warnings.push('this fm build exposes no quota command, so quota is not reported');
+  }
+  if (!features.streaming) {
+    warnings.push('this fm build does not document a streaming flag, so TTFT cannot be measured');
+  }
+  if (models.length === 0) {
+    warnings.push('could not discover any models from fm help output');
+  }
+  if (commands.length === 0) {
+    warnings.push('could not read the command list from fm --help');
+  }
+  return warnings;
+}
+
+/**
+ * Probe the installed `fm` binary once and describe what it can do.
+ *
+ * @param {string} fmBin
+ * @param {{ timeoutMs?: number, env?: NodeJS.ProcessEnv, help?: { text: string } }} [options]
+ */
+export async function detectFmCapabilities(fmBin, options = {}) {
+  const timeoutMs = options.timeoutMs ?? 10_000;
+  const env = options.env ?? process.env;
+  let helpText = options.help?.text ?? null;
+
+  if (helpText == null) {
+    const help = await runProcess(fmBin, ['--help'], { timeoutMs, env });
+    if (help.error) {
+      return {
+        ok: false,
+        bin: fmBin,
+        error: `Unable to execute ${fmBin}: ${help.stderr || help.error.message}`,
+        commands: [],
+        models: [],
+        features: buildFeatures('', '', []),
+        digest: null,
+        help: '',
+        warnings: [`cannot execute ${fmBin}`]
+      };
+    }
+    helpText = `${help.stdout}${help.stderr}`;
+  }
+
+  const cleanHelp = stripAnsi(helpText);
+  const commands = parseCommandsFromHelp(cleanHelp);
+
+  let respondHelpText = '';
+  if (commands.includes('respond')) {
+    const respondHelp = await runProcess(fmBin, ['respond', '--help'], { timeoutMs, env });
+    if (!respondHelp.error) respondHelpText = `${respondHelp.stdout}${respondHelp.stderr}`;
+  }
+
+  const features = buildFeatures(cleanHelp, respondHelpText, commands);
+  let models = parseModelsFromHelp(cleanHelp);
+
+  if (models.length === 0 && commands.includes('available')) {
+    models = await discoverModelsFromAvailability(fmBin, { ...options, env });
+  }
+
+  return {
+    ok: commands.length > 0,
+    bin: fmBin,
+    commands,
+    models,
+    features,
+    digest: helpDigest(cleanHelp),
+    help: cleanHelp,
+    warnings: capabilityWarnings(commands, features, models)
+  };
+}
+
+/**
+ * Fallback discovery: ask `fm available` without a model filter and read the
+ * model names it reports. Used when `fm --help` has no MODELS section.
+ */
+async function discoverModelsFromAvailability(fmBin, options = {}) {
+  const result = await runProcess(fmBin, ['available'], {
+    timeoutMs: options.timeoutMs ?? 15_000,
+    env: options.env ?? process.env
+  });
+  if (result.error) return [];
+  return parseAvailabilityList(`${result.stdout}${result.stderr}`)
+    .map((model) => ({ name: model.name, description: '' }));
+}
+
+function escapeRegExp(value) {
+  return String(value).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}