fm-bench 0.6.2 → 0.7.0

package/docs/report-format.md

@@ -1,6 +1,6 @@
 # Report format (schema v1)
 
-Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines.
+Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines. The schema version stayed at `1` in 0.7.0: the new `capabilities`, `metrics`, and per-result `attempts` fields are additive, so older readers and older reports both keep working.
 
 ## Top-level fields
 
@@ -13,12 +13,88 @@ Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** includ
 | `startedAt` / `finishedAt` | ISO-8601 timestamps |
 | `options` | Public run configuration (profile, runs, concurrency, SLOs, tags, note) |
 | `environment` | Host fingerprint: platform, arch, Node, hardware model, CPU, memory, macOS version/build, `fm` help digest, thermal/power snapshot |
+| `capabilities` | What the installed `fm` build exposes (see below) |
+| `metrics` | Per-metric availability and provenance for this run (see below) |
 | `suite` | Derived suite key + fingerprint for apples-to-apples comparison |
 | `prompts` | Prompt ids, text, and token counts |
 | `models` | Discovered models and availability |
 | `summary` | Per-model / per-concurrency roll-up statistics |
 | `results` | Per-run measurements (optional `output` when `--capture-output`) |
 
+## `capabilities`
+
+Detected once per run from `fm --help` and `fm respond --help`.
+
+```json
+{
+  "capabilities": {
+    "bin": "fm",
+    "digest": "921a7839714e3707",
+    "commands": ["available", "chat", "count-tokens", "license", "respond", "schema", "serve"],
+    "models": [{ "name": "system", "description": "On-device Apple Foundation Model" }],
+    "features": {
+      "tokenCounting": true,
+      "tokenCountCommand": "count-tokens",
+      "quota": false,
+      "streaming": true,
+      "modelSelection": true,
+      "instructions": true,
+      "greedy": true,
+      "useCase": true,
+      "guardrails": true,
+      "images": true,
+      "tools": true,
+      "structuredOutput": true,
+      "server": true
+    },
+    "warnings": ["this fm build exposes no quota command, so quota is not reported"]
+  }
+}
+```
+
+`digest` is a short hash of the normalized `fm --help` output, so a report records which CLI surface produced it. Reports from two different `fm` builds are not directly comparable even on the same machine.
+
+## `metrics`
+
+Each entry states how a metric was obtained for this run and whether it was available. `kind` is one of `measured`, `proxy`, `derived`, or `controlled`.
+
+```json
+{
+  "metrics": {
+    "ttft": {
+      "label": "TTFT",
+      "kind": "proxy",
+      "source": "arrival time of the first streamed stdout chunk",
+      "available": true,
+      "unavailableReason": ""
+    },
+    "quota": {
+      "label": "quota",
+      "kind": "measured",
+      "source": "fm quota-usage",
+      "available": false,
+      "unavailableReason": "this fm build exposes no quota command"
+    }
+  }
+}
+```
+
+Consumers should read `available` before trusting a metric. When a metric is unavailable, the corresponding fields are `null` (JSON) or blank (CSV) — never `0`.
+
+Per-run rows in `results` carry:
+
+| Field | Notes |
+|-------|-------|
+| `attempts` | Total `fm` invocations for this measured run, including retries. `1` when no retry was needed. |
+| `ok` | Whether the call produced a response and exited cleanly. |
+| `firstTokenMs`, `generationMs`, `tpotMs` | `null` when the run cannot supply them (no streaming, single-chunk answer, failed run). |
+| `promptTokens`, `outputTokens`, `tokensPerSecond`, `decodeTokensPerSecond`, `prefillTokensPerSecond` | `null` when the `fm` build cannot count tokens. |
+| `error` | Actionable failure text (`timed out after 30000ms`, `fm exited with code 3`, the first actionable line of `fm` stderr). |
+
+## CSV
+
+`--format csv` and `--out runs.csv` write per-run rows. Columns are stable; `attempts` was added in 0.7.0. Text cells that begin with `=`, `+`, `@`, or a non-numeric `-` are prefixed with a single quote so spreadsheets do not execute prompt or model output as a formula.
+
 ## Sharing results
 
 1. **JSON** — best for automation and `fm-bench compare`. Save with `--out bench.json` or `--output-dir reports/`.
@@ -38,6 +114,7 @@ For fair comparison, match:
 - Same `--profile` (or same `--prompt-file`)
 - Same `--runs` and `--warmup`
 - Same concurrency operating points (`--concurrency` or `--sweep-concurrency`)
+- Same `fm` build (compare `capabilities.digest` / `environment.fmHelpDigest`)
 - Same or intentionally changed macOS build (especially for beta-to-beta comparisons)
 - Similar power/thermal state (see `environment.power` and `environment.thermal`)
 
@@ -50,4 +127,4 @@ fm-bench compare before.json after.json --strict   # exit 2 if suites differ
 
 ## Legacy reports
 
-Reports from fm-bench before 0.6.0 remain valid JSON. They lack `schemaVersion`, `reportId`, `suite`, and enriched `environment`. `validate` still checks required fields; `compare` works on `summary` as before.
+Reports from fm-bench before 0.6.0 remain valid JSON. They lack `schemaVersion`, `reportId`, `suite`, `capabilities`, and the enriched `environment`. `validate` still checks required fields, and `compare` works on `summary` as before.

package/docs/supported-platforms.md

@@ -0,0 +1,43 @@
+# Supported platforms
+
+`fm-bench` benchmarks Apple's `fm` command, so support follows the platforms where Apple ships that CLI.
+
+## Requirements
+
+- **macOS 27.0 or newer** — Apple's `fm` CLI is preinstalled starting with macOS 27. Older macOS releases do not ship `fm`, so `fm-bench` cannot run there.
+- **Node.js 20 or newer**.
+- **Apple Intelligence enabled** on the device.
+
+## Version enforcement
+
+Commands that launch `fm` benchmarks — the default `run` command and `models` — check the macOS version first and refuse to start on anything older than macOS 27:
+
+```text
+fm-bench: unsupported macOS: detected macOS 26.1, but fm-bench requires macOS 27 or newer (Apple's fm CLI is preinstalled there).
+Latest supported: macOS 27.0 or newer (fm is not available on older macOS releases).
+```
+
+The process exits with code `2`. This is deliberate: running on an unsupported macOS could never produce a valid benchmark, so the CLI fails fast with the exact version it found and the latest supported macOS version.
+
+The gate guards the **default** `fm` discovery path, because Apple only ships the CLI from macOS 27. If you explicitly provide a binary with `--fm-bin <path>` or `FM_BIN`, the host version is not a constraint and fm-bench proceeds on any platform — the capability probe still exits `2` if that binary is unusable.
+
+Commands that only read local report files — `compare`, `history`, `validate`, `export`, and `legend` — are not gated and work anywhere Node.js runs.
+
+`fm-bench doctor` still runs on unsupported hosts so you can diagnose the environment: it prints the detected macOS version, an explicit `macOS support` line, and the latest supported version.
+
+## Which `fm` builds work
+
+Support is capability-based rather than version-pinned. The CLI probes the installed `fm` and reports what it can measure; a build that exposes different subcommand names or fewer flags still works, with the unsupported metrics reported as unavailable. See [compatibility.md](./compatibility.md) for the detection policy and the verified build table.
+
+## Models
+
+`fm-bench` benchmarks exactly the models the installed `fm` reports — it does not assume that any particular cloud or adapter model exists. On the verified macOS 27.0 build that is the on-device `system` model only. If a build adds models (for example a Private Cloud Compute model), they are discovered automatically and reported with their own availability.
+
+Requesting only models the build cannot run exits with code `2` before any benchmark starts, and `fm-bench models` shows the same reasons without failing:
+
+```text
+fm-bench: No benchmark was run: none of the requested models are usable right now.
+  requested: pcc
+  pcc: not supported by this fm build (supported: system)
+  run "fm-bench models" to see availability and reasons
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fm-bench",
-  "version": "0.6.2",
+  "version": "0.7.0",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {
@@ -14,13 +14,15 @@
     "LICENSE"
   ],
   "scripts": {
-    "test": "node --test",
-    "lint": "node --check bin/fm-bench.js && find src test -name '*.js' -print0 | xargs -0 -n1 node --check",
-    "prepack": "npm test && npm run lint",
+    "test": "node --test test/*.test.js",
+    "lint": "node --check bin/fm-bench.js && find src test scripts \\( -name '*.js' -o -name '*.mjs' \\) -print0 | xargs -0 -n1 node --check",
+    "check": "npm run lint && npm test && npm run check:pack",
+    "prepack": "npm run check >&2",
     "release:patch": "npm version patch && git push --follow-tags",
     "release:minor": "npm version minor && git push --follow-tags",
     "release:major": "npm version major && git push --follow-tags",
-    "publish:dry-run": "npm publish --dry-run --access public"
+    "publish:dry-run": "npm publish --dry-run --access public",
+    "check:pack": "node scripts/check-package.mjs"
   },
   "keywords": [
     "apple",
@@ -46,5 +48,8 @@
   "publishConfig": {
     "access": "public",
     "registry": "https://registry.npmjs.org/"
-  }
+  },
+  "os": [
+    "darwin"
+  ]
 }

package/src/bench.js

@@ -1,51 +1,85 @@
 import crypto from 'node:crypto';
-import { checkModelAvailability, collectEnvironment, countTokens, discoverModels, getQuotaUsage, respond } from './fm.js';
+import { detectFmCapabilities } from './capabilities.js';
+import { checkModelAvailability, collectEnvironment, countTokens, getQuotaUsage, respond } from './fm.js';
+import { metricAvailability } from './metrics.js';
 import { loadPrompts } from './prompts.js';
 import { finalizeReportPayload } from './schema.js';
 import { summarizeByModel } from './stats.js';
 
 export async function inspectModels(options = {}) {
-  const discovered = await discoverModels(options);
+  const fmBin = options.fmBin || process.env.FM_BIN || 'fm';
+  const capabilities = options.capabilities ?? await detectFmCapabilities(fmBin, options);
+  const discovered = {
+    fmBin,
+    models: capabilities.models,
+    help: capabilities.help,
+    capabilities
+  };
   const requested = normalizeModelSelection(options.models);
   const models = requested.length > 0
-    ? discovered.models.filter((model) => requested.includes(model.name))
+    ? requested.map((name) => discovered.models.find((model) => model.name === name)
+      ?? { name, description: 'Requested model not reported by this fm build' })
     : discovered.models;
 
-  const missing = requested.filter((name) => !models.some((model) => model.name === name));
-  for (const name of missing) {
-    models.push({ name, description: 'Requested model not reported by fm --help' });
-  }
-
   const inspected = [];
   for (const model of models) {
-    const availability = await checkModelAvailability(discovered.fmBin, model.name, options);
-    const quota = await getQuotaUsage(discovered.fmBin, model.name, options);
+    const availability = await checkModelAvailability(discovered.fmBin, model.name, {
+      ...options,
+      capabilities
+    });
+    const quota = await getQuotaUsage(discovered.fmBin, model.name, {
+      ...options,
+      capabilities
+    });
     inspected.push({
       ...model,
       available: availability.available,
-      reason: availability.reason || availability.raw,
-      quota: quota.raw
+      unsupported: Boolean(availability.unsupported),
+      reason: availability.available ? '' : (availability.reason || availability.raw || 'unavailable'),
+      quota: quota.supported ? (quota.raw || quota.reason) : '',
+      quotaSupported: quota.supported,
+      quotaReason: quota.reason
     });
   }
 
   return {
     fmBin: discovered.fmBin,
     models: inspected,
-    help: discovered.help
+    help: discovered.help,
+    capabilities
   };
 }
 
 export async function runBenchmark(options = {}) {
   const startedAt = new Date().toISOString();
+  const fmBin = options.fmBin || process.env.FM_BIN || 'fm';
+
+  notify(options, { type: 'phase', phase: 'capabilities', message: 'probing fm capabilities' });
+  const capabilities = options.capabilities ?? await detectFmCapabilities(fmBin, options);
+  if (!capabilities.ok) {
+    const error = new Error(capabilities.error
+      ? `${capabilities.error}\nInstall Apple's fm CLI (macOS 27+) or point --fm-bin / FM_BIN at a compatible binary.`
+      : `No usable fm commands were found in ${fmBin} --help.`);
+    error.exitCode = 2;
+    throw error;
+  }
+
   notify(options, { type: 'phase', phase: 'prompts', message: 'loading prompts' });
   const prompts = await loadPrompts(options);
   notify(options, { type: 'phase', phase: 'models', message: 'discovering models' });
-  const inspection = await inspectModels(options);
+  const inspection = await inspectModels({ ...options, capabilities });
   const modelStatuses = options.availableOnly
     ? inspection.models.filter((model) => model.available)
     : inspection.models;
   const runnableModels = modelStatuses.filter((model) => model.available);
-  const environment = await collectEnvironment(inspection.fmBin);
+  if (runnableModels.length === 0) {
+    throw noRunnableModelsError(inspection.models, options);
+  }
+  const environment = await collectEnvironment(inspection.fmBin, { ...options, capabilities });
+  const metrics = metricAvailability(capabilities, {
+    stream: options.stream,
+    slo: Boolean(options.sloTtftMs || options.sloE2eMs || options.sloTpotMs)
+  });
   const promptTokenCounts = new Map();
   const concurrencies = normalizeConcurrencySweep(options);
   const totalRuns = concurrencies.length * runnableModels.length * prompts.length * options.runs;
@@ -53,10 +87,13 @@ export async function runBenchmark(options = {}) {
   notify(options, {
     type: 'tokens:start',
     total: prompts.length,
-    message: 'counting prompt tokens'
+    supported: metrics.promptTokens.available,
+    message: metrics.promptTokens.available ? 'counting prompt tokens' : 'token counting unavailable'
   });
   for (const prompt of prompts) {
-    const counted = await countTokens(inspection.fmBin, prompt.prompt, options);
+    const counted = metrics.promptTokens.available
+      ? await countTokens(inspection.fmBin, prompt.prompt, { ...options, capabilities })
+      : { ok: false, count: null };
     promptTokenCounts.set(prompt.id, counted.ok ? counted.count : null);
     notify(options, {
       type: 'tokens:progress',
@@ -80,10 +117,12 @@ export async function runBenchmark(options = {}) {
   for (const [scenarioIndex, concurrency] of concurrencies.entries()) {
     const scenario = await runScenario({
       fmBin: inspection.fmBin,
+      capabilities,
       prompts,
       runnableModels,
       modelStatuses,
       promptTokenCounts,
+      tokenCounting: metrics.outputTokens.available,
       options,
       concurrency,
       scenarioIndex: scenarioIndex + 1,
@@ -123,6 +162,15 @@ export async function runBenchmark(options = {}) {
     finishedAt: new Date().toISOString(),
     options: publicOptions(options),
     environment,
+    capabilities: {
+      bin: capabilities.bin,
+      digest: capabilities.digest,
+      commands: capabilities.commands,
+      models: capabilities.models,
+      features: capabilities.features,
+      warnings: capabilities.warnings
+    },
+    metrics,
     prompts: prompts.map((prompt) => ({
       id: prompt.id,
       prompt: prompt.prompt,
@@ -145,10 +193,12 @@ export async function runBenchmark(options = {}) {
 async function runScenario(context) {
   const {
     fmBin,
+    capabilities,
     prompts,
     runnableModels,
     modelStatuses,
     promptTokenCounts,
+    tokenCounting,
     options,
     concurrency,
     scenarioIndex,
@@ -172,6 +222,7 @@ async function runScenario(context) {
     for (const model of runnableModels) {
       await respond(fmBin, model.name, prompts[0].prompt, {
         ...options,
+        capabilities,
         stream: false
       });
       warmupCompleted += 1;
@@ -206,7 +257,15 @@ async function runScenario(context) {
     total: jobs.length
   });
   await runLimited(jobs, concurrency, async (job) => {
-    const result = await runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt);
+    const result = await runSingleBenchmark({
+      fmBin,
+      capabilities,
+      job,
+      promptTokenCounts,
+      tokenCounting,
+      options,
+      benchmarkStartedAt
+    });
     results.push(result);
     if (onMeasuredResult) onMeasuredResult(result);
     if (!result.ok && options.failFast) {
@@ -229,13 +288,17 @@ async function runScenario(context) {
   };
 }
 
-async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt) {
+async function runSingleBenchmark(context) {
+  const { fmBin, capabilities, job, promptTokenCounts, tokenCounting, options, benchmarkStartedAt } = context;
   const maxAttempts = 1 + Math.max(0, options.retry ?? 0);
   const startOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
   let response;
+  let attempts = 0;
   for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
+    attempts = attempt;
     response = await respond(fmBin, job.model.name, job.prompt.prompt, {
       ...options,
+      capabilities,
       stream: options.stream
     });
     if (response.ok || attempt >= maxAttempts) break;
@@ -243,28 +306,39 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     await new Promise((resolve) => setTimeout(resolve, backoffMs));
   }
   const endOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
-  const outputTokens = response.ok
-    ? await countTokens(fmBin, response.output, options)
-    : { ok: false, count: null };
+
+  const ok = response.ok;
   const seconds = response.durationMs / 1000;
-  const firstTokenMs = response.firstOutputMs;
-  const generationMs = response.ok && firstTokenMs != null
+  const chunks = response.stdoutChunks ?? 0;
+
+  // A single stdout chunk carries the whole answer, so the streamed portion is
+  // not separable: report generation time and TPOT as unavailable rather than
+  // as a near-zero decode phase.
+  const firstTokenMs = ok ? response.firstOutputMs : null;
+  const generationMs = ok && firstTokenMs != null && chunks > 1
     ? Math.max(0, response.durationMs - firstTokenMs)
     : null;
+
+  const outputTokens = ok && tokenCounting
+    ? await countTokens(fmBin, response.output, { ...options, capabilities })
+    : { ok: false, count: null };
   const countedOutputTokens = outputTokens.ok ? outputTokens.count : null;
-  const decodeTokenCount = countedOutputTokens != null ? Math.max(0, countedOutputTokens - 1) : null;
-  const hasDecodeCadence = response.stdoutChunks > 2 && generationMs != null && generationMs > 0 && decodeTokenCount > 0;
-  const tpotMs = hasDecodeCadence ? generationMs / decodeTokenCount : null;
-  const decodeTokensPerSecond = hasDecodeCadence
-    ? decodeTokenCount / (generationMs / 1000)
+  // Two decode tokens is the minimum for an inter-token interval that is not
+  // simply the inverse of a single chunk gap.
+  const decodeTokenCount = countedOutputTokens != null && countedOutputTokens > 2
+    ? countedOutputTokens - 1
     : null;
+  const hasDecodeCadence = generationMs != null && generationMs > 0 && decodeTokenCount != null;
+  const tpotMs = hasDecodeCadence ? generationMs / decodeTokenCount : null;
+  const decodeTokensPerSecond = hasDecodeCadence ? decodeTokenCount / (generationMs / 1000) : null;
+
   const chars = response.output.length;
   const words = response.output.trim() ? response.output.trim().split(/\s+/).length : 0;
-  const promptTokens = promptTokenCounts.get(job.prompt.id);
-  const prefillTokensPerSecond = promptTokens != null && firstTokenMs > 0
+  const promptTokens = promptTokenCounts.get(job.prompt.id) ?? null;
+  const prefillTokensPerSecond = promptTokens != null && firstTokenMs != null && firstTokenMs > 0
     ? promptTokens / (firstTokenMs / 1000)
     : null;
-  const chunkGapsMs = chunkGaps(response.stdoutChunkTimesMs);
+  const chunkGapsMs = ok ? chunkGaps(response.stdoutChunkTimesMs) : [];
   const secondChunkMs = chunkGapsMs.length > 0 ? chunkGapsMs[0] : null;
 
   return {
@@ -272,7 +346,8 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     concurrency: job.concurrency,
     promptId: job.prompt.id,
     run: job.run,
-    ok: response.ok,
+    attempts,
+    ok,
     durationMs: response.durationMs,
     firstTokenMs,
     generationMs,
@@ -284,7 +359,7 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     tokensPerSecond: countedOutputTokens != null && seconds > 0 ? countedOutputTokens / seconds : null,
     decodeTokensPerSecond,
     prefillTokensPerSecond,
-    charsPerSecond: seconds > 0 ? chars / seconds : 0,
+    charsPerSecond: seconds > 0 ? chars / seconds : null,
     startOffsetMs,
     endOffsetMs,
     streamed: response.streamed,
@@ -293,14 +368,14 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     chunkGapsMs,
     chunkGapAvgMs: average(chunkGapsMs),
     chunkGapMaxMs: chunkGapsMs.length > 0 ? Math.max(...chunkGapsMs) : null,
-    outputHash: response.ok ? hashOutput(response.output) : null,
-    good: response.ok ? evaluateSlo({
+    outputHash: ok ? hashOutput(response.output) : null,
+    good: ok ? evaluateSlo({
       firstTokenMs,
       durationMs: response.durationMs,
       tpotMs
     }, options) : false,
     output: options.captureOutput ? response.output : undefined,
-    error: response.ok ? '' : response.stderr || `fm exited with code ${response.code ?? response.signal}`
+    error: ok ? '' : (response.error || `fm exited with code ${response.code ?? response.signal}`)
   };
 }
 
@@ -318,6 +393,23 @@ async function runLimited(items, concurrency, worker, options = {}) {
   await Promise.all(workers);
 }
 
+// A benchmark with nothing to run is a configuration error, not an empty
+// report: say which models were asked for and which ones the build supports.
+function noRunnableModelsError(models, options) {
+  const requested = normalizeModelSelection(options.models);
+  const supported = models.filter((model) => !model.unsupported).map((model) => model.name);
+  const lines = ['No benchmark was run: none of the requested models are usable right now.'];
+  if (requested.length > 0) lines.push(`  requested: ${requested.join(', ')}`);
+  if (supported.length > 0) lines.push(`  models reported by this fm build: ${supported.join(', ')}`);
+  for (const model of models) {
+    if (model.reason) lines.push(`  ${model.name}: ${model.reason}`);
+  }
+  lines.push('  run "fm-bench models" to see availability and reasons');
+  const error = new Error(lines.join('\n'));
+  error.exitCode = 2;
+  return error;
+}
+
 function normalizeModelSelection(models) {
   if (!models) return [];
   const values = Array.isArray(models) ? models : [models];