fm-bench 0.6.3 → 0.7.0

package/README.md

@@ -1,6 +1,7 @@
 # fm-bench
 
 [![CI](https://github.com/devinoldenburg/fm-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/devinoldenburg/fm-bench/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/fm-bench.svg)](https://www.npmjs.com/package/fm-bench)
 
 Benchmark Apple's `fm` command on macOS 27+.
 
@@ -8,7 +9,7 @@ Measure latency, throughput, streaming smoothness, stability, and goodput across
 
 ## Why fm-bench exists
 
-Apple's Foundation Models can run on-device, through Private Cloud Compute, and through model adapters. That makes raw model quality only half the story.
+Apple's Foundation Models run on-device, and the `fm` CLI is the terminal and script interface to them. That makes raw model quality only half the story.
 
 For real apps, the important questions are:
 
@@ -16,10 +17,12 @@ For real apps, the important questions are:
 - Does streaming stay smooth?
 - How stable is latency over repeated runs?
 - What happens under concurrency?
-- Which model/hardware pair meets an interactive SLO?
+- Does this Mac meet an interactive SLO?
 
 `fm-bench` answers those questions with repeatable local benchmarks. Think of it as GeekBench for Apple Foundation Models — run it, get numbers, compare across hardware, models, and macOS updates.
 
+It is deliberately honest about what it can measure: the installed `fm` is probed for its actual capabilities, and any metric that build cannot supply is reported as unavailable rather than estimated. See [docs/compatibility.md](docs/compatibility.md).
+
 ## Quick Start
 
 ```sh
@@ -27,22 +30,30 @@ npm install -g fm-bench
 fm-bench
 ```
 
-One command discovers your models, runs the standard prompt suite, and prints a full benchmark report:
+One command discovers your models, runs the standard prompt suite, and prints a full benchmark report (real output from a macOS 27.0 Apple M5 Pro):
 
 ```text
-fm-bench 0.6.3 | darwin/arm64 | fm
-prompts 3 | runs 5 | concurrency 1 | stream on | measured 15 | failed 0 | skipped 0 | elapsed 38.20s
-
-┌───┬────────┬────────┬─────────┬──────┬──────┬──────────┬──────┬──────────┬─────┬─────┐
-│ C │ MODEL  │ STATUS │ OK/RUNS │ SUCC │ GOOD │ GOOD RPS │ TTFT │ E2E P95  │ SYS │ CV  │
-├───┼────────┼────────┼─────────┼──────┼──────┼──────────┼──────┼──────────┼─────┼─────┤
-│ 1 │ system │ ok     │   15/15 │ 100% │  93% │      0.4 │ 318ms│    3.20s │  42 │ 12% │
-└───┴────────┴────────┴─────────┴──────┴──────┴──────────┴──────┴──────────┴─────┴─────┘
+fm-bench 0.7.0 | darwin/arm64 | fm
+prompts 3 | runs 3 | concurrency 1 | stream on | measured 9 | failed 0 | skipped 0 | elapsed 6.37s | SLO
+TTFT<=1.00s,E2E<=3.00s
+unavailable: quota unavailable — this fm build exposes no quota command
+
+┌───┬────────┬────────┬─────┬──────┬──────────┬───────┬───────┬─────────┬────────┬───────┬─────┬──────┐
+│ C │ MODEL  │ STATUS │ OK  │ GOOD │ GOOD RPS │ TTFT  │ E2E   │ E2E P95 │ USER/S │ SYS/S │ CV  │ NOTE │
+├───┼────────┼────────┼─────┼──────┼──────────┼───────┼───────┼─────────┼────────┼───────┼─────┼──────┤
+│ 1 │ system │ ok     │ 9/9 │ 100% │      1.7 │ 377ms │ 487ms │   749ms │   34.0 │  32.8 │ 25% │      │
+└───┴────────┴────────┴─────┴──────┴──────────┴───────┴───────┴─────────┴────────┴───────┴─────┴──────┘
+
+┌───┬────────┬────────┬─────────┬───────────┬──────────┬───────────┬─────────┬────────┐
+│ C │ MODEL  │ IN AVG │ OUT AVG │ PREFILL/S │ DECODE/S │ CHUNK P95 │ E2E P99 │ REPEAT │
+├───┼────────┼────────┼─────────┼───────────┼──────────┼───────────┼─────────┼────────┤
+│ 1 │ system │     13 │      20 │      35.9 │      113 │     197ms │   749ms │   100% │
+└───┴────────┴────────┴─────────┴───────────┴──────────┴───────────┴─────────┴────────┘
 ```
 
 Default profile is `standard` (3 prompts). Use `--runs 5`, `--sweep-concurrency 1,2`, or `--profile client` for heavier suites.
 
-Wide terminals add TTFT P95, TPOT, decode/prefill throughput, chunk-gap smoothness, and 95% CI columns. Narrow terminals switch to compact model cards automatically.
+Wide terminals add TTFT P95, TPOT, and RPS columns; medium terminals tighten the table; narrow terminals switch to compact model cards automatically. `--width <n>` previews any layout.
 
 ## Install
 
@@ -70,13 +81,16 @@ fm-bench doctor   # verify your setup
 | Command | What it does |
 |---------|-------------|
 | `fm-bench` | Run the full benchmark (default) |
-| `fm-bench models` | List discovered models, availability, and quota |
-| `fm-bench compare <a.json> <b.json>` | Regression diff with suite/hardware warnings; `--strict` for CI |
+| `fm-bench models` | Show the detected `fm` capabilities, discovered models, availability, and quota when supported |
+| `fm-bench compare <a.json> <b.json>` | Regression diff with suite/hardware/macOS warnings; `--strict` exits 2 when suites differ |
 | `fm-bench history [dir]` | Trend table from saved reports (sorted by time, tags visible) |
-| `fm-bench validate <report.json>` | Verify report JSON (schema v1) before sharing |
+| `fm-bench validate <report.json>` | Verify report JSON (schema v1) before sharing; `--json` for CI |
 | `fm-bench export <report.json>` | Standalone HTML report with embedded JSON |
-| `fm-bench legend` | Definitions for every table column and color rule |
-| `fm-bench doctor` | Environment check: Node, macOS, `fm`, CPU, memory, thermals, battery |
+| `fm-bench legend` | Definitions, provenance (measured/proxy/derived), and color rules for every column |
+| `fm-bench doctor` | Environment and `fm` capability check; `--json` for scripts |
+| `fm-bench metrics` | Alias for `legend` |
+
+Exit codes: `0` success, `1` operational failure (failed `--ci` gate, invalid reports), `2` usage or environment error (bad flags, unsupported macOS, unusable `fm`, no runnable model). Interrupting a run with Ctrl+C terminates in-flight `fm` processes.
 
 ## Common Recipes
 
@@ -84,15 +98,12 @@ fm-bench doctor   # verify your setup
 # Quick smoke test
 fm-bench --profile quick
 
-# Standard 5-run benchmark with SLO budgets
-fm-bench --runs 5 --slo-ttft-ms 750 --slo-e2e-ms 4000
+# Standard 5-run benchmark with SLO budgets and warmups
+fm-bench --runs 5 --warmup 1 --slo-ttft-ms 750 --slo-e2e-ms 4000
 
 # Sweep concurrency to find your throughput ceiling
 fm-bench --sweep-concurrency 1,2,4 --runs 3
 
-# Stress both on-device and PCC models
-fm-bench --models system,pcc --runs 3 --profile stress
-
 # Reasoning and coding workloads
 fm-bench --profile reasoning --runs 5
 fm-bench --profile coding --runs 3 --histogram
@@ -103,7 +114,7 @@ fm-bench --output-dir reports/ --tag after-update --export-html
 fm-bench validate reports/*.json
 fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json --strict
 
-# Fail CI when SLOs regress
+# Fail CI when SLOs regress or any run fails
 fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5
 
 # Save JSON for automation
@@ -117,9 +128,9 @@ fm-bench --format csv --out bench.csv
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `-m, --models <list>` | all | Comma-separated or repeated model names |
+| `-m, --models <list>` | discovered | Comma-separated or repeated model names |
 | `-r, --runs <n>` | 1 | Measured runs per prompt/model |
-| `--warmup <n>` | 0 | Warmup runs per model before measurement |
+| `--warmup <n>` | 0 | Unmeasured warmup runs per model before measurement |
 | `-c, --concurrency <n>` | 1 | Parallel `fm` processes |
 | `--sweep-concurrency <list>` | — | Separate operating points, e.g. `1,2,4` |
 | `--request-rate <rps>` | — | Pace request starts at a target rate |
@@ -129,7 +140,9 @@ fm-bench --format csv --out bench.csv
 | `--profile <name>` | standard | Built-in prompt suite (see Profiles) |
 | `-p, --prompt <text>` | — | Custom prompt, repeatable |
 | `--prompt-file <file>` | — | JSON, JSONL, or blank-line separated prompts |
-| `-i, --instructions <text>` | — | Passed to `fm respond` |
+| `-i, --instructions <text>` | — | Passed to `fm respond` when the build supports it |
+| `--use-case <case>` | — | System model use case, when supported |
+| `--guardrails <level>` | — | System model guardrail level, when supported |
 
 **Quality Gates**
 
@@ -159,11 +172,15 @@ fm-bench --format csv --out bench.csv
 
 | Flag | Description |
 |------|-------------|
+| `--no-stream` | Disable streaming; TTFT and decode metrics are reported as unavailable |
+| `--greedy` / `--no-greedy` | Request or omit greedy sampling (default: greedy) |
+| `--available-only` | Hide unavailable discovered models |
 | `--color` / `--no-color` | Force or disable ANSI colors (auto on TTYs) |
 | `--ascii` | Plain ASCII table borders instead of Unicode |
 | `--compact` | Force narrow terminal layout |
 | `--width <n>` | Render as if the terminal is `n` columns wide |
 | `--progress` / `--no-progress` | Force or disable the live progress line |
+| `--fm-bin <path>` | `fm` binary to execute (default: `FM_BIN` or `fm`) |
 
 ## Prompt Profiles
 
@@ -181,9 +198,38 @@ Nine built-in suites, choose the one that matches your use case:
 | `coding` | 5 | Code review, refactoring, algorithms, system design |
 | `creative` | 5 | Product copy, analogies, commit messages, docs |
 
+## Metrics
+
+**Latency** — TTFT (p50/p95), E2E (p50/p95/p99), TPOT, 95% confidence interval, coefficient of variation (CV).
+
+**Throughput** — prefill tokens/s, decode tokens/s, output tokens/s per request, aggregate system tokens/s, requests per second.
+
+**Streaming quality** — second-chunk delay, chunk-gap p95, captured from stdout chunk arrival during streaming runs.
+
+**Reliability** — success rate, goodput rate and RPS against SLO budgets, repeatability (most common output hash frequency across repeated runs).
+
+Every metric is labelled by provenance in JSON and in `fm-bench legend`:
+
+- **measured** — process wall clock, exit codes, chunk arrival, `fm` token counts
+- **proxy** — TTFT and chunk gaps (chunk granularity, not token timestamps); prefill tokens/s
+- **derived** — TPOT, decode tokens/s, throughput, CV, confidence intervals, goodput
+
+Token counts come from the `fm` build's own token-counting command (`count-tokens`, or `token-count` on older builds). If the build cannot count tokens, token metrics render as `-`, JSON carries `null`, and `metrics.promptTokens.available` is `false`. Spread statistics (`CV`, 95% CI) need at least two successful samples; with one sample they are unavailable rather than `0`. See [docs/methodology.md](docs/methodology.md).
+
+## fm compatibility
+
+`fm-bench` probes `fm --help` and `fm respond --help` once per run and adapts:
+
+- Subcommand names (`count-tokens` vs legacy `token-count`) are detected, not hardcoded.
+- Flags the build does not document (`--stream`, `--use-case`, `--guardrails`, `--model`) are not passed through.
+- Unsupported models are rejected before a benchmark starts, with the supported list in the error.
+- Raw `fm` argument-error text never reaches reports, tables, or JSON.
+
+`fm-bench doctor` shows exactly what was detected. Policy and the verified-build table: [docs/compatibility.md](docs/compatibility.md).
+
 ## Sharing and comparing results
 
-Reports from 0.6.0+ include **schema v1**: `reportId`, hardware fingerprint, and a **suite key** so you can tell if two JSON files used the same prompts and run settings. See [docs/report-format.md](docs/report-format.md).
+Reports from 0.6.0+ include **schema v1**: `reportId`, hardware fingerprint, a **suite key** so you can tell if two JSON files used the same prompts and run settings, plus the detected `fm` capabilities and per-metric availability. See [docs/report-format.md](docs/report-format.md).
 
 - Share **HTML** with teammates who do not use the CLI: `fm-bench export bench.json -o bench.html`
 - Gate uploads in CI: `fm-bench validate artifact.json`
@@ -194,33 +240,24 @@ Reports from 0.6.0+ include **schema v1**: `reportId`, hardware fingerprint, and
 Track performance across macOS updates, model changes, or hardware swaps:
 
 ```sh
-# Before
 fm-bench --profile coding --runs 5 --output-dir reports/ --tag before
-
-# After the change
 fm-bench --profile coding --runs 5 --output-dir reports/ --tag after
-
-# See what changed
 fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json
-```
-
-The compare output shows each model/concurrency row with the before value, a color-coded percent delta (green = improvement, red = regression), and the after value — for TTFT, E2E, TPOT, tokens/s, RPS, success rate, and CV.
-
-```sh
-# View the full trend over time
 fm-bench history reports/
 ```
 
+The compare output shows each model/concurrency row with the before value, a color-coded percent delta (green = improvement, red = regression), and the after value — for TTFT, E2E, TPOT, tokens/s, RPS, success rate, and CV. It warns when hardware, macOS version/build, or the prompt suite differ.
+
 ## CI Integration
 
 Gate deployments or model updates on benchmark quality:
 
 ```sh
-# Fails with exit code 1 if TTFT > 750ms or E2E > 4s on any run
+# Fails with exit code 1 if TTFT > 750ms, E2E > 4s, or any run fails
 fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5
 ```
 
-Prints `fm-bench ci: PASS` or `fm-bench ci: FAIL — <reason>` to stderr. Designed for GitHub Actions, Buildkite, or any shell-based pipeline.
+Prints `fm-bench ci: PASS` or `fm-bench ci: FAIL — <reason>` to stderr. Designed for GitHub Actions, Buildkite, or any shell-based pipeline. `--json` and `--csv` write only data to stdout; progress and diagnostics go to stderr.
 
 ## Prompt Files
 
@@ -240,36 +277,18 @@ JSONL:
 {"id":"latency","prompt":"Explain p95 latency in one sentence."}
 ```
 
-Plain text files are split on blank lines.
-
-## Metrics
-
-**Latency** — TTFT (p50/p95), E2E (p50/p95/p99), TPOT (p50/p95), 95% confidence interval, coefficient of variation (CV).
-
-**Throughput** — prefill tokens/s, decode tokens/s, output tokens/s per request, aggregate system tokens/s, requests per second.
-
-**Streaming quality** — second-chunk delay, chunk-gap p95. Captured from `stdout` chunks during streaming runs.
-
-**Reliability** — success rate, goodput rate and RPS against SLO budgets, repeatability (most common output hash frequency across repeated runs).
-
-**Stability** — CV (stddev/mean for E2E latency); green ≤10%, yellow ≤25%, red >25%.
-
-Token counts come from `fm token-count --quiet`. If `fm` cannot count tokens, those fields are blank while character throughput is still reported.
-
-Terminal layout is responsive: wide → full scoreboard + detail tables, medium → tighter single table, narrow → compact model cards. Use `--width` to preview any layout and `--ascii` for log-friendly output.
+Plain text files are split on blank lines. Parse errors name the file and, for JSONL, the line number.
 
 ## Colors and Legend
 
 Table output is color-coded on interactive terminals — **green** is better/passing, **yellow** is marginal/partial, **red** is failing/unstable. Fixed thresholds apply to success rate, goodput, CV, and repeatability. Latency uses SLO thresholds when set, otherwise lower-is-better relative ranking. Throughput uses higher-is-better relative ranking.
 
 ```sh
-fm-bench legend          # full column definitions and color rules
+fm-bench legend          # definitions, provenance, and color rules
 fm-bench legend --json   # machine-readable
 ```
 
-`NO_COLOR=1` disables color; `FORCE_COLOR=1` or `--color` enables it. `--ascii` switches to plain ASCII borders for log systems.
-
-A live single-line progress indicator runs on stderr during interactive sessions. The final report always goes to stdout — `--json`, `--csv`, and `--out` stay automation-friendly.
+`NO_COLOR=1` disables color; `FORCE_COLOR=1` or `--color` enables it. `--ascii` switches to plain ASCII borders for log systems. A live single-line progress indicator runs on stderr during interactive sessions; the final report always goes to stdout.
 
 ## Requirements
 
@@ -279,19 +298,18 @@ A live single-line progress indicator runs on stderr during interactive sessions
 
 Benchmark commands refuse to start on older macOS versions and report the detected version plus the latest supported macOS — see [docs/supported-platforms.md](docs/supported-platforms.md).
 
-`pcc` (Private Cloud Compute) availability depends on Apple's current eligibility. `fm-bench` shows it as skipped if `fm available --model pcc` reports unavailable.
-
-See [docs/methodology.md](docs/methodology.md) for benchmark methodology and metric references.
-
 ## Development
 
 ```sh
 npm install
-npm test     # node --test
-npm run lint
+npm test          # node --test (unit + integration against a fake fm)
+npm run lint      # node --check on every source file
+npm run check     # lint + tests + npm pack integrity — run this before pushing
 ```
 
-No runtime npm dependencies.
+No runtime npm dependencies. Integration tests drive the real CLI against `test/fixtures/fake-fm.mjs`, which emulates normal, streaming, slow, malformed, failing, timed-out, partial, and interrupted `fm` behaviour, so the suite runs on machines without `fm`.
+
+Documentation: [methodology](docs/methodology.md) · [report format](docs/report-format.md) · [compatibility](docs/compatibility.md) · [supported platforms](docs/supported-platforms.md) · [releasing](docs/releasing.md).
 
 ## License
 

package/bin/fm-bench.js

@@ -1,5 +1,19 @@
 #!/usr/bin/env node
 import { runCli } from '../src/cli.js';
+import { killActiveChildren } from '../src/process.js';
+
+// Terminate any in-flight `fm` child process before exiting, then use the
+// conventional 128 + signal exit code. A second signal exits immediately.
+let interrupting = false;
+for (const [signal, exitCode] of [['SIGINT', 130], ['SIGTERM', 143]]) {
+  process.on(signal, () => {
+    const killed = killActiveChildren(signal);
+    if (interrupting) process.exit(exitCode);
+    interrupting = true;
+    const graceMs = killed > 0 ? 500 : 0;
+    setTimeout(() => process.exit(exitCode), graceMs).unref();
+  });
+}
 
 runCli(process.argv.slice(2)).catch((error) => {
   const message = error?.message || String(error);

package/docs/compatibility.md

@@ -0,0 +1,46 @@
+# `fm` compatibility
+
+`fm-bench` is a client of whatever `fm` binary is installed. Apple has already changed that surface between macOS 27 builds: current builds expose `count-tokens`, older ones exposed `token-count`, and quota reporting is not available everywhere. fm-bench therefore detects capabilities at runtime instead of assuming a fixed subcommand list.
+
+## How detection works
+
+Once per run, `fm-bench` spawns two cheap calls:
+
+1. `fm --help` — the command list, the `MODELS` section, and (when present) a `--model` option list.
+2. `fm respond --help` — which flags `respond` actually accepts (`--model`, `--[no-]stream`, `--instructions`, `--greedy`, `--use-case`, `--guardrails`, `--image`, `--tool`, `--schema`).
+
+The result is recorded in the report (`capabilities`) and shown by `fm-bench models`, `fm-bench doctor`, and `fm-bench doctor --json`.
+
+Detection is tolerant of formatting changes: section headers are matched case-insensitively on uppercase lines, model lines by their column layout, and boolean flags in either `--flag` or Apple's `--[no-]flag` spelling. ANSI escapes are stripped before parsing.
+
+## Capability policy
+
+| Capability | When missing |
+|------------|--------------|
+| Token counting (`count-tokens`, fallback `token-count`) | The run still benchmarks latency and streaming. Every token metric is `null` with `available: false` in `metrics`, the table prints an `unavailable:` note, and the report carries a warning. `fm count-tokens` is never called. |
+| Streaming | TTFT, generation time, TPOT, prefill, and chunk-gap metrics are unavailable, and `--stream`/`--no-stream` is not passed through. E2E latency, success rate, and RPS still work. |
+| Quota command | The `models` table omits the quota column and reports quota as unavailable. |
+| Model selection (`--model`) | The flag is not passed; `fm` uses its default model. |
+| `--use-case`, `--guardrails`, `--instructions`, `--greedy` | Silently not passed, instead of failing the run with an argument error. |
+| No usable `fm` at all | The command exits `2` with the spawn error and a hint to install `fm` or point `--fm-bin` / `FM_BIN` at a compatible binary. |
+
+Models the build does not list are refused before any benchmark starts, with the message `not supported by this fm build (supported: ...)`. Raw `fm` argument-error text (usage blocks, `Unknown command`) never reaches reports, tables, or JSON output.
+
+## Exit codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | Success. A failed measured run is data, not a CLI failure — use `--ci` to make failures fatal. |
+| `1` | Operational failure: `--ci` gate failed, reports failed validation, or an `fm` run could not produce data. |
+| `2` | Usage or environment error: unknown flag, missing argument, unsupported macOS, unusable `fm`, malformed report input, or `compare --strict` suite mismatch. |
+| `130` / `143` | Interrupted by SIGINT / SIGTERM. In-flight `fm` child processes are terminated first. |
+
+The macOS 27 requirement is enforced only when `fm-bench` resolves the default `fm` from `PATH`. Supplying `--fm-bin <path>` or `FM_BIN` lets the CLI run on any host and instead fails on the binary's actual capabilities (exit `2` when it exposes no usable commands).
+
+## Verified builds
+
+| Platform | `fm` surface | Notes |
+|----------|--------------|-------|
+| macOS 27.0 (26A5425a), Apple M5 Pro (Mac17,9) | `available`, `chat`, `count-tokens`, `license`, `respond`, `schema`, `serve`; model `system` only; no quota command; no `--version` | Real-machine smoke tests in 0.7.0. `test/fixtures/fm-help-macos27.txt` captures this help output as a regression baseline. |
+
+If your `fm` build differs, `fm-bench doctor` shows exactly what was detected, and the report's `capabilities` block records it. Please open an issue with the `doctor --json` output and the report digest when something is missing.

package/docs/methodology.md

@@ -14,27 +14,45 @@ The metric set follows common LLM inference benchmark practice:
 - MLCommons describes varying concurrency and reporting verified operating points for TTFT, throughput, interactivity, and response latency rather than interpolated performance: <https://mlcommons.org/2026/03/mlperf-endpoints-gen-ai-benchmarking/>
 - MLPerf Client emphasizes local client workloads with multiple task types and varying prompt/response lengths: <https://mlcommons.org/benchmarks/client/>
 
+## What `fm` can actually tell us
+
+`fm-bench` never invents precision the CLI cannot provide. Every metric is classified by how it is obtained:
+
+| Kind | Meaning |
+|------|---------|
+| **measured** | Observed directly: process wall clock, exit codes, stdout chunk arrival, `fm`-reported token counts. |
+| **proxy** | Observed at a coarser granularity than the ideal metric (for example chunk arrival instead of token timestamps). |
+| **derived** | Computed from measured values (for example output tokens divided by elapsed seconds). |
+| **controlled** | An input setting rather than a measurement, such as the concurrency operating point. |
+
+The same classification is machine-readable in every JSON report under `metrics` and in the CLI via `fm-bench legend` (SOURCE column). A metric the installed `fm` build cannot support is `available: false` with a reason, and renders as `-` rather than a plausible-looking number.
+
 ## Metrics
 
 For a column-by-column terminal reference, run `fm-bench legend`.
 
-- `TTFT`: time from starting `fm respond` to the first streamed stdout chunk. This is a practical terminal-side proxy for time to first token.
-- `E2E latency`: time from starting `fm respond` until the process exits and the full response is captured.
-- `generation_ms`: `E2E - TTFT`.
-- `TPOT`: `(E2E - TTFT) / (output_tokens - 1)`. The first output token is excluded so TPOT focuses on decode cadence.
-- `second_chunk_ms`: time between the first and second streamed stdout chunks. This is a terminal-side proxy for time-to-second-token style startup smoothness.
-- `chunk_gap`: the distribution of time between consecutive streamed stdout chunks. It is useful for spotting streaming jitter, but it is chunk-based rather than token-based because the `fm` CLI writes stdout chunks, not token timestamp events.
-- `prefill_tokens_per_second`: input prompt tokens divided by TTFT seconds. This estimates prompt-processing speed for streaming runs.
-- `tokens_per_second`: output tokens divided by E2E seconds for one request.
-- `decode_tokens_per_second`: output tokens after the first token divided by generation seconds.
-- `total output token throughput`: all successful output tokens for a model divided by that model's measured wall-clock window.
-- `total token throughput`: successful prompt and output tokens divided by that model's measured wall-clock window.
-- `RPS`: successful requests divided by that model's measured wall-clock window.
-- `goodput`: successful requests that also satisfy all provided SLO thresholds.
-- `goodput RPS`: SLO-passing requests divided by that model's measured wall-clock window. If SLOs are set and no requests pass, this is reported as zero.
-- `repeatability`: for repeated runs of the same prompt, the average share of runs that produced the most common normalized output hash.
-- `CV`: coefficient of variation, or sample standard deviation divided by the mean. Lower values indicate steadier latency for that metric.
-- `95% CI`: a t-distribution confidence interval around the sample mean. Treat it as useful context, not proof, especially with very small sample sizes.
+- `TTFT` — *proxy*. Time from starting `fm respond` to the first streamed stdout chunk. `fm` exposes no per-token timestamps, so this is a terminal-side approximation of time to first token; the first chunk may already contain several tokens.
+- `E2E latency` — *measured*. Time from starting `fm respond` until the process exits and the full response is captured.
+- `generation_ms` — *derived*. `E2E - TTFT`, reported only when the answer arrives in more than one stdout chunk. When a single chunk carries the whole answer, prefill and decode are not separable and the value is unavailable instead of near zero.
+- `TPOT` — *derived*. `(E2E - TTFT) / (output_tokens - 1)`, reported when the answer has at least three output tokens so the interval is an average over two or more decode tokens rather than the inverse of a single chunk gap. Requires streaming and a token-counting `fm`.
+- `second_chunk_ms` — *proxy*. Time between the first and second streamed stdout chunks: a terminal-side signal for startup smoothness.
+- `chunk_gap` — *proxy*. Distribution of time between consecutive streamed stdout chunks. Useful for spotting streaming jitter; chunk-based, not token-based.
+- `prefill_tokens_per_second` — *proxy*. Input prompt tokens divided by TTFT seconds. Because TTFT includes process startup and first-token latency, this systematically understates true prefill speed and should be read as an upper-bound-constrained estimate, not a kernel measurement.
+- `tokens_per_second` — *derived*. Output tokens divided by E2E seconds for one request.
+- `decode_tokens_per_second` — *derived*. Output tokens after the first token divided by generation seconds.
+- `output token throughput` — *derived*. All successful output tokens for a model divided by that model's measured wall-clock window.
+- `total token throughput` — *derived*. Successful prompt and output tokens divided by the model's measured wall-clock window.
+- `RPS` — *measured*. Successful requests divided by the model's measured wall-clock window.
+- `goodput` — *derived*. Successful requests that also satisfy every provided SLO threshold. A run whose SLO metric is unavailable counts as not good, so an unverifiable SLO can never inflate goodput.
+- `goodput RPS` — *derived*. SLO-passing requests divided by the model's measured wall-clock window. Zero is reported when SLOs are set and nothing passes.
+- `repeatability` — *derived*. For repeated runs of the same prompt, the average share of runs that produced the most common normalized output hash.
+- `CV` — *derived*. Sample standard deviation divided by the mean. Reported only with two or more successful samples.
+- `95% CI` — *derived*. t-distribution confidence interval around the sample mean, using exact t critical values up to 30 degrees of freedom and the standard 2.0 / 1.96 approximations beyond that. Reported only with two or more successful samples. Treat it as context, not proof, at small sample sizes.
+- `quota` — *measured*, when the `fm` build exposes a quota command. Most current builds do not, in which case `fm-bench` reports quota as unavailable rather than empty.
+
+### Small samples
+
+Statistics are computed over the successful samples of one model/concurrency row. With zero samples the row reports `null`; with one sample, percentiles and the mean are reported while spread metrics (`stddev`, `cv`, `ci95Low`, `ci95High`) are `null`. A single run cannot demonstrate stability, so `fm-bench` does not print `0%` variation for it.
 
 ## Operating Points
 
@@ -44,20 +62,28 @@ Use `--request-rate <rps>` to pace request starts independently of concurrency.
 
 `fm-bench` does not interpolate between operating points. It reports only what was actually measured.
 
+## Warmups
+
+`--warmup <n>` runs `n` unmeasured calls per model at the start of each operating point. The first `fm respond` after a cold start includes model load time, which can dominate a short prompt (observed at several hundred milliseconds on Apple silicon). Warmups are never mixed into the measured results.
+
+## Failures and retries
+
+A failed call is recorded as a failed measurement with its error text, and is excluded from latency and throughput statistics. `--retry <n>` retries failed calls with exponential backoff before recording the failure; the report keeps the run count and records the number of attempts separately, so a retried success is still one sample and never silently duplicates work.
+
 ## Prompt Profiles
 
 The `client` profile is a pragmatic local-machine mix inspired by MLPerf Client's emphasis on multiple task categories and prompt/response lengths. It includes short chat, content generation, structured extraction, light summarization, and code analysis prompts. It is not a formal MLPerf submission suite; it is a convenient built-in workload for comparing your own Mac, OS build, and `fm` models over time.
 
 ## Caveats
 
-`fm-bench` uses `fm token-count --quiet` as the source of token counts, so token values follow Apple's local tokenizer behavior. It does not judge semantic quality unless you provide your own prompt suite and inspect captured outputs with `--capture-output`.
+Token counts come from the `fm` build's own token-counting command (`count-tokens` on current builds, `token-count` on older ones). If a build has no such command, token-derived metrics are reported as unavailable rather than estimated. `fm-bench` cannot judge semantic quality unless you provide your own prompt suite and inspect captured outputs with `--capture-output`.
 
 Client-side measurements include process startup, local queueing, model prefill, streaming, detokenization, and terminal pipe overhead. That is intentional for a command-line benchmark, but it is not the same as an internal model-kernel benchmark.
 
-Stream smoothness metrics use stdout chunk arrival times. A chunk can contain more than one token, and terminal or pipe buffering can affect chunk boundaries. Treat `second_chunk_ms` and `chunk_gap` as user-visible streaming diagnostics, not raw decoder telemetry.
+Stream smoothness metrics use stdout chunk arrival times. A chunk can contain more than one token, and pipe buffering can affect chunk boundaries. Treat `second_chunk_ms` and `chunk_gap` as user-visible streaming diagnostics, not raw decoder telemetry.
 
 For serious comparisons, prefer at least three runs per prompt, include warmups, benchmark both interactive and throughput or client profiles, compare models at the same concurrency operating points, set SLOs that match your real UX budget, and save JSON reports for later analysis.
 
 ## Report artifacts
 
-Saved JSON includes client-side environment metadata (hardware model, macOS build, `fm` help digest, power/thermal snapshot) so shared results remain interpretable on other machines. Use `fm-bench validate` before publishing and `fm-bench compare --strict` when you require identical prompt suites. Format details: [report-format.md](./report-format.md).
+Saved JSON includes client-side environment metadata (hardware model, macOS build, `fm` help digest, power/thermal snapshot) plus the detected `fm` capabilities and per-metric availability, so shared results remain interpretable on other machines. Use `fm-bench validate` before publishing and `fm-bench compare --strict` when you require identical prompt suites. Format details: [report-format.md](./report-format.md).

package/docs/releasing.md

@@ -1,12 +1,23 @@
 # Releasing
 
-`fm-bench` uses semver tags (`v*.*.*`). Pushing a tag runs the **Release** workflow: test, lint, npm publish (with provenance), and a GitHub release whose body is taken from the matching section in **`CHANGELOG.md`** (plus a compare link).
+`fm-bench` uses semver tags (`v*.*.*`). Pushing a tag runs the **Release** workflow: lint/test/package check, npm publish (with provenance), and a GitHub release whose body is taken from the matching section in **`CHANGELOG.md`** (plus a compare link).
 
 ## Prerequisites
 
 - Repository secret **`NPM_TOKEN`**: npm automation token with publish access to `fm-bench`.
 - **`main`** is green on CI.
 
+If `NPM_TOKEN` is missing or expired, the Release workflow warns, skips the npm publish step, and still creates the GitHub release. The release notes then state that npm publish was skipped. Re-run the Release workflow with the tag once the token is fixed — an already-published version is skipped automatically.
+
+## Before tagging
+
+```sh
+npm run check     # lint + tests + npm pack integrity
+npm run publish:dry-run
+```
+
+Then bump `CHANGELOG.md` with a `## X.Y.Z` section; the release notes come from that section.
+
 ## Option A — GitHub Actions (recommended)
 
 1. Open **Actions → Version → Run workflow**.
@@ -17,20 +28,34 @@
 ## Option B — Local
 
 ```sh
-npm ci && npm test && npm run lint && npm run check:pack
-npm version patch   # or minor / major
-git push --follow-tags
+npm ci && npm run check
+npm version minor   # or patch / major
+git push origin main --follow-tags
 ```
 
 ## Re-run Release without republishing
 
-If npm already has the version but GitHub release failed (or vice versa), use **Actions → Release → Run workflow** and enter the existing tag (for example `v0.5.3`). The workflow skips npm publish when that version is already on the registry. If the GitHub release already exists, it **updates the release notes** from `CHANGELOG.md`.
+If npm already has the version but the GitHub release failed (or vice versa), use **Actions → Release → Run workflow** and enter the existing tag (for example `v0.6.3`). The workflow skips npm publish when that version is already on the registry. If the GitHub release already exists, it **updates the release notes** from `CHANGELOG.md`.
 
 Refresh notes locally without re-publishing:
 
 ```sh
-node scripts/changelog-release-notes.mjs 0.6.0 > /tmp/notes.md
-gh release edit v0.6.0 --notes-file /tmp/notes.md --repo devinoldenburg/fm-bench
+node scripts/changelog-release-notes.mjs 0.6.3 > notes.md
+gh release edit v0.6.3 --notes-file notes.md --repo devinoldenburg/fm-bench
+```
+
+## Verifying a release
+
+```sh
+git tag --list 'v0.7.0'
+gh release view v0.7.0 --repo devinoldenburg/fm-bench
+npm view fm-bench version            # registry version
+
+mkdir -p fm-bench-check && cd fm-bench-check
+npm install fm-bench@0.7.0
+./node_modules/.bin/fm-bench --version
+./node_modules/.bin/fm-bench --help > /dev/null
+./node_modules/.bin/fm-bench doctor
 ```
 
 ## Dry run
@@ -40,4 +65,4 @@ npm run publish:dry-run
 npm run check:pack
 ```
 
-CI runs the same dry-run on every push to `main` and on pull requests.
+CI runs the dry run on every push to `main` and on pull requests.