fm-bench 0.7.1 → 0.8.0

package/README.md

@@ -1,7 +1,8 @@
 # fm-bench
 
-[![CI](https://github.com/devinoldenburg/fm-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/devinoldenburg/fm-bench/actions/workflows/ci.yml)
+[![CI](https://github.com/dvnold/fm-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/dvnold/fm-bench/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/fm-bench.svg)](https://www.npmjs.com/package/fm-bench)
+[![node](https://img.shields.io/node/v/fm-bench.svg)](https://nodejs.org/)
 
 Benchmark Apple's `fm` command on macOS 27+.
 
@@ -30,28 +31,27 @@ npm install -g fm-bench
 fm-bench
 ```
 
-One command discovers your models, runs the standard prompt suite, and prints a full benchmark report (real output from a macOS 27.0 Apple M5 Pro):
+One command discovers your models, warms each one up, runs the standard prompt suite, and prints a full benchmark report. Real output of `fm-bench --runs 3` on macOS 27.2 (26B5091g), Apple M5 Pro:
 
 ```text
-fm-bench 0.7.0 | darwin/arm64 | fm
-prompts 3 | runs 3 | concurrency 1 | stream on | measured 9 | failed 0 | skipped 0 | elapsed 6.37s | SLO
-TTFT<=1.00s,E2E<=3.00s
-unavailable: quota unavailable — this fm build exposes no quota command
+fm-bench 0.8.0 | darwin/arm64 | fm
+prompts 3 | runs 3 | concurrency 1 | stream on | measured 9 | failed 0 | skipped 0 | elapsed 11.07s
+models: system = AFM 3 Core Advanced
 
-┌───┬────────┬────────┬─────┬──────┬──────────┬───────┬───────┬─────────┬────────┬───────┬─────┬──────┐
-│ C │ MODEL  │ STATUS │ OK  │ GOOD │ GOOD RPS │ TTFT  │ E2E   │ E2E P95 │ USER/S │ SYS/S │ CV  │ NOTE │
-├───┼────────┼────────┼─────┼──────┼──────────┼───────┼───────┼─────────┼────────┼───────┼─────┼──────┤
-│ 1 │ system │ ok     │ 9/9 │ 100% │      1.7 │ 377ms │ 487ms │   749ms │   34.0 │  32.8 │ 25% │      │
-└───┴────────┴────────┴─────┴──────┴──────────┴───────┴───────┴─────────┴────────┴───────┴─────┴──────┘
+┌───┬────────┬────────┬─────┬───────┬───────┬─────────┬────────┬───────┬────┬──────┐
+│ C │ MODEL  │ STATUS │ OK  │ TTFT  │ E2E   │ E2E P95 │ USER/S │ SYS/S │ CV │ NOTE │
+├───┼────────┼────────┼─────┼───────┼───────┼─────────┼────────┼───────┼────┼──────┤
+│ 1 │ system │ ok     │ 9/9 │ 620ms │ 687ms │   1.43s │   40.6 │  37.7 │ 3% │      │
+└───┴────────┴────────┴─────┴───────┴───────┴─────────┴────────┴───────┴────┴──────┘
 
 ┌───┬────────┬────────┬─────────┬───────────┬──────────┬───────────┬─────────┬────────┐
 │ C │ MODEL  │ IN AVG │ OUT AVG │ PREFILL/S │ DECODE/S │ CHUNK P95 │ E2E P99 │ REPEAT │
 ├───┼────────┼────────┼─────────┼───────────┼──────────┼───────────┼─────────┼────────┤
-│ 1 │ system │     13 │      20 │      35.9 │      113 │     197ms │   749ms │   100% │
+│ 1 │ system │     27 │      38 │      40.7 │     78.1 │     197ms │   1.45s │   100% │
 └───┴────────┴────────┴─────────┴───────────┴──────────┴───────────┴─────────┴────────┘
 ```
 
-Default profile is `standard` (3 prompts). Use `--runs 5`, `--sweep-concurrency 1,2`, or `--profile client` for heavier suites.
+Default profile is `standard` (3 prompts) with one run each, so run-to-run CV shows `-` until you pass `--runs 2` or more. Use `--runs 5`, `--sweep-concurrency 1,2`, or `--profile client` for heavier suites. When a model is unavailable (for example `pcc` outside the Terminal app), it is listed as skipped with fm's reason; `--available-only` hides it.
 
 Wide terminals add TTFT P95, TPOT, and RPS columns; medium terminals tighten the table; narrow terminals switch to compact model cards automatically. `--width <n>` previews any layout.
 
@@ -64,7 +64,7 @@ npm install -g fm-bench
 Install directly from GitHub (always latest):
 
 ```sh
-npm install -g --install-links git+https://github.com/devinoldenburg/fm-bench.git
+npm install -g --install-links git+https://github.com/dvnold/fm-bench.git
 ```
 
 Local development:
@@ -74,24 +74,26 @@ npm install && npm link
 fm-bench doctor   # verify your setup
 ```
 
-**Requirements:** macOS 27+, Node.js 20+, Apple Intelligence enabled.
+**Requirements:** macOS 27+, Node.js 22+, Apple Intelligence enabled.
 
 ## Commands
 
 | Command | What it does |
 |---------|-------------|
 | `fm-bench` | Run the full benchmark (default) |
-| `fm-bench models` | Show the detected `fm` capabilities, discovered models, availability, and quota when supported |
+| `fm-bench models` | Show the detected `fm` capabilities, discovered models, their identity (for example `AFM 3 Core Advanced`), availability, and quota |
 | `fm-bench compare <a.json> <b.json>` | Regression diff with suite/hardware/macOS warnings; `--strict` exits 2 when suites differ |
 | `fm-bench history [dir]` | Trend table from saved reports (sorted by time, tags visible) |
 | `fm-bench validate <report.json>` | Verify report JSON (schema v1) before sharing; `--json` for CI |
 | `fm-bench export <report.json>` | Standalone HTML report with embedded JSON |
 | `fm-bench legend` | Definitions, provenance (measured/proxy/derived), and color rules for every column |
-| `fm-bench doctor` | Environment and `fm` capability check; `--json` for scripts |
+| `fm-bench doctor` | Environment, `fm` license, and capability check; `--json` for scripts |
 | `fm-bench metrics` | Alias for `legend` |
 
 Exit codes: `0` success, `1` operational failure (failed `--ci` gate, invalid reports), `2` usage or environment error (bad flags, unsupported macOS, unusable `fm`, no runnable model). Interrupting a run with Ctrl+C terminates in-flight `fm` processes.
 
+Mistyped commands and flags are caught with a suggestion (`Unknown command "modles". Did you mean "models"?`) instead of being benchmarked as a prompt. To benchmark such a word anyway, put it after `--`.
+
 ## Common Recipes
 
 ```sh
@@ -130,7 +132,7 @@ fm-bench --format csv --out bench.csv
 |------|---------|-------------|
 | `-m, --models <list>` | discovered | Comma-separated or repeated model names |
 | `-r, --runs <n>` | 1 | Measured runs per prompt/model |
-| `--warmup <n>` | 0 | Unmeasured warmup runs per model before measurement |
+| `--warmup <n>` | 1 | Unmeasured warmup runs per model before measurement; `0` measures cold start |
 | `-c, --concurrency <n>` | 1 | Parallel `fm` processes |
 | `--sweep-concurrency <list>` | — | Separate operating points, e.g. `1,2,4` |
 | `--request-rate <rps>` | — | Pace request starts at a target rate |
@@ -200,27 +202,32 @@ Nine built-in suites, choose the one that matches your use case:
 
 ## Metrics
 
-**Latency** — TTFT (p50/p95), E2E (p50/p95/p99), TPOT, 95% confidence interval, coefficient of variation (CV).
+**Latency** — TTFT (p50/p95), E2E (p50/p95/p99), TPOT, 95% confidence interval, and run-to-run CV (per prompt across repeated runs, so mixing short and long prompts does not read as instability).
 
 **Throughput** — prefill tokens/s, decode tokens/s, output tokens/s per request, aggregate system tokens/s, requests per second.
 
-**Streaming quality** — second-chunk delay, chunk-gap p95, captured from stdout chunk arrival during streaming runs.
+**Streaming quality** — first-chunk tokens, second-chunk delay, chunk-gap p95, captured from stdout chunk arrival during streaming runs.
 
 **Reliability** — success rate, goodput rate and RPS against SLO budgets, repeatability (most common output hash frequency across repeated runs).
 
+**Model identity** — the identity `fm models` reports (for example `system = AFM 3 Core Advanced`) is printed in the report header and saved in JSON, and `compare` warns when the model behind a name changed between two runs.
+
 Every metric is labelled by provenance in JSON and in `fm-bench legend`:
 
 - **measured** — process wall clock, exit codes, chunk arrival, `fm` token counts
 - **proxy** — TTFT and chunk gaps (chunk granularity, not token timestamps); prefill tokens/s
 - **derived** — TPOT, decode tokens/s, throughput, CV, confidence intervals, goodput
 
-Token counts come from the `fm` build's own token-counting command (`count-tokens`, or `token-count` on older builds). If the build cannot count tokens, token metrics render as `-`, JSON carries `null`, and `metrics.promptTokens.available` is `false`. Spread statistics (`CV`, 95% CI) need at least two successful samples; with one sample they are unavailable rather than `0`. See [docs/methodology.md](docs/methodology.md).
+Token counts come from the `fm` build's own token-counting command (`count-tokens`, or `token-count` on older builds). `count-tokens` adds one framing token to every count; fm-bench calibrates that overhead once per run and removes it from output counts (`tokenCounter` in JSON). If the build cannot count tokens, token metrics render as `-`, JSON carries `null`, and `metrics.promptTokens.available` is `false`. Spread statistics (`CV`, 95% CI) need at least two successful samples; with one sample they are unavailable rather than `0`.
+
+`fm` streams coarse deltas — on macOS 27.2 the first stdout chunk carries about 20 tokens — so TTFT is time to the first chunk, and TPOT and decode tokens/s are computed only over the tokens that arrive after it. See [docs/methodology.md](docs/methodology.md).
 
 ## fm compatibility
 
 `fm-bench` probes `fm --help` and `fm respond --help` once per run and adapts:
 
-- Subcommand names (`count-tokens` vs legacy `token-count`) are detected, not hardcoded.
+- Subcommand names (`count-tokens` vs legacy `token-count`, `models` vs the deprecated `available`) are detected, not hardcoded.
+- Model availability and identity come from one `fm models` call; reasons such as `Private Cloud Compute is not available in this context. Please use the Terminal app.` are shown in full.
 - Flags the build does not document (`--stream`, `--use-case`, `--guardrails`, `--model`) are not passed through.
 - Unsupported models are rejected before a benchmark starts, with the supported list in the error.
 - Raw `fm` argument-error text never reaches reports, tables, or JSON.
@@ -293,8 +300,9 @@ fm-bench legend --json   # machine-readable
 ## Requirements
 
 - macOS 27.0 or newer (Apple's `fm` CLI is preinstalled there).
-- Node.js 20 or newer.
-- Apple Intelligence enabled on the device.
+- Node.js 22 or newer (CI covers Node 22, 24, and 26).
+- Apple Intelligence enabled on the device, and the `fm` terms accepted once with `fm license` (`fm-bench doctor` checks this).
+- To benchmark the Private Cloud Compute model (`pcc`), run fm-bench from the Terminal app: `fm` reports `pcc` as unavailable in other contexts such as editor terminals.
 
 Benchmark commands refuse to start on older macOS versions and report the detected version plus the latest supported macOS — see [docs/supported-platforms.md](docs/supported-platforms.md).
 

package/bin/fm-bench.js

@@ -4,18 +4,29 @@ import { killActiveChildren } from '../src/process.js';
 
 // Terminate any in-flight `fm` child process before exiting, then use the
 // conventional 128 + signal exit code. A second signal exits immediately.
-let interrupting = false;
+let signalsReceived = 0;
+let interrupted = false;
 for (const [signal, exitCode] of [['SIGINT', 130], ['SIGTERM', 143]]) {
   process.on(signal, () => {
+    signalsReceived += 1;
+    interrupted = true;
+    // Carry the status on the process too, so the code survives even if the
+    // exit below is preempted.
+    process.exitCode = exitCode;
     const killed = killActiveChildren(signal);
-    if (interrupting) process.exit(exitCode);
-    interrupting = true;
-    const graceMs = killed > 0 ? 500 : 0;
-    setTimeout(() => process.exit(exitCode), graceMs).unref();
+    if (signalsReceived > 1 || killed === 0) process.exit(exitCode);
+    // Deliberately not unref'd: this timer performs the exit after giving the
+    // killed children a moment to be reaped.
+    setTimeout(() => process.exit(exitCode), 500);
   });
 }
 
 runCli(process.argv.slice(2)).catch((error) => {
+  if (interrupted) {
+    // The user asked to stop; a follow-up benchmark error is noise.
+    process.exitCode = typeof error?.exitCode === 'number' ? error.exitCode : 1;
+    return;
+  }
   const message = error?.message || String(error);
   console.error(`fm-bench: ${message}`);
   process.exitCode = typeof error?.exitCode === 'number' ? error.exitCode : 1;

package/docs/compatibility.md

@@ -1,15 +1,18 @@
 # `fm` compatibility
 
-`fm-bench` is a client of whatever `fm` binary is installed. Apple has already changed that surface between macOS 27 builds: current builds expose `count-tokens`, older ones exposed `token-count`, and quota reporting is not available everywhere. fm-bench therefore detects capabilities at runtime instead of assuming a fixed subcommand list.
+`fm-bench` is a client of whatever `fm` binary is installed. Apple has already changed that surface between macOS 27 builds: `count-tokens` replaced `token-count`, macOS 27.2 renamed `available` to `models` (the old name still works there but prints a deprecation warning) and added `quota-usage`, `config`, and the `pcc` model. fm-bench therefore detects capabilities at runtime instead of assuming a fixed subcommand list.
 
 ## How detection works
 
-Once per run, `fm-bench` spawns two cheap calls:
+Once per run, `fm-bench` spawns three cheap calls:
 
-1. `fm --help` — the command list, the `MODELS` section, and (when present) a `--model` option list.
+1. `fm --help` — the command list, the `MODELS` section, and (when present) a `--model` option list. The `<model>` custom-provider placeholder is not treated as a model.
 2. `fm respond --help` — which flags `respond` actually accepts (`--model`, `--[no-]stream`, `--instructions`, `--greedy`, `--use-case`, `--guardrails`, `--image`, `--tool`, `--schema`).
+3. `fm models` — one call that reports every model's availability, its identity (for example `✓ system (AFM 3 Core Advanced)`), and the reason for anything unavailable. Builds without `models` fall back to `fm available --model <name>` per model.
 
-The result is recorded in the report (`capabilities`) and shown by `fm-bench models`, `fm-bench doctor`, and `fm-bench doctor --json`.
+`warning:` lines that `fm` prints (such as the `available` rename notice) are discarded before parsing, so they can never be read as a model status or an error cause.
+
+The result is recorded in the report (`capabilities`, `models[].identity`) and shown by `fm-bench models`, `fm-bench doctor`, and `fm-bench doctor --json`. `doctor` also runs `fm license --status`, because `fm respond` cannot run until the terms are accepted.
 
 Detection is tolerant of formatting changes: section headers are matched case-insensitively on uppercase lines, model lines by their column layout, and boolean flags in either `--flag` or Apple's `--[no-]flag` spelling. ANSI escapes are stripped before parsing.
 
@@ -18,6 +21,7 @@ Detection is tolerant of formatting changes: section headers are matched case-in
 | Capability | When missing |
 |------------|--------------|
 | Token counting (`count-tokens`, fallback `token-count`) | The run still benchmarks latency and streaming. Every token metric is `null` with `available: false` in `metrics`, the table prints an `unavailable:` note, and the report carries a warning. `fm count-tokens` is never called. |
+| Model list (`models`, fallback `available`) | With neither command, models from `fm --help` are assumed runnable and any failure is recorded per run. Model identity is only available from `models`. |
 | Streaming | TTFT, generation time, TPOT, prefill, and chunk-gap metrics are unavailable, and `--stream`/`--no-stream` is not passed through. E2E latency, success rate, and RPS still work. |
 | Quota command | The `models` table omits the quota column and reports quota as unavailable. |
 | Model selection (`--model`) | The flag is not passed; `fm` uses its default model. |
@@ -41,6 +45,7 @@ The macOS 27 requirement is enforced only when `fm-bench` resolves the default `
 
 | Platform | `fm` surface | Notes |
 |----------|--------------|-------|
-| macOS 27.0 (26A5425a), Apple M5 Pro (Mac17,9) | `available`, `chat`, `count-tokens`, `license`, `respond`, `schema`, `serve`; model `system` only; no quota command; no `--version` | Real-machine smoke tests in 0.7.0. `test/fixtures/fm-help-macos27.txt` captures this help output as a regression baseline. |
+| macOS 27.2 (26B5091g), Apple M5 Pro (Mac17,9) | `chat`, `config`, `count-tokens`, `license`, `models`, `quota-usage`, `respond`, `schema`, `serve` (plus deprecated `available`); models `system` (`AFM 3 Core Advanced`) and `pcc`; `--model-provider` for custom Chat Completions providers; no `--version` | Real-machine runs in 0.8.0. `pcc` reports "not available in this context. Please use the Terminal app." outside Terminal. `count-tokens` adds one framing token per count. The first streamed chunk carries about 20 tokens. Fixtures: `test/fixtures/*-macos27.2.txt`. |
+| macOS 27.0 (26A5425a), Apple M5 Pro (Mac17,9) | `available`, `chat`, `count-tokens`, `license`, `respond`, `schema`, `serve`; model `system` only; no quota command; no `--version` | Real-machine smoke tests in 0.7.0. `test/fixtures/fm-help-macos27.txt` captures this help output as a regression baseline; the fake `fm` emulates it with `FAKE_FM_SCENARIO=legacy-available`. |
 
 If your `fm` build differs, `fm-bench doctor` shows exactly what was detected, and the report's `capabilities` block records it. Please open an issue with the `doctor --json` output and the report digest when something is missing.

package/docs/methodology.md

@@ -31,24 +31,25 @@ The same classification is machine-readable in every JSON report under `metrics`
 
 For a column-by-column terminal reference, run `fm-bench legend`.
 
-- `TTFT` — *proxy*. Time from starting `fm respond` to the first streamed stdout chunk. `fm` exposes no per-token timestamps, so this is a terminal-side approximation of time to first token; the first chunk may already contain several tokens.
+- `TTFT` — *proxy*. Time from starting `fm respond` to the first streamed stdout chunk. `fm` exposes no per-token timestamps, so this is a terminal-side approximation of time to first token. `fm` streams coarse deltas: on macOS 27.2 the first chunk consistently carries about 20 tokens (the same through a pseudo-terminal and through `fm serve`), so TTFT is closer to "time to the first ~20 tokens". `first_chunk_tokens` records the exact count per run.
 - `E2E latency` — *measured*. Time from starting `fm respond` until the process exits and the full response is captured.
-- `generation_ms` — *derived*. `E2E - TTFT`, reported only when the answer arrives in more than one stdout chunk. When a single chunk carries the whole answer, prefill and decode are not separable and the value is unavailable instead of near zero.
-- `TPOT` — *derived*. `(E2E - TTFT) / (output_tokens - 1)`, reported when the answer has at least three output tokens so the interval is an average over two or more decode tokens rather than the inverse of a single chunk gap. Requires streaming and a token-counting `fm`.
-- `second_chunk_ms` — *proxy*. Time between the first and second streamed stdout chunks: a terminal-side signal for startup smoothness.
-- `chunk_gap` — *proxy*. Distribution of time between consecutive streamed stdout chunks. Useful for spotting streaming jitter; chunk-based, not token-based.
+- **Deliveries.** `fm` often writes the tail of a short answer as several `write()` calls well under a millisecond apart. Stdout chunks that arrive less than 5 ms after the previous one are therefore grouped into one *delivery*. Real streaming deltas on macOS 27.2 arrive about 50–250 ms apart, so this only removes write bursts. Generation time, TPOT, decode rate, `second_chunk_ms`, and `chunk_gap` are computed over deliveries; `stdout_chunks` still counts raw chunks.
+- `generation_ms` — *derived*. Last streamed delivery minus first streamed delivery, reported only when the answer arrives in more than one delivery. When a single chunk carries the whole answer, prefill and decode are not separable and the value is unavailable instead of near zero. Process teardown after the last chunk is excluded.
+- `TPOT` — *derived*. `generation_ms / (output_tokens - first_chunk_tokens)`: the tokens that arrived during the generation window divided into it. Reported when at least two tokens arrived after the first chunk, so the interval is an average rather than the inverse of a single chunk gap. Requires streaming and a token-counting `fm`. Before 0.8.0 the denominator was `output_tokens - 1`, which credited the ~20 first-chunk tokens to the decode window and overstated decode speed by up to several times on short answers.
+- `second_chunk_ms` — *proxy*. Time between the first and second streamed deliveries: a terminal-side signal for startup smoothness.
+- `chunk_gap` — *proxy*. Distribution of time between consecutive streamed deliveries. Useful for spotting streaming jitter; delivery-based, not token-based.
 - `prefill_tokens_per_second` — *proxy*. Input prompt tokens divided by TTFT seconds. Because TTFT includes process startup and first-token latency, this systematically understates true prefill speed and should be read as an upper-bound-constrained estimate, not a kernel measurement.
 - `tokens_per_second` — *derived*. Output tokens divided by E2E seconds for one request.
-- `decode_tokens_per_second` — *derived*. Output tokens after the first token divided by generation seconds.
+- `decode_tokens_per_second` — *derived*. Per run: output tokens that arrived after the first delivery divided by generation seconds. Short answers finish in two or three deliveries, so their per-run rate rests on few tokens. The table's DECODE/S column is therefore the token-weighted `decodeThroughput` (decode tokens summed over runs divided by the summed generation time), so long generations dominate. The `throughput` profile gives the most trustworthy decode figure (about 51–53 tokens/s on an M5 Pro with macOS 27.2).
 - `output token throughput` — *derived*. All successful output tokens for a model divided by that model's measured wall-clock window.
 - `total token throughput` — *derived*. Successful prompt and output tokens divided by the model's measured wall-clock window.
 - `RPS` — *measured*. Successful requests divided by the model's measured wall-clock window.
 - `goodput` — *derived*. Successful requests that also satisfy every provided SLO threshold. A run whose SLO metric is unavailable counts as not good, so an unverifiable SLO can never inflate goodput.
 - `goodput RPS` — *derived*. SLO-passing requests divided by the model's measured wall-clock window. Zero is reported when SLOs are set and nothing passes.
 - `repeatability` — *derived*. For repeated runs of the same prompt, the average share of runs that produced the most common normalized output hash.
-- `CV` — *derived*. Sample standard deviation divided by the mean. Reported only with two or more successful samples.
+- `CV` — *derived*. Run-to-run stability: the E2E coefficient of variation (sample standard deviation divided by the mean) of each prompt across its repeated runs, averaged over prompts (`stabilityCv` in JSON). Reported once at least one prompt has two successful runs. The pooled `latency.cv` over every sample is still in the JSON, but it mostly reflects that prompts have different lengths, so the table no longer shows it. Before 0.8.0 the CV column used the pooled value.
 - `95% CI` — *derived*. t-distribution confidence interval around the sample mean, using exact t critical values up to 30 degrees of freedom and the standard 2.0 / 1.96 approximations beyond that. Reported only with two or more successful samples. Treat it as context, not proof, at small sample sizes.
-- `quota` — *measured*, when the `fm` build exposes a quota command. Most current builds do not, in which case `fm-bench` reports quota as unavailable rather than empty.
+- `quota` — *measured*, when the `fm` build exposes a quota command. macOS 27.2 has `quota-usage` (the on-device `system` model reports "Not applicable", since quota only applies to `pcc`); macOS 27.0 has none, in which case `fm-bench` reports quota as unavailable rather than empty.
 
 ### Small samples
 
@@ -66,6 +67,8 @@ Use `--request-rate <rps>` to pace request starts independently of concurrency.
 
 `--warmup <n>` runs `n` unmeasured calls per model at the start of each operating point. The first `fm respond` after a cold start includes model load time, which can dominate a short prompt (observed at several hundred milliseconds on Apple silicon). Warmups are never mixed into the measured results.
 
+Since 0.8.0 the default is one warmup per model, so a default run reports steady-state latency. Pass `--warmup 0` to measure cold start deliberately. `warmup` is part of the suite key, so `compare` flags a 0.7.x report (warmup 0) against a 0.8.0 default run as a different suite.
+
 ## Failures and retries
 
 A failed call is recorded as a failed measurement with its error text, and is excluded from latency and throughput statistics. `--retry <n>` retries failed calls with exponential backoff before recording the failure; the report keeps the run count and records the number of attempts separately, so a retried success is still one sample and never silently duplicates work.
@@ -76,7 +79,9 @@ The `client` profile is a pragmatic local-machine mix inspired by MLPerf Client'
 
 ## Caveats
 
-Token counts come from the `fm` build's own token-counting command (`count-tokens` on current builds, `token-count` on older ones). If a build has no such command, token-derived metrics are reported as unavailable rather than estimated. `fm-bench` cannot judge semantic quality unless you provide your own prompt suite and inspect captured outputs with `--capture-output`.
+Token counts come from the `fm` build's own token-counting command (`count-tokens` on current builds, `token-count` on older ones). If a build has no such command, token-derived metrics are reported as unavailable rather than estimated.
+
+`count-tokens` counts its input as a prompt and adds a constant framing token: on macOS 27.2 it reports 2 for `a`, 3 for `a a`, and 4 for `a a a`. Once per run fm-bench counts those three strings, derives the overhead from the consistent per-word step (`overhead = count(a) − step`), and subtracts it from output and first-chunk token counts. The result is saved as `tokenCounter: { command, overhead, calibrated }`. If the three counts are not consistent, no correction is applied and `calibrated` is `false`. Prompt token counts are left exactly as `fm` reports them, because the framing token is part of what the model processes. `count-tokens` always uses the on-device system tokenizer, so token counts for `pcc` are system-tokenizer counts. `fm-bench` cannot judge semantic quality unless you provide your own prompt suite and inspect captured outputs with `--capture-output`.
 
 Client-side measurements include process startup, local queueing, model prefill, streaming, detokenization, and terminal pipe overhead. That is intentional for a command-line benchmark, but it is not the same as an internal model-kernel benchmark.
 

package/docs/releasing.md

@@ -1,61 +1,78 @@
 # Releasing
 
-`fm-bench` uses semver tags (`v*.*.*`). Pushing a tag runs the **Release** workflow: lint/test/package check, npm publish (with provenance), and a GitHub release whose body is taken from the matching section in **`CHANGELOG.md`** (plus a compare link).
+`fm-bench` uses semver tags (`v*.*.*`). A release tag runs the **Release** workflow: lint/test/package check, npm publish with provenance, a GitHub release whose body is the matching section of **`CHANGELOG.md`** (plus a compare link), and an install check of the published package.
 
-## Prerequisites
+## npm authentication
 
-- Repository secret **`NPM_TOKEN`**: npm automation token with publish access to `fm-bench`.
-- **`main`** is green on CI.
+The Release workflow runs on Node 24 (npm 11) with `id-token: write`, so it publishes through **npm trusted publishing** (OIDC) when the package trusts this workflow. No long-lived token is needed, and provenance is attached automatically. Set it up once (requires an npm login with publish rights and 2FA):
 
-If `NPM_TOKEN` is missing or expired, the Release workflow warns, skips the npm publish step, and still creates the GitHub release. The release notes then state that npm publish was skipped. Re-run the Release workflow with the tag once the token is fixed — an already-published version is skipped automatically.
+```sh
+npm login
+npm trust github fm-bench --repo dvnold/fm-bench --file release.yml
+npm trust list fm-bench
+```
+
+or on npmjs.com: **fm-bench → Settings → Trusted publishing → GitHub Actions**, organization/user `dvnold`, repository `fm-bench`, workflow `release.yml`.
+
+Without trusted publishing, npm falls back to the **`NPM_TOKEN`** repository secret (a granular access token with publish rights). npm limits those tokens to 90 days, so an old token fails with `E401`/`E403`/`E404`.
+
+When publishing fails, the workflow still creates or updates the GitHub release (its notes say npm publish failed) and then fails the run, so the problem is visible. Fix the authentication, then re-run the workflow for the tag. An already-published version is skipped.
+
+Provenance also requires `repository.url` in `package.json` to match the repository the workflow runs in (`git+https://github.com/dvnold/fm-bench.git`). Update it if the repository is renamed or transferred.
 
 ## Before tagging
 
 ```sh
 npm run check     # lint + tests + npm pack integrity
 npm run publish:dry-run
+actionlint        # if workflows changed
 ```
 
-Then bump `CHANGELOG.md` with a `## X.Y.Z` section; the release notes come from that section.
+Then add a `## X.Y.Z` section to `CHANGELOG.md`; the release notes come from that section.
 
 ## Option A — GitHub Actions (recommended)
 
 1. Open **Actions → Version → Run workflow**.
 2. Choose `patch`, `minor`, `major`, or an exact semver.
-3. The job runs `npm version`, pushes the commit and tag to `main`.
-4. The tag push triggers **Release** automatically.
+3. The job runs `npm version`, pushes the commit and tag to `main`, and dispatches **Release** for the new tag. (A tag pushed with the workflow's `GITHUB_TOKEN` does not trigger other workflows by itself, so Version starts Release explicitly.)
 
 ## Option B — Local
 
 ```sh
 npm ci && npm run check
 npm version minor   # or patch / major
-git push origin main --follow-tags
+git push origin main --follow-tags   # the tag push triggers Release
 ```
 
 ## Re-run Release without republishing
 
-If npm already has the version but the GitHub release failed (or vice versa), use **Actions → Release → Run workflow** and enter the existing tag (for example `v0.6.3`). The workflow skips npm publish when that version is already on the registry. If the GitHub release already exists, it **updates the release notes** from `CHANGELOG.md`.
+If npm already has the version but the GitHub release failed (or vice versa), use **Actions → Release → Run workflow** and enter the existing tag (for example `v0.8.0`). The workflow skips npm publish when that version is already on the registry. If the GitHub release already exists, it **updates the release notes** from `CHANGELOG.md`.
+
+```sh
+gh workflow run release.yml --repo dvnold/fm-bench -f tag=v0.8.0
+```
 
 Refresh notes locally without re-publishing:
 
 ```sh
-node scripts/changelog-release-notes.mjs 0.6.3 > notes.md
-gh release edit v0.6.3 --notes-file notes.md --repo devinoldenburg/fm-bench
+node scripts/changelog-release-notes.mjs 0.8.0 > notes.md
+gh release edit v0.8.0 --notes-file notes.md --repo dvnold/fm-bench
 ```
 
 ## Verifying a release
 
+The workflow installs the published version into a clean directory and runs it against the fake `fm`. To check by hand:
+
 ```sh
-git tag --list 'v0.7.0'
-gh release view v0.7.0 --repo devinoldenburg/fm-bench
+gh release view v0.8.0 --repo dvnold/fm-bench
 npm view fm-bench version            # registry version
+npm view fm-bench@0.8.0 dist.attestations --json   # provenance
 
 mkdir -p fm-bench-check && cd fm-bench-check
-npm install fm-bench@0.7.0
+npm install fm-bench@0.8.0
 ./node_modules/.bin/fm-bench --version
-./node_modules/.bin/fm-bench --help > /dev/null
 ./node_modules/.bin/fm-bench doctor
+./node_modules/.bin/fm-bench --profile quick --runs 3
 ```
 
 ## Dry run

package/docs/report-format.md

@@ -1,6 +1,6 @@
 # Report format (schema v1)
 
-Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines. The schema version stayed at `1` in 0.7.0: the new `capabilities`, `metrics`, and per-result `attempts` fields are additive, so older readers and older reports both keep working.
+Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines. The schema version is still `1`: the `capabilities`, `metrics`, and per-result `attempts` fields (0.7.0) and the `tokenCounter`, `models[].identity`, `firstChunkTokens`, and `stabilityCv` fields (0.8.0) are additive, so older readers and older reports both keep working.
 
 ## Top-level fields
 
@@ -15,10 +15,11 @@ Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** includ
 | `environment` | Host fingerprint: platform, arch, Node, hardware model, CPU, memory, macOS version/build, `fm` help digest, thermal/power snapshot |
 | `capabilities` | What the installed `fm` build exposes (see below) |
 | `metrics` | Per-metric availability and provenance for this run (see below) |
-| `suite` | Derived suite key + fingerprint for apples-to-apples comparison |
+| `tokenCounter` | `{ command, overhead, calibrated }`: the token-counting command and the framing overhead removed from output counts (see [methodology](./methodology.md#caveats)) |
+| `suite` | Derived suite key + fingerprint for apples-to-apples comparison. `suite.fingerprint.modelIdentities` maps model names to identities, for example `{ "system": "AFM 3 Core Advanced" }` |
 | `prompts` | Prompt ids, text, and token counts |
-| `models` | Discovered models and availability |
-| `summary` | Per-model / per-concurrency roll-up statistics |
+| `models` | Discovered models, availability, `reason` when unavailable, and `identity` as reported by `fm models` |
+| `summary` | Per-model / per-concurrency roll-up statistics, including `stabilityCv` (run-to-run CV), `decodeThroughput` (token-weighted decode tokens/s), and `firstChunkTokens` |
 | `results` | Per-run measurements (optional `output` when `--capture-output`) |
 
 ## `capabilities`
@@ -29,13 +30,18 @@ Detected once per run from `fm --help` and `fm respond --help`.
 {
   "capabilities": {
     "bin": "fm",
-    "digest": "921a7839714e3707",
-    "commands": ["available", "chat", "count-tokens", "license", "respond", "schema", "serve"],
-    "models": [{ "name": "system", "description": "On-device Apple Foundation Model" }],
+    "digest": "a45f2838f399c5da",
+    "commands": ["chat", "config", "count-tokens", "license", "models", "quota-usage", "respond", "schema", "serve"],
+    "models": [
+      { "name": "system", "description": "On-device Apple Foundation Model" },
+      { "name": "pcc", "description": "Apple Foundation Model on Private Cloud Compute" }
+    ],
     "features": {
       "tokenCounting": true,
       "tokenCountCommand": "count-tokens",
-      "quota": false,
+      "modelListCommand": "models",
+      "license": true,
+      "quota": true,
       "streaming": true,
       "modelSelection": true,
       "instructions": true,
@@ -47,11 +53,13 @@ Detected once per run from `fm --help` and `fm respond --help`.
       "structuredOutput": true,
       "server": true
     },
-    "warnings": ["this fm build exposes no quota command, so quota is not reported"]
+    "warnings": []
   }
 }
 ```
 
+On a macOS 27.0 build the same block has `"modelListCommand": "available"`, `"quota": false`, and the warning `this fm build exposes no quota command, so quota is not reported`.
+
 `digest` is a short hash of the normalized `fm --help` output, so a report records which CLI surface produced it. Reports from two different `fm` builds are not directly comparable even on the same machine.
 
 ## `metrics`
@@ -72,8 +80,8 @@ Each entry states how a metric was obtained for this run and whether it was avai
       "label": "quota",
       "kind": "measured",
       "source": "fm quota-usage",
-      "available": false,
-      "unavailableReason": "this fm build exposes no quota command"
+      "available": true,
+      "unavailableReason": ""
     }
   }
 }
@@ -87,13 +95,14 @@ Per-run rows in `results` carry:
 |-------|-------|
 | `attempts` | Total `fm` invocations for this measured run, including retries. `1` when no retry was needed. |
 | `ok` | Whether the call produced a response and exited cleanly. |
-| `firstTokenMs`, `generationMs`, `tpotMs` | `null` when the run cannot supply them (no streaming, single-chunk answer, failed run). |
-| `promptTokens`, `outputTokens`, `tokensPerSecond`, `decodeTokensPerSecond`, `prefillTokensPerSecond` | `null` when the `fm` build cannot count tokens. |
+| `firstTokenMs`, `generationMs`, `tpotMs` | `null` when the run cannot supply them (no streaming, single-chunk answer, failed run). `generationMs` runs from the first to the last streamed chunk. |
+| `promptTokens`, `outputTokens`, `tokensPerSecond`, `decodeTokensPerSecond`, `prefillTokensPerSecond` | `null` when the `fm` build cannot count tokens. `promptTokens` is fm's own count; `outputTokens` has `tokenCounter.overhead` removed. |
+| `firstChunkTokens`, `decodeTokens` | Output tokens carried by the first streamed delivery (overhead removed), and the tokens that arrived after it. `tpotMs` and `decodeTokensPerSecond` use only `decodeTokens`. `null` for single-delivery or non-streamed answers. |
 | `error` | Actionable failure text (`timed out after 30000ms`, `fm exited with code 3`, the first actionable line of `fm` stderr). |
 
 ## CSV
 
-`--format csv` and `--out runs.csv` write per-run rows. Columns are stable; `attempts` was added in 0.7.0. Text cells that begin with `=`, `+`, `@`, or a non-numeric `-` are prefixed with a single quote so spreadsheets do not execute prompt or model output as a formula.
+`--format csv` and `--out runs.csv` write per-run rows. Columns are stable; `attempts` was added in 0.7.0, and `first_chunk_tokens` was appended as the last column in 0.8.0 so positional readers keep working. Text cells that begin with `=`, `+`, `@`, or a non-numeric `-` are prefixed with a single quote so spreadsheets do not execute prompt or model output as a formula.
 
 ## Sharing results
 

package/docs/supported-platforms.md

@@ -5,7 +5,7 @@
 ## Requirements
 
 - **macOS 27.0 or newer** — Apple's `fm` CLI is preinstalled starting with macOS 27. Older macOS releases do not ship `fm`, so `fm-bench` cannot run there.
-- **Node.js 20 or newer**.
+- **Node.js 22 or newer**. Node 20 reached end of life in April 2026; CI tests Node 22, 24, and 26.
 - **Apple Intelligence enabled** on the device.
 
 ## Version enforcement
@@ -31,13 +31,18 @@ Support is capability-based rather than version-pinned. The CLI probes the insta
 
 ## Models
 
-`fm-bench` benchmarks exactly the models the installed `fm` reports — it does not assume that any particular cloud or adapter model exists. On the verified macOS 27.0 build that is the on-device `system` model only. If a build adds models (for example a Private Cloud Compute model), they are discovered automatically and reported with their own availability.
+`fm-bench` benchmarks exactly the models the installed `fm` reports — it does not assume that any particular cloud or adapter model exists. macOS 27.0 ships the on-device `system` model only; macOS 27.2 adds `pcc` (Apple Foundation Model on Private Cloud Compute). New models are discovered automatically and reported with their own availability and identity.
 
-Requesting only models the build cannot run exits with code `2` before any benchmark starts, and `fm-bench models` shows the same reasons without failing:
+`fm` only serves `pcc` to the Terminal app. From editor terminals and other contexts it reports `Private Cloud Compute is not available in this context. Please use the Terminal app.`, and fm-bench shows that reason unchanged. Run fm-bench from Terminal to benchmark `pcc`.
+
+Requesting only models that cannot run right now exits with code `2` before any benchmark starts, and `fm-bench models` shows the same reasons without failing:
 
 ```text
 fm-bench: No benchmark was run: none of the requested models are usable right now.
   requested: pcc
-  pcc: not supported by this fm build (supported: system)
+  models reported by this fm build: system, pcc
+  pcc: Private Cloud Compute is not available in this context. Please use the Terminal app.
   run "fm-bench models" to see availability and reasons
 ```
+
+On a build that does not have the model at all, the reason reads `not supported by this fm build (supported: system)` instead.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fm-bench",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {
@@ -26,25 +26,31 @@
   },
   "keywords": [
     "apple",
+    "apple-intelligence",
+    "apple-silicon",
     "foundation-models",
     "fm",
+    "llm",
     "benchmark",
+    "benchmarking",
+    "latency",
+    "throughput",
     "macos",
     "cli"
   ],
   "author": "Devin Oldenburg",
   "license": "MIT",
   "engines": {
-    "node": ">=20"
+    "node": ">=22"
   },
   "repository": {
     "type": "git",
-    "url": "git+https://github.com/devinoldenburg/fm-bench.git"
+    "url": "git+https://github.com/dvnold/fm-bench.git"
   },
   "bugs": {
-    "url": "https://github.com/devinoldenburg/fm-bench/issues"
+    "url": "https://github.com/dvnold/fm-bench/issues"
   },
-  "homepage": "https://github.com/devinoldenburg/fm-bench#readme",
+  "homepage": "https://github.com/dvnold/fm-bench#readme",
   "publishConfig": {
     "access": "public",
     "registry": "https://registry.npmjs.org/"