fm-bench 0.4.4 → 0.5.1

package/README.md

@@ -1,41 +1,34 @@
 # fm-bench
 
-`fm-bench` is a dynamic benchmark CLI for Apple's `fm` command on macOS 27 and newer.
+Benchmark Apple's `fm` command on macOS 27+.
 
-It discovers the models reported by `fm --help`, checks availability with `fm available`, runs repeatable prompt suites through `fm respond`, counts tokens with `fm token-count`, shows live progress while it works, and prints terminal tables with latency, throughput, stability, goodput, and streaming-quality stats.
+Measure latency, throughput, streaming smoothness, stability, and goodput across Apple Foundation Models — with repeatable prompt suites and JSON/CSV reports for automation.
 
-Apple introduced the preinstalled `fm` command for macOS 27 as part of the Foundation Models tooling. `fm-bench` intentionally shells out to the system `fm` binary instead of linking private APIs, so it can adapt as Apple adds models or changes availability.
+## Why fm-bench exists
 
-## Install
+Apple's Foundation Models can run on-device, through Private Cloud Compute, and through model adapters. That makes raw model quality only half the story.
 
-```sh
-npm install -g fm-bench
-```
+For real apps, the important questions are:
 
-You can also install directly from GitHub:
+- How fast does the first token arrive?
+- Does streaming stay smooth?
+- How stable is latency over repeated runs?
+- What happens under concurrency?
+- Which model/hardware pair meets an interactive SLO?
 
-```sh
-npm install -g --install-links git+https://github.com/devinoldenburg/fm-bench.git
-```
-
-For local development from this repository:
-
-```sh
-npm install
-npm link
-fm-bench doctor
-```
+`fm-bench` answers those questions with repeatable local benchmarks. Think of it as GeekBench for Apple Foundation Models — run it, get numbers, compare across hardware, models, and macOS updates.
 
 ## Quick Start
 
 ```sh
+npm install -g fm-bench
 fm-bench
 ```
 
-Example output:
+One command discovers your models, runs the standard prompt suite, and prints a full benchmark report:
 
 ```text
-fm-bench 0.4.0 | darwin/arm64 | fm
+fm-bench 0.5.0 | darwin/arm64 | fm
 prompts 5 | runs 3 | concurrency 1,2 | stream on | measured 30 | failed 0 | skipped 0 | elapsed 42.10s | SLO TTFT<=750ms,E2E<=4.00s
 
 ┌───┬────────┬────────┬─────────┬──────┬──────┬──────────┬──────┬──────────┬─────┬─────┐
@@ -46,63 +39,173 @@ prompts 5 | runs 3 | concurrency 1,2 | stream on | measured 30 | failed 0 | skip
 └───┴────────┴────────┴─────────┴──────┴──────┴──────────┴──────┴──────────┴─────┴─────┘
 ```
 
-## Commands
+Wide terminals add TTFT P95, TPOT, decode/prefill throughput, chunk-gap smoothness, and 95% CI columns. Narrow terminals switch to compact model cards automatically.
+
+## Install
 
 ```sh
-fm-bench [run] [options]
-fm-bench models [options]
-fm-bench legend [options]
-fm-bench doctor [options]
+npm install -g fm-bench
 ```
 
-`run` is the default command. It benchmarks all models discovered from `fm` and skips models that are currently unavailable.
+Install directly from GitHub (always latest):
 
-`models` lists discovered models, availability, descriptions, and quota output.
+```sh
+npm install -g --install-links git+https://github.com/devinoldenburg/fm-bench.git
+```
+
+Local development:
+
+```sh
+npm install && npm link
+fm-bench doctor   # verify your setup
+```
+
+**Requirements:** macOS 27+, Node.js 20+, Apple Intelligence enabled.
 
-`legend` explains every terminal table column, compact-card field, model-list column, and color rule. It does not run `fm`.
+## Commands
 
-`doctor` checks Node, macOS, `fm`, and model availability.
+| Command | What it does |
+|---------|-------------|
+| `fm-bench` | Run the full benchmark (default) |
+| `fm-bench models` | List discovered models, availability, and quota |
+| `fm-bench compare <a.json> <b.json>` | Regression diff: before/after metrics with color-coded deltas |
+| `fm-bench history [dir]` | Chronological trend table from a directory of saved reports |
+| `fm-bench legend` | Definitions for every table column and color rule |
+| `fm-bench doctor` | Environment check: Node, macOS, `fm`, CPU, memory, thermals, battery |
 
-## Benchmark Options
+## Common Recipes
 
 ```sh
+# Quick smoke test
+fm-bench --profile quick
+
+# Standard 5-run benchmark with SLO budgets
+fm-bench --runs 5 --slo-ttft-ms 750 --slo-e2e-ms 4000
+
+# Sweep concurrency to find your throughput ceiling
+fm-bench --sweep-concurrency 1,2,4 --runs 3
+
+# Stress both on-device and PCC models
 fm-bench --models system,pcc --runs 3 --profile stress
-fm-bench --models system --runs 5 --profile interactive
-fm-bench --models system --runs 3 --profile throughput --warmup 1
-fm-bench --models system --profile interactive --sweep-concurrency 1,2,4
-fm-bench --profile client --sweep-concurrency 1,2 --request-rate 0.5 --ramp-up-ms 2000
-fm-bench --models system --runs 5 --slo-ttft-ms 750 --slo-e2e-ms 4000
-fm-bench --prompt "Reply with exactly: ok" --runs 5
-fm-bench --prompt-file prompts.json --format json --out reports/bench.json
-fm-bench --format csv --out reports/bench.csv
-fm-bench legend
-fm-bench legend --json
+
+# Reasoning and coding workloads
+fm-bench --profile reasoning --runs 5
+fm-bench --profile coding --runs 3 --histogram
+
+# Archive runs and compare before/after a macOS update
+fm-bench --output-dir reports/ --tag before-update
+fm-bench --output-dir reports/ --tag after-update
+fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json
+
+# Fail CI when SLOs regress
+fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5
+
+# Save JSON for automation
+fm-bench --json --out bench.json
+fm-bench --format csv --out bench.csv
 ```
 
-Useful flags:
-
-- `--models <list>`: comma-separated or repeated model names.
-- `--runs <n>`: measured runs per prompt/model.
-- `--warmup <n>`: warmup runs per model before measurement.
-- `--concurrency <n>`: parallel `fm` processes.
-- `--sweep-concurrency <list>`: run separate measured operating points, such as `1,2,4`.
-- `--request-rate <rps>`: pace request starts at a target requests-per-second rate.
-- `--ramp-up-ms <n>`: gradually ramp request pacing over `n` milliseconds.
-- `--timeout-ms <n>`: timeout per `fm` call.
-- `--slo-ttft-ms <n>`, `--slo-e2e-ms <n>`, `--slo-tpot-ms <n>`: count goodput against latency budgets.
-- `--profile quick|standard|interactive|throughput|client|stress`: built-in prompt suite.
-- `--prompt <text>`: custom prompt, repeatable.
-- `--prompt-file <file>`: JSON, JSONL, or blank-line separated text prompts.
-- `--instructions <text>`: passed to `fm respond`.
-- `--available-only`: hide unavailable discovered models.
-- `--capture-output`: include raw model output in JSON reports.
-- `--json`, `--csv`, `--format table|json|csv`: choose output format.
-- `--ascii`: use plain ASCII table borders.
-- `--color`, `--no-color`: force or disable semantic ANSI colors. Colors are automatic on TTYs.
-- `--progress`, `--no-progress`: force or disable the live progress status line on stderr.
-- `--compact`: force the narrow terminal layout.
-- `--width <n>`: render as if the terminal has `n` columns.
-- `--out <file>`: save a report.
+## Options Reference
+
+**Workload**
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `-m, --models <list>` | all | Comma-separated or repeated model names |
+| `-r, --runs <n>` | 1 | Measured runs per prompt/model |
+| `--warmup <n>` | 0 | Warmup runs per model before measurement |
+| `-c, --concurrency <n>` | 1 | Parallel `fm` processes |
+| `--sweep-concurrency <list>` | — | Separate operating points, e.g. `1,2,4` |
+| `--request-rate <rps>` | — | Pace request starts at a target rate |
+| `--ramp-up-ms <n>` | 0 | Gradually ramp pacing over `n` ms |
+| `--timeout-ms <n>` | 60000 | Timeout per `fm` call |
+| `--retry <n>` | 0 | Retry failed calls with exponential backoff (500ms–4s) |
+| `--profile <name>` | standard | Built-in prompt suite (see Profiles) |
+| `-p, --prompt <text>` | — | Custom prompt, repeatable |
+| `--prompt-file <file>` | — | JSON, JSONL, or blank-line separated prompts |
+| `-i, --instructions <text>` | — | Passed to `fm respond` |
+
+**Quality Gates**
+
+| Flag | Description |
+|------|-------------|
+| `--slo-ttft-ms <n>` | Count a run as good only if TTFT ≤ n ms |
+| `--slo-e2e-ms <n>` | Count a run as good only if E2E latency ≤ n ms |
+| `--slo-tpot-ms <n>` | Count a run as good only if TPOT ≤ n ms |
+| `--ci` | Exit 1 if any run fails or any SLO is violated (for pipelines) |
+| `--fail-fast` | Stop after the first failed run |
+
+**Output**
+
+| Flag | Description |
+|------|-------------|
+| `--json` / `--csv` | Output format (also `--format table\|json\|csv`) |
+| `-o, --out <file>` | Save a report to a file |
+| `--output-dir <dir>` | Auto-save a timestamped JSON report to a directory |
+| `--tag <name>` | Label this run; repeatable; appears in payload and header |
+| `--note <text>` | Freeform annotation in payload and header |
+| `--histogram` | Print ASCII latency distribution chart after the report |
+| `--capture-output` | Include raw model output in JSON reports |
+| `-v, --verbose` | Append per-run CSV after the summary table |
+
+**Display**
+
+| Flag | Description |
+|------|-------------|
+| `--color` / `--no-color` | Force or disable ANSI colors (auto on TTYs) |
+| `--ascii` | Plain ASCII table borders instead of Unicode |
+| `--compact` | Force narrow terminal layout |
+| `--width <n>` | Render as if the terminal is `n` columns wide |
+| `--progress` / `--no-progress` | Force or disable the live progress line |
+
+## Prompt Profiles
+
+Nine built-in suites, choose the one that matches your use case:
+
+| Profile | Prompts | Best for |
+|---------|---------|----------|
+| `quick` | 1 | Smoke test, fast health check |
+| `standard` | 3 | Default — short chat, JSON generation, medium output |
+| `interactive` | 3 | Conversational latency (TTFT-heavy) |
+| `throughput` | 3 | Longer generation, token throughput signal |
+| `client` | 5 | Real-world mix: chat, content, extraction, summarization, code |
+| `stress` | 5 | High-load mix with math and reasoning |
+| `reasoning` | 5 | Multi-step logic, estimation, debugging — capability + speed |
+| `coding` | 5 | Code review, refactoring, algorithms, system design |
+| `creative` | 5 | Product copy, analogies, commit messages, docs |
+
+## Regression Tracking
+
+Track performance across macOS updates, model changes, or hardware swaps:
+
+```sh
+# Before
+fm-bench --profile coding --runs 5 --output-dir reports/ --tag before
+
+# After the change
+fm-bench --profile coding --runs 5 --output-dir reports/ --tag after
+
+# See what changed
+fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json
+```
+
+The compare output shows each model/concurrency row with the before value, a color-coded percent delta (green = improvement, red = regression), and the after value — for TTFT, E2E, TPOT, tokens/s, RPS, success rate, and CV.
+
+```sh
+# View the full trend over time
+fm-bench history reports/
+```
+
+## CI Integration
+
+Gate deployments or model updates on benchmark quality:
+
+```sh
+# Fails with exit code 1 if TTFT > 750ms or E2E > 4s on any run
+fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5
+```
+
+Prints `fm-bench ci: PASS` or `fm-bench ci: FAIL — <reason>` to stderr. Designed for GitHub Actions, Buildkite, or any shell-based pipeline.
 
 ## Prompt Files
 
@@ -126,95 +229,52 @@ Plain text files are split on blank lines.
 
 ## Metrics
 
-`fm-bench` reports:
+**Latency** — TTFT (p50/p95), E2E (p50/p95/p99), TPOT (p50/p95), 95% confidence interval, coefficient of variation (CV).
 
-- TTFT, or time to first streamed output.
-- E2E latency, or full response wall-clock latency.
-- TPOT, or decode time per output token after the first output token.
-- second-chunk delay and chunk-gap p95 as terminal-side streaming smoothness signals.
-- prefill tokens per second, or prompt tokens divided by TTFT.
-- output tokens per second per request.
-- total output token throughput across the measured window.
-- total token throughput, including prompt and output tokens.
-- requests per second across the measured window.
-- goodput percentage and goodput RPS when SLO flags are set.
-- coefficient of variation (CV) and confidence interval context for stability.
-- prompt and output token counts.
-- p50, p95, and p99 tail latency views.
-- repeatability across repeated runs of the same prompt.
-- success and failure counts.
-- unavailable model notes.
+**Throughput** — prefill tokens/s, decode tokens/s, output tokens/s per request, aggregate system tokens/s, requests per second.
 
-Token counts come from `fm token-count --quiet`. If `fm` cannot count a response, token fields are left blank while character throughput is still reported.
+**Streaming quality** — second-chunk delay, chunk-gap p95. Captured from `stdout` chunks during streaming runs.
 
-Measured runs stream by default so `fm-bench` can capture TTFT and streaming smoothness. Use `--no-stream` if you need buffered `fm respond` behavior; TTFT, TPOT, second-chunk, and chunk-gap fields that depend on streaming will be blank.
+**Reliability** — success rate, goodput rate and RPS against SLO budgets, repeatability (most common output hash frequency across repeated runs).
 
-Terminal output is responsive. Wide terminals show full scoreboard and detail tables, medium terminals show a tighter operating-point table, and narrow terminals switch to compact model cards. Long `NOTE`, `DESCRIPTION`, and model quota cells wrap so error context stays visible instead of being hidden behind ellipses. Use `--width` to preview a layout and `--ascii` for log systems that do not render Unicode borders well.
+**Stability** — CV (stddev/mean for E2E latency); green ≤10%, yellow ≤25%, red >25%.
 
-## Table Legend
+Token counts come from `fm token-count --quiet`. If `fm` cannot count tokens, those fields are blank while character throughput is still reported.
 
-Benchmark output stays focused on results and does not print the metric legend footer. Use `fm-bench legend` when you want definitions for every table column and color rule:
+Terminal layout is responsive: wide → full scoreboard + detail tables, medium → tighter single table, narrow → compact model cards. Use `--width` to preview any layout and `--ascii` for log-friendly output.
 
-```sh
-fm-bench legend
-fm-bench legend --json
-fm-bench legend --csv
-```
+## Colors and Legend
 
-The legend wraps long definitions and rules to fit the terminal width instead of hiding text behind ellipses.
+Table output is color-coded on interactive terminals — **green** is better/passing, **yellow** is marginal/partial, **red** is failing/unstable. Fixed thresholds apply to success rate, goodput, CV, and repeatability. Latency uses SLO thresholds when set, otherwise lower-is-better relative ranking. Throughput uses higher-is-better relative ranking.
 
-## Live Progress
-
-Interactive terminal runs show a single-line status indicator on stderr while prompts are loaded, models are inspected, tokens are counted, warmups run, and benchmark jobs complete. The final report still prints to stdout, so `--json`, `--csv`, and `--out` remain automation-friendly.
-
-Progress is automatic for table output on TTYs. Use `--progress` to force it or `--no-progress` to keep the terminal completely quiet until the report is ready.
-
-## Terminal Colors
-
-Table output uses semantic ANSI color on interactive terminals:
-
-- green: passing, steadier, or better than the current comparison set.
-- yellow: marginal, partial, or near a budget.
-- red: failing a budget, unstable, or slower/lower than peers.
-
-Success rate, goodput, repeatability, and CV use fixed benchmark thresholds. CV is green at `<=10%`, yellow at `<=25%`, and red above `25%` because higher CV means less steady latency. Throughput columns use relative ranking within the current run because “good” depends on the machine, model, prompt mix, and concurrency. TTFT, E2E, and TPOT use SLO thresholds when you pass `--slo-ttft-ms`, `--slo-e2e-ms`, or `--slo-tpot-ms`; otherwise they use lower-is-better relative ranking across the models and operating points in the report.
+```sh
+fm-bench legend          # full column definitions and color rules
+fm-bench legend --json   # machine-readable
+```
 
-Use `--color` to force ANSI colors in captured logs, or `--no-color` for plain output. `NO_COLOR=1` disables automatic color and `FORCE_COLOR=1` enables it.
+`NO_COLOR=1` disables color; `FORCE_COLOR=1` or `--color` enables it. `--ascii` switches to plain ASCII borders for log systems.
 
-See [docs/methodology.md](docs/methodology.md) for the benchmark methodology and source references.
+A live single-line progress indicator runs on stderr during interactive sessions. The final report always goes to stdout — `--json`, `--csv`, and `--out` stay automation-friendly.
 
 ## Requirements
 
-- macOS 27 or newer for Apple's `fm` CLI.
+- macOS 27 or newer (Apple's `fm` CLI is preinstalled).
 - Node.js 20 or newer.
-- Apple Intelligence and model availability configured for the machine.
+- Apple Intelligence enabled on the device.
 
-Private Cloud Compute (`pcc`) availability depends on Apple's current eligibility and context. If `fm available --model pcc` reports unavailable, `fm-bench` will show it as skipped.
+`pcc` (Private Cloud Compute) availability depends on Apple's current eligibility. `fm-bench` shows it as skipped if `fm available --model pcc` reports unavailable.
+
+See [docs/methodology.md](docs/methodology.md) for benchmark methodology and metric references.
 
 ## Development
 
 ```sh
 npm install
-npm test
+npm test     # node --test
 npm run lint
-npm pack
-```
-
-The package has no runtime npm dependencies.
-
-## Releases
-
-Releases are tag-driven:
-
-```sh
-npm run release:patch
-npm run release:minor
-npm run release:major
 ```
 
-Pushing a `v*.*.*` tag runs the GitHub Release workflow, publishes to npm using the repository `NPM_TOKEN` secret, and creates a GitHub Release with generated notes.
-
-Maintainers can also run the **Version** workflow manually from GitHub Actions to bump the version and push the tag.
+No runtime npm dependencies.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fm-bench",
-  "version": "0.4.4",
+  "version": "0.5.1",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {

package/src/bench.js

@@ -229,11 +229,18 @@ async function runScenario(context) {
 }
 
 async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt) {
+  const maxAttempts = 1 + Math.max(0, options.retry ?? 0);
   const startOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
-  const response = await respond(fmBin, job.model.name, job.prompt.prompt, {
-    ...options,
-    stream: options.stream
-  });
+  let response;
+  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
+    response = await respond(fmBin, job.model.name, job.prompt.prompt, {
+      ...options,
+      stream: options.stream
+    });
+    if (response.ok || attempt >= maxAttempts) break;
+    const backoffMs = Math.min(500 * 2 ** (attempt - 1), 4000);
+    await new Promise((resolve) => setTimeout(resolve, backoffMs));
+  }
   const endOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
   const outputTokens = response.ok
     ? await countTokens(fmBin, response.output, options)
@@ -338,7 +345,10 @@ function publicOptions(options) {
       e2eMs: options.sloE2eMs || null,
       tpotMs: options.sloTpotMs || null
     },
-    instructions: options.instructions ? '[set]' : ''
+    instructions: options.instructions ? '[set]' : '',
+    retry: options.retry ?? 0,
+    tags: options.tags?.length ? options.tags : [],
+    note: options.note ?? null
   };
 }
 

package/src/cli.js

@@ -1,10 +1,12 @@
 import fs from 'node:fs/promises';
 import { createRequire } from 'node:module';
 import { inspectModels, runBenchmark } from './bench.js';
+import { diffReports, renderCompareReport } from './compare.js';
+import { loadHistory, renderHistoryReport } from './history.js';
 import { runProcess } from './process.js';
 import { createProgress } from './progress.js';
 import { flattenResults, toCsv, writeReport } from './report.js';
-import { legendEntries, renderBenchmarkReport, renderLegend, renderModelsTable } from './table.js';
+import { legendEntries, renderBenchmarkReport, renderLatencyHistogram, renderLegend, renderModelsTable } from './table.js';
 
 const require = createRequire(import.meta.url);
 const packageJson = require('../package.json');
@@ -38,6 +40,16 @@ export async function runCli(argv = process.argv.slice(2)) {
     return;
   }
 
+  if (parsed.command === 'compare') {
+    await runCompare(parsed, renderOptions(parsed));
+    return;
+  }
+
+  if (parsed.command === 'history') {
+    await runHistory(parsed, renderOptions(parsed));
+    return;
+  }
+
   if (parsed.command === 'models') {
     const inspection = await inspectModels(parsed);
     if (parsed.format === 'json') {
@@ -48,6 +60,11 @@ export async function runCli(argv = process.argv.slice(2)) {
     return;
   }
 
+  if (parsed.ci) {
+    if (parsed.color === 'auto') parsed.color = 'never';
+    if (parsed.progress === 'auto') parsed.progress = 'never';
+  }
+
   const progress = createProgress({
     ...renderOptions(parsed),
     enabled: resolveProgress(parsed),
@@ -70,6 +87,10 @@ export async function runCli(argv = process.argv.slice(2)) {
     console.log(toCsv(flattenResults(payload.results)));
   } else {
     console.log(renderBenchmarkReport(payload, renderOptions(parsed)));
+    if (parsed.histogram) {
+      console.log();
+      console.log(renderLatencyHistogram(payload.results, renderOptions(parsed)));
+    }
     if (parsed.verbose) {
       console.log();
       console.log(toCsv(flattenResults(payload.results)));
@@ -83,11 +104,62 @@ export async function runCli(argv = process.argv.slice(2)) {
       console.error(`Saved ${reportFormat.toUpperCase()} report to ${written}`);
     }
   }
+
+  if (parsed.outputDir) {
+    const stamp = payload.startedAt.replace(/[:.]/g, '-').replace('T', '_').slice(0, 19);
+    const firstModel = parsed.models?.flatMap((m) => String(m).split(',')).map((m) => m.trim()).filter(Boolean)[0] || 'all';
+    const modelSlug = firstModel.replace(/[^a-z0-9]/gi, '_');
+    const filename = `fm-bench_${stamp}_${modelSlug}.json`;
+    const filePath = `${parsed.outputDir}/${filename}`;
+    const written = await writeReport(filePath, payload, 'json');
+    if (parsed.format !== 'json') {
+      console.error(`Saved JSON report to ${written}`);
+    }
+  }
+
+  if (parsed.ci) {
+    const ciResult = evaluateCi(payload);
+    if (!ciResult.passed) {
+      const reasons = ciResult.reasons.join('; ');
+      console.error(`fm-bench ci: FAIL — ${reasons}`);
+      const error = new Error(`CI checks failed: ${reasons}`);
+      error.exitCode = 1;
+      throw error;
+    }
+    console.error(`fm-bench ci: PASS`);
+  }
+}
+
+function evaluateCi(payload) {
+  const reasons = [];
+  const totalFailed = payload.summary.reduce((sum, item) => sum + item.failures, 0);
+  if (totalFailed > 0) {
+    reasons.push(`${totalFailed} run(s) failed`);
+  }
+
+  const hasSlo = payload.options?.slo && (
+    payload.options.slo.ttftMs || payload.options.slo.e2eMs || payload.options.slo.tpotMs
+  );
+  if (hasSlo) {
+    for (const item of payload.summary) {
+      if (!item.available) continue;
+      if (item.goodputRate != null && item.goodputRate < 1) {
+        const pct = Math.round((1 - item.goodputRate) * 100);
+        reasons.push(`${item.model} c${item.concurrency ?? 1}: ${pct}% of runs violated SLO`);
+      }
+    }
+  }
+
+  return {
+    passed: reasons.length === 0,
+    reasons
+  };
 }
 
 export function parseArgs(argv) {
   const options = {
     command: 'run',
+    compareFiles: [],
     models: [],
     prompts: [],
     runs: 1,
@@ -107,16 +179,21 @@ export function parseArgs(argv) {
     captureOutput: false,
     availableOnly: false,
     failFast: false,
+    retry: 0,
+    ci: false,
+    tags: [],
+    note: null,
     verbose: false,
     ascii: false,
     color: 'auto',
     progress: 'auto',
     compact: false,
-    width: null
+    width: null,
+    histogram: false
   };
 
   const args = [...argv];
-  if (args[0] && !args[0].startsWith('-') && ['run', 'models', 'doctor', 'legend', 'metrics', 'help'].includes(args[0])) {
+  if (args[0] && !args[0].startsWith('-') && ['run', 'models', 'doctor', 'legend', 'metrics', 'compare', 'history', 'help'].includes(args[0])) {
     options.command = args.shift();
   }
 
@@ -189,6 +266,9 @@ export function parseArgs(argv) {
         break;
       case '--profile':
         options.profile = requireValue(arg, args);
+        if (!['quick', 'standard', 'interactive', 'throughput', 'client', 'stress', 'reasoning', 'coding', 'creative'].includes(options.profile)) {
+          throw new Error(`--profile must be one of: quick, standard, interactive, throughput, client, stress, reasoning, coding, creative`);
+        }
         break;
       case '-i':
       case '--instructions':
@@ -248,10 +328,16 @@ export function parseArgs(argv) {
       case '--width':
         options.width = parsePositiveInt(requireValue(arg, args), arg);
         break;
+      case '--histogram':
+        options.histogram = true;
+        break;
       case '-o':
       case '--out':
         options.out = requireValue(arg, args);
         break;
+      case '--output-dir':
+        options.outputDir = requireValue(arg, args);
+        break;
       case '--capture-output':
         options.captureOutput = true;
         break;
@@ -261,6 +347,18 @@ export function parseArgs(argv) {
       case '--fail-fast':
         options.failFast = true;
         break;
+      case '--retry':
+        options.retry = parseNonNegativeInt(requireValue(arg, args), arg);
+        break;
+      case '--ci':
+        options.ci = true;
+        break;
+      case '--tag':
+        options.tags.push(requireValue(arg, args));
+        break;
+      case '--note':
+        options.note = requireValue(arg, args);
+        break;
       case '-v':
       case '--verbose':
         options.verbose = true;
@@ -275,8 +373,14 @@ export function parseArgs(argv) {
         if (arg.startsWith('-')) {
           throw new Error(`Unknown option: ${arg}`);
         }
-        options.prompts.push([arg, ...args].join(' '));
-        args.length = 0;
+        if (options.command === 'compare') {
+          options.compareFiles.push(arg);
+        } else if (options.command === 'history') {
+          options.historyDir = arg;
+        } else {
+          options.prompts.push([arg, ...args].join(' '));
+          args.length = 0;
+        }
         break;
     }
   }
@@ -284,6 +388,64 @@ export function parseArgs(argv) {
   return options;
 }
 
+async function runHistory(options, renderOpts) {
+  const dir = options.historyDir || options.outputDir || '.';
+  const reports = await loadHistory(dir);
+
+  if (options.format === 'json') {
+    const data = reports.map(({ filePath, report }) => ({
+      file: filePath,
+      startedAt: report.startedAt,
+      version: report.version,
+      summary: report.summary
+    }));
+    console.log(JSON.stringify(data, null, 2));
+  } else {
+    console.log(renderHistoryReport(reports, renderOpts));
+  }
+}
+
+async function runCompare(options, renderOpts) {
+  const files = options.compareFiles;
+  if (files.length < 2) {
+    throw new Error('compare requires two JSON report files: fm-bench compare before.json after.json');
+  }
+  if (files.length > 2) {
+    throw new Error('compare accepts exactly two JSON report files');
+  }
+
+  const [beforePath, afterPath] = files;
+  const [beforeText, afterText] = await Promise.all([
+    fs.readFile(beforePath, 'utf8'),
+    fs.readFile(afterPath, 'utf8')
+  ]);
+
+  let before, after;
+  try {
+    before = JSON.parse(beforeText);
+  } catch {
+    throw new Error(`Cannot parse ${beforePath} as JSON`);
+  }
+  try {
+    after = JSON.parse(afterText);
+  } catch {
+    throw new Error(`Cannot parse ${afterPath} as JSON`);
+  }
+
+  const diff = diffReports(before, after);
+
+  if (options.format === 'json') {
+    console.log(JSON.stringify(diff, null, 2));
+  } else {
+    console.log(renderCompareReport(diff, renderOpts));
+  }
+
+  if (options.out) {
+    await fs.writeFile(options.out, `${JSON.stringify(diff, null, 2)}\n`, 'utf8');
+    console.error(`Saved compare report to ${options.out}`);
+  }
+}
+
 async function runDoctor(options) {
   const checks = [];
   checks.push(['node', process.version, true]);
@@ -295,13 +457,46 @@ async function runDoctor(options) {
   const major = versionMatch ? Number.parseInt(versionMatch[1].split('.')[0], 10) : null;
   checks.push(['macOS', versionMatch?.[1] || 'unknown', major == null || major >= 27]);
 
+  const hwModel = await runProcess('sysctl', ['-n', 'hw.model'], { timeoutMs: 3_000 });
+  const hwModelStr = (hwModel.stdout || '').trim();
+  if (hwModelStr) checks.push(['hw.model', hwModelStr, true]);
+
+  const cpuBrand = await runProcess('sysctl', ['-n', 'machdep.cpu.brand_string'], { timeoutMs: 3_000 });
+  const cpuBrandStr = (cpuBrand.stdout || '').trim();
+  if (cpuBrandStr) checks.push(['cpu', cpuBrandStr, true]);
+
+  const memBytes = await runProcess('sysctl', ['-n', 'hw.memsize'], { timeoutMs: 3_000 });
+  const memRaw = (memBytes.stdout || '').trim();
+  if (memRaw) {
+    const gb = (Number(memRaw) / (1024 ** 3)).toFixed(0);
+    checks.push(['memory', `${gb} GB`, true]);
+  }
+
+  const thermalResult = await runProcess('pmset', ['-g', 'therm'], { timeoutMs: 5_000 });
+  const thermalOut = (thermalResult.stdout || thermalResult.stderr || '').trim();
+  const thermalMatch = thermalOut.match(/CPU_Scheduler_Limit\s*=\s*(\d+)/);
+  if (thermalMatch) {
+    const limit = Number.parseInt(thermalMatch[1], 10);
+    checks.push(['thermal limit', `${limit}%`, limit >= 100]);
+  }
+
+  const batteryResult = await runProcess('pmset', ['-g', 'batt'], { timeoutMs: 5_000 });
+  const batteryOut = (batteryResult.stdout || '').trim();
+  const batteryLine = batteryOut.split('\n').find((line) => line.includes('%'));
+  if (batteryLine) {
+    const pctMatch = batteryLine.match(/(\d+)%/);
+    const charging = /charging|AC Power|charged/i.test(batteryLine);
+    const pct = pctMatch ? `${pctMatch[1]}%` : '?%';
+    checks.push(['battery', `${pct}${charging ? ' (charging/AC)' : ' (battery)'}`, charging || Number.parseInt(pctMatch?.[1], 10) >= 20]);
+  }
+
   const inspection = await inspectModels(options);
   checks.push(['fm', inspection.fmBin, inspection.models.length > 0]);
   for (const model of inspection.models) {
     checks.push([`model:${model.name}`, model.available ? 'available' : model.reason || 'unavailable', model.available]);
   }
 
-  const lines = checks.map(([name, detail, ok]) => `${ok ? 'ok  ' : 'warn'} ${name.padEnd(14)} ${String(detail).replace(/\s+/g, ' ').trim()}`);
+  const lines = checks.map(([name, detail, ok]) => `${ok ? 'ok  ' : 'warn'} ${name.padEnd(16)} ${String(detail).replace(/\s+/g, ' ').trim()}`);
   console.log(lines.join('\n'));
 
   if (options.out) {
@@ -374,12 +569,16 @@ Dynamic benchmark CLI for Apple's fm command on macOS 27+.
 Usage:
   fm-bench [run] [options]
   fm-bench models [options]
+  fm-bench compare <before.json> <after.json> [options]
+  fm-bench history [dir] [options]
   fm-bench legend [options]
   fm-bench doctor [options]
 
 Commands:
   run                  Benchmark discovered or selected fm models
   models               List discovered models and availability
+  compare              Compare two saved JSON reports and show metric deltas
+  history              Show a trend table from all fm-bench JSON reports in a directory
   legend               Explain every terminal table column and color rule
   doctor               Check Node, macOS, fm, and model availability
 
@@ -398,7 +597,7 @@ Run options:
       --slo-tpot-ms <n>     Count request as good only if TPOT is <= n
   -p, --prompt <text>       Prompt to benchmark; repeatable
       --prompt-file <file>  .json, .jsonl, or blank-line separated text prompts
-      --profile <name>      quick, standard, interactive, throughput, client, or stress
+      --profile <name>      quick, standard, interactive, throughput, client, stress, reasoning, coding, or creative
   -i, --instructions <text> Instructions passed to fm respond
       --use-case <case>     Pass a system model use case through to fm
       --guardrails <level>  Pass a system model guardrail level through to fm
@@ -409,6 +608,10 @@ Run options:
       --available-only      Hide unavailable discovered models
       --capture-output      Include raw model output in JSON reports
       --fail-fast           Stop after the first failed measured run
+      --retry <n>           Retry failed fm calls up to n times with exponential backoff
+      --ci                  Exit 1 if any run fails or any SLO is violated; disables color and progress
+      --tag <name>          Tag this run; repeatable; included in JSON payload and report header
+      --note <text>         Freeform note included in JSON payload and report header
 
 Output:
       --format <type>       table, json, or csv (default: table)
@@ -421,7 +624,9 @@ Output:
       --no-progress         Disable live progress on stderr
       --compact             Force compact terminal layout
       --width <n>           Render for a specific terminal width
+      --histogram           Print an ASCII latency distribution histogram after the report
   -o, --out <file>          Save JSON or CSV report based on file extension
+      --output-dir <dir>    Save a timestamped JSON report to a directory automatically
   -v, --verbose             Include per-run CSV after the summary table
 
 Environment:
@@ -434,6 +639,11 @@ Examples:
   fm-bench --models system,pcc --runs 3 --profile stress
   fm-bench --profile client --sweep-concurrency 1,2 --request-rate 0.5
   fm-bench --prompt "Reply with exactly: ok" --json --out bench.json
+  fm-bench --profile reasoning --runs 5 --retry 2
+  fm-bench compare before.json after.json
+  fm-bench compare before.json after.json --json
+  fm-bench history ./reports
+  fm-bench history ./reports --json
   fm-bench legend
   fm-bench models
   fm-bench doctor

package/src/compare.js

@@ -0,0 +1,241 @@
+import { formatMs, formatNumber, formatPercent } from './table.js';
+
+export function diffReports(before, after) {
+  const beforeByKey = indexSummary(before.summary ?? []);
+  const afterByKey = indexSummary(after.summary ?? []);
+
+  const keys = new Set([...beforeByKey.keys(), ...afterByKey.keys()]);
+  const rows = [];
+
+  for (const key of [...keys].sort()) {
+    const b = beforeByKey.get(key);
+    const a = afterByKey.get(key);
+    rows.push(buildDiffRow(key, b ?? null, a ?? null));
+  }
+
+  return {
+    before: reportMeta(before),
+    after: reportMeta(after),
+    rows
+  };
+}
+
+function reportMeta(report) {
+  return {
+    version: report.version ?? '?',
+    startedAt: report.startedAt ?? '',
+    finishedAt: report.finishedAt ?? '',
+    runs: report.options?.runs ?? null,
+    profile: report.options?.profile ?? null,
+    concurrency: report.options?.concurrency ?? null
+  };
+}
+
+function indexSummary(summary) {
+  const map = new Map();
+  for (const item of summary) {
+    const key = `${item.model}::${item.concurrency ?? 1}`;
+    map.set(key, item);
+  }
+  return map;
+}
+
+function buildDiffRow(key, before, after) {
+  const [model, concurrencyStr] = key.split('::');
+  return {
+    model,
+    concurrency: Number.parseInt(concurrencyStr, 10) || 1,
+    available: {
+      before: before?.available ?? null,
+      after: after?.available ?? null
+    },
+    ttftP50: diffMs(before?.ttft?.p50, after?.ttft?.p50),
+    ttftP95: diffMs(before?.ttft?.p95, after?.ttft?.p95),
+    e2eP50: diffMs(before?.latency?.p50, after?.latency?.p50),
+    e2eP95: diffMs(before?.latency?.p95, after?.latency?.p95),
+    e2eP99: diffMs(before?.latency?.p99, after?.latency?.p99),
+    tpotP50: diffMs(before?.tpot?.p50, after?.tpot?.p50),
+    successRate: diffPercent(before?.successRate, after?.successRate),
+    goodputRate: diffPercent(before?.goodputRate, after?.goodputRate),
+    tokensPerSecond: diffNumber(before?.tokensPerSecond?.avg, after?.tokensPerSecond?.avg, false),
+    rps: diffNumber(before?.rps, after?.rps, false),
+    cv: diffPercent(before?.latency?.cv, after?.latency?.cv)
+  };
+}
+
+function diffMs(before, after) {
+  return {
+    before: before ?? null,
+    after: after ?? null,
+    delta: numDelta(before, after),
+    deltaPercent: percentDelta(before, after)
+  };
+}
+
+function diffNumber(before, after, lowerIsBetter = true) {
+  return {
+    before: before ?? null,
+    after: after ?? null,
+    delta: numDelta(before, after),
+    deltaPercent: percentDelta(before, after),
+    lowerIsBetter
+  };
+}
+
+function diffPercent(before, after) {
+  return {
+    before: before ?? null,
+    after: after ?? null,
+    delta: numDelta(before, after),
+    deltaPercent: percentDelta(before, after)
+  };
+}
+
+function numDelta(before, after) {
+  if (!Number.isFinite(before) || !Number.isFinite(after)) return null;
+  return after - before;
+}
+
+function percentDelta(before, after) {
+  if (!Number.isFinite(before) || !Number.isFinite(after) || before === 0) return null;
+  return (after - before) / Math.abs(before);
+}
+
+export function renderCompareReport(diff, options = {}) {
+  const { color = false, ascii = false } = options;
+  const width = options.width || process.stdout.columns || 120;
+  const V = ascii ? '|' : '│';
+  const H = ascii ? '-' : '─';
+  const TL = ascii ? '+' : '┌';
+  const TR = ascii ? '+' : '┐';
+  const BL = ascii ? '+' : '└';
+  const BR = ascii ? '+' : '┘';
+  const ML = ascii ? '+' : '├';
+  const MR = ascii ? '+' : '┤';
+  const TJ = ascii ? '+' : '┬';
+  const MJ = ascii ? '+' : '┼';
+  const BJ = ascii ? '+' : '┴';
+
+  const lines = [];
+
+  const beforeLabel = `${diff.before.version}  ${diff.before.startedAt ? diff.before.startedAt.slice(0, 19).replace('T', ' ') : '?'}`;
+  const afterLabel = `${diff.after.version}  ${diff.after.startedAt ? diff.after.startedAt.slice(0, 19).replace('T', ' ') : '?'}`;
+
+  lines.push(`fm-bench compare`);
+  lines.push(`  before: ${beforeLabel}`);
+  lines.push(`  after:  ${afterLabel}`);
+  lines.push('');
+
+  const colWidths = [8, 3, 10, 10, 10, 10, 10, 10, 8, 8, 9, 7, 7];
+  const headers = ['MODEL', 'C', 'TTFT P50', 'TTFT P95', 'E2E P50', 'E2E P95', 'E2E P99', 'TPOT P50', 'SUCC', 'GOOD', 'USER T/S', 'RPS', 'CV'];
+
+  const renderRule = (l, j, r) => `${l}${colWidths.map((w) => H.repeat(w + 2)).join(j)}${r}`;
+  const renderRow = (cells) => `${V}${cells.map((cell, i) => ` ${fitCell(cell, colWidths[i])} `).join(V)}${V}`;
+
+  lines.push(renderRule(TL, TJ, TR));
+  lines.push(renderRow(headers.map((h, i) => pad(h, colWidths[i]))));
+  lines.push(renderRule(ML, MJ, MR));
+
+  for (const row of diff.rows) {
+    const bLine = renderRow([
+      fitCell(row.model, colWidths[0]),
+      fitCell(String(row.concurrency), colWidths[1]),
+      fmtDiffMs(row.ttftP50, 'before', color),
+      fmtDiffMs(row.ttftP95, 'before', color),
+      fmtDiffMs(row.e2eP50, 'before', color),
+      fmtDiffMs(row.e2eP95, 'before', color),
+      fmtDiffMs(row.e2eP99, 'before', color),
+      fmtDiffMs(row.tpotP50, 'before', color),
+      fitCell(row.successRate.before != null ? formatPercent(row.successRate.before) : '-', colWidths[8]),
+      fitCell(row.goodputRate.before != null ? formatPercent(row.goodputRate.before) : '-', colWidths[9]),
+      fitCell(row.tokensPerSecond.before != null ? formatNumber(row.tokensPerSecond.before) : '-', colWidths[10]),
+      fitCell(row.rps.before != null ? formatNumber(row.rps.before) : '-', colWidths[11]),
+      fitCell(row.cv.before != null ? formatPercent(row.cv.before) : '-', colWidths[12])
+    ]);
+
+    const deltaLine = renderRow([
+      fitCell('', colWidths[0]),
+      fitCell('', colWidths[1]),
+      fmtDelta(row.ttftP50, true, color, colWidths[2]),
+      fmtDelta(row.ttftP95, true, color, colWidths[3]),
+      fmtDelta(row.e2eP50, true, color, colWidths[4]),
+      fmtDelta(row.e2eP95, true, color, colWidths[5]),
+      fmtDelta(row.e2eP99, true, color, colWidths[6]),
+      fmtDelta(row.tpotP50, true, color, colWidths[7]),
+      fmtDelta(row.successRate, false, color, colWidths[8]),
+      fmtDelta(row.goodputRate, false, color, colWidths[9]),
+      fmtDelta(row.tokensPerSecond, false, color, colWidths[10]),
+      fmtDelta(row.rps, false, color, colWidths[11]),
+      fmtDelta(row.cv, true, color, colWidths[12])
+    ]);
+
+    const aLine = renderRow([
+      fitCell('', colWidths[0]),
+      fitCell('', colWidths[1]),
+      fmtDiffMs(row.ttftP50, 'after', color),
+      fmtDiffMs(row.ttftP95, 'after', color),
+      fmtDiffMs(row.e2eP50, 'after', color),
+      fmtDiffMs(row.e2eP95, 'after', color),
+      fmtDiffMs(row.e2eP99, 'after', color),
+      fmtDiffMs(row.tpotP50, 'after', color),
+      fitCell(row.successRate.after != null ? formatPercent(row.successRate.after) : '-', colWidths[8]),
+      fitCell(row.goodputRate.after != null ? formatPercent(row.goodputRate.after) : '-', colWidths[9]),
+      fitCell(row.tokensPerSecond.after != null ? formatNumber(row.tokensPerSecond.after) : '-', colWidths[10]),
+      fitCell(row.rps.after != null ? formatNumber(row.rps.after) : '-', colWidths[11]),
+      fitCell(row.cv.after != null ? formatPercent(row.cv.after) : '-', colWidths[12])
+    ]);
+
+    lines.push(bLine);
+    lines.push(deltaLine);
+    lines.push(aLine);
+  }
+
+  lines.push(renderRule(BL, BJ, BR));
+  lines.push('');
+  lines.push('Rows: before → delta (% change) → after  |  Lower is better for latency and CV; higher for throughput and success.');
+
+  return lines.join('\n');
+}
+
+function fmtDiffMs(diff, which, color) {
+  const value = diff[which];
+  if (value == null) return '-';
+  return formatMs(value);
+}
+
+function fmtDelta(diff, lowerIsBetter, color, width) {
+  const { delta, deltaPercent } = diff;
+  if (delta == null || deltaPercent == null) return fitCell('n/a', width ?? 10);
+
+  const pct = Math.round(deltaPercent * 100);
+  const sign = delta >= 0 ? '+' : '';
+  const label = `${sign}${pct}%`;
+
+  let tone = null;
+  if (lowerIsBetter) {
+    tone = pct < -5 ? 'green' : pct > 5 ? 'red' : 'yellow';
+  } else {
+    tone = pct > 5 ? 'green' : pct < -5 ? 'red' : 'yellow';
+  }
+
+  const text = fitCell(label, width ?? 10);
+  if (color) return applyTone(text, tone);
+  return text;
+}
+
+function applyTone(text, tone) {
+  const TONES = { green: '\x1b[32m', yellow: '\x1b[33m', red: '\x1b[31m' };
+  const reset = '\x1b[0m';
+  return tone && TONES[tone] ? `${TONES[tone]}${text}${reset}` : text;
+}
+
+function pad(text, width) {
+  const len = String(text ?? '').length;
+  return String(text ?? '') + ' '.repeat(Math.max(0, width - len));
+}
+
+function fitCell(text, width) {
+  const str = String(text ?? '');
+  if (str.length <= width) return str + ' '.repeat(width - str.length);
+  return `${str.slice(0, width - 1)}…`;
+}

package/src/history.js

@@ -0,0 +1,95 @@
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { formatMs, formatNumber, formatPercent } from './table.js';
+
+export async function loadHistory(dir) {
+  const absDir = path.resolve(dir);
+  let entries;
+  try {
+    entries = await fs.readdir(absDir);
+  } catch {
+    throw new Error(`Cannot read directory: ${absDir}`);
+  }
+
+  const jsonFiles = entries
+    .filter((name) => name.endsWith('.json'))
+    .map((name) => path.join(absDir, name))
+    .sort();
+
+  const reports = [];
+  for (const filePath of jsonFiles) {
+    try {
+      const text = await fs.readFile(filePath, 'utf8');
+      const parsed = JSON.parse(text);
+      if (parsed.tool === 'fm-bench' && parsed.summary) {
+        reports.push({ filePath, report: parsed });
+      }
+    } catch {
+      // skip unparseable files
+    }
+  }
+
+  return reports;
+}
+
+export function renderHistoryReport(reports, options = {}) {
+  if (reports.length === 0) {
+    return 'No fm-bench JSON reports found in the given directory.';
+  }
+
+  const { color = false, ascii = false } = options;
+  const V = ascii ? '|' : '│';
+  const H = ascii ? '-' : '─';
+  const TL = ascii ? '+' : '┌';
+  const TR = ascii ? '+' : '┐';
+  const BL = ascii ? '+' : '└';
+  const BR = ascii ? '+' : '┘';
+  const ML = ascii ? '+' : '├';
+  const MR = ascii ? '+' : '┤';
+  const TJ = ascii ? '+' : '┬';
+  const MJ = ascii ? '+' : '┼';
+  const BJ = ascii ? '+' : '┴';
+
+  const colWidths = [19, 8, 3, 6, 9, 9, 9, 9, 5];
+  const headers = ['STARTED AT', 'MODEL', 'C', 'RUNS', 'TTFT P50', 'E2E P50', 'E2E P95', 'USER T/S', 'SUCC'];
+
+  const renderRule = (l, j, r) => `${l}${colWidths.map((w) => H.repeat(w + 2)).join(j)}${r}`;
+  const renderRowLine = (cells) => `${V}${cells.map((text, i) => ` ${fit(text, colWidths[i])} `).join(V)}${V}`;
+
+  const lines = [];
+  lines.push(`fm-bench history (${reports.length} report${reports.length === 1 ? '' : 's'})`);
+  lines.push('');
+  lines.push(renderRule(TL, TJ, TR));
+  lines.push(renderRowLine(headers.map((h, i) => h)));
+  lines.push(renderRule(ML, MJ, MR));
+
+  for (const { filePath, report } of reports) {
+    const started = report.startedAt ? report.startedAt.slice(0, 19).replace('T', ' ') : '?';
+    const modelRows = report.summary ?? [];
+
+    for (const item of modelRows) {
+      if (!item.available) continue;
+      const row = [
+        started,
+        item.model ?? '-',
+        String(item.concurrency ?? 1),
+        String(item.successes ?? '-'),
+        item.ttft?.p50 != null ? formatMs(item.ttft.p50) : '-',
+        item.latency?.p50 != null ? formatMs(item.latency.p50) : '-',
+        item.latency?.p95 != null ? formatMs(item.latency.p95) : '-',
+        item.tokensPerSecond?.avg != null ? formatNumber(item.tokensPerSecond.avg) : '-',
+        item.successRate != null ? formatPercent(item.successRate) : '-'
+      ];
+      lines.push(renderRowLine(row));
+    }
+  }
+
+  lines.push(renderRule(BL, BJ, BR));
+  return lines.join('\n');
+}
+
+function fit(text, width) {
+  const str = String(text ?? '');
+  if (str.length <= width) return str + ' '.repeat(width - str.length);
+  return `${str.slice(0, width - 1)}…`;
+}

package/src/prompts.js

@@ -93,6 +93,72 @@ const PROFILES = {
       id: 'summarize',
       prompt: 'Summarize this in two bullets: local model benchmarks should measure first-token latency, total latency, throughput, failures, and the exact prompt suite so results can be compared later.'
     }
+  ],
+  reasoning: [
+    {
+      id: 'math-word',
+      prompt: 'A server processes 1,200 requests per minute at peak. Each request uses 0.8 ms of CPU time on average. How many CPU cores are needed to handle peak load at 70% utilization? Show your reasoning step-by-step, then give a single final answer.'
+    },
+    {
+      id: 'logic-sequence',
+      prompt: 'Five engineers — Alice, Bob, Carol, Dave, Eve — each deploy one service. Alice deploys before Bob. Carol deploys after Dave but before Eve. Bob deploys before Dave. List the deployment order from first to last.'
+    },
+    {
+      id: 'causal-chain',
+      prompt: 'A CI pipeline has: lint (2 min), unit tests (5 min, parallel), integration tests (8 min, depends on unit), build (3 min, depends on integration), deploy (1 min, depends on build). What is the minimum wall-clock time from start to deployed? Explain each step.'
+    },
+    {
+      id: 'estimation',
+      prompt: 'Estimate how many tokens per day a popular AI coding assistant might process if it has 500,000 daily active users, each averaging 30 completions of 200 output tokens. Show your calculation and state any assumptions.'
+    },
+    {
+      id: 'debug-logic',
+      prompt: 'This function should return the median of a list: def median(lst): lst.sort(); n=len(lst); return lst[n//2] if n%2 else (lst[n//2-1]+lst[n//2])/2. Find all bugs and explain why each is a bug.'
+    }
+  ],
+  coding: [
+    {
+      id: 'code-review',
+      prompt: 'Review this TypeScript for correctness, performance, and readability issues: async function fetchAll(urls: string[]) { const results = []; for (const url of urls) { const r = await fetch(url); results.push(await r.json()); } return results; } Give three specific improvements with brief explanations.'
+    },
+    {
+      id: 'refactor',
+      prompt: 'Refactor this JavaScript to be cleaner and handle edge cases: function getUser(id, cb) { db.query("SELECT * FROM users WHERE id=" + id, function(err, rows) { if (err) { cb(null, err); } else { cb(rows[0]); } }); } Return only the improved code and a two-sentence explanation.'
+    },
+    {
+      id: 'algorithm',
+      prompt: 'Write a JavaScript function findDuplicates(arr) that returns all duplicate values in O(n) time and O(n) space. Include a brief complexity explanation and two edge-case examples.'
+    },
+    {
+      id: 'explain-code',
+      prompt: 'Explain what this code does, why it might be used, and one potential problem: const cache = new WeakMap(); function memoize(fn) { return function(...args) { if (!cache.has(this)) cache.set(this, new Map()); const key = JSON.stringify(args); if (!cache.get(this).has(key)) cache.get(this).set(key, fn.apply(this, args)); return cache.get(this).get(key); }; }'
+    },
+    {
+      id: 'system-design',
+      prompt: 'Design a rate limiter in 120 words or fewer: specify the data structure, the algorithm (token bucket, sliding window, or fixed window), and how you handle distributed deployments. Be precise and practical.'
+    }
+  ],
+  creative: [
+    {
+      id: 'product-announcement',
+      prompt: 'Write a 100-word product announcement for a developer tool called "PulseDB" that shows real-time query performance heatmaps in the terminal. Tone: enthusiastic but not hypey. Include one concrete example of what a user would see.'
+    },
+    {
+      id: 'error-message',
+      prompt: 'Rewrite this cryptic error into a helpful, actionable message a junior developer could act on: "Error: ECONNREFUSED 127.0.0.1:5432 errno: -111 syscall: connect code: ECONNREFUSED". Include what likely caused it and the first two things to check.'
+    },
+    {
+      id: 'technical-analogy',
+      prompt: 'Explain CPU context switching to someone who has never programmed, using a single concrete analogy from everyday life. Keep it under 80 words and make sure the analogy captures the performance cost.'
+    },
+    {
+      id: 'commit-message',
+      prompt: 'Write a clear and conventional git commit message for a change that adds retry logic with exponential backoff and jitter to the HTTP client. Include a subject line and a 3-bullet body.'
+    },
+    {
+      id: 'doc-summary',
+      prompt: 'Write a one-paragraph README introduction for an open-source CLI tool called "logslice" that extracts time-bounded log windows from large log files without loading them fully into memory. Target audience: backend engineers.'
+    }
   ]
 };
 

package/src/table.js

@@ -50,8 +50,12 @@ export function renderBenchmarkReport(payload, options = {}) {
 
   const title = `fm-bench ${payload.version} | ${payload.environment.platform}/${payload.environment.arch} | ${payload.environment.fmBin}`;
   const meta = `prompts ${payload.prompts.length} | runs ${payload.options.runs} | concurrency ${concurrencies} | stream ${payload.options.stream ? 'on' : 'off'} | measured ${measured} | failed ${failed} | skipped ${skipped} | elapsed ${formatMs(elapsedMs)}${slo ? ` | ${slo}` : ''}`;
+  const tags = payload.options?.tags?.length ? payload.options.tags : [];
+  const note = payload.options?.note ?? null;
   lines.push(truncate(title, width));
   lines.push(truncate(meta, width));
+  if (tags.length > 0) lines.push(truncate(`tags: ${tags.join(', ')}`, width));
+  if (note) lines.push(truncate(`note: ${note}`, width));
   lines.push('');
 
   if (mode === 'compact') {
@@ -121,6 +125,49 @@ export function renderLegend(options = {}) {
   return renderWrappedTable(['table', 'column', 'definition', 'rule'], rows, legendColumnWidths(width), options);
 }
 
+export function renderLatencyHistogram(results, options = {}) {
+  const { ascii = false, color = false, width: termWidth = 80 } = options;
+  const successes = results.filter((r) => r.ok && Number.isFinite(r.durationMs));
+  if (successes.length === 0) return 'No successful results to histogram.';
+
+  const values = successes.map((r) => r.durationMs).sort((a, b) => a - b);
+  const min = values[0];
+  const max = values[values.length - 1];
+
+  if (min === max) {
+    return `E2E latency histogram: all ${values.length} values = ${formatMs(min)}`;
+  }
+
+  const numBuckets = Math.min(20, Math.max(5, Math.floor((termWidth - 20) / 4)));
+  const bucketSize = (max - min) / numBuckets;
+  const counts = new Array(numBuckets).fill(0);
+
+  for (const v of values) {
+    const idx = Math.min(numBuckets - 1, Math.floor((v - min) / bucketSize));
+    counts[idx] += 1;
+  }
+
+  const maxCount = Math.max(...counts);
+  const barWidth = Math.max(1, termWidth - 24);
+  const lines = [];
+  const bar = ascii ? '#' : '█';
+  const halfBar = ascii ? ':' : '▌';
+
+  lines.push(`E2E latency distribution (n=${values.length}, ${formatMs(min)}…${formatMs(max)}):`);
+
+  for (let i = 0; i < numBuckets; i += 1) {
+    const bucketMin = min + i * bucketSize;
+    const bucketMax = bucketMin + bucketSize;
+    const label = `${formatMs(bucketMin)}`.padStart(7);
+    const fillLength = Math.round((counts[i] / maxCount) * barWidth);
+    const filled = bar.repeat(fillLength);
+    const countStr = counts[i] > 0 ? String(counts[i]) : '';
+    lines.push(`${label} ${filled}${countStr ? ` ${countStr}` : ''}`);
+  }
+
+  return lines.join('\n');
+}
+
 export function formatMs(value) {
   if (value == null || !Number.isFinite(value)) return '-';
   if (value >= 1000) return `${(value / 1000).toFixed(2)}s`;