fm-bench 0.5.2 → 0.6.0

package/README.md

@@ -1,5 +1,7 @@
 # fm-bench
 
+[![CI](https://github.com/devinoldenburg/fm-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/devinoldenburg/fm-bench/actions/workflows/ci.yml)
+
 Benchmark Apple's `fm` command on macOS 27+.
 
 Measure latency, throughput, streaming smoothness, stability, and goodput across Apple Foundation Models — with repeatable prompt suites and JSON/CSV reports for automation.
@@ -28,17 +30,18 @@ fm-bench
 One command discovers your models, runs the standard prompt suite, and prints a full benchmark report:
 
 ```text
-fm-bench 0.5.0 | darwin/arm64 | fm
-prompts 5 | runs 3 | concurrency 1,2 | stream on | measured 30 | failed 0 | skipped 0 | elapsed 42.10s | SLO TTFT<=750ms,E2E<=4.00s
+fm-bench 0.6.0 | darwin/arm64 | fm
+prompts 3 | runs 5 | concurrency 1 | stream on | measured 15 | failed 0 | skipped 0 | elapsed 38.20s
 
 ┌───┬────────┬────────┬─────────┬──────┬──────┬──────────┬──────┬──────────┬─────┬─────┐
 │ C │ MODEL  │ STATUS │ OK/RUNS │ SUCC │ GOOD │ GOOD RPS │ TTFT │ E2E P95  │ SYS │ CV  │
 ├───┼────────┼────────┼─────────┼──────┼──────┼──────────┼──────┼──────────┼─────┼─────┤
 │ 1 │ system │ ok     │   15/15 │ 100% │  93% │      0.4 │ 318ms│    3.20s │  42 │ 12% │
-│ 2 │ system │ ok     │   15/15 │ 100% │  80% │      0.7 │ 501ms│    4.40s │  68 │ 21% │
 └───┴────────┴────────┴─────────┴──────┴──────┴──────────┴──────┴──────────┴─────┴─────┘
 ```
 
+Default profile is `standard` (3 prompts). Use `--runs 5`, `--sweep-concurrency 1,2`, or `--profile client` for heavier suites.
+
 Wide terminals add TTFT P95, TPOT, decode/prefill throughput, chunk-gap smoothness, and 95% CI columns. Narrow terminals switch to compact model cards automatically.
 
 ## Install
@@ -68,8 +71,10 @@ fm-bench doctor   # verify your setup
 |---------|-------------|
 | `fm-bench` | Run the full benchmark (default) |
 | `fm-bench models` | List discovered models, availability, and quota |
-| `fm-bench compare <a.json> <b.json>` | Regression diff: before/after metrics with color-coded deltas |
-| `fm-bench history [dir]` | Chronological trend table from a directory of saved reports |
+| `fm-bench compare <a.json> <b.json>` | Regression diff with suite/hardware warnings; `--strict` for CI |
+| `fm-bench history [dir]` | Trend table from saved reports (sorted by time, tags visible) |
+| `fm-bench validate <report.json>` | Verify report JSON (schema v1) before sharing |
+| `fm-bench export <report.json>` | Standalone HTML report with embedded JSON |
 | `fm-bench legend` | Definitions for every table column and color rule |
 | `fm-bench doctor` | Environment check: Node, macOS, `fm`, CPU, memory, thermals, battery |
 
@@ -93,9 +98,10 @@ fm-bench --profile reasoning --runs 5
 fm-bench --profile coding --runs 3 --histogram
 
 # Archive runs and compare before/after a macOS update
-fm-bench --output-dir reports/ --tag before-update
-fm-bench --output-dir reports/ --tag after-update
-fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json
+fm-bench --output-dir reports/ --tag before-update --export-html
+fm-bench --output-dir reports/ --tag after-update --export-html
+fm-bench validate reports/*.json
+fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json --strict
 
 # Fail CI when SLOs regress
 fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5
@@ -140,8 +146,9 @@ fm-bench --format csv --out bench.csv
 | Flag | Description |
 |------|-------------|
 | `--json` / `--csv` | Output format (also `--format table\|json\|csv`) |
-| `-o, --out <file>` | Save a report to a file |
-| `--output-dir <dir>` | Auto-save a timestamped JSON report to a directory |
+| `-o, --out <file>` | Save JSON (`.json`), per-run CSV (`.csv`), or shareable HTML (`.html`) |
+| `--output-dir <dir>` | Auto-save timestamped JSON (and optional HTML with `--export-html`) |
+| `--export-html` | With `--output-dir`, also write a matching `.html` report |
 | `--tag <name>` | Label this run; repeatable; appears in payload and header |
 | `--note <text>` | Freeform annotation in payload and header |
 | `--histogram` | Print ASCII latency distribution chart after the report |
@@ -174,6 +181,14 @@ Nine built-in suites, choose the one that matches your use case:
 | `coding` | 5 | Code review, refactoring, algorithms, system design |
 | `creative` | 5 | Product copy, analogies, commit messages, docs |
 
+## Sharing and comparing results
+
+Reports from 0.6.0+ include **schema v1**: `reportId`, hardware fingerprint, and a **suite key** so you can tell if two JSON files used the same prompts and run settings. See [docs/report-format.md](docs/report-format.md).
+
+- Share **HTML** with teammates who do not use the CLI: `fm-bench export bench.json -o bench.html`
+- Gate uploads in CI: `fm-bench validate artifact.json`
+- Apples-to-apples regressions: same `--profile` and `--runs`, then `fm-bench compare a.json b.json`
+
 ## Regression Tracking
 
 Track performance across macOS updates, model changes, or hardware swaps:

package/docs/methodology.md

@@ -57,3 +57,7 @@ Client-side measurements include process startup, local queueing, model prefill,
 Stream smoothness metrics use stdout chunk arrival times. A chunk can contain more than one token, and terminal or pipe buffering can affect chunk boundaries. Treat `second_chunk_ms` and `chunk_gap` as user-visible streaming diagnostics, not raw decoder telemetry.
 
 For serious comparisons, prefer at least three runs per prompt, include warmups, benchmark both interactive and throughput or client profiles, compare models at the same concurrency operating points, set SLOs that match your real UX budget, and save JSON reports for later analysis.
+
+## Report artifacts
+
+Saved JSON includes client-side environment metadata (hardware model, macOS build, `fm` help digest, power/thermal snapshot) so shared results remain interpretable on other machines. Use `fm-bench validate` before publishing and `fm-bench compare --strict` when you require identical prompt suites. Format details: [report-format.md](./report-format.md).

package/docs/releasing.md

@@ -0,0 +1,35 @@
+# Releasing
+
+`fm-bench` uses semver tags (`v*.*.*`). Pushing a tag runs the **Release** workflow: test, lint, npm publish (with provenance), and GitHub release notes.
+
+## Prerequisites
+
+- Repository secret **`NPM_TOKEN`**: npm automation token with publish access to `fm-bench`.
+- **`main`** is green on CI.
+
+## Option A — GitHub Actions (recommended)
+
+1. Open **Actions → Version → Run workflow**.
+2. Choose `patch`, `minor`, `major`, or an exact semver.
+3. The job runs `npm version`, pushes the commit and tag to `main`.
+4. The tag push triggers **Release** automatically.
+
+## Option B — Local
+
+```sh
+npm ci && npm test && npm run lint
+npm version patch   # or minor / major
+git push --follow-tags
+```
+
+## Re-run Release without republishing
+
+If npm already has the version but GitHub release failed (or vice versa), use **Actions → Release → Run workflow** and enter the existing tag (for example `v0.5.3`). The workflow skips npm publish when that version is already on the registry and skips creating a duplicate GitHub release.
+
+## Dry run
+
+```sh
+npm run publish:dry-run
+```
+
+CI runs the same dry-run on every push to `main` and on pull requests.

package/docs/report-format.md

@@ -0,0 +1,52 @@
+# Report format (schema v1)
+
+Every measured run can be saved as JSON. Reports from fm-bench **0.6.0+** include a versioned schema so you can validate, share, and compare results across machines.
+
+## Top-level fields
+
+| Field | Description |
+|-------|-------------|
+| `tool` | Always `"fm-bench"` |
+| `version` | fm-bench package version that produced the report |
+| `schemaVersion` | `"1"` for reports from 0.6.0+ (older reports omit this) |
+| `reportId` | Random hex id for citing a single run |
+| `startedAt` / `finishedAt` | ISO-8601 timestamps |
+| `options` | Public run configuration (profile, runs, concurrency, SLOs, tags, note) |
+| `environment` | Host fingerprint: platform, arch, Node, hardware model, CPU, memory, macOS version, `fm` help digest, thermal/power snapshot |
+| `suite` | Derived suite key + fingerprint for apples-to-apples comparison |
+| `prompts` | Prompt ids, text, and token counts |
+| `models` | Discovered models and availability |
+| `summary` | Per-model / per-concurrency roll-up statistics |
+| `results` | Per-run measurements (optional `output` when `--capture-output`) |
+
+## Sharing results
+
+1. **JSON** — best for automation and `fm-bench compare`. Save with `--out bench.json` or `--output-dir reports/`.
+2. **HTML** — self-contained page for humans: `--out bench.html`, `fm-bench export bench.json -o bench.html`, or `--output-dir reports/ --export-html`.
+3. **CSV** — per-run rows only: `--format csv` or `--out runs.csv`.
+
+Validate before publishing:
+
+```sh
+fm-bench validate my-report.json
+```
+
+## Comparable benchmarks
+
+For fair comparison, match:
+
+- Same `--profile` (or same `--prompt-file`)
+- Same `--runs` and `--warmup`
+- Same concurrency operating points (`--concurrency` or `--sweep-concurrency`)
+- Similar power/thermal state (see `environment.power` and `environment.thermal`)
+
+```sh
+fm-bench compare before.json after.json
+fm-bench compare before.json after.json --strict   # exit 2 if suites differ
+```
+
+`compare` warns when hardware, macOS, or suite configuration differ. Use `--tag` and `--note` so `fm-bench history` stays readable.
+
+## Legacy reports
+
+Reports from fm-bench before 0.6.0 remain valid JSON. They lack `schemaVersion`, `reportId`, `suite`, and enriched `environment`. `validate` still checks required fields; `compare` works on `summary` as before.

package/package.json

@@ -1,10 +1,10 @@
 {
   "name": "fm-bench",
-  "version": "0.5.2",
+  "version": "0.6.0",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {
-    "fm-bench": "./bin/fm-bench.js"
+    "fm-bench": "bin/fm-bench.js"
   },
   "files": [
     "bin",

package/src/bench.js

@@ -1,6 +1,7 @@
 import crypto from 'node:crypto';
 import { checkModelAvailability, collectEnvironment, countTokens, discoverModels, getQuotaUsage, respond } from './fm.js';
 import { loadPrompts } from './prompts.js';
+import { finalizeReportPayload } from './schema.js';
 import { summarizeByModel } from './stats.js';
 
 export async function inspectModels(options = {}) {
@@ -115,7 +116,7 @@ export async function runBenchmark(options = {}) {
     || a.run - b.run);
 
   const summary = summarizeByModel(results, modelStatuses, { concurrencies });
-  const payload = {
+  const payload = finalizeReportPayload({
     tool: 'fm-bench',
     version: options.version,
     startedAt,
@@ -131,7 +132,7 @@ export async function runBenchmark(options = {}) {
     scenarios,
     summary,
     results
-  };
+  });
   notify(options, {
     type: 'benchmark:complete',
     completed: completedRuns,

package/src/cli.js

@@ -2,6 +2,8 @@ import fs from 'node:fs/promises';
 import { createRequire } from 'node:module';
 import { inspectModels, runBenchmark } from './bench.js';
 import { diffReports, renderCompareReport } from './compare.js';
+import { renderHtmlReport } from './export.js';
+import { validateReport } from './schema.js';
 import { loadHistory, renderHistoryReport } from './history.js';
 import { runProcess } from './process.js';
 import { createProgress } from './progress.js';
@@ -46,6 +48,16 @@ export async function runCli(argv = process.argv.slice(2)) {
     return;
   }
 
+  if (parsed.command === 'validate') {
+    await runValidate(parsed);
+    return;
+  }
+
+  if (parsed.command === 'export') {
+    await runExport(parsed);
+    return;
+  }
+
   if (parsed.command === 'history') {
     await runHistory(parsed, renderOptions(parsed));
     return;
@@ -99,7 +111,9 @@ export async function runCli(argv = process.argv.slice(2)) {
   }
 
   if (parsed.out) {
-    const reportFormat = parsed.out.endsWith('.csv') ? 'csv' : 'json';
+    const reportFormat = parsed.out.endsWith('.csv') ? 'csv'
+      : parsed.out.endsWith('.html') ? 'html'
+        : 'json';
     const written = await writeReport(parsed.out, payload, reportFormat);
     if (parsed.format !== 'json') {
       console.error(`Saved ${reportFormat.toUpperCase()} report to ${written}`);
@@ -110,9 +124,17 @@ export async function runCli(argv = process.argv.slice(2)) {
     const stamp = payload.startedAt.replace(/[:.]/g, '-').replace('T', '_').slice(0, 19);
     const firstModel = parsed.models?.flatMap((m) => String(m).split(',')).map((m) => m.trim()).filter(Boolean)[0] || 'all';
     const modelSlug = firstModel.replace(/[^a-z0-9]/gi, '_');
-    const filename = `fm-bench_${stamp}_${modelSlug}.json`;
-    const filePath = `${parsed.outputDir}/${filename}`;
-    const written = await writeReport(filePath, payload, 'json');
+    const tagSlug = parsed.tags?.length ? `_${parsed.tags.join('-').replace(/[^a-z0-9-]/gi, '_')}` : '';
+    const base = `fm-bench_${stamp}_${modelSlug}${tagSlug}`;
+    const jsonPath = `${parsed.outputDir}/${base}.json`;
+    const written = await writeReport(jsonPath, payload, 'json');
+    if (parsed.exportHtml) {
+      const htmlPath = `${parsed.outputDir}/${base}.html`;
+      await writeReport(htmlPath, payload, 'html');
+      if (parsed.format !== 'json') {
+        console.error(`Saved HTML report to ${htmlPath}`);
+      }
+    }
     if (parsed.format !== 'json') {
       console.error(`Saved JSON report to ${written}`);
     }
@@ -190,11 +212,14 @@ export function parseArgs(argv) {
     progress: 'auto',
     compact: false,
     width: null,
-    histogram: false
+    histogram: false,
+    exportHtml: false,
+    strictCompare: false,
+    validateFiles: []
   };
 
   const args = [...argv];
-  if (args[0] && !args[0].startsWith('-') && ['run', 'models', 'doctor', 'legend', 'metrics', 'compare', 'history', 'help'].includes(args[0])) {
+  if (args[0] && !args[0].startsWith('-') && ['run', 'models', 'doctor', 'legend', 'metrics', 'compare', 'history', 'validate', 'export', 'help'].includes(args[0])) {
     options.command = args.shift();
   }
 
@@ -332,6 +357,12 @@ export function parseArgs(argv) {
       case '--histogram':
         options.histogram = true;
         break;
+      case '--export-html':
+        options.exportHtml = true;
+        break;
+      case '--strict':
+        options.strictCompare = true;
+        break;
       case '-o':
       case '--out':
         options.out = requireValue(arg, args);
@@ -378,6 +409,8 @@ export function parseArgs(argv) {
           options.compareFiles.push(arg);
         } else if (options.command === 'history') {
           options.historyDir = arg;
+        } else if (options.command === 'validate' || options.command === 'export') {
+          options.validateFiles.push(arg);
         } else {
           options.prompts.push([arg, ...args].join(' '));
           args.length = 0;
@@ -445,6 +478,75 @@ async function runCompare(options, renderOpts) {
     await fs.writeFile(options.out, `${JSON.stringify(diff, null, 2)}\n`, 'utf8');
     console.error(`Saved compare report to ${options.out}`);
   }
+
+  if (options.strictCompare && diff.compatibility && !diff.compatibility.suiteMatch) {
+    const error = new Error('compare: benchmark suites differ (--strict)');
+    error.exitCode = 2;
+    throw error;
+  }
+}
+
+async function runValidate(options) {
+  const files = options.validateFiles;
+  if (files.length === 0) {
+    throw new Error('validate requires at least one JSON report: fm-bench validate report.json');
+  }
+
+  let failed = 0;
+  for (const filePath of files) {
+    let parsed;
+    try {
+      const text = await fs.readFile(filePath, 'utf8');
+      parsed = JSON.parse(text);
+    } catch {
+      console.error(`invalid  ${filePath}  (cannot read or parse JSON)`);
+      failed += 1;
+      continue;
+    }
+    const result = validateReport(parsed);
+    if (result.ok) {
+      const id = result.report.reportId ?? '—';
+      const schema = result.report.schemaVersion ?? 'legacy';
+      console.log(`ok       ${filePath}  schema=${schema} id=${id}`);
+    } else {
+      console.error(`invalid  ${filePath}  ${result.errors.join('; ')}`);
+      failed += 1;
+    }
+  }
+
+  if (failed > 0) {
+    const error = new Error(`${failed} report(s) failed validation`);
+    error.exitCode = 1;
+    throw error;
+  }
+}
+
+async function runExport(options) {
+  const files = options.validateFiles;
+  if (files.length === 0) {
+    throw new Error('export requires a JSON report: fm-bench export report.json [-o out.html]');
+  }
+
+  const filePath = files[0];
+  const text = await fs.readFile(filePath, 'utf8');
+  let report;
+  try {
+    report = JSON.parse(text);
+  } catch {
+    throw new Error(`Cannot parse ${filePath} as JSON`);
+  }
+  const validation = validateReport(report);
+  if (!validation.ok) {
+    throw new Error(`Not a valid fm-bench report: ${validation.errors.join('; ')}`);
+  }
+
+  const html = renderHtmlReport(validation.report);
+  if (options.out) {
+    await fs.writeFile(options.out, html, 'utf8');
+    console.error(`Wrote HTML to ${options.out}`);
+  } else {
+    console.log(html);
+  }
 }
 
 async function runDoctor(options) {
@@ -574,6 +676,8 @@ Usage:
   fm-bench models [options]
   fm-bench compare <before.json> <after.json> [options]
   fm-bench history [dir] [options]
+  fm-bench validate <report.json> [more...]
+  fm-bench export <report.json> [-o report.html]
   fm-bench legend [options]
   fm-bench doctor [options]
 
@@ -582,6 +686,8 @@ Commands:
   models               List discovered models and availability
   compare              Compare two saved JSON reports and show metric deltas
   history              Show a trend table from all fm-bench JSON reports in a directory
+  validate             Verify report JSON structure (schema v1)
+  export               Render a shareable standalone HTML report from JSON
   legend               Explain every terminal table column and color rule
   doctor               Check Node, macOS, fm, and model availability
 
@@ -628,10 +734,14 @@ Output:
       --compact             Force compact terminal layout
       --width <n>           Render for a specific terminal width
       --histogram           Print an ASCII latency distribution histogram after the report
-  -o, --out <file>          Save JSON or CSV report based on file extension
+  -o, --out <file>          Save JSON, CSV, or HTML report based on file extension
       --output-dir <dir>    Save a timestamped JSON report to a directory automatically
+      --export-html         With --output-dir, also write a matching .html report
   -v, --verbose             Include per-run CSV after the summary table
 
+Compare:
+      --strict                Exit 2 when before/after benchmark suites differ
+
 Environment:
       --fm-bin <path>       fm binary to execute (default: FM_BIN or fm)
   -h, --help                Show this help
@@ -645,6 +755,10 @@ Examples:
   fm-bench --profile reasoning --runs 5 --retry 2
   fm-bench compare before.json after.json
   fm-bench compare before.json after.json --json
+  fm-bench compare before.json after.json --strict
+  fm-bench validate reports/*.json
+  fm-bench export bench.json -o bench.html
+  fm-bench --output-dir reports/ --export-html --tag nightly
   fm-bench history ./reports
   fm-bench history ./reports --json
   fm-bench legend

package/src/compare.js

@@ -1,6 +1,8 @@
+import { compareCompatibility } from './schema.js';
 import { formatMs, formatNumber, formatPercent } from './table.js';
 
 export function diffReports(before, after) {
+  const compatibility = compareCompatibility(before, after);
   const beforeByKey = indexSummary(before.summary ?? []);
   const afterByKey = indexSummary(after.summary ?? []);
 
@@ -16,18 +18,28 @@ export function diffReports(before, after) {
   return {
     before: reportMeta(before),
     after: reportMeta(after),
+    compatibility,
     rows
   };
 }
 
 function reportMeta(report) {
+  const tags = report.options?.tags;
   return {
     version: report.version ?? '?',
+    schemaVersion: report.schemaVersion ?? null,
+    reportId: report.reportId ?? null,
     startedAt: report.startedAt ?? '',
     finishedAt: report.finishedAt ?? '',
     runs: report.options?.runs ?? null,
     profile: report.options?.profile ?? null,
-    concurrency: report.options?.concurrency ?? null
+    concurrency: report.options?.concurrency ?? null,
+    tags: Array.isArray(tags) ? tags : [],
+    note: report.options?.note ?? null,
+    hwModel: report.environment?.hwModel ?? report.suite?.fingerprint?.hwModel ?? null,
+    macOS: report.suite?.fingerprint?.macOSProductVersion
+      ?? report.environment?.macOS?.match(/ProductVersion:\s*([^\n]+)/)?.[1]?.trim()
+      ?? null
   };
 }
 
@@ -122,8 +134,20 @@ export function renderCompareReport(diff, options = {}) {
   const afterLabel = `${diff.after.version}  ${diff.after.startedAt ? diff.after.startedAt.slice(0, 19).replace('T', ' ') : '?'}`;
 
   lines.push(`fm-bench compare`);
-  lines.push(`  before: ${beforeLabel}`);
-  lines.push(`  after:  ${afterLabel}`);
+  lines.push(`  before: ${beforeLabel}${diff.before.hwModel ? ` | ${diff.before.hwModel}` : ''}`);
+  lines.push(`  after:  ${afterLabel}${diff.after.hwModel ? ` | ${diff.after.hwModel}` : ''}`);
+  if (diff.before.profile || diff.after.profile) {
+    lines.push(`  suite:  profile=${diff.before.profile ?? '?'} runs=${diff.before.runs ?? '?'} (before) → profile=${diff.after.profile ?? '?'} runs=${diff.after.runs ?? '?'}`);
+  }
+  const compat = diff.compatibility;
+  if (compat?.warnings?.length) {
+    for (const w of compat.warnings) {
+      lines.push(`  warn:   ${w}`);
+    }
+  }
+  if (compat && !compat.suiteMatch) {
+    lines.push('  note:   use identical --profile, --runs, and prompts for apples-to-apples comparison');
+  }
   lines.push('');
 
   const colWidths = [8, 3, 10, 10, 10, 10, 10, 10, 8, 8, 9, 7, 7];

package/src/export.js

@@ -0,0 +1,145 @@
+import { environmentFingerprint } from './schema.js';
+
+/**
+ * Self-contained HTML report for sharing (paste, email, GitHub gist, static host).
+ * @param {Record<string, unknown>} report
+ */
+export function renderHtmlReport(report) {
+  const fp = environmentFingerprint(report);
+  const options = /** @type {Record<string, unknown>} */ (report.options ?? {});
+  const tags = Array.isArray(options.tags) ? options.tags : [];
+  const summary = /** @type {Array<Record<string, unknown>>} */ (report.summary ?? []);
+
+  const esc = (s) => String(s ?? '')
+    .replace(/&/g, '&amp;')
+    .replace(/</g, '&lt;')
+    .replace(/>/g, '&gt;')
+    .replace(/"/g, '&quot;');
+
+  const metaRows = [
+    ['Report ID', report.reportId ?? '—'],
+    ['fm-bench', report.version ?? '—'],
+    ['Schema', report.schemaVersion ?? 'legacy'],
+    ['Started', report.startedAt ?? '—'],
+    ['Finished', report.finishedAt ?? '—'],
+    ['Profile', options.profile ?? '—'],
+    ['Runs / prompt', options.runs ?? '—'],
+    ['Concurrency', formatConcurrency(options)],
+    ['Tags', tags.length ? tags.join(', ') : '—'],
+    ['Note', options.note ?? '—'],
+    ['Hardware', fp.hwModel ?? '—'],
+    ['CPU', fp.cpuBrand ?? '—'],
+    ['Memory', fp.memoryGb != null ? `${fp.memoryGb} GB` : '—'],
+    ['macOS', fp.macOSProductVersion ?? '—'],
+    ['Build', fp.macOSBuildVersion ?? '—'],
+    ['Node', fp.node ?? '—'],
+    ['fm CLI digest', fp.fmHelpDigest ?? '—']
+  ];
+
+  const summaryRows = summary.map((row) => {
+    const ttft = /** @type {{ p50?: number, p95?: number }} */ (row.ttft ?? {});
+    const lat = /** @type {{ p50?: number, p95?: number, cv?: number }} */ (row.latency ?? {});
+    const tps = /** @type {{ avg?: number }} */ (row.tokensPerSecond ?? {});
+    return `<tr>
+      <td>${esc(row.model)}</td>
+      <td>${esc(row.concurrency ?? 1)}</td>
+      <td>${row.available ? 'yes' : 'no'}</td>
+      <td>${fmtPct(row.successRate)}</td>
+      <td>${fmtPct(row.goodputRate)}</td>
+      <td>${fmtMs(ttft.p50)}</td>
+      <td>${fmtMs(ttft.p95)}</td>
+      <td>${fmtMs(lat.p50)}</td>
+      <td>${fmtMs(lat.p95)}</td>
+      <td>${fmtNum(tps.avg)}</td>
+      <td>${fmtPct(lat.cv)}</td>
+    </tr>`;
+  }).join('\n');
+
+  const jsonEmbed = esc(JSON.stringify(report, null, 2));
+
+  return `<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>fm-bench report ${esc(report.reportId ?? '')}</title>
+  <style>
+    :root { --bg: #0f1419; --card: #1a2332; --text: #e7ecf3; --muted: #8b9cb3; --accent: #3d8bfd; --border: #2a3548; }
+    * { box-sizing: border-box; }
+    body { font-family: ui-sans-serif, system-ui, -apple-system, sans-serif; background: var(--bg); color: var(--text); margin: 0; padding: 2rem 1.25rem; line-height: 1.5; }
+    h1 { font-size: 1.35rem; margin: 0 0 0.25rem; }
+    .sub { color: var(--muted); font-size: 0.9rem; margin-bottom: 1.5rem; }
+    section { background: var(--card); border: 1px solid var(--border); border-radius: 10px; padding: 1.25rem; margin-bottom: 1.25rem; }
+    h2 { font-size: 1rem; margin: 0 0 1rem; color: var(--accent); }
+    dl { display: grid; grid-template-columns: 10rem 1fr; gap: 0.35rem 1rem; margin: 0; font-size: 0.88rem; }
+    dt { color: var(--muted); }
+    dd { margin: 0; }
+    table { width: 100%; border-collapse: collapse; font-size: 0.82rem; }
+    th, td { text-align: left; padding: 0.45rem 0.5rem; border-bottom: 1px solid var(--border); }
+    th { color: var(--muted); font-weight: 600; }
+    pre { background: #0b0f14; border: 1px solid var(--border); border-radius: 8px; padding: 1rem; overflow: auto; font-size: 0.72rem; max-height: 320px; }
+    footer { color: var(--muted); font-size: 0.8rem; margin-top: 2rem; }
+    a { color: var(--accent); }
+  </style>
+</head>
+<body>
+  <h1>fm-bench benchmark report</h1>
+  <p class="sub">Share this file to compare runs on identical prompt suites. Attach JSON for automation.</p>
+
+  <section>
+    <h2>Run metadata</h2>
+    <dl>
+      ${metaRows.map(([k, v]) => `<dt>${esc(k)}</dt><dd>${esc(v)}</dd>`).join('\n      ')}
+    </dl>
+  </section>
+
+  <section>
+    <h2>Summary by model</h2>
+    <table>
+      <thead>
+        <tr>
+          <th>Model</th><th>C</th><th>Avail</th><th>Success</th><th>Goodput</th>
+          <th>TTFT P50</th><th>TTFT P95</th><th>E2E P50</th><th>E2E P95</th><th>User tok/s</th><th>CV</th>
+        </tr>
+      </thead>
+      <tbody>
+        ${summaryRows || '<tr><td colspan="11">No summary rows</td></tr>'}
+      </tbody>
+    </table>
+  </section>
+
+  <section>
+    <h2>Embedded JSON (machine-readable)</h2>
+    <pre>${jsonEmbed}</pre>
+  </section>
+
+  <footer>
+    Generated by <a href="https://github.com/devinoldenburg/fm-bench">fm-bench</a>.
+    Compare two reports: <code>fm-bench compare before.json after.json</code>.
+    Validate: <code>fm-bench validate report.json</code>.
+  </footer>
+</body>
+</html>
+`;
+}
+
+function formatConcurrency(options) {
+  const sweep = options.sweepConcurrency;
+  if (Array.isArray(sweep) && sweep.length > 0) return sweep.join(', ');
+  return String(options.concurrency ?? 1);
+}
+
+function fmtMs(v) {
+  if (!Number.isFinite(v)) return '—';
+  return v >= 1000 ? `${(v / 1000).toFixed(2)}s` : `${Math.round(v)}ms`;
+}
+
+function fmtPct(v) {
+  if (!Number.isFinite(v)) return '—';
+  return `${Math.round(v * 100)}%`;
+}
+
+function fmtNum(v) {
+  if (!Number.isFinite(v)) return '—';
+  return String(Math.round(v * 10) / 10);
+}

package/src/fm.js

@@ -1,6 +1,8 @@
+import crypto from 'node:crypto';
 import os from 'node:os';
 import { stripAnsi } from './ansi.js';
 import { runProcess } from './process.js';
+import { parseBatteryOutput, parseThermalOutput } from './system.js';
 
 const DEFAULT_MODELS = [
   { name: 'system', description: 'On-device Apple Foundation Model' },
@@ -183,13 +185,56 @@ export async function respond(fmBin, model, prompt, options = {}) {
 
 export async function collectEnvironment(fmBin) {
   const swVers = await runProcess('sw_vers', [], { timeoutMs: 5_000 });
+  const macOS = stripAnsi(swVers.stdout).trim() || null;
+
+  const hwModel = await runProcess('sysctl', ['-n', 'hw.model'], { timeoutMs: 3_000 });
+  const cpuBrand = await runProcess('sysctl', ['-n', 'machdep.cpu.brand_string'], { timeoutMs: 3_000 });
+  const memBytes = await runProcess('sysctl', ['-n', 'hw.memsize'], { timeoutMs: 3_000 });
+
+  const thermalResult = await runProcess('pmset', ['-g', 'therm'], { timeoutMs: 5_000 });
+  const thermal = parseThermalOutput(`${thermalResult.stdout || ''}${thermalResult.stderr || ''}`);
+  const batteryResult = await runProcess('pmset', ['-g', 'batt'], { timeoutMs: 5_000 });
+  const battery = parseBatteryOutput(`${batteryResult.stdout || ''}${batteryResult.stderr || ''}`);
+
+  let fmHelpDigest = null;
+  try {
+    const help = await getFmHelp(fmBin, 10_000);
+    const text = stripAnsi(help.text).trim();
+    if (text) {
+      fmHelpDigest = crypto.createHash('sha256').update(text).digest('hex').slice(0, 16);
+    }
+  } catch {
+    fmHelpDigest = null;
+  }
+
+  const memRaw = (memBytes.stdout || '').trim();
+  const memoryGb = memRaw && Number.isFinite(Number(memRaw))
+    ? Math.round(Number(memRaw) / (1024 ** 3))
+    : null;
+
   return {
     platform: process.platform,
     arch: process.arch,
     node: process.version,
     host: os.hostname(),
     fmBin,
-    macOS: stripAnsi(swVers.stdout).trim() || null
+    macOS,
+    hwModel: (hwModel.stdout || '').trim() || null,
+    cpuBrand: (cpuBrand.stdout || '').trim() || null,
+    memoryGb,
+    fmHelpDigest,
+    thermal: thermal.available
+      ? {
+        schedulerLimit: thermal.schedulerLimit,
+        healthyIdle: Boolean(thermal.healthyIdle)
+      }
+      : null,
+    power: battery.present
+      ? {
+        pct: battery.pct,
+        onAC: battery.onAC
+      }
+      : null
   };
 }
 

package/src/history.js

@@ -29,6 +29,13 @@ export async function loadHistory(dir) {
     }
   }
 
+  reports.sort((a, b) => {
+    const ta = Date.parse(a.report.startedAt ?? '') || 0;
+    const tb = Date.parse(b.report.startedAt ?? '') || 0;
+    if (tb !== ta) return tb - ta;
+    return a.filePath.localeCompare(b.filePath);
+  });
+
   return reports;
 }
 
@@ -50,8 +57,8 @@ export function renderHistoryReport(reports, options = {}) {
   const MJ = ascii ? '+' : '┼';
   const BJ = ascii ? '+' : '┴';
 
-  const colWidths = [19, 8, 3, 6, 9, 9, 9, 9, 5];
-  const headers = ['STARTED AT', 'MODEL', 'C', 'RUNS', 'TTFT P50', 'E2E P50', 'E2E P95', 'USER T/S', 'SUCC'];
+  const colWidths = [19, 8, 10, 3, 6, 9, 9, 9, 9, 5];
+  const headers = ['STARTED AT', 'MODEL', 'TAG / NOTE', 'C', 'RUNS', 'TTFT P50', 'E2E P50', 'E2E P95', 'USER T/S', 'SUCC'];
 
   const renderRule = (l, j, r) => `${l}${colWidths.map((w) => H.repeat(w + 2)).join(j)}${r}`;
   const renderRowLine = (cells) => `${V}${cells.map((text, i) => ` ${fit(text, colWidths[i])} `).join(V)}${V}`;
@@ -63,15 +70,19 @@ export function renderHistoryReport(reports, options = {}) {
   lines.push(renderRowLine(headers.map((h, i) => h)));
   lines.push(renderRule(ML, MJ, MR));
 
-  for (const { filePath, report } of reports) {
+  for (const { report } of reports) {
     const started = report.startedAt ? report.startedAt.slice(0, 19).replace('T', ' ') : '?';
     const modelRows = report.summary ?? [];
+    const tags = Array.isArray(report.options?.tags) ? report.options.tags.join(',') : '';
+    const note = report.options?.note ? String(report.options.note) : '';
+    const label = tags || note || report.options?.profile || '—';
 
     for (const item of modelRows) {
       if (!item.available) continue;
       const row = [
         started,
         item.model ?? '-',
+        label,
         String(item.concurrency ?? 1),
         String(item.successes ?? '-'),
         item.ttft?.p50 != null ? formatMs(item.ttft.p50) : '-',

package/src/report.js

@@ -43,9 +43,15 @@ export function flattenResults(results) {
 export async function writeReport(filePath, payload, format) {
   const target = path.resolve(filePath);
   await fs.mkdir(path.dirname(target), { recursive: true });
-  const content = format === 'csv'
-    ? toCsv(flattenResults(payload.results))
-    : `${JSON.stringify(payload, null, 2)}\n`;
+  let content;
+  if (format === 'csv') {
+    content = toCsv(flattenResults(payload.results));
+  } else if (format === 'html') {
+    const { renderHtmlReport } = await import('./export.js');
+    content = renderHtmlReport(payload);
+  } else {
+    content = `${JSON.stringify(payload, null, 2)}\n`;
+  }
   await fs.writeFile(target, content, 'utf8');
   return target;
 }

package/src/schema.js

@@ -0,0 +1,158 @@
+/** @typedef {'1'} ReportSchemaVersion */
+
+export const REPORT_SCHEMA_VERSION = '1';
+
+const REQUIRED_TOP_LEVEL = ['tool', 'version', 'startedAt', 'summary', 'options', 'environment'];
+
+/**
+ * @param {unknown} value
+ * @returns {value is Record<string, unknown>}
+ */
+export function isFmBenchReport(value) {
+  if (!value || typeof value !== 'object') return false;
+  const report = /** @type {Record<string, unknown>} */ (value);
+  if (report.tool !== 'fm-bench') return false;
+  for (const key of REQUIRED_TOP_LEVEL) {
+    if (!(key in report)) return false;
+  }
+  if (!Array.isArray(report.summary)) return false;
+  return true;
+}
+
+/**
+ * Stable key for comparing two runs (profile, prompts, runs, concurrency sweep).
+ * @param {Record<string, unknown>} report
+ */
+export function suiteKey(report) {
+  const options = /** @type {Record<string, unknown>} */ (report.options ?? {});
+  const prompts = Array.isArray(report.prompts)
+    ? report.prompts.map((p) => /** @type {{ id?: string }} */ (p).id).filter(Boolean).sort()
+    : [];
+  const sweep = Array.isArray(options.sweepConcurrency) ? [...options.sweepConcurrency].sort((a, b) => a - b) : [];
+  return JSON.stringify({
+    profile: options.profile ?? null,
+    runs: options.runs ?? null,
+    warmup: options.warmup ?? 0,
+    concurrency: options.concurrency ?? 1,
+    sweepConcurrency: sweep,
+    promptIds: prompts,
+    stream: options.stream ?? true,
+    greedy: options.greedy ?? true
+  });
+}
+
+/**
+ * Human-readable hardware + OS fingerprint for sharing and leaderboards.
+ * @param {Record<string, unknown>} report
+ */
+export function environmentFingerprint(report) {
+  const env = /** @type {Record<string, unknown>} */ (report.environment ?? {});
+  const mac = typeof env.macOS === 'string' ? env.macOS : '';
+  const product = mac.match(/ProductVersion:\s*([^\n]+)/)?.[1]?.trim() ?? null;
+  const build = mac.match(/BuildVersion:\s*([^\n]+)/)?.[1]?.trim() ?? null;
+  return {
+    platform: env.platform ?? null,
+    arch: env.arch ?? null,
+    node: env.node ?? null,
+    host: env.host ?? null,
+    hwModel: env.hwModel ?? null,
+    cpuBrand: env.cpuBrand ?? null,
+    memoryGb: env.memoryGb ?? null,
+    macOSProductVersion: product,
+    macOSBuildVersion: build,
+    fmBin: env.fmBin ?? null,
+    fmHelpDigest: env.fmHelpDigest ?? null,
+    thermal: env.thermal ?? null,
+    power: env.power ?? null
+  };
+}
+
+/**
+ * @param {Record<string, unknown>} before
+ * @param {Record<string, unknown>} after
+ */
+export function compareCompatibility(before, after) {
+  const warnings = [];
+  const errors = [];
+
+  if (!isFmBenchReport(before)) errors.push('before file is not a valid fm-bench report');
+  if (!isFmBenchReport(after)) errors.push('after file is not a valid fm-bench report');
+  if (errors.length > 0) {
+    return { compatible: false, warnings, errors, suiteMatch: false };
+  }
+
+  const beforeKey = suiteKey(before);
+  const afterKey = suiteKey(after);
+  const suiteMatch = beforeKey === afterKey;
+  if (!suiteMatch) {
+    warnings.push('benchmark suites differ (profile, runs, prompts, or concurrency). Summary deltas may be misleading.');
+  }
+
+  const bFp = environmentFingerprint(before);
+  const aFp = environmentFingerprint(after);
+  if (bFp.hwModel && aFp.hwModel && bFp.hwModel !== aFp.hwModel) {
+    warnings.push(`hardware model differs (${bFp.hwModel} vs ${aFp.hwModel})`);
+  }
+  if (bFp.macOSProductVersion && aFp.macOSProductVersion && bFp.macOSProductVersion !== aFp.macOSProductVersion) {
+    warnings.push(`macOS version differs (${bFp.macOSProductVersion} vs ${aFp.macOSProductVersion})`);
+  }
+
+  return {
+    compatible: errors.length === 0,
+    warnings,
+    errors,
+    suiteMatch,
+    beforeSuite: beforeKey,
+    afterSuite: afterKey,
+    beforeFingerprint: bFp,
+    afterFingerprint: aFp
+  };
+}
+
+/**
+ * @param {unknown} value
+ * @returns {{ ok: true, report: Record<string, unknown> } | { ok: false, errors: string[] }}
+ */
+export function validateReport(value) {
+  const errors = [];
+  if (!value || typeof value !== 'object') {
+    return { ok: false, errors: ['root must be a JSON object'] };
+  }
+  const report = /** @type {Record<string, unknown>} */ (value);
+  if (report.tool !== 'fm-bench') errors.push('tool must be "fm-bench"');
+  for (const key of REQUIRED_TOP_LEVEL) {
+    if (!(key in report)) errors.push(`missing required field: ${key}`);
+  }
+  if (!Array.isArray(report.summary)) errors.push('summary must be an array');
+  if (report.schemaVersion != null && report.schemaVersion !== REPORT_SCHEMA_VERSION) {
+    errors.push(`unsupported schemaVersion: ${report.schemaVersion} (expected ${REPORT_SCHEMA_VERSION})`);
+  }
+  if (errors.length > 0) return { ok: false, errors };
+  return { ok: true, report };
+}
+
+/**
+ * Attach schema version and suite metadata without mutating caller's object deeply.
+ * @param {Record<string, unknown>} payload
+ * @param {{ reportId?: string }} [meta]
+ */
+export function finalizeReportPayload(payload, meta = {}) {
+  const reportId = meta.reportId ?? cryptoRandomId();
+  return {
+    ...payload,
+    schemaVersion: REPORT_SCHEMA_VERSION,
+    reportId,
+    suite: {
+      key: suiteKey(payload),
+      profile: payload.options?.profile ?? null,
+      promptCount: Array.isArray(payload.prompts) ? payload.prompts.length : 0,
+      fingerprint: environmentFingerprint(payload)
+    }
+  };
+}
+
+function cryptoRandomId() {
+  const bytes = new Uint8Array(8);
+  globalThis.crypto.getRandomValues(bytes);
+  return [...bytes].map((b) => b.toString(16).padStart(2, '0')).join('');
+}