fm-bench 0.1.0 → 0.2.0

package/README.md

@@ -35,12 +35,20 @@ fm-bench
 Example output:
 
 ```text
-+--------+---------+------+----+------+-------+-------+-------+-------+--------+---------+------+
-| model  | status  | runs | ok | fail | p50   | p95   | avg   | tok/s | char/s | out tok | note |
-+--------+---------+------+----+------+-------+-------+-------+-------+--------+---------+------+
-| system | ok      |    3 |  3 |    - | 1.21s | 1.85s | 1.34s |  18.7 |    83  |      24 |      |
-| pcc    | skipped |    - |  - |    - | -     | -     | -     | -     | -      | -       | PCC inference is not available... |
-+--------+---------+------+----+------+-------+-------+-------+-------+--------+---------+------+
+fm-bench 0.2.0 | darwin/arm64 | fm
+prompts 3 | runs 1 | concurrency 1 | stream on | measured 3 | failed 0 | skipped models 0 | elapsed 11.36s
+
+┌────────┬────────┬──────┬────┬─────────┬──────────┬──────────┬─────────┬─────────┬──────────┬───────┬─────┬──────┐
+│ MODEL  │ STATUS │ RUNS │ OK │ SUCCESS │ TTFT P50 │ TTFT P95 │ E2E P50 │ E2E P95 │ TPOT P50 │ TOK/S │ RPS │ NOTE │
+├────────┼────────┼──────┼────┼─────────┼──────────┼──────────┼─────────┼─────────┼──────────┼───────┼─────┼──────┤
+│ system │ ok     │    3 │  3 │    100% │    409ms │    486ms │   2.29s │   5.97s │     14ms │  58.5 │ 0.3 │      │
+└────────┴────────┴──────┴────┴─────────┴──────────┴──────────┴─────────┴─────────┴──────────┴───────┴─────┴──────┘
+
+┌────────┬────────────┬─────────────┬─────────────┬──────────────┬─────────┬──────────┬────────┬──────────────────────────────────┐
+│ MODEL  │ IN TOK AVG │ OUT TOK AVG │ TOTAL TOK/S │ DECODE TOK/S │ E2E P99 │ TPOT P95 │ REPEAT │ DESCRIPTION                      │
+├────────┼────────────┼─────────────┼─────────────┼──────────────┼─────────┼──────────┼────────┼──────────────────────────────────┤
+│ system │         35 │         196 │        59.5 │         75.0 │   6.30s │     15ms │      - │ On-device Apple Foundation Model │
+└────────┴────────────┴─────────────┴─────────────┴──────────────┴─────────┴──────────┴────────┴──────────────────────────────────┘
 ```
 
 ## Commands
@@ -61,6 +69,8 @@ fm-bench doctor [options]
 
 ```sh
 fm-bench --models system,pcc --runs 3 --profile stress
+fm-bench --models system --runs 5 --profile interactive
+fm-bench --models system --runs 3 --profile throughput --warmup 1
 fm-bench --prompt "Reply with exactly: ok" --runs 5
 fm-bench --prompt-file prompts.json --format json --out reports/bench.json
 fm-bench --format csv --out reports/bench.csv
@@ -73,13 +83,14 @@ Useful flags:
 - `--warmup <n>`: warmup runs per model before measurement.
 - `--concurrency <n>`: parallel `fm` processes.
 - `--timeout-ms <n>`: timeout per `fm` call.
-- `--profile quick|standard|stress`: built-in prompt suite.
+- `--profile quick|standard|interactive|throughput|stress`: built-in prompt suite.
 - `--prompt <text>`: custom prompt, repeatable.
 - `--prompt-file <file>`: JSON, JSONL, or blank-line separated text prompts.
 - `--instructions <text>`: passed to `fm respond`.
 - `--available-only`: hide unavailable discovered models.
 - `--capture-output`: include raw model output in JSON reports.
 - `--json`, `--csv`, `--format table|json|csv`: choose output format.
+- `--ascii`: use plain ASCII table borders.
 - `--out <file>`: save a report.
 
 ## Prompt Files
@@ -106,14 +117,23 @@ Plain text files are split on blank lines.
 
 `fm-bench` reports:
 
-- `p50`, `p95`, and average wall-clock latency.
-- average output tokens per second.
-- average characters per second.
-- average output tokens.
+- TTFT, or time to first streamed output.
+- E2E latency, or full response wall-clock latency.
+- TPOT, or decode time per output token after the first output token.
+- output tokens per second per request.
+- total output token throughput across the measured window.
+- requests per second across the measured window.
+- prompt and output token counts.
+- p50, p95, and p99 tail latency views.
+- repeatability across repeated runs of the same prompt.
 - success and failure counts.
 - unavailable model notes.
 
-Token counts come from `fm token-count --quiet`. If `fm` cannot count a response, the token fields are left blank while character throughput is still reported.
+Token counts come from `fm token-count --quiet`. If `fm` cannot count a response, token fields are left blank while character throughput is still reported.
+
+Measured runs stream by default so `fm-bench` can capture TTFT. Use `--no-stream` if you need buffered `fm respond` behavior; TTFT and TPOT fields that depend on streaming will be blank.
+
+See [docs/methodology.md](docs/methodology.md) for the benchmark methodology and source references.
 
 ## Requirements
 

package/docs/methodology.md

@@ -0,0 +1,31 @@
+# Methodology
+
+`fm-bench` measures local `fm` command behavior from the client side. It is meant to answer: "What does this Mac deliver to a terminal user for this prompt suite right now?"
+
+## Sources
+
+The metric set follows common LLM inference benchmark practice:
+
+- Apple introduces the macOS 27 `fm` command as a preinstalled way to use Foundation Models from the terminal and scripts: <https://developer.apple.com/videos/play/wwdc2026/334/>
+- NVIDIA NIM benchmarking defines TTFT, end-to-end latency, inter-token latency / TPOT, tokens per second, and requests per second: <https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html>
+- NVIDIA GenAI-Perf reports TTFT, inter-token latency, request latency, sequence lengths, output token throughput, and JSON/CSV artifacts: <https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html>
+- vLLM benchmark tooling reports end-to-end latency and configurable percentiles, and can save JSON results: <https://docs.vllm.ai/en/latest/benchmarking/cli/>
+- MLCommons describes varying concurrency and reporting verified operating points for TTFT, throughput, interactivity, and response latency rather than interpolated performance: <https://mlcommons.org/2026/03/mlperf-endpoints-gen-ai-benchmarking/>
+
+## Metrics
+
+- `TTFT`: time from starting `fm respond` to the first streamed stdout chunk. This is a practical terminal-side proxy for time to first token.
+- `E2E latency`: time from starting `fm respond` until the process exits and the full response is captured.
+- `generation_ms`: `E2E - TTFT`.
+- `TPOT`: `(E2E - TTFT) / (output_tokens - 1)`. The first output token is excluded so TPOT focuses on decode cadence.
+- `tokens_per_second`: output tokens divided by E2E seconds for one request.
+- `decode_tokens_per_second`: output tokens after the first token divided by generation seconds.
+- `total output token throughput`: all successful output tokens for a model divided by that model's measured wall-clock window.
+- `RPS`: successful requests divided by that model's measured wall-clock window.
+- `repeatability`: for repeated runs of the same prompt, the average share of runs that produced the most common normalized output hash.
+
+## Caveats
+
+`fm-bench` uses `fm token-count --quiet` as the source of token counts, so token values follow Apple's local tokenizer behavior. It does not judge semantic quality unless you provide your own prompt suite and inspect captured outputs with `--capture-output`.
+
+Client-side measurements include process startup, local queueing, model prefill, streaming, detokenization, and terminal pipe overhead. That is intentional for a command-line benchmark, but it is not the same as an internal model-kernel benchmark.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fm-bench",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {
@@ -8,6 +8,7 @@
   },
   "files": [
     "bin",
+    "docs",
     "src",
     "README.md",
     "LICENSE"

package/src/bench.js

@@ -1,3 +1,4 @@
+import crypto from 'node:crypto';
 import { checkModelAvailability, collectEnvironment, countTokens, discoverModels, getQuotaUsage, respond } from './fm.js';
 import { loadPrompts } from './prompts.js';
 import { summarizeByModel } from './stats.js';
@@ -59,6 +60,7 @@ export async function runBenchmark(options = {}) {
   }
 
   const jobs = [];
+  const benchmarkStartedAt = process.hrtime.bigint();
   for (const model of runnableModels) {
     for (const prompt of prompts) {
       for (let run = 1; run <= options.runs; run += 1) {
@@ -69,7 +71,7 @@ export async function runBenchmark(options = {}) {
 
   const results = [];
   await runLimited(jobs, Math.max(1, options.concurrency), async (job) => {
-    const result = await runSingleBenchmark(inspection.fmBin, job, promptTokenCounts, options);
+    const result = await runSingleBenchmark(inspection.fmBin, job, promptTokenCounts, options, benchmarkStartedAt);
     results.push(result);
     if (!result.ok && options.failFast) {
       const error = new Error(result.error || `Benchmark failed for ${job.model.name}`);
@@ -78,6 +80,10 @@ export async function runBenchmark(options = {}) {
     }
   });
 
+  results.sort((a, b) => a.model.localeCompare(b.model)
+    || a.promptId.localeCompare(b.promptId)
+    || a.run - b.run);
+
   const summary = summarizeByModel(results, modelStatuses);
   return {
     tool: 'fm-bench',
@@ -97,15 +103,28 @@ export async function runBenchmark(options = {}) {
   };
 }
 
-async function runSingleBenchmark(fmBin, job, promptTokenCounts, options) {
+async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt) {
+  const startOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
   const response = await respond(fmBin, job.model.name, job.prompt.prompt, {
     ...options,
-    stream: false
+    stream: options.stream
   });
+  const endOffsetMs = Number(process.hrtime.bigint() - benchmarkStartedAt) / 1e6;
   const outputTokens = response.ok
     ? await countTokens(fmBin, response.output, options)
     : { ok: false, count: null };
   const seconds = response.durationMs / 1000;
+  const firstTokenMs = response.firstOutputMs;
+  const generationMs = response.ok && firstTokenMs != null
+    ? Math.max(0, response.durationMs - firstTokenMs)
+    : null;
+  const countedOutputTokens = outputTokens.ok ? outputTokens.count : null;
+  const decodeTokenCount = countedOutputTokens != null ? Math.max(0, countedOutputTokens - 1) : null;
+  const hasDecodeCadence = response.stdoutChunks > 2 && generationMs != null && generationMs > 0 && decodeTokenCount > 0;
+  const tpotMs = hasDecodeCadence ? generationMs / decodeTokenCount : null;
+  const decodeTokensPerSecond = hasDecodeCadence
+    ? decodeTokenCount / (generationMs / 1000)
+    : null;
   const chars = response.output.length;
   const words = response.output.trim() ? response.output.trim().split(/\s+/).length : 0;
 
@@ -115,12 +134,21 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options) {
     run: job.run,
     ok: response.ok,
     durationMs: response.durationMs,
+    firstTokenMs,
+    generationMs,
+    tpotMs,
     promptTokens: promptTokenCounts.get(job.prompt.id),
-    outputTokens: outputTokens.ok ? outputTokens.count : null,
+    outputTokens: countedOutputTokens,
     chars,
     words,
-    tokensPerSecond: outputTokens.ok && seconds > 0 ? outputTokens.count / seconds : null,
+    tokensPerSecond: countedOutputTokens != null && seconds > 0 ? countedOutputTokens / seconds : null,
+    decodeTokensPerSecond,
     charsPerSecond: seconds > 0 ? chars / seconds : 0,
+    startOffsetMs,
+    endOffsetMs,
+    streamed: response.streamed,
+    stdoutChunks: response.stdoutChunks,
+    outputHash: response.ok ? hashOutput(response.output) : null,
     output: options.captureOutput ? response.output : undefined,
     error: response.ok ? '' : response.stderr || `fm exited with code ${response.code ?? response.signal}`
   };
@@ -156,6 +184,14 @@ function publicOptions(options) {
     profile: options.profile,
     promptCount: options.promptCount,
     greedy: options.greedy,
+    stream: options.stream,
     instructions: options.instructions ? '[set]' : ''
   };
 }
+
+function hashOutput(output) {
+  return crypto.createHash('sha256')
+    .update(output.replace(/\s+/g, ' ').trim())
+    .digest('hex')
+    .slice(0, 16);
+}

package/src/cli.js

@@ -3,7 +3,7 @@ import { createRequire } from 'node:module';
 import { inspectModels, runBenchmark } from './bench.js';
 import { runProcess } from './process.js';
 import { flattenResults, toCsv, writeReport } from './report.js';
-import { renderModelsTable, renderSummaryTable } from './table.js';
+import { renderBenchmarkReport, renderModelsTable } from './table.js';
 
 const require = createRequire(import.meta.url);
 const packageJson = require('../package.json');
@@ -31,7 +31,7 @@ export async function runCli(argv = process.argv.slice(2)) {
     if (parsed.format === 'json') {
       console.log(JSON.stringify(inspection.models, null, 2));
     } else {
-      console.log(renderModelsTable(inspection.models));
+      console.log(renderModelsTable(inspection.models, { ascii: parsed.ascii }));
     }
     return;
   }
@@ -46,7 +46,7 @@ export async function runCli(argv = process.argv.slice(2)) {
   } else if (parsed.format === 'csv') {
     console.log(toCsv(flattenResults(payload.results)));
   } else {
-    console.log(renderSummaryTable(payload.summary));
+    console.log(renderBenchmarkReport(payload, { ascii: parsed.ascii }));
     if (parsed.verbose) {
       console.log();
       console.log(toCsv(flattenResults(payload.results)));
@@ -73,11 +73,13 @@ export function parseArgs(argv) {
     timeoutMs: 60_000,
     profile: 'standard',
     greedy: true,
+    stream: true,
     format: 'table',
     captureOutput: false,
     availableOnly: false,
     failFast: false,
-    verbose: false
+    verbose: false,
+    ascii: false
   };
 
   const args = [...argv];
@@ -149,6 +151,12 @@ export function parseArgs(argv) {
       case '--no-greedy':
         options.greedy = false;
         break;
+      case '--stream':
+        options.stream = true;
+        break;
+      case '--no-stream':
+        options.stream = false;
+        break;
       case '--json':
         options.format = 'json';
         break;
@@ -161,6 +169,9 @@ export function parseArgs(argv) {
           throw new Error('--format must be one of: table, json, csv');
         }
         break;
+      case '--ascii':
+        options.ascii = true;
+        break;
       case '-o':
       case '--out':
         options.out = requireValue(arg, args);
@@ -258,12 +269,14 @@ Run options:
       --timeout-ms <n>      Timeout per fm call in ms (default: 60000)
   -p, --prompt <text>       Prompt to benchmark; repeatable
       --prompt-file <file>  .json, .jsonl, or blank-line separated text prompts
-      --profile <name>      quick, standard, or stress (default: standard)
+      --profile <name>      quick, standard, interactive, throughput, or stress
   -i, --instructions <text> Instructions passed to fm respond
       --use-case <case>     Pass a system model use case through to fm
       --guardrails <level>  Pass a system model guardrail level through to fm
       --greedy              Use greedy sampling (default)
       --no-greedy           Do not request greedy sampling
+      --stream              Stream responses while measuring TTFT (default)
+      --no-stream           Disable streaming; TTFT fields will be blank
       --available-only      Hide unavailable discovered models
       --capture-output      Include raw model output in JSON reports
       --fail-fast           Stop after the first failed measured run
@@ -272,6 +285,7 @@ Output:
       --format <type>       table, json, or csv (default: table)
       --json                Alias for --format json
       --csv                 Alias for --format csv
+      --ascii               Use plain ASCII tables instead of Unicode
   -o, --out <file>          Save JSON or CSV report based on file extension
   -v, --verbose             Include per-run CSV after the summary table
 

package/src/fm.js

@@ -149,8 +149,9 @@ export async function countTokens(fmBin, text, options = {}) {
 
 export async function respond(fmBin, model, prompt, options = {}) {
   const args = ['respond', '--model', model];
+  const streamed = options.stream !== false;
 
-  if (options.stream === false) args.push('--no-stream');
+  if (!streamed) args.push('--no-stream');
   if (options.greedy) args.push('--greedy');
   if (options.instructions) args.push('--instructions', options.instructions);
   if (options.useCase) args.push('--use-case', options.useCase);
@@ -172,7 +173,10 @@ export async function respond(fmBin, model, prompt, options = {}) {
     code: result.code,
     signal: result.signal,
     timedOut: result.timedOut,
-    durationMs: result.durationMs
+    durationMs: result.durationMs,
+    firstOutputMs: streamed ? result.firstStdoutMs : null,
+    streamed,
+    stdoutChunks: result.stdoutChunks
   };
 }
 

package/src/process.js

@@ -18,6 +18,10 @@ export function runProcess(command, args = [], options = {}) {
 
     let stdout = '';
     let stderr = '';
+    let stdoutChunks = 0;
+    let stderrChunks = 0;
+    let firstStdoutMs = null;
+    let firstStderrMs = null;
     let timedOut = false;
     let settled = false;
 
@@ -34,9 +38,17 @@ export function runProcess(command, args = [], options = {}) {
     child.stdout.setEncoding('utf8');
     child.stderr.setEncoding('utf8');
     child.stdout.on('data', (chunk) => {
+      stdoutChunks += 1;
+      if (firstStdoutMs == null && chunk.length > 0) {
+        firstStdoutMs = Number(process.hrtime.bigint() - startedAt) / 1e6;
+      }
       stdout += chunk;
     });
     child.stderr.on('data', (chunk) => {
+      stderrChunks += 1;
+      if (firstStderrMs == null && chunk.length > 0) {
+        firstStderrMs = Number(process.hrtime.bigint() - startedAt) / 1e6;
+      }
       stderr += chunk;
     });
 
@@ -51,6 +63,10 @@ export function runProcess(command, args = [], options = {}) {
         signal: null,
         stdout,
         stderr: stderr || error.message,
+        stdoutChunks,
+        stderrChunks,
+        firstStdoutMs,
+        firstStderrMs,
         error,
         timedOut,
         durationMs: Number(endedAt - startedAt) / 1e6
@@ -68,6 +84,10 @@ export function runProcess(command, args = [], options = {}) {
         signal,
         stdout,
         stderr,
+        stdoutChunks,
+        stderrChunks,
+        firstStdoutMs,
+        firstStderrMs,
         timedOut,
         durationMs: Number(endedAt - startedAt) / 1e6
       });

package/src/prompts.js

@@ -10,22 +10,50 @@ const PROFILES = {
   ],
   standard: [
     {
-      id: 'echo-ok',
-      prompt: 'Reply with exactly: ok'
+      id: 'interactive-short',
+      prompt: 'Answer in one sentence: why should an on-device model benchmark report p95 latency?'
     },
     {
-      id: 'explain-latency',
-      prompt: 'In one concise paragraph, explain why AI model benchmarks should report latency percentiles.'
+      id: 'structured-json',
+      prompt: 'Return compact valid JSON with keys "summary" and "risks" for this text: Local model benchmarks should measure latency, throughput, failures, prompt size, and output size.'
     },
     {
-      id: 'json-transform',
-      prompt: 'Convert this list into a valid compact JSON array of strings: alpha, beta, gamma.'
+      id: 'medium-generation',
+      prompt: 'Write a concise four-bullet checklist for evaluating whether a local AI model is fast enough for an interactive coding assistant.'
+    }
+  ],
+  interactive: [
+    {
+      id: 'chat-short-1',
+      prompt: 'Reply in one sentence: what is time to first token?'
+    },
+    {
+      id: 'chat-short-2',
+      prompt: 'Give one practical reason to benchmark with multiple prompt lengths.'
+    },
+    {
+      id: 'chat-short-3',
+      prompt: 'In under 20 words, define throughput for text generation.'
+    }
+  ],
+  throughput: [
+    {
+      id: 'long-explain',
+      prompt: 'Write six concise bullets explaining the tradeoff between latency and throughput in local LLM inference.'
+    },
+    {
+      id: 'long-transform',
+      prompt: 'Rewrite this note as a polished release note with a title and five bullets: fm-bench now measures TTFT, end-to-end latency, output tokens per second, failures, and repeatability.'
+    },
+    {
+      id: 'long-plan',
+      prompt: 'Create a compact test plan for benchmarking a local foundation model across short, medium, and long prompts.'
     }
   ],
   stress: [
     {
-      id: 'echo-ok',
-      prompt: 'Reply with exactly: ok'
+      id: 'interactive-short',
+      prompt: 'Answer in one sentence: why should an on-device model benchmark report p95 latency?'
     },
     {
       id: 'explain-latency',

package/src/report.js

@@ -17,12 +17,19 @@ export function flattenResults(results) {
     run: result.run,
     ok: result.ok,
     duration_ms: round(result.durationMs),
+    ttft_ms: round(result.firstTokenMs),
+    generation_ms: round(result.generationMs),
+    tpot_ms: round(result.tpotMs),
     prompt_tokens: result.promptTokens ?? '',
     output_tokens: result.outputTokens ?? '',
     chars: result.chars,
     words: result.words,
     tokens_per_second: result.tokensPerSecond == null ? '' : round(result.tokensPerSecond),
+    decode_tokens_per_second: result.decodeTokensPerSecond == null ? '' : round(result.decodeTokensPerSecond),
     chars_per_second: round(result.charsPerSecond),
+    streamed: result.streamed,
+    stdout_chunks: result.stdoutChunks,
+    output_hash: result.outputHash || '',
     error: result.error || ''
   }));
 }

package/src/stats.js

@@ -6,8 +6,11 @@ export function summarizeNumbers(values) {
       min: null,
       max: null,
       avg: null,
+      sum: 0,
       p50: null,
-      p95: null
+      p90: null,
+      p95: null,
+      p99: null
     };
   }
 
@@ -17,8 +20,11 @@ export function summarizeNumbers(values) {
     min: clean[0],
     max: clean[clean.length - 1],
     avg: total / clean.length,
+    sum: total,
     p50: percentile(clean, 50),
-    p95: percentile(clean, 95)
+    p90: percentile(clean, 90),
+    p95: percentile(clean, 95),
+    p99: percentile(clean, 99)
   };
 }
 
@@ -64,9 +70,17 @@ export function summarizeByModel(results, modelStatuses = []) {
     const successes = entry.results.filter((result) => result.ok);
     const failures = entry.results.filter((result) => !result.ok);
     const latency = summarizeNumbers(successes.map((result) => result.durationMs));
+    const ttft = summarizeNumbers(successes.map((result) => result.firstTokenMs).filter((value) => value != null));
+    const generation = summarizeNumbers(successes.map((result) => result.generationMs).filter((value) => value != null));
+    const tpot = summarizeNumbers(successes.map((result) => result.tpotMs).filter((value) => value != null));
+    const promptTokens = summarizeNumbers(successes.map((result) => result.promptTokens).filter((value) => value != null));
     const outputTokens = summarizeNumbers(successes.map((result) => result.outputTokens).filter((value) => value != null));
     const charsPerSecond = summarizeNumbers(successes.map((result) => result.charsPerSecond));
     const tokensPerSecond = summarizeNumbers(successes.map((result) => result.tokensPerSecond).filter((value) => value != null));
+    const decodeTokensPerSecond = summarizeNumbers(successes.map((result) => result.decodeTokensPerSecond).filter((value) => value != null));
+    const windowMs = modelWindowMs(successes);
+    const rps = successes.length > 0 && windowMs > 0 ? successes.length / (windowMs / 1000) : null;
+    const outputTokenThroughput = outputTokens.sum > 0 && windowMs > 0 ? outputTokens.sum / (windowMs / 1000) : null;
 
     return {
       model: entry.model,
@@ -76,10 +90,46 @@ export function summarizeByModel(results, modelStatuses = []) {
       attempted: entry.results.length,
       successes: successes.length,
       failures: failures.length,
+      successRate: entry.results.length > 0 ? successes.length / entry.results.length : null,
+      rps,
+      outputTokenThroughput,
+      repeatability: summarizeRepeatability(successes),
       latency,
+      ttft,
+      generation,
+      tpot,
+      promptTokens,
       outputTokens,
       charsPerSecond,
-      tokensPerSecond
+      tokensPerSecond,
+      decodeTokensPerSecond
     };
   });
 }
+
+function modelWindowMs(results) {
+  const starts = results.map((result) => result.startOffsetMs).filter((value) => Number.isFinite(value));
+  const ends = results.map((result) => result.endOffsetMs).filter((value) => Number.isFinite(value));
+  if (starts.length === 0 || ends.length === 0) return null;
+  return Math.max(...ends) - Math.min(...starts);
+}
+
+function summarizeRepeatability(results) {
+  const byPrompt = new Map();
+  for (const result of results) {
+    if (!result.outputHash) continue;
+    if (!byPrompt.has(result.promptId)) byPrompt.set(result.promptId, []);
+    byPrompt.get(result.promptId).push(result.outputHash);
+  }
+
+  const scores = [];
+  for (const hashes of byPrompt.values()) {
+    if (hashes.length < 2) continue;
+    const counts = new Map();
+    for (const hash of hashes) counts.set(hash, (counts.get(hash) || 0) + 1);
+    scores.push(Math.max(...counts.values()) / hashes.length);
+  }
+
+  if (scores.length === 0) return null;
+  return scores.reduce((sum, score) => sum + score, 0) / scores.length;
+}

package/src/table.js

@@ -1,19 +1,42 @@
-export function renderTable(headers, rows) {
+export function renderTable(headers, rows, options = {}) {
+  const ascii = Boolean(options.ascii);
   const stringRows = rows.map((row) => row.map(formatCell));
   const widths = headers.map((header, index) => {
     const values = [header, ...stringRows.map((row) => row[index] ?? '')];
     return Math.max(...values.map(visibleLength));
   });
+  const style = ascii ? ASCII_TABLE : UNICODE_TABLE;
 
-  const separator = `+-${widths.map((width) => '-'.repeat(width)).join('-+-')}-+`;
-  const headerLine = `| ${headers.map((header, index) => pad(header, widths[index])).join(' | ')} |`;
-  const bodyLines = stringRows.map((row) => `| ${row.map((cell, index) => pad(cell, widths[index], isNumericCell(cell))).join(' | ')} |`);
+  const top = rule(style.topLeft, style.topJoin, style.topRight, style.horizontal, widths);
+  const middle = rule(style.midLeft, style.midJoin, style.midRight, style.horizontal, widths);
+  const bottom = rule(style.bottomLeft, style.bottomJoin, style.bottomRight, style.horizontal, widths);
+  const headerLine = rowLine(headers, widths, style, true);
+  const bodyLines = stringRows.map((row) => rowLine(row, widths, style));
 
-  return [separator, headerLine, separator, ...bodyLines, separator].join('\n');
+  return [top, headerLine, middle, ...bodyLines, bottom].join('\n');
+}
+
+export function renderBenchmarkReport(payload, options = {}) {
+  const lines = [];
+  const elapsedMs = Date.parse(payload.finishedAt) - Date.parse(payload.startedAt);
+  const skipped = payload.summary.filter((item) => !item.available).length;
+  const measured = payload.summary.reduce((sum, item) => sum + item.successes, 0);
+  const failed = payload.summary.reduce((sum, item) => sum + item.failures, 0);
+
+  lines.push(`fm-bench ${payload.version} | ${payload.environment.platform}/${payload.environment.arch} | ${payload.environment.fmBin}`);
+  lines.push(`prompts ${payload.prompts.length} | runs ${payload.options.runs} | concurrency ${payload.options.concurrency} | stream ${payload.options.stream ? 'on' : 'off'} | measured ${measured} | failed ${failed} | skipped models ${skipped} | elapsed ${formatMs(elapsedMs)}`);
+  lines.push('');
+  lines.push(renderSummaryTable(payload.summary, options));
+  lines.push('');
+  lines.push(renderDetailTable(payload.summary, options));
+  lines.push('');
+  lines.push('TTFT = time to first streamed output, E2E = full response latency, TPOT = decode time per output token.');
+
+  return lines.join('\n');
 }
 
 export function formatMs(value) {
-  if (value == null) return '-';
+  if (value == null || !Number.isFinite(value)) return '-';
   if (value >= 1000) return `${(value / 1000).toFixed(2)}s`;
   return `${Math.round(value)}ms`;
 }
@@ -24,7 +47,12 @@ export function formatNumber(value, digits = 1) {
   return value.toFixed(digits);
 }
 
-export function renderSummaryTable(summary) {
+export function formatPercent(value, digits = 0) {
+  if (value == null || !Number.isFinite(value)) return '-';
+  return `${(value * 100).toFixed(digits)}%`;
+}
+
+export function renderSummaryTable(summary, options = {}) {
   const rows = summary.map((item) => {
     const status = item.available ? (item.failures > 0 ? 'partial' : 'ok') : 'skipped';
     return [
@@ -32,13 +60,14 @@ export function renderSummaryTable(summary) {
       status,
       item.attempted || '-',
       item.successes || '-',
-      item.failures || '-',
+      formatPercent(item.successRate),
+      formatMs(item.ttft.p50),
+      formatMs(item.ttft.p95),
       formatMs(item.latency.p50),
       formatMs(item.latency.p95),
-      formatMs(item.latency.avg),
+      formatMs(item.tpot.p50),
       formatNumber(item.tokensPerSecond.avg),
-      formatNumber(item.charsPerSecond.avg, 0),
-      formatNumber(item.outputTokens.avg, 0),
+      formatNumber(item.rps),
       item.available ? '' : compactReason(item.skippedReason)
     ];
   });
@@ -48,24 +77,90 @@ export function renderSummaryTable(summary) {
     'status',
     'runs',
     'ok',
-    'fail',
-    'p50',
-    'p95',
-    'avg',
+    'success',
+    'ttft p50',
+    'ttft p95',
+    'e2e p50',
+    'e2e p95',
+    'tpot p50',
     'tok/s',
-    'char/s',
-    'out tok',
+    'rps',
     'note'
-  ], rows);
+  ], rows, options);
 }
 
-export function renderModelsTable(models) {
+export function renderDetailTable(summary, options = {}) {
+  const rows = summary.map((item) => [
+    item.model,
+    formatNumber(item.promptTokens.avg, 0),
+    formatNumber(item.outputTokens.avg, 0),
+    formatNumber(item.outputTokenThroughput),
+    formatNumber(item.decodeTokensPerSecond.avg),
+    formatMs(item.latency.p99),
+    formatMs(item.tpot.p95),
+    formatPercent(item.repeatability),
+    item.description || '-'
+  ]);
+
+  return renderTable([
+    'model',
+    'in tok avg',
+    'out tok avg',
+    'total tok/s',
+    'decode tok/s',
+    'e2e p99',
+    'tpot p95',
+    'repeat',
+    'description'
+  ], rows, options);
+}
+
+export function renderModelsTable(models, options = {}) {
   return renderTable(['model', 'available', 'description', 'quota'], models.map((model) => [
     model.name,
     model.available ? 'yes' : 'no',
     model.description || '-',
     compactReason(model.quota || model.reason || '-')
-  ]));
+  ]), options);
+}
+
+const ASCII_TABLE = {
+  topLeft: '+',
+  topJoin: '+',
+  topRight: '+',
+  midLeft: '+',
+  midJoin: '+',
+  midRight: '+',
+  bottomLeft: '+',
+  bottomJoin: '+',
+  bottomRight: '+',
+  horizontal: '-',
+  vertical: '|'
+};
+
+const UNICODE_TABLE = {
+  topLeft: '┌',
+  topJoin: '┬',
+  topRight: '┐',
+  midLeft: '├',
+  midJoin: '┼',
+  midRight: '┤',
+  bottomLeft: '└',
+  bottomJoin: '┴',
+  bottomRight: '┘',
+  horizontal: '─',
+  vertical: '│'
+};
+
+function rule(left, join, right, horizontal, widths) {
+  return `${left}${widths.map((width) => horizontal.repeat(width + 2)).join(join)}${right}`;
+}
+
+function rowLine(row, widths, style, header = false) {
+  return `${style.vertical}${row.map((cell, index) => {
+    const value = header ? String(cell).toUpperCase() : formatCell(cell);
+    return ` ${pad(value, widths[index], !header && isNumericCell(value))} `;
+  }).join(style.vertical)}${style.vertical}`;
 }
 
 function formatCell(value) {
@@ -80,7 +175,7 @@ function pad(value, width, left = false) {
 }
 
 function isNumericCell(value) {
-  return /^-?$|^[\d,.]+(?:ms|s)?$/.test(value);
+  return /^-?$|^[\d,.]+(?:ms|s|%)?$/.test(value);
 }
 
 function visibleLength(value) {