fm-bench 0.2.0 → 0.3.1

package/README.md

@@ -71,6 +71,8 @@ fm-bench doctor [options]
 fm-bench --models system,pcc --runs 3 --profile stress
 fm-bench --models system --runs 5 --profile interactive
 fm-bench --models system --runs 3 --profile throughput --warmup 1
+fm-bench --models system --profile interactive --sweep-concurrency 1,2,4
+fm-bench --models system --runs 5 --slo-ttft-ms 750 --slo-e2e-ms 4000
 fm-bench --prompt "Reply with exactly: ok" --runs 5
 fm-bench --prompt-file prompts.json --format json --out reports/bench.json
 fm-bench --format csv --out reports/bench.csv
@@ -82,7 +84,9 @@ Useful flags:
 - `--runs <n>`: measured runs per prompt/model.
 - `--warmup <n>`: warmup runs per model before measurement.
 - `--concurrency <n>`: parallel `fm` processes.
+- `--sweep-concurrency <list>`: run separate measured operating points, such as `1,2,4`.
 - `--timeout-ms <n>`: timeout per `fm` call.
+- `--slo-ttft-ms <n>`, `--slo-e2e-ms <n>`, `--slo-tpot-ms <n>`: count goodput against latency budgets.
 - `--profile quick|standard|interactive|throughput|stress`: built-in prompt suite.
 - `--prompt <text>`: custom prompt, repeatable.
 - `--prompt-file <file>`: JSON, JSONL, or blank-line separated text prompts.
@@ -91,6 +95,9 @@ Useful flags:
 - `--capture-output`: include raw model output in JSON reports.
 - `--json`, `--csv`, `--format table|json|csv`: choose output format.
 - `--ascii`: use plain ASCII table borders.
+- `--color`, `--no-color`: force or disable semantic ANSI colors. Colors are automatic on TTYs.
+- `--compact`: force the narrow terminal layout.
+- `--width <n>`: render as if the terminal has `n` columns.
 - `--out <file>`: save a report.
 
 ## Prompt Files
@@ -123,6 +130,8 @@ Plain text files are split on blank lines.
 - output tokens per second per request.
 - total output token throughput across the measured window.
 - requests per second across the measured window.
+- goodput percentage and goodput RPS when SLO flags are set.
+- coefficient of variation (CV) and confidence interval context for stability.
 - prompt and output token counts.
 - p50, p95, and p99 tail latency views.
 - repeatability across repeated runs of the same prompt.
@@ -133,6 +142,20 @@ Token counts come from `fm token-count --quiet`. If `fm` cannot count a response
 
 Measured runs stream by default so `fm-bench` can capture TTFT. Use `--no-stream` if you need buffered `fm respond` behavior; TTFT and TPOT fields that depend on streaming will be blank.
 
+Terminal output is responsive. Wide terminals show full scoreboard and detail tables, medium terminals show a tighter operating-point table, and narrow terminals switch to compact model cards. Use `--width` to preview a layout and `--ascii` for log systems that do not render Unicode borders well.
+
+## Terminal Colors
+
+Table output uses semantic ANSI color on interactive terminals:
+
+- green: passing, steadier, or better than the current comparison set.
+- yellow: marginal, partial, or near a budget.
+- red: failing a budget, unstable, or slower/lower than peers.
+
+Success rate, goodput, repeatability, and CV use fixed benchmark thresholds. Throughput columns use relative ranking within the current run because “good” depends on the machine, model, prompt mix, and concurrency. TTFT, E2E, and TPOT use SLO thresholds when you pass `--slo-ttft-ms`, `--slo-e2e-ms`, or `--slo-tpot-ms`; otherwise they use lower-is-better relative ranking across the models and operating points in the report.
+
+Use `--color` to force ANSI colors in captured logs, or `--no-color` for plain output. `NO_COLOR=1` disables automatic color and `FORCE_COLOR=1` enables it.
+
 See [docs/methodology.md](docs/methodology.md) for the benchmark methodology and source references.
 
 ## Requirements

package/docs/methodology.md

@@ -9,8 +9,9 @@ The metric set follows common LLM inference benchmark practice:
 - Apple introduces the macOS 27 `fm` command as a preinstalled way to use Foundation Models from the terminal and scripts: <https://developer.apple.com/videos/play/wwdc2026/334/>
 - NVIDIA NIM benchmarking defines TTFT, end-to-end latency, inter-token latency / TPOT, tokens per second, and requests per second: <https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html>
 - NVIDIA GenAI-Perf reports TTFT, inter-token latency, request latency, sequence lengths, output token throughput, and JSON/CSV artifacts: <https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html>
-- vLLM benchmark tooling reports end-to-end latency and configurable percentiles, and can save JSON results: <https://docs.vllm.ai/en/latest/benchmarking/cli/>
+- vLLM benchmark tooling reports TTFT, TPOT, ITL, E2E percentiles and SLO-oriented goodput: <https://docs.vllm.ai/en/stable/cli/bench/serve/>
 - MLCommons describes varying concurrency and reporting verified operating points for TTFT, throughput, interactivity, and response latency rather than interpolated performance: <https://mlcommons.org/2026/03/mlperf-endpoints-gen-ai-benchmarking/>
+- MLPerf Client emphasizes local client workloads with multiple task types and varying prompt/response lengths: <https://mlcommons.org/benchmarks/client/>
 
 ## Metrics
 
@@ -22,10 +23,21 @@ The metric set follows common LLM inference benchmark practice:
 - `decode_tokens_per_second`: output tokens after the first token divided by generation seconds.
 - `total output token throughput`: all successful output tokens for a model divided by that model's measured wall-clock window.
 - `RPS`: successful requests divided by that model's measured wall-clock window.
+- `goodput`: successful requests that also satisfy all provided SLO thresholds.
 - `repeatability`: for repeated runs of the same prompt, the average share of runs that produced the most common normalized output hash.
+- `CV`: coefficient of variation, or sample standard deviation divided by the mean. Lower values indicate steadier latency for that metric.
+- `95% CI`: a t-distribution confidence interval around the sample mean. Treat it as useful context, not proof, especially with very small sample sizes.
+
+## Operating Points
+
+Use `--sweep-concurrency 1,2,4` to measure separate concurrency operating points. This follows the same idea as MLCommons endpoint reporting: a single peak number hides the tradeoff between system throughput and per-user responsiveness.
+
+`fm-bench` does not interpolate between operating points. It reports only what was actually measured.
 
 ## Caveats
 
 `fm-bench` uses `fm token-count --quiet` as the source of token counts, so token values follow Apple's local tokenizer behavior. It does not judge semantic quality unless you provide your own prompt suite and inspect captured outputs with `--capture-output`.
 
 Client-side measurements include process startup, local queueing, model prefill, streaming, detokenization, and terminal pipe overhead. That is intentional for a command-line benchmark, but it is not the same as an internal model-kernel benchmark.
+
+For serious comparisons, prefer at least three runs per prompt, include warmups, benchmark both interactive and throughput profiles, and compare models at the same concurrency operating points.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fm-bench",
-  "version": "0.2.0",
+  "version": "0.3.1",
   "description": "Dynamic benchmark CLI for Apple's fm command on macOS 27+.",
   "type": "module",
   "bin": {

package/src/bench.js

@@ -44,15 +44,69 @@ export async function runBenchmark(options = {}) {
   const runnableModels = modelStatuses.filter((model) => model.available);
   const environment = await collectEnvironment(inspection.fmBin);
   const promptTokenCounts = new Map();
+  const concurrencies = normalizeConcurrencySweep(options);
 
   for (const prompt of prompts) {
     const counted = await countTokens(inspection.fmBin, prompt.prompt, options);
     promptTokenCounts.set(prompt.id, counted.ok ? counted.count : null);
   }
 
+  const results = [];
+  const scenarios = [];
+  for (const concurrency of concurrencies) {
+    const scenario = await runScenario({
+      fmBin: inspection.fmBin,
+      prompts,
+      runnableModels,
+      modelStatuses,
+      promptTokenCounts,
+      options,
+      concurrency
+    });
+    scenarios.push(scenario);
+    results.push(...scenario.results);
+  }
+
+  results.sort((a, b) => a.model.localeCompare(b.model)
+    || (a.concurrency ?? 0) - (b.concurrency ?? 0)
+    || a.promptId.localeCompare(b.promptId)
+    || a.run - b.run);
+
+  const summary = summarizeByModel(results, modelStatuses, { concurrencies });
+  return {
+    tool: 'fm-bench',
+    version: options.version,
+    startedAt,
+    finishedAt: new Date().toISOString(),
+    options: publicOptions(options),
+    environment,
+    prompts: prompts.map((prompt) => ({
+      id: prompt.id,
+      prompt: prompt.prompt,
+      promptTokens: promptTokenCounts.get(prompt.id)
+    })),
+    models: modelStatuses,
+    scenarios,
+    summary,
+    results
+  };
+}
+
+async function runScenario(context) {
+  const {
+    fmBin,
+    prompts,
+    runnableModels,
+    modelStatuses,
+    promptTokenCounts,
+    options,
+    concurrency
+  } = context;
+  const startedAt = new Date().toISOString();
+
   for (let warmupIndex = 0; warmupIndex < options.warmup; warmupIndex += 1) {
     for (const model of runnableModels) {
-      await respond(inspection.fmBin, model.name, prompts[0].prompt, {
+      await respond(fmBin, model.name, prompts[0].prompt, {
         ...options,
         stream: false
       });
@@ -64,14 +118,14 @@ export async function runBenchmark(options = {}) {
   for (const model of runnableModels) {
     for (const prompt of prompts) {
       for (let run = 1; run <= options.runs; run += 1) {
-        jobs.push({ model, prompt, run });
+        jobs.push({ model, prompt, run, concurrency });
       }
     }
   }
 
   const results = [];
-  await runLimited(jobs, Math.max(1, options.concurrency), async (job) => {
-    const result = await runSingleBenchmark(inspection.fmBin, job, promptTokenCounts, options, benchmarkStartedAt);
+  await runLimited(jobs, concurrency, async (job) => {
+    const result = await runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchmarkStartedAt);
     results.push(result);
     if (!result.ok && options.failFast) {
       const error = new Error(result.error || `Benchmark failed for ${job.model.name}`);
@@ -84,21 +138,11 @@ export async function runBenchmark(options = {}) {
     || a.promptId.localeCompare(b.promptId)
     || a.run - b.run);
 
-  const summary = summarizeByModel(results, modelStatuses);
   return {
-    tool: 'fm-bench',
-    version: options.version,
+    concurrency,
     startedAt,
     finishedAt: new Date().toISOString(),
-    options: publicOptions(options),
-    environment,
-    prompts: prompts.map((prompt) => ({
-      id: prompt.id,
-      prompt: prompt.prompt,
-      promptTokens: promptTokenCounts.get(prompt.id)
-    })),
-    models: modelStatuses,
-    summary,
+    summary: summarizeByModel(results, modelStatuses, { concurrencies: [concurrency] }),
     results
   };
 }
@@ -130,6 +174,7 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
 
   return {
     model: job.model.name,
+    concurrency: job.concurrency,
     promptId: job.prompt.id,
     run: job.run,
     ok: response.ok,
@@ -149,6 +194,11 @@ async function runSingleBenchmark(fmBin, job, promptTokenCounts, options, benchm
     streamed: response.streamed,
     stdoutChunks: response.stdoutChunks,
     outputHash: response.ok ? hashOutput(response.output) : null,
+    good: response.ok ? evaluateSlo({
+      firstTokenMs,
+      durationMs: response.durationMs,
+      tpotMs
+    }, options) : false,
     output: options.captureOutput ? response.output : undefined,
     error: response.ok ? '' : response.stderr || `fm exited with code ${response.code ?? response.signal}`
   };
@@ -175,23 +225,51 @@ function normalizeModelSelection(models) {
 }
 
 function publicOptions(options) {
+  const concurrencies = normalizeConcurrencySweep(options);
   return {
     models: normalizeModelSelection(options.models),
     runs: options.runs,
     warmup: options.warmup,
     concurrency: options.concurrency,
+    sweepConcurrency: concurrencies.length > 1 ? concurrencies : [],
     timeoutMs: options.timeoutMs,
     profile: options.profile,
     promptCount: options.promptCount,
     greedy: options.greedy,
     stream: options.stream,
+    slo: {
+      ttftMs: options.sloTtftMs || null,
+      e2eMs: options.sloE2eMs || null,
+      tpotMs: options.sloTpotMs || null
+    },
     instructions: options.instructions ? '[set]' : ''
   };
 }
 
+function normalizeConcurrencySweep(options) {
+  if (options.sweepConcurrency?.length) {
+    return [...new Set(options.sweepConcurrency)]
+      .filter((value) => Number.isInteger(value) && value > 0)
+      .sort((a, b) => a - b);
+  }
+  return [Math.max(1, options.concurrency || 1)];
+}
+
 function hashOutput(output) {
   return crypto.createHash('sha256')
     .update(output.replace(/\s+/g, ' ').trim())
     .digest('hex')
     .slice(0, 16);
 }
+
+function evaluateSlo(metrics, options) {
+  const thresholds = [
+    ['firstTokenMs', options.sloTtftMs],
+    ['durationMs', options.sloE2eMs],
+    ['tpotMs', options.sloTpotMs]
+  ].filter(([, threshold]) => Number.isFinite(threshold));
+
+  if (thresholds.length === 0) return null;
+
+  return thresholds.every(([field, threshold]) => metrics[field] != null && metrics[field] <= threshold);
+}

package/src/cli.js

@@ -31,7 +31,7 @@ export async function runCli(argv = process.argv.slice(2)) {
     if (parsed.format === 'json') {
       console.log(JSON.stringify(inspection.models, null, 2));
     } else {
-      console.log(renderModelsTable(inspection.models, { ascii: parsed.ascii }));
+      console.log(renderModelsTable(inspection.models, renderOptions(parsed)));
     }
     return;
   }
@@ -46,7 +46,7 @@ export async function runCli(argv = process.argv.slice(2)) {
   } else if (parsed.format === 'csv') {
     console.log(toCsv(flattenResults(payload.results)));
   } else {
-    console.log(renderBenchmarkReport(payload, { ascii: parsed.ascii }));
+    console.log(renderBenchmarkReport(payload, renderOptions(parsed)));
     if (parsed.verbose) {
       console.log();
       console.log(toCsv(flattenResults(payload.results)));
@@ -70,16 +70,23 @@ export function parseArgs(argv) {
     runs: 1,
     warmup: 0,
     concurrency: 1,
+    sweepConcurrency: [],
     timeoutMs: 60_000,
     profile: 'standard',
     greedy: true,
     stream: true,
+    sloTtftMs: null,
+    sloE2eMs: null,
+    sloTpotMs: null,
     format: 'table',
     captureOutput: false,
     availableOnly: false,
     failFast: false,
     verbose: false,
-    ascii: false
+    ascii: false,
+    color: 'auto',
+    compact: false,
+    width: null
   };
 
   const args = [...argv];
@@ -118,10 +125,25 @@ export function parseArgs(argv) {
       case '--concurrency':
         options.concurrency = parsePositiveInt(requireValue(arg, args), arg);
         break;
+      case '--sweep-concurrency':
+        options.sweepConcurrency = parsePositiveIntList(requireValue(arg, args), arg);
+        if (options.sweepConcurrency.length > 0) {
+          options.concurrency = options.sweepConcurrency[0];
+        }
+        break;
       case '--timeout':
       case '--timeout-ms':
         options.timeoutMs = parsePositiveInt(requireValue(arg, args), arg);
         break;
+      case '--slo-ttft-ms':
+        options.sloTtftMs = parsePositiveInt(requireValue(arg, args), arg);
+        break;
+      case '--slo-e2e-ms':
+        options.sloE2eMs = parsePositiveInt(requireValue(arg, args), arg);
+        break;
+      case '--slo-tpot-ms':
+        options.sloTpotMs = parsePositiveInt(requireValue(arg, args), arg);
+        break;
       case '-p':
       case '--prompt':
         options.prompts.push(requireValue(arg, args));
@@ -172,6 +194,18 @@ export function parseArgs(argv) {
       case '--ascii':
         options.ascii = true;
         break;
+      case '--color':
+        options.color = 'always';
+        break;
+      case '--no-color':
+        options.color = 'never';
+        break;
+      case '--compact':
+        options.compact = true;
+        break;
+      case '--width':
+        options.width = parsePositiveInt(requireValue(arg, args), arg);
+        break;
       case '-o':
       case '--out':
         options.out = requireValue(arg, args);
@@ -251,6 +285,33 @@ function parseNonNegativeInt(value, option) {
   return parsed;
 }
 
+function parsePositiveIntList(value, option) {
+  const parsed = String(value)
+    .split(',')
+    .map((item) => item.trim())
+    .filter(Boolean)
+    .map((item) => parsePositiveInt(item, option));
+  if (parsed.length === 0) throw new Error(`${option} requires at least one positive integer`);
+  return parsed;
+}
+
+function renderOptions(parsed) {
+  return {
+    ascii: parsed.ascii,
+    color: resolveColor(parsed.color),
+    compact: parsed.compact,
+    width: parsed.width
+  };
+}
+
+function resolveColor(value) {
+  if (value === 'always') return true;
+  if (value === 'never') return false;
+  if (process.env.NO_COLOR) return false;
+  if (process.env.FORCE_COLOR && process.env.FORCE_COLOR !== '0') return true;
+  return Boolean(process.stdout.isTTY);
+}
+
 function helpText() {
   return `fm-bench ${packageJson.version}
 
@@ -266,7 +327,12 @@ Run options:
   -r, --runs <n>            Runs per prompt/model (default: 1)
       --warmup <n>          Warmup runs per model before measurement
   -c, --concurrency <n>     Parallel fm processes (default: 1)
+      --sweep-concurrency <list>
+                            Run separate operating points, e.g. 1,2,4
       --timeout-ms <n>      Timeout per fm call in ms (default: 60000)
+      --slo-ttft-ms <n>     Count request as good only if TTFT is <= n
+      --slo-e2e-ms <n>      Count request as good only if E2E latency is <= n
+      --slo-tpot-ms <n>     Count request as good only if TPOT is <= n
   -p, --prompt <text>       Prompt to benchmark; repeatable
       --prompt-file <file>  .json, .jsonl, or blank-line separated text prompts
       --profile <name>      quick, standard, interactive, throughput, or stress
@@ -286,6 +352,10 @@ Output:
       --json                Alias for --format json
       --csv                 Alias for --format csv
       --ascii               Use plain ASCII tables instead of Unicode
+      --color               Force ANSI colors in table output
+      --no-color            Disable ANSI colors in table output
+      --compact             Force compact terminal layout
+      --width <n>           Render for a specific terminal width
   -o, --out <file>          Save JSON or CSV report based on file extension
   -v, --verbose             Include per-run CSV after the summary table
 

package/src/report.js

@@ -13,6 +13,7 @@ export function toCsv(rows) {
 export function flattenResults(results) {
   return results.map((result) => ({
     model: result.model,
+    concurrency: result.concurrency ?? '',
     prompt_id: result.promptId,
     run: result.run,
     ok: result.ok,
@@ -30,6 +31,7 @@ export function flattenResults(results) {
     streamed: result.streamed,
     stdout_chunks: result.stdoutChunks,
     output_hash: result.outputHash || '',
+    good: result.good == null ? '' : result.good,
     error: result.error || ''
   }));
 }

package/src/stats.js

@@ -7,6 +7,10 @@ export function summarizeNumbers(values) {
       max: null,
       avg: null,
       sum: 0,
+      stddev: null,
+      cv: null,
+      ci95Low: null,
+      ci95High: null,
       p50: null,
       p90: null,
       p95: null,
@@ -15,12 +19,22 @@ export function summarizeNumbers(values) {
   }
 
   const total = clean.reduce((sum, value) => sum + value, 0);
+  const avg = total / clean.length;
+  const variance = clean.length > 1
+    ? clean.reduce((sum, value) => sum + (value - avg) ** 2, 0) / (clean.length - 1)
+    : 0;
+  const stddev = Math.sqrt(variance);
+  const margin = clean.length > 1 ? tCritical95(clean.length) * (stddev / Math.sqrt(clean.length)) : 0;
   return {
     count: clean.length,
     min: clean[0],
     max: clean[clean.length - 1],
-    avg: total / clean.length,
+    avg,
     sum: total,
+    stddev,
+    cv: avg !== 0 ? stddev / Math.abs(avg) : null,
+    ci95Low: avg - margin,
+    ci95High: avg + margin,
     p50: percentile(clean, 50),
     p90: percentile(clean, 90),
     p95: percentile(clean, 95),
@@ -40,35 +54,44 @@ export function percentile(sortedValues, percentileValue) {
   return sortedValues[low] * (1 - weight) + sortedValues[high] * weight;
 }
 
-export function summarizeByModel(results, modelStatuses = []) {
+export function summarizeByModel(results, modelStatuses = [], options = {}) {
   const byModel = new Map();
+  const concurrencies = options.concurrencies?.length ? options.concurrencies : [undefined];
 
-  for (const status of modelStatuses) {
-    byModel.set(status.name, {
-      model: status.name,
-      description: status.description,
-      available: status.available,
-      skippedReason: status.available ? '' : status.reason || 'Unavailable',
-      results: []
-    });
+  for (const concurrency of concurrencies) {
+    for (const status of modelStatuses) {
+      const key = summaryKey(status.name, concurrency);
+      byModel.set(key, {
+        model: status.name,
+        concurrency,
+        description: status.description,
+        available: status.available,
+        skippedReason: status.available ? '' : status.reason || 'Unavailable',
+        results: []
+      });
+    }
   }
 
   for (const result of results) {
-    if (!byModel.has(result.model)) {
-      byModel.set(result.model, {
+    const key = summaryKey(result.model, result.concurrency);
+    if (!byModel.has(key)) {
+      byModel.set(key, {
         model: result.model,
+        concurrency: result.concurrency,
         description: '',
         available: true,
         skippedReason: '',
         results: []
       });
     }
-    byModel.get(result.model).results.push(result);
+    byModel.get(key).results.push(result);
   }
 
   return [...byModel.values()].map((entry) => {
     const successes = entry.results.filter((result) => result.ok);
     const failures = entry.results.filter((result) => !result.ok);
+    const goodResults = successes.filter((result) => result.good === true);
+    const goodMeasured = successes.filter((result) => result.good != null);
     const latency = summarizeNumbers(successes.map((result) => result.durationMs));
     const ttft = summarizeNumbers(successes.map((result) => result.firstTokenMs).filter((value) => value != null));
     const generation = summarizeNumbers(successes.map((result) => result.generationMs).filter((value) => value != null));
@@ -80,10 +103,12 @@ export function summarizeByModel(results, modelStatuses = []) {
     const decodeTokensPerSecond = summarizeNumbers(successes.map((result) => result.decodeTokensPerSecond).filter((value) => value != null));
     const windowMs = modelWindowMs(successes);
     const rps = successes.length > 0 && windowMs > 0 ? successes.length / (windowMs / 1000) : null;
+    const goodputRps = goodResults.length > 0 && windowMs > 0 ? goodResults.length / (windowMs / 1000) : null;
     const outputTokenThroughput = outputTokens.sum > 0 && windowMs > 0 ? outputTokens.sum / (windowMs / 1000) : null;
 
     return {
       model: entry.model,
+      concurrency: entry.concurrency,
       description: entry.description,
       available: entry.available,
       skippedReason: entry.skippedReason,
@@ -91,7 +116,9 @@ export function summarizeByModel(results, modelStatuses = []) {
       successes: successes.length,
       failures: failures.length,
       successRate: entry.results.length > 0 ? successes.length / entry.results.length : null,
+      goodputRate: goodMeasured.length > 0 ? goodResults.length / goodMeasured.length : null,
       rps,
+      goodputRps,
       outputTokenThroughput,
       repeatability: summarizeRepeatability(successes),
       latency,
@@ -107,6 +134,49 @@ export function summarizeByModel(results, modelStatuses = []) {
   });
 }
 
+function summaryKey(model, concurrency) {
+  return `${model}::${concurrency ?? 'default'}`;
+}
+
+function tCritical95(n) {
+  const df = Math.max(1, n - 1);
+  const table = {
+    1: 12.706,
+    2: 4.303,
+    3: 3.182,
+    4: 2.776,
+    5: 2.571,
+    6: 2.447,
+    7: 2.365,
+    8: 2.306,
+    9: 2.262,
+    10: 2.228,
+    11: 2.201,
+    12: 2.179,
+    13: 2.16,
+    14: 2.145,
+    15: 2.131,
+    16: 2.12,
+    17: 2.11,
+    18: 2.101,
+    19: 2.093,
+    20: 2.086,
+    21: 2.08,
+    22: 2.074,
+    23: 2.069,
+    24: 2.064,
+    25: 2.06,
+    26: 2.056,
+    27: 2.052,
+    28: 2.048,
+    29: 2.045,
+    30: 2.042
+  };
+  if (df <= 30) return table[df];
+  if (df <= 60) return 2;
+  return 1.96;
+}
+
 function modelWindowMs(results) {
   const starts = results.map((result) => result.startOffsetMs).filter((value) => Number.isFinite(value));
   const ends = results.map((result) => result.endOffsetMs).filter((value) => Number.isFinite(value));

package/src/table.js

@@ -1,8 +1,17 @@
+import { stripAnsi } from './ansi.js';
+
 export function renderTable(headers, rows, options = {}) {
   const ascii = Boolean(options.ascii);
-  const stringRows = rows.map((row) => row.map(formatCell));
+  const maxCellWidth = options.maxCellWidth || 60;
+  const normalizedRows = rows.map((row) => row.map((cell) => {
+    const normalized = normalizeCell(cell);
+    return {
+      ...normalized,
+      text: truncate(normalized.text, maxCellWidth)
+    };
+  }));
   const widths = headers.map((header, index) => {
-    const values = [header, ...stringRows.map((row) => row[index] ?? '')];
+    const values = [truncate(header, maxCellWidth), ...normalizedRows.map((row) => row[index]?.text ?? '')];
     return Math.max(...values.map(visibleLength));
   });
   const style = ascii ? ASCII_TABLE : UNICODE_TABLE;
@@ -10,27 +19,45 @@ export function renderTable(headers, rows, options = {}) {
   const top = rule(style.topLeft, style.topJoin, style.topRight, style.horizontal, widths);
   const middle = rule(style.midLeft, style.midJoin, style.midRight, style.horizontal, widths);
   const bottom = rule(style.bottomLeft, style.bottomJoin, style.bottomRight, style.horizontal, widths);
-  const headerLine = rowLine(headers, widths, style, true);
-  const bodyLines = stringRows.map((row) => rowLine(row, widths, style));
+  const headerLine = rowLine(headers.map((header) => truncate(header, maxCellWidth)), widths, style, true, options);
+  const bodyLines = normalizedRows.map((row) => rowLine(row, widths, style, false, options));
 
   return [top, headerLine, middle, ...bodyLines, bottom].join('\n');
 }
 
 export function renderBenchmarkReport(payload, options = {}) {
+  const width = terminalWidth(options);
+  const mode = options.compact || width < 88
+    ? 'compact'
+    : width < 140
+      ? 'medium'
+      : 'wide';
   const lines = [];
   const elapsedMs = Date.parse(payload.finishedAt) - Date.parse(payload.startedAt);
   const skipped = payload.summary.filter((item) => !item.available).length;
   const measured = payload.summary.reduce((sum, item) => sum + item.successes, 0);
   const failed = payload.summary.reduce((sum, item) => sum + item.failures, 0);
+  const concurrencies = payload.options.sweepConcurrency?.length
+    ? payload.options.sweepConcurrency.join(',')
+    : String(payload.options.concurrency);
+  const slo = formatSlo(payload.options.slo);
 
-  lines.push(`fm-bench ${payload.version} | ${payload.environment.platform}/${payload.environment.arch} | ${payload.environment.fmBin}`);
-  lines.push(`prompts ${payload.prompts.length} | runs ${payload.options.runs} | concurrency ${payload.options.concurrency} | stream ${payload.options.stream ? 'on' : 'off'} | measured ${measured} | failed ${failed} | skipped models ${skipped} | elapsed ${formatMs(elapsedMs)}`);
-  lines.push('');
-  lines.push(renderSummaryTable(payload.summary, options));
+  const title = `fm-bench ${payload.version} | ${payload.environment.platform}/${payload.environment.arch} | ${payload.environment.fmBin}`;
+  const meta = `prompts ${payload.prompts.length} | runs ${payload.options.runs} | concurrency ${concurrencies} | stream ${payload.options.stream ? 'on' : 'off'} | measured ${measured} | failed ${failed} | skipped ${skipped} | elapsed ${formatMs(elapsedMs)}${slo ? ` | ${slo}` : ''}`;
+  lines.push(truncate(title, width));
+  lines.push(truncate(meta, width));
   lines.push('');
-  lines.push(renderDetailTable(payload.summary, options));
+
+  if (mode === 'compact') {
+    lines.push(renderCompactSummary(payload.summary, { ...options, width, slo: payload.options.slo }));
+  } else {
+    lines.push(renderSummaryTable(payload.summary, { ...options, mode, width, slo: payload.options.slo }));
+    lines.push('');
+    lines.push(renderDetailTable(payload.summary, { ...options, mode, width, slo: payload.options.slo }));
+  }
+
   lines.push('');
-  lines.push('TTFT = time to first streamed output, E2E = full response latency, TPOT = decode time per output token.');
+  lines.push(compactLegend(width));
 
   return lines.join('\n');
 }
@@ -53,77 +80,181 @@ export function formatPercent(value, digits = 0) {
 }
 
 export function renderSummaryTable(summary, options = {}) {
+  const mode = options.mode || 'wide';
+  const hasGoodput = summary.some((item) => item.goodputRate != null);
+  const ranks = rankSummary(summary);
   const rows = summary.map((item) => {
     const status = item.available ? (item.failures > 0 ? 'partial' : 'ok') : 'skipped';
-    return [
-      item.model,
-      status,
-      item.attempted || '-',
-      item.successes || '-',
-      formatPercent(item.successRate),
-      formatMs(item.ttft.p50),
-      formatMs(item.ttft.p95),
-      formatMs(item.latency.p50),
-      formatMs(item.latency.p95),
-      formatMs(item.tpot.p50),
-      formatNumber(item.tokensPerSecond.avg),
-      formatNumber(item.rps),
-      item.available ? '' : compactReason(item.skippedReason)
+    const tones = metricTones(item, ranks, options.slo);
+    const base = [
+      cell(formatConcurrency(item.concurrency)),
+      cell(item.model, item.available ? null : 'muted'),
+      cell(status, statusTone(status)),
+      cell(item.attempted ? `${item.successes}/${item.attempted}` : '-', item.failures > 0 ? 'yellow' : item.available ? 'green' : 'muted'),
+      cell(formatPercent(item.successRate), percentTone(item.successRate, 0.95, 1)),
+      cell(formatPercent(item.goodputRate), percentTone(item.goodputRate, 0.8, 1)),
+      cell(formatMs(item.ttft.p50), tones.ttft),
+      cell(formatMs(item.ttft.p95), tones.ttftP95),
+      cell(formatMs(item.latency.p50), tones.e2e),
+      cell(formatMs(item.latency.p95), tones.e2eP95),
+      cell(formatNumber(item.tokensPerSecond.avg), tones.userTps),
+      cell(formatNumber(item.outputTokenThroughput), tones.systemTps),
+      cell(formatNumber(item.rps), tones.rps),
+      cell(formatPercent(item.latency.cv), cvTone(item.latency.cv)),
+      cell(item.available ? '' : compactReason(item.skippedReason), item.available ? null : 'yellow')
     ];
+
+    if (mode === 'medium') {
+      const medium = [
+        base[0],
+        base[1],
+        base[2],
+        base[3]
+      ];
+      if (hasGoodput) medium.push(base[5]);
+      medium.push(base[6], base[8], base[9], base[10], base[11], base[13], base[14]);
+      return medium;
+    }
+
+    const wide = [base[0], base[1], base[2], base[3], base[4]];
+    if (hasGoodput) wide.push(base[5]);
+    wide.push(
+      base[6],
+      base[7],
+      base[8],
+      base[9],
+      cell(formatMs(item.tpot.p50), tones.tpot),
+      cell(formatMs(item.tpot.p95), tones.tpotP95),
+      base[10],
+      base[11],
+      base[12],
+      base[13],
+      base[14]
+    );
+    return wide;
   });
 
-  return renderTable([
-    'model',
-    'status',
-    'runs',
-    'ok',
-    'success',
-    'ttft p50',
-    'ttft p95',
-    'e2e p50',
-    'e2e p95',
-    'tpot p50',
-    'tok/s',
-    'rps',
-    'note'
-  ], rows, options);
+  const mediumHeaders = ['c', 'model', 'status', 'ok', 'good', 'ttft', 'e2e', 'e2e p95', 'user/s', 'sys/s', 'cv', 'note'];
+  const wideHeaders = ['c', 'model', 'status', 'ok/runs', 'succ', 'good', 'ttft', 'ttft p95', 'e2e', 'e2e p95', 'tpot', 'tpot p95', 'user t/s', 'sys t/s', 'rps', 'cv', 'note'];
+  const headers = mode === 'medium'
+    ? (hasGoodput ? mediumHeaders : mediumHeaders.filter((header) => header !== 'good'))
+    : (hasGoodput ? wideHeaders : wideHeaders.filter((header) => header !== 'good'));
+
+  return renderTable(headers, rows, { ...options, maxCellWidth: mode === 'medium' ? 24 : 52 });
 }
 
 export function renderDetailTable(summary, options = {}) {
-  const rows = summary.map((item) => [
-    item.model,
-    formatNumber(item.promptTokens.avg, 0),
-    formatNumber(item.outputTokens.avg, 0),
-    formatNumber(item.outputTokenThroughput),
-    formatNumber(item.decodeTokensPerSecond.avg),
-    formatMs(item.latency.p99),
-    formatMs(item.tpot.p95),
-    formatPercent(item.repeatability),
-    item.description || '-'
-  ]);
-
-  return renderTable([
-    'model',
-    'in tok avg',
-    'out tok avg',
-    'total tok/s',
-    'decode tok/s',
-    'e2e p99',
-    'tpot p95',
-    'repeat',
-    'description'
-  ], rows, options);
+  const mode = options.mode || 'wide';
+  const ranks = rankSummary(summary);
+  const rows = summary.map((item) => {
+    const tones = metricTones(item, ranks, options.slo);
+    const base = [
+      cell(formatConcurrency(item.concurrency)),
+      cell(item.model, item.available ? null : 'muted'),
+      cell(formatNumber(item.promptTokens.avg, 0)),
+      cell(formatNumber(item.outputTokens.avg, 0)),
+      cell(formatNumber(item.decodeTokensPerSecond.avg), tones.decodeTps),
+      cell(formatMs(item.latency.p99), tones.e2eP99),
+      cell(formatRangeMs(item.latency.ci95Low, item.latency.ci95High), cvTone(item.latency.cv)),
+      cell(formatPercent(item.repeatability), percentTone(item.repeatability, 0.5, 0.9)),
+      cell(item.description || '-', 'muted')
+    ];
+
+    if (mode === 'medium') {
+      return base.slice(0, 8);
+    }
+
+    return base;
+  });
+
+  const headers = mode === 'medium'
+    ? ['c', 'model', 'in avg', 'out avg', 'decode t/s', 'e2e p99', 'e2e 95% ci', 'repeat']
+    : [
+      'c',
+      'model',
+      'in tok avg',
+      'out tok avg',
+      'decode tok/s',
+      'e2e p99',
+      'e2e 95% ci',
+      'repeat',
+      'description'
+    ];
+
+  return renderTable(headers, rows, { ...options, maxCellWidth: mode === 'medium' ? 34 : 52 });
+}
+
+export function renderCompactSummary(summary, options = {}) {
+  const width = options.width || 80;
+  const separator = options.ascii ? '-' : '─';
+  const lines = [];
+  const ranks = rankSummary(summary);
+
+  for (const item of summary) {
+    const status = item.available ? (item.failures > 0 ? 'partial' : 'ok') : 'skipped';
+    const title = `${item.model} c${formatConcurrency(item.concurrency)} ${status} ${item.attempted ? `${item.successes}/${item.attempted}` : '-'}`;
+    lines.push(toneText(truncate(title, width), statusTone(status), options));
+
+    if (item.available) {
+      const goodput = item.goodputRate == null ? '' : ` | good ${formatPercent(item.goodputRate)}`;
+      const tones = metricTones(item, ranks, options.slo);
+      lines.push(toneText(truncate(`  TTFT ${formatMs(item.ttft.p50)} p95 ${formatMs(item.ttft.p95)} | E2E ${formatMs(item.latency.p50)} p95 ${formatMs(item.latency.p95)} p99 ${formatMs(item.latency.p99)}`, width), worstTone(tones.ttft, tones.e2e, tones.e2eP95), options));
+      lines.push(toneText(truncate(`  user ${formatNumber(item.tokensPerSecond.avg)} tok/s | system ${formatNumber(item.outputTokenThroughput)} tok/s | RPS ${formatNumber(item.rps)} | CV ${formatPercent(item.latency.cv)}${goodput}`, width), worstTone(tones.userTps, tones.systemTps, cvTone(item.latency.cv), percentTone(item.goodputRate, 0.8, 1)), options));
+      lines.push(toneText(truncate(`  in/out ${formatNumber(item.promptTokens.avg, 0)}/${formatNumber(item.outputTokens.avg, 0)} tok avg | TPOT ${formatMs(item.tpot.p50)} | repeat ${formatPercent(item.repeatability)}`, width), worstTone(tones.tpot, percentTone(item.repeatability, 0.5, 0.9)), options));
+    } else {
+      lines.push(toneText(truncate(`  ${compactReason(item.skippedReason)}`, width), 'yellow', options));
+    }
+    lines.push(separator.repeat(Math.min(width, 72)));
+  }
+
+  if (lines.at(-1)?.startsWith(separator)) lines.pop();
+  return lines.join('\n');
 }
 
 export function renderModelsTable(models, options = {}) {
+  const width = terminalWidth(options);
+  const compact = options.compact || width < 88;
+  if (compact) {
+    return models.map((model) => {
+      const status = model.available ? 'yes' : 'no';
+      return `${model.name} ${toneText(status, model.available ? 'green' : 'yellow', options)} ${compactReason(model.reason || model.description || '-')}`;
+    }).join('\n');
+  }
+
   return renderTable(['model', 'available', 'description', 'quota'], models.map((model) => [
-    model.name,
-    model.available ? 'yes' : 'no',
+    cell(model.name),
+    cell(model.available ? 'yes' : 'no', model.available ? 'green' : 'yellow'),
     model.description || '-',
     compactReason(model.quota || model.reason || '-')
   ]), options);
 }
 
+function formatRangeMs(low, high) {
+  if (low == null || high == null || !Number.isFinite(low) || !Number.isFinite(high)) return '-';
+  return `${formatMs(Math.max(0, low))}..${formatMs(Math.max(0, high))}`;
+}
+
+function formatConcurrency(value) {
+  return value == null ? '1' : String(value);
+}
+
+function terminalWidth(options = {}) {
+  return options.width || process.stdout.columns || 120;
+}
+
+function compactLegend(width) {
+  const text = 'TTFT = first streamed output. E2E = full response. TPOT = post-first-token decode cadence. CV = lower is steadier.';
+  return truncate(text, width);
+}
+
+function formatSlo(slo = {}) {
+  const parts = [];
+  if (slo.ttftMs) parts.push(`TTFT<=${formatMs(slo.ttftMs)}`);
+  if (slo.e2eMs) parts.push(`E2E<=${formatMs(slo.e2eMs)}`);
+  if (slo.tpotMs) parts.push(`TPOT<=${formatMs(slo.tpotMs)}`);
+  return parts.length ? `SLO ${parts.join(',')}` : '';
+}
+
 const ASCII_TABLE = {
   topLeft: '+',
   topJoin: '+',
@@ -156,13 +287,32 @@ function rule(left, join, right, horizontal, widths) {
   return `${left}${widths.map((width) => horizontal.repeat(width + 2)).join(join)}${right}`;
 }
 
-function rowLine(row, widths, style, header = false) {
+function rowLine(row, widths, style, header = false, options = {}) {
   return `${style.vertical}${row.map((cell, index) => {
-    const value = header ? String(cell).toUpperCase() : formatCell(cell);
-    return ` ${pad(value, widths[index], !header && isNumericCell(value))} `;
+    const normalized = header ? { text: String(cell).toUpperCase(), tone: 'header' } : normalizeCell(cell);
+    const value = normalized.text;
+    const padded = pad(value, widths[index], !header && isNumericCell(value));
+    return ` ${toneText(padded, normalized.tone, options)} `;
   }).join(style.vertical)}${style.vertical}`;
 }
 
+function cell(text, tone = null) {
+  return { text, tone };
+}
+
+function normalizeCell(value) {
+  if (value && typeof value === 'object' && Object.hasOwn(value, 'text')) {
+    return {
+      text: formatCell(value.text),
+      tone: value.tone || null
+    };
+  }
+  return {
+    text: formatCell(value),
+    tone: null
+  };
+}
+
 function formatCell(value) {
   if (value == null) return '';
   return String(value);
@@ -179,7 +329,7 @@ function isNumericCell(value) {
 }
 
 function visibleLength(value) {
-  return String(value).length;
+  return stripAnsi(value).length;
 }
 
 function compactReason(value) {
@@ -187,3 +337,119 @@ function compactReason(value) {
   if (clean.length <= 58) return clean;
   return `${clean.slice(0, 55)}...`;
 }
+
+function truncate(value, width) {
+  const text = String(value ?? '');
+  if (visibleLength(text) <= width) return text;
+  if (width <= 1) return '…';
+  return `${text.slice(0, width - 1)}…`;
+}
+
+const TONES = {
+  header: ['\x1b[1m', '\x1b[0m'],
+  green: ['\x1b[32m', '\x1b[0m'],
+  yellow: ['\x1b[33m', '\x1b[0m'],
+  red: ['\x1b[31m', '\x1b[0m'],
+  muted: ['\x1b[2m', '\x1b[0m']
+};
+
+function toneText(text, tone, options = {}) {
+  if (!options.color || !tone || !TONES[tone]) return text;
+  const [open, close] = TONES[tone];
+  return `${open}${text}${close}`;
+}
+
+function statusTone(status) {
+  if (status === 'ok') return 'green';
+  if (status === 'partial') return 'yellow';
+  if (status === 'skipped') return 'yellow';
+  return 'red';
+}
+
+function percentTone(value, yellowAt, greenAt) {
+  if (value == null || !Number.isFinite(value)) return null;
+  if (value >= greenAt) return 'green';
+  if (value >= yellowAt) return 'yellow';
+  return 'red';
+}
+
+function cvTone(value) {
+  if (value == null || !Number.isFinite(value)) return null;
+  if (value <= 0.1) return 'green';
+  if (value <= 0.25) return 'yellow';
+  return 'red';
+}
+
+function thresholdTone(value, threshold) {
+  if (value == null || !Number.isFinite(value) || !Number.isFinite(threshold)) return null;
+  if (value <= threshold * 0.8) return 'green';
+  if (value <= threshold) return 'yellow';
+  return 'red';
+}
+
+function rankSummary(summary) {
+  return {
+    ttft: collectMetric(summary, (item) => item.ttft?.p50),
+    ttftP95: collectMetric(summary, (item) => item.ttft?.p95),
+    e2e: collectMetric(summary, (item) => item.latency?.p50),
+    e2eP95: collectMetric(summary, (item) => item.latency?.p95),
+    e2eP99: collectMetric(summary, (item) => item.latency?.p99),
+    tpot: collectMetric(summary, (item) => item.tpot?.p50),
+    tpotP95: collectMetric(summary, (item) => item.tpot?.p95),
+    userTps: collectMetric(summary, (item) => item.tokensPerSecond?.avg),
+    systemTps: collectMetric(summary, (item) => item.outputTokenThroughput),
+    rps: collectMetric(summary, (item) => item.rps),
+    decodeTps: collectMetric(summary, (item) => item.decodeTokensPerSecond?.avg)
+  };
+}
+
+function collectMetric(summary, accessor) {
+  return summary.map(accessor).filter((value) => Number.isFinite(value));
+}
+
+function rankTone(value, values) {
+  if (value == null || !Number.isFinite(value) || values.length === 0) return null;
+  if (values.length === 1) return 'green';
+  const min = Math.min(...values);
+  const max = Math.max(...values);
+  if (max === min) return 'green';
+  const position = (value - min) / (max - min);
+  if (position >= 0.67) return 'green';
+  if (position >= 0.34) return 'yellow';
+  return 'red';
+}
+
+function rankToneLower(value, values) {
+  if (value == null || !Number.isFinite(value) || values.length === 0) return null;
+  if (values.length === 1) return 'green';
+  const min = Math.min(...values);
+  const max = Math.max(...values);
+  if (max === min) return 'green';
+  const position = (max - value) / (max - min);
+  if (position >= 0.67) return 'green';
+  if (position >= 0.34) return 'yellow';
+  return 'red';
+}
+
+function metricTones(item, ranks, slo = {}) {
+  return {
+    ttft: thresholdTone(item.ttft?.p50, slo.ttftMs) || rankToneLower(item.ttft?.p50, ranks.ttft),
+    ttftP95: thresholdTone(item.ttft?.p95, slo.ttftMs) || rankToneLower(item.ttft?.p95, ranks.ttftP95),
+    e2e: thresholdTone(item.latency?.p50, slo.e2eMs) || rankToneLower(item.latency?.p50, ranks.e2e),
+    e2eP95: thresholdTone(item.latency?.p95, slo.e2eMs) || rankToneLower(item.latency?.p95, ranks.e2eP95),
+    e2eP99: thresholdTone(item.latency?.p99, slo.e2eMs) || rankToneLower(item.latency?.p99, ranks.e2eP99),
+    tpot: thresholdTone(item.tpot?.p50, slo.tpotMs) || rankToneLower(item.tpot?.p50, ranks.tpot),
+    tpotP95: thresholdTone(item.tpot?.p95, slo.tpotMs) || rankToneLower(item.tpot?.p95, ranks.tpotP95),
+    userTps: rankTone(item.tokensPerSecond?.avg, ranks.userTps),
+    systemTps: rankTone(item.outputTokenThroughput, ranks.systemTps),
+    rps: rankTone(item.rps, ranks.rps),
+    decodeTps: rankTone(item.decodeTokensPerSecond?.avg, ranks.decodeTps)
+  };
+}
+
+function worstTone(...tones) {
+  if (tones.includes('red')) return 'red';
+  if (tones.includes('yellow')) return 'yellow';
+  if (tones.includes('green')) return 'green';
+  return null;
+}