webperf-snippets 1.10.0 → 1.11.0

package/README.md

@@ -73,7 +73,7 @@ npx webperf-snippets https://example.com --json
 The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait, the version of the CLI and the machine: its CPU model, cores, architecture and memory, as the operating system reports them, and a benchmark index, a short CPU test in Node before Chromium starts, which only informs, since on the same machine it moves by up to 40 % with the load.
 
 ```json
-"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.10.0", "machine": { "cpu": "Apple M4", "cores": 10, "arch": "arm64", "memoryGB": 16, "benchmarkIndex": 2371 } }
+"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.11.0", "machine": { "cpu": "Apple M4", "cores": 10, "arch": "arm64", "memoryGB": 16, "benchmarkIndex": 2371 } }
 ```
 
 Single snippet:
@@ -138,13 +138,13 @@ Get a key for the [Chrome UX Report API](https://developer.chrome.com/docs/crux/
 
 With a key, the field data also gives what CrUX knows of the LCP when it is an image: the p75 of its sub-parts, TTFB, load delay, load duration and render delay. When the measurement ran LCP-Subparts, with `--snippet LCP-Subparts` or the `core-web-vitals` workflow over 2.5 s, a table puts each one next to its lab value, with a note where one side is at least 1.5 times the other and 100 ms apart: a slower server or network for users, an image the lab finds sooner, slower connections, slower devices, or a lab slower than users. The field sub-parts describe only the loads whose LCP is an image, and the table says which share that is; when the LCP of the lab is not an image, only TTFB and render delay compare. Each p75 is taken on its own, so the sub-parts do not add up to the p75 of LCP. A line gives the navigation types of the users, by share, and points to Back-Forward-Cache when over 5 % of the visits come back without the back/forward cache. The JSON output carries them in `crux.subParts`, `crux.lcpImageShare`, `crux.roundTripTime` and `crux.navigationTypes`.
 
-`--crux-history <weeks>` adds the trend of the field data over the last 1 to 40 weeks, from the [CrUX History API](https://developer.chrome.com/docs/crux/history-api), with the same key. Each week is the p75 of the 28 days before it, so the weeks overlap and the series is smooth. For LCP, CLS and INP the section gives the first and the last week, the change between them and its direction, `stable` within 5 %, and the weeks as a sparkline, where a gap is a week with too little data:
+`--crux-history <weeks>` adds the trend of the field data over the last 1 to 40 weeks, from the [CrUX History API](https://developer.chrome.com/docs/crux/history-api), with the same key. Each week is the p75 of the 28 days before it, so the weeks overlap and the series is smooth. For LCP, CLS and INP the section gives the first and the last week, the change between them and its direction, `stable` within 5 %, and the weeks as a sparkline, where a gap is a week with too little data. The bars span at least the smallest change `--baseline` counts for the metric (100 ms for LCP, 50 ms for INP, 0.05 for CLS), so a series that barely moves draws almost flat:
 
 ```
 Field data trend (CrUX p75, 40 weeks, mobile, origin-level)
   LCP   2.88s → 3.10s       ▲ +7.7%   worsening   ▃▃▄▃▃▃▄▆▇██▇▆▄▂▁▂▃▃▄▅▆▆▆▆▆▆
   CLS   0.2600 → 0.2100     ▼ −19.2%  improving   ▅▆▆▇▇█▇▆▆▅▄▃▃▃▃▃▂▂▁▂▂▂▂▂▂▂
-  INP   253ms → 251ms       ▼ −0.8%   stable      ▇▅▃▂▂▄▆██▇▅▃▂▃▄▆▅▃▂▁▂▃▃▄▅▆
+  INP   253ms → 251ms       ▼ −0.8%   stable      ▅▅▅▄▄▄▅▅▆▆▆▅▅▄▄▄▄▅▅▅▅▄▄▄▃▃▄▄▄▅▅
 ```
 
 The JSON output carries it in `crux.history`, with the periods, the p75 of each week and the trend of each metric. Like the p75, it uses the page when it has data of its own and its origin otherwise, and says which in `scope`.
@@ -171,7 +171,7 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.10.0" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.11.0" }
 }
 ```
 
@@ -398,8 +398,8 @@ It needs the Chromium of Playwright (`npx playwright install chromium`); without
 | Tool            | What it does                                                                 |
 | --------------- | ---------------------------------------------------------------------------- |
 | `list_snippets` | Lists the snippets with their category, title, short aliases and whether they track interactions. Takes an optional `category`. |
-| `run_snippet`   | Opens a URL in a headless Chromium, runs one snippet and returns a summary of its result. Takes `url`, `snippet` (name, alias or `Category/Name`), and optional `viewport` and `waitMs`. The snippets that record interactions run on a page opened with `open_page`. |
-| `run_workflow`  | Runs a workflow of the CLI (`core-web-vitals`, `audit`, `loading`, `interaction`) and the follow-ups of its decision tree. Takes `url`, `workflow`, and optional `viewport` and `waitMs`. |
+| `run_snippet`   | Opens a URL in a headless Chromium, runs one snippet and returns a summary of its result. Takes `url`, `snippet` (name, alias or `Category/Name`), and optional `viewport`, `waitMs`, `runs`, `baseline` and `thresholds`. The snippets that record interactions run on a page opened with `open_page`. |
+| `run_workflow`  | Runs a workflow of the CLI (`core-web-vitals`, `audit`, `loading`, `interaction`) and the follow-ups of its decision tree. Takes `url`, `workflow`, and optional `viewport`, `waitMs`, `runs`, `baseline` and `thresholds`. |
 | `get_crux`      | Returns the CrUX p75 of LCP, CLS and INP for a URL, or its origin. Takes `url` and an optional `formFactor` (`phone`, `desktop`, `tablet`). Needs `CRUX_API_KEY` in the environment of the server. |
 | `open_page`     | Opens a URL in a page that stays open between calls and installs tracking snippets (`track`, such as `INP`). Returns a `sessionId`. |
 | `interact`      | Runs steps on an open page: `click`, `type`, `hover`, `scroll` or `wait`. |
@@ -407,6 +407,14 @@ It needs the Chromium of Playwright (`npx playwright install chromium`); without
 | `get_details`   | Returns the whole object of one snippet of a run, by the `runId` that `run_snippet`, `run_workflow` or `read_results` returned, without measuring again. Takes `runId` and `snippet`. |
 | `close_page`    | Closes an open page. |
 
+`run_snippet` and `run_workflow` take `runs`, from 1 to 9, as `--runs`: the page loads that many times, one after the other, and the answer is the median run plus a `runs` field, the one of the JSON, with the median, min, max, the value of each run and the agreement of each metric. A run that fails counts in `failed` and is left out; when every run fails, the answer is a tool error. The server keeps the results of the median run only, as the JSON does, so `get_details` answers from that run and names it in `medianRun`; a metric other than the one that picked it (`medianBy`) can differ there from its median in `runs`.
+
+Each run is a page load plus `waitMs`. With the default wait of 3 seconds, five runs of a workflow take about 20 seconds on a local page, and about a minute on a page that takes a few seconds to load. When the client asks for progress, the server reports the three stages of each run, numbered across the runs. A client sets its own limit on a tool call: the MCP SDK stops waiting after 60 seconds unless it is given a longer `timeout` or `resetTimeoutOnProgress`, and in Claude Code `MCP_TOOL_TIMEOUT` sets the limit. When the client gives up, the run in progress ends and no other starts.
+
+`run_snippet` and `run_workflow` compare a run with an earlier one, with the verdict of `--baseline`. `baseline` takes the `runId` of an earlier run of the connection, which every run of these two tools returns, or the path of a page the CLI wrote with `--json`, absolute or relative to the working directory of the server. The answer adds a `baseline` field: the one of the JSON of the CLI, with the `runId` or the `file` it compared with, plus `regressions`, the same lines `--fail-on-regression` prints. `thresholds` takes the values of `--threshold`, such as `["LCP=15%", "CLS=0.1"]`. A baseline that cannot be read or a threshold the server does not understand is a tool error, before the page loads, and a baseline of a matrix is refused, since a tool measures one page.
+
+With five runs or more a side, a regression needs a Mann-Whitney U p-value under 0.05 on top of its threshold, as in the CLI, and each metric adds `stats` with the p-value, the shift and its 95 % interval. With fewer, a metric compares by its threshold and, against a baseline of several runs, by the range of those runs. A single run can be off by more than a threshold on a page that did not change, so read `warnings`, and measure both sides with `runs: 5`, or `--runs 5` in the CLI, when the verdict matters. An earlier run of the connection works for a change made in the same session, and a file measured with the CLI for a comparison with `main`.
+
 The server also offers resources, documents a client can attach to the context:
 
 | Resource                         | What it has                                                              |
@@ -414,6 +422,7 @@ The server also offers resources, documents a client can attach to the context:
 | `webperf://schema`               | The contract every snippet result follows: base shape, statuses, how to read a result, the rules of the issue codes. |
 | `webperf://schema/{snippet}`     | The fields one snippet returns, with an example, and the codes of its issues. |
 | `webperf://workflows/{category}` | The workflows and the decision tree of a category.                       |
+| `webperf://runs/{runId}`         | A run of `run_snippet` or `run_workflow` of the connection, in the shape of the JSON of the CLI. The list names each run with its snippet or workflow, its URL, its viewport, how many runs it measured when they are several, and its time. |
 
 And prompts, templates that chain the tools (in Claude Code, `/mcp__webperf__audit_lcp`): `audit_lcp`, `debug_inp` and `audit_page`, each with a `url` argument.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.10.0",
+  "version": "1.11.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",

package/snippets/SCHEMA.md

@@ -64,7 +64,7 @@ Every script must return an object matching this shape:
 
   // Tracking scripts (status: "tracking")
   message?: string;      // Human-readable status message
-  getDataFn?: string;    // window function to call for data: evaluate_script(`${getDataFn}()`). May be a dotted path such as "loafHelpers.getData"
+  getDataFn?: string;    // window function to call for data: evaluate_script(`() => ${getDataFn}()`). May be a dotted path such as "loafHelpers.getData"
 
   // Error info (status: "error" or "unsupported")
   error?: string;
@@ -169,7 +169,7 @@ return {
 ```
 1. evaluate_script(INP.js)          → { status: "tracking", getDataFn: "getINP" }
 2. (user interacts with the page)
-3. evaluate_script("getINP()")      → { script: "INP", status: "ok", value: 350, rating: "needs-improvement", ... }
+3. evaluate_script("() => getINP()") → { script: "INP", status: "ok", value: 350, rating: "needs-improvement", ... }
 ```
 
 **The window function must also return a structured object** matching the same schema, including when nothing has been recorded yet:
@@ -181,7 +181,7 @@ return {
 
 ### Pattern 4: Async scripts
 
-Scripts that use `async/await` or `setTimeout`. The IIFE returns a Promise, which `evaluate_script` can await (Chrome DevTools `awaitPromise`).
+Scripts that use `async/await` or `setTimeout`. The IIFE returns a Promise, which `evaluate_script` awaits with `format: "script"` (Chrome DevTools `awaitPromise`).
 
 Keep the existing `async () => {}` wrapper. Add a `return` statement with structured data at the end. The agent receives the resolved value.
 
@@ -1622,7 +1622,7 @@ Each issue names its video in `target`, except `no-videos`.
 // Read the return value: the skill scripts have no console output
 result = evaluate_script(scriptCode)
 if result.status == "ok" → use result.value, result.rating, result.details, result.items
-if result.status == "tracking" → call evaluate_script(`${result.getDataFn}()`) after user interaction
+if result.status == "tracking" → call evaluate_script(`() => ${result.getDataFn}()`) after user interaction
 if result.status == "error" → check result.error, the browser may not have loaded the page yet
 if result.status == "unsupported" → browser does not support the required API (check: Chrome 107+?)
 ```
@@ -1637,7 +1637,7 @@ result = evaluate_script(INP_js)
 // 2. Wait for/trigger user interactions
 
 // 3. Collect data
-data = evaluate_script("getINP()")
+data = evaluate_script("() => getINP()")
 // data = { status: "ok", value: 350, rating: "needs-improvement", ... }
 ```
 

package/src/baseline.js

@@ -23,7 +23,8 @@ export const DEFAULT_THRESHOLDS = {
 // The baseline in a file, as { kind: "page" | "matrix", file, data }. Throws with what to do when
 // the file is not an output of the CLI.
 export function readBaseline(text, file) {
-  const howTo = `save one with webperf-snippets <url> --json > ${file}`;
+  // Never this path as the place to save it: the file is something else, and an agent may follow the message
+  const howTo = "give a file saved with webperf-snippets <url> --json";
   let data;
   try {
     data = JSON.parse(text);
@@ -49,18 +50,18 @@ export function loadBaseline(path) {
 // (absolute), CLS=0.1 (CLS has only an absolute part, with no unit)
 export function parseThreshold(raw) {
   const match = /^([A-Za-z]+)=(\d+(?:\.\d+)?)(%|ms)?$/.exec(raw.trim());
-  if (!match) throw new Error(`Invalid --threshold "${raw}": give METRIC=VALUE with its unit, such as LCP=15%, LCP=200ms or CLS=0.1`);
+  if (!match) throw new Error(`Invalid threshold "${raw}": give METRIC=VALUE with its unit, such as LCP=15%, LCP=200ms or CLS=0.1`);
   const [, name, number, unit] = match;
   const metric = name.toUpperCase();
-  if (!METRICS.includes(metric)) throw new Error(`Unknown metric "${name}" in --threshold "${raw}". Choose from: ${METRICS.join(", ")}`);
+  if (!METRICS.includes(metric)) throw new Error(`Unknown metric "${name}" in threshold "${raw}". Choose from: ${METRICS.join(", ")}`);
   const value = Number(number);
   if (metric === "CLS") {
-    if (unit) throw new Error(`Invalid --threshold "${raw}": CLS is a score, with no unit and no relative part, such as CLS=0.1`);
+    if (unit) throw new Error(`Invalid threshold "${raw}": CLS is a score, with no unit and no relative part, such as CLS=0.1`);
     return { metric, absolute: value };
   }
   if (unit === "%") return { metric, relative: value / 100 };
   if (unit === "ms") return { metric, absolute: value };
-  throw new Error(`Invalid --threshold "${raw}": give the unit, such as ${metric}=15% or ${metric}=200ms`);
+  throw new Error(`Invalid threshold "${raw}": give the unit, such as ${metric}=15% or ${metric}=200ms`);
 }
 
 // The default thresholds with the parts that --threshold changes
@@ -153,8 +154,17 @@ function targetOf(target, pageUrl) {
 // (69.21 KB)…" and "Render-blocking CSS (71.80 KB)…"
 const byMessage = (error) => `${error.id}\u0000${error.message.replace(/\d+(?:[.,]\d+)*/g, "#")}`;
 
+// One snippet measured on both sides is the same result, whatever name it was asked by: the CLI names
+// the result of --snippet as it was typed (an alias, its name or Category/Name), and the MCP server by
+// its path. Its conditions name it by its path on both.
+function baseResultsOf(base, current) {
+  const path = base.conditions?.snippet;
+  if (!path || path !== current.conditions?.snippet || current.results?.length !== 1) return base.results ?? [];
+  return base.results.map((r) => ({ ...r, id: current.results[0].id }));
+}
+
 function compareErrors(base, current) {
-  const baseErrors = blockingErrors(base.results ?? []);
+  const baseErrors = blockingErrors(baseResultsOf(base, current));
   const currentErrors = blockingErrors(current.results ?? []);
   // The snippets whose errors on one side all lack a code pair by message on both
   const uncoded = (errors) => {
@@ -209,7 +219,8 @@ function warningsOf(base, current) {
   if (describeLoad(base.throttling) !== describeLoad(current.throttling)) {
     warnings.push(`the baseline was measured ${describeLoad(base.throttling)}, and this run ${describeLoad(current.throttling)}`);
   }
-  if (runCount(base) < 2) warnings.push("the baseline has 1 run, so only the thresholds apply; measure both with --runs");
+  // In words for both: the CLI asks for runs with --runs, and the tools of the MCP server with runs
+  if (runCount(base) < 2) warnings.push(`the baseline has 1 run, so only the thresholds apply; measure both with ${RUNS_FOR_STATISTICS} runs or more`);
   else if (runCount(base) !== runCount(current)) warnings.push(`the baseline has ${runCount(base)} runs, and this run ${runCount(current)}`);
   const machine = machineWarning(b?.machine, c?.machine);
   if (machine) warnings.push(machine);

package/src/bin.js

@@ -15,7 +15,7 @@ import { MissingBrowserError } from "./browser.js";
 import { parseHeader } from "./headers.js";
 import { NETWORK_PRESETS, parseCpuFactor } from "./throttling.js";
 import { parseOwnDomains } from "./own-domains.js";
-import { MEDIAN_METRICS, summarizeRuns } from "./runs.js";
+import { MEDIAN_METRICS, measureRuns } from "./runs.js";
 import { loadMatrix } from "./matrix.js";
 import { blockingErrors } from "./blocking.js";
 import { MEASURED_BY, checkBudgets, parseBudgets, unmeasuredMetrics, violationsOf } from "./budgets.js";
@@ -325,7 +325,7 @@ async function main() {
   if (runCount == null) {
     payload = await measureWith(conditions);
   } else {
-    const median = await measureRuns(() => measureWith(conditions), runCount, medianBy, (i) => `[${i}/${runCount}] ${url} · run ${i}/${runCount}`);
+    const median = await measureRunsWithProgress(() => measureWith(conditions), runCount, medianBy, (i) => `[${i}/${runCount}] ${url} · run ${i}/${runCount}`);
     if (!median.payload) throw new Error(`Every run failed: ${median.errors.join("; ")}`);
     payload = { ...median.payload, runs: median.runs };
   }
@@ -388,22 +388,16 @@ async function main() {
   process.exit(blockingErrors(payload.results).length > 0 ? 1 : 0);
 }
 
-// Measures `count` runs one after the other, never in parallel: they would compete for the CPU and
-// skew the timings. Each run is reported on stderr as it starts, so stdout keeps only the report. A
-// run that fails is kept as { error } and left out of the median.
-async function measureRuns(measure, count, medianBy, progress) {
-  const runs = [];
-  for (let i = 1; i <= count; i++) {
-    process.stderr.write(`${progress(i)}\n`);
-    try {
-      runs.push(await measure());
-    } catch (err) {
-      if (err instanceof MissingBrowserError) throw err;
-      runs.push({ error: err.message });
-    }
-  }
-  return summarizeRuns(runs, { medianBy });
-}
+// Each run is reported on stderr as it starts, so stdout keeps only the report
+const measureRunsWithProgress = (measure, count, medianBy, progress) =>
+  measureRuns(
+    (i) => {
+      process.stderr.write(`${progress(i)}\n`);
+      return measure();
+    },
+    count,
+    { medianBy, isFatal: (err) => err instanceof MissingBrowserError }
+  );
 
 // Every combination of the matrix, in order, each one `matrix.runs` times. Each combination reports
 // its median run, and the exit code follows the same rules as one page, for every combination.
@@ -417,7 +411,7 @@ async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, repo
   const combinations = [];
   for (const c of matrix.combinations) {
     const conditions = { url: c.url, viewport: VIEWPORT_PRESETS[c.viewport], storageState: c.storageState, headers: c.headers, cpu: c.cpu, network: c.network, ownDomains: c.ownDomains };
-    const median = await measureRuns(() => measureWith(conditions), matrix.runs, matrix.medianBy, (i) => `[${String(++started).padStart(width)}/${total}] ${c.id} · run ${i}/${matrix.runs}`);
+    const median = await measureRunsWithProgress(() => measureWith(conditions), matrix.runs, matrix.medianBy, (i) => `[${String(++started).padStart(width)}/${total}] ${c.id} · run ${i}/${matrix.runs}`);
     const identity = { id: c.id, name: c.name, device: c.device, ...(c.variant ? { variant: c.variant } : {}), url: c.url };
     combinations.push(
       median.payload

package/src/crux.js

@@ -1,3 +1,4 @@
+import { DEFAULT_THRESHOLDS } from "./baseline.js";
 import { rate } from "./ratings.js";
 
 const DEFAULT_ENDPOINT = "https://chromeuxreport.googleapis.com/v1/records:queryRecord";
@@ -122,12 +123,17 @@ export function trendOf(p75s) {
 
 const BARS = "▁▂▃▄▅▆▇█";
 
-// The series as bars between its lowest and highest values, a space for a period without data
-export function sparkline(values) {
+// The series as bars, a space for a period without data. The bars span the range of the series, or
+// `minRange` when the series is narrower, centred in it: a CLS of 0.20 to 0.21 drawn from its lowest
+// to its highest value looked like a big swing. Eight equal steps, so a flat series sits in the middle.
+export function sparkline(values, minRange = 0) {
   const known = values.filter((v) => v !== null);
   const min = Math.min(...known);
   const max = Math.max(...known);
-  return values.map((v) => (v === null ? " " : max === min ? BARS[3] : BARS[Math.round(((v - min) / (max - min)) * 7)])).join("");
+  const range = Math.max(max - min, minRange);
+  if (range === 0) return values.map((v) => (v === null ? " " : BARS[4])).join("");
+  const low = max - min >= minRange ? min : (min + max) / 2 - range / 2;
+  return values.map((v) => (v === null ? " " : BARS[Math.min(7, Math.floor(((v - low) / range) * 8))])).join("");
 }
 
 const dateOf = ({ year, month, day }) => `${year}-${String(month).padStart(2, "0")}-${String(day).padStart(2, "0")}`;
@@ -202,7 +208,8 @@ export function historyView(history) {
   const rows = METRICS.filter((m) => history.metrics[m.name]).map((m) => {
     const { p75s, first, last, change, trend } = history.metrics[m.name];
     const changeText = change === null ? "·" : `${change > 0 ? "▲ +" : change < 0 ? "▼ −" : "= "}${Math.abs(change)}%`;
-    return { metric: m.name, unit: m.unit, first, last, change: changeText, trend: trend ?? "·", sparkline: sparkline(p75s) };
+    // The smallest change --baseline counts for the metric is the least the bars span
+    return { metric: m.name, unit: m.unit, first, last, change: changeText, trend: trend ?? "·", sparkline: sparkline(p75s, DEFAULT_THRESHOLDS[m.name].absolute) };
   });
   return { title, rows };
 }

package/src/mcp/baseline.js

@@ -0,0 +1,26 @@
+import { loadBaseline, compareRun, parseThreshold, thresholdsWith } from "../baseline.js";
+import { describeRegressions } from "../reporters/baseline-view.js";
+
+// The baseline of run_snippet and run_workflow, as --baseline of the CLI: the runId of an earlier run of
+// the connection, or the path of a page the CLI wrote with --json. The comparison is the one of the CLI
+// (baseline.js), so both give the same verdict for the same two pages.
+
+const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
+
+// Read and checked before the page loads, as the CLI does before the browser opens: a wrong baseline
+// fails at once. Throws an error whose message says what to do.
+export function prepareBaseline({ baseline, thresholds = [] }, runs) {
+  const parsed = thresholdsWith(thresholds.map(parseThreshold));
+  // An id of a run the connection no longer keeps is not a file to look for: runs.page says so
+  if (runs.has(baseline) || UUID.test(baseline)) return { data: runs.page(baseline), source: { runId: baseline }, thresholds: parsed };
+  const { kind, data } = loadBaseline(baseline);
+  if (kind === "matrix") throw new Error(`${baseline} is the baseline of a matrix: this server measures one page; compare it with webperf-snippets --matrix <file> --baseline ${baseline}`);
+  return { data, source: { file: baseline }, thresholds: parsed };
+}
+
+// The page of this run against its baseline: the `baseline` of the JSON of the CLI, named by its runId or
+// its file, with the lines --fail-on-regression would print for it
+export function compareWithBaseline({ data, source, thresholds }, current) {
+  const { file, ...comparison } = compareRun(data, current, { file: source.file, thresholds });
+  return { ...source, ...comparison, regressions: describeRegressions(comparison) };
+}

package/src/mcp/server.js

@@ -1,4 +1,4 @@
-import { McpServer } from "@modelcontextprotocol/server";
+import { McpServer, ResourceTemplate, ResourceNotFoundError } from "@modelcontextprotocol/server";
 import * as z from "zod/v4";
 import { cliVersion } from "../report-to.js";
 import { CATEGORIES, aliasesOf, listSnippets, resolveSnippetName } from "../snippet-names.js";
@@ -16,6 +16,10 @@ import { registerPrompts } from "./prompts.js";
 import { shapeResults, createRuns, RunError } from "./summary.js";
 import { followUpCalls, trackingCall } from "./next-steps.js";
 import { isHttpUrl } from "../url.js";
+import { machineOf } from "../machine.js";
+import { measurementConditions } from "../conditions.js";
+import { prepareBaseline, compareWithBaseline } from "./baseline.js";
+import { MAX_RUNS, measureRuns } from "../runs.js";
 
 const VIEWPORTS = Object.keys(VIEWPORT_PRESETS);
 const STAGES = 3;
@@ -36,8 +40,59 @@ const SnippetResult = z
 const SessionId = z.string().describe("The id open_page returned");
 
 const RunId = z.string().describe("The id of the run, for get_details");
-// Only a summary names its run: whole objects leave nothing for get_details to add
+// Only a summary of read_results names its run: whole objects leave nothing for get_details to add
 const SummaryRunId = RunId.optional().describe("The id of the run, for get_details; only with summaries");
+// A run of a page load is also the baseline of a later one, so it always has its id
+const PageRunId = RunId.describe("The id of the run: the baseline of a later run_snippet or run_workflow; with summaries, also for get_details");
+
+const Baseline = z
+  .string()
+  .min(1)
+  .optional()
+  .describe(
+    "Compare this run with a baseline, as --baseline of the CLI: the runId of an earlier run_snippet or run_workflow of this connection (listed as webperf://runs/{runId}), or the path of a page the CLI wrote with --json, absolute or relative to the working directory of the server"
+  );
+const Thresholds = z
+  .array(z.string())
+  .optional()
+  .describe("With baseline, the threshold of a metric, as --threshold of the CLI: LCP=15% (relative), LCP=200ms (absolute), CLS=0.1. The part not given keeps its default");
+
+const Runs = z
+  .number()
+  .int()
+  .min(1)
+  .max(MAX_RUNS)
+  .default(1)
+  .describe(
+    "Measure the page this many times, one after the other, as --runs of the CLI: the answer is the median run, plus runs with the median, min, max and agreement of each metric. With baseline, 5 or more on both sides give the statistical verdict: use 5 for a comparison, and more only when agreement says noisy. Each run is a full page load and wait, so it takes that much longer"
+  );
+
+const LooseObject = (shape) => z.looseObject(shape).meta({ additionalProperties: true });
+const RunsSummary = LooseObject({
+  count: z.number(),
+  failed: z.number().describe("Runs left out because they failed"),
+  medianRun: z.number().describe("The run the answer comes from, from 1"),
+  medianBy: z.string().describe("The metric that picked it"),
+  summary: z.record(z.string(), LooseObject({ median: z.number() })).describe("By metric: median, min, max, values of each run (null for a failed one), unit, cv and agreement (consistent, variable, noisy)"),
+})
+  .optional()
+  .describe("With several runs, the runs field of the JSON of the CLI");
+const BaselineError = LooseObject({ id: z.string(), message: z.string().optional() });
+const Comparison = z
+  .object({
+    runId: z.string().optional().describe("The run the baseline was, when it was a runId"),
+    file: z.string().optional().describe("The file the baseline was, when it was a path"),
+    url: z.string().describe("The URL the baseline measured"),
+    runs: LooseObject({ count: z.number() }).describe("The runs of the baseline"),
+    metrics: z
+      .record(z.string(), LooseObject({ verdict: z.string() }))
+      .describe("By metric: baseline, current, delta, deltaPercent, threshold, verdict (regressed, improved, unchanged, not measured) and, with five runs a side, stats (p, shift, ci, method)"),
+    errors: z.object({ new: z.array(BaselineError), fixed: z.array(BaselineError) }).describe("The errors only this run has, and the ones only the baseline had"),
+    warnings: z.array(z.string()).describe("What differs in how both were measured: how far to trust the comparison"),
+    regressions: z.array(z.string()).describe("What got worse, one line each: the metrics that regressed and the new errors"),
+  })
+  .optional()
+  .describe("With baseline, the comparison, as the baseline field of the JSON of the CLI");
 
 // auto, the default, returns whole objects when they are small and summaries when they are large: a
 // summary of a small result costs the agent a turn for the details, which is more than it saves
@@ -55,6 +110,7 @@ const NextSteps = z
 
 const DETAIL_NOTE =
   "Large results come as summaries with a runId for get_details; nextSteps, when present, is the call to make next.";
+const BASELINE_NOTE = "With baseline, it also compares the run with an earlier one, as --baseline of the CLI, and says what regressed.";
 
 // A page can put instructions for the agent in what a snippet returns: an alt text, a class in a
 // selector, the URL of a script. The tools that return what a page holds say how to read it, and so
@@ -92,13 +148,28 @@ const toolError = (text) => ({ isError: true, content: [{ type: "text", text }]
 
 const firstLine = (message) => message.split("\n")[0];
 
-// A client that wants progress sends a token with the call; the stages of a run go to it
-function progressFor(ctx) {
+// A client that wants progress sends a token with the call; the stages of each run go to it, numbered
+// across the runs. Returns the reporter of a run, by its number.
+function progressFor(ctx, runs) {
   const progressToken = ctx.mcpReq._meta?.progressToken;
-  return (progress, message) =>
+  const total = STAGES * runs;
+  return (run) => (stage, message) =>
     progressToken === undefined
       ? undefined
-      : ctx.mcpReq.notify({ method: "notifications/progress", params: { progressToken, progress, total: STAGES, message } });
+      : ctx.mcpReq.notify({
+          method: "notifications/progress",
+          params: { progressToken, progress: (run - 1) * STAGES + stage, total, message: runs > 1 ? `Run ${run} of ${runs}: ${message}` : message },
+        });
+}
+
+// A page load measured once, or `count` times with the median run as the answer, as --runs of the CLI.
+// When the client gives up (a cancel, its timeout, or leaving), the runs left never start.
+async function measurePage(measure, count, ctx) {
+  const progress = progressFor(ctx, count);
+  if (count === 1) return measure(progress(1));
+  const { payload, runs, errors } = await measureRuns((i) => measure(progress(i)), count, { isFatal: isMissingBrowser, signal: ctx.mcpReq.signal });
+  if (!payload) throw new Error(`Every run failed: ${[...new Set(errors.map(firstLine))].join("; ")}`);
+  return { ...payload, runs };
 }
 
 const CRUX_FORM_FACTORS = { phone: "PHONE", desktop: "DESKTOP", tablet: "TABLET" };
@@ -116,9 +187,44 @@ function describeSnippet(name) {
   };
 }
 
+// The runs of the connection that measured a page, as resources: an agent picks the baseline of a run
+// among them. Each is the JSON the CLI prints with --json, so saved to a file it is a baseline of the
+// CLI too.
+function registerRuns(server, runs) {
+  const JSON_TYPE = "application/json";
+  server.registerResource(
+    "runs",
+    new ResourceTemplate("webperf://runs/{runId}", {
+      list: async () => ({
+        resources: runs.pages().map(({ runId, url, conditions, at, runs: count }) => ({
+          uri: `webperf://runs/${runId}`,
+          name: `${conditions.snippet ?? conditions.workflow} on ${url} (${conditions.viewport})${count ? `, ${count} runs` : ""}`,
+          description: `Measured at ${at}`,
+          mimeType: JSON_TYPE,
+        })),
+      }),
+    }),
+    {
+      title: "Runs of this connection",
+      description: `The page an earlier run_snippet or run_workflow of this connection measured, with its results and conditions, as the JSON of the CLI (--json). Its runId is the baseline argument of a later run. The server keeps the last 20 runs. ${PAGE_DATA}`,
+      mimeType: JSON_TYPE,
+    },
+    async (uri, { runId }) => {
+      let page;
+      try {
+        page = runs.page(runId);
+      } catch (err) {
+        if (err instanceof RunError) throw new ResourceNotFoundError(uri.href);
+        throw err;
+      }
+      return { contents: [{ uri: uri.href, mimeType: JSON_TYPE, text: JSON.stringify(page) }] };
+    }
+  );
+}
+
 // The MCP server of the CLI: the same snippets and runner, offered as tools an agent calls by name.
 // serveStdio creates one instance per connection, so the pages it keeps open belong to that client.
-export function createServer({ launchBrowser = () => launchChromium(), maxPages, idleMs, env = process.env } = {}) {
+export function createServer({ launchBrowser = () => launchChromium(), maxPages, idleMs, env = process.env, machine = machineOf } = {}) {
   const server = new McpServer({ name: "webperf-snippets", version: cliVersion }, { instructions: INSTRUCTIONS });
   const sessions = createSessions({ launchBrowser, maxPages, idleMs });
   const runs = createRuns();
@@ -126,6 +232,30 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
   server.server.onclose = () => sessions.closeAll();
   registerResources(server);
   registerPrompts(server);
+  registerRuns(server, runs);
+
+  // The machine goes in the conditions of every run, as in the JSON of the CLI, so a baseline measured
+  // on another one warns. It is measured once, with the first run, before its Chromium starts.
+  let machineOfServer;
+  const conditionsOf = (conditions) => measurementConditions({ ...conditions, machine: (machineOfServer ??= machine()) });
+
+  // A run of a page load, kept with its page in the shape of the JSON of the CLI: its runId is the
+  // baseline of a later run, and the resources list it
+  const keepPage = (run, results, conditions) => {
+    const page = { url: run.url, navMs: run.navMs, pageErrors: run.pageErrors, ...(run.throttling ? { throttling: run.throttling } : {}), ...(run.runs ? { runs: run.runs } : {}), conditions };
+    const runId = runs.keep(results, page);
+    server.sendResourceListChanged();
+    return { runId, current: { ...page, results } };
+  };
+
+  // The baseline of a tool, read and checked before the page loads; a wrong one is a tool error
+  const baselineFor = ({ baseline, thresholds }) => {
+    if (baseline === undefined) {
+      if (thresholds?.length) throw new Error("thresholds need a baseline");
+      return undefined;
+    }
+    return prepareBaseline({ baseline, thresholds }, runs);
+  };
 
   // Runs a session tool, and turns the errors the model can act on into tool errors
   const withSession = (handler) => async (args, ctx) => {
@@ -162,28 +292,33 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
     {
       title: "Run a snippet",
       description:
-        `Opens the URL in a headless Chromium, runs one web performance snippet on the loaded page and returns the object the snippet returns, as defined in snippets/SCHEMA.md. Use list_snippets for the names. The snippets that record interactions (INP, Interactions, Input-Latency-Breakdown) run on a page opened with open_page instead. ${DETAIL_NOTE} ${PAGE_DATA}`,
+        `Opens the URL in a headless Chromium, runs one web performance snippet on the loaded page and returns the object the snippet returns, as defined in snippets/SCHEMA.md. Use list_snippets for the names. The snippets that record interactions (INP, Interactions, Input-Latency-Breakdown) run on a page opened with open_page instead. ${BASELINE_NOTE} ${DETAIL_NOTE} ${PAGE_DATA}`,
       inputSchema: z.object({
         url: HttpUrl,
         snippet: z.string().describe("The snippet: its name (LCP-Subparts), an alias (render-blocking) or Category/Name"),
         viewport: z.enum(VIEWPORTS).default("mobile").describe("Device size of the page"),
         waitMs: z.number().int().min(0).max(10000).default(3000).describe("Wait after the load event before running the snippet, in ms"),
+        runs: Runs,
         detail: Detail,
+        baseline: Baseline,
+        thresholds: Thresholds,
       }),
       outputSchema: z.object({
         url: z.string(),
-        runId: SummaryRunId,
+        runId: PageRunId,
         detail: DetailOut,
         snippet: z.string().describe("Category/Name of the snippet that ran"),
         viewport: z.enum(VIEWPORTS),
         navMs: z.number().describe("From the start of the navigation to the end of the wait"),
         pageErrors: z.array(z.string()).describe("Uncaught errors of the page during the run"),
         result: SnippetResult,
+        runs: RunsSummary,
+        baseline: Comparison,
         nextSteps: NextSteps,
       }),
       annotations: { readOnlyHint: true, openWorldHint: true },
     },
-    async ({ url, snippet, viewport, waitMs, detail }, ctx) => {
+    async ({ url, snippet, viewport, waitMs, runs: count, detail, baseline, thresholds }, ctx) => {
       let path;
       try {
         path = resolveSnippetName(snippet);
@@ -194,22 +329,31 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
       if (isTrackingSnippet(path, source)) {
         return toolError(`${path} records interactions: open the page with open_page and this snippet in track, use it with interact, and read it with read_results.`);
       }
+      let base;
+      try {
+        base = baselineFor({ baseline, thresholds });
+      } catch (err) {
+        return toolError(err.message);
+      }
 
       try {
-        const run = await runSnippets({ url, items: [{ id: path, path, source }], waitMs, viewport: VIEWPORT_PRESETS[viewport], onProgress: progressFor(ctx) });
+        const conditions = conditionsOf({ viewport, snippet: path, waitMs });
+        const run = await measurePage((onProgress) => runSnippets({ url, items: [{ id: path, path, source }], waitMs, viewport: VIEWPORT_PRESETS[viewport], onProgress }), count, ctx);
         const shaped = shapeResults(run.results.map(({ id, ...result }) => result), detail);
-        const runId = shaped.detail === "summary" ? runs.keep(run.results) : undefined;
+        const { runId, current } = keepPage(run, run.results, conditions);
         const id = run.results[0].id;
         const nextSteps = followUpCalls({ results: run.results, url, viewport, waitMs, ran: [path] });
         return toolResult({
           url,
-          ...(runId ? { runId } : {}),
+          runId,
           detail: shaped.detail,
           snippet: id,
           viewport,
           navMs: run.navMs,
           pageErrors: run.pageErrors,
           result: shaped.results[0],
+          ...(run.runs ? { runs: run.runs } : {}),
+          ...(base ? { baseline: compareWithBaseline(base, current) } : {}),
           ...(nextSteps.length ? { nextSteps } : {}),
         });
       } catch (err) {
@@ -225,17 +369,20 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
     {
       title: "Run a workflow",
       description:
-        `Opens the URL in a headless Chromium and runs a workflow: a sequence of snippets plus the follow-ups its decision tree adds from the results (for example LCP-Subparts when LCP is over 2.5 s, with the reason). core-web-vitals: LCP and CLS. audit: render-blocking resources, resource hints, critical CSS, TTFB, scripts, images and more. loading: TTFB, FCP, render-blocking, scripts and fonts. interaction: long tasks, animation frames, scroll and layout shifts while it scrolls the page. ${DETAIL_NOTE} ${PAGE_DATA}`,
+        `Opens the URL in a headless Chromium and runs a workflow: a sequence of snippets plus the follow-ups its decision tree adds from the results (for example LCP-Subparts when LCP is over 2.5 s, with the reason). core-web-vitals: LCP and CLS. audit: render-blocking resources, resource hints, critical CSS, TTFB, scripts, images and more. loading: TTFB, FCP, render-blocking, scripts and fonts. interaction: long tasks, animation frames, scroll and layout shifts while it scrolls the page. ${BASELINE_NOTE} ${DETAIL_NOTE} ${PAGE_DATA}`,
       inputSchema: z.object({
         url: HttpUrl,
         workflow: z.enum(Object.keys(WORKFLOWS)).default("core-web-vitals"),
         viewport: z.enum(VIEWPORTS).default("mobile").describe("Device size of the page"),
         waitMs: z.number().int().min(0).max(10000).default(3000).describe("Wait after the load event before running the workflow, in ms"),
+        runs: Runs,
         detail: Detail,
+        baseline: Baseline,
+        thresholds: Thresholds,
       }),
       outputSchema: z.object({
         url: z.string(),
-        runId: SummaryRunId,
+        runId: PageRunId,
         detail: DetailOut,
         workflow: z.string(),
         viewport: z.enum(VIEWPORTS),
@@ -244,20 +391,26 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
         results: z
           .array(z.looseObject({ id: z.string(), status: z.string() }).meta({ additionalProperties: true }))
           .describe("One object per step, in order, then the follow-ups, each with the reason it was added"),
+        runs: RunsSummary,
+        baseline: Comparison,
         nextSteps: NextSteps,
       }),
       annotations: { readOnlyHint: true, openWorldHint: true },
     },
-    async ({ url, workflow, viewport, waitMs, detail }, ctx) => {
+    async ({ url, workflow, viewport, waitMs, runs: count, detail, baseline, thresholds }, ctx) => {
+      let base;
       try {
-        const run = await runMeasurement({
-          url,
-          workflow: WORKFLOWS[workflow],
-          rules: RULES,
-          waitMs,
-          viewport: VIEWPORT_PRESETS[viewport],
-          onProgress: progressFor(ctx),
-        });
+        base = baselineFor({ baseline, thresholds });
+      } catch (err) {
+        return toolError(err.message);
+      }
+      try {
+        const conditions = conditionsOf({ viewport, workflow, waitMs });
+        const run = await measurePage(
+          (onProgress) => runMeasurement({ url, workflow: WORKFLOWS[workflow], rules: RULES, waitMs, viewport: VIEWPORT_PRESETS[viewport], onProgress }),
+          count,
+          ctx
+        );
         // The runner explains a skipped step with the flag of the CLI; here the tools are what the model has
         const results = run.results.map((r) =>
           r.status === "skipped"
@@ -265,7 +418,7 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
             : r
         );
         const shaped = shapeResults(results, detail);
-        const runId = shaped.detail === "summary" ? runs.keep(results) : undefined;
+        const { runId, current } = keepPage(run, results, conditions);
         // The runner already ran the follow-ups of the decision tree; what is left is the steps it skipped
         const skipped = results
           .filter((r) => r.status === "skipped")
@@ -274,12 +427,14 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
         const nextSteps = skipped.length ? [trackingCall({ paths: skipped, url, viewport })] : [];
         return toolResult({
           url,
-          ...(runId ? { runId } : {}),
+          runId,
           ...shaped,
           workflow,
           viewport,
           navMs: run.navMs,
           pageErrors: run.pageErrors,
+          ...(run.runs ? { runs: run.runs } : {}),
+          ...(base ? { baseline: compareWithBaseline(base, current) } : {}),
           ...(nextSteps.length ? { nextSteps } : {}),
         });
       } catch (err) {
@@ -463,7 +618,12 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
         runId: RunId,
         snippet: z.string().describe("The id of the result in the summary, or its script"),
       }),
-      outputSchema: z.object({ runId: z.string(), snippet: z.string().describe("The id of the result in the run"), result: SnippetResult }),
+      outputSchema: z.object({
+        runId: z.string(),
+        snippet: z.string().describe("The id of the result in the run"),
+        medianRun: z.number().optional().describe("With several runs, the one this result comes from: the median run, the only one kept"),
+        result: SnippetResult,
+      }),
       annotations: { readOnlyHint: true, openWorldHint: false },
     },
     async ({ runId, snippet }) => {
@@ -482,7 +642,8 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
           if (!(err instanceof RunError) || !resolved || resolved === snippet) throw err;
           result = runs.get(runId, resolved);
         }
-        return toolResult({ runId, snippet: result.id, result });
+        const medianRun = runs.medianRun(runId);
+        return toolResult({ runId, snippet: result.id, ...(medianRun ? { medianRun } : {}), result });
       } catch (err) {
         if (err instanceof RunError) return toolError(err.message);
         throw err;

package/src/mcp/summary.js

@@ -29,27 +29,49 @@ export function shapeResults(results, detail = "auto", maxFullChars = FULL_UP_TO
 }
 
 // The full results of the last runs of an MCP connection, by run id, so get_details answers with the
-// values the summary came from: measuring again would give other ones
+// values the summary came from: measuring again would give other ones. A run of a page load also keeps
+// its page (url, navMs, pageErrors, conditions), so a later run can take it as its baseline.
 export function createRuns({ max = 20 } = {}) {
   const runs = new Map();
+  const unknown = (runId) =>
+    new RunError(`Unknown runId ${runId}: this server keeps the last ${max} runs of the connection. Run the tool again for a new one.`);
 
-  function keep(results) {
+  function keep(results, page) {
     const runId = randomUUID();
-    runs.set(runId, results);
+    runs.set(runId, { results, page, at: new Date().toISOString() });
     // A Map keeps the order of insertion, so the first key is the oldest run
     if (runs.size > max) runs.delete(runs.keys().next().value);
     return runId;
   }
 
   function get(runId, snippet) {
-    const results = runs.get(runId);
-    if (!results) {
-      throw new RunError(`Unknown runId ${runId}: this server keeps the last ${max} runs of the connection. Run the tool again for a new one.`);
-    }
+    const run = runs.get(runId);
+    if (!run) throw unknown(runId);
+    const { results } = run;
     const result = results.find((r) => r.id === snippet) ?? results.find((r) => r.script === snippet);
     if (!result) throw new RunError(`${snippet} is not in run ${runId}, which has: ${results.map((r) => r.id).join(", ")}.`);
     return result;
   }
 
-  return { keep, get };
+  // A run as the JSON of the CLI gives a page, which is what a baseline is
+  function page(runId) {
+    const run = runs.get(runId);
+    if (!run) throw unknown(runId);
+    if (!run.page) throw new RunError(`Run ${runId} read an open page; a baseline is a run of run_snippet or run_workflow.`);
+    return { ...run.page, results: run.results };
+  }
+
+  // The runs that can be a baseline, the oldest first, with the number of measurements of each when it
+  // has several: the statistics of a comparison need five a side
+  const pages = () =>
+    [...runs]
+      .filter(([, run]) => run.page)
+      .map(([runId, { page, at }]) => ({ runId, url: page.url, conditions: page.conditions, at, ...(page.runs ? { runs: page.runs.count } : {}) }));
+
+  // A measurement of several runs keeps the results of the median one, as the JSON of the CLI does
+  const medianRun = (runId) => runs.get(runId)?.page?.runs?.medianRun;
+
+  const has = (runId) => runs.has(runId);
+
+  return { keep, get, page, pages, medianRun, has };
 }

package/src/runs.js

@@ -4,6 +4,11 @@ import { DEFAULT_THRESHOLDS } from "./baseline.js";
 // Every result of the report comes from that run, so they agree with each other (the sub-parts of
 // the LCP add up to its LCP), and a summary gives the median, min and max of each metric.
 
+// The most runs of a measurement, the same in the tools of the MCP server as in --runs. Mann-Whitney U
+// needs five a side (baseline.js), so nine leave room for failed runs and for a noisy page, and a page
+// that takes a few seconds to load stays under two minutes.
+export const MAX_RUNS = 9;
+
 // The metrics the snippets report, and navMs, which every run has
 export const MEDIAN_METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP", "navMs"];
 
@@ -50,6 +55,23 @@ export function describeRuns({ count, failed, medianRun, medianBy, medianByReque
   return `${count} run${count === 1 ? "" : "s"}, median by ${by}: run ${medianRun}${failed ? ` (${failed} failed)` : ""}`;
 }
 
+// Measures `count` runs one after the other, never in parallel: they would compete for the CPU and
+// skew the timings. A run that fails is kept as { error } and left out of the median, unless isFatal
+// says the next run would fail the same way. An aborted signal, a caller that left, starts no more runs.
+export async function measureRuns(measure, count, { medianBy, isFatal = () => false, signal } = {}) {
+  const runs = [];
+  for (let i = 1; i <= count; i++) {
+    signal?.throwIfAborted();
+    try {
+      runs.push(await measure(i));
+    } catch (err) {
+      if (isFatal(err)) throw err;
+      runs.push({ error: err.message });
+    }
+  }
+  return summarizeRuns(runs, { medianBy });
+}
+
 // The runs as the CLI made them, in order, a run that failed being { error }. Returns the payload of
 // the median run and the `runs` field of the report, or nulls and the errors when every run failed.
 export function summarizeRuns(runs, { medianBy: requested } = {}) {