webperf-snippets 1.7.1 → 1.9.0

package/README.md

@@ -70,10 +70,10 @@ JSON output (for piping into `jq` or CI):
 npx webperf-snippets https://example.com --json
 ```
 
-The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait and the version of the CLI.
+The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait, the version of the CLI and the machine: its CPU model, cores, architecture and memory, as the operating system reports them, and a benchmark index, a short CPU test in Node before Chromium starts, which only informs, since on the same machine it moves by up to 40 % with the load.
 
 ```json
-"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.7.1" }
+"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.9.0", "machine": { "cpu": "Apple M4", "cores": 10, "arch": "arm64", "memoryGB": 16, "benchmarkIndex": 2371 } }
 ```
 
 Single snippet:
@@ -158,7 +158,7 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.7.1" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.9.0" }
 }
 ```
 
@@ -190,7 +190,7 @@ One page load is one sample, and lab values vary from one load to the next. `--r
 npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
 ```
 
-The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and the JSON output carries them in `runs`:
+The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and how much the runs agree, and the JSON output carries them in `runs`:
 
 ```json
 "runs": {
@@ -199,12 +199,12 @@ The median run is the one with the median LCP. A workflow without LCP uses the f
   "medianRun": 3,
   "medianBy": "LCP",
   "summary": {
-    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms" }
+    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms", "cv": 11.6, "agreement": "noisy" }
   }
 }
 ```
 
-`medianRun` counts from 1, and `values` keeps the order of the runs. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
+`medianRun` counts from 1, and `values` keeps the order of the runs. `cv` is the coefficient of variation of the runs, their standard deviation over their mean in percent, and `agreement` its band: `consistent` up to 5 %, `variable` up to 10 % and `noisy` above, where one median says little and more runs help. Runs that vary by less than half of the minimum change of `--baseline` (50 ms for LCP, FCP and TTFB, 25 ms for INP, 0.025 for CLS) are `consistent` whatever their CV, since a local TTFB of 3 to 18 ms has a CV of 64 %. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
 
 Measure several pages, on several devices and in several variants, in one command with `--matrix <file>`. The file lists them, and the CLI measures every URL on every device and in every variant, each combination `runs` times, one load after the other:
 
@@ -238,7 +238,7 @@ npx webperf-snippets --matrix matrix.json --budget-lcp 2500
 
 The options of the command line apply to every combination: `--workflow`, `--snippet`, `--wait`, `--interact-script`, `--header`, `--own-domains`, `--storage-state`, `--cpu`, `--network` and the budgets. The file replaces `cpu`, `network` and `storageState` where it sets them, and its `headers` and `ownDomains` add to the ones of the command line, the variant winning on the same header. A matrix file is meant to be versioned, so pass secrets, such as the token of an access proxy, with `--header` and an environment variable; a header of the file whose name looks like a secret raises a warning. A key the CLI does not know is an error, so a typo does not measure without its condition.
 
-The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
+The report is a table with one row per combination and the rated median of each metric, a variable or noisy one followed by the coefficient of variation of its runs (`🟡 2.71s ±14%`), then the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
 
 With CrUX, the CLI asks once for each URL and form factor, so the variants of a device share their field data. With `--report-to`, it posts one report for each combination, the body of one page with `meta.device`, `meta.variant` and `meta.runs`; the headers of the command line and of the file are conditions of the load, and no report carries their values.
 
@@ -257,6 +257,8 @@ npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.jso
 
 LCP, FCP, TTFB, CLS and INP compare when both runs measured them, the median of each with `--runs`; a metric that only one side measured is `not measured`. A metric regresses when it gets worse by more than its threshold, the larger of a share of the baseline and a minimum, and, when the baseline has two runs or more, ends above the slowest of them: a median inside the range of the baseline is noise the baseline already had. It improves under the same rules the other way, and otherwise it is unchanged.
 
+With five runs or more on each side, the range gives way to a test of the runs themselves: a change that beats its threshold also needs a one-sided Mann-Whitney U p-value under 0.05, the chance that runs of an unchanged page would differ this much. A median can move with two slow runs out of five; the test asks whether most runs moved. The JSON of the metric adds `stats`, with `p`, the `shift` (the Hodges-Lehmann estimate, the median of the differences between every run of one side and every run of the other) and its 95 % interval `ci`, and the reports add them to the verdict: `regressed (p = 0.028, +500ms, 95% CI +20ms to +600ms)`, or `unchanged (no evidence: p = 0.35)` for a change over the threshold that the runs do not back. Three runs a side can never reach p < 0.05, so under five the range decides.
+
 | Metric         | Share of the baseline | Minimum |
 | -------------- | --------------------- | ------- |
 | LCP, FCP, TTFB | 10%                   | 100 ms  |
@@ -282,7 +284,7 @@ The report adds the comparison after the usual one, and `--markdown` a table for
 }
 ```
 
-When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
+When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, on another CPU model or with other cores, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
 
 The comparison only informs, and the exit code is the same as without it, unless `--fail-on-regression`, which exits with `1` when a metric regressed or an error is new, and names them on stderr. A baseline that does not exist or is not an output of the CLI, a baseline of a page for a matrix or of a matrix for a page, or a `--threshold` the CLI does not understand exits with code `2`, before the page loads.
 
@@ -382,12 +384,13 @@ It needs the Chromium of Playwright (`npx playwright install chromium`); without
 | Tool            | What it does                                                                 |
 | --------------- | ---------------------------------------------------------------------------- |
 | `list_snippets` | Lists the snippets with their category, title, short aliases and whether they track interactions. Takes an optional `category`. |
-| `run_snippet`   | Opens a URL in a headless Chromium, runs one snippet and returns the object it returns. Takes `url`, `snippet` (name, alias or `Category/Name`), and optional `viewport` and `waitMs`. The snippets that record interactions run on a page opened with `open_page`. |
+| `run_snippet`   | Opens a URL in a headless Chromium, runs one snippet and returns a summary of its result. Takes `url`, `snippet` (name, alias or `Category/Name`), and optional `viewport` and `waitMs`. The snippets that record interactions run on a page opened with `open_page`. |
 | `run_workflow`  | Runs a workflow of the CLI (`core-web-vitals`, `audit`, `loading`, `interaction`) and the follow-ups of its decision tree. Takes `url`, `workflow`, and optional `viewport` and `waitMs`. |
 | `get_crux`      | Returns the CrUX p75 of LCP, CLS and INP for a URL, or its origin. Takes `url` and an optional `formFactor` (`phone`, `desktop`, `tablet`). Needs `CRUX_API_KEY` in the environment of the server. |
 | `open_page`     | Opens a URL in a page that stays open between calls and installs tracking snippets (`track`, such as `INP`). Returns a `sessionId`. |
 | `interact`      | Runs steps on an open page: `click`, `type`, `hover`, `scroll` or `wait`. |
 | `read_results`  | Returns what the tracking snippets of an open page have recorded. |
+| `get_details`   | Returns the whole object of one snippet of a run, by the `runId` that `run_snippet`, `run_workflow` or `read_results` returned, without measuring again. Takes `runId` and `snippet`. |
 | `close_page`    | Closes an open page. |
 
 The server also offers resources, documents a client can attach to the context:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.7.1",
+  "version": "1.9.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",

package/src/baseline.js

@@ -1,6 +1,7 @@
 import { readFileSync } from "node:fs";
 import { blockingErrors } from "./blocking.js";
 import { describeThrottling } from "./throttling.js";
+import { mannWhitneyGreater, hodgesLehmann } from "./stats.js";
 
 // A measurement compared with an earlier one, its baseline: which metrics got worse or better beyond
 // the noise of the lab, which errors appeared or went away, and what differs in how both were
@@ -71,23 +72,40 @@ export function thresholdsWith(overrides) {
 
 const round = (n) => Math.round(n * 10000) / 10000;
 
-// The value of a metric in a page: the median of its runs, with their range when there are two or
-// more, or its one result, as runs.js picks it
+// The value of a metric in a page: the median of its runs, with their range and their values when there
+// are two or more, or its one result, as runs.js picks it
 function measured(payload, metric) {
   const summary = payload.runs?.summary?.[metric];
   if (summary) {
-    const runs = summary.values.filter((value) => value !== null).length;
-    return { value: summary.median, ...(runs >= 2 ? { range: { min: summary.min, max: summary.max } } : {}) };
+    const values = summary.values.filter((value) => value !== null);
+    return { value: summary.median, ...(values.length >= 2 ? { range: { min: summary.min, max: summary.max }, values } : {}) };
   }
   const result = payload.results?.find((r) => r.metric === metric && r.status === "ok" && typeof r.value === "number");
   return result ? { value: result.value } : null;
 }
 
-// A regression must beat the threshold of its metric and, when the baseline has several runs, the
-// worst of them: a median inside the range of the baseline is noise the baseline already had.
-function verdict(base, current, threshold) {
-  if (current.value - base.value > threshold && (!base.range || current.value > base.range.max)) return "regressed";
-  if (base.value - current.value > threshold && (!base.range || current.value < base.range.min)) return "improved";
+// With five runs or more a side, Mann-Whitney U can tell a change from the noise of the runs; with
+// fewer, its p-value cannot go under 0.05 (three a side give 0.05 at best), so the range decides.
+export const RUNS_FOR_STATISTICS = 5;
+const ALPHA = 0.05;
+
+// The evidence of a change: the one-sided p-value in its direction, and the shift with its 95 % interval
+function statsOf(base, current, delta) {
+  if ((base.values?.length ?? 0) < RUNS_FOR_STATISTICS || (current.values?.length ?? 0) < RUNS_FOR_STATISTICS) return undefined;
+  const { p, method } = delta < 0 ? mannWhitneyGreater(base.values, current.values) : mannWhitneyGreater(current.values, base.values);
+  const { shift, ci } = hodgesLehmann(base.values, current.values);
+  return { p: round(p), shift: round(shift), ci: ci.map(round), method };
+}
+
+// A regression must beat the threshold of its metric and, with five runs a side, be unlikely to come
+// from the noise of the runs (p < 0.05). With fewer, when the baseline has several runs, it must beat
+// the worst of them: a median inside the range of the baseline is noise the baseline already had.
+function verdict(base, current, threshold, stats) {
+  const worse = current.value - base.value > threshold;
+  const better = base.value - current.value > threshold;
+  if (stats) return stats.p < ALPHA ? (worse ? "regressed" : better ? "improved" : "unchanged") : "unchanged";
+  if (worse && (!base.range || current.value > base.range.max)) return "regressed";
+  if (better && (!base.range || current.value < base.range.min)) return "improved";
   return "unchanged";
 }
 
@@ -104,13 +122,15 @@ function compareMetrics(base, current, thresholds) {
     const { relative = 0, absolute } = thresholds[metric];
     const threshold = round(Math.max(relative * b.value, absolute));
     const delta = c.value - b.value;
+    const stats = statsOf(b, c, delta);
     metrics[metric] = {
       baseline: b.value,
       current: c.value,
       delta: round(delta),
       deltaPercent: b.value === 0 ? null : Math.round((delta / b.value) * 1000) / 10,
       threshold,
-      verdict: verdict(b, c, threshold),
+      verdict: verdict(b, c, threshold, stats),
+      ...(stats ? { stats } : {}),
     };
   }
   return metrics;
@@ -166,6 +186,14 @@ const pathOf = (url) => {
   }
 };
 
+// Another CPU or other cores give other timings for the same page. A different benchmark index alone
+// does not warn: on the same machine it moves by 40 % with its load.
+function machineWarning(base, current) {
+  if (!base?.cpu || !current?.cpu) return null;
+  if (base.cpu === current.cpu && base.cores === current.cores) return null;
+  return `the baseline was measured on another machine (${base.cpu}, ${base.cores} cores) than this run (${current.cpu}, ${current.cores} cores); timings can differ for that alone`;
+}
+
 // What differs in how both were measured. The comparison goes on: the warnings say how far to trust it.
 function warningsOf(base, current) {
   const warnings = [];
@@ -183,6 +211,8 @@ function warningsOf(base, current) {
   }
   if (runCount(base) < 2) warnings.push("the baseline has 1 run, so only the thresholds apply; measure both with --runs");
   else if (runCount(base) !== runCount(current)) warnings.push(`the baseline has ${runCount(base)} runs, and this run ${runCount(current)}`);
+  const machine = machineWarning(b?.machine, c?.machine);
+  if (machine) warnings.push(machine);
   if (b && c && b.cliVersion !== c.cliVersion) {
     warnings.push(`the baseline was measured with CLI ${b.cliVersion}, and this run with ${c.cliVersion}; the snippets may have changed`);
   }

package/src/bin.js

@@ -20,6 +20,7 @@ import { loadMatrix } from "./matrix.js";
 import { blockingErrors } from "./blocking.js";
 import { MEASURED_BY, checkBudgets, parseBudgets, unmeasuredMetrics, violationsOf } from "./budgets.js";
 import { measurementConditions } from "./conditions.js";
+import { machineOf } from "./machine.js";
 import { isHttpUrl } from "./url.js";
 import { loadBaseline, parseThreshold, thresholdsWith, compareRun, compareMatrix } from "./baseline.js";
 import { describeRegressions, describeMatrixRegressions, matrixComparisonView } from "./reporters/baseline-view.js";
@@ -287,9 +288,11 @@ async function main() {
     const options = { waitMs, headless: !values.headed, interactScript, ...conditions };
     return items ? runSnippets({ ...options, items }) : runMeasurement({ ...options, workflow, rules: RULES });
   };
-  // What the JSON records of how each page was measured, for a later run to compare with
+  // What the JSON records of how each page was measured, for a later run to compare with. The machine is
+  // measured once, before Chromium starts, so its processes do not slow the benchmark down.
+  const machine = machineOf();
   const conditionsFor = (viewport) =>
-    measurementConditions({ viewport, waitMs, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
+    measurementConditions({ viewport, waitMs, machine, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
 
   if (matrixPath) {
     let matrix;

package/src/conditions.js

@@ -2,6 +2,6 @@ import { cliVersion } from "./report-to.js";
 
 // How a measurement was taken, kept in its JSON so that a later run compared with it can tell whether
 // both were measured the same way. A snippet is named by its path, so an alias and its path match.
-export function measurementConditions({ viewport, workflow, snippet, waitMs }) {
-  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion };
+export function measurementConditions({ viewport, workflow, snippet, waitMs, machine }) {
+  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion, ...(machine ? { machine } : {}) };
 }

package/src/machine.js

@@ -0,0 +1,42 @@
+import { arch as osArch, cpus as osCpus, totalmem as osTotalmem } from "node:os";
+
+// The machine a measurement ran on, kept in its conditions, so --baseline can say when two were measured
+// on different machines: timings move with the CPU as much as with the page. The machine is told apart by
+// what the operating system reports of it. The benchmark index only informs: on the same machine it
+// moves by 40 % with its load and with the cores the system picks (performance or efficiency).
+
+const WINDOW_MS = 40;
+const WINDOWS = 5;
+
+// A fixed piece of work: build a string and sort an array, as the benchmark index of Lighthouse builds
+// strings in the page. Its result is returned so the engine cannot drop it.
+function work() {
+  let s = "";
+  for (let i = 0; i < 2000; i++) s += String.fromCharCode(97 + (i % 26));
+  const numbers = Array.from({ length: 200 }, (_, i) => (i * 7919) % 211);
+  numbers.sort((a, b) => a - b);
+  return s.length + numbers[0];
+}
+
+// How many times the work fits in 40 ms, the best of five windows: the first one also warms up the
+// engine, and the best one is the least touched by other work of the machine. In Node, so --cpu, which
+// throttles the page, leaves it alone: it measures the machine. Alone, ten in a row vary by about 6 %.
+export function benchmarkIndex() {
+  let best = 0;
+  let sink = 0;
+  for (let w = 0; w < WINDOWS; w++) {
+    const end = performance.now() + WINDOW_MS;
+    let count = 0;
+    while (performance.now() < end) {
+      sink += work();
+      count++;
+    }
+    best = Math.max(best, count);
+  }
+  return sink > 0 ? best : 0;
+}
+
+export function machineOf({ cpus = osCpus, arch = osArch(), totalmem = osTotalmem, benchmark = benchmarkIndex } = {}) {
+  const list = cpus();
+  return { cpu: list[0]?.model ?? "unknown", cores: list.length, arch, memoryGB: Math.round(totalmem() / 2 ** 30), benchmarkIndex: benchmark() };
+}

package/src/mcp/next-steps.js

@@ -0,0 +1,29 @@
+import { RULES, nextSteps } from "../decision-tree.js";
+import { loadSnippet } from "../load-snippet.js";
+import { isTrackingSnippet } from "../tracking.js";
+
+const isTrackingPath = (path) => isTrackingSnippet(path, loadSnippet(path));
+
+// The call of the tool that runs a snippet: a tracking snippet needs a page the agent interacts with. A
+// snippet waits as long as the run that called for it, so it reads the same paint.
+const callFor = ({ path, reason, url, viewport, waitMs, isTracking }) =>
+  isTracking(path)
+    ? { tool: "open_page", args: { url, track: [path], viewport }, reason }
+    : { tool: "run_snippet", args: { url, snippet: path, viewport, ...(waitMs !== undefined ? { waitMs } : {}) }, reason };
+
+// The follow-ups the decision tree of the CLI asks for, as the calls an agent makes next. The rules name
+// the steps of a workflow (LCP, CLS, TTFB, FCP), which are the scripts of the snippets, so a result of
+// run_snippet, whose id is Category/Name, is matched by its script. `ran` leaves out what already ran.
+export function followUpCalls({ results, url, viewport, waitMs, ran = [], rules = RULES, isTracking = isTrackingPath }) {
+  const named = results.map((r) => ({ ...r, id: r.script ?? r.id }));
+  return nextSteps(named, rules, ran).map(({ path, reason }) => callFor({ path, reason, url, viewport, waitMs, isTracking }));
+}
+
+// The steps a workflow skipped because they measure clicks and key presses, in one page to open
+export function trackingCall({ paths, url, viewport }) {
+  return {
+    tool: "open_page",
+    args: { url, track: paths, viewport },
+    reason: "These steps measure clicks and key presses: open the page with them, interact with it and read them with read_results",
+  };
+}

package/src/mcp/prompts.js

@@ -31,7 +31,8 @@ export function registerPrompts(server) {
             `1. Call run_workflow with url "${url}", workflow "core-web-vitals" and viewport "${viewport}". It returns LCP and CLS, and adds LCP-Subparts with the reason when LCP is over 2.5 s.`,
             `2. Call get_crux for the same URL with formFactor "${viewport === "desktop" ? "desktop" : "phone"}", to compare the lab with what real users get. If it answers that there is no key, say so and go on with the lab data.`,
             '3. If LCP-Subparts did not run, call run_snippet with snippet "LCP-Subparts".',
-            "4. Report the LCP value and rating in the lab and in the field, the LCP element, the slowest sub-part and what it means, and the fixes for that sub-part, the most effective first.",
+            "4. If the answers are summaries (detail: summary), call get_details with the runId for LCP and LCP-Subparts: a summary leaves out the LCP element and the sub-parts.",
+            "5. Report the LCP value and rating in the lab and in the field, the LCP element, the slowest sub-part and what it means, and the fixes for that sub-part, the most effective first.",
             "",
             `${QUOTE} The result of each snippet is described below.`,
           ].join("\n")
@@ -63,7 +64,7 @@ export function registerPrompts(server) {
             selector
               ? `2. Call interact with a click on "${selector}". Add a few more clicks or key presses if the page has other controls a user would use.`
               : "2. Call interact with clicks on the controls a user would use first: main buttons, links that open menus, form fields. If you cannot tell which ones, ask me.",
-            "3. Call read_results. INP gives the worst interaction and its phases (input delay, processing, presentation); Input-Latency-Breakdown says which phase dominates for each type of event.",
+            "3. Call read_results; if it answers with summaries, call get_details with its runId for INP and Input-Latency-Breakdown. INP gives the worst interaction and its phases (input delay, processing, presentation); Input-Latency-Breakdown says which phase dominates for each type of event.",
             "4. Call close_page.",
             "5. Report the worst interaction, its target, the dominant phase and the fixes for that phase.",
             "",

package/src/mcp/server.js

@@ -13,6 +13,8 @@ import { createSessions, SessionError } from "./sessions.js";
 import { launchChromium, isMissingBrowser, MISSING_BROWSER } from "../browser.js";
 import { registerResources } from "./documents.js";
 import { registerPrompts } from "./prompts.js";
+import { shapeResults, createRuns, RunError } from "./summary.js";
+import { followUpCalls, trackingCall } from "./next-steps.js";
 import { isHttpUrl } from "../url.js";
 
 const VIEWPORTS = Object.keys(VIEWPORT_PRESETS);
@@ -33,6 +35,27 @@ const SnippetResult = z
 
 const SessionId = z.string().describe("The id open_page returned");
 
+const RunId = z.string().describe("The id of the run, for get_details");
+// Only a summary names its run: whole objects leave nothing for get_details to add
+const SummaryRunId = RunId.optional().describe("The id of the run, for get_details; only with summaries");
+
+// auto, the default, returns whole objects when they are small and summaries when they are large: a
+// summary of a small result costs the agent a turn for the details, which is more than it saves
+const Detail = z
+  .enum(["auto", "summary", "full"])
+  .default("auto")
+  .describe("auto: whole objects when small, summaries when large");
+const DetailOut = z.enum(["summary", "full"]).describe("Whether the results are summaries or the whole objects");
+
+// The call an agent makes next, with the arguments it takes as they are
+const NextSteps = z
+  .array(z.object({ tool: z.string(), args: z.record(z.string(), z.unknown()), reason: z.string() }))
+  .optional()
+  .describe("The calls to make next, with their arguments and why, when the results call for them");
+
+const DETAIL_NOTE =
+  "Large results come as summaries with a runId for get_details; nextSteps, when present, is the call to make next.";
+
 // A page can put instructions for the agent in what a snippet returns: an alt text, a class in a
 // selector, the URL of a script. The tools that return what a page holds say how to read it, and so
 // do the instructions of the server, which a client gives the model before any tool.
@@ -98,6 +121,7 @@ function describeSnippet(name) {
 export function createServer({ launchBrowser = () => launchChromium(), maxPages, idleMs, env = process.env } = {}) {
   const server = new McpServer({ name: "webperf-snippets", version: cliVersion }, { instructions: INSTRUCTIONS });
   const sessions = createSessions({ launchBrowser, maxPages, idleMs });
+  const runs = createRuns();
   // The client left (end of stdin, or a closed transport): close the pages and Chromium
   server.server.onclose = () => sessions.closeAll();
   registerResources(server);
@@ -138,24 +162,28 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
     {
       title: "Run a snippet",
       description:
-        `Opens the URL in a headless Chromium, runs one web performance snippet on the loaded page and returns the object the snippet returns, as defined in snippets/SCHEMA.md. Use list_snippets for the names. The snippets that record interactions (INP, Interactions, Input-Latency-Breakdown) run on a page opened with open_page instead. ${PAGE_DATA}`,
+        `Opens the URL in a headless Chromium, runs one web performance snippet on the loaded page and returns the object the snippet returns, as defined in snippets/SCHEMA.md. Use list_snippets for the names. The snippets that record interactions (INP, Interactions, Input-Latency-Breakdown) run on a page opened with open_page instead. ${DETAIL_NOTE} ${PAGE_DATA}`,
       inputSchema: z.object({
         url: HttpUrl,
         snippet: z.string().describe("The snippet: its name (LCP-Subparts), an alias (render-blocking) or Category/Name"),
         viewport: z.enum(VIEWPORTS).default("mobile").describe("Device size of the page"),
         waitMs: z.number().int().min(0).max(10000).default(3000).describe("Wait after the load event before running the snippet, in ms"),
+        detail: Detail,
       }),
       outputSchema: z.object({
         url: z.string(),
+        runId: SummaryRunId,
+        detail: DetailOut,
         snippet: z.string().describe("Category/Name of the snippet that ran"),
         viewport: z.enum(VIEWPORTS),
         navMs: z.number().describe("From the start of the navigation to the end of the wait"),
         pageErrors: z.array(z.string()).describe("Uncaught errors of the page during the run"),
         result: SnippetResult,
+        nextSteps: NextSteps,
       }),
       annotations: { readOnlyHint: true, openWorldHint: true },
     },
-    async ({ url, snippet, viewport, waitMs }, ctx) => {
+    async ({ url, snippet, viewport, waitMs, detail }, ctx) => {
       let path;
       try {
         path = resolveSnippetName(snippet);
@@ -169,8 +197,21 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
 
       try {
         const run = await runSnippets({ url, items: [{ id: path, path, source }], waitMs, viewport: VIEWPORT_PRESETS[viewport], onProgress: progressFor(ctx) });
-        const { id, ...result } = run.results[0];
-        return toolResult({ url, snippet: id, viewport, navMs: run.navMs, pageErrors: run.pageErrors, result });
+        const shaped = shapeResults(run.results.map(({ id, ...result }) => result), detail);
+        const runId = shaped.detail === "summary" ? runs.keep(run.results) : undefined;
+        const id = run.results[0].id;
+        const nextSteps = followUpCalls({ results: run.results, url, viewport, waitMs, ran: [path] });
+        return toolResult({
+          url,
+          ...(runId ? { runId } : {}),
+          detail: shaped.detail,
+          snippet: id,
+          viewport,
+          navMs: run.navMs,
+          pageErrors: run.pageErrors,
+          result: shaped.results[0],
+          ...(nextSteps.length ? { nextSteps } : {}),
+        });
       } catch (err) {
         if (isMissingBrowser(err)) return toolError(MISSING_BROWSER);
         if (/page\.goto/.test(err.message)) return toolError(`Could not load ${url}: ${firstLine(err.message)}`);
@@ -184,15 +225,18 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
     {
       title: "Run a workflow",
       description:
-        `Opens the URL in a headless Chromium and runs a workflow: a sequence of snippets plus the follow-ups its decision tree adds from the results (for example LCP-Subparts when LCP is over 2.5 s, with the reason). core-web-vitals: LCP and CLS. audit: render-blocking resources, resource hints, critical CSS, TTFB, scripts, images and more. loading: TTFB, FCP, render-blocking, scripts and fonts. interaction: long tasks, animation frames, scroll and layout shifts while it scrolls the page. ${PAGE_DATA}`,
+        `Opens the URL in a headless Chromium and runs a workflow: a sequence of snippets plus the follow-ups its decision tree adds from the results (for example LCP-Subparts when LCP is over 2.5 s, with the reason). core-web-vitals: LCP and CLS. audit: render-blocking resources, resource hints, critical CSS, TTFB, scripts, images and more. loading: TTFB, FCP, render-blocking, scripts and fonts. interaction: long tasks, animation frames, scroll and layout shifts while it scrolls the page. ${DETAIL_NOTE} ${PAGE_DATA}`,
       inputSchema: z.object({
         url: HttpUrl,
         workflow: z.enum(Object.keys(WORKFLOWS)).default("core-web-vitals"),
         viewport: z.enum(VIEWPORTS).default("mobile").describe("Device size of the page"),
         waitMs: z.number().int().min(0).max(10000).default(3000).describe("Wait after the load event before running the workflow, in ms"),
+        detail: Detail,
       }),
       outputSchema: z.object({
         url: z.string(),
+        runId: SummaryRunId,
+        detail: DetailOut,
         workflow: z.string(),
         viewport: z.enum(VIEWPORTS),
         navMs: z.number(),
@@ -200,10 +244,11 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
         results: z
           .array(z.looseObject({ id: z.string(), status: z.string() }).meta({ additionalProperties: true }))
           .describe("One object per step, in order, then the follow-ups, each with the reason it was added"),
+        nextSteps: NextSteps,
       }),
       annotations: { readOnlyHint: true, openWorldHint: true },
     },
-    async ({ url, workflow, viewport, waitMs }, ctx) => {
+    async ({ url, workflow, viewport, waitMs, detail }, ctx) => {
       try {
         const run = await runMeasurement({
           url,
@@ -219,7 +264,24 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
             ? { ...r, reason: "This step measures clicks and key presses: open the page with open_page and this snippet in track, use it with interact, and read it with read_results." }
             : r
         );
-        return toolResult({ url, workflow, viewport, navMs: run.navMs, pageErrors: run.pageErrors, results });
+        const shaped = shapeResults(results, detail);
+        const runId = shaped.detail === "summary" ? runs.keep(results) : undefined;
+        // The runner already ran the follow-ups of the decision tree; what is left is the steps it skipped
+        const skipped = results
+          .filter((r) => r.status === "skipped")
+          .map((r) => WORKFLOWS[workflow].steps.find((s) => s.id === r.id)?.path)
+          .filter(Boolean);
+        const nextSteps = skipped.length ? [trackingCall({ paths: skipped, url, viewport })] : [];
+        return toolResult({
+          url,
+          ...(runId ? { runId } : {}),
+          ...shaped,
+          workflow,
+          viewport,
+          navMs: run.navMs,
+          pageErrors: run.pageErrors,
+          ...(nextSteps.length ? { nextSteps } : {}),
+        });
       } catch (err) {
         if (isMissingBrowser(err)) return toolError(MISSING_BROWSER);
         if (/page\.goto/.test(err.message)) return toolError(`Could not load ${url}: ${firstLine(err.message)}`);
@@ -358,22 +420,58 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
     {
       title: "Read the tracking snippets",
       description:
-        `Returns what the tracking snippets installed by open_page have recorded on the page so far, as the objects they return. A snippet with nothing recorded yet, such as INP before any interaction, answers with status tracking and a message: interact with the page and read again. ${PAGE_DATA}`,
-      inputSchema: z.object({ sessionId: SessionId }),
-      outputSchema: z.object({ sessionId: z.string(), pageErrors: z.array(z.string()), results: z.array(SnippetResult) }),
+        `Returns what the tracking snippets installed by open_page have recorded on the page so far, as the objects they return. A snippet with nothing recorded yet, such as INP before any interaction, answers with status tracking and a message: interact with the page and read again. ${DETAIL_NOTE} ${PAGE_DATA}`,
+      inputSchema: z.object({ sessionId: SessionId, detail: Detail }),
+      outputSchema: z.object({ sessionId: z.string(), runId: SummaryRunId, detail: DetailOut, pageErrors: z.array(z.string()), results: z.array(SnippetResult) }),
       annotations: { readOnlyHint: true, openWorldHint: false },
     },
-    withSession(async ({ sessionId }) => {
+    withSession(async ({ sessionId, detail }) => {
       const session = sessions.get(sessionId);
-      const results = [];
-      for (const installed of session.tracking) {
-        const { id, ...result } = await collectTrackingData(session.page, installed, session.hadInput);
-        results.push(result);
-      }
-      return toolResult({ sessionId, pageErrors: session.pageErrors, results });
+      const collected = [];
+      for (const installed of session.tracking) collected.push(await collectTrackingData(session.page, installed, session.hadInput));
+      const shaped = shapeResults(collected.map(({ id, ...result }) => result), detail);
+      const runId = shaped.detail === "summary" ? runs.keep(collected) : undefined;
+      return toolResult({ sessionId, ...(runId ? { runId } : {}), ...shaped, pageErrors: session.pageErrors });
     })
   );
 
+  server.registerTool(
+    "get_details",
+    {
+      title: "Get the details of a result",
+      description:
+        `Returns the whole object of one snippet of a summarized run, by its runId: what the summary leaves out, from the same run, without measuring again. ${PAGE_DATA}`,
+      inputSchema: z.object({
+        runId: RunId,
+        snippet: z.string().describe("The id of the result in the summary, or its script"),
+      }),
+      outputSchema: z.object({ runId: z.string(), snippet: z.string().describe("The id of the result in the run"), result: SnippetResult }),
+      annotations: { readOnlyHint: true, openWorldHint: false },
+    },
+    async ({ runId, snippet }) => {
+      // The id the summary shows first, then the Category/Name an alias or a short name stands for
+      let resolved;
+      try {
+        resolved = resolveSnippetName(snippet);
+      } catch {
+        resolved = undefined;
+      }
+      try {
+        let result;
+        try {
+          result = runs.get(runId, snippet);
+        } catch (err) {
+          if (!(err instanceof RunError) || !resolved || resolved === snippet) throw err;
+          result = runs.get(runId, resolved);
+        }
+        return toolResult({ runId, snippet: result.id, result });
+      } catch (err) {
+        if (err instanceof RunError) return toolError(err.message);
+        throw err;
+      }
+    }
+  );
+
   server.registerTool(
     "close_page",
     {

package/src/mcp/summary.js

@@ -0,0 +1,55 @@
+import { randomUUID } from "node:crypto";
+
+// A failure of get_details the model can act on: a run that is not kept, or a snippet it did not run
+export class RunError extends Error {}
+
+// What an agent needs from a result to decide what to do next; the rest (details, items, thresholds)
+// is in get_details, from the same run
+const SUMMARY_FIELDS = ["id", "script", "status", "metric", "value", "unit", "rating", "error", "message", "reason"];
+const ISSUE_FIELDS = ["code", "severity", "message", "target"];
+
+const pick = (object, fields) => Object.fromEntries(fields.filter((f) => object[f] !== undefined).map((f) => [f, object[f]]));
+
+export function summarize(result) {
+  const summary = pick(result, SUMMARY_FIELDS);
+  if (result.issues?.length) summary.issues = result.issues.map((issue) => pick(issue, ISSUE_FIELDS));
+  return summary;
+}
+
+// About 2,000 tokens. Below it the whole objects cost the agent less than the turn it would spend on
+// get_details for what a summary leaves out: in the MCP evals of 2026-10-29, summaries of small results
+// made the agent call get_details in 4 of 9 runs, and each turn sent 8 to 12k tokens of context again.
+export const FULL_UP_TO_CHARS = 8000;
+
+// The results of a call as the agent asked for them; with auto, whole when they are small, and summaries
+// with get_details for the rest when they are large
+export function shapeResults(results, detail = "auto", maxFullChars = FULL_UP_TO_CHARS) {
+  const full = detail === "full" || (detail === "auto" && JSON.stringify(results).length <= maxFullChars);
+  return full ? { detail: "full", results } : { detail: "summary", results: results.map(summarize) };
+}
+
+// The full results of the last runs of an MCP connection, by run id, so get_details answers with the
+// values the summary came from: measuring again would give other ones
+export function createRuns({ max = 20 } = {}) {
+  const runs = new Map();
+
+  function keep(results) {
+    const runId = randomUUID();
+    runs.set(runId, results);
+    // A Map keeps the order of insertion, so the first key is the oldest run
+    if (runs.size > max) runs.delete(runs.keys().next().value);
+    return runId;
+  }
+
+  function get(runId, snippet) {
+    const results = runs.get(runId);
+    if (!results) {
+      throw new RunError(`Unknown runId ${runId}: this server keeps the last ${max} runs of the connection. Run the tool again for a new one.`);
+    }
+    const result = results.find((r) => r.id === snippet) ?? results.find((r) => r.script === snippet);
+    if (!result) throw new RunError(`${snippet} is not in run ${runId}, which has: ${results.map((r) => r.id).join(", ")}.`);
+    return result;
+  }
+
+  return { keep, get };
+}

package/src/reporters/baseline-view.js

@@ -15,9 +15,22 @@ function describeChange(metric, { delta, deltaPercent }) {
   return `${arrow(delta)} ${ms}${percent}`;
 }
 
+// Three decimals where the verdict is decided, two above it: "0.028", "0.35"
+const pValue = (p) => (p < 0.001 ? "< 0.001" : p < 0.05 ? p.toFixed(3) : p.toFixed(2));
+const amountOf = (metric, n) => (metric === "CLS" ? signed(n, Math.abs(n).toFixed(4)) : signed(n, `${Math.abs(Math.round(n))}ms`));
+
+// The evidence of a change measured with five runs a side: its p-value, its shift and the 95 % interval
+// of the shift; for a change over the threshold that stayed unchanged, that the runs gave no evidence
+function evidenceOf(metric, m) {
+  const { p, shift, ci } = m.stats;
+  if (m.verdict === "unchanged") return Math.abs(m.delta) > m.threshold ? `no evidence: p = ${pValue(p)}` : null;
+  return `p = ${pValue(p)}, ${amountOf(metric, shift)}, 95% CI ${amountOf(metric, ci[0])} to ${amountOf(metric, ci[1])}`;
+}
+
 // An unchanged metric that moved stayed under its threshold, or beat it inside the range of the runs
 // of the baseline: noise the baseline already had
 function noteOf(metric, m) {
+  if (m.stats) return evidenceOf(metric, m) ?? (m.verdict === "unchanged" && m.delta !== 0 ? `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}` : null);
   if (m.verdict !== "unchanged" || m.delta === 0) return null;
   if (Math.abs(m.delta) > m.threshold) return "inside the range of the baseline";
   return `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}`;

package/src/reporters/human.js

@@ -3,7 +3,7 @@ import { describeResult } from "../view.js";
 import { isSummaryResult, summaryNumbers } from "./summary-view.js";
 import { fieldDataView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
-import { describeRuns } from "../runs.js";
+import { describeRuns, describeAgreement } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
 import { baselineView } from "./baseline-view.js";
 import { budgetRows, matrixBudgetFailures } from "./budgets-view.js";
@@ -210,12 +210,13 @@ function renderFieldData(crux, results) {
   return lines;
 }
 
-// The runs line, and the median, min and max of each metric across the runs
+// The runs line, and the median, min and max of each metric across the runs, with how much they agree
 function renderRuns(runs) {
   const lines = [styleText("dim", describeRuns(runs))];
-  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
-    const cell = (label, value) => pad(`${label} ${formatValue(value, unit)}`, 16);
-    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max).trimEnd()}`));
+  for (const [metric, value] of Object.entries(runs.summary)) {
+    const { median, min, max, unit } = value;
+    const cell = (label, v) => pad(`${label} ${formatValue(v, unit)}`, 16);
+    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max)}${describeAgreement(value)}`.trimEnd()));
   }
   return lines;
 }
@@ -274,7 +275,7 @@ export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, th
 // One row per combination with the rated median of each metric, and below it the errors that decide
 // the exit code. With verbose, the report of the median run of each combination follows.
 export function reportMatrixHuman(report) {
-  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison, legend } = matrixView(report);
   const columns = [["name", "Page"], ["device", "Device"], ...(hasVariants ? [["variant", "Variant"]] : [])];
   const widths = columns.map(([key, label]) => Math.max(...report.combinations.map((c) => c[key].length), label.length) + 3);
   const identity = (values) => values.map((value, i) => pad(value, widths[i])).join("");
@@ -292,11 +293,13 @@ export function reportMatrixHuman(report) {
       continue;
     }
     const drawn = cells.map((cell, i) => {
-      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}`), 14) : pad("·", 14);
+      const width = legend ? 18 : 14;
+      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}${cell.spread ? ` ${cell.spread}` : ""}`), width) : pad("·", width);
       return value + (changes ? pad(styleText(VERDICT_STYLE[changes[i].verdict], changes[i].text), 12) : "");
     });
     lines.push(row + drawn.join("").trimEnd());
   }
+  if (legend) lines.push(styleText("dim", `  ${legend}`));
 
   if (errors.length) {
     lines.push("");

package/src/reporters/markdown.js

@@ -2,7 +2,7 @@ import { fieldDataView } from "../crux.js";
 import { describeResult } from "../view.js";
 import { isSummaryResult, summaryNumbers } from "./summary-view.js";
 import { describeThrottling } from "../throttling.js";
-import { describeRuns } from "../runs.js";
+import { describeRuns, describeAgreement } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
 import { baselineView } from "./baseline-view.js";
 import { budgetRows, matrixBudgetFailures } from "./budgets-view.js";
@@ -115,11 +115,12 @@ function renderFieldData(crux, results) {
   ];
 }
 
-// The median, min and max of each metric across the runs
+// The median, min and max of each metric across the runs, and how much they agree
 function renderRuns(runs) {
-  const lines = ["| Metric | Median | Min | Max |", "|--------|--------|-----|-----|"];
-  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
-    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} |`);
+  const lines = ["| Metric | Median | Min | Max | Runs |", "|--------|--------|-----|-----|------|"];
+  for (const [metric, value] of Object.entries(runs.summary)) {
+    const { median, min, max, unit } = value;
+    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} | ${describeAgreement(value) || "·"} |`);
   }
   return lines;
 }
@@ -196,7 +197,7 @@ export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttli
 // report of the median run of each combination folded into a details block, to keep a pull request
 // comment readable.
 export function reportMatrixMarkdown(report) {
-  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison, legend } = matrixView(report);
   // With --baseline, a change column follows each metric
   const metricHeadings = metrics.flatMap((m) => (comparison ? [m, "Δ"] : [m]));
   const headings = ["Page", "Device", ...(hasVariants ? ["Variant"] : []), ...metricHeadings];
@@ -208,12 +209,13 @@ export function reportMatrixMarkdown(report) {
   for (const { combination: c, cells, changes } of rows) {
     const drawn = cells
       ? cells.flatMap((cell, i) => {
-          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}` : "·";
+          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}${cell.spread ? ` ${cell.spread}` : ""}` : "·";
           return changes ? [value, changes[i].text] : [value];
         })
       : ["❌ every run failed", ...metricHeadings.slice(1).map(() => "")];
     lines.push(row([c.name, c.device, ...(hasVariants ? [c.variant] : []), ...drawn]));
   }
+  if (legend) lines.push("", `> ${legend}`);
 
   if (errors.length) {
     lines.push("", "### Errors", ...errors.map((e) => `- ${e}`));

package/src/reporters/matrix-view.js

@@ -6,6 +6,11 @@ import { changeCell, matrixComparisonView } from "./baseline-view.js";
 // The order of the metric columns of the matrix table
 const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
 
+// A median whose runs are variable or noisy carries their coefficient of variation in its cell; the
+// others carry nothing, so the table stays narrow and the marks stand out
+const varies = (value) => value?.agreement === "variable" || value?.agreement === "noisy";
+const LEGEND = "±: the coefficient of variation of the runs of a variable or noisy median";
+
 // What the human and markdown reports of a matrix show, so they only differ in how they draw it:
 // the title, the metric columns, one row per combination with the rated median of each metric, and
 // the errors that decide the exit code. With --baseline, each row also has the change of each metric
@@ -31,10 +36,13 @@ export function matrixView({ matrix, runs, combinations, baseline }) {
         ? null
         : metrics.map((metric) => {
             const value = c.runs.summary[metric];
-            return value ? { median: value.median, unit: value.unit, rating: rate(metric, value.median) } : null;
+            return value
+              ? { median: value.median, unit: value.unit, rating: rate(metric, value.median), ...(varies(value) ? { spread: `±${Math.round(value.cv)}%` } : {}) }
+              : null;
           }),
     })),
     ...(baseline ? { comparison: matrixComparisonView(baseline, combinations) } : {}),
+    ...(measured.some((c) => metrics.some((metric) => varies(c.runs.summary[metric]))) ? { legend: LEGEND } : {}),
     errors: combinations.flatMap((c) =>
       c.error ? [`${c.id}: ${c.error}`] : blockingErrors(c.results).map((e) => `${c.id}: ${e.id}: ${e.message}`)
     ),

package/src/runs.js

@@ -1,3 +1,5 @@
+import { DEFAULT_THRESHOLDS } from "./baseline.js";
+
 // Several runs of the same measurement, reduced to one: the median run, as WebPageTest reports it.
 // Every result of the report comes from that run, so they agree with each other (the sub-parts of
 // the LCP add up to its LCP), and a summary gives the median, min and max of each metric.
@@ -24,6 +26,24 @@ const measures = (runs, metric) => runs.some((run) => typeof valueOf(run, metric
 // the two middle ones, so the median is always a run that happened, never an average of two.
 const middle = (sorted) => sorted[Math.floor((sorted.length - 1) / 2)];
 
+// How much the runs of a metric agree: their coefficient of variation, the sample standard deviation
+// over the mean, in percent, and its band. Up to 5 % the runs agree; above 10 % one median says little.
+// None for one run, or for runs whose mean is 0, such as a CLS of 0 in every run. Runs that vary by
+// less than half of the change --baseline counts agree whatever their CV: a local TTFB of 3 to 18 ms
+// has a CV of 64 % and a standard deviation of 6 ms.
+function agreementOf(values, metric) {
+  if (values.length < 2) return {};
+  const mean = values.reduce((sum, v) => sum + v, 0) / values.length;
+  if (mean === 0) return {};
+  const sd = Math.sqrt(values.reduce((sum, v) => sum + (v - mean) ** 2, 0) / (values.length - 1));
+  const cv = Math.round((sd / mean) * 1000) / 10;
+  if (sd < DEFAULT_THRESHOLDS[metric].absolute / 2) return { cv, agreement: "consistent" };
+  return { cv, agreement: cv <= 5 ? "consistent" : cv <= 10 ? "variable" : "noisy" };
+}
+
+// "noisy (CV 14.4%)", for the reports
+export const describeAgreement = ({ cv, agreement }) => (cv === undefined ? "" : `${agreement} (CV ${cv}%)`);
+
 // "3 runs, median by LCP: run 2 (1 failed)", for the header of the reports
 export function describeRuns({ count, failed, medianRun, medianBy, medianByRequested }) {
   const by = medianByRequested ? `${medianBy} (no run measured ${medianByRequested})` : medianBy;
@@ -48,7 +68,7 @@ export function summarizeRuns(runs, { medianBy: requested } = {}) {
     const values = runs.map((run) => (run.error ? null : (valueOf(run, metric) ?? null)));
     const sorted = values.filter((value) => value !== null).sort((a, b) => a - b);
     const unit = completed.flatMap((run) => run.results).find((r) => r.metric === metric && r.unit)?.unit;
-    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit };
+    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit, ...agreementOf(sorted, metric) };
   }
 
   return {

package/src/stats.js

@@ -0,0 +1,108 @@
+// Statistics for comparing the runs of two measurements, without a browser: whether one sample tends to
+// be greater than the other (Mann-Whitney U, one-sided) and by how much (the Hodges-Lehmann shift with its
+// 95 % interval). Both work on ranks, so one slow run moves them less than it moves a mean.
+
+const EXACT_UP_TO = 16;
+const Z_975 = 1.959963984540054;
+
+// The ranks of the pooled values, with the mean rank for ties, and the sizes of the groups of ties
+function rank(values) {
+  const order = values.map((value, i) => ({ value, i })).sort((a, b) => a.value - b.value);
+  const ranks = new Array(values.length);
+  const ties = [];
+  for (let start = 0; start < order.length; ) {
+    let end = start;
+    while (end + 1 < order.length && order[end + 1].value === order[start].value) end++;
+    for (let k = start; k <= end; k++) ranks[order[k].i] = (start + end) / 2 + 1;
+    if (end > start) ties.push(end - start + 1);
+    start = end + 1;
+  }
+  return { ranks, ties };
+}
+
+// How many splits of m + n distinct values give each U, from U = 0 to m * n. A split with the largest
+// value in the first sample has U of the split without it plus n; otherwise, U of the split without it.
+function exactCounts(m, n) {
+  let prev = Array.from({ length: n + 1 }, () => [1]);
+  for (let i = 1; i <= m; i++) {
+    const row = [new Array(1).fill(1)];
+    for (let j = 1; j <= n; j++) {
+      const counts = new Array(i * j + 1).fill(0);
+      prev[j].forEach((c, u) => (counts[u + j] += c));
+      row[j - 1].forEach((c, u) => (counts[u] += c));
+      row.push(counts);
+    }
+    prev = row;
+  }
+  return prev[n];
+}
+
+// The complementary error function, with a fractional error under 1.2e-7 (Numerical Recipes, erfcc)
+function erfc(x) {
+  const z = Math.abs(x);
+  const t = 1 / (1 + 0.5 * z);
+  const r =
+    t *
+    Math.exp(
+      -z * z -
+        1.26551223 +
+        t * (1.00002368 + t * (0.37409196 + t * (0.09678418 + t * (-0.18628806 + t * (0.27886807 + t * (-1.13520398 + t * (1.48851587 + t * (-0.82215223 + t * 0.17087277))))))))
+    );
+  return x >= 0 ? r : 2 - r;
+}
+
+const upperTail = (z) => erfc(z / Math.SQRT2) / 2;
+
+const isExact = (m, n, ties) => m <= EXACT_UP_TO && n <= EXACT_UP_TO && ties.length === 0;
+
+// Whether `sample` tends to be greater than `reference`: U counts the pairs where it is (a tie counts
+// half), and p is the probability of a U this large or larger if both came from the same distribution.
+// For a regression of a timing, the sample is the current runs and the reference the baseline.
+export function mannWhitneyGreater(sample, reference) {
+  const m = sample.length;
+  const n = reference.length;
+  const { ranks, ties } = rank([...sample, ...reference]);
+  const rankSum = ranks.slice(0, m).reduce((sum, r) => sum + r, 0);
+  const u = rankSum - (m * (m + 1)) / 2;
+
+  if (isExact(m, n, ties)) {
+    const counts = exactCounts(m, n);
+    const total = counts.reduce((sum, c) => sum + c, 0);
+    const above = counts.slice(Math.ceil(u)).reduce((sum, c) => sum + c, 0);
+    return { u, p: above / total, method: "exact" };
+  }
+
+  const N = m + n;
+  const tieTerm = ties.reduce((sum, t) => sum + t ** 3 - t, 0) / (N * (N - 1));
+  const sigma = Math.sqrt(((m * n) / 12) * (N + 1 - tieTerm));
+  const z = (u - (m * n) / 2 - 0.5) / sigma;
+  return { u, p: upperTail(z), method: "normal" };
+}
+
+const median = (sorted) => {
+  const mid = sorted.length / 2;
+  return sorted.length % 2 ? sorted[Math.floor(mid)] : (sorted[mid - 1] + sorted[mid]) / 2;
+};
+
+// By how much `sample` is greater than `reference`: the median of the differences of every pair, and the
+// 95 % interval between two of those differences, chosen from the distribution of U. With samples too
+// small for 95 %, such as three runs a side, the interval is every difference.
+export function hodgesLehmann(reference, sample) {
+  const m = sample.length;
+  const n = reference.length;
+  const diffs = sample.flatMap((s) => reference.map((r) => s - r)).sort((a, b) => a - b);
+  const { ties } = rank([...sample, ...reference]);
+
+  let k;
+  if (isExact(m, n, ties)) {
+    // The largest k with P(U <= k - 1) at most 2.5 %
+    const counts = exactCounts(m, n);
+    const total = counts.reduce((sum, c) => sum + c, 0);
+    let below = 0;
+    k = 0;
+    while (k < counts.length && (below + counts[k]) / total <= 0.025) below += counts[k++];
+  } else {
+    k = Math.max(0, Math.floor((m * n) / 2 - Z_975 * Math.sqrt((m * n * (m + n + 1)) / 12)));
+  }
+  return { shift: median(diffs), ci: [diffs[k], diffs[m * n - 1 - k]] };
+}