webperf-snippets 1.8.0 → 1.9.0

package/README.md

@@ -70,10 +70,10 @@ JSON output (for piping into `jq` or CI):
 npx webperf-snippets https://example.com --json
 ```
 
-The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait and the version of the CLI.
+The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait, the version of the CLI and the machine: its CPU model, cores, architecture and memory, as the operating system reports them, and a benchmark index, a short CPU test in Node before Chromium starts, which only informs, since on the same machine it moves by up to 40 % with the load.
 
 ```json
-"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.8.0" }
+"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.9.0", "machine": { "cpu": "Apple M4", "cores": 10, "arch": "arm64", "memoryGB": 16, "benchmarkIndex": 2371 } }
 ```
 
 Single snippet:
@@ -158,7 +158,7 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.8.0" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.9.0" }
 }
 ```
 
@@ -190,7 +190,7 @@ One page load is one sample, and lab values vary from one load to the next. `--r
 npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
 ```
 
-The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and the JSON output carries them in `runs`:
+The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and how much the runs agree, and the JSON output carries them in `runs`:
 
 ```json
 "runs": {
@@ -199,12 +199,12 @@ The median run is the one with the median LCP. A workflow without LCP uses the f
   "medianRun": 3,
   "medianBy": "LCP",
   "summary": {
-    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms" }
+    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms", "cv": 11.6, "agreement": "noisy" }
   }
 }
 ```
 
-`medianRun` counts from 1, and `values` keeps the order of the runs. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
+`medianRun` counts from 1, and `values` keeps the order of the runs. `cv` is the coefficient of variation of the runs, their standard deviation over their mean in percent, and `agreement` its band: `consistent` up to 5 %, `variable` up to 10 % and `noisy` above, where one median says little and more runs help. Runs that vary by less than half of the minimum change of `--baseline` (50 ms for LCP, FCP and TTFB, 25 ms for INP, 0.025 for CLS) are `consistent` whatever their CV, since a local TTFB of 3 to 18 ms has a CV of 64 %. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
 
 Measure several pages, on several devices and in several variants, in one command with `--matrix <file>`. The file lists them, and the CLI measures every URL on every device and in every variant, each combination `runs` times, one load after the other:
 
@@ -238,7 +238,7 @@ npx webperf-snippets --matrix matrix.json --budget-lcp 2500
 
 The options of the command line apply to every combination: `--workflow`, `--snippet`, `--wait`, `--interact-script`, `--header`, `--own-domains`, `--storage-state`, `--cpu`, `--network` and the budgets. The file replaces `cpu`, `network` and `storageState` where it sets them, and its `headers` and `ownDomains` add to the ones of the command line, the variant winning on the same header. A matrix file is meant to be versioned, so pass secrets, such as the token of an access proxy, with `--header` and an environment variable; a header of the file whose name looks like a secret raises a warning. A key the CLI does not know is an error, so a typo does not measure without its condition.
 
-The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
+The report is a table with one row per combination and the rated median of each metric, a variable or noisy one followed by the coefficient of variation of its runs (`🟡 2.71s ±14%`), then the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
 
 With CrUX, the CLI asks once for each URL and form factor, so the variants of a device share their field data. With `--report-to`, it posts one report for each combination, the body of one page with `meta.device`, `meta.variant` and `meta.runs`; the headers of the command line and of the file are conditions of the load, and no report carries their values.
 
@@ -257,6 +257,8 @@ npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.jso
 
 LCP, FCP, TTFB, CLS and INP compare when both runs measured them, the median of each with `--runs`; a metric that only one side measured is `not measured`. A metric regresses when it gets worse by more than its threshold, the larger of a share of the baseline and a minimum, and, when the baseline has two runs or more, ends above the slowest of them: a median inside the range of the baseline is noise the baseline already had. It improves under the same rules the other way, and otherwise it is unchanged.
 
+With five runs or more on each side, the range gives way to a test of the runs themselves: a change that beats its threshold also needs a one-sided Mann-Whitney U p-value under 0.05, the chance that runs of an unchanged page would differ this much. A median can move with two slow runs out of five; the test asks whether most runs moved. The JSON of the metric adds `stats`, with `p`, the `shift` (the Hodges-Lehmann estimate, the median of the differences between every run of one side and every run of the other) and its 95 % interval `ci`, and the reports add them to the verdict: `regressed (p = 0.028, +500ms, 95% CI +20ms to +600ms)`, or `unchanged (no evidence: p = 0.35)` for a change over the threshold that the runs do not back. Three runs a side can never reach p < 0.05, so under five the range decides.
+
 | Metric         | Share of the baseline | Minimum |
 | -------------- | --------------------- | ------- |
 | LCP, FCP, TTFB | 10%                   | 100 ms  |
@@ -282,7 +284,7 @@ The report adds the comparison after the usual one, and `--markdown` a table for
 }
 ```
 
-When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
+When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, on another CPU model or with other cores, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
 
 The comparison only informs, and the exit code is the same as without it, unless `--fail-on-regression`, which exits with `1` when a metric regressed or an error is new, and names them on stderr. A baseline that does not exist or is not an output of the CLI, a baseline of a page for a matrix or of a matrix for a page, or a `--threshold` the CLI does not understand exits with code `2`, before the page loads.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",

package/src/baseline.js

@@ -1,6 +1,7 @@
 import { readFileSync } from "node:fs";
 import { blockingErrors } from "./blocking.js";
 import { describeThrottling } from "./throttling.js";
+import { mannWhitneyGreater, hodgesLehmann } from "./stats.js";
 
 // A measurement compared with an earlier one, its baseline: which metrics got worse or better beyond
 // the noise of the lab, which errors appeared or went away, and what differs in how both were
@@ -71,23 +72,40 @@ export function thresholdsWith(overrides) {
 
 const round = (n) => Math.round(n * 10000) / 10000;
 
-// The value of a metric in a page: the median of its runs, with their range when there are two or
-// more, or its one result, as runs.js picks it
+// The value of a metric in a page: the median of its runs, with their range and their values when there
+// are two or more, or its one result, as runs.js picks it
 function measured(payload, metric) {
   const summary = payload.runs?.summary?.[metric];
   if (summary) {
-    const runs = summary.values.filter((value) => value !== null).length;
-    return { value: summary.median, ...(runs >= 2 ? { range: { min: summary.min, max: summary.max } } : {}) };
+    const values = summary.values.filter((value) => value !== null);
+    return { value: summary.median, ...(values.length >= 2 ? { range: { min: summary.min, max: summary.max }, values } : {}) };
   }
   const result = payload.results?.find((r) => r.metric === metric && r.status === "ok" && typeof r.value === "number");
   return result ? { value: result.value } : null;
 }
 
-// A regression must beat the threshold of its metric and, when the baseline has several runs, the
-// worst of them: a median inside the range of the baseline is noise the baseline already had.
-function verdict(base, current, threshold) {
-  if (current.value - base.value > threshold && (!base.range || current.value > base.range.max)) return "regressed";
-  if (base.value - current.value > threshold && (!base.range || current.value < base.range.min)) return "improved";
+// With five runs or more a side, Mann-Whitney U can tell a change from the noise of the runs; with
+// fewer, its p-value cannot go under 0.05 (three a side give 0.05 at best), so the range decides.
+export const RUNS_FOR_STATISTICS = 5;
+const ALPHA = 0.05;
+
+// The evidence of a change: the one-sided p-value in its direction, and the shift with its 95 % interval
+function statsOf(base, current, delta) {
+  if ((base.values?.length ?? 0) < RUNS_FOR_STATISTICS || (current.values?.length ?? 0) < RUNS_FOR_STATISTICS) return undefined;
+  const { p, method } = delta < 0 ? mannWhitneyGreater(base.values, current.values) : mannWhitneyGreater(current.values, base.values);
+  const { shift, ci } = hodgesLehmann(base.values, current.values);
+  return { p: round(p), shift: round(shift), ci: ci.map(round), method };
+}
+
+// A regression must beat the threshold of its metric and, with five runs a side, be unlikely to come
+// from the noise of the runs (p < 0.05). With fewer, when the baseline has several runs, it must beat
+// the worst of them: a median inside the range of the baseline is noise the baseline already had.
+function verdict(base, current, threshold, stats) {
+  const worse = current.value - base.value > threshold;
+  const better = base.value - current.value > threshold;
+  if (stats) return stats.p < ALPHA ? (worse ? "regressed" : better ? "improved" : "unchanged") : "unchanged";
+  if (worse && (!base.range || current.value > base.range.max)) return "regressed";
+  if (better && (!base.range || current.value < base.range.min)) return "improved";
   return "unchanged";
 }
 
@@ -104,13 +122,15 @@ function compareMetrics(base, current, thresholds) {
     const { relative = 0, absolute } = thresholds[metric];
     const threshold = round(Math.max(relative * b.value, absolute));
     const delta = c.value - b.value;
+    const stats = statsOf(b, c, delta);
     metrics[metric] = {
       baseline: b.value,
       current: c.value,
       delta: round(delta),
       deltaPercent: b.value === 0 ? null : Math.round((delta / b.value) * 1000) / 10,
       threshold,
-      verdict: verdict(b, c, threshold),
+      verdict: verdict(b, c, threshold, stats),
+      ...(stats ? { stats } : {}),
     };
   }
   return metrics;
@@ -166,6 +186,14 @@ const pathOf = (url) => {
   }
 };
 
+// Another CPU or other cores give other timings for the same page. A different benchmark index alone
+// does not warn: on the same machine it moves by 40 % with its load.
+function machineWarning(base, current) {
+  if (!base?.cpu || !current?.cpu) return null;
+  if (base.cpu === current.cpu && base.cores === current.cores) return null;
+  return `the baseline was measured on another machine (${base.cpu}, ${base.cores} cores) than this run (${current.cpu}, ${current.cores} cores); timings can differ for that alone`;
+}
+
 // What differs in how both were measured. The comparison goes on: the warnings say how far to trust it.
 function warningsOf(base, current) {
   const warnings = [];
@@ -183,6 +211,8 @@ function warningsOf(base, current) {
   }
   if (runCount(base) < 2) warnings.push("the baseline has 1 run, so only the thresholds apply; measure both with --runs");
   else if (runCount(base) !== runCount(current)) warnings.push(`the baseline has ${runCount(base)} runs, and this run ${runCount(current)}`);
+  const machine = machineWarning(b?.machine, c?.machine);
+  if (machine) warnings.push(machine);
   if (b && c && b.cliVersion !== c.cliVersion) {
     warnings.push(`the baseline was measured with CLI ${b.cliVersion}, and this run with ${c.cliVersion}; the snippets may have changed`);
   }

package/src/bin.js

@@ -20,6 +20,7 @@ import { loadMatrix } from "./matrix.js";
 import { blockingErrors } from "./blocking.js";
 import { MEASURED_BY, checkBudgets, parseBudgets, unmeasuredMetrics, violationsOf } from "./budgets.js";
 import { measurementConditions } from "./conditions.js";
+import { machineOf } from "./machine.js";
 import { isHttpUrl } from "./url.js";
 import { loadBaseline, parseThreshold, thresholdsWith, compareRun, compareMatrix } from "./baseline.js";
 import { describeRegressions, describeMatrixRegressions, matrixComparisonView } from "./reporters/baseline-view.js";
@@ -287,9 +288,11 @@ async function main() {
     const options = { waitMs, headless: !values.headed, interactScript, ...conditions };
     return items ? runSnippets({ ...options, items }) : runMeasurement({ ...options, workflow, rules: RULES });
   };
-  // What the JSON records of how each page was measured, for a later run to compare with
+  // What the JSON records of how each page was measured, for a later run to compare with. The machine is
+  // measured once, before Chromium starts, so its processes do not slow the benchmark down.
+  const machine = machineOf();
   const conditionsFor = (viewport) =>
-    measurementConditions({ viewport, waitMs, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
+    measurementConditions({ viewport, waitMs, machine, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
 
   if (matrixPath) {
     let matrix;

package/src/conditions.js

@@ -2,6 +2,6 @@ import { cliVersion } from "./report-to.js";
 
 // How a measurement was taken, kept in its JSON so that a later run compared with it can tell whether
 // both were measured the same way. A snippet is named by its path, so an alias and its path match.
-export function measurementConditions({ viewport, workflow, snippet, waitMs }) {
-  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion };
+export function measurementConditions({ viewport, workflow, snippet, waitMs, machine }) {
+  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion, ...(machine ? { machine } : {}) };
 }

package/src/machine.js

@@ -0,0 +1,42 @@
+import { arch as osArch, cpus as osCpus, totalmem as osTotalmem } from "node:os";
+
+// The machine a measurement ran on, kept in its conditions, so --baseline can say when two were measured
+// on different machines: timings move with the CPU as much as with the page. The machine is told apart by
+// what the operating system reports of it. The benchmark index only informs: on the same machine it
+// moves by 40 % with its load and with the cores the system picks (performance or efficiency).
+
+const WINDOW_MS = 40;
+const WINDOWS = 5;
+
+// A fixed piece of work: build a string and sort an array, as the benchmark index of Lighthouse builds
+// strings in the page. Its result is returned so the engine cannot drop it.
+function work() {
+  let s = "";
+  for (let i = 0; i < 2000; i++) s += String.fromCharCode(97 + (i % 26));
+  const numbers = Array.from({ length: 200 }, (_, i) => (i * 7919) % 211);
+  numbers.sort((a, b) => a - b);
+  return s.length + numbers[0];
+}
+
+// How many times the work fits in 40 ms, the best of five windows: the first one also warms up the
+// engine, and the best one is the least touched by other work of the machine. In Node, so --cpu, which
+// throttles the page, leaves it alone: it measures the machine. Alone, ten in a row vary by about 6 %.
+export function benchmarkIndex() {
+  let best = 0;
+  let sink = 0;
+  for (let w = 0; w < WINDOWS; w++) {
+    const end = performance.now() + WINDOW_MS;
+    let count = 0;
+    while (performance.now() < end) {
+      sink += work();
+      count++;
+    }
+    best = Math.max(best, count);
+  }
+  return sink > 0 ? best : 0;
+}
+
+export function machineOf({ cpus = osCpus, arch = osArch(), totalmem = osTotalmem, benchmark = benchmarkIndex } = {}) {
+  const list = cpus();
+  return { cpu: list[0]?.model ?? "unknown", cores: list.length, arch, memoryGB: Math.round(totalmem() / 2 ** 30), benchmarkIndex: benchmark() };
+}

package/src/reporters/baseline-view.js

@@ -15,9 +15,22 @@ function describeChange(metric, { delta, deltaPercent }) {
   return `${arrow(delta)} ${ms}${percent}`;
 }
 
+// Three decimals where the verdict is decided, two above it: "0.028", "0.35"
+const pValue = (p) => (p < 0.001 ? "< 0.001" : p < 0.05 ? p.toFixed(3) : p.toFixed(2));
+const amountOf = (metric, n) => (metric === "CLS" ? signed(n, Math.abs(n).toFixed(4)) : signed(n, `${Math.abs(Math.round(n))}ms`));
+
+// The evidence of a change measured with five runs a side: its p-value, its shift and the 95 % interval
+// of the shift; for a change over the threshold that stayed unchanged, that the runs gave no evidence
+function evidenceOf(metric, m) {
+  const { p, shift, ci } = m.stats;
+  if (m.verdict === "unchanged") return Math.abs(m.delta) > m.threshold ? `no evidence: p = ${pValue(p)}` : null;
+  return `p = ${pValue(p)}, ${amountOf(metric, shift)}, 95% CI ${amountOf(metric, ci[0])} to ${amountOf(metric, ci[1])}`;
+}
+
 // An unchanged metric that moved stayed under its threshold, or beat it inside the range of the runs
 // of the baseline: noise the baseline already had
 function noteOf(metric, m) {
+  if (m.stats) return evidenceOf(metric, m) ?? (m.verdict === "unchanged" && m.delta !== 0 ? `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}` : null);
   if (m.verdict !== "unchanged" || m.delta === 0) return null;
   if (Math.abs(m.delta) > m.threshold) return "inside the range of the baseline";
   return `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}`;

package/src/reporters/human.js

@@ -3,7 +3,7 @@ import { describeResult } from "../view.js";
 import { isSummaryResult, summaryNumbers } from "./summary-view.js";
 import { fieldDataView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
-import { describeRuns } from "../runs.js";
+import { describeRuns, describeAgreement } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
 import { baselineView } from "./baseline-view.js";
 import { budgetRows, matrixBudgetFailures } from "./budgets-view.js";
@@ -210,12 +210,13 @@ function renderFieldData(crux, results) {
   return lines;
 }
 
-// The runs line, and the median, min and max of each metric across the runs
+// The runs line, and the median, min and max of each metric across the runs, with how much they agree
 function renderRuns(runs) {
   const lines = [styleText("dim", describeRuns(runs))];
-  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
-    const cell = (label, value) => pad(`${label} ${formatValue(value, unit)}`, 16);
-    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max).trimEnd()}`));
+  for (const [metric, value] of Object.entries(runs.summary)) {
+    const { median, min, max, unit } = value;
+    const cell = (label, v) => pad(`${label} ${formatValue(v, unit)}`, 16);
+    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max)}${describeAgreement(value)}`.trimEnd()));
   }
   return lines;
 }
@@ -274,7 +275,7 @@ export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, th
 // One row per combination with the rated median of each metric, and below it the errors that decide
 // the exit code. With verbose, the report of the median run of each combination follows.
 export function reportMatrixHuman(report) {
-  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison, legend } = matrixView(report);
   const columns = [["name", "Page"], ["device", "Device"], ...(hasVariants ? [["variant", "Variant"]] : [])];
   const widths = columns.map(([key, label]) => Math.max(...report.combinations.map((c) => c[key].length), label.length) + 3);
   const identity = (values) => values.map((value, i) => pad(value, widths[i])).join("");
@@ -292,11 +293,13 @@ export function reportMatrixHuman(report) {
       continue;
     }
     const drawn = cells.map((cell, i) => {
-      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}`), 14) : pad("·", 14);
+      const width = legend ? 18 : 14;
+      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}${cell.spread ? ` ${cell.spread}` : ""}`), width) : pad("·", width);
       return value + (changes ? pad(styleText(VERDICT_STYLE[changes[i].verdict], changes[i].text), 12) : "");
     });
     lines.push(row + drawn.join("").trimEnd());
   }
+  if (legend) lines.push(styleText("dim", `  ${legend}`));
 
   if (errors.length) {
     lines.push("");

package/src/reporters/markdown.js

@@ -2,7 +2,7 @@ import { fieldDataView } from "../crux.js";
 import { describeResult } from "../view.js";
 import { isSummaryResult, summaryNumbers } from "./summary-view.js";
 import { describeThrottling } from "../throttling.js";
-import { describeRuns } from "../runs.js";
+import { describeRuns, describeAgreement } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
 import { baselineView } from "./baseline-view.js";
 import { budgetRows, matrixBudgetFailures } from "./budgets-view.js";
@@ -115,11 +115,12 @@ function renderFieldData(crux, results) {
   ];
 }
 
-// The median, min and max of each metric across the runs
+// The median, min and max of each metric across the runs, and how much they agree
 function renderRuns(runs) {
-  const lines = ["| Metric | Median | Min | Max |", "|--------|--------|-----|-----|"];
-  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
-    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} |`);
+  const lines = ["| Metric | Median | Min | Max | Runs |", "|--------|--------|-----|-----|------|"];
+  for (const [metric, value] of Object.entries(runs.summary)) {
+    const { median, min, max, unit } = value;
+    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} | ${describeAgreement(value) || "·"} |`);
   }
   return lines;
 }
@@ -196,7 +197,7 @@ export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttli
 // report of the median run of each combination folded into a details block, to keep a pull request
 // comment readable.
 export function reportMatrixMarkdown(report) {
-  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison, legend } = matrixView(report);
   // With --baseline, a change column follows each metric
   const metricHeadings = metrics.flatMap((m) => (comparison ? [m, "Δ"] : [m]));
   const headings = ["Page", "Device", ...(hasVariants ? ["Variant"] : []), ...metricHeadings];
@@ -208,12 +209,13 @@ export function reportMatrixMarkdown(report) {
   for (const { combination: c, cells, changes } of rows) {
     const drawn = cells
       ? cells.flatMap((cell, i) => {
-          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}` : "·";
+          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}${cell.spread ? ` ${cell.spread}` : ""}` : "·";
           return changes ? [value, changes[i].text] : [value];
         })
       : ["❌ every run failed", ...metricHeadings.slice(1).map(() => "")];
     lines.push(row([c.name, c.device, ...(hasVariants ? [c.variant] : []), ...drawn]));
   }
+  if (legend) lines.push("", `> ${legend}`);
 
   if (errors.length) {
     lines.push("", "### Errors", ...errors.map((e) => `- ${e}`));

package/src/reporters/matrix-view.js

@@ -6,6 +6,11 @@ import { changeCell, matrixComparisonView } from "./baseline-view.js";
 // The order of the metric columns of the matrix table
 const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
 
+// A median whose runs are variable or noisy carries their coefficient of variation in its cell; the
+// others carry nothing, so the table stays narrow and the marks stand out
+const varies = (value) => value?.agreement === "variable" || value?.agreement === "noisy";
+const LEGEND = "±: the coefficient of variation of the runs of a variable or noisy median";
+
 // What the human and markdown reports of a matrix show, so they only differ in how they draw it:
 // the title, the metric columns, one row per combination with the rated median of each metric, and
 // the errors that decide the exit code. With --baseline, each row also has the change of each metric
@@ -31,10 +36,13 @@ export function matrixView({ matrix, runs, combinations, baseline }) {
         ? null
         : metrics.map((metric) => {
             const value = c.runs.summary[metric];
-            return value ? { median: value.median, unit: value.unit, rating: rate(metric, value.median) } : null;
+            return value
+              ? { median: value.median, unit: value.unit, rating: rate(metric, value.median), ...(varies(value) ? { spread: `±${Math.round(value.cv)}%` } : {}) }
+              : null;
           }),
     })),
     ...(baseline ? { comparison: matrixComparisonView(baseline, combinations) } : {}),
+    ...(measured.some((c) => metrics.some((metric) => varies(c.runs.summary[metric]))) ? { legend: LEGEND } : {}),
     errors: combinations.flatMap((c) =>
       c.error ? [`${c.id}: ${c.error}`] : blockingErrors(c.results).map((e) => `${c.id}: ${e.id}: ${e.message}`)
     ),

package/src/runs.js

@@ -1,3 +1,5 @@
+import { DEFAULT_THRESHOLDS } from "./baseline.js";
+
 // Several runs of the same measurement, reduced to one: the median run, as WebPageTest reports it.
 // Every result of the report comes from that run, so they agree with each other (the sub-parts of
 // the LCP add up to its LCP), and a summary gives the median, min and max of each metric.
@@ -24,6 +26,24 @@ const measures = (runs, metric) => runs.some((run) => typeof valueOf(run, metric
 // the two middle ones, so the median is always a run that happened, never an average of two.
 const middle = (sorted) => sorted[Math.floor((sorted.length - 1) / 2)];
 
+// How much the runs of a metric agree: their coefficient of variation, the sample standard deviation
+// over the mean, in percent, and its band. Up to 5 % the runs agree; above 10 % one median says little.
+// None for one run, or for runs whose mean is 0, such as a CLS of 0 in every run. Runs that vary by
+// less than half of the change --baseline counts agree whatever their CV: a local TTFB of 3 to 18 ms
+// has a CV of 64 % and a standard deviation of 6 ms.
+function agreementOf(values, metric) {
+  if (values.length < 2) return {};
+  const mean = values.reduce((sum, v) => sum + v, 0) / values.length;
+  if (mean === 0) return {};
+  const sd = Math.sqrt(values.reduce((sum, v) => sum + (v - mean) ** 2, 0) / (values.length - 1));
+  const cv = Math.round((sd / mean) * 1000) / 10;
+  if (sd < DEFAULT_THRESHOLDS[metric].absolute / 2) return { cv, agreement: "consistent" };
+  return { cv, agreement: cv <= 5 ? "consistent" : cv <= 10 ? "variable" : "noisy" };
+}
+
+// "noisy (CV 14.4%)", for the reports
+export const describeAgreement = ({ cv, agreement }) => (cv === undefined ? "" : `${agreement} (CV ${cv}%)`);
+
 // "3 runs, median by LCP: run 2 (1 failed)", for the header of the reports
 export function describeRuns({ count, failed, medianRun, medianBy, medianByRequested }) {
   const by = medianByRequested ? `${medianBy} (no run measured ${medianByRequested})` : medianBy;
@@ -48,7 +68,7 @@ export function summarizeRuns(runs, { medianBy: requested } = {}) {
     const values = runs.map((run) => (run.error ? null : (valueOf(run, metric) ?? null)));
     const sorted = values.filter((value) => value !== null).sort((a, b) => a - b);
     const unit = completed.flatMap((run) => run.results).find((r) => r.metric === metric && r.unit)?.unit;
-    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit };
+    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit, ...agreementOf(sorted, metric) };
   }
 
   return {

package/src/stats.js

@@ -0,0 +1,108 @@
+// Statistics for comparing the runs of two measurements, without a browser: whether one sample tends to
+// be greater than the other (Mann-Whitney U, one-sided) and by how much (the Hodges-Lehmann shift with its
+// 95 % interval). Both work on ranks, so one slow run moves them less than it moves a mean.
+
+const EXACT_UP_TO = 16;
+const Z_975 = 1.959963984540054;
+
+// The ranks of the pooled values, with the mean rank for ties, and the sizes of the groups of ties
+function rank(values) {
+  const order = values.map((value, i) => ({ value, i })).sort((a, b) => a.value - b.value);
+  const ranks = new Array(values.length);
+  const ties = [];
+  for (let start = 0; start < order.length; ) {
+    let end = start;
+    while (end + 1 < order.length && order[end + 1].value === order[start].value) end++;
+    for (let k = start; k <= end; k++) ranks[order[k].i] = (start + end) / 2 + 1;
+    if (end > start) ties.push(end - start + 1);
+    start = end + 1;
+  }
+  return { ranks, ties };
+}
+
+// How many splits of m + n distinct values give each U, from U = 0 to m * n. A split with the largest
+// value in the first sample has U of the split without it plus n; otherwise, U of the split without it.
+function exactCounts(m, n) {
+  let prev = Array.from({ length: n + 1 }, () => [1]);
+  for (let i = 1; i <= m; i++) {
+    const row = [new Array(1).fill(1)];
+    for (let j = 1; j <= n; j++) {
+      const counts = new Array(i * j + 1).fill(0);
+      prev[j].forEach((c, u) => (counts[u + j] += c));
+      row[j - 1].forEach((c, u) => (counts[u] += c));
+      row.push(counts);
+    }
+    prev = row;
+  }
+  return prev[n];
+}
+
+// The complementary error function, with a fractional error under 1.2e-7 (Numerical Recipes, erfcc)
+function erfc(x) {
+  const z = Math.abs(x);
+  const t = 1 / (1 + 0.5 * z);
+  const r =
+    t *
+    Math.exp(
+      -z * z -
+        1.26551223 +
+        t * (1.00002368 + t * (0.37409196 + t * (0.09678418 + t * (-0.18628806 + t * (0.27886807 + t * (-1.13520398 + t * (1.48851587 + t * (-0.82215223 + t * 0.17087277))))))))
+    );
+  return x >= 0 ? r : 2 - r;
+}
+
+const upperTail = (z) => erfc(z / Math.SQRT2) / 2;
+
+const isExact = (m, n, ties) => m <= EXACT_UP_TO && n <= EXACT_UP_TO && ties.length === 0;
+
+// Whether `sample` tends to be greater than `reference`: U counts the pairs where it is (a tie counts
+// half), and p is the probability of a U this large or larger if both came from the same distribution.
+// For a regression of a timing, the sample is the current runs and the reference the baseline.
+export function mannWhitneyGreater(sample, reference) {
+  const m = sample.length;
+  const n = reference.length;
+  const { ranks, ties } = rank([...sample, ...reference]);
+  const rankSum = ranks.slice(0, m).reduce((sum, r) => sum + r, 0);
+  const u = rankSum - (m * (m + 1)) / 2;
+
+  if (isExact(m, n, ties)) {
+    const counts = exactCounts(m, n);
+    const total = counts.reduce((sum, c) => sum + c, 0);
+    const above = counts.slice(Math.ceil(u)).reduce((sum, c) => sum + c, 0);
+    return { u, p: above / total, method: "exact" };
+  }
+
+  const N = m + n;
+  const tieTerm = ties.reduce((sum, t) => sum + t ** 3 - t, 0) / (N * (N - 1));
+  const sigma = Math.sqrt(((m * n) / 12) * (N + 1 - tieTerm));
+  const z = (u - (m * n) / 2 - 0.5) / sigma;
+  return { u, p: upperTail(z), method: "normal" };
+}
+
+const median = (sorted) => {
+  const mid = sorted.length / 2;
+  return sorted.length % 2 ? sorted[Math.floor(mid)] : (sorted[mid - 1] + sorted[mid]) / 2;
+};
+
+// By how much `sample` is greater than `reference`: the median of the differences of every pair, and the
+// 95 % interval between two of those differences, chosen from the distribution of U. With samples too
+// small for 95 %, such as three runs a side, the interval is every difference.
+export function hodgesLehmann(reference, sample) {
+  const m = sample.length;
+  const n = reference.length;
+  const diffs = sample.flatMap((s) => reference.map((r) => s - r)).sort((a, b) => a - b);
+  const { ties } = rank([...sample, ...reference]);
+
+  let k;
+  if (isExact(m, n, ties)) {
+    // The largest k with P(U <= k - 1) at most 2.5 %
+    const counts = exactCounts(m, n);
+    const total = counts.reduce((sum, c) => sum + c, 0);
+    let below = 0;
+    k = 0;
+    while (k < counts.length && (below + counts[k]) / total <= 0.025) below += counts[k++];
+  } else {
+    k = Math.max(0, Math.floor((m * n) / 2 - Z_975 * Math.sqrt((m * n * (m + n + 1)) / 12)));
+  }
+  return { shift: median(diffs), ci: [diffs[k], diffs[m * n - 1 - k]] };
+}