webperf-snippets 1.8.0 → 1.10.0

package/README.md

@@ -70,10 +70,10 @@ JSON output (for piping into `jq` or CI):
 npx webperf-snippets https://example.com --json
 ```
 
-The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait and the version of the CLI.
+The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait, the version of the CLI and the machine: its CPU model, cores, architecture and memory, as the operating system reports them, and a benchmark index, a short CPU test in Node before Chromium starts, which only informs, since on the same machine it moves by up to 40 % with the load.
 
 ```json
-"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.8.0" }
+"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.10.0", "machine": { "cpu": "Apple M4", "cores": 10, "arch": "arm64", "memoryGB": 16, "benchmarkIndex": 2371 } }
 ```
 
 Single snippet:
@@ -136,6 +136,19 @@ Field data (CrUX p75, mobile)
 
 Get a key for the [Chrome UX Report API](https://developer.chrome.com/docs/crux/api) and pass it with `--crux-key` or the `CRUX_API_KEY` environment variable; prefer the variable, because a flag ends up in shell history and CI logs. The section appears in the human and markdown output and under `crux` in the `--json` output.
 
+With a key, the field data also gives what CrUX knows of the LCP when it is an image: the p75 of its sub-parts, TTFB, load delay, load duration and render delay. When the measurement ran LCP-Subparts, with `--snippet LCP-Subparts` or the `core-web-vitals` workflow over 2.5 s, a table puts each one next to its lab value, with a note where one side is at least 1.5 times the other and 100 ms apart: a slower server or network for users, an image the lab finds sooner, slower connections, slower devices, or a lab slower than users. The field sub-parts describe only the loads whose LCP is an image, and the table says which share that is; when the LCP of the lab is not an image, only TTFB and render delay compare. Each p75 is taken on its own, so the sub-parts do not add up to the p75 of LCP. A line gives the navigation types of the users, by share, and points to Back-Forward-Cache when over 5 % of the visits come back without the back/forward cache. The JSON output carries them in `crux.subParts`, `crux.lcpImageShare`, `crux.roundTripTime` and `crux.navigationTypes`.
+
+`--crux-history <weeks>` adds the trend of the field data over the last 1 to 40 weeks, from the [CrUX History API](https://developer.chrome.com/docs/crux/history-api), with the same key. Each week is the p75 of the 28 days before it, so the weeks overlap and the series is smooth. For LCP, CLS and INP the section gives the first and the last week, the change between them and its direction, `stable` within 5 %, and the weeks as a sparkline, where a gap is a week with too little data:
+
+```
+Field data trend (CrUX p75, 40 weeks, mobile, origin-level)
+  LCP   2.88s → 3.10s       ▲ +7.7%   worsening   ▃▃▄▃▃▃▄▆▇██▇▆▄▂▁▂▃▃▄▅▆▆▆▆▆▆
+  CLS   0.2600 → 0.2100     ▼ −19.2%  improving   ▅▆▆▇▇█▇▆▆▅▄▃▃▃▃▃▂▂▁▂▂▂▂▂▂▂
+  INP   253ms → 251ms       ▼ −0.8%   stable      ▇▅▃▂▂▄▆██▇▅▃▂▃▄▆▅▃▂▁▂▃▃▄▅▆
+```
+
+The JSON output carries it in `crux.history`, with the periods, the p75 of each week and the trend of each metric. Like the p75, it uses the page when it has data of its own and its origin otherwise, and says which in `scope`.
+
 - The population follows `--viewport`: `desktop` uses desktop data, `mobile` and `tablet` use phone data. The heading says which one.
 - When the page itself has no data, the CLI looks the origin up instead and labels the section `origin-level`, so it is not mistaken for page-level data. If the origin has none either, the output says `Field data: not available`.
 - A CrUX error (quota, network, no answer within 10 seconds) only skips the section. It never changes the exit code, and the key is never printed.
@@ -158,7 +171,7 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.8.0" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.10.0" }
 }
 ```
 
@@ -190,7 +203,7 @@ One page load is one sample, and lab values vary from one load to the next. `--r
 npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
 ```
 
-The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and the JSON output carries them in `runs`:
+The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and how much the runs agree, and the JSON output carries them in `runs`:
 
 ```json
 "runs": {
@@ -199,12 +212,12 @@ The median run is the one with the median LCP. A workflow without LCP uses the f
   "medianRun": 3,
   "medianBy": "LCP",
   "summary": {
-    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms" }
+    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms", "cv": 11.6, "agreement": "noisy" }
   }
 }
 ```
 
-`medianRun` counts from 1, and `values` keeps the order of the runs. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
+`medianRun` counts from 1, and `values` keeps the order of the runs. `cv` is the coefficient of variation of the runs, their standard deviation over their mean in percent, and `agreement` its band: `consistent` up to 5 %, `variable` up to 10 % and `noisy` above, where one median says little and more runs help. Runs that vary by less than half of the minimum change of `--baseline` (50 ms for LCP, FCP and TTFB, 25 ms for INP, 0.025 for CLS) are `consistent` whatever their CV, since a local TTFB of 3 to 18 ms has a CV of 64 %. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
 
 Measure several pages, on several devices and in several variants, in one command with `--matrix <file>`. The file lists them, and the CLI measures every URL on every device and in every variant, each combination `runs` times, one load after the other:
 
@@ -238,7 +251,7 @@ npx webperf-snippets --matrix matrix.json --budget-lcp 2500
 
 The options of the command line apply to every combination: `--workflow`, `--snippet`, `--wait`, `--interact-script`, `--header`, `--own-domains`, `--storage-state`, `--cpu`, `--network` and the budgets. The file replaces `cpu`, `network` and `storageState` where it sets them, and its `headers` and `ownDomains` add to the ones of the command line, the variant winning on the same header. A matrix file is meant to be versioned, so pass secrets, such as the token of an access proxy, with `--header` and an environment variable; a header of the file whose name looks like a secret raises a warning. A key the CLI does not know is an error, so a typo does not measure without its condition.
 
-The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
+The report is a table with one row per combination and the rated median of each metric, a variable or noisy one followed by the coefficient of variation of its runs (`🟡 2.71s ±14%`), then the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
 
 With CrUX, the CLI asks once for each URL and form factor, so the variants of a device share their field data. With `--report-to`, it posts one report for each combination, the body of one page with `meta.device`, `meta.variant` and `meta.runs`; the headers of the command line and of the file are conditions of the load, and no report carries their values.
 
@@ -257,6 +270,8 @@ npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.jso
 
 LCP, FCP, TTFB, CLS and INP compare when both runs measured them, the median of each with `--runs`; a metric that only one side measured is `not measured`. A metric regresses when it gets worse by more than its threshold, the larger of a share of the baseline and a minimum, and, when the baseline has two runs or more, ends above the slowest of them: a median inside the range of the baseline is noise the baseline already had. It improves under the same rules the other way, and otherwise it is unchanged.
 
+With five runs or more on each side, the range gives way to a test of the runs themselves: a change that beats its threshold also needs a one-sided Mann-Whitney U p-value under 0.05, the chance that runs of an unchanged page would differ this much. A median can move with two slow runs out of five; the test asks whether most runs moved. The JSON of the metric adds `stats`, with `p`, the `shift` (the Hodges-Lehmann estimate, the median of the differences between every run of one side and every run of the other) and its 95 % interval `ci`, and the reports add them to the verdict: `regressed (p = 0.028, +500ms, 95% CI +20ms to +600ms)`, or `unchanged (no evidence: p = 0.35)` for a change over the threshold that the runs do not back. Three runs a side can never reach p < 0.05, so under five the range decides.
+
 | Metric         | Share of the baseline | Minimum |
 | -------------- | --------------------- | ------- |
 | LCP, FCP, TTFB | 10%                   | 100 ms  |
@@ -282,7 +297,7 @@ The report adds the comparison after the usual one, and `--markdown` a table for
 }
 ```
 
-When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
+When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, on another CPU model or with other cores, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
 
 The comparison only informs, and the exit code is the same as without it, unless `--fail-on-regression`, which exits with `1` when a metric regressed or an error is new, and names them on stderr. A baseline that does not exist or is not an output of the CLI, a baseline of a page for a matrix or of a matrix for a page, or a `--threshold` the CLI does not understand exits with code `2`, before the page loads.
 
@@ -325,6 +340,7 @@ Each budget compares the value measured, or the median with `--runs`, and fails
 | `--threshold <metric>=<v>`   | The threshold of a metric for `--baseline`, such as `LCP=15%`, `LCP=200ms` or `CLS=0.1`. Repeat it for more metrics. |
 | `--fail-on-regression`       | Exit `1` when a metric regressed or an error is new against `--baseline`. |
 | `--crux-key <key>`           | Add CrUX field data (p75) next to the measured values. Prefer `CRUX_API_KEY`. |
+| `--crux-history <weeks>`     | Add the trend of the field data over the last 1 to 40 weeks. Needs a CrUX key. |
 | `--report-to <url>`          | POST the results to this `https` URL after the run. No external calls without it. |
 | `--api-key <key>`            | Sent as a Bearer token with `--report-to`. Prefer `PERF_REVIEWS_API_KEY`. |
 | `--budget-lcp <ms>`          | Exit `1` if LCP exceeds this value or was not measured.                                    |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.8.0",
+  "version": "1.10.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",

package/src/baseline.js

@@ -1,6 +1,7 @@
 import { readFileSync } from "node:fs";
 import { blockingErrors } from "./blocking.js";
 import { describeThrottling } from "./throttling.js";
+import { mannWhitneyGreater, hodgesLehmann } from "./stats.js";
 
 // A measurement compared with an earlier one, its baseline: which metrics got worse or better beyond
 // the noise of the lab, which errors appeared or went away, and what differs in how both were
@@ -71,23 +72,40 @@ export function thresholdsWith(overrides) {
 
 const round = (n) => Math.round(n * 10000) / 10000;
 
-// The value of a metric in a page: the median of its runs, with their range when there are two or
-// more, or its one result, as runs.js picks it
+// The value of a metric in a page: the median of its runs, with their range and their values when there
+// are two or more, or its one result, as runs.js picks it
 function measured(payload, metric) {
   const summary = payload.runs?.summary?.[metric];
   if (summary) {
-    const runs = summary.values.filter((value) => value !== null).length;
-    return { value: summary.median, ...(runs >= 2 ? { range: { min: summary.min, max: summary.max } } : {}) };
+    const values = summary.values.filter((value) => value !== null);
+    return { value: summary.median, ...(values.length >= 2 ? { range: { min: summary.min, max: summary.max }, values } : {}) };
   }
   const result = payload.results?.find((r) => r.metric === metric && r.status === "ok" && typeof r.value === "number");
   return result ? { value: result.value } : null;
 }
 
-// A regression must beat the threshold of its metric and, when the baseline has several runs, the
-// worst of them: a median inside the range of the baseline is noise the baseline already had.
-function verdict(base, current, threshold) {
-  if (current.value - base.value > threshold && (!base.range || current.value > base.range.max)) return "regressed";
-  if (base.value - current.value > threshold && (!base.range || current.value < base.range.min)) return "improved";
+// With five runs or more a side, Mann-Whitney U can tell a change from the noise of the runs; with
+// fewer, its p-value cannot go under 0.05 (three a side give 0.05 at best), so the range decides.
+export const RUNS_FOR_STATISTICS = 5;
+const ALPHA = 0.05;
+
+// The evidence of a change: the one-sided p-value in its direction, and the shift with its 95 % interval
+function statsOf(base, current, delta) {
+  if ((base.values?.length ?? 0) < RUNS_FOR_STATISTICS || (current.values?.length ?? 0) < RUNS_FOR_STATISTICS) return undefined;
+  const { p, method } = delta < 0 ? mannWhitneyGreater(base.values, current.values) : mannWhitneyGreater(current.values, base.values);
+  const { shift, ci } = hodgesLehmann(base.values, current.values);
+  return { p: round(p), shift: round(shift), ci: ci.map(round), method };
+}
+
+// A regression must beat the threshold of its metric and, with five runs a side, be unlikely to come
+// from the noise of the runs (p < 0.05). With fewer, when the baseline has several runs, it must beat
+// the worst of them: a median inside the range of the baseline is noise the baseline already had.
+function verdict(base, current, threshold, stats) {
+  const worse = current.value - base.value > threshold;
+  const better = base.value - current.value > threshold;
+  if (stats) return stats.p < ALPHA ? (worse ? "regressed" : better ? "improved" : "unchanged") : "unchanged";
+  if (worse && (!base.range || current.value > base.range.max)) return "regressed";
+  if (better && (!base.range || current.value < base.range.min)) return "improved";
   return "unchanged";
 }
 
@@ -104,13 +122,15 @@ function compareMetrics(base, current, thresholds) {
     const { relative = 0, absolute } = thresholds[metric];
     const threshold = round(Math.max(relative * b.value, absolute));
     const delta = c.value - b.value;
+    const stats = statsOf(b, c, delta);
     metrics[metric] = {
       baseline: b.value,
       current: c.value,
       delta: round(delta),
       deltaPercent: b.value === 0 ? null : Math.round((delta / b.value) * 1000) / 10,
       threshold,
-      verdict: verdict(b, c, threshold),
+      verdict: verdict(b, c, threshold, stats),
+      ...(stats ? { stats } : {}),
     };
   }
   return metrics;
@@ -166,6 +186,14 @@ const pathOf = (url) => {
   }
 };
 
+// Another CPU or other cores give other timings for the same page. A different benchmark index alone
+// does not warn: on the same machine it moves by 40 % with its load.
+function machineWarning(base, current) {
+  if (!base?.cpu || !current?.cpu) return null;
+  if (base.cpu === current.cpu && base.cores === current.cores) return null;
+  return `the baseline was measured on another machine (${base.cpu}, ${base.cores} cores) than this run (${current.cpu}, ${current.cores} cores); timings can differ for that alone`;
+}
+
 // What differs in how both were measured. The comparison goes on: the warnings say how far to trust it.
 function warningsOf(base, current) {
   const warnings = [];
@@ -183,6 +211,8 @@ function warningsOf(base, current) {
   }
   if (runCount(base) < 2) warnings.push("the baseline has 1 run, so only the thresholds apply; measure both with --runs");
   else if (runCount(base) !== runCount(current)) warnings.push(`the baseline has ${runCount(base)} runs, and this run ${runCount(current)}`);
+  const machine = machineWarning(b?.machine, c?.machine);
+  if (machine) warnings.push(machine);
   if (b && c && b.cliVersion !== c.cliVersion) {
     warnings.push(`the baseline was measured with CLI ${b.cliVersion}, and this run with ${c.cliVersion}; the snippets may have changed`);
   }

package/src/bin.js

@@ -10,7 +10,7 @@ import { reportHuman, reportMatrixHuman } from "./reporters/human.js";
 import { reportJson } from "./reporters/json.js";
 import { reportMarkdown, reportMatrixMarkdown } from "./reporters/markdown.js";
 import { cliVersion, reportTo, validateReportUrl } from "./report-to.js";
-import { fetchCrux, formFactorFor } from "./crux.js";
+import { fetchCrux, fetchCruxHistory, formFactorFor } from "./crux.js";
 import { MissingBrowserError } from "./browser.js";
 import { parseHeader } from "./headers.js";
 import { NETWORK_PRESETS, parseCpuFactor } from "./throttling.js";
@@ -20,6 +20,7 @@ import { loadMatrix } from "./matrix.js";
 import { blockingErrors } from "./blocking.js";
 import { MEASURED_BY, checkBudgets, parseBudgets, unmeasuredMetrics, violationsOf } from "./budgets.js";
 import { measurementConditions } from "./conditions.js";
+import { machineOf } from "./machine.js";
 import { isHttpUrl } from "./url.js";
 import { loadBaseline, parseThreshold, thresholdsWith, compareRun, compareMatrix } from "./baseline.js";
 import { describeRegressions, describeMatrixRegressions, matrixComparisonView } from "./reporters/baseline-view.js";
@@ -79,6 +80,7 @@ Options:
                             LCP=200ms or CLS=0.1. Repeat it for more metrics
   --fail-on-regression      Exit 1 when a metric regressed or an error is new against --baseline
   --crux-key <key>          Add CrUX field data (p75) next to the measured values.
+  --crux-history <weeks>    Add the trend of the field data over the last 1 to 40 weeks (needs a CrUX key).
                             Also read from the CRUX_API_KEY environment variable
   --report-to <url>         POST the results to this https URL after the run
                             (without it the CLI makes no external calls)
@@ -174,6 +176,7 @@ async function main() {
         threshold: { type: "string", multiple: true },
         "fail-on-regression": { type: "boolean" },
         "crux-key": { type: "string" },
+        "crux-history": { type: "string" },
         "report-to": { type: "string" },
         "api-key": { type: "string" },
         verbose: { type: "boolean" },
@@ -252,6 +255,19 @@ async function main() {
   }
   if (medianBy != null && runCount == null && !matrixPath) fail("--median-by needs --runs");
 
+  // The field data: the p75 of the last 28 days with a key, and its trend week by week with --crux-history
+  const cruxKey = values["crux-key"] || process.env.CRUX_API_KEY;
+  let cruxWeeks;
+  if (values["crux-history"] !== undefined) {
+    cruxWeeks = /^\d+$/.test(values["crux-history"]) ? Number(values["crux-history"]) : NaN;
+    if (!(cruxWeeks >= 1 && cruxWeeks <= 40)) fail(`--crux-history takes 1 to 40 weeks, not "${values["crux-history"]}"`);
+    if (!cruxKey) fail("--crux-history needs a CrUX key: --crux-key or CRUX_API_KEY");
+  }
+  const fieldDataOf = async (url, formFactor) => {
+    const crux = await fetchCrux({ url, apiKey: cruxKey, formFactor });
+    return cruxWeeks ? { ...crux, history: await fetchCruxHistory({ url, apiKey: cruxKey, formFactor, weeks: cruxWeeks }) } : crux;
+  };
+
   // The baseline is read and checked before the browser opens: a wrong file fails at once
   let baseline;
   let thresholds;
@@ -287,9 +303,11 @@ async function main() {
     const options = { waitMs, headless: !values.headed, interactScript, ...conditions };
     return items ? runSnippets({ ...options, items }) : runMeasurement({ ...options, workflow, rules: RULES });
   };
-  // What the JSON records of how each page was measured, for a later run to compare with
+  // What the JSON records of how each page was measured, for a later run to compare with. The machine is
+  // measured once, before Chromium starts, so its processes do not slow the benchmark down.
+  const machine = machineOf();
   const conditionsFor = (viewport) =>
-    measurementConditions({ viewport, waitMs, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
+    measurementConditions({ viewport, waitMs, machine, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
 
   if (matrixPath) {
     let matrix;
@@ -298,7 +316,7 @@ async function main() {
     } catch (err) {
       fail(err.message);
     }
-    await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds, budgets });
+    await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds, budgets, cruxKey, fieldDataOf });
     return;
   }
 
@@ -313,9 +331,7 @@ async function main() {
   }
   payload.conditions = conditionsFor(viewportName);
 
-  const cruxKey = values["crux-key"] || process.env.CRUX_API_KEY;
-  const crux = cruxKey ? await fetchCrux({ url, apiKey: cruxKey, formFactor: formFactorFor(viewportName) }) : undefined;
-  if (crux) payload.crux = crux;
+  if (cruxKey) payload.crux = await fieldDataOf(url, formFactorFor(viewportName));
 
   if (baseline) {
     payload.baseline = compareRun(baseline.data, payload, { file: baseline.file, thresholds });
@@ -391,7 +407,7 @@ async function measureRuns(measure, count, medianBy, progress) {
 
 // Every combination of the matrix, in order, each one `matrix.runs` times. Each combination reports
 // its median run, and the exit code follows the same rules as one page, for every combination.
-async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds, budgets }) {
+async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds, budgets, cruxKey, fieldDataOf }) {
   for (const warning of matrix.warnings) process.stderr.write(`Warning: ${warning}\n`);
   const viewportOf = new Map(matrix.combinations.map((c) => [c.id, c.viewport]));
 
@@ -411,13 +427,12 @@ async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, repo
   }
 
   // The field data of a URL depends on the form factor, not on the variant: one query for each pair
-  const cruxKey = values["crux-key"] || process.env.CRUX_API_KEY;
   if (cruxKey) {
     const queries = new Map();
     for (const c of combinations.filter((c) => !c.error)) {
       const formFactor = formFactorFor(viewportOf.get(c.id));
       const key = `${c.url} ${formFactor}`;
-      if (!queries.has(key)) queries.set(key, fetchCrux({ url: c.url, apiKey: cruxKey, formFactor }));
+      if (!queries.has(key)) queries.set(key, fieldDataOf(c.url, formFactor));
       const crux = await queries.get(key);
       if (crux) c.crux = crux;
     }

package/src/conditions.js

@@ -2,6 +2,6 @@ import { cliVersion } from "./report-to.js";
 
 // How a measurement was taken, kept in its JSON so that a later run compared with it can tell whether
 // both were measured the same way. A snippet is named by its path, so an alias and its path match.
-export function measurementConditions({ viewport, workflow, snippet, waitMs }) {
-  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion };
+export function measurementConditions({ viewport, workflow, snippet, waitMs, machine }) {
+  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion, ...(machine ? { machine } : {}) };
 }

package/src/crux.js

@@ -1,6 +1,8 @@
 import { rate } from "./ratings.js";
 
 const DEFAULT_ENDPOINT = "https://chromeuxreport.googleapis.com/v1/records:queryRecord";
+// The History API lives next to the daily one; an endpoint given for tests or a proxy points at both
+const historyEndpointOf = (endpoint) => endpoint.replace(/records:queryRecord$/, "records:queryHistoryRecord");
 const CRUX_TIMEOUT_MS = 10000;
 
 const METRICS = [
@@ -27,6 +29,42 @@ function parseMetrics(record) {
   return metrics;
 }
 
+// The sub-parts of the LCP as CrUX reports them, for the loads whose LCP is an image, by the names of
+// the lab sub-parts of LCP-Subparts
+const SUB_PARTS = [
+  { part: "ttfb", label: "TTFB", key: "largest_contentful_paint_image_time_to_first_byte" },
+  { part: "resourceLoadDelay", label: "Load delay", key: "largest_contentful_paint_image_resource_load_delay" },
+  { part: "resourceLoadTime", label: "Load duration", key: "largest_contentful_paint_image_resource_load_duration" },
+  { part: "elementRenderDelay", label: "Render delay", key: "largest_contentful_paint_image_element_render_delay" },
+];
+
+const p75Of = (record, key) => {
+  const value = Number(record?.metrics?.[key]?.percentiles?.p75);
+  return Number.isFinite(value) ? value : undefined;
+};
+
+// What else the record of a page says besides LCP, CLS and INP: the sub-parts of an image LCP, the
+// share of loads whose LCP is an image, the round trip time and the navigation types, by share
+export function parseFieldExtras(record) {
+  const extras = {};
+  const subParts = {};
+  for (const { part, key } of SUB_PARTS) {
+    const value = p75Of(record, key);
+    if (value !== undefined) subParts[part] = value;
+  }
+  if (Object.keys(subParts).length) extras.subParts = subParts;
+  const imageShare = record?.metrics?.largest_contentful_paint_resource_type?.fractions?.image;
+  if (typeof imageShare === "number") extras.lcpImageShare = imageShare;
+  const rtt = p75Of(record, "round_trip_time");
+  if (rtt !== undefined) extras.roundTripTime = rtt;
+  const fractions = record?.metrics?.navigation_types?.fractions;
+  if (fractions) {
+    const sorted = Object.entries(fractions).filter(([, share]) => share > 0).sort((a, b) => b[1] - a[1]);
+    if (sorted.length) extras.navigationTypes = Object.fromEntries(sorted);
+  }
+  return extras;
+}
+
 async function query({ endpoint, apiKey, body, timeoutMs }) {
   const res = await fetch(`${endpoint}?key=${encodeURIComponent(apiKey)}`, {
     method: "POST",
@@ -61,7 +99,78 @@ export async function fetchCrux({
     if (answer.status) return { unavailable: `the CrUX API answered ${answer.status}`, failed: true };
     const metrics = parseMetrics(answer.record);
     if (Object.keys(metrics).length === 0) return { unavailable: "no Core Web Vitals in the CrUX record" };
-    return { scope, formFactor, metrics };
+    return { scope, formFactor, metrics, ...parseFieldExtras(answer.record) };
+  } catch (err) {
+    const reason = err.name === "TimeoutError" ? `no answer within ${timeoutMs}ms` : (err.cause?.code ?? "request failed");
+    return { unavailable: reason, failed: true };
+  }
+}
+
+// A trend compares the first and the last periods with data: each period already covers 28 days, so the
+// series is smooth enough. Within 5 % it is stable; lower is better for every metric.
+const STABLE_WITHIN = 5;
+
+export function trendOf(p75s) {
+  const values = p75s.filter((v) => v !== null);
+  const first = values[0];
+  const last = values.at(-1);
+  if (values.length < 2) return { first, last, change: null, trend: null };
+  const change = first === 0 ? null : Math.round(((last - first) / first) * 1000) / 10;
+  const trend = change === null || Math.abs(change) <= STABLE_WITHIN ? "stable" : change < 0 ? "improving" : "worsening";
+  return { first, last, change, trend };
+}
+
+const BARS = "▁▂▃▄▅▆▇█";
+
+// The series as bars between its lowest and highest values, a space for a period without data
+export function sparkline(values) {
+  const known = values.filter((v) => v !== null);
+  const min = Math.min(...known);
+  const max = Math.max(...known);
+  return values.map((v) => (v === null ? " " : max === min ? BARS[3] : BARS[Math.round(((v - min) / (max - min)) * 7)])).join("");
+}
+
+const dateOf = ({ year, month, day }) => `${year}-${String(month).padStart(2, "0")}-${String(day).padStart(2, "0")}`;
+
+// The record of the History API as periods, oldest first, and the p75s of each Core Web Vital
+export function parseHistory(record) {
+  const periods = (record?.collectionPeriods ?? []).map((p) => ({ start: dateOf(p.firstDate), end: dateOf(p.lastDate) }));
+  const metrics = {};
+  for (const m of METRICS) {
+    const raw = record?.metrics?.[m.key]?.percentilesTimeseries?.p75s;
+    if (!raw) continue;
+    // CLS comes as strings ("0.21"), and a period with too little data as null
+    const p75s = raw.map((v) => (v === null || v === undefined || !Number.isFinite(Number(v)) ? null : Number(v)));
+    if (p75s.every((v) => v === null)) continue;
+    metrics[m.name] = { unit: m.unit, p75s, ...trendOf(p75s) };
+  }
+  return { periods, metrics };
+}
+
+// The p75 of LCP, CLS and INP week by week over the last `weeks` periods (1 to 40), from the History
+// API: the URL, or its origin when the URL has too little traffic. Never throws, and like fetchCrux
+// never puts the key in a message.
+export async function fetchCruxHistory({
+  url,
+  apiKey,
+  weeks,
+  formFactor = "PHONE",
+  endpoint = process.env.WEBPERF_CRUX_ENDPOINT ?? DEFAULT_ENDPOINT,
+  timeoutMs = CRUX_TIMEOUT_MS,
+}) {
+  const history = historyEndpointOf(endpoint);
+  try {
+    let scope = "url";
+    let answer = await query({ endpoint: history, apiKey, body: { url, formFactor, collectionPeriodCount: weeks }, timeoutMs });
+    if (answer.notFound) {
+      scope = "origin";
+      answer = await query({ endpoint: history, apiKey, body: { origin: new URL(url).origin, formFactor, collectionPeriodCount: weeks }, timeoutMs });
+    }
+    if (answer.notFound) return { unavailable: "not in the CrUX dataset" };
+    if (answer.status) return { unavailable: `the CrUX History API answered ${answer.status}`, failed: true };
+    const { periods, metrics } = parseHistory(answer.record);
+    if (Object.keys(metrics).length === 0) return { unavailable: "no Core Web Vitals in the CrUX history" };
+    return { scope, formFactor, weeks: periods.length, periods, metrics };
   } catch (err) {
     const reason = err.name === "TimeoutError" ? `no answer within ${timeoutMs}ms` : (err.cause?.code ?? "request failed");
     return { unavailable: reason, failed: true };
@@ -84,3 +193,79 @@ export function fieldDataView(crux, results) {
   });
   return { title, rows };
 }
+
+// The trend of the field data, shared by both reporters: per metric the first and last p75 of the
+// weeks, the change and its direction, and the series as a sparkline
+export function historyView(history) {
+  const label = FORM_FACTOR_LABEL[history.formFactor] ?? history.formFactor.toLowerCase();
+  const title = `Field data trend (CrUX p75, ${history.weeks} weeks, ${label}${history.scope === "origin" ? ", origin-level" : ""})`;
+  const rows = METRICS.filter((m) => history.metrics[m.name]).map((m) => {
+    const { p75s, first, last, change, trend } = history.metrics[m.name];
+    const changeText = change === null ? "·" : `${change > 0 ? "▲ +" : change < 0 ? "▼ −" : "= "}${Math.abs(change)}%`;
+    return { metric: m.name, unit: m.unit, first, last, change: changeText, trend: trend ?? "·", sparkline: sparkline(p75s) };
+  });
+  return { title, rows };
+}
+
+const share = (fraction) => (fraction < 0.01 ? `${Math.round(fraction * 1000) / 10}%` : `${Math.round(fraction * 100)}%`);
+
+// A sub-part counts as different when one side is at least 1.5 times the other and 100 ms apart: less
+// than that is within the noise of the lab and of a percentile
+const differs = (slow, fast) => slow >= fast * 1.5 && slow - fast >= 100;
+
+const rttNote = (rtt) => (rtt === undefined ? "" : ` (RTT p75 ${rtt} ms)`);
+const FIELD_SLOWER = {
+  ttfb: (rtt) => `real users wait longer for the server: network distance${rttNote(rtt)}, the CDN or cache misses the lab does not see`,
+  resourceLoadDelay: () => 'the lab finds the image sooner: preload it or give it fetchpriority="high", or users run more scripts before it',
+  resourceLoadTime: (rtt) => `slower connections in the field${rttNote(rtt)}: compress or resize the LCP image`,
+  elementRenderDelay: () => "slower devices or more main-thread work in the field",
+};
+const LAB_SLOWER = "the lab is slower than users: check the CPU and network throttling of the lab";
+
+// The sub-parts of the field next to those of LCP-Subparts in the lab, when it ran, with a note where
+// they differ, and what the field sub-parts describe. Null without field sub-parts.
+export function subPartsView(extras, results) {
+  if (!extras.subParts) return null;
+  const labResult = results.find((r) => r.script === "LCP-Subparts" && r.status === "ok");
+  const labParts = labResult?.details?.subParts ?? {};
+  // The field sub-parts are those of image LCPs: a lab LCP without a resource, such as text, has no load
+  // delay or duration to set against them
+  const labImage = !labResult || Boolean(labResult.details?.url);
+  const rows = SUB_PARTS.filter(({ part }) => extras.subParts[part] !== undefined).map(({ part, label }) => {
+    const field = extras.subParts[part];
+    const comparable = labImage || part === "ttfb" || part === "elementRenderDelay";
+    const lab = comparable ? (labParts[part]?.value ?? null) : null;
+    let note = null;
+    if (lab !== null && differs(field, lab)) note = FIELD_SLOWER[part](extras.roundTripTime);
+    else if (lab !== null && differs(lab, field)) note = LAB_SLOWER;
+    return { part, label, field, lab, note };
+  });
+  const caveats = [
+    ...(extras.lcpImageShare !== undefined ? [`The field sub-parts describe the ${share(extras.lcpImageShare)} of loads whose LCP is an image.`] : []),
+    ...(labImage ? [] : ["The LCP of the lab is not an image, so it has no load delay or duration to compare."]),
+    "Each p75 is taken on its own, so the sub-parts do not add up to the p75 of LCP.",
+  ];
+  return { rows, caveats };
+}
+
+const NAVIGATION_LABELS = {
+  navigate: "navigate",
+  navigate_cache: "navigate from cache",
+  reload: "reload",
+  restore: "restore",
+  back_forward: "back/forward",
+  back_forward_cache: "back/forward cache",
+  prerender: "prerender",
+};
+
+// How users arrive at the page, by share, and a pointer to Back-Forward-Cache when over 5 % of the
+// visits come back without the back/forward cache, which would have shown the page at once
+export function navigationView(extras) {
+  if (!extras.navigationTypes) return null;
+  const parts = Object.entries(extras.navigationTypes).map(([type, fraction]) => `${NAVIGATION_LABELS[type] ?? type} ${share(fraction)}`);
+  const missed = extras.navigationTypes.back_forward ?? 0;
+  return {
+    line: `Navigations: ${parts.join(", ")}`,
+    hint: missed > 0.05 ? `${share(missed)} of the visits come back without the back/forward cache: run Back-Forward-Cache to see what blocks it.` : null,
+  };
+}

package/src/machine.js

@@ -0,0 +1,42 @@
+import { arch as osArch, cpus as osCpus, totalmem as osTotalmem } from "node:os";
+
+// The machine a measurement ran on, kept in its conditions, so --baseline can say when two were measured
+// on different machines: timings move with the CPU as much as with the page. The machine is told apart by
+// what the operating system reports of it. The benchmark index only informs: on the same machine it
+// moves by 40 % with its load and with the cores the system picks (performance or efficiency).
+
+const WINDOW_MS = 40;
+const WINDOWS = 5;
+
+// A fixed piece of work: build a string and sort an array, as the benchmark index of Lighthouse builds
+// strings in the page. Its result is returned so the engine cannot drop it.
+function work() {
+  let s = "";
+  for (let i = 0; i < 2000; i++) s += String.fromCharCode(97 + (i % 26));
+  const numbers = Array.from({ length: 200 }, (_, i) => (i * 7919) % 211);
+  numbers.sort((a, b) => a - b);
+  return s.length + numbers[0];
+}
+
+// How many times the work fits in 40 ms, the best of five windows: the first one also warms up the
+// engine, and the best one is the least touched by other work of the machine. In Node, so --cpu, which
+// throttles the page, leaves it alone: it measures the machine. Alone, ten in a row vary by about 6 %.
+export function benchmarkIndex() {
+  let best = 0;
+  let sink = 0;
+  for (let w = 0; w < WINDOWS; w++) {
+    const end = performance.now() + WINDOW_MS;
+    let count = 0;
+    while (performance.now() < end) {
+      sink += work();
+      count++;
+    }
+    best = Math.max(best, count);
+  }
+  return sink > 0 ? best : 0;
+}
+
+export function machineOf({ cpus = osCpus, arch = osArch(), totalmem = osTotalmem, benchmark = benchmarkIndex } = {}) {
+  const list = cpus();
+  return { cpu: list[0]?.model ?? "unknown", cores: list.length, arch, memoryGB: Math.round(totalmem() / 2 ** 30), benchmarkIndex: benchmark() };
+}

package/src/mcp/server.js

@@ -7,7 +7,7 @@ import { isTrackingSnippet } from "../tracking.js";
 import { runSnippets, runMeasurement, evaluateItems, collectTrackingData, VIEWPORT_PRESETS } from "../runner.js";
 import { WORKFLOWS } from "../workflows/index.js";
 import { RULES } from "../decision-tree.js";
-import { fetchCrux } from "../crux.js";
+import { fetchCrux, fetchCruxHistory } from "../crux.js";
 import { runInteractions, isInputStep, settle } from "../interactions.js";
 import { createSessions, SessionError } from "./sessions.js";
 import { launchChromium, isMissingBrowser, MISSING_BROWSER } from "../browser.js";
@@ -295,10 +295,11 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
     {
       title: "Get CrUX field data",
       description:
-        "Returns the p75 of LCP, CLS and INP that real Chrome users had on a URL over the last 28 days, from the Chrome UX Report, with their rating. Falls back to the origin when the URL has no data of its own. Compare it with the lab values of run_snippet: field data is what users get.",
+        "Returns the p75 of LCP, CLS and INP that real Chrome users had on a URL over the last 28 days, from the Chrome UX Report, with their rating. Falls back to the origin when the URL has no data of its own. Compare it with the lab values of run_snippet: field data is what users get. With weeks, it adds the trend: the p75 of each week, each the 28 days before it, with the change from the first week to the last.",
       inputSchema: z.object({
         url: HttpUrl,
         formFactor: z.enum(Object.keys(CRUX_FORM_FACTORS)).default("phone").describe("The devices the data describes"),
+        weeks: z.number().int().min(1).max(40).optional().describe("Add the trend of the last 1 to 40 weeks"),
       }),
       outputSchema: z.object({
         url: z.string(),
@@ -309,21 +310,38 @@ export function createServer({ launchBrowser = () => launchChromium(), maxPages,
           .optional()
           .describe("By metric name: LCP, CLS, INP"),
         unavailable: z.string().optional().describe("Why there is no field data, such as a page with too little traffic"),
+        subParts: z
+          .record(z.string(), z.number())
+          .optional()
+          .describe("The p75 of the sub-parts of an image LCP: ttfb, resourceLoadDelay, resourceLoadTime, elementRenderDelay"),
+        lcpImageShare: z.number().optional().describe("The share of loads whose LCP is an image, which the sub-parts describe"),
+        roundTripTime: z.number().optional().describe("The p75 of the round trip time of the users, in ms"),
+        navigationTypes: z.record(z.string(), z.number()).optional().describe("The share of each navigation type, such as navigate, reload or back_forward_cache"),
+        history: z
+          .looseObject({})
+          .meta({ additionalProperties: true })
+          .optional()
+          .describe("With weeks: scope, weeks, periods, and per metric its p75s, first, last, change in percent and trend"),
       }),
       annotations: { readOnlyHint: true, openWorldHint: true },
     },
-    async ({ url, formFactor }) => {
+    async ({ url, formFactor, weeks }) => {
       const apiKey = env.CRUX_API_KEY;
       if (!apiKey) return toolError(CRUX_SETUP);
-      const crux = await fetchCrux({
+      const options = {
         url,
         apiKey,
         formFactor: CRUX_FORM_FACTORS[formFactor],
         ...(env.WEBPERF_CRUX_ENDPOINT ? { endpoint: env.WEBPERF_CRUX_ENDPOINT } : {}),
-      });
+      };
+      const crux = await fetchCrux(options);
       if (crux.failed) return toolError(`Could not read the CrUX data: ${crux.unavailable}.`);
-      if (crux.unavailable) return toolResult({ url, formFactor, unavailable: crux.unavailable });
-      return toolResult({ url, formFactor, scope: crux.scope, metrics: crux.metrics });
+      const history = weeks ? await fetchCruxHistory({ ...options, weeks }) : undefined;
+      // A history that failed says why in its own unavailable, next to the p75 that did come
+      const trend = history ? { history: history.failed ? { unavailable: history.unavailable } : history } : {};
+      if (crux.unavailable) return toolResult({ url, formFactor, unavailable: crux.unavailable, ...trend });
+      const { scope, metrics, subParts, lcpImageShare, roundTripTime, navigationTypes } = crux;
+      return toolResult({ url, formFactor, scope, metrics, subParts, lcpImageShare, roundTripTime, navigationTypes, ...trend });
     }
   );
 

package/src/reporters/baseline-view.js

@@ -15,9 +15,22 @@ function describeChange(metric, { delta, deltaPercent }) {
   return `${arrow(delta)} ${ms}${percent}`;
 }
 
+// Three decimals where the verdict is decided, two above it: "0.028", "0.35"
+const pValue = (p) => (p < 0.001 ? "< 0.001" : p < 0.05 ? p.toFixed(3) : p.toFixed(2));
+const amountOf = (metric, n) => (metric === "CLS" ? signed(n, Math.abs(n).toFixed(4)) : signed(n, `${Math.abs(Math.round(n))}ms`));
+
+// The evidence of a change measured with five runs a side: its p-value, its shift and the 95 % interval
+// of the shift; for a change over the threshold that stayed unchanged, that the runs gave no evidence
+function evidenceOf(metric, m) {
+  const { p, shift, ci } = m.stats;
+  if (m.verdict === "unchanged") return Math.abs(m.delta) > m.threshold ? `no evidence: p = ${pValue(p)}` : null;
+  return `p = ${pValue(p)}, ${amountOf(metric, shift)}, 95% CI ${amountOf(metric, ci[0])} to ${amountOf(metric, ci[1])}`;
+}
+
 // An unchanged metric that moved stayed under its threshold, or beat it inside the range of the runs
 // of the baseline: noise the baseline already had
 function noteOf(metric, m) {
+  if (m.stats) return evidenceOf(metric, m) ?? (m.verdict === "unchanged" && m.delta !== 0 ? `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}` : null);
   if (m.verdict !== "unchanged" || m.delta === 0) return null;
   if (Math.abs(m.delta) > m.threshold) return "inside the range of the baseline";
   return `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}`;

package/src/reporters/human.js

@@ -1,9 +1,9 @@
 import { styleText } from "node:util";
 import { describeResult } from "../view.js";
 import { isSummaryResult, summaryNumbers } from "./summary-view.js";
-import { fieldDataView } from "../crux.js";
+import { fieldDataView, historyView, subPartsView, navigationView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
-import { describeRuns } from "../runs.js";
+import { describeRuns, describeAgreement } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
 import { baselineView } from "./baseline-view.js";
 import { budgetRows, matrixBudgetFailures } from "./budgets-view.js";
@@ -197,7 +197,8 @@ function renderResult(r, verbose) {
 }
 
 function renderFieldData(crux, results) {
-  if (crux.unavailable) return [styleText("dim", `Field data: not available (${crux.unavailable})`)];
+  const trend = crux.history ? ["", ...renderFieldTrend(crux.history)] : [];
+  if (crux.unavailable) return [styleText("dim", `Field data: not available (${crux.unavailable})`), ...trend];
   const { title, rows } = fieldDataView(crux, results);
   const lines = [styleText("bold", title)];
   for (const row of rows) {
@@ -207,15 +208,50 @@ function renderFieldData(crux, results) {
       `  ${icon} ${pad(row.metric, 6)} ${pad(paint(row.rating, formatValue(row.p75, row.unit)), 10)} ${pad(styleText("dim", row.rating), 20)} ${styleText("dim", `(synthetic: ${synthetic})`)}`
     );
   }
-  return lines;
+  return [...lines, ...renderFieldSubParts(crux, results), ...renderNavigations(crux), ...trend];
+}
+
+// "TTFB  1.53s  lab 400ms  real users wait longer for the server…"
+function renderFieldSubParts(crux, results) {
+  const view = subPartsView(crux, results);
+  if (!view) return [];
+  return [
+    "",
+    styleText("bold", "LCP sub-parts (CrUX p75 against the lab)"),
+    ...view.rows.map(
+      (row) => `  ${pad(row.label, 14)}${pad(formatValue(row.field, "ms"), 10)}${pad(`lab ${row.lab === null ? "n/a" : formatValue(row.lab, "ms")}`, 12)}${row.note ?? ""}`.trimEnd()
+    ),
+    ...view.caveats.map((caveat) => styleText("dim", `  ${caveat}`)),
+  ];
+}
+
+function renderNavigations(crux) {
+  const view = navigationView(crux);
+  if (!view) return [];
+  return ["", styleText("dim", view.line), ...(view.hint ? [`  ${view.hint}`] : [])];
+}
+
+// "LCP  2.88s → 3.10s  ▲ +7.7%  worsening  ▁▅█"
+function renderFieldTrend(history) {
+  if (history.unavailable) return [styleText("dim", `Field data trend: not available (${history.unavailable})`)];
+  const { title, rows } = historyView(history);
+  const style = { improving: "green", worsening: "red" };
+  return [
+    styleText("bold", title),
+    ...rows.map(
+      (row) =>
+        `  ${pad(row.metric, 6)}${pad(`${formatValue(row.first, row.unit)} → ${formatValue(row.last, row.unit)}`, 20)}${pad(row.change, 10)}${pad(style[row.trend] ? styleText(style[row.trend], row.trend) : row.trend, 12)}${row.sparkline}`
+    ),
+  ];
 }
 
-// The runs line, and the median, min and max of each metric across the runs
+// The runs line, and the median, min and max of each metric across the runs, with how much they agree
 function renderRuns(runs) {
   const lines = [styleText("dim", describeRuns(runs))];
-  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
-    const cell = (label, value) => pad(`${label} ${formatValue(value, unit)}`, 16);
-    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max).trimEnd()}`));
+  for (const [metric, value] of Object.entries(runs.summary)) {
+    const { median, min, max, unit } = value;
+    const cell = (label, v) => pad(`${label} ${formatValue(v, unit)}`, 16);
+    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max)}${describeAgreement(value)}`.trimEnd()));
   }
   return lines;
 }
@@ -274,7 +310,7 @@ export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, th
 // One row per combination with the rated median of each metric, and below it the errors that decide
 // the exit code. With verbose, the report of the median run of each combination follows.
 export function reportMatrixHuman(report) {
-  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison, legend } = matrixView(report);
   const columns = [["name", "Page"], ["device", "Device"], ...(hasVariants ? [["variant", "Variant"]] : [])];
   const widths = columns.map(([key, label]) => Math.max(...report.combinations.map((c) => c[key].length), label.length) + 3);
   const identity = (values) => values.map((value, i) => pad(value, widths[i])).join("");
@@ -292,11 +328,13 @@ export function reportMatrixHuman(report) {
       continue;
     }
     const drawn = cells.map((cell, i) => {
-      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}`), 14) : pad("·", 14);
+      const width = legend ? 18 : 14;
+      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}${cell.spread ? ` ${cell.spread}` : ""}`), width) : pad("·", width);
       return value + (changes ? pad(styleText(VERDICT_STYLE[changes[i].verdict], changes[i].text), 12) : "");
     });
     lines.push(row + drawn.join("").trimEnd());
   }
+  if (legend) lines.push(styleText("dim", `  ${legend}`));
 
   if (errors.length) {
     lines.push("");

package/src/reporters/markdown.js

@@ -1,8 +1,8 @@
-import { fieldDataView } from "../crux.js";
+import { fieldDataView, historyView, subPartsView, navigationView } from "../crux.js";
 import { describeResult } from "../view.js";
 import { isSummaryResult, summaryNumbers } from "./summary-view.js";
 import { describeThrottling } from "../throttling.js";
-import { describeRuns } from "../runs.js";
+import { describeRuns, describeAgreement } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
 import { baselineView } from "./baseline-view.js";
 import { budgetRows, matrixBudgetFailures } from "./budgets-view.js";
@@ -102,7 +102,8 @@ function renderSection(title, results) {
 }
 
 function renderFieldData(crux, results) {
-  if (crux.unavailable) return [`> Field data: not available (${crux.unavailable})`];
+  const trend = crux.history ? ["", ...renderFieldTrend(crux.history)] : [];
+  if (crux.unavailable) return [`> Field data: not available (${crux.unavailable})`, ...trend];
   const { title, rows } = fieldDataView(crux, results);
   return [
     `### ${title}`,
@@ -112,14 +113,49 @@ function renderFieldData(crux, results) {
       const synthetic = row.synthetic == null ? "n/a" : formatValue(row.synthetic, row.unit);
       return `| ${row.metric} | ${formatValue(row.p75, row.unit)} | ${RATING_ICON[row.rating] ?? "·"} ${row.rating} | ${synthetic} |`;
     }),
+    ...renderFieldSubParts(crux, results),
+    ...renderNavigations(crux),
+    ...trend,
   ];
 }
 
-// The median, min and max of each metric across the runs
+function renderFieldSubParts(crux, results) {
+  const view = subPartsView(crux, results);
+  if (!view) return [];
+  return [
+    "",
+    "### LCP sub-parts (CrUX p75 against the lab)",
+    "| Sub-part | Field p75 | Lab | Note |",
+    "|----------|-----------|-----|------|",
+    ...view.rows.map((row) => `| ${row.label} | ${formatValue(row.field, "ms")} | ${row.lab === null ? "n/a" : formatValue(row.lab, "ms")} | ${row.note ?? ""} |`),
+    "",
+    ...view.caveats.map((caveat) => `> ${caveat}`),
+  ];
+}
+
+function renderNavigations(crux) {
+  const view = navigationView(crux);
+  if (!view) return [];
+  return ["", `> ${view.line}`, ...(view.hint ? [">", `> ${view.hint}`] : [])];
+}
+
+function renderFieldTrend(history) {
+  if (history.unavailable) return [`> Field data trend: not available (${history.unavailable})`];
+  const { title, rows } = historyView(history);
+  return [
+    `### ${title}`,
+    "| Metric | First | Last | Change | Trend | Weeks |",
+    "|--------|-------|------|--------|-------|-------|",
+    ...rows.map((row) => `| ${row.metric} | ${formatValue(row.first, row.unit)} | ${formatValue(row.last, row.unit)} | ${row.change} | ${row.trend} | ${row.sparkline} |`),
+  ];
+}
+
+// The median, min and max of each metric across the runs, and how much they agree
 function renderRuns(runs) {
-  const lines = ["| Metric | Median | Min | Max |", "|--------|--------|-----|-----|"];
-  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
-    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} |`);
+  const lines = ["| Metric | Median | Min | Max | Runs |", "|--------|--------|-----|-----|------|"];
+  for (const [metric, value] of Object.entries(runs.summary)) {
+    const { median, min, max, unit } = value;
+    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} | ${describeAgreement(value) || "·"} |`);
   }
   return lines;
 }
@@ -196,7 +232,7 @@ export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttli
 // report of the median run of each combination folded into a details block, to keep a pull request
 // comment readable.
 export function reportMatrixMarkdown(report) {
-  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison, legend } = matrixView(report);
   // With --baseline, a change column follows each metric
   const metricHeadings = metrics.flatMap((m) => (comparison ? [m, "Δ"] : [m]));
   const headings = ["Page", "Device", ...(hasVariants ? ["Variant"] : []), ...metricHeadings];
@@ -208,12 +244,13 @@ export function reportMatrixMarkdown(report) {
   for (const { combination: c, cells, changes } of rows) {
     const drawn = cells
       ? cells.flatMap((cell, i) => {
-          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}` : "·";
+          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}${cell.spread ? ` ${cell.spread}` : ""}` : "·";
           return changes ? [value, changes[i].text] : [value];
         })
       : ["❌ every run failed", ...metricHeadings.slice(1).map(() => "")];
     lines.push(row([c.name, c.device, ...(hasVariants ? [c.variant] : []), ...drawn]));
   }
+  if (legend) lines.push("", `> ${legend}`);
 
   if (errors.length) {
     lines.push("", "### Errors", ...errors.map((e) => `- ${e}`));

package/src/reporters/matrix-view.js

@@ -6,6 +6,11 @@ import { changeCell, matrixComparisonView } from "./baseline-view.js";
 // The order of the metric columns of the matrix table
 const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
 
+// A median whose runs are variable or noisy carries their coefficient of variation in its cell; the
+// others carry nothing, so the table stays narrow and the marks stand out
+const varies = (value) => value?.agreement === "variable" || value?.agreement === "noisy";
+const LEGEND = "±: the coefficient of variation of the runs of a variable or noisy median";
+
 // What the human and markdown reports of a matrix show, so they only differ in how they draw it:
 // the title, the metric columns, one row per combination with the rated median of each metric, and
 // the errors that decide the exit code. With --baseline, each row also has the change of each metric
@@ -31,10 +36,13 @@ export function matrixView({ matrix, runs, combinations, baseline }) {
         ? null
         : metrics.map((metric) => {
             const value = c.runs.summary[metric];
-            return value ? { median: value.median, unit: value.unit, rating: rate(metric, value.median) } : null;
+            return value
+              ? { median: value.median, unit: value.unit, rating: rate(metric, value.median), ...(varies(value) ? { spread: `±${Math.round(value.cv)}%` } : {}) }
+              : null;
           }),
     })),
     ...(baseline ? { comparison: matrixComparisonView(baseline, combinations) } : {}),
+    ...(measured.some((c) => metrics.some((metric) => varies(c.runs.summary[metric]))) ? { legend: LEGEND } : {}),
     errors: combinations.flatMap((c) =>
       c.error ? [`${c.id}: ${c.error}`] : blockingErrors(c.results).map((e) => `${c.id}: ${e.id}: ${e.message}`)
     ),

package/src/runs.js

@@ -1,3 +1,5 @@
+import { DEFAULT_THRESHOLDS } from "./baseline.js";
+
 // Several runs of the same measurement, reduced to one: the median run, as WebPageTest reports it.
 // Every result of the report comes from that run, so they agree with each other (the sub-parts of
 // the LCP add up to its LCP), and a summary gives the median, min and max of each metric.
@@ -24,6 +26,24 @@ const measures = (runs, metric) => runs.some((run) => typeof valueOf(run, metric
 // the two middle ones, so the median is always a run that happened, never an average of two.
 const middle = (sorted) => sorted[Math.floor((sorted.length - 1) / 2)];
 
+// How much the runs of a metric agree: their coefficient of variation, the sample standard deviation
+// over the mean, in percent, and its band. Up to 5 % the runs agree; above 10 % one median says little.
+// None for one run, or for runs whose mean is 0, such as a CLS of 0 in every run. Runs that vary by
+// less than half of the change --baseline counts agree whatever their CV: a local TTFB of 3 to 18 ms
+// has a CV of 64 % and a standard deviation of 6 ms.
+function agreementOf(values, metric) {
+  if (values.length < 2) return {};
+  const mean = values.reduce((sum, v) => sum + v, 0) / values.length;
+  if (mean === 0) return {};
+  const sd = Math.sqrt(values.reduce((sum, v) => sum + (v - mean) ** 2, 0) / (values.length - 1));
+  const cv = Math.round((sd / mean) * 1000) / 10;
+  if (sd < DEFAULT_THRESHOLDS[metric].absolute / 2) return { cv, agreement: "consistent" };
+  return { cv, agreement: cv <= 5 ? "consistent" : cv <= 10 ? "variable" : "noisy" };
+}
+
+// "noisy (CV 14.4%)", for the reports
+export const describeAgreement = ({ cv, agreement }) => (cv === undefined ? "" : `${agreement} (CV ${cv}%)`);
+
 // "3 runs, median by LCP: run 2 (1 failed)", for the header of the reports
 export function describeRuns({ count, failed, medianRun, medianBy, medianByRequested }) {
   const by = medianByRequested ? `${medianBy} (no run measured ${medianByRequested})` : medianBy;
@@ -48,7 +68,7 @@ export function summarizeRuns(runs, { medianBy: requested } = {}) {
     const values = runs.map((run) => (run.error ? null : (valueOf(run, metric) ?? null)));
     const sorted = values.filter((value) => value !== null).sort((a, b) => a - b);
     const unit = completed.flatMap((run) => run.results).find((r) => r.metric === metric && r.unit)?.unit;
-    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit };
+    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit, ...agreementOf(sorted, metric) };
   }
 
   return {

package/src/stats.js

@@ -0,0 +1,108 @@
+// Statistics for comparing the runs of two measurements, without a browser: whether one sample tends to
+// be greater than the other (Mann-Whitney U, one-sided) and by how much (the Hodges-Lehmann shift with its
+// 95 % interval). Both work on ranks, so one slow run moves them less than it moves a mean.
+
+const EXACT_UP_TO = 16;
+const Z_975 = 1.959963984540054;
+
+// The ranks of the pooled values, with the mean rank for ties, and the sizes of the groups of ties
+function rank(values) {
+  const order = values.map((value, i) => ({ value, i })).sort((a, b) => a.value - b.value);
+  const ranks = new Array(values.length);
+  const ties = [];
+  for (let start = 0; start < order.length; ) {
+    let end = start;
+    while (end + 1 < order.length && order[end + 1].value === order[start].value) end++;
+    for (let k = start; k <= end; k++) ranks[order[k].i] = (start + end) / 2 + 1;
+    if (end > start) ties.push(end - start + 1);
+    start = end + 1;
+  }
+  return { ranks, ties };
+}
+
+// How many splits of m + n distinct values give each U, from U = 0 to m * n. A split with the largest
+// value in the first sample has U of the split without it plus n; otherwise, U of the split without it.
+function exactCounts(m, n) {
+  let prev = Array.from({ length: n + 1 }, () => [1]);
+  for (let i = 1; i <= m; i++) {
+    const row = [new Array(1).fill(1)];
+    for (let j = 1; j <= n; j++) {
+      const counts = new Array(i * j + 1).fill(0);
+      prev[j].forEach((c, u) => (counts[u + j] += c));
+      row[j - 1].forEach((c, u) => (counts[u] += c));
+      row.push(counts);
+    }
+    prev = row;
+  }
+  return prev[n];
+}
+
+// The complementary error function, with a fractional error under 1.2e-7 (Numerical Recipes, erfcc)
+function erfc(x) {
+  const z = Math.abs(x);
+  const t = 1 / (1 + 0.5 * z);
+  const r =
+    t *
+    Math.exp(
+      -z * z -
+        1.26551223 +
+        t * (1.00002368 + t * (0.37409196 + t * (0.09678418 + t * (-0.18628806 + t * (0.27886807 + t * (-1.13520398 + t * (1.48851587 + t * (-0.82215223 + t * 0.17087277))))))))
+    );
+  return x >= 0 ? r : 2 - r;
+}
+
+const upperTail = (z) => erfc(z / Math.SQRT2) / 2;
+
+const isExact = (m, n, ties) => m <= EXACT_UP_TO && n <= EXACT_UP_TO && ties.length === 0;
+
+// Whether `sample` tends to be greater than `reference`: U counts the pairs where it is (a tie counts
+// half), and p is the probability of a U this large or larger if both came from the same distribution.
+// For a regression of a timing, the sample is the current runs and the reference the baseline.
+export function mannWhitneyGreater(sample, reference) {
+  const m = sample.length;
+  const n = reference.length;
+  const { ranks, ties } = rank([...sample, ...reference]);
+  const rankSum = ranks.slice(0, m).reduce((sum, r) => sum + r, 0);
+  const u = rankSum - (m * (m + 1)) / 2;
+
+  if (isExact(m, n, ties)) {
+    const counts = exactCounts(m, n);
+    const total = counts.reduce((sum, c) => sum + c, 0);
+    const above = counts.slice(Math.ceil(u)).reduce((sum, c) => sum + c, 0);
+    return { u, p: above / total, method: "exact" };
+  }
+
+  const N = m + n;
+  const tieTerm = ties.reduce((sum, t) => sum + t ** 3 - t, 0) / (N * (N - 1));
+  const sigma = Math.sqrt(((m * n) / 12) * (N + 1 - tieTerm));
+  const z = (u - (m * n) / 2 - 0.5) / sigma;
+  return { u, p: upperTail(z), method: "normal" };
+}
+
+const median = (sorted) => {
+  const mid = sorted.length / 2;
+  return sorted.length % 2 ? sorted[Math.floor(mid)] : (sorted[mid - 1] + sorted[mid]) / 2;
+};
+
+// By how much `sample` is greater than `reference`: the median of the differences of every pair, and the
+// 95 % interval between two of those differences, chosen from the distribution of U. With samples too
+// small for 95 %, such as three runs a side, the interval is every difference.
+export function hodgesLehmann(reference, sample) {
+  const m = sample.length;
+  const n = reference.length;
+  const diffs = sample.flatMap((s) => reference.map((r) => s - r)).sort((a, b) => a - b);
+  const { ties } = rank([...sample, ...reference]);
+
+  let k;
+  if (isExact(m, n, ties)) {
+    // The largest k with P(U <= k - 1) at most 2.5 %
+    const counts = exactCounts(m, n);
+    const total = counts.reduce((sum, c) => sum + c, 0);
+    let below = 0;
+    k = 0;
+    while (k < counts.length && (below + counts[k]) / total <= 0.025) below += counts[k++];
+  } else {
+    k = Math.max(0, Math.floor((m * n) / 2 - Z_975 * Math.sqrt((m * n * (m + n + 1)) / 12)));
+  }
+  return { shift: median(diffs), ci: [diffs[k], diffs[m * n - 1 - k]] };
+}