webperf-snippets 1.2.0 → 1.4.0

package/README.md

@@ -70,6 +70,12 @@ JSON output (for piping into `jq` or CI):
 npx webperf-snippets https://example.com --json
 ```
 
+The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait and the version of the CLI.
+
+```json
+"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.4.0" }
+```
+
 Single snippet:
 
 ```bash
@@ -152,11 +158,11 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.2.0" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.4.0" }
 }
 ```
 
-For a `--snippet` run, `workflow` is `null` and a `snippet` field carries the name. With `--cpu` or `--network`, `meta` carries the conditions in `throttling`. `results` is the same list the `--json` output contains, so it can include URLs and details of the measured page; send it only to an endpoint you trust.
+For a `--snippet` run, `workflow` is `null` and a `snippet` field carries the name. With `--cpu` or `--network`, `meta` carries the conditions in `throttling`, and with `--runs`, the summary of the runs in `runs`. `results` is the same list the `--json` output contains, so it can include URLs and details of the measured page; send it only to an endpoint you trust.
 
 - The URL must be `https`. `http` is accepted only for `localhost`, so a key is never sent in clear text.
 - `--api-key` is sent as `Authorization: Bearer <key>`. Prefer the `PERF_REVIEWS_API_KEY` environment variable, because a flag ends up in shell history and CI logs. The key is never printed.
@@ -178,6 +184,116 @@ npx webperf-snippets https://example.com --cpu 4 --network slow-4g
 
 Chrome adds the latency to the body of each response, not to its headers. The values read from `responseStart`, such as the TTFB and its sub-parts, do not include it; the download of each resource, FCP, LCP and the load event do. The report names the conditions under its header, and the JSON output carries them in `throttling`, such as `{ "cpu": 4, "network": "slow-4g" }`. A CPU factor below `1` or an unknown preset exits with code `2`.
 
+One page load is one sample, and lab values vary from one load to the next. `--runs <n>` measures the page n times, from 1 to 9, one after the other, and reports the median run, as WebPageTest does: every result comes from that run, so they agree with each other (the sub-parts of the LCP add up to its LCP):
+
+```bash
+npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
+```
+
+The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and the JSON output carries them in `runs`:
+
+```json
+"runs": {
+  "count": 5,
+  "failed": 0,
+  "medianRun": 3,
+  "medianBy": "LCP",
+  "summary": {
+    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms" }
+  }
+}
+```
+
+`medianRun` counts from 1, and `values` keeps the order of the runs. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
+
+Measure several pages, on several devices and in several variants, in one command with `--matrix <file>`. The file lists them, and the CLI measures every URL on every device and in every variant, each combination `runs` times, one load after the other:
+
+```json
+{
+  "urls": {
+    "home": "https://example.com/",
+    "article": "https://example.com/news/some-article"
+  },
+  "devices": {
+    "mobile": { "viewport": "mobile", "cpu": 4, "network": "slow-4g" },
+    "high-end": { "viewport": "mobile", "cpu": 1, "network": { "downloadKbps": 50000, "uploadKbps": 10000, "latencyMs": 20 } },
+    "desktop": { "viewport": "desktop" }
+  },
+  "variants": {
+    "no-consent": {},
+    "consent": { "storageState": "states/consent.json" }
+  },
+  "runs": 3
+}
+```
+
+```bash
+npx webperf-snippets --matrix matrix.json --budget-lcp 2500
+```
+
+- `urls`: names and URLs. An array of URLs also works, and names each one by its path and query (`/`, `/news/`), with the host when the URLs have more than one.
+- `devices`: the `viewport` preset, the `cpu` factor and the `network`, a preset (`fast-4g`, `slow-4g`, `3g`) or a network of its own in kilobits per second and milliseconds, for the ones DevTools has no preset for, such as 5G. An array of viewport presets (`["mobile", "desktop"]`) is a shortcut for devices without throttling. Without `devices`, the matrix measures a mid-range phone, with CPU 4x and Slow 4G, as Lighthouse does.
+- `variants`: the conditions of each variant, `storageState` (relative to the file), `headers` and `ownDomains`. Without `variants`, the combinations are the pages on the devices: the report has no Variant column, and the `id` of a combination is its page and device (`/news/ · mobile`).
+- `runs` (1 to 9) and `medianBy`, which `--runs` and `--median-by` replace.
+
+The options of the command line apply to every combination: `--workflow`, `--snippet`, `--wait`, `--interact-script`, `--header`, `--own-domains`, `--storage-state`, `--cpu`, `--network` and the budgets. The file replaces `cpu`, `network` and `storageState` where it sets them, and its `headers` and `ownDomains` add to the ones of the command line, the variant winning on the same header. A matrix file is meant to be versioned, so pass secrets, such as the token of an access proxy, with `--header` and an environment variable; a header of the file whose name looks like a secret raises a warning. A key the CLI does not know is an error, so a typo does not measure without its condition.
+
+The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
+
+With CrUX, the CLI asks once for each URL and form factor, so the variants of a device share their field data. With `--report-to`, it posts one report for each combination, the body of one page with `meta.device`, `meta.variant` and `meta.runs`; the headers of the command line and of the file are conditions of the load, and no report carries their values.
+
+Compare a run with an earlier one with `--baseline <file>`, the `--json` output of that run. The comparison says which metrics got worse or better beyond the noise of the lab, which errors are new or fixed, and what differs in how both runs were measured:
+
+```bash
+# On main, once: the baseline
+npx webperf-snippets https://example.com --runs 5 --json > baseline.json
+
+# On the pull request: compare and report
+npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.json --markdown
+
+# Fail when the page gets worse
+npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.json --fail-on-regression
+```
+
+LCP, FCP, TTFB, CLS and INP compare when both runs measured them, the median of each with `--runs`; a metric that only one side measured is `not measured`. A metric regresses when it gets worse by more than its threshold, the larger of a share of the baseline and a minimum, and, when the baseline has two runs or more, ends above the slowest of them: a median inside the range of the baseline is noise the baseline already had. It improves under the same rules the other way, and otherwise it is unchanged.
+
+| Metric         | Share of the baseline | Minimum |
+| -------------- | --------------------- | ------- |
+| LCP, FCP, TTFB | 10%                   | 100 ms  |
+| INP            | 10%                   | 50 ms   |
+| CLS            | none                  | 0.05    |
+
+`--threshold` changes one part of a threshold and keeps the other: `--threshold LCP=15%` the share, `--threshold LCP=200ms` the minimum, and `--threshold CLS=0.1` the only part of CLS. Repeat it for more metrics.
+
+The errors that compare are the ones that decide the exit code: the issues of severity `error` of the steps of the workflow and the snippets that failed. Two errors are the same when they share a code or, without one, when their messages differ only in their figures, so "Render-blocking CSS (69.21 KB)…" and "Render-blocking CSS (71.80 KB)…" are one error. An error that only this run has is new, and one that only the baseline had is fixed. Warnings and information do not compare, since they change too easily.
+
+The report adds the comparison after the usual one, and `--markdown` a table for the pull request comment. The JSON output carries it in `baseline`:
+
+```json
+"baseline": {
+  "file": "baseline.json",
+  "url": "https://example.com/",
+  "runs": { "count": 5, "medianBy": "LCP" },
+  "metrics": {
+    "LCP": { "baseline": 2100, "current": 2480, "delta": 380, "deltaPercent": 18.1, "threshold": 210, "verdict": "regressed" }
+  },
+  "errors": { "new": [{ "id": "render-blocking", "message": "1 render-blocking resource(s) delay rendering until 900ms" }], "fixed": [] },
+  "warnings": []
+}
+```
+
+When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
+
+The comparison only informs, and the exit code is the same as without it, unless `--fail-on-regression`, which exits with `1` when a metric regressed or an error is new, and names them on stderr. A baseline that does not exist or is not an output of the CLI, a baseline of a page for a matrix or of a matrix for a page, or a `--threshold` the CLI does not understand exits with code `2`, before the page loads.
+
+A matrix compares with the `--json` output of an earlier matrix, each combination with the one of the same `id`:
+
+```bash
+npx webperf-snippets --matrix matrix.json --runs 3 --baseline last-week.json
+```
+
+The table adds a change column after each metric: the change when it counts (`▲ +380ms`, `▼ −90ms`), `=` when it is unchanged, and `new` for a combination the baseline did not have. Below the table come the warnings, once when every combination has them, the new and fixed errors of each combination, and the combinations that only one side has. `--markdown` gives the same, and the `<details>` block of each combination carries its own comparison. In the JSON output each combination carries its `baseline`, and the root `baseline` names the file and the `added` and `missing` combinations. `--fail-on-regression` exits with `1` when any combination regressed or has a new error.
+
 CI gating:
 
 ```bash
@@ -200,6 +316,12 @@ npx webperf-snippets https://example.com --budget-lcp 2500 --budget-cls 0.1
 | `--storage-state <path>`     | Playwright storage state (cookies and localStorage) for pages that require authentication. |
 | `--header "Name: value"`     | Request header for the origin of the URL only, such as the token of an access proxy. Repeat it for more headers. |
 | `--own-domains <list>`       | Domains of the site to count as first party, such as its own CDN. Comma-separated or repeated. |
+| `--runs <n>`                 | Measure the page n times, from 1 to 9, one after the other, and report the median run. |
+| `--median-by <metric>`       | Metric that picks the median run: `LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`. Default: the first of LCP, FCP, TTFB and `navMs` measured. |
+| `--matrix <file>`            | Measure every URL of a JSON file on every device and in every variant it lists, and report a table of the medians. |
+| `--baseline <file>`          | Compare the page, or each combination of a matrix, with the `--json` output of an earlier run: which metrics got worse or better beyond the noise, and which errors are new or fixed. |
+| `--threshold <metric>=<v>`   | The threshold of a metric for `--baseline`, such as `LCP=15%`, `LCP=200ms` or `CLS=0.1`. Repeat it for more metrics. |
+| `--fail-on-regression`       | Exit `1` when a metric regressed or an error is new against `--baseline`. |
 | `--crux-key <key>`           | Add CrUX field data (p75) next to the measured values. Prefer `CRUX_API_KEY`. |
 | `--report-to <url>`          | POST the results to this `https` URL after the run. No external calls without it. |
 | `--api-key <key>`            | Sent as a Bearer token with `--report-to`. Prefer `PERF_REVIEWS_API_KEY`. |
@@ -239,8 +361,8 @@ The results of a run can add steps on the same page load: LCP over 2.5 s adds `L
 | Code | Meaning                                       |
 | ---- | --------------------------------------------- |
 | `0`  | All checks passed.                            |
-| `1`  | Budget violation, a snippet errored, or a step of the workflow reported an `error` issue. |
-| `2`  | Usage error (missing URL, unknown workflow).  |
+| `1`  | Budget violation, a snippet errored, a step of the workflow reported an `error` issue, or, with `--fail-on-regression`, a regression against the baseline. |
+| `2`  | Usage error (missing URL, unknown workflow, a baseline or threshold that is not valid). |
 
 ## MCP server
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.2.0",
+  "version": "1.4.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",
@@ -21,7 +21,8 @@
   },
   "devDependencies": {
     "@modelcontextprotocol/client": "^2.3.1",
-    "vitest": "^5.0.5"
+    "vitest": "^5.0.5",
+    "web-vitals": "^6.2.3"
   },
   "engines": {
     "node": ">=20.12.0"

package/snippets/CoreWebVitals/CLS.js

@@ -53,10 +53,11 @@
 
   observer.observe({ type: "layout-shift", buffered: true });
 
-  // Update on visibility change (final CLS)
+  // Update on visibility change (final CLS). The shifts that the observer has not delivered yet are
+  // in takeRecords(): without them, the last shift before the page hides is lost.
   document.addEventListener("visibilitychange", () => {
     if (document.visibilityState === "hidden") {
-      observer.takeRecords();
+      for (const entry of observer.takeRecords()) addShift(entry);
       console.log("%c📊 Final CLS (on page hide):", "font-weight: bold;");
       logCLS();
     }

package/snippets/CoreWebVitals/LCP-Subparts.js

@@ -76,6 +76,23 @@
     return { waiting: firstByte - requestStart, download: responseEnd - firstByte };
   };
 
+  // The times that split the LCP into its subparts, as web-vitals computes them. The resource is the
+  // last request for the LCP URL that started before the render: an earlier download of the same URL
+  // did not paint the element. Its end is capped at the LCP, since a video keeps downloading after
+  // its first frame paints, so the four subparts always add up to the LCP.
+  const lcpSubpartTimes = (lcpEntry, navEntry) => {
+    const activationStart = navEntry.activationStart || 0;
+    const renderTime = Math.max(0, lcpEntry.startTime - activationStart);
+    const resEntry = performance
+      .getEntriesByType("resource")
+      .filter((e) => !isOwnRequest(e))
+      .findLast((e) => e.name === lcpEntry.url && (e.requestStart || e.startTime) <= lcpEntry.startTime);
+    const ttfb = Math.max(0, navEntry.responseStart - activationStart);
+    const requestStart = Math.max(ttfb, resEntry ? (resEntry.requestStart || resEntry.startTime) - activationStart : 0);
+    const responseEnd = Math.min(renderTime, Math.max(requestStart, resEntry ? resEntry.responseEnd - activationStart : 0));
+    return { resEntry, activationStart, ttfb, requestStart, responseEnd, renderTime };
+  };
+
   const getNavigationEntry = () => {
     const navEntry = performance.getEntriesByType("navigation")[0];
     if (navEntry?.responseStart > 0 && navEntry.responseStart < performance.now()) {
@@ -91,31 +108,14 @@
     const navEntry = getNavigationEntry();
     if (!navEntry) return;
 
-    const lcpResEntry = performance
-      .getEntriesByType("resource").filter((e) => !isOwnRequest(e))
-      .find((e) => e.name === lcpEntry.url);
-
-    const activationStart = navEntry.activationStart || 0;
-
-    // Calculate sub-parts
-    const ttfb = Math.max(0, navEntry.responseStart - activationStart);
-
-    const lcpRequestStart = Math.max(
+    const {
+      resEntry: lcpResEntry,
+      activationStart,
       ttfb,
-      lcpResEntry
-        ? (lcpResEntry.requestStart || lcpResEntry.startTime) - activationStart
-        : 0
-    );
-
-    const lcpResponseEnd = Math.max(
-      lcpRequestStart,
-      lcpResEntry ? lcpResEntry.responseEnd - activationStart : 0
-    );
-
-    const lcpRenderTime = Math.max(
-      lcpResponseEnd,
-      lcpEntry.startTime - activationStart
-    );
+      requestStart: lcpRequestStart,
+      responseEnd: lcpResponseEnd,
+      renderTime: lcpRenderTime,
+    } = lcpSubpartTimes(lcpEntry, navEntry);
 
     const loadSplit = splitLoadTime(lcpResEntry, lcpRequestStart, lcpResponseEnd, activationStart);
 
@@ -278,17 +278,14 @@
   if (!navEntrySync) {
     return { script: "LCP-Subparts", status: "error", error: "No navigation entry" };
   }
-  const lcpResEntrySync = performance.getEntriesByType("resource").filter((e) => !isOwnRequest(e))
-    .find((e) => e.name === lcpEntry.url);
-  const activationStartSync = navEntrySync.activationStart || 0;
-  const ttfbSync = Math.max(0, navEntrySync.responseStart - activationStartSync);
-  const lcpRequestStartSync = Math.max(ttfbSync,
-    lcpResEntrySync ? (lcpResEntrySync.requestStart || lcpResEntrySync.startTime) - activationStartSync : 0
-  );
-  const lcpResponseEndSync = Math.max(lcpRequestStartSync,
-    lcpResEntrySync ? lcpResEntrySync.responseEnd - activationStartSync : 0
-  );
-  const lcpRenderTimeSync = Math.max(lcpResponseEndSync, lcpEntry.startTime - activationStartSync);
+  const {
+    resEntry: lcpResEntrySync,
+    activationStart: activationStartSync,
+    ttfb: ttfbSync,
+    requestStart: lcpRequestStartSync,
+    responseEnd: lcpResponseEndSync,
+    renderTime: lcpRenderTimeSync,
+  } = lcpSubpartTimes(lcpEntry, navEntrySync);
   const totalSync = Math.round(lcpRenderTimeSync);
   const ratingSync = valueToRating(totalSync);
   const ttfbVal = Math.round(ttfbSync);

package/src/baseline.js

@@ -0,0 +1,201 @@
+import { readFileSync } from "node:fs";
+import { blockingErrors } from "./blocking.js";
+import { describeThrottling } from "./throttling.js";
+
+// A measurement compared with an earlier one, its baseline: which metrics got worse or better beyond
+// the noise of the lab, which errors appeared or went away, and what differs in how both were
+// measured. Pure functions over the JSON the CLI prints with --json, so a baseline is any earlier
+// output saved with --json > baseline.json.
+
+const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
+
+// A change counts when it beats the larger of a share of the baseline and a minimum: 10% of a 3 s LCP
+// is 300 ms, and 10% of a 200 ms TTFB would be noise. CLS is a score, with no relative part.
+export const DEFAULT_THRESHOLDS = {
+  LCP: { relative: 0.1, absolute: 100 },
+  FCP: { relative: 0.1, absolute: 100 },
+  TTFB: { relative: 0.1, absolute: 100 },
+  INP: { relative: 0.1, absolute: 50 },
+  CLS: { absolute: 0.05 },
+};
+
+// The baseline in a file, as { kind: "page" | "matrix", file, data }. Throws with what to do when
+// the file is not an output of the CLI.
+export function readBaseline(text, file) {
+  const howTo = `save one with webperf-snippets <url> --json > ${file}`;
+  let data;
+  try {
+    data = JSON.parse(text);
+  } catch {
+    throw new Error(`${file} is not JSON: ${howTo}`);
+  }
+  if (Array.isArray(data?.combinations)) return { kind: "matrix", file, data };
+  if (typeof data?.url === "string" && Array.isArray(data.results)) return { kind: "page", file, data };
+  throw new Error(`${file} is not an output of webperf-snippets --json: it has neither the results of a page nor the combinations of a matrix; ${howTo}`);
+}
+
+export function loadBaseline(path) {
+  let text;
+  try {
+    text = readFileSync(path, "utf8");
+  } catch (err) {
+    throw new Error(`Cannot read the baseline ${path}: ${err.code === "ENOENT" ? "the file does not exist" : err.message}`);
+  }
+  return readBaseline(text, path);
+}
+
+// One part of the threshold of a metric, from --threshold: LCP=15% (relative), LCP=200ms
+// (absolute), CLS=0.1 (CLS has only an absolute part, with no unit)
+export function parseThreshold(raw) {
+  const match = /^([A-Za-z]+)=(\d+(?:\.\d+)?)(%|ms)?$/.exec(raw.trim());
+  if (!match) throw new Error(`Invalid --threshold "${raw}": give METRIC=VALUE with its unit, such as LCP=15%, LCP=200ms or CLS=0.1`);
+  const [, name, number, unit] = match;
+  const metric = name.toUpperCase();
+  if (!METRICS.includes(metric)) throw new Error(`Unknown metric "${name}" in --threshold "${raw}". Choose from: ${METRICS.join(", ")}`);
+  const value = Number(number);
+  if (metric === "CLS") {
+    if (unit) throw new Error(`Invalid --threshold "${raw}": CLS is a score, with no unit and no relative part, such as CLS=0.1`);
+    return { metric, absolute: value };
+  }
+  if (unit === "%") return { metric, relative: value / 100 };
+  if (unit === "ms") return { metric, absolute: value };
+  throw new Error(`Invalid --threshold "${raw}": give the unit, such as ${metric}=15% or ${metric}=200ms`);
+}
+
+// The default thresholds with the parts that --threshold changes
+export function thresholdsWith(overrides) {
+  const thresholds = Object.fromEntries(Object.entries(DEFAULT_THRESHOLDS).map(([metric, parts]) => [metric, { ...parts }]));
+  for (const { metric, ...part } of overrides) Object.assign(thresholds[metric], part);
+  return thresholds;
+}
+
+const round = (n) => Math.round(n * 10000) / 10000;
+
+// The value of a metric in a page: the median of its runs, with their range when there are two or
+// more, or its one result, as runs.js picks it
+function measured(payload, metric) {
+  const summary = payload.runs?.summary?.[metric];
+  if (summary) {
+    const runs = summary.values.filter((value) => value !== null).length;
+    return { value: summary.median, ...(runs >= 2 ? { range: { min: summary.min, max: summary.max } } : {}) };
+  }
+  const result = payload.results?.find((r) => r.metric === metric && r.status === "ok" && typeof r.value === "number");
+  return result ? { value: result.value } : null;
+}
+
+// A regression must beat the threshold of its metric and, when the baseline has several runs, the
+// worst of them: a median inside the range of the baseline is noise the baseline already had.
+function verdict(base, current, threshold) {
+  if (current.value - base.value > threshold && (!base.range || current.value > base.range.max)) return "regressed";
+  if (base.value - current.value > threshold && (!base.range || current.value < base.range.min)) return "improved";
+  return "unchanged";
+}
+
+function compareMetrics(base, current, thresholds) {
+  const metrics = {};
+  for (const metric of METRICS) {
+    const b = measured(base, metric);
+    const c = measured(current, metric);
+    if (!b && !c) continue;
+    if (!b || !c) {
+      metrics[metric] = { ...(b ? { baseline: b.value } : { current: c.value }), verdict: "not measured" };
+      continue;
+    }
+    const { relative = 0, absolute } = thresholds[metric];
+    const threshold = round(Math.max(relative * b.value, absolute));
+    const delta = c.value - b.value;
+    metrics[metric] = {
+      baseline: b.value,
+      current: c.value,
+      delta: round(delta),
+      deltaPercent: b.value === 0 ? null : Math.round((delta / b.value) * 1000) / 10,
+      threshold,
+      verdict: verdict(b, c, threshold),
+    };
+  }
+  return metrics;
+}
+
+// Two errors are the same when they share a code or, without one, when their messages differ only in
+// their figures: "Render-blocking CSS (69.21 KB)…" and "Render-blocking CSS (71.80 KB)…"
+const keyOf = (error) => `${error.id}\u0000${error.code ?? error.message.replace(/\d+(?:[.,]\d+)*/g, "#")}`;
+
+function compareErrors(base, current) {
+  const byKey = (payload) => new Map(blockingErrors(payload.results ?? []).map((error) => [keyOf(error), error]));
+  const b = byKey(base);
+  const c = byKey(current);
+  return {
+    new: [...c].filter(([key]) => !b.has(key)).map(([, error]) => error),
+    fixed: [...b].filter(([key]) => !c.has(key)).map(([, error]) => error),
+  };
+}
+
+const describeWork = (conditions) => (conditions.snippet ? `snippet ${conditions.snippet}` : `workflow ${conditions.workflow}`);
+const describeLoad = (throttling) => (throttling ? `with ${describeThrottling(throttling)}` : "without throttling");
+const runCount = (payload) => payload.runs?.count ?? 1;
+const pathOf = (url) => {
+  try {
+    const { pathname, search } = new URL(url);
+    return pathname + search;
+  } catch {
+    return url;
+  }
+};
+
+// What differs in how both were measured. The comparison goes on: the warnings say how far to trust it.
+function warningsOf(base, current) {
+  const warnings = [];
+  const b = base.conditions;
+  const c = current.conditions;
+  if (!b) {
+    warnings.push("the baseline comes from a CLI without conditions (1.3.0 or earlier); it cannot say how it was measured");
+  } else if (c) {
+    if (describeWork(b) !== describeWork(c)) warnings.push(`the baseline was measured with ${describeWork(b)}, and this run with ${describeWork(c)}`);
+    if (b.viewport !== c.viewport) warnings.push(`the baseline was measured on ${b.viewport}, and this run on ${c.viewport}`);
+    if (b.waitMs !== c.waitMs) warnings.push(`the baseline waited ${b.waitMs} ms after the load, and this run ${c.waitMs} ms`);
+  }
+  if (describeLoad(base.throttling) !== describeLoad(current.throttling)) {
+    warnings.push(`the baseline was measured ${describeLoad(base.throttling)}, and this run ${describeLoad(current.throttling)}`);
+  }
+  if (runCount(base) < 2) warnings.push("the baseline has 1 run, so only the thresholds apply; measure both with --runs");
+  else if (runCount(base) !== runCount(current)) warnings.push(`the baseline has ${runCount(base)} runs, and this run ${runCount(current)}`);
+  if (b && c && b.cliVersion !== c.cliVersion) {
+    warnings.push(`the baseline was measured with CLI ${b.cliVersion}, and this run with ${c.cliVersion}; the snippets may have changed`);
+  }
+  // Another host is the review app of a pull request against production: the same page
+  if (pathOf(base.url) !== pathOf(current.url)) warnings.push(`the baseline measured ${pathOf(base.url)} and this run ${pathOf(current.url)}`);
+  return warnings;
+}
+
+// A page against its baseline, as the `baseline` field of its JSON. The runs of the baseline say how
+// much to trust its values, in the header of the reports.
+export function compareRun(base, current, { file, thresholds = DEFAULT_THRESHOLDS }) {
+  return {
+    file,
+    url: base.url,
+    runs: base.runs ? { count: base.runs.count, medianBy: base.runs.medianBy } : { count: 1 },
+    metrics: compareMetrics(base, current, thresholds),
+    errors: compareErrors(base, current),
+    warnings: warningsOf(base, current),
+  };
+}
+
+// A matrix against a baseline matrix: each combination with the one of the same id, and the ids that
+// only one side has
+export function compareMatrix(base, current, { file, thresholds = DEFAULT_THRESHOLDS }) {
+  const baseById = new Map(base.combinations.map((c) => [c.id, c]));
+  const currentIds = new Set(current.combinations.map((c) => c.id));
+  const combinations = {};
+  for (const c of current.combinations) {
+    if (baseById.has(c.id)) combinations[c.id] = compareRun(baseById.get(c.id), c, { file, thresholds });
+  }
+  return {
+    root: {
+      file,
+      added: current.combinations.filter((c) => !baseById.has(c.id)).map((c) => c.id),
+      missing: base.combinations.filter((c) => !currentIds.has(c.id)).map((c) => c.id),
+      warnings: [],
+    },
+    combinations,
+  };
+}