webperf-snippets 1.3.0 → 1.4.0

package/README.md

@@ -70,6 +70,12 @@ JSON output (for piping into `jq` or CI):
 npx webperf-snippets https://example.com --json
 ```
 
+The JSON output records how the page was measured in `conditions`, so a later run can tell whether it measured the same way: the viewport, the workflow (or the path of the snippet with `--snippet`), the wait and the version of the CLI.
+
+```json
+"conditions": { "viewport": "mobile", "workflow": "core-web-vitals", "waitMs": 3000, "cliVersion": "1.4.0" }
+```
+
 Single snippet:
 
 ```bash
@@ -152,7 +158,7 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.3.0" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.4.0" }
 }
 ```
 
@@ -232,10 +238,62 @@ npx webperf-snippets --matrix matrix.json --budget-lcp 2500
 
 The options of the command line apply to every combination: `--workflow`, `--snippet`, `--wait`, `--interact-script`, `--header`, `--own-domains`, `--storage-state`, `--cpu`, `--network` and the budgets. The file replaces `cpu`, `network` and `storageState` where it sets them, and its `headers` and `ownDomains` add to the ones of the command line, the variant winning on the same header. A matrix file is meant to be versioned, so pass secrets, such as the token of an access proxy, with `--header` and an environment variable; a header of the file whose name looks like a secret raises a warning. A key the CLI does not know is an error, so a typo does not measure without its condition.
 
-The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants) and `runs`. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
+The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants), `runs` and `conditions`, with the viewport of its device. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
 
 With CrUX, the CLI asks once for each URL and form factor, so the variants of a device share their field data. With `--report-to`, it posts one report for each combination, the body of one page with `meta.device`, `meta.variant` and `meta.runs`; the headers of the command line and of the file are conditions of the load, and no report carries their values.
 
+Compare a run with an earlier one with `--baseline <file>`, the `--json` output of that run. The comparison says which metrics got worse or better beyond the noise of the lab, which errors are new or fixed, and what differs in how both runs were measured:
+
+```bash
+# On main, once: the baseline
+npx webperf-snippets https://example.com --runs 5 --json > baseline.json
+
+# On the pull request: compare and report
+npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.json --markdown
+
+# Fail when the page gets worse
+npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.json --fail-on-regression
+```
+
+LCP, FCP, TTFB, CLS and INP compare when both runs measured them, the median of each with `--runs`; a metric that only one side measured is `not measured`. A metric regresses when it gets worse by more than its threshold, the larger of a share of the baseline and a minimum, and, when the baseline has two runs or more, ends above the slowest of them: a median inside the range of the baseline is noise the baseline already had. It improves under the same rules the other way, and otherwise it is unchanged.
+
+| Metric         | Share of the baseline | Minimum |
+| -------------- | --------------------- | ------- |
+| LCP, FCP, TTFB | 10%                   | 100 ms  |
+| INP            | 10%                   | 50 ms   |
+| CLS            | none                  | 0.05    |
+
+`--threshold` changes one part of a threshold and keeps the other: `--threshold LCP=15%` the share, `--threshold LCP=200ms` the minimum, and `--threshold CLS=0.1` the only part of CLS. Repeat it for more metrics.
+
+The errors that compare are the ones that decide the exit code: the issues of severity `error` of the steps of the workflow and the snippets that failed. Two errors are the same when they share a code or, without one, when their messages differ only in their figures, so "Render-blocking CSS (69.21 KB)…" and "Render-blocking CSS (71.80 KB)…" are one error. An error that only this run has is new, and one that only the baseline had is fixed. Warnings and information do not compare, since they change too easily.
+
+The report adds the comparison after the usual one, and `--markdown` a table for the pull request comment. The JSON output carries it in `baseline`:
+
+```json
+"baseline": {
+  "file": "baseline.json",
+  "url": "https://example.com/",
+  "runs": { "count": 5, "medianBy": "LCP" },
+  "metrics": {
+    "LCP": { "baseline": 2100, "current": 2480, "delta": 380, "deltaPercent": 18.1, "threshold": 210, "verdict": "regressed" }
+  },
+  "errors": { "new": [{ "id": "render-blocking", "message": "1 render-blocking resource(s) delay rendering until 900ms" }], "fixed": [] },
+  "warnings": []
+}
+```
+
+When both runs were measured differently, with another workflow or snippet, viewport, wait, throttling, number of runs or version of the CLI, or on another path or query, the CLI compares anyway and warns on stderr, in the reports and in `warnings`: whoever reads the pull request comment does not see the stderr of the CI. Another host does not warn, since it is the review app of a pull request against production. A baseline of one run warns that only the thresholds apply.
+
+The comparison only informs, and the exit code is the same as without it, unless `--fail-on-regression`, which exits with `1` when a metric regressed or an error is new, and names them on stderr. A baseline that does not exist or is not an output of the CLI, a baseline of a page for a matrix or of a matrix for a page, or a `--threshold` the CLI does not understand exits with code `2`, before the page loads.
+
+A matrix compares with the `--json` output of an earlier matrix, each combination with the one of the same `id`:
+
+```bash
+npx webperf-snippets --matrix matrix.json --runs 3 --baseline last-week.json
+```
+
+The table adds a change column after each metric: the change when it counts (`▲ +380ms`, `▼ −90ms`), `=` when it is unchanged, and `new` for a combination the baseline did not have. Below the table come the warnings, once when every combination has them, the new and fixed errors of each combination, and the combinations that only one side has. `--markdown` gives the same, and the `<details>` block of each combination carries its own comparison. In the JSON output each combination carries its `baseline`, and the root `baseline` names the file and the `added` and `missing` combinations. `--fail-on-regression` exits with `1` when any combination regressed or has a new error.
+
 CI gating:
 
 ```bash
@@ -261,6 +319,9 @@ npx webperf-snippets https://example.com --budget-lcp 2500 --budget-cls 0.1
 | `--runs <n>`                 | Measure the page n times, from 1 to 9, one after the other, and report the median run. |
 | `--median-by <metric>`       | Metric that picks the median run: `LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`. Default: the first of LCP, FCP, TTFB and `navMs` measured. |
 | `--matrix <file>`            | Measure every URL of a JSON file on every device and in every variant it lists, and report a table of the medians. |
+| `--baseline <file>`          | Compare the page, or each combination of a matrix, with the `--json` output of an earlier run: which metrics got worse or better beyond the noise, and which errors are new or fixed. |
+| `--threshold <metric>=<v>`   | The threshold of a metric for `--baseline`, such as `LCP=15%`, `LCP=200ms` or `CLS=0.1`. Repeat it for more metrics. |
+| `--fail-on-regression`       | Exit `1` when a metric regressed or an error is new against `--baseline`. |
 | `--crux-key <key>`           | Add CrUX field data (p75) next to the measured values. Prefer `CRUX_API_KEY`. |
 | `--report-to <url>`          | POST the results to this `https` URL after the run. No external calls without it. |
 | `--api-key <key>`            | Sent as a Bearer token with `--report-to`. Prefer `PERF_REVIEWS_API_KEY`. |
@@ -300,8 +361,8 @@ The results of a run can add steps on the same page load: LCP over 2.5 s adds `L
 | Code | Meaning                                       |
 | ---- | --------------------------------------------- |
 | `0`  | All checks passed.                            |
-| `1`  | Budget violation, a snippet errored, or a step of the workflow reported an `error` issue. |
-| `2`  | Usage error (missing URL, unknown workflow).  |
+| `1`  | Budget violation, a snippet errored, a step of the workflow reported an `error` issue, or, with `--fail-on-regression`, a regression against the baseline. |
+| `2`  | Usage error (missing URL, unknown workflow, a baseline or threshold that is not valid). |
 
 ## MCP server
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",
@@ -21,7 +21,8 @@
   },
   "devDependencies": {
     "@modelcontextprotocol/client": "^2.3.1",
-    "vitest": "^5.0.5"
+    "vitest": "^5.0.5",
+    "web-vitals": "^6.2.3"
   },
   "engines": {
     "node": ">=20.12.0"

package/snippets/CoreWebVitals/CLS.js

@@ -53,10 +53,11 @@
 
   observer.observe({ type: "layout-shift", buffered: true });
 
-  // Update on visibility change (final CLS)
+  // Update on visibility change (final CLS). The shifts that the observer has not delivered yet are
+  // in takeRecords(): without them, the last shift before the page hides is lost.
   document.addEventListener("visibilitychange", () => {
     if (document.visibilityState === "hidden") {
-      observer.takeRecords();
+      for (const entry of observer.takeRecords()) addShift(entry);
       console.log("%c📊 Final CLS (on page hide):", "font-weight: bold;");
       logCLS();
     }

package/snippets/CoreWebVitals/LCP-Subparts.js

@@ -76,6 +76,23 @@
     return { waiting: firstByte - requestStart, download: responseEnd - firstByte };
   };
 
+  // The times that split the LCP into its subparts, as web-vitals computes them. The resource is the
+  // last request for the LCP URL that started before the render: an earlier download of the same URL
+  // did not paint the element. Its end is capped at the LCP, since a video keeps downloading after
+  // its first frame paints, so the four subparts always add up to the LCP.
+  const lcpSubpartTimes = (lcpEntry, navEntry) => {
+    const activationStart = navEntry.activationStart || 0;
+    const renderTime = Math.max(0, lcpEntry.startTime - activationStart);
+    const resEntry = performance
+      .getEntriesByType("resource")
+      .filter((e) => !isOwnRequest(e))
+      .findLast((e) => e.name === lcpEntry.url && (e.requestStart || e.startTime) <= lcpEntry.startTime);
+    const ttfb = Math.max(0, navEntry.responseStart - activationStart);
+    const requestStart = Math.max(ttfb, resEntry ? (resEntry.requestStart || resEntry.startTime) - activationStart : 0);
+    const responseEnd = Math.min(renderTime, Math.max(requestStart, resEntry ? resEntry.responseEnd - activationStart : 0));
+    return { resEntry, activationStart, ttfb, requestStart, responseEnd, renderTime };
+  };
+
   const getNavigationEntry = () => {
     const navEntry = performance.getEntriesByType("navigation")[0];
     if (navEntry?.responseStart > 0 && navEntry.responseStart < performance.now()) {
@@ -91,31 +108,14 @@
     const navEntry = getNavigationEntry();
     if (!navEntry) return;
 
-    const lcpResEntry = performance
-      .getEntriesByType("resource").filter((e) => !isOwnRequest(e))
-      .find((e) => e.name === lcpEntry.url);
-
-    const activationStart = navEntry.activationStart || 0;
-
-    // Calculate sub-parts
-    const ttfb = Math.max(0, navEntry.responseStart - activationStart);
-
-    const lcpRequestStart = Math.max(
+    const {
+      resEntry: lcpResEntry,
+      activationStart,
       ttfb,
-      lcpResEntry
-        ? (lcpResEntry.requestStart || lcpResEntry.startTime) - activationStart
-        : 0
-    );
-
-    const lcpResponseEnd = Math.max(
-      lcpRequestStart,
-      lcpResEntry ? lcpResEntry.responseEnd - activationStart : 0
-    );
-
-    const lcpRenderTime = Math.max(
-      lcpResponseEnd,
-      lcpEntry.startTime - activationStart
-    );
+      requestStart: lcpRequestStart,
+      responseEnd: lcpResponseEnd,
+      renderTime: lcpRenderTime,
+    } = lcpSubpartTimes(lcpEntry, navEntry);
 
     const loadSplit = splitLoadTime(lcpResEntry, lcpRequestStart, lcpResponseEnd, activationStart);
 
@@ -278,17 +278,14 @@
   if (!navEntrySync) {
     return { script: "LCP-Subparts", status: "error", error: "No navigation entry" };
   }
-  const lcpResEntrySync = performance.getEntriesByType("resource").filter((e) => !isOwnRequest(e))
-    .find((e) => e.name === lcpEntry.url);
-  const activationStartSync = navEntrySync.activationStart || 0;
-  const ttfbSync = Math.max(0, navEntrySync.responseStart - activationStartSync);
-  const lcpRequestStartSync = Math.max(ttfbSync,
-    lcpResEntrySync ? (lcpResEntrySync.requestStart || lcpResEntrySync.startTime) - activationStartSync : 0
-  );
-  const lcpResponseEndSync = Math.max(lcpRequestStartSync,
-    lcpResEntrySync ? lcpResEntrySync.responseEnd - activationStartSync : 0
-  );
-  const lcpRenderTimeSync = Math.max(lcpResponseEndSync, lcpEntry.startTime - activationStartSync);
+  const {
+    resEntry: lcpResEntrySync,
+    activationStart: activationStartSync,
+    ttfb: ttfbSync,
+    requestStart: lcpRequestStartSync,
+    responseEnd: lcpResponseEndSync,
+    renderTime: lcpRenderTimeSync,
+  } = lcpSubpartTimes(lcpEntry, navEntrySync);
   const totalSync = Math.round(lcpRenderTimeSync);
   const ratingSync = valueToRating(totalSync);
   const ttfbVal = Math.round(ttfbSync);

package/src/baseline.js

@@ -0,0 +1,201 @@
+import { readFileSync } from "node:fs";
+import { blockingErrors } from "./blocking.js";
+import { describeThrottling } from "./throttling.js";
+
+// A measurement compared with an earlier one, its baseline: which metrics got worse or better beyond
+// the noise of the lab, which errors appeared or went away, and what differs in how both were
+// measured. Pure functions over the JSON the CLI prints with --json, so a baseline is any earlier
+// output saved with --json > baseline.json.
+
+const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
+
+// A change counts when it beats the larger of a share of the baseline and a minimum: 10% of a 3 s LCP
+// is 300 ms, and 10% of a 200 ms TTFB would be noise. CLS is a score, with no relative part.
+export const DEFAULT_THRESHOLDS = {
+  LCP: { relative: 0.1, absolute: 100 },
+  FCP: { relative: 0.1, absolute: 100 },
+  TTFB: { relative: 0.1, absolute: 100 },
+  INP: { relative: 0.1, absolute: 50 },
+  CLS: { absolute: 0.05 },
+};
+
+// The baseline in a file, as { kind: "page" | "matrix", file, data }. Throws with what to do when
+// the file is not an output of the CLI.
+export function readBaseline(text, file) {
+  const howTo = `save one with webperf-snippets <url> --json > ${file}`;
+  let data;
+  try {
+    data = JSON.parse(text);
+  } catch {
+    throw new Error(`${file} is not JSON: ${howTo}`);
+  }
+  if (Array.isArray(data?.combinations)) return { kind: "matrix", file, data };
+  if (typeof data?.url === "string" && Array.isArray(data.results)) return { kind: "page", file, data };
+  throw new Error(`${file} is not an output of webperf-snippets --json: it has neither the results of a page nor the combinations of a matrix; ${howTo}`);
+}
+
+export function loadBaseline(path) {
+  let text;
+  try {
+    text = readFileSync(path, "utf8");
+  } catch (err) {
+    throw new Error(`Cannot read the baseline ${path}: ${err.code === "ENOENT" ? "the file does not exist" : err.message}`);
+  }
+  return readBaseline(text, path);
+}
+
+// One part of the threshold of a metric, from --threshold: LCP=15% (relative), LCP=200ms
+// (absolute), CLS=0.1 (CLS has only an absolute part, with no unit)
+export function parseThreshold(raw) {
+  const match = /^([A-Za-z]+)=(\d+(?:\.\d+)?)(%|ms)?$/.exec(raw.trim());
+  if (!match) throw new Error(`Invalid --threshold "${raw}": give METRIC=VALUE with its unit, such as LCP=15%, LCP=200ms or CLS=0.1`);
+  const [, name, number, unit] = match;
+  const metric = name.toUpperCase();
+  if (!METRICS.includes(metric)) throw new Error(`Unknown metric "${name}" in --threshold "${raw}". Choose from: ${METRICS.join(", ")}`);
+  const value = Number(number);
+  if (metric === "CLS") {
+    if (unit) throw new Error(`Invalid --threshold "${raw}": CLS is a score, with no unit and no relative part, such as CLS=0.1`);
+    return { metric, absolute: value };
+  }
+  if (unit === "%") return { metric, relative: value / 100 };
+  if (unit === "ms") return { metric, absolute: value };
+  throw new Error(`Invalid --threshold "${raw}": give the unit, such as ${metric}=15% or ${metric}=200ms`);
+}
+
+// The default thresholds with the parts that --threshold changes
+export function thresholdsWith(overrides) {
+  const thresholds = Object.fromEntries(Object.entries(DEFAULT_THRESHOLDS).map(([metric, parts]) => [metric, { ...parts }]));
+  for (const { metric, ...part } of overrides) Object.assign(thresholds[metric], part);
+  return thresholds;
+}
+
+const round = (n) => Math.round(n * 10000) / 10000;
+
+// The value of a metric in a page: the median of its runs, with their range when there are two or
+// more, or its one result, as runs.js picks it
+function measured(payload, metric) {
+  const summary = payload.runs?.summary?.[metric];
+  if (summary) {
+    const runs = summary.values.filter((value) => value !== null).length;
+    return { value: summary.median, ...(runs >= 2 ? { range: { min: summary.min, max: summary.max } } : {}) };
+  }
+  const result = payload.results?.find((r) => r.metric === metric && r.status === "ok" && typeof r.value === "number");
+  return result ? { value: result.value } : null;
+}
+
+// A regression must beat the threshold of its metric and, when the baseline has several runs, the
+// worst of them: a median inside the range of the baseline is noise the baseline already had.
+function verdict(base, current, threshold) {
+  if (current.value - base.value > threshold && (!base.range || current.value > base.range.max)) return "regressed";
+  if (base.value - current.value > threshold && (!base.range || current.value < base.range.min)) return "improved";
+  return "unchanged";
+}
+
+function compareMetrics(base, current, thresholds) {
+  const metrics = {};
+  for (const metric of METRICS) {
+    const b = measured(base, metric);
+    const c = measured(current, metric);
+    if (!b && !c) continue;
+    if (!b || !c) {
+      metrics[metric] = { ...(b ? { baseline: b.value } : { current: c.value }), verdict: "not measured" };
+      continue;
+    }
+    const { relative = 0, absolute } = thresholds[metric];
+    const threshold = round(Math.max(relative * b.value, absolute));
+    const delta = c.value - b.value;
+    metrics[metric] = {
+      baseline: b.value,
+      current: c.value,
+      delta: round(delta),
+      deltaPercent: b.value === 0 ? null : Math.round((delta / b.value) * 1000) / 10,
+      threshold,
+      verdict: verdict(b, c, threshold),
+    };
+  }
+  return metrics;
+}
+
+// Two errors are the same when they share a code or, without one, when their messages differ only in
+// their figures: "Render-blocking CSS (69.21 KB)…" and "Render-blocking CSS (71.80 KB)…"
+const keyOf = (error) => `${error.id}\u0000${error.code ?? error.message.replace(/\d+(?:[.,]\d+)*/g, "#")}`;
+
+function compareErrors(base, current) {
+  const byKey = (payload) => new Map(blockingErrors(payload.results ?? []).map((error) => [keyOf(error), error]));
+  const b = byKey(base);
+  const c = byKey(current);
+  return {
+    new: [...c].filter(([key]) => !b.has(key)).map(([, error]) => error),
+    fixed: [...b].filter(([key]) => !c.has(key)).map(([, error]) => error),
+  };
+}
+
+const describeWork = (conditions) => (conditions.snippet ? `snippet ${conditions.snippet}` : `workflow ${conditions.workflow}`);
+const describeLoad = (throttling) => (throttling ? `with ${describeThrottling(throttling)}` : "without throttling");
+const runCount = (payload) => payload.runs?.count ?? 1;
+const pathOf = (url) => {
+  try {
+    const { pathname, search } = new URL(url);
+    return pathname + search;
+  } catch {
+    return url;
+  }
+};
+
+// What differs in how both were measured. The comparison goes on: the warnings say how far to trust it.
+function warningsOf(base, current) {
+  const warnings = [];
+  const b = base.conditions;
+  const c = current.conditions;
+  if (!b) {
+    warnings.push("the baseline comes from a CLI without conditions (1.3.0 or earlier); it cannot say how it was measured");
+  } else if (c) {
+    if (describeWork(b) !== describeWork(c)) warnings.push(`the baseline was measured with ${describeWork(b)}, and this run with ${describeWork(c)}`);
+    if (b.viewport !== c.viewport) warnings.push(`the baseline was measured on ${b.viewport}, and this run on ${c.viewport}`);
+    if (b.waitMs !== c.waitMs) warnings.push(`the baseline waited ${b.waitMs} ms after the load, and this run ${c.waitMs} ms`);
+  }
+  if (describeLoad(base.throttling) !== describeLoad(current.throttling)) {
+    warnings.push(`the baseline was measured ${describeLoad(base.throttling)}, and this run ${describeLoad(current.throttling)}`);
+  }
+  if (runCount(base) < 2) warnings.push("the baseline has 1 run, so only the thresholds apply; measure both with --runs");
+  else if (runCount(base) !== runCount(current)) warnings.push(`the baseline has ${runCount(base)} runs, and this run ${runCount(current)}`);
+  if (b && c && b.cliVersion !== c.cliVersion) {
+    warnings.push(`the baseline was measured with CLI ${b.cliVersion}, and this run with ${c.cliVersion}; the snippets may have changed`);
+  }
+  // Another host is the review app of a pull request against production: the same page
+  if (pathOf(base.url) !== pathOf(current.url)) warnings.push(`the baseline measured ${pathOf(base.url)} and this run ${pathOf(current.url)}`);
+  return warnings;
+}
+
+// A page against its baseline, as the `baseline` field of its JSON. The runs of the baseline say how
+// much to trust its values, in the header of the reports.
+export function compareRun(base, current, { file, thresholds = DEFAULT_THRESHOLDS }) {
+  return {
+    file,
+    url: base.url,
+    runs: base.runs ? { count: base.runs.count, medianBy: base.runs.medianBy } : { count: 1 },
+    metrics: compareMetrics(base, current, thresholds),
+    errors: compareErrors(base, current),
+    warnings: warningsOf(base, current),
+  };
+}
+
+// A matrix against a baseline matrix: each combination with the one of the same id, and the ids that
+// only one side has
+export function compareMatrix(base, current, { file, thresholds = DEFAULT_THRESHOLDS }) {
+  const baseById = new Map(base.combinations.map((c) => [c.id, c]));
+  const currentIds = new Set(current.combinations.map((c) => c.id));
+  const combinations = {};
+  for (const c of current.combinations) {
+    if (baseById.has(c.id)) combinations[c.id] = compareRun(baseById.get(c.id), c, { file, thresholds });
+  }
+  return {
+    root: {
+      file,
+      added: current.combinations.filter((c) => !baseById.has(c.id)).map((c) => c.id),
+      missing: base.combinations.filter((c) => !currentIds.has(c.id)).map((c) => c.id),
+      warnings: [],
+    },
+    combinations,
+  };
+}

package/src/bin.js

@@ -18,6 +18,9 @@ import { parseOwnDomains } from "./own-domains.js";
 import { MEDIAN_METRICS, summarizeRuns } from "./runs.js";
 import { loadMatrix } from "./matrix.js";
 import { blockingErrors } from "./blocking.js";
+import { measurementConditions } from "./conditions.js";
+import { loadBaseline, parseThreshold, thresholdsWith, compareRun, compareMatrix } from "./baseline.js";
+import { describeRegressions, describeMatrixRegressions, matrixComparisonView } from "./reporters/baseline-view.js";
 
 
 const USAGE = `webperf-snippets <url> [options]
@@ -63,6 +66,13 @@ Options:
                             variant it lists, and report a table of the medians
   --median-by <metric>      Metric that picks the median run: LCP, FCP, TTFB, CLS, INP,
                             navMs (default: the first of LCP, FCP, TTFB, navMs measured)
+  --baseline <file>         Compare the page, or each combination of a matrix, with the --json
+                            output of an earlier run: which metrics got worse or better beyond
+                            the noise, and which errors are new or fixed. It only informs,
+                            unless --fail-on-regression
+  --threshold <metric>=<v>  The threshold of a metric for --baseline, such as LCP=15%,
+                            LCP=200ms or CLS=0.1. Repeat it for more metrics
+  --fail-on-regression      Exit 1 when a metric regressed or an error is new against --baseline
   --crux-key <key>          Add CrUX field data (p75) next to the measured values.
                             Also read from the CRUX_API_KEY environment variable
   --report-to <url>         POST the results to this https URL after the run
@@ -86,6 +96,9 @@ Examples:
   npx webperf-snippets https://example.com --workflow loading --own-domains cdn.example.net
   npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
   npx webperf-snippets --matrix matrix.json --runs 3 --budget-lcp 2500
+  npx webperf-snippets https://example.com --runs 5 --json > baseline.json
+  npx webperf-snippets https://pr-123.example.com --runs 5 --baseline baseline.json --fail-on-regression
+  npx webperf-snippets --matrix matrix.json --baseline last-week.json
   npx webperf-snippets https://example.com --snippet INP --interact-script interactions.json
   npx webperf-snippets https://example.com/dashboard --storage-state auth.json
   npx webperf-snippets https://review.example.com --header "CF-Access-Client-Id: $CF_ID" --header "CF-Access-Client-Secret: $CF_SECRET"
@@ -166,6 +179,9 @@ async function main() {
         matrix: { type: "string" },
         runs: { type: "string" },
         "median-by": { type: "string" },
+        baseline: { type: "string" },
+        threshold: { type: "string", multiple: true },
+        "fail-on-regression": { type: "boolean" },
         "crux-key": { type: "string" },
         "report-to": { type: "string" },
         "api-key": { type: "string" },
@@ -242,6 +258,23 @@ async function main() {
   }
   if (medianBy != null && runCount == null && !matrixPath) fail("--median-by needs --runs");
 
+  // The baseline is read and checked before the browser opens: a wrong file fails at once
+  let baseline;
+  let thresholds;
+  if (values.baseline) {
+    try {
+      baseline = loadBaseline(values.baseline);
+      thresholds = thresholdsWith((values.threshold ?? []).map(parseThreshold));
+    } catch (err) {
+      fail(err.message);
+    }
+    if (!matrixPath && baseline.kind === "matrix") fail(`${values.baseline} is the baseline of a matrix: compare it with --matrix`);
+    if (matrixPath && baseline.kind === "page") fail(`${values.baseline} is the baseline of a page: compare it with a URL, not --matrix`);
+  } else {
+    if (values.threshold) fail("--threshold needs --baseline");
+    if (values["fail-on-regression"]) fail("--fail-on-regression needs --baseline");
+  }
+
   // What each page load runs, a snippet or a workflow, under the conditions of the load
   const items = values.snippet ? buildSnippetItem(values) : null;
   const workflow = values.snippet ? null : WORKFLOWS[values.workflow ?? "core-web-vitals"];
@@ -250,6 +283,9 @@ async function main() {
     const options = { waitMs, headless: !values.headed, interactScript, ...conditions };
     return items ? runSnippets({ ...options, items }) : runMeasurement({ ...options, workflow, rules: RULES });
   };
+  // What the JSON records of how each page was measured, for a later run to compare with
+  const conditionsFor = (viewport) =>
+    measurementConditions({ viewport, waitMs, ...(items ? { snippet: items[0].path } : { workflow: values.workflow ?? "core-web-vitals" }) });
 
   if (matrixPath) {
     let matrix;
@@ -258,7 +294,7 @@ async function main() {
     } catch (err) {
       fail(err.message);
     }
-    await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl });
+    await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds });
     return;
   }
 
@@ -271,11 +307,17 @@ async function main() {
     if (!median.payload) throw new Error(`Every run failed: ${median.errors.join("; ")}`);
     payload = { ...median.payload, runs: median.runs };
   }
+  payload.conditions = conditionsFor(viewportName);
 
   const cruxKey = values["crux-key"] || process.env.CRUX_API_KEY;
   const crux = cruxKey ? await fetchCrux({ url, apiKey: cruxKey, formFactor: formFactorFor(viewportName) }) : undefined;
   if (crux) payload.crux = crux;
 
+  if (baseline) {
+    payload.baseline = compareRun(baseline.data, payload, { file: baseline.file, thresholds });
+    for (const warning of payload.baseline.warnings) process.stderr.write(`Warning: ${warning}\n`);
+  }
+
   let output;
   if (values.markdown) {
     output = reportMarkdown(payload);
@@ -315,6 +357,12 @@ async function main() {
     }
     process.exit(1);
   }
+  // Without --fail-on-regression the comparison only informs, and the exit code is the one of today
+  const regressions = values["fail-on-regression"] ? describeRegressions(payload.baseline) : [];
+  if (regressions.length > 0) {
+    for (const r of regressions) process.stderr.write(`Regression against ${baseline.file}: ${r}\n`);
+    process.exit(1);
+  }
   process.exit(blockingErrors(payload.results).length > 0 ? 1 : 0);
 }
 
@@ -337,7 +385,7 @@ async function measureRuns(measure, count, medianBy, progress) {
 
 // Every combination of the matrix, in order, each one `matrix.runs` times. Each combination reports
 // its median run, and the exit code follows the same rules as one page, for every combination.
-async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl }) {
+async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds }) {
   for (const warning of matrix.warnings) process.stderr.write(`Warning: ${warning}\n`);
   const viewportOf = new Map(matrix.combinations.map((c) => [c.id, c.viewport]));
 
@@ -351,8 +399,8 @@ async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, repo
     const identity = { id: c.id, name: c.name, device: c.device, ...(c.variant ? { variant: c.variant } : {}), url: c.url };
     combinations.push(
       median.payload
-        ? { ...identity, ...median.payload, runs: median.runs }
-        : { ...identity, status: "error", error: `Every run failed: ${median.errors.join("; ")}`, runs: { count: matrix.runs, failed: matrix.runs } }
+        ? { ...identity, ...median.payload, runs: median.runs, conditions: conditionsFor(c.viewport) }
+        : { ...identity, status: "error", error: `Every run failed: ${median.errors.join("; ")}`, runs: { count: matrix.runs, failed: matrix.runs }, conditions: conditionsFor(c.viewport) }
     );
   }
 
@@ -369,8 +417,18 @@ async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, repo
     }
   }
 
-  const report = { matrix: matrixPath, runs: matrix.runs, combinations, verbose: values.verbose };
-  const output = values.markdown ? reportMatrixMarkdown(report) : values.json ? reportJson({ matrix: matrixPath, runs: matrix.runs, combinations }) : reportMatrixHuman(report);
+  // With --baseline, each combination is compared with the one of the same id in the baseline
+  let comparison;
+  if (baseline) {
+    const { root, combinations: byId } = compareMatrix(baseline.data, { combinations }, { file: baseline.file, thresholds });
+    for (const c of combinations) if (byId[c.id]) c.baseline = byId[c.id];
+    comparison = root;
+    for (const warning of matrixComparisonView(root, combinations).warnings) process.stderr.write(`Warning: ${warning}\n`);
+  }
+
+  const withBaseline = comparison ? { baseline: comparison } : {};
+  const report = { matrix: matrixPath, runs: matrix.runs, combinations, verbose: values.verbose, ...withBaseline };
+  const output = values.markdown ? reportMatrixMarkdown(report) : values.json ? reportJson({ matrix: matrixPath, runs: matrix.runs, ...withBaseline, combinations }) : reportMatrixHuman(report);
   process.stdout.write(output + "\n");
 
   // One report for each combination, with the body of one page and the device, variant and runs of
@@ -411,7 +469,10 @@ async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, repo
     for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
   }
   const failed = combinations.some((c) => c.error || blockingErrors(c.results).length > 0);
-  process.exit(violations.length > 0 || failed ? 1 : 0);
+  // Without --fail-on-regression the comparison only informs
+  const regressions = values["fail-on-regression"] ? describeMatrixRegressions(combinations) : [];
+  for (const r of regressions) process.stderr.write(`Regression against ${baseline.file} in ${r}\n`);
+  process.exit(violations.length > 0 || failed || regressions.length > 0 ? 1 : 0);
 }
 
 main().catch((err) => {

package/src/blocking.js

@@ -6,7 +6,9 @@ export function blockingErrors(results) {
   for (const r of results) {
     if (r.status === "error") errors.push({ id: r.id, message: r.error });
     if (r.reason || !Array.isArray(r.issues)) continue;
-    for (const issue of r.issues) if (issue.severity === "error") errors.push({ id: r.id, message: issue.message });
+    for (const issue of r.issues) {
+      if (issue.severity === "error") errors.push({ id: r.id, message: issue.message, ...(issue.code ? { code: issue.code } : {}) });
+    }
   }
   return errors;
 }

package/src/conditions.js

@@ -0,0 +1,7 @@
+import { cliVersion } from "./report-to.js";
+
+// How a measurement was taken, kept in its JSON so that a later run compared with it can tell whether
+// both were measured the same way. A snippet is named by its path, so an alias and its path match.
+export function measurementConditions({ viewport, workflow, snippet, waitMs }) {
+  return { viewport, ...(snippet ? { snippet } : { workflow }), waitMs, cliVersion };
+}

package/src/reporters/baseline-view.js

@@ -0,0 +1,83 @@
+// The comparison with a baseline as the human and markdown reports draw it: one row per metric with
+// its two values, the change and the verdict, and why an unchanged one did not count. The values stay
+// numbers with their unit, for each report to format.
+
+const MINUS = "−";
+const signed = (n, text) => (n > 0 ? `+${text}` : n < 0 ? `${MINUS}${text}` : text);
+const arrow = (delta) => (delta > 0 ? "▲" : delta < 0 ? "▼" : "=");
+const unitOf = (metric) => (metric === "CLS" ? "score" : "ms");
+
+// "▲ +380ms (+18%)", "▼ −90ms (−15%)", "▼ −0.0020"
+function describeChange(metric, { delta, deltaPercent }) {
+  if (metric === "CLS") return `${arrow(delta)} ${signed(delta, Math.abs(delta).toFixed(4))}`;
+  const ms = signed(delta, `${Math.abs(Math.round(delta))}ms`);
+  const percent = deltaPercent == null ? "" : ` (${signed(deltaPercent, `${Math.abs(Math.round(deltaPercent))}%`)})`;
+  return `${arrow(delta)} ${ms}${percent}`;
+}
+
+// An unchanged metric that moved stayed under its threshold, or beat it inside the range of the runs
+// of the baseline: noise the baseline already had
+function noteOf(metric, m) {
+  if (m.verdict !== "unchanged" || m.delta === 0) return null;
+  if (Math.abs(m.delta) > m.threshold) return "inside the range of the baseline";
+  return `under ${metric === "CLS" ? m.threshold : `${Math.round(m.threshold)}ms`}`;
+}
+
+export function baselineView({ file, url, runs, metrics, errors, warnings }) {
+  const summary = `${url}, ${runs.count === 1 ? "1 run" : `${runs.count} runs, median by ${runs.medianBy}`}`;
+  const rows = Object.entries(metrics).map(([metric, m]) => ({
+    metric,
+    unit: unitOf(metric),
+    baseline: m.baseline,
+    current: m.current,
+    change: m.verdict === "not measured" ? null : describeChange(metric, m),
+    verdict: m.verdict,
+    note: m.verdict === "not measured" ? null : noteOf(metric, m),
+  }));
+  return { file, summary, rows, newErrors: errors.new, fixedErrors: errors.fixed, warnings };
+}
+
+// What got worse, one line each, for --fail-on-regression: the metrics that regressed and the errors
+// that only this run has
+export function describeRegressions({ metrics, errors }) {
+  return [
+    ...Object.entries(metrics)
+      .filter(([, m]) => m.verdict === "regressed")
+      .map(([metric, m]) => `${metric} ${describeChange(metric, m)}`),
+    ...errors.new.map((e) => `new error ${e.id}: ${e.message}`),
+  ];
+}
+
+// The change of a metric in a cell of the matrix table: the figure only when it counts, so the table
+// reads at a glance. "▲ +380ms", "▼ −400ms", "=" for unchanged, "·" when one side did not measure it.
+export function changeCell(metric, m) {
+  if (m.verdict === "not measured") return { text: "·", verdict: m.verdict };
+  if (m.verdict === "unchanged") return { text: "=", verdict: m.verdict };
+  const amount = metric === "CLS" ? Math.abs(m.delta).toFixed(4) : `${Math.abs(Math.round(m.delta))}ms`;
+  return { text: `${arrow(m.delta)} ${signed(m.delta, amount)}`, verdict: m.verdict };
+}
+
+// The comparison of a matrix below its table. A warning that every compared combination has, such as
+// a baseline of one run, shows once; the others name their combinations.
+export function matrixComparisonView(root, combinations) {
+  const compared = combinations.filter((c) => c.baseline);
+  const idsByWarning = new Map();
+  for (const c of compared) {
+    for (const warning of c.baseline.warnings) idsByWarning.set(warning, [...(idsByWarning.get(warning) ?? []), c.id]);
+  }
+  const warnings = [...idsByWarning].map(([warning, ids]) => (ids.length === compared.length ? warning : `${ids.join(", ")}: ${warning}`));
+  const errorsOf = (kind) => compared.flatMap((c) => c.baseline.errors[kind].map((e) => `${c.id}: ${e.id}: ${e.message}`));
+  return {
+    file: root.file,
+    warnings: [...root.warnings, ...warnings],
+    newErrors: errorsOf("new"),
+    fixedErrors: errorsOf("fixed"),
+    added: root.added,
+    missing: root.missing,
+  };
+}
+
+// The regressions of every combination, each named by its combination
+export function describeMatrixRegressions(combinations) {
+  return combinations.filter((c) => c.baseline).flatMap((c) => describeRegressions(c.baseline).map((r) => `${c.id}: ${r}`));
+}

package/src/reporters/human.js

@@ -3,6 +3,7 @@ import { fieldDataView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
 import { describeRuns } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
+import { baselineView } from "./baseline-view.js";
 
 const RATING_ICON = {
   good: "🟢",
@@ -215,7 +216,26 @@ function renderRuns(runs) {
   return lines;
 }
 
-export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, throttling, runs, title = `WebPerf Snippets — ${url}` }) {
+const VERDICT_STYLE = { regressed: "red", improved: "green", unchanged: "dim", "not measured": "dim", new: "cyan" };
+
+// The comparison with --baseline, after the report: its warnings first, since they say how far to
+// trust the table, then a row per metric and the errors that appeared or went away
+function renderBaseline(comparison) {
+  const view = baselineView(comparison);
+  const lines = ["", styleText("bold", `Compared with ${view.file} (${view.summary})`)];
+  for (const warning of view.warnings) lines.push(`  ${styleText("yellow", `Warning: ${warning}`)}`);
+  lines.push(styleText("dim", `  ${pad("Metric", 9)}${pad("Baseline", 11)}${pad("Now", 10)}Change`));
+  for (const row of view.rows) {
+    const value = (v) => (v == null ? "·" : formatValue(v, row.unit));
+    const verdict = row.note ? `${row.verdict} (${row.note})` : row.verdict;
+    lines.push(`  ${pad(row.metric, 9)}${pad(value(row.baseline), 11)}${pad(value(row.current), 10)}${pad(row.change ?? "", 20)}${styleText(VERDICT_STYLE[row.verdict], verdict)}`);
+  }
+  for (const e of view.newErrors) lines.push(`  ${styleText("red", "+ new error")}  ${e.id}: ${e.message}`);
+  for (const e of view.fixedErrors) lines.push(`  ${styleText("green", "− fixed")}      ${e.id}: ${e.message}`);
+  return lines;
+}
+
+export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, throttling, runs, baseline, title = `WebPerf Snippets — ${url}` }) {
   const lines = [];
   lines.push(styleText(["bold"], title));
   lines.push(styleText("dim", `Navigated in ${navMs}ms`));
@@ -232,13 +252,14 @@ export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, th
     lines.push(styleText("yellow", "Page errors during run:"));
     for (const err of pageErrors) lines.push(`  ${styleText("dim", err)}`);
   }
+  if (baseline) lines.push(...renderBaseline(baseline));
   return lines.join("\n");
 }
 
 // One row per combination with the rated median of each metric, and below it the errors that decide
 // the exit code. With verbose, the report of the median run of each combination follows.
 export function reportMatrixHuman(report) {
-  const { file, summary, hasVariants, metrics, rows, errors } = matrixView(report);
+  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
   const columns = [["name", "Page"], ["device", "Device"], ...(hasVariants ? [["variant", "Variant"]] : [])];
   const widths = columns.map(([key, label]) => Math.max(...report.combinations.map((c) => c[key].length), label.length) + 3);
   const identity = (values) => values.map((value, i) => pad(value, widths[i])).join("");
@@ -246,14 +267,19 @@ export function reportMatrixHuman(report) {
   const lines = [];
   lines.push(styleText(["bold"], `WebPerf Snippets — ${file} (${summary})`));
   lines.push("");
-  lines.push(styleText("dim", `  ${identity(columns.map(([, label]) => label))}${metrics.map((m) => pad(m, 14)).join("").trimEnd()}`));
-  for (const { combination: c, cells } of rows) {
+  // With --baseline, a change column follows each metric
+  const headings = metrics.map((m) => pad(m, 14) + (comparison ? pad("Δ", 12) : "")).join("");
+  lines.push(styleText("dim", `  ${identity(columns.map(([, label]) => label))}${headings.trimEnd()}`));
+  for (const { combination: c, cells, changes } of rows) {
     const row = `  ${identity(columns.map(([key]) => c[key]))}`;
     if (!cells) {
       lines.push(row + styleText("red", "✗ every run failed"));
       continue;
     }
-    const drawn = cells.map((cell) => (cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}`), 14) : pad("·", 14)));
+    const drawn = cells.map((cell, i) => {
+      const value = cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}`), 14) : pad("·", 14);
+      return value + (changes ? pad(styleText(VERDICT_STYLE[changes[i].verdict], changes[i].text), 12) : "");
+    });
     lines.push(row + drawn.join("").trimEnd());
   }
 
@@ -263,6 +289,18 @@ export function reportMatrixHuman(report) {
     for (const e of errors) lines.push(`  ${styleText("red", "✗")} ${e}`);
   }
 
+  if (comparison) {
+    lines.push("", styleText("bold", `Compared with ${comparison.file}`));
+    for (const warning of comparison.warnings) lines.push(`  ${styleText("yellow", `Warning: ${warning}`)}`);
+    for (const e of comparison.newErrors) lines.push(`  ${styleText("red", "+ new error")}  ${e}`);
+    for (const e of comparison.fixedErrors) lines.push(`  ${styleText("green", "− fixed")}      ${e}`);
+    if (comparison.added.length) lines.push(`  Added: ${comparison.added.join(", ")}`);
+    if (comparison.missing.length) lines.push(`  Missing: ${comparison.missing.join(", ")}`);
+    if (!comparison.warnings.length && !comparison.newErrors.length && !comparison.fixedErrors.length && !comparison.added.length && !comparison.missing.length) {
+      lines.push(styleText("dim", "  The same combinations, no new or fixed errors, and no warnings"));
+    }
+  }
+
   if (report.verbose) {
     for (const c of report.combinations.filter((c) => !c.error)) {
       lines.push("", reportHuman({ ...c, verbose: true, title: `${c.id} — ${c.url}` }));

package/src/reporters/markdown.js

@@ -2,6 +2,7 @@ import { fieldDataView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
 import { describeRuns } from "../runs.js";
 import { matrixView } from "./matrix-view.js";
+import { baselineView } from "./baseline-view.js";
 
 const RATING_ICON = {
   good: "✅",
@@ -83,8 +84,27 @@ function renderRuns(runs) {
   return lines;
 }
 
+const VERDICT_ICON = { regressed: "❌", improved: "✅", unchanged: "➖", "not measured": "·" };
+
+// The comparison with --baseline. The warnings go in the comment itself: whoever reads the pull
+// request does not see the stderr of the CI.
+function renderBaseline(comparison) {
+  const view = baselineView(comparison);
+  const lines = ["", `### Compared with \`${view.file}\``, `> ${view.summary}`];
+  for (const warning of view.warnings) lines.push(">", `> ⚠️ ${warning}`);
+  lines.push("", "| Metric | Baseline | Now | Change | Verdict |", "|--------|----------|-----|--------|---------|");
+  for (const row of view.rows) {
+    const value = (v) => (v == null ? "·" : formatValue(v, row.unit));
+    const verdict = `${VERDICT_ICON[row.verdict]} ${row.verdict}${row.note ? ` (${row.note})` : ""}`;
+    lines.push(`| ${row.metric} | ${value(row.baseline)} | ${value(row.current)} | ${row.change ?? "·"} | ${verdict} |`);
+  }
+  if (view.newErrors.length) lines.push("", "#### New errors", ...view.newErrors.map((e) => `- \`${e.id}\`: ${e.message}`));
+  if (view.fixedErrors.length) lines.push("", "#### Fixed errors", ...view.fixedErrors.map((e) => `- \`${e.id}\`: ${e.message}`));
+  return lines;
+}
+
 // With a null title, the report has no heading, as inside the details block of a matrix
-export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttling, runs, title = `WebPerf Results — ${url}` }) {
+export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttling, runs, baseline, title = `WebPerf Results — ${url}` }) {
   const lines = [];
   if (title) lines.push(`## ${title}`);
   lines.push(`> Navigated in ${navMs}ms`);
@@ -120,6 +140,8 @@ export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttli
     for (const err of pageErrors) lines.push(`- ${err}`);
   }
 
+  if (baseline) lines.push(...renderBaseline(baseline));
+
   return lines.join("\n");
 }
 
@@ -127,17 +149,22 @@ export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttli
 // report of the median run of each combination folded into a details block, to keep a pull request
 // comment readable.
 export function reportMatrixMarkdown(report) {
-  const { file, summary, hasVariants, metrics, rows, errors } = matrixView(report);
-  const headings = ["Page", "Device", ...(hasVariants ? ["Variant"] : []), ...metrics];
+  const { file, summary, hasVariants, metrics, rows, errors, comparison } = matrixView(report);
+  // With --baseline, a change column follows each metric
+  const metricHeadings = metrics.flatMap((m) => (comparison ? [m, "Δ"] : [m]));
+  const headings = ["Page", "Device", ...(hasVariants ? ["Variant"] : []), ...metricHeadings];
   const row = (parts) => `|${parts.map((part) => (part ? ` ${part} ` : " ")).join("|")}|`;
 
   const lines = [`## WebPerf Results — ${file}`, `> ${summary}`, ""];
   lines.push(row(headings));
   lines.push(`|${headings.map((h) => "-".repeat(h.length + 2)).join("|")}|`);
-  for (const { combination: c, cells } of rows) {
+  for (const { combination: c, cells, changes } of rows) {
     const drawn = cells
-      ? cells.map((cell) => (cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}` : "·"))
-      : ["❌ every run failed", ...metrics.slice(1).map(() => "")];
+      ? cells.flatMap((cell, i) => {
+          const value = cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}` : "·";
+          return changes ? [value, changes[i].text] : [value];
+        })
+      : ["❌ every run failed", ...metricHeadings.slice(1).map(() => "")];
     lines.push(row([c.name, c.device, ...(hasVariants ? [c.variant] : []), ...drawn]));
   }
 
@@ -145,6 +172,19 @@ export function reportMatrixMarkdown(report) {
     lines.push("", "### Errors", ...errors.map((e) => `- ${e}`));
   }
 
+  if (comparison) {
+    lines.push("", `### Compared with \`${comparison.file}\``);
+    if (comparison.warnings.length) lines.push(...comparison.warnings.flatMap((w, i) => [...(i ? [">"] : []), `> ⚠️ ${w}`]), "");
+    const ids = (list) => list.map((id) => `\`${id}\``).join(", ");
+    const items = [
+      ...comparison.newErrors.map((e) => `- New error: ${e}`),
+      ...comparison.fixedErrors.map((e) => `- Fixed: ${e}`),
+      ...(comparison.added.length ? [`- Added: ${ids(comparison.added)}`] : []),
+      ...(comparison.missing.length ? [`- Missing: ${ids(comparison.missing)}`] : []),
+    ];
+    lines.push(...(items.length ? items : ["The same combinations, no new or fixed errors."]));
+  }
+
   for (const c of report.combinations.filter((c) => !c.error)) {
     lines.push("", `<details><summary>${c.id}</summary>`, "", reportMarkdown({ ...c, title: null }), "", "</details>");
   }

package/src/reporters/matrix-view.js

@@ -1,14 +1,16 @@
 import { basename } from "node:path";
 import { rate } from "../ratings.js";
 import { blockingErrors } from "../blocking.js";
+import { changeCell, matrixComparisonView } from "./baseline-view.js";
 
 // The order of the metric columns of the matrix table
 const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
 
 // What the human and markdown reports of a matrix show, so they only differ in how they draw it:
 // the title, the metric columns, one row per combination with the rated median of each metric, and
-// the errors that decide the exit code.
-export function matrixView({ matrix, runs, combinations }) {
+// the errors that decide the exit code. With --baseline, each row also has the change of each metric
+// ("new" for a combination the baseline did not have), and the comparison goes below the table.
+export function matrixView({ matrix, runs, combinations, baseline }) {
   const measured = combinations.filter((c) => !c.error);
   const medianBys = [...new Set(measured.map((c) => c.runs.medianBy))];
   const metrics = METRICS.filter((metric) => measured.some((c) => c.runs.summary[metric]));
@@ -22,6 +24,9 @@ export function matrixView({ matrix, runs, combinations }) {
     metrics,
     rows: combinations.map((c) => ({
       combination: c,
+      ...(baseline
+        ? { changes: metrics.map((metric) => (!c.baseline ? { text: "new", verdict: "new" } : c.baseline.metrics[metric] ? changeCell(metric, c.baseline.metrics[metric]) : { text: "·", verdict: "not measured" })) }
+        : {}),
       cells: c.error
         ? null
         : metrics.map((metric) => {
@@ -29,6 +34,7 @@ export function matrixView({ matrix, runs, combinations }) {
             return value ? { median: value.median, unit: value.unit, rating: rate(metric, value.median) } : null;
           }),
     })),
+    ...(baseline ? { comparison: matrixComparisonView(baseline, combinations) } : {}),
     errors: combinations.flatMap((c) =>
       c.error ? [`${c.id}: ${c.error}`] : blockingErrors(c.results).map((e) => `${c.id}: ${e.id}: ${e.message}`)
     ),