webperf-snippets 1.2.0 → 1.3.0

package/README.md

@@ -152,11 +152,11 @@ Without `--report-to` the CLI makes no external calls. With it, the results are
   "timestamp": "2026-05-06T10:00:00.000Z",
   "navMs": 1240,
   "results": [],
-  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.2.0" }
+  "meta": { "viewport": "mobile", "waitMs": 3000, "cli_version": "1.3.0" }
 }
 ```
 
-For a `--snippet` run, `workflow` is `null` and a `snippet` field carries the name. With `--cpu` or `--network`, `meta` carries the conditions in `throttling`. `results` is the same list the `--json` output contains, so it can include URLs and details of the measured page; send it only to an endpoint you trust.
+For a `--snippet` run, `workflow` is `null` and a `snippet` field carries the name. With `--cpu` or `--network`, `meta` carries the conditions in `throttling`, and with `--runs`, the summary of the runs in `runs`. `results` is the same list the `--json` output contains, so it can include URLs and details of the measured page; send it only to an endpoint you trust.
 
 - The URL must be `https`. `http` is accepted only for `localhost`, so a key is never sent in clear text.
 - `--api-key` is sent as `Authorization: Bearer <key>`. Prefer the `PERF_REVIEWS_API_KEY` environment variable, because a flag ends up in shell history and CI logs. The key is never printed.
@@ -178,6 +178,64 @@ npx webperf-snippets https://example.com --cpu 4 --network slow-4g
 
 Chrome adds the latency to the body of each response, not to its headers. The values read from `responseStart`, such as the TTFB and its sub-parts, do not include it; the download of each resource, FCP, LCP and the load event do. The report names the conditions under its header, and the JSON output carries them in `throttling`, such as `{ "cpu": 4, "network": "slow-4g" }`. A CPU factor below `1` or an unknown preset exits with code `2`.
 
+One page load is one sample, and lab values vary from one load to the next. `--runs <n>` measures the page n times, from 1 to 9, one after the other, and reports the median run, as WebPageTest does: every result comes from that run, so they agree with each other (the sub-parts of the LCP add up to its LCP):
+
+```bash
+npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
+```
+
+The median run is the one with the median LCP. A workflow without LCP uses the first of FCP, TTFB and the navigation time that it measures, and `--median-by <metric>` picks another one (`LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`). With an even count it takes the lower of the two middle runs, so the report is always a run that happened. The report adds the median, min and max of each metric across the runs, and the JSON output carries them in `runs`:
+
+```json
+"runs": {
+  "count": 5,
+  "failed": 0,
+  "medianRun": 3,
+  "medianBy": "LCP",
+  "summary": {
+    "LCP": { "median": 2140, "min": 1980, "max": 2610, "values": [2610, 2010, 2140, 1980, 2290], "unit": "ms" }
+  }
+}
+```
+
+`medianRun` counts from 1, and `values` keeps the order of the runs. The budgets compare the median of each metric, not one run. A run that fails, such as a navigation that does not finish, counts in `failed` and is left out of the median; when every run fails, the command exits with code `1`. Each run is reported on stderr as it starts, so stdout keeps only the report. An invalid count or metric exits with code `2`.
+
+Measure several pages, on several devices and in several variants, in one command with `--matrix <file>`. The file lists them, and the CLI measures every URL on every device and in every variant, each combination `runs` times, one load after the other:
+
+```json
+{
+  "urls": {
+    "home": "https://example.com/",
+    "article": "https://example.com/news/some-article"
+  },
+  "devices": {
+    "mobile": { "viewport": "mobile", "cpu": 4, "network": "slow-4g" },
+    "high-end": { "viewport": "mobile", "cpu": 1, "network": { "downloadKbps": 50000, "uploadKbps": 10000, "latencyMs": 20 } },
+    "desktop": { "viewport": "desktop" }
+  },
+  "variants": {
+    "no-consent": {},
+    "consent": { "storageState": "states/consent.json" }
+  },
+  "runs": 3
+}
+```
+
+```bash
+npx webperf-snippets --matrix matrix.json --budget-lcp 2500
+```
+
+- `urls`: names and URLs. An array of URLs also works, and names each one by its path and query (`/`, `/news/`), with the host when the URLs have more than one.
+- `devices`: the `viewport` preset, the `cpu` factor and the `network`, a preset (`fast-4g`, `slow-4g`, `3g`) or a network of its own in kilobits per second and milliseconds, for the ones DevTools has no preset for, such as 5G. An array of viewport presets (`["mobile", "desktop"]`) is a shortcut for devices without throttling. Without `devices`, the matrix measures a mid-range phone, with CPU 4x and Slow 4G, as Lighthouse does.
+- `variants`: the conditions of each variant, `storageState` (relative to the file), `headers` and `ownDomains`. Without `variants`, the combinations are the pages on the devices: the report has no Variant column, and the `id` of a combination is its page and device (`/news/ · mobile`).
+- `runs` (1 to 9) and `medianBy`, which `--runs` and `--median-by` replace.
+
+The options of the command line apply to every combination: `--workflow`, `--snippet`, `--wait`, `--interact-script`, `--header`, `--own-domains`, `--storage-state`, `--cpu`, `--network` and the budgets. The file replaces `cpu`, `network` and `storageState` where it sets them, and its `headers` and `ownDomains` add to the ones of the command line, the variant winning on the same header. A matrix file is meant to be versioned, so pass secrets, such as the token of an access proxy, with `--header` and an environment variable; a header of the file whose name looks like a secret raises a warning. A key the CLI does not know is an error, so a typo does not measure without its condition.
+
+The report is a table with one row per combination and the rated median of each metric, followed by the errors that decide the exit code; `--verbose` adds the report of the median run of each combination. `--markdown` gives the same table and folds the report of each combination into a `<details>` block, to keep a pull request comment readable. With `--json`, it is `{ "matrix", "runs", "combinations" }`, each combination being the report of its median run with its `id`, `name`, `device`, `variant` (when the file defines variants) and `runs`. The budgets compare the median of each combination. The command exits with code `1` when a combination exceeds a budget, has an error or fails every run, and with code `2` for an invalid file, before any page load, naming the place of the problem in the JSON. `--viewport` does not apply, since each device sets its own.
+
+With CrUX, the CLI asks once for each URL and form factor, so the variants of a device share their field data. With `--report-to`, it posts one report for each combination, the body of one page with `meta.device`, `meta.variant` and `meta.runs`; the headers of the command line and of the file are conditions of the load, and no report carries their values.
+
 CI gating:
 
 ```bash
@@ -200,6 +258,9 @@ npx webperf-snippets https://example.com --budget-lcp 2500 --budget-cls 0.1
 | `--storage-state <path>`     | Playwright storage state (cookies and localStorage) for pages that require authentication. |
 | `--header "Name: value"`     | Request header for the origin of the URL only, such as the token of an access proxy. Repeat it for more headers. |
 | `--own-domains <list>`       | Domains of the site to count as first party, such as its own CDN. Comma-separated or repeated. |
+| `--runs <n>`                 | Measure the page n times, from 1 to 9, one after the other, and report the median run. |
+| `--median-by <metric>`       | Metric that picks the median run: `LCP`, `FCP`, `TTFB`, `CLS`, `INP` or `navMs`. Default: the first of LCP, FCP, TTFB and `navMs` measured. |
+| `--matrix <file>`            | Measure every URL of a JSON file on every device and in every variant it lists, and report a table of the medians. |
 | `--crux-key <key>`           | Add CrUX field data (p75) next to the measured values. Prefer `CRUX_API_KEY`. |
 | `--report-to <url>`          | POST the results to this `https` URL after the run. No external calls without it. |
 | `--api-key <key>`            | Sent as a Bearer token with `--report-to`. Prefer `PERF_REVIEWS_API_KEY`. |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webperf-snippets",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "mcpName": "io.github.nucliweb/webperf-snippets",
   "description": "Run curated WebPerf Snippets headlessly via Playwright. Diagnose Core Web Vitals beyond what Lighthouse exposes.",
   "type": "module",

package/src/bin.js

@@ -6,18 +6,22 @@ import { resolveSnippetName } from "./snippet-names.js";
 import { runSnippets, runMeasurement, VIEWPORT_PRESETS } from "./runner.js";
 import { WORKFLOWS } from "./workflows/index.js";
 import { RULES } from "./decision-tree.js";
-import { reportHuman } from "./reporters/human.js";
+import { reportHuman, reportMatrixHuman } from "./reporters/human.js";
 import { reportJson } from "./reporters/json.js";
-import { reportMarkdown } from "./reporters/markdown.js";
+import { reportMarkdown, reportMatrixMarkdown } from "./reporters/markdown.js";
 import { cliVersion, reportTo, validateReportUrl } from "./report-to.js";
 import { fetchCrux, formFactorFor } from "./crux.js";
 import { MissingBrowserError } from "./browser.js";
 import { parseHeader } from "./headers.js";
 import { NETWORK_PRESETS, parseCpuFactor } from "./throttling.js";
 import { parseOwnDomains } from "./own-domains.js";
+import { MEDIAN_METRICS, summarizeRuns } from "./runs.js";
+import { loadMatrix } from "./matrix.js";
+import { blockingErrors } from "./blocking.js";
 
 
 const USAGE = `webperf-snippets <url> [options]
+webperf-snippets --matrix <file> [options]
 webperf-snippets mcp
 
 Run curated WebPerf Snippets headlessly via Playwright.
@@ -53,6 +57,12 @@ Options:
                             the token of an access proxy. Repeat it for more headers
   --own-domains <list>      Domains of the site to count as first party, such as its
                             own CDN, comma-separated or repeated
+  --runs <n>                Measure the page n times (1 to 9), one after the other, and
+                            report the median run
+  --matrix <file>           Measure every URL of a JSON file on every device and in every
+                            variant it lists, and report a table of the medians
+  --median-by <metric>      Metric that picks the median run: LCP, FCP, TTFB, CLS, INP,
+                            navMs (default: the first of LCP, FCP, TTFB, navMs measured)
   --crux-key <key>          Add CrUX field data (p75) next to the measured values.
                             Also read from the CRUX_API_KEY environment variable
   --report-to <url>         POST the results to this https URL after the run
@@ -74,6 +84,8 @@ Examples:
   npx webperf-snippets https://example.com --budget-lcp 2500
   npx webperf-snippets https://example.com --cpu 4 --network slow-4g
   npx webperf-snippets https://example.com --workflow loading --own-domains cdn.example.net
+  npx webperf-snippets https://example.com --runs 5 --budget-lcp 2500
+  npx webperf-snippets --matrix matrix.json --runs 3 --budget-lcp 2500
   npx webperf-snippets https://example.com --snippet INP --interact-script interactions.json
   npx webperf-snippets https://example.com/dashboard --storage-state auth.json
   npx webperf-snippets https://review.example.com --header "CF-Access-Client-Id: $CF_ID" --header "CF-Access-Client-Secret: $CF_SECRET"
@@ -94,11 +106,25 @@ function buildSnippetItem(values) {
   return [{ id: values.snippet, path, source: loadSnippet(path) }];
 }
 
-function checkBudgets(results, values) {
+// The count given to --runs, or null when it is not a whole number from 1 to 9
+function parseRunCount(raw) {
+  return /^[1-9]$/.test(raw.trim()) ? Number(raw) : null;
+}
+
+function checkBudgets({ results, runs }, values) {
   const violations = [];
   const lcpBudget = values["budget-lcp"] ? Number(values["budget-lcp"]) : null;
   const clsBudget = values["budget-cls"] ? Number(values["budget-cls"]) : null;
 
+  // With several runs the budgets compare the median of each metric, not one sample
+  if (runs) {
+    const lcp = runs.summary.LCP?.median;
+    const cls = runs.summary.CLS?.median;
+    if (lcp != null && lcpBudget != null && lcp > lcpBudget) violations.push(`LCP ${lcp}ms exceeds budget ${lcpBudget}ms`);
+    if (cls != null && clsBudget != null && cls > clsBudget) violations.push(`CLS ${cls} exceeds budget ${clsBudget}`);
+    return violations;
+  }
+
   for (const r of results) {
     if (r.status !== "ok") continue;
     if (r.metric === "LCP" && lcpBudget != null && r.value > lcpBudget) {
@@ -137,6 +163,9 @@ async function main() {
         "storage-state": { type: "string" },
         header: { type: "string", multiple: true },
         "own-domains": { type: "string", multiple: true },
+        matrix: { type: "string" },
+        runs: { type: "string" },
+        "median-by": { type: "string" },
         "crux-key": { type: "string" },
         "report-to": { type: "string" },
         "api-key": { type: "string" },
@@ -158,10 +187,13 @@ async function main() {
   }
 
   const url = positionals[0];
-  if (!url) {
+  const matrixPath = values.matrix;
+  if (url && matrixPath) fail("Use a URL or --matrix, not both");
+  if (!url && !matrixPath) {
     process.stdout.write(USAGE);
     process.exit(2);
   }
+  if (matrixPath && values.viewport) fail("--viewport does not apply to --matrix: set the viewport of each device in the file");
 
   const waitMs = values.wait ? Number(values.wait) : 3000;
   const viewportName = values.viewport ?? "mobile";
@@ -202,15 +234,42 @@ async function main() {
   }
   const ownDomains = values["own-domains"] && parseOwnDomains(values["own-domains"]);
 
+  const runCount = values.runs != null ? parseRunCount(values.runs) : undefined;
+  if (runCount === null) fail(`Invalid --runs "${values.runs}". Use a whole number from 1 to 9`);
+  const medianBy = values["median-by"];
+  if (medianBy != null && !MEDIAN_METRICS.includes(medianBy)) {
+    fail(`Unknown --median-by "${medianBy}". Choose from: ${MEDIAN_METRICS.join(", ")}`);
+  }
+  if (medianBy != null && runCount == null && !matrixPath) fail("--median-by needs --runs");
+
+  // What each page load runs, a snippet or a workflow, under the conditions of the load
+  const items = values.snippet ? buildSnippetItem(values) : null;
+  const workflow = values.snippet ? null : WORKFLOWS[values.workflow ?? "core-web-vitals"];
+  if (!values.snippet && !workflow) fail(`Unknown workflow: ${values.workflow}`);
+  const measureWith = (conditions) => {
+    const options = { waitMs, headless: !values.headed, interactScript, ...conditions };
+    return items ? runSnippets({ ...options, items }) : runMeasurement({ ...options, workflow, rules: RULES });
+  };
+
+  if (matrixPath) {
+    let matrix;
+    try {
+      matrix = loadMatrix(matrixPath, { cpu, network, storageState, headers, ownDomains, runs: runCount, medianBy });
+    } catch (err) {
+      fail(err.message);
+    }
+    await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl });
+    return;
+  }
+
+  const conditions = { url, viewport, storageState, headers, cpu, network, ownDomains };
   let payload;
-  if (values.snippet) {
-    const items = buildSnippetItem(values);
-    payload = await runSnippets({ url, items, waitMs, headless: !values.headed, viewport, interactScript, storageState, headers, cpu, network, ownDomains });
+  if (runCount == null) {
+    payload = await measureWith(conditions);
   } else {
-    const workflowName = values.workflow ?? "core-web-vitals";
-    const workflow = WORKFLOWS[workflowName];
-    if (!workflow) fail(`Unknown workflow: ${workflowName}`);
-    payload = await runMeasurement({ url, workflow, rules: RULES, waitMs, headless: !values.headed, viewport, interactScript, storageState, headers, cpu, network, ownDomains });
+    const median = await measureRuns(() => measureWith(conditions), runCount, medianBy, (i) => `[${i}/${runCount}] ${url} · run ${i}/${runCount}`);
+    if (!median.payload) throw new Error(`Every run failed: ${median.errors.join("; ")}`);
+    payload = { ...median.payload, runs: median.runs };
   }
 
   const cruxKey = values["crux-key"] || process.env.CRUX_API_KEY;
@@ -238,7 +297,7 @@ async function main() {
         timestamp: new Date().toISOString(),
         navMs: payload.navMs,
         results: payload.results,
-        meta: { viewport: viewportName, waitMs, ...(payload.throttling ? { throttling: payload.throttling } : {}), cli_version: cliVersion },
+        meta: { viewport: viewportName, waitMs, ...(payload.throttling ? { throttling: payload.throttling } : {}), ...(payload.runs ? { runs: payload.runs } : {}), cli_version: cliVersion },
       },
     });
     if (!outcome.ok) {
@@ -249,21 +308,110 @@ async function main() {
   }
 
   // Exit codes.
-  const violations = checkBudgets(payload.results, values);
+  const violations = checkBudgets(payload, values);
   if (violations.length > 0) {
     if (!values.json) {
       for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
     }
     process.exit(1);
   }
+  process.exit(blockingErrors(payload.results).length > 0 ? 1 : 0);
+}
+
+// Measures `count` runs one after the other, never in parallel: they would compete for the CPU and
+// skew the timings. Each run is reported on stderr as it starts, so stdout keeps only the report. A
+// run that fails is kept as { error } and left out of the median.
+async function measureRuns(measure, count, medianBy, progress) {
+  const runs = [];
+  for (let i = 1; i <= count; i++) {
+    process.stderr.write(`${progress(i)}\n`);
+    try {
+      runs.push(await measure());
+    } catch (err) {
+      if (err instanceof MissingBrowserError) throw err;
+      runs.push({ error: err.message });
+    }
+  }
+  return summarizeRuns(runs, { medianBy });
+}
+
+// Every combination of the matrix, in order, each one `matrix.runs` times. Each combination reports
+// its median run, and the exit code follows the same rules as one page, for every combination.
+async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl }) {
+  for (const warning of matrix.warnings) process.stderr.write(`Warning: ${warning}\n`);
+  const viewportOf = new Map(matrix.combinations.map((c) => [c.id, c.viewport]));
+
+  const total = matrix.combinations.length * matrix.runs;
+  const width = String(total).length;
+  let started = 0;
+  const combinations = [];
+  for (const c of matrix.combinations) {
+    const conditions = { url: c.url, viewport: VIEWPORT_PRESETS[c.viewport], storageState: c.storageState, headers: c.headers, cpu: c.cpu, network: c.network, ownDomains: c.ownDomains };
+    const median = await measureRuns(() => measureWith(conditions), matrix.runs, matrix.medianBy, (i) => `[${String(++started).padStart(width)}/${total}] ${c.id} · run ${i}/${matrix.runs}`);
+    const identity = { id: c.id, name: c.name, device: c.device, ...(c.variant ? { variant: c.variant } : {}), url: c.url };
+    combinations.push(
+      median.payload
+        ? { ...identity, ...median.payload, runs: median.runs }
+        : { ...identity, status: "error", error: `Every run failed: ${median.errors.join("; ")}`, runs: { count: matrix.runs, failed: matrix.runs } }
+    );
+  }
+
+  // The field data of a URL depends on the form factor, not on the variant: one query for each pair
+  const cruxKey = values["crux-key"] || process.env.CRUX_API_KEY;
+  if (cruxKey) {
+    const queries = new Map();
+    for (const c of combinations.filter((c) => !c.error)) {
+      const formFactor = formFactorFor(viewportOf.get(c.id));
+      const key = `${c.url} ${formFactor}`;
+      if (!queries.has(key)) queries.set(key, fetchCrux({ url: c.url, apiKey: cruxKey, formFactor }));
+      const crux = await queries.get(key);
+      if (crux) c.crux = crux;
+    }
+  }
 
-  const anyError = payload.results.some((r) => r.status === "error");
-  // A follow-up (a result with a reason) only informs: its error issues show in the report and
-  // leave the exit code to the budgets and to the steps the workflow runs itself.
-  const anyAuditViolation = payload.results.some(
-    (r) => !r.reason && Array.isArray(r.issues) && r.issues.some((i) => i.severity === "error"),
-  );
-  process.exit(anyError || anyAuditViolation ? 1 : 0);
+  const report = { matrix: matrixPath, runs: matrix.runs, combinations, verbose: values.verbose };
+  const output = values.markdown ? reportMatrixMarkdown(report) : values.json ? reportJson({ matrix: matrixPath, runs: matrix.runs, combinations }) : reportMatrixHuman(report);
+  process.stdout.write(output + "\n");
+
+  // One report for each combination, with the body of one page and the device, variant and runs of
+  // the combination. The headers are conditions of the load, never part of the body.
+  if (reportUrl) {
+    for (const c of combinations.filter((c) => !c.error)) {
+      const outcome = await reportTo({
+        url: reportUrl,
+        apiKey: values["api-key"] || process.env.PERF_REVIEWS_API_KEY,
+        body: {
+          url: c.url,
+          workflow: values.snippet ? null : (values.workflow ?? "core-web-vitals"),
+          ...(values.snippet ? { snippet: values.snippet } : {}),
+          timestamp: new Date().toISOString(),
+          navMs: c.navMs,
+          results: c.results,
+          meta: {
+            viewport: viewportOf.get(c.id),
+            device: c.device,
+            ...(c.variant ? { variant: c.variant } : {}),
+            waitMs,
+            ...(c.throttling ? { throttling: c.throttling } : {}),
+            runs: c.runs,
+            cli_version: cliVersion,
+          },
+        },
+      });
+      if (!outcome.ok) {
+        process.stderr.write(`Warning: could not report ${c.id} to ${new URL(reportUrl).origin} (${outcome.warning})\n`);
+      } else if (outcome.regressions.length > 0) {
+        process.stderr.write(`Regressions detected in ${c.id}:\n${outcome.regressions.map((r) => `  - ${typeof r === "string" ? r : JSON.stringify(r)}`).join("\n")}\n`);
+      }
+    }
+  }
+
+  const violations = combinations.filter((c) => !c.error).flatMap((c) => checkBudgets(c, values).map((v) => `${v} (${c.id})`));
+  if (!values.json) {
+    for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
+  }
+  const failed = combinations.some((c) => c.error || blockingErrors(c.results).length > 0);
+  process.exit(violations.length > 0 || failed ? 1 : 0);
 }
 
 main().catch((err) => {

package/src/blocking.js

@@ -0,0 +1,12 @@
+// The results that make the command exit with 1: a snippet that failed, and an issue of severity
+// error in a step the workflow runs itself. A follow-up (a result with a reason) only informs: its
+// error issues show in the report and leave the exit code to the budgets and to the steps.
+export function blockingErrors(results) {
+  const errors = [];
+  for (const r of results) {
+    if (r.status === "error") errors.push({ id: r.id, message: r.error });
+    if (r.reason || !Array.isArray(r.issues)) continue;
+    for (const issue of r.issues) if (issue.severity === "error") errors.push({ id: r.id, message: issue.message });
+  }
+  return errors;
+}

package/src/crux.js

@@ -1,16 +1,16 @@
+import { rate } from "./ratings.js";
+
 const DEFAULT_ENDPOINT = "https://chromeuxreport.googleapis.com/v1/records:queryRecord";
 const CRUX_TIMEOUT_MS = 10000;
 
-// Good / needs-improvement upper bounds from web.dev/articles/vitals.
 const METRICS = [
-  { name: "LCP", key: "largest_contentful_paint", unit: "ms", good: 2500, poor: 4000 },
-  { name: "CLS", key: "cumulative_layout_shift", unit: "score", good: 0.1, poor: 0.25 },
-  { name: "INP", key: "interaction_to_next_paint", unit: "ms", good: 200, poor: 500 },
+  { name: "LCP", key: "largest_contentful_paint", unit: "ms" },
+  { name: "CLS", key: "cumulative_layout_shift", unit: "score" },
+  { name: "INP", key: "interaction_to_next_paint", unit: "ms" },
 ];
 
 const FORM_FACTOR_LABEL = { PHONE: "mobile", DESKTOP: "desktop", TABLET: "tablet" };
 
-const rate = (value, { good, poor }) => (value <= good ? "good" : value <= poor ? "needs-improvement" : "poor");
 
 // The viewport preset decides which population the field data describes.
 export function formFactorFor(viewportName) {
@@ -22,7 +22,7 @@ function parseMetrics(record) {
   for (const m of METRICS) {
     // CLS p75 comes as a string ("0.08"); LCP and INP as numbers.
     const p75 = Number(record?.metrics?.[m.key]?.percentiles?.p75);
-    if (Number.isFinite(p75)) metrics[m.name] = { p75, rating: rate(p75, m), unit: m.unit };
+    if (Number.isFinite(p75)) metrics[m.name] = { p75, rating: rate(m.name, p75), unit: m.unit };
   }
   return metrics;
 }

package/src/matrix.js

@@ -0,0 +1,227 @@
+// A matrix file: several pages, measured on several devices and in several variants, each
+// combination one or more times. This module reads, validates and expands the file into the
+// combinations to measure, with the conditions of each one already combined with the flags of the
+// command line. It opens no browser, so an invalid file fails before any page load.
+//
+// {
+//   "urls": { "home": "https://example.com/" },              or an array of URLs
+//   "devices": { "mobile": { "viewport": "mobile", "cpu": 4, "network": "slow-4g" } },
+//   "variants": { "consent": { "storageState": "consent.json", "headers": {}, "ownDomains": [] } },
+//   "runs": 3,
+//   "medianBy": "LCP"
+// }
+//
+// The conditions of the file replace cpu, network and storageState of the command line where they
+// set them; headers and own domains add to the ones of the command line, which carries the secrets.
+import { existsSync, readFileSync } from "node:fs";
+import { dirname, resolve } from "node:path";
+import { VIEWPORT_PRESETS } from "./runner.js";
+import { resolveNetwork } from "./throttling.js";
+import { parseHeader } from "./headers.js";
+import { parseOwnDomains } from "./own-domains.js";
+import { MEDIAN_METRICS } from "./runs.js";
+
+const KEYS = {
+  matrix: ["urls", "devices", "variants", "runs", "medianBy"],
+  device: ["viewport", "cpu", "network"],
+  variant: ["storageState", "headers", "ownDomains"],
+};
+
+// Without devices, a mid-range phone, as Lighthouse measures by default
+const DEFAULT_DEVICE = { viewport: "mobile", cpu: 4, network: "slow-4g" };
+
+// A header whose name says it carries a secret, which a versioned file should not hold
+const LOOKS_SECRET = /secret|token|authorization|key|cookie/i;
+
+class MatrixError extends Error {
+  constructor(problem) {
+    super(`Invalid matrix: ${problem}`);
+  }
+}
+
+const isObject = (value) => value !== null && typeof value === "object" && !Array.isArray(value);
+
+function checkKeys(object, known, at) {
+  for (const key of Object.keys(object)) {
+    if (!known.includes(key)) {
+      throw new MatrixError(`unknown key "${at ? `${at}.` : ""}${key}". Known keys: ${known.join(", ")}`);
+    }
+  }
+}
+
+// The URLs by name. An array names each URL by its path and query, with the host when there are
+// several, which is what a reader recognises in a table of 5 rows or of 50.
+function readUrls(urls) {
+  if (urls === undefined) throw new MatrixError("urls is required");
+  const entries = Array.isArray(urls) ? urls.map((url, i) => [`[${i}]`, url]) : isObject(urls) ? Object.entries(urls).map(([name, url]) => [`.${name}`, url, name]) : null;
+  if (!entries) throw new MatrixError("urls must be an object of names and URLs, or an array of URLs");
+  if (entries.length === 0) throw new MatrixError("urls has no URL");
+
+  const parsed = entries.map(([at, url, name]) => {
+    let parsedUrl;
+    try {
+      parsedUrl = new URL(url);
+    } catch {
+      parsedUrl = null;
+    }
+    if (typeof url !== "string" || !parsedUrl || !["http:", "https:"].includes(parsedUrl.protocol)) {
+      throw new MatrixError(`urls${at} "${url}" is not an http or https URL`);
+    }
+    return { url, parsedUrl, name };
+  });
+  if (!Array.isArray(urls)) return parsed.map(({ name, url }) => ({ name, url }));
+
+  const severalHosts = new Set(parsed.map(({ parsedUrl }) => parsedUrl.host)).size > 1;
+  const named = parsed.map(({ url, parsedUrl }) => ({
+    name: `${severalHosts ? parsedUrl.host : ""}${parsedUrl.pathname}${parsedUrl.search}`,
+    url,
+  }));
+  const seen = new Set();
+  for (const { name, url } of named) {
+    if (seen.has(name)) {
+      const suggestion = name.replace(/[^a-z0-9]+/gi, "-").replace(/^-|-$/g, "") || "home";
+      throw new MatrixError(`two URLs of urls are named "${name}". Name them with an object, such as { "${suggestion}": "${url}" }`);
+    }
+    seen.add(name);
+  }
+  return named;
+}
+
+function checkViewport(viewport, at) {
+  if (!VIEWPORT_PRESETS[viewport]) {
+    throw new MatrixError(`${at} "${viewport}" is not a preset. Choose from: ${Object.keys(VIEWPORT_PRESETS).join(", ")}`);
+  }
+}
+
+// The devices by name. An array of viewport presets is a shortcut for devices without throttling.
+function readDevices(devices, flags) {
+  if (devices === undefined) {
+    return [{ name: "mobile", viewport: DEFAULT_DEVICE.viewport, cpu: flags.cpu ?? DEFAULT_DEVICE.cpu, network: flags.network ?? DEFAULT_DEVICE.network }];
+  }
+  if (Array.isArray(devices) ? devices.length === 0 : isObject(devices) && Object.keys(devices).length === 0) {
+    throw new MatrixError("devices has no device");
+  }
+  if (Array.isArray(devices)) {
+    devices.forEach((viewport, i) => checkViewport(viewport, `devices[${i}]`));
+    return devices.map((viewport) => ({ name: viewport, viewport, cpu: flags.cpu, network: flags.network }));
+  }
+  if (!isObject(devices)) throw new MatrixError("devices must be an object of names and devices, or an array of viewport presets");
+
+  return Object.entries(devices).map(([name, device]) => {
+    const at = `devices.${name}`;
+    if (!isObject(device)) throw new MatrixError(`${at} must be an object, such as { "viewport": "mobile", "cpu": 4, "network": "slow-4g" }`);
+    checkKeys(device, KEYS.device, at);
+    const viewport = device.viewport ?? "mobile";
+    checkViewport(viewport, `${at}.viewport`);
+    if (device.cpu !== undefined && !(typeof device.cpu === "number" && device.cpu >= 1)) {
+      throw new MatrixError(`${at}.cpu must be a number of at least 1, such as 4`);
+    }
+    if (device.network !== undefined) {
+      try {
+        resolveNetwork(device.network);
+      } catch (err) {
+        throw new MatrixError(`${at}.network: ${err.message}`);
+      }
+    }
+    return { name, viewport, cpu: device.cpu ?? flags.cpu, network: device.network ?? flags.network };
+  });
+}
+
+// The variants by name. Without variants, one variant with no name and no conditions of its own,
+// so the combinations are the pages on the devices and the reports leave the variant out.
+function readVariants(variants, base, warnings) {
+  if (variants === undefined) return [{ headers: {}, ownDomains: [] }];
+  if (!isObject(variants)) throw new MatrixError("variants must be an object of names and variants");
+  if (Object.keys(variants).length === 0) throw new MatrixError("variants has no variant");
+
+  return Object.entries(variants).map(([name, variant]) => {
+    const at = `variants.${name}`;
+    if (!isObject(variant)) throw new MatrixError(`${at} must be an object, such as { "storageState": "consent.json" }`);
+    checkKeys(variant, KEYS.variant, at);
+
+    let storageState;
+    if (variant.storageState !== undefined) {
+      storageState = resolve(base, String(variant.storageState));
+      if (!existsSync(storageState)) throw new MatrixError(`${at}.storageState "${variant.storageState}" not found at ${storageState}`);
+    }
+
+    const headers = {};
+    if (variant.headers !== undefined) {
+      if (!isObject(variant.headers)) throw new MatrixError(`${at}.headers must be an object of header names and values`);
+      for (const [header, value] of Object.entries(variant.headers)) {
+        if (!parseHeader(`${header}: `)) throw new MatrixError(`${at}.headers "${header}" is not a valid header`);
+        if (typeof value !== "string") throw new MatrixError(`${at}.headers "${header}" must be a string`);
+        if (LOOKS_SECRET.test(header)) {
+          warnings.push(`${at}.headers.${header} looks like a secret, and a matrix file is usually versioned. Pass it with --header "${header}: $VARIABLE" instead.`);
+        }
+        headers[header] = value;
+      }
+    }
+
+    let ownDomains = [];
+    if (variant.ownDomains !== undefined) {
+      const parsed = Array.isArray(variant.ownDomains) && variant.ownDomains.every((d) => typeof d === "string") ? parseOwnDomains(variant.ownDomains) : null;
+      if (!parsed) throw new MatrixError(`${at}.ownDomains must be a list of domains without spaces`);
+      ownDomains = parsed;
+    }
+
+    return { name, storageState, headers, ownDomains };
+  });
+}
+
+// The headers of the command line and of the variant; the variant wins on the same header, whatever its case
+function mergeHeaders(fromFlags = {}, fromVariant) {
+  const names = new Set(Object.keys(fromVariant).map((name) => name.toLowerCase()));
+  const kept = Object.entries(fromFlags).filter(([name]) => !names.has(name.toLowerCase()));
+  return Object.fromEntries([...kept, ...Object.entries(fromVariant)]);
+}
+
+// The file at `path`, with the flags of the command line: cpu, network, storageState, headers,
+// ownDomains, runs and medianBy. Returns the runs, the metric of the median, the combinations in the
+// order they run (URL, then device, then variant) and the warnings to print.
+export function loadMatrix(path, flags = {}) {
+  if (!existsSync(path)) throw new Error(`Matrix file not found: ${path}`);
+  let matrix;
+  try {
+    matrix = JSON.parse(readFileSync(path, "utf8"));
+  } catch (err) {
+    throw new MatrixError(`the file is not valid JSON (${err.message})`);
+  }
+  if (!isObject(matrix)) throw new MatrixError("the file must hold an object, such as { \"urls\": [\"https://example.com/\"] }");
+  checkKeys(matrix, KEYS.matrix);
+
+  if (matrix.runs !== undefined && !(Number.isInteger(matrix.runs) && matrix.runs >= 1 && matrix.runs <= 9)) {
+    throw new MatrixError("runs must be a whole number from 1 to 9");
+  }
+  if (matrix.medianBy !== undefined && !MEDIAN_METRICS.includes(matrix.medianBy)) {
+    throw new MatrixError(`medianBy "${matrix.medianBy}" is not a metric. Choose from: ${MEDIAN_METRICS.join(", ")}`);
+  }
+
+  const warnings = [];
+  const urls = readUrls(matrix.urls);
+  const devices = readDevices(matrix.devices, flags);
+  const variants = readVariants(matrix.variants, dirname(resolve(path)), warnings);
+
+  const combinations = [];
+  for (const { name, url } of urls) {
+    for (const device of devices) {
+      for (const variant of variants) {
+        combinations.push({
+          id: [name, device.name, variant.name].filter(Boolean).join(" · "),
+          name,
+          url,
+          device: device.name,
+          ...(variant.name ? { variant: variant.name } : {}),
+          viewport: device.viewport,
+          cpu: device.cpu,
+          network: device.network,
+          storageState: variant.storageState ?? flags.storageState,
+          headers: mergeHeaders(flags.headers, variant.headers),
+          ownDomains: [...(flags.ownDomains ?? []), ...variant.ownDomains],
+        });
+      }
+    }
+  }
+
+  return { runs: flags.runs ?? matrix.runs ?? 1, medianBy: flags.medianBy ?? matrix.medianBy, combinations, warnings };
+}

package/src/ratings.js

@@ -0,0 +1,16 @@
+// Good / needs-improvement upper bounds from web.dev/articles/vitals, and the ones web.dev gives for
+// FCP and TTFB, the same the snippets rate with.
+export const THRESHOLDS = {
+  LCP: { good: 2500, poor: 4000 },
+  FCP: { good: 1800, poor: 3000 },
+  TTFB: { good: 800, poor: 1800 },
+  CLS: { good: 0.1, poor: 0.25 },
+  INP: { good: 200, poor: 500 },
+};
+
+// "good", "needs-improvement" or "poor", or undefined for a value without thresholds
+export function rate(metric, value) {
+  const thresholds = THRESHOLDS[metric];
+  if (!thresholds) return undefined;
+  return value <= thresholds.good ? "good" : value <= thresholds.poor ? "needs-improvement" : "poor";
+}

package/src/reporters/human.js

@@ -1,6 +1,8 @@
 import { styleText } from "node:util";
 import { fieldDataView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
+import { describeRuns } from "../runs.js";
+import { matrixView } from "./matrix-view.js";
 
 const RATING_ICON = {
   good: "🟢",
@@ -203,11 +205,22 @@ function renderFieldData(crux, results) {
   return lines;
 }
 
-export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, throttling }) {
+// The runs line, and the median, min and max of each metric across the runs
+function renderRuns(runs) {
+  const lines = [styleText("dim", describeRuns(runs))];
+  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
+    const cell = (label, value) => pad(`${label} ${formatValue(value, unit)}`, 16);
+    lines.push(styleText("dim", `  ${pad(metric, 6)}${cell("median", median)}${cell("min", min)}${cell("max", max).trimEnd()}`));
+  }
+  return lines;
+}
+
+export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, throttling, runs, title = `WebPerf Snippets — ${url}` }) {
   const lines = [];
-  lines.push(styleText(["bold"], `WebPerf Snippets — ${url}`));
+  lines.push(styleText(["bold"], title));
   lines.push(styleText("dim", `Navigated in ${navMs}ms`));
   if (throttling) lines.push(styleText("dim", `Throttling: ${describeThrottling(throttling)}`));
+  if (runs) lines.push(...renderRuns(runs));
   lines.push("");
   for (const r of results) lines.push(renderResult(r, verbose));
   if (crux) {
@@ -221,3 +234,39 @@ export function reportHuman({ url, navMs, results, pageErrors, verbose, crux, th
   }
   return lines.join("\n");
 }
+
+// One row per combination with the rated median of each metric, and below it the errors that decide
+// the exit code. With verbose, the report of the median run of each combination follows.
+export function reportMatrixHuman(report) {
+  const { file, summary, hasVariants, metrics, rows, errors } = matrixView(report);
+  const columns = [["name", "Page"], ["device", "Device"], ...(hasVariants ? [["variant", "Variant"]] : [])];
+  const widths = columns.map(([key, label]) => Math.max(...report.combinations.map((c) => c[key].length), label.length) + 3);
+  const identity = (values) => values.map((value, i) => pad(value, widths[i])).join("");
+
+  const lines = [];
+  lines.push(styleText(["bold"], `WebPerf Snippets — ${file} (${summary})`));
+  lines.push("");
+  lines.push(styleText("dim", `  ${identity(columns.map(([, label]) => label))}${metrics.map((m) => pad(m, 14)).join("").trimEnd()}`));
+  for (const { combination: c, cells } of rows) {
+    const row = `  ${identity(columns.map(([key]) => c[key]))}`;
+    if (!cells) {
+      lines.push(row + styleText("red", "✗ every run failed"));
+      continue;
+    }
+    const drawn = cells.map((cell) => (cell ? pad(paint(cell.rating, `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}`), 14) : pad("·", 14)));
+    lines.push(row + drawn.join("").trimEnd());
+  }
+
+  if (errors.length) {
+    lines.push("");
+    lines.push(styleText("red", "Errors:"));
+    for (const e of errors) lines.push(`  ${styleText("red", "✗")} ${e}`);
+  }
+
+  if (report.verbose) {
+    for (const c of report.combinations.filter((c) => !c.error)) {
+      lines.push("", reportHuman({ ...c, verbose: true, title: `${c.id} — ${c.url}` }));
+    }
+  }
+  return lines.join("\n");
+}

package/src/reporters/markdown.js

@@ -1,5 +1,7 @@
 import { fieldDataView } from "../crux.js";
 import { describeThrottling } from "../throttling.js";
+import { describeRuns } from "../runs.js";
+import { matrixView } from "./matrix-view.js";
 
 const RATING_ICON = {
   good: "✅",
@@ -72,12 +74,24 @@ function renderFieldData(crux, results) {
   ];
 }
 
-export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttling }) {
+// The median, min and max of each metric across the runs
+function renderRuns(runs) {
+  const lines = ["| Metric | Median | Min | Max |", "|--------|--------|-----|-----|"];
+  for (const [metric, { median, min, max, unit }] of Object.entries(runs.summary)) {
+    lines.push(`| ${metric} | ${formatValue(median, unit)} | ${formatValue(min, unit)} | ${formatValue(max, unit)} |`);
+  }
+  return lines;
+}
+
+// With a null title, the report has no heading, as inside the details block of a matrix
+export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttling, runs, title = `WebPerf Results — ${url}` }) {
   const lines = [];
-  lines.push(`## WebPerf Results — ${url}`);
+  if (title) lines.push(`## ${title}`);
   lines.push(`> Navigated in ${navMs}ms`);
   if (throttling) lines.push(">", `> Throttling: ${describeThrottling(throttling)}`);
+  if (runs) lines.push(">", `> ${describeRuns(runs)}`);
   lines.push("");
+  if (runs) lines.push(...renderRuns(runs), "");
 
   const initial = results.filter((r) => !r.reason);
   if (initial.length > 0) {
@@ -108,3 +122,31 @@ export function reportMarkdown({ url, navMs, results, pageErrors, crux, throttli
 
   return lines.join("\n");
 }
+
+// A table with the rated median of each combination, the errors that decide the exit code, and the
+// report of the median run of each combination folded into a details block, to keep a pull request
+// comment readable.
+export function reportMatrixMarkdown(report) {
+  const { file, summary, hasVariants, metrics, rows, errors } = matrixView(report);
+  const headings = ["Page", "Device", ...(hasVariants ? ["Variant"] : []), ...metrics];
+  const row = (parts) => `|${parts.map((part) => (part ? ` ${part} ` : " ")).join("|")}|`;
+
+  const lines = [`## WebPerf Results — ${file}`, `> ${summary}`, ""];
+  lines.push(row(headings));
+  lines.push(`|${headings.map((h) => "-".repeat(h.length + 2)).join("|")}|`);
+  for (const { combination: c, cells } of rows) {
+    const drawn = cells
+      ? cells.map((cell) => (cell ? `${RATING_ICON[cell.rating] ?? "·"} ${formatValue(cell.median, cell.unit)}` : "·"))
+      : ["❌ every run failed", ...metrics.slice(1).map(() => "")];
+    lines.push(row([c.name, c.device, ...(hasVariants ? [c.variant] : []), ...drawn]));
+  }
+
+  if (errors.length) {
+    lines.push("", "### Errors", ...errors.map((e) => `- ${e}`));
+  }
+
+  for (const c of report.combinations.filter((c) => !c.error)) {
+    lines.push("", `<details><summary>${c.id}</summary>`, "", reportMarkdown({ ...c, title: null }), "", "</details>");
+  }
+  return lines.join("\n");
+}

package/src/reporters/matrix-view.js

@@ -0,0 +1,36 @@
+import { basename } from "node:path";
+import { rate } from "../ratings.js";
+import { blockingErrors } from "../blocking.js";
+
+// The order of the metric columns of the matrix table
+const METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP"];
+
+// What the human and markdown reports of a matrix show, so they only differ in how they draw it:
+// the title, the metric columns, one row per combination with the rated median of each metric, and
+// the errors that decide the exit code.
+export function matrixView({ matrix, runs, combinations }) {
+  const measured = combinations.filter((c) => !c.error);
+  const medianBys = [...new Set(measured.map((c) => c.runs.medianBy))];
+  const metrics = METRICS.filter((metric) => measured.some((c) => c.runs.summary[metric]));
+  const median = medianBys.length ? `, median by ${medianBys.join(", ")}` : "";
+
+  return {
+    file: basename(matrix),
+    // A matrix without variants has no Variant column
+    hasVariants: combinations.some((c) => c.variant),
+    summary: `${combinations.length} combinations, ${runs} run${runs === 1 ? "" : "s"} each${median}`,
+    metrics,
+    rows: combinations.map((c) => ({
+      combination: c,
+      cells: c.error
+        ? null
+        : metrics.map((metric) => {
+            const value = c.runs.summary[metric];
+            return value ? { median: value.median, unit: value.unit, rating: rate(metric, value.median) } : null;
+          }),
+    })),
+    errors: combinations.flatMap((c) =>
+      c.error ? [`${c.id}: ${c.error}`] : blockingErrors(c.results).map((e) => `${c.id}: ${e.id}: ${e.message}`)
+    ),
+  };
+}

package/src/runs.js

@@ -0,0 +1,65 @@
+// Several runs of the same measurement, reduced to one: the median run, as WebPageTest reports it.
+// Every result of the report comes from that run, so they agree with each other (the sub-parts of
+// the LCP add up to its LCP), and a summary gives the median, min and max of each metric.
+
+// The metrics the snippets report, and navMs, which every run has
+export const MEDIAN_METRICS = ["LCP", "FCP", "TTFB", "CLS", "INP", "navMs"];
+
+// Without --median-by, the median follows the first of these that the runs measured
+const DEFAULT_ORDER = ["LCP", "FCP", "TTFB", "navMs"];
+
+const SUMMARY_METRICS = MEDIAN_METRICS.filter((metric) => metric !== "navMs");
+
+// The value of a metric in a run: its first result with that metric, as LCP comes before its
+// follow-up LCP-Subparts, which reports the same metric
+function valueOf(run, metric) {
+  if (metric === "navMs") return run.navMs;
+  const result = run.results.find((r) => r.metric === metric && r.status === "ok" && typeof r.value === "number");
+  return result?.value;
+}
+
+const measures = (runs, metric) => runs.some((run) => typeof valueOf(run, metric) === "number");
+
+// The middle element of numbers sorted from low to high. With an even count it takes the lower of
+// the two middle ones, so the median is always a run that happened, never an average of two.
+const middle = (sorted) => sorted[Math.floor((sorted.length - 1) / 2)];
+
+// "3 runs, median by LCP: run 2 (1 failed)", for the header of the reports
+export function describeRuns({ count, failed, medianRun, medianBy, medianByRequested }) {
+  const by = medianByRequested ? `${medianBy} (no run measured ${medianByRequested})` : medianBy;
+  return `${count} run${count === 1 ? "" : "s"}, median by ${by}: run ${medianRun}${failed ? ` (${failed} failed)` : ""}`;
+}
+
+// The runs as the CLI made them, in order, a run that failed being { error }. Returns the payload of
+// the median run and the `runs` field of the report, or nulls and the errors when every run failed.
+export function summarizeRuns(runs, { medianBy: requested } = {}) {
+  const completed = runs.filter((run) => !run.error);
+  if (completed.length === 0) return { payload: null, runs: null, errors: runs.map((run) => run.error) };
+
+  const medianBy = requested && measures(completed, requested) ? requested : DEFAULT_ORDER.find((metric) => measures(completed, metric));
+  const ranked = completed
+    .filter((run) => typeof valueOf(run, medianBy) === "number")
+    .sort((a, b) => valueOf(a, medianBy) - valueOf(b, medianBy));
+  const payload = middle(ranked);
+
+  const summary = {};
+  for (const metric of SUMMARY_METRICS) {
+    if (!measures(completed, metric)) continue;
+    const values = runs.map((run) => (run.error ? null : (valueOf(run, metric) ?? null)));
+    const sorted = values.filter((value) => value !== null).sort((a, b) => a - b);
+    const unit = completed.flatMap((run) => run.results).find((r) => r.metric === metric && r.unit)?.unit;
+    summary[metric] = { median: middle(sorted), min: sorted[0], max: sorted[sorted.length - 1], values, unit };
+  }
+
+  return {
+    payload,
+    runs: {
+      count: runs.length,
+      failed: runs.length - completed.length,
+      medianRun: runs.indexOf(payload) + 1,
+      medianBy,
+      ...(requested && requested !== medianBy ? { medianByRequested: requested } : {}),
+      summary,
+    },
+  };
+}

package/src/throttling.js

@@ -1,6 +1,9 @@
 // CPU and network throttling through CDP, as the Performance panel of DevTools applies it. The
 // network presets copy the values of DevTools, so a measurement of the CLI compares with one made
 // there. Throughputs are in bytes per second and latencies in milliseconds.
+//
+// A network can also be given by its own values, for the ones DevTools has no preset for, such as
+// 5G or fibre: { downloadKbps, uploadKbps, latencyMs }, in kilobits per second and milliseconds.
 export const NETWORK_PRESETS = {
   "fast-4g": { title: "Fast 4G", download: (9 * 1000 * 1000) / 8 * 0.9, upload: (1.5 * 1000 * 1000) / 8 * 0.9, latency: 60 * 2.75 },
   "slow-4g": { title: "Slow 4G", download: (1.6 * 1000 * 1000) / 8 * 0.9, upload: (750 * 1000) / 8 * 0.9, latency: 150 * 3.75 },
@@ -13,25 +16,59 @@ export function parseCpuFactor(raw) {
   return raw.trim() !== "" && Number.isFinite(factor) && factor >= 1 ? factor : null;
 }
 
+const NETWORK_VALUES = ["downloadKbps", "uploadKbps", "latencyMs"];
+
+// The throughputs and latency of a preset name or of a network given by its values
+export function resolveNetwork(network) {
+  if (typeof network === "string") {
+    const preset = NETWORK_PRESETS[network];
+    if (!preset) throw new Error(`Unknown network preset: "${network}". Choose from: ${Object.keys(NETWORK_PRESETS).join(", ")}`);
+    return preset;
+  }
+  for (const key of NETWORK_VALUES) {
+    const value = network?.[key];
+    if (typeof value !== "number" || !Number.isFinite(value) || value < 0) {
+      throw new Error(`Invalid network: ${key} must be a number of at least 0, such as { "downloadKbps": 50000, "uploadKbps": 10000, "latencyMs": 20 }`);
+    }
+  }
+  return {
+    download: (network.downloadKbps * 1000) / 8,
+    upload: (network.uploadKbps * 1000) / 8,
+    latency: network.latencyMs,
+  };
+}
+
+// 400 kbps, 1.44 Mbps, 50 Mbps
+function formatKbps(kbps) {
+  return kbps < 1000 ? `${kbps} kbps` : `${Number((kbps / 1000).toFixed(2))} Mbps`;
+}
+
+function describeNetwork(network) {
+  if (typeof network === "string") return NETWORK_PRESETS[network]?.title ?? network;
+  return `${formatKbps(network.downloadKbps)}, ${network.latencyMs} ms`;
+}
+
 // The conditions of a run, as the payload reports them, or undefined without throttling
 export function throttlingOf({ cpu, network }) {
   if (cpu == null && network == null) return undefined;
   return { ...(cpu != null ? { cpu } : {}), ...(network != null ? { network } : {}) };
 }
 
-// "CPU 4x, Slow 4G", for the header of the reports
+// "CPU 4x, Slow 4G" or "CPU 1x, 50 Mbps, 20 ms", for the header of the reports
 export function describeThrottling({ cpu, network }) {
-  return [cpu != null ? `CPU ${cpu}x` : null, network != null ? (NETWORK_PRESETS[network]?.title ?? network) : null]
+  return [cpu != null ? `CPU ${cpu}x` : null, network != null ? describeNetwork(network) : null]
     .filter(Boolean)
     .join(", ");
 }
 
 export async function applyThrottling(context, page, { cpu, network }) {
   if (cpu == null && network == null) return;
+  // An invalid network fails before anything is applied to the page
+  const conditions = network != null ? resolveNetwork(network) : null;
   const cdp = await context.newCDPSession(page);
   if (cpu != null) await cdp.send("Emulation.setCPUThrottlingRate", { rate: cpu });
-  if (network != null) {
-    const { download, upload, latency } = NETWORK_PRESETS[network];
+  if (conditions) {
+    const { download, upload, latency } = conditions;
     await cdp.send("Network.enable");
     await cdp.send("Network.emulateNetworkConditions", {
       offline: false,