rolldown-perf-kit 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +345 -0
- package/LICENSE +25 -0
- package/README.md +621 -2
- package/harness.mjs +4860 -0
- package/lib/build.mjs +57 -0
- package/lib/cdp.mjs +217 -0
- package/lib/chunk-discovery.mjs +226 -0
- package/lib/coverage.mjs +498 -0
- package/lib/field/analysis.mjs +225 -0
- package/lib/field/browser.mjs +232 -0
- package/lib/field/build-match.mjs +86 -0
- package/lib/field/capture.mjs +501 -0
- package/lib/field/config.mjs +84 -0
- package/lib/field/run.mjs +612 -0
- package/lib/gen-app.mjs +399 -0
- package/lib/graph-diff.mjs +136 -0
- package/lib/interrupt.mjs +25 -0
- package/lib/measure.mjs +540 -0
- package/lib/min-cut.mjs +218 -0
- package/lib/module-graph.mjs +318 -0
- package/lib/network-evidence.mjs +84 -0
- package/lib/node/launch.mjs +351 -0
- package/lib/node/measure.mjs +289 -0
- package/lib/paint-diagnostics.mjs +76 -0
- package/lib/parity/anchor.mjs +186 -0
- package/lib/parity/capture.mjs +527 -0
- package/lib/parity/compare.mjs +813 -0
- package/lib/parity/png.mjs +166 -0
- package/lib/parity/run.mjs +145 -0
- package/lib/profile.mjs +281 -0
- package/lib/replay/cert.mjs +45 -0
- package/lib/replay/model.mjs +656 -0
- package/lib/replay/run.mjs +1140 -0
- package/lib/replay/server.mjs +499 -0
- package/lib/replay/shaper.mjs +158 -0
- package/lib/rolldown-support.mjs +161 -0
- package/lib/serve.mjs +142 -0
- package/lib/worker/attrib.mjs +125 -0
- package/lib/worker/config.mjs +116 -0
- package/lib/worker/launch.mjs +337 -0
- package/lib/worker/measure.mjs +296 -0
- package/package.json +38 -4
- package/replay.mjs +79 -0
package/README.md
CHANGED
|
@@ -1,3 +1,622 @@
|
|
|
1
|
-
|
|
1
|
+
> Experimental publish of `packages/perf-kit` from the rolldown repository (branch `feat/metrics-lab-field-replay`, commit `6e578fb57` (with uncommitted changes), 2026-10-29). The package is `@rolldown/perf-kit` in the repository; the name here is for experiments only. Installed, the harness is `npx rolldown-perf-kit` (or `npx perf-kit`) and its state goes to `.perf-kit/` in the current directory.
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
# perf-kit — performance harness for agent loops
|
|
4
|
+
|
|
5
|
+
## Install / run
|
|
6
|
+
|
|
7
|
+
Zero runtime dependencies (raw CDP over Node ≥22's built-in WebSocket + a system
|
|
8
|
+
Chrome/Edge), so it works three ways:
|
|
9
|
+
|
|
10
|
+
- **In this repo**: `node packages/perf-kit/harness.mjs …` from anywhere.
|
|
11
|
+
- **As a dependency**: `npm i -D <tarball or @rolldown/perf-kit>` → `npx perf-kit scan --app .`
|
|
12
|
+
(the `bin` shim replaces any `node node_modules/…` path-typing). When installed,
|
|
13
|
+
state lives in the consumer project at `.perf-kit/` (add it to your
|
|
14
|
+
`.gitignore`), never inside `node_modules`; override with `ROLLDOWN_PERF_KIT_STATE`.
|
|
15
|
+
The hints print the spelling that applies. No rolldown checkout sits next to an
|
|
16
|
+
installed copy, so `rolldown-install` needs `--from <a built packages/rolldown>`.
|
|
17
|
+
- **Once published**: `npx @rolldown/perf-kit scan --app .` with no install step.
|
|
18
|
+
|
|
19
|
+
Prototype of the metrics plan's **Phase 3b (lab runner)** and **3c (code coverage)**:
|
|
20
|
+
the measurement and mutation primitives an agent needs to run a
|
|
21
|
+
"measure → find unused-at-paint code → lazy-load it → re-measure → accept or revert"
|
|
22
|
+
optimization loop against a real headless browser. The harness deliberately does
|
|
23
|
+
**not** run the loop itself — deciding what to try next and whether to keep it is the
|
|
24
|
+
agent's job; every command here is one loop step with machine-readable output.
|
|
25
|
+
|
|
26
|
+
No dependencies: raw CDP over Node's built-in `WebSocket` (Node >= 22) against a
|
|
27
|
+
system Chrome/Edge, and the repo's own rolldown (`packages/rolldown/dist`, so run
|
|
28
|
+
`just build-rolldown` first). Builds run with `devtools: { mode: 'report' }`, so
|
|
29
|
+
every build also refreshes the build-side report under
|
|
30
|
+
`app/node_modules/.rolldown/report/dist/`.
|
|
31
|
+
|
|
32
|
+
## Commands
|
|
33
|
+
|
|
34
|
+
### Chunk discovery
|
|
35
|
+
|
|
36
|
+
`scan` keeps the full resource detail list in `runtime-metrics.json`. This includes
|
|
37
|
+
small preload facades. Each request records its initiator type and source location.
|
|
38
|
+
`requestPriority` records Chrome's initial priority, its last observed priority,
|
|
39
|
+
and changes with milliseconds since the request began. Use this history when
|
|
40
|
+
comparing preload priorities. The last priority is the value at collection, not
|
|
41
|
+
necessarily at paint. Repeated URLs and redirect hops have ambiguous matches;
|
|
42
|
+
the report omits their initiator and priority evidence. Missing evidence is not
|
|
43
|
+
proof of low priority. These fields use the
|
|
44
|
+
[CDP Network protocol](https://github.com/ChromeDevTools/devtools-protocol/blob/master/json/browser_protocol.json).
|
|
45
|
+
The report does not collect request headers or bodies. A browser timing buffer
|
|
46
|
+
overflow keeps the discovery check UNKNOWN.
|
|
47
|
+
|
|
48
|
+
The check reads `chunk-graph.json` from rolldown metrics. It can also use a Vite
|
|
49
|
+
manifest at `dist/.vite/manifest.json`. It follows static imports only. The report
|
|
50
|
+
stores separate results in `chunkDiscovery.fcp` and `chunkDiscovery.lcp`.
|
|
51
|
+
Each result uses that paint's time as the download deadline. A candidate starts
|
|
52
|
+
more than one configured RTT after its ancestor finishes. A download that finishes
|
|
53
|
+
after FCP can appear only in the LCP result. Missing LCP does not discard FCP evidence.
|
|
54
|
+
|
|
55
|
+
Each group includes filenames, bytes, and the number of affected runs. Its example
|
|
56
|
+
keeps one run's timings, import chains, and available source modules and initiators.
|
|
57
|
+
Groups and paint windows overlap. Do not add their bytes or delays.
|
|
58
|
+
|
|
59
|
+
The report marks each candidate's `paintDependency` as `unverified`. Check the
|
|
60
|
+
source to find whether the chunk renders the first visible content or the LCP
|
|
61
|
+
element. If paint needs it, test an earlier preload or a different chunk boundary.
|
|
62
|
+
If paint does not need it, test deferring its load and work. A discovery delay is
|
|
63
|
+
not a predicted paint gain. Earlier downloads can compete with other required assets.
|
|
64
|
+
|
|
65
|
+
The render-gap advice shows concurrent script, fetch/XHR, font, and image evidence.
|
|
66
|
+
Check which requests rendering awaits before changing load order. Keep the LCP
|
|
67
|
+
image request early. Compare these candidates with the CPU profile.
|
|
68
|
+
|
|
69
|
+
`paintComparison` checks both FCP and LCP against the pinned baseline. A change
|
|
70
|
+
must improve at least one metric beyond noise. Neither may regress beyond noise.
|
|
71
|
+
The threshold for each metric is max(30ms, 2% of its baseline). Missing metrics
|
|
72
|
+
leave the comparison unknown. Quick and early-stopped scans are probes. Use
|
|
73
|
+
`scan --no-early-stop` for a keep/revert decision. Check the app's features before
|
|
74
|
+
keeping a change. Check rendered output with `scan --parity` when the work is complete.
|
|
75
|
+
|
|
76
|
+
The render-gap report also counts scripts that start after `load` and finish
|
|
77
|
+
before FCP or LCP. The load event does not prove that downloads ended. Commands
|
|
78
|
+
that resolve a page target print its path, including remembered targets.
|
|
79
|
+
|
|
80
|
+
Static module analysis starts at build entries. It does not price a dynamic import
|
|
81
|
+
that startup immediately executes. Check coverage and CPU evidence before treating
|
|
82
|
+
small static entry totals as small startup totals. Manually grouped chunks can
|
|
83
|
+
also load modules outside an observed facade's static closure.
|
|
84
|
+
|
|
85
|
+
### CPU source attribution
|
|
86
|
+
|
|
87
|
+
`profile` maps sampled work through every same-origin chunk's sourcemap. This
|
|
88
|
+
includes static siblings and dynamic imports. `profile.json` records each source,
|
|
89
|
+
its chunk filenames, and sampled self-time. It separates source work, engine work,
|
|
90
|
+
and unattributed work. Missing or invalid maps, unmapped positions, inline scripts,
|
|
91
|
+
and external scripts remain visible. They do not count as engine work. A frame
|
|
92
|
+
without a URL (a builtin such as `JSON.parse` or a DOM call) counts for its nearest
|
|
93
|
+
caller with a URL, or as engine work when it has none. Top-level code of a bundled
|
|
94
|
+
chunk runs many modules in one frame: its time splits by the sampled lines, and a
|
|
95
|
+
line that holds more than one source stays `top-level` (unattributed). Minified
|
|
96
|
+
chunks keep most top-level time unattributed; profile an unminified build to split it.
|
|
97
|
+
|
|
98
|
+
The verdict keeps source modules with at least 15ms of sampled work OPEN for
|
|
99
|
+
inspection. A filename without a directory can also identify a source module.
|
|
100
|
+
Unattributed rows of at least 15ms keep the check UNKNOWN; a smaller row cannot
|
|
101
|
+
hide a source module that reaches the threshold. Old profiles that map only the entry chunk
|
|
102
|
+
must be collected again. A clear CPU signal means no source reaches the threshold;
|
|
103
|
+
it does not prove that all startup work is required.
|
|
104
|
+
|
|
105
|
+
The profile stops when the harness observes FCP. The polling tail can include
|
|
106
|
+
post-paint work. Sampled time is a lead, not a predicted paint gain. Check whether
|
|
107
|
+
the first render needs each source module. Test deferring optional work, then
|
|
108
|
+
compare both FCP and LCP in timed runs. The profile does not attribute CPU between
|
|
109
|
+
FCP and LCP when those paints differ.
|
|
110
|
+
|
|
111
|
+
### Command list
|
|
112
|
+
|
|
113
|
+
| Command | One loop step |
|
|
114
|
+
| -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
115
|
+
| `node harness.mjs scan --app <appDir>` | **The whole iteration in one browser session**: N timed runs + coverage + boot profile + the fused verdict. First scan of a target auto-pins the baseline; `--pin` re-pins after a kept change; `--quick` = 1 run + no profile, a fast mid-iteration probe on slow apps (indicative only — never pinned, and the verdict flags single-run measurements). The target is remembered — afterwards run everything bare, from any cwd. Per-target state dirs keep baselines/history from mixing across apps. |
|
|
116
|
+
| `node harness.mjs scan --bandwidth <Mbit/s> [--rtt <ms>] [--cpu <rate>]` | Pin a network in real units instead of a net-scale step: the network a deployed environment measured (a record's model), e.g. `--bandwidth 26 --rtt 120 --cpu 1` (1 = this machine's CPU). No calibration; RTT and CPU default to 150 ms and 4x. Pinned and compared like `--net-scale`: a change makes the pinned baseline incomparable (re-pin). Same flags on `measure` and `target`. |
|
|
117
|
+
| `node harness.mjs target [<appDir>] [--demo]` | Show / set / clear the remembered target. |
|
|
118
|
+
| `node harness.mjs gen [--force]` | Generate the demo app (deterministic; `--force` resets defers). |
|
|
119
|
+
| `node harness.mjs build` | Build `app/` → `app/dist/` + build metrics report. |
|
|
120
|
+
| `node harness.mjs measure [--runs 5] [--label X] [--no-throttle]` | N throttled runs (1 warmup discarded) → `state/runtime-metrics.json` with medians, guard, `delta`/`baselineDelta` — plus a **render-blocking CSS gate** flag (the last blocking stylesheet finished ≈ FCP: CSS, not JS, is the paint gate — via resource-timing `renderBlockingStatus`), a **render gap** flag (paint gated on post-load work; the gate is named: gating fetches, or per-type pre-paint resource weight — fonts/images) and a **pre-paint CPU** flag (long tasks before first paint). |
|
|
121
|
+
| `node harness.mjs coverage` | One instrumented run → per-module bytes executed **before first paint** vs **by settle** → `state/coverage.json` + four lead sections: defer candidates, **cold bytes at paint** (the unified ranking `totalBytes − paintBytes`, coldest first — catches the mid-band a partially-initializing vendor SDK sits in, which neither the <2% nor the ≥50% bucket sees; framework runtimes annotated), large-modules-executed-at-paint (data evaluates on import — executed ≠ needed), and sibling variant groups (locales/themes). Covers the **whole initial load**: the entry chunk plus every same-origin chunk that executed before first paint (modules tagged with their chunk; chunks without sourcemaps are called out). Post-paint-executed chunks are split by fetch timing: **static pre-paint transfer** (fetched before paint via static tags/preloads — the paint paid for the download; verdict lead when ≥100KB) vs genuinely-deferred (fetched after paint). Entry auto-detected from `dist/index.html` — module scripts, or webpack-style plain script tags (main-looking/biggest bundle, query strings stripped); override with `--entry`. |
|
|
122
|
+
| `node harness.mjs profile` | One profiled run → boot CPU by source module, navigation → first paint (`state/profile.json`). The follow-up to a pre-paint CPU flag. |
|
|
123
|
+
| `node harness.mjs graph` | **Static split candidates ranked by retained size** — per module, the bytes its dominator subtree would remove from the initial load if its import edge were deferred, with `via` naming the single import chain to cut. Reads `module-graph.json` from a rolldown devtools-metrics build (vite ≥ 8: `build.rolldownOptions.devtools = { mode: "report" }`; the demo app emits it natively). |
|
|
124
|
+
| `node harness.mjs what-if <module> [--keep a,b] [--root a,b]` | The exact modules + bytes one deferral frees (unique-reachable closure; equals the module's retained size). `--keep` marks sentry modules that stay eager. `--root` (also on `cut`) adds modules that load on every start without a static import from an entry - a dynamic import the app always runs: they join the entries, so the query prices that startup load. Instant candidate ranking; verify the winner's LCP effect with `scan`. |
|
|
125
|
+
| `node harness.mjs verdict` | Fuses all signals into an OPEN/clear/UNKNOWN lead checklist with staleness tracking. Refuses to say "done" while leads are open or signals are missing/stale; the all-clear states the tools' blind-spot boundary. While leads are OPEN it also instructs the operator/agent to copy the checklist into their summary and justify any early stop lead-by-lead — a re-pinned baseline records a gain, it does not close the checklist. |
|
|
126
|
+
| `node harness.mjs parity [--only /a,/b] [--label x] [--exit-code]` | **Rendered output vs the original build**, both rendered back to back in one browser session: first-screen + full-page pixels, page height, visible text, accessible names, new runtime errors, and the LCP element. Compared once, when the optimizing is done (`scan --parity` = final measurement + this). A difference starts recover mode. `--only` re-checks some pages, `--exit-code` exits 1 while a page differs (`git bisect run`), `--pages` / `--mask` add routes / masks (also on `scan`). |
|
|
127
|
+
| `node harness.mjs parity recover` / `strategy each-tweak\|on-demand` / `history` | Recover mode: what to do and how to bisect; re-check on every scan or when you decide; every check with its snapshot images. |
|
|
128
|
+
| `node harness.mjs parity anchor [--replace]` | Show the saved original (commit, pages, masks, its LCP element). `--replace` makes the current build the original — a human decision, recorded in every later verdict. |
|
|
129
|
+
| `node harness.mjs parity approve <page> --reason "…"` | A human accepts one exact, intended difference (keyed on the changed pixels and every judged difference — any further change reopens the page). Listed in every later verdict. |
|
|
130
|
+
| `node harness.mjs baseline` | Pin the last measurement (and the build-side `.state.json`) as the fixed reference for every following `baselineDelta`. Refused for a build known to render differently from the original (see [Rendered-output parity](#rendered-output-parity)). |
|
|
131
|
+
| `node harness.mjs defer <feature>` / `undefer <feature>` | Rewrite that feature's marker block in `app/src/main.ts` between static import and post-paint `import()`. Rebuild afterwards. |
|
|
132
|
+
| `node harness.mjs status` | Feature modes, entry size, last/baseline LCP. |
|
|
133
|
+
| `node harness.mjs serve [--port 4646]` | Serve `app/dist` for manual poking. |
|
|
134
|
+
|
|
135
|
+
`measure` and `coverage` also take `--dist <dir>` (plus optional `--entry`,
|
|
136
|
+
`--features a,b`) to point at any other built app; candidates are then advisory
|
|
137
|
+
per-module (the agent finds the import seams itself).
|
|
138
|
+
|
|
139
|
+
`graph`, `what-if`, `cut`, `why` and `graph-diff` read the report of the target dist
|
|
140
|
+
(`node_modules/.rolldown/report/<output dir>`), and stop with an error when none of
|
|
141
|
+
the report's entry chunks exists in that dist: a rebuild without the report leaves
|
|
142
|
+
the old report behind, and builds that share an outDir share a report. Rebuild with
|
|
143
|
+
the report on, or pass `--report <dir>`, which is never checked.
|
|
144
|
+
|
|
145
|
+
## Rendered-output parity
|
|
146
|
+
|
|
147
|
+
An LCP number only means something if the page still renders what it rendered
|
|
148
|
+
before. Agent runs showed the failure this section exists for: a change that
|
|
149
|
+
broke a style, deferred a font that then never loaded, or let a placeholder become
|
|
150
|
+
the LCP element — the page still loaded, the functional check still passed, and the
|
|
151
|
+
"gain" was measured on a broken page. Sometimes the break IS the gain (a hidden hero
|
|
152
|
+
makes LCP measure a smaller, earlier element).
|
|
153
|
+
|
|
154
|
+
So the finished work is compared with the **original** — **once, at the end**, not
|
|
155
|
+
per change: an agent that broke a page can usually find out why from a good
|
|
156
|
+
description of the difference, and most changes don't break anything, so a check on
|
|
157
|
+
every iteration would mostly buy time back for nothing.
|
|
158
|
+
|
|
159
|
+
- **While optimizing** nothing is captured. Each measurement only says whether the
|
|
160
|
+
LCP element is still the original's — free, the timed runs already observe it —
|
|
161
|
+
as information. Pinning does not wait on a check.
|
|
162
|
+
- **When done**, `scan --parity` takes the final measurement and compares. The
|
|
163
|
+
verdict holds the comparison as `[PENDING]` until then, so "done" needs it.
|
|
164
|
+
- **A difference starts recover mode.** The build's LCP numbers don't count until it
|
|
165
|
+
matches again (the keep line says "NOT a keeper", pinning refuses a build known to
|
|
166
|
+
differ). The agent fixes or reverts and picks how to re-check: `parity strategy
|
|
167
|
+
each-tweak` (every scan re-checks and records a snapshot) or `on-demand` (it runs
|
|
168
|
+
`parity` itself, `--only /page` for the broken pages). `parity --exit-code` exits 1
|
|
169
|
+
while a page differs, so `git bisect run` can find the kept change that broke it,
|
|
170
|
+
and `parity history` lists every check with its snapshot images. A check that
|
|
171
|
+
covers every page without a difference ends recover mode.
|
|
172
|
+
|
|
173
|
+
**The original is a copy of the build, frozen.** The first scan / measure / parity of
|
|
174
|
+
a target copies the build directory into the lab state (`parity/anchor/`), with the
|
|
175
|
+
app's commit and a content hash, and the first full scan records its LCP element. A
|
|
176
|
+
check serves it next to the current build from one origin and renders both in the
|
|
177
|
+
same browser session, so a Chrome update, a newly installed font or machine load
|
|
178
|
+
never shows up as a difference. Unlike the perf baseline, the loop never re-pins it:
|
|
179
|
+
the finished work is compared with what the app rendered before any change.
|
|
180
|
+
`parity anchor --replace` exists for a human who decides the reference must move,
|
|
181
|
+
and every later verdict says it was replaced.
|
|
182
|
+
|
|
183
|
+
**Pages are pinned.** The original's entry HTML files are found automatically (files
|
|
184
|
+
that load bundled scripts or stylesheets — a self-contained `stats.html` is skipped);
|
|
185
|
+
client-side routes are added with `--pages`. Pages can be added at any time (the
|
|
186
|
+
original build is kept, so a new page is rendered from it on demand), never removed.
|
|
187
|
+
|
|
188
|
+
**Captures are made repeatable before anything is compared**, with the sequence
|
|
189
|
+
hosted visual-regression tools converged on (Chromatic and Percy):
|
|
190
|
+
1280×900 at scale 1, no throttle, reduced motion, a CSS switch that zeroes
|
|
191
|
+
animations and transitions, `document.fonts.ready`, then animation-frame, DOM and
|
|
192
|
+
network quiet windows under one shared deadline (a page with an endless animation
|
|
193
|
+
loop stops being waited on after 2s), Web Animations and SMIL frozen right before
|
|
194
|
+
each screenshot, a pinned `Date` and a seeded `Math.random`, cleared storage per
|
|
195
|
+
capture, and a scroll through the page so sections that render on scroll exist in
|
|
196
|
+
both builds before the full-page shot.
|
|
197
|
+
|
|
198
|
+
**Noise is measured, not guessed.** Each page renders three times: original,
|
|
199
|
+
current, original again — the second original under a clock moved by 1d1h1m1s and
|
|
200
|
+
another random seed. Whatever differs between the two originals (a clock, random
|
|
201
|
+
content, a race) becomes that page's unstable mask; a difference there is reported
|
|
202
|
+
as "not judged", never as a break. Because both builds render on the same machine in
|
|
203
|
+
the same session, the remaining tolerance can be tiny (a channel move of more than
|
|
204
|
+
10/255 over at least 20 pixels) — a hosted service's 0.5%-of-the-screen default
|
|
205
|
+
would let a missing 24px icon through. A page whose original moves over more than
|
|
206
|
+
30% of the view is UNKNOWN: "nothing changed" would only cover the part that holds
|
|
207
|
+
still. A difference found once is confirmed by a second capture of the current
|
|
208
|
+
build before it is reported.
|
|
209
|
+
|
|
210
|
+
**What decides, and what explains.** Judged — any one opens the page:
|
|
211
|
+
first-screen and full-page pixels, page height, visible text (`innerText`, order-
|
|
212
|
+
insensitive), accessible names of controls/landmarks/headings/images, where the page
|
|
213
|
+
ended up, runtime errors the original never produced (uncaught exceptions,
|
|
214
|
+
`console.error`, failed requests, HTTP ≥ 400), and — for `/` — the LCP element from
|
|
215
|
+
the timed runs, compared with the original's: a different element, the same element
|
|
216
|
+
much smaller, or a placeholder (LCP fired on content the element later replaced: a
|
|
217
|
+
blurred image swapped for the real one, skeleton text swapped for copy, a skeleton
|
|
218
|
+
node removed). A same-size swap emits no new LCP entry, so the timestamp keeps
|
|
219
|
+
pointing at the placeholder; only the settled element shows it. Explanatory only —
|
|
220
|
+
they say why, never decide: computed styles per element, the font Chrome actually
|
|
221
|
+
drew each first-screen text element with (fallbacks included), images, loaded font
|
|
222
|
+
faces, and which elements sit under each changed region. A legitimate deferral
|
|
223
|
+
changes none of the judged facts but may change explanatory ones (a wrapper element,
|
|
224
|
+
a renamed image file).
|
|
225
|
+
|
|
226
|
+
**Masks and approvals are for humans, and visible.** `--mask <selector>` paints a
|
|
227
|
+
region opaque in every capture; one added after the original was saved is listed in
|
|
228
|
+
every verdict. `parity approve <page> --reason "…"` accepts one exact difference —
|
|
229
|
+
keyed on the changed pixels and every judged difference, so any further change
|
|
230
|
+
reopens the page — and is listed in every verdict with its reason.
|
|
231
|
+
|
|
232
|
+
Evidence per page lands in `parity/pages/<page>/`: `anchor.png`, `current.png`,
|
|
233
|
+
`diff.png` (changes red, unstable areas blue) and the `-full` variants, copied per
|
|
234
|
+
check into `parity/snapshots/<n>/`. A check costs three unthrottled loads per page
|
|
235
|
+
(four when a difference is confirmed); on the rolldown docs build that is ~2.5s per
|
|
236
|
+
load. A build with the original's exact files costs nothing: same files, same
|
|
237
|
+
rendering.
|
|
238
|
+
|
|
239
|
+
## Server startup mode (node / deno)
|
|
240
|
+
|
|
241
|
+
Same question — "what does the initial load pay for, and what can leave it?" — asked
|
|
242
|
+
of a process coming up instead of a page loading.
|
|
243
|
+
|
|
244
|
+
```bash
|
|
245
|
+
node harness.mjs node-scan --entry dist/server.js --ready port:3000
|
|
246
|
+
node harness.mjs node-scan # target + readiness are remembered
|
|
247
|
+
node harness.mjs node-scan --pin # make this the fixed reference
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
| Command | What it does |
|
|
251
|
+
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
252
|
+
| `node-scan` | Timed spawn→ready runs, then two instrumented runs: **what had to execute** to become ready (V8 precise coverage snapshotted at the readiness moment, attributed to source modules through the sourcemap) and **where the pre-ready CPU went** (sampling profile armed while the process is still paused at entry). Then the leads. |
|
|
253
|
+
| `node-measure` | Timing only — no instrumented runs. |
|
|
254
|
+
| `graph` / `what-if` / `cut` / `graph-diff` | Work unchanged on a server build (`--dist <builtDir>` or `--report <dir>`): a top-level import and a render-blocking import are the same edge in the module graph. |
|
|
255
|
+
|
|
256
|
+
**Readiness is part of the target.** A server has no equivalent of first paint, so the
|
|
257
|
+
target declares how it signals "up", and the spec is pinned with it — two scans that
|
|
258
|
+
measured different moments are not comparable.
|
|
259
|
+
|
|
260
|
+
| `--ready` spec | Ready when |
|
|
261
|
+
| ------------------------------ | ------------------------------------------------- |
|
|
262
|
+
| `port:3000` | a TCP listener is accepting |
|
|
263
|
+
| `stdout:listening on \d+` | a line matching this regex is printed |
|
|
264
|
+
| `http://127.0.0.1:3000/health` | an HTTP probe returns 2xx |
|
|
265
|
+
| `exit` | the process exits 0 (CLIs, lambda-shaped entries) |
|
|
266
|
+
|
|
267
|
+
Other options: `--cache cold|warm|off`, `--args "..."`, `--exec <bin>` (deno, another
|
|
268
|
+
node), `--cwd <dir>`, `--runs`, `--timeout <seconds>`, `--label`, `--pin`.
|
|
269
|
+
|
|
270
|
+
Metrics are `runtime.startup_ms`, `runtime.boot_floor_ms` (what an empty script costs on
|
|
271
|
+
this machine) and `runtime.app_startup_ms` (the difference — the part a bundle change can
|
|
272
|
+
actually move), plus the same `delta` / `baselineDelta` shape the browser side writes.
|
|
273
|
+
|
|
274
|
+
### What this mode does NOT do
|
|
275
|
+
|
|
276
|
+
- **No throttle.** Server startup does no network I/O, so a throttle would invent a
|
|
277
|
+
dimension the measurement does not have. There is therefore no net-scale calibration
|
|
278
|
+
and no cross-scale comparability problem.
|
|
279
|
+
- **The reported timing never runs under the inspector.** `--inspect-brk` shifts startup
|
|
280
|
+
by tens of ms; the instrumented runs are separate and used for attribution only.
|
|
281
|
+
- **Bun is not supported.** It runs JavaScriptCore, whose inspector is the WebKit
|
|
282
|
+
protocol and whose profile/coverage formats are not V8's, so attribution cannot work
|
|
283
|
+
there. Deno is V8 + CDP and rides the same driver via `--exec`.
|
|
284
|
+
- **Other edge runtimes cannot be measured this way** — no local process. Cloudflare
|
|
285
|
+
Workers have one (workerd); see the next section. The static half (`graph`, `what-if`,
|
|
286
|
+
`cut`) still applies to any edge bundle; the measured half does not.
|
|
287
|
+
|
|
288
|
+
## Worker startup mode (Cloudflare Workers)
|
|
289
|
+
|
|
290
|
+
The same question asked of a V8 isolate coming up under workerd: what does the Worker's
|
|
291
|
+
global scope pay for, and what can leave it? Cloudflare's startup limit counts that
|
|
292
|
+
scope, and `wrangler check startup` profiles it; this mode adds the loop around it:
|
|
293
|
+
a floor-corrected number per run, what executed, and what each deferral would free.
|
|
294
|
+
|
|
295
|
+
```bash
|
|
296
|
+
node harness.mjs worker-config --app <appDir> # writes perf-kit.vite.config.ts, prints the build commands
|
|
297
|
+
(cd <appDir> && npx vite build --config perf-kit.vite.config.ts)
|
|
298
|
+
node harness.mjs worker-scan --app <appDir> --request / --request /api/health # routes the app answers (2xx)
|
|
299
|
+
node harness.mjs worker-scan # target, requests and --tools are remembered
|
|
300
|
+
node harness.mjs worker-scan --pin # make this the fixed reference
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
| Command | What it does |
|
|
304
|
+
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
305
|
+
| `rolldown-install --app <dir>` | Copies a built rolldown package (with its native binding) over the copy the app's vite resolves; the default source is the rolldown package next to the lab, `--from <packages/rolldown>` names another. The npm release has no report mode; `worker-config` and the config shim refuse to build on it, because such a build writes the full event log (hundreds of MB) instead of the report. `<package manager> install --force` restores the release. |
|
|
306
|
+
| `worker-config` | Writes a Vite config that wraps the app's own: the rolldown metrics report on **every** environment (one directory per output dir), sourcemaps on the worker, a worker-only rebuild (`ROLLDOWN_PERF_KIT_WORKER_ONLY=1`) and the attribution build (`ROLLDOWN_PERF_KIT_ATTRIB=1`, `preserveModules` into `dist-attrib/`). |
|
|
307
|
+
| `worker-scan` | N cold runs, a fresh workerd each: the main module's import timed from the host minus an empty-import floor = **app startup**. Then the first requests on the real entry, a CPU profile of the import attributed through every chunk's sourcemap, precise coverage of the import taken from inside the isolate, and the leads: cold and idle bytes per module and per package, runtime modules loaded, class exports. |
|
|
308
|
+
| `worker-measure` | Timing and first requests only. |
|
|
309
|
+
| `worker-attrib` | CPU by **evaluating module** from the attribution build, each row with its import chain: the only view that charges a library's top-level work (zod schemas built at import, a YAML parse) to the module whose evaluation runs it. |
|
|
310
|
+
| `why <module>` | Direct static importers and one chain from the entry. |
|
|
311
|
+
| `cut <module> --root a,b` | The fewest imports to make dynamic, with the Durable Object / Workflow class modules as roots: their closure stays eager, so the cut lands below them instead of at the entry. |
|
|
312
|
+
| `graph` / `what-if` / `graph-diff` | Unchanged on the Worker's report (`node_modules/.rolldown/report/dist/<worker>`). |
|
|
313
|
+
|
|
314
|
+
**Why the startup sits inside a request.** workerd evaluates a Worker's main module when
|
|
315
|
+
it loads the worker, before any request or inspector can observe it. The lab therefore
|
|
316
|
+
runs a generated wrapper entry whose fetch handler does `await import('./index.js')`:
|
|
317
|
+
that import compiles and evaluates the main module and its whole static closure. The
|
|
318
|
+
first requests and parity use a plain instance with the real entry instead, because
|
|
319
|
+
Durable Object classes live in the main module. `wrangler check startup` uses the same
|
|
320
|
+
wrapper trick.
|
|
321
|
+
|
|
322
|
+
**How coverage differs from the other modes.** Browser and node mode arm coverage from
|
|
323
|
+
outside, over the inspector, before any code runs. workerd answers `Profiler.enable`
|
|
324
|
+
itself and never forwards it to V8, so `Profiler.startPreciseCoverage` fails there. The
|
|
325
|
+
coverage run adds two compatibility flags for that run only, `nodejs_compat` and
|
|
326
|
+
`enable_nodejs_inspector_local_dev`, under which a wrapper opens a `node:inspector`
|
|
327
|
+
session inside the isolate, starts coverage, imports the main module and returns the
|
|
328
|
+
result in the response. It needs Wrangler 4.108.0 or later (workerd 1.20260706.1); for an
|
|
329
|
+
app that pins an older one, `--tools <dir>` borrows a newer wrangler for that run only.
|
|
330
|
+
|
|
331
|
+
**Requirements the scan checks for and names when unmet**
|
|
332
|
+
|
|
333
|
+
| Need | Requirement |
|
|
334
|
+
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
335
|
+
| the target | a `@cloudflare/vite-plugin` build: `dist/<worker>/wrangler.json` next to `index.js` |
|
|
336
|
+
| the runtime | `wrangler` as a dev dependency of the app (brings miniflare and workerd) |
|
|
337
|
+
| the report | the `worker-config` build, or `devtools: { mode: 'report' }` on every environment by hand, on a rolldown with the report mode: the npm release has none yet, so `rolldown-install --app <dir>` first |
|
|
338
|
+
| sourcemaps | `build.sourcemap: true` on the worker environment (the config shim sets it) |
|
|
339
|
+
| precise coverage | Wrangler >= 4.108.0, or `--tools <dir>` |
|
|
340
|
+
| first-request numbers | the app's variables (a `.dev.vars` next to the config); without them both real apps answered 500 |
|
|
341
|
+
| the number for a PR | `wrangler check startup` (Wrangler >= 4.116.0), before and after; its dry-run build runs Docker for `containers` |
|
|
342
|
+
|
|
343
|
+
Metrics are `worker.startup_ms`, `worker.boot_floor_ms`, `worker.app_startup_ms` (the part a
|
|
344
|
+
bundle change can move), `worker.instance_boot_ms` (reported, never counted) and
|
|
345
|
+
`worker.request_ms:<path>` per pinned request, with the same `delta` / `baselineDelta`
|
|
346
|
+
shape as the other modes. A request's delta against the pinned baseline is the cost a
|
|
347
|
+
deferral **moved** into that route; the summary prints it next to the startup gain.
|
|
348
|
+
|
|
349
|
+
### What this mode does NOT do
|
|
350
|
+
|
|
351
|
+
- **No throttle, no production number.** Local CPU differs from Cloudflare's; the ratio
|
|
352
|
+
before and after on one machine is the claim, and Cloudflare's own number comes from a
|
|
353
|
+
deploy or version upload.
|
|
354
|
+
- **No isolate boot.** Process start, config load and wasm compilation are outside the
|
|
355
|
+
window and outside Cloudflare's startup limit too.
|
|
356
|
+
- **No timing from inside the Worker.** Timers in workerd advance only on I/O.
|
|
357
|
+
- **No response parity yet.** The first requests report status and size; a replayed
|
|
358
|
+
request set against the frozen original is not built.
|
|
359
|
+
- **The attribution build works around a rolldown limit (rolldown#11348).** With `preserveModules` every
|
|
360
|
+
CommonJS file ends with an eager `export default require_x()`, and a require cycle reaches
|
|
361
|
+
a wrapper before its own file ran (`require_x is not a function`). The config shim drops
|
|
362
|
+
that line in the attribution build only; its importers use the named wrapper. If an import
|
|
363
|
+
still fails, `worker-attrib` says so and marks its rows partial.
|
|
364
|
+
|
|
365
|
+
## Field mode (a deployed site)
|
|
366
|
+
|
|
367
|
+
The lab answers "what does this build pay for" on a local copy with a fixed throttle.
|
|
368
|
+
Field mode asks the same question of the **deployed** site, in your own logged-in
|
|
369
|
+
Chrome: real data, real network, real server time. Use it when the lab and production
|
|
370
|
+
disagree, or to check that a gain survives a fast network.
|
|
371
|
+
|
|
372
|
+
```bash
|
|
373
|
+
# 1. Chrome: open a logged-in tab on the site, turn on chrome://inspect/#remote-debugging
|
|
374
|
+
# 2. Describe how the site serves each build:
|
|
375
|
+
node harness.mjs field --config field.json # 6 alternating pairs + 1 diagnostic run each
|
|
376
|
+
node harness.mjs field --config field.json --runs 10 --cut lcp --out /tmp/my-field
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
```json
|
|
380
|
+
{
|
|
381
|
+
"url": "https://app.example.com/",
|
|
382
|
+
"variants": {
|
|
383
|
+
"base": {
|
|
384
|
+
"cookies": [{ "name": "version", "value": "stable" }],
|
|
385
|
+
"build": "/builds/stable/dist"
|
|
386
|
+
},
|
|
387
|
+
"candidate": { "query": { "preview": "pr-123" } }
|
|
388
|
+
},
|
|
389
|
+
"block": ["analytics.example.com"],
|
|
390
|
+
"ignoreLcp": "#splash",
|
|
391
|
+
"gate": "/api/items"
|
|
392
|
+
}
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
| Part | What it does |
|
|
396
|
+
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
397
|
+
| Timing runs | Pairs of loads, one per variant, in alternating order, so a slow minute of the network hits both sides. A hidden page (covered window) or a missing LCP makes a run invalid; it is retried, never averaged in. |
|
|
398
|
+
| Critical path | The request LCP waited for (`gate`: a substring of a fetch/XHR URL, matched after redaction; unset: the last fetch/XHR that ended before LCP), split into: document end, client time before the gate request, gate TTFB, download, client time after it. Only the client parts can move. |
|
|
399
|
+
| Script chain | What the gate request waited for: of all scripts on its initiator stack, the one that finished loading last, then that script's own initiator, back to the document. The shape comes from the diagnostic run (async stacks name the caller behind a fetch wrapper); the times come from the median timing run. |
|
|
400
|
+
| Diagnostic run | V8 precise coverage at the cut (`--cut gate`: when the gate request is sent; `lcp`; `fcp`) and at settle, plus a CPU profile to the cut. Slower by design and never used for timing. A run that does not reach its cut in `--timeout` is invalid. |
|
|
401
|
+
| Build match | Each served script is matched to a local build file by SHA-256 (or `//# debugId=`). A match gives module attribution and the module graph; no match still gives chunk-level coverage from the served body. |
|
|
402
|
+
| Report | `<out>/<variant>/runtime-metrics.json`, `coverage.json`, `profile.json` in the lab's shapes (`"paint"` means "at the cut", see `field.cut`), plus `field-report.json`: medians, paired deltas with a seeded bootstrap 90% interval, slow-server outlier pairs. |
|
|
403
|
+
|
|
404
|
+
**Privacy.** Output goes outside the repo (default `$TMPDIR/perf-kit-field/<site>`).
|
|
405
|
+
It holds no cookies, no headers, no page text and no screenshots. IDs, emails and long
|
|
406
|
+
numbers in URLs are redacted (add patterns with `redact`). Variant cookies change the
|
|
407
|
+
whole profile, so each run starts from the original values and they are restored at the
|
|
408
|
+
end, also on Ctrl-C. Block the site's analytics beacons in `block` so runs do not add
|
|
409
|
+
synthetic sessions.
|
|
410
|
+
|
|
411
|
+
**Getting a matching build.** Production rarely serves its sourcemaps. Build the same
|
|
412
|
+
commit locally with the deploy's settings and bundler version; identical inputs give
|
|
413
|
+
byte-identical chunks. Anything the build fetches at build time must be identical too.
|
|
414
|
+
`buildMatch` in `coverage.json` says how much of the page the build explains;
|
|
415
|
+
`different-build` means the path exists locally with other content.
|
|
416
|
+
|
|
417
|
+
### What this mode does NOT do
|
|
418
|
+
|
|
419
|
+
- **It measures one account, one network, one machine.** Gates, data size and network
|
|
420
|
+
differ per user. Population behavior needs field RUM.
|
|
421
|
+
- **Your extensions run in the measured page.** Their scripts show up in `profile.json`
|
|
422
|
+
as `external` rows (the full URL for `chrome-extension://` frames). Use a profile with
|
|
423
|
+
few extensions when CPU numbers matter.
|
|
424
|
+
- **No parity check.** Real data changes between loads.
|
|
425
|
+
- **Only navigations.** It never clicks or submits. `nonGetRequests` in each
|
|
426
|
+
`runtime-metrics.json` lists what the page itself sent.
|
|
427
|
+
- **`--throttle lab` adds the lab throttle on top of the real network.** The real RTT
|
|
428
|
+
stays in, so the result is not comparable to a lab scan.
|
|
429
|
+
|
|
430
|
+
## Replay (a build that is not deployed)
|
|
431
|
+
|
|
432
|
+
Field mode needs every variant deployed. Replay gives field-like numbers for a LOCAL
|
|
433
|
+
build: record the deployed site once, then serve that recording to a throwaway headless
|
|
434
|
+
Chrome with your dist in place of the deployed files.
|
|
435
|
+
|
|
436
|
+
```bash
|
|
437
|
+
# 1. Record one deployed variant (same config file as field mode; Chrome as for field)
|
|
438
|
+
node harness.mjs record --config field.json --variant base # 6 timing runs + 1 body run
|
|
439
|
+
# 2. Calibrate: the replay of the RECORDED build must reproduce the field medians
|
|
440
|
+
node harness.mjs replay --archive $TMPDIR/perf-kit-replay/<site>-base \
|
|
441
|
+
--dist recorded=/builds/stable/dist --calibrate
|
|
442
|
+
# 3. A/B: alternating pairs of two local builds on the same recorded network and server
|
|
443
|
+
node harness.mjs replay --archive <dir> --dist base=/builds/stable/dist --dist candidate=./dist
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
### Standalone: one agent records, another replays
|
|
447
|
+
|
|
448
|
+
`replay.mjs` (bin `perf-kit-replay`) runs record and replay with no harness target and
|
|
449
|
+
no lab state. The archive folder is the whole interface: the recording agent hands over
|
|
450
|
+
its path, and the replaying agent needs no browser login.
|
|
451
|
+
|
|
452
|
+
```bash
|
|
453
|
+
# Recording agent (the user's Chrome, remote debugging on). No config file needed.
|
|
454
|
+
node replay.mjs record --url https://app.example.com/ --cookie version=canary \
|
|
455
|
+
--block analytics.example.com --gate /api/items --ignore-lcp '#splash' \
|
|
456
|
+
--build /builds/stable/dist --settle 6000 --json
|
|
457
|
+
# -> stdout: { "archive": "<dir>", "complete": true, "gaps": [], "next": {...}, ... }
|
|
458
|
+
|
|
459
|
+
# Replaying agent
|
|
460
|
+
node replay.mjs inspect --archive <dir> --json # usable? complete? warnings? age?
|
|
461
|
+
node replay.mjs replay --archive <dir> --dist recorded=/builds/stable/dist \
|
|
462
|
+
--calibrate --json --exit-code # result.calibration.verdict
|
|
463
|
+
node replay.mjs replay --archive <dir> --dist base=/builds/stable/dist --dist candidate=./dist --json
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
- `--json` prints one result object on stdout and sends the log to stderr. An error
|
|
467
|
+
gives `{ "error": "..." }`.
|
|
468
|
+
- Exit codes: 0 done, 1 model error, 2 error, 3 record again. Codes 1 and 3 come only
|
|
469
|
+
with `replay --calibrate --exit-code`.
|
|
470
|
+
- **Gaps.** `record` and `inspect` compare the archive with the field timing runs. A
|
|
471
|
+
request that at least half of the runs made, that the archive has no response for,
|
|
472
|
+
and that the build does not serve is a gap: the replay answers it with a 404. They
|
|
473
|
+
return `complete` and `gaps`. A gap usually means that the body run stopped too early;
|
|
474
|
+
record again with a longer `--settle`.
|
|
475
|
+
- **Calibration verdict.** `calibration.verdict` says what to do next:
|
|
476
|
+
|
|
477
|
+
| Verdict | Meaning | Next step |
|
|
478
|
+
| -------------- | --------------------------------------------------------- | ------------------------------------------------ |
|
|
479
|
+
| `pass` | Every row is within tolerance and the archive has no gaps | An A/B on this archive is usable |
|
|
480
|
+
| `record-again` | The archive has gaps | Record again (longer `--settle`) |
|
|
481
|
+
| `model-error` | The archive is complete, but a row is off | Report the rows. A new recording does not fix it |
|
|
482
|
+
|
|
483
|
+
- `record` writes `HANDOFF.txt` next to `archive.json`. It holds the next commands
|
|
484
|
+
with absolute paths. `record --out` replaces an earlier archive, and refuses a folder
|
|
485
|
+
that is not empty and holds no `archive.json`.
|
|
486
|
+
- `archive.json` has `schemaVersion` and `recordedWith` (tool, Chrome and Node
|
|
487
|
+
versions, cookie names, never values). `inspect` and `replay` refuse an archive with
|
|
488
|
+
another schema or with missing bodies. Record again in that case.
|
|
489
|
+
- `inspect` warns when the archive has no network rating, no bandwidth, no named build,
|
|
490
|
+
or is more than 24 h old.
|
|
491
|
+
- `--build` on `record` names the local copy of the build that the site serves. The
|
|
492
|
+
replaying agent uses it for the calibration and the `next` commands.
|
|
493
|
+
- **Late chunks.** `inspect` runs the chunk discovery check (see "Chunk discovery") on
|
|
494
|
+
the field timing runs with the `--build` manifest. `replay` runs it on each dist's
|
|
495
|
+
replay runs. Both read the CDP requests, not the Resource Timing list, and use the
|
|
496
|
+
archive's `rtt` as the threshold. They return `lateChunksBeforeLcp`, and `replay`
|
|
497
|
+
writes the full result to `runtime-metrics.json` → `chunkDiscovery`. These are leads,
|
|
498
|
+
not predicted gains: a replay A/B decides. The check needs `.vite/manifest.json` that
|
|
499
|
+
is not older than the files it names; copy a build with `mv` or `cp -p`, not `cp -R`.
|
|
500
|
+
|
|
501
|
+
### How the replay works
|
|
502
|
+
|
|
503
|
+
| Part | What it does |
|
|
504
|
+
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
505
|
+
| Archive | Every response of the body run with safelisted headers (type, encoding, CORS, `vary`, `location`, `link`; never cookies or auth), request body hashes (never request bodies), the median server time of each call over the timing runs, volatile query parameters (values that change between loads), the field medians, and the timing runs (`timing.json`). |
|
|
506
|
+
| One server for every host | Chrome runs with `--host-resolver-rules`, so the page keeps its real origin (cookies, CORS, absolute URLs unchanged) and no request leaves the machine. HTTPS + HTTP/2, like the recorded site. |
|
|
507
|
+
| Document | The dist's `index.html` plus every tag that only the deployed HTML has (a host's nonce, analytics snippet, service-worker or bot-check loader), after the recorded TTFB. The head comes at once and the rest after the recorded download time, like the recorded early flush. `--document local\|recorded` serves either file bare. An iframe document is served like any recorded response. |
|
|
508
|
+
| Static files | From the dist, compressed like the recorded assets (encoding and level fitted on the recorded sizes). Each time to first byte is DRAWN from the recorded TTFB of files that loaded on an idle link (less than 50 kB of other downloads during the wait): a startup step waits for the slowest of its parallel chunks, so a constant median makes every step too fast. Both builds of a pair use the same seed. |
|
|
509
|
+
| Wire queue | On a busy slow link a response's first byte comes late because of the page's OWN downloads. The replay keeps a FIFO of `wireBytes` (the median extra TTFB of files fetched on a busy link × bandwidth) between the scheduler and the link. Static response headers wait behind the bytes in it, and the body follows them without a second wait. `--wire <kB>` overrides it. Drawing from ALL recorded TTFBs instead counts this queue two times. |
|
|
510
|
+
| API | The recorded response after its recorded server time, matched by method + host + path + query without volatile parameters, then call order (request bodies: exact hash, then normalized hash). A recorded late body sends its last byte after the recorded download time, outside the bandwidth queue (the recorded time already holds that wait). An unrecorded request gets a 404 and is listed in the report. |
|
|
511
|
+
| Bandwidth | One token bucket for all streams, most urgent first (`priority: u=N`). A delay per request would let 80 parallel chunks all run at full speed. The default rate is the median over the timing runs of the busiest 250 ms window. On a link whose rate changes (for example a VPN), this estimate is too high; use `--bandwidth <Mbit/s>` and calibrate. |
|
|
512
|
+
| Chrome's network rating | Over loopback Chrome rates the network as fast. On a slow real network (rating `3g` or slower) it holds back low-priority requests even on HTTP/2, which can add seconds. The replay forces the recorded rating (`navigator.connection.effectiveType`); `--ect` overrides it. |
|
|
513
|
+
| Storage | The field runs load the page in a profile that visited it before. A page that caches data in `localStorage` (a boot payload, flags) starts from that copy there and does not wait for the API. `--storage warm` first loads the page once, unmeasured, in the same browser context, so the build writes its own storage; the measured load reads it. The HTTP cache stays off and service workers are bypassed on every load, as in the field runs, so only storage and cookies carry over. Default: `warm` when the recorded page had `localStorage`, else `empty`. |
|
|
514
|
+
| Calibration | `--calibrate` compares the first dist's medians with the field medians (LCP, document end, gate start, gate - doc, gate TTFB, after gate). Each must be within max(15%, 150 ms). A row that the field has and the replay lacks (no gate found) is off. The verdict (`pass`, `record-again`, `model-error`) says what to do next; `timing.json` holds the field waterfalls to find a model error. |
|
|
515
|
+
|
|
516
|
+
**Privacy.** The archive holds REAL responses from your account. It goes to
|
|
517
|
+
`$TMPDIR/perf-kit-replay/` with owner-only permissions, and `record` refuses a path
|
|
518
|
+
inside a git work tree. Delete it when done.
|
|
519
|
+
|
|
520
|
+
### What replay does NOT do
|
|
521
|
+
|
|
522
|
+
- **It models one recorded network and server.** A result holds for those conditions;
|
|
523
|
+
record again when the network changes (a slower RTT changes Chrome's rating, and with
|
|
524
|
+
it which requests wait).
|
|
525
|
+
- **It does not copy the recorded profile's storage.** With `--storage warm` the build
|
|
526
|
+
writes its own storage in an unmeasured first load. Storage that only older visits or
|
|
527
|
+
other pages write (the archive's `page` field lists the cookie names and the number of
|
|
528
|
+
localStorage keys) is not there. A build that reads it can take another path; the
|
|
529
|
+
report warns when the replay lands on another path than the field.
|
|
530
|
+
- **It does not model packet loss.** A lost packet on a real connection holds every
|
|
531
|
+
stream until it is resent. The sampled TTFB covers only part of this.
|
|
532
|
+
- **New API calls of a candidate build have no response.** They get a 404 and the
|
|
533
|
+
report lists them; the result may not hold.
|
|
534
|
+
|
|
535
|
+
## The loop protocol (for an agent)
|
|
536
|
+
|
|
537
|
+
1. **Baseline**: `gen` → `build` → `measure --runs 5 --label baseline` → `baseline`.
|
|
538
|
+
2. **Find a candidate**: `coverage`. Candidates are modules ≥3KB with <2% of their
|
|
539
|
+
bytes executed at first paint, largest first. Modules hot at paint (e.g. the
|
|
540
|
+
demo's `i18n`, `hero_data`) are critical-path — never defer them, even though
|
|
541
|
+
`hero_data` structurally could be.
|
|
542
|
+
3. **Mutate**: `defer <top candidate>` → `build`.
|
|
543
|
+
4. **Judge**: `measure --runs 5 --label "defer <name>"`, then read
|
|
544
|
+
`state/runtime-metrics.json`:
|
|
545
|
+
- **Guard must pass**: `guard.allFeaturesReady && guard.heroRendered &&
|
|
546
|
+
guard.fcpObservedInAllRuns && guard.lcpObservedInAllRuns`, and `runtime.cls` must not grow by more than 0.02.
|
|
547
|
+
A faster build that broke a feature is a revert, not a win.
|
|
548
|
+
- **Improvement must beat noise**: check `paintComparison` and `baselineDelta`.
|
|
549
|
+
FCP or LCP must improve by at least max(30ms, 2% of its baseline). Neither
|
|
550
|
+
metric may regress by that amount. Use a full scan with `--no-early-stop`.
|
|
551
|
+
5. **Decide**:
|
|
552
|
+
- Accept → `baseline` (re-pin: this is the new reference).
|
|
553
|
+
- Revert → `undefer <name>` → `build`, and don't retry that candidate.
|
|
554
|
+
6. **Repeat** from 2. **Converged** when no candidates remain, or 2–3 consecutive
|
|
555
|
+
reverts, or the last accepted improvement is under ~2%.
|
|
556
|
+
7. **Compare once**: `scan --parity`. A difference from the original starts recover
|
|
557
|
+
mode — fix or revert until a check of every page is clear; the number you report
|
|
558
|
+
comes from that build.
|
|
559
|
+
|
|
560
|
+
Log the decision trail with `--label`; every measure also appends to
|
|
561
|
+
`state/history.jsonl`.
|
|
562
|
+
|
|
563
|
+
## Outputs
|
|
564
|
+
|
|
565
|
+
- `state/runtime-metrics.json` — flat metric ids (`runtime.lcp_ms`,
|
|
566
|
+
`runtime.lcp_p75_ms`, `runtime.fcp_ms`, `runtime.ttfb_ms`, `runtime.load_ms`,
|
|
567
|
+
`runtime.cls`, `runtime.transfer_bytes`, `runtime.js_request_count`), `guard`,
|
|
568
|
+
per-run `samples`, `delta`, `baselineDelta` — same delta/baseline conventions as
|
|
569
|
+
the build-side `metrics.json`.
|
|
570
|
+
- `state/coverage.json` — per-module `totalBytes` / `paintBytes` / `settleBytes`
|
|
571
|
+
(+ ratios), the sorted `candidates` list, and `coldAtPaint` (top modules by
|
|
572
|
+
`totalBytes − paintBytes`, framework runtimes flagged).
|
|
573
|
+
- `app/node_modules/.rolldown/report/dist/` — the build-side report (`output.max_initial_load_bytes`
|
|
574
|
+
should drop with every accepted defer while `output.total_bytes` stays flat).
|
|
575
|
+
- `state/parity/` — `anchor/` (the original build's copy + `manifest.json`: commit,
|
|
576
|
+
content hash, pinned pages and masks with the time each was added, the original's
|
|
577
|
+
LCP element), `report.json` (per page: state, the judged reasons, notes, region
|
|
578
|
+
explanations, approval signature), `approvals.json`, and `pages/<page>/*.png`.
|
|
579
|
+
|
|
580
|
+
## The demo app
|
|
581
|
+
|
|
582
|
+
Client-rendered page (LCP = the hero `<h1>` painted by `main.ts`), ~381KB entry.
|
|
583
|
+
Each module demonstrates one case the loop must get right:
|
|
584
|
+
|
|
585
|
+
| Module | ~KB | Behavior | Expected verdict |
|
|
586
|
+
| -------------------- | --- | ----------------------------------------- | -------------------------------------- |
|
|
587
|
+
| `features/charts` | 145 | runs post-paint (below fold) | defer → big LCP win |
|
|
588
|
+
| `features/markdown` | 107 | runs only on click | defer → big LCP win |
|
|
589
|
+
| `features/analytics` | 59 | runs post-paint | defer → win |
|
|
590
|
+
| `features/badges` | 4 | runs post-paint, tiny | defer → within noise → revert |
|
|
591
|
+
| `features/hero_data` | 25 | **executes before paint** (hero subtitle) | excluded by coverage, not by structure |
|
|
592
|
+
| `i18n` | 39 | executes before paint (hero copy) | excluded |
|
|
593
|
+
|
|
594
|
+
All weight lives inside function bodies so V8 coverage can separate "parsed" from
|
|
595
|
+
"ran before paint"; every feature reports readiness on `window.__ready` so the
|
|
596
|
+
guard catches a defer that broke behavior.
|
|
597
|
+
|
|
598
|
+
## Caveats
|
|
599
|
+
|
|
600
|
+
- Lab numbers are lab-only (fast-3G-ish throttle, 4× CPU, cold cache, localhost).
|
|
601
|
+
The signal is the delta between builds under identical conditions.
|
|
602
|
+
- V8 coverage counts a module's top level as executed at evaluation, so real-world
|
|
603
|
+
modules whose weight is top-level data look "used at paint" even if nothing reads
|
|
604
|
+
them — a known blind spot; judge those by size + manual inspection. The same blind
|
|
605
|
+
spot applies to `node-scan`'s cold-at-ready list.
|
|
606
|
+
- Server startup readiness discovered by polling (`port:` / `http://`) quantizes to the
|
|
607
|
+
poll interval. When the addressable time is only a few intervals wide, `node-scan`
|
|
608
|
+
says so rather than reporting millisecond deltas it cannot resolve — judge byte and
|
|
609
|
+
module counts there, not milliseconds.
|
|
610
|
+
- `defer`/`undefer` is a marker-block codemod, i.e. demo-app sugar for what an agent
|
|
611
|
+
does on a real codebase: rewriting the import site into a post-paint `import()`.
|
|
612
|
+
- Lab INP is meaningless (no real interaction); field metrics are Phase 3's
|
|
613
|
+
beacon path, not this harness.
|
|
614
|
+
- Parity sees what the build renders **served statically, without interaction**:
|
|
615
|
+
states behind a click, a login or live API data are not compared (both builds
|
|
616
|
+
render the same failed-fetch state, which is compared). Full pages are compared
|
|
617
|
+
down to 10 viewport heights, styles explain the first 4000 elements, shadow-DOM
|
|
618
|
+
content is compared by pixels only, and workers keep an unseeded `Math.random`.
|
|
619
|
+
Cross-origin resources (third-party fonts, CDNs) are fetched live by both builds:
|
|
620
|
+
a flaky one mostly shows up as instability between the two originals or fails the
|
|
621
|
+
confirmation capture, but one that fails exactly while the current build is being
|
|
622
|
+
captured (twice) reads as a difference.
|