@graphty/visual-review 0.0.1 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Adam Powers
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,7 +1,593 @@
1
1
  # @graphty/visual-review
2
2
 
3
- This version is a placeholder. It exists so that later versions can be published from GitHub
4
- Actions through npm trusted publishing, which requires the package to exist first.
3
+ Visual regression review for Storybook, hosted by nobody. GitHub Actions screenshots every story
4
+ on every pull request and compares each screenshot with an approved baseline PNG kept in your
5
+ repository (in Git LFS). You open a review page on your own machine, see every difference side by
6
+ side, and accept, reject or exclude each one. Accepting commits the new baselines to the pull
7
+ request's branch. A required check, the "Visual gate", keeps a pull request from merging while it
8
+ holds a difference nobody accepted.
5
9
 
6
- The first real release, and its documentation, will be published from
7
- https://github.com/graphty-org/graphty-monorepo (the `visual-review/` package).
10
+ There is no service, account or per-snapshot bill: the captures live in GitHub Actions artifacts
11
+ for 30 days, the baselines live in git, and the review page reads both through the `gh` CLI with
12
+ your login.
13
+
14
+ - [How it works](#how-it-works)
15
+ - [Requirements](#requirements)
16
+ - [Install and set up](#install-and-set-up)
17
+ - [Configuration](#configuration)
18
+ - [The GitHub Actions workflows](#the-github-actions-workflows)
19
+ - [Your first review: seeding baselines](#your-first-review-seeding-baselines)
20
+ - [Opening the review page](#opening-the-review-page), [the screens](#the-screens), [keys](#keys),
21
+ [decisions](#what-each-decision-does), [Finish](#finish)
22
+ - [Seeding one story at a time](#seeding-one-story-at-a-time)
23
+ - [Iterating on a story before a pull request exists](#iterating-on-a-story-before-a-pull-request-exists)
24
+ - [Story parameters](#story-parameters)
25
+ - [Reorganizing stories: renames](#reorganizing-stories-renames)
26
+ - [What the gate does and does not guarantee](#what-the-gate-does-and-does-not-guarantee)
27
+ - [Troubleshooting](#troubleshooting)
28
+
29
+ ## How it works
30
+
31
+ 1. **Capture (CI).** On every pull request and every push to your default branch, a workflow job
32
+ per Storybook builds it, opens every story in Chromium (1200 x 900 at device scale factor 2,
33
+ a fixed clock, software WebGL, WebGPU removed), screenshots it, and compares the screenshot
34
+ with its baseline. Anything that differs is captured a second time, so a real change, an
35
+ unstable story and a one-off flake are told apart. The results (`results.json` and the PNGs
36
+ worth looking at) are uploaded as an artifact.
37
+ 2. **Review (your machine).** `visual-review serve` lists your open pull requests, downloads their
38
+ captures, and serves a page with a grid of every change and a side-by-side, flash, highlight
39
+ and spotlight view of each. You accept, reject (with a reason) or exclude each item.
40
+ 3. **Finish.** One button applies your decisions: the accepted PNGs and a review record are
41
+ committed and pushed to the pull request's branch, and the rejects are posted as one comment.
42
+ CI captures again, and the accepted items now read `unchanged`.
43
+ 4. **Gate (CI).** The "Visual gate" job fails while a pull request holds a difference nobody
44
+ accepted, or a baseline file changed without a review record naming it.
45
+
46
+ A story with no baseline yet does not block anything until a pull request changes it, so you can
47
+ seed baselines a few stories at a time.
48
+
49
+ ## Requirements
50
+
51
+ - **A git repository on GitHub, with GitHub Actions.** The review page talks to GitHub through
52
+ the [`gh` CLI](https://cli.github.com), logged in (`gh auth login`) as someone who can push to
53
+ the repository's branches.
54
+ - **Node.js 20 or newer**, locally and in CI.
55
+ - **Storybook 7 or newer**, built as a static site (`storybook build`), which writes the
56
+ `index.json` the capture reads.
57
+ - **Playwright**, a peer dependency: install it next to this package. `visual-review
58
+ install-browser` installs the Chromium that version of Playwright drives.
59
+ - **git-lfs**, on every machine that accepts baselines or pushes a branch holding them. GitHub's
60
+ runners have it. Install it with your package manager (`brew install git-lfs`,
61
+ `sudo apt-get install git-lfs`), or without root put the `git-lfs` binary from
62
+ https://github.com/git-lfs/git-lfs/releases on your `PATH`; then run `git lfs install` once,
63
+ and `git lfs pull` in every checkout that already existed.
64
+ - **jq** on the runners, which GitHub's Ubuntu runners have.
65
+
66
+ ## Install and set up
67
+
68
+ ```bash
69
+ npm install --save-dev @graphty/visual-review playwright
70
+ npx visual-review init
71
+ ```
72
+
73
+ (`pnpm add -D` and `yarn add -D` work the same way; `init` notices the lockfile and writes
74
+ workflows for that package manager.)
75
+
76
+ `init` writes, at the root of your repository, and never overwrites a file you already have:
77
+
78
+ | File | What it is |
79
+ | ------------------------------------- | ---------------------------------------------------------------------------------------- |
80
+ | `visual-review.config.json` | Your Storybooks and your repository's settings ([Configuration](#configuration)) |
81
+ | `.gitattributes` | `visual-baselines/**/*.png filter=lfs diff=lfs merge=lfs -text`: baselines go to Git LFS |
82
+ | `.gitignore` | `/.visual-review/`, where the review page downloads captures and keeps its state |
83
+ | `.github/workflows/visual-review.yml` | Captures every pull request and push, and gates pull requests |
84
+ | `.github/workflows/visual-seed.yml` | Captures an older commit on demand, to seed baselines from |
85
+
86
+ Then:
87
+
88
+ 1. Edit `visual-review.config.json`: one entry under `projects` per Storybook, with the directory
89
+ its build writes and the command that builds it.
90
+ 2. Commit everything and open a pull request. Its "Visual review" run captures every story; with
91
+ no baselines yet, nothing blocks.
92
+ 3. Make **Visual gate** a required status check (Settings, then Branches or Rulesets).
93
+ 4. Merge, then seed your first baselines ([below](#your-first-review-seeding-baselines)).
94
+
95
+ `visual-review init --force` rewrites the two workflows when they still start with the line
96
+ `init` writes ("Generated by visual-review init"), to pick up a newer template after an upgrade.
97
+ It never replaces the config or a workflow you wrote yourself.
98
+
99
+ Every command has `--help`; `visual-review --help` lists them.
100
+
101
+ ## Configuration
102
+
103
+ `visual-review.config.json` sits at the root of the repository. Only `projects` is required.
104
+
105
+ ```json
106
+ {
107
+ "defaultBranch": "main",
108
+ "workflow": "visual-review.yml",
109
+ "baselines": "visual-baselines",
110
+ "workDir": ".visual-review",
111
+ "commitPrefix": "test",
112
+ "issueLabels": ["bug"],
113
+ "projects": {
114
+ "web": {
115
+ "storybook": "packages/web/storybook-static",
116
+ "build": "npm run build-storybook --workspace packages/web",
117
+ "workers": 4
118
+ },
119
+ "charts": {
120
+ "storybook": "packages/charts/storybook-static",
121
+ "build": "npm run build-storybook --workspace packages/charts",
122
+ "seedFromDefaultBranch": false,
123
+ "waitFor": { "selector": "my-chart", "method": "whenRendered", "failOnConsole": "render timeout" }
124
+ }
125
+ }
126
+ }
127
+ ```
128
+
129
+ | Key | Default | Meaning |
130
+ | --------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------- |
131
+ | `defaultBranch` | `main` | The branch baselines are seeded from and pull requests merge into |
132
+ | `workflow` | `visual-review.yml` | The workflow file whose runs hold the captures; the review page looks runs up by it |
133
+ | `baselines` | `visual-baselines` | Where baselines live: `<baselines>/<project>/<story id>[.<mode>].png`, and review records in `<baselines>/reviews/` |
134
+ | `workDir` | `.visual-review` | Where `serve` downloads captures and keeps its decisions and session token; keep it out of git |
135
+ | `commitPrefix` | `test` | The conventional-commit type and scope of the commits Finish makes, e.g. `test(ui)` |
136
+ | `issueLabels` | `["bug"]` | Labels of the issue Finish opens for rejects on the default branch; each must exist |
137
+ | `projects` | (required) | One entry per Storybook; the id names its baselines directory, CI job and artifact |
138
+
139
+ Per project:
140
+
141
+ | Key | Default | Meaning |
142
+ | ----------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
143
+ | `storybook` | (required) | The built Storybook's directory, relative to the repository root |
144
+ | `build` | none | The shell command CI runs to build it (from the repository root) |
145
+ | `workers` | `4` | How many browsers capture in parallel |
146
+ | `seedFromDefaultBranch` | `true` | `false`: the project's first baselines are accepted on a pull request, not seeded from the default branch |
147
+ | `waitFor` | none | After a story renders, call `method()` on every element matching `selector` and wait for the promise it returns, for a component that keeps drawing after Storybook says it is done. A console line containing `failOnConsole` fails the story |
148
+
149
+ Project ids are letters, digits, `.`, `_` and `-`. The pull request gate reads the config as it is
150
+ on the base branch, so a pull request cannot move `baselines` out from under it.
151
+
152
+ ## The GitHub Actions workflows
153
+
154
+ **`visual-review.yml`** runs on every pull request and every push to the default branch:
155
+
156
+ - **Plan** reads the projects from the config.
157
+ - **visual (&lt;project&gt;)**, one job per project: fetches that project's baselines from Git LFS
158
+ (cached), installs your dependencies, installs Chromium, runs the project's `build`, downloads
159
+ the default branch's newest capture as a reference (on pull requests), captures, and uploads
160
+ the artifact `visual-<project>-<attempt>` (kept 30 days). It never fails because of a
161
+ difference; `continue-on-error` keeps even a crash of the tool from failing the run.
162
+ - **Visual gate** (pull requests only) downloads every capture of the run and runs
163
+ `visual-review gate` at the version `init` pinned, with `npx`, so a pull request's own
164
+ dependencies cannot change it. Make it a required check.
165
+
166
+ The review page finds captures by the workflow's file name (the config's `workflow`), the jobs by
167
+ their names, `visual (<project>)`, and the artifacts by `visual-<project>-<attempt>`. If you would
168
+ rather capture inside an existing CI workflow (to reuse a Storybook your build job already made),
169
+ copy the `visual` job and the gate's steps into it, keep those names, and set `workflow` to that
170
+ file.
171
+
172
+ **`visual-seed.yml`** is started by hand to capture an older commit with the default branch's
173
+ tool: `gh workflow run visual-seed.yml --ref main -f ref=<sha>`. See
174
+ [Seeding](#seeding-one-story-at-a-time).
175
+
176
+ Both need nothing but the default `GITHUB_TOKEN`: the capture job reads Actions artifacts
177
+ (`actions: read`); nothing in CI writes to the repository.
178
+
179
+ ## Your first review: seeding baselines
180
+
181
+ A project has no baselines until you accept some, and until it has one the gate ignores it. After
182
+ the setup pull request merges, the default branch's push runs the capture:
183
+
184
+ 1. Find that run's id: `gh run list --workflow visual-review.yml --branch main --limit 1`.
185
+ 2. Start the page with `--master-run <run id>` ([Opening the review page](#opening-the-review-page))
186
+ and open "master (seed)" (the page calls the default branch's target "master", whatever its
187
+ name). Every story is `new` there.
188
+ 3. Accept what looks right, reject what does not (with a reason), leave the rest, and press
189
+ Finish. You get a pull request `visual/seed-<date>` holding the accepted baselines, and one
190
+ issue listing the rejects.
191
+ 4. Merge the seed pull request once its own run shows its accepted items `unchanged`. From then
192
+ on, every pull request that changes how a seeded story looks is blocked until you accept it.
193
+
194
+ ## Opening the review page
195
+
196
+ ```bash
197
+ PORT=4800 npx visual-review serve
198
+ ```
199
+
200
+ It prints the address to open, with a session token after `#token=`; open that exact URL. The
201
+ token is kept in the work directory, so the URL stays valid across restarts; delete
202
+ `<workDir>/state/token` to issue a new one. Without the token the page shows "No session token".
203
+
204
+ - Without a certificate it serves plain HTTP, and only on `localhost` (`HOST` defaults to it).
205
+ To open the page from another device (an iPad, say), serve HTTPS: set `HOST` to the machine's
206
+ name and `HTTPS_CERT_PATH` and `HTTPS_KEY_PATH` to a certificate and key for it.
207
+ - `serve` refuses to start without git-lfs: an accept would commit raw PNGs.
208
+ - `--master-run <run id>` also lists the default branch at that run, for seeding.
209
+ - `--results <dir>` serves local captures offline (a directory of `<project>/results.json`), for
210
+ looking at a story before a pull request exists. It is listed as "Local preview" and is look
211
+ only: no Accept, Reject or Exclude, and no Finish. Only CI captures of a pushed commit are
212
+ decided.
213
+
214
+ ### Links to a screen
215
+
216
+ The address always names the screen you are on, after the token: the targets list; a pull
217
+ request (or master) and project with the grid's filter and text; or one story with its view and
218
+ zoom, for example
219
+ `#token=...&target=123&project=web&filter=undecided&item=button--primary.dark.png&view=flash&zoom=2`.
220
+ Opening that address, in another tab or on another device, opens the same screen. **Copy link**
221
+ at the top right copies it. The link carries your session token, so it works on your iPad the way
222
+ the printed URL does; keep it to yourself as you would that URL. All of it sits after `#`, which a
223
+ browser never sends to any server or in a Referer, so the page never hands the token to another
224
+ site. Back and Forward move between the targets list, a grid and a story; moving between stories
225
+ or views of one grid updates the address in place.
226
+
227
+ A link to something that is gone opens the nearest screen that still exists, and the status line
228
+ says why: a story not in the newest CI run opens its grid, and a pull request no longer listed
229
+ (closed, or no CI run) opens the targets list.
230
+
231
+ Finish's commit is signed by the git configuration of the process that runs the server. If
232
+ someone else started it for you (an agent, a service manager), the commit carries their
233
+ identity: the page names the key that will sign before every Finish and prints the command that
234
+ starts the same server from your own shell.
235
+
236
+ ## The screens
237
+
238
+ 1. **Targets.** Each open pull request with a run of the capturing workflow, and the default
239
+ branch (shown as "master (seed)") when started with `--master-run`. Per project: how many items need a decision, how many you decided, and badges:
240
+ - **merge master first**: the default branch has newer baselines for this project than the
241
+ pull request. Merge the default branch into the pull request's branch (by merge, never
242
+ rebase) and wait for CI.
243
+ - **capture failed**: the `visual` job produced no results. Re-run that job in GitHub Actions.
244
+ - **incomplete: N of M stories**: the capture stopped part way. Re-run the job.
245
+ - **not seeded from master**: this project is not reviewed on the default branch
246
+ (`"seedFromDefaultBranch": false` in the config); its first baselines are accepted on a pull
247
+ request.
248
+ 2. **Grid.** It opens on **Needs a decision** (the undecided items, counted on the button); **All**
249
+ and one button per status show the rest, and **Accepted**, **Rejected** and **Excluded** show
250
+ what you decided, each counted, as Chromatic's review does. A line above the grid splits what is shown into
251
+ errors and images to compare, so the counts always add up. At the top, **Errors** lists every failed capture with its reason and,
252
+ under "console and stack", the story's console output and the thrown error's stack (a play
253
+ function's failed `expect` included). An error is never accepted: fix the story, re-run the
254
+ `visual` job for a one-off timeout, or exclude it with a reason. Below it the items are grouped
255
+ by component (the story id before `--`), components with a changed item first, then new,
256
+ unstable and removed ones; each story's modes (light, dark) sit side by side under its name.
257
+ Every tile is numbered, and the number is the story screen's "N of M". A component's
258
+ **Accept N undecided** accepts that component's undecided items without opening them, after
259
+ asking. Under every decided tile (and every decided error) its decision is spelled out:
260
+ "Accepted", "Accepted (not opened)" for one Accept all took, or "Rejected" or "Excluded" with
261
+ the reason. Its **Undo** clears it without opening the story. A component's **Undo N
262
+ decisions**, and **Undo all decisions** beside Accept all for the whole project, clear many at
263
+ once: the first press turns the button into "Confirm: undo N decisions", a second press undoes,
264
+ and Escape or any other change to the grid cancels. Every Undo here is the same request as the
265
+ story screen's U. A reject an earlier Finish already posted says "Posted by Finish: stays" and
266
+ has no Undo on the grid; the bulk Undo buttons leave it too. **Filter by story id** narrows the grid; **Go to** opens item N, or the first item
267
+ whose id contains the text. Coming back from a story, its tile is outlined and scrolled into
268
+ view.
269
+ 3. **Story.** One item, on one screen: the controls on top, then two panes of the same size side
270
+ by side, the baseline on the left and the new capture on the right, filling the rest of the
271
+ window. Images open at **Fit to screen**: both whole images fit their panes, across and down,
272
+ at one scale (never above real size), so two captures of the same size line up pixel for pixel
273
+ and nothing scrolls. With no baseline (a new story, or "no baseline yet") the left pane stays
274
+ as an empty frame labelled "No baseline", so the new image sits exactly where it would beside
275
+ one; a removed or failed story leaves the right pane empty the same way. **Real size (1x)** is
276
+ one CSS pixel of the page for each CSS pixel the story was drawn at (a capture holds two image
277
+ pixels per CSS pixel). **2x**, **4x** and **8x** enlarge it; from 4x pixels are drawn as hard
278
+ squares. Zoomed, the images grow past their panes, which scroll: scrolling one scrolls the
279
+ other to the same place, and **Fit to screen** returns to the whole image. (On an iPad,
280
+ pinching zooms the whole page; use the zoom buttons to zoom the images.) **Next changed box**
281
+ (N) scrolls both panes until the next region of changed pixels is in view and outlines it;
282
+ "box i of k" counts them. The views, each shown in the right pane at the same scale and place:
283
+ **Side by side**; **Flash**, which shows baseline and new one after the other in the same
284
+ place, about 1.5 times a second (the images themselves, not an overlay), keeping the zoom and
285
+ scroll it was opened at; **Highlight**, pixelmatch's changed pixels in red over the dimmed
286
+ baseline; and **Spotlight**, the new image dimmed everywhere except around the changed pixels
287
+ (each grown by 10 image pixels), which finds a one-pixel change. Flash, Highlight and
288
+ Spotlight need two images; on a new or removed story they are off and the page says why
289
+ ("New story, no baseline", "Only one image: this story was removed"). Badges here:
290
+ **size changed** (in image pixels), **flaky** (the two captures differed, then matched), and
291
+ **re-review** (an accept you made was replaced by the default branch's newer baseline).
292
+
293
+ **Next** and **Previous** (J and K) walk one pass: the items the grid showed when you opened
294
+ the story, in the grid's order, frozen until you go back to the grid. Accepting, rejecting or
295
+ excluding an item never drops it from the pass: the decision moves on to the next item, and
296
+ **Previous** comes back to the one just decided, showing its decision and an **Undo** (or U).
297
+ Going back to the grid shows what its filter now selects: under **Needs a decision** the items
298
+ you decided have left it, and the count has gone down; **Accepted**, **Rejected** and
299
+ **Excluded** show them with their decisions.
300
+
301
+ Statuses: `changed` (differs from its baseline), `moved` (a renamed story that looks exactly as
302
+ its old id's baseline; see [renames](#reorganizing-stories-renames)), `new` (no baseline, and on a pull request the
303
+ story is new or looks different from the default branch's newest capture of it), `no baseline yet` (status
304
+ `unseeded`: no baseline, and the pull request does not change it), `removed` (a baseline whose
305
+ story no longer exists, lost a mode, or whose story's own parameters now exclude it), `unstable`
306
+ (two captures of the same commit differed), `failed` (did not render, even after one retry).
307
+
308
+ `no baseline yet` items are listed under their own filter in the grid and never need a decision:
309
+ they do not block the pull request, Accept all skips them, and the story screen offers no buttons
310
+ for them. Seed them from the default branch (below), or accept them on the pull request that changes them.
311
+
312
+ ## Keys
313
+
314
+ | Key | Action |
315
+ | ------------ | ------------------------------------------------------------------------------ |
316
+ | J / K | Next / previous item of this pass (decided items stay in it) |
317
+ | A | Accept an undecided item |
318
+ | R | Reject an undecided item (asks for a reason, then Enter) |
319
+ | E | Exclude an undecided item (asks for a reason, then Enter, then a confirmation) |
320
+ | U | Undo the item's decision (on the grid: each tile's Undo button) |
321
+ | F | Flash between baseline and new; F again returns to side by side |
322
+ | H | Highlight changed pixels; H again returns to side by side |
323
+ | S | Spotlight the changes; S again returns to side by side |
324
+ | Z | Next zoom: fit to screen, real size, 2x, 4x, 8x, then fit again |
325
+ | N | Next changed box |
326
+ | Space (hold) | Flash while held |
327
+ | Shift+A | Accept every undecided item of this project without opening it (asks first) |
328
+ | Escape | Back to the grid from a story, wherever the focus is (the reason box included) |
329
+ | Escape | On the grid: cancel an Undo N decisions or Undo all decisions pressed once |
330
+
331
+ No key reverses a decision. A, R and E do nothing on an item that is already decided, and say
332
+ so; to change a decision, press U (or the Undo button) first. The same key twice never undoes.
333
+
334
+ ## What each decision does
335
+
336
+ - **Accept**: the new screenshot becomes the baseline (or, for `removed`, the baseline is
337
+ deleted). Allowed on `changed`, `moved`, `new` and `removed`. For a renamed story the baseline
338
+ is written under the new id and the old id's baseline is deleted, in the same commit.
339
+ - **Reject**: the difference is a regression. It always needs a reason, which is posted to the pull
340
+ request as a comment with a machine-readable block an agent can read. The pull request stays
341
+ blocked until its code changes so the capture matches the baseline again.
342
+ - **Exclude**: stops capturing the story. It needs a reason and writes
343
+ `<baselines>/<project>/<story id>.json` with `disableSnapshot: true`. It drops **every mode
344
+ of the story**, on every later pull request, until that file is deleted. It is the only
345
+ decision for `unstable` and `failed` items; for a one-off `failed` item (a timeout on a busy
346
+ runner), re-run the `visual` job instead, since the newest attempt replaces the old results.
347
+ - **Undo** (U, or a tile's Undo on the grid) clears a decision before Finish; it is the only way
348
+ to change one. The grid also undoes a whole component or project, after a second press.
349
+ Decisions are kept across server restarts.
350
+ - After Finish, accepts and exclusions are cleared; rejects stay, marked as already posted, and
351
+ still show as rejected on the next CI run while the capture is unchanged. Finish does not post
352
+ them twice. They live in the work directory's `state/` (the config's `workDir`), not in the
353
+ repository.
354
+
355
+ ## Finish
356
+
357
+ Finish applies every decision on one target at once:
358
+
359
+ - **A pull request:** one commit holding the accepted PNGs, the exclusion files and one review
360
+ record in `<baselines>/reviews/`, pushed to the pull request's branch, plus one comment
361
+ holding every reject. CI then recaptures, and the accepted items read `unchanged`.
362
+ - **One commit status**, "Visual review", posted once when Finish completes (never per
363
+ decision), on the commit Finish pushed, or on the captured commit when it pushed none. It
364
+ fails when anything was rejected, is pending while items are left undecided, and succeeds
365
+ otherwise; its description counts the accepts, rejects, exclusions and undecided items. It is
366
+ information for the pull request page, not a required check: the merge gate is the "Visual
367
+ gate" job. If posting it fails, the page says so; what was pushed and posted stays.
368
+ - **The default branch (seeding):** a branch `visual/seed-<date>` with the same commit and a pull
369
+ request from it, and one issue holding every reject (labelled with the config's `issueLabels`)
370
+ with the same machine-readable block, for a person or an agent to fix the stories. Rejects alone, with nothing accepted, open only the issue.
371
+
372
+ Finish runs on the server, not in the page. A seed of several hundred images takes minutes,
373
+ most of it uploading the images to Git LFS, which is longer than a browser (Safari on an iPad in
374
+ particular) keeps one request open. So pressing Finish only starts it, and the page then shows
375
+ each step as it happens: checking, writing the files, committing, uploading images to LFS (with a
376
+ count of the images uploaded so far), pushing, opening the pull request or posting the rejects,
377
+ and posting the status. When it ends, the page shows what was pushed and posted, or the error.
378
+ Closing or reloading the page does not stop it: reopen the page and it shows the running Finish
379
+ instead of a Finish button, and after it ends the result stays above the targets until the
380
+ server restarts. Only one Finish runs at a time, and decisions on that target are refused until
381
+ it ends.
382
+
383
+ The commit is signed by whatever git configuration the server process sees: yours when you
384
+ started it, someone else's when they (or an agent working for you) started it. The top of the
385
+ targets screen and Finish's confirmation name the key that will sign, where git found it and the
386
+ committer, and print the exact command that starts the same server from your own shell. If
387
+ Finish fails, your decisions are kept and the page shows git's or GitHub's message:
388
+
389
+ - **capture is stale, wait for CI**: someone pushed to the branch after the capture. Wait for the
390
+ new CI run, then decide again what still differs.
391
+ - **merge master first**: see the badge above.
392
+ - **failed to write commit object** or a signing error: unlock or plug in the signing key, then
393
+ Finish again.
394
+ - **the accepts were pushed ..., but the reject comment failed**: the accepts are done and cleared;
395
+ press Finish again to post the rejects.
396
+
397
+ ## Seeding: one story at a time
398
+
399
+ A story does not have to look right the first time, and nothing has to be seeded in one pass.
400
+ Seeding is per story:
401
+
402
+ 1. The review workflow captures every story on every push to the default branch. Start the
403
+ server with `--master-run <run id>` (that workflow's newest run on the default branch) and open
404
+ "master (seed)". Every story without a baseline is `new` there.
405
+ 2. **Accept** the stories that look right. **Reject** the ones that do not, with a reason saying
406
+ what is wrong. **Leave the rest** undecided; they simply stay without a baseline. Exclude only
407
+ stories that are unstable. Press Finish: the accepts become the seed pull request, and the
408
+ rejects become one issue whose machine-readable block says what to fix.
409
+ 3. Merge the seed pull request once its own capture shows its accepted items `unchanged`.
410
+
411
+ To seed from an older, known-good commit instead of the newest, capture it with the default
412
+ branch's tool: `gh workflow run visual-seed.yml --ref <default branch> -f ref=<sha>`, then start
413
+ the server with `--master-run <that run's id>`. It is listed as "master (seed)"; its results.json
414
+ names the captured commit, so Finish's seed branch starts from that commit.
415
+
416
+ A story with no baseline on the default branch is in the "no baseline yet" state. On every pull
417
+ request, CI compares its capture with the default branch's newest capture of that story:
418
+
419
+ - **The pull request does not change it:** `no baseline yet` (`unseeded`). It is shown, it does
420
+ not block the pull request, and it is never accepted by Accept all.
421
+ - **The pull request adds the story, or changes how it looks** (for example an agent fixing a
422
+ story you rejected): `new`. It blocks that pull request until you decide. Review it there;
423
+ accepting it creates its first baseline in that pull request's accept commit.
424
+
425
+ So seeding never restarts from scratch: each round accepts what now looks right, and the rest
426
+ waits, blocking nothing, until a pull request touches it. A project enters the merge gate when its
427
+ first baseline lands on the default branch; before that the gate ignores it entirely.
428
+
429
+ If the default branch's capture could not be downloaded (its artifacts expired, or no run there has
430
+ finished one), every story without a baseline is `new` on that pull request. Re-run its `visual` job
431
+ once the default branch's run has finished.
432
+
433
+ ## Iterating on a story before a pull request exists
434
+
435
+ To try a story's look quickly, capture it locally and look at it, as a PNG or in the page:
436
+
437
+ ```bash
438
+ npx visual-review install-browser # once per machine
439
+ npm run build-storybook # your project's build command
440
+ npx visual-review capture --project web --out .visual-review/preview/web --stories button--,badge--
441
+ ```
442
+
443
+ `--stories` captures only the story ids that start with one of the given prefixes, in seconds
444
+ rather than minutes, and then reports no baseline as removed. Start the server with
445
+ `--results .visual-review/preview` to see the capture beside its baseline. Capture and look again
446
+ after each change. A local preview is look only: its fonts and graphics stack are not CI's, so
447
+ only a CI capture of a pushed commit becomes a baseline. Push, let CI capture, and accept it on
448
+ the pull request.
449
+
450
+ ## How captures and baselines move
451
+
452
+ - **What a capture is.** Each story and mode is opened in a 1200 x 900 viewport at device scale
453
+ factor 2, as Chromatic captures, so a PNG holds two image pixels per CSS pixel. It is always
454
+ the whole canvas, in every project: the full page of the story iframe,
455
+ which is the whole viewport, or everything a scroll would reach when the story is taller or
456
+ wider. It is never cropped to the content, so a small component sits in the full canvas and
457
+ every capture of a project has the same size unless its story overflows. results.json records the scale as `scale`, and each review record
458
+ copies it into its `subject`.
459
+ - **From GitHub Actions to the page.** Each `visual` job uploads `results.json` and the PNGs to
460
+ review as an artifact `visual-<project>-<attempt>`, kept 30 days. The server lists open pull
461
+ requests with `gh`, finds each one's newest run of the capturing workflow, and downloads those
462
+ artifacts with `gh run download` into the work directory. It downloads nothing from Git LFS: the baselines a
463
+ capture was compared with travel inside the artifact.
464
+ - **What an accept does.** Finish writes the accepted PNGs (as LFS pointers, uploading the images
465
+ with `git lfs push`) and one review record in a throwaway worktree at the captured head,
466
+ commits, and pushes to the pull request's branch. CI then runs again on that branch.
467
+ - **Nothing restarts from scratch.** Every push captures again and compares with the baselines
468
+ the branch holds now, so after an accept the accepted items read `unchanged` and only what is
469
+ still undecided shows. Decisions you made but did not Finish are kept for every image whose
470
+ hash did not change.
471
+
472
+ ## Story parameters
473
+
474
+ Capture reads each story's `parameters.chromatic`, the same keys Chromatic reads, so stories
475
+ written for Chromatic work unchanged:
476
+
477
+ | Parameter | Effect |
478
+ | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
479
+ | `disableSnapshot: true` | The story is not captured (a baseline it still has is reported `removed`) |
480
+ | `diffThreshold` | pixelmatch's per-pixel colour threshold, 0 to 1 (default 0.063) |
481
+ | `diffIncludeAntiAliasing: true` | Count anti-aliased pixels as changes |
482
+ | `delay` | Milliseconds to wait after the render before the screenshot |
483
+ | `modes` | `{ "<name>": { <Storybook globals> } }`: one capture per mode, named `<story id>.<name>.png`; `disable: true` drops a mode |
484
+
485
+ Inside a story, `isChromatic()` from `chromatic/isChromatic` is true during capture (the URL carries
486
+ `chromatic=true`). A settings file `<baselines>/<project>/<story id>.json` overrides the story's
487
+ parameters; the page's Exclude writes one with `disableSnapshot: true` and your reason.
488
+
489
+ ## Reorganizing stories: renames
490
+
491
+ Storybook derives a story's id from its title, so moving stories in the sidebar (a new `title`,
492
+ a new folder) gives every moved story a new id. Without help, each old id's baseline is reported
493
+ `removed` and each new id is `new`, with no before and after to compare. A renames file keeps them
494
+ paired. Add it in the pull request that moves the stories, at `<baselines>/<project>/renames.json`:
495
+
496
+ ```json
497
+ [
498
+ { "from": "building-a-panel-fieldrow--default", "to": "components-panels-and-rows-fieldrow--default" },
499
+ { "from": "compact-theme-mantine-components-badge--dot", "to": "components-display-badge--dot" }
500
+ ]
501
+ ```
502
+
503
+ Each entry is one story: `from` is its old id, `to` its new one. Modes map on their own, so
504
+ `<from>.dark.png` is compared with the capture of `<to>` in the dark mode, and `<from>.light.png`
505
+ with the light one; a mode the new story no longer has is still reported `removed`.
506
+
507
+ - **Capture** compares the new id's capture with the old id's baseline. It reads `moved` when the
508
+ two look the same (only the name changed) and `changed` when they differ; either way the item
509
+ carries `from`, and the old baseline is not reported `removed`.
510
+ - **The review page** shows each one as a pair, labeled "moved from &lt;old id&gt;" on the tile
511
+ and on the story screen, with the old id's baseline on the left. The grid has a **moved** filter,
512
+ and a component's Accept N undecided takes its moved stories too.
513
+ - **Accepting** writes the baseline under the new id and deletes the old id's baseline in the same
514
+ accept commit. For a `moved` item the image is the same, so git sees a rename and the Git LFS
515
+ pointer does not change. The review record names both paths.
516
+ - **The gate** blocks a `moved` item until it is accepted, like any change: a pull request that
517
+ moves stories cannot land without their baselines moving with them.
518
+ - **A rename whose new id is not a story** in the Storybook is an error: capture reports it as a
519
+ `failed` item under the new id, with the reason, and the page lists it under Errors. Fix the
520
+ entry in `renames.json`. A rename whose old id is still a story is not a move, and does
521
+ nothing.
522
+ - **An entry whose move was accepted** does nothing any more: the old baseline is gone and the new
523
+ id has its own. You can delete the file once the pull request has merged, or leave it.
524
+
525
+ `renames.json` is read from the pull request's own checkout, like the baselines. Only baseline
526
+ PNGs move: a settings file (`<old id>.json`) is not renamed; rename it in the same pull request.
527
+
528
+ ## What the gate does and does not guarantee
529
+
530
+ - A pull request cannot pass the gate while its capture of a seeded project holds anything but
531
+ `unchanged`, `excluded` or `no baseline yet` items, including after "Re-run failed jobs" (the
532
+ highest attempt's artifact counts); a missing, unfinished or invalid capture blocks it too. A
533
+ rejected item stays blocking until a code change makes it match the baseline.
534
+ - `no baseline yet` rests on the default branch's capture being honest and recent: a story is
535
+ `new` (blocking) only when it looks different from the default branch's newest complete capture
536
+ of it. That capture may be a few merges older than the pull request's base; a story changed in
537
+ between shows as `new`, which blocks rather than passes.
538
+ - Every baseline PNG, and every settings file that excludes a story, that the pull request adds,
539
+ changes or deletes must be named with its new hash in a review record the pull request adds
540
+ under `<baselines>/reviews/`. A baseline PNG is a Git LFS pointer in git, and the gate reads the
541
+ image's hash from the pointer, so it never downloads an image. Existing records may not be
542
+ edited or deleted. This stops the shortcut of copying captured PNGs, or an exclusion, straight
543
+ into the baselines directory.
544
+ - **It does not prove a person reviewed anything.** A record is a plain JSON file: anyone who can
545
+ push to the branch can write one that names copied PNGs, and the gate cannot tell it from one
546
+ Finish wrote. Records are marked `"unproven": true` for that reason. What the gate shows is that
547
+ the captures match the pull request's baselines and that each baseline change carries a record.
548
+ - **Only the repository owner should approve.** The page runs on a development machine, where
549
+ anything running as you (an AI coding agent included) has your GitHub login and signing key and
550
+ could press Accept or call the page's API. Nothing technical prevents that today; tell your
551
+ agents not to, and keep the review to yourself.
552
+ - The projects the gate checks are the ones with baselines on the base branch, so removing a
553
+ project from the config does not remove it from the gate.
554
+ - The gate is part of a workflow file, which a pull request can edit, and a pull request can
555
+ loosen a story's own `diffThreshold` or `delay`, or a settings file's non-excluding keys,
556
+ without a review item. Read changes to those in code review.
557
+
558
+ ## Troubleshooting
559
+
560
+ - **"baseline is an LFS pointer; run git lfs pull".** The checkout was made without git-lfs, so
561
+ the baselines are small pointer files. Install git-lfs, run `git lfs install`, then
562
+ `git lfs pull`.
563
+ - **`serve` refuses to start, or Finish says git-lfs is missing or its filter is not
564
+ configured.** Install git-lfs and run `git lfs install` (it sets up the filters in your global
565
+ git configuration).
566
+ - **A push of baselines left CI unable to fetch them.** `git lfs install` adds a pre-push hook
567
+ that uploads the images a push points at. If your repository's hooks belong to husky (or any
568
+ other `core.hooksPath`), that hook is not installed: call `git lfs pre-push "$@"` from your own
569
+ pre-push hook. `git push --no-verify` skips the upload too; after one that carried baselines,
570
+ run `git lfs push origin <branch>`.
571
+ - **capture failed / no capture** on a target. The `visual` job produced no results. Open its
572
+ log from the page and re-run the job. **incomplete: N of M stories**: the job stopped part way
573
+ (a timeout); re-run it.
574
+ - **Every story is `new` on a pull request.** No reference capture of the default branch could be
575
+ downloaded (none finished yet, or its artifacts expired after 30 days). Re-run the `visual` job
576
+ once a run on the default branch has finished.
577
+ - **Finish says "capture is stale, wait for CI".** Someone pushed to the branch after the capture.
578
+ Wait for the new run, then decide again what still differs.
579
+ - **"merge master first".** The default branch has newer baselines for that project than the
580
+ pull request. Merge the default branch into the branch (by merge, never by rebase, so an accept
581
+ commit stays as it was made) and wait for CI.
582
+ - **Finish fails with "failed to write commit object"** or another signing error: unlock or plug
583
+ in your signing key, then press Finish again. Your decisions are kept.
584
+ - **"the accepts were pushed ..., but the reject comment failed".** The accepts are done; press
585
+ Finish again to post the rejects.
586
+ - **Opening the seed issue fails.** Every label in `issueLabels` must exist in the repository.
587
+ - **The pnpm setup step fails in CI.** `pnpm/action-setup` reads the pnpm version from the
588
+ `packageManager` field of your root `package.json`; add one.
589
+ - **A merge conflict under the baselines directory.** Take the default branch's side for every
590
+ file there and let CI capture again; review what still differs.
591
+ - **Captures differ from what you see locally.** Only CI's captures are compared: fonts and the
592
+ graphics stack differ from machine to machine. Look locally with `capture --stories` and
593
+ `serve --results`, but let CI's capture become the baseline.