@graphty/visual-review 0.1.3 → 0.2.1

package/README.md

@@ -43,8 +43,9 @@ your login.
 4. **Gate (CI).** The "Visual gate" job fails while a pull request holds a difference nobody
    accepted, or a baseline file changed without a review record naming it.
 
-A story with no baseline yet does not block anything until a pull request changes it, so you can
-seed baselines a few stories at a time.
+Every story needs a baseline you approved before a pull request can merge. A story with no
+baseline blocks every pull request until you accept it, either on a pull request or by seeding it
+from the default branch, so seed a project's baselines before its stories start blocking work.
 
 ## Requirements
 
@@ -146,8 +147,12 @@ Per project:
 | `seedFromDefaultBranch` | `true`     | `false`: the project's first baselines are accepted on a pull request, not seeded from the default branch                                                                                                                                      |
 | `waitFor`               | none       | After a story renders, call `method()` on every element matching `selector` and wait for the promise it returns, for a component that keeps drawing after Storybook says it is done. A console line containing `failOnConsole` fails the story |
 
-Project ids are letters, digits, `.`, `_` and `-`. The pull request gate reads the config as it is
-on the base branch, so a pull request cannot move `baselines` out from under it.
+Project ids are lowercase letters, digits and `-` (results.json allows no others), and so are
+the names of Chromatic modes, which a capture refuses before it starts. Every project is gated;
+there is no setting that turns the gate off, and a config that sets `gate` is refused. The pull
+request gate reads the config as it is on the base branch, so a pull request cannot move
+`baselines` out from under it or drop a project from the gate; a project that a pull request adds
+to its own config is gated too.
 
 ## The GitHub Actions workflows
 
@@ -180,8 +185,10 @@ Both need nothing but the default `GITHUB_TOKEN`: the capture job reads Actions
 
 ## Your first review: seeding baselines
 
-A project has no baselines until you accept some, and until it has one the gate ignores it. After
-the setup pull request merges, the default branch's push runs the capture:
+A project has no baselines until you accept some, and the gate fails closed until it has them:
+every pull request is blocked by each of its stories (`new` or `no baseline yet`), with a message
+saying to seed the project. After the setup pull request merges, the default branch's push runs
+the capture:
 
 1. Find that run's id: `gh run list --workflow visual-review.yml --branch main --limit 1`.
 2. Start the page with `--master-run <run id>` ([Opening the review page](#opening-the-review-page))
@@ -217,10 +224,10 @@ token is kept in the work directory, so the URL stays valid across restarts; del
 
 The address always names the screen you are on, after the token: the targets list; a pull
 request (or master) and project with the grid's filter and text; or one story with its view,
-zoom, changed box and blink, for example
-`#token=...&target=123&project=web&filter=undecided&item=button--primary.dark.png&view=flash&zoom=2&box=on&blink=off`.
-A link's `box` and `blink` apply to the page it opens; the choice this browser remembers for B
-and L is left as it was.
+zoom, changed box, blink and Spotlight flash, for example
+`#token=...&target=123&project=web&filter=undecided&item=button--primary.dark.png&view=flash&zoom=2&box=on&blink=off&flash=off`.
+A link's `box`, `blink` and `flash` apply to the page it opens; the choice this browser remembers
+for B, L and F in Spotlight is left as it was.
 Opening that address, in another tab or on another device, opens the same screen. **Copy link**
 at the top right copies it. The link carries your session token, so it works on your iPad the way
 the printed URL does; keep it to yourself as you would that URL. All of it sits after `#`, which a
@@ -245,6 +252,10 @@ starts the same server from your own shell.
       pull request. Merge the default branch into the pull request's branch (by merge, never
       rebase) and wait for CI.
     - **capture failed**: the `visual` job produced no results. Re-run that job in GitHub Actions.
+    - **CI still running**, **waiting for CI**, **downloading the captures**: there is nothing
+      to review yet; reload the page in a moment.
+    - **artifact expired**: GitHub deleted the capture after 30 days and it was never
+      downloaded here. Re-run the `visual` job.
     - **incomplete: N of M stories**: the capture stopped part way. Re-run the job.
     - **not seeded from master**: this project is not reviewed on the default branch
       (`"seedFromDefaultBranch": false` in the config); its first baselines are accepted on a pull
@@ -290,7 +301,9 @@ starts the same server from your own shell.
    scroll it was opened at; **Highlight**, the changed pixels in solid red laid over both images
    themselves, in both panes, where **Blink** (L) flashes the red pixels on and off at Flash's
    pace (remembered in this browser); and **Spotlight**, the new image dimmed everywhere except around the changed pixels
-   (each grown by 10 image pixels), which finds a one-pixel change. Flash, Highlight and
+   (each grown by 10 image pixels), which finds a one-pixel change, where **Spotlight flash** (F
+   in Spotlight) shows the spotlighted baseline and the spotlighted new image one after the other
+   at Flash's pace, the pane's label saying which (remembered in this browser). Flash, Highlight and
    Spotlight need two images; on a new or removed story they are off and the page says why
    ("New story, no baseline", "Only one image: this story was removed"). Badges here:
    **size changed** (in image pixels), **flaky** (the two captures differed, then matched), and
@@ -311,20 +324,22 @@ story is new or looks different from the default branch's newest capture of it),
 story no longer exists, lost a mode, or whose story's own parameters now exclude it), `unstable`
 (two captures of the same commit differed), `failed` (did not render, even after one retry).
 
-`no baseline yet` items are listed under their own filter in the grid and never need a decision:
-they do not block the pull request, Accept all skips them, and the story screen offers no buttons
-for them. Seed them from the default branch (below), or accept them on the pull request that changes them.
+`no baseline yet` items block the pull request like `new` ones: they count as needing a decision,
+Accept all includes them, and the story screen offers Accept, Reject and Exclude for them. Accepting
+one makes its capture the story's first baseline. The grid also lists them under their own filter.
+Seed them from the default branch (below), or accept them on the pull request.
 
 ## Keys
 
 | Key          | Action                                                                         |
 | ------------ | ------------------------------------------------------------------------------ |
 | J / K        | Next / previous item of this pass (decided items stay in it)                   |
-| A            | Accept an undecided item                                                       |
+| A            | Accept an undecided item, once both its images are shown                       |
 | R            | Reject an undecided item (asks for a reason, then Enter)                       |
 | E            | Exclude an undecided item (asks for a reason, then Enter, then a confirmation) |
 | U            | Undo the item's decision (on the grid: each tile's Undo button)                |
 | F            | Flash between baseline and new; F again returns to side by side                |
+| F            | In Spotlight: flash the spotlighted baseline and new, or stop flashing         |
 | H            | Highlight changed pixels; H again returns to side by side                      |
 | S            | Spotlight the changes; S again returns to side by side                         |
 | Z            | Next zoom: fit to screen, real size, 2x, 4x, 8x, then fit again                |
@@ -338,6 +353,8 @@ for them. Seed them from the default branch (below), or accept them on the pull
 
 No key reverses a decision. A, R and E do nothing on an item that is already decided, and say
 so; to change a decision, press U (or the Undo button) first. The same key twice never undoes.
+A held A, R, E or U decides once, and a double click on a decision button decides only the item
+it was clicked on, never the next one.
 
 ## What each decision does
 
@@ -354,7 +371,9 @@ so; to change a decision, press U (or the Undo button) first. The same key twice
   runner), re-run the `visual` job instead, since the newest attempt replaces the old results.
 - **Undo** (U, or a tile's Undo on the grid) clears a decision before Finish; it is the only way
   to change one. The grid also undoes a whole component or project, after a second press.
-  Decisions are kept across server restarts.
+  Decisions are kept across server restarts. A decision applies only to the image it was taken
+  on: when a new run or a re-run attempt captures that item differently, it is undecided again
+  (the old decision stays saved, and comes back if the image does).
 - After Finish, accepts and exclusions are cleared; rejects stay, marked as already posted, and
   still show as rejected on the next CI run while the capture is unchanged. Finish does not post
   them twice. They live in the work directory's `state/` (the config's `workDir`), not in the
@@ -424,15 +443,23 @@ names the captured commit, so Finish's seed branch starts from that commit.
 A story with no baseline on the default branch is in the "no baseline yet" state. On every pull
 request, CI compares its capture with the default branch's newest capture of that story:
 
-- **The pull request does not change it:** `no baseline yet` (`unseeded`). It is shown, it does
-  not block the pull request, and it is never accepted by Accept all.
+- **The pull request does not change it:** `no baseline yet` (`unseeded`).
 - **The pull request adds the story, or changes how it looks** (for example an agent fixing a
-  story you rejected): `new`. It blocks that pull request until you decide. Review it there;
-  accepting it creates its first baseline in that pull request's accept commit.
-
-So seeding never restarts from scratch: each round accepts what now looks right, and the rest
-waits, blocking nothing, until a pull request touches it. A project enters the merge gate when its
-first baseline lands on the default branch; before that the gate ignores it entirely.
+  story you rejected): `new`.
+
+Both block the pull request until you decide. Review them there; accepting one creates its first
+baseline in that pull request's accept commit. Seeding never restarts from scratch: each round
+accepts what now looks right, and every story still without a baseline keeps blocking pull
+requests until it is accepted or seeded.
+
+The same holds for a project with no baselines at all: the gate fails closed. Every project in the
+config is gated from the start, so every pull request is blocked by the stories of an unseeded
+project, and the gate's message says how to unblock it: seed the project (capture a known-good
+commit with `visual-seed.yml`, or take the default branch's newest run, review it with
+`serve --master-run <run id>`, merge the seed pull request, then merge the default branch into the
+blocked one), or accept the items on that pull request. A project with
+`"seedFromDefaultBranch": false` is told to accept them on the pull request. Seed only from a commit
+whose images a person already reviewed.
 
 If the default branch's capture could not be downloaded (its artifacts expired, or no run there has
 finished one), every story without a baseline is `new` on that pull request. Re-run its `visual` job
@@ -543,14 +570,13 @@ PNGs move: a settings file (`<old id>.json`) is not renamed; rename it in the sa
 
 ## What the gate does and does not guarantee
 
-- A pull request cannot pass the gate while its capture of a seeded project holds anything but
-  `unchanged`, `excluded` or `no baseline yet` items, including after "Re-run failed jobs" (the
+- A pull request cannot pass the gate while its capture of a gated project holds anything but
+  `unchanged` or `excluded` items, including after "Re-run failed jobs" (the
   highest attempt's artifact counts); a missing, unfinished or invalid capture blocks it too. A
   rejected item stays blocking until a code change makes it match the baseline.
-- `no baseline yet` rests on the default branch's capture being honest and recent: a story is
-  `new` (blocking) only when it looks different from the default branch's newest complete capture
-  of it. That capture may be a few merges older than the pull request's base; a story changed in
-  between shows as `new`, which blocks rather than passes.
+- A story with no baseline always blocks. `new` and `no baseline yet` only tell the reviewer
+  whether the pull request changed it, measured against the default branch's newest complete
+  capture, which may be a few merges older than the pull request's base.
 - Every baseline PNG, and every settings file that excludes a story, that the pull request adds,
   changes or deletes must be named with its new hash in a review record the pull request adds
   under `<baselines>/reviews/`. A baseline PNG is a Git LFS pointer in git, and the gate reads the
@@ -565,8 +591,9 @@ PNGs move: a settings file (`<old id>.json`) is not renamed; rename it in the sa
   anything running as you (an AI coding agent included) has your GitHub login and signing key and
   could press Accept or call the page's API. Nothing technical prevents that today; tell your
   agents not to, and keep the review to yourself.
-- The projects the gate checks are the ones with baselines on the base branch, so removing a
-  project from the config does not remove it from the gate.
+- The projects the gate checks are every project in the base branch's config and in the pull
+  request's config, seeded or not, plus every project with baselines on the base branch. So
+  removing a project from the config does not remove it from the gate.
 - The gate is part of a workflow file, which a pull request can edit, and a pull request can
   loosen a story's own `diffThreshold` or `delay`, or a settings file's non-excluding keys,
   without a review item. Read changes to those in code review.
@@ -584,6 +611,21 @@ PNGs move: a settings file (`<old id>.json`) is not renamed; rename it in the sa
   other `core.hooksPath`), that hook is not installed: call `git lfs pre-push "$@"` from your own
   pre-push hook. `git push --no-verify` skips the upload too; after one that carried baselines,
   run `git lfs push origin <branch>`.
+- **download failed / failed to load: ...; reload the page to retry.** `serve` starts
+  downloading every capture as soon as it starts, and retries a gh call that fails on the network
+  (DNS, a dropped connection, a GitHub 5xx) three times over about 20 seconds; it logs each failed
+  call and each retry to stderr. A project whose download still fails shows "download failed", a
+  pull request (or the default branch's run) GitHub would not answer for shows "failed to load",
+  or, when it loaded before, keeps what it showed with "could not refresh", and everything else
+  loads as usual. Reload the page to try again; captures already downloaded are kept, and a
+  damaged one is downloaded again.
+- **A gh or git call hangs.** Every gh and git call `serve` and Finish make is stopped after 10
+  minutes (`VISUAL_REVIEW_TIMEOUT_MS` sets another limit, in milliseconds), and git never waits
+  for a credential prompt. A stopped Finish names the step it was on and keeps your decisions.
+- **"the server stopped while this Finish was at ..."** The server restarted during a Finish.
+  Look at the branch on origin to see whether its commit was pushed before pressing Finish again.
+- **"... was pushed as ..., but opening its pull request failed"** (the seed). Press Finish
+  again: it opens the pull request for the branch already pushed.
 - **capture failed / no capture** on a target. The `visual` job produced no results. Open its
   log from the page and re-run the job. **incomplete: N of M stories**: the job stopped part way
   (a timeout); re-run it.

package/capture/capture.mjs

@@ -154,6 +154,13 @@ export function storySettings(parameters, file) {
             name,
             globals: Object.fromEntries(Object.entries(m).filter(([k]) => k !== "disable")),
         }));
+    // A mode names files and results.json items, which allow only these: refused before any story
+    // is captured, not when results.json is checked at the end.
+    for (const { name } of modes) {
+        if (!/^[a-z0-9][a-z0-9-]*$/.test(name) || name.length > 50) {
+            throw new Error(`Chromatic mode "${name}": a mode name is lowercase letters, digits and "-", at most 50`);
+        }
+    }
     const where = byFile ? "settings file" : "story's parameters";
     return {
         disableSnapshot,
@@ -506,7 +513,10 @@ async function loadReference(dir) {
     if (!results || validateResults(results).length > 0 || !results.complete) {
         return { images, runId: null };
     }
-    for (const item of results.items.filter((i) => i.status === "new")) {
+    // Both statuses mean "captured with no baseline, hash recorded". Reading only `new` made a story
+    // alternate: a run that matched its reference says `unseeded`, so the next run found no image
+    // for it and said `new` again.
+    for (const item of results.items.filter((i) => i.status === "new" || i.status === "unseeded")) {
         const bytes = await readFile(join(dir, item.file)).catch(() => null);
         if (bytes && sha256(bytes) === item.capture) {
             images.set(item.file, bytes);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@graphty/visual-review",
-  "version": "0.1.3",
+  "version": "0.2.1",
   "description": "Self-hosted visual review of Storybook stories: capture in GitHub Actions, keep the baselines in git (Git LFS), accept or reject each change in a local page, and gate pull requests",
   "author": "Adam Powers <apowers@ato.ms>",
   "type": "module",

package/templates/visual-review.yml

@@ -79,7 +79,7 @@ jobs:
               run: ${{ matrix.build }}
 
             # A story with no baseline that looks as it does in the default branch's newest capture is
-            # "unseeded" and does not block this pull request; without a reference every one is "new".
+            # "unseeded"; without a reference every one is "new". Both block until accepted or seeded.
             - name: Download the default branch's newest capture
               id: reference
               if: github.event_name == 'pull_request'
@@ -115,8 +115,9 @@ jobs:
                       fi
                   } >> "$GITHUB_STEP_SUMMARY"
 
-    # Fails while a project with baselines on the base branch holds anything but unchanged, excluded
-    # or unseeded items in its newest capture, or a baseline file changed without a review record.
+    # Fails while a project of the base branch's or the pull request's config, or with baselines on
+    # the base branch (seeded or not), holds anything but unchanged or excluded items in its newest
+    # capture, or a baseline file changed without a review record.
     # It runs the published gate at a pinned version, so a pull request's own dependencies cannot
     # change it. Depth 2: HEAD^1, the merge commit's first parent, is the base branch tip.
     visual-gate:

package/trusted/cli.mjs

@@ -214,6 +214,7 @@ async function serve(args) {
         origin,
         masterRun,
         results: values.results && resolve(values.results),
+        warm: true,
         // What the owner types to run this server from their own shell, so Finish signs with
         // their key rather than the environment of whoever started it (an agent, say).
         startCommand:

package/trusted/gate.mjs

@@ -1,20 +1,21 @@
 #!/usr/bin/env node
 /**
- * The pull request gate: fails while a project that has baselines on the base branch holds visual
- * changes the owner has not reviewed.
+ * The pull request gate: fails while a project holds visual changes the owner has not reviewed.
  *
- * Seeding is per story, so a seeded project can hold stories with no baseline yet. Those that the
- * pull request did not change are `unseeded` (capture compared them with master's newest capture)
- * and pass; a story the pull request adds or changes is `new` and blocks until the owner accepts
- * it there, which creates its first baseline.
+ * It fails closed: nothing merges with an image nobody approved. Every project with baselines on the
+ * base branch, and every project in the base branch's config or the pull request's, is gated, seeded
+ * or not. Every story needs an approved baseline. A story with none blocks: `new` when the pull
+ * request adds or changes it, `unseeded` when it looks as in master's newest capture. Either way
+ * the owner accepts it (on the pull request, or by seeding it from the default branch), which
+ * creates its first baseline.
  *
  * <dir> holds the downloaded `visual-<project>-<attempt>` artifacts of this CI run, every attempt
  * of it. For each project only the highest attempt counts, so re-running failed jobs (which
  * leaves the visual jobs' old attempt as the newest) can neither hide nor resurrect a capture.
  * Which projects exist and are seeded is read from <ref> (the base branch tip, fetched by the
- * caller), not from the pull request, and so is visual-review.config.json (where the baselines
- * live), so neither deleting a project's baselines nor moving the baselines directory in the pull
- * request turns the gate off. A seeded project with no results.json, or an incomplete one, fails: a capture that
+ * caller), and so is visual-review.config.json (the projects and where the baselines live), so
+ * neither deleting a project's baselines, nor removing it from the config, nor moving the baselines
+ * directory in the pull request turns the gate off. The pull request's config can only add projects. A gated project with no results.json, or an incomplete one, fails: a capture that
  * crashed has shown the owner nothing. An invalid results.json counts as missing.
  *
  * It also fails when a baseline PNG, or a settings file that excludes a story, differs from the
@@ -40,7 +41,7 @@ import { parseArgs } from "node:util";
 import { loadConfigAt, repoRoot } from "./lib/config.mjs";
 import { validateResults } from "./lib/results.mjs";
 
-const PASSING = new Set(["unchanged", "excluded", "unseeded"]);
+const PASSING = new Set(["unchanged", "excluded"]);
 
 /**
  * The newest attempt's results.json of every project in a directory of downloaded artifacts.
@@ -62,24 +63,41 @@ export function newestResults(dir) {
             continue;
         }
         const file = join(dir, name, "results.json");
-        out[project] = { attempt, results: existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : null };
+        let results = null;
+        try {
+            results = JSON.parse(readFileSync(file, "utf8"));
+        } catch {
+            // Missing or not JSON: counted as missing, like any invalid results.json.
+        }
+        out[project] = { attempt, results };
     }
     return out;
 }
 
+/**
+ * The projects the gate checks: every project with baselines at the base, and every project of the
+ * base's config and of the pull request's.
+ * @param {{ projects: Record<string, object> }} config the base branch's config
+ * @param {Set<string>} seeded the projects with baselines at the base
+ * @param {{ projects: Record<string, object> }} [headConfig] the pull request's config
+ * @returns {string[]} the gated project ids
+ */
+export function gatedProjects(config, seeded, headConfig) {
+    return [...new Set([...seeded, ...Object.keys(config.projects), ...Object.keys(headConfig?.projects ?? {})])];
+}
+
 /**
  * What blocks the pull request.
- * @param {{ projects: string[], seeded: Set<string>, captures: Record<string, { attempt: number,
- *     results: object | null }> }} input every captured project, those with baselines on the base
- *     branch, and the newest capture of each
+ * @param {{ config: { defaultBranch: string, projects: Record<string, { seedFromDefaultBranch: boolean }> },
+ *     headConfig: { projects: Record<string, object> } | undefined, seeded: Set<string>,
+ *     captures: Record<string, { attempt: number, results: object | null }> }} input the base
+ *     branch's config, the pull request's config (if any), the projects with baselines on the base
+ *     branch, and the newest capture of each project
  * @returns {string[]} one line per blocked project; empty when the gate passes
  */
-export function gateProblems({ projects, seeded, captures }) {
+export function gateProblems({ config, headConfig, seeded, captures }) {
     const problems = [];
-    for (const p of projects) {
-        if (!seeded.has(p)) {
-            continue;
-        }
+    for (const p of gatedProjects(config, seeded, headConfig)) {
         const r = captures[p]?.results;
         if (!r) {
             problems.push(`${p}: no capture results (the visual job failed or uploaded nothing); re-run it`);
@@ -101,15 +119,36 @@ export function gateProblems({ projects, seeded, captures }) {
             for (const s of open) {
                 counts.set(s, (counts.get(s) ?? 0) + 1);
             }
-            problems.push(
-                `${p}: ${[...counts].map(([s, n]) => `${n} ${s}`).join(", ")} ` +
-                    "(not accepted; a rejected item needs a code change, not another review)",
-            );
+            const what = [...counts].map(([s, n]) => `${n} ${s}`).join(", ");
+            problems.push(`${p}: ${what} ${seeded.has(p) ? NOT_ACCEPTED : notSeeded(config, p)}`);
         }
     }
     return problems;
 }
 
+const NOT_ACCEPTED = "(not accepted; a rejected item needs a code change, not another review)";
+
+/**
+ * Why an unseeded project blocks, and how to seed it.
+ * @param {{ defaultBranch: string, projects: Record<string, { seedFromDefaultBranch: boolean }> }} config the
+ *     base branch's config
+ * @param {string} p the project
+ * @returns {string} the rest of the gate's line
+ */
+function notSeeded(config, p) {
+    const branch = config.defaultBranch;
+    const here = "accept them on this pull request with `visual-review serve`";
+    if (config.projects[p]?.seedFromDefaultBranch === false) {
+        return `(not accepted; ${p} has no baselines on ${branch} yet, so ${here} to create its first ones)`;
+    }
+    return (
+        `(not accepted; ${p} has no baselines on ${branch} yet. Seed it: capture a known-good commit with ` +
+        `\`gh workflow run visual-seed.yml --ref ${branch} -f ref=<sha>\` (or take ${branch}'s newest run), ` +
+        `review that run with \`visual-review serve --master-run <run id>\` and merge the seed pull ` +
+        `request, then merge ${branch} into this branch; or ${here})`
+    );
+}
+
 const gitOut = (cwd, args) => execFileSync("git", args, { cwd, maxBuffer: 1 << 28 });
 
 /**
@@ -189,8 +228,7 @@ function parseOr(bytes) {
 
 /**
  * The projects with at least one baseline PNG at a git ref: the directories under the baselines
- * directory at the base tip, so a pull request cannot drop a project from the gate by editing
- * visual-review.config.json.
+ * directory at the base tip. They are gated whatever the config says.
  * @param {string} ref the base branch tip
  * @param {string} [cwd] the repository
  * @param {string} [baselines] the baselines directory
@@ -245,17 +283,14 @@ export function runGate(args) {
         return 2;
     }
     const root = repoRoot();
-    const { baselines } = loadConfigAt(values.base, root);
-    const seeded = seededAt(values.base, root, baselines);
-    const problems = gateProblems({
-        projects: [...seeded],
-        seeded,
-        captures: newestResults(values.captures),
-    });
+    const config = loadConfigAt(values.base, root);
+    const seeded = seededAt(values.base, root, config.baselines);
+    const headConfig = loadConfigAt(values.head, root);
+    const problems = gateProblems({ config, headConfig, seeded, captures: newestResults(values.captures) });
     for (const line of problems) {
         console.log(`::error::visual changes not accepted -- ${line}`);
     }
-    const unrecorded = unrecordedChanges(values.base, values.head, root, baselines);
+    const unrecorded = unrecordedChanges(values.base, values.head, root, config.baselines);
     for (const line of unrecorded) {
         console.log(`::error::baseline without a review -- ${line}`);
     }