@graphty/visual-review 0.1.3 → 0.2.0

package/README.md

@@ -43,8 +43,9 @@ your login.
 4. **Gate (CI).** The "Visual gate" job fails while a pull request holds a difference nobody
    accepted, or a baseline file changed without a review record naming it.
 
-A story with no baseline yet does not block anything until a pull request changes it, so you can
-seed baselines a few stories at a time.
+Every story needs a baseline you approved before a pull request can merge. A story with no
+baseline blocks every pull request until you accept it, either on a pull request or by seeding it
+from the default branch, so seed a project's baselines before its stories start blocking work.
 
 ## Requirements
 
@@ -146,8 +147,10 @@ Per project:
 | `seedFromDefaultBranch` | `true`     | `false`: the project's first baselines are accepted on a pull request, not seeded from the default branch                                                                                                                                      |
 | `waitFor`               | none       | After a story renders, call `method()` on every element matching `selector` and wait for the promise it returns, for a component that keeps drawing after Storybook says it is done. A console line containing `failOnConsole` fails the story |
 
-Project ids are letters, digits, `.`, `_` and `-`. The pull request gate reads the config as it is
-on the base branch, so a pull request cannot move `baselines` out from under it.
+Project ids are letters, digits, `.`, `_` and `-`. Every project is gated; there is no setting
+that turns the gate off, and a config that sets `gate` is refused. The pull request gate reads the
+config as it is on the base branch, so a pull request cannot move `baselines` out from under it or
+drop a project from the gate; a project that a pull request adds to its own config is gated too.
 
 ## The GitHub Actions workflows
 
@@ -180,8 +183,10 @@ Both need nothing but the default `GITHUB_TOKEN`: the capture job reads Actions
 
 ## Your first review: seeding baselines
 
-A project has no baselines until you accept some, and until it has one the gate ignores it. After
-the setup pull request merges, the default branch's push runs the capture:
+A project has no baselines until you accept some, and the gate fails closed until it has them:
+every pull request is blocked by each of its stories (`new` or `no baseline yet`), with a message
+saying to seed the project. After the setup pull request merges, the default branch's push runs
+the capture:
 
 1. Find that run's id: `gh run list --workflow visual-review.yml --branch main --limit 1`.
 2. Start the page with `--master-run <run id>` ([Opening the review page](#opening-the-review-page))
@@ -311,9 +316,10 @@ story is new or looks different from the default branch's newest capture of it),
 story no longer exists, lost a mode, or whose story's own parameters now exclude it), `unstable`
 (two captures of the same commit differed), `failed` (did not render, even after one retry).
 
-`no baseline yet` items are listed under their own filter in the grid and never need a decision:
-they do not block the pull request, Accept all skips them, and the story screen offers no buttons
-for them. Seed them from the default branch (below), or accept them on the pull request that changes them.
+`no baseline yet` items block the pull request like `new` ones: they count as needing a decision,
+Accept all includes them, and the story screen offers Accept, Reject and Exclude for them. Accepting
+one makes its capture the story's first baseline. The grid also lists them under their own filter.
+Seed them from the default branch (below), or accept them on the pull request.
 
 ## Keys
 
@@ -424,15 +430,23 @@ names the captured commit, so Finish's seed branch starts from that commit.
 A story with no baseline on the default branch is in the "no baseline yet" state. On every pull
 request, CI compares its capture with the default branch's newest capture of that story:
 
-- **The pull request does not change it:** `no baseline yet` (`unseeded`). It is shown, it does
-  not block the pull request, and it is never accepted by Accept all.
+- **The pull request does not change it:** `no baseline yet` (`unseeded`).
 - **The pull request adds the story, or changes how it looks** (for example an agent fixing a
-  story you rejected): `new`. It blocks that pull request until you decide. Review it there;
-  accepting it creates its first baseline in that pull request's accept commit.
-
-So seeding never restarts from scratch: each round accepts what now looks right, and the rest
-waits, blocking nothing, until a pull request touches it. A project enters the merge gate when its
-first baseline lands on the default branch; before that the gate ignores it entirely.
+  story you rejected): `new`.
+
+Both block the pull request until you decide. Review them there; accepting one creates its first
+baseline in that pull request's accept commit. Seeding never restarts from scratch: each round
+accepts what now looks right, and every story still without a baseline keeps blocking pull
+requests until it is accepted or seeded.
+
+The same holds for a project with no baselines at all: the gate fails closed. Every project in the
+config is gated from the start, so every pull request is blocked by the stories of an unseeded
+project, and the gate's message says how to unblock it: seed the project (capture a known-good
+commit with `visual-seed.yml`, or take the default branch's newest run, review it with
+`serve --master-run <run id>`, merge the seed pull request, then merge the default branch into the
+blocked one), or accept the items on that pull request. A project with
+`"seedFromDefaultBranch": false` is told to accept them on the pull request. Seed only from a commit
+whose images a person already reviewed.
 
 If the default branch's capture could not be downloaded (its artifacts expired, or no run there has
 finished one), every story without a baseline is `new` on that pull request. Re-run its `visual` job
@@ -543,14 +557,13 @@ PNGs move: a settings file (`<old id>.json`) is not renamed; rename it in the sa
 
 ## What the gate does and does not guarantee
 
-- A pull request cannot pass the gate while its capture of a seeded project holds anything but
-  `unchanged`, `excluded` or `no baseline yet` items, including after "Re-run failed jobs" (the
+- A pull request cannot pass the gate while its capture of a gated project holds anything but
+  `unchanged` or `excluded` items, including after "Re-run failed jobs" (the
   highest attempt's artifact counts); a missing, unfinished or invalid capture blocks it too. A
   rejected item stays blocking until a code change makes it match the baseline.
-- `no baseline yet` rests on the default branch's capture being honest and recent: a story is
-  `new` (blocking) only when it looks different from the default branch's newest complete capture
-  of it. That capture may be a few merges older than the pull request's base; a story changed in
-  between shows as `new`, which blocks rather than passes.
+- A story with no baseline always blocks. `new` and `no baseline yet` only tell the reviewer
+  whether the pull request changed it, measured against the default branch's newest complete
+  capture, which may be a few merges older than the pull request's base.
 - Every baseline PNG, and every settings file that excludes a story, that the pull request adds,
   changes or deletes must be named with its new hash in a review record the pull request adds
   under `<baselines>/reviews/`. A baseline PNG is a Git LFS pointer in git, and the gate reads the
@@ -565,8 +578,9 @@ PNGs move: a settings file (`<old id>.json`) is not renamed; rename it in the sa
   anything running as you (an AI coding agent included) has your GitHub login and signing key and
   could press Accept or call the page's API. Nothing technical prevents that today; tell your
   agents not to, and keep the review to yourself.
-- The projects the gate checks are the ones with baselines on the base branch, so removing a
-  project from the config does not remove it from the gate.
+- The projects the gate checks are every project in the base branch's config and in the pull
+  request's config, seeded or not, plus every project with baselines on the base branch. So
+  removing a project from the config does not remove it from the gate.
 - The gate is part of a workflow file, which a pull request can edit, and a pull request can
   loosen a story's own `diffThreshold` or `delay`, or a settings file's non-excluding keys,
   without a review item. Read changes to those in code review.

package/capture/capture.mjs

@@ -506,7 +506,10 @@ async function loadReference(dir) {
     if (!results || validateResults(results).length > 0 || !results.complete) {
         return { images, runId: null };
     }
-    for (const item of results.items.filter((i) => i.status === "new")) {
+    // Both statuses mean "captured with no baseline, hash recorded". Reading only `new` made a story
+    // alternate: a run that matched its reference says `unseeded`, so the next run found no image
+    // for it and said `new` again.
+    for (const item of results.items.filter((i) => i.status === "new" || i.status === "unseeded")) {
         const bytes = await readFile(join(dir, item.file)).catch(() => null);
         if (bytes && sha256(bytes) === item.capture) {
             images.set(item.file, bytes);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@graphty/visual-review",
-  "version": "0.1.3",
+  "version": "0.2.0",
   "description": "Self-hosted visual review of Storybook stories: capture in GitHub Actions, keep the baselines in git (Git LFS), accept or reject each change in a local page, and gate pull requests",
   "author": "Adam Powers <apowers@ato.ms>",
   "type": "module",

package/templates/visual-review.yml

@@ -79,7 +79,7 @@ jobs:
               run: ${{ matrix.build }}
 
             # A story with no baseline that looks as it does in the default branch's newest capture is
-            # "unseeded" and does not block this pull request; without a reference every one is "new".
+            # "unseeded"; without a reference every one is "new". Both block until accepted or seeded.
             - name: Download the default branch's newest capture
               id: reference
               if: github.event_name == 'pull_request'
@@ -115,8 +115,9 @@ jobs:
                       fi
                   } >> "$GITHUB_STEP_SUMMARY"
 
-    # Fails while a project with baselines on the base branch holds anything but unchanged, excluded
-    # or unseeded items in its newest capture, or a baseline file changed without a review record.
+    # Fails while a project of the base branch's or the pull request's config, or with baselines on
+    # the base branch (seeded or not), holds anything but unchanged or excluded items in its newest
+    # capture, or a baseline file changed without a review record.
     # It runs the published gate at a pinned version, so a pull request's own dependencies cannot
     # change it. Depth 2: HEAD^1, the merge commit's first parent, is the base branch tip.
     visual-gate:

package/trusted/gate.mjs

@@ -1,20 +1,21 @@
 #!/usr/bin/env node
 /**
- * The pull request gate: fails while a project that has baselines on the base branch holds visual
- * changes the owner has not reviewed.
+ * The pull request gate: fails while a project holds visual changes the owner has not reviewed.
  *
- * Seeding is per story, so a seeded project can hold stories with no baseline yet. Those that the
- * pull request did not change are `unseeded` (capture compared them with master's newest capture)
- * and pass; a story the pull request adds or changes is `new` and blocks until the owner accepts
- * it there, which creates its first baseline.
+ * It fails closed: nothing merges with an image nobody approved. Every project with baselines on the
+ * base branch, and every project in the base branch's config or the pull request's, is gated, seeded
+ * or not. Every story needs an approved baseline. A story with none blocks: `new` when the pull
+ * request adds or changes it, `unseeded` when it looks as in master's newest capture. Either way
+ * the owner accepts it (on the pull request, or by seeding it from the default branch), which
+ * creates its first baseline.
  *
  * <dir> holds the downloaded `visual-<project>-<attempt>` artifacts of this CI run, every attempt
  * of it. For each project only the highest attempt counts, so re-running failed jobs (which
  * leaves the visual jobs' old attempt as the newest) can neither hide nor resurrect a capture.
  * Which projects exist and are seeded is read from <ref> (the base branch tip, fetched by the
- * caller), not from the pull request, and so is visual-review.config.json (where the baselines
- * live), so neither deleting a project's baselines nor moving the baselines directory in the pull
- * request turns the gate off. A seeded project with no results.json, or an incomplete one, fails: a capture that
+ * caller), and so is visual-review.config.json (the projects and where the baselines live), so
+ * neither deleting a project's baselines, nor removing it from the config, nor moving the baselines
+ * directory in the pull request turns the gate off. The pull request's config can only add projects. A gated project with no results.json, or an incomplete one, fails: a capture that
  * crashed has shown the owner nothing. An invalid results.json counts as missing.
  *
  * It also fails when a baseline PNG, or a settings file that excludes a story, differs from the
@@ -40,7 +41,7 @@ import { parseArgs } from "node:util";
 import { loadConfigAt, repoRoot } from "./lib/config.mjs";
 import { validateResults } from "./lib/results.mjs";
 
-const PASSING = new Set(["unchanged", "excluded", "unseeded"]);
+const PASSING = new Set(["unchanged", "excluded"]);
 
 /**
  * The newest attempt's results.json of every project in a directory of downloaded artifacts.
@@ -67,19 +68,30 @@ export function newestResults(dir) {
     return out;
 }
 
+/**
+ * The projects the gate checks: every project with baselines at the base, and every project of the
+ * base's config and of the pull request's.
+ * @param {{ projects: Record<string, object> }} config the base branch's config
+ * @param {Set<string>} seeded the projects with baselines at the base
+ * @param {{ projects: Record<string, object> }} [headConfig] the pull request's config
+ * @returns {string[]} the gated project ids
+ */
+export function gatedProjects(config, seeded, headConfig) {
+    return [...new Set([...seeded, ...Object.keys(config.projects), ...Object.keys(headConfig?.projects ?? {})])];
+}
+
 /**
  * What blocks the pull request.
- * @param {{ projects: string[], seeded: Set<string>, captures: Record<string, { attempt: number,
- *     results: object | null }> }} input every captured project, those with baselines on the base
- *     branch, and the newest capture of each
+ * @param {{ config: { defaultBranch: string, projects: Record<string, { seedFromDefaultBranch: boolean }> },
+ *     headConfig: { projects: Record<string, object> } | undefined, seeded: Set<string>,
+ *     captures: Record<string, { attempt: number, results: object | null }> }} input the base
+ *     branch's config, the pull request's config (if any), the projects with baselines on the base
+ *     branch, and the newest capture of each project
  * @returns {string[]} one line per blocked project; empty when the gate passes
  */
-export function gateProblems({ projects, seeded, captures }) {
+export function gateProblems({ config, headConfig, seeded, captures }) {
     const problems = [];
-    for (const p of projects) {
-        if (!seeded.has(p)) {
-            continue;
-        }
+    for (const p of gatedProjects(config, seeded, headConfig)) {
         const r = captures[p]?.results;
         if (!r) {
             problems.push(`${p}: no capture results (the visual job failed or uploaded nothing); re-run it`);
@@ -101,15 +113,36 @@ export function gateProblems({ projects, seeded, captures }) {
             for (const s of open) {
                 counts.set(s, (counts.get(s) ?? 0) + 1);
             }
-            problems.push(
-                `${p}: ${[...counts].map(([s, n]) => `${n} ${s}`).join(", ")} ` +
-                    "(not accepted; a rejected item needs a code change, not another review)",
-            );
+            const what = [...counts].map(([s, n]) => `${n} ${s}`).join(", ");
+            problems.push(`${p}: ${what} ${seeded.has(p) ? NOT_ACCEPTED : notSeeded(config, p)}`);
         }
     }
     return problems;
 }
 
+const NOT_ACCEPTED = "(not accepted; a rejected item needs a code change, not another review)";
+
+/**
+ * Why an unseeded project blocks, and how to seed it.
+ * @param {{ defaultBranch: string, projects: Record<string, { seedFromDefaultBranch: boolean }> }} config the
+ *     base branch's config
+ * @param {string} p the project
+ * @returns {string} the rest of the gate's line
+ */
+function notSeeded(config, p) {
+    const branch = config.defaultBranch;
+    const here = "accept them on this pull request with `visual-review serve`";
+    if (config.projects[p]?.seedFromDefaultBranch === false) {
+        return `(not accepted; ${p} has no baselines on ${branch} yet, so ${here} to create its first ones)`;
+    }
+    return (
+        `(not accepted; ${p} has no baselines on ${branch} yet. Seed it: capture a known-good commit with ` +
+        `\`gh workflow run visual-seed.yml --ref ${branch} -f ref=<sha>\` (or take ${branch}'s newest run), ` +
+        `review that run with \`visual-review serve --master-run <run id>\` and merge the seed pull ` +
+        `request, then merge ${branch} into this branch; or ${here})`
+    );
+}
+
 const gitOut = (cwd, args) => execFileSync("git", args, { cwd, maxBuffer: 1 << 28 });
 
 /**
@@ -189,8 +222,7 @@ function parseOr(bytes) {
 
 /**
  * The projects with at least one baseline PNG at a git ref: the directories under the baselines
- * directory at the base tip, so a pull request cannot drop a project from the gate by editing
- * visual-review.config.json.
+ * directory at the base tip. They are gated whatever the config says.
  * @param {string} ref the base branch tip
  * @param {string} [cwd] the repository
  * @param {string} [baselines] the baselines directory
@@ -245,17 +277,14 @@ export function runGate(args) {
         return 2;
     }
     const root = repoRoot();
-    const { baselines } = loadConfigAt(values.base, root);
-    const seeded = seededAt(values.base, root, baselines);
-    const problems = gateProblems({
-        projects: [...seeded],
-        seeded,
-        captures: newestResults(values.captures),
-    });
+    const config = loadConfigAt(values.base, root);
+    const seeded = seededAt(values.base, root, config.baselines);
+    const headConfig = loadConfigAt(values.head, root);
+    const problems = gateProblems({ config, headConfig, seeded, captures: newestResults(values.captures) });
     for (const line of problems) {
         console.log(`::error::visual changes not accepted -- ${line}`);
     }
-    const unrecorded = unrecordedChanges(values.base, values.head, root, baselines);
+    const unrecorded = unrecordedChanges(values.base, values.head, root, config.baselines);
     for (const line of unrecorded) {
         console.log(`::error::baseline without a review -- ${line}`);
     }

package/trusted/lib/accept.mjs

@@ -26,7 +26,7 @@ import { isLfsPointer } from "./compare.mjs";
 const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex");
 
 /** The statuses an item can be accepted or rejected in; unstable and failed are only excluded. */
-const DECIDABLE = new Set(["changed", "moved", "new", "removed"]);
+const DECIDABLE = new Set(["changed", "moved", "new", "unseeded", "removed"]);
 const EXCLUDABLE = new Set([...DECIDABLE, "unstable", "failed"]);
 
 /**

package/trusted/lib/config.mjs

@@ -90,6 +90,9 @@ export function normalizeConfig(input) {
         if (p.seedFromDefaultBranch !== undefined && typeof p.seedFromDefaultBranch !== "boolean") {
             fail(`${where}.seedFromDefaultBranch must be true or false`);
         }
+        if (p.gate !== undefined) {
+            fail(`${where}.gate is not a setting: every project is gated, and every story needs an approved baseline`);
+        }
         let waitFor = null;
         if (p.waitFor !== undefined && p.waitFor !== null) {
             const w = p.waitFor;

package/trusted/lib/serve.mjs

@@ -44,7 +44,7 @@ const HEADERS = {
  */
 const componentOf = (id) => id.split("--")[0];
 
-const REVIEWABLE = new Set(["changed", "moved", "new", "removed", "unstable", "failed"]);
+const REVIEWABLE = new Set(["changed", "moved", "new", "unseeded", "removed", "unstable", "failed"]);
 const WRITES = new Set(["decide", "accept-all", "finish"]);
 
 /**

package/trusted/page/review.js

@@ -11,9 +11,10 @@ const app = document.getElementById("app");
 const crumbs = document.getElementById("crumbs");
 const statusLine = document.getElementById("status");
 
-const REVIEWABLE = ["changed", "moved", "new", "removed", "unstable", "failed"];
-const ACCEPTABLE = ["changed", "moved", "new", "removed"];
-// A story with no baseline that this pull request did not change: shown, never a decision here.
+const REVIEWABLE = ["changed", "moved", "new", "unseeded", "removed", "unstable", "failed"];
+const ACCEPTABLE = ["changed", "moved", "new", "unseeded", "removed"];
+// A story with no baseline that this pull request did not change. It blocks like `new`: accepting it
+// creates its first baseline.
 const UNSEEDED = "unseeded";
 const NO_BASELINE = "no baseline yet";
 const statusLabel = (status) => (status === UNSEEDED ? NO_BASELINE : status);
@@ -58,7 +59,7 @@ const state = {
 };
 const running = () => state.job?.running === true;
 const VIEWS = ["side", "flash", "highlight", "spotlight"];
-const FILTERS = ["undecided", "all", ...REVIEWABLE, UNSEEDED, ...Object.keys(DECISIONS)];
+const FILTERS = ["undecided", "all", ...REVIEWABLE, ...Object.keys(DECISIONS)];
 let routing = false; // true while the page follows the address (a link opened, Back, Forward)
 const images = new Map();
 const diffs = new Map();
@@ -182,18 +183,13 @@ function ordered(items) {
 
 function visibleItems() {
     const text = state.text.trim().toLowerCase();
-    let items;
-    if (state.filter === UNSEEDED) {
-        items = state.data.items.filter((i) => i.status === UNSEEDED);
-    } else {
-        items = state.data.items.filter((i) => REVIEWABLE.includes(i.status));
-        if (state.filter === "undecided") {
-            items = items.filter((i) => !decisionOf(i));
-        } else if (Object.hasOwn(DECISIONS, state.filter)) {
-            items = items.filter((i) => decisionOf(i)?.decision === state.filter);
-        } else if (state.filter !== "all") {
-            items = items.filter((i) => i.status === state.filter);
-        }
+    let items = state.data.items.filter((i) => REVIEWABLE.includes(i.status));
+    if (state.filter === "undecided") {
+        items = items.filter((i) => !decisionOf(i));
+    } else if (Object.hasOwn(DECISIONS, state.filter)) {
+        items = items.filter((i) => decisionOf(i)?.decision === state.filter);
+    } else if (state.filter !== "all") {
+        items = items.filter((i) => i.status === state.filter);
     }
     return ordered(text ? items.filter((i) => i.file.includes(text)) : items);
 }
@@ -384,7 +380,7 @@ function targetCard(t) {
                     el(
                         "td",
                         {},
-                        p.reviewable > 0 || p.counts[UNSEEDED]
+                        p.reviewable > 0
                             ? el("button", { type: "button", onclick: () => openProject(t, p.project) }, "Review")
                             : null,
                     ),
@@ -864,7 +860,7 @@ function showStory() {
     const view = note ? "side" : state.view;
     const onlyExclude = item.status === "unstable" || item.status === "failed";
     const unseeded = item.status === UNSEEDED;
-    const decidable = !isLocal() && !unseeded;
+    const decidable = !isLocal();
     setCrumbs(
         el("button", { type: "button", class: "link", onclick: showGrid }, `${targetLabel()} / ${state.project}`),
         el("span", {}, itemName(item)),
@@ -1069,7 +1065,7 @@ function showStory() {
                           "p",
                           {},
                           "No baseline yet, and this pull request does not change it: it looks as on master. " +
-                              "Seed it from master's capture, or accept it on the pull request that changes it.",
+                              "Accepting it makes this image its first baseline.",
                       )
                     : null,
                 decidable ? decisionButtons : null,