@recursica/adapter-tester 2.2.0 → 4.0.0

package/README.md

@@ -8,13 +8,13 @@ The `@recursica/adapter-tester` ensures that a Recursica adapter's components vi
 
 This package provides three primary utilities:
 
-1. **Automated Visual Tests:** Headless, pixel-by-pixel regression checks for all stories, run as a single CLI command (`adapter-tester`) reading a small JSON config — no Playwright config or spec files to author.
+1. **Automated Visual Tests:** Headless, golden-image regression checks for all stories, run as a single CLI command (`adapter-tester`) reading a small JSON config — no Playwright config or spec files to author. See [Golden Images](#golden-images) below for the model.
 2. **Interactive Dev Mode:** A side-by-side synchronized browser environment with built-in note taking for auditing components and feeding fixes directly to an AI agent.
-3. **An installable devDependency**, currently wired into `@recursica/mui-adapter` (not `@recursica/mantine-adapter` — Mantine is itself the source of truth, so it has nothing to diff against). Any other adapter repo — including one that never checked out this monorepo — can install it the same way. See [Using this package in another adapter repo](#using-this-package-in-another-adapter-repo).
+3. **An installable devDependency**, wired into `@recursica/mantine-adapter` — the source-of-truth adapter, run in `isSourceOfTruthAdapter` mode (own-drift check only, since it has nothing above it to diverge from). Any other adapter repo — including one that never checked out this monorepo — can install it the same way. See [Using this package in another adapter repo](#using-this-package-in-another-adapter-repo).
 
-**Default mode** (used by `@recursica/mui-adapter`, and any external adapter repo installing this package): compares this project's own Storybook — the one already defined by its `storybook` npm script — against a throwaway harness that installs the _published_ `@recursica/mantine-adapter` from npm. No monorepo checkout, no manual config, in most cases no config file at all.
+**Default mode** (used by any adapter repo installing this package): checks this project's own Storybook — the one already defined by its `storybook` npm script — against its own committed golden images. No monorepo checkout, no manual config, in most cases no config file at all. Divergence from the source-of-truth adapter's (`@recursica/mantine-adapter`) own golden images is a separate, opt-in check — see [`--divergence-only`](#automated-visual-tests).
 
-**Non-standard mode**: this monorepo also runs adapter-tester against itself — Mantine and MUI both as local sibling workspace packages — to catch regressions in _uncommitted_ code before it's published. See `adapter-tester.config.json` in this package for that config; it's the same CLI, just pointed at `sourceOfTruth.type: "url"` instead of the default Mantine harness.
+`@recursica/adapter-tester` itself has no default mode and no `adapter-tester.config.json` of its own — it's purely a tool other adapters install and configure. `mantine-adapter` and `mui-adapter` are the ones that run it, each with their own config.
 
 ---
 
@@ -26,13 +26,15 @@ Installing `@recursica/adapter-tester` as a devDependency (see [below](#using-th
 {
   "scripts": {
     "adapter-tester": "adapter-tester --serve",
-    "adapter-tester:automated": "adapter-tester"
+    "adapter-tester:automated": "adapter-tester",
+    "adapter-tester:source-of-truth": "adapter-tester --divergence-only"
   }
 }
 ```
 
 - `npm run adapter-tester` — Interactive Dev Mode: boots both Storybooks (reusing them if already running) and opens the synced side-by-side comparison browser.
-- `npm run adapter-tester:automated` — the headless Playwright pixel-diff suite; see [Automated Visual Tests](#automated-visual-tests) below.
+- `npm run adapter-tester:automated` — the headless own-drift check against this project's own committed goldens; see [Automated Visual Tests](#automated-visual-tests) below. This is the one to run normally — fast, no network calls.
+- `npm run adapter-tester:source-of-truth` — the separate, opt-in divergence check against the source-of-truth adapter's published goldens. Not wired up for `@recursica/mantine-adapter` itself — it _is_ the source of truth, so it has nothing to diverge from.
 
 Both boot Storybooks themselves — nothing needs to be running beforehand.
 
@@ -76,10 +78,11 @@ The Dev Mode server fetches this file dynamically, so any changes you make to `r
 
 ## Automated Visual Tests
 
-To execute a full headless visual regression scan across all components:
+Two independent checks, gated by `--divergence-only` — the normal, everyday one and a separate opt-in one:
 
 ```bash
-npm run adapter-tester:automated   # runs the `adapter-tester` CLI
+npm run adapter-tester:automated          # own-drift check only — the normal one to run
+npm run adapter-tester:source-of-truth    # divergence check only — opt-in, against Mantine
 ```
 
 `adapter-tester` reads `adapter-tester.config.json` from the current directory (or falls back to defaults if the file doesn't exist), generates a throwaway Playwright config + spec under `.adapter-tester/run/` (git-ignore this directory), and runs it. There's no Playwright config or spec file to hand-author.
@@ -92,13 +95,22 @@ Any other args are forwarded to `playwright test`, so you can scope a run to spe
 npm run adapter-tester:automated -- --grep "Toast"
 ```
 
+Pin the source-of-truth adapter's version for the divergence check (instead of resolving `@recursica/mantine-adapter`'s `latest` on npm) with `--source-of-truth-version`:
+
+```bash
+npm run adapter-tester:source-of-truth -- --source-of-truth-version 0.53.0
+```
+
+Only meaningful with the default `sourceOfTruth.type: "mantine-harness"` mode — throws if combined with `sourceOfTruth.type: "url"` (that mode reads a sibling package's local checkout directly, there's no version to pin) or with `isSourceOfTruthAdapter: true` (nothing to pin a version for).
+
 ### What happens during execution?
 
-1. **Automatic Storybook Bootup**: `adapter-tester` boots both sides of the comparison — this project's own Storybook, and the source-of-truth Storybook (a throwaway Mantine harness by default) — reusing them if already running.
-2. **Story Discovery**: The test suite fetches the source of truth's Storybook index (`/index.json`) to dynamically discover and parameterize tests for every component story, excluding `@recursica/storybook-template`'s own default token/theme demo stories (`Theme/*`, `Tokens/*`) and each adapter's own onboarding stories (`Introduction/*`).
-3. **Headless Snapshots**: It launches headless Chrome in the background, navigates to the isolated iframe views for both targets, and takes side-by-side snapshots.
-4. **Pixel Diffing**: It diffs the images using `pixelmatch` against the configured mismatch threshold.
-5. **Native Report Generation**: All raw snapshots, highlighted pixel diff overlays, and extracted computed CSS DOM trees are embedded directly as test attachments.
+1. **Automatic Storybook Bootup**: `adapter-tester` boots only this project's own Storybook — the divergence check reads the source-of-truth adapter's stored golden files, never a live Storybook, so nothing else needs to boot.
+2. **Story Discovery**: The test suite fetches this project's own Storybook index (`/index.json`) to dynamically discover and parameterize tests for every component story, excluding `@recursica/storybook-template`'s own default token/theme demo stories (`Theme/*`, `Tokens/*`) and each adapter's own onboarding stories (`Introduction/*`).
+3. **Headless Snapshot**: It launches headless Chrome, navigates to the isolated iframe view, and takes a screenshot. A missing golden is captured from this screenshot regardless of which check is running.
+4. **Own-Drift Check** (`adapter-tester:automated`): Diffs the screenshot against this project's own `test/golden/<story-id>.png` using `pixelmatch`, against the configured mismatch threshold. No golden yet for a story is not a failure — one is captured from this run.
+5. **Divergence Check** (`adapter-tester:source-of-truth`): Diffs this project's own golden against the source-of-truth adapter's golden (skipped for the source-of-truth adapter's own config). Different and not yet reviewed via `--approve-divergence`? Flagged as a report annotation — never a failure on its own.
+6. **Native Report Generation**: Screenshots, diff overlays, and divergence flags are embedded directly as test attachments/annotations.
 
 ### Output & Reports
 
@@ -113,6 +125,36 @@ All test outcomes and visual outputs are compiled into the standard, git-ignored
 
 ---
 
+## Golden Images
+
+Each adapter's `test/golden/` directory — committed to git, alongside a `manifest.json` tracking when each was captured — is its own baseline:
+
+- **Own-drift check (hard fail):** this run's live render vs this project's own golden. Catches unintended CSS regressions.
+- **Divergence check (soft flag, never fails a run):** this project's own golden vs the source-of-truth adapter's golden — different underlying UI libraries can legitimately render a component differently, so a divergence isn't automatically wrong, just unreviewed.
+
+These are explicit, developer-run workflows — not run in CI, and not scoped down automatically; use `--grep` to target specific stories.
+
+```bash
+# Capture/refresh this project's own goldens for stories matching "Toast",
+# from the current live render. Use after an intentional styling change.
+npm run adapter-tester:automated -- --grep "Toast" --update-golden
+
+# Review a flagged divergence from the source of truth, and accept it as
+# intentional. Only valid alongside --divergence-only — there's nothing to
+# approve without the divergence check running.
+npm run adapter-tester:source-of-truth -- --grep "Toast" --approve-divergence
+```
+
+`--approve-divergence` records the source-of-truth adapter's own `manifest.json` `createdAt` for that story at review time. If the source of truth's golden is recaptured later, the divergence flags again for re-review — approval isn't a permanent exemption.
+
+A story with no golden yet (new story, or one never captured) is not a failure in either mode — one is captured from the current run automatically.
+
+Renamed/removed stories just leave an orphaned `test/golden/<story-id>.png`/manifest entry behind; nothing prunes these automatically.
+
+`manifest.json` is validated against [`src/golden/manifest.schema.json`](./src/golden/manifest.schema.json) on every read/write — don't hand-edit it.
+
+---
+
 ## Configuring `adapter-tester.config.json`
 
 All fields are optional — an adapter repo with a normal `storybook` npm script (e.g. `storybook dev -p 6012`) needs no config file at all.
@@ -121,13 +163,15 @@ Every `adapter-tester.config.json` is validated against [`src/adapter-tester.sch
 
 ```json
 {
-  "$schema": "./node_modules/@recursica/adapter-tester/src/adapter-tester.schema.json",
+  "$schema": "https://raw.githubusercontent.com/borderux/recursica/main/packages/adapter-tester/src/adapter-tester.schema.json",
   "name": "MyAdapter",
   "storybook": { "port": 6006, "command": "npm run storybook" },
   "sourceOfTruth": { "type": "mantine-harness" },
   "diffThresholdPixels": 3500,
-  "relaxedThresholdStoryIds": ["ui-kit-slider"],
-  "relaxedThresholdPixels": 15000,
+  "stories": {
+    "ui-kit-slider": { "threshold": 15000 },
+    "ui-kit-slider--range-mode-with-icons-and-inputs": { "exclude": true }
+  },
   "excludeTitlePrefixes": ["Theme", "Tokens", "Introduction"]
 }
 ```
@@ -135,9 +179,10 @@ Every `adapter-tester.config.json` is validated against [`src/adapter-tester.sch
 - `name` — label for this project's own target. Defaults to the unscoped `package.json` name.
 - `storybook.port` — defaults to whatever `-p`/`--port` is set on this project's own `storybook` script; falls back to Storybook's default of `6006`.
 - `storybook.command`/`storybook.cwd` — default to `npm run storybook` in the current directory.
-- `sourceOfTruth` — defaults to `{ "type": "mantine-harness" }`: a throwaway harness that installs the _published_ `@recursica/mantine-adapter` from npm, so you never need this monorepo checked out. This is the standard mode every external adapter repo (and `@recursica/mui-adapter`) uses.
-- `sourceOfTruth.type: "url"` — the **non-standard** mode: points at an already-addressable Storybook via `command`/`port`/`cwd` instead of the harness. Used inside this monorepo to compare local, uncommitted `mantine-adapter`/`mui-adapter` workspace packages against each other — see `adapter-tester.config.json` in this package.
-- `diffThresholdPixels`/`relaxedThresholdStoryIds`/`relaxedThresholdPixels`/`excludeTitlePrefixes` — same meaning as before; see `src/adapter-tester.schema.json` for full docs.
+- `sourceOfTruth` — defaults to `{ "type": "mantine-harness" }`: a throwaway harness that installs the _published_ `@recursica/mantine-adapter` from npm, so you never need this monorepo checked out. This is the standard mode every external adapter repo uses. `sourceOfTruth.mantineAdapterVersion` pins the version installed/fetched (defaults to `latest`) — or override it per-run with `--source-of-truth-version` instead of hardcoding it in the config.
+- `sourceOfTruth.type: "url"` — the **non-standard** mode: points at an already-addressable Storybook via `command`/`port`/`cwd` instead of the harness, and reads that sibling package's `test/golden/` directly from disk for the divergence check (including uncommitted local changes) instead of resolving a published npm version. For comparing two locally checked-out sibling adapters instead of a published one.
+- `isSourceOfTruthAdapter` — set `true` only in the source-of-truth adapter's own config (`@recursica/mantine-adapter`). Skips `sourceOfTruth`/the divergence check entirely; runs the own-drift check standalone. Also disables Dev Mode (`--serve`) — there's nothing to sync against.
+- `diffThresholdPixels`/`stories`/`excludeTitlePrefixes` — same meaning as before; see `src/adapter-tester.schema.json` for full docs.
 
 ---
 
@@ -150,11 +195,12 @@ npm install --save-dev @recursica/adapter-tester @playwright/test
 npx playwright install chromium
 ```
 
-If your repo already has a `storybook` npm script with a port set (`storybook dev -p 6006`), no config file is required. Add the two scripts from [npm scripts](#npm-scripts) above to your `package.json`, then:
+If your repo already has a `storybook` npm script with a port set (`storybook dev -p 6006`), no config file is required. Add the scripts from [npm scripts](#npm-scripts) above to your `package.json`, then:
 
 ```bash
-npm run adapter-tester              # Interactive Dev Mode
-npm run adapter-tester:automated    # headless Playwright suite
+npm run adapter-tester                    # Interactive Dev Mode
+npm run adapter-tester:automated          # own-drift check — run this normally
+npm run adapter-tester:source-of-truth    # divergence check — opt-in, against Mantine
 ```
 
 Any extra args after `adapter-tester:automated` are passed straight through to `playwright test`, so you can scope a run instead of executing the full suite:
@@ -163,4 +209,4 @@ Any extra args after `adapter-tester:automated` are passed straight through to `
 npm run adapter-tester:automated -- --grep "Toast"
 ```
 
-Add `adapter-tester.config.json` only to override defaults (see [Configuring `adapter-tester.config.json`](#configuring-adapter-testerconfigjson) above), e.g. to raise the diff threshold or relax specific stories. Add `.adapter-tester/` to your `.gitignore` — it's regenerated on every run.
+Add `adapter-tester.config.json` only to override defaults (see [Configuring `adapter-tester.config.json`](#configuring-adapter-testerconfigjson) above), e.g. to raise the diff threshold or relax specific stories. Add `.adapter-tester/` to your `.gitignore` — it's regenerated on every run. Commit `test/golden/` — that's your project's actual baseline, not a build artifact.

package/dist/adapter-tester.schema.json.d.ts

@@ -8,11 +8,11 @@ declare const _default: {
   "properties": {
     "$schema": {
       "type": "string",
-      "description": "Optional pointer to this schema for editor autocomplete, e.g. \"./node_modules/@recursica/adapter-tester/src/adapter-tester.schema.json\"."
+      "description": "Optional pointer to this schema for editor autocomplete, e.g. \"https://raw.githubusercontent.com/borderux/recursica/main/packages/adapter-tester/src/adapter-tester.schema.json\"."
     },
     "name": {
       "type": "string",
-      "description": "Label for this project's own target. Defaults to the unscoped name in this project's package.json (e.g. \"@recursica/mui-adapter\" -> \"mui-adapter\")."
+      "description": "Label for this project's own target. Defaults to the unscoped name in this project's package.json (e.g. \"@recursica/some-adapter\" -> \"some-adapter\")."
     },
     "storybook": {
       "type": "object",
@@ -89,20 +89,33 @@ declare const _default: {
         }
       ]
     },
+    "isSourceOfTruthAdapter": {
+      "type": "boolean",
+      "description": "True only for the source-of-truth adapter's own config (mantine-adapter). Skips `sourceOfTruth` entirely — there's nothing above it to diverge from — and runs the own-drift golden check standalone, against just this project's own Storybook. Defaults to false."
+    },
     "diffThresholdPixels": {
       "type": "number",
       "minimum": 0,
       "description": "Global visual diff threshold, in mismatched pixels, before a story is considered a failure. Defaults to 3500. AI agents must not modify this field — see AGENT.md."
     },
-    "relaxedThresholdStoryIds": {
-      "type": "array",
-      "items": { "type": "string" },
-      "description": "Story id prefixes that use `relaxedThresholdPixels` instead of the global `diffThresholdPixels`."
-    },
-    "relaxedThresholdPixels": {
-      "type": "number",
-      "minimum": 0,
-      "description": "Diff threshold applied to stories matching `relaxedThresholdStoryIds`."
+    "stories": {
+      "type": "object",
+      "additionalProperties": {
+        "type": "object",
+        "additionalProperties": false,
+        "properties": {
+          "threshold": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Diff threshold override for matching stories, in mismatched pixels. Overrides `diffThresholdPixels`."
+          },
+          "exclude": {
+            "type": "boolean",
+            "description": "Skip matching stories entirely — no own-drift check, no divergence check, no golden captured. For stories that don't have a cross-adapter counterpart; use `excludeTitlePrefixes` to drop a whole title instead."
+          }
+        }
+      },
+      "description": "Per-story overrides, keyed by story id prefix (a story matches if its id equals the key or starts with it). When more than one key matches a story, the longest (most specific) key wins."
     },
     "excludeTitlePrefixes": {
       "type": "array",