@recursica/adapter-tester 1.1.0 → 2.0.0

package/README.md

@@ -8,38 +8,47 @@ The `@recursica/adapter-tester` ensures that a Recursica adapter's components vi
 
 This package provides three primary utilities:
 
-1. **Automated Visual Tests:** Headless, pixel-by-pixel regression checks for all stories.
+1. **Automated Visual Tests:** Headless, pixel-by-pixel regression checks for all stories, run as a single CLI command (`adapter-tester`) reading a small JSON config — no Playwright config or spec files to author.
 2. **Interactive Dev Mode:** A side-by-side synchronized browser environment with built-in note taking for auditing components and feeding fixes directly to an AI agent.
-3. **An installable library** (`import ... from "@recursica/adapter-tester"`) so any adapter repo — including ones that never checked out this monorepo — can run the same comparison against its own local Storybook. See [Using this package in another adapter repo](#using-this-package-in-another-adapter-repo).
+3. **An installable devDependency**, currently wired into `@recursica/mui-adapter` (not `@recursica/mantine-adapter` — Mantine is itself the source of truth, so it has nothing to diff against). Any other adapter repo — including one that never checked out this monorepo — can install it the same way. See [Using this package in another adapter repo](#using-this-package-in-another-adapter-repo).
 
-Inside this monorepo, it runs its bundled config comparing `@recursica/mantine-adapter` (source of truth) against `@recursica/mui-adapter`, both launched as sibling workspace packages — see `adapter-tester.config.ts`.
+**Default mode** (used by `@recursica/mui-adapter`, and any external adapter repo installing this package): compares this project's own Storybook — the one already defined by its `storybook` npm script — against a throwaway harness that installs the _published_ `@recursica/mantine-adapter` from npm. No monorepo checkout, no manual config, in most cases no config file at all.
+
+**Non-standard mode**: this monorepo also runs adapter-tester against itself — Mantine and MUI both as local sibling workspace packages — to catch regressions in _uncommitted_ code before it's published. See `adapter-tester.config.json` in this package for that config; it's the same CLI, just pointed at `sourceOfTruth.type: "url"` instead of the default Mantine harness.
 
 ---
 
-## Prerequisites
+## npm scripts
+
+Installing `@recursica/adapter-tester` as a devDependency (see [below](#using-this-package-in-another-adapter-repo)) wires up two scripts:
 
-Before running the tester or dev mode, ensure that both adapter Storybooks are running locally on their assigned odd ports to prevent clashing:
+```json
+{
+  "scripts": {
+    "adapter-tester": "adapter-tester --serve",
+    "adapter-tester:automated": "adapter-tester"
+  }
+}
+```
 
-- **Mantine Adapter (Source of Truth)**: `http://localhost:6011`
-- **MUI Adapter (Target)**: `http://localhost:6012`
+- `npm run adapter-tester` — Interactive Dev Mode: boots both Storybooks (reusing them if already running) and opens the synced side-by-side comparison browser.
+- `npm run adapter-tester:automated` — the headless Playwright pixel-diff suite; see [Automated Visual Tests](#automated-visual-tests) below.
 
-_(Launch these from their respective package directories using `npm run storybook`)_
+Both boot Storybooks themselves — nothing needs to be running beforehand.
 
 ---
 
 ## Interactive Dev Mode
 
-The **Dev Mode** is a powerful visual auditing tool that syncs the Mantine and MUI Storybooks in real-time. When you interact with controls (like toggling 'disabled' or changing 'size') in the Mantine pane, the Dev Mode instantly updates the MUI pane to match that exact state.
+The **Dev Mode** is a powerful visual auditing tool that syncs the source-of-truth and target Storybooks in real-time. When you interact with controls (like toggling 'disabled' or changing 'size') in the left pane, Dev Mode instantly updates the right pane to match that exact state.
 
 ### How to Start
 
-1. Ensure the Storybooks are running on `6011` and `6012`.
-2. Run the Dev Mode server:
-   ```bash
-   cd packages/adapter-tester
-   npm run dev
-   ```
-3. Open your browser to [http://localhost:6010](http://localhost:6010).
+```bash
+npm run adapter-tester
+```
+
+This boots both Storybooks (reusing them if already running) and opens [http://localhost:6010](http://localhost:6010) automatically.
 
 ### Taking Component Notes
 
@@ -70,95 +79,76 @@ The Dev Mode server fetches this file dynamically, so any changes you make to `r
 To execute a full headless visual regression scan across all components:
 
 ```bash
-cd packages/adapter-tester
-npm run test:visual
+npm run adapter-tester:automated   # runs the `adapter-tester` CLI
 ```
 
+`adapter-tester` reads `adapter-tester.config.json` from the current directory (or falls back to defaults if the file doesn't exist), generates a throwaway Playwright config + spec under `.adapter-tester/run/` (git-ignore this directory), and runs it. There's no Playwright config or spec file to hand-author.
+
+Pass `--dry-run` to print the resolved config and generated files without booting anything — useful for checking what a config resolves to.
+
 ### What happens during execution?
 
-1. **Automatic Storybook Bootup**: The testing engine checks ports `6011` and `6012`. If they are inactive, it automatically starts the Mantine and MUI Storybooks in the background and cleans them up upon exit. If they are already active, it instantly reuses them.
-2. **Story Discovery**: The test suite fetches the source of truth's Storybook index (`http://localhost:6011/index.json`) to dynamically discover and parameterize tests for every component story, excluding `@recursica/storybook-template`'s own default token/theme demo stories (`Theme/*`, `Tokens/*`) and each adapter's own onboarding stories (`Introduction/*`).
-3. **Headless Snapshots**: It launches headless Chrome in the background, navigates to the isolated iframe views for both adapters, and takes side-by-side snapshots.
-4. **Pixel Diffing**: It diffs the images using `pixelmatch` against our global mismatch threshold.
+1. **Automatic Storybook Bootup**: `adapter-tester` boots both sides of the comparison — this project's own Storybook, and the source-of-truth Storybook (a throwaway Mantine harness by default) — reusing them if already running.
+2. **Story Discovery**: The test suite fetches the source of truth's Storybook index (`/index.json`) to dynamically discover and parameterize tests for every component story, excluding `@recursica/storybook-template`'s own default token/theme demo stories (`Theme/*`, `Tokens/*`) and each adapter's own onboarding stories (`Introduction/*`).
+3. **Headless Snapshots**: It launches headless Chrome in the background, navigates to the isolated iframe views for both targets, and takes side-by-side snapshots.
+4. **Pixel Diffing**: It diffs the images using `pixelmatch` against the configured mismatch threshold.
 5. **Native Report Generation**: All raw snapshots, highlighted pixel diff overlays, and extracted computed CSS DOM trees are embedded directly as test attachments.
 
 ### Output & Reports
 
 All test outcomes and visual outputs are compiled into the standard, git-ignored Playwright reports folder:
 
-- **Interactive HTML Dashboard**: View the full side-by-side browser layout, visual difference overlays, and style audits by opening:
-  `packages/adapter-tester/playwright-report/index.html`
+- **Interactive HTML Dashboard**: View the full side-by-side browser layout, visual difference overlays, and style audits by opening `playwright-report/index.html`.
 - **Review in Browser**: Open the interactive dashboard directly from your terminal by running:
   ```bash
   npx playwright show-report
   ```
-- **Raw Screenshot Assets**: Individual screenshot buffers and visual diff outputs are retained in the standard `packages/adapter-tester/test-results/` folder for reference.
+- **Raw Screenshot Assets**: Individual screenshot buffers and visual diff outputs are retained in the standard `test-results/` folder for reference.
 
 ---
 
-## Using this package in another adapter repo
-
-`@recursica/adapter-tester` is a real, installable devDependency — a standalone adapter repo (not checked out inside this monorepo) can use it to diff its own Storybook against Mantine, Recursica's source-of-truth adapter, without cloning this monorepo at all.
-
-```bash
-npm install --save-dev @recursica/adapter-tester
-```
+## Configuring `adapter-tester.config.json`
 
-Add a config listing your adapter's already-running local Storybook alongside a Mantine source-of-truth target:
+All fields are optional — an adapter repo with a normal `storybook` npm script (e.g. `storybook dev -p 6012`) needs no config file at all.
 
-```ts
-// adapter-tester.config.ts
-import { defineAdapterTesterConfig } from "@recursica/adapter-tester";
+Every `adapter-tester.config.json` is validated against [`src/adapter-tester.schema.json`](./src/adapter-tester.schema.json) before a run — unknown fields, wrong types, and invalid `sourceOfTruth` shapes fail fast with a specific error instead of silently doing the wrong thing. Point your editor at it for autocomplete/inline docs:
 
-export default defineAdapterTesterConfig({
-  targets: [
-    { name: "Mantine", url: "http://localhost:6011", sourceOfTruth: true },
-    { name: "MyAdapter", url: "http://localhost:6006" },
-  ],
-  diffThresholdPixels: 3500,
-});
+```json
+{
+  "$schema": "./node_modules/@recursica/adapter-tester/src/adapter-tester.schema.json",
+  "name": "MyAdapter",
+  "storybook": { "port": 6006, "command": "npm run storybook" },
+  "sourceOfTruth": { "type": "mantine-harness" },
+  "diffThresholdPixels": 3500,
+  "relaxedThresholdStoryIds": ["ui-kit-slider"],
+  "relaxedThresholdPixels": 15000,
+  "excludeTitlePrefixes": ["Theme", "Tokens", "Introduction"]
+}
 ```
 
-Wire the Mantine side up as a `playwright.config.ts` `webServer` entry with `mantineSourceOfTruthWebServer()` — it scaffolds a small throwaway project that installs the real, published `@recursica/mantine-adapter` package and boots a real Storybook from it, so you never need this monorepo checked out:
-
-```ts
-// playwright.config.ts
-import { defineConfig } from "@playwright/test";
-import { mantineSourceOfTruthWebServer } from "@recursica/adapter-tester";
-import config from "./adapter-tester.config";
-
-export default defineConfig({
-  testDir: "./tests",
-  webServer: [
-    mantineSourceOfTruthWebServer({
-      dir: "./.adapter-tester/mantine-harness", // add to .gitignore
-      port: 6011,
-    }),
-    {
-      command: "npm run storybook", // your own adapter's Storybook
-      port: 6006,
-      reuseExistingServer: !process.env.CI,
-    },
-  ],
-});
-```
+- `name` — label for this project's own target. Defaults to the unscoped `package.json` name.
+- `storybook.port` — defaults to whatever `-p`/`--port` is set on this project's own `storybook` script; falls back to Storybook's default of `6006`.
+- `storybook.command`/`storybook.cwd` — default to `npm run storybook` in the current directory.
+- `sourceOfTruth` — defaults to `{ "type": "mantine-harness" }`: a throwaway harness that installs the _published_ `@recursica/mantine-adapter` from npm, so you never need this monorepo checked out. This is the standard mode every external adapter repo (and `@recursica/mui-adapter`) uses.
+- `sourceOfTruth.type: "url"` — the **non-standard** mode: points at an already-addressable Storybook via `command`/`port`/`cwd` instead of the harness. Used inside this monorepo to compare local, uncommitted `mantine-adapter`/`mui-adapter` workspace packages against each other — see `adapter-tester.config.json` in this package.
+- `diffThresholdPixels`/`relaxedThresholdStoryIds`/`relaxedThresholdPixels`/`excludeTitlePrefixes` — same meaning as before; see `src/adapter-tester.schema.json` for full docs.
 
-Then define the test suite itself in a spec file — it diffs every non-source-of-truth target against the source of truth, story by story:
+---
 
-```ts
-// tests/visual-regression.spec.ts
-import { runVisualRegression } from "@recursica/adapter-tester/testing";
-import config from "../adapter-tester.config";
+## Using this package in another adapter repo
 
-await runVisualRegression(config);
-```
+`@recursica/adapter-tester` is a real, installable devDependency — a standalone adapter repo (not checked out inside this monorepo) can use it to diff its own Storybook against Mantine, Recursica's source-of-truth adapter, without cloning this monorepo at all.
 
 ```bash
-npx playwright test
+npm install --save-dev @recursica/adapter-tester @playwright/test
+npx playwright install chromium
 ```
 
-Notes:
+If your repo already has a `storybook` npm script with a port set (`storybook dev -p 6006`), no config file is required. Add the two scripts from [npm scripts](#npm-scripts) above to your `package.json`, then:
+
+```bash
+npm run adapter-tester              # Interactive Dev Mode
+npm run adapter-tester:automated    # headless Playwright suite
+```
 
-- `mantineSourceOfTruthWebServer()` regenerates its harness directory on every run — commit `.adapter-tester/` to your `.gitignore` rather than checking it in.
-- Pass `mantineAdapterVersion`/`storybookTemplateVersion` to pin the harness to a specific published release instead of tracking `latest`.
-- `defineAdapterTesterConfig` requires exactly one target marked `sourceOfTruth: true`; it throws otherwise.
+Add `adapter-tester.config.json` only to override defaults (see [Configuring `adapter-tester.config.json`](#configuring-adapter-testerconfigjson) above), e.g. to raise the diff threshold or relax specific stories. Add `.adapter-tester/` to your `.gitignore` — it's regenerated on every run.

package/dist/adapter-tester.schema.json.d.ts

@@ -0,0 +1,116 @@
+declare const _default: {
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://github.com/borderux/recursica/tree/main/packages/adapter-tester/src/adapter-tester.schema.json",
+  "title": "adapter-tester.config.json",
+  "description": "Configuration for @recursica/adapter-tester. All fields are optional — an adapter repo with a normal `storybook` npm script needs no config file at all.",
+  "type": "object",
+  "additionalProperties": false,
+  "properties": {
+    "$schema": {
+      "type": "string",
+      "description": "Optional pointer to this schema for editor autocomplete, e.g. \"./node_modules/@recursica/adapter-tester/src/adapter-tester.schema.json\"."
+    },
+    "name": {
+      "type": "string",
+      "description": "Label for this project's own target. Defaults to the unscoped name in this project's package.json (e.g. \"@recursica/mui-adapter\" -> \"mui-adapter\")."
+    },
+    "storybook": {
+      "type": "object",
+      "additionalProperties": false,
+      "description": "This project's own Storybook target.",
+      "properties": {
+        "port": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 65535,
+          "description": "Port the target's Storybook is served on. Auto-detected from this project's own `scripts.storybook` (a `-p <port>`/`--port <port>` flag) when omitted, falling back to Storybook's own default of 6006."
+        },
+        "command": {
+          "type": "string",
+          "description": "Command that boots the target's Storybook. Defaults to `npm run storybook`."
+        },
+        "cwd": {
+          "type": "string",
+          "description": "Directory the command runs in, relative to the config file. Defaults to \".\"."
+        }
+      }
+    },
+    "sourceOfTruth": {
+      "description": "The comparison target. Defaults to `{ \"type\": \"mantine-harness\" }`.",
+      "oneOf": [
+        {
+          "type": "object",
+          "additionalProperties": false,
+          "description": "Default mode: boots a throwaway harness that installs the published @recursica/mantine-adapter from npm — no monorepo checkout required.",
+          "properties": {
+            "type": { "const": "mantine-harness" },
+            "port": {
+              "type": "integer",
+              "minimum": 1,
+              "maximum": 65535,
+              "description": "Port the throwaway harness's Storybook is served on. Defaults to 6011."
+            },
+            "mantineAdapterVersion": {
+              "type": "string",
+              "description": "npm version/tag of @recursica/mantine-adapter to install in the harness. Defaults to `latest`."
+            },
+            "storybookTemplateVersion": {
+              "type": "string",
+              "description": "npm version/tag of @recursica/storybook-template to install in the harness. Defaults to `latest`."
+            }
+          }
+        },
+        {
+          "type": "object",
+          "additionalProperties": false,
+          "description": "Non-standard mode: points at an already-addressable Storybook — e.g. a sibling workspace package's own Storybook inside this monorepo.",
+          "required": ["type", "port"],
+          "properties": {
+            "type": { "const": "url" },
+            "name": {
+              "type": "string",
+              "description": "Label for the source-of-truth target. Defaults to \"Mantine\"."
+            },
+            "port": {
+              "type": "integer",
+              "minimum": 1,
+              "maximum": 65535,
+              "description": "Port the source-of-truth Storybook is served on."
+            },
+            "command": {
+              "type": "string",
+              "description": "Command that boots the source-of-truth Storybook. Defaults to `npm run storybook`."
+            },
+            "cwd": {
+              "type": "string",
+              "description": "Directory the command runs in, relative to the config file. Defaults to \".\"."
+            }
+          }
+        }
+      ]
+    },
+    "diffThresholdPixels": {
+      "type": "number",
+      "minimum": 0,
+      "description": "Global visual diff threshold, in mismatched pixels, before a story is considered a failure. Defaults to 3500. AI agents must not modify this field — see AGENT.md."
+    },
+    "relaxedThresholdStoryIds": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Story id prefixes that use `relaxedThresholdPixels` instead of the global `diffThresholdPixels`."
+    },
+    "relaxedThresholdPixels": {
+      "type": "number",
+      "minimum": 0,
+      "description": "Diff threshold applied to stories matching `relaxedThresholdStoryIds`."
+    },
+    "excludeTitlePrefixes": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Storybook title prefixes to exclude from comparison, in addition to the built-in denylist (`Theme/*`, `Tokens/*`, `Introduction`)."
+    }
+  }
+}
+;
+
+export default _default;