@appliqation/visual-regression 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -16
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -1,18 +1,18 @@
|
|
|
1
1
|
# Appliqation Visual-Regression
|
|
2
2
|
|
|
3
|
-
**Checks one route for a real visual regression by diffing it against its own live production counterpart
|
|
3
|
+
**Checks one route for a real visual regression by diffing it against its own live production counterpart: no stored baseline files, no manual baseline-approval workflow.**
|
|
4
4
|
|
|
5
|
-
Point it at a route and a test case
|
|
5
|
+
Point it at a route and a test case. It navigates to that route on production and a target environment, masks any known dynamic regions, full-page screenshots both, pixel-diffs them for real, and has a model judge the result from the actual evidence: a genuine regression, an expected divergence, or a route that simply doesn't exist on production yet (declines rather than substituting any other baseline).
|
|
6
6
|
|
|
7
7
|
## Why this exists
|
|
8
8
|
|
|
9
|
-
A Playwright `getByRole` assertion can pass while the page looks broken
|
|
9
|
+
A Playwright `getByRole` assertion can pass while the page looks broken: an overlapping modal, a button pushed off-screen by a CSS regression, invisible text from a contrast bug. Nothing else in this agent family checks appearance, only behaviour. Traditional visual-regression tooling solves this with a stored baseline image that a human has to keep re-approving every time a legitimate UI change ships; in practice, that maintenance burden is what kills most setups. This agent sidesteps it entirely by using **production itself as the baseline, fetched live at comparison time**. Production always represents current truth by definition; there's nothing to store or re-approve.
|
|
10
10
|
|
|
11
11
|
## The one rule that matters more than anything else here
|
|
12
12
|
|
|
13
|
-
**If the route doesn't exist on production, that's not a failure
|
|
13
|
+
**If the route doesn't exist on production, that's not a failure: it's not applicable.** This agent never falls back to comparing against a design file or mock (a live render vs. a Figma file is a different, unreliable problem: design fidelity, not regression detection). No production equivalent means no comparison, reported plainly, nothing forced.
|
|
14
14
|
|
|
15
|
-
The mechanical work
|
|
15
|
+
The mechanical work (navigating both environments, masking, screenshotting, pixel-diffing) is entirely code-owned, never something the model claims: it happens inside one atomic `capture_and_diff` call, and the real diff statistics it returns are what the model is required to cite, never a number it asserts on its own. The model's only real job is judgment, reported back through a structured `submit_verdict` call rather than free-text prose, the same discipline `appq:autotest-validator` already uses for its own verdicts.
|
|
16
16
|
|
|
17
17
|
## Quick start
|
|
18
18
|
|
|
@@ -25,7 +25,7 @@ Create a `.env` file (in whatever directory you'll run it from) with:
|
|
|
25
25
|
|
|
26
26
|
```
|
|
27
27
|
APPQ_API_KEY=your-appliqation-api-key # read-only is enough
|
|
28
|
-
ANTHROPIC_API_KEY=your-anthropic-key # or OPENAI_API_KEY
|
|
28
|
+
ANTHROPIC_API_KEY=your-anthropic-key # or OPENAI_API_KEY (pick one)
|
|
29
29
|
```
|
|
30
30
|
|
|
31
31
|
```bash
|
|
@@ -38,33 +38,33 @@ appliqation-visual-regression check \
|
|
|
38
38
|
--mask "[data-testid=timestamp]"
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
-
Add `--json`/`--ci` for a structured summary. The exit code is 0 for `expected-divergence`/`not-applicable`, 1 for `regression`/`inconclusive` (fail-closed on ambiguity)
|
|
41
|
+
Add `--json`/`--ci` for a structured summary. The exit code is 0 for `expected-divergence`/`not-applicable`, 1 for `regression`/`inconclusive` (fail-closed on ambiguity). The JSON summary's `verdict` field is what actually distinguishes the outcomes, not the exit code alone.
|
|
42
42
|
|
|
43
43
|
## What this agent does not do (on purpose)
|
|
44
44
|
|
|
45
|
-
- **No route enumeration or inference.** `--route` is always explicit
|
|
45
|
+
- **No route enumeration or inference.** `--route` is always explicit; there's no structured route data on a scenario/test case to derive it from (confirmed: routes only ever exist inside free-text step descriptions). A real caller with a just-completed run derives it from real observed navigation data (`get_execution_evidence`), never by guessing at step text.
|
|
46
46
|
- **No auto-detection of dynamic regions.** `--mask` is caller-supplied CSS selectors only. The model already has to reason about "is this difference data-driven or a real break" regardless, so masking is an optimization, not a prerequisite.
|
|
47
|
-
- **No write capability.** This agent never calls an Appliqation write tool and never files a defect
|
|
48
|
-
- **No ID/slug-based dynamic-content routes.** `/blog/123` on staging has no reliable way to be matched to its "equivalent" content on production (`/blog/345`) without this agent guessing at content equivalence
|
|
49
|
-
- **No multi-step workflow replay.** This agent navigates directly to a URL
|
|
47
|
+
- **No write capability.** This agent never calls an Appliqation write tool and never files a defect; it reports a verdict, nothing else. What happens to a confirmed regression (or a secondary observation) is entirely the caller's decision.
|
|
48
|
+
- **No ID/slug-based dynamic-content routes.** `/blog/123` on staging has no reliable way to be matched to its "equivalent" content on production (`/blog/345`) without this agent guessing at content equivalence: a single `--route` assumes path identity means content identity. Static, stable routes only.
|
|
49
|
+
- **No multi-step workflow replay.** This agent navigates directly to a URL; it doesn't replay a login flow or rebuild cart state. Auth-gated pages are covered via `--storage-state`; deeper application state built up through a workflow is not.
|
|
50
50
|
|
|
51
51
|
## Primary finding vs. secondary observations
|
|
52
52
|
|
|
53
|
-
A full-page diff can surface something real that's unrelated to what the check was actually for. The verdict carries one **primary finding** (what drove the classification) and a separate list of **secondary observations
|
|
53
|
+
A full-page diff can surface something real that's unrelated to what the check was actually for. The verdict carries one **primary finding** (what drove the classification) and a separate list of **secondary observations**: other real differences noticed elsewhere on the page. Secondary observations are always reported, never silently dropped, and never affect the verdict or exit code.
|
|
54
54
|
|
|
55
55
|
## Configuration
|
|
56
56
|
|
|
57
|
-
Copy `.env.example` to `.env`. Requires `APPQ_API_KEY` (read-only access is sufficient
|
|
57
|
+
Copy `.env.example` to `.env`. Requires `APPQ_API_KEY` (read-only access is sufficient; this agent never calls an appq write tool) and one of `ANTHROPIC_API_KEY`/`OPENAI_API_KEY`.
|
|
58
58
|
|
|
59
59
|
## Running this safely
|
|
60
60
|
|
|
61
|
-
This agent has a real browser and navigates to whatever URLs `--baseline-environment`/`--target-environment` resolve to
|
|
61
|
+
This agent has a real browser and navigates to whatever URLs `--baseline-environment`/`--target-environment` resolve to. It has no filesystem write access and no shell surface at all (unlike `heal-selector`/`scriptgen`, it never patches anything).
|
|
62
62
|
|
|
63
63
|
**Run this inside a container with an egress allowlist**, same as every agent in this family. This process only ever legitimately needs to reach:
|
|
64
64
|
|
|
65
65
|
- your LLM provider (`api.anthropic.com` or `api.openai.com`)
|
|
66
66
|
- your configured `APPQ_ORIGIN` (`appq.appliqation.io` by default)
|
|
67
|
-
- the two sites under test
|
|
67
|
+
- the two sites under test (whatever `--baseline-environment`/`--target-environment` resolve to)
|
|
68
68
|
|
|
69
69
|
Anything else this process tries to reach is unexpected and worth investigating.
|
|
70
70
|
|
|
@@ -84,4 +84,4 @@ See `CLAUDE.md` for a map of this repo if you're working in it with an AI coding
|
|
|
84
84
|
|
|
85
85
|
## License
|
|
86
86
|
|
|
87
|
-
MIT
|
|
87
|
+
MIT. See [LICENSE](./LICENSE).
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@appliqation/visual-regression",
|
|
3
|
-
"version": "0.1.
|
|
4
|
-
"description": "Standalone agent that checks a page for real visual regressions by diffing it against its own live production counterpart
|
|
3
|
+
"version": "0.1.1",
|
|
4
|
+
"description": "Standalone agent that checks a page for real visual regressions by diffing it against its own live production counterpart: no stored baseline files, no manual baseline-approval workflow. Declines rather than compares when no production equivalent exists.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
7
7
|
"type": "git",
|