cypress-annotate 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. package/AGENTS.md +65 -0
  2. package/DOCS/pipeline.md +258 -0
  3. package/LICENSE +21 -0
  4. package/README.md +360 -0
  5. package/dist/annotate-image.d.ts +60 -0
  6. package/dist/annotate-image.d.ts.map +1 -0
  7. package/dist/annotate-image.js +160 -0
  8. package/dist/annotate.d.ts +9 -0
  9. package/dist/annotate.d.ts.map +1 -0
  10. package/dist/annotate.js +82 -0
  11. package/dist/browser/collect-elements.js +158 -0
  12. package/dist/browser/measure-target.js +69 -0
  13. package/dist/claude-reasoner.d.ts +17 -0
  14. package/dist/claude-reasoner.d.ts.map +1 -0
  15. package/dist/claude-reasoner.js +192 -0
  16. package/dist/cli.d.ts +2 -0
  17. package/dist/cli.d.ts.map +1 -0
  18. package/dist/cli.js +74 -0
  19. package/dist/cypress/commands.d.ts +50 -0
  20. package/dist/cypress/commands.d.ts.map +1 -0
  21. package/dist/cypress/commands.js +68 -0
  22. package/dist/cypress/failure-explainer.d.ts +45 -0
  23. package/dist/cypress/failure-explainer.d.ts.map +1 -0
  24. package/dist/cypress/failure-explainer.js +123 -0
  25. package/dist/cypress/failure-hook.d.ts +30 -0
  26. package/dist/cypress/failure-hook.d.ts.map +1 -0
  27. package/dist/cypress/failure-hook.js +148 -0
  28. package/dist/cypress/failure-report.d.ts +41 -0
  29. package/dist/cypress/failure-report.d.ts.map +1 -0
  30. package/dist/cypress/failure-report.js +14 -0
  31. package/dist/cypress/failure-selector.d.ts +45 -0
  32. package/dist/cypress/failure-selector.d.ts.map +1 -0
  33. package/dist/cypress/failure-selector.js +107 -0
  34. package/dist/cypress/measure-dom.d.ts +52 -0
  35. package/dist/cypress/measure-dom.d.ts.map +1 -0
  36. package/dist/cypress/measure-dom.js +132 -0
  37. package/dist/cypress/scroll-into-view.d.ts +15 -0
  38. package/dist/cypress/scroll-into-view.d.ts.map +1 -0
  39. package/dist/cypress/scroll-into-view.js +17 -0
  40. package/dist/cypress/task.d.ts +47 -0
  41. package/dist/cypress/task.d.ts.map +1 -0
  42. package/dist/cypress/task.js +69 -0
  43. package/dist/draw.d.ts +19 -0
  44. package/dist/draw.d.ts.map +1 -0
  45. package/dist/draw.js +177 -0
  46. package/dist/finder.d.ts +68 -0
  47. package/dist/finder.d.ts.map +1 -0
  48. package/dist/finder.js +64 -0
  49. package/dist/index.d.ts +32 -0
  50. package/dist/index.d.ts.map +1 -0
  51. package/dist/index.js +40 -0
  52. package/dist/inventory.d.ts +33 -0
  53. package/dist/inventory.d.ts.map +1 -0
  54. package/dist/inventory.js +88 -0
  55. package/dist/measure.d.ts +11 -0
  56. package/dist/measure.d.ts.map +1 -0
  57. package/dist/measure.js +145 -0
  58. package/dist/pipeline.d.ts +60 -0
  59. package/dist/pipeline.d.ts.map +1 -0
  60. package/dist/pipeline.js +115 -0
  61. package/dist/types.d.ts +115 -0
  62. package/dist/types.d.ts.map +1 -0
  63. package/dist/types.js +1 -0
  64. package/package.json +113 -0
  65. package/skills/annotate-bug/SKILL.md +121 -0
  66. package/skills/cypress-annotate/SKILL.md +154 -0
  67. package/src/browser/collect-elements.js +158 -0
  68. package/src/browser/measure-target.js +69 -0
package/AGENTS.md ADDED
@@ -0,0 +1,65 @@
1
+ # Notes for coding agents
2
+
3
+ `cypress-annotate` draws a pixel-accurate box on exactly what's wrong in a
4
+ Cypress screenshot. If you are adding it to someone's Cypress suite, read
5
+ `skills/cypress-annotate/SKILL.md`, which is written for that job and covers the
6
+ setup, the options, and the traps.
7
+
8
+ This file is for working inside this repo.
9
+
10
+ ## The one rule that matters
11
+
12
+ Measure, then screenshot, with no navigation, scroll, or click in between. A
13
+ rectangle is only valid for the scroll position it was measured at. Everything
14
+ else in the pipeline can recover from being wrong. This it cannot.
15
+
16
+ ## Layout
17
+
18
+ | Path | What it is |
19
+ | --- | --- |
20
+ | `src/cypress/` | The Cypress plugin. Needs no browser automation library. |
21
+ | `src/annotate-image.ts`, `src/draw.ts` | The shared engine. sharp and SVG, no browser. |
22
+ | `src/annotate.ts`, `src/measure.ts`, `src/pipeline.ts`, `src/finder.ts` | The Playwright pipeline. |
23
+ | `src/claude-reasoner.ts` | The optional reasoning layer. |
24
+ | `scripts/` | Every suite, plus the CLIs. No test framework, they are plain scripts. |
25
+ | `fixtures/` | Flat-coloured targets the alignment suite scans for. |
26
+
27
+ Three entry points, deliberately independent: the root, `./reasoner`, and
28
+ `./cypress/*`. `playwright`, `@anthropic-ai/sdk` and `cypress` are all optional
29
+ peer dependencies.
30
+
31
+ **Do not import `ClaudeReasoner` into anything the root barrel re-exports.** That
32
+ already shipped as a bug once. An ES module evaluates a whole file's top-level
33
+ code the moment anything is imported from it, so putting the reasoner beside a
34
+ re-exported utility made every root import throw `ERR_MODULE_NOT_FOUND` for the
35
+ Anthropic SDK, even for callers who never touched it. The `package install` CI
36
+ job exists to catch a repeat.
37
+
38
+ ## Before you claim something works
39
+
40
+ This repo's standard is that claims are verified, not assumed. Several comments
41
+ say so explicitly, and they mean it.
42
+
43
+ ```bash
44
+ npm run typecheck
45
+ npm run test:cypress # 7 cases, real Cypress
46
+ npm run test:failure-selector # 12 cases, no browser
47
+ npm run test:cypress-failures # 4 cases end to end, real Cypress
48
+ npm run test:svg-escaping # 5 cases
49
+ npm run verify # alignment, 20 cases, needs playwright chromium
50
+ npm run verify:image # 10 cases, needs playwright chromium
51
+ ```
52
+
53
+ Eyeballing a screenshot does not prove pixel accuracy. `scripts/verify.ts` scans
54
+ for a target's flat colour, recovers its true pixel footprint, and compares that
55
+ against the computed rect. Add a case there rather than looking at an image.
56
+
57
+ ## Writing
58
+
59
+ No em dashes or en dashes, in code comments, docs, or commit messages. Use a
60
+ spaced hyphen, a comma, or parentheses. CI fails the build on one. The single
61
+ exemption is the element-inventory separator in `src/inventory.ts`, which is
62
+ program output that goes into a model's prompt.
63
+
64
+ Avoid the "it's not X, it's Y" construction, including the compressed trailing
65
+ form ("a mechanical pass, no rewording required").
@@ -0,0 +1,258 @@
1
+ # The Playwright pipeline and the reasoning layer
2
+
3
+ Two things ship in `cypress-annotate` that the Cypress plugin never touches: a
4
+ general Playwright-driven annotation pipeline, and a Claude-based reasoning layer
5
+ that looks at a page and works out *what's* wrong before drawing the box. This
6
+ document covers both. For the Cypress plugin, see the [README](../README.md).
7
+
8
+ Both are optional. `playwright` and `@anthropic-ai/sdk` are optional
9
+ `peerDependencies`, so install whichever your own usage calls into:
10
+
11
+ ```bash
12
+ npm install playwright @anthropic-ai/sdk
13
+ npx playwright install chromium
14
+ ```
15
+
16
+ The repo's own suites for this half:
17
+
18
+ ```bash
19
+ npm run verify # coordinate-pipeline alignment suite, 20 cases
20
+ npm run verify:image # driver-agnostic (MCP) alignment suite, 10 cases
21
+ npm run test:reasoner # Claude reasoning request/parse suite, 12 cases, no API key needed
22
+ npm run demo # end-to-end on the buggy fixture, using recorded findings
23
+ npm run annotate -- --url https://example.com --selector 'a' --label 'Broken link'
24
+ ```
25
+
26
+ ## API
27
+
28
+ ```ts
29
+ import { captureAnnotated } from 'cypress-annotate';
30
+
31
+ const result = await captureAnnotated('https://example.com', [
32
+ { selector: '#submit', label: 'Misaligned by 12px', shape: 'box' },
33
+ { selector: ['iframe#checkout', '.pay-button'], label: 'Overlaps' },
34
+ ], { mode: 'element', cropPadding: 80, style: { dimOutside: 0.5 } });
35
+
36
+ await writeFile('bug.png', result.image);
37
+ console.log(result.elements, result.drawnRects, result.warnings);
38
+ ```
39
+
40
+ `captureAnnotated()` is the only export in the package that launches a browser,
41
+ and it imports playwright lazily on first call. `annotateImage`, `runBugHunt`,
42
+ and `findingsToAnnotations` all work with no browser installed.
43
+
44
+ Or drive an existing Playwright page directly with
45
+ `annotate(page, annotations, options)`. That's the entry point the reasoning step
46
+ uses, since the agent already has a page open.
47
+
48
+ **Modes.** `viewport` captures what's on screen, scrolling the first target into
49
+ view first. `fullPage` captures the whole document. `element` captures full-page
50
+ then crops to the annotated region plus `cropPadding`, which is usually the most
51
+ useful thing to attach to a ticket.
52
+
53
+ **Result.** Alongside `image`, you get `rawImage` (same capture, same crop, no
54
+ overlay), the measured `elements`, the final `drawnRects` in image pixels, and
55
+ `warnings`.
56
+
57
+ **Warnings worth acting on:** selector matched more than one element, element has
58
+ a zero-sized box, element is outside the viewport, element is fixed during a
59
+ full-page capture.
60
+
61
+ ### The region fallback
62
+
63
+ When no element corresponds to the problem (blank space where something should
64
+ be), a target can be a rectangle in percentages instead:
65
+
66
+ ```ts
67
+ { region: { xPct: 10, yPct: 50, widthPct: 30, heightPct: 8, basis: 'viewport' } }
68
+ ```
69
+
70
+ `basis: 'viewport'` percentages are relative to the scroll position they were
71
+ measured at, which is why the hunt never scrolls between measuring and drawing.
72
+ `basis: 'document'` is absolute. Both paths are covered by the alignment suite
73
+ and land on the same pixels the selector path does.
74
+
75
+ ## The reasoning step
76
+
77
+ Point it at a URL. That's the whole invocation:
78
+
79
+ ```bash
80
+ npm run find-bugs -- --url https://news.ycombinator.com
81
+ ```
82
+
83
+ A ticket is optional. With one, findings are judged against it. Without one, the
84
+ page is judged on its own terms.
85
+
86
+ ```bash
87
+ npm run find-bugs -- --url https://your.app/checkout \
88
+ --ticket "CHK-482: customer reviews the order and places it"
89
+ ```
90
+
91
+ **The problem this solves.** A model cannot reliably invent a CSS selector by
92
+ looking at a screenshot. So it is never asked to. Before the screenshot is taken,
93
+ every candidate element is tagged with a `data-annot-ref` attribute and listed
94
+ for the model with its tag, accessible name and bounding box:
95
+
96
+ ```
97
+ [e17] <button> "Apply" @ 1090,642 87x39 — #promo-apply
98
+ [e24] <span> "£244.99" @ 1262,1046 57x22 — #grand-total-value
99
+ ```
100
+
101
+ The model returns a handle, which maps back to a selector guaranteed to resolve
102
+ because we injected it. The readable selector (`#promo-apply`) still goes in the
103
+ ticket. Those bounding boxes are also the model's best evidence for geometric
104
+ bugs, since overlapping boxes and boxes that run past a parent show up in the
105
+ numbers as clearly as in the pixels.
106
+
107
+ **Structured output.** Findings come back through a strict tool schema:
108
+
109
+ ```json
110
+ { "description": "...", "evidence": "...", "severity": "blocker", "ref": "e17", "region": null }
111
+ ```
112
+
113
+ Severity drives the annotation colour. Every finding gets a cropped, dimmed
114
+ close-up, and each pass also produces an overview with numbered markers, because
115
+ full-sentence labels on an overview cover the rest of the page.
116
+
117
+ **Three resolution paths, all tested:** a valid handle resolves to its injected
118
+ selector; a handle the model invented falls back to the region it supplied; a
119
+ finding with neither is reported as unresolved rather than silently dropped.
120
+
121
+ **Viewport sweep.** `runBugHunt` works down the page one viewport at a time,
122
+ overlapping passes by 10%. A pass is the unit of correctness: the inventory, the
123
+ screenshot the model reasons over, and any percentage region it returns all share
124
+ one scroll position. Findings are de-duplicated across passes by readable
125
+ selector, since handles are re-issued each pass.
126
+
127
+ ## Using this from a live agent session
128
+
129
+ When Claude is already driving a real browser, say a Playwright MCP or Chrome
130
+ DevTools MCP session that is logged in, the thing to avoid is opening a second
131
+ one. A second browser would not share the session and would land on the SSO login
132
+ page, so this path never opens one.
133
+
134
+ Three pieces:
135
+
136
+ - **[src/browser/measure-target.js](../src/browser/measure-target.js)** is a
137
+ snippet to paste into whichever `evaluate` tool is live. Edit one line, run it,
138
+ get `{ metrics, targets }` back. Both backends can write the result straight to
139
+ a file (`filename` on Playwright MCP, `filePath` on Chrome DevTools MCP).
140
+ - **`npm run annotate-shot`** takes a PNG plus that JSON and draws the boxes. It
141
+ never opens a browser and knows nothing about either backend.
142
+ - **[skills/annotate-bug/SKILL.md](../skills/annotate-bug/SKILL.md)** is the
143
+ protocol written as a skill, with the exact tool calls for both backends. Copy
144
+ it into `.claude/skills/`.
145
+
146
+ ```bash
147
+ npm run annotate-shot -- --image shot.png --measurement measure.json \
148
+ --label "Apply button escapes its card" --out annotated.png --crop
149
+ ```
150
+
151
+ **The ordering rule that makes it work:** measure, then screenshot, with no
152
+ navigation, scroll, or click in between. A rectangle is only valid for the scroll
153
+ position it was measured at. Everything else the pipeline can recover from. This
154
+ it cannot.
155
+
156
+ **The scale is measured on every capture.** Both backends can hand back a resized
157
+ image, Playwright MCP via `scale: "css"` and Chrome DevTools MCP via
158
+ `--screenshot-max-width`, so `devicePixelRatio` describes the browser and says
159
+ nothing about the file on disk. `resolveScale()` compares the image against the
160
+ viewport it was supposed to cover, uses the measured ratio when the two disagree,
161
+ and prints a warning. `npm run verify:image` proves this holds at scales from 0.5
162
+ to 2.0, including a case downscaled to 900px from a 2560px capture.
163
+
164
+ If there is no element to point at (blank space where something should render),
165
+ `--region x,y,w,h` takes percentages of the viewport instead.
166
+
167
+ ## How the reasoning step is verified
168
+
169
+ The live call has been run against the real API (see below). It is also tested
170
+ without credentials, so the suite stays runnable and free. `npm run test:reasoner`
171
+ drives `ClaudeReasoner` through a stubbed `fetch` that returns a real SSE stream,
172
+ covering request construction (model, strict tool schema, adaptive thinking, the
173
+ image and inventory blocks), stream handling, tool-use parsing, all three
174
+ resolution paths, refusals, and malformed model output. 12 checks.
175
+
176
+ `npm run demo` then runs the whole pipeline against
177
+ [fixtures/buggy-checkout.html](../fixtures/buggy-checkout.html), a page with five
178
+ deliberate bugs, using recorded findings in place of the model. The recorded
179
+ findings name real CSS selectors, which the replay translates into the handles
180
+ the model would have returned, so the resolution path under test is identical to
181
+ the live one. Two of the seven entries are deliberate fallback tests.
182
+
183
+ **To run it live**, copy `.env.example` to `.env` and put your key in it:
184
+
185
+ ```bash
186
+ cp .env.example .env
187
+ # ANTHROPIC_API_KEY=sk-ant-...
188
+
189
+ npm run find-bugs -- --url https://your.app/checkout --ticket "CHK-482: ..."
190
+ ```
191
+
192
+ `.env` is read by `scripts/find-bugs.ts` via `process.loadEnvFile` and is
193
+ gitignored. Exporting `ANTHROPIC_API_KEY` in your shell works too, as does an
194
+ `ant auth login` profile, and the SDK resolves credentials in that order. It
195
+ resolves them lazily, at request time rather than when the client is constructed,
196
+ so a missing key surfaces on the first request. The CLI catches that and prints
197
+ setup instructions instead of a stack trace.
198
+
199
+ The library itself never reads `.env`. Only the CLI does, so importing
200
+ `annotate()` or `ClaudeReasoner` into your own program leaves credential handling
201
+ to you.
202
+
203
+ ### What the first live runs showed
204
+
205
+ On [fixtures/buggy-checkout.html](../fixtures/buggy-checkout.html) with **no
206
+ ticket**, Claude found 4 of the 5 planted bugs, every selector resolved, and
207
+ nothing was unresolved. It missed the `#place-order` button overlapping the terms
208
+ text.
209
+
210
+ The evidence it gave is the part that matters, because it shows the design
211
+ premise holding up. It reasons from the bounding boxes as much as from the
212
+ pixels:
213
+
214
+ > `#postcode-field`: "e10/e11/e12 start at x=419 while all other fields (e3, e7,
215
+ > e9, e13) start at x=385, and the input is narrower (476 vs 510)."
216
+
217
+ Two real bugs surfaced from pointing it at Hacker News, both now fixed and
218
+ covered by the alignment suite:
219
+
220
+ - **Quirks mode.** HN has no doctype, so `documentElement.clientHeight` returns
221
+ the *content* height (1214) instead of the viewport height (900). The sweep
222
+ concluded the page fit on one screen, and percentage regions were measured
223
+ against the wrong height. Metrics now use `window.innerWidth/innerHeight`, and
224
+ [fixtures/quirks-page.html](../fixtures/quirks-page.html) covers it.
225
+ - **Stale handles.** Each pass restarts its counter at `e1`, but the previous
226
+ pass's attributes were still in the DOM, so `[data-annot-ref="e5"]` could match
227
+ two elements and annotate the wrong one. The collector now clears stale handles
228
+ before tagging.
229
+
230
+ Both surfaced from the pipeline reporting its own warnings, which is the argument
231
+ for keeping `warnings` in the report.
232
+
233
+ ## Known limits
234
+
235
+ - Findings vary between runs on the same page, as you'd expect. Two runs of the
236
+ Hacker News front page returned three findings and one.
237
+ - Judgement is not calibrated. The missing-domain finding on Hacker News is
238
+ arguably a false positive, since HN legitimately shows no domain for text
239
+ posts. The model considered that and reported it anyway.
240
+ - `element` mode captures full-page and then crops, which is wasteful on very
241
+ long pages. Fine at this stage, worth revisiting if it becomes a bottleneck.
242
+ - The inventory covers the main document and same-origin iframes. Cross-origin
243
+ frames are skipped, so bugs inside them will not be found.
244
+ - Overview labels can still overlap each other when findings are close together.
245
+
246
+ ## What's next here
247
+
248
+ Selector accuracy looks like the solved part. Across the live runs so far, every
249
+ handle Claude picked resolved to the element it described, and no real page has
250
+ needed the region fallback yet.
251
+
252
+ The open question has shifted to **judgement**, meaning precision and recall.
253
+ Claude missed the button and text overlap on the fixture, and its Hacker News
254
+ finding is arguably a false positive. Worth running against pages with known
255
+ defects and counting both the misses and the false alarms. `report.json` already
256
+ carries what that needs: `selector` is what Claude chose, `resolvedTo` is what
257
+ got annotated, `warnings` flags ambiguity, and `unresolved` lists what it could
258
+ not point at.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 bidemiajala
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.