@regressionbot/sdk 1.2.3 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -13,7 +13,7 @@ RegressionBot is a declarative visual regression testing platform that helps you
13
13
 
14
14
  Unlike traditional visual diffing libraries, RegressionBot is designed for modern, automated development loops and agentic pipelines:
15
15
 
16
- - **Highly Accurate Regressions (Zero Noise)**: Leveraging advanced pixel-matching algorithms and element masking (using CSS selectors or the automatic `data-vr-mask` attribute), RegressionBot eliminates false positives caused by dynamic data, layout shifting, or third-party widgets.
16
+ - **Highly Accurate Regressions (Less Noise)**: Leveraging advanced pixel-matching algorithms and element masking (using CSS selectors or the automatic `data-vr-mask` attribute), RegressionBot minimizes false positives caused by dynamic data, layout shifting, or third-party widgets.
17
17
  - **Plain-English Summaries**: No more manual screenshot comparisons. RegressionBot translates visual diffs into concise, plain-English descriptions of what changed, so you know exactly what was modified at a glance.
18
18
  - **Agentic Workflow Ready**: Built from the ground up to support autonomous coding agents (like Gemini or Claude) and automated developer loops. Through standard API endpoints, CLI commands, and Model Context Protocol (MCP) integrations, agents can trigger tests, read plain-English results, and approve baseline changes programmatically without human intervention.
19
19
 
@@ -24,6 +24,8 @@ Unlike traditional visual diffing libraries, RegressionBot is designed for moder
24
24
  - **Matrix Testing**: Test multiple devices and viewports in a single job.
25
25
  - **Auto-Discovery**: Scan sitemaps with glob patterns and limits.
26
26
  - **RegressionBot Summaries**: Plain-English change descriptions for every regression, generated on-demand via the API.
27
+ - **Intent-Aware Verdicts**: Describe what a run is meant to change, and every regression comes back judged as intentional, a bug, or noise.
28
+ - **Scheduled Checks**: Run a saved project unattended, hourly, daily, or weekly — and at a set UTC hour.
27
29
  - **Project-Based Baselines**: Save and reuse test configurations; share visual history across environments.
28
30
  - **Auto-Approval**: Automatically promote screenshots to baselines on jobs that pass your criteria.
29
31
  - **Zero Infrastructure**: No browser maintenance or server provisioning — RegressionBot handles it all.
@@ -34,6 +36,84 @@ Unlike traditional visual diffing libraries, RegressionBot is designed for moder
34
36
  npm install @regressionbot/sdk
35
37
  ```
36
38
 
39
+ ## Migration from 1.x
40
+
41
+ 2.0.0 re-types the SDK against what the API actually returns, taken from the
42
+ orchestrator's implementation rather than from a spec that had drifted in both
43
+ directions. Most of the breakage is caught by the compiler. One is not:
44
+
45
+ ```typescript
46
+ // 1.x — updateProject() returned the raw { message, project } envelope,
47
+ // so every field read off it was undefined at runtime
48
+ const project = await rb.updateProject('site', { schedule: 'daily' });
49
+ project.project.schedule; // this is where the value actually was
50
+
51
+ // 2.0 — the envelope is unwrapped for you
52
+ const project = await rb.updateProject('site', { schedule: 'daily' });
53
+ project.schedule; // 'daily'
54
+ ```
55
+
56
+ The rest surface as type errors:
57
+
58
+ - **`PageResult.diffCount` is gone.** No endpoint ever returned it — both
59
+ `getJobStatus` and `getJobSummary` strip it. `PageResult` gains the fields they
60
+ do return: `status`, `changed`, `isNewBaseline`, `errorMessage`,
61
+ `elementsChanged`, `domAssistSkipReason` and `verdict`. Image URLs are typed
62
+ nullable, which they always were.
63
+ - **`updateProject()` takes `ProjectConfigUpdate`, not `Partial<ProjectConfig>`.**
64
+ The read and write shapes genuinely differ: a gate credential is written as an
65
+ `EnvGate` and read back only as `{ configured: true }`.
66
+ - **`ProjectConfig.orgId` is gone** — the API stopped returning it.
67
+ - **`summaryStatus` can be `FAILED`**, which the old union omitted. `error`,
68
+ `progress`, `results`, `createdAt` and `intentAssessment` are always returned.
69
+ - **`approve()` gains `conflictedUrls`**, returned when another job updated a
70
+ baseline first.
71
+
72
+ New in 2.0 and requiring a current API: `.customCss()` and `.withContext()` on
73
+ the builder, intent-aware verdicts, environment gates, baseline policies, and
74
+ scheduling. `scheduleHourUtc` needs API 2.7.0 or later — on an older API the
75
+ field is accepted and ignored, and the schedule stays anchored to its first run.
76
+
77
+ ### 2.0.1 — never published separately; included in 2.1.0
78
+
79
+ Upgrading from 2.0.0 goes straight to 2.1.0, so read this section as well as the
80
+ next one.
81
+
82
+ - **The builder no longer invents defaults.** It used to send `concurrency: 10`
83
+ on every run, plus an empty `devices` list. The API compares every parameter it
84
+ receives against the saved project config and rejects the run on a mismatch, so
85
+ a project storing any other concurrency — including none, which is every project
86
+ created outside the SDK — failed with *"Params differ from stored config"* and
87
+ there was no way to unset the field. Call `.concurrency(n)` only when you mean
88
+ it; unset now means the API's default of 4, not 10.
89
+ - **CLI: `--concurrency` now rejects a missing or non-numeric value** instead of
90
+ coercing it. `--concurrency` with no value used to run at 1, and a
91
+ non-numeric one used to run at 10; both now fail with a message. Passing a
92
+ valid `1`–`20` is unchanged. Misuse exits **2**, so CI can tell a bad flag
93
+ from a run that worked and found regressions — both of which used to exit 1.
94
+ - **`PageResult.maskUrl` is gone.** It was never on the public contract: both
95
+ endpoints emit it to internal callers only, so code reading it was already
96
+ getting `undefined` and nothing changes at runtime. It was listed here in
97
+ 2.0.0 by mistake. There is no public replacement — the mask is a transparent
98
+ overlay built for the dashboard, and `diffUrl` is the annotated diff image you
99
+ want instead.
100
+ - **`PageResult.changed` added**, with `contentChanged` alongside it. This is the
101
+ rule the API uses to split `regressions` from `matches`. Read it instead of
102
+ testing `diffPercentage === 0` — a text edit that moves no pixels is a
103
+ regression at 0.00%.
104
+
105
+ ### 2.1.0
106
+
107
+ - **`--fail-on unintended`** on the CLI: exit 1 only on changes judged a bug or
108
+ needing review, so expected changes keep CI green. Comes with
109
+ `--change-description`, `--expected-changes`, `--pr-title` and `--commit` to
110
+ describe the run. Default behaviour is unchanged.
111
+ - **`--flag=value` is now accepted** for every flag. Required when the value starts
112
+ with `--`, such as a CSS custom property: `--custom-css="--brand: #fff"` used to
113
+ drop its value silently.
114
+ - The run command now reports capture errors even when there are also regressions.
115
+ They used to be hidden behind the regression branch.
116
+
37
117
  ## Usage
38
118
 
39
119
  ### Basic Example
@@ -84,13 +164,20 @@ const job = await rb
84
164
  // Discovery: Auto-discover up to 20 blog posts
85
165
  .scan('/blog/**', { limit: 20 })
86
166
 
87
- // Concurrency: Max parallel browser instances
88
- .concurrency(10)
167
+ // Concurrency: pages captured in parallel, 1-20. Omit it to take the default of 4 —
168
+ // the load lands on the site being captured, not on RegressionBot.
169
+ .concurrency(8)
89
170
 
90
171
  // Masking: Automatic and manual masking
91
172
  .mask(['.ads', '#modal']) // Manual selectors
92
173
  // Tip: Adding 'data-vr-mask' to your HTML elements masks them automatically!
93
174
 
175
+ // Custom CSS: Injected before every screenshot (max 4096 chars)
176
+ .customCss('#chat-widget { display: none !important; }')
177
+
178
+ // Intent: What this run is testing, so changes can be judged against it
179
+ .withContext({ changeDescription: 'Restyle the pricing table' })
180
+
94
181
  // Execute: Compiles manifest and triggers the API
95
182
  .run();
96
183
 
@@ -112,9 +199,9 @@ const summary = await job.getSummary();
112
199
  if (summary.regressions.length > 0) {
113
200
  console.log(`\n${summary.regressions.length} regressions found:`);
114
201
  for (const regression of summary.regressions) {
115
- if (regression.regressionbotSummary) {
116
- console.log(`\nRegressionBot Summary for ${regression.url}:`);
117
- console.log(`> ${regression.regressionbotSummary}`);
202
+ for (const item of regression.regressionbotSummary ?? []) {
203
+ // `label` is the region letter (A, B, C…), or '' for a whole-page change
204
+ console.log(`[${item.label}] ${item.text}`);
118
205
  }
119
206
  }
120
207
  }
@@ -124,6 +211,107 @@ const aiResult = await job.generateAiSummary();
124
211
  console.log(`Generated summaries for ${aiResult.summaries.length} regressions.`);
125
212
  ```
126
213
 
214
+ ### Reading exactly what changed
215
+
216
+ `changes` is computed by diffing the two documents, so unlike the prose summary it is
217
+ measured rather than described — you can quote it directly.
218
+
219
+ ```typescript
220
+ for (const r of summary.regressions) {
221
+ for (const c of r.changes ?? []) {
222
+ switch (c.type) {
223
+ case 'text-edit':
224
+ console.log(`${c.element}: "${c.before}" → "${c.after}"`);
225
+ break;
226
+ case 'style-only':
227
+ for (const [prop, [from, to]] of Object.entries(c.style ?? {})) {
228
+ console.log(`${c.element}: ${prop} ${from} → ${to}`);
229
+ }
230
+ break;
231
+ default:
232
+ console.log(`${c.type} on ${c.element}`);
233
+ }
234
+ }
235
+ }
236
+ ```
237
+
238
+ Two things to know before you rely on it:
239
+
240
+ - **It can be truncated.** At most 40 changes per page, and fewer if they would exceed the
241
+ API's 8 KB budget for the field. Treat it as the most significant changes, not a
242
+ guaranteed-complete list.
243
+ - **`changes` absent is not "nothing changed".** It means the document comparison could not
244
+ run on that page — check `domAssistSkipReason`.
245
+
246
+ `box` gives you the location in the capture, in CSS pixels from the top-left of the
247
+ full-page image, so it indexes straight into `currentUrl` if you want to crop:
248
+
249
+ ```typescript
250
+ const aboveTheFold = r.changes?.filter(c => c.box && c.box.y < 800) ?? [];
251
+ ```
252
+
253
+ ### Intent-Aware Verdicts
254
+
255
+ Tell RegressionBot what a run is meant to change, and every regression comes back judged
256
+ against that intent — `intentional`, `bug`, `noise`, or `needs_review`.
257
+
258
+ ```typescript
259
+ const job = await rb
260
+ .test(process.env.VERCEL_PREVIEW_URL)
261
+ .forProject('marketing-site-v2')
262
+ .withContext({
263
+ changeDescription: 'Restyle the pricing table',
264
+ gitCommitSha: process.env.GITHUB_SHA,
265
+ prTitle: process.env.PR_TITLE,
266
+ expectedChanges: ['Pricing table background is now dark'],
267
+ scope: ['components/PricingTable.tsx'],
268
+ })
269
+ .run();
270
+
271
+ await job.waitForCompletion(2000, undefined, { waitForSummaries: true });
272
+ const summary = await job.getSummary();
273
+
274
+ // Whole-job roll-up: does everything that changed line up with the stated intent?
275
+ console.log(summary.intentAssessment?.summary);
276
+ console.log(`Bugs: ${summary.intentAssessment?.bugCount}`);
277
+
278
+ // Per-result verdict, plus the per-region verdicts behind it
279
+ for (const r of summary.regressions) {
280
+ console.log(`${r.url}: ${r.verdict?.decision} (${r.verdict?.avgConfidence})`);
281
+ }
282
+
283
+ // Fail CI only on changes that were not intended
284
+ const bugs = summary.regressions.filter(r => r.verdict?.decision === 'bug');
285
+ if (bugs.length > 0) process.exit(1);
286
+ ```
287
+
288
+ `runProject()` takes the same context, so a run from a saved project can be
289
+ intent-aware too:
290
+
291
+ ```typescript
292
+ const job = await rb.runProject('marketing-site-v2', {
293
+ testOrigin: process.env.VERCEL_PREVIEW_URL,
294
+ runContext: { changeDescription: 'Restyle the pricing table' },
295
+ });
296
+ ```
297
+
298
+ ### Hiding Dynamic Content
299
+
300
+ `.mask()` hides elements by selector. `.customCss()` injects arbitrary CSS before each
301
+ screenshot when masking isn't enough — collapsing an animation, pinning a carousel, or
302
+ neutralising a third-party widget. Max 4096 characters.
303
+
304
+ ```typescript
305
+ const job = await rb
306
+ .test('https://preview.myapp.com')
307
+ .forProject('my-app-web')
308
+ .customCss(`
309
+ #chat-widget { display: none !important; }
310
+ * { animation: none !important; transition: none !important; }
311
+ `)
312
+ .run();
313
+ ```
314
+
127
315
  ### Progress Tracking
128
316
 
129
317
  Pass a callback to `waitForCompletion` to receive status updates while the job runs:
@@ -177,6 +365,69 @@ const summary = await job.getSummary();
177
365
  console.log(`Score: ${summary.overallScore}/100`);
178
366
  ```
179
367
 
368
+ ### Environment Gates
369
+
370
+ If an origin sits behind basic auth, a bypass header, or a session cookie, store the
371
+ credential on the project. It is encrypted at rest and never returned — reads report
372
+ only `{ configured: true }`.
373
+
374
+ ```typescript
375
+ await rb.updateProject('marketing-site-v2', {
376
+ testAuth: { basic: { username: 'preview', password: process.env.PREVIEW_PASSWORD! } },
377
+ // or: { headers: { 'x-vercel-protection-bypass': token } }
378
+ // or: { cookies: [{ name: 'session', value: token }] }
379
+ });
380
+
381
+ const project = await rb.getProject('marketing-site-v2');
382
+ console.log(project.testAuth); // { configured: true }
383
+
384
+ // Clear it
385
+ await rb.updateProject('marketing-site-v2', { testAuth: null });
386
+ ```
387
+
388
+ Setting or clearing a gate credential invalidates the stored baselines, since it changes
389
+ what the capture can reach.
390
+
391
+ ### Scheduled Checks
392
+
393
+ A project can run unattended on a fixed cadence. Scheduling requires
394
+ `baselinePolicy: 'rolling'` on a managed project — on the default `approved` policy an
395
+ unapproved change would be re-reported on every subsequent run, so the API rejects the
396
+ combination. Live-vs-live projects store no baseline and are exempt.
397
+
398
+ ```typescript
399
+ // Turn on a daily unattended check
400
+ await rb.updateProject('marketing-site-v2', {
401
+ baselinePolicy: 'rolling',
402
+ schedule: 'daily', // 'hourly' | 'daily' | 'weekly'
403
+ });
404
+
405
+ // Or pin it to a UTC hour — this one runs at 03:00 UTC every day
406
+ await rb.updateProject('marketing-site-v2', {
407
+ baselinePolicy: 'rolling',
408
+ schedule: 'daily',
409
+ scheduleHourUtc: 3, // 0–23, UTC
410
+ });
411
+
412
+ // Turn it back off (this clears scheduleHourUtc too)
413
+ await rb.updateProject('marketing-site-v2', { schedule: null });
414
+
415
+ // Read back when the scheduler last fired — the next run is due one interval after this
416
+ const project = await rb.getProject('marketing-site-v2');
417
+ console.log(project.schedule, project.scheduleHourUtc, project.lastScheduledRunAt);
418
+ ```
419
+
420
+ Without `scheduleHourUtc` the first run starts at the next hourly sweep and the cadence
421
+ anchors to it. With it, the first run waits for that hour as well. Hours only, since the
422
+ scheduler sweeps once an hour, and UTC only. It is rejected on an `hourly` schedule, and
423
+ rejected on a project with no schedule to apply it to. Clearing the schedule clears the
424
+ hour with it, and a missed slot waits for the next one rather than catching up.
425
+
426
+ Setting a schedule, its hour, or a baseline policy does **not** invalidate
427
+ baselines — only changes that affect what a capture looks like (`testOrigin`,
428
+ `baseOrigin`, `sitemapUrl`, `paths`, `scans`, `devices`, `masks`, `customCss`) do that.
429
+ Billing is per comparison, so cost scales with frequency.
430
+
180
431
  ### Reconnecting to an Existing Job
181
432
 
182
433
  If you have a job ID from a previous run (e.g., stored in CI state), you can attach to it without re-running the test:
@@ -210,7 +461,39 @@ Test an entire site using glob patterns.
210
461
  npx @regressionbot/sdk https://example.com --project my-project --scan "/**" --exclude "/admin/**" --concurrency 20
211
462
  ```
212
463
 
213
- #### 3. Job Summary
464
+ Use `--mask` to hide elements by selector, or `--custom-css` to inject CSS before every
465
+ screenshot:
466
+ ```bash
467
+ npx @regressionbot/sdk https://example.com --project my-site --custom-css "#chat-widget { display: none !important; }"
468
+ ```
469
+
470
+ Any flag also accepts `--flag=value`. You need that form when the value itself starts with
471
+ `--`, such as a CSS custom property — otherwise it looks like the next flag and is dropped:
472
+ ```bash
473
+ npx @regressionbot/sdk https://example.com --project my-site --custom-css="--brand-bg: #fff"
474
+ ```
475
+
476
+ #### 3. Failing CI only on unintended changes
477
+
478
+ By default any regression exits 1. Describe what the run is meant to change and
479
+ `--fail-on unintended` will keep the build green for changes that match, failing only on
480
+ what looks like a bug:
481
+
482
+ ```bash
483
+ npx @regressionbot/sdk https://preview.example.com \
484
+ --project my-site \
485
+ --change-description "Restyle the pricing table" \
486
+ --expected-changes "Pricing table is now dark,Buttons are rounded" \
487
+ --commit "$GITHUB_SHA" \
488
+ --fail-on unintended
489
+ ```
490
+
491
+ A change with no verdict always fails the build — if the run carried no intent, or the
492
+ summary pass hasn't finished, nothing judged it, and treating that as expected would turn a
493
+ missing verdict into a silent pass. So `--fail-on unintended` without
494
+ `--change-description` behaves like `--fail-on any`, and warns that it will.
495
+
496
+ #### 4. Job Summary
214
497
  Get detailed results and diff URLs for a completed job.
215
498
  ```bash
216
499
  npx @regressionbot/sdk summary <jobId>
@@ -226,7 +509,7 @@ Use the `--download-full` flag to save baseline, current, and diff images:
226
509
  npx @regressionbot/sdk summary <jobId> --download-full
227
510
  ```
228
511
 
229
- #### 4. Approve Changes
512
+ #### 5. Approve Changes
230
513
  Promote the current screenshots of a job to be the new baselines.
231
514
  ```bash
232
515
  npx @regressionbot/sdk approve <jobId>
package/dist/cli.d.ts CHANGED
@@ -1,2 +1,28 @@
1
1
  #!/usr/bin/env node
2
- export {};
2
+ import { PageResult, RunContext } from './types';
3
+ declare function printRegression(r: PageResult): void;
4
+ type FailOn = 'any' | 'unintended';
5
+ declare function parseFailOn(raw: unknown): FailOn;
6
+ /**
7
+ * Whether one regression should fail the build under `--fail-on unintended`.
8
+ *
9
+ * A regression with no verdict always blocks. Nothing judged it — the run carried no
10
+ * intent, or the summary pass had not finished — and treating unjudged as wanted would
11
+ * turn a missing verdict into a silent pass, which is the one outcome a regression
12
+ * detector must never produce.
13
+ */
14
+ declare function isBlocking(r: PageResult): boolean;
15
+ /**
16
+ * Splits regressions into the ones that fail the build and a count of the ones excused.
17
+ *
18
+ * Pure, and exported, because this is the decision that sets the exit code — the whole
19
+ * point of --fail-on. Testing isBlocking alone proves nothing about whether the caller
20
+ * applies it correctly for each mode.
21
+ */
22
+ declare function selectBlocking(regressions: PageResult[], failOn: FailOn): {
23
+ blocking: PageResult[];
24
+ excused: number;
25
+ };
26
+ declare function buildRunContext(options: any): RunContext | undefined;
27
+ declare function parseArgs(args: string[]): any;
28
+ export { parseArgs, parseFailOn, isBlocking, selectBlocking, buildRunContext, printRegression };