game-harness 1.1.0 → 1.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/MIGRATION.md +8 -1
- package/README.md +25 -4
- package/dist/cjs/bin/visual-battery.js +19 -1
- package/dist/cjs/visual-battery.js +70 -6
- package/dist/esm/bin/visual-battery.js +19 -1
- package/dist/esm/visual-battery.js +71 -7
- package/dist/types/visual-battery.d.cts +8 -5
- package/dist/types/visual-battery.d.ts +8 -5
- package/docs/architecture.md +6 -2
- package/docs/decisions.md +11 -0
- package/docs/guides/visual-battery.md +23 -2
- package/docs/introduction.md +1 -1
- package/package.json +6 -2
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,13 @@ All notable changes are recorded here. Releases follow
|
|
|
4
4
|
[Semantic Versioning](https://semver.org/) and are generated from Conventional
|
|
5
5
|
Commits by release-please.
|
|
6
6
|
|
|
7
|
+
## [1.1.1](https://github.com/jbcom/game-harness/compare/game-harness-v1.1.0...game-harness-v1.1.1) (2026-10-07)
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
### Bug Fixes
|
|
11
|
+
|
|
12
|
+
* **visual-battery:** tolerate rasterization noise while failing real drift ([e2ef323](https://github.com/jbcom/game-harness/commit/e2ef323643dc6b09a4991489f579f068c915d078))
|
|
13
|
+
|
|
7
14
|
## [1.1.0](https://github.com/jbcom/game-harness/compare/game-harness-v1.0.0...game-harness-v1.1.0) (2026-10-07)
|
|
8
15
|
|
|
9
16
|
|
package/MIGRATION.md
CHANGED
|
@@ -64,6 +64,13 @@ New in Game Harness: `activateSilentQaAsync` (`/silent-qa`),
|
|
|
64
64
|
`VisualBatteryCommand` (`/visual-battery` and root) and
|
|
65
65
|
`CHROMIUM_ANTI_THROTTLING_ARGS` (`/chromium`).
|
|
66
66
|
|
|
67
|
+
Modified PNG baselines now use decoded RGBA comparisons instead of byte equality.
|
|
68
|
+
`maxChannelDelta` defaults to 2 and `maxDifferentPixelRatio` to 0, so every
|
|
69
|
+
pixel beyond ±2 is drift. Noise-only files are restored to committed bytes;
|
|
70
|
+
dimensions and new/deleted files still fail. Use `maxChannelDelta: 0` for exact
|
|
71
|
+
decoded pixels. CLI flags are `--max-channel-delta` and
|
|
72
|
+
`--max-different-pixel-ratio`; see the visual-battery guide for their ranges.
|
|
73
|
+
|
|
67
74
|
## 3. Options
|
|
68
75
|
|
|
69
76
|
Every option keeps its name and meaning:
|
|
@@ -79,7 +86,7 @@ Every option keeps its name and meaning:
|
|
|
79
86
|
| `ProductionRuntimeServerOptions` | `command`, `args`, `cwd`, `env`, `readyUrl`, `startupTimeoutMs`, `shutdownTimeoutMs` |
|
|
80
87
|
| `AvailableProductionPortOptions` | `host` |
|
|
81
88
|
| `ActivateSilentQaOptions` | `search`, `queryParameter`, `markerTarget`, `markerAttribute`, `markerValue` |
|
|
82
|
-
| `VisualBatteryOptions` | `ci`, `cwd`, `testCommand`, `baselinesDir`, `baselineProfile`, `isolatedHarnessFiles`, `log`, `error`
|
|
89
|
+
| `VisualBatteryOptions` | `ci`, `cwd`, `testCommand`, `baselinesDir`, `baselineProfile`, `isolatedHarnessFiles`, `maxChannelDelta`, `maxDifferentPixelRatio`, `log`, `error` |
|
|
83
90
|
| `LighthouseAssertionsOverrides` | `staticDistDir`, `url`, `numberOfRuns`, `assertions` |
|
|
84
91
|
| `ChromiumLaunchProfileOptions` | `gpuMode`, `args`, `env` |
|
|
85
92
|
| `VerifyReleaseLadderOptions` | `log`, `error` |
|
package/README.md
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
Release-grade browser QA primitives for TypeScript games. Game Harness turns a
|
|
10
10
|
successful build into evidence: fresh silent browser sessions, deterministic
|
|
11
|
-
device tiers,
|
|
11
|
+
device tiers, pixel-tolerant screenshot gates, production-runtime assertions,
|
|
12
12
|
Lighthouse policy, and an ordered release ladder.
|
|
13
13
|
|
|
14
14
|
It is intentionally a focused library rather than a test framework. Your game
|
|
@@ -24,7 +24,8 @@ and Vitest Browser Mode.
|
|
|
24
24
|
commands use strict-port semantics, and production verification refuses an
|
|
25
25
|
already-reachable readiness URL.
|
|
26
26
|
- **Visual evidence that fails closed.** Screenshot baselines are scoped,
|
|
27
|
-
profile-aware, and checked
|
|
27
|
+
profile-aware, and checked against Git with bounded channel tolerance.
|
|
28
|
+
Every pixel beyond that tolerance fails by default; commands avoid shell
|
|
28
29
|
interpolation.
|
|
29
30
|
- **Real package boundaries.** Framework peers stay optional and isolated to
|
|
30
31
|
subpath exports, with clean ESM, CommonJS, type, CLI, and install smoke tests.
|
|
@@ -377,6 +378,26 @@ server to bind with strict-port semantics so an external race fails closed.
|
|
|
377
378
|
|
|
378
379
|
## Visual battery contract
|
|
379
380
|
|
|
381
|
+
Modified PNGs are decoded and compared pixel by pixel. `maxChannelDelta`
|
|
382
|
+
(default 2, integer 0–255) tolerates that maximum RGBA channel delta.
|
|
383
|
+
`maxDifferentPixelRatio` (default 0, range 0–1) limits the fraction of pixels
|
|
384
|
+
beyond the channel tolerance: any such pixel fails by default. Dimension
|
|
385
|
+
changes, new/deleted files, and unreadable PNGs always count as drift.
|
|
386
|
+
Accepted renders are restored to committed bytes; noise-only files are logged
|
|
387
|
+
as `rasterization noise (N px within ±2)`. The CI dirty-start check remains strict.
|
|
388
|
+
Use `maxChannelDelta: 0` for exact decoded pixel equality. Raising the ratio
|
|
389
|
+
deliberately permits pixels beyond the channel tolerance.
|
|
390
|
+
|
|
391
|
+
```ts
|
|
392
|
+
runVisualBattery('tests/harness', {
|
|
393
|
+
ci: true,
|
|
394
|
+
maxChannelDelta: 2,
|
|
395
|
+
maxDifferentPixelRatio: 0,
|
|
396
|
+
});
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
The CLI exposes `--max-channel-delta 2` and `--max-different-pixel-ratio 0`.
|
|
400
|
+
|
|
380
401
|
Run the default harness directory in update mode, or enforce committed
|
|
381
402
|
baselines in CI:
|
|
382
403
|
|
|
@@ -397,8 +418,8 @@ battery rejects any second `__screenshots__` directory nested elsewhere under
|
|
|
397
418
|
the harness tree; otherwise an apparently green run could leave an important
|
|
398
419
|
screenshot outside the Git diff gate.
|
|
399
420
|
|
|
400
|
-
When
|
|
401
|
-
|
|
421
|
+
When renderers produce genuinely different pixels, keep separate profiles.
|
|
422
|
+
Pass `baselineProfile: 'linux'`; the
|
|
402
423
|
battery compares `__screenshots__/linux/` and exposes the same value to Vite as
|
|
403
424
|
`VITE_VISUAL_BASELINE_PROFILE`. Screenshot helpers should include that optional
|
|
404
425
|
directory in their path:
|
|
@@ -15,11 +15,29 @@ Arguments:
|
|
|
15
15
|
|
|
16
16
|
Options:
|
|
17
17
|
--ci Fail on drift and refuse a dirty starting state
|
|
18
|
+
--max-channel-delta N RGBA channel tolerance, integer 0–255 (default: 2)
|
|
19
|
+
--max-different-pixel-ratio N Allowed fraction beyond tolerance (default: 0)
|
|
18
20
|
-h, --help Show this help
|
|
19
21
|
|
|
20
22
|
The legacy executable name test-harness-visual-battery remains available.`);
|
|
21
23
|
process.exit(0);
|
|
22
24
|
}
|
|
25
|
+
const thresholds = {};
|
|
26
|
+
for (const [flag, key] of [
|
|
27
|
+
['--max-channel-delta', 'maxChannelDelta'],
|
|
28
|
+
['--max-different-pixel-ratio', 'maxDifferentPixelRatio'],
|
|
29
|
+
]) {
|
|
30
|
+
const index = args.indexOf(flag);
|
|
31
|
+
if (index === -1)
|
|
32
|
+
continue;
|
|
33
|
+
const value = args[index + 1];
|
|
34
|
+
if (value === undefined || value.trim() === '' || !Number.isFinite(Number(value))) {
|
|
35
|
+
console.error(`${flag} requires a finite numeric value.`);
|
|
36
|
+
process.exit(2);
|
|
37
|
+
}
|
|
38
|
+
thresholds[key] = Number(value);
|
|
39
|
+
args.splice(index, 2);
|
|
40
|
+
}
|
|
23
41
|
const unknownFlags = args.filter((argument) => argument.startsWith('-') && argument !== '--ci');
|
|
24
42
|
if (unknownFlags.length > 0) {
|
|
25
43
|
console.error(`Unknown option(s): ${unknownFlags.join(', ')}. Use --help for usage.`);
|
|
@@ -33,7 +51,7 @@ if (positionalArgs.length > 1) {
|
|
|
33
51
|
const ci = args.includes('--ci');
|
|
34
52
|
const harnessDirArg = positionalArgs[0] ?? 'tests/harness';
|
|
35
53
|
try {
|
|
36
|
-
(0, visual_battery_js_1.runVisualBattery)(harnessDirArg, { ci });
|
|
54
|
+
(0, visual_battery_js_1.runVisualBattery)(harnessDirArg, { ci, ...thresholds });
|
|
37
55
|
}
|
|
38
56
|
catch (err) {
|
|
39
57
|
if (err instanceof visual_battery_js_1.VisualBatteryError) {
|
|
@@ -9,6 +9,8 @@ const node_child_process_1 = require("node:child_process");
|
|
|
9
9
|
const node_fs_1 = require("node:fs");
|
|
10
10
|
const node_path_1 = require("node:path");
|
|
11
11
|
const cross_spawn_1 = __importDefault(require("cross-spawn"));
|
|
12
|
+
const pngjs_1 = require("pngjs");
|
|
13
|
+
const which_1 = __importDefault(require("which"));
|
|
12
14
|
/**
|
|
13
15
|
* Thrown by every `runVisualBattery()` failure path — a missing harness dir,
|
|
14
16
|
* no discovered `.browser.test.ts(x)` files, a misplaced `__screenshots__`
|
|
@@ -59,9 +61,9 @@ function canonicalizePath(target) {
|
|
|
59
61
|
*
|
|
60
62
|
* Runs every `.browser.test.tsx` harness file under `harnessDir`, then
|
|
61
63
|
* diffs the resulting `__screenshots__/*.png` baselines against what's
|
|
62
|
-
* committed in git.
|
|
63
|
-
*
|
|
64
|
-
*
|
|
64
|
+
* committed in git. Modified PNGs tolerate RGBA channel deltas up to 2 by
|
|
65
|
+
* default; any pixel beyond that precise tolerance is drift. Noise-only
|
|
66
|
+
* renders are restored to committed bytes. New/deleted baselines always drift.
|
|
65
67
|
*
|
|
66
68
|
* - Update mode (`ci: false`, the default): runs the harnesses, lets new
|
|
67
69
|
* baselines land on disk, reports what changed so a human can review
|
|
@@ -76,11 +78,19 @@ function canonicalizePath(target) {
|
|
|
76
78
|
* exits the process.
|
|
77
79
|
*/
|
|
78
80
|
function runVisualBattery(harnessDir, options = {}) {
|
|
79
|
-
const { ci = false, cwd = process.cwd(), testCommand = 'pnpm test:browser', baselineProfile, isolatedHarnessFiles = [], log = defaultLog, error = defaultError, } = options;
|
|
81
|
+
const { ci = false, cwd = process.cwd(), testCommand = 'pnpm test:browser', baselineProfile, isolatedHarnessFiles = [], log = defaultLog, error = defaultError, maxChannelDelta = 2, maxDifferentPixelRatio = 0, } = options;
|
|
80
82
|
const die = (msg) => {
|
|
81
83
|
error(msg);
|
|
82
84
|
throw new VisualBatteryError(msg);
|
|
83
85
|
};
|
|
86
|
+
if (!Number.isInteger(maxChannelDelta) || maxChannelDelta < 0 || maxChannelDelta > 255) {
|
|
87
|
+
die('maxChannelDelta must be an integer between 0 and 255');
|
|
88
|
+
}
|
|
89
|
+
if (!Number.isFinite(maxDifferentPixelRatio) ||
|
|
90
|
+
maxDifferentPixelRatio < 0 ||
|
|
91
|
+
maxDifferentPixelRatio > 1) {
|
|
92
|
+
die('maxDifferentPixelRatio must be between 0 and 1');
|
|
93
|
+
}
|
|
84
94
|
const resolvedCwd = (0, node_path_1.resolve)(cwd);
|
|
85
95
|
const canonicalCwd = (() => {
|
|
86
96
|
try {
|
|
@@ -141,10 +151,20 @@ function runVisualBattery(harnessDir, options = {}) {
|
|
|
141
151
|
log(`running ${harnessFiles.length} harness file(s):`);
|
|
142
152
|
for (const f of harnessFiles)
|
|
143
153
|
log(` - ${f}`);
|
|
154
|
+
// Resolve the consumer's Git installation once, including PATHEXT on Windows.
|
|
155
|
+
// All Git subprocesses use the resulting absolute executable path.
|
|
156
|
+
const gitExecutable = (() => {
|
|
157
|
+
try {
|
|
158
|
+
return (0, node_path_1.resolve)(which_1.default.sync('git'));
|
|
159
|
+
}
|
|
160
|
+
catch (err) {
|
|
161
|
+
return die(`git executable not found: ${err}`);
|
|
162
|
+
}
|
|
163
|
+
})();
|
|
144
164
|
if (ci) {
|
|
145
165
|
let beforeStatus = '';
|
|
146
166
|
try {
|
|
147
|
-
beforeStatus = (0, node_child_process_1.execFileSync)(
|
|
167
|
+
beforeStatus = (0, node_child_process_1.execFileSync)(gitExecutable, ['status', '--porcelain', '--', relativeBaselinesDir], {
|
|
148
168
|
cwd,
|
|
149
169
|
encoding: 'utf-8',
|
|
150
170
|
});
|
|
@@ -214,7 +234,7 @@ function runVisualBattery(harnessDir, options = {}) {
|
|
|
214
234
|
log(`baseline screenshots produced: ${baselineCount}`);
|
|
215
235
|
let afterStatus = '';
|
|
216
236
|
try {
|
|
217
|
-
afterStatus = (0, node_child_process_1.execFileSync)(
|
|
237
|
+
afterStatus = (0, node_child_process_1.execFileSync)(gitExecutable, ['status', '--porcelain', '-z', '--untracked-files=all', '--', relativeBaselinesDir], {
|
|
218
238
|
cwd,
|
|
219
239
|
encoding: 'utf-8',
|
|
220
240
|
});
|
|
@@ -222,6 +242,50 @@ function runVisualBattery(harnessDir, options = {}) {
|
|
|
222
242
|
catch (err) {
|
|
223
243
|
die(`git status failed after harness run: ${err}`);
|
|
224
244
|
}
|
|
245
|
+
const drift = [];
|
|
246
|
+
const entries = afterStatus.split('\0').filter(Boolean);
|
|
247
|
+
for (let index = 0; index < entries.length; index += 1) {
|
|
248
|
+
const entry = entries[index];
|
|
249
|
+
const status = entry.slice(0, 2);
|
|
250
|
+
const path = entry.slice(3);
|
|
251
|
+
// Renames/copies have a second NUL-delimited path; neither is noise.
|
|
252
|
+
if (/[RC]/.test(status))
|
|
253
|
+
index += 1;
|
|
254
|
+
if (status !== ' M' || !path.endsWith('.png')) {
|
|
255
|
+
drift.push(entry);
|
|
256
|
+
continue;
|
|
257
|
+
}
|
|
258
|
+
try {
|
|
259
|
+
const committed = pngjs_1.PNG.sync.read((0, node_child_process_1.execFileSync)(gitExecutable, ['show', `HEAD:./${path}`], { cwd }));
|
|
260
|
+
const current = pngjs_1.PNG.sync.read((0, node_fs_1.readFileSync)((0, node_path_1.resolve)(cwd, path)));
|
|
261
|
+
if (committed.width !== current.width || committed.height !== current.height) {
|
|
262
|
+
drift.push(entry);
|
|
263
|
+
continue;
|
|
264
|
+
}
|
|
265
|
+
let changed = 0;
|
|
266
|
+
let beyond = 0;
|
|
267
|
+
for (let offset = 0; offset < current.data.length; offset += 4) {
|
|
268
|
+
const delta = Math.max(Math.abs(current.data[offset] - committed.data[offset]), Math.abs(current.data[offset + 1] - committed.data[offset + 1]), Math.abs(current.data[offset + 2] - committed.data[offset + 2]), Math.abs(current.data[offset + 3] - committed.data[offset + 3]));
|
|
269
|
+
if (delta > 0)
|
|
270
|
+
changed += 1;
|
|
271
|
+
if (delta > maxChannelDelta)
|
|
272
|
+
beyond += 1;
|
|
273
|
+
}
|
|
274
|
+
if (beyond / (current.width * current.height) > maxDifferentPixelRatio) {
|
|
275
|
+
drift.push(entry);
|
|
276
|
+
continue;
|
|
277
|
+
}
|
|
278
|
+
(0, node_child_process_1.execFileSync)(gitExecutable, ['checkout', '--', path], { cwd });
|
|
279
|
+
log(beyond === 0
|
|
280
|
+
? `${path}: rasterization noise (${changed} px within ±${maxChannelDelta})`
|
|
281
|
+
: `${path}: accepted pixel tolerance (${beyond} px beyond ±${maxChannelDelta})`);
|
|
282
|
+
}
|
|
283
|
+
catch {
|
|
284
|
+
// Missing/unreadable committed PNGs or failed restoration must fail closed.
|
|
285
|
+
drift.push(entry);
|
|
286
|
+
}
|
|
287
|
+
}
|
|
288
|
+
afterStatus = drift.join('\n');
|
|
225
289
|
if (afterStatus.trim().length === 0) {
|
|
226
290
|
log('baselines clean — no visual drift detected.');
|
|
227
291
|
return;
|
|
@@ -13,11 +13,29 @@ Arguments:
|
|
|
13
13
|
|
|
14
14
|
Options:
|
|
15
15
|
--ci Fail on drift and refuse a dirty starting state
|
|
16
|
+
--max-channel-delta N RGBA channel tolerance, integer 0–255 (default: 2)
|
|
17
|
+
--max-different-pixel-ratio N Allowed fraction beyond tolerance (default: 0)
|
|
16
18
|
-h, --help Show this help
|
|
17
19
|
|
|
18
20
|
The legacy executable name test-harness-visual-battery remains available.`);
|
|
19
21
|
process.exit(0);
|
|
20
22
|
}
|
|
23
|
+
const thresholds = {};
|
|
24
|
+
for (const [flag, key] of [
|
|
25
|
+
['--max-channel-delta', 'maxChannelDelta'],
|
|
26
|
+
['--max-different-pixel-ratio', 'maxDifferentPixelRatio'],
|
|
27
|
+
]) {
|
|
28
|
+
const index = args.indexOf(flag);
|
|
29
|
+
if (index === -1)
|
|
30
|
+
continue;
|
|
31
|
+
const value = args[index + 1];
|
|
32
|
+
if (value === undefined || value.trim() === '' || !Number.isFinite(Number(value))) {
|
|
33
|
+
console.error(`${flag} requires a finite numeric value.`);
|
|
34
|
+
process.exit(2);
|
|
35
|
+
}
|
|
36
|
+
thresholds[key] = Number(value);
|
|
37
|
+
args.splice(index, 2);
|
|
38
|
+
}
|
|
21
39
|
const unknownFlags = args.filter((argument) => argument.startsWith('-') && argument !== '--ci');
|
|
22
40
|
if (unknownFlags.length > 0) {
|
|
23
41
|
console.error(`Unknown option(s): ${unknownFlags.join(', ')}. Use --help for usage.`);
|
|
@@ -31,7 +49,7 @@ if (positionalArgs.length > 1) {
|
|
|
31
49
|
const ci = args.includes('--ci');
|
|
32
50
|
const harnessDirArg = positionalArgs[0] ?? 'tests/harness';
|
|
33
51
|
try {
|
|
34
|
-
runVisualBattery(harnessDirArg, { ci });
|
|
52
|
+
runVisualBattery(harnessDirArg, { ci, ...thresholds });
|
|
35
53
|
}
|
|
36
54
|
catch (err) {
|
|
37
55
|
if (err instanceof VisualBatteryError) {
|
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
import { execFileSync } from 'node:child_process';
|
|
2
|
-
import { existsSync, readdirSync, realpathSync, statSync } from 'node:fs';
|
|
2
|
+
import { existsSync, readFileSync, readdirSync, realpathSync, statSync } from 'node:fs';
|
|
3
3
|
import { basename, dirname, isAbsolute, relative, resolve, sep } from 'node:path';
|
|
4
4
|
import spawn from 'cross-spawn';
|
|
5
|
+
import { PNG } from 'pngjs';
|
|
6
|
+
import which from 'which';
|
|
5
7
|
/**
|
|
6
8
|
* Thrown by every `runVisualBattery()` failure path — a missing harness dir,
|
|
7
9
|
* no discovered `.browser.test.ts(x)` files, a misplaced `__screenshots__`
|
|
@@ -51,9 +53,9 @@ function canonicalizePath(target) {
|
|
|
51
53
|
*
|
|
52
54
|
* Runs every `.browser.test.tsx` harness file under `harnessDir`, then
|
|
53
55
|
* diffs the resulting `__screenshots__/*.png` baselines against what's
|
|
54
|
-
* committed in git.
|
|
55
|
-
*
|
|
56
|
-
*
|
|
56
|
+
* committed in git. Modified PNGs tolerate RGBA channel deltas up to 2 by
|
|
57
|
+
* default; any pixel beyond that precise tolerance is drift. Noise-only
|
|
58
|
+
* renders are restored to committed bytes. New/deleted baselines always drift.
|
|
57
59
|
*
|
|
58
60
|
* - Update mode (`ci: false`, the default): runs the harnesses, lets new
|
|
59
61
|
* baselines land on disk, reports what changed so a human can review
|
|
@@ -68,11 +70,19 @@ function canonicalizePath(target) {
|
|
|
68
70
|
* exits the process.
|
|
69
71
|
*/
|
|
70
72
|
export function runVisualBattery(harnessDir, options = {}) {
|
|
71
|
-
const { ci = false, cwd = process.cwd(), testCommand = 'pnpm test:browser', baselineProfile, isolatedHarnessFiles = [], log = defaultLog, error = defaultError, } = options;
|
|
73
|
+
const { ci = false, cwd = process.cwd(), testCommand = 'pnpm test:browser', baselineProfile, isolatedHarnessFiles = [], log = defaultLog, error = defaultError, maxChannelDelta = 2, maxDifferentPixelRatio = 0, } = options;
|
|
72
74
|
const die = (msg) => {
|
|
73
75
|
error(msg);
|
|
74
76
|
throw new VisualBatteryError(msg);
|
|
75
77
|
};
|
|
78
|
+
if (!Number.isInteger(maxChannelDelta) || maxChannelDelta < 0 || maxChannelDelta > 255) {
|
|
79
|
+
die('maxChannelDelta must be an integer between 0 and 255');
|
|
80
|
+
}
|
|
81
|
+
if (!Number.isFinite(maxDifferentPixelRatio) ||
|
|
82
|
+
maxDifferentPixelRatio < 0 ||
|
|
83
|
+
maxDifferentPixelRatio > 1) {
|
|
84
|
+
die('maxDifferentPixelRatio must be between 0 and 1');
|
|
85
|
+
}
|
|
76
86
|
const resolvedCwd = resolve(cwd);
|
|
77
87
|
const canonicalCwd = (() => {
|
|
78
88
|
try {
|
|
@@ -133,10 +143,20 @@ export function runVisualBattery(harnessDir, options = {}) {
|
|
|
133
143
|
log(`running ${harnessFiles.length} harness file(s):`);
|
|
134
144
|
for (const f of harnessFiles)
|
|
135
145
|
log(` - ${f}`);
|
|
146
|
+
// Resolve the consumer's Git installation once, including PATHEXT on Windows.
|
|
147
|
+
// All Git subprocesses use the resulting absolute executable path.
|
|
148
|
+
const gitExecutable = (() => {
|
|
149
|
+
try {
|
|
150
|
+
return resolve(which.sync('git'));
|
|
151
|
+
}
|
|
152
|
+
catch (err) {
|
|
153
|
+
return die(`git executable not found: ${err}`);
|
|
154
|
+
}
|
|
155
|
+
})();
|
|
136
156
|
if (ci) {
|
|
137
157
|
let beforeStatus = '';
|
|
138
158
|
try {
|
|
139
|
-
beforeStatus = execFileSync(
|
|
159
|
+
beforeStatus = execFileSync(gitExecutable, ['status', '--porcelain', '--', relativeBaselinesDir], {
|
|
140
160
|
cwd,
|
|
141
161
|
encoding: 'utf-8',
|
|
142
162
|
});
|
|
@@ -206,7 +226,7 @@ export function runVisualBattery(harnessDir, options = {}) {
|
|
|
206
226
|
log(`baseline screenshots produced: ${baselineCount}`);
|
|
207
227
|
let afterStatus = '';
|
|
208
228
|
try {
|
|
209
|
-
afterStatus = execFileSync(
|
|
229
|
+
afterStatus = execFileSync(gitExecutable, ['status', '--porcelain', '-z', '--untracked-files=all', '--', relativeBaselinesDir], {
|
|
210
230
|
cwd,
|
|
211
231
|
encoding: 'utf-8',
|
|
212
232
|
});
|
|
@@ -214,6 +234,50 @@ export function runVisualBattery(harnessDir, options = {}) {
|
|
|
214
234
|
catch (err) {
|
|
215
235
|
die(`git status failed after harness run: ${err}`);
|
|
216
236
|
}
|
|
237
|
+
const drift = [];
|
|
238
|
+
const entries = afterStatus.split('\0').filter(Boolean);
|
|
239
|
+
for (let index = 0; index < entries.length; index += 1) {
|
|
240
|
+
const entry = entries[index];
|
|
241
|
+
const status = entry.slice(0, 2);
|
|
242
|
+
const path = entry.slice(3);
|
|
243
|
+
// Renames/copies have a second NUL-delimited path; neither is noise.
|
|
244
|
+
if (/[RC]/.test(status))
|
|
245
|
+
index += 1;
|
|
246
|
+
if (status !== ' M' || !path.endsWith('.png')) {
|
|
247
|
+
drift.push(entry);
|
|
248
|
+
continue;
|
|
249
|
+
}
|
|
250
|
+
try {
|
|
251
|
+
const committed = PNG.sync.read(execFileSync(gitExecutable, ['show', `HEAD:./${path}`], { cwd }));
|
|
252
|
+
const current = PNG.sync.read(readFileSync(resolve(cwd, path)));
|
|
253
|
+
if (committed.width !== current.width || committed.height !== current.height) {
|
|
254
|
+
drift.push(entry);
|
|
255
|
+
continue;
|
|
256
|
+
}
|
|
257
|
+
let changed = 0;
|
|
258
|
+
let beyond = 0;
|
|
259
|
+
for (let offset = 0; offset < current.data.length; offset += 4) {
|
|
260
|
+
const delta = Math.max(Math.abs(current.data[offset] - committed.data[offset]), Math.abs(current.data[offset + 1] - committed.data[offset + 1]), Math.abs(current.data[offset + 2] - committed.data[offset + 2]), Math.abs(current.data[offset + 3] - committed.data[offset + 3]));
|
|
261
|
+
if (delta > 0)
|
|
262
|
+
changed += 1;
|
|
263
|
+
if (delta > maxChannelDelta)
|
|
264
|
+
beyond += 1;
|
|
265
|
+
}
|
|
266
|
+
if (beyond / (current.width * current.height) > maxDifferentPixelRatio) {
|
|
267
|
+
drift.push(entry);
|
|
268
|
+
continue;
|
|
269
|
+
}
|
|
270
|
+
execFileSync(gitExecutable, ['checkout', '--', path], { cwd });
|
|
271
|
+
log(beyond === 0
|
|
272
|
+
? `${path}: rasterization noise (${changed} px within ±${maxChannelDelta})`
|
|
273
|
+
: `${path}: accepted pixel tolerance (${beyond} px beyond ±${maxChannelDelta})`);
|
|
274
|
+
}
|
|
275
|
+
catch {
|
|
276
|
+
// Missing/unreadable committed PNGs or failed restoration must fail closed.
|
|
277
|
+
drift.push(entry);
|
|
278
|
+
}
|
|
279
|
+
}
|
|
280
|
+
afterStatus = drift.join('\n');
|
|
217
281
|
if (afterStatus.trim().length === 0) {
|
|
218
282
|
log('baselines clean — no visual drift detected.');
|
|
219
283
|
return;
|
|
@@ -5,6 +5,10 @@ export interface VisualBatteryCommand {
|
|
|
5
5
|
args?: readonly string[];
|
|
6
6
|
}
|
|
7
7
|
export interface VisualBatteryOptions {
|
|
8
|
+
/** Maximum absolute RGBA channel delta per pixel (0–255 integer). Defaults to 2. */
|
|
9
|
+
maxChannelDelta?: number;
|
|
10
|
+
/** Allowed fraction of pixels beyond the channel tolerance (0–1). Defaults to 0. */
|
|
11
|
+
maxDifferentPixelRatio?: number;
|
|
8
12
|
/** CI mode: refuses to run with a dirty baseline dir, fails on drift instead of updating. */
|
|
9
13
|
ci?: boolean;
|
|
10
14
|
/** Repo root the git commands run relative to. Defaults to `process.cwd()`. */
|
|
@@ -21,8 +25,7 @@ export interface VisualBatteryOptions {
|
|
|
21
25
|
/**
|
|
22
26
|
* Optional platform/profile directory below the baseline root (for example
|
|
23
27
|
* `linux`). The profile is also exposed to Vite browser tests as
|
|
24
|
-
* `VITE_VISUAL_BASELINE_PROFILE`,
|
|
25
|
-
* strict on renderers that cannot produce identical PNGs.
|
|
28
|
+
* `VITE_VISUAL_BASELINE_PROFILE`, keeping distinct renderer baselines separate.
|
|
26
29
|
*/
|
|
27
30
|
baselineProfile?: string;
|
|
28
31
|
/**
|
|
@@ -50,9 +53,9 @@ export declare class VisualBatteryError extends Error {
|
|
|
50
53
|
*
|
|
51
54
|
* Runs every `.browser.test.tsx` harness file under `harnessDir`, then
|
|
52
55
|
* diffs the resulting `__screenshots__/*.png` baselines against what's
|
|
53
|
-
* committed in git.
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
+
* committed in git. Modified PNGs tolerate RGBA channel deltas up to 2 by
|
|
57
|
+
* default; any pixel beyond that precise tolerance is drift. Noise-only
|
|
58
|
+
* renders are restored to committed bytes. New/deleted baselines always drift.
|
|
56
59
|
*
|
|
57
60
|
* - Update mode (`ci: false`, the default): runs the harnesses, lets new
|
|
58
61
|
* baselines land on disk, reports what changed so a human can review
|
|
@@ -5,6 +5,10 @@ export interface VisualBatteryCommand {
|
|
|
5
5
|
args?: readonly string[];
|
|
6
6
|
}
|
|
7
7
|
export interface VisualBatteryOptions {
|
|
8
|
+
/** Maximum absolute RGBA channel delta per pixel (0–255 integer). Defaults to 2. */
|
|
9
|
+
maxChannelDelta?: number;
|
|
10
|
+
/** Allowed fraction of pixels beyond the channel tolerance (0–1). Defaults to 0. */
|
|
11
|
+
maxDifferentPixelRatio?: number;
|
|
8
12
|
/** CI mode: refuses to run with a dirty baseline dir, fails on drift instead of updating. */
|
|
9
13
|
ci?: boolean;
|
|
10
14
|
/** Repo root the git commands run relative to. Defaults to `process.cwd()`. */
|
|
@@ -21,8 +25,7 @@ export interface VisualBatteryOptions {
|
|
|
21
25
|
/**
|
|
22
26
|
* Optional platform/profile directory below the baseline root (for example
|
|
23
27
|
* `linux`). The profile is also exposed to Vite browser tests as
|
|
24
|
-
* `VITE_VISUAL_BASELINE_PROFILE`,
|
|
25
|
-
* strict on renderers that cannot produce identical PNGs.
|
|
28
|
+
* `VITE_VISUAL_BASELINE_PROFILE`, keeping distinct renderer baselines separate.
|
|
26
29
|
*/
|
|
27
30
|
baselineProfile?: string;
|
|
28
31
|
/**
|
|
@@ -50,9 +53,9 @@ export declare class VisualBatteryError extends Error {
|
|
|
50
53
|
*
|
|
51
54
|
* Runs every `.browser.test.tsx` harness file under `harnessDir`, then
|
|
52
55
|
* diffs the resulting `__screenshots__/*.png` baselines against what's
|
|
53
|
-
* committed in git.
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
+
* committed in git. Modified PNGs tolerate RGBA channel deltas up to 2 by
|
|
57
|
+
* default; any pixel beyond that precise tolerance is drift. Noise-only
|
|
58
|
+
* renders are restored to committed bytes. New/deleted baselines always drift.
|
|
56
59
|
*
|
|
57
60
|
* - Update mode (`ci: false`, the default): runs the harnesses, lets new
|
|
58
61
|
* baselines land on disk, reports what changed so a human can review
|
package/docs/architecture.md
CHANGED
|
@@ -52,8 +52,12 @@ runs selected harnesses in direct child processes, requires at least one PNG,
|
|
|
52
52
|
and asks Git for scoped status with argument arrays rather than shell commands.
|
|
53
53
|
|
|
54
54
|
Update mode reports reviewed changes. CI mode also refuses a dirty starting
|
|
55
|
-
state and fails on any resulting drift.
|
|
56
|
-
|
|
55
|
+
state and fails on any resulting drift. Modified PNGs are decoded with pngjs;
|
|
56
|
+
dimensions must match, and every pixel whose maximum RGBA channel delta exceeds
|
|
57
|
+
`maxChannelDelta` (default 2) counts toward `maxDifferentPixelRatio` (default 0).
|
|
58
|
+
Accepted renders are restored to committed bytes. New/deleted or unreadable
|
|
59
|
+
baselines remain drift. Renderer-specific output belongs in a separate profile
|
|
60
|
+
when the difference exceeds rasterization noise.
|
|
57
61
|
|
|
58
62
|
## Configuration philosophy
|
|
59
63
|
|
package/docs/decisions.md
CHANGED
|
@@ -6,6 +6,17 @@ description: Why the package is shaped the way it is, and how a predecessor harn
|
|
|
6
6
|
Each entry records a decision, the reason for it, and what it means for a
|
|
7
7
|
consumer. Newest first.
|
|
8
8
|
|
|
9
|
+
## Pixel tolerance for screenshot baselines
|
|
10
|
+
|
|
11
|
+
Byte equality is not stable across renders, even on the same machine. Decode
|
|
12
|
+
modified PNGs with pngjs and compare the maximum absolute RGBA channel delta
|
|
13
|
+
per pixel. `maxChannelDelta` defaults to 2 (out of 255), and
|
|
14
|
+
`maxDifferentPixelRatio` defaults to 0: any pixel beyond that tolerance is
|
|
15
|
+
drift. Dimension changes, new/deleted files, and unreadable PNGs remain drift.
|
|
16
|
+
Noise-only files are restored to committed bytes, keeping the Git diff clean.
|
|
17
|
+
Both thresholds are configurable; raising the ratio explicitly permits real
|
|
18
|
+
changes and should be a deliberate consumer decision.
|
|
19
|
+
|
|
9
20
|
## Converging a predecessor `test-harness` package
|
|
10
21
|
|
|
11
22
|
This entry compares the supported migration paths export by export so that
|
|
@@ -3,6 +3,27 @@ title: Visual battery
|
|
|
3
3
|
description: Deterministic, fail-closed screenshot baseline orchestration.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
+
Modified PNGs are decoded and compared against `HEAD` pixel by pixel.
|
|
7
|
+
`maxChannelDelta` defaults to 2 (integer 0–255): a pixel differs only when
|
|
8
|
+
its maximum absolute RGBA channel delta exceeds this tolerance.
|
|
9
|
+
`maxDifferentPixelRatio` defaults to 0 (range 0–1): any pixel beyond the
|
|
10
|
+
channel tolerance is drift. Dimension changes, new/deleted baselines, and
|
|
11
|
+
unreadable PNGs always count as drift. Accepted renders are restored to
|
|
12
|
+
committed bytes; noise-only files log `rasterization noise (N px within ±2)`.
|
|
13
|
+
The CI dirty-start check still rejects all uncommitted baseline changes.
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
runVisualBattery('tests/harness', {
|
|
17
|
+
ci: true,
|
|
18
|
+
maxChannelDelta: 2,
|
|
19
|
+
maxDifferentPixelRatio: 0,
|
|
20
|
+
});
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
CLI equivalents: `--max-channel-delta 2 --max-different-pixel-ratio 0`.
|
|
24
|
+
Use a channel delta of 0 for exact decoded pixels. Raising the ratio explicitly
|
|
25
|
+
permits some pixels beyond the tolerance; do so deliberately.
|
|
26
|
+
|
|
6
27
|
Run the default harness directory in update mode, or enforce committed
|
|
7
28
|
baselines in CI:
|
|
8
29
|
|
|
@@ -24,8 +45,8 @@ The battery rejects any second `__screenshots__` directory nested elsewhere
|
|
|
24
45
|
under the harness tree; otherwise an apparently green run could leave an
|
|
25
46
|
important screenshot outside the Git diff gate.
|
|
26
47
|
|
|
27
|
-
When
|
|
28
|
-
|
|
48
|
+
When renderers produce genuinely different pixels, keep separate profiles.
|
|
49
|
+
Pass `baselineProfile: 'linux'`; the
|
|
29
50
|
battery compares `__screenshots__/linux/` and exposes the same value to Vite
|
|
30
51
|
as `VITE_VISUAL_BASELINE_PROFILE`. Screenshot helpers should include that
|
|
31
52
|
optional directory in their path:
|
package/docs/introduction.md
CHANGED
|
@@ -4,7 +4,7 @@ description: Release-grade browser QA primitives for TypeScript games.
|
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
Game Harness turns a successful build into evidence: silent browser sessions,
|
|
7
|
-
deterministic device tiers,
|
|
7
|
+
deterministic device tiers, pixel-tolerant screenshot gates, production-runtime proof,
|
|
8
8
|
and Lighthouse gates.
|
|
9
9
|
|
|
10
10
|
It is intentionally a focused library rather than a test framework. Your game
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "game-harness",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.1",
|
|
4
4
|
"description": "Release-grade browser QA primitives for TypeScript games: silent Playwright/Vitest sessions, deterministic screenshots, runtime proof, and Lighthouse gates.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -122,7 +122,9 @@
|
|
|
122
122
|
"types": "./dist/types/index.d.ts",
|
|
123
123
|
"dependencies": {
|
|
124
124
|
"cross-spawn": "7.0.6",
|
|
125
|
-
"get-port": "7.2.0"
|
|
125
|
+
"get-port": "7.2.0",
|
|
126
|
+
"pngjs": "^7.0.0",
|
|
127
|
+
"which": "^5.0.0"
|
|
126
128
|
},
|
|
127
129
|
"peerDependencies": {
|
|
128
130
|
"@playwright/test": ">=1.62.1 <2",
|
|
@@ -145,6 +147,8 @@
|
|
|
145
147
|
"@playwright/test": "1.62.1",
|
|
146
148
|
"@types/cross-spawn": "6.0.6",
|
|
147
149
|
"@types/node": "24.13.3",
|
|
150
|
+
"@types/pngjs": "^6.0.5",
|
|
151
|
+
"@types/which": "^3.0.4",
|
|
148
152
|
"@vitest/browser-playwright": "5.0.3",
|
|
149
153
|
"@vitest/coverage-v8": "5.0.3",
|
|
150
154
|
"oxlint": "1.79.0",
|