@roku-sdk/device-test-client 0.0.0-stage → 2.19.0

package/docs/screenshot-assertions.md

@@ -0,0 +1,316 @@
+# Screenshot assertions
+
+Screenshot assertions commit a rendered frame to disk as the ground
+truth for a UI state and verify future runs stay within a configurable
+mismatch budget. They share the baseline layout, resolution routing,
+and `UPDATE_BASELINES=1` workflow with
+[DOM assertions](dom-assertions.md), but the comparison engine is
+[pixelmatch](https://github.com/mapbox/pixelmatch) and failures write
+an annotated diff PNG to `test-results/diffs/`.
+
+- [Core types](#core-types)
+- [Stable capture](#stable-capture)
+- `[expectScreenshot().toMatchBaseline(...)](#expectscreenshottomatchbaseline)`
+- [Thresholds — aggregate vs per-pixel](#thresholds--aggregate-vs-per-pixel)
+- [Masks](#masks)
+- [Alignment tolerance](#alignment-tolerance)
+- [Diff PNGs](#diff-pngs)
+- [Diff-root override](#diff-root-override)
+- `[UPDATE_BASELINES=1` refresh](#update_baselines1-refresh)
+- [Errors thrown](#errors-thrown)
+
+## Core types
+
+```ts
+interface ScreenshotMatchOptions {
+    /** Mismatch budget as a fraction of UNMASKED pixels. Default 0.01 (1%). */
+    threshold?: number;
+    /** Regions zeroed out in BOTH images before comparison. */
+    mask?: MaskRegion[];
+    /** Whole-pixel slack for device-dependent re-centering. Default none. */
+    alignTolerance?: AlignTolerance;
+}
+
+interface MaskRegion {
+    x: number;
+    y: number;
+    width: number;
+    height: number;
+}
+
+interface AlignTolerance {
+    /** Max horizontal translation, 0..ALIGN_TOLERANCE_MAX_PX. Default 0. */
+    x?: number;
+    /** Max vertical translation, 0..ALIGN_TOLERANCE_MAX_PX. Default 0. */
+    y?: number;
+}
+
+interface CaptureStableOptions {
+    /** Overall wait budget (ms). */
+    timeout: number;
+    /** Minimum gap between captures (ms). Default 500. */
+    interval?: number;
+    /** Regions ignored when deciding "two frames are identical". */
+    mask?: MaskRegion[];
+}
+```
+
+## Stable capture
+
+Raw `client.captureScreenshot()` returns whatever is on screen at that
+instant — typically a mid-animation frame. For baseline comparison you
+want a **settled** frame: that's `captureStableScreenshot`.
+
+```ts
+captureStableScreenshot(opts: CaptureStableOptions): Promise<Buffer>;
+```
+
+Polls until two successive captures are pixel-identical outside `mask`
+regions and returns the newer PNG. `assertAppAlive` runs before each
+capture so a crash during polling surfaces as `AppCrashedError`
+instead of a timeout.
+
+**Example.**
+
+```ts
+const png = await client.captureStableScreenshot({
+    timeout: 5_000,
+    mask: [{ x: 0, y: 0, width: 1920, height: 80 }]  // hide the clock/status bar
+});
+```
+
+Throws `ScreenshotNotStableError` with `iterations`, `lastDiffCount`,
+`firstDiffPoint`, `maskedRegions`, and `lastPng` on timeout.
+
+## `expectScreenshot().toMatchBaseline()`
+
+```ts
+function expectScreenshot(png: Buffer): ScreenshotAssertion;
+
+interface ScreenshotAssertion {
+    toMatchBaseline(baselinePath: string, opts?: ScreenshotMatchOptions): Promise<void>;
+}
+```
+
+Resolves `baselinePath` against the active baseline root + resolution
+subdir (`hd/` or `fhd/`), reads the stored PNG, applies masks, runs
+pixelmatch, then either:
+
+- **equal within threshold** → assertion passes;
+- **missing baseline + `UPDATE_BASELINES=1`** → atomically writes the
+actual PNG at the baseline path and passes;
+- **missing baseline, normal run** → throws
+`ScreenshotBaselineMissingError`;
+- **dimensions differ** → throws `ScreenshotDimensionsMismatchError`
+(no diff PNG — pixelmatch cannot compare at different sizes);
+- **mismatch above threshold** → writes a diff PNG and throws
+`ScreenshotMismatchError`.
+
+**Minimal example.**
+
+```ts
+import { expectScreenshot } from "@roku-sdk/device-test-client";
+
+const png = await client.captureStableScreenshot({ timeout: 5_000 });
+await expectScreenshot(png).toMatchBaseline("home/initial.png");
+```
+
+**Realistic example — mask, explicit threshold.**
+
+```ts
+const png = await client.captureStableScreenshot({
+    timeout: 5_000,
+    mask: [{ x: 0, y: 0, width: 1920, height: 80 }]
+});
+
+await expectScreenshot(png).toMatchBaseline("home/initial.png", {
+    threshold: 0.005,  // 0.5% of unmasked pixels may differ
+    mask: [{ x: 0, y: 0, width: 1920, height: 80 }]
+});
+```
+
+Note that `mask` is applied in **both** the capture step and the
+assertion. The capture mask prevents animating content from ever
+being considered "stable"; the assertion mask makes sure the same
+region is ignored during comparison.
+
+## Thresholds — aggregate vs per-pixel
+
+The SDK decouples two thresholds on purpose:
+
+- **Aggregate ratio** (`opts.threshold`, default `0.01`) — the
+fraction of unmasked pixels that may differ for the assertion to
+pass.
+- **Per-pixel YIQ color distance** — fixed internal constant
+(`PIXELMATCH_COLOR_THRESHOLD = 0.1`), not exposed. Tuning this
+would quietly shift what counts as a "different" pixel, which is
+not a knob most tests actually want.
+
+So `{ threshold: 0.5 }` means "up to 50% of pixels may differ before
+I fail" — it does **not** mean "also loosen per-pixel sensitivity."
+
+## Masks
+
+Masks are validated and clamped against the image canvas:
+
+- Zero-width / zero-height → `ScreenshotMaskError`.
+- Fully out-of-bounds → `ScreenshotMaskError`.
+- Partially out-of-bounds → silently clamped to canvas.
+
+The SDK zeroes masked pixels in **both** images before invoking
+pixelmatch and excludes them from the ratio denominator, so a large
+mask does not inflate the effective threshold.
+
+**Example — masking the player clock.**
+
+```ts
+await expectScreenshot(png).toMatchBaseline("player/playing.png", {
+    mask: [
+        { x: 1600, y: 1000, width: 300, height: 60 },   // time counter
+        { x: 0,    y: 1040, width: 1920, height: 40 }   // progress bar
+    ]
+});
+```
+
+## Alignment tolerance
+
+Some surfaces are drawn by firmware rather than by your app — the
+system OSD raised by `StandardKeyboardDialog` and
+`StandardPinPadDialog`, for example. Their internal row heights round
+differently per device and firmware, so the whole surface re-centers by
+1-3px between otherwise-identical runs. Every separator line and text
+row then lands a pixel or two off, which is easily enough to blow a 1%
+budget even though nothing about the app changed.
+
+`alignTolerance` lets the comparator translate the capture by a few
+whole pixels before scoring it:
+
+```ts
+await expectScreenshot(png).toMatchBaseline("dialogs/keyboard-open.png", {
+    threshold: 0.01,
+    mask: KEYBOARD_MASK,
+    alignTolerance: { y: 3 }   // absorb vertical re-centering
+});
+```
+
+Three properties make this safe to reach for:
+
+- **It can only rescue a failing assertion, never relax a passing
+one.** The untranslated comparison runs first; the search happens only
+if that already exceeded `threshold`. A test that passes today keeps
+passing with byte-identical semantics, and the happy path does no
+extra work.
+- **`threshold` still applies in full** at the winning offset. The
+tolerance buys positional slack, not a bigger mismatch budget, so a
+missing row or a wrong color still fails.
+- **The window is capped** at `ALIGN_TOLERANCE_MAX_PX` (4) per axis.
+Needing more than that means the content genuinely differs and the
+baseline should be re-captured, not slid into place.
+
+Only the overlapping region is compared: at `dy=2` the top or bottom
+two rows have no counterpart and are dropped from both the mismatch
+count and the denominator.
+
+Use this only for firmware-drawn chrome. App-rendered content is
+positionally deterministic, and reaching for `alignTolerance` there
+hides real layout regressions.
+
+When the assertion fails even after aligning, the error names the best
+offset it found and the untranslated ratio, which distinguishes "the
+surface drifted further than the tolerance" from "the content is
+genuinely different":
+
+```
+... (unmasked ratio 0.00412 > threshold 0.001). Best alignment dx=0, dy=2
+(untranslated ratio was 0.01027). Diff: .../keyboard-open.diff.png.
+```
+
+A malformed tolerance (non-integer, negative, or over the cap) throws
+`ScreenshotAlignToleranceError`. Validation runs up front, so a typo
+surfaces on a passing run too rather than lying dormant until the day
+the assertion drifts.
+
+## Diff PNGs
+
+On mismatch, the SDK renders a side-by-side diff PNG under
+`getScreenshotDiffRoot()` (default `<cwd>/test-results/diffs/`). The
+filename flattens the baseline path:
+
+- `home/initial.png` → `home--initial.diff.png`
+- `grid/after-select.png` → `grid--after-select.diff.png`
+
+This keeps every failure's artifact in a single flat directory, easy
+for CI reporters to upload. The `ScreenshotMismatchError` exposes the
+full path in its `diffPath` field.
+
+Diff PNGs are always full canvas size with baseline-aligned
+coordinates, so a pixel in the diff names the same position as that
+pixel in the baseline. When an `alignTolerance` search won, the rows or
+columns that had no counterpart to compare are left transparent.
+
+## Diff-root override
+
+The diff-root is configured through `installDeviceTestHooks`:
+
+```ts
+import {
+    createDefaultClient,
+    installDeviceTestHooks
+} from "@roku-sdk/device-test-client";
+import { resolve } from "node:path";
+
+const client = createDefaultClient();
+
+installDeviceTestHooks({
+    client,
+    diffRoot: resolve(__dirname, "../../test-results/diffs")
+});
+```
+
+**Internal / deep-import.** The underlying
+`setScreenshotDiffRoot(absPath)` / `getScreenshotDiffRoot()`
+functions are no longer re-exported from the barrel in v2.0.0 — the
+public path is `installDeviceTestHooks({ diffRoot })`. If a bespoke
+runner needs to set the diff root without using
+`installDeviceTestHooks`, the functions are reachable via
+`@roku-sdk/device-test-client/src/assertions/screenshot-artifacts`
+(deep import, not SemVer-covered).
+
+## `UPDATE_BASELINES=1` refresh
+
+Re-run the suite with the env var set to regenerate baselines after a
+legitimate UI change:
+
+```bash
+UPDATE_BASELINES=1 pnpm jest path/to/file.test.ts
+```
+
+- Missing baseline → created from the actual PNG.
+- Mismatched baseline → overwritten with the actual PNG.
+**No diff PNG is written in the update path.**
+- Dimensions mismatch → overwritten with the actual PNG.
+- CI guard still applies — `CI=true` blocks writes.
+
+## Errors thrown
+
+- `ScreenshotBaselineMissingError` — no baseline + no
+`UPDATE_BASELINES=1`. Fields: `baselinePath`, `resolution`.
+- `ScreenshotDimensionsMismatchError` — baseline and actual disagree
+on size. Fields: `baselinePath`, `resolution`, `baseline.w/h`,
+`actual.w/h`.
+- `ScreenshotMismatchError` — pixel diff over threshold. Fields:
+`baselinePath`, `resolution`, `actual.w/h`, `mismatchedPixels`,
+`totalPixels`, `ratio`, `threshold`, `diffPath`, and `alignment`
+(`{ dx, dy }`, present only when an `alignTolerance` search improved on
+the untranslated comparison).
+- `ScreenshotNotStableError` — `captureStableScreenshot` timed out.
+Fields: `iterations`, `lastDiffCount`, `firstDiffPoint`,
+`maskedRegions`, `lastPng`.
+- `ScreenshotMaskError` — invalid `MaskRegion`. Fields: `mask`,
+`reason`.
+- `ScreenshotAlignToleranceError` — invalid `alignTolerance` axis.
+Fields: `axis`, `value`, `max`, `reason`.
+- `AppCrashedError` — crash probe during stable-capture polling.
+
+Full field lists and recovery advice in
+`[errors.md](errors.md)`.

package/package.json

@@ -1,6 +1,84 @@
 {
   "name": "@roku-sdk/device-test-client",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
+  "version": "2.19.0",
+  "description": "Roku SDK on-device integration test client (RkClient, DOM/screenshot/log assertions, Jest wiring).",
+  "main": "src/index.ts",
+  "types": "src/index.ts",
+  "exports": {
+    ".": {
+      "types": "./src/index.ts",
+      "default": "./src/index.ts"
+    },
+    "./environment": {
+      "types": "./src/jest/device-test-environment.ts",
+      "default": "./src/jest/device-test-environment.ts"
+    },
+    "./support/deploy": {
+      "types": "./support/deploy.ts",
+      "default": "./support/deploy.ts"
+    },
+    "./support/constants": {
+      "types": "./support/constants.ts",
+      "default": "./support/constants.ts"
+    },
+    "./support/paths": {
+      "types": "./support/paths.ts",
+      "default": "./support/paths.ts"
+    },
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "src/**/*",
+    "support/**/*",
+    "BREAKING_CHANGES.md",
+    "docs/**/*",
+    "README.md"
+  ],
+  "engines": {
+    "node": ">=24.0.0"
+  },
+  "author": "Roku Inc",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://gitlab.com/rokudev/rsg-sdk.git",
+    "directory": "packages/device-test-client"
+  },
+  "homepage": "https://gitlab.com/rokudev/rsg-sdk/-/tree/main/packages/device-test-client#readme",
+  "bugs": {
+    "url": "https://gitlab.com/rokudev/rsg-sdk/-/issues"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "pixelmatch": "^5.3.0",
+    "pngjs": "^7.0.0",
+    "semver": "^7.6.0",
+    "@roku-sdk/tooling": "2.19.0"
+  },
+  "peerDependencies": {
+    "jest-environment-node": ">=29.7.0",
+    "@roku-sdk/tooling": "^2.19.0"
+  },
+  "devDependencies": {
+    "@types/jest": "^30.0.0",
+    "@types/node": "^20.0.0",
+    "@types/pixelmatch": "^5.2.6",
+    "@types/pngjs": "^6.0.5",
+    "@types/semver": "^7.5.8",
+    "fast-check": "^4.7.0",
+    "jest": "^30.4.1",
+    "jest-environment-node": "^30.4.1",
+    "ts-jest": "^29.4.6",
+    "typescript": "5.5.4",
+    "ulid": "^2.3.0",
+    "@roku-sdk/dev-tools": "2.19.0",
+    "@roku-sdk/tooling": "2.19.0"
+  },
+  "scripts": {
+    "test": "jest",
+    "lint": "eslint --no-error-on-unmatched-pattern \"src/**/*.{ts,tsx,mjs,cjs,js}\" \"__tests__/**/*.{ts,tsx,mjs,cjs,js}\"",
+    "typecheck": "tsc --noEmit"
+  }
 }