@roku-sdk/device-test-client 0.0.0-stage → 2.18.0

package/docs/screenshot-assertions.md

@@ -0,0 +1,316 @@
+# Screenshot assertions
+
+Screenshot assertions commit a rendered frame to disk as the ground
+truth for a UI state and verify future runs stay within a configurable
+mismatch budget. They share the baseline layout, resolution routing,
+and `UPDATE_BASELINES=1` workflow with
+[DOM assertions](dom-assertions.md), but the comparison engine is
+[pixelmatch](https://github.com/mapbox/pixelmatch) and failures write
+an annotated diff PNG to `test-results/diffs/`.
+
+- [Core types](#core-types)
+- [Stable capture](#stable-capture)
+- `[expectScreenshot().toMatchBaseline(...)](#expectscreenshottomatchbaseline)`
+- [Thresholds — aggregate vs per-pixel](#thresholds--aggregate-vs-per-pixel)
+- [Masks](#masks)
+- [Alignment tolerance](#alignment-tolerance)
+- [Diff PNGs](#diff-pngs)
+- [Diff-root override](#diff-root-override)
+- `[UPDATE_BASELINES=1` refresh](#update_baselines1-refresh)
+- [Errors thrown](#errors-thrown)
+
+## Core types
+
+```ts
+interface ScreenshotMatchOptions {
+    /** Mismatch budget as a fraction of UNMASKED pixels. Default 0.01 (1%). */
+    threshold?: number;
+    /** Regions zeroed out in BOTH images before comparison. */
+    mask?: MaskRegion[];
+    /** Whole-pixel slack for device-dependent re-centering. Default none. */
+    alignTolerance?: AlignTolerance;
+}
+
+interface MaskRegion {
+    x: number;
+    y: number;
+    width: number;
+    height: number;
+}
+
+interface AlignTolerance {
+    /** Max horizontal translation, 0..ALIGN_TOLERANCE_MAX_PX. Default 0. */
+    x?: number;
+    /** Max vertical translation, 0..ALIGN_TOLERANCE_MAX_PX. Default 0. */
+    y?: number;
+}
+
+interface CaptureStableOptions {
+    /** Overall wait budget (ms). */
+    timeout: number;
+    /** Minimum gap between captures (ms). Default 500. */
+    interval?: number;
+    /** Regions ignored when deciding "two frames are identical". */
+    mask?: MaskRegion[];
+}
+```
+
+## Stable capture
+
+Raw `client.captureScreenshot()` returns whatever is on screen at that
+instant — typically a mid-animation frame. For baseline comparison you
+want a **settled** frame: that's `captureStableScreenshot`.
+
+```ts
+captureStableScreenshot(opts: CaptureStableOptions): Promise<Buffer>;
+```
+
+Polls until two successive captures are pixel-identical outside `mask`
+regions and returns the newer PNG. `assertAppAlive` runs before each
+capture so a crash during polling surfaces as `AppCrashedError`
+instead of a timeout.
+
+**Example.**
+
+```ts
+const png = await client.captureStableScreenshot({
+    timeout: 5_000,
+    mask: [{ x: 0, y: 0, width: 1920, height: 80 }]  // hide the clock/status bar
+});
+```
+
+Throws `ScreenshotNotStableError` with `iterations`, `lastDiffCount`,
+`firstDiffPoint`, `maskedRegions`, and `lastPng` on timeout.
+
+## `expectScreenshot().toMatchBaseline()`
+
+```ts
+function expectScreenshot(png: Buffer): ScreenshotAssertion;
+
+interface ScreenshotAssertion {
+    toMatchBaseline(baselinePath: string, opts?: ScreenshotMatchOptions): Promise<void>;
+}
+```
+
+Resolves `baselinePath` against the active baseline root + resolution
+subdir (`hd/` or `fhd/`), reads the stored PNG, applies masks, runs
+pixelmatch, then either:
+
+- **equal within threshold** → assertion passes;
+- **missing baseline + `UPDATE_BASELINES=1`** → atomically writes the
+actual PNG at the baseline path and passes;
+- **missing baseline, normal run** → throws
+`ScreenshotBaselineMissingError`;
+- **dimensions differ** → throws `ScreenshotDimensionsMismatchError`
+(no diff PNG — pixelmatch cannot compare at different sizes);
+- **mismatch above threshold** → writes a diff PNG and throws
+`ScreenshotMismatchError`.
+
+**Minimal example.**
+
+```ts
+import { expectScreenshot } from "@roku-sdk/device-test-client";
+
+const png = await client.captureStableScreenshot({ timeout: 5_000 });
+await expectScreenshot(png).toMatchBaseline("home/initial.png");
+```
+
+**Realistic example — mask, explicit threshold.**
+
+```ts
+const png = await client.captureStableScreenshot({
+    timeout: 5_000,
+    mask: [{ x: 0, y: 0, width: 1920, height: 80 }]
+});
+
+await expectScreenshot(png).toMatchBaseline("home/initial.png", {
+    threshold: 0.005,  // 0.5% of unmasked pixels may differ
+    mask: [{ x: 0, y: 0, width: 1920, height: 80 }]
+});
+```
+
+Note that `mask` is applied in **both** the capture step and the
+assertion. The capture mask prevents animating content from ever
+being considered "stable"; the assertion mask makes sure the same
+region is ignored during comparison.
+
+## Thresholds — aggregate vs per-pixel
+
+The SDK decouples two thresholds on purpose:
+
+- **Aggregate ratio** (`opts.threshold`, default `0.01`) — the
+fraction of unmasked pixels that may differ for the assertion to
+pass.
+- **Per-pixel YIQ color distance** — fixed internal constant
+(`PIXELMATCH_COLOR_THRESHOLD = 0.1`), not exposed. Tuning this
+would quietly shift what counts as a "different" pixel, which is
+not a knob most tests actually want.
+
+So `{ threshold: 0.5 }` means "up to 50% of pixels may differ before
+I fail" — it does **not** mean "also loosen per-pixel sensitivity."
+
+## Masks
+
+Masks are validated and clamped against the image canvas:
+
+- Zero-width / zero-height → `ScreenshotMaskError`.
+- Fully out-of-bounds → `ScreenshotMaskError`.
+- Partially out-of-bounds → silently clamped to canvas.
+
+The SDK zeroes masked pixels in **both** images before invoking
+pixelmatch and excludes them from the ratio denominator, so a large
+mask does not inflate the effective threshold.
+
+**Example — masking the player clock.**
+
+```ts
+await expectScreenshot(png).toMatchBaseline("player/playing.png", {
+    mask: [
+        { x: 1600, y: 1000, width: 300, height: 60 },   // time counter
+        { x: 0,    y: 1040, width: 1920, height: 40 }   // progress bar
+    ]
+});
+```
+
+## Alignment tolerance
+
+Some surfaces are drawn by firmware rather than by your app — the
+system OSD raised by `StandardKeyboardDialog` and
+`StandardPinPadDialog`, for example. Their internal row heights round
+differently per device and firmware, so the whole surface re-centers by
+1-3px between otherwise-identical runs. Every separator line and text
+row then lands a pixel or two off, which is easily enough to blow a 1%
+budget even though nothing about the app changed.
+
+`alignTolerance` lets the comparator translate the capture by a few
+whole pixels before scoring it:
+
+```ts
+await expectScreenshot(png).toMatchBaseline("dialogs/keyboard-open.png", {
+    threshold: 0.01,
+    mask: KEYBOARD_MASK,
+    alignTolerance: { y: 3 }   // absorb vertical re-centering
+});
+```
+
+Three properties make this safe to reach for:
+
+- **It can only rescue a failing assertion, never relax a passing
+one.** The untranslated comparison runs first; the search happens only
+if that already exceeded `threshold`. A test that passes today keeps
+passing with byte-identical semantics, and the happy path does no
+extra work.
+- **`threshold` still applies in full** at the winning offset. The
+tolerance buys positional slack, not a bigger mismatch budget, so a
+missing row or a wrong color still fails.
+- **The window is capped** at `ALIGN_TOLERANCE_MAX_PX` (4) per axis.
+Needing more than that means the content genuinely differs and the
+baseline should be re-captured, not slid into place.
+
+Only the overlapping region is compared: at `dy=2` the top or bottom
+two rows have no counterpart and are dropped from both the mismatch
+count and the denominator.
+
+Use this only for firmware-drawn chrome. App-rendered content is
+positionally deterministic, and reaching for `alignTolerance` there
+hides real layout regressions.
+
+When the assertion fails even after aligning, the error names the best
+offset it found and the untranslated ratio, which distinguishes "the
+surface drifted further than the tolerance" from "the content is
+genuinely different":
+
+```
+... (unmasked ratio 0.00412 > threshold 0.001). Best alignment dx=0, dy=2
+(untranslated ratio was 0.01027). Diff: .../keyboard-open.diff.png.
+```
+
+A malformed tolerance (non-integer, negative, or over the cap) throws
+`ScreenshotAlignToleranceError`. Validation runs up front, so a typo
+surfaces on a passing run too rather than lying dormant until the day
+the assertion drifts.
+
+## Diff PNGs
+
+On mismatch, the SDK renders a side-by-side diff PNG under
+`getScreenshotDiffRoot()` (default `<cwd>/test-results/diffs/`). The
+filename flattens the baseline path:
+
+- `home/initial.png` → `home--initial.diff.png`
+- `grid/after-select.png` → `grid--after-select.diff.png`
+
+This keeps every failure's artifact in a single flat directory, easy
+for CI reporters to upload. The `ScreenshotMismatchError` exposes the
+full path in its `diffPath` field.
+
+Diff PNGs are always full canvas size with baseline-aligned
+coordinates, so a pixel in the diff names the same position as that
+pixel in the baseline. When an `alignTolerance` search won, the rows or
+columns that had no counterpart to compare are left transparent.
+
+## Diff-root override
+
+The diff-root is configured through `installDeviceTestHooks`:
+
+```ts
+import {
+    createDefaultClient,
+    installDeviceTestHooks
+} from "@roku-sdk/device-test-client";
+import { resolve } from "node:path";
+
+const client = createDefaultClient();
+
+installDeviceTestHooks({
+    client,
+    diffRoot: resolve(__dirname, "../../test-results/diffs")
+});
+```
+
+**Internal / deep-import.** The underlying
+`setScreenshotDiffRoot(absPath)` / `getScreenshotDiffRoot()`
+functions are no longer re-exported from the barrel in v2.0.0 — the
+public path is `installDeviceTestHooks({ diffRoot })`. If a bespoke
+runner needs to set the diff root without using
+`installDeviceTestHooks`, the functions are reachable via
+`@roku-sdk/device-test-client/src/assertions/screenshot-artifacts`
+(deep import, not SemVer-covered).
+
+## `UPDATE_BASELINES=1` refresh
+
+Re-run the suite with the env var set to regenerate baselines after a
+legitimate UI change:
+
+```bash
+UPDATE_BASELINES=1 pnpm jest path/to/file.test.ts
+```
+
+- Missing baseline → created from the actual PNG.
+- Mismatched baseline → overwritten with the actual PNG.
+**No diff PNG is written in the update path.**
+- Dimensions mismatch → overwritten with the actual PNG.
+- CI guard still applies — `CI=true` blocks writes.
+
+## Errors thrown
+
+- `ScreenshotBaselineMissingError` — no baseline + no
+`UPDATE_BASELINES=1`. Fields: `baselinePath`, `resolution`.
+- `ScreenshotDimensionsMismatchError` — baseline and actual disagree
+on size. Fields: `baselinePath`, `resolution`, `baseline.w/h`,
+`actual.w/h`.
+- `ScreenshotMismatchError` — pixel diff over threshold. Fields:
+`baselinePath`, `resolution`, `actual.w/h`, `mismatchedPixels`,
+`totalPixels`, `ratio`, `threshold`, `diffPath`, and `alignment`
+(`{ dx, dy }`, present only when an `alignTolerance` search improved on
+the untranslated comparison).
+- `ScreenshotNotStableError` — `captureStableScreenshot` timed out.
+Fields: `iterations`, `lastDiffCount`, `firstDiffPoint`,
+`maskedRegions`, `lastPng`.
+- `ScreenshotMaskError` — invalid `MaskRegion`. Fields: `mask`,
+`reason`.
+- `ScreenshotAlignToleranceError` — invalid `alignTolerance` axis.
+Fields: `axis`, `value`, `max`, `reason`.
+- `AppCrashedError` — crash probe during stable-capture polling.
+
+Full field lists and recovery advice in
+`[errors.md](errors.md)`.

package/internal/constants.ts

@@ -0,0 +1,54 @@
+// Copyright (c) 2026 Roku, Inc.
+// SPDX-License-Identifier: MIT
+
+/**
+ * `@roku-sdk/device-test-client/internal/constants` — config-load-safe
+ * entry point for the small subset of `src/constants.ts` symbols that
+ * Jest configs (and other CommonJS-mode loaders) legitimately need at
+ * **parse time**.
+ *
+ * Why this exists, and why the public barrel is not enough:
+ *
+ *   `src/index.ts` re-exports `RkClient` from `src/client/client.ts`,
+ *   which imports `@roku-sdk/tooling/src/device-test-helpers.ts`. The
+ *   `@roku-sdk/tooling` package declares `"type": "module"` and uses
+ *   `import.meta.url`. Loaders that parse `*.config.ts` in CommonJS
+ *   mode (ts-node default, esbuild-register, the loader Jest invokes
+ *   from `jest-config/readConfigFileAndSetRootDir`) blow up on the
+ *   first hop with `ERR_REQUIRE_ESM` — *before* any test, hook, or
+ *   transform runs. The stack trace points at the loader, not at the
+ *   import that pulled the ESM file in, so the failure is opaque to
+ *   the reader of `jest.config.ts`.
+ *
+ *   `src/constants.ts` itself has only `type` imports (all erased by
+ *   TypeScript) and value-level constants, so a file that imports
+ *   exclusively from it is parse-safe under CJS. This subpath is the
+ *   one sanctioned "deep import past the barrel" surface for that
+ *   purpose.
+ *
+ * Why a re-export shim instead of a direct entry in `exports`:
+ *   `device-tests/tsconfig.json` uses `moduleResolution: "Node"`
+ *   (classic), which falls back to a filesystem walk and does NOT honor
+ *   the package `exports` map. A physical file at this path resolves
+ *   under every resolution mode; the `exports` map entry covers the
+ *   publish-time / `Node16` / `Bundler` case symmetrically. The same
+ *   pattern is used by `./internal/deploy`.
+ *
+ * SemVer / stability contract:
+ *   Symbols here are reachable, typed, and stable across patch releases
+ *   — but explicitly NOT covered by SemVer (`/internal/` in the path is
+ *   the tell). Partner code MUST NOT import from this subpath; the
+ *   public, SemVer-covered surface for these constants stays
+ *   `@roku-sdk/device-test-client` (curated barrel in `src/index.ts`).
+ *
+ * Adding new symbols:
+ *   Only constants that a config-loader truly needs at parse time
+ *   belong here (file-name constants, env-var names, default budgets
+ *   referenced by `jest.config.ts` reporters / global setup). Pure-
+ *   implementation magic numbers stay file-internal in
+ *   `src/constants.ts`. Prefer extending this barrel over minting a
+ *   parallel `/internal/<other>` subpath — keeps the internal contract
+ *   grep-able.
+ */
+
+export { ARTIFACT_FILE } from "../src/constants";

package/internal/deploy.ts

@@ -0,0 +1,60 @@
+// Copyright (c) 2026 Roku, Inc.
+// SPDX-License-Identifier: MIT
+
+/**
+ * `@roku-sdk/device-test-client/internal/deploy` — explicit internal
+ * subpath for the **deploy-side** tooling helpers that the curated
+ * public API intentionally does NOT carry.
+ *
+ * Currently exposes:
+ *
+ *   - `resolveAppPath` — integration-test deploy-guard directory-wipe
+ *     path resolution.
+ *   - `invalidateBuildCacheForApp` — release the in-process
+ *     `RkClient.buildAndDeploy` build memoisation entries scoped to
+ *     a single `absRoot`. Pairs with the deploy-guard's
+ *     `wipeIntegrationBuildOutput` so a "we rebuilt because the
+ *     previous bundle was dirty" recovery path also invalidates the
+ *     "we already built this" memoisation. Both calls must run in
+ *     lockstep; calling one without the other is a bug.
+ *
+ * Symbols here are reachable, typed, and stable across patch
+ * releases — but explicitly NOT covered by SemVer (`/internal/` in
+ * the path is the tell). Partner code MUST NOT import from this
+ * subpath; downstream consumers should treat it as private.
+ *
+ * Why a re-export shim instead of a direct entry in `exports`:
+ * `device-tests/tsconfig.json` uses `moduleResolution: "Node"`
+ * (classic), which falls back to a filesystem walk and does NOT honor
+ * the package `exports` map. A physical file at this path resolves
+ * under every resolution mode; the `exports` map entry covers the
+ * publish-time / `Node16` / `Bundler` case symmetrically.
+ *
+ * Adding new symbols (load-chain caveat):
+ *   This barrel re-exports from `../src/client/deploy`, which imports
+ *   `@roku-sdk/tooling/src/device-test-helpers` — an ESM module
+ *   (`"type": "module"`, `import.meta.url`). Anything importing this
+ *   subpath therefore pulls that ESM chain in transitively. That is
+ *   FINE for runtime callers (Jest transforms `@roku-sdk/*` via babel,
+ *   see `device-tests/jest.config.ts → transform`) but it is FATAL
+ *   for parse-time loaders such as ts-node in CJS mode that the IDE
+ *   and Jest itself use to read `*.config.ts`; those crash with
+ *   `ERR_REQUIRE_ESM` before any transform runs.
+ *
+ *   Group new `@internal` symbols by **load-time safety**, not just
+ *   by concern:
+ *     - Runtime-only helpers (need the live client / build pipeline /
+ *       tooling shim) → add here.
+ *     - Symbols a `*.config.ts` legitimately reads at parse time →
+ *       add to `./internal/constants` (or another sibling subpath
+ *       whose source file imports only types and value-level
+ *       literals). Re-confirm by tracing the new file's imports to
+ *       leaves; one accidental hop into `@roku-sdk/tooling` re-breaks
+ *       the config-loader path.
+ *
+ *   Prefer extending an existing `/internal/<name>` subpath over
+ *   minting a parallel one — keeps the internal contract grep-able.
+ */
+
+export { resolveAppPath } from "../src/client/deploy";
+export { invalidateBuildCacheForApp } from "../src/client/build-cache";

package/internal/paths.ts

@@ -0,0 +1,15 @@
+// Copyright (c) 2026 Roku, Inc.
+// SPDX-License-Identifier: MIT
+
+/**
+ * `@roku-sdk/device-test-client/internal/paths` — explicit internal
+ * subpath for load-safe path constants/helpers used by test config and
+ * setup code.
+ *
+ * This subpath intentionally re-exports only from `src/paths.ts`, whose
+ * runtime imports are Node built-ins. It is safe for CommonJS-mode config
+ * loaders (`jest.config.ts`, ts-node) and must stay free of `RkClient`,
+ * `@roku-sdk/tooling`, or other ESM-heavy runtime chains.
+ */
+
+export { TEST_RESULTS_DIR_ENV_VAR, TEST_RESULTS_DIR_NAME, defaultTestResultsDir } from "../src/paths";

package/package.json

@@ -1,6 +1,83 @@
 {
   "name": "@roku-sdk/device-test-client",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
+  "version": "2.18.0",
+  "description": "Roku SDK on-device integration test client (RkClient, DOM/screenshot/log assertions, Jest wiring).",
+  "main": "src/index.ts",
+  "types": "src/index.ts",
+  "exports": {
+    ".": {
+      "types": "./src/index.ts",
+      "default": "./src/index.ts"
+    },
+    "./environment": {
+      "types": "./src/jest/device-test-environment.ts",
+      "default": "./src/jest/device-test-environment.ts"
+    },
+    "./internal/deploy": {
+      "types": "./internal/deploy.ts",
+      "default": "./internal/deploy.ts"
+    },
+    "./internal/constants": {
+      "types": "./internal/constants.ts",
+      "default": "./internal/constants.ts"
+    },
+    "./internal/paths": {
+      "types": "./internal/paths.ts",
+      "default": "./internal/paths.ts"
+    },
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "src/**/*",
+    "internal/**/*",
+    "docs/**/*",
+    "README.md"
+  ],
+  "engines": {
+    "node": ">=24.0.0"
+  },
+  "author": "Roku Inc",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://gitlab.com/rokudev/rsg-sdk.git",
+    "directory": "packages/device-test-client"
+  },
+  "homepage": "https://gitlab.com/rokudev/rsg-sdk/-/tree/main/packages/device-test-client#readme",
+  "bugs": {
+    "url": "https://gitlab.com/rokudev/rsg-sdk/-/issues"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "pixelmatch": "^5.3.0",
+    "pngjs": "^7.0.0",
+    "semver": "^7.6.0",
+    "@roku-sdk/tooling": "2.18.0"
+  },
+  "peerDependencies": {
+    "jest-environment-node": ">=29.7.0",
+    "@roku-sdk/tooling": "^2.18.0"
+  },
+  "devDependencies": {
+    "@types/jest": "^30.0.0",
+    "@types/node": "^20.0.0",
+    "@types/pixelmatch": "^5.2.6",
+    "@types/pngjs": "^6.0.5",
+    "@types/semver": "^7.5.8",
+    "fast-check": "^4.7.0",
+    "jest": "^30.4.1",
+    "jest-environment-node": "^30.4.1",
+    "ts-jest": "^29.4.6",
+    "typescript": "5.5.4",
+    "ulid": "^2.3.0",
+    "@roku-sdk/dev-tools": "2.18.0",
+    "@roku-sdk/tooling": "2.18.0"
+  },
+  "scripts": {
+    "test": "jest",
+    "lint": "eslint --no-error-on-unmatched-pattern \"src/**/*.{ts,tsx,mjs,cjs,js}\" \"__tests__/**/*.{ts,tsx,mjs,cjs,js}\"",
+    "typecheck": "tsc --noEmit"
+  }
 }