@norskvideo/ctl-dev-kit 0.1.6 → 0.1.8

package/build/flake.nix

@@ -77,6 +77,7 @@
             pkgs.cargo # builds the native product-signer addon (probe/native)
             pkgs.rustc
             pkgs.git
+            pkgs.ffmpeg # integration suites spawn a local ffmpeg (e.g. SRT egress capture)
           ];
         in {
           # Everyday dev shell — build tools + the pinned norsk-ctl daemon.

package/conventions/integration-testing.md

@@ -0,0 +1,140 @@
+# Product integration testing on the DooD runners
+
+Guidelines for writing (and debugging) a split product's slow-tier integration
+suite against the org's self-hosted CI. These are hard-won: the failures below
+_look_ like license, networking, or engine bugs but are almost always
+**runner/harness setup** — mounts, ports, hosts, image provenance. Chasing them
+as product bugs wastes time.
+
+The reference implementations are the `norsk-ctl-product-playout` and
+`norsk-ctl-product-commentary` harnesses; the shared helpers live in
+`@norskvideo/ctl-test-harness`.
+
+## Runner topology (docker-outside-of-docker)
+
+The CI runner is itself a **container**. The daemon (`norsk-ctl`) and the product
+backend are spawned **in-process inside that container**. Studio + the Norsk
+media engine launch as **host siblings** (the runner shares the host docker
+socket), with their ports **published on the host**.
+
+Consequence — two address classes the harness must not confuse:
+
+| Reaching…                            | Address                                   |
+| ------------------------------------ | ----------------------------------------- |
+| the daemon, the product backend      | `localhost:<port>` (in-process)           |
+| studio / media host-published ports  | `${NORSK_TEST_HOST}:<port>`               |
+| a launched container by compose name | `<instance>-<service>-1` **on norsk-net** |
+
+`NORSK_TEST_HOST` is `host.docker.internal` in CI and unset (→ `localhost`)
+locally. Set it in the workflow env. The shared `studio-state` fetchers
+(`fetchComponents`, `fetchComponentState`, …) already honour it; **any local
+support code that hits a studio/media port must route through a `TEST_HOST`
+helper** (`process.env.NORSK_TEST_HOST ?? "localhost"`), never a hardcoded
+`localhost`. Daemon/backend fetches stay on `localhost`.
+
+## Host-shared working directory (or studio EACCESes on boot)
+
+Studio bind-mounts its instance working directory as `${workdir}:/data` and runs
+as a **non-root** auto-detected user. If the workdir is the daemon default
+(`~/norsk-runtime`, container-only), the sibling mount materialises **root-owned**
+and studio dies at boot with `EACCES: mkdir '/data/moq-certs'` — the workflow
+never starts, the SRT graph never builds, and every downstream test times out
+looking like a networking/license fault.
+
+Fix: point the daemon at a **host-shared** workdir under the test store dir.
+Pre-write `config.yaml` with `defaultWorkingDirectory` under `makeStoreDir(...)`
+(which roots in the gitignored, host-shared `test-temp/`):
+
+```ts
+writeFileSync(
+  join(storeDir, "config.yaml"),
+  `networkMode: docker\ndefaultWorkingDirectory: ${JSON.stringify(join(storeDir, "norsk-runtime"))}\n`,
+);
+```
+
+Docker still leaves **root-owned** read-only bind-mount target dirs inside that
+workdir, which the non-root runner can't `rm`. So `teardown()` must nuke the
+store dir from a throwaway **root** container, or `afterAll` (and the runner's
+own checkout cleanup) blocks:
+
+```ts
+spawnSync("docker", ["run", "--rm", "--user", "0:0", "-v", `${dirname(storeDir)}:/base`,
+  "--entrypoint", "sh", STUDIO_IMAGE, "-c", `rm -rf /base/${basename(storeDir)}`]);
+```
+
+## Reaching served media / HLS output
+
+Studio reports playback URLs (HLS multivariant, WHIP endpoints) with the host it
+was **configured with** — loopback locally, which the in-container test can't
+reach. Two cases:
+
+- **Proxied harness** — the URL is on the daemon's `publicHost` (default
+  `:443`). Route the fetch through `applyTestHost(url)` (swaps a loopback host
+  for `NORSK_TEST_HOST`). `assertMultivariantHasRenditions` does this by default.
+- **Proxy-less harness** — the advertised `/instance/<id>/media/...` route is
+  served **only by the daemon's nginx proxy**; with no proxy, nothing answers on
+  `:443`. Reach the media container **directly on norsk-net** instead:
+  `mediaDirectUrl(url)` rewrites it to `http://<id>-media-1:8080/<native-path>`
+  (the same bypass `whip-driver` uses; the media route is unauthenticated). Pass
+  it via `assertMultivariantHasRenditions`'s `resolveFetchUrl` hook, and **join
+  the runner to norsk-net** in setup so the compose service name resolves:
+
+  ```ts
+  spawnSync("docker", ["network", "create", "norsk-net"]);          // idempotent
+  spawnSync("docker", ["network", "connect", "norsk-net", hostname()]);
+  ```
+
+## Capture servers (studio pushes → the runner)
+
+When a test stands up an HLS/SCTE-35 capture server that **studio pushes to**,
+studio (a host sibling on norsk-net) can't reach the runner via the host LAN IP.
+Join norsk-net and hand studio the runner's **norsk-net IP** (`docker inspect`
+self), falling back to the host LAN IP only when `NORSK_TEST_HOST` is unset. See
+`manifest-capture.ts`'s `captureHost()`.
+
+## Build the product image in the integration job
+
+A `kind: dev` product still launches an instance whose compose references the
+product's **image** (e.g. the runtime-screen `funke-backend` service on
+`<product>:dev`). The `build-image` job builds it on a _different_ runner, so the
+integration job must build it too — otherwise `docker compose up` tries to
+**pull** the local-only tag and dies with `pull access denied` before any test
+runs. Add `bun run build:image` inside the `nix develop .#build` block (docker is
+reachable there); it also builds the dist the dev backend serves `/configure`
+from.
+
+## Stale node_modules on `clean: false` runners
+
+The runners check out `clean: false` (to preserve `test-temp/`), so
+`node_modules` **persists between runs**, and `bun install --frozen-lockfile`
+does **not** reliably relink a workspace-hoisted package whose version moved. A
+stale `@norskvideo/ctl-test-harness@0.1.2` once survived a bump to `0.1.3`, so a
+newly-added export was `SyntaxError: export 'X' not found` and the suite crashed
+in <1s. The bun **global cache is integrity-checked** (it can't serve a wrong
+version); only the persisted `node_modules` dir goes stale. Guard the install:
+
+```yaml
+rm -rf node_modules/@norskvideo/ctl-test-harness
+bun install --frozen-lockfile
+```
+
+When a product pins the shared harness in **both** `package.json` **and**
+`tests/package.json`, bump **both** — a lone root bump can leave the tests
+workspace resolving the old version.
+
+## GPU
+
+Instances launch with `--hardware nvidia` and encode on **nvenc** on the runners
+(the nvenc log-grep + no-software-fallback assertions pass). Note the
+`/usr/bin/nvidia-smi` **binary** may be absent in the media container even when
+nvenc works — assert on the encoder logs, not on `nvidia-smi` exit status.
+
+## License
+
+The org secret `NORSK_LICENSE_V2` + a modern engine accept the license (a green
+`probe` run proves it). V2 licenses need a **post-2026-07-17** engine, and SDK
+clients that create media nodes (e.g. a WHIP driver) must answer the license
+**handshake** via `createLicenseChallengeResponder` or the engine aborts node
+creation with `10 ABORTED: License handshake required`. The `license is marked
+NON-PRODUCTION` warning is expected in CI, not an error. The media engine's
+`hugepages / rte_eal_init failed` line is **benign noise** — do not chase it.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@norskvideo/ctl-dev-kit",
-  "version": "0.1.6",
+  "version": "0.1.8",
   "type": "module",
   "exports": {
     "./package.json": "./package.json"