@norskvideo/ctl-dev-kit 0.1.84 → 0.1.86

package/conventions/checks.yml

@@ -30,23 +30,61 @@ jobs:
   drift:
     runs-on: x64
     steps:
-      # Root-owned leftovers under test-temp/ are what `clean: false` used to route
-      # around: the integration suite bind-mounts host dirs into containers, and a
-      # non-root git clean then EACCESes on what they wrote -- taking down whatever
-      # innocent job checked out next. Skipping the clean cured that symptom and
-      # caused another: every gitignored artifact survived between runs, so a test
-      # could pass on build output an earlier job left behind and go red the first
-      # time it landed on a cold runner (probe, 2026-09-01). Clear the leftovers
-      # from a root container FIRST, and the checkout can clean properly again.
-      # Best-effort by construction: no docker, no test-temp, or a failed run all
-      # fall through without failing the job.
-      - name: Clear stale root-owned test-temp (pre-checkout)
+      # Runs BEFORE checkout, and does two things: points the harness's temp base
+      # out of the workspace for every later step in this job, and clears whatever
+      # the old repo-local default left behind on this runner. The body carries the
+      # why. It fails the job only when legacy litter survives -- which would take
+      # the checkout down a step later anyway, but report nothing useful.
+      - name: Route test-temp out of the workspace (pre-checkout)
         run: |
           set -uo pipefail
+          # These runners are docker-OUTSIDE-of-docker, so a bind-mount source the
+          # HOST daemon has not seen before is created by dockerd, as ROOT, on the
+          # host -- even when the launch passes --container-user (reproduced
+          # 2026-09-04). The harness's default base is repo-local for an OrbStack
+          # inotify constraint that binds macOS dev and nothing here, so every run
+          # seeded the CHECKOUT with root-owned dirs a non-root `git clean -ffdx`
+          # cannot remove. That took commentary's checks AND integration red on
+          # 2026-09-03. Use the sanctioned override, per-run and outside the
+          # checkout -- as 8ff17a8d did in June for one debug workflow, and stopped.
+          base="$RUNNER_TEMP/nctt-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"
+          mkdir -p "$base"
+          echo "NORSK_CTL_TEST_TMP=$base" >> "$GITHUB_ENV"
+          # Yesterday's bases hold dockerd-created root dirs, so a non-root rm
+          # EACCESes. They cannot wedge a checkout, but they do fill the disk.
+          for old in $(find "$RUNNER_TEMP" -maxdepth 1 -name 'nctt-*' -mtime +0 -printf '%f\n' 2>/dev/null || true); do
+            docker run --rm --user 0:0 -v "$RUNNER_TEMP":/t alpine sh -c 'rm -rf -- "/t/$1"' sh "$old" >/dev/null 2>&1 || true
+          done
+          # LEGACY: a runner that ran the old default still carries a workspace
+          # test-temp, and a checkout with clean:true dies on it. Reap the holders
+          # first -- rm cannot unlink a live mountpoint even as root (EBUSY), and a
+          # RESTARTING leaked container re-creates the path as root seconds after
+          # any clean. Match on mount SOURCE, not a label: the holders are untracked
+          # by construction, because a killed run labels nothing.
           tt="$GITHUB_WORKSPACE/test-temp"
           [ -d "$tt" ] || exit 0
-          docker run --rm --user 0:0 -v "$tt":/t alpine sh \
-            -c 'rm -rf /t/* /t/.[!.]* 2>/dev/null || true' || rm -rf "$tt"/* 2>/dev/null || true
+          stuck=""
+          for c in $(docker ps -aq 2>/dev/null || true); do
+            if docker inspect -f '{{range .Mounts}}{{println .Source}}{{end}}' "$c" 2>/dev/null | grep -q "^$tt/"; then
+              stuck="$stuck $c"
+            fi
+          done
+          if [ -n "$stuck" ]; then
+            echo "reaping containers holding mounts under $tt:$stuck"
+            docker rm -f $stuck || true
+          fi
+          # The DIRECTORY, not just its contents: an empty but root-owned base
+          # still EACCESes a later mkdtemp. Mounting the PARENT is what lets a root
+          # container unlink the leaf.
+          docker run --rm --user 0:0 -v "$GITHUB_WORKSPACE":/w alpine \
+            sh -c 'rm -rf /w/test-temp' || rm -rf "$tt" || true
+          # Fail HERE if it survived. The checkout fails on it either way, but
+          # reports only an EACCES rmdir with no clue what was holding the path.
+          if [ -e "$tt" ]; then
+            echo "::error::$tt survived the pre-checkout clean"
+            ls -lan "$tt" || true
+            exit 1
+          fi
 
       - uses: actions/checkout@v5
         with:
@@ -68,23 +106,61 @@ jobs:
   quality:
     runs-on: x64
     steps:
-      # Root-owned leftovers under test-temp/ are what `clean: false` used to route
-      # around: the integration suite bind-mounts host dirs into containers, and a
-      # non-root git clean then EACCESes on what they wrote -- taking down whatever
-      # innocent job checked out next. Skipping the clean cured that symptom and
-      # caused another: every gitignored artifact survived between runs, so a test
-      # could pass on build output an earlier job left behind and go red the first
-      # time it landed on a cold runner (probe, 2026-09-01). Clear the leftovers
-      # from a root container FIRST, and the checkout can clean properly again.
-      # Best-effort by construction: no docker, no test-temp, or a failed run all
-      # fall through without failing the job.
-      - name: Clear stale root-owned test-temp (pre-checkout)
+      # Runs BEFORE checkout, and does two things: points the harness's temp base
+      # out of the workspace for every later step in this job, and clears whatever
+      # the old repo-local default left behind on this runner. The body carries the
+      # why. It fails the job only when legacy litter survives -- which would take
+      # the checkout down a step later anyway, but report nothing useful.
+      - name: Route test-temp out of the workspace (pre-checkout)
         run: |
           set -uo pipefail
+          # These runners are docker-OUTSIDE-of-docker, so a bind-mount source the
+          # HOST daemon has not seen before is created by dockerd, as ROOT, on the
+          # host -- even when the launch passes --container-user (reproduced
+          # 2026-09-04). The harness's default base is repo-local for an OrbStack
+          # inotify constraint that binds macOS dev and nothing here, so every run
+          # seeded the CHECKOUT with root-owned dirs a non-root `git clean -ffdx`
+          # cannot remove. That took commentary's checks AND integration red on
+          # 2026-09-03. Use the sanctioned override, per-run and outside the
+          # checkout -- as 8ff17a8d did in June for one debug workflow, and stopped.
+          base="$RUNNER_TEMP/nctt-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"
+          mkdir -p "$base"
+          echo "NORSK_CTL_TEST_TMP=$base" >> "$GITHUB_ENV"
+          # Yesterday's bases hold dockerd-created root dirs, so a non-root rm
+          # EACCESes. They cannot wedge a checkout, but they do fill the disk.
+          for old in $(find "$RUNNER_TEMP" -maxdepth 1 -name 'nctt-*' -mtime +0 -printf '%f\n' 2>/dev/null || true); do
+            docker run --rm --user 0:0 -v "$RUNNER_TEMP":/t alpine sh -c 'rm -rf -- "/t/$1"' sh "$old" >/dev/null 2>&1 || true
+          done
+          # LEGACY: a runner that ran the old default still carries a workspace
+          # test-temp, and a checkout with clean:true dies on it. Reap the holders
+          # first -- rm cannot unlink a live mountpoint even as root (EBUSY), and a
+          # RESTARTING leaked container re-creates the path as root seconds after
+          # any clean. Match on mount SOURCE, not a label: the holders are untracked
+          # by construction, because a killed run labels nothing.
           tt="$GITHUB_WORKSPACE/test-temp"
           [ -d "$tt" ] || exit 0
-          docker run --rm --user 0:0 -v "$tt":/t alpine sh \
-            -c 'rm -rf /t/* /t/.[!.]* 2>/dev/null || true' || rm -rf "$tt"/* 2>/dev/null || true
+          stuck=""
+          for c in $(docker ps -aq 2>/dev/null || true); do
+            if docker inspect -f '{{range .Mounts}}{{println .Source}}{{end}}' "$c" 2>/dev/null | grep -q "^$tt/"; then
+              stuck="$stuck $c"
+            fi
+          done
+          if [ -n "$stuck" ]; then
+            echo "reaping containers holding mounts under $tt:$stuck"
+            docker rm -f $stuck || true
+          fi
+          # The DIRECTORY, not just its contents: an empty but root-owned base
+          # still EACCESes a later mkdtemp. Mounting the PARENT is what lets a root
+          # container unlink the leaf.
+          docker run --rm --user 0:0 -v "$GITHUB_WORKSPACE":/w alpine \
+            sh -c 'rm -rf /w/test-temp' || rm -rf "$tt" || true
+          # Fail HERE if it survived. The checkout fails on it either way, but
+          # reports only an EACCES rmdir with no clue what was holding the path.
+          if [ -e "$tt" ]; then
+            echo "::error::$tt survived the pre-checkout clean"
+            ls -lan "$tt" || true
+            exit 1
+          fi
 
       - uses: actions/checkout@v5
         with:
@@ -160,23 +236,61 @@ jobs:
     if: ${{ !cancelled() && github.event_name == 'push' }}
     runs-on: x64
     steps:
-      # Root-owned leftovers under test-temp/ are what `clean: false` used to route
-      # around: the integration suite bind-mounts host dirs into containers, and a
-      # non-root git clean then EACCESes on what they wrote -- taking down whatever
-      # innocent job checked out next. Skipping the clean cured that symptom and
-      # caused another: every gitignored artifact survived between runs, so a test
-      # could pass on build output an earlier job left behind and go red the first
-      # time it landed on a cold runner (probe, 2026-09-01). Clear the leftovers
-      # from a root container FIRST, and the checkout can clean properly again.
-      # Best-effort by construction: no docker, no test-temp, or a failed run all
-      # fall through without failing the job.
-      - name: Clear stale root-owned test-temp (pre-checkout)
+      # Runs BEFORE checkout, and does two things: points the harness's temp base
+      # out of the workspace for every later step in this job, and clears whatever
+      # the old repo-local default left behind on this runner. The body carries the
+      # why. It fails the job only when legacy litter survives -- which would take
+      # the checkout down a step later anyway, but report nothing useful.
+      - name: Route test-temp out of the workspace (pre-checkout)
         run: |
           set -uo pipefail
+          # These runners are docker-OUTSIDE-of-docker, so a bind-mount source the
+          # HOST daemon has not seen before is created by dockerd, as ROOT, on the
+          # host -- even when the launch passes --container-user (reproduced
+          # 2026-09-04). The harness's default base is repo-local for an OrbStack
+          # inotify constraint that binds macOS dev and nothing here, so every run
+          # seeded the CHECKOUT with root-owned dirs a non-root `git clean -ffdx`
+          # cannot remove. That took commentary's checks AND integration red on
+          # 2026-09-03. Use the sanctioned override, per-run and outside the
+          # checkout -- as 8ff17a8d did in June for one debug workflow, and stopped.
+          base="$RUNNER_TEMP/nctt-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"
+          mkdir -p "$base"
+          echo "NORSK_CTL_TEST_TMP=$base" >> "$GITHUB_ENV"
+          # Yesterday's bases hold dockerd-created root dirs, so a non-root rm
+          # EACCESes. They cannot wedge a checkout, but they do fill the disk.
+          for old in $(find "$RUNNER_TEMP" -maxdepth 1 -name 'nctt-*' -mtime +0 -printf '%f\n' 2>/dev/null || true); do
+            docker run --rm --user 0:0 -v "$RUNNER_TEMP":/t alpine sh -c 'rm -rf -- "/t/$1"' sh "$old" >/dev/null 2>&1 || true
+          done
+          # LEGACY: a runner that ran the old default still carries a workspace
+          # test-temp, and a checkout with clean:true dies on it. Reap the holders
+          # first -- rm cannot unlink a live mountpoint even as root (EBUSY), and a
+          # RESTARTING leaked container re-creates the path as root seconds after
+          # any clean. Match on mount SOURCE, not a label: the holders are untracked
+          # by construction, because a killed run labels nothing.
           tt="$GITHUB_WORKSPACE/test-temp"
           [ -d "$tt" ] || exit 0
-          docker run --rm --user 0:0 -v "$tt":/t alpine sh \
-            -c 'rm -rf /t/* /t/.[!.]* 2>/dev/null || true' || rm -rf "$tt"/* 2>/dev/null || true
+          stuck=""
+          for c in $(docker ps -aq 2>/dev/null || true); do
+            if docker inspect -f '{{range .Mounts}}{{println .Source}}{{end}}' "$c" 2>/dev/null | grep -q "^$tt/"; then
+              stuck="$stuck $c"
+            fi
+          done
+          if [ -n "$stuck" ]; then
+            echo "reaping containers holding mounts under $tt:$stuck"
+            docker rm -f $stuck || true
+          fi
+          # The DIRECTORY, not just its contents: an empty but root-owned base
+          # still EACCESes a later mkdtemp. Mounting the PARENT is what lets a root
+          # container unlink the leaf.
+          docker run --rm --user 0:0 -v "$GITHUB_WORKSPACE":/w alpine \
+            sh -c 'rm -rf /w/test-temp' || rm -rf "$tt" || true
+          # Fail HERE if it survived. The checkout fails on it either way, but
+          # reports only an EACCES rmdir with no clue what was holding the path.
+          if [ -e "$tt" ]; then
+            echo "::error::$tt survived the pre-checkout clean"
+            ls -lan "$tt" || true
+            exit 1
+          fi
 
       - uses: actions/checkout@v5
         with:

package/conventions/integration-testing.md

@@ -138,3 +138,43 @@ clients that create media nodes (e.g. a WHIP driver) must answer the license
 creation with `10 ABORTED: License handshake required`. The `license is marked
 NON-PRODUCTION` warning is expected in CI, not an error. The media engine's
 `hugepages / rte_eal_init failed` line is **benign noise** — do not chase it.
+
+## The workflow: `integration.yml` is a convention
+
+`.github/workflows/integration.yml` is single-sourced in this dev-kit
+(`conventions/integration.yml`) and drift-gated like `smoke.yml`, with the same
+two per-repo lines: the `product:` dashboard key and the matrix `runner:` label
+array. Everything else a repo used to hand-edit into the workflow now lives in
+three optional hooks the workflow calls if they are executable:
+
+| Hook                                   | When                              | What goes there                                                                           |
+| -------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------- |
+| `scripts/integration/prepull-extra.sh` | before the suite, outside nix     | `docker pull` of images the suite launches beyond media + studio (a driver, a sink)       |
+| `scripts/integration/prepare.sh`       | before `bun run test:integration` | builds the suite needs on disk — a frontend dist the daemon fetches, an image it launches |
+| `scripts/integration/verify.sh`        | after the suite                   | post-suite checks; absent, a repo with `tests/demo.spec.ts` gets `bun run demo -- check`  |
+
+`bun run test:integration` is the product's claim of what its tier is, and
+ctl's release gate believes it: a `ctl-candidate` dispatch runs this workflow
+and the reported result decides whether `latest` moves.
+
+**Converging a repo.** The gate only takes over a copy whose first line is the
+canonical's marker (`# ctl-dev-kit convention: integration`); a copy without it
+is hand-owned and ungated, which is the state every repo started in. To
+converge: move the repo's bespoke steps into the hooks above, replace the
+workflow with the canonical (product and runner filled in), and prove it with
+`gh workflow run integration.yml --ref <branch>` before merging — the tier
+only fires on `main` and on dispatch, so a PR's checks never run it. Once the
+marker is in, `sync-dev-kit` keeps the copy current and `check:drift` names any
+hand edit.
+
+**Why the pipe is shaped the way it is.** The suite runs inside a shell
+function piped into `tee` so the log survives as an artifact, and the step's
+first line is `set -euo pipefail`. GitHub's default `run:` shell is `bash -e
+{0}` with no pipefail, so without it the step exits with tee's status and a
+dead suite reads green; a `set -euo pipefail` inside the nested `bash -c` body
+cannot reach the pipeline one shell up. Both turnkeys ran a void tier for weeks
+that way. `tee-pipefail.test.ts` guards every convention workflow for it.
+
+The workflow also uploads `test-results/` beside the log, and reaps the
+instances the harness recorded in `$NORSK_LAUNCHED_MANIFEST` after a cancelled
+or failed run, so one killed job cannot poison the next with a held port.