@norskvideo/ctl-dev-kit 0.2.62 → 0.2.64

package/conventions/CLAUDE.core.md

@@ -90,6 +90,12 @@
   opens a PR, and `bot-prs` merges it once checks, smoke, integration and the
   guides gate (build-image) are green on it. A red bot PR is the alarm for a bad
   upstream change; main stays green.
+- **Flaky checks stay out of gates, never out of sight.** The guides gate runs
+  the deterministic tiers; the engine tier runs beside it and reports (summary,
+  warning, cell detail) without gating until norsk-ctl task 57 is fixed. A
+  known-flaky test goes in `.github/quarantine.json` with an owner, a date and
+  an exit condition: `checks` leaves it out of the gating run, then runs it on
+  its own and reports it. Fix it and take it out; don't let an entry age.
 - **A config-driven workflow needs a parity-checked invariant contract.** If this
   product emits its Studio graph dynamically from config, its load-bearing rules
   are documented as stable-ID statements (`INV-<AREA>-<NNN>`) in an

package/conventions/build-docs.yml

@@ -164,6 +164,31 @@ jobs:
           NORSK_LICENSE_V2: ${{ secrets.NORSK_LICENSE_V2 }}
         run: printf '%s' "$NORSK_LICENSE_V2" > "$RUNNER_TEMP/norsk-license.json"
 
+      # Every norsk-ctl daemon starts the node MoQ relay (norsk-moq-relay) unless
+      # told not to, and nothing removes it: it is a node singleton, and the reap
+      # step only takes containers that mount this job's test-temp. So the first
+      # test daemon on a shared runner left one, and every later one adopted it
+      # (probe's smoke ran in relay mode on 2026-10-10; relay's smoke lost 4443
+      # to it). This job's daemons run without the node relay and discovery, so
+      # they neither start nor adopt one, and whatever is already on the host is
+      # named here. It reports and never fails: the leftover may be someone's.
+      - name: Isolate this job from node singletons other jobs left on this host
+        run: |
+          set -uo pipefail
+          echo "NORSK_CTL_MOQ_RELAY_ENABLE=0" >> "$GITHUB_ENV"
+          echo "NORSK_CTL_DISCOVERY_ENABLE=0" >> "$GITHUB_ENV"
+          left="$(docker ps -a --filter label=norsk-ctl.role --format '{{.Names}} {{.Label "norsk-ctl.role"}} {{.Status}} (created {{.CreatedAt}})' 2>/dev/null || true)"
+          if [ -z "$left" ]; then echo "no norsk-ctl containers on this host"; exit 0; fi
+          echo "norsk-ctl containers already on this host (other jobs' running instances are expected):"
+          echo "$left"
+          while IFS= read -r line; do
+            role="$(awk '{print $2}' <<<"$line")"
+            case "$role" in
+              moq-relay|discovery)
+                echo "::warning title=node singleton left on this host::$line -- left by an earlier job (a daemon's auto-start, or a run killed before its reap). This job's daemons run without it and will not adopt it; remove it with docker rm -f if nothing on this host needs it." ;;
+            esac
+          done <<<"$left"
+
       # The norsk-ctl this commit pins (flake.nix ctlVersion, moved by
       # upgrade-latest's bump PR), never S3's moving `latest`: a guide run is then
       # a real repeat, and a red belongs to a commit (ci-fragility proposal A).

package/conventions/build-image.yml

@@ -23,7 +23,8 @@
 #
 # BOT PRs: on a pull request from a bot branch (upgrade-latest/*,
 # sync-dev-kit/*, chore/sync-ctl-packages) the workflow builds both arches and
-# runs the guides gate exactly as a push would, and publishes, records and
+# runs the guides gate exactly as a push would (the deterministic tiers gate;
+# the engine tier reports beside it), and publishes, records and
 # reports nothing. A bump reaches main only once that has passed on it, and
 # bot-prs merges on it (ci-workflows docs/dashboard/ci-fragility.md, proposal
 # B): a dev-kit sync that brought a newer copy of THIS file broke
@@ -302,15 +303,13 @@ jobs:
   gate-guides:
     needs: label
     runs-on: x64
-    # One guide run per repo at a time, nightly and publish gate alike: both
-    # launch the same instance ids on the x64 pool's shared docker daemon, and
-    # ctl names each compose project after its instance id. Queue, never cancel.
-    concurrency:
-      group: doc-guides-${{ github.repository }}
-      cancel-in-progress: false
+    # No doc-guides concurrency group: the deterministic tiers launch no
+    # instances, so there is nothing to serialise, and a pending job in a group
+    # is cancelled by the next one queued. guides-engine takes the group.
     outputs:
       guides: ${{ steps.plan.outputs.guides }}
       reason: ${{ steps.plan.outputs.reason }}
+      engine: ${{ steps.plan.outputs.engine }}
     steps:
       # Runs BEFORE checkout, and does two things: points the harness's temp base
       # out of the workspace for every later step in this job, and clears whatever
@@ -390,6 +389,10 @@ jobs:
             ')"
             if [ "$decision" = "run" ]; then
               echo "guides=run" >> "$GITHUB_OUTPUT"
+              # The engine tier runs in guides-engine, beside this gate.
+              if jq -e '.scripts["docs:guides:engine"]' package.json >/dev/null; then
+                echo "engine=run" >> "$GITHUB_OUTPUT"
+              fi
             else
               skip "${decision#skip$'\t'}"
             fi
@@ -401,6 +404,31 @@ jobs:
           NORSK_LICENSE_V2: ${{ secrets.NORSK_LICENSE_V2 }}
         run: printf '%s' "$NORSK_LICENSE_V2" > "$RUNNER_TEMP/norsk-license.json"
 
+      # Every norsk-ctl daemon starts the node MoQ relay (norsk-moq-relay) unless
+      # told not to, and nothing removes it: it is a node singleton, and the reap
+      # step only takes containers that mount this job's test-temp. So the first
+      # test daemon on a shared runner left one, and every later one adopted it
+      # (probe's smoke ran in relay mode on 2026-10-10; relay's smoke lost 4443
+      # to it). This job's daemons run without the node relay and discovery, so
+      # they neither start nor adopt one, and whatever is already on the host is
+      # named here. It reports and never fails: the leftover may be someone's.
+      - name: Isolate this job from node singletons other jobs left on this host
+        run: |
+          set -uo pipefail
+          echo "NORSK_CTL_MOQ_RELAY_ENABLE=0" >> "$GITHUB_ENV"
+          echo "NORSK_CTL_DISCOVERY_ENABLE=0" >> "$GITHUB_ENV"
+          left="$(docker ps -a --filter label=norsk-ctl.role --format '{{.Names}} {{.Label "norsk-ctl.role"}} {{.Status}} (created {{.CreatedAt}})' 2>/dev/null || true)"
+          if [ -z "$left" ]; then echo "no norsk-ctl containers on this host"; exit 0; fi
+          echo "norsk-ctl containers already on this host (other jobs' running instances are expected):"
+          echo "$left"
+          while IFS= read -r line; do
+            role="$(awk '{print $2}' <<<"$line")"
+            case "$role" in
+              moq-relay|discovery)
+                echo "::warning title=node singleton left on this host::$line -- left by an earlier job (a daemon's auto-start, or a run killed before its reap). This job's daemons run without it and will not adopt it; remove it with docker rm -f if nothing on this host needs it." ;;
+            esac
+          done <<<"$left"
+
       # The norsk-ctl this commit pins (flake.nix ctlVersion, moved by
       # upgrade-latest's bump PR), never S3's moving `latest`: a guide run is then
       # a real repeat, and a red belongs to a commit (ci-fragility proposal A).
@@ -433,13 +461,15 @@ jobs:
             ./scripts/doc-guide/prepull-extra.sh
           fi
 
-      # Rebuild every workspace the manual depends on (a guide's engine tier packs
-      # the freshly-built dashboard + frontend dist into the launched template),
-      # then run the product's doc-guide tiers and assemble the manual.
-      # regen-manual.sh owns the slug list and which tiers run; with-display.sh
-      # wraps the whole run in an X display for headless chromium (the engine tier
-      # drives a raw chromium that needs one).
-      - name: Regenerate the illustrated manual (all tiers)
+      # The deterministic tiers (fixture and frontend: no daemon, no instances),
+      # then the manual assembled from what they captured. These gate. The engine
+      # tier runs on its own below: it stands up real instances, is known-flaky
+      # on container DNS (norsk-ctl task 57), and reports without gating until
+      # that is fixed (ci-workflows docs/dashboard/ci-fragility.md, proposal E).
+      # regen-manual.sh owns the slug list and honours SKIP_ENGINE; it rebuilds
+      # every workspace the manual depends on first, and with-display.sh gives
+      # headless chromium an X display.
+      - name: Regenerate the illustrated manual (deterministic tiers)
         if: ${{ steps.plan.outputs.guides == 'run' }}
         env:
           NORSK_CTL_BINARY: ${{ runner.temp }}/norsk-ctl
@@ -453,6 +483,7 @@ jobs:
           # them binding no host ports — so concurrent doc-guide instances can't
           # collide on the product's deterministic host-port bands.
           NORSK_TEST_NET: direct
+          SKIP_ENGINE: "1"
         run: |
           nix develop .#build --command bash -c '
             set -euo pipefail
@@ -486,6 +517,228 @@ jobs:
             docker rm -f $s
           fi
 
+  # The engine tier of the guides, beside the gate rather than in it (see the
+  # deterministic step above): it runs whenever the gate runs the guides and the
+  # product has an engine tier, and reports -- a warning, the run summary and
+  # the build-image cell's detail -- but publish does not wait on it, and a
+  # failure leaves the job green so bot-prs can still merge. To make it gate
+  # again once norsk-ctl task 57 is fixed, add it to publish's needs and drop
+  # continue-on-error from the engine step.
+  guides-engine:
+    needs: [label, gate-guides]
+    # !cancelled(): a red deterministic gate must not hide the engine tier's
+    # result. The plan's output is set before any guide runs.
+    if: ${{ !cancelled() && needs.gate-guides.outputs.engine == 'run' }}
+    runs-on: x64
+    # One engine-tier run per repo at a time, nightly (publish-docs.yml) and
+    # this alike: both launch the same instance ids on the x64 pool's shared
+    # docker daemon, and ctl names each compose project after its instance id.
+    # Queue, never cancel.
+    concurrency:
+      group: doc-guides-${{ github.repository }}
+      cancel-in-progress: false
+    outputs:
+      result: ${{ steps.engine.outcome }}
+    steps:
+      # Runs BEFORE checkout, and does two things: points the harness's temp base
+      # out of the workspace for every later step in this job, and clears whatever
+      # the old repo-local default left behind on this runner. The body carries the
+      # why. It fails the job only when legacy litter survives -- which would take
+      # the checkout down a step later anyway, but report nothing useful.
+      - name: Route test-temp out of the workspace (pre-checkout)
+        run: |
+          set -uo pipefail
+          # These runners are docker-OUTSIDE-of-docker, so a bind-mount source the
+          # HOST daemon has not seen before is created by dockerd, as ROOT, on the
+          # host -- even when the launch passes --container-user (reproduced
+          # 2026-09-04). The harness's default base is repo-local for an OrbStack
+          # inotify constraint that binds macOS dev and nothing here, so every run
+          # seeded the CHECKOUT with root-owned dirs a non-root `git clean -ffdx`
+          # cannot remove. That took commentary's checks AND integration red on
+          # 2026-09-03. Use the sanctioned override, per-run and outside the
+          # checkout -- as 8ff17a8d did in June for one debug workflow, and stopped.
+          base="$RUNNER_TEMP/nctt-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"
+          mkdir -p "$base"
+          echo "NORSK_CTL_TEST_TMP=$base" >> "$GITHUB_ENV"
+          # Yesterday's bases hold dockerd-created root dirs, so a non-root rm
+          # EACCESes. They cannot wedge a checkout, but they do fill the disk.
+          for old in $(find "$RUNNER_TEMP" -maxdepth 1 -name 'nctt-*' -mtime +0 -printf '%f\n' 2>/dev/null || true); do
+            docker run --rm --user 0:0 -v "$RUNNER_TEMP":/t alpine sh -c 'rm -rf -- "/t/$1"' sh "$old" >/dev/null 2>&1 || true
+          done
+          # LEGACY: a runner that ran the old default still carries a workspace
+          # test-temp, and a checkout with clean:true dies on it. Reap the holders
+          # first -- rm cannot unlink a live mountpoint even as root (EBUSY), and a
+          # RESTARTING leaked container re-creates the path as root seconds after
+          # any clean. Match on mount SOURCE, not a label: the holders are untracked
+          # by construction, because a killed run labels nothing.
+          tt="$GITHUB_WORKSPACE/test-temp"
+          [ -d "$tt" ] || exit 0
+          stuck=""
+          for c in $(docker ps -aq 2>/dev/null || true); do
+            if docker inspect -f '{{range .Mounts}}{{println .Source}}{{end}}' "$c" 2>/dev/null | grep -q "^$tt/"; then
+              stuck="$stuck $c"
+            fi
+          done
+          if [ -n "$stuck" ]; then
+            echo "reaping containers holding mounts under $tt:$stuck"
+            docker rm -f $stuck || true
+          fi
+          # The DIRECTORY, not just its contents: an empty but root-owned base
+          # still EACCESes a later mkdtemp. Mounting the PARENT is what lets a root
+          # container unlink the leaf.
+          docker run --rm --user 0:0 -v "$GITHUB_WORKSPACE":/w alpine \
+            sh -c 'rm -rf /w/test-temp' || rm -rf "$tt" || true
+          # Fail HERE if it survived. The checkout fails on it either way, but
+          # reports only an EACCES rmdir with no clue what was holding the path.
+          if [ -e "$tt" ]; then
+            echo "::error::$tt survived the pre-checkout clean"
+            ls -lan "$tt" || true
+            exit 1
+          fi
+
+      - uses: actions/checkout@v5
+        with:
+          clean: true
+
+      - name: Write the Norsk license (from the org secret)
+        env:
+          NORSK_LICENSE_V2: ${{ secrets.NORSK_LICENSE_V2 }}
+        run: printf '%s' "$NORSK_LICENSE_V2" > "$RUNNER_TEMP/norsk-license.json"
+      # Every norsk-ctl daemon starts the node MoQ relay (norsk-moq-relay) unless
+      # told not to, and nothing removes it: it is a node singleton, and the reap
+      # step only takes containers that mount this job's test-temp. So the first
+      # test daemon on a shared runner left one, and every later one adopted it
+      # (probe's smoke ran in relay mode on 2026-10-10; relay's smoke lost 4443
+      # to it). This job's daemons run without the node relay and discovery, so
+      # they neither start nor adopt one, and whatever is already on the host is
+      # named here. It reports and never fails: the leftover may be someone's.
+      - name: Isolate this job from node singletons other jobs left on this host
+        run: |
+          set -uo pipefail
+          echo "NORSK_CTL_MOQ_RELAY_ENABLE=0" >> "$GITHUB_ENV"
+          echo "NORSK_CTL_DISCOVERY_ENABLE=0" >> "$GITHUB_ENV"
+          left="$(docker ps -a --filter label=norsk-ctl.role --format '{{.Names}} {{.Label "norsk-ctl.role"}} {{.Status}} (created {{.CreatedAt}})' 2>/dev/null || true)"
+          if [ -z "$left" ]; then echo "no norsk-ctl containers on this host"; exit 0; fi
+          echo "norsk-ctl containers already on this host (other jobs' running instances are expected):"
+          echo "$left"
+          while IFS= read -r line; do
+            role="$(awk '{print $2}' <<<"$line")"
+            case "$role" in
+              moq-relay|discovery)
+                echo "::warning title=node singleton left on this host::$line -- left by an earlier job (a daemon's auto-start, or a run killed before its reap). This job's daemons run without it and will not adopt it; remove it with docker rm -f if nothing on this host needs it." ;;
+            esac
+          done <<<"$left"
+
+      # The norsk-ctl this commit pins (flake.nix ctlVersion, moved by
+      # upgrade-latest's bump PR), never S3's moving `latest`: a guide run is then
+      # a real repeat, and a red belongs to a commit (ci-fragility proposal A).
+      - name: Download the pinned norsk-ctl binary (flake.nix ctlVersion)
+        run: |
+          set -euo pipefail
+          S3="https://s3.eu-west-1.amazonaws.com/norsk.video/norsk-ctl"
+          ver="$( { sed -nE 's/^[[:space:]]*ctlVersion = "([^"]+)";.*/\1/p' flake.nix 2>/dev/null || true; } | head -1)"
+          [ -n "$ver" ] || { echo "::error::flake.nix pins no ctlVersion. CI runs the norsk-ctl this commit pins (upgrade-latest moves it); add ctlVersion = \"<version>\"; to flake.nix."; exit 1; }
+          echo "norsk-ctl pinned by flake.nix ctlVersion -> $ver"
+          curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 "$S3/$ver/norsk-ctl-$ver-linux-x64" -o "$RUNNER_TEMP/norsk-ctl"
+          chmod +x "$RUNNER_TEMP/norsk-ctl"
+          "$RUNNER_TEMP/norsk-ctl" --version || true
+      # Cold-runner first pulls overrun the harness's per-test launch timeouts;
+      # pre-pull the universal media + studio images (every product pins them in
+      # manifest.seed.json) so compose/run hit local images. A product needing
+      # more (e.g. a WHIP driver) pulls them in scripts/doc-guide/prepull-extra.sh.
+      - name: Pre-pull the media + studio images
+        run: |
+          set -euo pipefail
+          for img in "$(jq -r '.latest.media' manifest.seed.json)" "$(jq -r '.latest.studio' manifest.seed.json)"; do
+            echo "pre-pulling $img"
+            docker pull "$img"
+          done
+          if [ -x scripts/doc-guide/prepull-extra.sh ]; then
+            echo "running scripts/doc-guide/prepull-extra.sh"
+            ./scripts/doc-guide/prepull-extra.sh
+          fi
+      - name: Install and build the workspaces the guides drive
+        run: |
+          nix develop .#build --command bash -c '
+            set -euo pipefail
+            find . -name node_modules -type d -prune -exec rm -rf {} + 2>/dev/null || true
+            bun install --frozen-lockfile
+            bun run build:no-lint
+          '
+
+      # The engine tier: real instances, driven in a browser. Reported, not
+      # gating (see the deterministic step). A non-zero exit is forgiven only
+      # when the tier's own results say a teardown hook timed out after every
+      # guide's test passed (doc-guide/engine-verdict.ts), as regen-manual.sh
+      # forgives it. A product with no docs:guides:engine script has no engine
+      # tier. Runs on the workspaces the step before it installed and built.
+      - name: Run the engine-tier guides (reported, not gating)
+        id: engine
+        continue-on-error: true
+        env:
+          NORSK_CTL_BINARY: ${{ runner.temp }}/norsk-ctl
+          NORSK_LICENSE_FILE: ${{ runner.temp }}/norsk-license.json
+          # This runner launches Studio as a HOST sibling (DooD), so the harness
+          # reaches host-published ports via the host-gateway alias, not
+          # localhost — same as the integration suite.
+          NORSK_TEST_HOST: host.docker.internal
+          # Reach launched instances over norsk-net by service DNS (as the
+          # integration tier already does) AND, on a ctl that supports it, launch
+          # them binding no host ports — so concurrent doc-guide instances can't
+          # collide on the product's deterministic host-port bands.
+          NORSK_TEST_NET: direct
+        run: |
+          nix develop .#build --command bash -c '
+            set -euo pipefail
+            if ! jq -e ".scripts[\"docs:guides:engine\"]" package.json >/dev/null; then
+              echo "no docs:guides:engine script: this product has no engine tier"
+              exit 0
+            fi
+            export DOCS_GENERATE=1
+            log="$(mktemp)"
+            if bash scripts/with-display.sh bun run docs:guides:engine >"$log" 2>&1; then
+              cat "$log"
+            else
+              cat "$log"
+              verdict="$(bun -e "console.log(Bun.resolveSync(\"@norskvideo/ctl-dev-kit/doc-guide/engine-verdict\", process.cwd()))")"
+              bun "$verdict" "$log"
+            fi
+          '
+
+      # Said three ways, so a failure is never silent: a warning on the run, a
+      # line in its summary, and the cell's detail (the notify job).
+      - name: Report the engine tier (not gating)
+        if: ${{ !cancelled() }}
+        env:
+          OUTCOME: ${{ steps.engine.outcome }}
+        run: |
+          case "$OUTCOME" in
+            success) echo "engine-tier guides passed (or this product has none)" ;;
+            failure)
+              echo "::warning title=engine-tier guides failed (not gating)::the engine tier does not gate until norsk-ctl task 57 is fixed. See the engine step's log."
+              printf '### Engine-tier guides: failed (not gating)\n\nThe deterministic guide tiers gate; the engine tier only reports until its container networking is fixed (norsk-ctl task 57). See the step log.\n' >> "$GITHUB_STEP_SUMMARY" ;;
+            *) echo "engine tier: ${OUTCOME:-not run}" ;;
+          esac
+
+      - name: Reap the containers this job launched
+        # Whatever the tests did. A teardown cut short by a test timeout leaves
+        # its instance running on the shared Docker daemon (see JOB_REAP_STEP in
+        # conventions/workflow-shell.ts); everything this job launched mounts its
+        # own test-temp, so that is what goes.
+        if: always()
+        run: |
+          base="${NORSK_CTL_TEST_TMP:-}"
+          tt="$GITHUB_WORKSPACE/test-temp"
+          s=""
+          for c in $(docker ps -aq); do
+            src="$(docker inspect -f '{{range .Mounts}}{{println .Source}}{{end}}' "$c" 2>/dev/null)" || continue
+            if { [ -n "$base" ] && grep -q "^$base/" <<<"$src"; } || grep -q "^$tt/" <<<"$src"; then s="$s $c"; fi
+          done
+          if [ -n "$s" ]; then
+            echo "::warning::removing containers this job left running:$s"
+            docker rm -f $s
+          fi
+
   # Combine the per-arch images into a multi-arch manifest and write the two
   # published tags. Skipped for a build-only product (empty PUBLISH_IMAGE) and on
   # a dry run.
@@ -733,7 +986,15 @@ jobs:
   # the pony/emoji from the dispatched result. always() so a red or manual run
   # still reports.
   notify:
-    needs: [label, build, gate-checks, gate-guides, publish, record]
+    needs: [
+      label,
+      build,
+      gate-checks,
+      gate-guides,
+      guides-engine,
+      publish,
+      record
+    ]
     if: ${{ !cancelled() && github.event_name != 'pull_request' }}
     runs-on: x64
     steps:
@@ -796,9 +1057,14 @@ jobs:
       - uses: actions/checkout@v5
         with:
           clean: true
+      # guides-engine reports in the detail below and never sets the state: a
+      # failed engine tier is not a failed build, nor is one cancelled in its
+      # concurrency queue.
       - id: meta
+        env:
+          NEEDS: ${{ toJSON(needs) }}
         run: |
-          if [ "${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}" = "true" ]; then
+          if jq -e 'del(.["guides-engine"]) | [.[].result] | any(. == "failure" or . == "cancelled")' <<<"$NEEDS" >/dev/null; then
             echo "state=failure" >> "$GITHUB_OUTPUT"
           else
             echo "state=success" >> "$GITHUB_OUTPUT"
@@ -811,7 +1077,8 @@ jobs:
           status: ${{ steps.meta.outputs.state }}
           # Only where something is published: on a build-only repo the gate has
           # nothing to gate, and saying so on every build is noise.
-          detail: "${{ needs.label.outputs.publish == 'true' && needs.gate-guides.outputs.guides == 'skip' && format('guides not gating: {0}', needs.gate-guides.outputs.reason) || '' }}"
+          # And when the engine-tier guides failed: they report, never gate.
+          detail: "${{ needs.label.outputs.publish == 'true' && needs.gate-guides.outputs.guides == 'skip' && format('guides not gating: {0}', needs.gate-guides.outputs.reason) || needs.guides-engine.outputs.result == 'failure' && 'engine-tier guides failed (not gating; norsk-ctl task 57)' || '' }}"
       - name: Reap the containers this job launched
         # Whatever the tests did. A teardown cut short by a test timeout leaves
         # its instance running on the shared Docker daemon (see JOB_REAP_STEP in

package/conventions/checks.sdk-app.yml

@@ -127,6 +127,9 @@ jobs:
 
   quality:
     runs-on: x64
+    outputs:
+      quarantined: ${{ steps.quarantine.outputs.quarantined }}
+      quarantine_failed: ${{ steps.quarantine.outputs.quarantine_failed }}
     steps:
       # Runs BEFORE checkout, and does two things: points the harness's temp base
       # out of the workspace for every later step in this job, and clears whatever
@@ -200,7 +203,11 @@ jobs:
             bun install --frozen-lockfile
             bun run lint
             bun run typecheck
-            bun run test:unit
+            # Known-flaky tests (.github/quarantine.json) leave the gating run
+            # here and run on their own in the next step (conventions/quarantine.ts).
+            q=node_modules/@norskvideo/ctl-dev-kit/conventions/quarantine.ts
+            bun "$q" check
+            BUN_OPTIONS="$(bun "$q" bun-options)" bun run test:unit
             bun run docs:check
             # The tree must resolve ONE zod 4 (conventions/zod-singleton.ts).
             # studio guards this in-repo but never shipped the guard; products
@@ -208,6 +215,16 @@ jobs:
             # in here and breaks a schema handed across the instance boundary.
             bun node_modules/@norskvideo/ctl-dev-kit/conventions/zod-singleton.ts
           '
+      # The quarantined tests, each run on its own and reported: a summary
+      # table, a warning per failure, and counts for the checks cell's detail.
+      # It never fails the job -- that is what quarantine means -- and it runs
+      # even when the gate is red, so the flake's record has no holes. Nothing
+      # listed, nothing run.
+      - name: Quarantined tests (run and reported, not gating)
+        id: quarantine
+        if: ${{ !cancelled() }}
+        continue-on-error: true
+        run: nix develop .#build --command bun node_modules/@norskvideo/ctl-dev-kit/conventions/quarantine.ts run
       # licenseTemplate.json records the entry a licence is minted from: this
       # product at the image CI builds. One naming another image mints a licence
       # that entitles nothing, first seen as a refused launch at the customer.
@@ -335,6 +352,8 @@ jobs:
           product: __PRODUCT__
           pipeline: checks
           status: ${{ steps.meta.outputs.state }}
+          # Quarantined tests leave the gate, never the board: say how many.
+          detail: "${{ needs.quality.outputs.quarantined > 0 && format('quarantined: {0} test(s), {1} not passing this run (not gating; .github/quarantine.json)', needs.quality.outputs.quarantined, needs.quality.outputs.quarantine_failed) || '' }}"
       - name: Reap the containers this job launched
         # Whatever the tests did. A teardown cut short by a test timeout leaves
         # its instance running on the shared Docker daemon (see JOB_REAP_STEP in

package/conventions/checks.yml

@@ -130,6 +130,9 @@ jobs:
 
   quality:
     runs-on: x64
+    outputs:
+      quarantined: ${{ steps.quarantine.outputs.quarantined }}
+      quarantine_failed: ${{ steps.quarantine.outputs.quarantine_failed }}
     steps:
       # Runs BEFORE checkout, and does two things: points the harness's temp base
       # out of the workspace for every later step in this job, and clears whatever
@@ -203,7 +206,11 @@ jobs:
             bun install --frozen-lockfile
             bun run lint
             bun run typecheck
-            bun run test:unit
+            # Known-flaky tests (.github/quarantine.json) leave the gating run
+            # here and run on their own in the next step (conventions/quarantine.ts).
+            q=node_modules/@norskvideo/ctl-dev-kit/conventions/quarantine.ts
+            bun "$q" check
+            BUN_OPTIONS="$(bun "$q" bun-options)" bun run test:unit
             bun run docs:check
             # The tree must resolve ONE zod 4 (conventions/zod-singleton.ts).
             # studio guards this in-repo but never shipped the guard; products
@@ -211,6 +218,16 @@ jobs:
             # in here and breaks a schema handed across the instance boundary.
             bun node_modules/@norskvideo/ctl-dev-kit/conventions/zod-singleton.ts
           '
+      # The quarantined tests, each run on its own and reported: a summary
+      # table, a warning per failure, and counts for the checks cell's detail.
+      # It never fails the job -- that is what quarantine means -- and it runs
+      # even when the gate is red, so the flake's record has no holes. Nothing
+      # listed, nothing run.
+      - name: Quarantined tests (run and reported, not gating)
+        id: quarantine
+        if: ${{ !cancelled() }}
+        continue-on-error: true
+        run: nix develop .#build --command bun node_modules/@norskvideo/ctl-dev-kit/conventions/quarantine.ts run
 
       # The two halves of one licence guarantee, run together because they only
       # work as a pair: check-template asserts this repo RECORDS the entitlement
@@ -356,6 +373,8 @@ jobs:
           product: __PRODUCT__
           pipeline: checks
           status: ${{ steps.meta.outputs.state }}
+          # Quarantined tests leave the gate, never the board: say how many.
+          detail: "${{ needs.quality.outputs.quarantined > 0 && format('quarantined: {0} test(s), {1} not passing this run (not gating; .github/quarantine.json)', needs.quality.outputs.quarantined, needs.quality.outputs.quarantine_failed) || '' }}"
       - name: Reap the containers this job launched
         # Whatever the tests did. A teardown cut short by a test timeout leaves
         # its instance running on the shared Docker daemon (see JOB_REAP_STEP in

package/conventions/integration.yml

@@ -283,6 +283,31 @@ jobs:
           NORSK_LICENSE_V2: ${{ secrets.NORSK_LICENSE_V2 }}
         run: printf '%s' "$NORSK_LICENSE_V2" > "$RUNNER_TEMP/norsk-license.json"
 
+      # Every norsk-ctl daemon starts the node MoQ relay (norsk-moq-relay) unless
+      # told not to, and nothing removes it: it is a node singleton, and the reap
+      # step only takes containers that mount this job's test-temp. So the first
+      # test daemon on a shared runner left one, and every later one adopted it
+      # (probe's smoke ran in relay mode on 2026-10-10; relay's smoke lost 4443
+      # to it). This job's daemons run without the node relay and discovery, so
+      # they neither start nor adopt one, and whatever is already on the host is
+      # named here. It reports and never fails: the leftover may be someone's.
+      - name: Isolate this job from node singletons other jobs left on this host
+        run: |
+          set -uo pipefail
+          echo "NORSK_CTL_MOQ_RELAY_ENABLE=0" >> "$GITHUB_ENV"
+          echo "NORSK_CTL_DISCOVERY_ENABLE=0" >> "$GITHUB_ENV"
+          left="$(docker ps -a --filter label=norsk-ctl.role --format '{{.Names}} {{.Label "norsk-ctl.role"}} {{.Status}} (created {{.CreatedAt}})' 2>/dev/null || true)"
+          if [ -z "$left" ]; then echo "no norsk-ctl containers on this host"; exit 0; fi
+          echo "norsk-ctl containers already on this host (other jobs' running instances are expected):"
+          echo "$left"
+          while IFS= read -r line; do
+            role="$(awk '{print $2}' <<<"$line")"
+            case "$role" in
+              moq-relay|discovery)
+                echo "::warning title=node singleton left on this host::$line -- left by an earlier job (a daemon's auto-start, or a run killed before its reap). This job's daemons run without it and will not adopt it; remove it with docker rm -f if nothing on this host needs it." ;;
+            esac
+          done <<<"$left"
+
       # Pick the binary for THIS runner's architecture: the GPU box is aarch64,
       # the x64 pool is x86_64, and the wrong one dies with "Exec format error"
       # deep in the suite. No `|| true` on --version: a bad binary must fail here.

package/conventions/publish-docs.yml

@@ -22,6 +22,9 @@
 #
 # Nightly-only by design: a guide's engine tier stands up real instances (slow,
 # and subject to teardown flakes), so it does not ride every push. The
+# deterministic tiers decide this workflow's result; the engine tier reports
+# without failing it (norsk-ctl task 57), and a failed engine tier keeps
+# yesterday's manual rather than publishing one with gaps. The
 # licensed-runner setup mirrors integration.yml — ctl comes from the released
 # binary via NORSK_CTL_BINARY, Docker is used directly on the x64 pool.
 name: publish-docs
@@ -46,6 +49,8 @@ concurrency:
 jobs:
   publish:
     runs-on: x64
+    outputs:
+      engine: ${{ steps.engine.outcome }}
     # One guide run per repo at a time, nightly and publish gate alike: both
     # launch the same instance ids on the x64 pool's shared docker daemon, and
     # ctl names each compose project after its instance id. Queue, never cancel.
@@ -174,6 +179,31 @@ jobs:
           NORSK_LICENSE_V2: ${{ secrets.NORSK_LICENSE_V2 }}
         run: printf '%s' "$NORSK_LICENSE_V2" > "$RUNNER_TEMP/norsk-license.json"
 
+      # Every norsk-ctl daemon starts the node MoQ relay (norsk-moq-relay) unless
+      # told not to, and nothing removes it: it is a node singleton, and the reap
+      # step only takes containers that mount this job's test-temp. So the first
+      # test daemon on a shared runner left one, and every later one adopted it
+      # (probe's smoke ran in relay mode on 2026-10-10; relay's smoke lost 4443
+      # to it). This job's daemons run without the node relay and discovery, so
+      # they neither start nor adopt one, and whatever is already on the host is
+      # named here. It reports and never fails: the leftover may be someone's.
+      - name: Isolate this job from node singletons other jobs left on this host
+        run: |
+          set -uo pipefail
+          echo "NORSK_CTL_MOQ_RELAY_ENABLE=0" >> "$GITHUB_ENV"
+          echo "NORSK_CTL_DISCOVERY_ENABLE=0" >> "$GITHUB_ENV"
+          left="$(docker ps -a --filter label=norsk-ctl.role --format '{{.Names}} {{.Label "norsk-ctl.role"}} {{.Status}} (created {{.CreatedAt}})' 2>/dev/null || true)"
+          if [ -z "$left" ]; then echo "no norsk-ctl containers on this host"; exit 0; fi
+          echo "norsk-ctl containers already on this host (other jobs' running instances are expected):"
+          echo "$left"
+          while IFS= read -r line; do
+            role="$(awk '{print $2}' <<<"$line")"
+            case "$role" in
+              moq-relay|discovery)
+                echo "::warning title=node singleton left on this host::$line -- left by an earlier job (a daemon's auto-start, or a run killed before its reap). This job's daemons run without it and will not adopt it; remove it with docker rm -f if nothing on this host needs it." ;;
+            esac
+          done <<<"$left"
+
       # The norsk-ctl this commit pins (flake.nix ctlVersion, moved by
       # upgrade-latest's bump PR), never S3's moving `latest`: a guide run is then
       # a real repeat, and a red belongs to a commit (ci-fragility proposal A).
@@ -204,13 +234,15 @@ jobs:
             ./scripts/doc-guide/prepull-extra.sh
           fi
 
-      # Rebuild every workspace the manual depends on (a guide's engine tier packs
-      # the freshly-built dashboard + frontend dist into the launched template),
-      # then run the product's doc-guide tiers and assemble the manual.
-      # regen-manual.sh owns the slug list and which tiers run; with-display.sh
-      # wraps the whole run in an X display for headless chromium (the engine tier
-      # drives a raw chromium that needs one).
-      - name: Regenerate the illustrated manual (all tiers)
+      # The deterministic tiers (fixture and frontend: no daemon, no instances),
+      # then the manual assembled from what they captured. These gate. The engine
+      # tier runs on its own below: it stands up real instances, is known-flaky
+      # on container DNS (norsk-ctl task 57), and reports without gating until
+      # that is fixed (ci-workflows docs/dashboard/ci-fragility.md, proposal E).
+      # regen-manual.sh owns the slug list and honours SKIP_ENGINE; it rebuilds
+      # every workspace the manual depends on first, and with-display.sh gives
+      # headless chromium an X display.
+      - name: Regenerate the illustrated manual (deterministic tiers)
         env:
           NORSK_CTL_BINARY: ${{ runner.temp }}/norsk-ctl
           NORSK_LICENSE_FILE: ${{ runner.temp }}/norsk-license.json
@@ -223,6 +255,7 @@ jobs:
           # them binding no host ports — so concurrent doc-guide instances can't
           # collide on the product's deterministic host-port bands.
           NORSK_TEST_NET: direct
+          SKIP_ENGINE: "1"
         run: |
           nix develop .#build --command bash -c '
             set -euo pipefail
@@ -237,11 +270,71 @@ jobs:
             bash scripts/with-display.sh scripts/doc-guide/regen-manual.sh
           '
 
+      # The engine tier: real instances, driven in a browser. Reported, not
+      # gating (see the deterministic step). A non-zero exit is forgiven only
+      # when the tier's own results say a teardown hook timed out after every
+      # guide's test passed (doc-guide/engine-verdict.ts), as regen-manual.sh
+      # forgives it. A product with no docs:guides:engine script has no engine
+      # tier. Runs on the workspaces the step before it installed and built.
+      - name: Run the engine-tier guides (reported, not gating)
+        id: engine
+        continue-on-error: true
+        env:
+          NORSK_CTL_BINARY: ${{ runner.temp }}/norsk-ctl
+          NORSK_LICENSE_FILE: ${{ runner.temp }}/norsk-license.json
+          # This runner launches Studio as a HOST sibling (DooD), so the harness
+          # reaches host-published ports via the host-gateway alias, not
+          # localhost — same as the integration suite.
+          NORSK_TEST_HOST: host.docker.internal
+          # Reach launched instances over norsk-net by service DNS (as the
+          # integration tier already does) AND, on a ctl that supports it, launch
+          # them binding no host ports — so concurrent doc-guide instances can't
+          # collide on the product's deterministic host-port bands.
+          NORSK_TEST_NET: direct
+        run: |
+          nix develop .#build --command bash -c '
+            set -euo pipefail
+            if ! jq -e ".scripts[\"docs:guides:engine\"]" package.json >/dev/null; then
+              echo "no docs:guides:engine script: this product has no engine tier"
+              exit 0
+            fi
+            export DOCS_GENERATE=1
+            log="$(mktemp)"
+            if bash scripts/with-display.sh bun run docs:guides:engine >"$log" 2>&1; then
+              cat "$log"
+            else
+              cat "$log"
+              verdict="$(bun -e "console.log(Bun.resolveSync(\"@norskvideo/ctl-dev-kit/doc-guide/engine-verdict\", process.cwd()))")"
+              bun "$verdict" "$log"
+            fi
+          '
+
+      # Only after a passing engine tier: the deterministic step assembled a
+      # manual with the engine pages as gaps.
+      - name: Assemble the manual with the engine captures
+        if: ${{ steps.engine.outcome == 'success' }}
+        run: nix develop .#build --command bun run docs:manual
+
+      # Said three ways, so a failure is never silent: a warning on the run, a
+      # line in its summary, and the cell's detail (the notify job).
+      - name: Report the engine tier (not gating)
+        if: ${{ !cancelled() }}
+        env:
+          OUTCOME: ${{ steps.engine.outcome }}
+        run: |
+          case "$OUTCOME" in
+            success) echo "engine-tier guides passed (or this product has none)" ;;
+            failure)
+              echo "::warning title=engine-tier guides failed (not gating)::the engine tier does not gate until norsk-ctl task 57 is fixed. The manual was not republished: yesterday's stays. See the engine step's log."
+              printf '### Engine-tier guides: failed (not gating)\n\nThe deterministic guide tiers gate; the engine tier only reports until its container networking is fixed (norsk-ctl task 57). The manual was not republished: yesterday's stays. See the step log.\n' >> "$GITHUB_STEP_SUMMARY" ;;
+            *) echo "engine tier: ${OUTCOME:-not run}" ;;
+          esac
+
       # Publish the assembled manual as the moving docs-latest release asset.
       # gh is on PATH inside the nix shell; `upload --clobber` replaces the
       # same-named asset in place, so the release stays latest-only.
       - name: Publish manual.html to the moving docs-latest release
-        if: ${{ github.event_name != 'workflow_dispatch' || inputs.publish }}
+        if: ${{ steps.engine.outcome == 'success' && (github.event_name != 'workflow_dispatch' || inputs.publish) }}
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
@@ -290,7 +383,7 @@ jobs:
       # exports map, so this works whether ctl-dev-kit is the published package or
       # a workspace link (scripts/set-ctl-source.ts local mode).
       - name: Publish the versioned docs bundle and move the pointer
-        if: ${{ github.event_name != 'workflow_dispatch' || inputs.publish }}
+        if: ${{ steps.engine.outcome == 'success' && (github.event_name != 'workflow_dispatch' || inputs.publish) }}
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           # How many bundles to keep on the release, the current one included.
@@ -325,7 +418,7 @@ jobs:
       # key through ci-workflows' registry, so a new product needs no new mapping
       # here. Not wrapped in nix -- plain curl, and the quoting stays readable.
       - name: Tell norsk-ctl a new bundle is published
-        if: ${{ github.event_name != 'workflow_dispatch' || inputs.publish }}
+        if: ${{ steps.engine.outcome == 'success' && (github.event_name != 'workflow_dispatch' || inputs.publish) }}
         env:
           DISPATCH_TOKEN: ${{ secrets.CI_DISPATCH_TOKEN }}
         run: |
@@ -437,6 +530,8 @@ jobs:
           product: __PRODUCT__
           pipeline: docs
           status: ${{ steps.meta.outputs.state }}
+          # The engine tier reports, never gates; a failure keeps yesterday's manual.
+          detail: "${{ needs.publish.outputs.engine == 'failure' && 'engine-tier guides failed (not gating; norsk-ctl task 57): the manual was not republished' || '' }}"
       - name: Reap the containers this job launched
         # Whatever the tests did. A teardown cut short by a test timeout leaves
         # its instance running on the shared Docker daemon (see JOB_REAP_STEP in

package/conventions/quarantine.ts

@@ -0,0 +1,209 @@
+#!/usr/bin/env bun
+
+// Known-flaky tests, kept out of the gate but not out of the run (ci-fragility
+// proposal E). A test that fails one run in three on a loaded runner turns the
+// whole checks job red; a person re-runs it, and a real red hides among the
+// flaky ones. Quarantine takes it out of the gating run, runs it on its own
+// straight after, and reports the result without failing the job.
+//
+// The list is the repo's `.github/quarantine.json`, product-owned (not a
+// drift-checked copy), so a reader on GitHub, the dashboard and the job all see
+// the same thing. Every entry carries an owner, the date it went in and the
+// condition for taking it out, and an entry older than STALE_DAYS is flagged on
+// every run: quarantine is a holding pen, not a way to skip a test for ever.
+//
+//   {"tests": [{"file": "tests/unit/x.test.ts", "name": "<test name>",
+//               "owner": "<who>", "since": "YYYY-MM-DD",
+//               "reason": "<what fails>", "exit": "<when it comes out>"}]}
+//
+// Run by its node_modules path from the checks quality job (conventions/
+// checks.yml), like zod-singleton.ts:
+//   quarantine.ts check        validate the list; exit 1 if it is malformed
+//   quarantine.ts bun-options  the BUN_OPTIONS that leave the listed tests out
+//   quarantine.ts run          run each listed test alone and report; exit 0
+
+import { appendFileSync, existsSync, readFileSync } from "node:fs";
+
+export const QUARANTINE_FILE = ".github/quarantine.json";
+export const STALE_DAYS = 30;
+
+export interface QuarantineEntry {
+  file: string;
+  name: string;
+  owner: string;
+  since: string;
+  reason: string;
+  exit: string;
+  issue?: string;
+}
+
+const REQUIRED = ["file", "name", "owner", "since", "reason", "exit"] as const;
+
+export interface ParsedQuarantine {
+  entries: QuarantineEntry[];
+  errors: string[];
+  warnings: string[];
+}
+
+const DAY_MS = 86_400_000;
+const isDate = (s: string) => /^\d{4}-\d{2}-\d{2}$/.test(s) && !Number.isNaN(Date.parse(`${s}T00:00:00Z`));
+
+/** `text` is the file's contents, undefined when the repo has none. */
+export function parseQuarantine(
+  text: string | undefined,
+  opts: { today: string; exists: (file: string) => boolean },
+): ParsedQuarantine {
+  const out: ParsedQuarantine = { entries: [], errors: [], warnings: [] };
+  if (text === undefined) return out;
+  let doc: unknown;
+  try {
+    doc = JSON.parse(text);
+  } catch (e) {
+    out.errors.push(`${QUARANTINE_FILE} is not JSON: ${e instanceof Error ? e.message : String(e)}`);
+    return out;
+  }
+  const tests = (doc as { tests?: unknown } | null)?.tests;
+  if (!Array.isArray(tests)) {
+    out.errors.push(`${QUARANTINE_FILE} has no "tests" array`);
+    return out;
+  }
+  const seen = new Set<string>();
+  tests.forEach((raw, i) => {
+    const e = (raw ?? {}) as Record<string, unknown>;
+    const at = `${QUARANTINE_FILE} tests[${i}]`;
+    const missing = REQUIRED.filter((f) => typeof e[f] !== "string" || (e[f] as string).trim() === "");
+    if (missing.length > 0) {
+      out.errors.push(
+        `${at}: missing ${missing.join(", ")} (every entry needs an owner, a date and an exit condition)`,
+      );
+      return;
+    }
+    const entry = e as unknown as QuarantineEntry;
+    if (!isDate(entry.since)) {
+      out.errors.push(`${at}: since "${entry.since}" is not a YYYY-MM-DD date`);
+      return;
+    }
+    const age = Math.round((Date.parse(`${opts.today}T00:00:00Z`) - Date.parse(`${entry.since}T00:00:00Z`)) / DAY_MS);
+    if (age < 0) {
+      out.errors.push(`${at}: since ${entry.since} is in the future`);
+      return;
+    }
+    if (!opts.exists(entry.file)) {
+      out.errors.push(`${at}: ${entry.file} is not in the repo; drop the entry or fix the path`);
+      return;
+    }
+    const key = `${entry.file}\0${entry.name}`;
+    if (seen.has(key)) {
+      out.errors.push(`${at}: "${entry.name}" is listed twice`);
+      return;
+    }
+    seen.add(key);
+    if (age > STALE_DAYS) {
+      out.warnings.push(
+        `"${entry.name}" has been quarantined since ${entry.since} (${age} days, owner ${entry.owner}): fix it and take it out, or delete it`,
+      );
+    }
+    out.entries.push(entry);
+  });
+  return out;
+}
+
+const escapeRegex = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+
+/** A regex matching the one test, wherever its describe blocks put it. */
+export function namePattern(name: string): string {
+  return `${escapeRegex(name)}$`;
+}
+
+/** BUN_OPTIONS for the gating run: every test except the listed ones. Bun
+ *  splits BUN_OPTIONS on whitespace and has no exclude-by-name flag, so it is
+ *  one negative lookahead with each space written as \s. */
+export function bunOptions(entries: readonly QuarantineEntry[]): string {
+  if (entries.length === 0) return "";
+  const names = entries.map((e) => namePattern(e.name).replace(/\s/g, "\\s")).join("|");
+  return `--test-name-pattern=^(?!.*(?:${names})).*`;
+}
+
+export type RunResult = "passed" | "failed" | "missing";
+
+/** A run that matched nothing means the test was renamed or deleted, and the
+ *  entry is stale; that is not a pass, whatever the exit code. */
+export function classifyRun(exitCode: number, output: string): RunResult {
+  if (/matched 0 tests/.test(output) || /Ran 0 tests/.test(output)) return "missing";
+  return exitCode === 0 ? "passed" : "failed";
+}
+
+export function summary(results: readonly { entry: QuarantineEntry; result: RunResult }[]): string {
+  const lines = [
+    `### Quarantined tests (run and reported, not gating)`,
+    "",
+    `From \`${QUARANTINE_FILE}\`. A failure here does not fail the job; it is the flake the entry is waiting on.`,
+    "",
+    "| Result | Test | File | Owner | Since | Exit condition |",
+    "|---|---|---|---|---|---|",
+  ];
+  for (const { entry, result } of results) {
+    const cell = (s: string) => s.replace(/\|/g, "\\|");
+    lines.push(
+      `| ${result} | ${cell(entry.name)} | \`${entry.file}\` | ${cell(entry.owner)} | ${entry.since} | ${cell(entry.exit)} |`,
+    );
+  }
+  return `${lines.join("\n")}\n`;
+}
+
+function load(): ParsedQuarantine {
+  const text = existsSync(QUARANTINE_FILE) ? readFileSync(QUARANTINE_FILE, "utf8") : undefined;
+  return parseQuarantine(text, { today: new Date().toISOString().slice(0, 10), exists: existsSync });
+}
+
+function report(parsed: ParsedQuarantine): void {
+  for (const w of parsed.warnings) console.log(`::warning title=quarantine::${w}`);
+  for (const e of parsed.errors) console.log(`::error title=quarantine::${e}`);
+}
+
+async function main(cmd: string | undefined): Promise<number> {
+  const parsed = load();
+  if (cmd === "check") {
+    report(parsed);
+    console.log(`${parsed.entries.length} quarantined test(s) in ${QUARANTINE_FILE}`);
+    return parsed.errors.length > 0 ? 1 : 0;
+  }
+  if (cmd === "bun-options") {
+    if (parsed.errors.length > 0) return 1;
+    process.stdout.write(bunOptions(parsed.entries));
+    return 0;
+  }
+  if (cmd === "run") {
+    report(parsed);
+    const results: { entry: QuarantineEntry; result: RunResult }[] = [];
+    for (const entry of parsed.entries) {
+      console.log(`== quarantined: ${entry.name} (${entry.file})`);
+      const proc = Bun.spawn(["bun", "test", entry.file, `--test-name-pattern=${namePattern(entry.name)}`], {
+        stdout: "pipe",
+        stderr: "pipe",
+      });
+      const [out, err] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text()]);
+      const code = await proc.exited;
+      process.stdout.write(out);
+      process.stdout.write(err);
+      const result = classifyRun(code, `${out}${err}`);
+      results.push({ entry, result });
+      if (result !== "passed") {
+        console.log(`::warning title=quarantined test ${result} (not gating)::${entry.name} (${entry.file})`);
+      }
+    }
+    const failed = results.filter((r) => r.result !== "passed").length;
+    if (process.env.GITHUB_STEP_SUMMARY && results.length > 0) {
+      appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary(results));
+    }
+    if (process.env.GITHUB_OUTPUT) {
+      appendFileSync(process.env.GITHUB_OUTPUT, `quarantined=${results.length}\nquarantine_failed=${failed}\n`);
+    }
+    console.log(`${results.length} quarantined test(s) run, ${failed} not passing (not gating)`);
+    return 0;
+  }
+  console.error("usage: quarantine.ts check | bun-options | run");
+  return 2;
+}
+
+if (import.meta.main) process.exit(await main(process.argv[2]));

package/conventions/smoke.yml

@@ -193,6 +193,31 @@ jobs:
           NORSK_LICENSE_V2: ${{ secrets.NORSK_LICENSE_V2 }}
         run: printf '%s' "$NORSK_LICENSE_V2" > "$RUNNER_TEMP/norsk-license.json"
 
+      # Every norsk-ctl daemon starts the node MoQ relay (norsk-moq-relay) unless
+      # told not to, and nothing removes it: it is a node singleton, and the reap
+      # step only takes containers that mount this job's test-temp. So the first
+      # test daemon on a shared runner left one, and every later one adopted it
+      # (probe's smoke ran in relay mode on 2026-10-10; relay's smoke lost 4443
+      # to it). This job's daemons run without the node relay and discovery, so
+      # they neither start nor adopt one, and whatever is already on the host is
+      # named here. It reports and never fails: the leftover may be someone's.
+      - name: Isolate this job from node singletons other jobs left on this host
+        run: |
+          set -uo pipefail
+          echo "NORSK_CTL_MOQ_RELAY_ENABLE=0" >> "$GITHUB_ENV"
+          echo "NORSK_CTL_DISCOVERY_ENABLE=0" >> "$GITHUB_ENV"
+          left="$(docker ps -a --filter label=norsk-ctl.role --format '{{.Names}} {{.Label "norsk-ctl.role"}} {{.Status}} (created {{.CreatedAt}})' 2>/dev/null || true)"
+          if [ -z "$left" ]; then echo "no norsk-ctl containers on this host"; exit 0; fi
+          echo "norsk-ctl containers already on this host (other jobs' running instances are expected):"
+          echo "$left"
+          while IFS= read -r line; do
+            role="$(awk '{print $2}' <<<"$line")"
+            case "$role" in
+              moq-relay|discovery)
+                echo "::warning title=node singleton left on this host::$line -- left by an earlier job (a daemon's auto-start, or a run killed before its reap). This job's daemons run without it and will not adopt it; remove it with docker rm -f if nothing on this host needs it." ;;
+            esac
+          done <<<"$left"
+
       # Pick the binary for THIS runner's architecture: the GPU box is aarch64,
       # the x64 pool is x86_64, and the wrong one dies with "Exec format error"
       # deep in the suite. No `|| true` on --version: a bad binary must fail here.

package/doc-guide/engine-verdict.js

@@ -5,9 +5,9 @@
  * product's scripts/doc-guide/regen-manual.sh asks this, from the tier's own
  * output, instead of "are the capture files on disk": that rule passed a
  * composition guide that hung after two captures (2026-10-03) and passed any
- * guide whose assertions failed after its captures were written. The engine
- * tier is also half of the publish gate (dev-kit conventions/build-image.yml),
- * so what it forgives, the gate forgives.
+ * guide whose assertions failed after its captures were written. The
+ * conventions' engine-tier step (build-image.yml guides-engine, publish-docs.yml)
+ * asks it the same way; that tier reports but does not gate (norsk-ctl task 57).
  *
  * Forgiven: every result line a test passed, apart from unnamed hook failures
  * in guides that also passed a test. Anything else -- a named test failing, a

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@norskvideo/ctl-dev-kit",
-  "version": "0.2.62",
+  "version": "0.2.64",
   "type": "module",
   "exports": {
     "./create-product": "./create-product/create-product.ts",
@@ -39,9 +39,9 @@
   "devDependencies": {
     "@norskvideo/ctl-oas-to-ts": "^0.1.0",
     "@norskvideo/norsk-sdk": "1.3.1-2026-10-08-846f25e3",
-    "@norskvideo/norsk-studio": "1.30.0-2026-10-08-8f4b6c11",
-    "@norskvideo/norsk-studio-builder": "1.30.0-2026-10-08-8f4b6c11",
-    "@norskvideo/norsk-studio-built-ins": "1.30.0-2026-10-08-8f4b6c11",
+    "@norskvideo/norsk-studio": "1.30.0-2026-10-09-0cd6f070",
+    "@norskvideo/norsk-studio-builder": "1.30.0-2026-10-09-0cd6f070",
+    "@norskvideo/norsk-studio-built-ins": "1.30.0-2026-10-09-0cd6f070",
     "@playwright/test": "^1.59.0",
     "@types/bun": "latest",
     "@types/cors": "^2.8.17",