@celilo/e2e 0.7.16 → 0.7.17

package/AGENTS.md

@@ -0,0 +1,117 @@
+# AGENTS.md — `@celilo/e2e`
+
+`@celilo/e2e` is a **simulated internet in Docker** for end-to-end testing of
+[celilo](https://celilo.computer)-deployed apps. It stands up a DNS hierarchy
+(root → TLD → registrar → recursive resolver), an ACME server (Pebble, standing
+in for Let's Encrypt), an ISP router + customer firewall with real iptables NAT,
+zoned LAN networks, and systemd target machines — then runs celilo *inside* it as
+if it were deploying to a real home lab. This file orients an AI agent to write
+and run those tests. **Read the in-package docs below before searching the web** —
+they are the source of truth.
+
+This file ships **inside the npm package**: the `./`-prefixed docs in the doc map
+are installed alongside it (grep them offline). Repo-relative entries point at the
+source repository, which the tarball does not carry.
+
+## Mental model (learn these five things)
+
+- **The sim is a faithful internet, not a convenience harness.** Public DNS only
+  ever answers with the customer firewall's *external* IP; the firewall DNATs
+  inbound to the right internal host. ACME challenges validate over the public
+  path. A test that "passes" by short-circuiting any of this is testing a placebo
+  — see **Inviolable rules** below.
+- **`network()` builder** — declare zones and the machines in them, `.start()` the
+  topology, drive celilo with `.celilo(...)`, assert, then `.stop()`. This is the
+  whole library surface (`import { network } from '@celilo/e2e'`).
+- **Zones** — `internal` (home LAN), `dmz`, `app`, `secure` are the customer-side
+  networks; `isp-external` / `internet-external` model the public internet. Public
+  IPs live only on the public networks; the firewall containers are the only place
+  the two address spaces meet (via NAT/DNAT).
+- **Two consumption modes** — as a **library** (`network()` in a `bun test` file)
+  or via the **`cele2e` CLI** (`cele2e run <test>`, `build-infra`, `list`). The
+  shared infra (`celilo-e2e-shared`) is a *global* Docker compose project,
+  independent of which checkout started it.
+- **`build-infra` gate** — anything baked into a Docker image (`docker/`,
+  `config/`, `simulators/`, installed package tarballs) needs `cele2e build-infra`
+  before it takes effect. Pure `src/*.ts` changes do not (bun runs TS directly).
+
+## Writing a test (the happy path)
+
+```ts
+import { test, expect } from 'bun:test';
+import { network } from '@celilo/e2e';
+
+test('caddy serves TLS in the dmz', async () => {
+  const net = await network()
+    .dmz({ caddy: '10.0.10.10' })   // machine name → zone-side IP
+    .start();
+  try {
+    await net.celilo('module import caddy');
+    await net.celilo('module deploy caddy --no-interactive');
+    // assert against the deployed service via the public path, not the container IP
+  } finally {
+    await net.stop();
+  }
+});
+```
+
+- Prefer the **fixtures** in `./src/fixtures.ts` (`CADDY_DEPLOYMENT`,
+  `FULL_STACK`, `TECHNITIUM_STACK`, …) over hand-rolling topology.
+- Drive **everything through the celilo CLI** or the module's documented surface —
+  never `docker exec` to hand-edit a deployed container's config (that's the bug
+  you're trying to catch).
+- Poll on the **observable condition** the deploy is meant to satisfy
+  (`VantageProbe`, DNS resolution, an HTTP header) — never `setTimeout` to mask a
+  race.
+
+## CLI you'll use
+
+```
+cele2e list                     # enumerate every test (module suites + top-level)
+cele2e build-infra              # rebuild Docker images (required after image-baked changes)
+cele2e run <test> [--keep]      # run one test; --keep leaves the network up for debugging
+cele2e run --all                # full regression (every suite); --all-modules = modules only
+```
+
+For Claude-driven / detached runs, fire via the repo wrapper
+`infra/scripts/cele2e-run.sh <test> [--keep|--build]` (fire-and-forget RUNID/LOG
+contract) rather than calling `cele2e run` from a long-lived watcher.
+
+## Inviolable rules (breaking any turns the suite into a placebo)
+
+1. **No RFC 1918 leaks past the public boundary.** Public DNS never serves an A
+   record pointing at `10/8`, `172.16/12`, or `192.168/16` — only the firewall's
+   external IP.
+2. **Don't collapse the topology to fix a timing/connectivity bug.** Fix the
+   *deployment ordering* (DNS record / DNAT rule / firewall opening in place
+   before the dependent step), not the simulation.
+3. **Public IPs only on the public networks.** Firewall containers bridge public
+   and private — the only place those spaces meet, and only via NAT/DNAT.
+4. **ACME challenges go through the public path** to the firewall's external IP.
+5. **DDNS updates modify *public* DNS** with the firewall's external IP, never an
+   internal target IP.
+
+Cheats (`/etc/hosts` overrides, `iptables -F`, `--insecure` in module code,
+hardcoded internal IPs, `|| true` on a celilo command, sleeping past a race) are
+fine for **bisecting** a failure but are never a **final solution** — confirm the
+hypothesis, then ship the real fix. Full list + rationale in `./README.md` and the
+repo `CLAUDE.md`.
+
+## Doc map (read in this order)
+
+Shipped **inside this package** (offline, grep-friendly):
+
+- `./README.md` — network architecture, machines, routing, DNS, simulators,
+  manual usage. Start here.
+- `./COVERAGE.md` — the array/object/map coverage audit (where N>1 paths are or
+  aren't exercised). Consult before adding a multi-element feature test.
+- `./src/index.ts` — the public library exports (`network`, types, probes).
+- `./src/fixtures.ts` — ready-made topology presets to build tests on.
+- `./src/types.ts` — `NetworkHandle`, `Zone`, `MachineSpec`, `Vantage`, and the
+  zone subnet/gateway constants.
+
+In the **source repo** (not carried in the tarball):
+
+- `infra/docs/RUNNING_CELE2E_TESTS.md` — the full operator guide; read before
+  running the suite for real.
+- `https://celilo.computer/docs` — hosted docs (LAN; the repo docs lead).

package/COVERAGE.md

@@ -0,0 +1,60 @@
+# E2E Test Coverage Audit
+
+This document tracks how well the e2e suite exercises capability and module
+fields whose type is `array`, `object`, or `map` — anywhere the framework's
+public API admits "more than one of X."
+
+## Why this exists
+
+A test that only ever uses N=1 will pass even when the multi-element path is
+broken. The bug fixed by `dns_registrar` 4.0.0 (see
+`apps/celilo/designs/DNS_REGISTRAR_MULTI_DOMAIN.md`) escaped the existing e2e
+suite for exactly this reason: the website-deploy-simple test put caddy, namecheap,
+and the website all on a single domain, and the multi-domain code path was
+never exercised.
+
+## Audit checklist
+
+For every capability and every module manifest, identify each field whose
+type is `array`, `object`, or `map`. For each one, ask:
+
+1. Does at least one e2e test exercise this field with **N > 1** distinct
+   values?
+2. Are those values **meaningfully different** (different domains, different
+   zones, different ports — not just "two of the same kind of thing")?
+3. Does the test assert behavior that would be **wrong if only the first
+   element worked**?
+
+Anywhere all three answers aren't "yes," that's a gap to either close (write
+the test) or document why it's intentionally deferred.
+
+## Coverage table
+
+| Field | N>1? | Meaningfully different? | Asserts non-first works? | Status |
+|---|---|---|---|---|
+| `namecheap.domains` | yes (`website-deploy-cross-domain`) | yes (different TLDs: `iamtheinternet.org` + `example.net`) | yes (verifies example.net DNS+TLS while iamtheinternet.org is canonical) | covered |
+| `namecheap.ddns_passwords` (per-domain map) | yes (same test) | yes (distinct passwords per domain) | yes (uses example.net's own password to update its A record) | covered |
+| `caddy.hostnames` | yes (`website-deploy-simple`) | partial (same domain in `website-deploy-simple.test.ts`); yes (cross-TLD in cross-domain test) | yes (cross-domain test serves a hostname that isn't `hostnames[0]`) | covered |
+| `iptables.exposed_services` | unclear — needs audit | — | — | **GAP** |
+| `public_web` registrations across modules | partial (`website-deploy-simple.test.ts` registers auth + website on the same domain) | needs cross-domain variant | partial | **GAP** |
+| `<every secret: object field>` | typically only one key | usually no | — | **AUDIT EACH** |
+
+## Closing the gaps
+
+- **`iptables.exposed_services`**: write an e2e that exposes two services
+  on the same machine via different ports and verifies both are reachable
+  from outside.
+- **`public_web` cross-domain**: extend `website-deploy-simple.test.ts` (or a new
+  test) to register routes on two different domains and verify each is
+  served by the right backend.
+- **secret object fields**: as modules using these grow (e.g. third-party
+  API integrations with multiple keys), add per-key tests.
+
+## New-module checklist
+
+When authoring a module, list every field whose type permits multiple
+values. For each, write at least one e2e or integration test that uses
+**N ≥ 2** with meaningfully different values. Add a row to the table
+above (or document why you've deferred it).
+
+See `design/MODULE_DEVELOPMENT_GUIDE.md` for the full new-module checklist.

package/bin/e2e-build

@@ -22,6 +22,7 @@ NETAPPS_DIR="$E2E_DIR/netapps"
 cd "$E2E_DIR"
 
 GREEN='\033[0;32m'
+YELLOW='\033[0;33m'
 DIM='\033[2m'
 BOLD='\033[1m'
 NC='\033[0m'
@@ -160,6 +161,25 @@ BAKE_END=$(date +%s)
 echo -e "${GREEN}Baked in $((BAKE_END - BAKE_START))s${NC}"
 echo ""
 
+# ── Step 2d: Pre-seed heavy app images into the app-zone preload cache ─────────
+#
+# Heavy application images (authentik server, postgres, redis) are pulled ONCE
+# on the host here and saved to docker-image-cache/, which app-zone target
+# machines bind-mount and load at startup via docker-image-preload.service.
+# Without this the cache is empty on a fresh checkout / the builder, so every
+# app-module deploy pulls GB through the simulated internet — slow enough to
+# time the authentik deploy out (exit 124) and take down forgejo-deploy +
+# full-stack-pipeline (ISS-0154). Best-effort: e2e-cache-images is per-image
+# resilient and the preload service skips a missing tarball, so a transient
+# registry hiccup degrades to a slow pull rather than failing build-infra.
+echo -e "${BOLD}Pre-seeding app-zone image cache...${NC}"
+if bash "$E2E_DIR/bin/e2e-cache-images"; then
+  :
+else
+  echo -e "${YELLOW}⚠ image cache step failed — app-zone deploys may pull over the sim${NC}" >&2
+fi
+echo ""
+
 # ── Step 3: Optionally save images to tarball ─────────────────────────────────
 
 if [ "$1" = "--save" ]; then

package/bin/e2e-cache-images

@@ -23,6 +23,7 @@ mkdir -p "$CACHE_DIR"
 
 BOLD='\033[1m'
 GREEN='\033[0;32m'
+YELLOW='\033[0;33m'
 DIM='\033[2m'
 NC='\033[0m'
 
@@ -44,12 +45,18 @@ for filename in "${!IMAGES[@]}"; do
   printf "  %-50s " "$image"
   START=$(date +%s)
 
-  docker pull "$image" > /dev/null 2>&1
-  docker save "$image" -o "$tarball"
-
-  END=$(date +%s)
-  SIZE=$(du -h "$tarball" | cut -f1)
-  echo -e "${GREEN}✔${NC} ${DIM}$((END - START))s  ${SIZE}${NC}"
+  # Per-image best-effort: a transient registry failure for one image must not
+  # abort caching the others (this runs inside build-infra now). A guarded
+  # command does not trip `set -e`. The preload service skips a missing tarball
+  # gracefully — that module just pulls over the sim at deploy time.
+  if docker pull "$image" > /dev/null 2>&1 && docker save "$image" -o "$tarball"; then
+    END=$(date +%s)
+    SIZE=$(du -h "$tarball" | cut -f1)
+    echo -e "${GREEN}✔${NC} ${DIM}$((END - START))s  ${SIZE}${NC}"
+  else
+    rm -f "$tarball"
+    echo -e "${YELLOW}✘ failed — will pull over the sim at deploy time${NC}"
+  fi
 done
 
 echo ""

package/config/dns/iamtheinternet.org.zone

@@ -0,0 +1,16 @@
+$ORIGIN iamtheinternet.org.
+$TTL 300
+
+@ IN SOA ns1.iamtheinternet.org. admin.iamtheinternet.org. (
+    2024020247 300 60 604800 300
+)
+
+; Public DNS — RFC 1918 addresses NEVER appear here. The whole point of
+; the e2e simulation is to model real internet connectivity, where
+; private IPs are unreachable from outside. All public-facing names
+; resolve to the firewall's external IP (100.100.0.100), which DNATs
+; inbound to the right internal host.
+@       IN NS   ns1.iamtheinternet.org.
+ns1     IN A    100.64.0.55
+@       IN A    0.0.0.0
+www     IN A    100.100.0.100

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/e2e",
-  "version": "0.7.16",
+  "version": "0.7.17",
   "description": "E2E test infrastructure for Celilo-deployed applications. Provides a simulated internet with DNS hierarchy, ACME server, firewalls, and target machines in Docker.",
   "type": "module",
   "main": "./src/index.ts",
@@ -19,7 +19,8 @@
   },
   "scripts": {
     "test": "bun test tests/",
-    "test:completion": "bun test tests/completion"
+    "test:completion": "bun test tests/completion",
+    "test:integration": "bun test tests-integration/"
   },
   "files": [
     "src/",
@@ -33,7 +34,9 @@
     "npm-registry-server/tsconfig.json",
     "npm-registry-server/src/",
     "scripts/",
-    "README.md"
+    "README.md",
+    "AGENTS.md",
+    "COVERAGE.md"
   ],
   "dependencies": {
     "@celilo/cli-display": "^0.1.9",

package/scripts/stage-apt-repo.ts

@@ -65,11 +65,24 @@ export function stageAptRepo(repoRoot: string, pkgDir: string): boolean {
   process.stdout.write(`  ${'apt-repo (build debs)'.padEnd(28)} `);
   const t0 = Date.now();
 
+  // The deb must install the EXACT version the npm-registry-sim serves locally
+  // (= apps/celilo/package.json). Pin CELILO_VERSION so the postinst runs
+  // `bun add @celilo/cli@<version>` rather than `@celilo/cli@alpha` (build-deb.sh's
+  // default): the sim has no local `alpha` dist-tag, so an `@alpha` install leaks
+  // to the real npm uplink and pulls a stale published alpha (ISS-0153). Pinning
+  // makes the deb version, the installed npm version, and the test expectation a
+  // single source of truth.
+  const version = currentCeliloVersion(repoRoot);
+
   // Reuse the canonical build-deb scripts so versioning/contents match a
   // real release exactly. build:deb builds both arches; build:deb:bootstrap
   // builds the arch-all meta-package.
   for (const script of ['build:deb', 'build:deb:bootstrap']) {
-    const result = spawnSync('bun', ['run', script], { cwd: repoRoot, stdio: 'pipe' });
+    const result = spawnSync('bun', ['run', script], {
+      cwd: repoRoot,
+      stdio: 'pipe',
+      env: { ...process.env, CELILO_VERSION: version },
+    });
     if (result.status !== 0) {
       console.log(`${red}✗${reset}`);
       console.error(result.stderr?.toString());
@@ -81,7 +94,6 @@ export function stageAptRepo(repoRoot: string, pkgDir: string): boolean {
   // Stage every freshly-built .deb into the pool. dist/ may also hold older
   // versions; copy only the current ones so the repo doesn't advertise
   // stale celilo versions the npm-registry-sim no longer serves.
-  const version = currentCeliloVersion(repoRoot);
   const distDir = join(repoRoot, 'dist');
 
   const debs = readdirSync(distDir).filter((f) => f.endsWith('.deb') && f.includes(version));

package/src/cli/build.ts

@@ -48,6 +48,7 @@ const bold = '\x1b[1m';
 const dim = '\x1b[2m';
 const green = '\x1b[32m';
 const red = '\x1b[31m';
+const yellow = '\x1b[33m';
 const reset = '\x1b[0m';
 
 interface BuildOptions {
@@ -320,6 +321,30 @@ function bakeManagement(pkgDir: string): void {
   console.log(`\n${green}Baked in ${elapsed}s${reset}\n`);
 }
 
+/**
+ * Pre-seed heavy application images (authentik server, postgres, redis) into
+ * the app-zone preload cache. Without this the `docker-image-cache` is empty on
+ * a fresh checkout / the builder, so every app-module deploy pulls GB through
+ * the simulated internet — slow enough to time the authentik deploy out
+ * (exit 124) and take down forgejo-deploy + full-stack-pipeline (ISS-0154).
+ * Best-effort: `e2e-cache-images` is per-image resilient and the preload
+ * service skips a missing tarball, so a transient registry hiccup degrades to a
+ * slow pull rather than failing build-infra.
+ */
+function cacheAppImages(pkgDir: string): void {
+  const cacheScript = join(pkgDir, 'bin', 'e2e-cache-images');
+  if (!existsSync(cacheScript)) return;
+
+  console.log(`${bold}Pre-seeding app-zone image cache...${reset}\n`);
+  const result = spawnSync('bash', [cacheScript], { stdio: 'inherit' });
+  if (result.status !== 0) {
+    console.error(
+      `${yellow}⚠ image cache step failed — app-zone deploys may pull over the sim${reset}`,
+    );
+  }
+  console.log('');
+}
+
 function saveImages(pkgDir: string): void {
   const cacheDir = join(pkgDir, '.docker-cache');
   mkdirSync(cacheDir, { recursive: true });
@@ -406,6 +431,10 @@ export function runBuild(options: BuildOptions): void {
   // install.sh, rather than a parallel `bun add -g` pinning.
   bakeManagement(pkgDir);
 
+  // Pre-seed heavy app images so app-zone deploys load them from the cache
+  // instead of pulling GB through the sim (ISS-0154).
+  cacheAppImages(pkgDir);
+
   if (save) {
     saveImages(pkgDir);
   }

package/src/cli/command-registry.ts

@@ -26,6 +26,8 @@ export const COMMANDS: CommandDef[] = [
     description: 'Run e2e tests — pass a module path or pattern, or --all / --all-modules',
     flags: [
       { name: '--all', description: 'Run the full regression: every module suite + top-level e2e/tests/' },
+      { name: '--complete', description: 'Alias of --all: run every test, including ci-unsafe (quarantined) ones' },
+      { name: '--ci-safe', description: 'Full regression EXCEPT tests marked cele2e-ci-unsafe (used by the nightly)' },
       { name: '--all-modules', description: 'Run e2e tests for all modules with e2e/ directories' },
       { name: '--keep', description: 'Keep network running after tests' },
       { name: '--reuse', description: 'Reuse existing network if running' },
@@ -99,7 +101,7 @@ export const COMMANDS: CommandDef[] = [
 
 export const UP_PRESETS = ['--caddy', '--full-stack', '--infrastructure'];
 
-export const RUN_FLAGS = ['--all', '--all-modules', '--keep', '--reuse', '--live', '--published'];
+export const RUN_FLAGS = ['--all', '--complete', '--ci-safe', '--all-modules', '--keep', '--reuse', '--live', '--published'];
 
 export const DOWN_FLAGS = ['--keep', '--all'];
 

package/src/cli/completion.ts

@@ -119,6 +119,8 @@ _cele2e() {
             '--reuse[Reuse existing network if running]' \\
             '--live[Use live (non-simulated) internet]' \\
             '--published[Use published .netapp packages]' \\
+            '--ci[Plain log output: one line per step, no spinner]' \\
+            '--no-ci[Force spinner even when a CI env var is set]' \\
             '*::test name or module path:_cele2e_run_args'
           ;;
         list)

package/src/cli/index.ts

@@ -139,6 +139,8 @@ Options for \`run\`:
   --reuse           Reuse existing network if running
   --live            Use live (non-simulated) internet
   --published       Use published .netapp packages
+  --ci              Plain log output: one ✔/✗ line per step, no spinner
+                    (auto-on when a CI env var is set; --no-ci forces off)
 
 Options for \`up\`:
   --caddy           Start with caddy machine in DMZ
@@ -187,8 +189,17 @@ switch (command) {
     // repo's top-level e2e/tests/ in one invocation; `--all-modules` runs just
     // the module suites. Both resolve the repo root via resolveSuiteRoot() so
     // they work through the `./cele2e` wrapper's cd into packages/e2e.
-    if (args.includes('--all') || args.includes('--all-modules')) {
-      const wantTopLevel = args.includes('--all');
+    if (
+      args.includes('--all') ||
+      args.includes('--all-modules') ||
+      args.includes('--complete') ||
+      args.includes('--ci-safe')
+    ) {
+      // --complete and --ci-safe both run the full discovery (every module +
+      // the repo's top-level e2e/tests/). --ci-safe additionally has the runner
+      // skip tests marked `cele2e-ci-unsafe`. --all-modules stays modules-only.
+      const wantTopLevel =
+        args.includes('--all') || args.includes('--complete') || args.includes('--ci-safe');
       const suiteRoot = resolveSuiteRoot();
       const allModules = findAllModules(suiteRoot);
       const env: Record<string, string> = {
@@ -206,7 +217,9 @@ switch (command) {
       }
       runScript(
         'e2e-run',
-        args.filter((a) => a !== '--all' && a !== '--all-modules'),
+        // Strip the discovery aliases; KEEP --ci-safe so the runner applies the
+        // `cele2e-ci-unsafe` quarantine exclusion.
+        args.filter((a) => a !== '--all' && a !== '--all-modules' && a !== '--complete'),
         env,
       );
     }

package/src/docker-compose-generator.ts

@@ -396,6 +396,17 @@ export function generateTestComposeYaml(config: NetworkConfig, celiloRoot?: stri
         variant === 'vanilla' ? 'celilo-e2e/management:vanilla' : 'celilo-e2e/management:latest',
       networks: { internal: { ipv4_address: ip } },
       cap_add: ['NET_ADMIN'],
+      // ISS-0157: `module import`'s `bun install` hard-links packages from the
+      // bun cache into node_modules, which on overlayfs forces a copy-up + fsync
+      // per file. On a slow/contended builder disk that storm blows past the
+      // 120s import timeout (intermittently). Put the celilo data dir (the
+      // node_modules install target) on tmpfs (RAM): bun then copies cache→tmpfs
+      // (cross-fs, no overlay copy-up) at RAM speed, immune to host disk. NOT
+      // /root/.bun itself — that holds the baked `celilo` CLI (`bun add -g`).
+      // Tmpfs the bun *cache* subdir too: the copy-up + fsync storm is the
+      // overlay upper layer under /root/.bun/install/cache. RAM-backing it (cold
+      // → re-fetch over the fast sim/net) keeps the CLI intact. Ephemeral e2e.
+      tmpfs: ['/root/.local/share/celilo', '/root/.bun/install/cache'],
       volumes: [
         ...(celiloRoot ? [`${celiloRoot}:/celilo`] : []),
         'ssh-keys:/ssh-keys',

package/src/runner.ts

@@ -19,9 +19,9 @@
 
 import { spawn, spawnSync, execSync } from 'node:child_process';
 import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
-import { basename, join, resolve } from 'node:path';
+import { basename, dirname, join, resolve } from 'node:path';
 import { parse as parseYaml } from 'yaml';
-import { ProgressDisplay } from '@celilo/cli-display';
+import { type DisplayMode, ProgressDisplay } from '@celilo/cli-display';
 import {
   emitRunCompleted,
   emitRunFailed,
@@ -70,7 +70,23 @@ const flagReuse = rawArgs.includes('--reuse');
 const flagLive = rawArgs.includes('--live');
 const flagPublished = rawArgs.includes('--published');
 const flagVerbose = rawArgs.includes('--verbose');
+// CI mode: no animated spinner, no [progress:*] markers — just one clean
+// ✔/✗ line per step (plus sub-events) written straight to the log. Forgejo's
+// runner allocates a PTY, so process.stdout.isTTY is truthy there and the
+// interactive footer (redrawn via `\r` on every spinner tick) gets linearized
+// into one log line per tick — dozens of identical braille lines per step.
+// `--ci` (or any standard `CI` env var, e.g. Forgejo/GitHub Actions) forces
+// the non-interactive render path that sidesteps that entirely. `--no-ci`
+// opts back out when a CI env var is set but an operator wants the spinner.
+const flagCi = rawArgs.includes('--ci');
+const flagNoCi = rawArgs.includes('--no-ci');
+const ci = flagCi || (!flagNoCi && !!process.env.CI);
 const patterns = rawArgs.filter((a) => !a.startsWith('--'));
+// --ci-safe: skip tests marked `cele2e-ci-unsafe` in their file (quarantined —
+// known-failing or builder-flaky, each tracked by an ISS). `--complete`/`--all`
+// run everything. Passed through from the CLI (not stripped) so the runner can
+// apply the exclusion.
+const flagCiSafe = rawArgs.includes('--ci-safe');
 
 // ─── Colors ──────────────────────────────────────────────────────────
 
@@ -81,7 +97,8 @@ const red = '\x1b[31m';
 const cyan = '\x1b[36m';
 const reset = '\x1b[0m';
 
-const interactive = process.stdout.isTTY && !process.argv.includes('--no-interactive') && !flagVerbose;
+const interactive =
+  process.stdout.isTTY && !process.argv.includes('--no-interactive') && !flagVerbose && !ci;
 
 // ─── Timing ──────────────────────────────────────────────────────────
 
@@ -454,12 +471,45 @@ async function main() {
       .filter((f) => f.endsWith('.test.ts'))
       .sort()
       .map((f) => resolve(testsPath, f));
+    // Docker-backed integration tests live in a sibling `tests-integration/`
+    // so they stay OUT of the hermetic `bun test tests/` gate. cele2e is the
+    // heavy runner (Docker + build-infra present), so it covers BOTH tiers —
+    // this is what keeps the documented `cele2e run ansible-output` working
+    // after the test moved out of tests/.
+    const integrationPath = join(dirname(testsPath), 'tests-integration');
+    if (existsSync(integrationPath)) {
+      testFiles.push(
+        ...readdirSync(integrationPath)
+          .filter((f) => f.endsWith('.test.ts'))
+          .sort()
+          .map((f) => resolve(integrationPath, f)),
+      );
+    }
   }
 
   if (patterns.length > 0) {
     testFiles = testFiles.filter((f) => patterns.some((p) => basename(f, '.test.ts').includes(p)));
   }
 
+  // --ci-safe: drop quarantined tests — those whose file contains the marker
+  // `cele2e-ci-unsafe` (a comment naming the reason + ISS). Keeps the nightly
+  // green on the CI-safe subset while the quarantined tests are tracked + fixed
+  // separately. `--complete`/`--all` run everything (no exclusion).
+  if (flagCiSafe) {
+    const before = testFiles.length;
+    testFiles = testFiles.filter((f) => {
+      try {
+        return !readFileSync(f, 'utf-8').includes('cele2e-ci-unsafe');
+      } catch {
+        return true;
+      }
+    });
+    const skipped = before - testFiles.length;
+    if (skipped > 0) {
+      console.log(`${dim}--ci-safe: skipped ${skipped} quarantined (cele2e-ci-unsafe) test(s)${reset}`);
+    }
+  }
+
   if (testFiles.length === 0) {
     console.error(`${red}No test files matched${reset}`);
     process.exit(1);
@@ -524,14 +574,21 @@ async function main() {
 
   // 'auto' resolves to 'render' when isTTY is true (operator at a
   // terminal sees animated spinners) and 'protocol' when isTTY is
-  // false (CI / cele2e-run.sh wrapper / any redirected stdout gets
-  // ASCII `[progress:*]` markers — grep-friendly, no braille noise).
+  // false (cele2e-run.sh wrapper / any redirected stdout gets ASCII
+  // `[progress:*]` markers — grep-friendly, no braille noise).
   // Prior to this change the mode was hardcoded 'render', so non-TTY
   // callers got `⣾ phase started` / `✔ phase done` lines per phase
   // (e.g. ~29 braille codepoints in a typical test log) — cosmetic
   // but actively unhelpful for log scanning and tooling.
+  //
+  // CI mode forces 'render' with isTTY:false (non-interactive render):
+  // one ✔/✗ line per step, sub-events emitted inline, NO spinner and
+  // NO `[progress:*]` markers. We can't rely on 'auto' here because
+  // Forgejo's PTY makes isTTY truthy — which would pick the animated
+  // footer and spam the log with one braille line per spinner tick.
+  const displayMode: DisplayMode = ci ? 'render' : 'auto';
   const display = new ProgressDisplay({
-    mode: 'auto',
+    mode: displayMode,
     out: { write: process.stdout.write.bind(process.stdout), isTTY: interactive },
   });
   let passed = 0;

package/src/shared-infra.ts

@@ -172,11 +172,16 @@ export async function ensureSharedInfra(): Promise<void> {
   run(`docker compose -f ${SHARED_COMPOSE_FILE} -p ${SHARED_PROJECT_NAME} up -d`,
     { cwd: e2eDir, timeout: 120_000 });
 
-  // Wait for DNS convergence
+  // Wait for DNS convergence — robustly. The old single 60s window with no
+  // recovery was an intermittent build-infra failure on the builder (#319:
+  // namecheap-dns hadn't served its zone within 60s, and the bare throw gave
+  // no clue why). Now: a longer window, RESTART the DNS container if it's
+  // stuck (re-triggers the zone load), and DUMP diagnostics before giving up.
   console.log('[progress:start] waiting for DNS convergence | DNS converged');
+  const dnsTimeout = 180_000;
   const start = Date.now();
-  const timeout = 60_000;
-  while (Date.now() - start < timeout) {
+  let lastRestart = start;
+  while (Date.now() - start < dnsTimeout) {
     try {
       const result = run(
         `docker compose -f ${SHARED_COMPOSE_FILE} -p ${SHARED_PROJECT_NAME} exec -T namecheap-dns dig @127.0.0.1 iamtheinternet.org SOA +short +timeout=2`,
@@ -190,9 +195,34 @@ export async function ensureSharedInfra(): Promise<void> {
     } catch {
       // retry
     }
+    // Recovery: if ~45s have passed since the last (re)start without
+    // converging, the DNS container likely didn't load its zone — restart
+    // it to re-trigger the load instead of waiting out the whole window.
+    if (Date.now() - lastRestart > 45_000) {
+      console.log('[progress:sub] DNS not converged yet — restarting namecheap-dns');
+      try {
+        run(`docker compose -f ${SHARED_COMPOSE_FILE} -p ${SHARED_PROJECT_NAME} restart namecheap-dns`,
+          { cwd: e2eDir, timeout: 30_000 });
+      } catch {}
+      lastRestart = Date.now();
+    }
     await new Promise(r => setTimeout(r, 2000));
   }
-  throw new Error('Timeout waiting for shared infrastructure DNS convergence');
+  // Diagnostics before failing, so the log shows WHY it didn't converge.
+  let psOut = '';
+  try {
+    psOut = run(`docker compose -f ${SHARED_COMPOSE_FILE} -p ${SHARED_PROJECT_NAME} ps`,
+      { cwd: e2eDir, timeout: 10_000 });
+  } catch {}
+  let dnsLog = '';
+  try {
+    dnsLog = run(`docker compose -f ${SHARED_COMPOSE_FILE} -p ${SHARED_PROJECT_NAME} logs --tail 40 namecheap-dns`,
+      { cwd: e2eDir, timeout: 10_000 });
+  } catch {}
+  throw new Error(
+    `Timeout waiting for shared infrastructure DNS convergence (${dnsTimeout / 1000}s)\n` +
+      `--- compose ps ---\n${psOut}\n--- namecheap-dns logs (tail 40) ---\n${dnsLog}`,
+  );
 }
 
 /**