@cat-factory/executor-harness 1.147.0 → 1.151.0

package/README.md

@@ -155,6 +155,21 @@ Three rules bind anything added to it:
   payload on this machine, a daemon whose architecture the payload is not built for, `docker load`
   refusing the archive) AND the halves of a failed run that are ours rather than the daemon's,
   which is what docker's exit 126/127, a tag that did not resolve and an unexecutable payload are.
+- **A daemon that runs containers is not a daemon whose containers have a NETWORK**, which is the
+  same mistake one level in again. Loading and running a local image needs no network at all, so the
+  workload check above passes identically on a daemon whose nested containers are cut off: every
+  published image ran its rootless daemon with `--iptables=false`, which drops the MASQUERADE rule
+  for its bridge, and the harness reported `usable` while every `docker build` that fetched a
+  dependency was guaranteed to fail (issue #2174). So `usable` carries its own egress verdict,
+  measured from INSIDE a nested container (a TCP connect to a raw address, and a name to resolve),
+  and the rendered line says something different for each. The `blocked` wording is precise about
+  which commands break, because "docker has no network" is false and would have an agent skip work
+  it could do: the daemon pulls base images and `docker compose up` of pre-built images works, while
+  every `RUN` line that fetches anything fails, SLOWLY, since npm reports `EAI_AGAIN` only once its
+  retry backoff gives up and the build reads as a hang. `reachable` needs the connect AND the
+  lookup, since nothing an agent installs is fetched by address; a resolved name with a refused
+  connect is undetermined rather than blocked, because the resolution proves a path out exists and
+  the likelier cause is a deployment that filters the address the check was pointed at.
 - **A daemon that is STARTING is not a daemon that is absent.** Because the entrypoint does not
   wait, the backend dispatches seconds before there is a socket, and `docker info` is then refused
   at once rather than slowly. So a refusal is read against `DOCKER_HOST`, which the entrypoint sets
@@ -389,12 +404,15 @@ Docker socket would hand the container root on the host.
   concluded about itself.
 - The compose stand-up REFUSES on a decided negative and says why, instead of running compose
   against nothing and handing the agent a connection error to interpret. The refusal rides back on
-  the Tester step as the cause plus the two facts that decide where a human should look:
-  `infraSetup.dockerAvailable` (was anything answering) and `infraSetup.dockerWorkload` (what a
-  container did on it). Two fields rather than one, because the daemon has two ways to stop a
-  stand-up: nothing to talk to, and a daemon that answers and cannot run a container. Flattening
-  the second onto `dockerAvailable: false` renders as "no Docker daemon in the executor" and sends
-  an operator to restart a daemon that is already up.
+  the Tester step as the cause plus the three facts that decide where a human should look:
+  `infraSetup.dockerAvailable` (was anything answering), `infraSetup.dockerWorkload` (what a
+  container did on it) and `infraSetup.dockerEgress` (what that container could REACH). Three
+  fields rather than one, because the daemon has three ways to stop the work: nothing to talk to, a
+  daemon that answers and cannot run a container, and a daemon that runs containers and gives them
+  no network. Flattening the second onto `dockerAvailable: false` renders as "no Docker daemon in
+  the executor" and sends an operator to restart a daemon that is already up; flattening the third
+  onto `dockerWorkload: 'usable'` renders as a sandbox where the stack works, which is the reading
+  that let every `docker build` in a `--iptables=false` container fail unexplained for months.
 
 **What the entrypoint probes for is a SOCKET, and serving is not usable.** That is the whole of
 what a boot record can know, and it is weaker than what either consumer wants: a rootless daemon in
@@ -433,6 +451,39 @@ about to run, and the record supplies what only the record holds, the cause and
 log tail. `GET /health` deliberately keeps reporting the boot record rather than probing per poll,
 since it is not the surface that acts on the answer.
 
+**Which daemon the container ends up with is a choice made on evidence.** `--iptables=false`
+arrived because the daemon could not start at all without it: a sandbox like Cloudflare Containers
+gives it no way to install its firewall rules, and it refuses to start. What went unnoticed is what
+the flag costs once the daemon DOES start. The rule it drops is the MASQUERADE for the bridge, so a
+nested container is never NATed and has no egress whatsoever, no DNS and no raw IP either. The
+daemon's own `docker pull` keeps working, which is most of why it stayed hidden for so long, and the
+cost lands on a `docker build` whose `RUN npm ci` sits in npm's retry backoff for some seven minutes
+before failing (issue #2173). So the entrypoint starts the daemon that manages its own rules first
+and falls back to `--iptables=false` only when that one EXITS without serving.
+
+The death, and not a clock, is what decides. A sandbox with no iptables binary and no NAT module
+does not make the daemon slow, it makes `dockerd` exit at once with the reason on its log, so an
+exit is the one observation here that is actually about the flags. A first arm still starting when
+the budget runs out is a cold sandbox as often as a wedged one, and swapping there would take a
+capable daemon away for the container's whole life on a guess; it is recorded as undecided and LEFT
+RUNNING instead, so `resolveDockerVerdict`'s live re-probe can still find it with its NAT intact. A
+sandbox that genuinely cannot do iptables ends up exactly where it was; a privileged Docker or
+Podman host, which is what local mode runs on, gets working nested networking.
+
+Each arm gets its OWN rootlesskit state directory, image store and pid file, and the abandon path
+waits for the process to be gone after `kill -9`. SIGKILL is not propagated, so a launcher that had
+already forked the real `dockerd` dies while its child holds an exclusive lock on the data root and
+a live pid file: sharing either would make the fallback fail with `pid file found` for a reason that
+has nothing to do with why it was started, which is the worst outcome available here (both arms
+record `failed` and the container ends up with no daemon at all). The socket is the one thing that
+cannot be per-arm, since `DOCKER_HOST` names it, and dockerd unlinks and rebinds it.
+
+The two arms record different `reason` words (`serving` and `serving-without-nat`) with a detail
+that names the CONSEQUENCE, which the workload check measures from inside a nested container without
+ever learning why. What the detail does NOT do is name a cause: "iptables is unavailable here" is
+not something the entrypoint measures, and the real cause is the daemon's own log tail, which rides
+the stderr line announcing the switch.
+
 Why it is written down at all: the image shipped for months with `docker-ce-rootless-extras` (the
 wrappers that START a daemon) and no `docker-ce` (the daemon itself), and no `iproute2` for the
 network rootlesskit builds. The entrypoint backgrounded the start in a subshell where its exit
@@ -483,8 +534,8 @@ verdict is stated.
 | `src/agent-shared.ts` | The few helpers every agent MODE shares (effort-report folding, the capability fields forwarded to `runAgentInWorkspace`). |
 | `src/logger.ts`    | Structured logging.                                                                                     |
 | `src/docker-status.ts` | This container's own verdict about its Docker daemon, as recorded by `entrypoint.sh`. Three-valued on purpose: a daemon that FAILED and a daemon nobody asked about are different facts, and only a DECIDED negative refuses a stand-up. See [Local infra: the container's Docker daemon](#local-infra-the-containers-docker-daemon). |
-| `src/docker-capability.ts` | Whether the daemon can RUN A CONTAINER, which is the fact every caller wanted and `docker info` does not answer. Loads a one-layer image built in-process and runs it; `usable` / `unusable` / `unknown`, and only the container RUN may produce the middle one, AND only where the daemon is what refused it (docker's 126/127, a tag that did not resolve and an unexecutable payload are the platform's own machinery, so they say "could not tell"). Total: it answers even if it throws. One budget for the whole pass, cancelled when the last caller abandons it, memoised per container for a positive and re-measured for a negative. |
-| `src/docker-probe-image.ts` | The one-layer docker-archive that check loads, assembled here from a statically linked binary already in the image, so the whole thing is local: no registry, no network, no second image. Pure, byte-stable, and it names the architecture the DAEMON reported rather than this process's. |
+| `src/docker-capability.ts` | Whether the daemon can RUN A CONTAINER, which is the fact every caller wanted and `docker info` does not answer. Loads a one-layer image built in-process and runs it; `usable` / `unusable` / `unknown`, and only the container RUN may produce the middle one, AND only where the daemon is what refused it (docker's 126/127, a tag that did not resolve and an unexecutable payload are the platform's own machinery, so they say "could not tell"). `usable` then carries what a SECOND container, on the default network, could reach: `reachable` / `blocked` / `undetermined`, on the same asymmetry (a missing busybox applet is the platform's gap, never an absent network). Total: it answers even if it throws. One budget for the whole pass plus a separate one for the egress container, cancelled when the last caller abandons it, memoised per container once SETTLED: any positive, plus every negative whose cause cannot change under a running container (a rejected target, a payload with no `nc`, a filtered address). Only a genuinely transient failure is re-measured, because re-measuring is two container starts and an image load per job on the critical path. |
+| `src/docker-probe-image.ts` | The one-layer docker-archive that check loads, assembled here from a statically linked binary already in the image, so the whole thing is local: no registry, no network, no second image. Pure, byte-stable, and it names the architecture the DAEMON reported rather than this process's. Also builds the egress container's argv, and validates the address it aims at (an IPv4 literal, never a name: a connect that has to resolve first cannot separate a broken route from broken DNS). The connect uses busybox `nc -z` where the payload has it and no `-w` where it does not, both under `busybox timeout`, because `-w` covers the FINAL NET READ: on a TLS port, a connect that succeeded then exits non-zero and reads as a route that is not there. |
 | `src/docker-command.ts` | How the harness runs one `docker …` command on its own behalf: an argv (no shell), a stdin body, stdout kept apart from stderr, one required timeout, the job's signal, and `killChildProcess` for the kill. Not a second `captured-command.ts`, which stays the one way a DECLARED shell command runs; the header says which of its choices this needs to differ on and why. |
 | `src/agent-env.ts` | The env for anything the harness spawns into the agent's CHECKOUT: its own environment minus the variables that are facts about the HARNESS. Today that is `NODE_ENV` (the harness runs in production mode, and an inherited `NODE_ENV=production` makes npm omit devDependencies in a checkout that never asked for it) and `PORT` (the port this harness is listening on, so a service that reads it would bind the one address in the container's network namespace that is already taken). |
 
@@ -510,8 +561,10 @@ runner):
 | `REPRODUCTION_TOTAL_BUDGET_MS` | `2700000` (45m) | Wall-clock ceiling on the WHOLE proof phase (every attempt, both trees, setup included). Attempts multiply two full tree runs each and the heartbeat above deliberately stops the inactivity watchdog from firing, so this is what bounds the phase. Checked at phase boundaries; exceeding it settles `inconclusive`, never a run failure. |
 | `HARNESS_TRANSCRIPT_TTL_MS` | `259200000` (3d) | How long lifted subscription-CLI session transcripts are kept before the retention sweep prunes them. |
 | `HARNESS_TRANSCRIPT_ROOT`   | `<tmpdir>/cf-agent-transcripts` | Where retained session transcripts are moved to (one dir per run). Meaningful only on a reused (warm-pool) container; a per-run container is torn down with the job. The TTL sweep deletes only dirs it created (each carries a `.cf-retained` marker), so pointing this at a shared directory never touches unrelated content, though a dedicated dir is still recommended. An override on a different filesystem than the config home falls back to copy-then-remove. |
-| `HARNESS_DOCKER_READY_TIMEOUT_SECONDS` | `60` | How long `entrypoint.sh` waits for the container's Docker daemon before recording it unavailable. Only a HUNG daemon pays this in full: the wait ends early both when the socket answers and when the daemon process is gone. It runs in the BACKGROUND, so it never delays the container's boot. |
+| `HARNESS_DOCKER_READY_TIMEOUT_SECONDS` | `60` | How long `entrypoint.sh` spends GETTING a Docker daemon to answer, in total, before recording it unavailable. One budget for the whole sequence, not one per arm: the rootless path may start a second daemon after the first exits, and the fallback gets what is left of this (floored at 10s, the only thing that can push the sequence past it, and only when a daemon took the whole window to die). Only a HUNG daemon pays it in full: the wait ends early both when the socket answers and when the daemon process is gone. It runs in the BACKGROUND, so it never delays the container's boot. |
 | `HARNESS_DOCKER_STATUS_FILE` | `/tmp/harness-docker-status.json` | Where that verdict is recorded. `entrypoint.sh` writes it and the harness reads it, so an override must be set for BOTH (they share one process env). |
+| `HARNESS_DOCKER_EGRESS_TARGET` | `1.1.1.1:443` | Where the egress check connects FROM INSIDE a nested container, to find out whether this daemon's containers have a route out. An `IPv4:port`, never a name (a connect that has to resolve first cannot separate a broken route from broken DNS). The default aims at the PUBLIC internet: a deployment that deliberately has none should point this, and the DNS name below, at what it does run, or the verdict is an honest `blocked` about two addresses that were never the ones that matter. A value that is not an address is REPORTED as a check that could not be carried out, never silently replaced. |
+| `HARNESS_DOCKER_EGRESS_DNS_NAME` | `registry.npmjs.org` | The name that same container resolves, which is the other half of egress: nothing an agent installs is fetched by address, so a working route with broken DNS is still reported as blocked, with the detail naming DNS as the half to fix. |
 | `HARNESS_DOCKER_PROBE_BINARY` | `/bin/busybox` | The statically linked binary the platform builds its one-layer probe image from, for the check that answers whether this daemon can RUN a container. Only an image variant that ships it elsewhere needs this. Absent is a supported answer, not a failure: the check then reports that it could not be carried out, which is what the native host transport gets on a developer laptop. |
 
 ## Build / test

package/dist/agent.js

@@ -139,9 +139,12 @@ export async function handleAgent(job, opts = {}) {
         // single post-clone place to put this). The pass is sized for that: everything in it runs
         // concurrently, and every probe either answers in milliseconds or is bounded. Two of them are
         // deliberate waits rather than instant answers: one short retry for a daemon that is still
-        // starting, and, only once a daemon has answered, the container the platform runs to find out
-        // whether it can run one at all (`docker-capability.ts`, budgeted and memoised per container
-        // for a positive). Both take the job's signal, so an abandoned run stops paying at once.
+        // starting, and, only once a daemon has answered, the CONTAINERS the platform runs to find out
+        // what this daemon can do (`docker-capability.ts`: one to prove it runs a container at all,
+        // then one on the default network to see what that container reaches). Both are budgeted, and
+        // the pair is memoised per container once it has SETTLED, which is any positive plus every
+        // negative that cannot change under a running container. Both take the job's signal, so an
+        // abandoned run stops paying at once.
         const staged = {
             ...job,
             systemPrompt: await appendEnvironmentInventory(job.systemPrompt, {

package/dist/docker-capability.d.ts

@@ -1,8 +1,51 @@
 import { type DockerCommandRunner } from './docker-command.js';
 import { type Logger } from './logger.js';
+/**
+ * What a NESTED container could reach, measured from inside one.
+ *
+ * Measured there and nowhere else, which is the whole point. The harness container's own network
+ * is fine in both cases: it resolves and fetches normally, and the daemon pulls images
+ * successfully, so every check one layer out reports a working network over a daemon whose
+ * containers have none.
+ *
+ * `reachable` needs BOTH halves. A route with no DNS is not a network an agent can use: nothing
+ * it installs is fetched by address, so `blocked` is the honest verdict and the detail names DNS
+ * as the half to fix. The reverse (a name resolved, the configured address refused) is
+ * `undetermined` instead of `blocked`, because a resolved name proves a path out exists and the
+ * likeliest cause is that this deployment filters the address the check was pointed at.
+ */
+export type ContainerEgress = 
+/** A container opened a TCP connection out AND resolved a public name. */
+{
+    status: 'reachable';
+}
+/** It could not get out. `detail` names how far it got, since the two have different fixes. */
+ | {
+    status: 'blocked';
+    detail: string;
+}
+/** The check could not be carried out, or could not be read as evidence about the network. */
+ | {
+    status: 'undetermined';
+    reason: string;
+    /**
+     * Whether asking again could give a different answer.
+     *
+     * Load-bearing, because the probe re-measures an undetermined egress and re-measuring
+     * means two container starts plus an image load, per job, on the critical path ahead of
+     * the clone. Most of the ways to land here cannot change while this container lives: a
+     * rejected `HARNESS_DOCKER_EGRESS_TARGET`, a payload with no `nc` applet, an address this
+     * deployment filters. Latching those is not a stale verdict, it is the same measurement
+     * with the same inputs; re-running it forever costs every job about forty seconds for a
+     * value that is settled. Only a genuinely transient failure (a timeout, a cancelled job,
+     * a daemon that could not attach the bridge yet) is worth asking about again.
+     */
+    recheck: boolean;
+};
 /** What one measurement concluded. See the three answers above; nothing collapses them. */
 export type DockerWorkload = {
     status: 'usable';
+    egress: ContainerEgress;
 } | {
     status: 'unusable';
     detail: string;
@@ -38,6 +81,11 @@ export interface DockerWorkloadDeps {
     runDocker: DockerCommandRunner;
     /** This process's architecture, as `process.arch` spells it: the PAYLOAD's, never the daemon's. */
     arch: string;
+    /** Where the egress container aims, as configured. Validated at use, never here. */
+    egress: {
+        target: string;
+        dnsName: string;
+    };
     logger?: Logger;
     archives?: ProbeArchiveMemo;
 }
@@ -76,6 +124,17 @@ export interface DockerWorkloadProbe {
  * the whole container into saying so. Re-measuring a negative is cheap; a daemon that cannot
  * mount fails at once.
  *
+ * A `usable` verdict whose EGRESS could not be determined is re-measured on the same rule and for
+ * the same reason. Whether the bridge is NATed is settled once and for the daemon's life, so a
+ * measured `reachable` or `blocked` is kept; a check that timed out measured nothing, and latching
+ * that would leave the container permanently unable to say which of the two it is.
+ *
+ * But only where asking again could ANSWER differently, which is what `ContainerEgress.recheck`
+ * carries. Most ways to reach `undetermined` are standing facts about this container: a rejected
+ * target setting, a payload with no `nc`, an address the deployment filters. Re-running the whole
+ * measurement on those never converges, and it is not cheap: it is `docker version`, the archive,
+ * `docker load`, two container starts and an `image rm`, per job, ahead of the clone.
+ *
  * Concurrent callers share one in-flight measurement rather than each starting a container, and
  * the measurement is cancelled when the LAST of them has abandoned it. Neither half is optional:
  * one job's abort may not kill a measurement a sibling job is still waiting on (the local native

package/dist/docker-capability.js

@@ -1,14 +1,32 @@
 import { readFile } from 'node:fs/promises';
 import { spawnDockerCommand, } from './docker-command.js';
-import { buildProbeArchive, payloadArchitecture, PROBE_COMMAND, PROBE_IMAGE_TAG, PROBE_SENTINEL, } from './docker-probe-image.js';
+import { buildEgressCommand, buildProbeArchive, EGRESS_DNS_MARKER, EGRESS_TCP_MARKER, parseEgressTarget, payloadArchitecture, PROBE_COMMAND, PROBE_IMAGE_TAG, PROBE_SENTINEL, } from './docker-probe-image.js';
 import { log } from './logger.js';
-import { redactSecrets } from './redact.js';
+import { scrubbedExcerpt } from './redact.js';
 /**
  * The statically linked binary the probe image is built from, overridable for an image variant
  * that ships it elsewhere. Absent is a supported answer, not a failure: under
  * `LOCAL_NATIVE_AGENTS` the harness runs on a developer's machine that never saw this image.
  */
 const PAYLOAD_PATH = process.env.HARNESS_DOCKER_PROBE_BINARY?.trim() || '/bin/busybox';
+/**
+ * Where the egress check aims, overridable for a deployment whose network permits something else.
+ *
+ * A raw IPv4 address rather than a name, so the connect answers a question about ROUTING alone:
+ * pointing it at a hostname would make every verdict depend on DNS, which is the other half and
+ * is measured separately. `1.1.1.1:443` is an anycast address that answers TLS from everywhere
+ * and belongs to no API this repo calls; the name is npm's because npm is what the outage
+ * actually broke. Neither is validated here (see `parseEgressTarget`), so a rejected setting is
+ * reported rather than replaced.
+ *
+ * Both defaults aim at the PUBLIC internet, and a deployment that deliberately has none should
+ * point these at what it does run (an internal registry mirror and its own DNS zone). Left at
+ * the defaults there, the measurement is honest but narrow: it establishes that a container
+ * cannot reach these two, which is why the prompt built from a `blocked` verdict says which
+ * targets were tried rather than that nothing at all is reachable.
+ */
+const EGRESS_TARGET = process.env.HARNESS_DOCKER_EGRESS_TARGET?.trim() || '1.1.1.1:443';
+const EGRESS_DNS_NAME = process.env.HARNESS_DOCKER_EGRESS_DNS_NAME?.trim() || 'registry.npmjs.org';
 /**
  * The ceiling on ONE WHOLE measurement, shared out across the docker commands it makes: each
  * gets what is left of it, down to {@link MIN_COMMAND_MS}.
@@ -29,6 +47,19 @@ const PAYLOAD_PATH = process.env.HARNESS_DOCKER_PROBE_BINARY?.trim() || '/bin/bu
 const WORKLOAD_BUDGET_MS = 20_000;
 /** The floor on one command's share of the budget, so an exhausted budget still gets an answer. */
 const MIN_COMMAND_MS = 1_000;
+/**
+ * The ceiling on the egress container, which gets its OWN budget rather than a share of the one
+ * above.
+ *
+ * The argument for a single shared budget is that a per-command ceiling multiplies on a WEDGED
+ * daemon, and that argument does not reach here: this container is started only after another one
+ * has already run to completion, so the daemon is known to work by the time it is spawned. What
+ * it does have to allow for is a check that is SUPPOSED to be slow in the failing case, since a
+ * blocked route is silent rather than refused and both in-container timeouts have to expire.
+ * Taking that out of the workload budget would have starved the step this whole module exists
+ * for; leaving it unbounded would hand a wedged network the whole job.
+ */
+const EGRESS_BUDGET_MS = 20_000;
 /**
  * The ceiling on removing the probe image again.
  *
@@ -54,6 +85,7 @@ const realDeps = {
     payloadPath: PAYLOAD_PATH,
     runDocker: spawnDockerCommand,
     arch: process.arch,
+    egress: { target: EGRESS_TARGET, dnsName: EGRESS_DNS_NAME },
     archives: oneSlotArchiveMemo(),
 };
 /**
@@ -105,7 +137,7 @@ async function measure(deps, seen, signal) {
     seen.daemonAnswered = true;
     const daemonArch = asked.stdout.trim();
     if (!/^[a-z0-9_]+$/.test(daemonArch)) {
-        return undeterminable(`the Docker daemon did not name an architecture the platform can build an image for (${redactSecrets(daemonArch).slice(0, 40) || 'it answered nothing'})`, true);
+        return undeterminable(`the Docker daemon did not name an architecture the platform can build an image for (${scrubbedExcerpt(daemonArch, 40) || 'it answered nothing'})`, true);
     }
     const payloadArch = payloadArchitecture(deps.arch);
     if (!payloadArch) {
@@ -131,22 +163,135 @@ async function measure(deps, seen, signal) {
         PROBE_IMAGE_TAG,
         ...PROBE_COMMAND,
     ]);
-    // Before the verdict, and whatever the verdict is: the probe image is the platform's, and an
-    // agent that runs `docker images` should not have to wonder whose it is.
+    // A daemon that ran that container has answered the first question, and only then is there a
+    // second one worth asking. An `unusable` daemon cannot run the egress container either, and a
+    // check that could not be carried out has nothing to measure egress against.
+    const verdict = classifyRun(run);
+    const measured = verdict.status === 'usable'
+        ? { status: 'usable', egress: await measureEgress(deps, signal) }
+        : verdict;
+    // After both runs, whatever the verdict is: the probe image is the platform's, and an agent
+    // that runs `docker images` should not have to wonder whose it is.
     await removeProbeImage(deps);
-    return classifyRun(run);
+    return measured;
+}
+/**
+ * Run the second container and read what it reached.
+ *
+ * On the DEFAULT network, deliberately, which is the one thing that separates it from the marker
+ * run above (`--network none`). What an agent's own `docker build` and `docker run` get is the
+ * bridge, and the bridge is exactly what a daemon started with `--iptables=false` fails to NAT.
+ *
+ * Never concludes anything about the DAEMON. Every failure here is either evidence about the
+ * network or evidence about this check, and the caller has already established that the daemon
+ * runs containers.
+ */
+async function measureEgress(deps, signal) {
+    const setting = parseEgressTarget(deps.egress.target, deps.egress.dnsName);
+    // A rejected setting is read from this container's own environment, so it answers the same way
+    // on every job: latched rather than re-measured, which would otherwise spend two container
+    // starts per job re-reading one unchanged string.
+    if ('invalid' in setting) {
+        return { status: 'undetermined', reason: setting.invalid, recheck: false };
+    }
+    const run = await deps.runDocker(['run', '--rm', '--pull', 'never', PROBE_IMAGE_TAG, ...buildEgressCommand(setting.target)], {
+        ...(signal ? { signal } : {}),
+        timeoutMs: EGRESS_BUDGET_MS,
+        ...(deps.logger ? { logger: deps.logger } : {}),
+    });
+    return classifyEgress(run, setting.target);
+}
+/**
+ * What the egress container's output proves, over the four combinations its two markers can
+ * carry.
+ *
+ * Read off the STATUS each command printed rather than off the run's own exit code, because the
+ * two failures that look alike from outside need opposite answers: a refused connection is
+ * evidence about the network, and a 126/127 is busybox saying the image has no such applet, which
+ * is evidence about the platform's own payload and may never be reported as a network that is not
+ * there.
+ */
+function classifyEgress(run, target) {
+    const where = `${target.host}:${target.port}`;
+    if (run.outcome === 'failed') {
+        return {
+            status: 'undetermined',
+            reason: `the platform's egress check did not run (${run.reason})`,
+            // A spawn failure or a timeout is about this attempt and not about the container.
+            recheck: true,
+        };
+    }
+    const tcp = readMarker(run.stdout, EGRESS_TCP_MARKER);
+    const dns = readMarker(run.stdout, EGRESS_DNS_MARKER);
+    if (tcp === undefined || dns === undefined) {
+        // Nothing was measured, and the two ways to get here need different words. `docker run`
+        // failing at the DAEMON level (125, or its own "no such image") means the container never
+        // started, so the network was not the thing that did not answer; a container that started
+        // and printed something unrecognisable is the platform's own payload misbehaving. Reporting
+        // the first as "printed no verdict" tells an operator to go looking at the check's output
+        // for a container that produced none.
+        const refused = platformSideRunFailure(run) ?? (run.code === 125 ? daemonRefusedEgressRun : undefined);
+        return refused
+            ? {
+                status: 'undetermined',
+                reason: `the platform's egress container did not start (${refused}: ${describeOutcome(run)})`,
+                // The daemon just ran the marker container, so a bridge it could not attach now is
+                // the sort of thing that can differ on the next job.
+                recheck: true,
+            }
+            : {
+                status: 'undetermined',
+                reason: `the platform's egress check printed no verdict (${describeOutcome(run)})`,
+                // Same image, same argv, same output: asking again re-reads the same non-answer.
+                recheck: false,
+            };
+    }
+    if ([tcp, dns].some((code) => code === 126 || code === 127)) {
+        return {
+            status: 'undetermined',
+            reason: "the platform's egress check could not run inside its own probe container (the payload " +
+                'has no `nc` or `nslookup` applet)',
+            // A fact about the image this repo builds, which does not change under a running container.
+            recheck: false,
+        };
+    }
+    if (tcp === 0 && dns === 0)
+        return { status: 'reachable' };
+    if (tcp === 0) {
+        return {
+            status: 'blocked',
+            detail: `a container reached ${where} but could not resolve ${target.dnsName}: the route out works and DNS does not`,
+        };
+    }
+    if (dns === 0) {
+        // A resolved name proves a path out of the container exists, so the connect failing is far
+        // more likely to be about the ADDRESS than about the network. Saying "blocked" here would
+        // condemn a working sandbox over a target it happens to filter.
+        return {
+            status: 'undetermined',
+            reason: `a container resolved ${target.dnsName} but could not connect to ${where}, so this ` +
+                'deployment probably filters that address; point HARNESS_DOCKER_EGRESS_TARGET at one it permits',
+            // A filtered address is a standing property of the network this container sits in.
+            recheck: false,
+        };
+    }
+    return {
+        status: 'blocked',
+        detail: `a container could reach neither ${where} nor ${target.dnsName}, the two the platform is configured to try`,
+    };
 }
+/** What a 125 from the egress run means, kept beside the other platform-side run messages. */
+const daemonRefusedEgressRun = 'the daemon refused to create or start it';
 /**
- * What the container RUN proves. The one step allowed to conclude `unusable`, and even here not
- * every non-zero exit is evidence about the daemon.
+ * The exit status printed after `marker`, or undefined when the container never printed one.
  *
- * Docker splits its own failures from the container's by exit code: 125 is `docker run` itself
- * failing, 126 is a command that could not be invoked and 127 one that was not found. The last
- * two are facts about THE PAYLOAD, a binary this platform put in an image it built, and the
- * container had to be created and started to produce them. 125 covers both the daemon refusing to
- * create the container (the verdict this whole module exists for) and the tag not being there to
- * run, which is our own load, so that one is split by what docker SAID.
+ * The LAST occurrence wins, so a marker that somehow reached the stream twice is read at its
+ * final value rather than at whichever came first.
  */
+function readMarker(stdout, marker) {
+    const status = [...stdout.matchAll(new RegExp(`${marker}(\\d{1,3})`, 'g'))].pop()?.[1];
+    return status === undefined ? undefined : Number(status);
+}
 function classifyRun(run) {
     if (run.outcome === 'failed') {
         return undeterminable(`the platform's container check did not run (${run.reason})`, true);
@@ -261,8 +406,7 @@ function describeThrown(err) {
     return bounded(err instanceof Error ? err.message : String(err)) || 'it said nothing';
 }
 function bounded(text) {
-    const scrubbed = redactSecrets(text);
-    return scrubbed.length > DETAIL_CHARS ? `${scrubbed.slice(0, DETAIL_CHARS)}…` : scrubbed;
+    return scrubbedExcerpt(text, DETAIL_CHARS);
 }
 /**
  * Build a probe that measures at most once per container for a POSITIVE answer.
@@ -274,6 +418,17 @@ function bounded(text) {
  * the whole container into saying so. Re-measuring a negative is cheap; a daemon that cannot
  * mount fails at once.
  *
+ * A `usable` verdict whose EGRESS could not be determined is re-measured on the same rule and for
+ * the same reason. Whether the bridge is NATed is settled once and for the daemon's life, so a
+ * measured `reachable` or `blocked` is kept; a check that timed out measured nothing, and latching
+ * that would leave the container permanently unable to say which of the two it is.
+ *
+ * But only where asking again could ANSWER differently, which is what `ContainerEgress.recheck`
+ * carries. Most ways to reach `undetermined` are standing facts about this container: a rejected
+ * target setting, a payload with no `nc`, an address the deployment filters. Re-running the whole
+ * measurement on those never converges, and it is not cheap: it is `docker version`, the archive,
+ * `docker load`, two container starts and an `image rm`, per job, ahead of the clone.
+ *
  * Concurrent callers share one in-flight measurement rather than each starting a container, and
  * the measurement is cancelled when the LAST of them has abandoned it. Neither half is optional:
  * one job's abort may not kill a measurement a sibling job is still waiting on (the local native
@@ -299,7 +454,7 @@ export function createDockerWorkloadProbe(deps = realDeps) {
         return measurement;
     };
     const probe = (async (signal) => {
-        if (latest?.status === 'usable')
+        if (latest?.status === 'usable' && !isWorthReMeasuring(latest.egress))
             return latest;
         const measurement = (inFlight ??= begin());
         measurement.waiters += 1;
@@ -319,6 +474,10 @@ export function createDockerWorkloadProbe(deps = realDeps) {
     probe.last = () => latest;
     return probe;
 }
+/** Whether a kept verdict's egress half is one that asking again could still settle. */
+function isWorthReMeasuring(egress) {
+    return egress.status === 'undetermined' && egress.recheck;
+}
 /**
  * A verdict for the caller whose job was cancelled while it waited, and the listener teardown
  * that keeps a long-lived native-transport process from accumulating one per job.

package/dist/docker-probe-image.d.ts

@@ -10,6 +10,67 @@ export declare const PROBE_IMAGE_TAG = "cat-factory-docker-probe:1";
 export declare const PROBE_SENTINEL = "cat-factory-docker-probe-ok";
 /** The argv the probe container runs. `busybox` dispatches on its own name, so this is an echo. */
 export declare const PROBE_COMMAND: readonly string[];
+/**
+ * What the egress container prints for each observation: the marker, then the exit STATUS of the
+ * command that made it.
+ *
+ * The status rather than a pass/fail marker, because the two failures need different answers.
+ * A refused connection is evidence about the network; a 127 is busybox saying it has no such
+ * applet, which is evidence about the platform's own probe image and may never be reported as a
+ * network that is not there.
+ */
+export declare const EGRESS_TCP_MARKER = "cat-factory-egress-tcp=";
+export declare const EGRESS_DNS_MARKER = "cat-factory-egress-dns=";
+/** Where the egress check aims: a raw address, plus a name to resolve. */
+export interface EgressTarget {
+    host: string;
+    port: number;
+    dnsName: string;
+}
+/**
+ * Read the configured target, or say why it cannot be used.
+ *
+ * Validated rather than trusted, and strictly, for two reasons that both matter. The host and the
+ * name are interpolated into a `sh -c` script INSIDE the probe container, so anything else there
+ * would be running whatever a deployment's environment happened to hold; and a target that is
+ * quietly wrong produces a confident `blocked` about a daemon that is fine, which is the exact
+ * class of lie this whole module exists to remove. A rejected setting is REPORTED as a check that
+ * could not be carried out, never silently swapped for the default: an operator who pointed this
+ * at an address their network permits is entitled to find out that it was ignored.
+ */
+export declare function parseEgressTarget(target: string, dnsName: string): {
+    target: EgressTarget;
+} | {
+    invalid: string;
+};
+/**
+ * The argv the egress container runs: connect, say what that returned, resolve, say the same.
+ *
+ * Both observations are made and BOTH are reported, because they fail for different reasons and
+ * have different fixes. A connect to a raw address needs only a route; a lookup needs the
+ * daemon's embedded resolver to be reachable and to forward. Reporting only the first would call
+ * a container with a working route and broken DNS "reachable", and nothing an agent fetches by
+ * name would work there.
+ *
+ * Every applet is called by its full path (`/busybox nc`) rather than by name. The image holds
+ * one file and no PATH, and busybox's standalone-shell dispatch is a build-time option nothing
+ * here may assume.
+ *
+ * The connect is the half that is easy to get silently wrong, and `nc -w SEC` alone gets it
+ * wrong. busybox documents that flag as the timeout for connects AND FINAL NET READS: once stdin
+ * hits EOF `nc` half-closes and then waits to be spoken to, so a connect that SUCCEEDED to a peer
+ * which expects the client to speak first (every TLS port, the default `1.1.1.1:443` included)
+ * hits the alarm and exits non-zero. Read off the exit status alone that is indistinguishable
+ * from a refusal, so a working network reports a route that is not there. `-z` means "connect,
+ * then stop", which is the question being asked, so it is used wherever the payload's busybox was
+ * built with it; where it was not, the connect runs with no `-w`, which is the build whose `nc`
+ * exits on its own when stdin closes.
+ *
+ * Both halves are wrapped in `${busybox} timeout` either way, since a blackholed route is silent
+ * rather than refused and the applet's own ceiling is the thing this comment exists because we
+ * cannot assume.
+ */
+export declare function buildEgressCommand(target: EgressTarget): readonly string[];
 /** The docker name for the architecture THIS process's payload is built for, when there is one. */
 export declare function payloadArchitecture(arch?: string): string | undefined;
 interface TarEntry {