@cat-factory/executor-harness 1.144.1 → 1.147.0

package/README.md

@@ -142,6 +142,19 @@ Three rules bind anything added to it:
   installed in this image unconditionally, `entrypoint.sh` starts the rootless daemon best-effort
   and execs the server without waiting for it, so at job start this probe is the only thing that
   knows how that went.
+- **A daemon that ANSWERS is not a daemon that WORKS**, which is the same mistake one level in and
+  the one this block used to make. A rootless daemon nested inside a sandbox serves throughout
+  while its snapshotter cannot mount an image layer, so `docker info` succeeds and `docker build`,
+  `docker run` and `docker pull` of anything multi-layer all fail on one EINVAL. Issue #2120 is
+  three agents in a single run each paying to disprove the claim, against a block that also tells
+  them not to re-check it. Only a container that RAN settles it, so the reachable case is split by
+  a real workload (`src/docker-capability.ts`) into `usable`, `unusable` and a daemon that answered
+  while the check could not be carried out. **Only `usable` may say the commands work**, and the
+  asymmetry runs the other way too: a failure of the platform's own machinery reports that it
+  could not tell, never that the daemon is broken. That covers every step before the run (no probe
+  payload on this machine, a daemon whose architecture the payload is not built for, `docker load`
+  refusing the archive) AND the halves of a failed run that are ours rather than the daemon's,
+  which is what docker's exit 126/127, a tag that did not resolve and an unexecutable payload are.
 - **A daemon that is STARTING is not a daemon that is absent.** Because the entrypoint does not
   wait, the backend dispatches seconds before there is a socket, and `docker info` is then refused
   at once rather than slowly. So a refusal is read against `DOCKER_HOST`, which the entrypoint sets
@@ -374,19 +387,47 @@ Docker socket would hand the container root on the host.
 
 - `GET /health` reports it, so an operator (and a boot-time probe) can see what the container
   concluded about itself.
-- The compose stand-up REFUSES on a decided absence and says why, instead of running compose
+- The compose stand-up REFUSES on a decided negative and says why, instead of running compose
   against nothing and handing the agent a connection error to interpret. The refusal rides back on
-  the Tester step as `infraSetup.dockerAvailable: false` with the cause.
+  the Tester step as the cause plus the two facts that decide where a human should look:
+  `infraSetup.dockerAvailable` (was anything answering) and `infraSetup.dockerWorkload` (what a
+  container did on it). Two fields rather than one, because the daemon has two ways to stop a
+  stand-up: nothing to talk to, and a daemon that answers and cannot run a container. Flattening
+  the second onto `dockerAvailable: false` renders as "no Docker daemon in the executor" and sends
+  an operator to restart a daemon that is already up.
+
+**What the entrypoint probes for is a SOCKET, and serving is not usable.** That is the whole of
+what a boot record can know, and it is weaker than what either consumer wants: a rootless daemon in
+a sandbox answers `docker version` while being unable to mount an image, so compose ran and died on
+a mount error inside the one mechanism whose job is to explain why the dependencies did not come up
+(issue #2120). The live half of the verdict is therefore a real workload (load a one-layer image
+and run a container from it, `src/docker-capability.ts`), and `resolveDockerVerdict` consults it in
+BOTH directions: a recorded absence a working daemon contradicts, and a recorded presence that
+cannot run anything. `GET /health` reports the last measurement beside the boot record under
+`docker.workload` and never takes one itself, since it is polled; `unmeasured` is one of its answers.
+
+**The weaker fact did not stop mattering, though, and it is what a stale record is read against.**
+A workload check can come back undeterminable for reasons that have nothing to do with whether a
+daemon is up (no probe payload in a deployment's own image variant, an architecture it is not built
+for, a `docker load` the engine refuses, a timeout), so falling straight back to the boot record
+there would re-latch the very refusal the paragraph below rules out. The check therefore reports
+`daemonAnswered` alongside its `unknown`, established on its way past at no extra cost, and a
+daemon that merely ANSWERED overrules a recorded absence exactly as the old `docker version` probe
+did. Only a check that never reached a daemon at all leaves the record to decide.
+
+The workload check is memoised per container for a POSITIVE answer only. A daemon that has run a
+container proved something that does not stop being true; a negative is re-measured for the same
+reason a recorded absence is, and it fails fast anyway.
 
 The verdict is three-valued, and that is the point. `false` is a decided absence. `undefined` is
 "nothing decided" — the probe is still in flight, or nothing recorded anything at all, which is the
 normal state under the native host transport (`LOCAL_NATIVE_AGENTS`) where the harness runs on a
-developer's machine with no entrypoint. Undecided attempts the stand-up; only a decided absence
-refuses it.
+developer's machine with no entrypoint. Undecided attempts the stand-up, and nothing probes it into
+a refusal; only a decided negative refuses.
 
 What is recorded describes BOOT, and a container outlives its boot: a warm pool serves many jobs
 from one, and a sidecar daemon that took longer to come up than the entrypoint's bounded wait
-allows is serving perfectly well by the second job. So a recorded absence is a hypothesis, not the
+allows is serving perfectly well by the second job. So a recorded verdict is a hypothesis, not the
 refusal: `resolveDockerVerdict` re-checks it against a live daemon at the moment a stand-up is
 about to run, and the record supplies what only the record holds, the cause and the daemon's own
 log tail. `GET /health` deliberately keeps reporting the boot record rather than probing per poll,
@@ -438,10 +479,13 @@ verdict is stated.
 | `src/bootstrap-mode.ts` | The repo-bootstrap MODE: clone-a-reference-or-scaffold → run the agent → refuse to push an empty tree → reinit + force-push to the pre-created target repo. |
 | `src/artifact-upload.ts` | The OUTBOUND half of the artifact seam: parses the body's `artifactUpload` and projects it onto the agent's env as `ARTIFACT_UPLOAD_URL` / `ARTIFACT_UPLOAD_TOKEN`, registering the token for redaction first. Passes through what the body carries and decides nothing: which kinds get the seam is the backend's call. |
 | `src/codex-images.ts` | Codex's own `image_gen` output, staged where the agent can reach it: creates `$CODEX_HOME/generated_images` as a symlink into `.cat-context/binary-output/generated/` before the CLI starts, sweeps anything a failed redirect left behind, and unlinks (never follows) the redirect at teardown — a failed unlink is REPORTED, because that unlink is what stops the recursive delete reaching the checkout. Exists because codex exposes no path for what it generated AND `$CODEX_HOME` holds the run's decrypted credential, so neither asking the agent nor sending it there is available. |
-| `src/environment-inventory.ts` | What the MACHINE holds, probed once per job and appended to the agent's system prompt as an ENVIRONMENT INVENTORY block. The only layer that can state it: the backend composes its prompt before a transport is chosen, and the same body serves this image, a deployment's own variant and the developer's laptop under `LOCAL_NATIVE_AGENTS`. Three-valued on purpose, so a probe that failed renders as unknown rather than as an absence, and the Docker DAEMON is answered by running `docker info` rather than by finding the CLI, which is installed here either way. See [The environment is probed once, not by the agent](#the-environment-is-probed-once-not-by-the-agent). |
+| `src/environment-inventory.ts` | What the MACHINE holds, probed once per job and appended to the agent's system prompt as an ENVIRONMENT INVENTORY block. The only layer that can state it: the backend composes its prompt before a transport is chosen, and the same body serves this image, a deployment's own variant and the developer's laptop under `LOCAL_NATIVE_AGENTS`. Three-valued on purpose, so a probe that failed renders as unknown rather than as an absence; Docker gets FIVE, because a daemon that answers `docker info` is not a daemon that can run a container. See [The environment is probed once, not by the agent](#the-environment-is-probed-once-not-by-the-agent). |
 | `src/agent-shared.ts` | The few helpers every agent MODE shares (effort-report folding, the capability fields forwarded to `runAgentInWorkspace`). |
 | `src/logger.ts`    | Structured logging.                                                                                     |
-| `src/docker-status.ts` | This container's own verdict about its Docker daemon, as recorded by `entrypoint.sh`. Three-valued on purpose: a daemon that FAILED and a daemon nobody asked about are different facts, and only a DECIDED absence refuses a stand-up. See [Local infra: the container's Docker daemon](#local-infra-the-containers-docker-daemon). |
+| `src/docker-status.ts` | This container's own verdict about its Docker daemon, as recorded by `entrypoint.sh`. Three-valued on purpose: a daemon that FAILED and a daemon nobody asked about are different facts, and only a DECIDED negative refuses a stand-up. See [Local infra: the container's Docker daemon](#local-infra-the-containers-docker-daemon). |
+| `src/docker-capability.ts` | Whether the daemon can RUN A CONTAINER, which is the fact every caller wanted and `docker info` does not answer. Loads a one-layer image built in-process and runs it; `usable` / `unusable` / `unknown`, and only the container RUN may produce the middle one, AND only where the daemon is what refused it (docker's 126/127, a tag that did not resolve and an unexecutable payload are the platform's own machinery, so they say "could not tell"). Total: it answers even if it throws. One budget for the whole pass, cancelled when the last caller abandons it, memoised per container for a positive and re-measured for a negative. |
+| `src/docker-probe-image.ts` | The one-layer docker-archive that check loads, assembled here from a statically linked binary already in the image, so the whole thing is local: no registry, no network, no second image. Pure, byte-stable, and it names the architecture the DAEMON reported rather than this process's. |
+| `src/docker-command.ts` | How the harness runs one `docker …` command on its own behalf: an argv (no shell), a stdin body, stdout kept apart from stderr, one required timeout, the job's signal, and `killChildProcess` for the kill. Not a second `captured-command.ts`, which stays the one way a DECLARED shell command runs; the header says which of its choices this needs to differ on and why. |
 | `src/agent-env.ts` | The env for anything the harness spawns into the agent's CHECKOUT: its own environment minus the variables that are facts about the HARNESS. Today that is `NODE_ENV` (the harness runs in production mode, and an inherited `NODE_ENV=production` makes npm omit devDependencies in a checkout that never asked for it) and `PORT` (the port this harness is listening on, so a service that reads it would bind the one address in the container's network namespace that is already taken). |
 
 ## Runner lifecycle knobs
@@ -468,6 +512,7 @@ runner):
 | `HARNESS_TRANSCRIPT_ROOT`   | `<tmpdir>/cf-agent-transcripts` | Where retained session transcripts are moved to (one dir per run). Meaningful only on a reused (warm-pool) container; a per-run container is torn down with the job. The TTL sweep deletes only dirs it created (each carries a `.cf-retained` marker), so pointing this at a shared directory never touches unrelated content, though a dedicated dir is still recommended. An override on a different filesystem than the config home falls back to copy-then-remove. |
 | `HARNESS_DOCKER_READY_TIMEOUT_SECONDS` | `60` | How long `entrypoint.sh` waits for the container's Docker daemon before recording it unavailable. Only a HUNG daemon pays this in full: the wait ends early both when the socket answers and when the daemon process is gone. It runs in the BACKGROUND, so it never delays the container's boot. |
 | `HARNESS_DOCKER_STATUS_FILE` | `/tmp/harness-docker-status.json` | Where that verdict is recorded. `entrypoint.sh` writes it and the harness reads it, so an override must be set for BOTH (they share one process env). |
+| `HARNESS_DOCKER_PROBE_BINARY` | `/bin/busybox` | The statically linked binary the platform builds its one-layer probe image from, for the check that answers whether this daemon can RUN a container. Only an image variant that ships it elsewhere needs this. Absent is a supported answer, not a failure: the check then reports that it could not be carried out, which is what the native host transport gets on a developer laptop. |
 
 ## Build / test
 

package/dist/agent.js

@@ -137,8 +137,11 @@ export async function handleAgent(job, opts = {}) {
         // This sits on the critical path AHEAD of the clone, which is the cost of having one
         // composition point instead of one per mode (each mode owns its own clone, so there is no
         // single post-clone place to put this). The pass is sized for that: everything in it runs
-        // concurrently, every probe is a call that answers in milliseconds or is wedged, and the one
-        // deliberate wait is a single short retry for a daemon that is still starting.
+        // concurrently, and every probe either answers in milliseconds or is bounded. Two of them are
+        // deliberate waits rather than instant answers: one short retry for a daemon that is still
+        // starting, and, only once a daemon has answered, the container the platform runs to find out
+        // whether it can run one at all (`docker-capability.ts`, budgeted and memoised per container
+        // for a positive). Both take the job's signal, so an abandoned run stops paying at once.
         const staged = {
             ...job,
             systemPrompt: await appendEnvironmentInventory(job.systemPrompt, {

package/dist/docker-capability.d.ts

@@ -0,0 +1,99 @@
+import { type DockerCommandRunner } from './docker-command.js';
+import { type Logger } from './logger.js';
+/** What one measurement concluded. See the three answers above; nothing collapses them. */
+export type DockerWorkload = {
+    status: 'usable';
+} | {
+    status: 'unusable';
+    detail: string;
+} | {
+    status: 'unknown';
+    reason: string;
+    /**
+     * Whether a daemon ANSWERED before the check ran out of things it could do. A weaker fact
+     * than the check exists to establish, and the one a stale boot record must be read against.
+     */
+    daemonAnswered: boolean;
+};
+/**
+ * A one-slot memo for the assembled archive.
+ *
+ * The archive is byte-stable for one `(payload path, architecture)` pair by construction
+ * (docker-probe-image.ts pins every timestamp for exactly this reason), and neither half changes
+ * within a process. Since a NEGATIVE verdict is re-measured on purpose and two sites ask per job,
+ * without this the same two megabytes are re-read and re-sha256'd four times per job for a value
+ * that cannot differ. One slot rather than a map: there is only ever one key.
+ */
+export interface ProbeArchiveMemo {
+    read(key: string): Buffer | undefined;
+    write(key: string, archive: Buffer): void;
+}
+/** Build a {@link ProbeArchiveMemo}. Supplied only by {@link realDeps}, so a test memoises nothing. */
+export declare function oneSlotArchiveMemo(): ProbeArchiveMemo;
+/** What a measurement needs from the machine, so a test can supply all of it. */
+export interface DockerWorkloadDeps {
+    /** Read the probe payload. Rejecting (ENOENT) is the supported "this machine has none". */
+    readPayload: (path: string) => Promise<Buffer>;
+    payloadPath: string;
+    runDocker: DockerCommandRunner;
+    /** This process's architecture, as `process.arch` spells it: the PAYLOAD's, never the daemon's. */
+    arch: string;
+    logger?: Logger;
+    archives?: ProbeArchiveMemo;
+}
+/**
+ * Carry out one measurement. Pure of caching, so the suite states every branch directly.
+ *
+ * TOTAL: it never rejects, whatever happens inside it. The thing it replaced was total by
+ * construction (a `try/catch` around one `execFile`), and it is consulted from a stand-up path
+ * documented as best-effort, so a throw here would fail a job over a probe whose whole purpose is
+ * to make a failure legible. A throw is also, by definition, the platform's own machinery
+ * breaking, which is the `unknown` disposition and never the `unusable` one.
+ *
+ * That asymmetry is the design. Only the RUN produces `unusable`; everything before it produces
+ * `unknown`, because everything before it is the platform's own machinery and a bug in it must be
+ * able to say "I could not tell" and never "your daemon is broken". The load step in particular is
+ * the one this repo wrote itself.
+ */
+export declare function measureDockerWorkload(deps?: DockerWorkloadDeps, signal?: AbortSignal): Promise<DockerWorkload>;
+/**
+ * A measurement, plus what the last one concluded without taking another.
+ *
+ * Callable because every caller wants the verdict; `last()` exists for `GET /health`, which is
+ * polled and must not spawn a container per poll to answer a question it does not act on.
+ */
+export interface DockerWorkloadProbe {
+    (signal?: AbortSignal): Promise<DockerWorkload>;
+    last(): DockerWorkload | undefined;
+}
+/**
+ * Build a probe that measures at most once per container for a POSITIVE answer.
+ *
+ * A daemon that has run a container has proved something that does not stop being true, so that
+ * verdict is kept and every later job reads it for free. A negative is NOT kept, for the reason
+ * `resolveDockerVerdict` gives about the boot record: a container outlives its boot, a warm pool
+ * serves many jobs from one, and a daemon that was not ready for the first job must not latch
+ * the whole container into saying so. Re-measuring a negative is cheap; a daemon that cannot
+ * mount fails at once.
+ *
+ * Concurrent callers share one in-flight measurement rather than each starting a container, and
+ * the measurement is cancelled when the LAST of them has abandoned it. Neither half is optional:
+ * one job's abort may not kill a measurement a sibling job is still waiting on (the local native
+ * transport serves every concurrent job from one process), and a measurement nobody is waiting
+ * for is a container start no job will read, which is what an abandoned run should stop paying
+ * for the moment it is abandoned.
+ */
+export declare function createDockerWorkloadProbe(deps?: DockerWorkloadDeps): DockerWorkloadProbe;
+/** The process-wide probe. One per container, which is what makes the positive memo worth having. */
+export declare const probeDockerWorkload: DockerWorkloadProbe;
+/**
+ * What `GET /health` reports about the workload check.
+ *
+ * `unmeasured` is its own word rather than an omitted key or a `null`: this endpoint is polled
+ * from boot, so "nothing has needed the daemon yet" is the normal early answer and it must not
+ * read as either a broken daemon or a build that cannot report one.
+ */
+export declare function reportedDockerWorkload(probe?: DockerWorkloadProbe): DockerWorkload | {
+    status: 'unmeasured';
+    reason: string;
+};

package/dist/docker-capability.js

@@ -0,0 +1,351 @@
+import { readFile } from 'node:fs/promises';
+import { spawnDockerCommand, } from './docker-command.js';
+import { buildProbeArchive, payloadArchitecture, PROBE_COMMAND, PROBE_IMAGE_TAG, PROBE_SENTINEL, } from './docker-probe-image.js';
+import { log } from './logger.js';
+import { redactSecrets } from './redact.js';
+/**
+ * The statically linked binary the probe image is built from, overridable for an image variant
+ * that ships it elsewhere. Absent is a supported answer, not a failure: under
+ * `LOCAL_NATIVE_AGENTS` the harness runs on a developer's machine that never saw this image.
+ */
+const PAYLOAD_PATH = process.env.HARNESS_DOCKER_PROBE_BINARY?.trim() || '/bin/busybox';
+/**
+ * The ceiling on ONE WHOLE measurement, shared out across the docker commands it makes: each
+ * gets what is left of it, down to {@link MIN_COMMAND_MS}.
+ *
+ * One budget rather than a per-command ceiling, because a per-command one multiplies: three
+ * commands at 30s each is a minute and a half of dead time on a wedged daemon, on the critical
+ * path ahead of the clone. Sized for a WEDGED daemon and not for a slow one: the payload is a
+ * couple of megabytes already on local disk, so a daemon that works answers in about the time it
+ * takes to start one container, and a daemon that cannot mount fails immediately.
+ *
+ * What it does NOT do is bound the cost per JOB, and the comment that once claimed so was wrong.
+ * A POSITIVE verdict is memoised for the container's life; a negative is deliberately
+ * re-measured (see {@link createDockerWorkloadProbe}), and two independent sites ask per job (the
+ * environment inventory and the compose stand-up), so a serving-but-wedged daemon costs this
+ * twice per job. That is the price of not latching a warm container into a stale refusal, which
+ * is why the budget is the size it is and why an abandoned job stops paying it at once.
+ */
+const WORKLOAD_BUDGET_MS = 20_000;
+/** The floor on one command's share of the budget, so an exhausted budget still gets an answer. */
+const MIN_COMMAND_MS = 1_000;
+/**
+ * The ceiling on removing the probe image again.
+ *
+ * Its own, and deliberately NOT given the caller's abort signal: the image is the platform's, and
+ * a cancelled job is the one case where nobody is left to clean up after it. Bounded separately
+ * so a wedged daemon cannot turn the cleanup into a second full budget.
+ */
+const CLEANUP_TIMEOUT_MS = 5_000;
+/** How much of a failing command's output is kept. It is quoted into an agent's system prompt. */
+const DETAIL_CHARS = 300;
+/** Build a {@link ProbeArchiveMemo}. Supplied only by {@link realDeps}, so a test memoises nothing. */
+export function oneSlotArchiveMemo() {
+    let held;
+    return {
+        read: (key) => (held?.key === key ? held.archive : undefined),
+        write: (key, archive) => {
+            held = { key, archive };
+        },
+    };
+}
+const realDeps = {
+    readPayload: (path) => readFile(path),
+    payloadPath: PAYLOAD_PATH,
+    runDocker: spawnDockerCommand,
+    arch: process.arch,
+    archives: oneSlotArchiveMemo(),
+};
+/**
+ * Carry out one measurement. Pure of caching, so the suite states every branch directly.
+ *
+ * TOTAL: it never rejects, whatever happens inside it. The thing it replaced was total by
+ * construction (a `try/catch` around one `execFile`), and it is consulted from a stand-up path
+ * documented as best-effort, so a throw here would fail a job over a probe whose whole purpose is
+ * to make a failure legible. A throw is also, by definition, the platform's own machinery
+ * breaking, which is the `unknown` disposition and never the `unusable` one.
+ *
+ * That asymmetry is the design. Only the RUN produces `unusable`; everything before it produces
+ * `unknown`, because everything before it is the platform's own machinery and a bug in it must be
+ * able to say "I could not tell" and never "your daemon is broken". The load step in particular is
+ * the one this repo wrote itself.
+ */
+export async function measureDockerWorkload(deps = realDeps, signal) {
+    const seen = { daemonAnswered: false };
+    try {
+        return await measure(deps, seen, signal);
+    }
+    catch (err) {
+        const cause = describeThrown(err);
+        (deps.logger ?? log).warn('docker capability: the container check itself fell over', {
+            error: cause,
+        });
+        return undeterminable(`the platform's own container check could not be completed (${cause})`, seen.daemonAnswered);
+    }
+}
+async function measure(deps, seen, signal) {
+    const deadline = Date.now() + WORKLOAD_BUDGET_MS;
+    const share = () => Math.max(MIN_COMMAND_MS, deadline - Date.now());
+    const command = (args, stdin) => deps.runDocker(args, {
+        ...(stdin ? { stdin } : {}),
+        ...(signal ? { signal } : {}),
+        timeoutMs: share(),
+        ...(deps.logger ? { logger: deps.logger } : {}),
+    });
+    // Ask the DAEMON which architecture it runs, rather than assuming it shares this process's.
+    // An external `DOCKER_HOST` is a supported path, and an arm64 harness against an amd64 sidecar
+    // (or a remote x86_64 daemon reached from an arm64 laptop) shares nothing with it but a socket:
+    // declaring the wrong one in the image config gets the run refused, which would report a
+    // perfectly good daemon as one that cannot run containers. It is also the cheapest proof that a
+    // daemon is answering at all, which is the fact `resolveDockerVerdict` reads back.
+    const asked = await command(['version', '--format', '{{.Server.Arch}}']);
+    if (asked.outcome !== 'ran' || asked.code !== 0) {
+        return undeterminable(`no Docker daemon answered the platform's container check (${describeOutcome(asked)})`, false);
+    }
+    seen.daemonAnswered = true;
+    const daemonArch = asked.stdout.trim();
+    if (!/^[a-z0-9_]+$/.test(daemonArch)) {
+        return undeterminable(`the Docker daemon did not name an architecture the platform can build an image for (${redactSecrets(daemonArch).slice(0, 40) || 'it answered nothing'})`, true);
+    }
+    const payloadArch = payloadArchitecture(deps.arch);
+    if (!payloadArch) {
+        return undeterminable(`the platform has no container check for the ${deps.arch} architecture`, true);
+    }
+    if (payloadArch !== daemonArch) {
+        return undeterminable(`the platform's container check is built for ${payloadArch} and this daemon runs ${daemonArch}`, true);
+    }
+    const assembled = await assembleArchive(deps, daemonArch);
+    if ('reason' in assembled)
+        return undeterminable(assembled.reason, true);
+    const load = await command(['load'], assembled.archive);
+    if (load.outcome !== 'ran' || load.code !== 0) {
+        return undeterminable(`the platform could not load its own probe image (${describeOutcome(load)})`, true);
+    }
+    const run = await command([
+        'run',
+        '--rm',
+        '--pull',
+        'never',
+        '--network',
+        'none',
+        PROBE_IMAGE_TAG,
+        ...PROBE_COMMAND,
+    ]);
+    // Before the verdict, and whatever the verdict is: the probe image is the platform's, and an
+    // agent that runs `docker images` should not have to wonder whose it is.
+    await removeProbeImage(deps);
+    return classifyRun(run);
+}
+/**
+ * What the container RUN proves. The one step allowed to conclude `unusable`, and even here not
+ * every non-zero exit is evidence about the daemon.
+ *
+ * Docker splits its own failures from the container's by exit code: 125 is `docker run` itself
+ * failing, 126 is a command that could not be invoked and 127 one that was not found. The last
+ * two are facts about THE PAYLOAD, a binary this platform put in an image it built, and the
+ * container had to be created and started to produce them. 125 covers both the daemon refusing to
+ * create the container (the verdict this whole module exists for) and the tag not being there to
+ * run, which is our own load, so that one is split by what docker SAID.
+ */
+function classifyRun(run) {
+    if (run.outcome === 'failed') {
+        return undeterminable(`the platform's container check did not run (${run.reason})`, true);
+    }
+    if (run.code === 0) {
+        if (run.stdout.includes(PROBE_SENTINEL))
+            return { status: 'usable' };
+        // Nothing explains this: the container was reported as having run cleanly and produced none
+        // of the output it exists to produce. That is a fact about the check, not about the daemon.
+        return undeterminable("the platform's probe container exited cleanly without printing its marker", true);
+    }
+    const ours = platformSideRunFailure(run);
+    return ours
+        ? undeterminable(`${ours} (${describeOutcome(run)})`, true)
+        : { status: 'unusable', detail: describeOutcome(run) };
+}
+/** Messages that name the PLATFORM's half of a failed run rather than the daemon's. */
+const PLATFORM_SIDE_RUN_MESSAGES = [
+    {
+        pattern: /no such image|unable to find image/i,
+        cause: "the platform's probe image was not there to be run",
+    },
+    {
+        pattern: /exec format error/i,
+        cause: "the platform's probe binary cannot be executed on this daemon's machine",
+    },
+];
+function platformSideRunFailure(run) {
+    if (run.code === 126 || run.code === 127) {
+        return "the platform's probe binary could not be invoked inside the container";
+    }
+    const said = `${run.stderr}\n${run.stdout}`;
+    return PLATFORM_SIDE_RUN_MESSAGES.find((m) => m.pattern.test(said))?.cause;
+}
+/**
+ * Assemble the archive for the daemon's architecture, reusing the last one built.
+ *
+ * A read that fails is classified rather than asserted away: `HARNESS_DOCKER_PROBE_BINARY`
+ * pointing at a directory, at a path this user may not read, or at a failing mount is a
+ * misconfiguration an operator can fix, and "this machine does not have it" states the opposite
+ * fact. The sentence goes into an agent's system prompt and into `GET /health`, so a discarded
+ * cause is a cause nobody ever sees.
+ */
+async function assembleArchive(deps, architecture) {
+    const key = `${deps.payloadPath}::${architecture}`;
+    const held = deps.archives?.read(key);
+    if (held)
+        return { archive: held };
+    let payload;
+    try {
+        payload = await deps.readPayload(deps.payloadPath);
+    }
+    catch (err) {
+        return { reason: describePayloadFailure(err, deps.payloadPath) };
+    }
+    const archive = buildProbeArchive(payload, architecture);
+    deps.archives?.write(key, archive);
+    return { archive };
+}
+function describePayloadFailure(err, path) {
+    const needs = `the platform's own container check needs ${path}`;
+    switch (err.code) {
+        case 'ENOENT':
+        case 'ENOTDIR':
+            return `${needs}, which this machine does not have`;
+        case 'EACCES':
+        case 'EPERM':
+            return `${needs}, which it is not permitted to read`;
+        case 'EISDIR':
+            return `${needs} to be a file, and it is a directory`;
+        default:
+            return `${needs}, which could not be read (${describeThrown(err)})`;
+    }
+}
+/**
+ * Remove the probe image, and SAY SO when that did not work.
+ *
+ * The one line above it promises an agent will never find a `cat-factory-docker-probe` and wonder
+ * whose it is, and the daemon has two ordinary ways to refuse: a `--rm` teardown still in flight
+ * holds the image ("image is being used by stopped container"), and a wedged daemon does not
+ * answer at all. Discarding the outcome left both silent, so the promise was unverifiable in
+ * exactly the states that break it.
+ */
+async function removeProbeImage(deps) {
+    const removed = await deps.runDocker(['image', 'rm', '-f', PROBE_IMAGE_TAG], {
+        timeoutMs: CLEANUP_TIMEOUT_MS,
+        ...(deps.logger ? { logger: deps.logger } : {}),
+    });
+    if (removed.outcome === 'ran' && removed.code === 0)
+        return;
+    (deps.logger ?? log).warn('docker capability: the probe image could not be removed', {
+        image: PROBE_IMAGE_TAG,
+        error: describeOutcome(removed),
+    });
+}
+function undeterminable(reason, daemonAnswered) {
+    return { status: 'unknown', reason, daemonAnswered };
+}
+/** A bounded, scrubbed one-line summary of what a command said, for a prompt or a log field. */
+function describeOutcome(outcome) {
+    if (outcome.outcome === 'failed')
+        return outcome.reason;
+    const said = `${outcome.stderr}\n${outcome.stdout}`
+        .split('\n')
+        .map((line) => line.trim())
+        .filter(Boolean)
+        .join('; ');
+    return bounded(said) || `docker exited ${outcome.code} without saying why`;
+}
+/** The one describer for a thrown value here: scrubbed and bounded, like any other detail. */
+function describeThrown(err) {
+    return bounded(err instanceof Error ? err.message : String(err)) || 'it said nothing';
+}
+function bounded(text) {
+    const scrubbed = redactSecrets(text);
+    return scrubbed.length > DETAIL_CHARS ? `${scrubbed.slice(0, DETAIL_CHARS)}…` : scrubbed;
+}
+/**
+ * Build a probe that measures at most once per container for a POSITIVE answer.
+ *
+ * A daemon that has run a container has proved something that does not stop being true, so that
+ * verdict is kept and every later job reads it for free. A negative is NOT kept, for the reason
+ * `resolveDockerVerdict` gives about the boot record: a container outlives its boot, a warm pool
+ * serves many jobs from one, and a daemon that was not ready for the first job must not latch
+ * the whole container into saying so. Re-measuring a negative is cheap; a daemon that cannot
+ * mount fails at once.
+ *
+ * Concurrent callers share one in-flight measurement rather than each starting a container, and
+ * the measurement is cancelled when the LAST of them has abandoned it. Neither half is optional:
+ * one job's abort may not kill a measurement a sibling job is still waiting on (the local native
+ * transport serves every concurrent job from one process), and a measurement nobody is waiting
+ * for is a container start no job will read, which is what an abandoned run should stop paying
+ * for the moment it is abandoned.
+ */
+export function createDockerWorkloadProbe(deps = realDeps) {
+    let latest;
+    let inFlight;
+    const begin = () => {
+        const cancel = new AbortController();
+        const measurement = {
+            cancel,
+            waiters: 0,
+            result: measureDockerWorkload(deps, cancel.signal).then((verdict) => {
+                latest = verdict;
+                if (inFlight === measurement)
+                    inFlight = undefined;
+                return verdict;
+            }),
+        };
+        return measurement;
+    };
+    const probe = (async (signal) => {
+        if (latest?.status === 'usable')
+            return latest;
+        const measurement = (inFlight ??= begin());
+        measurement.waiters += 1;
+        const watch = signal ? watchAbandonment(signal) : undefined;
+        try {
+            return watch
+                ? await Promise.race([measurement.result, watch.abandoned])
+                : await measurement.result;
+        }
+        finally {
+            watch?.dispose();
+            measurement.waiters -= 1;
+            if (measurement.waiters === 0 && inFlight === measurement)
+                measurement.cancel.abort();
+        }
+    });
+    probe.last = () => latest;
+    return probe;
+}
+/**
+ * A verdict for the caller whose job was cancelled while it waited, and the listener teardown
+ * that keeps a long-lived native-transport process from accumulating one per job.
+ */
+function watchAbandonment(signal) {
+    let give = () => { };
+    const abandoned = new Promise((resolve) => {
+        give = () => resolve(undeterminable("the job was cancelled before the platform's container check answered", false));
+        if (signal.aborted)
+            give();
+        else
+            signal.addEventListener('abort', give, { once: true });
+    });
+    return { abandoned, dispose: () => signal.removeEventListener('abort', give) };
+}
+/** The process-wide probe. One per container, which is what makes the positive memo worth having. */
+export const probeDockerWorkload = createDockerWorkloadProbe();
+/**
+ * What `GET /health` reports about the workload check.
+ *
+ * `unmeasured` is its own word rather than an omitted key or a `null`: this endpoint is polled
+ * from boot, so "nothing has needed the daemon yet" is the normal early answer and it must not
+ * read as either a broken daemon or a build that cannot report one.
+ */
+export function reportedDockerWorkload(probe = probeDockerWorkload) {
+    return (probe.last() ?? {
+        status: 'unmeasured',
+        reason: 'nothing in this container has needed the docker daemon yet',
+    });
+}

package/dist/docker-command.d.ts

@@ -0,0 +1,30 @@
+import { type Logger } from './logger.js';
+/** What running one docker command did, kept as raw as the spawn. */
+export type CommandOutcome = {
+    outcome: 'ran';
+    code: number;
+    stdout: string;
+    stderr: string;
+} | {
+    outcome: 'failed';
+    reason: string;
+};
+/** What one docker invocation is given. `timeoutMs` is required: an unbounded one has no caller. */
+export interface DockerCommandOptions {
+    /** Piped to the command's stdin and closed. */
+    stdin?: Buffer;
+    /** The job's signal. An abandoned job's command is killed rather than left running. */
+    signal?: AbortSignal;
+    timeoutMs: number;
+    logger?: Logger;
+}
+/** Run one `docker …` command. Injected so the suite drives every branch with no daemon. */
+export type DockerCommandRunner = (args: string[], opts: DockerCommandOptions) => Promise<CommandOutcome>;
+/**
+ * The real runner: spawn docker, feed it `stdin` when there is any, and report what happened.
+ *
+ * Never rejects. Every way a spawn can go wrong is one of the two outcomes, because the caller
+ * classifies them differently and an exception would collapse that distinction into whichever
+ * `catch` caught it first.
+ */
+export declare const spawnDockerCommand: DockerCommandRunner;