@cat-factory/executor-harness 1.94.0 → 1.98.0

package/README.md

@@ -1,7 +1,7 @@
 # @cat-factory/executor-harness
 
 The payload that runs **inside** a per-run Cloudflare Container (or a
-[self-hosted runner](../../docs/runner-pool-integration.md)) to perform real
+[self-hosted runner](https://github.com/kibertoad/cat-factory/blob/main/backend/docs/runner-pool-integration.md)) to perform real
 repo work with the [Pi coding agent](https://github.com/earendil-works/pi).
 
 It is a thin TypeScript wrapper (a `node:http` server on `:8080`) that the
@@ -33,7 +33,7 @@ replayed `POST` **re-attaches** to the running job rather than starting a
 duplicate (the durable driver's retries/replays are safe). Pi's todo-tool counts
 are surfaced as `progress` while a job runs. The exact request/response shapes
 cat-factory sends are documented in
-[`docs/runner-pool-integration.md`](../../docs/runner-pool-integration.md).
+[`docs/runner-pool-integration.md`](https://github.com/kibertoad/cat-factory/blob/main/backend/docs/runner-pool-integration.md).
 
 `GET /jobs/{id}` is also the harness's observability channel: `spans`, `followUps`
 and `callMetrics` are **drain-on-read**; each poll returns what accumulated since
@@ -62,7 +62,7 @@ The implementation job (`POST /run`) is the canonical sequence:
    body's `proxyPhasePath` says the backend serves it, which is how a repair round's model spend
    stays distinguishable from the first pass's in telemetry; without that flag the plain path is
    used and the calls are recorded as unattributed
-   (see [token-burn instrumentation](../../../docs/initiatives/token-burn-instrumentation.md)),
+   (see [token-burn instrumentation](https://github.com/kibertoad/cat-factory/blob/main/docs/initiatives/token-burn-instrumentation.md)),
 3. **prepopulate dependencies**, when the job body carries `dependencyInstall`: the
    service's install command is run with `sh -c` in the checkout BEFORE the agent starts, so
    it reads real installed packages instead of inferring a library's capabilities from a
@@ -71,7 +71,7 @@ The implementation job (`POST /run`) is the canonical sequence:
    steps 6 and 7, which start a fresh agent) and the run continues either way. Whatever the
    install materialises is excluded from git first, so no later `git add -A` can sweep a
    dependency tree into the pull request (see
-   [dependency prepopulation](../../../docs/initiatives/agent-dependency-prepopulation.md)),
+   [dependency prepopulation](https://github.com/kibertoad/cat-factory/blob/main/docs/initiatives/agent-dependency-prepopulation.md)),
 4. **resolve the repo's pull-request template**, when this dispatch opens a PR (`src/pr-template.ts`):
   `.github/PULL_REQUEST_TEMPLATE.md` and its root/`docs/`/multi-template-directory variants, or
    GitLab's `.gitlab/merge_request_templates/`, read straight off the checkout (a symlinked template
@@ -85,11 +85,11 @@ The implementation job (`POST /run`) is the canonical sequence:
 6. **validate** the checkout, when the job body carries `validationChecks`: the service's
    configured check commands (install/lint/test/build) run with `sh -c` in the checkout, and
    while they fail and the attempt budget remains the agent is re-run with the captured output
-   as its instruction (see [pre-PR validation](../../../docs/initiatives/pre-pr-validation.md)),
+   as its instruction (see [pre-PR validation](https://github.com/kibertoad/cat-factory/blob/main/docs/initiatives/pre-pr-validation.md)),
 7. **prove the reproduction**, when the job body carries `reproduction`: the declared check is
    run against the pre-fix tree and the tree the PR will open from, in two freshly-created
    symmetric `git worktree` checkouts, and only red-then-green is reported as proof (see
-   [bugfix reproduction proof](../../docs/adr/0033-bugfix-reproduction-proof.md)). Unlike
+   [bugfix reproduction proof](https://github.com/kibertoad/cat-factory/blob/main/backend/docs/adr/0033-bugfix-reproduction-proof.md)). Unlike
    step 6 this NEVER gates the PR: a failed verification is fed back to the agent while budget
    remains, then recorded as `inconclusive`. It runs BEFORE step 6 so validation stays the last
    thing to touch the tree,
@@ -120,7 +120,7 @@ PR. Blueprint **commits onto a branch** (no history reset) and returns the tree.
 A job body may carry `skills[]` (procedural playbooks) and `mcpServers[]` (MCP tool servers): the
 harness MATERIALISES both and decides nothing about them; the backend has already resolved which
 apply and dropped what this harness cannot serve (see
-[`backend/docs/adr/0029-agent-kind-capabilities.md`](../../docs/adr/0029-agent-kind-capabilities.md)).
+[`backend/docs/adr/0029-agent-kind-capabilities.md`](https://github.com/kibertoad/cat-factory/blob/main/backend/docs/adr/0029-agent-kind-capabilities.md)).
 
 - **Skills** install natively under `CLAUDE_CONFIG_DIR/skills/<name>/` for a leased-credential
   claude-code run (the CLI discovers and invokes them), and under
@@ -210,9 +210,12 @@ Kimi / DeepSeek) and meters spend. The provider key never enters the container.
 | File               | Responsibility                                                                                          |
 | ------------------ | ------------------------------------------------------------------------------------------------------- |
 | `src/server.ts`    | HTTP entry point; routes `/health`, `/run`, `/bootstrap`, `/blueprint`, `/jobs/{id}`.                   |
-| `src/runner.ts`    | `JobRegistry`: async job lifecycle, idempotent on `jobId`, progress tracking.                          |
+| `src/runner.ts`    | `JobRegistry`: async job lifecycle, idempotent on `jobId`, progress tracking, and the three per-job watchdogs (max-duration, inactivity, tool-silence).                          |
+| `src/jsonl-stream.ts` | The BOUNDS on a child CLI's streams, shared by both runners: `JsonlLineReader` frames its JSONL stdout while refusing to buffer a runaway record, `BoundedTail` keeps a capped tail of raw output for failure quoting. Both watchdog timers and the poll endpoints share one event loop with this parsing, so an unbounded buffer here is how a container stops answering polls with no watchdog having fired. |
 | `src/job.ts`       | Request types + validators for the job specs.                                                           |
 | `src/pi.ts`        | Pi provider config, non-interactive run, JSON-line event + todo-progress parsing, global `AGENTS.md` guidance. |
+| `src/pi-reduction.ts` | Reducing a Pi event stream to what the run PRODUCED (summary, stats, diagnostics, terminal failure), FOLDED as records stream rather than over a retained array — memory is O(largest record), not O(records). The array-taking entry points offline tooling uses are defined in terms of the same reducer. |
+| `src/tool-silence.ts` | The tool-silence watchdog (F13) and the `ToolProgressWindow` an agent stream opens, beats and closes. Separate from the phase marker on purpose: a window is only meaningful while something able to reset it is running. |
 | `src/git.ts`       | clone / branch / commit / push + GitHub PR creation; bootstrap history reset + force-push.              |
 | `src/bootstrap.ts` | The `/bootstrap` handler (clone-or-empty → adapt → reinit + force-push).                                |
 | `src/blueprint.ts` | The `/blueprint` handler (decompose → render `blueprints/` → commit on branch).                         |
@@ -240,6 +243,7 @@ runner):
 | `PORT`                | `8080`          | HTTP port the harness listens on.                           |
 | `JOB_MAX_DURATION_MS` | `3600000` (60m) | Hard ceiling on a job's wall-clock time; force-fails after. |
 | `JOB_INACTIVITY_MS`   | `600000` (10m)  | Kills a hung agent that produces no output for this long.   |
+| `JOB_TOOL_SILENCE_MS` | half `JOB_MAX_DURATION_MS` (30m at its default) | Kills an agent that keeps producing output but completes no tool call for this long: the "chatty hang" neither watchdog above can see, since streamed output resets the inactivity timer on every chunk while nothing gets done (stuck-run audit F13). Armed ONLY while an agent CLI that reports completed tool calls is running (each runner opens its own window and closes it on exit), so clone / dependency install / push / a validation loop's check commands sit outside it — they are activity-silent by nature and bounded by their own per-command timeouts — and each repair pass opens a fresh window. Derived from the job ceiling rather than fixed. It fires only when output arrived during the window that elapsed, which is what leaves a genuinely quiet run to `JOB_INACTIVITY_MS` and its clearer diagnostic. `0` disables it. |
 | `JOB_MAX_CONSECUTIVE_MCP_CALLS` | `40` | Consecutive tool-server (`mcp__*`) calls with no other tool call between before the run counts as a lookup loop. The counter-bound the no-edit exemption above owes; a per-kind `tuning.guardLimits` entry can only RAISE it. |
 | `JOB_MAX_CONSECUTIVE_NON_ACTION_CALLS` | `200` | Consecutive calls of ANY no-edit-exempt family (reads, searches, web, tool servers, subagent dispatches) with no action call between them. The backstop above the per-family caps, since each of those resets on a call outside its own family; sized as a backstop rather than a research judgement, and reset by any `bash`/edit. |
 | `JOB_COLD_START_MS`   | `120000` (2m)   | First-output window (ADR 0026 D4). A job that has produced nothing this long records a cold-start diagnostic (a likely onboarding/auth wedge) WITHOUT being killed: logged, exposed on `GET /jobs/{id}`, and folded into the failure `detail` if the job goes on to fail. `0` disables it. |
@@ -279,7 +283,7 @@ docker.io/<org>/cat-factory-executor:<version>
 Each is tagged with the package `version`, the commit `sha-…`, and `latest`.
 
 **CI** does this automatically:
-[`.github/workflows/docker-publish.yml`](../../../.github/workflows/docker-publish.yml)
+[`.github/workflows/docker-publish.yml`](https://github.com/kibertoad/cat-factory/blob/main/.github/workflows/docker-publish.yml)
 republishes on every push to `main` that touches image content (`src/**`,
 `Dockerfile`, `tsconfig.json`, `package.json`). Docker Hub is gated on the
 `DOCKERHUB_USERNAME` / `DOCKERHUB_TOKEN` repo secrets; without them it publishes
@@ -303,9 +307,9 @@ via env vars (`REGISTRIES`, `GHCR_OWNER`, `DOCKERHUB_ORG`, `TAG`, `PUSH_LATEST`,
 
 A backend deployment references the image from `wrangler.toml`
 (`[[containers]] image = "ghcr.io/<owner>/cat-factory-executor:<version>"`: see
-[`deploy/backend`](../../../deploy/backend)); a self-hosted runner pool pulls the
-same image (see [`docs/runner-pool-integration.md`](../../docs/runner-pool-integration.md)).
+[`deploy/backend`](https://github.com/kibertoad/cat-factory/tree/main/deploy/backend)); a self-hosted runner pool pulls the
+same image (see [`docs/runner-pool-integration.md`](https://github.com/kibertoad/cat-factory/blob/main/backend/docs/runner-pool-integration.md)).
 The worker library's own test/dev `wrangler.toml` still references this
 `Dockerfile` by local path so the acceptance suite can build it. Because the
 version is the image tag, **bump this package via a changeset whenever you change
-image content** (see [`CONTRIBUTING.md`](../../../CONTRIBUTING.md)).
+image content** (see [`CONTRIBUTING.md`](https://github.com/kibertoad/cat-factory/blob/main/CONTRIBUTING.md)).

package/dist/agent-capabilities.d.ts

@@ -42,6 +42,67 @@ export interface McpServerSpec {
      */
     secretKeys?: string[];
 }
+/**
+ * What the agent's CLI reported about ONE wired tool server when it started up.
+ *
+ * This is the OBSERVED half of the run's tool-server record, and it answers a question the
+ * backend's own half structurally cannot: the dispatch record says why the platform WITHHELD a
+ * tool, while this says a server the platform wired failed to start anyway. A vendor endpoint
+ * that 500s, an `npx` package that no longer resolves, a credential the vendor has since revoked
+ * — every one of those leaves the prompt promising a tool the agent then cannot call, and before
+ * this the only evidence was the agent saying so in prose, if it noticed at all.
+ *
+ * OBSERVED, never decided: nothing here changes what the run does. The harness reports what the
+ * CLI said and the backend records it beside what it decided; no code path branches on it.
+ */
+export interface ObservedMcpServer {
+    /** The server id the CLI named — the same id the backend declared (`--strict-mcp-config`). */
+    id: string;
+    status: ObservedMcpStatus;
+    /**
+     * How many of the CLI's exposed tools belong to this server (`mcp__<id>__…`).
+     *
+     * ABSENT and `0` are different facts and both are worth having: absent means this image counted
+     * nothing for the server, while `0` means it counted and the server contributed none: a server
+     * that connected and exposes nothing, which reads to the agent exactly like a server that was
+     * never wired. Never defaulted to 0.
+     *
+     * Two things leave it absent, and neither is "the server has no tools": the CLI listed no tools
+     * at all, or the tool namespace cannot say which of two declared servers a name belongs to (see
+     * {@link tallyToolsByServer}).
+     */
+    toolCount?: number;
+}
+/**
+ * The status vocabulary of {@link ObservedMcpServer}, normalised from the CLI's own word.
+ *
+ * A CLOSED list mapped from an OPEN one, which is why `unknown` is a member rather than a reason
+ * to drop the row. The CLI's status strings are a third party's vocabulary and it may add to them;
+ * a server whose status this image cannot name is still a server the CLI knows about, and the
+ * honest report is "it was there, this image could not read its state" rather than silence
+ * (which reads as a server the CLI never mentioned) or a guess at `ready` (which would report a
+ * dead tool as a live one, the precise failure the whole unavailability vocabulary exists to
+ * prevent).
+ *
+ * `unknown` covers two causes that share one remedy, which is why they share one member: a word
+ * this image cannot map, and a word the CLI uses for a state that is not resolved YET. Neither
+ * says anything about the server, and the surface paints neither as a fault.
+ */
+export type ObservedMcpStatus = 'ready' | 'failed' | 'needs_auth' | 'unknown';
+/**
+ * Read the claude-code CLI's startup report (`{"type":"system","subtype":"init"}`) into one
+ * {@link ObservedMcpServer} per server the CLI knows about.
+ *
+ * The CLI announces its resolved session ONCE, before the first model call: which MCP servers it
+ * loaded and with what status, and the flat list of tool names it will expose. Both halves are
+ * read here because neither answers the question alone — a `ready` server exposing no tools is as
+ * useless to the agent as a failed one, and a tool count with no status cannot say why.
+ *
+ * Returns `undefined` when the event names no servers at all, which keeps "this run wired none"
+ * and "this image observed none" from collapsing into an empty list on the backend's record.
+ * Pure, so the parsing is testable without a CLI: {@link runClaudeCode} feeds it the raw event.
+ */
+export declare function observeClaudeMcpInit(event: Record<string, unknown>): ObservedMcpServer[] | undefined;
 /**
  * The credential values carried by a run's tool servers, for {@link registerKnownSecrets}. An MCP
  * server that fails to start routinely echoes its own argv or request headers into stderr, and

package/dist/agent-capabilities.js

@@ -1,5 +1,118 @@
 import { mkdir, writeFile } from 'node:fs/promises';
 import { dirname, join } from 'node:path';
+/**
+ * Map one status word from the CLI onto {@link ObservedMcpStatus}.
+ *
+ * The synonyms are grouped rather than listed one-to-one because the CLI has spelled the same
+ * two states more than one way across versions (`connected`/`ready`, `failed`/`error`), and an
+ * image that pinned the exact spelling would silently start reporting `unknown` for every server
+ * on a CLI upgrade — a regression that looks identical to a genuine outage.
+ */
+function normalizeMcpStatus(value) {
+    if (typeof value !== 'string')
+        return 'unknown';
+    const status = value.trim().toLowerCase();
+    if (status === 'connected' || status === 'ready' || status === 'ok')
+        return 'ready';
+    if (status === 'failed' || status === 'error')
+        return 'failed';
+    // The vendor spells the OAuth-required state with a hyphen; the underscore form costs nothing
+    // to accept and is what a JSON-ish vocabulary tends to drift toward.
+    if (status === 'needs-auth' || status === 'needs_auth')
+        return 'needs_auth';
+    // Everything else, INCLUDING the CLI's `pending`. A server still handshaking when the session
+    // was announced has no resolved state, which is exactly what `unknown` says, and `needs_auth` is
+    // the tempting wrong guess for it: the surface paints that one amber as "waiting for you to
+    // authorize it", sending an operator to re-issue a working credential for a server that was
+    // merely slow and came up a second later.
+    return 'unknown';
+}
+/**
+ * Read the claude-code CLI's startup report (`{"type":"system","subtype":"init"}`) into one
+ * {@link ObservedMcpServer} per server the CLI knows about.
+ *
+ * The CLI announces its resolved session ONCE, before the first model call: which MCP servers it
+ * loaded and with what status, and the flat list of tool names it will expose. Both halves are
+ * read here because neither answers the question alone — a `ready` server exposing no tools is as
+ * useless to the agent as a failed one, and a tool count with no status cannot say why.
+ *
+ * Returns `undefined` when the event names no servers at all, which keeps "this run wired none"
+ * and "this image observed none" from collapsing into an empty list on the backend's record.
+ * Pure, so the parsing is testable without a CLI: {@link runClaudeCode} feeds it the raw event.
+ */
+export function observeClaudeMcpInit(event) {
+    if (event.type !== 'system' || event.subtype !== 'init')
+        return undefined;
+    const reported = event.mcp_servers;
+    if (!Array.isArray(reported) || reported.length === 0)
+        return undefined;
+    const rows = [];
+    const declared = new Set();
+    for (const entry of reported) {
+        if (typeof entry !== 'object' || entry === null)
+            continue;
+        const record = entry;
+        const id = sanitizeServerId(record.name);
+        // An id this image cannot hold is dropped rather than reported under a mangled name: the
+        // whole row is only useful if it JOINS the backend's declaration, and `--strict-mcp-config`
+        // means every server the CLI loaded came from the config this harness wrote.
+        if (!id || declared.has(id))
+            continue;
+        declared.add(id);
+        rows.push({ id, status: normalizeMcpStatus(record.status) });
+    }
+    if (rows.length === 0)
+        return undefined;
+    // Counted from the CLI's own tool list rather than from a per-server field, because there is no
+    // per-server field: the CLI flattens every server's tools into one array namespaced by server
+    // id. A missing/non-array list leaves every count ABSENT rather than 0 (see `toolCount`).
+    const tally = tallyToolsByServer(event.tools, declared);
+    return rows.map((row) => ({
+        ...row,
+        ...(tally && !tally.ambiguous.has(row.id) ? { toolCount: tally.counts.get(row.id) ?? 0 } : {}),
+    }));
+}
+/**
+ * Tally the CLI's flat tool list (`mcp__<id>__<tool>`) against the servers the SAME event
+ * declared, or `undefined` when it carried no list, which is the distinction
+ * {@link ObservedMcpServer.toolCount} preserves.
+ *
+ * Matched against the declared ids rather than split on the first `__`, because the id vocabulary
+ * ({@link MCP_SERVER_ID_PATTERN}) permits an underscore: a server named `code__search` owns
+ * `mcp__code__search__query`, which a first-separator split files under a server called `code`,
+ * leaving the real one reporting `toolCount: 0`. That is the single most diagnostic value on the
+ * field, so the mis-split renders a fully healthy server as one that started and exposes nothing.
+ *
+ * The same underscore makes genuine ambiguity representable: with both `code` and `code__search`
+ * declared, `mcp__code__search__query` is a name either could own and nothing in the report says
+ * which. Neither server is counted then, and both are named `ambiguous` so their count stays
+ * absent. Guessing an owner would move a real tool onto the wrong server and take the other's
+ * count to a `0` that reads as a fault.
+ */
+function tallyToolsByServer(tools, declared) {
+    if (!Array.isArray(tools))
+        return undefined;
+    const counts = new Map();
+    const ambiguous = new Set();
+    for (const tool of tools) {
+        if (typeof tool !== 'string' || !tool.startsWith('mcp__'))
+            continue;
+        const owners = [];
+        for (const id of declared) {
+            const prefix = `mcp__${id}__`;
+            // The tool name after the prefix must be non-empty: `mcp__slack__` names no tool.
+            if (tool.length > prefix.length && tool.startsWith(prefix))
+                owners.push(id);
+        }
+        const [owner] = owners;
+        if (owners.length === 1 && owner)
+            counts.set(owner, (counts.get(owner) ?? 0) + 1);
+        else
+            for (const id of owners)
+                ambiguous.add(id);
+    }
+    return { counts, ambiguous };
+}
 /**
  * The credential values carried by a run's tool servers, for {@link registerKnownSecrets}. An MCP
  * server that fails to start routinely echoes its own argv or request headers into stderr, and

package/dist/agent-runner.d.ts

@@ -1,6 +1,7 @@
-import type { Logger } from './logger.js';
+import { type Logger } from './logger.js';
+import { type ToolProgressWindow } from './tool-silence.js';
 import { type HarnessCallMetric, type PiRunOutcome, type TodoProgress, type ToolSpan } from './pi.js';
-import { type McpServerSpec, type SkillSpec } from './agent-capabilities.js';
+import { type McpServerSpec, type ObservedMcpServer, type SkillSpec } from './agent-capabilities.js';
 import { type ProgressGuardLimits } from './progress-guard.js';
 import { type SliceReview } from './subagents.js';
 /** Which subscription harness to run (the Pi harness uses `runPi` directly). */
@@ -79,6 +80,18 @@ export interface SubscriptionRunOptions {
      * DID dies with the container.
      */
     onSpan?: (span: ToolSpan) => void;
+    /**
+     * Opens this stream's tool-silence window (see `RunOptions.beginToolWindow`), closed when the
+     * CLI exits. Both subscription CLIs report tool activity — claude-code on the `tool_result`
+     * turn that answers each call, codex on its tool/command/exec events — so a window either
+     * opens is one the run can beat. It is deliberately NOT tied to {@link onSpan}: the trajectory
+     * is an observability opt-in, and the codex stream produces none at all while still doing tool
+     * work, which a span-keyed window would have read as a run making no progress.
+     *
+     * A caller with no tool loop (the inline one-shot completion) passes nothing; see the note at
+     * `handleInline`.
+     */
+    beginToolWindow?: () => ToolProgressWindow;
     /**
      * Called with the FULL set of per-slice reviews each time one lands, so the backend can persist
      * a parallel review's completed work as it happens instead of only from the terminal result.
@@ -92,6 +105,21 @@ export interface SubscriptionRunOptions {
      * same row still rides the result, so a lost poll response costs nothing.
      */
     onCallMetric?: (call: HarnessCallMetric) => void;
+    /**
+     * Called once with what the CLI reported about the tool servers it loaded, the moment it
+     * announces its resolved session (see {@link observeClaudeMcpInit}).
+     *
+     * The one thing the backend's own dispatch record cannot answer: it knows why it WITHHELD a
+     * tool, and this says a server it wired failed to start anyway. Reported even when every server
+     * came up, because "observed, all healthy" and "this image observed nothing" are different
+     * facts about a run and only the first one clears a wired server of suspicion.
+     *
+     * Whole-value latest-wins, not a delta — the CLI announces its session once, so a second call
+     * would only ever be a re-announcement of the same set. A harness whose CLI reports nothing
+     * (codex today) never calls this, which is what leaves the backend's record honestly empty
+     * rather than claiming every server failed.
+     */
+    onToolServers?: (observed: ObservedMcpServer[]) => void;
     /**
      * The per-job child logger (jobId/repo/branch correlation). Threaded so the retained
      * session-transcript path is logged for the run when the isolated config home is torn down.