@cat-factory/executor-harness 1.110.0 → 1.112.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +48 -0
- package/dist/agent-capabilities.js +6 -1
- package/dist/agent-runner.d.ts +15 -0
- package/dist/agent-runner.js +14 -51
- package/dist/agent-shared.d.ts +1 -0
- package/dist/agent-shared.js +1 -0
- package/dist/agent.js +6 -17
- package/dist/artifact-upload.d.ts +38 -0
- package/dist/artifact-upload.js +39 -0
- package/dist/codex-home.d.ts +79 -0
- package/dist/codex-home.js +109 -0
- package/dist/codex-images.d.ts +52 -0
- package/dist/codex-images.js +157 -0
- package/dist/job.d.ts +23 -0
- package/dist/job.js +5 -1
- package/dist/json-reply.d.ts +12 -0
- package/dist/json-reply.js +114 -0
- package/dist/pi-workspace.d.ts +9 -0
- package/dist/pi-workspace.js +5 -0
- package/dist/pi.d.ts +6 -0
- package/dist/pi.js +13 -2
- package/package.json +4 -4
- package/src/agent-capabilities.ts +6 -1
- package/src/agent-runner.ts +30 -54
- package/src/agent-shared.ts +2 -0
- package/src/agent.ts +6 -17
- package/src/artifact-upload.ts +71 -0
- package/src/codex-home.ts +187 -0
- package/src/codex-images.ts +167 -0
- package/src/job.ts +30 -0
- package/src/json-reply.ts +112 -0
- package/src/pi-workspace.ts +14 -0
- package/src/pi.ts +12 -2
package/README.md
CHANGED
|
@@ -148,6 +148,51 @@ workspace once per repair round: a non-empty file already on disk is counted and
|
|
|
148
148
|
and only a view that MISSED is retried. The per-image ceiling is enforced against the declared
|
|
149
149
|
length and against the stream as it arrives, so an oversized body is refused rather than buffered.
|
|
150
150
|
|
|
151
|
+
### Uploading what a job PRODUCES
|
|
152
|
+
|
|
153
|
+
The return leg of the same seam. A job body for a kind the backend gave a browser image to carries
|
|
154
|
+
`artifactUpload: { url, token }`, which the harness surfaces to the agent as `ARTIFACT_UPLOAD_URL`
|
|
155
|
+
/ `ARTIFACT_UPLOAD_TOKEN` — the variables the capturing prompt already names. The token is the run's
|
|
156
|
+
EXISTING container session token, so this grants no reach the job did not already have, and it is
|
|
157
|
+
registered for redaction before it can reach a log.
|
|
158
|
+
|
|
159
|
+
Which kinds get it is the BACKEND's decision (it keys off the kind's declared `ui` image), so the
|
|
160
|
+
harness passes it through for every mode rather than testing the agent kind: a container-side kind
|
|
161
|
+
list would be the same decision made twice, in the half that cannot see the registry. An unusable
|
|
162
|
+
spec drops the WHOLE seam — a URL with no token is an endpoint nothing can call — and the prompt
|
|
163
|
+
branches on the variable being unset, which is what makes an absent capability visible as manual
|
|
164
|
+
mode rather than as an upload that 401s.
|
|
165
|
+
|
|
166
|
+
### Generating binaries with the CLI's own tool
|
|
167
|
+
|
|
168
|
+
Codex ships an `image_gen` tool that works only on ChatGPT subscription auth (an `OPENAI_API_KEY`
|
|
169
|
+
session is routed elsewhere and never offered it). A job body carrying `generateImages: true`
|
|
170
|
+
enables it in the per-run `CODEX_HOME/config.toml` and redirects what it writes.
|
|
171
|
+
|
|
172
|
+
The redirect is the point. Codex writes to `$CODEX_HOME/generated_images/` and tells the model no
|
|
173
|
+
path for it, and `$CODEX_HOME` is where the run's decrypted subscription credential lives — so
|
|
174
|
+
neither "ask the agent where it saved the file" nor "send the agent to look there" is available.
|
|
175
|
+
Instead `generated_images` is created as a symlink into `.cat-context/binary-output/generated/`
|
|
176
|
+
before the CLI starts, so the file is where the agent was told to look the moment the tool returns,
|
|
177
|
+
with no polling and no race, and `$CODEX_HOME` stays unread. A post-run sweep moves anything a
|
|
178
|
+
failed redirect left behind and NAMES it, because an image that arrived too late to be stored is a
|
|
179
|
+
different fact from a run that generated none.
|
|
180
|
+
|
|
181
|
+
Opt-in per job because the tool bills the leased ChatGPT plan at several times an ordinary turn.
|
|
182
|
+
|
|
183
|
+
Unavailable under `ambientAuth`: there is no per-run home to configure or redirect, and
|
|
184
|
+
reconfiguring the developer's own `~/.codex` is the HOME-global mutation this harness never makes.
|
|
185
|
+
Unavailable is not silent — the backend has already composed a brief naming the staging directory,
|
|
186
|
+
so `createCodexHome` reports the gap and one sentence is folded into the prompt saying the tool
|
|
187
|
+
could not be enabled and nothing will appear there. A refused redirect gets its own wording (the
|
|
188
|
+
tool IS on, its output is only unreachable until the post-run sweep), and the teardown report reads
|
|
189
|
+
the same outcome, so a rescued file is never reported as a late arrival when the redirect never
|
|
190
|
+
existed at all.
|
|
191
|
+
|
|
192
|
+
`generateImages` is also a `/health` capability, so a runner pool on an image that predates it is
|
|
193
|
+
refused rather than run blind: the brief names the staging directory whatever the image does with
|
|
194
|
+
the flag.
|
|
195
|
+
|
|
151
196
|
### Skills and tool servers
|
|
152
197
|
|
|
153
198
|
A job body may carry `skills[]` (procedural playbooks) and `mcpServers[]` (MCP tool servers): the
|
|
@@ -211,6 +256,7 @@ under a per-job directory:
|
|
|
211
256
|
| Tester secrets | child env | child env (same path: the old `process.env` set/restore is gone) |
|
|
212
257
|
| Private-registry auth | `~/.npmrc`; cleared when a job has no entries | per-job `.npmrc` + `npm_config_userconfig`, seeded from the developer's; theirs is never written or removed |
|
|
213
258
|
| Repo-sourced Claude Skill | installed into the isolated `CLAUDE_CONFIG_DIR` | not installed: read from the checkout's `.cat-context/skill/`, like codex |
|
|
259
|
+
| Codex image output | redirected out of the per-run `CODEX_HOME` into the checkout | not redirected: no per-run home exists, so the capability is reported unavailable rather than pointed at the developer's own `~/.codex` |
|
|
214
260
|
|
|
215
261
|
Two consequences worth knowing:
|
|
216
262
|
|
|
@@ -267,6 +313,8 @@ Kimi / DeepSeek) and meters spend. The provider key never enters the container.
|
|
|
267
313
|
| `src/design-images.ts` | The task's DESIGN PICTURES: downloads the manifest a building job body carries into `.cat-context/design-renders/`, for an agent CLI that can read an image into its turn. Says NOTHING on success (the backend's prompt already names every file and its view) and speaks only to correct that list when a picture is not here, because an agent told to open a file that is absent goes looking for the design rather than for the transfer. |
|
|
268
314
|
| `src/reference-screenshots.ts` | The task's REFERENCE DESIGN images: downloads the manifest a capturing job body carries into `.cat-context/reference-screenshots/` (on the run's own container session token) and composes the prompt block naming each file's view. Best-effort, time-bounded and IDEMPOTENT over the checkout, so a repair round re-costs a stat rather than a transfer. A reference that is not on disk is NAMED to the agent, whether a transfer failed or the backend's cap dropped the view, because on disk an absent file and a screen the design does not have are the same thing. Backend-authored throughout, including the file names. |
|
|
269
315
|
| `src/bootstrap-mode.ts` | The repo-bootstrap MODE: clone-a-reference-or-scaffold → run the agent → refuse to push an empty tree → reinit + force-push to the pre-created target repo. |
|
|
316
|
+
| `src/artifact-upload.ts` | The OUTBOUND half of the artifact seam: parses the body's `artifactUpload` and projects it onto the agent's env as `ARTIFACT_UPLOAD_URL` / `ARTIFACT_UPLOAD_TOKEN`, registering the token for redaction first. Passes through what the body carries and decides nothing: which kinds get the seam is the backend's call. |
|
|
317
|
+
| `src/codex-images.ts` | Codex's own `image_gen` output, staged where the agent can reach it: creates `$CODEX_HOME/generated_images` as a symlink into `.cat-context/binary-output/generated/` before the CLI starts, sweeps anything a failed redirect left behind, and unlinks (never follows) the redirect at teardown — a failed unlink is REPORTED, because that unlink is what stops the recursive delete reaching the checkout. Exists because codex exposes no path for what it generated AND `$CODEX_HOME` holds the run's decrypted credential, so neither asking the agent nor sending it there is available. |
|
|
270
318
|
| `src/agent-shared.ts` | The few helpers every agent MODE shares (effort-report folding, the capability fields forwarded to `runAgentInWorkspace`). |
|
|
271
319
|
| `src/logger.ts` | Structured logging. |
|
|
272
320
|
|
|
@@ -241,7 +241,12 @@ export function parseSkillSpecs(value) {
|
|
|
241
241
|
* A member is added here in the SAME change that teaches the parser the field, never ahead of it:
|
|
242
242
|
* the whole value of the list is that it is the image's own honest answer.
|
|
243
243
|
*/
|
|
244
|
-
export const HARNESS_BODY_CAPABILITIES = [
|
|
244
|
+
export const HARNESS_BODY_CAPABILITIES = [
|
|
245
|
+
'mcpServers',
|
|
246
|
+
'skills',
|
|
247
|
+
'designImages',
|
|
248
|
+
'generateImages',
|
|
249
|
+
];
|
|
245
250
|
/**
|
|
246
251
|
* A safe MCP server id: it becomes a tool-name fragment AND a TOML table key.
|
|
247
252
|
*
|
package/dist/agent-runner.d.ts
CHANGED
|
@@ -48,6 +48,21 @@ export interface SubscriptionRunOptions {
|
|
|
48
48
|
* carries this job's credentials. Absent ⇒ the CLI's built-in tools only.
|
|
49
49
|
*/
|
|
50
50
|
mcpServers?: McpServerSpec[];
|
|
51
|
+
/**
|
|
52
|
+
* CODEX ONLY: enable the CLI's built-in `image_gen` tool for this job, and redirect what it
|
|
53
|
+
* writes into the checkout (see `codex-images.ts`).
|
|
54
|
+
*
|
|
55
|
+
* Opt-in per job rather than a property of the image, because the tool bills against the leased
|
|
56
|
+
* ChatGPT plan at 3-5x an ordinary turn: every non-generating run would pay for a capability it
|
|
57
|
+
* was never asked for. Set when the dispatch resolved a HARNESS-transport binary generator whose
|
|
58
|
+
* `harness` is `codex`, which is the one signal that says this step exists to make pictures.
|
|
59
|
+
*
|
|
60
|
+
* A no-op under `ambientAuth`: there is no per-run `CODEX_HOME` to write a config into or
|
|
61
|
+
* redirect, and the alternative — reconfiguring the developer's own `~/.codex` and staging into
|
|
62
|
+
* their real output directory — is the HOME-global mutation this harness never makes. The
|
|
63
|
+
* backend states the capability as unavailable there rather than half-enabling it.
|
|
64
|
+
*/
|
|
65
|
+
generateImages?: boolean;
|
|
51
66
|
/**
|
|
52
67
|
* Extra environment for the CLI child, scoped to this job (the tester's secrets, a
|
|
53
68
|
* private-registry npmrc pointer). Merged over the inherited `process.env` at spawn, so the
|
package/dist/agent-runner.js
CHANGED
|
@@ -8,7 +8,8 @@ import { ToolCallTracker, recordClaudeToolResults, } from './tool-trajectory.js'
|
|
|
8
8
|
import { log } from './logger.js';
|
|
9
9
|
import { NO_TOOL_WINDOW } from './tool-silence.js';
|
|
10
10
|
import { createCallMetricPublisher, publishCallMetric, } from './pi.js';
|
|
11
|
-
import { claudeAllowedToolPatterns,
|
|
11
|
+
import { claudeAllowedToolPatterns, mcpServerSecretValues, observeClaudeMcpInit, writeClaudeMcpConfig, } from './agent-capabilities.js';
|
|
12
|
+
import { codexImageGapNote, createCodexHome, disposeCodexHome } from './codex-home.js';
|
|
12
13
|
import { ProgressGuard } from './progress-guard.js';
|
|
13
14
|
import { BoundedTail, JsonlLineReader } from './jsonl-stream.js';
|
|
14
15
|
import { killChildProcess, spawnDetached } from './process.js';
|
|
@@ -892,48 +893,19 @@ export async function runCodex(opts) {
|
|
|
892
893
|
// weight — while the fallback call metric below needs the split, so both are derived from
|
|
893
894
|
// this one value rather than one being reconstructed from the other.
|
|
894
895
|
let cumulative;
|
|
895
|
-
//
|
|
896
|
-
//
|
|
897
|
-
//
|
|
898
|
-
|
|
899
|
-
// subscription `auth.json` (access + refresh tokens) to the PR branch. An
|
|
900
|
-
// isolated, per-run temp dir keeps the credential out of the working tree and is
|
|
901
|
-
// removed in `finally`.
|
|
902
|
-
//
|
|
903
|
-
// KNOWN LIMITATION: Codex refreshes its OAuth access token in-place by rewriting
|
|
904
|
-
// this `auth.json` mid-run. Because the home is a per-run temp dir wiped in
|
|
905
|
-
// `finally`, that refreshed credential is discarded and never written back to the
|
|
906
|
-
// pool — there is no write-back path. The stored bundle keeps working as long as
|
|
907
|
-
// its refresh token stays valid (ChatGPT refresh tokens are long-lived and reused,
|
|
908
|
-
// not rotated per refresh today), so each run re-refreshes from the same stored
|
|
909
|
-
// copy; if OpenAI ever rotates refresh tokens on use, a pooled Codex token would
|
|
910
|
-
// eventually need to be re-connected by the user. Claude OAuth tokens (from
|
|
911
|
-
// `claude setup-token`) are long-lived and unaffected.
|
|
912
|
-
// Native (ambient) mode: run the developer's installed `codex` with its OWN login —
|
|
913
|
-
// no isolated CODEX_HOME, no injected auth.json. Otherwise write the leased credential
|
|
914
|
-
// to a per-run temp home kept OUTSIDE the checkout (and removed in `finally`).
|
|
915
|
-
if (!opts.ambientAuth && !opts.subscriptionToken) {
|
|
916
|
-
throw new Error('codex harness requires a subscription token (or ambientAuth)');
|
|
917
|
-
}
|
|
918
|
-
const codexHome = opts.ambientAuth ? undefined : await mkdtemp(join(tmpdir(), 'cf-codex-'));
|
|
919
|
-
if (codexHome) {
|
|
920
|
-
await writeFile(join(codexHome, 'auth.json'), opts.subscriptionToken, { mode: 0o600 });
|
|
921
|
-
// Tool servers (MCP) ride the SAME per-run config.toml, so they are scoped to this job and
|
|
922
|
-
// torn down with the home. Under AMBIENT auth there is no per-run home — and writing servers
|
|
923
|
-
// into the developer's own `~/.codex/config.toml` would outlive the run and race a concurrent
|
|
924
|
-
// job — so an ambient codex run gets no MCP servers; the backend states them as unavailable
|
|
925
|
-
// the same way it does for a harness with no MCP client at all.
|
|
926
|
-
// Registered before the CLI starts, for the same reason the claude path does it: a server that
|
|
927
|
-
// fails to launch puts its own command line into the stderr tail we keep.
|
|
928
|
-
if (opts.mcpServers?.length)
|
|
929
|
-
registerKnownSecrets(mcpServerSecretValues(opts.mcpServers));
|
|
930
|
-
const mcpToml = opts.mcpServers?.length ? codexMcpConfigToml(opts.mcpServers) : '';
|
|
931
|
-
await writeFile(join(codexHome, 'config.toml'), `cli_auth_credentials_store = "file"\n${mcpToml ? `\n${mcpToml}` : ''}`, { encoding: 'utf8', mode: 0o600 });
|
|
932
|
-
}
|
|
896
|
+
// The per-run `CODEX_HOME` — the credential, the config and the generated-output redirect — is
|
|
897
|
+
// a lifecycle of its own, in `codex-home.ts`. Ambient mode answers no home: the developer's
|
|
898
|
+
// own CLI login, with nothing written and nothing to tear down.
|
|
899
|
+
const { home: codexHome, images } = await createCodexHome(opts);
|
|
933
900
|
// Codex has no system-prompt flag, so fold the composed role + best-practice
|
|
934
901
|
// context into the prompt itself (Claude Code instead rides --append-system-prompt,
|
|
935
902
|
// falling back to this same fold when the prompt overflows argv).
|
|
936
|
-
|
|
903
|
+
//
|
|
904
|
+
// An image capability that could NOT be honoured is stated in the same fold, because the
|
|
905
|
+
// backend's brief has already promised it and only this half knows it is missing. Absent for
|
|
906
|
+
// every ordinary run, which is byte-for-byte the prompt it composed before.
|
|
907
|
+
const gap = codexImageGapNote(images);
|
|
908
|
+
const prompt = foldSystemPrompt(opts.systemPrompt, gap ? `${opts.userPrompt}\n\n${gap}` : opts.userPrompt);
|
|
937
909
|
// This stream's tool-silence window (see the claude runner for the shape); opened just before
|
|
938
910
|
// the CLI starts and closed in the `finally` below.
|
|
939
911
|
let toolWindow = NO_TOOL_WINDOW;
|
|
@@ -1041,17 +1013,8 @@ export async function runCodex(opts) {
|
|
|
1041
1013
|
}
|
|
1042
1014
|
finally {
|
|
1043
1015
|
toolWindow.close();
|
|
1044
|
-
if (codexHome)
|
|
1045
|
-
|
|
1046
|
-
// home is deleted — the credential (`auth.json`) lives at the home root, never in
|
|
1047
|
-
// `sessions/`, so this keeps the debugging artifact without leaking it. Best-effort.
|
|
1048
|
-
await retainSessionTranscripts(codexHome, ['sessions'], {
|
|
1049
|
-
label: 'codex',
|
|
1050
|
-
...(opts.log ? { log: opts.log } : {}),
|
|
1051
|
-
});
|
|
1052
|
-
// Never leave the decrypted credential on disk past the run.
|
|
1053
|
-
await rm(codexHome, { recursive: true, force: true }).catch(() => { });
|
|
1054
|
-
}
|
|
1016
|
+
if (codexHome)
|
|
1017
|
+
await disposeCodexHome(codexHome, opts, images);
|
|
1055
1018
|
}
|
|
1056
1019
|
}
|
|
1057
1020
|
/**
|
package/dist/agent-shared.d.ts
CHANGED
|
@@ -16,6 +16,7 @@ export declare function mergeEffort(result: AgentResult, effortReport: EffortRep
|
|
|
16
16
|
export declare function agentCapabilities(job: AgentJob): {
|
|
17
17
|
skills?: SkillSpec[];
|
|
18
18
|
mcpServers?: McpServerSpec[];
|
|
19
|
+
generateImages?: boolean;
|
|
19
20
|
referenceScreenshots?: ImageManifestSpec;
|
|
20
21
|
designImages?: ImageManifestSpec;
|
|
21
22
|
};
|
package/dist/agent-shared.js
CHANGED
|
@@ -20,6 +20,7 @@ export function agentCapabilities(job) {
|
|
|
20
20
|
return {
|
|
21
21
|
...(job.skills?.length ? { skills: job.skills } : {}),
|
|
22
22
|
...(job.mcpServers?.length ? { mcpServers: job.mcpServers } : {}),
|
|
23
|
+
...(job.generateImages ? { generateImages: true } : {}),
|
|
23
24
|
...(job.referenceScreenshots ? { referenceScreenshots: job.referenceScreenshots } : {}),
|
|
24
25
|
...(job.designImages ? { designImages: job.designImages } : {}),
|
|
25
26
|
};
|
package/dist/agent.js
CHANGED
|
@@ -4,6 +4,7 @@ import { mkdir, mkdtemp, rm } from 'node:fs/promises';
|
|
|
4
4
|
import { execFile } from 'node:child_process';
|
|
5
5
|
import { promisify } from 'node:util';
|
|
6
6
|
import { standUpFrontend, tearDownFrontend } from './frontend-infra.js';
|
|
7
|
+
import { artifactUploadEnv } from './artifact-upload.js';
|
|
7
8
|
import { configurePackageRegistries } from './package-registries.js';
|
|
8
9
|
import { captureRedactedOutput, redactSecrets, registerKnownSecrets } from './redact.js';
|
|
9
10
|
import { cloneRepo, commitAll, conflictDiff, fetchPullRequestHead, fetchReferenceBranches, headCommit, mergeBranch, prepareExistingCheckout, pushBranch, unmergedPaths, } from './git.js';
|
|
@@ -18,6 +19,7 @@ import { agentCapabilities, mergeEffort } from './agent-shared.js';
|
|
|
18
19
|
import { runBootstrap } from './bootstrap-mode.js';
|
|
19
20
|
import { acquireRepoCheckout, agentNeverActed, agentOutputTail, NEVER_ACTED_CAUSE, runAgentInWorkspace, unusableFinalAnswerCause, withWorkspace, } from './pi-workspace.js';
|
|
20
21
|
import { diagnosticsSuffix, resolveStructuredOutput, } from './structured-output.js';
|
|
22
|
+
import { extractJsonObject } from './json-reply.js';
|
|
21
23
|
import { log } from './logger.js';
|
|
22
24
|
// The single generic agent handler — the manifest-driven replacement for the bespoke
|
|
23
25
|
// per-kind handlers. It runs an LLM over an optional checkout and returns text/JSON
|
|
@@ -194,23 +196,6 @@ async function resolveReplyCustom(job, summary, signal) {
|
|
|
194
196
|
});
|
|
195
197
|
return { value: resolved.value, diagnostics: resolved.diagnostics };
|
|
196
198
|
}
|
|
197
|
-
/** Extract the first JSON object from an agent's final message (tolerating fences/prose). */
|
|
198
|
-
function extractJsonObject(text) {
|
|
199
|
-
const trimmed = text.trim();
|
|
200
|
-
const fenced = /^```(?:json)?\s*([\s\S]*?)\s*```$/i.exec(trimmed);
|
|
201
|
-
const body = fenced ? (fenced[1] ?? '') : trimmed;
|
|
202
|
-
try {
|
|
203
|
-
return JSON.parse(body);
|
|
204
|
-
}
|
|
205
|
-
catch {
|
|
206
|
-
const start = body.indexOf('{');
|
|
207
|
-
const end = body.lastIndexOf('}');
|
|
208
|
-
if (start === -1 || end === -1 || end <= start) {
|
|
209
|
-
throw new Error('agent did not return a JSON object');
|
|
210
|
-
}
|
|
211
|
-
return JSON.parse(body.slice(start, end + 1));
|
|
212
|
-
}
|
|
213
|
-
}
|
|
214
199
|
/**
|
|
215
200
|
* The service work directory for a checkout at `dir`: the monorepo service subtree
|
|
216
201
|
* (`repo.serviceDirectory`, created if missing) when the job is service-scoped, else the clone
|
|
@@ -256,9 +241,13 @@ export async function handleAgent(job, opts = {}) {
|
|
|
256
241
|
// not the other would be an integration that works or 401s depending on how its step was
|
|
257
242
|
// registered. Per-job env like everything else here — never `process.env`, which the shared
|
|
258
243
|
// native host process makes a cross-job leak.
|
|
244
|
+
// The platform's own artifact ingest, layered on for EVERY mode for the same reason: which
|
|
245
|
+
// kinds get the seam is the backend's call (it keys off the kind's declared `ui` image), so a
|
|
246
|
+
// mode check here would be that decision made twice, in the half that cannot see the registry.
|
|
259
247
|
const scoped = withAgentEnv(opts, {
|
|
260
248
|
...registryEnv,
|
|
261
249
|
...secretEnv(job.generatorSecrets),
|
|
250
|
+
...artifactUploadEnv(job.artifactUpload),
|
|
262
251
|
});
|
|
263
252
|
if (job.mode === 'preview')
|
|
264
253
|
return await runPreviewMode(job, scoped);
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Where this job uploads the artifacts it produces, and the credential to do it with.
|
|
3
|
+
*
|
|
4
|
+
* The token is the run's EXISTING container session token, not a second credential: the ingest
|
|
5
|
+
* route authenticates it the same way the LLM proxy does and scopes the stored bytes to that
|
|
6
|
+
* token's workspace + execution. So a body carrying this grants no reach the job did not already
|
|
7
|
+
* have, which is why it needs no allow-list of its own beyond the transport check below.
|
|
8
|
+
*/
|
|
9
|
+
export interface ArtifactUploadSpec {
|
|
10
|
+
/** Absolute http(s) URL of the ingest endpoint. */
|
|
11
|
+
url: string;
|
|
12
|
+
/** Bearer credential — the run's container session token. */
|
|
13
|
+
token: string;
|
|
14
|
+
}
|
|
15
|
+
/** The env var naming the ingest endpoint, as the producing prompts already reference it. */
|
|
16
|
+
export declare const ARTIFACT_UPLOAD_URL_ENV = "ARTIFACT_UPLOAD_URL";
|
|
17
|
+
/** The env var carrying the ingest credential. */
|
|
18
|
+
export declare const ARTIFACT_UPLOAD_TOKEN_ENV = "ARTIFACT_UPLOAD_TOKEN";
|
|
19
|
+
/**
|
|
20
|
+
* Parse the job body's upload seam, or undefined when absent/unusable.
|
|
21
|
+
*
|
|
22
|
+
* The whole spec is dropped when either half is unusable, exactly as `parseImageManifest` drops a
|
|
23
|
+
* manifest whose transport half is: a URL with no token and a token with no URL are both
|
|
24
|
+
* an endpoint nothing can call, and the agent is told the capability is absent rather than handed
|
|
25
|
+
* half of it. Absent is the NORMAL case — only a kind the backend gave a browser image to ever
|
|
26
|
+
* receives one.
|
|
27
|
+
*/
|
|
28
|
+
export declare function parseArtifactUpload(value: unknown): ArtifactUploadSpec | undefined;
|
|
29
|
+
/**
|
|
30
|
+
* Project the seam into the agent's child env, registering the credential for redaction first.
|
|
31
|
+
*
|
|
32
|
+
* Returns the env rather than writing `process.env`, for the reason every other per-job value here
|
|
33
|
+
* does: the native host transport serves every concurrent ambient job from ONE process, so a
|
|
34
|
+
* global would hand one job's ingest credential to a sibling. Absent spec ⇒ `{}`, which is what
|
|
35
|
+
* makes the capability's absence visible to the agent as an unset variable (the prompts that use
|
|
36
|
+
* it already branch on that) rather than as an endpoint that 401s.
|
|
37
|
+
*/
|
|
38
|
+
export declare function artifactUploadEnv(spec: ArtifactUploadSpec | undefined): Record<string, string>;
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
import { registerKnownSecrets } from './redact.js';
|
|
2
|
+
/** The env var naming the ingest endpoint, as the producing prompts already reference it. */
|
|
3
|
+
export const ARTIFACT_UPLOAD_URL_ENV = 'ARTIFACT_UPLOAD_URL';
|
|
4
|
+
/** The env var carrying the ingest credential. */
|
|
5
|
+
export const ARTIFACT_UPLOAD_TOKEN_ENV = 'ARTIFACT_UPLOAD_TOKEN';
|
|
6
|
+
/**
|
|
7
|
+
* Parse the job body's upload seam, or undefined when absent/unusable.
|
|
8
|
+
*
|
|
9
|
+
* The whole spec is dropped when either half is unusable, exactly as `parseImageManifest` drops a
|
|
10
|
+
* manifest whose transport half is: a URL with no token and a token with no URL are both
|
|
11
|
+
* an endpoint nothing can call, and the agent is told the capability is absent rather than handed
|
|
12
|
+
* half of it. Absent is the NORMAL case — only a kind the backend gave a browser image to ever
|
|
13
|
+
* receives one.
|
|
14
|
+
*/
|
|
15
|
+
export function parseArtifactUpload(value) {
|
|
16
|
+
if (typeof value !== 'object' || value === null)
|
|
17
|
+
return undefined;
|
|
18
|
+
const o = value;
|
|
19
|
+
const url = typeof o.url === 'string' ? o.url.trim() : '';
|
|
20
|
+
const token = typeof o.token === 'string' ? o.token : '';
|
|
21
|
+
if (!url || !token || !/^https?:\/\//i.test(url))
|
|
22
|
+
return undefined;
|
|
23
|
+
return { url, token };
|
|
24
|
+
}
|
|
25
|
+
/**
|
|
26
|
+
* Project the seam into the agent's child env, registering the credential for redaction first.
|
|
27
|
+
*
|
|
28
|
+
* Returns the env rather than writing `process.env`, for the reason every other per-job value here
|
|
29
|
+
* does: the native host transport serves every concurrent ambient job from ONE process, so a
|
|
30
|
+
* global would hand one job's ingest credential to a sibling. Absent spec ⇒ `{}`, which is what
|
|
31
|
+
* makes the capability's absence visible to the agent as an unset variable (the prompts that use
|
|
32
|
+
* it already branch on that) rather than as an endpoint that 401s.
|
|
33
|
+
*/
|
|
34
|
+
export function artifactUploadEnv(spec) {
|
|
35
|
+
if (!spec)
|
|
36
|
+
return {};
|
|
37
|
+
registerKnownSecrets([spec.token]);
|
|
38
|
+
return { [ARTIFACT_UPLOAD_URL_ENV]: spec.url, [ARTIFACT_UPLOAD_TOKEN_ENV]: spec.token };
|
|
39
|
+
}
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
import { type McpServerSpec } from './agent-capabilities.js';
|
|
2
|
+
import type { Logger } from './logger.js';
|
|
3
|
+
/** What one codex job needs written into its own home. */
|
|
4
|
+
export interface CodexHomeOptions {
|
|
5
|
+
/** The decrypted `auth.json` bundle. Required unless `ambientAuth`. */
|
|
6
|
+
subscriptionToken?: string;
|
|
7
|
+
/** Run the developer's own CLI login instead: no isolated home, nothing written. */
|
|
8
|
+
ambientAuth?: boolean;
|
|
9
|
+
/** Tool servers to scope to this job's config. */
|
|
10
|
+
mcpServers?: McpServerSpec[];
|
|
11
|
+
/** Enable the CLI's built-in image tool and redirect its output into the checkout. */
|
|
12
|
+
generateImages?: boolean;
|
|
13
|
+
/** The checkout, which is where generated output is staged to. */
|
|
14
|
+
cwd: string;
|
|
15
|
+
log?: Logger;
|
|
16
|
+
}
|
|
17
|
+
/**
|
|
18
|
+
* What became of the image capability a job asked for, so the run can SAY so.
|
|
19
|
+
*
|
|
20
|
+
* Its own value rather than a boolean, because the two failures need different words and the
|
|
21
|
+
* teardown report needs to tell them apart: an image found in the home afterwards is a LATE
|
|
22
|
+
* arrival when the redirect was live, and a file that was never reachable at all when it was not.
|
|
23
|
+
*/
|
|
24
|
+
export type CodexImageOutcome =
|
|
25
|
+
/** The redirect is in place: the agent reads what the tool writes, the moment it writes it. */
|
|
26
|
+
{
|
|
27
|
+
state: 'staged';
|
|
28
|
+
}
|
|
29
|
+
/** Enabled nowhere: an ambient run has no per-run home to configure or redirect. */
|
|
30
|
+
| {
|
|
31
|
+
state: 'unavailable';
|
|
32
|
+
reason: 'ambient-home';
|
|
33
|
+
}
|
|
34
|
+
/** The tool is on and its output goes somewhere the agent cannot reach during the run. */
|
|
35
|
+
| {
|
|
36
|
+
state: 'unavailable';
|
|
37
|
+
reason: 'redirect-refused';
|
|
38
|
+
};
|
|
39
|
+
/** This job's `CODEX_HOME` (absent for an ambient run) and what its image capability came to. */
|
|
40
|
+
export interface CodexHomeSetup {
|
|
41
|
+
home?: string;
|
|
42
|
+
/** Absent when the job asked for no image generation. */
|
|
43
|
+
images?: CodexImageOutcome;
|
|
44
|
+
}
|
|
45
|
+
/**
|
|
46
|
+
* Create and populate this job's `CODEX_HOME`, or answer no home for an ambient run.
|
|
47
|
+
*
|
|
48
|
+
* Ambient mode writes nothing deliberately: there is no per-run home, so there is nowhere to put
|
|
49
|
+
* MCP servers (writing them into the developer's own `~/.codex/config.toml` would outlive the run
|
|
50
|
+
* and race a concurrent job) and nowhere to redirect generated images.
|
|
51
|
+
*
|
|
52
|
+
* What it does NOT do is drop the image capability quietly. The backend composed a brief naming
|
|
53
|
+
* the staging directory and told the agent to collect from it, so an ambient run with the tool
|
|
54
|
+
* silently off leaves the agent hunting for files nothing wrote and reporting a vendor problem
|
|
55
|
+
* for a configuration one. The outcome comes back so the caller can state it, which is what the
|
|
56
|
+
* brief's own "if the tool is unavailable, say so" instruction exists to be paired with.
|
|
57
|
+
*/
|
|
58
|
+
export declare function createCodexHome(opts: CodexHomeOptions): Promise<CodexHomeSetup>;
|
|
59
|
+
/**
|
|
60
|
+
* What to TELL THE AGENT when the image capability it was briefed on is not there, or undefined
|
|
61
|
+
* when there is nothing to say.
|
|
62
|
+
*
|
|
63
|
+
* Appended to the user prompt rather than left to the backend, because only this half knows: the
|
|
64
|
+
* backend resolved a harness-served generator and composed a brief naming the staging directory,
|
|
65
|
+
* and whether that directory can be written to is decided here, one process later. Silence is the
|
|
66
|
+
* one answer that is never right — it reads to the agent exactly like a working tool that returned
|
|
67
|
+
* nothing.
|
|
68
|
+
*/
|
|
69
|
+
export declare function codexImageGapNote(images: CodexImageOutcome | undefined): string | undefined;
|
|
70
|
+
/**
|
|
71
|
+
* Tear the home down: rescue anything generated, keep the transcripts, delete the credential.
|
|
72
|
+
*
|
|
73
|
+
* ORDER is load-bearing. The image sweep runs first because the files are about to be deleted with
|
|
74
|
+
* the home; the redirect is unlinked next so the recursive delete cannot follow it into the
|
|
75
|
+
* checkout; the transcripts are lifted before the delete (the credential lives at the home ROOT,
|
|
76
|
+
* never in `sessions/`, which is what makes that safe); and the delete is last, because nothing
|
|
77
|
+
* else may leave a decrypted credential on disk past the run.
|
|
78
|
+
*/
|
|
79
|
+
export declare function disposeCodexHome(codexHome: string, opts: CodexHomeOptions, images?: CodexImageOutcome): Promise<void>;
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
|
|
2
|
+
import { tmpdir } from 'node:os';
|
|
3
|
+
import { join } from 'node:path';
|
|
4
|
+
import { codexMcpConfigToml, mcpServerSecretValues, } from './agent-capabilities.js';
|
|
5
|
+
import { GENERATED_BINARY_DIR, stageCodexImages, sweepCodexImages, unstageCodexImages, } from './codex-images.js';
|
|
6
|
+
import { registerKnownSecrets } from './redact.js';
|
|
7
|
+
import { retainSessionTranscripts } from './transcript-retention.js';
|
|
8
|
+
/**
|
|
9
|
+
* Create and populate this job's `CODEX_HOME`, or answer no home for an ambient run.
|
|
10
|
+
*
|
|
11
|
+
* Ambient mode writes nothing deliberately: there is no per-run home, so there is nowhere to put
|
|
12
|
+
* MCP servers (writing them into the developer's own `~/.codex/config.toml` would outlive the run
|
|
13
|
+
* and race a concurrent job) and nowhere to redirect generated images.
|
|
14
|
+
*
|
|
15
|
+
* What it does NOT do is drop the image capability quietly. The backend composed a brief naming
|
|
16
|
+
* the staging directory and told the agent to collect from it, so an ambient run with the tool
|
|
17
|
+
* silently off leaves the agent hunting for files nothing wrote and reporting a vendor problem
|
|
18
|
+
* for a configuration one. The outcome comes back so the caller can state it, which is what the
|
|
19
|
+
* brief's own "if the tool is unavailable, say so" instruction exists to be paired with.
|
|
20
|
+
*/
|
|
21
|
+
export async function createCodexHome(opts) {
|
|
22
|
+
if (!opts.ambientAuth && !opts.subscriptionToken) {
|
|
23
|
+
throw new Error('codex harness requires a subscription token (or ambientAuth)');
|
|
24
|
+
}
|
|
25
|
+
if (opts.ambientAuth) {
|
|
26
|
+
return opts.generateImages ? { images: { state: 'unavailable', reason: 'ambient-home' } } : {};
|
|
27
|
+
}
|
|
28
|
+
const codexHome = await mkdtemp(join(tmpdir(), 'cf-codex-'));
|
|
29
|
+
await writeFile(join(codexHome, 'auth.json'), opts.subscriptionToken, { mode: 0o600 });
|
|
30
|
+
// Registered before the CLI starts, for the same reason the claude path does it: a server that
|
|
31
|
+
// fails to launch puts its own command line into the stderr tail we keep.
|
|
32
|
+
if (opts.mcpServers?.length)
|
|
33
|
+
registerKnownSecrets(mcpServerSecretValues(opts.mcpServers));
|
|
34
|
+
const mcpToml = opts.mcpServers?.length ? codexMcpConfigToml(opts.mcpServers) : '';
|
|
35
|
+
// `image_generation` is OPT-IN per job rather than always-on: the tool bills the leased ChatGPT
|
|
36
|
+
// plan at 3-5x an ordinary turn, so every non-generating run would pay for a capability it was
|
|
37
|
+
// never asked for. Enabled only when the dispatch selected a harness-served generator, which is
|
|
38
|
+
// the one thing that knows the step exists to make pictures.
|
|
39
|
+
const imagesToml = opts.generateImages ? '\n[features]\nimage_generation = true\n' : '';
|
|
40
|
+
await writeFile(join(codexHome, 'config.toml'), `cli_auth_credentials_store = "file"\n${mcpToml ? `\n${mcpToml}` : ''}${imagesToml}`, { encoding: 'utf8', mode: 0o600 });
|
|
41
|
+
if (!opts.generateImages)
|
|
42
|
+
return { home: codexHome };
|
|
43
|
+
// Redirect the tool's output into the checkout BEFORE the CLI starts, so the agent never has to
|
|
44
|
+
// read this directory (which holds the decrypted credential) to find what it generated. The
|
|
45
|
+
// answer is KEPT rather than discarded: a refused redirect leaves the tool enabled and its
|
|
46
|
+
// output unreachable until the post-run sweep, which is a different fact from a live one and
|
|
47
|
+
// the difference the teardown report would otherwise get wrong.
|
|
48
|
+
const staged = await stageCodexImages(codexHome, opts.cwd, opts.log);
|
|
49
|
+
return {
|
|
50
|
+
home: codexHome,
|
|
51
|
+
images: staged ? { state: 'staged' } : { state: 'unavailable', reason: 'redirect-refused' },
|
|
52
|
+
};
|
|
53
|
+
}
|
|
54
|
+
/**
|
|
55
|
+
* What to TELL THE AGENT when the image capability it was briefed on is not there, or undefined
|
|
56
|
+
* when there is nothing to say.
|
|
57
|
+
*
|
|
58
|
+
* Appended to the user prompt rather than left to the backend, because only this half knows: the
|
|
59
|
+
* backend resolved a harness-served generator and composed a brief naming the staging directory,
|
|
60
|
+
* and whether that directory can be written to is decided here, one process later. Silence is the
|
|
61
|
+
* one answer that is never right — it reads to the agent exactly like a working tool that returned
|
|
62
|
+
* nothing.
|
|
63
|
+
*/
|
|
64
|
+
export function codexImageGapNote(images) {
|
|
65
|
+
if (!images || images.state === 'staged')
|
|
66
|
+
return undefined;
|
|
67
|
+
const shared = `Nothing will appear in \`${GENERATED_BINARY_DIR}/\` while you are working. Report the ` +
|
|
68
|
+
`artifacts you could not produce, exactly as your instructions for an unavailable generation ` +
|
|
69
|
+
`tool describe, and do not substitute another generator or describe an image you did not make.`;
|
|
70
|
+
return images.reason === 'ambient-home'
|
|
71
|
+
? `NOTE: this run's built-in image generation tool could NOT be enabled. It needs an isolated ` +
|
|
72
|
+
`per-run CLI home, and this run uses the host's own CLI login, which is not reconfigured ` +
|
|
73
|
+
`for a job. ${shared}`
|
|
74
|
+
: `NOTE: this run's built-in image generation tool is enabled, but its output could NOT be ` +
|
|
75
|
+
`redirected into the checkout, so anything it writes lands somewhere you cannot read. ` +
|
|
76
|
+
`${shared}`;
|
|
77
|
+
}
|
|
78
|
+
/**
|
|
79
|
+
* Tear the home down: rescue anything generated, keep the transcripts, delete the credential.
|
|
80
|
+
*
|
|
81
|
+
* ORDER is load-bearing. The image sweep runs first because the files are about to be deleted with
|
|
82
|
+
* the home; the redirect is unlinked next so the recursive delete cannot follow it into the
|
|
83
|
+
* checkout; the transcripts are lifted before the delete (the credential lives at the home ROOT,
|
|
84
|
+
* never in `sessions/`, which is what makes that safe); and the delete is last, because nothing
|
|
85
|
+
* else may leave a decrypted credential on disk past the run.
|
|
86
|
+
*/
|
|
87
|
+
export async function disposeCodexHome(codexHome, opts, images) {
|
|
88
|
+
if (opts.generateImages) {
|
|
89
|
+
const stranded = await sweepCodexImages(codexHome, opts.cwd, opts.log);
|
|
90
|
+
if (stranded.length > 0) {
|
|
91
|
+
// REPORTED rather than quietly rescued: an image that arrived too late for the agent to
|
|
92
|
+
// store is a different fact from a run that generated none.
|
|
93
|
+
//
|
|
94
|
+
// And WHICH fact depends on what the setup came to, which is why the outcome is threaded in
|
|
95
|
+
// rather than inferred here. With a live redirect these really are late arrivals (the CLI
|
|
96
|
+
// wrote after the agent's last turn); with a refused one they were never reachable at all,
|
|
97
|
+
// and calling those "late" points the next reader at the model instead of at the filesystem.
|
|
98
|
+
opts.log?.warn(images?.state === 'unavailable'
|
|
99
|
+
? 'generated images were rescued after the run: the output redirect was never in place'
|
|
100
|
+
: 'generated images were staged after the agent finished', { count: stranded.length, dir: GENERATED_BINARY_DIR });
|
|
101
|
+
}
|
|
102
|
+
await unstageCodexImages(codexHome, opts.log);
|
|
103
|
+
}
|
|
104
|
+
await retainSessionTranscripts(codexHome, ['sessions'], {
|
|
105
|
+
label: 'codex',
|
|
106
|
+
...(opts.log ? { log: opts.log } : {}),
|
|
107
|
+
});
|
|
108
|
+
await rm(codexHome, { recursive: true, force: true }).catch(() => { });
|
|
109
|
+
}
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
import type { Logger } from './logger.js';
|
|
2
|
+
/**
|
|
3
|
+
* Where a harness-generated binary artifact is staged for the agent, relative to the checkout.
|
|
4
|
+
*
|
|
5
|
+
* Under {@link CONTEXT_DIR} so it inherits that directory's git exclude: an image the agent has
|
|
6
|
+
* not uploaded yet must never be swept into a commit by the `git add -A` a coding run ends with.
|
|
7
|
+
* Part of the backend↔harness path contract (`HARNESS_SENTINEL_PATHS.generatedBinaries`), because
|
|
8
|
+
* the agent's brief has to NAME this path and the two halves are written independently.
|
|
9
|
+
*/
|
|
10
|
+
export declare const GENERATED_BINARY_SUBDIR = "binary-output/generated";
|
|
11
|
+
/** The staging directory's repo-relative path, as the prompt names it. */
|
|
12
|
+
export declare const GENERATED_BINARY_DIR = ".cat-context/binary-output/generated";
|
|
13
|
+
/**
|
|
14
|
+
* Point codex's image output at the checkout, before the CLI starts.
|
|
15
|
+
*
|
|
16
|
+
* Returns whether the redirect is in place. FALSE is a real and reportable answer rather than a
|
|
17
|
+
* throw: a run whose images cannot be staged is still a run worth doing (the agent may have plenty
|
|
18
|
+
* of non-generating work), and the caller states the gap instead of failing the job. The sweep
|
|
19
|
+
* below is what keeps that case from losing files outright.
|
|
20
|
+
*
|
|
21
|
+
* Best-effort by construction, and deliberately NOT idempotent-by-overwrite: an existing
|
|
22
|
+
* `generated_images` is left exactly as it is. On the per-run home this is always a fresh
|
|
23
|
+
* directory, so anything already there on the ambient path is the DEVELOPER's own history, and
|
|
24
|
+
* replacing it with a symlink into a throwaway checkout would destroy it.
|
|
25
|
+
*/
|
|
26
|
+
export declare function stageCodexImages(codexHome: string, cwd: string, log?: Logger): Promise<boolean>;
|
|
27
|
+
/**
|
|
28
|
+
* Move anything codex wrote into a REAL `generated_images` directory across to the staging path.
|
|
29
|
+
*
|
|
30
|
+
* The backstop for a redirect that did not take. Returns the file names moved, so the caller can
|
|
31
|
+
* say what arrived after the agent had already finished — those are images the run generated and
|
|
32
|
+
* the agent never had a chance to upload, which is a different fact from generating none, and the
|
|
33
|
+
* kind of distinction this codebase refuses to let collapse into silence.
|
|
34
|
+
*
|
|
35
|
+
* A live redirect yields nothing here, detected by `lstat` on the directory itself: reading THROUGH
|
|
36
|
+
* the link would list files that are already where they belong and report every one of them as
|
|
37
|
+
* stranded.
|
|
38
|
+
*/
|
|
39
|
+
export declare function sweepCodexImages(codexHome: string, cwd: string, log?: Logger): Promise<string[]>;
|
|
40
|
+
/**
|
|
41
|
+
* Remove the redirect before the per-run home is torn down.
|
|
42
|
+
*
|
|
43
|
+
* Only ever the LINK: `rm` on a symlink unlinks it and leaves the target alone, which is what must
|
|
44
|
+
* happen here — the target is inside the checkout and holds the run's actual output. Passed the
|
|
45
|
+
* home rather than the link path so a caller cannot accidentally hand it the staging directory.
|
|
46
|
+
*
|
|
47
|
+
* A failure is SWALLOWED (teardown must not take a completed run down with it) and REPORTED,
|
|
48
|
+
* because this unlink is the property that stops the recursive delete that follows from reaching
|
|
49
|
+
* the checkout. Dropping it silently would forfeit that with no line anywhere saying so, and the
|
|
50
|
+
* evidence would be missing artifacts nobody could trace back to this call.
|
|
51
|
+
*/
|
|
52
|
+
export declare function unstageCodexImages(codexHome: string, log?: Logger): Promise<void>;
|