agency-lang 0.6.3 → 0.6.5

package/dist/lib/agents/docs/appendix/appendix/callbacks.md

@@ -98,6 +98,21 @@ Called after an LLM call completes. You can return a `MessageJSON[]` array to ov
 - `timeTaken`: how long the call took in milliseconds
 - `messages`: the messages that were sent
 
+### onLLMRetry
+Called just before the backend waits to retry an LLM call after a transient failure (see [retries and timeouts](../guide/llm.md#resilience-retries-and-timeouts)). Side-effect only — it cannot change whether the retry happens.
+
+- `attempt`: the 1-based retry number (retry 1, 2, …)
+- `maxRetries`: the configured retry count
+- `delayMs`: how long the backend will wait before this retry
+- `reason`: why we're retrying — `"timeout"`, `"connectionLost"`, `"streamInterrupted"`, `"rateLimit"`, `"serverError"`, or `"overloaded"`
+- `detail`: the raw provider message
+
+### onLLMTimeout
+Called whenever an LLM call exceeds its per-call deadline (`timeout`), whether or not a retry follows.
+
+- `limitMs`: the deadline that was exceeded
+- `attempt`: the 0-based attempt that timed out
+
 ### onFunctionStart
 Called when a function (tool) begins executing.
 

package/dist/lib/agents/docs/appendix/callbacks.md

@@ -98,6 +98,21 @@ Called after an LLM call completes. You can return a `MessageJSON[]` array to ov
 - `timeTaken`: how long the call took in milliseconds
 - `messages`: the messages that were sent
 
+### onLLMRetry
+Called just before the backend waits to retry an LLM call after a transient failure (see [retries and timeouts](../guide/llm.md#resilience-retries-and-timeouts)). Side-effect only — it cannot change whether the retry happens.
+
+- `attempt`: the 1-based retry number (retry 1, 2, …)
+- `maxRetries`: the configured retry count
+- `delayMs`: how long the backend will wait before this retry
+- `reason`: why we're retrying — `"timeout"`, `"connectionLost"`, `"streamInterrupted"`, `"rateLimit"`, `"serverError"`, or `"overloaded"`
+- `detail`: the raw provider message
+
+### onLLMTimeout
+Called whenever an LLM call exceeds its per-call deadline (`timeout`), whether or not a retry follows.
+
+- `limitMs`: the deadline that was exceeded
+- `attempt`: the 0-based attempt that timed out
+
 ### onFunctionStart
 Called when a function (tool) begins executing.
 

package/dist/lib/agents/docs/guide/guide/llm.md

@@ -177,4 +177,24 @@ def myTool() {
 
 It does **not** work at module top level, inside a `callback(...)` registration block, or inside the `onAgentStart` lifecycle hook — those scopes run before any agent has started and have no conversation to append to. If you call `llm` from one of them you'll get a runtime error like *"Message threads are not available in this scope."* Move the call inside a `node` or `def` body to fix it.
 
-See [Message history and threads](./message-history-and-threads.md) for the full picture.
+See [Message history and threads](./message-history-and-threads.md) for the full picture.
+## Resilience: retries and timeouts
+
+Transient LLM failures — a dropped connection, a `429` rate limit, a `5xx`, or a call that simply hangs — are common and usually self-heal. `llm()` retries them automatically with exponential backoff and an optional per-call deadline, so your happy path stays clean and you don't have to check every call.
+
+```ts
+llm("Summarize this", {
+  retries: 2,                                  // max retry attempts (default 2; 0 disables)
+  timeout: 30s,                                // per-attempt deadline (default 10min; 0 disables)
+  backoff: { initial: 500ms, factor: 2, max: 10s },   // exponential, capped (these are the defaults)
+})
+```
+
+- **What's retried:** connection drops (ECONNRESET, fetch failed, …), `5xx`, `429` (honoring the server's `retry-after`), and `529` overloaded. A `429`'s `retry-after` overrides the computed backoff.
+- **What's not:** `400`/auth/content-policy/context-window errors (terminal — no point retrying), and user cancels / guard trips (those propagate immediately and are never swallowed by the retry loop). A guard's time budget keeps ticking through backoff, so a `guard(time:)` still wins.
+- **After retries exhaust** (or with `retries: 0`), the call surfaces a normal `Failure` you can handle with `try` / `isFailure` — it does not abort the run.
+- **Cancellation:** pressing Esc during a backoff wait cancels the whole loop.
+
+Set defaults for a whole branch with `setLlmOptions({ retries, timeout, backoff })` (per-call options still win). Classification is provider-neutral — it reads HTTP status from the LLM client adapter, so a custom (non-smoltalk) client works too, falling back to message matching.
+
+To be notified of retries/timeouts (for a status line, logging, etc.), use the [`onLLMRetry` and `onLLMTimeout` callbacks](../appendix/callbacks.md).

package/dist/lib/agents/docs/guide/llm.md

@@ -177,4 +177,24 @@ def myTool() {
 
 It does **not** work at module top level, inside a `callback(...)` registration block, or inside the `onAgentStart` lifecycle hook — those scopes run before any agent has started and have no conversation to append to. If you call `llm` from one of them you'll get a runtime error like *"Message threads are not available in this scope."* Move the call inside a `node` or `def` body to fix it.
 
-See [Message history and threads](./message-history-and-threads.md) for the full picture.
+See [Message history and threads](./message-history-and-threads.md) for the full picture.
+## Resilience: retries and timeouts
+
+Transient LLM failures — a dropped connection, a `429` rate limit, a `5xx`, or a call that simply hangs — are common and usually self-heal. `llm()` retries them automatically with exponential backoff and an optional per-call deadline, so your happy path stays clean and you don't have to check every call.
+
+```ts
+llm("Summarize this", {
+  retries: 2,                                  // max retry attempts (default 2; 0 disables)
+  timeout: 30s,                                // per-attempt deadline (default 10min; 0 disables)
+  backoff: { initial: 500ms, factor: 2, max: 10s },   // exponential, capped (these are the defaults)
+})
+```
+
+- **What's retried:** connection drops (ECONNRESET, fetch failed, …), `5xx`, `429` (honoring the server's `retry-after`), and `529` overloaded. A `429`'s `retry-after` overrides the computed backoff.
+- **What's not:** `400`/auth/content-policy/context-window errors (terminal — no point retrying), and user cancels / guard trips (those propagate immediately and are never swallowed by the retry loop). A guard's time budget keeps ticking through backoff, so a `guard(time:)` still wins.
+- **After retries exhaust** (or with `retries: 0`), the call surfaces a normal `Failure` you can handle with `try` / `isFailure` — it does not abort the run.
+- **Cancellation:** pressing Esc during a backoff wait cancels the whole loop.
+
+Set defaults for a whole branch with `setLlmOptions({ retries, timeout, backoff })` (per-call options still win). Classification is provider-neutral — it reads HTTP status from the LLM client adapter, so a custom (non-smoltalk) client works too, falling back to message matching.
+
+To be notified of retries/timeouts (for a status line, logging, etc.), use the [`onLLMRetry` and `onLLMTimeout` callbacks](../appendix/callbacks.md).

package/dist/lib/cli/eval/run.d.ts

@@ -20,10 +20,27 @@ export type EvalRunLoadedInputsOptions = {
     continueOnError?: boolean;
     verbose?: boolean;
     config?: AgencyConfig;
-    /** Suppress compile progress lines for the agent compile. */
+    /** Suppress compile progress lines for the agent compile.
+     *  (Currently unused — kept for API compatibility.) */
     quietCompile?: boolean;
     /** Pipe agent subprocess stdout/stderr through to the console. Defaults to true. */
     pipeAgentOutput?: boolean;
+    /**
+     * Workdir seed override. When set, used verbatim for every input — no
+     * closure-base derivation, no relpath computation. The optimizer always
+     * sets this from `source.baseDir` + `source.entryFile`, so the seed
+     * cannot silently diverge from the closure walk.
+     *
+     * Mutually exclusive with per-input `working_dir`: setting both is an
+     * error.
+     */
+    seed?: {
+        dir: string;
+        agentRelPath: string;
+    };
+    /** Candidate-file overlay applied to every input in this call (over the
+     *  seeded copy, before compile). Plain eval never sets this. */
+    overlayFiles?: Record<string, string>;
 };
 export declare function resolveEvalRunTarget(target: string): {
     agentFile: string;

package/dist/lib/cli/eval/run.js

@@ -2,13 +2,12 @@ import * as fs from "fs";
 import * as path from "path";
 import { fork } from "child_process";
 import { nanoid } from "nanoid";
-import { compile } from "../../cli/commands.js";
 import { parseTarget } from "../../cli/util.js";
-import { RunStrategy } from "../../importStrategy.js";
 import { StatelogParser } from "../../eval/statelogParser.js";
 import { loadInputs, inputFromGoal } from "../../eval/loadInputs.js";
 import { initializeEvalRun, writeEvalRunSummary, } from "../../eval/runArtifacts.js";
 import { runEvalInput, } from "../../eval/runEvalInput.js";
+import { agentClosureBaseDir } from "../../optimize/targets.js";
 import { buildForkOptions, buildRunInstruction, subprocessBootstrapPath, } from "../../runtime/ipc.js";
 /**
  * Per-task resource limits for subprocess invocations driven by `agency eval
@@ -66,14 +65,13 @@ export async function evalRun(opts, overrides = {}) {
 }
 export async function evalRunLoadedInputs(opts, overrides = {}) {
     const target = resolveEvalRunTarget(opts.agent);
+    const absoluteAgent = fs.realpathSync(target.agentFile);
     const runsDir = path.resolve(opts.runsDir ?? opts.config?.eval?.runsDir ?? "runs");
     const runId = opts.runId ?? nanoid();
     const continueOnError = opts.continueOnError ?? true;
-    const compiled = await compileAgentForEvalRun({
-        config: opts.config ?? {},
-        agentFile: target.agentFile,
-        quiet: opts.quietCompile ?? false,
-    });
+    const config = opts.config ?? {};
+    // One closure walk per call when no caller-supplied seed; never per input.
+    const defaultSeed = opts.seed ?? deriveSeedFromAgent(target.agentFile, absoluteAgent);
     const state = initializeEvalRun({
         runId,
         runsDir,
@@ -87,10 +85,33 @@ export async function evalRunLoadedInputs(opts, overrides = {}) {
     const extractor = overrides.extractor ?? defaultEvalRecordExtractor;
     const results = [];
     for (const input of opts.inputs) {
+        let seed;
+        try {
+            seed = resolveInputSeed(input, defaultSeed, absoluteAgent, opts.seed !== undefined);
+        }
+        catch (err) {
+            // Seed-resolution errors (working_dir misuse, seed/working_dir conflict)
+            // are per-input prepare failures — never escape `evalRunLoadedInputs`.
+            const message = err instanceof Error ? err.message : String(err);
+            console.error(`[evalRun] seed resolution failed for input ${input.id ?? "(no id)"}: ${message}`);
+            results.push({
+                inputId: input.id ?? "",
+                status: "error",
+                evalRecordPath: "",
+                statelogPath: "",
+                workdirPath: "",
+                errorMessage: message,
+            });
+            if (!continueOnError)
+                break;
+            continue;
+        }
         const result = await runEvalInput({
             state,
             input,
-            compiled,
+            seed,
+            overlayFiles: opts.overlayFiles,
+            config,
             defaultNode: target.node,
             runner,
             extractor,
@@ -101,18 +122,31 @@ export async function evalRunLoadedInputs(opts, overrides = {}) {
     }
     return writeEvalRunSummary(state, results);
 }
-async function compileAgentForEvalRun(args) {
-    const compiledPath = compile(args.config, args.agentFile, undefined, {
-        importStrategy: new RunStrategy(),
-        quiet: args.quiet,
-    });
-    if (compiledPath === null) {
-        throw new Error(`Failed to compile ${args.agentFile}`);
+/** Derive seed from agent file via closure walk. Called at most once per
+ *  `evalRunLoadedInputs` invocation, never per input. */
+function deriveSeedFromAgent(agentFile, absoluteAgent) {
+    const dir = agentClosureBaseDir(agentFile);
+    return { dir, agentRelPath: path.relative(dir, absoluteAgent) };
+}
+/** Apply the per-input seed rule:
+ *  - `opts.seed` (callerSetSeed=true) + `input.working_dir` → conflict, throw.
+ *  - `input.working_dir` only → validate (must be a directory; must contain the agent).
+ *  - neither → use the default seed. */
+function resolveInputSeed(input, defaultSeed, absoluteAgent, callerSetSeed) {
+    if (!input.working_dir)
+        return defaultSeed;
+    if (callerSetSeed) {
+        throw new Error(`input.working_dir cannot be combined with a caller-supplied seed (input id=${input.id ?? "(no id)"})`);
     }
-    return {
-        moduleId: path.basename(compiledPath, ".js"),
-        path: compiledPath,
-    };
+    const resolved = fs.realpathSync(path.resolve(input.working_dir));
+    if (!fs.statSync(resolved).isDirectory()) {
+        throw new Error(`Eval input working_dir is not a directory: ${input.working_dir}`);
+    }
+    const rel = path.relative(resolved, absoluteAgent);
+    if (rel.startsWith("..") || path.isAbsolute(rel)) {
+        throw new Error(`working_dir must contain the agent file (working_dir=${resolved}, agent=${absoluteAgent})`);
+    }
+    return { dir: resolved, agentRelPath: rel };
 }
 const defaultEvalRecordExtractor = async ({ statelogPath, outPath, }) => {
     const record = new StatelogParser(statelogPath).evalRecord();
@@ -132,12 +166,9 @@ export const optimizeEvalRecordExtractor = async ({ statelogPath, outPath, }) =>
     fs.writeFileSync(outPath, JSON.stringify(record, null, 2));
 };
 function makeSubprocessEvalInputRunner(pipeAgentOutput) {
-    return async ({ compiled, node, args, cwd, statelogPath }) => {
-        if (!compiled.path) {
-            return { ok: false, errorMessage: "Compiled agent has no path" };
-        }
+    return async ({ compiledEntryPath, node, args, cwd, statelogPath }) => {
         return runCompiledAgentInSubprocess({
-            compiledPath: compiled.path,
+            compiledPath: compiledEntryPath,
             node,
             args,
             cwd,

package/dist/lib/cli/eval/run.workdir.test.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/lib/cli/eval/run.workdir.test.js

@@ -0,0 +1,111 @@
+import * as fs from "fs";
+import * as os from "os";
+import * as path from "path";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { evalRunLoadedInputs } from "./run.js";
+describe("evalRunLoadedInputs compiles + runs inside each input workdir", () => {
+    let proj;
+    beforeEach(() => {
+        proj = fs.mkdtempSync(path.join(os.tmpdir(), "evalwd-"));
+        fs.writeFileSync(path.join(proj, "agent.agency"), "node main() { return 1 }\n");
+    });
+    afterEach(() => {
+        fs.rmSync(proj, { recursive: true, force: true });
+    });
+    it("module-dir == cwd: compiled entry lives inside each input's workdir", async () => {
+        const runsDir = path.join(proj, "runs");
+        const seen = [];
+        const runner = vi.fn(async (args) => {
+            seen.push({ compiledEntryPath: args.compiledEntryPath, cwd: args.cwd });
+            return { ok: true };
+        });
+        const result = await evalRunLoadedInputs({
+            agent: path.join(proj, "agent.agency"),
+            inputs: [{ id: "input-1", goal: "g", args: {} }],
+            inputsSource: "test",
+            runsDir,
+            runId: "r1",
+            config: {},
+            pipeAgentOutput: false,
+        }, { runner });
+        const workdir = result.inputs[0].workdirPath;
+        expect(seen).toHaveLength(1);
+        expect(seen[0].cwd).toBe(workdir);
+        expect(seen[0].compiledEntryPath.startsWith(workdir + path.sep)).toBe(true);
+        expect(fs.existsSync(seen[0].compiledEntryPath)).toBe(true);
+    });
+    it("rejects working_dir that does not contain the agent file", async () => {
+        const outside = fs.mkdtempSync(path.join(os.tmpdir(), "outside-"));
+        const runner = vi.fn(async () => ({ ok: true }));
+        try {
+            const result = await evalRunLoadedInputs({
+                agent: path.join(proj, "agent.agency"),
+                inputs: [{ id: "input-1", goal: "g", args: {}, working_dir: outside }],
+                inputsSource: "test",
+                runsDir: path.join(proj, "runs"),
+                runId: "r-outside",
+                config: {},
+                pipeAgentOutput: false,
+            }, { runner });
+            expect(result.inputs[0].status).toBe("error");
+            expect(result.inputs[0].errorMessage).toMatch(/working_dir must contain the agent file/);
+            expect(runner).not.toHaveBeenCalled();
+        }
+        finally {
+            fs.rmSync(outside, { recursive: true, force: true });
+        }
+    });
+    it("rejects working_dir values that point to files", async () => {
+        const file = path.join(proj, "not-a-dir.txt");
+        fs.writeFileSync(file, "x");
+        const runner = vi.fn(async () => ({ ok: true }));
+        const result = await evalRunLoadedInputs({
+            agent: path.join(proj, "agent.agency"),
+            inputs: [{ id: "input-1", goal: "g", args: {}, working_dir: file }],
+            inputsSource: "test",
+            runsDir: path.join(proj, "runs"),
+            runId: "r-file",
+            config: {},
+            pipeAgentOutput: false,
+        }, { runner });
+        expect(result.inputs[0].status).toBe("error");
+        expect(result.inputs[0].errorMessage).toMatch(/working_dir is not a directory/);
+    });
+    it("overlayFiles overwrite the seed copy inside each input's workdir", async () => {
+        fs.writeFileSync(path.join(proj, "config.txt"), "original\n");
+        const observedCwd = [];
+        const runner = vi.fn(async (args) => {
+            observedCwd.push(fs.readFileSync(path.join(args.cwd, "config.txt"), "utf8"));
+            return { ok: true };
+        });
+        await evalRunLoadedInputs({
+            agent: path.join(proj, "agent.agency"),
+            inputs: [{ id: "input-1", goal: "g", args: {} }],
+            inputsSource: "test",
+            runsDir: path.join(proj, "runs"),
+            runId: "r-overlay",
+            config: {},
+            pipeAgentOutput: false,
+            seed: { dir: proj, agentRelPath: "agent.agency" },
+            overlayFiles: { "config.txt": "patched\n" },
+        }, { runner });
+        // The overlay wins inside the workdir; the source tree is untouched.
+        expect(observedCwd).toEqual(["patched\n"]);
+        expect(fs.readFileSync(path.join(proj, "config.txt"), "utf8")).toBe("original\n");
+    });
+    it("rejects combining a caller-supplied seed with input.working_dir", async () => {
+        const runner = vi.fn(async () => ({ ok: true }));
+        const result = await evalRunLoadedInputs({
+            agent: path.join(proj, "agent.agency"),
+            inputs: [{ id: "input-1", goal: "g", args: {}, working_dir: proj }],
+            inputsSource: "test",
+            runsDir: path.join(proj, "runs"),
+            runId: "r-seed-conflict",
+            config: {},
+            pipeAgentOutput: false,
+            seed: { dir: proj, agentRelPath: "agent.agency" },
+        }, { runner });
+        expect(result.inputs[0].status).toBe("error");
+        expect(result.inputs[0].errorMessage).toMatch(/cannot be combined with a caller-supplied seed/);
+    });
+});

package/dist/lib/eval/runArtifacts.js

@@ -33,16 +33,10 @@ export function prepareInput(state, input) {
     const inputDir = path.join(state.inputsDir, id);
     const workdirPath = path.join(inputDir, "workdir");
     fs.mkdirSync(inputDir, { recursive: true });
-    if (input.working_dir) {
-        const workingDirStat = fs.statSync(input.working_dir);
-        if (!workingDirStat.isDirectory()) {
-            throw new Error("Eval input working_dir must be a directory");
-        }
-        fs.cpSync(input.working_dir, workdirPath, { recursive: true });
-    }
-    else {
-        fs.mkdirSync(workdirPath, { recursive: true });
-    }
+    // workdir is materialized by prepareRunDir (seed + overlay + compile); we
+    // only allocate the path here. working_dir validation moves to
+    // evalRunLoadedInputs, where both working_dir and the agent file are in
+    // scope (it must enforce "working_dir contains the agent file").
     const prepared = {
         input,
         inputDir,

package/dist/lib/eval/runArtifacts.test.js

@@ -46,11 +46,13 @@ Choose a different --run-id or delete the existing directory.`);
         expect(fs.existsSync(path.join(tmpDir, "existing", "inputs"))).toBe(false);
         expect(fs.existsSync(path.join(tmpDir, "existing", "config.json"))).toBe(false);
     });
-    it("prepares per-input artifact paths and an empty workdir", () => {
+    it("prepares per-input artifact paths but leaves workdir to prepareRunDir", () => {
         const state = initializeState();
         const prepared = prepareInput(state, { id: "t1", goal: "goal", args: {} });
         expect(JSON.parse(fs.readFileSync(path.join(state.runDir, "inputs", "t1", "input.json"), "utf-8"))).toMatchObject({ id: "t1", goal: "goal" });
-        expect(fs.existsSync(prepared.workdirPath)).toBe(true);
+        // prepareInput allocates the path but no longer creates the workdir — that's
+        // prepareRunDir's job (seed + overlay + compile inside the workdir).
+        expect(fs.existsSync(prepared.workdirPath)).toBe(false);
         expect(prepared.statelogPath).toBe(path.join(state.runDir, "inputs", "t1", "statelog.jsonl"));
         expect(prepared.evalRecordPath).toBe(path.join(state.runDir, "inputs", "t1", "eval-record.json"));
     });
@@ -69,26 +71,12 @@ Choose a different --run-id or delete the existing directory.`);
         const state = initializeState();
         expect(() => prepareInput(state, { id: "../escape", goal: "goal", args: {} })).toThrow("Invalid id");
     });
-    it("copies a fixture working_dir into the input workdir", () => {
-        const state = initializeState();
-        const fixture = path.join(tmpDir, "fixture");
-        fs.mkdirSync(fixture);
-        fs.writeFileSync(path.join(fixture, "input.txt"), "fixture-data");
-        const prepared = prepareInput(state, { id: "t1", goal: "goal", args: {}, working_dir: fixture });
-        expect(fs.readFileSync(path.join(prepared.workdirPath, "input.txt"), "utf-8")).toBe("fixture-data");
-        expect(fs.readFileSync(path.join(fixture, "input.txt"), "utf-8")).toBe("fixture-data");
-    });
-    it("rejects working_dir values that point to files", () => {
-        const state = initializeState();
-        const fixtureFile = path.join(tmpDir, "fixture.txt");
-        fs.writeFileSync(fixtureFile, "fixture-data");
-        expect(() => prepareInput(state, {
-            id: "t1",
-            goal: "goal",
-            args: {},
-            working_dir: fixtureFile,
-        })).toThrow("working_dir must be a directory");
-    });
+    // Notes on `working_dir` handling: prepareInput is no longer responsible
+    // for materializing the workdir from a `working_dir` fixture or for
+    // validating that the value points to a directory. Both responsibilities
+    // moved to `evalRunLoadedInputs.resolveInputSeed`/`prepareRunDir`, where
+    // the agent file is in scope (enabling the "working_dir must contain the
+    // agent file" check). See `lib/cli/eval/run.workdir.test.ts`.
     it("records prepare failures without touching artifact paths", () => {
         const result = recordInputPrepareFailure("t1", "invalid id");
         expect(result).toEqual({

package/dist/lib/eval/runEvalInput.d.ts

@@ -1,5 +1,6 @@
+import type { AgencyConfig } from "../config.js";
 import { type EvalRunState } from "./runArtifacts.js";
-import type { EvalRunCompiledAgent, Input, EvalRunInputResult } from "./runTypes.js";
+import type { Input, EvalRunInputResult } from "./runTypes.js";
 /**
  * How to actually invoke the compiled agent for an input. The CLI plugs in a
  * subprocess fork; alternative callers (tests, in-process variants) can
@@ -11,7 +12,7 @@ import type { EvalRunCompiledAgent, Input, EvalRunInputResult } from "./runTypes
  * to the runner.
  */
 export type EvalInputRunner = (args: {
-    compiled: EvalRunCompiledAgent;
+    compiledEntryPath: string;
     node: string;
     args: Record<string, any>;
     cwd: string;
@@ -48,7 +49,14 @@ export type EvalRecordExtractor = (args: {
 export declare function runEvalInput(args: {
     state: EvalRunState;
     input: Input;
-    compiled: EvalRunCompiledAgent;
+    /** The workdir spec for this input: seed dir + agent path within it. */
+    seed: {
+        dir: string;
+        agentRelPath: string;
+    };
+    /** Optional candidate-file overlay applied on top of the seed before compile. */
+    overlayFiles?: Record<string, string>;
+    config: AgencyConfig;
     defaultNode: string;
     runner: EvalInputRunner;
     extractor: EvalRecordExtractor;

package/dist/lib/eval/runEvalInput.js

@@ -1,5 +1,6 @@
 import * as fs from "fs";
 import * as path from "path";
+import { prepareRunDir } from "./runWorkdir.js";
 import { prepareInput, recordInputPrepareFailure, recordInputRunFailure, recordInputSuccess, shouldExtractStatelog, } from "./runArtifacts.js";
 /**
  * Single source of truth for "run one eval input end-to-end."
@@ -23,8 +24,22 @@ export async function runEvalInput(args) {
         console.error(`[evalRun] prepare failed for input ${inputId}: ${message}`);
         return recordInputPrepareFailure(inputId, message);
     }
+    let dir;
+    try {
+        const spec = {
+            seedDir: args.seed.dir,
+            agentRelPath: args.seed.agentRelPath,
+            overlayFiles: args.overlayFiles,
+        };
+        dir = prepareRunDir(spec, prepared.workdirPath, args.config);
+    }
+    catch (err) {
+        const message = errMessage(err);
+        console.error(`[evalRun] prepareRunDir failed for input ${inputId}: ${message}`);
+        return recordInputRunFailure(prepared, message);
+    }
     const runResult = await args.runner({
-        compiled: args.compiled,
+        compiledEntryPath: dir.compiledEntryPath,
         node: args.input.node ?? args.defaultNode,
         args: args.input.args,
         cwd: prepared.workdirPath,

package/dist/lib/eval/runTypes.d.ts

@@ -44,7 +44,3 @@ export type EvalRunConfig = {
     continueOnError: boolean;
     verbose?: boolean;
 };
-export type EvalRunCompiledAgent = {
-    moduleId: string;
-    path?: string;
-};

package/dist/lib/eval/runWorkdir.d.ts

@@ -0,0 +1,25 @@
+import type { AgencyConfig } from "../config.js";
+/** Declarative description of a per-input run directory: where the project
+ *  tree is seeded from, where the entry agent lives within it, and any
+ *  candidate-file overlay applied before compile. */
+export type RunWorkdirSpec = {
+    /** Absolute dir whose contents seed the workdir (the agent's project tree). */
+    seedDir: string;
+    /** Entry agent path relative to seedDir, e.g. "examples/08-optimize.agency". */
+    agentRelPath: string;
+    /** Optional overlay applied after seeding — the optimizer's mutated candidate
+     *  files, keyed by path relative to seedDir. Absent for plain eval. */
+    overlayFiles?: Record<string, string>;
+};
+export type PreparedRunDir = {
+    workdirPath: string;
+    /** Absolute path of the compiled entry JS inside the workdir. */
+    compiledEntryPath: string;
+};
+/**
+ * Materialize an isolated run directory: copy the seed project tree into
+ * `workdirPath`, overlay the candidate's mutated files (if any), then compile
+ * the entry agent in place. The result has module-dir == cwd == workdir, so the
+ * agent's reads, writes, and execs all resolve to this one isolated copy.
+ */
+export declare function prepareRunDir(spec: RunWorkdirSpec, workdirPath: string, config: AgencyConfig): PreparedRunDir;

package/dist/lib/eval/runWorkdir.js

@@ -0,0 +1,46 @@
+import * as fs from "fs";
+import * as path from "path";
+import { compile } from "../cli/commands.js";
+import { RunStrategy } from "../importStrategy.js";
+import { copyProjectTree } from "../utils/projectTree.js";
+/** Resolve `rel` against `workdirPath` and refuse paths that escape it (`..`,
+ *  absolute). Overlay keys come from optimizer candidates today but may flow in
+ *  from less-trusted callers later; this is the same guard the prior
+ *  `WorkspaceManager.resolveWithin` provided. */
+function resolveWithin(workdirPath, rel) {
+    const root = path.resolve(workdirPath);
+    const abs = path.resolve(root, rel);
+    if (abs !== root && !abs.startsWith(root + path.sep)) {
+        throw new Error(`Path ${JSON.stringify(rel)} escapes the workdir ${root}`);
+    }
+    return abs;
+}
+/** Write each overlay file (path relative to the workdir root) over the seeded copy. */
+function applyOverlay(workdirPath, overlayFiles) {
+    for (const [rel, source] of Object.entries(overlayFiles)) {
+        const abs = resolveWithin(workdirPath, rel);
+        fs.mkdirSync(path.dirname(abs), { recursive: true });
+        fs.writeFileSync(abs, source);
+    }
+}
+/**
+ * Materialize an isolated run directory: copy the seed project tree into
+ * `workdirPath`, overlay the candidate's mutated files (if any), then compile
+ * the entry agent in place. The result has module-dir == cwd == workdir, so the
+ * agent's reads, writes, and execs all resolve to this one isolated copy.
+ */
+export function prepareRunDir(spec, workdirPath, config) {
+    copyProjectTree(spec.seedDir, workdirPath);
+    if (spec.overlayFiles) {
+        applyOverlay(workdirPath, spec.overlayFiles);
+    }
+    const entryAgency = resolveWithin(workdirPath, spec.agentRelPath);
+    const compiledEntryPath = compile(config, entryAgency, undefined, {
+        importStrategy: new RunStrategy(),
+        quiet: true,
+    });
+    if (compiledEntryPath === null) {
+        throw new Error(`Failed to compile ${entryAgency}`);
+    }
+    return { workdirPath, compiledEntryPath };
+}

package/dist/lib/eval/runWorkdir.test.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/lib/eval/runWorkdir.test.js

@@ -0,0 +1,42 @@
+import * as fs from "fs";
+import * as os from "os";
+import * as path from "path";
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { prepareRunDir } from "./runWorkdir.js";
+describe("prepareRunDir", () => {
+    let root;
+    let seed;
+    beforeEach(() => {
+        root = fs.mkdtempSync(path.join(os.tmpdir(), "prd-"));
+        seed = path.join(root, "proj");
+        fs.mkdirSync(path.join(seed, "demo"), { recursive: true });
+        fs.writeFileSync(path.join(seed, "agent.agency"), "node main() { return 1 }\n");
+        fs.writeFileSync(path.join(seed, "demo", "data.txt"), "original\n");
+    });
+    afterEach(() => {
+        fs.rmSync(root, { recursive: true, force: true });
+    });
+    it("seeds the workdir, applies overlay, and compiles the entry in place", () => {
+        const workdir = path.join(root, "wd");
+        const prepared = prepareRunDir({ seedDir: seed, agentRelPath: "agent.agency", overlayFiles: { "demo/data.txt": "patched\n" } }, workdir, {});
+        // overlay applied on top of the seeded copy
+        expect(fs.readFileSync(path.join(workdir, "demo", "data.txt"), "utf8")).toBe("patched\n");
+        // compiled entry lives inside the workdir, next to its source
+        expect(prepared.compiledEntryPath).toBe(path.join(workdir, "agent.js"));
+        expect(fs.existsSync(prepared.compiledEntryPath)).toBe(true);
+        expect(prepared.workdirPath).toBe(workdir);
+    });
+    it("refuses overlay keys that escape the workdir (traversal / absolute)", () => {
+        const workdir = path.join(root, "wd-escape");
+        const outside = path.join(root, "outside.txt");
+        expect(() => prepareRunDir({ seedDir: seed, agentRelPath: "agent.agency", overlayFiles: { "../outside.txt": "x" } }, workdir, {})).toThrow(/escapes the workdir/);
+        expect(() => prepareRunDir({ seedDir: seed, agentRelPath: "agent.agency", overlayFiles: { [outside]: "x" } }, path.join(root, "wd-escape-abs"), {})).toThrow(/escapes the workdir/);
+        expect(fs.existsSync(outside)).toBe(false);
+    });
+    it("works without an overlay", () => {
+        const workdir = path.join(root, "wd2");
+        const prepared = prepareRunDir({ seedDir: seed, agentRelPath: "agent.agency" }, workdir, {});
+        expect(fs.readFileSync(path.join(workdir, "demo", "data.txt"), "utf8")).toBe("original\n");
+        expect(fs.existsSync(prepared.compiledEntryPath)).toBe(true);
+    });
+});