@tangle-network/agent-bench 0.9.4 → 0.11.0

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.11.0
+
+Adds caller-controlled start and resume inside managed benchmark shots.
+Retains setup, final extraction, grading, and cleanup while recording every prompt and its observed usage.
+The callback composes with outer retry attempts; learning policy and policy inference accounting remain consumer responsibilities.
+
 ## 0.9.4
 
 Requires Runtime 0.202.0 so Bench consumers receive explicit failed-execution settlement and retained failure recovery.

package/HARNESS.md

@@ -8,9 +8,11 @@
 |---|---|
 | **agent-runtime** | exact execution, reusable benchmark adapters, packed-consumer checks, one full-fidelity integration fixture |
 | **discovery** | research questions, preregistrations, acceptance criteria, negative results, and decisions about what is worth testing |
-| **discovery-lab** | paid campaigns, upstream method reproductions, long-horizon hill climbs, immutable receipts, and result archives |
+| **supervisor-lab** | registered adaptive-agent and retained-learning comparisons, including carried versus revised profiles |
+| **discovery-lab** | research campaigns and evidence for Discovery, including immutable inputs, run records, and result archives |
 
-A benchmark implementation may begin here while it is becoming a reusable adapter. Once the question is “does method X improve benchmark Y?”, the campaign belongs in Discovery Lab.
+A benchmark implementation may begin here while it is becoming a reusable adapter. Once the question is “does method X improve benchmark Y?”, the campaign belongs in the consuming lab.
+Supervisor Lab owns adaptive-learning comparisons; Discovery Lab owns research campaigns for Discovery.
 
 ## Evidence levels
 
@@ -20,8 +22,8 @@ Use these labels literally. Do not promote one level into another in prose.
 |---|---|---|
 | **contract proof** | packages install; identities, budgets, callbacks, resume, and receipts have the expected shape | root `pnpm verify:official-optimizers`, `pnpm verify:bench` |
 | **evaluator proof** | the benchmark's own evaluator can distinguish known fail/pass artifacts in the exact environment | adapter preflight and gold/self-check |
-| **reproduction proof** | an upstream method is run at a pinned revision on its claimed benchmark under a matched protocol | Discovery Lab reproduction manifest and runner |
-| **value proof** | the integrated method beats the preregistered baseline on frozen evidence with uncertainty and complete cost accounting | Discovery Lab result receipt |
+| **reproduction proof** | an upstream method is run at a pinned revision on its claimed benchmark under a matched protocol | Consuming lab reproduction manifest and runner |
+| **value proof** | the integrated method beats the preregistered baseline on frozen evidence with uncertainty and complete cost accounting | Consuming lab result receipt |
 | **production proof** | a promoted artifact transfers to real traffic under a canary or controlled rollout | product repository / platform telemetry |
 
 A localization score, output-shape check, LLM quality judge, or toy deterministic reward can be useful for development. None is a substitute for the benchmark's outcome evaluator.
@@ -62,27 +64,63 @@ Use `LOOP_ATTEMPTS=N` only when the benchmark's own visible feedback is allowed
 `runBenchmarks()` returns each judged artifact, worker events, and observed usage in `perTask`.
 Retry usage includes every attempt; missing receipts leave the measured subtotal explicitly incomplete.
 Judge failures retain completed worker evidence.
+Each task separates `execution` from `measurement` availability.
+Captured empty output and explicit failed turns remain measured failures when the evaluator runs successfully.
+Read, extraction, and judge failures leave measurement unavailable while retaining observed usage.
+Missing dispatch evidence remains unknown; `ok: false` never establishes permission to retry.
 Errors propagated by `close()` remain in `detail` beside the settled task outcome.
 The current Runtime lineage suppresses sandbox deletion errors, so a returned result does not confirm resource deletion.
 The caller's abort signal stops queued shots and reaches active sandbox turns.
 `modelApiKey` supplies sandbox inference authorization separately from the `routerKey` used for sandbox control.
 
-### Full-fidelity improvement fixture
+### Caller-controlled prompts within a task
+
+`runBenchmarks({ execute })` invokes the callback inside each managed sandbox shot.
+The context supplies the task prompt, executed profile, benchmark and task identities, attempt number, and abort signal.
+Its managed `run` exposes Runtime's `start`, `resume`, `box`, and `sessionId`.
+Use the live box for permitted working checks in the same session.
+Submit worker prompts through managed `start` and `resume` so Bench captures their outcomes and usage.
+Direct sandbox prompt calls bypass this accounting.
+The callback must leave session lifecycle, extraction, and cleanup to Bench.
+It receives no adapter, final grader, or task metadata containing gold material.
+This callback is trusted consumer code, not an isolation boundary for arbitrary code.
+
+Return after the final prompt completes.
+Bench extracts the last captured prompt's artifact and applies the adapter's final grading outside the callback.
+Start and resume calls must be sequential.
+Bench waits for an unawaited active invocation before extraction and refuses calls after the callback settles.
+The callback must use its signal to cancel external work.
+Bench stops awaiting policy work when cancelled, but cannot stop external effects that ignore cancellation.
+
+Each task's `prompts` retains ordered method, prompt, attempt, session identity, outcome, events, usage, and capture errors.
+Prompt indices start at zero; attempt numbers start at one.
+Usage sums each prompt separately, including failed and interrupted prompts.
+A partial capture retains observed counters and marks accounting incomplete.
+These counters cover sandbox workers only; consumers must account for policy and working-evaluator inference separately.
+Consumers must bind their callback source, configuration, profiles, and initial state to their execution identity.
+
+`execute` runs inside every `loopAttempts` shot.
+Those outer attempts still create fresh sandboxes and use the existing checker feedback policy.
+Use one outer attempt when final grading must remain unavailable to adaptation.
+A custom `runShot` receives `execute` and owns whether it consumes the callback.
+
+A sandbox prompt can contain multiple native model requests.
+Same-session continuation does not prove a barrier before every native request, profile reload, coordinator restart, or fresh-session state transfer.
+Offline fake-sandbox tests prove the managed contract and correction consumption only.
+They establish no live learning gain or provider session restoration.
+
+### Retained strategy driver
 
 ```bash
 cd bench
 pnpm tsx src/swe-self-improve.mts
 ```
 
-This is the canonical real-task Runtime fixture:
-
-- SWE-bench Verified instances;
-- repository state as the produced artifact;
-- the official Docker judge outside the candidate agent;
-- explicit train, selection, and frozen final-test partitions;
-- Runtime's `improve()` boundary and complete cost receipts.
-
-It proves the integrated execution path can support a real value campaign. A paid powered result still belongs in Discovery Lab.
+This driver uses `runStrategyEvolution` with SWE-bench tasks and a frozen holdout.
+It does not exercise `improve`, and it deletes its temporary run directory on exit.
+It therefore cannot provide retained improvement or lineage evidence.
+Use `examples/improve` for the maintained offline API fixture.
+Use the consuming labs for registered learning campaigns with retained execution and comparison evidence.
 
 ### Offline diagnostics
 
@@ -133,4 +171,4 @@ A new file under `bench/src` must be one of:
 - a package-consumer or evaluator calibration test;
 - one canonical full-fidelity fixture that exercises a public Runtime contract.
 
-A one-off campaign, generation-N optimizer script, bespoke dashboard, or historical result belongs in Discovery Lab. If an older file has no package script, no importer, and no unique reusable primitive, delete it rather than adding another index entry.
+A one-off campaign, generation-N optimizer script, bespoke dashboard, or historical result belongs in the consuming lab. If an older file has no package script, no importer, and no unique reusable primitive, delete it rather than adding another index entry.

package/README.md

@@ -100,3 +100,25 @@ Pier owns the task container and verifier; protected model usage and traces stay
 `FilePierCandidateTrialController` atomically reserves a unique Pier job, then persists the supervisor PID, process-session identity, and that job's exact Docker projects so a fresh evaluator process can stop and remove an abandoned trial.
 Run `PIER_REPO=/path/to/pier pnpm verify:pier` for the zero-model failure/pass and fresh-process recovery proof, and see `HARNESS.md` for the exact invocation and failure contract.
 From an installed npm package, expose the shipped Python module with `export PYTHONPATH="$(npm root)/@tangle-network/agent-bench${PYTHONPATH:+:$PYTHONPATH}"` before invoking Pier.
+
+## Control execution within a managed shot
+
+Supply `execute` to control whole sandbox prompts while Bench owns setup, extraction, grading, and cleanup.
+Your policy can inspect permitted working checks through the existing sandbox handle.
+
+```ts
+import { runBenchmarks, type BenchExecution } from '@tangle-network/agent-bench'
+import { executionOptions, chooseNextPrompt } from './policy.js'
+
+const execute: BenchExecution = async ({ run, prompt, signal }) => {
+  const first = await run.start(prompt)
+  const correction = await chooseNextPrompt({ first, box: run.box, sessionId: run.sessionId, signal })
+  if (correction !== undefined) await run.resume(correction)
+}
+
+const report = await runBenchmarks({ ...executionOptions, execute })
+```
+
+The consumer supplies and identifies `chooseNextPrompt`; Bench provides no learning policy.
+`perTask[].prompts` retains each invocation and its observed worker cost, including partial failures.
+See [the execution contract](./HARNESS.md#caller-controlled-prompts-within-a-task) for accounting, retry composition, and lifecycle limits.

package/dist/index.d.ts

@@ -8,10 +8,10 @@ import { FINAL_ANSWER_SENTINEL, RagAnswerScore, RagContext, answerScoreToBenchSc
 import { createRagBenchAdapter } from "./benchmarks/ragbench.js";
 import { SweBenchAdapterOptions, SweBenchArtifactCaptureContext, SweBenchCacheLevel, createSweBenchAdapter, scoreSweReport, sweEvaluationArgv, swePatchOutput } from "./benchmarks/swe-bench.js";
 import { createT2RagBenchAdapter } from "./benchmarks/t2-ragbench.js";
-import { AgentProfile, SandboxClient, sumSandboxUsage } from "@tangle-network/agent-runtime/kernel";
+import { AgentProfile, SandboxClient, SandboxRun, TurnResult, sumSandboxUsage } from "@tangle-network/agent-runtime/kernel";
 import { SandboxEvent } from "@tangle-network/sandbox";
 import { AgentCandidateBenchmarkGraderPort, AgentCandidateExecutionClaimStore, AgentCandidateExecutorPort, AgentCandidateExecutorRequest, AgentCandidateExecutorStopRequest, AgentCandidateOutputArtifactPort, AgentCandidateRunFinalization, PreparedAgentCandidateExecution } from "@tangle-network/agent-runtime/candidate-execution";
-import { TraceStore } from "@tangle-network/agent-eval";
+import { RunTerminalOutcome, TraceStore } from "@tangle-network/agent-eval";
 //#region src/resolve-client.d.ts
 interface ResolveBenchClientOptions {
   /** The selector (the `BACKEND` env value): `router` = off-box; anything else = in-box Sandbox. */
@@ -48,6 +48,31 @@ interface BenchCell {
   /** The agent under test. Defaults to a minimal `{ name, metadata.backendType }` profile. */
   readonly profile?: AgentProfile;
 }
+/** Caller-owned work inside a managed benchmark shot. Final grading stays outside this callback. */
+interface BenchExecutionContext {
+  readonly prompt: string;
+  readonly profile: AgentProfile;
+  readonly benchmark: string;
+  readonly taskId: string;
+  readonly attempt: number;
+  readonly signal: AbortSignal;
+  /** Runtime owns session continuity. Bench owns capture, extraction, and close. */
+  readonly run: Pick<SandboxRun<string>, 'start' | 'resume' | 'box' | 'sessionId'>;
+}
+type BenchExecution = (context: BenchExecutionContext) => Promise<void>;
+/** One submitted sandbox prompt, including partial evidence when its invocation throws. */
+interface BenchPromptResult {
+  readonly attempt: number;
+  readonly index: number;
+  readonly method: 'start' | 'resume';
+  readonly prompt: string;
+  readonly sessionId?: string;
+  readonly outcome?: TurnResult<string>['outcome'];
+  readonly events: readonly SandboxEvent[];
+  readonly usage: ReturnType<typeof sumSandboxUsage>;
+  readonly readError?: string;
+  readonly error?: string;
+}
 /** A worker's artifact and observed execution evidence, before external grading. */
 interface BenchShotResult {
   readonly artifact: string;
@@ -56,6 +81,14 @@ interface BenchShotResult {
   /** Provider observations, including explicit unknown counters. Omitted when the shot reports none. */
   readonly usage?: ReturnType<typeof sumSandboxUsage>;
   readonly events?: readonly SandboxEvent[];
+  readonly prompts?: readonly BenchPromptResult[];
+  /** Observed dispatch and terminal state, independent of artifact quality. */
+  readonly execution?: {
+    readonly phase: 'not-started' | 'started' | 'unknown';
+    readonly terminalOutcome: RunTerminalOutcome;
+  };
+  /** Whether the artifact was captured without a read or extraction failure. */
+  readonly artifactAvailable?: boolean;
 }
 /** Runs one (adapter, task, cell) shot. Defaults to `openSandboxRun`. */
 type BenchShot = (input: {
@@ -66,6 +99,8 @@ type BenchShot = (input: {
   readonly prompt?: string;
   /** 1-based attempt index for looped runs. */
   readonly attempt?: number;
+  /** Custom shots own whether they consume this managed execution callback. */
+  readonly execute?: BenchExecution;
   readonly routerBaseUrl: string;
   readonly routerKey: string;
   /** Optional inference credential for the box; routerKey continues to authorize sandbox control. */
@@ -109,6 +144,8 @@ interface RunBenchmarksOptions {
   /** Self-verify each benchmark's judge against its gold artifact on the first task before spending
    *  model tokens; a benchmark whose judge rejects its own gold is recorded unavailable. Default true. */
   readonly verifyJudge?: boolean;
+  /** Caller policy inside each managed shot, including every refine-loop attempt. */
+  readonly execute?: BenchExecution;
   /** Test seam: a deterministic shot runner. Defaults to the `openSandboxRun` leaf. */
   readonly runShot?: BenchShot;
   /** Test seam: resolve a benchmark key to an adapter. Defaults to the registry `resolveAdapter`. */
@@ -122,9 +159,11 @@ interface BenchCellTaskResult {
   readonly rep: number;
   readonly resolved: boolean;
   readonly score: number;
-  /** false = the shot threw or produced no artifact (infra/empty), excluded from the resolve
-   *  denominator so a harness outage can't masquerade as a 0% capability result. */
+  /** Whether execution completed successfully and produced a readable, nonempty artifact. */
   readonly ok: boolean;
+  readonly execution?: BenchShotResult['execution'];
+  /** Available failed attempts remain in comparisons; unavailable measurement is reported separately. */
+  readonly measurement?: 'available' | 'unavailable';
   readonly detail?: string;
   readonly wallMs: number;
   /** Exact bytes given to the benchmark judge, retained even when judging fails. */
@@ -132,6 +171,7 @@ interface BenchCellTaskResult {
   readonly usage?: ReturnType<typeof sumSandboxUsage>;
   /** Worker events only; benchmark grading remains outside this trace. */
   readonly events?: readonly SandboxEvent[];
+  readonly prompts?: readonly BenchPromptResult[];
 }
 interface BenchLeaderboardRow {
   readonly benchmark: string;
@@ -319,5 +359,5 @@ declare class FilePierCandidateTrialController implements PierCandidateTrialCont
 //#region src/pier-result-grader.d.ts
 declare function createPierResultGrader(descriptor: Pick<PierCandidateGraderPort, 'name' | 'version' | 'artifact'>): PierCandidateGraderPort;
 //#endregion
-export { ADAPTERS, type BenchCell, type BenchCellTaskResult, type BenchLeaderboardRow, type BenchScore, type BenchShot, type BenchShotResult, type BenchTask, type BenchmarkAdapter, type ExecutePreparedPierCandidateOptions, FINAL_ANSWER_SENTINEL, FilePierCandidateTrialController, type FilePierCandidateTrialControllerOptions, type JudgeArtifactFileReceipt, type JudgeArtifactReceipt, type LoadOptions, type PierCandidateGraderPort, type PierCandidateOfficialResult, type PierCandidateProcessSpec, type PierCandidateTerminationAcknowledgement, type PierCandidateTrialController, type PierCandidateTrialHandle, type PierCandidateTrialIdentity, type PierCandidateTrialResult, type PierDockerConnection, type RagAnswerScore, type RagContext, type RunBenchmarksOptions, type RunBenchmarksReport, StagedJudgeError, type StagedPierCandidateExecution, type StagedRunCaptureSpec, type StagedRunSpec, type SweBenchAdapterOptions, type SweBenchArtifactCaptureContext, type SweBenchCacheLevel, answerScoreToBenchScore, contextBlock, contextsFrom, createCragAdapter, createNoMiraclAdapter, createOpenRagBenchAdapter, createPierCandidateRecoveryExecutor, createPierResultGrader, createRagBenchAdapter, createSweBenchAdapter, createT2RagBenchAdapter, executePreparedPierCandidate, normalizeAnswer, parseCitations, parseFinalAnswer, printBenchmarksReport, ragAnswerOutput, resolveAdapter, runBenchmarks, runStagedJudge, scoreAnswerArtifact, scoreSweReport, sweEvaluationArgv, swePatchOutput, tokenF1 };
+export { ADAPTERS, type BenchCell, type BenchCellTaskResult, type BenchExecution, type BenchExecutionContext, type BenchLeaderboardRow, type BenchPromptResult, type BenchScore, type BenchShot, type BenchShotResult, type BenchTask, type BenchmarkAdapter, type ExecutePreparedPierCandidateOptions, FINAL_ANSWER_SENTINEL, FilePierCandidateTrialController, type FilePierCandidateTrialControllerOptions, type JudgeArtifactFileReceipt, type JudgeArtifactReceipt, type LoadOptions, type PierCandidateGraderPort, type PierCandidateOfficialResult, type PierCandidateProcessSpec, type PierCandidateTerminationAcknowledgement, type PierCandidateTrialController, type PierCandidateTrialHandle, type PierCandidateTrialIdentity, type PierCandidateTrialResult, type PierDockerConnection, type RagAnswerScore, type RagContext, type RunBenchmarksOptions, type RunBenchmarksReport, StagedJudgeError, type StagedPierCandidateExecution, type StagedRunCaptureSpec, type StagedRunSpec, type SweBenchAdapterOptions, type SweBenchArtifactCaptureContext, type SweBenchCacheLevel, answerScoreToBenchScore, contextBlock, contextsFrom, createCragAdapter, createNoMiraclAdapter, createOpenRagBenchAdapter, createPierCandidateRecoveryExecutor, createPierResultGrader, createRagBenchAdapter, createSweBenchAdapter, createT2RagBenchAdapter, executePreparedPierCandidate, normalizeAnswer, parseCitations, parseFinalAnswer, printBenchmarksReport, ragAnswerOutput, resolveAdapter, runBenchmarks, runStagedJudge, scoreAnswerArtifact, scoreSweReport, sweEvaluationArgv, swePatchOutput, tokenF1 };
 //# sourceMappingURL=index.d.ts.map

package/dist/index.js

@@ -252,7 +252,7 @@ function finalText(events) {
 }
 /** The default real-agent shot: one `openSandboxRun` over the cell's harness+model, deliverable
 *  extracted by the adapter's parser (or final text), abortable on `timeoutMs`. */
-const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, routerKey, modelApiKey, bridgeUrl, bridgeBearer, sandboxBaseUrl, timeoutMs, signal, resolveClient }) => {
+const openSandboxShot = async ({ adapter, task, cell, prompt, attempt = 1, execute, routerBaseUrl, routerKey, modelApiKey, bridgeUrl, bridgeBearer, sandboxBaseUrl, timeoutMs, signal, resolveClient }) => {
 	signal?.throwIfAborted();
 	const client = (resolveClient ?? resolveBenchClient)({
 		backend: cell.backend ?? "router",
@@ -304,11 +304,15 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 	};
 	const controller = new AbortController();
 	const timer = timeoutMs ? setTimeout(() => controller.abort(), timeoutMs) : void 0;
+	const observedEvents = [];
 	const runOptions = {
 		agentRun,
 		signal: signal ? AbortSignal.any([controller.signal, signal]) : controller.signal,
 		runId: `bench:${adapter.name}:${task.id}:${uniq}`,
-		scenarioId: task.id
+		scenarioId: task.id,
+		onSandboxEvent: (event) => {
+			observedEvents.push(event);
+		}
 	};
 	const boxSetup = adapter.boxSetup;
 	if (boxSetup) runOptions.beforeStart = async ({ box, sessionId }) => {
@@ -325,14 +329,119 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 		artifact: "",
 		ok: false
 	};
+	const prompts = [];
+	const execution = { accepting: true };
 	try {
 		run = await openSandboxRun(client, runOptions, deliverable);
-		const turn = await run.start(prompt ?? task.prompt);
+		result = {
+			...result,
+			execution: {
+				phase: "unknown",
+				terminalOutcome: "unknown"
+			}
+		};
+		const managedRun = run;
+		const invoke = (method, input) => {
+			if (!execution.accepting || execution.active) {
+				execution.coordinationError = /* @__PURE__ */ new Error(!execution.accepting ? "benchmark execution callback has settled" : "benchmark execution requires sequential start/resume calls");
+				throw execution.coordinationError;
+			}
+			runOptions.signal.throwIfAborted();
+			const index = prompts.length;
+			const eventStart = observedEvents.length;
+			execution.lastTurn = void 0;
+			const capture = async () => {
+				let turn;
+				let failure;
+				try {
+					turn = await managedRun[method](input);
+					return turn;
+				} catch (error) {
+					failure = error;
+					throw error;
+				} finally {
+					const events = observedEvents.slice(eventStart);
+					if (turn) execution.lastTurn = {
+						...turn,
+						events,
+						outcome: { ...turn.outcome }
+					};
+					let sessionId;
+					try {
+						sessionId = managedRun.sessionId;
+					} catch {}
+					const readError = turn?.readError ?? (failure instanceof SandboxRunAbortError ? failure.readError : void 0);
+					prompts.push({
+						attempt,
+						index,
+						method,
+						prompt: input,
+						...sessionId === void 0 ? {} : { sessionId },
+						...turn === void 0 ? {} : { outcome: { ...turn.outcome } },
+						events,
+						usage: turn === void 0 ? {
+							...sumSandboxUsage(events),
+							tokensKnown: false,
+							usdKnown: false
+						} : sumSandboxUsage(events),
+						...readError === void 0 ? {} : { readError },
+						...turn !== void 0 ? {} : { error: failure instanceof Error ? failure.message : String(failure) }
+					});
+				}
+			};
+			const pending = capture();
+			execution.active = pending;
+			pending.then(() => {
+				execution.active = void 0;
+			}, () => {
+				execution.active = void 0;
+			});
+			return pending;
+		};
+		const context = {
+			prompt: prompt ?? task.prompt,
+			profile: structuredClone(profile),
+			benchmark: adapter.name,
+			taskId: task.id,
+			attempt,
+			signal: runOptions.signal,
+			run: {
+				start: (input) => invoke("start", input),
+				resume: (input) => invoke("resume", input),
+				get box() {
+					return managedRun.box;
+				},
+				get sessionId() {
+					return managedRun.sessionId;
+				}
+			}
+		};
+		try {
+			await executeWithSignal(async () => {
+				if (execute) await execute(context);
+				else await context.run.start(context.prompt);
+			}, runOptions.signal);
+		} catch (error) {
+			if (execution.active) controller.abort();
+			throw error;
+		} finally {
+			execution.accepting = false;
+			await execution.active?.catch(() => void 0);
+		}
+		if (execution.coordinationError) throw execution.coordinationError;
+		runOptions.signal.throwIfAborted();
+		const turn = execution.lastTurn;
+		if (!turn) throw new Error(prompts.at(-1)?.error ?? "benchmark execution returned without a completed prompt");
 		result = {
 			artifact: "",
 			ok: false,
-			usage: sumSandboxUsage(turn.events),
-			events: turn.events
+			usage: promptUsage(prompts),
+			events: observedEvents,
+			prompts,
+			execution: {
+				phase: "started",
+				terminalOutcome: turn.outcome.success ? "succeeded" : turn.outcome.status === "failed" ? "failed" : "incomplete"
+			}
 		};
 		let artifact = (turn.out ?? "").trim();
 		let boxExtractError;
@@ -379,19 +488,27 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 		result = {
 			artifact,
 			ok: turn.outcome.success && artifact.length > 0 && turn.readError === void 0 && boxExtractError === void 0,
+			execution: result.execution,
+			artifactAvailable: turn.readError === void 0 && boxExtractError === void 0,
 			usage: result.usage,
-			events: turn.events,
+			events: observedEvents,
+			prompts,
 			...detail ? { detail } : {}
 		};
 	} catch (err) {
+		const events = observedEvents;
 		result = {
 			...result,
 			ok: false,
+			artifactAvailable: false,
+			execution: {
+				phase: events.length > 0 ? "started" : "unknown",
+				terminalOutcome: result.execution?.terminalOutcome ?? "unknown"
+			},
 			detail: err instanceof Error ? err.message : String(err),
-			...err instanceof SandboxRunAbortError ? {
-				usage: sumSandboxUsage(err.events),
-				events: err.events
-			} : {}
+			usage: promptUsage(prompts),
+			events,
+			prompts
 		};
 	} finally {
 		if (timer) clearTimeout(timer);
@@ -407,6 +524,27 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 	}
 	return result;
 };
+/** Stop waiting on policy work when cancelled; the caller must also cancel its external effects. */
+async function executeWithSignal(execute, signal) {
+	signal.throwIfAborted();
+	let onAbort;
+	const aborted = new Promise((_resolve, reject) => {
+		onAbort = () => reject(signal.reason ?? /* @__PURE__ */ new Error("aborted"));
+		signal.addEventListener("abort", onAbort, { once: true });
+	});
+	try {
+		await Promise.race([Promise.resolve().then(execute), aborted]);
+	} finally {
+		if (onAbort) signal.removeEventListener("abort", onAbort);
+	}
+}
+function promptUsage(prompts) {
+	return combinedUsage(prompts.map((prompt) => ({
+		artifact: "",
+		ok: false,
+		usage: prompt.usage
+	})));
+}
 function parseMaybeJson(value) {
 	try {
 		return JSON.parse(value);
@@ -496,17 +634,25 @@ async function loopedShot(input, shot, attempts) {
 		return {
 			artifact: completed.at(-1)?.artifact ?? "",
 			ok: false,
+			execution: pendingShot ? {
+				phase: "unknown",
+				terminalOutcome: "unknown"
+			} : completed.at(-1)?.execution,
+			artifactAvailable: false,
 			usage: combinedUsage(pendingShot ? [...completed, {
 				artifact: "",
 				ok: false
 			}] : completed),
 			events: completed.flatMap((shot) => shot.events ?? []),
+			prompts: completed.flatMap((shot) => shot.prompts ?? []),
 			detail: err instanceof Error ? err.message : String(err)
 		};
 	}
 	const best = result.rounds.reduce((winner, candidate) => {
-		if (shots.get(candidate.round)?.ok !== true) return winner;
-		if (shots.get(winner.round)?.ok !== true) return candidate;
+		const candidateShot = shots.get(candidate.round);
+		const winnerShot = shots.get(winner.round);
+		const rank = (shot) => shot?.ok ? 2 : shot?.artifactAvailable ? 1 : 0;
+		if (rank(candidateShot) !== rank(winnerShot)) return rank(candidateShot) > rank(winnerShot) ? candidate : winner;
 		const a = scores.get(winner.round);
 		const b = scores.get(candidate.round);
 		if (!a) return candidate;
@@ -519,8 +665,11 @@ async function loopedShot(input, shot, attempts) {
 	return {
 		artifact: best.artifact,
 		ok: shots.get(best.round)?.ok === true && best.artifact.trim().length > 0,
+		execution: shots.get(best.round)?.execution,
+		artifactAvailable: shots.get(best.round)?.artifactAvailable,
 		usage: combinedUsage([...shots.values()]),
 		events: [...shots.values()].flatMap((shot) => shot.events ?? []),
+		prompts: [...shots.values()].flatMap((shot) => shot.prompts ?? []),
 		detail: JSON.stringify({
 			mode: "refine-loop",
 			attempts: result.rounds.length,
@@ -641,6 +790,7 @@ async function runBenchmarks(opts) {
 		const startedAt = Date.now();
 		let result;
 		let out;
+		let invoked = false;
 		try {
 			opts.signal?.throwIfAborted();
 			const shotInput = {
@@ -655,8 +805,10 @@ async function runBenchmarks(opts) {
 				...opts.sandboxBaseUrl ? { sandboxBaseUrl: opts.sandboxBaseUrl } : {},
 				...opts.timeoutMs ? { timeoutMs: opts.timeoutMs } : {},
 				...opts.signal ? { signal: opts.signal } : {},
-				...opts.resolveClient ? { resolveClient: opts.resolveClient } : {}
+				...opts.resolveClient ? { resolveClient: opts.resolveClient } : {},
+				...opts.execute ? { execute: opts.execute } : {}
 			};
+			invoked = true;
 			out = loopAttempts > 1 ? await loopedShot(shotInput, shot, loopAttempts) : await shot(shotInput);
 			const score = await job.adapter.judge(job.task, out.artifact);
 			result = {
@@ -667,11 +819,17 @@ async function runBenchmarks(opts) {
 				resolved: out.ok && score.resolved,
 				score: out.ok ? score.score : 0,
 				ok: out.ok,
+				execution: out.execution ?? {
+					phase: out.ok ? "started" : "unknown",
+					terminalOutcome: out.ok ? "succeeded" : "unknown"
+				},
+				measurement: out.artifactAvailable ?? out.ok ? "available" : "unavailable",
 				...out.detail ?? score.detail ? { detail: combineDetails(out.detail, score.detail) } : {},
 				wallMs: Date.now() - startedAt,
 				artifact: out.artifact,
 				...out.usage === void 0 ? {} : { usage: out.usage },
-				...out.events === void 0 ? {} : { events: out.events }
+				...out.events === void 0 ? {} : { events: out.events },
+				...out.prompts === void 0 ? {} : { prompts: out.prompts }
 			};
 		} catch (err) {
 			result = {
@@ -682,11 +840,17 @@ async function runBenchmarks(opts) {
 				resolved: false,
 				score: 0,
 				ok: false,
+				execution: out?.execution ?? {
+					phase: !invoked ? "not-started" : out?.ok ? "started" : "unknown",
+					terminalOutcome: out?.ok ? "succeeded" : "unknown"
+				},
+				measurement: "unavailable",
 				detail: err instanceof Error ? err.message.slice(0, 200) : String(err),
 				wallMs: Date.now() - startedAt,
 				...out === void 0 ? {} : { artifact: out.artifact },
 				...out?.usage === void 0 ? {} : { usage: out.usage },
-				...out?.events === void 0 ? {} : { events: out.events }
+				...out?.events === void 0 ? {} : { events: out.events },
+				...out?.prompts === void 0 ? {} : { prompts: out.prompts }
 			};
 		}
 		perTask.push(result);
@@ -714,7 +878,7 @@ function aggregate(perTask) {
 			scoreSum: 0
 		};
 		e.n += 1;
-		if (!r.ok) e.errored += 1;
+		if ((r.measurement ?? (r.ok ? "available" : "unavailable")) === "unavailable") e.errored += 1;
 		else {
 			if (r.resolved) e.resolved += 1;
 			e.scoreSum += r.score;