@tangle-network/agent-bench 0.10.0 → 0.11.1

package/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 0.11.1
+
+The published benchmark package now admits sandbox SDK 0.38.x, including consumer-ready runtime-edge readiness.
+
+## 0.11.0
+
+Adds caller-controlled start and resume inside managed benchmark shots.
+Retains setup, final extraction, grading, and cleanup while recording every prompt and its observed usage.
+The callback composes with outer retry attempts; learning policy and policy inference accounting remain consumer responsibilities.
+
 ## 0.9.4
 
 Requires Runtime 0.202.0 so Bench consumers receive explicit failed-execution settlement and retained failure recovery.

package/HARNESS.md

@@ -8,9 +8,11 @@
 |---|---|
 | **agent-runtime** | exact execution, reusable benchmark adapters, packed-consumer checks, one full-fidelity integration fixture |
 | **discovery** | research questions, preregistrations, acceptance criteria, negative results, and decisions about what is worth testing |
-| **discovery-lab** | paid campaigns, upstream method reproductions, long-horizon hill climbs, immutable receipts, and result archives |
+| **supervisor-lab** | registered adaptive-agent and retained-learning comparisons, including carried versus revised profiles |
+| **discovery-lab** | research campaigns and evidence for Discovery, including immutable inputs, run records, and result archives |
 
-A benchmark implementation may begin here while it is becoming a reusable adapter. Once the question is “does method X improve benchmark Y?”, the campaign belongs in Discovery Lab.
+A benchmark implementation may begin here while it is becoming a reusable adapter. Once the question is “does method X improve benchmark Y?”, the campaign belongs in the consuming lab.
+Supervisor Lab owns adaptive-learning comparisons; Discovery Lab owns research campaigns for Discovery.
 
 ## Evidence levels
 
@@ -20,8 +22,8 @@ Use these labels literally. Do not promote one level into another in prose.
 |---|---|---|
 | **contract proof** | packages install; identities, budgets, callbacks, resume, and receipts have the expected shape | root `pnpm verify:official-optimizers`, `pnpm verify:bench` |
 | **evaluator proof** | the benchmark's own evaluator can distinguish known fail/pass artifacts in the exact environment | adapter preflight and gold/self-check |
-| **reproduction proof** | an upstream method is run at a pinned revision on its claimed benchmark under a matched protocol | Discovery Lab reproduction manifest and runner |
-| **value proof** | the integrated method beats the preregistered baseline on frozen evidence with uncertainty and complete cost accounting | Discovery Lab result receipt |
+| **reproduction proof** | an upstream method is run at a pinned revision on its claimed benchmark under a matched protocol | Consuming lab reproduction manifest and runner |
+| **value proof** | the integrated method beats the preregistered baseline on frozen evidence with uncertainty and complete cost accounting | Consuming lab result receipt |
 | **production proof** | a promoted artifact transfers to real traffic under a canary or controlled rollout | product repository / platform telemetry |
 
 A localization score, output-shape check, LLM quality judge, or toy deterministic reward can be useful for development. None is a substitute for the benchmark's outcome evaluator.
@@ -71,6 +73,42 @@ The current Runtime lineage suppresses sandbox deletion errors, so a returned re
 The caller's abort signal stops queued shots and reaches active sandbox turns.
 `modelApiKey` supplies sandbox inference authorization separately from the `routerKey` used for sandbox control.
 
+### Caller-controlled prompts within a task
+
+`runBenchmarks({ execute })` invokes the callback inside each managed sandbox shot.
+The context supplies the task prompt, executed profile, benchmark and task identities, attempt number, and abort signal.
+Its managed `run` exposes Runtime's `start`, `resume`, `box`, and `sessionId`.
+Use the live box for permitted working checks in the same session.
+Submit worker prompts through managed `start` and `resume` so Bench captures their outcomes and usage.
+Direct sandbox prompt calls bypass this accounting.
+The callback must leave session lifecycle, extraction, and cleanup to Bench.
+It receives no adapter, final grader, or task metadata containing gold material.
+This callback is trusted consumer code, not an isolation boundary for arbitrary code.
+
+Return after the final prompt completes.
+Bench extracts the last captured prompt's artifact and applies the adapter's final grading outside the callback.
+Start and resume calls must be sequential.
+Bench waits for an unawaited active invocation before extraction and refuses calls after the callback settles.
+The callback must use its signal to cancel external work.
+Bench stops awaiting policy work when cancelled, but cannot stop external effects that ignore cancellation.
+
+Each task's `prompts` retains ordered method, prompt, attempt, session identity, outcome, events, usage, and capture errors.
+Prompt indices start at zero; attempt numbers start at one.
+Usage sums each prompt separately, including failed and interrupted prompts.
+A partial capture retains observed counters and marks accounting incomplete.
+These counters cover sandbox workers only; consumers must account for policy and working-evaluator inference separately.
+Consumers must bind their callback source, configuration, profiles, and initial state to their execution identity.
+
+`execute` runs inside every `loopAttempts` shot.
+Those outer attempts still create fresh sandboxes and use the existing checker feedback policy.
+Use one outer attempt when final grading must remain unavailable to adaptation.
+A custom `runShot` receives `execute` and owns whether it consumes the callback.
+
+A sandbox prompt can contain multiple native model requests.
+Same-session continuation does not prove a barrier before every native request, profile reload, coordinator restart, or fresh-session state transfer.
+Offline fake-sandbox tests prove the managed contract and correction consumption only.
+They establish no live learning gain or provider session restoration.
+
 ### Retained strategy driver
 
 ```bash
@@ -133,4 +171,4 @@ A new file under `bench/src` must be one of:
 - a package-consumer or evaluator calibration test;
 - one canonical full-fidelity fixture that exercises a public Runtime contract.
 
-A one-off campaign, generation-N optimizer script, bespoke dashboard, or historical result belongs in Discovery Lab. If an older file has no package script, no importer, and no unique reusable primitive, delete it rather than adding another index entry.
+A one-off campaign, generation-N optimizer script, bespoke dashboard, or historical result belongs in the consuming lab. If an older file has no package script, no importer, and no unique reusable primitive, delete it rather than adding another index entry.

package/README.md

@@ -100,3 +100,25 @@ Pier owns the task container and verifier; protected model usage and traces stay
 `FilePierCandidateTrialController` atomically reserves a unique Pier job, then persists the supervisor PID, process-session identity, and that job's exact Docker projects so a fresh evaluator process can stop and remove an abandoned trial.
 Run `PIER_REPO=/path/to/pier pnpm verify:pier` for the zero-model failure/pass and fresh-process recovery proof, and see `HARNESS.md` for the exact invocation and failure contract.
 From an installed npm package, expose the shipped Python module with `export PYTHONPATH="$(npm root)/@tangle-network/agent-bench${PYTHONPATH:+:$PYTHONPATH}"` before invoking Pier.
+
+## Control execution within a managed shot
+
+Supply `execute` to control whole sandbox prompts while Bench owns setup, extraction, grading, and cleanup.
+Your policy can inspect permitted working checks through the existing sandbox handle.
+
+```ts
+import { runBenchmarks, type BenchExecution } from '@tangle-network/agent-bench'
+import { executionOptions, chooseNextPrompt } from './policy.js'
+
+const execute: BenchExecution = async ({ run, prompt, signal }) => {
+  const first = await run.start(prompt)
+  const correction = await chooseNextPrompt({ first, box: run.box, sessionId: run.sessionId, signal })
+  if (correction !== undefined) await run.resume(correction)
+}
+
+const report = await runBenchmarks({ ...executionOptions, execute })
+```
+
+The consumer supplies and identifies `chooseNextPrompt`; Bench provides no learning policy.
+`perTask[].prompts` retains each invocation and its observed worker cost, including partial failures.
+See [the execution contract](./HARNESS.md#caller-controlled-prompts-within-a-task) for accounting, retry composition, and lifecycle limits.

package/dist/index.d.ts

@@ -8,7 +8,7 @@ import { FINAL_ANSWER_SENTINEL, RagAnswerScore, RagContext, answerScoreToBenchSc
 import { createRagBenchAdapter } from "./benchmarks/ragbench.js";
 import { SweBenchAdapterOptions, SweBenchArtifactCaptureContext, SweBenchCacheLevel, createSweBenchAdapter, scoreSweReport, sweEvaluationArgv, swePatchOutput } from "./benchmarks/swe-bench.js";
 import { createT2RagBenchAdapter } from "./benchmarks/t2-ragbench.js";
-import { AgentProfile, SandboxClient, sumSandboxUsage } from "@tangle-network/agent-runtime/kernel";
+import { AgentProfile, SandboxClient, SandboxRun, TurnResult, sumSandboxUsage } from "@tangle-network/agent-runtime/kernel";
 import { SandboxEvent } from "@tangle-network/sandbox";
 import { AgentCandidateBenchmarkGraderPort, AgentCandidateExecutionClaimStore, AgentCandidateExecutorPort, AgentCandidateExecutorRequest, AgentCandidateExecutorStopRequest, AgentCandidateOutputArtifactPort, AgentCandidateRunFinalization, PreparedAgentCandidateExecution } from "@tangle-network/agent-runtime/candidate-execution";
 import { RunTerminalOutcome, TraceStore } from "@tangle-network/agent-eval";
@@ -48,6 +48,31 @@ interface BenchCell {
   /** The agent under test. Defaults to a minimal `{ name, metadata.backendType }` profile. */
   readonly profile?: AgentProfile;
 }
+/** Caller-owned work inside a managed benchmark shot. Final grading stays outside this callback. */
+interface BenchExecutionContext {
+  readonly prompt: string;
+  readonly profile: AgentProfile;
+  readonly benchmark: string;
+  readonly taskId: string;
+  readonly attempt: number;
+  readonly signal: AbortSignal;
+  /** Runtime owns session continuity. Bench owns capture, extraction, and close. */
+  readonly run: Pick<SandboxRun<string>, 'start' | 'resume' | 'box' | 'sessionId'>;
+}
+type BenchExecution = (context: BenchExecutionContext) => Promise<void>;
+/** One submitted sandbox prompt, including partial evidence when its invocation throws. */
+interface BenchPromptResult {
+  readonly attempt: number;
+  readonly index: number;
+  readonly method: 'start' | 'resume';
+  readonly prompt: string;
+  readonly sessionId?: string;
+  readonly outcome?: TurnResult<string>['outcome'];
+  readonly events: readonly SandboxEvent[];
+  readonly usage: ReturnType<typeof sumSandboxUsage>;
+  readonly readError?: string;
+  readonly error?: string;
+}
 /** A worker's artifact and observed execution evidence, before external grading. */
 interface BenchShotResult {
   readonly artifact: string;
@@ -56,6 +81,7 @@ interface BenchShotResult {
   /** Provider observations, including explicit unknown counters. Omitted when the shot reports none. */
   readonly usage?: ReturnType<typeof sumSandboxUsage>;
   readonly events?: readonly SandboxEvent[];
+  readonly prompts?: readonly BenchPromptResult[];
   /** Observed dispatch and terminal state, independent of artifact quality. */
   readonly execution?: {
     readonly phase: 'not-started' | 'started' | 'unknown';
@@ -73,6 +99,8 @@ type BenchShot = (input: {
   readonly prompt?: string;
   /** 1-based attempt index for looped runs. */
   readonly attempt?: number;
+  /** Custom shots own whether they consume this managed execution callback. */
+  readonly execute?: BenchExecution;
   readonly routerBaseUrl: string;
   readonly routerKey: string;
   /** Optional inference credential for the box; routerKey continues to authorize sandbox control. */
@@ -116,6 +144,8 @@ interface RunBenchmarksOptions {
   /** Self-verify each benchmark's judge against its gold artifact on the first task before spending
    *  model tokens; a benchmark whose judge rejects its own gold is recorded unavailable. Default true. */
   readonly verifyJudge?: boolean;
+  /** Caller policy inside each managed shot, including every refine-loop attempt. */
+  readonly execute?: BenchExecution;
   /** Test seam: a deterministic shot runner. Defaults to the `openSandboxRun` leaf. */
   readonly runShot?: BenchShot;
   /** Test seam: resolve a benchmark key to an adapter. Defaults to the registry `resolveAdapter`. */
@@ -141,6 +171,7 @@ interface BenchCellTaskResult {
   readonly usage?: ReturnType<typeof sumSandboxUsage>;
   /** Worker events only; benchmark grading remains outside this trace. */
   readonly events?: readonly SandboxEvent[];
+  readonly prompts?: readonly BenchPromptResult[];
 }
 interface BenchLeaderboardRow {
   readonly benchmark: string;
@@ -328,5 +359,5 @@ declare class FilePierCandidateTrialController implements PierCandidateTrialCont
 //#region src/pier-result-grader.d.ts
 declare function createPierResultGrader(descriptor: Pick<PierCandidateGraderPort, 'name' | 'version' | 'artifact'>): PierCandidateGraderPort;
 //#endregion
-export { ADAPTERS, type BenchCell, type BenchCellTaskResult, type BenchLeaderboardRow, type BenchScore, type BenchShot, type BenchShotResult, type BenchTask, type BenchmarkAdapter, type ExecutePreparedPierCandidateOptions, FINAL_ANSWER_SENTINEL, FilePierCandidateTrialController, type FilePierCandidateTrialControllerOptions, type JudgeArtifactFileReceipt, type JudgeArtifactReceipt, type LoadOptions, type PierCandidateGraderPort, type PierCandidateOfficialResult, type PierCandidateProcessSpec, type PierCandidateTerminationAcknowledgement, type PierCandidateTrialController, type PierCandidateTrialHandle, type PierCandidateTrialIdentity, type PierCandidateTrialResult, type PierDockerConnection, type RagAnswerScore, type RagContext, type RunBenchmarksOptions, type RunBenchmarksReport, StagedJudgeError, type StagedPierCandidateExecution, type StagedRunCaptureSpec, type StagedRunSpec, type SweBenchAdapterOptions, type SweBenchArtifactCaptureContext, type SweBenchCacheLevel, answerScoreToBenchScore, contextBlock, contextsFrom, createCragAdapter, createNoMiraclAdapter, createOpenRagBenchAdapter, createPierCandidateRecoveryExecutor, createPierResultGrader, createRagBenchAdapter, createSweBenchAdapter, createT2RagBenchAdapter, executePreparedPierCandidate, normalizeAnswer, parseCitations, parseFinalAnswer, printBenchmarksReport, ragAnswerOutput, resolveAdapter, runBenchmarks, runStagedJudge, scoreAnswerArtifact, scoreSweReport, sweEvaluationArgv, swePatchOutput, tokenF1 };
+export { ADAPTERS, type BenchCell, type BenchCellTaskResult, type BenchExecution, type BenchExecutionContext, type BenchLeaderboardRow, type BenchPromptResult, type BenchScore, type BenchShot, type BenchShotResult, type BenchTask, type BenchmarkAdapter, type ExecutePreparedPierCandidateOptions, FINAL_ANSWER_SENTINEL, FilePierCandidateTrialController, type FilePierCandidateTrialControllerOptions, type JudgeArtifactFileReceipt, type JudgeArtifactReceipt, type LoadOptions, type PierCandidateGraderPort, type PierCandidateOfficialResult, type PierCandidateProcessSpec, type PierCandidateTerminationAcknowledgement, type PierCandidateTrialController, type PierCandidateTrialHandle, type PierCandidateTrialIdentity, type PierCandidateTrialResult, type PierDockerConnection, type RagAnswerScore, type RagContext, type RunBenchmarksOptions, type RunBenchmarksReport, StagedJudgeError, type StagedPierCandidateExecution, type StagedRunCaptureSpec, type StagedRunSpec, type SweBenchAdapterOptions, type SweBenchArtifactCaptureContext, type SweBenchCacheLevel, answerScoreToBenchScore, contextBlock, contextsFrom, createCragAdapter, createNoMiraclAdapter, createOpenRagBenchAdapter, createPierCandidateRecoveryExecutor, createPierResultGrader, createRagBenchAdapter, createSweBenchAdapter, createT2RagBenchAdapter, executePreparedPierCandidate, normalizeAnswer, parseCitations, parseFinalAnswer, printBenchmarksReport, ragAnswerOutput, resolveAdapter, runBenchmarks, runStagedJudge, scoreAnswerArtifact, scoreSweReport, sweEvaluationArgv, swePatchOutput, tokenF1 };
 //# sourceMappingURL=index.d.ts.map

package/dist/index.js

@@ -252,7 +252,7 @@ function finalText(events) {
 }
 /** The default real-agent shot: one `openSandboxRun` over the cell's harness+model, deliverable
 *  extracted by the adapter's parser (or final text), abortable on `timeoutMs`. */
-const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, routerKey, modelApiKey, bridgeUrl, bridgeBearer, sandboxBaseUrl, timeoutMs, signal, resolveClient }) => {
+const openSandboxShot = async ({ adapter, task, cell, prompt, attempt = 1, execute, routerBaseUrl, routerKey, modelApiKey, bridgeUrl, bridgeBearer, sandboxBaseUrl, timeoutMs, signal, resolveClient }) => {
 	signal?.throwIfAborted();
 	const client = (resolveClient ?? resolveBenchClient)({
 		backend: cell.backend ?? "router",
@@ -329,6 +329,8 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 		artifact: "",
 		ok: false
 	};
+	const prompts = [];
+	const execution = { accepting: true };
 	try {
 		run = await openSandboxRun(client, runOptions, deliverable);
 		result = {
@@ -338,12 +340,104 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 				terminalOutcome: "unknown"
 			}
 		};
-		const turn = await run.start(prompt ?? task.prompt);
+		const managedRun = run;
+		const invoke = (method, input) => {
+			if (!execution.accepting || execution.active) {
+				execution.coordinationError = /* @__PURE__ */ new Error(!execution.accepting ? "benchmark execution callback has settled" : "benchmark execution requires sequential start/resume calls");
+				throw execution.coordinationError;
+			}
+			runOptions.signal.throwIfAborted();
+			const index = prompts.length;
+			const eventStart = observedEvents.length;
+			execution.lastTurn = void 0;
+			const capture = async () => {
+				let turn;
+				let failure;
+				try {
+					turn = await managedRun[method](input);
+					return turn;
+				} catch (error) {
+					failure = error;
+					throw error;
+				} finally {
+					const events = observedEvents.slice(eventStart);
+					if (turn) execution.lastTurn = {
+						...turn,
+						events,
+						outcome: { ...turn.outcome }
+					};
+					let sessionId;
+					try {
+						sessionId = managedRun.sessionId;
+					} catch {}
+					const readError = turn?.readError ?? (failure instanceof SandboxRunAbortError ? failure.readError : void 0);
+					prompts.push({
+						attempt,
+						index,
+						method,
+						prompt: input,
+						...sessionId === void 0 ? {} : { sessionId },
+						...turn === void 0 ? {} : { outcome: { ...turn.outcome } },
+						events,
+						usage: turn === void 0 ? {
+							...sumSandboxUsage(events),
+							tokensKnown: false,
+							usdKnown: false
+						} : sumSandboxUsage(events),
+						...readError === void 0 ? {} : { readError },
+						...turn !== void 0 ? {} : { error: failure instanceof Error ? failure.message : String(failure) }
+					});
+				}
+			};
+			const pending = capture();
+			execution.active = pending;
+			pending.then(() => {
+				execution.active = void 0;
+			}, () => {
+				execution.active = void 0;
+			});
+			return pending;
+		};
+		const context = {
+			prompt: prompt ?? task.prompt,
+			profile: structuredClone(profile),
+			benchmark: adapter.name,
+			taskId: task.id,
+			attempt,
+			signal: runOptions.signal,
+			run: {
+				start: (input) => invoke("start", input),
+				resume: (input) => invoke("resume", input),
+				get box() {
+					return managedRun.box;
+				},
+				get sessionId() {
+					return managedRun.sessionId;
+				}
+			}
+		};
+		try {
+			await executeWithSignal(async () => {
+				if (execute) await execute(context);
+				else await context.run.start(context.prompt);
+			}, runOptions.signal);
+		} catch (error) {
+			if (execution.active) controller.abort();
+			throw error;
+		} finally {
+			execution.accepting = false;
+			await execution.active?.catch(() => void 0);
+		}
+		if (execution.coordinationError) throw execution.coordinationError;
+		runOptions.signal.throwIfAborted();
+		const turn = execution.lastTurn;
+		if (!turn) throw new Error(prompts.at(-1)?.error ?? "benchmark execution returned without a completed prompt");
 		result = {
 			artifact: "",
 			ok: false,
-			usage: sumSandboxUsage(turn.events),
-			events: turn.events,
+			usage: promptUsage(prompts),
+			events: observedEvents,
+			prompts,
 			execution: {
 				phase: "started",
 				terminalOutcome: turn.outcome.success ? "succeeded" : turn.outcome.status === "failed" ? "failed" : "incomplete"
@@ -397,11 +491,12 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 			execution: result.execution,
 			artifactAvailable: turn.readError === void 0 && boxExtractError === void 0,
 			usage: result.usage,
-			events: turn.events,
+			events: observedEvents,
+			prompts,
 			...detail ? { detail } : {}
 		};
 	} catch (err) {
-		const events = err instanceof SandboxRunAbortError ? err.events : observedEvents;
+		const events = observedEvents;
 		result = {
 			...result,
 			ok: false,
@@ -411,12 +506,9 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 				terminalOutcome: result.execution?.terminalOutcome ?? "unknown"
 			},
 			detail: err instanceof Error ? err.message : String(err),
-			usage: {
-				...sumSandboxUsage(events),
-				tokensKnown: false,
-				usdKnown: false
-			},
-			events
+			usage: promptUsage(prompts),
+			events,
+			prompts
 		};
 	} finally {
 		if (timer) clearTimeout(timer);
@@ -432,6 +524,27 @@ const openSandboxShot = async ({ adapter, task, cell, prompt, routerBaseUrl, rou
 	}
 	return result;
 };
+/** Stop waiting on policy work when cancelled; the caller must also cancel its external effects. */
+async function executeWithSignal(execute, signal) {
+	signal.throwIfAborted();
+	let onAbort;
+	const aborted = new Promise((_resolve, reject) => {
+		onAbort = () => reject(signal.reason ?? /* @__PURE__ */ new Error("aborted"));
+		signal.addEventListener("abort", onAbort, { once: true });
+	});
+	try {
+		await Promise.race([Promise.resolve().then(execute), aborted]);
+	} finally {
+		if (onAbort) signal.removeEventListener("abort", onAbort);
+	}
+}
+function promptUsage(prompts) {
+	return combinedUsage(prompts.map((prompt) => ({
+		artifact: "",
+		ok: false,
+		usage: prompt.usage
+	})));
+}
 function parseMaybeJson(value) {
 	try {
 		return JSON.parse(value);
@@ -531,6 +644,7 @@ async function loopedShot(input, shot, attempts) {
 				ok: false
 			}] : completed),
 			events: completed.flatMap((shot) => shot.events ?? []),
+			prompts: completed.flatMap((shot) => shot.prompts ?? []),
 			detail: err instanceof Error ? err.message : String(err)
 		};
 	}
@@ -555,6 +669,7 @@ async function loopedShot(input, shot, attempts) {
 		artifactAvailable: shots.get(best.round)?.artifactAvailable,
 		usage: combinedUsage([...shots.values()]),
 		events: [...shots.values()].flatMap((shot) => shot.events ?? []),
+		prompts: [...shots.values()].flatMap((shot) => shot.prompts ?? []),
 		detail: JSON.stringify({
 			mode: "refine-loop",
 			attempts: result.rounds.length,
@@ -690,7 +805,8 @@ async function runBenchmarks(opts) {
 				...opts.sandboxBaseUrl ? { sandboxBaseUrl: opts.sandboxBaseUrl } : {},
 				...opts.timeoutMs ? { timeoutMs: opts.timeoutMs } : {},
 				...opts.signal ? { signal: opts.signal } : {},
-				...opts.resolveClient ? { resolveClient: opts.resolveClient } : {}
+				...opts.resolveClient ? { resolveClient: opts.resolveClient } : {},
+				...opts.execute ? { execute: opts.execute } : {}
 			};
 			invoked = true;
 			out = loopAttempts > 1 ? await loopedShot(shotInput, shot, loopAttempts) : await shot(shotInput);
@@ -712,7 +828,8 @@ async function runBenchmarks(opts) {
 				wallMs: Date.now() - startedAt,
 				artifact: out.artifact,
 				...out.usage === void 0 ? {} : { usage: out.usage },
-				...out.events === void 0 ? {} : { events: out.events }
+				...out.events === void 0 ? {} : { events: out.events },
+				...out.prompts === void 0 ? {} : { prompts: out.prompts }
 			};
 		} catch (err) {
 			result = {
@@ -732,7 +849,8 @@ async function runBenchmarks(opts) {
 				wallMs: Date.now() - startedAt,
 				...out === void 0 ? {} : { artifact: out.artifact },
 				...out?.usage === void 0 ? {} : { usage: out.usage },
-				...out?.events === void 0 ? {} : { events: out.events }
+				...out?.events === void 0 ? {} : { events: out.events },
+				...out?.prompts === void 0 ? {} : { prompts: out.prompts }
 			};
 		}
 		perTask.push(result);