humanish 0.131.1 → 0.133.0

package/CHANGELOG.md

@@ -8,6 +8,31 @@ The Unreleased section holds the full notes for the next version until it is tag
 
 ## Unreleased
 
+## 0.133.0: An OpenAI participant names how long it waits (2026-10-14)
+
+humanish 0.133.0 lets an OpenAI computer-use participant name how long it waits, with a
+`wait_minutes` tool, and cuts what a participant costs while nothing on its screen changes. A turn of
+only waits is watched, and each further quiet wait in a row lasts twice as long, starting from the
+length the participant named, up to 10 minutes. In live 60-minute runs against 0.132.0, median model
+spend per participant-hour fell 13% on a resident portal, 52% on a support queue, 46% on an issue
+list that shows new issues only on reload, and 78% on a dispatch board, and all 6 participants on
+the issue list reloaded and reported every new issue. verify and analysis also no longer fail on a
+run whose traces together pass 512 MiB.
+
+[Release notes](https://github.com/danielgwilson/humanish/releases/tag/v0.133.0)
+
+## 0.132.0: An OpenAI participant's closing report reads the prompt cache (2026-10-13)
+
+humanish 0.132.0 cuts the cost of an OpenAI computer-use participant's closing report. The closing
+report and impressions requests set `text.format`, which is part of the prompt cache's prefix, so
+each one was billed as a fresh write of the whole session: 25% to 42% of a run's model spend in the
+8 runs measured on 0.131.0. They now send the schema in their input instead and read the session
+from the cache. In two live 12-minute runs the closing request cost $0.026 and $0.025 where it
+would have cost $0.14. A reply that does not match the schema is asked once more with the strict
+schema, after the same budget and time checks.
+
+[Release notes](https://github.com/danielgwilson/humanish/releases/tag/v0.132.0)
+
 ## 0.131.1: Hosted desktops keep their screens on while participants wait (2026-10-13)
 
 humanish 0.131.1 keeps a hosted desktop's screen on while a participant waits without input.

package/dist/actors/codex/restricted-participant.d.ts

@@ -22,6 +22,8 @@ export interface RestrictedParticipantOptions {
      * defaults the loop applies without it (resolveWaitSettings).
      */
     maxWaitMs?: number;
+    /** The participant's idleWaitMs, the shortest a call of only waits lasts, stated the same way. */
+    idleWaitMs?: number;
 }
 /** What doctor's hosted-participant check found, before any turn. */
 export interface ParticipantReadiness {

package/dist/actors/codex/restricted-participant.js

@@ -41,7 +41,7 @@ function deferred() {
     return { promise, resolve, reject };
 }
 // prose-check: model prompt (Codex reads this tool description, not a person)
-const toolDescription = (speechEnabled, { maxWaitMs, watchMaxMs }) => `Act on the participant's browser through humanish. Submit one to four UI actions${speechEnabled ? ", including speak when you need to reply aloud," : ""} and a short public comment; never private reasoning. humanish returns a JSON STRING with execution acknowledgments${speechEnabled ? ", speech heard from the actual participant speaker sink," : ""} and a fresh screenshot. In Code Mode use: const r = JSON.parse(await tools.humanish_ui({narration: "...", actions: [...]})); text({acknowledgments: r.acknowledgments${speechEnabled ? ", heardSpeech: r.heardSpeech" : ""}, contextHint: r.contextHint, closing: r.closing}); image(r.imageUrl). Call serially and inspect each returned screenshot${speechEnabled ? " and heardSpeech array" : ""} before deciding what to do next. A call whose actions are only waits returns when the screen changes and holds still${speechEnabled ? ", when speech is heard" : ""}, or after its waits' ms, together at most ${watchMaxMs} ms. While no change on the screen holds still, each further such call in a row waits twice as long as the one before, up to ${watchMaxMs} ms, and contextHint says how long it waited and why it returned. A wait among other actions lasts its ms, at most ${maxWaitMs} ms.${speechEnabled ? " Use speak only after the visible UI shows that you joined the call and your microphone is unmuted; it sends audio into that call." : ""} Acknowledged input does not prove an application outcome. If closing is true, stop calling tools and give your final account.`;
+const toolDescription = (speechEnabled, { maxWaitMs, watchMaxMs, idleWaitMs, }) => `Act on the participant's browser through humanish. Submit one to four UI actions${speechEnabled ? ", including speak when you need to reply aloud," : ""} and a short public comment; never private reasoning. humanish returns a JSON STRING with execution acknowledgments${speechEnabled ? ", speech heard from the actual participant speaker sink," : ""} and a fresh screenshot. In Code Mode use: const r = JSON.parse(await tools.humanish_ui({narration: "...", actions: [...]})); text({acknowledgments: r.acknowledgments${speechEnabled ? ", heardSpeech: r.heardSpeech" : ""}, contextHint: r.contextHint, closing: r.closing}); image(r.imageUrl). Call serially and inspect each returned screenshot${speechEnabled ? " and heardSpeech array" : ""} before deciding what to do next. A call whose actions are only waits and that asks for some wait time asks to wait its waits' ms together; an ask under ${idleWaitMs} ms counts as ${idleWaitMs} ms, and named ms count up to ${watchMaxMs} ms${speechEnabled ? ". It returns when speech is heard" : ""}. contextHint says how long it waited and why it returned. A wait among other actions lasts its ms, at most ${maxWaitMs} ms.${speechEnabled ? " Use speak only after the visible UI shows that you joined the call and your microphone is unmuted; it sends audio into that call." : ""} Acknowledged input does not prove an application outcome. If closing is true, stop calling tools and give your final account.`;
 const participantInstructions = (prompt, speechEnabled) => `${prompt}\n\nYou are the study participant throughout this conversation, including its closing account. Use only the supplied screenshots and humanish_ui tool to interact. The tool returns a JSON string: parse it, inspect acknowledgments${speechEnabled ? " and heardSpeech captured from the actual participant speaker sink" : ""}, and display imageUrl with Code Mode image(). Do not print the image data URL as text. Keep your persona and earlier observations throughout the session.${speechEnabled ? " A speak action plays into the participant microphone; use it only after the visible UI shows that you joined the call and the microphone is unmuted." : ""} Speak publicly about your experience, never reveal private reasoning. Completed inputs do not prove application outcomes; verify on the next screenshot. When the task ends, return only the required final JSON with outcome, summary, frictionReports and impressions. Report observed confusion and recovered mistakes as well as blockers. ${IMPRESSIONS_ASK} Do not invent observations.`;
 /** Turns and failures from the native run, handed in order to the next provider request. */
 class ParticipantEvents {
@@ -192,7 +192,11 @@ function participantSession(options, speechEnabled, effort, call) {
             reasoningEffort: effort,
             tool: {
                 name: "humanish_ui",
-                description: toolDescription(speechEnabled, resolveWaitSettings({ maxWaitMs: options.maxWaitMs, speechEnabled })),
+                description: toolDescription(speechEnabled, resolveWaitSettings({
+                    maxWaitMs: options.maxWaitMs,
+                    idleWaitMs: options.idleWaitMs,
+                    speechEnabled,
+                })),
                 inputSchema: participantToolSchema(speechEnabled),
                 call,
             },

package/dist/actors/computer-use/loop/actions.js

@@ -209,6 +209,7 @@ export async function runActionBatch(session, actions, watched) {
                 ? { coord: { x: action.x, y: action.y } }
                 : {}),
             ...(watched !== undefined && waitIndex === watched.endedOn ? { watch: watched.watch } : {}),
+            ...(requested.kind === "wait" && requested.ms !== undefined ? { namedMs: requested.ms } : {}),
         }));
     }
     return { execution, hint: hint() };

package/dist/actors/computer-use/loop/debrief.js

@@ -86,36 +86,68 @@ function planDebrief(session, ask, trigger) {
     }
     return { send, boundMs };
 }
-async function debriefExchange(session, context, trigger, ask, { send, boundMs }, record) {
+async function debriefExchange(session, context, trigger, ask, plan, record) {
+    const { provider } = session;
+    const request = {
+        instructions: session.settings.instructions,
+        observation: trigger.observation,
+        ...(provider.requestPolicy !== "fail_closed" || context.previousExecution === undefined
+            ? {}
+            : { previousExecution: context.previousExecution }),
+        ...(context.previousResponseId === undefined
+            ? {}
+            : { previousResponseId: context.previousResponseId }),
+        ...(context.acknowledgedSafetyChecks === undefined
+            ? {}
+            : { acknowledgedSafetyChecks: context.acknowledgedSafetyChecks }),
+        contextHint: ask.contextHint,
+    };
+    const first = await sendClosing(session, ask, request, plan);
+    if (!("turn" in first)) {
+        return first.status === "skipped"
+            ? record("skipped", `${first.detail}; no closing request was sent`)
+            : record("failed", first.detail, first.usageReported);
+    }
+    if (first.turn.closingRetry !== true)
+        return acceptDebriefTurn(session, first.turn, trigger, ask, record, first.usageReported);
+    // The first reply's usage is already recorded, so the checks see it. Under a cap, a reply
+    // without complete usage stops another paid request, as it does after a participant turn.
+    const again = session.capDeclared && !first.usageReported
+        ? {
+            skip: "the closing request did not report complete usage, so the remaining model budget is unknown",
+        }
+        : planDebrief(session, ask, trigger);
+    const second = "skip" in again
+        ? { status: "skipped", detail: again.skip }
+        : await sendClosing(session, ask, request, again);
+    if ("turn" in second) {
+        const usageReported = first.usageReported && second.usageReported;
+        return acceptDebriefTurn(session, second.turn, trigger, ask, record, usageReported, "; asked twice: the first reply was not the JSON the request asked for");
+    }
+    // The first reply stands, with why no second one replaced it.
+    const skipped = second.status === "skipped";
+    return acceptDebriefTurn(session, first.turn, trigger, ask, record, first.usageReported && (skipped || second.usageReported), `; ${skipped ? "not asked again" : "asked once more"}: ${second.detail}`);
+}
+/** Send one closing request within its time bound, and record the usage its reply reports. */
+async function sendClosing(session, ask, request, { send, boundMs }) {
     const { signal, provider } = session;
     const scope = requestScope(signal);
     const timer = setTimeout(() => scope.end(), boundMs);
     timer.unref?.();
     session.trace.bump("debriefCalls");
     try {
-        const request = {
-            instructions: session.settings.instructions,
-            observation: trigger.observation,
-            ...(provider.requestPolicy !== "fail_closed" || context.previousExecution === undefined
-                ? {}
-                : { previousExecution: context.previousExecution }),
-            ...(context.previousResponseId === undefined
-                ? {}
-                : { previousResponseId: context.previousResponseId }),
-            ...(context.acknowledgedSafetyChecks === undefined
-                ? {}
-                : { acknowledgedSafetyChecks: context.acknowledgedSafetyChecks }),
-            contextHint: ask.contextHint,
-        };
         const turn = provider.requestPolicy === "fail_closed"
             ? await singleDispatch(session, "debrief", (dispatchSignal) => send(request, dispatchSignal), boundMs)
             : await raceSessionDeadline(send(request, scope.signal), boundMs, signal);
-        return acceptDebriefTurn(session, turn, trigger, ask, record);
+        return { turn, usageReported: recordClosingUsage(session, turn) };
     }
     catch (error) {
         // A failed optional report cannot rewrite the already observed structured completion.
         if (isComputerUseAdmissionLimitError(error)) {
-            return record("skipped", "the adapter reported a local admission limit before provider dispatch; no closing request was sent");
+            return {
+                status: "skipped",
+                detail: "the adapter reported a local admission limit before provider dispatch",
+            };
         }
         const detail = signal?.aborted
             ? "cancelled"
@@ -126,14 +158,19 @@ async function debriefExchange(session, context, trigger, ask, { send, boundMs }
                     : String(error);
         const debriefReceipt = provider.requestPolicy === "fail_closed" ? session.usage.requests.at(-1) : undefined;
         const usageReported = debriefReceipt?.usageComplete === true && isCompleteTurnUsage(debriefReceipt.usage);
-        return record("failed", `${detail}; closing request usage is ${usageReported ? "reported" : "unreported"}`, usageReported);
+        return {
+            status: "failed",
+            detail: `${detail}; closing request usage is ${usageReported ? "reported" : "unreported"}`,
+            usageReported,
+        };
     }
     finally {
         clearTimeout(timer);
         scope.end();
     }
 }
-function acceptDebriefTurn(session, turn, trigger, ask, record) {
+/** Record a closing reply's usage, and say whether it was complete. */
+function recordClosingUsage(session, turn) {
     const { maxUsd, overRunBudget, estimateTurnCostUsd } = session.settings;
     session.usage.record(turn, "debrief");
     session.lastResponseId = turn.responseId ?? session.lastResponseId;
@@ -144,13 +181,16 @@ function acceptDebriefTurn(session, turn, trigger, ask, record) {
         (maxUsd !== undefined && finalEstimate != null && finalEstimate > maxUsd)) {
         session.trace.record("notice", () => notice("warn", "model budget reached during closing report", "The closing request crossed an estimated budget; no further requests or actions followed. Task completion is unchanged."));
     }
-    const usageReported = isCompleteTurnUsage(turn.usage) && turn.providerRequest?.usageComplete !== false;
+    return isCompleteTurnUsage(turn.usage) && turn.providerRequest?.usageComplete !== false;
+}
+/** Accept a closing reply as the debrief's result; `note` adds why no later reply replaced it. */
+function acceptDebriefTurn(session, turn, trigger, ask, record, usageReported, note = "") {
     if (turn.actions.length > 0 || turn.pendingSafetyChecks.length > 0) {
-        return record("failed", "the closing response requested actions or safety checks; none were executed and its report was not accepted", usageReported);
+        return record("failed", `the closing response requested actions or safety checks; none were executed and its report was not accepted${note}`, usageReported);
     }
     // First-party providers parse their replies already; these parses cover third-party ports.
     const { status, detail, ...added } = ask.read(session, turn, trigger);
-    return { ...record(status, detail, usageReported), ...added };
+    return { ...record(status, `${detail}${note}`, usageReported), ...added };
 }
 /**
  * A closing report: its summary and friction as one participant message, then its impressions.
@@ -205,6 +245,6 @@ function readImpressions(session, turn) {
     session.impressions = recordImpressions(session, impressions.data);
     return {
         status: "completed",
-        detail: "one read-only impressions request; no additional desktop actions; the participant's own ending is unchanged",
+        detail: "a read-only impressions request; no additional desktop actions; the participant's own ending is unchanged",
     };
 }

package/dist/actors/computer-use/loop/types.d.ts

@@ -203,6 +203,12 @@ export interface CuaTurn {
     closingReport?: ParticipantClosingReport;
     /** Present only for an accepted impressions-only closing reply. */
     impressions?: ParticipantImpression[];
+    /**
+     * Set on a closing reply that was not the JSON its request asked for, when the provider's next
+     * closing request would ask for it more strictly. The loop may then call the same closing method
+     * once more, after the checks the first request passed and with a time bound of its own.
+     */
+    closingRetry?: true;
 }
 /** The model side of the loop. Self-describes its identity and capabilities. */
 export interface CuaProvider {
@@ -246,7 +252,10 @@ export interface CuaProvider {
     /** Latest known usage of the continuing request. Used only for runtime spend guards. */
     readonly pendingRequestUsage?: CuaTurn["usage"];
     nextTurn(req: CuaTurnRequest, signal: AbortSignal, spend?: CuaSpendGate): Promise<CuaTurn>;
-    /** Optional read-only closing report. Implementations must disable tools and make no retries. */
+    /**
+     * Optional read-only closing report. Implementations must disable tools and send one request per
+     * call, with no retries of their own; a second request is the loop's call (`closingRetry`).
+     */
     debrief?: ((req: CuaTurnRequest, signal: AbortSignal) => Promise<CuaTurn>) | undefined;
     /**
      * Optional read-only request for impressions only, after the participant ended the session
@@ -319,12 +328,12 @@ export interface CuaLoopOptions {
     maxWaitMs?: number;
     /**
      * How long a `wait` that names no duration lasts, in ms, when its turn only waits or takes
-     * screenshots: the study's actor.idleWaitMs. It is also how often the loop looks at the screen
-     * while such a turn is watched, at most every 30000 ms when the executor is speechEnabled. After
-     * an action in the same turn such a wait lasts 500 ms. A whole number from 1000 to 600000 and no
-     * longer than an explicit maxWaitMs; any other value throws a RangeError before the session
-     * starts. Default 5000. The session never applies more than its maxWaitMs, and the trace records
-     * the applied lengths as `waitSettings`.
+     * screenshots: the study's actor.idleWaitMs. It is also the shortest such a turn is watched for,
+     * and how often the loop looks at the screen while it is watched, at most every 30000 ms when the
+     * executor is speechEnabled. After an action in the same turn such a wait lasts 500 ms. A whole
+     * number from 1000 to 600000 and no longer than an explicit maxWaitMs; any other value throws a
+     * RangeError before the session starts. Default 5000. The session never applies more than its
+     * maxWaitMs, and the trace records the applied lengths as `waitSettings`.
      */
     idleWaitMs?: number;
     /**

package/dist/actors/computer-use/loop/watch.js

@@ -50,10 +50,13 @@ export class IdleWatcher {
         if (plannedMs === 0)
             return undefined;
         const capMs = Math.max(plannedMs, session.waits.watchMaxMs);
-        // Never shorter than the turn's own planned length, so a long named wait is not cut short.
+        // Never shorter than idleWaitMs, so a participant naming a few milliseconds on a static screen
+        // does not send requests back to back, and never shorter than the turn's own planned length, so
+        // a long named wait is not cut short.
+        const shortestMs = Math.max(plannedMs, session.waits.idleWaitMs);
         const limitMs = this.grownFromMs === undefined
-            ? plannedMs
-            : Math.min(capMs, Math.max(plannedMs, 2 * this.grownFromMs));
+            ? shortestMs
+            : Math.min(capMs, Math.max(shortestMs, 2 * this.grownFromMs));
         const ending = await watchScreen(session, this.observer, turnNumber, reference, limitMs);
         // A small change that held still (a status label, or a counter the rule did not hold back)
         // wakes the participant and keeps the doubling, so a repeating small change cannot keep every
@@ -64,7 +67,7 @@ export class IdleWatcher {
             wake: ending.wake,
             ...(ending.observation === undefined ? {} : { observation: ending.observation }),
             ...(ending.stop === undefined ? {} : { stop: ending.stop }),
-            hint: hintFor(ending, grows && limitMs < capMs ? capMs : undefined),
+            hint: hintFor(ending),
             waits: {
                 ...splitWatch(lengths, limitMs, ending.ms),
                 sessionEnded: ending.stop !== undefined,
@@ -215,21 +218,18 @@ function splitWatch(lengths, limitMs, ms) {
     };
 }
 /**
- * The next request's note. A clock ending says nothing: the clock text that follows says it. A
+ * The next request's note. A clock ending says only how long the wait lasted. A
  * small change is named by its place, so a participant is not told the screen changed when what
- * changed was a counter or a timestamp. `growsToMs` adds the growth sentence when the next watch
- * doubles.
+ * changed was a counter or a timestamp. The note says nothing of the next wait's length: a
+ * participant told it names that length and keeps waiting.
  */
-function hintFor(ending, growsToMs) {
+function hintFor(ending) {
     const lasted = `Your wait lasted ${duration(ending.ms)}`;
-    const grows = growsToMs === undefined
-        ? ""
-        : ` A wait right after it can last twice as long, at most ${duration(growsToMs)}.`;
     const { seen } = ending;
     switch (ending.wake) {
         case "changed":
             return seen?.verdict === "settled" && seen.small
-                ? `${lasted} and ended when a small part of the screen changed and then stayed the same for over a minute, ${place(seen.area)}.${grows}`
+                ? `${lasted} and ended when a small part of the screen changed and then stayed the same for over a minute, ${place(seen.area)}.`
                 : `${lasted} and ended when the screen changed.`;
         case "speech":
             return `${lasted} and ended when you heard speech.`;
@@ -239,13 +239,15 @@ function hintFor(ending, growsToMs) {
         // never says the screen stayed the same.
         case "limit":
             if (seen?.verdict === "changing" && seen.lookSoon)
-                return `${lasted}; a large part of the screen was changing at its last check.${grows}`;
+                return `${lasted}; a large part of the screen was changing at its last check.`;
             if (seen?.verdict !== "settled" && seen?.held !== undefined)
-                return `${lasted}; a small part of the screen changed and stayed the same, ${place(seen.held)}. A small change does not end a wait.${grows}`;
-            return `${lasted}; no change on the screen held still long enough to end it.${grows}`;
+                return `${lasted}; a small part of the screen changed and stayed the same, ${place(seen.held)}. A small change does not end a wait.`;
+            return `${lasted}; no change on the screen held still long enough to end it.`;
         case "stalled":
             return `${lasted}; the screen could not be checked during it.`;
+        // The clock text that follows says why the wait ended.
         case "clock":
+            return `${lasted}.`;
         case "stopped":
             return undefined;
     }

package/dist/actors/computer-use/openai-context.js

@@ -4,16 +4,17 @@ import { asArray, asRecord, inputMessage, openAiActionToCua } from "./openai-wir
 // provider still threads on previous_response_id feed it too, so a switch partway through a
 // session keeps everything before the switch.
 //
-// The Responses API pairs items: every computer_call needs its computer_call_output, and a
-// reasoning item travels with the item it preceded. The conversation is therefore cut only at
-// exchange boundaries: one reply's output items together with the items that answered them.
-// When the estimate passes the budget, the conversation is cut down to half the budget in one
-// step: the opening screenshot goes first, then the oldest exchanges become lines of a progress
-// note that keeps their reasoning summaries, messages and actions, and the text that answered
-// them, such as a note that an action was not run. The note is an assistant message, so text the
-// model wrote (which can quote a web page) keeps the trust it had when the model wrote it. The newest MIN_KEPT_EXCHANGES exchanges are never collapsed, so a
-// request passes the budget when they alone exceed it; the trace's per-request estimate shows when
-// that happens.
+// The Responses API pairs items: every computer_call needs its computer_call_output, every
+// function_call (wait_minutes) its function_call_output, and a reasoning item travels with the
+// item it preceded. The conversation is therefore cut only at exchange boundaries: one reply's
+// output items together with the items that answered them. When the estimate passes the budget, the
+// conversation is cut down to half the budget in one step: the opening screenshot goes first, then
+// the oldest exchanges become lines of a progress note that keeps their reasoning summaries,
+// messages, actions and function calls, and the text that answered them, such as a note that an
+// action was not run or how a wait ended. The note is an assistant message, so text the model wrote
+// (which can quote a web page) keeps the trust it had when the model wrote it. The newest
+// MIN_KEPT_EXCHANGES exchanges are never collapsed, so a request passes the budget when they alone
+// exceed it; the trace's per-request estimate shows when that happens.
 //
 // A cut rewrites the start of the prompt, so the provider's prompt cache misses on the request
 // after it. Between cuts nothing already sent changes: each request starts with the whole of the
@@ -156,12 +157,15 @@ function describeExchange(exchange) {
     const said = [];
     const did = [];
     const told = [];
-    // The answers are the call outputs, whose content is a screenshot, and hint messages.
+    // The answers are the computer_call outputs, whose content is a screenshot, the function call
+    // outputs, which say how a wait ended as text beside any screenshot, and hint messages.
     for (const raw of exchange.answers) {
         const item = asRecord(raw);
         if (item.type === "computer_call_output")
             continue;
-        for (const entry of asArray(item.content)) {
+        if (typeof item.output === "string" && item.output.length > 0)
+            told.push(item.output);
+        for (const entry of [...asArray(item.content), ...asArray(item.output)]) {
             const text = asRecord(entry).text;
             if (typeof text === "string" && text.length > 0)
                 told.push(text);
@@ -189,6 +193,11 @@ function describeExchange(exchange) {
                     did.push(describeAction(mapped));
             }
         }
+        // A function call is noted as it was made, such as wait_minutes {"minutes":3}.
+        if (item.type === "function_call" && typeof item.name === "string")
+            did.push(typeof item.arguments === "string"
+                ? `${item.name} ${clip(item.arguments, 120)}`
+                : item.name);
     }
     const lead = `Turn ${exchange.turn}: `;
     const before = [
@@ -224,14 +233,17 @@ function clip(text, limit) {
     const flat = text.replace(/\s+/g, " ").trim();
     return flat.length <= limit ? flat : `${flat.slice(0, limit - 3)}...`;
 }
-/** The data URLs of the images an input item carries. */
+/**
+ * The data URLs of the images an input item carries: a computer_call output's screenshot, and the
+ * image parts of a message's content or a function call output's output.
+ */
 function imagesOf(item) {
     const record = asRecord(item);
     const output = asRecord(record.output);
     const urls = [];
     if (typeof output.image_url === "string")
         urls.push(output.image_url);
-    for (const part of asArray(record.content)) {
+    for (const part of [...asArray(record.content), ...asArray(record.output)]) {
         const url = asRecord(part).image_url;
         if (typeof url === "string")
             urls.push(url);

package/dist/actors/computer-use/openai-conversation.d.ts

@@ -40,6 +40,10 @@ export declare class OpenAiConversation {
     private lastResponseId;
     /** computer_call ids the next request owes an output for. */
     private pendingCallIds;
+    /** The function calls the next request owes an output for. */
+    private pendingFunctionCalls;
+    /** Whether the latest reply acted, so its waits went to the desktop and were not watched. */
+    private pendingActed;
     /** Accepted replies so far; the next request is number `replies + 1`. */
     private replies;
     /** The whole conversation, which explicit-context requests carry within its token budget. */
@@ -48,12 +52,22 @@ export declare class OpenAiConversation {
     /** Undefined once the account or model rejected reasoning summaries. */
     private reasoningSummary;
     private readonly evidence;
+    /** The items of a closing request whose reply was not the stated JSON, for its strict retry. */
+    private openClosing;
     constructor(settings: OpenAiConversationSettings);
     /**
-     * The next request, carrying `req`'s screen: the opening request, or the outputs the latest
-     * reply's computer calls are owed. A closing request asks for strict JSON and no tools.
+     * The next participant turn's request, carrying `req`'s screen: the opening request, or the
+     * outputs the latest reply's computer and function calls are owed.
      */
-    request(req: CuaTurnRequest, closing?: ClosingRequest): ConversationRequest;
+    request(req: CuaTurnRequest): ConversationRequest;
+    /**
+     * The read-only closing request carrying `req`'s screen, sent once with `send`. It goes out
+     * without text.format, which is part of the cached prefix, so it reads the session from the
+     * cache. A completed reply that is not JSON the schema accepts comes back marked closingRetry and
+     * leaves the conversation where it was, so the next closing request is the same one with the
+     * strict schema set. Returns the reply's turn with the account it gave.
+     */
+    close(req: CuaTurnRequest, closing: ClosingRequest, send: (body: Record<string, unknown>) => Promise<unknown>): Promise<CuaTurn>;
     /**
      * The organization rejected server-side state: carry the conversation on the client from the
      * request being sent, and record which answer made the switch. False when the conversation is
@@ -69,9 +83,20 @@ export declare class OpenAiConversation {
     record(): ActorConversation;
     /**
      * The items a request adds to the conversation: the opening message with the first screen, or
-     * an output for each pending computer_call with the latest screenshot and any hint.
+     * an output for each pending call with the latest screenshot, and any hint. Each computer_call
+     * output carries the screenshot. Function call outputs follow in reply order: with no
+     * computer_call the first carries the screenshot, and every other one carries text only, so the
+     * screenshot is sent once. A turn's hint goes into the first function call output, so it reads
+     * as how the wait ended; a closing request's hint asks for the closing account and stays a user
+     * message. A call to a tool that does not exist gets an output that says so (callNote).
      */
     private newItems;
+    /**
+     * What a call's output says when no note does, so every reply says how the wait went: a wait
+     * among actions ran for its length (a shortened or unsent one has a note), and a further call in
+     * a reply of only waits was part of the one watch the first call's output reports.
+     */
+    private waitOutcome;
     private context;
     /**
      * A participant request: the opening request, or a continuation carrying `sent`. A threaded
@@ -79,12 +104,17 @@ export declare class OpenAiConversation {
      * ahead of `sent`.
      */
     private turnBody;
-    /** A read-only closing request: no tools, its own output limit, a strict schema. */
+    /**
+     * A read-only closing request: the next turn's body with tools off and its own output limit,
+     * and the strict schema only when `schema` is set.
+     */
     private closingBody;
     /**
-     * Move the conversation on to a reply to a request that added `sent`. A participant reply cut
-     * off by the output limit is set aside, so the next request is the same one again
-     * (outputLimitRetry); its actions never run, so no call output is owed for them.
+     * A participant turn's reply to a request that added `sent`. A reply cut off by the output
+     * limit is set aside, so the next request is the same one again (outputLimitRetry); its actions
+     * never run, so no call output is owed for them.
      */
     private accept;
+    /** Move the conversation on to a reply to a request that added `sent`. */
+    private moveOn;
 }