@isomorph.ai/app-sdk 1.38.1 → 1.40.0

package/README.md

@@ -397,9 +397,10 @@ const computerUse = createComputerUseTool({ runKey: ctx.runId ?? `my-job/${ctx.s
 ```
 
 - The tool calls the task API at `ISOMORPH_COMPUTER_USE_URL` (region `ISOMORPH_COMPUTER_USE_REGION`, default `us-east-1`); Provision sets both when the cell has the API. Each request is signed with SigV4 (`execute-api`) as the runtime's own role: the Lambda environment's credentials, or on a cluster pod its ServiceAccount role through STS `AssumeRoleWithWebIdentity`.
-- It starts the task with `POST /tasks`, then checks `POST /tasks/check` every 5 s until the task is `done` or `failed`.
+- It starts the task with `POST /tasks`, then checks `POST /tasks/check` every 5 s until the task is `done`, `failed` or `waiting_for_input`. Waiting returns immediately; end the app job rather than holding a worker while a person signs in.
 - It calls `progress(step count, note)` on every new step and at least once a minute, so the job's lease does not run out.
 - When `cancelled()` turns true it sends `POST /tasks/cancel` once and keeps checking until the task has `failed`.
+- While the task is `queued` and the CUA reports `waiting` (no computer is free), the progress note says so with the minutes waited. After `maxQueueMs` (20 minutes unless set) the tool cancels the task and returns `failed` with `No computer was free within N minutes; the task was cancelled before it started.`
 - One task record per `runKey`: a retried job run sends the same key and gets the queued, running or finished task back. Only a failed task starts a new attempt.
 - `finalUrl` is the page the browser ended on. A website's sign-in page there means a saved sign-in has expired: ask the person to sign in again.
 
@@ -407,13 +408,56 @@ Options the app passes, never the model:
 
 - `files`: PDFs the task may upload, at most five and 2 MiB in all.
 - `session`: the name of a runtime secret a sign-in was saved under (`finishSignIn`'s `saveAs`, below). The task starts signed in and can use only that sign-in's websites. When the task is `done` the refreshed session is saved back to the same secret. A missing one fails with `No saved sign-in named <name>.` Run one task per saved sign-in at a time: two tasks would both save back, and some websites sign out the older copy.
-- `secrets`: `{ name: value }`, at most 10, names `^[A-Za-z][A-Za-z0-9_-]{0,63}$`, values up to 4096 bytes. The computer types a value into a password field by its name; the model sees only the names. Never put a password or any other credential in the task text; pass it here and say "sign in with the `password` secret".
+- `secrets`: `{ name: value }`, at most 10, names `^[A-Za-z][A-Za-z0-9_-]{0,63}$`, values up to 4096 bytes. The computer types a value into a password field by its name, and only there: it refuses to type a secret into a visible field, since the model reads the page after every step. The model sees only the names. Never put a password or any other credential in the task text; pass it here and say "sign in with the `password` secret". A user ID, customer ID or email is not a secret: give it in the task text.
+- `maxQueueMs`: how long the task may wait for a free computer before the tool cancels it (20 minutes unless set).
 - `allowedHosts`: the websites (1 to 10 lower-case hostnames) a task with `secrets` and no `session` may use. A saved sign-in carries its own.
 - `onDownload({ name, contentType, size, blob })`: called once for each file a `done` task downloaded, one at a time, before the tool returns (at most 10 files, 100 MiB each). Store the `blob` yourself; the CUA keeps the files for a day at most. Without it nothing is fetched. A link that does not answer 200 fails the tool. The model only sees `downloads: [{ name, size }]`.
 
 A task with `session` or `secrets` works the page through clicks and typing only. `connectionId` is gone: pass `session`.
 
-(1.25.0, 1.36.0, 1.37.0; data plane ADR 0078, ADR 0090, ADR 0091)
+(1.25.0, 1.36.0, 1.37.0, 1.39.0; data plane ADR 0078, ADR 0090, ADR 0091, ADR 0092)
+
+## Help on the task's existing computer
+
+`dist/computer-use-handoff.js` adds short server calls for a paused task. It does not start another computer and does not import `ai`. It requires `runKey` and `authorize`, a server callback checked before **every** call. The callback must check the current authenticated person's access to the app's task row. An app-role signature, client-supplied task ID or pause ID alone does not prove that person owns the task.
+
+```text
+Job's browser tool → waiting_for_input → job ends
+                                             │
+App action → authorized screen/input or private login
+                                             │
+User/app explicitly requests resume → new job attaches to original runKey
+```
+
+The app keeps the original `runKey`, task ownership and returned pause metadata in its existing task row. Neither model nor browser chooses the backend `runKey`. For example, in an action backed by an app-owned `computer_tasks` table with `id`, `owner_id` and `run_key` columns:
+
+```ts
+import { createClient } from '@isomorph.ai/app-sdk';
+import { createComputerUseHandoff } from '@isomorph.ai/app-sdk/dist/computer-use-handoff.js';
+
+export default async function viewComputer(input: { taskId: string; pauseId: string }, ctx: { user: { id: string } }) {
+  const app = createClient();
+  const owned = async () => (await app.data.from('computer_tasks').select('run_key')
+    .eq('id', input.taskId).eq('owner_id', ctx.user.id).single()).data;
+  const task = await owned();
+  if (!task) throw new Error('Unknown task.');
+  const computer = createComputerUseHandoff({ runKey: task.run_key, authorize: async () => Boolean(await owned()) });
+  return computer.view(input.pauseId);
+}
+```
+
+Keep that table protected by the app's user row rules. A trusted action must still check `owner_id` against `ctx.user.id`; never use workload identity as proof of human ownership.
+
+- `requestTakeover(reason, requiredInput?)` requests a pause; `requiredInput` is `human` (default) or `login`. Poll `check()` until `waiting_for_input` gives `taskId`, `pauseId`, `reason`, `requiredInput` and optional login result.
+- `view(pauseId)` returns `{status:'pausing'}` or `{status:'ready', screenshot, url}`. Reuse the existing sign-in screenshot UI: display the base64 PNG as an image, map clicks to image pixels, and forward keyboard/scroll events through app actions. Viewing does not itself resume the agent.
+- `input(pauseId, action)` sends the same `SignInAction` click/type/key/scroll commands used by `sendSignInInput`. The CUA accepts human input only while the task is paused; it blocks the helper and main agent from acting at the same time.
+- `completeLogin(pauseId, {allowedHosts, username, password})`, or `completeLogin(pauseId, {otp})`, queues the separate browser-action-only login agent on this computer. **Call only from private server code, never expose this method or credentials as model tool inputs, prompts, progress, logs or job input/results.** Poll `check()` for `login.status: success | needs_input | human_help_needed`, with `input:'otp'` when another code is needed. Helper success leaves the main agent paused.
+- `resume(pauseId)` explicitly resumes; it does not wait for completion. On Continue, call this from the authorized app action, then enqueue the app's continuation job. That job creates `createComputerUseTool({...jobOptions, runKey: originalRunKey, attach:true})`: it polls the same task without sending `/tasks` again. Keep the original saved-session and download options if needed. Stale/duplicate resumes fail with 409; do not silently retry with a new pause ID. If enqueue fails, retry attaching to the already resumed task, not another resume.
+- `cancel()` stops the task, including while waiting.
+
+The outer agent may use app-defined tools for takeover and opening the authorized view. Resume requires an explicit user or app decision. Keep any outer-agent conversation in the app's existing job/task storage when ending a waiting job; this SDK does not persist the outer model's messages. The CUA persists its own conversation/resume state under the same task folder as screenshots and progress in its existing result bucket, without passwords or OTPs.
+
+P0 `getSavedLogin(accountRef, website)` and `getOtp(taskId, website)` return `{status:'unavailable'}` and call no integration. Use human takeover when no approved backend input is supplied. Governance saved-login/website storage and email/SMS OTP lookup are P0.5, not part of this SDK change. These task helpers require the matching CUA handoff release; existing new-computer sign-in helpers below remain separate.
 
 ## Browser sign-in for computer use
 
@@ -431,7 +475,7 @@ await sendSignInInput(handle, { op: 'click', x: 640, y: 400 });
 await finishSignIn(handle, { saveAs: `vendor-central/${ctx.user.id}` }); // { url }
 ```
 
-- `startSignIn` starts a computer that may open only `allowedHosts` and returns an opaque handle for the calls that follow; the computer answers only this app's role and stops after 30 minutes.
+- `startSignIn` starts a computer that may open only `allowedHosts` and returns an opaque handle for the calls that follow; the computer answers only this app's role and stops after 30 minutes. It opens `startUrl`, an HTTPS page on one of those websites, or the first website's front page: an account on `accounts.zoho.in` lists that host first or passes it as `startUrl`.
 - `showSignIn` gives `{ status: 'starting' }` until the browser is up, then `{ status: 'ready', screenshot, url }`: a base64 PNG of the 1280×800 screen and the page's address.
 - `sendSignInInput` takes `{ op: 'click', x, y }` (screen pixels), `{ op: 'typeText', value }` (up to 512 characters), `{ op: 'pressKey', key }` (such as `Enter` or `Tab`) or `{ op: 'scroll', direction: 'up' | 'down' }`.
 - `finishSignIn` saves the session (cookies and site storage of the allowed websites, at most 60 KB) as the runtime secret `saveAs`, stops the computer and returns the page it was on. App code never holds the cookies.
@@ -652,3 +696,61 @@ keep the workflow run ID and tool-call ID stable across retries. Direct writes
 must supply their own persisted UUID `idempotencyKey`. A tool marked read-only
 does not require a key. An incomplete replay requires reconciliation, not a new
 write. Account consent and app permissions are enforced by the platform.
+
+### Tools for the outer app agent
+
+`createComputerUseHandoffTools` from `dist/computer-use-handoff-tool.js` creates
+`request_computer_input` and `resume_computer_task`. Register these beside the
+existing computer-use tool. Bind them to the original task's `runKey` in server
+code; the model supplies only the current `pauseId`.
+
+```ts
+import { generateText, hasToolCall, stepCountIs } from 'ai';
+import { createComputerUseHandoffTools } from '@isomorph.ai/app-sdk/dist/computer-use-handoff-tool.js';
+
+const handoffTools = createComputerUseHandoffTools({
+  runKey: task.run_key,
+  progress: ctx.progress,
+  cancelled: ctx.cancelled,
+  authorize: () => userOwnsTask(ctx.user.id, task.id),
+  onInputRequired: (pause, page) => showComputerInApp(task.id, pause, page),
+  canResume: pauseId => userPressedContinue(ctx.user.id, task.id, pauseId),
+});
+const result = await generateText({
+  model,
+  messages: savedMessages,
+  tools: { computer_use: computerUse, ...handoffTools },
+  stopWhen: [stepCountIs(10), hasToolCall('request_computer_input')],
+});
+await saveOuterMessages(task.id, [...savedMessages, ...result.response.messages]);
+```
+
+The ownership, screen, Continue and conversation functions in this example are
+app-provided functions, not SDK exports. The app must save the completed tool
+result before returning its waiting response to the UI. The stop condition ends
+the outer loop after the input request; creating tools alone does not stop a loop.
+
+The request tool checks the current pause and calls `view()`. Its callback receives
+the screenshot privately, so the app can show its existing computer view. The
+model receives `human_input_required`, task/pause references, reason and required
+input, without screenshot bytes or a computer endpoint. Screen refreshes and human
+input still use authorized backend `view()` and `input()` calls.
+
+When the human presses Continue, the app records approval for that user, task and
+pause and queues its outer-agent continuation with the saved conversation and
+original run key. Only this trusted user action may make `canResume` return true;
+never derive approval from model arguments or a chat string. Retain that approval
+for retries of the same continuation. A new pause needs new approval. The resume
+tool checks approval, calls resume and follows the existing task with `attach:true`.
+If an earlier attempt already resumed the task, it attaches without resuming twice.
+It may return another waiting result; use the input tool again for that new pause.
+Credentials and OTPs remain private backend `completeLogin()` inputs, not model tools.
+
+`onMessage` on the computer-use tool receives allowlisted CUA progress/handoff
+messages when they change; the progress callback also receives their text. Waiting
+results and handoff `check()` preserve the latest valid `message`. This is latest
+message delivery, not a full message history.
+
+This SDK does not supply an app-specific screen, conversation database or job
+scheduler. It supplies the tools and waiting contract; the example wires them to
+an app's existing actions, task rows, screen and job runner.

package/dist/computer-use-handoff-tool.d.ts

@@ -0,0 +1,590 @@
+import { type ComputerUseHandoffOptions, type ComputerUsePage, type ComputerUseWaiting } from './computer-use-handoff.js';
+import { type ComputerUseOptions } from './computer-use.js';
+export interface ComputerUseHandoffToolOptions extends ComputerUseHandoffOptions, ComputerUseOptions {
+    onInputRequired(input: ComputerUseWaiting, page: ComputerUsePage): Promise<void> | void;
+    canResume(pauseId: string): Promise<boolean> | boolean;
+}
+export declare function createComputerUseHandoffTools(options: ComputerUseHandoffToolOptions): {
+    request_computer_input: ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type?: undefined | "function";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type: "dynamic";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: false;
+        supportsDeferredResults?: never;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: true;
+        supportsDeferredResults?: boolean;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    });
+    resume_computer_task: ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type?: undefined | "function";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type: "dynamic";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: false;
+        supportsDeferredResults?: never;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: true;
+        supportsDeferredResults?: boolean;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    });
+};

package/dist/computer-use-handoff-tool.js

@@ -0,0 +1,45 @@
+import { jsonSchema, tool } from 'ai';
+import { createComputerUseHandoff } from './computer-use-handoff.js';
+import { createComputerUseTool } from './computer-use.js';
+export function createComputerUseHandoffTools(options) {
+    if (typeof options.onInputRequired !== 'function' || typeof options.canResume !== 'function') {
+        throw new Error('Computer use handoff tools need onInputRequired and canResume callbacks.');
+    }
+    const handoff = createComputerUseHandoff(options);
+    const schema = jsonSchema({
+        type: 'object', properties: { pauseId: { type: 'string', minLength: 1 } }, required: ['pauseId'], additionalProperties: false,
+    });
+    return {
+        request_computer_input: tool({
+            description: 'Show the paused task computer to the user and request outside input. Use the current pauseId from the computer-use result. This returns human_input_required: end this agent turn and wait for the user. Never ask for passwords or one-time codes in chat.',
+            inputSchema: schema,
+            execute: async ({ pauseId }) => {
+                const input = await handoff.check();
+                if (input.status !== 'waiting_for_input' || !('pauseId' in input) || input.pauseId !== pauseId)
+                    throw new Error('The task is not waiting at this pause.');
+                const page = await handoff.view(pauseId);
+                await options.onInputRequired(input, page);
+                return { ...input, status: 'human_input_required' };
+            },
+        }),
+        resume_computer_task: tool({
+            description: 'Resume the original computer-use task only after the user has supplied input and explicitly pressed Continue. Server approval is required. Follow that same task until it finishes or needs more input; never start a new task.',
+            inputSchema: schema,
+            execute: async ({ pauseId }, execution) => {
+                const current = await handoff.check();
+                if (await options.canResume(pauseId) !== true)
+                    throw new Error('The user has not approved continuing this pause.');
+                if (current.status === 'waiting_for_input') {
+                    if (!('pauseId' in current) || current.pauseId !== pauseId)
+                        throw new Error('The task is waiting at a different pause.');
+                    await handoff.resume(pauseId);
+                }
+                else if (!['resume_starting', 'resume_queued', 'resuming', 'running', 'done', 'failed'].includes(current.status)) {
+                    throw new Error('The task cannot continue in its current state.');
+                }
+                const follow = createComputerUseTool({ ...options, attach: true });
+                return follow.execute({ task: 'Continue the original task.' }, execution);
+            },
+        }),
+    };
+}

package/dist/computer-use-handoff.d.ts

@@ -0,0 +1,78 @@
+import { type ComputerUseTransport } from './computer-use-api.js';
+import type { SignInAction } from './computer-use-sign-in.js';
+export interface ComputerUseMessage {
+    type: 'progress' | 'handoff';
+    text: string;
+    required_input?: 'login' | 'human';
+}
+export declare function computerUseMessage(value: unknown): ComputerUseMessage | undefined;
+export interface ComputerUseWaiting {
+    status: 'waiting_for_input';
+    taskId: string;
+    pauseId: string;
+    reason: string;
+    requiredInput: 'login' | 'otp' | 'human';
+    message?: ComputerUseMessage;
+    login?: {
+        status: 'success' | 'needs_input' | 'human_help_needed';
+        input?: 'otp';
+    };
+}
+export declare function waitingInput(answer: {
+    taskId?: unknown;
+    pauseId?: unknown;
+    reason?: unknown;
+    requiredInput?: unknown;
+    login?: unknown;
+    message?: unknown;
+}): ComputerUseWaiting;
+export type ComputerUsePage = {
+    status: 'pausing';
+} | {
+    status: 'ready';
+    screenshot: string;
+    url: string;
+};
+export type ComputerUseLoginInput = {
+    allowedHosts: readonly string[];
+    username: string;
+    password: string;
+    otp?: never;
+} | {
+    otp: string;
+    allowedHosts?: never;
+    username?: never;
+    password?: never;
+};
+export interface ComputerUseHandoffOptions extends ComputerUseTransport {
+    runKey: string;
+    authorize(): Promise<boolean> | boolean;
+}
+export declare function createComputerUseHandoff(options: ComputerUseHandoffOptions): {
+    requestTakeover(reason: string, requiredInput?: "login" | "human"): Promise<{
+        status: string;
+    }>;
+    view(pauseId: string): Promise<ComputerUsePage>;
+    input(pauseId: string, action: SignInAction): Promise<{
+        status: "accepted";
+    }>;
+    completeLogin(pauseId: string, input: ComputerUseLoginInput): Promise<{
+        status: "login_queued";
+    }>;
+    check(): Promise<ComputerUseWaiting | {
+        message?: ComputerUseMessage | undefined;
+        status: string;
+    }>;
+    resume(pauseId: string): Promise<{
+        status: "resume_queued";
+    }>;
+    cancel(): Promise<{
+        status: string;
+    }>;
+};
+export declare function getSavedLogin(_accountRef: string, _website: string): Promise<{
+    status: 'unavailable';
+}>;
+export declare function getOtp(_taskId: string, _website: string): Promise<{
+    status: 'unavailable';
+}>;

package/dist/computer-use-handoff.js

@@ -0,0 +1,109 @@
+import { allowedHostsOf, computerUseApi, refused } from './computer-use-api.js';
+export function computerUseMessage(value) {
+    if (!value || typeof value !== 'object')
+        return undefined;
+    const message = value;
+    if (!['progress', 'handoff'].includes(message.type) || typeof message.text !== 'string' || !message.text.trim() || message.text.length > 500)
+        return undefined;
+    return { type: message.type, text: message.text,
+        ...(message.type === 'handoff' && ['login', 'human'].includes(message.required_input ?? '') ? { required_input: message.required_input } : {}) };
+}
+export function waitingInput(answer) {
+    if (typeof answer.taskId !== 'string' || !answer.taskId || typeof answer.pauseId !== 'string' || !answer.pauseId
+        || !['login', 'otp', 'human'].includes(String(answer.requiredInput)))
+        throw new Error('The computer use API returned an incomplete pause.');
+    const result = {
+        status: 'waiting_for_input', taskId: answer.taskId, pauseId: answer.pauseId,
+        reason: typeof answer.reason === 'string' ? answer.reason : 'Outside input needed.',
+        requiredInput: answer.requiredInput,
+    };
+    const message = computerUseMessage(answer.message);
+    if (message)
+        result.message = message;
+    const login = answer.login;
+    if (login && ['success', 'needs_input', 'human_help_needed'].includes(login.status)) {
+        result.login = { status: login.status, ...(login.input === 'otp' ? { input: 'otp' } : {}) };
+    }
+    return result;
+}
+export function createComputerUseHandoff(options) {
+    if (typeof options.runKey !== 'string' || !options.runKey || typeof options.authorize !== 'function')
+        throw new Error('Computer use handoff needs runKey and an authorize check for the current user.');
+    const api = computerUseApi(options);
+    const post = async (path, body = {}, privateInput = false) => {
+        if (await options.authorize() !== true)
+            throw new Error('You cannot access this computer use task.');
+        let answer;
+        try {
+            answer = await api.call(path, { run_key: options.runKey, ...body });
+        }
+        catch (error) {
+            if (privateInput)
+                throw new Error('The computer use login request could not be completed.');
+            throw error;
+        }
+        if (answer.status !== 200)
+            throw refused(path, answer.status, privateInput ? undefined : answer.body);
+        return answer.body;
+    };
+    const paused = (pauseId) => {
+        if (!pauseId || typeof pauseId !== 'string')
+            throw new Error('Computer use handoff needs the current pauseId.');
+        return { pauseId };
+    };
+    return {
+        requestTakeover(reason, requiredInput = 'human') {
+            if (!reason.trim() || !['login', 'human'].includes(requiredInput))
+                throw new Error('Takeover needs a reason and login or human input.');
+            return post('/tasks/takeover', { reason, requiredInput });
+        },
+        async view(pauseId) {
+            const page = await post('/tasks/view', paused(pauseId));
+            if (page.status === 'pausing')
+                return { status: 'pausing' };
+            if (page.status !== 'ready' || typeof page.screenshot !== 'string' || typeof page.url !== 'string')
+                throw new Error('The computer use API returned an incomplete screen.');
+            return { status: 'ready', screenshot: page.screenshot, url: page.url };
+        },
+        input(pauseId, action) {
+            return post('/tasks/input', { ...paused(pauseId), action });
+        },
+        completeLogin(pauseId, input) {
+            let body;
+            if ('otp' in input && input.otp !== undefined) {
+                if (Object.keys(input).some(key => key !== 'otp') || !validSecret(input.otp))
+                    throw new Error('Pass only a non-empty OTP of at most 4096 bytes.');
+                body = { otp: input.otp };
+            }
+            else {
+                if (!validSecret(input.username) || !validSecret(input.password))
+                    throw new Error('Login needs a non-empty username and password of at most 4096 bytes.');
+                body = { allowedHosts: allowedHostsOf(input.allowedHosts), username: input.username, password: input.password };
+            }
+            return post('/tasks/login', { ...paused(pauseId), ...body }, true)
+                .then(() => ({ status: 'login_queued' }));
+        },
+        async check() {
+            const answer = await post('/tasks/check');
+            if (answer.status === 'waiting_for_input')
+                return waitingInput(answer);
+            const message = computerUseMessage(answer.message);
+            return { status: String(answer.status), ...(message ? { message } : {}) };
+        },
+        resume(pauseId) {
+            return post('/tasks/resume', paused(pauseId));
+        },
+        cancel() {
+            return post('/tasks/cancel');
+        },
+    };
+}
+function validSecret(value) {
+    return typeof value === 'string' && value.length > 0 && Buffer.byteLength(value) <= 4096;
+}
+export async function getSavedLogin(_accountRef, _website) {
+    return { status: 'unavailable' };
+}
+export async function getOtp(_taskId, _website) {
+    return { status: 'unavailable' };
+}

package/dist/computer-use-sign-in.d.ts

@@ -22,9 +22,13 @@ export type SignInPage = {
     screenshot: string;
     url: string;
 };
-/** Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. */
-export declare function startSignIn({ allowedHosts, ...transport }: {
+/**
+ * Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. The
+ * computer opens `startUrl`, an HTTPS page on one of those websites, or the first website's front page.
+ */
+export declare function startSignIn({ allowedHosts, startUrl, ...transport }: {
     allowedHosts: readonly string[];
+    startUrl?: string;
 } & ComputerUseTransport): Promise<string>;
 export declare function showSignIn(handle: string, transport?: ComputerUseTransport): Promise<SignInPage>;
 export declare function sendSignInInput(handle: string, action: SignInAction, transport?: ComputerUseTransport): Promise<void>;

package/dist/computer-use-sign-in.js

@@ -5,6 +5,19 @@
 // jobs. The computer answers only the app role that started it.
 import { allowedHostsOf, computerUseApi } from './computer-use-api.js';
 import { runtimeSecretLocation, runtimeSecrets } from './secrets.js';
+function startUrlOf(startUrl, hosts) {
+    let url;
+    try {
+        url = new URL(String(startUrl));
+    }
+    catch {
+        url = undefined;
+    }
+    if (typeof startUrl !== 'string' || !url || url.protocol !== 'https:' || url.username || url.password || !hosts.includes(url.hostname)) {
+        throw new Error('startUrl must be an HTTPS address on one of the allowedHosts.');
+    }
+    return startUrl;
+}
 function computerOf(handle) {
     let computer;
     try {
@@ -17,10 +30,13 @@ function computerOf(handle) {
         throw new Error('This is not a sign-in handle from startSignIn.');
     return { computerId: computer.computerId, endpoint: computer.endpoint };
 }
-/** Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. */
-export async function startSignIn({ allowedHosts, ...transport }) {
+/**
+ * Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. The
+ * computer opens `startUrl`, an HTTPS page on one of those websites, or the first website's front page.
+ */
+export async function startSignIn({ allowedHosts, startUrl, ...transport }) {
     const hosts = allowedHostsOf(allowedHosts);
-    const { computerId, endpoint } = await computerUseApi(transport).post('/computers/start', { allowedHosts: hosts });
+    const { computerId, endpoint } = await computerUseApi(transport).post('/computers/start', { allowedHosts: hosts, ...(startUrl === undefined ? {} : { startUrl: startUrlOf(startUrl, hosts) }) });
     return Buffer.from(JSON.stringify({ computerId, endpoint })).toString('base64url');
 }
 export async function showSignIn(handle, transport = {}) {

package/dist/computer-use.d.ts

@@ -1,3 +1,4 @@
+import { type ComputerUseWaiting, type ComputerUseMessage } from './computer-use-handoff.js';
 export interface ComputerUseFile {
     name: string;
     contentType: "application/pdf";
@@ -21,24 +22,32 @@ export interface ComputerUseOptions {
     allowedHosts?: readonly string[];
     /**
      * Values the computer types into a password field by name, at most 10; the model sees only the names.
-     * Never put a credential in the task text.
+     * Never put a credential in the task text. A user ID, customer ID or email is not a secret: the
+     * computer refuses to type a secret into a visible field, so give those in the task text.
      */
     secrets?: Readonly<Record<string, string>>;
     /** The app's code gets each file the task downloaded, one at a time, when the task is done. Never seen by the model. */
     onDownload?(download: ComputerUseDownload): Promise<void> | void;
     /** The job's runId, or `<job name>/<scheduledAt>` for a scheduled job: one task record per job run. */
     runKey: string;
+    attach?: boolean;
+    onMessage?(message: ComputerUseMessage): Promise<void> | void;
     /** The job's progress: called on each new step and at least once a minute, which renews the job's lease. */
     progress(percent: number, note?: string): Promise<void> | void;
     /** The job's cancelled(): when true the tool cancels the task and waits until the CUA stops. */
     cancelled(): Promise<boolean> | boolean;
     env?: NodeJS.ProcessEnv;
     fetch?: typeof fetch;
+    /**
+     * How long the task may wait for a free computer before the tool cancels it and returns `failed`;
+     * 20 minutes unless set. The wait is reported through `progress` while it lasts.
+     */
+    maxQueueMs?: number;
     /** The pause between checks; the tests shorten it. */
     pollIntervalMs?: number;
     now?: () => number;
 }
-export interface ComputerUseResult {
+export interface ComputerUseFinished {
     status: 'done' | 'failed';
     summary: string | null;
     /** The page the browser was on when the task ended; a sign-in page there means the saved sign-in has expired. */
@@ -49,6 +58,7 @@ export interface ComputerUseResult {
         size: number;
     }[];
 }
+export type ComputerUseResult = ComputerUseFinished | ComputerUseWaiting;
 export declare function createComputerUseTool(options: ComputerUseOptions): ({
     deferLoading?: boolean;
     title?: string;

package/dist/computer-use.js

@@ -8,9 +8,20 @@ import { setTimeout as sleep } from 'node:timers/promises';
 import { jsonSchema, tool } from 'ai';
 import { allowedHostsOf, computerUseApi, refused } from './computer-use-api.js';
 import { runtimeSecretLocation, runtimeSecrets } from './secrets.js';
+import { computerUseMessage, waitingInput } from './computer-use-handoff.js';
 const HEARTBEAT_MS = 60_000;
+const DEFAULT_MAX_QUEUE_MS = 20 * 60_000;
 // Checks the API did not answer (a throttle, a 5xx, no connection) before the tool gives up: about a minute.
 const MAX_MISSED_CHECKS = 12;
+/** The progress note of a task that has not started: the wait and its reason when the CUA names one. */
+function queueNote(answer, waitedMs) {
+    if (answer.status !== 'queued' || !answer.waiting)
+        return `Computer use task ${answer.status}.`;
+    const minutes = Math.max(1, Math.round(waitedMs / 60_000));
+    return answer.waiting.reason === 'capacity'
+        ? `Computer use task queued: no computer is free yet (${minutes} min).`
+        : `Computer use task queued: ${answer.waiting.reason} (${minutes} min).`;
+}
 function secretsOf(secrets) {
     const entries = secrets && typeof secrets === 'object' && !Array.isArray(secrets) ? Object.entries(secrets) : [];
     if (entries.length < 1 || entries.length > 10)
@@ -72,18 +83,23 @@ export function createComputerUseTool(options) {
         }
     };
     return tool({
-        description: 'Do one task in a web browser, such as filling in and submitting a form on a website. Describe the whole task in plain words, with the website address and any details it needs. Waits until the task ends and returns its status and a summary. Files the task downloads go to the app; the result lists their names.',
+        description: 'Do one task in a web browser, such as filling in and submitting a form on a website. Describe the whole task in plain words, with the website address and any details it needs. Returns when the task ends or pauses for outside input. When waiting_for_input, ask the app or human for help; do not start another task or resume without their explicit request. Files the task downloads go to the app; the result lists their names.',
         inputSchema: jsonSchema({ type: 'object', properties: { task: { type: 'string', minLength: 1 } }, required: ['task'], additionalProperties: false }),
         execute: async ({ task }) => {
-            await api.post('/tasks', {
-                run_key: options.runKey, task, ...(files?.length ? { files } : {}), ...(session === undefined ? {} : { session: await savedSession(session) }),
-                ...(allowedHosts ? { allowedHosts } : {}), ...(secrets ? { secrets } : {}),
-            });
-            await options.progress(0, 'Computer use task started.');
+            if (!options.attach)
+                await api.post('/tasks', {
+                    run_key: options.runKey, task, ...(files?.length ? { files } : {}), ...(session === undefined ? {} : { session: await savedSession(session) }),
+                    ...(allowedHosts ? { allowedHosts } : {}), ...(secrets ? { secrets } : {}),
+                });
+            await options.progress(0, options.attach ? 'Computer use task reattached.' : 'Computer use task started.');
+            const queuedAt = now();
+            const maxQueueMs = options.maxQueueMs ?? DEFAULT_MAX_QUEUE_MS;
             let reported = now();
             let steps = 0;
             let cancelSent = false;
+            let queueExpired = false;
             let missed = 0;
+            let lastMessage = "";
             for (;;) {
                 if (!cancelSent && await options.cancelled()) {
                     await api.post('/tasks/cancel', { run_key: options.runKey });
@@ -99,7 +115,19 @@ export function createComputerUseTool(options) {
                 if (check?.status === 200) {
                     missed = 0;
                     const answer = check.body;
+                    const message = computerUseMessage(answer.message);
+                    if (message && JSON.stringify(message) !== lastMessage) {
+                        await options.onMessage?.(message);
+                        await options.progress(Math.min(steps, 99), message.text);
+                        reported = now();
+                        lastMessage = JSON.stringify(message);
+                    }
+                    if (answer.status === 'waiting_for_input' && !cancelSent)
+                        return waitingInput(answer);
                     if (answer.status === 'done' || answer.status === 'failed') {
+                        if (queueExpired && answer.status === 'failed') {
+                            return { status: 'failed', summary: `No computer was free within ${Math.round(maxQueueMs / 60_000)} minutes; the task was cancelled before it started.`, finalUrl: null, downloads: [] };
+                        }
                         if (answer.status === 'done' && session !== undefined && answer.session)
                             await runtimeSecrets.put(session, JSON.stringify(answer.session), transport);
                         const downloads = answer.downloads ?? [];
@@ -114,9 +142,14 @@ export function createComputerUseTool(options) {
                         }
                         return { status: answer.status, summary: answer.summary ?? null, finalUrl: answer.final_url ?? null, downloads: downloads.map(({ name, size }) => ({ name, size })) };
                     }
+                    if (answer.status === 'queued' && !cancelSent && now() - queuedAt >= maxQueueMs) {
+                        await api.post('/tasks/cancel', { run_key: options.runKey });
+                        cancelSent = true;
+                        queueExpired = true;
+                    }
                     if (answer.steps > steps || now() - reported >= HEARTBEAT_MS) {
                         steps = Math.max(steps, answer.steps);
-                        await options.progress(Math.min(steps, 99), answer.last_step ? `Computer use step ${steps}: ${answer.last_step}` : `Computer use task ${answer.status}.`);
+                        await options.progress(Math.min(steps, 99), answer.last_step ? `Computer use step ${steps}: ${answer.last_step}` : queueNote(answer, now() - queuedAt));
                         reported = now();
                     }
                 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@isomorph.ai/app-sdk",
-  "version": "1.38.1",
+  "version": "1.40.0",
   "type": "module",
   "repository": {
     "type": "git",