@isomorph.ai/app-sdk 1.39.0 → 1.41.0

package/README.md

@@ -397,7 +397,7 @@ const computerUse = createComputerUseTool({ runKey: ctx.runId ?? `my-job/${ctx.s
 ```
 
 - The tool calls the task API at `ISOMORPH_COMPUTER_USE_URL` (region `ISOMORPH_COMPUTER_USE_REGION`, default `us-east-1`); Provision sets both when the cell has the API. Each request is signed with SigV4 (`execute-api`) as the runtime's own role: the Lambda environment's credentials, or on a cluster pod its ServiceAccount role through STS `AssumeRoleWithWebIdentity`.
-- It starts the task with `POST /tasks`, then checks `POST /tasks/check` every 5 s until the task is `done` or `failed`.
+- It starts the task with `POST /tasks`, then checks `POST /tasks/check` every 5 s until the task is `done`, `failed` or `waiting_for_input`. Waiting returns immediately; end the app job rather than holding a worker while a person signs in.
 - It calls `progress(step count, note)` on every new step and at least once a minute, so the job's lease does not run out.
 - When `cancelled()` turns true it sends `POST /tasks/cancel` once and keeps checking until the task has `failed`.
 - While the task is `queued` and the CUA reports `waiting` (no computer is free), the progress note says so with the minutes waited. After `maxQueueMs` (20 minutes unless set) the tool cancels the task and returns `failed` with `No computer was free within N minutes; the task was cancelled before it started.`
@@ -417,6 +417,65 @@ A task with `session` or `secrets` works the page through clicks and typing only
 
 (1.25.0, 1.36.0, 1.37.0, 1.39.0; data plane ADR 0078, ADR 0090, ADR 0091, ADR 0092)
 
+## Help on the task's existing computer
+
+`dist/computer-use-handoff.js` adds short server calls for a paused task. It does not start another computer and does not import `ai`. It requires `runKey` and `authorize`, a server callback checked before **every** call. The callback must check the current authenticated person's access to the app's task row. An app-role signature, client-supplied task ID or pause ID alone does not prove that person owns the task.
+
+```text
+Job's browser tool → waiting_for_input → job ends
+                                             │
+App action → authorized screen/input or private login
+                                             │
+User/app explicitly requests resume → new job attaches to original runKey
+```
+
+The app keeps the original `runKey`, task ownership and returned pause metadata in its existing task row. Neither model nor browser chooses the backend `runKey`. For example, in an action backed by an app-owned `computer_tasks` table with `id`, `owner_id` and `run_key` columns:
+
+```ts
+import { createClient } from '@isomorph.ai/app-sdk';
+import { createComputerUseHandoff } from '@isomorph.ai/app-sdk/dist/computer-use-handoff.js';
+
+export default async function viewComputer(input: { taskId: string; pauseId: string }, ctx: { user: { id: string } }) {
+  const app = createClient();
+  const owned = async () => (await app.data.from('computer_tasks').select('run_key')
+    .eq('id', input.taskId).eq('owner_id', ctx.user.id).single()).data;
+  const task = await owned();
+  if (!task) throw new Error('Unknown task.');
+  const computer = createComputerUseHandoff({ runKey: task.run_key, authorize: async () => Boolean(await owned()) });
+  return computer.view(input.pauseId);
+}
+```
+
+Keep that table protected by the app's user row rules. A trusted action must still check `owner_id` against `ctx.user.id`; never use workload identity as proof of human ownership.
+
+- `requestTakeover(reason, requiredInput?)` requests a pause; `requiredInput` is `human` (default) or `login`. Poll `check()` until `waiting_for_input` gives `taskId`, `pauseId`, `reason`, `requiredInput` and optional login result.
+- `view(pauseId)` returns `{status:'pausing'}` or `{status:'ready', screenshot, url}`. Reuse the existing sign-in screenshot UI: display the base64 PNG as an image, map clicks to image pixels, and forward keyboard/scroll events through app actions. Viewing does not itself resume the agent.
+- `input(pauseId, action)` sends the same `SignInAction` click/type/key/scroll commands used by `sendSignInInput`. The CUA accepts human input only while the task is paused; it blocks the helper and main agent from acting at the same time.
+- `completeLogin(pauseId, {allowedHosts, username, password})`, or `completeLogin(pauseId, {otp})`, queues the separate browser-action-only login agent on this computer. **Call only from private server code, never expose this method or credentials as model tool inputs, prompts, progress, logs or job input/results.** Poll `check()` for `login.status: success | needs_input | human_help_needed`, with `input:'otp'` when another code is needed. Helper success leaves the main agent paused.
+- `resume(pauseId)` explicitly resumes; it does not wait for completion. On Continue, call this from the authorized app action, then enqueue the app's continuation job. That job creates `createComputerUseTool({...jobOptions, runKey: originalRunKey, attach:true})`: it polls the same task without sending `/tasks` again. Keep the original saved-session and download options if needed. Stale/duplicate resumes fail with 409; do not silently retry with a new pause ID. If enqueue fails, retry attaching to the already resumed task, not another resume.
+- `cancel()` stops the task, including while waiting.
+
+The outer agent may use app-defined tools for takeover and opening the authorized view. Resume requires an explicit user or app decision. Keep any outer-agent conversation in the app's existing job/task storage when ending a waiting job; this SDK does not persist the outer model's messages. The CUA persists its own conversation/resume state under the same task folder as screenshots and progress in its existing result bucket, without passwords or OTPs.
+
+A website that texts or mails a one-time code: `signInWithOtp` from `dist/computer-use-otp.js` does the password login, waits for the helper to ask for the code, reads it from the company's own phone number or mailbox through the app's integration grant, answers it once and returns when the helper succeeded or needs a person. The app process does the reading; the computer-use service never calls governance.
+
+```ts
+import { signInWithOtp } from '@isomorph.ai/app-sdk/dist/computer-use-otp.js';
+const answer = await signInWithOtp(computer, input.pauseId, { allowedHosts: ['netbanking.hdfcbank.com'], username, password }, {
+  execute: app.integrations.execute,
+  sms: { connection: 'company-phone', from: 'HDFCBK' },                                   // phone.sms.list on the company line
+  // email: { connection: 'company-gmail', provider: 'gmail', from: 'alerts@hdfcbank.net' }, // or the connected mailbox (gmail | outlook)
+});
+// { status: 'success' } | { status: 'human_help_needed', reason } | { status: 'ended', taskStatus }
+```
+
+- Only messages that arrived after the login started count; the code is the 4 to 8 digits next to "OTP", "code" or "password" (`extract` overrides it). It waits `timeoutMs` (3 min) for the message, checking every `pollIntervalMs` (5 s).
+- One code per pause: a website that asks for a second one gets a person (`human_help_needed`), as does a helper that gives up. Success leaves the task paused; resume it as above.
+- `readOtp(source, since)` reads a code on its own, for a page that asks for one outside a login.
+- The connection needs the app's grant for `phone.sms.list`, or `gmail.thread.list` + `gmail.message.read` (Outlook: `outlook.message.list` + `outlook.message.read`), and the portal account must be registered to that number or mailbox.
+
+`getSavedLogin(accountRef, website)` still returns `{status:'unavailable'}`; governance saved-login storage is not part of the SDK. These task helpers require the matching CUA handoff release; existing new-computer sign-in helpers below remain separate.
+
 ## Browser sign-in for computer use
 
 A person signs in to a website once on a short-lived computer the app shows as screenshots; the app keeps the browser session as a runtime secret and passes its name to later computer use tasks as `session`. Each call is short, so these run from actions as well as jobs.
@@ -654,3 +713,61 @@ keep the workflow run ID and tool-call ID stable across retries. Direct writes
 must supply their own persisted UUID `idempotencyKey`. A tool marked read-only
 does not require a key. An incomplete replay requires reconciliation, not a new
 write. Account consent and app permissions are enforced by the platform.
+
+### Tools for the outer app agent
+
+`createComputerUseHandoffTools` from `dist/computer-use-handoff-tool.js` creates
+`request_computer_input` and `resume_computer_task`. Register these beside the
+existing computer-use tool. Bind them to the original task's `runKey` in server
+code; the model supplies only the current `pauseId`.
+
+```ts
+import { generateText, hasToolCall, stepCountIs } from 'ai';
+import { createComputerUseHandoffTools } from '@isomorph.ai/app-sdk/dist/computer-use-handoff-tool.js';
+
+const handoffTools = createComputerUseHandoffTools({
+  runKey: task.run_key,
+  progress: ctx.progress,
+  cancelled: ctx.cancelled,
+  authorize: () => userOwnsTask(ctx.user.id, task.id),
+  onInputRequired: (pause, page) => showComputerInApp(task.id, pause, page),
+  canResume: pauseId => userPressedContinue(ctx.user.id, task.id, pauseId),
+});
+const result = await generateText({
+  model,
+  messages: savedMessages,
+  tools: { computer_use: computerUse, ...handoffTools },
+  stopWhen: [stepCountIs(10), hasToolCall('request_computer_input')],
+});
+await saveOuterMessages(task.id, [...savedMessages, ...result.response.messages]);
+```
+
+The ownership, screen, Continue and conversation functions in this example are
+app-provided functions, not SDK exports. The app must save the completed tool
+result before returning its waiting response to the UI. The stop condition ends
+the outer loop after the input request; creating tools alone does not stop a loop.
+
+The request tool checks the current pause and calls `view()`. Its callback receives
+the screenshot privately, so the app can show its existing computer view. The
+model receives `human_input_required`, task/pause references, reason and required
+input, without screenshot bytes or a computer endpoint. Screen refreshes and human
+input still use authorized backend `view()` and `input()` calls.
+
+When the human presses Continue, the app records approval for that user, task and
+pause and queues its outer-agent continuation with the saved conversation and
+original run key. Only this trusted user action may make `canResume` return true;
+never derive approval from model arguments or a chat string. Retain that approval
+for retries of the same continuation. A new pause needs new approval. The resume
+tool checks approval, calls resume and follows the existing task with `attach:true`.
+If an earlier attempt already resumed the task, it attaches without resuming twice.
+It may return another waiting result; use the input tool again for that new pause.
+Credentials and OTPs remain private backend `completeLogin()` inputs, not model tools.
+
+`onMessage` on the computer-use tool receives allowlisted CUA progress/handoff
+messages when they change; the progress callback also receives their text. Waiting
+results and handoff `check()` preserve the latest valid `message`. This is latest
+message delivery, not a full message history.
+
+This SDK does not supply an app-specific screen, conversation database or job
+scheduler. It supplies the tools and waiting contract; the example wires them to
+an app's existing actions, task rows, screen and job runner.

package/dist/computer-use-handoff-tool.d.ts

@@ -0,0 +1,590 @@
+import { type ComputerUseHandoffOptions, type ComputerUsePage, type ComputerUseWaiting } from './computer-use-handoff.js';
+import { type ComputerUseOptions } from './computer-use.js';
+export interface ComputerUseHandoffToolOptions extends ComputerUseHandoffOptions, ComputerUseOptions {
+    onInputRequired(input: ComputerUseWaiting, page: ComputerUsePage): Promise<void> | void;
+    canResume(pauseId: string): Promise<boolean> | boolean;
+}
+export declare function createComputerUseHandoffTools(options: ComputerUseHandoffToolOptions): {
+    request_computer_input: ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type?: undefined | "function";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type: "dynamic";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: false;
+        supportsDeferredResults?: never;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<{
+                status: "human_input_required";
+                taskId: string;
+                pauseId: string;
+                reason: string;
+                requiredInput: "login" | "otp" | "human";
+                message?: import("./computer-use-handoff.js").ComputerUseMessage;
+                login?: {
+                    status: "success" | "needs_input" | "human_help_needed";
+                    input?: "otp";
+                };
+            }>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<{
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: true;
+        supportsDeferredResults?: boolean;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, {
+            status: "human_input_required";
+            taskId: string;
+            pauseId: string;
+            reason: string;
+            requiredInput: "login" | "otp" | "human";
+            message?: import("./computer-use-handoff.js").ComputerUseMessage;
+            login?: {
+                status: "success" | "needs_input" | "human_help_needed";
+                input?: "otp";
+            };
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    });
+    resume_computer_task: ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type?: undefined | "function";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        description?: string | ((options: {
+            context: NoInfer<import("@ai-sdk/provider-utils").Context>;
+            experimental_sandbox?: import("ai").Experimental_SandboxSession;
+        }) => string) | undefined;
+        strict?: boolean;
+        inputExamples?: {
+            input: NoInfer<{
+                pauseId: string;
+            }>;
+        }[] | undefined;
+        id?: never;
+        isProviderExecuted?: never;
+        args?: never;
+        supportsDeferredResults?: never;
+    } & {
+        type: "dynamic";
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: false;
+        supportsDeferredResults?: never;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    }) | ({
+        deferLoading?: boolean;
+        title?: string;
+        providerOptions?: import("@ai-sdk/provider-utils").ProviderOptions;
+        metadata?: import("@ai-sdk/provider").JSONObject;
+        inputSchema: import("ai").FlexibleSchema<{
+            pauseId: string;
+        }>;
+        contextSchema?: import("ai").FlexibleSchema<import("@ai-sdk/provider-utils").Context> | undefined;
+        needsApproval?: boolean | import("@ai-sdk/provider-utils").ToolNeedsApprovalFunction<{
+            pauseId: string;
+        }, NoInfer<import("@ai-sdk/provider-utils").Context>> | undefined;
+        onInputStart?: ((options: import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputDelta?: ((options: {
+            inputTextDelta: string;
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        onInputAvailable?: ((options: {
+            input: {
+                pauseId: string;
+            };
+        } & import("ai").ToolExecutionOptions<NoInfer<import("@ai-sdk/provider-utils").Context>>) => void | PromiseLike<void>) | undefined;
+        toModelOutput?: ((options: {
+            toolCallId: string;
+            input: {
+                pauseId: string;
+            };
+            output: NoInfer<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>>;
+        }) => import("@ai-sdk/provider-utils").ToolResultOutput | PromiseLike<import("@ai-sdk/provider-utils").ToolResultOutput>) | undefined;
+    } & {
+        outputSchema?: import("ai").FlexibleSchema<import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>> | undefined;
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    } & {
+        type: "provider";
+        id: `${string}.${string}`;
+        args: Record<string, unknown>;
+        description?: never;
+        strict?: never;
+        inputExamples?: never;
+    } & {
+        isProviderExecuted: true;
+        supportsDeferredResults?: boolean;
+    } & {
+        execute: import("ai").ToolExecuteFunction<{
+            pauseId: string;
+        }, import("./computer-use.js").ComputerUseResult | AsyncIterable<import("./computer-use.js").ComputerUseResult>, NoInfer<import("@ai-sdk/provider-utils").Context>>;
+    });
+};

package/dist/computer-use-handoff-tool.js

@@ -0,0 +1,45 @@
+import { jsonSchema, tool } from 'ai';
+import { createComputerUseHandoff } from './computer-use-handoff.js';
+import { createComputerUseTool } from './computer-use.js';
+export function createComputerUseHandoffTools(options) {
+    if (typeof options.onInputRequired !== 'function' || typeof options.canResume !== 'function') {
+        throw new Error('Computer use handoff tools need onInputRequired and canResume callbacks.');
+    }
+    const handoff = createComputerUseHandoff(options);
+    const schema = jsonSchema({
+        type: 'object', properties: { pauseId: { type: 'string', minLength: 1 } }, required: ['pauseId'], additionalProperties: false,
+    });
+    return {
+        request_computer_input: tool({
+            description: 'Show the paused task computer to the user and request outside input. Use the current pauseId from the computer-use result. This returns human_input_required: end this agent turn and wait for the user. Never ask for passwords or one-time codes in chat.',
+            inputSchema: schema,
+            execute: async ({ pauseId }) => {
+                const input = await handoff.check();
+                if (input.status !== 'waiting_for_input' || !('pauseId' in input) || input.pauseId !== pauseId)
+                    throw new Error('The task is not waiting at this pause.');
+                const page = await handoff.view(pauseId);
+                await options.onInputRequired(input, page);
+                return { ...input, status: 'human_input_required' };
+            },
+        }),
+        resume_computer_task: tool({
+            description: 'Resume the original computer-use task only after the user has supplied input and explicitly pressed Continue. Server approval is required. Follow that same task until it finishes or needs more input; never start a new task.',
+            inputSchema: schema,
+            execute: async ({ pauseId }, execution) => {
+                const current = await handoff.check();
+                if (await options.canResume(pauseId) !== true)
+                    throw new Error('The user has not approved continuing this pause.');
+                if (current.status === 'waiting_for_input') {
+                    if (!('pauseId' in current) || current.pauseId !== pauseId)
+                        throw new Error('The task is waiting at a different pause.');
+                    await handoff.resume(pauseId);
+                }
+                else if (!['resume_starting', 'resume_queued', 'resuming', 'running', 'done', 'failed'].includes(current.status)) {
+                    throw new Error('The task cannot continue in its current state.');
+                }
+                const follow = createComputerUseTool({ ...options, attach: true });
+                return follow.execute({ task: 'Continue the original task.' }, execution);
+            },
+        }),
+    };
+}

package/dist/computer-use-handoff.d.ts

@@ -0,0 +1,77 @@
+import { type ComputerUseTransport } from './computer-use-api.js';
+import type { SignInAction } from './computer-use-sign-in.js';
+export interface ComputerUseMessage {
+    type: 'progress' | 'handoff';
+    text: string;
+    required_input?: 'login' | 'human';
+}
+export declare function computerUseMessage(value: unknown): ComputerUseMessage | undefined;
+export interface ComputerUseWaiting {
+    status: 'waiting_for_input';
+    taskId: string;
+    pauseId: string;
+    reason: string;
+    requiredInput: 'login' | 'otp' | 'human';
+    message?: ComputerUseMessage;
+    login?: {
+        status: 'success' | 'needs_input' | 'human_help_needed';
+        input?: 'otp';
+    };
+}
+export declare function waitingInput(answer: {
+    taskId?: unknown;
+    pauseId?: unknown;
+    reason?: unknown;
+    requiredInput?: unknown;
+    login?: unknown;
+    message?: unknown;
+}): ComputerUseWaiting;
+export type ComputerUsePage = {
+    status: 'pausing';
+} | {
+    status: 'ready';
+    screenshot: string;
+    url: string;
+};
+export type ComputerUseLoginInput = {
+    allowedHosts: readonly string[];
+    username: string;
+    password: string;
+    otp?: never;
+} | {
+    otp: string;
+    allowedHosts?: never;
+    username?: never;
+    password?: never;
+};
+export interface ComputerUseHandoffOptions extends ComputerUseTransport {
+    runKey: string;
+    authorize(): Promise<boolean> | boolean;
+}
+export declare function createComputerUseHandoff(options: ComputerUseHandoffOptions): {
+    requestTakeover(reason: string, requiredInput?: "login" | "human"): Promise<{
+        status: string;
+    }>;
+    view(pauseId: string): Promise<ComputerUsePage>;
+    input(pauseId: string, action: SignInAction): Promise<{
+        status: "accepted";
+    }>;
+    completeLogin(pauseId: string, input: ComputerUseLoginInput): Promise<{
+        status: "login_queued";
+    }>;
+    check(): Promise<ComputerUseWaiting | {
+        message?: ComputerUseMessage | undefined;
+        status: string;
+    }>;
+    resume(pauseId: string): Promise<{
+        status: "resume_queued";
+    }>;
+    cancel(): Promise<{
+        status: string;
+    }>;
+};
+export declare function getSavedLogin(_accountRef: string, _website: string): Promise<{
+    status: 'unavailable';
+}>;
+/** A one-time code comes from the app's own mailbox or phone number: `signInWithOtp` in `computer-use-otp.js`. */
+export { readOtp, signInWithOtp } from './computer-use-otp.js';

package/dist/computer-use-handoff.js

@@ -0,0 +1,108 @@
+import { allowedHostsOf, computerUseApi, refused } from './computer-use-api.js';
+export function computerUseMessage(value) {
+    if (!value || typeof value !== 'object')
+        return undefined;
+    const message = value;
+    if (!['progress', 'handoff'].includes(message.type) || typeof message.text !== 'string' || !message.text.trim() || message.text.length > 500)
+        return undefined;
+    return { type: message.type, text: message.text,
+        ...(message.type === 'handoff' && ['login', 'human'].includes(message.required_input ?? '') ? { required_input: message.required_input } : {}) };
+}
+export function waitingInput(answer) {
+    if (typeof answer.taskId !== 'string' || !answer.taskId || typeof answer.pauseId !== 'string' || !answer.pauseId
+        || !['login', 'otp', 'human'].includes(String(answer.requiredInput)))
+        throw new Error('The computer use API returned an incomplete pause.');
+    const result = {
+        status: 'waiting_for_input', taskId: answer.taskId, pauseId: answer.pauseId,
+        reason: typeof answer.reason === 'string' ? answer.reason : 'Outside input needed.',
+        requiredInput: answer.requiredInput,
+    };
+    const message = computerUseMessage(answer.message);
+    if (message)
+        result.message = message;
+    const login = answer.login;
+    if (login && ['success', 'needs_input', 'human_help_needed'].includes(login.status)) {
+        result.login = { status: login.status, ...(login.input === 'otp' ? { input: 'otp' } : {}) };
+    }
+    return result;
+}
+export function createComputerUseHandoff(options) {
+    if (typeof options.runKey !== 'string' || !options.runKey || typeof options.authorize !== 'function')
+        throw new Error('Computer use handoff needs runKey and an authorize check for the current user.');
+    const api = computerUseApi(options);
+    const post = async (path, body = {}, privateInput = false) => {
+        if (await options.authorize() !== true)
+            throw new Error('You cannot access this computer use task.');
+        let answer;
+        try {
+            answer = await api.call(path, { run_key: options.runKey, ...body });
+        }
+        catch (error) {
+            if (privateInput)
+                throw new Error('The computer use login request could not be completed.');
+            throw error;
+        }
+        if (answer.status !== 200)
+            throw refused(path, answer.status, privateInput ? undefined : answer.body);
+        return answer.body;
+    };
+    const paused = (pauseId) => {
+        if (!pauseId || typeof pauseId !== 'string')
+            throw new Error('Computer use handoff needs the current pauseId.');
+        return { pauseId };
+    };
+    return {
+        requestTakeover(reason, requiredInput = 'human') {
+            if (!reason.trim() || !['login', 'human'].includes(requiredInput))
+                throw new Error('Takeover needs a reason and login or human input.');
+            return post('/tasks/takeover', { reason, requiredInput });
+        },
+        async view(pauseId) {
+            const page = await post('/tasks/view', paused(pauseId));
+            if (page.status === 'pausing')
+                return { status: 'pausing' };
+            if (page.status !== 'ready' || typeof page.screenshot !== 'string' || typeof page.url !== 'string')
+                throw new Error('The computer use API returned an incomplete screen.');
+            return { status: 'ready', screenshot: page.screenshot, url: page.url };
+        },
+        input(pauseId, action) {
+            return post('/tasks/input', { ...paused(pauseId), action });
+        },
+        completeLogin(pauseId, input) {
+            let body;
+            if ('otp' in input && input.otp !== undefined) {
+                if (Object.keys(input).some(key => key !== 'otp') || !validSecret(input.otp))
+                    throw new Error('Pass only a non-empty OTP of at most 4096 bytes.');
+                body = { otp: input.otp };
+            }
+            else {
+                if (!validSecret(input.username) || !validSecret(input.password))
+                    throw new Error('Login needs a non-empty username and password of at most 4096 bytes.');
+                body = { allowedHosts: allowedHostsOf(input.allowedHosts), username: input.username, password: input.password };
+            }
+            return post('/tasks/login', { ...paused(pauseId), ...body }, true)
+                .then(() => ({ status: 'login_queued' }));
+        },
+        async check() {
+            const answer = await post('/tasks/check');
+            if (answer.status === 'waiting_for_input')
+                return waitingInput(answer);
+            const message = computerUseMessage(answer.message);
+            return { status: String(answer.status), ...(message ? { message } : {}) };
+        },
+        resume(pauseId) {
+            return post('/tasks/resume', paused(pauseId));
+        },
+        cancel() {
+            return post('/tasks/cancel');
+        },
+    };
+}
+function validSecret(value) {
+    return typeof value === 'string' && value.length > 0 && Buffer.byteLength(value) <= 4096;
+}
+export async function getSavedLogin(_accountRef, _website) {
+    return { status: 'unavailable' };
+}
+/** A one-time code comes from the app's own mailbox or phone number: `signInWithOtp` in `computer-use-otp.js`. */
+export { readOtp, signInWithOtp } from './computer-use-otp.js';

package/dist/computer-use-otp.d.ts

@@ -0,0 +1,62 @@
+import type { createComputerUseHandoff } from './computer-use-handoff.js';
+type Handoff = ReturnType<typeof createComputerUseHandoff>;
+/** `app.integrations.execute` from `createClient()`: the app's own grant, identity and rate limit. */
+export type IntegrationExecute = (connection: string, request: {
+    operation: string;
+    resource: string;
+    input?: Record<string, unknown>;
+}) => Promise<unknown>;
+export interface OtpSource {
+    execute: IntegrationExecute;
+    /** The connected mailbox the website mails the code to. `from` is the website's sender address. */
+    email?: {
+        connection: string;
+        provider: 'gmail' | 'outlook';
+        from: string;
+        resource?: string;
+    };
+    /** The company phone number the website texts the code to. `from` is the sender id, such as `HDFCBK`. */
+    sms?: {
+        connection: string;
+        from?: string;
+        resource?: string;
+    };
+    /** The code in a message's text; the default takes 4 to 8 digits next to "OTP", "code" or "password", else the first such group. */
+    extract?: (text: string) => string | undefined;
+    /** How long to wait for the message, default 3 minutes; checked every `pollIntervalMs`, default 5 s. */
+    timeoutMs?: number;
+    pollIntervalMs?: number;
+}
+export type OtpSignIn = {
+    status: 'success';
+}
+/** The website wants a person: no code arrived in time, a second code was asked for, or the helper gave up. The task is still paused. */
+ | {
+    status: 'human_help_needed';
+    reason: string;
+}
+/** The task ended while signing in. */
+ | {
+    status: 'ended';
+    taskStatus: string;
+};
+export declare function extractOtp(text: string): string | undefined;
+/** Reads the website's one-time code sent since `since`, waiting up to `timeoutMs` for it to arrive. */
+export declare function readOtp(options: OtpSource, since: Date, now?: () => number): Promise<string | undefined>;
+/**
+ * Signs in on a paused task's computer with a username and password, answers
+ * the website's one-time code from the mailbox or phone the app may read, and
+ * returns once the helper succeeded or needs a person. Call it only from
+ * private server code: the values never reach a model, a log or a job result.
+ * Success leaves the task paused; the app resumes it when it decides to.
+ */
+export declare function signInWithOtp(handoff: Handoff, pauseId: string, login: {
+    allowedHosts: readonly string[];
+    username: string;
+    password: string;
+}, otp: OtpSource, options?: {
+    pollIntervalMs?: number;
+    timeoutMs?: number;
+    now?: () => number;
+}): Promise<OtpSignIn>;
+export {};

package/dist/computer-use-otp.js

@@ -0,0 +1,92 @@
+const CODE_NEAR_WORD = /(?:otp|one[\s-]?time|passcode|password|code|pin)\D{0,40}?(\d{4,8})\b/i;
+const CODE_ALONE = /\b(\d{4,8})\b/;
+export function extractOtp(text) {
+    return CODE_NEAR_WORD.exec(text)?.[1] ?? CODE_ALONE.exec(text)?.[1];
+}
+function source(options) {
+    if (typeof options.execute !== 'function' || (options.email === undefined) === (options.sms === undefined))
+        throw new Error('An OTP source is execute plus exactly one of email or sms.');
+    return { kind: options.email ? 'email' : 'sms' };
+}
+/** The newest message from the website since `since`, as text, or undefined when none has arrived. */
+async function latestMessage(options, since) {
+    const arrived = (at) => new Date(at).getTime() >= since.getTime();
+    if (options.sms) {
+        const { connection, from, resource = 'company-line' } = options.sms;
+        const answer = await options.execute(connection, { operation: 'phone.sms.list', resource, input: { since: since.toISOString(), ...(from ? { from } : {}) } });
+        const message = (answer.messages ?? []).filter(item => arrived(item.receivedAt)).sort((a, b) => b.receivedAt.localeCompare(a.receivedAt))[0];
+        return message?.body;
+    }
+    const { connection, provider, from, resource = 'inbox' } = options.email;
+    if (provider === 'gmail') {
+        const list = await options.execute(connection, { operation: 'gmail.thread.list', resource, input: { q: `from:${from} newer_than:1d`, limit: 10 } });
+        const thread = (list.threads ?? []).filter(item => arrived(item.date)).sort((a, b) => b.date.localeCompare(a.date))[0];
+        if (!thread)
+            return undefined;
+        const message = await options.execute(connection, { operation: 'gmail.message.read', resource, input: { messageId: thread.messageId } });
+        return `${message.subject}\n${message.text}`;
+    }
+    const list = await options.execute(connection, { operation: 'outlook.message.list', resource, input: { q: `from:${from}` } });
+    const summary = (list.mailMessages ?? []).filter(item => arrived(item.receivedAt) && item.from.toLowerCase().includes(from.toLowerCase())).sort((a, b) => b.receivedAt.localeCompare(a.receivedAt))[0];
+    if (!summary)
+        return undefined;
+    const message = await options.execute(connection, { operation: 'outlook.message.read', resource, input: { messageId: summary.id } });
+    return `${message.subject}\n${message.text}`;
+}
+const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
+/** Reads the website's one-time code sent since `since`, waiting up to `timeoutMs` for it to arrive. */
+export async function readOtp(options, since, now = Date.now) {
+    source(options);
+    const extract = options.extract ?? extractOtp;
+    const deadline = now() + (options.timeoutMs ?? 180_000);
+    for (;;) {
+        const text = await latestMessage(options, since);
+        const code = text === undefined ? undefined : extract(text);
+        if (code)
+            return code;
+        if (now() >= deadline)
+            return undefined;
+        await sleep(options.pollIntervalMs ?? 5_000);
+    }
+}
+/**
+ * Signs in on a paused task's computer with a username and password, answers
+ * the website's one-time code from the mailbox or phone the app may read, and
+ * returns once the helper succeeded or needs a person. Call it only from
+ * private server code: the values never reach a model, a log or a job result.
+ * Success leaves the task paused; the app resumes it when it decides to.
+ */
+export async function signInWithOtp(handoff, pauseId, login, otp, options = {}) {
+    source(otp);
+    const now = options.now ?? Date.now;
+    // A clock a minute ahead of the mail server must not hide the code.
+    const since = new Date(now() - 60_000);
+    await handoff.completeLogin(pauseId, login);
+    let answered = false;
+    const deadline = now() + (options.timeoutMs ?? 600_000);
+    for (;;) {
+        await sleep(options.pollIntervalMs ?? 5_000);
+        const check = await handoff.check();
+        if (check.status === 'done' || check.status === 'failed')
+            return { status: 'ended', taskStatus: check.status };
+        if (check.status !== 'waiting_for_input') {
+            if (now() >= deadline)
+                return { status: 'human_help_needed', reason: 'The sign-in helper did not finish in time.' };
+            continue;
+        }
+        const login = check.login;
+        if (login?.status === 'success')
+            return { status: 'success' };
+        if (login?.status === 'human_help_needed')
+            return { status: 'human_help_needed', reason: 'The sign-in helper could not finish on its own.' };
+        if (login?.status === 'needs_input' && login.input === 'otp') {
+            if (answered)
+                return { status: 'human_help_needed', reason: 'The website asked for a second code.' };
+            const code = await readOtp(otp, since, now);
+            if (!code)
+                return { status: 'human_help_needed', reason: `No code arrived by ${otp.email ? 'email' : 'SMS'} in time.` };
+            await handoff.completeLogin(pauseId, { otp: code });
+            answered = true;
+        }
+    }
+}

package/dist/computer-use.d.ts

@@ -1,3 +1,4 @@
+import { type ComputerUseWaiting, type ComputerUseMessage } from './computer-use-handoff.js';
 export interface ComputerUseFile {
     name: string;
     contentType: "application/pdf";
@@ -29,6 +30,8 @@ export interface ComputerUseOptions {
     onDownload?(download: ComputerUseDownload): Promise<void> | void;
     /** The job's runId, or `<job name>/<scheduledAt>` for a scheduled job: one task record per job run. */
     runKey: string;
+    attach?: boolean;
+    onMessage?(message: ComputerUseMessage): Promise<void> | void;
     /** The job's progress: called on each new step and at least once a minute, which renews the job's lease. */
     progress(percent: number, note?: string): Promise<void> | void;
     /** The job's cancelled(): when true the tool cancels the task and waits until the CUA stops. */
@@ -44,7 +47,7 @@ export interface ComputerUseOptions {
     pollIntervalMs?: number;
     now?: () => number;
 }
-export interface ComputerUseResult {
+export interface ComputerUseFinished {
     status: 'done' | 'failed';
     summary: string | null;
     /** The page the browser was on when the task ended; a sign-in page there means the saved sign-in has expired. */
@@ -55,6 +58,7 @@ export interface ComputerUseResult {
         size: number;
     }[];
 }
+export type ComputerUseResult = ComputerUseFinished | ComputerUseWaiting;
 export declare function createComputerUseTool(options: ComputerUseOptions): ({
     deferLoading?: boolean;
     title?: string;

package/dist/computer-use.js

@@ -8,6 +8,7 @@ import { setTimeout as sleep } from 'node:timers/promises';
 import { jsonSchema, tool } from 'ai';
 import { allowedHostsOf, computerUseApi, refused } from './computer-use-api.js';
 import { runtimeSecretLocation, runtimeSecrets } from './secrets.js';
+import { computerUseMessage, waitingInput } from './computer-use-handoff.js';
 const HEARTBEAT_MS = 60_000;
 const DEFAULT_MAX_QUEUE_MS = 20 * 60_000;
 // Checks the API did not answer (a throttle, a 5xx, no connection) before the tool gives up: about a minute.
@@ -82,14 +83,15 @@ export function createComputerUseTool(options) {
         }
     };
     return tool({
-        description: 'Do one task in a web browser, such as filling in and submitting a form on a website. Describe the whole task in plain words, with the website address and any details it needs. Waits until the task ends and returns its status and a summary. Files the task downloads go to the app; the result lists their names.',
+        description: 'Do one task in a web browser, such as filling in and submitting a form on a website. Describe the whole task in plain words, with the website address and any details it needs. Returns when the task ends or pauses for outside input. When waiting_for_input, ask the app or human for help; do not start another task or resume without their explicit request. Files the task downloads go to the app; the result lists their names.',
         inputSchema: jsonSchema({ type: 'object', properties: { task: { type: 'string', minLength: 1 } }, required: ['task'], additionalProperties: false }),
         execute: async ({ task }) => {
-            await api.post('/tasks', {
-                run_key: options.runKey, task, ...(files?.length ? { files } : {}), ...(session === undefined ? {} : { session: await savedSession(session) }),
-                ...(allowedHosts ? { allowedHosts } : {}), ...(secrets ? { secrets } : {}),
-            });
-            await options.progress(0, 'Computer use task started.');
+            if (!options.attach)
+                await api.post('/tasks', {
+                    run_key: options.runKey, task, ...(files?.length ? { files } : {}), ...(session === undefined ? {} : { session: await savedSession(session) }),
+                    ...(allowedHosts ? { allowedHosts } : {}), ...(secrets ? { secrets } : {}),
+                });
+            await options.progress(0, options.attach ? 'Computer use task reattached.' : 'Computer use task started.');
             const queuedAt = now();
             const maxQueueMs = options.maxQueueMs ?? DEFAULT_MAX_QUEUE_MS;
             let reported = now();
@@ -97,6 +99,7 @@ export function createComputerUseTool(options) {
             let cancelSent = false;
             let queueExpired = false;
             let missed = 0;
+            let lastMessage = "";
             for (;;) {
                 if (!cancelSent && await options.cancelled()) {
                     await api.post('/tasks/cancel', { run_key: options.runKey });
@@ -112,6 +115,15 @@ export function createComputerUseTool(options) {
                 if (check?.status === 200) {
                     missed = 0;
                     const answer = check.body;
+                    const message = computerUseMessage(answer.message);
+                    if (message && JSON.stringify(message) !== lastMessage) {
+                        await options.onMessage?.(message);
+                        await options.progress(Math.min(steps, 99), message.text);
+                        reported = now();
+                        lastMessage = JSON.stringify(message);
+                    }
+                    if (answer.status === 'waiting_for_input' && !cancelSent)
+                        return waitingInput(answer);
                     if (answer.status === 'done' || answer.status === 'failed') {
                         if (queueExpired && answer.status === 'failed') {
                             return { status: 'failed', summary: `No computer was free within ${Math.round(maxQueueMs / 60_000)} minutes; the task was cancelled before it started.`, finalUrl: null, downloads: [] };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@isomorph.ai/app-sdk",
-  "version": "1.39.0",
+  "version": "1.41.0",
   "type": "module",
   "repository": {
     "type": "git",