@isomorph.ai/app-sdk 1.40.0 → 1.42.0

package/README.md

@@ -448,16 +448,33 @@ export default async function viewComputer(input: { taskId: string; pauseId: str
 
 Keep that table protected by the app's user row rules. A trusted action must still check `owner_id` against `ctx.user.id`; never use workload identity as proof of human ownership.
 
-- `requestTakeover(reason, requiredInput?)` requests a pause; `requiredInput` is `human` (default) or `login`. Poll `check()` until `waiting_for_input` gives `taskId`, `pauseId`, `reason`, `requiredInput` and optional login result.
-- `view(pauseId)` returns `{status:'pausing'}` or `{status:'ready', screenshot, url}`. Reuse the existing sign-in screenshot UI: display the base64 PNG as an image, map clicks to image pixels, and forward keyboard/scroll events through app actions. Viewing does not itself resume the agent.
-- `input(pauseId, action)` sends the same `SignInAction` click/type/key/scroll commands used by `sendSignInInput`. The CUA accepts human input only while the task is paused; it blocks the helper and main agent from acting at the same time.
+- `requestTakeover(reason, requiredInput?)` requests a pause; `requiredInput` is `human` (default) or `login`. It answers the pause (`taskId`, `pauseId`, `status`, `reason`, `requiredInput`; the same pause again if one exists). Poll `check()` until `waiting_for_input` gives the pause plus the optional login result; while the task is not waiting, `check()` answers its progress (`status`, `steps`, `lastStep`, `summary`, `finalUrl`, `waiting`, `message`).
+- `view(pauseId)` returns `{status:'ready', screenshot, url}`; while the task is still pausing, or once it resumed, the API answers 409 and the call throws. Reuse the existing sign-in screenshot UI: display the base64 PNG as an image, map clicks to image pixels, and forward keyboard/scroll events through app actions. Viewing does not itself resume the agent.
+- `input(pauseId, action)` sends the same `SignInAction` click/type/key/scroll commands used by `sendSignInInput` (`typeText` may be up to 4096 characters here) and answers what the computer did. The CUA accepts human input only while the task is paused; it blocks the helper and main agent from acting at the same time.
 - `completeLogin(pauseId, {allowedHosts, username, password})`, or `completeLogin(pauseId, {otp})`, queues the separate browser-action-only login agent on this computer. **Call only from private server code, never expose this method or credentials as model tool inputs, prompts, progress, logs or job input/results.** Poll `check()` for `login.status: success | needs_input | human_help_needed`, with `input:'otp'` when another code is needed. Helper success leaves the main agent paused.
-- `resume(pauseId)` explicitly resumes; it does not wait for completion. On Continue, call this from the authorized app action, then enqueue the app's continuation job. That job creates `createComputerUseTool({...jobOptions, runKey: originalRunKey, attach:true})`: it polls the same task without sending `/tasks` again. Keep the original saved-session and download options if needed. Stale/duplicate resumes fail with 409; do not silently retry with a new pause ID. If enqueue fails, retry attaching to the already resumed task, not another resume.
+- `resume(pauseId)` explicitly resumes and answers the pause with `status: 'resume_queued'`; it does not wait for completion. On Continue, call this from the authorized app action, then enqueue the app's continuation job. That job creates `createComputerUseTool({...jobOptions, runKey: originalRunKey, attach:true})`: it polls the same task without sending `/tasks` again. Keep the original saved-session and download options if needed. Stale/duplicate resumes fail with 409; do not silently retry with a new pause ID. If enqueue fails, retry attaching to the already resumed task, not another resume.
 - `cancel()` stops the task, including while waiting.
 
 The outer agent may use app-defined tools for takeover and opening the authorized view. Resume requires an explicit user or app decision. Keep any outer-agent conversation in the app's existing job/task storage when ending a waiting job; this SDK does not persist the outer model's messages. The CUA persists its own conversation/resume state under the same task folder as screenshots and progress in its existing result bucket, without passwords or OTPs.
 
-P0 `getSavedLogin(accountRef, website)` and `getOtp(taskId, website)` return `{status:'unavailable'}` and call no integration. Use human takeover when no approved backend input is supplied. Governance saved-login/website storage and email/SMS OTP lookup are P0.5, not part of this SDK change. These task helpers require the matching CUA handoff release; existing new-computer sign-in helpers below remain separate.
+A website that texts or mails a one-time code: `signInWithOtp` from `dist/computer-use-otp.js` does the password login, waits for the helper to ask for the code, reads it from the company's own phone number or mailbox through the app's integration grant, answers it once and returns when the helper succeeded or needs a person. The app process does the reading; the computer-use service never calls governance.
+
+```ts
+import { signInWithOtp } from '@isomorph.ai/app-sdk/dist/computer-use-otp.js';
+const answer = await signInWithOtp(computer, input.pauseId, { allowedHosts: ['netbanking.hdfcbank.com'], username, password }, {
+  execute: app.integrations.execute,
+  sms: { connection: 'company-phone', from: 'HDFCBK' },                                   // phone.sms.list on the company line
+  // email: { connection: 'company-gmail', provider: 'gmail', from: 'alerts@hdfcbank.net' }, // or the connected mailbox (gmail | outlook)
+});
+// { status: 'success' } | { status: 'human_help_needed', reason } | { status: 'ended', taskStatus }
+```
+
+- Only messages that arrived after the login started count; the code is the 4 to 8 digits next to "OTP", "code" or "password" (`extract` overrides it). It waits `timeoutMs` (3 min) for the message, checking every `pollIntervalMs` (5 s).
+- One code per pause: a website that asks for a second one gets a person (`human_help_needed`), as does a helper that gives up. Success leaves the task paused; resume it as above.
+- `readOtp(source, since)` reads a code on its own, for a page that asks for one outside a login.
+- The connection needs the app's grant for `phone.sms.list`, or `gmail.thread.list` + `gmail.message.read` (Outlook: `outlook.message.list` + `outlook.message.read`), and the portal account must be registered to that number or mailbox.
+
+`getSavedLogin(accountRef, website)` still returns `{status:'unavailable'}`; governance saved-login storage is not part of the SDK. These task helpers require the matching CUA handoff release; existing new-computer sign-in helpers below remain separate.
 
 ## Browser sign-in for computer use
 

package/dist/computer-use-handoff.d.ts

@@ -27,12 +27,34 @@ export declare function waitingInput(answer: {
     message?: unknown;
 }): ComputerUseWaiting;
 export type ComputerUsePage = {
-    status: 'pausing';
-} | {
     status: 'ready';
     screenshot: string;
     url: string;
 };
+/** A pause as the computer-use API describes it on takeover, login and resume; `status` is the task's, not only `waiting_for_input`. */
+export type ComputerUsePause = {
+    taskId: string;
+    pauseId: string;
+    status: string;
+    reason: string;
+    requiredInput: 'login' | 'otp' | 'human';
+};
+/** What the paused computer answered to one input: the desktop function's own result, passed through. */
+export type ComputerUseInputResult = Record<string, unknown>;
+/** Progress of a task that is not waiting: the same fields `createComputerUseTool` reads while it polls. */
+export type ComputerUseProgress = {
+    status: string;
+    steps?: number;
+    lastStep?: string | null;
+    summary?: string | null;
+    finalUrl?: string | null;
+    waiting?: {
+        reason: string;
+        since: string;
+        attempts: number;
+    } | null;
+    message?: ComputerUseMessage;
+};
 export type ComputerUseLoginInput = {
     allowedHosts: readonly string[];
     username: string;
@@ -49,23 +71,14 @@ export interface ComputerUseHandoffOptions extends ComputerUseTransport {
     authorize(): Promise<boolean> | boolean;
 }
 export declare function createComputerUseHandoff(options: ComputerUseHandoffOptions): {
-    requestTakeover(reason: string, requiredInput?: "login" | "human"): Promise<{
-        status: string;
-    }>;
+    requestTakeover(reason: string, requiredInput?: "login" | "human"): Promise<ComputerUsePause>;
     view(pauseId: string): Promise<ComputerUsePage>;
-    input(pauseId: string, action: SignInAction): Promise<{
-        status: "accepted";
-    }>;
+    input(pauseId: string, action: SignInAction): Promise<ComputerUseInputResult>;
     completeLogin(pauseId: string, input: ComputerUseLoginInput): Promise<{
         status: "login_queued";
     }>;
-    check(): Promise<ComputerUseWaiting | {
-        message?: ComputerUseMessage | undefined;
-        status: string;
-    }>;
-    resume(pauseId: string): Promise<{
-        status: "resume_queued";
-    }>;
+    check(): Promise<ComputerUseWaiting | ComputerUseProgress>;
+    resume(pauseId: string): Promise<ComputerUsePause>;
     cancel(): Promise<{
         status: string;
     }>;
@@ -73,6 +86,5 @@ export declare function createComputerUseHandoff(options: ComputerUseHandoffOpti
 export declare function getSavedLogin(_accountRef: string, _website: string): Promise<{
     status: 'unavailable';
 }>;
-export declare function getOtp(_taskId: string, _website: string): Promise<{
-    status: 'unavailable';
-}>;
+/** A one-time code comes from the app's own mailbox or phone number: `signInWithOtp` in `computer-use-otp.js`. */
+export { readOtp, signInWithOtp } from './computer-use-otp.js';

package/dist/computer-use-handoff.js

@@ -46,6 +46,11 @@ export function createComputerUseHandoff(options) {
             throw refused(path, answer.status, privateInput ? undefined : answer.body);
         return answer.body;
     };
+    const pauseOf = (answer) => {
+        if (typeof answer.taskId !== 'string' || typeof answer.pauseId !== 'string' || !['login', 'otp', 'human'].includes(String(answer.requiredInput)))
+            throw new Error('The computer use API returned an incomplete pause.');
+        return { taskId: answer.taskId, pauseId: answer.pauseId, status: String(answer.status), reason: typeof answer.reason === 'string' ? answer.reason : 'Outside input needed.', requiredInput: answer.requiredInput };
+    };
     const paused = (pauseId) => {
         if (!pauseId || typeof pauseId !== 'string')
             throw new Error('Computer use handoff needs the current pauseId.');
@@ -55,12 +60,11 @@ export function createComputerUseHandoff(options) {
         requestTakeover(reason, requiredInput = 'human') {
             if (!reason.trim() || !['login', 'human'].includes(requiredInput))
                 throw new Error('Takeover needs a reason and login or human input.');
-            return post('/tasks/takeover', { reason, requiredInput });
+            return post('/tasks/takeover', { reason, requiredInput }).then(pauseOf);
         },
+        // The API answers 409 while the task is still pausing or after it resumed; `ready` is the only 200.
         async view(pauseId) {
             const page = await post('/tasks/view', paused(pauseId));
-            if (page.status === 'pausing')
-                return { status: 'pausing' };
             if (page.status !== 'ready' || typeof page.screenshot !== 'string' || typeof page.url !== 'string')
                 throw new Error('The computer use API returned an incomplete screen.');
             return { status: 'ready', screenshot: page.screenshot, url: page.url };
@@ -88,10 +92,23 @@ export function createComputerUseHandoff(options) {
             if (answer.status === 'waiting_for_input')
                 return waitingInput(answer);
             const message = computerUseMessage(answer.message);
-            return { status: String(answer.status), ...(message ? { message } : {}) };
+            const progress = { status: String(answer.status) };
+            if (typeof answer.steps === 'number')
+                progress.steps = answer.steps;
+            if (answer.last_step !== undefined)
+                progress.lastStep = answer.last_step;
+            if (answer.summary !== undefined)
+                progress.summary = answer.summary;
+            if (answer.final_url !== undefined)
+                progress.finalUrl = answer.final_url;
+            if (answer.waiting !== undefined)
+                progress.waiting = answer.waiting;
+            if (message)
+                progress.message = message;
+            return progress;
         },
         resume(pauseId) {
-            return post('/tasks/resume', paused(pauseId));
+            return post('/tasks/resume', paused(pauseId)).then(pauseOf);
         },
         cancel() {
             return post('/tasks/cancel');
@@ -104,6 +121,5 @@ function validSecret(value) {
 export async function getSavedLogin(_accountRef, _website) {
     return { status: 'unavailable' };
 }
-export async function getOtp(_taskId, _website) {
-    return { status: 'unavailable' };
-}
+/** A one-time code comes from the app's own mailbox or phone number: `signInWithOtp` in `computer-use-otp.js`. */
+export { readOtp, signInWithOtp } from './computer-use-otp.js';

package/dist/computer-use-otp.d.ts

@@ -0,0 +1,62 @@
+import type { createComputerUseHandoff } from './computer-use-handoff.js';
+type Handoff = ReturnType<typeof createComputerUseHandoff>;
+/** `app.integrations.execute` from `createClient()`: the app's own grant, identity and rate limit. */
+export type IntegrationExecute = (connection: string, request: {
+    operation: string;
+    resource: string;
+    input?: Record<string, unknown>;
+}) => Promise<unknown>;
+export interface OtpSource {
+    execute: IntegrationExecute;
+    /** The connected mailbox the website mails the code to. `from` is the website's sender address. */
+    email?: {
+        connection: string;
+        provider: 'gmail' | 'outlook';
+        from: string;
+        resource?: string;
+    };
+    /** The company phone number the website texts the code to. `from` is the sender id, such as `HDFCBK`. */
+    sms?: {
+        connection: string;
+        from?: string;
+        resource?: string;
+    };
+    /** The code in a message's text; the default takes 4 to 8 digits next to "OTP", "code" or "password", else the first such group. */
+    extract?: (text: string) => string | undefined;
+    /** How long to wait for the message, default 3 minutes; checked every `pollIntervalMs`, default 5 s. */
+    timeoutMs?: number;
+    pollIntervalMs?: number;
+}
+export type OtpSignIn = {
+    status: 'success';
+}
+/** The website wants a person: no code arrived in time, a second code was asked for, or the helper gave up. The task is still paused. */
+ | {
+    status: 'human_help_needed';
+    reason: string;
+}
+/** The task ended while signing in. */
+ | {
+    status: 'ended';
+    taskStatus: string;
+};
+export declare function extractOtp(text: string): string | undefined;
+/** Reads the website's one-time code sent since `since`, waiting up to `timeoutMs` for it to arrive. */
+export declare function readOtp(options: OtpSource, since: Date, now?: () => number): Promise<string | undefined>;
+/**
+ * Signs in on a paused task's computer with a username and password, answers
+ * the website's one-time code from the mailbox or phone the app may read, and
+ * returns once the helper succeeded or needs a person. Call it only from
+ * private server code: the values never reach a model, a log or a job result.
+ * Success leaves the task paused; the app resumes it when it decides to.
+ */
+export declare function signInWithOtp(handoff: Handoff, pauseId: string, login: {
+    allowedHosts: readonly string[];
+    username: string;
+    password: string;
+}, otp: OtpSource, options?: {
+    pollIntervalMs?: number;
+    timeoutMs?: number;
+    now?: () => number;
+}): Promise<OtpSignIn>;
+export {};

package/dist/computer-use-otp.js

@@ -0,0 +1,91 @@
+const CODE_NEAR_WORD = /(?:otp|one[\s-]?time|passcode|password|code|pin)\D{0,40}?(\d{4,8})\b/i;
+const CODE_ALONE = /\b(\d{4,8})\b/;
+export function extractOtp(text) {
+    return CODE_NEAR_WORD.exec(text)?.[1] ?? CODE_ALONE.exec(text)?.[1];
+}
+function source(options) {
+    if (typeof options.execute !== 'function' || (options.email === undefined) === (options.sms === undefined))
+        throw new Error('An OTP source is execute plus exactly one of email or sms.');
+    return { kind: options.email ? 'email' : 'sms' };
+}
+/** The newest message from the website since `since`, as text, or undefined when none has arrived. */
+async function latestMessage(options, since) {
+    const arrived = (at) => new Date(at).getTime() >= since.getTime();
+    if (options.sms) {
+        const { connection, from, resource = 'company-line' } = options.sms;
+        const answer = await options.execute(connection, { operation: 'phone.sms.list', resource, input: { since: since.toISOString(), ...(from ? { from } : {}) } });
+        const message = (answer.messages ?? []).filter(item => arrived(item.receivedAt)).sort((a, b) => b.receivedAt.localeCompare(a.receivedAt))[0];
+        return message?.body;
+    }
+    const { connection, provider, from, resource = 'inbox' } = options.email;
+    if (provider === 'gmail') {
+        const list = await options.execute(connection, { operation: 'gmail.thread.list', resource, input: { q: `from:${from} newer_than:1d`, limit: 10 } });
+        const thread = (list.threads ?? []).filter(item => arrived(item.date)).sort((a, b) => b.date.localeCompare(a.date))[0];
+        if (!thread)
+            return undefined;
+        const message = await options.execute(connection, { operation: 'gmail.message.read', resource, input: { messageId: thread.messageId } });
+        return `${message.subject}\n${message.text}`;
+    }
+    const list = await options.execute(connection, { operation: 'outlook.message.list', resource, input: { q: `from:${from}` } });
+    const summary = (list.mailMessages ?? []).filter(item => arrived(item.receivedAt) && item.from.toLowerCase().includes(from.toLowerCase())).sort((a, b) => b.receivedAt.localeCompare(a.receivedAt))[0];
+    if (!summary)
+        return undefined;
+    const message = await options.execute(connection, { operation: 'outlook.message.read', resource, input: { messageId: summary.id } });
+    return `${message.subject}\n${message.text}`;
+}
+const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
+/** Reads the website's one-time code sent since `since`, waiting up to `timeoutMs` for it to arrive. */
+export async function readOtp(options, since, now = Date.now) {
+    source(options);
+    const extract = options.extract ?? extractOtp;
+    const deadline = now() + (options.timeoutMs ?? 180_000);
+    for (;;) {
+        const text = await latestMessage(options, since);
+        const code = text === undefined ? undefined : extract(text);
+        if (code)
+            return code;
+        if (now() >= deadline)
+            return undefined;
+        await sleep(options.pollIntervalMs ?? 5_000);
+    }
+}
+/**
+ * Signs in on a paused task's computer with a username and password, answers
+ * the website's one-time code from the mailbox or phone the app may read, and
+ * returns once the helper succeeded or needs a person. Call it only from
+ * private server code: the values never reach a model, a log or a job result.
+ * Success leaves the task paused; the app resumes it when it decides to.
+ */
+export async function signInWithOtp(handoff, pauseId, login, otp, options = {}) {
+    source(otp);
+    const now = options.now ?? Date.now;
+    // A clock a minute ahead of the mail server must not hide the code.
+    const since = new Date(now() - 60_000);
+    await handoff.completeLogin(pauseId, login);
+    let answered = false;
+    const deadline = now() + (options.timeoutMs ?? 600_000);
+    for (;;) {
+        await sleep(options.pollIntervalMs ?? 5_000);
+        const check = await handoff.check();
+        if (check.status === 'done' || check.status === 'failed')
+            return { status: 'ended', taskStatus: check.status };
+        if (now() >= deadline)
+            return { status: 'human_help_needed', reason: 'The sign-in helper did not finish in time.' };
+        if (check.status !== 'waiting_for_input')
+            continue;
+        const login = check.login;
+        if (login?.status === 'success')
+            return { status: 'success' };
+        if (login?.status === 'human_help_needed')
+            return { status: 'human_help_needed', reason: 'The sign-in helper could not finish on its own.' };
+        if (login?.status === 'needs_input' && login.input === 'otp') {
+            if (answered)
+                return { status: 'human_help_needed', reason: 'The website asked for a second code.' };
+            const code = await readOtp(otp, since, now);
+            if (!code)
+                return { status: 'human_help_needed', reason: `No code arrived by ${otp.email ? 'email' : 'SMS'} in time.` };
+            await handoff.completeLogin(pauseId, { otp: code });
+            answered = true;
+        }
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@isomorph.ai/app-sdk",
-  "version": "1.40.0",
+  "version": "1.42.0",
   "type": "module",
   "repository": {
     "type": "git",