@isomorph.ai/app-sdk 1.46.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -26
- package/dist/computer-use-handoff.d.ts +15 -2
- package/dist/computer-use.d.ts +1 -1
- package/package.json +2 -2
- package/dist/computer-use-sign-in.d.ts +0 -42
- package/dist/computer-use-sign-in.js +0 -59
package/README.md
CHANGED
|
@@ -407,7 +407,7 @@ const computerUse = createComputerUseTool({ runKey: ctx.runId ?? `my-job/${ctx.s
|
|
|
407
407
|
Options the app passes, never the model:
|
|
408
408
|
|
|
409
409
|
- `files`: PDFs the task may upload, at most five and 2 MiB in all.
|
|
410
|
-
- `session`: the name of
|
|
410
|
+
- `session`: the name of an existing browser-session runtime secret. The task starts signed in and can use only that sign-in's websites. When the task is `done` the refreshed session is saved back to the same secret. A missing one fails with `No saved sign-in named <name>.` Run one task per saved sign-in at a time: two tasks would both save back, and some websites sign out the older copy.
|
|
411
411
|
- `secrets`: `{ name: value }`, at most 10, names `^[A-Za-z][A-Za-z0-9_-]{0,63}$`, values up to 4096 bytes. The computer types a value into a password field by its name, and only there: it refuses to type a secret into a visible field, since the model reads the page after every step. The model sees only the names. Never put a password or any other credential in the task text; pass it here and say "sign in with the `password` secret". A user ID, customer ID or email is not a secret: give it in the task text.
|
|
412
412
|
- `maxQueueMs`: how long the task may wait for a free computer before the tool cancels it (20 minutes unless set).
|
|
413
413
|
- `allowedHosts`: the websites (1 to 10 lower-case hostnames) the browser may use; without it a task with no sign-in may use any website. A task with `secrets` and no `session` needs it: the websites the secrets may be typed into. A saved sign-in carries its own.
|
|
@@ -451,7 +451,7 @@ Keep that table protected by the app's user row rules. A trusted action must sti
|
|
|
451
451
|
|
|
452
452
|
- `requestTakeover(reason, requiredInput?)` requests a pause; `requiredInput` is `human` (default) or `login`. It answers the pause (`taskId`, `pauseId`, `status`, `reason`, `requiredInput`; the same pause again if one exists). Poll `check()` until `waiting_for_input` gives the pause plus the optional login result; while the task is not waiting, `check()` answers its progress (`status`, `steps`, `lastStep`, `summary`, `finalUrl`, `waiting`, `message`).
|
|
453
453
|
- `view(pauseId)` returns `{status:'ready', screenshot, url}`; while the task is still pausing, or once it resumed, the API answers 409 and the call throws. Reuse the existing sign-in screenshot UI: display the base64 PNG as an image, map clicks to image pixels, and forward keyboard/scroll events through app actions. Viewing does not itself resume the agent.
|
|
454
|
-
- `input(pauseId, action)` sends
|
|
454
|
+
- `input(pauseId, action)` sends `ComputerUseInputAction` click/type/key/scroll commands (`typeText` may be up to 4096 characters here) and answers what the computer did. The CUA accepts human input only while the task is paused; it blocks the helper and main agent from acting at the same time.
|
|
455
455
|
- `completeLogin(pauseId, {allowedHosts, username, password})`, or `completeLogin(pauseId, {otp})`, queues the separate browser-action-only login agent on this computer. **Call only from private server code, never expose this method or credentials as model tool inputs, prompts, progress, logs or job input/results.** Poll `check()` for `login.status: success | needs_input | human_help_needed`, with `input:'otp'` when another code is needed, `reason` (`unverified_success`, `helper_gave_up`, `second_code_requested`, `helper_error:<type>`) when a person is needed, and the helper's short `evidence` (never a private value). Helper success leaves the main agent paused.
|
|
456
456
|
- `resume(pauseId)` explicitly resumes and answers the pause with `status: 'resume_queued'`; it does not wait for completion. On Continue, call this from the authorized app action, then enqueue the app's continuation job. That job creates `createComputerUseTool({...jobOptions, runKey: originalRunKey, attach:true})`: it polls the same task without sending `/tasks` again. Keep the original saved-session and download options if needed. Stale/duplicate resumes fail with 409; do not silently retry with a new pause ID. If enqueue fails, retry attaching to the already resumed task, not another resume.
|
|
457
457
|
- `cancel()` stops the task, including while waiting.
|
|
@@ -475,32 +475,16 @@ const answer = await signInWithOtp(computer, input.pauseId, { allowedHosts: ['ne
|
|
|
475
475
|
- `readOtp(source, since)` reads a code on its own, for a page that asks for one outside a login.
|
|
476
476
|
- The connection needs the app's grant for `phone.sms.list`, or `gmail.thread.list` + `gmail.message.read` (Outlook: `outlook.message.list` + `outlook.message.read`), and the portal account must be registered to that number or mailbox.
|
|
477
477
|
|
|
478
|
-
`getSavedLogin(accountRef, website)` still returns `{status:'unavailable'}`; governance saved-login storage is not part of the SDK. These task helpers require the matching CUA handoff release;
|
|
478
|
+
`getSavedLogin(accountRef, website)` still returns `{status:'unavailable'}`; governance saved-login storage is not part of the SDK. These task helpers require the matching CUA handoff release; separate sign-in computer helpers are removed.
|
|
479
479
|
|
|
480
|
-
##
|
|
480
|
+
## Removed separate sign-in computers
|
|
481
481
|
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
// actions/connect-page.ts: poll until ready, then show the screenshot
|
|
489
|
-
const page = await showSignIn(handle); // { status: 'starting' } or { status: 'ready', screenshot, url }
|
|
490
|
-
// actions/connect-input.ts: what the person did on the screenshot
|
|
491
|
-
await sendSignInInput(handle, { op: 'click', x: 640, y: 400 });
|
|
492
|
-
// actions/connect-finish.ts: once the person is signed in
|
|
493
|
-
await finishSignIn(handle, { saveAs: `vendor-central/${ctx.user.id}` }); // { url }
|
|
494
|
-
```
|
|
495
|
-
|
|
496
|
-
- `startSignIn` starts a computer that may open only `allowedHosts` and returns an opaque handle for the calls that follow; the computer answers only this app's role and stops after 30 minutes. It opens `startUrl`, an HTTPS page on one of those websites, or the first website's front page: an account on `accounts.zoho.in` lists that host first or passes it as `startUrl`.
|
|
497
|
-
- `showSignIn` gives `{ status: 'starting' }` until the browser is up, then `{ status: 'ready', screenshot, url }`: a base64 PNG of the 1280×800 screen and the page's address.
|
|
498
|
-
- `sendSignInInput` takes `{ op: 'click', x, y }` (screen pixels), `{ op: 'typeText', value }` (up to 512 characters), `{ op: 'pressKey', key }` (such as `Enter` or `Tab`) or `{ op: 'scroll', direction: 'up' | 'down' }`.
|
|
499
|
-
- `finishSignIn` saves the session (cookies and site storage of the allowed websites, at most 60 KB) as the runtime secret `saveAs`, stops the computer and returns the page it was on. App code never holds the cookies.
|
|
500
|
-
- `cancelSignIn` stops the computer and keeps nothing.
|
|
501
|
-
- Deleting the sign-in is `runtimeSecrets.delete(name)`. The app decides whose sign-in it is, when it has expired and when to delete it.
|
|
502
|
-
|
|
503
|
-
(1.36.0; data plane ADR 0090)
|
|
482
|
+
SDK 2.0 removes the separate sign-in module. The matching CUA release removes `/computers/*` API routes.
|
|
483
|
+
Start a computer-use task. When it returns `waiting_for_input`, use the task
|
|
484
|
+
handoff operations above. Pass private login input through `completeLogin`,
|
|
485
|
+
or use `view` and `input` for human help. Explicitly resume the same task.
|
|
486
|
+
Existing apps that imported the removed module must migrate; there is no alias
|
|
487
|
+
that starts another computer. Existing saved-session input remains supported.
|
|
504
488
|
|
|
505
489
|
## Runtime secrets
|
|
506
490
|
|
|
@@ -1,5 +1,18 @@
|
|
|
1
1
|
import { type ComputerUseTransport } from './computer-use-api.js';
|
|
2
|
-
|
|
2
|
+
export type ComputerUseInputAction = {
|
|
3
|
+
op: 'click';
|
|
4
|
+
x: number;
|
|
5
|
+
y: number;
|
|
6
|
+
} | {
|
|
7
|
+
op: 'typeText';
|
|
8
|
+
value: string;
|
|
9
|
+
} | {
|
|
10
|
+
op: 'pressKey';
|
|
11
|
+
key: string;
|
|
12
|
+
} | {
|
|
13
|
+
op: 'scroll';
|
|
14
|
+
direction: 'up' | 'down';
|
|
15
|
+
};
|
|
3
16
|
export interface ComputerUseMessage {
|
|
4
17
|
type: 'progress' | 'handoff';
|
|
5
18
|
text: string;
|
|
@@ -77,7 +90,7 @@ export interface ComputerUseHandoffOptions extends ComputerUseTransport {
|
|
|
77
90
|
export declare function createComputerUseHandoff(options: ComputerUseHandoffOptions): {
|
|
78
91
|
requestTakeover(reason: string, requiredInput?: "login" | "human"): Promise<ComputerUsePause>;
|
|
79
92
|
view(pauseId: string): Promise<ComputerUsePage>;
|
|
80
|
-
input(pauseId: string, action:
|
|
93
|
+
input(pauseId: string, action: ComputerUseInputAction): Promise<ComputerUseInputResult>;
|
|
81
94
|
completeLogin(pauseId: string, input: ComputerUseLoginInput): Promise<{
|
|
82
95
|
status: "login_queued";
|
|
83
96
|
}>;
|
package/dist/computer-use.d.ts
CHANGED
|
@@ -48,7 +48,7 @@ export type ComputerUseSecrets = Readonly<Record<string, string>>;
|
|
|
48
48
|
*/
|
|
49
49
|
export type ComputerUseSignIn = {
|
|
50
50
|
/**
|
|
51
|
-
*
|
|
51
|
+
* An existing browser-session runtime secret: the task starts signed in,
|
|
52
52
|
* and the refreshed session is saved back to it when the task is done. Never supplied by the model.
|
|
53
53
|
*/
|
|
54
54
|
session: string;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@isomorph.ai/app-sdk",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "2.0.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
"dist"
|
|
17
17
|
],
|
|
18
18
|
"scripts": {
|
|
19
|
-
"build": "tsc -p tsconfig.json",
|
|
19
|
+
"build": "node -e \"require('node:fs').rmSync('dist', {recursive: true, force: true})\" && tsc -p tsconfig.json",
|
|
20
20
|
"test": "npm run build && node --test test/*.test.mjs"
|
|
21
21
|
},
|
|
22
22
|
"devDependencies": {
|
|
@@ -1,42 +0,0 @@
|
|
|
1
|
-
import { type ComputerUseTransport } from './computer-use-api.js';
|
|
2
|
-
/** What the person did on the screenshot: the sign-in computer's input actions. */
|
|
3
|
-
export type SignInAction = {
|
|
4
|
-
op: 'click';
|
|
5
|
-
x: number;
|
|
6
|
-
y: number;
|
|
7
|
-
} | {
|
|
8
|
-
op: 'typeText';
|
|
9
|
-
value: string;
|
|
10
|
-
} | {
|
|
11
|
-
op: 'pressKey';
|
|
12
|
-
key: string;
|
|
13
|
-
} | {
|
|
14
|
-
op: 'scroll';
|
|
15
|
-
direction: 'up' | 'down';
|
|
16
|
-
};
|
|
17
|
-
/** The computer's screen: `starting` until its browser is up, then a base64 PNG and the page's address. */
|
|
18
|
-
export type SignInPage = {
|
|
19
|
-
status: 'starting';
|
|
20
|
-
} | {
|
|
21
|
-
status: 'ready';
|
|
22
|
-
screenshot: string;
|
|
23
|
-
url: string;
|
|
24
|
-
};
|
|
25
|
-
/**
|
|
26
|
-
* Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. The
|
|
27
|
-
* computer opens `startUrl`, an HTTPS page on one of those websites, or the first website's front page.
|
|
28
|
-
*/
|
|
29
|
-
export declare function startSignIn({ allowedHosts, startUrl, ...transport }: {
|
|
30
|
-
allowedHosts: readonly string[];
|
|
31
|
-
startUrl?: string;
|
|
32
|
-
} & ComputerUseTransport): Promise<string>;
|
|
33
|
-
export declare function showSignIn(handle: string, transport?: ComputerUseTransport): Promise<SignInPage>;
|
|
34
|
-
export declare function sendSignInInput(handle: string, action: SignInAction, transport?: ComputerUseTransport): Promise<void>;
|
|
35
|
-
/** Saves the signed-in browser session as the runtime secret `saveAs`, stops the computer and returns the page it was on. */
|
|
36
|
-
export declare function finishSignIn(handle: string, { saveAs, ...transport }: {
|
|
37
|
-
saveAs: string;
|
|
38
|
-
} & ComputerUseTransport): Promise<{
|
|
39
|
-
url: string;
|
|
40
|
-
}>;
|
|
41
|
-
/** Stops the computer and keeps nothing. */
|
|
42
|
-
export declare function cancelSignIn(handle: string, transport?: ComputerUseTransport): Promise<void>;
|
|
@@ -1,59 +0,0 @@
|
|
|
1
|
-
// Browser sign-in for computer use (data plane ADR 0090): a person signs in
|
|
2
|
-
// to a website on a short-lived computer the app shows as screenshots, and the
|
|
3
|
-
// app saves the browser session as one of its runtime secrets for later
|
|
4
|
-
// computer use tasks. Each call is short, so these run in actions as well as
|
|
5
|
-
// jobs. The computer answers only the app role that started it.
|
|
6
|
-
import { allowedHostsOf, computerUseApi } from './computer-use-api.js';
|
|
7
|
-
import { runtimeSecretLocation, runtimeSecrets } from './secrets.js';
|
|
8
|
-
function startUrlOf(startUrl, hosts) {
|
|
9
|
-
let url;
|
|
10
|
-
try {
|
|
11
|
-
url = new URL(String(startUrl));
|
|
12
|
-
}
|
|
13
|
-
catch {
|
|
14
|
-
url = undefined;
|
|
15
|
-
}
|
|
16
|
-
if (typeof startUrl !== 'string' || !url || url.protocol !== 'https:' || url.username || url.password || !hosts.includes(url.hostname)) {
|
|
17
|
-
throw new Error('startUrl must be an HTTPS address on one of the allowedHosts.');
|
|
18
|
-
}
|
|
19
|
-
return startUrl;
|
|
20
|
-
}
|
|
21
|
-
function computerOf(handle) {
|
|
22
|
-
let computer;
|
|
23
|
-
try {
|
|
24
|
-
computer = JSON.parse(Buffer.from(String(handle), 'base64url').toString('utf8'));
|
|
25
|
-
}
|
|
26
|
-
catch {
|
|
27
|
-
computer = undefined;
|
|
28
|
-
}
|
|
29
|
-
if (typeof computer?.computerId !== 'string' || typeof computer.endpoint !== 'string')
|
|
30
|
-
throw new Error('This is not a sign-in handle from startSignIn.');
|
|
31
|
-
return { computerId: computer.computerId, endpoint: computer.endpoint };
|
|
32
|
-
}
|
|
33
|
-
/**
|
|
34
|
-
* Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. The
|
|
35
|
-
* computer opens `startUrl`, an HTTPS page on one of those websites, or the first website's front page.
|
|
36
|
-
*/
|
|
37
|
-
export async function startSignIn({ allowedHosts, startUrl, ...transport }) {
|
|
38
|
-
const hosts = allowedHostsOf(allowedHosts);
|
|
39
|
-
const { computerId, endpoint } = await computerUseApi(transport).post('/computers/start', { allowedHosts: hosts, ...(startUrl === undefined ? {} : { startUrl: startUrlOf(startUrl, hosts) }) });
|
|
40
|
-
return Buffer.from(JSON.stringify({ computerId, endpoint })).toString('base64url');
|
|
41
|
-
}
|
|
42
|
-
export async function showSignIn(handle, transport = {}) {
|
|
43
|
-
return computerUseApi(transport).post('/computers/page', computerOf(handle));
|
|
44
|
-
}
|
|
45
|
-
export async function sendSignInInput(handle, action, transport = {}) {
|
|
46
|
-
await computerUseApi(transport).post('/computers/input', { ...computerOf(handle), action });
|
|
47
|
-
}
|
|
48
|
-
/** Saves the signed-in browser session as the runtime secret `saveAs`, stops the computer and returns the page it was on. */
|
|
49
|
-
export async function finishSignIn(handle, { saveAs, ...transport }) {
|
|
50
|
-
const computer = computerOf(handle);
|
|
51
|
-
runtimeSecretLocation(saveAs, transport.env);
|
|
52
|
-
const { session, url } = await computerUseApi(transport).post('/computers/finish', computer);
|
|
53
|
-
await runtimeSecrets.put(saveAs, JSON.stringify(session), transport);
|
|
54
|
-
return { url };
|
|
55
|
-
}
|
|
56
|
-
/** Stops the computer and keeps nothing. */
|
|
57
|
-
export async function cancelSignIn(handle, transport = {}) {
|
|
58
|
-
await computerUseApi(transport).post('/computers/stop', computerOf(handle));
|
|
59
|
-
}
|