@isomorph.ai/app-sdk 1.35.1 → 1.36.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +56 -4
- package/dist/aws.d.ts +6 -2
- package/dist/aws.js +6 -3
- package/dist/computer-use-api.d.ts +15 -0
- package/dist/computer-use-api.js +33 -0
- package/dist/computer-use-sign-in.d.ts +38 -0
- package/dist/computer-use-sign-in.js +43 -0
- package/dist/computer-use.d.ts +14 -2
- package/dist/computer-use.js +53 -25
- package/dist/secrets.d.ts +17 -0
- package/dist/secrets.js +62 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -387,13 +387,13 @@ App code declares exact HTTPS pages and named click/fill selectors reviewed for
|
|
|
387
387
|
|
|
388
388
|
## Computer use tool
|
|
389
389
|
|
|
390
|
-
A job's agent can hand one browser task, writes included, to
|
|
390
|
+
A job's agent can hand one browser task, writes included, to the computer use service. Use it only in `jobs/*.ts`. Install the optional peer `ai`.
|
|
391
391
|
|
|
392
392
|
```ts
|
|
393
393
|
import { createComputerUseTool } from '@isomorph.ai/app-sdk/dist/computer-use.js';
|
|
394
394
|
// Inside a job handler:
|
|
395
395
|
const computerUse = createComputerUseTool({ runKey: ctx.runId ?? `my-job/${ctx.scheduledAt}`, progress: ctx.progress, cancelled: ctx.cancelled });
|
|
396
|
-
// Add { computerUse } to generateText's tools. It returns { status: 'done' | 'failed', summary }.
|
|
396
|
+
// Add { computerUse } to generateText's tools. It returns { status: 'done' | 'failed', summary, finalUrl }.
|
|
397
397
|
```
|
|
398
398
|
|
|
399
399
|
- The tool calls the task API at `ISOMORPH_COMPUTER_USE_URL` (region `ISOMORPH_COMPUTER_USE_REGION`, default `us-east-1`); Provision sets both when the cell has the API. Each request is signed with SigV4 (`execute-api`) as the runtime's own role: the Lambda environment's credentials, or on a cluster pod its ServiceAccount role through STS `AssumeRoleWithWebIdentity`.
|
|
@@ -401,9 +401,61 @@ const computerUse = createComputerUseTool({ runKey: ctx.runId ?? `my-job/${ctx.s
|
|
|
401
401
|
- It calls `progress(step count, note)` on every new step and at least once a minute, so the job's lease does not run out.
|
|
402
402
|
- When `cancelled()` turns true it sends `POST /tasks/cancel` once and keeps checking until the task has `failed`.
|
|
403
403
|
- One task record per `runKey`: a retried job run sends the same key and gets the queued, running or finished task back. Only a failed task starts a new attempt.
|
|
404
|
-
-
|
|
404
|
+
- `finalUrl` is the page the browser ended on. A website's sign-in page there means a saved sign-in has expired: ask the person to sign in again.
|
|
405
405
|
|
|
406
|
-
|
|
406
|
+
Options the app passes, never the model:
|
|
407
|
+
|
|
408
|
+
- `files`: PDFs the task may upload, at most five and 2 MiB in all.
|
|
409
|
+
- `session`: the name of a runtime secret a sign-in was saved under (`finishSignIn`'s `saveAs`, below). The task starts signed in and can use only that sign-in's websites. When the task is `done` the refreshed session is saved back to the same secret. A missing one fails with `No saved sign-in named <name>.` Run one task per saved sign-in at a time: two tasks would both save back, and some websites sign out the older copy.
|
|
410
|
+
- `secrets`: `{ name: value }`, at most 10, names `^[A-Za-z][A-Za-z0-9_-]{0,63}$`, values up to 4096 bytes. The computer types a value into a password field by its name; the model sees only the names. Never put a password or any other credential in the task text; pass it here and say "sign in with the `password` secret".
|
|
411
|
+
- `allowedHosts`: the websites (1 to 10 lower-case hostnames) a task with `secrets` and no `session` may use. A saved sign-in carries its own.
|
|
412
|
+
|
|
413
|
+
A task with `session` or `secrets` works the page through clicks and typing only. `connectionId` is gone: pass `session`.
|
|
414
|
+
|
|
415
|
+
(1.25.0, 1.36.0; data plane ADR 0078, ADR 0090)
|
|
416
|
+
|
|
417
|
+
## Browser sign-in for computer use
|
|
418
|
+
|
|
419
|
+
A person signs in to a website once on a short-lived computer the app shows as screenshots; the app keeps the browser session as a runtime secret and passes its name to later computer use tasks as `session`. Each call is short, so these run from actions as well as jobs.
|
|
420
|
+
|
|
421
|
+
```ts
|
|
422
|
+
import { startSignIn, showSignIn, sendSignInInput, finishSignIn, cancelSignIn } from '@isomorph.ai/app-sdk/dist/computer-use-sign-in.js';
|
|
423
|
+
// actions/connect-start.ts
|
|
424
|
+
const handle = await startSignIn({ allowedHosts: ['vendorcentral.amazon.com'] });
|
|
425
|
+
// actions/connect-page.ts: poll until ready, then show the screenshot
|
|
426
|
+
const page = await showSignIn(handle); // { status: 'starting' } or { status: 'ready', screenshot, url }
|
|
427
|
+
// actions/connect-input.ts: what the person did on the screenshot
|
|
428
|
+
await sendSignInInput(handle, { op: 'click', x: 640, y: 400 });
|
|
429
|
+
// actions/connect-finish.ts: once the person is signed in
|
|
430
|
+
await finishSignIn(handle, { saveAs: `vendor-central/${ctx.user.id}` }); // { url }
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
- `startSignIn` starts a computer that may open only `allowedHosts` and returns an opaque handle for the calls that follow; the computer answers only this app's role and stops after 30 minutes.
|
|
434
|
+
- `showSignIn` gives `{ status: 'starting' }` until the browser is up, then `{ status: 'ready', screenshot, url }`: a base64 PNG of the 1280×800 screen and the page's address.
|
|
435
|
+
- `sendSignInInput` takes `{ op: 'click', x, y }` (screen pixels), `{ op: 'typeText', value }` (up to 512 characters), `{ op: 'pressKey', key }` (such as `Enter` or `Tab`) or `{ op: 'scroll', direction: 'up' | 'down' }`.
|
|
436
|
+
- `finishSignIn` saves the session (cookies and site storage of the allowed websites, at most 60 KB) as the runtime secret `saveAs`, stops the computer and returns the page it was on. App code never holds the cookies.
|
|
437
|
+
- `cancelSignIn` stops the computer and keeps nothing.
|
|
438
|
+
- Deleting the sign-in is `runtimeSecrets.delete(name)`. The app decides whose sign-in it is, when it has expired and when to delete it.
|
|
439
|
+
|
|
440
|
+
(1.36.0; data plane ADR 0090)
|
|
441
|
+
|
|
442
|
+
## Runtime secrets
|
|
443
|
+
|
|
444
|
+
An app can save its own secrets while it runs, such as a saved sign-in, an OAuth token a person granted or a key a person typed into the app's settings. They work in actions and jobs.
|
|
445
|
+
|
|
446
|
+
```ts
|
|
447
|
+
import { runtimeSecrets } from '@isomorph.ai/app-sdk/dist/secrets.js';
|
|
448
|
+
await runtimeSecrets.put(`supplier-key/${ctx.user.id}`, key); // creates it the first time
|
|
449
|
+
const key = await runtimeSecrets.get(`supplier-key/${ctx.user.id}`); // string, or null when there is none
|
|
450
|
+
await runtimeSecrets.delete(`supplier-key/${ctx.user.id}`);
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
- Names match `^[A-Za-z0-9][A-Za-z0-9/_+=.@-]{0,199}$`; values are strings up to 64 KB.
|
|
454
|
+
- Each is one Secrets Manager secret in the app's own cell, under `ISOMORPH_RUNTIME_SECRETS_PREFIX` (the app's secret path plus `runtime/`, in `ISOMORPH_RUNTIME_SECRETS_REGION`); Provision sets both. The calls are signed as the runtime's own role, which may use only this app's runtime secrets. Secrets the builder set stay read-only.
|
|
455
|
+
- `delete` removes the secret at once, with no recovery window, so the same name can be saved again straight away.
|
|
456
|
+
- Each one costs $0.40 a month while it exists.
|
|
457
|
+
|
|
458
|
+
(1.36.0; data plane ADR 0090)
|
|
407
459
|
|
|
408
460
|
## Running and rollout
|
|
409
461
|
|
package/dist/aws.d.ts
CHANGED
|
@@ -27,10 +27,14 @@ export declare function runtimeCredentials(env?: NodeJS.ProcessEnv, { fetch: fet
|
|
|
27
27
|
fetch?: typeof fetch;
|
|
28
28
|
now?: () => number;
|
|
29
29
|
}): Promise<Credentials>;
|
|
30
|
-
/**
|
|
31
|
-
|
|
30
|
+
/**
|
|
31
|
+
* A JSON POST signed for `service` in `aws.region`, answered with its status and JSON body. `headers`
|
|
32
|
+
* (lower-case names) are added to the signed request and may replace its `content-type`.
|
|
33
|
+
*/
|
|
34
|
+
export declare function signedJsonPost(aws: AwsEnvironment, service: string, target: string, payload: unknown, { fetch: fetcher, signingDate, headers: extra }?: {
|
|
32
35
|
fetch?: typeof fetch;
|
|
33
36
|
signingDate?: Date;
|
|
37
|
+
headers?: Record<string, string>;
|
|
34
38
|
}): Promise<{
|
|
35
39
|
status: number;
|
|
36
40
|
body: unknown;
|
package/dist/aws.js
CHANGED
|
@@ -79,13 +79,16 @@ export async function runtimeCredentials(env = process.env, { fetch: fetcher = f
|
|
|
79
79
|
webIdentity = { roleArn, credentials: { accessKeyId, secretAccessKey, sessionToken }, expiresAt: expiration };
|
|
80
80
|
return webIdentity.credentials;
|
|
81
81
|
}
|
|
82
|
-
/**
|
|
83
|
-
|
|
82
|
+
/**
|
|
83
|
+
* A JSON POST signed for `service` in `aws.region`, answered with its status and JSON body. `headers`
|
|
84
|
+
* (lower-case names) are added to the signed request and may replace its `content-type`.
|
|
85
|
+
*/
|
|
86
|
+
export async function signedJsonPost(aws, service, target, payload, { fetch: fetcher = fetch, signingDate = new Date(), headers: extra = {} } = {}) {
|
|
84
87
|
const url = new URL(target);
|
|
85
88
|
const body = JSON.stringify(payload);
|
|
86
89
|
const request = new HttpRequest({
|
|
87
90
|
method: 'POST', protocol: url.protocol, hostname: url.hostname, ...(url.port ? { port: Number(url.port) } : {}), path: url.pathname,
|
|
88
|
-
headers: { host: url.host, 'content-type': 'application/json' }, body,
|
|
91
|
+
headers: { host: url.host, 'content-type': 'application/json', ...extra }, body,
|
|
89
92
|
});
|
|
90
93
|
const { host: _host, ...headers } = (await signer(aws, service).sign(request, { signingDate })).headers;
|
|
91
94
|
const response = await fetcher(url.href, { method: 'POST', headers, body, signal: AbortSignal.timeout(30_000) });
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
export interface ComputerUseTransport {
|
|
2
|
+
env?: NodeJS.ProcessEnv;
|
|
3
|
+
fetch?: typeof fetch;
|
|
4
|
+
}
|
|
5
|
+
export declare function refused(path: string, status: number, body: unknown): Error;
|
|
6
|
+
export declare function computerUseApi({ env, fetch }?: ComputerUseTransport): {
|
|
7
|
+
call: (path: string, body: Record<string, unknown>) => Promise<{
|
|
8
|
+
status: number;
|
|
9
|
+
body: unknown;
|
|
10
|
+
}>;
|
|
11
|
+
/** A call the API must answer with 200; its body. */
|
|
12
|
+
post<T>(path: string, body: Record<string, unknown>): Promise<T>;
|
|
13
|
+
};
|
|
14
|
+
/** The websites a sign-in or a task may use: 1 to 10 lower-case hostnames, copied. */
|
|
15
|
+
export declare function allowedHostsOf(hosts: unknown): string[];
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
// Harbour's computer use API (CUA) as the runtime's own role, signed with
|
|
2
|
+
// SigV4 (ADR 0078): what the computer use tool and the sign-in helpers share.
|
|
3
|
+
import { runtimeCredentials, signedJsonPost } from './aws.js';
|
|
4
|
+
export function refused(path, status, body) {
|
|
5
|
+
const error = body?.error;
|
|
6
|
+
return new Error(`The computer use API refused ${path} (${status}${typeof error === 'string' ? `: ${error}` : ''}).`);
|
|
7
|
+
}
|
|
8
|
+
export function computerUseApi({ env = process.env, fetch } = {}) {
|
|
9
|
+
const base = env.ISOMORPH_COMPUTER_USE_URL?.replace(/\/+$/, '');
|
|
10
|
+
if (!base)
|
|
11
|
+
throw new Error('This app has no computer use API (ISOMORPH_COMPUTER_USE_URL is not set).');
|
|
12
|
+
const region = env.ISOMORPH_COMPUTER_USE_REGION || 'us-east-1';
|
|
13
|
+
const transport = fetch ? { fetch } : {};
|
|
14
|
+
const call = async (path, body) => signedJsonPost({ ...await runtimeCredentials(env, transport), region }, 'execute-api', `${base}${path}`, body, transport);
|
|
15
|
+
return {
|
|
16
|
+
call,
|
|
17
|
+
/** A call the API must answer with 200; its body. */
|
|
18
|
+
async post(path, body) {
|
|
19
|
+
const answer = await call(path, body);
|
|
20
|
+
if (answer.status !== 200)
|
|
21
|
+
throw refused(path, answer.status, answer.body);
|
|
22
|
+
return answer.body;
|
|
23
|
+
},
|
|
24
|
+
};
|
|
25
|
+
}
|
|
26
|
+
const HOST = /^[a-z0-9](?:[a-z0-9.-]{0,251}[a-z0-9])?$/;
|
|
27
|
+
/** The websites a sign-in or a task may use: 1 to 10 lower-case hostnames, copied. */
|
|
28
|
+
export function allowedHostsOf(hosts) {
|
|
29
|
+
if (!Array.isArray(hosts) || hosts.length < 1 || hosts.length > 10 || hosts.some(host => typeof host !== 'string' || !HOST.test(host))) {
|
|
30
|
+
throw new Error('Computer use needs 1 to 10 lower-case website hostnames in allowedHosts, such as vendorcentral.amazon.com.');
|
|
31
|
+
}
|
|
32
|
+
return [...hosts];
|
|
33
|
+
}
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
import { type ComputerUseTransport } from './computer-use-api.js';
|
|
2
|
+
/** What the person did on the screenshot: the sign-in computer's input actions. */
|
|
3
|
+
export type SignInAction = {
|
|
4
|
+
op: 'click';
|
|
5
|
+
x: number;
|
|
6
|
+
y: number;
|
|
7
|
+
} | {
|
|
8
|
+
op: 'typeText';
|
|
9
|
+
value: string;
|
|
10
|
+
} | {
|
|
11
|
+
op: 'pressKey';
|
|
12
|
+
key: string;
|
|
13
|
+
} | {
|
|
14
|
+
op: 'scroll';
|
|
15
|
+
direction: 'up' | 'down';
|
|
16
|
+
};
|
|
17
|
+
/** The computer's screen: `starting` until its browser is up, then a base64 PNG and the page's address. */
|
|
18
|
+
export type SignInPage = {
|
|
19
|
+
status: 'starting';
|
|
20
|
+
} | {
|
|
21
|
+
status: 'ready';
|
|
22
|
+
screenshot: string;
|
|
23
|
+
url: string;
|
|
24
|
+
};
|
|
25
|
+
/** Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. */
|
|
26
|
+
export declare function startSignIn({ allowedHosts, ...transport }: {
|
|
27
|
+
allowedHosts: readonly string[];
|
|
28
|
+
} & ComputerUseTransport): Promise<string>;
|
|
29
|
+
export declare function showSignIn(handle: string, transport?: ComputerUseTransport): Promise<SignInPage>;
|
|
30
|
+
export declare function sendSignInInput(handle: string, action: SignInAction, transport?: ComputerUseTransport): Promise<void>;
|
|
31
|
+
/** Saves the signed-in browser session as the runtime secret `saveAs`, stops the computer and returns the page it was on. */
|
|
32
|
+
export declare function finishSignIn(handle: string, { saveAs, ...transport }: {
|
|
33
|
+
saveAs: string;
|
|
34
|
+
} & ComputerUseTransport): Promise<{
|
|
35
|
+
url: string;
|
|
36
|
+
}>;
|
|
37
|
+
/** Stops the computer and keeps nothing. */
|
|
38
|
+
export declare function cancelSignIn(handle: string, transport?: ComputerUseTransport): Promise<void>;
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
// Browser sign-in for computer use (data plane ADR 0090): a person signs in
|
|
2
|
+
// to a website on a short-lived computer the app shows as screenshots, and the
|
|
3
|
+
// app saves the browser session as one of its runtime secrets for later
|
|
4
|
+
// computer use tasks. Each call is short, so these run in actions as well as
|
|
5
|
+
// jobs. The computer answers only the app role that started it.
|
|
6
|
+
import { allowedHostsOf, computerUseApi } from './computer-use-api.js';
|
|
7
|
+
import { runtimeSecretLocation, runtimeSecrets } from './secrets.js';
|
|
8
|
+
function computerOf(handle) {
|
|
9
|
+
let computer;
|
|
10
|
+
try {
|
|
11
|
+
computer = JSON.parse(Buffer.from(String(handle), 'base64url').toString('utf8'));
|
|
12
|
+
}
|
|
13
|
+
catch {
|
|
14
|
+
computer = undefined;
|
|
15
|
+
}
|
|
16
|
+
if (typeof computer?.computerId !== 'string' || typeof computer.endpoint !== 'string')
|
|
17
|
+
throw new Error('This is not a sign-in handle from startSignIn.');
|
|
18
|
+
return { computerId: computer.computerId, endpoint: computer.endpoint };
|
|
19
|
+
}
|
|
20
|
+
/** Starts a sign-in computer limited to `allowedHosts`; the handle names it in the calls that follow. */
|
|
21
|
+
export async function startSignIn({ allowedHosts, ...transport }) {
|
|
22
|
+
const hosts = allowedHostsOf(allowedHosts);
|
|
23
|
+
const { computerId, endpoint } = await computerUseApi(transport).post('/computers/start', { allowedHosts: hosts });
|
|
24
|
+
return Buffer.from(JSON.stringify({ computerId, endpoint })).toString('base64url');
|
|
25
|
+
}
|
|
26
|
+
export async function showSignIn(handle, transport = {}) {
|
|
27
|
+
return computerUseApi(transport).post('/computers/page', computerOf(handle));
|
|
28
|
+
}
|
|
29
|
+
export async function sendSignInInput(handle, action, transport = {}) {
|
|
30
|
+
await computerUseApi(transport).post('/computers/input', { ...computerOf(handle), action });
|
|
31
|
+
}
|
|
32
|
+
/** Saves the signed-in browser session as the runtime secret `saveAs`, stops the computer and returns the page it was on. */
|
|
33
|
+
export async function finishSignIn(handle, { saveAs, ...transport }) {
|
|
34
|
+
const computer = computerOf(handle);
|
|
35
|
+
runtimeSecretLocation(saveAs, transport.env);
|
|
36
|
+
const { session, url } = await computerUseApi(transport).post('/computers/finish', computer);
|
|
37
|
+
await runtimeSecrets.put(saveAs, JSON.stringify(session), transport);
|
|
38
|
+
return { url };
|
|
39
|
+
}
|
|
40
|
+
/** Stops the computer and keeps nothing. */
|
|
41
|
+
export async function cancelSignIn(handle, transport = {}) {
|
|
42
|
+
await computerUseApi(transport).post('/computers/stop', computerOf(handle));
|
|
43
|
+
}
|
package/dist/computer-use.d.ts
CHANGED
|
@@ -6,8 +6,18 @@ export interface ComputerUseFile {
|
|
|
6
6
|
export interface ComputerUseOptions {
|
|
7
7
|
/** App-authorized PDFs, at most five files and 2 MiB total. Never supplied by the model. */
|
|
8
8
|
files?: readonly ComputerUseFile[];
|
|
9
|
-
/**
|
|
10
|
-
|
|
9
|
+
/**
|
|
10
|
+
* The runtime secret a sign-in was saved under (finishSignIn's `saveAs`): the task starts signed in,
|
|
11
|
+
* and the refreshed session is saved back to it when the task is done. Never supplied by the model.
|
|
12
|
+
*/
|
|
13
|
+
session?: string;
|
|
14
|
+
/** The websites a task with `secrets` and no `session` may use. */
|
|
15
|
+
allowedHosts?: readonly string[];
|
|
16
|
+
/**
|
|
17
|
+
* Values the computer types into a password field by name, at most 10; the model sees only the names.
|
|
18
|
+
* Never put a credential in the task text.
|
|
19
|
+
*/
|
|
20
|
+
secrets?: Readonly<Record<string, string>>;
|
|
11
21
|
/** The job's runId, or `<job name>/<scheduledAt>` for a scheduled job: one task record per job run. */
|
|
12
22
|
runKey: string;
|
|
13
23
|
/** The job's progress: called on each new step and at least once a minute, which renews the job's lease. */
|
|
@@ -23,6 +33,8 @@ export interface ComputerUseOptions {
|
|
|
23
33
|
export interface ComputerUseResult {
|
|
24
34
|
status: 'done' | 'failed';
|
|
25
35
|
summary: string | null;
|
|
36
|
+
/** The page the browser was on when the task ended; a sign-in page there means the saved sign-in has expired. */
|
|
37
|
+
finalUrl: string | null;
|
|
26
38
|
}
|
|
27
39
|
export declare function createComputerUseTool(options: ComputerUseOptions): ({
|
|
28
40
|
deferLoading?: boolean;
|
package/dist/computer-use.js
CHANGED
|
@@ -1,17 +1,30 @@
|
|
|
1
1
|
// The computer use tool: hands one browser task to Harbour's computer use
|
|
2
2
|
// agent (CUA) and waits for its result. Calls go to the CUA's task API, signed
|
|
3
|
-
// with SigV4 as the runtime's own role (ADR 0078).
|
|
3
|
+
// with SigV4 as the runtime's own role (ADR 0078). A saved sign-in is a runtime
|
|
4
|
+
// secret: its session goes with the task and the refreshed one is saved back
|
|
5
|
+
// to it (ADR 0090).
|
|
4
6
|
import { setTimeout as sleep } from 'node:timers/promises';
|
|
5
7
|
import { jsonSchema, tool } from 'ai';
|
|
6
|
-
import {
|
|
8
|
+
import { allowedHostsOf, computerUseApi, refused } from './computer-use-api.js';
|
|
9
|
+
import { runtimeSecretLocation, runtimeSecrets } from './secrets.js';
|
|
7
10
|
const HEARTBEAT_MS = 60_000;
|
|
8
11
|
// Checks the API did not answer (a throttle, a 5xx, no connection) before the tool gives up: about a minute.
|
|
9
12
|
const MAX_MISSED_CHECKS = 12;
|
|
10
|
-
|
|
11
|
-
const
|
|
12
|
-
if (
|
|
13
|
-
throw new Error('Computer use
|
|
13
|
+
function secretsOf(secrets) {
|
|
14
|
+
const entries = secrets && typeof secrets === 'object' && !Array.isArray(secrets) ? Object.entries(secrets) : [];
|
|
15
|
+
if (entries.length < 1 || entries.length > 10)
|
|
16
|
+
throw new Error('Computer use secrets are 1 to 10 named values.');
|
|
17
|
+
for (const [name, value] of entries) {
|
|
18
|
+
if (!/^[A-Za-z][A-Za-z0-9_-]{0,63}$/.test(name))
|
|
19
|
+
throw new Error(`${JSON.stringify(name)} is not a computer use secret name: up to 64 letters, digits, _ and -, starting with a letter.`);
|
|
20
|
+
if (typeof value !== 'string' || !value || Buffer.byteLength(value) > 4096)
|
|
21
|
+
throw new Error(`The computer use secret ${name} must be a non-empty string of at most 4096 bytes.`);
|
|
14
22
|
}
|
|
23
|
+
return Object.fromEntries(entries);
|
|
24
|
+
}
|
|
25
|
+
export function createComputerUseTool(options) {
|
|
26
|
+
if ('connectionId' in options)
|
|
27
|
+
throw new Error('connectionId is no longer supported; pass session, the runtime secret a sign-in was saved under.');
|
|
15
28
|
const files = options.files?.map(file => ({ ...file }));
|
|
16
29
|
if (files) {
|
|
17
30
|
const names = new Set();
|
|
@@ -30,27 +43,41 @@ export function createComputerUseTool(options) {
|
|
|
30
43
|
names.add(file.name);
|
|
31
44
|
}
|
|
32
45
|
}
|
|
33
|
-
const
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
46
|
+
const { session } = options;
|
|
47
|
+
if (session !== undefined && options.allowedHosts !== undefined)
|
|
48
|
+
throw new Error('Pass session or allowedHosts, not both: a saved sign-in carries its own websites.');
|
|
49
|
+
if (options.allowedHosts !== undefined && options.secrets === undefined)
|
|
50
|
+
throw new Error('allowedHosts is only for a task with secrets and no session.');
|
|
51
|
+
if (options.secrets !== undefined && session === undefined && options.allowedHosts === undefined)
|
|
52
|
+
throw new Error('A task with secrets needs session or allowedHosts: the websites the computer may type them into.');
|
|
53
|
+
const allowedHosts = options.allowedHosts === undefined ? undefined : allowedHostsOf(options.allowedHosts);
|
|
54
|
+
const secrets = options.secrets === undefined ? undefined : secretsOf(options.secrets);
|
|
55
|
+
const transport = { env: options.env ?? process.env, fetch: options.fetch };
|
|
56
|
+
if (session !== undefined)
|
|
57
|
+
runtimeSecretLocation(session, transport.env);
|
|
58
|
+
const api = computerUseApi(transport);
|
|
37
59
|
if (!options.runKey)
|
|
38
60
|
throw new Error('createComputerUseTool needs the job run as runKey.');
|
|
39
|
-
const region = env.ISOMORPH_COMPUTER_USE_REGION || 'us-east-1';
|
|
40
61
|
const now = options.now ?? Date.now;
|
|
41
|
-
const
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
62
|
+
const savedSession = async (name) => {
|
|
63
|
+
const stored = await runtimeSecrets.get(name, transport);
|
|
64
|
+
if (stored === null)
|
|
65
|
+
throw new Error(`No saved sign-in named ${name}.`);
|
|
66
|
+
try {
|
|
67
|
+
return JSON.parse(stored);
|
|
68
|
+
}
|
|
69
|
+
catch {
|
|
70
|
+
throw new Error(`The saved sign-in named ${name} is not a browser session.`);
|
|
71
|
+
}
|
|
46
72
|
};
|
|
47
73
|
return tool({
|
|
48
74
|
description: 'Do one task in a web browser, such as filling in and submitting a form on a website. Describe the whole task in plain words, with the website address and any details it needs. Waits until the task ends and returns its status and a summary.',
|
|
49
75
|
inputSchema: jsonSchema({ type: 'object', properties: { task: { type: 'string', minLength: 1 } }, required: ['task'], additionalProperties: false }),
|
|
50
76
|
execute: async ({ task }) => {
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
77
|
+
await api.post('/tasks', {
|
|
78
|
+
run_key: options.runKey, task, ...(files?.length ? { files } : {}), ...(session === undefined ? {} : { session: await savedSession(session) }),
|
|
79
|
+
...(allowedHosts ? { allowedHosts } : {}), ...(secrets ? { secrets } : {}),
|
|
80
|
+
});
|
|
54
81
|
await options.progress(0, 'Computer use task started.');
|
|
55
82
|
let reported = now();
|
|
56
83
|
let steps = 0;
|
|
@@ -58,14 +85,12 @@ export function createComputerUseTool(options) {
|
|
|
58
85
|
let missed = 0;
|
|
59
86
|
for (;;) {
|
|
60
87
|
if (!cancelSent && await options.cancelled()) {
|
|
61
|
-
|
|
62
|
-
if (cancel.status !== 200)
|
|
63
|
-
throw refused('/tasks/cancel', cancel.status, cancel.body);
|
|
88
|
+
await api.post('/tasks/cancel', { run_key: options.runKey });
|
|
64
89
|
cancelSent = true;
|
|
65
90
|
}
|
|
66
91
|
let check;
|
|
67
92
|
try {
|
|
68
|
-
check = await call('/tasks/check', { run_key: options.runKey });
|
|
93
|
+
check = await api.call('/tasks/check', { run_key: options.runKey });
|
|
69
94
|
}
|
|
70
95
|
catch {
|
|
71
96
|
check = undefined;
|
|
@@ -73,8 +98,11 @@ export function createComputerUseTool(options) {
|
|
|
73
98
|
if (check?.status === 200) {
|
|
74
99
|
missed = 0;
|
|
75
100
|
const answer = check.body;
|
|
76
|
-
if (answer.status === 'done' || answer.status === 'failed')
|
|
77
|
-
|
|
101
|
+
if (answer.status === 'done' || answer.status === 'failed') {
|
|
102
|
+
if (answer.status === 'done' && session !== undefined && answer.session)
|
|
103
|
+
await runtimeSecrets.put(session, JSON.stringify(answer.session), transport);
|
|
104
|
+
return { status: answer.status, summary: answer.summary ?? null, finalUrl: answer.final_url ?? null };
|
|
105
|
+
}
|
|
78
106
|
if (answer.steps > steps || now() - reported >= HEARTBEAT_MS) {
|
|
79
107
|
steps = Math.max(steps, answer.steps);
|
|
80
108
|
await options.progress(Math.min(steps, 99), answer.last_step ? `Computer use step ${steps}: ${answer.last_step}` : `Computer use task ${answer.status}.`);
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
export interface RuntimeSecretsOptions {
|
|
2
|
+
env?: NodeJS.ProcessEnv;
|
|
3
|
+
fetch?: typeof fetch;
|
|
4
|
+
}
|
|
5
|
+
/** Where a runtime secret lives; throws, before any call, for a bad name or an app without runtime secrets. */
|
|
6
|
+
export declare function runtimeSecretLocation(name: string, env?: NodeJS.ProcessEnv): {
|
|
7
|
+
secretId: string;
|
|
8
|
+
region: string;
|
|
9
|
+
};
|
|
10
|
+
export declare const runtimeSecrets: {
|
|
11
|
+
/** Saves `value` under `name`, creating the secret the first time. */
|
|
12
|
+
put(name: string, value: string, options?: RuntimeSecretsOptions): Promise<void>;
|
|
13
|
+
/** The value saved under `name`, or null when there is none. */
|
|
14
|
+
get(name: string, options?: RuntimeSecretsOptions): Promise<string | null>;
|
|
15
|
+
/** Deletes the secret at once, so the same name can be saved again straight away; a missing one is already deleted. */
|
|
16
|
+
delete(name: string, options?: RuntimeSecretsOptions): Promise<void>;
|
|
17
|
+
};
|
package/dist/secrets.js
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
// App runtime secrets (data plane ADR 0090): secrets an app saves, reads and
|
|
2
|
+
// deletes while it runs, such as a saved browser sign-in, an OAuth token a
|
|
3
|
+
// user granted or a key a user typed into the app's settings. Each is one
|
|
4
|
+
// Secrets Manager secret under the app's own `<prefix><org>/<app>/runtime/`
|
|
5
|
+
// path in its cell, which Provision names in ISOMORPH_RUNTIME_SECRETS_PREFIX.
|
|
6
|
+
// The calls are signed as the runtime's own role, as the computer use tool's
|
|
7
|
+
// are (ADR 0078); no AWS SDK client is added.
|
|
8
|
+
import { randomUUID } from 'node:crypto';
|
|
9
|
+
import { runtimeCredentials, signedJsonPost } from './aws.js';
|
|
10
|
+
const NAME = /^[A-Za-z0-9][A-Za-z0-9/_+=.@-]{0,199}$/;
|
|
11
|
+
const MAX_VALUE_BYTES = 64 * 1024;
|
|
12
|
+
/** Where a runtime secret lives; throws, before any call, for a bad name or an app without runtime secrets. */
|
|
13
|
+
export function runtimeSecretLocation(name, env = process.env) {
|
|
14
|
+
if (typeof name !== 'string' || !NAME.test(name))
|
|
15
|
+
throw new Error(`${JSON.stringify(name)} is not a runtime secret name: up to 200 letters, digits and / _ + = . @ -, starting with a letter or digit.`);
|
|
16
|
+
const prefix = env.ISOMORPH_RUNTIME_SECRETS_PREFIX;
|
|
17
|
+
const region = env.ISOMORPH_RUNTIME_SECRETS_REGION;
|
|
18
|
+
if (!prefix || !region)
|
|
19
|
+
throw new Error('This app has no runtime secrets (ISOMORPH_RUNTIME_SECRETS_PREFIX and ISOMORPH_RUNTIME_SECRETS_REGION are not set).');
|
|
20
|
+
return { secretId: prefix + name, region };
|
|
21
|
+
}
|
|
22
|
+
function secretsManager(name, { env = process.env, fetch }) {
|
|
23
|
+
const { secretId, region } = runtimeSecretLocation(name, env);
|
|
24
|
+
const transport = fetch ? { fetch } : {};
|
|
25
|
+
const send = async (action, payload) => {
|
|
26
|
+
const { status, body } = await signedJsonPost({ ...await runtimeCredentials(env, transport), region }, 'secretsmanager', `https://secretsmanager.${region}.amazonaws.com/`, payload, {
|
|
27
|
+
...transport, headers: { 'content-type': 'application/x-amz-json-1.1', 'x-amz-target': `secretsmanager.${action}` },
|
|
28
|
+
});
|
|
29
|
+
if (status === 200)
|
|
30
|
+
return { missing: false, body };
|
|
31
|
+
const answer = body;
|
|
32
|
+
const type = typeof answer?.__type === 'string' ? answer.__type.replace(/^.*#/, '').replace(/:.*$/, '') : '';
|
|
33
|
+
if (type === 'ResourceNotFoundException')
|
|
34
|
+
return { missing: true, body };
|
|
35
|
+
const message = answer?.message ?? answer?.Message;
|
|
36
|
+
throw new Error(`Secrets Manager refused ${action} on the runtime secret ${name} (${status}${type ? ` ${type}` : ''}${typeof message === 'string' ? `: ${message}` : ''}).`);
|
|
37
|
+
};
|
|
38
|
+
return { secretId, send };
|
|
39
|
+
}
|
|
40
|
+
export const runtimeSecrets = {
|
|
41
|
+
/** Saves `value` under `name`, creating the secret the first time. */
|
|
42
|
+
async put(name, value, options = {}) {
|
|
43
|
+
const { secretId, send } = secretsManager(name, options);
|
|
44
|
+
if (typeof value !== 'string' || Buffer.byteLength(value) > MAX_VALUE_BYTES)
|
|
45
|
+
throw new Error(`The runtime secret ${name} must be a string of at most 64 KB.`);
|
|
46
|
+
// Secrets Manager wants each new version's idempotency token from a caller that is not an AWS SDK.
|
|
47
|
+
const saved = await send('PutSecretValue', { SecretId: secretId, SecretString: value, ClientRequestToken: randomUUID() });
|
|
48
|
+
if (saved.missing)
|
|
49
|
+
await send('CreateSecret', { Name: secretId, SecretString: value, ClientRequestToken: randomUUID() });
|
|
50
|
+
},
|
|
51
|
+
/** The value saved under `name`, or null when there is none. */
|
|
52
|
+
async get(name, options = {}) {
|
|
53
|
+
const { secretId, send } = secretsManager(name, options);
|
|
54
|
+
const read = await send('GetSecretValue', { SecretId: secretId });
|
|
55
|
+
return read.missing ? null : read.body.SecretString ?? null;
|
|
56
|
+
},
|
|
57
|
+
/** Deletes the secret at once, so the same name can be saved again straight away; a missing one is already deleted. */
|
|
58
|
+
async delete(name, options = {}) {
|
|
59
|
+
const { secretId, send } = secretsManager(name, options);
|
|
60
|
+
await send('DeleteSecret', { SecretId: secretId, ForceDeleteWithoutRecovery: true });
|
|
61
|
+
},
|
|
62
|
+
};
|