@allternit/computer-driver 0.1.0 → 0.2.0

package/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## 0.2.0 — 2026-10-10
+
+### Added
+
+- **Contract v2 (`allternit.computer.v2`) structured members.** Typed client methods for all ten driver-backed members — `read_ui`, `act`, `run_batch`, `verify`, `request_human`, `use_credential`, `run_subtask`, `run_parallel`, `run_skill`, `skills` — on the new `ComputerV2Driver` (exported from the package root). Input and result types are generated from `contracts/computer-toolset/allternit-computer-v2.json` (`src/v2-generated.ts`, emitted by `contracts/computer-toolset/generate.mjs`), so they cannot drift from the server contract.
+- **`computer_v2` function tool next to the pixel tool in every adapter**, with the same steering guidance gizzi sends: `computerV2AnthropicTool` + `runAnthropicComputerV2` (anthropic subpath), `openaiComputerV2Tool` + `runOpenAIV2Call`, `geminiComputerV2Declaration` + `runGeminiV2Call`, plus the provider-neutral `computerV2Tool()`.
+- **Approval flow helper.** `client.toolsetWithApproval(id, call, onApproval)` answers a 409 `approval_required` hold (approve + resend with the single-use `approval_grant`) for any member, pixel or structured.
+- **Typed errors.** `ComputerV2Error` (a structured member answered `is_error`), `ComputerBusyError` (423 `computer_busy` / `computer_controlled_elsewhere`), `SandboxRequiredError` (409 `sandbox_required`) and `ComputerConflictError` (409 `computer_conflict`).
+
+### Notes
+
+- Requires the Allternit Driver sidecar on the computer for the structured members (this-device today). The 17 pixel members are unchanged.

package/README.md

@@ -8,6 +8,59 @@ const client = new AllternitComputers({ apiKey: process.env.ALLTERNIT_API_KEY })
 const computer = await client.create({ name: "agent-1" })
 ```
 
+## Structured UI driving (allternit.computer.v2)
+
+Contract v2 adds ten driver-backed structured members — `read_ui`, `act`, `run_batch`, `verify`, `request_human`, `use_credential`, `run_subtask`, `run_parallel`, `run_skill`, `skills` — typed end to end. `ComputerV2Driver` calls them directly; every adapter also exposes them to the model as the `computer_v2` function tool. The structured members need the Allternit Driver on the computer (this-device today).
+
+```ts
+import { ComputerV2Driver } from "@allternit/computer-driver"
+const v2 = new ComputerV2Driver({ client, computerId: computer.id, onApproval: async (a) => askAPerson(a) })
+
+// Read the UI as an element tree — no screenshots.
+const ui = await v2.read_ui({ app: "Safari" })
+const field = ui.elements?.find((e) => e.role === "textfield")
+if (field) await v2.act({ id: field.id, op: "set_value", value: "hello@example.com", version: ui.version })
+
+// Hand a bounded step sequence to the fast decision loop instead of clicking through it yourself.
+const sub = await v2.run_subtask({
+  goal: "Fill the signup form and submit it",
+  inputs: [{ name: "email", value: "hello@example.com" }],
+  success: [{ role: "button", name: "Submit" }],
+})
+if (sub.status !== "done") console.log(sub.status, sub.next)
+```
+
+### Subtask safety statuses
+
+`run_subtask` / `run_skill` / `run_parallel` results end with a `status`:
+
+- `done` — the goal is met.
+- `escalated` — handed back with a `reason` and the current `screen`; continue from there yourself.
+- `needs_confirmation` — the next step (`held_step`) needs the person's confirmation; run it yourself with `act`/`run_batch` and the server asks them.
+- `paused` — the safety monitor paused the subtask; stop and call `request_human`.
+- `denied` — the step isn't allowed on this computer (app/domain lists or a credential binding); find another way.
+- `use_api` — make the named API/MCP call yourself (`api.tool`); the screen is untouched.
+- `failed` — ran out of steps, budget or a hard error.
+
+### The computer_v2 function tool in a model loop
+
+Each adapter exposes the structured members with the same steering guidance gizzi sends:
+
+```ts
+import { computerV2AnthropicTool, runAnthropicComputerV2 } from "@allternit/computer-driver/anthropic"
+import { openaiComputerV2Tool, runOpenAIV2Call } from "@allternit/computer-driver"
+import { geminiComputerV2Declaration, runGeminiV2Call } from "@allternit/computer-driver"
+
+// Anthropic: put computerV2AnthropicTool() next to the toolset in tools, and answer
+// tool_use blocks whose name is "computer_v2" with runAnthropicComputerV2({ client, computerId, onApproval }, name, input).
+// OpenAI: put openaiComputerV2Tool() in tools and answer function_call items with
+// runOpenAIV2Call({ client, computerId }, call.call_id, JSON.parse(call.arguments)).
+// Gemini: put geminiComputerV2Declaration() in functionDeclarations and answer
+// functionCall parts with runGeminiV2Call({ client, computerId }, part.functionCall.name, part.functionCall.args).
+```
+
+`computerV2Tool()` returns the provider-neutral `{ name, description, schema }` if you wire tools yourself.
+
 ## Claude (`computer_toolset_20260801` / `browser_toolset_20260801`)
 
 Requires `@anthropic-ai/sdk` >= 0.132.0 (optional peer).
@@ -18,7 +71,7 @@ const toolset = new AllternitComputerToolset({
   client, computerId: computer.id,
   onApproval: async (approval) => askAPerson(approval), // server-held risky calls
 })
-// pass `toolset` to client.beta.messages.toolRunner({ tools: [toolset], ... })
+// pass `toolset` to client.beta.messages.toolRunner({ tools: [toolset, computerV2AnthropicTool()], ... })
 ```
 
 Every member runs on the server through `POST /v1/computers/{id}/toolset`. When the server holds a call (409 `approval_required`), `onApproval` decides: `true` approves it and resends with the grant, `false` returns the held result to the model as an error. The SDK's own `confirm` option still runs first. Approving with an API key needs the project's `approval_mode` set to `api_key`.
@@ -40,6 +93,13 @@ const { functionResponse, inlineData } = await runGeminiCall({ client, computerI
 
 Gemini's 0–999 coordinates are sent as `coordinate_space: "normalized_1000"`; the server scales them.
 
-The OpenAI and Gemini adapters let `ApprovalRequiredError` propagate; catch it, call `client.approve()`, and resend the action.
+The OpenAI and Gemini pixel adapters let `ApprovalRequiredError` propagate; catch it, call `client.approve()`, and resend the action.
+
+## Errors
 
-Errors: `ApprovalRequiredError` (409, carries `approval` and `result`) and `AllternitApiError` (everything else, with `status`, `code`, `type`).
+- `ApprovalRequiredError` — 409 `approval_required`; carries `approval` (id, member, `approve_url`) and the held `result`. `client.toolsetWithApproval(id, call, onApproval)` approves and resends with the `approval_grant` for you.
+- `ComputerV2Error` — a structured member answered `is_error: true`; carries the member name and raw `result`.
+- `ComputerBusyError` — 423 `computer_busy` / `computer_controlled_elsewhere`.
+- `SandboxRequiredError` — 409 `sandbox_required` (the call needs a sandbox computer).
+- `ComputerConflictError` — 409 `computer_conflict` (another subtask or lease conflicts).
+- `AllternitApiError` — everything else, with `status`, `code`, `type`.

package/dist/anthropic.d.ts

@@ -23,3 +23,21 @@ export declare class AllternitBrowserToolset extends BetaAbstractBrowserToolset2
     constructor(o: DriverOptions & Omit<BetaBrowserToolsetOptions, "browserState"> & Partial<Pick<BetaBrowserToolsetOptions, "browserState">>);
     protected execute(ctx: BetaToolsetCallContext, name: string, input: unknown): Promise<BetaBrowserMemberResult>;
 }
+/**
+ * The `computer_v2` function tool for Anthropic models: the ten structured
+ * members (read_ui … skills) next to the native pixel toolset. The Anthropic
+ * wire hook only rewrites tools carrying the v1 pixel marker, so this passes
+ * through untouched.
+ */
+export declare function computerV2AnthropicTool(): {
+    name: string;
+    description: string;
+    input_schema: Record<string, unknown>;
+};
+/**
+ * Run one `computer_v2` function call (`{action, ...fields}`) and return the
+ * tool-result content blocks, like any toolset member. Throws ToolError on a
+ * failed action; without `onApproval`, a held call (409 approval_required)
+ * resolves its held result as an error, the same rule as the toolsets.
+ */
+export declare function runAnthropicComputerV2(o: DriverOptions, name: string, input: unknown): Promise<any[]>;

package/dist/anthropic.js

@@ -3,6 +3,7 @@
 // member on an Allternit hosted computer via POST /v1/computers/{id}/toolset.
 import { BetaAbstractBrowserToolset20260801, BetaAbstractComputerToolset20260801, ToolError, } from "@anthropic-ai/sdk/helpers/beta/toolsets";
 import { ApprovalRequiredError, resultImage, resultText, } from "./client.js";
+import { computerV2Tool, runComputerV2Member } from "./v2.js";
 /**
  * Default SDK `confirm`: let the call through. The Allternit server is the gate: it applies the
  * project's member policy and holds risky calls with a 409, which `onApproval` answers. Pass your
@@ -128,3 +129,29 @@ export class AllternitBrowserToolset extends BetaAbstractBrowserToolset20260801
         }
     }
 }
+// ------------------------------------------------------------------ computer_v2
+/**
+ * The `computer_v2` function tool for Anthropic models: the ten structured
+ * members (read_ui … skills) next to the native pixel toolset. The Anthropic
+ * wire hook only rewrites tools carrying the v1 pixel marker, so this passes
+ * through untouched.
+ */
+export function computerV2AnthropicTool() {
+    const t = computerV2Tool();
+    return { name: t.name, description: t.description, input_schema: t.schema };
+}
+/**
+ * Run one `computer_v2` function call (`{action, ...fields}`) and return the
+ * tool-result content blocks, like any toolset member. Throws ToolError on a
+ * failed action; without `onApproval`, a held call (409 approval_required)
+ * resolves its held result as an error, the same rule as the toolsets.
+ */
+export async function runAnthropicComputerV2(o, name, input) {
+    const { action, ...fields } = (input ?? {});
+    if (!action)
+        throw new ToolError("The computer_v2 call needs an action.");
+    const res = await runComputerV2Member({ client: o.client, computerId: o.computerId, onApproval: o.onApproval }, action, fields);
+    if (res.is_error)
+        throw new ToolError(toAnthropicBlocks(res));
+    return toAnthropicBlocks(res);
+}

package/dist/client.d.ts

@@ -92,6 +92,18 @@ export declare class ApprovalRequiredError extends AllternitApiError {
     readonly result: ToolsetResult;
     constructor(err: Partial<ApiErrorBody>, approval: Approval, result: ToolsetResult, body: unknown);
 }
+/** 423 computer_busy / computer_controlled_elsewhere: someone else holds the computer right now; retry shortly or request_human. */
+export declare class ComputerBusyError extends AllternitApiError {
+    constructor(status: number, err: Partial<ApiErrorBody>, body: unknown);
+}
+/** 409 sandbox_required: the call only runs on a sandbox (cloud/bot) computer. */
+export declare class SandboxRequiredError extends AllternitApiError {
+    constructor(status: number, err: Partial<ApiErrorBody>, body: unknown);
+}
+/** 409 computer_conflict: the call conflicts with another subtask or lease on the computer. */
+export declare class ComputerConflictError extends AllternitApiError {
+    constructor(status: number, err: Partial<ApiErrorBody>, body: unknown);
+}
 export interface ClientOptions {
     /** Project key (alt_live_… / alt_test_…). Defaults to ALLTERNIT_API_KEY. */
     apiKey?: string;
@@ -117,6 +129,14 @@ export declare class AllternitComputers {
     delete(id: string): Promise<Computer>;
     /** Run one toolset member. Action failures resolve with is_error:true; a held call throws ApprovalRequiredError. */
     toolset(id: string, call: ToolsetCall): Promise<ToolsetResult>;
+    /**
+     * Run one toolset member, answering a 409 approval_required hold. When the
+     * server holds the call and `onApproval` returns true, the approval is
+     * granted (POST /approvals/{id}) and the same call resent with the
+     * single-use `approval_grant`. When `onApproval` is missing or returns
+     * false, the held result resolves (is_error:true) so the model sees it.
+     */
+    toolsetWithApproval(id: string, call: ToolsetCall, onApproval?: (approval: Approval) => boolean | Promise<boolean>): Promise<ToolsetResult>;
     schema(id: string, toolset?: ToolsetName): Promise<Record<string, unknown>>;
     events(id: string, q?: {
         after?: string;

package/dist/client.js

@@ -28,6 +28,41 @@ export class ApprovalRequiredError extends AllternitApiError {
         this.result = result;
     }
 }
+/** 423 computer_busy / computer_controlled_elsewhere: someone else holds the computer right now; retry shortly or request_human. */
+export class ComputerBusyError extends AllternitApiError {
+    constructor(status, err, body) {
+        super(status, err, body);
+        this.name = "ComputerBusyError";
+    }
+}
+/** 409 sandbox_required: the call only runs on a sandbox (cloud/bot) computer. */
+export class SandboxRequiredError extends AllternitApiError {
+    constructor(status, err, body) {
+        super(status, err, body);
+        this.name = "SandboxRequiredError";
+    }
+}
+/** 409 computer_conflict: the call conflicts with another subtask or lease on the computer. */
+export class ComputerConflictError extends AllternitApiError {
+    constructor(status, err, body) {
+        super(status, err, body);
+        this.name = "ComputerConflictError";
+    }
+}
+/** Maps a toolset error body onto the typed error for its code, if one exists. */
+function typedApiError(status, err, body) {
+    switch (err.code) {
+        case "computer_busy":
+        case "computer_controlled_elsewhere":
+            return new ComputerBusyError(status, err, body);
+        case "sandbox_required":
+            return new SandboxRequiredError(status, err, body);
+        case "computer_conflict":
+            return new ComputerConflictError(status, err, body);
+        default:
+            return new AllternitApiError(status, err, body);
+    }
+}
 export class AllternitComputers {
     baseUrl;
     #apiKey;
@@ -63,6 +98,26 @@ export class AllternitComputers {
     toolset(id, call) {
         return this.#req("POST", `/v1/computers/${enc(id)}/toolset`, call);
     }
+    /**
+     * Run one toolset member, answering a 409 approval_required hold. When the
+     * server holds the call and `onApproval` returns true, the approval is
+     * granted (POST /approvals/{id}) and the same call resent with the
+     * single-use `approval_grant`. When `onApproval` is missing or returns
+     * false, the held result resolves (is_error:true) so the model sees it.
+     */
+    async toolsetWithApproval(id, call, onApproval) {
+        try {
+            return await this.toolset(id, call);
+        }
+        catch (e) {
+            if (!(e instanceof ApprovalRequiredError))
+                throw e;
+            if (!onApproval || !(await onApproval(e.approval)))
+                return e.result;
+            await this.approve(id, e.approval.id);
+            return this.toolset(id, { ...call, approval_grant: e.approval.id });
+        }
+    }
     schema(id, toolset = "computer") {
         return this.#req("GET", `/v1/computers/${enc(id)}/toolset/schema${qs({ toolset })}`);
     }
@@ -97,7 +152,7 @@ export class AllternitComputers {
         if (res.status === 409 && err.code === "approval_required" && json?.approval) {
             throw new ApprovalRequiredError(err, json.approval, json.result ?? { is_error: true, content: [] }, json);
         }
-        throw new AllternitApiError(res.status, err, json);
+        throw typedApiError(res.status, err, json);
     }
 }
 /** Text of a result, joined. */

package/dist/gemini.d.ts

@@ -1,4 +1,4 @@
-import { type AllternitComputers, type ToolsetCall, type ToolsetResult } from "./client.ts";
+import { type AllternitComputers, type Approval, type ToolsetCall, type ToolsetResult } from "./client.ts";
 export interface GeminiFunctionCall {
     name: string;
     args?: Record<string, unknown>;
@@ -29,4 +29,28 @@ export interface GeminiStepResult {
 }
 /** Run one Gemini function call and return the parts Gemini expects back. */
 export declare function runGeminiCall(o: GeminiAdapterOptions, fc: GeminiFunctionCall): Promise<GeminiStepResult>;
+/**
+ * The `computer_v2` function declaration for Gemini models: the ten structured
+ * members next to the `computer_use` declaration. Route matching
+ * `functionCall` parts through runGeminiV2Call.
+ */
+export declare function geminiComputerV2Declaration(): {
+    name: string;
+    description: string;
+    parameters: Record<string, unknown>;
+};
+export interface GeminiV2AdapterOptions extends GeminiAdapterOptions {
+    /** Called when the server holds the call (409 approval_required). Without it the ApprovalRequiredError propagates, like the pixel adapter. */
+    onApproval?: (approval: Approval) => boolean | Promise<boolean>;
+}
+export interface GeminiV2StepResult {
+    /** A `functionResponse` part for the next turn. */
+    functionResponse: {
+        name: string;
+        response: Record<string, unknown>;
+    };
+    error?: ToolsetResult;
+}
+/** Run one `computer_v2` function call (`{action, ...fields}`) and return the function response part. */
+export declare function runGeminiV2Call(o: GeminiV2AdapterOptions, name: string, input: unknown): Promise<GeminiV2StepResult>;
 export {};

package/dist/gemini.js

@@ -1,7 +1,8 @@
 // Gemini computer_use adapter: runs one predefined function call on an Allternit
 // hosted computer. Gemini coordinates are 0–999 normalized, so every call is sent
 // with coordinate_space "normalized_1000" and the server scales to the screen.
-import { resultImage } from "./client.js";
+import { resultImage, resultText } from "./client.js";
+import { computerV2Tool, runComputerV2Member } from "./v2.js";
 const pt = (x, y) => [Math.round(Number(x)), Math.round(Number(y))];
 const key = (text) => ({ member: "key", input: { text } });
 const typed = (text) => ({ member: "type", input: { text } });
@@ -79,3 +80,28 @@ export async function runGeminiCall(o, fc) {
         ...(error ? { error } : {}),
     };
 }
+// ------------------------------------------------------------------ computer_v2
+/**
+ * The `computer_v2` function declaration for Gemini models: the ten structured
+ * members next to the `computer_use` declaration. Route matching
+ * `functionCall` parts through runGeminiV2Call.
+ */
+export function geminiComputerV2Declaration() {
+    const t = computerV2Tool();
+    return { name: t.name, description: t.description, parameters: t.schema };
+}
+/** Run one `computer_v2` function call (`{action, ...fields}`) and return the function response part. */
+export async function runGeminiV2Call(o, name, input) {
+    const { action, ...fields } = (input ?? {});
+    if (!action)
+        throw new Error("The computer_v2 call needs an action.");
+    const base = { client: o.client, computerId: o.computerId };
+    const res = o.onApproval
+        ? await runComputerV2Member({ ...base, onApproval: o.onApproval }, action, fields)
+        : await o.client.toolset(o.computerId, { toolset: "computer", member: action, input: fields });
+    const text = resultText(res);
+    const response = { result: text };
+    if (res.is_error)
+        response.error = text || "The action failed.";
+    return { functionResponse: { name, response }, ...(res.is_error ? { error: res } : {}) };
+}

package/dist/index.d.ts

@@ -1,3 +1,4 @@
 export * from "./client.ts";
+export * from "./v2.ts";
 export * from "./openai.ts";
 export * from "./gemini.ts";

package/dist/index.js

@@ -2,5 +2,6 @@
 // The Anthropic toolsets live on the "@allternit/computer-driver/anthropic" subpath
 // so this entry never loads the optional @anthropic-ai/sdk peer.
 export * from "./client.js";
+export * from "./v2.js";
 export * from "./openai.js";
 export * from "./gemini.js";

package/dist/openai.d.ts

@@ -1,4 +1,4 @@
-import { type AllternitComputers, type ToolsetCall, type ToolsetResult } from "./client.ts";
+import { type AllternitComputers, type Approval, type ToolsetCall, type ToolsetResult } from "./client.ts";
 type Pt = {
     x: number;
     y: number;
@@ -67,4 +67,32 @@ export interface OpenAIStepResult {
 }
 /** Run one OpenAI `computer_call` (its `call_id` and `action`) and return the output item. */
 export declare function runOpenAIAction(o: OpenAIAdapterOptions, callId: string, action: OpenAIComputerAction): Promise<OpenAIStepResult>;
+/**
+ * The `computer_v2` function tool for the Responses API: the ten structured
+ * members next to the `computer_use_preview` tool. Register it in `tools`;
+ * route matching `function_call` items through runOpenAIV2Call.
+ */
+export declare function openaiComputerV2Tool(): {
+    type: "function";
+    name: string;
+    description: string;
+    parameters: Record<string, unknown>;
+    strict: false;
+};
+export interface OpenAIV2AdapterOptions extends OpenAIAdapterOptions {
+    /** Called when the server holds the call (409 approval_required). Without it the ApprovalRequiredError propagates, like the pixel adapter. */
+    onApproval?: (approval: Approval) => boolean | Promise<boolean>;
+}
+export interface OpenAIV2StepResult {
+    /** Ready for the Responses API `input`: `{type:"function_call_output", call_id, output}`. */
+    output: {
+        type: "function_call_output";
+        call_id: string;
+        output: string;
+    };
+    /** The failed action result, if the action failed. */
+    error?: ToolsetResult;
+}
+/** Run one `computer_v2` function call (`{action, ...fields}`) and return the `function_call_output` item. */
+export declare function runOpenAIV2Call(o: OpenAIV2AdapterOptions, callId: string, input: unknown): Promise<OpenAIV2StepResult>;
 export {};

package/dist/openai.js

@@ -1,6 +1,7 @@
 // OpenAI computer-use adapter: runs one `computer_call` action on an Allternit
 // hosted computer and returns a `computer_call_output` with the next screenshot.
 import { resultImage } from "./client.js";
+import { computerV2Tool, runComputerV2Member } from "./v2.js";
 const KEYS = {
     CTRL: "ctrl", CONTROL: "ctrl", ALT: "alt", OPTION: "alt", SHIFT: "shift", CMD: "super", META: "super", SUPER: "super",
     WIN: "super", ENTER: "Return", RETURN: "Return", ESC: "Escape", ESCAPE: "Escape", TAB: "Tab", SPACE: "space",
@@ -85,3 +86,28 @@ export async function runOpenAIAction(o, callId, action) {
         ...(error ? { error } : {}),
     };
 }
+// ------------------------------------------------------------------ computer_v2
+/**
+ * The `computer_v2` function tool for the Responses API: the ten structured
+ * members next to the `computer_use_preview` tool. Register it in `tools`;
+ * route matching `function_call` items through runOpenAIV2Call.
+ */
+export function openaiComputerV2Tool() {
+    const t = computerV2Tool();
+    return { type: "function", name: t.name, description: t.description, parameters: t.schema, strict: false };
+}
+/** Run one `computer_v2` function call (`{action, ...fields}`) and return the `function_call_output` item. */
+export async function runOpenAIV2Call(o, callId, input) {
+    const { action, ...fields } = (input ?? {});
+    if (!action)
+        throw new Error("The computer_v2 call needs an action.");
+    const base = { client: o.client, computerId: o.computerId };
+    const res = o.onApproval
+        ? await runComputerV2Member({ ...base, onApproval: o.onApproval }, action, fields)
+        : await o.client.toolset(o.computerId, { toolset: "computer", member: action, input: fields });
+    const text = res.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("\n");
+    return {
+        output: { type: "function_call_output", call_id: callId, output: text || "Done." },
+        ...(res.is_error ? { error: res } : {}),
+    };
+}