@agentorg/computer 0.0.19 → 0.0.20

package/README.md

@@ -23,10 +23,10 @@ That gives an agent this computer instead: the machine the service runs on, whol
 
 ## This computer and containers
 
-Each computer is this computer, attached with `attach`, or one made in a container of its own with `create`. A container needs Docker on the service's machine. The service follows your `DOCKER_HOST` or current context and never changes them, and comes up without Docker.
+Each computer is this computer, attached with `attach`, or one made in a container of its own with `create`. A service started with `serve --lend` only lends this computer: it makes nothing, `info.made` is empty, and `create` is refused with 409. It listens only on loopback, as this computer is offered only there, and a container it finds in its data folder from before can't be started or removed until a service without `--lend` has the folder. A container needs Docker on the service's machine. The service follows your `DOCKER_HOST` or current context and never changes them, and comes up without Docker.
 
 - **`info` says what the service offers**, each part ready or not and why, in a line a person reads:
-  - `thisComputer`: its capabilities (`about`, `terminal`, `files` and `screen`) and the browsers that can go with it. Its screen isn't reached yet: `This service can't reach the screen yet.`
+  - `thisComputer`: its capabilities (`about`, `terminal`, `files` and `screen`) and the browsers that can go with it. The screen is ready where Cua Driver is installed and granted.
   - `made`, each kind it makes: today a container, ready while Docker answers, with its capabilities and browsers.
 - **A container is refused with 409 where Docker doesn't answer:** `A container needs Docker, which isn't answering on this machine.` One already made isn't started there, and isn't killed there either, since its container couldn't be removed: both are refused with the same words, and it is kept.
 - **A computer with no isolation is no longer made.** `create --isolation none` is refused with 400, and `attach` is what does that now.
@@ -38,6 +38,37 @@ This computer:
 - **`--workspace`, which `attach` needs, is its work folder.** Commands and relative paths start there, and `about` names it. But its files reach the whole machine, as whoever started the service, and so do its commands at `--access terminal`.
 - **`kill` detaches it.** What the service kept for it goes, and nothing of the machine's.
 - **Its browser,** with `--browser`, is a headless one with a profile of its own. It runs the Chrome installed on the machine, or what `serve --browser-name` or `--browser-path` names. `info` says whether that program is there.
+- **Or the person's own browser,** with `--own-browser` (`browser: "own"`): their Chrome or Edge, whichever has remote debugging on, Chrome where both do, as they use it, signed in. Its agent works in it with the same tools, and the person watches it in the same live view.
+  - **The agent has the person's sign-ins, not their open tabs.** It works only in tabs it opens, and in the pages those open: its first call opens a tab of its own, and its tools list and act on its tabs alone. The person's own tabs are left as they are: no dialog of theirs is answered and no file picker of theirs taken over, and nothing that changes the browser as a whole, such as its cookies or its network, is the agent's to do. A page opened with a modifier or middle click has no opener, so it isn't the agent's. Sharing one of their tabs with the agent can come later. The live view shows the person every tab, and takes over no file picker here, so theirs open as they always do.
+  - It is reached through the browser's own attach, from version 144, with nothing installed. The person switches on remote debugging once, on the machine, at `chrome://inspect/#remote-debugging` (`edge://inspect/#remote-debugging` in Edge), with the browser open. The browser then asks them to allow each connection: attaching waits up to 2 minutes for that, and is refused with 409 if it isn't allowed. That and Edge's switch are as their makers describe them; here they have been tried only against a browser started with its remote debugging on, which asks nobody.
+  - `info` says whether it is ready (`browsers`, `kind` `own`), and if not, which browser and what to do on that machine. Attaching with it is refused with 409 until then. `serve --own-browser-data DIR` names the browser's user data folder in place of looking for Chrome's and Edge's. It is connected to only where what listens on its port is the person's own: another user's process can't stand in for it.
+  - The connection changes nothing of the person's: their downloads, colours and focus stay as they have them. It is never closed: when the computer stops or is detached, the browser is only let go, with the tabs the agent opened. When the browser closes, or the person ends the connection, the computer is `failed`.
+- **Its screen,** with `--screen` (`screen: true`), is the machine's own, through [Cua Driver](https://github.com/trycua/cua/tree/main/libs/cua-driver). It acts as the person, whatever `--access` says: its agent can type into any window, a terminal's too. Its agent gets a `screen` endpoint with three tools, and none of the driver's others:
+  - `see`: the whole screen as a picture (`apps` lists its windows), or one window's elements, each with a reference, and a picture of it;
+  - `act`: one thing on the screen (click, double or right click, drag, scroll, type, a key or keys together), aimed at an element's reference or at a point in the picture just seen. Click an element that can't be pressed, such as a text area, by its point; the driver's refusal to press it says so. It acts in the background by default, without moving the person's pointer or taking their focus. Where that can't be done it is refused, and `delivery_mode: "foreground"` brings the window forward and acts there.
+  - `apps`: the running apps and their windows, and bringing one forward. Each window's `app_name`, `title`, `pid`, `window_id`, `z_index` (higher in front) and `bounds` are also given as fields. The bounds are in the screen's points, not the pixels `act` aims in: to aim at a window, see it and use its picture.
+  - On a machine the service only lends (`serve --lend`), the agent acts only in the background: `act` takes no `delivery_mode`, and `apps` takes no window to bring forward, which would take the person's front app and their typing. Where the driver's answer to `act` advises the foreground, the agent is told in plain words instead. What `see` and `apps` say of the screen is never rewritten.
+  - The driver's hints and suggested next calls, which can name its own tools, aren't passed on with any of the three.
+- **The person installs Cua Driver once,** the signed app from cua.ai/download, and on macOS grants it Screen Recording and Accessibility once. AgentComputer never installs or ships it, nor its optional perception extension, which is AGPL. `info` says whether the screen is ready, at Cua Driver 0.49.7 or later, and if not, what to do; attaching with it is refused with 409 until then. `serve --screen-driver PATH` names the program; otherwise it is `cua-driver` on the path, in `~/.local/bin`, or inside `/Applications/CuaDriver.app`.
+- **An app can carry Cua Driver and host it,** as the driver's embedding guide describes, so that the grants are the app's.
+  - The app runs `cua-driver serve --embedded --socket PATH` itself. The service, started with `serve --screen-driver DRIVER --screen-socket PATH`, where DRIVER is the driver inside the app, reaches it there through `cua-driver mcp --embedded`.
+  - The grants are read from that driver, and listed under the outermost app the driver is inside.
+  - Asked for, they are asked for by the app: the service opens it through the system with `--args grant` and each grant to ask for, `screen-recording` or `accessibility`, which it asks for in turn.
+  - Once a grant changes, the app restarts its driver, which reads grants once. That ends the connections of computers using the screen, which then say Cua Driver has stopped.
+- **What the service starts of Cua Driver sends no telemetry.** The person's own Cua Driver app follows its own setting, which `cua-driver telemetry disable` turns off.
+
+**Upgrading from 0.0.19:**
+
+- **A computer's `browser` can be `"own"`,** for this computer attached with the person's own browser, where it was only `true` or `false`. At run time this breaks only for such a computer, in a client that reads `browser` as a boolean: `browser === true` takes it for a computer with no browser, and a strict schema that expects a boolean refuses it. At build time it breaks TypeScript code that puts `browser` into a `boolean`, whatever the computer, as the SDK's type is now `boolean | "own"`. Test it as truthy to know whether a computer has a browser.
+- **A computer has a new `state`, `starting`:** while it is made, started, or brought back as the service starts, until it is running. A client that knows only `running`, `stopped` and `failed` should take it as neither running nor stopped. `list --state running`, and the SDK's `isRunning()`, leave it out.
+- **The service says it is up, with `AgentComputer on …` and its record, before what was running is back.** That is brought back after, all at once, each `starting` until it is running: `Its container is being started.`, or for this computer `It is being started.` A script that starts the service and then uses a computer should wait until its `state` isn't `starting`; meanwhile what needs it running is refused with 409 and its `reason`.
+- **While `create` waits, `list` shows the computer as `starting`;** it showed `running`. `create` and `start` still answer once it is running, except `start` on one already starting, which answers at once, still `starting`.
+- **Every computer has `reason` and `progress`.** `reason` says why a failed one failed, which only its logs said.
+- **Computers carry other new fields,** which a client with a strict schema needs: `screen` and `endpoints.screen`, and in `get`, `tools.screen`. Every running computer's `surfaces` now lists `presence`, and `screen` where it was attached with it. `info` lists the person's own browser, `kind: "own"`, beside `headless`.
+- **`about` has a new row, `Acting now`,** before `CPU`, so `CPU`, `Memory` and `Disk` each come one later.
+- **A file a browser call names,** by `paths` or `filename`, has to be in the computer's workspace or its output folder. The MCP client's roots no longer move or widen that, so a path under a client's root elsewhere is refused.
+- **An image that can't be fetched is refused with 409,** `Its image couldn't be fetched: ` and Docker's words; it was 500.
+- **This computer's own endpoint takes one JSON-RPC request in each POST,** as the browser's does, and refuses a batch with 400. MCP clients send no batches.
 
 **Upgrading from 0.0.18 or earlier:**
 
@@ -55,7 +86,11 @@ A computer in a container:
 - Its image holds AgentExec, Playwright MCP and a headless Chromium. It is `ghcr.io/uniac-ai/agentcomputer/computer` at the service's version; `--image` names another.
 - `--workspace` is mounted at `/workspace`, read-only at `--access read`, and nothing else of the machine is visible. The computer is told that folder's path on the machine and the machine's name, which `about` answers. Docker has to be able to share that folder, and the data folder, where its home is. Without `--workspace`, the service makes one, in the data folder.
 - `--copy-from DIR` (`copyFrom`), used instead of `--workspace`, has the folder the service makes start with a copy of what is in `DIR`, taken once when the computer is made. `DIR` itself is never mounted or changed. Links are copied as they are, and sockets, pipes and devices are left out. It needs `--access` other than `none`.
-- The image is fetched the first time it is needed, which can take minutes: when the first computer in a container is made, a call that then waits for it. `docker pull` it beforehand to spare that wait.
+- **The image is fetched the first time it is needed,** which can take minutes: when the first computer in a container is made or started under this version, a call that then waits for it. `docker pull` it beforehand to spare that wait.
+  - Meanwhile the computer is listed with `state` `starting`. Its `reason` says what it waits on, in a line a person reads with no numbers, `Its image is being fetched.`, and its `progress` how far that has come: `{ "title": "Downloading its image", "done": …, "total": …, "moved": … }`, in bytes, with `total` as far as is known (Docker gives each layer's size as the layer begins, so it can grow) and `moved` the time it last moved: bytes came, or a layer was unpacked. A download that has stopped is still waited on: `moved` says since when. What needs the computer running is refused with 409 and its `reason`.
+  - `create` answers only once the computer is running, with its id and token. To show the download meanwhile, give the computer a name or metadata of your own and find it among the computers listed (`GET /v1/computers`, `Computer.list`): it is there once any `--copy-from` copy is done. One the service brings back when it starts keeps its id: `GET /v1/computers/{computerID}` shows it.
+  - Where Docker is on a socket here, as Docker Desktop and colima are, the image is fetched through Docker's API, which counts the bytes. Where it is elsewhere, or where that can't fetch the image, as from a registry that needs a sign-in, `docker pull` fetches it, which counts nothing: `progress` is null.
+  - When it can't be fetched, making the computer is refused with 409, `Its image couldn't be fetched: ` and Docker's words.
 - It runs as the user who started the service, with no capabilities.
 - Its home, `/home/computer`, is a folder of the machine's, in the computer's own folder in the data folder, mounted as the workspace is. What the agent keeps there stays, the browser's profile with its sign-ins among it. `stop` stops the container and keeps it, and `kill` removes it with the computer's folder.
 - **After an update,** a computer in a container is made again at its first start under a new version of the service, or under another `--image`: its container is removed and made again from the service's image. It keeps its workspace, its home, and its id, token, endpoints and settings. It loses its running commands, as at any restart, and everything else in the container: software installed there, and files outside `/workspace` and `/home/computer`, such as `/tmp`. Keep what must last in those two folders, or in an image of your own. Every computer's answer says the `version` of the program it runs.
@@ -63,26 +98,27 @@ A computer in a container:
 
 ## Access
 
-| `--access` | Tools at the `computer` endpoint                          |
-| ---------- | --------------------------------------------------------- |
-| `read`     | `about`, `read`, `list`, `search`                         |
-| `files`    | Those, and `edit`                                         |
-| `terminal` | Those, and `exec_command`, `write_stdin`, `list_sessions` |
-| `none`     | No `computer` endpoint; needs `--browser`                 |
+| `--access` | Tools at the `computer` endpoint                                   |
+| ---------- | ------------------------------------------------------------------ |
+| `read`     | `about`, `read`, `list`, `search`                                  |
+| `files`    | Those, and `edit`                                                  |
+| `terminal` | Those, and `exec_command`, `write_stdin`, `list_sessions`          |
+| `none`     | No `computer` endpoint; needs a browser, or `--screen` on `attach` |
 
 `read` answers a text file as numbered lines, and a PNG, JPEG, GIF or WebP of up to 3 MiB as the picture itself: an image an MCP client can pass on to a model. The browser's `browser_take_screenshot` answers an image too, but not one of more than 3 MiB, which a `fullPage` screenshot of a long page can be many times over: it answers an error that gives the size and says to take it again without `fullPage`, or of one element. The most counts bytes only: a model may still refuse a smaller picture for its height or width.
 
 `about` tells the agent what a person looking at the computer reads. It takes nothing and answers JSON text, `{"rows": [...]}`, each row a `label` and a `value`, with `"path": true` on a value that is a path and `"caution": true` on one to be read as a caution; a row that is neither has neither. The rows come in this order:
 
-| `label`     | `value`                                                                                   |
-| ----------- | ----------------------------------------------------------------------------------------- |
-| `Where`     | The machine's own name                                                                    |
-| `Folder`    | The workspace's path on that machine, as a path                                           |
-| `Reaches`   | `Read-only files`, `Files`, or `Files and terminal`; with a browser, `, Browser` after it |
-| `Isolation` | `None: it acts as you, on the whole machine`, as a caution, or `A container of its own`   |
-| `CPU`       | Used, of all the machine's CPUs, as a whole percentage: `37%`                             |
-| `Memory`    | Used of the machine's total, counted in 1024s and written GB: `9.2 of 16 GB`              |
-| `Disk`      | Used of total, of the volume the workspace is on, counted in 1000s: `212 of 494 GB`       |
+| `label`      | `value`                                                                                                                                   |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
+| `Where`      | The machine's own name                                                                                                                    |
+| `Folder`     | The workspace's path on that machine, as a path                                                                                           |
+| `Reaches`    | `Read-only files`, `Files`, or `Files and terminal`; with a browser, `, Browser` after it                                                 |
+| `Isolation`  | `None: it acts as you, on the whole machine`, as a caution, or `A container of its own`                                                   |
+| `Acting now` | Who else acted on it in the last 10 s, and on what, the most recent first: `Scout: the files; The person: terminal s_…`, or `Nobody else` |
+| `CPU`        | Used, of all the machine's CPUs, as a whole percentage: `37%`                                                                             |
+| `Memory`     | Used of the machine's total, counted in 1024s and written GB: `9.2 of 16 GB`                                                              |
+| `Disk`       | Used of total, of the volume the workspace is on, counted in 1000s: `212 of 494 GB`                                                       |
 
 - **In a container** the agent's commands find the folder somewhere else, so `Folder` says both and is not marked as a path: `/home/me/Research on the machine, mounted at /workspace in it`. CPU and Memory there are the computer's own; on this computer they are the whole machine's.
 - **The last three** are a reading of its load taken over half a second when it is asked, in the same way the `load` surface reads. They come together, or not at all where a reading can't be taken.
@@ -90,26 +126,26 @@ A computer in a container:
 
 A client that calls a computer's tools can put the commands it starts in a group, and stop them together. At `--access terminal` it sends a word as `agentexec/group` in the `_meta` of an `exec_command` call, and the request `agentexec/group/stop` with `{"group": "<word>"}` on the `computer` endpoint, which ends every running command of that group and answers `{"stopped": <how many>}`. Giving up a call does not stop its command; only that request does. A shell a person opened is in no group. This is AgentExec's, described in [its README](https://github.com/uniac-ai/AgentExec#sessions), and the service passes both through, on this computer and in a container.
 
-In a container the file tools stay inside the workspace, which is a check and not a boundary: the container is the boundary. On this computer they reach the whole machine. Below `terminal`, the browser's `browser_run_code_unsafe` tool is hidden and refused, because it would otherwise be a terminal. The browser's endpoint takes one JSON-RPC request in each POST, as a JSON object, and refuses anything else with 400, a batch among them: it checks each request before passing it on, and passes on only what it read. The service enforces all of this, whatever the client does.
+In a container the file tools stay inside the workspace, which is a check and not a boundary: the container is the boundary. On this computer they reach the whole machine. Below `terminal`, the browser's `browser_run_code_unsafe` tool is hidden and refused, because it would otherwise be a terminal. The browser's endpoint takes one JSON-RPC request in each POST, as a JSON object, and refuses anything else with 400, a batch among them: it checks each request before passing it on, and passes on only what it read. A file a browser call names, such as a snapshot's `filename`, is in the computer's folder, whatever roots the client gives. The service enforces all of this, whatever the client does.
 
 Before 0.0.14 the browser's endpoint passed on a body it couldn't read as one request, unchecked. So on a computer made with `--access none`, `read` or `files` and `--browser`, a caller holding that computer's token could run code in its browser by sending the call in a batch or behind a byte order mark. It needed the computer's own token and a client that sends such a body, which MCP clients don't. 0.0.14 fixes it; a computer in a container made by an earlier version keeps the fault until its container is made again.
 
-The browser has no debug port: a small host process starts it and shares it on a Unix socket in a folder only its user can enter, which the agent's tools and the live view both use. Playwright MCP, which serves the tools, listens on a Unix socket in a second such folder, and no port either. Only the service's user can reach them; callers go through the computer's `browser` endpoint and its token. It lives as long as its computer does: a client that ends its session or goes away leaves the browser and its pages as they are, and the agent's `browser_close` is refused. Only an agent with `--access terminal` can stop it, with `browser_run_code_unsafe` or from its terminal.
+A browser of the computer's own has no debug port: a small host process starts it and shares it on a Unix socket in a folder only its user can enter, which the agent's tools and the live view both use. The person's own browser has the port its own attach opens on loopback, which it asks the person about; the host connects to it once, and shares that connection the same way. Playwright MCP, which serves the tools, listens on a Unix socket in a second such folder, and no port either. Only the service's user can reach them; callers go through the computer's `browser` endpoint and its token. It lives as long as its computer does: a client that ends its session or goes away leaves the browser and its pages as they are, and the agent's `browser_close` is refused. Only an agent with `--access terminal` can stop it, with `browser_run_code_unsafe` or from its terminal.
 
 ## Commands
 
-|                                           |                                                                                                                                        |
-| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
-| `serve`                                   | Run the service: `--listen` (127.0.0.1:7878), `--data`                                                                                 |
-| `up`, `down`, `status`                    | Run the service in the background, stop it, say where it is running                                                                    |
-| `info`                                    | The service's version and what it offers: this computer, and the kinds it makes, each ready or not                                     |
-| `create`                                  | Make a computer: `--access`, `--browser`, `--isolation`, `--workspace` or `--copy-from`, `--name`, `--timeout`, `--metadata KEY=VALUE` |
-| `attach`                                  | Attach this computer: `--access`, `--workspace`, `--browser`, `--name`, `--timeout`, `--metadata KEY=VALUE`                            |
-| `list [--state]`, `get <id>`, `logs <id>` | Look at them                                                                                                                           |
-| `timeout <id> <seconds>`                  | It now stops this long from now                                                                                                        |
-| `stop <id>`, `start <id> [--timeout]`     | Stop it, keeping its folder and token; start it again                                                                                  |
-| `live <id> --surface [--mode]`            | A one-time ticket to open one of its surfaces live; see [Live sessions](#live-sessions)                                                |
-| `kill <id>`                               | Stop it and remove what the service made; this computer, detach it                                                                     |
+|                                           |                                                                                                                                            |
+| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
+| `serve`                                   | Run the service: `--listen` (127.0.0.1:7878), `--data`                                                                                     |
+| `up`, `down`, `status`                    | Run the service in the background, stop it, say where it is running                                                                        |
+| `info`                                    | The service's version and what it offers: this computer, and the kinds it makes, each ready or not                                         |
+| `create`                                  | Make a computer: `--access`, `--browser`, `--isolation`, `--workspace` or `--copy-from`, `--name`, `--timeout`, `--metadata KEY=VALUE`     |
+| `attach`                                  | Attach this computer: `--access`, `--workspace`, `--browser` or `--own-browser`, `--screen`, `--name`, `--timeout`, `--metadata KEY=VALUE` |
+| `list [--state]`, `get <id>`, `logs <id>` | Look at them                                                                                                                               |
+| `timeout <id> <seconds>`                  | It now stops this long from now                                                                                                            |
+| `stop <id>`, `start <id> [--timeout]`     | Stop it, keeping its folder and token; start it again                                                                                      |
+| `live <id> --surface [--mode]`            | A one-time ticket to open one of its surfaces live; see [Live sessions](#live-sessions)                                                    |
+| `kill <id>`                               | Stop it and remove what the service made; this computer, detach it                                                                         |
 
 **In the background.** `up` takes what `serve` takes, runs the service in the background for you, and returns once it answers, printing its `url`, `pid` and version. If one already answers for the data folder it prints that one and starts nothing. `status` prints the same, or exits 1 with none running. `down` ends it and says nothing; its computers stop with it, and start again when it next does. `serve` won't run beside a service that already answers for the same data folder. `--version` prints the program's version, with no service.
 
@@ -141,6 +177,11 @@ The data folder is `--data`, else `$AGENTCOMPUTER_DATA`, else `$XDG_DATA_HOME/ag
 | `POST /v1/computers/{computerID}/start`   | `computer_start`       |
 | `GET /v1/computers/{computerID}/logs`     | `computer_logs`        |
 | `POST /v1/computers/{computerID}/live`    | `computer_live`        |
+| `GET /v1/connection`                      |                        |
+| `GET /v1/acting`                          |                        |
+| `GET /v1/setup`                           |                        |
+| `PUT /v1/setup/{capability}`              |                        |
+| `POST /v1/setup/{capability}/ask`         |                        |
 
 These take the control token, kept in the data folder. A computer's own endpoints take its `accessToken`, and neither token works for the other:
 
@@ -151,6 +192,26 @@ These take the control token, kept in the data folder. A computer's own endpoint
 
 `POST /v1/computers` and `POST /v1/computers/attach` are the only places an `accessToken` is shown. Failures are `{ "code": 404, "message": "..." }`.
 
+- **A client that relies on the service can hold a connection to it:** a WebSocket at `/v1/connection?name=NAME`, with the control token in its `Authorization` header. Nothing is sent either way; the service pings, and drops one that stops answering. `GET /v1/connection` says who holds one: `{ "connected": true, "clients": [{ "name", "since", "local" }] }`, where `name` is its first 100 characters and `local` is whether it connects from the service's machine, false behind `--tls-proxy`.
+- **The person's switches.** Each part of this computer an agent can use (`terminal`, `files`, `screen`, and `browser`, the person's own) has the person's own switch.
+  - Off, agents' new calls to that part refuse with "The person turned off the screen for agents on this Mac" (or "this machine" off macOS), `info` says the same, and attaching with the screen or the person's browser is refused with 409. What an agent already started, such as a command, runs on.
+  - The terminal and the screen each reach the whole machine: with the terminal on, commands read and change any file, whatever the files' switch; with the screen on, the agent acts in any app.
+  - Any browser's calls that reach the machine answer to its switches too: `browser_run_code_unsafe` to the terminal's, and a call that names files, by `paths` or `filename`, to the files'.
+  - The computer's own endpoint on this machine reads each call whole and checks it as it arrives, so it takes one JSON-RPC request in each POST, as the browser's does, and refuses a batch.
+  - A service started with `--lend` starts with every switch off, for the person to turn on. Any other starts with every one on. They are kept in the data folder, so they last across a restart. A switches file that can't be read leaves every one off.
+- **`GET /v1/setup`** gives `{ "capabilities": [...] }`, each part as `{ "capability", "on", "ready", "needs" }`, read afresh each time.
+  - `ready` is whether the part has all it needs, apart from its switch. `needs` lists each thing it needs, always, each with its state.
+  - The terminal and files need nothing, and are always ready; on macOS the Mac asks folder by folder.
+  - The screen needs Cua Driver, and on macOS its `screen-recording` and `accessibility` grants, each `granted`, `not-granted` or `unknown`; it is ready once both are granted. While Cua Driver isn't installed, the one item is `{ "need": "install", "state": "missing" }`. `holder` names whom System Settings lists the grants under: the outermost app Cua Driver is inside, CuaDriver for its own app, or an app that carries the driver.
+  - The browser lists Chrome, then Edge, each `on` (open, with remote debugging on), `off`, or `missing` (not installed); a service given one browser's folder, with `--own-browser-data`, lists just that one as `named`. It is ready with one on. With both on, the agent's is Chrome.
+- **`POST /v1/setup/{capability}/ask`**, with `{ "need": "..." }` or no body, asks the person, on the service's machine, macOS only, for one thing the part needs, or for everything it is missing. It answers once the steps have started, with each part then:
+  - `screen-recording` or `accessibility`, while it is missing: Cua Driver's own grant, its app started first, through the system, so the Mac names that app. Cua Driver asks in its own order, Accessibility first. For a driver an app hosts, the app asks, for that grant alone. Then, missing or not, that grant's own pane in System Settings opens.
+  - `chrome` or `edge`: that browser opens at its inspect page, where remote debugging is switched on. One the service was given only the folder of, the person opens themselves.
+  - With none named: the screen's grant while either is missing; the first browser installed, Chrome before Edge, while none is on.
+  - What the part doesn't need, a browser that isn't installed, or steps that couldn't be started answer 409, saying why.
+- **`PUT /v1/setup/{capability}`** with `{ "on": true }` or `{ "on": false }` sets a switch. Turning it on, or on again, also asks for everything the part is missing, as asking with no need does. Where that couldn't be started, it answers 409, saying why, and the switch stays on.
+- **`GET /v1/acting`** is who is acting now on every computer attached to this machine, in one list, as each one's `presence` surface has it. A computer in a container isn't included: its own `presence` surface has it.
+
 ## SDK
 
 ```ts
@@ -166,7 +227,7 @@ await computer.kill();
 
 It reads the control token from `AGENTCOMPUTER_API_KEY` and the address from `AGENTCOMPUTER_API_URL`, or takes `apiKey` and `apiUrl` as options. `computer.getMcpUrl("browser")` and `computer.getMcpToken()` give an MCP client what it needs, and `Computer.connect(id, { accessToken })` picks up a computer made earlier.
 
-`files.write` makes a new file or fills an empty one; `files.edit` changes one that has content. `files.read` returns the whole text, or fails if a line is longer than the tool can return.
+`files.write` makes a new file or fills an empty one; `files.edit` changes one that has content. `files.read` returns the whole text, or fails if a line is longer than the tool can return. The file calls return the tool's answer alone, without the line on who else just acted there. They count as the agent's calls, so that line isn't given again on the agent's next one.
 
 ## Across a network
 
@@ -232,12 +293,23 @@ A computer with `--browser` has another:
   - **A dialog, a file chooser and a list belong to a page.** A dialog is told to the sockets shown its page until someone answers it, you or the agent. Nothing says when the agent answers a file chooser, so when it answers one every open chooser is taken as closed: press again if yours was.
   - **A client written for the earlier `screencast@1` still connects.** Its `release` is ignored and no `driver` event comes: nobody holds the browser any more.
 
-Every running computer has one more, to watch only:
+A computer attached with `--screen` has another:
+
+- **`screen`** (protocol `screen@1`): the machine's screen, the one the agent sees and acts on. Binary messages are PNG frames of the whole screen, at most 1,280 pixels on their longest side. Text messages are JSON: the service sends `size` and `refused`, and an `operate` socket sends `mouse` (`click`, `double_click`, `right_click`, `drag` or `scroll`), `key` (`keys`: one, or several pressed together, by Cua Driver's names, such as `["enter"]` or `["ctrl", "c"]`) and `text`. `openapi.yaml` has each one's fields.
+  - **A frame is sent only when the picture changed.** The screen is taken twice a second while a socket is open, and not at all while nobody watches. A socket is sent the last frame when it opens, after a `size`, and another `size` comes before a frame of a different size. One more than 4 MiB behind is sent nothing until it catches up.
+  - **Your input is done as you,** through Cua Driver, in the order you sent it. A point is in pixels of the frames, scaled to the screen; keys and text go to the window in front, which the agent may just have changed. What isn't done is answered with `refused`, and why, only to the socket that sent it.
+  - No message is over 65,536 bytes: a larger one closes the socket. Binary messages, and whatever a `watch` socket sends, are dropped.
+
+Every running computer has two more, to watch only:
 
 - **`load`** (protocol `metrics@1`): how busy it is. The service sends a reading every two seconds while a socket is open, the first two seconds after the first socket opens, and the last one at once to a socket that opens later: `{"type":"reading","scope":"machine","cpuUsedPct":37.2,"memUsed":…,"memTotal":…,"diskUsed":…,"diskTotal":…}`, in bytes. `openapi.yaml` has each field, as `LoadReading`.
   - **`scope` says whose numbers they are.** A computer in a container reads its own (`computer`), against the machine's totals, since a container has no limit of its own. This computer has no edge to measure at, so it reads the whole machine's (`machine`); so does a container that can't read its own.
   - **The disk is the volume its workspace is on,** not the workspace's own size.
   - **Nothing depends on it.** Nothing is read while nobody watches or asks, a reading that can't be taken is left out, and a socket that hasn't read the one before is not sent the next.
+- **`presence`** (protocol `presence@1`): who is acting on it now, and on what: a terminal, the files, the browser or the screen. It sends the whole list, `{"type":"presence","acting":[…]}`, when it opens and each time someone starts or stops acting on something. Someone stops 10 seconds after they last acted there. `openapi.yaml` has each field, as `PresenceEvent` and `Acting`, with what counts as acting.
+  - **The agent is told too.** `about` names who else is acting. The answer to the agent's call on something someone else acted on since its last call there ends with a line naming each of them: `The person used this terminal 3 s ago.`, `The person wrote a file here 2 s ago.`, `The person used the browser 4 s ago.` or `The person used the screen 1 s ago.`
+  - **Computers attached to the same machine share its files and its screen.** Each is shown the person and the other computers' agents acting on them, by name, and its agent's line names each one since its last call: `The person wrote a file here 1 s ago. Helper used the files here 2 s ago.` A computer's terminals and browser stay its own; the person's own browser is shared by the computers attached with it, and each shows only its own agent acting there.
+  - **Nothing is refused by it,** for the person or the agent.
 
 `POST /v1/computers/{computerID}/live` with `{ "surface": "terminal", "mode": "operate" }` gives a ticket: a `url` to open once within a minute, which needs no token. A request with an `Origin` header is refused unless `serve --allow-origin` names that origin, so a page can't open one. The SDK does the same with `computer.terminals.connect({ onEvent })`, or gives the ticket alone with `computer.liveSession("files")` or `computer.liveSession("browser", "operate")`, and the CLI with `agentcomputer live <id> --surface terminal --mode operate`.
 
@@ -247,12 +319,12 @@ A shell of your own needs [node-pty](https://github.com/microsoft/node-pty), an
 
 - A container has no CPU or memory limit, and its network is Docker's default, so it reaches the internet and anything else Docker's network does.
 - In a container the browser runs without its own sandbox, which needs privileges a computer doesn't have. The container is the boundary.
-- A browser that stops, or a container that ends by itself, shows the computer as `failed`, and what can no longer be opened live says so in `surfaces`. So does a computer that couldn't be brought back when the service started, for one in a container because Docker didn't answer; its `logs` say why. Stop the computer and start it again.
+- A browser that stops, or a container that ends by itself, shows the computer as `failed`, with why in its `reason`, and what can no longer be opened live says so in `surfaces`. So does a computer that couldn't be brought back when the service started, for one in a container because Docker didn't answer or its image couldn't be fetched. Its `logs` say more. Stop the computer and start it again.
 - A file being written through the `files` surface sits beside its path as `.agentexec-…` until its last part, where a listing, and the agent, can see it. It stays if the service dies meanwhile.
 
 ## Tests
 
-`npm test` runs the built service through the official MCP client, and one scenario through HTTP, MCP, the SDK and the CLI that fails if any operation in the schema isn't reached by each. Set `AGENTCOMPUTER_TEST_IMAGE` to a built image (`npm run build`, then `docker build .`) to run the same suite on containers, and `AGENTCOMPUTER_TEST_SERVICE_IMAGE` to one from `Dockerfile.service` to run the service itself as a container, with `AGENTCOMPUTER_TEST_ROOT` naming a folder Docker can mount if the temporary one isn't. After changing `openapi.yaml`, `npm run generate` writes what comes from it; CI fails if that changes anything. Set `AGENTCOMPUTER_BROWSER_PATH` to a Chromium program, such as Playwright's headless shell, to run the browser tests that are skipped without it.
+`npm test` runs the built service through the official MCP client, and one scenario through HTTP, MCP, the SDK and the CLI that fails if any operation in the schema with an MCP tool isn't reached by each. Set `AGENTCOMPUTER_TEST_IMAGE` to a built image (`npm run build`, then `docker build .`) to run the same suite on containers, and `AGENTCOMPUTER_TEST_SERVICE_IMAGE` to one from `Dockerfile.service` to run the service itself as a container, with `AGENTCOMPUTER_TEST_ROOT` naming a folder Docker can mount if the temporary one isn't. After changing `openapi.yaml`, `npm run generate` writes what comes from it; CI fails if that changes anything. Set `AGENTCOMPUTER_BROWSER_PATH` to a Chromium program, such as Playwright's headless shell, to run the browser tests that are skipped without it.
 
 ## Licence
 

package/dist/THIRD_PARTY_LICENSES.txt

@@ -1,4 +1,4 @@
-@agentorg/exec 0.0.9 (Apache-2.0)
+@agentorg/exec 0.0.10 (Apache-2.0)
 
 Apache License
                            Version 2.0, January 2004
@@ -1114,6 +1114,32 @@ the full license text.
 
 ========================================================================
 
+cross-spawn 7.0.6 (MIT)
+
+The MIT License (MIT)
+
+Copyright (c) 2018 Made With MOXY Lda <hello@moxy.studio>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+========================================================================
+
 eventsource 3.0.7 (MIT)
 
 The MIT License
@@ -1193,6 +1219,26 @@ SOFTWARE.
 
 ========================================================================
 
+isexe 2.0.0 (ISC)
+
+The ISC License
+
+Copyright (c) Isaac Z. Schlueter and Contributors
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
+IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+========================================================================
+
 jose 6.2.17 (MIT)
 
 The MIT License (MIT)
@@ -1245,6 +1291,20 @@ SOFTWARE.
 
 ========================================================================
 
+path-key 3.1.1 (MIT)
+
+MIT License
+
+Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+========================================================================
+
 pkce-challenge 5.0.1 (MIT)
 
 MIT License
@@ -1271,6 +1331,54 @@ SOFTWARE.
 
 ========================================================================
 
+shebang-command 2.0.0 (MIT)
+
+MIT License
+
+Copyright (c) Kevin Mårtensson <kevinmartensson@gmail.com> (github.com/kevva)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+========================================================================
+
+shebang-regex 3.0.0 (MIT)
+
+MIT License
+
+Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+========================================================================
+
+which 2.0.2 (ISC)
+
+The ISC License
+
+Copyright (c) Isaac Z. Schlueter and Contributors
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
+IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+========================================================================
+
 ws 8.22.2 (MIT)
 
 Copyright (c) 2011 Einar Otto Stangvik <einaros@gmail.com>