@vashp2029/glovebox 0.0.0-stage → 0.3.0

package/docs/backend-protocol.md

@@ -0,0 +1,118 @@
+# Backend protocol
+
+A backend is the native piece for one OS. The scripting layer talks to it with
+JSON lines: one request `{"id", "method", "params"}` per line, one reply
+`{"id", "result"}` or `{"id", "error": {"message", "code"}}`, and unsolicited
+events `{"event": "escape"}`. On macOS the transport is the helper's Unix
+socket; a new backend may use a socket, a named pipe, or stdio.
+
+All coordinates are global screen points with a top-left origin. The
+scripting layer owns the image-pixel mapping: it picks each window's image
+size and converts targets to screen points before calling the backend. The
+one place a backend sees image dimensions is `screenshot`, which receives the
+`width` and `height` to capture at and must return an image of exactly that
+size. A backend does not choose a scale of its own.
+
+The backend enforces approval: every method that names a `pid` (or launches
+an app) must pass the policy in `config.json` before it acts, and refuses with
+a coded error otherwise. Methods that only read the app list or the backend's
+own state are not gated: `hello`, `ping`, `permissions`, `listApps`,
+`resolveApp`, `forget`, `cursor`. See `design.md` and `SECURITY.md`.
+
+| Method | Params | Result |
+|---|---|---|
+| `hello` | `client` (name shown in dialogs and the menu) | `{version, platform, pid}` |
+| `ping` | | `{version, platform, pid, background}` |
+| `permissions` | `prompt`, `inputMonitoring` | `{accessibility, screenRecording, escapeWatcher, inputMonitoring}` |
+| `listApps` | | `[{name, bundleId, pid?, path?, running, active?}]` |
+| `resolveApp` | `query` (name, bundle id, `pid:N`, path) | one app |
+| `launch` | `bundleId`, `path`, `name`, `activate` | the running app |
+| `activate` | `pid` | |
+| `isActive` | `pid` | `{active}` |
+| `windows` | `pid` | `[{index, title, frame: [x,y,w,h], scale, focused, main, minimized, subrole?, windowId?}]` |
+| `axTree` | `pid`, `window`, `maxNodes`, `maxDepth`, `maxText`, `menus` | `{window, nodes, truncated}` |
+| `hitTest` | `pid`, `x`, `y` | `{index, role, actions, frame}` or `{}` |
+| `elementFrame` | `pid`, `index` | `{frame}` |
+| `actionNames` | `pid`, `index` | `{actions}` |
+| `performAction` | `pid`, `index`, `action` | |
+| `setValue` | `pid`, `index`, `value`, `attribute?` | |
+| `focusElement` | `pid`, `index` | |
+| `selectText` | `pid`, `index`, `start`, `length` | |
+| `keys` | `pid`, `chords: [{key, mods}]`, input options | input result |
+| `type` | `pid`, `text`, input options | input result |
+| `click` | `pid`, `x`, `y`, `button`, `count`, `mods`, input options | input result |
+| `scroll` | `pid`, `x`, `y`, `dy`, `dx`, input options | input result |
+| `drag` | `pid`, `path: [[x, y], ...]`, `button`, input options | input result |
+| `move` | `pid` (for the gate), `x`, `y` | |
+| `cursor` | `enabled`, `hide`, `x`, `y`, `windowId` | `{enabled}` |
+| `makeKey` | `pid`, `windowId` | `{keyed}` |
+| `screenshot` | `pid`, `windowId`, `width`, `height`, `format`, `quality` | `{data (base64), width, height, mimeType}` |
+| `forget` | `pid` | |
+
+`cursor` draws the agent cursor (a decoration; clients ignore its failures).
+`move` moves the real pointer. `makeKey` makes a window key inside its app
+without activating the app. `forget` drops the element table kept for `pid`.
+The scripting layer does not call `move`, `makeKey`, or `forget` today; a
+port may leave them out. `format` is `png` or `jpeg`; `quality` is 0 to 1.
+
+`ping.background` reports which private functions the macOS helper found, as
+booleans: `postEventRecord`, `postToPid`, `setWindowLocation`,
+`setIntegerField`, `processSerialNumber`. Background input needs
+`postEventRecord`, `setIntegerField`, and `processSerialNumber`.
+`glovebox doctor` prints this; it is informational. Clients decide to fall
+back from the error codes below, not from `ping`.
+
+`permissions` returns `inputMonitoring` (listening for key events, which
+Escape-to-stop needs on macOS) next to `accessibility` and `screenRecording`.
+`prompt: true` asks the OS to prompt; with `inputMonitoring: true` it also
+asks for Input Monitoring. `escapeWatcher` says whether an Escape watcher is
+running.
+
+## Background input
+
+`keys`, `type`, `click`, `scroll`, and `drag` take `mode: "background"` with
+the target `windowId` (from `windows`) to deliver input without activating
+the app or moving the user's pointer. `windowLocation: "screen"` switches the
+event location from window-local to screen coordinates. Without `mode`, input
+goes out on the foreground route: the client activates the app first. A
+background result is `{route, focusRestored}`, where `route` is `background`
+or `background-skylight` (Chromium and Electron apps), and `focusRestored` is
+true if the target took focus and the backend gave it back to the user's app.
+
+A backend with no background route should answer `mode: "background"` with
+`background_unavailable`; the client then falls back to foreground.
+
+## Errors
+
+Replies carry `{message, code}`. Codes the macOS helper uses:
+
+| Code | Meaning |
+|---|---|
+| `blocked` | The app is blocked by default (terminals, password managers, ...). |
+| `denied` | On the `deny` list, or the user chose Deny. |
+| `not_allowed` | `approval` is `allowlist` and the app is not on `allow`. |
+| `background_unavailable` | The private functions background input needs are missing. |
+| `background_unconfirmed` | The window could not be made key in the background. |
+| `no_window` | No such window, or it is gone or not capturable. |
+| `no_frame` | The element has no on-screen frame. |
+| `stale_index` | The element index is not in the table; read the tree again. |
+| `action_failed` | An accessibility action or attribute write failed. |
+| `activate_failed` | The app would not come forward. |
+| `not_found` | No app matches the query, or it cannot be located to launch. |
+| `not_running` | No running app has that `pid`. |
+| `permission` | A needed OS permission (Screen Recording) is missing. |
+| `bad_key` | A key name has no key code. |
+| `bad_request` | A parameter is missing or invalid, or the method is unknown. |
+| `failed` | Anything else. |
+
+Clients act on `background_unavailable` and `background_unconfirmed` by
+falling back to the foreground route; every other code is an error.
+
+Node shape in `axTree`: `{i, d, role, subrole?, title?, value?, desc?, help?,
+placeholder?, id?, frame?, actions?, focused?, selected?, disabled?,
+settable?}`. `i` must be stable for the same element for as long as the
+backend runs. `role` uses the macOS AX names (`AXButton`, `AXTextField`, ...);
+other backends map their native roles onto those.
+
+Key names are xdotool keysyms after parsing (`Return`, `Tab`, `Page_Down`,
+single characters); modifiers are `ctrl`, `shift`, `alt`, `super`, `fn`.

package/docs/cli.md

@@ -0,0 +1,150 @@
+# CLI reference
+
+`glovebox` is one executable (`bin/glovebox.mjs`). This page lists every
+subcommand and flag, the environment variables, the config file, the two MCP
+tools, and how the helper and daemon start and stop. For attaching an agent
+see `agents.md`; for the scripting API see `api.md`.
+
+Run `glovebox` with no arguments, `help`, `-h`, or `--help` to print usage.
+A failing command exits with status 1.
+
+## Subcommands
+
+| Command | What it does |
+|---|---|
+| `glovebox setup [--rebuild] [--restart]` | Builds Glovebox Helper if it is missing or its source changed, then walks through Accessibility and Screen Recording, one at a time, and offers Input Monitoring. Exits 1 if the first two are not granted. macOS only. |
+| `glovebox signing [--create]` | Reports whether the local signing certificate exists. With `--create`, makes it in the login keychain. Run `setup --rebuild` afterwards to re-sign. macOS only. |
+| `glovebox doctor` | Checks Node (20.10+), the config, the helper build, that the helper answers, background input, Accessibility, Screen Recording, and Input Monitoring. Exits 1 on any `FAIL`. |
+| `glovebox mcp` | Runs the stdio MCP server (tools `js`, `js_reset`). |
+| `glovebox run [code] [-f file] [-s name] [-t title] [--timeout ms]` | Runs a script in a persistent CLI session and prints the result. |
+| `glovebox reset [-s name]` | Drops a CLI session's bindings. |
+| `glovebox stop` | Stops the CLI daemon and every session in it. |
+| `glovebox smoke [--full]` | End-to-end test on Calculator. See `testing.md`. |
+| `glovebox allow <app>...` | Adds apps to `allow`, and removes them from `deny`. |
+| `glovebox deny <app>...` | Adds apps to `deny`, and removes them from `allow`. |
+| `glovebox revoke <app>...` | Removes apps from both lists. |
+| `glovebox approval <ask\|allowlist\|allow-all>` | Sets `approval` in the config. |
+| `glovebox config` | Prints the effective configuration as JSON: defaults, then the config file, then `GLOVEBOX_APPROVAL`. |
+| `glovebox log [-n N]` | Prints the last N (default 20) lines of the action log. |
+| `glovebox version` | Prints the version. Also `--version` and `-v`. |
+| `glovebox daemon` | Internal. Runs the CLI daemon; `run` starts it when needed. Do not call it directly. |
+
+### Flags
+
+| Flag | Applies to | Meaning |
+|---|---|---|
+| `-f`, `--file path` | `run` | Read the script from a file. |
+| `-s`, `--session name` | `run`, `reset` | Session name. Default `default`. Each name has its own bindings. `stop` ignores it. |
+| `-t`, `--title text` | `run` | A few words naming the step, written to the action log as `step`. |
+| `--timeout ms` | `run` | Time limit for the script. Default is `timeoutMs` from the config (120000). |
+| `--rebuild` | `setup` | Rebuild the helper even if its source is unchanged. Quits a running helper first. |
+| `--restart` | `setup` | Quit the running helper so it relaunches with new grants (needed after enabling Input Monitoring). |
+| `--create` | `signing` | Create the certificate `glovebox local signing` if it does not exist. |
+| `--full` | `smoke` | Also run the background click, drag, and scroll checks in TextEdit. |
+| `-n N` | `log` | Number of lines. |
+
+Notes:
+
+- `run` takes code from the arguments (joined with spaces), else from
+  `-f file`, else from stdin. With none of them it exits with an error. Text
+  results print to stdout; screenshots are saved under
+  `~/.glovebox/screenshots/` and the path is printed. Exit status is 1 if the
+  script failed or was stopped.
+- `run` starts the daemon if none is listening. `reset` and `stop` do not; with
+  no daemon they print `No daemon running.`
+- `allow`, `deny`, and `revoke` match names and bundle ids case-insensitively,
+  rewrite `config.json`, and do not affect approvals a running session already
+  holds. Use `glovebox reset` or `js_reset` to make a session re-check.
+- `approval allow-all` prints a warning: every app except the blocked ones is
+  then controllable without asking.
+- `setup` and `signing` are macOS only; elsewhere they exit with an error.
+
+## Environment variables
+
+| Variable | Effect |
+|---|---|
+| `GLOVEBOX_HOME` | State directory. Default `~/.glovebox`. Holds `config.json`, `actions.log`, `screenshots/`, `daemon.sock`, `helper.sock`, and `Glovebox Helper.app`. The helper receives it when glovebox launches it. |
+| `GLOVEBOX_APPROVAL` | Overrides `approval` (`ask`, `allowlist`, `allow-all`) without editing the file. glovebox passes it to the helper when it launches it; a helper that is already running keeps the value it started with. |
+| `GLOVEBOX_SOCKET` | Path of the helper's socket. Default `$GLOVEBOX_HOME/helper.sock`. Also passed to the helper at launch. It does not move the CLI daemon's socket. |
+| `GLOVEBOX_CLIENT` | Name of the agent, shown in the approval dialog and menu-bar item and written to the log as `client`. `glovebox run` sends it when it creates a session; the default is `glovebox CLI (<session>)`. Over MCP the name is the client's own `clientInfo.name`. |
+| `GLOVEBOX_DEBUG` | When set, the session process's stderr is copied to the host's stderr. Useful with `glovebox mcp`; the CLI daemon runs detached with no stderr. |
+
+glovebox also sets `GLOVEBOX_IMAGE_MODE` (`inline` over MCP, `file` over the
+CLI) on its session processes. It is internal.
+
+## Config file
+
+Location: `$GLOVEBOX_HOME/config.json` (default `~/.glovebox/config.json`),
+created with mode 0600 by `allow`, `deny`, `revoke`, `approval`, or the
+"Always Allow" button. A missing file means all defaults; a file that is not
+valid JSON is an error. Any subset of keys may be set. Nested objects are
+merged key by key; arrays and scalars replace the default.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `approval` | `"ask"` | `ask`: native dialog on first use of each app per session. `allowlist`: never prompt; only apps in `allow` work. `allow-all`: no prompts; blocked apps still need `allow`. |
+| `allow` | `[]` | App names or bundle ids that are always approved. Also unlocks blocked apps. |
+| `deny` | `[]` | App names or bundle ids that are always refused. Wins over `allow`. |
+| `input` | `"background"` | `background`: input reaches apps without focusing them, with a foreground fallback. `foreground`: always bring the app forward. |
+| `foregroundApps` | `[]` | App names or bundle ids that always use the foreground route. |
+| `cursor` | `true` | Draw the agent cursor where actions land. `false` turns it off. |
+| `screenshot.maxLongSide` | `1568` | Longest side of a window image, in pixels. |
+| `screenshot.maxPixels` | `1150000` | Pixel budget of a window image. |
+| `screenshot.format` | `"png"` | `png` or `jpeg`. |
+| `screenshot.quality` | `80` | JPEG quality as a percentage. Ignored for PNG. |
+| `ax.maxNodes` | `800` | Elements per `getAXState` read. |
+| `ax.maxDepth` | `40` | Tree depth per read. |
+| `ax.maxTextLength` | `160` | Longest label or value kept, in characters. |
+| `log.enabled` | `true` | Write `actions.log`. |
+| `log.text` | `false` | Log typed text instead of only its length. |
+| `daemon.idleMinutes` | `60` | The CLI daemon exits after this long without requests. `0` never exits. |
+| `timeoutMs` | `120000` | Default script time limit for `js` and `run`. |
+| `settleMs` | `80` | Pause after each action, in milliseconds. |
+
+`screenshot.maxLongSide` and `maxPixels` set the image scale; see "Coordinates
+and images" in `design.md`. Check order for each app: `deny`, `allow`, the
+blocked list (`SECURITY.md`), then `approval`.
+
+When changes apply: the session process reloads the config before every call,
+and the helper reads it on every approval check. The CLI daemon reads it once
+at startup, so `timeoutMs` and `daemon.idleMinutes` need `glovebox stop`.
+
+## MCP tools
+
+`glovebox mcp` speaks MCP over stdio and exposes two tools. The session starts
+on the first `js` call and lives until `js_reset` or the connection closes.
+
+| Tool | Parameters | Notes |
+|---|---|---|
+| `js` | `code` (string, required), `title` (string), `timeout_ms` (integer) | Runs `code` in the persistent session. `title` names the step in the action log. `timeout_ms` defaults to `timeoutMs`; a value that is not a positive integer is ignored. Result content is text, plus inline images for screenshots; `isError` is true if the script threw or was stopped. |
+| `js_reset` | none | Restarts the session and drops every binding. |
+
+On timeout the script is asked to stop at its next action and the result says
+`Stopped: the script ran longer than N ms.` If it has not stopped five seconds
+later, the session process is killed and the bindings are lost.
+
+## Helper and daemon lifecycle
+
+Glovebox Helper (macOS):
+
+- `setup` builds it into `$GLOVEBOX_HOME/Glovebox Helper.app`. It is rebuilt
+  only when its source changes; until then, or after an upgrade, clients fail
+  with a message to run `glovebox setup`.
+- The first client request launches it through LaunchServices; no agent has to
+  start it. It listens on `helper.sock`, mode 0600, and refuses other users.
+- While at least one client is connected it shows a menu-bar item with the
+  connected sessions and their approved apps, and the entries Stop Running
+  Scripts, Revoke Session Approvals, Hide Agent Cursor, and Quit Glovebox
+  Helper.
+- It quits by itself after 30 minutes with no connected clients, checked once a
+  minute. The next client relaunches it. Quit Glovebox Helper, `setup
+  --restart`, and `setup --rebuild` also stop it.
+
+CLI daemon:
+
+- `glovebox run` starts it on first use, detached, listening on
+  `$GLOVEBOX_HOME/daemon.sock` (mode 0600).
+- Each session name is its own session process, created by the first `run`
+  that names it.
+- It exits after `daemon.idleMinutes` of no requests, or on `glovebox stop`.
+  Its sessions go with it.

package/docs/design.md

@@ -0,0 +1,212 @@
+# Design
+
+This is the design glovebox follows. It records what we chose and why, so
+changes can argue with the reasons rather than the code.
+
+## The bar
+
+Measured against a real Codex computer-use trace (786 calls in one session,
+median script about 130 characters), the properties that made that loop cheap:
+
+1. One tool call holds a whole step: several actions, then one read.
+2. The session persists. A handle bound in one call exists in the next, and
+   the model can define its own helpers.
+3. The default read is accessibility text that diffs after the first read.
+4. Screenshots are an explicit choice, never a side effect of an action.
+5. Clicks land in the same pixel space as the image the model saw.
+6. Pixel actions and element actions are both cheap, because some apps
+   (Citrix, canvases) expose a single accessibility node.
+7. No documentation dump in tool results, and no idle teardown that throws
+   away bindings.
+
+The shapes we compared against:
+
+| | Codex (`@oai/sky`) | Claude computer use | Orca | glovebox |
+|---|---|---|---|---|
+| Calls per step | 1 script | 1 per action, or a fixed batch list | 1 CLI call per action | 1 script |
+| Persistent bindings | yes | no | no | yes |
+| Default read | AX text, diffed | screenshot plus AX summary | AX text plus screenshot | AX text, diffed |
+| Model-defined helpers | yes | no | no | yes |
+| Pixel space | screenshot pixels | screenshot frame | window coordinates | window image pixels |
+| Background input | yes | yes | partial | yes (keys, clicks, drag, scroll; foreground fallback) |
+| Drawn agent cursor | yes | yes | no | yes |
+| Works without vendor app | no | no | no | yes |
+
+Claude's tools and Orca's CLI both make the model pay a round trip per
+action, and every state read carries an image or a full tree. Codex's REPL is
+the right shape but is tied to OpenAI's signed binaries and service.
+
+## Interface: one persistent JavaScript session
+
+We expose a JavaScript session, not a set of action tools, because it is the
+only shape that gets properties 1, 2, and 6 at once. Every agent we target
+writes JavaScript fluently, and the surface stays close to the one Codex
+models already know (`getApp`, `click`, `pressKey`, `typeText`,
+`getAXState`), with `cua` kept as an alias for `gb`.
+
+The same session is reachable two ways:
+
+- **MCP** (`glovebox mcp`): two tools, `js` and `js_reset`. The tool
+  description is the cheat sheet (about 1.8 KB, sent once with `tools/list`).
+  `help()` prints the full reference only when asked.
+- **CLI** (`glovebox run`): a background daemon keeps named sessions alive,
+  so agents that can only run shell commands get the same persistence.
+  Images go to files and the path is printed.
+
+### Persistence
+
+A cell is rewritten (with acorn) so that top-level `const`, `let`, `class`,
+and `import` become assignments to context-level `var`s, function
+declarations are hoisted, and the body runs in an async function. The last
+expression is the result. This is what Node's REPL does, without its
+problem that a rejected top-level `await` is printed and swallowed instead of
+failing the call.
+
+The session runs in its own process (the kernel), so the host can stop a
+runaway script. A timeout first sets an abort flag that every action checks;
+if the script still has not stopped after five seconds, the kernel is killed
+and the result says the bindings were lost. MCP sessions live as long as the
+connection. CLI sessions live until `glovebox reset`, `glovebox stop`, or an
+idle limit (an hour by default).
+
+## Reads and output
+
+`getAXState` prints one line per element: index, role, label, value, id,
+flags, and non-obvious actions, indented by depth. The format is close to
+Codex's on purpose. Element indices are stable for the session, because the
+backend keeps a table of the elements it has seen (`CFEqual` on macOS), so a
+diff only shows real changes: `~` changed, `+` added, then a list of removed
+indices. Frames are off by default (they are tokens most steps do not need)
+and on with `{frames: true}`.
+
+Reads emit into the result by default and return their value; `{emit:
+false}` returns only. Actions emit nothing.
+
+## Coordinates and images
+
+Image space is per window: `(0, 0)` is the window's top-left corner, and one
+image pixel is `1 / s` screen points, where
+
+    s = min(backing scale, maxLongSide / max(w, h), sqrt(maxPixels / (w * h)))
+
+with defaults `maxLongSide = 1568` and `maxPixels = 1.15 M`. The scale depends
+only on the window size and display, so screenshots, `frames`, `center()`, and
+`click([x, y])` agree even when no screenshot was taken. The screenshot is
+captured at exactly that size, so there is no resize without a matching
+transform. The header of every read prints the image size.
+
+## Input delivery
+
+Input goes to apps in the background, the way Codex computer use does it:
+the user's frontmost app keeps focus and the real cursor never moves.
+
+- A plain left click on a pressable element (button, checkbox, link, menu
+  item) is sent as an accessibility press.
+- Keys and typing: the helper makes the target window key *inside its app*
+  without activating the app, using two private SkyLight event records
+  (`SLPSPostEventRecordTo`: a focus record and a make-key pair, 248 bytes
+  each). It confirms through Accessibility that the window is key, then posts
+  the key events to the process (`SLEventPostToPid`). Plain
+  `CGEventPostToPid` is not enough: Calculator dropped those keys when another
+  app was frontmost, because a background app has no key window.
+- Mouse clicks, drags, and scrolls: a focus record for the target window only,
+  then a `mouseMoved` and the button events posted to the process, each
+  stamped with the target pid and window (event fields 40, 51, 58, 91, 92)
+  and a window-local location (`CGEventSetWindowLocation`). AppKit apps get
+  the public `CGEventPostToPid`; Chromium/Electron apps get the SkyLight route
+  plus an off-screen primer click. Scroll events go through both routes.
+  Background events carry no click count unless one is set, and AppKit
+  ignores a mouse-down without it.
+- A focus guard notes the frontmost app before each background action and
+  gives it focus back if the target took it anyway.
+- Each private function is looked up at runtime. If one is missing (a future
+  macOS may remove it), the helper says so and glovebox falls back to the
+  foreground route: bring the app forward once per call, then post events.
+  On that route, switching to another app stops the script. `glovebox doctor`
+  reports whether background input is available, and `app.lastRoute` shows
+  which route the last action took.
+- The record layouts and event fields come from open-source tools that do the
+  same: yabai, trycua's cua-driver, and Peekaboo (all MIT). Codex's service
+  links only public frameworks and resolves these calls by hashed name at
+  runtime, and Claude desktop's helper references the same calls, so this is
+  the established approach, not a novel hack. It is still private API.
+- A drawn agent cursor (a borderless, click-through window ordered just above
+  the target window, not above everything) animates to each action's point.
+  It is included in screen recordings and hides after a few seconds idle.
+- Pixel targets outside the window image are refused, so a bad coordinate
+  cannot click into another app.
+- While any agent is connected the helper shows a menu-bar item listing the
+  sessions and their approved apps, with Stop Running Scripts, Revoke Session
+  Approvals, Hide Agent Cursor, and Quit.
+
+Measured on macOS 27 with Firefox frontmost throughout (`glovebox smoke
+--full`): background keys in Calculator, a pixel click that placed TextEdit's
+caret at the start of the text, a drag that selected a line, and a scroll
+that moved the scroll bar, with the frontmost app unchanged.
+
+## Approval and safety
+
+Computer use can click anything, so approval is part of the product:
+
+- The gate lives in the helper, the process that holds the macOS grants, not
+  in the scripting layer. A client cannot skip it by talking to the socket.
+- Default `approval: "ask"`: the first action in each app shows a native
+  dialog naming the agent (from the MCP `clientInfo`) and the app: Allow for
+  This Session, Always Allow (persisted to `config.json`), or Deny.
+- `allowlist` mode never prompts. `allow-all` is an explicit opt-in.
+- Terminals, password managers, Keychain Access, System Settings, Script
+  Editor, Automator, and the helper itself are blocked by default and never
+  offered in a dialog. Only an explicit `glovebox allow` unlocks one. The
+  full list is in `SECURITY.md`.
+- Switching to another app stops the running script. A script brings an app
+  forward at most once per call; if the app is no longer in front later in
+  the same call, the user took over, and the script stops rather than taking
+  focus back. (An earlier version re-activated before every key and fought
+  the user for focus.)
+- With Input Monitoring granted, Escape pressed by the user (glovebox marks
+  its own events) stops the running script. In testing on macOS 27, an
+  Accessibility-only global key monitor saw no key events, so Escape is
+  documented as needing Input Monitoring.
+- Every action is appended to `~/.glovebox/actions.log` as one JSON object:
+  `ts`, `client`, `step` (the call's title, when given), then the action's own
+  fields (`app`, `action`, and for example `target`, `keys`, or `length`).
+  Typed text is logged as a length unless `log.text` is true.
+- The helper's socket is mode 0600 and checks the peer's uid.
+
+## Signing and permissions
+
+macOS ties Accessibility, Screen Recording, and Input Monitoring grants to the
+helper's designated requirement. An ad-hoc signature's requirement is the
+binary's hash, so every rebuild lost the grants (and stale entries kept
+showing as on). `glovebox signing --create` makes a self-signed code-signing
+certificate in the login keychain; signed with it, the requirement becomes
+"identifier + certificate", and grants survive rebuilds. The certificate
+does not need to be trusted. Verified by rebuilding twice without re-granting.
+
+## The macOS backend
+
+Glovebox Helper is a small Swift app (`native/macos/GloveboxHelper.swift`)
+built locally by `glovebox setup` with the Xcode command line tools. It is
+signed with the local certificate from `glovebox signing --create` when one
+exists, and ad hoc otherwise. It runs as its own app, launched through LaunchServices, so
+macOS attributes Accessibility and Screen Recording to "Glovebox Helper"
+rather than to whatever terminal or agent started it. One grant covers every
+agent. (The first prototype ran the native code as a child process; macOS
+then charges the permission to the parent, and some hosts disclaim
+responsibility for their children, so the grant never applies.)
+
+It uses AXUIElement for the tree and actions, CGEvent for input, and
+ScreenCaptureKit for window capture, all public APIs. Background input
+(see Input delivery) also uses private SkyLight functions, looked up at
+runtime, with a foreground fallback when they are missing. It speaks JSON
+lines over `~/.glovebox/helper.sock`. See `backend-protocol.md`.
+
+## What we did not do
+
+- **Wrap OpenAI's Sky service.** It works on a Mac with ChatGPT installed, but
+  only by launching OpenAI's signed binaries to satisfy the service's
+  sender check. That is fragile across updates, depends on a vendor install,
+  and works around an access control, which is a poor fit for a public tool.
+- **One tool per action.** Every action becomes a model round trip.
+- **Screenshots after every action.** Images are the most expensive thing in
+  the context; the model asks when it needs one.

package/docs/recipes.md

@@ -0,0 +1,162 @@
+# Recipes
+
+Short scripts an agent can paste into one `js` call (or `glovebox run`). They
+use only the API in `api.md`.
+
+Only recipes 1 and 2, the Calculator and TextEdit flows, match what `glovebox
+smoke` has exercised on a real machine (see `testing.md`). Every other recipe
+is marked "not yet run on a real machine": it follows the documented API and
+the code, but nobody has run it. Treat the details, such as app names and menu
+labels, as things to check against the tree.
+
+## 1. Calculator: one batched step
+
+Bind the app, press keys, read once. This is the loop `glovebox smoke` runs.
+
+```js
+const calc = await gb.getApp("Calculator");
+await calc.pressKey("Escape");
+await calc.typeText("6*7=");
+await calc.getAXState();   // the display line shows value="42"
+```
+
+Clicking by element index works in a later call, because `calc` persists. Find
+the index in the tree (`14 button "7"` means index 14), then:
+
+```js
+await calc.click(14);
+await calc.getAXState();   // prints only what changed
+```
+
+## 2. TextEdit: click into the text and type
+
+Uses the tree for the text area's box, a pixel click inside it, then typing.
+Assumes TextEdit already has a document window open. This follows the
+`glovebox smoke --full` flow.
+
+```js
+const te = await gb.getApp("TextEdit");
+const tree = await te.getAXState({ frames: true, disableDiffing: true, emit: false });
+const [, x, y] = tree.match(/text area[^\n]*@(\d+),(\d+)/);
+await te.click([Number(x) + 8, Number(y) + 10]);   // inside the text area
+await te.typeText("Hello from glovebox\n");
+await te.getAXState();
+```
+
+## 3. Open a menu item through the AX tree
+
+Not yet run on a real machine. There is no menu API. `menus: true` adds the
+menu bar to the tree, and menu bar items and menu items are pressed through
+accessibility like buttons. This clicks the bar item, reads again, then
+clicks the item.
+
+```js
+const app = await gb.getApp("TextEdit");
+const find = (tree, role, label) => {
+  const m = tree.match(new RegExp(`^\\s*(\\d+) ${role} "${label}"`, "m"));
+  if (!m) throw new Error(`No ${role} "${label}" in the tree`);
+  return Number(m[1]);
+};
+const clickMenu = async (app, menu, item) => {
+  let tree = await app.getAXState({ menus: true, disableDiffing: true, emit: false });
+  await app.click(find(tree, "menu bar item", menu));
+  tree = await app.getAXState({ menus: true, disableDiffing: true, emit: false });
+  await app.click(find(tree, "menu item", item));
+};
+await clickMenu(app, "Edit", "Select All");
+await app.getAXState();
+```
+
+## 4. Pixel fallback for Citrix and canvases
+
+Not yet run on a real machine. When the tree is one node, take a screenshot,
+read coordinates off the image, and click `[x, y]`. Pixels are the
+screenshot's own: `(0, 0)` is the window's top-left corner.
+
+```js
+const app = await gb.getApp("Citrix Viewer");   // any app with a one-node tree
+const shot = await app.getScreenshot();         // the image comes back in the result
+console.log(shot.width, shot.height);
+await app.click([412, 230]);                    // a point you read off the image
+await app.typeText("hello");
+await app.getScreenshot();                      // look again before the next step
+```
+
+Re-take the screenshot if the window was resized: a click outside the window
+image throws instead of landing in another app.
+
+## 5. Define a helper once, reuse it
+
+Not yet run on a real machine. Functions and constants persist across calls
+until `js_reset`, so a repeated lookup becomes one name.
+
+```js
+const clickLabel = async (app, label) => {
+  const tree = await app.getAXState({ emit: false, disableDiffing: true });
+  const m = tree.match(new RegExp(`^\\s*(\\d+) button "${label}"`, "m"));
+  if (!m) throw new Error(`No button "${label}"`);
+  await app.click(Number(m[1]));
+};
+```
+
+Later calls:
+
+```js
+const calc = await gb.getApp("Calculator");
+for (const label of ["7", "Add", "1", "Equals"]) await clickLabel(calc, label);
+await calc.getAXState();
+```
+
+## 6. Set a field's value directly
+
+Not yet run on a real machine. A line with `settable` in its flags can take
+`setValue`, which is faster and more exact than typing. Read it back to check.
+
+```js
+const tree = await app.getAXState({ disableDiffing: true });
+// e.g.  9 text field "Name" (settable)
+await app.setValue(9, "Ada Lovelace");
+await app.getAXState();
+```
+
+If the app ignores the new value, fall back to `await app.focus(9)` then
+`await app.typeText("Ada Lovelace")`. `help("setValue")` prints the reference.
+
+## 7. Handle a "Stopped:" result
+
+Not yet run on a real machine. A result that starts with `Stopped:` means the
+user took over (switched apps, pressed Escape with Input Monitoring granted,
+or the script hit its time limit). Do not retry or continue; tell the user and
+ask. Inside a script, never swallow it (`clickLabel` is from recipe 5):
+
+```js
+try {
+  for (const label of ["7", "Add", "1", "Equals"]) await clickLabel(calc, label);
+} catch (error) {
+  if (error.name === "Stopped") throw error;   // let the result say Stopped:
+  console.log(`click failed: ${error.message}`);
+}
+```
+
+Continue only after the user asks, and start with a fresh read:
+`await calc.getAXState({ disableDiffing: true })`.
+
+## 8. Force foreground for an app that ignores background input
+
+Not yet run on a real machine. If background clicks or keys do nothing (some
+Java, Tk, and game windows), bring the app forward for those actions. A plain
+click on a button is pressed through accessibility and never needs this; it
+matters for mouse clicks, keys, typing, scroll, and drag.
+
+```js
+const app = await gb.getApp("Some Java App");
+await app.click([120, 80], { foreground: true });
+await app.typeText("hello", { foreground: true });
+app.lastRoute;   // "foreground"
+```
+
+To make every action on an app use the foreground route, pass
+`gb.getApp(name, {foreground: true})` on the call that first binds it, or ask
+the user to add the app to `foregroundApps` in the config (`cli.md`). In
+foreground mode a script brings the app forward once per call and stops if
+the user switches away.