@vashp2029/glovebox 0.0.0-stage → 0.3.0

package/CHANGELOG.md

@@ -0,0 +1,66 @@
+# Changelog
+
+## 0.3.0 (2026-11-03)
+
+- First npm release, as `@vashp2029/glovebox`. CI runs the unit tests on every
+  push and pull request.
+- A clearer error when the helper was built from different source than the
+  running glovebox, including what a rebuild does to macOS permissions.
+- Fix: a read with `{emit: false}` no longer becomes the diff baseline, so
+  the next printed read shows every change since the last printed one.
+  Before, changes made before a silent read never appeared in a diff.
+- `app.waitForChange()` and `app.settle()`: wait silently for a late update
+  instead of polling from the script.
+- Fix: the default window ignores tooltips, popovers, and Finder's rename
+  field, so reads and screenshots no longer land on a 50-pixel window.
+- Fix: diffs survive a window title change, a `{frames: true}` read, and
+  elements that only moved deeper, instead of reprinting the whole tree.
+- Fix: a read that is also returned or passed to `console.log` prints once.
+- Fix: a raw click on an open menu's item works in the background, and any
+  click on a menu item waits for the menu to close before the next action.
+- Fix: keys reach Finder's rename field in the background, and `setValue` on a
+  Finder file name fails with the rename steps instead of only relabeling.
+- Fix: horizontal background scroll sets the point and fixed-point deltas
+  some apps read.
+- `settleMs` is in the documented defaults; `npm test` works on Node 22;
+  the npm package includes `SECURITY.md` and `CHANGELOG.md`.
+
+## 0.2.0 (2026-10-27)
+
+- Background input by default: keys, typing, clicks, drags, and scrolls reach
+  an app without bringing it forward or moving the real cursor (private
+  SkyLight event records and per-process events, with a foreground fallback).
+- A drawn agent cursor shows where each action lands; Hide Agent Cursor in
+  the menu-bar item, or `"cursor": false` in the config.
+- `glovebox signing --create`: a local signing certificate so macOS keeps the
+  helper's permissions across rebuilds.
+- `glovebox setup` asks for one permission at a time and waits for each.
+- `glovebox smoke --full` checks background click, drag, and scroll in TextEdit
+  and that the frontmost app never changed.
+- `input`, `foregroundApps`, and `cursor` config keys; `app.lastRoute`.
+
+### Upgrading
+
+If the helper was signed ad hoc (you did not run `glovebox signing --create`),
+rebuilding it makes macOS treat it as a new app. Reset the old grants and run
+setup again:
+
+```bash
+for s in Accessibility ScreenCapture ListenEvent; do
+  tccutil reset $s io.github.vpatel9202.glovebox.helper
+done
+glovebox setup
+```
+
+## 0.1.0 (2026-10-27)
+
+First release.
+
+- One persistent JavaScript session, reachable over MCP (`js`, `js_reset`) and
+  the CLI (`glovebox run`, backed by a daemon).
+- macOS backend: Glovebox Helper.app (Swift, public APIs) with accessibility
+  trees, stable element indices, diffs, window screenshots, keyboard and mouse
+  input, and background key delivery.
+- Image-pixel coordinates shared by screenshots, frames, and clicks.
+- Per-app approval dialogs enforced in the helper, blocked-by-default apps,
+  Escape to stop, and an action log.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vash Patel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,239 @@
-# Temporary Holding Version
+# glovebox
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![CI](https://github.com/vpatel9202/glovebox/actions/workflows/test.yml/badge.svg)](https://github.com/vpatel9202/glovebox/actions/workflows/test.yml)
+[![npm](https://img.shields.io/npm/v/@vashp2029/glovebox.svg)](https://www.npmjs.com/package/@vashp2029/glovebox)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Platform: macOS](https://img.shields.io/badge/platform-macOS-lightgrey.svg)](#status)
+
+**Computer use for coding agents on macOS: one JavaScript call per step,
+accessibility diffs instead of screenshots, and your screen stays yours.**
+
+<!-- demo: docs/demo.gif goes here once recorded -->
+
+```js
+const calc = await gb.getApp("Calculator");
+await calc.pressKey("Escape");
+await calc.typeText("6*7=");
+await calc.getAXState();
+```
+
+```
+Calculator — window 0 "Calculator" — image 460x816 — changes since last read (~ changed, + added):
+~					8 scroll area id=StandardInputView;value:42
+~						9 text "Edit field" value="42"
+```
+
+On the next call, `calc` is still bound. An agent writes a few lines per step
+instead of making one tool call per click, and reads what changed instead of a
+fresh picture of the window.
+
+- **A script per step, not a tool call per click.** Bindings and helper
+  functions persist across calls, so one call can click, type, loop, and read.
+- **Diffs, not screenshots.** The default read is the accessibility tree as
+  text, and later reads show only changes. Screenshots and pixel clicks are
+  there when the tree is empty (Citrix, canvases).
+- **Background by default.** Keys, clicks, drags, and scrolls reach the app
+  while you keep working in another one. Your cursor stays put, and a drawn
+  cursor shows what the agent is doing.
+
+In an [author-run test](#benchmark) of six setups on 12 tasks (2 runs each),
+glovebox had the lowest total cost, $0.57 against $1.41 to $5.67, and the
+fewest round trips and the most passes.
+
+**What to know first:** macOS only, tested on macOS 27 on Apple silicon. You
+need the Xcode command line tools to build a small helper app, and you grant it
+Accessibility and Screen Recording. Background input uses two private macOS
+calls, with a foreground fallback. Only Claude Code has been tested end to end.
+Despite the name, this is not a sandbox, and it is unrelated to
+[glovebox-mcp](https://github.com/segentic-lab/glovebox-mcp).
+
+## Quickstart
+
+Requires Node 20.10+ and the Xcode command line tools (`xcode-select --install`).
+
+```bash
+npm install -g @vashp2029/glovebox
+glovebox signing --create   # optional: macOS keeps the grants across helper rebuilds
+glovebox setup              # builds the helper, walks through permissions
+glovebox doctor             # every line should say ok
+glovebox smoke              # a real end-to-end test on Calculator
+claude mcp add --scope user glovebox -- glovebox mcp
+```
+
+`setup` builds **Glovebox Helper.app** into `~/.glovebox/` and asks you to turn
+it on under System Settings > Privacy & Security > **Accessibility** and
+**Screen & System Audio Recording**. **Input Monitoring** is optional and only
+enables Escape-to-stop. The helper is its own app, so one grant covers every
+agent and terminal. `signing --create` makes a self-signed certificate in your
+login keychain that signs only the helper. Without it, macOS treats each
+rebuild as a new app; see the [upgrade note](CHANGELOG.md#upgrading).
+
+**Works with:** Claude Code (tested end to end). Codex, Grok CLI, OpenCode,
+Gemini CLI, and Cursor have documented configs that have not been exercised.
+Any agent that can run shell commands can use `glovebox run`.
+See [docs/agents.md](docs/agents.md).
+
+## Why not Claude Code's built-in computer use?
+
+If it fits your setup, it is the easier choice: nothing to install, supported
+by Anthropic, and it keeps your terminal out of screenshots. glovebox is for
+the cases where it does not fit. From [Anthropic's docs](https://code.claude.com/docs/en/computer-use),
+checked 2026-11-03:
+
+| | Claude Code built-in | glovebox |
+|---|---|---|
+| Your other apps | Hidden while Claude works | Stay visible; input goes to the target app in the background |
+| Default read | Downscaled screenshot | Accessibility text diff |
+| Who can use it | Pro, Max, or Ultra through claude.ai; not Team, Enterprise, Bedrock, Google Cloud, or Foundry | Anyone; it never talks to a model or an account |
+| Scripted runs | Not in `-p` mode | The benchmark drives it unattended from a script; `claude -p` itself not yet tested |
+| Sessions | One at a time, enforced by a lock file | No global lock; named sessions run side by side |
+| Remembering approvals | This session only | Allow for This Session or Always Allow |
+| Browsers | View-only | Full control once you approve the browser, which is more power and more risk |
+
+**Claude Code's built-in tool was not in the benchmark below**, so this table
+compares documented behavior, not measured results.
+
+## Benchmark
+
+Twelve macOS tasks, six setups, two runs each, 2026-11-03. Same model (Claude
+Sonnet 5.5, medium effort), prompt, and agent loop for every setup; only the
+tool changes. Task prompts name no tool, and a separate check that doesn't use
+the tool under test decides each result.
+
+| Setup | Passed | Round trips | Images returned | Cost at list price |
+|---|---|---|---|---|
+| glovebox | 24/24 | 128 | 21 | $0.57 |
+| Codex `cua_repl`, driven by Claude | 20/24 | 166 | 30 | $1.41 |
+| Anthropic toolset, run by glovebox | 22/24 | 187 | 179 | $1.65 |
+| agent-desktop | 22/24 | 204 | 22 | $2.32 |
+| open-computer-use | 21/24 | 244 | 385 | $2.76 |
+| Peekaboo | 20/24 | 359 | 183 | $5.67 |
+
+![Median round trips and cost per task for each setup](bench/results/v2/chart.svg)
+
+**Where glovebox did not win.** It was not cheapest on `finder-rename`
+($0.061 against $0.041 for Codex `cua_repl`). It did not have the fewest round
+trips on 3 of the 12 tasks, or the fastest time on 3 of the 12. On
+`remote-desktop` it took 36 seconds against 22 and 23 for two others.
+
+**How much to trust this:**
+
+- I wrote the tasks and ran the test, on one Mac, in one sitting. Several tasks
+  (`bulk-entry`, `remote-desktop`, `coexistence`) were designed around things
+  glovebox set out to do.
+- With two runs per task, pass counts mean little: 24/24 against 20/24 is not
+  statistically significant. The cost gap is the sturdier result, because it
+  holds task by task: glovebox cost less than each other setup on at least 11
+  of 12 tasks.
+- The glovebox tested is the code released as 0.3.0. Result files say 0.2.0,
+  because the version had not been bumped yet.
+- "Anthropic toolset" is Anthropic's API computer-use toolset, executed through
+  glovebox's foreground route. It failed `coexistence` because of that route,
+  so that result doesn't describe a shipping product. "Codex `cua_repl`" measures
+  the tool shape under Claude, not Codex with its own model.
+- Peekaboo's first pass was rerun with foreground input allowed, which helped
+  it. Its two `bulk-entry` failures were my $0.75 per-run budget cap stopping
+  the run. agent-desktop ran through its CLI, not its MCP mode.
+
+Method, per-task numbers, and raw results: [bench/](bench/README.md) and
+[bench/results/v2/summary.md](bench/results/v2/summary.md).
+
+## How it compares
+
+Checked against each project's README and docs on 2026-11-03. "—" means I could
+not confirm it from those.
+
+| | Platforms | Actions per call | State between calls | Default read | Diffed reads | Background input | Needs |
+|---|---|---|---|---|---|---|---|
+| **glovebox** | macOS | many (one script) | persistent JS session | accessibility text | yes | yes | Xcode CLT to build the helper |
+| Claude Code built-in | macOS (CLI) | — | — | screenshot | — | no, hides other apps | Pro, Max, or Ultra |
+| Codex computer use | macOS, Windows | many (one script) | persistent JS session | accessibility text | yes | yes on macOS | ChatGPT app and plugin |
+| [lcu](https://github.com/amontlabs/lcu) | macOS (Apple silicon), Linux | as Codex; it runs Codex's runtime | as Codex | as Codex | as Codex | — | ChatGPT app installed |
+| [agent-desktop](https://github.com/vercel-labs/agent-desktop) | macOS, Windows, Linux | one per command, plus `batch` | refs tied to a snapshot | accessibility tree JSON | — | yes | Homebrew or npm |
+| [Peekaboo](https://github.com/openclaw/Peekaboo) | macOS 15+ | one per command | element IDs from a snapshot | screenshot or JSON UI map | — | yes | Homebrew or npm |
+| [open-computer-use](https://github.com/iFurySt/open-computer-use) | macOS, Linux, Windows | one per MCP call; CLI has sequences | — | app state | — | — | npm |
+| [computer-use-mcp](https://github.com/minghinmatthewlam/computer-use-mcp) | macOS 14+ | one per call | element IDs | screenshot and tree per action | — | yes | notarized binary |
+| [cua-driver](https://github.com/trycua/cua) | macOS, Windows, Linux | — | — | — | — | yes | see its docs |
+
+**Pick something else if** you need Windows or Linux (agent-desktop, cua-driver,
+lcu), a notarized binary with no build step (computer-use-mcp), or want
+OpenAI's actual runtime and already have the ChatGPT app (lcu).
+
+## Safety
+
+The first time a session touches an app, you get a dialog: Allow for This
+Session, Always Allow, or Deny. Terminals, password managers, System Settings,
+and a few others are blocked unless you run `glovebox allow <bundle id>`; the
+list is in [SECURITY.md](SECURITY.md#blocked-apps). Switching to another app
+stops a running script (Escape too, with Input Monitoring). Every action is
+logged to `~/.glovebox/actions.log`.
+
+An approved app is approved for everything in it: allowing a browser lets the
+agent use every site you are signed in to. glovebox does not screen page
+content for prompt injection. Treat it like giving someone remote control of
+that app. Details: [SECURITY.md](SECURITY.md).
+
+**Known issue:** while a background app's pop-up menu is open, macOS sends your
+keystrokes to that app, so a word you type can land in it
+([details](docs/api.md#known-limits)).
+
+## How it works
+
+```
+agent ──MCP stdio──▶ glovebox mcp ──▶ kernel (persistent JS session)
+agent ──shell──▶ glovebox run ──▶ daemon ──▶ kernel        │
+                                                            ▼ Unix socket
+                                        Glovebox Helper.app (Swift: AX, CGEvent,
+                                        ScreenCaptureKit, approval dialogs)
+```
+
+Background input uses two private macOS (SkyLight) functions, looked up at
+runtime with a foreground fallback; `glovebox doctor` reports which route you
+have. Everything else is public API. No OpenAI, Anthropic, or other vendor
+binaries.
+
+## Status
+
+| Platform | Status |
+|---|---|
+| macOS | Works. Built for macOS 14+; tested only on macOS 27 (Apple silicon). |
+| Windows | Planned (UI Automation). See [roadmap](docs/roadmap.md). |
+| Linux | Planned (AT-SPI; Wayland limited). See [roadmap](docs/roadmap.md). |
+
+Run for real, not mocked: `glovebox smoke` and `smoke --full` over MCP (batched
+steps, persistent bindings, diffs, pixel clicks, and background click, drag, and
+scroll with another app frontmost), and Claude Code attached through
+`glovebox mcp`. Full list: [docs/testing.md](docs/testing.md).
+
+## The API in one screen
+
+- `gb.getApp(name)` launches or finds an app without stealing focus. `gb.listApps()`.
+- `app.getAXState()` returns the accessibility tree as text; later reads are
+  diffs. `{frames: true}` adds element boxes, `{disableDiffing: true}` the full tree.
+- `app.getScreenshot()` is the only way to get an image.
+- `app.click(target)`, `doubleClick`, `rightClick`, `scroll`, `drag`,
+  `setValue`, `focus`, `selectText`, `performSecondaryAction`.
+- `app.pressKey("ctrl+a")` uses xdotool names (`super` is Cmd). `app.typeText("text\n")`.
+- `app.waitForChange()` and `app.settle()` wait for a late update.
+- A target is an element index from the tree or `[x, y]` pixels in the
+  window image, which is the screenshot's pixel space.
+
+Full reference: [docs/api.md](docs/api.md), or `help()` inside a session.
+
+## Docs
+
+- [docs/api.md](docs/api.md): the scripting API, for agents and script authors.
+- [docs/agents.md](docs/agents.md): attaching each MCP client or a shell-only agent.
+- [docs/cli.md](docs/cli.md): every subcommand, flag, config key, and environment variable.
+- [docs/recipes.md](docs/recipes.md): short scripts to paste into one `js` call.
+- [docs/design.md](docs/design.md): decisions and their reasons, for contributors and evaluators.
+- [docs/backend-protocol.md](docs/backend-protocol.md): the helper protocol, for porting to a new platform.
+- [docs/testing.md](docs/testing.md): unit and smoke tests, and what has been run on real hardware.
+- [docs/roadmap.md](docs/roadmap.md): planned work; nothing there is shipped.
+- [SECURITY.md](SECURITY.md): what protects you, what does not, and how to report a vulnerability.
+- [CONTRIBUTING.md](CONTRIBUTING.md): setup from a clone, tests, and pull request rules.
+- [CHANGELOG.md](CHANGELOG.md): changes per release and upgrade notes.
+
+## License
+
+MIT

package/SECURITY.md

@@ -0,0 +1,81 @@
+# Security
+
+glovebox lets an AI agent click, type, and read the screen in apps on your
+computer. Treat it with the same care as giving someone remote control.
+
+## What protects you
+
+- **Per-app approval.** By default, the first action a session takes in an app
+  shows a dialog: Allow for This Session, Always Allow, or Deny. The check runs
+  inside Glovebox Helper, the process that holds the macOS permissions, so
+  scripts cannot skip it.
+- **Blocked apps.** Apps that can run commands or hold secrets are refused,
+  and never offered in a dialog, unless you add them with `glovebox allow`
+  (see the list below). An agent with terminal access could otherwise use the
+  GUI to run commands its sandbox would not allow. An app on the `deny` list
+  is refused even if it is also allowed.
+- **Taking over stops a script.** If you switch to another app while a
+  script is driving one, the script stops instead of taking focus back. With
+  the optional Input Monitoring permission, pressing Escape also stops it
+  (glovebox marks its own key events, so its own Escapes do not count).
+- **Menu-bar item.** While an agent is connected, Glovebox Helper shows a
+  menu-bar item listing each session and its approved apps. Stop Running
+  Scripts stops every running script. Revoke Session Approvals clears the
+  per-session approvals, so apps must be approved again; it does not touch
+  `allow` entries in `config.json`. Quit Glovebox Helper exits the helper; the
+  next agent call starts it again.
+- **Action log.** `~/.glovebox/actions.log` has one JSON object per action:
+  `ts`, `client` (the agent's name), `step` (the call's title, when given),
+  then the action's own fields (`app`, `action`, and for example `target`,
+  `keys`, or `length`). Typed text is recorded as a length only, unless you
+  set `"log": {"text": true}`.
+- **Local only.** Nothing is sent over the network. The helper listens on a
+  Unix socket that only your user can open.
+
+## Blocked apps
+
+This is the full default list, from `blockedByDefault` in
+`native/macos/GloveboxHelper.swift`. Bundle ids are matched case-insensitively.
+`glovebox allow <name or bundle id>` unlocks one.
+
+| Kind | Bundle ids |
+|---|---|
+| Terminals | `com.apple.terminal`, `com.googlecode.iterm2`, `com.mitchellh.ghostty`, `dev.warp.warp-stable`, `com.github.wez.wezterm`, `org.alacritty`, `net.kovidgoyal.kitty`, `org.tabby`, `co.zeit.hyper` |
+| System settings | `com.apple.systempreferences` |
+| Password managers | `com.apple.keychainaccess`, `com.apple.passwords`, `com.1password.1password`, `com.agilebits.onepassword7`, `com.bitwarden.desktop`, `com.lastpass.lastpassmacdesktop` |
+| Scripting | `com.apple.scripteditor2`, `com.apple.automator` |
+| Glovebox itself | `io.github.vpatel9202.glovebox.helper` |
+
+Order of checks for each app: `deny`, then `allow`, then this list, then the
+`approval` mode.
+
+## What does not
+
+- Code running as your user can edit `~/.glovebox/config.json` and approve
+  apps for itself, or talk to the helper directly. glovebox does not defend
+  against malware already running as you.
+- `approval: "allow-all"` removes the dialogs for every non-blocked app.
+  Use it only when you trust every agent you attach.
+- An approved app is approved for everything in it. Allowing a browser lets
+  the agent use every site you are signed in to.
+- Background input uses two private macOS (SkyLight) functions, the same
+  approach as yabai, cua-driver, and Peekaboo. They are looked up at runtime;
+  if a macOS update removes them, glovebox falls back to bringing the app
+  forward. They do not bypass any permission: the helper still needs
+  Accessibility.
+- Scripts have no `require`, `fs`, or dynamic `import`, and see an empty
+  `process.env`. That limits a script to the `gb` API and approved apps, but an
+  approved app can still reach whatever that app can reach. Finder is not on the
+  blocked list, and approving it lets an agent read or change any file Finder
+  can.
+- glovebox does not screen app or page content for prompt injection. Text an
+  agent reads in an approved app can try to steer it.
+- The optional "glovebox local signing" certificate lives in your login
+  keychain and signs only the helper you build. Without it, macOS ties
+  privacy grants to an ad-hoc signature and a rebuild means granting again.
+
+## Reporting a vulnerability
+
+Please open a private security advisory at
+https://github.com/vpatel9202/glovebox/security/advisories/new rather than a
+public issue.