@vashp2029/glovebox 0.0.0-stage → 0.3.1

package/docs/roadmap.md

@@ -0,0 +1,167 @@
+# Roadmap
+
+What works today is macOS (14+, tested on 27). Everything below is planned,
+not shipped. Nothing here has been tested unless it says so.
+
+## Found by the benchmark and the field run
+
+The diff fixes and the bugs found by the 2026-10-28 baseline
+([bench/results](../bench/results/summary.md)) and the 2026-10-29 Citrix field
+run ([summary](../bench/results/field/2026-10-29-citrix/summary.md)) are
+fixed; see the [changelog](../CHANGELOG.md). One is not yet confirmed:
+
+1. **Horizontal background scroll in Citrix.** Scroll events now set the
+   point and fixed-point deltas for the horizontal axis, but this has not been
+   tried in Citrix Viewer. (gb-1iz.12)
+
+## Credibility and distribution
+
+Ordered by expected impact. These come before new platforms.
+
+1. **Benchmark harness (`bench/`).** Done 2026-10-28: harness, raw results,
+   and chart are in [bench/](../bench/README.md). The original scope follows.
+   About five fixed tasks: Calculator
+   arithmetic, editing a TextEdit document, a Finder rename, toggling a
+   setting in a non-blocked app, and filling a form in a native app. Add two
+   more: a Citrix-or-canvas task (pixel fallback, one-node tree) and a replay
+   task (run the same task again from a saved script), so the differentiators
+   below are measured against this baseline. Compare glovebox, Anthropic
+   computer use (screenshot-first), and one per-action AX MCP server
+   (open-computer-use or agent-desktop). Measure model round trips,
+   input tokens including image tokens, output tokens, wall-clock time, and
+   success over N runs per task, plus the token cost of a diff versus a full
+   tree. For the token work below, also count full re-prints caused by a
+   baseline mismatch, "no change" reads followed by another read, screenshots
+   per text task, and per-result text characters and image pixels (logged by
+   the kernel). Results must split input tokens into uncached, cache write,
+   and cache read, because with prompt caching round trips dominate cost.
+   Done when the raw results and the harness are in the repo and one chart is
+   in the README.
+2. **A prebuilt helper.** Ship a Developer ID signed, notarized helper in each
+   release. Done when `npx -y @vashp2029/glovebox setup` works on a Mac without
+   the Xcode command line tools and grants survive an upgrade.
+3. **Demo media, then launch.** Record a GIF and a short video. Launch (Show HN, X, Codex computer-use
+   threads) only after one or two of these exist. Done when the GIF is at the
+   top of the README.
+4. **Exercise every documented client.** Run the Codex, Grok CLI, OpenCode,
+   Gemini CLI, and Cursor configs end to end, with a short clip for each.
+   Done when `docs/agents.md` and the README's "Works with" line list each
+   client as verified or say what failed.
+5. **CI on real GUI runs.** Unit tests run on every push and PR. Smoke tests
+   still need a Mac with permissions granted, so they run by hand.
+6. **Listings.** The official MCP registry, the Claude Code plugin
+   marketplace, and skills.sh (for `skills/glovebox/SKILL.md`). Done when each
+   listing is live and links back to the repo.
+7. **Packaging and docs fixes found in review.** Done; see the changelog.
+
+After these, Windows is the highest-priority platform: cross-platform support
+is what lcu, agent-desktop, open-computer-use, and cua have over glovebox, and
+Windows-MCP has about 10.5k stars.
+
+## Token efficiency
+
+Ranked by round trips saved. Each call re-reads the cached context and adds
+reasoning and output tokens, so a round trip costs more than trimming one
+read. Savings are estimates from reading the code, to be confirmed by the
+benchmark. Effort: S is small, M is medium.
+
+1. **Silent structured lookups.** Add `app.find({role, label, id})`,
+   `app.text(i)`, and `app.nodes()`, which return data without printing, plus
+   one instruction line in the tool description and `skills/glovebox/SKILL.md`.
+   `app.text` also gives the full value of a field, which a read truncates at
+   160 characters. Saves 1-2 round trips on text and web tasks. S.
+2. **A read that waits for change.** Done: `app.waitForChange()` and
+   `app.settle()`.
+3. **Checks inside the script.** An `expect` helper lets one call chain
+   several steps, stop at the first failed check, and print state on failure
+   or error. Per-action tools cannot do this. Saves 0.5-1 round trip per task
+   and per error. S.
+4. **Outline for large trees.** Above about 250 nodes, show an outline
+   instead of the full tree, and prune offscreen nodes with a summary line.
+   Helps Electron and web apps and saves 5-8k tokens per full read. Adds no
+   round trip, because the script can call `find` in the same call. M.
+5. **Field labels and window notices.** Resolve `AXTitleUIElement` and
+   `AXLabelUIElements` into field labels, and add a one-line notice when a
+   window, sheet, or alert opens. Helps forms and save sheets by avoiding
+   screenshots, and makes scripts that skip the read safe. S.
+6. **Line-format cleanup.** Quiet `AXShowMenu` (it is not in `QUIET_ACTIONS`),
+   drop auto-generated `_NS:` ids, show `help=` only when there is no label,
+   collapse scroll bars into `scrolled N%`, and drop single-child unlabeled
+   groups. A guess of 20-40% off a full tree, to be verified. S.
+7. **Diffs that survive context clearing.** A `read #N` header on each read,
+   and role plus label in short removed lists, so a diff still makes sense
+   after the earlier tree has left the context. S.
+8. **Cheaper images.** `getScreenshot({region})` crops, and a hint when the
+   tree has two or fewer nodes. JPEG and grayscale save no tokens, because
+   images are charged by pixel size. Lowering the default scale would change
+   the click coordinate space, so it is not planned. S-M.
+
+Not doing (backfires):
+
+- An automatic read after every action: it adds tokens to calls that did not
+  need a read.
+- Aggressive pruning: the agent loses elements it needs and spends a round
+  trip finding them.
+- Diffs that show only changed attributes: they are harder to read and the
+  line no longer identifies the element.
+- A smaller default screenshot scale: it changes the click coordinate space.
+- An outline-first read for small trees: it adds a round trip below about
+  2-3k tokens.
+
+## Differentiators
+
+Two features no other tool checked has, as far as their READMEs show (eight
+competitors checked on 2026-10-27).
+
+1. **Local OCR text.** `app.getText()` runs Vision `VNRecognizeTextRequest`
+   in the helper and returns diffable `"text" @x,y` lines in click
+   space. It targets Citrix, canvases, and remote desktops, where the tree is
+   one node. None of the eight competitors checked documents this. The OCR
+   runs locally, but the recognized text still goes to the model.
+2. **Durable scripts and replay.** Selectors (token item 1) and `waitFor`
+   (token item 2) feed functions saved to `~/.glovebox/lib/*.mjs`, which load
+   automatically. `glovebox run file.mjs` replays a script with zero model
+   calls. For comparison, ghost-os recipes appear to need a small model, and
+   Peekaboo and open-computer-use replay JSON step lists with no branching.
+
+## Windows backend
+
+Plan: a PowerShell helper (no install beyond Windows itself) that speaks the
+backend protocol over a named pipe.
+
+- Tree and actions: UI Automation through `System.Windows.Automation`
+  (`InvokePattern` for presses, `ValuePattern` for `setValue`,
+  `TextPattern`/`SelectionItemPattern` where present). Stable indices from
+  `RuntimeId`.
+- Input: `SendInput` via a small `Add-Type` C# shim; `PostMessage` for
+  background keys where the target accepts it.
+- Screenshots: `PrintWindow` with `PW_RENDERFULLCONTENT`, resized to the
+  requested size.
+- Approval dialog: a WinForms message box from the helper.
+
+## Linux backend
+
+Plan: a Python helper.
+
+- Tree and actions: AT-SPI over D-Bus (`pyatspi` or `gi.repository.Atspi`).
+- Input: XTest on X11. Wayland compositors restrict synthetic input and
+  screen capture; support will be per compositor (portal-based capture,
+  `ydotool` or the RemoteDesktop portal for input), and some will not work.
+- Screenshots: X11 `XGetImage` on the window; the ScreenCast portal on Wayland.
+- Approval dialog: `zenity` or `kdialog`, else allowlist mode only.
+
+## macOS improvements
+
+- Chromium/Electron keyboard input on macOS 15+ may need an
+  `SLSEventAuthenticationMessage` attached to each key event; not done yet.
+- Menu shortcuts (⌘S and friends) in background apps: press the menu item
+  through accessibility instead of sending the key.
+- Detect apps that ignore background input and fall back automatically,
+  instead of relying on `foregroundApps` in the config.
+- Test matrix beyond AppKit: Safari/WebKit, Electron apps, SwiftUI-only apps,
+  Citrix Viewer.
+- MCP elicitation as an alternative to the native approval dialog when the
+  client supports it.
+- `app.menu(["File", "Export…"])` through the menu bar tree.
+- Optional settle detection (wait until the tree stops changing) instead of
+  a fixed delay after actions.

package/docs/testing.md

@@ -0,0 +1,79 @@
+# Testing
+
+glovebox has two kinds of tests. Unit tests cover the pure parts and run
+anywhere. The smoke test drives real apps through the real MCP server and
+helper, so it runs only on macOS with the helper built and permissions granted.
+Nothing in the smoke test is mocked.
+
+## What has been run for real on macOS 27 (Apple silicon)
+
+- `glovebox smoke` over MCP: one batched call types `6*7=` and reads `42`;
+  the binding persists to the next call; index clicks compute `7+1=8` and
+  the read comes back as a diff; a pixel click lands on the `9` key; the
+  screenshot is 460x816, the same space as clicks; `js_reset` drops bindings.
+- Claude Code (headless) attached through `glovebox mcp`: two calls, the
+  second reusing the first call's binding.
+- `glovebox smoke --full` with Firefox frontmost the whole time: background
+  keys in Calculator; in TextEdit, a pixel click placed the caret, a drag
+  selected a line, a scroll moved the view; the frontmost app never changed.
+- The approval dialog, a blocked app (Terminal) being refused, and Escape to
+  stop a script.
+
+The Grok, OpenCode, and Codex configs in `agents.md` follow each client's
+documented MCP format but have not been exercised end to end yet.
+
+## Unit tests
+
+```bash
+npm test    # node --test test/ (Node 22+: node --test test/unit.test.mjs)
+```
+
+`test/unit.test.mjs` needs only Node. It checks:
+
+- cell rewriting: top-level bindings persist across cells and can be
+  redeclared; destructuring and imports become bindings; a rejected top-level
+  `await` rejects instead of hanging; syntax errors report a line.
+- key parsing: xdotool chords, modifier aliases, and unknown modifiers.
+- tree text: compact lines, and diffs keyed by element index (`~`, `+`,
+  removed).
+- image space: window pixels round-trip to screen points, and large windows
+  are scaled to the pixel budget.
+
+## Smoke test
+
+```bash
+glovebox smoke
+```
+
+Starts `glovebox mcp` as a client would and works on Calculator. It prints one
+`PASS` or `FAIL` line per check and exits 0 only if all pass. Calculator's
+state changes during the run. The first time, macOS may ask to approve
+Calculator for "glovebox smoke"; run `glovebox allow Calculator TextEdit`
+first to skip the dialogs. Checks:
+
+1. MCP handshake: the server is named `glovebox` and lists two tools.
+2. A batched call types `6*7=` and the read shows `42`.
+3. The binding persists to the next call; clicking by element index computes
+   `7+1=8`; the read is a diff ("changes since last read").
+4. A pixel click at an element's `center()` presses the `9` key.
+5. A screenshot is one image whose size equals `calc.view`, the pixel space
+   clicks use.
+6. `js_reset` drops the binding: `typeof calc` is `undefined`.
+7. The frontmost app was the same before and after (checked with
+   `osascript`; skipped if it cannot run).
+
+```bash
+glovebox smoke --full
+```
+
+Runs the checks above, then writes two temporary text files, opens them in
+TextEdit without activating it (`open -g`), and checks raw mouse input in the
+background:
+
+1. A pixel click places the caret: typing `Y` lands at the start of the text.
+2. A drag selects text: typing `Q` replaces the selection.
+3. A scroll moves the view: a scroll bar's value changes on a 300-line file.
+
+The frontmost-app check then covers these steps too. The two documents stay
+open in TextEdit; close them without saving (the temp directory is printed).
+For a meaningful result, keep some other app frontmost while it runs.