dsh-inner-browser-use 0.11.44 → 0.18.0

package/docs/COMPATIBILITY.md

@@ -1,104 +1,74 @@
 # Compatibility
 
-What this plugin assumes, and what it degrades to when an assumption fails.
-
-## Requirements
-
-| Capability | Needed for | Missing means |
-| --- | --- | --- |
-| Desktop app (`dshDesktop.browser`) | Leasing the app's own browser tab | The call answers that the built-in browser is not available; tools stay registered. |
-| `sidebar.right.pane.tab` slot | The tab body | Nothing registers; no tab appears in the **+** menu. |
-| `ctx.sidebarRightTabs` | Tab type registration | Registration is skipped silently; the tab never appears. |
-| `ctx.sidebarRight.openTabIn` | Placing a tab in the column | Tools need the tab opened by hand; the call says so in one sentence. |
-| `ctx.webServer.register` | The whole wire | Plugin fails to load, visibly. |
-| `ctx.tools.register` | The tool surface | Bridge still runs; the model sees no `browser_*` tools. |
-| `EventSource` in the page | Learning that a tab is wanted | Same as the row above. |
-
-## What it never touches
-
-Installing this plugin changes nothing a user already had:
-
-- Nothing shipped is wrapped or shadowed: the browser tab, its kind, its tools'
-  behaviour and its partition are the product's; this plugin only drives them.
-- No guide card: the `开始` pane renders exactly the entries it had before. A
-  client test asserts the registered tab type carries no `guide` array.
-- No global CSS: every rule is scoped to the page it drives.
-- No slot occupant on any slot it does not own, and no `computerUse`/`browserUse`
-  provider registration, so the shipped browser's own providers keep their seats.
-- Removing the bundle removes every one of the above.
-
-## Version notes
-
-- **Node 20+** on the Host. The plugin uses only `node:http`, `node:fs/promises`,
-  `URL`, and `AbortSignal`.
-- **No dependencies.** No bundler, no build step, no post-install script.
-- `engines.dsh` is `>= 0.2.0`. Checked against Desktop builds whose sidebar
-  browser tab kind is `browser` and whose guest handshake is an `about:blank#`
-  webview carrying the workspace partition; on any other shape the plugin treats
-  the Desktop bridge as absent rather than guessing.
-
-## Coexistence
-
-- The tab kind is `browser` — the built-in browser's own. This plugin registers no
-  tab kind of its own.
-- The route prefix is `/dsh-inner-browser-use`. Nothing is mounted under `/api`.
-- The guest partition is the one the app leases for the workspace
-  (`dsh-sidebar-browser-<uuid>`). Two plugins leasing the same workspace key share
-  storage, exactly as two built-in tabs do.
-
-## Known limits
-
-| Limit | Why | Workaround |
-| --- | --- | --- |
-| A login is kept only if the page can read its cookie | `HttpOnly` cookies are not visible to the page half | Sites that keep the session in a readable cookie come back signed in |
-| One storage partition per workspace | The lease key is the workspace CWD | Use one workspace per account you need isolated |
-| One chat is on screen at a time | The column shows one chat, so another chat's page does not exist | Open the chat that asks; the call answers in about a second |
-| Text-only observations | Frames are JSON over one SSE socket | Use `browser_script` to pull exact values |
-| A capture needs that page drawn | The app has no surface to read for a page another tab covers | Bring the chat's tab forward; the call says so |
-
-## Downloads land in the chat folder
-
-The app cancels its own downloads, so a page cannot save a file by itself. The
-page half therefore collects what a page offers — an `<a download>`, a `blob:`
-address, a `data:` address, or a link whose path ends in an archive, office
-document, PDF, text, image or media extension — and hands the bytes to the Host,
-which writes them under `<chat folder>/.dsh-inner-browser-use/downloads/`. The
-call returns that path, and `browser_download` with no argument prints the files
-already collected.
-
-Limits that come from the page, not from the plugin:
-
-- One offered file is capped at 7 MB and six are taken in one go; a file the
-  page could not hand over is named with its size in the answer, so the page can
-  be asked again. The Host refuses an upload over 8 MB with the size named.
-- A page whose `connect-src` forbids the site's own address cannot fetch it; the
-  plugin then has only what the page already holds, so ask the page for the file
-  through its own UI.
-- A file streams only while the page keeps answering. A page that navigates away
-  mid-save ends the transfer, and the call says so instead of writing half a file.
-
-## Checking an install
-
-Run these in the chat whose tab is open, in this order. Each line is what a
-working install answers.
-
-1. `browser_navigate` to `https://example.com/` → title `Example Domain`, one control.
-2. `browser_view` → the same page, so the tab is the one this chat owns.
-3. `browser_script` adding a button, then `browser_click` it → `Clicked #2 [button] clicked at 120,160`.
-4. `browser_screenshot` → a path under `.dsh-inner-browser-use/`, or the sentence
-   naming the tab to bring forward.
-5. Restart the app once if `browser_selection` answers `unknown tool`: the tool
-   list is read at start.
-
-## A tab that is not the active one
-
-The strip mounts only the tab it shows, and after a restart the restored tab is
-not the active one. Nothing in the tab face switches it: `openTabIn` with
-`activate`/`focus`, `focus(id)`, `focus(session, id)`, `active(id)`,
-`tab.activate` and `tab.focus` were each tried and left the active tab
-unchanged; the tab bar's controls are not inside `sidebarRight.host`, so there is
-no element to click.
-
-A call therefore never depends on that switch. On load, the chat on screen gets
-its own tab back and the page it had; every other chat opens nothing, because a
-tab lands in the column that is showing.
+What this plugin assumes, and what it does when an assumption fails.
+
+| | |
+| --- | --- |
+| Harness | 0.2.0 or newer, with the right column |
+| Runtime | Node 20 or newer |
+| Browser | Google Chrome, Chromium or Canary, 112 or newer |
+| System | macOS 12 or newer; Linux x86_64 or arm64 |
+| License | MIT |
+
+## Where Chrome is looked for
+
+The plugin starts a browser it finds at one of these paths, in this order:
+
+```
+/Applications/Google Chrome.app/Contents/MacOS/Google Chrome
+/Applications/Chromium.app/Contents/MacOS/Chromium
+/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary
+/usr/bin/google-chrome
+/usr/bin/chromium
+```
+
+That is the whole list. A Chromium kept elsewhere (snap, flatpak, a home directory, a bundled
+runtime) is not found, and there is no setting to point at one. Installing Chrome the ordinary
+way, or placing the binary at `/usr/bin/chromium`, is the supported arrangement.
+
+If nothing in the list exists, the first tool call returns `sidecar: no Chrome found on this
+machine` and nothing is written to disk.
+
+Chrome 112 is the floor: the mirror needs `--headless=new` and a screencast that survives an
+idle page.
+
+## macOS
+
+The browser is launched with `--use-mock-keychain` and `--password-store=basic`, so no keychain
+dialog appears on the person's screen and no password is read from the OS keyring. Chrome's own
+password store inside the profile still works.
+
+The profile directory is created under the home directory of whoever runs DSH, which is the
+same user the browser runs as. Running that process as a different user leaves the profile
+unwritable, and running it as root leaves Chrome unable to start its sandbox: run DSH as the
+person using it.
+
+## Linux
+
+The two Linux entries above are the same lookup list; put Chrome or Chromium there. The browser
+runs without a display, so no X or Wayland session is needed for the mirror itself.
+
+Linux has not been part of the release test pass; macOS is where this plugin is developed and
+checked.
+
+## Windows
+
+Not supported. There is no Windows path in the lookup list, and no installation of this plugin
+on Windows has been checked.
+
+## When an assumption fails
+
+| Assumption | What happens |
+| --- | --- |
+| No Chrome at a listed path | First tool call returns the error above; no profile, no process. |
+| Harness older than 0.2.0 | The plugin does not load. |
+| A Harness build with no right column | The tools still drive the browser; there is nowhere to show the mirror. |
+| A site that refuses headless | The user agent is replaced with an ordinary desktop Chrome string; a site that insists on a window can be opened with `headless: false`. |
+| A page wider than 1440×900 | The viewport is fixed, so the page lays out for 1440 wide; nothing is scaled. |
+| Two plugin instances, one profile | The second adopts the Chrome already running on that profile rather than starting another. |
+| The debugging file left by a crashed Chrome | It is removed and the browser is started again. |
+
+## Limits by design
+
+One page per browser, fixed 1440×900 viewport, no mobile emulation, no downloads, no uploads, no
+script evaluation. The eight tools in the README are the whole surface.

package/docs/ROADMAP.md

@@ -0,0 +1,176 @@
+<!-- ui-copy-ignore -->
+# Roadmap: from a working mirror to the best computer-use plugin
+
+The bar is not "another browser driver". The bar is: an agent can finish a real
+task in this browser with fewer turns, fewer tokens, fewer surprises than with
+Codex, Claude Code, browser-use, Stagehand, Playwright MCP or any screenshot-only
+computer-use model — and the person can watch it happen in the sidebar of the
+chat that asked, while every other chat stays untouched.
+
+Written as phases with a number attached, because a phase without a number is a
+wish. Baselines come from `node scripts/bench.mjs`; every phase must move one.
+
+## Where the competition actually wins
+
+| Project | What it does well | What it costs |
+| --- | --- | --- |
+| Codex / Claude Code (computer + bash) | free-form shell, one action per turn, human can take over mid-run | no page structure, no refs, screenshots dominate context |
+| browser-use | indexed DOM tree with stable element ids, action model, per-step memory | tree is big; a full read is thousands of tokens |
+| Stagehand | `act` / `observe` / `extract` split, self-healing selectors, natural-language actions | server round trips; two model calls per step |
+| Playwright MCP | accessibility tree with `ref` handles, deterministic waits | text-only; a page read is large; no live picture |
+| Skyvern / CUA screenshot models | works on canvas apps and hostile DOM | ~1 image per step, slow and token-expensive |
+
+Our position: **a real Chrome with a live, interactive mirror in the owning
+chat, and a read model that is small by construction** — accessibility refs that
+survive a read, viewport-only text, diffs instead of full re-reads, and one
+batched tool for multi-step work. The mirror is not decoration: it is the human
+takeover surface, and no other plugin can show it inside the asking chat without
+stealing the column from other chats.
+
+## Phase 0 — measurement (done when numbers exist)
+
+- `scripts/bench.mjs`: launch→first frame, steady fps, frame KB, read ms /
+  chars / approx tokens for a dense page, a text page and a light page, click,
+  key, screenshot KB.
+- Record every run in `docs/BENCH.md` with the commit; no phase is "done"
+  without a before/after row.
+
+## Phase 1 — read model that costs less (target: dense page read ≤ 1.2k tokens)
+
+- Accessibility-tree walk with `ref` handles that stay valid across reads of the
+  same page; only elements that exist in the tree get a number.
+- Viewport-limited by default (`inView`), with `mode`: `text` (default),
+  `links`, `forms`, `full`; `max_chars` budget; text deduplicated and
+  truncation marked, never silent.
+- `changed_only`: after an action, return only added / removed / changed refs —
+  the browser-use and Playwright MCP gap.
+- Stable refs: a read re-numbers nothing that did not change, so the model can
+  act on numbers from an earlier read and gets a clear "ref 3 is gone, re-read"
+  instead of clicking the wrong thing.
+
+## Phase 2 — one call, many steps (target: a login flow in 1 tool call)
+
+- `sidecar_do`: ordered steps with per-step waits (`wait_text`, `wait_gone`,
+  `wait_idle`, `wait_url`), one result: what changed, the final URL, tokens spent
+  on one read instead of five.
+- Auto-wait after every action: navigation start/finish, DOM quiet, and
+  `document.readyState`, so the next step never races the page.
+- Ref errors are actionable: which ref, why it is gone, and one call that
+  re-reads and retries once.
+
+## Phase 3 — any operation (target: no "can't do that" left)
+
+- Hover, drag, double-click, right-click, wheel with modifiers.
+- `select` option by label or value; checkboxes and radios by label.
+- File upload (`DOM.setFileInputFiles`), download capture with a returned path,
+  print-to-PDF, full-page and element screenshots.
+- Tabs: new / switch / close / list; iframes addressed by frame id; dialog
+  accept/dismiss; `sidecar_eval` for the last 1% of cases; cookies read/write for
+  debugging; console + failed requests surfaced in the state read.
+- Keyboard: modifiers, shortcuts, clipboard paste, IME composition unchanged.
+
+## Phase 4 — mirror latency and clarity (target: ≤120 ms input→painted, 60 fps idle)
+
+- Adaptive JPEG quality and frame size: drop quality before dropping fps; send a
+  small first frame, refine when idle.
+- `Page.startScreencast` tuning (maxWidth/quality/everyNthFrame), Chrome flags
+  that stop background throttling, `--disable-frame-rate-limit`.
+- Client: decode off the main thread (`createImageBitmap`), draw only the newest
+  frame, no layout thrash, ResizeObserver-driven backing-store sizing for a crisp
+  picture at any pane width, DPR-aware.
+- Pointer: keep the 1 ms path; add remote cursor shape to the pane cursor with a
+  proper fallback, and a visible "the agent is acting" pulse that never moves the
+  system cursor.
+
+## Phase 5 — reliability and safety
+
+- Human takeover (0.17.1): real input in the pane (click / text / wheel / key)
+  holds the agent's next action for 6 s and says so; hover and move do not count;
+  reads stay open; `sidecar_resume` takes the page back at once.
+- Guarded actions (upload, download, close tab, buy-ish submits) can require a
+  confirm flag. Shipped in 0.14.0 for the narrow case that matters most: an
+  overlay in the way is cleared only when its own way out is a safe word
+  (accept / agree / close / 同意 / 关闭 …) and the overlay says nothing about
+  deleting, paying or signing out; otherwise it is reported, never taken.
+  Since 0.15.0 the same clearing runs before every control-naming step of a
+  `sidecar_do` batch, so a multi-step flow through a cookie wall fails zero times.
+- Crash recovery: attach to a still-running Chrome, rebuild the page, keep the
+  profile; a killed host never leaves a zombie Chrome.
+- Every failure names what happened, what state the page is in, and the next
+  call to make.
+- A failure that is the browser's, not the page's, is said in those words: on
+  Chrome's own error page a read, a click and a batch step all refuse with the
+  `ERR_…` code and the way out, while the tab list, console, failed requests and
+  address still answer, because those are how a stuck person finds out what
+  happened.
+
+## Phase 6 — proof and packaging
+
+- `docs/BENCH.md` with before/after per phase; a competitor comparison table with
+  measured numbers, not adjectives.
+- `docs/VERIFY.md` walk-through that a stranger can run; `scripts/doctor.mjs`
+  green; 3 linters zero errors; tests covering every new tool.
+- Packed tarball, CHANGELOG per release, `0.x` bumps that match behaviour.
+  0.13.4 and 0.14.0 shipped tarballs without `tools.js` and `read.js` — a clone
+  hid it because the dev profile symlinks the repo. Since 0.15.0 a test walks the
+  import graph against `files`, and another unpacks the tarball and imports it.
+
+## Phase 7 — recovery inside the call, 0.16.0
+
+The measured weak spot after Phase 5 was not speed but turns spent recovering:
+every framework redraw invalidated the numbers from the last read, and the model
+paid a full read plus a turn to get them back. 0.16.0 moves that into the call by
+finding the control again by identity (kind + exact words + link destination),
+refusing when the words are not unique. Measured at 462 ms against 445 ms for a
+fresh control — 17 ms when needed, nothing when not.
+
+Since 0.16.1 the count is visible: `sidecar_state` reports `refs followed N`,
+and breadth asserts it moved after a redraw, so the next rounds can tell from
+real sites whether redrawing is frequent enough to justify more work here.
+Next in this direction: the same identity recovery for `sidecar_do` steps that
+name a field by its label when the label element is rebuilt (today the step
+falls back to a fresh read).
+
+## Order of work
+
+1. Phase 0 baseline. 2. Phase 1 + 2 together (they share the read model).
+3. Phase 3 breadth. 4. Phase 4 numbers. 5. Phase 5 hardening. 6. Phase 6 proof,
+then repeat the loop with whatever the numbers say is worst.
+
+## Phase 8 — the shape of the browser, 0.17.0
+
+Two things were wrong at the edge of the tool surface, and both were only
+reachable on real sites.
+
+The first was that the shape of the browser could not change. `sidecar_open`
+accepted `headless: false` and answered success; when a browser was already
+running nothing happened, because the flag was only read at launch. A site that
+refuses a windowless browser (chatgpt.com on this machine: a "just a moment"
+page forever, versus a login page in a windowed one) therefore had no way out
+except killing a process by hand. Since 0.17.0 a request for the other shape
+restarts the browser on the same profile, cookie jar and address — 21.6 s
+windowless → windowed, 22.1 s back, 0 ms when the shape already matches — and
+both `sidecar_open` and `sidecar_state` say which shape is in front of the
+person.
+
+The second was silence on a check page. A site asking for proof of a person is
+not broken and not an error page, so every read returned it as content. Now
+every read of it carries the way past: `headless false` on the same profile.
+
+Next in this direction: a site that only accepts a window is still discovered by
+the model, not by the plugin. The plugin could try the page in a window by itself
+when the read shows a challenge, but that puts a window on the person's screen
+without being asked — the current rule is that a person's screen is theirs.
+
+## Phase 9 — the mirror has to exist before it can be seen, 0.18.0
+
+Everything the plugin does is worth nothing if the person cannot see it, and the
+failure is silent: the browser runs, reads work, and the column is empty because
+the tab that draws it was never loaded. 0.18.0 makes that state say itself out
+loud, with the one action that fixes it, and only pays the wait when a column was
+actually requested.
+
+Next in this direction: the drawing half could re-attach by itself. A tab that
+reloads re-subscribes on its own, but a tab that never loaded the half cannot be
+told to — that is a load-order problem, so the fix belongs in how the panel half
+is delivered, not in the wait.