dsh-inner-browser-use 0.11.44 → 0.13.0

package/docs/ARCHITECTURE.md

@@ -1,172 +1,204 @@
 # Architecture
 
-Two halves, one wire.
+Two halves in two processes, and a Chrome that neither of them is.
 
 ```
-Host (plugin index.js)                Client (client.js, in the page)
-──────────────────────                ────────────────────────────────
-tools.register(browser_*)             the product's browser tab → <webview>
-        │                                       │
-        │  bridge.call(chat, op, args)          │  executeJavaScript(RUNTIME)
-        ▼                                       ▼
-   POST /push ────── SSE ──────►  dispatch op, reply via POST /uplink
-        ▲                                       │
-        └────────── /uplink ◄───────────────────┘
+server side  (index.js, in the DSH process)
+  sidecar_* tools ──> Sidecar ── CDP over a socket ──> headless Chrome
+                         │
+  HTTP + SSE routes <────┘
+        │
+        │ same origin as the product document, no other channel
+        │
+page side  (client.js, in the product document)
+  right column: JPEG picture, black arrow, input handlers, address row
 ```
 
-## Why the Host needs a wire at all
+## Why a wire
 
-The sidebar browser is a Client object. The `<webview>` is created in the product
-document, leased from the Desktop main process, and it only exists while the tab
-is mounted. A tool, however, executes on the Host. Nothing in the Host can reach
-a DOM element that lives in another process, so the two halves talk.
+The page half runs in the product document. It can paint a picture and read the mouse, and
+that is the limit: a page may not read another origin's DOM, and the product's own browser
+tab belongs to the product. The agent needs the page's text, its controls and real input
+events, which only the browser itself can give.
 
-## The transport
+So the server half owns a Chrome process and speaks CDP to it, and the two halves exchange
+two things over the plugin's own routes: frames going one way, input and navigation going the
+other. The picture is a consequence of that, not the mechanism.
 
-The Client `fetch`es relative URLs under `/dsh-inner-browser-use/`; the Host
-registers that prefix on its web server.
+## Chrome lives outside the product
 
-Same-origin is the whole trick, and it holds on both builds:
+The browser is a normal Chrome launched with `--user-data-dir` pointing at
+`~/.dsh/inner-browser-use/chrome/<profile>`, the profile name lowercased to `default`.
+Chrome's own storage is not used, and the product's browser storage is not used either: that
+one is created per process, so every sign-in in it is gone at the next restart.
 
-- **Web build:** the page already sits on the Host origin.
-- **Desktop build:** the product document is `dsh-app://app`. Its protocol
-  handler forwards every path it does not serve itself to the Host, with the
-  Host cookie attached. A relative `fetch` therefore lands on these routes with
-  no CORS preflight, no port number, and no token for the Client to learn.
+Other launch flags:
 
-| Route | Direction | Purpose |
-| --- | --- | --- |
-| `GET /downlink?session=` | Host → tab | Server-sent events, one long-lived response per open tab. |
-| `POST /uplink?session=` | tab → Host | Replies and lifecycle frames. |
-| `POST /push?session=` | Host → tab | Diagnostic push without a waiting call. Not used by tools. |
-| `GET /watch` | Host → page | Server-sent events, one long-lived response per page (not per tab): the channel the page uses to learn that some chat needs a tab. |
-| `GET /state` | either | Which chats have a tab, how many calls are in flight, last client heartbeat. |
-
-## Surviving a reload of the other half
-
-Each host half names itself (`epoch`) in every heartbeat reply. A page whose ask
-stream predates that name still holds an open socket to the old half, which can
-never open a tab. The next heartbeat (≤20 s) answers with a new name, and the page
-reopens the stream. Installing, or reloading the host, therefore needs no page
-reload.
-
-## Routing a chat to its tab
-
-The tool layer names a chat by its agent id; the tab the page opens may carry a
-different session id. Right after an ask, the page reports which tab it opened
-(`phase:"open"`, `opened:"current"`), and the host records that pair for the next
-30 s. Later calls resolve chat → tab through the pair, so two open tabs stay
-apart instead of both falling back to the single-seat rule.
-
-## Opening the tab on demand
-
-A tool cannot ask a user to click `+`, so the call carries the ask: when no seat
-exists, the Host pushes `wanted` down `/watch`, the page opens this chat's tab
-through `ctx.sidebarRight.openTabIn(sessionId, "browser")` — the product's own
-kind, the one with the address bar — and the call waits up to `openWaitMs` (8 s)
-for the seat to appear before it fails.
-
-`/watch` is per page rather than per tab on purpose: a page with no tab open has
-no `/downlink` stream, and it is exactly that page which must be able to obey.
-
-## Seats
-
-One open tab is one seat, keyed by the chat id. A tool resolves its seat by the
-agent id that called it and falls back to the only seat when exactly one tab is
-open — that fallback is what lets a chat drive a tab another chat opened, which
-is occasionally what you want while debugging.
-
-A seat that disappears mid-call rejects the waiter; it does not hang until the
-timeout. A seat that goes silent does hang until the timeout, which is
-deliberate: a page can be slow, and the model should see "did not finish" rather
-than a misleading success.
-
-## Frames
-
-```jsonc
-// Host → tab
-{ "type": "call", "id": "<random>", "tool": "click", "args": { "ref": 4 } }
-// tab → Host
-{ "type": "reply", "id": "<same>", "ok": true, "value": { "text": "…" } }
-{ "type": "reply", "id": "<same>", "ok": false, "error": "no control 4 on this page" }
-{ "type": "wanted", "session": "<chat id>" }
-{ "type": "client-alive", "phase": "registered", "tabType": true }
-```
-
-A navigation to the address the tab already shows is answered by `reloadIgnoringCache()`
-rather than `loadURL()`: a browser serves that second request from its own cache, so a
-page whose file changed on the machine would otherwise come back stale and the numbers
-handed to the model would point at text that no longer exists.
+| Flag | Why |
+| --- | --- |
+| `--headless=new` | No window on the desktop, still a full rendering engine. |
+| `--remote-debugging-port=0` | Chrome picks its own debugging address. | <!-- ui-copy-ignore -->
+| `--use-mock-keychain`, `--password-store=basic` | No keychain prompt on macOS. |
+| `--remote-allow-origins=*` | The CDP client connects over a plain WebSocket. |
+| `--disable-blink-features=AutomationControlled` | Sites that sniff for automation get the ordinary path. |
+| `--window-size=1440,900`, `--lang=zh-CN` | The viewport the pane mirrors, and a Chinese page locale. |
 
-Frames are JSON, not binary, and they carry text, not screenshots. A page
-observation is a few kilobytes of text; a screenshot would be megabytes over the
-same socket, and it is not what the model needs to decide where to click.
+The binary comes from a fixed list: Google Chrome, Chromium or Canary under `/Applications`,
+then `/usr/bin/google-chrome` and `/usr/bin/chromium`. Nothing else is probed and there is no
+setting to add a path.
 
-## The guest runtime
+## Attach and lifetime
 
-`client.js` keeps the whole in-page side as one string, evaluated per call:
-
-```js
-el.executeJavaScript(`(async function(){ ${RUNTIME}; return await run(${op}, ${args}) })()`)
-```
+The browser outlives the plugin:
 
-Rewriting it on every call costs a few milliseconds and buys immunity from the
-one failure that would otherwise break everything: the page navigating and
-taking an earlier injection with it. Nothing persists in the guest except what
-the call itself creates.
+1. `ensure()` reads `DevToolsActivePort` from the profile directory.
+2. It probes `GET http://127.0.0.1:<address>/json/version`. A stale file is deleted and Chrome
+   is launched again.
+3. Otherwise the running Chrome is adopted: the page target is read from `/json/list` (with
+   `/json/new?about:blank` as a fallback) and a client connects to it.
 
-`view` numbers every control it can see (`a`, `button`, `input`, `select`,
-`textarea`, `[role=…]`, `[onclick]`, `[tabindex]`) in document order, trimming
-invisible and zero-sized nodes, and returns the first 2400 characters of visible
-text. The numbers are only valid for the page the model last read, which is why
-every mutating tool returns a fresh `view` afterwards.
+One page per browser. `apply()` only constructs the `Sidecar`; no process exists until the
+first tool call, and `sidecar_state` reports a stopped browser without starting one.
 
-`click` and `type` scroll the target into view, move the pointer there, ripple,
-and dispatch real `MouseEvent`s (full `pointerdown/mousedown/mouseup/click`
-sequence) so frameworks that listen on any of them react.
+Shutdown asks Chrome to close (`Browser.close`), waits briefly, then sends `SIGTERM`, which is
+the point where Chrome flushes cookies. An idle timer closes the browser after 15 minutes; a
+tool call resets the clock, and a watching column defers the close when the timer fires, so a
+page someone is reading outlives the idle window.
 
-## The lease
+## Routes
 
-Only the rollback path (`NATIVE_MODE = false`) uses this. In the shipped mode the
-page drives the `<webview>` the product's browser tab already holds — it finds
-that node (right of the chat, `[data-sidebar-browser-frame="webview"]`, or a
-`webview` no other half parked) and never creates, moves, or disposes one.
+Everything hangs under `/dsh-inner-browser-use`, all same-origin, bodies capped at 1 MB.
 
-On the rollback path the tab does not create its own browser. It asks the Desktop
-main process:
-
-```js
-const reservation = await dshDesktop.browser.acquire("cwd:/path/to/workspace")
-webview.name = reservation.lease
-webview.partition = reservation.partition
-webview.src = `about:blank#${reservation.lease}`
-```
-
-All three attributes matter. The main process cancels `will-attach-webview` when
-the `src` hash does not match the lease it issued, so a webview configured
-loosely never attaches — it silently shows nothing. The partition is
-`dsh-sidebar-browser-<uuid>`, in memory, scoped to the workspace key: `cwd:<path>`
-for a chat in a workspace, `session:<id>` for an ungrouped one. Same key, same
-cookies as the built-in browser tab.
-
-In the shipped mode the partition is the product's own and outlives a restart, so
-no cookie bookkeeping is needed. On the rollback path that partition is in memory,
-so the page half files the cookies the page itself can read under `~/.dsh/inner-browser-use/cookies/<hash>.json`, mode 0600,
-keyed by the same workspace key: name, value, and the origin the page reported,
-nothing else, on this machine only. When that workspace's pane opens again, its
-remembered page loads, the kept cookies for that origin are written back, and the
-page loads once more — a cookie set from the page cannot affect a document that is
-already open. A workspace holds several sites, so a save replaces only the origin
-the page is on.
-
-The debugger is not an option for this: a `<webview>` element in the shipped app
-does not expose one, and the main process keeps the guest session to itself.
-Cookies marked `HttpOnly` are invisible to the page, so a login that relies only on
-those does not survive; anything the page reads from `document.cookie` does.
-
-## Failure modes, by design
-
-- **No tab:** the call asks for one, waits, and fails with one sentence if none opens.
-- **No Desktop:** the body renders "Desktop app only"; nothing else registers.
-- **Page refuses:** the error string reaches the model as a tool error.
-- **Plugin stops:** every seat closes, every waiter rejects.
+| Route | Kind | Purpose |
+| --- | --- | --- |
+| `GET /sidecar/frames?session=` | SSE | The mirror: `hello`, then frame messages, plus pointer messages. |
+| `GET /sidecar/state` | JSON | What the browser is doing. |
+| `POST /sidecar/open` | JSON | Start or reuse the browser, optionally navigating. |
+| `POST /sidecar/input` | JSON | `hover`, `move`, `click`, `wheel`, `text`, `key`; answers with the page's cursor. |
+| `POST /sidecar/nav` | JSON | `back`, `forward`, `reload`, or a `goto` with an address. |
+| `POST /sidecar/stop` | JSON | Close the browser now. |
+| `GET /downlink?session=` | SSE | Per-session notices; the only message today is `sidecar-wanted`. |
+
+The page half never learns the debugging address. It knows the route prefix and nothing else;
+the CDP socket stays in the server half.
+
+## Mirror
+
+The server half starts `Page.startScreencast` with JPEG quality 72, `maxWidth` and `maxHeight`
+at 1440×900, `optimizeForSpeed`, `everyNthFrame: 1`, and acknowledges every frame it receives
+so Chrome keeps sending. Frames are never queued: with two frames in flight, further frames
+are dropped and the next one that fits takes their place. A slow reader loses frames, never
+freshness.
+
+The viewport is pinned with `Emulation.setDeviceMetricsOverride` to the same 1440×900, so a
+frame is the whole page as laid out, not a crop.
+
+If `Browser.getVersion` reports `HeadlessChrome`, the user agent is overridden with an ordinary
+Chrome string, `acceptLanguage: 'zh-CN,zh;q=0.9,en;q=0.8'` and `platform: 'MacIntel'`, so a site
+that gates on headless gets the normal rendering.
+
+The stream is SSE with a comment beat every 15 seconds. A response that falls more than 3 MB
+behind is given up rather than buffered, Nagle is off on the socket, and a session that opens a
+second stream replaces its first seat: one column per session, no fan-out of stale frames.
+
+## Pointer, input, cursor
+
+Two pointers, two sources:
+
+- The agent's arrow comes from the tool calls. `movePointer()` sends `{type: 'pointer', w, h,
+  pointer}` on the same stream and the pane draws it at once, so the arrow does not wait for
+  the next frame. Hovering with your own mouse sends `mouseMoved` and never touches that state.
+- Your own cursor stays the system cursor. The pane asks the page which CSS cursor sits under
+  the hand (`cursorAt()`), the reply to `/sidecar/input` carries it, and the pane applies it
+  only when it is in the pane's allow-list.
+
+Clicks are `Input.dispatchMouseEvent` down and up at one point. Typing is `Input.insertText`:
+a composed Chinese word or a pasted string lands as one insertion rather than key by key.
+Named keys go through `Input.dispatchKeyEvent` with a table mapping Enter, Tab, Escape,
+Backspace, Delete, the arrows, Home, End, PageUp and PageDown to their key and code, which is
+what makes Backspace delete a character on the page instead of nothing.
+
+## Session scoping
+
+The right column is shared by every plugin and belongs to whichever session is on screen. A
+tool call pushes `sidecar-wanted` down that session's own `/downlink`, and the page half opens
+the mirror tab only when `sidebarRight.mounted` equals the session that asked. An unknown or
+empty `mounted` is a refusal, not permission.
+
+The frame stream is opened by the pane with its own `session` in the query. A chat whose column
+sits on another session gets no frames, and a tool call in the background drives the same
+browser without pulling the column away from the person reading it.
+
+## Tool surface
+
+`sidecarTools()` in `tools.js` builds the 25 definitions and `index.js` registers them on the
+plugin's tool scope. All of them return one text block. A page read numbers the controls and
+keeps those numbers across later reads, so `sidecar_click` by ref means the element the model
+last saw; navigation drops them and the model gets `REF_GONE` rather than a wrong click.
+
+An action answers with one line — address, title, focused field — and reads the page only when
+`read: true` is passed. That is what keeps a long batch cheap (see `BENCH.md`).
+
+Calls that touch the browser are marked not concurrency-safe; reading state, console and failed
+requests is. `sidecar_open`, `sidecar_view`, `sidecar_wait` and `sidecar_shot` carry longer
+timeouts than the rest.
+
+Sign-in state lives in the profile, and session cookies — the kind a sign-in usually sets,
+which Chrome would otherwise drop on exit — are saved next to it and restored on the next
+launch, so a restart of the app does not sign the person out.
+
+A system-prompt section (`browser-in-this-chat`) tells the model to reach for these tools
+without asking first when someone wants something done on a site.
+
+## Network
+
+`Network.enable` is already on for the mirror. A request is kept when it is
+sent and settled when its response arrives, so the list holds method, status,
+type and address of what the page actually got — 120 rows, oldest dropped.
+`sidecar_network` reads that list without touching the page, which is why it
+can answer while a dialog is open.
+
+## Downloads
+
+Chrome is pointed at `<profile>/downloads` when the page starts, so a click
+that saves a file writes it there instead of nowhere. The folder belongs to
+the profile: it is the same one on the next launch, and nothing outside the
+profile is touched. `sidecar_downloads` lists what is on disk, newest first;
+a `.crdownload` still in flight is not listed.
+
+## Frames and popups
+
+A page is not one document. Chrome keeps a table of frames and gives each one the
+JavaScript context its scripts run in; the plugin listens for those contexts as they
+are created and destroyed, so a read can be sent to the context that owns the
+controls it wants.
+
+- A read walks the frame tree main frame first and sends one snapshot per frame, so a
+  form inside an iframe is numbered with the page's own controls.
+- Refs from a frame are offset: `900000 + slot * 100000 + local`. The slot is the
+  frame's position in the tree, so the number says which context to resolve in, and a
+  frame's refs never collide with the page's.
+- A click on a framed control resolves in that frame, then adds the frame's rectangle
+  to the point it found, because input is always dispatched in viewport coordinates.
+- A frame Chrome runs in another process is a target of its own: the plugin keeps the
+  session Chrome hands it, so a read can be sent there and its controls still get a slot.
+  Its corner is asked from the page, through the frame's owner element, because the
+  frame's own process cannot see where it sits.
+
+A click can open a tab. A page target gets no handle on its own popups, so after an
+action the plugin asks the DevTools address list for page targets other than the one it
+drives. If one appears, the mirror navigates to its address and the popup is closed with
+the target HTTP endpoints. A person watching the column sees where the click went,
+and there is still one page to drive.
+
+## Files
+
+| File | Half |
+| --- | --- |
+| `index.js` | Server: routes, tools, CDP page reads, system prompt. |
+| `sidecar.js` | Server: Chrome launch, attach, screencast, input, shutdown. |
+| `client.js` | Page: the column, its stream, the arrow, the address row. |
+| `locale/*.json` | Page: pane labels. |
+
+Data on disk is confined to `~/.dsh/inner-browser-use/`: `chrome/<profile>` for the browser,
+`screenshots/` for `sidecar_shot`.

package/docs/BENCH.md

@@ -0,0 +1,124 @@
+<!-- ui-copy-ignore -->
+# Measured, not claimed
+
+`node scripts/bench.mjs` on a local page, headless, profile of its own. One JSON
+line per measurement; a phase is only done when its row improves.
+
+Method: a dense page (20 inputs, 120 links, two buttons), a text page (~60
+paragraphs), a light page. The dense page carries a viewport-fixed counter,
+because Chrome sends screencast frames only for visible change — measuring an
+idle page measures nothing.
+
+## Baseline — 0.12.1, before phase 1
+
+| What | Number |
+| --- | --- |
+| launch → first frame | 506 ms (cold profile) |
+| steady mirror | 60 fps, 72 KB/frame |
+| click → next frame | 13 ms |
+| click round trip | 1 ms |
+| key round trip | 13 ms |
+| screenshot | 82 ms, 133 KB PNG |
+| read: dense page | 4.7k chars ≈ **1383 tokens** |
+| read: text page | 8.0k chars ≈ **2363 tokens** |
+| read: light page | 69 chars ≈ 20 tokens |
+
+Targets: dense read ≤ 1200 tokens, text read ≤ 1500 tokens with the useful part
+kept, click → painted ≤ 120 ms (already 13 ms; the number to protect), and a
+batch of five steps in one tool call.
+
+## After phase 1 + 2 — read model v2 and batched steps
+
+Same bench, same machine. Read numbers are what a model actually receives with
+the default budget (4600 chars of page text).
+
+| What | Baseline | Now |
+| --- | --- | --- |
+| read: dense page | 1383 tokens | **436** tokens (90 controls, 1.5k chars) — `links` 514, `forms` 127, `full` 624 |
+| read: text page | 2363 tokens | **1130** tokens (screenful only, budget 4600) |
+| read: light page | 20 tokens | 17 tokens |
+| read round trip | ~30 ms | **1–4 ms** |
+| refs across reads | new numbers every read | **stable** (same refs, ticker-only text changes) |
+| click round trip | 1 ms | 1 ms |
+| key round trip | 13 ms | 0–2 ms |
+| launch → first frame | 506 ms | 129–492 ms (cold/warm profile) |
+| steady mirror | 60 fps, 72 KB/frame | 60 fps, 76 KB/frame |
+| screenshot | 82 ms / 133 KB | 79–86 ms / 115 KB |
+
+The win is structural: the old read took `document.body.innerText` (8k chars on
+the text page); the new one walks the viewport, dedupes parent text, numbers
+controls with refs that survive reads, and caps the text at the budget. Reading
+again returns the same numbers, so a batch of steps costs one read instead of
+one per step.
+
+## What one step costs the model
+
+Measured on the live tool surface (`/tmp/probe6.mjs`, 2026-10-27). Before this
+pass every action answered with a full page read; now it answers with one line
+and reads only when `read: true` is passed.
+
+| step | reply | ≈ tokens |
+| --- | --- | --- |
+| click | `ok click 1 · https://… · Title` (60 chars) | 18 |
+| type | `ok type · https://… · Title · focus email = a@b.c` (77) | 23 |
+| key | `ok Enter · https://… · Title` (65) | 20 |
+| check | `I agree is now on` (17) | 6 |
+| select | `chose Beta · https://… · Title · focus …` (65) | 20 |
+| wait | `saw hello` (9) | 3 |
+| console / failed | `error: message` or `no request failed` (17–47) | 6–14 |
+| dialog | `alert: sure?` (12) | 4 |
+| cookie | one line per cookie, up to 40 | — |
+| full read (dense page) | 1484 chars | 436 |
+
+A ten-step batch therefore spends ~150 tokens on confirmations instead of
+~4 400, and the page itself is read once when it changes.
+
+## Notes
+
+- Screencast frames are change-driven. A page with nothing moving sends nothing;
+  that is Chrome behaving, not the mirror failing. After an agent action the host
+  should force one frame so the person sees the result of a step that changed
+  nothing visible.
+- A tall page whose only animation is below the fold produces no frames: the
+  viewport is what is captured.
+
+## Phase 3 — frames and popups, 0.13.0
+
+Same bench, same page. Two changes landed: controls are capped at the read's own
+`max` after viewport-first sorting (the cap was applied to candidates, not to the
+answer), and a read now also asks for the frame tree.
+
+| What | Before | After |
+| --- | --- | --- |
+| read: dense, `full` | 3118 chars, 142 controls ≈ 917 tokens | 2128 chars, 90 controls ≈ **626 tokens** |
+| read: dense, `text` | 1510 chars, 142 controls ≈ 444 tokens | 1510 chars, 90 controls ≈ 444 tokens |
+| read: text page | 13 ms | 13 ms (one frame-tree call, ~1 ms) |
+| launch → first frame | 506 ms | 479 ms |
+| click round trip | 1 ms | 2 ms |
+| key round trip | 13 ms | 2 ms |
+| screenshot | 82 ms, 133 KB | 77 ms, 113 KB |
+
+Breadth run (`npm run breadth`), 25 checks, all true: open shadow DOM read and
+click, form typing, select, checkbox, submit, `confirm()` seen and accepted,
+iframe read, iframe typing, iframe click, cross-site frame read and click,
+popup followed, download saved, network responses listed, a file uploaded
+into a file input, the page printed to a PDF, a changed-only read, names from `for=` and `aria-labelledby`, a click found by its words, a login filled by field names in one call,
+stale ref refused, a crashed renderer picked back up on the same page.
+
+## Phase 4 — what a change costs
+
+| What | Number |
+| --- | --- |
+| change in the page → frame ready for the column | 9 ms median, 17 ms max (12 samples) |
+| `changed_only` after a field changes, dense page | 58 chars ≈ **17 tokens** (whole page: 444) |
+| read: text page, `changed_only` | 42 chars ≈ 12 tokens (whole page: 1517) |
+| renderer crashed → readable page again | **1.4 s** (detect, fresh tab, same address) |
+
+The mirror's delay is measured from the moment the page is told to change to
+the frame object the column would draw — capture, JPEG, and the CDP hop. It is
+the part of the latency that belongs to this side; what a person sees adds the
+browser's compositing and the SSE hop, both smaller.
+
+A cross-site frame (Chrome runs it in its own process) costs one more round
+trip than a same-process one: its controls arrive in the same read, and the
+frame's corner is one `DOM.getFrameOwner` plus one call on the owner element.