dsh-inner-browser-use 0.13.0 → 0.27.0

package/README.md

@@ -12,7 +12,8 @@ live picture of that browser, and your own cursor keeps working on the picture.
 | --- | --- |
 | `sidecar_open` | Starts the browser (or reuses the running one) and opens an address. `headless: false` shows a real window on the desktop instead. |
 | `sidecar_view` | Returns the address, the title, every control as `1 … 120`, and the visible text. |
-| `sidecar_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. |
+| `sidecar_extract` | Returns the items a CSS selector matches, one compact line each with its link, instead of the whole page. |
+| `sidecar_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. A cookie banner or modal over the control is cleared first, then the click happens. |
 | `sidecar_type` | Types text into the focused field, or clicks a numbered control first. Chinese goes in as one string. `submit: true` presses Enter. |
 | `sidecar_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, ArrowUp/Down, Home, End, PageUp, PageDown. |
 | `sidecar_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
@@ -94,8 +95,8 @@ In the app: **Plugins → Install from folder**, and pick the clone. To install
 tarball instead, build it from that directory and install the file:
 
 ```bash
-npm pack                                  # writes dsh-inner-browser-use-0.13.0.tgz
-dsh plugin install ./dsh-inner-browser-use-0.13.0.tgz
+npm pack                                  # writes dsh-inner-browser-use-0.16.0.tgz
+dsh plugin install ./dsh-inner-browser-use-0.16.0.tgz
 ```
 
 The plugin appears under **Plugins** as *Background browser*. It needs Google Chrome or
@@ -118,6 +119,9 @@ opens before a call: a chat that never asked for a page keeps its column as it w
   profile folder, and `sidecar_downloads` names it, its size and when it arrived.
 - **The network is visible.** `sidecar_network` lists what the page asked for, newest first,
   so a page that fills itself in from an address can be understood without a screenshot.
+- **An overlay does not eat the action.** A cookie banner over the button is cleared the way a
+  person would clear it, then the click lands — in the same call, and in every step of a
+  `sidecar_do` batch. Anything offering to delete, pay or sign out is reported instead.
 
 ## What it does not do
 

package/README.zh.md

@@ -11,7 +11,8 @@ profile，不是产品自带的那个浏览页签：右栏放的是它的画面
 | --- | --- |
 | `sidecar_open` | 启动浏览器（已在跑就复用）并打开一个地址。传 `headless: false` 会改为在桌面上开真实窗口。 |
 | `sidecar_view` | 返回地址、标题、编号成 `1 … 120` 的控件，以及可见文字。 |
-| `sidecar_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。 |
+| `sidecar_extract` | 按 CSS 选择器把列表项读成一行一条、带链接，比整页读省得多。 |
+| `sidecar_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。控件被横幅或弹层挡着时，先关掉它再点。 |
 | `sidecar_type` | 往焦点输入框输入文字，也可以先点一个编号控件。中文整段送进去。`submit: true` 回车提交。 |
 | `sidecar_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
 | `sidecar_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
@@ -102,6 +103,7 @@ dsh plugin install ./dsh-inner-browser-use-0.12.1.tgz
 - **iframe 能读，跨站也能。** 不管是和页面共用进程，还是 Chrome 单独给一个进程，里面的控件都和页面自己的控件一样有编号；点击按框架在页面上的实际位置落点。
 - **下载会落到磁盘。** 点一下就下载的文件写在这个浏览器自己的 profile 目录里，`sidecar_downloads` 报出文件名、大小和到达时间。
 - **能看到网络。** `sidecar_network` 倒序列出页面请求了什么，自己用接口填内容、屏幕上又看不出原因的页面，不用截图去猜。
+- **遮罩不吞动作。** 盖在按钮上的 cookie 横幅会按人的做法先关掉再点下去，同一次调用里完成；`sidecar_do` 的每一步都一样。写着删除、支付、退出的遮罩只报告，不代按。
 
 ## 不做什么
 

package/docs/ARCHITECTURE.md

@@ -131,11 +131,25 @@ browser without pulling the column away from the person reading it.
 
 ## Tool surface
 
-`sidecarTools()` in `tools.js` builds the 25 definitions and `index.js` registers them on the
+`sidecarTools()` in `tools.js` builds the 28 definitions and `index.js` registers them on the
 plugin's tool scope. All of them return one text block. A page read numbers the controls and
 keeps those numbers across later reads, so `sidecar_click` by ref means the element the model
 last saw; navigation drops them and the model gets `REF_GONE` rather than a wrong click.
 
+Before a ref click or a ref focus, one page-side probe asks what is on top of that control.
+When something is, and that something carries its own safe way out (a banner with Accept /
+Close, a modal with 同意 / 关闭), the probe's answer names that button's point, the driver takes
+it, checks the control is clear, and then performs the action the model asked for. The result
+says both things. An overlay whose wording involves deleting, paying or signing out is only
+reported, never accepted on the person's behalf. The whole exchange is one `Runtime.evaluate`
+and a possible `mousePressed`/`mouseReleased` pair on an already-open CDP connection, so an
+uncovered click pays nothing.
+
+`sidecar_do` runs the same probe before a step that names a control — by `ref`, by
+text, or by field label — so a banner the first step clears is gone for the rest of
+the batch, and the step's own line says it in one clause (`after Accept on a banner,`).
+A batch therefore pays for one overlay, not one per step.
+
 An action answers with one line — address, title, focused field — and reads the page only when
 `read: true` is passed. That is what keeps a long batch cheap (see `BENCH.md`).
 

package/docs/BENCH.md

@@ -41,7 +41,7 @@ the default budget (4600 chars of page text).
 | refs across reads | new numbers every read | **stable** (same refs, ticker-only text changes) |
 | click round trip | 1 ms | 1 ms |
 | key round trip | 13 ms | 0–2 ms |
-| launch → first frame | 506 ms | 129–492 ms (cold/warm profile) |
+| launch → first frame | 506 ms | 129–539 ms (cold/warm profile) |
 | steady mirror | 60 fps, 72 KB/frame | 60 fps, 76 KB/frame |
 | screenshot | 82 ms / 133 KB | 79–86 ms / 115 KB |
 
@@ -98,21 +98,41 @@ answer), and a read now also asks for the frame tree.
 | key round trip | 13 ms | 2 ms |
 | screenshot | 82 ms, 133 KB | 77 ms, 113 KB |
 
-Breadth run (`npm run breadth`), 25 checks, all true: open shadow DOM read and
+`sidecar_extract` on a 30-row list: 1106 chars for all 30 rows (≈330 tokens),
+against 1291 chars for the same page read as text — and at a 500-char read
+budget the text read stops around row 12 while the list still returns 30.
+
+Live run (`npm run live`), 8 checks, all true on the open internet: open
+example.com in 566 ms, read it for 785 chars, click the language link on
+wikipedia.org and land on en.wikipedia.org in 2.3 s, go back to
+www.wikipedia.org, extract ten list rows from the Web scraping article in 3 ms
+(1813 chars), visit a host that cannot resolve (1.3 s to the browser's own
+error page, where a read, a click and a batch all refuse instead of quoting it),
+recover from there to a real page, and type into a search box and reach a
+results URL in 1.3 s. These are wall-clock end to end including the site's own
+load; the local numbers below are where the driver's own cost lives. Two hosts
+were unusable from this machine and are not plugin problems: Hacker News (12 s
+connect timeout, confirmed with curl) and Wikipedia's search redirect, which
+answers on the home page and times out on the query — the search check runs on
+Bing now, driven by the box's words rather than a control number.
+
+Breadth run (`npm run breadth`), 33 checks, all true: open shadow DOM read and
 click, form typing, select, checkbox, submit, `confirm()` seen and accepted,
 iframe read, iframe typing, iframe click, cross-site frame read and click,
 popup followed, download saved, network responses listed, a file uploaded
 into a file input, the page printed to a PDF, a changed-only read, names from `for=` and `aria-labelledby`, a click found by its words, a login filled by field names in one call,
-stale ref refused, a crashed renderer picked back up on the same page.
+stale ref refused, a crash picked back up on the same page, a list read as 30 rows, a click under a full-screen cookie banner clears the banner and lands on the
+control, typing into a field under that banner works the same way, and an
+overlay that offers to delete the account is reported but never accepted.
 
 ## Phase 4 — what a change costs
 
 | What | Number |
 | --- | --- |
-| change in the page → frame ready for the column | 9 ms median, 17 ms max (12 samples) |
+| change in the page → frame ready for the column | 7 ms median, 7 ms max (12 samples) |
 | `changed_only` after a field changes, dense page | 58 chars ≈ **17 tokens** (whole page: 444) |
 | read: text page, `changed_only` | 42 chars ≈ 12 tokens (whole page: 1517) |
-| renderer crashed → readable page again | **1.4 s** (detect, fresh tab, same address) |
+| renderer crashed → readable page again | **1.39 s** (detect, fresh tab, same address) |
 
 The mirror's delay is measured from the moment the page is told to change to
 the frame object the column would draw — capture, JPEG, and the CDP hop. It is
@@ -122,3 +142,202 @@ browser's compositing and the SSE hop, both smaller.
 A cross-site frame (Chrome runs it in its own process) costs one more round
 trip than a same-process one: its controls arrive in the same read, and the
 frame's corner is one `DOM.getFrameOwner` plus one call on the owner element.
+
+## Phase 5 — what an overlay costs
+
+| What | Number |
+| --- | --- |
+| the "what is on top of this control" probe | **1 ms median, 2 ms max** (5 samples) — the same `Runtime.evaluate` hop, no round trip added |
+| `sidecar_click` on an uncovered control | 460 ms median (3 runs, page-load settle included) |
+| `sidecar_click` on the same control under a cookie banner | 630 ms median (3 runs) — the banner's own Accept click, a 150 ms beat, a re-probe, then the intended click |
+| the same under a modal that offers to delete the account | 456 ms — reported, nothing pressed |
+| one `sidecar_do` step on an uncovered control | 458 ms median (3 runs) |
+| the same step under the cookie banner | 633 ms median (3 runs) — cleared, then the step ran |
+
+A batch pays it once per step that has something over it, not once per call: a
+banner cleared by step 1 is gone for steps 2..n, so a five-step flow through a
+cookie wall costs one extra step's worth of time, not five.
+
+The probe costs nothing when nothing is in the way, which is the common case;
+the covered case pays about 170 ms to clear the banner and land the click in
+the same call, instead of making the model spend a whole extra turn (a read, a
+click, another read) telling it to close the banner first.
+
+## Phase 6 — what a stale number costs, 0.16.0
+
+A ref is a number for an element. A page that redraws a list, a table, a card
+grid — React, Vue, Svelte, or plain `innerHTML` — throws every element away and
+makes new ones, so the numbers from the last read stop pointing anywhere. The
+control is still on screen, saying the same words, usually in the same place.
+
+Before this phase that cost two round trips: the call came back `That control is
+gone`, the model read the page again to get a new number, then used it. The work
+is now done inside the call, by looking the control up by what it is.
+
+| What | Number |
+| --- | --- |
+| `sidecar_click` on a control the page just redrew | **462 ms median, 483 ms max** (8 samples) — one extra read of the page's visible text, 600 chars, to find the control again |
+| `sidecar_click` on a control that was not redrawn | 445 ms median, 446 ms max (8 runs) — the probe costs nothing when the number is still good |
+| stale refs recovered this way | **8/8** |
+| a control that really left the page | still refused, `Call sidecar_view and use the new number` — 2/2 in the breadth suite |
+
+So the recovery costs about **17 ms** when it is needed and nothing when it is
+not, against a model round trip of two calls plus the tokens of a fresh read.
+
+What counts as the same control: the same words and the same kind. A button does
+not become a field; a field never passes for the button beside it; a link keeps
+its destination, because the same words pointing somewhere else is a different
+thing to click. The words have to appear **exactly once** on the page — two
+"Delete" buttons in two rows are a choice only the person can make, so that case
+still asks for a read.
+
+The call says what it did: `(the page redrew, so "Learn more" has a new number — used where it is now)`, so the model knows its old number is not the one to keep
+using. On the live network the same thing happened on example.com: a link
+replaced by an identical clone, one click, 4.4 s including the iana.org page
+load, straight to `https://www.iana.org/help/example-domains`.
+
+## Phase 7 — what changing the shape of the browser costs, 0.17.0
+
+A site that refuses a windowless browser is the one case where the shape has to
+change mid-session. Changing it restarts the browser, so it costs a cold start:
+
+| what | median | notes |
+| --- | --- | --- |
+| windowless → windowed, same profile and page | 21.6 s | one process, cold start |
+| windowed → windowless, same profile and page | 22.1 s | one process, cold start |
+| asking for the shape it already has | 0 ms | nothing is restarted |
+
+Measured through the tool layer (`sidecar_open` → `sidecar_state` → `ps` spawn
+args), profile `probe-tool`, 1 run each; the same-shape row is the third call in
+the same run. A browser start on this machine is 20–22 s either way, so the
+switch is not what costs — the process is.
+
+Reading a check page costs nothing: the sentence is built from the title already
+in the read, and a normal page gets `''` for one comparison.
+
+## Phase 8 — saying it when nobody is drawing, 0.18.0
+
+The mirror is drawn by the tab in the chatting column, and that tab attaches
+while the page is loading. Waiting for it costs nothing when it arrives, and a
+bounded wait is the difference between a silent `viewers 0` and a sentence:
+
+| what | cost |
+| --- | --- |
+| `sidecar_open` with a tab watching | 0 ms |
+| `sidecar_open` with nothing watching | ≤ 1.2 s, and the line says so |
+| `sidecar_open` when the chat has no column to draw into | 0 ms (no wait) |
+| building the sentence on a normal state read | 0 ms, one comparison |
+
+Measured with `/tmp/probe-unseen.mjs` (tool layer, example.com, profile
+`probe-tool`): 1344 ms with nothing watching and no column asked for, 3236 ms in
+the first version of the same wait that did not skip the case where no column was
+ever requested — the whole difference was the wasted 1.2 s.
+
+## Phase 9 — finding the field behind the words, 0.19.0
+
+A read of a `/bare` form (two div labels, two bare inputs, no aria anywhere on
+the fields) and one `{field: "Work email"}` step:
+
+| what | result |
+| --- | --- |
+| the click lands on the field, not on the div | yes, value `me@x.y` in `input[name=wk]` |
+| the second, untouched field | still empty |
+| the tool's own words | `typed into "Work email"` |
+| worst case when nothing matches | one extra in-page query, then the old behaviour |
+
+The query walks the open shadow roots (capped at 400) and, per visible editable
+element, builds a list of candidate names from real labels, `aria-labelledby`,
+`aria-label`, `placeholder`, `title`, `name`, `label[for]`, the element right
+before it, and its wrapper's text — then scores exact (3) / contains (2) /
+reverse-contains (1).
+
+## Phase 10 — typing by name, 0.20.0
+
+| what the caller has | what it used to take | now |
+| --- | --- | --- |
+| a name for the field, no number | read the page, find the number, then type | one `sidecar_type` with `field`, one round trip |
+| the field's words live in a div | click landed on the div, text went wherever focus was | resolved to the field, `typed into "Work email"` |
+| a banner over the field | typed under the banner | banner cleared first, then typed |
+
+Breadth check `type-by-words` on the `/bare` form: value lands in
+`input[name=wk]`, the page does not move (no stray Enter).
+
+## Phase 11 — choosing a dropdown by name, 0.21.0
+
+`/bare` now carries a `select[name=seats]` whose only name on screen is a div
+saying `Seats`. One `sidecar_select` with `field: "Seats"`, `value: "Two"`:
+
+| what | result |
+| --- | --- |
+| the value after the call | `2` |
+| the tool's own words | `chose Two in Seats` (on-screen name, not `name="seats"`) |
+| a wrong option | answered with the first 12 options instead of a page read |
+| round trips to do it | one; no read beforehand, no read after |
+
+The naming rules (`labelSpot` and `selectSpot`) now live in one shared in-page
+block, so the two steps cannot drift apart.
+
+## Phase 12 — words split across nodes, 0.22.0
+
+`/words` now carries `<button><span>Sign</span><span>in</span></button>`, which
+the page reports as `Signin`.
+
+| what | result |
+| --- | --- |
+| `{text: "Sign in"}` before | not found; caller had to fall back to a number |
+| `{text: "Sign in"}` now | clicked, title changes, `ok click "Sign in"` |
+| extra work | none: the same single control list, one comparison changed |
+
+Exact equality still wins over the squeezed form, so `Sign in` never lands on
+`Sign up` when both exist.
+
+## Phase 13 — the field behind the box, 0.23.0
+
+`/bare` gains the date-picker shape: a visible rounded box with a transparent
+`<input>` laid over it, its words in a div two levels up.
+
+| what | before | now |
+| --- | --- | --- |
+| `{field: "Birthday"}` | not found, fell back to clicking the div | found the hidden input, clicked its visible box |
+| value after one `sidecar_do` step | whatever already had focus | `2001-02-03` in `#bd`, `typed into "Birthday"` |
+| extra round trips | read the page, guess a number | none |
+
+The visible-box rule: a candidate is usable when it, or its nearest visible
+ancestor, has a real box — so a 0-opacity input still counts if it covers
+something the person can see.
+
+## Phase 15 — the answer is the picture, 0.25.0
+
+`/pic` is one 140x90 PNG and nothing else.
+
+| what | result |
+| --- | --- |
+| `sidecar_grab` | 6 ms, `picture-check.png`, 140x90 |
+| before | three round trips: list the images, ask for the full address, fetch it |
+
+A picture page has no text at all, so a read of it costs one line and no tokens;
+the picture comes back as a file path the person can open.
+
+## Phase 16 — say what the picture looks like, 0.26.0
+
+Two 140x90 PNGs: one a gradient, one a single colour.
+
+| page | what `sidecar_grab` answers | time |
+| --- | --- | --- |
+| `/pic` | `... 140x90 a picture with detail` | 21 ms (first, page cold) |
+| `/flat` | `... 140x90 mostly one colour` | 4 ms |
+
+The sampling is 32x32 inside the page, so it costs no round trip and its result
+travels in the same line as the path.
+
+## Phase 17 — the answer is a picture, in the read itself, 0.27.0
+
+| page | read line | extra round trips |
+| --- | --- | --- |
+| `/pic` (gradient, no words) | `picture 140x90 a picture with detail "Chart" — sidecar_grab saves it as a file` | 0 |
+| `/flat` (single colour) | `picture 140x90 mostly one colour "Chart" — …` | 0 |
+| `/words` (text page) | no picture line | 0 |
+
+The sampling is 32x32 inside the page, in the same call that reads the page, so a
+blank chart is visible before anything is saved.
+

package/docs/ROADMAP.md

@@ -84,14 +84,25 @@ stealing the column from other chats.
 
 ## Phase 5 — reliability and safety
 
-- Human takeover: typing in the pane pauses the agent's next action and says so;
-  a resume action exists.
+- Human takeover (0.17.1): real input in the pane (click / text / wheel / key)
+  holds the agent's next action for 6 s and says so; hover and move do not count;
+  reads stay open; `sidecar_resume` takes the page back at once.
 - Guarded actions (upload, download, close tab, buy-ish submits) can require a
-  confirm flag.
+  confirm flag. Shipped in 0.14.0 for the narrow case that matters most: an
+  overlay in the way is cleared only when its own way out is a safe word
+  (accept / agree / close / 同意 / 关闭 …) and the overlay says nothing about
+  deleting, paying or signing out; otherwise it is reported, never taken.
+  Since 0.15.0 the same clearing runs before every control-naming step of a
+  `sidecar_do` batch, so a multi-step flow through a cookie wall fails zero times.
 - Crash recovery: attach to a still-running Chrome, rebuild the page, keep the
   profile; a killed host never leaves a zombie Chrome.
 - Every failure names what happened, what state the page is in, and the next
   call to make.
+- A failure that is the browser's, not the page's, is said in those words: on
+  Chrome's own error page a read, a click and a batch step all refuse with the
+  `ERR_…` code and the way out, while the tab list, console, failed requests and
+  address still answer, because those are how a stuck person finds out what
+  happened.
 
 ## Phase 6 — proof and packaging
 
@@ -100,9 +111,124 @@ stealing the column from other chats.
 - `docs/VERIFY.md` walk-through that a stranger can run; `scripts/doctor.mjs`
   green; 3 linters zero errors; tests covering every new tool.
 - Packed tarball, CHANGELOG per release, `0.x` bumps that match behaviour.
+  0.13.4 and 0.14.0 shipped tarballs without `tools.js` and `read.js` — a clone
+  hid it because the dev profile symlinks the repo. Since 0.15.0 a test walks the
+  import graph against `files`, and another unpacks the tarball and imports it.
+
+## Phase 7 — recovery inside the call, 0.16.0
+
+The measured weak spot after Phase 5 was not speed but turns spent recovering:
+every framework redraw invalidated the numbers from the last read, and the model
+paid a full read plus a turn to get them back. 0.16.0 moves that into the call by
+finding the control again by identity (kind + exact words + link destination),
+refusing when the words are not unique. Measured at 462 ms against 445 ms for a
+fresh control — 17 ms when needed, nothing when not.
+
+Since 0.16.1 the count is visible: `sidecar_state` reports `refs followed N`,
+and breadth asserts it moved after a redraw, so the next rounds can tell from
+real sites whether redrawing is frequent enough to justify more work here.
+Next in this direction: the same identity recovery for `sidecar_do` steps that
+name a field by its label when the label element is rebuilt (today the step
+falls back to a fresh read).
 
 ## Order of work
 
 1. Phase 0 baseline. 2. Phase 1 + 2 together (they share the read model).
 3. Phase 3 breadth. 4. Phase 4 numbers. 5. Phase 5 hardening. 6. Phase 6 proof,
 then repeat the loop with whatever the numbers say is worst.
+
+## Phase 8 — the shape of the browser, 0.17.0
+
+Two things were wrong at the edge of the tool surface, and both were only
+reachable on real sites.
+
+The first was that the shape of the browser could not change. `sidecar_open`
+accepted `headless: false` and answered success; when a browser was already
+running nothing happened, because the flag was only read at launch. A site that
+refuses a windowless browser (chatgpt.com on this machine: a "just a moment"
+page forever, versus a login page in a windowed one) therefore had no way out
+except killing a process by hand. Since 0.17.0 a request for the other shape
+restarts the browser on the same profile, cookie jar and address — 21.6 s
+windowless → windowed, 22.1 s back, 0 ms when the shape already matches — and
+both `sidecar_open` and `sidecar_state` say which shape is in front of the
+person.
+
+The second was silence on a check page. A site asking for proof of a person is
+not broken and not an error page, so every read returned it as content. Now
+every read of it carries the way past: `headless false` on the same profile.
+
+Next in this direction: a site that only accepts a window is still discovered by
+the model, not by the plugin. The plugin could try the page in a window by itself
+when the read shows a challenge, but that puts a window on the person's screen
+without being asked — the current rule is that a person's screen is theirs.
+
+## Phase 9 — the mirror has to exist before it can be seen, 0.18.0
+
+Everything the plugin does is worth nothing if the person cannot see it, and the
+failure is silent: the browser runs, reads work, and the column is empty because
+the tab that draws it was never loaded. 0.18.0 makes that state say itself out
+loud, with the one action that fixes it, and only pays the wait when a column was
+actually requested.
+
+Next in this direction: the drawing half could re-attach by itself. A tab that
+reloads re-subscribes on its own, but a tab that never loaded the half cannot be
+told to — that is a load-order problem, so the fix belongs in how the panel half
+is delivered, not in the wait.
+
+## Phase 10 — words are not where the field is, 0.19.0
+
+The tool used to treat "the control that says Email" as "the field called
+Email". Real forms separate the two, so the writing step now resolves the words
+to the element that takes text, and says which one it used.
+
+Next in this direction: the same resolution for `sidecar_type` when the caller
+names a field by its words instead of a ref, and for date/time pickers where the
+visible box is a div and the real input is hidden beside it.
+
+## Phase 11 — say where, not which number, 0.20.0
+
+Typing by name is in. The same resolution is what other steps should be able to
+use when the caller knows the destination but not the number — the next place
+worth doing this is `sidecar_select` (choose an option by the field it belongs
+to), and after that the date/time case where the visible box is a div and the
+real input sits hidden beside it.
+
+## Phase 12 — the same words, the same place, 0.21.0
+
+`sidecar_select` joined `sidecar_type` and `sidecar_do` on the shared resolver.
+Left in this direction: the date/time case where the box the user sees is a div
+and the real input is hidden beside it, and `sidecar_click` on a control whose
+words are split across sibling nodes (`<span>Sign</span><span>in</span>`).
+
+## Phase 13 — the page's spelling is not the caller's, 0.22.0
+
+Whitespace is normalised in both halves now (tool side and in-page side). Next
+in this line: the date/time box where the visible box is a div and the real
+input is hidden beside it, which needs the resolver to accept a hidden target
+and click the thing in front of it.
+
+## Phase 14 — behind the box, 0.23.0
+
+Both halves now resolve a name to a place: visible field, hidden field behind a
+visible box, words split over nodes, words outside the wrapper. Next: a page
+whose visible box is a `<canvas>` or an `<iframe>` — nothing in the tree to
+name, so the fallback has to be a picture plus a click, not a name lookup.
+
+## Phase 16 — a file, not a path, 0.25.0
+
+`sidecar_grab` returns a path. Next: say one line about the picture (size, and
+whether it is mostly one colour) so a model that cannot open files still learns
+"the chart is blank" without a full read.
+
+## Phase 17 — a picture nobody can see, 0.26.0
+
+The description is coarse on purpose. Next: when a page's answer is a picture,
+let a read say so in one line ("the answer is a picture, and it is blank") so a
+model that cannot open files does not need the tool at all to notice a failure.
+
+## Phase 18 — a picture that never arrived, 0.27.0
+
+A picture is only an answer if it loaded. Next: report a large picture element
+whose source failed (`naturalWidth === 0`) as a broken picture rather than
+staying silent, since that is the commonest way a generated image "disappears".
+

package/docs/VERIFY.md

@@ -87,12 +87,79 @@ still hold the cookies.
   内置浏览器 means the page half is stale: reload the GUI page.
 - Guest-era logins are not migrated; sign in once in the new profile.
 
+## Live
+
+`node scripts/live.mjs` does the same work on the open internet, with no local
+fixture: it opens `example.com` (566 ms), reads it, clicks the language link on
+`wikipedia.org` and lands on `en.wikipedia.org` (2.3 s, the page's own load
+included), goes back (URL back to `wikipedia.org`), reads ten list items out of
+the Web scraping article in one call (3 ms, 1813 chars), visits a host that
+cannot resolve, and types into a search box then presses Enter (1.3 s to a
+results URL). Eight JSON lines; eight `true` is the bar. A site being
+unreachable is reported as such and is not a plugin failure — the run says
+which site and how long it waited.
+
+Two of those checks are about failure rather than success. `live-dead-host`
+opens `https://never-resolves-this.invalid/`, waits for the browser to give up,
+then asks three tools — a read, a click, a batch — what the page says: all
+three must answer "the network could not reach that address", and none may
+quote the browser's own `ERR_…` text or `This site can't be reached` back as
+page content. `live-dead-host-recovers` navigates from there to a real address
+and requires the real page. The search check drives Bing by the box's words
+(`field: '搜索词'`) rather than a control number, because a search box is a
+`combobox` on one engine and an `input` on another; it used to use Wikipedia,
+whose home page answers while its search redirect times out on this network —
+three runs in a row failed there and said nothing about the tools.
+
+None of the numbers are the plugin's own latency: they are dominated by the
+site and the network, which is the point of having them next to the local
+measurements in [BENCH.md](BENCH.md).
+
+## The packed file is whole
+
+A clone works even when `package.json`'s `files` list is wrong, because the repo
+is right there. A tarball does not. After `npm pack`, the listing must contain
+the four modules the host half imports:
+
+```bash
+tar tzf dsh-inner-browser-use-*.tgz | grep -E 'package/(tools|read)\.js'
+```
+
+Two lines, or the install will fail with `Cannot find module` on first call.
+`node --test test/*.test.mjs` covers both halves of this: one test walks the
+import graph and fails when an imported file is not in `files`, another unpacks
+the tarball and imports it from outside the repo.
+
 ## Breadth
 
 `node scripts/breadth.mjs` drives a shadow-DOM component, a form (type, select,
 checkbox, submit), a native dialog, an iframe (read, type, click), a click that
 opens a tab, and a dead ref against local pages, and prints one JSON line per
-check. Twenty-seven `true` lines is the bar; a `false` line names the check that
+check. Forty-two `true` lines is the bar; a `false` line names the check that
 broke. Two of them cover a cross-site frame, which Chrome runs in its own
-process.
+process. Three cover a control under an overlay: the click clears a cookie
+banner and lands (`under clicked`), typing into a field under the same banner
+works, and a modal that offers to delete the account is reported without
+anything being pressed. Two more drive the same overlay through `sidecar_do`:
+a batch step clears the banner before it clicks, and a batch step fills a field
+that was under it. One more counts the recoveries: after the redraw check, the
+state line says at least one ref was followed to its new number. Three cover a
+hand on the mirror: while the person is using it a read still works
+(`hold-allows-read`), an action is refused and names both ways out
+(`hold-refuses-action`), and `sidecar_resume` hands the page back so the same
+click goes through (`resume-lets-go`).
+
+## Who the mirror serves
+
+The bar that matters most cannot be reached from `node --test`, because it is
+about a column that belongs to one conversation. With DSH running:
+
+```sh
+node scripts/live-safety.mjs http://127.0.0.1:49552
+```
 
+It reads the state, then tries the two ways in — a frame stream and a
+navigation — as a session that is not the owner. Both must answer 403, and the
+owner's stream must answer 200. Three `true` lines is the bar; `open · owner
+none` as the first line only means nothing is on screen, and the two refusals
+still have to hold.

package/index.js

@@ -285,6 +285,9 @@ export function apply(ctx) {
           await page.key('keyDown', spec)
           await page.key('keyUp', { ...spec, text: undefined })
         }
+        /* Hovering and moving are looking, not acting: only real input counts as
+           the person taking the page. */
+        if (kind !== 'hover' && kind !== 'move') sidecar.humanUse({ kind, x, y })
         json(200, cursor === undefined ? { ok: true } : { ok: true, cursor })
       } catch (error) {
         json(500, { ok: false, text: String(error?.message ?? error) })

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-inner-browser-use",
-  "version": "0.13.0",
+  "version": "0.27.0",
   "description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
   "type": "module",
   "main": "index.js",
@@ -14,6 +14,8 @@
   "files": [
     "index.js",
     "sidecar.js",
+    "tools.js",
+    "read.js",
     "client.js",
     "locale/*.json",
     "cordis.patch.yml",
@@ -42,10 +44,14 @@
   "scripts": {
     "test": "node --test test/*.test.mjs",
     "doctor": "node scripts/doctor.mjs",
-    "breadth": "node scripts/breadth.mjs"
+    "breadth": "node scripts/breadth.mjs",
+    "live": "node scripts/live.mjs"
   },
   "license": "MIT",
-  "author": "regrevia",
+  "author": {
+    "name": "regrevia",
+    "url": "https://github.com/regrevia"
+  },
   "repository": {
     "type": "git",
     "url": "git+https://github.com/regrevia/dsh-inner-browser-use.git"