dsh-inner-browser-use 0.27.0 → 0.34.95

package/README.md

@@ -10,17 +10,22 @@ live picture of that browser, and your own cursor keeps working on the picture.
 
 | Tool | What it does |
 | --- | --- |
-| `sidecar_open` | Starts the browser (or reuses the running one) and opens an address. `headless: false` shows a real window on the desktop instead. |
-| `sidecar_view` | Returns the address, the title, every control as `1 … 120`, and the visible text. |
-| `sidecar_extract` | Returns the items a CSS selector matches, one compact line each with its link, instead of the whole page. |
-| `sidecar_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. A cookie banner or modal over the control is cleared first, then the click happens. |
-| `sidecar_type` | Types text into the focused field, or clicks a numbered control first. Chinese goes in as one string. `submit: true` presses Enter. |
-| `sidecar_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, ArrowUp/Down, Home, End, PageUp, PageDown. |
-| `sidecar_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
-| `sidecar_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
-| `sidecar_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
-
-`sidecar_view` answers with one page of text:
+| `browser_open` | Starts the browser (or reuses the running one) and opens an address. `headless: false` shows a real window on the desktop instead. |
+| `browser_view` | Returns the address, the title, every control as `1 … 120`, and the visible text. |
+| `browser_read` | Reads the page the way a person does: one Markdown block of headings, paragraphs, lists and links, no controls. Far fewer tokens than a snapshot on long pages, and the links stay usable. Give it a `query` and only the passages that answer it come back, ranked (BM25 over Readability-style blocks, link-heavy menus down-weighted). |
+| `browser_find` | Finds words (or a regex) anywhere on the page: matching controls come back with a clickable number, matching text with its point, each saying whether it is on screen. |
+| `browser_extract` | Returns the items a CSS selector matches, one compact line each with its link, instead of the whole page. |
+| `browser_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. A cookie banner or modal over the control is cleared first, then the click happens. |
+| `browser_type` | Types text into the focused field, or clicks a numbered control first. Chinese goes in as one string. `submit: true` presses Enter. |
+| `browser_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, Insert, arrows, Home, End, PageUp, PageDown, F1-F12, the numeric keypad, letters, digits, punctuation and every shifted symbol, plus ContextMenu, CapsLock, PrintScreen and media keys. `Ctrl`/`Alt`/`Shift`/`Meta`/`Mod` modifiers, chords with `+`, and `Ctrl+C/X/V/A/Z/Y` run the real editing commands. |
+| `browser_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
+| `browser_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
+| `browser_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
+
+`browser_read` on the same page is the words only, in Markdown (measured: 4,520
+characters of an about page against 2,987 characters of snapshot — and the
+snapshot spends most of its budget on controls; on a long article the gap grows
+with the number of controls skipped). `browser_view` answers with one page of text:
 
 ```
 https://example.com
@@ -34,8 +39,9 @@ Numbers stay valid until the next read. Anything that changes the page returns a
 
 ## The column
 
-The pane is a picture of the browser, sent as JPEG over the session's own stream: 1440×900,
-quality 72, every frame, about 82 KB and 25 per second on a page at rest.
+The pane is a picture of the browser, sent as JPEG over the session's own stream: quality 72,
+every frame, about 82 KB and 25 per second on a page at rest. The page is laid out at the size
+of the column, so the picture fills the frame instead of sitting letterboxed inside it.
 
 Two pointers live on that picture:
 
@@ -82,7 +88,9 @@ Screenshots are the only thing written outside the profile:
 | Read, dense page | 444 tokens (1510 chars, 90 controls) |
 | Read round trip | 1–4 ms |
 | Click round trip | 2 ms |
-| Screenshot | 77 ms, 113 KB |
+| Key round trip | 7 ms |
+| Key to next frame | 15–18 ms |
+| Screenshot | 75 ms, 115 KB |
 
 ## Install
 
@@ -102,6 +110,22 @@ dsh plugin install ./dsh-inner-browser-use-0.16.0.tgz
 The plugin appears under **Plugins** as *Background browser*. It needs Google Chrome or
 Chromium on the machine; see [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
 
+## When it opens
+
+Only when the person asks for it by name: "in the browser", "on the web page",
+"open this site", "click it for me". A link on its own is not a request.
+
+- A public article, one link's text, docs, a GitHub repo or PR: read it with
+  `web_fetch` or Envoy. No browser, no Chrome process.
+- If the page needs a login, is an empty shell to `web_fetch`, or needs a click,
+  say why and ask once instead of starting a browser on your own.
+- Asked for, it starts without a question and never opens a second, external
+  Chrome. The picture stays in this chat's right column and stays open when the
+  work is done.
+
+One browser is shared by every chat (see [Sharing](#sharing)), so opening one is
+a tab, not a second Chrome.
+
 ## Use
 
 1. Open the chat you want the agent to browse for.
@@ -110,24 +134,35 @@ Chromium on the machine; see [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
 The first call starts the browser and puts its mirror in this chat's right column. Nothing
 opens before a call: a chat that never asked for a page keeps its column as it was.
 
+## Sharing
+
+Every chat uses the same background Chrome: one process, one tab per chat, and
+each tab's address saved so it can be closed and reopened cheaply. Past six
+chats the oldest tab nobody is watching is closed (a watched tab — the one on
+someone's screen — never is). Tabs an earlier plugin load left behind are closed
+on first use: one profile used for weeks otherwise keeps a renderer per chat that
+ever opened it. Nothing is launched per chat, and the browser itself is closed 15
+minutes after the last mirror is detached.
+
 ## Frames, downloads, network
 
 - **Frames are read, including cross-site ones.** Controls inside an iframe are numbered like
   the page's own, whether Chrome shares the page's process or runs the frame apart; a click in
   either lands where the frame actually sits.
 - **A download lands on disk.** A click that saves a file writes it into the browser's own
-  profile folder, and `sidecar_downloads` names it, its size and when it arrived.
-- **The network is visible.** `sidecar_network` lists what the page asked for, newest first,
+  profile folder, and `browser_downloads` names it, its size and when it arrived.
+- **The network is visible.** `browser_network` lists what the page asked for, newest first,
   so a page that fills itself in from an address can be understood without a screenshot.
 - **An overlay does not eat the action.** A cookie banner over the button is cleared the way a
   person would clear it, then the click lands — in the same call, and in every step of a
-  `sidecar_do` batch. Anything offering to delete, pay or sign out is reported instead.
+  `browser_do` batch. Anything offering to delete, pay or sign out is reported instead.
 
 ## What it does not do
 
 - **It does not touch your Chrome.** The browser runs from a separate profile with no
   connection to the window you are using.
-- **The viewport is fixed.** 1440×900, desktop layout, no resizing and no mobile emulation.
+- **The layout viewport is the column, nothing more.** Desktop layout, no mobile emulation, no
+touch emulation; a very narrow column is laid out narrow rather than zoomed out.
 - **One page at a time.** A link that opens a tab is followed — the mirror lands on the new
   page and the popup closes — so there is never a second page to choose between.
 

package/README.zh.md

@@ -9,17 +9,21 @@ profile，不是产品自带的那个浏览页签：右栏放的是它的画面
 
 | 工具 | 作用 |
 | --- | --- |
-| `sidecar_open` | 启动浏览器（已在跑就复用）并打开一个地址。传 `headless: false` 会改为在桌面上开真实窗口。 |
-| `sidecar_view` | 返回地址、标题、编号成 `1 … 120` 的控件，以及可见文字。 |
-| `sidecar_extract` | 按 CSS 选择器把列表项读成一行一条、带链接，比整页读省得多。 |
-| `sidecar_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。控件被横幅或弹层挡着时，先关掉它再点。 |
-| `sidecar_type` | 往焦点输入框输入文字，也可以先点一个编号控件。中文整段送进去。`submit: true` 回车提交。 |
-| `sidecar_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
-| `sidecar_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
-| `sidecar_shot` | 存一张原尺寸 PNG 到 `~/.dsh-inner-browser-use/screenshots`，返回路径。 |
-| `sidecar_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
-
-`sidecar_view` 一次给一页文本：
+| `browser_open` | 启动浏览器（已在跑就复用）并打开一个地址。传 `headless: false` 会改为在桌面上开真实窗口。 |
+| `browser_view` | 返回地址、标题、编号成 `1 … 120` 的控件，以及可见文字。 |
+| `browser_read` | 按人读文章的方式给 Markdown（标题、段落、列表、链接）。带 `query` 只回能回答的段落；问不到时自动改为整页，不再空手回来。页面和上次读的没变就只回一行，不重发正文（`force: true` 强制重发）。 |
+| `browser_extract` | 按 CSS 选择器把列表项读成一行一条、带链接，比整页读省得多。 |
+| `browser_view` 的 `map` | 大页面先看地标：各类控件各多少个、去重后的控件和编号、正文多少字，但不带回正文。 |
+| `browser_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。控件被横幅或弹层挡着时，先关掉它再点。 |
+| `browser_type` | 往焦点输入框输入文字，也可以先点一个编号控件。中文整段送进去。`submit: true` 回车提交。 |
+| `browser_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
+| `browser_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
+| `browser_shot` | 存一张原尺寸 PNG 到 `~/.dsh-inner-browser-use/screenshots`，返回路径。 |
+| `browser_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
+
+点、输入、按键之后的那一行会多一句结果：`· navigated to …`（换页了）、`· content changed`（页内有变化）、`· no visible change`（点了没动）、Flash 页则是 `· screen updated` / `· no frame change`；`read: true` 时不再多说。
+
+`browser_view` 一次给一页文本：
 
 ```
 https://example.com
@@ -72,6 +76,7 @@ profile 之外只写一样东西：`~/.dsh-inner-browser-use/screenshots/` 里
 | 单帧 | 1440×900 JPEG，约 82 KB |
 | 画面 | 约 25 帧每秒 |
 | 指针移动到箭头跟上 | 0.8–7 ms |
+| 右栏里手动点击到画面更新 | 中位 13 ms，最坏 17 ms（点击派发 2 ms、悬停 8 ms） |
 
 ## 安装
 
@@ -98,12 +103,19 @@ dsh plugin install ./dsh-inner-browser-use-0.12.1.tgz
 第一次调用会启动浏览器，并把镜像放进这个会话的右栏。没有调用就不会有东西自己打开：没要过
 页面的会话，右栏保持原样。
 
+## Flash 页面
+
+`browser_open` 带 `flash: true` 时，这个地址改用插件自带的 Flash 运行时（Electron 87 + Pepper Flash）在**离屏窗口**里播放：窗口一次都不显示，Dock 与菜单栏都不出现，画面只在发起会话的右栏里。切换回来再调一次 `flash: false`，选择会记住。运行时首次使用时下载到 `~/.dsh/inner-browser-use/flash/`，不需要单独安装播放器。
+
+`npm run flash-hidden` 会真起一次运行时并断言「窗口未显示 / 帧持续产出 / 两帧不同」；`npm run flash-mirror` 量镜像帧率与按下/抬起/按键耗时；`npm run flash-tools` 量工具层成本（奥拉星登录页实测：点击 28 ms、输入 14 ms、`browser_view` 31 ms；同一页后续回话只说标题，点击 22 字符、输入 13 字符）；`npm run flash-local` 用仓库自带的 SWF 在本机起页，不依赖网络，量首帧与输入直达。
+
 ## 框架、下载、网络
 
 - **iframe 能读，跨站也能。** 不管是和页面共用进程，还是 Chrome 单独给一个进程，里面的控件都和页面自己的控件一样有编号；点击按框架在页面上的实际位置落点。
-- **下载会落到磁盘。** 点一下就下载的文件写在这个浏览器自己的 profile 目录里，`sidecar_downloads` 报出文件名、大小和到达时间。
-- **能看到网络。** `sidecar_network` 倒序列出页面请求了什么，自己用接口填内容、屏幕上又看不出原因的页面，不用截图去猜。
-- **遮罩不吞动作。** 盖在按钮上的 cookie 横幅会按人的做法先关掉再点下去，同一次调用里完成；`sidecar_do` 的每一步都一样。写着删除、支付、退出的遮罩只报告，不代按。
+- **下载会落到磁盘。** 点一下就下载的文件写在这个浏览器自己的 profile 目录里，`browser_downloads` 报出文件名、大小和到达时间。
+- **PDF 有去处。** Chrome 无头进程画不出 PDF，右栏只会是白图；读到 PDF 时把文件取回归档到该浏览器 profile 的 `downloads/`，结果里直接给绝对路径，交给 `envoy_read` 或本地阅读器。
+- **能看到网络。** `browser_network` 倒序列出页面请求了什么，自己用接口填内容、屏幕上又看不出原因的页面，不用截图去猜。
+- **遮罩不吞动作。** 盖在按钮上的 cookie 横幅会按人的做法先关掉再点下去，同一次调用里完成；`browser_do` 的每一步都一样。写着删除、支付、退出的遮罩只报告，不代按。
 
 ## 不做什么