dsh-inner-browser-use 0.11.44 → 0.18.0

package/README.md

@@ -1,119 +1,156 @@
 # dsh-inner-browser-use
 
-**Lets an agent drive the built-in sidebar browser of DeepSeek Harness — the one already in the right panel, not a new window.**
+A Chrome that the agent owns, mirrored into the right column of DeepSeek Harness.
 
-The agent navigates, reads the page as numbered controls, clicks, types, and runs JavaScript. You watch it happen: a pointer moves to each target and ripples where it lands. Everything runs inside the `<webview>` the product already leases, with the same partition and the same cookies, so a page you signed into by hand is the page the agent works on.
+The agent opens pages, reads them, clicks, types and scrolls. The browser is a separate
+Chrome with its own profile on disk, not the tab the product ships: the column shows a
+live picture of that browser, and your own cursor keeps working on the picture.
 
-> **On the name:** *inner* means the browser built into the Harness sidebar. This plugin opens no window of its own and never attaches to Chrome.
-
-## What you get
+## Tools
 
 | Tool | What it does |
 | --- | --- |
-| `browser_navigate` | Opens a URL in this chat's built-in browser and returns the page. |
-| `browser_view` | Returns the address, title, visible text, and every control as `#1 … #80`. |
-| `browser_click` | Clicks one numbered control and returns the page it leads to. |
-| `browser_type` | Types into one numbered field, optionally pressing Enter. |
-| `browser_script` | Evaluates JavaScript and returns the last expression. |
-| `browser_selection` | Returns the text currently selected in the page. |
-| `browser_copy` | Copies the selection, or one control's text, to the clipboard. |
-| `browser_screenshot` | Saves a PNG of the tab and returns its path. |
-| `browser_download` | Saves a file the page offers into this chat's project folder. A file the page starts saving on its own lands there too, and the call that caused it reports the path. |
-| `browser_upload` | Attaches a local file to a numbered file field. |
+| `sidecar_open` | Starts the browser (or reuses the running one) and opens an address. `headless: false` shows a real window on the desktop instead. |
+| `sidecar_view` | Returns the address, the title, every control as `1 … 120`, and the visible text. |
+| `sidecar_extract` | Returns the items a CSS selector matches, one compact line each with its link, instead of the whole page. |
+| `sidecar_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. A cookie banner or modal over the control is cleared first, then the click happens. |
+| `sidecar_type` | Types text into the focused field, or clicks a numbered control first. Chinese goes in as one string. `submit: true` presses Enter. |
+| `sidecar_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, ArrowUp/Down, Home, End, PageUp, PageDown. |
+| `sidecar_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
+| `sidecar_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
+| `sidecar_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
+
+`sidecar_view` answers with one page of text:
 
-Every call is visible: the pointer travels to the control, a ring marks it, and the page scrolls the target into view first.
+```
+https://example.com
+Example Domain
 
-The first call opens the tab for you: the page half hears that a chat needs one and opens **内置浏览器** in the right sidebar. That tab is the product's own — same tab kind, same address bar, same partition — and this plugin adds nothing to the guide pane. A tab you opened by hand is the tab the agent drives.
+1 [a] More information
+7 [input] Search
+```
 
-## Where files land
+Numbers stay valid until the next read. Anything that changes the page returns a fresh read.
 
-The tab reports the folder its chat works in when it connects, and everything this plugin writes goes under one
-folder of that working directory:
+## The column
 
-```
-.dsh-inner-browser-use/
-  .gitignore     keeps the folder out of your project's own version control
-  screenshots/   PNGs from browser_screenshot
-  downloads/     files the page offered to browser_download
-```
+The pane is a picture of the browser, sent as JPEG over the session's own stream: 1440×900,
+quality 72, every frame, about 82 KB and 25 per second on a page at rest.
 
-## Install
+Two pointers live on that picture:
 
-```bash
-git clone https://github.com/regrevia/dsh-inner-browser-use
-```
+- **The black arrow** is the agent. It marks where its last click landed, and it moves the
+  moment a new one is dispatched, without waiting for the page to repaint.
+- **Your cursor** is the real one. Hovering, clicking, the wheel and keystrokes are sent to
+  the browser as real mouse and keyboard input, so a site that rejects scripted clicks still
+  works. Moving your mouse never moves the black arrow.
+
+The address row above the picture is an address bar: type an address and press Enter, or use
+back, forward and reload. The field follows the page except while you are editing it.
+
+Typing arrives two ways: composed text (Chinese, emoji, anything the input method produces)
+lands as one insertion, and keys arrive as themselves, so Backspace deletes one character on
+the page.
+
+The pane opens only in the session that asked for it. A tool call in another session drives
+the same browser without taking the column that someone else is reading.
+
+## Logins
+
+The profile is a directory of its own: `~/.dsh/inner-browser-use/chrome/default`. It sits
+outside the product's browser storage, which is created per process and gone on restart, so a
+sign-in here is still there the next day.
 
-Then in DeepSeek Harness: **Plugins → Install from folder**, and pick the clone. Or from a shell:
+- **Plugin reload.** A reload adopts the Chrome that is still running, with the page it was on.
+- **Harness restart.** The same directory is opened again, so cookies, `localStorage` and
+  passwords that Chrome saved all survive.
+- **Shutdown.** The browser is asked to close rather than killed, which is when Chrome writes
+  the last of its cookies; an idle browser is closed after 15 minutes, and the clock stops
+  while a column is watching.
+
+Screenshots are the only thing written outside the profile:
+`~/.dsh-inner-browser-use/screenshots/`.
+
+## Measured
+
+| | |
+| --- | --- |
+| Navigate to first frame | ~850 ms, network included |
+| Frame | 1440×900 JPEG, ~82 KB |
+| Stream | ~25 frames per second |
+| Pointer move to arrow | 0.8–7 ms |
+| Read, dense page | 444 tokens (1510 chars, 90 controls) |
+| Read round trip | 1–4 ms |
+| Click round trip | 2 ms |
+| Screenshot | 77 ms, 113 KB |
+
+## Install
 
 ```bash
+git clone https://github.com/regrevia/dsh-inner-browser-use
 dsh plugin install /path/to/dsh-inner-browser-use
 ```
 
-Or install the packed release, no clone and no npm account:
+In the app: **Plugins → Install from folder**, and pick the clone. To install the packed
+tarball instead, build it from that directory and install the file:
 
 ```bash
-dsh plugin install https://github.com/regrevia/dsh-inner-browser-use/releases/download/v0.11.40/dsh-inner-browser-use-0.11.40.tgz
+npm pack                                  # writes dsh-inner-browser-use-0.16.0.tgz
+dsh plugin install ./dsh-inner-browser-use-0.16.0.tgz
 ```
 
-The plugin appears under **Plugins** as *内置浏览器*. It needs the Desktop app, because the sidebar browser is a Desktop capability.
+The plugin appears under **Plugins** as *Background browser*. It needs Google Chrome or
+Chromium on the machine; see [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
 
 ## Use
 
 1. Open the chat you want the agent to browse for.
-2. Ask the agent for something on a page: *"打开 example.com，告诉我标题"*. The first call opens the built-in browser in the right panel; pressing **+ → 内置浏览器** yourself lands on the same tab.
-
-The tab is per chat: each chat has its own in the same built-in browser. A call for a chat that never opened one opens it there.
-
-## Drag data both ways
-
-- **Page → chat:** `browser_script` returns strings and `browser_screenshot` saves a PNG, so the agent can read what the page shows and quote it — or hand you the image path — in the conversation.
-- **Chat → page:** `browser_type` puts text into a field; `browser_script` can set values the page cannot be typed into.
-- **Page → chat, by hand:** select text on the page and a **Drag to chat** handle appears in the tab's
-  status row. Drag it into the conversation to carry that text with you. After the agent downloads something
-  for this chat, the same handle carries that file, so a page's picture or PDF can be dropped into the chat
-  as an attachment without leaving the app. The agent gets the same content through `browser_selection`,
-  `browser_copy` and `browser_download`.
-
-## What it will not do
-
-- **It does not create a second browser.** No new window, no Chrome profile, no CDP mirror.
-- **A sign-in outlives a restart only when the page can read its cookie.** The guest partition lives in memory; the cookies this page can see are kept per workspace in `~/.dsh/inner-browser-use/cookies/` and written back when that workspace's pane opens again. A login held only in an `HttpOnly` cookie needs the password again.
-- **Screenshots are viewport-sized.** A `<webview>` captures what is visible, not the whole scroll height; scroll first, then capture.
-- **One partition per workspace.** Two chats in the same workspace share the browser's storage, exactly as two tabs of the built-in browser do.
-- **No popups or permission prompts.** A link that wants a new window stays in the tab it is in. A file the page
-  offers comes out through `browser_download`, which saves it into this chat's project folder and hands the path
-  back to the conversation. A file the page starts saving by itself needs no call: it lands in the same folder,
-  and the call that pressed the link reports where it went.
-- **It opens the tab only when a call needs it.** A call for the chat you are reading may open the right
-  panel; a call for any other chat gets a page out of sight instead, so the panel stays as you arranged it.
-  At most three such pages are kept, and the oldest goes first.
-- **A capture of a chat you are not looking at.** Reading, clicking and typing work on a tab that chat already
-  had, because they drive the page itself; a tab is opened only for the chat on screen. Pixels come from a surface the app is drawing, and a view
-  behind another tab of the same pane is not drawn: the capture tries to bring this chat's tab forward, and if the
-  page still has no surface it says so in a couple of seconds instead of waiting. Read what you need from the page
-  instead — `browser_script` returns the same content.
+2. Ask for something on a page: *"open example.com and tell me the title"*.
+
+The first call starts the browser and puts its mirror in this chat's right column. Nothing
+opens before a call: a chat that never asked for a page keeps its column as it was.
+
+## Frames, downloads, network
+
+- **Frames are read, including cross-site ones.** Controls inside an iframe are numbered like
+  the page's own, whether Chrome shares the page's process or runs the frame apart; a click in
+  either lands where the frame actually sits.
+- **A download lands on disk.** A click that saves a file writes it into the browser's own
+  profile folder, and `sidecar_downloads` names it, its size and when it arrived.
+- **The network is visible.** `sidecar_network` lists what the page asked for, newest first,
+  so a page that fills itself in from an address can be understood without a screenshot.
+- **An overlay does not eat the action.** A cookie banner over the button is cleared the way a
+  person would clear it, then the click lands — in the same call, and in every step of a
+  `sidecar_do` batch. Anything offering to delete, pay or sign out is reported instead.
+
+## What it does not do
+
+- **It does not touch your Chrome.** The browser runs from a separate profile with no
+  connection to the window you are using.
+- **The viewport is fixed.** 1440×900, desktop layout, no resizing and no mobile emulation.
+- **One page at a time.** A link that opens a tab is followed — the mirror lands on the new
+  page and the popup closes — so there is never a second page to choose between.
 
 ## Compatibility
 
 | | |
 | --- | --- |
-| Harness | Desktop app, sidebar browser present |
-| Tab kind | `browser` — the built-in browser's own kind |
-| Host | Node 20+ |
+| Harness | 0.2.0 or newer |
+| Chrome | Google Chrome, Chromium or Canary, 112 or newer |
+| Runtime | Node 20 or newer |
 | License | MIT |
 
-See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the wire and [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md) for version limits.
+Details in [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md); how the halves connect in
+[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 
-## Develop
+## Development
 
 ```bash
 node --test test/*.test.mjs
 ```
 
-The host half is plain ESM with no dependencies; the client half is a dynamic client module with no build step. Both are loaded by the app as-is.
-
-- **The page comes back on its own.** The pane is rebuilt from time to time; when that leaves the view blank, the
-  next tool call reopens the page that chat was last told to open, so a run is not interrupted by it.
+The server half is zero-dependency ESM, the page half is a client module loaded at runtime.
+Neither has a build step.
 
 ## License
 

package/README.zh.md

@@ -1,82 +1,127 @@
 # dsh-inner-browser-use
 
-**让 agent 直接操作 DeepSeek Harness 右侧栏里那个内置浏览器——不是再开一个窗口。**
+一个由 agent 自己养的 Chrome，实时投到 DeepSeek Harness 的右栏里。
 
-模型可以打开网址、把页面读成带编号的控件、点击、输入、执行脚本。你能看见它做每一步：指针移到目标上，落点有涟漪。页面就跑在产品自己租用的 `<webview>` 里，分区和 cookie 都没换，所以你手动登录过的那页，就是 agent 操作的那页。
+agent 打开网页、读页面、点、输入、滚动。这个浏览器是独立的 Chrome 进程，用自己的磁盘
+profile，不是产品自带的那个浏览页签：右栏放的是它的画面，而画面上的鼠标仍然是你自己的。
 
-> **关于名字：** *inner* 指 Harness 侧边栏里内置的浏览器。这个插件不另开窗口，也不接管 Chrome。
-
-## 提供的工具
+## 工具
 
 | 工具 | 作用 |
 | --- | --- |
-| `browser_navigate` | 在本文的内置浏览器里打开网址，返回页面。 |
-| `browser_view` | 返回地址、标题、可见文字，以及 `#1 … #80` 编号的控件。 |
-| `browser_click` | 点击一个编号控件，返回跳转后的页面。 |
-| `browser_type` | 往一个编号输入框里写字，可选回车。 |
-| `browser_script` | 执行 JavaScript，返回最后一个表达式。 |
-| `browser_selection` | 返回页面里选中的文字。 |
-| `browser_copy` | 把选中内容、或某个编号控件的文字，复制到剪贴板。 |
-| `browser_screenshot` | 存一张当前画面的 PNG，返回文件路径。 |
-| `browser_download` | 把网页给的文件存进本对话的项目目录。网页自己开始的下载也落在同一个目录，触发它的那次调用会报出路径。 |
-| `browser_upload` | 把本机文件塞进一个编号的文件框。 |
+| `sidecar_open` | 启动浏览器（已在跑就复用）并打开一个地址。传 `headless: false` 会改为在桌面上开真实窗口。 |
+| `sidecar_view` | 返回地址、标题、编号成 `1 … 120` 的控件，以及可见文字。 |
+| `sidecar_extract` | 按 CSS 选择器把列表项读成一行一条、带链接，比整页读省得多。 |
+| `sidecar_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。控件被横幅或弹层挡着时，先关掉它再点。 |
+| `sidecar_type` | 往焦点输入框输入文字，也可以先点一个编号控件。中文整段送进去。`submit: true` 回车提交。 |
+| `sidecar_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
+| `sidecar_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
+| `sidecar_shot` | 存一张原尺寸 PNG 到 `~/.dsh-inner-browser-use/screenshots`，返回路径。 |
+| `sidecar_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
+
+`sidecar_view` 一次给一页文本：
 
-每次调用都可见：指针先滑到控件上，套一个光圈，页面会先把目标滚进视野。
+```
+https://example.com
+Example Domain
 
-第一次调用会自己把标签页开出来：侧边栏里出现**内置浏览器**，不用先点 `+`。那是产品自己的标签页——同样的标签类型、同样的地址栏、同样的分区——插件只借用它，不往「开始」里加任何东西。你手动开的那页，就是 agent 操作的那页。
+1 [a] More information
+7 [input] Search
+```
 
-## 安装
+编号到下一次读之前都有效。任何改动页面的操作都会顺带返回一次新的读取。
 
-```bash
-git clone https://github.com/regrevia/dsh-inner-browser-use
-```
+## 右栏
+
+那一栏显示的是浏览器的画面，按会话自己的流发 JPEG：1440×900，质量 72，每帧都发；静止
+页面上约 82 KB 一帧、每秒 25 帧。
+
+画面上有两个指针：
+
+- **黑色箭头是 agent。** 它标在 agent 上一次点击的落点上，新的点击一发出箭头就动，不等
+  页面重绘。
+- **鼠标是你自己的。** 悬停、点击、滚轮、按键都当作真实输入交给浏览器，所以对拦截脚本点击
+  的站也照常管用。移动鼠标不会带动黑色箭头。
+
+画面上方那行地址就是地址栏：输入域名回车，页面换成它；旁边是后退、前进、刷新。你在编辑时
+它不会跟着页面跳。
+
+输入分两路进来：输入法合成的文字（中文、emoji 等）整段插入，按键则按原样送过去，所以
+Backspace 在页面上删掉的是一个字。
 
-然后在 DeepSeek Harness 里：**插件 → 从文件夹安装**，选中该目录。或命令行：
+右栏只在向它要过页面的那个会话里打开。别的会话调用工具时用的是同一个浏览器，但不会抢走
+别人正在看的这一栏。
+
+## 登录态
+
+profile 是独立目录：`~/.dsh/inner-browser-use/chrome/default`。它在产品自己的浏览数据之外
+——那份数据是每个进程临时建的，进程一退就没了，所以在这里登过的账号第二天还在。
+
+- **插件重载。** 重载会接上仍在运行的那个 Chrome，页面也还在原处。
+- **Harness 重启。** 重新打开同一个目录，cookie、`localStorage` 和 Chrome 存下的密码都留着。
+- **退出。** 关浏览器是请它自己关（`Browser.close`），Chrome 在这一步才把 cookie 落盘；空闲
+  15 分钟会关掉，有右栏在看时不计这段空闲。
+
+profile 之外只写一样东西：`~/.dsh-inner-browser-use/screenshots/` 里的截图。
+
+## 实测
+
+| | |
+| --- | --- |
+| 打开页面到第一帧 | 约 850 ms，含网络加载 |
+| 单帧 | 1440×900 JPEG，约 82 KB |
+| 画面 | 约 25 帧每秒 |
+| 指针移动到箭头跟上 | 0.8–7 ms |
+
+## 安装
 
 ```bash
+git clone https://github.com/regrevia/dsh-inner-browser-use
 dsh plugin install /path/to/dsh-inner-browser-use
 ```
 
-也可以直接装打包好的发布版，不用 clone、不用 npm 账号：
+在应用里走 **插件 → 从文件夹安装**，选这个仓库目录。要装打包好的 tgz，就在该目录里打一个再装：
 
 ```bash
-dsh plugin install https://github.com/regrevia/dsh-inner-browser-use/releases/download/v0.11.40/dsh-inner-browser-use-0.11.40.tgz
+npm pack                                  # 得到 dsh-inner-browser-use-0.12.1.tgz
+dsh plugin install ./dsh-inner-browser-use-0.12.1.tgz
 ```
 
-装好后**插件**列表里出现 *内置浏览器*。需要桌面版，因为侧边栏浏览器是桌面能力。
+装好后在 **插件** 里显示为 *后台浏览器*。机器上要有 Google Chrome 或 Chromium，细节见
+[docs/COMPATIBILITY.md](docs/COMPATIBILITY.md)。
 
 ## 使用
 
-1. 打开你想让它浏览的那个会话。
-2. 直接提要求：*「打开 example.com，告诉我标题」*。第一次调用会把内置浏览器在右侧栏开出来；自己按 **+ → 内置浏览器** 也是同一个标签页。
+1. 打开你想让 agent 代跑的会话。
+2. 直接说要页面上的什么事：*「打开 example.com，告诉我标题」*。
 
-标签属于会话：每个会话在同一套内置浏览器里各有一页。会话里还没开过时，工具会替你在那里开。
+第一次调用会启动浏览器，并把镜像放进这个会话的右栏。没有调用就不会有东西自己打开：没要过
+页面的会话，右栏保持原样。
 
-## 数据双向流动
+## 框架、下载、网络
 
-- **页面 → 聊天：** `browser_script` 返回字符串，`browser_screenshot` 存成 PNG，模型可以把页面内容写进对话，也可以把图片路径交给你。
-- **选中即用：** `browser_selection` 把你在页面里选中的文字交到对话里；`browser_copy` 把它压进剪贴板，⌘V 就粘进输入框。
-- **聊天 → 页面：** `browser_type` 写进输入框；`browser_script` 可以设置那些没法输入的字段。
+- **iframe 能读，跨站也能。** 不管是和页面共用进程，还是 Chrome 单独给一个进程，里面的控件都和页面自己的控件一样有编号；点击按框架在页面上的实际位置落点。
+- **下载会落到磁盘。** 点一下就下载的文件写在这个浏览器自己的 profile 目录里，`sidecar_downloads` 报出文件名、大小和到达时间。
+- **能看到网络。** `sidecar_network` 倒序列出页面请求了什么，自己用接口填内容、屏幕上又看不出原因的页面，不用截图去猜。
+- **遮罩不吞动作。** 盖在按钮上的 cookie 横幅会按人的做法先关掉再点下去，同一次调用里完成；`sidecar_do` 的每一步都一样。写着删除、支付、退出的遮罩只报告，不代按。
 
-## 它不做什么
+## 不做什么
 
-- **不另开浏览器。** 没有新窗口，没有 Chrome 配置目录，没有 CDP 镜像。
-- **只有页面能读到的 cookie 才活过重启。** 访客分区在内存里，页面能读到的 cookie 按工作区存到 `~/.dsh/inner-browser-use/cookies/`，该工作区的侧边栏再次打开时写回；只存在 `HttpOnly` cookie 里的登录要重新输密码。
-- **截图只有可视区域。** `<webview>` 抓到的是看得见的那块，不是整页；要拍长页面先滚动。
-- **一个工作区一个分区。** 同一工作区的两个会话共用浏览器存储，和内置浏览器开两个标签时一样。
-- **没有新窗口、没有权限弹窗。** 想在新的窗口打开的链接留在当前页里。
-- **没有标签时会替你开。** 会话里还没有内置浏览器时，第一次调用就在那里开一页；你自己开的那页同样会被接管。
+- **不动你自己的 Chrome。** 浏览器跑在单独的 profile 上，和你正在用的窗口没有关系。
+- **视口固定。** 1440×900、桌面布局，不缩放，也不模拟手机。
+- **一次一页。** 点开新页签的链接会被跟过去：镜像落到新页面，弹窗随即关掉，不会留下第二个页面要选。
 
-## 兼容
+## 兼容性
 
 | | |
 | --- | --- |
-| Harness | 桌面版，且侧边栏浏览器存在 |
-| 标签 kind | `browser`，就是内置浏览器自己那个 |
-| 运行环境 | Node 20+ |
+| Harness | 0.2.0 及以上 |
+| Chrome | Google Chrome、Chromium 或 Canary，112 及以上 |
+| 运行时 | Node 20 及以上 |
 | 许可 | MIT |
 
-链路细节见 [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)，版本边界见 [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md)。
+细节在 [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md)，两半怎么连起来在
+[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)。
 
 ## 开发
 
@@ -84,7 +129,7 @@ dsh plugin install https://github.com/regrevia/dsh-inner-browser-use/releases/do
 node --test test/*.test.mjs
 ```
 
-宿主半边是零依赖 ESM，客户端半边是动态客户端模块，都没有构建步骤，应用直接加载。
+服务端那一半是零依赖 ESM，页面那一半是运行时加载的客户端模块，都不需要构建。
 
 ## 许可