dsh-inner-browser-use 0.34.28 → 0.34.95

package/README.md

@@ -10,17 +10,22 @@ live picture of that browser, and your own cursor keeps working on the picture.
 
 | Tool | What it does |
 | --- | --- |
-| `sidecar_open` | Starts the browser (or reuses the running one) and opens an address. `headless: false` shows a real window on the desktop instead. |
-| `sidecar_view` | Returns the address, the title, every control as `1 … 120`, and the visible text. |
-| `sidecar_extract` | Returns the items a CSS selector matches, one compact line each with its link, instead of the whole page. |
-| `sidecar_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. A cookie banner or modal over the control is cleared first, then the click happens. |
-| `sidecar_type` | Types text into the focused field, or clicks a numbered control first. Chinese goes in as one string. `submit: true` presses Enter. |
-| `sidecar_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, ArrowUp/Down, Home, End, PageUp, PageDown. |
-| `sidecar_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
-| `sidecar_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
-| `sidecar_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
-
-`sidecar_view` answers with one page of text:
+| `browser_open` | Starts the browser (or reuses the running one) and opens an address. `headless: false` shows a real window on the desktop instead. |
+| `browser_view` | Returns the address, the title, every control as `1 … 120`, and the visible text. |
+| `browser_read` | Reads the page the way a person does: one Markdown block of headings, paragraphs, lists and links, no controls. Far fewer tokens than a snapshot on long pages, and the links stay usable. Give it a `query` and only the passages that answer it come back, ranked (BM25 over Readability-style blocks, link-heavy menus down-weighted). |
+| `browser_find` | Finds words (or a regex) anywhere on the page: matching controls come back with a clickable number, matching text with its point, each saying whether it is on screen. |
+| `browser_extract` | Returns the items a CSS selector matches, one compact line each with its link, instead of the whole page. |
+| `browser_click` | Clicks a numbered control, or an `x`/`y` point. `double: true` double-clicks. A cookie banner or modal over the control is cleared first, then the click happens. |
+| `browser_type` | Types text into the focused field, or clicks a numbered control first. Chinese goes in as one string. `submit: true` presses Enter. |
+| `browser_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, Insert, arrows, Home, End, PageUp, PageDown, F1-F12, the numeric keypad, letters, digits, punctuation and every shifted symbol, plus ContextMenu, CapsLock, PrintScreen and media keys. `Ctrl`/`Alt`/`Shift`/`Meta`/`Mod` modifiers, chords with `+`, and `Ctrl+C/X/V/A/Z/Y` run the real editing commands. |
+| `browser_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
+| `browser_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
+| `browser_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
+
+`browser_read` on the same page is the words only, in Markdown (measured: 4,520
+characters of an about page against 2,987 characters of snapshot — and the
+snapshot spends most of its budget on controls; on a long article the gap grows
+with the number of controls skipped). `browser_view` answers with one page of text:
 
 ```
 https://example.com
@@ -34,8 +39,9 @@ Numbers stay valid until the next read. Anything that changes the page returns a
 
 ## The column
 
-The pane is a picture of the browser, sent as JPEG over the session's own stream: 1440×900,
-quality 72, every frame, about 82 KB and 25 per second on a page at rest.
+The pane is a picture of the browser, sent as JPEG over the session's own stream: quality 72,
+every frame, about 82 KB and 25 per second on a page at rest. The page is laid out at the size
+of the column, so the picture fills the frame instead of sitting letterboxed inside it.
 
 Two pointers live on that picture:
 
@@ -82,7 +88,9 @@ Screenshots are the only thing written outside the profile:
 | Read, dense page | 444 tokens (1510 chars, 90 controls) |
 | Read round trip | 1–4 ms |
 | Click round trip | 2 ms |
-| Screenshot | 77 ms, 113 KB |
+| Key round trip | 7 ms |
+| Key to next frame | 15–18 ms |
+| Screenshot | 75 ms, 115 KB |
 
 ## Install
 
@@ -102,6 +110,22 @@ dsh plugin install ./dsh-inner-browser-use-0.16.0.tgz
 The plugin appears under **Plugins** as *Background browser*. It needs Google Chrome or
 Chromium on the machine; see [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
 
+## When it opens
+
+Only when the person asks for it by name: "in the browser", "on the web page",
+"open this site", "click it for me". A link on its own is not a request.
+
+- A public article, one link's text, docs, a GitHub repo or PR: read it with
+  `web_fetch` or Envoy. No browser, no Chrome process.
+- If the page needs a login, is an empty shell to `web_fetch`, or needs a click,
+  say why and ask once instead of starting a browser on your own.
+- Asked for, it starts without a question and never opens a second, external
+  Chrome. The picture stays in this chat's right column and stays open when the
+  work is done.
+
+One browser is shared by every chat (see [Sharing](#sharing)), so opening one is
+a tab, not a second Chrome.
+
 ## Use
 
 1. Open the chat you want the agent to browse for.
@@ -110,24 +134,35 @@ Chromium on the machine; see [docs/COMPATIBILITY.md](docs/COMPATIBILITY.md).
 The first call starts the browser and puts its mirror in this chat's right column. Nothing
 opens before a call: a chat that never asked for a page keeps its column as it was.
 
+## Sharing
+
+Every chat uses the same background Chrome: one process, one tab per chat, and
+each tab's address saved so it can be closed and reopened cheaply. Past six
+chats the oldest tab nobody is watching is closed (a watched tab — the one on
+someone's screen — never is). Tabs an earlier plugin load left behind are closed
+on first use: one profile used for weeks otherwise keeps a renderer per chat that
+ever opened it. Nothing is launched per chat, and the browser itself is closed 15
+minutes after the last mirror is detached.
+
 ## Frames, downloads, network
 
 - **Frames are read, including cross-site ones.** Controls inside an iframe are numbered like
   the page's own, whether Chrome shares the page's process or runs the frame apart; a click in
   either lands where the frame actually sits.
 - **A download lands on disk.** A click that saves a file writes it into the browser's own
-  profile folder, and `sidecar_downloads` names it, its size and when it arrived.
-- **The network is visible.** `sidecar_network` lists what the page asked for, newest first,
+  profile folder, and `browser_downloads` names it, its size and when it arrived.
+- **The network is visible.** `browser_network` lists what the page asked for, newest first,
   so a page that fills itself in from an address can be understood without a screenshot.
 - **An overlay does not eat the action.** A cookie banner over the button is cleared the way a
   person would clear it, then the click lands — in the same call, and in every step of a
-  `sidecar_do` batch. Anything offering to delete, pay or sign out is reported instead.
+  `browser_do` batch. Anything offering to delete, pay or sign out is reported instead.
 
 ## What it does not do
 
 - **It does not touch your Chrome.** The browser runs from a separate profile with no
   connection to the window you are using.
-- **The viewport is fixed.** 1440×900, desktop layout, no resizing and no mobile emulation.
+- **The layout viewport is the column, nothing more.** Desktop layout, no mobile emulation, no
+touch emulation; a very narrow column is laid out narrow rather than zoomed out.
 - **One page at a time.** A link that opens a tab is followed — the mirror lands on the new
   page and the popup closes — so there is never a second page to choose between.
 

package/README.zh.md

@@ -9,17 +9,21 @@ profile，不是产品自带的那个浏览页签：右栏放的是它的画面
 
 | 工具 | 作用 |
 | --- | --- |
-| `sidecar_open` | 启动浏览器（已在跑就复用）并打开一个地址。传 `headless: false` 会改为在桌面上开真实窗口。 |
-| `sidecar_view` | 返回地址、标题、编号成 `1 … 120` 的控件，以及可见文字。 |
-| `sidecar_extract` | 按 CSS 选择器把列表项读成一行一条、带链接，比整页读省得多。 |
-| `sidecar_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。控件被横幅或弹层挡着时，先关掉它再点。 |
-| `sidecar_type` | 往焦点输入框输入文字，也可以先点一个编号控件。中文整段送进去。`submit: true` 回车提交。 |
-| `sidecar_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
-| `sidecar_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
-| `sidecar_shot` | 存一张原尺寸 PNG 到 `~/.dsh-inner-browser-use/screenshots`，返回路径。 |
-| `sidecar_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
-
-`sidecar_view` 一次给一页文本：
+| `browser_open` | 启动浏览器（已在跑就复用）并打开一个地址。传 `headless: false` 会改为在桌面上开真实窗口。 |
+| `browser_view` | 返回地址、标题、编号成 `1 … 120` 的控件，以及可见文字。 |
+| `browser_read` | 按人读文章的方式给 Markdown（标题、段落、列表、链接）。带 `query` 只回能回答的段落；问不到时自动改为整页，不再空手回来。页面和上次读的没变就只回一行，不重发正文（`force: true` 强制重发）。 |
+| `browser_extract` | 按 CSS 选择器把列表项读成一行一条、带链接，比整页读省得多。 |
+| `browser_view` 的 `map` | 大页面先看地标：各类控件各多少个、去重后的控件和编号、正文多少字，但不带回正文。 |
+| `browser_click` | 点编号控件，或点 `x`/`y` 坐标。`double: true` 双击。控件被横幅或弹层挡着时，先关掉它再点。 |
+| `browser_type` | 往焦点输入框输入文字，也可以先点一个编号控件。中文整段送进去。`submit: true` 回车提交。 |
+| `browser_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
+| `browser_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
+| `browser_shot` | 存一张原尺寸 PNG 到 `~/.dsh-inner-browser-use/screenshots`，返回路径。 |
+| `browser_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
+
+点、输入、按键之后的那一行会多一句结果：`· navigated to …`（换页了）、`· content changed`（页内有变化）、`· no visible change`（点了没动）、Flash 页则是 `· screen updated` / `· no frame change`；`read: true` 时不再多说。
+
+`browser_view` 一次给一页文本：
 
 ```
 https://example.com
@@ -72,6 +76,7 @@ profile 之外只写一样东西：`~/.dsh-inner-browser-use/screenshots/` 里
 | 单帧 | 1440×900 JPEG，约 82 KB |
 | 画面 | 约 25 帧每秒 |
 | 指针移动到箭头跟上 | 0.8–7 ms |
+| 右栏里手动点击到画面更新 | 中位 13 ms，最坏 17 ms（点击派发 2 ms、悬停 8 ms） |
 
 ## 安装
 
@@ -98,12 +103,19 @@ dsh plugin install ./dsh-inner-browser-use-0.12.1.tgz
 第一次调用会启动浏览器，并把镜像放进这个会话的右栏。没有调用就不会有东西自己打开：没要过
 页面的会话，右栏保持原样。
 
+## Flash 页面
+
+`browser_open` 带 `flash: true` 时，这个地址改用插件自带的 Flash 运行时（Electron 87 + Pepper Flash）在**离屏窗口**里播放：窗口一次都不显示，Dock 与菜单栏都不出现，画面只在发起会话的右栏里。切换回来再调一次 `flash: false`，选择会记住。运行时首次使用时下载到 `~/.dsh/inner-browser-use/flash/`，不需要单独安装播放器。
+
+`npm run flash-hidden` 会真起一次运行时并断言「窗口未显示 / 帧持续产出 / 两帧不同」；`npm run flash-mirror` 量镜像帧率与按下/抬起/按键耗时；`npm run flash-tools` 量工具层成本（奥拉星登录页实测：点击 28 ms、输入 14 ms、`browser_view` 31 ms；同一页后续回话只说标题，点击 22 字符、输入 13 字符）；`npm run flash-local` 用仓库自带的 SWF 在本机起页，不依赖网络，量首帧与输入直达。
+
 ## 框架、下载、网络
 
 - **iframe 能读，跨站也能。** 不管是和页面共用进程，还是 Chrome 单独给一个进程，里面的控件都和页面自己的控件一样有编号；点击按框架在页面上的实际位置落点。
-- **下载会落到磁盘。** 点一下就下载的文件写在这个浏览器自己的 profile 目录里，`sidecar_downloads` 报出文件名、大小和到达时间。
-- **能看到网络。** `sidecar_network` 倒序列出页面请求了什么，自己用接口填内容、屏幕上又看不出原因的页面，不用截图去猜。
-- **遮罩不吞动作。** 盖在按钮上的 cookie 横幅会按人的做法先关掉再点下去，同一次调用里完成；`sidecar_do` 的每一步都一样。写着删除、支付、退出的遮罩只报告，不代按。
+- **下载会落到磁盘。** 点一下就下载的文件写在这个浏览器自己的 profile 目录里，`browser_downloads` 报出文件名、大小和到达时间。
+- **PDF 有去处。** Chrome 无头进程画不出 PDF，右栏只会是白图；读到 PDF 时把文件取回归档到该浏览器 profile 的 `downloads/`，结果里直接给绝对路径，交给 `envoy_read` 或本地阅读器。
+- **能看到网络。** `browser_network` 倒序列出页面请求了什么，自己用接口填内容、屏幕上又看不出原因的页面，不用截图去猜。
+- **遮罩不吞动作。** 盖在按钮上的 cookie 横幅会按人的做法先关掉再点下去，同一次调用里完成；`browser_do` 的每一步都一样。写着删除、支付、退出的遮罩只报告，不代按。
 
 ## 不做什么
 

package/client.js

@@ -32,7 +32,7 @@ window.__ModuleLoader__.load({
     const DICT = {
       zh: {
         'side.label': '后台浏览器',
-        'side.empty': '还没有打开的页面，在上方地址栏输入网址',
+        'side.empty': '后台的独立 Chrome 还没有页面，在上方地址栏输入网址',
         'side.restore': '恢复上次的页面',
         'side.connecting': '正在连接…',
         'side.failed': '画面没有连上',
@@ -45,7 +45,7 @@ window.__ModuleLoader__.load({
       },
       en: {
         'side.label': 'Background browser',
-        'side.empty': 'No page open; type an address above',
+        'side.empty': 'The background Chrome has no page yet; type an address above',
         'side.restore': 'Reopen the last page',
         'side.connecting': 'Connecting…',
         'side.failed': 'The picture did not connect',
@@ -167,6 +167,12 @@ window.__ModuleLoader__.load({
             single: true,
             priority: 'extension',
             title: () => L('side.label'),
+            /* The tab strip's close control is not the pane's own button: a
+               tab closed there never told the host, the ask stayed alive, and
+               the keepalive put the mirror back a few seconds later — a tab
+               the person closed that would not stay closed. Every close path
+               ends here. */
+            onClose: (_tab, where) => closeHere(String(where?.sessionId ?? mountedOf())),
           })))
           disposers.push(scope.slots.inject(SLOT, () => scope.slots.register({
             name: SLOT, // ui-copy-ignore
@@ -227,7 +233,10 @@ window.__ModuleLoader__.load({
           const watchAsk = () => {
             try {
               if (typeof surface?.mounted?.subscribe === 'function') {
-                const off = surface.mounted.subscribe(syncAsk)
+                const off = surface.mounted.subscribe(() => {
+                  syncAsk()
+                  restore()
+                })
                 if (typeof off === 'function') return off
               }
             } catch {
@@ -236,10 +245,41 @@ window.__ModuleLoader__.load({
             return null
           }
           syncAsk()
+          /* An app restart leaves the browser alive but no ask on the wire, so
+             the column would come back empty in a chat that never closed it.
+             This chat asking for the state of the browser it still owns is what
+             puts the mirror back without anyone pressing anything. */
+          const restoreTimers = []
+          const restore = (tries = 0) => {
+            const who = mountedOf()
+            if (who === '') return
+            const again = () => {
+              if (tries >= 2) return
+              restoreTimers.push(setTimeout(() => restore(tries + 1), 1500))
+            }
+            try {
+              fetch(`${ROUTE}/sidecar/state?session=${encodeURIComponent(who)}`) // ui-copy-ignore
+                .then((said) => (said.ok === true ? said.json() : null))
+                .then((state) => {
+                  if (state?.owner !== who) return
+                  const live = state?.alive === true || state?.chat?.alive === true
+                  if (live !== true) {
+                    again()
+                    return
+                  }
+                  ask(who)
+                })
+                .catch(again)
+            } catch {
+              again()
+            }
+          }
+          restore()
           const offAsk = watchAsk()
           const pollAsk = setInterval(syncAsk, 1000)
           const stopAsk = () => {
             clearInterval(pollAsk)
+            for (const timer of restoreTimers) clearTimeout(timer)
             if (typeof offAsk === 'function') {
               try {
                 offAsk()
@@ -274,6 +314,32 @@ window.__ModuleLoader__.load({
     function SidecarBody(props) {
       const [frame, setFrame] = React.useState(null)
       const [ptr, setPtr] = React.useState(null)
+      /* The page is laid out at the size of this column, so the picture it
+         sends already has the right shape; these are its pixels. */
+      const [view, setView] = React.useState({ w: SIDE_VIEW.width, h: SIDE_VIEW.height })
+      /* The person at the mirror drives with their own hand: while they are,
+         the agent's arrow stays out of the way, and it comes back by itself
+         once their hold runs out. */
+      const [person, setPerson] = React.useState(false)
+      const personTimer = React.useRef(0)
+      /* The person's own hand hides the arrow the moment it lands, without
+         waiting for the browser to report a hold: the arrow is a claim about
+         what the agent did, and a claim that outlives the agent's turn is a lie
+         the person has to look past. It returns when their hand stops. */
+      const [hand, setHand] = React.useState(false)
+      const handTimer = React.useRef(0)
+      const touched = () => {
+        setHand(true)
+        clearTimeout(handTimer.current)
+        handTimer.current = setTimeout(() => setHand(false), 1500)
+      }
+      const hearHold = (message) => {
+        const hold = Number(message?.hold ?? 0)
+        if (!Number.isFinite(hold) || hold <= 0) return
+        setPerson(true)
+        clearTimeout(personTimer.current)
+        personTimer.current = setTimeout(() => setPerson(false), hold + 120)
+      }
       const [phase, setPhase] = React.useState('connecting')
       const [note, setNote] = React.useState('')
       /* The address this chat had before its tab went away: the pane reopens
@@ -384,11 +450,15 @@ window.__ModuleLoader__.load({
           pending = null
           setFrame(message)
           setPtr(message.pointer ?? null)
+          if (Number.isFinite(Number(message.w)) && Number.isFinite(Number(message.h))) {
+            setView({ w: Number(message.w), h: Number(message.h) })
+          }
           setPhase('live')
           drawnAt.current = Date.now()
           if (typeof message.url === 'string' && message.url !== '') setUrl(message.url)
         }
         const draw = (message) => {
+          hearHold(message)
           pending = message
           if (raf !== 0) return
           if (typeof requestAnimationFrame === 'function') raf = requestAnimationFrame(paint)
@@ -417,6 +487,7 @@ window.__ModuleLoader__.load({
           } else if (message.type === 'pointer') {
             /* The arrow is drawn the moment the agent moves, not when the page
                next repaints: it stays honest even on a page that is idle. */
+            hearHold(message)
             setPtr(message.pointer ?? null)
           } else if (typeof message.jpeg === 'string') {
             draw(message)
@@ -425,6 +496,8 @@ window.__ModuleLoader__.load({
         source.onerror = () => setPhase('lost')
         return () => {
           clearInterval(stuck)
+          clearTimeout(personTimer.current)
+          clearTimeout(handTimer.current)
           if (raf !== 0 && typeof cancelAnimationFrame === 'function') cancelAnimationFrame(raf)
           source.close()
         }
@@ -448,9 +521,49 @@ window.__ModuleLoader__.load({
         return () => observer.disconnect()
       }, [])
 
-      const scale = size.w > 0 && size.h > 0 ? Math.min(size.w / SIDE_VIEW.width, size.h / SIDE_VIEW.height) : 0
-      const padX = (size.w - SIDE_VIEW.width * scale) / 2
-      const padY = (size.h - SIDE_VIEW.height * scale) / 2
+      const scale = size.w > 0 && size.h > 0 ? Math.min(size.w / view.w, size.h / view.h) : 0
+      const padX = (size.w - view.w * scale) / 2
+      const padY = (size.h - view.h * scale) / 2
+
+      /* The column tells the page how big it is, a beat after the resize stops:
+         a page that relayouts on every pointer step would never settle. */
+      React.useEffect(() => {
+        if (mine !== true || size.w < 200 || size.h < 200) return undefined
+        /* The page already has that shape: a resize that lands between two
+           viewport sizes would only cost a relayout and another frame. */
+        if (Math.abs(Number(view?.w ?? 0) - size.w) < 1 && Math.abs(Number(view?.h ?? 0) - size.h) < 1) return undefined
+        /* The page is told the column's size a beat after the resize stops, and
+           again until the browser says it took it: a request that arrives while
+           the page is still starting is refused, and a pane that gives up then
+           keeps a letterboxed picture for as long as it stays open. */
+        let live = true
+        let tries = 0
+        let timer = 0
+        const send = () => {
+          if (!live) return
+          fetch(`${ROUTE}/sidecar/viewport`, {
+            method: 'POST',
+            headers: { 'content-type': 'application/json' },
+            body: JSON.stringify({ session: String(props.sessionId ?? ''), w: Math.round(size.w), h: Math.round(size.h) }),
+          })
+            .then((answer) => (answer.ok === true ? answer.json() : { ok: false }))
+            .then((said) => {
+              if (!live || said?.ok === true || tries >= 5) return
+              tries += 1
+              timer = setTimeout(send, 400)
+            })
+            .catch(() => {
+              if (!live || tries >= 5) return
+              tries += 1
+              timer = setTimeout(send, 600)
+            })
+        }
+        timer = setTimeout(send, 250)
+        return () => {
+          live = false
+          clearTimeout(timer)
+        }
+      }, [mine, size.w, size.h, view?.w, view?.h])
 
       const who = String(props.sessionId ?? '')
       const post = (path, body) => {
@@ -487,7 +600,7 @@ window.__ModuleLoader__.load({
         const box = el.getBoundingClientRect()
         const x = (event.clientX - box.left - padX) / scale
         const y = (event.clientY - box.top - padY) / scale
-        if (x < 0 || y < 0 || x > SIDE_VIEW.width || y > SIDE_VIEW.height) return null
+        if (x < 0 || y < 0 || x > view.w || y > view.h) return null
         return { x: Math.round(x), y: Math.round(y) }
       }
       const ime = React.useRef(null)
@@ -503,6 +616,7 @@ window.__ModuleLoader__.load({
       }
       const stageRef = stage
       const onPointerDown = (event) => {
+        touched()
         const point = toView(event)
         if (point === null) return
         const field = ime.current
@@ -515,6 +629,7 @@ window.__ModuleLoader__.load({
       }
       const movedAt = React.useRef(0)
       const onPointerMove = (event) => {
+        touched()
         const point = toView(event)
         if (point === null) return
         /* Hover is part of a page: menus open under the hand, so the movement
@@ -526,24 +641,46 @@ window.__ModuleLoader__.load({
         hover(point)
       }
       const onWheel = (event) => {
+        touched()
         event.preventDefault()
-        const point = toView(event) ?? { x: Math.round(SIDE_VIEW.width / 2), y: Math.round(SIDE_VIEW.height / 2) }
+        const point = toView(event) ?? { x: Math.round(view.w / 2), y: Math.round(view.h / 2) }
         post('/sidecar/input', { kind: 'wheel', x: point.x, y: point.y, dx: event.deltaX, dy: event.deltaY })
       }
+      /* The page's keys are the person's keys: shift makes a capital, a
+         shortcut stays a shortcut so Ctrl+V pastes and Ctrl+A selects, and the
+         key that goes down also goes up so a game can hold it. */
+      const modsOf = (event) => (event.altKey === true ? 1 : 0)
+        | (event.ctrlKey === true ? 2 : 0)
+        | (event.metaKey === true ? 4 : 0)
+        | (event.shiftKey === true ? 8 : 0)
+      const MODIFIER_KEYS = ['Alt', 'Control', 'Meta', 'Shift', 'AltGraph', 'OS', 'Super', 'Hyper']
+      const keyBody = (event, down) => {
+        const modifiers = modsOf(event)
+        const shortcut = (modifiers & 7) !== 0
+        const typing = down === true && event.key.length === 1 && (modifiers & 7) === 0 && event.repeat !== true
+        return {
+          kind: 'key',
+          key: event.key,
+          code: event.code || event.key,
+          keyCode: event.keyCode || 0,
+          modifiers,
+          ...(typing ? { text: event.key } : {}),
+          ...(down === false ? { down: false } : {}),
+        }
+      }
       const onKeyDown = (event) => {
+        touched()
         /* A key still being composed belongs to the IME, not to the page. */
         if (event.isComposing === true) return
-        const named = ['Enter', 'Tab', 'Escape', 'Backspace', 'Delete', 'ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight', 'Home', 'End', 'PageUp', 'PageDown']
-        if (named.includes(event.key) === true) {
-          event.preventDefault()
-          post('/sidecar/input', { kind: 'key', key: event.key, code: event.code || event.key, keyCode: event.keyCode || 0 }) // ui-copy-ignore
-          return
-        }
-        if (event.metaKey === true || event.ctrlKey === true || event.altKey === true) return
-        if (event.key != null && event.key.length === 1) {
-          event.preventDefault()
-          post('/sidecar/input', { kind: 'text', text: event.key })
-        }
+        if (event.key == null || MODIFIER_KEYS.includes(event.key) === true) return
+        event.preventDefault()
+        post('/sidecar/input', event.repeat === true ? { ...keyBody(event, true), repeat: true } : keyBody(event, true))
+      }
+      const onKeyUp = (event) => {
+        if (event.isComposing === true) return
+        if (event.key == null || MODIFIER_KEYS.includes(event.key) === true) return
+        event.preventDefault()
+        post('/sidecar/input', keyBody(event, false))
       }
       const goto = () => {
         const address = String(draft ?? '').trim()
@@ -557,7 +694,7 @@ window.__ModuleLoader__.load({
         return h('div', { className: 'dshib-side' },
           h('div', { className: 'dshib-side-note' }, L('side.foreign')))
       }
-      const pointer = ptr
+      const pointer = person === true || hand === true ? null : ptr
       const noteText = phase === 'failed' ? note || L('side.failed') : phase === 'live' ? L('side.empty') : L('side.connecting')
       return h('div', { className: 'dshib-side' },
         h('div', { className: 'dshib-side-bar' },
@@ -587,6 +724,7 @@ window.__ModuleLoader__.load({
           onPointerMove,
           onWheel,
           onKeyDown,
+          onKeyUp,
         },
           frame === null && back !== ''
             ? h('button', {
@@ -627,12 +765,13 @@ window.__ModuleLoader__.load({
               clearIme()
             },
             onKeyDown,
+            onKeyUp,
           }),
           frame === null ? null : h('img', {
             className: 'dshib-side-frame',
             src: `data:image/jpeg;base64,${frame.jpeg}`,
-            width: Math.max(1, Math.round(SIDE_VIEW.width * scale)),
-            height: Math.max(1, Math.round(SIDE_VIEW.height * scale)),
+            width: Math.max(1, Math.round(view.w * scale)),
+            height: Math.max(1, Math.round(view.h * scale)),
             alt: '', // ui-copy-ignore
             draggable: false,
           }),

package/docs/ARCHITECTURE.md

@@ -4,7 +4,7 @@ Two halves in two processes, and a Chrome that neither of them is.
 
 ```
 server side  (index.js, in the DSH process)
-  sidecar_* tools ──> Sidecar ── CDP over a socket ──> headless Chrome
+  browser_* tools ──> Sidecar ── CDP over a socket ──> headless Chrome
                          │
   HTTP + SSE routes <────┘
         │
@@ -41,7 +41,7 @@ Other launch flags:
 | `--use-mock-keychain`, `--password-store=basic` | No keychain prompt on macOS. |
 | `--remote-allow-origins=*` | The CDP client connects over a plain WebSocket. |
 | `--disable-blink-features=AutomationControlled` | Sites that sniff for automation get the ordinary path. |
-| `--window-size=1440,900`, `--lang=zh-CN` | The viewport the pane mirrors, and a Chinese page locale. |
+| `--window-size=1440,900`, `--lang=zh-CN` | The window the browser starts in, and a Chinese page locale; the layout viewport is set to the column instead. |
 
 The binary comes from a fixed list: Google Chrome, Chromium or Canary under `/Applications`,
 then `/usr/bin/google-chrome` and `/usr/bin/chromium`. Nothing else is probed and there is no
@@ -58,7 +58,7 @@ The browser outlives the plugin:
    `/json/new?about:blank` as a fallback) and a client connects to it.
 
 One page per browser. `apply()` only constructs the `Sidecar`; no process exists until the
-first tool call, and `sidecar_state` reports a stopped browser without starting one.
+first tool call, and `browser_state` reports a stopped browser without starting one.
 
 Shutdown asks Chrome to close (`Browser.close`), waits briefly, then sends `SIGTERM`, which is
 the point where Chrome flushes cookies. An idle timer closes the browser after 15 minutes; a
@@ -85,13 +85,16 @@ the CDP socket stays in the server half.
 ## Mirror
 
 The server half starts `Page.startScreencast` with JPEG quality 72, `maxWidth` and `maxHeight`
-at 1440×900, `optimizeForSpeed`, `everyNthFrame: 1`, and acknowledges every frame it receives
+at the pane's size, `optimizeForSpeed`, `everyNthFrame: 1`, and acknowledges every frame it receives
 so Chrome keeps sending. Frames are never queued: with two frames in flight, further frames
 are dropped and the next one that fits takes their place. A slow reader loses frames, never
 freshness.
 
-The viewport is pinned with `Emulation.setDeviceMetricsOverride` to the same 1440×900, so a
-frame is the whole page as laid out, not a crop.
+The layout viewport follows the column: the pane measures itself, posts the size to
+`/sidecar/viewport` a beat after the resize settles, and the server pins
+`Emulation.setDeviceMetricsOverride` to that size and restarts the screencast at it. So a frame
+is the whole page as laid out at the shape it is shown in — no letterbox bands, no crop. Sizes
+below 320 px or above 1920×1600 are ignored rather than applied as a sliver.
 
 If `Browser.getVersion` reports `HeadlessChrome`, the user agent is overridden with an ordinary
 Chrome string, `acceptLanguage: 'zh-CN,zh;q=0.9,en;q=0.8'` and `platform: 'MacIntel'`, so a site
@@ -133,7 +136,7 @@ browser without pulling the column away from the person reading it.
 
 `sidecarTools()` in `tools.js` builds the 28 definitions and `index.js` registers them on the
 plugin's tool scope. All of them return one text block. A page read numbers the controls and
-keeps those numbers across later reads, so `sidecar_click` by ref means the element the model
+keeps those numbers across later reads, so `browser_click` by ref means the element the model
 last saw; navigation drops them and the model gets `REF_GONE` rather than a wrong click.
 
 Before a ref click or a ref focus, one page-side probe asks what is on top of that control.
@@ -145,7 +148,7 @@ reported, never accepted on the person's behalf. The whole exchange is one `Runt
 and a possible `mousePressed`/`mouseReleased` pair on an already-open CDP connection, so an
 uncovered click pays nothing.
 
-`sidecar_do` runs the same probe before a step that names a control — by `ref`, by
+`browser_do` runs the same probe before a step that names a control — by `ref`, by
 text, or by field label — so a banner the first step clears is gone for the rest of
 the batch, and the step's own line says it in one clause (`after Accept on a banner,`).
 A batch therefore pays for one overlay, not one per step.
@@ -154,7 +157,7 @@ An action answers with one line — address, title, focused field — and reads
 `read: true` is passed. That is what keeps a long batch cheap (see `BENCH.md`).
 
 Calls that touch the browser are marked not concurrency-safe; reading state, console and failed
-requests is. `sidecar_open`, `sidecar_view`, `sidecar_wait` and `sidecar_shot` carry longer
+requests is. `browser_open`, `browser_view`, `browser_wait` and `browser_shot` carry longer
 timeouts than the rest.
 
 Sign-in state lives in the profile, and session cookies — the kind a sign-in usually sets,
@@ -169,7 +172,7 @@ without asking first when someone wants something done on a site.
 `Network.enable` is already on for the mirror. A request is kept when it is
 sent and settled when its response arrives, so the list holds method, status,
 type and address of what the page actually got — 120 rows, oldest dropped.
-`sidecar_network` reads that list without touching the page, which is why it
+`browser_network` reads that list without touching the page, which is why it
 can answer while a dialog is open.
 
 ## Downloads
@@ -177,7 +180,7 @@ can answer while a dialog is open.
 Chrome is pointed at `<profile>/downloads` when the page starts, so a click
 that saves a file writes it there instead of nowhere. The folder belongs to
 the profile: it is the same one on the next launch, and nothing outside the
-profile is touched. `sidecar_downloads` lists what is on disk, newest first;
+profile is touched. `browser_downloads` lists what is on disk, newest first;
 a `.crdownload` still in flight is not listed.
 
 ## Frames and popups
@@ -215,4 +218,4 @@ and there is still one page to drive.
 | `locale/*.json` | Page: pane labels. |
 
 Data on disk is confined to `~/.dsh/inner-browser-use/`: `chrome/<profile>` for the browser,
-`screenshots/` for `sidecar_shot`.
+`screenshots/` for `browser_shot`.

package/docs/ARTIFACTS.md

@@ -8,7 +8,7 @@
 
 - **项目**：会话工作目录的文件夹名，加路径哈希前 6 位，例如 `dsh-inner-browser-use-a1b2c3`。同名文件夹、不同路径不会混。
 - **会话**：会话 id 去掉 `session-` 后的前 8 位，例如 `72557d6e`。同一项目里两个会话各存各的。
-- **种类**：`shots/` 是 `sidecar_shot` 的截图，`pictures/` 是 `sidecar_grab` 抓的图，`pdf/` 是 `sidecar_pdf` 打印的文档。
+- **种类**：`shots/` 是 `browser_shot` 的截图，`pictures/` 是 `browser_grab` 抓的图，`pdf/` 是 `browser_pdf` 打印的文档。
 
 文件名默认带时间戳（`shot-20261028-124500.png`），给了 `name` 就用它；名字里的分隔符、连续点、开头点都会被去掉，跳不出这个目录。