dsh-inner-browser-use 0.34.28 → 0.34.95

package/docs/VERIFY.md

@@ -33,7 +33,7 @@ Expected before any tool call: `"alive":false`, `"profile":"default"`,
 
 ## 2. A tool opens its own page
 
-Ask the model `用 sidecar_open 打开 https://www.bing.com`, or call it yourself.
+Ask the model `用 browser_open 打开 https://www.bing.com`, or call it yourself.
 Expected: the result names the address and title, and this chat's right column
 shows the picture in a tab called **后台浏览器** or **Background browser**.
 `doctor.mjs` now reads `attached, headless`, `viewers 1`, frames climbing.
@@ -43,8 +43,8 @@ shows the picture in a tab called **后台浏览器** or **Background browser**.
 - `"alive":false` with a non-empty `"error"`: Chrome did not start. Check the
   binary at a listed path, and on macOS that the launch carried
   `--use-mock-keychain`.
-- Frames flat while viewers is 1: the screencast stalled. `sidecar_stop`, then
-  `sidecar_open` again.
+- Frames flat while viewers is 1: the screencast stalled. `browser_stop`, then
+  `browser_open` again.
 - `"pointer":null`: no tool input yet; it fills on the first click or move.
 
 ## 3. Only the chat that asked
@@ -60,18 +60,28 @@ A mirror that opens in every chat means the page half is not comparing
 - **Hover** the search field: your cursor turns into an I-beam, the black arrow
   does not move. An arrow that follows the mouse means the pointer paths are
   crossed.
-- **Type** Chinese with `sidecar_type { "text": "…", "submit": true }`: the
+- **Type** Chinese with `browser_type { "text": "…", "submit": true }`: the
   results page appears and the field holds characters, not boxes.
-- **Backspace** with `sidecar_key { "key": "Backspace" }` in a filled field:
+- **Backspace** with `browser_key { "key": "Backspace" }` in a filled field:
   one character goes. Nothing happening means the key reached the document.
-- **Screenshot** with `sidecar_shot {}`: an absolute path under
+- **Screenshot** with `browser_shot {}`: an absolute path under
   `~/.dsh-inner-browser-use/screenshots`. A black picture means the screencast
   was not running when the frame was taken.
+- **The whole keyboard, on this machine**: `npm run keys` starts a real
+  background Chrome, presses all 150 keys into a field, and checks what the page
+  reported for each one (`key`, `code`, `keyCode`, shift) plus what the field
+  typed. It then runs copy / cut / paste / select-all / undo / redo, caret keys,
+  Tab focus, Enter, and a 180 ms hold. One JSON line, exit code 0 when every key
+  matched. CI runs it on macOS and Windows in the `keys` job.
+- **A shortcut by hand**: type `Hi`, press `browser_key { "key": "Mod+a" }`,
+  type `X`: the field holds `X` and nothing before it. `Mod` is Command on macOS
+  and Control elsewhere; `Ctrl+C/X/V/A/Z/Y` and `Ctrl+Shift+Z` run the same
+  editing commands a person would.
 
 ## 5. Stop, then open again
 
-`sidecar_stop` reads `"alive":false` and leaves the profile alone.
-`sidecar_open https://www.bing.com` brings the same profile back, still signed
+`browser_stop` reads `"alive":false` and leaves the profile alone.
+`browser_open https://www.bing.com` brings the same profile back, still signed
 in from the first pass.
 
 Signed out after the restart: Chrome came up on another profile, or was killed
@@ -140,13 +150,13 @@ broke. Two of them cover a cross-site frame, which Chrome runs in its own
 process. Three cover a control under an overlay: the click clears a cookie
 banner and lands (`under clicked`), typing into a field under the same banner
 works, and a modal that offers to delete the account is reported without
-anything being pressed. Two more drive the same overlay through `sidecar_do`:
+anything being pressed. Two more drive the same overlay through `browser_do`:
 a batch step clears the banner before it clicks, and a batch step fills a field
 that was under it. One more counts the recoveries: after the redraw check, the
 state line says at least one ref was followed to its new number. Three cover a
 hand on the mirror: while the person is using it a read still works
 (`hold-allows-read`), an action is refused and names both ways out
-(`hold-refuses-action`), and `sidecar_resume` hands the page back so the same
+(`hold-refuses-action`), and `browser_resume` hands the page back so the same
 click goes through (`resume-lets-go`).
 
 ## Who the mirror serves

package/index.js

@@ -20,7 +20,7 @@
  * learns no port and no secret. // ui-copy-ignore
  */
 
-import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
+import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
 import { homedir } from 'node:os'
 import { join } from 'node:path'
 import { Sidecar } from './sidecar.js?v=1'
@@ -106,6 +106,17 @@ export class Bridge {
     this.wanted = { session: kept.session, at: kept.at }
   }
 
+  /**
+   * A chat is the chat that asked, and nothing else. Guessing at an alias once
+   * put the mirror in a stranger's column: two conversations are never the same
+   * conversation, however few pages happen to be on screen.
+   * @param {string} session - whatever id the caller has.
+   * @returns {string} the same id.
+   */
+  canon(session) {
+    return String(session ?? '')
+  }
+
   /**
    * Give the mirror to one chat and take it away from every earlier one.
    * @param {string} sessionId - chat that asked for it.
@@ -123,11 +134,12 @@ export class Bridge {
    * @returns {boolean} whether a live tab took the frame.
    */
   want(sessionId, at = Date.now()) {
-    if (sessionId === '') return false
-    this.claim(sessionId)
-    this.wanted = { session: sessionId, at }
-    if (this.persist === true) writeMirror({ owner: this.owner, session: sessionId, at })
-    return this.push(sessionId, { type: 'sidecar-wanted', session: sessionId, at })
+    const id = this.canon(sessionId)
+    if (id === '') return false
+    this.claim(id)
+    this.wanted = { session: id, at }
+    if (this.persist === true) writeMirror({ owner: this.owner, session: id, at })
+    return this.push(id, { type: 'sidecar-wanted', session: id, at })
   }
 
   /**
@@ -195,7 +207,7 @@ export class Bridge {
 /** Page read: address, title, controls, then text, as JSON the Host parses. */ // ui-copy-ignore
 /* The read model and the tool surface live in their own files: this module is
    the Host half (routes, bridge, lifetimes), not the model's vocabulary. */ // ui-copy-ignore
-import { KEYS, sidecarTools, sessionScope, workScope } from './tools.js?v=1'
+import { keyEvent, sidecarTools, sessionScope, workScope } from './tools.js?v=1'
 
 /**
  * A state poll may pick up a browser that outlived a reload, and only for the
@@ -252,6 +264,36 @@ export function shouldAdoptOnState(said, chat, owner) {
   return said?.alive === false && chat !== '' && owner === chat
 }
 
+/**
+ * Whether a running browser still needs its column told to draw it.
+ * @param {{alive: boolean, owner: string, wanted: string, listeners: number}} what - current facts.
+ * @returns {boolean} true when the owner is known, the browser runs, and nothing listens.
+ */
+export function wantsMirrorAgain(what) {
+  if (what?.alive !== true) return false
+  const owner = String(what?.owner ?? '')
+  if (owner === '') return false
+  if (String(what?.wanted ?? '') !== owner) return false
+  return Number(what?.listeners ?? 0) <= 0
+}
+
+/**
+ * Whether asking again can achieve anything. A pane that is polling this very
+ * endpoint is awake and already knows the frames are missing, so a second ask
+ * every few seconds only writes log lines; a pane that went quiet may have lost
+ * the stream and does need one. One ask per gap at most.
+ * @param {any} what - { askedAgo, sinceLast, quiet, gap } in milliseconds.
+ * @returns {boolean} true when the ask should be sent now.
+ */
+export function nudgeDue(what) {
+  const askedAgo = Number(what?.askedAgo ?? -1)
+  const sinceLast = Number(what?.sinceLast ?? Infinity)
+  const quiet = Number(what?.quiet ?? 15_000)
+  const gap = Number(what?.gap ?? 10_000)
+  if (!Number.isFinite(askedAgo) || askedAgo < quiet) return false
+  return sinceLast >= gap
+}
+
 export const name = 'dsh-inner-browser-use'
 export const inject = ['webServer']
 
@@ -268,6 +310,31 @@ export function apply(ctx) {
   /** @type {{refs: any[], at: number, last?: any}} */
   const sidecarRefs = { refs: [], at: 0 }
   ctx.effect(() => () => bridge.dispose(), 'dsh-inner-browser-use: bridge lifetime')
+  /* A live picture that nobody is drawing is the one failure people notice
+     first: the browser is fine, the column just lost its pane. The chat that
+     owns the browser is asked again, on a slow beat, until the frames stream
+     has a listener — the client keeps the rule that only the mounted chat's
+     column may open. */
+  /** @type {Map<string, number>} when each chat last polled for state. */
+  const lastAsk = new Map()
+  ctx.effect(() => {
+    let last = 0
+    const timer = setInterval(() => {
+      try {
+        const owner = bridge.owner
+        const listeners = sidecar.page?.frames?.size ?? 0
+        const again = wantsMirrorAgain({ alive: sidecar.page !== null && sidecar.page !== undefined, owner, wanted: bridge.wanted.session, listeners }) // ui-copy-ignore
+        if (again !== true) return
+        const due = nudgeDue({ askedAgo: Date.now() - (lastAsk.get(owner) ?? 0), sinceLast: Date.now() - last }) // ui-copy-ignore
+        if (due !== true) return
+        last = Date.now()
+        bridge.want(owner)
+        appendFileSync(join(STATE_DIR, 'asks.log'), `${new Date().toISOString()} asked ${owner.slice(-8)} to draw the mirror again\n`)
+      } catch {}
+    }, 4000)
+    timer.unref?.()
+    return () => clearInterval(timer)
+  }, 'dsh-inner-browser-use: mirror keepalive')
   /* Closing the browser cleanly on quit is what keeps a sign-in alive: a
      killed Chrome never writes the cookies a page set this minute. */
   ctx.effect(() => {
@@ -285,6 +352,10 @@ export function apply(ctx) {
   /** @param {import('node:http').IncomingMessage} req - request. @param {import('node:http').ServerResponse} res - response. */
   /** @type {{at: number, path: string, session: string}[]} the last few calls the column made. */
   const recent = []
+  /* Whether the pane is asking at all is the difference between "the browser
+     stopped" and "nothing is drawing it": the poll is every 2.5 s, so a chat
+     that asks is a chat with a mounted pane. Kept on disk because the answer
+     has to survive a plugin reload, which is when the question is asked. */
   const handler = async (req, res) => {
     const url = new URL(req.url ?? ROUTE, 'http://dsh.invalid')
     const path = url.pathname.slice(ROUTE.length) || '/'
@@ -334,6 +405,8 @@ export function apply(ctx) {
       const asked = String(url.searchParams.get('session') ?? '')
       const watching = asked !== '' ? asked : sessionId
       if (held(watching) && refuse()) return
+      /* Only the owner's own page may stream the picture. */
+      const streaming = watching
       res.writeHead(200, {
         'content-type': 'text/event-stream; charset=utf-8',
         'cache-control': 'no-cache, no-transform',
@@ -353,7 +426,7 @@ export function apply(ctx) {
              page wait: the newest picture is the only one worth sending. */
           if (res.writableLength > 3 * 1024 * 1024) return
           res.write(`data: ${JSON.stringify(frame)}\n\n`)
-        }, watching)
+        }, streaming)
         const info = await sidecar.state()
         /* The pane needs one address to offer back when this chat's own page
            went away; the state route answers the same question for the tab
@@ -426,6 +499,15 @@ export function apply(ctx) {
           said = await sidecar.state()
         } catch {}
       }
+      if (chat !== '') {
+        const before = lastAsk.get(chat) ?? 0
+        if (Date.now() - before > 2000) {
+          lastAsk.set(chat, Date.now())
+          try {
+            appendFileSync(join(STATE_DIR, 'asks.log'), `${new Date().toISOString()} ${chat.slice(-8)} alive=${String(said?.alive === true)} viewers=${String(said?.viewers ?? 0)}\n`)
+          } catch {}
+        }
+      }
       json(200, { ...said, ...(chat === '' ? {} : { chat: mineChat ?? (await sidecar.chatState(chat)) }), owner: bridge.owner, chats: [...bridge.seats.keys()], recent: mine })
       return
     }
@@ -463,6 +545,35 @@ export function apply(ctx) {
       return
     }
 
+    if (path === '/sidecar/viewport' && req.method === 'POST') {
+      /* The pane tells the page how big it is, so the picture fills the column
+         instead of being letterboxed inside it. Only the owner may resize what
+         its own mirror shows. */
+      let body
+      try {
+        body = JSON.parse(await readBody(req))
+      } catch {
+        json(400, { ok: false, text: 'bad frame' }) // ui-copy-ignore
+        return
+      }
+      const actor = acting(body)
+      if (actor === '') {
+        json(403, { ok: false, text: 'this chat does not own the browser' }) // ui-copy-ignore
+        return
+      }
+      try {
+        const page = sidecar.page
+        if (page === null || page === undefined) {
+          json(409, { ok: false, text: 'the browser is not open' }) // ui-copy-ignore
+          return
+        }
+        json(200, { ok: true, view: await page.setViewport(body.w, body.h) })
+      } catch (error) {
+        json(500, { ok: false, text: String(error?.message ?? error) })
+      }
+      return
+    }
+
     if (path === '/sidecar/input' && req.method === 'POST') {
       let body
       try {
@@ -495,21 +606,26 @@ export function apply(ctx) {
         } else if (kind === 'wheel') await page.wheel({ x, y, dx: Number(body.dx ?? 0), dy: Number(body.dy ?? 0) })
         else if (kind === 'text') await page.insertText(String(body.text ?? ''))
         else if (kind === 'key') {
-          const name = String(body.key ?? '')
-          const known = KEYS[name]
-          const spec = {
-            key: name,
-            code: String(body.code ?? known?.code ?? name),
-            keyCode: Number(body.keyCode ?? 0) || known?.keyCode || 0,
-            text: body.text === undefined ? known?.text : String(body.text),
-            modifiers: Number(body.modifiers ?? 0),
-          }
-          await page.key('keyDown', spec)
-          await page.key('keyUp', { ...spec, text: undefined })
+          const key = keyEvent(body)
+          if (key.down !== true) await page.key('keyUp', { ...key.spec, text: undefined })
+          else if (key.repeat === true) {
+            /* Typing repeat: the page draws a held key as a fresh keydown. */
+            await page.key('keyDown', { ...key.spec, autoRepeat: true })
+          } else if (key.command === null) await page.key('keyDown', key.spec)
+          else await page.key('keyDown', { ...key.spec, text: undefined, commands: [key.command] })
         }
         /* Hovering and moving are looking, not acting: only real input counts as
            the person taking the page. */
-        if (kind !== 'hover' && kind !== 'move') sidecar.humanUse({ kind, x, y })
+        if (kind !== 'hover' && kind !== 'move') {
+          sidecar.humanUse({ kind, x, y })
+          /* The arrow goes now, not on the next repaint: a still page may not
+             paint again for a second and a half. */
+          try {
+            page.announceHold()
+          } catch {
+            /* the stream is gone; the next frame carries the same answer */
+          }
+        }
         json(200, cursor === undefined ? { ok: true } : { ok: true, cursor })
       } catch (error) {
         json(500, { ok: false, text: String(error?.message ?? error) })
@@ -668,11 +784,15 @@ export function apply(ctx) {
       name: 'browser-in-this-chat',
       order: 858,
       text: [
-        '## 网页任务：用 sidecar 浏览器做，不问直接开始',
+        '## 网页任务：人明确点名才开右栏浏览器',
+        '右栏「后台浏览器」是一台独立的后台 Chrome，模型能点、能读、能执行脚本；画面只出现在本会话右栏，不占别的会话。它只做「必须动手」的活，默认不开。',
         '',
-        '人说「用网页端 / 在浏览器里 / 打开某个网站 / 帮我上 ChatGPT 生成 …」时，直接用 `sidecar_*` 工具；不要问要不要打开，也不要另起一个外部 Chrome。它的画面会出现在本会话右栏，做完保留，不主动关。',
-        '先接着用已有的页：`sidecar_view`（拿编号控件与可见文字）→ `sidecar_click` / `sidecar_type` → 再 `sidecar_view` 直到出现要的结果；页面要等就重复读，不 sleep。`sidecar_open` 只用于没有页、要换新地址、或要明确刷新；同一个地址不要反复重开。',
-        '要保存页面样子用 `sidecar_shot`，它会给出 PNG 绝对路径；要交给人看的文件，把绝对路径写进回复。',
+        '只有人明确说要看网页/用浏览器时才开始：「用网页端 / 在浏览器里 / 打开某个网站 / 上 ChatGPT 生成 / 帮我在这个页面点…」。人只给了链接、没说要用浏览器时，先用 `web_fetch` 或 envoy 读；读不到（登录墙、空壳、要交互）就说明原因再问一句，不要自己开。',
+        '不要用它：读公开文章、单个链接正文、文档、搜索结果摘要、GitHub 仓库或 PR 页面——这些能用 `web_fetch` / envoy 读就不用浏览器；一打开就是一个后台 Chrome 进程加一个页签，会话多了很占内存。',
+        '人点名之后不问、不另起外部 Chrome；画面留在本会话右栏，做完保留，不主动关。',
+        '先接着用已有的页：`browser_view`（拿编号控件与可见文字）→ `browser_click` / `browser_type` → 再 `browser_view` 直到出现要的结果；页面要等就重复读，不 sleep。`browser_open` 只用于没有页、要换新地址、或要明确刷新；同一个地址不要反复重开。',
+        '页面很长、只想确认某句话/按钮在哪时用 `browser_find`，别整页读。要读长文（文章、文档、issue、论坛帖）用 `browser_read`：它回 Markdown，只花几十分之一的 token，链接可点；要动手点、填表才用 `browser_view` 拿编号。',
+        '要保存页面样子用 `browser_shot`，它会给出 PNG 绝对路径；要交给人看的文件，把绝对路径写进回复。',
       ].join('\n'),
     }), 'dsh-inner-browser-use: system prompt section')
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-inner-browser-use",
-  "version": "0.34.28",
+  "version": "0.34.95",
   "description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
   "type": "module",
   "main": "index.js",
@@ -22,6 +22,11 @@
     "cordis.patch.yml",
     "icon.svg",
     "scripts/doctor.mjs",
+    "scripts/check-flash-window.mjs",
+    "scripts/check-flash-mirror.mjs",
+    "scripts/check-flash-local.mjs",
+    "scripts/check-flash-tools.mjs",
+    "runtime/",
     "docs/",
     "README.md",
     "README.zh.md",
@@ -43,12 +48,21 @@
     "dsh": ">=0.2.0"
   },
   "scripts": {
-    "test": "node --test test/*.test.mjs",
+    "test": "node --test test/artifacts.test.mjs test/client.test.mjs test/cookies.test.mjs test/diff.test.mjs test/fresh.test.mjs test/host.test.mjs test/input.test.mjs test/isolation.test.mjs test/keyboard.test.mjs test/leak.test.mjs test/login.test.mjs test/no-duplicate.test.mjs test/package.test.mjs test/readfault.test.mjs test/refs.test.mjs test/review.test.mjs test/session.test.mjs test/settle.test.mjs test/sidecar.test.mjs test/swf.test.mjs test/tabs.test.mjs test/trouble.test.mjs",
     "doctor": "node scripts/doctor.mjs",
     "breadth": "node scripts/breadth.mjs",
     "live": "node scripts/live.mjs",
     "adopt": "node scripts/check-adopt.mjs",
-    "game": "node scripts/check-game.mjs"
+    "game": "node scripts/check-game.mjs",
+    "modes": "node scripts/check-modes.mjs",
+    "ready": "node scripts/check-prepare.mjs",
+    "pdf": "node scripts/check-pdf.mjs",
+    "swf": "node scripts/swf-inspect.mjs",
+    "flash-hidden": "node scripts/check-flash-window.mjs",
+    "flash-mirror": "node scripts/check-flash-mirror.mjs",
+    "flash-tools": "node scripts/check-flash-tools.mjs",
+    "flash-local": "node scripts/check-flash-local.mjs",
+    "keys": "node scripts/keyboard.mjs"
   },
   "license": "MIT",
   "author": {