dsh-inner-browser-use 0.27.0 → 0.34.95

package/docs/FLASH.md

@@ -0,0 +1,84 @@
+# Flash 之后：把兼容性放在正确的位置上
+
+Chrome 88（2021 年 1 月）拿掉 Flash，不是商业决定，是插件模型的失败：一个能直接操作音频、图形、网络和本地文件的第三方二进制，安全面等于浏览器本身，而它承担的内容又都可以用开放标准重写。结论对自研同样成立：**不要再造插件槽位，要造内容层的运行时。**
+
+## 兼容的到底是什么
+
+一个 SWF 只有四样东西是「约定」：
+
+1. **标签流**——Shape/Sprite/Bitmap/Font/Sound/Frame 的二进制结构，决定画面怎么组出来；
+2. **时间线**——`ShowFrame` 与帧率构成保留模式动画，帧可以跳、可以循环；
+3. **AVM1 / AVM2 语义**——脚本看到的 DisplayObject、事件、`MovieClip`、`SharedObject`；
+4. **安全模型**——跨域、socket 策略、本地存储。
+
+只要这四样对得上，游戏就能跑。渲染用 Canvas2D 还是 WebGPU、时钟是实时还是虚拟、存储落 IndexedDB 还是宿主磁盘，**游戏看不见**。这就是「参考原理重做一个更好的」的立足点：兼容性锁在 AVM + 标签流，创新空间在其余全部。
+
+## 分层设计
+
+```
+SWF ──▶ 摄入(读取标签流, 建 IR) ──▶ 显示列表 + 时间线 ──▶ 渲染(WebGPU, 损伤跟踪)
+                                     │
+                                     ├─ AVM 兼容核(Ruffle wasm / 自研 AVM1 解释器)
+                                     ├─ 虚拟时钟(固定步长, 输入带时间戳)
+                                     ├─ 能力沙箱(网络/存储/剪贴板 走宿主代理, 可录可回放)
+                                     └─ 对象图内省(每帧把状态压成 JSON 差异, 给 agent 读)
+```
+
+四层的分工决定了「比 Flash 好」不是同一维度的竞速：
+
+| 维度 | Flash 的做法 | 这里的做法 | 可测指标 |
+|---|---|---|---|
+| 启动 | 装插件、NPAPI/PPAPI 握手 | 页面内 wasm，零安装 | 首帧 ms |
+| 安全 | 插件进程 + 沙箱补丁 | 无原生代码，网络存储全代理 | 无提权面 |
+| 确定性 | 真实时钟，帧率随机器 | 固定步长虚拟时钟 + 输入时间戳 | 同输入同结果 |
+| 内省 | 无（黑盒像素） | 显示列表与 AVM 状态可导出 | 状态 → JSON 字节数 |
+| 传输 | 屏幕像素 | 损伤区域 + 增量 | 每帧 KB |
+| 回放 | 无 | 事件日志可重放 | 重放一致率 |
+
+## 已落地：PPAPI 分支做成插件里的开关（本版）
+
+「兼容层」和「能立刻玩」不是二选一。插件现在有两套运行时，同一个 CDP 管线、同一个镜像、同一套工具，切换只是一次重启：
+
+```
+browser_open({ flash: true, url: 'http://aola.100bt.com/play/play.html?from_h5=true' })
+```
+
+开的时候宿主用 Electron 11.3.0（Chromium 87，最后一个还认 PPAPI 的 Chromium）当浏览器，关的时候切回 Chrome。输入注入、导航、读页面、镜像全都不用改，因为两边都说 CDP。
+
+三条必须同时成立的条件（缺一条就报「Flash 缺失」，且症状会骗人）：
+
+1. `--ppapi-flash-path` 指向 bundle **里的 Mach-O 可执行文件** `PepperFlashPlayer.plugin/Contents/MacOS/PepperFlashPlayer`，不是 `.plugin` 目录。指错目录时插件报 `No PPP_GetInterface in plugin library`——那是假象，不是导出缺失。
+2. 必须加 `--no-sandbox`。否则 ppapi 进程报 `dlopen(...) (file system sandbox blocked open())`，页面提示没有 Flash。沙箱只对这一个隐藏的独立 profile 关，正常浏览器不关。
+3. 宿主必须是 x86_64（Apple 芯片跑 Rosetta）：插件只有 i386/x86_64 切片，arm64 Electron 装不进它。所以下载固定取 `darwin-x64`。
+
+Electron 11 的两个真实限制，代码里已按它们写：
+
+- 它不会用当前 Chrome 的 profile 目录（会降级并上锁），Flash 分支固定用 `<profile>-flash` 独立目录；两边登录态各自独立。
+- `/json/new` 在 Chromium 87 上是 404，单窗口应用也没有第二个窗口，所以 Flash 分支不建新页签：会话要页签时把唯一那个窗口导航过去。
+
+实测数字（Apple 芯片 + Rosetta，1440×900）：
+
+| 指标 | 值 |
+|---|---|
+| 冷启动到能驱动页面 | 1389 ms（首次含解压；同机热启 545–927 ms） |
+| 页面首帧出图 | 约 8–10 s（SWF 自己的加载），之前是 11 KB 白帧 |
+| 单帧 JPEG q72 | 96–103 KB |
+| `capturePage` 取一帧 | 16 ms |
+| 镜像 SSE 节奏 | 15 fps（66 ms 一轮），4 s 收 60 帧，均值 97.8 KB |
+| PC 版判定 | 页面里出现 `EMBED .../weblogin/20171026_1.swf`、登录框是「多多号/手机/邮箱」而不是 H5 的样式 |
+
+**不能随包分发 Adobe Flash Player**：插件本体留在本机状态目录（`~/.dsh/inner-browser-use/flash/plugin/`），由 `npm run flash:setup` 从本机已有的安装里找/拷，找不到就明说；Ruffle 那条路（MIT/BSD）仍是可分发的那条。
+
+## 分阶段
+
+- **M0 摄入（已发布）**：读 SWF 标签流，给出尺寸/帧率/AVM/资源构成。`runtime/swf.mjs` + `npm run swf`。
+- **MP 立刻能玩（本版）**：PPAPI 分支 + 插件开关，见上节；`npm run flash:setup` 备运行时。
+- **M1 能跑**：自托管 Ruffle + 宿主路由 + `embed/object` 自动 polyfill，在一个真实 SWF 上量首帧、fps、字节数。
+- **M2 能读**：把 Ruffle 的 player 对象与显示列表接到 `browser_eval`，agent 直接读游戏状态，不用截图。
+- **M3 更快**：损伤跟踪 + 增量帧传输；对同一游戏给出「Flash 版 vs 这里」的毫秒与 CPU 对比。
+
+## 诚实边界
+
+- AVM2 的兼容上限由兼容核决定；Ruffle 官方口径是「多数游戏能玩，仍有 API 缺口」，具体游戏要实测。
+- Stage3D、加密 SWF、依赖本地插件的网游登录器，不在 M1–M3 的承诺内。
+- ZWS（LZMA）本版明确拒绝；补 LZMA 是一天的事，但先要有真实影片来验证。

package/docs/ROADMAP.md

@@ -50,7 +50,7 @@ stealing the column from other chats.
 
 ## Phase 2 — one call, many steps (target: a login flow in 1 tool call)
 
-- `sidecar_do`: ordered steps with per-step waits (`wait_text`, `wait_gone`,
+- `browser_do`: ordered steps with per-step waits (`wait_text`, `wait_gone`,
   `wait_idle`, `wait_url`), one result: what changed, the final URL, tokens spent
   on one read instead of five.
 - Auto-wait after every action: navigation start/finish, DOM quiet, and
@@ -65,7 +65,7 @@ stealing the column from other chats.
 - File upload (`DOM.setFileInputFiles`), download capture with a returned path,
   print-to-PDF, full-page and element screenshots.
 - Tabs: new / switch / close / list; iframes addressed by frame id; dialog
-  accept/dismiss; `sidecar_eval` for the last 1% of cases; cookies read/write for
+  accept/dismiss; `browser_eval` for the last 1% of cases; cookies read/write for
   debugging; console + failed requests surfaced in the state read.
 - Keyboard: modifiers, shortcuts, clipboard paste, IME composition unchanged.
 
@@ -86,14 +86,14 @@ stealing the column from other chats.
 
 - Human takeover (0.17.1): real input in the pane (click / text / wheel / key)
   holds the agent's next action for 6 s and says so; hover and move do not count;
-  reads stay open; `sidecar_resume` takes the page back at once.
+  reads stay open; `browser_resume` takes the page back at once.
 - Guarded actions (upload, download, close tab, buy-ish submits) can require a
   confirm flag. Shipped in 0.14.0 for the narrow case that matters most: an
   overlay in the way is cleared only when its own way out is a safe word
   (accept / agree / close / 同意 / 关闭 …) and the overlay says nothing about
   deleting, paying or signing out; otherwise it is reported, never taken.
   Since 0.15.0 the same clearing runs before every control-naming step of a
-  `sidecar_do` batch, so a multi-step flow through a cookie wall fails zero times.
+  `browser_do` batch, so a multi-step flow through a cookie wall fails zero times.
 - Crash recovery: attach to a still-running Chrome, rebuild the page, keep the
   profile; a killed host never leaves a zombie Chrome.
 - Every failure names what happened, what state the page is in, and the next
@@ -124,10 +124,10 @@ finding the control again by identity (kind + exact words + link destination),
 refusing when the words are not unique. Measured at 462 ms against 445 ms for a
 fresh control — 17 ms when needed, nothing when not.
 
-Since 0.16.1 the count is visible: `sidecar_state` reports `refs followed N`,
+Since 0.16.1 the count is visible: `browser_state` reports `refs followed N`,
 and breadth asserts it moved after a redraw, so the next rounds can tell from
 real sites whether redrawing is frequent enough to justify more work here.
-Next in this direction: the same identity recovery for `sidecar_do` steps that
+Next in this direction: the same identity recovery for `browser_do` steps that
 name a field by its label when the label element is rebuilt (today the step
 falls back to a fresh read).
 
@@ -142,7 +142,7 @@ then repeat the loop with whatever the numbers say is worst.
 Two things were wrong at the edge of the tool surface, and both were only
 reachable on real sites.
 
-The first was that the shape of the browser could not change. `sidecar_open`
+The first was that the shape of the browser could not change. `browser_open`
 accepted `headless: false` and answered success; when a browser was already
 running nothing happened, because the flag was only read at launch. A site that
 refuses a windowless browser (chatgpt.com on this machine: a "just a moment"
@@ -150,7 +150,7 @@ page forever, versus a login page in a windowed one) therefore had no way out
 except killing a process by hand. Since 0.17.0 a request for the other shape
 restarts the browser on the same profile, cookie jar and address — 21.6 s
 windowless → windowed, 22.1 s back, 0 ms when the shape already matches — and
-both `sidecar_open` and `sidecar_state` say which shape is in front of the
+both `browser_open` and `browser_state` say which shape is in front of the
 person.
 
 The second was silence on a check page. A site asking for proof of a person is
@@ -181,7 +181,7 @@ The tool used to treat "the control that says Email" as "the field called
 Email". Real forms separate the two, so the writing step now resolves the words
 to the element that takes text, and says which one it used.
 
-Next in this direction: the same resolution for `sidecar_type` when the caller
+Next in this direction: the same resolution for `browser_type` when the caller
 names a field by its words instead of a ref, and for date/time pickers where the
 visible box is a div and the real input is hidden beside it.
 
@@ -189,15 +189,15 @@ visible box is a div and the real input is hidden beside it.
 
 Typing by name is in. The same resolution is what other steps should be able to
 use when the caller knows the destination but not the number — the next place
-worth doing this is `sidecar_select` (choose an option by the field it belongs
+worth doing this is `browser_select` (choose an option by the field it belongs
 to), and after that the date/time case where the visible box is a div and the
 real input sits hidden beside it.
 
 ## Phase 12 — the same words, the same place, 0.21.0
 
-`sidecar_select` joined `sidecar_type` and `sidecar_do` on the shared resolver.
+`browser_select` joined `browser_type` and `browser_do` on the shared resolver.
 Left in this direction: the date/time case where the box the user sees is a div
-and the real input is hidden beside it, and `sidecar_click` on a control whose
+and the real input is hidden beside it, and `browser_click` on a control whose
 words are split across sibling nodes (`<span>Sign</span><span>in</span>`).
 
 ## Phase 13 — the page's spelling is not the caller's, 0.22.0
@@ -216,7 +216,7 @@ name, so the fallback has to be a picture plus a click, not a name lookup.
 
 ## Phase 16 — a file, not a path, 0.25.0
 
-`sidecar_grab` returns a path. Next: say one line about the picture (size, and
+`browser_grab` returns a path. Next: say one line about the picture (size, and
 whether it is mostly one colour) so a model that cannot open files still learns
 "the chart is blank" without a full read.
 
@@ -232,3 +232,97 @@ A picture is only an answer if it loaded. Next: report a large picture element
 whose source failed (`naturalWidth === 0`) as a broken picture rather than
 staying silent, since that is the commonest way a generated image "disappears".
 
+## Phase 19 — the same picture again, 0.28.0
+
+Reads are stateless: the same page costs the same tokens twice. Next: remember
+the last read of a page by address and say only what changed, with a hard reset
+when the address or the document changes.
+
+## Phase 20 — two browsers, one conversation, 0.29.0
+
+The column is a real browser, so the model inside it can be a reviewer, not just
+a tool: hand it the repository address, ask for a review, read the answer back
+through the same mirror, and take the next round from it. Nothing opens on the
+desktop while that happens: a site that refuses a headless build gets a window
+at -3000,-3000, and the column shows the only picture.
+
+## Phase 21 — a review round that closes the loop, 0.30.0
+
+`browser_ask` sends and returns an answer. Next: a round trip that carries the
+repository address there and the review back, and records in the round log which
+commit the review was about, so a later round can tell whether the advice was
+already taken.
+
+## Phase 22 — the reviewer sees the code, not a description of it, 0.31.0
+
+The round log now records what was asked and what came back. Next: paste the
+diff the review is about into the same message (bounded, so the column does not
+fill with code), and let a later round say which points were already acted on,
+so the loop stops repeating advice.
+
+## Phase 23 — the drawing half, 0.31.0
+
+A page that never shipped an image still has a canvas or a video frame; take
+that surface as the picture when the page has no file to grab.
+
+## Phases 24–39, condensed
+
+The phases above were written one at a time as the work happened; the run from
+0.32 to 0.34.18 is dense enough that the per-phase prose lives in
+`docs/BENCH.md` and the changelog. What each one changed, in one line:
+
+| Version | Phase | What it moved |
+| --- | --- | --- |
+| 0.32.0 | 22 | the diff a review is about travels with the request |
+| 0.33.0 | 22b | the mirror outlives the visit: claim, takeover, remembered address |
+| 0.33.2 | 22c | a duplicate registration cannot take the pane down |
+| 0.34.0 | 23 | one tab per chat, a UA that does not say "headless", typing that looks like hands |
+| 0.34.1 | 24 | a ref is checked against what the control says before it is used |
+| 0.34.2 | 25 | one chat cannot take another chat's page; a password never comes back in a result |
+| 0.34.3 | 26 | a picture reaches the model as a picture; read state is per chat |
+| 0.34.4 | 27 | an empty mirror says what to do; remembering an address is written, not crashed |
+| 0.34.5 | 28 | one ownership gate for every acting route |
+| 0.34.6 | 29 | one tab per chat even when two calls race |
+| 0.34.7 | 30 | a live cookie is never overwritten by a stale copy |
+| 0.34.8 | 31 | a cold first tab in 0.6 s instead of 21 s |
+| 0.34.9 | 32 | a plugin reload no longer closes the browser |
+| 0.34.10 | 33 | a call log never shows another chat's calls |
+| 0.34.11 | 34 | the column offers "restore the last page" in one click |
+| 0.34.12 | 35 | every write needs the chat that asked for it |
+| 0.34.13 | 36 | a review round is verifiable: each point marked fixed / verified / invalid / open |
+| 0.34.14 | 36b | a reload must not cost the browser: the watchdog follows the application pid |
+| 0.34.15 | 37 | an observation admits its blind spot: pictures, canvases, videos, frames; an error page says so |
+| 0.34.16 | 38 | a reload does not blank the mirror: the owning chat's poll adopts the running browser |
+| 0.34.17 | 39 | the blind-spot frame count follows the screen, not the DOM |
+| 0.34.18 | — | the adoption rule is a named function with a test |
+
+## 页游这一档（2026-10-28 起）
+
+- 操作方式已分叉（0.34.29）：`browser_mode` 每个聊天记一种，`human` / `dom` 单次可覆盖。页游默认 human，登录/表单默认 dom 更省事。
+- Flash 判死：headless Chrome 里 Ruffle 交得上 swf 但起不了画面（26 秒，canvas 0、metadata 0），不承诺。真页游只能走站方 H5 / 微端 / 老 360 浏览器，页面上线前先用 `scripts/probe-flash.mjs` 问一句它要不要 Flash。
+- 开玩前先体检：`browser_prepare` 一次调用给出输入面、帧率、按键是否真的到得了框架、以及 Flash 结论（0.34.30）。4399《怪物小汽车》实测页面 58 fps、框架 59 fps —— 不少「Flash 小游戏」其实已经是 H5，先体检再下结论；`npm run ready <url>` 可复现。
+- 读 SWF 标签流（`npm run swf`）：尺寸、帧率、AVM 版本、资源构成，FWS/CWS 都读，ZWS 明确报不支持（0.34.32）。
+- 空白画面的两种原因分开报（`viewers` / `listening chats` / `asks.log`），并在「浏览器在跑、没人画」时自己向所属会话补发一次开页签请求（0.34.31）。
+
+输入这一层已经到位：按住键、组合键、按住鼠标，一项 300 ms 的按住整次调用 305 ms，事件由页面自己证实。实时画面沿用同一条 CDP 帧路（60 fps、镜像延迟中位数 8 ms）。
+
+还差的一层是「看」：4399 / 7k7k / 奥拉星这类页游把画面画在 canvas 上，文本树里没有可读的东西，判断走位、Boss 血条、按钮位置都要靠像素。本会话的模型没有视觉能力，所以这里的正确做法不是假装能玩，而是把可判读的信号做出来：canvas 局部像素采样、颜色/区域变化率、画面是否在动（`blind` 行已经在报 canvas 数量）。有视觉能力的模型接着用同一套帧路就能玩。
+
+## Where the next round should go
+
+Ranked by what a real task still loses:
+
+1. **End-to-end proof of the places unit tests cannot reach.** The restore-in-one-click
+   path (0.34.11) and the mirror coming back after a reload (0.34.16) were checked by
+   HTTP and by pid, not by a person clicking. A browser-level test that opens the
+   column, kills the driver, and clicks restore is the missing evidence.
+2. **One-click "open here" when this chat has no seat.** `state.chats` can lack this
+   chat, and the column then only tells the model to call `browser_open`. The button
+   belongs in the column.
+3. **Competitor behaviours still unmatched.** Playwright MCP's accessibility refs are
+   ours, and the blind line covers the vision-less read; Stagehand v4's
+   act/observe/extract split with cached selectors is only partly ours, and its
+   self-healing of a selector between rounds has no equivalent here.
+4. **Token numbers for a real session, not a synthetic page.** Reads are measured on
+   fixed pages; an actual 20-step task has never had its cumulative token cost
+   recorded.

package/docs/VERIFY.md

@@ -33,7 +33,7 @@ Expected before any tool call: `"alive":false`, `"profile":"default"`,
 
 ## 2. A tool opens its own page
 
-Ask the model `用 sidecar_open 打开 https://www.bing.com`, or call it yourself.
+Ask the model `用 browser_open 打开 https://www.bing.com`, or call it yourself.
 Expected: the result names the address and title, and this chat's right column
 shows the picture in a tab called **后台浏览器** or **Background browser**.
 `doctor.mjs` now reads `attached, headless`, `viewers 1`, frames climbing.
@@ -43,8 +43,8 @@ shows the picture in a tab called **后台浏览器** or **Background browser**.
 - `"alive":false` with a non-empty `"error"`: Chrome did not start. Check the
   binary at a listed path, and on macOS that the launch carried
   `--use-mock-keychain`.
-- Frames flat while viewers is 1: the screencast stalled. `sidecar_stop`, then
-  `sidecar_open` again.
+- Frames flat while viewers is 1: the screencast stalled. `browser_stop`, then
+  `browser_open` again.
 - `"pointer":null`: no tool input yet; it fills on the first click or move.
 
 ## 3. Only the chat that asked
@@ -60,18 +60,28 @@ A mirror that opens in every chat means the page half is not comparing
 - **Hover** the search field: your cursor turns into an I-beam, the black arrow
   does not move. An arrow that follows the mouse means the pointer paths are
   crossed.
-- **Type** Chinese with `sidecar_type { "text": "…", "submit": true }`: the
+- **Type** Chinese with `browser_type { "text": "…", "submit": true }`: the
   results page appears and the field holds characters, not boxes.
-- **Backspace** with `sidecar_key { "key": "Backspace" }` in a filled field:
+- **Backspace** with `browser_key { "key": "Backspace" }` in a filled field:
   one character goes. Nothing happening means the key reached the document.
-- **Screenshot** with `sidecar_shot {}`: an absolute path under
+- **Screenshot** with `browser_shot {}`: an absolute path under
   `~/.dsh-inner-browser-use/screenshots`. A black picture means the screencast
   was not running when the frame was taken.
+- **The whole keyboard, on this machine**: `npm run keys` starts a real
+  background Chrome, presses all 150 keys into a field, and checks what the page
+  reported for each one (`key`, `code`, `keyCode`, shift) plus what the field
+  typed. It then runs copy / cut / paste / select-all / undo / redo, caret keys,
+  Tab focus, Enter, and a 180 ms hold. One JSON line, exit code 0 when every key
+  matched. CI runs it on macOS and Windows in the `keys` job.
+- **A shortcut by hand**: type `Hi`, press `browser_key { "key": "Mod+a" }`,
+  type `X`: the field holds `X` and nothing before it. `Mod` is Command on macOS
+  and Control elsewhere; `Ctrl+C/X/V/A/Z/Y` and `Ctrl+Shift+Z` run the same
+  editing commands a person would.
 
 ## 5. Stop, then open again
 
-`sidecar_stop` reads `"alive":false` and leaves the profile alone.
-`sidecar_open https://www.bing.com` brings the same profile back, still signed
+`browser_stop` reads `"alive":false` and leaves the profile alone.
+`browser_open https://www.bing.com` brings the same profile back, still signed
 in from the first pass.
 
 Signed out after the restart: Chrome came up on another profile, or was killed
@@ -140,13 +150,13 @@ broke. Two of them cover a cross-site frame, which Chrome runs in its own
 process. Three cover a control under an overlay: the click clears a cookie
 banner and lands (`under clicked`), typing into a field under the same banner
 works, and a modal that offers to delete the account is reported without
-anything being pressed. Two more drive the same overlay through `sidecar_do`:
+anything being pressed. Two more drive the same overlay through `browser_do`:
 a batch step clears the banner before it clicks, and a batch step fills a field
 that was under it. One more counts the recoveries: after the redraw check, the
 state line says at least one ref was followed to its new number. Three cover a
 hand on the mirror: while the person is using it a read still works
 (`hold-allows-read`), an action is refused and names both ways out
-(`hold-refuses-action`), and `sidecar_resume` hands the page back so the same
+(`hold-refuses-action`), and `browser_resume` hands the page back so the same
 click goes through (`resume-lets-go`).
 
 ## Who the mirror serves