dsh-inner-browser-use 0.43.0 → 0.45.0

package/README.md

@@ -20,10 +20,13 @@ live picture of that browser, and your own cursor keeps working on the picture.
 | `browser_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, Insert, arrows, Home, End, PageUp, PageDown, F1-F12, the numeric keypad, letters, digits, punctuation and every shifted symbol, plus ContextMenu, CapsLock, PrintScreen and media keys. `Ctrl`/`Alt`/`Shift`/`Meta`/`Mod` modifiers, chords with `+`, and `Ctrl+C/X/V/A/Z/Y` run the real editing commands. |
 | `browser_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
 | `browser_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
+| `browser_look` | Looks at a page that draws itself — a game, a canvas, a Flash movie — and returns it as a grid of letters, one per block of screen, the same letter for the same colour: 12×8 costs about 200 characters, so a picture costs no image tokens. A second look reports only the cells that changed. `browser_shot` is still the tool for colour, layout and anything a person should see. |
 | `browser_media` | Lists the video and sound the page already fetched, with sizes, the one its own player shows first and marked `main`; `save: <row>`, `save: main` or part of the address keeps one. Ads and beacons stay out unless `all: true`; `watch: false` stops remembering the page. A stream is followed to its pieces and joined, so nothing has to be installed. |
 | `browser_login` | Keeps a sign-in by name: `save` stores this window's cookies and each site's local storage, `list` shows what is kept, `use` signs that account back in and reopens the page, `import` takes a sign-in that is already in Chrome, Brave, Edge or Chromium on this Mac and keeps it under a name, `drop` forgets it. Names survive a restart and belong to the browser rather than to one chat, so two accounts on one site are two names. What sits on disk is encrypted, with the key in the system keychain. |
 | `browser_state` | Reports whether the browser runs, which profile it uses, its address and title, how many columns are showing it, how much memory it holds right now and when it would close on its own if left alone. |
 
+A site you read twice is remembered by origin in `~/.dsh/inner-browser-use/sites.json` (override with `DSH_INNER_BROWSER_STATE_DIR`). After two reads, a site whose page text or control count is big opens in `browser_view` as a `map:` of what is there, and a site whose articles are long has `browser_read` cut at 3,500 characters with a line saying so. Both come with one sentence naming the measurement, and both yield to an explicit `mode`, `query` or `max_chars` — learning changes the default, not the answer to a direct question. `scripts/check-sites.mjs` is the live proof: three views of a 300-button page and three reads of a 12,000-character document, saying where the cut fell.
+
 `browser_read` on the same page is the words only, in Markdown (measured: 4,520
 characters of an about page against 2,987 characters of snapshot — and the
 snapshot spends most of its budget on controls; on a long article the gap grows

package/README.zh.md

@@ -19,10 +19,13 @@ profile，不是产品自带的那个浏览页签：右栏放的是它的画面
 | `browser_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
 | `browser_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
 | `browser_shot` | 存一张 PNG 到 `~/.dsh/inner-browser-use/screenshots`，返回路径；`region: [x, y, w, h]` 只截一块（游戏、图表、看局部）。 |
+| `browser_look` | 看自己画自己的页面（游戏、canvas、Flash）：把画面压成字母网格，一块屏幕一个字母、同色同字母，12×8 约 200 字符，不花图片 token；再看一次只报变了的格子。要颜色、布局或给人看图仍用 `browser_shot`。 |
 | `browser_media` | 列出页面已经请求过的视频和声音（带大小），页面自己播放器里那个排第一、标 `main`；`save: <行号>`、`save: main` 或地址片段存下来。广告和统计默认不出现（`all: true` 才看），`watch: false` 关掉这个页面的记录。流会顺着播放列表把片段取回拼好，不用装任何东西。 |
 | `browser_login` | 按名字留住登录：`save` 存下当前窗口的 cookie 和各站的 localStorage，`list` 看有哪些，`use` 切回那个账号并重开页面，`import` 把本机 Chrome/Brave/Edge 里已经登着的站直接留成一个名字，`drop` 忘掉。名字跟着浏览器走、不跟着一个会话走，重启也在；同一站点两个账号就存两个名字。磁盘上那一份是加密的，钥匙在系统钥匙串里。 |
 | `browser_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题、有几个右栏在看它，以及它此刻占多少内存、没人用的话还有多久关闭。 |
 
+同一个站读两次就会被按 origin 记住（`~/.dsh/inner-browser-use/sites.json`，可用 `DSH_INNER_BROWSER_STATE_DIR` 改位置）。第二次之后：正文或控件多的站，`browser_view` 默认先给 `map:` 地标；文章长的站，`browser_read` 默认截到 3500 字符并说明截在哪。两者都带一句实测数字，也都会被显式 `mode`、`query`、`max_chars` 覆盖——学到的是默认值，不是对直接提问的答案。`scripts/check-sites.mjs` 是现场证据：300 个按钮的页看三次、12000 字的文档读三次，报出截断位置。
+
 点、输入、按键之后的那一行会多一句结果：`· navigated to …`（换页了）、`· content changed`（页内有变化）、`· no visible change`（点了没动）、Flash 页则是 `· screen updated` / `· no frame change`；`read: true` 时不再多说。
 
 `browser_view` 一次给一页文本：

package/docs/ARCHITECTURE.md

@@ -134,7 +134,7 @@ browser without pulling the column away from the person reading it.
 
 ## Tool surface
 
-`sidecarTools()` in `tools.js` builds the 28 definitions and `index.js` registers them on the
+`sidecarTools()` in `tools.js` builds the 39 definitions and `index.js` registers them on the
 plugin's tool scope. All of them return one text block. A page read numbers the controls and
 keeps those numbers across later reads, so `browser_click` by ref means the element the model
 last saw; navigation drops them and the model gets `REF_GONE` rather than a wrong click.
@@ -168,8 +168,12 @@ window) costs one round trip on an animation-heavy page instead of one per frame
 ## Reading for an answer
 
 `browser_read` with a `query` does not return the page; it returns the sentences that answer the
-question. The page-side block reader already returns paragraphs with their heading and link
-density, so the work stays in `read.js` and costs no extra round trip.
+question. The page-side block reader already returns paragraphs with their heading trail
+(`Plans \u203a Trial`), their link density, a 0-1 quality score and table rows as one
+`column: value | column: value` line, so the work stays in `read.js` and costs no extra round trip.
+When the page's own menu would eat the budget, the blocks past the ceiling are chosen by quality and
+then put back in page order, so prose survives where document order would have spent everything
+above it.
 
 Paragraphs are ranked first (BM25 over the question's terms, link-heavy menus down-weighted), then
 split into sentences. Each sentence is scored on its own with the same saturation and length
@@ -177,7 +181,9 @@ normalisation, and that score is multiplied by three priors: the words of its he
 the paragraph it sits in, and inverse link density. Words the page says everywhere are dropped
 before scoring — a term in most sentences cannot point at anything — and a sentence that matches
 only part of the question is scaled by the share of the question's idf weight it actually covers,
-which is what stops a paragraph containing "to" from competing with one containing "refund".
+which is what stops a paragraph containing "to" from competing with one containing "refund". The
+block's quality score and its being a table row are priors as well, so a menu that repeats the
+question's words cannot outrank the sentence that explains them.
 
 Selection is not "the top N scores": each pick subtracts its Jaccard overlap with what is already
 chosen (MMR), selection stops when the best remaining sentence is worth nothing, and picked
@@ -192,6 +198,32 @@ characters) and no text, and the tool turns that into a line saying nothing dire
 question plus those passages. The whole page is still one `browser_view` away, so the expensive
 fallback — a page pasted into the context for a question it never addressed — is gone.
 
+## Looking at a page that draws itself
+
+A canvas, a game and a Flash movie have no text tree and no controls: a snapshot reads
+an empty page and a screenshot costs image tokens on every look. `browser_look` takes
+the captured frame, decodes the PNG inside Node (so no page policy and no browser
+decode step is involved) and prints a grid of palette letters — one letter per block of
+screen, the same letter for the same colour. A 12×8 look costs about 200 characters; a
+second look diffs against the first grid and reports only the cells that moved, plus the
+box they moved in. `browser_prepare` is the way in: it reports the surface, the frame
+rate and whether keys arrive, and says when the page needs the Flash runtime.
+
+## Sites that teach the next read
+
+Two calls into the same origin, the browser knows what it is looking at. A record per
+origin in `sites.json` holds an exponentially drifted average of the page's character
+count, its control count and its article length, updated by every `browser_view` and
+`browser_read` — measured before the budget cuts the output, so a cut does not shrink
+the site on the next visit. Past two reads, a site whose text or control count is big
+opens `browser_view` as the `map:` view instead of a page of numbered controls, and a
+site whose articles are long has `browser_read` land at 3,500 characters and say where
+it cut. Every learned default carries the measurement that produced it, and every one
+yields to what the model asked for: an explicit `mode`, `query` or `max_chars` is used
+as given. The record lives in `~/.dsh/inner-browser-use/sites.json` (override with
+`DSH_INNER_BROWSER_STATE_DIR`), is written at most once every two seconds, and never
+holds page text — only counts.
+
 ## Answering across tabs
 
 A question can be answered by a tab that is not the one on screen. Moving the mirror there and

package/docs/ARTIFACTS.md

@@ -1,35 +1,40 @@
 # 文件落在哪里
 
-浏览器产生的东西——截图、抓下来的图、打印的 PDF——不落在项目根目录，也不散在 home 下。固定规则：
+浏览器产生的东西——截图、抓下来的图、打印的 PDF、存下的视频——落在**当前项目里**，跟会话工作目录走；不在项目根目录，也不散在 home 下。
 
 ```
-~/.dsh-inner-browser-use/artifacts/<项目>/<会话>/<种类>/
+<项目>/.dsh/browser/<会话>/<种类>/
 ```
 
-- **项目**：会话工作目录的文件夹名，加路径哈希前 6 位，例如 `dsh-inner-browser-use-a1b2c3`。同名文件夹、不同路径不会混。
-- **会话**：会话 id 去掉 `session-` 后的前 8 位，例如 `72557d6e`。同一项目里两个会话各存各的。
+- **项目**：会话的工作目录。换项目就换目录，产出跟着项目走，方便直接拷进仓库或用项目里的工具看。
+- **会话**：会话 id 去掉 `session-` 后的前 8 位，例如 `72557d6e`。同一项目里两个会话各存各的，不会互相覆盖。
 - **种类**：`shots/` 是 `browser_shot` 的截图，`pictures/` 是 `browser_grab` 抓的图，`pdf/` 是 `browser_pdf` 打印的文档，`media/` 是 `browser_media` 存下的视频、音频和流。
 
-文件名默认带时间戳（`shot-20261028-124500.png`），给了 `name` 就用它；名字里的分隔符、连续点、开头点都会被去掉，跳不出这个目录。
+`browser_look` 不落文件：它把屏幕读成字母网格直接返回，用来反复看画面变化。
+
+文件名默认带时间戳（`shot-20261028-124500.png`），给了 `name` 就用它；名字里的分隔符、连续点、开头点都会被去掉，跳不出这个目录。`browser_state` 的 `files` 一行给出当前实际落点。
 
 每个种类保留最新 40 个文件，旧的自动删，免得跑一天的会话留下一个没人翻得动的文件夹。开着的文件不会被删掉（尽力删除，失败就留着）。
 
-调用环境里没有项目或会话信息时，落到 `no-project/no-chat/<种类>/`，不猜。
+`<项目>/.dsh/browser/.gitignore` 写一行 `*`，所以截图不会进 `git status`；这个文件只在项目里写。调用环境里没有项目信息时，落到 `~/.dsh-inner-browser-use/artifacts/no-project/`，不猜，也不写 `.gitignore`。
+
+`DSH_INNER_BROWSER_ARTIFACTS` 能把整个落点换到一个固定目录（测试用，也方便把产出收到别处），此时不写 `.gitignore`。
 
 ## 整个目录
 
 ```
+<项目>/.dsh/browser/                 截图、抓图、PDF、媒体（见上）
 ~/.dsh-inner-browser-use/
-  artifacts/<项目>/<会话>/<种类>/   给人看的产出，只在这里（见上）
-  chrome/                          浏览器自己的数据：登录态、Cookie、缓存
-  logins/                          按名字留住的登录，一个名字一个文件，加密
-  vault.key                        钥匙串拿不到钥匙时的替代文件，只有本人可读
-  flash/                           Flash 播放器运行时，第一次用 flash:true 才装
-  browser.log                     最近一次浏览器启动和页面记录
-  asks.log                        人问过什么、答了什么
+  artifacts/no-project/              没有项目时的产出
+  chrome/                            浏览器自己的数据：登录态、Cookie、缓存
+  logins/                            按名字留住的登录，一个名字一个文件，加密
+  vault.key                          钥匙串拿不到钥匙时的替代文件，只有本人可读
+  flash/                             Flash 播放器运行时，第一次用 flash:true 才装
+  browser.log                        最近一次浏览器启动和页面记录
+  asks.log                           人问过什么、答了什么
   tabs.json / mirror.json / flash.json / watcher.json   上次开着哪些页
 ```
 
-`chrome/session-cookies.json` 是 Chrome 退出时会丢的会话 cookie，加密后存在 profile 旁边；`logins/` 与 `vault.key` 由 `browser_login` 和启动时自动保存的登录备份共用，不要手动改或删，要清就用 `browser_login({action:"drop"})`。`chrome/` 其余部分是浏览器进程自己在读写，不要手动改；要清登录态，用 `browser_open` 的档位或插件自己的入口，别删目录里的某个文件。上面这些都在一个根目录下，卸载插件前想留产出就拷 `artifacts/`。
+`chrome/session-cookies.json` 是 Chrome 退出时会丢的会话 cookie，加密后存在 profile 旁边；`logins/` 与 `vault.key` 由 `browser_login` 和启动时自动保存的登录备份共用，不要手动改或删，要清就用 `browser_login({action:"drop"})`。`chrome/` 其余部分是浏览器进程自己在读写，不要手动改；要清登录态，用 `browser_open` 的档位或插件自己的入口，别删目录里的某个文件。
 
 测试里用 `DSH_INNER_BROWSER_ARTIFACTS` 指向临时目录，不写真实 home。

package/docs/BENCH.md

@@ -1,3 +1,33 @@
+## 读过的站第二次更便宜（0.45.0，本地页面，`scripts/check-sites.mjs`）
+
+同一台机器、无头 Chrome、两个本地 origin。第一次走老路（整页读），第二次之后用学到的默认值。
+所有数字来自一次真实运行，不是估算。
+
+| 页面 | 第一次 | 学到之后 | 省下 | 说明 |
+| --- | --- | --- | --- | --- |
+| 300 个按钮的页面，`browser_view` | 1801 字符 | 760 字符（`map: 90 controls`） | 58% | 两次之后自动改成地标视图，并说明「this site reads big」 |
+| 12000 字文档，`browser_read` | 6096 字符（满预算） | 3726 字符（截到 3500） | 39% | 截断处自带 `cut at 3500 characters`，用 `max_chars` 可要全文 |
+
+- 学的尺子量的是页面本身，不是这次输出：`read.js` 的 `chars` 在截断前取值，所以第一次就把整篇长度记下来，不会被 3500 的预算越读越小
+- 显式 `mode` / `query` / `max_chars` 一律优先；学到的是默认值，附一句带数字的理由
+- 记录写在 `sites.json`，每进程最多每 2 秒一次；两次运行之间只留计数，不留正文
+- `scripts/check-look.mjs`：12×8 的字母网格，三次看共 639 字符，全程没有图片 token
+- `scripts/soak.mjs`：无头浏览器 24 轮开页/读页/截图/重读，34.9 秒，进程树内存 1440 → 1287 MB（−153 MB），页数始终 1，8 次截图、24 次读；没有页签或内存随轮数上涨
+
+## 读页先读正文（0.44.0，本地页面，`scripts/bench.mjs`）
+
+同一台机器、同一档预算。旧法是按页面顺序把预算灌满的整页读。
+
+| 页面 | 旧法字符 | 新法字符 | 省下 | 答案 |
+| --- | --- | --- | --- | --- |
+| 九成是菜单、答案在正文（900 字预算） | 1500 | 88 | 94% | 旧法读不到，新法读到 |
+| 定价表（900 字预算） | 679 | 109 | 84% | 价格与列名同一行 |
+
+- `read-noise` 2 ms、`read-table` 2 ms，两项预算 300 ms 全过（26 项全过）
+- 表格行形如 `Plan: Pro | Price per month: $84`，列名来自表头，一行就是一条记录
+- 标题路径 `Plans \u203a Trial` 印在证据上方；与最近标题相同、或超过 90 字时退回最近标题
+- 既有数字没有回退：`read-answer` 90 字 / 92%、`read-miss` 438 字 / 79%、`read-tabs` 90 字 / 90%（与 0.43.0 相同）
+
 ## 一次读穿过登录墙（0.43.0，本地页面，`scripts/bench.mjs`）
 
 同一页：带 cookie 时是账户页，不带时是一个只有密码框的登录页。留一个登录（vault），清掉页面的 cookie 与 localStorage 后读它。

package/docs/FLASH.md

@@ -56,6 +56,16 @@ Electron 11 的两个真实限制，代码里已按它们写：
 - 它不会用当前 Chrome 的 profile 目录（会降级并上锁），Flash 分支固定用 `<profile>-flash` 独立目录；两边登录态各自独立。
 - `/json/new` 在 Chromium 87 上是 404，单窗口应用也没有第二个窗口，所以 Flash 分支不建新页签：会话要页签时把唯一那个窗口导航过去。
 
+### 起来之后还要证明它真在
+
+端口能连上不等于插件装上了：`/json/version` 由浏览器进程回答，插件加载失败时它照样 200，页面只是画出一块黑。所以启动流程在端口就绪之后还问页面一句 `navigator.plugins`，只认名字里带 `Shockwave Flash` 的那条，结果存成浏览器自己的状态：
+
+- `browser_open` 在正文里给一行 `flash runtime: plugin loaded (Shockwave Flash)`，或者 `open, but no Flash plugin is loaded — movies will not play; run scripts/flash-setup.mjs`；还没问到时写 `plugin not checked yet`，不猜。
+- `browser_state` 的运行时行把「模式」和「插件」分开：`Flash-era runtime — plugin loaded (…)` / `— no Flash plugin is loaded`。
+- `browser_prepare` 在真的找到 movie、但插件缺失时补一句「它们跑不起来」，免得把黑帧当成游戏还在加载。
+
+问不到（页面崩溃、CDP 早断）时记 `null`——没有证据就不下结论，这跟「没装运行时」是两回事：后者在启动阶段就抛 `sidecar: no Flash runtime on this machine`。
+
 实测数字（Apple 芯片 + Rosetta，1440×900）：
 
 | 指标 | 值 |

package/docs/VERIFY.md

@@ -97,6 +97,47 @@ still hold the cookies.
   内置浏览器 means the page half is stale: reload the GUI page.
 - Guest-era logins are not migrated; sign in once in the new profile.
 
+## 7. A page that draws itself
+
+```bash
+node scripts/check-look.mjs
+```
+
+Headless Chrome paints a page, `browser_look` returns a 12×8 letter grid, the page
+paints a red square, and the second look names only the cells that moved. Expected:
+`three looks cost 639 characters and no picture`. A look that returns words instead
+of a grid, or a diff of the whole screen, means the PNG was not decoded locally.
+
+## 8. The next visit is cheaper
+
+```bash
+node scripts/check-sites.mjs
+```
+
+Two local origins: a page of 300 buttons and a 12,000-character document. Expected,
+in order: the first view is the page, the second is still the page (one read is not a
+lesson), the third opens as `map: 90 controls` and says `this site reads big (2 read(s),
+~2946 chars of page, ~90 controls)`; the first document read is the whole thing (6,096
+characters), a forced second read is the whole thing again, and the third comes back
+`cut at 3500 characters` with `this site reads long (2 read(s), ~6000 chars of text)`.
+The run ends with `sites: ok` and leaves `sites.json` in a temporary directory.
+
+A third view that is still the page, or a third read that is not cut, means the record
+is written after the default is chosen, or the size recorded is the output's rather
+than the page's: a cut must not shrink the site for the next visit.
+
+## 9. It does not grow
+
+```bash
+npm run soak
+```
+
+Twenty-four cycles of open, read, snapshot, re-read through one headless browser.
+Expected: `soak: ok · 24 cycles` with `pages 1` and memory lower at the end than
+at the start. Memory climbing with the cycle count, or a second page appearing,
+means something is kept per read rather than per page. The run is bounded on
+purpose — under a minute, no internet.
+
 ## Live
 
 `node scripts/live.mjs` does the same work on the open internet, with no local

package/index.js

@@ -947,7 +947,7 @@ export function apply(ctx) {
         '页面很长、只想确认某句话/按钮在哪时用 `browser_find`，别整页读。要读长文（文章、文档、issue、论坛帖）用 `browser_read`：它回 Markdown，只花几十分之一的 token，链接可点；要动手点、填表才用 `browser_view` 拿编号。',
         '要保存页面样子用 `browser_shot`，它会给出 PNG 绝对路径；要交给人看的文件，把绝对路径写进回复。',
         '页面放视频或音乐、人想留下它时用 `browser_media`：它列的是页面自己已经请求过的媒体（带大小），页面自己播放器里那个标 `main` 排第一，广告和统计请求默认不出现（`all:true` 才看），`save: main` 存第一个；视频是流的话会顺着播放列表把片段取回拼成整段，不用装 yt-dlp 或 ffmpeg。先让页面播一下再列，播放请求才是文件。不想要这个页面的记录就 `watch:false`。',
-        '游戏、canvas、Flash：画面自己画的地方，`browser_view` 和 `browser_read` 读不出里面是什么。先 `browser_prepare` 拿画面位置、帧率、按键是否到达和下一步；之后看 `browser_shot` 的图定位置，按坐标点、按住、按组合键，再拍一张看画面变了没有——变了就是这一步生效了。当年用 Flash 做的游戏用 `browser_open` 带 `flash:true` 在栏内跑，不装播放器、不开窗口；Flash 默认关，`flash:true` 开、`flash:false` 回到普通 Chrome，两边都只在本会话右栏。',
+        '游戏、canvas、Flash：画面自己画的地方，`browser_view` 和 `browser_read` 读不出里面是什么。先 `browser_prepare` 拿画面位置、帧率、按键是否到达和下一步；之后用 `browser_look` 读屏：它把画面压成字母网格、点完再看一次只报变了哪块，不花图片 token；要定位置或看颜色再 `browser_shot`。按坐标点、按住、按组合键后先 `browser_look` 确认这一步生效，别每步截图。当年用 Flash 做的游戏用 `browser_open` 带 `flash:true` 在栏内跑，不装播放器、不开窗口；Flash 默认关，`flash:true` 开、`flash:false` 回到普通 Chrome，两边都只在本会话右栏。',
       ].join('\n'),
     }), 'dsh-inner-browser-use: system prompt section')
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-inner-browser-use",
-  "version": "0.43.0",
+  "version": "0.45.0",
   "description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
   "type": "module",
   "main": "index.js",
@@ -18,6 +18,7 @@
     "read.js",
     "review.js",
     "media.js",
+    "pixels.js",
     "vault.js",
     "chromecookies.js",
     "client.js",
@@ -29,6 +30,9 @@
     "scripts/check-flash-mirror.mjs",
     "scripts/check-flash-local.mjs",
     "scripts/check-flash-tools.mjs",
+    "scripts/check-look.mjs",
+    "scripts/check-sites.mjs",
+    "scripts/soak.mjs",
     "runtime/",
     "docs/",
     "README.md",
@@ -51,7 +55,7 @@
     "dsh": ">=0.2.0"
   },
   "scripts": {
-    "test": "node --test test/artifacts.test.mjs test/client.test.mjs test/cookies.test.mjs test/diff.test.mjs test/fresh.test.mjs test/host.test.mjs test/input.test.mjs test/isolation.test.mjs test/keyboard.test.mjs test/leak.test.mjs test/login.test.mjs test/media.test.mjs test/no-duplicate.test.mjs test/package.test.mjs test/readfault.test.mjs test/refs.test.mjs test/review.test.mjs test/session.test.mjs test/settle.test.mjs test/sidecar.test.mjs test/swf.test.mjs test/tabs.test.mjs test/trouble.test.mjs test/vault.test.mjs",
+    "test": "node --test test/artifacts.test.mjs test/client.test.mjs test/cookies.test.mjs test/diff.test.mjs test/fresh.test.mjs test/host.test.mjs test/input.test.mjs test/isolation.test.mjs test/keyboard.test.mjs test/leak.test.mjs test/login.test.mjs test/media.test.mjs test/no-duplicate.test.mjs test/package.test.mjs test/blocks.test.mjs test/readfault.test.mjs test/refs.test.mjs test/review.test.mjs test/session.test.mjs test/settle.test.mjs test/sidecar.test.mjs test/swf.test.mjs test/tabs.test.mjs test/trouble.test.mjs test/vault.test.mjs",
     "doctor": "node scripts/doctor.mjs",
     "breadth": "node scripts/breadth.mjs",
     "live": "node scripts/live.mjs",
@@ -65,7 +69,10 @@
     "flash-mirror": "node scripts/check-flash-mirror.mjs",
     "flash-tools": "node scripts/check-flash-tools.mjs",
     "flash-local": "node scripts/check-flash-local.mjs",
-    "keys": "node scripts/keyboard.mjs"
+    "keys": "node scripts/keyboard.mjs",
+    "look": "node scripts/check-look.mjs",
+    "sites": "node scripts/check-sites.mjs",
+    "soak": "node scripts/soak.mjs"
   },
   "license": "MIT",
   "author": {

package/pixels.js

@@ -0,0 +1,217 @@
+/**
+ * The screen without the picture: a page that draws itself — a game, a canvas
+ * app, a Flash movie — has nothing a text tree can read, and a screenshot costs
+ * image tokens on every step. This turns the screen into a small grid of
+ * characters instead: one letter per block of pixels, the same letters for the
+ * same colours, so a caller can see what changed by reading text.
+ *
+ * Everything here is pure: decode, average, quantise, diff. No browser, no
+ * files, so it can be tested without one.
+ */
+
+import { inflateSync } from 'node:zlib'
+
+/** Colours a screen block can be called, most common things first. */
+export const PALETTE = [
+  { char: '.', name: 'black', rgb: [16, 16, 16] },
+  { char: ':', name: 'dark', rgb: [64, 64, 64] },
+  { char: '-', name: 'gray', rgb: [112, 112, 112] },
+  { char: '+', name: 'light', rgb: [168, 168, 168] },
+  { char: '*', name: 'bright', rgb: [216, 216, 216] },
+  { char: '#', name: 'white', rgb: [248, 248, 248] },
+  { char: 'r', name: 'red', rgb: [208, 48, 48] },
+  { char: 'g', name: 'green', rgb: [64, 176, 72] },
+  { char: 'b', name: 'blue', rgb: [56, 104, 216] },
+  { char: 'y', name: 'yellow', rgb: [232, 208, 72] },
+  { char: 'c', name: 'cyan', rgb: [64, 200, 208] },
+  { char: 'm', name: 'magenta', rgb: [200, 72, 200] },
+  { char: 'o', name: 'orange', rgb: [224, 136, 48] },
+  { char: 'p', name: 'purple', rgb: [112, 56, 168] },
+  { char: 'n', name: 'brown', rgb: [136, 88, 48] },
+]
+
+/**
+ * The letter closest to a colour. Grays and bright colours are judged by eye
+ * first so a dark sky does not read as blue and a red button does not read as
+ * brown.
+ * @param {number} r - red 0..255.
+ * @param {number} g - green 0..255.
+ * @param {number} b - blue 0..255.
+ * @returns {string} one character.
+ */
+export function quantize(r, g, b) {
+  /* Red, green and blue are counted more heavily than a plain euclidean
+     distance would: two colours a person calls different should not collapse
+     into the same letter. */
+  const red = r * 1.30
+  const green = g * 1.15
+  const blue = b * 0.90
+  let best = PALETTE[0].char
+  let near = Infinity
+  for (const entry of PALETTE) {
+    const dr = red - entry.rgb[0] * 1.30
+    const dg = green - entry.rgb[1] * 1.15
+    const db = blue - entry.rgb[2] * 0.90
+    const d = dr * dr + dg * dg + db * db
+    if (d < near) {
+      near = d
+      best = entry.char
+    }
+  }
+  return best
+}
+
+/**
+ * Read a PNG made by Chrome into plain pixels. Chrome writes 8-bit
+ * non-interlaced images, RGB or RGBA; anything else is refused with a reason
+ * rather than a guess.
+ * @param {Buffer} file - the PNG bytes.
+ * @returns {{width: number, height: number, data: Uint8Array, channels: number}} pixels, row-major.
+ */
+export function decodePng(file) {
+  const bytes = new Uint8Array(file.buffer !== undefined && file.byteOffset === 0 && file.byteLength === file.buffer.byteLength ? file.buffer : file)
+  const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
+  const signature = [137, 80, 78, 71, 13, 10, 26, 10]
+  for (let i = 0; i < signature.length; i += 1) {
+    if (bytes[i] !== signature[i]) throw new Error('pixels: not a PNG')
+  }
+  let width = 0
+  let height = 0
+  let depth = 0
+  let type = 0
+  let interlace = 0
+  const chunks = []
+  let at = 8
+  while (at + 8 <= bytes.length) {
+    const length = view.getUint32(at)
+    const name = String.fromCharCode(bytes[at + 4], bytes[at + 5], bytes[at + 6], bytes[at + 7])
+    const body = bytes.subarray(at + 8, at + 8 + length)
+    if (name === 'IHDR') {
+      width = view.getUint32(at + 8)
+      height = view.getUint32(at + 12)
+      depth = body[8]
+      type = body[9]
+      interlace = body[12]
+    } else if (name === 'IDAT') chunks.push(body)
+    else if (name === 'IEND') break
+    at += 12 + length
+  }
+  if (depth !== 8) throw new Error(`pixels: PNG depth ${depth} is not 8`)
+  const channels = type === 6 ? 4 : type === 2 ? 3 : type === 0 ? 1 : 0
+  if (channels === 0) throw new Error(`pixels: PNG colour type ${type} is not RGB`)
+  if (interlace !== 0) throw new Error('pixels: interlaced PNG is not supported')
+  const packed = inflateSync(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk))))
+  const stride = width * channels
+  const out = new Uint8Array(width * height * channels)
+  let raw = 0
+  for (let y = 0; y < height; y += 1) {
+    const filter = packed[raw]
+    raw += 1
+    const row = out.subarray(y * stride, (y + 1) * stride)
+    const up = y === 0 ? null : out.subarray((y - 1) * stride, y * stride)
+    for (let x = 0; x < stride; x += 1) {
+      const value = packed[raw + x]
+      const left = x >= channels ? row[x - channels] : 0
+      const above = up === null ? 0 : up[x]
+      const corner = up !== null && x >= channels ? up[x - channels] : 0
+      let said = value
+      if (filter === 1) said = value + left
+      else if (filter === 2) said = value + above
+      else if (filter === 3) said = value + ((left + above) >> 1)
+      else if (filter === 4) {
+        const guess = left + above - corner
+        const dl = Math.abs(guess - left)
+        const du = Math.abs(guess - above)
+        const dc = Math.abs(guess - corner)
+        said = value + (dl <= du && dl <= dc ? left : du <= dc ? above : corner)
+      } else if (filter !== 0) throw new Error(`pixels: PNG filter ${filter} is not known`)
+      row[x] = said & 0xff
+    }
+    raw += stride
+  }
+  return { width, height, data: out, channels }
+}
+
+/**
+ * Average each block of the picture into one character.
+ * @param {{width: number, height: number, data: Uint8Array, channels: number}} png - decoded picture.
+ * @param {number} cols - cells across.
+ * @returns {{rows: string[], cols: number, rowCount: number, blockW: number, blockH: number}} the grid.
+ */
+export function gridOf(png, cols) {
+  const across = Math.min(64, Math.max(4, Math.round(cols)))
+  const rows = Math.min(64, Math.max(2, Math.round((across * png.height) / Math.max(1, png.width))))
+  const blockW = png.width / across
+  const blockH = png.height / rows
+  const lines = []
+  let seen = ''
+  for (let ry = 0; ry < rows; ry += 1) {
+    let line = ''
+    for (let rx = 0; rx < across; rx += 1) {
+      const x0 = Math.floor(rx * blockW)
+      const x1 = Math.max(x0 + 1, Math.min(png.width, Math.floor((rx + 1) * blockW)))
+      const y0 = Math.floor(ry * blockH)
+      const y1 = Math.max(y0 + 1, Math.min(png.height, Math.floor((ry + 1) * blockH)))
+      let r = 0
+      let g = 0
+      let b = 0
+      let n = 0
+      for (let y = y0; y < y1; y += 1) {
+        for (let x = x0; x < x1; x += 1) {
+          const at = (y * png.width + x) * png.channels
+          r += png.data[at]
+          g += png.data[at + 1] ?? png.data[at]
+          b += png.data[at + 2] ?? png.data[at]
+          n += 1
+        }
+      }
+      line += quantize(r / n, g / n, b / n)
+    }
+    lines.push(line)
+    seen += line
+  }
+  return { rows: lines, cols: across, rowCount: rows, blockW, blockH }
+}
+
+/**
+ * What changed between two grids of the same screen.
+ * @param {string[]} before - earlier rows.
+ * @param {string[]} after - later rows.
+ * @returns {{cells: number, box: {x: number, y: number, w: number, h: number} | null}} the change.
+ */
+export function diffGrid(before, after) {
+  if (before.length !== after.length || before.some((row, i) => row.length !== after[i].length)) {
+    return { cells: Number.POSITIVE_INFINITY, box: null }
+  }
+  let cells = 0
+  let x0 = Infinity
+  let y0 = Infinity
+  let x1 = -1
+  let y1 = -1
+  for (let y = 0; y < after.length; y += 1) {
+    for (let x = 0; x < after[y].length; x += 1) {
+      if (before[y][x] === after[y][x]) continue
+      cells += 1
+      if (x < x0) x0 = x
+      if (y < y0) y0 = y
+      if (x > x1) x1 = x
+      if (y > y1) y1 = y
+    }
+  }
+  if (cells === 0) return { cells: 0, box: null }
+  return { cells, box: { x: x0, y: y0, w: x1 - x0 + 1, h: y1 - y0 + 1 } }
+}
+
+/**
+ * Cut one box out of a grid, with a cell of margin around it.
+ * @param {string[]} rows - the whole grid.
+ * @param {{x: number, y: number, w: number, h: number}} box - changed cells.
+ * @returns {{rows: string[], x: number, y: number}} the cut grid and where it starts.
+ */
+export function around(rows, box) {
+  const x = Math.max(0, box.x - 1)
+  const y = Math.max(0, box.y - 1)
+  const to = Math.min(rows[0].length, box.x + box.w + 1)
+  const down = Math.min(rows.length, box.y + box.h + 1)
+  return { rows: rows.slice(y, down).map((row) => row.slice(x, to)), x, y }
+}