dsh-inner-browser-use 0.35.2 → 0.36.0

package/README.md

@@ -20,6 +20,7 @@ live picture of that browser, and your own cursor keeps working on the picture.
 | `browser_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, Insert, arrows, Home, End, PageUp, PageDown, F1-F12, the numeric keypad, letters, digits, punctuation and every shifted symbol, plus ContextMenu, CapsLock, PrintScreen and media keys. `Ctrl`/`Alt`/`Shift`/`Meta`/`Mod` modifiers, chords with `+`, and `Ctrl+C/X/V/A/Z/Y` run the real editing commands. |
 | `browser_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
 | `browser_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
+| `browser_media` | Lists the video and sound the page already fetched, with sizes; `save: <row>` or part of the address keeps one. A stream is followed to its pieces and joined, so nothing has to be installed. |
 | `browser_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
 
 `browser_read` on the same page is the words only, in Markdown (measured: 4,520

package/README.zh.md

@@ -19,6 +19,7 @@ profile，不是产品自带的那个浏览页签：右栏放的是它的画面
 | `browser_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
 | `browser_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
 | `browser_shot` | 存一张 PNG 到 `~/.dsh/inner-browser-use/screenshots`，返回路径；`region: [x, y, w, h]` 只截一块（游戏、图表、看局部）。 |
+| `browser_media` | 列出页面已经请求过的视频和声音（带大小），`save: <行号>` 或地址片段存下来；流会顺着播放列表把片段取回拼好，不用装任何东西。 |
 | `browser_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
 
 点、输入、按键之后的那一行会多一句结果：`· navigated to …`（换页了）、`· content changed`（页内有变化）、`· no visible change`（点了没动）、Flash 页则是 `· screen updated` / `· no frame change`；`read: true` 时不再多说。

package/docs/ARTIFACTS.md

@@ -8,7 +8,7 @@
 
 - **项目**：会话工作目录的文件夹名，加路径哈希前 6 位，例如 `dsh-inner-browser-use-a1b2c3`。同名文件夹、不同路径不会混。
 - **会话**：会话 id 去掉 `session-` 后的前 8 位，例如 `72557d6e`。同一项目里两个会话各存各的。
-- **种类**：`shots/` 是 `browser_shot` 的截图，`pictures/` 是 `browser_grab` 抓的图，`pdf/` 是 `browser_pdf` 打印的文档。
+- **种类**：`shots/` 是 `browser_shot` 的截图，`pictures/` 是 `browser_grab` 抓的图，`pdf/` 是 `browser_pdf` 打印的文档，`media/` 是 `browser_media` 存下的视频、音频和流。
 
 文件名默认带时间戳（`shot-20261028-124500.png`），给了 `name` 就用它；名字里的分隔符、连续点、开头点都会被去掉，跳不出这个目录。
 

package/index.js

@@ -944,7 +944,8 @@ export function apply(ctx) {
         '先接着用已有的页：`browser_view`（拿编号控件与可见文字）→ `browser_click` / `browser_type` → 再 `browser_view` 直到出现要的结果；页面要等就重复读，不 sleep。`browser_open` 只用于没有页、要换新地址、或要明确刷新；同一个地址不要反复重开。',
         '页面很长、只想确认某句话/按钮在哪时用 `browser_find`，别整页读。要读长文（文章、文档、issue、论坛帖）用 `browser_read`：它回 Markdown，只花几十分之一的 token，链接可点；要动手点、填表才用 `browser_view` 拿编号。',
         '要保存页面样子用 `browser_shot`，它会给出 PNG 绝对路径；要交给人看的文件，把绝对路径写进回复。',
-        '游戏、canvas、Flash：画面自己画的地方，`browser_view` 和 `browser_read` 读不出里面是什么。先 `browser_prepare` 拿画面位置、帧率、按键是否到达和下一步；之后看 `browser_shot` 的图定位置，按坐标点、按住、按组合键，再拍一张看画面变了没有——变了就是这一步生效了。当年用 Flash 做的游戏用 `browser_open` 带 `flash:true` 在栏内跑，不装播放器、不开窗口。',
+        '页面放视频或音乐、人想留下它时用 `browser_media`：它列的是页面自己已经请求过的媒体（带大小），用行号或地址片段存一个；视频是流的话会顺着播放列表把片段取回拼成整段，不用装 yt-dlp 或 ffmpeg。先让页面播一下再列，播放请求才是文件。',
+        '游戏、canvas、Flash：画面自己画的地方，`browser_view` 和 `browser_read` 读不出里面是什么。先 `browser_prepare` 拿画面位置、帧率、按键是否到达和下一步；之后看 `browser_shot` 的图定位置，按坐标点、按住、按组合键，再拍一张看画面变了没有——变了就是这一步生效了。当年用 Flash 做的游戏用 `browser_open` 带 `flash:true` 在栏内跑，不装播放器、不开窗口；Flash 默认关，`flash:true` 开、`flash:false` 回到普通 Chrome，两边都只在本会话右栏。',
       ].join('\n'),
     }), 'dsh-inner-browser-use: system prompt section')
   }

package/media.js

@@ -0,0 +1,239 @@
+/**
+ * Media a page plays, and the way to the file behind it.
+ *
+ * yt-dlp solves this by teaching itself every site. The page already knows
+ * better than any list: it asked for the file, and this browser saw the
+ * request. So the sniffing half is free — the network log this plugin already
+ * keeps says which URL is the movie — and the rest is playlist arithmetic:
+ * a master playlist names its variants, a media playlist names its segments,
+ * and the segments are plain HTTP with the cookies the page already has.
+ *
+ * Nothing here touches the disk or the network: parsing and planning are pure,
+ * so they are the part that can be tested without a page.
+ */
+
+/** A mime that means "this is the movie itself", not a page about it. */
+const FILE_MIME = /^(video|audio)\/|^application\/(vnd\.apple\.mpegurl|x-mpegurl|dash\+xml|octet-stream)$/i
+
+/** A URL that ends like a file a player would stream. */
+const FILE_PATH = /\.(m3u8|mpd|mp4|m4v|webm|mkv|mov|flv|ts|m4s|mp3|m4a|aac|ogg|opus|wav)(\?|#|$)/i
+
+/** Same-origin noise that sniffing must not report as a movie. */
+const NOT_MEDIA = /\.(m3u8|mpd|mp4|webm|mov|flv|ts|m4s|mp3|m4a|aac|ogg|opus|wav)\b|^data:|^blob:/i
+
+/**
+ * Is this request something a person would call a video or a song?
+ * @param {string} url - request URL.
+ * @param {string} [mime] - response mime, when known.
+ * @param {string} [type] - CDP resource type, when known.
+ * @returns {boolean} true when it is worth listing.
+ */
+export function isMedia(url, mime = '', type = '') {
+  const at = String(url ?? '')
+  const said = String(mime ?? '').toLowerCase()
+  if (at === '' || at.startsWith('data:') || at.startsWith('blob:')) return false
+  if (said.startsWith('image/') || said.startsWith('font/') || said.startsWith('text/html') || said.includes('javascript') || said.includes('css')) return false
+  if (said === 'application/json' || said.startsWith('application/json')) return false
+  if (FILE_MIME.test(said)) return true
+  if (String(type ?? '') === 'Media') return true
+  return FILE_PATH.test(at) && !/\.(html?|php|aspx?)$/i.test(at)
+}
+
+/**
+ * Which word a person would use for it.
+ * @param {string} url - request URL.
+ * @param {string} [mime] - response mime.
+ * @returns {'video'|'audio'|'stream'|'file'} the kind.
+ */
+export function kindOf(url, mime = '') {
+  const said = String(mime ?? '').toLowerCase()
+  const at = String(url ?? '')
+  if (/mpegurl|dash\+xml/.test(said) || /\.(m3u8|mpd)(\?|#|$)/i.test(at)) return 'stream'
+  if (said.startsWith('audio/')) return 'audio'
+  if (/\.(mp3|m4a|aac|ogg|opus|wav)(\?|#|$)/i.test(at)) return 'audio'
+  if (said.startsWith('video/') || /\.(mp4|m4v|webm|mkv|mov|flv|ts|m4s)(\?|#|$)/i.test(at)) return 'video'
+  return 'file'
+}
+
+/**
+ * Turn the relative parts of a playlist into addresses that can be fetched.
+ * @param {string} base - the playlist's own URL.
+ * @param {string} href - a line from it.
+ * @returns {string} absolute URL, or '' when it is a comment.
+ */
+export function absoluteUrl(base, href) {
+  const line = String(href ?? '').trim()
+  if (line === '' || line.startsWith('#')) return ''
+  try {
+    return new URL(line, base).toString()
+  } catch {
+    return ''
+  }
+}
+
+/**
+ * Read the attributes on one `#EXT-X-STREAM-INF:` line.
+ * @param {string} line - the tag line.
+ * @returns {{bandwidth: number, width: number, height: number, codecs: string}} what it says.
+ */
+export function streamInfo(line) {
+  const text = String(line ?? '')
+  const number = (name) => {
+    const found = new RegExp(`${name}=(\\d+)`, 'i').exec(text)
+    return found === null ? 0 : Number(found[1])
+  }
+  const size = /RESOLUTION=(\d+)x(\d+)/i.exec(text)
+  const codecs = /CODECS="([^"]*)"/i.exec(text)
+  return {
+    bandwidth: number('BANDWIDTH') || number('AVERAGE-BANDWIDTH'),
+    width: size === null ? 0 : Number(size[1]),
+    height: size === null ? 0 : Number(size[2]),
+    codecs: codecs === null ? '' : String(codecs[1]).slice(0, 40),
+  }
+}
+
+/**
+ * A master playlist is a menu of the same movie at different sizes.
+ * @param {string} text - playlist body.
+ * @param {string} base - playlist URL.
+ * @returns {Array<{bandwidth: number, width: number, height: number, codecs: string, url: string}>} variants, best first.
+ */
+export function parseMaster(text, base) {
+  const lines = String(text ?? '').split(/\r?\n/)
+  const variants = []
+  for (let i = 0; i < lines.length; i += 1) {
+    const line = lines[i].trim()
+    if (!line.toUpperCase().startsWith('#EXT-X-STREAM-INF')) continue
+    const info = streamInfo(line)
+    let at = i + 1
+    while (at < lines.length && (lines[at].trim() === '' || lines[at].trim().startsWith('#'))) at += 1
+    const url = at < lines.length ? absoluteUrl(base, lines[at]) : ''
+    if (url === '') continue
+    variants.push({ ...info, url })
+    i = at
+  }
+  variants.sort((left, right) => (right.bandwidth || right.height) - (left.bandwidth || left.height))
+  return variants
+}
+
+/**
+ * A media playlist is the movie cut into pieces.
+ * @param {string} text - playlist body.
+ * @param {string} base - playlist URL.
+ * @returns {{segments: string[], init: string, duration: number, encrypted: boolean}} the pieces and what they need first.
+ */
+export function parsePlaylist(text, base) {
+  const lines = String(text ?? '').split(/\r?\n/)
+  const segments = []
+  let init = ''
+  let duration = 0
+  let encrypted = false
+  for (const raw of lines) {
+    const line = raw.trim()
+    if (line === '') continue
+    if (line.startsWith('#EXTINF:')) duration += Number.parseFloat(line.slice(8)) || 0
+    else if (line.toUpperCase().startsWith('#EXT-X-MAP')) {
+      const found = /URI="([^"]+)"/i.exec(line)
+      if (found !== null) init = absoluteUrl(base, found[1])
+    } else if (line.toUpperCase().startsWith('#EXT-X-KEY')) {
+      if (!/METHOD=NONE/i.test(line)) encrypted = true
+    } else if (!line.startsWith('#')) {
+      const url = absoluteUrl(base, line)
+      if (url !== '') segments.push(url)
+    }
+  }
+  return { segments, init, duration: Math.round(duration), encrypted }
+}
+
+/**
+ * Which playlist this body is.
+ * @param {string} text - playlist body.
+ * @returns {'master'|'media'|null} the kind.
+ */
+export function playlistKind(text) {
+  const body = String(text ?? '')
+  if (!body.trimStart().startsWith('#EXTM3U')) return null
+  if (/#EXT-X-STREAM-INF/i.test(body)) return 'master'
+  return 'media'
+}
+
+/**
+ * The one variant a person would want: tall enough to be worth watching,
+ * cheap enough to fetch. A high one is never the answer to "save this".
+ * @param {Array<{bandwidth: number, width: number, height: number, url: string}>} variants - from parseMaster.
+ * @param {{height?: number, bandwidth?: number}} [wanted] - caps.
+ * @returns {{bandwidth: number, width: number, height: number, url: string}|null} the pick.
+ */
+export function pickVariant(variants, wanted = {}) {
+  const list = (variants ?? []).filter((each) => each !== null && typeof each === 'object' && String(each.url ?? '') !== '')
+  if (list.length === 0) return null
+  const capHeight = Number(wanted.height ?? 0)
+  const capBandwidth = Number(wanted.bandwidth ?? 0)
+  let allowed = list
+  if (capHeight > 0) allowed = list.filter((each) => (each.height || 0) <= capHeight)
+  if (capBandwidth > 0) allowed = allowed.filter((each) => (each.bandwidth || 0) <= capBandwidth)
+  /* A cap nobody fits is not an empty answer: the smallest is still better
+     than nothing. */
+  if (allowed.length === 0) allowed = [...list].sort((left, right) => (left.bandwidth || left.height) - (right.bandwidth || right.height)).slice(0, 1)
+  return allowed.slice().sort((left, right) => (right.bandwidth || right.height) - (left.bandwidth || left.height))[0]
+}
+
+/**
+ * Follow a manifest down to the URLs that hold bytes.
+ * @param {string} url - what the page asked for.
+ * @param {string} body - what came back.
+ * @param {{height?: number, bandwidth?: number}} [wanted] - size cap.
+ * @returns {{kind: 'file'|'hls', parts: string[], init: string, duration: number, variant: object|null, encrypted: boolean}} what to fetch.
+ */
+export function planOf(url, body, wanted = {}) {
+  const kind = playlistKind(body)
+  if (kind === 'master') {
+    const variant = pickVariant(parseMaster(body, url), wanted)
+    return { kind: 'hls', parts: [], init: '', duration: 0, variant, encrypted: false }
+  }
+  if (kind === 'media') {
+    const list = parsePlaylist(body, url)
+    return { kind: 'hls', parts: list.segments, init: list.init, duration: list.duration, variant: null, encrypted: list.encrypted }
+  }
+  return { kind: 'file', parts: [url], init: '', duration: 0, variant: null, encrypted: false }
+}
+
+/**
+ * Fetch many URLs, but only a few at a time: a hundred segments asked for at
+ * once is a hundred sockets and a server that starts refusing.
+ * @template T
+ * @param {string[]} urls - in order.
+ * @param {(url: string) => Promise<T>} get - one URL.
+ * @param {number} [width] - how many at once.
+ * @returns {Promise<T[]>} in the same order as the input.
+ */
+export async function gather(urls, get, width = 6) {
+  const out = new Array(urls.length)
+  let next = 0
+  const take = async () => {
+    for (;;) {
+      const at = next
+      next += 1
+      if (at >= urls.length) return
+      out[at] = await get(urls[at])
+    }
+  }
+  const taking = []
+  for (let i = 0; i < Math.max(1, Math.min(width, urls.length)); i += 1) taking.push(take())
+  await Promise.all(taking)
+  return out
+}
+
+/**
+ * How a person would see the size of a pile of bytes.
+ * @param {number} bytes - byte count.
+ * @returns {string} "12 MB" and the like.
+ */
+export function sizeText(bytes) {
+  const n = Number(bytes ?? 0)
+  if (!(n > 0)) return '?'
+  if (n >= 1_000_000_000) return `${(n / 1_000_000_000).toFixed(1)} GB`
+  if (n >= 1_000_000) return `${(n / 1_000_000).toFixed(1)} MB`
+  if (n >= 1000) return `${Math.round(n / 1000)} kB`
+  return `${String(n)} B`
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-inner-browser-use",
-  "version": "0.35.2",
+  "version": "0.36.0",
   "description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
   "type": "module",
   "main": "index.js",
@@ -17,6 +17,7 @@
     "tools.js",
     "read.js",
     "review.js",
+    "media.js",
     "client.js",
     "locale/*.json",
     "cordis.patch.yml",
@@ -48,7 +49,7 @@
     "dsh": ">=0.2.0"
   },
   "scripts": {
-    "test": "node --test test/artifacts.test.mjs test/client.test.mjs test/cookies.test.mjs test/diff.test.mjs test/fresh.test.mjs test/host.test.mjs test/input.test.mjs test/isolation.test.mjs test/keyboard.test.mjs test/leak.test.mjs test/login.test.mjs test/no-duplicate.test.mjs test/package.test.mjs test/readfault.test.mjs test/refs.test.mjs test/review.test.mjs test/session.test.mjs test/settle.test.mjs test/sidecar.test.mjs test/swf.test.mjs test/tabs.test.mjs test/trouble.test.mjs",
+    "test": "node --test test/artifacts.test.mjs test/client.test.mjs test/cookies.test.mjs test/diff.test.mjs test/fresh.test.mjs test/host.test.mjs test/input.test.mjs test/isolation.test.mjs test/keyboard.test.mjs test/leak.test.mjs test/login.test.mjs test/media.test.mjs test/no-duplicate.test.mjs test/package.test.mjs test/readfault.test.mjs test/refs.test.mjs test/review.test.mjs test/session.test.mjs test/settle.test.mjs test/sidecar.test.mjs test/swf.test.mjs test/tabs.test.mjs test/trouble.test.mjs",
     "doctor": "node scripts/doctor.mjs",
     "breadth": "node scripts/breadth.mjs",
     "live": "node scripts/live.mjs",

package/read.js

@@ -432,6 +432,66 @@ export function resolve(ref) {
 })()`
 }
 
+/**
+ * The first of several controls that still says these words, found in one
+ * evaluate. A caller hunts through a dozen numbers after a redraw: one page
+ * round trip answers "which of them is still there", instead of a dozen.
+ * @param {number[]} refs - numbers to try, in the order a person would.
+ * @param {string} wanted - the words, compared with the spaces taken out.
+ * @returns {string} an expression that answers with the first that is live.
+ */
+export function resolveFirst(refs, wanted) {
+  const order = refs.map((ref) => Number(ref)).filter((ref) => Number.isFinite(ref))
+  return `(() => {
+  const reg = window.${REGISTRY}
+  if (!reg) return JSON.stringify({ gone: true })
+  const squeeze = (text) => String(text).toLowerCase().replace(/\\s+/g, '')
+  const want = ${JSON.stringify(String(wanted))}
+  const spotOf = (el) => {
+    const b = el.getBoundingClientRect()
+    if (b.width < 2 || b.height < 2) return null
+    const clipped = () => {
+      const me = el.getBoundingClientRect()
+      for (let up = el.parentElement; up; up = up.parentElement) {
+        const css = getComputedStyle(up)
+        if (!/auto|scroll|hidden|clip/.test(css.overflowX + ' ' + css.overflowY)) continue
+        const box = up.getBoundingClientRect()
+        if (me.top < box.top - 1 || me.bottom > box.bottom + 1 || me.left < box.left - 1 || me.right > box.right + 1) return true
+      }
+      return false
+    }
+    if (b.top < 0 || b.left < 0 || b.bottom > innerHeight || b.right > innerWidth || clipped()) el.scrollIntoView({ block: 'center', inline: 'center' })
+    const after = el.getBoundingClientRect()
+    return { x: Math.round(after.left + after.width / 2), y: Math.round(after.top + after.height / 2) }
+  }
+  for (const ref of ${JSON.stringify(order)}) {
+    const hit = reg.find(ref)
+    if (!hit) continue
+    const el = hit.el
+    const name = String(el.innerText ?? el.value ?? el.getAttribute?.('aria-label') ?? '').trim().slice(0, 80)
+    if (want !== '' && !squeeze(name).includes(want)) continue
+    const spot = spotOf(el)
+    if (spot === null) continue
+    return JSON.stringify({ gone: false, ref, name, x: spot.x, y: spot.y })
+  }
+  return JSON.stringify({ gone: true })
+})()`
+}
+
+/**
+ * Ask the page which of these numbers still carries the words.
+ * @param {any} page - sidecar page.
+ * @param {number[]} refs - numbers to try, in order.
+ * @param {string} wanted - the words said by the caller.
+ * @returns {Promise<{gone: boolean, ref?: number, name?: string, x?: number, y?: number}>} the first live one.
+ */
+export async function locateFirst(page, refs, wanted) {
+  const said = await evaluateIn(page, resolveFirst(refs, wanted)).catch(() => undefined)
+  let spot = { gone: true }
+  try { spot = JSON.parse(String(said ?? '{"gone":true}')) } catch (error) { spot = { gone: true } }
+  return spot
+}
+
 /**
  * Click the control the page itself registered, instead of moving the mouse to
  * it. A page whose own handlers want a click, or a control an overlay covers,
@@ -685,7 +745,7 @@ export function covering(ref) {
 }
 
 /** Where refs of controls inside iframes start. Main-frame refs stay small. */
-const FRAME_BASE = 900_000
+export const FRAME_BASE = 900_000
 
 /** Room for the refs of one frame between FRAME_BASE steps. */
 const FRAME_SLOT = 100_000

package/sidecar.js

@@ -11,6 +11,7 @@ import { homedir } from 'node:os'
 import { join } from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { readyFlash } from './runtime/flash-setup.mjs'
+import { isMedia, kindOf } from './media.js?m=1'
 
 /**
  * Windows keeps a browser under Program Files (machine-wide) or LocalAppData
@@ -575,6 +576,9 @@ export class SidecarPage {
        be understood without a screenshot or a retry. */
     this.network = []
     this.requests = new Map()
+    /* The movie a page plays is one of the requests it already made, so the
+       file behind it costs no second look: the network log is the sniff. */
+    this.media = new Map()
     /* A PDF is a document the mirror cannot draw: Chrome leaves the page blank
        and a model looking at it would think the address was empty. What it is
        and where the bytes went is state, not a guess. */
@@ -713,6 +717,7 @@ export class SidecarPage {
          it navigates the same tab to its own page, and the PDF is still what
          the tab shows. */
       if (String(params.type ?? '') === 'Document' && !String(request.url ?? '').startsWith('chrome-extension://')) this.pdf = null
+      if (String(params.type ?? '') === 'Document' && !String(request.url ?? '').startsWith('chrome-extension://')) this.media = new Map()
       this.requests.set(id, {
         method: String(request.method ?? 'GET'),
         url: String(request.url ?? '').slice(0, 300), // ui-copy-ignore
@@ -732,6 +737,8 @@ export class SidecarPage {
           bytes: Number(params.response?.encodedDataLength ?? 0),
         }
       }
+      if (known !== undefined) known.status = Number(params.response?.status ?? 0)
+      this.noteMedia(params)
       if (known === undefined) return
       known.status = Number(params.response?.status ?? 0)
       this.troubleSeq += 1
@@ -749,6 +756,43 @@ export class SidecarPage {
     }
   }
 
+  /**
+   * Keep the requests that hold a movie or a song, so asking "what can be saved
+   * here" is a memory read instead of a network sweep.
+   * @param {{requestId?: string, type?: string, response?: object}} params - a Network.responseReceived payload.
+   * @returns {void} nothing.
+   */
+  noteMedia(params) {
+    const response = params?.response ?? {}
+    const url = String(response.url ?? '')
+    const mime = String(response.mimeType ?? '')
+    if (!isMedia(url, mime, String(params?.type ?? ''))) return
+    const id = String(params?.requestId ?? '')
+    if (id === '') return
+    const headers = response.headers ?? {}
+    const length = Number(headers['content-length'] ?? headers['Content-Length'] ?? response.encodedDataLength ?? 0)
+    const existing = this.media.get(id)
+    this.media.set(id, {
+      url,
+      mime,
+      kind: kindOf(url, mime),
+      bytes: Number.isFinite(length) ? length : 0,
+      status: Number(response.status ?? 0),
+      seq: existing?.seq ?? this.troubleSeq,
+    })
+    if (this.media.size > 60) this.media.delete(this.media.keys().next().value)
+  }
+
+  /**
+   * What on this page is worth saving, newest first.
+   * @returns {Array<{url: string, mime: string, kind: string, bytes: number, status: number}>} the media.
+   */
+  mediaList() {
+    const byUrl = new Map()
+    for (const each of this.media.values()) byUrl.set(each.url, each)
+    return [...byUrl.values()].sort((left, right) => right.seq - left.seq)
+  }
+
   /**
    * Follow a tab the page just opened: the mirror shows the new page and the
    * popup is closed, so there is still one page to drive.

package/tools.js

@@ -14,10 +14,29 @@ import { createHash } from 'node:crypto'
 import { homedir } from 'node:os'
 import { basename, join } from 'node:path'
 import { mkdir, readdir, stat, unlink, writeFile } from 'node:fs/promises'
-import { VIEW } from './sidecar.js?v=1'
-import { readPage, renderPage, readArticle, readRelevant, findOnPage, locate, covering, contextFor, identityOf, sameThing, differs, labelSpot, selectSpot, bounds, pictures, clickInDom, writeInDom, REF_GONE } from './read.js?v=1'
+import { VIEW, DESKTOP_UA } from './sidecar.js?v=1'
+import { planOf, gather, sizeText } from './media.js?m=1'
+import { readPage, renderPage, readArticle, readRelevant, findOnPage, locate, locateFirst, FRAME_BASE, covering, contextFor, identityOf, sameThing, differs, labelSpot, selectSpot, bounds, pictures, clickInDom, writeInDom, REF_GONE } from './read.js?v=1'
 import { brief as askBrief, recordRound, lastRoundNote, lastRound, markRound, MARK_STATUS } from './review.js?v=1'
 
+/**
+ * What a saved file should be called, from the address when it says so and the
+ * type when it does not.
+ * @param {string} url - the media's address.
+ * @param {string} [mime] - what the server called it.
+ * @returns {string} an extension with the dot.
+ */
+function extensionOf(url, mime = '') {
+  const known = /\.(mp4|m4v|webm|mkv|mov|flv|ts|m4s|mp3|m4a|aac|ogg|opus|wav)(\?|#|$)/i.exec(String(url ?? ''))
+  if (known !== null) return `.${known[1].toLowerCase()}`
+  const said = String(mime ?? '').toLowerCase()
+  if (said.includes('mp4') || said.includes('m4v')) return '.mp4'
+  if (said.includes('webm')) return '.webm'
+  if (said.includes('mpegurl')) return '.ts'
+  if (said.startsWith('audio/')) return '.m4a'
+  return '.bin'
+}
+
 /** Which chat the running call belongs to, so each gets its own tab. */
 export const sessionScope = new AsyncLocalStorage()
 
@@ -1050,6 +1069,49 @@ async function until(page, expression, timeout) {
  * @returns {Promise<string>} a compact report.
  */
 /** The first control whose words match, scrolled into view. Text can beat a number: it costs no read. */
+/** Words a label can lose without changing what it asks for. */
+const SMALL_WORDS = new Set(['a', 'an', 'the', 'to', 'of', 'and', 'or', 'in', 'on', 'at', 'for', 'with', 'my', 'your', 'please', 'click', 'tap', 'button', 'btn', 'link', 'now'])
+
+/**
+ * The significant words of a label, lowercased.
+ * @param {unknown} text - a label.
+ * @returns {string[]} words worth comparing.
+ */
+export function tokensOf(text) {
+  return String(text ?? '').toLowerCase().split(/[^a-z0-9\u4e00-\u9fff]+/).filter((word) => word !== '' && !SMALL_WORDS.has(word))
+}
+
+/**
+ * The one label that means what the caller said when no label repeats it.
+ * Only an unambiguous winner is offered, so a paraphrase cannot silently click
+ * a different control than the one asked for.
+ * @param {Array<{ref: number, name?: string, view?: boolean}>} controls - controls from a read.
+ * @param {string} words - what the caller asked for.
+ * @returns {any} the best control, or null when nothing stands out.
+ */
+export function nearestByWords(controls, words) {
+  const want = tokensOf(words)
+  if (want.length === 0 || !Array.isArray(controls)) return null
+  const scored = []
+  for (const control of controls) {
+    if (control?.view === false) continue
+    const have = new Set(tokensOf(control?.name))
+    if (have.size === 0) continue
+    let shared = 0
+    for (const word of want) if (have.has(word)) shared += 1
+    const cover = shared / want.length
+    if (shared === 0 || cover < 0.5) continue
+    scored.push({ control, score: cover * 0.7 + (shared / have.size) * 0.3 })
+  }
+  if (scored.length === 0) return null
+  scored.sort((a, b) => b.score - a.score)
+  const best = scored[0]
+  const second = scored[1]
+  if (second !== undefined && second.score > best.score - 0.05) return null
+  if (String(best.control.name ?? '').length > String(words).length * 3 + 12) return null
+  return { ...best.control, ref: Number(best.control.ref) }
+}
+
 export async function findWords(page, words, known) {
   /* The words a caller used usually came from a read that already happened, so
      look among those numbers first: one position check is a single evaluate,
@@ -1066,7 +1128,19 @@ export async function findWords(page, words, known) {
      end is only the fallback. */
   const firstInOrder = remembered.filter((control) => control.view !== false).sort((a, b) => Number(a.ref) - Number(b.ref))
   const nearEnd = remembered.slice(-8).reverse()
-  for (const control of [...firstInOrder.slice(0, 4), ...nearEnd]) {
+  const order = [...firstInOrder.slice(0, 4), ...nearEnd]
+  /* One round trip answers which of these the page still has: a dozen numbers
+     left over from a read that a redraw invalidated used to cost a dozen
+     checks. Numbers inside a frame keep the per-number route below. */
+  const plain = order.map((control) => Number(control.ref)).filter((ref) => Number.isFinite(ref) && ref < FRAME_BASE)
+  if (plain.length > 0) {
+    const any = await locateFirst(page, plain, wantedFirst)
+    if (any?.gone !== true && Number.isFinite(Number(any?.x))) {
+      const byRef = order.find((control) => Number(control.ref) === Number(any.ref)) ?? { ref: Number(any.ref) }
+      return { ...byRef, ref: Number(any.ref), name: String(any.name ?? byRef.name ?? ''), x: Number(any.x), y: Number(any.y) }
+    }
+  }
+  for (const control of order.filter((control) => Number(control.ref) >= FRAME_BASE)) {
     const spot = await locate(page, Number(control.ref))
     if (spot.gone === true) continue
     const name = String(spot.name ?? control.name ?? '')
@@ -1080,7 +1154,17 @@ export async function findWords(page, words, known) {
   const squeeze = (text) => String(text).toLowerCase().replace(/\s+/g, '')
   const wanted = squeeze(words)
   const found = shown.find((control) => squeeze(control.name) === wanted) ?? shown.find((control) => squeeze(control.name).includes(wanted))
-  if (found === undefined) return null
+  if (found === undefined) {
+    /* A button the page redrew often only swaps one word, and a person says
+       "submit order" for "Place order". Accept only an unambiguous winner. */
+    const near = nearestByWords(shown, words)
+    if (near !== null) {
+      const spot = await locate(page, near.ref)
+      if (spot.gone === true) return null
+      return { ...near, x: Number(spot.x), y: Number(spot.y) }
+    }
+    return null
+  }
   /* When only the squeezed form matched, say the words the caller used: the
      page spread them over nodes and "Signin" is not what anyone wrote. */
   const hit = squeeze(found.name) === wanted && found.name.toLowerCase() !== words.toLowerCase() ? { ...found, name: words } : found
@@ -3248,6 +3332,91 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
       }),
     },
 
+    {
+      name: 'browser_media', // ui-copy-ignore
+      description: 'What this page played, and how to keep it: lists the video and sound requests the page already made, with their sizes; save one by row number or part of its address. A stream playlist is followed to its pieces and joined, so nothing has to be installed.',
+      parameters: {
+        type: 'object',
+        properties: {
+          find: { type: 'string', description: 'Only rows whose address contains this.' },
+          save: { type: ['integer', 'string'], description: 'Row number from the list, or part of its address.' },
+          height: { type: 'number', description: 'Tallest version to take when one address offers several (default 1080).' },
+          name: { type: 'string', description: 'File name without extension.' },
+        },
+        additionalProperties: false,
+      },
+      output: TEXT_OUTPUT, // ui-copy-ignore
+      timeoutMs: 180_000,
+      isConcurrencySafe: () => false,
+      /** @returns {Promise<{text: string}>} the list, or where the file went. */
+      execute: async (args) => withRawPage(async (page) => {
+        const all = typeof page.mediaList === 'function' ? page.mediaList() : []
+        const needle = String(args?.find ?? '').toLowerCase()
+        const rows = needle === '' ? all : all.filter((each) => `${String(each.url)} ${String(each.mime)}`.toLowerCase().includes(needle))
+        const save = args?.save
+        if (save === undefined || save === null || save === '') {
+          if (rows.length === 0) {
+            return all.length === 0
+              ? 'Nothing to save yet: play the video or sound once, then ask again — the request it makes is the file.'
+              : `Nothing matches ${String(args?.find)}; this page has ${String(all.length)} other media.`
+          }
+          const lines = rows.slice(0, 12).map((each, at) => `${String(at + 1)}. ${String(each.kind)} ${sizeText(each.bytes)} · ${String(each.url).slice(0, 160)}`)
+          if (rows.length > 12) lines.push(`… ${String(rows.length - 12)} more; narrow it with find`)
+          lines.push('save: <row> keeps one')
+          return lines.join('\n')
+        }
+        const wanted = typeof save === 'number' ? rows[save - 1] : rows.find((each) => String(each.url).includes(String(save)))
+        if (wanted === undefined) {
+          return `No media row ${String(save)}. ${rows.length === 0 ? 'Nothing to save yet.' : 'Call browser_media without save to see the rows.'}`
+        }
+        const cookieAnswer = await page.cdp.send('Network.getCookies', { urls: [String(wanted.url)] }).catch(() => ({ cookies: [] }))
+        const cookie = (cookieAnswer?.cookies ?? []).map((each) => `${String(each.name)}=${String(each.value)}`).join('; ')
+        const headers = { 'user-agent': DESKTOP_UA, accept: '*/*' }
+        if (cookie !== '') headers.cookie = cookie
+        if (/^https?:/i.test(String(page.url ?? ''))) headers.referer = String(page.url)
+        const started = Date.now()
+        const getBinary = async (address) => {
+          const answer = await fetch(address, { headers, redirect: 'follow', signal: AbortSignal.timeout(60_000) })
+          if (!answer.ok) throw new Error(`${String(answer.status)} for ${address.slice(0, 120)}`)
+          return Buffer.from(await answer.arrayBuffer())
+        }
+        const cap = Number(args?.height ?? 1080)
+        const wantedSize = cap > 0 ? { height: cap } : {}
+        let parts
+        let ext = extensionOf(wanted.url, wanted.mime)
+        try {
+          let body = await getBinary(String(wanted.url))
+          parts = [body]
+          const playlist = /\.m3u8(\?|#|$)/i.test(String(wanted.url)) || /mpegurl/i.test(String(wanted.mime ?? ''))
+          if (playlist) {
+            let plan = null
+            let from = String(wanted.url)
+            for (let hop = 0; hop < 2; hop += 1) {
+              plan = planOf(from, body.toString('utf8'), wantedSize)
+              if (plan.kind !== 'hls' || plan.variant === null) break
+              from = plan.variant.url
+              body = await getBinary(from)
+            }
+            if (plan.encrypted === true) return 'That stream is encrypted: the browser can play it, but the pieces are not a file to keep.'
+            if (plan.parts.length === 0) return 'That playlist names no pieces. Play the movie first, then ask again.'
+            const init = plan.init === '' ? [] : [await getBinary(plan.init)]
+            parts = [...init, ...(await gather(plan.parts, getBinary, 6))]
+            ext = plan.init !== '' ? '.mp4' : (/\.ts(\?|#|$)/i.test(plan.parts[0]) ? '.ts' : '.m4s')
+          }
+        } catch (error) {
+          return `Could not fetch it: ${String(error?.message ?? error)}`
+        }
+        const body = Buffer.concat(parts)
+        const dir = artifactDir('media')
+        await mkdir(dir, { recursive: true })
+        const stamp = new Date().toISOString().replace(/[-:]/g, '').replace('T', '-').slice(0, 15)
+        const safe = String(args?.name ?? '').replace(/[^\w.-]/g, '').replace(/\.{2,}/g, '.').replace(/^\.+/, '')
+        const file = join(dir, `${safe === '' ? `media-${stamp}` : safe}${ext}`)
+        await writeFile(file, body)
+        return `${file} — ${sizeText(body.length)} in ${String(Date.now() - started)} ms${parts.length > 1 ? ` (${String(parts.length)} parts joined)` : ''}`
+      }),
+    },
+
     {
       name: 'browser_pdf', // ui-copy-ignore
       description: 'Print the page to a PDF and return its path. Useful when the person wants the document rather than a picture of it.',