dsh-inner-browser-use 0.36.0 → 0.37.0

package/README.md

@@ -20,7 +20,7 @@ live picture of that browser, and your own cursor keeps working on the picture.
 | `browser_key` | Presses one key: Enter, Tab, Escape, Backspace, Delete, Insert, arrows, Home, End, PageUp, PageDown, F1-F12, the numeric keypad, letters, digits, punctuation and every shifted symbol, plus ContextMenu, CapsLock, PrintScreen and media keys. `Ctrl`/`Alt`/`Shift`/`Meta`/`Mod` modifiers, chords with `+`, and `Ctrl+C/X/V/A/Z/Y` run the real editing commands. |
 | `browser_scroll` | Scrolls the wheel at the centre of the page, or at `x`/`y`. `dy: 600` is about a screen down. |
 | `browser_shot` | Saves a full-size PNG into `~/.dsh-inner-browser-use/screenshots` and returns the path. |
-| `browser_media` | Lists the video and sound the page already fetched, with sizes; `save: <row>` or part of the address keeps one. A stream is followed to its pieces and joined, so nothing has to be installed. |
+| `browser_media` | Lists the video and sound the page already fetched, with sizes, the one its own player shows first and marked `main`; `save: <row>`, `save: main` or part of the address keeps one. Ads and beacons stay out unless `all: true`; `watch: false` stops remembering the page. A stream is followed to its pieces and joined, so nothing has to be installed. |
 | `browser_state` | Reports whether the browser runs, which profile it uses, its address and title, and how many columns are showing it. |
 
 `browser_read` on the same page is the words only, in Markdown (measured: 4,520

package/README.zh.md

@@ -19,7 +19,7 @@ profile，不是产品自带的那个浏览页签：右栏放的是它的画面
 | `browser_key` | 按一个键：Enter、Tab、Escape、Backspace、Delete、上下左右、Home、End、PageUp、PageDown。 |
 | `browser_scroll` | 在页面中心滚轮，也可以指定 `x`/`y`。`dy: 600` 约一屏。 |
 | `browser_shot` | 存一张 PNG 到 `~/.dsh/inner-browser-use/screenshots`，返回路径；`region: [x, y, w, h]` 只截一块（游戏、图表、看局部）。 |
-| `browser_media` | 列出页面已经请求过的视频和声音（带大小），`save: <行号>` 或地址片段存下来；流会顺着播放列表把片段取回拼好，不用装任何东西。 |
+| `browser_media` | 列出页面已经请求过的视频和声音（带大小），页面自己播放器里那个排第一、标 `main`；`save: <行号>`、`save: main` 或地址片段存下来。广告和统计默认不出现（`all: true` 才看），`watch: false` 关掉这个页面的记录。流会顺着播放列表把片段取回拼好，不用装任何东西。 |
 | `browser_state` | 报浏览器在不在跑、用的哪个 profile、地址、标题，以及有几个右栏在看它。 |
 
 点、输入、按键之后的那一行会多一句结果：`· navigated to …`（换页了）、`· content changed`（页内有变化）、`· no visible change`（点了没动）、Flash 页则是 `· screen updated` / `· no frame change`；`read: true` 时不再多说。

package/docs/ARTIFACTS.md

@@ -16,4 +16,18 @@
 
 调用环境里没有项目或会话信息时，落到 `no-project/no-chat/<种类>/`，不猜。
 
+## 整个目录
+
+```
+~/.dsh-inner-browser-use/
+  artifacts/<项目>/<会话>/<种类>/   给人看的产出，只在这里（见上）
+  chrome/                          浏览器自己的数据：登录态、Cookie、缓存
+  flash/                           Flash 播放器运行时，第一次用 flash:true 才装
+  browser.log                     最近一次浏览器启动和页面记录
+  asks.log                        人问过什么、答了什么
+  tabs.json / mirror.json / flash.json / watcher.json   上次开着哪些页
+```
+
+`chrome/` 是浏览器进程自己在读写，不要手动改；要清登录态，用 `browser_open` 的档位或插件自己的入口，别删目录里的某个文件。上面这些都在一个根目录下，卸载插件前想留产出就拷 `artifacts/`。
+
 测试里用 `DSH_INNER_BROWSER_ARTIFACTS` 指向临时目录，不写真实 home。

package/docs/BENCH.md

@@ -1,3 +1,20 @@
+## 只列该看的那一个（0.37.0，同一台本地页面，加了两个广告请求）
+
+页面自己放一个 `<video src="/movie.mp4">`、一个 `<audio>`、fetch 一个 master 播放列表，再 fetch 两条广告（`/ads/preroll.mp4` 9 MB、`/px/beacon.mp4` 2 B）。不筛选时 5 条都在；默认列表剩 3 条，页面播放器那个标 `main` 排第一，`save: main` 存的就是它。
+
+| 项 | 0.36.0 | 0.37.0 |
+|---|---|---|
+| 列表行数（同一页面 5 条请求） | 5 | 3，广告单独一行说数量 |
+| 首行 | `1. video 20 B · /movie.mp4` | `1. main video 156 B · /movie.mp4` |
+| 找人要的那个 | 数行号 / 猜地址 | `save: main` |
+| 列表耗时 | 0 ms（纯内存） | 1 ms（多一次页面查询拿播放器与时长） |
+
+`media-pick` 1 ms，预算 `media-pick: 150 ms`；`all: true` 回到 5 行。`media-sniff` 仍 0 ms（记录仍是网络日志附带的），`media-save` 4 ms、96 B、`6 parts joined`，`budget checks 17 ok`。
+
+## 媒体嗅探与流保存（0.36.0，本地 http 页面，`scripts/bench.mjs`）
+
+页面自己放 `<video>`、`<audio>` 并 fetch 一个 master 播放列表：加载完的那一刻网络记录里已经有 3 条，`media-sniff` 0 ms（没有额外请求，嗅探就是已有的网络记录）。保存 master → 1080 → 6 个片段，`media-save` 5 ms、96 B，报 `6 parts joined`；列表首行 59 字符。预算 `media-sniff: 400 ms`、`media-save: 500 ms` 均通过，`budget checks 16 ok`。
+
 ## Flash 镜像按栏改窗口（0.34.48，Electron Flash 运行时，右栏 747×1196）
 
 Flash 路径原来采集一个固定的 1440×900 窗口，竖高右栏里上下留黑边。现在 `setViewport` 直接改窗口大小，采集源与栏同比例。

package/index.js

@@ -944,7 +944,7 @@ export function apply(ctx) {
         '先接着用已有的页：`browser_view`（拿编号控件与可见文字）→ `browser_click` / `browser_type` → 再 `browser_view` 直到出现要的结果；页面要等就重复读，不 sleep。`browser_open` 只用于没有页、要换新地址、或要明确刷新；同一个地址不要反复重开。',
         '页面很长、只想确认某句话/按钮在哪时用 `browser_find`，别整页读。要读长文（文章、文档、issue、论坛帖）用 `browser_read`：它回 Markdown，只花几十分之一的 token，链接可点；要动手点、填表才用 `browser_view` 拿编号。',
         '要保存页面样子用 `browser_shot`，它会给出 PNG 绝对路径；要交给人看的文件，把绝对路径写进回复。',
-        '页面放视频或音乐、人想留下它时用 `browser_media`：它列的是页面自己已经请求过的媒体（带大小），用行号或地址片段存一个；视频是流的话会顺着播放列表把片段取回拼成整段，不用装 yt-dlp 或 ffmpeg。先让页面播一下再列，播放请求才是文件。',
+        '页面放视频或音乐、人想留下它时用 `browser_media`：它列的是页面自己已经请求过的媒体（带大小），页面自己播放器里那个标 `main` 排第一，广告和统计请求默认不出现（`all:true` 才看），`save: main` 存第一个；视频是流的话会顺着播放列表把片段取回拼成整段，不用装 yt-dlp 或 ffmpeg。先让页面播一下再列，播放请求才是文件。不想要这个页面的记录就 `watch:false`。',
         '游戏、canvas、Flash：画面自己画的地方，`browser_view` 和 `browser_read` 读不出里面是什么。先 `browser_prepare` 拿画面位置、帧率、按键是否到达和下一步；之后看 `browser_shot` 的图定位置，按坐标点、按住、按组合键，再拍一张看画面变了没有——变了就是这一步生效了。当年用 Flash 做的游戏用 `browser_open` 带 `flash:true` 在栏内跑，不装播放器、不开窗口；Flash 默认关，`flash:true` 开、`flash:false` 回到普通 Chrome，两边都只在本会话右栏。',
       ].join('\n'),
     }), 'dsh-inner-browser-use: system prompt section')

package/media.js

@@ -237,3 +237,73 @@ export function sizeText(bytes) {
   if (n >= 1000) return `${Math.round(n / 1000)} kB`
   return `${String(n)} B`
 }
+
+/**
+ * Ad servers and ad slots, recognised by themselves rather than by size: an ad
+ * is often the biggest file on the page, and a real clip can be small.
+ */
+const AD_HOST = /^(?:(?:[\w-]+\.)*(?:doubleclick\.net|googlesyndication\.com|googleadservices\.com|amazon-adsystem\.com|adsrvr\.org|taboola\.com|outbrain\.com|criteo\.(?:net|com)|scorecardresearch\.com|moatads\.com|adnxs\.com|pubmatic\.com|rubiconproject\.com|openx\.net)$)$/i;
+const AD_SLOT = /^(?:ad|ads|adv|adserver|advert(?:isement|ising)?|pre-?roll|mid-?roll|post-?roll|banner|sponsor|promo|tracking|tracker|pixel|beacon|analytics|telemetry)\d*$/i
+
+/**
+ * Whether an address is an ad or a measurement call.
+ * @param {string} url - the address.
+ * @returns {boolean} true when it is worth hiding from a person picking a file.
+ */
+export function adLike(url) {
+  let parsed
+  try {
+    parsed = new URL(String(url))
+  } catch {
+    return false
+  }
+  if (AD_HOST.test(parsed.hostname)) return true
+  const pieces = parsed.pathname.split('/').filter((piece) => piece !== '')
+  const named = pieces.slice(0, -1).some((piece) => AD_SLOT.test(piece))
+  if (named) return true
+  const file = pieces.length === 0 ? '' : pieces[pieces.length - 1]
+  return /^(?:ad|ads|adv|preroll|banner|pixel|beacon|track|tracking)[-_.]/i.test(file)
+}
+
+/**
+ * How much a row looks like the thing a person came for: the player on the page
+ * first, then a real file, and ads or beacons last.
+ * @param {{url: string, mime?: string, kind?: string, bytes?: number}} row - one media request.
+ * @param {Array<{url: string, dur?: number}>} [players] - what the page's own `<video>` / `<audio>` plays.
+ * @returns {number} the score; higher is more likely the point of the page.
+ */
+export function coreScore(row, players = []) {
+  const url = String(row?.url ?? '')
+  const bytes = Number(row?.bytes ?? 0)
+  const kind = String(row?.kind ?? '')
+  const mime = String(row?.mime ?? '')
+  let score = 0
+  const mine = players.some((one) => {
+    const said = String(one?.url ?? '')
+    return said !== '' && (said === url || url.startsWith(said) || said.startsWith(url))
+  })
+  if (mine) score += 100
+  if (adLike(url)) score -= 80
+  if (bytes >= 1_000_000) score += 25
+  else if (bytes >= 200_000) score += 10
+  else if (bytes > 0 && bytes < 20_000 && kind !== 'stream') score -= 20
+  if (kind === 'stream') score += 15
+  if (/^video\//i.test(mime)) score += 5
+  if (/^audio\//i.test(mime)) score += 3
+  return score
+}
+
+/**
+ * The rows in the order a person would want them, with ads gone unless asked.
+ * @param {Array<object>} rows - media rows, any order.
+ * @param {Array<{url: string, dur?: number}>} [players] - the page's own players.
+ * @param {boolean} [all] - keep the ads and the beacons too.
+ * @returns {Array<object>} the rows to show.
+ */
+export function coreRows(rows, players = [], all = false) {
+  const ranked = rows.map((row) => ({ row, score: coreScore(row, players) }))
+  return ranked
+    .filter((one) => all || one.score >= 0)
+    .sort((left, right) => right.score - left.score || Number(right.row?.seq ?? 0) - Number(left.row?.seq ?? 0))
+    .map((one) => one.row)
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-inner-browser-use",
-  "version": "0.36.0",
+  "version": "0.37.0",
   "description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
   "type": "module",
   "main": "index.js",

package/sidecar.js

@@ -579,6 +579,7 @@ export class SidecarPage {
     /* The movie a page plays is one of the requests it already made, so the
        file behind it costs no second look: the network log is the sniff. */
     this.media = new Map()
+    this.mediaOn = true
     /* A PDF is a document the mirror cannot draw: Chrome leaves the page blank
        and a model looking at it would think the address was empty. What it is
        and where the bytes went is state, not a guess. */
@@ -763,6 +764,7 @@ export class SidecarPage {
    * @returns {void} nothing.
    */
   noteMedia(params) {
+    if (this.mediaOn === false) return
     const response = params?.response ?? {}
     const url = String(response.url ?? '')
     const mime = String(response.mimeType ?? '')
@@ -783,6 +785,18 @@ export class SidecarPage {
     if (this.media.size > 60) this.media.delete(this.media.keys().next().value)
   }
 
+  /**
+   * Stop or resume remembering media on this page; stopping empties the list so
+   * nothing is kept from before the switch.
+   * @param {boolean} on - false stops, true resumes.
+   * @returns {boolean} what it is set to.
+   */
+  mediaWatch(on) {
+    this.mediaOn = on !== false
+    if (!this.mediaOn) this.media = new Map()
+    return this.mediaOn
+  }
+
   /**
    * What on this page is worth saving, newest first.
    * @returns {Array<{url: string, mime: string, kind: string, bytes: number, status: number}>} the media.

package/tools.js

@@ -15,7 +15,7 @@ import { homedir } from 'node:os'
 import { basename, join } from 'node:path'
 import { mkdir, readdir, stat, unlink, writeFile } from 'node:fs/promises'
 import { VIEW, DESKTOP_UA } from './sidecar.js?v=1'
-import { planOf, gather, sizeText } from './media.js?m=1'
+import { planOf, gather, sizeText, coreRows } from './media.js?m=2'
 import { readPage, renderPage, readArticle, readRelevant, findOnPage, locate, locateFirst, FRAME_BASE, covering, contextFor, identityOf, sameThing, differs, labelSpot, selectSpot, bounds, pictures, clickInDom, writeInDom, REF_GONE } from './read.js?v=1'
 import { brief as askBrief, recordRound, lastRoundNote, lastRound, markRound, MARK_STATUS } from './review.js?v=1'
 
@@ -37,6 +37,24 @@ function extensionOf(url, mime = '') {
   return '.bin'
 }
 
+/** What the page's own players are showing, in one look: the address, the length, whether it runs. */
+async function playersOf(page) {
+  const answer = await page.cdp.send('Runtime.evaluate', {
+    expression: `(() => [...document.querySelectorAll('video,audio')].map((one) => ({ url: one.currentSrc || one.src || '', playing: !one.paused, dur: Number.isFinite(one.duration) ? Math.round(one.duration) : 0 })))()`,
+    returnByValue: true,
+  }).catch(() => null)
+  const seen = answer?.result?.value
+  return Array.isArray(seen) ? seen.filter((one) => String(one?.url ?? '') !== '') : []
+}
+
+/** Seconds as a person writes them: 2:13, 1:02:00. */
+function clockOf(seconds) {
+  const whole = Math.max(0, Math.round(Number(seconds) || 0))
+  const rest = whole % 3600
+  const pad = (value) => String(value).padStart(2, '0')
+  return whole >= 3600 ? `${String(Math.floor(whole / 3600))}:${pad(Math.floor(rest / 60))}:${pad(rest % 60)}` : `${String(Math.floor(whole / 60))}:${pad(rest % 60)}`
+}
+
 /** Which chat the running call belongs to, so each gets its own tab. */
 export const sessionScope = new AsyncLocalStorage()
 
@@ -3334,12 +3352,14 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
 
     {
       name: 'browser_media', // ui-copy-ignore
-      description: 'What this page played, and how to keep it: lists the video and sound requests the page already made, with their sizes; save one by row number or part of its address. A stream playlist is followed to its pieces and joined, so nothing has to be installed.',
+      description: 'What this page played, and how to keep it: lists the video and sound requests the page already made, the one its own player shows first, with sizes; save one by row number, by part of its address, or save: main. Ads and measurement beacons stay out of the list unless all: true. A stream playlist is followed to its pieces and joined, so nothing has to be installed. watch: false stops remembering this page.',
       parameters: {
         type: 'object',
         properties: {
           find: { type: 'string', description: 'Only rows whose address contains this.' },
-          save: { type: ['integer', 'string'], description: 'Row number from the list, or part of its address.' },
+          save: { type: ['integer', 'string'], description: 'Row number from the list, part of its address, or main for the first row.' },
+          all: { type: 'boolean', description: 'Include ads and beacons too (default false).' },
+          watch: { type: 'boolean', description: 'false stops remembering what this page plays and empties the list; true resumes.' },
           height: { type: 'number', description: 'Tallest version to take when one address offers several (default 1080).' },
           name: { type: 'string', description: 'File name without extension.' },
         },
@@ -3350,24 +3370,39 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
       isConcurrencySafe: () => false,
       /** @returns {Promise<{text: string}>} the list, or where the file went. */
       execute: async (args) => withRawPage(async (page) => {
-        const all = typeof page.mediaList === 'function' ? page.mediaList() : []
+        if (typeof args?.watch === 'boolean') {
+          const on = typeof page.mediaWatch === 'function' ? page.mediaWatch(args.watch) : false
+          if (!on) return 'Off: this page keeps nothing from now on. watch: true puts it back.'
+          if (args.save === undefined || args.save === null || args.save === '') return 'On: ask again once it has played something.'
+        }
+        const known = typeof page.mediaList === 'function' ? page.mediaList() : []
+        const players = known.length === 0 ? [] : await playersOf(page)
+        const ranked = coreRows(known, players, args?.all === true)
         const needle = String(args?.find ?? '').toLowerCase()
-        const rows = needle === '' ? all : all.filter((each) => `${String(each.url)} ${String(each.mime)}`.toLowerCase().includes(needle))
+        const rows = needle === '' ? ranked : ranked.filter((each) => `${String(each.url)} ${String(each.mime)}`.toLowerCase().includes(needle))
         const save = args?.save
         if (save === undefined || save === null || save === '') {
           if (rows.length === 0) {
-            return all.length === 0
-              ? 'Nothing to save yet: play the video or sound once, then ask again — the request it makes is the file.'
-              : `Nothing matches ${String(args?.find)}; this page has ${String(all.length)} other media.`
+            if (known.length === 0) return 'Nothing to save yet: play the video or sound once, then ask again — the request it makes is the file.'
+            if (needle !== '') return `Nothing matches ${String(args?.find)}; ${String(known.length)} played here, the ones worth keeping are gone.`
+            return `Only ads and beacons played here (${String(known.length)}); all: true shows them.`
           }
-          const lines = rows.slice(0, 12).map((each, at) => `${String(at + 1)}. ${String(each.kind)} ${sizeText(each.bytes)} · ${String(each.url).slice(0, 160)}`)
+          const hidden = known.length - ranked.length
+          const lines = rows.slice(0, 12).map((each, at) => {
+            const mine = players.find((one) => one.url === each.url || String(one.url).startsWith(String(each.url)) || String(each.url).startsWith(String(one.url)))
+            const mark = at === 0 && mine !== undefined ? 'main ' : ''
+            const length = mine !== undefined && mine.dur > 0 ? ` ${clockOf(mine.dur)}` : ''
+            return `${String(at + 1)}. ${mark}${String(each.kind)}${length} ${sizeText(each.bytes)} · ${String(each.url).slice(0, 160)}`
+          })
           if (rows.length > 12) lines.push(`… ${String(rows.length - 12)} more; narrow it with find`)
-          lines.push('save: <row> keeps one')
+          if (hidden > 0) lines.push(`… ${String(hidden)} ad or beacon kept out; all: true shows them`)
+          lines.push('save: <row> or main')
           return lines.join('\n')
         }
-        const wanted = typeof save === 'number' ? rows[save - 1] : rows.find((each) => String(each.url).includes(String(save)))
+        const named = typeof save === 'string' && save.toLowerCase() === 'main'
+        const wanted = named ? rows[0] : typeof save === 'number' ? rows[save - 1] : rows.find((each) => String(each.url).includes(String(save)))
         if (wanted === undefined) {
-          return `No media row ${String(save)}. ${rows.length === 0 ? 'Nothing to save yet.' : 'Call browser_media without save to see the rows.'}`
+          return `No media row ${String(save)}. ${rows.length === 0 ? 'Nothing to save yet.' : 'Call browser_media without save to see the rows; all: true includes ads.'}`
         }
         const cookieAnswer = await page.cdp.send('Network.getCookies', { urls: [String(wanted.url)] }).catch(() => ({ cookies: [] }))
         const cookie = (cookieAnswer?.cookies ?? []).map((each) => `${String(each.name)}=${String(each.value)}`).join('; ')