dsh-web-fetch-playwright 0.2.15 → 0.2.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -9,13 +9,14 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
9
9
  ## Features
10
10
 
11
11
  - **Real browser rendering** — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
12
- - **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `![alt](data:image/png;base64,...8.9KB)` so they cannot flood the body.
12
+ - **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `![alt](data:image/png;base64,...8.9KB)` so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content *is* the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content. An accepted extraction is then pruned of trailing link-dense sections — the post-article recommendation rails and share bars that ride *inside* the winning wrapper and survive every vocabulary-based rule (163's `post_recommend` cards measure 0.42–0.48 per-card link density, just under Readability's own 0.5 removal bar, and arrive as bare divs with no class to match); a trailing run of such blocks is removed only when it is a minority of its container *and* clearly denser than the content before it, so link-list extractions (feed rows, portal headlines) are untouched.
13
13
  - **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
14
14
  - **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
15
15
  - **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
16
16
  - **Live configuration** — the Plugins page (侧边栏 → 插件 → *dsh-web-fetch-playwright*, dsh ≥ 0.1.7) shows the full form right on the plugin's package page (and on the row's own page via its configure control); it edits the backend, context mode, denoise toggle, and concurrency, and a committed change reaches the next fetch without a restart.
17
17
  - **Budget-aware** — per-fetch deadline (45s); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
18
18
  - **Bounded Cloudflare-challenge wait** — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented `cf-mitigated: challenge` response header plus structural page markers), the fetch keeps the **same tab and context** and waits for the browser's own verification to clear it — tracking the *last* main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded and configurable (`challengeWaitMs`, default 15s; 0 restores the legacy first-response behavior), with a bounded same-tab retry (`challengeRetries`, default 1). When the budget runs out, the fetch fails with the distinct `WEB_FETCH_CHALLENGE` error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.
19
+ - **Deep capture for viewport-virtualized feeds and app-shell documents** — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by `data-block-id` with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/`data-block-id`), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot and splices the removed ones back beside their surviving siblings, in a structural clone of the page that it serializes itself (the live page is never rewritten). Pages that never react to scrolling — most of the web — are detected and skip the slow walk. Bounded by `scrollBudgetMs` (default 20s; 0 restores the legacy single-instant behavior); any failure silently degrades to the legacy capture.
19
20
 
20
21
  ## How it works
21
22
 
@@ -29,8 +30,8 @@ web_fetch (tool-web)
29
30
  └─ ctx.web.fetchProvider = playwright
30
31
  ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
31
32
  ├─ cdp: connectOverCDP(endpoint)
32
- ├─ page.goto → settle (networkidle, best-effort) → strip scripting subtrees in the live DOM → page.content()
33
- ├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
33
+ ├─ page.goto → settle (networkidle, best-effort) → deep-capture scroll walk (virtualized feeds; returns its merged, stripped capture when it restored anything) → otherwise strip scripting subtrees in the live DOM → page.content()
34
+ ├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → trailing link-dense prune → DOMPurify → Turndown(GFM)
34
35
  └─ Markdown (or raw HTML when denoise is off)
35
36
  ```
36
37
 
@@ -79,6 +80,7 @@ The Plugins page (侧边栏 → 插件) opens the form one click from the list:
79
80
  | `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
80
81
  | `challengeWaitMs` | `15000` | Bounded wait (ms, 0–60000) for a Cloudflare challenge to clear naturally in the same tab. `0` disables the whole challenge path — the first response is returned as-is (the pre-0.2.5 behavior). |
81
82
  | `challengeRetries` | `1` | Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 45s per-fetch deadline. |
83
+ | `scrollBudgetMs` | `20000` | Bounded budget (ms, 0–30000) for the deep-capture scroll walk. `0` disables it — the page is captured as it rests (the pre-walk behavior), which loses the bodies of viewport-virtualized feeds. |
82
84
 
83
85
  Local backend resolution order:
84
86
 
@@ -130,6 +132,31 @@ With the wait on (default):
130
132
 
131
133
  Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by `challengeWaitMs` and the 45s per-fetch deadline; nothing blocks forever.
132
134
 
135
+ ### Deep capture (viewport-virtualized feeds and app-shell documents)
136
+
137
+ Feed-shaped pages — Discourse topic streams (linux.do, meta.discourse.org), comment threads, timelines — render only the items near the current scroll position. At rest the live DOM holds a window's worth of real content while every item outside it is a height-only placeholder that keeps the item's identity but none of it:
138
+
139
+ ```html
140
+ <div class="post-stream--cloaked" data-post-number="1" id="post_1" style="height:297.9px !important;…">
141
+ <h2 class="sr-only">由 Diver-QC 于 2 小时前 发布</h2>
142
+ </div>
143
+ ```
144
+
145
+ A fetch that renders, never scrolls, and serializes one instant of that DOM hands back the skeleton — the screen-reader-only "posted by" headings survive the pipeline as if they were content, the bodies vanish, and even the denoise pipeline's coverage gate cannot notice (the page's own visible text is the same windowed fragment as the extraction). Scrolling to the bottom once does not help either: the render window *moves* — measured on a 40-reply linux.do topic, the resting DOM held only the first ~6 replies, a bottom jump left only the last 10, and replies past the first chunk were not in the DOM at all until a load-more sentinel near the bottom fetched them.
146
+
147
+ App-shell document editors have the same pathology one level deeper — the measured badcase is a Feishu wiki page (`bocha-ai.feishu.cn/wiki/…`). There the body never scrolls at all: html/body are locked at exactly one viewport and the document scrolls inside `div.bear-web-x-container` (measured: `documentElement.scrollHeight` 720 == `innerHeight`, `window.scrollTo(0, 800)` left `scrollY` at 0, while the container held 4534px against a 656px client). Its blocks are virtualized with two twists over Discourse: they are keyed by `data-block-id` and carry **no id**, and off-window blocks are **removed** outright — the whole un-rendered tail is one anonymous `.bear-virtual-renderUnit-placeholder` spacer with an explicit height — so there is no window scrollbar to walk, no id to stash by, and no degraded-in-place row to swap back. A fetch just returns the statically rendered first screen (measured: 7 blocks, 892 chars; the full document is ~190 blocks).
148
+
149
+ The fix is a bounded in-page scroll walk (`src/autoscroll.ts`), run after the settle and before the serialization strip for every HTML document:
150
+
151
+ 1. **Resolve the walk target**: the window when the document itself scrolls (the Discourse shape); otherwise the page's dominant inner scroll container (the app-shell shape) — the element with the largest scrollable distance among computed `overflow-y: auto|scroll` elements, gated at half a viewport so dropdown-sized scrollers never trigger a walk. Neither scrollable means no walk.
152
+ 2. **Stash** every keyed element — keyed by id, else `data-block-id` (namespaced so the two can never collide) — as a detached clone, keeping the *richest* sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus `data-preloaded` and `svg-sprites` ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
153
+ 3. **Walk** down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; `scrollTop` drives an inner container exactly the way `window.scrollTo` drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s *patience window* with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A `MutationObserver` watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does: a ~30-viewport static article costs ~4.5s instead of exhausting the 20s budget.
154
+ 4. **Merge** into a structural clone (`document.cloneNode(true)`): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal `</script>` in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. *Splice* (container walks): stashed keys from inside the scroll container that the clone no longer holds were **removed** by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk *saw* rows in — including an anchored open that starts mid-document. *Swap*: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except *containers* (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. The clone is then stripped of scripting elements (the same `src/strip.ts` routine the live read uses) and serialized; the provider reads that string instead of the live page. A walk that restored nothing hands back no HTML, and the live read proceeds exactly as before.
155
+
156
+ The walk runs in both denoise modes and is bounded by `scrollBudgetMs` (default 20s; `0` restores the legacy single-instant behavior) and by the remaining fetch deadline minus a decode reserve. Any failure — a page without `evaluate`, a rejected evaluation, an engine that refuses the walk — silently degrades to the legacy capture; a walk can only add content, never fail a fetch. In CDP profile mode the walk scrolls the fetch-owned tab in the remote browser, exactly where the fetch already renders it.
157
+
158
+ Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. One residual limitation, documented as measured: Feishu table blocks lazy-render their *cells* behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.
159
+
133
160
  ## Development
134
161
 
135
162
  ```sh
package/README.zh-CN.md CHANGED
@@ -9,13 +9,14 @@
9
9
  ## 特性
10
10
 
11
11
  - **真实浏览器渲染** —— 以用户视角加载页面,SPA 客户端渲染内容也能抓到,而非只有原始 HTML。
12
- - **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 `![alt](data:image/png;base64,...8.9KB)`,避免 base64 字符流刷屏。
12
+ - **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 `![alt](data:image/png;base64,...8.9KB)`,避免 base64 字符流刷屏。对列表型页面(门户首页、商品目录——正文本身就是链接),Readability 的链接密度打分会误选出噪声碎片(实测 163.com 首页:96 字符的页脚备案块胜出,320 条新闻标题得分趋零),因此抽取结果需通过**覆盖率置信门**——不足 250 字符、或占页面可见文本不足 10% 即不予采信,转而回退到整页消毒转换,此类页面上可基本完整保留列表内容。通过置信门的抽取结果再做**尾部链接密集块修剪**——正文之后的推荐流、分享栏往往随正文一起被裹进胜出的容器里,且能躲过所有基于类名的规则(163 的 `post_recommend` 卡片单卡链接密度 0.42–0.48,恰好低于 Readability 自己 0.5 的删除线,且抽取后类名尽失、无从匹配);只有当尾部连续块**既占容器文本少数、又明显比其前的正文更"链接化"**时才整体移除,因此链接列表型抽取(信息流、门户头条)不受影响。
13
13
  - **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
14
14
  - **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
15
15
  - **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
16
16
  - **热配置** —— 插件页(侧边栏 → 插件 → *dsh-web-fetch-playwright*,dsh ≥ 0.1.7)在插件自己的包页面直接展示完整表单(也可通过该行的「配置」入口进入同一表单);可随时切换后端、上下文模式、降噪开关与并发数,提交的改动对下一次抓取即时生效,无需重启。
17
17
  - **预算控制** —— 单次抓取 45s 超时;并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 20s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
18
18
  - **Cloudflare 挑战有界等待** —— 导航落到验证中间页("Just a moment…" 及其多语言同族,通过官方 `cf-mitigated: challenge` 响应头 + 结构性页面标记识别)时,抓取保持**同一标签页与上下文**,等待浏览器自行通过验证:跟踪*最后一次*主 frame 响应(真实页面随后重载进来),并轮询活 DOM 以捕获 SPA 式清除。有界且可配置(`challengeWaitMs`,默认 15s;`0` 恢复旧版首响应行为),附带同标签页有界重试(`challengeRetries`,默认 1)。预算耗尽时以独立的 `WEB_FETCH_CHALLENGE` 错误码明确失败,而不是把中间页当正文返回。全程不点击、不注入验证码答案、不伪造浏览器状态、不导出或复制 cookie。
19
+ - **视口虚拟化信息流与应用壳文档的深度抓取** —— 论坛帖子、评论区、时间线类页面越来越多只渲染滚动位置附近的内容(Discourse 帖子流把视口外的每一楼都替换成只保留楼层 id、不含任何正文的高度占位符),应用壳文档编辑器(飞书文档一族)更进一步:body 根本不滚动、块以 `data-block-id` 标识而没有 id、窗口外的块被直接移除、只留匿名高度占位符。静止快照只会拿到骨架或首屏而正文静默丢失。抓取因此在序列化前执行一次有界的页内滚动遍历:文档可滚时走窗口,否则走页面主导的内层滚动容器;暂存每个渲染窗口的内容(每个 id/`data-block-id` 保留最丰满的快照),逐步走到底部——只有"触底且高度停止增长"才意味着分页加载完毕——然后在页面的结构化克隆里用暂存快照再水合所有退化的键元素、把被移除的键插回其存活兄弟旁边,并由遍历自行序列化该克隆(活文档从不被改写)。对滚动毫无反应的页面(绝大多数网页)会被识别出来并跳过慢速遍历。由 `scrollBudgetMs` 约束(默认 20s;`0` 恢复旧版单瞬间抓取行为);任何失败都静默降级为旧版抓取。
19
20
 
20
21
  ## 工作原理
21
22
 
@@ -29,8 +30,8 @@ web_fetch (tool-web)
29
30
  └─ ctx.web.fetchProvider = playwright
30
31
  ├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
31
32
  ├─ cdp: connectOverCDP(endpoint)
32
- ├─ page.goto → 等待稳定(networkidle,尽力而为)→ 在活 DOM 中剥离脚本子树 → page.content()
33
- ├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → DOMPurify → Turndown(GFM)
33
+ ├─ page.goto → 等待稳定(networkidle,尽力而为)→ 深度抓取滚动遍历(虚拟化信息流;有还原时直接返回其合并且已剥离的结果)→ 否则在活 DOM 中剥离脚本子树 → page.content()
34
+ ├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → 尾部链接密集块修剪 → DOMPurify → Turndown(GFM)
34
35
  └─ Markdown(关闭降噪时返回原始 HTML)
35
36
  ```
36
37
 
@@ -79,6 +80,7 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
79
80
  | `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;20s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
80
81
  | `challengeWaitMs` | `15000` | Cloudflare 挑战的**有界**自然等待上限(毫秒,0–60000),在同一标签页内等待浏览器自行通过验证。`0` 关闭整条挑战处理链路——直接返回首次响应(0.2.5 之前的旧行为)。 |
81
82
  | `challengeRetries` | `1` | 一个等待窗口耗尽后的**同标签页**重新导航次数(0–3);浏览器已拿到的通关 cookie 留在上下文里供重试使用。总耗时始终受 45s 单次抓取预算约束。 |
83
+ | `scrollBudgetMs` | `20000` | 深度抓取滚动遍历的**有界**预算(毫秒,0–30000)。`0` 关闭——按页面静止状态抓取(旧行为),会丢失视口虚拟化信息流的正文。 |
82
84
 
83
85
  本地后端解析顺序:
84
86
 
@@ -130,6 +132,31 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
130
132
 
131
133
  安全边界(刻意为之):不点击 Turnstile、不解验证码、不注入 token、不伪装指纹/UA、不轮换代理、不导出 cookie——CDP 隔离模式下通关态保留在对应会话 / 全局 context 中,直到该 context 被清理(本地 context 仍随每次抓取结束);profile 模式下它留在远端浏览器自己的 profile 里,插件从不复制或清理。等待始终受 `challengeWaitMs` 与 45s 单次抓取预算双重约束,永不无限阻塞。
132
134
 
135
+ ### 深度抓取(视口虚拟化信息流与应用壳文档)
136
+
137
+ 信息流形态的页面——Discourse 帖子流(linux.do、meta.discourse.org)、评论区、时间线——只渲染滚动位置附近的条目。静止时活 DOM 里只有一个窗口的真实内容,窗口外的每个条目都是只保留身份、不含内容的高度占位符:
138
+
139
+ ```html
140
+ <div class="post-stream--cloaked" data-post-number="1" id="post_1" style="height:297.9px !important;…">
141
+ <h2 class="sr-only">由 Diver-QC 于 2 小时前 发布</h2>
142
+ </div>
143
+ ```
144
+
145
+ 渲染后从不滚动、只序列化某一瞬间的抓取,交回的就是骨架——屏幕阅读器专用的"由 X 发布"标题穿过管线存活下来、看起来像正文,真正的楼层正文全部消失;连降噪管线的覆盖率门控也发现不了(页面自身的可见文本和提取结果是同一份被窗口化的残片)。只滚动到底部一次同样不行:渲染窗口是**移动的**——在一个 40 楼的 linux.do 帖子上实测,静止 DOM 只有前 ~6 楼,一步跳到底只剩最后 10 楼,而且第一块之后的楼层根本不在 DOM 里,要等触底 sentinel 拉取。
146
+
147
+ 应用壳文档编辑器把同样的病理又下沉了一层——实测坏例是飞书 wiki 页面(`bocha-ai.feishu.cn/wiki/…`):body 根本不滚动,html/body 被锁死在恰好一屏,文档在 `div.bear-web-x-container` 里滚(实测:`documentElement.scrollHeight` 720 == `innerHeight`,`window.scrollTo(0, 800)` 后 `scrollY` 仍为 0,而容器以 656px 的 client 撑着 4534px)。它的块虚拟化比 Discourse 多两个变化:块以 `data-block-id` 标识、**没有 id**;窗口外的块被**直接移除**——整段未渲染尾部只是一个匿名的 `.bear-virtual-renderUnit-placeholder` 显式高度占位符——于是既没有窗口滚动条可走、没有 id 可暂存、也没有原地退化的行可换回,抓取只会返回静态渲染的首屏(实测:7 个块、892 字;完整文档约 190 块)。
148
+
149
+ 修复是一次有界的页内滚动遍历(`src/autoscroll.ts`),在每个 HTML 文档的 settle 之后、序列化剥除之前执行:
150
+
151
+ 1. **解析遍历目标**:文档自身可滚时走窗口(Discourse 形态);否则找页面的主导内层滚动容器(应用壳形态)——在 computed `overflow-y: auto|scroll` 的元素里取可滚距离最大者,且门槛为半个视口,下拉框级别的小滚动容器永远不会触发遍历。两者皆无则不遍历。
152
+ 2. **暂存**每个键元素——以元素 id 为键,无 id 则用 `data-block-id`(加命名空间前缀,两种属性永不冲突)——为脱离文档的克隆,每个键保留**最丰满**的一次观测(先以占位符形态被看到的行,会在渲染形态出现时升级),并记录其最近的带键兄弟:即它在自己所属列表中的位置。单条体积上限约 64KB:虚拟化行只有几 KB,而几十 KB 的元素只可能是整个信息流的容器(linux.do 实测:六层嵌套包装加上 `data-preloaded` 和 `svg-sprites`,一次收集就把 2MB 预算烧掉 1.9MB),这类键会被记住、此后不再序列化。
153
+ 3. **遍历**:从顶部起每次下移 70% 目标视口高度,每步为 300ms 平滑动画(按时间插值;`scrollTop` 驱动内层容器与 `window.scrollTo` 驱动文档完全同效)加 300ms 静置供信息流引擎的防抖 uncloak 生效;直到能在底部持续 2.5s **耐心窗口**而高度无增长(真实信息流的分块拉取是网络往返,任何增长都会重新计时),或步数/预算上限关闭遍历。一个 `MutationObserver` 观察遍历存在的全部理由——页面因滚动而增删元素。连续两个谨慎步都无反应时,页面被视为对滚动惰性,剩余部分用一次跳到底部来探测;跳跃后一旦出现任何反应(uncloak、重新 cloak、加载更多的追加),遍历回退到起跳位置并谨慎走完。正确性从不依赖热身阶段的判断,只有惰性页面的开销依赖它:约 30 屏的静态长文从耗尽 20s 预算降到约 4.5s。
154
+ 4. **合并**到结构化克隆(`document.cloneNode(true)`)中:没有序列化/重解析往返,运行时改写进脚本文本的字面 `</script>` 无法污染合并结果;克隆没有浏览上下文,不执行脚本、不升级自定义元素、不加载 frame。**插回**(仅容器遍历):来自滚动容器内、而克隆中已不存在的暂存键是被引擎**移除**的(飞书形态,占位符是匿名的),插回到其记录的兄弟旁边,迭代求解,使一段被移除的连续块能从任一存活端链式还原;随已插回父级一同回来的键直接跳过。定位是结构性的,与遍历**看到**各行的顺序无关——包括从文档中部起步的锚点打开。**替换**:每个比快照更空的键元素(子元素更少,或文本明显更少——即 Discourse 占位符)被替换为快照,但**容器**除外(其后代中有待替换元素的祖先),容器由这些后代拼装还原——容器自身的快照只是某一窗口的局部视图。随后克隆经过与活文档读取相同的脚本剥除(`src/strip.ts`)并序列化,provider 读取该字符串而非活页面。没有还原任何内容的遍历不返回 HTML,活文档读取与以往完全一致。
155
+
156
+ 遍历在两种降噪模式下都运行;受 `scrollBudgetMs`(默认 20s;`0` 恢复旧版单瞬间行为)与剩余抓取预算减去解码储备的双重约束。任何失败——页面没有 `evaluate`、求值被拒、引擎拒绝遍历——都静默降级为旧版抓取:遍历只会增加内容,不会让抓取失败。CDP profile 模式下遍历滚动的是远端浏览器里抓取自有的那个标签页,与渲染发生在同一处。
157
+
158
+ 真实坏例实测:linux.do 帖子(51 楼、约 18kpx)在 20s 预算内一次遍历走完,每楼都被再水合、零占位符残留;飞书 wiki 页面在 20s 预算中用约 11.5s 走完内层容器(15 步、插回 38 个被移除的块、最终 DOM 191 块——静止抓取只有 7 块),深部尾文(两处 JSON 示例与完整段落)都进入 markdown。一项如实记录的残余限制:飞书表格块的**单元格**藏在第二层块内虚拟化后面,遍历不追——表头与行能还原,超大载荷单元格可能保持为空。
159
+
133
160
  ## 开发
134
161
 
135
162
  ```sh
package/lib/client.js CHANGED
@@ -371,7 +371,8 @@ window.__ModuleLoader__.load({
371
371
  radioField("contextIsolationScope", ["session", "global"]),
372
372
  checkboxField("denoise"),
373
373
  numberField("maxConcurrency", 1, 200),
374
- numberField("challengeWaitMs", 0, 6e4)
374
+ numberField("challengeWaitMs", 0, 6e4),
375
+ numberField("scrollBudgetMs", 0, 3e4)
375
376
  ]);
376
377
  this.store = this.form.bind(() => this.projection());
377
378
  }
@@ -385,7 +386,8 @@ window.__ModuleLoader__.load({
385
386
  contextIsolationScope: this.form.field("contextIsolationScope"),
386
387
  denoise: this.form.field("denoise"),
387
388
  maxConcurrency: this.form.field("maxConcurrency"),
388
- challengeWaitMs: this.form.field("challengeWaitMs")
389
+ challengeWaitMs: this.form.field("challengeWaitMs"),
390
+ scrollBudgetMs: this.form.field("scrollBudgetMs")
389
391
  };
390
392
  }
391
393
  /**
@@ -415,12 +417,12 @@ window.__ModuleLoader__.load({
415
417
  document.head.appendChild(tag);
416
418
  }
417
419
  var PluginCard_module_css_default = {
418
- "footer": "UM8-Rq_footer",
419
420
  "form": "UM8-Rq_form",
421
+ "footer": "UM8-Rq_footer",
420
422
  "failed": "UM8-Rq_failed",
421
- "save": "UM8-Rq_save",
422
423
  "readOnly": "UM8-Rq_readOnly",
423
- "discard": "UM8-Rq_discard"
424
+ "discard": "UM8-Rq_discard",
425
+ "save": "UM8-Rq_save"
424
426
  };
425
427
  //#endregion
426
428
  //#region src/client/PluginCard.tsx
@@ -480,27 +482,27 @@ window.__ModuleLoader__.load({
480
482
  document.head.appendChild(tag);
481
483
  }
482
484
  var fields_module_css_default = {
483
- "radioHint": "CFSaVG_radioHint",
484
- "checkboxRow": "CFSaVG_checkboxRow",
485
- "label": "CFSaVG_label",
486
- "field": "CFSaVG_field",
487
- "hint": "CFSaVG_hint",
488
- "radioGroup": "CFSaVG_radioGroup",
489
- "radioContent": "CFSaVG_radioContent",
485
+ "fieldEmbedded": "CFSaVG_fieldEmbedded",
490
486
  "head": "CFSaVG_head",
491
- "badges": "CFSaVG_badges",
492
- "radioText": "CFSaVG_radioText",
493
487
  "badge": "CFSaVG_badge",
494
- "radioOptionPlain": "CFSaVG_radioOptionPlain",
488
+ "reset": "CFSaVG_reset",
489
+ "radioGroup": "CFSaVG_radioGroup",
490
+ "radioText": "CFSaVG_radioText",
491
+ "radioContent": "CFSaVG_radioContent",
492
+ "hint": "CFSaVG_hint",
493
+ "checkboxRow": "CFSaVG_checkboxRow",
494
+ "radioLabel": "CFSaVG_radioLabel",
495
+ "radioHint": "CFSaVG_radioHint",
495
496
  "radioPick": "CFSaVG_radioPick",
496
- "fieldEmbedded": "CFSaVG_fieldEmbedded",
497
497
  "input": "CFSaVG_input",
498
- "inputInvalid": "CFSaVG_inputInvalid",
499
- "radioOption": "CFSaVG_radioOption",
500
- "invalid": "CFSaVG_invalid",
501
498
  "hintIndented": "CFSaVG_hintIndented",
502
- "reset": "CFSaVG_reset",
503
- "radioLabel": "CFSaVG_radioLabel"
499
+ "badges": "CFSaVG_badges",
500
+ "invalid": "CFSaVG_invalid",
501
+ "radioOption": "CFSaVG_radioOption",
502
+ "radioOptionPlain": "CFSaVG_radioOptionPlain",
503
+ "label": "CFSaVG_label",
504
+ "field": "CFSaVG_field",
505
+ "inputInvalid": "CFSaVG_inputInvalid"
504
506
  };
505
507
  //#endregion
506
508
  //#region src/client/fields.tsx
@@ -846,6 +848,23 @@ window.__ModuleLoader__.load({
846
848
  onReset: () => {
847
849
  props.resetField("challengeWaitMs");
848
850
  }
851
+ }),
852
+ /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
853
+ id: "plugin-config-playwright-scroll-budget",
854
+ label: t("scrollBudgetMs"),
855
+ hint: t("scrollBudgetMsHint"),
856
+ placeholder: t("scrollBudgetMsPlaceholder"),
857
+ overriddenLabel: t("overridden"),
858
+ resetLabel: t("reset"),
859
+ invalidLabel: t("invalidText"),
860
+ disabled,
861
+ ...state.scrollBudgetMs,
862
+ onEdit: (text) => {
863
+ props.edit("scrollBudgetMs", text);
864
+ },
865
+ onReset: () => {
866
+ props.resetField("scrollBudgetMs");
867
+ }
849
868
  })
850
869
  ]
851
870
  });
@@ -883,6 +902,9 @@ window.__ModuleLoader__.load({
883
902
  challengeWaitMs: "Cloudflare challenge wait (ms)",
884
903
  challengeWaitMsHint: "Bounded wait for a Cloudflare challenge to clear naturally in the same tab (0–60000; 0 = off). 0 disables: the first response is returned as-is.",
885
904
  challengeWaitMsPlaceholder: "(default: 15000)",
905
+ scrollBudgetMs: "Deep-capture scroll budget (ms)",
906
+ scrollBudgetMsHint: "Bounded in-page scroll walk that captures viewport-virtualized feeds (forum topics, comment threads) whole: posts outside the render window are empty placeholders at rest. 0–30000; 0 = off (capture the page as it rests).",
907
+ scrollBudgetMsPlaceholder: "(default: 20000)",
886
908
  overridden: "Overridden",
887
909
  reset: "Reset to default",
888
910
  readOnly: "This deployment stores settings read-only.",
@@ -923,6 +945,9 @@ window.__ModuleLoader__.load({
923
945
  challengeWaitMs: "Cloudflare 挑战等待上限(毫秒)",
924
946
  challengeWaitMsHint: "在同一标签页内有界等待 Cloudflare 验证自然通过(0–60000;0 = 关闭)。关闭时直接返回首次响应——旧版行为。",
925
947
  challengeWaitMsPlaceholder: "(默认:15000)",
948
+ scrollBudgetMs: "深度抓取滚动预算(毫秒)",
949
+ scrollBudgetMsHint: "有界的页内滚动遍历,完整捕获视口虚拟化的信息流(论坛帖子、评论区):静止时渲染窗口外的帖子只是空占位符。0–30000;0 = 关闭(按页面静止状态抓取)。",
950
+ scrollBudgetMsPlaceholder: "(默认:20000)",
926
951
  overridden: "已覆盖",
927
952
  reset: "恢复默认",
928
953
  readOnly: "本部署的设置为只读。",