dsh-web-fetch-playwright 0.2.19 → 0.2.22

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -15,9 +15,9 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
15
15
  - **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
16
16
  - **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
17
17
  - **Live configuration** — the Plugins page (侧边栏 → 插件 → *dsh-web-fetch-playwright*, dsh ≥ 0.1.7) shows the full form right on the plugin's package page (and on the row's own page via its configure control); it edits the backend, context mode, denoise toggle, and concurrency, and a committed change reaches the next fetch without a restart.
18
- - **Budget-aware** — per-fetch deadline (45s); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
19
- - **Bounded Cloudflare-challenge wait** — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented `cf-mitigated: challenge` response header plus structural page markers), the fetch keeps the **same tab and context** and waits for the browser's own verification to clear it — tracking the *last* main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded and configurable (`challengeWaitMs`, default 15s; 0 restores the legacy first-response behavior), with a bounded same-tab retry (`challengeRetries`, default 1). When the budget runs out, the fetch fails with the distinct `WEB_FETCH_CHALLENGE` error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.
20
- - **Deep capture for viewport-virtualized feeds and app-shell documents** — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by `data-block-id` with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/`data-block-id`), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot and splices the removed ones back beside their surviving siblings, in a structural clone of the page that it serializes itself (the live page is never rewritten). Pages that never react to scrolling — most of the web — are detected and skip the slow walk. Bounded by `scrollBudgetMs` (default 20s; 0 restores the legacy single-instant behavior); any failure silently degrades to the legacy capture.
18
+ - **Budget-aware** — per-fetch waiting budget (25s) — when it runs out (a slow load, an endless scroll feed) the fetch stops waiting and extracts what the page already rendered instead of failing (`truncated` when the document never finished loading; `WEB_FETCH_TIMEOUT` only when nothing arrived at all or extraction itself hangs past a 10s grace); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 10s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
19
+ - **Bounded Cloudflare-challenge wait** — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented `cf-mitigated: challenge` response header plus structural page markers), the fetch keeps the **same tab and context** and waits for the browser's own verification to clear it — tracking the *last* main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded by a built-in 15s wait, with a bounded same-tab retry (`challengeRetries`, default 1). When the window runs out, a bounded **plain-HTTP refetch** of the same URL runs through the page's own request context (`challengeHttpFallback`, default on) — the same cookie jar and the browser's UA, but none of Chromium's render decisions — because many edges serve the real document to such a request while withholding the interactive challenge from the browser session; the answer must itself be a non-challenge document or the fetch still fails. Failing that, the fetch fails with the distinct `WEB_FETCH_CHALLENGE` error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.
20
+ - **Deep capture for viewport-virtualized feeds and app-shell documents** — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by `data-block-id` with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/`data-block-id`), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot, splices the removed ones back beside their surviving siblings, and restores rows that carry no DOM identity at all (react-window lists, X-style timelines) from the moment the engine removed them, in a structural clone of the page that it serializes itself (the live page is never rewritten). Pages that never react to scrolling — most of the web — are detected and skip the slow walk. Bounded by a built-in 20s budget; any failure silently degrades to the legacy capture.
21
21
 
22
22
  ## How it works
23
23
 
@@ -79,10 +79,9 @@ The Plugins page (侧边栏 → 插件) opens the form one click from the list:
79
79
  | `shareBrowserContext` | `true` | CDP only: Share / Isolate radio. `true` uses the remote default context and logins; `false` uses a separate context with the lifecycle determined by the scope below. Ignored locally. |
80
80
  | `contextIsolationScope` | `global` | CDP isolation only: `session` creates a context on demand per DSH session and closes it when no active/pending fetches or open tabs remain (next fetch starts fresh); `global` reuses one fixed isolated context across all DSH sessions within this plugin instance. |
81
81
  | `denoise` | `true` | Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert. |
82
- | `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
83
- | `challengeWaitMs` | `15000` | Bounded wait (ms, 0–60000) for a Cloudflare challenge to clear naturally in the same tab. `0` disables the whole challenge path — the first response is returned as-is (the pre-0.2.5 behavior). |
84
- | `challengeRetries` | `1` | Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 45s per-fetch deadline. |
85
- | `scrollBudgetMs` | `20000` | Bounded budget (ms, 0–30000) for the deep-capture scroll walk. `0` disables it — the page is captured as it rests (the pre-walk behavior), which loses the bodies of viewport-virtualized feeds. |
82
+ | `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 10s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
83
+ | `challengeRetries` | `1` | Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 25s per-fetch waiting budget. |
84
+ | `challengeHttpFallback` | `true` | When the browser cannot clear a challenge inside its window, refetch the same URL once through the page's plain-HTTP request context (same cookies, the browser's UA, no render decision) and return that document when it is not itself a challenge. `false` keeps the browser-only posture. |
86
85
 
87
86
  Local backend resolution order:
88
87
 
@@ -123,16 +122,17 @@ Headless server (pre-seed logins in a headful environment first): `chromium --he
123
122
 
124
123
  ### Cloudflare challenge handling (bounded natural wait)
125
124
 
126
- Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior; reproduce it any time with `challengeWaitMs: 0`, or — from a repo checkout, after `pnpm build` — run `node scripts/challenge-demo.mjs` for a local simulated before/after, and `node scripts/challenge-online.mjs <url>` against a real site).
125
+ Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior; from a repo checkout, after `pnpm build` — run `node scripts/challenge-demo.mjs` for a local simulated before/after, and `node scripts/challenge-online.mjs <url>` against a real site).
127
126
 
128
127
  With the wait on (default):
129
128
 
130
129
  1. **Detection** — a response carrying `cf-mitigated: challenge` (the documented signal for every challenge page type), or a 403/503 HTML document from a `server: cloudflare` edge, or the localized interstitial itself (title family like "Just a moment…", `请稍候…`, "Минутку…", plus structural markers: the challenge-platform runtime's `<script src>`, `#challenge-*` elements, `cf-chl-widget-` frames, `window._cf_chl_opt`). The content-level markers are the *fallback tier* and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (`server: cloudflare` / `cf-ray`) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Two refinements keep that promise for real pages: the challenge-platform script check reads real `<script src>` attributes only (prose, and escaped code samples that quote a path, are ignored), and it matches the runtime mount `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/` *positively*. Everything else Cloudflare injects under that prefix — Bot Management's JavaScript Detections (`scripts/jsd/`, `h/[b|g]/scripts/jsd/<hash>/`, `h/[b|g]/jsd/r/`), and Precursor (`scripts/precursor/`) — is passive instrumentation present on *every normal page* of a protected zone (e.g. openrouter.ai, nodeseek.com), so it is never read as a challenge. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
131
130
  2. **Bounded wait, same tab and context** — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the *last* main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
132
131
  3. **Bounded retry** — when a window runs out, the same tab re-navigates once (default; `challengeRetries`) with whatever clearance cookies the context already holds.
133
- 4. **Clear failure** — `WEB_FETCH_CHALLENGE` (a provider-specific code the web seam's open-string `code` allows) naming the site, the budget spent, and the last challenge status.
132
+ 4. **Dead-end refetch** — before failing, one plain-HTTP request for the same URL through the page's request context (`challengeHttpFallback`, default on): the same cookie jar and the browser's User-Agent, none of Chromium's render decisions. Many Cloudflare edges serve the real document to such a request while withholding the interactive challenge from the browser session itself (measured live: v2ex.com from a datacenter IP — the browser drew an empty widget frame for 15s×2, the plain request drew the full article). The answer must itself be a non-challenge HTML document — checked at response level AND content level by the same classifiers that gate the browser path — or the fetch keeps the browser-only verdict.
133
+ 5. **Clear failure** — `WEB_FETCH_CHALLENGE` (a provider-specific code the web seam's open-string `code` allows) naming the site, the budget spent, and the last challenge status.
134
134
 
135
- Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by `challengeWaitMs` and the 45s per-fetch deadline; nothing blocks forever.
135
+ Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by the built-in 15s wait and the 25s per-fetch waiting budget; nothing blocks forever.
136
136
 
137
137
  ### Deep capture (viewport-virtualized feeds and app-shell documents)
138
138
 
@@ -151,13 +151,13 @@ App-shell document editors have the same pathology one level deeper — the meas
151
151
  The fix is a bounded in-page scroll walk (`src/autoscroll.ts`), run after the settle and before the serialization strip for every HTML document:
152
152
 
153
153
  1. **Resolve the walk target**: the window when the document itself scrolls (the Discourse shape); otherwise the page's dominant inner scroll container (the app-shell shape) — the element with the largest scrollable distance among computed `overflow-y: auto|scroll` elements, gated at half a viewport so dropdown-sized scrollers never trigger a walk. Neither scrollable means no walk.
154
- 2. **Stash** every keyed element — keyed by id, else `data-block-id` (namespaced so the two can never collide) — as a detached clone, keeping the *richest* sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus `data-preloaded` and `svg-sprites` ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
155
- 3. **Walk** down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; `scrollTop` drives an inner container exactly the way `window.scrollTo` drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s *patience window* with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A `MutationObserver` watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does: a ~30-viewport static article costs ~4.5s instead of exhausting the 20s budget.
156
- 4. **Merge** into a structural clone (`document.cloneNode(true)`): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal `</script>` in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. *Splice* (container walks): stashed keys from inside the scroll container that the clone no longer holds were **removed** by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk *saw* rows in — including an anchored open that starts mid-document. *Swap*: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except *containers* (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. The clone is then stripped of scripting elements (the same `src/strip.ts` routine the live read uses) and serialized; the provider reads that string instead of the live page. A walk that restored nothing hands back no HTML, and the live read proceeds exactly as before.
154
+ 2. **Stash** every keyed element — keyed by id, else `data-block-id` (namespaced so the two can never collide) — as a detached clone (framework-generated ids are not keys: Ember's `ember123` and React `useId` values such as `:r1:`, `«r1»` and `_r_1_` change on every render, so such elements take the unkeyed path below), keeping the *richest* sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus `data-preloaded` and `svg-sprites` ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
155
+ 3. **Walk** down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; `scrollTop` drives an inner container exactly the way `window.scrollTo` drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s *patience window* with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A `MutationObserver` watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does. The same observer *harvests unkeyed rows*: when the engine removes an element that has no key, the record still hands over the detached element whole, plus the sibling nodes it sat between, and that is where the merge puts it back. No layout is read, so absolute positioning and `transform` do not matter. Each row identity is stashed once per list (an engine that re-renders every frame removes the same row thousands of times; later sightings only anchor), from its own character budget so it can never crowd out the keyed stash. Transient UI is never harvested (`role=status|alert|dialog|tooltip|progressbar|menu|…`, `aria-hidden`, `aria-busy`, elements without text). A page that never reacted and issued no fetch/XHR holds only 900ms at the bottom instead of the full 2.5s patience. Measured: a ~30-viewport static article costs ~2.4s instead of exhausting the 20s budget, and a page that does not scroll pays nothing.
156
+ 4. **Merge** into a structural clone (`document.cloneNode(true)`): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal `</script>` in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. *Splice* (container walks): stashed keys from inside the scroll container that the clone no longer holds were **removed** by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk *saw* rows in — including an anchored open that starts mid-document. *Swap*: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except *containers* (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. *Restore* (unkeyed rows, before the swap): a list qualifies only if the engine both removed text-bearing rows from it and added elements to it during the walk. A lone toast or a load-more append does not qualify, and neither does anything inside an `aria-live="assertive"`, `role=alert|status|timer|marquee` or carousel region. Each removed row the list no longer holds goes back next to the sibling it was removed beside. An anchor that was itself removed is followed to *its* neighbor, and a repeat sighting resolves to wherever the row's stashed copy lands. Every sighting of a row is tried, so the first window of an anchored open is placed by the later pass that saw it beside its neighbors. A list re-rendered wholesale (`innerHTML`) leaves no surviving anchor; its rows go before the first text-bearing row, in removal order, because a top-down walk that ends at the bottom only loses what lay above the final window. De-duplication against the rows still present uses a per-list identity attribute when the engine sets one (`data-index`, `data-item-index`, `aria-rowindex`, `aria-posinset`, `data-key`, `data-id`, …, scoped to the list so repeated values across lists cannot collide), else the row's text. A present row with the same identity attribute that is far emptier is a placeholder and is replaced. An unkeyed stub the engine put in a row's place (short text the row contains, right beside it) is absorbed. A restored list and its ancestors count as containers for the swap, so a keyed wrapper is never replaced by its one-window snapshot. The clone is then stripped of scripting elements (the same `src/strip.ts` routine the live read uses) and serialized; the provider reads that string instead of the live page. If a page reacted but nothing could be restored (rows without text, canvas rendering), the walk scrolls back to where the page opened and waits for it to settle before the live read. Otherwise the provider would read the walk's last (bottom) window instead of the opening screen the legacy capture saw. A scroll-inert page is left where it is, since its DOM is the same either way.
157
157
 
158
- The walk runs in both denoise modes and is bounded by `scrollBudgetMs` (default 20s; `0` restores the legacy single-instant behavior) and by the remaining fetch deadline minus a decode reserve. Any failure — a page without `evaluate`, a rejected evaluation, an engine that refuses the walk — silently degrades to the legacy capture; a walk can only add content, never fail a fetch. In CDP profile mode the walk scrolls the fetch-owned tab in the remote browser, exactly where the fetch already renders it.
158
+ The walk runs in both denoise modes and is bounded by a built-in 20s budget and by the fetch's remaining waiting budget; a walk cut short by the budget still merges and returns what it stashed, and the content read plus decode run in the 10s extraction grace window after it. Any failure — a page without `evaluate`, a rejected evaluation, an engine that refuses the walk — silently degrades to the legacy capture; a walk can only add content, never fail a fetch. In CDP profile mode the walk scrolls the fetch-owned tab in the remote browser, exactly where the fetch already renders it.
159
159
 
160
- Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. One residual limitation, documented as measured: Feishu table blocks lazy-render their *cells* behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.
160
+ Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. On the public react-window demo (`/list/fixed-row-height`, 1,000 unkeyed rows), the walk previously returned only its final window (rows 129–142); the unkeyed restore returns rows 1–155 in order with no duplicates, stopping where the 20s budget ends. Remaining limits of the unkeyed path: rows that hold only images or render to canvas cannot be restored (the walk rewinds instead), and engines that *recycle* row nodes in place (rewriting a pooled node's text rather than removing it) are not harvested. One residual limitation, documented as measured: Feishu table blocks lazy-render their *cells* behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.
161
161
 
162
162
  ### PDF links
163
163
 
package/README.zh-CN.md CHANGED
@@ -15,9 +15,9 @@
15
15
  - **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
16
16
  - **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
17
17
  - **热配置** —— 插件页(侧边栏 → 插件 → *dsh-web-fetch-playwright*,dsh ≥ 0.1.7)在插件自己的包页面直接展示完整表单(也可通过该行的「配置」入口进入同一表单);可随时切换后端、上下文模式、降噪开关与并发数,提交的改动对下一次抓取即时生效,无需重启。
18
- - **预算控制** —— 单次抓取 45s 超时;并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 20s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
19
- - **Cloudflare 挑战有界等待** —— 导航落到验证中间页("Just a moment…" 及其多语言同族,通过官方 `cf-mitigated: challenge` 响应头 + 结构性页面标记识别)时,抓取保持**同一标签页与上下文**,等待浏览器自行通过验证:跟踪*最后一次*主 frame 响应(真实页面随后重载进来),并轮询活 DOM 以捕获 SPA 式清除。有界且可配置(`challengeWaitMs`,默认 15s;`0` 恢复旧版首响应行为),附带同标签页有界重试(`challengeRetries`,默认 1)。预算耗尽时以独立的 `WEB_FETCH_CHALLENGE` 错误码明确失败,而不是把中间页当正文返回。全程不点击、不注入验证码答案、不伪造浏览器状态、不导出或复制 cookie。
20
- - **视口虚拟化信息流与应用壳文档的深度抓取** —— 论坛帖子、评论区、时间线类页面越来越多只渲染滚动位置附近的内容(Discourse 帖子流把视口外的每一楼都替换成只保留楼层 id、不含任何正文的高度占位符),应用壳文档编辑器(飞书文档一族)更进一步:body 根本不滚动、块以 `data-block-id` 标识而没有 id、窗口外的块被直接移除、只留匿名高度占位符。静止快照只会拿到骨架或首屏而正文静默丢失。抓取因此在序列化前执行一次有界的页内滚动遍历:文档可滚时走窗口,否则走页面主导的内层滚动容器;暂存每个渲染窗口的内容(每个 id/`data-block-id` 保留最丰满的快照),逐步走到底部——只有"触底且高度停止增长"才意味着分页加载完毕——然后在页面的结构化克隆里用暂存快照再水合所有退化的键元素、把被移除的键插回其存活兄弟旁边,并由遍历自行序列化该克隆(活文档从不被改写)。对滚动毫无反应的页面(绝大多数网页)会被识别出来并跳过慢速遍历。由 `scrollBudgetMs` 约束(默认 20s;`0` 恢复旧版单瞬间抓取行为);任何失败都静默降级为旧版抓取。
18
+ - **预算控制** —— 单次抓取 25s 等待预算——预算耗尽时(加载过慢、无限下拉的信息流)不再继续等待,而是直接对页面已渲染的内容做提取并返回,而不是报错(文档未加载完成时标记 `truncated`;只有完全没收到响应、或提取本身超出 10s 宽限时才报 `WEB_FETCH_TIMEOUT`);并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 10s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
19
+ - **Cloudflare 挑战有界等待** —— 导航落到验证中间页("Just a moment…" 及其多语言同族,通过官方 `cf-mitigated: challenge` 响应头 + 结构性页面标记识别)时,抓取保持**同一标签页与上下文**,等待浏览器自行通过验证:跟踪*最后一次*主 frame 响应(真实页面随后重载进来),并轮询活 DOM 以捕获 SPA 式清除。有界(内置 15s 上限),附带同标签页有界重试(`challengeRetries`,默认 1)。窗口耗尽时,先经由页面自身的纯 HTTP 请求通道对同一 URL 做一次**再拉取**(`challengeHttpFallback`,默认开启)——同一个 cookie 罐、浏览器自身的 UA,但没有 Chromium 的渲染决策——因为很多 Cloudflare 边缘会对这样的请求直接下发真实文档,却对浏览器会话扣住交互式挑战不放;回取的响应必须本身不是挑战页,否则仍按失败处理。仍失败时,才以独立的 `WEB_FETCH_CHALLENGE` 错误码明确失败,而不是把中间页当正文返回。全程不点击、不注入验证码答案、不伪造浏览器状态、不导出或复制 cookie。
20
+ - **视口虚拟化信息流与应用壳文档的深度抓取** —— 论坛帖子、评论区、时间线类页面越来越多只渲染滚动位置附近的内容(Discourse 帖子流把视口外的每一楼都替换成只保留楼层 id、不含任何正文的高度占位符),应用壳文档编辑器(飞书文档一族)更进一步:body 根本不滚动、块以 `data-block-id` 标识而没有 id、窗口外的块被直接移除、只留匿名高度占位符。静止快照只会拿到骨架或首屏而正文静默丢失。抓取因此在序列化前执行一次有界的页内滚动遍历:文档可滚时走窗口,否则走页面主导的内层滚动容器;暂存每个渲染窗口的内容(每个 id/`data-block-id` 保留最丰满的快照),逐步走到底部——只有"触底且高度停止增长"才意味着分页加载完毕——然后在页面的结构化克隆里用暂存快照再水合所有退化的键元素、把被移除的键插回其存活兄弟旁边,并把完全没有 DOM 身份的行(react-window 列表、X 式时间线)从引擎移除它们的那一刻起还原回来,并由遍历自行序列化该克隆(活文档从不被改写)。对滚动毫无反应的页面(绝大多数网页)会被识别出来并跳过慢速遍历。受内置 20s 预算约束;任何失败都静默降级为旧版抓取。
21
21
 
22
22
  ## 工作原理
23
23
 
@@ -79,10 +79,9 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
79
79
  | `shareBrowserContext` | `true` | 仅 CDP:单选共享 / 隔离。`true` 复用远端默认 context 与登录态;`false` 使用独立 context,生命周期由下方范围决定。本地后端忽略。 |
80
80
  | `contextIsolationScope` | `global` | 仅 CDP 隔离模式:`session` 按 DSH 会话按需创建 context,无活动/待处理抓取且无打开的标签页后关闭(下次抓取使用全新状态);`global` 在同一插件实例内跨DSH会话复用一个固定的隔离上下文。 |
81
81
  | `denoise` | `true` | 是否启用降噪;关闭时返回整页渲染 HTML,交由工具层转换。 |
82
- | `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;20s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
83
- | `challengeWaitMs` | `15000` | Cloudflare 挑战的**有界**自然等待上限(毫秒,0–60000),在同一标签页内等待浏览器自行通过验证。`0` 关闭整条挑战处理链路——直接返回首次响应(0.2.5 之前的旧行为)。 |
84
- | `challengeRetries` | `1` | 一个等待窗口耗尽后的**同标签页**重新导航次数(0–3);浏览器已拿到的通关 cookie 留在上下文里供重试使用。总耗时始终受 45s 单次抓取预算约束。 |
85
- | `scrollBudgetMs` | `20000` | 深度抓取滚动遍历的**有界**预算(毫秒,0–30000)。`0` 关闭——按页面静止状态抓取(旧行为),会丢失视口虚拟化信息流的正文。 |
82
+ | `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;10s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
83
+ | `challengeRetries` | `1` | 一个等待窗口耗尽后的**同标签页**重新导航次数(0–3);浏览器已拿到的通关 cookie 留在上下文里供重试使用。总耗时始终受 25s 单次抓取等待预算约束。 |
84
+ | `challengeHttpFallback` | `true` | 浏览器在窗口内清不掉挑战时,经页面自身的纯 HTTP 请求通道对同一 URL 再拉取一次(同 cookie、浏览器 UA、无渲染决策),回取到非挑战文档则直接返回。`false` 保持纯浏览器姿态。 |
86
85
 
87
86
  本地后端解析顺序:
88
87
 
@@ -123,16 +122,17 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
123
122
 
124
123
  ### Cloudflare 挑战处理(有界自然等待)
125
124
 
126
- 部分严格站点会在返回真实页面前先给一个 Cloudflare 验证中间页。真实浏览器通常几秒内就能**自行**通过验证;但只看第一次响应的抓取会把中间页当成正文返回(0.2.5 之前的旧行为;把 `challengeWaitMs` 设为 `0` 可随时复现,或在仓库检出、执行 `pnpm build` 后运行 `node scripts/challenge-demo.mjs` 看本地模拟站点的前后对比、`node scripts/challenge-online.mjs <url>` 对真实站点做在线对比)。
125
+ 部分严格站点会在返回真实页面前先给一个 Cloudflare 验证中间页。真实浏览器通常几秒内就能**自行**通过验证;但只看第一次响应的抓取会把中间页当成正文返回(0.2.5 之前的旧行为;可在仓库检出、执行 `pnpm build` 后运行 `node scripts/challenge-demo.mjs` 看本地模拟站点的前后对比、`node scripts/challenge-online.mjs <url>` 对真实站点做在线对比)。
127
126
 
128
127
  开启等待(默认)后的流程:
129
128
 
130
129
  1. **识别** —— 响应带 `cf-mitigated: challenge`(Cloudflare 官方文档注明所有挑战页类型都带此头),或 403/503 且 `server: cloudflare` 的 HTML 文档,或本地化的中间页本身("Just a moment…" / "请稍候…" / "Минутку…" 等 title 家族,以及结构性标记:挑战运行时的 `<script src>`、`#challenge-*` 元素、`cf-chl-widget-` 框架、`window._cf_chl_opt`)。内容级标记只是**兜底层**,且仅对"挑战兼容"的响应(403/429/503 或来自 Cloudflare 边缘——`server: cloudflare` / `cf-ray`)运行——因为中间页从不会以普通 200 返回,所以正文里引用了挑战文案的普通文章绝不可能被误判。两处收紧让这个承诺真正成立:挑战平台脚本只读真实的 `<script src>` 属性(正文、以及转义后的代码示例里引用的路径一律忽略),并且**正面**匹配运行时挂载点 `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/`。该前缀下 Cloudflare 注入的其余脚本——Bot Management 的 JavaScript Detections(`scripts/jsd/`、`h/[b|g]/scripts/jsd/<hash>/`、`h/[b|g]/jsd/r/`)与 Precursor(`scripts/precursor/`)——都是会注入受保护站点**每一个正常页面**的被动探测脚本(如 openrouter.ai、nodeseek.com),永远不算挑战信号。硬封锁页("Sorry, you have been blocked")单独分类并立即失败——等待无法解除。
131
130
  2. **同标签页、同上下文的有界等待** —— 每 500ms 轮询活 DOM,等浏览器跑完自己的验证;同时跟踪**最后一次主 frame 导航响应**,所以重载进来的真实文档的状态码和响应头才是最终上报的。SPA 式清除(无导航、纯内容替换)由同一个 DOM 探测捕获。
132
131
  3. **有界重试** —— 窗口耗尽后,同一标签页默认再导航一次(`challengeRetries`),上下文里已有的通关 cookie 继续生效。
133
- 4. **明确失败** —— 返回独立的 `WEB_FETCH_CHALLENGE` 错误码(web seam 的 `code` 是开放字符串,允许 provider 专属码),消息中写明站点、等待预算与最后一次挑战响应的状态。
132
+ 4. **死局回取** —— 失败之前,先经页面自身的请求通道对同一 URL 做一次纯 HTTP 再拉取(`challengeHttpFallback`,默认开启):同一个 cookie 罐、浏览器自身的 User-Agent,没有 Chromium 的渲染决策。很多 Cloudflare 边缘对这样的请求直接下发真实文档,却对浏览器会话扣住交互式挑战(实测:数据中心 IP 访问 v2ex.com——浏览器 15s×2 只拿到空的 widget 帧,纯 HTTP 请求拿到完整正文)。回取的响应必须本身不是挑战 HTML——响应级与内容级双重校验,与浏览器路径同一条分类链——否则保持浏览器-only 的判定。
133
+ 5. **明确失败** —— 返回独立的 `WEB_FETCH_CHALLENGE` 错误码(web seam 的 `code` 是开放字符串,允许 provider 专属码),消息中写明站点、等待预算与最后一次挑战响应的状态。
134
134
 
135
- 安全边界(刻意为之):不点击 Turnstile、不解验证码、不注入 token、不伪装指纹/UA、不轮换代理、不导出 cookie——CDP 隔离模式下通关态保留在对应会话 / 全局 context 中,直到该 context 被清理(本地 context 仍随每次抓取结束);profile 模式下它留在远端浏览器自己的 profile 里,插件从不复制或清理。等待始终受 `challengeWaitMs` 与 45s 单次抓取预算双重约束,永不无限阻塞。
135
+ 安全边界(刻意为之):不点击 Turnstile、不解验证码、不注入 token、不伪装指纹/UA、不轮换代理、不导出 cookie——CDP 隔离模式下通关态保留在对应会话 / 全局 context 中,直到该 context 被清理(本地 context 仍随每次抓取结束);profile 模式下它留在远端浏览器自己的 profile 里,插件从不复制或清理。等待始终受内置 15s 上限与 25s 单次抓取等待预算双重约束,永不无限阻塞。
136
136
 
137
137
  ### 深度抓取(视口虚拟化信息流与应用壳文档)
138
138
 
@@ -151,13 +151,13 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
151
151
  修复是一次有界的页内滚动遍历(`src/autoscroll.ts`),在每个 HTML 文档的 settle 之后、序列化剥除之前执行:
152
152
 
153
153
  1. **解析遍历目标**:文档自身可滚时走窗口(Discourse 形态);否则找页面的主导内层滚动容器(应用壳形态)——在 computed `overflow-y: auto|scroll` 的元素里取可滚距离最大者,且门槛为半个视口,下拉框级别的小滚动容器永远不会触发遍历。两者皆无则不遍历。
154
- 2. **暂存**每个键元素——以元素 id 为键,无 id 则用 `data-block-id`(加命名空间前缀,两种属性永不冲突)——为脱离文档的克隆,每个键保留**最丰满**的一次观测(先以占位符形态被看到的行,会在渲染形态出现时升级),并记录其最近的带键兄弟:即它在自己所属列表中的位置。单条体积上限约 64KB:虚拟化行只有几 KB,而几十 KB 的元素只可能是整个信息流的容器(linux.do 实测:六层嵌套包装加上 `data-preloaded` 和 `svg-sprites`,一次收集就把 2MB 预算烧掉 1.9MB),这类键会被记住、此后不再序列化。
155
- 3. **遍历**:从顶部起每次下移 70% 目标视口高度,每步为 300ms 平滑动画(按时间插值;`scrollTop` 驱动内层容器与 `window.scrollTo` 驱动文档完全同效)加 300ms 静置供信息流引擎的防抖 uncloak 生效;直到能在底部持续 2.5s **耐心窗口**而高度无增长(真实信息流的分块拉取是网络往返,任何增长都会重新计时),或步数/预算上限关闭遍历。一个 `MutationObserver` 观察遍历存在的全部理由——页面因滚动而增删元素。连续两个谨慎步都无反应时,页面被视为对滚动惰性,剩余部分用一次跳到底部来探测;跳跃后一旦出现任何反应(uncloak、重新 cloak、加载更多的追加),遍历回退到起跳位置并谨慎走完。正确性从不依赖热身阶段的判断,只有惰性页面的开销依赖它:约 30 屏的静态长文从耗尽 20s 预算降到约 4.5s。
156
- 4. **合并**到结构化克隆(`document.cloneNode(true)`)中:没有序列化/重解析往返,运行时改写进脚本文本的字面 `</script>` 无法污染合并结果;克隆没有浏览上下文,不执行脚本、不升级自定义元素、不加载 frame。**插回**(仅容器遍历):来自滚动容器内、而克隆中已不存在的暂存键是被引擎**移除**的(飞书形态,占位符是匿名的),插回到其记录的兄弟旁边,迭代求解,使一段被移除的连续块能从任一存活端链式还原;随已插回父级一同回来的键直接跳过。定位是结构性的,与遍历**看到**各行的顺序无关——包括从文档中部起步的锚点打开。**替换**:每个比快照更空的键元素(子元素更少,或文本明显更少——即 Discourse 占位符)被替换为快照,但**容器**除外(其后代中有待替换元素的祖先),容器由这些后代拼装还原——容器自身的快照只是某一窗口的局部视图。随后克隆经过与活文档读取相同的脚本剥除(`src/strip.ts`)并序列化,provider 读取该字符串而非活页面。没有还原任何内容的遍历不返回 HTML,活文档读取与以往完全一致。
154
+ 2. **暂存**每个键元素——以元素 id 为键,无 id 则用 `data-block-id`(加命名空间前缀,两种属性永不冲突)——为脱离文档的克隆(框架生成的 id 不算键:Ember 的 `ember123`、React `useId` 的 `:r1:`、`«r1»`、`_r_1_` 每次渲染都会变,这类元素走下文的无键路径),每个键保留**最丰满**的一次观测(先以占位符形态被看到的行,会在渲染形态出现时升级),并记录其最近的带键兄弟:即它在自己所属列表中的位置。单条体积上限约 64KB:虚拟化行只有几 KB,而几十 KB 的元素只可能是整个信息流的容器(linux.do 实测:六层嵌套包装加上 `data-preloaded` 和 `svg-sprites`,一次收集就把 2MB 预算烧掉 1.9MB),这类键会被记住、此后不再序列化。
155
+ 3. **遍历**:从顶部起每次下移 70% 目标视口高度,每步为 300ms 平滑动画(按时间插值;`scrollTop` 驱动内层容器与 `window.scrollTo` 驱动文档完全同效)加 300ms 静置供信息流引擎的防抖 uncloak 生效;直到能在底部持续 2.5s **耐心窗口**而高度无增长(真实信息流的分块拉取是网络往返,任何增长都会重新计时),或步数/预算上限关闭遍历。一个 `MutationObserver` 观察遍历存在的全部理由——页面因滚动而增删元素。连续两个谨慎步都无反应时,页面被视为对滚动惰性,剩余部分用一次跳到底部来探测;跳跃后一旦出现任何反应(uncloak、重新 cloak、加载更多的追加),遍历回退到起跳位置并谨慎走完。正确性从不依赖热身阶段的判断,只有惰性页面的开销依赖它。同一个观察器还负责**收割无键行**:引擎移除一个没有键的元素时,记录里仍有完整的已脱离元素,以及它被移除时左右的兄弟节点,合并时就把它放回那里。全程不读布局,绝对定位和 `transform` 都不影响。每个列表里,每个行身份只暂存一次(每帧重渲染的引擎会把同一行移除成千上万次,之后的观测只作锚点用),并且使用独立的字符预算,不会挤占带键暂存。瞬时 UI 从不收割(`role=status|alert|dialog|tooltip|progressbar|menu|…`、`aria-hidden`、`aria-busy`、无文本元素)。从未产生反应、也没有发出 fetch/XHR 的页面,在底部只停留 900ms,而不是完整的 2.5s 耐心窗口。实测:约 30 屏的静态长文开销约 2.4s,而不是耗尽 20s 预算;不需要滚动的页面没有任何额外开销。
156
+ 4. **合并**到结构化克隆(`document.cloneNode(true)`)中:没有序列化/重解析往返,运行时改写进脚本文本的字面 `</script>` 无法污染合并结果;克隆没有浏览上下文,不执行脚本、不升级自定义元素、不加载 frame。**插回**(仅容器遍历):来自滚动容器内、而克隆中已不存在的暂存键是被引擎**移除**的(飞书形态,占位符是匿名的),插回到其记录的兄弟旁边,迭代求解,使一段被移除的连续块能从任一存活端链式还原;随已插回父级一同回来的键直接跳过。定位是结构性的,与遍历**看到**各行的顺序无关——包括从文档中部起步的锚点打开。**替换**:每个比快照更空的键元素(子元素更少,或文本明显更少——即 Discourse 占位符)被替换为快照,但**容器**除外(其后代中有待替换元素的祖先),容器由这些后代拼装还原——容器自身的快照只是某一窗口的局部视图。**还原**(无键行,在替换之前进行):只有在遍历期间引擎既从某个列表移除过带文本的行、又向它添加过元素时,该列表才符合条件。单个 toast 或加载更多的追加不算,位于 `aria-live="assertive"`、`role=alert|status|timer|marquee` 或轮播区域内的也不算。列表里已不存在的被移除行,会放回它被移除时所在的兄弟旁边。如果锚点本身也被移除了,就顺着它自己的邻居继续找;同一行的重复观测,解析到该行暂存副本最终落下的位置。每一次观测都会尝试,所以锚点打开时的首屏行,可以由后来看到它与邻居相邻的那一轮来定位。被整体重渲染(`innerHTML`)的列表没有存活锚点,这些行按移除顺序放到第一个带文本的行之前:自上而下、止于底部的遍历只会丢失最终窗口上方的内容。与仍存在的行去重时,引擎设置了按列表的身份属性就用它(`data-index`、`data-item-index`、`aria-rowindex`、`aria-posinset`、`data-key`、`data-id` 等,限定在本列表内,不同列表间的重复值不会冲突),否则用行文本。身份属性相同但明显更空的现存行视为占位符,会被替换。引擎放在原行位置上的无键占位(紧挨该行、文本很短且被该行包含)会被吸收。已还原的列表及其祖先在替换阶段一律视为容器,带键的包装元素不会被替换成只有一个窗口的快照。随后克隆经过与活文档读取相同的脚本剥除(`src/strip.ts`)并序列化,provider 读取该字符串而非活页面。页面有反应但什么都没能还原时(行没有文本、canvas 渲染),遍历会滚回页面打开时的位置,等页面静止后再交给活文档读取;否则 provider 读到的是遍历停在底部的最后一个窗口,而不是旧版抓取看到的首屏。对滚动惰性的页面保持原位,它的 DOM 怎样都一样。
157
157
 
158
- 遍历在两种降噪模式下都运行;受 `scrollBudgetMs`(默认 20s;`0` 恢复旧版单瞬间行为)与剩余抓取预算减去解码储备的双重约束。任何失败——页面没有 `evaluate`、求值被拒、引擎拒绝遍历——都静默降级为旧版抓取:遍历只会增加内容,不会让抓取失败。CDP profile 模式下遍历滚动的是远端浏览器里抓取自有的那个标签页,与渲染发生在同一处。
158
+ 遍历在两种降噪模式下都运行;受内置 20s 预算与剩余等待预算的双重约束——在预算处截停的遍历仍会合并并返回已暂存的内容,正文读取与解码在其后的 10s 提取宽限窗内完成。任何失败——页面没有 `evaluate`、求值被拒、引擎拒绝遍历——都静默降级为旧版抓取:遍历只会增加内容,不会让抓取失败。CDP profile 模式下遍历滚动的是远端浏览器里抓取自有的那个标签页,与渲染发生在同一处。
159
159
 
160
- 真实坏例实测:linux.do 帖子(51 楼、约 18kpx)在 20s 预算内一次遍历走完,每楼都被再水合、零占位符残留;飞书 wiki 页面在 20s 预算中用约 11.5s 走完内层容器(15 步、插回 38 个被移除的块、最终 DOM 191 块——静止抓取只有 7 块),深部尾文(两处 JSON 示例与完整段落)都进入 markdown。一项如实记录的残余限制:飞书表格块的**单元格**藏在第二层块内虚拟化后面,遍历不追——表头与行能还原,超大载荷单元格可能保持为空。
160
+ 真实坏例实测:linux.do 帖子(51 楼、约 18kpx)在 20s 预算内一次遍历走完,每楼都被再水合、零占位符残留;飞书 wiki 页面在 20s 预算中用约 11.5s 走完内层容器(15 步、插回 38 个被移除的块、最终 DOM 191 块——静止抓取只有 7 块),深部尾文(两处 JSON 示例与完整段落)都进入 markdown。公开的 react-window 演示页(`/list/fixed-row-height`,1000 个无键行)上,遍历以前只返回最后一个窗口(第 129–142 行);无键还原后按顺序返回第 1–155 行,没有重复,在 20s 预算耗尽处停止。无键路径的剩余限制:只有图片或用 canvas 渲染的行无法还原(遍历改为回滚);原地**复用**行节点的引擎(改写池中节点的文本,而不是移除它)不会被收割。一项如实记录的残余限制:飞书表格块的**单元格**藏在第二层块内虚拟化后面,遍历不追——表头与行能还原,超大载荷单元格可能保持为空。
161
161
 
162
162
  ### PDF 链接
163
163
 
package/cordis.patch.yml CHANGED
@@ -26,10 +26,6 @@
26
26
 
27
27
  - id: tool-web
28
28
  disabled: false
29
- config:
30
- fetch: true
31
- fetchTimeoutMs: 60000
32
- searchTimeoutMs: 60000
33
29
 
34
30
  - insert:
35
31
  - id: web-fetch-playwright
package/lib/client.js CHANGED
@@ -370,9 +370,7 @@ window.__ModuleLoader__.load({
370
370
  checkboxField("shareBrowserContext"),
371
371
  radioField("contextIsolationScope", ["session", "global"]),
372
372
  checkboxField("denoise"),
373
- numberField("maxConcurrency", 1, 200),
374
- numberField("challengeWaitMs", 0, 6e4),
375
- numberField("scrollBudgetMs", 0, 3e4)
373
+ numberField("maxConcurrency", 1, 200)
376
374
  ]);
377
375
  this.store = this.form.bind(() => this.projection());
378
376
  }
@@ -385,9 +383,7 @@ window.__ModuleLoader__.load({
385
383
  shareBrowserContext: this.form.field("shareBrowserContext"),
386
384
  contextIsolationScope: this.form.field("contextIsolationScope"),
387
385
  denoise: this.form.field("denoise"),
388
- maxConcurrency: this.form.field("maxConcurrency"),
389
- challengeWaitMs: this.form.field("challengeWaitMs"),
390
- scrollBudgetMs: this.form.field("scrollBudgetMs")
386
+ maxConcurrency: this.form.field("maxConcurrency")
391
387
  };
392
388
  }
393
389
  /**
@@ -417,12 +413,12 @@ window.__ModuleLoader__.load({
417
413
  document.head.appendChild(tag);
418
414
  }
419
415
  var PluginCard_module_css_default = {
420
- "footer": "UM8-Rq_footer",
421
- "form": "UM8-Rq_form",
422
- "readOnly": "UM8-Rq_readOnly",
423
- "failed": "UM8-Rq_failed",
424
416
  "save": "UM8-Rq_save",
425
- "discard": "UM8-Rq_discard"
417
+ "discard": "UM8-Rq_discard",
418
+ "readOnly": "UM8-Rq_readOnly",
419
+ "form": "UM8-Rq_form",
420
+ "footer": "UM8-Rq_footer",
421
+ "failed": "UM8-Rq_failed"
426
422
  };
427
423
  //#endregion
428
424
  //#region src/client/PluginCard.tsx
@@ -482,27 +478,27 @@ window.__ModuleLoader__.load({
482
478
  document.head.appendChild(tag);
483
479
  }
484
480
  var fields_module_css_default = {
481
+ "head": "CFSaVG_head",
485
482
  "inputInvalid": "CFSaVG_inputInvalid",
486
- "radioOptionPlain": "CFSaVG_radioOptionPlain",
487
- "badge": "CFSaVG_badge",
483
+ "label": "CFSaVG_label",
488
484
  "radioPick": "CFSaVG_radioPick",
489
- "field": "CFSaVG_field",
490
- "invalid": "CFSaVG_invalid",
491
- "hint": "CFSaVG_hint",
492
- "radioGroup": "CFSaVG_radioGroup",
493
- "input": "CFSaVG_input",
494
- "radioText": "CFSaVG_radioText",
495
485
  "radioHint": "CFSaVG_radioHint",
496
- "radioContent": "CFSaVG_radioContent",
497
- "label": "CFSaVG_label",
498
- "reset": "CFSaVG_reset",
499
- "radioLabel": "CFSaVG_radioLabel",
500
- "hintIndented": "CFSaVG_hintIndented",
501
- "checkboxRow": "CFSaVG_checkboxRow",
486
+ "radioText": "CFSaVG_radioText",
487
+ "radioOptionPlain": "CFSaVG_radioOptionPlain",
488
+ "fieldEmbedded": "CFSaVG_fieldEmbedded",
489
+ "radioGroup": "CFSaVG_radioGroup",
502
490
  "radioOption": "CFSaVG_radioOption",
491
+ "reset": "CFSaVG_reset",
492
+ "invalid": "CFSaVG_invalid",
493
+ "field": "CFSaVG_field",
494
+ "hint": "CFSaVG_hint",
495
+ "badge": "CFSaVG_badge",
503
496
  "badges": "CFSaVG_badges",
504
- "fieldEmbedded": "CFSaVG_fieldEmbedded",
505
- "head": "CFSaVG_head"
497
+ "hintIndented": "CFSaVG_hintIndented",
498
+ "radioLabel": "CFSaVG_radioLabel",
499
+ "radioContent": "CFSaVG_radioContent",
500
+ "input": "CFSaVG_input",
501
+ "checkboxRow": "CFSaVG_checkboxRow"
506
502
  };
507
503
  //#endregion
508
504
  //#region src/client/fields.tsx
@@ -831,40 +827,6 @@ window.__ModuleLoader__.load({
831
827
  onReset: () => {
832
828
  props.resetField("maxConcurrency");
833
829
  }
834
- }),
835
- /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
836
- id: "plugin-config-playwright-challenge-wait",
837
- label: t("challengeWaitMs"),
838
- hint: t("challengeWaitMsHint"),
839
- placeholder: t("challengeWaitMsPlaceholder"),
840
- overriddenLabel: t("overridden"),
841
- resetLabel: t("reset"),
842
- invalidLabel: t("invalidText"),
843
- disabled,
844
- ...state.challengeWaitMs,
845
- onEdit: (text) => {
846
- props.edit("challengeWaitMs", text);
847
- },
848
- onReset: () => {
849
- props.resetField("challengeWaitMs");
850
- }
851
- }),
852
- /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
853
- id: "plugin-config-playwright-scroll-budget",
854
- label: t("scrollBudgetMs"),
855
- hint: t("scrollBudgetMsHint"),
856
- placeholder: t("scrollBudgetMsPlaceholder"),
857
- overriddenLabel: t("overridden"),
858
- resetLabel: t("reset"),
859
- invalidLabel: t("invalidText"),
860
- disabled,
861
- ...state.scrollBudgetMs,
862
- onEdit: (text) => {
863
- props.edit("scrollBudgetMs", text);
864
- },
865
- onReset: () => {
866
- props.resetField("scrollBudgetMs");
867
- }
868
830
  })
869
831
  ]
870
832
  });
@@ -899,12 +861,6 @@ window.__ModuleLoader__.load({
899
861
  maxConcurrency: "Max concurrent fetches",
900
862
  maxConcurrencyHint: "How many pages may render at once (1–200). Blank = auto: 4 local browsers, 50 CDP tabs in the remote browser.",
901
863
  maxConcurrencyPlaceholder: "(auto: local 4 / CDP 50)",
902
- challengeWaitMs: "Cloudflare challenge wait (ms)",
903
- challengeWaitMsHint: "Bounded wait for a Cloudflare challenge to clear naturally in the same tab (0–60000; 0 = off). 0 disables: the first response is returned as-is.",
904
- challengeWaitMsPlaceholder: "(default: 15000)",
905
- scrollBudgetMs: "Deep-capture scroll budget (ms)",
906
- scrollBudgetMsHint: "Bounded in-page scroll walk that captures viewport-virtualized feeds (forum topics, comment threads) whole: posts outside the render window are empty placeholders at rest. 0–30000; 0 = off (capture the page as it rests).",
907
- scrollBudgetMsPlaceholder: "(default: 20000)",
908
864
  overridden: "Overridden",
909
865
  reset: "Reset to default",
910
866
  readOnly: "This deployment stores settings read-only.",
@@ -942,12 +898,6 @@ window.__ModuleLoader__.load({
942
898
  maxConcurrency: "最大并发抓取数",
943
899
  maxConcurrencyHint: "同时渲染的页面上限(1–200);留空自动:本地 4 个浏览器,CDP 在远端浏览器里 50 个标签页。",
944
900
  maxConcurrencyPlaceholder: "(自动:本地 4 / CDP 50)",
945
- challengeWaitMs: "Cloudflare 挑战等待上限(毫秒)",
946
- challengeWaitMsHint: "在同一标签页内有界等待 Cloudflare 验证自然通过(0–60000;0 = 关闭)。关闭时直接返回首次响应——旧版行为。",
947
- challengeWaitMsPlaceholder: "(默认:15000)",
948
- scrollBudgetMs: "深度抓取滚动预算(毫秒)",
949
- scrollBudgetMsHint: "有界的页内滚动遍历,完整捕获视口虚拟化的信息流(论坛帖子、评论区):静止时渲染窗口外的帖子只是空占位符。0–30000;0 = 关闭(按页面静止状态抓取)。",
950
- scrollBudgetMsPlaceholder: "(默认:20000)",
951
901
  overridden: "已覆盖",
952
902
  reset: "恢复默认",
953
903
  readOnly: "本部署的设置为只读。",