dsh-web-fetch-playwright 0.2.16 → 0.2.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -4
- package/README.zh-CN.md +47 -4
- package/lib/client.js +45 -20
- package/lib/index.js +1869 -75
- package/lib/types/autoscroll.d.ts +277 -0
- package/lib/types/client/controller.d.ts +7 -0
- package/lib/types/client/locales.d.ts +1 -1
- package/lib/types/config.d.ts +38 -3
- package/lib/types/index.d.ts +5 -1
- package/lib/types/markdown.d.ts +8 -5
- package/lib/types/pdf.d.ts +143 -0
- package/lib/types/provider.d.ts +34 -0
- package/lib/types/strip.d.ts +59 -0
- package/lib/types/types.d.ts +5 -0
- package/package.json +3 -2
- package/src/autoscroll.ts +653 -0
- package/src/client/card.tsx +13 -0
- package/src/client/controller.ts +9 -0
- package/src/client/locales.ts +7 -0
- package/src/config.ts +46 -3
- package/src/index.ts +7 -0
- package/src/markdown.ts +252 -6
- package/src/pdf.ts +1114 -0
- package/src/provider.ts +174 -76
- package/src/session-window.ts +8 -0
- package/src/strip.ts +72 -0
- package/src/types.ts +5 -0
package/README.md
CHANGED
|
@@ -9,13 +9,15 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
|
|
|
9
9
|
## Features
|
|
10
10
|
|
|
11
11
|
- **Real browser rendering** — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
|
|
12
|
-
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `` so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content *is* the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content.
|
|
12
|
+
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `` so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content *is* the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content. An accepted extraction is then pruned of trailing link-dense sections — the post-article recommendation rails and share bars that ride *inside* the winning wrapper and survive every vocabulary-based rule (163's `post_recommend` cards measure 0.42–0.48 per-card link density, just under Readability's own 0.5 removal bar, and arrive as bare divs with no class to match); a trailing run of such blocks is removed only when it is a minority of its container *and* clearly denser than the content before it, so link-list extractions (feed rows, portal headlines) are untouched.
|
|
13
|
+
- **PDF links** — a URL that serves a PDF (inline `application/pdf`, a `Content-Disposition: attachment` download, or an `application/octet-stream` whose bytes start with `%PDF-`) is read as bytes and converted to Markdown instead of failing: pdf.js (via `unpdf`) supplies positioned text and real font names, and layout heuristics rebuild headings (outline/bookmarks, section numbering such as `3.1` / `第二章` / `一、`, type size, standalone bold lines), paragraphs (gaps, first-line indents, cross-column/cross-page continuation, dictionary-checked de-hyphenation, CJK-aware joining), lists, monospace code blocks, column-aligned GFM tables, inline bold, and `<!-- page N -->` markers, while dropping running headers/footers and page numbers. See [PDF links](#pdf-links).
|
|
13
14
|
- **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
|
|
14
15
|
- **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
|
|
15
16
|
- **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
|
|
16
17
|
- **Live configuration** — the Plugins page (侧边栏 → 插件 → *dsh-web-fetch-playwright*, dsh ≥ 0.1.7) shows the full form right on the plugin's package page (and on the row's own page via its configure control); it edits the backend, context mode, denoise toggle, and concurrency, and a committed change reaches the next fetch without a restart.
|
|
17
18
|
- **Budget-aware** — per-fetch deadline (45s); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
|
|
18
19
|
- **Bounded Cloudflare-challenge wait** — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented `cf-mitigated: challenge` response header plus structural page markers), the fetch keeps the **same tab and context** and waits for the browser's own verification to clear it — tracking the *last* main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded and configurable (`challengeWaitMs`, default 15s; 0 restores the legacy first-response behavior), with a bounded same-tab retry (`challengeRetries`, default 1). When the budget runs out, the fetch fails with the distinct `WEB_FETCH_CHALLENGE` error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.
|
|
20
|
+
- **Deep capture for viewport-virtualized feeds and app-shell documents** — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by `data-block-id` with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/`data-block-id`), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot and splices the removed ones back beside their surviving siblings, in a structural clone of the page that it serializes itself (the live page is never rewritten). Pages that never react to scrolling — most of the web — are detected and skip the slow walk. Bounded by `scrollBudgetMs` (default 20s; 0 restores the legacy single-instant behavior); any failure silently degrades to the legacy capture.
|
|
19
21
|
|
|
20
22
|
## How it works
|
|
21
23
|
|
|
@@ -29,12 +31,13 @@ web_fetch (tool-web)
|
|
|
29
31
|
└─ ctx.web.fetchProvider = playwright
|
|
30
32
|
├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
|
|
31
33
|
├─ cdp: connectOverCDP(endpoint)
|
|
32
|
-
├─
|
|
33
|
-
├─
|
|
34
|
+
├─ PDF (inline / attachment / sniffed octet-stream) → bytes (navigation body, else request-context refetch) → pdf.js → layout heuristics → Markdown
|
|
35
|
+
├─ page.goto → settle (networkidle, best-effort) → deep-capture scroll walk (virtualized feeds; returns its merged, stripped capture when it restored anything) → otherwise strip scripting subtrees in the live DOM → page.content()
|
|
36
|
+
├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → trailing link-dense prune → DOMPurify → Turndown(GFM)
|
|
34
37
|
└─ Markdown (or raw HTML when denoise is off)
|
|
35
38
|
```
|
|
36
39
|
|
|
37
|
-
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content;
|
|
40
|
+
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; a PDF download is extracted to Markdown (see [PDF links](#pdf-links)), and other undecodable downloads (a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
|
|
38
41
|
|
|
39
42
|
**Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction.** The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses `page.content()` with jsdom, and a script's *text* is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal `</script>` embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`, un-escaping the wire's 712 `\u003c` sequences into raw `<` — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the *first* embedded end tag and parses the remaining ~270KB of JSON *as markup*. The phantom elements that survive (unknown tags like `<id>`, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and `CachedAsset` strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary `script` and `template` subtrees are stripped inside the live page (`page.evaluate`, best-effort) before `page.content()`. This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back **byte-identical on every page**. Two deliberate carve-outs keep even the indirect consumers intact: `script[type="application/ld+json"]` is *defused* rather than deleted (`</` rewritten to `<\/` — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and `noscript` is kept entirely because Readability's `_unwrapNoscriptImages` salvages the real image out of lazy-load `<noscript><img …>` wrappers. The strip also applies with `denoise` off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).
|
|
40
43
|
|
|
@@ -79,6 +82,7 @@ The Plugins page (侧边栏 → 插件) opens the form one click from the list:
|
|
|
79
82
|
| `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
|
|
80
83
|
| `challengeWaitMs` | `15000` | Bounded wait (ms, 0–60000) for a Cloudflare challenge to clear naturally in the same tab. `0` disables the whole challenge path — the first response is returned as-is (the pre-0.2.5 behavior). |
|
|
81
84
|
| `challengeRetries` | `1` | Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 45s per-fetch deadline. |
|
|
85
|
+
| `scrollBudgetMs` | `20000` | Bounded budget (ms, 0–30000) for the deep-capture scroll walk. `0` disables it — the page is captured as it rests (the pre-walk behavior), which loses the bodies of viewport-virtualized feeds. |
|
|
82
86
|
|
|
83
87
|
Local backend resolution order:
|
|
84
88
|
|
|
@@ -130,6 +134,45 @@ With the wait on (default):
|
|
|
130
134
|
|
|
131
135
|
Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by `challengeWaitMs` and the 45s per-fetch deadline; nothing blocks forever.
|
|
132
136
|
|
|
137
|
+
### Deep capture (viewport-virtualized feeds and app-shell documents)
|
|
138
|
+
|
|
139
|
+
Feed-shaped pages — Discourse topic streams (linux.do, meta.discourse.org), comment threads, timelines — render only the items near the current scroll position. At rest the live DOM holds a window's worth of real content while every item outside it is a height-only placeholder that keeps the item's identity but none of it:
|
|
140
|
+
|
|
141
|
+
```html
|
|
142
|
+
<div class="post-stream--cloaked" data-post-number="1" id="post_1" style="height:297.9px !important;…">
|
|
143
|
+
<h2 class="sr-only">由 Diver-QC 于 2 小时前 发布</h2>
|
|
144
|
+
</div>
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
A fetch that renders, never scrolls, and serializes one instant of that DOM hands back the skeleton — the screen-reader-only "posted by" headings survive the pipeline as if they were content, the bodies vanish, and even the denoise pipeline's coverage gate cannot notice (the page's own visible text is the same windowed fragment as the extraction). Scrolling to the bottom once does not help either: the render window *moves* — measured on a 40-reply linux.do topic, the resting DOM held only the first ~6 replies, a bottom jump left only the last 10, and replies past the first chunk were not in the DOM at all until a load-more sentinel near the bottom fetched them.
|
|
148
|
+
|
|
149
|
+
App-shell document editors have the same pathology one level deeper — the measured badcase is a Feishu wiki page (`bocha-ai.feishu.cn/wiki/…`). There the body never scrolls at all: html/body are locked at exactly one viewport and the document scrolls inside `div.bear-web-x-container` (measured: `documentElement.scrollHeight` 720 == `innerHeight`, `window.scrollTo(0, 800)` left `scrollY` at 0, while the container held 4534px against a 656px client). Its blocks are virtualized with two twists over Discourse: they are keyed by `data-block-id` and carry **no id**, and off-window blocks are **removed** outright — the whole un-rendered tail is one anonymous `.bear-virtual-renderUnit-placeholder` spacer with an explicit height — so there is no window scrollbar to walk, no id to stash by, and no degraded-in-place row to swap back. A fetch just returns the statically rendered first screen (measured: 7 blocks, 892 chars; the full document is ~190 blocks).
|
|
150
|
+
|
|
151
|
+
The fix is a bounded in-page scroll walk (`src/autoscroll.ts`), run after the settle and before the serialization strip for every HTML document:
|
|
152
|
+
|
|
153
|
+
1. **Resolve the walk target**: the window when the document itself scrolls (the Discourse shape); otherwise the page's dominant inner scroll container (the app-shell shape) — the element with the largest scrollable distance among computed `overflow-y: auto|scroll` elements, gated at half a viewport so dropdown-sized scrollers never trigger a walk. Neither scrollable means no walk.
|
|
154
|
+
2. **Stash** every keyed element — keyed by id, else `data-block-id` (namespaced so the two can never collide) — as a detached clone, keeping the *richest* sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus `data-preloaded` and `svg-sprites` ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
|
|
155
|
+
3. **Walk** down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; `scrollTop` drives an inner container exactly the way `window.scrollTo` drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s *patience window* with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A `MutationObserver` watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does: a ~30-viewport static article costs ~4.5s instead of exhausting the 20s budget.
|
|
156
|
+
4. **Merge** into a structural clone (`document.cloneNode(true)`): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal `</script>` in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. *Splice* (container walks): stashed keys from inside the scroll container that the clone no longer holds were **removed** by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk *saw* rows in — including an anchored open that starts mid-document. *Swap*: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except *containers* (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. The clone is then stripped of scripting elements (the same `src/strip.ts` routine the live read uses) and serialized; the provider reads that string instead of the live page. A walk that restored nothing hands back no HTML, and the live read proceeds exactly as before.
|
|
157
|
+
|
|
158
|
+
The walk runs in both denoise modes and is bounded by `scrollBudgetMs` (default 20s; `0` restores the legacy single-instant behavior) and by the remaining fetch deadline minus a decode reserve. Any failure — a page without `evaluate`, a rejected evaluation, an engine that refuses the walk — silently degrades to the legacy capture; a walk can only add content, never fail a fetch. In CDP profile mode the walk scrolls the fetch-owned tab in the remote browser, exactly where the fetch already renders it.
|
|
159
|
+
|
|
160
|
+
Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. One residual limitation, documented as measured: Feishu table blocks lazy-render their *cells* behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.
|
|
161
|
+
|
|
162
|
+
### PDF links
|
|
163
|
+
|
|
164
|
+
When the fetched URL turns out to be a PDF, the provider extracts it instead of failing with `WEB_UNSUPPORTED_CONTENT_TYPE`:
|
|
165
|
+
|
|
166
|
+
| How the PDF arrives | What the browser does | How the bytes are obtained |
|
|
167
|
+
| --- | --- | --- |
|
|
168
|
+
| `application/pdf` (inline) | headless Chromium downloads it; a headed or CDP-attached Chrome opens its viewer | navigation body when it is the PDF, else a refetch through the page's API request context |
|
|
169
|
+
| `Content-Disposition: attachment` | `goto` rejects with `Download is starting` | refetch through the API request context |
|
|
170
|
+
| `application/octet-stream` (and kin), `.pdf` path or filename | download | refetch, then the `%PDF-` magic decides |
|
|
171
|
+
|
|
172
|
+
The refetch uses the same browser context, so cookies and logins (CDP profile mode) apply to gated PDFs. The bytes go through pdf.js (`unpdf`'s serverless build, lazily imported — plugin start never loads it) and a layout pass (`src/pdf.ts`) that turns positioned text into Markdown: the largest first-page type becomes the `#` title (else the PDF's metadata title), headings come from the outline, numbering, larger type, or standalone bold lines; running headers/footers and page numbers are removed; paragraphs are rebuilt and joined across columns and pages; bullets, numbered items, monospace blocks, and column-aligned rows become lists, fenced code, and GFM tables; each page starts with a `<!-- page N -->` marker.
|
|
173
|
+
|
|
174
|
+
Limits and errors: PDFs over 50MB fail with `WEB_PDF_TOO_LARGE` (checked against `Content-Length` before downloading); reading stops after 500 pages, 100k collected characters, or the fetch deadline, and the result is flagged `truncated` with a note naming the last page read. A scan with no text layer fails with `WEB_PDF_NO_TEXT` (OCR is not supported — PDFs that carry an OCR text layer extract normally), a password-protected PDF with `WEB_PDF_ENCRYPTED`, and bytes pdf.js cannot parse with `WEB_PDF_INVALID`. Known limitations: tables without consistent column gaps (and figure labels) come out as plain text, multi-column layouts rely on the PDF's content order, and bullets drawn as vector shapes (rather than glyphs) are not recognized as list markers.
|
|
175
|
+
|
|
133
176
|
## Development
|
|
134
177
|
|
|
135
178
|
```sh
|
package/README.zh-CN.md
CHANGED
|
@@ -9,13 +9,15 @@
|
|
|
9
9
|
## 特性
|
|
10
10
|
|
|
11
11
|
- **真实浏览器渲染** —— 以用户视角加载页面,SPA 客户端渲染内容也能抓到,而非只有原始 HTML。
|
|
12
|
-
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 ``,避免 base64 字符流刷屏。对列表型页面(门户首页、商品目录——正文本身就是链接),Readability 的链接密度打分会误选出噪声碎片(实测 163.com 首页:96 字符的页脚备案块胜出,320 条新闻标题得分趋零),因此抽取结果需通过**覆盖率置信门**——不足 250 字符、或占页面可见文本不足 10%
|
|
12
|
+
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 ``,避免 base64 字符流刷屏。对列表型页面(门户首页、商品目录——正文本身就是链接),Readability 的链接密度打分会误选出噪声碎片(实测 163.com 首页:96 字符的页脚备案块胜出,320 条新闻标题得分趋零),因此抽取结果需通过**覆盖率置信门**——不足 250 字符、或占页面可见文本不足 10% 即不予采信,转而回退到整页消毒转换,此类页面上可基本完整保留列表内容。通过置信门的抽取结果再做**尾部链接密集块修剪**——正文之后的推荐流、分享栏往往随正文一起被裹进胜出的容器里,且能躲过所有基于类名的规则(163 的 `post_recommend` 卡片单卡链接密度 0.42–0.48,恰好低于 Readability 自己 0.5 的删除线,且抽取后类名尽失、无从匹配);只有当尾部连续块**既占容器文本少数、又明显比其前的正文更"链接化"**时才整体移除,因此链接列表型抽取(信息流、门户头条)不受影响。
|
|
13
|
+
- **PDF 链接** —— 当 URL 返回的是 PDF(inline 的 `application/pdf`、`Content-Disposition: attachment` 下载,或文件头为 `%PDF-` 的 `application/octet-stream`)时,按字节读取并转换为 Markdown,而不是报错:pdf.js(通过 `unpdf`)提供带坐标的文本和真实字体名,版面启发式规则据此还原标题(书签、`3.1` / `第二章` / `一、` 等章节编号、字号、独立粗体行)、段落(间距、首行缩进、跨栏跨页衔接、基于全文词表的断词合并、中日韩文字无空格拼接)、列表、等宽代码块、列对齐的 GFM 表格、行内粗体和 `<!-- page N -->` 页码标记,同时去除页眉页脚和页码。详见 [PDF 链接](#pdf-链接)。
|
|
13
14
|
- **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
|
|
14
15
|
- **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
|
|
15
16
|
- **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
|
|
16
17
|
- **热配置** —— 插件页(侧边栏 → 插件 → *dsh-web-fetch-playwright*,dsh ≥ 0.1.7)在插件自己的包页面直接展示完整表单(也可通过该行的「配置」入口进入同一表单);可随时切换后端、上下文模式、降噪开关与并发数,提交的改动对下一次抓取即时生效,无需重启。
|
|
17
18
|
- **预算控制** —— 单次抓取 45s 超时;并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 20s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
|
|
18
19
|
- **Cloudflare 挑战有界等待** —— 导航落到验证中间页("Just a moment…" 及其多语言同族,通过官方 `cf-mitigated: challenge` 响应头 + 结构性页面标记识别)时,抓取保持**同一标签页与上下文**,等待浏览器自行通过验证:跟踪*最后一次*主 frame 响应(真实页面随后重载进来),并轮询活 DOM 以捕获 SPA 式清除。有界且可配置(`challengeWaitMs`,默认 15s;`0` 恢复旧版首响应行为),附带同标签页有界重试(`challengeRetries`,默认 1)。预算耗尽时以独立的 `WEB_FETCH_CHALLENGE` 错误码明确失败,而不是把中间页当正文返回。全程不点击、不注入验证码答案、不伪造浏览器状态、不导出或复制 cookie。
|
|
20
|
+
- **视口虚拟化信息流与应用壳文档的深度抓取** —— 论坛帖子、评论区、时间线类页面越来越多只渲染滚动位置附近的内容(Discourse 帖子流把视口外的每一楼都替换成只保留楼层 id、不含任何正文的高度占位符),应用壳文档编辑器(飞书文档一族)更进一步:body 根本不滚动、块以 `data-block-id` 标识而没有 id、窗口外的块被直接移除、只留匿名高度占位符。静止快照只会拿到骨架或首屏而正文静默丢失。抓取因此在序列化前执行一次有界的页内滚动遍历:文档可滚时走窗口,否则走页面主导的内层滚动容器;暂存每个渲染窗口的内容(每个 id/`data-block-id` 保留最丰满的快照),逐步走到底部——只有"触底且高度停止增长"才意味着分页加载完毕——然后在页面的结构化克隆里用暂存快照再水合所有退化的键元素、把被移除的键插回其存活兄弟旁边,并由遍历自行序列化该克隆(活文档从不被改写)。对滚动毫无反应的页面(绝大多数网页)会被识别出来并跳过慢速遍历。由 `scrollBudgetMs` 约束(默认 20s;`0` 恢复旧版单瞬间抓取行为);任何失败都静默降级为旧版抓取。
|
|
19
21
|
|
|
20
22
|
## 工作原理
|
|
21
23
|
|
|
@@ -29,12 +31,13 @@ web_fetch (tool-web)
|
|
|
29
31
|
└─ ctx.web.fetchProvider = playwright
|
|
30
32
|
├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
|
|
31
33
|
├─ cdp: connectOverCDP(endpoint)
|
|
32
|
-
├─
|
|
33
|
-
├─
|
|
34
|
+
├─ PDF(inline / attachment / 嗅探出的 octet-stream)→ 字节(导航响应体,否则 request context 重新拉取)→ pdf.js → 版面启发式 → Markdown
|
|
35
|
+
├─ page.goto → 等待稳定(networkidle,尽力而为)→ 深度抓取滚动遍历(虚拟化信息流;有还原时直接返回其合并且已剥离的结果)→ 否则在活 DOM 中剥离脚本子树 → page.content()
|
|
36
|
+
├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → 尾部链接密集块修剪 → DOMPurify → Turndown(GFM)
|
|
34
37
|
└─ Markdown(关闭降噪时返回原始 HTML)
|
|
35
38
|
```
|
|
36
39
|
|
|
37
|
-
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context
|
|
40
|
+
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;PDF 下载会被提取为 Markdown(见 [PDF 链接](#pdf-链接)),其他不可解码的下载(ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;传输错误与取消也依旧按错误处理。
|
|
38
41
|
|
|
39
42
|
**回读文档前,先在活 DOM 中剥离非内容脚本子树 —— 构造上即输出中性。** 降噪管线(以及关闭降噪时工具层自己的渲染器)会用 jsdom 重新解析 `page.content()`,而脚本的*文本内容*是这条往返链路中唯一无法信任的东西:页面自身的运行时 JS 可能把内嵌 `</script>` 的文本写回某个数据脚本的 textContent —— 实测于 techcommunity.microsoft.com(Khoros/Aurora 平台),其客户端把整个 Apollo 缓存经 `JSON.stringify` 写回 `script#__NEXT_DATA__`,把线上响应里 712 处 `\u003c` 转义还原成了裸 `<` —— 而浏览器序列化脚本文本时原样输出,于是这份扁平化的 HTML 字符串会在*第一个*内嵌结束标签处终止脚本,把余下约 270KB 的 JSON *当作标记语言*解析。解析出的幽灵元素(`<id>` 之类的未知标签、塞满 CSS 与 i18n 字符串的锚点)恰好是 Readability 评分器眼里的"正文质量"(实测 126 分对真实正文的 45 分),于是抓取 RDP 10 AVC/H.264 一文时返回的是 CSS 规则和 `CachedAsset` 字符串而不是文章本体。事后再对字符串做补丁无法消除这种歧义 —— 只有浏览器自己的解析持有文档的真实结构 —— 因此普通的 `script` 与 `template` 子树在 `page.content()` 之前先在活页面里剥掉(`page.evaluate`,尽力而为)。这不会改变管线以往的任何输出:Readability 本就在抽取与评分前删除 script/noscript 节点,DOMPurify 的白名单本就丢弃 script,工具层渲染器也整体移除 SCRIPT/TEMPLATE —— 在真实页面语料(Wikipedia、MDN、Hacker News、坏例页 wire 版,加上 noscript/JSON-LD/template 边界用例)上的差分 A/B **每一页都字节级一致**。两个刻意的保留让间接消费方也不受损:`script[type="application/ld+json"]` 只做*消险*而非删除(`</` 改写为 `<\/` —— 仍是合法 JSON,Readability 的元数据读取解析到同一对象,文章标题保持无站点后缀的干净形态,如 "Remote Desktop Protocol" 而非 "… - Wikipedia");`noscript` 完整保留,因为 Readability 的 `_unwrapNoscriptImages` 会从懒加载的 `<noscript><img …>` 包裹中抢救出真实图片。该剥离在关闭降噪时同样生效,让返回给工具层渲染器的原始 HTML 也是序列化安全的,并把管线要解析的体积压缩约 70%(以该坏例页计)。
|
|
40
43
|
|
|
@@ -79,6 +82,7 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
|
|
|
79
82
|
| `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;20s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
|
|
80
83
|
| `challengeWaitMs` | `15000` | Cloudflare 挑战的**有界**自然等待上限(毫秒,0–60000),在同一标签页内等待浏览器自行通过验证。`0` 关闭整条挑战处理链路——直接返回首次响应(0.2.5 之前的旧行为)。 |
|
|
81
84
|
| `challengeRetries` | `1` | 一个等待窗口耗尽后的**同标签页**重新导航次数(0–3);浏览器已拿到的通关 cookie 留在上下文里供重试使用。总耗时始终受 45s 单次抓取预算约束。 |
|
|
85
|
+
| `scrollBudgetMs` | `20000` | 深度抓取滚动遍历的**有界**预算(毫秒,0–30000)。`0` 关闭——按页面静止状态抓取(旧行为),会丢失视口虚拟化信息流的正文。 |
|
|
82
86
|
|
|
83
87
|
本地后端解析顺序:
|
|
84
88
|
|
|
@@ -130,6 +134,45 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
|
|
|
130
134
|
|
|
131
135
|
安全边界(刻意为之):不点击 Turnstile、不解验证码、不注入 token、不伪装指纹/UA、不轮换代理、不导出 cookie——CDP 隔离模式下通关态保留在对应会话 / 全局 context 中,直到该 context 被清理(本地 context 仍随每次抓取结束);profile 模式下它留在远端浏览器自己的 profile 里,插件从不复制或清理。等待始终受 `challengeWaitMs` 与 45s 单次抓取预算双重约束,永不无限阻塞。
|
|
132
136
|
|
|
137
|
+
### 深度抓取(视口虚拟化信息流与应用壳文档)
|
|
138
|
+
|
|
139
|
+
信息流形态的页面——Discourse 帖子流(linux.do、meta.discourse.org)、评论区、时间线——只渲染滚动位置附近的条目。静止时活 DOM 里只有一个窗口的真实内容,窗口外的每个条目都是只保留身份、不含内容的高度占位符:
|
|
140
|
+
|
|
141
|
+
```html
|
|
142
|
+
<div class="post-stream--cloaked" data-post-number="1" id="post_1" style="height:297.9px !important;…">
|
|
143
|
+
<h2 class="sr-only">由 Diver-QC 于 2 小时前 发布</h2>
|
|
144
|
+
</div>
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
渲染后从不滚动、只序列化某一瞬间的抓取,交回的就是骨架——屏幕阅读器专用的"由 X 发布"标题穿过管线存活下来、看起来像正文,真正的楼层正文全部消失;连降噪管线的覆盖率门控也发现不了(页面自身的可见文本和提取结果是同一份被窗口化的残片)。只滚动到底部一次同样不行:渲染窗口是**移动的**——在一个 40 楼的 linux.do 帖子上实测,静止 DOM 只有前 ~6 楼,一步跳到底只剩最后 10 楼,而且第一块之后的楼层根本不在 DOM 里,要等触底 sentinel 拉取。
|
|
148
|
+
|
|
149
|
+
应用壳文档编辑器把同样的病理又下沉了一层——实测坏例是飞书 wiki 页面(`bocha-ai.feishu.cn/wiki/…`):body 根本不滚动,html/body 被锁死在恰好一屏,文档在 `div.bear-web-x-container` 里滚(实测:`documentElement.scrollHeight` 720 == `innerHeight`,`window.scrollTo(0, 800)` 后 `scrollY` 仍为 0,而容器以 656px 的 client 撑着 4534px)。它的块虚拟化比 Discourse 多两个变化:块以 `data-block-id` 标识、**没有 id**;窗口外的块被**直接移除**——整段未渲染尾部只是一个匿名的 `.bear-virtual-renderUnit-placeholder` 显式高度占位符——于是既没有窗口滚动条可走、没有 id 可暂存、也没有原地退化的行可换回,抓取只会返回静态渲染的首屏(实测:7 个块、892 字;完整文档约 190 块)。
|
|
150
|
+
|
|
151
|
+
修复是一次有界的页内滚动遍历(`src/autoscroll.ts`),在每个 HTML 文档的 settle 之后、序列化剥除之前执行:
|
|
152
|
+
|
|
153
|
+
1. **解析遍历目标**:文档自身可滚时走窗口(Discourse 形态);否则找页面的主导内层滚动容器(应用壳形态)——在 computed `overflow-y: auto|scroll` 的元素里取可滚距离最大者,且门槛为半个视口,下拉框级别的小滚动容器永远不会触发遍历。两者皆无则不遍历。
|
|
154
|
+
2. **暂存**每个键元素——以元素 id 为键,无 id 则用 `data-block-id`(加命名空间前缀,两种属性永不冲突)——为脱离文档的克隆,每个键保留**最丰满**的一次观测(先以占位符形态被看到的行,会在渲染形态出现时升级),并记录其最近的带键兄弟:即它在自己所属列表中的位置。单条体积上限约 64KB:虚拟化行只有几 KB,而几十 KB 的元素只可能是整个信息流的容器(linux.do 实测:六层嵌套包装加上 `data-preloaded` 和 `svg-sprites`,一次收集就把 2MB 预算烧掉 1.9MB),这类键会被记住、此后不再序列化。
|
|
155
|
+
3. **遍历**:从顶部起每次下移 70% 目标视口高度,每步为 300ms 平滑动画(按时间插值;`scrollTop` 驱动内层容器与 `window.scrollTo` 驱动文档完全同效)加 300ms 静置供信息流引擎的防抖 uncloak 生效;直到能在底部持续 2.5s **耐心窗口**而高度无增长(真实信息流的分块拉取是网络往返,任何增长都会重新计时),或步数/预算上限关闭遍历。一个 `MutationObserver` 观察遍历存在的全部理由——页面因滚动而增删元素。连续两个谨慎步都无反应时,页面被视为对滚动惰性,剩余部分用一次跳到底部来探测;跳跃后一旦出现任何反应(uncloak、重新 cloak、加载更多的追加),遍历回退到起跳位置并谨慎走完。正确性从不依赖热身阶段的判断,只有惰性页面的开销依赖它:约 30 屏的静态长文从耗尽 20s 预算降到约 4.5s。
|
|
156
|
+
4. **合并**到结构化克隆(`document.cloneNode(true)`)中:没有序列化/重解析往返,运行时改写进脚本文本的字面 `</script>` 无法污染合并结果;克隆没有浏览上下文,不执行脚本、不升级自定义元素、不加载 frame。**插回**(仅容器遍历):来自滚动容器内、而克隆中已不存在的暂存键是被引擎**移除**的(飞书形态,占位符是匿名的),插回到其记录的兄弟旁边,迭代求解,使一段被移除的连续块能从任一存活端链式还原;随已插回父级一同回来的键直接跳过。定位是结构性的,与遍历**看到**各行的顺序无关——包括从文档中部起步的锚点打开。**替换**:每个比快照更空的键元素(子元素更少,或文本明显更少——即 Discourse 占位符)被替换为快照,但**容器**除外(其后代中有待替换元素的祖先),容器由这些后代拼装还原——容器自身的快照只是某一窗口的局部视图。随后克隆经过与活文档读取相同的脚本剥除(`src/strip.ts`)并序列化,provider 读取该字符串而非活页面。没有还原任何内容的遍历不返回 HTML,活文档读取与以往完全一致。
|
|
157
|
+
|
|
158
|
+
遍历在两种降噪模式下都运行;受 `scrollBudgetMs`(默认 20s;`0` 恢复旧版单瞬间行为)与剩余抓取预算减去解码储备的双重约束。任何失败——页面没有 `evaluate`、求值被拒、引擎拒绝遍历——都静默降级为旧版抓取:遍历只会增加内容,不会让抓取失败。CDP profile 模式下遍历滚动的是远端浏览器里抓取自有的那个标签页,与渲染发生在同一处。
|
|
159
|
+
|
|
160
|
+
真实坏例实测:linux.do 帖子(51 楼、约 18kpx)在 20s 预算内一次遍历走完,每楼都被再水合、零占位符残留;飞书 wiki 页面在 20s 预算中用约 11.5s 走完内层容器(15 步、插回 38 个被移除的块、最终 DOM 191 块——静止抓取只有 7 块),深部尾文(两处 JSON 示例与完整段落)都进入 markdown。一项如实记录的残余限制:飞书表格块的**单元格**藏在第二层块内虚拟化后面,遍历不追——表头与行能还原,超大载荷单元格可能保持为空。
|
|
161
|
+
|
|
162
|
+
### PDF 链接
|
|
163
|
+
|
|
164
|
+
当抓取的 URL 实际返回的是 PDF 时,provider 会提取其内容,而不是以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败:
|
|
165
|
+
|
|
166
|
+
| PDF 的返回方式 | 浏览器的行为 | 字节的获取方式 |
|
|
167
|
+
| --- | --- | --- |
|
|
168
|
+
| `application/pdf`(inline) | 无头 Chromium 会下载它;有界面或经 CDP 连接的 Chrome 会打开内置查看器 | 导航响应体本身是 PDF 时直接使用,否则通过页面的 API request context 重新拉取 |
|
|
169
|
+
| `Content-Disposition: attachment` | `goto` 以 `Download is starting` 拒绝 | 通过 API request context 重新拉取 |
|
|
170
|
+
| `application/octet-stream` 等通用二进制类型、`.pdf` 路径或文件名 | 下载 | 重新拉取后,以 `%PDF-` 魔数判定 |
|
|
171
|
+
|
|
172
|
+
重新拉取使用同一个浏览器上下文,因此 Cookie 和登录态(CDP profile 模式)对需要登录的 PDF 同样生效。字节交给 pdf.js(`unpdf` 的 serverless 构建,按需动态加载,插件启动时不会加载)解析,再由版面分析(`src/pdf.ts`)把带坐标的文本还原成 Markdown:第一页最大字号的文字作为 `#` 标题(否则用 PDF 元数据中的标题);标题层级来自书签、章节编号(`3.1` / `第二章` / `一、`)、更大的字号或独立成行的粗体;去除页眉页脚和页码;重建段落,并跨栏、跨页衔接;项目符号、编号条目、等宽字体块和列对齐的行分别转为列表、代码块和 GFM 表格;每页开头带 `<!-- page N -->` 标记。
|
|
173
|
+
|
|
174
|
+
限制与错误码:超过 50MB 的 PDF 以 `WEB_PDF_TOO_LARGE` 失败(下载前先按 `Content-Length` 检查);读取到 500 页、累计 10 万字符或抓取截止时间时停止,结果标记为 `truncated`,并注明最后读取的页码。没有文本层的扫描件以 `WEB_PDF_NO_TEXT` 失败(不支持 OCR;自带 OCR 文本层的 PDF 可以正常提取),加密 PDF 以 `WEB_PDF_ENCRYPTED` 失败,pdf.js 无法解析的数据以 `WEB_PDF_INVALID` 失败。已知局限:没有稳定列间距的表格(以及图中的标注文字)会以普通文本输出;多栏版面依赖 PDF 内容流的顺序;用矢量图形(而非字形)绘制的项目符号不会被识别为列表标记。
|
|
175
|
+
|
|
133
176
|
## 开发
|
|
134
177
|
|
|
135
178
|
```sh
|
package/lib/client.js
CHANGED
|
@@ -371,7 +371,8 @@ window.__ModuleLoader__.load({
|
|
|
371
371
|
radioField("contextIsolationScope", ["session", "global"]),
|
|
372
372
|
checkboxField("denoise"),
|
|
373
373
|
numberField("maxConcurrency", 1, 200),
|
|
374
|
-
numberField("challengeWaitMs", 0, 6e4)
|
|
374
|
+
numberField("challengeWaitMs", 0, 6e4),
|
|
375
|
+
numberField("scrollBudgetMs", 0, 3e4)
|
|
375
376
|
]);
|
|
376
377
|
this.store = this.form.bind(() => this.projection());
|
|
377
378
|
}
|
|
@@ -385,7 +386,8 @@ window.__ModuleLoader__.load({
|
|
|
385
386
|
contextIsolationScope: this.form.field("contextIsolationScope"),
|
|
386
387
|
denoise: this.form.field("denoise"),
|
|
387
388
|
maxConcurrency: this.form.field("maxConcurrency"),
|
|
388
|
-
challengeWaitMs: this.form.field("challengeWaitMs")
|
|
389
|
+
challengeWaitMs: this.form.field("challengeWaitMs"),
|
|
390
|
+
scrollBudgetMs: this.form.field("scrollBudgetMs")
|
|
389
391
|
};
|
|
390
392
|
}
|
|
391
393
|
/**
|
|
@@ -415,12 +417,12 @@ window.__ModuleLoader__.load({
|
|
|
415
417
|
document.head.appendChild(tag);
|
|
416
418
|
}
|
|
417
419
|
var PluginCard_module_css_default = {
|
|
418
|
-
"discard": "UM8-Rq_discard",
|
|
419
|
-
"save": "UM8-Rq_save",
|
|
420
|
-
"readOnly": "UM8-Rq_readOnly",
|
|
421
|
-
"form": "UM8-Rq_form",
|
|
422
420
|
"footer": "UM8-Rq_footer",
|
|
423
|
-
"
|
|
421
|
+
"form": "UM8-Rq_form",
|
|
422
|
+
"readOnly": "UM8-Rq_readOnly",
|
|
423
|
+
"failed": "UM8-Rq_failed",
|
|
424
|
+
"save": "UM8-Rq_save",
|
|
425
|
+
"discard": "UM8-Rq_discard"
|
|
424
426
|
};
|
|
425
427
|
//#endregion
|
|
426
428
|
//#region src/client/PluginCard.tsx
|
|
@@ -480,27 +482,27 @@ window.__ModuleLoader__.load({
|
|
|
480
482
|
document.head.appendChild(tag);
|
|
481
483
|
}
|
|
482
484
|
var fields_module_css_default = {
|
|
483
|
-
"
|
|
485
|
+
"inputInvalid": "CFSaVG_inputInvalid",
|
|
486
|
+
"radioOptionPlain": "CFSaVG_radioOptionPlain",
|
|
484
487
|
"badge": "CFSaVG_badge",
|
|
488
|
+
"radioPick": "CFSaVG_radioPick",
|
|
485
489
|
"field": "CFSaVG_field",
|
|
490
|
+
"invalid": "CFSaVG_invalid",
|
|
491
|
+
"hint": "CFSaVG_hint",
|
|
492
|
+
"radioGroup": "CFSaVG_radioGroup",
|
|
493
|
+
"input": "CFSaVG_input",
|
|
486
494
|
"radioText": "CFSaVG_radioText",
|
|
487
495
|
"radioHint": "CFSaVG_radioHint",
|
|
488
496
|
"radioContent": "CFSaVG_radioContent",
|
|
489
|
-
"
|
|
490
|
-
"radioLabel": "CFSaVG_radioLabel",
|
|
491
|
-
"badges": "CFSaVG_badges",
|
|
492
|
-
"radioOptionPlain": "CFSaVG_radioOptionPlain",
|
|
493
|
-
"invalid": "CFSaVG_invalid",
|
|
497
|
+
"label": "CFSaVG_label",
|
|
494
498
|
"reset": "CFSaVG_reset",
|
|
499
|
+
"radioLabel": "CFSaVG_radioLabel",
|
|
500
|
+
"hintIndented": "CFSaVG_hintIndented",
|
|
495
501
|
"checkboxRow": "CFSaVG_checkboxRow",
|
|
496
|
-
"radioGroup": "CFSaVG_radioGroup",
|
|
497
|
-
"radioPick": "CFSaVG_radioPick",
|
|
498
502
|
"radioOption": "CFSaVG_radioOption",
|
|
499
|
-
"
|
|
500
|
-
"
|
|
501
|
-
"
|
|
502
|
-
"hintIndented": "CFSaVG_hintIndented",
|
|
503
|
-
"inputInvalid": "CFSaVG_inputInvalid"
|
|
503
|
+
"badges": "CFSaVG_badges",
|
|
504
|
+
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
505
|
+
"head": "CFSaVG_head"
|
|
504
506
|
};
|
|
505
507
|
//#endregion
|
|
506
508
|
//#region src/client/fields.tsx
|
|
@@ -846,6 +848,23 @@ window.__ModuleLoader__.load({
|
|
|
846
848
|
onReset: () => {
|
|
847
849
|
props.resetField("challengeWaitMs");
|
|
848
850
|
}
|
|
851
|
+
}),
|
|
852
|
+
/* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
|
|
853
|
+
id: "plugin-config-playwright-scroll-budget",
|
|
854
|
+
label: t("scrollBudgetMs"),
|
|
855
|
+
hint: t("scrollBudgetMsHint"),
|
|
856
|
+
placeholder: t("scrollBudgetMsPlaceholder"),
|
|
857
|
+
overriddenLabel: t("overridden"),
|
|
858
|
+
resetLabel: t("reset"),
|
|
859
|
+
invalidLabel: t("invalidText"),
|
|
860
|
+
disabled,
|
|
861
|
+
...state.scrollBudgetMs,
|
|
862
|
+
onEdit: (text) => {
|
|
863
|
+
props.edit("scrollBudgetMs", text);
|
|
864
|
+
},
|
|
865
|
+
onReset: () => {
|
|
866
|
+
props.resetField("scrollBudgetMs");
|
|
867
|
+
}
|
|
849
868
|
})
|
|
850
869
|
]
|
|
851
870
|
});
|
|
@@ -883,6 +902,9 @@ window.__ModuleLoader__.load({
|
|
|
883
902
|
challengeWaitMs: "Cloudflare challenge wait (ms)",
|
|
884
903
|
challengeWaitMsHint: "Bounded wait for a Cloudflare challenge to clear naturally in the same tab (0–60000; 0 = off). 0 disables: the first response is returned as-is.",
|
|
885
904
|
challengeWaitMsPlaceholder: "(default: 15000)",
|
|
905
|
+
scrollBudgetMs: "Deep-capture scroll budget (ms)",
|
|
906
|
+
scrollBudgetMsHint: "Bounded in-page scroll walk that captures viewport-virtualized feeds (forum topics, comment threads) whole: posts outside the render window are empty placeholders at rest. 0–30000; 0 = off (capture the page as it rests).",
|
|
907
|
+
scrollBudgetMsPlaceholder: "(default: 20000)",
|
|
886
908
|
overridden: "Overridden",
|
|
887
909
|
reset: "Reset to default",
|
|
888
910
|
readOnly: "This deployment stores settings read-only.",
|
|
@@ -923,6 +945,9 @@ window.__ModuleLoader__.load({
|
|
|
923
945
|
challengeWaitMs: "Cloudflare 挑战等待上限(毫秒)",
|
|
924
946
|
challengeWaitMsHint: "在同一标签页内有界等待 Cloudflare 验证自然通过(0–60000;0 = 关闭)。关闭时直接返回首次响应——旧版行为。",
|
|
925
947
|
challengeWaitMsPlaceholder: "(默认:15000)",
|
|
948
|
+
scrollBudgetMs: "深度抓取滚动预算(毫秒)",
|
|
949
|
+
scrollBudgetMsHint: "有界的页内滚动遍历,完整捕获视口虚拟化的信息流(论坛帖子、评论区):静止时渲染窗口外的帖子只是空占位符。0–30000;0 = 关闭(按页面静止状态抓取)。",
|
|
950
|
+
scrollBudgetMsPlaceholder: "(默认:20000)",
|
|
926
951
|
overridden: "已覆盖",
|
|
927
952
|
reset: "恢复默认",
|
|
928
953
|
readOnly: "本部署的设置为只读。",
|