dsh-web-fetch-playwright 0.2.20 → 0.2.23
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -17
- package/README.zh-CN.md +19 -17
- package/cordis.patch.yml +0 -4
- package/lib/client.js +18 -68
- package/lib/index.js +24049 -22754
- package/lib/types/autoscroll.d.ts +46 -10
- package/lib/types/candidates.d.ts +55 -0
- package/lib/types/challenge.d.ts +2 -2
- package/lib/types/client/controller.d.ts +0 -14
- package/lib/types/client/locales.d.ts +1 -1
- package/lib/types/config.d.ts +80 -30
- package/lib/types/index.d.ts +1 -1
- package/lib/types/provider.d.ts +101 -9
- package/lib/types/types.d.ts +26 -6
- package/package.json +1 -1
- package/src/autoscroll.ts +459 -17
- package/src/block-model.ts +1 -1
- package/src/candidates.ts +184 -35
- package/src/challenge.ts +2 -2
- package/src/client/card.tsx +0 -26
- package/src/client/controller.ts +0 -18
- package/src/client/locales.ts +0 -14
- package/src/config.ts +97 -37
- package/src/index.ts +4 -1
- package/src/markdown.ts +13 -2
- package/src/provider.ts +482 -56
- package/src/selector-model.ts +1 -1
- package/src/selector.ts +18 -7
- package/src/types.ts +27 -7
package/README.md
CHANGED
|
@@ -15,9 +15,9 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
|
|
|
15
15
|
- **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
|
|
16
16
|
- **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
|
|
17
17
|
- **Live configuration** — the Plugins page (侧边栏 → 插件 → *dsh-web-fetch-playwright*, dsh ≥ 0.1.7) shows the full form right on the plugin's package page (and on the row's own page via its configure control); it edits the backend, context mode, denoise toggle, and concurrency, and a committed change reaches the next fetch without a restart.
|
|
18
|
-
- **Budget-aware** — per-fetch
|
|
19
|
-
- **Bounded Cloudflare-challenge wait** — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented `cf-mitigated: challenge` response header plus structural page markers), the fetch keeps the **same tab and context** and waits for the browser's own verification to clear it — tracking the *last* main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded
|
|
20
|
-
- **Deep capture for viewport-virtualized feeds and app-shell documents** — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by `data-block-id` with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/`data-block-id`), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot
|
|
18
|
+
- **Budget-aware** — per-fetch waiting budget (25s) — when it runs out (a slow load, an endless scroll feed) the fetch stops waiting and extracts what the page already rendered instead of failing (`truncated` when the document never finished loading; `WEB_FETCH_TIMEOUT` only when nothing arrived at all or extraction itself hangs past a 10s grace); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 10s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
|
|
19
|
+
- **Bounded Cloudflare-challenge wait** — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented `cf-mitigated: challenge` response header plus structural page markers), the fetch keeps the **same tab and context** and waits for the browser's own verification to clear it — tracking the *last* main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded by a built-in 15s wait, with a bounded same-tab retry (`challengeRetries`, default 1). When the window runs out, a bounded **plain-HTTP refetch** of the same URL runs through the page's own request context (`challengeHttpFallback`, default on) — the same cookie jar and the browser's UA, but none of Chromium's render decisions — because many edges serve the real document to such a request while withholding the interactive challenge from the browser session; the answer must itself be a non-challenge document or the fetch still fails. Failing that, the fetch fails with the distinct `WEB_FETCH_CHALLENGE` error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.
|
|
20
|
+
- **Deep capture for viewport-virtualized feeds and app-shell documents** — forum topics, comment threads, and timelines increasingly render only the items near the scroll position (Discourse topic streams keep every other post as an empty height-only placeholder that carries the post's id but none of its content), and app-shell document editors (Feishu docs and their kin) go one step further: the body never scrolls at all, blocks are keyed by `data-block-id` with no id, and off-window blocks are removed outright behind anonymous height spacers. A resting snapshot returns the skeleton or just the first screen while the bodies silently vanish. The fetch therefore runs one bounded in-page scroll walk before serialization: it walks the window or, when the document does not scroll, the page's dominant inner scroll container; it stashes what each rendered window shows (richest snapshot per id/`data-block-id`), steps to the bottom — the only place "the page has stopped growing" means anything — then rehydrates every degraded keyed element from its stashed snapshot, splices the removed ones back beside their surviving siblings, and restores rows that carry no DOM identity at all (react-window lists, X-style timelines) from the moment the engine removed them, in a structural clone of the page that it serializes itself (the live page is never rewritten). Pages that never react to scrolling — most of the web — are detected and skip the slow walk. Bounded by a built-in 20s budget; any failure silently degrades to the legacy capture.
|
|
21
21
|
|
|
22
22
|
## How it works
|
|
23
23
|
|
|
@@ -32,12 +32,14 @@ web_fetch (tool-web)
|
|
|
32
32
|
├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
|
|
33
33
|
├─ cdp: connectOverCDP(endpoint)
|
|
34
34
|
├─ PDF (inline / attachment / sniffed octet-stream) → bytes (navigation body, else request-context refetch) → pdf.js → layout heuristics → Markdown
|
|
35
|
-
├─ page.goto → settle (
|
|
35
|
+
├─ page.goto → settle (load baseline + bounded network-quiet window with a stale-request discount, best-effort — never Playwright's networkidle: worker-script and cross-origin-auth requests never complete on the page session, so strict idle is unreachable on modern SPA pages) → deep-capture scroll walk (virtualized feeds; returns its merged, stripped capture when it restored anything) → otherwise strip scripting subtrees in the live DOM → page.content()
|
|
36
36
|
├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → trailing link-dense prune → DOMPurify → Turndown(GFM)
|
|
37
37
|
└─ Markdown (or raw HTML when denoise is off)
|
|
38
38
|
```
|
|
39
39
|
|
|
40
|
-
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; a PDF download is extracted to Markdown (see [PDF links](#pdf-links)), and other undecodable downloads (a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
|
|
40
|
+
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; a PDF download is extracted to Markdown (see [PDF links](#pdf-links)), and other undecodable downloads (a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; persistent transport errors and cancellations stay errors — but *transient* transport blips are retried first: `net::ERR_NETWORK_CHANGED` (Chromium's NetworkChangeNotifier killed the in-flight navigation because the host's network configuration shifted under the browser — a route/DNS/VPN/Docker-DNS change; measured live: one session's source.android.com fetch died this way and the same URL fetched clean seconds later) and its sibling codes (`ERR_CONNECTION_RESET`, `ERR_NAME_NOT_RESOLVED`, …) re-navigate the same page up to twice with a short backoff, bounded by the remaining fetch budget; `ERR_ABORTED` and Playwright timeout errors are never retried (their shapes are deterministic and already owned by the recoveries above).
|
|
41
|
+
|
|
42
|
+
**Web-component app shells skip the Readability family they cannot win, and large pages stop re-parsing the same DOM.** The denoise pipeline is synchronous and jsdom-bound, so a heavy page pays its whole cost as dead air after the browser already holds the full document — measured on the badcase (`aistudio.google.com/docs/computer-use`, fetched over CDP into the user's real browser): the page was fully rendered ~7s in, then the pipeline spent another 13.4s on the 1.1MB capture, of which ~6.5s was three Readability passes and ~2.9s eight DOMPurify re-parses. Two of those passes and most of those parses produced *provably dead* candidates: the page is an Angular app whose entire DOM lives under `<ms-app>`, and DOMPurify's allowlist has no hyphenated (custom-element) tags, so with `KEEP_CONTENT: false` every extraction carrying that wrapper sanitizes to a corpse (Readability's pick: a root with zero words; the whole-document candidate: 39 of ~4.5k words). When ≥90% of the body's visible text sits inside custom-element subtrees (`customWrappedTextShare`), candidate generation therefore skips the Readability family (`art`/`artu`/`artx`) and marks the skip in the facts (`readabilitySkipped`); the selector is seeded with the empty-`art` reference stats a dead pick would have measured, so the surviving candidates' features are unchanged, and the DOM-backed fallbacks, noscript paths and block classifier — what the selector picks on that shape anyway — carry the extraction. Independently, candidates the generator already holds as DOM nodes (`doc`, `docc`, `main`, `ns_doc`) are sanitized by importing the node (identical output, measured 555ms → 304ms for the whole-body candidate) instead of re-parsing their serialization; the Readability article is parsed once for the coverage gate and the trailing prune instead of once each; and rail-name recognition short-circuits on a literal hint before the per-token split (~360ms of the walk over every element on the badcase). Net effect on the badcase fetch: pipeline 13.4s → 5.0s, end-to-end 21–23s → 13–15s, same winning strategy and complete content; over a 63-page benchmark sample (where the gate fires on none of them) outputs are byte-identical except one additional label line on the badcase itself.
|
|
41
43
|
|
|
42
44
|
**Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction.** The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses `page.content()` with jsdom, and a script's *text* is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal `</script>` embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`, un-escaping the wire's 712 `\u003c` sequences into raw `<` — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the *first* embedded end tag and parses the remaining ~270KB of JSON *as markup*. The phantom elements that survive (unknown tags like `<id>`, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and `CachedAsset` strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary `script` and `template` subtrees are stripped inside the live page (`page.evaluate`, best-effort) before `page.content()`. This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back **byte-identical on every page**. Two deliberate carve-outs keep even the indirect consumers intact: `script[type="application/ld+json"]` is *defused* rather than deleted (`</` rewritten to `<\/` — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and `noscript` is kept entirely because Readability's `_unwrapNoscriptImages` salvages the real image out of lazy-load `<noscript><img …>` wrappers. The strip also applies with `denoise` off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).
|
|
43
45
|
|
|
@@ -79,10 +81,9 @@ The Plugins page (侧边栏 → 插件) opens the form one click from the list:
|
|
|
79
81
|
| `shareBrowserContext` | `true` | CDP only: Share / Isolate radio. `true` uses the remote default context and logins; `false` uses a separate context with the lifecycle determined by the scope below. Ignored locally. |
|
|
80
82
|
| `contextIsolationScope` | `global` | CDP isolation only: `session` creates a context on demand per DSH session and closes it when no active/pending fetches or open tabs remain (next fetch starts fresh); `global` reuses one fixed isolated context across all DSH sessions within this plugin instance. |
|
|
81
83
|
| `denoise` | `true` | Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert. |
|
|
82
|
-
| `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within
|
|
83
|
-
| `
|
|
84
|
-
| `
|
|
85
|
-
| `scrollBudgetMs` | `20000` | Bounded budget (ms, 0–30000) for the deep-capture scroll walk. `0` disables it — the page is captured as it rests (the pre-walk behavior), which loses the bodies of viewport-virtualized feeds. |
|
|
84
|
+
| `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 10s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
|
|
85
|
+
| `challengeRetries` | `1` | Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 25s per-fetch waiting budget. |
|
|
86
|
+
| `challengeHttpFallback` | `true` | When the browser cannot clear a challenge inside its window, refetch the same URL once through the page's plain-HTTP request context (same cookies, the browser's UA, no render decision) and return that document when it is not itself a challenge. `false` keeps the browser-only posture. |
|
|
86
87
|
|
|
87
88
|
Local backend resolution order:
|
|
88
89
|
|
|
@@ -123,16 +124,17 @@ Headless server (pre-seed logins in a headful environment first): `chromium --he
|
|
|
123
124
|
|
|
124
125
|
### Cloudflare challenge handling (bounded natural wait)
|
|
125
126
|
|
|
126
|
-
Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior;
|
|
127
|
+
Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior; from a repo checkout, after `pnpm build` — run `node scripts/challenge-demo.mjs` for a local simulated before/after, and `node scripts/challenge-online.mjs <url>` against a real site).
|
|
127
128
|
|
|
128
129
|
With the wait on (default):
|
|
129
130
|
|
|
130
131
|
1. **Detection** — a response carrying `cf-mitigated: challenge` (the documented signal for every challenge page type), or a 403/503 HTML document from a `server: cloudflare` edge, or the localized interstitial itself (title family like "Just a moment…", `请稍候…`, "Минутку…", plus structural markers: the challenge-platform runtime's `<script src>`, `#challenge-*` elements, `cf-chl-widget-` frames, `window._cf_chl_opt`). The content-level markers are the *fallback tier* and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (`server: cloudflare` / `cf-ray`) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Two refinements keep that promise for real pages: the challenge-platform script check reads real `<script src>` attributes only (prose, and escaped code samples that quote a path, are ignored), and it matches the runtime mount `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/` *positively*. Everything else Cloudflare injects under that prefix — Bot Management's JavaScript Detections (`scripts/jsd/`, `h/[b|g]/scripts/jsd/<hash>/`, `h/[b|g]/jsd/r/`), and Precursor (`scripts/precursor/`) — is passive instrumentation present on *every normal page* of a protected zone (e.g. openrouter.ai, nodeseek.com), so it is never read as a challenge. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
|
|
131
132
|
2. **Bounded wait, same tab and context** — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the *last* main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
|
|
132
133
|
3. **Bounded retry** — when a window runs out, the same tab re-navigates once (default; `challengeRetries`) with whatever clearance cookies the context already holds.
|
|
133
|
-
4. **
|
|
134
|
+
4. **Dead-end refetch** — before failing, one plain-HTTP request for the same URL through the page's request context (`challengeHttpFallback`, default on): the same cookie jar and the browser's User-Agent, none of Chromium's render decisions. Many Cloudflare edges serve the real document to such a request while withholding the interactive challenge from the browser session itself (measured live: v2ex.com from a datacenter IP — the browser drew an empty widget frame for 15s×2, the plain request drew the full article). The answer must itself be a non-challenge HTML document — checked at response level AND content level by the same classifiers that gate the browser path — or the fetch keeps the browser-only verdict.
|
|
135
|
+
5. **Clear failure** — `WEB_FETCH_CHALLENGE` (a provider-specific code the web seam's open-string `code` allows) naming the site, the budget spent, and the last challenge status.
|
|
134
136
|
|
|
135
|
-
Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by
|
|
137
|
+
Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in CDP isolated mode clearance remains within its session/global context until that context is discarded (local contexts still end with each fetch); in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by the built-in 15s wait and the 25s per-fetch waiting budget; nothing blocks forever.
|
|
136
138
|
|
|
137
139
|
### Deep capture (viewport-virtualized feeds and app-shell documents)
|
|
138
140
|
|
|
@@ -151,13 +153,13 @@ App-shell document editors have the same pathology one level deeper — the meas
|
|
|
151
153
|
The fix is a bounded in-page scroll walk (`src/autoscroll.ts`), run after the settle and before the serialization strip for every HTML document:
|
|
152
154
|
|
|
153
155
|
1. **Resolve the walk target**: the window when the document itself scrolls (the Discourse shape); otherwise the page's dominant inner scroll container (the app-shell shape) — the element with the largest scrollable distance among computed `overflow-y: auto|scroll` elements, gated at half a viewport so dropdown-sized scrollers never trigger a walk. Neither scrollable means no walk.
|
|
154
|
-
2. **Stash** every keyed element — keyed by id, else `data-block-id` (namespaced so the two can never collide) — as a detached clone, keeping the *richest* sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus `data-preloaded` and `svg-sprites` ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
|
|
155
|
-
3. **Walk** down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; `scrollTop` drives an inner container exactly the way `window.scrollTo` drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s *patience window* with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A `MutationObserver` watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does: a ~30-viewport static article costs ~
|
|
156
|
-
4. **Merge** into a structural clone (`document.cloneNode(true)`): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal `</script>` in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. *Splice* (container walks): stashed keys from inside the scroll container that the clone no longer holds were **removed** by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk *saw* rows in — including an anchored open that starts mid-document. *Swap*: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except *containers* (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. The clone is then stripped of scripting elements (the same `src/strip.ts` routine the live read uses) and serialized; the provider reads that string instead of the live page.
|
|
156
|
+
2. **Stash** every keyed element — keyed by id, else `data-block-id` (namespaced so the two can never collide) — as a detached clone (framework-generated ids are not keys: Ember's `ember123` and React `useId` values such as `:r1:`, `«r1»` and `_r_1_` change on every render, so such elements take the unkeyed path below), keeping the *richest* sighting per key (a row first seen as its placeholder is upgraded when its rendered form appears), together with its nearest keyed siblings: where it sits in its own list. Entries are size-capped (~64KB each): a virtualized row is a few KB, while an element tens of KB big is a container whose serialization embeds the whole stream (measured on linux.do: six nested wrappers plus `data-preloaded` and `svg-sprites` ballooned one pass to 1.9MB of the 2MB budget); such keys are remembered and never serialized again.
|
|
157
|
+
3. **Walk** down from the top — 70% of the target's viewport per step, each a smooth 300ms-animated descent (time-interpolated ticks; `scrollTop` drives an inner container exactly the way `window.scrollTo` drives the document) plus a 300ms quiet settle for the feed engine's debounced uncloak — until the walk can hold at the bottom through a 2.5s *patience window* with no height growth (real feed engines fetch their next chunk with network latency; growth restarts the clock), or the step/budget caps close it. A `MutationObserver` watches for the walk's whole reason to exist — the page adding or removing elements in reaction to the scroll. After two careful steps with no reaction the page is treated as scroll-inert and the rest is probed with one jump to the bottom; any reaction after that jump (an uncloak, a re-cloak, a load-more append) rewinds the walk to where it jumped from and it finishes carefully. Correctness never rests on the warm-up guess, only the cost of an inert page does. The same observer *harvests unkeyed rows*: when the engine removes an element that has no key, the record still hands over the detached element whole, plus the sibling nodes it sat between, and that is where the merge puts it back. No layout is read, so absolute positioning and `transform` do not matter. Each row identity is stashed once per list (an engine that re-renders every frame removes the same row thousands of times; later sightings only anchor), from its own character budget so it can never crowd out the keyed stash. Transient UI is never harvested (`role=status|alert|dialog|tooltip|progressbar|menu|…`, `aria-hidden`, `aria-busy`, elements without text). A page that never reacted and issued no fetch/XHR holds only 900ms at the bottom instead of the full 2.5s patience. Measured: a ~30-viewport static article costs ~2.4s instead of exhausting the 20s budget, and a page that does not scroll pays nothing.
|
|
158
|
+
4. **Merge** into a structural clone (`document.cloneNode(true)`): no serialize/re-parse round trip, so a runtime-rewritten data script with a literal `</script>` in its text cannot corrupt the merge, and — having no browsing context — the clone runs no scripts, upgrades no custom elements, and loads no frames. *Splice* (container walks): stashed keys from inside the scroll container that the clone no longer holds were **removed** by the engine (the Feishu shape, whose placeholders are anonymous) and go back next to their recorded sibling, resolved iteratively so a removed run chains off whichever end survived; a key that rode in inside its spliced parent is skipped. Placement is structural, so it holds whatever order the walk *saw* rows in — including an anchored open that starts mid-document. *Swap*: every keyed element emptier than its snapshot (fewer child elements, or meaningfully less text — the Discourse placeholder) is replaced by it, except *containers* (ancestors of another element being swapped), which are assembled from those descendants instead — a container's own snapshot is one window's partial view. *Restore* (unkeyed rows, before the swap): a list qualifies only if the engine both removed text-bearing rows from it and added elements to it during the walk. A lone toast or a load-more append does not qualify, and neither does anything inside an `aria-live="assertive"`, `role=alert|status|timer|marquee` or carousel region. Each removed row the list no longer holds goes back next to the sibling it was removed beside. An anchor that was itself removed is followed to *its* neighbor, and a repeat sighting resolves to wherever the row's stashed copy lands. Every sighting of a row is tried, so the first window of an anchored open is placed by the later pass that saw it beside its neighbors. A list re-rendered wholesale (`innerHTML`) leaves no surviving anchor; its rows go before the first text-bearing row, in removal order, because a top-down walk that ends at the bottom only loses what lay above the final window. De-duplication against the rows still present uses a per-list identity attribute when the engine sets one (`data-index`, `data-item-index`, `aria-rowindex`, `aria-posinset`, `data-key`, `data-id`, …, scoped to the list so repeated values across lists cannot collide), else the row's text. A present row with the same identity attribute that is far emptier is a placeholder and is replaced. An unkeyed stub the engine put in a row's place (short text the row contains, right beside it) is absorbed. A restored list and its ancestors count as containers for the swap, so a keyed wrapper is never replaced by its one-window snapshot. The clone is then stripped of scripting elements (the same `src/strip.ts` routine the live read uses) and serialized; the provider reads that string instead of the live page. If a page reacted but nothing could be restored (rows without text, canvas rendering), the walk scrolls back to where the page opened and waits for it to settle before the live read. Otherwise the provider would read the walk's last (bottom) window instead of the opening screen the legacy capture saw. A scroll-inert page is left where it is, since its DOM is the same either way.
|
|
157
159
|
|
|
158
|
-
The walk runs in both denoise modes and is bounded by
|
|
160
|
+
The walk runs in both denoise modes and is bounded by a built-in 20s budget and by the fetch's remaining waiting budget; a walk cut short by the budget still merges and returns what it stashed, and the content read plus decode run in the 10s extraction grace window after it. Any failure — a page without `evaluate`, a rejected evaluation, an engine that refuses the walk — silently degrades to the legacy capture; a walk can only add content, never fail a fetch. In CDP profile mode the walk scrolls the fetch-owned tab in the remote browser, exactly where the fetch already renders it.
|
|
159
161
|
|
|
160
|
-
Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. One residual limitation, documented as measured: Feishu table blocks lazy-render their *cells* behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.
|
|
162
|
+
Measured on the live badcases: the linux.do topic (51 replies, ~18kpx) walks fully in one 20s budget with every post rehydrated and zero cloaked placeholders left; the Feishu wiki page walks its inner container in ~11.5s of the 20s budget (15 steps, 38 removed blocks spliced back, 191 blocks in the final DOM — the resting capture had 7) and the deep tail (both JSON examples, the full passages) lands in the markdown. On the public react-window demo (`/list/fixed-row-height`, 1,000 unkeyed rows), the walk previously returned only its final window (rows 129–142); the unkeyed restore returns rows 1–155 in order with no duplicates, stopping where the 20s budget ends. Remaining limits of the unkeyed path: rows that hold only images or render to canvas cannot be restored (the walk rewinds instead), and engines that *recycle* row nodes in place (rewriting a pooled node's text rather than removing it) are not harvested. One residual limitation, documented as measured: Feishu table blocks lazy-render their *cells* behind a second, in-block virtualization the walk does not chase — table headers and rows are restored, very large payload cells may stay empty.
|
|
161
163
|
|
|
162
164
|
### PDF links
|
|
163
165
|
|
package/README.zh-CN.md
CHANGED
|
@@ -15,9 +15,9 @@
|
|
|
15
15
|
- **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
|
|
16
16
|
- **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
|
|
17
17
|
- **热配置** —— 插件页(侧边栏 → 插件 → *dsh-web-fetch-playwright*,dsh ≥ 0.1.7)在插件自己的包页面直接展示完整表单(也可通过该行的「配置」入口进入同一表单);可随时切换后端、上下文模式、降噪开关与并发数,提交的改动对下一次抓取即时生效,无需重启。
|
|
18
|
-
- **预算控制** —— 单次抓取
|
|
19
|
-
- **Cloudflare 挑战有界等待** —— 导航落到验证中间页("Just a moment…" 及其多语言同族,通过官方 `cf-mitigated: challenge` 响应头 + 结构性页面标记识别)时,抓取保持**同一标签页与上下文**,等待浏览器自行通过验证:跟踪*最后一次*主 frame 响应(真实页面随后重载进来),并轮询活 DOM 以捕获 SPA
|
|
20
|
-
- **视口虚拟化信息流与应用壳文档的深度抓取** —— 论坛帖子、评论区、时间线类页面越来越多只渲染滚动位置附近的内容(Discourse 帖子流把视口外的每一楼都替换成只保留楼层 id、不含任何正文的高度占位符),应用壳文档编辑器(飞书文档一族)更进一步:body 根本不滚动、块以 `data-block-id` 标识而没有 id、窗口外的块被直接移除、只留匿名高度占位符。静止快照只会拿到骨架或首屏而正文静默丢失。抓取因此在序列化前执行一次有界的页内滚动遍历:文档可滚时走窗口,否则走页面主导的内层滚动容器;暂存每个渲染窗口的内容(每个 id/`data-block-id` 保留最丰满的快照),逐步走到底部——只有"触底且高度停止增长"
|
|
18
|
+
- **预算控制** —— 单次抓取 25s 等待预算——预算耗尽时(加载过慢、无限下拉的信息流)不再继续等待,而是直接对页面已渲染的内容做提取并返回,而不是报错(文档未加载完成时标记 `truncated`;只有完全没收到响应、或提取本身超出 10s 宽限时才报 `WEB_FETCH_TIMEOUT`);并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 10s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
|
|
19
|
+
- **Cloudflare 挑战有界等待** —— 导航落到验证中间页("Just a moment…" 及其多语言同族,通过官方 `cf-mitigated: challenge` 响应头 + 结构性页面标记识别)时,抓取保持**同一标签页与上下文**,等待浏览器自行通过验证:跟踪*最后一次*主 frame 响应(真实页面随后重载进来),并轮询活 DOM 以捕获 SPA 式清除。有界(内置 15s 上限),附带同标签页有界重试(`challengeRetries`,默认 1)。窗口耗尽时,先经由页面自身的纯 HTTP 请求通道对同一 URL 做一次**再拉取**(`challengeHttpFallback`,默认开启)——同一个 cookie 罐、浏览器自身的 UA,但没有 Chromium 的渲染决策——因为很多 Cloudflare 边缘会对这样的请求直接下发真实文档,却对浏览器会话扣住交互式挑战不放;回取的响应必须本身不是挑战页,否则仍按失败处理。仍失败时,才以独立的 `WEB_FETCH_CHALLENGE` 错误码明确失败,而不是把中间页当正文返回。全程不点击、不注入验证码答案、不伪造浏览器状态、不导出或复制 cookie。
|
|
20
|
+
- **视口虚拟化信息流与应用壳文档的深度抓取** —— 论坛帖子、评论区、时间线类页面越来越多只渲染滚动位置附近的内容(Discourse 帖子流把视口外的每一楼都替换成只保留楼层 id、不含任何正文的高度占位符),应用壳文档编辑器(飞书文档一族)更进一步:body 根本不滚动、块以 `data-block-id` 标识而没有 id、窗口外的块被直接移除、只留匿名高度占位符。静止快照只会拿到骨架或首屏而正文静默丢失。抓取因此在序列化前执行一次有界的页内滚动遍历:文档可滚时走窗口,否则走页面主导的内层滚动容器;暂存每个渲染窗口的内容(每个 id/`data-block-id` 保留最丰满的快照),逐步走到底部——只有"触底且高度停止增长"才意味着分页加载完毕——然后在页面的结构化克隆里用暂存快照再水合所有退化的键元素、把被移除的键插回其存活兄弟旁边,并把完全没有 DOM 身份的行(react-window 列表、X 式时间线)从引擎移除它们的那一刻起还原回来,并由遍历自行序列化该克隆(活文档从不被改写)。对滚动毫无反应的页面(绝大多数网页)会被识别出来并跳过慢速遍历。受内置 20s 预算约束;任何失败都静默降级为旧版抓取。
|
|
21
21
|
|
|
22
22
|
## 工作原理
|
|
23
23
|
|
|
@@ -32,12 +32,14 @@ web_fetch (tool-web)
|
|
|
32
32
|
├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
|
|
33
33
|
├─ cdp: connectOverCDP(endpoint)
|
|
34
34
|
├─ PDF(inline / attachment / 嗅探出的 octet-stream)→ 字节(导航响应体,否则 request context 重新拉取)→ pdf.js → 版面启发式 → Markdown
|
|
35
|
-
├─ page.goto → 等待稳定(networkidle
|
|
35
|
+
├─ page.goto → 等待稳定(load 基线 + 有界网络静默窗,陈旧在途请求不再阻塞——尽力而为;不再用 Playwright 的 networkidle:worker 脚本与跨域认证请求在页面会话上永不完成,严格空闲在现代 SPA 页面上不可达)→ 深度抓取滚动遍历(虚拟化信息流;有还原时直接返回其合并且已剥离的结果)→ 否则在活 DOM 中剥离脚本子树 → page.content()
|
|
36
36
|
├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → 尾部链接密集块修剪 → DOMPurify → Turndown(GFM)
|
|
37
37
|
└─ Markdown(关闭降噪时返回原始 HTML)
|
|
38
38
|
```
|
|
39
39
|
|
|
40
|
-
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;PDF 下载会被提取为 Markdown(见 [PDF 链接](#pdf-链接)),其他不可解码的下载(ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE`
|
|
40
|
+
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;PDF 下载会被提取为 Markdown(见 [PDF 链接](#pdf-链接)),其他不可解码的下载(ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;持续性传输错误与取消依旧按错误处理——但*瞬时*传输抖动会先重试:`net::ERR_NETWORK_CHANGED`(Chromium 的 NetworkChangeNotifier 因宿主机网络配置在导航途中变化——路由/DNS/VPN/Docker 内嵌 DNS 改写——而掐灭了在途请求;实测:某会话抓取 source.android.com 一次即如此失败,同一 URL 数秒后从同一宿主机干净抓回)及其同族错误码(`ERR_CONNECTION_RESET`、`ERR_NAME_NOT_RESOLVED` 等)会在同一页面上带短退避地重导航至多两次,且受剩余抓取预算约束;`ERR_ABORTED` 与 Playwright 超时错误从不重试(它们的形态是确定性的,已由上述各恢复逻辑分别接管)。
|
|
41
|
+
|
|
42
|
+
**Web Component 应用外壳跳过它注定赢不了的 Readability 家族,大页面也不再反复重解析同一份 DOM。** 降噪管线是同步的且以 jsdom 为主,重页面会在浏览器已拿到完整文档之后把全部开销变成"死等"——实测坏例(`aistudio.google.com/docs/computer-use`,经 CDP 抓取进用户的真实浏览器):页面 ~7 秒渲染完整,之后管线又在 1.1MB 的捕获上花掉 13.4 秒,其中 ~6.5 秒是三次 Readability、~2.9 秒是八次 DOMPurify 重解析。而这些开销里有相当一部分产出的是*可证明的死候选*:该页是 Angular 应用,整个 DOM 都在 `<ms-app>` 之下,而 DOMPurify 的白名单不含带连字符的自定义元素标签,配合 `KEEP_CONTENT: false`,凡携带该包装的提取物净化后只剩空壳(Readability 的选中项:词数为零的根;整文档候选:约 4.5k 词只剩 39 个)。因此当 body 可见文本有 ≥90% 位于自定义元素子树内时(`customWrappedTextShare`),候选生成直接跳过 Readability 家族(`art`/`artu`/`artx`)并在 facts 中标记(`readabilitySkipped`);选择器会被注入死选中项本应测得的空 `art` 参考统计,存活候选的特征因此不变,由 DOM 兜底候选、noscript 路径和块分类器——选择器在这种页面上本来就会选中的那些——承担提取(标题回退到 `og:title`/`<title>`)。另外,生成器本来就持有 DOM 节点的候选(`doc`、`docc`、`main`、`ns_doc`)改为按节点净化(输出逐字节一致,整文档候选实测 555ms → 304ms),不再重解析其序列化文本;Readability 文章对覆盖门限与尾部修剪只解析一次;推荐位名称识别先用字面子串短路再做逐 token 切分(坏例页上省 ~360ms)。坏例页的净效果:管线 13.4s → 5.0s,端到端 21–23s → 13–15s,胜出策略与内容不变;在 63 页的基准输出对比样本上(门控一页都未触发)输出逐字节一致,仅坏例页本身多保留了一行标签文本。
|
|
41
43
|
|
|
42
44
|
**回读文档前,先在活 DOM 中剥离非内容脚本子树 —— 构造上即输出中性。** 降噪管线(以及关闭降噪时工具层自己的渲染器)会用 jsdom 重新解析 `page.content()`,而脚本的*文本内容*是这条往返链路中唯一无法信任的东西:页面自身的运行时 JS 可能把内嵌 `</script>` 的文本写回某个数据脚本的 textContent —— 实测于 techcommunity.microsoft.com(Khoros/Aurora 平台),其客户端把整个 Apollo 缓存经 `JSON.stringify` 写回 `script#__NEXT_DATA__`,把线上响应里 712 处 `\u003c` 转义还原成了裸 `<` —— 而浏览器序列化脚本文本时原样输出,于是这份扁平化的 HTML 字符串会在*第一个*内嵌结束标签处终止脚本,把余下约 270KB 的 JSON *当作标记语言*解析。解析出的幽灵元素(`<id>` 之类的未知标签、塞满 CSS 与 i18n 字符串的锚点)恰好是 Readability 评分器眼里的"正文质量"(实测 126 分对真实正文的 45 分),于是抓取 RDP 10 AVC/H.264 一文时返回的是 CSS 规则和 `CachedAsset` 字符串而不是文章本体。事后再对字符串做补丁无法消除这种歧义 —— 只有浏览器自己的解析持有文档的真实结构 —— 因此普通的 `script` 与 `template` 子树在 `page.content()` 之前先在活页面里剥掉(`page.evaluate`,尽力而为)。这不会改变管线以往的任何输出:Readability 本就在抽取与评分前删除 script/noscript 节点,DOMPurify 的白名单本就丢弃 script,工具层渲染器也整体移除 SCRIPT/TEMPLATE —— 在真实页面语料(Wikipedia、MDN、Hacker News、坏例页 wire 版,加上 noscript/JSON-LD/template 边界用例)上的差分 A/B **每一页都字节级一致**。两个刻意的保留让间接消费方也不受损:`script[type="application/ld+json"]` 只做*消险*而非删除(`</` 改写为 `<\/` —— 仍是合法 JSON,Readability 的元数据读取解析到同一对象,文章标题保持无站点后缀的干净形态,如 "Remote Desktop Protocol" 而非 "… - Wikipedia");`noscript` 完整保留,因为 Readability 的 `_unwrapNoscriptImages` 会从懒加载的 `<noscript><img …>` 包裹中抢救出真实图片。该剥离在关闭降噪时同样生效,让返回给工具层渲染器的原始 HTML 也是序列化安全的,并把管线要解析的体积压缩约 70%(以该坏例页计)。
|
|
43
45
|
|
|
@@ -79,10 +81,9 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
|
|
|
79
81
|
| `shareBrowserContext` | `true` | 仅 CDP:单选共享 / 隔离。`true` 复用远端默认 context 与登录态;`false` 使用独立 context,生命周期由下方范围决定。本地后端忽略。 |
|
|
80
82
|
| `contextIsolationScope` | `global` | 仅 CDP 隔离模式:`session` 按 DSH 会话按需创建 context,无活动/待处理抓取且无打开的标签页后关闭(下次抓取使用全新状态);`global` 在同一插件实例内跨DSH会话复用一个固定的隔离上下文。 |
|
|
81
83
|
| `denoise` | `true` | 是否启用降噪;关闭时返回整页渲染 HTML,交由工具层转换。 |
|
|
82
|
-
| `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;
|
|
83
|
-
| `
|
|
84
|
-
| `
|
|
85
|
-
| `scrollBudgetMs` | `20000` | 深度抓取滚动遍历的**有界**预算(毫秒,0–30000)。`0` 关闭——按页面静止状态抓取(旧行为),会丢失视口虚拟化信息流的正文。 |
|
|
84
|
+
| `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;10s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
|
|
85
|
+
| `challengeRetries` | `1` | 一个等待窗口耗尽后的**同标签页**重新导航次数(0–3);浏览器已拿到的通关 cookie 留在上下文里供重试使用。总耗时始终受 25s 单次抓取等待预算约束。 |
|
|
86
|
+
| `challengeHttpFallback` | `true` | 浏览器在窗口内清不掉挑战时,经页面自身的纯 HTTP 请求通道对同一 URL 再拉取一次(同 cookie、浏览器 UA、无渲染决策),回取到非挑战文档则直接返回。`false` 保持纯浏览器姿态。 |
|
|
86
87
|
|
|
87
88
|
本地后端解析顺序:
|
|
88
89
|
|
|
@@ -123,16 +124,17 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
|
|
|
123
124
|
|
|
124
125
|
### Cloudflare 挑战处理(有界自然等待)
|
|
125
126
|
|
|
126
|
-
部分严格站点会在返回真实页面前先给一个 Cloudflare 验证中间页。真实浏览器通常几秒内就能**自行**通过验证;但只看第一次响应的抓取会把中间页当成正文返回(0.2.5
|
|
127
|
+
部分严格站点会在返回真实页面前先给一个 Cloudflare 验证中间页。真实浏览器通常几秒内就能**自行**通过验证;但只看第一次响应的抓取会把中间页当成正文返回(0.2.5 之前的旧行为;可在仓库检出、执行 `pnpm build` 后运行 `node scripts/challenge-demo.mjs` 看本地模拟站点的前后对比、`node scripts/challenge-online.mjs <url>` 对真实站点做在线对比)。
|
|
127
128
|
|
|
128
129
|
开启等待(默认)后的流程:
|
|
129
130
|
|
|
130
131
|
1. **识别** —— 响应带 `cf-mitigated: challenge`(Cloudflare 官方文档注明所有挑战页类型都带此头),或 403/503 且 `server: cloudflare` 的 HTML 文档,或本地化的中间页本身("Just a moment…" / "请稍候…" / "Минутку…" 等 title 家族,以及结构性标记:挑战运行时的 `<script src>`、`#challenge-*` 元素、`cf-chl-widget-` 框架、`window._cf_chl_opt`)。内容级标记只是**兜底层**,且仅对"挑战兼容"的响应(403/429/503 或来自 Cloudflare 边缘——`server: cloudflare` / `cf-ray`)运行——因为中间页从不会以普通 200 返回,所以正文里引用了挑战文案的普通文章绝不可能被误判。两处收紧让这个承诺真正成立:挑战平台脚本只读真实的 `<script src>` 属性(正文、以及转义后的代码示例里引用的路径一律忽略),并且**正面**匹配运行时挂载点 `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/`。该前缀下 Cloudflare 注入的其余脚本——Bot Management 的 JavaScript Detections(`scripts/jsd/`、`h/[b|g]/scripts/jsd/<hash>/`、`h/[b|g]/jsd/r/`)与 Precursor(`scripts/precursor/`)——都是会注入受保护站点**每一个正常页面**的被动探测脚本(如 openrouter.ai、nodeseek.com),永远不算挑战信号。硬封锁页("Sorry, you have been blocked")单独分类并立即失败——等待无法解除。
|
|
131
132
|
2. **同标签页、同上下文的有界等待** —— 每 500ms 轮询活 DOM,等浏览器跑完自己的验证;同时跟踪**最后一次主 frame 导航响应**,所以重载进来的真实文档的状态码和响应头才是最终上报的。SPA 式清除(无导航、纯内容替换)由同一个 DOM 探测捕获。
|
|
132
133
|
3. **有界重试** —— 窗口耗尽后,同一标签页默认再导航一次(`challengeRetries`),上下文里已有的通关 cookie 继续生效。
|
|
133
|
-
4.
|
|
134
|
+
4. **死局回取** —— 失败之前,先经页面自身的请求通道对同一 URL 做一次纯 HTTP 再拉取(`challengeHttpFallback`,默认开启):同一个 cookie 罐、浏览器自身的 User-Agent,没有 Chromium 的渲染决策。很多 Cloudflare 边缘对这样的请求直接下发真实文档,却对浏览器会话扣住交互式挑战(实测:数据中心 IP 访问 v2ex.com——浏览器 15s×2 只拿到空的 widget 帧,纯 HTTP 请求拿到完整正文)。回取的响应必须本身不是挑战 HTML——响应级与内容级双重校验,与浏览器路径同一条分类链——否则保持浏览器-only 的判定。
|
|
135
|
+
5. **明确失败** —— 返回独立的 `WEB_FETCH_CHALLENGE` 错误码(web seam 的 `code` 是开放字符串,允许 provider 专属码),消息中写明站点、等待预算与最后一次挑战响应的状态。
|
|
134
136
|
|
|
135
|
-
安全边界(刻意为之):不点击 Turnstile、不解验证码、不注入 token、不伪装指纹/UA、不轮换代理、不导出 cookie——CDP 隔离模式下通关态保留在对应会话 / 全局 context 中,直到该 context 被清理(本地 context 仍随每次抓取结束);profile 模式下它留在远端浏览器自己的 profile
|
|
137
|
+
安全边界(刻意为之):不点击 Turnstile、不解验证码、不注入 token、不伪装指纹/UA、不轮换代理、不导出 cookie——CDP 隔离模式下通关态保留在对应会话 / 全局 context 中,直到该 context 被清理(本地 context 仍随每次抓取结束);profile 模式下它留在远端浏览器自己的 profile 里,插件从不复制或清理。等待始终受内置 15s 上限与 25s 单次抓取等待预算双重约束,永不无限阻塞。
|
|
136
138
|
|
|
137
139
|
### 深度抓取(视口虚拟化信息流与应用壳文档)
|
|
138
140
|
|
|
@@ -151,13 +153,13 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
|
|
|
151
153
|
修复是一次有界的页内滚动遍历(`src/autoscroll.ts`),在每个 HTML 文档的 settle 之后、序列化剥除之前执行:
|
|
152
154
|
|
|
153
155
|
1. **解析遍历目标**:文档自身可滚时走窗口(Discourse 形态);否则找页面的主导内层滚动容器(应用壳形态)——在 computed `overflow-y: auto|scroll` 的元素里取可滚距离最大者,且门槛为半个视口,下拉框级别的小滚动容器永远不会触发遍历。两者皆无则不遍历。
|
|
154
|
-
2. **暂存**每个键元素——以元素 id 为键,无 id 则用 `data-block-id
|
|
155
|
-
3. **遍历**:从顶部起每次下移 70% 目标视口高度,每步为 300ms 平滑动画(按时间插值;`scrollTop` 驱动内层容器与 `window.scrollTo` 驱动文档完全同效)加 300ms 静置供信息流引擎的防抖 uncloak 生效;直到能在底部持续 2.5s **耐心窗口**而高度无增长(真实信息流的分块拉取是网络往返,任何增长都会重新计时),或步数/预算上限关闭遍历。一个 `MutationObserver` 观察遍历存在的全部理由——页面因滚动而增删元素。连续两个谨慎步都无反应时,页面被视为对滚动惰性,剩余部分用一次跳到底部来探测;跳跃后一旦出现任何反应(uncloak、重新 cloak
|
|
156
|
-
4. **合并**到结构化克隆(`document.cloneNode(true)`)中:没有序列化/重解析往返,运行时改写进脚本文本的字面 `</script>` 无法污染合并结果;克隆没有浏览上下文,不执行脚本、不升级自定义元素、不加载 frame。**插回**(仅容器遍历):来自滚动容器内、而克隆中已不存在的暂存键是被引擎**移除**的(飞书形态,占位符是匿名的),插回到其记录的兄弟旁边,迭代求解,使一段被移除的连续块能从任一存活端链式还原;随已插回父级一同回来的键直接跳过。定位是结构性的,与遍历**看到**各行的顺序无关——包括从文档中部起步的锚点打开。**替换**:每个比快照更空的键元素(子元素更少,或文本明显更少——即 Discourse
|
|
156
|
+
2. **暂存**每个键元素——以元素 id 为键,无 id 则用 `data-block-id`(加命名空间前缀,两种属性永不冲突)——为脱离文档的克隆(框架生成的 id 不算键:Ember 的 `ember123`、React `useId` 的 `:r1:`、`«r1»`、`_r_1_` 每次渲染都会变,这类元素走下文的无键路径),每个键保留**最丰满**的一次观测(先以占位符形态被看到的行,会在渲染形态出现时升级),并记录其最近的带键兄弟:即它在自己所属列表中的位置。单条体积上限约 64KB:虚拟化行只有几 KB,而几十 KB 的元素只可能是整个信息流的容器(linux.do 实测:六层嵌套包装加上 `data-preloaded` 和 `svg-sprites`,一次收集就把 2MB 预算烧掉 1.9MB),这类键会被记住、此后不再序列化。
|
|
157
|
+
3. **遍历**:从顶部起每次下移 70% 目标视口高度,每步为 300ms 平滑动画(按时间插值;`scrollTop` 驱动内层容器与 `window.scrollTo` 驱动文档完全同效)加 300ms 静置供信息流引擎的防抖 uncloak 生效;直到能在底部持续 2.5s **耐心窗口**而高度无增长(真实信息流的分块拉取是网络往返,任何增长都会重新计时),或步数/预算上限关闭遍历。一个 `MutationObserver` 观察遍历存在的全部理由——页面因滚动而增删元素。连续两个谨慎步都无反应时,页面被视为对滚动惰性,剩余部分用一次跳到底部来探测;跳跃后一旦出现任何反应(uncloak、重新 cloak、加载更多的追加),遍历回退到起跳位置并谨慎走完。正确性从不依赖热身阶段的判断,只有惰性页面的开销依赖它。同一个观察器还负责**收割无键行**:引擎移除一个没有键的元素时,记录里仍有完整的已脱离元素,以及它被移除时左右的兄弟节点,合并时就把它放回那里。全程不读布局,绝对定位和 `transform` 都不影响。每个列表里,每个行身份只暂存一次(每帧重渲染的引擎会把同一行移除成千上万次,之后的观测只作锚点用),并且使用独立的字符预算,不会挤占带键暂存。瞬时 UI 从不收割(`role=status|alert|dialog|tooltip|progressbar|menu|…`、`aria-hidden`、`aria-busy`、无文本元素)。从未产生反应、也没有发出 fetch/XHR 的页面,在底部只停留 900ms,而不是完整的 2.5s 耐心窗口。实测:约 30 屏的静态长文开销约 2.4s,而不是耗尽 20s 预算;不需要滚动的页面没有任何额外开销。
|
|
158
|
+
4. **合并**到结构化克隆(`document.cloneNode(true)`)中:没有序列化/重解析往返,运行时改写进脚本文本的字面 `</script>` 无法污染合并结果;克隆没有浏览上下文,不执行脚本、不升级自定义元素、不加载 frame。**插回**(仅容器遍历):来自滚动容器内、而克隆中已不存在的暂存键是被引擎**移除**的(飞书形态,占位符是匿名的),插回到其记录的兄弟旁边,迭代求解,使一段被移除的连续块能从任一存活端链式还原;随已插回父级一同回来的键直接跳过。定位是结构性的,与遍历**看到**各行的顺序无关——包括从文档中部起步的锚点打开。**替换**:每个比快照更空的键元素(子元素更少,或文本明显更少——即 Discourse 占位符)被替换为快照,但**容器**除外(其后代中有待替换元素的祖先),容器由这些后代拼装还原——容器自身的快照只是某一窗口的局部视图。**还原**(无键行,在替换之前进行):只有在遍历期间引擎既从某个列表移除过带文本的行、又向它添加过元素时,该列表才符合条件。单个 toast 或加载更多的追加不算,位于 `aria-live="assertive"`、`role=alert|status|timer|marquee` 或轮播区域内的也不算。列表里已不存在的被移除行,会放回它被移除时所在的兄弟旁边。如果锚点本身也被移除了,就顺着它自己的邻居继续找;同一行的重复观测,解析到该行暂存副本最终落下的位置。每一次观测都会尝试,所以锚点打开时的首屏行,可以由后来看到它与邻居相邻的那一轮来定位。被整体重渲染(`innerHTML`)的列表没有存活锚点,这些行按移除顺序放到第一个带文本的行之前:自上而下、止于底部的遍历只会丢失最终窗口上方的内容。与仍存在的行去重时,引擎设置了按列表的身份属性就用它(`data-index`、`data-item-index`、`aria-rowindex`、`aria-posinset`、`data-key`、`data-id` 等,限定在本列表内,不同列表间的重复值不会冲突),否则用行文本。身份属性相同但明显更空的现存行视为占位符,会被替换。引擎放在原行位置上的无键占位(紧挨该行、文本很短且被该行包含)会被吸收。已还原的列表及其祖先在替换阶段一律视为容器,带键的包装元素不会被替换成只有一个窗口的快照。随后克隆经过与活文档读取相同的脚本剥除(`src/strip.ts`)并序列化,provider 读取该字符串而非活页面。页面有反应但什么都没能还原时(行没有文本、canvas 渲染),遍历会滚回页面打开时的位置,等页面静止后再交给活文档读取;否则 provider 读到的是遍历停在底部的最后一个窗口,而不是旧版抓取看到的首屏。对滚动惰性的页面保持原位,它的 DOM 怎样都一样。
|
|
157
159
|
|
|
158
|
-
|
|
160
|
+
遍历在两种降噪模式下都运行;受内置 20s 预算与剩余等待预算的双重约束——在预算处截停的遍历仍会合并并返回已暂存的内容,正文读取与解码在其后的 10s 提取宽限窗内完成。任何失败——页面没有 `evaluate`、求值被拒、引擎拒绝遍历——都静默降级为旧版抓取:遍历只会增加内容,不会让抓取失败。CDP profile 模式下遍历滚动的是远端浏览器里抓取自有的那个标签页,与渲染发生在同一处。
|
|
159
161
|
|
|
160
|
-
真实坏例实测:linux.do 帖子(51 楼、约 18kpx)在 20s 预算内一次遍历走完,每楼都被再水合、零占位符残留;飞书 wiki 页面在 20s 预算中用约 11.5s 走完内层容器(15 步、插回 38 个被移除的块、最终 DOM 191 块——静止抓取只有 7 块),深部尾文(两处 JSON 示例与完整段落)都进入 markdown
|
|
162
|
+
真实坏例实测:linux.do 帖子(51 楼、约 18kpx)在 20s 预算内一次遍历走完,每楼都被再水合、零占位符残留;飞书 wiki 页面在 20s 预算中用约 11.5s 走完内层容器(15 步、插回 38 个被移除的块、最终 DOM 191 块——静止抓取只有 7 块),深部尾文(两处 JSON 示例与完整段落)都进入 markdown。公开的 react-window 演示页(`/list/fixed-row-height`,1000 个无键行)上,遍历以前只返回最后一个窗口(第 129–142 行);无键还原后按顺序返回第 1–155 行,没有重复,在 20s 预算耗尽处停止。无键路径的剩余限制:只有图片或用 canvas 渲染的行无法还原(遍历改为回滚);原地**复用**行节点的引擎(改写池中节点的文本,而不是移除它)不会被收割。一项如实记录的残余限制:飞书表格块的**单元格**藏在第二层块内虚拟化后面,遍历不追——表头与行能还原,超大载荷单元格可能保持为空。
|
|
161
163
|
|
|
162
164
|
### PDF 链接
|
|
163
165
|
|
package/cordis.patch.yml
CHANGED
package/lib/client.js
CHANGED
|
@@ -370,9 +370,7 @@ window.__ModuleLoader__.load({
|
|
|
370
370
|
checkboxField("shareBrowserContext"),
|
|
371
371
|
radioField("contextIsolationScope", ["session", "global"]),
|
|
372
372
|
checkboxField("denoise"),
|
|
373
|
-
numberField("maxConcurrency", 1, 200)
|
|
374
|
-
numberField("challengeWaitMs", 0, 6e4),
|
|
375
|
-
numberField("scrollBudgetMs", 0, 3e4)
|
|
373
|
+
numberField("maxConcurrency", 1, 200)
|
|
376
374
|
]);
|
|
377
375
|
this.store = this.form.bind(() => this.projection());
|
|
378
376
|
}
|
|
@@ -385,9 +383,7 @@ window.__ModuleLoader__.load({
|
|
|
385
383
|
shareBrowserContext: this.form.field("shareBrowserContext"),
|
|
386
384
|
contextIsolationScope: this.form.field("contextIsolationScope"),
|
|
387
385
|
denoise: this.form.field("denoise"),
|
|
388
|
-
maxConcurrency: this.form.field("maxConcurrency")
|
|
389
|
-
challengeWaitMs: this.form.field("challengeWaitMs"),
|
|
390
|
-
scrollBudgetMs: this.form.field("scrollBudgetMs")
|
|
386
|
+
maxConcurrency: this.form.field("maxConcurrency")
|
|
391
387
|
};
|
|
392
388
|
}
|
|
393
389
|
/**
|
|
@@ -418,11 +414,11 @@ window.__ModuleLoader__.load({
|
|
|
418
414
|
}
|
|
419
415
|
var PluginCard_module_css_default = {
|
|
420
416
|
"footer": "UM8-Rq_footer",
|
|
421
|
-
"discard": "UM8-Rq_discard",
|
|
422
|
-
"form": "UM8-Rq_form",
|
|
423
417
|
"save": "UM8-Rq_save",
|
|
424
418
|
"readOnly": "UM8-Rq_readOnly",
|
|
425
|
-
"
|
|
419
|
+
"form": "UM8-Rq_form",
|
|
420
|
+
"failed": "UM8-Rq_failed",
|
|
421
|
+
"discard": "UM8-Rq_discard"
|
|
426
422
|
};
|
|
427
423
|
//#endregion
|
|
428
424
|
//#region src/client/PluginCard.tsx
|
|
@@ -482,27 +478,27 @@ window.__ModuleLoader__.load({
|
|
|
482
478
|
document.head.appendChild(tag);
|
|
483
479
|
}
|
|
484
480
|
var fields_module_css_default = {
|
|
485
|
-
"
|
|
486
|
-
"
|
|
487
|
-
"
|
|
481
|
+
"badges": "CFSaVG_badges",
|
|
482
|
+
"reset": "CFSaVG_reset",
|
|
483
|
+
"checkboxRow": "CFSaVG_checkboxRow",
|
|
484
|
+
"badge": "CFSaVG_badge",
|
|
488
485
|
"radioOptionPlain": "CFSaVG_radioOptionPlain",
|
|
486
|
+
"field": "CFSaVG_field",
|
|
487
|
+
"input": "CFSaVG_input",
|
|
488
|
+
"radioOption": "CFSaVG_radioOption",
|
|
489
489
|
"radioPick": "CFSaVG_radioPick",
|
|
490
490
|
"inputInvalid": "CFSaVG_inputInvalid",
|
|
491
491
|
"radioLabel": "CFSaVG_radioLabel",
|
|
492
|
+
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
493
|
+
"invalid": "CFSaVG_invalid",
|
|
494
|
+
"radioContent": "CFSaVG_radioContent",
|
|
492
495
|
"radioGroup": "CFSaVG_radioGroup",
|
|
493
|
-
"
|
|
494
|
-
"radioText": "CFSaVG_radioText",
|
|
496
|
+
"label": "CFSaVG_label",
|
|
495
497
|
"hint": "CFSaVG_hint",
|
|
496
|
-
"invalid": "CFSaVG_invalid",
|
|
497
498
|
"radioHint": "CFSaVG_radioHint",
|
|
498
|
-
"
|
|
499
|
-
"radioContent": "CFSaVG_radioContent",
|
|
500
|
-
"radioOption": "CFSaVG_radioOption",
|
|
501
|
-
"badges": "CFSaVG_badges",
|
|
499
|
+
"hintIndented": "CFSaVG_hintIndented",
|
|
502
500
|
"head": "CFSaVG_head",
|
|
503
|
-
"
|
|
504
|
-
"label": "CFSaVG_label",
|
|
505
|
-
"reset": "CFSaVG_reset"
|
|
501
|
+
"radioText": "CFSaVG_radioText"
|
|
506
502
|
};
|
|
507
503
|
//#endregion
|
|
508
504
|
//#region src/client/fields.tsx
|
|
@@ -831,40 +827,6 @@ window.__ModuleLoader__.load({
|
|
|
831
827
|
onReset: () => {
|
|
832
828
|
props.resetField("maxConcurrency");
|
|
833
829
|
}
|
|
834
|
-
}),
|
|
835
|
-
/* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
|
|
836
|
-
id: "plugin-config-playwright-challenge-wait",
|
|
837
|
-
label: t("challengeWaitMs"),
|
|
838
|
-
hint: t("challengeWaitMsHint"),
|
|
839
|
-
placeholder: t("challengeWaitMsPlaceholder"),
|
|
840
|
-
overriddenLabel: t("overridden"),
|
|
841
|
-
resetLabel: t("reset"),
|
|
842
|
-
invalidLabel: t("invalidText"),
|
|
843
|
-
disabled,
|
|
844
|
-
...state.challengeWaitMs,
|
|
845
|
-
onEdit: (text) => {
|
|
846
|
-
props.edit("challengeWaitMs", text);
|
|
847
|
-
},
|
|
848
|
-
onReset: () => {
|
|
849
|
-
props.resetField("challengeWaitMs");
|
|
850
|
-
}
|
|
851
|
-
}),
|
|
852
|
-
/* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
|
|
853
|
-
id: "plugin-config-playwright-scroll-budget",
|
|
854
|
-
label: t("scrollBudgetMs"),
|
|
855
|
-
hint: t("scrollBudgetMsHint"),
|
|
856
|
-
placeholder: t("scrollBudgetMsPlaceholder"),
|
|
857
|
-
overriddenLabel: t("overridden"),
|
|
858
|
-
resetLabel: t("reset"),
|
|
859
|
-
invalidLabel: t("invalidText"),
|
|
860
|
-
disabled,
|
|
861
|
-
...state.scrollBudgetMs,
|
|
862
|
-
onEdit: (text) => {
|
|
863
|
-
props.edit("scrollBudgetMs", text);
|
|
864
|
-
},
|
|
865
|
-
onReset: () => {
|
|
866
|
-
props.resetField("scrollBudgetMs");
|
|
867
|
-
}
|
|
868
830
|
})
|
|
869
831
|
]
|
|
870
832
|
});
|
|
@@ -899,12 +861,6 @@ window.__ModuleLoader__.load({
|
|
|
899
861
|
maxConcurrency: "Max concurrent fetches",
|
|
900
862
|
maxConcurrencyHint: "How many pages may render at once (1–200). Blank = auto: 4 local browsers, 50 CDP tabs in the remote browser.",
|
|
901
863
|
maxConcurrencyPlaceholder: "(auto: local 4 / CDP 50)",
|
|
902
|
-
challengeWaitMs: "Cloudflare challenge wait (ms)",
|
|
903
|
-
challengeWaitMsHint: "Bounded wait for a Cloudflare challenge to clear naturally in the same tab (0–60000; 0 = off). 0 disables: the first response is returned as-is.",
|
|
904
|
-
challengeWaitMsPlaceholder: "(default: 15000)",
|
|
905
|
-
scrollBudgetMs: "Deep-capture scroll budget (ms)",
|
|
906
|
-
scrollBudgetMsHint: "Bounded in-page scroll walk that captures viewport-virtualized feeds (forum topics, comment threads) whole: posts outside the render window are empty placeholders at rest. 0–30000; 0 = off (capture the page as it rests).",
|
|
907
|
-
scrollBudgetMsPlaceholder: "(default: 20000)",
|
|
908
864
|
overridden: "Overridden",
|
|
909
865
|
reset: "Reset to default",
|
|
910
866
|
readOnly: "This deployment stores settings read-only.",
|
|
@@ -942,12 +898,6 @@ window.__ModuleLoader__.load({
|
|
|
942
898
|
maxConcurrency: "最大并发抓取数",
|
|
943
899
|
maxConcurrencyHint: "同时渲染的页面上限(1–200);留空自动:本地 4 个浏览器,CDP 在远端浏览器里 50 个标签页。",
|
|
944
900
|
maxConcurrencyPlaceholder: "(自动:本地 4 / CDP 50)",
|
|
945
|
-
challengeWaitMs: "Cloudflare 挑战等待上限(毫秒)",
|
|
946
|
-
challengeWaitMsHint: "在同一标签页内有界等待 Cloudflare 验证自然通过(0–60000;0 = 关闭)。关闭时直接返回首次响应——旧版行为。",
|
|
947
|
-
challengeWaitMsPlaceholder: "(默认:15000)",
|
|
948
|
-
scrollBudgetMs: "深度抓取滚动预算(毫秒)",
|
|
949
|
-
scrollBudgetMsHint: "有界的页内滚动遍历,完整捕获视口虚拟化的信息流(论坛帖子、评论区):静止时渲染窗口外的帖子只是空占位符。0–30000;0 = 关闭(按页面静止状态抓取)。",
|
|
950
|
-
scrollBudgetMsPlaceholder: "(默认:20000)",
|
|
951
901
|
overridden: "已覆盖",
|
|
952
902
|
reset: "恢复默认",
|
|
953
903
|
readOnly: "本部署的设置为只读。",
|