dsh-web-fetch-playwright 0.2.14 → 0.2.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -3
- package/README.zh-CN.md +5 -3
- package/lib/client.js +17 -17
- package/lib/index.js +212 -21
- package/lib/types/challenge.d.ts +23 -9
- package/lib/types/markdown.d.ts +18 -10
- package/package.json +1 -1
- package/src/challenge.ts +36 -11
- package/src/markdown.ts +127 -16
- package/src/provider.ts +79 -0
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
|
|
|
9
9
|
## Features
|
|
10
10
|
|
|
11
11
|
- **Real browser rendering** — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
|
|
12
|
-
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `` so they cannot flood the body.
|
|
12
|
+
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `` so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content *is* the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content.
|
|
13
13
|
- **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
|
|
14
14
|
- **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
|
|
15
15
|
- **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
|
|
@@ -29,13 +29,15 @@ web_fetch (tool-web)
|
|
|
29
29
|
└─ ctx.web.fetchProvider = playwright
|
|
30
30
|
├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
|
|
31
31
|
├─ cdp: connectOverCDP(endpoint)
|
|
32
|
-
├─ page.goto → settle (networkidle, best-effort) → page.content()
|
|
33
|
-
├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
|
|
32
|
+
├─ page.goto → settle (networkidle, best-effort) → strip scripting subtrees in the live DOM → page.content()
|
|
33
|
+
├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → DOMPurify → Turndown(GFM)
|
|
34
34
|
└─ Markdown (or raw HTML when denoise is off)
|
|
35
35
|
```
|
|
36
36
|
|
|
37
37
|
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; undecodable downloads (a PDF, a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
|
|
38
38
|
|
|
39
|
+
**Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction.** The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses `page.content()` with jsdom, and a script's *text* is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal `</script>` embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`, un-escaping the wire's 712 `\u003c` sequences into raw `<` — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the *first* embedded end tag and parses the remaining ~270KB of JSON *as markup*. The phantom elements that survive (unknown tags like `<id>`, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and `CachedAsset` strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary `script` and `template` subtrees are stripped inside the live page (`page.evaluate`, best-effort) before `page.content()`. This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back **byte-identical on every page**. Two deliberate carve-outs keep even the indirect consumers intact: `script[type="application/ld+json"]` is *defused* rather than deleted (`</` rewritten to `<\/` — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and `noscript` is kept entirely because Readability's `_unwrapNoscriptImages` salvages the real image out of lazy-load `<noscript><img …>` wrappers. The strip also applies with `denoise` off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).
|
|
40
|
+
|
|
39
41
|
## Requirements
|
|
40
42
|
|
|
41
43
|
- DSH web profile (`dsh web`), Node.js ≥ 20.
|
package/README.zh-CN.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
## 特性
|
|
10
10
|
|
|
11
11
|
- **真实浏览器渲染** —— 以用户视角加载页面,SPA 客户端渲染内容也能抓到,而非只有原始 HTML。
|
|
12
|
-
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 ``,避免 base64
|
|
12
|
+
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 ``,避免 base64 字符流刷屏。对列表型页面(门户首页、商品目录——正文本身就是链接),Readability 的链接密度打分会误选出噪声碎片(实测 163.com 首页:96 字符的页脚备案块胜出,320 条新闻标题得分趋零),因此抽取结果需通过**覆盖率置信门**——不足 250 字符、或占页面可见文本不足 10% 即不予采信,转而回退到整页消毒转换,此类页面上可基本完整保留列表内容。
|
|
13
13
|
- **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
|
|
14
14
|
- **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
|
|
15
15
|
- **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
|
|
@@ -29,13 +29,15 @@ web_fetch (tool-web)
|
|
|
29
29
|
└─ ctx.web.fetchProvider = playwright
|
|
30
30
|
├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
|
|
31
31
|
├─ cdp: connectOverCDP(endpoint)
|
|
32
|
-
├─ page.goto → 等待稳定(networkidle,尽力而为)→ page.content()
|
|
33
|
-
├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → DOMPurify → Turndown(GFM)
|
|
32
|
+
├─ page.goto → 等待稳定(networkidle,尽力而为)→ 在活 DOM 中剥离脚本子树 → page.content()
|
|
33
|
+
├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → DOMPurify → Turndown(GFM)
|
|
34
34
|
└─ Markdown(关闭降噪时返回原始 HTML)
|
|
35
35
|
```
|
|
36
36
|
|
|
37
37
|
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;不可解码的下载(PDF、ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;传输错误与取消也依旧按错误处理。
|
|
38
38
|
|
|
39
|
+
**回读文档前,先在活 DOM 中剥离非内容脚本子树 —— 构造上即输出中性。** 降噪管线(以及关闭降噪时工具层自己的渲染器)会用 jsdom 重新解析 `page.content()`,而脚本的*文本内容*是这条往返链路中唯一无法信任的东西:页面自身的运行时 JS 可能把内嵌 `</script>` 的文本写回某个数据脚本的 textContent —— 实测于 techcommunity.microsoft.com(Khoros/Aurora 平台),其客户端把整个 Apollo 缓存经 `JSON.stringify` 写回 `script#__NEXT_DATA__`,把线上响应里 712 处 `\u003c` 转义还原成了裸 `<` —— 而浏览器序列化脚本文本时原样输出,于是这份扁平化的 HTML 字符串会在*第一个*内嵌结束标签处终止脚本,把余下约 270KB 的 JSON *当作标记语言*解析。解析出的幽灵元素(`<id>` 之类的未知标签、塞满 CSS 与 i18n 字符串的锚点)恰好是 Readability 评分器眼里的"正文质量"(实测 126 分对真实正文的 45 分),于是抓取 RDP 10 AVC/H.264 一文时返回的是 CSS 规则和 `CachedAsset` 字符串而不是文章本体。事后再对字符串做补丁无法消除这种歧义 —— 只有浏览器自己的解析持有文档的真实结构 —— 因此普通的 `script` 与 `template` 子树在 `page.content()` 之前先在活页面里剥掉(`page.evaluate`,尽力而为)。这不会改变管线以往的任何输出:Readability 本就在抽取与评分前删除 script/noscript 节点,DOMPurify 的白名单本就丢弃 script,工具层渲染器也整体移除 SCRIPT/TEMPLATE —— 在真实页面语料(Wikipedia、MDN、Hacker News、坏例页 wire 版,加上 noscript/JSON-LD/template 边界用例)上的差分 A/B **每一页都字节级一致**。两个刻意的保留让间接消费方也不受损:`script[type="application/ld+json"]` 只做*消险*而非删除(`</` 改写为 `<\/` —— 仍是合法 JSON,Readability 的元数据读取解析到同一对象,文章标题保持无站点后缀的干净形态,如 "Remote Desktop Protocol" 而非 "… - Wikipedia");`noscript` 完整保留,因为 Readability 的 `_unwrapNoscriptImages` 会从懒加载的 `<noscript><img …>` 包裹中抢救出真实图片。该剥离在关闭降噪时同样生效,让返回给工具层渲染器的原始 HTML 也是序列化安全的,并把管线要解析的体积压缩约 70%(以该坏例页计)。
|
|
40
|
+
|
|
39
41
|
## 环境要求
|
|
40
42
|
|
|
41
43
|
- DSH web profile(`dsh web`),Node.js ≥ 20。
|
package/lib/client.js
CHANGED
|
@@ -419,8 +419,8 @@ window.__ModuleLoader__.load({
|
|
|
419
419
|
"save": "UM8-Rq_save",
|
|
420
420
|
"readOnly": "UM8-Rq_readOnly",
|
|
421
421
|
"form": "UM8-Rq_form",
|
|
422
|
-
"
|
|
423
|
-
"
|
|
422
|
+
"footer": "UM8-Rq_footer",
|
|
423
|
+
"failed": "UM8-Rq_failed"
|
|
424
424
|
};
|
|
425
425
|
//#endregion
|
|
426
426
|
//#region src/client/PluginCard.tsx
|
|
@@ -480,27 +480,27 @@ window.__ModuleLoader__.load({
|
|
|
480
480
|
document.head.appendChild(tag);
|
|
481
481
|
}
|
|
482
482
|
var fields_module_css_default = {
|
|
483
|
-
"head": "CFSaVG_head",
|
|
484
483
|
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
485
|
-
"radioLabel": "CFSaVG_radioLabel",
|
|
486
|
-
"radioOption": "CFSaVG_radioOption",
|
|
487
|
-
"input": "CFSaVG_input",
|
|
488
|
-
"field": "CFSaVG_field",
|
|
489
|
-
"reset": "CFSaVG_reset",
|
|
490
484
|
"badge": "CFSaVG_badge",
|
|
491
|
-
"
|
|
492
|
-
"
|
|
493
|
-
"
|
|
485
|
+
"field": "CFSaVG_field",
|
|
486
|
+
"radioText": "CFSaVG_radioText",
|
|
487
|
+
"radioHint": "CFSaVG_radioHint",
|
|
494
488
|
"radioContent": "CFSaVG_radioContent",
|
|
495
|
-
"
|
|
496
|
-
"
|
|
497
|
-
"
|
|
489
|
+
"head": "CFSaVG_head",
|
|
490
|
+
"radioLabel": "CFSaVG_radioLabel",
|
|
491
|
+
"badges": "CFSaVG_badges",
|
|
498
492
|
"radioOptionPlain": "CFSaVG_radioOptionPlain",
|
|
493
|
+
"invalid": "CFSaVG_invalid",
|
|
494
|
+
"reset": "CFSaVG_reset",
|
|
499
495
|
"checkboxRow": "CFSaVG_checkboxRow",
|
|
500
496
|
"radioGroup": "CFSaVG_radioGroup",
|
|
501
|
-
"
|
|
502
|
-
"
|
|
503
|
-
"
|
|
497
|
+
"radioPick": "CFSaVG_radioPick",
|
|
498
|
+
"radioOption": "CFSaVG_radioOption",
|
|
499
|
+
"label": "CFSaVG_label",
|
|
500
|
+
"input": "CFSaVG_input",
|
|
501
|
+
"hint": "CFSaVG_hint",
|
|
502
|
+
"hintIndented": "CFSaVG_hintIndented",
|
|
503
|
+
"inputInvalid": "CFSaVG_inputInvalid"
|
|
504
504
|
};
|
|
505
505
|
//#endregion
|
|
506
506
|
//#region src/client/fields.tsx
|
package/lib/index.js
CHANGED
|
@@ -161,14 +161,22 @@ function normalizeCdpEndpoint(input) {
|
|
|
161
161
|
* 2. Content-level — the interstitial's localized `<title>` family
|
|
162
162
|
* ("Just a moment...", "请稍候…", ...) plus structural markers
|
|
163
163
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
164
|
-
* `
|
|
165
|
-
*
|
|
166
|
-
*
|
|
167
|
-
*
|
|
168
|
-
*
|
|
169
|
-
*
|
|
170
|
-
*
|
|
171
|
-
*
|
|
164
|
+
* `window._cf_chl_opt`). Markers are deliberately structural: a blog
|
|
165
|
+
* post that merely MENTIONS Cloudflare must not classify as a challenge,
|
|
166
|
+
* so the challenge-platform script check reads `<script src>` attributes
|
|
167
|
+
* only (never prose or escaped code samples) and identifies the runtime
|
|
168
|
+
* POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
|
|
169
|
+
* instrumentation Cloudflare injects into every NORMAL page of a
|
|
170
|
+
* protected zone, on several moving mounts, is not a challenge signal.
|
|
171
|
+
* The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
|
|
172
|
+
* marker: sites embed real Turnstile widgets into normal pages
|
|
173
|
+
* (linux.do — a Discourse forum with the Turnstile plugin mounts one on
|
|
174
|
+
* every topic page), so the widget alone cannot distinguish an
|
|
175
|
+
* interstitial from a normal page behind a Cloudflare edge. Every real
|
|
176
|
+
* interstitial carries at least one independent signal beyond its
|
|
177
|
+
* widget — the localized title, the orchestrate runtime script,
|
|
178
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
|
|
179
|
+
* stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
|
|
172
180
|
*
|
|
173
181
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
174
182
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -314,9 +322,21 @@ function isChallengeRuntimeScript(src) {
|
|
|
314
322
|
* The challenge-platform prefix is deliberately absent: script evidence goes
|
|
315
323
|
* through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
|
|
316
324
|
* distinguish a real `src` from prose and runtime from instrumentation.
|
|
325
|
+
*
|
|
326
|
+
* The Turnstile widget id (`id="cf-chl-widget-…"`) is deliberately ABSENT
|
|
327
|
+
* (badcase: linux.do/t/topic/3143588 — a Discourse forum behind Cloudflare
|
|
328
|
+
* whose Turnstile plugin mounts a real widget, id and all, into every normal
|
|
329
|
+
* topic page it serves at HTTP 200). The widget is minted by the same
|
|
330
|
+
* Turnstile runtime whether it renders inside an interstitial or inside a
|
|
331
|
+
* site's own UI, so its presence proves only "this page embeds Turnstile".
|
|
332
|
+
* On a real interstitial the widget always co-occurs with independent proof
|
|
333
|
+
* — the localized title, the orchestrate runtime script,
|
|
334
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding, each sufficient
|
|
335
|
+
* above — so dropping the widget loses no real interstitial while every
|
|
336
|
+
* widget-embedding normal page (linux.do, login portals, contact forms)
|
|
337
|
+
* stops dying as WEB_FETCH_CHALLENGE after a wait that can never clear.
|
|
317
338
|
*/
|
|
318
339
|
const CHALLENGE_HTML_MARKERS = [
|
|
319
|
-
"id=\"cf-chl-widget-",
|
|
320
340
|
"window._cf_chl_opt",
|
|
321
341
|
"id=\"challenge-form\"",
|
|
322
342
|
"id=\"challenge-running\"",
|
|
@@ -410,7 +430,13 @@ function classifyChallengeHtml(html) {
|
|
|
410
430
|
* check uses, so the two can never drift apart. Script evidence follows the
|
|
411
431
|
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
412
432
|
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
413
|
-
* protected zone is ignored.
|
|
433
|
+
* protected zone is ignored. The Turnstile widget selector is likewise
|
|
434
|
+
* absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
|
|
435
|
+
* widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
|
|
436
|
+
* the probe reporting "still challenged" forever, so a challenge that HAD
|
|
437
|
+
* cleared could never be confirmed — the wait burned its whole budget and
|
|
438
|
+
* the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
|
|
439
|
+
* layer's timeout.
|
|
414
440
|
*/
|
|
415
441
|
const CHALLENGE_DOM_PROBE = `(() => {
|
|
416
442
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
@@ -418,7 +444,6 @@ const CHALLENGE_DOM_PROBE = `(() => {
|
|
|
418
444
|
const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
|
|
419
445
|
if (re.test(document.title.trim())) return true
|
|
420
446
|
if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
|
|
421
|
-
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
422
447
|
for (const script of Array.from(document.scripts)) {
|
|
423
448
|
const src = (script.getAttribute('src') || '').toLowerCase()
|
|
424
449
|
if (src.includes(platform) && src.includes(runtime)) return true
|
|
@@ -1064,11 +1089,14 @@ function installBuiltinSlashResolveCompat() {
|
|
|
1064
1089
|
* inline `data:` image payloads are elided to size placeholders (build
|
|
1065
1090
|
* tools inline images as base64, which would otherwise dominate the body);
|
|
1066
1091
|
* Mozilla Readability extracts the article (dropping nav bars, sidebars,
|
|
1067
|
-
* footers, and ad chrome by scoring link density and text mass)
|
|
1068
|
-
*
|
|
1069
|
-
*
|
|
1070
|
-
*
|
|
1071
|
-
*
|
|
1092
|
+
* footers, and ad chrome by scoring link density and text mass) — its result
|
|
1093
|
+
* must then pass a coverage gate that distrusts tiny or sliver-thin
|
|
1094
|
+
* extractions, which on list-shaped pages (portals, catalogs) is exactly
|
|
1095
|
+
* what the link-density penalty leaves behind; DOMPurify sanitizes whatever
|
|
1096
|
+
* HTML remains and forbids the layout tags noise lives in; Turndown with
|
|
1097
|
+
* the GFM plugin converts to markdown using the same style options and
|
|
1098
|
+
* span-safe table rules as the shipped `dsh-tool-web` renderer, so output
|
|
1099
|
+
* is consistent with what `web_fetch` produces elsewhere.
|
|
1072
1100
|
*
|
|
1073
1101
|
* Synchronous for its callers and unit-tested against fixture pages: jsdom
|
|
1074
1102
|
* itself is loaded lazily on the first call — and, inside a dsh profile, only
|
|
@@ -1167,6 +1195,99 @@ function humanSize(bytes) {
|
|
|
1167
1195
|
if (bytes < 1048576) return `${(bytes / 1024).toFixed(1)}KB`;
|
|
1168
1196
|
return `${(bytes / 1048576).toFixed(1)}MB`;
|
|
1169
1197
|
}
|
|
1198
|
+
/**
|
|
1199
|
+
* Tags whose subtrees are noise when measuring how much of a page an
|
|
1200
|
+
* extraction kept: DOMPurify drops these outright (`KEEP_CONTENT: false`),
|
|
1201
|
+
* and its default policy always removes `script`/`style` — so a text walk
|
|
1202
|
+
* that skips them measures what the sanitized document fallback could
|
|
1203
|
+
* actually return, not text that is already destined for deletion. Measuring
|
|
1204
|
+
* the raw body instead would let a huge `<nav>`/`<footer>` mass shrink an
|
|
1205
|
+
* article's honest coverage share and demote good extractions.
|
|
1206
|
+
*/
|
|
1207
|
+
const NOISE_SUBTREE_TAGS = /* @__PURE__ */ new Set([
|
|
1208
|
+
...FORBID_TAGS,
|
|
1209
|
+
"script",
|
|
1210
|
+
"style"
|
|
1211
|
+
]);
|
|
1212
|
+
/**
|
|
1213
|
+
* A page must have at least this much visible (post-noise) text before the
|
|
1214
|
+
* article acceptance gate below has an opinion. Sparse pages — logins,
|
|
1215
|
+
* stubs, `login-like` fixtures — have nowhere for content to hide, so
|
|
1216
|
+
* Readability's pick, however short, is still the best reading of the page
|
|
1217
|
+
* and keeps flowing through the article path exactly as before.
|
|
1218
|
+
*/
|
|
1219
|
+
const GATE_MIN_BODY_TEXT_CHARS = 1e3;
|
|
1220
|
+
/**
|
|
1221
|
+
* Minimum non-whitespace characters a Readability extraction must carry
|
|
1222
|
+
* before it can be trusted as a page's content. Measured on the 163.com
|
|
1223
|
+
* homepage, the scorer's pick was a 96-character footer licensing block —
|
|
1224
|
+
* real articles and even sparse pages clear two-to-three times this bar,
|
|
1225
|
+
* while link-dense portal pages lose whole news columns to it.
|
|
1226
|
+
*/
|
|
1227
|
+
const GATE_MIN_ARTICLE_TEXT_CHARS = 250;
|
|
1228
|
+
/**
|
|
1229
|
+
* Minimum share of a page's visible text an extraction must cover to be
|
|
1230
|
+
* trusted. Measured separation on real pages is wide: portal homepages
|
|
1231
|
+
* (163 0.01, sina 0.03, sohu 0.02) vs article pages (0.28–0.45), so 10%
|
|
1232
|
+
* sits in an empty band with margin on both sides.
|
|
1233
|
+
*/
|
|
1234
|
+
const GATE_MIN_ARTICLE_COVERAGE = .1;
|
|
1235
|
+
/**
|
|
1236
|
+
* Count non-whitespace characters under `root`, skipping noise subtrees
|
|
1237
|
+
* ({@link NOISE_SUBTREE_TAGS}). Whitespace is ignored so CJK pages (no
|
|
1238
|
+
* spaces) and latin pages (space-separated) measure comparable densities,
|
|
1239
|
+
* mirroring how the extraction-quality ratios in the docs were computed.
|
|
1240
|
+
*
|
|
1241
|
+
* @param root - the subtree to measure; `null` counts as nothing.
|
|
1242
|
+
* @returns the number of non-whitespace characters in kept subtrees.
|
|
1243
|
+
*/
|
|
1244
|
+
function visibleTextLength(root) {
|
|
1245
|
+
if (root === null) return 0;
|
|
1246
|
+
let length = 0;
|
|
1247
|
+
const visit = (node) => {
|
|
1248
|
+
if (node.nodeType === 3) {
|
|
1249
|
+
length += (node.nodeValue ?? "").replace(/\s+/g, "").length;
|
|
1250
|
+
return;
|
|
1251
|
+
}
|
|
1252
|
+
if (node.nodeType !== 1 || NOISE_SUBTREE_TAGS.has(node.nodeName.toLowerCase())) return;
|
|
1253
|
+
for (let child = node.firstChild; child !== null; child = child.nextSibling) visit(child);
|
|
1254
|
+
};
|
|
1255
|
+
visit(root);
|
|
1256
|
+
return length;
|
|
1257
|
+
}
|
|
1258
|
+
/**
|
|
1259
|
+
* Whether a Readability extraction is trustworthy as *the* content of the
|
|
1260
|
+
* page it came from — the acceptance gate that keeps list-shaped pages whole.
|
|
1261
|
+
*
|
|
1262
|
+
* Readability is an article extractor: it scores text mass and penalizes
|
|
1263
|
+
* link density, so on a portal/catalog page — where the content *is* the
|
|
1264
|
+
* links — every news column scores near zero, some sparse low-link block (a
|
|
1265
|
+
* footer licensing notice, a province-switcher popup) wins by default, and
|
|
1266
|
+
* Readability's own 500-char quality check cannot fail the parse because its
|
|
1267
|
+
* retry loop eventually returns the longest of its bad attempts. The plugin
|
|
1268
|
+
* used to accept any non-null result, so the 163.com homepage rendered as a
|
|
1269
|
+
* 162-character licensing fragment while 320 headlines were dropped — a
|
|
1270
|
+
* recall loss far worse than any precision the gate could cost.
|
|
1271
|
+
*
|
|
1272
|
+
* The gate therefore distrusts an extraction that is tiny in absolute terms
|
|
1273
|
+
* or covers a sliver of the page's visible text; the pipeline then falls
|
|
1274
|
+
* back to the sanitized whole document, which DOMPurify keeps clean and
|
|
1275
|
+
* which on such pages retains effectively all list content (measured 0%
|
|
1276
|
+
* text loss on the 163.com homepage). Only text-rich pages are gated
|
|
1277
|
+
* ({@link GATE_MIN_BODY_TEXT_CHARS}); below that, behavior is unchanged.
|
|
1278
|
+
*
|
|
1279
|
+
* @param articleHtml - the `article.content` HTML Readability produced.
|
|
1280
|
+
* @param document - the live document the article was extracted from.
|
|
1281
|
+
* @returns whether the article should flow through the article path.
|
|
1282
|
+
*/
|
|
1283
|
+
function articleCoversPage(articleHtml, document) {
|
|
1284
|
+
const bodyText = visibleTextLength(document.body);
|
|
1285
|
+
if (bodyText < GATE_MIN_BODY_TEXT_CHARS) return true;
|
|
1286
|
+
const container = document.createElement("div");
|
|
1287
|
+
container.innerHTML = articleHtml;
|
|
1288
|
+
const articleText = visibleTextLength(container);
|
|
1289
|
+
return articleText >= GATE_MIN_ARTICLE_TEXT_CHARS && articleText / bodyText >= GATE_MIN_ARTICLE_COVERAGE;
|
|
1290
|
+
}
|
|
1170
1291
|
/** The shared converter: same style options as `dsh-tool-web`'s renderer. */
|
|
1171
1292
|
const turndown = new TurndownService({
|
|
1172
1293
|
headingStyle: "atx",
|
|
@@ -1217,10 +1338,11 @@ turndown.addRule("tableRowWithoutSpanExpansion", {
|
|
|
1217
1338
|
/**
|
|
1218
1339
|
* Convert one rendered HTML document to denoised markdown.
|
|
1219
1340
|
*
|
|
1220
|
-
* Readability failure (non-article pages
|
|
1221
|
-
*
|
|
1222
|
-
*
|
|
1223
|
-
*
|
|
1341
|
+
* Readability failure (non-article pages, or extractions the coverage gate
|
|
1342
|
+
* {@link articleCoversPage} distrusts) degrades to converting the sanitized
|
|
1343
|
+
* whole document — layout tags are still forbidden, so the fallback stays
|
|
1344
|
+
* cleaner than raw turndown, and a degraded page beats an error for a body
|
|
1345
|
+
* the browser already rendered.
|
|
1224
1346
|
*
|
|
1225
1347
|
* @param html - the rendered page HTML (`page.content()`).
|
|
1226
1348
|
* @param url - the page URL, used to resolve relative links during parsing.
|
|
@@ -1240,7 +1362,7 @@ function htmlToMarkdown(html, url) {
|
|
|
1240
1362
|
try {
|
|
1241
1363
|
const cloned = document.cloneNode(true);
|
|
1242
1364
|
const article = new Readability(cloned).parse();
|
|
1243
|
-
if (article !== null && typeof article.content === "string" && article.content !== "") {
|
|
1365
|
+
if (article !== null && typeof article.content === "string" && article.content !== "" && articleCoversPage(article.content, document)) {
|
|
1244
1366
|
source = article.content;
|
|
1245
1367
|
title = article.title ?? void 0;
|
|
1246
1368
|
}
|
|
@@ -1856,6 +1978,7 @@ var PlaywrightFetchProvider = class {
|
|
|
1856
1978
|
content: finalResponse !== null ? await finalResponse.text() : await page.content()
|
|
1857
1979
|
});
|
|
1858
1980
|
await page.waitForLoadState("networkidle", { timeout: Math.min(SETTLE_MS, deadline.remainingMs()) }).catch(() => {});
|
|
1981
|
+
await stripScriptingElements(page);
|
|
1859
1982
|
const html = await page.content();
|
|
1860
1983
|
if (!config.denoise) return capResult(finalUrl, statusCode, {
|
|
1861
1984
|
kind: "html",
|
|
@@ -2168,6 +2291,74 @@ async function probeStillOnChallenge(page) {
|
|
|
2168
2291
|
return true;
|
|
2169
2292
|
}
|
|
2170
2293
|
}
|
|
2294
|
+
/**
|
|
2295
|
+
* In-page script that removes non-content scripting subtrees from the live
|
|
2296
|
+
* document before `page.content()` reads it as an HTML string.
|
|
2297
|
+
*
|
|
2298
|
+
* WHAT goes, and why each removal is output-neutral versus the old pipeline:
|
|
2299
|
+
*
|
|
2300
|
+
* - `script` — three consumers ever saw script text before, and none loses
|
|
2301
|
+
* it here: Readability's `parse()` deletes script/noscript nodes BEFORE
|
|
2302
|
+
* extraction and scoring (so script text never reached the article or
|
|
2303
|
+
* influenced the scorer); DOMPurify's default allowlist drops script; the
|
|
2304
|
+
* tool layer's renderer removes SCRIPT wholesale on the raw path. What the
|
|
2305
|
+
* removal buys is the reason this exists: a page's runtime JS may rewrite
|
|
2306
|
+
* a data script's text with a literal `<\/script>` embedded — measured on
|
|
2307
|
+
* techcommunity.microsoft.com (Khoros/Aurora), whose client writes its
|
|
2308
|
+
* Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`,
|
|
2309
|
+
* turning the wire's 712 `\u003c` escapes back into raw `<` — and
|
|
2310
|
+
* browsers serialize script text verbatim, so the flattened string ends
|
|
2311
|
+
* that script at the FIRST embedded end tag and re-parses the remaining
|
|
2312
|
+
* ~270KB of JSON as markup. The phantom elements that survive (unknown
|
|
2313
|
+
* tags like `<id>`, anchors full of JSON text) are exactly what
|
|
2314
|
+
* Readability's scorer reads as article mass, so the extracted "article"
|
|
2315
|
+
* is CSS and i18n strings while the real body — still present in the DOM —
|
|
2316
|
+
* loses. No string-level patch can undo this after serialization (the
|
|
2317
|
+
* string is genuinely ambiguous); only the browser's own parse holds the
|
|
2318
|
+
* document's true structure.
|
|
2319
|
+
* - `script[type="application/ld+json"]` — KEPT, but defused: Readability
|
|
2320
|
+
* extracts metadata (title/byline/excerpt — e.g. the site-suffix-free
|
|
2321
|
+
* "Remote Desktop Protocol" instead of "… - Wikipedia") from JSON-LD
|
|
2322
|
+
* scripts before removing them, so deleting them would regress the title
|
|
2323
|
+
* on every JSON-LD-carrying page. Rewriting `</` as `<\/` keeps the text
|
|
2324
|
+
* valid JSON (`\/` is a legal escape, so the re-parsed text parses to the
|
|
2325
|
+
* same object) while making the serialization harmless: the tokenizer
|
|
2326
|
+
* only terminates on a literal `<\/script`.
|
|
2327
|
+
* - `noscript` — KEPT: Readability's `_unwrapNoscriptImages` salvages the
|
|
2328
|
+
* real image out of lazy-load `<noscript><img …>` wrappers before
|
|
2329
|
+
* `_removeScripts` deletes the noscript; pre-stripping it would silently
|
|
2330
|
+
* drop those images from the markdown.
|
|
2331
|
+
* - `template` — removed: its children parse into the inert `.content`
|
|
2332
|
+
* fragment (invisible to Readability's traversal), DOMPurify's
|
|
2333
|
+
* FORBID_TAGS drops it, and the tool layer's renderer removes it too.
|
|
2334
|
+
* - `style` stays — its text cannot terminate a re-parse except at its own
|
|
2335
|
+
* end tag, and the pipeline already drops it.
|
|
2336
|
+
*/
|
|
2337
|
+
const STRIP_SCRIPTING_ELEMENTS = `(() => {
|
|
2338
|
+
for (const el of Array.from(document.querySelectorAll('script, template'))) {
|
|
2339
|
+
if (el.tagName === 'SCRIPT' && (el.getAttribute('type') || '').trim().toLowerCase() === 'application/ld+json') {
|
|
2340
|
+
// '<\\/' is still valid JSON, so the re-parsed textContent parses to
|
|
2341
|
+
// the same object — while the serialized form can no longer close the
|
|
2342
|
+
// script element early.
|
|
2343
|
+
el.textContent = el.textContent.replace(/<\\//g, '<\\\\/')
|
|
2344
|
+
continue
|
|
2345
|
+
}
|
|
2346
|
+
el.remove()
|
|
2347
|
+
}
|
|
2348
|
+
})()`;
|
|
2349
|
+
/**
|
|
2350
|
+
* Remove non-content scripting subtrees from the live document before
|
|
2351
|
+
* serialization. Best-effort, mirroring {@link probeStillOnChallenge}: a
|
|
2352
|
+
* backend without `evaluate` (fakes, minimal engines) simply keeps the
|
|
2353
|
+
* document as-is — the fetch never fails here, and the pipeline degrades to
|
|
2354
|
+
* whatever the serialized string itself says.
|
|
2355
|
+
*/
|
|
2356
|
+
async function stripScriptingElements(page) {
|
|
2357
|
+
if (typeof page.evaluate !== "function") return;
|
|
2358
|
+
try {
|
|
2359
|
+
await page.evaluate(STRIP_SCRIPTING_ELEMENTS);
|
|
2360
|
+
} catch {}
|
|
2361
|
+
}
|
|
2171
2362
|
/** Apply the decoded-body cap and flag the cut. */
|
|
2172
2363
|
function capResult(url, statusCode, body) {
|
|
2173
2364
|
const truncated = body.content.length > MAX_BODY_CHARS;
|
package/lib/types/challenge.d.ts
CHANGED
|
@@ -14,14 +14,22 @@
|
|
|
14
14
|
* 2. Content-level — the interstitial's localized `<title>` family
|
|
15
15
|
* ("Just a moment...", "请稍候…", ...) plus structural markers
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
|
-
* `
|
|
18
|
-
*
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
17
|
+
* `window._cf_chl_opt`). Markers are deliberately structural: a blog
|
|
18
|
+
* post that merely MENTIONS Cloudflare must not classify as a challenge,
|
|
19
|
+
* so the challenge-platform script check reads `<script src>` attributes
|
|
20
|
+
* only (never prose or escaped code samples) and identifies the runtime
|
|
21
|
+
* POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
|
|
22
|
+
* instrumentation Cloudflare injects into every NORMAL page of a
|
|
23
|
+
* protected zone, on several moving mounts, is not a challenge signal.
|
|
24
|
+
* The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
|
|
25
|
+
* marker: sites embed real Turnstile widgets into normal pages
|
|
26
|
+
* (linux.do — a Discourse forum with the Turnstile plugin mounts one on
|
|
27
|
+
* every topic page), so the widget alone cannot distinguish an
|
|
28
|
+
* interstitial from a normal page behind a Cloudflare edge. Every real
|
|
29
|
+
* interstitial carries at least one independent signal beyond its
|
|
30
|
+
* widget — the localized title, the orchestrate runtime script,
|
|
31
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
|
|
32
|
+
* stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
|
|
25
33
|
*
|
|
26
34
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
27
35
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -86,6 +94,12 @@ export declare function classifyChallengeHtml(html: string): ChallengeVerdict;
|
|
|
86
94
|
* check uses, so the two can never drift apart. Script evidence follows the
|
|
87
95
|
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
88
96
|
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
89
|
-
* protected zone is ignored.
|
|
97
|
+
* protected zone is ignored. The Turnstile widget selector is likewise
|
|
98
|
+
* absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
|
|
99
|
+
* widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
|
|
100
|
+
* the probe reporting "still challenged" forever, so a challenge that HAD
|
|
101
|
+
* cleared could never be confirmed — the wait burned its whole budget and
|
|
102
|
+
* the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
|
|
103
|
+
* layer's timeout.
|
|
90
104
|
*/
|
|
91
105
|
export declare const CHALLENGE_DOM_PROBE: string;
|
package/lib/types/markdown.d.ts
CHANGED
|
@@ -5,11 +5,14 @@
|
|
|
5
5
|
* inline `data:` image payloads are elided to size placeholders (build
|
|
6
6
|
* tools inline images as base64, which would otherwise dominate the body);
|
|
7
7
|
* Mozilla Readability extracts the article (dropping nav bars, sidebars,
|
|
8
|
-
* footers, and ad chrome by scoring link density and text mass)
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
8
|
+
* footers, and ad chrome by scoring link density and text mass) — its result
|
|
9
|
+
* must then pass a coverage gate that distrusts tiny or sliver-thin
|
|
10
|
+
* extractions, which on list-shaped pages (portals, catalogs) is exactly
|
|
11
|
+
* what the link-density penalty leaves behind; DOMPurify sanitizes whatever
|
|
12
|
+
* HTML remains and forbids the layout tags noise lives in; Turndown with
|
|
13
|
+
* the GFM plugin converts to markdown using the same style options and
|
|
14
|
+
* span-safe table rules as the shipped `dsh-tool-web` renderer, so output
|
|
15
|
+
* is consistent with what `web_fetch` produces elsewhere.
|
|
13
16
|
*
|
|
14
17
|
* Synchronous for its callers and unit-tested against fixture pages: jsdom
|
|
15
18
|
* itself is loaded lazily on the first call — and, inside a dsh profile, only
|
|
@@ -24,16 +27,21 @@ export type DenoiseMode = 'article' | 'document';
|
|
|
24
27
|
export interface DenoiseResult {
|
|
25
28
|
/** The markdown body (title, when found, already prepended). */
|
|
26
29
|
markdown: string;
|
|
27
|
-
/**
|
|
30
|
+
/**
|
|
31
|
+
* `article` = Readability extraction that passed the coverage gate;
|
|
32
|
+
* `document` = whole-document fallback (Readability returned nothing, or
|
|
33
|
+
* the gate distrusted what it returned — list-shaped pages).
|
|
34
|
+
*/
|
|
28
35
|
mode: DenoiseMode;
|
|
29
36
|
}
|
|
30
37
|
/**
|
|
31
38
|
* Convert one rendered HTML document to denoised markdown.
|
|
32
39
|
*
|
|
33
|
-
* Readability failure (non-article pages
|
|
34
|
-
*
|
|
35
|
-
*
|
|
36
|
-
*
|
|
40
|
+
* Readability failure (non-article pages, or extractions the coverage gate
|
|
41
|
+
* {@link articleCoversPage} distrusts) degrades to converting the sanitized
|
|
42
|
+
* whole document — layout tags are still forbidden, so the fallback stays
|
|
43
|
+
* cleaner than raw turndown, and a degraded page beats an error for a body
|
|
44
|
+
* the browser already rendered.
|
|
37
45
|
*
|
|
38
46
|
* @param html - the rendered page HTML (`page.content()`).
|
|
39
47
|
* @param url - the page URL, used to resolve relative links during parsing.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-web-fetch-playwright",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.16",
|
|
4
4
|
"description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/src/challenge.ts
CHANGED
|
@@ -14,14 +14,22 @@
|
|
|
14
14
|
* 2. Content-level — the interstitial's localized `<title>` family
|
|
15
15
|
* ("Just a moment...", "请稍候…", ...) plus structural markers
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
|
-
* `
|
|
18
|
-
*
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
17
|
+
* `window._cf_chl_opt`). Markers are deliberately structural: a blog
|
|
18
|
+
* post that merely MENTIONS Cloudflare must not classify as a challenge,
|
|
19
|
+
* so the challenge-platform script check reads `<script src>` attributes
|
|
20
|
+
* only (never prose or escaped code samples) and identifies the runtime
|
|
21
|
+
* POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
|
|
22
|
+
* instrumentation Cloudflare injects into every NORMAL page of a
|
|
23
|
+
* protected zone, on several moving mounts, is not a challenge signal.
|
|
24
|
+
* The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
|
|
25
|
+
* marker: sites embed real Turnstile widgets into normal pages
|
|
26
|
+
* (linux.do — a Discourse forum with the Turnstile plugin mounts one on
|
|
27
|
+
* every topic page), so the widget alone cannot distinguish an
|
|
28
|
+
* interstitial from a normal page behind a Cloudflare edge. Every real
|
|
29
|
+
* interstitial carries at least one independent signal beyond its
|
|
30
|
+
* widget — the localized title, the orchestrate runtime script,
|
|
31
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
|
|
32
|
+
* stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
|
|
25
33
|
*
|
|
26
34
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
27
35
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -176,9 +184,21 @@ function isChallengeRuntimeScript(src: string): boolean {
|
|
|
176
184
|
* The challenge-platform prefix is deliberately absent: script evidence goes
|
|
177
185
|
* through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
|
|
178
186
|
* distinguish a real `src` from prose and runtime from instrumentation.
|
|
187
|
+
*
|
|
188
|
+
* The Turnstile widget id (`id="cf-chl-widget-…"`) is deliberately ABSENT
|
|
189
|
+
* (badcase: linux.do/t/topic/3143588 — a Discourse forum behind Cloudflare
|
|
190
|
+
* whose Turnstile plugin mounts a real widget, id and all, into every normal
|
|
191
|
+
* topic page it serves at HTTP 200). The widget is minted by the same
|
|
192
|
+
* Turnstile runtime whether it renders inside an interstitial or inside a
|
|
193
|
+
* site's own UI, so its presence proves only "this page embeds Turnstile".
|
|
194
|
+
* On a real interstitial the widget always co-occurs with independent proof
|
|
195
|
+
* — the localized title, the orchestrate runtime script,
|
|
196
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding, each sufficient
|
|
197
|
+
* above — so dropping the widget loses no real interstitial while every
|
|
198
|
+
* widget-embedding normal page (linux.do, login portals, contact forms)
|
|
199
|
+
* stops dying as WEB_FETCH_CHALLENGE after a wait that can never clear.
|
|
179
200
|
*/
|
|
180
201
|
const CHALLENGE_HTML_MARKERS = [
|
|
181
|
-
'id="cf-chl-widget-',
|
|
182
202
|
'window._cf_chl_opt',
|
|
183
203
|
'id="challenge-form"',
|
|
184
204
|
'id="challenge-running"',
|
|
@@ -295,7 +315,13 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
|
|
|
295
315
|
* check uses, so the two can never drift apart. Script evidence follows the
|
|
296
316
|
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
297
317
|
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
298
|
-
* protected zone is ignored.
|
|
318
|
+
* protected zone is ignored. The Turnstile widget selector is likewise
|
|
319
|
+
* absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
|
|
320
|
+
* widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
|
|
321
|
+
* the probe reporting "still challenged" forever, so a challenge that HAD
|
|
322
|
+
* cleared could never be confirmed — the wait burned its whole budget and
|
|
323
|
+
* the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
|
|
324
|
+
* layer's timeout.
|
|
299
325
|
*/
|
|
300
326
|
export const CHALLENGE_DOM_PROBE = `(() => {
|
|
301
327
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
@@ -303,7 +329,6 @@ export const CHALLENGE_DOM_PROBE = `(() => {
|
|
|
303
329
|
const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
|
|
304
330
|
if (re.test(document.title.trim())) return true
|
|
305
331
|
if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
|
|
306
|
-
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
307
332
|
for (const script of Array.from(document.scripts)) {
|
|
308
333
|
const src = (script.getAttribute('src') || '').toLowerCase()
|
|
309
334
|
if (src.includes(platform) && src.includes(runtime)) return true
|
package/src/markdown.ts
CHANGED
|
@@ -5,11 +5,14 @@
|
|
|
5
5
|
* inline `data:` image payloads are elided to size placeholders (build
|
|
6
6
|
* tools inline images as base64, which would otherwise dominate the body);
|
|
7
7
|
* Mozilla Readability extracts the article (dropping nav bars, sidebars,
|
|
8
|
-
* footers, and ad chrome by scoring link density and text mass)
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
8
|
+
* footers, and ad chrome by scoring link density and text mass) — its result
|
|
9
|
+
* must then pass a coverage gate that distrusts tiny or sliver-thin
|
|
10
|
+
* extractions, which on list-shaped pages (portals, catalogs) is exactly
|
|
11
|
+
* what the link-density penalty leaves behind; DOMPurify sanitizes whatever
|
|
12
|
+
* HTML remains and forbids the layout tags noise lives in; Turndown with
|
|
13
|
+
* the GFM plugin converts to markdown using the same style options and
|
|
14
|
+
* span-safe table rules as the shipped `dsh-tool-web` renderer, so output
|
|
15
|
+
* is consistent with what `web_fetch` produces elsewhere.
|
|
13
16
|
*
|
|
14
17
|
* Synchronous for its callers and unit-tested against fixture pages: jsdom
|
|
15
18
|
* itself is loaded lazily on the first call — and, inside a dsh profile, only
|
|
@@ -108,6 +111,105 @@ function humanSize(bytes: number): string {
|
|
|
108
111
|
return `${(bytes / 1_048_576).toFixed(1)}MB`
|
|
109
112
|
}
|
|
110
113
|
|
|
114
|
+
/**
|
|
115
|
+
* Tags whose subtrees are noise when measuring how much of a page an
|
|
116
|
+
* extraction kept: DOMPurify drops these outright (`KEEP_CONTENT: false`),
|
|
117
|
+
* and its default policy always removes `script`/`style` — so a text walk
|
|
118
|
+
* that skips them measures what the sanitized document fallback could
|
|
119
|
+
* actually return, not text that is already destined for deletion. Measuring
|
|
120
|
+
* the raw body instead would let a huge `<nav>`/`<footer>` mass shrink an
|
|
121
|
+
* article's honest coverage share and demote good extractions.
|
|
122
|
+
*/
|
|
123
|
+
const NOISE_SUBTREE_TAGS = new Set([...FORBID_TAGS, 'script', 'style'])
|
|
124
|
+
|
|
125
|
+
/**
|
|
126
|
+
* A page must have at least this much visible (post-noise) text before the
|
|
127
|
+
* article acceptance gate below has an opinion. Sparse pages — logins,
|
|
128
|
+
* stubs, `login-like` fixtures — have nowhere for content to hide, so
|
|
129
|
+
* Readability's pick, however short, is still the best reading of the page
|
|
130
|
+
* and keeps flowing through the article path exactly as before.
|
|
131
|
+
*/
|
|
132
|
+
const GATE_MIN_BODY_TEXT_CHARS = 1_000
|
|
133
|
+
|
|
134
|
+
/**
|
|
135
|
+
* Minimum non-whitespace characters a Readability extraction must carry
|
|
136
|
+
* before it can be trusted as a page's content. Measured on the 163.com
|
|
137
|
+
* homepage, the scorer's pick was a 96-character footer licensing block —
|
|
138
|
+
* real articles and even sparse pages clear two-to-three times this bar,
|
|
139
|
+
* while link-dense portal pages lose whole news columns to it.
|
|
140
|
+
*/
|
|
141
|
+
const GATE_MIN_ARTICLE_TEXT_CHARS = 250
|
|
142
|
+
|
|
143
|
+
/**
|
|
144
|
+
* Minimum share of a page's visible text an extraction must cover to be
|
|
145
|
+
* trusted. Measured separation on real pages is wide: portal homepages
|
|
146
|
+
* (163 0.01, sina 0.03, sohu 0.02) vs article pages (0.28–0.45), so 10%
|
|
147
|
+
* sits in an empty band with margin on both sides.
|
|
148
|
+
*/
|
|
149
|
+
const GATE_MIN_ARTICLE_COVERAGE = 0.1
|
|
150
|
+
|
|
151
|
+
/**
|
|
152
|
+
* Count non-whitespace characters under `root`, skipping noise subtrees
|
|
153
|
+
* ({@link NOISE_SUBTREE_TAGS}). Whitespace is ignored so CJK pages (no
|
|
154
|
+
* spaces) and latin pages (space-separated) measure comparable densities,
|
|
155
|
+
* mirroring how the extraction-quality ratios in the docs were computed.
|
|
156
|
+
*
|
|
157
|
+
* @param root - the subtree to measure; `null` counts as nothing.
|
|
158
|
+
* @returns the number of non-whitespace characters in kept subtrees.
|
|
159
|
+
*/
|
|
160
|
+
function visibleTextLength(root: Node | null): number {
|
|
161
|
+
if (root === null) return 0
|
|
162
|
+
let length = 0
|
|
163
|
+
const visit = (node: Node): void => {
|
|
164
|
+
// Text nodes contribute their non-whitespace characters (nodeType 3).
|
|
165
|
+
if (node.nodeType === 3) {
|
|
166
|
+
length += (node.nodeValue ?? '').replace(/\s+/g, '').length
|
|
167
|
+
return
|
|
168
|
+
}
|
|
169
|
+
// Only elements branch further (nodeType 1); comments and friends carry
|
|
170
|
+
// no text, and noise subtrees are skipped wholesale.
|
|
171
|
+
if (node.nodeType !== 1 || NOISE_SUBTREE_TAGS.has(node.nodeName.toLowerCase())) return
|
|
172
|
+
for (let child = node.firstChild; child !== null; child = child.nextSibling) visit(child)
|
|
173
|
+
}
|
|
174
|
+
visit(root)
|
|
175
|
+
return length
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
/**
|
|
179
|
+
* Whether a Readability extraction is trustworthy as *the* content of the
|
|
180
|
+
* page it came from — the acceptance gate that keeps list-shaped pages whole.
|
|
181
|
+
*
|
|
182
|
+
* Readability is an article extractor: it scores text mass and penalizes
|
|
183
|
+
* link density, so on a portal/catalog page — where the content *is* the
|
|
184
|
+
* links — every news column scores near zero, some sparse low-link block (a
|
|
185
|
+
* footer licensing notice, a province-switcher popup) wins by default, and
|
|
186
|
+
* Readability's own 500-char quality check cannot fail the parse because its
|
|
187
|
+
* retry loop eventually returns the longest of its bad attempts. The plugin
|
|
188
|
+
* used to accept any non-null result, so the 163.com homepage rendered as a
|
|
189
|
+
* 162-character licensing fragment while 320 headlines were dropped — a
|
|
190
|
+
* recall loss far worse than any precision the gate could cost.
|
|
191
|
+
*
|
|
192
|
+
* The gate therefore distrusts an extraction that is tiny in absolute terms
|
|
193
|
+
* or covers a sliver of the page's visible text; the pipeline then falls
|
|
194
|
+
* back to the sanitized whole document, which DOMPurify keeps clean and
|
|
195
|
+
* which on such pages retains effectively all list content (measured 0%
|
|
196
|
+
* text loss on the 163.com homepage). Only text-rich pages are gated
|
|
197
|
+
* ({@link GATE_MIN_BODY_TEXT_CHARS}); below that, behavior is unchanged.
|
|
198
|
+
*
|
|
199
|
+
* @param articleHtml - the `article.content` HTML Readability produced.
|
|
200
|
+
* @param document - the live document the article was extracted from.
|
|
201
|
+
* @returns whether the article should flow through the article path.
|
|
202
|
+
*/
|
|
203
|
+
function articleCoversPage(articleHtml: string, document: Document): boolean {
|
|
204
|
+
const bodyText = visibleTextLength(document.body)
|
|
205
|
+
if (bodyText < GATE_MIN_BODY_TEXT_CHARS) return true
|
|
206
|
+
const container = document.createElement('div')
|
|
207
|
+
container.innerHTML = articleHtml
|
|
208
|
+
const articleText = visibleTextLength(container)
|
|
209
|
+
return articleText >= GATE_MIN_ARTICLE_TEXT_CHARS
|
|
210
|
+
&& articleText / bodyText >= GATE_MIN_ARTICLE_COVERAGE
|
|
211
|
+
}
|
|
212
|
+
|
|
111
213
|
/** The shared converter: same style options as `dsh-tool-web`'s renderer. */
|
|
112
214
|
const turndown = new TurndownService({
|
|
113
215
|
headingStyle: 'atx',
|
|
@@ -170,17 +272,22 @@ export type DenoiseMode = 'article' | 'document'
|
|
|
170
272
|
export interface DenoiseResult {
|
|
171
273
|
/** The markdown body (title, when found, already prepended). */
|
|
172
274
|
markdown: string
|
|
173
|
-
/**
|
|
275
|
+
/**
|
|
276
|
+
* `article` = Readability extraction that passed the coverage gate;
|
|
277
|
+
* `document` = whole-document fallback (Readability returned nothing, or
|
|
278
|
+
* the gate distrusted what it returned — list-shaped pages).
|
|
279
|
+
*/
|
|
174
280
|
mode: DenoiseMode
|
|
175
281
|
}
|
|
176
282
|
|
|
177
283
|
/**
|
|
178
284
|
* Convert one rendered HTML document to denoised markdown.
|
|
179
285
|
*
|
|
180
|
-
* Readability failure (non-article pages
|
|
181
|
-
*
|
|
182
|
-
*
|
|
183
|
-
*
|
|
286
|
+
* Readability failure (non-article pages, or extractions the coverage gate
|
|
287
|
+
* {@link articleCoversPage} distrusts) degrades to converting the sanitized
|
|
288
|
+
* whole document — layout tags are still forbidden, so the fallback stays
|
|
289
|
+
* cleaner than raw turndown, and a degraded page beats an error for a body
|
|
290
|
+
* the browser already rendered.
|
|
184
291
|
*
|
|
185
292
|
* @param html - the rendered page HTML (`page.content()`).
|
|
186
293
|
* @param url - the page URL, used to resolve relative links during parsing.
|
|
@@ -204,14 +311,18 @@ export function htmlToMarkdown(html: string, url: string): DenoiseResult {
|
|
|
204
311
|
let source: string | null = null
|
|
205
312
|
let title: string | undefined
|
|
206
313
|
try {
|
|
207
|
-
// Ungated: isProbablyReaderable is a conservative hint that
|
|
208
|
-
// sparse-but-real articles (measured on a browser-rendered fixture),
|
|
209
|
-
// the extraction is simply attempted; a null
|
|
210
|
-
// to the sanitized whole document below. parse() mutates,
|
|
211
|
-
// (typed as Document: DOM lib types cloneNode's return as
|
|
314
|
+
// Ungated readability: isProbablyReaderable is a conservative hint that
|
|
315
|
+
// rejects sparse-but-real articles (measured on a browser-rendered fixture),
|
|
316
|
+
// so the extraction is simply attempted; a null, empty, or gate-rejected
|
|
317
|
+
// result falls back to the sanitized whole document below. parse() mutates,
|
|
318
|
+
// hence the clone (typed as Document: DOM lib types cloneNode's return as
|
|
319
|
+
// Node).
|
|
212
320
|
const cloned = document.cloneNode(true) as typeof document
|
|
213
321
|
const article = new Readability(cloned).parse()
|
|
214
|
-
if (
|
|
322
|
+
if (
|
|
323
|
+
article !== null && typeof article.content === 'string' && article.content !== ''
|
|
324
|
+
&& articleCoversPage(article.content, document)
|
|
325
|
+
) {
|
|
215
326
|
source = article.content
|
|
216
327
|
title = article.title ?? undefined
|
|
217
328
|
}
|
package/src/provider.ts
CHANGED
|
@@ -535,6 +535,13 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
535
535
|
// what domcontentloaded already produced.
|
|
536
536
|
await page.waitForLoadState('networkidle', { timeout: Math.min(SETTLE_MS, deadline.remainingMs()) }).catch(() => {})
|
|
537
537
|
|
|
538
|
+
// Serialization safety: the document is about to be flattened into an
|
|
539
|
+
// HTML string and re-parsed by jsdom, and scripting elements are the one
|
|
540
|
+
// place that round-trip can break — see {@link STRIP_SCRIPTING_ELEMENTS}
|
|
541
|
+
// for exactly what is removed, what is kept (JSON-LD, noscript), and why
|
|
542
|
+
// each removal cannot change what the pipeline extracts.
|
|
543
|
+
await stripScriptingElements(page)
|
|
544
|
+
|
|
538
545
|
const html = await page.content()
|
|
539
546
|
if (!config.denoise) {
|
|
540
547
|
// The tool layer's own turndown renders raw HTML; the checkbox only
|
|
@@ -900,6 +907,78 @@ async function probeStillOnChallenge(page: PlaywrightPage): Promise<boolean> {
|
|
|
900
907
|
}
|
|
901
908
|
}
|
|
902
909
|
|
|
910
|
+
/**
|
|
911
|
+
* In-page script that removes non-content scripting subtrees from the live
|
|
912
|
+
* document before `page.content()` reads it as an HTML string.
|
|
913
|
+
*
|
|
914
|
+
* WHAT goes, and why each removal is output-neutral versus the old pipeline:
|
|
915
|
+
*
|
|
916
|
+
* - `script` — three consumers ever saw script text before, and none loses
|
|
917
|
+
* it here: Readability's `parse()` deletes script/noscript nodes BEFORE
|
|
918
|
+
* extraction and scoring (so script text never reached the article or
|
|
919
|
+
* influenced the scorer); DOMPurify's default allowlist drops script; the
|
|
920
|
+
* tool layer's renderer removes SCRIPT wholesale on the raw path. What the
|
|
921
|
+
* removal buys is the reason this exists: a page's runtime JS may rewrite
|
|
922
|
+
* a data script's text with a literal `</script>` embedded — measured on
|
|
923
|
+
* techcommunity.microsoft.com (Khoros/Aurora), whose client writes its
|
|
924
|
+
* Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`,
|
|
925
|
+
* turning the wire's 712 `\u003c` escapes back into raw `<` — and
|
|
926
|
+
* browsers serialize script text verbatim, so the flattened string ends
|
|
927
|
+
* that script at the FIRST embedded end tag and re-parses the remaining
|
|
928
|
+
* ~270KB of JSON as markup. The phantom elements that survive (unknown
|
|
929
|
+
* tags like `<id>`, anchors full of JSON text) are exactly what
|
|
930
|
+
* Readability's scorer reads as article mass, so the extracted "article"
|
|
931
|
+
* is CSS and i18n strings while the real body — still present in the DOM —
|
|
932
|
+
* loses. No string-level patch can undo this after serialization (the
|
|
933
|
+
* string is genuinely ambiguous); only the browser's own parse holds the
|
|
934
|
+
* document's true structure.
|
|
935
|
+
* - `script[type="application/ld+json"]` — KEPT, but defused: Readability
|
|
936
|
+
* extracts metadata (title/byline/excerpt — e.g. the site-suffix-free
|
|
937
|
+
* "Remote Desktop Protocol" instead of "… - Wikipedia") from JSON-LD
|
|
938
|
+
* scripts before removing them, so deleting them would regress the title
|
|
939
|
+
* on every JSON-LD-carrying page. Rewriting `</` as `<\/` keeps the text
|
|
940
|
+
* valid JSON (`\/` is a legal escape, so the re-parsed text parses to the
|
|
941
|
+
* same object) while making the serialization harmless: the tokenizer
|
|
942
|
+
* only terminates on a literal `</script`.
|
|
943
|
+
* - `noscript` — KEPT: Readability's `_unwrapNoscriptImages` salvages the
|
|
944
|
+
* real image out of lazy-load `<noscript><img …>` wrappers before
|
|
945
|
+
* `_removeScripts` deletes the noscript; pre-stripping it would silently
|
|
946
|
+
* drop those images from the markdown.
|
|
947
|
+
* - `template` — removed: its children parse into the inert `.content`
|
|
948
|
+
* fragment (invisible to Readability's traversal), DOMPurify's
|
|
949
|
+
* FORBID_TAGS drops it, and the tool layer's renderer removes it too.
|
|
950
|
+
* - `style` stays — its text cannot terminate a re-parse except at its own
|
|
951
|
+
* end tag, and the pipeline already drops it.
|
|
952
|
+
*/
|
|
953
|
+
const STRIP_SCRIPTING_ELEMENTS = `(() => {
|
|
954
|
+
for (const el of Array.from(document.querySelectorAll('script, template'))) {
|
|
955
|
+
if (el.tagName === 'SCRIPT' && (el.getAttribute('type') || '').trim().toLowerCase() === 'application/ld+json') {
|
|
956
|
+
// '<\\/' is still valid JSON, so the re-parsed textContent parses to
|
|
957
|
+
// the same object — while the serialized form can no longer close the
|
|
958
|
+
// script element early.
|
|
959
|
+
el.textContent = el.textContent.replace(/<\\//g, '<\\\\/')
|
|
960
|
+
continue
|
|
961
|
+
}
|
|
962
|
+
el.remove()
|
|
963
|
+
}
|
|
964
|
+
})()`
|
|
965
|
+
|
|
966
|
+
/**
|
|
967
|
+
* Remove non-content scripting subtrees from the live document before
|
|
968
|
+
* serialization. Best-effort, mirroring {@link probeStillOnChallenge}: a
|
|
969
|
+
* backend without `evaluate` (fakes, minimal engines) simply keeps the
|
|
970
|
+
* document as-is — the fetch never fails here, and the pipeline degrades to
|
|
971
|
+
* whatever the serialized string itself says.
|
|
972
|
+
*/
|
|
973
|
+
async function stripScriptingElements(page: PlaywrightPage): Promise<void> {
|
|
974
|
+
if (typeof page.evaluate !== 'function') return
|
|
975
|
+
try {
|
|
976
|
+
await page.evaluate(STRIP_SCRIPTING_ELEMENTS)
|
|
977
|
+
} catch {
|
|
978
|
+
// keep going without the strip
|
|
979
|
+
}
|
|
980
|
+
}
|
|
981
|
+
|
|
903
982
|
/** Apply the decoded-body cap and flag the cut. */
|
|
904
983
|
function capResult(url: string, statusCode: number, body: { kind: 'html' | 'text'; content: string }): WebFetchResult {
|
|
905
984
|
const truncated = body.content.length > MAX_BODY_CHARS
|