dsh-web-fetch-playwright 0.2.14 → 0.2.16

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -9,7 +9,7 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
9
9
  ## Features
10
10
 
11
11
  - **Real browser rendering** — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
12
- - **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `![alt](data:image/png;base64,...8.9KB)` so they cannot flood the body.
12
+ - **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `![alt](data:image/png;base64,...8.9KB)` so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content *is* the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content.
13
13
  - **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
14
14
  - **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
15
15
  - **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
@@ -29,13 +29,15 @@ web_fetch (tool-web)
29
29
  └─ ctx.web.fetchProvider = playwright
30
30
  ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
31
31
  ├─ cdp: connectOverCDP(endpoint)
32
- ├─ page.goto → settle (networkidle, best-effort) → page.content()
33
- ├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
32
+ ├─ page.goto → settle (networkidle, best-effort) → strip scripting subtrees in the live DOM → page.content()
33
+ ├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → DOMPurify → Turndown(GFM)
34
34
  └─ Markdown (or raw HTML when denoise is off)
35
35
  ```
36
36
 
37
37
  **Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; undecodable downloads (a PDF, a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
38
38
 
39
+ **Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction.** The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses `page.content()` with jsdom, and a script's *text* is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal `</script>` embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`, un-escaping the wire's 712 `\u003c` sequences into raw `<` — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the *first* embedded end tag and parses the remaining ~270KB of JSON *as markup*. The phantom elements that survive (unknown tags like `<id>`, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and `CachedAsset` strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary `script` and `template` subtrees are stripped inside the live page (`page.evaluate`, best-effort) before `page.content()`. This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back **byte-identical on every page**. Two deliberate carve-outs keep even the indirect consumers intact: `script[type="application/ld+json"]` is *defused* rather than deleted (`</` rewritten to `<\/` — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and `noscript` is kept entirely because Readability's `_unwrapNoscriptImages` salvages the real image out of lazy-load `<noscript><img …>` wrappers. The strip also applies with `denoise` off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).
40
+
39
41
  ## Requirements
40
42
 
41
43
  - DSH web profile (`dsh web`), Node.js ≥ 20.
package/README.zh-CN.md CHANGED
@@ -9,7 +9,7 @@
9
9
  ## 特性
10
10
 
11
11
  - **真实浏览器渲染** —— 以用户视角加载页面,SPA 客户端渲染内容也能抓到,而非只有原始 HTML。
12
- - **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 `![alt](data:image/png;base64,...8.9KB)`,避免 base64 字符流刷屏。
12
+ - **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 `![alt](data:image/png;base64,...8.9KB)`,避免 base64 字符流刷屏。对列表型页面(门户首页、商品目录——正文本身就是链接),Readability 的链接密度打分会误选出噪声碎片(实测 163.com 首页:96 字符的页脚备案块胜出,320 条新闻标题得分趋零),因此抽取结果需通过**覆盖率置信门**——不足 250 字符、或占页面可见文本不足 10% 即不予采信,转而回退到整页消毒转换,此类页面上可基本完整保留列表内容。
13
13
  - **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
14
14
  - **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
15
15
  - **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
@@ -29,13 +29,15 @@ web_fetch (tool-web)
29
29
  └─ ctx.web.fetchProvider = playwright
30
30
  ├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
31
31
  ├─ cdp: connectOverCDP(endpoint)
32
- ├─ page.goto → 等待稳定(networkidle,尽力而为)→ page.content()
33
- ├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → DOMPurify → Turndown(GFM)
32
+ ├─ page.goto → 等待稳定(networkidle,尽力而为)→ 在活 DOM 中剥离脚本子树 → page.content()
33
+ ├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → DOMPurify → Turndown(GFM)
34
34
  └─ Markdown(关闭降噪时返回原始 HTML)
35
35
  ```
36
36
 
37
37
  **非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;不可解码的下载(PDF、ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;传输错误与取消也依旧按错误处理。
38
38
 
39
+ **回读文档前,先在活 DOM 中剥离非内容脚本子树 —— 构造上即输出中性。** 降噪管线(以及关闭降噪时工具层自己的渲染器)会用 jsdom 重新解析 `page.content()`,而脚本的*文本内容*是这条往返链路中唯一无法信任的东西:页面自身的运行时 JS 可能把内嵌 `</script>` 的文本写回某个数据脚本的 textContent —— 实测于 techcommunity.microsoft.com(Khoros/Aurora 平台),其客户端把整个 Apollo 缓存经 `JSON.stringify` 写回 `script#__NEXT_DATA__`,把线上响应里 712 处 `\u003c` 转义还原成了裸 `<` —— 而浏览器序列化脚本文本时原样输出,于是这份扁平化的 HTML 字符串会在*第一个*内嵌结束标签处终止脚本,把余下约 270KB 的 JSON *当作标记语言*解析。解析出的幽灵元素(`<id>` 之类的未知标签、塞满 CSS 与 i18n 字符串的锚点)恰好是 Readability 评分器眼里的"正文质量"(实测 126 分对真实正文的 45 分),于是抓取 RDP 10 AVC/H.264 一文时返回的是 CSS 规则和 `CachedAsset` 字符串而不是文章本体。事后再对字符串做补丁无法消除这种歧义 —— 只有浏览器自己的解析持有文档的真实结构 —— 因此普通的 `script` 与 `template` 子树在 `page.content()` 之前先在活页面里剥掉(`page.evaluate`,尽力而为)。这不会改变管线以往的任何输出:Readability 本就在抽取与评分前删除 script/noscript 节点,DOMPurify 的白名单本就丢弃 script,工具层渲染器也整体移除 SCRIPT/TEMPLATE —— 在真实页面语料(Wikipedia、MDN、Hacker News、坏例页 wire 版,加上 noscript/JSON-LD/template 边界用例)上的差分 A/B **每一页都字节级一致**。两个刻意的保留让间接消费方也不受损:`script[type="application/ld+json"]` 只做*消险*而非删除(`</` 改写为 `<\/` —— 仍是合法 JSON,Readability 的元数据读取解析到同一对象,文章标题保持无站点后缀的干净形态,如 "Remote Desktop Protocol" 而非 "… - Wikipedia");`noscript` 完整保留,因为 Readability 的 `_unwrapNoscriptImages` 会从懒加载的 `<noscript><img …>` 包裹中抢救出真实图片。该剥离在关闭降噪时同样生效,让返回给工具层渲染器的原始 HTML 也是序列化安全的,并把管线要解析的体积压缩约 70%(以该坏例页计)。
40
+
39
41
  ## 环境要求
40
42
 
41
43
  - DSH web profile(`dsh web`),Node.js ≥ 20。
package/lib/client.js CHANGED
@@ -419,8 +419,8 @@ window.__ModuleLoader__.load({
419
419
  "save": "UM8-Rq_save",
420
420
  "readOnly": "UM8-Rq_readOnly",
421
421
  "form": "UM8-Rq_form",
422
- "failed": "UM8-Rq_failed",
423
- "footer": "UM8-Rq_footer"
422
+ "footer": "UM8-Rq_footer",
423
+ "failed": "UM8-Rq_failed"
424
424
  };
425
425
  //#endregion
426
426
  //#region src/client/PluginCard.tsx
@@ -480,27 +480,27 @@ window.__ModuleLoader__.load({
480
480
  document.head.appendChild(tag);
481
481
  }
482
482
  var fields_module_css_default = {
483
- "head": "CFSaVG_head",
484
483
  "fieldEmbedded": "CFSaVG_fieldEmbedded",
485
- "radioLabel": "CFSaVG_radioLabel",
486
- "radioOption": "CFSaVG_radioOption",
487
- "input": "CFSaVG_input",
488
- "field": "CFSaVG_field",
489
- "reset": "CFSaVG_reset",
490
484
  "badge": "CFSaVG_badge",
491
- "invalid": "CFSaVG_invalid",
492
- "hint": "CFSaVG_hint",
493
- "radioPick": "CFSaVG_radioPick",
485
+ "field": "CFSaVG_field",
486
+ "radioText": "CFSaVG_radioText",
487
+ "radioHint": "CFSaVG_radioHint",
494
488
  "radioContent": "CFSaVG_radioContent",
495
- "label": "CFSaVG_label",
496
- "inputInvalid": "CFSaVG_inputInvalid",
497
- "hintIndented": "CFSaVG_hintIndented",
489
+ "head": "CFSaVG_head",
490
+ "radioLabel": "CFSaVG_radioLabel",
491
+ "badges": "CFSaVG_badges",
498
492
  "radioOptionPlain": "CFSaVG_radioOptionPlain",
493
+ "invalid": "CFSaVG_invalid",
494
+ "reset": "CFSaVG_reset",
499
495
  "checkboxRow": "CFSaVG_checkboxRow",
500
496
  "radioGroup": "CFSaVG_radioGroup",
501
- "badges": "CFSaVG_badges",
502
- "radioText": "CFSaVG_radioText",
503
- "radioHint": "CFSaVG_radioHint"
497
+ "radioPick": "CFSaVG_radioPick",
498
+ "radioOption": "CFSaVG_radioOption",
499
+ "label": "CFSaVG_label",
500
+ "input": "CFSaVG_input",
501
+ "hint": "CFSaVG_hint",
502
+ "hintIndented": "CFSaVG_hintIndented",
503
+ "inputInvalid": "CFSaVG_inputInvalid"
504
504
  };
505
505
  //#endregion
506
506
  //#region src/client/fields.tsx
package/lib/index.js CHANGED
@@ -161,14 +161,22 @@ function normalizeCdpEndpoint(input) {
161
161
  * 2. Content-level — the interstitial's localized `<title>` family
162
162
  * ("Just a moment...", "请稍候…", ...) plus structural markers
163
163
  * (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
164
- * `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
165
- * deliberately structural: a blog post that merely MENTIONS Cloudflare
166
- * must not classify as a challenge, so the challenge-platform script
167
- * check reads `<script src>` attributes only (never prose or escaped
168
- * code samples) and identifies the runtime POSITIVELY by its
169
- * `/orchestrate/` mount — the JSD/Precursor instrumentation Cloudflare
170
- * injects into every NORMAL page of a protected zone, on several moving
171
- * mounts, is not a challenge signal.
164
+ * `window._cf_chl_opt`). Markers are deliberately structural: a blog
165
+ * post that merely MENTIONS Cloudflare must not classify as a challenge,
166
+ * so the challenge-platform script check reads `<script src>` attributes
167
+ * only (never prose or escaped code samples) and identifies the runtime
168
+ * POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
169
+ * instrumentation Cloudflare injects into every NORMAL page of a
170
+ * protected zone, on several moving mounts, is not a challenge signal.
171
+ * The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
172
+ * marker: sites embed real Turnstile widgets into normal pages
173
+ * (linux.do — a Discourse forum with the Turnstile plugin mounts one on
174
+ * every topic page), so the widget alone cannot distinguish an
175
+ * interstitial from a normal page behind a Cloudflare edge. Every real
176
+ * interstitial carries at least one independent signal beyond its
177
+ * widget — the localized title, the orchestrate runtime script,
178
+ * `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
179
+ * stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
172
180
  *
173
181
  * The security stance (issue #2): this module only DETECTS and classifies.
174
182
  * Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
@@ -314,9 +322,21 @@ function isChallengeRuntimeScript(src) {
314
322
  * The challenge-platform prefix is deliberately absent: script evidence goes
315
323
  * through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
316
324
  * distinguish a real `src` from prose and runtime from instrumentation.
325
+ *
326
+ * The Turnstile widget id (`id="cf-chl-widget-…"`) is deliberately ABSENT
327
+ * (badcase: linux.do/t/topic/3143588 — a Discourse forum behind Cloudflare
328
+ * whose Turnstile plugin mounts a real widget, id and all, into every normal
329
+ * topic page it serves at HTTP 200). The widget is minted by the same
330
+ * Turnstile runtime whether it renders inside an interstitial or inside a
331
+ * site's own UI, so its presence proves only "this page embeds Turnstile".
332
+ * On a real interstitial the widget always co-occurs with independent proof
333
+ * — the localized title, the orchestrate runtime script,
334
+ * `window._cf_chl_opt`, or the `#challenge-*` scaffolding, each sufficient
335
+ * above — so dropping the widget loses no real interstitial while every
336
+ * widget-embedding normal page (linux.do, login portals, contact forms)
337
+ * stops dying as WEB_FETCH_CHALLENGE after a wait that can never clear.
317
338
  */
318
339
  const CHALLENGE_HTML_MARKERS = [
319
- "id=\"cf-chl-widget-",
320
340
  "window._cf_chl_opt",
321
341
  "id=\"challenge-form\"",
322
342
  "id=\"challenge-running\"",
@@ -410,7 +430,13 @@ function classifyChallengeHtml(html) {
410
430
  * check uses, so the two can never drift apart. Script evidence follows the
411
431
  * same rule as {@link classifyChallengeHtml}: only the challenge runtime
412
432
  * counts, and the instrumentation Cloudflare injects into normal pages of a
413
- * protected zone is ignored.
433
+ * protected zone is ignored. The Turnstile widget selector is likewise
434
+ * absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
435
+ * widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
436
+ * the probe reporting "still challenged" forever, so a challenge that HAD
437
+ * cleared could never be confirmed — the wait burned its whole budget and
438
+ * the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
439
+ * layer's timeout.
414
440
  */
415
441
  const CHALLENGE_DOM_PROBE = `(() => {
416
442
  const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
@@ -418,7 +444,6 @@ const CHALLENGE_DOM_PROBE = `(() => {
418
444
  const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
419
445
  if (re.test(document.title.trim())) return true
420
446
  if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
421
- if (document.querySelector('[id^="cf-chl-widget-"]')) return true
422
447
  for (const script of Array.from(document.scripts)) {
423
448
  const src = (script.getAttribute('src') || '').toLowerCase()
424
449
  if (src.includes(platform) && src.includes(runtime)) return true
@@ -1064,11 +1089,14 @@ function installBuiltinSlashResolveCompat() {
1064
1089
  * inline `data:` image payloads are elided to size placeholders (build
1065
1090
  * tools inline images as base64, which would otherwise dominate the body);
1066
1091
  * Mozilla Readability extracts the article (dropping nav bars, sidebars,
1067
- * footers, and ad chrome by scoring link density and text mass); DOMPurify
1068
- * sanitizes whatever HTML remains and forbids the layout tags noise lives in;
1069
- * Turndown with the GFM plugin converts to markdown using the same style
1070
- * options and span-safe table rules as the shipped `dsh-tool-web` renderer,
1071
- * so output is consistent with what `web_fetch` produces elsewhere.
1092
+ * footers, and ad chrome by scoring link density and text mass) — its result
1093
+ * must then pass a coverage gate that distrusts tiny or sliver-thin
1094
+ * extractions, which on list-shaped pages (portals, catalogs) is exactly
1095
+ * what the link-density penalty leaves behind; DOMPurify sanitizes whatever
1096
+ * HTML remains and forbids the layout tags noise lives in; Turndown with
1097
+ * the GFM plugin converts to markdown using the same style options and
1098
+ * span-safe table rules as the shipped `dsh-tool-web` renderer, so output
1099
+ * is consistent with what `web_fetch` produces elsewhere.
1072
1100
  *
1073
1101
  * Synchronous for its callers and unit-tested against fixture pages: jsdom
1074
1102
  * itself is loaded lazily on the first call — and, inside a dsh profile, only
@@ -1167,6 +1195,99 @@ function humanSize(bytes) {
1167
1195
  if (bytes < 1048576) return `${(bytes / 1024).toFixed(1)}KB`;
1168
1196
  return `${(bytes / 1048576).toFixed(1)}MB`;
1169
1197
  }
1198
+ /**
1199
+ * Tags whose subtrees are noise when measuring how much of a page an
1200
+ * extraction kept: DOMPurify drops these outright (`KEEP_CONTENT: false`),
1201
+ * and its default policy always removes `script`/`style` — so a text walk
1202
+ * that skips them measures what the sanitized document fallback could
1203
+ * actually return, not text that is already destined for deletion. Measuring
1204
+ * the raw body instead would let a huge `<nav>`/`<footer>` mass shrink an
1205
+ * article's honest coverage share and demote good extractions.
1206
+ */
1207
+ const NOISE_SUBTREE_TAGS = /* @__PURE__ */ new Set([
1208
+ ...FORBID_TAGS,
1209
+ "script",
1210
+ "style"
1211
+ ]);
1212
+ /**
1213
+ * A page must have at least this much visible (post-noise) text before the
1214
+ * article acceptance gate below has an opinion. Sparse pages — logins,
1215
+ * stubs, `login-like` fixtures — have nowhere for content to hide, so
1216
+ * Readability's pick, however short, is still the best reading of the page
1217
+ * and keeps flowing through the article path exactly as before.
1218
+ */
1219
+ const GATE_MIN_BODY_TEXT_CHARS = 1e3;
1220
+ /**
1221
+ * Minimum non-whitespace characters a Readability extraction must carry
1222
+ * before it can be trusted as a page's content. Measured on the 163.com
1223
+ * homepage, the scorer's pick was a 96-character footer licensing block —
1224
+ * real articles and even sparse pages clear two-to-three times this bar,
1225
+ * while link-dense portal pages lose whole news columns to it.
1226
+ */
1227
+ const GATE_MIN_ARTICLE_TEXT_CHARS = 250;
1228
+ /**
1229
+ * Minimum share of a page's visible text an extraction must cover to be
1230
+ * trusted. Measured separation on real pages is wide: portal homepages
1231
+ * (163 0.01, sina 0.03, sohu 0.02) vs article pages (0.28–0.45), so 10%
1232
+ * sits in an empty band with margin on both sides.
1233
+ */
1234
+ const GATE_MIN_ARTICLE_COVERAGE = .1;
1235
+ /**
1236
+ * Count non-whitespace characters under `root`, skipping noise subtrees
1237
+ * ({@link NOISE_SUBTREE_TAGS}). Whitespace is ignored so CJK pages (no
1238
+ * spaces) and latin pages (space-separated) measure comparable densities,
1239
+ * mirroring how the extraction-quality ratios in the docs were computed.
1240
+ *
1241
+ * @param root - the subtree to measure; `null` counts as nothing.
1242
+ * @returns the number of non-whitespace characters in kept subtrees.
1243
+ */
1244
+ function visibleTextLength(root) {
1245
+ if (root === null) return 0;
1246
+ let length = 0;
1247
+ const visit = (node) => {
1248
+ if (node.nodeType === 3) {
1249
+ length += (node.nodeValue ?? "").replace(/\s+/g, "").length;
1250
+ return;
1251
+ }
1252
+ if (node.nodeType !== 1 || NOISE_SUBTREE_TAGS.has(node.nodeName.toLowerCase())) return;
1253
+ for (let child = node.firstChild; child !== null; child = child.nextSibling) visit(child);
1254
+ };
1255
+ visit(root);
1256
+ return length;
1257
+ }
1258
+ /**
1259
+ * Whether a Readability extraction is trustworthy as *the* content of the
1260
+ * page it came from — the acceptance gate that keeps list-shaped pages whole.
1261
+ *
1262
+ * Readability is an article extractor: it scores text mass and penalizes
1263
+ * link density, so on a portal/catalog page — where the content *is* the
1264
+ * links — every news column scores near zero, some sparse low-link block (a
1265
+ * footer licensing notice, a province-switcher popup) wins by default, and
1266
+ * Readability's own 500-char quality check cannot fail the parse because its
1267
+ * retry loop eventually returns the longest of its bad attempts. The plugin
1268
+ * used to accept any non-null result, so the 163.com homepage rendered as a
1269
+ * 162-character licensing fragment while 320 headlines were dropped — a
1270
+ * recall loss far worse than any precision the gate could cost.
1271
+ *
1272
+ * The gate therefore distrusts an extraction that is tiny in absolute terms
1273
+ * or covers a sliver of the page's visible text; the pipeline then falls
1274
+ * back to the sanitized whole document, which DOMPurify keeps clean and
1275
+ * which on such pages retains effectively all list content (measured 0%
1276
+ * text loss on the 163.com homepage). Only text-rich pages are gated
1277
+ * ({@link GATE_MIN_BODY_TEXT_CHARS}); below that, behavior is unchanged.
1278
+ *
1279
+ * @param articleHtml - the `article.content` HTML Readability produced.
1280
+ * @param document - the live document the article was extracted from.
1281
+ * @returns whether the article should flow through the article path.
1282
+ */
1283
+ function articleCoversPage(articleHtml, document) {
1284
+ const bodyText = visibleTextLength(document.body);
1285
+ if (bodyText < GATE_MIN_BODY_TEXT_CHARS) return true;
1286
+ const container = document.createElement("div");
1287
+ container.innerHTML = articleHtml;
1288
+ const articleText = visibleTextLength(container);
1289
+ return articleText >= GATE_MIN_ARTICLE_TEXT_CHARS && articleText / bodyText >= GATE_MIN_ARTICLE_COVERAGE;
1290
+ }
1170
1291
  /** The shared converter: same style options as `dsh-tool-web`'s renderer. */
1171
1292
  const turndown = new TurndownService({
1172
1293
  headingStyle: "atx",
@@ -1217,10 +1338,11 @@ turndown.addRule("tableRowWithoutSpanExpansion", {
1217
1338
  /**
1218
1339
  * Convert one rendered HTML document to denoised markdown.
1219
1340
  *
1220
- * Readability failure (non-article pages) degrades to converting the
1221
- * sanitized whole document — layout tags are still forbidden, so the
1222
- * fallback stays cleaner than raw turndown, and a degraded page beats an
1223
- * error for a body the browser already rendered.
1341
+ * Readability failure (non-article pages, or extractions the coverage gate
1342
+ * {@link articleCoversPage} distrusts) degrades to converting the sanitized
1343
+ * whole document — layout tags are still forbidden, so the fallback stays
1344
+ * cleaner than raw turndown, and a degraded page beats an error for a body
1345
+ * the browser already rendered.
1224
1346
  *
1225
1347
  * @param html - the rendered page HTML (`page.content()`).
1226
1348
  * @param url - the page URL, used to resolve relative links during parsing.
@@ -1240,7 +1362,7 @@ function htmlToMarkdown(html, url) {
1240
1362
  try {
1241
1363
  const cloned = document.cloneNode(true);
1242
1364
  const article = new Readability(cloned).parse();
1243
- if (article !== null && typeof article.content === "string" && article.content !== "") {
1365
+ if (article !== null && typeof article.content === "string" && article.content !== "" && articleCoversPage(article.content, document)) {
1244
1366
  source = article.content;
1245
1367
  title = article.title ?? void 0;
1246
1368
  }
@@ -1856,6 +1978,7 @@ var PlaywrightFetchProvider = class {
1856
1978
  content: finalResponse !== null ? await finalResponse.text() : await page.content()
1857
1979
  });
1858
1980
  await page.waitForLoadState("networkidle", { timeout: Math.min(SETTLE_MS, deadline.remainingMs()) }).catch(() => {});
1981
+ await stripScriptingElements(page);
1859
1982
  const html = await page.content();
1860
1983
  if (!config.denoise) return capResult(finalUrl, statusCode, {
1861
1984
  kind: "html",
@@ -2168,6 +2291,74 @@ async function probeStillOnChallenge(page) {
2168
2291
  return true;
2169
2292
  }
2170
2293
  }
2294
+ /**
2295
+ * In-page script that removes non-content scripting subtrees from the live
2296
+ * document before `page.content()` reads it as an HTML string.
2297
+ *
2298
+ * WHAT goes, and why each removal is output-neutral versus the old pipeline:
2299
+ *
2300
+ * - `script` — three consumers ever saw script text before, and none loses
2301
+ * it here: Readability's `parse()` deletes script/noscript nodes BEFORE
2302
+ * extraction and scoring (so script text never reached the article or
2303
+ * influenced the scorer); DOMPurify's default allowlist drops script; the
2304
+ * tool layer's renderer removes SCRIPT wholesale on the raw path. What the
2305
+ * removal buys is the reason this exists: a page's runtime JS may rewrite
2306
+ * a data script's text with a literal `<\/script>` embedded — measured on
2307
+ * techcommunity.microsoft.com (Khoros/Aurora), whose client writes its
2308
+ * Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`,
2309
+ * turning the wire's 712 `\u003c` escapes back into raw `<` — and
2310
+ * browsers serialize script text verbatim, so the flattened string ends
2311
+ * that script at the FIRST embedded end tag and re-parses the remaining
2312
+ * ~270KB of JSON as markup. The phantom elements that survive (unknown
2313
+ * tags like `<id>`, anchors full of JSON text) are exactly what
2314
+ * Readability's scorer reads as article mass, so the extracted "article"
2315
+ * is CSS and i18n strings while the real body — still present in the DOM —
2316
+ * loses. No string-level patch can undo this after serialization (the
2317
+ * string is genuinely ambiguous); only the browser's own parse holds the
2318
+ * document's true structure.
2319
+ * - `script[type="application/ld+json"]` — KEPT, but defused: Readability
2320
+ * extracts metadata (title/byline/excerpt — e.g. the site-suffix-free
2321
+ * "Remote Desktop Protocol" instead of "… - Wikipedia") from JSON-LD
2322
+ * scripts before removing them, so deleting them would regress the title
2323
+ * on every JSON-LD-carrying page. Rewriting `</` as `<\/` keeps the text
2324
+ * valid JSON (`\/` is a legal escape, so the re-parsed text parses to the
2325
+ * same object) while making the serialization harmless: the tokenizer
2326
+ * only terminates on a literal `<\/script`.
2327
+ * - `noscript` — KEPT: Readability's `_unwrapNoscriptImages` salvages the
2328
+ * real image out of lazy-load `<noscript><img …>` wrappers before
2329
+ * `_removeScripts` deletes the noscript; pre-stripping it would silently
2330
+ * drop those images from the markdown.
2331
+ * - `template` — removed: its children parse into the inert `.content`
2332
+ * fragment (invisible to Readability's traversal), DOMPurify's
2333
+ * FORBID_TAGS drops it, and the tool layer's renderer removes it too.
2334
+ * - `style` stays — its text cannot terminate a re-parse except at its own
2335
+ * end tag, and the pipeline already drops it.
2336
+ */
2337
+ const STRIP_SCRIPTING_ELEMENTS = `(() => {
2338
+ for (const el of Array.from(document.querySelectorAll('script, template'))) {
2339
+ if (el.tagName === 'SCRIPT' && (el.getAttribute('type') || '').trim().toLowerCase() === 'application/ld+json') {
2340
+ // '<\\/' is still valid JSON, so the re-parsed textContent parses to
2341
+ // the same object — while the serialized form can no longer close the
2342
+ // script element early.
2343
+ el.textContent = el.textContent.replace(/<\\//g, '<\\\\/')
2344
+ continue
2345
+ }
2346
+ el.remove()
2347
+ }
2348
+ })()`;
2349
+ /**
2350
+ * Remove non-content scripting subtrees from the live document before
2351
+ * serialization. Best-effort, mirroring {@link probeStillOnChallenge}: a
2352
+ * backend without `evaluate` (fakes, minimal engines) simply keeps the
2353
+ * document as-is — the fetch never fails here, and the pipeline degrades to
2354
+ * whatever the serialized string itself says.
2355
+ */
2356
+ async function stripScriptingElements(page) {
2357
+ if (typeof page.evaluate !== "function") return;
2358
+ try {
2359
+ await page.evaluate(STRIP_SCRIPTING_ELEMENTS);
2360
+ } catch {}
2361
+ }
2171
2362
  /** Apply the decoded-body cap and flag the cut. */
2172
2363
  function capResult(url, statusCode, body) {
2173
2364
  const truncated = body.content.length > MAX_BODY_CHARS;
@@ -14,14 +14,22 @@
14
14
  * 2. Content-level — the interstitial's localized `<title>` family
15
15
  * ("Just a moment...", "请稍候…", ...) plus structural markers
16
16
  * (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
17
- * `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
18
- * deliberately structural: a blog post that merely MENTIONS Cloudflare
19
- * must not classify as a challenge, so the challenge-platform script
20
- * check reads `<script src>` attributes only (never prose or escaped
21
- * code samples) and identifies the runtime POSITIVELY by its
22
- * `/orchestrate/` mount — the JSD/Precursor instrumentation Cloudflare
23
- * injects into every NORMAL page of a protected zone, on several moving
24
- * mounts, is not a challenge signal.
17
+ * `window._cf_chl_opt`). Markers are deliberately structural: a blog
18
+ * post that merely MENTIONS Cloudflare must not classify as a challenge,
19
+ * so the challenge-platform script check reads `<script src>` attributes
20
+ * only (never prose or escaped code samples) and identifies the runtime
21
+ * POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
22
+ * instrumentation Cloudflare injects into every NORMAL page of a
23
+ * protected zone, on several moving mounts, is not a challenge signal.
24
+ * The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
25
+ * marker: sites embed real Turnstile widgets into normal pages
26
+ * (linux.do — a Discourse forum with the Turnstile plugin mounts one on
27
+ * every topic page), so the widget alone cannot distinguish an
28
+ * interstitial from a normal page behind a Cloudflare edge. Every real
29
+ * interstitial carries at least one independent signal beyond its
30
+ * widget — the localized title, the orchestrate runtime script,
31
+ * `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
32
+ * stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
25
33
  *
26
34
  * The security stance (issue #2): this module only DETECTS and classifies.
27
35
  * Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
@@ -86,6 +94,12 @@ export declare function classifyChallengeHtml(html: string): ChallengeVerdict;
86
94
  * check uses, so the two can never drift apart. Script evidence follows the
87
95
  * same rule as {@link classifyChallengeHtml}: only the challenge runtime
88
96
  * counts, and the instrumentation Cloudflare injects into normal pages of a
89
- * protected zone is ignored.
97
+ * protected zone is ignored. The Turnstile widget selector is likewise
98
+ * absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
99
+ * widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
100
+ * the probe reporting "still challenged" forever, so a challenge that HAD
101
+ * cleared could never be confirmed — the wait burned its whole budget and
102
+ * the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
103
+ * layer's timeout.
90
104
  */
91
105
  export declare const CHALLENGE_DOM_PROBE: string;
@@ -5,11 +5,14 @@
5
5
  * inline `data:` image payloads are elided to size placeholders (build
6
6
  * tools inline images as base64, which would otherwise dominate the body);
7
7
  * Mozilla Readability extracts the article (dropping nav bars, sidebars,
8
- * footers, and ad chrome by scoring link density and text mass); DOMPurify
9
- * sanitizes whatever HTML remains and forbids the layout tags noise lives in;
10
- * Turndown with the GFM plugin converts to markdown using the same style
11
- * options and span-safe table rules as the shipped `dsh-tool-web` renderer,
12
- * so output is consistent with what `web_fetch` produces elsewhere.
8
+ * footers, and ad chrome by scoring link density and text mass) — its result
9
+ * must then pass a coverage gate that distrusts tiny or sliver-thin
10
+ * extractions, which on list-shaped pages (portals, catalogs) is exactly
11
+ * what the link-density penalty leaves behind; DOMPurify sanitizes whatever
12
+ * HTML remains and forbids the layout tags noise lives in; Turndown with
13
+ * the GFM plugin converts to markdown using the same style options and
14
+ * span-safe table rules as the shipped `dsh-tool-web` renderer, so output
15
+ * is consistent with what `web_fetch` produces elsewhere.
13
16
  *
14
17
  * Synchronous for its callers and unit-tested against fixture pages: jsdom
15
18
  * itself is loaded lazily on the first call — and, inside a dsh profile, only
@@ -24,16 +27,21 @@ export type DenoiseMode = 'article' | 'document';
24
27
  export interface DenoiseResult {
25
28
  /** The markdown body (title, when found, already prepended). */
26
29
  markdown: string;
27
- /** `article` = Readability extraction; `document` = whole-document fallback. */
30
+ /**
31
+ * `article` = Readability extraction that passed the coverage gate;
32
+ * `document` = whole-document fallback (Readability returned nothing, or
33
+ * the gate distrusted what it returned — list-shaped pages).
34
+ */
28
35
  mode: DenoiseMode;
29
36
  }
30
37
  /**
31
38
  * Convert one rendered HTML document to denoised markdown.
32
39
  *
33
- * Readability failure (non-article pages) degrades to converting the
34
- * sanitized whole document — layout tags are still forbidden, so the
35
- * fallback stays cleaner than raw turndown, and a degraded page beats an
36
- * error for a body the browser already rendered.
40
+ * Readability failure (non-article pages, or extractions the coverage gate
41
+ * {@link articleCoversPage} distrusts) degrades to converting the sanitized
42
+ * whole document — layout tags are still forbidden, so the fallback stays
43
+ * cleaner than raw turndown, and a degraded page beats an error for a body
44
+ * the browser already rendered.
37
45
  *
38
46
  * @param html - the rendered page HTML (`page.content()`).
39
47
  * @param url - the page URL, used to resolve relative links during parsing.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-web-fetch-playwright",
3
- "version": "0.2.14",
3
+ "version": "0.2.16",
4
4
  "description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/src/challenge.ts CHANGED
@@ -14,14 +14,22 @@
14
14
  * 2. Content-level — the interstitial's localized `<title>` family
15
15
  * ("Just a moment...", "请稍候…", ...) plus structural markers
16
16
  * (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
17
- * `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
18
- * deliberately structural: a blog post that merely MENTIONS Cloudflare
19
- * must not classify as a challenge, so the challenge-platform script
20
- * check reads `<script src>` attributes only (never prose or escaped
21
- * code samples) and identifies the runtime POSITIVELY by its
22
- * `/orchestrate/` mount — the JSD/Precursor instrumentation Cloudflare
23
- * injects into every NORMAL page of a protected zone, on several moving
24
- * mounts, is not a challenge signal.
17
+ * `window._cf_chl_opt`). Markers are deliberately structural: a blog
18
+ * post that merely MENTIONS Cloudflare must not classify as a challenge,
19
+ * so the challenge-platform script check reads `<script src>` attributes
20
+ * only (never prose or escaped code samples) and identifies the runtime
21
+ * POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
22
+ * instrumentation Cloudflare injects into every NORMAL page of a
23
+ * protected zone, on several moving mounts, is not a challenge signal.
24
+ * The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
25
+ * marker: sites embed real Turnstile widgets into normal pages
26
+ * (linux.do — a Discourse forum with the Turnstile plugin mounts one on
27
+ * every topic page), so the widget alone cannot distinguish an
28
+ * interstitial from a normal page behind a Cloudflare edge. Every real
29
+ * interstitial carries at least one independent signal beyond its
30
+ * widget — the localized title, the orchestrate runtime script,
31
+ * `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
32
+ * stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
25
33
  *
26
34
  * The security stance (issue #2): this module only DETECTS and classifies.
27
35
  * Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
@@ -176,9 +184,21 @@ function isChallengeRuntimeScript(src: string): boolean {
176
184
  * The challenge-platform prefix is deliberately absent: script evidence goes
177
185
  * through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
178
186
  * distinguish a real `src` from prose and runtime from instrumentation.
187
+ *
188
+ * The Turnstile widget id (`id="cf-chl-widget-…"`) is deliberately ABSENT
189
+ * (badcase: linux.do/t/topic/3143588 — a Discourse forum behind Cloudflare
190
+ * whose Turnstile plugin mounts a real widget, id and all, into every normal
191
+ * topic page it serves at HTTP 200). The widget is minted by the same
192
+ * Turnstile runtime whether it renders inside an interstitial or inside a
193
+ * site's own UI, so its presence proves only "this page embeds Turnstile".
194
+ * On a real interstitial the widget always co-occurs with independent proof
195
+ * — the localized title, the orchestrate runtime script,
196
+ * `window._cf_chl_opt`, or the `#challenge-*` scaffolding, each sufficient
197
+ * above — so dropping the widget loses no real interstitial while every
198
+ * widget-embedding normal page (linux.do, login portals, contact forms)
199
+ * stops dying as WEB_FETCH_CHALLENGE after a wait that can never clear.
179
200
  */
180
201
  const CHALLENGE_HTML_MARKERS = [
181
- 'id="cf-chl-widget-',
182
202
  'window._cf_chl_opt',
183
203
  'id="challenge-form"',
184
204
  'id="challenge-running"',
@@ -295,7 +315,13 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
295
315
  * check uses, so the two can never drift apart. Script evidence follows the
296
316
  * same rule as {@link classifyChallengeHtml}: only the challenge runtime
297
317
  * counts, and the instrumentation Cloudflare injects into normal pages of a
298
- * protected zone is ignored.
318
+ * protected zone is ignored. The Turnstile widget selector is likewise
319
+ * absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
320
+ * widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
321
+ * the probe reporting "still challenged" forever, so a challenge that HAD
322
+ * cleared could never be confirmed — the wait burned its whole budget and
323
+ * the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
324
+ * layer's timeout.
299
325
  */
300
326
  export const CHALLENGE_DOM_PROBE = `(() => {
301
327
  const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
@@ -303,7 +329,6 @@ export const CHALLENGE_DOM_PROBE = `(() => {
303
329
  const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
304
330
  if (re.test(document.title.trim())) return true
305
331
  if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
306
- if (document.querySelector('[id^="cf-chl-widget-"]')) return true
307
332
  for (const script of Array.from(document.scripts)) {
308
333
  const src = (script.getAttribute('src') || '').toLowerCase()
309
334
  if (src.includes(platform) && src.includes(runtime)) return true
package/src/markdown.ts CHANGED
@@ -5,11 +5,14 @@
5
5
  * inline `data:` image payloads are elided to size placeholders (build
6
6
  * tools inline images as base64, which would otherwise dominate the body);
7
7
  * Mozilla Readability extracts the article (dropping nav bars, sidebars,
8
- * footers, and ad chrome by scoring link density and text mass); DOMPurify
9
- * sanitizes whatever HTML remains and forbids the layout tags noise lives in;
10
- * Turndown with the GFM plugin converts to markdown using the same style
11
- * options and span-safe table rules as the shipped `dsh-tool-web` renderer,
12
- * so output is consistent with what `web_fetch` produces elsewhere.
8
+ * footers, and ad chrome by scoring link density and text mass) — its result
9
+ * must then pass a coverage gate that distrusts tiny or sliver-thin
10
+ * extractions, which on list-shaped pages (portals, catalogs) is exactly
11
+ * what the link-density penalty leaves behind; DOMPurify sanitizes whatever
12
+ * HTML remains and forbids the layout tags noise lives in; Turndown with
13
+ * the GFM plugin converts to markdown using the same style options and
14
+ * span-safe table rules as the shipped `dsh-tool-web` renderer, so output
15
+ * is consistent with what `web_fetch` produces elsewhere.
13
16
  *
14
17
  * Synchronous for its callers and unit-tested against fixture pages: jsdom
15
18
  * itself is loaded lazily on the first call — and, inside a dsh profile, only
@@ -108,6 +111,105 @@ function humanSize(bytes: number): string {
108
111
  return `${(bytes / 1_048_576).toFixed(1)}MB`
109
112
  }
110
113
 
114
+ /**
115
+ * Tags whose subtrees are noise when measuring how much of a page an
116
+ * extraction kept: DOMPurify drops these outright (`KEEP_CONTENT: false`),
117
+ * and its default policy always removes `script`/`style` — so a text walk
118
+ * that skips them measures what the sanitized document fallback could
119
+ * actually return, not text that is already destined for deletion. Measuring
120
+ * the raw body instead would let a huge `<nav>`/`<footer>` mass shrink an
121
+ * article's honest coverage share and demote good extractions.
122
+ */
123
+ const NOISE_SUBTREE_TAGS = new Set([...FORBID_TAGS, 'script', 'style'])
124
+
125
+ /**
126
+ * A page must have at least this much visible (post-noise) text before the
127
+ * article acceptance gate below has an opinion. Sparse pages — logins,
128
+ * stubs, `login-like` fixtures — have nowhere for content to hide, so
129
+ * Readability's pick, however short, is still the best reading of the page
130
+ * and keeps flowing through the article path exactly as before.
131
+ */
132
+ const GATE_MIN_BODY_TEXT_CHARS = 1_000
133
+
134
+ /**
135
+ * Minimum non-whitespace characters a Readability extraction must carry
136
+ * before it can be trusted as a page's content. Measured on the 163.com
137
+ * homepage, the scorer's pick was a 96-character footer licensing block —
138
+ * real articles and even sparse pages clear two-to-three times this bar,
139
+ * while link-dense portal pages lose whole news columns to it.
140
+ */
141
+ const GATE_MIN_ARTICLE_TEXT_CHARS = 250
142
+
143
+ /**
144
+ * Minimum share of a page's visible text an extraction must cover to be
145
+ * trusted. Measured separation on real pages is wide: portal homepages
146
+ * (163 0.01, sina 0.03, sohu 0.02) vs article pages (0.28–0.45), so 10%
147
+ * sits in an empty band with margin on both sides.
148
+ */
149
+ const GATE_MIN_ARTICLE_COVERAGE = 0.1
150
+
151
+ /**
152
+ * Count non-whitespace characters under `root`, skipping noise subtrees
153
+ * ({@link NOISE_SUBTREE_TAGS}). Whitespace is ignored so CJK pages (no
154
+ * spaces) and latin pages (space-separated) measure comparable densities,
155
+ * mirroring how the extraction-quality ratios in the docs were computed.
156
+ *
157
+ * @param root - the subtree to measure; `null` counts as nothing.
158
+ * @returns the number of non-whitespace characters in kept subtrees.
159
+ */
160
+ function visibleTextLength(root: Node | null): number {
161
+ if (root === null) return 0
162
+ let length = 0
163
+ const visit = (node: Node): void => {
164
+ // Text nodes contribute their non-whitespace characters (nodeType 3).
165
+ if (node.nodeType === 3) {
166
+ length += (node.nodeValue ?? '').replace(/\s+/g, '').length
167
+ return
168
+ }
169
+ // Only elements branch further (nodeType 1); comments and friends carry
170
+ // no text, and noise subtrees are skipped wholesale.
171
+ if (node.nodeType !== 1 || NOISE_SUBTREE_TAGS.has(node.nodeName.toLowerCase())) return
172
+ for (let child = node.firstChild; child !== null; child = child.nextSibling) visit(child)
173
+ }
174
+ visit(root)
175
+ return length
176
+ }
177
+
178
+ /**
179
+ * Whether a Readability extraction is trustworthy as *the* content of the
180
+ * page it came from — the acceptance gate that keeps list-shaped pages whole.
181
+ *
182
+ * Readability is an article extractor: it scores text mass and penalizes
183
+ * link density, so on a portal/catalog page — where the content *is* the
184
+ * links — every news column scores near zero, some sparse low-link block (a
185
+ * footer licensing notice, a province-switcher popup) wins by default, and
186
+ * Readability's own 500-char quality check cannot fail the parse because its
187
+ * retry loop eventually returns the longest of its bad attempts. The plugin
188
+ * used to accept any non-null result, so the 163.com homepage rendered as a
189
+ * 162-character licensing fragment while 320 headlines were dropped — a
190
+ * recall loss far worse than any precision the gate could cost.
191
+ *
192
+ * The gate therefore distrusts an extraction that is tiny in absolute terms
193
+ * or covers a sliver of the page's visible text; the pipeline then falls
194
+ * back to the sanitized whole document, which DOMPurify keeps clean and
195
+ * which on such pages retains effectively all list content (measured 0%
196
+ * text loss on the 163.com homepage). Only text-rich pages are gated
197
+ * ({@link GATE_MIN_BODY_TEXT_CHARS}); below that, behavior is unchanged.
198
+ *
199
+ * @param articleHtml - the `article.content` HTML Readability produced.
200
+ * @param document - the live document the article was extracted from.
201
+ * @returns whether the article should flow through the article path.
202
+ */
203
+ function articleCoversPage(articleHtml: string, document: Document): boolean {
204
+ const bodyText = visibleTextLength(document.body)
205
+ if (bodyText < GATE_MIN_BODY_TEXT_CHARS) return true
206
+ const container = document.createElement('div')
207
+ container.innerHTML = articleHtml
208
+ const articleText = visibleTextLength(container)
209
+ return articleText >= GATE_MIN_ARTICLE_TEXT_CHARS
210
+ && articleText / bodyText >= GATE_MIN_ARTICLE_COVERAGE
211
+ }
212
+
111
213
  /** The shared converter: same style options as `dsh-tool-web`'s renderer. */
112
214
  const turndown = new TurndownService({
113
215
  headingStyle: 'atx',
@@ -170,17 +272,22 @@ export type DenoiseMode = 'article' | 'document'
170
272
  export interface DenoiseResult {
171
273
  /** The markdown body (title, when found, already prepended). */
172
274
  markdown: string
173
- /** `article` = Readability extraction; `document` = whole-document fallback. */
275
+ /**
276
+ * `article` = Readability extraction that passed the coverage gate;
277
+ * `document` = whole-document fallback (Readability returned nothing, or
278
+ * the gate distrusted what it returned — list-shaped pages).
279
+ */
174
280
  mode: DenoiseMode
175
281
  }
176
282
 
177
283
  /**
178
284
  * Convert one rendered HTML document to denoised markdown.
179
285
  *
180
- * Readability failure (non-article pages) degrades to converting the
181
- * sanitized whole document — layout tags are still forbidden, so the
182
- * fallback stays cleaner than raw turndown, and a degraded page beats an
183
- * error for a body the browser already rendered.
286
+ * Readability failure (non-article pages, or extractions the coverage gate
287
+ * {@link articleCoversPage} distrusts) degrades to converting the sanitized
288
+ * whole document — layout tags are still forbidden, so the fallback stays
289
+ * cleaner than raw turndown, and a degraded page beats an error for a body
290
+ * the browser already rendered.
184
291
  *
185
292
  * @param html - the rendered page HTML (`page.content()`).
186
293
  * @param url - the page URL, used to resolve relative links during parsing.
@@ -204,14 +311,18 @@ export function htmlToMarkdown(html: string, url: string): DenoiseResult {
204
311
  let source: string | null = null
205
312
  let title: string | undefined
206
313
  try {
207
- // Ungated: isProbablyReaderable is a conservative hint that rejects
208
- // sparse-but-real articles (measured on a browser-rendered fixture), so
209
- // the extraction is simply attempted; a null or empty result falls back
210
- // to the sanitized whole document below. parse() mutates, hence the clone
211
- // (typed as Document: DOM lib types cloneNode's return as Node).
314
+ // Ungated readability: isProbablyReaderable is a conservative hint that
315
+ // rejects sparse-but-real articles (measured on a browser-rendered fixture),
316
+ // so the extraction is simply attempted; a null, empty, or gate-rejected
317
+ // result falls back to the sanitized whole document below. parse() mutates,
318
+ // hence the clone (typed as Document: DOM lib types cloneNode's return as
319
+ // Node).
212
320
  const cloned = document.cloneNode(true) as typeof document
213
321
  const article = new Readability(cloned).parse()
214
- if (article !== null && typeof article.content === 'string' && article.content !== '') {
322
+ if (
323
+ article !== null && typeof article.content === 'string' && article.content !== ''
324
+ && articleCoversPage(article.content, document)
325
+ ) {
215
326
  source = article.content
216
327
  title = article.title ?? undefined
217
328
  }
package/src/provider.ts CHANGED
@@ -535,6 +535,13 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
535
535
  // what domcontentloaded already produced.
536
536
  await page.waitForLoadState('networkidle', { timeout: Math.min(SETTLE_MS, deadline.remainingMs()) }).catch(() => {})
537
537
 
538
+ // Serialization safety: the document is about to be flattened into an
539
+ // HTML string and re-parsed by jsdom, and scripting elements are the one
540
+ // place that round-trip can break — see {@link STRIP_SCRIPTING_ELEMENTS}
541
+ // for exactly what is removed, what is kept (JSON-LD, noscript), and why
542
+ // each removal cannot change what the pipeline extracts.
543
+ await stripScriptingElements(page)
544
+
538
545
  const html = await page.content()
539
546
  if (!config.denoise) {
540
547
  // The tool layer's own turndown renders raw HTML; the checkbox only
@@ -900,6 +907,78 @@ async function probeStillOnChallenge(page: PlaywrightPage): Promise<boolean> {
900
907
  }
901
908
  }
902
909
 
910
+ /**
911
+ * In-page script that removes non-content scripting subtrees from the live
912
+ * document before `page.content()` reads it as an HTML string.
913
+ *
914
+ * WHAT goes, and why each removal is output-neutral versus the old pipeline:
915
+ *
916
+ * - `script` — three consumers ever saw script text before, and none loses
917
+ * it here: Readability's `parse()` deletes script/noscript nodes BEFORE
918
+ * extraction and scoring (so script text never reached the article or
919
+ * influenced the scorer); DOMPurify's default allowlist drops script; the
920
+ * tool layer's renderer removes SCRIPT wholesale on the raw path. What the
921
+ * removal buys is the reason this exists: a page's runtime JS may rewrite
922
+ * a data script's text with a literal `</script>` embedded — measured on
923
+ * techcommunity.microsoft.com (Khoros/Aurora), whose client writes its
924
+ * Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`,
925
+ * turning the wire's 712 `\u003c` escapes back into raw `<` — and
926
+ * browsers serialize script text verbatim, so the flattened string ends
927
+ * that script at the FIRST embedded end tag and re-parses the remaining
928
+ * ~270KB of JSON as markup. The phantom elements that survive (unknown
929
+ * tags like `<id>`, anchors full of JSON text) are exactly what
930
+ * Readability's scorer reads as article mass, so the extracted "article"
931
+ * is CSS and i18n strings while the real body — still present in the DOM —
932
+ * loses. No string-level patch can undo this after serialization (the
933
+ * string is genuinely ambiguous); only the browser's own parse holds the
934
+ * document's true structure.
935
+ * - `script[type="application/ld+json"]` — KEPT, but defused: Readability
936
+ * extracts metadata (title/byline/excerpt — e.g. the site-suffix-free
937
+ * "Remote Desktop Protocol" instead of "… - Wikipedia") from JSON-LD
938
+ * scripts before removing them, so deleting them would regress the title
939
+ * on every JSON-LD-carrying page. Rewriting `</` as `<\/` keeps the text
940
+ * valid JSON (`\/` is a legal escape, so the re-parsed text parses to the
941
+ * same object) while making the serialization harmless: the tokenizer
942
+ * only terminates on a literal `</script`.
943
+ * - `noscript` — KEPT: Readability's `_unwrapNoscriptImages` salvages the
944
+ * real image out of lazy-load `<noscript><img …>` wrappers before
945
+ * `_removeScripts` deletes the noscript; pre-stripping it would silently
946
+ * drop those images from the markdown.
947
+ * - `template` — removed: its children parse into the inert `.content`
948
+ * fragment (invisible to Readability's traversal), DOMPurify's
949
+ * FORBID_TAGS drops it, and the tool layer's renderer removes it too.
950
+ * - `style` stays — its text cannot terminate a re-parse except at its own
951
+ * end tag, and the pipeline already drops it.
952
+ */
953
+ const STRIP_SCRIPTING_ELEMENTS = `(() => {
954
+ for (const el of Array.from(document.querySelectorAll('script, template'))) {
955
+ if (el.tagName === 'SCRIPT' && (el.getAttribute('type') || '').trim().toLowerCase() === 'application/ld+json') {
956
+ // '<\\/' is still valid JSON, so the re-parsed textContent parses to
957
+ // the same object — while the serialized form can no longer close the
958
+ // script element early.
959
+ el.textContent = el.textContent.replace(/<\\//g, '<\\\\/')
960
+ continue
961
+ }
962
+ el.remove()
963
+ }
964
+ })()`
965
+
966
+ /**
967
+ * Remove non-content scripting subtrees from the live document before
968
+ * serialization. Best-effort, mirroring {@link probeStillOnChallenge}: a
969
+ * backend without `evaluate` (fakes, minimal engines) simply keeps the
970
+ * document as-is — the fetch never fails here, and the pipeline degrades to
971
+ * whatever the serialized string itself says.
972
+ */
973
+ async function stripScriptingElements(page: PlaywrightPage): Promise<void> {
974
+ if (typeof page.evaluate !== 'function') return
975
+ try {
976
+ await page.evaluate(STRIP_SCRIPTING_ELEMENTS)
977
+ } catch {
978
+ // keep going without the strip
979
+ }
980
+ }
981
+
903
982
  /** Apply the decoded-body cap and flag the cut. */
904
983
  function capResult(url: string, statusCode: number, body: { kind: 'html' | 'text'; content: string }): WebFetchResult {
905
984
  const truncated = body.content.length > MAX_BODY_CHARS