dsh-web-fetch-playwright 0.2.14 → 0.2.15

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -29,13 +29,15 @@ web_fetch (tool-web)
29
29
  └─ ctx.web.fetchProvider = playwright
30
30
  ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
31
31
  ├─ cdp: connectOverCDP(endpoint)
32
- ├─ page.goto → settle (networkidle, best-effort) → page.content()
32
+ ├─ page.goto → settle (networkidle, best-effort) → strip scripting subtrees in the live DOM → page.content()
33
33
  ├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
34
34
  └─ Markdown (or raw HTML when denoise is off)
35
35
  ```
36
36
 
37
37
  **Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; undecodable downloads (a PDF, a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
38
38
 
39
+ **Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction.** The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses `page.content()` with jsdom, and a script's *text* is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal `</script>` embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`, un-escaping the wire's 712 `\u003c` sequences into raw `<` — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the *first* embedded end tag and parses the remaining ~270KB of JSON *as markup*. The phantom elements that survive (unknown tags like `<id>`, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and `CachedAsset` strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary `script` and `template` subtrees are stripped inside the live page (`page.evaluate`, best-effort) before `page.content()`. This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back **byte-identical on every page**. Two deliberate carve-outs keep even the indirect consumers intact: `script[type="application/ld+json"]` is *defused* rather than deleted (`</` rewritten to `<\/` — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and `noscript` is kept entirely because Readability's `_unwrapNoscriptImages` salvages the real image out of lazy-load `<noscript><img …>` wrappers. The strip also applies with `denoise` off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).
40
+
39
41
  ## Requirements
40
42
 
41
43
  - DSH web profile (`dsh web`), Node.js ≥ 20.
package/README.zh-CN.md CHANGED
@@ -29,13 +29,15 @@ web_fetch (tool-web)
29
29
  └─ ctx.web.fetchProvider = playwright
30
30
  ├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
31
31
  ├─ cdp: connectOverCDP(endpoint)
32
- ├─ page.goto → 等待稳定(networkidle,尽力而为)→ page.content()
32
+ ├─ page.goto → 等待稳定(networkidle,尽力而为)→ 在活 DOM 中剥离脚本子树 → page.content()
33
33
  ├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → DOMPurify → Turndown(GFM)
34
34
  └─ Markdown(关闭降噪时返回原始 HTML)
35
35
  ```
36
36
 
37
37
  **非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;不可解码的下载(PDF、ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;传输错误与取消也依旧按错误处理。
38
38
 
39
+ **回读文档前,先在活 DOM 中剥离非内容脚本子树 —— 构造上即输出中性。** 降噪管线(以及关闭降噪时工具层自己的渲染器)会用 jsdom 重新解析 `page.content()`,而脚本的*文本内容*是这条往返链路中唯一无法信任的东西:页面自身的运行时 JS 可能把内嵌 `</script>` 的文本写回某个数据脚本的 textContent —— 实测于 techcommunity.microsoft.com(Khoros/Aurora 平台),其客户端把整个 Apollo 缓存经 `JSON.stringify` 写回 `script#__NEXT_DATA__`,把线上响应里 712 处 `\u003c` 转义还原成了裸 `<` —— 而浏览器序列化脚本文本时原样输出,于是这份扁平化的 HTML 字符串会在*第一个*内嵌结束标签处终止脚本,把余下约 270KB 的 JSON *当作标记语言*解析。解析出的幽灵元素(`<id>` 之类的未知标签、塞满 CSS 与 i18n 字符串的锚点)恰好是 Readability 评分器眼里的"正文质量"(实测 126 分对真实正文的 45 分),于是抓取 RDP 10 AVC/H.264 一文时返回的是 CSS 规则和 `CachedAsset` 字符串而不是文章本体。事后再对字符串做补丁无法消除这种歧义 —— 只有浏览器自己的解析持有文档的真实结构 —— 因此普通的 `script` 与 `template` 子树在 `page.content()` 之前先在活页面里剥掉(`page.evaluate`,尽力而为)。这不会改变管线以往的任何输出:Readability 本就在抽取与评分前删除 script/noscript 节点,DOMPurify 的白名单本就丢弃 script,工具层渲染器也整体移除 SCRIPT/TEMPLATE —— 在真实页面语料(Wikipedia、MDN、Hacker News、坏例页 wire 版,加上 noscript/JSON-LD/template 边界用例)上的差分 A/B **每一页都字节级一致**。两个刻意的保留让间接消费方也不受损:`script[type="application/ld+json"]` 只做*消险*而非删除(`</` 改写为 `<\/` —— 仍是合法 JSON,Readability 的元数据读取解析到同一对象,文章标题保持无站点后缀的干净形态,如 "Remote Desktop Protocol" 而非 "… - Wikipedia");`noscript` 完整保留,因为 Readability 的 `_unwrapNoscriptImages` 会从懒加载的 `<noscript><img …>` 包裹中抢救出真实图片。该剥离在关闭降噪时同样生效,让返回给工具层渲染器的原始 HTML 也是序列化安全的,并把管线要解析的体积压缩约 70%(以该坏例页计)。
40
+
39
41
  ## 环境要求
40
42
 
41
43
  - DSH web profile(`dsh web`),Node.js ≥ 20。
package/lib/client.js CHANGED
@@ -415,12 +415,12 @@ window.__ModuleLoader__.load({
415
415
  document.head.appendChild(tag);
416
416
  }
417
417
  var PluginCard_module_css_default = {
418
- "discard": "UM8-Rq_discard",
419
- "save": "UM8-Rq_save",
420
- "readOnly": "UM8-Rq_readOnly",
418
+ "footer": "UM8-Rq_footer",
421
419
  "form": "UM8-Rq_form",
422
420
  "failed": "UM8-Rq_failed",
423
- "footer": "UM8-Rq_footer"
421
+ "save": "UM8-Rq_save",
422
+ "readOnly": "UM8-Rq_readOnly",
423
+ "discard": "UM8-Rq_discard"
424
424
  };
425
425
  //#endregion
426
426
  //#region src/client/PluginCard.tsx
@@ -480,27 +480,27 @@ window.__ModuleLoader__.load({
480
480
  document.head.appendChild(tag);
481
481
  }
482
482
  var fields_module_css_default = {
483
- "head": "CFSaVG_head",
484
- "fieldEmbedded": "CFSaVG_fieldEmbedded",
485
- "radioLabel": "CFSaVG_radioLabel",
486
- "radioOption": "CFSaVG_radioOption",
487
- "input": "CFSaVG_input",
483
+ "radioHint": "CFSaVG_radioHint",
484
+ "checkboxRow": "CFSaVG_checkboxRow",
485
+ "label": "CFSaVG_label",
488
486
  "field": "CFSaVG_field",
489
- "reset": "CFSaVG_reset",
490
- "badge": "CFSaVG_badge",
491
- "invalid": "CFSaVG_invalid",
492
487
  "hint": "CFSaVG_hint",
493
- "radioPick": "CFSaVG_radioPick",
494
- "radioContent": "CFSaVG_radioContent",
495
- "label": "CFSaVG_label",
496
- "inputInvalid": "CFSaVG_inputInvalid",
497
- "hintIndented": "CFSaVG_hintIndented",
498
- "radioOptionPlain": "CFSaVG_radioOptionPlain",
499
- "checkboxRow": "CFSaVG_checkboxRow",
500
488
  "radioGroup": "CFSaVG_radioGroup",
489
+ "radioContent": "CFSaVG_radioContent",
490
+ "head": "CFSaVG_head",
501
491
  "badges": "CFSaVG_badges",
502
492
  "radioText": "CFSaVG_radioText",
503
- "radioHint": "CFSaVG_radioHint"
493
+ "badge": "CFSaVG_badge",
494
+ "radioOptionPlain": "CFSaVG_radioOptionPlain",
495
+ "radioPick": "CFSaVG_radioPick",
496
+ "fieldEmbedded": "CFSaVG_fieldEmbedded",
497
+ "input": "CFSaVG_input",
498
+ "inputInvalid": "CFSaVG_inputInvalid",
499
+ "radioOption": "CFSaVG_radioOption",
500
+ "invalid": "CFSaVG_invalid",
501
+ "hintIndented": "CFSaVG_hintIndented",
502
+ "reset": "CFSaVG_reset",
503
+ "radioLabel": "CFSaVG_radioLabel"
504
504
  };
505
505
  //#endregion
506
506
  //#region src/client/fields.tsx
package/lib/index.js CHANGED
@@ -1856,6 +1856,7 @@ var PlaywrightFetchProvider = class {
1856
1856
  content: finalResponse !== null ? await finalResponse.text() : await page.content()
1857
1857
  });
1858
1858
  await page.waitForLoadState("networkidle", { timeout: Math.min(SETTLE_MS, deadline.remainingMs()) }).catch(() => {});
1859
+ await stripScriptingElements(page);
1859
1860
  const html = await page.content();
1860
1861
  if (!config.denoise) return capResult(finalUrl, statusCode, {
1861
1862
  kind: "html",
@@ -2168,6 +2169,74 @@ async function probeStillOnChallenge(page) {
2168
2169
  return true;
2169
2170
  }
2170
2171
  }
2172
+ /**
2173
+ * In-page script that removes non-content scripting subtrees from the live
2174
+ * document before `page.content()` reads it as an HTML string.
2175
+ *
2176
+ * WHAT goes, and why each removal is output-neutral versus the old pipeline:
2177
+ *
2178
+ * - `script` — three consumers ever saw script text before, and none loses
2179
+ * it here: Readability's `parse()` deletes script/noscript nodes BEFORE
2180
+ * extraction and scoring (so script text never reached the article or
2181
+ * influenced the scorer); DOMPurify's default allowlist drops script; the
2182
+ * tool layer's renderer removes SCRIPT wholesale on the raw path. What the
2183
+ * removal buys is the reason this exists: a page's runtime JS may rewrite
2184
+ * a data script's text with a literal `<\/script>` embedded — measured on
2185
+ * techcommunity.microsoft.com (Khoros/Aurora), whose client writes its
2186
+ * Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`,
2187
+ * turning the wire's 712 `\u003c` escapes back into raw `<` — and
2188
+ * browsers serialize script text verbatim, so the flattened string ends
2189
+ * that script at the FIRST embedded end tag and re-parses the remaining
2190
+ * ~270KB of JSON as markup. The phantom elements that survive (unknown
2191
+ * tags like `<id>`, anchors full of JSON text) are exactly what
2192
+ * Readability's scorer reads as article mass, so the extracted "article"
2193
+ * is CSS and i18n strings while the real body — still present in the DOM —
2194
+ * loses. No string-level patch can undo this after serialization (the
2195
+ * string is genuinely ambiguous); only the browser's own parse holds the
2196
+ * document's true structure.
2197
+ * - `script[type="application/ld+json"]` — KEPT, but defused: Readability
2198
+ * extracts metadata (title/byline/excerpt — e.g. the site-suffix-free
2199
+ * "Remote Desktop Protocol" instead of "… - Wikipedia") from JSON-LD
2200
+ * scripts before removing them, so deleting them would regress the title
2201
+ * on every JSON-LD-carrying page. Rewriting `</` as `<\/` keeps the text
2202
+ * valid JSON (`\/` is a legal escape, so the re-parsed text parses to the
2203
+ * same object) while making the serialization harmless: the tokenizer
2204
+ * only terminates on a literal `<\/script`.
2205
+ * - `noscript` — KEPT: Readability's `_unwrapNoscriptImages` salvages the
2206
+ * real image out of lazy-load `<noscript><img …>` wrappers before
2207
+ * `_removeScripts` deletes the noscript; pre-stripping it would silently
2208
+ * drop those images from the markdown.
2209
+ * - `template` — removed: its children parse into the inert `.content`
2210
+ * fragment (invisible to Readability's traversal), DOMPurify's
2211
+ * FORBID_TAGS drops it, and the tool layer's renderer removes it too.
2212
+ * - `style` stays — its text cannot terminate a re-parse except at its own
2213
+ * end tag, and the pipeline already drops it.
2214
+ */
2215
+ const STRIP_SCRIPTING_ELEMENTS = `(() => {
2216
+ for (const el of Array.from(document.querySelectorAll('script, template'))) {
2217
+ if (el.tagName === 'SCRIPT' && (el.getAttribute('type') || '').trim().toLowerCase() === 'application/ld+json') {
2218
+ // '<\\/' is still valid JSON, so the re-parsed textContent parses to
2219
+ // the same object — while the serialized form can no longer close the
2220
+ // script element early.
2221
+ el.textContent = el.textContent.replace(/<\\//g, '<\\\\/')
2222
+ continue
2223
+ }
2224
+ el.remove()
2225
+ }
2226
+ })()`;
2227
+ /**
2228
+ * Remove non-content scripting subtrees from the live document before
2229
+ * serialization. Best-effort, mirroring {@link probeStillOnChallenge}: a
2230
+ * backend without `evaluate` (fakes, minimal engines) simply keeps the
2231
+ * document as-is — the fetch never fails here, and the pipeline degrades to
2232
+ * whatever the serialized string itself says.
2233
+ */
2234
+ async function stripScriptingElements(page) {
2235
+ if (typeof page.evaluate !== "function") return;
2236
+ try {
2237
+ await page.evaluate(STRIP_SCRIPTING_ELEMENTS);
2238
+ } catch {}
2239
+ }
2171
2240
  /** Apply the decoded-body cap and flag the cut. */
2172
2241
  function capResult(url, statusCode, body) {
2173
2242
  const truncated = body.content.length > MAX_BODY_CHARS;
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-web-fetch-playwright",
3
- "version": "0.2.14",
3
+ "version": "0.2.15",
4
4
  "description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/src/provider.ts CHANGED
@@ -535,6 +535,13 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
535
535
  // what domcontentloaded already produced.
536
536
  await page.waitForLoadState('networkidle', { timeout: Math.min(SETTLE_MS, deadline.remainingMs()) }).catch(() => {})
537
537
 
538
+ // Serialization safety: the document is about to be flattened into an
539
+ // HTML string and re-parsed by jsdom, and scripting elements are the one
540
+ // place that round-trip can break — see {@link STRIP_SCRIPTING_ELEMENTS}
541
+ // for exactly what is removed, what is kept (JSON-LD, noscript), and why
542
+ // each removal cannot change what the pipeline extracts.
543
+ await stripScriptingElements(page)
544
+
538
545
  const html = await page.content()
539
546
  if (!config.denoise) {
540
547
  // The tool layer's own turndown renders raw HTML; the checkbox only
@@ -900,6 +907,78 @@ async function probeStillOnChallenge(page: PlaywrightPage): Promise<boolean> {
900
907
  }
901
908
  }
902
909
 
910
+ /**
911
+ * In-page script that removes non-content scripting subtrees from the live
912
+ * document before `page.content()` reads it as an HTML string.
913
+ *
914
+ * WHAT goes, and why each removal is output-neutral versus the old pipeline:
915
+ *
916
+ * - `script` — three consumers ever saw script text before, and none loses
917
+ * it here: Readability's `parse()` deletes script/noscript nodes BEFORE
918
+ * extraction and scoring (so script text never reached the article or
919
+ * influenced the scorer); DOMPurify's default allowlist drops script; the
920
+ * tool layer's renderer removes SCRIPT wholesale on the raw path. What the
921
+ * removal buys is the reason this exists: a page's runtime JS may rewrite
922
+ * a data script's text with a literal `</script>` embedded — measured on
923
+ * techcommunity.microsoft.com (Khoros/Aurora), whose client writes its
924
+ * Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`,
925
+ * turning the wire's 712 `\u003c` escapes back into raw `<` — and
926
+ * browsers serialize script text verbatim, so the flattened string ends
927
+ * that script at the FIRST embedded end tag and re-parses the remaining
928
+ * ~270KB of JSON as markup. The phantom elements that survive (unknown
929
+ * tags like `<id>`, anchors full of JSON text) are exactly what
930
+ * Readability's scorer reads as article mass, so the extracted "article"
931
+ * is CSS and i18n strings while the real body — still present in the DOM —
932
+ * loses. No string-level patch can undo this after serialization (the
933
+ * string is genuinely ambiguous); only the browser's own parse holds the
934
+ * document's true structure.
935
+ * - `script[type="application/ld+json"]` — KEPT, but defused: Readability
936
+ * extracts metadata (title/byline/excerpt — e.g. the site-suffix-free
937
+ * "Remote Desktop Protocol" instead of "… - Wikipedia") from JSON-LD
938
+ * scripts before removing them, so deleting them would regress the title
939
+ * on every JSON-LD-carrying page. Rewriting `</` as `<\/` keeps the text
940
+ * valid JSON (`\/` is a legal escape, so the re-parsed text parses to the
941
+ * same object) while making the serialization harmless: the tokenizer
942
+ * only terminates on a literal `</script`.
943
+ * - `noscript` — KEPT: Readability's `_unwrapNoscriptImages` salvages the
944
+ * real image out of lazy-load `<noscript><img …>` wrappers before
945
+ * `_removeScripts` deletes the noscript; pre-stripping it would silently
946
+ * drop those images from the markdown.
947
+ * - `template` — removed: its children parse into the inert `.content`
948
+ * fragment (invisible to Readability's traversal), DOMPurify's
949
+ * FORBID_TAGS drops it, and the tool layer's renderer removes it too.
950
+ * - `style` stays — its text cannot terminate a re-parse except at its own
951
+ * end tag, and the pipeline already drops it.
952
+ */
953
+ const STRIP_SCRIPTING_ELEMENTS = `(() => {
954
+ for (const el of Array.from(document.querySelectorAll('script, template'))) {
955
+ if (el.tagName === 'SCRIPT' && (el.getAttribute('type') || '').trim().toLowerCase() === 'application/ld+json') {
956
+ // '<\\/' is still valid JSON, so the re-parsed textContent parses to
957
+ // the same object — while the serialized form can no longer close the
958
+ // script element early.
959
+ el.textContent = el.textContent.replace(/<\\//g, '<\\\\/')
960
+ continue
961
+ }
962
+ el.remove()
963
+ }
964
+ })()`
965
+
966
+ /**
967
+ * Remove non-content scripting subtrees from the live document before
968
+ * serialization. Best-effort, mirroring {@link probeStillOnChallenge}: a
969
+ * backend without `evaluate` (fakes, minimal engines) simply keeps the
970
+ * document as-is — the fetch never fails here, and the pipeline degrades to
971
+ * whatever the serialized string itself says.
972
+ */
973
+ async function stripScriptingElements(page: PlaywrightPage): Promise<void> {
974
+ if (typeof page.evaluate !== 'function') return
975
+ try {
976
+ await page.evaluate(STRIP_SCRIPTING_ELEMENTS)
977
+ } catch {
978
+ // keep going without the strip
979
+ }
980
+ }
981
+
903
982
  /** Apply the decoded-body cap and flag the cut. */
904
983
  function capResult(url: string, statusCode: number, body: { kind: 'html' | 'text'; content: string }): WebFetchResult {
905
984
  const truncated = body.content.length > MAX_BODY_CHARS