dsh-web-fetch-playwright 0.2.22 → 0.2.23

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -32,12 +32,14 @@ web_fetch (tool-web)
32
32
  ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
33
33
  ├─ cdp: connectOverCDP(endpoint)
34
34
  ├─ PDF (inline / attachment / sniffed octet-stream) → bytes (navigation body, else request-context refetch) → pdf.js → layout heuristics → Markdown
35
- ├─ page.goto → settle (networkidle, best-effort) → deep-capture scroll walk (virtualized feeds; returns its merged, stripped capture when it restored anything) → otherwise strip scripting subtrees in the live DOM → page.content()
35
+ ├─ page.goto → settle (load baseline + bounded network-quiet window with a stale-request discount, best-effort — never Playwright's networkidle: worker-script and cross-origin-auth requests never complete on the page session, so strict idle is unreachable on modern SPA pages) → deep-capture scroll walk (virtualized feeds; returns its merged, stripped capture when it restored anything) → otherwise strip scripting subtrees in the live DOM → page.content()
36
36
  ├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → trailing link-dense prune → DOMPurify → Turndown(GFM)
37
37
  └─ Markdown (or raw HTML when denoise is off)
38
38
  ```
39
39
 
40
- **Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; a PDF download is extracted to Markdown (see [PDF links](#pdf-links)), and other undecodable downloads (a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
40
+ **Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; a PDF download is extracted to Markdown (see [PDF links](#pdf-links)), and other undecodable downloads (a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; persistent transport errors and cancellations stay errors — but *transient* transport blips are retried first: `net::ERR_NETWORK_CHANGED` (Chromium's NetworkChangeNotifier killed the in-flight navigation because the host's network configuration shifted under the browser — a route/DNS/VPN/Docker-DNS change; measured live: one session's source.android.com fetch died this way and the same URL fetched clean seconds later) and its sibling codes (`ERR_CONNECTION_RESET`, `ERR_NAME_NOT_RESOLVED`, …) re-navigate the same page up to twice with a short backoff, bounded by the remaining fetch budget; `ERR_ABORTED` and Playwright timeout errors are never retried (their shapes are deterministic and already owned by the recoveries above).
41
+
42
+ **Web-component app shells skip the Readability family they cannot win, and large pages stop re-parsing the same DOM.** The denoise pipeline is synchronous and jsdom-bound, so a heavy page pays its whole cost as dead air after the browser already holds the full document — measured on the badcase (`aistudio.google.com/docs/computer-use`, fetched over CDP into the user's real browser): the page was fully rendered ~7s in, then the pipeline spent another 13.4s on the 1.1MB capture, of which ~6.5s was three Readability passes and ~2.9s eight DOMPurify re-parses. Two of those passes and most of those parses produced *provably dead* candidates: the page is an Angular app whose entire DOM lives under `<ms-app>`, and DOMPurify's allowlist has no hyphenated (custom-element) tags, so with `KEEP_CONTENT: false` every extraction carrying that wrapper sanitizes to a corpse (Readability's pick: a root with zero words; the whole-document candidate: 39 of ~4.5k words). When ≥90% of the body's visible text sits inside custom-element subtrees (`customWrappedTextShare`), candidate generation therefore skips the Readability family (`art`/`artu`/`artx`) and marks the skip in the facts (`readabilitySkipped`); the selector is seeded with the empty-`art` reference stats a dead pick would have measured, so the surviving candidates' features are unchanged, and the DOM-backed fallbacks, noscript paths and block classifier — what the selector picks on that shape anyway — carry the extraction. Independently, candidates the generator already holds as DOM nodes (`doc`, `docc`, `main`, `ns_doc`) are sanitized by importing the node (identical output, measured 555ms → 304ms for the whole-body candidate) instead of re-parsing their serialization; the Readability article is parsed once for the coverage gate and the trailing prune instead of once each; and rail-name recognition short-circuits on a literal hint before the per-token split (~360ms of the walk over every element on the badcase). Net effect on the badcase fetch: pipeline 13.4s → 5.0s, end-to-end 21–23s → 13–15s, same winning strategy and complete content; over a 63-page benchmark sample (where the gate fires on none of them) outputs are byte-identical except one additional label line on the badcase itself.
41
43
 
42
44
  **Non-content scripting subtrees are removed in the live DOM before the document is read back — output-neutral by construction.** The denoise pipeline (and the tool layer's own renderer on the raw-HTML path) re-parses `page.content()` with jsdom, and a script's *text* is the one thing that round-trip cannot trust: a page's runtime JS may rewrite a data script's text with a literal `</script>` embedded — measured on techcommunity.microsoft.com (Khoros/Aurora), whose client writes its whole Apollo cache back into `script#__NEXT_DATA__` via `JSON.stringify`, un-escaping the wire's 712 `\u003c` sequences into raw `<` — and browsers serialize script text verbatim, so the flattened HTML string ends that script at the *first* embedded end tag and parses the remaining ~270KB of JSON *as markup*. The phantom elements that survive (unknown tags like `<id>`, anchors full of CSS and i18n strings) are exactly what Readability's scorer reads as article mass (measured: 126 points vs the real body's 45), so the badcase fetch of the RDP 10 AVC/H.264 article returned CSS rules and `CachedAsset` strings instead of the post. No string-level patch can disambiguate that serialization after the fact — only the browser's own parse holds the document's true structure — so ordinary `script` and `template` subtrees are stripped inside the live page (`page.evaluate`, best-effort) before `page.content()`. This changes nothing the pipeline ever returned: Readability deletes script/noscript nodes before extraction and scoring anyway, DOMPurify's allowlist drops script, and the tool layer's renderer removes SCRIPT/TEMPLATE wholesale — differential A/B over a real-page corpus (Wikipedia, MDN, Hacker News, the badcase page's wire HTML, plus noscript/JSON-LD/template edge cases) came back **byte-identical on every page**. Two deliberate carve-outs keep even the indirect consumers intact: `script[type="application/ld+json"]` is *defused* rather than deleted (`</` rewritten to `<\/` — still valid JSON, so Readability's metadata pass reads the same object and keeps the site-suffix-free article title, e.g. "Remote Desktop Protocol" instead of "… - Wikipedia"), and `noscript` is kept entirely because Readability's `_unwrapNoscriptImages` salvages the real image out of lazy-load `<noscript><img …>` wrappers. The strip also applies with `denoise` off, keeps the raw HTML serialization-safe for the tool layer's renderer, and shrinks what the pipeline parses (~70% on the badcase page).
43
45
 
package/README.zh-CN.md CHANGED
@@ -32,12 +32,14 @@ web_fetch (tool-web)
32
32
  ├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
33
33
  ├─ cdp: connectOverCDP(endpoint)
34
34
  ├─ PDF(inline / attachment / 嗅探出的 octet-stream)→ 字节(导航响应体,否则 request context 重新拉取)→ pdf.js → 版面启发式 → Markdown
35
- ├─ page.goto → 等待稳定(networkidle,尽力而为)→ 深度抓取滚动遍历(虚拟化信息流;有还原时直接返回其合并且已剥离的结果)→ 否则在活 DOM 中剥离脚本子树 → page.content()
35
+ ├─ page.goto → 等待稳定(load 基线 + 有界网络静默窗,陈旧在途请求不再阻塞——尽力而为;不再用 Playwright 的 networkidle:worker 脚本与跨域认证请求在页面会话上永不完成,严格空闲在现代 SPA 页面上不可达)→ 深度抓取滚动遍历(虚拟化信息流;有还原时直接返回其合并且已剥离的结果)→ 否则在活 DOM 中剥离脚本子树 → page.content()
36
36
  ├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → 尾部链接密集块修剪 → DOMPurify → Turndown(GFM)
37
37
  └─ Markdown(关闭降噪时返回原始 HTML)
38
38
  ```
39
39
 
40
- **非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;PDF 下载会被提取为 Markdown(见 [PDF 链接](#pdf-链接)),其他不可解码的下载(ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;传输错误与取消也依旧按错误处理。
40
+ **非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;PDF 下载会被提取为 Markdown(见 [PDF 链接](#pdf-链接)),其他不可解码的下载(ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;持续性传输错误与取消依旧按错误处理——但*瞬时*传输抖动会先重试:`net::ERR_NETWORK_CHANGED`(Chromium 的 NetworkChangeNotifier 因宿主机网络配置在导航途中变化——路由/DNS/VPN/Docker 内嵌 DNS 改写——而掐灭了在途请求;实测:某会话抓取 source.android.com 一次即如此失败,同一 URL 数秒后从同一宿主机干净抓回)及其同族错误码(`ERR_CONNECTION_RESET`、`ERR_NAME_NOT_RESOLVED` 等)会在同一页面上带短退避地重导航至多两次,且受剩余抓取预算约束;`ERR_ABORTED` 与 Playwright 超时错误从不重试(它们的形态是确定性的,已由上述各恢复逻辑分别接管)。
41
+
42
+ **Web Component 应用外壳跳过它注定赢不了的 Readability 家族,大页面也不再反复重解析同一份 DOM。** 降噪管线是同步的且以 jsdom 为主,重页面会在浏览器已拿到完整文档之后把全部开销变成"死等"——实测坏例(`aistudio.google.com/docs/computer-use`,经 CDP 抓取进用户的真实浏览器):页面 ~7 秒渲染完整,之后管线又在 1.1MB 的捕获上花掉 13.4 秒,其中 ~6.5 秒是三次 Readability、~2.9 秒是八次 DOMPurify 重解析。而这些开销里有相当一部分产出的是*可证明的死候选*:该页是 Angular 应用,整个 DOM 都在 `<ms-app>` 之下,而 DOMPurify 的白名单不含带连字符的自定义元素标签,配合 `KEEP_CONTENT: false`,凡携带该包装的提取物净化后只剩空壳(Readability 的选中项:词数为零的根;整文档候选:约 4.5k 词只剩 39 个)。因此当 body 可见文本有 ≥90% 位于自定义元素子树内时(`customWrappedTextShare`),候选生成直接跳过 Readability 家族(`art`/`artu`/`artx`)并在 facts 中标记(`readabilitySkipped`);选择器会被注入死选中项本应测得的空 `art` 参考统计,存活候选的特征因此不变,由 DOM 兜底候选、noscript 路径和块分类器——选择器在这种页面上本来就会选中的那些——承担提取(标题回退到 `og:title`/`<title>`)。另外,生成器本来就持有 DOM 节点的候选(`doc`、`docc`、`main`、`ns_doc`)改为按节点净化(输出逐字节一致,整文档候选实测 555ms → 304ms),不再重解析其序列化文本;Readability 文章对覆盖门限与尾部修剪只解析一次;推荐位名称识别先用字面子串短路再做逐 token 切分(坏例页上省 ~360ms)。坏例页的净效果:管线 13.4s → 5.0s,端到端 21–23s → 13–15s,胜出策略与内容不变;在 63 页的基准输出对比样本上(门控一页都未触发)输出逐字节一致,仅坏例页本身多保留了一行标签文本。
41
43
 
42
44
  **回读文档前,先在活 DOM 中剥离非内容脚本子树 —— 构造上即输出中性。** 降噪管线(以及关闭降噪时工具层自己的渲染器)会用 jsdom 重新解析 `page.content()`,而脚本的*文本内容*是这条往返链路中唯一无法信任的东西:页面自身的运行时 JS 可能把内嵌 `</script>` 的文本写回某个数据脚本的 textContent —— 实测于 techcommunity.microsoft.com(Khoros/Aurora 平台),其客户端把整个 Apollo 缓存经 `JSON.stringify` 写回 `script#__NEXT_DATA__`,把线上响应里 712 处 `\u003c` 转义还原成了裸 `<` —— 而浏览器序列化脚本文本时原样输出,于是这份扁平化的 HTML 字符串会在*第一个*内嵌结束标签处终止脚本,把余下约 270KB 的 JSON *当作标记语言*解析。解析出的幽灵元素(`<id>` 之类的未知标签、塞满 CSS 与 i18n 字符串的锚点)恰好是 Readability 评分器眼里的"正文质量"(实测 126 分对真实正文的 45 分),于是抓取 RDP 10 AVC/H.264 一文时返回的是 CSS 规则和 `CachedAsset` 字符串而不是文章本体。事后再对字符串做补丁无法消除这种歧义 —— 只有浏览器自己的解析持有文档的真实结构 —— 因此普通的 `script` 与 `template` 子树在 `page.content()` 之前先在活页面里剥掉(`page.evaluate`,尽力而为)。这不会改变管线以往的任何输出:Readability 本就在抽取与评分前删除 script/noscript 节点,DOMPurify 的白名单本就丢弃 script,工具层渲染器也整体移除 SCRIPT/TEMPLATE —— 在真实页面语料(Wikipedia、MDN、Hacker News、坏例页 wire 版,加上 noscript/JSON-LD/template 边界用例)上的差分 A/B **每一页都字节级一致**。两个刻意的保留让间接消费方也不受损:`script[type="application/ld+json"]` 只做*消险*而非删除(`</` 改写为 `<\/` —— 仍是合法 JSON,Readability 的元数据读取解析到同一对象,文章标题保持无站点后缀的干净形态,如 "Remote Desktop Protocol" 而非 "… - Wikipedia");`noscript` 完整保留,因为 Readability 的 `_unwrapNoscriptImages` 会从懒加载的 `<noscript><img …>` 包裹中抢救出真实图片。该剥离在关闭降噪时同样生效,让返回给工具层渲染器的原始 HTML 也是序列化安全的,并把管线要解析的体积压缩约 70%(以该坏例页计)。
43
45
 
package/lib/client.js CHANGED
@@ -413,12 +413,12 @@ window.__ModuleLoader__.load({
413
413
  document.head.appendChild(tag);
414
414
  }
415
415
  var PluginCard_module_css_default = {
416
+ "footer": "UM8-Rq_footer",
416
417
  "save": "UM8-Rq_save",
417
- "discard": "UM8-Rq_discard",
418
418
  "readOnly": "UM8-Rq_readOnly",
419
419
  "form": "UM8-Rq_form",
420
- "footer": "UM8-Rq_footer",
421
- "failed": "UM8-Rq_failed"
420
+ "failed": "UM8-Rq_failed",
421
+ "discard": "UM8-Rq_discard"
422
422
  };
423
423
  //#endregion
424
424
  //#region src/client/PluginCard.tsx
@@ -478,27 +478,27 @@ window.__ModuleLoader__.load({
478
478
  document.head.appendChild(tag);
479
479
  }
480
480
  var fields_module_css_default = {
481
- "head": "CFSaVG_head",
482
- "inputInvalid": "CFSaVG_inputInvalid",
483
- "label": "CFSaVG_label",
484
- "radioPick": "CFSaVG_radioPick",
485
- "radioHint": "CFSaVG_radioHint",
486
- "radioText": "CFSaVG_radioText",
481
+ "badges": "CFSaVG_badges",
482
+ "reset": "CFSaVG_reset",
483
+ "checkboxRow": "CFSaVG_checkboxRow",
484
+ "badge": "CFSaVG_badge",
487
485
  "radioOptionPlain": "CFSaVG_radioOptionPlain",
488
- "fieldEmbedded": "CFSaVG_fieldEmbedded",
489
- "radioGroup": "CFSaVG_radioGroup",
486
+ "field": "CFSaVG_field",
487
+ "input": "CFSaVG_input",
490
488
  "radioOption": "CFSaVG_radioOption",
491
- "reset": "CFSaVG_reset",
489
+ "radioPick": "CFSaVG_radioPick",
490
+ "inputInvalid": "CFSaVG_inputInvalid",
491
+ "radioLabel": "CFSaVG_radioLabel",
492
+ "fieldEmbedded": "CFSaVG_fieldEmbedded",
492
493
  "invalid": "CFSaVG_invalid",
493
- "field": "CFSaVG_field",
494
+ "radioContent": "CFSaVG_radioContent",
495
+ "radioGroup": "CFSaVG_radioGroup",
496
+ "label": "CFSaVG_label",
494
497
  "hint": "CFSaVG_hint",
495
- "badge": "CFSaVG_badge",
496
- "badges": "CFSaVG_badges",
498
+ "radioHint": "CFSaVG_radioHint",
497
499
  "hintIndented": "CFSaVG_hintIndented",
498
- "radioLabel": "CFSaVG_radioLabel",
499
- "radioContent": "CFSaVG_radioContent",
500
- "input": "CFSaVG_input",
501
- "checkboxRow": "CFSaVG_checkboxRow"
500
+ "head": "CFSaVG_head",
501
+ "radioText": "CFSaVG_radioText"
502
502
  };
503
503
  //#endregion
504
504
  //#region src/client/fields.tsx