dsh-web-fetch-playwright 0.2.15 → 0.2.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/README.zh-CN.md +2 -2
- package/lib/client.js +17 -17
- package/lib/index.js +143 -21
- package/lib/types/challenge.d.ts +23 -9
- package/lib/types/markdown.d.ts +18 -10
- package/package.json +1 -1
- package/src/challenge.ts +36 -11
- package/src/markdown.ts +127 -16
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
|
|
|
9
9
|
## Features
|
|
10
10
|
|
|
11
11
|
- **Real browser rendering** — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
|
|
12
|
-
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `` so they cannot flood the body.
|
|
12
|
+
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer. Inline `data:` images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as `` so they cannot flood the body. On list-shaped pages (portals, catalogs) where the content *is* the links, Readability's link-density scoring picks a noise fragment — a coverage gate therefore distrusts an extraction under 250 characters or covering under 10% of the page's visible text (measured on the 163.com homepage: 96 characters of footer licensing won while 320 headlines scored near zero), and the pipeline falls back to the sanitized whole document, which on such pages keeps effectively all list content.
|
|
13
13
|
- **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
|
|
14
14
|
- **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
|
|
15
15
|
- **Shared profile or reusable isolation (CDP)** — each fetch opens and closes its own tab; session isolation uses a dedicated CDP connection per DSH session, while global/profile modes share one connection. By default it uses the browser's existing default context and logins (`connectOverCDP` → `browser.contexts()[0]`, like CLI CDP attach with isolation disabled, not `playwright-cli open`). Session isolation creates an in-memory context on demand and shares it only while same-session fetches or other tabs keep it in use; global isolation retains one fixed isolated context across all sessions within the same plugin instance. Local launches still close their browser per fetch.
|
|
@@ -30,7 +30,7 @@ web_fetch (tool-web)
|
|
|
30
30
|
├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
|
|
31
31
|
├─ cdp: connectOverCDP(endpoint)
|
|
32
32
|
├─ page.goto → settle (networkidle, best-effort) → strip scripting subtrees in the live DOM → page.content()
|
|
33
|
-
├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
|
|
33
|
+
├─ denoise: jsdom → elide data-URI images → Readability → coverage gate → DOMPurify → Turndown(GFM)
|
|
34
34
|
└─ Markdown (or raw HTML when denoise is off)
|
|
35
35
|
```
|
|
36
36
|
|
package/README.zh-CN.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
## 特性
|
|
10
10
|
|
|
11
11
|
- **真实浏览器渲染** —— 以用户视角加载页面,SPA 客户端渲染内容也能抓到,而非只有原始 HTML。
|
|
12
|
-
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 ``,避免 base64
|
|
12
|
+
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。内联 `data:` 图片(Docusaurus 等构建工具会把截图以 base64 内嵌进 HTML)会被替换为带大小的占位符,如 ``,避免 base64 字符流刷屏。对列表型页面(门户首页、商品目录——正文本身就是链接),Readability 的链接密度打分会误选出噪声碎片(实测 163.com 首页:96 字符的页脚备案块胜出,320 条新闻标题得分趋零),因此抽取结果需通过**覆盖率置信门**——不足 250 字符、或占页面可见文本不足 10% 即不予采信,转而回退到整页消毒转换,此类页面上可基本完整保留列表内容。
|
|
13
13
|
- **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
|
|
14
14
|
- **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
|
|
15
15
|
- **共享 profile 或复用隔离上下文(CDP)** —— 每次抓取通过共享 CDP 连接创建自己的标签页,用完即关。默认复用浏览器已有默认 context 与登录态(`connectOverCDP` → `browser.contexts()[0]`,类似关闭隔离后的 CLI CDP attach,而不是 `playwright-cli open`)。会话隔离按需创建独立的内存 context,仅在同一会话的抓取或其他标签页仍在使用时复用;全局隔离则在同一插件实例内跨会话保留并复用一个固定的隔离 context。本地后端仍每次抓取启动并关闭浏览器。
|
|
@@ -30,7 +30,7 @@ web_fetch (tool-web)
|
|
|
30
30
|
├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
|
|
31
31
|
├─ cdp: connectOverCDP(endpoint)
|
|
32
32
|
├─ page.goto → 等待稳定(networkidle,尽力而为)→ 在活 DOM 中剥离脚本子树 → page.content()
|
|
33
|
-
├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → DOMPurify → Turndown(GFM)
|
|
33
|
+
├─ 降噪:jsdom → 内联 data: 图片改占位符 → Readability → 覆盖率置信门 → DOMPurify → Turndown(GFM)
|
|
34
34
|
└─ Markdown(关闭降噪时返回原始 HTML)
|
|
35
35
|
```
|
|
36
36
|
|
package/lib/client.js
CHANGED
|
@@ -415,12 +415,12 @@ window.__ModuleLoader__.load({
|
|
|
415
415
|
document.head.appendChild(tag);
|
|
416
416
|
}
|
|
417
417
|
var PluginCard_module_css_default = {
|
|
418
|
-
"
|
|
419
|
-
"form": "UM8-Rq_form",
|
|
420
|
-
"failed": "UM8-Rq_failed",
|
|
418
|
+
"discard": "UM8-Rq_discard",
|
|
421
419
|
"save": "UM8-Rq_save",
|
|
422
420
|
"readOnly": "UM8-Rq_readOnly",
|
|
423
|
-
"
|
|
421
|
+
"form": "UM8-Rq_form",
|
|
422
|
+
"footer": "UM8-Rq_footer",
|
|
423
|
+
"failed": "UM8-Rq_failed"
|
|
424
424
|
};
|
|
425
425
|
//#endregion
|
|
426
426
|
//#region src/client/PluginCard.tsx
|
|
@@ -480,27 +480,27 @@ window.__ModuleLoader__.load({
|
|
|
480
480
|
document.head.appendChild(tag);
|
|
481
481
|
}
|
|
482
482
|
var fields_module_css_default = {
|
|
483
|
-
"
|
|
484
|
-
"
|
|
485
|
-
"label": "CFSaVG_label",
|
|
483
|
+
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
484
|
+
"badge": "CFSaVG_badge",
|
|
486
485
|
"field": "CFSaVG_field",
|
|
487
|
-
"
|
|
488
|
-
"
|
|
486
|
+
"radioText": "CFSaVG_radioText",
|
|
487
|
+
"radioHint": "CFSaVG_radioHint",
|
|
489
488
|
"radioContent": "CFSaVG_radioContent",
|
|
490
489
|
"head": "CFSaVG_head",
|
|
490
|
+
"radioLabel": "CFSaVG_radioLabel",
|
|
491
491
|
"badges": "CFSaVG_badges",
|
|
492
|
-
"radioText": "CFSaVG_radioText",
|
|
493
|
-
"badge": "CFSaVG_badge",
|
|
494
492
|
"radioOptionPlain": "CFSaVG_radioOptionPlain",
|
|
493
|
+
"invalid": "CFSaVG_invalid",
|
|
494
|
+
"reset": "CFSaVG_reset",
|
|
495
|
+
"checkboxRow": "CFSaVG_checkboxRow",
|
|
496
|
+
"radioGroup": "CFSaVG_radioGroup",
|
|
495
497
|
"radioPick": "CFSaVG_radioPick",
|
|
496
|
-
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
497
|
-
"input": "CFSaVG_input",
|
|
498
|
-
"inputInvalid": "CFSaVG_inputInvalid",
|
|
499
498
|
"radioOption": "CFSaVG_radioOption",
|
|
500
|
-
"
|
|
499
|
+
"label": "CFSaVG_label",
|
|
500
|
+
"input": "CFSaVG_input",
|
|
501
|
+
"hint": "CFSaVG_hint",
|
|
501
502
|
"hintIndented": "CFSaVG_hintIndented",
|
|
502
|
-
"
|
|
503
|
-
"radioLabel": "CFSaVG_radioLabel"
|
|
503
|
+
"inputInvalid": "CFSaVG_inputInvalid"
|
|
504
504
|
};
|
|
505
505
|
//#endregion
|
|
506
506
|
//#region src/client/fields.tsx
|
package/lib/index.js
CHANGED
|
@@ -161,14 +161,22 @@ function normalizeCdpEndpoint(input) {
|
|
|
161
161
|
* 2. Content-level — the interstitial's localized `<title>` family
|
|
162
162
|
* ("Just a moment...", "请稍候…", ...) plus structural markers
|
|
163
163
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
164
|
-
* `
|
|
165
|
-
*
|
|
166
|
-
*
|
|
167
|
-
*
|
|
168
|
-
*
|
|
169
|
-
*
|
|
170
|
-
*
|
|
171
|
-
*
|
|
164
|
+
* `window._cf_chl_opt`). Markers are deliberately structural: a blog
|
|
165
|
+
* post that merely MENTIONS Cloudflare must not classify as a challenge,
|
|
166
|
+
* so the challenge-platform script check reads `<script src>` attributes
|
|
167
|
+
* only (never prose or escaped code samples) and identifies the runtime
|
|
168
|
+
* POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
|
|
169
|
+
* instrumentation Cloudflare injects into every NORMAL page of a
|
|
170
|
+
* protected zone, on several moving mounts, is not a challenge signal.
|
|
171
|
+
* The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
|
|
172
|
+
* marker: sites embed real Turnstile widgets into normal pages
|
|
173
|
+
* (linux.do — a Discourse forum with the Turnstile plugin mounts one on
|
|
174
|
+
* every topic page), so the widget alone cannot distinguish an
|
|
175
|
+
* interstitial from a normal page behind a Cloudflare edge. Every real
|
|
176
|
+
* interstitial carries at least one independent signal beyond its
|
|
177
|
+
* widget — the localized title, the orchestrate runtime script,
|
|
178
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
|
|
179
|
+
* stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
|
|
172
180
|
*
|
|
173
181
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
174
182
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -314,9 +322,21 @@ function isChallengeRuntimeScript(src) {
|
|
|
314
322
|
* The challenge-platform prefix is deliberately absent: script evidence goes
|
|
315
323
|
* through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
|
|
316
324
|
* distinguish a real `src` from prose and runtime from instrumentation.
|
|
325
|
+
*
|
|
326
|
+
* The Turnstile widget id (`id="cf-chl-widget-…"`) is deliberately ABSENT
|
|
327
|
+
* (badcase: linux.do/t/topic/3143588 — a Discourse forum behind Cloudflare
|
|
328
|
+
* whose Turnstile plugin mounts a real widget, id and all, into every normal
|
|
329
|
+
* topic page it serves at HTTP 200). The widget is minted by the same
|
|
330
|
+
* Turnstile runtime whether it renders inside an interstitial or inside a
|
|
331
|
+
* site's own UI, so its presence proves only "this page embeds Turnstile".
|
|
332
|
+
* On a real interstitial the widget always co-occurs with independent proof
|
|
333
|
+
* — the localized title, the orchestrate runtime script,
|
|
334
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding, each sufficient
|
|
335
|
+
* above — so dropping the widget loses no real interstitial while every
|
|
336
|
+
* widget-embedding normal page (linux.do, login portals, contact forms)
|
|
337
|
+
* stops dying as WEB_FETCH_CHALLENGE after a wait that can never clear.
|
|
317
338
|
*/
|
|
318
339
|
const CHALLENGE_HTML_MARKERS = [
|
|
319
|
-
"id=\"cf-chl-widget-",
|
|
320
340
|
"window._cf_chl_opt",
|
|
321
341
|
"id=\"challenge-form\"",
|
|
322
342
|
"id=\"challenge-running\"",
|
|
@@ -410,7 +430,13 @@ function classifyChallengeHtml(html) {
|
|
|
410
430
|
* check uses, so the two can never drift apart. Script evidence follows the
|
|
411
431
|
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
412
432
|
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
413
|
-
* protected zone is ignored.
|
|
433
|
+
* protected zone is ignored. The Turnstile widget selector is likewise
|
|
434
|
+
* absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
|
|
435
|
+
* widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
|
|
436
|
+
* the probe reporting "still challenged" forever, so a challenge that HAD
|
|
437
|
+
* cleared could never be confirmed — the wait burned its whole budget and
|
|
438
|
+
* the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
|
|
439
|
+
* layer's timeout.
|
|
414
440
|
*/
|
|
415
441
|
const CHALLENGE_DOM_PROBE = `(() => {
|
|
416
442
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
@@ -418,7 +444,6 @@ const CHALLENGE_DOM_PROBE = `(() => {
|
|
|
418
444
|
const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
|
|
419
445
|
if (re.test(document.title.trim())) return true
|
|
420
446
|
if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
|
|
421
|
-
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
422
447
|
for (const script of Array.from(document.scripts)) {
|
|
423
448
|
const src = (script.getAttribute('src') || '').toLowerCase()
|
|
424
449
|
if (src.includes(platform) && src.includes(runtime)) return true
|
|
@@ -1064,11 +1089,14 @@ function installBuiltinSlashResolveCompat() {
|
|
|
1064
1089
|
* inline `data:` image payloads are elided to size placeholders (build
|
|
1065
1090
|
* tools inline images as base64, which would otherwise dominate the body);
|
|
1066
1091
|
* Mozilla Readability extracts the article (dropping nav bars, sidebars,
|
|
1067
|
-
* footers, and ad chrome by scoring link density and text mass)
|
|
1068
|
-
*
|
|
1069
|
-
*
|
|
1070
|
-
*
|
|
1071
|
-
*
|
|
1092
|
+
* footers, and ad chrome by scoring link density and text mass) — its result
|
|
1093
|
+
* must then pass a coverage gate that distrusts tiny or sliver-thin
|
|
1094
|
+
* extractions, which on list-shaped pages (portals, catalogs) is exactly
|
|
1095
|
+
* what the link-density penalty leaves behind; DOMPurify sanitizes whatever
|
|
1096
|
+
* HTML remains and forbids the layout tags noise lives in; Turndown with
|
|
1097
|
+
* the GFM plugin converts to markdown using the same style options and
|
|
1098
|
+
* span-safe table rules as the shipped `dsh-tool-web` renderer, so output
|
|
1099
|
+
* is consistent with what `web_fetch` produces elsewhere.
|
|
1072
1100
|
*
|
|
1073
1101
|
* Synchronous for its callers and unit-tested against fixture pages: jsdom
|
|
1074
1102
|
* itself is loaded lazily on the first call — and, inside a dsh profile, only
|
|
@@ -1167,6 +1195,99 @@ function humanSize(bytes) {
|
|
|
1167
1195
|
if (bytes < 1048576) return `${(bytes / 1024).toFixed(1)}KB`;
|
|
1168
1196
|
return `${(bytes / 1048576).toFixed(1)}MB`;
|
|
1169
1197
|
}
|
|
1198
|
+
/**
|
|
1199
|
+
* Tags whose subtrees are noise when measuring how much of a page an
|
|
1200
|
+
* extraction kept: DOMPurify drops these outright (`KEEP_CONTENT: false`),
|
|
1201
|
+
* and its default policy always removes `script`/`style` — so a text walk
|
|
1202
|
+
* that skips them measures what the sanitized document fallback could
|
|
1203
|
+
* actually return, not text that is already destined for deletion. Measuring
|
|
1204
|
+
* the raw body instead would let a huge `<nav>`/`<footer>` mass shrink an
|
|
1205
|
+
* article's honest coverage share and demote good extractions.
|
|
1206
|
+
*/
|
|
1207
|
+
const NOISE_SUBTREE_TAGS = /* @__PURE__ */ new Set([
|
|
1208
|
+
...FORBID_TAGS,
|
|
1209
|
+
"script",
|
|
1210
|
+
"style"
|
|
1211
|
+
]);
|
|
1212
|
+
/**
|
|
1213
|
+
* A page must have at least this much visible (post-noise) text before the
|
|
1214
|
+
* article acceptance gate below has an opinion. Sparse pages — logins,
|
|
1215
|
+
* stubs, `login-like` fixtures — have nowhere for content to hide, so
|
|
1216
|
+
* Readability's pick, however short, is still the best reading of the page
|
|
1217
|
+
* and keeps flowing through the article path exactly as before.
|
|
1218
|
+
*/
|
|
1219
|
+
const GATE_MIN_BODY_TEXT_CHARS = 1e3;
|
|
1220
|
+
/**
|
|
1221
|
+
* Minimum non-whitespace characters a Readability extraction must carry
|
|
1222
|
+
* before it can be trusted as a page's content. Measured on the 163.com
|
|
1223
|
+
* homepage, the scorer's pick was a 96-character footer licensing block —
|
|
1224
|
+
* real articles and even sparse pages clear two-to-three times this bar,
|
|
1225
|
+
* while link-dense portal pages lose whole news columns to it.
|
|
1226
|
+
*/
|
|
1227
|
+
const GATE_MIN_ARTICLE_TEXT_CHARS = 250;
|
|
1228
|
+
/**
|
|
1229
|
+
* Minimum share of a page's visible text an extraction must cover to be
|
|
1230
|
+
* trusted. Measured separation on real pages is wide: portal homepages
|
|
1231
|
+
* (163 0.01, sina 0.03, sohu 0.02) vs article pages (0.28–0.45), so 10%
|
|
1232
|
+
* sits in an empty band with margin on both sides.
|
|
1233
|
+
*/
|
|
1234
|
+
const GATE_MIN_ARTICLE_COVERAGE = .1;
|
|
1235
|
+
/**
|
|
1236
|
+
* Count non-whitespace characters under `root`, skipping noise subtrees
|
|
1237
|
+
* ({@link NOISE_SUBTREE_TAGS}). Whitespace is ignored so CJK pages (no
|
|
1238
|
+
* spaces) and latin pages (space-separated) measure comparable densities,
|
|
1239
|
+
* mirroring how the extraction-quality ratios in the docs were computed.
|
|
1240
|
+
*
|
|
1241
|
+
* @param root - the subtree to measure; `null` counts as nothing.
|
|
1242
|
+
* @returns the number of non-whitespace characters in kept subtrees.
|
|
1243
|
+
*/
|
|
1244
|
+
function visibleTextLength(root) {
|
|
1245
|
+
if (root === null) return 0;
|
|
1246
|
+
let length = 0;
|
|
1247
|
+
const visit = (node) => {
|
|
1248
|
+
if (node.nodeType === 3) {
|
|
1249
|
+
length += (node.nodeValue ?? "").replace(/\s+/g, "").length;
|
|
1250
|
+
return;
|
|
1251
|
+
}
|
|
1252
|
+
if (node.nodeType !== 1 || NOISE_SUBTREE_TAGS.has(node.nodeName.toLowerCase())) return;
|
|
1253
|
+
for (let child = node.firstChild; child !== null; child = child.nextSibling) visit(child);
|
|
1254
|
+
};
|
|
1255
|
+
visit(root);
|
|
1256
|
+
return length;
|
|
1257
|
+
}
|
|
1258
|
+
/**
|
|
1259
|
+
* Whether a Readability extraction is trustworthy as *the* content of the
|
|
1260
|
+
* page it came from — the acceptance gate that keeps list-shaped pages whole.
|
|
1261
|
+
*
|
|
1262
|
+
* Readability is an article extractor: it scores text mass and penalizes
|
|
1263
|
+
* link density, so on a portal/catalog page — where the content *is* the
|
|
1264
|
+
* links — every news column scores near zero, some sparse low-link block (a
|
|
1265
|
+
* footer licensing notice, a province-switcher popup) wins by default, and
|
|
1266
|
+
* Readability's own 500-char quality check cannot fail the parse because its
|
|
1267
|
+
* retry loop eventually returns the longest of its bad attempts. The plugin
|
|
1268
|
+
* used to accept any non-null result, so the 163.com homepage rendered as a
|
|
1269
|
+
* 162-character licensing fragment while 320 headlines were dropped — a
|
|
1270
|
+
* recall loss far worse than any precision the gate could cost.
|
|
1271
|
+
*
|
|
1272
|
+
* The gate therefore distrusts an extraction that is tiny in absolute terms
|
|
1273
|
+
* or covers a sliver of the page's visible text; the pipeline then falls
|
|
1274
|
+
* back to the sanitized whole document, which DOMPurify keeps clean and
|
|
1275
|
+
* which on such pages retains effectively all list content (measured 0%
|
|
1276
|
+
* text loss on the 163.com homepage). Only text-rich pages are gated
|
|
1277
|
+
* ({@link GATE_MIN_BODY_TEXT_CHARS}); below that, behavior is unchanged.
|
|
1278
|
+
*
|
|
1279
|
+
* @param articleHtml - the `article.content` HTML Readability produced.
|
|
1280
|
+
* @param document - the live document the article was extracted from.
|
|
1281
|
+
* @returns whether the article should flow through the article path.
|
|
1282
|
+
*/
|
|
1283
|
+
function articleCoversPage(articleHtml, document) {
|
|
1284
|
+
const bodyText = visibleTextLength(document.body);
|
|
1285
|
+
if (bodyText < GATE_MIN_BODY_TEXT_CHARS) return true;
|
|
1286
|
+
const container = document.createElement("div");
|
|
1287
|
+
container.innerHTML = articleHtml;
|
|
1288
|
+
const articleText = visibleTextLength(container);
|
|
1289
|
+
return articleText >= GATE_MIN_ARTICLE_TEXT_CHARS && articleText / bodyText >= GATE_MIN_ARTICLE_COVERAGE;
|
|
1290
|
+
}
|
|
1170
1291
|
/** The shared converter: same style options as `dsh-tool-web`'s renderer. */
|
|
1171
1292
|
const turndown = new TurndownService({
|
|
1172
1293
|
headingStyle: "atx",
|
|
@@ -1217,10 +1338,11 @@ turndown.addRule("tableRowWithoutSpanExpansion", {
|
|
|
1217
1338
|
/**
|
|
1218
1339
|
* Convert one rendered HTML document to denoised markdown.
|
|
1219
1340
|
*
|
|
1220
|
-
* Readability failure (non-article pages
|
|
1221
|
-
*
|
|
1222
|
-
*
|
|
1223
|
-
*
|
|
1341
|
+
* Readability failure (non-article pages, or extractions the coverage gate
|
|
1342
|
+
* {@link articleCoversPage} distrusts) degrades to converting the sanitized
|
|
1343
|
+
* whole document — layout tags are still forbidden, so the fallback stays
|
|
1344
|
+
* cleaner than raw turndown, and a degraded page beats an error for a body
|
|
1345
|
+
* the browser already rendered.
|
|
1224
1346
|
*
|
|
1225
1347
|
* @param html - the rendered page HTML (`page.content()`).
|
|
1226
1348
|
* @param url - the page URL, used to resolve relative links during parsing.
|
|
@@ -1240,7 +1362,7 @@ function htmlToMarkdown(html, url) {
|
|
|
1240
1362
|
try {
|
|
1241
1363
|
const cloned = document.cloneNode(true);
|
|
1242
1364
|
const article = new Readability(cloned).parse();
|
|
1243
|
-
if (article !== null && typeof article.content === "string" && article.content !== "") {
|
|
1365
|
+
if (article !== null && typeof article.content === "string" && article.content !== "" && articleCoversPage(article.content, document)) {
|
|
1244
1366
|
source = article.content;
|
|
1245
1367
|
title = article.title ?? void 0;
|
|
1246
1368
|
}
|
package/lib/types/challenge.d.ts
CHANGED
|
@@ -14,14 +14,22 @@
|
|
|
14
14
|
* 2. Content-level — the interstitial's localized `<title>` family
|
|
15
15
|
* ("Just a moment...", "请稍候…", ...) plus structural markers
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
|
-
* `
|
|
18
|
-
*
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
17
|
+
* `window._cf_chl_opt`). Markers are deliberately structural: a blog
|
|
18
|
+
* post that merely MENTIONS Cloudflare must not classify as a challenge,
|
|
19
|
+
* so the challenge-platform script check reads `<script src>` attributes
|
|
20
|
+
* only (never prose or escaped code samples) and identifies the runtime
|
|
21
|
+
* POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
|
|
22
|
+
* instrumentation Cloudflare injects into every NORMAL page of a
|
|
23
|
+
* protected zone, on several moving mounts, is not a challenge signal.
|
|
24
|
+
* The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
|
|
25
|
+
* marker: sites embed real Turnstile widgets into normal pages
|
|
26
|
+
* (linux.do — a Discourse forum with the Turnstile plugin mounts one on
|
|
27
|
+
* every topic page), so the widget alone cannot distinguish an
|
|
28
|
+
* interstitial from a normal page behind a Cloudflare edge. Every real
|
|
29
|
+
* interstitial carries at least one independent signal beyond its
|
|
30
|
+
* widget — the localized title, the orchestrate runtime script,
|
|
31
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
|
|
32
|
+
* stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
|
|
25
33
|
*
|
|
26
34
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
27
35
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -86,6 +94,12 @@ export declare function classifyChallengeHtml(html: string): ChallengeVerdict;
|
|
|
86
94
|
* check uses, so the two can never drift apart. Script evidence follows the
|
|
87
95
|
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
88
96
|
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
89
|
-
* protected zone is ignored.
|
|
97
|
+
* protected zone is ignored. The Turnstile widget selector is likewise
|
|
98
|
+
* absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
|
|
99
|
+
* widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
|
|
100
|
+
* the probe reporting "still challenged" forever, so a challenge that HAD
|
|
101
|
+
* cleared could never be confirmed — the wait burned its whole budget and
|
|
102
|
+
* the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
|
|
103
|
+
* layer's timeout.
|
|
90
104
|
*/
|
|
91
105
|
export declare const CHALLENGE_DOM_PROBE: string;
|
package/lib/types/markdown.d.ts
CHANGED
|
@@ -5,11 +5,14 @@
|
|
|
5
5
|
* inline `data:` image payloads are elided to size placeholders (build
|
|
6
6
|
* tools inline images as base64, which would otherwise dominate the body);
|
|
7
7
|
* Mozilla Readability extracts the article (dropping nav bars, sidebars,
|
|
8
|
-
* footers, and ad chrome by scoring link density and text mass)
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
8
|
+
* footers, and ad chrome by scoring link density and text mass) — its result
|
|
9
|
+
* must then pass a coverage gate that distrusts tiny or sliver-thin
|
|
10
|
+
* extractions, which on list-shaped pages (portals, catalogs) is exactly
|
|
11
|
+
* what the link-density penalty leaves behind; DOMPurify sanitizes whatever
|
|
12
|
+
* HTML remains and forbids the layout tags noise lives in; Turndown with
|
|
13
|
+
* the GFM plugin converts to markdown using the same style options and
|
|
14
|
+
* span-safe table rules as the shipped `dsh-tool-web` renderer, so output
|
|
15
|
+
* is consistent with what `web_fetch` produces elsewhere.
|
|
13
16
|
*
|
|
14
17
|
* Synchronous for its callers and unit-tested against fixture pages: jsdom
|
|
15
18
|
* itself is loaded lazily on the first call — and, inside a dsh profile, only
|
|
@@ -24,16 +27,21 @@ export type DenoiseMode = 'article' | 'document';
|
|
|
24
27
|
export interface DenoiseResult {
|
|
25
28
|
/** The markdown body (title, when found, already prepended). */
|
|
26
29
|
markdown: string;
|
|
27
|
-
/**
|
|
30
|
+
/**
|
|
31
|
+
* `article` = Readability extraction that passed the coverage gate;
|
|
32
|
+
* `document` = whole-document fallback (Readability returned nothing, or
|
|
33
|
+
* the gate distrusted what it returned — list-shaped pages).
|
|
34
|
+
*/
|
|
28
35
|
mode: DenoiseMode;
|
|
29
36
|
}
|
|
30
37
|
/**
|
|
31
38
|
* Convert one rendered HTML document to denoised markdown.
|
|
32
39
|
*
|
|
33
|
-
* Readability failure (non-article pages
|
|
34
|
-
*
|
|
35
|
-
*
|
|
36
|
-
*
|
|
40
|
+
* Readability failure (non-article pages, or extractions the coverage gate
|
|
41
|
+
* {@link articleCoversPage} distrusts) degrades to converting the sanitized
|
|
42
|
+
* whole document — layout tags are still forbidden, so the fallback stays
|
|
43
|
+
* cleaner than raw turndown, and a degraded page beats an error for a body
|
|
44
|
+
* the browser already rendered.
|
|
37
45
|
*
|
|
38
46
|
* @param html - the rendered page HTML (`page.content()`).
|
|
39
47
|
* @param url - the page URL, used to resolve relative links during parsing.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-web-fetch-playwright",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.16",
|
|
4
4
|
"description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/src/challenge.ts
CHANGED
|
@@ -14,14 +14,22 @@
|
|
|
14
14
|
* 2. Content-level — the interstitial's localized `<title>` family
|
|
15
15
|
* ("Just a moment...", "请稍候…", ...) plus structural markers
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
|
-
* `
|
|
18
|
-
*
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
17
|
+
* `window._cf_chl_opt`). Markers are deliberately structural: a blog
|
|
18
|
+
* post that merely MENTIONS Cloudflare must not classify as a challenge,
|
|
19
|
+
* so the challenge-platform script check reads `<script src>` attributes
|
|
20
|
+
* only (never prose or escaped code samples) and identifies the runtime
|
|
21
|
+
* POSITIVELY by its `/orchestrate/` mount — the JSD/Precursor
|
|
22
|
+
* instrumentation Cloudflare injects into every NORMAL page of a
|
|
23
|
+
* protected zone, on several moving mounts, is not a challenge signal.
|
|
24
|
+
* The Turnstile widget id (`cf-chl-widget-…`) is deliberately NOT a
|
|
25
|
+
* marker: sites embed real Turnstile widgets into normal pages
|
|
26
|
+
* (linux.do — a Discourse forum with the Turnstile plugin mounts one on
|
|
27
|
+
* every topic page), so the widget alone cannot distinguish an
|
|
28
|
+
* interstitial from a normal page behind a Cloudflare edge. Every real
|
|
29
|
+
* interstitial carries at least one independent signal beyond its
|
|
30
|
+
* widget — the localized title, the orchestrate runtime script,
|
|
31
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding — and those
|
|
32
|
+
* stay sufficient on their own (see {@link CHALLENGE_HTML_MARKERS}).
|
|
25
33
|
*
|
|
26
34
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
27
35
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -176,9 +184,21 @@ function isChallengeRuntimeScript(src: string): boolean {
|
|
|
176
184
|
* The challenge-platform prefix is deliberately absent: script evidence goes
|
|
177
185
|
* through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
|
|
178
186
|
* distinguish a real `src` from prose and runtime from instrumentation.
|
|
187
|
+
*
|
|
188
|
+
* The Turnstile widget id (`id="cf-chl-widget-…"`) is deliberately ABSENT
|
|
189
|
+
* (badcase: linux.do/t/topic/3143588 — a Discourse forum behind Cloudflare
|
|
190
|
+
* whose Turnstile plugin mounts a real widget, id and all, into every normal
|
|
191
|
+
* topic page it serves at HTTP 200). The widget is minted by the same
|
|
192
|
+
* Turnstile runtime whether it renders inside an interstitial or inside a
|
|
193
|
+
* site's own UI, so its presence proves only "this page embeds Turnstile".
|
|
194
|
+
* On a real interstitial the widget always co-occurs with independent proof
|
|
195
|
+
* — the localized title, the orchestrate runtime script,
|
|
196
|
+
* `window._cf_chl_opt`, or the `#challenge-*` scaffolding, each sufficient
|
|
197
|
+
* above — so dropping the widget loses no real interstitial while every
|
|
198
|
+
* widget-embedding normal page (linux.do, login portals, contact forms)
|
|
199
|
+
* stops dying as WEB_FETCH_CHALLENGE after a wait that can never clear.
|
|
179
200
|
*/
|
|
180
201
|
const CHALLENGE_HTML_MARKERS = [
|
|
181
|
-
'id="cf-chl-widget-',
|
|
182
202
|
'window._cf_chl_opt',
|
|
183
203
|
'id="challenge-form"',
|
|
184
204
|
'id="challenge-running"',
|
|
@@ -295,7 +315,13 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
|
|
|
295
315
|
* check uses, so the two can never drift apart. Script evidence follows the
|
|
296
316
|
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
297
317
|
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
298
|
-
* protected zone is ignored.
|
|
318
|
+
* protected zone is ignored. The Turnstile widget selector is likewise
|
|
319
|
+
* absent for the same reason {@link CHALLENGE_HTML_MARKERS} omits it: a
|
|
320
|
+
* widget embedded in a NORMAL page (linux.do's Turnstile plugin) would keep
|
|
321
|
+
* the probe reporting "still challenged" forever, so a challenge that HAD
|
|
322
|
+
* cleared could never be confirmed — the wait burned its whole budget and
|
|
323
|
+
* the fetch died as WEB_FETCH_CHALLENGE (last status 200) or the tool
|
|
324
|
+
* layer's timeout.
|
|
299
325
|
*/
|
|
300
326
|
export const CHALLENGE_DOM_PROBE = `(() => {
|
|
301
327
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
@@ -303,7 +329,6 @@ export const CHALLENGE_DOM_PROBE = `(() => {
|
|
|
303
329
|
const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
|
|
304
330
|
if (re.test(document.title.trim())) return true
|
|
305
331
|
if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
|
|
306
|
-
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
307
332
|
for (const script of Array.from(document.scripts)) {
|
|
308
333
|
const src = (script.getAttribute('src') || '').toLowerCase()
|
|
309
334
|
if (src.includes(platform) && src.includes(runtime)) return true
|
package/src/markdown.ts
CHANGED
|
@@ -5,11 +5,14 @@
|
|
|
5
5
|
* inline `data:` image payloads are elided to size placeholders (build
|
|
6
6
|
* tools inline images as base64, which would otherwise dominate the body);
|
|
7
7
|
* Mozilla Readability extracts the article (dropping nav bars, sidebars,
|
|
8
|
-
* footers, and ad chrome by scoring link density and text mass)
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
8
|
+
* footers, and ad chrome by scoring link density and text mass) — its result
|
|
9
|
+
* must then pass a coverage gate that distrusts tiny or sliver-thin
|
|
10
|
+
* extractions, which on list-shaped pages (portals, catalogs) is exactly
|
|
11
|
+
* what the link-density penalty leaves behind; DOMPurify sanitizes whatever
|
|
12
|
+
* HTML remains and forbids the layout tags noise lives in; Turndown with
|
|
13
|
+
* the GFM plugin converts to markdown using the same style options and
|
|
14
|
+
* span-safe table rules as the shipped `dsh-tool-web` renderer, so output
|
|
15
|
+
* is consistent with what `web_fetch` produces elsewhere.
|
|
13
16
|
*
|
|
14
17
|
* Synchronous for its callers and unit-tested against fixture pages: jsdom
|
|
15
18
|
* itself is loaded lazily on the first call — and, inside a dsh profile, only
|
|
@@ -108,6 +111,105 @@ function humanSize(bytes: number): string {
|
|
|
108
111
|
return `${(bytes / 1_048_576).toFixed(1)}MB`
|
|
109
112
|
}
|
|
110
113
|
|
|
114
|
+
/**
|
|
115
|
+
* Tags whose subtrees are noise when measuring how much of a page an
|
|
116
|
+
* extraction kept: DOMPurify drops these outright (`KEEP_CONTENT: false`),
|
|
117
|
+
* and its default policy always removes `script`/`style` — so a text walk
|
|
118
|
+
* that skips them measures what the sanitized document fallback could
|
|
119
|
+
* actually return, not text that is already destined for deletion. Measuring
|
|
120
|
+
* the raw body instead would let a huge `<nav>`/`<footer>` mass shrink an
|
|
121
|
+
* article's honest coverage share and demote good extractions.
|
|
122
|
+
*/
|
|
123
|
+
const NOISE_SUBTREE_TAGS = new Set([...FORBID_TAGS, 'script', 'style'])
|
|
124
|
+
|
|
125
|
+
/**
|
|
126
|
+
* A page must have at least this much visible (post-noise) text before the
|
|
127
|
+
* article acceptance gate below has an opinion. Sparse pages — logins,
|
|
128
|
+
* stubs, `login-like` fixtures — have nowhere for content to hide, so
|
|
129
|
+
* Readability's pick, however short, is still the best reading of the page
|
|
130
|
+
* and keeps flowing through the article path exactly as before.
|
|
131
|
+
*/
|
|
132
|
+
const GATE_MIN_BODY_TEXT_CHARS = 1_000
|
|
133
|
+
|
|
134
|
+
/**
|
|
135
|
+
* Minimum non-whitespace characters a Readability extraction must carry
|
|
136
|
+
* before it can be trusted as a page's content. Measured on the 163.com
|
|
137
|
+
* homepage, the scorer's pick was a 96-character footer licensing block —
|
|
138
|
+
* real articles and even sparse pages clear two-to-three times this bar,
|
|
139
|
+
* while link-dense portal pages lose whole news columns to it.
|
|
140
|
+
*/
|
|
141
|
+
const GATE_MIN_ARTICLE_TEXT_CHARS = 250
|
|
142
|
+
|
|
143
|
+
/**
|
|
144
|
+
* Minimum share of a page's visible text an extraction must cover to be
|
|
145
|
+
* trusted. Measured separation on real pages is wide: portal homepages
|
|
146
|
+
* (163 0.01, sina 0.03, sohu 0.02) vs article pages (0.28–0.45), so 10%
|
|
147
|
+
* sits in an empty band with margin on both sides.
|
|
148
|
+
*/
|
|
149
|
+
const GATE_MIN_ARTICLE_COVERAGE = 0.1
|
|
150
|
+
|
|
151
|
+
/**
|
|
152
|
+
* Count non-whitespace characters under `root`, skipping noise subtrees
|
|
153
|
+
* ({@link NOISE_SUBTREE_TAGS}). Whitespace is ignored so CJK pages (no
|
|
154
|
+
* spaces) and latin pages (space-separated) measure comparable densities,
|
|
155
|
+
* mirroring how the extraction-quality ratios in the docs were computed.
|
|
156
|
+
*
|
|
157
|
+
* @param root - the subtree to measure; `null` counts as nothing.
|
|
158
|
+
* @returns the number of non-whitespace characters in kept subtrees.
|
|
159
|
+
*/
|
|
160
|
+
function visibleTextLength(root: Node | null): number {
|
|
161
|
+
if (root === null) return 0
|
|
162
|
+
let length = 0
|
|
163
|
+
const visit = (node: Node): void => {
|
|
164
|
+
// Text nodes contribute their non-whitespace characters (nodeType 3).
|
|
165
|
+
if (node.nodeType === 3) {
|
|
166
|
+
length += (node.nodeValue ?? '').replace(/\s+/g, '').length
|
|
167
|
+
return
|
|
168
|
+
}
|
|
169
|
+
// Only elements branch further (nodeType 1); comments and friends carry
|
|
170
|
+
// no text, and noise subtrees are skipped wholesale.
|
|
171
|
+
if (node.nodeType !== 1 || NOISE_SUBTREE_TAGS.has(node.nodeName.toLowerCase())) return
|
|
172
|
+
for (let child = node.firstChild; child !== null; child = child.nextSibling) visit(child)
|
|
173
|
+
}
|
|
174
|
+
visit(root)
|
|
175
|
+
return length
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
/**
|
|
179
|
+
* Whether a Readability extraction is trustworthy as *the* content of the
|
|
180
|
+
* page it came from — the acceptance gate that keeps list-shaped pages whole.
|
|
181
|
+
*
|
|
182
|
+
* Readability is an article extractor: it scores text mass and penalizes
|
|
183
|
+
* link density, so on a portal/catalog page — where the content *is* the
|
|
184
|
+
* links — every news column scores near zero, some sparse low-link block (a
|
|
185
|
+
* footer licensing notice, a province-switcher popup) wins by default, and
|
|
186
|
+
* Readability's own 500-char quality check cannot fail the parse because its
|
|
187
|
+
* retry loop eventually returns the longest of its bad attempts. The plugin
|
|
188
|
+
* used to accept any non-null result, so the 163.com homepage rendered as a
|
|
189
|
+
* 162-character licensing fragment while 320 headlines were dropped — a
|
|
190
|
+
* recall loss far worse than any precision the gate could cost.
|
|
191
|
+
*
|
|
192
|
+
* The gate therefore distrusts an extraction that is tiny in absolute terms
|
|
193
|
+
* or covers a sliver of the page's visible text; the pipeline then falls
|
|
194
|
+
* back to the sanitized whole document, which DOMPurify keeps clean and
|
|
195
|
+
* which on such pages retains effectively all list content (measured 0%
|
|
196
|
+
* text loss on the 163.com homepage). Only text-rich pages are gated
|
|
197
|
+
* ({@link GATE_MIN_BODY_TEXT_CHARS}); below that, behavior is unchanged.
|
|
198
|
+
*
|
|
199
|
+
* @param articleHtml - the `article.content` HTML Readability produced.
|
|
200
|
+
* @param document - the live document the article was extracted from.
|
|
201
|
+
* @returns whether the article should flow through the article path.
|
|
202
|
+
*/
|
|
203
|
+
function articleCoversPage(articleHtml: string, document: Document): boolean {
|
|
204
|
+
const bodyText = visibleTextLength(document.body)
|
|
205
|
+
if (bodyText < GATE_MIN_BODY_TEXT_CHARS) return true
|
|
206
|
+
const container = document.createElement('div')
|
|
207
|
+
container.innerHTML = articleHtml
|
|
208
|
+
const articleText = visibleTextLength(container)
|
|
209
|
+
return articleText >= GATE_MIN_ARTICLE_TEXT_CHARS
|
|
210
|
+
&& articleText / bodyText >= GATE_MIN_ARTICLE_COVERAGE
|
|
211
|
+
}
|
|
212
|
+
|
|
111
213
|
/** The shared converter: same style options as `dsh-tool-web`'s renderer. */
|
|
112
214
|
const turndown = new TurndownService({
|
|
113
215
|
headingStyle: 'atx',
|
|
@@ -170,17 +272,22 @@ export type DenoiseMode = 'article' | 'document'
|
|
|
170
272
|
export interface DenoiseResult {
|
|
171
273
|
/** The markdown body (title, when found, already prepended). */
|
|
172
274
|
markdown: string
|
|
173
|
-
/**
|
|
275
|
+
/**
|
|
276
|
+
* `article` = Readability extraction that passed the coverage gate;
|
|
277
|
+
* `document` = whole-document fallback (Readability returned nothing, or
|
|
278
|
+
* the gate distrusted what it returned — list-shaped pages).
|
|
279
|
+
*/
|
|
174
280
|
mode: DenoiseMode
|
|
175
281
|
}
|
|
176
282
|
|
|
177
283
|
/**
|
|
178
284
|
* Convert one rendered HTML document to denoised markdown.
|
|
179
285
|
*
|
|
180
|
-
* Readability failure (non-article pages
|
|
181
|
-
*
|
|
182
|
-
*
|
|
183
|
-
*
|
|
286
|
+
* Readability failure (non-article pages, or extractions the coverage gate
|
|
287
|
+
* {@link articleCoversPage} distrusts) degrades to converting the sanitized
|
|
288
|
+
* whole document — layout tags are still forbidden, so the fallback stays
|
|
289
|
+
* cleaner than raw turndown, and a degraded page beats an error for a body
|
|
290
|
+
* the browser already rendered.
|
|
184
291
|
*
|
|
185
292
|
* @param html - the rendered page HTML (`page.content()`).
|
|
186
293
|
* @param url - the page URL, used to resolve relative links during parsing.
|
|
@@ -204,14 +311,18 @@ export function htmlToMarkdown(html: string, url: string): DenoiseResult {
|
|
|
204
311
|
let source: string | null = null
|
|
205
312
|
let title: string | undefined
|
|
206
313
|
try {
|
|
207
|
-
// Ungated: isProbablyReaderable is a conservative hint that
|
|
208
|
-
// sparse-but-real articles (measured on a browser-rendered fixture),
|
|
209
|
-
// the extraction is simply attempted; a null
|
|
210
|
-
// to the sanitized whole document below. parse() mutates,
|
|
211
|
-
// (typed as Document: DOM lib types cloneNode's return as
|
|
314
|
+
// Ungated readability: isProbablyReaderable is a conservative hint that
|
|
315
|
+
// rejects sparse-but-real articles (measured on a browser-rendered fixture),
|
|
316
|
+
// so the extraction is simply attempted; a null, empty, or gate-rejected
|
|
317
|
+
// result falls back to the sanitized whole document below. parse() mutates,
|
|
318
|
+
// hence the clone (typed as Document: DOM lib types cloneNode's return as
|
|
319
|
+
// Node).
|
|
212
320
|
const cloned = document.cloneNode(true) as typeof document
|
|
213
321
|
const article = new Readability(cloned).parse()
|
|
214
|
-
if (
|
|
322
|
+
if (
|
|
323
|
+
article !== null && typeof article.content === 'string' && article.content !== ''
|
|
324
|
+
&& articleCoversPage(article.content, document)
|
|
325
|
+
) {
|
|
215
326
|
source = article.content
|
|
216
327
|
title = article.title ?? undefined
|
|
217
328
|
}
|