dsh-web-fetch-playwright 0.2.5 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -1
- package/README.zh-CN.md +3 -1
- package/lib/client.js +23 -23
- package/lib/index.js +40 -15
- package/lib/types/challenge.d.ts +8 -2
- package/lib/types/index.d.ts +1 -1
- package/lib/types/playwright-resolve.d.ts +3 -1
- package/package.json +14 -13
- package/src/challenge.ts +34 -5
- package/src/index.ts +11 -9
- package/src/playwright-resolve.ts +5 -3
- package/dsh.plugin.json +0 -16
package/README.md
CHANGED
|
@@ -81,6 +81,8 @@ Local backend resolution order:
|
|
|
81
81
|
2. A `playwright` executable on `$PATH` (its package knows that installation's browser registry).
|
|
82
82
|
3. The bundled `playwright-core` — requires `PLAYWRIGHT_BROWSERS_PATH` or browsers in the default cache; otherwise the error suggests `playwright install chromium`.
|
|
83
83
|
|
|
84
|
+
> **Windows note** — `$PATH` is scanned with the platform delimiter (`;`), but npm/pnpm global installs expose `playwright` as `.cmd`/`.ps1` shims whose location does not walk up to the package root, so discovery may still land on step 3 (the bundled core). To drive a specific installation's browser registry, point `playwrightPath` at the `playwright` package directory or a browser executable.
|
|
85
|
+
|
|
84
86
|
CDP mode needs no local browser: the provider holds **one shared connection** for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a tab in the remote browser that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection (the remote browser itself is never closed).
|
|
85
87
|
|
|
86
88
|
### The CDP context modes (share the browser's profile or not)
|
|
@@ -107,7 +109,7 @@ Some strict sites serve a Cloudflare interstitial before the real page. A real b
|
|
|
107
109
|
|
|
108
110
|
With the wait on (default):
|
|
109
111
|
|
|
110
|
-
1. **Detection** — a response carrying `cf-mitigated: challenge` (the documented signal for every challenge page type), or a 403/503 HTML document from a `server: cloudflare` edge, or the localized interstitial itself (title family like "Just a moment…", `请稍候…`, "Минутку…", plus structural markers: `/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements, `cf-chl-widget-` frames, `window._cf_chl_opt`). The content-level markers are the *fallback tier* and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (`server: cloudflare` / `cf-ray`) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
|
|
112
|
+
1. **Detection** — a response carrying `cf-mitigated: challenge` (the documented signal for every challenge page type), or a 403/503 HTML document from a `server: cloudflare` edge, or the localized interstitial itself (title family like "Just a moment…", `请稍候…`, "Минутку…", plus structural markers: `/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements, `cf-chl-widget-` frames, `window._cf_chl_opt`). The content-level markers are the *fallback tier* and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (`server: cloudflare` / `cf-ray`) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Cloudflare Bot Management's passive JavaScript-Detections telemetry (`/cdn-cgi/challenge-platform/scripts/jsd/`, injected into every *normal* page of a protected zone — e.g. openrouter.ai) is neutralized before the prefix scan, so protected 200 pages with real content never misclassify. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
|
|
111
113
|
2. **Bounded wait, same tab and context** — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the *last* main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
|
|
112
114
|
3. **Bounded retry** — when a window runs out, the same tab re-navigates once (default; `challengeRetries`) with whatever clearance cookies the context already holds.
|
|
113
115
|
4. **Clear failure** — `WEB_FETCH_CHALLENGE` (a provider-specific code the web seam's open-string `code` allows) naming the site, the budget spent, and the last challenge status.
|
package/README.zh-CN.md
CHANGED
|
@@ -81,6 +81,8 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
|
|
|
81
81
|
2. `$PATH` 上的 `playwright`(其包自带该安装的浏览器注册表);
|
|
82
82
|
3. 插件内置的 `playwright-core`——需要 `PLAYWRIGHT_BROWSERS_PATH` 或默认缓存里有浏览器,否则报错会提示 `playwright install chromium`。
|
|
83
83
|
|
|
84
|
+
> **Windows 说明** —— `$PATH` 已按平台分隔符(`;`)扫描,但 npm/pnpm 全局安装暴露的 `playwright` 是 `.cmd`/`.ps1` 垫片,从垫片位置向上找不到包根,自动发现可能仍落在第 3 步(内置 core)。要使用指定安装的浏览器注册表,请把 `playwrightPath` 显式指向 `playwright` 包目录或浏览器二进制。
|
|
85
|
+
|
|
84
86
|
CDP 模式不需要本地浏览器:插件在生命周期内对远端浏览器保持**一条共享连接**(连接断开自动重连,地址改动后自动换连),每次抓取只租用远端浏览器里的一个标签页,抓取结束即关闭。因此并发数按"标签页"计,默认也更高(50)。插件卸载时断开共享连接(绝不会关闭远端浏览器本身)。
|
|
85
87
|
|
|
86
88
|
### CDP 上下文模式(是否共享浏览器 profile)
|
|
@@ -107,7 +109,7 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
|
|
|
107
109
|
|
|
108
110
|
开启等待(默认)后的流程:
|
|
109
111
|
|
|
110
|
-
1. **识别** —— 响应带 `cf-mitigated: challenge`(Cloudflare 官方文档注明所有挑战页类型都带此头),或 403/503 且 `server: cloudflare` 的 HTML 文档,或本地化的中间页本身("Just a moment…" / "请稍候…" / "Минутку…" 等 title 家族,以及结构性标记:`/cdn-cgi/challenge-platform/` 脚本、`#challenge-*` 元素、`cf-chl-widget-` 框架、`window._cf_chl_opt`)。内容级标记只是**兜底层**,且仅对"挑战兼容"的响应(403/429/503 或来自 Cloudflare 边缘——`server: cloudflare` / `cf-ray`)运行——因为中间页从不会以普通 200
|
|
112
|
+
1. **识别** —— 响应带 `cf-mitigated: challenge`(Cloudflare 官方文档注明所有挑战页类型都带此头),或 403/503 且 `server: cloudflare` 的 HTML 文档,或本地化的中间页本身("Just a moment…" / "请稍候…" / "Минутку…" 等 title 家族,以及结构性标记:`/cdn-cgi/challenge-platform/` 脚本、`#challenge-*` 元素、`cf-chl-widget-` 框架、`window._cf_chl_opt`)。内容级标记只是**兜底层**,且仅对"挑战兼容"的响应(403/429/503 或来自 Cloudflare 边缘——`server: cloudflare` / `cf-ray`)运行——因为中间页从不会以普通 200 返回,所以正文里引用了挑战文案的普通文章绝不可能被误判。Cloudflare Bot Management 的被动 JavaScript-Detections 遥测(`/cdn-cgi/challenge-platform/scripts/jsd/`,会注入受保护站点的每一个**正常**页面——如 openrouter.ai)在前缀扫描前被显式中和,因此带真实内容的 200 正常页绝不会被误判。硬封锁页("Sorry, you have been blocked")单独分类并立即失败——等待无法解除。
|
|
111
113
|
2. **同标签页、同上下文的有界等待** —— 每 500ms 轮询活 DOM,等浏览器跑完自己的验证;同时跟踪**最后一次主 frame 导航响应**,所以重载进来的真实文档的状态码和响应头才是最终上报的。SPA 式清除(无导航、纯内容替换)由同一个 DOM 探测捕获。
|
|
112
114
|
3. **有界重试** —— 窗口耗尽后,同一标签页默认再导航一次(`challengeRetries`),上下文里已有的通关 cookie 继续生效。
|
|
113
115
|
4. **明确失败** —— 返回独立的 `WEB_FETCH_CHALLENGE` 错误码(web seam 的 `code` 是开放字符串,允许 provider 专属码),消息中写明站点、等待预算与最后一次挑战响应的状态。
|
package/lib/client.js
CHANGED
|
@@ -390,21 +390,21 @@ window.__ModuleLoader__.load({
|
|
|
390
390
|
document.head.appendChild(tag);
|
|
391
391
|
}
|
|
392
392
|
var PluginCard_module_css_default = {
|
|
393
|
-
"footer": "UM8-Rq_footer",
|
|
394
|
-
"pending": "UM8-Rq_pending",
|
|
395
|
-
"discard": "UM8-Rq_discard",
|
|
396
393
|
"cardOpen": "UM8-Rq_cardOpen",
|
|
397
|
-
"
|
|
398
|
-
"card": "UM8-Rq_card",
|
|
399
|
-
"chevronOpen": "UM8-Rq_chevronOpen",
|
|
394
|
+
"discard": "UM8-Rq_discard",
|
|
400
395
|
"body": "UM8-Rq_body",
|
|
401
|
-
"description": "UM8-Rq_description",
|
|
402
396
|
"failed": "UM8-Rq_failed",
|
|
403
|
-
"header": "UM8-Rq_header",
|
|
404
397
|
"name": "UM8-Rq_name",
|
|
398
|
+
"chevronOpen": "UM8-Rq_chevronOpen",
|
|
399
|
+
"headText": "UM8-Rq_headText",
|
|
400
|
+
"card": "UM8-Rq_card",
|
|
401
|
+
"footer": "UM8-Rq_footer",
|
|
405
402
|
"save": "UM8-Rq_save",
|
|
406
403
|
"chevron": "UM8-Rq_chevron",
|
|
407
|
-
"
|
|
404
|
+
"pending": "UM8-Rq_pending",
|
|
405
|
+
"readOnly": "UM8-Rq_readOnly",
|
|
406
|
+
"header": "UM8-Rq_header",
|
|
407
|
+
"description": "UM8-Rq_description"
|
|
408
408
|
};
|
|
409
409
|
//#endregion
|
|
410
410
|
//#region src/client/PluginCard.tsx
|
|
@@ -521,26 +521,26 @@ window.__ModuleLoader__.load({
|
|
|
521
521
|
document.head.appendChild(tag);
|
|
522
522
|
}
|
|
523
523
|
var fields_module_css_default = {
|
|
524
|
+
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
525
|
+
"radioText": "CFSaVG_radioText",
|
|
526
|
+
"radioHint": "CFSaVG_radioHint",
|
|
524
527
|
"radioContent": "CFSaVG_radioContent",
|
|
525
|
-
"
|
|
526
|
-
"radioGroup": "CFSaVG_radioGroup",
|
|
527
|
-
"input": "CFSaVG_input",
|
|
528
|
-
"radioLabel": "CFSaVG_radioLabel",
|
|
528
|
+
"reset": "CFSaVG_reset",
|
|
529
529
|
"radioPick": "CFSaVG_radioPick",
|
|
530
|
+
"checkboxRow": "CFSaVG_checkboxRow",
|
|
531
|
+
"radioLabel": "CFSaVG_radioLabel",
|
|
532
|
+
"radioOption": "CFSaVG_radioOption",
|
|
533
|
+
"input": "CFSaVG_input",
|
|
534
|
+
"inputInvalid": "CFSaVG_inputInvalid",
|
|
535
|
+
"hint": "CFSaVG_hint",
|
|
530
536
|
"badges": "CFSaVG_badges",
|
|
537
|
+
"hintIndented": "CFSaVG_hintIndented",
|
|
538
|
+
"label": "CFSaVG_label",
|
|
531
539
|
"badge": "CFSaVG_badge",
|
|
532
|
-
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
533
|
-
"hint": "CFSaVG_hint",
|
|
534
|
-
"reset": "CFSaVG_reset",
|
|
535
|
-
"field": "CFSaVG_field",
|
|
536
540
|
"head": "CFSaVG_head",
|
|
537
|
-
"
|
|
538
|
-
"label": "CFSaVG_label",
|
|
539
|
-
"radioOption": "CFSaVG_radioOption",
|
|
541
|
+
"field": "CFSaVG_field",
|
|
540
542
|
"invalid": "CFSaVG_invalid",
|
|
541
|
-
"
|
|
542
|
-
"inputInvalid": "CFSaVG_inputInvalid",
|
|
543
|
-
"radioText": "CFSaVG_radioText"
|
|
543
|
+
"radioGroup": "CFSaVG_radioGroup"
|
|
544
544
|
};
|
|
545
545
|
//#endregion
|
|
546
546
|
//#region src/client/fields.tsx
|
package/lib/index.js
CHANGED
|
@@ -1,5 +1,4 @@
|
|
|
1
1
|
import { createRequire } from "node:module";
|
|
2
|
-
import { installSettingsSection, settingsNamespace } from "@deepseek-ai/dsh-settings";
|
|
3
2
|
import z from "@deepseek-ai/schemastery";
|
|
4
3
|
import { WebError } from "@deepseek-ai/dsh-web";
|
|
5
4
|
import { JSDOM, VirtualConsole } from "jsdom";
|
|
@@ -8,7 +7,7 @@ import { Readability } from "@mozilla/readability";
|
|
|
8
7
|
import TurndownService from "turndown";
|
|
9
8
|
import { gfm } from "@joplin/turndown-plugin-gfm";
|
|
10
9
|
import { accessSync, constants, readFileSync, realpathSync, statSync } from "node:fs";
|
|
11
|
-
import { dirname, join } from "node:path";
|
|
10
|
+
import { delimiter, dirname, join } from "node:path";
|
|
12
11
|
import { pathToFileURL } from "node:url";
|
|
13
12
|
//#region src/config.ts
|
|
14
13
|
/**
|
|
@@ -127,7 +126,10 @@ function normalizeCdpEndpoint(input) {
|
|
|
127
126
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
128
127
|
* `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
|
|
129
128
|
* deliberately structural: a blog post that merely MENTIONS Cloudflare
|
|
130
|
-
* must not classify as a challenge.
|
|
129
|
+
* must not classify as a challenge. Bot Management's passive JSD
|
|
130
|
+
* telemetry (`/cdn-cgi/challenge-platform/scripts/jsd/`), which
|
|
131
|
+
* Cloudflare injects into every NORMAL page of a protected zone, is
|
|
132
|
+
* explicitly neutralized — it is not a challenge signal.
|
|
131
133
|
*
|
|
132
134
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
133
135
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -195,14 +197,29 @@ function escapeRegExp(text) {
|
|
|
195
197
|
*/
|
|
196
198
|
const CHALLENGE_TITLE_RE = new RegExp(`^(?:${CHALLENGE_TITLE_CORES.map(escapeRegExp).join("|")})[\\s.!??…。!]*$`, "i");
|
|
197
199
|
/**
|
|
200
|
+
* Cloudflare Bot Management's passive JavaScript-Detections (JSD)
|
|
201
|
+
* telemetry directory. Cloudflare injects this into EVERY normal page of
|
|
202
|
+
* a protected zone — as a direct `<script src>` or as the URL strung
|
|
203
|
+
* inside the hidden-1x1-iframe bootstrap's inline script text — with the
|
|
204
|
+
* page itself a plain 200 of real content. The path therefore proves
|
|
205
|
+
* nothing about a challenge and must be neutralized before the
|
|
206
|
+
* challenge-platform prefix check: real interstitials load orchestrate
|
|
207
|
+
* scripts (`/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/...`), which
|
|
208
|
+
* never live under `scripts/jsd/`.
|
|
209
|
+
*/
|
|
210
|
+
const TELEMETRY_JSD_DIR = "/cdn-cgi/challenge-platform/scripts/jsd/";
|
|
211
|
+
/**
|
|
198
212
|
* Structural markers a challenge document carries in its HTML. Kept in
|
|
199
213
|
* attribute/assignment form (`id="…"`, `window._cf_chl_opt`) rather than bare
|
|
200
214
|
* substrings so an article that merely QUOTES these strings in prose or code
|
|
201
215
|
* samples does not classify — the marker must look like the real element.
|
|
216
|
+
* The bare `/cdn-cgi/challenge-platform/` prefix is checked only after
|
|
217
|
+
* {@link TELEMETRY_JSD_DIR} occurrences are stripped (see
|
|
218
|
+
* {@link classifyChallengeHtml}); `scripts/jsd/main.js` in any spelling is
|
|
219
|
+
* passive telemetry and deliberately absent from this list.
|
|
202
220
|
*/
|
|
203
221
|
const CHALLENGE_HTML_MARKERS = [
|
|
204
222
|
"/cdn-cgi/challenge-platform/",
|
|
205
|
-
"/cdn-cgi/scripts/jsd/main.js",
|
|
206
223
|
"id=\"cf-chl-widget-",
|
|
207
224
|
"window._cf_chl_opt",
|
|
208
225
|
"id=\"challenge-form\"",
|
|
@@ -283,7 +300,8 @@ function classifyChallengeHtml(html) {
|
|
|
283
300
|
const lowered = html.toLowerCase();
|
|
284
301
|
if (lowered.includes(BLOCKED_BODY_PHRASE) && BLOCKED_BODY_MARKERS.some((marker) => lowered.includes(marker))) return "blocked";
|
|
285
302
|
if (CHALLENGE_TITLE_RE.test(title)) return "challenge";
|
|
286
|
-
|
|
303
|
+
const withoutTelemetry = lowered.split(TELEMETRY_JSD_DIR).join("");
|
|
304
|
+
if (CHALLENGE_HTML_MARKERS.some((marker) => withoutTelemetry.includes(marker))) return "challenge";
|
|
287
305
|
if (lowered.includes("class=\"ray-id\"") && lowered.includes("class=\"footer-inner\"")) return "challenge";
|
|
288
306
|
return "none";
|
|
289
307
|
}
|
|
@@ -293,7 +311,10 @@ function classifyChallengeHtml(html) {
|
|
|
293
311
|
* what makes SPA-style clears visible — the document swaps its content
|
|
294
312
|
* without any navigation, so only a DOM read can see the change. The title
|
|
295
313
|
* regex is baked in at build time from the same single source the Node-side
|
|
296
|
-
* check uses, so the two can never drift apart.
|
|
314
|
+
* check uses, so the two can never drift apart. Script srcs under
|
|
315
|
+
* `/cdn-cgi/challenge-platform/scripts/jsd/` are Bot Management's passive
|
|
316
|
+
* JSD telemetry — present on every normal page of a protected zone — and are
|
|
317
|
+
* excluded just like in {@link classifyChallengeHtml}.
|
|
297
318
|
*/
|
|
298
319
|
const CHALLENGE_DOM_PROBE = `(() => {
|
|
299
320
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
@@ -302,7 +323,7 @@ const CHALLENGE_DOM_PROBE = `(() => {
|
|
|
302
323
|
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
303
324
|
for (const script of Array.from(document.scripts)) {
|
|
304
325
|
const src = script.getAttribute('src') || ''
|
|
305
|
-
if (src.includes('/cdn-cgi/challenge-platform/')
|
|
326
|
+
if (src.includes('/cdn-cgi/challenge-platform/') && !src.includes('/cdn-cgi/challenge-platform/scripts/jsd/')) return true
|
|
306
327
|
}
|
|
307
328
|
if (document.querySelector('.footer .footer-inner .ray-id')) return true
|
|
308
329
|
return false
|
|
@@ -693,13 +714,15 @@ const EXECUTABLE_MAGICS = [
|
|
|
693
714
|
Buffer.from([77, 90])
|
|
694
715
|
];
|
|
695
716
|
/**
|
|
696
|
-
* Find `name` as an executable file on `$PATH
|
|
717
|
+
* Find `name` as an executable file on `$PATH`, split on the platform
|
|
718
|
+
* delimiter (`;` on Windows, `:` elsewhere — issue #1: a literal `:` made
|
|
719
|
+
* Windows discovery always miss).
|
|
697
720
|
* @param name - the executable basename.
|
|
698
721
|
* @returns the first matching path, or undefined.
|
|
699
722
|
*/
|
|
700
723
|
function findOnPath(name) {
|
|
701
724
|
const pathValue = process.env.PATH ?? "";
|
|
702
|
-
for (const dir of pathValue.split(
|
|
725
|
+
for (const dir of pathValue.split(delimiter)) {
|
|
703
726
|
if (dir === "") continue;
|
|
704
727
|
const candidate = join(dir, name);
|
|
705
728
|
try {
|
|
@@ -1404,15 +1427,17 @@ const name = "dsh-web-fetch-playwright";
|
|
|
1404
1427
|
/** The web seam this provider registers into. */
|
|
1405
1428
|
const inject = ["web"];
|
|
1406
1429
|
/** Settings namespace carrying this provider's configuration card. */
|
|
1407
|
-
const WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE =
|
|
1430
|
+
const WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE = "web-fetch-playwright";
|
|
1408
1431
|
/** Register the Playwright fetch provider with `ctx.web`. */
|
|
1409
1432
|
function apply(ctx, config) {
|
|
1410
1433
|
let current = () => config;
|
|
1411
|
-
|
|
1412
|
-
|
|
1413
|
-
|
|
1414
|
-
|
|
1415
|
-
|
|
1434
|
+
ctx.inject(["settings"], (settingsCtx) => {
|
|
1435
|
+
settingsCtx.settings.installSection(ctx, WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE, Config, config, {
|
|
1436
|
+
setSource: (source) => {
|
|
1437
|
+
current = source;
|
|
1438
|
+
},
|
|
1439
|
+
onChange: () => {}
|
|
1440
|
+
});
|
|
1416
1441
|
});
|
|
1417
1442
|
const provider = new PlaywrightFetchProvider(() => current());
|
|
1418
1443
|
ctx.effect(() => () => {
|
package/lib/types/challenge.d.ts
CHANGED
|
@@ -16,7 +16,10 @@
|
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
17
|
* `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
|
|
18
18
|
* deliberately structural: a blog post that merely MENTIONS Cloudflare
|
|
19
|
-
* must not classify as a challenge.
|
|
19
|
+
* must not classify as a challenge. Bot Management's passive JSD
|
|
20
|
+
* telemetry (`/cdn-cgi/challenge-platform/scripts/jsd/`), which
|
|
21
|
+
* Cloudflare injects into every NORMAL page of a protected zone, is
|
|
22
|
+
* explicitly neutralized — it is not a challenge signal.
|
|
20
23
|
*
|
|
21
24
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
22
25
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -78,6 +81,9 @@ export declare function classifyChallengeHtml(html: string): ChallengeVerdict;
|
|
|
78
81
|
* what makes SPA-style clears visible — the document swaps its content
|
|
79
82
|
* without any navigation, so only a DOM read can see the change. The title
|
|
80
83
|
* regex is baked in at build time from the same single source the Node-side
|
|
81
|
-
* check uses, so the two can never drift apart.
|
|
84
|
+
* check uses, so the two can never drift apart. Script srcs under
|
|
85
|
+
* `/cdn-cgi/challenge-platform/scripts/jsd/` are Bot Management's passive
|
|
86
|
+
* JSD telemetry — present on every normal page of a protected zone — and are
|
|
87
|
+
* excluded just like in {@link classifyChallengeHtml}.
|
|
82
88
|
*/
|
|
83
89
|
export declare const CHALLENGE_DOM_PROBE: string;
|
package/lib/types/index.d.ts
CHANGED
|
@@ -25,6 +25,6 @@ export declare const name = "dsh-web-fetch-playwright";
|
|
|
25
25
|
/** The web seam this provider registers into. */
|
|
26
26
|
export declare const inject: string[];
|
|
27
27
|
/** Settings namespace carrying this provider's configuration card. */
|
|
28
|
-
export declare const WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE
|
|
28
|
+
export declare const WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE = "web-fetch-playwright";
|
|
29
29
|
/** Register the Playwright fetch provider with `ctx.web`. */
|
|
30
30
|
export declare function apply(ctx: Context, config: Config): void;
|
|
@@ -26,7 +26,9 @@ export interface ResolvedPlaywright {
|
|
|
26
26
|
source: string;
|
|
27
27
|
}
|
|
28
28
|
/**
|
|
29
|
-
* Find `name` as an executable file on `$PATH
|
|
29
|
+
* Find `name` as an executable file on `$PATH`, split on the platform
|
|
30
|
+
* delimiter (`;` on Windows, `:` elsewhere — issue #1: a literal `:` made
|
|
31
|
+
* Windows discovery always miss).
|
|
30
32
|
* @param name - the executable basename.
|
|
31
33
|
* @returns the first matching path, or undefined.
|
|
32
34
|
*/
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-web-fetch-playwright",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.7",
|
|
4
4
|
"description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -26,13 +26,12 @@
|
|
|
26
26
|
"lib/types/**/*.d.ts",
|
|
27
27
|
"src",
|
|
28
28
|
"cordis.patch.yml",
|
|
29
|
-
"dsh.plugin.json",
|
|
30
29
|
"README.md",
|
|
31
30
|
"README.zh-CN.md",
|
|
32
31
|
"LICENSE"
|
|
33
32
|
],
|
|
34
33
|
"scripts": {
|
|
35
|
-
"build": "
|
|
34
|
+
"build": "node -e \"require('node:fs').rmSync('lib',{recursive:true,force:true})\" && tsc -p tsconfig.build.json --emitDeclarationOnly && tsdown",
|
|
36
35
|
"prepare": "pnpm run build",
|
|
37
36
|
"prepublishOnly": "pnpm run build",
|
|
38
37
|
"typecheck": "tsc --noEmit",
|
|
@@ -89,7 +88,7 @@
|
|
|
89
88
|
"retention": "none (profile mode: site data lands in the remote browser's own persistent user-data-dir, which this plugin never cleans)"
|
|
90
89
|
},
|
|
91
90
|
"dependencies": {
|
|
92
|
-
"@deepseek-ai/schemastery": ">=3.18.
|
|
91
|
+
"@deepseek-ai/schemastery": ">=3.18.2 <4",
|
|
93
92
|
"@joplin/turndown-plugin-gfm": "^1.0.56",
|
|
94
93
|
"@mozilla/readability": "^0.6.0",
|
|
95
94
|
"dompurify": "^3.2.4",
|
|
@@ -99,8 +98,8 @@
|
|
|
99
98
|
},
|
|
100
99
|
"peerDependencies": {
|
|
101
100
|
"@deepseek-ai/cordis": ">=4.0.1-rc.1 <5",
|
|
102
|
-
"@deepseek-ai/dsh-settings": ">=0.1.
|
|
103
|
-
"@deepseek-ai/dsh-web": ">=0.1.
|
|
101
|
+
"@deepseek-ai/dsh-settings": ">=0.1.2-alpha.0 <0.3.0",
|
|
102
|
+
"@deepseek-ai/dsh-web": ">=0.1.2-alpha.0 <0.3.0"
|
|
104
103
|
},
|
|
105
104
|
"peerDependenciesMeta": {
|
|
106
105
|
"@deepseek-ai/cordis": {
|
|
@@ -109,13 +108,15 @@
|
|
|
109
108
|
},
|
|
110
109
|
"devDependencies": {
|
|
111
110
|
"@deepseek-ai/cordis": "4.0.1",
|
|
112
|
-
"@deepseek-ai/dsh-
|
|
113
|
-
"@deepseek-ai/dsh-client-
|
|
114
|
-
"@deepseek-ai/dsh-client-
|
|
115
|
-
"@deepseek-ai/dsh-client-
|
|
116
|
-
"@deepseek-ai/dsh-client-ui-
|
|
117
|
-
"@deepseek-ai/dsh-settings": "0.1.
|
|
118
|
-
"@deepseek-ai/dsh-
|
|
111
|
+
"@deepseek-ai/dsh-brand": "0.1.2-alpha.5",
|
|
112
|
+
"@deepseek-ai/dsh-client-locale": "0.1.2-alpha.5",
|
|
113
|
+
"@deepseek-ai/dsh-client-runtime": "0.1.1-rc.2",
|
|
114
|
+
"@deepseek-ai/dsh-client-store": "0.1.2-alpha.5",
|
|
115
|
+
"@deepseek-ai/dsh-client-ui-settings": "0.1.2-alpha.5",
|
|
116
|
+
"@deepseek-ai/dsh-client-ui-settings-plugins": "0.1.2-alpha.5",
|
|
117
|
+
"@deepseek-ai/dsh-client-ui-slots": "0.1.2-alpha.5",
|
|
118
|
+
"@deepseek-ai/dsh-settings": "0.1.2-alpha.5",
|
|
119
|
+
"@deepseek-ai/dsh-web": "0.1.2-alpha.5",
|
|
119
120
|
"@types/jsdom": "^30.0.0",
|
|
120
121
|
"@types/node": "^24.0.0",
|
|
121
122
|
"@types/react": "^18.3.1",
|
package/src/challenge.ts
CHANGED
|
@@ -16,7 +16,10 @@
|
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
17
|
* `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
|
|
18
18
|
* deliberately structural: a blog post that merely MENTIONS Cloudflare
|
|
19
|
-
* must not classify as a challenge.
|
|
19
|
+
* must not classify as a challenge. Bot Management's passive JSD
|
|
20
|
+
* telemetry (`/cdn-cgi/challenge-platform/scripts/jsd/`), which
|
|
21
|
+
* Cloudflare injects into every NORMAL page of a protected zone, is
|
|
22
|
+
* explicitly neutralized — it is not a challenge signal.
|
|
20
23
|
*
|
|
21
24
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
22
25
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -85,15 +88,31 @@ export const CHALLENGE_TITLE_RE = new RegExp(
|
|
|
85
88
|
'i',
|
|
86
89
|
)
|
|
87
90
|
|
|
91
|
+
/**
|
|
92
|
+
* Cloudflare Bot Management's passive JavaScript-Detections (JSD)
|
|
93
|
+
* telemetry directory. Cloudflare injects this into EVERY normal page of
|
|
94
|
+
* a protected zone — as a direct `<script src>` or as the URL strung
|
|
95
|
+
* inside the hidden-1x1-iframe bootstrap's inline script text — with the
|
|
96
|
+
* page itself a plain 200 of real content. The path therefore proves
|
|
97
|
+
* nothing about a challenge and must be neutralized before the
|
|
98
|
+
* challenge-platform prefix check: real interstitials load orchestrate
|
|
99
|
+
* scripts (`/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/...`), which
|
|
100
|
+
* never live under `scripts/jsd/`.
|
|
101
|
+
*/
|
|
102
|
+
const TELEMETRY_JSD_DIR = '/cdn-cgi/challenge-platform/scripts/jsd/'
|
|
103
|
+
|
|
88
104
|
/**
|
|
89
105
|
* Structural markers a challenge document carries in its HTML. Kept in
|
|
90
106
|
* attribute/assignment form (`id="…"`, `window._cf_chl_opt`) rather than bare
|
|
91
107
|
* substrings so an article that merely QUOTES these strings in prose or code
|
|
92
108
|
* samples does not classify — the marker must look like the real element.
|
|
109
|
+
* The bare `/cdn-cgi/challenge-platform/` prefix is checked only after
|
|
110
|
+
* {@link TELEMETRY_JSD_DIR} occurrences are stripped (see
|
|
111
|
+
* {@link classifyChallengeHtml}); `scripts/jsd/main.js` in any spelling is
|
|
112
|
+
* passive telemetry and deliberately absent from this list.
|
|
93
113
|
*/
|
|
94
114
|
const CHALLENGE_HTML_MARKERS = [
|
|
95
115
|
'/cdn-cgi/challenge-platform/',
|
|
96
|
-
'/cdn-cgi/scripts/jsd/main.js',
|
|
97
116
|
'id="cf-chl-widget-',
|
|
98
117
|
'window._cf_chl_opt',
|
|
99
118
|
'id="challenge-form"',
|
|
@@ -187,7 +206,14 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
|
|
|
187
206
|
const lowered = html.toLowerCase()
|
|
188
207
|
if (lowered.includes(BLOCKED_BODY_PHRASE) && BLOCKED_BODY_MARKERS.some(marker => lowered.includes(marker))) return 'blocked'
|
|
189
208
|
if (CHALLENGE_TITLE_RE.test(title)) return 'challenge'
|
|
190
|
-
|
|
209
|
+
// Strip the passive JSD telemetry directory before the marker scan: its
|
|
210
|
+
// URL embeds the `/cdn-cgi/challenge-platform/` prefix, so an unstripped
|
|
211
|
+
// scan misreads every Bot-Management-protected normal page (a plain 200
|
|
212
|
+
// with real content — e.g. openrouter.ai) as an interstitial, the wait
|
|
213
|
+
// can then never confirm a clear, and the fetch dies with
|
|
214
|
+
// WEB_FETCH_CHALLENGE despite the page having loaded fine.
|
|
215
|
+
const withoutTelemetry = lowered.split(TELEMETRY_JSD_DIR).join('')
|
|
216
|
+
if (CHALLENGE_HTML_MARKERS.some(marker => withoutTelemetry.includes(marker))) return 'challenge'
|
|
191
217
|
// Crawlee's footer marker, as a two-class combo so prose quoting one class
|
|
192
218
|
// alone stays clean.
|
|
193
219
|
if (lowered.includes('class="ray-id"') && lowered.includes('class="footer-inner"')) return 'challenge'
|
|
@@ -200,7 +226,10 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
|
|
|
200
226
|
* what makes SPA-style clears visible — the document swaps its content
|
|
201
227
|
* without any navigation, so only a DOM read can see the change. The title
|
|
202
228
|
* regex is baked in at build time from the same single source the Node-side
|
|
203
|
-
* check uses, so the two can never drift apart.
|
|
229
|
+
* check uses, so the two can never drift apart. Script srcs under
|
|
230
|
+
* `/cdn-cgi/challenge-platform/scripts/jsd/` are Bot Management's passive
|
|
231
|
+
* JSD telemetry — present on every normal page of a protected zone — and are
|
|
232
|
+
* excluded just like in {@link classifyChallengeHtml}.
|
|
204
233
|
*/
|
|
205
234
|
export const CHALLENGE_DOM_PROBE = `(() => {
|
|
206
235
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
@@ -209,7 +238,7 @@ export const CHALLENGE_DOM_PROBE = `(() => {
|
|
|
209
238
|
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
210
239
|
for (const script of Array.from(document.scripts)) {
|
|
211
240
|
const src = script.getAttribute('src') || ''
|
|
212
|
-
if (src.includes('/cdn-cgi/challenge-platform/')
|
|
241
|
+
if (src.includes('/cdn-cgi/challenge-platform/') && !src.includes('/cdn-cgi/challenge-platform/scripts/jsd/')) return true
|
|
213
242
|
}
|
|
214
243
|
if (document.querySelector('.footer .footer-inner .ray-id')) return true
|
|
215
244
|
return false
|
package/src/index.ts
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
*/
|
|
12
12
|
|
|
13
13
|
import type { Context } from '@deepseek-ai/cordis'
|
|
14
|
-
import {
|
|
14
|
+
import type {} from '@deepseek-ai/dsh-settings'
|
|
15
15
|
import type {} from '@deepseek-ai/dsh-web'
|
|
16
16
|
import { Config } from './config.ts'
|
|
17
17
|
import type { ResolvedConfig } from './config.ts'
|
|
@@ -34,7 +34,7 @@ export const name = 'dsh-web-fetch-playwright'
|
|
|
34
34
|
export const inject = ['web']
|
|
35
35
|
|
|
36
36
|
/** Settings namespace carrying this provider's configuration card. */
|
|
37
|
-
export const WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE =
|
|
37
|
+
export const WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE = 'web-fetch-playwright'
|
|
38
38
|
|
|
39
39
|
/** Register the Playwright fetch provider with `ctx.web`. */
|
|
40
40
|
export function apply(ctx: Context, config: Config): void {
|
|
@@ -44,13 +44,15 @@ export function apply(ctx: Context, config: Config): void {
|
|
|
44
44
|
// Passing `current` directly would pin the composition entry forever (the
|
|
45
45
|
// web-search-deepseek provider uses the same wrapper idiom).
|
|
46
46
|
let current: () => ResolvedConfig = () => config as ResolvedConfig
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
47
|
+
ctx.inject(['settings'], (settingsCtx) => {
|
|
48
|
+
settingsCtx.settings.installSection(ctx, WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE, Config, config, {
|
|
49
|
+
setSource: (source) => {
|
|
50
|
+
current = source as () => ResolvedConfig
|
|
51
|
+
},
|
|
52
|
+
// The provider projects the section per fetch, so a committed change
|
|
53
|
+
// needs no re-registration.
|
|
54
|
+
onChange: () => {},
|
|
55
|
+
})
|
|
54
56
|
})
|
|
55
57
|
// The CDP backend holds one shared connection for the provider's lifetime;
|
|
56
58
|
// drop it when this plugin unloads so restarts don't strand sockets.
|
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
*/
|
|
18
18
|
|
|
19
19
|
import { accessSync, constants, readFileSync, realpathSync, statSync } from 'node:fs'
|
|
20
|
-
import { dirname, join } from 'node:path'
|
|
20
|
+
import { delimiter, dirname, join } from 'node:path'
|
|
21
21
|
import { createRequire } from 'node:module'
|
|
22
22
|
import { pathToFileURL } from 'node:url'
|
|
23
23
|
import type { PlaywrightChromium } from './types.ts'
|
|
@@ -41,13 +41,15 @@ export interface ResolvedPlaywright {
|
|
|
41
41
|
}
|
|
42
42
|
|
|
43
43
|
/**
|
|
44
|
-
* Find `name` as an executable file on `$PATH
|
|
44
|
+
* Find `name` as an executable file on `$PATH`, split on the platform
|
|
45
|
+
* delimiter (`;` on Windows, `:` elsewhere — issue #1: a literal `:` made
|
|
46
|
+
* Windows discovery always miss).
|
|
45
47
|
* @param name - the executable basename.
|
|
46
48
|
* @returns the first matching path, or undefined.
|
|
47
49
|
*/
|
|
48
50
|
export function findOnPath(name: string): string | undefined {
|
|
49
51
|
const pathValue = process.env.PATH ?? ''
|
|
50
|
-
for (const dir of pathValue.split(
|
|
52
|
+
for (const dir of pathValue.split(delimiter)) {
|
|
51
53
|
if (dir === '') continue
|
|
52
54
|
const candidate = join(dir, name)
|
|
53
55
|
try {
|
package/dsh.plugin.json
DELETED
|
@@ -1,16 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"name": "dsh-web-fetch-playwright",
|
|
3
|
-
"version": "0.2.5",
|
|
4
|
-
"description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
|
|
5
|
-
"entry": {
|
|
6
|
-
"name": "dsh-web-fetch-playwright",
|
|
7
|
-
"inject": []
|
|
8
|
-
},
|
|
9
|
-
"client": {
|
|
10
|
-
"platform": "web"
|
|
11
|
-
},
|
|
12
|
-
"contributes": {
|
|
13
|
-
"tools": [],
|
|
14
|
-
"skills": []
|
|
15
|
-
}
|
|
16
|
-
}
|