dsh-web-fetch-playwright 0.2.10 → 0.2.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/README.zh-CN.md +2 -2
- package/lib/client.js +19 -19
- package/lib/index.js +210 -35
- package/lib/types/challenge.d.ts +10 -8
- package/lib/types/provider.d.ts +16 -8
- package/lib/types/types.d.ts +33 -0
- package/package.json +1 -1
- package/src/challenge.ts +101 -33
- package/src/provider.ts +136 -9
- package/src/types.ts +32 -0
package/README.md
CHANGED
|
@@ -34,7 +34,7 @@ web_fetch (tool-web)
|
|
|
34
34
|
└─ Markdown (or raw HTML when denoise is off)
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
|
|
37
|
+
**Non-2xx statuses are results, not errors** — the web seam's contract (`@deepseek-ai/dsh-web`): a fetched 404/418/500 comes back with its status code, exactly what `dsh-web-fetch-http` does. Chromium complicates this for *status-only* responses — an error status with an empty body (httpbin's `/status/404`) fails the whole navigation with `net::ERR_HTTP_RESPONSE_CODE_FAILURE`, and a 204/205 is aborted (`net::ERR_ABORTED`) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so `web_fetch` reports `Fetched … (HTTP 404)` instead of a `WEB_PROVIDER_ERROR` throw. The same recovery covers *download* responses — a body served with `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: `goto` rejects with `Download is starting` and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; undecodable downloads (a PDF, a ZIP) still fail with `WEB_UNSUPPORTED_CONTENT_TYPE`. A recovered response that is itself a Cloudflare challenge edge still fails with `WEB_FETCH_CHALLENGE`; transport errors and cancellations stay errors.
|
|
38
38
|
|
|
39
39
|
## Requirements
|
|
40
40
|
|
|
@@ -111,7 +111,7 @@ Some strict sites serve a Cloudflare interstitial before the real page. A real b
|
|
|
111
111
|
|
|
112
112
|
With the wait on (default):
|
|
113
113
|
|
|
114
|
-
1. **Detection** — a response carrying `cf-mitigated: challenge` (the documented signal for every challenge page type), or a 403/503 HTML document from a `server: cloudflare` edge, or the localized interstitial itself (title family like "Just a moment…", `请稍候…`, "Минутку…", plus structural markers:
|
|
114
|
+
1. **Detection** — a response carrying `cf-mitigated: challenge` (the documented signal for every challenge page type), or a 403/503 HTML document from a `server: cloudflare` edge, or the localized interstitial itself (title family like "Just a moment…", `请稍候…`, "Минутку…", plus structural markers: the challenge-platform runtime's `<script src>`, `#challenge-*` elements, `cf-chl-widget-` frames, `window._cf_chl_opt`). The content-level markers are the *fallback tier* and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (`server: cloudflare` / `cf-ray`) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Two refinements keep that promise for real pages: the challenge-platform script check reads real `<script src>` attributes only (prose, and escaped code samples that quote a path, are ignored), and it matches the runtime mount `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/` *positively*. Everything else Cloudflare injects under that prefix — Bot Management's JavaScript Detections (`scripts/jsd/`, `h/[b|g]/scripts/jsd/<hash>/`, `h/[b|g]/jsd/r/`), and Precursor (`scripts/precursor/`) — is passive instrumentation present on *every normal page* of a protected zone (e.g. openrouter.ai, nodeseek.com), so it is never read as a challenge. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
|
|
115
115
|
2. **Bounded wait, same tab and context** — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the *last* main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
|
|
116
116
|
3. **Bounded retry** — when a window runs out, the same tab re-navigates once (default; `challengeRetries`) with whatever clearance cookies the context already holds.
|
|
117
117
|
4. **Clear failure** — `WEB_FETCH_CHALLENGE` (a provider-specific code the web seam's open-string `code` allows) naming the site, the budget spent, and the last challenge status.
|
package/README.zh-CN.md
CHANGED
|
@@ -34,7 +34,7 @@ web_fetch (tool-web)
|
|
|
34
34
|
└─ Markdown(关闭降噪时返回原始 HTML)
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR
|
|
37
|
+
**非 2xx 状态码是结果而非错误** —— 这是 web seam 的契约(`@deepseek-ai/dsh-web`):抓取到的 404/418/500 会连同状态码一起正常返回,与 `dsh-web-fetch-http` 的行为一致。但对*仅有状态*的响应 Chromium 会把事情搞砸 —— 空 body 的错误状态(httpbin 的 `/status/404`)会让整个导航以 `net::ERR_HTTP_RESPONSE_CODE_FAILURE` 失败,204/205 则因无可渲染内容被中止(`net::ERR_ABORTED`)。因此 provider 在每次抓取中都跟踪最后一个主框架响应(该响应先于 Chromium 判定导航失败到达),并把这类失败恢复成带状态码的正常结果,`web_fetch` 会报告 `Fetched … (HTTP 404)` 而不是抛出 `WEB_PROVIDER_ERROR`。同样的恢复也覆盖*下载型*响应 —— 带 `Content-Disposition: attachment` 的 body(publish.twitter.com 的 oembed 端点——agent 读取推文常用的替代 URL——返回的 JSON 正是如此)永远不会被提交为文档:`goto` 以 `Download is starting` 拒绝,且 body 已从网络栈中流失,provider 因此通过页面的 API request context 重新拉取并返回可解码内容;不可解码的下载(PDF、ZIP)仍以 `WEB_UNSUPPORTED_CONTENT_TYPE` 失败。若恢复出的响应本身来自 Cloudflare 挑战边缘,仍以 `WEB_FETCH_CHALLENGE` 失败;传输错误与取消也依旧按错误处理。
|
|
38
38
|
|
|
39
39
|
## 环境要求
|
|
40
40
|
|
|
@@ -111,7 +111,7 @@ google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome
|
|
|
111
111
|
|
|
112
112
|
开启等待(默认)后的流程:
|
|
113
113
|
|
|
114
|
-
1. **识别** —— 响应带 `cf-mitigated: challenge`(Cloudflare 官方文档注明所有挑战页类型都带此头),或 403/503 且 `server: cloudflare` 的 HTML 文档,或本地化的中间页本身("Just a moment…" / "请稍候…" / "Минутку…" 等 title
|
|
114
|
+
1. **识别** —— 响应带 `cf-mitigated: challenge`(Cloudflare 官方文档注明所有挑战页类型都带此头),或 403/503 且 `server: cloudflare` 的 HTML 文档,或本地化的中间页本身("Just a moment…" / "请稍候…" / "Минутку…" 等 title 家族,以及结构性标记:挑战运行时的 `<script src>`、`#challenge-*` 元素、`cf-chl-widget-` 框架、`window._cf_chl_opt`)。内容级标记只是**兜底层**,且仅对"挑战兼容"的响应(403/429/503 或来自 Cloudflare 边缘——`server: cloudflare` / `cf-ray`)运行——因为中间页从不会以普通 200 返回,所以正文里引用了挑战文案的普通文章绝不可能被误判。两处收紧让这个承诺真正成立:挑战平台脚本只读真实的 `<script src>` 属性(正文、以及转义后的代码示例里引用的路径一律忽略),并且**正面**匹配运行时挂载点 `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/`。该前缀下 Cloudflare 注入的其余脚本——Bot Management 的 JavaScript Detections(`scripts/jsd/`、`h/[b|g]/scripts/jsd/<hash>/`、`h/[b|g]/jsd/r/`)与 Precursor(`scripts/precursor/`)——都是会注入受保护站点**每一个正常页面**的被动探测脚本(如 openrouter.ai、nodeseek.com),永远不算挑战信号。硬封锁页("Sorry, you have been blocked")单独分类并立即失败——等待无法解除。
|
|
115
115
|
2. **同标签页、同上下文的有界等待** —— 每 500ms 轮询活 DOM,等浏览器跑完自己的验证;同时跟踪**最后一次主 frame 导航响应**,所以重载进来的真实文档的状态码和响应头才是最终上报的。SPA 式清除(无导航、纯内容替换)由同一个 DOM 探测捕获。
|
|
116
116
|
3. **有界重试** —— 窗口耗尽后,同一标签页默认再导航一次(`challengeRetries`),上下文里已有的通关 cookie 继续生效。
|
|
117
117
|
4. **明确失败** —— 返回独立的 `WEB_FETCH_CHALLENGE` 错误码(web seam 的 `code` 是开放字符串,允许 provider 专属码),消息中写明站点、等待预算与最后一次挑战响应的状态。
|
package/lib/client.js
CHANGED
|
@@ -412,12 +412,12 @@ window.__ModuleLoader__.load({
|
|
|
412
412
|
document.head.appendChild(tag);
|
|
413
413
|
}
|
|
414
414
|
var PluginCard_module_css_default = {
|
|
415
|
+
"footer": "UM8-Rq_footer",
|
|
415
416
|
"readOnly": "UM8-Rq_readOnly",
|
|
416
|
-
"form": "UM8-Rq_form",
|
|
417
417
|
"failed": "UM8-Rq_failed",
|
|
418
|
-
"
|
|
419
|
-
"
|
|
420
|
-
"
|
|
418
|
+
"discard": "UM8-Rq_discard",
|
|
419
|
+
"form": "UM8-Rq_form",
|
|
420
|
+
"save": "UM8-Rq_save"
|
|
421
421
|
};
|
|
422
422
|
//#endregion
|
|
423
423
|
//#region src/client/PluginCard.tsx
|
|
@@ -477,26 +477,26 @@ window.__ModuleLoader__.load({
|
|
|
477
477
|
document.head.appendChild(tag);
|
|
478
478
|
}
|
|
479
479
|
var fields_module_css_default = {
|
|
480
|
-
"
|
|
481
|
-
"inputInvalid": "CFSaVG_inputInvalid",
|
|
482
|
-
"hintIndented": "CFSaVG_hintIndented",
|
|
483
|
-
"radioText": "CFSaVG_radioText",
|
|
480
|
+
"head": "CFSaVG_head",
|
|
484
481
|
"radioPick": "CFSaVG_radioPick",
|
|
485
|
-
"radioLabel": "CFSaVG_radioLabel",
|
|
486
|
-
"invalid": "CFSaVG_invalid",
|
|
487
|
-
"radioHint": "CFSaVG_radioHint",
|
|
488
|
-
"radioOption": "CFSaVG_radioOption",
|
|
489
|
-
"badges": "CFSaVG_badges",
|
|
490
|
-
"label": "CFSaVG_label",
|
|
491
|
-
"hint": "CFSaVG_hint",
|
|
492
482
|
"badge": "CFSaVG_badge",
|
|
483
|
+
"radioHint": "CFSaVG_radioHint",
|
|
484
|
+
"radioContent": "CFSaVG_radioContent",
|
|
493
485
|
"field": "CFSaVG_field",
|
|
494
|
-
"
|
|
486
|
+
"label": "CFSaVG_label",
|
|
495
487
|
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
496
|
-
"radioGroup": "CFSaVG_radioGroup",
|
|
497
488
|
"checkboxRow": "CFSaVG_checkboxRow",
|
|
498
|
-
"
|
|
499
|
-
"
|
|
489
|
+
"hintIndented": "CFSaVG_hintIndented",
|
|
490
|
+
"radioGroup": "CFSaVG_radioGroup",
|
|
491
|
+
"reset": "CFSaVG_reset",
|
|
492
|
+
"radioText": "CFSaVG_radioText",
|
|
493
|
+
"radioLabel": "CFSaVG_radioLabel",
|
|
494
|
+
"radioOption": "CFSaVG_radioOption",
|
|
495
|
+
"hint": "CFSaVG_hint",
|
|
496
|
+
"badges": "CFSaVG_badges",
|
|
497
|
+
"invalid": "CFSaVG_invalid",
|
|
498
|
+
"input": "CFSaVG_input",
|
|
499
|
+
"inputInvalid": "CFSaVG_inputInvalid"
|
|
500
500
|
};
|
|
501
501
|
//#endregion
|
|
502
502
|
//#region src/client/fields.tsx
|
package/lib/index.js
CHANGED
|
@@ -160,10 +160,12 @@ function normalizeCdpEndpoint(input) {
|
|
|
160
160
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
161
161
|
* `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
|
|
162
162
|
* deliberately structural: a blog post that merely MENTIONS Cloudflare
|
|
163
|
-
* must not classify as a challenge
|
|
164
|
-
*
|
|
165
|
-
*
|
|
166
|
-
*
|
|
163
|
+
* must not classify as a challenge, so the challenge-platform script
|
|
164
|
+
* check reads `<script src>` attributes only (never prose or escaped
|
|
165
|
+
* code samples) and identifies the runtime POSITIVELY by its
|
|
166
|
+
* `/orchestrate/` mount — the JSD/Precursor instrumentation Cloudflare
|
|
167
|
+
* injects into every NORMAL page of a protected zone, on several moving
|
|
168
|
+
* mounts, is not a challenge signal.
|
|
167
169
|
*
|
|
168
170
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
169
171
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -231,29 +233,86 @@ function escapeRegExp(text) {
|
|
|
231
233
|
*/
|
|
232
234
|
const CHALLENGE_TITLE_RE = new RegExp(`^(?:${CHALLENGE_TITLE_CORES.map(escapeRegExp).join("|")})[\\s.!??…。!]*$`, "i");
|
|
233
235
|
/**
|
|
234
|
-
* Cloudflare
|
|
235
|
-
*
|
|
236
|
-
*
|
|
237
|
-
*
|
|
238
|
-
*
|
|
239
|
-
*
|
|
240
|
-
*
|
|
241
|
-
*
|
|
242
|
-
*
|
|
236
|
+
* Cloudflare's challenge-platform namespace holds two very different things,
|
|
237
|
+
* and telling them apart is the whole game:
|
|
238
|
+
*
|
|
239
|
+
* - the challenge RUNTIME — `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/…`
|
|
240
|
+
* — which real interstitials load to run the verification;
|
|
241
|
+
* - PASSIVE INSTRUMENTATION — everything else Cloudflare injects into NORMAL
|
|
242
|
+
* pages of a protected zone, as a direct `<script src>` or as the URL
|
|
243
|
+
* strung inside the hidden-1x1-iframe bootstrap's inline script text. The
|
|
244
|
+
* page itself is a plain 200 of real content.
|
|
245
|
+
*
|
|
246
|
+
* Observed instrumentation layouts (a thread that lists all of them:
|
|
247
|
+
* `community.cloudflare.com/t/3-javascripts-are-added-unable-to-disable`):
|
|
248
|
+
*
|
|
249
|
+
* - `/cdn-cgi/challenge-platform/scripts/jsd/…` — JavaScript Detections;
|
|
250
|
+
* - `/cdn-cgi/challenge-platform/scripts/precursor/…` — Precursor, the
|
|
251
|
+
* continuous behavioral-validation engine;
|
|
252
|
+
* - `/cdn-cgi/challenge-platform/h/[b|g]/scripts/jsd/<hash>/…` and
|
|
253
|
+
* `/cdn-cgi/challenge-platform/h/[b|g]/jsd/r/…` — the same detections
|
|
254
|
+
* served from a hash-scoped runtime mount, with no `scripts/` segment.
|
|
255
|
+
*
|
|
256
|
+
* Enumerating instrumentation names (`jsd`, `precursor`) or carving out a
|
|
257
|
+
* `scripts/` namespace is a standing false-positive subscription: Cloudflare
|
|
258
|
+
* keeps ADDING packages and keeps moving them onto new mounts (openrouter.ai
|
|
259
|
+
* broke on JSD, nodeseek.com broke on Precursor, and the hash-scoped variant
|
|
260
|
+
* defeats a `scripts/`-only carve-out). So the runtime is identified
|
|
261
|
+
* POSITIVELY instead: `orchestrate` is the challenge runtime's own mount and
|
|
262
|
+
* never appears in instrumentation. Everything else under the prefix is
|
|
263
|
+
* treated as passive — the directive that a protected page must stay
|
|
264
|
+
* readable.
|
|
265
|
+
*
|
|
266
|
+
* Losing an unknown future runtime mount costs little here: the response
|
|
267
|
+
* header, the localized title family, and the challenge DOM elements remain
|
|
268
|
+
* independent signals, and the content-level script check is only their
|
|
269
|
+
* fallback.
|
|
270
|
+
*/
|
|
271
|
+
const CHALLENGE_PLATFORM_PREFIX = "/cdn-cgi/challenge-platform/";
|
|
272
|
+
/** The path segment that identifies the challenge runtime (not telemetry). */
|
|
273
|
+
const CHALLENGE_RUNTIME_SEGMENT = "/orchestrate/";
|
|
274
|
+
/**
|
|
275
|
+
* A `<script src>` value in document HTML: double-quoted, single-quoted, or
|
|
276
|
+
* bare. Only real `src` attributes are read, so prose and escaped code
|
|
277
|
+
* samples that merely quote a path can never match (badcase:
|
|
278
|
+
* `community.cloudflare.com/t/3-javascripts-are-added-unable-to-disable`
|
|
279
|
+
* — a thread ABOUT these injected scripts, which lists
|
|
280
|
+
* `/cdn-cgi/challenge-platform/scripts/jsd/main.js` in its body text and was
|
|
281
|
+
* misclassified as a challenge by the old whole-document substring scan).
|
|
282
|
+
*/
|
|
283
|
+
const SCRIPT_SRC_RE = /<script\b[^>]*?\bsrc\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>]+))/gi;
|
|
284
|
+
/**
|
|
285
|
+
* Extract every `<script src>` value from a document's HTML, lowercased.
|
|
286
|
+
*
|
|
287
|
+
* @param html - the document HTML as `page.content()` returns it.
|
|
288
|
+
* @returns the script source URLs, in document order.
|
|
289
|
+
*/
|
|
290
|
+
function scriptSrcsOf(html) {
|
|
291
|
+
const srcs = [];
|
|
292
|
+
SCRIPT_SRC_RE.lastIndex = 0;
|
|
293
|
+
for (let match = SCRIPT_SRC_RE.exec(html); match !== null; match = SCRIPT_SRC_RE.exec(html)) srcs.push((match[1] ?? match[2] ?? match[3] ?? "").toLowerCase());
|
|
294
|
+
return srcs;
|
|
295
|
+
}
|
|
296
|
+
/**
|
|
297
|
+
* Whether a script src belongs to the challenge RUNTIME rather than to the
|
|
298
|
+
* passive instrumentation Cloudflare injects into normal pages.
|
|
299
|
+
*
|
|
300
|
+
* @param src - a lowercased script source URL.
|
|
301
|
+
* @returns true when the script is the challenge runtime.
|
|
243
302
|
*/
|
|
244
|
-
|
|
303
|
+
function isChallengeRuntimeScript(src) {
|
|
304
|
+
return src.includes(CHALLENGE_PLATFORM_PREFIX) && src.includes(CHALLENGE_RUNTIME_SEGMENT);
|
|
305
|
+
}
|
|
245
306
|
/**
|
|
246
307
|
* Structural markers a challenge document carries in its HTML. Kept in
|
|
247
308
|
* attribute/assignment form (`id="…"`, `window._cf_chl_opt`) rather than bare
|
|
248
309
|
* substrings so an article that merely QUOTES these strings in prose or code
|
|
249
310
|
* samples does not classify — the marker must look like the real element.
|
|
250
|
-
* The
|
|
251
|
-
* {@link
|
|
252
|
-
*
|
|
253
|
-
* passive telemetry and deliberately absent from this list.
|
|
311
|
+
* The challenge-platform prefix is deliberately absent: script evidence goes
|
|
312
|
+
* through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
|
|
313
|
+
* distinguish a real `src` from prose and runtime from instrumentation.
|
|
254
314
|
*/
|
|
255
315
|
const CHALLENGE_HTML_MARKERS = [
|
|
256
|
-
"/cdn-cgi/challenge-platform/",
|
|
257
316
|
"id=\"cf-chl-widget-",
|
|
258
317
|
"window._cf_chl_opt",
|
|
259
318
|
"id=\"challenge-form\"",
|
|
@@ -334,8 +393,8 @@ function classifyChallengeHtml(html) {
|
|
|
334
393
|
const lowered = html.toLowerCase();
|
|
335
394
|
if (lowered.includes(BLOCKED_BODY_PHRASE) && BLOCKED_BODY_MARKERS.some((marker) => lowered.includes(marker))) return "blocked";
|
|
336
395
|
if (CHALLENGE_TITLE_RE.test(title)) return "challenge";
|
|
337
|
-
|
|
338
|
-
if (CHALLENGE_HTML_MARKERS.some((marker) =>
|
|
396
|
+
if (scriptSrcsOf(html).some(isChallengeRuntimeScript)) return "challenge";
|
|
397
|
+
if (CHALLENGE_HTML_MARKERS.some((marker) => lowered.includes(marker))) return "challenge";
|
|
339
398
|
if (lowered.includes("class=\"ray-id\"") && lowered.includes("class=\"footer-inner\"")) return "challenge";
|
|
340
399
|
return "none";
|
|
341
400
|
}
|
|
@@ -345,19 +404,21 @@ function classifyChallengeHtml(html) {
|
|
|
345
404
|
* what makes SPA-style clears visible — the document swaps its content
|
|
346
405
|
* without any navigation, so only a DOM read can see the change. The title
|
|
347
406
|
* regex is baked in at build time from the same single source the Node-side
|
|
348
|
-
* check uses, so the two can never drift apart. Script
|
|
349
|
-
*
|
|
350
|
-
*
|
|
351
|
-
*
|
|
407
|
+
* check uses, so the two can never drift apart. Script evidence follows the
|
|
408
|
+
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
409
|
+
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
410
|
+
* protected zone is ignored.
|
|
352
411
|
*/
|
|
353
412
|
const CHALLENGE_DOM_PROBE = `(() => {
|
|
354
413
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
414
|
+
const platform = ${JSON.stringify(CHALLENGE_PLATFORM_PREFIX)}
|
|
415
|
+
const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
|
|
355
416
|
if (re.test(document.title.trim())) return true
|
|
356
417
|
if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
|
|
357
418
|
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
358
419
|
for (const script of Array.from(document.scripts)) {
|
|
359
|
-
const src = script.getAttribute('src') || ''
|
|
360
|
-
if (src.includes(
|
|
420
|
+
const src = (script.getAttribute('src') || '').toLowerCase()
|
|
421
|
+
if (src.includes(platform) && src.includes(runtime)) return true
|
|
361
422
|
}
|
|
362
423
|
if (document.querySelector('.footer .footer-inner .ray-id')) return true
|
|
363
424
|
return false
|
|
@@ -998,11 +1059,16 @@ function existsAsFile(path) {
|
|
|
998
1059
|
* an error. Chromium can make that awkward: it refuses to render status-only
|
|
999
1060
|
* responses (error statuses with an empty body — httpbin's `/status/N` —
|
|
1000
1061
|
* plus 204/205), failing the whole navigation with
|
|
1001
|
-
* `net::ERR_HTTP_RESPONSE_CODE_FAILURE` / `net::ERR_ABORTED
|
|
1002
|
-
*
|
|
1003
|
-
*
|
|
1004
|
-
*
|
|
1005
|
-
*
|
|
1062
|
+
* `net::ERR_HTTP_RESPONSE_CODE_FAILURE` / `net::ERR_ABORTED`, and it hands a
|
|
1063
|
+
* `Content-Disposition: attachment` response (publish.twitter.com's oembed
|
|
1064
|
+
* JSON, the URL agents use to read a tweet) to its download manager instead
|
|
1065
|
+
* of committing a document, failing `goto` with "Download is starting" —
|
|
1066
|
+
* even though the response itself arrived and is decodable. The last
|
|
1067
|
+
* main-frame response is therefore tracked on every fetch, and such failed
|
|
1068
|
+
* `goto`s are recovered — the status-only shape into the status-bearing
|
|
1069
|
+
* result ({@link recoverUnrenderedStatus}), the download shape via a
|
|
1070
|
+
* refetch through the page's API request context
|
|
1071
|
+
* ({@link recoverDownloadResponse}) — instead of a `WEB_PROVIDER_ERROR`.
|
|
1006
1072
|
*
|
|
1007
1073
|
* Lifecycle: the local backend launches a browser per fetch and closes it —
|
|
1008
1074
|
* nothing outlives the call. The CDP backend keeps ONE shared connection to
|
|
@@ -1284,6 +1350,7 @@ var PlaywrightFetchProvider = class {
|
|
|
1284
1350
|
const lease = await this.cdpPool.acquire(endpoint, timeout, effectiveContextMode(config));
|
|
1285
1351
|
await installResourceFilter(lease.page);
|
|
1286
1352
|
guardPopups(lease.page);
|
|
1353
|
+
guardDownloads(lease.page);
|
|
1287
1354
|
return {
|
|
1288
1355
|
browser: lease.browser,
|
|
1289
1356
|
context: lease.context,
|
|
@@ -1307,6 +1374,7 @@ var PlaywrightFetchProvider = class {
|
|
|
1307
1374
|
const page = await context.newPage();
|
|
1308
1375
|
await installResourceFilter(page);
|
|
1309
1376
|
guardPopups(page);
|
|
1377
|
+
guardDownloads(page);
|
|
1310
1378
|
return {
|
|
1311
1379
|
browser,
|
|
1312
1380
|
context,
|
|
@@ -1385,9 +1453,12 @@ var PlaywrightFetchProvider = class {
|
|
|
1385
1453
|
}
|
|
1386
1454
|
/**
|
|
1387
1455
|
* One tracked `goto`: the navigation response on success, or — when
|
|
1388
|
-
* Chromium failed the navigation
|
|
1389
|
-
*
|
|
1390
|
-
*
|
|
1456
|
+
* Chromium failed the navigation for a reason that is not a transport
|
|
1457
|
+
* error — the recovered finished result. Two such reasons are recovered:
|
|
1458
|
+
* the HTTP status it received (an error status with an empty body, or a
|
|
1459
|
+
* status-only 204/205 — {@link recoverUnrenderedStatus}), and the response
|
|
1460
|
+
* being a download the browser refuses to commit as a document
|
|
1461
|
+
* (`Content-Disposition: attachment` — {@link recoverDownloadResponse}).
|
|
1391
1462
|
*
|
|
1392
1463
|
* @returns the response `goto` resolved with, plus `recovered` when the
|
|
1393
1464
|
* navigation failure was recovered into a finished result the caller must
|
|
@@ -1406,6 +1477,11 @@ var PlaywrightFetchProvider = class {
|
|
|
1406
1477
|
response: null,
|
|
1407
1478
|
recovered
|
|
1408
1479
|
};
|
|
1480
|
+
const download = await recoverDownloadResponse(error, page, tracker, url, deadline, denoise);
|
|
1481
|
+
if (download !== void 0) return {
|
|
1482
|
+
response: null,
|
|
1483
|
+
recovered: download
|
|
1484
|
+
};
|
|
1409
1485
|
}
|
|
1410
1486
|
throw error;
|
|
1411
1487
|
}
|
|
@@ -1570,6 +1646,89 @@ async function recoverUnrenderedStatus(error, tracker, url, denoise) {
|
|
|
1570
1646
|
});
|
|
1571
1647
|
}
|
|
1572
1648
|
/**
|
|
1649
|
+
* Whether a `goto` throw is Chromium handing the navigation to its download
|
|
1650
|
+
* manager instead of committing a document. Playwright rejects with
|
|
1651
|
+
* "Download is starting" when the navigation response carries
|
|
1652
|
+
* `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint —
|
|
1653
|
+
* the workaround URL agents use to read a tweet — serves exactly that:
|
|
1654
|
+
* decodable `application/json` the browser still refuses to render). The
|
|
1655
|
+
* attachment disposition on the tracked response is a second,
|
|
1656
|
+
* engine-version-proof signal for engines that report a plain `ERR_ABORTED`
|
|
1657
|
+
* instead; note `ERR_ABORTED` alone is NOT enough (the status recovery
|
|
1658
|
+
* above legitimately owns the 204/205 shape, and SPA cancels abort too).
|
|
1659
|
+
*
|
|
1660
|
+
* @param error - the value `page.goto` rejected with.
|
|
1661
|
+
* @param response - the last tracked main-frame response, if any.
|
|
1662
|
+
* @returns true when the navigation failed because it became a download.
|
|
1663
|
+
*/
|
|
1664
|
+
function isDownloadFailure(error, response) {
|
|
1665
|
+
if (!(error instanceof Error)) return false;
|
|
1666
|
+
if (error.message.includes("Download is starting")) return true;
|
|
1667
|
+
if (!error.message.includes("ERR_ABORTED")) return false;
|
|
1668
|
+
return (response?.headers()["content-disposition"] ?? "").toLowerCase().includes("attachment");
|
|
1669
|
+
}
|
|
1670
|
+
/**
|
|
1671
|
+
* Recover a download-turned navigation into the finished fetch result the
|
|
1672
|
+
* seam's contract demands. The resource itself is often perfectly decodable
|
|
1673
|
+
* — the oembed endpoint's `application/json` classifies as `text` — the
|
|
1674
|
+
* browser just will not commit it as a document, and the response it tracked
|
|
1675
|
+
* cannot yield its body either (Chromium streams a download to its download
|
|
1676
|
+
* manager, so `Network.getResponseBody` answers "No resource with given
|
|
1677
|
+
* identifier found"; both verified against a real browser). The body is
|
|
1678
|
+
* therefore refetched through the page's API request context: plain HTTP
|
|
1679
|
+
* from the Playwright client — same cookie jar, same URL, no render-or-
|
|
1680
|
+
* download decision — which works identically on the local and CDP
|
|
1681
|
+
* backends because it never touches the remote artifact.
|
|
1682
|
+
*
|
|
1683
|
+
* Undecodable downloads (a PDF, a ZIP) still fail with
|
|
1684
|
+
* `WEB_UNSUPPORTED_CONTENT_TYPE`, classified from what the navigation
|
|
1685
|
+
* actually served; a playwright build without a request context surfaces a
|
|
1686
|
+
* clear "the browser will not render this" error instead of the raw
|
|
1687
|
+
* "Download is starting".
|
|
1688
|
+
*
|
|
1689
|
+
* @param error - the value `page.goto` rejected with.
|
|
1690
|
+
* @param page - the fetch's tab, for its API request context.
|
|
1691
|
+
* @param tracker - the fetch's main-frame response record.
|
|
1692
|
+
* @param url - the request URL (fallback when the response carries none).
|
|
1693
|
+
* @param deadline - the fetch budget, applied to the refetch.
|
|
1694
|
+
* @param denoise - the settings toggle, honored as on the normal HTML path.
|
|
1695
|
+
* @returns the finished fetch result, or undefined when the failure is not
|
|
1696
|
+
* a download navigation (the throw should propagate).
|
|
1697
|
+
*/
|
|
1698
|
+
async function recoverDownloadResponse(error, page, tracker, url, deadline, denoise) {
|
|
1699
|
+
const tracked = tracker.last();
|
|
1700
|
+
if (!isDownloadFailure(error, tracked)) return void 0;
|
|
1701
|
+
if (tracked !== null) {
|
|
1702
|
+
if (classifyContentType(tracked.headers()["content-type"]) === void 0) throw new WebError(`unsupported content type "${tracked.headers()["content-type"] ?? "unknown"}" (served as a download)`, "WEB_UNSUPPORTED_CONTENT_TYPE");
|
|
1703
|
+
}
|
|
1704
|
+
if (page.request === void 0) throw new WebError("the response is a file download (Content-Disposition: attachment) the browser will not render, and this playwright build exposes no request context to refetch it", "WEB_PROVIDER_ERROR", { cause: error });
|
|
1705
|
+
const fetchUrl = tracked?.url?.() ?? url.toString();
|
|
1706
|
+
const refetched = await page.request.get(fetchUrl, {
|
|
1707
|
+
timeout: deadline.remainingMs(),
|
|
1708
|
+
signal: deadline.signal
|
|
1709
|
+
});
|
|
1710
|
+
const headers = refetched.headers();
|
|
1711
|
+
const servedType = (headers["content-type"] ?? "").trim() !== "" ? headers["content-type"] : tracked?.headers()["content-type"];
|
|
1712
|
+
const kind = classifyContentType(servedType);
|
|
1713
|
+
if (kind === void 0) throw new WebError(`unsupported content type "${servedType ?? "unknown"}"`, "WEB_UNSUPPORTED_CONTENT_TYPE");
|
|
1714
|
+
const finalUrl = refetched.url?.() ?? fetchUrl;
|
|
1715
|
+
const statusCode = refetched.status();
|
|
1716
|
+
const content = await refetched.text();
|
|
1717
|
+
if (kind === "text") return capResult(finalUrl, statusCode, {
|
|
1718
|
+
kind: "text",
|
|
1719
|
+
content
|
|
1720
|
+
});
|
|
1721
|
+
if (!denoise) return capResult(finalUrl, statusCode, {
|
|
1722
|
+
kind: "html",
|
|
1723
|
+
content
|
|
1724
|
+
});
|
|
1725
|
+
const { markdown } = htmlToMarkdown(content, finalUrl);
|
|
1726
|
+
return capResult(finalUrl, statusCode, {
|
|
1727
|
+
kind: "text",
|
|
1728
|
+
content: markdown
|
|
1729
|
+
});
|
|
1730
|
+
}
|
|
1731
|
+
/**
|
|
1573
1732
|
* Probe the live document for challenge markers. Prefers the in-page probe
|
|
1574
1733
|
* (`page.evaluate`) — the only thing that can see SPA-style clears — and
|
|
1575
1734
|
* falls back to reading and classifying the serialized HTML when the backend
|
|
@@ -1654,6 +1813,22 @@ async function installResourceFilter(owner) {
|
|
|
1654
1813
|
} catch {}
|
|
1655
1814
|
}
|
|
1656
1815
|
/**
|
|
1816
|
+
* Cancel any download Chromium starts on the fetch's tab. A navigation whose
|
|
1817
|
+
* response carries `Content-Disposition: attachment` never commits a
|
|
1818
|
+
* document — `goto` rejects with "Download is starting" and the recovery
|
|
1819
|
+
* refetches the bytes (see {@link recoverDownloadResponse}) — but the
|
|
1820
|
+
* browser still hands the original body to its download manager, which in
|
|
1821
|
+
* profile mode would drop files into the remote machine's download dir.
|
|
1822
|
+
* Best-effort: a page without the listener just loses the guard.
|
|
1823
|
+
*/
|
|
1824
|
+
function guardDownloads(page) {
|
|
1825
|
+
try {
|
|
1826
|
+
page.on?.("download", (download) => {
|
|
1827
|
+
download.cancel().catch(() => {});
|
|
1828
|
+
});
|
|
1829
|
+
} catch {}
|
|
1830
|
+
}
|
|
1831
|
+
/**
|
|
1657
1832
|
* Close any popup a fetched page spawns so nothing outlives the fetch's tab
|
|
1658
1833
|
* — `page.close()` does not auto-close windows the page opened, and in
|
|
1659
1834
|
* profile mode a stray tab would stay in the user's remote browser.
|
package/lib/types/challenge.d.ts
CHANGED
|
@@ -16,10 +16,12 @@
|
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
17
|
* `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
|
|
18
18
|
* deliberately structural: a blog post that merely MENTIONS Cloudflare
|
|
19
|
-
* must not classify as a challenge
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
19
|
+
* must not classify as a challenge, so the challenge-platform script
|
|
20
|
+
* check reads `<script src>` attributes only (never prose or escaped
|
|
21
|
+
* code samples) and identifies the runtime POSITIVELY by its
|
|
22
|
+
* `/orchestrate/` mount — the JSD/Precursor instrumentation Cloudflare
|
|
23
|
+
* injects into every NORMAL page of a protected zone, on several moving
|
|
24
|
+
* mounts, is not a challenge signal.
|
|
23
25
|
*
|
|
24
26
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
25
27
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -81,9 +83,9 @@ export declare function classifyChallengeHtml(html: string): ChallengeVerdict;
|
|
|
81
83
|
* what makes SPA-style clears visible — the document swaps its content
|
|
82
84
|
* without any navigation, so only a DOM read can see the change. The title
|
|
83
85
|
* regex is baked in at build time from the same single source the Node-side
|
|
84
|
-
* check uses, so the two can never drift apart. Script
|
|
85
|
-
*
|
|
86
|
-
*
|
|
87
|
-
*
|
|
86
|
+
* check uses, so the two can never drift apart. Script evidence follows the
|
|
87
|
+
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
88
|
+
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
89
|
+
* protected zone is ignored.
|
|
88
90
|
*/
|
|
89
91
|
export declare const CHALLENGE_DOM_PROBE: string;
|
package/lib/types/provider.d.ts
CHANGED
|
@@ -7,11 +7,16 @@
|
|
|
7
7
|
* an error. Chromium can make that awkward: it refuses to render status-only
|
|
8
8
|
* responses (error statuses with an empty body — httpbin's `/status/N` —
|
|
9
9
|
* plus 204/205), failing the whole navigation with
|
|
10
|
-
* `net::ERR_HTTP_RESPONSE_CODE_FAILURE` / `net::ERR_ABORTED
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
14
|
-
*
|
|
10
|
+
* `net::ERR_HTTP_RESPONSE_CODE_FAILURE` / `net::ERR_ABORTED`, and it hands a
|
|
11
|
+
* `Content-Disposition: attachment` response (publish.twitter.com's oembed
|
|
12
|
+
* JSON, the URL agents use to read a tweet) to its download manager instead
|
|
13
|
+
* of committing a document, failing `goto` with "Download is starting" —
|
|
14
|
+
* even though the response itself arrived and is decodable. The last
|
|
15
|
+
* main-frame response is therefore tracked on every fetch, and such failed
|
|
16
|
+
* `goto`s are recovered — the status-only shape into the status-bearing
|
|
17
|
+
* result ({@link recoverUnrenderedStatus}), the download shape via a
|
|
18
|
+
* refetch through the page's API request context
|
|
19
|
+
* ({@link recoverDownloadResponse}) — instead of a `WEB_PROVIDER_ERROR`.
|
|
15
20
|
*
|
|
16
21
|
* Lifecycle: the local backend launches a browser per fetch and closes it —
|
|
17
22
|
* nothing outlives the call. The CDP backend keeps ONE shared connection to
|
|
@@ -132,9 +137,12 @@ export declare class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
132
137
|
private retrieve;
|
|
133
138
|
/**
|
|
134
139
|
* One tracked `goto`: the navigation response on success, or — when
|
|
135
|
-
* Chromium failed the navigation
|
|
136
|
-
*
|
|
137
|
-
*
|
|
140
|
+
* Chromium failed the navigation for a reason that is not a transport
|
|
141
|
+
* error — the recovered finished result. Two such reasons are recovered:
|
|
142
|
+
* the HTTP status it received (an error status with an empty body, or a
|
|
143
|
+
* status-only 204/205 — {@link recoverUnrenderedStatus}), and the response
|
|
144
|
+
* being a download the browser refuses to commit as a document
|
|
145
|
+
* (`Content-Disposition: attachment` — {@link recoverDownloadResponse}).
|
|
138
146
|
*
|
|
139
147
|
* @returns the response `goto` resolved with, plus `recovered` when the
|
|
140
148
|
* navigation failure was recovered into a finished result the caller must
|
package/lib/types/types.d.ts
CHANGED
|
@@ -20,6 +20,27 @@ export interface PlaywrightResponse {
|
|
|
20
20
|
*/
|
|
21
21
|
request?(): PlaywrightRequest;
|
|
22
22
|
}
|
|
23
|
+
/**
|
|
24
|
+
* A download Chromium started instead of committing a document (a response
|
|
25
|
+
* with `Content-Disposition: attachment`).
|
|
26
|
+
*/
|
|
27
|
+
export interface PlaywrightDownload {
|
|
28
|
+
cancel(): Promise<void>;
|
|
29
|
+
}
|
|
30
|
+
/**
|
|
31
|
+
* The page's `APIRequestContext` surface the download recovery refetches
|
|
32
|
+
* through — plain HTTP from the Playwright client, the context's cookie jar,
|
|
33
|
+
* and none of Chromium's render-or-download semantics. Reuses
|
|
34
|
+
* {@link PlaywrightResponse} because that is structurally what an
|
|
35
|
+
* `APIResponse` is for the members this plugin reads. Absent on fakes that
|
|
36
|
+
* do not exercise the recovery.
|
|
37
|
+
*/
|
|
38
|
+
export interface PlaywrightRequestContext {
|
|
39
|
+
get(url: string, options?: {
|
|
40
|
+
timeout?: number;
|
|
41
|
+
signal?: AbortSignal;
|
|
42
|
+
}): Promise<PlaywrightResponse>;
|
|
43
|
+
}
|
|
23
44
|
/** The request side of a response, for main-frame filtering. */
|
|
24
45
|
export interface PlaywrightRequest {
|
|
25
46
|
/** True for navigations (document loads and their redirect hops). */
|
|
@@ -54,6 +75,13 @@ export interface PlaywrightPage {
|
|
|
54
75
|
* document). Absent on minimal fakes (content polling covers them).
|
|
55
76
|
*/
|
|
56
77
|
on?(event: 'response', listener: (response: PlaywrightResponse) => void): unknown;
|
|
78
|
+
/**
|
|
79
|
+
* Download notification — a navigation the browser handed to its download
|
|
80
|
+
* manager instead of committing a document (`Content-Disposition:
|
|
81
|
+
* attachment`); the fetch's guard cancels it so nothing lands on disk.
|
|
82
|
+
* Absent on minimal fakes (the recovery does not depend on it).
|
|
83
|
+
*/
|
|
84
|
+
on?(event: 'download', listener: (download: PlaywrightDownload) => void): unknown;
|
|
57
85
|
/**
|
|
58
86
|
* Evaluate an expression in the page — the challenge probe's live-DOM
|
|
59
87
|
* path. Absent on minimal fakes (content polling covers them).
|
|
@@ -61,6 +89,11 @@ export interface PlaywrightPage {
|
|
|
61
89
|
evaluate?(script: string, arg?: unknown): Promise<unknown>;
|
|
62
90
|
/** The main frame handle; compare with `request.frame()` for filtering. */
|
|
63
91
|
mainFrame?(): unknown;
|
|
92
|
+
/**
|
|
93
|
+
* The page's API request context — the download recovery's refetch path
|
|
94
|
+
* (see {@link PlaywrightRequestContext}). Absent on minimal fakes.
|
|
95
|
+
*/
|
|
96
|
+
request?: PlaywrightRequestContext;
|
|
64
97
|
}
|
|
65
98
|
/**
|
|
66
99
|
* A browser context: fetch-owned and isolated (local backend, or CDP
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-web-fetch-playwright",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.12",
|
|
4
4
|
"description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/src/challenge.ts
CHANGED
|
@@ -16,10 +16,12 @@
|
|
|
16
16
|
* (`/cdn-cgi/challenge-platform/` scripts, `#challenge-*` elements,
|
|
17
17
|
* `cf-chl-widget-` Turnstile frames, `window._cf_chl_opt`). Markers are
|
|
18
18
|
* deliberately structural: a blog post that merely MENTIONS Cloudflare
|
|
19
|
-
* must not classify as a challenge
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
19
|
+
* must not classify as a challenge, so the challenge-platform script
|
|
20
|
+
* check reads `<script src>` attributes only (never prose or escaped
|
|
21
|
+
* code samples) and identifies the runtime POSITIVELY by its
|
|
22
|
+
* `/orchestrate/` mount — the JSD/Precursor instrumentation Cloudflare
|
|
23
|
+
* injects into every NORMAL page of a protected zone, on several moving
|
|
24
|
+
* mounts, is not a challenge signal.
|
|
23
25
|
*
|
|
24
26
|
* The security stance (issue #2): this module only DETECTS and classifies.
|
|
25
27
|
* Solving, spoofing, CAPTCHA answering, and cookie lifting live outside the
|
|
@@ -89,30 +91,93 @@ export const CHALLENGE_TITLE_RE = new RegExp(
|
|
|
89
91
|
)
|
|
90
92
|
|
|
91
93
|
/**
|
|
92
|
-
* Cloudflare
|
|
93
|
-
*
|
|
94
|
-
*
|
|
95
|
-
*
|
|
96
|
-
*
|
|
97
|
-
*
|
|
98
|
-
*
|
|
99
|
-
*
|
|
100
|
-
*
|
|
94
|
+
* Cloudflare's challenge-platform namespace holds two very different things,
|
|
95
|
+
* and telling them apart is the whole game:
|
|
96
|
+
*
|
|
97
|
+
* - the challenge RUNTIME — `/cdn-cgi/challenge-platform/h/[b|g]/orchestrate/…`
|
|
98
|
+
* — which real interstitials load to run the verification;
|
|
99
|
+
* - PASSIVE INSTRUMENTATION — everything else Cloudflare injects into NORMAL
|
|
100
|
+
* pages of a protected zone, as a direct `<script src>` or as the URL
|
|
101
|
+
* strung inside the hidden-1x1-iframe bootstrap's inline script text. The
|
|
102
|
+
* page itself is a plain 200 of real content.
|
|
103
|
+
*
|
|
104
|
+
* Observed instrumentation layouts (a thread that lists all of them:
|
|
105
|
+
* `community.cloudflare.com/t/3-javascripts-are-added-unable-to-disable`):
|
|
106
|
+
*
|
|
107
|
+
* - `/cdn-cgi/challenge-platform/scripts/jsd/…` — JavaScript Detections;
|
|
108
|
+
* - `/cdn-cgi/challenge-platform/scripts/precursor/…` — Precursor, the
|
|
109
|
+
* continuous behavioral-validation engine;
|
|
110
|
+
* - `/cdn-cgi/challenge-platform/h/[b|g]/scripts/jsd/<hash>/…` and
|
|
111
|
+
* `/cdn-cgi/challenge-platform/h/[b|g]/jsd/r/…` — the same detections
|
|
112
|
+
* served from a hash-scoped runtime mount, with no `scripts/` segment.
|
|
113
|
+
*
|
|
114
|
+
* Enumerating instrumentation names (`jsd`, `precursor`) or carving out a
|
|
115
|
+
* `scripts/` namespace is a standing false-positive subscription: Cloudflare
|
|
116
|
+
* keeps ADDING packages and keeps moving them onto new mounts (openrouter.ai
|
|
117
|
+
* broke on JSD, nodeseek.com broke on Precursor, and the hash-scoped variant
|
|
118
|
+
* defeats a `scripts/`-only carve-out). So the runtime is identified
|
|
119
|
+
* POSITIVELY instead: `orchestrate` is the challenge runtime's own mount and
|
|
120
|
+
* never appears in instrumentation. Everything else under the prefix is
|
|
121
|
+
* treated as passive — the directive that a protected page must stay
|
|
122
|
+
* readable.
|
|
123
|
+
*
|
|
124
|
+
* Losing an unknown future runtime mount costs little here: the response
|
|
125
|
+
* header, the localized title family, and the challenge DOM elements remain
|
|
126
|
+
* independent signals, and the content-level script check is only their
|
|
127
|
+
* fallback.
|
|
101
128
|
*/
|
|
102
|
-
const
|
|
129
|
+
const CHALLENGE_PLATFORM_PREFIX = '/cdn-cgi/challenge-platform/'
|
|
130
|
+
|
|
131
|
+
/** The path segment that identifies the challenge runtime (not telemetry). */
|
|
132
|
+
const CHALLENGE_RUNTIME_SEGMENT = '/orchestrate/'
|
|
133
|
+
|
|
134
|
+
/**
|
|
135
|
+
* A `<script src>` value in document HTML: double-quoted, single-quoted, or
|
|
136
|
+
* bare. Only real `src` attributes are read, so prose and escaped code
|
|
137
|
+
* samples that merely quote a path can never match (badcase:
|
|
138
|
+
* `community.cloudflare.com/t/3-javascripts-are-added-unable-to-disable`
|
|
139
|
+
* — a thread ABOUT these injected scripts, which lists
|
|
140
|
+
* `/cdn-cgi/challenge-platform/scripts/jsd/main.js` in its body text and was
|
|
141
|
+
* misclassified as a challenge by the old whole-document substring scan).
|
|
142
|
+
*/
|
|
143
|
+
const SCRIPT_SRC_RE = /<script\b[^>]*?\bsrc\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>]+))/gi
|
|
144
|
+
|
|
145
|
+
/**
|
|
146
|
+
* Extract every `<script src>` value from a document's HTML, lowercased.
|
|
147
|
+
*
|
|
148
|
+
* @param html - the document HTML as `page.content()` returns it.
|
|
149
|
+
* @returns the script source URLs, in document order.
|
|
150
|
+
*/
|
|
151
|
+
function scriptSrcsOf(html: string): string[] {
|
|
152
|
+
const srcs: string[] = []
|
|
153
|
+
SCRIPT_SRC_RE.lastIndex = 0
|
|
154
|
+
for (let match = SCRIPT_SRC_RE.exec(html); match !== null; match = SCRIPT_SRC_RE.exec(html)) {
|
|
155
|
+
srcs.push((match[1] ?? match[2] ?? match[3] ?? '').toLowerCase())
|
|
156
|
+
}
|
|
157
|
+
return srcs
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
/**
|
|
161
|
+
* Whether a script src belongs to the challenge RUNTIME rather than to the
|
|
162
|
+
* passive instrumentation Cloudflare injects into normal pages.
|
|
163
|
+
*
|
|
164
|
+
* @param src - a lowercased script source URL.
|
|
165
|
+
* @returns true when the script is the challenge runtime.
|
|
166
|
+
*/
|
|
167
|
+
function isChallengeRuntimeScript(src: string): boolean {
|
|
168
|
+
return src.includes(CHALLENGE_PLATFORM_PREFIX) && src.includes(CHALLENGE_RUNTIME_SEGMENT)
|
|
169
|
+
}
|
|
103
170
|
|
|
104
171
|
/**
|
|
105
172
|
* Structural markers a challenge document carries in its HTML. Kept in
|
|
106
173
|
* attribute/assignment form (`id="…"`, `window._cf_chl_opt`) rather than bare
|
|
107
174
|
* substrings so an article that merely QUOTES these strings in prose or code
|
|
108
175
|
* samples does not classify — the marker must look like the real element.
|
|
109
|
-
* The
|
|
110
|
-
* {@link
|
|
111
|
-
*
|
|
112
|
-
* passive telemetry and deliberately absent from this list.
|
|
176
|
+
* The challenge-platform prefix is deliberately absent: script evidence goes
|
|
177
|
+
* through {@link scriptSrcsOf} + {@link isChallengeRuntimeScript}, which
|
|
178
|
+
* distinguish a real `src` from prose and runtime from instrumentation.
|
|
113
179
|
*/
|
|
114
180
|
const CHALLENGE_HTML_MARKERS = [
|
|
115
|
-
'/cdn-cgi/challenge-platform/',
|
|
116
181
|
'id="cf-chl-widget-',
|
|
117
182
|
'window._cf_chl_opt',
|
|
118
183
|
'id="challenge-form"',
|
|
@@ -206,14 +271,15 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
|
|
|
206
271
|
const lowered = html.toLowerCase()
|
|
207
272
|
if (lowered.includes(BLOCKED_BODY_PHRASE) && BLOCKED_BODY_MARKERS.some(marker => lowered.includes(marker))) return 'blocked'
|
|
208
273
|
if (CHALLENGE_TITLE_RE.test(title)) return 'challenge'
|
|
209
|
-
//
|
|
210
|
-
//
|
|
211
|
-
//
|
|
212
|
-
//
|
|
213
|
-
//
|
|
214
|
-
//
|
|
215
|
-
|
|
216
|
-
if (
|
|
274
|
+
// Script evidence BEFORE the text markers: only a real `<script src>`
|
|
275
|
+
// counts, and only when it is the challenge runtime rather than passive
|
|
276
|
+
// instrumentation. Both halves matter — scanning prose false-positived on
|
|
277
|
+
// an article quoting these paths, and reading the prefix as a challenge
|
|
278
|
+
// false-positived on every protected normal page (openrouter.ai's JSD,
|
|
279
|
+
// nodeseek.com's Precursor). Either way the wait can never confirm a
|
|
280
|
+
// "clear", and a page that loaded fine dies with WEB_FETCH_CHALLENGE.
|
|
281
|
+
if (scriptSrcsOf(html).some(isChallengeRuntimeScript)) return 'challenge'
|
|
282
|
+
if (CHALLENGE_HTML_MARKERS.some(marker => lowered.includes(marker))) return 'challenge'
|
|
217
283
|
// Crawlee's footer marker, as a two-class combo so prose quoting one class
|
|
218
284
|
// alone stays clean.
|
|
219
285
|
if (lowered.includes('class="ray-id"') && lowered.includes('class="footer-inner"')) return 'challenge'
|
|
@@ -226,19 +292,21 @@ export function classifyChallengeHtml(html: string): ChallengeVerdict {
|
|
|
226
292
|
* what makes SPA-style clears visible — the document swaps its content
|
|
227
293
|
* without any navigation, so only a DOM read can see the change. The title
|
|
228
294
|
* regex is baked in at build time from the same single source the Node-side
|
|
229
|
-
* check uses, so the two can never drift apart. Script
|
|
230
|
-
*
|
|
231
|
-
*
|
|
232
|
-
*
|
|
295
|
+
* check uses, so the two can never drift apart. Script evidence follows the
|
|
296
|
+
* same rule as {@link classifyChallengeHtml}: only the challenge runtime
|
|
297
|
+
* counts, and the instrumentation Cloudflare injects into normal pages of a
|
|
298
|
+
* protected zone is ignored.
|
|
233
299
|
*/
|
|
234
300
|
export const CHALLENGE_DOM_PROBE = `(() => {
|
|
235
301
|
const re = new RegExp(${JSON.stringify(CHALLENGE_TITLE_RE.source)}, ${JSON.stringify(CHALLENGE_TITLE_RE.flags)})
|
|
302
|
+
const platform = ${JSON.stringify(CHALLENGE_PLATFORM_PREFIX)}
|
|
303
|
+
const runtime = ${JSON.stringify(CHALLENGE_RUNTIME_SEGMENT)}
|
|
236
304
|
if (re.test(document.title.trim())) return true
|
|
237
305
|
if (document.querySelector('#challenge-form, #challenge-running, #challenge-stage, #challenge-error-text')) return true
|
|
238
306
|
if (document.querySelector('[id^="cf-chl-widget-"]')) return true
|
|
239
307
|
for (const script of Array.from(document.scripts)) {
|
|
240
|
-
const src = script.getAttribute('src') || ''
|
|
241
|
-
if (src.includes(
|
|
308
|
+
const src = (script.getAttribute('src') || '').toLowerCase()
|
|
309
|
+
if (src.includes(platform) && src.includes(runtime)) return true
|
|
242
310
|
}
|
|
243
311
|
if (document.querySelector('.footer .footer-inner .ray-id')) return true
|
|
244
312
|
return false
|
package/src/provider.ts
CHANGED
|
@@ -7,11 +7,16 @@
|
|
|
7
7
|
* an error. Chromium can make that awkward: it refuses to render status-only
|
|
8
8
|
* responses (error statuses with an empty body — httpbin's `/status/N` —
|
|
9
9
|
* plus 204/205), failing the whole navigation with
|
|
10
|
-
* `net::ERR_HTTP_RESPONSE_CODE_FAILURE` / `net::ERR_ABORTED
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
14
|
-
*
|
|
10
|
+
* `net::ERR_HTTP_RESPONSE_CODE_FAILURE` / `net::ERR_ABORTED`, and it hands a
|
|
11
|
+
* `Content-Disposition: attachment` response (publish.twitter.com's oembed
|
|
12
|
+
* JSON, the URL agents use to read a tweet) to its download manager instead
|
|
13
|
+
* of committing a document, failing `goto` with "Download is starting" —
|
|
14
|
+
* even though the response itself arrived and is decodable. The last
|
|
15
|
+
* main-frame response is therefore tracked on every fetch, and such failed
|
|
16
|
+
* `goto`s are recovered — the status-only shape into the status-bearing
|
|
17
|
+
* result ({@link recoverUnrenderedStatus}), the download shape via a
|
|
18
|
+
* refetch through the page's API request context
|
|
19
|
+
* ({@link recoverDownloadResponse}) — instead of a `WEB_PROVIDER_ERROR`.
|
|
15
20
|
*
|
|
16
21
|
* Lifecycle: the local backend launches a browser per fetch and closes it —
|
|
17
22
|
* nothing outlives the call. The CDP backend keeps ONE shared connection to
|
|
@@ -55,7 +60,7 @@ import type { ResolvedConfig } from './config.ts'
|
|
|
55
60
|
import { CdpConnectionPool } from './cdp-pool.ts'
|
|
56
61
|
import { htmlToMarkdown } from './markdown.ts'
|
|
57
62
|
import { resolveCdpBackend, resolvePlaywrightBackend } from './playwright-resolve.ts'
|
|
58
|
-
import type { PlaywrightBrowser, PlaywrightContext, PlaywrightPage, PlaywrightResponse, PlaywrightRoute } from './types.ts'
|
|
63
|
+
import type { PlaywrightBrowser, PlaywrightContext, PlaywrightDownload, PlaywrightPage, PlaywrightResponse, PlaywrightRoute } from './types.ts'
|
|
59
64
|
|
|
60
65
|
/** Stable id this provider registers under (the bundle patch pins it). */
|
|
61
66
|
export const PLAYWRIGHT_FETCH_PROVIDER_ID = 'playwright'
|
|
@@ -373,6 +378,7 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
373
378
|
const lease = await this.cdpPool.acquire(endpoint, timeout, effectiveContextMode(config))
|
|
374
379
|
await installResourceFilter(lease.page)
|
|
375
380
|
guardPopups(lease.page)
|
|
381
|
+
guardDownloads(lease.page)
|
|
376
382
|
return {
|
|
377
383
|
browser: lease.browser,
|
|
378
384
|
context: lease.context,
|
|
@@ -396,6 +402,7 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
396
402
|
const page = await context.newPage()
|
|
397
403
|
await installResourceFilter(page)
|
|
398
404
|
guardPopups(page)
|
|
405
|
+
guardDownloads(page)
|
|
399
406
|
return { browser, context, page }
|
|
400
407
|
} catch (error: unknown) {
|
|
401
408
|
// A partial setup (browser launched, then newContext/newPage failed)
|
|
@@ -515,9 +522,12 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
515
522
|
|
|
516
523
|
/**
|
|
517
524
|
* One tracked `goto`: the navigation response on success, or — when
|
|
518
|
-
* Chromium failed the navigation
|
|
519
|
-
*
|
|
520
|
-
*
|
|
525
|
+
* Chromium failed the navigation for a reason that is not a transport
|
|
526
|
+
* error — the recovered finished result. Two such reasons are recovered:
|
|
527
|
+
* the HTTP status it received (an error status with an empty body, or a
|
|
528
|
+
* status-only 204/205 — {@link recoverUnrenderedStatus}), and the response
|
|
529
|
+
* being a download the browser refuses to commit as a document
|
|
530
|
+
* (`Content-Disposition: attachment` — {@link recoverDownloadResponse}).
|
|
521
531
|
*
|
|
522
532
|
* @returns the response `goto` resolved with, plus `recovered` when the
|
|
523
533
|
* navigation failure was recovered into a finished result the caller must
|
|
@@ -539,6 +549,8 @@ export class PlaywrightFetchProvider implements WebFetchProvider {
|
|
|
539
549
|
if (!deadline.signal.aborted) {
|
|
540
550
|
const recovered = await recoverUnrenderedStatus(error, tracker, url, denoise)
|
|
541
551
|
if (recovered !== undefined) return { response: null, recovered }
|
|
552
|
+
const download = await recoverDownloadResponse(error, page, tracker, url, deadline, denoise)
|
|
553
|
+
if (download !== undefined) return { response: null, recovered: download }
|
|
542
554
|
}
|
|
543
555
|
throw error
|
|
544
556
|
}
|
|
@@ -737,6 +749,104 @@ async function recoverUnrenderedStatus(
|
|
|
737
749
|
return capResult(finalUrl, status, { kind: 'text', content: markdown })
|
|
738
750
|
}
|
|
739
751
|
|
|
752
|
+
/**
|
|
753
|
+
* Whether a `goto` throw is Chromium handing the navigation to its download
|
|
754
|
+
* manager instead of committing a document. Playwright rejects with
|
|
755
|
+
* "Download is starting" when the navigation response carries
|
|
756
|
+
* `Content-Disposition: attachment` (publish.twitter.com's oembed endpoint —
|
|
757
|
+
* the workaround URL agents use to read a tweet — serves exactly that:
|
|
758
|
+
* decodable `application/json` the browser still refuses to render). The
|
|
759
|
+
* attachment disposition on the tracked response is a second,
|
|
760
|
+
* engine-version-proof signal for engines that report a plain `ERR_ABORTED`
|
|
761
|
+
* instead; note `ERR_ABORTED` alone is NOT enough (the status recovery
|
|
762
|
+
* above legitimately owns the 204/205 shape, and SPA cancels abort too).
|
|
763
|
+
*
|
|
764
|
+
* @param error - the value `page.goto` rejected with.
|
|
765
|
+
* @param response - the last tracked main-frame response, if any.
|
|
766
|
+
* @returns true when the navigation failed because it became a download.
|
|
767
|
+
*/
|
|
768
|
+
function isDownloadFailure(error: unknown, response: PlaywrightResponse | null): boolean {
|
|
769
|
+
if (!(error instanceof Error)) return false
|
|
770
|
+
if (error.message.includes('Download is starting')) return true
|
|
771
|
+
if (!error.message.includes('ERR_ABORTED')) return false
|
|
772
|
+
const disposition = (response?.headers()['content-disposition'] ?? '').toLowerCase()
|
|
773
|
+
return disposition.includes('attachment')
|
|
774
|
+
}
|
|
775
|
+
|
|
776
|
+
/**
|
|
777
|
+
* Recover a download-turned navigation into the finished fetch result the
|
|
778
|
+
* seam's contract demands. The resource itself is often perfectly decodable
|
|
779
|
+
* — the oembed endpoint's `application/json` classifies as `text` — the
|
|
780
|
+
* browser just will not commit it as a document, and the response it tracked
|
|
781
|
+
* cannot yield its body either (Chromium streams a download to its download
|
|
782
|
+
* manager, so `Network.getResponseBody` answers "No resource with given
|
|
783
|
+
* identifier found"; both verified against a real browser). The body is
|
|
784
|
+
* therefore refetched through the page's API request context: plain HTTP
|
|
785
|
+
* from the Playwright client — same cookie jar, same URL, no render-or-
|
|
786
|
+
* download decision — which works identically on the local and CDP
|
|
787
|
+
* backends because it never touches the remote artifact.
|
|
788
|
+
*
|
|
789
|
+
* Undecodable downloads (a PDF, a ZIP) still fail with
|
|
790
|
+
* `WEB_UNSUPPORTED_CONTENT_TYPE`, classified from what the navigation
|
|
791
|
+
* actually served; a playwright build without a request context surfaces a
|
|
792
|
+
* clear "the browser will not render this" error instead of the raw
|
|
793
|
+
* "Download is starting".
|
|
794
|
+
*
|
|
795
|
+
* @param error - the value `page.goto` rejected with.
|
|
796
|
+
* @param page - the fetch's tab, for its API request context.
|
|
797
|
+
* @param tracker - the fetch's main-frame response record.
|
|
798
|
+
* @param url - the request URL (fallback when the response carries none).
|
|
799
|
+
* @param deadline - the fetch budget, applied to the refetch.
|
|
800
|
+
* @param denoise - the settings toggle, honored as on the normal HTML path.
|
|
801
|
+
* @returns the finished fetch result, or undefined when the failure is not
|
|
802
|
+
* a download navigation (the throw should propagate).
|
|
803
|
+
*/
|
|
804
|
+
async function recoverDownloadResponse(
|
|
805
|
+
error: unknown,
|
|
806
|
+
page: PlaywrightPage,
|
|
807
|
+
tracker: MainFrameTracker,
|
|
808
|
+
url: URL,
|
|
809
|
+
deadline: Deadline,
|
|
810
|
+
denoise: boolean,
|
|
811
|
+
): Promise<WebFetchResult | undefined> {
|
|
812
|
+
const tracked = tracker.last()
|
|
813
|
+
if (!isDownloadFailure(error, tracked)) return undefined
|
|
814
|
+
if (tracked !== null) {
|
|
815
|
+
// What the site served the navigation is the truth about the resource:
|
|
816
|
+
// an undecodable download (PDF/ZIP/…) has no business being refetched.
|
|
817
|
+
const trackedKind = classifyContentType(tracked.headers()['content-type'])
|
|
818
|
+
if (trackedKind === undefined) {
|
|
819
|
+
throw new WebError(`unsupported content type "${tracked.headers()['content-type'] ?? 'unknown'}" (served as a download)`, 'WEB_UNSUPPORTED_CONTENT_TYPE')
|
|
820
|
+
}
|
|
821
|
+
}
|
|
822
|
+
if (page.request === undefined) {
|
|
823
|
+
throw new WebError(
|
|
824
|
+
'the response is a file download (Content-Disposition: attachment) the browser will not render, and this playwright build exposes no request context to refetch it',
|
|
825
|
+
'WEB_PROVIDER_ERROR',
|
|
826
|
+
{ cause: error },
|
|
827
|
+
)
|
|
828
|
+
}
|
|
829
|
+
const fetchUrl = tracked?.url?.() ?? url.toString()
|
|
830
|
+
const refetched = await page.request.get(fetchUrl, { timeout: deadline.remainingMs(), signal: deadline.signal })
|
|
831
|
+
const headers = refetched.headers()
|
|
832
|
+
// The refetch speaks for the bytes it returned; only a content-type-free
|
|
833
|
+
// answer falls back to what the navigation served.
|
|
834
|
+
const servedType = (headers['content-type'] ?? '').trim() !== '' ? headers['content-type'] : tracked?.headers()['content-type']
|
|
835
|
+
const kind = classifyContentType(servedType)
|
|
836
|
+
if (kind === undefined) {
|
|
837
|
+
throw new WebError(`unsupported content type "${servedType ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE')
|
|
838
|
+
}
|
|
839
|
+
const finalUrl = refetched.url?.() ?? fetchUrl
|
|
840
|
+
const statusCode = refetched.status()
|
|
841
|
+
const content = await refetched.text()
|
|
842
|
+
if (kind === 'text') {
|
|
843
|
+
return capResult(finalUrl, statusCode, { kind: 'text', content })
|
|
844
|
+
}
|
|
845
|
+
if (!denoise) return capResult(finalUrl, statusCode, { kind: 'html', content })
|
|
846
|
+
const { markdown } = htmlToMarkdown(content, finalUrl)
|
|
847
|
+
return capResult(finalUrl, statusCode, { kind: 'text', content: markdown })
|
|
848
|
+
}
|
|
849
|
+
|
|
740
850
|
/**
|
|
741
851
|
* Probe the live document for challenge markers. Prefers the in-page probe
|
|
742
852
|
* (`page.evaluate`) — the only thing that can see SPA-style clears — and
|
|
@@ -829,6 +939,23 @@ async function installResourceFilter(owner: {
|
|
|
829
939
|
}
|
|
830
940
|
}
|
|
831
941
|
|
|
942
|
+
/**
|
|
943
|
+
* Cancel any download Chromium starts on the fetch's tab. A navigation whose
|
|
944
|
+
* response carries `Content-Disposition: attachment` never commits a
|
|
945
|
+
* document — `goto` rejects with "Download is starting" and the recovery
|
|
946
|
+
* refetches the bytes (see {@link recoverDownloadResponse}) — but the
|
|
947
|
+
* browser still hands the original body to its download manager, which in
|
|
948
|
+
* profile mode would drop files into the remote machine's download dir.
|
|
949
|
+
* Best-effort: a page without the listener just loses the guard.
|
|
950
|
+
*/
|
|
951
|
+
function guardDownloads(page: PlaywrightPage): void {
|
|
952
|
+
try {
|
|
953
|
+
page.on?.('download', (download: PlaywrightDownload) => { void download.cancel().catch(() => {}) })
|
|
954
|
+
} catch {
|
|
955
|
+
// keep going without the guard
|
|
956
|
+
}
|
|
957
|
+
}
|
|
958
|
+
|
|
832
959
|
/**
|
|
833
960
|
* Close any popup a fetched page spawns so nothing outlives the fetch's tab
|
|
834
961
|
* — `page.close()` does not auto-close windows the page opened, and in
|
package/src/types.ts
CHANGED
|
@@ -22,6 +22,26 @@ export interface PlaywrightResponse {
|
|
|
22
22
|
request?(): PlaywrightRequest
|
|
23
23
|
}
|
|
24
24
|
|
|
25
|
+
/**
|
|
26
|
+
* A download Chromium started instead of committing a document (a response
|
|
27
|
+
* with `Content-Disposition: attachment`).
|
|
28
|
+
*/
|
|
29
|
+
export interface PlaywrightDownload {
|
|
30
|
+
cancel(): Promise<void>
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
/**
|
|
34
|
+
* The page's `APIRequestContext` surface the download recovery refetches
|
|
35
|
+
* through — plain HTTP from the Playwright client, the context's cookie jar,
|
|
36
|
+
* and none of Chromium's render-or-download semantics. Reuses
|
|
37
|
+
* {@link PlaywrightResponse} because that is structurally what an
|
|
38
|
+
* `APIResponse` is for the members this plugin reads. Absent on fakes that
|
|
39
|
+
* do not exercise the recovery.
|
|
40
|
+
*/
|
|
41
|
+
export interface PlaywrightRequestContext {
|
|
42
|
+
get(url: string, options?: { timeout?: number; signal?: AbortSignal }): Promise<PlaywrightResponse>
|
|
43
|
+
}
|
|
44
|
+
|
|
25
45
|
/** The request side of a response, for main-frame filtering. */
|
|
26
46
|
export interface PlaywrightRequest {
|
|
27
47
|
/** True for navigations (document loads and their redirect hops). */
|
|
@@ -52,6 +72,13 @@ export interface PlaywrightPage {
|
|
|
52
72
|
* document). Absent on minimal fakes (content polling covers them).
|
|
53
73
|
*/
|
|
54
74
|
on?(event: 'response', listener: (response: PlaywrightResponse) => void): unknown
|
|
75
|
+
/**
|
|
76
|
+
* Download notification — a navigation the browser handed to its download
|
|
77
|
+
* manager instead of committing a document (`Content-Disposition:
|
|
78
|
+
* attachment`); the fetch's guard cancels it so nothing lands on disk.
|
|
79
|
+
* Absent on minimal fakes (the recovery does not depend on it).
|
|
80
|
+
*/
|
|
81
|
+
on?(event: 'download', listener: (download: PlaywrightDownload) => void): unknown
|
|
55
82
|
/**
|
|
56
83
|
* Evaluate an expression in the page — the challenge probe's live-DOM
|
|
57
84
|
* path. Absent on minimal fakes (content polling covers them).
|
|
@@ -59,6 +86,11 @@ export interface PlaywrightPage {
|
|
|
59
86
|
evaluate?(script: string, arg?: unknown): Promise<unknown>
|
|
60
87
|
/** The main frame handle; compare with `request.frame()` for filtering. */
|
|
61
88
|
mainFrame?(): unknown
|
|
89
|
+
/**
|
|
90
|
+
* The page's API request context — the download recovery's refetch path
|
|
91
|
+
* (see {@link PlaywrightRequestContext}). Absent on minimal fakes.
|
|
92
|
+
*/
|
|
93
|
+
request?: PlaywrightRequestContext
|
|
62
94
|
}
|
|
63
95
|
|
|
64
96
|
/**
|