dsh-web-fetch-playwright 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -4
- package/README.zh-CN.md +25 -4
- package/dsh.plugin.json +1 -1
- package/lib/client.js +57 -30
- package/lib/index.js +116 -42
- package/lib/types/cdp-pool.d.ts +42 -16
- package/lib/types/client/card.d.ts +4 -3
- package/lib/types/client/controller.d.ts +8 -0
- package/lib/types/client/fields.d.ts +7 -1
- package/lib/types/client/locales.d.ts +1 -1
- package/lib/types/config.d.ts +24 -0
- package/lib/types/index.d.ts +3 -3
- package/lib/types/provider.d.ts +23 -15
- package/lib/types/types.d.ts +18 -1
- package/package.json +12 -9
- package/src/cdp-pool.ts +73 -21
- package/src/client/card.tsx +33 -17
- package/src/client/controller.ts +10 -0
- package/src/client/fields.module.css +12 -0
- package/src/client/fields.tsx +9 -3
- package/src/client/locales.ts +5 -0
- package/src/config.ts +30 -0
- package/src/index.ts +3 -3
- package/src/provider.ts +79 -34
- package/src/types.ts +18 -1
package/README.md
CHANGED
|
@@ -12,8 +12,8 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
|
|
|
12
12
|
- **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer.
|
|
13
13
|
- **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
|
|
14
14
|
- **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
|
|
15
|
-
- **Isolated sessions** — every fetch
|
|
16
|
-
- **Live configuration** — a settings card (设置 → 插件 → 插件配置) edits the backend, denoise toggle, and concurrency; changes apply to the next fetch without a restart.
|
|
15
|
+
- **Isolated or profile sessions (CDP)** — every fetch is scoped to exactly one tab. Local launches close their browser per fetch; the CDP backend keeps **one shared connection** to the remote browser and each fetch opens a tab inside it, closed when done. By default that tab lives in the remote browser's **real profile** (its cookies, localStorage, and persistent logins apply — like `playwright-cli open`); unchecking *Share the browser context* switches to a throwaway isolated context per fetch.
|
|
16
|
+
- **Live configuration** — a settings card (设置 → 插件 → 插件配置) edits the backend, context mode, denoise toggle, and concurrency; changes apply to the next fetch without a restart.
|
|
17
17
|
- **Budget-aware** — per-fetch deadline (45s); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
|
|
18
18
|
|
|
19
19
|
## How it works
|
|
@@ -61,11 +61,14 @@ After a bundle plugin is added to the profile layer stack, **restart `dsh web`**
|
|
|
61
61
|
|
|
62
62
|
The settings card (设置 → 插件 → 插件配置 → *Playwright 网页爬取*) edits the `web-fetch-playwright` settings section live:
|
|
63
63
|
|
|
64
|
+

|
|
65
|
+
|
|
64
66
|
| Field | Default | Description |
|
|
65
67
|
| --- | --- | --- |
|
|
66
68
|
| `backend` | `local` | Radio: *Local Playwright* or *Remote CDP endpoint*, each with its own nested input. |
|
|
67
69
|
| `playwrightPath` | (blank) | Local backend: path to a `playwright` executable or a Chromium-family browser binary. Blank = discover on `$PATH`, then fall back to the bundled `playwright-core`. |
|
|
68
70
|
| `cdpEndpoint` | `127.0.0.1:9222` | Remote backend: `host:port`, `http(s)://…` or `ws(s)://…`. |
|
|
71
|
+
| `shareBrowserContext` | `true` | CDP backend only. **Checked (profile mode)**: each fetch is a tab in the remote browser's default context — its real profile — so cookies/localStorage are shared and its persistent logins apply; only the tab closes when the fetch ends. **Unchecked (isolated mode)**: a fresh incognito-like context per fetch, nothing shared. Local backend ignores this field. |
|
|
69
72
|
| `denoise` | `true` | Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert. |
|
|
70
73
|
| `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
|
|
71
74
|
|
|
@@ -75,7 +78,25 @@ Local backend resolution order:
|
|
|
75
78
|
2. A `playwright` executable on `$PATH` (its package knows that installation's browser registry).
|
|
76
79
|
3. The bundled `playwright-core` — requires `PLAYWRIGHT_BROWSERS_PATH` or browsers in the default cache; otherwise the error suggests `playwright install chromium`.
|
|
77
80
|
|
|
78
|
-
CDP mode needs no local browser: the provider holds **one shared connection** for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a
|
|
81
|
+
CDP mode needs no local browser: the provider holds **one shared connection** for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a tab in the remote browser that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection (the remote browser itself is never closed).
|
|
82
|
+
|
|
83
|
+
### The CDP context modes (share the browser's profile or not)
|
|
84
|
+
|
|
85
|
+
With *Share the browser context* **checked** (default, profile mode), each fetch is a tab in the remote browser's default context — the real profile. Cookies and localStorage come from and are written back to that profile, so sites the browser is logged into are fetched logged-in, exactly like a tab you open by hand. The shared context is never closed; resource filtering and popup guards attach to the fetch's tab only, so your other tabs are untouched. With it **unchecked** (isolated mode), every fetch gets a fresh incognito-like context — anonymous reads, nothing persists.
|
|
86
|
+
|
|
87
|
+
**Profile-mode risk notes** — it upgrades `web_fetch` from "anonymous read" to "acts as the browser's logged-in user":
|
|
88
|
+
|
|
89
|
+
- A fetched page that talks the agent into a GET-style state-changing URL (logout, settings change, API call) will send it with your session cookies.
|
|
90
|
+
- Concurrent fetches to the same site share one cookie jar; one fetch's logout or `Set-Cookie` affects the others.
|
|
91
|
+
- Output starts depending on the browser's history (A/B buckets, language preferences). The remote profile also keeps accumulating site data; the plugin never cleans it.
|
|
92
|
+
|
|
93
|
+
Persistent logins require a persistent user-data-dir. Headful (recommended — log in by hand once):
|
|
94
|
+
|
|
95
|
+
```sh
|
|
96
|
+
google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome-dsh-profile"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Headless server (pre-seed logins in a headful environment first): `chromium --headless=new --remote-debugging-port=9222 --user-data-dir=/data/chrome-dsh-profile`. Do **not** add `--incognito` or a throwaway user-data-dir — either defeats profile mode. Design notes and verified playwright-core facts live in [`docs/context-mode-profile.md`](./docs/context-mode-profile.md).
|
|
79
100
|
|
|
80
101
|
## Development
|
|
81
102
|
|
|
@@ -104,7 +125,7 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development and release workflo
|
|
|
104
125
|
|
|
105
126
|
## Security
|
|
106
127
|
|
|
107
|
-
Same stance as the built-in HTTP provider: **no SSRF / private-network protection is implemented** — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself.
|
|
128
|
+
Same stance as the built-in HTTP provider: **no SSRF / private-network protection is implemented** — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself — but note that in profile mode the requests (and any state changes a malicious page tricks the agent into) carry the remote browser's logged-in sessions (see the risk notes above).
|
|
108
129
|
|
|
109
130
|
## License
|
|
110
131
|
|
package/README.zh-CN.md
CHANGED
|
@@ -12,8 +12,8 @@
|
|
|
12
12
|
- **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。
|
|
13
13
|
- **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
|
|
14
14
|
- **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
|
|
15
|
-
-
|
|
16
|
-
- **热配置** —— 「设置 → 插件 →
|
|
15
|
+
- **共享或隔离会话(CDP)** —— 每次抓取严格限定为一个标签页。本地后端每次抓取启动并关闭自己的浏览器;CDP 后端对远端浏览器保持**一条共享连接**,每次抓取只在其里开一个标签页、用完即关。默认该标签页位于远端浏览器的**真实 profile**(沿用其 cookie、localStorage 与已登录会话,效果类似 `playwright-cli open`);取消勾选「共享浏览器上下文」则切换为每次抓取全新隔离 context。
|
|
16
|
+
- **热配置** —— 「设置 → 插件 → 插件配置」卡片可随时切换后端、上下文模式、降噪开关与并发数,改动对下一次抓取即时生效,无需重启。
|
|
17
17
|
- **预算控制** —— 单次抓取 45s 超时;并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 20s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
|
|
18
18
|
|
|
19
19
|
## 工作原理
|
|
@@ -61,11 +61,14 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
|
|
|
61
61
|
|
|
62
62
|
设置卡片(设置 → 插件 → 插件配置 → *Playwright 网页爬取*)实时编辑 `web-fetch-playwright` 设置段:
|
|
63
63
|
|
|
64
|
+

|
|
65
|
+
|
|
64
66
|
| 字段 | 默认 | 说明 |
|
|
65
67
|
| --- | --- | --- |
|
|
66
68
|
| `backend` | `local` | radio:本地 Playwright / 远端 CDP 地址,每个选项内嵌各自的填空。 |
|
|
67
69
|
| `playwrightPath` | 空 | 本地后端:`playwright` 可执行文件或 Chromium 系浏览器二进制路径;留空按 `$PATH` 查找,再回退到内置 `playwright-core`。 |
|
|
68
70
|
| `cdpEndpoint` | `127.0.0.1:9222` | 远端后端:`host:port`、`http(s)://…` 或 `ws(s)://…`。 |
|
|
71
|
+
| `shareBrowserContext` | `true` | 仅 CDP 后端。**勾选(profile 模式)**:每次抓取是远端浏览器默认 context(真实 profile)里的一个标签页,cookie/localStorage 与之互通、已登录会话直接生效,抓取结束只关标签页;**取消勾选(隔离模式)**:每次抓取使用全新隐身式 context,互不共享。本地后端忽略此字段。 |
|
|
69
72
|
| `denoise` | `true` | 是否启用降噪;关闭时返回整页渲染 HTML,交由工具层转换。 |
|
|
70
73
|
| `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;20s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
|
|
71
74
|
|
|
@@ -75,7 +78,25 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
|
|
|
75
78
|
2. `$PATH` 上的 `playwright`(其包自带该安装的浏览器注册表);
|
|
76
79
|
3. 插件内置的 `playwright-core`——需要 `PLAYWRIGHT_BROWSERS_PATH` 或默认缓存里有浏览器,否则报错会提示 `playwright install chromium`。
|
|
77
80
|
|
|
78
|
-
CDP
|
|
81
|
+
CDP 模式不需要本地浏览器:插件在生命周期内对远端浏览器保持**一条共享连接**(连接断开自动重连,地址改动后自动换连),每次抓取只租用远端浏览器里的一个标签页,抓取结束即关闭。因此并发数按"标签页"计,默认也更高(50)。插件卸载时断开共享连接(绝不会关闭远端浏览器本身)。
|
|
82
|
+
|
|
83
|
+
### CDP 上下文模式(是否共享浏览器 profile)
|
|
84
|
+
|
|
85
|
+
「共享浏览器上下文」**勾选**(默认,profile 模式)时,每次抓取是远端浏览器默认 context——真实 profile——里的一个标签页:cookie 与 localStorage 双向互通,浏览器里已登录的站点会以登录态被抓取,和你手动开标签页一样。共享 context 永不关闭;资源过滤与弹窗守卫只挂在本次抓取自己的标签页上,不会干扰你人工打开的其他标签页。**取消勾选**(隔离模式)时,每次抓取使用全新隐身式 context——匿名读取,什么都不保留。
|
|
86
|
+
|
|
87
|
+
**profile 模式风险须知** —— 它把 `web_fetch` 从"匿名读网页"升级为"以浏览器登录身份行动":
|
|
88
|
+
|
|
89
|
+
- 被抓取的恶意页面若诱导 agent 请求 GET 型状态变更 URL(登出、改设置、API 操作),请求会自动携带会话 cookie。
|
|
90
|
+
- 同一站点的并发抓取共享一个 cookie jar,一方的登出 / `Set-Cookie` 会影响另一方。
|
|
91
|
+
- 输出开始依赖浏览器历史(A/B 分桶、语言偏好);远端 profile 的站点数据只增不减,插件不做清理。
|
|
92
|
+
|
|
93
|
+
登录态持久的前提是 user-data-dir 持久化。有头(推荐,手动登录一次):
|
|
94
|
+
|
|
95
|
+
```sh
|
|
96
|
+
google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome-dsh-profile"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
无头服务器(先在有头环境预置登录态):`chromium --headless=new --remote-debugging-port=9222 --user-data-dir=/data/chrome-dsh-profile`。**不要**叠加 `--incognito` 或一次性 user-data-dir——都会让 profile 模式失效。设计依据与已核实的 playwright-core 源码事实见 [`docs/context-mode-profile.md`](./docs/context-mode-profile.md)。
|
|
79
100
|
|
|
80
101
|
## 开发
|
|
81
102
|
|
|
@@ -104,7 +125,7 @@ tests/ # 单元 + provider + 浏览器集成(可自跳过
|
|
|
104
125
|
|
|
105
126
|
## 安全边界
|
|
106
127
|
|
|
107
|
-
与内置 HTTP provider 同立场:**未实现 SSRF/私网防护**——浏览器能访问的目标,本 provider 就能抓。CDP
|
|
128
|
+
与内置 HTTP provider 同立场:**未实现 SSRF/私网防护**——浏览器能访问的目标,本 provider 就能抓。CDP 地址由设置页配置,不做回环限制,请在可信环境暴露设置页。抓取仅在本地渲染,除目标页面自身外不会向任何地方发送数据——但 profile 模式下请求(以及恶意页面诱导 agent 触发的状态变更)会携带远端浏览器的登录会话,见上文风险须知。
|
|
108
129
|
|
|
109
130
|
## 许可证
|
|
110
131
|
|
package/dsh.plugin.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-web-fetch-playwright",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.2",
|
|
4
4
|
"description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
|
|
5
5
|
"entry": {
|
|
6
6
|
"name": "dsh-web-fetch-playwright",
|
package/lib/client.js
CHANGED
|
@@ -348,6 +348,7 @@ window.__ModuleLoader__.load({
|
|
|
348
348
|
radioField("backend", ["local", "cdp"]),
|
|
349
349
|
textField("playwrightPath"),
|
|
350
350
|
textField("cdpEndpoint"),
|
|
351
|
+
checkboxField("shareBrowserContext"),
|
|
351
352
|
checkboxField("denoise"),
|
|
352
353
|
numberField("maxConcurrency", 1, 200)
|
|
353
354
|
]);
|
|
@@ -359,6 +360,7 @@ window.__ModuleLoader__.load({
|
|
|
359
360
|
backend: this.form.field("backend"),
|
|
360
361
|
playwrightPath: this.form.field("playwrightPath"),
|
|
361
362
|
cdpEndpoint: this.form.field("cdpEndpoint"),
|
|
363
|
+
shareBrowserContext: this.form.field("shareBrowserContext"),
|
|
362
364
|
denoise: this.form.field("denoise"),
|
|
363
365
|
maxConcurrency: this.form.field("maxConcurrency")
|
|
364
366
|
};
|
|
@@ -386,21 +388,21 @@ window.__ModuleLoader__.load({
|
|
|
386
388
|
document.head.appendChild(tag);
|
|
387
389
|
}
|
|
388
390
|
var PluginCard_module_css_default = {
|
|
389
|
-
"
|
|
391
|
+
"chevron": "UM8-Rq_chevron",
|
|
390
392
|
"chevronOpen": "UM8-Rq_chevronOpen",
|
|
391
|
-
"readOnly": "UM8-Rq_readOnly",
|
|
392
|
-
"footer": "UM8-Rq_footer",
|
|
393
|
-
"pending": "UM8-Rq_pending",
|
|
394
|
-
"header": "UM8-Rq_header",
|
|
395
393
|
"save": "UM8-Rq_save",
|
|
396
|
-
"
|
|
394
|
+
"name": "UM8-Rq_name",
|
|
395
|
+
"discard": "UM8-Rq_discard",
|
|
397
396
|
"failed": "UM8-Rq_failed",
|
|
398
|
-
"
|
|
399
|
-
"body": "UM8-Rq_body",
|
|
397
|
+
"header": "UM8-Rq_header",
|
|
400
398
|
"cardOpen": "UM8-Rq_cardOpen",
|
|
401
|
-
"
|
|
402
|
-
"
|
|
403
|
-
"
|
|
399
|
+
"description": "UM8-Rq_description",
|
|
400
|
+
"body": "UM8-Rq_body",
|
|
401
|
+
"card": "UM8-Rq_card",
|
|
402
|
+
"footer": "UM8-Rq_footer",
|
|
403
|
+
"readOnly": "UM8-Rq_readOnly",
|
|
404
|
+
"headText": "UM8-Rq_headText",
|
|
405
|
+
"pending": "UM8-Rq_pending"
|
|
404
406
|
};
|
|
405
407
|
//#endregion
|
|
406
408
|
//#region src/client/PluginCard.tsx
|
|
@@ -507,7 +509,7 @@ window.__ModuleLoader__.load({
|
|
|
507
509
|
}
|
|
508
510
|
//#endregion
|
|
509
511
|
//#region \0dsh-web-fetch-playwright-css:/opt/dsh/plugins/dsh-web-fetch-playwright/src/client/fields.module.css.mjs
|
|
510
|
-
const css = ".CFSaVG_field{border:0;border-top:1px solid var(--dsw-alias-border-l2);flex-direction:column;gap:6px;margin:0;padding:12px 0;display:flex}.CFSaVG_field:first-child{border-top:0}.CFSaVG_fieldEmbedded{border:0;flex-direction:column;gap:6px;margin:0;padding:0;display:flex}.CFSaVG_head{align-items:center;gap:8px;display:flex}.CFSaVG_label{min-width:0;color:var(--dsw-alias-label-primary);flex:1;font-size:13px;font-weight:500;line-height:1.5}.CFSaVG_badges{align-items:center;gap:8px;display:inline-flex}.CFSaVG_badge{white-space:nowrap;background:var(--dsw-alias-bg-module-platform);color:var(--dsw-alias-label-secondary);border-radius:999px;padding:1px 8px;font-size:11px;font-weight:500;line-height:17px}.CFSaVG_reset{font:inherit;color:var(--dsw-alias-label-secondary);cursor:pointer;background:0 0;border:none;padding:0;font-size:12px;line-height:1.5}.CFSaVG_reset:hover:not(:disabled){color:var(--dsw-alias-label-primary)}.CFSaVG_reset:disabled{cursor:default}.CFSaVG_input{border:1px solid var(--dsw-alias-border-l2);background:var(--dsw-alias-bg-layer-3);height:34px;font:inherit;color:var(--dsw-alias-label-primary);border-radius:8px;padding:0 12px;font-size:13px;line-height:1.5}.CFSaVG_input:focus-visible{border-color:var(--dsw-alias-brand-primary);outline:none}.CFSaVG_input:disabled{color:var(--dsw-alias-label-tertiary);cursor:default}.CFSaVG_inputInvalid{border:1px solid var(--dsw-alias-label-error);background:var(--dsw-alias-bg-layer-3);height:34px;font:inherit;color:var(--dsw-alias-label-primary);border-radius:8px;padding:0 12px;font-size:13px;line-height:1.5}.CFSaVG_invalid{color:var(--dsw-alias-label-error);margin:0;font-size:12px;line-height:1.5}.CFSaVG_hint{color:var(--dsw-alias-label-tertiary);margin:0;font-size:12px;line-height:1.5}.CFSaVG_radioGroup{flex-direction:column;gap:8px;display:flex}.CFSaVG_radioOption{border:1px solid var(--dsw-alias-border-l2);background:var(--dsw-alias-bg-layer-3);border-radius:8px;flex-direction:column;gap:8px;padding:10px 12px;display:flex}.CFSaVG_radioOption:has(input[type=radio]:checked){border-color:var(--dsw-alias-brand-primary)}.CFSaVG_radioPick{cursor:pointer;align-items:flex-start;gap:10px;display:flex}.CFSaVG_radioPick input[type=radio]{width:14px;height:14px;accent-color:var(--dsw-alias-brand-primary);flex:none;margin-top:2px}.CFSaVG_radioPick input[type=radio]:focus-visible{outline:2px solid var(--dsw-alias-brand-primary);outline-offset:1px}.CFSaVG_radioText{flex-direction:column;gap:2px;min-width:0;display:flex}.CFSaVG_radioLabel{color:var(--dsw-alias-label-primary);font-size:13px;font-weight:500;line-height:1.5}.CFSaVG_radioHint{color:var(--dsw-alias-label-tertiary);font-size:12px;line-height:1.5}.CFSaVG_radioContent{margin-left:24px}.CFSaVG_radioContent input[type=text]:enabled{cursor:text}.CFSaVG_checkboxRow{align-items:center;gap:10px;display:flex}.CFSaVG_checkboxRow input[type=checkbox]{width:16px;height:16px;accent-color:var(--dsw-alias-brand-primary);flex:none;margin:0}.CFSaVG_checkboxRow input[type=checkbox]:focus-visible{outline:2px solid var(--dsw-alias-brand-primary);outline-offset:1px}";
|
|
512
|
+
const css = ".CFSaVG_field{border:0;border-top:1px solid var(--dsw-alias-border-l2);flex-direction:column;gap:6px;margin:0;padding:12px 0;display:flex}.CFSaVG_field:first-child{border-top:0}.CFSaVG_fieldEmbedded{border:0;flex-direction:column;gap:6px;margin:0;padding:0;display:flex}.CFSaVG_head{align-items:center;gap:8px;display:flex}.CFSaVG_label{min-width:0;color:var(--dsw-alias-label-primary);flex:1;font-size:13px;font-weight:500;line-height:1.5}.CFSaVG_badges{align-items:center;gap:8px;display:inline-flex}.CFSaVG_badge{white-space:nowrap;background:var(--dsw-alias-bg-module-platform);color:var(--dsw-alias-label-secondary);border-radius:999px;padding:1px 8px;font-size:11px;font-weight:500;line-height:17px}.CFSaVG_reset{font:inherit;color:var(--dsw-alias-label-secondary);cursor:pointer;background:0 0;border:none;padding:0;font-size:12px;line-height:1.5}.CFSaVG_reset:hover:not(:disabled){color:var(--dsw-alias-label-primary)}.CFSaVG_reset:disabled{cursor:default}.CFSaVG_input{border:1px solid var(--dsw-alias-border-l2);background:var(--dsw-alias-bg-layer-3);height:34px;font:inherit;color:var(--dsw-alias-label-primary);border-radius:8px;padding:0 12px;font-size:13px;line-height:1.5}.CFSaVG_input:focus-visible{border-color:var(--dsw-alias-brand-primary);outline:none}.CFSaVG_input:disabled{color:var(--dsw-alias-label-tertiary);cursor:default}.CFSaVG_inputInvalid{border:1px solid var(--dsw-alias-label-error);background:var(--dsw-alias-bg-layer-3);height:34px;font:inherit;color:var(--dsw-alias-label-primary);border-radius:8px;padding:0 12px;font-size:13px;line-height:1.5}.CFSaVG_invalid{color:var(--dsw-alias-label-error);margin:0;font-size:12px;line-height:1.5}.CFSaVG_hint{color:var(--dsw-alias-label-tertiary);margin:0;font-size:12px;line-height:1.5}.CFSaVG_hintIndented{margin-left:26px}.CFSaVG_radioGroup{flex-direction:column;gap:8px;display:flex}.CFSaVG_radioOption{border:1px solid var(--dsw-alias-border-l2);background:var(--dsw-alias-bg-layer-3);border-radius:8px;flex-direction:column;gap:8px;padding:10px 12px;display:flex}.CFSaVG_radioOption:has(input[type=radio]:checked){border-color:var(--dsw-alias-brand-primary)}.CFSaVG_radioPick{cursor:pointer;align-items:flex-start;gap:10px;display:flex}.CFSaVG_radioPick input[type=radio]{width:14px;height:14px;accent-color:var(--dsw-alias-brand-primary);flex:none;margin-top:2px}.CFSaVG_radioPick input[type=radio]:focus-visible{outline:2px solid var(--dsw-alias-brand-primary);outline-offset:1px}.CFSaVG_radioText{flex-direction:column;gap:2px;min-width:0;display:flex}.CFSaVG_radioLabel{color:var(--dsw-alias-label-primary);font-size:13px;font-weight:500;line-height:1.5}.CFSaVG_radioHint{color:var(--dsw-alias-label-tertiary);font-size:12px;line-height:1.5}.CFSaVG_radioContent{margin-left:24px}.CFSaVG_radioContent>*+*{margin-top:12px}.CFSaVG_radioContent input[type=text]:enabled{cursor:text}.CFSaVG_checkboxRow{align-items:center;gap:10px;display:flex}.CFSaVG_checkboxRow input[type=checkbox]{width:16px;height:16px;accent-color:var(--dsw-alias-brand-primary);flex:none;margin:0}.CFSaVG_checkboxRow input[type=checkbox]:focus-visible{outline:2px solid var(--dsw-alias-brand-primary);outline-offset:1px}";
|
|
511
513
|
const tagId = "dsh-web-fetch-playwright/fields.module.css";
|
|
512
514
|
if (typeof document !== "undefined" && document.querySelector("style[data-plugin-css=" + JSON.stringify(tagId) + "]") === null) {
|
|
513
515
|
const tag = document.createElement("style");
|
|
@@ -517,25 +519,26 @@ window.__ModuleLoader__.load({
|
|
|
517
519
|
document.head.appendChild(tag);
|
|
518
520
|
}
|
|
519
521
|
var fields_module_css_default = {
|
|
520
|
-
"checkboxRow": "CFSaVG_checkboxRow",
|
|
521
|
-
"radioOption": "CFSaVG_radioOption",
|
|
522
|
-
"radioLabel": "CFSaVG_radioLabel",
|
|
523
|
-
"head": "CFSaVG_head",
|
|
524
|
-
"radioPick": "CFSaVG_radioPick",
|
|
525
522
|
"badges": "CFSaVG_badges",
|
|
523
|
+
"invalid": "CFSaVG_invalid",
|
|
524
|
+
"radioContent": "CFSaVG_radioContent",
|
|
525
|
+
"hint": "CFSaVG_hint",
|
|
526
|
+
"radioLabel": "CFSaVG_radioLabel",
|
|
526
527
|
"radioGroup": "CFSaVG_radioGroup",
|
|
527
|
-
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
528
528
|
"radioText": "CFSaVG_radioText",
|
|
529
|
-
"radioHint": "CFSaVG_radioHint",
|
|
530
|
-
"radioContent": "CFSaVG_radioContent",
|
|
531
|
-
"field": "CFSaVG_field",
|
|
532
|
-
"input": "CFSaVG_input",
|
|
533
529
|
"badge": "CFSaVG_badge",
|
|
534
|
-
"
|
|
535
|
-
"
|
|
530
|
+
"fieldEmbedded": "CFSaVG_fieldEmbedded",
|
|
531
|
+
"checkboxRow": "CFSaVG_checkboxRow",
|
|
532
|
+
"hintIndented": "CFSaVG_hintIndented",
|
|
533
|
+
"input": "CFSaVG_input",
|
|
534
|
+
"radioHint": "CFSaVG_radioHint",
|
|
536
535
|
"label": "CFSaVG_label",
|
|
537
536
|
"reset": "CFSaVG_reset",
|
|
538
|
-
"inputInvalid": "CFSaVG_inputInvalid"
|
|
537
|
+
"inputInvalid": "CFSaVG_inputInvalid",
|
|
538
|
+
"field": "CFSaVG_field",
|
|
539
|
+
"radioPick": "CFSaVG_radioPick",
|
|
540
|
+
"head": "CFSaVG_head",
|
|
541
|
+
"radioOption": "CFSaVG_radioOption"
|
|
539
542
|
};
|
|
540
543
|
//#endregion
|
|
541
544
|
//#region src/client/fields.tsx
|
|
@@ -641,10 +644,14 @@ window.__ModuleLoader__.load({
|
|
|
641
644
|
})]
|
|
642
645
|
});
|
|
643
646
|
}
|
|
644
|
-
/**
|
|
647
|
+
/**
|
|
648
|
+
* A staged checkbox: the denoise toggle and the CDP shared-context toggle.
|
|
649
|
+
* An absent stored value formats as '' — callers render the schema default
|
|
650
|
+
* (denoise on, context shared).
|
|
651
|
+
*/
|
|
645
652
|
function CheckboxField(props) {
|
|
646
653
|
return /* @__PURE__ */ (0, react_jsx_runtime.jsxs)("div", {
|
|
647
|
-
className: fields_module_css_default.field,
|
|
654
|
+
className: props.embedded ? fields_module_css_default.fieldEmbedded : fields_module_css_default.field,
|
|
648
655
|
children: [/* @__PURE__ */ (0, react_jsx_runtime.jsxs)("div", {
|
|
649
656
|
className: fields_module_css_default.checkboxRow,
|
|
650
657
|
children: [
|
|
@@ -677,7 +684,7 @@ window.__ModuleLoader__.load({
|
|
|
677
684
|
}) : null
|
|
678
685
|
]
|
|
679
686
|
}), /* @__PURE__ */ (0, react_jsx_runtime.jsx)("p", {
|
|
680
|
-
className: fields_module_css_default.hint,
|
|
687
|
+
className: props.embedded ? `${fields_module_css_default.hint} ${fields_module_css_default.hintIndented}` : fields_module_css_default.hint,
|
|
681
688
|
children: props.hint
|
|
682
689
|
})]
|
|
683
690
|
});
|
|
@@ -739,7 +746,7 @@ window.__ModuleLoader__.load({
|
|
|
739
746
|
value: "cdp",
|
|
740
747
|
label: t("backendCdp"),
|
|
741
748
|
hint: t("backendCdpHint"),
|
|
742
|
-
content: /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
|
|
749
|
+
content: /* @__PURE__ */ (0, react_jsx_runtime.jsxs)(react_jsx_runtime.Fragment, { children: [/* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
|
|
743
750
|
embedded: true,
|
|
744
751
|
id: "plugin-config-playwright-cdp",
|
|
745
752
|
label: t("cdpEndpoint"),
|
|
@@ -756,7 +763,23 @@ window.__ModuleLoader__.load({
|
|
|
756
763
|
onReset: () => {
|
|
757
764
|
props.resetField("cdpEndpoint");
|
|
758
765
|
}
|
|
759
|
-
})
|
|
766
|
+
}), /* @__PURE__ */ (0, react_jsx_runtime.jsx)(CheckboxField, {
|
|
767
|
+
embedded: true,
|
|
768
|
+
id: "plugin-config-playwright-share-context",
|
|
769
|
+
label: t("shareBrowserContext"),
|
|
770
|
+
hint: t("shareBrowserContextHint"),
|
|
771
|
+
checked: state.shareBrowserContext.text !== "false",
|
|
772
|
+
overridden: state.shareBrowserContext.overridden,
|
|
773
|
+
overriddenLabel: t("overridden"),
|
|
774
|
+
resetLabel: t("reset"),
|
|
775
|
+
disabled: disabled || backend !== "cdp",
|
|
776
|
+
onEdit: (text) => {
|
|
777
|
+
props.edit("shareBrowserContext", text);
|
|
778
|
+
},
|
|
779
|
+
onReset: () => {
|
|
780
|
+
props.resetField("shareBrowserContext");
|
|
781
|
+
}
|
|
782
|
+
})] })
|
|
760
783
|
}],
|
|
761
784
|
text: state.backend.text,
|
|
762
785
|
overridden: state.backend.overridden,
|
|
@@ -822,6 +845,8 @@ window.__ModuleLoader__.load({
|
|
|
822
845
|
playwrightPathPlaceholder: "(auto: playwright from $PATH)",
|
|
823
846
|
cdpEndpoint: "CDP endpoint",
|
|
824
847
|
cdpEndpointHint: "host:port or http(s)/ws URL. Leave blank for 127.0.0.1:9222.",
|
|
848
|
+
shareBrowserContext: "Share the browser context (profile logins)",
|
|
849
|
+
shareBrowserContextHint: "Each fetch opens a tab in the remote browser’s real profile — cookies and localStorage are shared and its persistent logins apply. Unchecked: every fetch uses a fresh isolated context.",
|
|
825
850
|
denoise: "Enable the denoise algorithm",
|
|
826
851
|
denoiseHint: "Readability + DOMPurify strip nav bars, sidebars, footers, and ads before converting to markdown.",
|
|
827
852
|
maxConcurrency: "Max concurrent fetches",
|
|
@@ -853,6 +878,8 @@ window.__ModuleLoader__.load({
|
|
|
853
878
|
playwrightPathPlaceholder: "(自动:按 $PATH 查找 playwright)",
|
|
854
879
|
cdpEndpoint: "CDP 地址",
|
|
855
880
|
cdpEndpointHint: "host:port 或 http(s)/ws 地址;留空默认 127.0.0.1:9222。",
|
|
881
|
+
shareBrowserContext: "共享浏览器上下文(复用登录态)",
|
|
882
|
+
shareBrowserContextHint: "每次抓取在远端浏览器的真实 profile 里开一个标签页:共享 cookie 与 localStorage、复用已登录会话,页面会看到你的登录身份;取消勾选则每次抓取使用全新隔离上下文。",
|
|
856
883
|
denoise: "启用降噪算法",
|
|
857
884
|
denoiseHint: "使用 Readability + DOMPurify 清洗导航栏、侧边栏、页脚与贴片广告后再转为 Markdown。",
|
|
858
885
|
maxConcurrency: "最大并发抓取数",
|
package/lib/index.js
CHANGED
|
@@ -25,6 +25,7 @@ const Config = z.object({
|
|
|
25
25
|
backend: z.union([z.const("local"), z.const("cdp")]).default("local"),
|
|
26
26
|
playwrightPath: z.string().default(""),
|
|
27
27
|
cdpEndpoint: z.string().default(""),
|
|
28
|
+
shareBrowserContext: z.boolean().default(true),
|
|
28
29
|
denoise: z.boolean().default(true),
|
|
29
30
|
maxConcurrency: z.number().step(1).min(1).max(200)
|
|
30
31
|
});
|
|
@@ -41,6 +42,20 @@ function effectiveMaxConcurrency(config) {
|
|
|
41
42
|
return config.backend === "cdp" ? 50 : 4;
|
|
42
43
|
}
|
|
43
44
|
/**
|
|
45
|
+
* The context mode a fetch actually runs with: profile (a tab in the remote
|
|
46
|
+
* browser's real profile) requires the CDP backend — a local launch has no
|
|
47
|
+
* meaningful shared profile — and the checkbox (which defaults on, so an
|
|
48
|
+
* absent value reads as `true`, mirroring the schema default); everything
|
|
49
|
+
* else collapses to isolated.
|
|
50
|
+
*
|
|
51
|
+
* @param config - the resolved settings section (or any partial of it).
|
|
52
|
+
* @returns the mode the pool acquires its lease with.
|
|
53
|
+
*/
|
|
54
|
+
function effectiveContextMode(config) {
|
|
55
|
+
if (config.backend === "cdp" && config.shareBrowserContext !== false) return "profile";
|
|
56
|
+
return "isolated";
|
|
57
|
+
}
|
|
58
|
+
/**
|
|
44
59
|
* Normalize a configured CDP endpoint for `chromium.connectOverCDP`:
|
|
45
60
|
* blank becomes the default loopback endpoint; `wss?://` and `https?://`
|
|
46
61
|
* pass through; a bare `host:port` gains the `http://` scheme Playwright's
|
|
@@ -59,7 +74,7 @@ function normalizeCdpEndpoint(input) {
|
|
|
59
74
|
}
|
|
60
75
|
//#endregion
|
|
61
76
|
//#region src/cdp-pool.ts
|
|
62
|
-
/** A reusable `connectOverCDP` session handing out per-fetch
|
|
77
|
+
/** A reusable `connectOverCDP` session handing out per-fetch pages. */
|
|
63
78
|
var CdpConnectionPool = class {
|
|
64
79
|
connect;
|
|
65
80
|
browser;
|
|
@@ -76,40 +91,70 @@ var CdpConnectionPool = class {
|
|
|
76
91
|
this.connect = connect;
|
|
77
92
|
}
|
|
78
93
|
/**
|
|
79
|
-
* Lease a
|
|
94
|
+
* Lease a fetch's page on the shared connection, connecting (or
|
|
80
95
|
* reconnecting) first if needed. Concurrent first fetches share one
|
|
81
96
|
* connect attempt.
|
|
82
97
|
*
|
|
83
98
|
* @param endpoint - normalized CDP endpoint URL.
|
|
84
99
|
* @param timeoutMs - connect timeout budget.
|
|
85
|
-
* @
|
|
100
|
+
* @param mode - `isolated` (default): a fresh throwaway context plus a page
|
|
101
|
+
* in it; `profile`: a page in the remote browser's default context — the
|
|
102
|
+
* real profile — which is never closed, only the page is.
|
|
103
|
+
* @returns the shared browser, the page's context, and the page the caller
|
|
104
|
+
* must release.
|
|
86
105
|
* @throws whatever the connect function throws when no connection can be
|
|
87
|
-
*
|
|
106
|
+
* established (the provider wraps it in a structured WebError), or a
|
|
107
|
+
* diagnostic error when `profile` mode finds no default context.
|
|
88
108
|
*/
|
|
89
|
-
async acquire(endpoint, timeoutMs) {
|
|
109
|
+
async acquire(endpoint, timeoutMs, mode = "isolated") {
|
|
90
110
|
const browser = await this.ensure(endpoint, timeoutMs);
|
|
91
111
|
try {
|
|
92
|
-
return
|
|
93
|
-
browser,
|
|
94
|
-
context: await browser.newContext()
|
|
95
|
-
};
|
|
112
|
+
return await this.openLease(browser, mode);
|
|
96
113
|
} catch (error) {
|
|
97
114
|
if (this.isLive(browser)) throw error;
|
|
98
115
|
this.drop(browser);
|
|
99
116
|
const fresh = await this.ensure(endpoint, timeoutMs);
|
|
117
|
+
return await this.openLease(fresh, mode);
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
/**
|
|
121
|
+
* Open one lease's page on a live connection. The profile-mode handle is
|
|
122
|
+
* taken fresh every call (never cached across connections): a reconnect
|
|
123
|
+
* yields a new Browser object whose `contexts()[0]` must be re-read.
|
|
124
|
+
*/
|
|
125
|
+
async openLease(browser, mode) {
|
|
126
|
+
if (mode === "profile") {
|
|
127
|
+
const context = browser.contexts?.()[0];
|
|
128
|
+
if (context === void 0) throw new Error("the CDP endpoint exposed no default browser context (profile mode requires a real browser profile)");
|
|
100
129
|
return {
|
|
101
|
-
browser
|
|
102
|
-
context
|
|
130
|
+
browser,
|
|
131
|
+
context,
|
|
132
|
+
page: await context.newPage(),
|
|
133
|
+
persistent: true
|
|
103
134
|
};
|
|
104
135
|
}
|
|
136
|
+
const context = await browser.newContext();
|
|
137
|
+
try {
|
|
138
|
+
return {
|
|
139
|
+
browser,
|
|
140
|
+
context,
|
|
141
|
+
page: await context.newPage(),
|
|
142
|
+
persistent: false
|
|
143
|
+
};
|
|
144
|
+
} catch (error) {
|
|
145
|
+
await context.close().catch(() => {});
|
|
146
|
+
throw error;
|
|
147
|
+
}
|
|
105
148
|
}
|
|
106
149
|
/**
|
|
107
|
-
* Close a fetch-owned context
|
|
108
|
-
*
|
|
109
|
-
*
|
|
150
|
+
* Close a fetch-owned page, plus its context when the lease owns one. The
|
|
151
|
+
* remote default context (persistent leases) and the shared connection
|
|
152
|
+
* stay for the next fetch.
|
|
153
|
+
* @param lease - the lease whose page (and, when isolated, context) goes away.
|
|
110
154
|
*/
|
|
111
155
|
async release(lease) {
|
|
112
|
-
await lease.
|
|
156
|
+
await lease.page.close().catch(() => {});
|
|
157
|
+
if (!lease.persistent) await lease.context.close().catch(() => {});
|
|
113
158
|
}
|
|
114
159
|
/**
|
|
115
160
|
* Drop the shared connection (plugin teardown or endpoint change). Fetches
|
|
@@ -512,18 +557,21 @@ function existsAsFile(path) {
|
|
|
512
557
|
*
|
|
513
558
|
* Lifecycle: the local backend launches a browser per fetch and closes it —
|
|
514
559
|
* nothing outlives the call. The CDP backend keeps ONE shared connection to
|
|
515
|
-
* the remote browser for the provider's lifetime; each fetch only opens
|
|
516
|
-
*
|
|
517
|
-
*
|
|
518
|
-
*
|
|
519
|
-
*
|
|
520
|
-
* browsers
|
|
521
|
-
*
|
|
522
|
-
*
|
|
560
|
+
* the remote browser for the provider's lifetime; each fetch only opens a
|
|
561
|
+
* page (tab) inside it and closes that on completion — in a throwaway
|
|
562
|
+
* isolated context, or (the default, `shareBrowserContext`) in the remote
|
|
563
|
+
* browser's default context so its profile, cookies, and persistent logins
|
|
564
|
+
* apply; that default context is never closed. The concurrency cap counts
|
|
565
|
+
* tabs, not browsers. Either way, an aborted signal closes the fetch's page.
|
|
566
|
+
* Concurrency is capped at the `maxConcurrency` setting (explicit, or the
|
|
567
|
+
* backend default — {@link DEFAULT_MAX_CONCURRENCY_LOCAL} browsers /
|
|
568
|
+
* {@link DEFAULT_MAX_CONCURRENCY_CDP} tabs); further fetches wait briefly in
|
|
569
|
+
* a queue and fail fast (rather than hang until abort) when no slot frees.
|
|
523
570
|
*
|
|
524
571
|
* Private-network and SSRF protection is not implemented (same stance as the
|
|
525
572
|
* shipped HTTP provider); a page this provider can reach is whatever the
|
|
526
|
-
* browser can reach.
|
|
573
|
+
* browser can reach. Profile mode additionally acts WITH the remote
|
|
574
|
+
* browser's logged-in sessions (see the README's risk notes).
|
|
527
575
|
*
|
|
528
576
|
* @module dsh-web-fetch-playwright/provider
|
|
529
577
|
*/
|
|
@@ -752,7 +800,7 @@ var PlaywrightFetchProvider = class {
|
|
|
752
800
|
}
|
|
753
801
|
}
|
|
754
802
|
/**
|
|
755
|
-
* Open the configured backend and its per-fetch
|
|
803
|
+
* Open the configured backend and its per-fetch page. Split out so the
|
|
756
804
|
* test suite can substitute a fake browser.
|
|
757
805
|
* @param config - the resolved settings section.
|
|
758
806
|
* @param deadline - the fetch budget, applied to launch/connect timeouts.
|
|
@@ -764,37 +812,45 @@ var PlaywrightFetchProvider = class {
|
|
|
764
812
|
const endpoint = normalizeCdpEndpoint(config.cdpEndpoint);
|
|
765
813
|
const { source } = await resolveCdpBackend();
|
|
766
814
|
try {
|
|
767
|
-
const lease = await this.cdpPool.acquire(endpoint, timeout);
|
|
768
|
-
await installResourceFilter(lease.
|
|
815
|
+
const lease = await this.cdpPool.acquire(endpoint, timeout, effectiveContextMode(config));
|
|
816
|
+
await installResourceFilter(lease.page);
|
|
817
|
+
guardPopups(lease.page);
|
|
769
818
|
return {
|
|
770
819
|
browser: lease.browser,
|
|
771
820
|
context: lease.context,
|
|
772
|
-
|
|
821
|
+
page: lease.page,
|
|
822
|
+
sharedBrowser: true,
|
|
823
|
+
persistent: lease.persistent
|
|
773
824
|
};
|
|
774
825
|
} catch (error) {
|
|
775
826
|
throw new WebError(`cannot connect to the CDP endpoint ${endpoint} (${source}); is the browser started with --remote-debugging-port? ${String(error instanceof Error ? error.message : error)}`, "WEB_PROVIDER_ERROR", { cause: error });
|
|
776
827
|
}
|
|
777
828
|
}
|
|
778
829
|
const { chromium, executablePath, source } = await resolvePlaywrightBackend(config.playwrightPath);
|
|
830
|
+
let browser;
|
|
779
831
|
try {
|
|
780
|
-
|
|
832
|
+
browser = await chromium.launch({
|
|
781
833
|
headless: true,
|
|
782
834
|
...executablePath !== void 0 ? { executablePath } : {},
|
|
783
835
|
timeout
|
|
784
836
|
});
|
|
785
837
|
const context = await browser.newContext();
|
|
786
|
-
await
|
|
838
|
+
const page = await context.newPage();
|
|
839
|
+
await installResourceFilter(page);
|
|
840
|
+
guardPopups(page);
|
|
787
841
|
return {
|
|
788
842
|
browser,
|
|
789
|
-
context
|
|
843
|
+
context,
|
|
844
|
+
page
|
|
790
845
|
};
|
|
791
846
|
} catch (error) {
|
|
847
|
+
await browser?.close().catch(() => {});
|
|
792
848
|
throw new WebError(`cannot launch the local browser (${source}); run \`playwright install chromium\` or point the settings path at a playwright/browser executable. ${String(error instanceof Error ? error.message : error)}`, "WEB_PROVIDER_ERROR", { cause: error });
|
|
793
849
|
}
|
|
794
850
|
}
|
|
795
|
-
/** Navigate, settle, and decode one URL inside an open session. */
|
|
851
|
+
/** Navigate, settle, and decode one URL inside an open session's tab. */
|
|
796
852
|
async retrieve(session, url, config, deadline) {
|
|
797
|
-
const page =
|
|
853
|
+
const page = session.page;
|
|
798
854
|
const response = await page.goto(url.toString(), {
|
|
799
855
|
waitUntil: "domcontentloaded",
|
|
800
856
|
timeout: deadline.remainingMs()
|
|
@@ -841,14 +897,17 @@ function capResult(url, statusCode, body) {
|
|
|
841
897
|
};
|
|
842
898
|
}
|
|
843
899
|
/**
|
|
844
|
-
* Close a session's fetch-owned
|
|
845
|
-
*
|
|
846
|
-
*
|
|
847
|
-
*
|
|
900
|
+
* Close a session's fetch-owned tab, then its context unless the context is
|
|
901
|
+
* the remote default context (profile mode — closing it would tear down the
|
|
902
|
+
* whole shared connection), then its browser when the session launched one
|
|
903
|
+
* (local backend). CDP sessions ride the shared connection, which stays
|
|
904
|
+
* open. Every close is grace-bounded so cleanup always completes and the
|
|
905
|
+
* concurrency slot is released even when Playwright hangs.
|
|
848
906
|
*/
|
|
849
907
|
async function closeSession(session) {
|
|
850
908
|
if (session === void 0) return;
|
|
851
|
-
await closeWithGrace(session.
|
|
909
|
+
await closeWithGrace(session.page);
|
|
910
|
+
if (session.persistent !== true) await closeWithGrace(session.context);
|
|
852
911
|
if (session.sharedBrowser !== true) await closeWithGrace(session.browser);
|
|
853
912
|
}
|
|
854
913
|
/** One best-effort `close()` that resolves within the grace period. */
|
|
@@ -876,17 +935,32 @@ async function defaultCdpConnect(endpoint, timeoutMs) {
|
|
|
876
935
|
/**
|
|
877
936
|
* Abort image/font/media subrequests: the markdown output keeps their URLs
|
|
878
937
|
* but never renders them, so downloading them only spends the budget.
|
|
879
|
-
*
|
|
938
|
+
* Installed on the PAGE (never the context): in profile mode a context-level
|
|
939
|
+
* route would intercept the remote browser's other tabs too. Best-effort —
|
|
940
|
+
* a page that refuses interception still fetches.
|
|
880
941
|
*/
|
|
881
|
-
async function installResourceFilter(
|
|
942
|
+
async function installResourceFilter(owner) {
|
|
882
943
|
try {
|
|
883
|
-
await
|
|
944
|
+
await owner.route("**/*", async (route) => {
|
|
884
945
|
const type = route.request().resourceType();
|
|
885
946
|
if (type === "image" || type === "font" || type === "media") await route.abort();
|
|
886
947
|
else await route.continue();
|
|
887
948
|
});
|
|
888
949
|
} catch {}
|
|
889
950
|
}
|
|
951
|
+
/**
|
|
952
|
+
* Close any popup a fetched page spawns so nothing outlives the fetch's tab
|
|
953
|
+
* — `page.close()` does not auto-close windows the page opened, and in
|
|
954
|
+
* profile mode a stray tab would stay in the user's remote browser.
|
|
955
|
+
* Best-effort: a page that refuses listeners just loses the guard.
|
|
956
|
+
*/
|
|
957
|
+
function guardPopups(page) {
|
|
958
|
+
try {
|
|
959
|
+
page.on?.("popup", (popup) => {
|
|
960
|
+
popup.close().catch(() => {});
|
|
961
|
+
});
|
|
962
|
+
} catch {}
|
|
963
|
+
}
|
|
890
964
|
//#endregion
|
|
891
965
|
//#region src/index.ts
|
|
892
966
|
/** Cordis plugin name used by loader diagnostics. */
|
|
@@ -911,4 +985,4 @@ function apply(ctx, config) {
|
|
|
911
985
|
ctx.web.registerFetchProvider(provider);
|
|
912
986
|
}
|
|
913
987
|
//#endregion
|
|
914
|
-
export { CdpConnectionPool, Config, DEFAULT_CDP_ENDPOINT, PLAYWRIGHT_FETCH_PROVIDER_ID, PlaywrightFetchProvider, WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE, apply, htmlToMarkdown, inject, name, normalizeCdpEndpoint };
|
|
988
|
+
export { CdpConnectionPool, Config, DEFAULT_CDP_ENDPOINT, PLAYWRIGHT_FETCH_PROVIDER_ID, PlaywrightFetchProvider, WEB_FETCH_PLAYWRIGHT_SETTINGS_NAMESPACE, apply, effectiveContextMode, htmlToMarkdown, inject, name, normalizeCdpEndpoint };
|