dsh-web-fetch-playwright 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 chendefine
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,108 @@
1
+ # dsh-web-fetch-playwright
2
+
3
+ [中文](./README.zh-CN.md) · [npm](https://www.npmjs.com/package/dsh-web-fetch-playwright) · [GitHub](https://github.com/chendefine/dsh-web-fetch-playwright)
4
+
5
+ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plugin that gives the built-in `web_fetch` tool a **Playwright/CDP backend**: pages are rendered in a real browser, denoised with **Readability + DOMPurify + Turndown + GFM**, and returned as Markdown.
6
+
7
+ ![npm](https://img.shields.io/npm/v/dsh-web-fetch-playwright) ![license](https://img.shields.io/npm/l/dsh-web-fetch-playwright) ![node](https://img.shields.io/node/v/dsh-web-fetch-playwright) ![CI](https://img.shields.io/github/actions/workflow/status/chendefine/dsh-web-fetch-playwright/ci.yml) ![stars](https://img.shields.io/github/stars/chendefine/dsh-web-fetch-playwright)
8
+
9
+ ## Features
10
+
11
+ - **Real browser rendering** — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
12
+ - **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer.
13
+ - **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
14
+ - **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
15
+ - **Isolated sessions** — every fetch uses its own browser context; local launches close their browser, CDP connections only disconnect. Nothing outlives the call that opened it.
16
+ - **Live configuration** — a settings card (设置 → 插件 → 插件配置) edits the backend and denoise toggle; changes apply to the next fetch without a restart.
17
+ - **Budget-aware** — per-fetch deadline (45s), concurrent-fetch semaphore (2 browsers), image/font/media subrequests aborted, body capped at 100k chars.
18
+
19
+ ## How it works
20
+
21
+ | Half | Location | Responsibility |
22
+ | --- | --- | --- |
23
+ | Host (server) | `src/` | Registers the fetch provider (id `playwright`) into `ctx.web`; `cordis.patch.yml` pins the web seam's `fetchProvider` to it and enables the `web_fetch` tool with a 60s budget. |
24
+ | Browser (client) | `src/client/` | Registers the *Playwright 网页爬取* configuration card, which hot-writes the settings section into `$DSH_HOME/settings.yaml`. |
25
+
26
+ ```
27
+ web_fetch (tool-web)
28
+ └─ ctx.web.fetchProvider = playwright
29
+ ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
30
+ ├─ cdp: connectOverCDP(endpoint)
31
+ ├─ page.goto → settle (networkidle, best-effort) → page.content()
32
+ ├─ denoise: jsdom → Readability → DOMPurify → Turndown(GFM)
33
+ └─ Markdown (or raw HTML when denoise is off)
34
+ ```
35
+
36
+ ## Requirements
37
+
38
+ - DSH web profile (`dsh web`), Node.js ≥ 20.
39
+ - For the **local** backend: a Playwright installation with Chromium, a Chromium-family browser binary, or `playwright-core` with a browser in the default cache.
40
+ - For the **CDP** backend: any browser already running with `--remote-debugging-port` (e.g. `chromium --headless --remote-debugging-port=9222`).
41
+
42
+ ## Installation
43
+
44
+ From the npm registry (prebuilt — no build permission needed):
45
+
46
+ ```sh
47
+ dsh plugin --profile web add dsh-web-fetch-playwright
48
+ ```
49
+
50
+ From a GitHub repository (source — pnpm runs the `prepare` build; allowlist the package in `profiles/web/pnpm-workspace.yaml` if pnpm blocks the build script):
51
+
52
+ ```sh
53
+ dsh plugin --profile web add github:chendefine/dsh-web-fetch-playwright
54
+ ```
55
+
56
+ Or through the DSH plugin marketplace (设置 → DSH插件市场) — the repo carries the `dsh-plugin` topic and is indexed automatically.
57
+
58
+ After a bundle plugin is added to the profile layer stack, **restart `dsh web`** for it to load; uninstall with `dsh plugin --profile web remove dsh-web-fetch-playwright` and restart again.
59
+
60
+ ## Configuration
61
+
62
+ The settings card (设置 → 插件 → 插件配置 → *Playwright 网页爬取*) edits the `web-fetch-playwright` settings section live:
63
+
64
+ | Field | Default | Description |
65
+ | --- | --- | --- |
66
+ | `backend` | `local` | Radio: *Local Playwright* or *Remote CDP endpoint*, each with its own nested input. |
67
+ | `playwrightPath` | (blank) | Local backend: path to a `playwright` executable or a Chromium-family browser binary. Blank = discover on `$PATH`, then fall back to the bundled `playwright-core`. |
68
+ | `cdpEndpoint` | `127.0.0.1:9222` | Remote backend: `host:port`, `http(s)://…` or `ws(s)://…`. |
69
+ | `denoise` | `true` | Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert. |
70
+
71
+ Local backend resolution order:
72
+
73
+ 1. The configured path (auto-detected as Playwright CLI or browser binary).
74
+ 2. A `playwright` executable on `$PATH` (its package knows that installation's browser registry).
75
+ 3. The bundled `playwright-core` — requires `PLAYWRIGHT_BROWSERS_PATH` or browsers in the default cache; otherwise the error suggests `playwright install chromium`.
76
+
77
+ CDP mode needs no local browser: it connects fresh per fetch and uses an isolated context, so it never pollutes an existing browser session.
78
+
79
+ ## Development
80
+
81
+ ```sh
82
+ pnpm install
83
+ pnpm typecheck # tsc --noEmit
84
+ pnpm test # vitest run (browser smoke self-skips without a browser)
85
+ pnpm build # tsc declarations + tsdown (host ESM + client module-registration bundle)
86
+ ```
87
+
88
+ Repository layout:
89
+
90
+ ```
91
+ src/
92
+ ├── index.ts # host entry: registers provider + settings section
93
+ ├── config.ts # schemastery schema, CDP endpoint normalizer
94
+ ├── provider.ts # WebFetchProvider: navigation, deadline, semaphore, caps
95
+ ├── markdown.ts # denoise pipeline (Readability + DOMPurify + Turndown/GFM)
96
+ ├── playwright-resolve.ts # local backend discovery (path / $PATH / bundled core)
97
+ ├── types.ts # structural Playwright types (runtime module discovered dynamically)
98
+ └── client/ # browser half: settings card, form model, locales
99
+ tests/ # unit + provider + browser integration (self-skipping)
100
+ ```
101
+
102
+ ## Security
103
+
104
+ Same stance as the built-in HTTP provider: **no SSRF / private-network protection is implemented** — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself.
105
+
106
+ ## License
107
+
108
+ [MIT](./LICENSE) © 2026 chendefine
@@ -0,0 +1,108 @@
1
+ # dsh-web-fetch-playwright
2
+
3
+ [English](./README.md) · [npm](https://www.npmjs.com/package/dsh-web-fetch-playwright) · [GitHub](https://github.com/chendefine/dsh-web-fetch-playwright)
4
+
5
+ [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(DSH)双端插件:为内置 `web_fetch` 工具提供 **Playwright/CDP 后端**——用真实浏览器渲染网页,经 **Readability + DOMPurify + Turndown + GFM** 降噪(清洗导航栏、侧边栏、页脚、贴片广告)后输出 Markdown。
6
+
7
+ ![npm](https://img.shields.io/npm/v/dsh-web-fetch-playwright) ![license](https://img.shields.io/npm/l/dsh-web-fetch-playwright) ![node](https://img.shields.io/node/v/dsh-web-fetch-playwright) ![CI](https://img.shields.io/github/actions/workflow/status/chendefine/dsh-web-fetch-playwright/ci.yml) ![stars](https://img.shields.io/github/stars/chendefine/dsh-web-fetch-playwright)
8
+
9
+ ## 特性
10
+
11
+ - **真实浏览器渲染** —— 以用户视角加载页面,SPA 客户端渲染内容也能抓到,而非只有原始 HTML。
12
+ - **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。
13
+ - **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
14
+ - **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
15
+ - **隔离会话** —— 每次抓取使用独立 browser context;本地启动用完即关,CDP 连接仅断开,浏览器不会比调用它的一次抓取活得更久。
16
+ - **热配置** —— 「设置 → 插件 → 插件配置」卡片可随时切换后端与降噪开关,改动对下一次抓取即时生效,无需重启。
17
+ - **预算控制** —— 单次抓取 45s 超时、并发上限 2 个浏览器、拦截图片/字体/媒体子请求、返回体 10 万字符封顶。
18
+
19
+ ## 工作原理
20
+
21
+ | 半端 | 位置 | 职责 |
22
+ | --- | --- | --- |
23
+ | 宿主(服务端) | `src/` | 向 `ctx.web` 注册 fetch provider(id `playwright`);`cordis.patch.yml` 把 web seam 的 `fetchProvider` 固定为本插件,并启用 `web_fetch` 工具(60s 预算)。 |
24
+ | 浏览器(客户端) | `src/client/` | 注册 *Playwright 网页爬取* 配置卡片,通过 settings 服务把改动热写入 `$DSH_HOME/settings.yaml`。 |
25
+
26
+ ```
27
+ web_fetch (tool-web)
28
+ └─ ctx.web.fetchProvider = playwright
29
+ ├─ local: 解析(路径 → $PATH → 内置 playwright-core)→ chromium.launch
30
+ ├─ cdp: connectOverCDP(endpoint)
31
+ ├─ page.goto → 等待稳定(networkidle,尽力而为)→ page.content()
32
+ ├─ 降噪:jsdom → Readability → DOMPurify → Turndown(GFM)
33
+ └─ Markdown(关闭降噪时返回原始 HTML)
34
+ ```
35
+
36
+ ## 环境要求
37
+
38
+ - DSH web profile(`dsh web`),Node.js ≥ 20。
39
+ - **本地**后端:装有 Chromium 的 Playwright、Chromium 系浏览器可执行文件,或默认缓存里有浏览器的 `playwright-core`。
40
+ - **CDP** 后端:任意已带 `--remote-debugging-port` 启动的浏览器(如 `chromium --headless --remote-debugging-port=9222`)。
41
+
42
+ ## 安装
43
+
44
+ 从 npm registry 安装(预构建产物,无需构建授权):
45
+
46
+ ```sh
47
+ dsh plugin --profile web add dsh-web-fetch-playwright
48
+ ```
49
+
50
+ 从 GitHub 仓库安装(源码型,pnpm 会在安装时跑 `prepare` 构建;若 pnpm 拦截构建脚本,请在 `profiles/web/pnpm-workspace.yaml` 中放行该包):
51
+
52
+ ```sh
53
+ dsh plugin --profile web add github:chendefine/dsh-web-fetch-playwright
54
+ ```
55
+
56
+ 或通过 DSH 插件市场(设置 → DSH插件市场)一键安装——本仓库带 `dsh-plugin` topic,会被自动收录。
57
+
58
+ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用 `dsh plugin --profile web remove dsh-web-fetch-playwright` 后重启。
59
+
60
+ ## 配置项
61
+
62
+ 设置卡片(设置 → 插件 → 插件配置 → *Playwright 网页爬取*)实时编辑 `web-fetch-playwright` 设置段:
63
+
64
+ | 字段 | 默认 | 说明 |
65
+ | --- | --- | --- |
66
+ | `backend` | `local` | radio:本地 Playwright / 远端 CDP 地址,每个选项内嵌各自的填空。 |
67
+ | `playwrightPath` | 空 | 本地后端:`playwright` 可执行文件或 Chromium 系浏览器二进制路径;留空按 `$PATH` 查找,再回退到内置 `playwright-core`。 |
68
+ | `cdpEndpoint` | `127.0.0.1:9222` | 远端后端:`host:port`、`http(s)://…` 或 `ws(s)://…`。 |
69
+ | `denoise` | `true` | 是否启用降噪;关闭时返回整页渲染 HTML,交由工具层转换。 |
70
+
71
+ 本地后端解析顺序:
72
+
73
+ 1. 配置的路径(自动判别 Playwright CLI / 浏览器二进制);
74
+ 2. `$PATH` 上的 `playwright`(其包自带该安装的浏览器注册表);
75
+ 3. 插件内置的 `playwright-core`——需要 `PLAYWRIGHT_BROWSERS_PATH` 或默认缓存里有浏览器,否则报错会提示 `playwright install chromium`。
76
+
77
+ CDP 模式不需要本地浏览器:每次抓取独立连接并使用隔离 context,不污染已有浏览器会话。
78
+
79
+ ## 开发
80
+
81
+ ```sh
82
+ pnpm install
83
+ pnpm typecheck # tsc --noEmit
84
+ pnpm test # vitest run(无浏览器时浏览器集成用例自动跳过)
85
+ pnpm build # tsc 声明 + tsdown(宿主 ESM + 客户端 module-registration bundle)
86
+ ```
87
+
88
+ 仓库结构:
89
+
90
+ ```
91
+ src/
92
+ ├── index.ts # 宿主入口:注册 provider 与设置段
93
+ ├── config.ts # schemastery schema、CDP 端点归一化
94
+ ├── provider.ts # WebFetchProvider:导航、超时、信号量、截断
95
+ ├── markdown.ts # 降噪管线(Readability + DOMPurify + Turndown/GFM)
96
+ ├── playwright-resolve.ts # 本地后端发现(路径 / $PATH / 内置 core)
97
+ ├── types.ts # Playwright 结构化类型(运行时模块动态发现)
98
+ └── client/ # 浏览器半端:设置卡片、表单模型、多语言
99
+ tests/ # 单元 + provider + 浏览器集成(可自跳过)
100
+ ```
101
+
102
+ ## 安全边界
103
+
104
+ 与内置 HTTP provider 同立场:**未实现 SSRF/私网防护**——浏览器能访问的目标,本 provider 就能抓。CDP 地址由设置页配置,不做回环限制,请在可信环境暴露设置页。抓取仅在本地渲染,除目标页面自身外不会向任何地方发送数据。
105
+
106
+ ## 许可证
107
+
108
+ [MIT](./LICENSE) © 2026 chendefine
@@ -0,0 +1,31 @@
1
+ # dsh-web-fetch-playwright bundle layer.
2
+ #
3
+ # 1. Pin the web seam's fetch provider to this plugin's provider (id
4
+ # 'playwright'). A patch replaces the targeted row's WHOLE config, so the
5
+ # shipped search selection is restated verbatim — losing it would break
6
+ # web_search.
7
+ # 2. Enable the model-facing web_fetch tool with a browser-realistic budget
8
+ # (launch + render + denoise takes longer than a plain HTTP GET; 60s
9
+ # matches the deployment's search budget). `disabled: false` re-enables
10
+ # the row the dsh-web-app layer disables (that layer parks tool-web on the
11
+ # preset plane with fetch: false; this layer — applied after it — turns
12
+ # the host-plane row back on so every session gets web_fetch).
13
+ # 3. Insert this package's host plugin row.
14
+ #
15
+ # Bundle layers load at the next profile start; the settings section
16
+ # ('web-fetch-playwright') layering over the row config is hot-reloaded.
17
+ - id: web
18
+ config:
19
+ searchProvider: deepseek-official
20
+ fetchProvider: playwright
21
+
22
+ - id: tool-web
23
+ disabled: false
24
+ config:
25
+ fetch: true
26
+ fetchTimeoutMs: 60000
27
+ searchTimeoutMs: 60000
28
+
29
+ - insert:
30
+ - id: web-fetch-playwright
31
+ name: dsh-web-fetch-playwright
@@ -0,0 +1,16 @@
1
+ {
2
+ "name": "dsh-web-fetch-playwright",
3
+ "version": "0.1.0",
4
+ "description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
5
+ "entry": {
6
+ "name": "dsh-web-fetch-playwright",
7
+ "inject": []
8
+ },
9
+ "client": {
10
+ "platform": "web"
11
+ },
12
+ "contributes": {
13
+ "tools": [],
14
+ "skills": []
15
+ }
16
+ }