dsh-web-fetch-playwright 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -12,9 +12,9 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plug
12
12
  - **Denoise pipeline** — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped `tool-web` renderer.
13
13
  - **Two backends** — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
14
14
  - **Browser resolution** — a configured path, a `playwright` CLI on `$PATH`, or the bundled `playwright-core`; CDP needs no local browser at all.
15
- - **Isolated sessions** — every fetch uses its own browser context; local launches close their browser, CDP connections only disconnect. Nothing outlives the call that opened it.
16
- - **Live configuration** — a settings card (设置 → 插件 → 插件配置) edits the backend and denoise toggle; changes apply to the next fetch without a restart.
17
- - **Budget-aware** — per-fetch deadline (45s), concurrent-fetch semaphore (2 browsers), image/font/media subrequests aborted, body capped at 100k chars.
15
+ - **Isolated sessions** — every fetch uses its own browser context. Local launches close their browser per fetch; the CDP backend keeps **one shared connection** to the remote browser and each fetch opens an isolated context (a tab) inside it, closed when done.
16
+ - **Live configuration** — a settings card (设置 → 插件 → 插件配置) edits the backend, denoise toggle, and concurrency; changes apply to the next fetch without a restart.
17
+ - **Budget-aware** — per-fetch deadline (45s); concurrency is backend-priced (`maxConcurrency`, default 4 local browsers / **50 CDP tabs**; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
18
18
 
19
19
  ## How it works
20
20
 
@@ -67,6 +67,7 @@ The settings card (设置 → 插件 → 插件配置 → *Playwright 网页爬
67
67
  | `playwrightPath` | (blank) | Local backend: path to a `playwright` executable or a Chromium-family browser binary. Blank = discover on `$PATH`, then fall back to the bundled `playwright-core`. |
68
68
  | `cdpEndpoint` | `127.0.0.1:9222` | Remote backend: `host:port`, `http(s)://…` or `ws(s)://…`. |
69
69
  | `denoise` | `true` | Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert. |
70
+ | `maxConcurrency` | *(auto)* | How many fetches may render at once (1–200). Blank = backend default: **4** for local (each slot launches a browser) / **50** for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with `WEB_FETCH_TIMEOUT` and a hint to retry or raise this setting, rather than hanging until the tool budget aborts. |
70
71
 
71
72
  Local backend resolution order:
72
73
 
@@ -74,7 +75,7 @@ Local backend resolution order:
74
75
  2. A `playwright` executable on `$PATH` (its package knows that installation's browser registry).
75
76
  3. The bundled `playwright-core` — requires `PLAYWRIGHT_BROWSERS_PATH` or browsers in the default cache; otherwise the error suggests `playwright install chromium`.
76
77
 
77
- CDP mode needs no local browser: it connects fresh per fetch and uses an isolated context, so it never pollutes an existing browser session.
78
+ CDP mode needs no local browser: the provider holds **one shared connection** for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a fresh isolated context — a tab in your browser — that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection.
78
79
 
79
80
  ## Development
80
81
 
@@ -99,6 +100,8 @@ src/
99
100
  tests/ # unit + provider + browser integration (self-skipping)
100
101
  ```
101
102
 
103
+ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development and release workflow, and [SECURITY.md](./SECURITY.md) for the security model and reporting policy.
104
+
102
105
  ## Security
103
106
 
104
107
  Same stance as the built-in HTTP provider: **no SSRF / private-network protection is implemented** — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself.
package/README.zh-CN.md CHANGED
@@ -12,9 +12,9 @@
12
12
  - **降噪管线** —— Mozilla Readability 提取正文,DOMPurify 移除布局/噪音标签(导航、侧边栏、页脚、广告、表单),Turndown + GFM 插件按与内置 `tool-web` 渲染器一致的风格转成 Markdown。
13
13
  - **两种后端** —— 本地启动 Playwright 浏览器,或通过 DevTools 协议(CDP)驱动一个已在运行的浏览器。
14
14
  - **浏览器解析** —— 配置路径 → `$PATH` 上的 `playwright` CLI → 插件自带的 `playwright-core`;CDP 模式完全不需要本地浏览器。
15
- - **隔离会话** —— 每次抓取使用独立 browser context;本地启动用完即关,CDP 连接仅断开,浏览器不会比调用它的一次抓取活得更久。
16
- - **热配置** —— 「设置 → 插件 → 插件配置」卡片可随时切换后端与降噪开关,改动对下一次抓取即时生效,无需重启。
17
- - **预算控制** —— 单次抓取 45s 超时、并发上限 2 个浏览器、拦截图片/字体/媒体子请求、返回体 10 万字符封顶。
15
+ - **隔离会话** —— 每次抓取使用独立 browser context。本地后端每次抓取启动并关闭自己的浏览器;CDP 后端对远端浏览器保持**一条共享连接**,每次抓取只在其中开一个隔离 context(标签页),用完即关。
16
+ - **热配置** —— 「设置 → 插件 → 插件配置」卡片可随时切换后端、降噪开关与并发数,改动对下一次抓取即时生效,无需重启。
17
+ - **预算控制** —— 单次抓取 45s 超时;并发按后端定价(`maxConcurrency`,默认本地 4 个浏览器 / **CDP 50 个标签页**;排队的抓取等不到空位会在 20s 内尽快报错并提示重试,而不是一直挂到被工具层中止);拦截图片/字体/媒体子请求;返回体 10 万字符封顶。
18
18
 
19
19
  ## 工作原理
20
20
 
@@ -67,6 +67,7 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
67
67
  | `playwrightPath` | 空 | 本地后端:`playwright` 可执行文件或 Chromium 系浏览器二进制路径;留空按 `$PATH` 查找,再回退到内置 `playwright-core`。 |
68
68
  | `cdpEndpoint` | `127.0.0.1:9222` | 远端后端:`host:port`、`http(s)://…` 或 `ws(s)://…`。 |
69
69
  | `denoise` | `true` | 是否启用降噪;关闭时返回整页渲染 HTML,交由工具层转换。 |
70
+ | `maxConcurrency` | *(自动)* | 同时渲染的页面上限(1–200)。留空按后端取默认:本地 **4**(每个槽位启动一个浏览器)/ CDP **50**(远端浏览器已就位,每个槽位只是一个标签页)。超出的请求短暂排队;20s 内等不到空位则以 `WEB_FETCH_TIMEOUT` 尽快失败并提示重试或调大该值,而不是一直挂起直到工具层预算中止。 |
70
71
 
71
72
  本地后端解析顺序:
72
73
 
@@ -74,7 +75,7 @@ bundle 插件加入 profile 层栈后需**重启 `dsh web`** 生效;卸载用
74
75
  2. `$PATH` 上的 `playwright`(其包自带该安装的浏览器注册表);
75
76
  3. 插件内置的 `playwright-core`——需要 `PLAYWRIGHT_BROWSERS_PATH` 或默认缓存里有浏览器,否则报错会提示 `playwright install chromium`。
76
77
 
77
- CDP 模式不需要本地浏览器:每次抓取独立连接并使用隔离 context,不污染已有浏览器会话。
78
+ CDP 模式不需要本地浏览器:插件在生命周期内对远端浏览器保持**一条共享连接**(连接断开自动重连,地址改动后自动换连),每次抓取只租用一个全新的隔离 context——即远端浏览器里的一个标签页——抓取结束即关闭、不污染已有会话。因此并发数按"标签页"计,默认也更高(50)。插件卸载时断开共享连接。
78
79
 
79
80
  ## 开发
80
81
 
@@ -99,6 +100,8 @@ src/
99
100
  tests/ # 单元 + provider + 浏览器集成(可自跳过)
100
101
  ```
101
102
 
103
+ 开发与发布流程见 [CONTRIBUTING.md](./CONTRIBUTING.md),安全模型与漏洞报告见 [SECURITY.md](./SECURITY.md)。
104
+
102
105
  ## 安全边界
103
106
 
104
107
  与内置 HTTP provider 同立场:**未实现 SSRF/私网防护**——浏览器能访问的目标,本 provider 就能抓。CDP 地址由设置页配置,不做回环限制,请在可信环境暴露设置页。抓取仅在本地渲染,除目标页面自身外不会向任何地方发送数据。
package/dsh.plugin.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-web-fetch-playwright",
3
- "version": "0.1.0",
3
+ "version": "0.2.0",
4
4
  "description": "Playwright/CDP web-fetch provider for DeepSeek Harness: renders pages in a real browser, denoises them (Readability + DOMPurify), and returns markdown.",
5
5
  "entry": {
6
6
  "name": "dsh-web-fetch-playwright",
package/lib/client.js CHANGED
@@ -101,6 +101,31 @@ window.__ModuleLoader__.load({
101
101
  };
102
102
  }
103
103
  /**
104
+ * An integer field over an inclusive range. The draft is the decimal string;
105
+ * an empty draft clears the field (re-inherit the schema default), and
106
+ * anything but an integer in range blocks the save as invalid.
107
+ * @param field - field name inside the namespace section.
108
+ * @param min - smallest accepted integer (inclusive).
109
+ * @param max - largest accepted integer (inclusive).
110
+ * @returns the field's conversion spec.
111
+ */
112
+ function numberField(field, min, max) {
113
+ return {
114
+ field,
115
+ format: (value) => typeof value === "number" && Number.isInteger(value) ? String(value) : "",
116
+ parse: (text) => {
117
+ const trimmed = text.trim();
118
+ if (trimmed === "") return { kind: "clear" };
119
+ if (!/^\d+$/.test(trimmed)) return void 0;
120
+ const value = Number(trimmed);
121
+ return value >= min && value <= max ? {
122
+ kind: "set",
123
+ value
124
+ } : void 0;
125
+ }
126
+ };
127
+ }
128
+ /**
104
129
  * Stages the Playwright card's edits over one settings namespace and writes
105
130
  * them on save. Publishes through a snapshot store because slot components
106
131
  * read through a selector while both the scope and the drafts change below.
@@ -323,7 +348,8 @@ window.__ModuleLoader__.load({
323
348
  radioField("backend", ["local", "cdp"]),
324
349
  textField("playwrightPath"),
325
350
  textField("cdpEndpoint"),
326
- checkboxField("denoise")
351
+ checkboxField("denoise"),
352
+ numberField("maxConcurrency", 1, 200)
327
353
  ]);
328
354
  this.store = this.form.bind(() => this.projection());
329
355
  }
@@ -333,7 +359,8 @@ window.__ModuleLoader__.load({
333
359
  backend: this.form.field("backend"),
334
360
  playwrightPath: this.form.field("playwrightPath"),
335
361
  cdpEndpoint: this.form.field("cdpEndpoint"),
336
- denoise: this.form.field("denoise")
362
+ denoise: this.form.field("denoise"),
363
+ maxConcurrency: this.form.field("maxConcurrency")
337
364
  };
338
365
  }
339
366
  /**
@@ -359,21 +386,21 @@ window.__ModuleLoader__.load({
359
386
  document.head.appendChild(tag);
360
387
  }
361
388
  var PluginCard_module_css_default = {
389
+ "card": "UM8-Rq_card",
362
390
  "chevronOpen": "UM8-Rq_chevronOpen",
363
- "failed": "UM8-Rq_failed",
364
- "name": "UM8-Rq_name",
365
391
  "readOnly": "UM8-Rq_readOnly",
366
- "headText": "UM8-Rq_headText",
367
- "discard": "UM8-Rq_discard",
368
- "cardOpen": "UM8-Rq_cardOpen",
369
392
  "footer": "UM8-Rq_footer",
370
- "card": "UM8-Rq_card",
371
- "header": "UM8-Rq_header",
372
- "body": "UM8-Rq_body",
373
393
  "pending": "UM8-Rq_pending",
394
+ "header": "UM8-Rq_header",
374
395
  "save": "UM8-Rq_save",
375
- "description": "UM8-Rq_description",
376
- "chevron": "UM8-Rq_chevron"
396
+ "chevron": "UM8-Rq_chevron",
397
+ "failed": "UM8-Rq_failed",
398
+ "headText": "UM8-Rq_headText",
399
+ "body": "UM8-Rq_body",
400
+ "cardOpen": "UM8-Rq_cardOpen",
401
+ "discard": "UM8-Rq_discard",
402
+ "name": "UM8-Rq_name",
403
+ "description": "UM8-Rq_description"
377
404
  };
378
405
  //#endregion
379
406
  //#region src/client/PluginCard.tsx
@@ -490,25 +517,25 @@ window.__ModuleLoader__.load({
490
517
  document.head.appendChild(tag);
491
518
  }
492
519
  var fields_module_css_default = {
493
- "invalid": "CFSaVG_invalid",
494
- "badge": "CFSaVG_badge",
495
- "reset": "CFSaVG_reset",
496
- "input": "CFSaVG_input",
497
- "radioHint": "CFSaVG_radioHint",
498
520
  "checkboxRow": "CFSaVG_checkboxRow",
499
- "field": "CFSaVG_field",
500
- "badges": "CFSaVG_badges",
501
521
  "radioOption": "CFSaVG_radioOption",
522
+ "radioLabel": "CFSaVG_radioLabel",
502
523
  "head": "CFSaVG_head",
503
- "radioGroup": "CFSaVG_radioGroup",
504
524
  "radioPick": "CFSaVG_radioPick",
505
- "hint": "CFSaVG_hint",
506
- "radioContent": "CFSaVG_radioContent",
525
+ "badges": "CFSaVG_badges",
526
+ "radioGroup": "CFSaVG_radioGroup",
507
527
  "fieldEmbedded": "CFSaVG_fieldEmbedded",
508
- "inputInvalid": "CFSaVG_inputInvalid",
509
- "radioLabel": "CFSaVG_radioLabel",
528
+ "radioText": "CFSaVG_radioText",
529
+ "radioHint": "CFSaVG_radioHint",
530
+ "radioContent": "CFSaVG_radioContent",
531
+ "field": "CFSaVG_field",
532
+ "input": "CFSaVG_input",
533
+ "badge": "CFSaVG_badge",
534
+ "invalid": "CFSaVG_invalid",
535
+ "hint": "CFSaVG_hint",
510
536
  "label": "CFSaVG_label",
511
- "radioText": "CFSaVG_radioText"
537
+ "reset": "CFSaVG_reset",
538
+ "inputInvalid": "CFSaVG_inputInvalid"
512
539
  };
513
540
  //#endregion
514
541
  //#region src/client/fields.tsx
@@ -683,80 +710,100 @@ window.__ModuleLoader__.load({
683
710
  state,
684
711
  onSave: props.save,
685
712
  onDiscard: props.discard,
686
- children: [/* @__PURE__ */ (0, react_jsx_runtime.jsx)(RadioGroupField, {
687
- label: t("backendLabel"),
688
- options: [{
689
- value: "local",
690
- label: t("backendLocal"),
691
- hint: t("backendLocalHint"),
692
- content: /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
693
- embedded: true,
694
- id: "plugin-config-playwright-path",
695
- label: t("playwrightPath"),
696
- hint: t("playwrightPathHint"),
697
- placeholder: t("playwrightPathPlaceholder"),
698
- overriddenLabel: t("overridden"),
699
- resetLabel: t("reset"),
700
- invalidLabel: t("invalidText"),
701
- disabled: disabled || backend !== "local",
702
- ...state.playwrightPath,
703
- onEdit: (text) => {
704
- props.edit("playwrightPath", text);
705
- },
706
- onReset: () => {
707
- props.resetField("playwrightPath");
708
- }
709
- })
710
- }, {
711
- value: "cdp",
712
- label: t("backendCdp"),
713
- hint: t("backendCdpHint"),
714
- content: /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
715
- embedded: true,
716
- id: "plugin-config-playwright-cdp",
717
- label: t("cdpEndpoint"),
718
- hint: t("cdpEndpointHint"),
719
- placeholder: "127.0.0.1:9222",
720
- overriddenLabel: t("overridden"),
721
- resetLabel: t("reset"),
722
- invalidLabel: t("invalidText"),
723
- disabled: disabled || backend !== "cdp",
724
- ...state.cdpEndpoint,
725
- onEdit: (text) => {
726
- props.edit("cdpEndpoint", text);
727
- },
728
- onReset: () => {
729
- props.resetField("cdpEndpoint");
730
- }
731
- })
732
- }],
733
- text: state.backend.text,
734
- overridden: state.backend.overridden,
735
- overriddenLabel: t("overridden"),
736
- resetLabel: t("reset"),
737
- disabled,
738
- onEdit: (text) => {
739
- props.edit("backend", text);
740
- },
741
- onReset: () => {
742
- props.resetField("backend");
743
- }
744
- }), /* @__PURE__ */ (0, react_jsx_runtime.jsx)(CheckboxField, {
745
- id: "plugin-config-playwright-denoise",
746
- label: t("denoise"),
747
- hint: t("denoiseHint"),
748
- checked: state.denoise.text !== "false",
749
- overridden: state.denoise.overridden,
750
- overriddenLabel: t("overridden"),
751
- resetLabel: t("reset"),
752
- disabled,
753
- onEdit: (text) => {
754
- props.edit("denoise", text);
755
- },
756
- onReset: () => {
757
- props.resetField("denoise");
758
- }
759
- })]
713
+ children: [
714
+ /* @__PURE__ */ (0, react_jsx_runtime.jsx)(RadioGroupField, {
715
+ label: t("backendLabel"),
716
+ options: [{
717
+ value: "local",
718
+ label: t("backendLocal"),
719
+ hint: t("backendLocalHint"),
720
+ content: /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
721
+ embedded: true,
722
+ id: "plugin-config-playwright-path",
723
+ label: t("playwrightPath"),
724
+ hint: t("playwrightPathHint"),
725
+ placeholder: t("playwrightPathPlaceholder"),
726
+ overriddenLabel: t("overridden"),
727
+ resetLabel: t("reset"),
728
+ invalidLabel: t("invalidText"),
729
+ disabled: disabled || backend !== "local",
730
+ ...state.playwrightPath,
731
+ onEdit: (text) => {
732
+ props.edit("playwrightPath", text);
733
+ },
734
+ onReset: () => {
735
+ props.resetField("playwrightPath");
736
+ }
737
+ })
738
+ }, {
739
+ value: "cdp",
740
+ label: t("backendCdp"),
741
+ hint: t("backendCdpHint"),
742
+ content: /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
743
+ embedded: true,
744
+ id: "plugin-config-playwright-cdp",
745
+ label: t("cdpEndpoint"),
746
+ hint: t("cdpEndpointHint"),
747
+ placeholder: "127.0.0.1:9222",
748
+ overriddenLabel: t("overridden"),
749
+ resetLabel: t("reset"),
750
+ invalidLabel: t("invalidText"),
751
+ disabled: disabled || backend !== "cdp",
752
+ ...state.cdpEndpoint,
753
+ onEdit: (text) => {
754
+ props.edit("cdpEndpoint", text);
755
+ },
756
+ onReset: () => {
757
+ props.resetField("cdpEndpoint");
758
+ }
759
+ })
760
+ }],
761
+ text: state.backend.text,
762
+ overridden: state.backend.overridden,
763
+ overriddenLabel: t("overridden"),
764
+ resetLabel: t("reset"),
765
+ disabled,
766
+ onEdit: (text) => {
767
+ props.edit("backend", text);
768
+ },
769
+ onReset: () => {
770
+ props.resetField("backend");
771
+ }
772
+ }),
773
+ /* @__PURE__ */ (0, react_jsx_runtime.jsx)(CheckboxField, {
774
+ id: "plugin-config-playwright-denoise",
775
+ label: t("denoise"),
776
+ hint: t("denoiseHint"),
777
+ checked: state.denoise.text !== "false",
778
+ overridden: state.denoise.overridden,
779
+ overriddenLabel: t("overridden"),
780
+ resetLabel: t("reset"),
781
+ disabled,
782
+ onEdit: (text) => {
783
+ props.edit("denoise", text);
784
+ },
785
+ onReset: () => {
786
+ props.resetField("denoise");
787
+ }
788
+ }),
789
+ /* @__PURE__ */ (0, react_jsx_runtime.jsx)(ValueField, {
790
+ id: "plugin-config-playwright-concurrency",
791
+ label: t("maxConcurrency"),
792
+ hint: t("maxConcurrencyHint"),
793
+ placeholder: t("maxConcurrencyPlaceholder"),
794
+ overriddenLabel: t("overridden"),
795
+ resetLabel: t("reset"),
796
+ invalidLabel: t("invalidText"),
797
+ disabled,
798
+ ...state.maxConcurrency,
799
+ onEdit: (text) => {
800
+ props.edit("maxConcurrency", text);
801
+ },
802
+ onReset: () => {
803
+ props.resetField("maxConcurrency");
804
+ }
805
+ })
806
+ ]
760
807
  });
761
808
  }
762
809
  //#endregion
@@ -777,6 +824,9 @@ window.__ModuleLoader__.load({
777
824
  cdpEndpointHint: "host:port or http(s)/ws URL. Leave blank for 127.0.0.1:9222.",
778
825
  denoise: "Enable the denoise algorithm",
779
826
  denoiseHint: "Readability + DOMPurify strip nav bars, sidebars, footers, and ads before converting to markdown.",
827
+ maxConcurrency: "Max concurrent fetches",
828
+ maxConcurrencyHint: "How many pages may render at once (1–200). Blank = auto: 4 local browsers, 50 CDP tabs in the remote browser.",
829
+ maxConcurrencyPlaceholder: "(auto: local 4 / CDP 50)",
780
830
  overridden: "Overridden",
781
831
  reset: "Reset to default",
782
832
  readOnly: "This deployment stores settings read-only.",
@@ -805,6 +855,9 @@ window.__ModuleLoader__.load({
805
855
  cdpEndpointHint: "host:port 或 http(s)/ws 地址;留空默认 127.0.0.1:9222。",
806
856
  denoise: "启用降噪算法",
807
857
  denoiseHint: "使用 Readability + DOMPurify 清洗导航栏、侧边栏、页脚与贴片广告后再转为 Markdown。",
858
+ maxConcurrency: "最大并发抓取数",
859
+ maxConcurrencyHint: "同时渲染的页面上限(1–200);留空自动:本地 4 个浏览器,CDP 在远端浏览器里 50 个标签页。",
860
+ maxConcurrencyPlaceholder: "(自动:本地 4 / CDP 50)",
808
861
  overridden: "已覆盖",
809
862
  reset: "恢复默认",
810
863
  readOnly: "本部署的设置为只读。",