dsh-web-fetch-enhanced 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/NOTICE +14 -0
- package/README.en.md +235 -0
- package/README.md +235 -0
- package/cordis.patch.yml +11 -0
- package/docs/design.zh-CN.md +140 -0
- package/docs/security.zh-CN.md +82 -0
- package/examples/coexist.cordis.yml +10 -0
- package/examples/drop-in.cordis.yml +16 -0
- package/examples/manual.cordis.patch.yml +13 -0
- package/lib/client.js +341 -0
- package/lib/index.js +645 -0
- package/lib/types/address-policy.d.ts +24 -0
- package/lib/types/client/AllowlistCard.d.ts +16 -0
- package/lib/types/client/index.d.ts +13 -0
- package/lib/types/client/locales.d.ts +26 -0
- package/lib/types/index.d.ts +44 -0
- package/lib/types/policy.d.ts +16 -0
- package/lib/types/provider.d.ts +24 -0
- package/lib/types/resolver.d.ts +31 -0
- package/package.json +122 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-web-fetch-enhanced contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/NOTICE
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
dsh-web-fetch-enhanced
|
|
2
|
+
|
|
3
|
+
The HTTP URL policy, DNS64 discovery, address-pinned Undici transport, redirect
|
|
4
|
+
handling, bounded body reader, and error classification in this project are
|
|
5
|
+
based in part on @deepseek-ai/dsh-web-fetch-http from DeepSeek Harness:
|
|
6
|
+
|
|
7
|
+
https://github.com/deepseek-ai/deepseek-harness
|
|
8
|
+
|
|
9
|
+
DeepSeek Harness is distributed under the MIT License.
|
|
10
|
+
Copyright (c) 2026 DeepSeek.
|
|
11
|
+
|
|
12
|
+
This project modifies that design to support explicit operator-configured CIDR
|
|
13
|
+
and hostname exceptions for non-public destinations. The project is not an
|
|
14
|
+
official DeepSeek package unless explicitly stated by its publisher.
|
package/README.en.md
ADDED
|
@@ -0,0 +1,235 @@
|
|
|
1
|
+
# dsh-web-fetch-enhanced
|
|
2
|
+
|
|
3
|
+
[中文](README.md) · [English](README.en.md)
|
|
4
|
+
|
|
5
|
+
> Safe, operator-controlled web fetching for DeepSeek Harness. It solves Clash / Mihomo fake-IP and trusted private-network access without changing how agents use `web_fetch`.
|
|
6
|
+
|
|
7
|
+
## What does it solve?
|
|
8
|
+
|
|
9
|
+
The native DeepSeek Harness HTTP provider rejects non-public addresses by default. That is an important SSRF boundary, but it can also block trusted targets in environments where:
|
|
10
|
+
|
|
11
|
+
- Clash, Mihomo, or another transparent proxy resolves public domains into a fake-IP range such as `198.18.0.0/15`;
|
|
12
|
+
- an agent must read a trusted intranet documentation site or self-hosted knowledge base; or
|
|
13
|
+
- a local proxy needs to take over the connection after Harness performs its address check.
|
|
14
|
+
|
|
15
|
+
This plugin adds two explicit exception layers:
|
|
16
|
+
|
|
17
|
+
1. a **CIDR allowlist** for non-public IPv4 and IPv6 ranges; and
|
|
18
|
+
2. an optional **hostname allowlist** that limits which hosts may use those CIDR exceptions.
|
|
19
|
+
|
|
20
|
+
With an empty CIDR allowlist, the security boundary remains equivalent to the native public-only provider.
|
|
21
|
+
|
|
22
|
+
## Highlights
|
|
23
|
+
|
|
24
|
+
- **No new model tool** — agents continue to call the existing `web_fetch` tool;
|
|
25
|
+
- **Live Web settings** — edit the allowlist in DSH Web and use it on the next fetch;
|
|
26
|
+
- **Deny non-public destinations by default** — only explicit CIDR matches become exceptions;
|
|
27
|
+
- **Optional hostname second factor** — restrict an allowed range to exact names or `*.example.com` rules;
|
|
28
|
+
- **Whole-answer DNS validation and address pinning** — every DNS answer must pass, and transport uses only validated addresses;
|
|
29
|
+
- **Redirect revalidation** — same-origin redirects are resolved, validated, and pinned again at every hop;
|
|
30
|
+
- **Anonymous bounded GET requests** — no cookies, Authorization header, or URL credentials;
|
|
31
|
+
- **Resource limits** — URL length, response bytes, decoded characters, redirects, and time are bounded;
|
|
32
|
+
- **IPv4 and IPv6 coverage** — including IPv4-mapped IPv6 and active DNS64 / NAT64 checks.
|
|
33
|
+
|
|
34
|
+
## Quick start
|
|
35
|
+
|
|
36
|
+
### 1. Install into the Web Profile
|
|
37
|
+
|
|
38
|
+
Use the DSH plugin manager. The package contributes its published `cordis.patch.yml` as a Profile patch layer:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
dsh plugin --profile web add dsh-web-fetch-enhanced
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
If the Web Profile is already running, restart its Host process using your deployment's normal method so the new Host plugin and Client face are loaded.
|
|
45
|
+
|
|
46
|
+
For local source development, use an absolute link:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
dsh plugin --profile web add link:/absolute/path/to/dsh-web-fetch-enhanced
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
### 2. Configure the allowlist
|
|
53
|
+
|
|
54
|
+
Open DSH Web and go to:
|
|
55
|
+
|
|
56
|
+
**Settings → Plugins → Configurable plugins → Enhanced web fetch allowlist**
|
|
57
|
+
|
|
58
|
+
Expand the card and enter one network per line under **Allowed CIDRs**. For a typical Clash / Mihomo fake-IP setup:
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
198.18.0.0/15
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
To let only selected sites use this exception, add hostnames as a second factor:
|
|
65
|
+
|
|
66
|
+
```text
|
|
67
|
+
api.example.com
|
|
68
|
+
*.docs.example.com
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Choose **Save**. The next `web_fetch` uses the new policy; no Profile restart is required.
|
|
72
|
+
|
|
73
|
+
### 3. Keep using `web_fetch` normally
|
|
74
|
+
|
|
75
|
+
The agent-facing workflow does not change. Ask naturally, for example:
|
|
76
|
+
|
|
77
|
+
> Read https://docs.example.com/guide and summarize the deployment steps.
|
|
78
|
+
|
|
79
|
+
The agent calls `web_fetch` as usual. This plugin performs address resolution, policy checks, and pinned transport underneath it.
|
|
80
|
+
|
|
81
|
+
## Common setups
|
|
82
|
+
|
|
83
|
+
### Clash / Mihomo fake-IP
|
|
84
|
+
|
|
85
|
+
When the entire fake-IP range is controlled by a trusted proxy:
|
|
86
|
+
|
|
87
|
+
| Setting | Value |
|
|
88
|
+
| --- | --- |
|
|
89
|
+
| Allowed CIDRs | `198.18.0.0/15` |
|
|
90
|
+
| Allowed hostnames | Empty |
|
|
91
|
+
|
|
92
|
+
An empty hostname list means any hostname may use an allowed CIDR exception. Public addresses remain available normally.
|
|
93
|
+
|
|
94
|
+
### Restrict fake-IP access to selected hosts
|
|
95
|
+
|
|
96
|
+
| Setting | Value |
|
|
97
|
+
| --- | --- |
|
|
98
|
+
| Allowed CIDRs | `198.18.0.0/15` |
|
|
99
|
+
| Allowed hostnames | `api.example.com`, `*.docs.example.com` |
|
|
100
|
+
|
|
101
|
+
This is the safer choice when you do not fully control the proxy's routing rules.
|
|
102
|
+
|
|
103
|
+
### Access a trusted intranet site
|
|
104
|
+
|
|
105
|
+
```text
|
|
106
|
+
# Allowed CIDRs
|
|
107
|
+
10.20.0.0/16
|
|
108
|
+
|
|
109
|
+
# Allowed hostnames
|
|
110
|
+
wiki.corp.example
|
|
111
|
+
*.docs.corp.example
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Allow only the smallest range you actually need. Do not add all RFC 1918 space for convenience.
|
|
115
|
+
|
|
116
|
+
## Allowlist syntax
|
|
117
|
+
|
|
118
|
+
### CIDRs
|
|
119
|
+
|
|
120
|
+
- Enter one standard IPv4 or IPv6 CIDR per line.
|
|
121
|
+
- IPv4 uses four-part decimal network addresses, such as `192.168.1.0/24`.
|
|
122
|
+
- IPv6 zone IDs such as `%eth0` are not accepted.
|
|
123
|
+
- Use the network base address, not an address with host bits set.
|
|
124
|
+
- Blank lines are ignored. Duplicate entries are reported by the settings card and block saving.
|
|
125
|
+
|
|
126
|
+
### Hostnames
|
|
127
|
+
|
|
128
|
+
- Exact rule: `api.example.com`.
|
|
129
|
+
- Left-most wildcard rule: `*.example.com`.
|
|
130
|
+
- `*.example.com` matches subdomains but **does not match** `example.com` itself.
|
|
131
|
+
- Do not include a scheme, path, or port. Values such as `https://example.com`, `example.com/path`, and `example.com:8080` are invalid.
|
|
132
|
+
- Hostname rules apply only to non-public CIDR exceptions. They do not restrict destinations that were already public.
|
|
133
|
+
|
|
134
|
+
### How the two layers combine
|
|
135
|
+
|
|
136
|
+
| Destination | CIDR match | Hostname allowlist | Result |
|
|
137
|
+
| --- | --- | --- | --- |
|
|
138
|
+
| Public address | Not required | Not required | Allow |
|
|
139
|
+
| Non-public address | No | Any | Deny |
|
|
140
|
+
| Non-public address | Yes | Not configured | Allow |
|
|
141
|
+
| Non-public address | Yes | Match | Allow |
|
|
142
|
+
| Non-public address | Yes | No match | Deny |
|
|
143
|
+
|
|
144
|
+
## Settings actions
|
|
145
|
+
|
|
146
|
+
- **Save** writes the current draft to the user settings layer.
|
|
147
|
+
- **Discard** drops unsaved edits and restores the effective values.
|
|
148
|
+
- **Reset to Profile** stages removal of the user overrides for `allowCidrs` and `allowHostnames`, restoring Profile inheritance after you choose **Save**.
|
|
149
|
+
- A **read-only** card means the current connection cannot persist Host Profile settings. Normally, open DSH Web from a loopback address on the Host.
|
|
150
|
+
|
|
151
|
+
Saving explicit empty lists is different from resetting: empty lists override the Profile with nothing, while reset restores inheritance.
|
|
152
|
+
|
|
153
|
+
## Security guidance
|
|
154
|
+
|
|
155
|
+
> **An allowlist expands the network locations an agent can request. Allow only the smallest destinations you understand and trust.**
|
|
156
|
+
|
|
157
|
+
Do not allow broad or sensitive targets such as:
|
|
158
|
+
|
|
159
|
+
- `0.0.0.0/0` or `::/0`;
|
|
160
|
+
- cloud metadata endpoints such as `169.254.169.254/32`;
|
|
161
|
+
- all of `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16` unless genuinely required; or
|
|
162
|
+
- ranges jointly controlled by untrusted proxy, DNS, or tenant infrastructure.
|
|
163
|
+
|
|
164
|
+
The Host still performs complete CIDR, hostname, DNS, and provider validation after a Web edit. Browser validation is never the security boundary. See the [security design](docs/security.zh-CN.md) for the threat model and launch checklist.
|
|
165
|
+
|
|
166
|
+
## Advanced configuration
|
|
167
|
+
|
|
168
|
+
Most users only need the two allowlists exposed in DSH Web. Manage other values in the Profile composition:
|
|
169
|
+
|
|
170
|
+
| Field | Default | Purpose |
|
|
171
|
+
| --- | --- | --- |
|
|
172
|
+
| `providerId` | `http-enhanced` | Fetch provider ID registered in `ctx.web` |
|
|
173
|
+
| `allowCidrs` | `[]` | Non-public IPv4 / IPv6 CIDR exceptions |
|
|
174
|
+
| `allowHostnames` | `[]` | Optional exact hosts or left-most wildcard rules |
|
|
175
|
+
| `maxResponseBytes` | `5,000,000` | Maximum response bytes read |
|
|
176
|
+
| `maxBodyChars` | `100,000` | Maximum decoded characters |
|
|
177
|
+
| `timeoutMs` | `30,000` | Fetch timeout in milliseconds |
|
|
178
|
+
| `maxRedirects` | `5` | Maximum same-origin redirect hops; `0` disables following |
|
|
179
|
+
| `userAgent` | `dsh-web-fetch-enhanced/0.1.0` | User-Agent sent with each request |
|
|
180
|
+
|
|
181
|
+
The default bundle uses the separate `http-enhanced` provider ID and leaves the native `http` provider installed. For manual composition or drop-in replacement, see:
|
|
182
|
+
|
|
183
|
+
- [coexisting provider example](examples/coexist.cordis.yml)
|
|
184
|
+
- [drop-in replacement example](examples/drop-in.cordis.yml)
|
|
185
|
+
- [manual composition patch](examples/manual.cordis.patch.yml)
|
|
186
|
+
- [architecture and provider selection](docs/design.zh-CN.md)
|
|
187
|
+
|
|
188
|
+
> Never register the native provider and this plugin under the same ID. The Host fails with `WEB_DUPLICATE_PROVIDER`; it does not use last-wins replacement.
|
|
189
|
+
|
|
190
|
+
## FAQ
|
|
191
|
+
|
|
192
|
+
<details>
|
|
193
|
+
<summary><strong>Why does enabling Clash / Mihomo make an ordinary public site look non-public?</strong></summary>
|
|
194
|
+
|
|
195
|
+
Fake-IP mode resolves the domain into a reserved range such as `198.18.0.0/15`, then takes over routing. The native provider checks the answer before the proxy can take over. Add the actual fake-IP range used by your proxy to Allowed CIDRs.
|
|
196
|
+
</details>
|
|
197
|
+
|
|
198
|
+
<details>
|
|
199
|
+
<summary><strong>Do allowlist changes require a restart?</strong></summary>
|
|
200
|
+
|
|
201
|
+
No. After a successful Web settings save, the next `web_fetch` uses the new policy. Installing, removing, or upgrading the plugin itself may require a Host restart depending on your deployment.
|
|
202
|
+
</details>
|
|
203
|
+
|
|
204
|
+
<details>
|
|
205
|
+
<summary><strong>Why is the settings card read-only?</strong></summary>
|
|
206
|
+
|
|
207
|
+
The current browser connection cannot persist Host settings. Open DSH Web from `127.0.0.1` or `localhost` on the Host and verify that the Profile settings service is writable.
|
|
208
|
+
</details>
|
|
209
|
+
|
|
210
|
+
<details>
|
|
211
|
+
<summary><strong>Why does `*.example.com` not allow `example.com`?</strong></summary>
|
|
212
|
+
|
|
213
|
+
Wildcard rules match subdomains only. Add both `example.com` and `*.example.com` when you need both.
|
|
214
|
+
</details>
|
|
215
|
+
|
|
216
|
+
<details>
|
|
217
|
+
<summary><strong>Why is a cross-site redirect blocked?</strong></summary>
|
|
218
|
+
|
|
219
|
+
The provider follows same-origin redirects only. The agent can issue a separate fetch for the new URL, preventing trust in one validated origin from being inherited by another.
|
|
220
|
+
</details>
|
|
221
|
+
|
|
222
|
+
## Development
|
|
223
|
+
|
|
224
|
+
For local development:
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
pnpm install
|
|
228
|
+
pnpm run check
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
`pnpm run check` runs type checking, lint, builds, tests, and package publication checks. The Host bundle is emitted as `lib/index.js`; the browser Client bundle is emitted as `lib/client.js`.
|
|
232
|
+
|
|
233
|
+
## License and attribution
|
|
234
|
+
|
|
235
|
+
This project is licensed under the [MIT License](LICENSE). Its network security model and parts of the implementation are based on DeepSeek Harness `@deepseek-ai/dsh-web-fetch-http`; see [NOTICE](NOTICE). This project is not an official DeepSeek package unless explicitly stated by its publisher.
|
package/README.md
ADDED
|
@@ -0,0 +1,235 @@
|
|
|
1
|
+
# dsh-web-fetch-enhanced
|
|
2
|
+
|
|
3
|
+
[中文](README.md) · [English](README.en.md)
|
|
4
|
+
|
|
5
|
+
> 为 DeepSeek Harness 提供安全、可控的增强型网页抓取:解决 Clash / Mihomo Fake-IP 与受信任内网目标被原生公网地址检查拦截的问题,同时保持模型侧 `web_fetch` 用法不变。
|
|
6
|
+
|
|
7
|
+
## 它解决什么问题?
|
|
8
|
+
|
|
9
|
+
DeepSeek Harness 原生 HTTP provider 默认拒绝所有非公网地址,这是重要的 SSRF 安全边界。但在以下场景中,目标明明可信,也可能被提前拦截:
|
|
10
|
+
|
|
11
|
+
- Clash、Mihomo 等透明代理的 Fake-IP 模式把公网域名解析到 `198.18.0.0/15`;
|
|
12
|
+
- Agent 需要读取企业内网文档站、自建知识库或其他受信任的私有网络服务;
|
|
13
|
+
- 本地代理在接管连接前,需要先通过 Harness 的地址检查。
|
|
14
|
+
|
|
15
|
+
本插件允许管理员显式配置两层例外:
|
|
16
|
+
|
|
17
|
+
1. **CIDR 白名单**:哪些非公网 IPv4 / IPv6 网段可以访问;
|
|
18
|
+
2. **域名白名单(可选)**:哪些域名可以使用上述 CIDR 例外。
|
|
19
|
+
|
|
20
|
+
没有配置任何 CIDR 时,行为与原生 provider 的“仅公网”默认值一致。
|
|
21
|
+
|
|
22
|
+
## 主要特性
|
|
23
|
+
|
|
24
|
+
- **与原生工具完全兼容**:不增加新模型工具,Agent 继续调用 `web_fetch`;
|
|
25
|
+
- **网页端即时配置**:在 DSH Web 设置中编辑白名单,保存后下一次抓取立即生效;
|
|
26
|
+
- **默认拒绝非公网目标**:只有明确命中的 CIDR 例外才会放行;
|
|
27
|
+
- **可选域名第二因子**:将可访问网段进一步限制到精确域名或 `*.example.com`;
|
|
28
|
+
- **DNS 全答案校验与连接固定**:所有解析结果都必须合规,连接只使用已验证的地址;
|
|
29
|
+
- **重定向逐跳复核**:仅跟随同源重定向,并在每一跳重新解析、校验和固定地址;
|
|
30
|
+
- **受限匿名请求**:只发送无 Cookie、无 Authorization、无 URL 凭据的 GET 请求;
|
|
31
|
+
- **完整资源上限**:限制 URL、响应字节、解码字符、重定向次数和超时时间;
|
|
32
|
+
- **IPv4 / IPv6 防护**:覆盖 IPv4-mapped IPv6 与活动 DNS64 / NAT64 目标检查。
|
|
33
|
+
|
|
34
|
+
## 快速开始
|
|
35
|
+
|
|
36
|
+
### 1. 安装到 Web Profile
|
|
37
|
+
|
|
38
|
+
推荐使用 DSH 的插件管理命令。安装包会把随包发布的 `cordis.patch.yml` 作为 Profile patch 层应用:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
dsh plugin --profile web add dsh-web-fetch-enhanced
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
如果 Web Profile 已经在运行,请按你的部署方式重启对应的 Host 进程,使新插件和 Client face 完成装载。
|
|
45
|
+
|
|
46
|
+
本地源码开发时可以使用绝对路径:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
dsh plugin --profile web add link:/absolute/path/to/dsh-web-fetch-enhanced
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
### 2. 配置白名单
|
|
53
|
+
|
|
54
|
+
打开 DSH Web,进入:
|
|
55
|
+
|
|
56
|
+
**设置 → 插件 → 可配置插件 → WebFetch白名单**
|
|
57
|
+
|
|
58
|
+
展开卡片,在“允许的 CIDR”中每行填写一个网段。例如 Clash / Mihomo 的常见 Fake-IP 配置:
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
198.18.0.0/15
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
如果希望只有指定网站可以使用这个例外,再填写“允许的域名”:
|
|
65
|
+
|
|
66
|
+
```text
|
|
67
|
+
api.example.com
|
|
68
|
+
*.docs.example.com
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
点击“保存”。无需重启 Profile,下一次 `web_fetch` 就会使用新规则。
|
|
72
|
+
|
|
73
|
+
### 3. 正常使用 `web_fetch`
|
|
74
|
+
|
|
75
|
+
插件不会改变 Agent 的使用方式。你仍然可以直接用自然语言提出请求,例如:
|
|
76
|
+
|
|
77
|
+
> 读取 https://docs.example.com/guide,并总结部署步骤。
|
|
78
|
+
|
|
79
|
+
Agent 会照常调用 `web_fetch`;地址解析、白名单判断和安全传输由本插件在底层完成。
|
|
80
|
+
|
|
81
|
+
## 常见配置场景
|
|
82
|
+
|
|
83
|
+
### Clash / Mihomo Fake-IP
|
|
84
|
+
|
|
85
|
+
如果整个 Fake-IP 网段都由可信代理接管:
|
|
86
|
+
|
|
87
|
+
| 设置项 | 内容 |
|
|
88
|
+
| --- | --- |
|
|
89
|
+
| 允许的 CIDR | `198.18.0.0/15` |
|
|
90
|
+
| 允许的域名 | 留空 |
|
|
91
|
+
|
|
92
|
+
留空域名白名单表示:任意域名只要解析到已放行 CIDR,就可以使用该 CIDR 例外。公网地址仍照常访问。
|
|
93
|
+
|
|
94
|
+
### 只允许少量域名使用 Fake-IP
|
|
95
|
+
|
|
96
|
+
| 设置项 | 内容 |
|
|
97
|
+
| --- | --- |
|
|
98
|
+
| 允许的 CIDR | `198.18.0.0/15` |
|
|
99
|
+
| 允许的域名 | `api.example.com`、`*.docs.example.com` |
|
|
100
|
+
|
|
101
|
+
这是更严格的配置,适合代理规则不完全受你控制的环境。
|
|
102
|
+
|
|
103
|
+
### 访问受信任的内网站点
|
|
104
|
+
|
|
105
|
+
```text
|
|
106
|
+
# 允许的 CIDR
|
|
107
|
+
10.20.0.0/16
|
|
108
|
+
|
|
109
|
+
# 允许的域名
|
|
110
|
+
wiki.corp.example
|
|
111
|
+
*.docs.corp.example
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
请只放行实际需要的最小网段,不要为了方便加入整个 RFC1918 地址空间。
|
|
115
|
+
|
|
116
|
+
## 白名单规则
|
|
117
|
+
|
|
118
|
+
### CIDR
|
|
119
|
+
|
|
120
|
+
- 每行一个标准 IPv4 或 IPv6 CIDR;
|
|
121
|
+
- IPv4 必须使用四段十进制网络地址,例如 `192.168.1.0/24`;
|
|
122
|
+
- IPv6 不接受 `%eth0` 之类的 Zone ID;
|
|
123
|
+
- 必须填写网络基址,不能用带主机位的地址代替网段;
|
|
124
|
+
- 空白行会被忽略,重复条目会在设置页中提示并阻止保存。
|
|
125
|
+
|
|
126
|
+
### 域名
|
|
127
|
+
|
|
128
|
+
- 精确规则:`api.example.com`;
|
|
129
|
+
- 最左侧通配规则:`*.example.com`;
|
|
130
|
+
- `*.example.com` 匹配其子域名,但**不匹配** `example.com` 本身;
|
|
131
|
+
- 不要填写协议、路径或端口,例如 `https://example.com`、`example.com/path`、`example.com:8080` 都不是合法规则;
|
|
132
|
+
- 域名白名单只约束“非公网 CIDR 例外”,不会限制原本就允许访问的公网地址。
|
|
133
|
+
|
|
134
|
+
### 两层规则如何组合?
|
|
135
|
+
|
|
136
|
+
| 目标地址 | CIDR 命中 | 已配置域名白名单且域名命中 | 结果 |
|
|
137
|
+
| --- | --- | --- | --- |
|
|
138
|
+
| 公网地址 | 不需要 | 不需要 | 允许 |
|
|
139
|
+
| 非公网地址 | 否 | 任意 | 拒绝 |
|
|
140
|
+
| 非公网地址 | 是 | 未配置域名白名单 | 允许 |
|
|
141
|
+
| 非公网地址 | 是 | 是 | 允许 |
|
|
142
|
+
| 非公网地址 | 是 | 否 | 拒绝 |
|
|
143
|
+
|
|
144
|
+
## 设置页按钮说明
|
|
145
|
+
|
|
146
|
+
- **保存**:把当前草稿写入用户设置层;
|
|
147
|
+
- **放弃修改**:丢弃尚未保存的编辑,恢复当前生效值;
|
|
148
|
+
- **重置为 Profile 配置**:把“删除 `allowCidrs` 和 `allowHostnames` 用户覆盖、重新继承 Profile composition”的操作加入草稿;仍需点击“保存”才会生效;
|
|
149
|
+
- **只读状态**:当前连接没有持久化 Host Profile 设置的权限。通常应从 Host 本机的 loopback 地址打开 Web GUI。
|
|
150
|
+
|
|
151
|
+
“保存空列表”和“重置为 Profile 配置”含义不同:前者显式覆盖为空,后者恢复继承。
|
|
152
|
+
|
|
153
|
+
## 安全提示
|
|
154
|
+
|
|
155
|
+
> **白名单会扩大 Agent 可发起 HTTP 请求的网络范围。只放行你理解并信任的最小目标。**
|
|
156
|
+
|
|
157
|
+
请勿加入以下宽泛或敏感目标:
|
|
158
|
+
|
|
159
|
+
- `0.0.0.0/0` 或 `::/0`;
|
|
160
|
+
- 云元数据地址,例如 `169.254.169.254/32`;
|
|
161
|
+
- 不必要的整个 `10.0.0.0/8`、`172.16.0.0/12` 或 `192.168.0.0/16`;
|
|
162
|
+
- 由不可信代理、DNS 或租户共同控制的网段。
|
|
163
|
+
|
|
164
|
+
即使配置了白名单,Host 仍会重新执行完整的 CIDR、域名、DNS 和 provider 身份校验;浏览器端校验不是安全边界。详细威胁模型见 [安全设计](docs/security.zh-CN.md)。
|
|
165
|
+
|
|
166
|
+
## 高级配置
|
|
167
|
+
|
|
168
|
+
通常只需要在 Web GUI 中维护两项白名单。其他参数应由 Profile composition 管理:
|
|
169
|
+
|
|
170
|
+
| 字段 | 默认值 | 说明 |
|
|
171
|
+
| --- | --- | --- |
|
|
172
|
+
| `providerId` | `http-enhanced` | 注册到 `ctx.web` 的 fetch provider ID |
|
|
173
|
+
| `allowCidrs` | `[]` | 允许作为非公网例外的 IPv4 / IPv6 CIDR |
|
|
174
|
+
| `allowHostnames` | `[]` | 可选的精确域名或最左侧通配规则 |
|
|
175
|
+
| `maxResponseBytes` | `5,000,000` | 响应正文最大读取字节数 |
|
|
176
|
+
| `maxBodyChars` | `100,000` | 解码后最大字符数 |
|
|
177
|
+
| `timeoutMs` | `30,000` | 单次抓取超时(毫秒) |
|
|
178
|
+
| `maxRedirects` | `5` | 同源重定向最大跳数;`0` 表示不跟随 |
|
|
179
|
+
| `userAgent` | `dsh-web-fetch-enhanced/0.1.0` | 每个请求使用的 User-Agent |
|
|
180
|
+
|
|
181
|
+
默认安装采用独立的 `http-enhanced` provider ID,并保留原生 `http` provider。需要手动组合或 drop-in 替换时,参考:
|
|
182
|
+
|
|
183
|
+
- [独立 provider 示例](examples/coexist.cordis.yml)
|
|
184
|
+
- [drop-in 替换示例](examples/drop-in.cordis.yml)
|
|
185
|
+
- [手动 composition patch](examples/manual.cordis.patch.yml)
|
|
186
|
+
- [架构与 provider 选择语义](docs/design.zh-CN.md)
|
|
187
|
+
|
|
188
|
+
> 不要让原生 provider 和本插件同时注册同一个 ID,否则 Host 会以 `WEB_DUPLICATE_PROVIDER` 拒绝启动,而不是执行 last-wins 覆盖。
|
|
189
|
+
|
|
190
|
+
## 常见问题
|
|
191
|
+
|
|
192
|
+
<details>
|
|
193
|
+
<summary><strong>开启 Clash / Mihomo 后,为什么普通公网网页也被判定为非公网地址?</strong></summary>
|
|
194
|
+
|
|
195
|
+
Fake-IP 模式会把域名解析到 `198.18.0.0/15` 等保留网段,再由代理接管连接。原生 provider 在代理接管前执行地址检查,因此会拒绝该结果。将代理实际使用的 Fake-IP 网段加入 CIDR 白名单即可。
|
|
196
|
+
</details>
|
|
197
|
+
|
|
198
|
+
<details>
|
|
199
|
+
<summary><strong>修改白名单后需要重启吗?</strong></summary>
|
|
200
|
+
|
|
201
|
+
不需要。Web 设置保存成功后,下一次 `web_fetch` 会立即读取新策略。只有首次安装、移除或升级插件时,才可能需要按部署方式重新启动 Host。
|
|
202
|
+
</details>
|
|
203
|
+
|
|
204
|
+
<details>
|
|
205
|
+
<summary><strong>为什么设置卡片是只读的?</strong></summary>
|
|
206
|
+
|
|
207
|
+
当前浏览器连接不能持久化 Host 设置。请确认你从 Host 本机通过 `127.0.0.1` 或 `localhost` 访问,并检查 Profile 的设置服务是否允许写入。
|
|
208
|
+
</details>
|
|
209
|
+
|
|
210
|
+
<details>
|
|
211
|
+
<summary><strong>为什么 `*.example.com` 不能访问 `example.com`?</strong></summary>
|
|
212
|
+
|
|
213
|
+
通配规则只匹配子域名。若两者都需要,请分别加入 `example.com` 和 `*.example.com`。
|
|
214
|
+
</details>
|
|
215
|
+
|
|
216
|
+
<details>
|
|
217
|
+
<summary><strong>为什么某些跨站跳转会被拒绝?</strong></summary>
|
|
218
|
+
|
|
219
|
+
插件只自动跟随同源重定向。跨源地址需要由 Agent 对新的 URL 发起一次独立抓取,这可以避免把已验证目标的信任隐式传递给另一个站点。
|
|
220
|
+
</details>
|
|
221
|
+
|
|
222
|
+
## 开发与贡献
|
|
223
|
+
|
|
224
|
+
需要二次开发时:
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
pnpm install
|
|
228
|
+
pnpm run check
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
`pnpm run check` 会执行类型检查、lint、构建、测试与发布包校验。Host bundle 输出到 `lib/index.js`,浏览器 Client bundle 输出到 `lib/client.js`。
|
|
232
|
+
|
|
233
|
+
## 许可证与来源
|
|
234
|
+
|
|
235
|
+
本项目采用 [MIT License](LICENSE)。网络安全模型与部分实现基于 DeepSeek Harness 的 `@deepseek-ai/dsh-web-fetch-http`,详见 [NOTICE](NOTICE)。本项目不是 DeepSeek 官方包,除非发布者另有明确说明。
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# Optional Profile Bundle: select the enhanced HTTP fetch provider on the Host plane.
|
|
2
|
+
# This layer is applied after @deepseek-ai/dsh-base. A patch replaces a row's
|
|
3
|
+
# entire config, so both WebRuntime selector fields are restated here.
|
|
4
|
+
- id: web
|
|
5
|
+
config:
|
|
6
|
+
searchProvider: deepseek-official
|
|
7
|
+
fetchProvider: http-enhanced
|
|
8
|
+
|
|
9
|
+
- insert:
|
|
10
|
+
- id: web-fetch-enhanced
|
|
11
|
+
name: dsh-web-fetch-enhanced
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# 设计文档
|
|
2
|
+
|
|
3
|
+
## 1. 目标
|
|
4
|
+
|
|
5
|
+
本插件只解决一个问题:在 DeepSeek Harness 的 HTTP fetch provider 中,为非公网地址过滤增加显式、可审计的 CIDR 白名单,同时维持原生 <code>web_fetch</code> 的工具名称、参数、结果与安全约束。
|
|
6
|
+
|
|
7
|
+
设计目标:
|
|
8
|
+
|
|
9
|
+
1. 空白名单与原生 public-only 策略等价;
|
|
10
|
+
2. fake-IP 等保留地址路由可以按 CIDR 放行;
|
|
11
|
+
3. 可用域名规则作为第二因子缩小例外范围;
|
|
12
|
+
4. DNS 校验结果必须直接用于连接,避免 rebinding/TOCTOU;
|
|
13
|
+
5. provider 可与原生实现共存,也可在 composition 中做 drop-in 替换;
|
|
14
|
+
6. 不改变模型侧 <code>web_fetch</code> schema 和结果呈现。
|
|
15
|
+
|
|
16
|
+
非目标:
|
|
17
|
+
|
|
18
|
+
- 不读取或注入 cookie、Authorization、客户端证书;
|
|
19
|
+
- 不允许模型设置任意 header、HTTP method 或代理地址;
|
|
20
|
+
- 不加载图片、脚本、CSS 等子资源;
|
|
21
|
+
- 不支持跨源自动重定向;
|
|
22
|
+
- 不把白名单配置暴露成模型参数;
|
|
23
|
+
- 不提供 provider fallback 或优先级链;
|
|
24
|
+
- 不修改 agent preset 权限边界。
|
|
25
|
+
|
|
26
|
+
## 2. Cordis 与 DSH 接入面
|
|
27
|
+
|
|
28
|
+
DeepSeek Harness 的 <code>@deepseek-ai/dsh-web</code> 提供 <code>ctx.web</code> 服务和 fetch provider 注册表。模型侧 <code>@deepseek-ai/dsh-tool-web</code> 只调用 <code>ctx.web.fetch()</code>,因此替换 provider 不会改变工具接口。
|
|
29
|
+
|
|
30
|
+
~~~mermaid
|
|
31
|
+
flowchart LR
|
|
32
|
+
M[Model] --> T[web_fetch / dsh-tool-web]
|
|
33
|
+
T --> W[ctx.web / dsh-web]
|
|
34
|
+
W -->|fetchProvider: http-enhanced| P[dsh-web-fetch-enhanced]
|
|
35
|
+
P --> D[resolve all DNS answers]
|
|
36
|
+
D --> A[address and hostname policy]
|
|
37
|
+
A --> U[pinned Undici dispatcher]
|
|
38
|
+
U --> R[bounded text response]
|
|
39
|
+
~~~
|
|
40
|
+
|
|
41
|
+
本插件是 Host 侧 provider:它消费已有的 <code>web</code> 服务并注册一个 fetch provider,但不发布新的 Cordis 服务。因此它属于 Host composition,不属于 agent preset。模型工具仍由 preset 中原有的 <code>tool-web</code> 行提供。
|
|
42
|
+
|
|
43
|
+
入口采用 namespace plugin 约定,只命名导出 <code>name</code>、<code>inject</code>、<code>Config</code>、安全的 <code>createProvider</code> 和 <code>apply</code>,没有 default export,也不公开低级 pinned transport/custom resolver helper。与 DeepSeek Harness 内部插件一致,源码使用显式 <code>.ts</code> 相对导入,<code>cordis.source.patch.yml</code> 在开发时直接加载源码;发布构建由 TypeScript 生成 <code>lib/types</code>,再由 tsdown 分别输出 Host ESM <code>lib/index.js</code> 和浏览器 Client ESM <code>lib/client.js</code>。Client face 以 <code>web-fetch-enhanced</code> 命名空间键向 <code>settings.plugin.item</code> 注册白名单卡片;Host face 通过 <code>installSettingsSection</code> 将 Profile composition 作为 base layer,并让 provider 在每次请求时读取最新 resolved section。随包发布的 <code>cordis.patch.yml</code> 供操作者显式合并到 Profile composition。
|
|
44
|
+
|
|
45
|
+
## 3. Provider 选择
|
|
46
|
+
|
|
47
|
+
<code>ctx.web</code> 的选择是 ID 驱动而不是注册顺序驱动:
|
|
48
|
+
|
|
49
|
+
- 配置 <code>fetchProvider: http-enhanced</code> 时只选择本插件默认 ID;
|
|
50
|
+
- 保留原生 <code>http</code> provider 不会产生歧义,因为选择已显式固定;
|
|
51
|
+
- 未配置 ID 且存在多个可用 provider 时会报 <code>WEB_PROVIDER_AMBIGUOUS</code>;
|
|
52
|
+
- 两个 provider 注册相同 ID 会立即报 <code>WEB_DUPLICATE_PROVIDER</code>。
|
|
53
|
+
|
|
54
|
+
所以“覆盖原生”有两种明确模式:
|
|
55
|
+
|
|
56
|
+
1. 推荐模式:本插件使用 <code>http-enhanced</code>,修改现有 web 行的 <code>fetchProvider</code>;
|
|
57
|
+
2. drop-in 模式:禁用原生 provider,本插件配置 <code>providerId: http</code>。
|
|
58
|
+
|
|
59
|
+
不存在 last-wins 或自动 fallback。
|
|
60
|
+
|
|
61
|
+
## 4. 请求数据流
|
|
62
|
+
|
|
63
|
+
一次 fetch 的顺序如下:
|
|
64
|
+
|
|
65
|
+
1. 检查 URL 长度不超过 2048;
|
|
66
|
+
2. 由 WHATWG URL 解析,仅接受 HTTP(S),拒绝 URL 内嵌凭据;
|
|
67
|
+
3. 对域名执行一次 <code>lookup(all: true, order: verbatim)</code>;IP literal 直接进入相同策略;
|
|
68
|
+
4. 验证每个答案的 family 与文本格式;
|
|
69
|
+
5. 逐个判断公网单播或白名单例外;任何一个失败就拒绝整个答案集;
|
|
70
|
+
6. 若存在 IPv6,解析 <code>ipv4only.arpa</code> 发现活动 DNS64 前缀,并检查嵌入的 IPv4;
|
|
71
|
+
7. 为当前请求创建私有 Undici Agent,其 lookup 回调只返回已验证答案;
|
|
72
|
+
8. 保持原始 URL hostname,用于 Host header 与 TLS SNI;
|
|
73
|
+
9. 手动处理响应;同源重定向回到步骤 1,每一跳重新解析和固定;
|
|
74
|
+
10. 验证 MIME 与 charset,按字节和字符上限读取;
|
|
75
|
+
11. 返回 <code>WebFetchResult</code>,非 2xx 状态不转为异常。
|
|
76
|
+
|
|
77
|
+
## 5. 白名单语义
|
|
78
|
+
|
|
79
|
+
### 5.1 地址条件
|
|
80
|
+
|
|
81
|
+
<code>allowCidrs</code> 只对非公网地址生效。公网单播地址始终允许,不会因为未命中白名单而被拒绝。
|
|
82
|
+
|
|
83
|
+
非公网地址必须命中至少一个 CIDR。IPv4-mapped 和废弃的 IPv4-compatible IPv6 会先转换为嵌入的 IPv4,再执行分类与 CIDR 匹配;配置中拒绝 IPv4-mapped CIDR,要求直接写对应 IPv4 CIDR。IPv4 CIDR 只接受四段十进制,IPv6 拒绝 zone ID,且所有 CIDR 必须填写规范的网络基址,避免审计语义歧义。
|
|
84
|
+
|
|
85
|
+
### 5.2 域名第二因子
|
|
86
|
+
|
|
87
|
+
<code>allowHostnames</code> 为空时,命中 CIDR 就足以使用例外。非空时,非公网地址还必须命中一个域名规则:
|
|
88
|
+
|
|
89
|
+
- <code>api.example.com</code> 为精确匹配;
|
|
90
|
+
- <code>*.example.com</code> 匹配其子域名,不匹配根域名;
|
|
91
|
+
- 只允许最左侧完整标签通配符;
|
|
92
|
+
- 规则不得包含端口、反斜杠、路径、查询、片段或凭据分隔符;
|
|
93
|
+
- 域名按 URL 规则规范化、转小写并移除末尾点。
|
|
94
|
+
|
|
95
|
+
域名规则不是 DNS 信任替代物;它只是 operator 配置的第二个静态条件。
|
|
96
|
+
|
|
97
|
+
### 5.3 NAT64
|
|
98
|
+
|
|
99
|
+
活动 DNS64 前缀通过 RFC 7050 保留名发现。若 IPv6 答案映射到一个非公网 IPv4,则嵌入的 IPv4 也必须满足同一白名单和域名条件。一个地址同时匹配多个重叠 Pref64 时会验证所有候选,而不是依赖 DNS 返回顺序;这避免内核最长前缀路由与策略选择不一致。
|
|
100
|
+
|
|
101
|
+
## 6. 配置与生命周期
|
|
102
|
+
|
|
103
|
+
Cordis Config schema 提供所有默认值;<code>createProvider()</code> 也在直接调用时应用同一默认值。启动阶段校验:
|
|
104
|
+
|
|
105
|
+
- provider ID 字符集和长度;
|
|
106
|
+
- CIDR 与域名规则语法;
|
|
107
|
+
- 字节、字符与超时必须为正有限数;
|
|
108
|
+
- 超时不能超过 Node timer 上限;
|
|
109
|
+
- redirect 上限必须为非负整数。
|
|
110
|
+
|
|
111
|
+
<code>ctx.web.registerFetchProvider()</code> 返回的注册由 web 服务绑定到当前 Cordis Fiber,插件停止、更新或 HMR 时会自动注销,没有进程级残留。
|
|
112
|
+
|
|
113
|
+
## 7. 错误契约
|
|
114
|
+
|
|
115
|
+
插件沿用 DSH 的 <code>WebError</code>:
|
|
116
|
+
|
|
117
|
+
| code | 含义 |
|
|
118
|
+
| --- | --- |
|
|
119
|
+
| <code>WEB_INVALID_URL</code> | URL 无效、scheme 不支持或超长 |
|
|
120
|
+
| <code>WEB_BLOCKED_URL</code> | 凭据、非公网且未白名单、NAT64 目标未白名单 |
|
|
121
|
+
| <code>WEB_REDIRECT_BLOCKED</code> | 跨源、缺少预算或不安全 redirect |
|
|
122
|
+
| <code>WEB_FETCH_TOO_LARGE</code> | 声明响应长度超过上限 |
|
|
123
|
+
| <code>WEB_FETCH_TIMEOUT</code> | provider 自身资源超时 |
|
|
124
|
+
| <code>WEB_UNSUPPORTED_CONTENT_TYPE</code> | 二进制 MIME 或不支持的 charset |
|
|
125
|
+
| <code>WEB_ABORTED</code> | 调用方或外层工具策略取消 |
|
|
126
|
+
| <code>WEB_PROVIDER_ERROR</code> | DNS、连接、TLS、流读取等 transport 故障 |
|
|
127
|
+
|
|
128
|
+
模型侧工具的 timeout policy 可能把外层截止时间呈现为 <code>TOOL_TIMEOUT</code>;这是 DSH 已有的两层超时设计。
|
|
129
|
+
|
|
130
|
+
## 8. 兼容策略
|
|
131
|
+
|
|
132
|
+
插件只依赖 <code>@deepseek-ai/dsh-web</code> 和 <code>@deepseek-ai/dsh-timeout</code> 的公开包根 API,不依赖 DSH 包的 <code>src/*</code> 深路径。安全 transport 在本包内维护,相关行为通过契约测试固定。
|
|
133
|
+
|
|
134
|
+
版本升级时重点回归:
|
|
135
|
+
|
|
136
|
+
- <code>WebFetchProvider</code> 与 <code>WebFetchResult</code> 类型;
|
|
137
|
+
- <code>WebError</code> 构造参数;
|
|
138
|
+
- deadline/timeout 分类;
|
|
139
|
+
- Cordis namespace plugin 的无 default export 约束;
|
|
140
|
+
- DSH 原生 provider 的安全策略变化。
|