@xneog/dsh-web-fetch-http 0.1.0 → 0.1.3-alpha.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.i18n.yaml CHANGED
@@ -2,5 +2,5 @@
2
2
  # side as of the last confirmed-consistent state. Both languages carry equal authority;
3
3
  # after editing either side, bring the other along and re-record with:
4
4
  # pnpm run verify-translation-pairing --write packages/web/web-fetch-http/README.md
5
- README.md: 5589a8e8605a64ae9ef5f6d9978a9b63331d5b0d
6
- README.zh.md: e9cf98feb9065947af1321c87562a2ae9ac21315
5
+ README.md: 44aad8a83078587057230927e5cb69ba715a0934
6
+ README.zh.md: e70be385f6180175f2b51ef269968c235e9b31cd
package/README.md CHANGED
@@ -1,44 +1,125 @@
1
+ ---
2
+ description: "The anonymous public HTTP(S) fetch backend for ctx.web: how deployments mount bounded, safe URL retrieval with same-origin redirects and text-only decoding."
3
+ kind: "package-reference"
4
+ ---
5
+
1
6
  # @xneog/dsh-web-fetch-http
2
7
 
3
8
  English | [中文](README.zh.md)
4
9
 
5
- An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content.
10
+ ## Summary
11
+
12
+ With `dsh-web-fetch-http`, the harness can fetch public HTTP(S) pages through the web service (`ctx.web`) and get their status code plus bounded, decoded content without sending credentials. Choose it when a composition needs safe retrieval with URL validation, public-address resolution, connection pinning, same-origin redirects, byte and character caps, and an explicit product `User-Agent`. It returns non-2xx responses as results rather than errors, and rejects non-public destinations, binary data, and unsupported content types. The model-facing `web_fetch` tool lives in `dsh-tool-web`, which renders this provider's bodies.
13
+
14
+ ## Table of Contents
6
15
 
7
- This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`).
16
+ - [Use this package](#use-this-package)
17
+ - [Understand the implementation](#understand-the-implementation)
18
+ - [Further Exploration](#further-exploration)
19
+ - [Model Experience](#model-experience)
20
+ - [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
21
+ - [Dev Note](#dev-note)
8
22
 
9
- ## Responsibility split
23
+ -----
10
24
 
11
- The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@xneog/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource.
25
+ <a id="use-this-package"></a>
26
+ ## Use this package
12
27
 
13
- The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers and misconfigured deployments, not the model-facing tool-call budget. [`dsh-tool-call-timeout-policy`](../../guard/timeout-policy/README.md) owns the `web_fetch` tool-call budget by arming `exec.signal`.
28
+ Mount the provider in a composition that already loads the web service; it registers as the `http` fetch provider, so `ctx.web.fetch()` resolves it automatically when it is the only usable fetch backend or pin it with `fetchProvider: http`.
14
29
 
15
- A shipping web-tool deployment sets the provider backstop above the tool budget, so model calls normally return `TOOL_TIMEOUT`. If the outer deadline reaches the provider first, the provider reports `WEB_ABORTED` and the outer policy replaces it with `TOOL_TIMEOUT`. `WEB_FETCH_TIMEOUT` therefore identifies a direct service caller whose provider budget elapsed.
30
+ ### When to choose it
16
31
 
17
- ## Transport hygiene
32
+ Choose this backend when a deployment must fetch public pages with bounded output and safe transport: no credentials are sent, every resolved address must be public, each connection is pinned to the validated answer set, redirects cannot escape the origin, and every response is capped.
18
33
 
19
- - Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`).
20
- - Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap.
21
- - Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read.
22
- - Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch).
23
- - Sends an explicit product `User-Agent`, never a browser disguise.
24
- - Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`.
34
+ ### Minimal configuration
25
35
 
26
- ## Config
36
+ Load the web service and the provider; configurable limits have safe defaults and validate at plugin construction, so an invalid value fails loudly instead of building a provider with nonsensical caps. The URL security limit is fixed at 2,048 characters.
27
37
 
28
- | Key | Default | Meaning |
38
+ ```yaml
39
+ - name: '@xneog/dsh-web'
40
+ - name: '@xneog/dsh-web-fetch-http'
41
+ ```
42
+
43
+ | Field | Default | Meaning |
29
44
  |---|---|---|
30
- | `maxUrlLength` | `2048` | Maximum accepted request URL length. |
31
- | `maxResponseBytes` | `5_000_000` | Maximum response body size in bytes. |
32
- | `maxBodyChars` | `100_000` | Maximum decoded body length in characters. |
33
- | `timeoutMs` | `30_000` | Fetch timeout within Node's timer range — a resource backstop for direct `ctx.web.fetch()` callers, not the model-facing tool-call budget (that is `dsh-tool-call-timeout-policy`). |
34
- | `maxRedirects` | `5` | Maximum same-origin redirect hops (`0` follows none). |
35
- | `userAgent` | `deepseek-harness/…` | `User-Agent` header. |
45
+ | `maxResponseBytes` | `5,000,000` | Maximum response body size in bytes |
46
+ | `maxBodyChars` | `100,000` | Maximum decoded body length in characters |
47
+ | `timeoutMs` | `30,000` | Fetch timeout a resource backstop, not the model-facing tool budget |
48
+ | `maxRedirects` | `5` | Maximum same-origin redirect hops (`0` follows none) |
49
+ | `userAgent` | `xneog-harness/…` | `User-Agent` header sent on every request |
50
+
51
+ The generated [configuration catalog](../../../docs/config-catalog.md#xneogdsh-web-fetch-http) is the exhaustive source for every accepted field and its JSDoc.
52
+
53
+ ### What a fetch returns
54
+
55
+ A successful call yields a `WebFetchResult`: the final URL after allowed redirects, the HTTP status code, a decoded body classified as `html` or `text`, and a `truncated` flag. A non-2xx response is a result, not an error — the status code is part of the fetched resource state; `WebError` is reserved for failures to safely retrieve or represent the resource.
56
+
57
+ ```text
58
+ const page = await ctx.web.fetch({ url: 'https://example.com' })
59
+ // page.body.kind === 'html' | 'text'; page.statusCode === 200 | 404 | ...
60
+ ```
61
+
62
+ ### Transport behavior
63
+
64
+ The provider keeps requests anonymous and bounded: it accepts only `http:` and `https:` URLs without embedded credentials and rejects URLs over 2,048 characters. It resolves each hostname once, rejects the complete result if any IPv4 or IPv6 address is not public unicast, and pins the connection to that validated set. IPv6 checks discover the active DNS64 prefix and reject translations to non-public IPv4. Each same-origin redirect repeats resolution and pinning; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product `User-Agent`.
65
+
66
+ ### Failures and recovery
67
+
68
+ Failures throw `WebError` with a machine-routable code: `WEB_INVALID_URL`, `WEB_BLOCKED_URL`, `WEB_FETCH_TOO_LARGE`, `WEB_FETCH_TIMEOUT`, `WEB_REDIRECT_BLOCKED`, `WEB_UNSUPPORTED_CONTENT_TYPE`, `WEB_ABORTED`, or `WEB_PROVIDER_ERROR`. Direct callers can route on the code; the model-facing `web_fetch` tool surfaces the failure text to the model under its own error wrapper.
69
+
70
+ -----
71
+
72
+ <a id="understand-the-implementation"></a>
73
+ ## Understand the implementation
74
+
75
+ <details>
76
+ <summary>Implementation internals — click to expand</summary>
77
+
78
+ This section explains the design decisions behind the provider; the observable behavior is fully covered in [Use this package](#use-this-package).
79
+
80
+ ### Design philosophy
81
+
82
+ The package is built on one separation and one layered timeout:
36
83
 
37
- The numeric limits are validated at plugin construction: every cap except `maxRedirects` must be a positive finite number, and `maxRedirects` must be a non-negative integer. An invalid value throws rather than silently constructing a provider with nonsensical limits.
84
+ - **Safe retrieval vs. presentation.** This provider owns URL validation, public-address enforcement, connection pinning, HTTP transport, redirect policy, caps, charset decoding, and binary rejection; `dsh-tool-web` owns HTML→markdown and truncation formatting. A non-2xx response is data, not failure.
85
+ - **Two timeout layers.** The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers; the model-facing tool-call budget belongs to `dsh-tool-call-timeout-policy`, which arms `exec.signal`. When the outer deadline fires first the provider reports `WEB_ABORTED` and the policy replaces it with `TOOL_TIMEOUT`; `WEB_FETCH_TIMEOUT` therefore identifies a direct service caller whose provider budget elapsed.
38
86
 
87
+ ### Source map
88
+
89
+ | File | Role |
90
+ |---|---|
91
+ | [`src/index.ts`](src/index.ts) | Plugin entry: config schema, limit validation, provider registration |
92
+ | [`src/provider.ts`](src/provider.ts) | The `HttpFetchProvider`: pinned transport, redirect following, capped reads, charset decoding |
93
+ | [`src/network.ts`](src/network.ts) | Public-address resolution, DNS64 discovery, and connection pinning |
94
+ | [`src/policy.ts`](src/policy.ts) | URL validation, same-origin checks, content-type classification, charset parsing |
95
+ | — | No runtime invariant companion is published; this package exposes no independent event sequence or mutable data relation beyond contracts enforced at its owning seam. |
96
+
97
+ ### Read path
98
+
99
+ A fetch validates the URL, resolves the hostname once, rejects the complete answer set when any address is not public, and pins the connection to the accepted addresses. It repeats that check for each same-origin redirect; a cross-origin redirect or non-public target fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset, and read under the byte cap; the decoded text is then truncated to the character cap.
100
+
101
+ </details>
102
+
103
+ -----
104
+
105
+ <a id="further-exploration"></a>
106
+ ## Further Exploration
107
+
108
+ Read these pages when the package-level contract is not enough. They move from the shared vocabulary to the service, the model-facing tools, and the design rationale.
109
+
110
+ - [Web subsystem](../../../docs/subsystems/web.md) — the exhaustive fetch request/result vocabulary and error codes.
111
+ - [Web package map](../README.md) — the six-package family and each role.
112
+ - [dsh-web](../web/README.md) — the web service this provider registers into.
113
+ - [dsh-tool-web](../tool-web/README.md) — the model-facing `web_fetch` tool that renders this provider's bodies.
114
+ - [Generated configuration catalog](../../../docs/config-catalog.md#xneogdsh-web-fetch-http) — every accepted config field and its source declaration.
115
+ - [Web capability seam decision](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) — why search and fetch share one provider-selection service.
116
+
117
+ -----
118
+
119
+ <a id="model-experience"></a>
39
120
  ## Model Experience
40
121
 
41
- Indirectly, through [`dsh-tool-web`](../tool-web/README.md), which places this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper and retains provider failures while redirects, headers, and transport mechanics remain hidden.
122
+ Indirectly, through `dsh-tool-web`, which renders this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper while redirects, headers, and transport limits remain hidden.
42
123
 
43
124
  #### KV Cache effect
44
125
 
@@ -46,6 +127,20 @@ No direct invalidation; the named consumer owns any request-prefix changes.
46
127
 
47
128
  ## Known Limitations and Deferred Work
48
129
 
49
- - **SSRF / private-network protection is deferred** — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
50
- - **Only textual content decodes** — html/xhtml and `text/*`-plus-JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
130
+ <a id="known-limitations-and-deferred-work"></a>
131
+
132
+
133
+ These limits define when the provider is unsafe or a poor fit. They are current package constraints.
134
+
135
+ - **Only textual content decodes** — html/xhtml and `text/*` plus JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
51
136
  - **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
137
+
138
+ <a id="dev-note"></a>
139
+ ### Dev Note
140
+
141
+ <details>
142
+ <summary>Working context for maintainers — click to expand</summary>
143
+
144
+ None.
145
+
146
+ </details>
package/README.zh.md CHANGED
@@ -1,51 +1,146 @@
1
+ ---
2
+ description: "ctx.web 的匿名公共 HTTP(S) 抓取后端:部署方如何挂载有界、安全的 URL 抓取,含同源重定向与仅文本解码。"
3
+ kind: "package-reference"
4
+ ---
5
+
1
6
  # @xneog/dsh-web-fetch-http
2
7
 
3
8
  [English](README.md) | 中文
4
9
 
5
- 一个匿名公共 HTTP(S) `WebFetchProvider`,用于 harness [web 能力 seam](../web/README.md)(`ctx.web`)。它获取具体 URL,返回状态码和长度受限的解码内容。
10
+ ## 概述
11
+
12
+ 有了 `dsh-web-fetch-http`,harness 可以通过 web 服务(`ctx.web`)抓取公共 HTTP(S) 页面,并在不发送凭据的情况下获得状态码与有界、解码后的内容。当组合需要 URL 校验、公开地址解析、连接固定、仅同源重定向、字节和字符上限及显式产品 `User-Agent` 时选择它。它把非 2xx 响应作为结果而非错误返回,并拒绝非公开目标、二进制数据与不受支持的内容类型。面向模型的 `web_fetch` 工具位于 `dsh-tool-web`,由它渲染本提供方的正文。
13
+
14
+ ## 目录
6
15
 
7
- 这是一个**实现**包:它向 `ctx.web` 注册提供方,不拥有该键,也不注册面向模型的工具。它是函数/命名空间插件(`inject: ['web']`)。
16
+ - [使用本包](#use-this-package)
17
+ - [理解实现](#understand-the-implementation)
18
+ - [进一步探索](#further-exploration)
19
+ - [模型体验](#model-experience)
20
+ - [已知限制与延期工作](#known-limitations-and-deferred-work)
21
+ - [开发备注](#dev-note)
8
22
 
9
- ## 职责拆分
23
+ -----
10
24
 
11
- 提供方拥有**安全资源获取**:URL 验证、HTTP 传输、重定向策略、资源兜底超时、中止传播、字节上限、charset 解码、内容类型分类与二进制拒绝。`@xneog/dsh-tool-web` 拥有**呈现**(HTML→markdown、截断格式)。非 2xx HTTP 响应是*结果*(状态码 + 解码主体),不是错误;`WebError` 只用于无法安全获取或表示资源的失败。
25
+ <a id="use-this-package"></a>
26
+ ## 使用本包
12
27
 
13
- 提供方的 `timeoutMs` 是直接 `ctx.web.fetch()` 调用方和配置有误的部署所用的资源兜底,不是面向模型的工具调用预算。[`dsh-tool-call-timeout-policy`](../../guard/timeout-policy/README.md) 拥有 `web_fetch` 工具调用预算,并让 `exec.signal` 在超时时触发,以强制执行该预算。
28
+ 在已加载 web 服务的组合中挂载本提供方;它以 `http` 抓取提供方身份注册,因此当它是唯一可用的抓取后端时,`ctx.web.fetch()` 会自动解析到它——也可以用 `fetchProvider: http` 固定。
14
29
 
15
- 已交付的 web 工具部署会把提供方兜底设为高于工具预算,因此模型调用通常返回 `TOOL_TIMEOUT`。如果外层截止期限先于提供方的兜底超时触发,提供方会报告 `WEB_ABORTED`,外层策略再将其替换为 `TOOL_TIMEOUT`。因此,`WEB_FETCH_TIMEOUT` 表明直接服务调用方的提供方预算已经耗尽。
30
+ ### 何时选择
16
31
 
17
- ## 传输卫生
32
+ 当部署必须以有界输出和安全传输抓取公共页面时选择此后端:不发送凭据,每个已解析地址必须是公共地址,每次连接都固定到已校验的地址集合,重定向无法逃出源站,每个响应都有上限。
18
33
 
19
- - 只接受 `http:` 和 `https:` URL;拒绝 URL 中的凭据(`WEB_BLOCKED_URL`)以及过长/格式错误的 URL(`WEB_INVALID_URL`)。
20
- - 强制执行 URL 最大长度、响应字节上限(`WEB_FETCH_TOO_LARGE`)、解码主体字符上限、超时(`WEB_FETCH_TIMEOUT`)和重定向跳数上限。
21
- - 把调用方的中止信号(`WEB_ABORTED`)传播到网络请求与流式读取。
22
- - 只跟随**同源**重定向;跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败,要求发起新的工具调用(沿用 Claude Code 的 WebFetch 模式)。
23
- - 发送显式的产品 `User-Agent`,绝不伪装成浏览器。
24
- - 不受支持的内容类型(例如二进制)以 `WEB_UNSUPPORTED_CONTENT_TYPE` 拒绝。
34
+ ### 最小配置
25
35
 
26
- ## 配置
36
+ 加载 web 服务与本提供方;可配置上限都有安全默认值,并在插件构造时验证,因此无效值会响亮地失败,而不是构造出上限荒谬的提供方。URL 安全上限固定为 2,048 个字符。
27
37
 
28
- | 配置键 | 默认值 | 含义 |
38
+ ```yaml
39
+ - name: '@xneog/dsh-web'
40
+ - name: '@xneog/dsh-web-fetch-http'
41
+ ```
42
+
43
+ | 字段 | 默认值 | 含义 |
29
44
  |---|---|---|
30
- | `maxUrlLength` | `2048` | 接受的请求 URL 最大长度。 |
31
- | `maxResponseBytes` | `5_000_000` | 响应主体最大字节数。 |
32
- | `maxBodyChars` | `100_000` | 解码主体最大字符数。 |
33
- | `timeoutMs` | `30_000` | Node 定时器范围内的抓取超时:直接 `ctx.web.fetch()` 调用方的资源兜底,而非面向模型的工具调用预算(后者属于 `dsh-tool-call-timeout-policy`)。 |
34
- | `maxRedirects` | `5` | 同源重定向最大跳数(`0` 表示完全不跟随)。 |
35
- | `userAgent` | `deepseek-harness/…` | `User-Agent` 标头。 |
45
+ | `maxResponseBytes` | `5,000,000` | 响应主体最大字节数 |
46
+ | `maxBodyChars` | `100,000` | 解码主体最大字符数 |
47
+ | `timeoutMs` | `30,000` | 抓取超时——资源兜底,不是面向模型的工具预算 |
48
+ | `maxRedirects` | `5` | 同源重定向最大跳数(`0` 表示不跟随) |
49
+ | `userAgent` | `xneog-harness/…` | 每次请求发送的 `User-Agent` 标头 |
50
+
51
+ 生成的[配置目录](../../../docs/config-catalog.zh.md#xneogdsh-web-fetch-http)是每个受支持字段及其 JSDoc 的穷尽式真源。
52
+
53
+ ### 抓取返回什么
54
+
55
+ 成功调用产生 `WebFetchResult`:允许的重定向之后的最终 URL、HTTP 状态码、分类为 `html` 或 `text` 的解码正文,以及 `truncated` 标志。非 2xx 响应是结果而非错误——状态码是被抓取资源状态的一部分;`WebError` 只用于无法安全获取或表示资源的失败。
56
+
57
+ ```text
58
+ const page = await ctx.web.fetch({ url: 'https://example.com' })
59
+ // page.body.kind === 'html' | 'text'; page.statusCode === 200 | 404 | ...
60
+ ```
61
+
62
+ ### 传输行为
63
+
64
+ 提供方保持请求匿名且有界:只接受不含内嵌凭据且不超过 2,048 个字符的 `http:` 与 `https:` URL。它只解析一次主机名;只要结果中有任何 IPv4 或 IPv6 地址不是公共单播地址,就拒绝整个结果,并把连接固定到已校验的地址集合。IPv6 检查会发现活动 DNS64 前缀,并拒绝指向非公开 IPv4 的转换地址。每次同源重定向都会重复解析与固定;跨源重定向会失败并要求重新调用。提供方还强制执行字节、字符、跳数和时间上限,拒绝不支持的内容类型,并发送显式产品 `User-Agent`。
65
+
66
+ ### 失败与恢复
67
+
68
+ 失败抛出携带可按机器路由 code 的 `WebError`:`WEB_INVALID_URL`、`WEB_BLOCKED_URL`、`WEB_FETCH_TOO_LARGE`、`WEB_FETCH_TIMEOUT`、`WEB_REDIRECT_BLOCKED`、`WEB_UNSUPPORTED_CONTENT_TYPE`、`WEB_ABORTED` 或 `WEB_PROVIDER_ERROR`。直接调用方可以按 code 路由;面向模型的 `web_fetch` 工具会在自己的错误包装层内把失败文本呈现给模型。
69
+
70
+ -----
71
+
72
+ <a id="understand-the-implementation"></a>
73
+ ## 理解实现
74
+
75
+ <details>
76
+ <summary>实现细节——点击展开</summary>
77
+
78
+ 本节解释提供方背后的设计决策;可观察行为已在[使用本包](#use-this-package)中完整说明。
79
+
80
+ ### 设计理念
81
+
82
+ 本包建立在一个分离与一个分层超时之上:
36
83
 
37
- 数值限制会在插件构造时验证:除 `maxRedirects` 外,每个上限都必须是正的有限数;`maxRedirects` 必须是非负整数。无效值会抛出异常,不会静默构造限制荒谬的提供方。
84
+ - **安全获取与呈现分离。** 本提供方拥有 URL 校验、公开地址强制规则、连接固定、HTTP 传输、重定向策略、上限、charset 解码与二进制拒绝;`dsh-tool-web` 拥有 HTML→markdown 与截断格式化。非 2xx 响应是数据,不是失败。
85
+ - **两层超时。** 提供方的 `timeoutMs` 是直接 `ctx.web.fetch()` 调用方的资源兜底;面向模型的工具调用预算属于 `dsh-tool-call-timeout-policy`,由它触发 `exec.signal`。外层截止期限先到时,提供方报告 `WEB_ABORTED`,策略再以 `TOOL_TIMEOUT` 替换;因此 `WEB_FETCH_TIMEOUT` 标识的是提供方预算耗尽的直接服务调用方。
38
86
 
87
+ ### 源码地图
88
+
89
+ | 文件 | 职责 |
90
+ |---|---|
91
+ | [`src/index.ts`](src/index.ts) | 插件入口:配置 schema、上限验证、提供方注册 |
92
+ | [`src/provider.ts`](src/provider.ts) | `HttpFetchProvider`:固定连接、重定向跟随、有界读取、charset 解码 |
93
+ | [`src/network.ts`](src/network.ts) | 公开地址解析、DNS64 发现与连接固定 |
94
+ | [`src/policy.ts`](src/policy.ts) | URL 校验、同源检查、内容类型分类、charset 解析 |
95
+ | — | 不发布运行时不变式伴生入口;上限在提供方处强制执行。 |
96
+
97
+ ### 读取路径
98
+
99
+ 抓取先校验 URL,只解析一次主机名,结果中只要有非公开地址就拒绝,并把连接固定到已接受地址。每次同源重定向都重复该检查;跨源重定向或非公开目标在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码,并在字节上限内读取;解码后的文本再截断到字符上限。
100
+
101
+ </details>
102
+
103
+ -----
104
+
105
+ <a id="further-exploration"></a>
106
+ ## 进一步探索
107
+
108
+ 当包级约定不够用时阅读以下页面。它们从共享词汇逐步进入服务、面向模型的工具与设计依据。
109
+
110
+ - [web 子系统](../../../docs/subsystems/web.zh.md)——穷尽式的抓取请求/结果词汇与错误码。
111
+ - [web 包映射](../README.zh.md)——六包家族与各角色。
112
+ - [dsh-web](../web/README.zh.md)——本提供方注册进入的 web 服务。
113
+ - [dsh-tool-web](../tool-web/README.zh.md)——渲染本提供方正文的面向模型 `web_fetch` 工具。
114
+ - [生成配置目录](../../../docs/config-catalog.zh.md#xneogdsh-web-fetch-http)——每个受支持配置字段及其源声明。
115
+ - [web 能力 seam 决策](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md)——搜索与抓取为何共用一项提供方选择服务。
116
+
117
+ -----
118
+
119
+ <a id="model-experience"></a>
39
120
  ## 模型体验
40
121
 
41
- 通过 [`dsh-tool-web`](../tool-web/README.md) 间接影响;该工具把此提供方经 `maxBodyChars` 限制的解码文本或由 HTML 转换得到的 markdown 置于抓取结果包装层中,并保留提供方失败;重定向、标头与传输机制保持隐藏。
122
+ 间接地,通过 `dsh-tool-web`:该工具把本提供方经 `maxBodyChars` 限制的解码文本或由 HTML 转换得到的 markdown 置于抓取结果包装层内,而重定向、标头与传输上限保持隐藏。
42
123
 
43
124
  #### KV Cache 影响
44
125
 
45
126
  不会直接导致 KV Cache 失效;请求前缀变更由上述消费方负责。
46
127
 
47
- ## 已知限制与暂缓事项
128
+ ## 已知限制与延期工作
129
+
130
+ <a id="known-limitations-and-deferred-work"></a>
131
+
132
+
133
+ 这些限制说明提供方何时不安全或不合适。它们是当前包约束。
134
+
135
+ - **只解码文本内容**——包括 html/xhtml 与 `text/*` 加 JSON/XML 家族;缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`,可提取文本的 PDF 解码属于明确的延期工作。
136
+ - **charset 只来自 `Content-Type` 标头**(默认 UTF-8)——HTML `<meta charset>` 声明会被忽略;声明但无法识别的 charset 标签会抛出异常,而非回退。
137
+
138
+ <a id="dev-note"></a>
139
+ ### 开发备注
140
+
141
+ <details>
142
+ <summary>维护者的工作上下文——点击展开</summary>
143
+
144
+ 无。
48
145
 
49
- - **SSRF/私有网络防护暂缓**:不会阻止私有、loopback、link-local、multicast 或其他非公开目标,也不进行 DNS 解析后验证或逐跳重新验证(见 [web 能力 seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md))。在此功能落地前,该提供方是 SSRF 原语;能够访问敏感内部网络目标的部署**禁止启用它**。
50
- - **只解码文本内容**:包括 html/xhtml 与 `text/*` 加 JSON/XML 家族;缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`,可提取文本的 PDF 解码属于明确的暂缓工作。
51
- - **charset 只来自 `Content-Type` 标头**(默认为 UTF-8):HTML `<meta charset>` 声明会被忽略;声明但无法识别的 charset 标签会抛出异常,而非回退。
146
+ </details>
package/lib/index.js CHANGED
@@ -1,6 +1,257 @@
1
1
  import z from "@xneog/schemastery";
2
2
  import { WebError } from "@xneog/dsh-web";
3
3
  import { deadline, timeoutOf } from "@xneog/dsh-timeout";
4
+ import { proxyRouteFor } from "@xneog/dsh-http-proxy";
5
+ import { lookup } from "node:dns/promises";
6
+ import { isIP } from "node:net";
7
+ import ipaddr from "ipaddr.js";
8
+ //#region lib/types/network.js
9
+ /**
10
+ * Public-network resolution and address-pinned HTTP transport for `web-fetch-http`.
11
+ * One DNS answer set is validated before Undici receives it through a custom lookup,
12
+ * so the connection cannot resolve the hostname again to a private address.
13
+ *
14
+ * @module @xneog/dsh-web-fetch-http/network
15
+ */
16
+ /** RFC 6052 prefix lengths that may carry an IPv4 destination through NAT64. */
17
+ const RFC6052_PREFIX_LENGTHS = [
18
+ 32,
19
+ 40,
20
+ 48,
21
+ 56,
22
+ 64,
23
+ 96
24
+ ];
25
+ const IPV4ONLY_DISCOVERY_HOST = "ipv4only.arpa";
26
+ const IPV4ONLY_SENTINELS = new Set(["192.0.0.170", "192.0.0.171"]);
27
+ /**
28
+ * Return whether an address is globally reachable unicast. IPv4-mapped IPv6 is
29
+ * classified by its embedded IPv4 address; transition and translation prefixes
30
+ * remain blocked because their eventual IPv4 destination cannot be pinned here.
31
+ *
32
+ * @param input - textual IPv4 or IPv6 address.
33
+ * @returns true only for a public unicast destination.
34
+ */
35
+ function isPublicIpAddress(input) {
36
+ let parsed;
37
+ try {
38
+ parsed = ipaddr.parse(stripIpv6Brackets(input));
39
+ } catch {
40
+ return false;
41
+ }
42
+ if (parsed instanceof ipaddr.IPv4) return parsed.range() === "unicast";
43
+ if (parsed.isIPv4MappedAddress()) return parsed.toIPv4Address().range() === "unicast";
44
+ return parsed.range() === "unicast";
45
+ }
46
+ /**
47
+ * Resolve a hostname once and reject the complete answer set if any destination
48
+ * is not public. The returned addresses are the only ones the transport may use.
49
+ *
50
+ * @param hostname - URL hostname, including brackets when it is an IPv6 literal.
51
+ * @param signal - aborts the wait for system resolution; an in-flight OS lookup may finish unused.
52
+ * @param resolver - lookup implementation, overridden only by focused tests.
53
+ * @returns the validated, non-empty address set.
54
+ */
55
+ async function resolvePublicAddresses(hostname, signal, resolver = lookup) {
56
+ const unbracketed = stripIpv6Brackets(hostname);
57
+ const literalFamily = isIP(unbracketed);
58
+ const resolved = literalFamily === 0 ? await raceWithSignal(resolver(unbracketed, {
59
+ all: true,
60
+ order: "verbatim"
61
+ }), signal) : [{
62
+ address: unbracketed,
63
+ family: literalFamily
64
+ }];
65
+ if (resolved.length === 0) throw new WebError(`hostname "${hostname}" resolved to no addresses`, "WEB_PROVIDER_ERROR");
66
+ const nat64Prefixes = resolved.some((entry) => entry.family === 6 && isIP(entry.address) === 6) ? await discoverNat64Prefixes(signal, resolver) : [];
67
+ const addresses = [];
68
+ for (const entry of resolved) {
69
+ if (entry.family !== 4 && entry.family !== 6 || isIP(entry.address) !== entry.family) throw new WebError(`hostname "${hostname}" resolved to an invalid IP address`, "WEB_PROVIDER_ERROR");
70
+ if (!isPublicIpAddress(entry.address)) throw new WebError(`URL hostname "${hostname}" resolves to a non-public IP address`, "WEB_BLOCKED_URL");
71
+ const translatedIpv4 = translatedIpv4Address(entry.address, nat64Prefixes);
72
+ if (translatedIpv4 !== void 0 && !isPublicIpAddress(translatedIpv4)) throw new WebError(`URL hostname "${hostname}" resolves through NAT64 to a non-public IPv4 address`, "WEB_BLOCKED_URL");
73
+ addresses.push({
74
+ address: entry.address,
75
+ family: entry.family
76
+ });
77
+ }
78
+ return addresses;
79
+ }
80
+ /** Discover the active DNS64 prefix set using RFC 7050's reserved hostname. */
81
+ async function discoverNat64Prefixes(signal, resolver) {
82
+ const discovered = await raceWithSignal(resolver(IPV4ONLY_DISCOVERY_HOST, {
83
+ all: true,
84
+ order: "verbatim"
85
+ }), signal);
86
+ const prefixes = [];
87
+ const seen = /* @__PURE__ */ new Set();
88
+ for (const entry of discovered) {
89
+ if (entry.family !== 6 || isIP(entry.address) !== 6) continue;
90
+ const bytes = ipaddr.parse(entry.address).toByteArray();
91
+ for (const length of RFC6052_PREFIX_LENGTHS) {
92
+ const embedded = embeddedIpv4Address(bytes, length);
93
+ if (embedded === void 0 || !IPV4ONLY_SENTINELS.has(embedded)) continue;
94
+ const prefixBytes = bytes.slice(0, length / 8);
95
+ const key = `${String(length)}:${prefixBytes.join(".")}`;
96
+ if (seen.has(key)) continue;
97
+ seen.add(key);
98
+ prefixes.push({
99
+ bytes: prefixBytes,
100
+ length
101
+ });
102
+ }
103
+ }
104
+ return prefixes;
105
+ }
106
+ /** Return the RFC 6052-embedded IPv4 address when an IPv6 address matches a discovered prefix. */
107
+ function translatedIpv4Address(input, prefixes) {
108
+ if (isIP(input) !== 6) return void 0;
109
+ const bytes = ipaddr.parse(input).toByteArray();
110
+ for (const prefix of prefixes) {
111
+ if (!prefix.bytes.every((byte, index) => bytes[index] === byte)) continue;
112
+ const embedded = embeddedIpv4Address(bytes, prefix.length);
113
+ if (embedded !== void 0) return embedded;
114
+ }
115
+ }
116
+ /** Extract one IPv4 address from an RFC 6052 IPv6 layout. */
117
+ function embeddedIpv4Address(bytes, prefixLength) {
118
+ if (prefixLength === 96) return bytes.slice(12, 16).join(".");
119
+ if (bytes[8] !== 0) return void 0;
120
+ const prefixBytes = prefixLength / 8;
121
+ const beforeReservedOctet = 8 - prefixBytes;
122
+ return [...bytes.slice(prefixBytes, prefixBytes + beforeReservedOctet), ...bytes.slice(9, 13 - beforeReservedOctet)].join(".");
123
+ }
124
+ /**
125
+ * Whether a hostname is an IP literal that {@link resolvePublicAddresses} would refuse.
126
+ *
127
+ * A proxied hop skips those checks because the proxy resolves the origin, but a literal needs no
128
+ * resolution: the address is already stated, and handing it to a proxy running on this machine
129
+ * would reach exactly the loopback or private service the checks exist to keep out of reach.
130
+ *
131
+ * @param hostname - a URL's hostname, bracketed or not.
132
+ * @returns true when the host is a literal address no request may be sent to.
133
+ */
134
+ function isNonPublicIpLiteral(hostname) {
135
+ const unbracketed = stripIpv6Brackets(hostname);
136
+ return isIP(unbracketed) !== 0 && !isPublicIpAddress(unbracketed);
137
+ }
138
+ /**
139
+ * Fetch through an agent whose lookup callback returns only the already validated address set. The
140
+ * URL hostname remains intact for HTTP Host and TLS SNI.
141
+ *
142
+ * The agent is this request's own because the address set is: pinning is how this package refuses a
143
+ * DNS answer that changes between validation and connection, and it may not apply process-wide —
144
+ * an operator-configured MCP server or model endpoint on loopback is a supported destination, and
145
+ * only the URLs this tool fetches are the model's to choose.
146
+ *
147
+ * @param url - validated HTTP(S) URL the policy does not route through a proxy.
148
+ * @param addresses - public addresses returned by {@link resolvePublicAddresses}.
149
+ * @param headers - request headers.
150
+ * @param signal - request and body-read cancellation signal.
151
+ * @returns a response plus the disposer its consumer must call.
152
+ */
153
+ async function requestPinned(url, addresses, headers, signal) {
154
+ const { Agent, fetch } = await import("undici");
155
+ const dispatcher = new Agent({
156
+ autoSelectFamily: true,
157
+ connect: { lookup: createPinnedLookup(addresses) }
158
+ });
159
+ try {
160
+ return {
161
+ response: await fetch(url, {
162
+ method: "GET",
163
+ redirect: "manual",
164
+ headers,
165
+ signal,
166
+ dispatcher
167
+ }),
168
+ close: async () => {
169
+ await dispatcher.close();
170
+ }
171
+ };
172
+ } catch (error) {
173
+ await dispatcher.close();
174
+ throw error;
175
+ }
176
+ }
177
+ /**
178
+ * Fetch through the dispatcher the proxy policy already installed, letting the proxy resolve the
179
+ * origin.
180
+ *
181
+ * No address set is pinned because none exists to pin: the proxy performs the lookup, and a
182
+ * connection pinned to a locally resolved address would reach the origin directly and defeat the
183
+ * proxy. The dispatcher is the process-wide one, so hops share its connection pool and no caller
184
+ * closes it.
185
+ *
186
+ * @param dispatcher - the route's dispatcher, from `proxyRouteFor`.
187
+ * @param url - validated HTTP(S) URL the policy routes through a proxy.
188
+ * @param headers - request headers.
189
+ * @param signal - request and body-read cancellation signal.
190
+ * @returns a response plus a disposer that releases nothing, so both paths close alike.
191
+ */
192
+ async function requestVia(dispatcher, url, headers, signal) {
193
+ const { fetch } = await import("undici");
194
+ return {
195
+ response: await fetch(url, {
196
+ method: "GET",
197
+ redirect: "manual",
198
+ headers,
199
+ signal,
200
+ dispatcher
201
+ }),
202
+ close: () => Promise.resolve()
203
+ };
204
+ }
205
+ /** Production network operations kept as an object so provider tests can replace resolution only. */
206
+ const publicHttpNetwork = {
207
+ resolve: resolvePublicAddresses,
208
+ request: requestPinned,
209
+ requestVia
210
+ };
211
+ /**
212
+ * Build the connector lookup that serves a fixed validated answer set.
213
+ *
214
+ * @param addresses - public addresses retained from the preceding resolution.
215
+ * @returns a Node-compatible lookup callback that performs no network resolution.
216
+ */
217
+ function createPinnedLookup(addresses) {
218
+ return (hostname, options, callback) => {
219
+ const family = typeof options.family === "number" ? options.family : options.family === "IPv4" ? 4 : options.family === "IPv6" ? 6 : 0;
220
+ const eligible = family === 0 ? addresses : addresses.filter((address) => address.family === family);
221
+ const selected = eligible[0];
222
+ if (selected === void 0) {
223
+ callback(Object.assign(/* @__PURE__ */ new Error(`no validated address for ${hostname} in family ${family}`), {
224
+ code: "ENOTFOUND",
225
+ hostname
226
+ }), options.all === true ? [] : "", family);
227
+ return;
228
+ }
229
+ if (options.all === true) {
230
+ callback(null, eligible.map((address) => ({ ...address })));
231
+ return;
232
+ }
233
+ callback(null, selected.address, selected.family);
234
+ };
235
+ }
236
+ /** Race a non-cancellable OS lookup without letting it delay tool cancellation. */
237
+ function raceWithSignal(promise, signal) {
238
+ const abortError = () => new Error("web fetch aborted during hostname resolution", { cause: signal.reason });
239
+ if (signal.aborted) return Promise.reject(abortError());
240
+ return new Promise((resolve, reject) => {
241
+ const abort = () => {
242
+ reject(abortError());
243
+ };
244
+ signal.addEventListener("abort", abort, { once: true });
245
+ promise.then(resolve, reject).finally(() => {
246
+ signal.removeEventListener("abort", abort);
247
+ });
248
+ });
249
+ }
250
+ /** WHATWG URL retains brackets around IPv6 hostnames; IP parsers do not. */
251
+ function stripIpv6Brackets(hostname) {
252
+ return hostname.startsWith("[") && hostname.endsWith("]") ? hostname.slice(1, -1) : hostname;
253
+ }
254
+ //#endregion
4
255
  //#region lib/types/policy.js
5
256
  /**
6
257
  * URL validation and content-type classification for the local HTTP(S) fetch
@@ -9,18 +260,17 @@ import { deadline, timeoutOf } from "@xneog/dsh-timeout";
9
260
  *
10
261
  * @module @xneog/dsh-web-fetch-http/policy
11
262
  */
263
+ /** Maximum accepted request URL length enforced by the public fetch provider. */
264
+ const WEB_FETCH_MAX_URL_LENGTH = 2048;
12
265
  /**
13
- * Validate a request URL against the basic transport hygiene the provider
14
- * enforces before any network access: http(s) only, no embedded credentials,
15
- * bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise.
16
- * (SSRF / private-network blocking is deferred — see the package Agent Note.)
266
+ * Parse a request URL and enforce network-independent transport restrictions:
267
+ * HTTP(S) only and no embedded credentials. The provider applies this before
268
+ * resolving a destination.
17
269
  *
18
270
  * @param input - the raw URL string from the fetch request.
19
- * @param maxUrlLength - inclusive upper bound on `input`'s length.
20
271
  * @returns the parsed `URL`.
21
272
  */
22
- function validateFetchUrl(input, maxUrlLength) {
23
- if (input.length > maxUrlLength) throw new WebError(`URL exceeds the maximum length of ${maxUrlLength}`, "WEB_INVALID_URL");
273
+ function parseFetchUrl(input) {
24
274
  let url;
25
275
  try {
26
276
  url = new URL(input);
@@ -32,9 +282,21 @@ function validateFetchUrl(input, maxUrlLength) {
32
282
  return url;
33
283
  }
34
284
  /**
285
+ * Validate a request URL against the provider's complete pre-network policy:
286
+ * bounded length plus the restrictions enforced by {@link parseFetchUrl}.
287
+ * Public-address resolution and connection pinning run after this check.
288
+ *
289
+ * @param input - the raw URL string from the fetch request.
290
+ * @returns the parsed `URL`.
291
+ */
292
+ function validateFetchUrl(input) {
293
+ if (input.length > 2048) throw new WebError(`URL exceeds the maximum length of ${WEB_FETCH_MAX_URL_LENGTH}`, "WEB_INVALID_URL");
294
+ return parseFetchUrl(input);
295
+ }
296
+ /**
35
297
  * Two URLs are same-origin when scheme, hostname, and port match. A redirect
36
298
  * that crosses origins is refused so each new origin requires a fresh tool call
37
- * (and thus a fresh provider/permission decision).
299
+ * and public-address validation.
38
300
  *
39
301
  * @param a - one of the two URLs to compare.
40
302
  * @param b - the other URL to compare.
@@ -92,12 +354,10 @@ function decoderForCharset(charset) {
92
354
  //#endregion
93
355
  //#region lib/types/provider.js
94
356
  /**
95
- * Safe HTTP(S) retrieval for `ctx.web`: validates URLs, follows only same-origin redirects,
96
- * enforces time and size limits, classifies and decodes text, and leaves presentation to
97
- * `@xneog/dsh-tool-web`. Requests carry no browser cookies or ambient credentials.
98
- *
99
- * Private-network and SSRF protection is not implemented; do not enable this provider where
100
- * it can reach sensitive internal targets.
357
+ * Safe HTTP(S) retrieval for `ctx.web`: validates and pins public IP destinations, follows
358
+ * only same-origin redirects, enforces time and size limits, classifies and decodes text,
359
+ * and leaves presentation to `@xneog/dsh-tool-web`. Requests carry no browser cookies
360
+ * or ambient credentials.
101
361
  * @module @xneog/dsh-web-fetch-http/provider
102
362
  */
103
363
  var __addDisposableResource = function(env, value, async) {
@@ -163,9 +423,15 @@ const LOCAL_FETCH_PROVIDER_ID = "http";
163
423
  /** The anonymous public HTTP(S) fetch provider. */
164
424
  var HttpFetchProvider = class {
165
425
  limits;
426
+ resolveAddresses;
166
427
  id = LOCAL_FETCH_PROVIDER_ID;
167
- constructor(limits) {
428
+ /**
429
+ * @param limits - resolved transport and response limits.
430
+ * @param resolveAddresses - resolver that rejects non-public destinations before returning.
431
+ */
432
+ constructor(limits, resolveAddresses = publicHttpNetwork.resolve) {
168
433
  this.limits = limits;
434
+ this.resolveAddresses = resolveAddresses;
169
435
  }
170
436
  /** No credentials to check — an anonymous public fetcher is always usable. */
171
437
  available() {
@@ -190,49 +456,54 @@ var HttpFetchProvider = class {
190
456
  }
191
457
  /** Follow same-origin redirects up to the hop cap, then read the final response. */
192
458
  async followAndRead(initialUrl, signal) {
193
- let currentUrl = validateFetchUrl(initialUrl, this.limits.maxUrlLength);
459
+ let currentUrl = validateFetchUrl(initialUrl);
194
460
  let redirectsFollowed = 0;
195
461
  for (;;) {
196
- const response = await this.requestOnce(currentUrl, signal);
197
- if (isRedirectStatus(response.status)) {
198
- if (redirectsFollowed >= this.limits.maxRedirects) {
199
- await response.body?.cancel();
200
- throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, "WEB_REDIRECT_BLOCKED");
201
- }
202
- const location = response.headers.get("location");
203
- if (location === null) {
204
- await response.body?.cancel();
205
- throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, "WEB_PROVIDER_ERROR");
206
- }
207
- const target = resolveRedirect(location, currentUrl);
208
- let validatedTarget;
209
- try {
210
- validatedTarget = validateFetchUrl(target.toString(), this.limits.maxUrlLength);
211
- if (!isSameOrigin(validatedTarget, currentUrl)) throw new WebError(`cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`, "WEB_REDIRECT_BLOCKED");
212
- } catch (error) {
462
+ const request = await this.requestOnce(currentUrl, signal);
463
+ const { response } = request;
464
+ try {
465
+ if (isRedirectStatus(response.status)) {
466
+ if (redirectsFollowed >= this.limits.maxRedirects) {
467
+ await response.body?.cancel();
468
+ throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, "WEB_REDIRECT_BLOCKED");
469
+ }
470
+ const location = response.headers.get("location");
471
+ if (location === null) {
472
+ await response.body?.cancel();
473
+ throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, "WEB_PROVIDER_ERROR");
474
+ }
475
+ const target = resolveRedirect(location, currentUrl);
476
+ let validatedTarget;
477
+ try {
478
+ validatedTarget = validateFetchUrl(target.toString());
479
+ if (!isSameOrigin(validatedTarget, currentUrl)) throw new WebError(`cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`, "WEB_REDIRECT_BLOCKED");
480
+ } catch (error) {
481
+ await response.body?.cancel();
482
+ throw error;
483
+ }
213
484
  await response.body?.cancel();
214
- throw error;
485
+ currentUrl = validatedTarget;
486
+ redirectsFollowed++;
487
+ continue;
215
488
  }
216
- await response.body?.cancel();
217
- currentUrl = validatedTarget;
218
- redirectsFollowed++;
219
- continue;
489
+ return await this.readBody(response, currentUrl, signal);
490
+ } finally {
491
+ await request.close();
220
492
  }
221
- return await this.readBody(response, currentUrl, signal);
222
493
  }
223
494
  }
224
495
  async requestOnce(url, signal) {
496
+ const headers = {
497
+ "user-agent": this.limits.userAgent,
498
+ "accept": "text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8"
499
+ };
225
500
  try {
226
- return await fetch(url, {
227
- method: "GET",
228
- redirect: "manual",
229
- headers: {
230
- "user-agent": this.limits.userAgent,
231
- "accept": "text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8"
232
- },
233
- signal
234
- });
501
+ const route = proxyRouteFor(url);
502
+ if (route.proxied && !isNonPublicIpLiteral(url.hostname)) return await publicHttpNetwork.requestVia(route.dispatcher, url, headers, signal);
503
+ const addresses = await this.resolveAddresses(url.hostname, signal);
504
+ return await publicHttpNetwork.request(url, addresses, headers, signal);
235
505
  } catch (error) {
506
+ if (error instanceof WebError) throw error;
236
507
  throw translateAbortOrNetwork(error, signal);
237
508
  }
238
509
  }
@@ -358,22 +629,19 @@ function translateAbortOrNetwork(error, signal) {
358
629
  //#endregion
359
630
  //#region lib/types/index.js
360
631
  /**
361
- * `@xneog/dsh-web-fetch-http`: registers an anonymous public HTTP(S)
362
- * `WebFetchProvider` with `ctx.web`. A function/namespace plugin (NOT a
363
- * default-export service): it registers INTO the seam's fetch registry, like the
364
- * search providers register into the search registry.
632
+ * Anonymous public HTTP(S) `WebFetchProvider` plugin. It contributes to the
633
+ * `ctx.web` registry without owning the service.
365
634
  *
366
635
  * @module @xneog/dsh-web-fetch-http
367
636
  */
368
637
  const MAX_NODE_TIMER_DELAY_MS = 2147483647;
369
638
  /** Default `User-Agent`: an explicit product agent, never a browser disguise. */
370
- const DEFAULT_USER_AGENT = "deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)";
639
+ const DEFAULT_USER_AGENT = "xneog-harness/0.0.1 (+https://github.com/gomes007-alt)";
371
640
  /** Cordis plugin name used by loader diagnostics. */
372
641
  const name = "web-fetch-http";
373
642
  /** The web seam this provider registers into. */
374
643
  const inject = ["web"];
375
644
  const Config = z.object({
376
- maxUrlLength: z.number().default(2048),
377
645
  maxResponseBytes: z.number().default(5e6),
378
646
  maxBodyChars: z.number().default(1e5),
379
647
  timeoutMs: z.number().default(3e4),
@@ -396,13 +664,11 @@ function assertNonNegativeInteger(name, value) {
396
664
  /** Register the local HTTP(S) fetch provider with `ctx.web`. */
397
665
  function apply(ctx, config) {
398
666
  const resolved = config;
399
- assertPositiveFinite("maxUrlLength", resolved.maxUrlLength);
400
667
  assertPositiveFinite("maxResponseBytes", resolved.maxResponseBytes);
401
668
  assertPositiveFinite("maxBodyChars", resolved.maxBodyChars);
402
669
  assertTimeoutMs(resolved.timeoutMs);
403
670
  assertNonNegativeInteger("maxRedirects", resolved.maxRedirects);
404
671
  const limits = {
405
- maxUrlLength: resolved.maxUrlLength,
406
672
  maxResponseBytes: resolved.maxResponseBytes,
407
673
  maxBodyChars: resolved.maxBodyChars,
408
674
  timeoutMs: resolved.timeoutMs,
@@ -1,25 +1,21 @@
1
1
  /**
2
- * `@xneog/dsh-web-fetch-http`: registers an anonymous public HTTP(S)
3
- * `WebFetchProvider` with `ctx.web`. A function/namespace plugin (NOT a
4
- * default-export service): it registers INTO the seam's fetch registry, like the
5
- * search providers register into the search registry.
2
+ * Anonymous public HTTP(S) `WebFetchProvider` plugin. It contributes to the
3
+ * `ctx.web` registry without owning the service.
6
4
  *
7
5
  * @module @xneog/dsh-web-fetch-http
8
6
  */
9
7
  import type { Context } from '@xneog/cordis';
10
8
  import z from '@xneog/schemastery';
11
9
  export { LOCAL_FETCH_PROVIDER_ID, HttpFetchProvider, } from './provider.ts';
12
- export type { HttpFetchLimits } from './provider.ts';
10
+ export type { HttpFetchLimits, HttpFetchResolver } from './provider.ts';
13
11
  /** Default `User-Agent`: an explicit product agent, never a browser disguise. */
14
- export declare const DEFAULT_USER_AGENT = "deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)";
12
+ export declare const DEFAULT_USER_AGENT = "xneog-harness/0.0.1 (+https://github.com/gomes007-alt)";
15
13
  /** Cordis plugin name used by loader diagnostics. */
16
14
  export declare const name = "web-fetch-http";
17
15
  /** The web seam this provider registers into. */
18
16
  export declare const inject: string[];
19
17
  /** Plugin config: the provider's transport and size limits plus its `User-Agent` (all defaulted). */
20
18
  export interface Config {
21
- /** Maximum accepted request URL length. */
22
- maxUrlLength?: number;
23
19
  /** Maximum response body size in bytes. */
24
20
  maxResponseBytes?: number;
25
21
  /** Maximum decoded body length in characters. */
@@ -0,0 +1,106 @@
1
+ /**
2
+ * Public-network resolution and address-pinned HTTP transport for `web-fetch-http`.
3
+ * One DNS answer set is validated before Undici receives it through a custom lookup,
4
+ * so the connection cannot resolve the hostname again to a private address.
5
+ *
6
+ * @module @xneog/dsh-web-fetch-http/network
7
+ */
8
+ import type { LookupAddress, LookupOptions } from 'node:dns';
9
+ import type { Dispatcher, Response } from 'undici';
10
+ /** One address resolved and retained for the subsequent pinned connection. */
11
+ export interface PublicAddress {
12
+ /** Canonical textual IPv4 or IPv6 address. */
13
+ readonly address: string;
14
+ /** Address family accepted by Node's connection lookup callback. */
15
+ readonly family: 4 | 6;
16
+ }
17
+ /** The result of one address-pinned request; closing releases its private pool. */
18
+ export interface PinnedResponse {
19
+ /** HTTP response whose body remains readable until `close()` is called. */
20
+ readonly response: Response;
21
+ /** Release the request's dispatcher after the response body is consumed or cancelled. */
22
+ close(): Promise<void>;
23
+ }
24
+ /** Resolver signature used to test public-address policy without process DNS changes. */
25
+ export type AddressResolver = (hostname: string, options: {
26
+ all: true;
27
+ order: 'verbatim';
28
+ }) => Promise<LookupAddress[]>;
29
+ /**
30
+ * Return whether an address is globally reachable unicast. IPv4-mapped IPv6 is
31
+ * classified by its embedded IPv4 address; transition and translation prefixes
32
+ * remain blocked because their eventual IPv4 destination cannot be pinned here.
33
+ *
34
+ * @param input - textual IPv4 or IPv6 address.
35
+ * @returns true only for a public unicast destination.
36
+ */
37
+ export declare function isPublicIpAddress(input: string): boolean;
38
+ /**
39
+ * Resolve a hostname once and reject the complete answer set if any destination
40
+ * is not public. The returned addresses are the only ones the transport may use.
41
+ *
42
+ * @param hostname - URL hostname, including brackets when it is an IPv6 literal.
43
+ * @param signal - aborts the wait for system resolution; an in-flight OS lookup may finish unused.
44
+ * @param resolver - lookup implementation, overridden only by focused tests.
45
+ * @returns the validated, non-empty address set.
46
+ */
47
+ export declare function resolvePublicAddresses(hostname: string, signal: AbortSignal, resolver?: AddressResolver): Promise<PublicAddress[]>;
48
+ /**
49
+ * Whether a hostname is an IP literal that {@link resolvePublicAddresses} would refuse.
50
+ *
51
+ * A proxied hop skips those checks because the proxy resolves the origin, but a literal needs no
52
+ * resolution: the address is already stated, and handing it to a proxy running on this machine
53
+ * would reach exactly the loopback or private service the checks exist to keep out of reach.
54
+ *
55
+ * @param hostname - a URL's hostname, bracketed or not.
56
+ * @returns true when the host is a literal address no request may be sent to.
57
+ */
58
+ export declare function isNonPublicIpLiteral(hostname: string): boolean;
59
+ /**
60
+ * Fetch through an agent whose lookup callback returns only the already validated address set. The
61
+ * URL hostname remains intact for HTTP Host and TLS SNI.
62
+ *
63
+ * The agent is this request's own because the address set is: pinning is how this package refuses a
64
+ * DNS answer that changes between validation and connection, and it may not apply process-wide —
65
+ * an operator-configured MCP server or model endpoint on loopback is a supported destination, and
66
+ * only the URLs this tool fetches are the model's to choose.
67
+ *
68
+ * @param url - validated HTTP(S) URL the policy does not route through a proxy.
69
+ * @param addresses - public addresses returned by {@link resolvePublicAddresses}.
70
+ * @param headers - request headers.
71
+ * @param signal - request and body-read cancellation signal.
72
+ * @returns a response plus the disposer its consumer must call.
73
+ */
74
+ export declare function requestPinned(url: URL, addresses: readonly PublicAddress[], headers: Record<string, string>, signal: AbortSignal): Promise<PinnedResponse>;
75
+ /**
76
+ * Fetch through the dispatcher the proxy policy already installed, letting the proxy resolve the
77
+ * origin.
78
+ *
79
+ * No address set is pinned because none exists to pin: the proxy performs the lookup, and a
80
+ * connection pinned to a locally resolved address would reach the origin directly and defeat the
81
+ * proxy. The dispatcher is the process-wide one, so hops share its connection pool and no caller
82
+ * closes it.
83
+ *
84
+ * @param dispatcher - the route's dispatcher, from `proxyRouteFor`.
85
+ * @param url - validated HTTP(S) URL the policy routes through a proxy.
86
+ * @param headers - request headers.
87
+ * @param signal - request and body-read cancellation signal.
88
+ * @returns a response plus a disposer that releases nothing, so both paths close alike.
89
+ */
90
+ export declare function requestVia(dispatcher: Dispatcher, url: URL, headers: Record<string, string>, signal: AbortSignal): Promise<PinnedResponse>;
91
+ /** Production network operations kept as an object so provider tests can replace resolution only. */
92
+ export declare const publicHttpNetwork: {
93
+ resolve: typeof resolvePublicAddresses;
94
+ request: typeof requestPinned;
95
+ requestVia: typeof requestVia;
96
+ };
97
+ type LookupCallback = (error: NodeJS.ErrnoException | null, address: string | LookupAddress[], family?: number) => void;
98
+ /**
99
+ * Build the connector lookup that serves a fixed validated answer set.
100
+ *
101
+ * @param addresses - public addresses retained from the preceding resolution.
102
+ * @returns a Node-compatible lookup callback that performs no network resolution.
103
+ */
104
+ export declare function createPinnedLookup(addresses: readonly PublicAddress[]): (hostname: string, options: LookupOptions, callback: LookupCallback) => void;
105
+ export {};
106
+ //# sourceMappingURL=network.d.ts.map
@@ -5,23 +5,32 @@
5
5
  *
6
6
  * @module @xneog/dsh-web-fetch-http/policy
7
7
  */
8
+ /** Maximum accepted request URL length enforced by the public fetch provider. */
9
+ export declare const WEB_FETCH_MAX_URL_LENGTH = 2048;
8
10
  /** The body kinds this provider decodes. */
9
11
  export type FetchableKind = 'html' | 'text';
10
12
  /**
11
- * Validate a request URL against the basic transport hygiene the provider
12
- * enforces before any network access: http(s) only, no embedded credentials,
13
- * bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise.
14
- * (SSRF / private-network blocking is deferred — see the package Agent Note.)
13
+ * Parse a request URL and enforce network-independent transport restrictions:
14
+ * HTTP(S) only and no embedded credentials. The provider applies this before
15
+ * resolving a destination.
15
16
  *
16
17
  * @param input - the raw URL string from the fetch request.
17
- * @param maxUrlLength - inclusive upper bound on `input`'s length.
18
18
  * @returns the parsed `URL`.
19
19
  */
20
- export declare function validateFetchUrl(input: string, maxUrlLength: number): URL;
20
+ export declare function parseFetchUrl(input: string): URL;
21
+ /**
22
+ * Validate a request URL against the provider's complete pre-network policy:
23
+ * bounded length plus the restrictions enforced by {@link parseFetchUrl}.
24
+ * Public-address resolution and connection pinning run after this check.
25
+ *
26
+ * @param input - the raw URL string from the fetch request.
27
+ * @returns the parsed `URL`.
28
+ */
29
+ export declare function validateFetchUrl(input: string): URL;
21
30
  /**
22
31
  * Two URLs are same-origin when scheme, hostname, and port match. A redirect
23
32
  * that crosses origins is refused so each new origin requires a fresh tool call
24
- * (and thus a fresh provider/permission decision).
33
+ * and public-address validation.
25
34
  *
26
35
  * @param a - one of the two URLs to compare.
27
36
  * @param b - the other URL to compare.
@@ -1,17 +1,14 @@
1
1
  /**
2
- * Safe HTTP(S) retrieval for `ctx.web`: validates URLs, follows only same-origin redirects,
3
- * enforces time and size limits, classifies and decodes text, and leaves presentation to
4
- * `@xneog/dsh-tool-web`. Requests carry no browser cookies or ambient credentials.
5
- *
6
- * Private-network and SSRF protection is not implemented; do not enable this provider where
7
- * it can reach sensitive internal targets.
2
+ * Safe HTTP(S) retrieval for `ctx.web`: validates and pins public IP destinations, follows
3
+ * only same-origin redirects, enforces time and size limits, classifies and decodes text,
4
+ * and leaves presentation to `@xneog/dsh-tool-web`. Requests carry no browser cookies
5
+ * or ambient credentials.
8
6
  * @module @xneog/dsh-web-fetch-http/provider
9
7
  */
10
8
  import type { WebFetchProvider, WebFetchRequest, WebFetchResult } from '@xneog/dsh-web';
9
+ import type { PublicAddress } from './network.ts';
11
10
  /** Resolved provider limits (the plugin's schemastery Config supplies defaults). */
12
11
  export interface HttpFetchLimits {
13
- /** Maximum accepted request URL length. */
14
- maxUrlLength: number;
15
12
  /** Maximum response body size in bytes (read is aborted past this). */
16
13
  maxResponseBytes: number;
17
14
  /** Maximum decoded body length in characters (truncated past this). */
@@ -23,13 +20,20 @@ export interface HttpFetchLimits {
23
20
  /** `User-Agent` header sent on every request. */
24
21
  userAgent: string;
25
22
  }
23
+ /** Resolve one hostname to an already policy-validated address set. */
24
+ export type HttpFetchResolver = (hostname: string, signal: AbortSignal) => Promise<PublicAddress[]>;
26
25
  /** Stable id this provider registers under. */
27
26
  export declare const LOCAL_FETCH_PROVIDER_ID = "http";
28
27
  /** The anonymous public HTTP(S) fetch provider. */
29
28
  export declare class HttpFetchProvider implements WebFetchProvider {
30
29
  private readonly limits;
30
+ private readonly resolveAddresses;
31
31
  readonly id = "http";
32
- constructor(limits: HttpFetchLimits);
32
+ /**
33
+ * @param limits - resolved transport and response limits.
34
+ * @param resolveAddresses - resolver that rejects non-public destinations before returning.
35
+ */
36
+ constructor(limits: HttpFetchLimits, resolveAddresses?: HttpFetchResolver);
33
37
  /** No credentials to check — an anonymous public fetcher is always usable. */
34
38
  available(): boolean;
35
39
  fetch(request: WebFetchRequest, signal?: AbortSignal): Promise<WebFetchResult>;
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@xneog/dsh-web-fetch-http",
3
3
  "description": "Anonymous public HTTP(S) fetch provider for the xneog web capability seam (ctx.web)",
4
- "version": "0.1.0",
4
+ "version": "0.1.3-alpha.1",
5
5
  "publishConfig": {
6
6
  "access": "public"
7
7
  },
@@ -18,32 +18,29 @@
18
18
  "types": "./lib/types/index.d.ts",
19
19
  "default": "./lib/index.js"
20
20
  },
21
- "./invariant": {
22
- "types": "./lib/types/invariant.d.ts",
23
- "default": "./lib/invariant.js"
24
- },
25
21
  "./src/*": "./src/*",
26
22
  "./package.json": "./package.json"
27
23
  },
28
24
  "files": [
29
25
  "lib/index.js",
30
- "lib/invariant.js",
31
26
  "lib/types/**/*.d.ts"
32
27
  ],
33
28
  "license": "MIT",
34
29
  "peerDependencies": {
35
- "@xneog/dsh-invariants": "0.1.0",
36
- "@xneog/dsh-timeout": "0.1.0",
37
- "@xneog/dsh-web": "0.1.0",
38
- "@xneog/cordis": "0.1.0"
30
+ "@xneog/cordis": "^4.0.2",
31
+ "@xneog/dsh-timeout": "^0.1.3-alpha.1",
32
+ "@xneog/dsh-http-proxy": "^0.1.3-alpha.1",
33
+ "@xneog/dsh-web": "^0.1.3-alpha.1"
39
34
  },
40
35
  "dependencies": {
41
- "@xneog/schemastery": "0.1.0"
36
+ "ipaddr.js": "^2.5.0",
37
+ "undici": "^8.10.0",
38
+ "@xneog/schemastery": "^3.18.2"
42
39
  },
43
40
  "devDependencies": {
44
- "@xneog/dsh-invariants": "0.1.0",
45
- "@xneog/dsh-timeout": "0.1.0",
46
- "@xneog/dsh-web": "0.1.0",
47
- "@xneog/cordis": "0.1.0"
41
+ "@xneog/cordis": "^4.0.2",
42
+ "@xneog/dsh-http-proxy": "^0.1.3-alpha.1",
43
+ "@xneog/dsh-timeout": "^0.1.3-alpha.1",
44
+ "@xneog/dsh-web": "^0.1.3-alpha.1"
48
45
  }
49
46
  }
package/lib/invariant.js DELETED
@@ -1,23 +0,0 @@
1
- //#region lib/types/invariant.js
2
- /**
3
- * Package-owned invariant companion for `@xneog/dsh-web-fetch-http`.
4
- * @module @xneog/dsh-web-fetch-http/invariant
5
- */
6
- const PACKAGE_NAME = "@xneog/dsh-web-fetch-http";
7
- /** Cordis companion plugin name. */
8
- const name = "web-fetch-http-invariant";
9
- /** Service required before the companion can reserve package ownership. */
10
- const inject = ["invariants"];
11
- /**
12
- * No runtime invariant: this package exposes no independent event sequence or mutable data relation
13
- * beyond contracts enforced at its owning seam.
14
- */
15
- const install = () => {};
16
- /**
17
- * Register this package's invariant companion.
18
- * @param ctx - Cordis context carrying the invariant service.
19
- * @returns the installed registration's disposer after setup succeeds.
20
- */
21
- const apply = (ctx) => Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install));
22
- //#endregion
23
- export { apply, inject, name };
@@ -1,16 +0,0 @@
1
- /**
2
- * Package-owned invariant companion for `@xneog/dsh-web-fetch-http`.
3
- * @module @xneog/dsh-web-fetch-http/invariant
4
- */
5
- import type { Context } from '@xneog/cordis';
6
- /** Cordis companion plugin name. */
7
- export declare const name = "web-fetch-http-invariant";
8
- /** Service required before the companion can reserve package ownership. */
9
- export declare const inject: string[];
10
- /**
11
- * Register this package's invariant companion.
12
- * @param ctx - Cordis context carrying the invariant service.
13
- * @returns the installed registration's disposer after setup succeeds.
14
- */
15
- export declare const apply: (ctx: Context) => Promise<() => void>;
16
- //# sourceMappingURL=invariant.d.ts.map