@qilin-agent/web-fetch-http 0.0.0-stage → 3.1.3

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 kkutysllb
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.i18n.yaml

@@ -0,0 +1,61 @@
+# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
+# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
+# After editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write packages/web/web-fetch-http/README.md
+/:
+  en: a55f23f2363fdbd6
+  zh: b1fd7328e041ec52
+/qilin-web-fetch-http:
+  en: 38829ee8fd1e419d
+  zh: 4db2b1a68b753c01
+/qilin-web-fetch-http/summary:
+  en: 4ad5cf8fe6b3121f
+  zh: d405bb551d343807
+/qilin-web-fetch-http/table-of-contents:
+  en: d152484eb41ac6b4
+  zh: 09388d293f9be9cb
+/qilin-web-fetch-http/use-this-package:
+  en: 630b12516d53e905
+  zh: eed124cfe9c8dc1f
+/qilin-web-fetch-http/use-this-package/when-to-choose-it:
+  en: eb52e77279c21d6f
+  zh: 0560e89caa377bfc
+/qilin-web-fetch-http/use-this-package/minimal-configuration:
+  en: b4eecdca0c9f5e68
+  zh: 34f208d7ff205c20
+/qilin-web-fetch-http/use-this-package/what-a-fetch-returns:
+  en: 2d18cdcc9b53dd0f
+  zh: 3bc6ca7c5933ee5d
+/qilin-web-fetch-http/use-this-package/transport-behavior:
+  en: c432147daf2877cf
+  zh: c14d507dd66e124a
+/qilin-web-fetch-http/use-this-package/failures-and-recovery:
+  en: 35407f106dc65deb
+  zh: 02118072fba68b8f
+/qilin-web-fetch-http/understand-the-implementation:
+  en: 9842c69ad3636bb3
+  zh: 3cdc3f816cbdd0cb
+/qilin-web-fetch-http/understand-the-implementation/design-philosophy:
+  en: b09832c1e63fee37
+  zh: 88427bd3fc022d34
+/qilin-web-fetch-http/understand-the-implementation/source-map:
+  en: 023ca0c9a9ef4968
+  zh: 74a5785a493f5111
+/qilin-web-fetch-http/understand-the-implementation/read-path:
+  en: 749c0ffeac145807
+  zh: 1468821a33a9518a
+/qilin-web-fetch-http/further-exploration:
+  en: a74a50853efdc0c5
+  zh: 0d9c9b6340e07932
+/qilin-web-fetch-http/model-experience:
+  en: 771846af95daa886
+  zh: 20b3cb9193b74660
+/qilin-web-fetch-http/model-experience/kv-cache-effect:
+  en: 40294c1c31b74824
+  zh: d75d48f485447a74
+/qilin-web-fetch-http/known-limitations-and-deferred-work:
+  en: 4d935fd8f466a234
+  zh: c398eb7aa737f5ea
+/qilin-web-fetch-http/known-limitations-and-deferred-work/dev-note:
+  en: e67f633dc6fe7773
+  zh: 3bb0c20c93f16f98

package/README.md

@@ -1,3 +1,145 @@
-# Temporary Holding Version
+---
+description: "The anonymous public HTTP(S) fetch backend for ctx.web: how deployments mount bounded, safe URL retrieval with same-origin redirects and text-only decoding."
+kind: "package-reference"
+---
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# @qilin-agent/web-fetch-http
+
+English | [中文](README.zh.md)
+
+## Summary
+
+With `qilin-web-fetch-http`, the harness can fetch public HTTP(S) pages through the web service (`ctx.web`) and get their status code plus bounded, decoded content without sending credentials. Choose it when a composition needs safe retrieval with URL validation, public-address resolution, connection pinning, same-origin redirects, byte and character caps, and an explicit product `User-Agent`. It returns non-2xx responses as results rather than errors, and rejects non-public destinations, binary data, and unsupported content types. The model-facing `web_fetch` tool lives in `qilin-tool-web`, which renders this provider's bodies.
+
+## Table of Contents
+
+- [Use this package](#use-this-package)
+- [Understand the implementation](#understand-the-implementation)
+- [Further Exploration](#further-exploration)
+- [Model Experience](#model-experience)
+- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
+- [Dev Note](#dev-note)
+
+-----
+
+<a id="use-this-package"></a>
+## Use this package
+
+Mount the provider in a composition that already loads the web service; it registers as the `http` fetch provider, so `ctx.web.fetch()` resolves it automatically when it is the only usable fetch backend — or pin it with `fetchProvider: http`.
+
+### When to choose it
+
+Choose this backend when a deployment must fetch public pages with bounded output and safe transport: no credentials are sent, every resolved address must be public, each connection is pinned to the validated answer set, redirects cannot escape the origin, and every response is capped.
+
+### Minimal configuration
+
+Load the web service and the provider; configurable limits have safe defaults and validate at plugin construction, so an invalid value fails loudly instead of building a provider with nonsensical caps. The URL security limit is fixed at 2,048 characters.
+
+```yaml
+- name: '@qilin-agent/web'
+- name: '@qilin-agent/web-fetch-http'
+```
+
+| Field | Default | Meaning |
+|---|---|---|
+| `maxResponseBytes` | `5,000,000` | Maximum response body size in bytes |
+| `maxBodyChars` | `100,000` | Maximum decoded body length in characters |
+| `timeoutMs` | `30,000` | Fetch timeout — a resource backstop, not the model-facing tool budget |
+| `maxRedirects` | `5` | Maximum same-origin redirect hops (`0` follows none) |
+| `userAgent` | `deepseek-harness/…` | `User-Agent` header sent on every request |
+
+The generated [configuration catalog](../../../docs/config-catalog.md#qilin-agentweb-fetch-http) is the exhaustive source for every accepted field and its JSDoc.
+
+### What a fetch returns
+
+A successful call yields a `WebFetchResult`: the final URL after allowed redirects, the HTTP status code, a decoded body classified as `html` or `text`, and a `truncated` flag. A non-2xx response is a result, not an error — the status code is part of the fetched resource state; `WebError` is reserved for failures to safely retrieve or represent the resource.
+
+```text
+const page = await ctx.web.fetch({ url: 'https://example.com' })
+// page.body.kind === 'html' | 'text'; page.statusCode === 200 | 404 | ...
+```
+
+### Transport behavior
+
+The provider keeps requests anonymous and bounded: it accepts only `http:` and `https:` URLs without embedded credentials and rejects URLs over 2,048 characters. It resolves each hostname once, rejects the complete result if any IPv4 or IPv6 address is not public unicast, and pins the connection to that validated set. IPv6 checks discover the active DNS64 prefix and reject translations to non-public IPv4. Each same-origin redirect repeats resolution and pinning; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product `User-Agent`.
+
+### Failures and recovery
+
+Failures throw `WebError` with a machine-routable code: `WEB_INVALID_URL`, `WEB_BLOCKED_URL`, `WEB_FETCH_TOO_LARGE`, `WEB_FETCH_TIMEOUT`, `WEB_REDIRECT_BLOCKED`, `WEB_UNSUPPORTED_CONTENT_TYPE`, `WEB_ABORTED`, or `WEB_PROVIDER_ERROR`. Direct callers can route on the code; the model-facing `web_fetch` tool surfaces the failure text to the model under its own error wrapper.
+
+-----
+
+<a id="understand-the-implementation"></a>
+## Understand the implementation
+
+<details>
+<summary>Implementation internals — click to expand</summary>
+
+This section explains the design decisions behind the provider; the observable behavior is fully covered in [Use this package](#use-this-package).
+
+### Design philosophy
+
+The package is built on one separation and one layered timeout:
+
+- **Safe retrieval vs. presentation.** This provider owns URL validation, public-address enforcement, connection pinning, HTTP transport, redirect policy, caps, charset decoding, and binary rejection; `qilin-tool-web` owns HTML→markdown and truncation formatting. A non-2xx response is data, not failure.
+- **Two timeout layers.** The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers; the model-facing tool-call budget belongs to `qilin-tool-call-timeout-policy`, which arms `exec.signal`. When the outer deadline fires first the provider reports `WEB_ABORTED` and the policy replaces it with `TOOL_TIMEOUT`; `WEB_FETCH_TIMEOUT` therefore identifies a direct service caller whose provider budget elapsed.
+
+### Source map
+
+| File | Role |
+|---|---|
+| [`src/index.ts`](src/index.ts) | Plugin entry: config schema, limit validation, provider registration |
+| [`src/provider.ts`](src/provider.ts) | The `HttpFetchProvider`: pinned transport, redirect following, capped reads, charset decoding |
+| [`src/network.ts`](src/network.ts) | Public-address resolution, DNS64 discovery, and connection pinning |
+| [`src/policy.ts`](src/policy.ts) | URL validation, same-origin checks, content-type classification, charset parsing |
+
+### Read path
+
+A fetch validates the URL, resolves the hostname once, rejects the complete answer set when any address is not public, and pins the connection to the accepted addresses. It repeats that check for each same-origin redirect; a cross-origin redirect or non-public target fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset, and read under the byte cap; the decoded text is then truncated to the character cap.
+
+</details>
+
+-----
+
+<a id="further-exploration"></a>
+## Further Exploration
+
+Read these pages when the package-level contract is not enough. They move from the shared vocabulary to the service, the model-facing tools, and the design rationale.
+
+- [Web subsystem](../../../docs/subsystems/web.md) — the exhaustive fetch request/result vocabulary and error codes.
+- [Web package map](../README.md) — the six-package family and each role.
+- [qilin-web](../web/README.md) — the web service this provider registers into.
+- [qilin-tool-web](../tool-web/README.md) — the model-facing `web_fetch` tool that renders this provider's bodies.
+- [Generated configuration catalog](../../../docs/config-catalog.md#qilin-agentweb-fetch-http) — every accepted config field and its source declaration.
+- [Web capability seam decision](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) — why search and fetch share one provider-selection service.
+
+-----
+
+<a id="model-experience"></a>
+## Model Experience
+
+Indirectly, through `qilin-tool-web`, which renders this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper while redirects, headers, and transport limits remain hidden.
+
+#### KV Cache effect
+
+No direct invalidation; the named consumer owns any request-prefix changes.
+
+## Known Limitations and Deferred Work
+
+<a id="known-limitations-and-deferred-work"></a>
+
+
+These limits define when the provider is unsafe or a poor fit. They are current package constraints.
+
+- **Only textual content decodes** — html/xhtml and `text/*` plus JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
+- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
+
+<a id="dev-note"></a>
+### Dev Note
+
+<details>
+<summary>Working context for maintainers — click to expand</summary>
+
+None.
+
+</details>

package/README.zh.md

@@ -0,0 +1,145 @@
+---
+description: "ctx.web 的匿名公共 HTTP(S) 抓取后端：部署方如何挂载有界、安全的 URL 抓取，含同源重定向与仅文本解码。"
+kind: "package-reference"
+---
+
+# @qilin-agent/web-fetch-http
+
+[English](README.md) | 中文
+
+## 概述
+
+有了 `qilin-web-fetch-http`，harness 可以通过 web 服务（`ctx.web`）抓取公共 HTTP(S) 页面，并在不发送凭据的情况下获得状态码与有界、解码后的内容。当组合需要 URL 校验、公开地址解析、连接固定、仅同源重定向、字节和字符上限及显式产品 `User-Agent` 时选择它。它把非 2xx 响应作为结果而非错误返回，并拒绝非公开目标、二进制数据与不受支持的内容类型。面向模型的 `web_fetch` 工具位于 `qilin-tool-web`，由它渲染本提供方的正文。
+
+## 目录
+
+- [使用本包](#use-this-package)
+- [理解实现](#understand-the-implementation)
+- [进一步探索](#further-exploration)
+- [模型体验](#model-experience)
+- [已知限制与延期工作](#known-limitations-and-deferred-work)
+- [开发备注](#dev-note)
+
+-----
+
+<a id="use-this-package"></a>
+## 使用本包
+
+在已加载 web 服务的组合中挂载本提供方；它以 `http` 抓取提供方身份注册，因此当它是唯一可用的抓取后端时，`ctx.web.fetch()` 会自动解析到它——也可以用 `fetchProvider: http` 固定。
+
+### 何时选择
+
+当部署必须以有界输出和安全传输抓取公共页面时选择此后端：不发送凭据，每个已解析地址必须是公共地址，每次连接都固定到已校验的地址集合，重定向无法逃出源站，每个响应都有上限。
+
+### 最小配置
+
+加载 web 服务与本提供方；可配置上限都有安全默认值，并在插件构造时验证，因此无效值会直接报错，而不是构造出上限荒谬的提供方。URL 安全上限固定为 2,048 个字符。
+
+```yaml
+- name: '@qilin-agent/web'
+- name: '@qilin-agent/web-fetch-http'
+```
+
+| 字段 | 默认值 | 含义 |
+|---|---|---|
+| `maxResponseBytes` | `5,000,000` | 响应主体最大字节数 |
+| `maxBodyChars` | `100,000` | 解码主体最大字符数 |
+| `timeoutMs` | `30,000` | 抓取超时——资源兜底，不是面向模型的工具预算 |
+| `maxRedirects` | `5` | 同源重定向最大跳数（`0` 表示不跟随） |
+| `userAgent` | `deepseek-harness/…` | 每次请求发送的 `User-Agent` 标头 |
+
+生成的[配置目录](../../../docs/config-catalog.zh.md#qilin-agentweb-fetch-http)是每个受支持字段及其 JSDoc 的穷尽式真源。
+
+### 抓取返回什么
+
+成功调用产生 `WebFetchResult`：允许的重定向之后的最终 URL、HTTP 状态码、分类为 `html` 或 `text` 的解码正文，以及 `truncated` 标志。非 2xx 响应是结果而非错误——状态码是被抓取资源状态的一部分；`WebError` 只用于无法安全获取或表示资源的失败。
+
+```text
+const page = await ctx.web.fetch({ url: 'https://example.com' })
+// page.body.kind === 'html' | 'text'; page.statusCode === 200 | 404 | ...
+```
+
+### 传输行为
+
+提供方保持请求匿名且有界：只接受不含内嵌凭据且不超过 2,048 个字符的 `http:` 与 `https:` URL。它只解析一次主机名；只要结果中有任何 IPv4 或 IPv6 地址不是公共单播地址，就拒绝整个结果，并把连接固定到已校验的地址集合。IPv6 检查会发现活动 DNS64 前缀，并拒绝指向非公开 IPv4 的转换地址。每次同源重定向都会重复解析与固定；跨源重定向会失败并要求重新调用。提供方还强制执行字节、字符、跳数和时间上限，拒绝不支持的内容类型，并发送显式产品 `User-Agent`。
+
+### 失败与恢复
+
+失败会抛出 `WebError`，其中包含可供程序路由的错误码：`WEB_INVALID_URL`、`WEB_BLOCKED_URL`、`WEB_FETCH_TOO_LARGE`、`WEB_FETCH_TIMEOUT`、`WEB_REDIRECT_BLOCKED`、`WEB_UNSUPPORTED_CONTENT_TYPE`、`WEB_ABORTED` 或 `WEB_PROVIDER_ERROR`。直接调用方可以按错误码路由；面向模型的 `web_fetch` 工具会在自己的错误包装层内把失败文本呈现给模型。
+
+-----
+
+<a id="understand-the-implementation"></a>
+## 理解实现
+
+<details>
+<summary>实现细节——点击展开</summary>
+
+本节解释提供方背后的设计决策；可观察行为已在[使用本包](#use-this-package)中完整说明。
+
+### 设计理念
+
+本包基于一项职责分离和一套分层超时机制：
+
+- **安全获取与呈现分离。** 本提供方拥有 URL 校验、公开地址强制规则、连接固定、HTTP 传输、重定向策略、上限、charset 解码与二进制拒绝；`qilin-tool-web` 拥有 HTML→markdown 与截断格式化。非 2xx 响应是数据，不是失败。
+- **两层超时。** 提供方的 `timeoutMs` 是直接 `ctx.web.fetch()` 调用方的资源兜底；面向模型的工具调用预算属于 `qilin-tool-call-timeout-policy`，由它触发 `exec.signal`。外层截止期限先到时，提供方报告 `WEB_ABORTED`，策略再以 `TOOL_TIMEOUT` 替换；因此 `WEB_FETCH_TIMEOUT` 标识的是提供方预算耗尽的直接服务调用方。
+
+### 源码地图
+
+| 文件 | 职责 |
+|---|---|
+| [`src/index.ts`](src/index.ts) | 插件入口：配置 schema、上限验证、提供方注册 |
+| [`src/provider.ts`](src/provider.ts) | `HttpFetchProvider`：固定连接、重定向跟随、有界读取、charset 解码 |
+| [`src/network.ts`](src/network.ts) | 公开地址解析、DNS64 发现与连接固定 |
+| [`src/policy.ts`](src/policy.ts) | URL 校验、同源检查、内容类型分类、charset 解析 |
+
+### 读取路径
+
+抓取先校验 URL，只解析一次主机名，结果中只要有非公开地址就拒绝，并把连接固定到已接受地址。每次同源重定向都重复该检查；跨源重定向或非公开目标在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码，并在字节上限内读取；解码后的文本再截断到字符上限。
+
+</details>
+
+-----
+
+<a id="further-exploration"></a>
+## 进一步探索
+
+当包级约定不够用时阅读以下页面。它们从共享词汇逐步进入服务、面向模型的工具与设计依据。
+
+- [web 子系统](../../../docs/subsystems/web.zh.md)——穷尽式的抓取请求／结果词汇与错误码。
+- [web 包映射](../README.zh.md)——六包家族与各角色。
+- [qilin-web](../web/README.zh.md)——本提供方注册进入的 web 服务。
+- [qilin-tool-web](../tool-web/README.zh.md)——渲染本提供方正文的面向模型 `web_fetch` 工具。
+- [生成配置目录](../../../docs/config-catalog.zh.md#qilin-agentweb-fetch-http)——每个受支持配置字段及其源声明。
+- [web 能力 seam 决策](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md)——搜索与抓取为何共用一项提供方选择服务。
+
+-----
+
+<a id="model-experience"></a>
+## 模型体验
+
+间接地，通过 `qilin-tool-web`：该工具把本提供方经 `maxBodyChars` 限制的解码文本或由 HTML 转换得到的 markdown 置于抓取结果包装层内，而重定向、标头与传输上限保持隐藏。
+
+#### KV Cache 影响
+
+不会直接导致 KV Cache 失效；请求前缀变更由上述消费方负责。
+
+## 已知限制与延期工作
+
+<a id="known-limitations-and-deferred-work"></a>
+
+
+这些限制说明提供方何时不安全或不合适。它们是当前包约束。
+
+- **只解码文本内容**——包括 html/xhtml 与 `text/*` 加 JSON/XML 家族；缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`，可提取文本的 PDF 解码属于明确的延期工作。
+- **charset 只来自 `Content-Type` 标头**（默认 UTF-8）——HTML `<meta charset>` 声明会被忽略；声明但无法识别的 charset 标签会抛出异常，而非回退。
+
+<a id="dev-note"></a>
+### 开发备注
+
+<details>
+<summary>维护者的工作上下文——点击展开</summary>
+
+无。
+
+</details>