dsh-read-url 0.4.7

package/CHANGELOG.md

@@ -0,0 +1,260 @@
+# Changelog
+
+## [0.4.7] - 2026-08-19
+
+### 省 token：参数 schema 描述精简（固定开销 -9%）
+
+- **新增审查维度**：固定开销 = 工具 description + **参数 schema 描述**——参数 schema 合计 2048 字符，比工具 description（1000）还大一倍，此前从未专项精简
+- **精简内容**：4 个工具 14 个参数的 description 压缩冗余词（保留范围/默认/关键行为语义）；batch/site 工具 description 尾句瘦身
+- **量化**：固定开销总计 **3048 → 2779 字符（-9%）**，每次工具调用省 ~270 字符（≈70 token）
+- **防回归**：新增"参数 schema 总预算 <2000 字符"断言（与 description 预算互补）
+- **测试**：37 → **38 断言** + test-spa 10 全绿
+- 明确排除（因小失大）：blockMd 递归（5000 层实测无栈溢出）、render 元数据（已紧凑 ~9.6%）、正文提前截断（破坏提取语义）、remove lang（省 8 字符不值）
+
+## [0.4.6] - 2026-08-19
+
+### 日常维护：逻辑缺陷 + 效率 + 边界加固
+
+**修复（逻辑/边界）**
+- `raceFirstSuccess([])` 空数组永不 resolve（死锁）→ 立即返回 `{ success:false, failures:[] }`
+- `renderPage` 不响应外部取消：协作超时/模型取消后 playwright 仍跑满 poll 窗口 → goto 传 signal + poll 循环检查 aborted，取消立即释放浏览器（spa.js）
+- `extractLinks` 无迭代上限：导航重复链接上千次的页面会扫到 HTML 末尾 → 扫描上限 `limit*4`
+- `crawlSite` queue 无上限：海量链接站点内存膨胀 → queue 上限 `maxPages*20`
+
+**效率**
+- `fetchViaCurlProxy` 支持 `proxyOverride`：竞速路径已探测过代理，不再二次读 env/注册表
+
+**代码卫生**
+- 移除 `raceFirstSuccess` 上方重复注释（v0.4.5 修复残留）
+
+**验证**：36 → **37 断言**（+空竞速防挂起；修正 1 个 async 回调断言为顶层 await）+ test-spa 10 全绿；真实 e2e：SPA 渲染/竞速 1.3s/取消立即响应 全过；29 站回归 **18 OK / 4 预期 / 7 边界 / 0 崩溃**
+
+## [0.4.5] - 2026-08-18
+
+### 直连 + 代理并发竞速（海外站提速 ~94%）
+
+- **fetchPage 重构为双通道竞速**：有代理时同时发起直连 fetch 与代理 curl，`raceFirstSuccess` 取先完成且成功者，输者立即 abort（curl 进程 kill / fetch AbortController）——不再等待直连超时兜底
+- **实测收益**（本机真实环境）：BBC 中文 11s → **633ms**（-94%）、V2EX 11s → **782ms**（-93%）；新浪等国内站 182ms 零感知（直连赢）；DNS 失败 / w3c 403 归因完整保留
+- **代理探测前置**：`detectProxy()` 提前执行（有进程级缓存），无代理时退化为纯直连（行为与 v0.4.3 完全一致）
+- **失败归因不变**：双方均失败时仍返回"直连原因 + 已尝试代理 X（代理返回/未连通）"
+- **测试**：33 → **36 断言**（+raceFirstSuccess：成功优先/慢成功胜快失败/失败按序收集）；test-spa 10 全绿
+
+## [0.4.4] - 2026-08-18
+
+### 健壮性提升
+
+- **4 个工具 execute 空参防御**：`undefined`/`null` args 不再抛 TypeError（返回明确错误）——应对异常调用
+- **proxy-fallback 空响应防御**：curl 输出无元数据尾部时视为失败而非成功（防御异常响应）
+
+### 隐私说明（README 新增）
+
+- 明确声明：插件**绝不使用开发者的任何网络配置**——代理回退只读取**用户自己机器**的代理（环境变量或 Windows 系统代理）；无遥测、无统计、无数据收集，唯一对外动作是抓取用户指定读取的 URL
+
+### 测试
+
+- test.mjs 32 → **33 断言**（新增 4 工具空参容错断言）
+
+## [0.4.3] - 2026-08-18
+
+### 新增：直连失败自动代理回退（用户"默认直连，不行再走用户自己的代理"）
+
+- **新模块 `proxy-fallback.js`**：直连失败（连接类错误：被墙/拒绝/DNS/连接超时）→ 检测用户代理 → curl 经代理重试一次
+- **代理来源优先级**：环境变量 `HTTPS_PROXY/HTTP_PROXY` → **Windows 系统代理**（注册表 `HKCU Internet Settings`，Clash 等代理软件的真正落点；环境变量经常不存在）
+- **无感设计**：代理读取成功返回与直连完全一致的结果（模型无感知）；失败返回原始直连错误 + 明确归因（"已尝试代理 X，代理未连通 / 代理返回 HTTP 403"）
+- **不误伤**：插件自身超时（页面慢）不算连接失败，不触发代理；国内站直连成功零开销（新浪 230ms）
+- **零 npm 依赖**：调用系统 curl（Win10+/macOS/Linux 标配），`-x` 显式传代理；`--max-filesize` 单位修正为字节
+- 工具整体超时（非连接类）不触发代理回退
+
+### 验证（29 站实测）
+
+- **16 → 18 OK**：BBC 中文 / V2EX 从 ERR 变 OK（直连 10s 超时 → 自动代理 → 干净正文）
+- 海外站错误全部准确归因：Wikipedia 403（代理可达但反爬）、httpbin 503（服务端故障）、PDF 404、DNS 未连通、代理 403——模型能区分"服务端错误/反爬/网络边界"
+- 国内站零影响：新浪 230ms 直连；site crawl 5/5；offset 续读 / batch 隔离不变
+- test.mjs 32 + test-spa 10 断言全绿（新增 proxy-fallback 函数验证）
+
+## [0.4.2] - 2026-08-18
+
+### 性能
+
+- **looksLikeSpa 计数优化**：`match()` 建大数组 → `exec()` 循环计数（多 MB HTML 不再为每个 script 标签分配数组项）
+- **SPA 空页轮询提前退出**：DOM 稳定检测改为"连续两次一致即停"（含空 body）——空白页不再白等满 10 秒
+- **ctx.web seam 调用加协作超时**：官方 provider 挂起不再阻塞工具调用（与 fetchPage 同款 `timeoutMs` 约束）
+
+### 修复 / 省 token
+
+- **extractLinks URL 去重**：导航栏重复链接只保留一条（链接列表更紧凑，token 更省，覆盖不减）
+- **缓存键去除 URL fragment**：`#锚点` 不再分裂缓存（续读命中率提升）
+- **fetch 错误明确归因**：提取 undici `e.cause`——DNS 失败显示 `getaddrinfo ENOTFOUND`、被墙站显示超时，模型能区分"域名不存在"与"网络被阻断"，不再盲目重试；错误信息截断 ≤120 字符
+- **blockMd 表格分隔行转义修复**：含 `|` 的表头单元格不再让 `---` 分隔行带反斜杠
+- **batch render 链接截断**：每页 links 最多内联 6 条 + `…共 N 个`
+
+### 测试 / 验证
+
+- test.mjs 29 → **32 断言**（SPA 阈值 4/5 边界、链接去重、表格分隔行转义）
+- **multi-site.mjs 扩展 18 → 29 站**：新增门户（搜狐/凤凰）、SPA（少数派）、海外与边界站（BBC-中文/维基英文/httpbin 404·重定向·PNG/PDF 类型拦截/DNS 失败）
+- 29 站实测：**16 OK / 3 预期边界 / 10 环境网络边界 / 0 崩溃**（64s）；海外站 Node 直连被墙（curl 走代理可达）为环境限制非插件缺陷，错误信息现已明确归因；site crawl 阮一峰 5/5
+- **海外站专项验证（08-18）**：curl 走 7897 代理拉取 BBC 中文（427KB）/V2EX（107KB）真实 HTML 后，本地提取管线输出 4112 / 6422 字符干净正文——**提取逻辑对海外站完全正常**，失败根因仅为 Node fetch 直连被墙（`NODE_USE_ENV_PROXY` 在 Node 22/24 均实测无效，不引入代理，网络通道由 DSH 官方 ctx.web seam 负责）；Wikipedia 代理 IP 被限流（403 Too Many Reqs）
+
+## [0.4.1] - 2026-08-17
+
+### 省 token 复审（18 站实测驱动）
+
+- **4 个工具 description 精简 ~23%**（约 1290 → 990 字符）：`read_url` 主描述删冗余引导（引擎/缓存机制细节移出），links/batch/site 同步压缩——固定开销每次调用都在，这是最大单项
+- **实体解码扩展 6 → 45 个命名实体**：`&mdash;` `&hellip;` `&ensp;` `&ldquo;` `&copy;` `&deg;` `&times;` 等不再残留——残留实体既浪费 token 又显示为乱码
+- **SPA 安装提示精简**、batch render 文案中文化（`(no readable content)` → `(无可读内容)`）
+
+### 修复（18 站实测发现）
+
+- **SPA 渲染 `networkidle` 30s 超时**（qq-news 持续轮询永不空闲 → 渲染失败）——改为 `domcontentloaded` + DOM 稳定轮询（内容停止增长即收，上限 10s）；18 站全量测试 **1m51s → 54s**，qq-news EMPTY → OK
+
+### 测试 / 验证
+
+- test.mjs 27 → **29 断言**（+实体解码 7 项、+description 预算与静态性守卫）
+- **multi-site.mjs 入库（18 站可复跑）**：12 OK / 3 预期边界（知乎·微博登录墙、example.com 简单页）/ 3 网络·反爬边界（wikipedia 地域 403、w3c 对 Chrome UA 403、github TLS 指纹超时——均 curl 对照定性，非插件 bug）；offset 续读、batch 失败隔离、site 爬取 5/5 通过
+
+## [0.4.0] - 2026-08-16
+
+### 新增（Roadmap 收官）
+
+- **`read_url_site`** — 整站递归爬取（BFS）：从入口 URL 发现同域名页面，返回紧凑站点地图
+  - 只爬同域名；登录/API/静态资源路径自动跳过（`isNoiseUrl`）；URL 去重（去 fragment）
+  - `maxPages`（2–50，默认 15）/ `maxDepth`（1–5，默认 2）双重上限防 token 爆炸；**batch 轮次不超发页数预算**
+  - `includeContent` 附每页摘要（默认关，结构优先省 token）；输出缩进树 `[深度] 标题 (字符数) URL`
+  - 并发 2 对目标站友好；单页失败 `[失败]` 隔离不中断
+  - **不做 SPA 渲染**（批量轻量抓取优先速度，SPA 单读用 `read_url`）
+
+### 修复
+
+- `sameHost` 对裸 hostname 误判（`new URL(hostname)` 抛错 → 所有链接被当站外）——改为 `new URL(url).hostname === hostname`
+
+### 测试
+
+- test.mjs 23 → **27 断言**（站点爬取：范围/去重/噪音跳过/深度/maxPages 上限/失败隔离/includeContent/render）；真实站点验证：阮一峰博客 8/8 页树状输出
+
+## [0.3.4] - 2026-08-16
+
+### 省 token 特调（核心原则复审）+ 修复
+
+- **状态提示精简**：截断/续读/渲染/空内容提示全部收敛为一行短文本（`chars 6000/12000 — 截断，offset 续读`），删掉每次截断都重复输出的长英文引导语（每调用省 ~20 token，批量场景累积更多）；`spaHint` 安装指引同步精简；
+- **失败结果缓存（30s）**：坏 URL 不再触发重复 fetch 循环——第一次失败即入缓存，重复请求直接返回缓存错误（省网络 + 省模型等待 + 防重试风暴）；
+- **offset 越界保护**：`offset >= 全文长度` 时返回空而非重复开头（防模型续读错位）；
+- **`includeLinks` 严格布尔判断**：与 batch 对齐（`=== true`），字符串 `"false"` 不再被当真值；cacheKey 同步严格化；
+- 测试 21 → **23 断言**（offset 越界、错误缓存）；README 中英「省 token 设计」章节更新为 7 点（含双层缓存、批量共用缓存）。
+
+## [0.3.3] - 2026-08-16
+
+### 新增
+
+- **`read_url_batch`** — 批量读多个 URL（1–10 个）：并行（并发 4 防限流）、逐页复用 `read_url` 全部能力（编码识别/正文净化/SPA 渲染/会话缓存），合并成紧凑报告。**单页失败隔离**（`[失败]` + 原因），不影响其他页；每页带标题/字符数/cached 标记。
+
+### 测试
+
+- test.mjs 18 → **21 断言**（批量并行+错误隔离、缓存复用、URL 数上限 10）；
+- 真实站点批量验证：example.com + 腾讯（成功）+ 百度 404（失败隔离）输出正确；
+- test.mjs 内两处 fire-and-forget async 断言改为顶层 await（保证断言执行、退出码可信）。
+
+## [0.3.2] - 2026-08-16
+
+### 修复（playwright 就绪后的真实 SPA 验证发现的 bug）
+
+- **SPA 渲染等待不足**：`renderPage` 用 `networkidle` 后立即取 DOM，但 JS 异步渲染（setTimeout/fetch 回调）常在其后完成——拿到的是渲染前占位内容。现在 networkidle 后额外等待 `SETTLE_MS=1000ms` 让 SPA 完成绘制；
+- **`read_url_links` 崩溃**：SPA 兜底分支 `const links` 被重新赋值导致 `TypeError: Assignment to constant variable`（v0.3.1 引入）。改为 `let`；
+- **缓存键不含 `includeLinks`**：先读（无链接）再读（带链接）会命中旧缓存返回空链接。缓存键加入 links 变体；
+- **相对链接被丢弃**：`extractLinks` 只匹配 `https?://` 绝对链接，页面内相对路径（`/page2`、`../x`）全部丢失——套娃入口不完整。现支持 base URL resolve 成绝对链接。
+
+### 测试
+
+- 新增 **`test-spa.mjs`**：本地起 SPA 服务（骨架 HTML + JS 异步渲染正文和链接），验证渲染触发/JS 正文/渲染后链接提取/工具不崩溃/缓存隔离——**10 断言**（需 playwright，缺失时自动 SKIP）；
+- 原有 18 断言回归通过。
+
+## [0.3.1] - 2026-08-16
+
+### 修复（SPA 套娃链路补全）
+
+- `read_url` 的 `includeLinks` 改为**从渲染后的 DOM 提取链接**（之前用静态 HTML，SPA 页面渲染出正文却拿不到链接——"正文→链接→下一页"的套娃浏览在 SPA 上断链）；
+- `read_url_links` 增加 **SPA 兜底**：静态提取的链接 <3 条且页面疑似 SPA 时，自动渲染后重新提取（playwright 未装时优雅回退静态结果）；
+- 工具描述同步更新（注明 SPA 渲染支持）。
+
+### 测试
+
+- 18 个零依赖断言（新增 read_url_links SPA 降级用例）。
+
+## [0.3.0] - 2026-08-16
+
+### 新增（SPA 页面渲染增强，Roadmap 收官）
+
+- **可选 Playwright 渲染**：在 DSH profile 目录 `npm i playwright && npx playwright install chromium` 后自动启用——检测到正文为空且页面脚本密集（疑似 Vue/React 客户端渲染）时，自动用无头 Chromium 渲染后再提取，返回结果带 `rendered` 标记（模型可知内容来自 JS 执行）；未安装时优雅降级：返回明确安装提示、不报错，核心保持零依赖；
+- **SPA 检测启发式**：`<script>` 标签数量 ≥5 且正文提取 <200 字符 → 疑似 SPA（独立 `spa.js` 模块，`looksLikeSpa` 可测）；
+- **渲染生命周期合规**：浏览器实例模块级单例复用（多次调用不重复启动），插件卸载时经 `ctx.effect` 关闭；
+- 新配置项 `spaRender`（默认 `true`，可关闭）。
+
+### 测试
+
+- 17 个零依赖断言（新增 SPA 检测 / 正常页不误判 / 未装 playwright 优雅降级）；真实链路验证降级提示。
+
+## [0.2.3] - 2026-08-15
+
+### 新增（模型后续处理便捷性特调）
+
+- **`offset` 续读参数**：长文续读从指定字符偏移开始，不再重复返回前文；缓存改为按 url+mode 存全文，续读直接命中缓存切片（实测新浪 0+500 → 500+500，cached 且无重复）；
+- **render 消费引导**：截断提示明确"用 offset 或更大 maxChars 续读"；续读时显示 `chars 500+500/12199` 位置信息；
+- **空内容归因**：正文为空时提示"可能是登录墙 / JS 渲染 / 空页面"，避免模型对空结果误判。
+
+## [0.2.2] - 2026-08-15
+
+### 修复（12 站真实测试发现）
+
+- **HTML 注释残留**：`<!-- 注释 -->` 只删边界不删内容，注释文本（含广告位配置）混入正文（新浪 `-->` 残留）；
+- **HTML 实体未解码**：`&nbsp;` / `&quot;` / `&amp;` / 数字实体按原样输出（新浪 `&nbsp`、CSDN `&quot;` 残留）——新增 `decodeTextEntities`，在标签剥离后解码；
+- **多 article 聚合页**：博客园首页每篇文章一个 `<article>`，原先只取第一个（165 字符）——改为聚合所有 article 块（3,560 字符）。
+
+## [0.2.1] - 2026-08-15
+
+### 优化（DeepSeek 成本特调）
+
+- **KV Cache 友好**：工具 schema/description 静态化，不再嵌入配置值——配置变更不影响 prompt 前缀复用，KV 缓存持续命中（DeepSeek 缓存命中 token 约 1/10 价格）；
+- **render 元数据精简**：`siteName` 与域名相同时省略；
+- **缓存参数可配置**：`cacheTtlMs` / `cacheMax` 加入插件级配置。
+
+## [0.2.0] - 2026-08-15
+
+### 新增
+
+- **`read_url_links` 独立工具**：只返回页面链接清单（标题+URL），不返回正文，适合找来源/摸站点结构；
+- **插件级配置**：`timeoutMs` / `maxBytes` / `maxChars` / `maxLinks` / `userAgent` 可通过 profile 的 `cordis.patch.yml` 覆盖；
+- **可选正文提取增强**：安装 `@mozilla/readability` + `happy-dom` 后自动启用 Firefox Reader Mode 算法，未安装回退内置启发式（核心保持零依赖）。
+
+## [0.1.1] - 2026-08-15
+
+### 修复
+
+- **隐藏容器噪音（百度首页实测 bug）**：部分站点（如 baidu.com）把整块 CSS 存放在隐藏 `<textarea>` 中、以实体化形式（`&lt;style&gt;`）输出，导致正文混入 25~36 万字符 CSS/JS 噪音。修复：
+  - 删除 `<textarea>` / `<style>` / `<script>` / `<template>` / `<noscript>` 隐藏块；
+  - 实体化标签还原（`&lt;` → `<`、`&gt;` → `>`）后二次清理；
+  - `blockMd` / `inlineMd` 防御性跳过以上标签。
+- 修复后百度首页：text 模式 250,545 → 868 字符（-99.65%），markdown 模式 365,753 → 4,744 字符（-98.7%）。
+
+## [0.1.0] - 2026-08-15
+
+首个发布版本。URL 阅读插件：抓取网页并提取干净正文，面向 DSH 深度优化。
+
+### 功能
+
+- `read_url` 工具：URL → 干净正文（text / markdown 两种模式）
+- 中文编码自动检测：GBK / GB2312 / UTF-8 / Big5（三级探测 + 乱码回退）
+- 容器级正文净化：优先 `<article>` / `role="main"`，剥离导航/页脚/广告
+- 省 token：默认 6000 字符 + 段落级智能截断 + 紧凑文本 render
+- 会话级缓存（5 分钟 TTL，插件卸载自动清理）
+- 可选返回页面链接清单（`includeLinks`，上限 20 条）
+
+### 架构合规（DSH 官方文档）
+
+- 网络访问优先走 `ctx.web` 能力缝，缺失时回退全局 fetch
+- 缓存副作用注册于 `ctx.effect`，满足时间可组合性
+- 协作式工具调用超时（`timeoutMs` + `exec.signal`），不暴露给模型
+- 零运行时依赖（Node 20+ 内置能力），即装即用
+
+### 测试
+
+- 8 个零依赖单元断言（编码 / 提取 / Markdown / 截断）
+- 真实链路验证：example.com（EN）、qq.com（ZH 截断）、ctx.web seam 路径、缓存命中

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 dsh-read-url contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,228 @@
+# dsh-read-url
+
+🌐 **English** | [中文](README.zh.md)
+
+![dsh-read-url](docs/banner.svg)
+
+URL reader plugin for DeepSeek Harness: fetch any webpage, **auto-detect encoding (GBK/GB2312/UTF-8/Big5)**, extract the clean main content, and return **token-efficient compact text or structured Markdown**.
+
+Zero dependencies (Node 20+ built-ins), no API key, no server side — install and use.
+
+## Why
+
+DSH agents can search (getting links and snippets) but lack the step of "reading a URL into clean body text". The official `tool-web` `web_fetch` does a **whole-page turndown conversion** (nav/ads/sidebars all preserved) with a default cap of 200,000 characters — a token black hole. This plugin returns only what the model actually needs: **cleaned body + essential metadata**, truncated by default.
+
+### Competitor comparison (measured from source/docs, 2026-08-15)
+
+| Capability | Official `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
+|---|---|---|---|---|
+| Body cleaning (container-level) | ❌ whole page | ⚠️ tag-level, nav/footer leak in | ⚠️ custom, noisy | ✅ article/main containers + noise stripping |
+| Default output cap | 200,000 chars | 50,000 chars | 30,000 chars | **6,000 chars + paragraph-aligned truncation** |
+| Chinese GBK/GB2312 | provider-dependent | ⚠️ not normalized, GB2312 garbles | ❌ not handled | ✅ normalized + mojibake fallback |
+| Session-level cache | ❌ | ❌ | ❌ | ✅ 5-min TTL |
+| `ctx.web` seam | ✅ (official core) | ❌ global fetch | ❌ | ✅ seam-first, fallback included |
+| `ctx.effect` unload cleanup | ✅ | ❌ | ❌ | ✅ |
+| Cooperative timeout (hidden from model) | ✅ | ⚠️ self-managed | ⚠️ self-managed | ✅ `timeoutMs` + `exec.signal` |
+| Model-facing output | whole-page Markdown | compact text | 15-field JSON | **compact text (no JSON parsing)** |
+| Dependencies | official | TypeScript build | zero deps | zero deps (JS ESM, drop-in) |
+| Anti-bot / degraded responses (UA & TLS fingerprint) | ⚠️ Node default UA; measured: https intercepted by middlebox TLS fingerprinting, Baidu returns a degraded page without trending topics | ❓ not disclosed | ❓ not disclosed | ✅ full browser UA; measured: full page fetched (Baidu trending topics intact) |
+
+> Measured 2026-08-16 (local environment): with this plugin removed, the official `web_fetch` hitting `https://www.baidu.com` had its TLS handshake intercepted by a middlebox using program fingerprints (fell back to http to succeed), and Baidu returned a **server-side degraded page** (trending topics moved to JS loading, absent from static HTML). With `dsh-read-url` restored, https worked and trending topics were fully readable. Root cause: the request's UA and TLS characteristics decide whether sites/middleboxes treat you as a bot.
+
+## DSH architecture compliance
+
+Implemented per official docs (`docs/capability-seams.md`, `docs/cordis-primer.md`, `docs/tool-execution-pipeline.md`):
+
+1. **Web access via the `ctx.web` capability seam** — all web requests go through `ctx.web.fetch()` first (provider resolved inside the seam, same as official `tool-web`), falling back to global fetch when the seam is absent. The network layer is replaceable, not bound to any provider;
+2. **Reversible side effects** — the session cache is registered under `ctx.effect`, auto-cleared on plugin unload (temporal composability);
+3. **Cooperative tool-call timeout** — `ToolDefinition.timeoutMs` declares the budget, `execute(args, exec)` forwards `exec.signal` to fetch; the timeout policy is enforced by the pipeline, never exposed to the model;
+4. **Model-facing simplicity** — render emits compact text (`title:` header + body); the model consumes it directly with no JSON parsing. Defaults are the most token-efficient; structured output is opt-in.
+
+## Install
+
+```bash
+# From GitHub (recommended, easy updates)
+npx @deepseek-ai/dsh plugin --profile web add github:2672243194/dsh-read-url
+
+# Local development
+npx @deepseek-ai/dsh plugin --profile web add ./dsh-read-url
+```
+
+Restart DSH (Web/TUI); you should see `dsh-read-url` enabled in Settings → Plugins.
+
+## Usage
+
+Just talk to the agent:
+
+```
+Read https://example.com/article and summarize the key points
+Read https://docs.example.org/guide in markdown mode
+```
+
+### Tools
+
+**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body
+
+| Param | Type | Default | Description |
+|---|---|---|---|
+| `url` | string | required | http(s) URL |
+| `maxChars` | number | 6000 | Max body characters returned (500–20000) |
+| `offset` | number | 0 | Resume reading from this character offset (long-article continuation; served from cache without repeating earlier text) |
+| `mode` | string | `text` | `text` = plain (most token-efficient); `markdown` = structured |
+| `includeLinks` | boolean | `false` | Also return up to 20 page links (title+URL) |
+
+**`read_url_batch(urls, maxChars?, mode?, includeLinks?)`** — read multiple URLs (1–10) in parallel, each cleaned individually, merged into one compact report
+
+| Param | Type | Default | Description |
+|---|---|---|---|
+| `urls` | string[] | required | http(s) URL list (1–10) |
+| `maxChars` | number | 3000 | Max body characters per page (500–20000) |
+| `mode` | string | `text` | `text` = plain; `markdown` = structured |
+| `includeLinks` | boolean | `false` | Also return links per page (title+URL) |
+
+- Concurrency capped at 4 (avoids rate-limiting); a failing page is **isolated** (`[失败]` + reason in the output) and does not affect the others;
+- Reuses every `read_url` capability and the session cache (encoding, cleaning, SPA rendering, 5-min cache — repeat batches hit the cache).
+
+**`read_url_site(url, maxPages?, maxDepth?, includeContent?, maxCharsPerPage?)`** — recursive site crawl: BFS from the entry URL across same-host pages, returns a compact site map
+
+| Param | Type | Default | Description |
+|---|---|---|---|
+| `url` | string | required | http(s) entry URL |
+| `maxPages` | number | 15 | Max pages to crawl (2–50; bounds output) |
+| `maxDepth` | number | 2 | Max link depth from entry (1–5) |
+| `includeContent` | boolean | `false` | Attach a short body summary per page (default off — structure first, token-efficient) |
+| `maxCharsPerPage` | number | 500 | Summary length per page when includeContent=true (200–2000) |
+
+- **Same-host only**; login/API/static-asset paths are skipped; URLs deduped (fragment stripped);
+- Concurrency 2 (gentle on the target site); per-page failures recorded as `[失败]` without aborting;
+- Output is an indented tree: `[depth] title (chars) URL`;
+- **No SPA rendering here** (crawling favors speed/breadth) — use `read_url` for JS-only pages.
+
+**`read_url_links(url, limit?)`** — list the page's links without returning body text (lighter; good for sourcing / mapping a site)
+
+| Param | Type | Default | Description |
+|---|---|---|---|
+| `url` | string | required | http(s) URL |
+| `limit` | number | 20 | Max links returned (1–50) |
+
+### Configuration (optional)
+
+Plugin-level config is overridable via the profile's `cordis.patch.yml` (defaults in the plugin's own `cordis.patch.yml`):
+
+```yaml
+- id: dsh-read-url
+  config:
+    timeoutMs: 15000      # per-request timeout
+    maxBytes: 3145728     # response body cap (bytes)
+    maxChars: 6000        # default body truncation
+    maxLinks: 20          # read_url_links default count
+    spaRender: true       # SPA rendering enhancement (needs playwright installed; degrades with a hint otherwise)
+    userAgent: '...'      # request UA
+```
+
+### Output (compact)
+
+```json
+{
+  "url": "...",
+  "title": "...",
+  "siteName": "...",
+  "lang": "zh-CN",
+  "charset": "gbk",
+  "mode": "text",
+  "truncated": true,
+  "charsTotal": 12990,
+  "charsReturned": 6000,
+  "text": "...",
+  "links": []          // only when includeLinks=true
+}
+```
+
+### PTC mode
+
+Output is pure JSON and composable; orchestrate parallel multi-URL reads in PTC mode:
+
+```ts
+const results = await Promise.all([
+  read_url({ url: 'https://a.example.com', maxChars: 4000 }),
+  read_url({ url: 'https://b.example.com', maxChars: 4000 }),
+])
+```
+
+## Token economy (core)
+
+1. **Body text only by default** — no redundant headings/keywords/images/word-count fields; take them via params only when needed;
+2. **Paragraph-aligned truncation + offset continuation** — 6,000 chars by default (~3,000 tokens), cut at paragraph boundaries to keep semantics; output notes a single line `(chars 6000/12990 — truncated, continue via offset)`; resume starts at the given offset, sliced from cache — **no repetition of already-read text** (measured 0+500 → 500+500, no overlap); offset past the end returns empty instead of repeating the head;
+3. **`text` mode first** — Markdown structure is opt-in;
+4. **Compact text render** — the model sees a `title:` header + body directly, no JSON parsing; `siteName` is omitted when identical to the hostname; every status hint is one short line (truncated / cached / rendered), no verbose paragraphs;
+5. **Two-tier cache** — successful results cached per URL for 5 minutes (repeat reads hit cache: fewer network calls and fewer model retries); **failed results cached for 30 seconds** so a broken URL never triggers a re-fetch loop;
+6. **KV-cache friendly (DeepSeek cost tuning)** — tool schema/description stay **static text** (no config values embedded), so changing config never invalidates the reusable prompt prefix and KV cache keeps hitting. DeepSeek's cache-hit tokens cost about 1/10 of misses — the more stable the prefix, the cheaper the run (same analysis as the official `tool-web` docs);
+7. **Batch shares the cache** — `read_url_batch` reuses the same cache (repeat batches hit it directly) and caps each page at 3,000 chars (below the single-page 6,000) to bound total output;
+8. **Compact fixed cost** — the four tool descriptions total ~990 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `—`/`…` never waste tokens or render as mojibake.
+
+## Technical notes
+
+- **Encoding**: three-level detection (HTTP `Content-Type` charset → HTML meta → BOM), built-in `TextDecoder` transcoding (Node 20+ full-icu), GB2312 normalized to GBK, auto-fallback to UTF-8 on mojibake;
+- **Extraction**: prefers `<article>` / `role="main"`, strips `nav/footer/header/aside/form/iframe` and ad-like containers, heuristic fallback to `<body>`;
+- **Markdown**: self-written lightweight tag state machine (headings/paragraphs/lists/blockquotes/code/tables/inline bold-italic-links), zero deps;
+- **Safety**: http/https only; no page scripts executed; responses over 3 MB rejected; 15s timeout; structured errors (HTTP status / timeout / unsupported type / DNS cause such as `getaddrinfo ENOTFOUND` vs blocked-network timeout);
+- **Network fallback (proxy, raced)**: when a proxy is configured (env `HTTPS_PROXY`/`HTTP_PROXY`, falling back to the Windows system proxy registry where Clash-type apps persist it), the plugin starts the direct fetch and a proxy `curl` (`-x` passed explicitly, zero npm deps) **at the same time** and uses whichever completes first — so overseas sites that are blocked on direct connect are served through the user's own proxy in ~0.6s instead of waiting out the direct-connect timeout (~11s measured, -94%). The loser is aborted (curl killed / fetch aborted) and never enters the model context, so **token cost is unchanged**. A failed race returns the original direct error with the proxy attempt noted (`已尝试代理 …`); with no proxy configured the behaviour is exactly the plain direct connect;
+- **Privacy**: the plugin never uses the developer's network configuration — the proxy fallback only reads **your own machine's** proxy (env vars or Windows system proxy) at runtime. No telemetry, no analytics, no data collection: the only outbound action is fetching the URL you asked it to read;
+- **Optional enhancement 1 (Firefox Reader Mode algorithm)**: run `npm i @mozilla/readability happy-dom` in the DSH profile directory to auto-enable `@mozilla/readability` (MPL-2.0, referenced unmodified) for higher-quality extraction; falls back to the built-in heuristic when not installed — the core stays zero-dependency;
+- **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
+- **Boundaries**: login-walled pages are not readable; SPA pages need the Playwright enhancement; **structured data (e.g. which like-count belongs to which comment) is out of text-extraction scope** — this plugin flattens HTML into readable text, so exact field↔value associations are lost; for precise fields, intercept the page's actual data API (see "Real-world validation" below).
+
+## Real-world validation (2026-08-19, v0.4.6)
+
+29-site sweep driven by `multi-site.mjs` (committed, re-runnable): **18 OK / 4 expected boundaries / 7 network boundaries / 0 crashes** — overseas sites (BBC/V2EX) are served through the user's proxy in ~1s via the direct+proxy race; wikipedia/httpbin remain clear attributed network boundaries.
+
+| Category | Sites | Result |
+|---|---|---|
+| Portal navigation cleaning | Baidu / QQ / NetEase / Sina / Douban / CSDN / Sohu / Ifeng | ✅ clean text, no CSS noise |
+| SPA rendering | Bilibili / Xiaoheihe / Juejin / QQ News / SSPAI | ✅ `rendered` flag, post-JS body |
+| Multi-article aggregation | Cnblogs / Ruan Yifeng blog | ✅ 800+ chars across articles |
+| Static doc pages | MDN / Ruan Yifeng / example.com / GitHub | ✅ clean extraction (example.com = short page, expected) |
+| Login wall | Zhihu / Weibo | ✅ clear content or empty (expected) |
+| **Proxy fallback (overseas)** | BBC Chinese / V2EX | ✅ direct connect blocked → auto-retried through the user's system proxy → clean body |
+| Network / anti-bot boundary | W3C (403 for Chrome UA); Wikipedia (403 via proxy); httpbin (503 service-down); PDF (404); DNS-fail (proxy unreachable) | ✅ accurately attributed errors (HTTP status / 403 / 503 / ENOTFOUND) — not plugin defects |
+| offset continuation | Sina News (12,284 chars) | ✅ 800+800 seamless, cache hit |
+| Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
+| Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
+
+- **38 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget) + **10 SPA-test assertions** all green;
+- Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
+
+## Roadmap
+
+- [x] Single-page continuation (`offset` parameter)
+- [x] On-demand SPA rendering (optional Playwright enhancement, auto-enabled once the browser is installed)
+- [x] Batch reading (`read_url_batch`)
+- [x] Recursive site crawl (`read_url_site`)
+
+## Development
+
+```bash
+node test.mjs          # zero-dependency self-tests (charset/extract/markdown/truncate)
+node test-spa.mjs      # SPA rendering tests (10 assertions; SKIPs if playwright absent)
+node multi-site.mjs    # 29-site real-world sweep (needs network): portals/SPA/login-walls/static/anti-bot/net-boundaries
+
+# End-to-end (requires DSH CLI)
+npx @deepseek-ai/dsh plugin --profile headless add .        # run from the parent dir of the plugin
+npx @deepseek-ai/dsh --profile headless "use read_url to read https://example.com and output the title"
+```
+
+Verified against real DSH v0.1.0-rc.6: plugin loads, `read_url` registers, model calls it, real page content returned.
+
+## Support
+
+If dsh-read-url helps you, please give it a ⭐ Star on [GitHub](https://github.com/2672243194/dsh-read-url).
+
+- Completely free and open source (MIT): zero dependencies, no API key, fully local processing, no data collection;
+- Independently developed and maintained — your Star is the direct signal for whether I keep investing in it;
+- More users means more features — the next one might be exactly what you need.
+
+A Star costs nothing but helps this project go further. Thanks ⭐
+
+## License
+
+MIT