dsh-read-url 1.0.0 → 1.1.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## [1.1.0] - 2026-08-22
+
+### 适配 DSH 0.1.1-rc.2：并行工具调用声明
+
+对 `@deepseek-ai/dsh` 0.1.0-rc.8 → 0.1.1-rc.2 做了逐包静态对比（dsh-tools / dsh-tool-web / dsh-base / dsh-headless / dsh-tool-call-timeout-policy / dsh-agent-loop 的 lib 逐字节一致，唯一变化是 dsh-tool-cordis 新增 host 侧 API——authorization flows、session projections host-only units、image attachment policy、webserver index injections，全部为增量，无删除），插件所依赖的 API 面（`ctx.tools.register` / `ctx.effect` / `ctx.web` seam / `ToolDefinition.timeoutMs` / `exec.signal`）**零破坏性变化**，cordis 核心 4.0.1 两版一致。
+
+**新增：`isConcurrencySafe` 并行声明（4 个工具全部启用）**
+- DSH agent loop 会把连续的并行安全工具调用归入有界滚动池——此前 read_url 系工具未声明，即使模型同时发多个请求也被逐个串行执行
+- 全部共享状态审查确认交换性安全：会话缓存 Map（get/set 原子，重叠 miss 最多重复抓一次）、解码器缓存（TextDecoder 无状态）、readability 延迟 import（幂等）、SPA 渲染（共享 browser 单例 + 每次调用独立 `newPage`，page 间隔离）、系统代理检测缓存（读环境变量，双写同值）、爬虫去重集合（调用局部）
+- 效果：多源阅读任务（一次读 3–5 个 URL 对比观点）由串行变并行，墙钟时间按最慢一站计
+- 新增单测：4 工具声明锁定 + 6 路并发 read_url 缓存竞态烟雾测试（每个结果必须携带自己的正文，绝不串扰）
+
+**验证**：79 单元断言（+2）+ 12 SPA 断言全绿；新版 DSH `--dump-config` 配置树正常解析；headless 端到端实测 read_url 真实调用成功（example.com 返回正确标题）。
+
 ## [1.0.0] - 2026-08-21
 
 ### 量大管饱的持久版本：一次把「读网页」这件事做全

package/README.md

@@ -41,7 +41,8 @@ Implemented per official docs (`docs/capability-seams.md`, `docs/cordis-primer.m
 1. **Web access via the `ctx.web` capability seam** — all web requests go through `ctx.web.fetch()` first (provider resolved inside the seam, same as official `tool-web`), falling back to global fetch when the seam is absent. The network layer is replaceable, not bound to any provider;
 2. **Reversible side effects** — the session cache is registered under `ctx.effect`, auto-cleared on plugin unload (temporal composability);
 3. **Cooperative tool-call timeout** — `ToolDefinition.timeoutMs` declares the budget, `execute(args, exec)` forwards `exec.signal` to fetch; the timeout policy is enforced by the pipeline, never exposed to the model;
-4. **Model-facing simplicity** — render emits compact text (`title:` header + body); the model consumes it directly with no JSON parsing. Defaults are the most token-efficient; structured output is opt-in.
+4. **Model-facing simplicity** — render emits compact text (`title:` header + body); the model consumes it directly with no JSON parsing. Defaults are the most token-efficient; structured output is opt-in;
+5. **Parallel tool calls** — all four tools declare `isConcurrencySafe` (since v1.1.0): every piece of shared state (session cache, decoder cache, per-render isolated browser pages) is commutative-safe, so the agent loop may fan out several read_url calls into one parallel group — the wall-clock time of a multi-source reading task is that of the slowest site, not the sum.
 
 ## Install
 
@@ -232,7 +233,7 @@ const results = await Promise.all([
 - **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered) or a JS-redirect shell (few scripts but an empty `<body>`), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); the rendered text is accepted only when it meaningfully beats the static one (≥20 chars from an empty static extraction — short real content is no longer rejected); **bot-challenge defense** — Cloudflare-style interstitials ("Just a moment...") get an extra 8s poll for the auto-redirect, and if the page still looks like a challenge the rendered result is rejected and the static body kept (an interstitial can carry MORE text than the real body — measured 259 vs 135 chars — and must never be mistaken for an improvement); the interstitial DOM never feeds pagination/link extraction either; rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
 - **Boundaries**: login-walled pages are not readable; SPA pages need the Playwright enhancement; **structured data (e.g. which like-count belongs to which comment) is out of text-extraction scope** — this plugin flattens HTML into readable text, so exact field↔value associations are lost; for precise fields, intercept the page's actual data API (see "Real-world validation" below).
 
-## Real-world validation (2026-08-21, v1.0.0)
+## Real-world validation (2026-08-21, v1.0.0; re-verified 2026-08-22 against DSH 0.1.1-rc.2)
 
 152-site sweep driven by `multi-site.mjs` (committed, re-runnable, 8-way concurrent): **115 OK / 17 expected boundaries (login-walls·captcha·short static pages) / 20 attributed network·anti-bot errors / 0 crashes** (final round with all pre-release fixes in; network-class errors vary ±5 between runs, all environment-attributed). Coverage: CN portals/media, e-commerce (JD/Taobao/Pinduoduo/Suning/Dangdang), video (Bilibili/iQiyi/Youku/Mango), music, games, novels (Qidian/Zongheng/JJWXC legacy GBK), Q&A/forums, government, universities (Tsinghua/PKU/Fudan/SJTU…), Traditional-Chinese TW/HK (PTT/LTN/UDN), JP/KR portals (Yahoo JP/Hatena/goo/naver/daum), overseas tech docs (GitHub/dev.to/react.dev/nodejs.org/rust/go/python), feeds, JSON APIs, encoding stress (GBK/GB2312/Big5/gb18030), anti-bot & network boundaries. All errors are environment-attributed (Wikipedia/Reddit/UDN connect-timeout; W3C/Tieba/NGA/StackOverflow 403; BUPT 412; DNS-fail…) — every one returned as a structured, correctly-attributed error, never a crash.
 
@@ -257,7 +258,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **77 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection) + **12 SPA-test assertions** all green;
+- **79 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green;
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap
@@ -272,6 +273,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - [x] Text-density fallback + metadata harvesting (v1.0.0)
 - [x] Bot-challenge defense (Cloudflare interstitial detection + auto-redirect wait + rejection, v1.0.0)
 - [x] Byline fallback for meta-less pages (author / published, v1.0.0)
+- [x] Parallel tool-call declaration (isConcurrencySafe; multi-source wall-clock time = slowest site, v1.1.0)
 
 > From v1.0.0 the plugin enters maintenance mode: bug fixes first, updates kept rare.
 

package/README.zh.md

@@ -41,7 +41,8 @@ DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净
 1. **网络访问走 `ctx.web` 能力缝**——所有 web 访问优先通过 `ctx.web.fetch()`（seam 内解析 provider，与官方 `tool-web` 一致），seam 缺失时回退全局 fetch。网络层可替换，不绑定任何具体 provider；
 2. **可逆副作用**——会话缓存注册在 `ctx.effect` 下，插件卸载即自动清理（时间可组合性）；
 3. **协作式工具调用超时**——`ToolDefinition.timeoutMs` 声明预算，`execute(args, exec)` 把 `exec.signal` 转发给 fetch，超时策略由管线强制执行，不把超时暴露给模型；
-4. **模型视角精简**——render 输出紧凑文本（`title:` 头部 + 正文），模型直接消费，无需解析 JSON；默认参数最省 token，结构化能力按需开启。
+4. **模型视角精简**——render 输出紧凑文本（`title:` 头部 + 正文），模型直接消费，无需解析 JSON；默认参数最省 token，结构化能力按需开启；
+5. **并行工具调用**——4 个工具全部声明 `isConcurrencySafe`（v1.1.0 起）：共享状态（会话缓存 / 解码器缓存 / SPA 浏览器 page 隔离）均满足交换性安全，agent loop 可把多个 read_url 调用归入并行池——多源阅读任务的墙钟时间按最慢一站计，而非各站之和。
 
 ## 安装
 
@@ -232,7 +233,7 @@ const results = await Promise.all([
 - **可选增强二（SPA 页面渲染）**：在 DSH profile 目录执行 `npm i playwright && npx playwright install chromium` 后自动启用。检测到正文为空且（页面脚本密集 疑似 Vue/React 客户端渲染，或 body 为空的 JS 跳转壳——script 数不多但正文全靠跳转）时，自动用无头 Chromium 渲染后再提取（`rendered` 标记告知模型）；渲染结果仅在**显著优于**静态提取时采用（静态为空时 ≥20 字符即接受，防短正文被拒）；**人机验证页防御**——Cloudflare「Just a moment...」等指纹页识别后额外轮询 8s 供其自动跳转，仍未通过则拒绝渲染结果、保留静态正文（指纹页文字量可能超过真实正文，实测 259 vs 135 字符，绝不当作提升），指纹页 DOM 也不污染分页/链接提取；渲染采用 `domcontentloaded` + **DOM 稳定轮询**（内容停止增长即收，上限 10s）而非 `networkidle`——心跳轮询站永不空闲，避免 30s 超时；未安装时优雅提示安装方法、不报错——核心保持零依赖；
 - **边界**：登录墙页面无法读取；SPA 页面需安装 Playwright 增强后渲染读取（未安装时返回明确提示）；**结构化数据（如评论的点赞数归属、榜单数值）不在文本提取范围**——本插件把 HTML 扁平化为可读文本，字段与数值的精确对应关系会丢失；需要精确字段时，用 Playwright 拦截页面实际调用的数据 API 获取（见下方「真实世界验证」）。
 
-## 真实世界验证（2026-08-21，v1.0.0）
+## 真实世界验证（2026-08-21，v1.0.0；2026-08-22 适配 DSH 0.1.1-rc.2 复验）
 
 152 站全量实测（`multi-site.mjs` 已提交可复跑，8 并发）：**115 OK / 17 预期边界（登录墙·验证页·静态小页） / 20 网络·反爬归因错误 / 0 崩溃**（含全部发布前修复的终态轮；网络类错误逐轮有 ±5 波动，均为环境归因）。覆盖国内门户 / 媒体 / 电商（京东·淘宝·拼多多·苏宁·当当）/ 视频（B 站·爱奇艺·优酷·芒果）/ 音乐 / 游戏 / 小说（起点·纵横·晋江 legacy GBK）/ 问答 / 论坛 / 政府 / 高校（清北复交等 8 所）/ 港台繁体（PTT·自由时报·联合报）/ 日韩（Yahoo JP·Hatena·goo·naver·daum）/ 海外技术站（GitHub·dev.to·react.dev·nodejs·rust·go·python docs）/ 订阅源 / JSON API / 编码压力（GBK·GB2312·Big5·gb18030）/ 反爬与网络边界。错误全部环境归因（维基/Reddit/UDN 连接超时；W3C/贴吧/NGA/StackOverflow 403；北邮 412；DNS 失败等）——每一个都返回结构化、准确归因的错误，无一崩溃。
 
@@ -257,7 +258,7 @@ const results = await Promise.all([
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **77 个单元断言**（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别）+ **12 个 SPA 测试断言**全绿；
+- **79 个单元断言**（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap
@@ -272,6 +273,7 @@ const results = await Promise.all([
 - [x] 文本密度兜底 + 元数据提取（v1.0.0）
 - [x] 人机验证页防御（Cloudflare 指纹页识别 + 等待自动跳转 + 拒绝误采，v1.0.0）
 - [x] 无 meta 页面署名行兜底（author / published，v1.0.0）
+- [x] 并行工具调用声明（isConcurrencySafe，多源阅读墙钟时间按最慢一站计，v1.1.0）
 
 > v1.0.0 起进入维护期：以修 bug 为主，减少更新频率。
 

package/index.js

@@ -1105,6 +1105,7 @@ function readLinksTool(ctx, cfg) {
     },
     // Covers SPA-render fallback: full fetch timeout + playwright render cap.
     timeoutMs: Math.max(45000, cfg.timeoutMs + 45000),
+    isConcurrencySafe: () => true,
     output: {
       schema: {
         type: 'object',
@@ -1209,6 +1210,9 @@ function readUrlBatchTool(ctx, cfg) {
     },
     // Worst case: ceil(10/4)=3 concurrency waves, each (fetch + SPA render).
     timeoutMs: Math.max(60000, cfg.timeoutMs * 3 + 135000),
+    // Concurrent batch calls multiply fetch pressure but stay isolated (each
+    // call owns its result array; the session cache is commutative).
+    isConcurrencySafe: () => true,
     output: {
       schema: {
         type: 'object',
@@ -1379,6 +1383,10 @@ function readUrlSiteTool(ctx, cfg) {
     // Worst case: ceil(maxPages/2)=25 waves of paired fetches (no SPA render
     // here); the cooperative signal stops the loop early once the budget is hit.
     timeoutMs: Math.max(120000, cfg.timeoutMs * 25),
+    // Crawl state (visited set, queue) is call-local; only the shared session
+    // cache is global and commutative — parallel crawls of different hosts
+    // are safe and useful (one task mapping several sites).
+    isConcurrencySafe: () => true,
     output: {
       schema: {
         type: 'object',
@@ -1474,6 +1482,11 @@ export function apply(ctx, config) {
     // continuation fetches (static, no render). Derived from load-time cfg —
     // scale with it, never clamp it.
     timeoutMs: Math.max(45000, cfg.timeoutMs * (cfg.paginate === false ? 1 : cfg.paginateMax + 1) + 45000),
+    // All shared state is commutative-safe under concurrent calls (session
+    // cache Map get/set, decoder cache, one shared browser with isolated
+    // pages per render), so the agent may fan out several read_url calls in
+    // one parallel group — a real win for multi-source reading tasks.
+    isConcurrencySafe: () => true,
     output: {
       schema: {
         type: 'object',

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in.",
   "type": "module",
   "main": "index.js",