dsh-read-url 0.4.8 → 0.4.9

package/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## [0.4.9] - 2026-08-20
+
+### 稳定性：配置健壮性 + 严格宿主兼容 + 资源泄漏
+
+**修复（真 bug）**
+- `cordis.patch.yml` 数字加引号（`timeoutMs: "15000"`）时按字符串拼接进超时预算（`"1500020000"`）→ 加载时统一强转数字并钳制到合理范围（timeoutMs 500–120000 等），非法值回退默认
+- 严格 cordis 宿主上 `ctx.get('web')` 对未注入服务直接抛错（同 `settings` 踩过的坑）→ 移入 try，降级为全局 fetch
+- 超时预算公式低估最坏链路：SPA 渲染实际是 goto 30s 上限 + DOM 轮询 10s = 40s，原公式只留 20s → 修正为 `timeoutMs + 45s`，且随配置缩放（60s 配置得 105s 预算，不再钳制）
+- `read_url_site` 预算 120s 覆盖不了 50 页 × 长超时的爬取波次 → 改为随配置缩放（`timeoutMs × 25` 波）
+- `crawlSite` 在协作超时触发后不退出循环（对已中止的 fetch 空转清空队列）→ while 增加 signal aborted 检查
+- 无 body 响应（204/304）导致 `res.body` 迭代抛 TypeError → 空响应防护返回空 buffer
+- offset 续读到文末时误报"登录墙 / SPA 页" → 识别文末，提示"offset 已到文末"
+- playwright 启动失败被永久缓存为 rejected promise（装好后也不重试）→ 失败自愈重置；`closeBrowser` 对未完成的 launch 健壮化
+
+**测试**：39 → **42 断言**（+最坏链路预算随配置缩放、yml 字符串强转+钳制、严格宿主 seam 降级）+ test-spa 10 全绿
+
+**明确不做**：Web UI 设置卡片——宿主侧注册成功但官方 UI 只渲染自带浏览器半边的插件（"A served namespace no card claims renders nothing"），为一个 6 字段低频配置维护 React 构建链不划算；配置走 `cordis.patch.yml` 已足够
+
 ## [0.4.8] - 2026-08-19
 
 ### 41 站多类型实测 + 提取修复

package/README.md

@@ -9,7 +9,7 @@
 
 URL reader plugin for DeepSeek Harness: fetch any webpage, **auto-detect encoding (GBK/GB2312/UTF-8/Big5)**, extract the clean main content, and return **token-efficient compact text or structured Markdown**.
 
-Zero dependencies (Node 20+ built-ins), no API key, no server side — install and use.
+Zero runtime dependencies (Node 20+ built-ins handle fetch/decode/extract), no API key, no server side — install and use.
 
 ## Why
 
@@ -149,19 +149,23 @@ chars 800+800/12398 · cached
 
 ### Configuration (optional)
 
-Plugin-level config is overridable via the profile's `cordis.patch.yml` (defaults in the plugin's own `cordis.patch.yml`):
+Override via the profile's `cordis.patch.yml` (defaults in the plugin's own `cordis.patch.yml`):
 
 ```yaml
 - id: dsh-read-url
   config:
-    timeoutMs: 15000      # per-request timeout
+    timeoutMs: 15000      # per-request timeout (500-120000, clamped)
     maxBytes: 3145728     # response body cap (bytes)
     maxChars: 6000        # default body truncation
     maxLinks: 20          # read_url_links default count
     spaRender: true       # SPA rendering enhancement (needs playwright installed; degrades with a hint otherwise)
     userAgent: '...'      # request UA
+    cacheTtlMs: 300000    # success-cache TTL
+    cacheMax: 32          # cache entry cap
 ```
 
+Values are coerced and clamped to sane ranges at load — quoted numbers in YAML work, garbage falls back to defaults.
+
 ### Output (compact)
 
 ```json
@@ -231,7 +235,7 @@ const results = await Promise.all([
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **39 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick) + **10 SPA-test assertions** all green;
+- **42 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation) + **10 SPA-test assertions** all green;
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap

package/README.zh.md

@@ -9,7 +9,7 @@
 
 DeepSeek Harness 的 URL 阅读插件：抓取任意网页，**自动识别编码（GBK/GB2312/UTF-8/Big5）**，提取干净正文，输出**省 token 的紧凑文本或结构化 Markdown**。
 
-零依赖（Node 20+ 内置能力），免 API key，免服务端，装完即用。
+核心零运行时依赖（Node 20+ 内置能力完成抓取/转码/提取），免 API key，免服务端，装完即用。
 
 ## 为什么做它
 
@@ -27,7 +27,7 @@ DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净
 | `ctx.effect` 卸载清理 | ✅ | ❌ | ❌ | ✅ |
 | 协作式超时（不暴露给模型） | ✅ | ⚠️ 自管 | ⚠️ 自管 | ✅ `timeoutMs` + `exec.signal` |
 | 模型视角输出 | 整页 Markdown | 紧凑文本 | 15 字段 JSON | **紧凑文本（无需解析 JSON）** |
-| 依赖 | 官方 | TS 需构建 | 零依赖 | 零依赖（JS ESM 即装即用） |
+| 依赖 | 官方 | TS 需构建 | 零依赖 | 核心零运行时依赖（JS ESM 即装即用） |
 | 反爬/降级响应（UA 与 TLS 指纹） | ⚠️ Node 默认 UA，实测 https 被中间设备按 TLS 指纹拦截、百度返回无热搜的降级版 | ❓ 未披露 | ❓ 未披露 | ✅ 完整浏览器 UA，实测获取完整版页面（百度热搜正常） |
 
 > 2026-08-16 实测（本机环境）：停用本插件后用官方 `web_fetch` 读 `https://www.baidu.com`——TLS 握手被中间设备按程序指纹拦截（退回 http 才成功），且百度对 Node UA 返回**服务端降级版**（热搜词条改由 JS 异步加载，静态 HTML 不含）；换回 `dsh-read-url` 后 https 正常、热搜完整可读。差异根因：请求的 UA 与 TLS 特征决定网站/中间设备是否按 bot 处理。
@@ -150,19 +150,23 @@ chars 800+800/12398 · cached
 
 ### 配置（可选）
 
-插件级配置通过 profile 的 `cordis.patch.yml` 覆盖（默认值见插件自带 `cordis.patch.yml`）：
+通过 profile 的 `cordis.patch.yml` 覆盖（默认值见插件自带 `cordis.patch.yml`）：
 
 ```yaml
 - id: dsh-read-url
   config:
-    timeoutMs: 15000      # 单请求超时
+    timeoutMs: 15000      # 单请求超时（500-120000，自动钳制）
     maxBytes: 3145728     # 响应体上限（字节）
     maxChars: 6000        # 默认正文截断
     maxLinks: 20          # read_url_links 默认条数
     spaRender: true       # SPA 渲染增强（需 playwright 已安装，未装自动降级提示）
     userAgent: '...'      # 请求 UA
+    cacheTtlMs: 300000    # 成功缓存 TTL
+    cacheMax: 32          # 缓存条目上限
 ```
 
+配置在加载时统一强转数字并钳制到合理范围——YAML 里加引号的数字也能用，非法值回退默认。
+
 ### 输出结构（紧凑）
 
 ```json
@@ -232,7 +236,7 @@ const results = await Promise.all([
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **39 个零依赖断言**（含实体解码、description 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、schema 预算、裸 main 提取）+ **10 个 SPA 测试断言**全绿；
+- **42 个单元断言**（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级）+ **10 个 SPA 测试断言**全绿；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap
@@ -245,7 +249,7 @@ const results = await Promise.all([
 ## 开发
 
 ```bash
-node test.mjs          # 零依赖自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离）
+node test.mjs          # 单元自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离/配置钳制）
 
 # SPA 渲染真实测试（需 playwright 已安装，未装自动 SKIP）
 node test-spa.mjs      # 10 断言：JS 正文/渲染后链接/工具不崩溃/缓存隔离

package/index.js

@@ -32,6 +32,8 @@ const DEFAULTS = {
   userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36',
 }
 
+// Session-level cache: url -> { time, full } on success, { time, error } on
+// failure (short TTL so a broken URL is not re-fetched in a loop).
 const cache = new Map()
 const decoders = new Map()
 
@@ -85,16 +87,21 @@ async function directFetch(url, signal, cfg) {
       signal,
       headers: { 'user-agent': cfg.userAgent, accept: 'text/html,application/xhtml+xml,*/*;q=0.8' },
     })
-    if (!res.ok) return { error: `HTTP ${res.status} ${res.statusText}` }
+    // Early-return paths must cancel the body stream, or the connection is
+    // held open undrained and cannot return to the keep-alive pool.
+    const discardBody = () => { if (res.body) res.body.cancel().catch(() => {}) }
+    if (!res.ok) { discardBody(); return { error: `HTTP ${res.status} ${res.statusText}` } }
     const contentType = res.headers.get('content-type') || ''
     if (contentType && !/text\/html|application\/xhtml|text\/plain/i.test(contentType)) {
+      discardBody()
       return { error: `Unsupported content-type: ${contentType.split(';')[0]}` }
     }
+    if (!res.body) return { buffer: Buffer.alloc(0), contentType, finalUrl: res.url }
     const chunks = []
     let size = 0
     for await (const chunk of res.body) {
       size += chunk.length
-      if (size > cfg.maxBytes) return { error: `Page exceeds ${cfg.maxBytes} bytes` }
+      if (size > cfg.maxBytes) { discardBody(); return { error: `Page exceeds ${cfg.maxBytes} bytes` } }
       chunks.push(chunk)
     }
     return { buffer: Buffer.concat(chunks), contentType, finalUrl: res.url }
@@ -178,9 +185,11 @@ async function fetchPage(url, externalSignal, cfg) {
 // Seam calls get the same cooperative timeout as fetchPage (a hanging
 // provider should not block the tool call forever).
 async function fetchViaWebSeam(ctx, url, externalSignal, cfg) {
-  const web = ctx && typeof ctx.get === 'function' ? ctx.get('web') : undefined
-  if (!web || typeof web.fetch !== 'function') return null
   try {
+    // ctx.get for a non-injected service throws on strict cordis hosts (seen
+    // with 'settings') — treat that as "seam absent" and fall through.
+    const web = ctx && typeof ctx.get === 'function' ? ctx.get('web') : undefined
+    if (!web || typeof web.fetch !== 'function') return null
     const timeoutSignal = AbortSignal.timeout(cfg.timeoutMs)
     const signal = externalSignal ? AbortSignal.any([externalSignal, timeoutSignal]) : timeoutSignal
     const res = await web.fetch(url, { signal })
@@ -655,9 +664,13 @@ function renderResult(value) {
   if (r.cached) lines.push('(cached)')
   if (r.rendered) lines.push('(rendered — JS 执行后内容)')
   if (!r.text) {
-    lines.push(r.spaHint
-      ? `(无可读内容 — ${r.spaHint})`
-      : '(无可读内容 — 登录墙 / SPA 页 / 空页面；SPA 需安装 playwright)')
+    if (r.charsStart > 0 && !r.truncated) {
+      lines.push('(offset 已到文末，无更多内容)')
+    } else {
+      lines.push(r.spaHint
+        ? `(无可读内容 — ${r.spaHint})`
+        : '(无可读内容 — 登录墙 / SPA 页 / 空页面；SPA 需安装 playwright)')
+    }
   }
   lines.push('', r.text || '')
   if (Array.isArray(r.links) && r.links.length) {
@@ -692,7 +705,8 @@ function readLinksTool(ctx, cfg) {
       },
       required: ['url'],
     },
-    timeoutMs: 20000,
+    // Covers SPA-render fallback: full fetch timeout + playwright render cap.
+    timeoutMs: Math.max(45000, cfg.timeoutMs + 45000),
     output: {
       schema: {
         type: 'object',
@@ -791,7 +805,8 @@ function readUrlBatchTool(ctx, cfg) {
       },
       required: ['urls'],
     },
-    timeoutMs: Math.max(30000, cfg.timeoutMs + 15000),
+    // Worst case: ceil(10/4)=3 concurrency waves, each (fetch + SPA render).
+    timeoutMs: Math.max(60000, cfg.timeoutMs * 3 + 135000),
     output: {
       schema: {
         type: 'object',
@@ -877,7 +892,9 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal) {
   const pages = []
   const failures = []
   const queue = [{ url: entryUrl, depth: 0 }]
-  while (queue.length && pages.length < maxPages) {
+  // Stop once the host's cooperative signal fires (tool budget hit): without
+  // this the loop keeps draining the queue on instantly-aborted fetches.
+  while (queue.length && pages.length < maxPages && !(externalSignal && externalSignal.aborted)) {
     // Never overshoot the page budget within a batch round.
     const batch = queue.splice(0, Math.min(2, maxPages - pages.length, queue.length))
     const results = await Promise.all(
@@ -957,7 +974,9 @@ function readUrlSiteTool(ctx, cfg) {
       },
       required: ['url'],
     },
-    timeoutMs: 120000,
+    // Worst case: ceil(maxPages/2)=25 waves of paired fetches (no SPA render
+    // here); the cooperative signal stops the loop early once the budget is hit.
+    timeoutMs: Math.max(120000, cfg.timeoutMs * 25),
     output: {
       schema: {
         type: 'object',
@@ -1002,7 +1021,22 @@ function readUrlSiteTool(ctx, cfg) {
 
 export function apply(ctx, config) {
   // Plugin-level config from cordis.patch.yml (config row), merged over defaults.
+  // YAML is free-form: numbers can arrive quoted as strings, which would poison
+  // arithmetic and AbortSignal timeouts downstream — coerce to numbers once
+  // here, clamp to sane ranges, and fall back to defaults on garbage.
   const cfg = { ...DEFAULTS, ...(config || {}) }
+  const clamp = (v, d, min, max) => {
+    const n = Number(v)
+    return Number.isFinite(n) ? Math.min(max, Math.max(min, n)) : d
+  }
+  cfg.timeoutMs = clamp(cfg.timeoutMs, DEFAULTS.timeoutMs, 500, 120000)
+  cfg.maxBytes = clamp(cfg.maxBytes, DEFAULTS.maxBytes, 65536, 16 * 1024 * 1024)
+  cfg.maxChars = clamp(cfg.maxChars, DEFAULTS.maxChars, 500, 20000)
+  cfg.maxLinks = clamp(cfg.maxLinks, DEFAULTS.maxLinks, 1, 50)
+  cfg.cacheTtlMs = clamp(cfg.cacheTtlMs, DEFAULTS.cacheTtlMs, 0, 3600000)
+  cfg.cacheMax = clamp(cfg.cacheMax, DEFAULTS.cacheMax, 1, 256)
+  if (typeof cfg.userAgent !== 'string' || !cfg.userAgent.trim()) cfg.userAgent = DEFAULTS.userAgent
+  if (typeof cfg.spaRender !== 'boolean') cfg.spaRender = DEFAULTS.spaRender
 
   // Temporal composability: unload must fully revert side effects.
   ctx.effect(() => () => {
@@ -1031,7 +1065,10 @@ export function apply(ctx, config) {
       },
       required: ['url'],
     },
-    timeoutMs: Math.max(5000, cfg.timeoutMs + 5000),
+    // Pipeline budget must cover the real worst case: full fetch timeout
+    // (cfg.timeoutMs) + SPA render (playwright goto 30s cap + DOM-stability
+    // poll 10s) + margin. Derived from load-time cfg — scale with it, never clamp it.
+    timeoutMs: Math.max(45000, cfg.timeoutMs + 45000),
     output: {
       schema: {
         type: 'object',

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-read-url",
-  "version": "0.4.8",
-  "description": "DeepSeek Harness URL reader: fetch any webpage, auto-detect encoding (GBK/GB2312/UTF-8/Big5), extract clean main content, output compact text/Markdown to save tokens. Zero runtime dependencies.",
+  "version": "0.4.9",
+  "description": "DeepSeek Harness URL reader: fetch any webpage, auto-detect encoding (GBK/GB2312/UTF-8/Big5), extract clean main content, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in.",
   "type": "module",
   "main": "index.js",
   "files": [

package/spa.js

@@ -20,8 +20,14 @@ export function looksLikeSpa(html) {
 
 async function getBrowser() {
   if (!browserPromise) {
-    const { chromium } = await import('playwright')
-    browserPromise = chromium.launch({ headless: true })
+    // A failed launch (playwright later installed / chromium download finished)
+    // must not stay cached as a rejected promise — reset so the next call retries.
+    browserPromise = import('playwright')
+      .then(({ chromium }) => chromium.launch({ headless: true }))
+      .catch((e) => {
+        browserPromise = null
+        throw e
+      })
   }
   return browserPromise
 }
@@ -73,9 +79,13 @@ export async function renderPage(url, externalSignal) {
 
 // Called on plugin unload (temporal composability): release the browser.
 export async function closeBrowser() {
-  if (browserPromise) {
-    const b = await browserPromise
+  if (!browserPromise) return
+  const p = browserPromise
+  browserPromise = null
+  try {
+    const b = await p
     await b.close().catch(() => {})
-    browserPromise = null
+  } catch {
+    // launch never completed — nothing to close
   }
 }