dsh-read-url 1.1.0 → 1.2.0

package/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## [1.2.0] - 2026-08-22
+
+### 修复三处隐患 + 进一步省 token
+
+秉持核心理念不变：零运行时依赖、只返回干净正文 + 关键元数据、静态 schema 保 KV-cache。
+
+**修复**
+- **raceFetch 错误归因去位置化**：`fetchPage` 不再假设 `failures[0]`=直连、`failures[1]`=代理，race 失败时由 `raceFetch` 在定义尝试顺序处附加 `label: 'direct'|'proxy'`，调用方按 label 取错——调整尝试顺序不会再静默归因错误
+- **失败缓存独立**：30s 失败条目原先与成功条目共用同一个 FIFO Map（占 `cacheMax=32` 名额，可能挤掉热点成功页）。拆为独立 `failCache`（容量 8），成功缓存容量完整保留
+- **smartTruncate 长段落退化**：offset 后首段超过 maxChars 时原先直接硬切 slice，丢失段落对齐承诺。现在先按句读（。！？!?. 换行）二次对齐，单句仍超长才硬切
+
+**省 token**
+- **JSON 紧凑渲染**：`JSON.stringify(x, null, 1)` 的逐层缩进改为无缩进紧凑输出，且超长字符串值（内嵌 HTML / base64，>1500 字符）截断并附 `…[+N chars]` 标记。典型 API 响应实测省约 21% 字符，大字段响应节省无上限
+- **状态行合并**：renderResult 的 `(chars …)`、`(cached)`、`(已自动拼接 N 页)`、`(rendered)` 多行括号提示合并为单行 ` · ` 分隔；`charset utf-8`（绝大多数响应）不再输出
+
+**效果增强**
+- **分页识别放宽**：NEXT_TEXT_RE 允许箭头包裹的变体（`下一页 ›`、`Next »`、`下一頁` 繁体等），仍保持全匹配保守策略，不做模糊猜测；findNextLink 锚点扫描加上限（400），巨型导航栏不再全文扫描
+
+### 同日第二轮：对抗性审查 + 多轮实网测试
+
+静态通读全部源码 + 对抗性探针（`probe.mjs`）+ 扩展类别实测（`extended-sites.mjs`，JSON API / 文档生成器站 / 学术 / raw 内容 / 韩文 euc-kr / 重定向等 24 类）+ 152 站三轮回归，发现并修复 6 个问题：
+
+**修复（含 1 个二次复杂度 DoS 向量）**
+- **对抗页面线性化（重要）**：正则标签扫描存在多类二次复杂度路径——无 `>` 配对的 `<` 尾部（每个位置重扫至串尾）、主正则属性组的组合回溯、无闭合标签的懒扫描（`<li>`/`<tr>`/feed `<item>`/`<article>`）、`[^>]+` 无界属性扫描。实测 60KB 恶意输入耗时 2.2s，3MB 上限页面可放大到分钟级。修复：`defuseLt`（中和必然无法成标签的尾部 `<`）、属性扫描统一加 `{0,1000}` 界、无闭合守卫（`includes('</li')` 类短路）、markdown 递归深度帽 100 层（防栈溢出）。修复后全部向量线性（输入翻倍耗时翻倍）
+- **散文 `&lt;` 内容丢失（长期存在）**：`revealEscapedTags` 无条件解码 `&lt;` 后，正文里 "a &lt; b" 的裸 `<` 被标签剥离器吞到下一个 `>`，丢一段正文。改为只揭示"标签形状"（后跟字母或 `/`）的 `&lt;`；裸 `&gt;`/`&quot;` 安全揭示
+- **代理路径重定向丢最终 URL**：curl 代理路径 `finalUrl` 硬编码原始 URL，跨域重定向后分页与链接解析落在错误主机。改用 `%{url_effective}`
+- **compactJson 深嵌套崩溃**：5000 层嵌套 JSON 触发原生 stringify 栈溢出直接抛异常（生产路径有 JSON.parse 兜底，PTC 直调会崩）。加 try/catch 降级为裁剪的字符串渲染
+- **unload 漏清 failCache**：v1.2.0 拆出的失败缓存未在插件卸载时清理，违反"unload 完全回退"承诺
+- **extractLinks 协议过滤**：`ftp:` 等非 web 协议链接原先能通过 URL 归一化进入链接列表与爬虫队列，现在只保留 http/https
+
+**验证**：88 单元断言（新增对抗输入限时回归、散文保留、深 JSON 降级）+ 12 SPA 断言全绿；对抗探针全部向量线性化（80k 恶意标签 <5s，修复前 80s+）；扩展类别 20/24 OK（失败均为环境边界：Stack Overflow/Wikipedia/doi 403 或超时）；152 站三轮回归 85/83/100 OK、0 崩溃、0 内容性回归，轮间抖动经交叉比对全部为环境性（反爬 403/503、连接超时），三连败站点均为已知边界（README 已记载的登录墙/反爬/境外不可达）
+
 ## [1.1.0] - 2026-08-22
 
 ### 适配 DSH 0.1.1-rc.2：并行工具调用声明

package/README.md

@@ -107,7 +107,7 @@ chars 800+800/12398 · cached
 
 ### Tools
 
-**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body. Handles HTML pages, JSON APIs (compact re-indented render) and RSS/Atom feeds (entry list + `feedCount`); paginated articles (novel/news/forum) are auto-joined up to `paginateMax` pages; page metadata (`published`, `author`) is harvested automatically — meta tags first, with a byline fallback from the body text when meta is absent
+**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body. Handles HTML pages, JSON APIs (fully compact render with long-value clipping) and RSS/Atom feeds (entry list + `feedCount`); paginated articles (novel/news/forum) are auto-joined up to `paginateMax` pages; page metadata (`published`, `author`) is harvested automatically — meta tags first, with a byline fallback from the body text when meta is absent
 
 | Param | Type | Default | Description |
 |---|---|---|---|
@@ -221,7 +221,7 @@ const results = await Promise.all([
 ## Technical notes
 
 - **Encoding**: BOM-first detection (UTF-8 / UTF-16LE / UTF-16BE — byte-level evidence beats any declared charset), then HTTP `Content-Type` charset → HTML meta; built-in `TextDecoder` transcoding (Node 20+ full-icu, so declared Shift-JIS/EUC-JP/GBK/Big5 all decode), GB2312 normalized to GBK, auto-fallback to UTF-8 on mojibake;
-- **Content dispatch**: a URL is not always an HTML page — JSON APIs are re-rendered compact (indent 1), RSS 2.0 / Atom feeds parse into an entry list (`title — url` + summary, `feedCount` field, full items via `includeLinks`; entry summaries are iteratively tag-stripped and entity-decoded until stable, so double-escaped `<a>` never leaks as a literal tag); XML sitemaps are rejected with a clear reason (no reading value); everything else falls through to the HTML pipeline;
+- **Content dispatch**: a URL is not always an HTML page — JSON APIs are re-rendered fully compact with no indentation, and string values over 1500 chars are clipped with a visible marker (since v1.2.0), RSS 2.0 / Atom feeds parse into an entry list (`title — url` + summary, `feedCount` field, full items via `includeLinks`; entry summaries are iteratively tag-stripped and entity-decoded until stable, so double-escaped `<a>` never leaks as a literal tag); XML sitemaps are rejected with a clear reason (no reading value); everything else falls through to the HTML pipeline;
 - **Extraction**: prefers `<article>` (all articles joined on aggregation pages; an unrelated tiny article — e.g. a newsletter card — falls through to `<main>` when its text is under 200 chars and `<main>` exists) / `<main>` / `role="main"`; `role="main"` containers are closed via **depth counting of balanced tags** (nested divs no longer cut the block at the first `</div>` — measured: gnu.org used to lose 7/8 of its body this way); strips `nav/footer/header/aside/form/iframe` and ad-like containers, heuristic fallback to `<body>`; on the body path a **text-density pass** drops link-dominated segments (related posts, category sidebars, hot-article widgets) — pages with standard containers never reach it;
 - **Pagination**: auto-follows `rel=next` or a short next-page anchor (下一页 / next / › / » — conservative, no fuzzy guessing); same-host only, loop-guarded, repeated paragraphs across the seam de-duplicated; continuation pages take the fast static path (a paginated SPA chain would cost one full render per page);
 - **Metadata**: `published` / `author` harvested from meta tags into the output and the status line; when the page has no such meta (ruanyifeng.com ships zero author/date tags) a **byline fallback** harvests the body head (600-char window: `作者：X / 日期：Y` — meta wins, deep-body mentions ignored, markdown link syntax stripped); empty-body pages (login walls, JS shells) fall back to `og:description` as a hint instead of nothing;
@@ -258,7 +258,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **79 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green;
+- **88 zero-dep assertions** (v1.2.0 adds: sentence-aligned truncation, compact JSON, pagination variants, adversarial-input time bounds, prose &lt; preservation, deep-JSON degradation) (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green;
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap
@@ -274,6 +274,8 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - [x] Bot-challenge defense (Cloudflare interstitial detection + auto-redirect wait + rejection, v1.0.0)
 - [x] Byline fallback for meta-less pages (author / published, v1.0.0)
 - [x] Parallel tool-call declaration (isConcurrencySafe; multi-source wall-clock time = slowest site, v1.1.0)
+- [x] Separate failure cache, label-based race attribution, sentence-aligned truncation, fully compact JSON render (~21% smaller), merged status flags, arrow-wrapped pagination variants (v1.2.0)
+- [x] Adversarial-page linearization (unclosed-tag/unpaired-lt/attribute-scan bounds + recursion depth cap, fixing a quadratic DoS vector), prose &lt; preservation, proxy-path redirects keep the final URL, deep-JSON degradation instead of crash (v1.2.0 round 2)
 
 > From v1.0.0 the plugin enters maintenance mode: bug fixes first, updates kept rare.
 

package/README.zh.md

@@ -221,7 +221,7 @@ const results = await Promise.all([
 ## 技术说明
 
 - **编码**：BOM 优先探测（UTF-8 / UTF-16LE / UTF-16BE——字节级证据优先于任何声明），其次 HTTP `Content-Type` charset → HTML meta；内置 `TextDecoder` 转码（Node 20+ full-icu，页面声明的 Shift-JIS/EUC-JP/GBK/Big5 均可正确解码），GB2312 归一为 GBK，检测到乱码自动回退 UTF-8；
-- **内容类型分发**：URL 不一定是 HTML——JSON 接口紧凑重排渲染（缩进 1 格），RSS 2.0 / Atom 订阅源解析为条目列表（`标题 — 链接` + 摘要，`feedCount` 字段，`includeLinks` 时附完整 items；条目摘要迭代「剥标签+解实体」直到稳定，双重转义的 `<a>` 不会漏成字面标签）；XML sitemap 明确拒绝（对模型无阅读价值）；其余全部走 HTML 管线；
+- **内容类型分发**：URL 不一定是 HTML——JSON 接口紧凑重排渲染（无缩进，超长字符串值 >1500 字符截断并标注，v1.2.0 起缩进 1 格改为完全紧凑），RSS 2.0 / Atom 订阅源解析为条目列表（`标题 — 链接` + 摘要，`feedCount` 字段，`includeLinks` 时附完整 items；条目摘要迭代「剥标签+解实体」直到稳定，双重转义的 `<a>` 不会漏成字面标签）；XML sitemap 明确拒绝（对模型无阅读价值）；其余全部走 HTML 管线；
 - **正文提取**：优先 `<article>`（聚合页多篇合并；无关小卡片 article——如订阅挂件——文本不足 200 字符且页面有 `<main>` 时自动回落 main）/ `<main>` / `role="main"`，`role="main"` 容器用**深度计数找平衡闭合标签**（嵌套 div 不会在第一个 `</div>` 被截断——实测 gnu.org 曾因此只取到 1/8 正文）；剥离 `nav/footer/header/aside/form/iframe` 及广告类容器，启发式回归到 `<body>`；body 路径上追加**文本密度过滤**——丢弃链接主导的短块（相关推荐/分类侧栏/热门文章挂件），标准容器页面完全不走此路径；
 - **分页拼接**：识别 `rel=next`（标准）或纯「下一页 / next / › / »」短锚文本（刻意保守，不做模糊猜测）；同域限定 + 防环；跨页重复段落自动去重；续页走静态快路径（分页 SPA 链每页一次完整渲染不划算）；
 - **元数据**：`published` / `author` 从 meta 标签提取进输出字段与状态行；页面无相关 meta 时（如阮一峰博客零 meta 标签）从**正文头部 600 字符的署名行兜底**（「作者：X / 日期：2026年8月21日」，meta 优先、正文深部提及不误采、markdown 链接语法不泄漏）；空正文页（登录墙/JS 壳）回落到 `og:description` 作为提示，不再返回空；
@@ -258,7 +258,7 @@ const results = await Promise.all([
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **79 个单元断言**（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；
+- **88 个单元断言**（v1.2.0 新增：长段落句级对齐、紧凑 JSON、分页变体、对抗输入限时、散文 &lt; 保留、深 JSON 降级）（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap
@@ -274,6 +274,8 @@ const results = await Promise.all([
 - [x] 人机验证页防御（Cloudflare 指纹页识别 + 等待自动跳转 + 拒绝误采，v1.0.0）
 - [x] 无 meta 页面署名行兜底（author / published，v1.0.0）
 - [x] 并行工具调用声明（isConcurrencySafe，多源阅读墙钟时间按最慢一站计，v1.1.0）
+- [x] 失败缓存独立（不再挤占成功缓存名额）、race 错误按 label 归因、长段落句级对齐截断、JSON 完全紧凑渲染 + 长值截断（约省 21%）、状态行合并、分页箭头变体（v1.2.0）
+- [x] 对抗页面线性化（无配对尖括号/无闭合标签/属性扫描加界 + 递归深度帽，修复 60KB→2.2s 的二次复杂度 DoS 向量）、散文 &lt; 不再被吞、代理重定向保留最终 URL、深 JSON 降级不崩溃（v1.2.0 第二轮）
 
 > v1.0.0 起进入维护期：以修 bug 为主，减少更新频率。