dsh-read-url 1.3.1 → 1.6.1

package/CHANGELOG.md

@@ -1,6 +1,158 @@
 # Changelog
 
-## [1.3.0] - 2026-08-24
+## [1.6.1] - 2026-08-30
+
+> 迭代 3（效率与缓存轮）：缓存 FIFO → LRU、分页/跟随总时间预算、全量收官回归。零 schema 变更。
+
+### 改进
+
+- **成功/失败缓存 FIFO → LRU**：原实现按插入序驱逐，32 条容量下「老但热」的条目（长会话中反复读取的同一页）会被后来者挤掉触发重复 fetch。命中即重插为最新槽（Map delete+set），热点页面在 TTL 内永不被驱逐；failCache 同样处理（反复失败的坏 URL 也不会在 TTL 内被挤出导致重复请求）。
+- **meta-refresh 跟随链总预算**：每跳自带 fetch 超时，3 跳最坏 3×timeout——此前未计入工具超时公式（该公式只预留 paginateMax+1 次 fetch + 渲染余量）。整条跟随链共享一个 timeout 预算，超限即停止跟随（保留当前 HTML，fail-open）。
+- **分页链总时间预算**：while 循环增加墙钟守卫 `startedAt + timeoutMs × (paginateMax+1)`——首页或 SPA 渲染变慢时，分页链自动让出预算而不是顶穿工具超时（顶穿意味着整个调用被宿主杀掉、结果全丢；现在至少返回已拼好的部分）。
+
+### 验证
+
+- 单元断言 **175 → 176**（+1：LRU 驱逐语义——填满 3 条、命中最老条 a 提升、加入 d 驱逐次老条 b、b 重新 fetch 而 a 仍命中；测试内通过插件自身 dispose 清空共享缓存保证确定性）+ 12 SPA + probe 47 PASS 零 THROW；
+- 收官全量回归：152 站扫描 + site-types 语义探针 + 对抗 probe（见下轮备注/日志）。
+
+## [1.6.0] - 2026-08-30
+
+> 迭代 2（站点类型扩展轮）：AMP/响应式图片、纯文本家族原生路径、非标准 JSON 消毒。零 schema 变更。
+
+### 新增
+
+- **AMP 页面图片**：AMP HTML 用 `<amp-img>` 代替 `<img>`，markdown 模式图片此前全丢——图片预处理同等对待 amp-img（alt/src/data-src 同一套规则）。
+- **`<picture>/<source srcset>` 回退**：响应式布局把真实图片 URL 放在 `<source srcset>`、`<img>` 只留 data: 占位时，首个 source URL 注入 img 兜底；img 自带可用 src 的 picture 不动。
+- **纯文本家族原生路径**：`text/plain` / `text/markdown` / `text/csv` 且前 5000 字符无 HTML 标签形状时不再走 HTML 管线（那会把每行压平成一段）——行结构原样保留（CRLF 归一、行尾空白清理、3+ 空行折叠）。.txt 日志、.md 文档、robots.txt、CSV 数据集逐行可读。被错误标成 text/plain 的真 HTML 页仍走 HTML 管线。
+- **JSON NaN/Infinity 消毒**：JS 手写序列化产物含 `NaN`/`Infinity`（非法 JSON）时 `JSON.parse` 整页失败、退化成 HTML 管线输出原始转储。解析失败后以字面量置 null 重试一次（仅在已非法的 JSON 上应用）。
+
+### 验证
+
+- 单元断言 **170 → 175**（+5：amp-img、picture 注入、picture 不动、text/plain 行结构 + 误标 HTML 回归、NaN JSON）+ 12 SPA + probe 47 全绿；
+- site-types.mjs **12 类 10/12 语义通过**（2 例境外连接超时）：text/markdown（155 行结构保留）、robots.txt（164 行）实网验证；WordPress feed / `<time>` 博客 / 懒加载图片复验通过。
+
+## [1.5.2] - 2026-08-30
+
+> 迭代 1（底层管线审查轮）：fetchPage/重试/竞速/解码/spa/爬虫链路的例行审查，2 个真 bug + 2 处健壮性，零 schema 变更。
+
+### 修复
+
+- **双次 429 形状崩溃（P0）**：`directFetchOnce` 在任意一次 429/503+Retry-After 都返回 `{ retryAfterMs }` 形状；`directFetch` 对第二次结果不检查原样返回——无代理路径绕过 raceFirstSuccess 的形状过滤，`decodeBuffer(undefined)` 在 `buffer[0]` 处裸抛 TypeError。现在重试预算耗尽时返回明确的 rate-limited error；等待 Retry-After 期间被 abort 则返回 cancelled（原先仍会补打一次请求）。`directFetch` 导出供测试直测（本机 Windows 系统代理无法通过环境变量屏蔽，无代理路径无法端到端复现）。
+- **爬虫不跟随 meta-refresh 壳页（P0）**：`crawlSite` 直连 fetchPage——read_url 已跟随跳板壳，read_url_site 却把壳页当正文爬（无标题跳板文本 + 永远到不了真实内容）。爬虫页处理接入 `followMetaRefresh`（fail-open），提取与出站链接均基于跟随后的文档。
+
+### 加固
+
+- **spa.js 渲染轮询**：`evaluate` 连续失败两次（页面已崩）即放弃 DOM 稳定等待，不再对崩溃页烧满 10s；移除 playwright `page.goto` 不支持的 `signal` 选项（静默忽略、误导读者），abort 语义由循环内检查与 catch 承担。
+
+### 验证
+
+- 单元断言 **168 → 170**（+2：directFetch 双 429 error 形状（本地 429 服务器直测，确认重试恰一次）；read_url_site 跟随壳页落在真实页）+ 12 SPA 断言 + probe 47 PASS 零 THROW 全绿。
+
+## [1.5.1] - 2026-08-30
+
+> 例行审查轮：3 个实测坐实的 bug 修复 + 3 项健壮性加固 + 2 项站点适配，零 schema 变更。
+
+### 修复（P0，均有复现用例）
+
+- **锚点读 × ld+json 兜底越权**：`extract()` 的 articleBody 兜底不感知锚点语义——定位到 <200 字符的小节时整篇 LD 全文覆盖小节内容（`anchored: true` 却返回全文）。现在锚点命中时跳过该兜底（与 readability 升级的既有守卫对齐）；无锚点的薄页仍正常兜底。
+- **feed `content:encoded` 全文丢失**：闭标签正则要求字面 `</content>`，`</content:encoded>` 永不匹配——WordPress 等博客 feed 的全文字段整段丢失只剩标题。改为捕获字段名 + 反向引用配对（`<content:encoded>` 必须配 `</content:encoded>`），顺带杜绝与 `</description>` 跨配对吞错内容。实测 WPJAM/WP学院 feed 每条恢复 200 字符实质摘要。
+- **隐藏/consent 嵌套泄漏**：容器内有先闭合的嵌套 `<div>` 时，非贪婪正则止步于第一个内层闭标签，其余横幅文本泄漏进正文（真实 consent 横幅普遍深嵌套）。改为深度计数平衡扫描（复用 tagBlockAt 思路）；扫描窗口 100KB 上限防「大量未闭合开标签」退化；未闭合 fail-open 保留内容。
+
+### 加固（P1）
+
+- **batch 单页异常隔离**：`mapLimit` 的 `Promise.all` 无守卫，单页意外 throw 会炸整批。每页 try/catch 转 `{error}`。
+- **batch 列表去重**：同一 URL 多次出现会并发双 fetch。按精确字符串去重（带 `#fragment` 的锚点读语义不同，不与裸 URL 合并）。
+- **`<time datetime>` 作用域收窄**：优先 `itemprop="datePublished"`/`pubdate` 标记的标签；扫描范围从全文档改为剥离噪声后的正文区——页脚/评论区的 time 标签不再劫持发布时间。
+
+### 站点适配（P2）
+
+- **noscript SEO 兜底**：SPA 壳把正文放 `<noscript>` 时（stripNoise 剥离的块），正文过薄则取最长的 noscript 块兜底（≥150 字符阈值；"请启用 JS" 类短横幅不触发）。
+- **懒加载图片**：markdown 模式 `<img src>` 缺失或为 data: 占位图时回退 `data-src`/`data-original`/`data-lazy-src`，描述性图片不再丢失。
+
+### 验证
+
+- 单元断言 **158 → 168**（+10：锚点守卫正反例 2、content:encoded + 跨配对 1、嵌套隐藏/consent 平衡移除 1、未闭合 fail-open 1、noscript 兜底正反例 2、懒加载回退 1、time 标记优先级 1；batch 上限断言改用 15 个不同 URL 并新增去重断言）
+- probe.mjs **39 → 47**（+8 对抗项：3000 个分散未闭合隐藏 div（窗口有界 408ms）、3000 层 consent 深嵌套完整移除 0.9ms、2000 未闭合 consent、隐藏 table、50k articleBody、畸形 LDJSON、3000 noscript 块、未闭合 noscript——全部无 THROW）
+- 新增 `site-types.mjs` 站点类型语义探针（10 类，按类型断言而非字符数）：**8/10 语义通过**，2 例境外连接超时（环境边界）。关键实锤——WordPress feed（content:encoded）摘要恢复、`<time datetime>` 博客 published 提取（2026-08-28）、懒加载站点 markdown 图片 15 张、MediaWiki 变体百科正文干净
+- 152 站全量回归：**103 OK / 25 THIN+EMPTY / 24 ERR / 0 THREW**，OK 站零噪声；5 个首窗字符下降站点经 v1.3.1/v1.4.0 模块 A/B 比对确认均为段落化重排效应，内容完整（总字符量普遍增加）
+
+## [1.5.0] - 2026-08-29
+
+> 元数据补全与 token 边角打磨轮。152 站回归确认 1.4.0 无内容性回归后，聚焦三处已核实的缺口（响应大小上限 3MB、RSS 摘要、nav 剔除等此前已就位的项不再重复）。
+
+### JSON-LD `articleBody` 正文兜底
+
+不少新闻/门户站把**全文**放在 `<script type="application/ld+json">` 的 `articleBody` 字段，可见 HTML 却是反爬壳或薄片段——DOM 提取器够不到。现在正文提取不足 200 字符时自动尝试 ld+json 正文兜底（strip 标签 → 段落化，块扫描沿用有界 LDJSON_RE，上限 200K 字符；DOM 正文充分时不触发，绝不用兜底覆盖真实正文）。
+
+### `<time datetime>` 元数据源 + 日期归一化
+
+- 发布时间合并链扩展为：显式 `article:`/`og:` meta → JSON-LD datePublished → **`<time datetime>`** → 通用 date meta → byline 文本。语义化 `<time>` 标记通常标示发布时间，优先于常为 last-modified 的通用 date meta。
+- 可识别格式的日期统一归一为 ISO（`2026年8月24日` → `2026-08-24`），覆盖 meta/JSON-LD/time/byline 全部来源；不匹配的格式原样透传。
+
+### markdown 表格行数帽
+
+兼容性对照表（40+ 行 × 多列）是 markdown 模式最大的单点 token 开销。表格超过 25 行时只保留前 25 行并追加一行 `…+N rows` 提示；text 模式不受影响。
+
+### 零宽字符剥离
+
+零宽空格/连接符（U+200B–200D、U+2060、BOM）对读者不可见却计 token，且 152 站实测中确有泄漏（如 vuejs-doc 输出）。textOnly / textLines / blockMd 三条渲染路径统一剥离。
+
+### 评估后保持不变：batch 默认 maxChars = 3000
+
+实测对比：ruanyifeng 全文不足 2000（两窗口无差异）；solidot 的 2000–3000 段是第二篇文章的真实正文——下调到 2000 会在聚合页切掉实际内容。且 batch schema 描述含字面 `default 3000`，改动必须动 schema 文本，违背静态 schema 承诺。**结论：维持 3000。**
+
+### 验证
+
+- 单元断言 **150 → 158**（+8：articleBody 兜底 2、无日期源不编造 1、time datetime 1、CJK 归一 1、表格帽 1、零宽 2；既有 byline 日期断言同步归一化行为更新）+ 12 SPA 断言全绿；
+- probe.mjs 39 PASS 无回归；
+- 实网：MDN markdown 模式表格静态 HTML 未超帽（兼容表为 JS 渲染），帽由单测覆盖；ruanyifeng/solidot 窗口评估见上。
+
+## [1.4.0] - 2026-08-28
+
+> 省token主打轮。经实测复核，原方案中的「nav 导航块剔除」与「RSS/Atom 条目摘要」在 1.3.1 已就位（python-docs 实测 235K 全文 → 20K 内干净正文，导航零泄漏；feed 输出已含每条 ≤200 字符摘要），本轮聚焦真实缺口：隐藏元素剥离、URL 片段定位阅读、段落化正文。
+
+### URL 片段定位阅读（url#section-anchor）——token 大头
+
+超长参考页（MDN / Python docs 一页 20 万+ 字符）此前只能从文档头翻起。现在 `#fragment` 直接定位目标小节：
+
+- **容器锚点**（`<section id>` / `<dl id>` 等）→ 深度计数平衡块提取（复用 tagBlockAt，正确处理嵌套），**精确切出该小节**
+- **标题/内联锚点**（`<h2 id>`、`<dt id>`、`<a name>`）→ 从锚点起读到文末（标题引出其后正文），smartTruncate 兜底
+- percent-encoded 中文锚点自动尝试解码形态；未命中锚点降级为全文（不报错）
+- **cache key 保留 fragment**：`#a` 与 `#b` 各自独立缓存条目，绝不互串；offset 续读相对小节起点
+- 命中锚点时跳过 readability 升级（其会以全文覆盖小节语义）与自动分页（拼接后续整页违背定位意图）；输出 meta 行加 `#anchor` 标记说明正文起点
+- 实测：python-docs stdtypes.html `#str.startswith` **235,393 → 111,587 字符（-52.6%）**，且默认 6000 字符窗口直接落在目标方法而非文档头
+
+### 隐藏元素与同意弹窗剥离
+
+SPA 装饰树（`style="display:none"`）、`hidden` 属性、`aria-hidden="true"`、OneTrust/Cookiebot/GDPR 等 id 挂载的同意横幅此前全部漏网进正文。新增两条有界正则过滤器：
+
+- 覆盖 `hidden`（裸属性与带值）、`aria-hidden="true"`（false 保留）、`style` 含 `display:none` / `visibility:hidden`
+- consent id 匹配：`onetrust|cookiebot|cybot|gdpr|consent|cookie-law|cookie_banner|cmp-`（大小写不敏感）
+- **`[\s"']` 属性界卫兵**：`data-hidden` / `data-id` 等复合属性名不误伤
+- 未闭合的隐藏元素安全降级为保留内容（lazy 扫描 50K 有界，探针实测 4.9ms）
+
+### 正文段落化（textLines）
+
+text 模式原先把全文压成单行平文本，offset 续读从句子中间切开。现在块级边界成为段落分隔：
+
+- 标题独立成行、`<br>` 保单换行，段落以 `\n\n` 分隔——smartTruncate 的段落对齐真正生效，续读切口落在真实段落边界
+- 标题行尾的永久链接装饰符（`¶` / `§`）剥离（text 与 markdown 双模式）
+- readability 升级路径的 text 输出同步走段落化（可选增强安装时行为一致）
+
+### 其他
+
+- 正文 <60 字符且无渲染提示的页面补一条恒定成本的提示行（「正文极短（可能登录墙或反爬拦截）」），模型不再对 40 字符页面盲目 offset 续读
+- `renderResult` 导出（供测试断言输出文案）
+
+### 验证
+
+- 单元断言 **126 → 150**（+24：隐藏/consent 剥离 11、锚点切片 8 含 e2e 缓存隔离与 offset 语义、段落接缝 2、装饰符 1、短正文提示 1、既有回归 1）+ 12 SPA 断言全绿
+- probe.mjs 对抗探针 +7（3000 隐藏 div、未闭合隐藏、2000 consent 横幅、超长 style 前缀、正则特殊字符锚点、4000 层嵌套平衡块、畸形百分号解码）全绿，最慢 4.9ms
+- 实网：python-docs 锚点定位实测（上文数据）；152 站回归基线不变
+
+## [1.3.1] - 2026-08-25
+
+> 注：v1.3.0（2026-08-24 开发完成）未独立发布——三项工作（v1.3.0 四项改进 + 08-25 健壮性轮 + 审查收尾轮）合并发布为 1.3.1。以下内容即该合并版本的完整变更。
 
 ### 四项改进：NUL 防护 + JSON-LD 元数据 + base href + meta-refresh 跟随
 

package/README.md

@@ -210,19 +210,23 @@ const results = await Promise.all([
 ## Token economy (core)
 
 1. **Body text only by default** — no redundant headings/keywords/images/word-count fields; take them via params only when needed;
-2. **Paragraph-aligned truncation + offset continuation** — 6,000 chars by default (~3,000 tokens), cut at paragraph boundaries to keep semantics; output notes a single line `(chars 6000/12990 — truncated, continue via offset)`; resume starts at the given offset, sliced from cache — **no repetition of already-read text** (measured 0+500 → 500+500, no overlap); offset past the end returns empty instead of repeating the head;
-3. **`text` mode first** — Markdown structure is opt-in;
-4. **Compact text render** — the model sees a `title:` header + body directly, no JSON parsing; `siteName` is omitted when identical to the hostname; every status hint is one short line (truncated / cached / rendered), no verbose paragraphs;
-5. **Two-tier cache** — successful results cached per URL for 5 minutes (repeat reads hit cache: fewer network calls and fewer model retries); **failed results cached for 30 seconds** so a broken URL never triggers a re-fetch loop;
-6. **KV-cache friendly (DeepSeek cost tuning)** — tool schema/description stay **static text** (no config values embedded), so changing config never invalidates the reusable prompt prefix and KV cache keeps hitting. DeepSeek's cache-hit tokens cost about 1/10 of misses — the more stable the prefix, the cheaper the run (same analysis as the official `tool-web` docs);
-7. **Batch shares the cache** — `read_url_batch` reuses the same cache (repeat batches hit it directly) and caps each page at 3,000 chars (below the single-page 6,000) to bound total output;
-8. **Compact fixed cost** — the four tool descriptions total ~900 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `—`/`…` never waste tokens or render as mojibake.
+2. **Paragraph-aligned truncation + offset continuation** — 6,000 chars by default (~3,000 tokens), cut at paragraph boundaries to keep semantics (text mode separates paragraphs with `\n\n`, so the cut lands on a real boundary, not mid-sentence); output notes a single line `(chars 6000/12990 — truncated, continue via offset)`; resume starts at the given offset, sliced from cache — **no repetition of already-read text** (measured 0+500 → 500+500, no overlap); offset past the end returns empty instead of repeating the head;
+3. **URL fragment reading (v1.4.0)** — `url#section-anchor` jumps straight to the target section of a long reference page: container anchors slice out exactly that block (depth-counted balanced extraction), heading anchors read from the heading onward; the cache key keeps the fragment so `#a`/`#b` never cross-pollute, and offset continues relative to the section start. Measured on python-docs `#str.startswith`: **235,393 → 111,587 chars (-52.6%)**, the default 6000-char window landing on the target method instead of the document head;
+4. **Hidden-element & consent-banner stripping (v1.4.0)** — `display:none`/`visibility:hidden` decoration trees, the bare `hidden` attribute, `aria-hidden="true"`, and OneTrust/Cookiebot/GDPR banners mounted by id no longer leak into the body; a bounded attribute-separator guard keeps compound names (`data-hidden`/`data-id`) from false matches;
+5. **Three-way metadata fallback (v1.5.0)** — the published-time merge chain runs `article:`/`og:` meta → JSON-LD → `<time datetime>` → generic date meta → byline, with recognized formats normalized to ISO; when an anti-bot shell leaves the body too thin, the full ld+json `articleBody` serves as a body fallback; markdown tables beyond 25 rows truncate with a `…+N rows` hint; zero-width characters (U+200B etc.) are stripped uniformly;
+6. **`text` mode first** — Markdown structure is opt-in;
+7. **Compact text render** — the model sees a `title:` header + body directly, no JSON parsing; `siteName` is omitted when identical to the hostname; every status hint is one short line (truncated / cached / rendered), no verbose paragraphs;
+8. **Two-tier cache** — successful results cached per URL for 5 minutes (repeat reads hit cache: fewer network calls and fewer model retries); **failed results cached for 30 seconds** so a broken URL never triggers a re-fetch loop;
+9. **KV-cache friendly (DeepSeek cost tuning)** — tool schema/description stay **static text** (no config values embedded), so changing config never invalidates the reusable prompt prefix and KV cache keeps hitting. DeepSeek's cache-hit tokens cost about 1/10 of misses — the more stable the prefix, the cheaper the run (same analysis as the official `tool-web` docs);
+10. **Batch shares the cache** — `read_url_batch` reuses the same cache (repeat batches hit it directly) and caps each page at 3,000 chars (below the single-page 6,000) to bound total output;
+11. **Compact fixed cost** — the four tool descriptions total ~900 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `&mdash;`/`&hellip;` never waste tokens or render as mojibake.
 
 ## Technical notes
 
 - **Encoding**: BOM-first detection (UTF-8 / UTF-16LE / UTF-16BE — byte-level evidence beats any declared charset), then HTTP `Content-Type` charset → HTML meta; built-in `TextDecoder` transcoding (Node 20+ full-icu, so declared Shift-JIS/EUC-JP/GBK/Big5 all decode), GB2312 normalized to GBK, auto-fallback to UTF-8 on mojibake;
 - **Content dispatch**: a URL is not always an HTML page — JSON APIs are re-rendered fully compact with no indentation, and string values over 1500 chars are clipped with a visible marker (since v1.2.0), RSS 2.0 / Atom feeds parse into an entry list (`title — url` + summary, `feedCount` field, full items via `includeLinks`; entry summaries are iteratively tag-stripped and entity-decoded until stable, so double-escaped `&lt;a&gt;` never leaks as a literal tag); XML sitemaps are rejected with a clear reason (no reading value); everything else falls through to the HTML pipeline;
-- **Extraction**: prefers `<article>` (all articles joined on aggregation pages; an unrelated tiny article — e.g. a newsletter card — falls through to `<main>` when its text is under 200 chars and `<main>` exists) / `<main>` / `role="main"`; `role="main"` containers are closed via **depth counting of balanced tags** (nested divs no longer cut the block at the first `</div>` — measured: gnu.org used to lose 7/8 of its body this way); strips `nav/footer/header/aside/form/iframe` and ad-like containers, heuristic fallback to `<body>`; on the body path a **text-density pass** drops link-dominated segments (related posts, category sidebars, hot-article widgets) — pages with standard containers never reach it;
+- **Extraction**: prefers `<article>` (all articles joined on aggregation pages; an unrelated tiny article — e.g. a newsletter card — falls through to `<main>` when its text is under 200 chars and `<main>` exists) / `<main>` / `role="main"`; `role="main"` containers are closed via **depth counting of balanced tags** (nested divs no longer cut the block at the first `</div>` — measured: gnu.org used to lose 7/8 of its body this way); strips `nav/footer/header/aside/form/iframe`, ad-like containers, and **hidden elements & consent banners** (the `hidden` attribute, `aria-hidden="true"`, styles containing `display:none`/`visibility:hidden`, and GDPR banners mounted under ids like onetrust/cookiebot/cybot/gdpr/consent/cookie-law; a `[\s"']` attribute-separator guard keeps compound names such as `data-hidden`/`data-id` from false matches; unclosed hidden elements degrade safely to keeping the content), heuristic fallback to `<body>`; on the body path a **text-density pass** drops link-dominated segments (related posts, category sidebars, hot-article widgets) — pages with standard containers never reach it;
+- **URL fragment anchoring (v1.4.0)**: `url#section-anchor` targets one section of a long document — container tags (section/div/dl/table…) take the **depth-counted balanced block** (nesting handled correctly, exactly that section), heading/inline anchors (`<h2 id>`/`<dt id>`/`<a name>`) read from the anchor to the end (a heading introduces the content after it); percent-encoded non-ASCII anchors are also tried in decoded form; a miss degrades to the full document. The cache key keeps the fragment (`#a`/`#b` cached as separate entries); a hit skips the readability upgrade and auto-pagination (both override located semantics with whole-document semantics); text mode is **paragraph-structured** (block-level boundaries → `\n\n` paragraph gaps, headings on their own lines — paragraph-aligned truncation actually lands on real boundaries and offset continuation cuts at paragraph seams; trailing permalink glyphs `¶`/`§` are stripped from headings);
 - **Pagination**: auto-follows `rel=next` or a short next-page anchor (下一页 / next / › / » — conservative, no fuzzy guessing); same-host only, loop-guarded, repeated paragraphs across the seam de-duplicated; continuation pages take the fast static path (a paginated SPA chain would cost one full render per page);
 - **Metadata**: `published` / `author` harvested from meta tags into the output and the status line; when the page has no such meta (ruanyifeng.com ships zero author/date tags) a **byline fallback** harvests the body head (600-char window: `作者：X / 日期：Y` — meta wins, deep-body mentions ignored, markdown link syntax stripped); empty-body pages (login walls, JS shells) fall back to `og:description` as a hint instead of nothing;
 - **Markdown**: self-written lightweight tag state machine (headings/paragraphs/lists/blockquotes/code/tables/inline bold-italic-links), zero deps; images with alt text render as `![alt](src)` (decorative empty-alt images are dropped), code fences carry the language hint from `language-*` highlighter classes;
@@ -233,12 +237,14 @@ const results = await Promise.all([
 - **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered) or a JS-redirect shell (few scripts but an empty `<body>`), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); the rendered text is accepted only when it meaningfully beats the static one (≥20 chars from an empty static extraction — short real content is no longer rejected); **bot-challenge defense** — Cloudflare-style interstitials ("Just a moment...") get an extra 8s poll for the auto-redirect, and if the page still looks like a challenge the rendered result is rejected and the static body kept (an interstitial can carry MORE text than the real body — measured 259 vs 135 chars — and must never be mistaken for an improvement); the interstitial DOM never feeds pagination/link extraction either; rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
 - **Boundaries**: login-walled pages are not readable; SPA pages need the Playwright enhancement; **structured data (e.g. which like-count belongs to which comment) is out of text-extraction scope** — this plugin flattens HTML into readable text, so exact field↔value associations are lost; for precise fields, intercept the page's actual data API (see "Real-world validation" below).
 
-## Real-world validation (2026-08-21, v1.0.0; re-verified 2026-08-22 against DSH 0.1.1-rc.2; 2026-08-24 v1.3.0)
+## Real-world validation (2026-08-21, v1.0.0; re-verified 2026-08-22 against DSH 0.1.1-rc.2; 2026-08-24 v1.3.0; 2026-08-28 v1.4.0)
 
 152-site sweep driven by `multi-site.mjs` (committed, re-runnable, 8-way concurrent): **115 OK / 17 expected boundaries (login-walls·captcha·short static pages) / 20 attributed network·anti-bot errors / 0 crashes** (final round with all pre-release fixes in; network-class errors vary ±5 between runs, all environment-attributed). Coverage: CN portals/media, e-commerce (JD/Taobao/Pinduoduo/Suning/Dangdang), video (Bilibili/iQiyi/Youku/Mango), music, games, novels (Qidian/Zongheng/JJWXC legacy GBK), Q&A/forums, government, universities (Tsinghua/PKU/Fudan/SJTU…), Traditional-Chinese TW/HK (PTT/LTN/UDN), JP/KR portals (Yahoo JP/Hatena/goo/naver/daum), overseas tech docs (GitHub/dev.to/react.dev/nodejs.org/rust/go/python), feeds, JSON APIs, encoding stress (GBK/GB2312/Big5/gb18030), anti-bot & network boundaries. All errors are environment-attributed (Wikipedia/Reddit/UDN connect-timeout; W3C/Tieba/NGA/StackOverflow 403; BUPT 412; DNS-fail…) — every one returned as a structured, correctly-attributed error, never a crash. v1.3.0 re-run (direct connect, no proxy): **93 OK / 24 THIN+EMPTY / 35 ERR / 0 THREW**, ERR all overseas connect-timeouts and 403/412 anti-bot (baseline-consistent), zero content regressions.
 
 Sweep-driven pre-release fixes (each locked by unit tests): double-escaped RSS descriptions, headerless binary sniffing, JS-redirect shell rendering, **`role="main"` container cut short by nested divs** (gnu.org 165→800 chars), **tiny `<article>` hijacking the main content** (gitlab 71→800 chars), relaxed render-acceptance threshold.
 
+v1.4.0 re-verification (2026-08-28, proxy environment): anchor read measured on python-docs stdtypes.html — `#str.startswith` takes the full 235,393 chars down to a 111,587-char located slice (**-52.6%**), with both text and markdown output starting exactly at the target method definition; 152-site sweep baseline unchanged.
+
 Plus a **15-item DSH acceptance round** (a real agent driving every read_url tool end-to-end): 12 items fully conformant, the 3 findings all closed — ① Cloudflare's "Just a moment..." interstitial carries MORE text than the real body (259 vs 135 chars) and was mistaken for a render improvement → challenge-marker detection + an extra 8s wait for the auto-redirect + interstitial results rejected; ② ruanyifeng.com ships ZERO author/date meta tags, the byline only exists in body text → byline fallback harvesting the body head (verified: `author=阮一峰 published=2026年8月21日`); ③ PDF rejection now attributes clearly: `Unsupported content-type: application/pdf`.
 
 | Category | Sites | Result |
@@ -258,7 +264,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **126 zero-dep assertions** (v1.3.1 round 2 adds: pre-block entity decoding, follow-charset propagation, read_url_links shell following; v1.3.1 robustness round adds: meta attribute-order, quoted content values, http-equiv charset, oversized JSON-LD + nested mainEntity, markdown paren escaping, m3u8 crawl noise, bounded oversized attributes; v1.3.0 adds: control-char entity sanitization, JSON-LD metadata, base-href link resolution, meta-refresh following ×11; v1.2.0 adds: sentence-aligned truncation, compact JSON, pagination variants, adversarial-input time bounds, prose &lt; preservation, deep-JSON degradation) (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green;
+- **150 zero-dep assertions** (v1.4.0 adds 24: hidden/consent stripping ×11 — hidden attribute / aria-hidden / style-hidden / onetrust / cybot / gdpr / compound-attribute guard / aria-hidden=false kept; anchor slicing ×8 — container balanced block / heading-to-end / markdown anchoring / miss-degrades-to-full / percent-decoding / data-id guard / e2e cache isolation & offset semantics / bare URL unaffected; paragraph seams & permalink glyphs ×3; thin-page hint ×1; regression ×1; v1.3.1 round 2 adds: pre-block entity decoding, follow-charset propagation, read_url_links shell following; v1.3.1 robustness round adds: meta attribute-order, quoted content values, http-equiv charset, oversized JSON-LD + nested mainEntity, markdown paren escaping, m3u8 crawl noise, bounded oversized attributes; v1.3.0 adds: control-char entity sanitization, JSON-LD metadata, base-href link resolution, meta-refresh following ×11; v1.2.0 adds: sentence-aligned truncation, compact JSON, pagination variants, adversarial-input time bounds, prose &lt; preservation, deep-JSON degradation) (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green; probe.mjs adversarial probes +7 (every new v1.4.0 regex time-bounded, slowest 4.9ms);
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap
@@ -278,6 +284,9 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - [x] Adversarial-page linearization (unclosed-tag/unpaired-lt/attribute-scan bounds + recursion depth cap, fixing a quadratic DoS vector), prose &lt; preservation, proxy-path redirects keep the final URL, deep-JSON degradation instead of crash (v1.2.0 round 2)
 - [x] Control-char entity sanitization (`&#0;`/`&#127;` → space, surrogates → U+FFFD), JSON-LD metadata (meta → JSON-LD → byline), `<base href>` link resolution (frame/legacy-forum pages), `<meta refresh>` shell following (≤3 hops, loop-safe, fail-open, v1.3.0)
 - [x] Robustness round: order-independent meta extraction (content-first no longer missed), recursive JSON-LD nesting (ItemList→mainEntity), legacy `<meta http-equiv="Content-Type">` charset, bounded link/pagination regexes, markdown paren escaping, crawl stream-URL noise filtering (v1.3.1)
+- [x] URL fragment reading (`url#section-anchor`: container balanced-block / heading-to-end, cache isolation, offset relative to the section; python-docs measured -52.6%, v1.4.0)
+- [x] Hidden-element & consent-banner stripping (hidden/aria-hidden/style-hidden + onetrust/cookiebot/gdpr id matching, v1.4.0)
+- [x] Paragraph-structured text output (block boundaries → paragraph gaps, offset continuation cuts at real paragraph boundaries, v1.4.0)
 
 > From v1.0.0 the plugin enters maintenance mode: bug fixes first, updates kept rare.
 

package/README.zh.md

@@ -210,19 +210,23 @@ const results = await Promise.all([
 ## 省 token 设计（核心）
 
 1. **默认只给正文**——不返回 headings/keywords/images/字数统计等冗余字段，需要时按参数取；
-2. **段落级智能截断 + offset 续读**——默认 6000 字符（约 3000 token），在段落边界截断保证语义完整，输出行仅一行 `(chars 6000/12990 — 截断，offset 续读)` 引导；续读从指定偏移开始、命中缓存切片，**不重复返回已读前文**（实测 0+500 → 500+500，无重复）；offset 越界返回空而非重复开头；
-3. **text 模式优先**——Markdown 结构按需开启；
-4. **紧凑文本 render**——模型直接看到 `title:` 头部 + 正文，无需解析 JSON；`siteName` 与域名相同时省略；状态提示全部一行内（截断/续读/缓存/渲染标记），无长段落废话；
-5. **双层缓存**——成功结果按 URL 缓存 5 分钟（重复读取直接命中，省网络也省模型重试）；**失败结果缓存 30 秒**（坏 URL 不会触发重复 fetch 循环）；
-6. **KV Cache 友好（DeepSeek 成本特调）**——工具 schema/description 保持**静态文本**（不嵌入配置值），配置变更不会使可复用的 prompt 前缀失效，KV 缓存持续命中。DeepSeek 缓存命中 token 价格约为未命中的 1/10，前缀越稳定越省钱（官方 `tool-web` 文档同款分析）；
-7. **批量共用缓存**——`read_url_batch` 内部复用同一套缓存，重复批量读直接命中，且每页默认 3000 字符（低于单页 6000）控制总量；
-8. **固定开销压缩**——4 个工具 description 合计约 900 字符（有断言守卫，保持静态利于 KV 缓存）；HTML 实体解码扩展至 45 个命名实体，`—`/`…` 等残留不再浪费 token 或显示为乱码。
+2. **段落级智能截断 + offset 续读**——默认 6000 字符（约 3000 token），在段落边界截断保证语义完整（text 模式段落以 `\n\n` 分隔，切口落在真实段落边界而非句中），输出行仅一行 `(chars 6000/12990 — 截断，offset 续读)` 引导；续读从指定偏移开始、命中缓存切片，**不重复返回已读前文**（实测 0+500 → 500+500，无重复）；offset 越界返回空而非重复开头；
+3. **URL 片段定位阅读（v1.4.0）**——`url#section-anchor` 直接定位长文档目标小节：容器锚点精确切块（深度计数平衡提取）、标题锚点从该处起读；cache key 保留 fragment 保证 `#a`/`#b` 互不串页，offset 相对小节起点。实测 python-docs `#str.startswith`：**235,393 → 111,587 字符（-52.6%）**，默认 6000 字符窗口直接落在目标方法而非文档头；
+4. **隐藏元素与同意弹窗剥离（v1.4.0）**——`display:none`/`visibility:hidden` 装饰树、`hidden` 属性、`aria-hidden="true"`、OneTrust/Cookiebot/GDPR id 挂载的横幅不再混入正文；复合属性名（`data-hidden`/`data-id`）有界卫兵不误伤；
+5. **元数据三路兜底（v1.5.0）**——发布时间合并链 `article:`/`og:` meta → JSON-LD → `<time datetime>` → 通用 date meta → byline，可识别格式归一为 ISO；反爬壳页正文过薄时自动用 ld+json `articleBody` 全文兜底；markdown 表格超 25 行截断并附 `…+N rows` 提示；零宽字符（U+200B 等）统一剥离；
+6. **text 模式优先**——Markdown 结构按需开启；
+7. **紧凑文本 render**——模型直接看到 `title:` 头部 + 正文，无需解析 JSON；`siteName` 与域名相同时省略；状态提示全部一行内（截断/续读/缓存/渲染标记），无长段落废话；
+8. **双层缓存**——成功结果按 URL 缓存 5 分钟（重复读取直接命中，省网络也省模型重试）；**失败结果缓存 30 秒**（坏 URL 不会触发重复 fetch 循环）；
+9. **KV Cache 友好（DeepSeek 成本特调）**——工具 schema/description 保持**静态文本**（不嵌入配置值），配置变更不会使可复用的 prompt 前缀失效，KV 缓存持续命中。DeepSeek 缓存命中 token 价格约为未命中的 1/10，前缀越稳定越省钱（官方 `tool-web` 文档同款分析）；
+10. **批量共用缓存**——`read_url_batch` 内部复用同一套缓存，重复批量读直接命中，且每页默认 3000 字符（低于单页 6000）控制总量；
+11. **固定开销压缩**——4 个工具 description 合计约 900 字符（有断言守卫，保持静态利于 KV 缓存）；HTML 实体解码扩展至 45 个命名实体，`&mdash;`/`&hellip;` 等残留不再浪费 token 或显示为乱码。
 
 ## 技术说明
 
 - **编码**：BOM 优先探测（UTF-8 / UTF-16LE / UTF-16BE——字节级证据优先于任何声明），其次 HTTP `Content-Type` charset → HTML meta；内置 `TextDecoder` 转码（Node 20+ full-icu，页面声明的 Shift-JIS/EUC-JP/GBK/Big5 均可正确解码），GB2312 归一为 GBK，检测到乱码自动回退 UTF-8；
 - **内容类型分发**：URL 不一定是 HTML——JSON 接口紧凑重排渲染（无缩进，超长字符串值 >1500 字符截断并标注，v1.2.0 起缩进 1 格改为完全紧凑），RSS 2.0 / Atom 订阅源解析为条目列表（`标题 — 链接` + 摘要，`feedCount` 字段，`includeLinks` 时附完整 items；条目摘要迭代「剥标签+解实体」直到稳定，双重转义的 `&lt;a&gt;` 不会漏成字面标签）；XML sitemap 明确拒绝（对模型无阅读价值）；其余全部走 HTML 管线；
-- **正文提取**：优先 `<article>`（聚合页多篇合并；无关小卡片 article——如订阅挂件——文本不足 200 字符且页面有 `<main>` 时自动回落 main）/ `<main>` / `role="main"`，`role="main"` 容器用**深度计数找平衡闭合标签**（嵌套 div 不会在第一个 `</div>` 被截断——实测 gnu.org 曾因此只取到 1/8 正文）；剥离 `nav/footer/header/aside/form/iframe` 及广告类容器，启发式回归到 `<body>`；body 路径上追加**文本密度过滤**——丢弃链接主导的短块（相关推荐/分类侧栏/热门文章挂件），标准容器页面完全不走此路径；
+- **正文提取**：优先 `<article>`（聚合页多篇合并；无关小卡片 article——如订阅挂件——文本不足 200 字符且页面有 `<main>` 时自动回落 main）/ `<main>` / `role="main"`，`role="main"` 容器用**深度计数找平衡闭合标签**（嵌套 div 不会在第一个 `</div>` 被截断——实测 gnu.org 曾因此只取到 1/8 正文）；剥离 `nav/footer/header/aside/form/iframe` 及广告类容器、**隐藏元素与同意弹窗**（`hidden` 属性、`aria-hidden="true"`、`style` 含 `display:none`/`visibility:hidden`、`onetrust/cookiebot/cybot/gdpr/consent/cookie-law` 等 id 挂载的 GDPR 横幅；`[\s"']` 属性界卫兵保证 `data-hidden`/`data-id` 复合属性名不误伤；未闭合元素安全降级为保留内容），启发式回归到 `<body>`；body 路径上追加**文本密度过滤**——丢弃链接主导的短块（相关推荐/分类侧栏/热门文章挂件），标准容器页面完全不走此路径；
+- **URL 片段定位（v1.4.0）**：`url#section-anchor` 定位长文档小节——容器标签（section/div/dl/table 等）取**深度计数平衡块**（嵌套正确、精确切出该节），标题/内联锚点（`<h2 id>`/`<dt id>`/`<a name>`）从该处读到文末（标题引出其后正文）；percent-encoded 中文锚点自动尝试解码形态；未命中降级全文。cache key 保留 fragment（`#a`/`#b` 独立缓存）；命中锚点时跳过 readability 升级与自动分页（二者都以全文语义覆盖定位语义）；text 模式**段落化输出**（块级边界 → `\n\n` 段落分隔，标题独立成行，smartTruncate 段落对齐真正生效，offset 续读切口落在真实段落边界；标题行尾永久链接装饰符 `¶`/`§` 剥离）；
 - **分页拼接**：识别 `rel=next`（标准）或纯「下一页 / next / › / »」短锚文本（刻意保守，不做模糊猜测）；同域限定 + 防环；跨页重复段落自动去重；续页走静态快路径（分页 SPA 链每页一次完整渲染不划算）；
 - **元数据**：`published` / `author` 从 meta 标签提取进输出字段与状态行；页面无相关 meta 时（如阮一峰博客零 meta 标签）从**正文头部 600 字符的署名行兜底**（「作者：X / 日期：2026年8月21日」，meta 优先、正文深部提及不误采、markdown 链接语法不泄漏）；空正文页（登录墙/JS 壳）回落到 `og:description` 作为提示，不再返回空；
 - **Markdown**：自研轻量标签状态机（标题/段落/列表/引用/代码块/表格/行内加粗斜体链接），零依赖；带 alt 的图片渲染为 `![alt](src)`（空 alt 装饰图丢弃），代码块围栏带 `language-*` class 里的语言标注；
@@ -233,12 +237,14 @@ const results = await Promise.all([
 - **可选增强二（SPA 页面渲染）**：在 DSH profile 目录执行 `npm i playwright && npx playwright install chromium` 后自动启用。检测到正文为空且（页面脚本密集 疑似 Vue/React 客户端渲染，或 body 为空的 JS 跳转壳——script 数不多但正文全靠跳转）时，自动用无头 Chromium 渲染后再提取（`rendered` 标记告知模型）；渲染结果仅在**显著优于**静态提取时采用（静态为空时 ≥20 字符即接受，防短正文被拒）；**人机验证页防御**——Cloudflare「Just a moment...」等指纹页识别后额外轮询 8s 供其自动跳转，仍未通过则拒绝渲染结果、保留静态正文（指纹页文字量可能超过真实正文，实测 259 vs 135 字符，绝不当作提升），指纹页 DOM 也不污染分页/链接提取；渲染采用 `domcontentloaded` + **DOM 稳定轮询**（内容停止增长即收，上限 10s）而非 `networkidle`——心跳轮询站永不空闲，避免 30s 超时；未安装时优雅提示安装方法、不报错——核心保持零依赖；
 - **边界**：登录墙页面无法读取；SPA 页面需安装 Playwright 增强后渲染读取（未安装时返回明确提示）；**结构化数据（如评论的点赞数归属、榜单数值）不在文本提取范围**——本插件把 HTML 扁平化为可读文本，字段与数值的精确对应关系会丢失；需要精确字段时，用 Playwright 拦截页面实际调用的数据 API 获取（见下方「真实世界验证」）。
 
-## 真实世界验证（2026-08-21，v1.0.0；2026-08-22 适配 DSH 0.1.1-rc.2 复验；2026-08-24 v1.3.0 复验）
+## 真实世界验证（2026-08-21，v1.0.0；2026-08-22 适配 DSH 0.1.1-rc.2 复验；2026-08-24 v1.3.0 复验；2026-08-28 v1.4.0 复验）
 
 152 站全量实测（`multi-site.mjs` 已提交可复跑，8 并发）：**115 OK / 17 预期边界（登录墙·验证页·静态小页） / 20 网络·反爬归因错误 / 0 崩溃**（含全部发布前修复的终态轮；网络类错误逐轮有 ±5 波动，均为环境归因）。覆盖国内门户 / 媒体 / 电商（京东·淘宝·拼多多·苏宁·当当）/ 视频（B 站·爱奇艺·优酷·芒果）/ 音乐 / 游戏 / 小说（起点·纵横·晋江 legacy GBK）/ 问答 / 论坛 / 政府 / 高校（清北复交等 8 所）/ 港台繁体（PTT·自由时报·联合报）/ 日韩（Yahoo JP·Hatena·goo·naver·daum）/ 海外技术站（GitHub·dev.to·react.dev·nodejs·rust·go·python docs）/ 订阅源 / JSON API / 编码压力（GBK·GB2312·Big5·gb18030）/ 反爬与网络边界。错误全部环境归因（维基/Reddit/UDN 连接超时；W3C/贴吧/NGA/StackOverflow 403；北邮 412；DNS 失败等）——每一个都返回结构化、准确归因的错误，无一崩溃。v1.3.0 复验轮（本机直连，无代理）：**93 OK / 24 THIN+EMPTY / 35 ERR / 0 THREW**，ERR 全部为境外连接超时与 403/412 反爬（与基线一致），无内容性回归。
 
 扫描驱动的发布前修复（全部带单测锁定）：RSS 双重转义、无头二进制嗅探、JS 跳转壳渲染、**`role="main"` 嵌套 div 截断**（gnu.org 165→800 字符）、**小 article 劫持主内容**（gitlab 71→800 字符）、渲染接受门槛放宽。
 
+v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点定位实测——`#str.startswith` 全文 235,393 → 定位切片 111,587 字符（-52.6%），text/markdown 双模式内容均精确起始于目标方法定义；152 站回归基线不变。
+
 另有 **DSH 15 项全功能验收轮**（真实 Agent 调用 read_url 系工具逐项核对）：12 项完全符合，3 项问题全部修复闭环——① Cloudflare「Just a moment...」验证页文字量超过真实正文（259 vs 135 字符）被误当作渲染提升 → 挑战页特征识别 + 渲染后额外等待 8s 自动跳转 + 拒绝接受指纹页；② 阮一峰页面零 author/date meta 标签，作者/日期只在正文里 → byline 兜底从正文头部 600 字符提取署名行（实测 `author=阮一峰 published=2026年8月21日`）；③ PDF 类型拒绝归因 → 明确报 `Unsupported content-type: application/pdf`。
 
 | 类别 | 站点 | 结果 |
@@ -258,7 +264,7 @@ const results = await Promise.all([
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **126 个单元断言**（v1.3.1 收尾轮新增：pre 代码块实体解码、meta-refresh 跟随 charset 更新、read_url_links 壳页跟随；v1.3.1 健壮性轮新增：meta 属性序颠倒、content 内嵌引号、http-equiv charset、JSON-LD 大块/嵌套 mainEntity、markdown 括号转义、m3u8 噪音、超长属性限时；v1.3.0 新增：控制字符实体防护、JSON-LD 元数据、base href 链接、meta-refresh 跟随 11 条；v1.2.0 新增：长段落句级对齐、紧凑 JSON、分页变体、对抗输入限时、散文 &lt; 保留、深 JSON 降级）（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；
+- **150 个单元断言**（v1.4.0 新增 24：隐藏/consent 剥离 11（hidden 属性/aria-hidden/style 隐藏/onetrust/cybot/gdpr/复合属性卫兵/aria-hidden=false 保留）、锚点切片 8（容器平衡块/标题到文末/markdown 定位/未命中降级/百分号解码/data-id 卫兵/e2e 缓存隔离与 offset 语义/裸 URL 不受影响）、段落接缝与装饰符 3、短正文提示 1、回归 1；v1.3.1 收尾轮新增：pre 代码块实体解码、meta-refresh 跟随 charset 更新、read_url_links 壳页跟随；v1.3.1 健壮性轮新增：meta 属性序颠倒、content 内嵌引号、http-equiv charset、JSON-LD 大块/嵌套 mainEntity、markdown 括号转义、m3u8 噪音、超长属性限时；v1.3.0 新增：控制字符实体防护、JSON-LD 元数据、base href 链接、meta-refresh 跟随 11 条；v1.2.0 新增：长段落句级对齐、紧凑 JSON、分页变体、对抗输入限时、散文 &lt; 保留、深 JSON 降级）（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；probe.mjs 对抗探针 +7（v1.4.0 新正则全部限时通过，最慢 4.9ms）；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap
@@ -278,6 +284,9 @@ const results = await Promise.all([
 - [x] 对抗页面线性化（无配对尖括号/无闭合标签/属性扫描加界 + 递归深度帽，修复 60KB→2.2s 的二次复杂度 DoS 向量）、散文 &lt; 不再被吞、代理重定向保留最终 URL、深 JSON 降级不崩溃（v1.2.0 第二轮）
 - [x] NUL/控制字符实体防护（`&#0;`/`&#127;` 输出空格、surrogate 输出 U+FFFD，不污染模型上下文）、JSON-LD 元数据三级提取（meta → JSON-LD → byline）、`<base href>` 链接解析（框架/老论坛页）、`<meta refresh>` 壳页自动跟随（最多 3 跳、防循环、fail-open，v1.3.0）
 - [x] 健壮性轮：meta 属性序无关提取（content 在前不再漏）、JSON-LD 嵌套结构递归（ItemList→mainEntity）、`<meta http-equiv="Content-Type">` 老式 charset、链接/分页正则统一加界、markdown 括号转义、爬虫流媒体噪音过滤（v1.3.1）
+- [x] URL 片段定位阅读（`url#section-anchor` 容器平衡块/标题到文末、缓存隔离、offset 相对小节，python-docs 实测 -52.6%，v1.4.0）
+- [x] 隐藏元素与同意弹窗剥离（hidden/aria-hidden/style 隐藏 + onetrust/cookiebot/gdpr id 匹配，v1.4.0）
+- [x] text 模式段落化输出（块级边界 → 段落分隔，offset 续读切口落在真实段落边界，v1.4.0）
 
 > v1.0.0 起进入维护期：以修 bug 为主，减少更新频率。
 

package/index.js

@@ -100,10 +100,11 @@ export function decodeBuffer(buffer, contentType) {
   return { text, charset: enc }
 }
 
-// Content types this plugin can consume. Beyond HTML: JSON (data APIs) and
-// XML (RSS/Atom feeds) are read natively — see readUrl's type dispatch.
+// Content types this plugin can consume. Beyond HTML: JSON (data APIs),
+// XML (RSS/Atom feeds) and plain-text families (txt/md/csv — logs, docs,
+// datasets) are read natively — see readUrl's type dispatch.
 // Note "application/rss+xml": the separator before xml is '+', not '/'.
-const FETCHABLE_CT = /text\/html|application\/xhtml|text\/plain|\/json|[+/]xml/i
+const FETCHABLE_CT = /text\/html|application\/xhtml|text\/(?:plain|markdown|csv)|\/json|[+/]xml/i
 
 // Retry-After header (seconds form) → ms, capped; undefined when absent/invalid.
 function retryAfterMs(res) {
@@ -168,14 +169,24 @@ async function directFetchOnce(url, signal, cfg) {
 }
 
 // Direct-connect fetch with a single Retry-After-aware retry on 429/503.
-async function directFetch(url, signal, cfg) {
+// Exported for tests: the double-throttle path must always produce an { error }
+// shape — a second `{ retryAfterMs }` would otherwise leak past the race
+// filter into decodeBuffer as a bufferless "success" (bare TypeError there).
+export async function directFetch(url, signal, cfg) {
   const first = await directFetchOnce(url, signal, cfg)
   if (first && first.retryAfterMs !== undefined && !(signal && signal.aborted)) {
     await new Promise((resolve) => {
       const t = setTimeout(resolve, first.retryAfterMs)
       if (signal) signal.addEventListener('abort', () => { clearTimeout(t); resolve() }, { once: true })
     })
-    return directFetchOnce(url, signal, cfg)
+    if (signal && signal.aborted) return { error: 'cancelled' }
+    const second = await directFetchOnce(url, signal, cfg)
+    // The retry budget is spent — report the rate limit instead of returning
+    // a shapeless { retryAfterMs } (no buffer, no error) to the caller.
+    if (second && second.retryAfterMs !== undefined) {
+      return { error: 'HTTP 429/503 rate limited (retried once after Retry-After, still throttled)' }
+    }
+    return second
   }
   return first
 }
@@ -329,6 +340,10 @@ function defuseLt(html) {
   return html.slice(0, from) + html.slice(from).replace(/</g, ' ')
 }
 
+// Invisible-to-readers joiner characters (zero-width space/joiners, word
+// joiner, BOM) cost tokens and leak into copy/paste text — strip at render.
+const ZW_RE = /[\u200b-\u200d\u2060\ufeff]/g
+
 function textOnly(html) {
   return decodeTextEntities(defuseLt(html)
     .replace(/<!--[\s\S]*?-->/g, ' ')
@@ -338,9 +353,36 @@ function textOnly(html) {
     .replace(/<noscript[\s>][\s\S]*?<\/noscript>/gi, ' ')
     .replace(/<[^>]{1,1000}>/g, ' ')
     .replace(/[ \t\r\n]+/g, ' ')
+    .replace(ZW_RE, '')
     .trim())
 }
 
+// Paragraph-aware variant of textOnly for the body extraction path: block
+// boundaries become paragraph breaks so offset continuation slices at real
+// paragraph seams (smartTruncate aligns on /\n\n+/) instead of mid-sentence,
+// and long reference pages stay readable instead of one flattened line.
+// <br> keeps single newlines; every other tag runs are stripped as spaces.
+function textLines(html) {
+  const cleaned = decodeTextEntities(defuseLt(html)
+    .replace(/<!--[\s\S]*?-->/g, ' ')
+    .replace(/<script[\s>][\s\S]*?<\/script>/gi, ' ')
+    .replace(/<style[\s>][\s\S]*?<\/style>/gi, ' ')
+    .replace(/<textarea[\s>][\s\S]*?<\/textarea\s*>/gi, ' ')
+    .replace(/<noscript[\s>][\s\S]*?<\/noscript>/gi, ' ')
+    .replace(/<br\b[^>]{0,1000}\/?>/gi, '\n')
+    .replace(/<\/?(?:p|div|section|article|main|h[1-6]|li|ul|ol|table|tbody|thead|tfoot|tr|blockquote|pre|figcaption|dt|dd|dl|figure|fieldset|option)\b[^>]{0,1000}>/gi, '\n')
+    .replace(/<[^>]{1,1000}>/g, ' '))
+  const lines = []
+  for (const rawLine of cleaned.replace(ZW_RE, '').split('\n')) {
+    let line = rawLine.replace(/[ \t\r\f\v]+/g, ' ').trim()
+    // A heading becomes its own short paragraph in this layout; a trailing
+    // permalink glyph on one is anchor decoration, not content.
+    if (line.length <= 120) line = line.replace(/\s*[¶§]$/, '')
+    if (line) lines.push(line)
+  }
+  return lines.join('\n\n')
+}
+
 // ---- JSON / RSS / Atom content dispatch ----
 // A URL is not always an HTML page: data APIs answer JSON and content sources
 // answer feeds. Both are model-readable when compacted, so read_url handles
@@ -374,7 +416,10 @@ function parseFeed(xml, limit) {
     const linkTag = /<link[^>]*>([\s\S]*?)<\/link>/i.exec(it)
     const linkHref = /<link[^>]+href=["']([^"']+)["']/i.exec(it)
     const link = (linkTag && xmlText(linkTag[1])) || (linkHref && linkHref[1]) || ''
-    const desc = xmlText(/<(?:description|summary|content)[^>]*>([\s\S]*?)<\/(?:description|summary|content)>/i.exec(it)?.[1] || '')
+    // Namespaced fields (WordPress <content:encoded>) pair via a captured
+    // name + backreference: the closer must match the SAME name, so a
+    // <content:encoded> opener can never pair with a distant </description>.
+    const desc = xmlText(/<((?:description|summary|content)(?::[a-zA-Z0-9]+)?)(?:\s[^>]*)?>([\s\S]*?)<\/\1\s*>/i.exec(it)?.[2] || '')
     if (!title && !link && !desc) continue
     items.push({ title: title.slice(0, 120), url: link, summary: desc.slice(0, 200) })
   }
@@ -427,13 +472,16 @@ export function densityFilter(html) {
 
 // Depth-counted block extraction: returns html from the opening tag at
 // openIdx through its BALANCED closing tag. A non-greedy regex would stop at
-// the first nested </div>, truncating heavily nested container divs.
-function tagBlockAt(html, openIdx, tagName) {
+// the first nested </div>, truncating heavily nested container divs. maxScan
+// bounds the search window — a closer further out than that is treated as
+// absent (keeps hostile many-unclosed-opens inputs linear).
+function tagBlockAt(html, openIdx, tagName, maxScan = Infinity) {
   const re = new RegExp(`</?${tagName}(?=[\\s>])`, 'gi')
   re.lastIndex = openIdx
   let depth = 0
   let m
   while ((m = re.exec(html))) {
+    if (m.index - openIdx > maxScan) return null
     depth += m[0][1] === '/' ? -1 : 1
     if (depth === 0) {
       const close = html.indexOf('>', m.index)
@@ -443,6 +491,29 @@ function tagBlockAt(html, openIdx, tagName) {
   return null
 }
 
+// Remove every element matched by an OPEN-TAG-ONLY pattern together with its
+// balanced content. The pattern must capture the tag name (group 1) so the
+// scanner counts only same-name open/close pairs. Unclosed matches (no closer
+// within the 100k scan window) are left in place — fail-open. Each match
+// continues from the removed block's end, so removed interiors are never
+// rescanned and total cost stays near-linear in the surviving input.
+const BALANCED_SCAN_WINDOW = 100000
+function stripBalanced(html, openTagRe) {
+  openTagRe.lastIndex = 0
+  let out = ''
+  let last = 0
+  let m
+  while ((m = openTagRe.exec(html))) {
+    const tag = m[1].toLowerCase()
+    const block = tagBlockAt(html, m.index, tag, BALANCED_SCAN_WINDOW)
+    if (!block) continue // unclosed container — keep the content
+    out += html.slice(last, m.index) + ' '
+    last = m.index + block.length
+    openTagRe.lastIndex = last
+  }
+  return out + html.slice(last)
+}
+
 export function pickMain(html) {
   // Aggregation pages (e.g. blog homepages) have one <article> per item:
   // collect all of them instead of only the first. A tiny <article> (e.g. a
@@ -509,6 +580,16 @@ function stripNoise(mainHtml) {
   if (box.length) {
     out = out.replace(new RegExp(`<(${box.join('|')})[\\s>][\\s\\S]*?<\\/\\1>`, 'gi'), ' ')
   }
+  // Elements explicitly hidden from users are invisible decoration or stateful
+  // UI (collapsed panels, modal templates) — their text must not leak into the
+  // body. The [\s"'] guard keeps `hidden`/`style` from matching inside
+  // compound attr names (data-hidden, aria-hidden). Consent/GDPR banners mount
+  // by id (onetrust, cookiebot, cybot, gdpr) rather than class, so a second
+  // pass keys on the id attribute. Removal is a depth-counted balanced scan —
+  // banners are deeply nested and a lazy `[\s\S]*?` would stop at the first
+  // inner closer, leaking the rest of the banner text.
+  out = stripBalanced(out, /<(div|span|section|p|ul|table)\b[^>]{0,1000}[\s"'](?:hidden\b|aria-hidden\s*=\s*["']?true|style\s*=\s*["'][^"']{0,200}(?:display:\s*none|visibility:\s*hidden))[^>]{0,1000}>/gi)
+  out = stripBalanced(out, /<([a-z][a-z0-9]*)\s+id=["'][^"']{0,120}(?:onetrust|cookiebot|cybot|gdpr|consent|cookie-law|cookie_banner|cmp-)[^"']{0,120}["'][^>]{0,1000}>/gi)
   return out.replace(
     /<([a-z][a-z0-9]*)[^>]{1,1000}class=["'][^"']{0,300}(ad-|ads|advert|banner|sidebar|social|share|comment|popup|modal|cookie)[^"']{0,300}["'][^>]{0,1000}>[\s\S]*?<\/\1>/gi,
     ' ',
@@ -527,10 +608,29 @@ function escInline(s) {
 const IMG_SENTINEL = '\u0001'
 function imgsToMarkdown(html) {
   const store = []
-  const out = html.replace(/<img\b[^>]{0,1000}>/gi, (tag) => {
+  // AMP pages use <amp-img> instead of <img>; responsive layouts wrap images
+  // in <picture><source srcset>. When the <img> fallback carries no usable
+  // src (lazy placeholder), the first source URL is injected into it so the
+  // normal img pass below still emits the image.
+  html = html.replace(/<picture\b[^>]*>([\s\S]{0,50000}?)<\/picture\s*>/gi, (block) => {
+    const imgTag = /<img\b[^>]{0,1000}>/i.exec(block)
+    const imgSrc = imgTag && /(?:^|\s)src=["']([^"']+)["']/i.exec(imgTag[0])
+    if (imgTag && imgSrc && !/^data:/i.test(imgSrc[1])) return block
+    const source = /<source\b[^>]{0,1000}\bsrcset=["']([^"'\s>]+)/i.exec(block)
+    if (!source || !imgTag) return block
+    return block.replace(/<img\b[^>]{0,1000}>/i, (img0) => img0.replace(/<img/i, `<img src="${source[1]}"`))
+  })
+  const out = html.replace(/<(?:amp-)?img\b[^>]{0,1000}>/gi, (tag) => {
     const alt = /alt=["']([^"']*)["']/i.exec(tag)
-    const src = /src=["']([^"']+)["']/i.exec(tag)
-    store.push(src && alt && alt[1].trim() ? `![${alt[1].trim()}](${src[1]})` : '')
+    let src = /src=["']([^"']+)["']/i.exec(tag)
+    let srcVal = src && src[1]
+    // Lazy-load wrappers keep the real URL in data-* attributes; a data: URI
+    // src is an inline placeholder spacer, not a usable image reference.
+    if (!srcVal || /^data:/i.test(srcVal)) {
+      const lazy = /data-(?:src|original|lazy-src)=["']([^"']+)["']/i.exec(tag)
+      if (lazy) srcVal = lazy[1]
+    }
+    store.push(srcVal && alt && alt[1].trim() ? `![${alt[1].trim()}](${srcVal})` : '')
     return `${IMG_SENTINEL}${store.length - 1}${IMG_SENTINEL}`
   })
   return {
@@ -552,6 +652,7 @@ function imgsToMarkdown(html) {
 // the walker degrades to plain text.
 const OPEN_TAG_RE = /<([a-zA-Z0-9]+)((?:"[^"]*"|'[^']*'|[^'">]){0,1000})>/g
 const MD_MAX_DEPTH = 100
+const MD_TABLE_MAX_ROWS = 25
 function stripUnmatchedOpeners(html) {
   const closers = new Set()
   for (const m of html.matchAll(/<\/([a-zA-Z0-9]+)/g)) closers.add(m[1].toLowerCase())
@@ -615,7 +716,8 @@ export function blockMd(html, depth = 0) {
     }
     if (tag === 'h1' || tag === 'h2' || tag === 'h3' || tag === 'h4' || tag === 'h5' || tag === 'h6') {
       const level = '#'.repeat(Number(tag[1]))
-      out += `\n\n${level} ${inlineMd(inner, depth + 1)}`
+      // Trailing permalink glyphs are anchor decorations, not heading text.
+      out += `\n\n${level} ${inlineMd(inner, depth + 1).replace(/\s*[¶§]\s*$/, '')}`
     } else if (tag === 'p') {
       const t = inlineMd(inner, depth + 1)
       if (t) out += `\n\n${t}`
@@ -676,7 +778,12 @@ export function blockMd(html, depth = 0) {
         // unescape cells' escaped pipes before deriving the separator row,
         // otherwise the --- line is one char wider per escaped pipe
         const sep = header.replace(/\\\|/g, '|').replace(/[^|]/g, '-')
-        out += `\n\n${rows.join('\n')}\n${sep}`
+        // Compatibility tables (40+ rows × many columns) are the single
+        // largest token sink in markdown mode; the hint keeps the model
+        // aware the table continues without repeating the cell noise.
+        const kept = rows.length > MD_TABLE_MAX_ROWS ? rows.slice(0, MD_TABLE_MAX_ROWS) : rows
+        const more = rows.length - kept.length
+        out += `\n\n${kept.join('\n')}\n${sep}${more > 0 ? `\n…+${more} rows` : ''}`
       }
     } else if (tag === 'a' || tag === 'strong' || tag === 'b' || tag === 'em' || tag === 'i') {
       const t = inlineMd(inner, depth + 1)
@@ -686,7 +793,7 @@ export function blockMd(html, depth = 0) {
       if (t) out += t
     }
   }
-  return img.restore(out)
+  return img.restore(out).replace(ZW_RE, '')
 }
 
 // Read the content attribute of the FIRST <meta> whose property/name equals
@@ -781,6 +888,76 @@ function jsonLdMeta(html) {
   return { published: '', author: '' }
 }
 
+// News portals frequently ship the FULL article text inside ld+json
+// (articleBody) while the visible HTML is an anti-bot shell or a thin
+// fragment — a fallback body source the DOM extractors cannot reach. Returns
+// the first string articleBody found (array values joined), stripped to
+// paragraphs via textLines. Block scan is the same bounded LDJSON_RE.
+function jsonLdArticleBody(html) {
+  for (const m of html.matchAll(LDJSON_RE)) {
+    let json
+    try {
+      json = JSON.parse(m[1])
+    } catch {
+      continue
+    }
+    const nodes = []
+    const collect = (node, depth = 0) => {
+      if (!node || typeof node !== 'object' || depth > 20) return
+      if (Array.isArray(node)) {
+        for (const it of node) collect(it, depth + 1)
+        return
+      }
+      nodes.push(node)
+      for (const k of Object.keys(node)) {
+        const v = node[k]
+        if (v && typeof v === 'object') collect(v, depth + 1)
+      }
+    }
+    collect(json)
+    for (const n of nodes) {
+      const raw = Array.isArray(n.articleBody) ? n.articleBody.join('\n\n') : n.articleBody
+      if (typeof raw === 'string' && raw.trim()) return textLines(raw).slice(0, 200000)
+    }
+  }
+  return ''
+}
+
+// SPA shells ship their SEO copy inside <noscript> (the block the stripNoise
+// pass removes from main content). A thin body falls back to the LONGEST
+// noscript block when it carries substantive text; short ones are UI noise
+// ("please enable JavaScript"). Unclosed blocks simply never match — fail-open.
+function noscriptBody(html) {
+  let best = ''
+  for (const m of html.matchAll(/<noscript[^>]{0,500}>([\s\S]{0,50000}?)<\/noscript\s*>/gi)) {
+    const t = textLines(m[1])
+    if (t.length > best.length) best = t
+  }
+  return best.length >= 150 ? best : ''
+}
+
+// Semantic <time datetime="..."> — common on blogs/docs that ship no meta or
+// ld+json date. Tags explicitly marked as publication time (itemprop/pubdate)
+// win; otherwise the FIRST time tag in the scoped html is used. Callers pass
+// the noise-stripped main content so footer/comment timestamps (which carry
+// their own <time> tags) are out of scope. Empty attribute is a no-op.
+function timeTagMeta(html) {
+  if (!html) return ''
+  const marked = /<time\b[^>]{0,1000}(?:itemprop=["']datePublished["']|pubdate\b)[^>]{0,1000}datetime=["']([^"']{1,60})["']/i.exec(html)
+  if (marked) return marked[1].trim().slice(0, 40)
+  const m = /<time\b[^>]{0,1000}datetime=["']([^"']{1,60})["']/i.exec(html)
+  return m ? m[1].trim().slice(0, 40) : ''
+}
+
+// CJK / loose-variant dates normalize to ISO so the metadata line stays
+// comparable across sources ("2026年8月24日" → "2026-08-24"). Anything that
+// does not match a recognized shape passes through untouched.
+function normalizeDate(s) {
+  const m = /^(\d{4})\s*[年/\-.]\s*(\d{1,2})\s*[月/\-.]\s*(\d{1,2})\s*日?$/.exec((s || '').trim())
+  if (!m) return s
+  return `${m[1]}-${m[2].padStart(2, '0')}-${m[3].padStart(2, '0')}`
+}
+
 // Meta-less pages still print the byline inside the body head: "作者：阮一峰
 // 日期：2026年8月21日". Fill ONLY the missing fields, and only from the first
 // 600 chars of body text — deeper matches are content, not bylines.
@@ -796,12 +973,47 @@ function bylineFallback(text, meta) {
   }
   if (!out.published) {
     const m = BYLINE_DATE_RE.exec(head)
-    if (m) out.published = m[1].replace(/\s+/g, ' ').trim().slice(0, 40)
+    if (m) out.published = normalizeDate(m[1].replace(/\s+/g, ' ').trim().slice(0, 40))
   }
   return out
 }
 
-export function extract(html, mode) {
+// Locate the element whose id/name equals `anchor` and return the html of its
+// section: for container tags (section/div/dl/…) the element's own balanced
+// block IS the section; for heading/inline anchors (<h2 id>, <a name>, <dt
+// id>) the anchor only marks where the section starts, so everything FROM it
+// onward is kept (the heading introduces the content that follows). Returns
+// null when no element matches — callers fall back to full text.
+// The [\s"'] guard rejects compound attr names (data-id, data-name), and the
+// exact-quoted value means id="prefix-anchor" never matches anchor "prefix".
+const ANCHOR_CONTAINER = /^(?:section|div|article|aside|main|dl|table|ul|ol|details|figure|blockquote|fieldset|form)$/
+function sliceAtAnchor(mainHtml, anchor) {
+  if (!anchor) return null
+  const esc = anchor.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
+  const m = new RegExp(`<([a-z][a-z0-9]*)\\b[^>]{0,1000}[\\s"'](?:id|name)\\s*=\\s*["']${esc}["'][^>]{0,1000}>`, 'i').exec(mainHtml)
+  if (!m) return null
+  const tag = m[1].toLowerCase()
+  if (ANCHOR_CONTAINER.test(tag)) {
+    const block = tagBlockAt(mainHtml, m.index, tag)
+    if (block) return block
+  }
+  return mainHtml.slice(m.index)
+}
+
+// Anchors may arrive percent-encoded (CJK section names); the document can
+// carry either form. Try the raw fragment first, then its decoded variant.
+function anchorCandidates(anchor) {
+  const list = [anchor]
+  try {
+    const dec = decodeURIComponent(anchor)
+    if (dec !== anchor) list.push(dec)
+  } catch {
+    /* malformed percent-encoding — raw form only */
+  }
+  return list
+}
+
+export function extract(html, mode, anchor = '') {
   html = defuseLt(html) // kills the no-'>'-tail quadratic scans before any picker runs
   const titleTag = /<title[^>]{0,1000}>([\s\S]*?)<\/title>/i.exec(html)
   const ogTitle = metaContent(html, 'og:title')
@@ -809,21 +1021,36 @@ export function extract(html, mode) {
   const langMatch = /<html[^>]{1,1000}lang=["']([\w-]+)["']/i.exec(html)
   let main = stripNoise(pickMain(html))
   main = stripNoise(revealEscapedTags(main))
+  // A URL fragment scopes the read to one section of a long reference page —
+  // slice at the anchor so only that section is extracted. Unmatched anchors
+  // degrade to the full document instead of failing.
+  let scoped = null
+  for (const cand of anchorCandidates(anchor)) {
+    scoped = sliceAtAnchor(main, cand)
+    if (scoped) break
+  }
   let bodyText
   if (mode === 'markdown') {
-    bodyText = blockMd(main).replace(/\n{3,}/g, '\n\n').trim()
+    bodyText = blockMd(scoped || main).replace(/\n{3,}/g, '\n\n').trim()
   } else {
-    bodyText = textOnly(main).replace(/ +/g, ' ').trim()
+    bodyText = textLines(scoped || main).replace(/ +/g, ' ').trim()
   }
   const meta = extractMeta(html)
   const ld = jsonLdMeta(html)
-  // Merge chain: <meta> → JSON-LD → byline. An explicit article:/og: meta date
-  // stays authoritative; otherwise a JSON-LD datePublished beats the generic
-  // date meta key (last-modified stamps are not publication times).
-  const published = ld.published
-    ? (meta.publishedExplicit ? meta.published : ld.published)
-    : meta.published
+  // Merge chain: explicit article:/og: meta date → JSON-LD datePublished →
+  // <time datetime> (semantic markup usually marks publication, unlike the
+  // generic date meta key which is often last-modified) → generic date meta →
+  // byline text. jsonLdArticleBody doubles as a thin-body fallback below.
+  const published = meta.publishedExplicit
+    ? meta.published
+    : normalizeDate(ld.published || timeTagMeta(main) || meta.published)
   const author = ld.author || meta.author
+  // Thin-body fallback: anti-bot shells and JS-only pages still carry the
+  // full article in ld+json — prefer it over the short og:description hint.
+  // An anchored read scopes the answer to one section: the ld+json body is
+  // the WHOLE article and would override the located section, so it applies
+  // only to full-document reads.
+  const ldBody = !scoped && bodyText.length < 200 ? jsonLdArticleBody(html) : ''
   // Empty-body pages (login walls, JS-only shells) still carry og:description
   // — surface it as a fallback hint instead of nothing.
   let text = bodyText
@@ -832,6 +1059,17 @@ export function extract(html, mode) {
     text = meta.description
     usedDescription = true
   }
+  if (ldBody && ldBody.length > text.length) {
+    text = ldBody
+    usedDescription = false
+  }
+  // Same thin-body shape, next fallback layer: SPA shells put their SEO copy
+  // in <noscript> (stripped from main) — surface it when substantive.
+  const nosBody = !scoped && text.length < 200 ? noscriptBody(html) : ''
+  if (nosBody && nosBody.length > text.length) {
+    text = nosBody
+    usedDescription = false
+  }
   const byline = bylineFallback(text, { published, author })
   return {
     title: titleTag ? textOnly(titleTag[1]) || titleTag[1].trim() : ogTitle,
@@ -841,6 +1079,7 @@ export function extract(html, mode) {
     author: byline.author,
     text,
     usedDescription,
+    anchored: !!scoped,
   }
 }
 
@@ -1053,6 +1292,10 @@ function sliceFrom(full, offset, maxChars) {
   }
   if (full.rendered) out.rendered = true
   if (full.spaHint) out.spaHint = full.spaHint
+  if (full.anchored) {
+    out.anchored = true
+    out.anchor = full.anchor || ''
+  }
   if (full.paginated > 1) out.paginated = full.paginated
   if (full.feedCount) out.feedCount = full.feedCount
   if (full.published) out.published = full.published
@@ -1142,9 +1385,13 @@ async function followMetaRefresh(html, finalUrl, externalSignal, cfg) {
   let curUrl = finalUrl
   let curCharset = '' // set only when a hop actually happened
   const seen = new Set([normalizeUrl(finalUrl)])
+  // Chain budget: each hop carries its own fetch timeout, so 3 hops could
+  // triple the call's cost — the whole follow chain shares ONE timeout.
+  const t0 = Date.now()
   for (let i = 0; i < MAX_META_REFRESH_HOPS; i++) {
     const target = metaRefreshTarget(cur, curUrl)
     if (!target) break
+    if (Date.now() - t0 > cfg.timeoutMs) break
     const norm = normalizeUrl(target)
     if (!norm || seen.has(norm)) break
     seen.add(norm)
@@ -1166,29 +1413,52 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   const maxChars = Math.max(500, Math.min(20000, Number(args.maxChars) || cfg.maxChars))
   const mode = args.mode === 'markdown' ? 'markdown' : 'text'
   const offset = Math.max(0, Number(args.offset) || 0)
+  // Wall-clock anchor for the pagination budget below (the tool timeout
+  // formula reserves paginateMax+1 fetches + render slack; the in-code guard
+  // keeps a slow page from consuming the budget meant for the rest).
+  const startedAt = Date.now()
 
   // Cache stores the FULL extracted text keyed by url+mode+includeLinks, so
   // continuation reads (offset) and different maxChars hit the same entry —
   // but a links-request must not be served from a links-less cached copy.
-  // The key drops the URL fragment: #sections would otherwise split the cache.
+  // The fragment STAYS in the key: a #section read scopes the extraction to
+  // that anchor, so two fragments must never share one cache entry.
   let cacheUrl = url
   try {
-    const u = new URL(url)
-    u.hash = ''
-    cacheUrl = u.href
+    cacheUrl = new URL(url).href
   } catch {
     /* keep raw url on parse failure */
   }
+  // A #fragment scopes the read to one section of a long page. The browser
+  // never sends it to the server, so it must be taken from the REQUEST url —
+  // finalUrl (after redirects/renders) has no fragment.
+  let anchor = ''
+  try {
+    const h = new URL(url).hash
+    if (h.length > 1) anchor = h.slice(1)
+  } catch {
+    /* keep bare url */
+  }
   const cacheKey = `${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
   // Failed fetches are cached briefly so the model doesn't re-request a
   // broken URL in a loop (token + latency saver).
   const fh = failCache.get(cacheKey)
   if (fh) {
-    if (Date.now() - fh.time < 30000) return { error: fh.error, cached: true }
+    if (Date.now() - fh.time < 30000) {
+      // LRU touch: re-insert as the newest slot so repeated failures on a
+      // busy session don't evict (and re-fetch) this URL before its TTL ends.
+      failCache.delete(cacheKey)
+      failCache.set(cacheKey, fh)
+      return { error: fh.error, cached: true }
+    }
     failCache.delete(cacheKey)
   }
   const hit = cache.get(cacheKey)
   if (hit && Date.now() - hit.time < cfg.cacheTtlMs) {
+    // LRU touch: hot pages survive eviction even when cached long ago — the
+    // Map re-insertion moves the entry to the newest slot.
+    cache.delete(cacheKey)
+    cache.set(cacheKey, hit)
     const sliced = sliceFrom(hit.full, offset, maxChars)
     return { ...sliced, cached: true }
   }
@@ -1198,16 +1468,26 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     return { error }
   }
 
-  // Non-HTML payloads (JSON APIs, RSS/Atom feeds) are read natively: fetch,
-  // dispatch by content-type / payload shape, render compact. Falls back to
-  // the HTML pipeline for everything else.
+  // Non-HTML payloads (JSON APIs, RSS/Atom feeds, plain text files) are read
+  // natively: fetch, dispatch by content-type / payload shape, render compact.
+  // Falls back to the HTML pipeline for everything else.
+  // JSON with JS-only literals (NaN/Infinity — hand-rolled serializers) is
+  // not valid JSON: retry once with the literals nulled rather than dropping
+  // the whole page into the HTML pipeline as a raw-text dump.
+  const parseJsonLoose = (t) => {
+    try {
+      return JSON.parse(t)
+    } catch {
+      return JSON.parse(t.replace(/-?(?:NaN|Infinity)\b/g, 'null'))
+    }
+  }
   const dispatch = async () => {
     const viaSeam = await fetchViaWebSeam(ctx, url, externalSignal, cfg)
     if (viaSeam) {
       const t = viaSeam.html.trim()
       if (t.startsWith('{') || t.startsWith('[')) {
         try {
-          return { kind: 'json', text: compactJson(JSON.parse(t)) }
+          return { kind: 'json', text: compactJson(parseJsonLoose(t)) }
         } catch { /* not JSON — keep HTML pipeline */ }
       }
       return { kind: 'html', html: viaSeam.html, finalUrl: viaSeam.finalUrl, charset: 'provider-decoded' }
@@ -1218,7 +1498,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     const decoded = decodeBuffer(page.buffer, page.contentType)
     if (/json$/.test(ct)) {
       try {
-        return { kind: 'json', text: compactJson(JSON.parse(decoded.text)) }
+        return { kind: 'json', text: compactJson(parseJsonLoose(decoded.text)) }
       } catch { /* invalid JSON — serve through the HTML pipeline as text */ }
     }
     if (/[+/]xml$/.test(ct)) {
@@ -1229,11 +1509,30 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
         return { kind: 'feed', feed: parseFeed(decoded.text, cfg.maxLinks) }
       }
     }
+    // Plain-text families (txt/md/csv): line structure IS the content — the
+    // HTML pipeline would flatten every newline into spaces. Pages that are
+    // actually HTML but mislabelled text/plain keep the HTML pipeline.
+    if (/^text\/(?:plain|markdown|csv)$/.test(ct)) {
+      const head = decoded.text.slice(0, 5000)
+      if (!/<(?:html|body|div|p|table)\b[\s>]/i.test(head)) {
+        const text = decoded.text
+          .replace(/\r\n?/g, '\n')
+          .replace(/[ \t]+\n/g, '\n')
+          .replace(/\n{4,}/g, '\n\n\n')
+          .trim()
+        return { kind: 'text', text, charset: decoded.charset }
+      }
+    }
     return { kind: 'html', html: decoded.text, finalUrl: page.finalUrl || url, charset: decoded.charset }
   }
   const payload = await dispatch()
   if (payload.error) return failWith(payload.error)
 
+  if (payload.kind === 'text') {
+    const full = { url, title: '', siteName: hostOf(url), lang: '', charset: payload.charset, mode: 'text', fullText: payload.text, paginated: 1 }
+    cacheStore(cacheKey, full, cfg.cacheMax)
+    return sliceFrom(full, offset, maxChars)
+  }
   if (payload.kind === 'json') {
     const full = { url, title: '', siteName: hostOf(url), lang: '', charset: 'json', mode: 'json', fullText: payload.text, paginated: 1 }
     cacheStore(cacheKey, full, cfg.cacheMax)
@@ -1264,7 +1563,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   // The target may use a different encoding than the shell page — reflect it.
   if (followed.charset) charset = followed.charset
 
-  let extracted = extract(html, mode)
+  let extracted = extract(html, mode, anchor)
   // Optional readability upgrade: cleaner article extraction when installed.
   const ext = await getReadabilityExtractor()
   // The base passed to readability must be the CURRENT final URL — after a
@@ -1277,14 +1576,16 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
       if (clean && clean.length > 0) {
         target.text = mode === 'markdown'
           ? blockMd(clean).replace(/\n{3,}/g, '\n\n').trim()
-          : textOnly(clean).replace(/ +/g, ' ').trim()
+          : textLines(clean).replace(/ +/g, ' ').trim()
       }
     } catch {
       // keep heuristic result
     }
     return target
   }
-  extracted = upgrade(extracted, html)
+  // Readability would replace the scoped section with the FULL article text;
+  // when an anchor matched, the scoped extraction IS the answer.
+  extracted = extracted.anchored ? extracted : upgrade(extracted, html)
 
   // Optional SPA enhancement: if the page looks client-rendered and static
   // extraction found almost nothing, try headless rendering (playwright).
@@ -1311,7 +1612,8 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
           ? '已渲染但被人机验证页拦截，保留静态提取结果'
           : '已渲染但被人机验证页拦截（无静态内容可用）'
       } else {
-        const r2 = upgrade(extract(rr.html, mode), rr.html, rr.finalUrl || finalUrl)
+        const r2raw = extract(rr.html, mode, anchor)
+        const r2 = r2raw.anchored ? r2raw : upgrade(r2raw, rr.html, rr.finalUrl || finalUrl)
         // Accept the rendered text when it meaningfully beats the static one.
         // From an EMPTY static extraction even a short body is a real gain
         // (JS shells can render just 30-60 chars of real content, below the
@@ -1338,10 +1640,16 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   const paginateMax = cfg.paginate === false ? 1 : Math.max(1, Math.min(10, Number(cfg.paginateMax) || 3))
   const startHost = hostOf(finalUrl || url)
   const seen = new Set([normalizeUrl(finalUrl || url)])
-  let nextUrl = paginateMax > 1 ? findNextLink(renderedHtml || html, finalUrl || url) : null
+  // An anchored read is scoped to one section — joining the "next page" chain
+  // would append whole pages the fragment never asked for.
+  let nextUrl = paginateMax > 1 && !extracted.anchored ? findNextLink(renderedHtml || html, finalUrl || url) : null
   while (
     nextUrl && paginated < paginateMax &&
     sameHost(nextUrl, startHost) &&
+    // Budget guard: the whole read (initial fetch + renders + chain) must fit
+    // the tool timeout formula (paginateMax+1 fetches). A slow first page or
+    // SPA render eats into the chain's share here instead of overshooting.
+    Date.now() - startedAt < cfg.timeoutMs * (paginateMax + 1) &&
     !(externalSignal && externalSignal.aborted)
   ) {
     const nu = normalizeUrl(nextUrl)
@@ -1359,6 +1667,13 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     nextUrl = findNextLink(nextHtml, page.finalUrl || nextUrl)
   }
 
+  // A near-empty body (login wall, anti-bot shell) with no render hint yet
+  // still looks like "the page content" to the model — one constant-cost line
+  // tells it not to keep offset-reading a 40-char page.
+  if (!spaHint && extracted.text && extracted.text.length < 60) {
+    spaHint = '正文极短（可能登录墙或反爬拦截）'
+  }
+
   const full = {
     url: finalUrl,
     title: extracted.title || '',
@@ -1369,6 +1684,8 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     fullText: extracted.text,
     rendered,
     spaHint,
+    anchored: extracted.anchored || false,
+    anchor: extracted.anchored ? anchor : '',
     paginated,
     published: extracted.published || '',
     author: extracted.author || '',
@@ -1381,7 +1698,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   return sliceFrom(full, offset, maxChars)
 }
 
-function renderResult(value) {
+export function renderResult(value) {
   if (typeof value === 'string') return value
   const r = value || {}
   if (r.error) return `Error: ${r.error}`
@@ -1396,6 +1713,9 @@ function renderResult(value) {
   }
   if (r.published) meta.push(r.published)
   if (r.author) meta.push(`by ${r.author}`)
+  // Tells the model why the text starts mid-document: the read was scoped to
+  // this fragment, and offset continues from the section start.
+  if (r.anchored) meta.push(`#${r.anchor || ''}`)
   if (r.feedCount) meta.push(`feed · ${r.feedCount} 条`)
   if (r.mode === 'json') meta.push('json')
   if (meta.length) lines.push(meta.join(' · '))
@@ -1408,6 +1728,9 @@ function renderResult(value) {
   if (r.cached) flags.push('cached')
   if (r.paginated > 1) flags.push(`已拼接${r.paginated}页`)
   if (r.rendered) flags.push('rendered')
+  // With text present the hint rides the flags line; with empty text the
+  // dedicated "无可读内容" line below already carries it.
+  if (r.text && r.spaHint) flags.push(r.spaHint)
   if (flags.length) lines.push(`(${flags.join(' · ')})`)
   if (!r.text) {
     if (r.charsStart > 0 && !r.truncated) {
@@ -1582,15 +1905,34 @@ function readUrlBatchTool(ctx, cfg) {
     },
     async execute(args, exec) {
       const a = args || {}
-      const urls = (Array.isArray(a.urls) ? a.urls : []).map((u) => String(u).trim()).filter(Boolean)
+      // Duplicate URLs collapse to one fetch (parallel copies would race the
+      // cache and fetch the same page twice); order is preserved. Exact-string
+      // identity: a #fragment read is an ANCHORED read with different
+      // semantics, so it must not merge with its bare URL.
+      const seenUrl = new Set()
+      const urls = (Array.isArray(a.urls) ? a.urls : [])
+        .map((u) => String(u).trim())
+        .filter(Boolean)
+        .filter((u) => {
+          if (seenUrl.has(u)) return false
+          seenUrl.add(u)
+          return true
+        })
       if (!urls.length) return { error: 'urls array is required (1-10 http(s) URLs)' }
       const list = urls.slice(0, 10)
       const perMax = Math.max(500, Math.min(20000, Number(a.maxChars) || 3000))
       const mode = a.mode === 'markdown' ? 'markdown' : 'text'
       const signal = exec && exec.signal
-      const pages = await mapLimit(list, 4, (u) =>
-        readUrl({ url: u, maxChars: perMax, mode, includeLinks: a.includeLinks === true }, ctx, signal, cfg),
-      )
+      // One page throwing must not reject the whole batch — readUrl returns
+      // error objects for expected failures, but any unexpected exception is
+      // converted to one here too.
+      const pages = await mapLimit(list, 4, async (u) => {
+        try {
+          return await readUrl({ url: u, maxChars: perMax, mode, includeLinks: a.includeLinks === true }, ctx, signal, cfg)
+        } catch (e) {
+          return { error: `Unexpected error: ${e && e.message ? e.message : e}` }
+        }
+      })
       const ok = pages.filter((p) => !p.error)
       return {
         total: list.length,
@@ -1663,11 +2005,19 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal) {
         visited.add(norm)
         const page = await fetchPage(url, externalSignal, cfg)
         if (page.error) return { url, depth, error: page.error }
-        const html = decodeBuffer(page.buffer, page.contentType).text
+        let html = decodeBuffer(page.buffer, page.contentType).text
+        let pageUrl = page.finalUrl || url
+        // Same shell-following as read_url: a meta-refresh stub would otherwise
+        // be crawled AS the page (title-less hop text pollutes results and the
+        // real content is never reached). Fails open — fetch problems keep the
+        // stub's own HTML.
+        const followed = await followMetaRefresh(html, pageUrl, externalSignal, cfg)
+        html = followed.html
+        pageUrl = followed.finalUrl
         const ex = extract(html, 'text')
-        const links = extractLinks(html, 100, page.finalUrl || url)
+        const links = extractLinks(html, 100, pageUrl)
         return {
-          url: page.finalUrl || url,
+          url: pageUrl,
           depth,
           title: ex.title || '',
           chars: ex.text.length,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.3.1",
+  "version": "1.6.1",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",

package/spa.js

@@ -58,13 +58,13 @@ export async function renderPage(url, externalSignal) {
     // 'domcontentloaded' instead of 'networkidle': heartbeat-polling sites
     // (qq-news, juejin) never go idle and would time out at 30s. Instead wait
     // for the DOM to stabilize (content stops growing) up to 10s — SPA paint
-    // usually lands within a couple of seconds of DOM-ready.
-    // The cooperative signal aborts both the navigation and the poll loop, so
-    // a cancelled/timed-out call releases the browser immediately instead of
-    // burning the full poll window.
-    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000, signal: externalSignal || undefined })
+    // usually lands within a couple of seconds of DOM-ready. Two consecutive
+    // evaluate failures mean the page crashed — stop waiting for growth that
+    // can never come.
+    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 })
     const t0 = Date.now()
     let prevLen = -1
+    let evalFails = 0
     for (;;) {
       if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
       await page.waitForTimeout(500)
@@ -72,6 +72,7 @@ export async function renderPage(url, externalSignal) {
         .evaluate(() => (document.body ? document.body.innerHTML.length : 0))
         .catch(() => -1)
       if (Date.now() - t0 > 10000) break
+      if (len === -1 && ++evalFails >= 2) break
       // stop once two consecutive reads agree — including empty bodies (a
       // blank page should not burn the full 10s poll)
       if (len === prevLen && prevLen >= 0) break