dsh-read-url 1.2.0 → 1.4.0

package/CHANGELOG.md

@@ -1,5 +1,100 @@
 # Changelog
 
+## [1.4.0] - 2026-08-28
+
+> 省token主打轮。经实测复核，原方案中的「nav 导航块剔除」与「RSS/Atom 条目摘要」在 1.3.1 已就位（python-docs 实测 235K 全文 → 20K 内干净正文，导航零泄漏；feed 输出已含每条 ≤200 字符摘要），本轮聚焦真实缺口：隐藏元素剥离、URL 片段定位阅读、段落化正文。
+
+### URL 片段定位阅读（url#section-anchor）——token 大头
+
+超长参考页（MDN / Python docs 一页 20 万+ 字符）此前只能从文档头翻起。现在 `#fragment` 直接定位目标小节：
+
+- **容器锚点**（`<section id>` / `<dl id>` 等）→ 深度计数平衡块提取（复用 tagBlockAt，正确处理嵌套），**精确切出该小节**
+- **标题/内联锚点**（`<h2 id>`、`<dt id>`、`<a name>`）→ 从锚点起读到文末（标题引出其后正文），smartTruncate 兜底
+- percent-encoded 中文锚点自动尝试解码形态；未命中锚点降级为全文（不报错）
+- **cache key 保留 fragment**：`#a` 与 `#b` 各自独立缓存条目，绝不互串；offset 续读相对小节起点
+- 命中锚点时跳过 readability 升级（其会以全文覆盖小节语义）与自动分页（拼接后续整页违背定位意图）；输出 meta 行加 `#anchor` 标记说明正文起点
+- 实测：python-docs stdtypes.html `#str.startswith` **235,393 → 111,587 字符（-52.6%）**，且默认 6000 字符窗口直接落在目标方法而非文档头
+
+### 隐藏元素与同意弹窗剥离
+
+SPA 装饰树（`style="display:none"`）、`hidden` 属性、`aria-hidden="true"`、OneTrust/Cookiebot/GDPR 等 id 挂载的同意横幅此前全部漏网进正文。新增两条有界正则过滤器：
+
+- 覆盖 `hidden`（裸属性与带值）、`aria-hidden="true"`（false 保留）、`style` 含 `display:none` / `visibility:hidden`
+- consent id 匹配：`onetrust|cookiebot|cybot|gdpr|consent|cookie-law|cookie_banner|cmp-`（大小写不敏感）
+- **`[\s"']` 属性界卫兵**：`data-hidden` / `data-id` 等复合属性名不误伤
+- 未闭合的隐藏元素安全降级为保留内容（lazy 扫描 50K 有界，探针实测 4.9ms）
+
+### 正文段落化（textLines）
+
+text 模式原先把全文压成单行平文本，offset 续读从句子中间切开。现在块级边界成为段落分隔：
+
+- 标题独立成行、`<br>` 保单换行，段落以 `\n\n` 分隔——smartTruncate 的段落对齐真正生效，续读切口落在真实段落边界
+- 标题行尾的永久链接装饰符（`¶` / `§`）剥离（text 与 markdown 双模式）
+- readability 升级路径的 text 输出同步走段落化（可选增强安装时行为一致）
+
+### 其他
+
+- 正文 <60 字符且无渲染提示的页面补一条恒定成本的提示行（「正文极短（可能登录墙或反爬拦截）」），模型不再对 40 字符页面盲目 offset 续读
+- `renderResult` 导出（供测试断言输出文案）
+
+### 验证
+
+- 单元断言 **126 → 150**（+24：隐藏/consent 剥离 11、锚点切片 8 含 e2e 缓存隔离与 offset 语义、段落接缝 2、装饰符 1、短正文提示 1、既有回归 1）+ 12 SPA 断言全绿
+- probe.mjs 对抗探针 +7（3000 隐藏 div、未闭合隐藏、2000 consent 横幅、超长 style 前缀、正则特殊字符锚点、4000 层嵌套平衡块、畸形百分号解码）全绿，最慢 4.9ms
+- 实网：python-docs 锚点定位实测（上文数据）；152 站回归基线不变
+
+## [1.3.1] - 2026-08-25
+
+> 注：v1.3.0（2026-08-24 开发完成）未独立发布——三项工作（v1.3.0 四项改进 + 08-25 健壮性轮 + 审查收尾轮）合并发布为 1.3.1。以下内容即该合并版本的完整变更。
+
+### 四项改进：NUL 防护 + JSON-LD 元数据 + base href + meta-refresh 跟随
+
+延续维护期原则：零运行时依赖、静态 schema 保 KV-cache（工具 schema 无任何变更）、fail-open 不报错。
+
+**修复（实体解码安全）**
+- **NUL/控制字符实体防护**：`decodeTextEntities` 原先对 `&#0;`（NUL）、`&#127;`（DEL）及 C1 控制区直接 `String.fromCodePoint`，控制字符会混入正文与模型上下文。新增 `fromSafeCodePoint`：C0/C1 中非空白控制字符（0–8、11–12、14–31、127–159）输出空格；`\t\n\r` 作为合法 HTML 空白实体保留（页面用 `&#10;` 表示换行，压成空格会破坏 markdown 段落结构）；surrogate 区（U+D800–DFFF）输出 U+FFFD，不再泄漏孤立代码单元。
+
+**功能补全**
+- **JSON-LD 元数据（jsonLdMeta）**：大量新闻/文章站的发布时间、作者只存在于 schema.org `<script type="application/ld+json">`。解析首个含 datePublished/author 的 JSON-LD 块（容忍损坏 JSON；支持 @graph、@type 数组、author 为字符串/对象/数组形态；块大小有界防恶意大 script）。合并链：**meta → JSON-LD → byline**——显式 `article:`/`og:` meta 日期保持权威；JSON-LD 的 datePublished 高于通用 `date`/`dc.date` meta（后者常是 last-modified 而非发布时间）。
+- **base href 链接解析（detectBaseHref）**：老论坛/框架页声明 `<base href>` 时，页面内相对链接按文档 URL 解析会落错主机。新增 `detectBaseHref(html, finalUrl)`（head 内第一个 base，属性序无关、协议相对按文档协议解析、空值忽略），`extractLinks` 与 `findNextLink` 统一使用——链接列表、分页跟随、整站爬虫三处同时受益。
+- **meta-refresh 跟随（metaRefreshTarget + 有界循环）**：`<meta http-equiv="refresh" content="0;url=...">` 壳页（链接跳板、反盗链中转、无 JS 的 SPA 入口）只返回跳板 HTML，正文在第二个 URL。新增纯函数 `metaRefreshTarget`（属性顺序无关、大小写不敏感、content 内引号剥离、相对目标按文档 URL 或 base href 解析、非 http(s) 拒绝、**只跟随立即跳转**——定时自刷新如 `content="30; url=..."` 是正常页面，不跟随）。插入点：dispatch 返回 HTML 之后、extract() 之前——先于 SPA 渲染判断，命中时省掉一次 playwright 全渲染。有界跟随最多 3 跳 + normalizeUrl seen-set 防循环；复用 fetchPage 直连/代理竞速路径；**fail-open**：跟随失败（网络错/非 HTML）保留原 HTML 继续静态提取，不报错；跟随后的页面不单独写全局缓存（调用方缓存键仍是原始 URL，壳页重请求会重新跟随）。输出 schema 不变，`finalUrl` 自然反映最终地址。
+
+**验证**
+- 单元断言 **88 → 114**（新增 26 条：控制字符实体 6、JSON-LD 5、metaRefreshTarget 纯函数 9、base href 链接 3、readUrl 集成跟随 4——含 2 跳链、循环终止、fail-open）+ 12 SPA 断言全绿；
+- probe.mjs 对抗探针全绿，性能无回归（新增路径全部线性）；
+- 实网：24 扩展站 **18/24 OK**（失败均境外连接超时/403 环境边界）；152 站回归 **93 OK / 24 THIN+EMPTY / 35 ERR / 0 THREW**，0 崩溃、0 内容性回归（ERR 全为境外超时与 403/412 反爬，与 v1.2.0 基线一致）；
+- taptap.cn 当前已上阿里云 WAF JS 验证页（非 meta-refresh 壳页），壳页场景由本地集成测试覆盖（标准/相对/2 跳/循环/fail-open）。
+
+### 同日第二轮：健壮性审查（6 处修复 + JSON-LD 嵌套递归缺陷）
+
+静态通读 + 新测试驱动的健壮性审查：
+
+**修复**
+- **extractLinks / findNextLink 正则加界**：锚点属性 `[^>]+`/`[^>]*` 无界——超长属性（或无 `>`）的 `<a` 会让每个锚点位置扫描到串尾，与 v1.2.0「属性扫描统一加 {0,1000} 界」原则不一致。统一改为 `[^>]{0,1000}`，与全项目对抗性防御对齐；超长属性链接被安全跳过，正常链接不受影响（实测 300KB 属性 5–10ms）
+- **meta 属性提取顺序无关（metaContent 两段式）**：`extractMeta` / og:title / og:site_name 原为「property/name 必须在 content 之前」的单段正则——content 在前（`<meta content="..." property="...">`）的真实页面提取为空。改为先定位标签再读 content，与 metaRefreshTarget 同手法；content 值内嵌引号（`他说'你好'`）不再截断
+- **JSON-LD 嵌套结构（新测试抓出）**：原 collect 只展开数组与 @graph，`ItemList → mainEntity → Article`、`WebPage → mainEntityOfPage` 等嵌套结构的 datePublished/author 全部漏掉。改为递归展开全部对象属性（深度帽 20 防恶意嵌套）
+- **charset 嗅探补老式写法**：`<meta http-equiv="Content-Type" content="text/html; charset=gb2312">`（无 charset 属性、HTTP 头无 charset 的 legacy GBK 页）原先识别不到。sniffCharset 增加 http-equiv 分支
+- **JSON-LD 块上限 20k → 100k**：真实新闻站的 article-list JSON-LD 常超 20k，元数据随之丢失。上限放宽（防御性有界不变，元数据在块首节点，扫描成本可控）
+- **markdown 链接括号转义**：`[inner](url)` 的 URL 含 `)`（wiki/框架页常见）会提前终止 markdown 链接。href 中 `(` `)` 百分号编码为 `%28`/`%29`
+- **爬虫噪音扩展**：NOISE_EXT 增加 `m3u8|m3u|mpd|flv|ts`（流媒体 URL 不再进爬虫队列）
+
+**验证**
+- 单元断言 **114 → 123**（新增 9 条：meta 属性序颠倒 2、content 引号 1、og 顺序 1、http-equiv charset 1、JSON-LD 大块 1、嵌套 mainEntity 1、markdown 括号 1、爬虫 m3u8 噪音 + 超长属性限时 2）+ 12 SPA 断言全绿；
+- probe.mjs 全绿，无性能回归；
+
+### 同日第二轮：全量审查收尾（4 处一致性修复）
+
+针对全量代码审查（index.js 全流程逐段核查 + 实测验证）发现的一致性/元数据缺陷：
+
+**修复**
+- **meta-refresh 跟随后 charset 未更新**：`charset` 在 follow 之前取自壳页，壳页 utf-8 → 目标 GBK 时返回的 charset 标记与正文实际编码不符。`followMetaRefresh` 现在返回跟随目标的 charset，`readUrl` 在跟随后更新
+- **readability base URL 用原始请求地址**：`upgrade()` 传 `url`（原始 URL）给 readability，meta-refresh 跟随/SPA 渲染后文档实际在另一主机，相对链接/图片会按错误 base 解析。改为传当前 `finalUrl`（SPA 路径显式传 `rr.finalUrl`）
+- **markdown 代码块实体未解码**：`<pre>` 内 `&lt;div&gt;` 输出字面量（与普通文本不一致，浪费 token）。改为「先剥标签、再解码实体」——代码中的实体还原为可读字符
+- **read_url_links 不跟随 meta-refresh 壳页**：read_url 会跟随、read_url_links 只返回跳板页链接——工具间行为不一致。复用 `followMetaRefresh`（fail-open、3 跳有界）
+
+**验证**
+- 单元断言 **123 → 126**（新增 3 条：pre 实体解码、跟随 charset 更新、read_url_links 壳页跟随）+ 12 SPA 断言全绿；probe.mjs 32 PASS 无回归
+
 ## [1.2.0] - 2026-08-22
 
 ### 修复三处隐患 + 进一步省 token

package/README.md

@@ -107,7 +107,7 @@ chars 800+800/12398 · cached
 
 ### Tools
 
-**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body. Handles HTML pages, JSON APIs (fully compact render with long-value clipping) and RSS/Atom feeds (entry list + `feedCount`); paginated articles (novel/news/forum) are auto-joined up to `paginateMax` pages; page metadata (`published`, `author`) is harvested automatically — meta tags first, with a byline fallback from the body text when meta is absent
+**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body. Handles HTML pages, JSON APIs (fully compact render with long-value clipping) and RSS/Atom feeds (entry list + `feedCount`); paginated articles (novel/news/forum) are auto-joined up to `paginateMax` pages; page metadata (`published`, `author`) is harvested automatically — **meta → schema.org JSON-LD → byline, three-tier fallback** (v1.3.0: JSON-LD fills publication time/author on news sites); `<meta http-equiv="refresh">` shell pages (link hops / anti-hotlink relays / no-JS SPA entries) are followed to the real page automatically (≤3 hops, loop-safe, fails open on fetch errors, v1.3.0); pages declaring `<base href>` (frame sites / legacy forums) resolve relative links against it (v1.3.0)
 
 | Param | Type | Default | Description |
 |---|---|---|---|
@@ -210,19 +210,22 @@ const results = await Promise.all([
 ## Token economy (core)
 
 1. **Body text only by default** — no redundant headings/keywords/images/word-count fields; take them via params only when needed;
-2. **Paragraph-aligned truncation + offset continuation** — 6,000 chars by default (~3,000 tokens), cut at paragraph boundaries to keep semantics; output notes a single line `(chars 6000/12990 — truncated, continue via offset)`; resume starts at the given offset, sliced from cache — **no repetition of already-read text** (measured 0+500 → 500+500, no overlap); offset past the end returns empty instead of repeating the head;
-3. **`text` mode first** — Markdown structure is opt-in;
-4. **Compact text render** — the model sees a `title:` header + body directly, no JSON parsing; `siteName` is omitted when identical to the hostname; every status hint is one short line (truncated / cached / rendered), no verbose paragraphs;
-5. **Two-tier cache** — successful results cached per URL for 5 minutes (repeat reads hit cache: fewer network calls and fewer model retries); **failed results cached for 30 seconds** so a broken URL never triggers a re-fetch loop;
-6. **KV-cache friendly (DeepSeek cost tuning)** — tool schema/description stay **static text** (no config values embedded), so changing config never invalidates the reusable prompt prefix and KV cache keeps hitting. DeepSeek's cache-hit tokens cost about 1/10 of misses — the more stable the prefix, the cheaper the run (same analysis as the official `tool-web` docs);
-7. **Batch shares the cache** — `read_url_batch` reuses the same cache (repeat batches hit it directly) and caps each page at 3,000 chars (below the single-page 6,000) to bound total output;
-8. **Compact fixed cost** — the four tool descriptions total ~900 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `&mdash;`/`&hellip;` never waste tokens or render as mojibake.
+2. **Paragraph-aligned truncation + offset continuation** — 6,000 chars by default (~3,000 tokens), cut at paragraph boundaries to keep semantics (text mode separates paragraphs with `\n\n`, so the cut lands on a real boundary, not mid-sentence); output notes a single line `(chars 6000/12990 — truncated, continue via offset)`; resume starts at the given offset, sliced from cache — **no repetition of already-read text** (measured 0+500 → 500+500, no overlap); offset past the end returns empty instead of repeating the head;
+3. **URL fragment reading (v1.4.0)** — `url#section-anchor` jumps straight to the target section of a long reference page: container anchors slice out exactly that block (depth-counted balanced extraction), heading anchors read from the heading onward; the cache key keeps the fragment so `#a`/`#b` never cross-pollute, and offset continues relative to the section start. Measured on python-docs `#str.startswith`: **235,393 → 111,587 chars (-52.6%)**, the default 6000-char window landing on the target method instead of the document head;
+4. **Hidden-element & consent-banner stripping (v1.4.0)** — `display:none`/`visibility:hidden` decoration trees, the bare `hidden` attribute, `aria-hidden="true"`, and OneTrust/Cookiebot/GDPR banners mounted by id no longer leak into the body; a bounded attribute-separator guard keeps compound names (`data-hidden`/`data-id`) from false matches;
+5. **`text` mode first** — Markdown structure is opt-in;
+6. **Compact text render** — the model sees a `title:` header + body directly, no JSON parsing; `siteName` is omitted when identical to the hostname; every status hint is one short line (truncated / cached / rendered), no verbose paragraphs;
+7. **Two-tier cache** — successful results cached per URL for 5 minutes (repeat reads hit cache: fewer network calls and fewer model retries); **failed results cached for 30 seconds** so a broken URL never triggers a re-fetch loop;
+8. **KV-cache friendly (DeepSeek cost tuning)** — tool schema/description stay **static text** (no config values embedded), so changing config never invalidates the reusable prompt prefix and KV cache keeps hitting. DeepSeek's cache-hit tokens cost about 1/10 of misses — the more stable the prefix, the cheaper the run (same analysis as the official `tool-web` docs);
+9. **Batch shares the cache** — `read_url_batch` reuses the same cache (repeat batches hit it directly) and caps each page at 3,000 chars (below the single-page 6,000) to bound total output;
+10. **Compact fixed cost** — the four tool descriptions total ~900 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `&mdash;`/`&hellip;` never waste tokens or render as mojibake.
 
 ## Technical notes
 
 - **Encoding**: BOM-first detection (UTF-8 / UTF-16LE / UTF-16BE — byte-level evidence beats any declared charset), then HTTP `Content-Type` charset → HTML meta; built-in `TextDecoder` transcoding (Node 20+ full-icu, so declared Shift-JIS/EUC-JP/GBK/Big5 all decode), GB2312 normalized to GBK, auto-fallback to UTF-8 on mojibake;
 - **Content dispatch**: a URL is not always an HTML page — JSON APIs are re-rendered fully compact with no indentation, and string values over 1500 chars are clipped with a visible marker (since v1.2.0), RSS 2.0 / Atom feeds parse into an entry list (`title — url` + summary, `feedCount` field, full items via `includeLinks`; entry summaries are iteratively tag-stripped and entity-decoded until stable, so double-escaped `&lt;a&gt;` never leaks as a literal tag); XML sitemaps are rejected with a clear reason (no reading value); everything else falls through to the HTML pipeline;
-- **Extraction**: prefers `<article>` (all articles joined on aggregation pages; an unrelated tiny article — e.g. a newsletter card — falls through to `<main>` when its text is under 200 chars and `<main>` exists) / `<main>` / `role="main"`; `role="main"` containers are closed via **depth counting of balanced tags** (nested divs no longer cut the block at the first `</div>` — measured: gnu.org used to lose 7/8 of its body this way); strips `nav/footer/header/aside/form/iframe` and ad-like containers, heuristic fallback to `<body>`; on the body path a **text-density pass** drops link-dominated segments (related posts, category sidebars, hot-article widgets) — pages with standard containers never reach it;
+- **Extraction**: prefers `<article>` (all articles joined on aggregation pages; an unrelated tiny article — e.g. a newsletter card — falls through to `<main>` when its text is under 200 chars and `<main>` exists) / `<main>` / `role="main"`; `role="main"` containers are closed via **depth counting of balanced tags** (nested divs no longer cut the block at the first `</div>` — measured: gnu.org used to lose 7/8 of its body this way); strips `nav/footer/header/aside/form/iframe`, ad-like containers, and **hidden elements & consent banners** (the `hidden` attribute, `aria-hidden="true"`, styles containing `display:none`/`visibility:hidden`, and GDPR banners mounted under ids like onetrust/cookiebot/cybot/gdpr/consent/cookie-law; a `[\s"']` attribute-separator guard keeps compound names such as `data-hidden`/`data-id` from false matches; unclosed hidden elements degrade safely to keeping the content), heuristic fallback to `<body>`; on the body path a **text-density pass** drops link-dominated segments (related posts, category sidebars, hot-article widgets) — pages with standard containers never reach it;
+- **URL fragment anchoring (v1.4.0)**: `url#section-anchor` targets one section of a long document — container tags (section/div/dl/table…) take the **depth-counted balanced block** (nesting handled correctly, exactly that section), heading/inline anchors (`<h2 id>`/`<dt id>`/`<a name>`) read from the anchor to the end (a heading introduces the content after it); percent-encoded non-ASCII anchors are also tried in decoded form; a miss degrades to the full document. The cache key keeps the fragment (`#a`/`#b` cached as separate entries); a hit skips the readability upgrade and auto-pagination (both override located semantics with whole-document semantics); text mode is **paragraph-structured** (block-level boundaries → `\n\n` paragraph gaps, headings on their own lines — paragraph-aligned truncation actually lands on real boundaries and offset continuation cuts at paragraph seams; trailing permalink glyphs `¶`/`§` are stripped from headings);
 - **Pagination**: auto-follows `rel=next` or a short next-page anchor (下一页 / next / › / » — conservative, no fuzzy guessing); same-host only, loop-guarded, repeated paragraphs across the seam de-duplicated; continuation pages take the fast static path (a paginated SPA chain would cost one full render per page);
 - **Metadata**: `published` / `author` harvested from meta tags into the output and the status line; when the page has no such meta (ruanyifeng.com ships zero author/date tags) a **byline fallback** harvests the body head (600-char window: `作者：X / 日期：Y` — meta wins, deep-body mentions ignored, markdown link syntax stripped); empty-body pages (login walls, JS shells) fall back to `og:description` as a hint instead of nothing;
 - **Markdown**: self-written lightweight tag state machine (headings/paragraphs/lists/blockquotes/code/tables/inline bold-italic-links), zero deps; images with alt text render as `![alt](src)` (decorative empty-alt images are dropped), code fences carry the language hint from `language-*` highlighter classes;
@@ -233,12 +236,14 @@ const results = await Promise.all([
 - **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered) or a JS-redirect shell (few scripts but an empty `<body>`), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); the rendered text is accepted only when it meaningfully beats the static one (≥20 chars from an empty static extraction — short real content is no longer rejected); **bot-challenge defense** — Cloudflare-style interstitials ("Just a moment...") get an extra 8s poll for the auto-redirect, and if the page still looks like a challenge the rendered result is rejected and the static body kept (an interstitial can carry MORE text than the real body — measured 259 vs 135 chars — and must never be mistaken for an improvement); the interstitial DOM never feeds pagination/link extraction either; rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
 - **Boundaries**: login-walled pages are not readable; SPA pages need the Playwright enhancement; **structured data (e.g. which like-count belongs to which comment) is out of text-extraction scope** — this plugin flattens HTML into readable text, so exact field↔value associations are lost; for precise fields, intercept the page's actual data API (see "Real-world validation" below).
 
-## Real-world validation (2026-08-21, v1.0.0; re-verified 2026-08-22 against DSH 0.1.1-rc.2)
+## Real-world validation (2026-08-21, v1.0.0; re-verified 2026-08-22 against DSH 0.1.1-rc.2; 2026-08-24 v1.3.0; 2026-08-28 v1.4.0)
 
-152-site sweep driven by `multi-site.mjs` (committed, re-runnable, 8-way concurrent): **115 OK / 17 expected boundaries (login-walls·captcha·short static pages) / 20 attributed network·anti-bot errors / 0 crashes** (final round with all pre-release fixes in; network-class errors vary ±5 between runs, all environment-attributed). Coverage: CN portals/media, e-commerce (JD/Taobao/Pinduoduo/Suning/Dangdang), video (Bilibili/iQiyi/Youku/Mango), music, games, novels (Qidian/Zongheng/JJWXC legacy GBK), Q&A/forums, government, universities (Tsinghua/PKU/Fudan/SJTU…), Traditional-Chinese TW/HK (PTT/LTN/UDN), JP/KR portals (Yahoo JP/Hatena/goo/naver/daum), overseas tech docs (GitHub/dev.to/react.dev/nodejs.org/rust/go/python), feeds, JSON APIs, encoding stress (GBK/GB2312/Big5/gb18030), anti-bot & network boundaries. All errors are environment-attributed (Wikipedia/Reddit/UDN connect-timeout; W3C/Tieba/NGA/StackOverflow 403; BUPT 412; DNS-fail…) — every one returned as a structured, correctly-attributed error, never a crash.
+152-site sweep driven by `multi-site.mjs` (committed, re-runnable, 8-way concurrent): **115 OK / 17 expected boundaries (login-walls·captcha·short static pages) / 20 attributed network·anti-bot errors / 0 crashes** (final round with all pre-release fixes in; network-class errors vary ±5 between runs, all environment-attributed). Coverage: CN portals/media, e-commerce (JD/Taobao/Pinduoduo/Suning/Dangdang), video (Bilibili/iQiyi/Youku/Mango), music, games, novels (Qidian/Zongheng/JJWXC legacy GBK), Q&A/forums, government, universities (Tsinghua/PKU/Fudan/SJTU…), Traditional-Chinese TW/HK (PTT/LTN/UDN), JP/KR portals (Yahoo JP/Hatena/goo/naver/daum), overseas tech docs (GitHub/dev.to/react.dev/nodejs.org/rust/go/python), feeds, JSON APIs, encoding stress (GBK/GB2312/Big5/gb18030), anti-bot & network boundaries. All errors are environment-attributed (Wikipedia/Reddit/UDN connect-timeout; W3C/Tieba/NGA/StackOverflow 403; BUPT 412; DNS-fail…) — every one returned as a structured, correctly-attributed error, never a crash. v1.3.0 re-run (direct connect, no proxy): **93 OK / 24 THIN+EMPTY / 35 ERR / 0 THREW**, ERR all overseas connect-timeouts and 403/412 anti-bot (baseline-consistent), zero content regressions.
 
 Sweep-driven pre-release fixes (each locked by unit tests): double-escaped RSS descriptions, headerless binary sniffing, JS-redirect shell rendering, **`role="main"` container cut short by nested divs** (gnu.org 165→800 chars), **tiny `<article>` hijacking the main content** (gitlab 71→800 chars), relaxed render-acceptance threshold.
 
+v1.4.0 re-verification (2026-08-28, proxy environment): anchor read measured on python-docs stdtypes.html — `#str.startswith` takes the full 235,393 chars down to a 111,587-char located slice (**-52.6%**), with both text and markdown output starting exactly at the target method definition; 152-site sweep baseline unchanged.
+
 Plus a **15-item DSH acceptance round** (a real agent driving every read_url tool end-to-end): 12 items fully conformant, the 3 findings all closed — ① Cloudflare's "Just a moment..." interstitial carries MORE text than the real body (259 vs 135 chars) and was mistaken for a render improvement → challenge-marker detection + an extra 8s wait for the auto-redirect + interstitial results rejected; ② ruanyifeng.com ships ZERO author/date meta tags, the byline only exists in body text → byline fallback harvesting the body head (verified: `author=阮一峰 published=2026年8月21日`); ③ PDF rejection now attributes clearly: `Unsupported content-type: application/pdf`.
 
 | Category | Sites | Result |
@@ -258,7 +263,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **88 zero-dep assertions** (v1.2.0 adds: sentence-aligned truncation, compact JSON, pagination variants, adversarial-input time bounds, prose &lt; preservation, deep-JSON degradation) (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green;
+- **150 zero-dep assertions** (v1.4.0 adds 24: hidden/consent stripping ×11 — hidden attribute / aria-hidden / style-hidden / onetrust / cybot / gdpr / compound-attribute guard / aria-hidden=false kept; anchor slicing ×8 — container balanced block / heading-to-end / markdown anchoring / miss-degrades-to-full / percent-decoding / data-id guard / e2e cache isolation & offset semantics / bare URL unaffected; paragraph seams & permalink glyphs ×3; thin-page hint ×1; regression ×1; v1.3.1 round 2 adds: pre-block entity decoding, follow-charset propagation, read_url_links shell following; v1.3.1 robustness round adds: meta attribute-order, quoted content values, http-equiv charset, oversized JSON-LD + nested mainEntity, markdown paren escaping, m3u8 crawl noise, bounded oversized attributes; v1.3.0 adds: control-char entity sanitization, JSON-LD metadata, base-href link resolution, meta-refresh following ×11; v1.2.0 adds: sentence-aligned truncation, compact JSON, pagination variants, adversarial-input time bounds, prose &lt; preservation, deep-JSON degradation) (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection, isConcurrencySafe declaration + concurrent cache-race smoke) + **12 SPA-test assertions** all green; probe.mjs adversarial probes +7 (every new v1.4.0 regex time-bounded, slowest 4.9ms);
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap
@@ -276,6 +281,11 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - [x] Parallel tool-call declaration (isConcurrencySafe; multi-source wall-clock time = slowest site, v1.1.0)
 - [x] Separate failure cache, label-based race attribution, sentence-aligned truncation, fully compact JSON render (~21% smaller), merged status flags, arrow-wrapped pagination variants (v1.2.0)
 - [x] Adversarial-page linearization (unclosed-tag/unpaired-lt/attribute-scan bounds + recursion depth cap, fixing a quadratic DoS vector), prose &lt; preservation, proxy-path redirects keep the final URL, deep-JSON degradation instead of crash (v1.2.0 round 2)
+- [x] Control-char entity sanitization (`&#0;`/`&#127;` → space, surrogates → U+FFFD), JSON-LD metadata (meta → JSON-LD → byline), `<base href>` link resolution (frame/legacy-forum pages), `<meta refresh>` shell following (≤3 hops, loop-safe, fail-open, v1.3.0)
+- [x] Robustness round: order-independent meta extraction (content-first no longer missed), recursive JSON-LD nesting (ItemList→mainEntity), legacy `<meta http-equiv="Content-Type">` charset, bounded link/pagination regexes, markdown paren escaping, crawl stream-URL noise filtering (v1.3.1)
+- [x] URL fragment reading (`url#section-anchor`: container balanced-block / heading-to-end, cache isolation, offset relative to the section; python-docs measured -52.6%, v1.4.0)
+- [x] Hidden-element & consent-banner stripping (hidden/aria-hidden/style-hidden + onetrust/cookiebot/gdpr id matching, v1.4.0)
+- [x] Paragraph-structured text output (block boundaries → paragraph gaps, offset continuation cuts at real paragraph boundaries, v1.4.0)
 
 > From v1.0.0 the plugin enters maintenance mode: bug fixes first, updates kept rare.
 

package/README.zh.md

@@ -107,7 +107,7 @@ chars 800+800/12398 · cached
 
 ### 工具
 
-**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — 抓取并提取干净正文。支持 HTML 网页、JSON 接口（紧凑重排渲染）、RSS/Atom 订阅源（条目列表 + `feedCount`）；分页长文（小说/新闻/论坛）自动拼接至 `paginateMax` 页；页面元数据（`published`、`author`）自动提取——meta 标签优先，无 meta 时从正文署名行兜底
+**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — 抓取并提取干净正文。支持 HTML 网页、JSON 接口（紧凑重排渲染）、RSS/Atom 订阅源（条目列表 + `feedCount`）；分页长文（小说/新闻/论坛）自动拼接至 `paginateMax` 页；页面元数据（`published`、`author`）自动提取——**meta → JSON-LD → 正文署名行 三级兜底**（v1.3.0 起解析 schema.org JSON-LD，新闻站发布时间/作者显著补全）；`<meta http-equiv="refresh">` 壳页（链接跳板 / 反盗链中转 / 无 JS 的 SPA 入口）自动跟随到真实页（最多 3 跳、防循环、失败保留原页不报错，v1.3.0）；声明 `<base href>` 的框架/老论坛页，页面内相对链接按 base 解析（v1.3.0）
 
 | 参数 | 类型 | 默认 | 说明 |
 |---|---|---|---|
@@ -210,19 +210,22 @@ const results = await Promise.all([
 ## 省 token 设计（核心）
 
 1. **默认只给正文**——不返回 headings/keywords/images/字数统计等冗余字段，需要时按参数取；
-2. **段落级智能截断 + offset 续读**——默认 6000 字符（约 3000 token），在段落边界截断保证语义完整，输出行仅一行 `(chars 6000/12990 — 截断，offset 续读)` 引导；续读从指定偏移开始、命中缓存切片，**不重复返回已读前文**（实测 0+500 → 500+500，无重复）；offset 越界返回空而非重复开头；
-3. **text 模式优先**——Markdown 结构按需开启；
-4. **紧凑文本 render**——模型直接看到 `title:` 头部 + 正文，无需解析 JSON；`siteName` 与域名相同时省略；状态提示全部一行内（截断/续读/缓存/渲染标记），无长段落废话；
-5. **双层缓存**——成功结果按 URL 缓存 5 分钟（重复读取直接命中，省网络也省模型重试）；**失败结果缓存 30 秒**（坏 URL 不会触发重复 fetch 循环）；
-6. **KV Cache 友好（DeepSeek 成本特调）**——工具 schema/description 保持**静态文本**（不嵌入配置值），配置变更不会使可复用的 prompt 前缀失效，KV 缓存持续命中。DeepSeek 缓存命中 token 价格约为未命中的 1/10，前缀越稳定越省钱（官方 `tool-web` 文档同款分析）；
-7. **批量共用缓存**——`read_url_batch` 内部复用同一套缓存，重复批量读直接命中，且每页默认 3000 字符（低于单页 6000）控制总量；
-8. **固定开销压缩**——4 个工具 description 合计约 900 字符（有断言守卫，保持静态利于 KV 缓存）；HTML 实体解码扩展至 45 个命名实体，`&mdash;`/`&hellip;` 等残留不再浪费 token 或显示为乱码。
+2. **段落级智能截断 + offset 续读**——默认 6000 字符（约 3000 token），在段落边界截断保证语义完整（text 模式段落以 `\n\n` 分隔，切口落在真实段落边界而非句中），输出行仅一行 `(chars 6000/12990 — 截断，offset 续读)` 引导；续读从指定偏移开始、命中缓存切片，**不重复返回已读前文**（实测 0+500 → 500+500，无重复）；offset 越界返回空而非重复开头；
+3. **URL 片段定位阅读（v1.4.0）**——`url#section-anchor` 直接定位长文档目标小节：容器锚点精确切块（深度计数平衡提取）、标题锚点从该处起读；cache key 保留 fragment 保证 `#a`/`#b` 互不串页，offset 相对小节起点。实测 python-docs `#str.startswith`：**235,393 → 111,587 字符（-52.6%）**，默认 6000 字符窗口直接落在目标方法而非文档头；
+4. **隐藏元素与同意弹窗剥离（v1.4.0）**——`display:none`/`visibility:hidden` 装饰树、`hidden` 属性、`aria-hidden="true"`、OneTrust/Cookiebot/GDPR id 挂载的横幅不再混入正文；复合属性名（`data-hidden`/`data-id`）有界卫兵不误伤；
+5. **text 模式优先**——Markdown 结构按需开启；
+6. **紧凑文本 render**——模型直接看到 `title:` 头部 + 正文，无需解析 JSON；`siteName` 与域名相同时省略；状态提示全部一行内（截断/续读/缓存/渲染标记），无长段落废话；
+7. **双层缓存**——成功结果按 URL 缓存 5 分钟（重复读取直接命中，省网络也省模型重试）；**失败结果缓存 30 秒**（坏 URL 不会触发重复 fetch 循环）；
+8. **KV Cache 友好（DeepSeek 成本特调）**——工具 schema/description 保持**静态文本**（不嵌入配置值），配置变更不会使可复用的 prompt 前缀失效，KV 缓存持续命中。DeepSeek 缓存命中 token 价格约为未命中的 1/10，前缀越稳定越省钱（官方 `tool-web` 文档同款分析）；
+9. **批量共用缓存**——`read_url_batch` 内部复用同一套缓存，重复批量读直接命中，且每页默认 3000 字符（低于单页 6000）控制总量；
+10. **固定开销压缩**——4 个工具 description 合计约 900 字符（有断言守卫，保持静态利于 KV 缓存）；HTML 实体解码扩展至 45 个命名实体，`&mdash;`/`&hellip;` 等残留不再浪费 token 或显示为乱码。
 
 ## 技术说明
 
 - **编码**：BOM 优先探测（UTF-8 / UTF-16LE / UTF-16BE——字节级证据优先于任何声明），其次 HTTP `Content-Type` charset → HTML meta；内置 `TextDecoder` 转码（Node 20+ full-icu，页面声明的 Shift-JIS/EUC-JP/GBK/Big5 均可正确解码），GB2312 归一为 GBK，检测到乱码自动回退 UTF-8；
 - **内容类型分发**：URL 不一定是 HTML——JSON 接口紧凑重排渲染（无缩进，超长字符串值 >1500 字符截断并标注，v1.2.0 起缩进 1 格改为完全紧凑），RSS 2.0 / Atom 订阅源解析为条目列表（`标题 — 链接` + 摘要，`feedCount` 字段，`includeLinks` 时附完整 items；条目摘要迭代「剥标签+解实体」直到稳定，双重转义的 `&lt;a&gt;` 不会漏成字面标签）；XML sitemap 明确拒绝（对模型无阅读价值）；其余全部走 HTML 管线；
-- **正文提取**：优先 `<article>`（聚合页多篇合并；无关小卡片 article——如订阅挂件——文本不足 200 字符且页面有 `<main>` 时自动回落 main）/ `<main>` / `role="main"`，`role="main"` 容器用**深度计数找平衡闭合标签**（嵌套 div 不会在第一个 `</div>` 被截断——实测 gnu.org 曾因此只取到 1/8 正文）；剥离 `nav/footer/header/aside/form/iframe` 及广告类容器，启发式回归到 `<body>`；body 路径上追加**文本密度过滤**——丢弃链接主导的短块（相关推荐/分类侧栏/热门文章挂件），标准容器页面完全不走此路径；
+- **正文提取**：优先 `<article>`（聚合页多篇合并；无关小卡片 article——如订阅挂件——文本不足 200 字符且页面有 `<main>` 时自动回落 main）/ `<main>` / `role="main"`，`role="main"` 容器用**深度计数找平衡闭合标签**（嵌套 div 不会在第一个 `</div>` 被截断——实测 gnu.org 曾因此只取到 1/8 正文）；剥离 `nav/footer/header/aside/form/iframe` 及广告类容器、**隐藏元素与同意弹窗**（`hidden` 属性、`aria-hidden="true"`、`style` 含 `display:none`/`visibility:hidden`、`onetrust/cookiebot/cybot/gdpr/consent/cookie-law` 等 id 挂载的 GDPR 横幅；`[\s"']` 属性界卫兵保证 `data-hidden`/`data-id` 复合属性名不误伤；未闭合元素安全降级为保留内容），启发式回归到 `<body>`；body 路径上追加**文本密度过滤**——丢弃链接主导的短块（相关推荐/分类侧栏/热门文章挂件），标准容器页面完全不走此路径；
+- **URL 片段定位（v1.4.0）**：`url#section-anchor` 定位长文档小节——容器标签（section/div/dl/table 等）取**深度计数平衡块**（嵌套正确、精确切出该节），标题/内联锚点（`<h2 id>`/`<dt id>`/`<a name>`）从该处读到文末（标题引出其后正文）；percent-encoded 中文锚点自动尝试解码形态；未命中降级全文。cache key 保留 fragment（`#a`/`#b` 独立缓存）；命中锚点时跳过 readability 升级与自动分页（二者都以全文语义覆盖定位语义）；text 模式**段落化输出**（块级边界 → `\n\n` 段落分隔，标题独立成行，smartTruncate 段落对齐真正生效，offset 续读切口落在真实段落边界；标题行尾永久链接装饰符 `¶`/`§` 剥离）；
 - **分页拼接**：识别 `rel=next`（标准）或纯「下一页 / next / › / »」短锚文本（刻意保守，不做模糊猜测）；同域限定 + 防环；跨页重复段落自动去重；续页走静态快路径（分页 SPA 链每页一次完整渲染不划算）；
 - **元数据**：`published` / `author` 从 meta 标签提取进输出字段与状态行；页面无相关 meta 时（如阮一峰博客零 meta 标签）从**正文头部 600 字符的署名行兜底**（「作者：X / 日期：2026年8月21日」，meta 优先、正文深部提及不误采、markdown 链接语法不泄漏）；空正文页（登录墙/JS 壳）回落到 `og:description` 作为提示，不再返回空；
 - **Markdown**：自研轻量标签状态机（标题/段落/列表/引用/代码块/表格/行内加粗斜体链接），零依赖；带 alt 的图片渲染为 `![alt](src)`（空 alt 装饰图丢弃），代码块围栏带 `language-*` class 里的语言标注；
@@ -233,12 +236,14 @@ const results = await Promise.all([
 - **可选增强二（SPA 页面渲染）**：在 DSH profile 目录执行 `npm i playwright && npx playwright install chromium` 后自动启用。检测到正文为空且（页面脚本密集 疑似 Vue/React 客户端渲染，或 body 为空的 JS 跳转壳——script 数不多但正文全靠跳转）时，自动用无头 Chromium 渲染后再提取（`rendered` 标记告知模型）；渲染结果仅在**显著优于**静态提取时采用（静态为空时 ≥20 字符即接受，防短正文被拒）；**人机验证页防御**——Cloudflare「Just a moment...」等指纹页识别后额外轮询 8s 供其自动跳转，仍未通过则拒绝渲染结果、保留静态正文（指纹页文字量可能超过真实正文，实测 259 vs 135 字符，绝不当作提升），指纹页 DOM 也不污染分页/链接提取；渲染采用 `domcontentloaded` + **DOM 稳定轮询**（内容停止增长即收，上限 10s）而非 `networkidle`——心跳轮询站永不空闲，避免 30s 超时；未安装时优雅提示安装方法、不报错——核心保持零依赖；
 - **边界**：登录墙页面无法读取；SPA 页面需安装 Playwright 增强后渲染读取（未安装时返回明确提示）；**结构化数据（如评论的点赞数归属、榜单数值）不在文本提取范围**——本插件把 HTML 扁平化为可读文本，字段与数值的精确对应关系会丢失；需要精确字段时，用 Playwright 拦截页面实际调用的数据 API 获取（见下方「真实世界验证」）。
 
-## 真实世界验证（2026-08-21，v1.0.0；2026-08-22 适配 DSH 0.1.1-rc.2 复验）
+## 真实世界验证（2026-08-21，v1.0.0；2026-08-22 适配 DSH 0.1.1-rc.2 复验；2026-08-24 v1.3.0 复验；2026-08-28 v1.4.0 复验）
 
-152 站全量实测（`multi-site.mjs` 已提交可复跑，8 并发）：**115 OK / 17 预期边界（登录墙·验证页·静态小页） / 20 网络·反爬归因错误 / 0 崩溃**（含全部发布前修复的终态轮；网络类错误逐轮有 ±5 波动，均为环境归因）。覆盖国内门户 / 媒体 / 电商（京东·淘宝·拼多多·苏宁·当当）/ 视频（B 站·爱奇艺·优酷·芒果）/ 音乐 / 游戏 / 小说（起点·纵横·晋江 legacy GBK）/ 问答 / 论坛 / 政府 / 高校（清北复交等 8 所）/ 港台繁体（PTT·自由时报·联合报）/ 日韩（Yahoo JP·Hatena·goo·naver·daum）/ 海外技术站（GitHub·dev.to·react.dev·nodejs·rust·go·python docs）/ 订阅源 / JSON API / 编码压力（GBK·GB2312·Big5·gb18030）/ 反爬与网络边界。错误全部环境归因（维基/Reddit/UDN 连接超时；W3C/贴吧/NGA/StackOverflow 403；北邮 412；DNS 失败等）——每一个都返回结构化、准确归因的错误，无一崩溃。
+152 站全量实测（`multi-site.mjs` 已提交可复跑，8 并发）：**115 OK / 17 预期边界（登录墙·验证页·静态小页） / 20 网络·反爬归因错误 / 0 崩溃**（含全部发布前修复的终态轮；网络类错误逐轮有 ±5 波动，均为环境归因）。覆盖国内门户 / 媒体 / 电商（京东·淘宝·拼多多·苏宁·当当）/ 视频（B 站·爱奇艺·优酷·芒果）/ 音乐 / 游戏 / 小说（起点·纵横·晋江 legacy GBK）/ 问答 / 论坛 / 政府 / 高校（清北复交等 8 所）/ 港台繁体（PTT·自由时报·联合报）/ 日韩（Yahoo JP·Hatena·goo·naver·daum）/ 海外技术站（GitHub·dev.to·react.dev·nodejs·rust·go·python docs）/ 订阅源 / JSON API / 编码压力（GBK·GB2312·Big5·gb18030）/ 反爬与网络边界。错误全部环境归因（维基/Reddit/UDN 连接超时；W3C/贴吧/NGA/StackOverflow 403；北邮 412；DNS 失败等）——每一个都返回结构化、准确归因的错误，无一崩溃。v1.3.0 复验轮（本机直连，无代理）：**93 OK / 24 THIN+EMPTY / 35 ERR / 0 THREW**，ERR 全部为境外连接超时与 403/412 反爬（与基线一致），无内容性回归。
 
 扫描驱动的发布前修复（全部带单测锁定）：RSS 双重转义、无头二进制嗅探、JS 跳转壳渲染、**`role="main"` 嵌套 div 截断**（gnu.org 165→800 字符）、**小 article 劫持主内容**（gitlab 71→800 字符）、渲染接受门槛放宽。
 
+v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点定位实测——`#str.startswith` 全文 235,393 → 定位切片 111,587 字符（-52.6%），text/markdown 双模式内容均精确起始于目标方法定义；152 站回归基线不变。
+
 另有 **DSH 15 项全功能验收轮**（真实 Agent 调用 read_url 系工具逐项核对）：12 项完全符合，3 项问题全部修复闭环——① Cloudflare「Just a moment...」验证页文字量超过真实正文（259 vs 135 字符）被误当作渲染提升 → 挑战页特征识别 + 渲染后额外等待 8s 自动跳转 + 拒绝接受指纹页；② 阮一峰页面零 author/date meta 标签，作者/日期只在正文里 → byline 兜底从正文头部 600 字符提取署名行（实测 `author=阮一峰 published=2026年8月21日`）；③ PDF 类型拒绝归因 → 明确报 `Unsupported content-type: application/pdf`。
 
 | 类别 | 站点 | 结果 |
@@ -258,7 +263,7 @@ const results = await Promise.all([
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **88 个单元断言**（v1.2.0 新增：长段落句级对齐、紧凑 JSON、分页变体、对抗输入限时、散文 &lt; 保留、深 JSON 降级）（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；
+- **150 个单元断言**（v1.4.0 新增 24：隐藏/consent 剥离 11（hidden 属性/aria-hidden/style 隐藏/onetrust/cybot/gdpr/复合属性卫兵/aria-hidden=false 保留）、锚点切片 8（容器平衡块/标题到文末/markdown 定位/未命中降级/百分号解码/data-id 卫兵/e2e 缓存隔离与 offset 语义/裸 URL 不受影响）、段落接缝与装饰符 3、短正文提示 1、回归 1；v1.3.1 收尾轮新增：pre 代码块实体解码、meta-refresh 跟随 charset 更新、read_url_links 壳页跟随；v1.3.1 健壮性轮新增：meta 属性序颠倒、content 内嵌引号、http-equiv charset、JSON-LD 大块/嵌套 mainEntity、markdown 括号转义、m3u8 噪音、超长属性限时；v1.3.0 新增：控制字符实体防护、JSON-LD 元数据、base href 链接、meta-refresh 跟随 11 条；v1.2.0 新增：长段落句级对齐、紧凑 JSON、分页变体、对抗输入限时、散文 &lt; 保留、深 JSON 降级）（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别、isConcurrencySafe 声明 + 并发缓存竞态烟雾）+ **12 个 SPA 测试断言**全绿；probe.mjs 对抗探针 +7（v1.4.0 新正则全部限时通过，最慢 4.9ms）；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap
@@ -276,13 +281,18 @@ const results = await Promise.all([
 - [x] 并行工具调用声明（isConcurrencySafe，多源阅读墙钟时间按最慢一站计，v1.1.0）
 - [x] 失败缓存独立（不再挤占成功缓存名额）、race 错误按 label 归因、长段落句级对齐截断、JSON 完全紧凑渲染 + 长值截断（约省 21%）、状态行合并、分页箭头变体（v1.2.0）
 - [x] 对抗页面线性化（无配对尖括号/无闭合标签/属性扫描加界 + 递归深度帽，修复 60KB→2.2s 的二次复杂度 DoS 向量）、散文 &lt; 不再被吞、代理重定向保留最终 URL、深 JSON 降级不崩溃（v1.2.0 第二轮）
+- [x] NUL/控制字符实体防护（`&#0;`/`&#127;` 输出空格、surrogate 输出 U+FFFD，不污染模型上下文）、JSON-LD 元数据三级提取（meta → JSON-LD → byline）、`<base href>` 链接解析（框架/老论坛页）、`<meta refresh>` 壳页自动跟随（最多 3 跳、防循环、fail-open，v1.3.0）
+- [x] 健壮性轮：meta 属性序无关提取（content 在前不再漏）、JSON-LD 嵌套结构递归（ItemList→mainEntity）、`<meta http-equiv="Content-Type">` 老式 charset、链接/分页正则统一加界、markdown 括号转义、爬虫流媒体噪音过滤（v1.3.1）
+- [x] URL 片段定位阅读（`url#section-anchor` 容器平衡块/标题到文末、缓存隔离、offset 相对小节，python-docs 实测 -52.6%，v1.4.0）
+- [x] 隐藏元素与同意弹窗剥离（hidden/aria-hidden/style 隐藏 + onetrust/cookiebot/gdpr id 匹配，v1.4.0）
+- [x] text 模式段落化输出（块级边界 → 段落分隔，offset 续读切口落在真实段落边界，v1.4.0）
 
 > v1.0.0 起进入维护期：以修 bug 为主，减少更新频率。
 
 ## 开发
 
 ```bash
-node test.mjs          # 单元自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离/配置钳制）
+node test.mjs          # 单元自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离/配置钳制/控制字符/JSON-LD/base href/meta-refresh）
 
 # SPA 渲染真实测试（需 playwright 已安装，未装自动 SKIP）
 node test-spa.mjs      # 12 断言：JS 正文/渲染后链接/工具不崩溃/缓存隔离/JS 跳转壳