dsh-read-url 0.4.9 → 1.0.0

package/CHANGELOG.md

@@ -1,5 +1,60 @@
 # Changelog
 
+## [1.0.0] - 2026-08-21
+
+### 量大管饱的持久版本：一次把「读网页」这件事做全
+
+目标：省 token 不变的前提下，覆盖绝大多数 URL 类型（HTML / JSON / RSS / Atom / 多页长文 / 各国编码），把后续更新频率降到最低。
+
+**新增：内容类型分发（HTML 不再是唯一入口）**
+- JSON 数据 API（如 `api.github.com`）→ 原生紧凑渲染（`JSON.stringify` 缩进 1 格），不再报 "Unsupported content-type"
+- RSS 2.0 / Atom 订阅源 → 解析为紧凑条目列表（`标题 — 链接 + 摘要`），返回 `feedCount`，`includeLinks` 时附完整 items；XML sitemap 仍明确拒绝（对模型无阅读价值）
+- 修复 proxy-fallback 的 content-type 正则：`application/rss+xml` 分隔符是 `+` 不是 `/`，此前被误拦
+- web seam 路径同样分流：seam 返回体以 `{`/`[` 开头时尝试 JSON 渲染
+
+**新增：分页长文自动拼接（默认 3 页）**
+- 识别 `rel=next`（标准）或纯「下一页 / 下页 / 后一页 / 下一頁 / next / › / » / older entries」短锚文本——刻意保守，不做模糊猜测
+- 同域限定 + 已见 URL 集合防环；续页走静态快路径（分页 SPA 链每页一次完整渲染不划算）；`joinPageText` 去除跨页重复段落/标题（30–300 字符重叠窗口）
+- 输出新增 `paginated` 字段，正文标注 `(已自动拼接 N 页)`；`paginate: false` 关闭，`paginateMax` 可调（1–10）
+- 工具超时预算随之修正：`timeoutMs × (paginateMax + 1) + 45s`（首页 + SPA 渲染 + 续页抓取），随配置缩放
+
+**新增：文本密度兜底（无标准容器页）**
+- 无 `<article>`/`<main>` 而退化到整页 body 的页面：按块级标签切块，丢弃「短且链接主导」（正文 < 300 字符且锚文本占比 > 65%）的相关推荐 / 分类侧栏 / 热门文章挂件
+- 标准容器页面完全不走此路径，零行为变化
+
+**新增：编码增强**
+- BOM 优先探测：UTF-8 / UTF-16LE / UTF-16BE（字节级证据优先于任何声明——UTF-16 body 用 latin1 读 `<meta>` 探针本来就匹配不上）
+- Shift-JIS 等日韩编码：页面声明 charset 即可由内置 TextDecoder 正确解码（实测日文页面全绿）
+
+**新增：网络韧性**
+- 429/503 感知 `Retry-After`（秒形式，封顶 5s 保证协作超时仍约束调用）重试一次
+- 空响应（204/304）沿用 v0.4.9 防护
+
+**新增：Markdown / 元数据增强**
+- `<img>` 带 alt+src → `![alt](src)`（哨兵占位法穿过转义层；空 alt 装饰图丢弃——模型反正看不见像素）
+- 代码块围栏带语言标注（识别 `language-*` / `lang-*` class，```js 远比无标注准确）
+- 页级元数据提取：`published`（article:published_time / og:published_time / pubdate / dc.date 等）、`author`（author / og:article:author 等）进输出字段与状态行
+- 空正文页（登录墙 / JS 壳）回落到 `og:description` 作为提示，不再返回空
+
+**测试**：42 → **60 断言**（+UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 重试、图片 alt、代码语言、元数据、og:description 回落）+ test-spa 10 全绿 + **47 站真实回归**（新增 ruanyifeng Atom / github-blog Atom / v2ex Atom / GitHub API JSON / 起点·开源中国分页候选站）
+
+**发布前实测修复（152 站全量扫描 + 逐站诊断定位）**
+- RSS 描述双重转义（`&lt;a&gt;` 嵌套实体）→ `xmlText` 迭代「剥标签+解实体」直到稳定，正文不再出现字面 `<a href>`
+- 无 Content-Type 头的二进制响应（字节级嗅探：NUL 字节或控制字符占比 > 30%）被当 HTML 解码出乱码 → `looksBinary` 嗅探拒绝，直连与代理路径统一防护
+- JS 跳转壳（script 数低于 SPA 阈值但 body 为空，实测 taptap.cn 形态）不触发渲染 → 空 body 且含任意 `<script>` 即渲染
+- **`role="main"` 容器被非贪婪正则截断**：`<div role="main">` 内嵌套 div 时匹配到第一个 `</div>` 就停（实测 gnu.org 只取到 2700/20000+ 字符，naver.com 只取到 115 字符即 0 文本）→ `tagBlockAt` 深度计数找平衡闭合标签
+- **小 `<article>` 劫持主内容**：页面存在无关小 article（如订阅卡片，实测 about.gitlab.com 71 字符）时优先级高于 `<main>` → article 文本 < 200 字符且页面有 `<main>` 时回落 main
+- **渲染结果接受门槛过严**：静态提取为空时，渲染出 20–50 字符的短正文（JS 壳典型形态）被 `+50` 增量门槛拒绝 → 空静态 + 渲染 ≥ 20 字符即接受
+
+**发布前实测修复二（DSH 15 项全功能验收驱动）**
+- **Cloudflare 人机验证页劫持正文**：挑战过渡页（"Just a moment..."）文字量可超过真实静态正文（实测 259 vs 135 字符），被渲染提升逻辑误当作更优结果 → 三层防御：`looksLikeChallenge` 高特异性特征（cf-chl / cdn-cgi/challenge-platform 等）识别；renderPage 检测到挑战页时额外轮询 8s 等待自动跳转（验证常在数秒内自行通过）；仍是指纹页则拒绝渲染结果、保留静态正文 + 明确提示，且挑战页 DOM 不污染分页/链接提取
+- **元数据 byline 兜底**：无任何 author/date meta 的页面（实测阮一峰博客零 meta 标签）从正文头部 600 字符提取「作者：XXX / 日期：2026年8月21日」等署名行——仅填空缺字段、meta 优先、深部提及不误采、markdown 链接语法不泄漏
+- 测试 PDF 样本 URL 失效（africau.edu sample.pdf 已 404）→ 换 mozilla.github.io 稳定样本，PDF 类型拒绝路径恢复验证（明确报 `Unsupported content-type: application/pdf`）
+
+**测试（发布前终态）**：**77 断言**（+双转义 feed、无头二进制拒绝、嵌套 role=main、薄 article 回落 main、不平衡标签降级、byline 兜底四态、挑战页识别不误伤正文）+ test-spa **12 全绿**（+JS 跳转壳渲染）+ **152 站真实回归**（113 OK / 16 预期边界 THIN / 23 环境类 ERR / 0 崩溃；对比修复前 THIN 20 → 16，gnu.org 165→800 字符、gitlab 71→800 字符）+ **DSH 15 项全功能验收**（12 项完全符合，3 项问题全部修复闭环：验证页防御、byline 字段、PDF 类型归因）
+
+**发布策略**：大版本收敛——此后以修 bug 为主，减少更新频率。
+
 ## [0.4.9] - 2026-08-20
 
 ### 稳定性：配置健壮性 + 严格宿主兼容 + 资源泄漏

package/README.md

@@ -7,7 +7,7 @@
 [![npm](https://img.shields.io/npm/v/dsh-read-url)](https://www.npmjs.com/package/dsh-read-url)
 [![License](https://img.shields.io/github/license/2672243194/dsh-read-url)](https://github.com/2672243194/dsh-read-url/blob/main/LICENSE)
 
-URL reader plugin for DeepSeek Harness: fetch any webpage, **auto-detect encoding (GBK/GB2312/UTF-8/Big5)**, extract the clean main content, and return **token-efficient compact text or structured Markdown**.
+URL reader plugin for DeepSeek Harness: fetch any URL — **webpages (HTML), JSON APIs, RSS/Atom feeds** — auto-detect encoding (UTF-16 BOM / GBK/GB2312 / UTF-8 / Big5 / Shift-JIS), extract the clean main content (**auto-joining paginated articles**), and return **token-efficient compact text or structured Markdown**.
 
 Zero runtime dependencies (Node 20+ built-ins handle fetch/decode/extract), no API key, no server side — install and use.
 
@@ -21,7 +21,9 @@ DSH agents can search (getting links and snippets) but lack the step of "reading
 |---|---|---|---|---|
 | Body cleaning (container-level) | ❌ whole page | ⚠️ tag-level, nav/footer leak in | ⚠️ custom, noisy | ✅ article/main containers + noise stripping |
 | Default output cap | 200,000 chars | 50,000 chars | 30,000 chars | **6,000 chars + paragraph-aligned truncation** |
-| Chinese GBK/GB2312 | provider-dependent | ⚠️ not normalized, GB2312 garbles | ❌ not handled | ✅ normalized + mojibake fallback |
+| Chinese GBK/GB2312 | provider-dependent | ⚠️ not normalized, GB2312 garbles | ❌ not handled | ✅ normalized + mojibake fallback + UTF-16 BOM + Shift-JIS |
+| JSON / RSS / Atom URLs | ❌ HTML only | ❌ | ❌ | ✅ native compact rendering |
+| Paginated articles | ❌ manual per-page calls | ❌ | ❌ | ✅ auto-joined (default 3 pages) |
 | Session-level cache | ❌ | ❌ | ❌ | ✅ 5-min TTL |
 | `ctx.web` seam | ✅ (official core) | ❌ global fetch | ❌ | ✅ seam-first, fallback included |
 | `ctx.effect` unload cleanup | ✅ | ❌ | ❌ | ✅ |
@@ -60,6 +62,7 @@ Just talk to the agent:
 ```
 Read https://example.com/article and summarize the key points
 Read https://docs.example.org/guide in markdown mode
+Read these URLs together and compare their viewpoints: <url1> <url2> <url3>
 ```
 
 ### Real examples (measured)
@@ -103,7 +106,7 @@ chars 800+800/12398 · cached
 
 ### Tools
 
-**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body
+**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body. Handles HTML pages, JSON APIs (compact re-indented render) and RSS/Atom feeds (entry list + `feedCount`); paginated articles (novel/news/forum) are auto-joined up to `paginateMax` pages; page metadata (`published`, `author`) is harvested automatically — meta tags first, with a byline fallback from the body text when meta is absent
 
 | Param | Type | Default | Description |
 |---|---|---|---|
@@ -159,6 +162,8 @@ Override via the profile's `cordis.patch.yml` (defaults in the plugin's own `cor
     maxChars: 6000        # default body truncation
     maxLinks: 20          # read_url_links default count
     spaRender: true       # SPA rendering enhancement (needs playwright installed; degrades with a hint otherwise)
+    paginate: true        # auto-join paginated articles
+    paginateMax: 3        # pages per article, incl. the first (1-10, clamped)
     userAgent: '...'      # request UA
     cacheTtlMs: 300000    # success-cache TTL
     cacheMax: 32          # cache entry cap
@@ -179,8 +184,14 @@ Values are coerced and clamped to sane ranges at load — quoted numbers in YAML
   "truncated": true,
   "charsTotal": 12990,
   "charsReturned": 6000,
+  "charsStart": 0,
   "text": "...",
-  "links": []          // only when includeLinks=true
+  "rendered": true,        // only when extracted after SPA rendering
+  "paginated": 3,          // pages auto-joined (only when > 1)
+  "feedCount": 20,         // RSS/Atom entries (feeds only)
+  "published": "2026-08-01",  // when the page declares it (incl. byline fallback)
+  "author": "...",           // when the page declares it (incl. byline fallback)
+  "links": []               // only when includeLinks=true
 }
 ```
 
@@ -204,38 +215,49 @@ const results = await Promise.all([
 5. **Two-tier cache** — successful results cached per URL for 5 minutes (repeat reads hit cache: fewer network calls and fewer model retries); **failed results cached for 30 seconds** so a broken URL never triggers a re-fetch loop;
 6. **KV-cache friendly (DeepSeek cost tuning)** — tool schema/description stay **static text** (no config values embedded), so changing config never invalidates the reusable prompt prefix and KV cache keeps hitting. DeepSeek's cache-hit tokens cost about 1/10 of misses — the more stable the prefix, the cheaper the run (same analysis as the official `tool-web` docs);
 7. **Batch shares the cache** — `read_url_batch` reuses the same cache (repeat batches hit it directly) and caps each page at 3,000 chars (below the single-page 6,000) to bound total output;
-8. **Compact fixed cost** — the four tool descriptions total ~990 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `&mdash;`/`&hellip;` never waste tokens or render as mojibake.
+8. **Compact fixed cost** — the four tool descriptions total ~900 chars (audited by a budget assertion in tests, kept static for KV-cache); extended HTML-entity decoding (45 named entities) means leftovers like `&mdash;`/`&hellip;` never waste tokens or render as mojibake.
 
 ## Technical notes
 
-- **Encoding**: three-level detection (HTTP `Content-Type` charset → HTML meta → BOM), built-in `TextDecoder` transcoding (Node 20+ full-icu), GB2312 normalized to GBK, auto-fallback to UTF-8 on mojibake;
-- **Extraction**: prefers `<article>` / `role="main"`, strips `nav/footer/header/aside/form/iframe` and ad-like containers, heuristic fallback to `<body>`;
-- **Markdown**: self-written lightweight tag state machine (headings/paragraphs/lists/blockquotes/code/tables/inline bold-italic-links), zero deps;
-- **Safety**: http/https only; no page scripts executed; responses over 3 MB rejected; 15s timeout; structured errors (HTTP status / timeout / unsupported type / DNS cause such as `getaddrinfo ENOTFOUND` vs blocked-network timeout);
+- **Encoding**: BOM-first detection (UTF-8 / UTF-16LE / UTF-16BE — byte-level evidence beats any declared charset), then HTTP `Content-Type` charset → HTML meta; built-in `TextDecoder` transcoding (Node 20+ full-icu, so declared Shift-JIS/EUC-JP/GBK/Big5 all decode), GB2312 normalized to GBK, auto-fallback to UTF-8 on mojibake;
+- **Content dispatch**: a URL is not always an HTML page — JSON APIs are re-rendered compact (indent 1), RSS 2.0 / Atom feeds parse into an entry list (`title — url` + summary, `feedCount` field, full items via `includeLinks`; entry summaries are iteratively tag-stripped and entity-decoded until stable, so double-escaped `&lt;a&gt;` never leaks as a literal tag); XML sitemaps are rejected with a clear reason (no reading value); everything else falls through to the HTML pipeline;
+- **Extraction**: prefers `<article>` (all articles joined on aggregation pages; an unrelated tiny article — e.g. a newsletter card — falls through to `<main>` when its text is under 200 chars and `<main>` exists) / `<main>` / `role="main"`; `role="main"` containers are closed via **depth counting of balanced tags** (nested divs no longer cut the block at the first `</div>` — measured: gnu.org used to lose 7/8 of its body this way); strips `nav/footer/header/aside/form/iframe` and ad-like containers, heuristic fallback to `<body>`; on the body path a **text-density pass** drops link-dominated segments (related posts, category sidebars, hot-article widgets) — pages with standard containers never reach it;
+- **Pagination**: auto-follows `rel=next` or a short next-page anchor (下一页 / next / › / » — conservative, no fuzzy guessing); same-host only, loop-guarded, repeated paragraphs across the seam de-duplicated; continuation pages take the fast static path (a paginated SPA chain would cost one full render per page);
+- **Metadata**: `published` / `author` harvested from meta tags into the output and the status line; when the page has no such meta (ruanyifeng.com ships zero author/date tags) a **byline fallback** harvests the body head (600-char window: `作者：X / 日期：Y` — meta wins, deep-body mentions ignored, markdown link syntax stripped); empty-body pages (login walls, JS shells) fall back to `og:description` as a hint instead of nothing;
+- **Markdown**: self-written lightweight tag state machine (headings/paragraphs/lists/blockquotes/code/tables/inline bold-italic-links), zero deps; images with alt text render as `![alt](src)` (decorative empty-alt images are dropped), code fences carry the language hint from `language-*` highlighter classes;
+- **Safety**: http/https only; no page scripts executed; responses over 3 MB rejected; 15s timeout; **429/503 honored with one Retry-After-aware retry** (capped at 5s so the cooperative timeout still bounds the call); **headerless responses are byte-sniffed** (a NUL byte or >30% control characters marks the body binary — PDFs/images rejected explicitly instead of mojibake through the HTML pipeline); structured errors (HTTP status / timeout / unsupported type / DNS cause such as `getaddrinfo ENOTFOUND` vs blocked-network timeout);
 - **Network fallback (proxy, raced)**: when a proxy is configured (env `HTTPS_PROXY`/`HTTP_PROXY`, falling back to the Windows system proxy registry where Clash-type apps persist it), the plugin starts the direct fetch and a proxy `curl` (`-x` passed explicitly, zero npm deps) **at the same time** and uses whichever completes first — so overseas sites that are blocked on direct connect are served through the user's own proxy in ~0.6s instead of waiting out the direct-connect timeout (~11s measured, -94%). The loser is aborted (curl killed / fetch aborted) and never enters the model context, so **token cost is unchanged**. A failed race returns the original direct error with the proxy attempt noted (`已尝试代理 …`); with no proxy configured the behaviour is exactly the plain direct connect;
 - **Privacy**: the plugin never uses the developer's network configuration — the proxy fallback only reads **your own machine's** proxy (env vars or Windows system proxy) at runtime. No telemetry, no analytics, no data collection: the only outbound action is fetching the URL you asked it to read;
 - **Optional enhancement 1 (Firefox Reader Mode algorithm)**: run `npm i @mozilla/readability happy-dom` in the DSH profile directory to auto-enable `@mozilla/readability` (MPL-2.0, referenced unmodified) for higher-quality extraction; falls back to the built-in heuristic when not installed — the core stays zero-dependency;
-- **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
+- **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered) or a JS-redirect shell (few scripts but an empty `<body>`), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); the rendered text is accepted only when it meaningfully beats the static one (≥20 chars from an empty static extraction — short real content is no longer rejected); **bot-challenge defense** — Cloudflare-style interstitials ("Just a moment...") get an extra 8s poll for the auto-redirect, and if the page still looks like a challenge the rendered result is rejected and the static body kept (an interstitial can carry MORE text than the real body — measured 259 vs 135 chars — and must never be mistaken for an improvement); the interstitial DOM never feeds pagination/link extraction either; rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
 - **Boundaries**: login-walled pages are not readable; SPA pages need the Playwright enhancement; **structured data (e.g. which like-count belongs to which comment) is out of text-extraction scope** — this plugin flattens HTML into readable text, so exact field↔value associations are lost; for precise fields, intercept the page's actual data API (see "Real-world validation" below).
 
-## Real-world validation (2026-08-19, v0.4.8)
+## Real-world validation (2026-08-21, v1.0.0)
 
-41-site sweep driven by `multi-site.mjs` (committed, re-runnable): **23 OK / 8 expected boundaries / 10 network·anti-bot boundaries / 0 crashes** — overseas sites (BBC/V2EX) served in ~1s via the direct+proxy race; doc sites (vuejs.org) fixed via bare-`<main>` picking; wikipedia/httpbin remain clear attributed network boundaries.
+152-site sweep driven by `multi-site.mjs` (committed, re-runnable, 8-way concurrent): **115 OK / 17 expected boundaries (login-walls·captcha·short static pages) / 20 attributed network·anti-bot errors / 0 crashes** (final round with all pre-release fixes in; network-class errors vary ±5 between runs, all environment-attributed). Coverage: CN portals/media, e-commerce (JD/Taobao/Pinduoduo/Suning/Dangdang), video (Bilibili/iQiyi/Youku/Mango), music, games, novels (Qidian/Zongheng/JJWXC legacy GBK), Q&A/forums, government, universities (Tsinghua/PKU/Fudan/SJTU…), Traditional-Chinese TW/HK (PTT/LTN/UDN), JP/KR portals (Yahoo JP/Hatena/goo/naver/daum), overseas tech docs (GitHub/dev.to/react.dev/nodejs.org/rust/go/python), feeds, JSON APIs, encoding stress (GBK/GB2312/Big5/gb18030), anti-bot & network boundaries. All errors are environment-attributed (Wikipedia/Reddit/UDN connect-timeout; W3C/Tieba/NGA/StackOverflow 403; BUPT 412; DNS-fail…) — every one returned as a structured, correctly-attributed error, never a crash.
+
+Sweep-driven pre-release fixes (each locked by unit tests): double-escaped RSS descriptions, headerless binary sniffing, JS-redirect shell rendering, **`role="main"` container cut short by nested divs** (gnu.org 165→800 chars), **tiny `<article>` hijacking the main content** (gitlab 71→800 chars), relaxed render-acceptance threshold.
+
+Plus a **15-item DSH acceptance round** (a real agent driving every read_url tool end-to-end): 12 items fully conformant, the 3 findings all closed — ① Cloudflare's "Just a moment..." interstitial carries MORE text than the real body (259 vs 135 chars) and was mistaken for a render improvement → challenge-marker detection + an extra 8s wait for the auto-redirect + interstitial results rejected; ② ruanyifeng.com ships ZERO author/date meta tags, the byline only exists in body text → byline fallback harvesting the body head (verified: `author=阮一峰 published=2026年8月21日`); ③ PDF rejection now attributes clearly: `Unsupported content-type: application/pdf`.
 
 | Category | Sites | Result |
 |---|---|---|
 | Portal navigation cleaning | Baidu / QQ / NetEase / Sina / Douban / CSDN / Sohu / Ifeng | ✅ clean text, no CSS noise |
-| SPA rendering | Bilibili / Xiaoheihe / Juejin / QQ News / SSPAI | ✅ `rendered` flag, post-JS body |
-| Multi-article aggregation | Cnblogs / Ruan Yifeng blog | ✅ 800+ chars across articles |
-| Static doc pages | MDN / Ruan Yifeng / example.com / GitHub | ✅ clean extraction (example.com = short page, expected) |
-| Login wall | Zhihu / Weibo | ✅ clear content or empty (expected) |
-| **Proxy fallback (overseas)** | BBC Chinese / V2EX | ✅ direct connect blocked → auto-retried through the user's system proxy → clean body |
-| Network / anti-bot boundary | W3C (403 for Chrome UA); Wikipedia (403 via proxy); httpbin (503 service-down); PDF (404); DNS-fail (proxy unreachable) | ✅ accurately attributed errors (HTTP status / 403 / 503 / ENOTFOUND) — not plugin defects |
-| offset continuation | Sina News (12,284 chars) | ✅ 800+800 seamless, cache hit |
+| SPA rendering | Bilibili / Xiaoheihe / Juejin / QQ News / SSPAI / oschina / ThePaper / Xueqiu / Vipshop / TapTap | ✅ `rendered` flag, post-JS body |
+| Multi-article aggregation | Cnblogs / Ruan Yifeng blog / Douban group / Hacker News | ✅ 800+ chars across articles (pagination-joined paginated=2~3) |
+| **Feeds & data APIs (v1.0.0)** | Ruan Yifeng Atom / github.blog Atom / V2EX Atom / Solidot RSS / SSPAI RSS / GitHub API / HN API | ✅ `charset=feed` / `charset=json` compact render |
+| Static doc pages | MDN / Ruan Yifeng / example.com / GitHub / vuejs.org / react.dev / nodejs.org / go.dev / svelte.dev | ✅ clean extraction (example.com = short page, expected) |
+| **Legacy-layout fixes** | gnu.org (nested-div container) / gitlab (tiny-article hijack) | ✅ 800 chars after fix (165 / 71 chars before) |
+| Q&A / forum / encyclopedia | Zhihu column / Hupu / Baidu Zhidao / Baike | ✅ clean text (Zhihu homepage = login-wall, expected) |
+| GBK legacy encoding | ZOL / Dangdang / JJWXC (gb18030) / People.com.cn (GB2312) | ✅ no mojibake |
+| E-commerce / video / music | JD / Taobao / Pinduoduo / Bilibili / iQiyi / NetEase Music / QQ Music | ✅ clean extraction (Amazon CN button page & Kuaishou login-wall, expected) |
+| Government / education | gov.cn / MOE / MIIT / Stats Bureau / 8 universities | ✅ clean; BUPT 412 attributed (WAF) |
+| Network / anti-bot boundary | W3C·Tieba·NGA·StackOverflow (403); Wikipedia·Reddit·UDN (connect-timeout); httpbin (503/binary rejected); DNS-fail (ENOTFOUND) | ✅ accurately attributed errors (HTTP status / timeout / ENOTFOUND) — not plugin defects |
+| offset continuation | Sina News (1,602 chars) | ✅ 800+800 seamless, cache hit |
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **42 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation) + **10 SPA-test assertions** all green;
+- **77 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick, worst-case timeout budgets, yml string coercion + clamping, strict-host seam degradation, UTF-16 BOM, Shift-JIS, density filter, pagination join/cap/disable, JSON render, RSS parse, sitemap rejection, 429 Retry-After retry, image alt, code-fence language, metadata, og:description fallback, double-escaped feed, headerless binary sniff, nested role=main, tiny-article fallback, unbalanced-tag degradation, byline fallback, challenge-page detection) + **12 SPA-test assertions** all green;
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap
@@ -244,13 +266,21 @@ const results = await Promise.all([
 - [x] On-demand SPA rendering (optional Playwright enhancement, auto-enabled once the browser is installed)
 - [x] Batch reading (`read_url_batch`)
 - [x] Recursive site crawl (`read_url_site`)
+- [x] Native JSON / RSS / Atom support (v1.0.0)
+- [x] Auto-joined paginated articles (v1.0.0)
+- [x] UTF-16 BOM / Shift-JIS encoding boost (v1.0.0)
+- [x] Text-density fallback + metadata harvesting (v1.0.0)
+- [x] Bot-challenge defense (Cloudflare interstitial detection + auto-redirect wait + rejection, v1.0.0)
+- [x] Byline fallback for meta-less pages (author / published, v1.0.0)
+
+> From v1.0.0 the plugin enters maintenance mode: bug fixes first, updates kept rare.
 
 ## Development
 
 ```bash
 node test.mjs          # zero-dependency self-tests (charset/extract/markdown/truncate)
-node test-spa.mjs      # SPA rendering tests (10 assertions; SKIPs if playwright absent)
-node multi-site.mjs    # 29-site real-world sweep (needs network): portals/SPA/login-walls/static/anti-bot/net-boundaries
+node test-spa.mjs      # SPA rendering tests (12 assertions; SKIPs if playwright absent)
+node multi-site.mjs    # 152-site real-world sweep (needs network, CONC=8 tunable): portals/SPA/login-walls/static/feeds/JSON/anti-bot/net-boundaries
 
 # End-to-end (requires DSH CLI)
 npx @deepseek-ai/dsh plugin --profile headless add .        # run from the parent dir of the plugin

package/README.zh.md

@@ -7,7 +7,7 @@
 [![npm](https://img.shields.io/npm/v/dsh-read-url)](https://www.npmjs.com/package/dsh-read-url)
 [![License](https://img.shields.io/github/license/2672243194/dsh-read-url)](https://github.com/2672243194/dsh-read-url/blob/main/LICENSE)
 
-DeepSeek Harness 的 URL 阅读插件：抓取任意网页，**自动识别编码（GBK/GB2312/UTF-8/Big5）**，提取干净正文，输出**省 token 的紧凑文本或结构化 Markdown**。
+DeepSeek Harness 的 URL 阅读插件：抓取任意 URL——**网页（HTML）、JSON 接口、RSS/Atom 订阅源**——**自动识别编码（UTF-16 BOM / GBK/GB2312 / UTF-8 / Big5 / Shift-JIS）**，提取干净正文（**分页长文自动拼接**），输出**省 token 的紧凑文本或结构化 Markdown**。
 
 核心零运行时依赖（Node 20+ 内置能力完成抓取/转码/提取），免 API key，免服务端，装完即用。
 
@@ -21,7 +21,9 @@ DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净
 |---|---|---|---|---|
 | 正文净化（容器级提取） | ❌ 整页渲染 | ⚠️ 标签级去噪，nav/footer 仍混入 | ⚠️ 自研，含噪音 | ✅ article/main 容器 + 噪音剥离 |
 | 默认输出上限 | 200000 字符 | 50000 字符 | 30000 字符 | **6000 字符 + 段落级截断** |
-| 中文 GBK/GB2312 | 视 provider | ⚠️ 未归一化，GB2312 易乱码 | ❌ 未处理 | ✅ 归一化 + 乱码回退 |
+| 中文 GBK/GB2312 | 视 provider | ⚠️ 未归一化，GB2312 易乱码 | ❌ 未处理 | ✅ 归一化 + 乱码回退 + UTF-16 BOM + Shift-JIS |
+| JSON / RSS / Atom URL | ❌ 仅 HTML | ❌ | ❌ | ✅ 原生紧凑渲染 |
+| 分页长文 | ❌ 需逐页手动调用 | ❌ | ❌ | ✅ 自动拼接（默认 3 页） |
 | 会话级缓存 | ❌ | ❌ | ❌ | ✅ 5 分钟 TTL |
 | 走 `ctx.web` seam | ✅ 官方本体 | ❌ 全局 fetch | ❌ | ✅ 优先 seam，缺失回退 |
 | `ctx.effect` 卸载清理 | ✅ | ❌ | ❌ | ✅ |
@@ -104,7 +106,7 @@ chars 800+800/12398 · cached
 
 ### 工具
 
-**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — 抓取并提取干净正文
+**`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — 抓取并提取干净正文。支持 HTML 网页、JSON 接口（紧凑重排渲染）、RSS/Atom 订阅源（条目列表 + `feedCount`）；分页长文（小说/新闻/论坛）自动拼接至 `paginateMax` 页；页面元数据（`published`、`author`）自动提取——meta 标签优先，无 meta 时从正文署名行兜底
 
 | 参数 | 类型 | 默认 | 说明 |
 |---|---|---|---|
@@ -160,6 +162,8 @@ chars 800+800/12398 · cached
     maxChars: 6000        # 默认正文截断
     maxLinks: 20          # read_url_links 默认条数
     spaRender: true       # SPA 渲染增强（需 playwright 已安装，未装自动降级提示）
+    paginate: true        # 分页长文自动拼接
+    paginateMax: 3        # 每篇最多拼接页数，含首页（1-10，自动钳制）
     userAgent: '...'      # 请求 UA
     cacheTtlMs: 300000    # 成功缓存 TTL
     cacheMax: 32          # 缓存条目上限
@@ -180,8 +184,14 @@ chars 800+800/12398 · cached
   "truncated": true,
   "charsTotal": 12990,
   "charsReturned": 6000,
+  "charsStart": 0,
   "text": "……",
-  "links": []          // 仅 includeLinks=true 时
+  "rendered": true,        // 仅 SPA 渲染后提取时出现
+  "paginated": 3,          // 自动拼接的页数（仅 > 1 时出现）
+  "feedCount": 20,         // RSS/Atom 条目数（仅订阅源）
+  "published": "2026-08-01", // 页面声明了发布时间时（含署名行兜底）
+  "author": "……",            // 页面声明了作者时（含署名行兜底）
+  "links": []              // 仅 includeLinks=true 时
 }
 ```
 
@@ -205,38 +215,49 @@ const results = await Promise.all([
 5. **双层缓存**——成功结果按 URL 缓存 5 分钟（重复读取直接命中，省网络也省模型重试）；**失败结果缓存 30 秒**（坏 URL 不会触发重复 fetch 循环）；
 6. **KV Cache 友好（DeepSeek 成本特调）**——工具 schema/description 保持**静态文本**（不嵌入配置值），配置变更不会使可复用的 prompt 前缀失效，KV 缓存持续命中。DeepSeek 缓存命中 token 价格约为未命中的 1/10，前缀越稳定越省钱（官方 `tool-web` 文档同款分析）；
 7. **批量共用缓存**——`read_url_batch` 内部复用同一套缓存，重复批量读直接命中，且每页默认 3000 字符（低于单页 6000）控制总量；
-8. **固定开销压缩**——4 个工具 description 合计约 990 字符（有断言守卫，保持静态利于 KV 缓存）；HTML 实体解码扩展至 45 个命名实体，`—`/`…` 等残留不再浪费 token 或显示为乱码。
+8. **固定开销压缩**——4 个工具 description 合计约 900 字符（有断言守卫，保持静态利于 KV 缓存）；HTML 实体解码扩展至 45 个命名实体，`—`/`…` 等残留不再浪费 token 或显示为乱码。
 
 ## 技术说明
 
-- **编码**：HTTP `Content-Type` charset → HTML meta → BOM 三级探测，内置 `TextDecoder` 转码（Node 20+ full-icu），GB2312 归一为 GBK，检测到乱码自动回退 UTF-8；
-- **正文提取**：优先 `<article>` / `role="main"`，剥离 `nav/footer/header/aside/form/iframe` 及广告类容器，启发式回归到 `<body>`；
-- **Markdown**：自研轻量标签状态机（标题/段落/列表/引用/代码块/表格/行内加粗斜体链接），零依赖；
-- **安全**：仅 http/https；不执行页面脚本；响应超 3MB 拒绝；15s 超时；错误信息结构化返回（HTTP 状态/超时/类型不支持/DNS 归因如 `getaddrinfo ENOTFOUND` vs 被墙超时）；
+- **编码**：BOM 优先探测（UTF-8 / UTF-16LE / UTF-16BE——字节级证据优先于任何声明），其次 HTTP `Content-Type` charset → HTML meta；内置 `TextDecoder` 转码（Node 20+ full-icu，页面声明的 Shift-JIS/EUC-JP/GBK/Big5 均可正确解码），GB2312 归一为 GBK，检测到乱码自动回退 UTF-8；
+- **内容类型分发**：URL 不一定是 HTML——JSON 接口紧凑重排渲染（缩进 1 格），RSS 2.0 / Atom 订阅源解析为条目列表（`标题 — 链接` + 摘要，`feedCount` 字段，`includeLinks` 时附完整 items；条目摘要迭代「剥标签+解实体」直到稳定，双重转义的 `&lt;a&gt;` 不会漏成字面标签）；XML sitemap 明确拒绝（对模型无阅读价值）；其余全部走 HTML 管线；
+- **正文提取**：优先 `<article>`（聚合页多篇合并；无关小卡片 article——如订阅挂件——文本不足 200 字符且页面有 `<main>` 时自动回落 main）/ `<main>` / `role="main"`，`role="main"` 容器用**深度计数找平衡闭合标签**（嵌套 div 不会在第一个 `</div>` 被截断——实测 gnu.org 曾因此只取到 1/8 正文）；剥离 `nav/footer/header/aside/form/iframe` 及广告类容器，启发式回归到 `<body>`；body 路径上追加**文本密度过滤**——丢弃链接主导的短块（相关推荐/分类侧栏/热门文章挂件），标准容器页面完全不走此路径；
+- **分页拼接**：识别 `rel=next`（标准）或纯「下一页 / next / › / »」短锚文本（刻意保守，不做模糊猜测）；同域限定 + 防环；跨页重复段落自动去重；续页走静态快路径（分页 SPA 链每页一次完整渲染不划算）；
+- **元数据**：`published` / `author` 从 meta 标签提取进输出字段与状态行；页面无相关 meta 时（如阮一峰博客零 meta 标签）从**正文头部 600 字符的署名行兜底**（「作者：X / 日期：2026年8月21日」，meta 优先、正文深部提及不误采、markdown 链接语法不泄漏）；空正文页（登录墙/JS 壳）回落到 `og:description` 作为提示，不再返回空；
+- **Markdown**：自研轻量标签状态机（标题/段落/列表/引用/代码块/表格/行内加粗斜体链接），零依赖；带 alt 的图片渲染为 `![alt](src)`（空 alt 装饰图丢弃），代码块围栏带 `language-*` class 里的语言标注；
+- **安全**：仅 http/https；不执行页面脚本；响应超 3MB 拒绝；15s 超时；**429/503 感知 Retry-After 重试一次**（封顶 5s，保证协作超时仍约束调用）；**无 Content-Type 头的响应按字节特征嗅探**（NUL 字节或控制字符占比 >30% 判为二进制，PDF/图片等明确拒绝，不进 HTML 管线产出乱码）；错误信息结构化返回（HTTP 状态/超时/类型不支持/DNS 归因如 `getaddrinfo ENOTFOUND` vs 被墙超时）；
 - **网络回退（代理，并发竞速）**：检测到代理时（环境变量 `HTTPS_PROXY`/`HTTP_PROXY` → **Windows 系统代理**注册表，Clash 类软件的真正落点），插件**同时发起**直连 fetch 与代理 curl（`-x` 显式传参，零 npm 依赖），**先完成且成功者胜**——海外站（直连被墙）经用户自己的代理 ~0.6s 读到，不再等直连超时兜底（实测 11s → 633ms，-94%）。输者立即 abort（curl 进程 kill / fetch 中止），**结果从不进入模型上下文，token 消耗零变化**。双方均失败时返回原始直连错误并注明代理尝试（`已尝试代理 …`）；无代理配置时退化为纯直连（行为与 v0.4.3 完全一致）；
 - **隐私**：插件**绝不使用开发者的任何网络配置**——代理回退只在运行时读取**你自己机器**的代理（环境变量或 Windows 系统代理）。无遥测、无统计、无数据收集：唯一的对外动作就是抓取你让它读的那个 URL；
 - **可选增强一（Firefox Reader Mode 算法）**：在 DSH profile 目录执行 `npm i @mozilla/readability happy-dom` 后自动启用，正文提取升级为 `@mozilla/readability`（MPL-2.0，引用不改写），未安装时回退内置启发式提取器，核心保持零依赖；
-- **可选增强二（SPA 页面渲染）**：在 DSH profile 目录执行 `npm i playwright && npx playwright install chromium` 后自动启用。检测到正文为空且页面脚本密集（疑似 Vue/React 客户端渲染）时，自动用无头 Chromium 渲染后再提取（`rendered` 标记告知模型）；渲染采用 `domcontentloaded` + **DOM 稳定轮询**（内容停止增长即收，上限 10s）而非 `networkidle`——心跳轮询站永不空闲，避免 30s 超时；未安装时优雅提示安装方法、不报错——核心保持零依赖；
+- **可选增强二（SPA 页面渲染）**：在 DSH profile 目录执行 `npm i playwright && npx playwright install chromium` 后自动启用。检测到正文为空且（页面脚本密集 疑似 Vue/React 客户端渲染，或 body 为空的 JS 跳转壳——script 数不多但正文全靠跳转）时，自动用无头 Chromium 渲染后再提取（`rendered` 标记告知模型）；渲染结果仅在**显著优于**静态提取时采用（静态为空时 ≥20 字符即接受，防短正文被拒）；**人机验证页防御**——Cloudflare「Just a moment...」等指纹页识别后额外轮询 8s 供其自动跳转，仍未通过则拒绝渲染结果、保留静态正文（指纹页文字量可能超过真实正文，实测 259 vs 135 字符，绝不当作提升），指纹页 DOM 也不污染分页/链接提取；渲染采用 `domcontentloaded` + **DOM 稳定轮询**（内容停止增长即收，上限 10s）而非 `networkidle`——心跳轮询站永不空闲，避免 30s 超时；未安装时优雅提示安装方法、不报错——核心保持零依赖；
 - **边界**：登录墙页面无法读取；SPA 页面需安装 Playwright 增强后渲染读取（未安装时返回明确提示）；**结构化数据（如评论的点赞数归属、榜单数值）不在文本提取范围**——本插件把 HTML 扁平化为可读文本，字段与数值的精确对应关系会丢失；需要精确字段时，用 Playwright 拦截页面实际调用的数据 API 获取（见下方「真实世界验证」）。
 
-## 真实世界验证（2026-08-19，v0.4.8）
+## 真实世界验证（2026-08-21，v1.0.0）
 
-41 站全量实测（`multi-site.mjs` 已提交可复跑）：**23 OK / 8 预期边界 / 10 网络·反爬边界 / 0 崩溃**——海外站（BBC/V2EX）经竞速通道 ~1s 读回；文档站（vuejs.org）经裸 `<main>` 修复后正文聚焦；wikipedia/httpbin 保持清晰的网络边界归因。
+152 站全量实测（`multi-site.mjs` 已提交可复跑，8 并发）：**115 OK / 17 预期边界（登录墙·验证页·静态小页） / 20 网络·反爬归因错误 / 0 崩溃**（含全部发布前修复的终态轮；网络类错误逐轮有 ±5 波动，均为环境归因）。覆盖国内门户 / 媒体 / 电商（京东·淘宝·拼多多·苏宁·当当）/ 视频（B 站·爱奇艺·优酷·芒果）/ 音乐 / 游戏 / 小说（起点·纵横·晋江 legacy GBK）/ 问答 / 论坛 / 政府 / 高校（清北复交等 8 所）/ 港台繁体（PTT·自由时报·联合报）/ 日韩（Yahoo JP·Hatena·goo·naver·daum）/ 海外技术站（GitHub·dev.to·react.dev·nodejs·rust·go·python docs）/ 订阅源 / JSON API / 编码压力（GBK·GB2312·Big5·gb18030）/ 反爬与网络边界。错误全部环境归因（维基/Reddit/UDN 连接超时；W3C/贴吧/NGA/StackOverflow 403；北邮 412；DNS 失败等）——每一个都返回结构化、准确归因的错误，无一崩溃。
+
+扫描驱动的发布前修复（全部带单测锁定）：RSS 双重转义、无头二进制嗅探、JS 跳转壳渲染、**`role="main"` 嵌套 div 截断**（gnu.org 165→800 字符）、**小 article 劫持主内容**（gitlab 71→800 字符）、渲染接受门槛放宽。
+
+另有 **DSH 15 项全功能验收轮**（真实 Agent 调用 read_url 系工具逐项核对）：12 项完全符合，3 项问题全部修复闭环——① Cloudflare「Just a moment...」验证页文字量超过真实正文（259 vs 135 字符）被误当作渲染提升 → 挑战页特征识别 + 渲染后额外等待 8s 自动跳转 + 拒绝接受指纹页；② 阮一峰页面零 author/date meta 标签，作者/日期只在正文里 → byline 兜底从正文头部 600 字符提取署名行（实测 `author=阮一峰 published=2026年8月21日`）；③ PDF 类型拒绝归因 → 明确报 `Unsupported content-type: application/pdf`。
 
 | 类别 | 站点 | 结果 |
 |---|---|---|
 | 门户导航净化 | 百度 / 腾讯 / 网易 / 新浪 / 豆瓣 / CSDN / 搜狐 / 凤凰 | ✅ 干净正文，无 CSS 噪音 |
-| **SPA 渲染** | B 站 / 小黑盒 / 掘金 / QQ 新闻 / 少数派 | ✅ `rendered` 标记 + JS 执行后正文 |
-| 多 article 聚合 | 博客园 / 阮一峰博客 | ✅ 800+ 字符多篇聚合 |
-| 静态文档页 | MDN / 阮一峰 / example.com / GitHub | ✅ 干净提取（example.com 简单页，预期） |
-| 登录墙 | 知乎 / 微博 | ✅ 清晰内容或空（预期） |
-| **代理回退（海外站）** | BBC 中文 / V2EX | ✅ 直连被墙 → 自动经系统代理重试 → 干净正文 |
-| 网络·反爬边界 | W3C（对 Chrome UA 403）；维基（代理 403）；httpbin（503 服务端故障）；PDF（404）；DNS 失败（代理未连通） | ✅ 错误全部准确归因（HTTP 状态/403/503/ENOTFOUND），非插件缺陷 |
-| **offset 续读** | 新浪新闻（12284 字符） | ✅ 800+800 无缝衔接、命中缓存 |
+| **SPA 渲染** | B 站 / 小黑盒 / 掘金 / QQ 新闻 / 少数派 / 开源中国 / 澎湃 / 雪球 / 唯品会 / TapTap | ✅ `rendered` 标记 + JS 执行后正文 |
+| 多 article 聚合 | 博客园 / 阮一峰博客 / 豆瓣小组 / Hacker News | ✅ 800+ 字符多篇聚合（含分页拼接 paginated=2~3） |
+| **订阅源与数据 API（v1.0.0）** | 阮一峰 Atom / github.blog Atom / v2ex Atom / Solidot RSS / 少数派 RSS / GitHub API / HN API | ✅ `charset=feed` / `charset=json` 紧凑渲染 |
+| 静态文档页 | MDN / 阮一峰 / example.com / GitHub / vuejs.org / react.dev / nodejs.org / go.dev / svelte.dev | ✅ 干净提取（example.com 简单页，预期） |
+| **legacy 布局修复** | gnu.org（嵌套 div 容器）/ gitlab（小 article 劫持） | ✅ 修复后 800 字符满额（修复前 165 / 71 字符） |
+| 问答 / 论坛 / 百科 | 知乎专栏 / 虎扑 / 百度知道 / 百科 | ✅ 干净正文（知乎首页登录墙，预期） |
+| GBK 老编码 | 中关村在线 / 当当 / 晋江（gb18030） / 人民网（GB2312） | ✅ 无乱码 |
+| 电商 / 视频 / 音乐 | 京东 / 淘宝 / 拼多多 / B 站 / 爱奇艺 / 网易云 / QQ 音乐 | ✅ 干净提取（亚马逊跳转按钮页、快手登录墙，预期） |
+| 政府 / 教育 | gov.cn / 教育部 / 工信部 / 统计局 / 清华北大等 8 所高校 | ✅ 干净；北邮 412 已归因（WAF） |
+| 网络·反爬边界 | W3C·贴吧·NGA·StackOverflow（403）；维基·Reddit·UDN（连接超时）；httpbin（503/二进制拒绝）；DNS 失败（ENOTFOUND） | ✅ 错误全部准确归因（HTTP 状态/超时/ENOTFOUND），非插件缺陷 |
+| **offset 续读** | 新浪新闻（1602 字符） | ✅ 800+800 无缝衔接、命中缓存 |
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **42 个单元断言**（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级）+ **10 个 SPA 测试断言**全绿；
+- **77 个单元断言**（含实体解码、description/schema 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、裸 main 提取、最坏链路超时预算、yml 字符串强转+钳制、严格宿主 seam 降级、UTF-16 BOM、Shift-JIS、密度过滤、分页拼接/封顶/关闭、JSON 渲染、RSS 解析、sitemap 拒绝、429 Retry-After 重试、图片 alt、代码语言、元数据、og:description 回落、双转义 feed、无头二进制嗅探、嵌套 role=main、薄 article 回落、不平衡标签降级、byline 兜底、人机验证页识别）+ **12 个 SPA 测试断言**全绿；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap
@@ -245,6 +266,14 @@ const results = await Promise.all([
 - [x] SPA 页面按需渲染（可选 Playwright 增强，装浏览器后自动启用）
 - [x] 批量读取（`read_url_batch`）
 - [x] 整站递归爬取（`read_url_site`）
+- [x] JSON / RSS / Atom 原生支持（v1.0.0）
+- [x] 分页长文自动拼接（v1.0.0）
+- [x] UTF-16 BOM / Shift-JIS 编码增强（v1.0.0）
+- [x] 文本密度兜底 + 元数据提取（v1.0.0）
+- [x] 人机验证页防御（Cloudflare 指纹页识别 + 等待自动跳转 + 拒绝误采，v1.0.0）
+- [x] 无 meta 页面署名行兜底（author / published，v1.0.0）
+
+> v1.0.0 起进入维护期：以修 bug 为主，减少更新频率。
 
 ## 开发
 
@@ -252,10 +281,10 @@ const results = await Promise.all([
 node test.mjs          # 单元自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离/配置钳制）
 
 # SPA 渲染真实测试（需 playwright 已安装，未装自动 SKIP）
-node test-spa.mjs      # 10 断言：JS 正文/渲染后链接/工具不崩溃/缓存隔离
+node test-spa.mjs      # 12 断言：JS 正文/渲染后链接/工具不崩溃/缓存隔离/JS 跳转壳
 
-# 29 站真实世界验证（需联网）
-node multi-site.mjs    # 门户/SPA/登录墙/静态/反爬/网络边界，输出分级结果
+# 152 站真实世界验证（需联网，CONC=8 可调并发）
+node multi-site.mjs    # 门户/SPA/登录墙/静态/订阅源/JSON/反爬/网络边界，输出分级结果
 
 # 端到端验证（需已安装 DSH CLI）
 npx @deepseek-ai/dsh plugin --profile headless add .        # 在插件目录的上一级执行

package/cordis.patch.yml

@@ -9,4 +9,6 @@
         cacheTtlMs: 300000
         cacheMax: 32
         spaRender: true
+        paginate: true
+        paginateMax: 3
         userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'

package/index.js

@@ -7,15 +7,18 @@
 //     (temporal composability, docs/cordis-primer.md)
 //   - cooperative tool-call timeout via ToolDefinition.timeoutMs + exec.signal
 // Focus: token economy + smarter extraction for DSH agents.
-//   - charset auto-detect (GBK/GB2312/UTF-8/Big5) via built-in TextDecoder
+//   - charset auto-detect (BOM UTF-16 / GBK/GB2312/UTF-8/Big5/Shift-JIS)
+//   - content dispatch: HTML / JSON APIs / RSS-Atom feeds natively
 //   - clean main-content extraction (heuristic; optional @mozilla/readability upgrade path)
+//     + text-density fallback on the body path
+//   - auto-joined pagination, page metadata (published/author)
 //   - HTML -> Markdown / plain text, paragraph-aligned smart truncation
 //   - session-level cache, compact text render (lowest token cost)
 import { TextDecoder } from 'node:util'
-import { looksLikeSpa, renderPage, closeBrowser } from './spa.js'
-import { detectProxy, fetchViaCurlProxy } from './proxy-fallback.js'
+import { looksLikeSpa, renderPage, closeBrowser, looksLikeChallenge } from './spa.js'
+import { detectProxy, fetchViaCurlProxy, looksBinary } from './proxy-fallback.js'
 // Re-export for tests / programmatic (PTC) cleanup.
-export { closeBrowser, renderPage, looksLikeSpa } from './spa.js'
+export { closeBrowser, renderPage, looksLikeSpa, looksLikeChallenge } from './spa.js'
 
 export const name = 'dsh-read-url'
 export const inject = ['tools']
@@ -29,6 +32,8 @@ const DEFAULTS = {
   cacheTtlMs: 300000,
   cacheMax: 32,
   spaRender: true,
+  paginate: true,
+  paginateMax: 3,
   userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36',
 }
 
@@ -51,12 +56,16 @@ function getDecoder(enc) {
 }
 
 function sniffCharset(buffer, contentType) {
+  // BOM first — byte-level evidence beats any declared charset (a UTF-16 body
+  // with a latin1-read <meta> probe garbles the meta match anyway).
+  if (buffer[0] === 0xef && buffer[1] === 0xbb && buffer[2] === 0xbf) return 'utf-8'
+  if (buffer[0] === 0xff && buffer[1] === 0xfe) return 'utf-16le'
+  if (buffer[0] === 0xfe && buffer[1] === 0xff) return 'utf-16be'
   const m = /charset=["']?([\w-]+)/i.exec(contentType || '')
   if (m) return m[1]
   const head = buffer.subarray(0, 2048).toString('latin1').toLowerCase()
   const meta = /<meta[^>]+charset=["']?([\w-]+)/i.exec(head)
   if (meta) return meta[1]
-  if (buffer[0] === 0xef && buffer[1] === 0xbb && buffer[2] === 0xbf) return 'utf-8'
   return null
 }
 
@@ -77,10 +86,22 @@ export function decodeBuffer(buffer, contentType) {
   return { text, charset: enc }
 }
 
-// Direct-connect fetch, extracted so the race path can run it concurrently
-// with the proxy attempt and abort it. Returns { buffer, contentType, finalUrl }
-// on success or { error } — never throws (abort/timeout/network all mapped).
-async function directFetch(url, signal, cfg) {
+// Content types this plugin can consume. Beyond HTML: JSON (data APIs) and
+// XML (RSS/Atom feeds) are read natively — see readUrl's type dispatch.
+// Note "application/rss+xml": the separator before xml is '+', not '/'.
+const FETCHABLE_CT = /text\/html|application\/xhtml|text\/plain|\/json|[+/]xml/i
+
+// Retry-After header (seconds form) → ms, capped; undefined when absent/invalid.
+function retryAfterMs(res) {
+  const v = res.headers.get('retry-after')
+  if (!v) return undefined
+  const sec = Number(v)
+  if (!Number.isFinite(sec) || sec < 0) return undefined
+  return Math.min(sec * 1000, 5000)
+}
+
+// Single fetch attempt. Never throws; maps abort/timeout/network to { error }.
+async function directFetchOnce(url, signal, cfg) {
   try {
     const res = await fetch(url, {
       redirect: 'follow',
@@ -90,9 +111,19 @@ async function directFetch(url, signal, cfg) {
     // Early-return paths must cancel the body stream, or the connection is
     // held open undrained and cannot return to the keep-alive pool.
     const discardBody = () => { if (res.body) res.body.cancel().catch(() => {}) }
-    if (!res.ok) { discardBody(); return { error: `HTTP ${res.status} ${res.statusText}` } }
+    if (!res.ok) {
+      // Rate-limit / temporary-overload: honor Retry-After with one retry
+      // (capped at 5s so the cooperative timeout still bounds the call).
+      if (res.status === 429 || res.status === 503) {
+        const ra = retryAfterMs(res)
+        discardBody()
+        if (ra !== undefined) return { retryAfterMs: ra }
+      }
+      discardBody()
+      return { error: `HTTP ${res.status} ${res.statusText}` }
+    }
     const contentType = res.headers.get('content-type') || ''
-    if (contentType && !/text\/html|application\/xhtml|text\/plain/i.test(contentType)) {
+    if (contentType && !FETCHABLE_CT.test(contentType)) {
       discardBody()
       return { error: `Unsupported content-type: ${contentType.split(';')[0]}` }
     }
@@ -104,7 +135,12 @@ async function directFetch(url, signal, cfg) {
       if (size > cfg.maxBytes) { discardBody(); return { error: `Page exceeds ${cfg.maxBytes} bytes` } }
       chunks.push(chunk)
     }
-    return { buffer: Buffer.concat(chunks), contentType, finalUrl: res.url }
+    const buffer = Buffer.concat(chunks)
+    // Headerless responses: gate binary bodies out of the HTML pipeline.
+    if (!contentType && looksBinary(buffer)) {
+      return { error: 'Unsupported content-type (no header, binary body)' }
+    }
+    return { buffer, contentType, finalUrl: res.url }
   } catch (e) {
     if (e.name === 'AbortError' || e.name === 'TimeoutError') {
       return { error: `Timeout after ${cfg.timeoutMs}ms or cancelled` }
@@ -117,6 +153,19 @@ async function directFetch(url, signal, cfg) {
   }
 }
 
+// Direct-connect fetch with a single Retry-After-aware retry on 429/503.
+async function directFetch(url, signal, cfg) {
+  const first = await directFetchOnce(url, signal, cfg)
+  if (first && first.retryAfterMs !== undefined && !(signal && signal.aborted)) {
+    await new Promise((resolve) => {
+      const t = setTimeout(resolve, first.retryAfterMs)
+      if (signal) signal.addEventListener('abort', () => { clearTimeout(t); resolve() }, { once: true })
+    })
+    return directFetchOnce(url, signal, cfg)
+  }
+  return first
+}
+
 // Settle-all race that resolves with the FIRST SUCCESSFUL result (a value
 // with a buffer and no error). If every attempt fails it resolves with
 // { success:false, failures:[...] } once the slowest one has settled —
@@ -245,26 +294,136 @@ function textOnly(html) {
     .trim())
 }
 
+// ---- JSON / RSS / Atom content dispatch ----
+// A URL is not always an HTML page: data APIs answer JSON and content sources
+// answer feeds. Both are model-readable when compacted, so read_url handles
+// them natively instead of erroring with "Unsupported content-type".
+
+function xmlText(s) {
+  let t = String(s || '').replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, '$1')
+  // Feeds double-escape HTML in descriptions (&lt;a href=…&gt;查看全文&lt;/a&gt;):
+  // one strip-then-decode pass leaves literal tags in the rendered text
+  // (measured on sspai.com/feed). Iterate strip → decode until stable.
+  for (let i = 0; i < 3; i++) {
+    const prev = t
+    t = decodeTextEntities(t.replace(/<[^>]+>/g, ' '))
+    if (t === prev) break
+  }
+  return t.replace(/\s+/g, ' ').trim()
+}
+
+// RSS 2.0 <item> / Atom <entry> → compact list "title — url\n  summary".
+function parseFeed(xml, limit) {
+  const isAtom = /<feed[\s>]/i.test(xml)
+  const itemRe = isAtom ? /<entry[\s>][\s\S]*?<\/entry\s*>/gi : /<item[\s>][\s\S]*?<\/item\s*>/gi
+  const items = []
+  let m
+  while ((m = itemRe.exec(xml)) && items.length < limit) {
+    const it = m[0]
+    const title = xmlText(/<title[^>]*>([\s\S]*?)<\/title>/i.exec(it)?.[1] || '')
+    const linkTag = /<link[^>]*>([\s\S]*?)<\/link>/i.exec(it)
+    const linkHref = /<link[^>]+href=["']([^"']+)["']/i.exec(it)
+    const link = (linkTag && xmlText(linkTag[1])) || (linkHref && linkHref[1]) || ''
+    const desc = xmlText(/<(?:description|summary|content)[^>]*>([\s\S]*?)<\/(?:description|summary|content)>/i.exec(it)?.[1] || '')
+    if (!title && !link && !desc) continue
+    items.push({ title: title.slice(0, 120), url: link, summary: desc.slice(0, 200) })
+  }
+  const feedTitle = xmlText(/<title[^>]*>([\s\S]*?)<\/title>/i.exec(xml)?.[1] || '')
+  const lines = []
+  for (const it of items) {
+    lines.push(`- ${it.title}${it.url ? ` — ${it.url}` : ''}`)
+    if (it.summary) lines.push(`  ${it.summary}`)
+  }
+  return { title: feedTitle, count: items.length, items, text: lines.join('\n') }
+}
+
+// ---- Text-density fallback (body path only) ----
+// Pages without <article>/<main> degrade to the whole <body>: semantic noise
+// stripping then still leaves link-dominated blocks (related posts, category
+// sidebars, "hot articles" widgets) that carry little reading value. Split the
+// body at block-level open tags and drop segments whose text is mostly links.
+// Standard-container pages (article/main) never reach this code path.
+const BLOCK_OPEN_RE = /<(?:p|div|section|article|main|h[1-6]|table|tbody|tr|ul|ol|li|blockquote|pre|dl|figure|figcaption|fieldset)\b[^>]*>/gi
+
+function linkTextLength(html) {
+  let n = 0
+  const re = /<a[\s>][\s\S]*?<\/a\s*>/gi
+  let m
+  while ((m = re.exec(html))) n += textOnly(m[0]).length
+  return n
+}
+
+export function densityFilter(html) {
+  const marks = []
+  BLOCK_OPEN_RE.lastIndex = 0
+  let m
+  while ((m = BLOCK_OPEN_RE.exec(html))) marks.push(m.index)
+  if (marks.length < 2) return html
+  const segs = []
+  if (marks[0] > 0) segs.push(html.slice(0, marks[0]))
+  for (let k = 0; k < marks.length; k++) {
+    segs.push(html.slice(marks[k], k + 1 < marks.length ? marks[k + 1] : html.length))
+  }
+  const kept = segs.filter((seg) => {
+    const text = textOnly(seg)
+    if (text.length < 2) return false
+    // short link-dominated segment = nav item / recommendation widget entry
+    if (text.length < 300 && linkTextLength(seg) > text.length * 0.65) return false
+    return true
+  })
+  return kept.length ? kept.join('') : html
+}
+
+// Depth-counted block extraction: returns html from the opening tag at
+// openIdx through its BALANCED closing tag. A non-greedy regex stops at the
+// first nested </div> — measured: gnu.org "content" div cut to 2700 chars,
+// naver.com "container" to 115 (zero text). Container divs nest heavily.
+function tagBlockAt(html, openIdx, tagName) {
+  const re = new RegExp(`</?${tagName}(?=[\\s>])`, 'gi')
+  re.lastIndex = openIdx
+  let depth = 0
+  let m
+  while ((m = re.exec(html))) {
+    depth += m[0][1] === '/' ? -1 : 1
+    if (depth === 0) {
+      const close = html.indexOf('>', m.index)
+      return close > 0 ? html.slice(openIdx, close + 1) : null
+    }
+  }
+  return null
+}
+
 export function pickMain(html) {
   // Aggregation pages (e.g. blog homepages) have one <article> per item:
-  // collect all of them instead of only the first.
+  // collect all of them instead of only the first. A tiny <article> (e.g. a
+  // newsletter card — measured on about.gitlab.com: 71 chars of text while
+  // the real content sits in <main>) falls through to <main> when present.
   const articles = [...html.matchAll(/<article[\s>][\s\S]*?<\/article\s*>/gi)]
-  if (articles.length) return articles.map((a) => a[0]).join('\n')
-  const mainTag =
-    /<(main|div)[^>]*role=["']main["'][\s>][\s\S]*?<\/\1\s*>/i.exec(html) ||
-    // Bare <main> without role="main": doc generators (VitePress/VuePress,
-    // MDN, MDX sites) mark the body with a plain <main> tag. Falling back to
-    // the whole <body> would drag in the top/local nav (menu, outline, skip
-    // link) — the extraction quality loss we saw on vuejs.org.
-    /<main[\s>][\s\S]*?<\/main\s*>/i.exec(html)
-  if (mainTag) return mainTag[0]
+  const hasMain = /<main[\s>]/i.test(html)
+  if (articles.length && (!hasMain || textOnly(articles.map((a) => a[0]).join('')).length >= 200)) {
+    return articles.map((a) => a[0]).join('\n')
+  }
+  const roleMain = /<(main|div)[^>]*role=["']main["'][\s>]/i.exec(html)
+  if (roleMain) {
+    const block = tagBlockAt(html, roleMain.index, roleMain[1].toLowerCase())
+    if (block) return block
+  }
+  // Bare <main> without role="main": doc generators (VitePress/VuePress,
+  // MDN, MDX sites) mark the body with a plain <main> tag. Falling back to
+  // the whole <body> would drag in the top/local nav (menu, outline, skip
+  // link) — the extraction quality loss we saw on vuejs.org.
+  const bareMain = /<main[\s>]/i.exec(html)
+  if (bareMain) {
+    const block = tagBlockAt(html, bareMain.index, 'main')
+    if (block) return block
+  }
   const body = /<body[\s>]/i.exec(html)
   if (body) {
     const after = html.slice(body.index)
     const end = after.search(/<\/body>/i)
-    return end > 0 ? after.slice(0, end) : after
+    return densityFilter(end > 0 ? after.slice(0, end) : after)
   }
-  return html
+  return densityFilter(html)
 }
 
 // Some sites (e.g. baidu.com) ship CSS/HTML inside hidden <textarea> with
@@ -290,7 +449,33 @@ function escInline(s) {
   return s.replace(/([\\`*_[\]])/g, '\\$1')
 }
 
+// <img> is a void element (never matches the paired-tag regex in either md
+// walker): convert descriptive images to markdown BEFORE the tag walk, via a
+// sentinel so escInline cannot escape the generated brackets. Decorative
+// images (empty alt) are dropped — the model cannot see pixels anyway.
+const IMG_SENTINEL = '\u0001'
+function imgsToMarkdown(html) {
+  const store = []
+  const out = html.replace(/<img\b[^>]*>/gi, (tag) => {
+    const alt = /alt=["']([^"']*)["']/i.exec(tag)
+    const src = /src=["']([^"']+)["']/i.exec(tag)
+    store.push(src && alt && alt[1].trim() ? `![${alt[1].trim()}](${src[1]})` : '')
+    return `${IMG_SENTINEL}${store.length - 1}${IMG_SENTINEL}`
+  })
+  return {
+    html: out,
+    // Unknown sentinel indexes are PRESERVED (a nested walker's restore must
+    // not eat sentinels owned by an outer frame — blockMd recurses into itself).
+    restore: (s) => s.replace(new RegExp(`${IMG_SENTINEL}(\\d+)${IMG_SENTINEL}`, 'g'), (m, i) => {
+      const v = store[Number(i)]
+      return v === undefined ? m : v
+    }),
+  }
+}
+
 export function inlineMd(html) {
+  const img = imgsToMarkdown(html)
+  html = img.html
   let out = ''
   const re = /<([a-zA-Z0-9]+)((?:"[^"]*"|'[^']*'|[^'">])*)>([\s\S]*?)<\/\1>|<[^>]+>|([^<]+)/g
   let m
@@ -312,11 +497,13 @@ export function inlineMd(html) {
     else if (tag === 'br') out += '  \n'
     else out += inner
   }
-  return out.replace(/[ \t]+/g, ' ').trim()
+  return img.restore(out.replace(/[ \t]+/g, ' ').trim())
 }
 
 export function blockMd(html) {
   let out = ''
+  const img = imgsToMarkdown(html)
+  html = img.html
   const re = /<([a-zA-Z0-9]+)((?:"[^"]*"|'[^']*'|[^'">])*)>([\s\S]*?)<\/\1>|<[^>]+>|([^<]+)/g
   let m
   while ((m = re.exec(html))) {
@@ -345,7 +532,11 @@ export function blockMd(html) {
       out += `\n\n> ${t.replace(/\n/g, '\n> ')}`
     } else if (tag === 'pre') {
       const code = inner.replace(/<[^>]+>/g, '').trim()
-      out += `\n\n\`\`\`\n${code}\n\`\`\``
+      // Language hint from common highlighter conventions: the model reads
+      // ```js fenced blocks far more accurately than unlabelled ones.
+      const lang =
+        /<code[^>]+class=["'][^"']*(?:language|lang)-([\w#+.-]+)["']/i.exec(inner)?.[1] || ''
+      out += `\n\n\`\`\`${lang}\n${code}\n\`\`\``
     } else if (tag === 'code') {
       out += `\`${textOnly(inner)}\``
     } else if (tag === 'ul' || tag === 'ol') {
@@ -388,6 +579,44 @@ export function blockMd(html) {
       if (t) out += t
     }
   }
+  return img.restore(out)
+}
+
+// Page-level metadata the model actually asks about (who wrote it, when) —
+// cheap to harvest from <meta>, previously thrown away.
+function extractMeta(html) {
+  const get = (names) => {
+    for (const n of names) {
+      const m = new RegExp(`<meta[^>]+(?:property|name)=["']${n}["'][^>]+content=["']([^"']+)["']`, 'i').exec(html)
+      if (m) return decodeTextEntities(m[1]).replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim().slice(0, 80)
+    }
+    return ''
+  }
+  return {
+    published: get(['article:published_time', 'og:published_time', 'pubdate', 'publishdate', 'date', 'dc.date']),
+    author: get(['author', 'article:author', 'og:article:author']),
+    description: get(['og:description', 'description']),
+  }
+}
+
+// Meta-less pages (measured: ruanyifeng.com carries ZERO author/date meta
+// tags) still print the byline inside the body head: "作者：阮一峰 日期：
+// 2026年8月21日". Fill ONLY the missing fields, and only from the first
+// 600 chars of body text — deeper matches are content, not bylines.
+const BYLINE_AUTHOR_RE = /(?:作者|責任編輯|責任主编|editor)[：:\s]{1,3}([^\s，,。/|()（）[\]]{2,24})/
+const BYLINE_DATE_RE = /(?:发表日期|发布日期|发表时间|发布时间|日期|发表于|发布于|published(?:\s+on)?)[：:\s]{0,3}(\d{4}[-/.年]\s?\d{1,2}[-/.月]\s?\d{1,2}日?|\d{4}-\d{2}-\d{2}(?:[T ]\d{2}:\d{2})?)/i
+
+function bylineFallback(text, meta) {
+  const head = typeof text === 'string' ? text.slice(0, 600) : ''
+  const out = { published: meta.published, author: meta.author }
+  if (!out.author) {
+    const m = BYLINE_AUTHOR_RE.exec(head)
+    if (m) out.author = m[1].trim().slice(0, 40)
+  }
+  if (!out.published) {
+    const m = BYLINE_DATE_RE.exec(head)
+    if (m) out.published = m[1].replace(/\s+/g, ' ').trim().slice(0, 40)
+  }
   return out
 }
 
@@ -403,11 +632,24 @@ export function extract(html, mode) {
   } else {
     bodyText = textOnly(main).replace(/ +/g, ' ').trim()
   }
+  const meta = extractMeta(html)
+  // Empty-body pages (login walls, JS-only shells) still carry og:description
+  // — surface it as a fallback hint instead of nothing.
+  let text = bodyText
+  let usedDescription = false
+  if (!text && meta.description) {
+    text = meta.description
+    usedDescription = true
+  }
+  const byline = bylineFallback(text, meta)
   return {
     title: titleMatch ? textOnly(titleMatch[1]) || titleMatch[1].trim() : '',
     siteName: siteName ? siteName[1] : '',
     lang: langMatch ? langMatch[1] : '',
-    text: bodyText,
+    published: byline.published,
+    author: byline.author,
+    text,
+    usedDescription,
   }
 }
 
@@ -508,6 +750,53 @@ function hostOf(url) {
   }
 }
 
+// ---- Pagination: auto-follow "next page" ----
+// Long articles on novel/news/forum sites are split into pages; without this
+// the model had to notice the truncation and re-call the tool per page. The
+// recognition is deliberately conservative: rel=next (standard) or a short
+// anchor whose whole text is a next-page marker — no fuzzy guessing.
+const NEXT_TEXT_RE = /^(?:下一页|下页|后一页|下一頁|下頁|next page|next|›{1,3}|»{1,3}|>|older entries)$/i
+
+export function findNextLink(html, baseUrl) {
+  const resolve = (href) => {
+    try {
+      const u = new URL(href, baseUrl)
+      return u.protocol === 'http:' || u.protocol === 'https:' ? u.href : null
+    } catch {
+      return null
+    }
+  }
+  const rel = /<(?:link|a)\b[^>]+rel=["']next["'][^>]*>/i.exec(html || '')
+  if (rel) {
+    const href = /href=["']([^"']+)["']/i.exec(rel[0])
+    const r = href && resolve(href[1])
+    if (r) return r
+  }
+  const re = /<a\b[^>]+href=["']([^"']+)["'][^>]*>([\s\S]*?)<\/a\s*>/gi
+  let m
+  while ((m = re.exec(html || ''))) {
+    const t = textOnly(m[2])
+    if (t.length <= 16 && NEXT_TEXT_RE.test(t)) {
+      const r = resolve(m[1])
+      if (r) return r
+    }
+  }
+  return null
+}
+
+// Join two page bodies, dropping a repeated suffix/prefix overlap (many
+// paginated sites repeat the last paragraph or a heading on the next page).
+function joinPageText(a, b) {
+  const max = Math.min(300, a.length, b.length)
+  for (let n = max; n >= 30; n--) {
+    if (a.slice(a.length - n) === b.slice(0, n)) {
+      const rest = b.slice(n).trim()
+      return rest ? `${a}\n\n${rest}` : a
+    }
+  }
+  return `${a}\n\n${b}`
+}
+
 function sliceFrom(full, offset, maxChars) {
   const s = smartTruncate(full.fullText, maxChars, offset)
   const out = {
@@ -525,6 +814,10 @@ function sliceFrom(full, offset, maxChars) {
   }
   if (full.rendered) out.rendered = true
   if (full.spaHint) out.spaHint = full.spaHint
+  if (full.paginated > 1) out.paginated = full.paginated
+  if (full.feedCount) out.feedCount = full.feedCount
+  if (full.published) out.published = full.published
+  if (full.author) out.author = full.author
   if (Array.isArray(full.links)) out.links = full.links
   return out
 }
@@ -562,27 +855,70 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     }
   }
 
-  let html, finalUrl, charset
-  const viaSeam = await fetchViaWebSeam(ctx, url, externalSignal, cfg)
-  if (viaSeam) {
-    html = viaSeam.html
-    finalUrl = viaSeam.finalUrl
-    charset = 'provider-decoded'
-  } else {
-    const page = await fetchPage(url, externalSignal, cfg)
-    if (page.error) {
-      // Brief error cache: a failing URL usually stays failing for a while,
-      // so serve the same error from cache instead of re-fetching.
-      if (cache.size >= cfg.cacheMax) cache.delete(cache.keys().next().value)
-      cache.set(cacheKey, { time: Date.now(), error: page.error })
-      return { error: page.error }
+  const failWith = (error) => {
+    if (cache.size >= cfg.cacheMax) cache.delete(cache.keys().next().value)
+    cache.set(cacheKey, { time: Date.now(), error })
+    return { error }
+  }
+
+  // Non-HTML payloads (JSON APIs, RSS/Atom feeds) are read natively: fetch,
+  // dispatch by content-type / payload shape, render compact. Falls back to
+  // the HTML pipeline for everything else.
+  const dispatch = async () => {
+    const viaSeam = await fetchViaWebSeam(ctx, url, externalSignal, cfg)
+    if (viaSeam) {
+      const t = viaSeam.html.trim()
+      if (t.startsWith('{') || t.startsWith('[')) {
+        try {
+          return { kind: 'json', text: JSON.stringify(JSON.parse(t), null, 1) }
+        } catch { /* not JSON — keep HTML pipeline */ }
+      }
+      return { kind: 'html', html: viaSeam.html, finalUrl: viaSeam.finalUrl, charset: 'provider-decoded' }
     }
+    const page = await fetchPage(url, externalSignal, cfg)
+    if (page.error) return { error: page.error }
+    const ct = (page.contentType || '').split(';')[0].toLowerCase().trim()
     const decoded = decodeBuffer(page.buffer, page.contentType)
-    html = decoded.text
-    finalUrl = page.finalUrl || url
-    charset = decoded.charset
+    if (/json$/.test(ct)) {
+      try {
+        return { kind: 'json', text: JSON.stringify(JSON.parse(decoded.text), null, 1) }
+      } catch { /* invalid JSON — serve through the HTML pipeline as text */ }
+    }
+    if (/[+/]xml$/.test(ct)) {
+      if (/<urlset[\s>]/i.test(decoded.text) || /<sitemapindex[\s>]/i.test(decoded.text)) {
+        return { error: 'Unsupported content (XML sitemap)' }
+      }
+      if (/<rss[\s>]/i.test(decoded.text) || /<feed[\s>]/i.test(decoded.text)) {
+        return { kind: 'feed', feed: parseFeed(decoded.text, cfg.maxLinks) }
+      }
+    }
+    return { kind: 'html', html: decoded.text, finalUrl: page.finalUrl || url, charset: decoded.charset }
+  }
+  const payload = await dispatch()
+  if (payload.error) return failWith(payload.error)
+
+  if (payload.kind === 'json') {
+    const full = { url, title: '', siteName: hostOf(url), lang: '', charset: 'json', mode: 'json', fullText: payload.text, paginated: 1 }
+    if (cache.size >= cfg.cacheMax) cache.delete(cache.keys().next().value)
+    cache.set(cacheKey, { time: Date.now(), full })
+    return sliceFrom(full, offset, maxChars)
+  }
+  if (payload.kind === 'feed') {
+    const f = payload.feed
+    const full = {
+      url, title: f.title || '', siteName: hostOf(url), lang: '', charset: 'feed',
+      mode: 'feed', fullText: f.text, paginated: 1, feedCount: f.count,
+      links: args.includeLinks === true ? f.items : undefined,
+    }
+    if (cache.size >= cfg.cacheMax) cache.delete(cache.keys().next().value)
+    cache.set(cacheKey, { time: Date.now(), full })
+    return sliceFrom(full, offset, maxChars)
   }
 
+  let html = payload.html
+  let finalUrl = payload.finalUrl || url
+  const charset = payload.charset
+
   let extracted = extract(html, mode)
   // Optional readability upgrade: cleaner article extraction when installed.
   const ext = await getReadabilityExtractor()
@@ -604,24 +940,78 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
 
   // Optional SPA enhancement: if the page looks client-rendered and static
   // extraction found almost nothing, try headless rendering (playwright).
+  // Beyond script-heavy shells (looksLikeSpa, ≥5 <script> tags), a page whose
+  // extraction is EMPTY but carries any <script> may be a JS-redirect shell
+  // (measured: taptap.cn — 2 scripts, empty <body>, redirects via JS) — with
+  // no text to lose, rendering is always worth one bounded attempt. Pages
+  // with no scripts at all cannot change under rendering and are skipped.
   let rendered = false
   let spaHint = ''
   let renderedHtml = null
-  if (cfg.spaRender !== false && looksLikeSpa(html) && (!extracted.text || extracted.text.length < 200)) {
+  const needsRender = !extracted.text || extracted.text.length < 200
+  const scriptShell = !extracted.text && /<script[\s>]/i.test(html)
+  if (cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
     const rr = await renderPage(finalUrl || url, externalSignal)
     if (rr.html) {
       renderedHtml = rr.html
-      const r2 = upgrade(extract(rr.html, mode), rr.html)
-      if (r2.text && r2.text.length > extracted.text.length + 50) {
-        extracted = r2
-        finalUrl = rr.finalUrl || finalUrl
-        rendered = true
+      // A challenge interstitial (Cloudflare "Just a moment...") must never
+      // be accepted as an improvement — it can carry MORE text than the real
+      // static body (measured: 259 chars of verification prose vs 135 chars
+      // of real content on ruanyifeng.com) and would evict the latter.
+      if (looksLikeChallenge(rr.html)) {
+        renderedHtml = null // pagination/links must not read the interstitial DOM
+        spaHint = extracted.text
+          ? '已渲染但被人机验证页拦截，保留静态提取结果'
+          : '已渲染但被人机验证页拦截（无静态内容可用）'
+      } else {
+        const r2 = upgrade(extract(rr.html, mode), rr.html)
+        // Accept the rendered text when it meaningfully beats the static one.
+        // From an EMPTY static extraction even a short body is a real gain
+        // (measured: taptap-style JS shells render 30-60 chars of real content
+        // that the fixed +50 margin would reject).
+        const gain = r2.text.length - extracted.text.length
+        const accept = r2.text && (gain > 50 || (!extracted.text && r2.text.length >= 20))
+        if (accept) {
+          extracted = r2
+          finalUrl = rr.finalUrl || finalUrl
+          rendered = true
+        } else if (!extracted.text) {
+          spaHint = '已渲染仍无可读内容（可能登录墙或反爬拦截）'
+        }
       }
     } else {
       spaHint = rr.error
     }
   }
 
+  // Auto-pagination: follow the "next page" chain (bounded, same-host, no
+  // loops). Continuation pages go through the fast static path — a paginated
+  // SPA chain would cost one full render per page, out of proportion.
+  let paginated = 1
+  const paginateMax = cfg.paginate === false ? 1 : Math.max(1, Math.min(10, Number(cfg.paginateMax) || 3))
+  const startHost = hostOf(finalUrl || url)
+  const seen = new Set([normalizeUrl(finalUrl || url)])
+  let nextUrl = paginateMax > 1 ? findNextLink(renderedHtml || html, finalUrl || url) : null
+  while (
+    nextUrl && paginated < paginateMax &&
+    sameHost(nextUrl, startHost) &&
+    !(externalSignal && externalSignal.aborted)
+  ) {
+    const nu = normalizeUrl(nextUrl)
+    if (!nu || seen.has(nu)) break
+    seen.add(nu)
+    const page = await fetchPage(nextUrl, externalSignal, cfg)
+    if (page.error) break
+    const ct = (page.contentType || '').split(';')[0].toLowerCase().trim()
+    if (!/html|xhtml/.test(ct)) break
+    const nextHtml = decodeBuffer(page.buffer, page.contentType).text
+    const nextEx = extract(nextHtml, mode)
+    if (!nextEx.text || nextEx.text.length < 20) break
+    extracted.text = joinPageText(extracted.text, nextEx.text)
+    paginated++
+    nextUrl = findNextLink(nextHtml, page.finalUrl || nextUrl)
+  }
+
   const full = {
     url: finalUrl,
     title: extracted.title || '',
@@ -632,6 +1022,9 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     fullText: extracted.text,
     rendered,
     spaHint,
+    paginated,
+    published: extracted.published || '',
+    author: extracted.author || '',
     // Links come from the rendered DOM when available — otherwise the SPA
     // chain (content -> links -> next page) would break for JS-only pages.
     links: args.includeLinks === true ? extractLinks(renderedHtml || html, cfg.maxLinks, finalUrl) : undefined,
@@ -652,7 +1045,11 @@ function renderResult(value) {
   const meta = []
   if (r.siteName && r.siteName !== host) meta.push(r.siteName)
   if (r.lang) meta.push(r.lang)
-  if (r.charset) meta.push(`charset ${r.charset}`)
+  if (r.charset && r.charset !== 'json' && r.charset !== 'feed') meta.push(`charset ${r.charset}`)
+  if (r.published) meta.push(r.published)
+  if (r.author) meta.push(`by ${r.author}`)
+  if (r.feedCount) meta.push(`feed · ${r.feedCount} 条`)
+  if (r.mode === 'json') meta.push('json')
   if (meta.length) lines.push(meta.join(' · '))
   if (r.charsStart > 0) {
     lines.push(`(chars ${r.charsStart}+${r.charsReturned}/${r.charsTotal} — offset 续读)`)
@@ -662,6 +1059,7 @@ function renderResult(value) {
     lines.push(`(chars ${r.charsTotal})`)
   }
   if (r.cached) lines.push('(cached)')
+  if (r.paginated > 1) lines.push(`(已自动拼接 ${r.paginated} 页)`)
   if (r.rendered) lines.push('(rendered — JS 执行后内容)')
   if (!r.text) {
     if (r.charsStart > 0 && !r.truncated) {
@@ -737,10 +1135,14 @@ function readLinksTool(ctx, cfg) {
       }
       let links = extractLinks(html, limit, finalUrl)
       // SPA fallback: a JS-only page yields no links from its static HTML;
-      // render it and re-extract when the static result looks empty.
-      if (links.length < 3 && cfg.spaRender !== false && looksLikeSpa(html)) {
+      // render it and re-extract when the static result looks empty. Any
+      // <script> counts as renderable when there are no links at all — the
+      // static shell may be a JS-redirect (same rationale as read_url).
+      if (links.length < 3 && cfg.spaRender !== false && (looksLikeSpa(html) || (links.length === 0 && /<script[\s>]/i.test(html)))) {
         const rr = await renderPage(finalUrl || url, exec && exec.signal)
-        if (rr.html) {
+        // A challenge interstitial's links are the challenge's own scripts —
+        // never an improvement over whatever the static HTML offered.
+        if (rr.html && !looksLikeChallenge(rr.html)) {
           const rl = extractLinks(rr.html, limit, rr.finalUrl || finalUrl)
           if (rl.length > links.length) {
             links = rl
@@ -805,7 +1207,7 @@ function readUrlBatchTool(ctx, cfg) {
       },
       required: ['urls'],
     },
-    // Worst case: ceil(10/4)=3 concurrency waves, each (fetch + SPA render).
+    // Worst case: ceil(10/4)=3 concurrency waves, each (fetch + SPA render).
     timeoutMs: Math.max(60000, cfg.timeoutMs * 3 + 135000),
     output: {
       schema: {
@@ -974,8 +1376,8 @@ function readUrlSiteTool(ctx, cfg) {
       },
       required: ['url'],
     },
-    // Worst case: ceil(maxPages/2)=25 waves of paired fetches (no SPA render
-    // here); the cooperative signal stops the loop early once the budget is hit.
+    // Worst case: ceil(maxPages/2)=25 waves of paired fetches (no SPA render
+    // here); the cooperative signal stops the loop early once the budget is hit.
     timeoutMs: Math.max(120000, cfg.timeoutMs * 25),
     output: {
       schema: {
@@ -1035,8 +1437,10 @@ export function apply(ctx, config) {
   cfg.maxLinks = clamp(cfg.maxLinks, DEFAULTS.maxLinks, 1, 50)
   cfg.cacheTtlMs = clamp(cfg.cacheTtlMs, DEFAULTS.cacheTtlMs, 0, 3600000)
   cfg.cacheMax = clamp(cfg.cacheMax, DEFAULTS.cacheMax, 1, 256)
+  cfg.paginateMax = clamp(cfg.paginateMax, DEFAULTS.paginateMax, 1, 10)
   if (typeof cfg.userAgent !== 'string' || !cfg.userAgent.trim()) cfg.userAgent = DEFAULTS.userAgent
   if (typeof cfg.spaRender !== 'boolean') cfg.spaRender = DEFAULTS.spaRender
+  if (typeof cfg.paginate !== 'boolean') cfg.paginate = DEFAULTS.paginate
 
   // Temporal composability: unload must fully revert side effects.
   ctx.effect(() => () => {
@@ -1065,10 +1469,11 @@ export function apply(ctx, config) {
       },
       required: ['url'],
     },
-    // Pipeline budget must cover the real worst case: full fetch timeout
-    // (cfg.timeoutMs) + SPA render (playwright goto 30s cap + DOM-stability
-    // poll 10s) + margin. Derived from load-time cfg — scale with it, never clamp it.
-    timeoutMs: Math.max(45000, cfg.timeoutMs + 45000),
+    // Pipeline budget must cover the real worst case: first page fetch +
+    // SPA render (goto 30s cap + DOM-stability poll 10s) + up to paginateMax
+    // continuation fetches (static, no render). Derived from load-time cfg —
+    // scale with it, never clamp it.
+    timeoutMs: Math.max(45000, cfg.timeoutMs * (cfg.paginate === false ? 1 : cfg.paginateMax + 1) + 45000),
     output: {
       schema: {
         type: 'object',

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-read-url",
-  "version": "0.4.9",
-  "description": "DeepSeek Harness URL reader: fetch any webpage, auto-detect encoding (GBK/GB2312/UTF-8/Big5), extract clean main content, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in.",
+  "version": "1.0.0",
+  "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in.",
   "type": "module",
   "main": "index.js",
   "files": [

package/proxy-fallback.js

@@ -15,6 +15,23 @@
 // model/user can act (turn the proxy on, or accept the network boundary).
 import { execFile, execFileSync } from 'node:child_process'
 
+// Some servers omit the Content-Type header entirely; the content-type gate
+// then has nothing to test and a binary body (PDF/PNG/zip) would flow into
+// the HTML pipeline as mojibake. Sniff the leading bytes instead: text
+// documents never contain NUL bytes or a run of control characters.
+// Shared by the direct path (index.js) and the curl proxy path below.
+export function looksBinary(buffer) {
+  const n = Math.min(buffer.length, 512)
+  if (!n) return false
+  let ctrl = 0
+  for (let i = 0; i < n; i++) {
+    const b = buffer[i]
+    if (b === 0) return true
+    if (b < 9 || (b > 13 && b < 32)) ctrl++
+  }
+  return ctrl / n > 0.3
+}
+
 let sysProxyCache = null // '' = checked & absent; string = proxy URL
 let sysProxyCheckedAt = 0 // cache timestamp; a failed read is re-tried after 60s
 const SYS_PROXY_RETRY_MS = 60 * 1000
@@ -98,8 +115,15 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
     const [codeStr, ...ctParts] = meta.split(' ')
     const code = Number(codeStr)
     const contentType = ctParts.join(' ')
+    if (code >= 200 && code < 300) {
+      // Headerless responses: gate binary bodies out (same rule as the direct
+      // path — curl writes an empty %{content_type} then).
+      if (!contentType && looksBinary(body)) {
+        return { error: 'Unsupported content-type (no header, binary body)' }
+      }
+    }
     if (!(code >= 200 && code < 300)) return { error: `HTTP ${code}` }
-    if (contentType && !/text\/html|application\/xhtml|text\/plain/i.test(contentType)) {
+    if (contentType && !/text\/html|application\/xhtml|text\/plain|\/json|[+/]xml/i.test(contentType)) {
       return { error: `Unsupported content-type: ${contentType.split(';')[0]}` }
     }
     return { buffer: body, contentType, finalUrl: url, viaProxy: proxy }

package/spa.js

@@ -18,6 +18,17 @@ export function looksLikeSpa(html) {
   return n >= 5
 }
 
+// High-specificity markers of a bot-challenge interstitial (Cloudflare and
+// friends). Measured: ruanyifeng.com served "Just a moment..." whose DOM
+// stabilizes while the challenge sits in "Verification successful. Waiting
+// for ... to respond" — the poll loop returned that transitional page and
+// the caller mistook it for an improvement over static extraction.
+const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s*[|·]?\s*cloudflare|enable javascript and cookies to continue/i
+
+export function looksLikeChallenge(html) {
+  return typeof html === 'string' && CHALLENGE_RE.test(html)
+}
+
 async function getBrowser() {
   if (!browserPromise) {
     // A failed launch (playwright later installed / chromium download finished)
@@ -66,7 +77,21 @@ export async function renderPage(url, externalSignal) {
       if (len === prevLen && prevLen >= 0) break
       prevLen = len
     }
-    const html = await page.content()
+    // A bot-challenge interstitial "stabilizes" the same way real content
+    // does while it waits to redirect. When the DOM settled on a challenge
+    // page, keep polling briefly: the challenge often clears and navigates
+    // on its own within a few seconds (measured: Cloudflare "Verification
+    // successful. Waiting for ... to respond").
+    let html = await page.content()
+    if (looksLikeChallenge(html)) {
+      const t1 = Date.now()
+      while (Date.now() - t1 < 8000) {
+        if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+        await page.waitForTimeout(700)
+        html = await page.content()
+        if (!looksLikeChallenge(html)) break
+      }
+    }
     const finalUrl = page.url()
     return { html, finalUrl }
   } catch (e) {