dsh-read-url 0.4.7 → 0.4.8

package/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## [0.4.8] - 2026-08-19
+
+### 41 站多类型实测 + 提取修复
+
+**新增 12 类站点测试**（29 → 41）：问答（知乎专栏/百度知道）、论坛（虎扑/贴吧）、百科（百度百科）、小说（起点）、政府（gov.cn）、教育（北邮）、视频（B 站视频页）、技术媒体（36 氪/中关村在线 GBK）、技术文档（Vue 文档）
+- 结果：**23 OK / 8 预期边界 / 10 网络·反爬边界 / 0 崩溃**；新增站 gov.cn/知乎专栏/虎扑/百度知道/B 站视频页/中关村在线(GBK) 全部干净提取
+
+**修复（真 bug）**
+- `pickMain` 只匹配 `role="main"` 属性形式，**裸 `<main>` 标签漏选**——VitePress 系文档站（vuejs.org）退化为整页 body，顶部导航（VPLocalNav "菜单/本页目录/赞助位"）混入正文 → 新增裸 `<main>` 匹配分支，文档站正文聚焦、噪音清除（实测 vuejs.org 修复）
+
+**修复（防御性）**
+- `readWindowsSystemProxy` 失败时**永久缓存空代理**——若 PowerShell 子进程被 EDR/策略临时拦截，整个会话都失去代理 → 失败缓存 60s 后允许重新探测
+
+**修复（测试脚本）**
+- `noiseCheck` 的 css 对象正则误报 JS 代码块（`{ data ()`）→ 收紧为 CSS 属性名白名单
+
+**验证**：38 → **39 断言**（+裸 main 优先于 body）+ test-spa 10 全绿；41 站 0 THREW
+
+### README 加真实案例
+- 中英「真实案例」章节：省 token（新浪 800/12398 截断）、竞速（BBC 633ms）、offset 续读（800+800 cached）、批量失败隔离——全部实测数据
+
 ## [0.4.7] - 2026-08-19
 
 ### 省 token：参数 schema 描述精简（固定开销 -9%）

package/README.md

@@ -4,6 +4,9 @@
 
 ![dsh-read-url](docs/banner.svg)
 
+[![npm](https://img.shields.io/npm/v/dsh-read-url)](https://www.npmjs.com/package/dsh-read-url)
+[![License](https://img.shields.io/github/license/2672243194/dsh-read-url)](https://github.com/2672243194/dsh-read-url/blob/main/LICENSE)
+
 URL reader plugin for DeepSeek Harness: fetch any webpage, **auto-detect encoding (GBK/GB2312/UTF-8/Big5)**, extract the clean main content, and return **token-efficient compact text or structured Markdown**.
 
 Zero dependencies (Node 20+ built-ins), no API key, no server side — install and use.
@@ -59,6 +62,45 @@ Read https://example.com/article and summarize the key points
 Read https://docs.example.org/guide in markdown mode
 ```
 
+### Real examples (measured)
+
+**1. Token economy — return only what the model needs**
+
+`read_url` on a portal returns cleaned body capped at `maxChars` (default 6,000) — not the raw page with nav/ads/footers. Repeat reads hit the 5-min cache (`(cached)`), so the agent never re-fetches:
+
+```
+title: 新闻中心首页_新浪网
+charset utf-8
+(chars 800/12398 — 截断，offset 续读)
+```
+
+**2. Overseas sites — direct + proxy race**
+
+When a proxy is configured, the direct fetch and the proxy `curl` start together and the first to complete wins. A blocked overseas site that used to cost ~11s (direct-connect timeout + fallback) now reads in under a second:
+
+```
+BBC 中文: OK (633ms) — clean 4,000+ chars of headline news
+```
+
+**3. Long-article continuation (`offset`)**
+
+A 12,000-char article is read in slices; `offset` resumes from cache without repeating earlier text — the model keeps exactly what it needs in context:
+
+```
+chars 800+800/12398 · cached
+```
+
+**4. Batch research across pages**
+
+`read_url_batch` reads up to 10 pages in parallel (concurrency 4), each cleaned individually, failures isolated:
+
+```
+读取 2/4 页成功，2 页失败
+--- 阮一峰的网络日志 (491 字符) ---
+--- Example Domain (127 字符) ---
+[失败] https://zh.wikipedia.org/... — Fetch failed: HTTP 403 ...
+```
+
 ### Tools
 
 **`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — fetch and extract clean body
@@ -172,9 +214,9 @@ const results = await Promise.all([
 - **Optional enhancement 2 (SPA page rendering)**: run `npm i playwright && npx playwright install chromium` in the DSH profile directory to auto-enable it. When the extracted body is empty and the page is script-heavy (likely Vue/React client-rendered), the plugin automatically renders it with headless Chromium before extracting (a `rendered` flag tells the model); rendering waits for the DOM to stabilize (content stops growing) instead of `networkidle` — heartbeat-polling sites never idle, so this avoids 30s timeouts; when not installed it degrades with a clear install hint, never errors — the core stays zero-dependency;
 - **Boundaries**: login-walled pages are not readable; SPA pages need the Playwright enhancement; **structured data (e.g. which like-count belongs to which comment) is out of text-extraction scope** — this plugin flattens HTML into readable text, so exact field↔value associations are lost; for precise fields, intercept the page's actual data API (see "Real-world validation" below).
 
-## Real-world validation (2026-08-19, v0.4.6)
+## Real-world validation (2026-08-19, v0.4.8)
 
-29-site sweep driven by `multi-site.mjs` (committed, re-runnable): **18 OK / 4 expected boundaries / 7 network boundaries / 0 crashes** — overseas sites (BBC/V2EX) are served through the user's proxy in ~1s via the direct+proxy race; wikipedia/httpbin remain clear attributed network boundaries.
+41-site sweep driven by `multi-site.mjs` (committed, re-runnable): **23 OK / 8 expected boundaries / 10 network·anti-bot boundaries / 0 crashes** — overseas sites (BBC/V2EX) served in ~1s via the direct+proxy race; doc sites (vuejs.org) fixed via bare-`<main>` picking; wikipedia/httpbin remain clear attributed network boundaries.
 
 | Category | Sites | Result |
 |---|---|---|
@@ -189,7 +231,7 @@ const results = await Promise.all([
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
-- **38 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget) + **10 SPA-test assertions** all green;
+- **39 zero-dep assertions** (incl. entity decoding, description-budget guard, link dedupe, table-separator escaping, proxy-fallback function, missing-args tolerance, race logic, empty-race guard, schema budget, bare-main pick) + **10 SPA-test assertions** all green;
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
 ## Roadmap

package/README.zh.md

@@ -4,6 +4,9 @@
 
 ![dsh-read-url](docs/banner.svg)
 
+[![npm](https://img.shields.io/npm/v/dsh-read-url)](https://www.npmjs.com/package/dsh-read-url)
+[![License](https://img.shields.io/github/license/2672243194/dsh-read-url)](https://github.com/2672243194/dsh-read-url/blob/main/LICENSE)
+
 DeepSeek Harness 的 URL 阅读插件：抓取任意网页，**自动识别编码（GBK/GB2312/UTF-8/Big5）**，提取干净正文，输出**省 token 的紧凑文本或结构化 Markdown**。
 
 零依赖（Node 20+ 内置能力），免 API key，免服务端，装完即用。
@@ -60,6 +63,45 @@ npx @deepseek-ai/dsh plugin --profile web add ./dsh-read-url
 同时读一下这几个网址，对比它们的观点：<url1> <url2> <url3>
 ```
 
+### 真实案例（实测数据）
+
+**1. 省 token——只返回模型真正需要的内容**
+
+门户页 `read_url` 返回按 `maxChars`（默认 6000）截断的干净正文——不是带导航/广告/页脚的原始页面。重复读取命中 5 分钟会话缓存（`(cached)`），模型不会重复抓取：
+
+```
+title: 新闻中心首页_新浪网
+charset utf-8
+(chars 800/12398 — 截断，offset 续读)
+```
+
+**2. 海外站——直连 + 代理并发竞速**
+
+检测到代理时，直连与代理 `curl` 同时发起、先完成者胜。原本被墙的海外站（直连超时 + 回退约 11s）现在 1 秒内读到：
+
+```
+BBC 中文: OK (633ms) — 4000+ 字符干净头条新闻
+```
+
+**3. 长文续读（`offset`）**
+
+1.2 万字符的文章分片读取，`offset` 从缓存续读且不重复前文——模型上下文只保留需要的部分：
+
+```
+chars 800+800/12398 · cached
+```
+
+**4. 跨页批量研究**
+
+`read_url_batch` 并行读最多 10 页（并发 4），逐页净化、失败隔离：
+
+```
+读取 2/4 页成功，2 页失败
+--- 阮一峰的网络日志 (491 字符) ---
+--- Example Domain (127 字符) ---
+[失败] https://zh.wikipedia.org/... — Fetch failed: HTTP 403 ...
+```
+
 ### 工具
 
 **`read_url(url, maxChars?, offset?, mode?, includeLinks?)`** — 抓取并提取干净正文
@@ -173,9 +215,9 @@ const results = await Promise.all([
 - **可选增强二（SPA 页面渲染）**：在 DSH profile 目录执行 `npm i playwright && npx playwright install chromium` 后自动启用。检测到正文为空且页面脚本密集（疑似 Vue/React 客户端渲染）时，自动用无头 Chromium 渲染后再提取（`rendered` 标记告知模型）；渲染采用 `domcontentloaded` + **DOM 稳定轮询**（内容停止增长即收，上限 10s）而非 `networkidle`——心跳轮询站永不空闲，避免 30s 超时；未安装时优雅提示安装方法、不报错——核心保持零依赖；
 - **边界**：登录墙页面无法读取；SPA 页面需安装 Playwright 增强后渲染读取（未安装时返回明确提示）；**结构化数据（如评论的点赞数归属、榜单数值）不在文本提取范围**——本插件把 HTML 扁平化为可读文本，字段与数值的精确对应关系会丢失；需要精确字段时，用 Playwright 拦截页面实际调用的数据 API 获取（见下方「真实世界验证」）。
 
-## 真实世界验证（2026-08-19，v0.4.6）
+## 真实世界验证（2026-08-19，v0.4.8）
 
-29 站全量实测（`multi-site.mjs` 已提交可复跑）：**18 OK / 4 预期边界 / 7 网络·反爬边界 / 0 崩溃**——海外站（BBC/V2EX）经竞速通道 ~1s 读回；wikipedia/httpbin 保持清晰的网络边界归因。
+41 站全量实测（`multi-site.mjs` 已提交可复跑）：**23 OK / 8 预期边界 / 10 网络·反爬边界 / 0 崩溃**——海外站（BBC/V2EX）经竞速通道 ~1s 读回；文档站（vuejs.org）经裸 `<main>` 修复后正文聚焦；wikipedia/httpbin 保持清晰的网络边界归因。
 
 | 类别 | 站点 | 结果 |
 |---|---|---|
@@ -190,7 +232,7 @@ const results = await Promise.all([
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
-- **38 个零依赖断言**（含实体解码、description 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、schema 预算）+ **10 个 SPA 测试断言**全绿；
+- **39 个零依赖断言**（含实体解码、description 预算守卫、链接去重、表格分隔行转义、代理回退函数、空参容错、竞速逻辑、空竞速守卫、schema 预算、裸 main 提取）+ **10 个 SPA 测试断言**全绿；
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
 ## Roadmap

package/index.js

@@ -241,7 +241,13 @@ export function pickMain(html) {
   // collect all of them instead of only the first.
   const articles = [...html.matchAll(/<article[\s>][\s\S]*?<\/article\s*>/gi)]
   if (articles.length) return articles.map((a) => a[0]).join('\n')
-  const mainTag = /<(main|div)[^>]*role=["']main["'][\s>][\s\S]*?<\/\1\s*>/i.exec(html)
+  const mainTag =
+    /<(main|div)[^>]*role=["']main["'][\s>][\s\S]*?<\/\1\s*>/i.exec(html) ||
+    // Bare <main> without role="main": doc generators (VitePress/VuePress,
+    // MDN, MDX sites) mark the body with a plain <main> tag. Falling back to
+    // the whole <body> would drag in the top/local nav (menu, outline, skip
+    // link) — the extraction quality loss we saw on vuejs.org.
+    /<main[\s>][\s\S]*?<\/main\s*>/i.exec(html)
   if (mainTag) return mainTag[0]
   const body = /<body[\s>]/i.exec(html)
   if (body) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "0.4.7",
+  "version": "0.4.8",
   "description": "DeepSeek Harness URL reader: fetch any webpage, auto-detect encoding (GBK/GB2312/UTF-8/Big5), extract clean main content, output compact text/Markdown to save tokens. Zero runtime dependencies.",
   "type": "module",
   "main": "index.js",

package/proxy-fallback.js

@@ -16,14 +16,21 @@
 import { execFile, execFileSync } from 'node:child_process'
 
 let sysProxyCache = null // '' = checked & absent; string = proxy URL
+let sysProxyCheckedAt = 0 // cache timestamp; a failed read is re-tried after 60s
+const SYS_PROXY_RETRY_MS = 60 * 1000
 
 // Windows system proxy from the registry (HKCU Internet Settings). PowerShell
 // is used because `reg.exe` is frequently blocked by security policy; the
 // read is synchronous and cached after the first call (it only runs after a
 // direct-connect failure, so the ~1s cost is invisible on the happy path).
+// A FAILED read is cached as '' but re-tried after 60s — an environment that
+// transiently blocks the PowerShell child (EDR, policy) should not starve
+// the whole session of the proxy.
 function readWindowsSystemProxy() {
-  if (sysProxyCache !== null) return sysProxyCache
+  const now = Date.now()
+  if (sysProxyCache !== null && now - sysProxyCheckedAt < SYS_PROXY_RETRY_MS) return sysProxyCache
   sysProxyCache = ''
+  sysProxyCheckedAt = now
   try {
     const out = execFileSync(
       'powershell',
@@ -38,10 +45,11 @@ function readWindowsSystemProxy() {
     // ProxyServer may be "host:port" or "http=...;https=host:port"
     const httpsPart = server.split(';').map((s) => s.trim()).find((s) => /^https=/i.test(s))
     sysProxyCache = (httpsPart ? httpsPart.replace(/^https=/i, '') : server) || ''
+    return sysProxyCache
   } catch {
     sysProxyCache = ''
+    return sysProxyCache
   }
-  return sysProxyCache
 }
 
 // The user's proxy, in priority order: env vars, then Windows system proxy.