dsh-read-url 1.9.0 → 1.9.1

package/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## [1.9.1] - 2026-09-28
+
+### 修复
+
+- 链接、分页、base 与 meta-refresh 按真实属性读取，支持等号空格和合法无引号属性；忽略 data-href/data-rel 等伪属性，识别 rel 中的 next 标记。
+- meta-refresh 正确解码实体并保留 URL 中的分号及查询参数，避免跳转地址被截断。
+- 有界标签扫描识别引号内的 `>`，空 href 不吞掉后续锚点；锚点正文单向扫描，避免损坏 HTML 触发反复搜索。
+- 链接扫描与站点爬取在取消后返回 cancelled，避免返回旧跳转壳或部分结果作为完整成功；Retry-After 等待结束或取消时清理监听与计时器。
+- 站点正文摘要按 maxCharsPerPage 显示，取消渲染层额外的 80 字符截断；默认仍只输出站点结构。
+- 对抗探针等待异步检查完成，异常设置非零退出码，避免未完成的 Promise 被打印为通过。
+
+### 验证
+
+- 352 个零依赖单元断言、15 个真实 Chromium SPA 断言、59 项对抗探针通过；新增 30 条单元检查与 6 条未闭合锚点探针。原始维护缺陷在 v1.8.1/v1.9.0 同输入均可复现，属于既有问题。
+- 152 站最终实网回归：103 OK / 20 THIN / 5 EMPTY / 24 ERR / 0 THREW，OK 站零噪声。Vue、React、酷狗及 Hatena 同 HTML 的全文续读与 800 字输出对照 v1.9.0 一致；微博动态波动经同 HTML/DOM 对照一致，错误站点集合未变。
+- DSH 0.1.7-rc.2 官方包字节差异及社区升级卡（截至 rc.1）交叉核查完成；v1.9.0 和当前维护代码各通过 17 项真实官方服务集成检查，发布版 patch 另通过 4 项真实加载器检查。已核查接口无需适配修改；受控 provider 检查不等同于完整宿主 UI/模型端到端验证。
+- 未闭合且含 href 的锚点 2.5 万/5 万/10 万输入本机约 27/57/118 ms；保留有界属性策略与零运行时依赖。四工具 description/schema 仍为 1,159/1,876 字符。
+
 ## [1.9.0] - 2026-09-19
 
 ### 新增

package/README.md

@@ -315,6 +315,14 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - Vue, React and Kugou's known NOISE cases matched v1.8.1 exactly using identical HTML and optional dependencies, in both text/Markdown and full/800-character outputs. Errors were network, anti-bot, invalid-address or expected content-type boundaries.
 - All four tools enforce the explicit origin policy, including redirect/crawl boundaries, cache isolation and visible static-reading hints. Model input remains **1,159 description characters / 1,876 schema characters**, within the 1,250 / 2,000 budgets.
 
+### v1.9.1 maintenance validation (2026-09-28)
+
+- **352 zero-dependency unit assertions, 15 real Chromium SPA assertions and 59 adversarial probes** passed. Navigation tests cover real attributes, quoted `>` characters, empty anchors, refresh entities and semicolons; cancellation, retry cleanup and the rendered site-summary limit have regression coverage. Asynchronous probes now await completion and report failures through the exit code.
+- The final 152-site sweep returned **103 OK / 25 THIN+EMPTY / 24 ERR / 0 THREW**, with no noise in OK results. Vue, React, Kugou and Hatena matched v1.9.0 using identical HTML, including full continuation and 800-character outputs. Weibo's variable static/SPA results also matched on identical HTML/DOM snapshots; the error-site set was unchanged.
+- Navigation uses bounded, quote-aware attribute scanning and advances through each anchor body once. Runs with 25k/50k/100k unclosed anchors containing `href` took about **27/57/118 ms** on this machine. The existing 1,000-character attribute boundary remains in place.
+- [DSH 0.1.7-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.7-rc.2) compatibility was checked against official npm package byte differences and community upgrade notes (available through rc.1). Both published v1.9.0 and this maintenance code passed **17 actual official-service integration checks each**; the published patch passed **4 loader checks** covering loading, execution, configuration remount and removal. Web-provider/runtime contracts remain compatible; no adaptation change is needed for the checked interfaces. These checks used controlled provider responses, not a complete host UI/model session.
+- Core runtime dependencies remain zero. Model input stays at **1,159 description characters / 1,876 schema characters**; site body summaries remain opt-in and respect `maxCharsPerPage` without an extra 80-character render cutoff.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)

package/README.zh.md

@@ -315,6 +315,14 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 - Vue、React、酷狗三个已知 NOISE 站采用同份 HTML 与相同可选依赖对照 v1.8.1，text/Markdown 全文及 800 字符窗口逐字一致；ERR 均为网络、反爬、无效地址或预期不支持的类型。
 - 四个工具统一遵守显式来源策略，覆盖跳转与爬取边界、缓存隔离及可见的静态读取提示。模型固定输入成本为 **description 1,159 字符 / schema 1,876 字符**，低于 1,250 / 2,000 预算。
 
+### v1.9.1 维护验证（2026-09-28）
+
+- **352 个零依赖单元断言、15 个真实 Chromium SPA 断言、59 项对抗探针**通过。导航测试覆盖真实属性、引号内的 `>`、空锚点、刷新实体和分号；取消、重试清理及站点摘要渲染上限均有回归覆盖。异步探针现在等待完成，并通过退出码报告失败。
+- 152 站最终结果为 **103 OK / 25 THIN+EMPTY / 24 ERR / 0 THREW**，OK 站零噪声。Vue、React、酷狗与 Hatena 使用同份 HTML 对照 v1.9.0，全文续读及 800 字输出一致；微博静态/SPA 波动也经同份 HTML/DOM 对照一致，错误站点集合未变。
+- 导航采用有界、识别引号的属性扫描，每段锚点正文只扫描一次。本机 2.5 万/5 万/10 万个含 `href` 的未闭合锚点耗时约 **27/57/118 ms**，保留原有 1,000 字符属性边界。
+- [DSH 0.1.7-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.7-rc.2) 已完成官方 npm 接触包字节差异核查，并与社区升级卡交叉验证（卡片目前到 rc.1）。发布版 v1.9.0 与当前维护代码各通过 **17 项真实官方服务集成检查**；发布版 patch 另通过 **4 项加载器检查**，覆盖加载、调用、配置重挂和卸载。Web provider/runtime 契约兼容，已核查接口无需适配修改；测试使用受控 provider 响应，未覆盖完整宿主 UI/模型会话。
+- 核心仍为零运行时依赖，模型固定输入仍为 **description 1,159 字符 / schema 1,876 字符**；站点正文摘要仍默认关闭，启用后遵循 `maxCharsPerPage`，不再额外截成 80 字符。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）

package/index.js

@@ -247,10 +247,19 @@ async function directFetchOnce(url, signal, cfg, origins) {
 // filter into decodeBuffer as a bufferless "success" (bare TypeError there).
 export async function directFetch(url, signal, cfg, origins) {
   const first = await directFetchOnce(url, signal, cfg, origins)
-  if (first && first.retryAfterMs !== undefined && !(signal && signal.aborted)) {
+  if (first && first.retryAfterMs !== undefined) {
+    if (signal && signal.aborted) return { error: 'cancelled' }
     await new Promise((resolve) => {
-      const t = setTimeout(resolve, first.retryAfterMs)
-      if (signal) signal.addEventListener('abort', () => { clearTimeout(t); resolve() }, { once: true })
+      const done = () => {
+        clearTimeout(timer)
+        if (signal) signal.removeEventListener('abort', done)
+        resolve()
+      }
+      const timer = setTimeout(done, first.retryAfterMs)
+      if (signal) {
+        signal.addEventListener('abort', done, { once: true })
+        if (signal.aborted) done()
+      }
     })
     if (signal && signal.aborted) return { error: 'cancelled' }
     const second = await directFetchOnce(url, signal, cfg, origins)
@@ -1427,22 +1436,60 @@ function decodeUrlAttribute(value) {
 }
 
 // <base href> overrides the document base for relative URL resolution (old
-// forums / frame sites). Per HTML spec only the FIRST <base> in <head> counts;
+// forums / frame sites). Only the FIRST <base href> in <head> counts;
 // attribute order is arbitrary and the href may be protocol-relative. Empty
 // href (= document URL) is a no-op; non-http(s) targets are rejected. The
 // head scan is bounded — the tag cannot legally appear beyond it.
 function detectBaseHref(html, finalUrl) {
   const headEnd = html.search(/<\/head\s*>/i)
   const head = headEnd >= 0 ? html.slice(0, headEnd) : html.slice(0, 16384)
-  const base = /<base\b[^>]{0,1000}href=["']([^"']+)["'][^>]{0,1000}>/i.exec(head)
-  if (!base) return finalUrl
-  const href = decodeUrlAttribute(base[1]).trim()
-  if (!href) return finalUrl
-  try {
-    const u = new URL(href, finalUrl)
-    return u.protocol === 'http:' || u.protocol === 'https:' ? u.href : finalUrl
-  } catch {
-    return finalUrl
+  for (const { attrs } of navigationTags(head, /<base(?=[\s/>])/gi)) {
+    if (!Object.hasOwn(attrs, 'href')) continue
+    const href = decodeUrlAttribute(attrs.href).trim()
+    if (!href) return finalUrl
+    try {
+      const u = new URL(href, finalUrl)
+      return u.protocol === 'http:' || u.protocol === 'https:' ? u.href : finalUrl
+    } catch {
+      return finalUrl
+    }
+  }
+  return finalUrl
+}
+
+// Honor quoted '>' without nested regex alternatives. Keep the same total
+// tag-length bound as htmlAttributes; malformed/overlong tags stay inert.
+function* navigationTags(html, open) {
+  let m
+  while ((m = open.exec(html))) {
+    let quote = '', end = -1
+    const limit = Math.min(html.length, m.index + 1002)
+    for (let i = open.lastIndex; i < limit; i++) {
+      const ch = html[i]
+      if (quote) {
+        if (ch === quote) quote = ''
+      } else if (ch === '"' || ch === "'") quote = ch
+      else if (ch === '<') break
+      else if (ch === '>') { end = i + 1; break }
+    }
+    if (end < 0) continue
+    open.lastIndex = end
+    yield { attrs: htmlAttributes(html.slice(m.index, end)), end }
+  }
+}
+
+// Scan each anchor body once; tags without a real href cannot navigate.
+function* navigationAnchors(html) {
+  const open = /<a(?=[\s>])/gi
+  const close = /<\/a\s{0,1000}>/gi
+  for (const { attrs, end: start } of navigationTags(html, open)) {
+    if (!attrs.href?.trim()) continue
+    close.lastIndex = start
+    const end = close.exec(html)
+    if (!end) break
+    const body = html.slice(start, end.index)
+    open.lastIndex = close.lastIndex
+    yield { attrs, body }
   }
 }
 
@@ -1454,18 +1501,14 @@ function extractLinks(html, limit, baseUrl) {
   // A declared <base href> wins over the document URL (frame-site links would
   // otherwise land on the wrong host).
   const linkBase = detectBaseHref(html, baseUrl)
-  // Attribute runs bounded: an unclosed <a whose '>' is far away (or missing)
-  // must not make the regex scan the rest of the string at every anchor
-  // position. Real attributes never approach 1000 chars.
-  const re = /<a\b[^>]{0,1000}href=["']([^"']+)["'][^>]{0,1000}>([\s\S]*?)<\/a\s*>/gi
-  let m
   // Iteration cap: a page whose nav repeats the same links thousands of times
   // would otherwise scan the whole HTML to fill `limit` unique entries. Scan
   // at most limit*4 anchors; dedupe keeps the result bounded regardless.
   let scans = 0
   const maxScans = limit * 4
-  while ((m = re.exec(html)) && links.length < limit && scans++ < maxScans) {
-    const href = decodeUrlAttribute(m[1]).trim()
+  for (const { attrs, body } of navigationAnchors(html)) {
+    if (links.length >= limit || scans++ >= maxScans) break
+    const href = decodeUrlAttribute(attrs.href).trim()
     if (!href || /^(javascript|mailto|tel|data):/i.test(href)) continue
     let url
     try {
@@ -1481,7 +1524,7 @@ function extractLinks(html, limit, baseUrl) {
     // keeps the link list compact (tokens) without losing coverage
     if (seen.has(url)) continue
     seen.add(url)
-    const t = textOnly(m[2])
+    const t = textOnly(body)
     if (t) links.push({ title: t.slice(0, 80), url })
   }
   return links
@@ -1507,6 +1550,7 @@ export function findNextLink(html, baseUrl) {
   // document base, and a "next page" link resolves against it.
   const base = detectBaseHref(html || '', baseUrl)
   const resolve = (href) => {
+    if (!href?.trim()) return null
     try {
       const u = new URL(decodeUrlAttribute(href), base)
       return u.protocol === 'http:' || u.protocol === 'https:' ? u.href : null
@@ -1514,21 +1558,20 @@ export function findNextLink(html, baseUrl) {
       return null
     }
   }
-  const rel = /<(?:link|a)\b[^>]{0,1000}rel=["']next["'][^>]{0,1000}>/i.exec(html || '')
-  if (rel) {
-    const href = /href=["']([^"']+)["']/i.exec(rel[0])
-    const r = href && resolve(href[1])
+  for (const { attrs } of navigationTags(html || '', /<(?:link|a)(?=[\s/>])/gi)) {
+    const rel = decodeUrlAttribute(attrs.rel || '').toLowerCase().split(/[ \t\r\n\f]{1,1000}/)
+    if (!rel.includes('next')) continue
+    const r = resolve(attrs.href)
     if (r) return r
   }
-  const re = /<a\b[^>]{0,1000}href=["']([^"']+)["'][^>]{0,1000}>([\s\S]*?)<\/a\s*>/gi
-  let m
   let scans = 0
   // Bounded like extractLinks: a nav bar with thousands of anchors must not
   // scan the whole DOM hunting for a next-marker.
-  while ((m = re.exec(html || '')) && scans++ < 400) {
-    const t = textOnly(m[2])
+  for (const { attrs, body } of navigationAnchors(html || '')) {
+    if (scans++ >= 400) break
+    const t = textOnly(body)
     if (t.length <= 16 && NEXT_TEXT_RE.test(t)) {
-      const r = resolve(m[1])
+      const r = resolve(attrs.href)
       if (r) return r
     }
   }
@@ -1621,18 +1664,15 @@ export function metaRefreshTarget(html, baseUrl) {
   // Locate the refresh <meta> tag first (any attribute order / case), then
   // read content from inside that tag — a forward regex would demand
   // http-equiv before content and miss the reversed order.
-  const re = /<meta\b[^>]{0,1000}(?:http-equiv|name)=["']refresh["'][^>]{0,1000}>/gi
-  let m
-  while ((m = re.exec(html || ''))) {
-    const tag = m[0]
-    const contentM = /content=("([^"]*)"|'([^']*)')/i.exec(tag)
-    if (!contentM) continue // a refresh meta without content is inert
-    const content = (contentM[2] || contentM[3] || '').trim()
+  for (const { attrs } of navigationTags(html || '', /<meta(?=[\s/>])/gi)) {
+    if (![attrs['http-equiv'], attrs.name].some(value => value?.toLowerCase() === 'refresh')) continue
+    if (!Object.hasOwn(attrs, 'content')) continue
+    const content = decodeUrlAttribute(attrs.content).trim()
     // content="5" | content="0; url=http://…" | content=";url='…'" — the url
     // part may be single/double quoted or bare; "url=" must be a real token.
-    const urlM = /\burl\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s;]+))/i.exec(content)
+    const urlM = /\burl\s{0,1000}=\s{0,1000}(?:"([^"]{0,1000})"|'([^']{0,1000})'|([^\s]{1,1000}))/i.exec(content)
     if (!urlM) return null
-    const delayM = /^\s*(\d+(?:\.\d+)?)/.exec(content)
+    const delayM = /^\s{0,1000}(\d{1,1000}(?:\.\d{1,1000})?)/.exec(content)
     if (delayM && Number(delayM[1]) > 0) return null // timed refresh: keep the page
     const target = (urlM[1] || urlM[2] || urlM[3] || '').trim()
     if (!target) return null
@@ -2130,14 +2170,18 @@ function readLinksTool(ctx, cfg) {
       const url = String((args && args.url) || '').trim()
       if (!/^https?:\/\//i.test(url)) return { error: 'Only http/https URLs are supported' }
       const limit = Math.max(1, Math.min(50, Number(args.limit) || cfg.maxLinks))
+      const signal = exec && exec.signal
+      if (signal && signal.aborted) return { error: 'cancelled' }
       const route = createRoute(cfg)
-      const page = await fetchResource(ctx, url, exec && exec.signal, cfg, route)
+      const page = await fetchResource(ctx, url, signal, cfg, route)
+      if (signal && signal.aborted) return { error: 'cancelled' }
       if (page.error) return { error: page.error }
       let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
       let finalUrl = page.finalUrl || url
       // Same meta-refresh shell following as read_url: a link hop serves a
       // stub whose real links live at the target. Fails open, hop-bounded.
-      const followed = await followMetaRefresh(html, finalUrl, exec && exec.signal, cfg, ctx, route)
+      const followed = await followMetaRefresh(html, finalUrl, signal, cfg, ctx, route)
+      if (signal && signal.aborted) return { error: 'cancelled' }
       html = followed.html
       finalUrl = followed.finalUrl
       let links = extractLinks(html, limit, finalUrl)
@@ -2147,7 +2191,8 @@ function readLinksTool(ctx, cfg) {
       // static shell may be a JS-redirect (same rationale as read_url).
       const needsRender = links.length < 3 && cfg.spaRender !== false && (looksLikeSpa(html) || (links.length === 0 && /<script[\s>]/i.test(html)))
       if (!route.restricted && needsRender) {
-        const rr = await renderPage(finalUrl || url, exec && exec.signal)
+        const rr = await renderPage(finalUrl || url, signal)
+        if (signal && signal.aborted) return { error: 'cancelled' }
         // A challenge interstitial's links are the challenge's own scripts —
         // never an improvement over whatever the static HTML offered.
         if (rr.html && !looksLikeChallenge(rr.html)) {
@@ -2349,6 +2394,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         attempted++
         const route = { ...policy, restricted }
         const page = await fetchResource(ctx, url, externalSignal, cfg, route)
+        if (externalSignal && externalSignal.aborted) return null
         if (page.error) return { url, depth, error: page.error }
         let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
         let pageUrl = page.finalUrl || url
@@ -2360,6 +2406,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         // real content is never reached). Fails open — fetch problems keep the
         // stub's own HTML.
         const followed = await followMetaRefresh(html, pageUrl, externalSignal, cfg, ctx, route)
+        if (externalSignal && externalSignal.aborted) return null
         html = followed.html
         pageUrl = followed.finalUrl
         if (depth === 0) host = hostOf(pageUrl)
@@ -2380,6 +2427,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         }
       }),
     )
+    if (externalSignal && externalSignal.aborted) break
     for (const r of results) {
       if (!r) continue
       if (r.error) {
@@ -2417,7 +2465,7 @@ function renderSite(value) {
     const head = p.title || p.url
     lines.push(`${indent}[${p.depth}] ${head} (${p.chars} 字符)  ${p.url}`)
     if (p.spaHint) lines.push(`${indent}提示: ${p.spaHint}`)
-    if (p.text) lines.push(`${indent}   ${p.text.slice(0, 80)}`)
+    if (p.text) lines.push(`${indent}   ${p.text}`)
   }
   for (const f of v.failures) lines.push(`[失败] ${f.url} — ${f.error}`)
   return lines.join('\n')
@@ -2471,12 +2519,15 @@ function readUrlSiteTool(ctx, cfg) {
       const url = String((args && args.url) || '').trim()
       if (!/^https?:\/\//i.test(url)) return { error: 'Only http/https URLs are supported' }
       const maxPages = Math.floor(Math.max(2, Math.min(50, Number(args.maxPages) || 15)))
+      const signal = exec && exec.signal
+      if (signal && signal.aborted) return { error: 'cancelled' }
       const maxDepth = Math.max(1, Math.min(5, Number(args.maxDepth) || 2))
       const includeContent = args.includeContent === true
       const perMax = Math.max(200, Math.min(2000, Number(args.maxCharsPerPage) || 500))
       const { host, pages, failures } = await crawlSite(
-        url, cfg, { maxPages, maxDepth, includeContent, perMax }, exec && exec.signal, ctx,
+        url, cfg, { maxPages, maxDepth, includeContent, perMax }, signal, ctx,
       )
+      if (signal && signal.aborted) return { error: 'cancelled' }
       return {
         host,
         total: pages.length + failures.length,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",