dsh-read-url 1.8.0 → 1.8.1

package/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## [1.8.1] - 2026-09-19
+
+### 修复
+
+- SPA 导航取消及时关闭当前页面；启动期间取消不再创建页面，卸载等待浏览器关闭，并发清理共享等待，断开连接后可重新启动；旧启动失败不会清空新浏览器引用。
+- 浏览器启动异常保留简短真实原因，避免把执行权限错误误判为缺少 Playwright；DOM 轮询按连续失败次数判断停止。
+- 有效成功缓存优先于并发失败，成功写入或命中清理同键失败记录。
+- 站点爬取按 URL 路径边界过滤，保留 feedback、admin-guide 及查询参数中包含登录地址的正常页面。
+- 编码声明支持等号空格和无引号属性，排除 data-charset 伪属性；Markdown 链接与图片按真实属性读取，正确处理实体、括号和图片说明文本。
+
+### 验证
+
+- 297 个零依赖单元断言、15 个真实 Chromium SPA 断言、53 项对抗探针通过；新增 21 条单元断言和 3 条 SPA 断言。
+- 152 站实网回归：104 OK / 23 THIN+EMPTY / 25 ERR / 0 THREW，OK 站零噪声；Vue、React、酷狗三站已知 NOISE 在同输入 v1.8.0 A/B 中逐字一致。
+- 修复前 v1.7.0 / v1.8.0 同输入均复现相关缺陷，属于既有问题；核心零运行时依赖，新增或修改的正则保持有界。
+- 四工具 description 共 1,159 字符，参数 schema 共 1,876 字符；预算仍为 1,250 / 2,000。
+
 ## [1.8.0] - 2026-09-07
 
 ### 内容与准确性

package/README.md

@@ -274,10 +274,19 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
+### Released v1.8.0 validation snapshot
+
 - **276 zero-dependency assertions + 12 SPA assertions** pass. `node test.mjs` covers provider contracts, cancellation/retry, extraction/metadata, RSS/Atom, Markdown, continuation, batching and crawl boundaries. `node probe.mjs` checks pathological tags, nesting, long whitespace and bounded scanning.
 - Local maintenance validation (2026-09-07): **102 OK / 27 THIN+EMPTY / 23 ERR / 0 THREW** across 152 sites, with no noise among OK results; 53 adversarial probes pass. ERR results are network/anti-bot failures, invalid addresses or intentionally unsupported binary types. Known NOISE flags were checked against the previous code using identical inputs.
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
+### v1.8.1 maintenance validation (2026-09-19)
+
+- **297 zero-dependency assertions + 15 real Chromium SPA assertions + 53 adversarial probes** pass. `test-lifecycle.mjs`, included in `test.mjs`, uses a local browser fixture for cancellation, launch failures, concurrent cleanup and restart checks.
+- The 152-site run returned **104 OK / 23 THIN+EMPTY / 25 ERR / 0 THREW**, with no noise among OK results. The three known NOISE sites have identical decoded content and extracted output against v1.8.0 using the same input. Network errors and intentionally unsupported types remain separate from extraction results.
+- SPA cancellation closes the active page; plugin disposal awaits browser shutdown. Valid success-cache entries take precedence over concurrent failures. Crawl filtering uses path boundaries. Charset and Markdown attributes use bounded parsing.
+- Fixed model input cost remains **1,159 description characters / 1,876 schema characters**, within the 1,250 / 2,000 budgets.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)
@@ -306,7 +315,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 
 ```bash
 node test.mjs          # zero-dependency self-tests (charset/extract/markdown/truncate)
-node test-spa.mjs      # SPA rendering tests (12 assertions; SKIPs if playwright absent)
+node test-spa.mjs      # SPA rendering tests (15 assertions; SKIPs if playwright absent)
 node multi-site.mjs    # 152-site real-world sweep (needs network, CONC=8 tunable): portals/SPA/login-walls/static/feeds/JSON/anti-bot/net-boundaries
 
 # End-to-end (requires DSH CLI)

package/README.zh.md

@@ -274,10 +274,19 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
+### 已发布 v1.8.0 验证快照
+
 - **276 个零依赖单元断言 + 12 个 SPA 断言**通过；`node test.mjs` 包含网络契约、取消重试、正文/元数据、RSS/Atom、Markdown、续读、批量与站点边界测试。`node probe.mjs` 验证极端标签、嵌套、长空白和有界扫描性能。
 - 本机维护验证（2026-09-07）：152 站为 **102 OK / 27 THIN+EMPTY / 23 ERR / 0 THREW**，OK 站零噪声；53 项对抗探针通过。ERR 为网络、反爬、无效地址或预期不支持的二进制类型；已知 NOISE 标记经同输入新旧 A/B 核查。
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
+### v1.8.1 维护验证（2026-09-19）
+
+- **297 个零依赖单元断言 + 15 个真实 Chromium SPA 断言 + 53 项对抗探针**通过。`test.mjs` 已包含 `test-lifecycle.mjs`，通过本地浏览器桩验证取消、启动失败、并发清理和重启。
+- 152 站实网结果为 **104 OK / 23 THIN+EMPTY / 25 ERR / 0 THREW**，OK 站零噪声。三个已知 NOISE 站在同输入 v1.8.0 对照中，解码内容与提取结果逐字一致；网络错误与预期不支持的类型单独归类。
+- SPA 取消关闭当前页面，插件卸载等待浏览器关闭；有效成功缓存优先于并发失败；爬取过滤遵循路径边界；编码与 Markdown 属性使用有界解析。
+- 模型固定输入成本仍为 **description 1,159 字符 / schema 1,876 字符**，低于 1,250 / 2,000 预算。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）
@@ -308,7 +317,7 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 node test.mjs          # 单元自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离/配置钳制/控制字符/JSON-LD/base href/meta-refresh）
 
 # SPA 渲染真实测试（需 playwright 已安装，未装自动 SKIP）
-node test-spa.mjs      # 12 断言：JS 正文/渲染后链接/工具不崩溃/缓存隔离/JS 跳转壳
+node test-spa.mjs      # 15 断言：JS 正文/渲染后链接/缓存隔离/导航取消/浏览器重启
 
 # 152 站真实世界验证（需联网，CONC=8 可调并发）
 node multi-site.mjs    # 门户/SPA/登录墙/静态/订阅源/JSON/反爬/网络边界，输出分级结果

package/index.js

@@ -46,9 +46,10 @@ function cacheFail(key, error) {
   if (failCache.size >= FAIL_CACHE_MAX) failCache.delete(failCache.keys().next().value)
   failCache.set(key, { time: Date.now(), error })
 }
-function cacheStore(key, full, cacheMax) {
-  if (cache.size >= cacheMax) cache.delete(cache.keys().next().value)
-  cache.set(key, { time: Date.now(), full })
+function cacheStore(key, full, cacheMax) {
+  if (cache.size >= cacheMax) cache.delete(cache.keys().next().value)
+  cache.set(key, { time: Date.now(), full })
+  failCache.delete(key)
 }
 const decoders = new Map()
 
@@ -65,21 +66,29 @@ function getDecoder(enc) {
   return d
 }
 
-function sniffCharset(buffer, contentType) {
+function declaredCharset(contentType) {
+  const m = /(?:^|;)[ \t]{0,1000}charset[ \t]{0,1000}=[ \t]{0,1000}(?:"([\w.:-]{1,80})"|'([\w.:-]{1,80})'|([\w.:-]{1,80}))(?=[ \t;]|$)/i.exec(contentType || '')
+  return m ? m[1] || m[2] || m[3] : null
+}
+
+function sniffCharset(buffer, contentType) {
   // BOM first — byte-level evidence beats any declared charset (a UTF-16 body
   // with a latin1-read <meta> probe garbles the meta match anyway).
   if (buffer[0] === 0xef && buffer[1] === 0xbb && buffer[2] === 0xbf) return 'utf-8'
   if (buffer[0] === 0xff && buffer[1] === 0xfe) return 'utf-16le'
   if (buffer[0] === 0xfe && buffer[1] === 0xff) return 'utf-16be'
-  const m = /charset=["']?([\w-]+)/i.exec(contentType || '')
-  if (m) return m[1]
-  const head = buffer.subarray(0, 2048).toString('latin1').toLowerCase()
-  const meta = /<meta[^>]{0,1000}charset=["']?([\w-]+)/i.exec(head)
-  if (meta) return meta[1]
-  // Legacy form: <meta http-equiv="Content-Type" content="text/html; charset=gb2312">
-  // (no charset= attribute anywhere) — common on old GBK pages with no header charset.
-  const metaEquiv = /<meta[^>]{0,1000}(?:http-equiv)=["']content-type["'][^>]{0,1000}content=["'][^"']*charset=([\w-]+)/i.exec(head)
-  if (metaEquiv) return metaEquiv[1]
+  const header = declaredCharset(contentType)
+  if (header) return header
+  const head = buffer.subarray(0, 2048).toString('latin1')
+  for (const meta of head.matchAll(/<meta(?=[\s/>])[^>]{0,1000}>/gi)) {
+    const attrs = htmlAttributes(meta[0])
+    const charset = attrs.charset?.trim()
+    if (charset && /^[\w.:-]{1,80}$/.test(charset)) return charset
+    if (attrs['http-equiv']?.toLowerCase() === 'content-type') {
+      const legacy = declaredCharset(attrs.content)
+      if (legacy) return legacy
+    }
+  }
   return null
 }
 
@@ -708,9 +717,15 @@ function stripNoise(mainHtml) {
 }
 
 // ---- HTML -> Markdown (lightweight, tag-state machine) ----
-function escInline(s) {
-  return s.replace(/([\\`*_[\]])/g, '\\$1')
-}
+function escInline(s) {
+  return s.replace(/([\\`*_[\]])/g, '\\$1')
+}
+
+function markdownTarget(value) {
+  return decodeUrlAttribute(value).trim()
+    .replace(/\(/g, '%28').replace(/\)/g, '%29')
+    .replace(/[<>\s\\]/g, encodeURIComponent)
+}
 
 // <img> is a void element (never matches the paired-tag regex in either md
 // walker): convert descriptive images to markdown BEFORE the tag walk, via a
@@ -729,25 +744,30 @@ function imgsToMarkdown(html) {
   // in <picture><source srcset>. When the <img> fallback carries no usable
   // src (lazy placeholder), the first source URL is injected into it so the
   // normal img pass below still emits the image.
-  html = html.replace(/<picture\b[^>]*>([\s\S]{0,50000}?)<\/picture\s*>/gi, (block) => {
-    const imgTag = /<img\b[^>]{0,1000}>/i.exec(block)
-    const imgSrc = imgTag && /(?:^|\s)src=["']([^"']+)["']/i.exec(imgTag[0])
-    if (imgTag && imgSrc && !/^data:/i.test(imgSrc[1])) return block
-    const source = /<source\b[^>]{0,1000}\bsrcset=["']([^"'\s>]+)/i.exec(block)
-    if (!source || !imgTag) return block
-    return block.replace(/<img\b[^>]{0,1000}>/i, (img0) => img0.replace(/<img/i, `<img src="${source[1]}"`))
-  })
-  const out = html.replace(/<(?:amp-)?img\b[^>]{0,1000}>/gi, (tag) => {
-    const alt = /alt=["']([^"']*)["']/i.exec(tag)
-    let src = /src=["']([^"']+)["']/i.exec(tag)
-    let srcVal = src && src[1]
+  html = html.replace(/<picture\b[^>]{0,1000}>([\s\S]{0,50000}?)<\/picture\s{0,1000}>/gi, (block) => {
+    const imgTag = /<img\b[^>]{0,1000}>/i.exec(block)
+    const imgSrc = imgTag && htmlAttributes(imgTag[0]).src
+    if (!imgTag || (imgSrc && !/^data:/i.test(imgSrc))) return block
+    let source = ''
+    for (const tag of block.matchAll(/<source\b[^>]{0,1000}>/gi)) {
+      source = htmlAttributes(tag[0]).srcset?.trim().split(/[\s,]{1,1000}/, 1)[0] || ''
+      if (source) break
+    }
+    if (!source) return block
+    const escaped = decodeUrlAttribute(source).replace(/&/g, '&amp;').replace(/"/g, '&quot;')
+    return block.replace(imgTag[0], () => `<img src="${escaped}"${imgTag[0].slice(4)}`)
+  })
+  const out = html.replace(/<(?:amp-)?img\b[^>]{0,1000}>/gi, (tag) => {
+    const attrs = htmlAttributes(tag)
+    const alt = decodeTextEntities(attrs.alt || '').replace(/\s{1,1000}/g, ' ').trim()
+    let srcVal = attrs.src?.trim()
     // Lazy-load wrappers keep the real URL in data-* attributes; a data: URI
     // src is an inline placeholder spacer, not a usable image reference.
     if (!srcVal || /^data:/i.test(srcVal)) {
-      const lazy = /data-(?:src|original|lazy-src)=["']([^"']+)["']/i.exec(tag)
-      if (lazy) srcVal = lazy[1]
-    }
-    return protect(srcVal && alt && alt[1].trim() ? `![${alt[1].trim()}](${srcVal})` : '')
+      const lazy = attrs['data-src'] || attrs['data-original'] || attrs['data-lazy-src']
+      if (lazy) srcVal = lazy.trim()
+    }
+    return protect(srcVal && alt ? `![${escInline(alt)}](${markdownTarget(srcVal)})` : '')
   })
   return {
     html: out,
@@ -826,13 +846,13 @@ export function inlineMd(html, depth = 0) {
       out += img.protect(inlineCode(m[3]))
       continue
     }
-    const inner = inlineMd(m[3], depth + 1)
-    if (tag === 'a') {
-      const href = /href=["']([^"']+)["']/i.exec(m[2])
+    const inner = inlineMd(m[3], depth + 1)
+    if (tag === 'a') {
+      const href = htmlAttributes(`<a${m[2]}>`).href
       // A raw ')' in the target would terminate the markdown link early
       // (common on wiki/framework URLs) — percent-encode parens to keep the
       // link parseable by the model.
-      out += href ? `[${inner}](${href[1].replace(/\)/g, '%29').replace(/\(/g, '%28')})` : inner
+      out += href?.trim() ? `[${inner}](${markdownTarget(href)})` : inner
     } else if (tag === 'strong' || tag === 'b') out += `**${inner}**`
     else if (tag === 'em' || tag === 'i') out += `*${inner}*`
     else out += inner
@@ -1612,8 +1632,18 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   } catch {
     /* keep bare url */
   }
-  const cacheKey = `${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
-  // Failed fetches are cached briefly so the model doesn't re-request a
+  const cacheKey = `${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
+  const hit = cache.get(cacheKey)
+  if (hit && Date.now() - hit.time < cfg.cacheTtlMs) {
+    // Valid content takes precedence over a concurrent fetch failure.
+    failCache.delete(cacheKey)
+    // LRU touch keeps hot pages available throughout their TTL.
+    cache.delete(cacheKey)
+    cache.set(cacheKey, hit)
+    const sliced = sliceFrom(hit.full, offset, maxChars)
+    return { ...sliced, cached: true }
+  }
+  // Failed fetches are cached briefly so the model doesn't re-request a
   // broken URL in a loop (token + latency saver).
   const fh = failCache.get(cacheKey)
   if (fh) {
@@ -1626,15 +1656,6 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     }
     failCache.delete(cacheKey)
   }
-  const hit = cache.get(cacheKey)
-  if (hit && Date.now() - hit.time < cfg.cacheTtlMs) {
-    // LRU touch: hot pages survive eviction even when cached long ago — the
-    // Map re-insertion moves the entry to the newest slot.
-    cache.delete(cacheKey)
-    cache.set(cacheKey, hit)
-    const sliced = sliceFrom(hit.full, offset, maxChars)
-    return { ...sliced, cached: true }
-  }
 
   const failWith = (error) => {
     if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
@@ -2189,11 +2210,16 @@ function sameHost(a, b) {
 }
 
 // URLs not worth crawling: static assets, login/auth paths, feeds, sitemaps.
-const NOISE_EXT = /\.(png|jpe?g|gif|svg|webp|ico|bmp|css|js|json|xml|pdf|zip|gz|tar|7z|mp3|m3u8?|mpd|flv|ts|mp4|avi|mov|webm|woff2?|ttf|eot|map)(\?|#|$)/i
-const NOISE_PATH = /(\/login|\/signin|\/register|\/logout|\/signup|\/api\/|\/admin|\/wp-admin|\/wp-login|\/feed|\/rss|\/sitemap|\/robots\.txt|\/cdn-cgi)/i
-function isNoiseUrl(url) {
-  return NOISE_EXT.test(url) || NOISE_PATH.test(url)
-}
+const NOISE_EXT = /\.(png|jpe?g|gif|svg|webp|ico|bmp|css|js|json|xml|pdf|zip|gz|tar|7z|mp3|m3u8?|mpd|flv|ts|mp4|avi|mov|webm|woff2?|ttf|eot|map)$/i
+const NOISE_PATH = /\/(?:login|signin|register|logout|signup|api|admin|wp-admin|wp-login|feed|rss|sitemap|robots\.txt|cdn-cgi)(?:\/|[.;]|$)/i
+function isNoiseUrl(url) {
+  try {
+    const path = new URL(url).pathname
+    return NOISE_EXT.test(path) || NOISE_PATH.test(path)
+  } catch {
+    return true
+  }
+}
 
 // BFS crawl of a single site. Only same-host http(s) pages are followed;
 // static assets / auth paths are skipped; visited URLs are deduped; a per-page
@@ -2382,11 +2408,11 @@ export function apply(ctx, config) {
   if (typeof cfg.paginate !== 'boolean') cfg.paginate = DEFAULTS.paginate
 
   // Temporal composability: unload must fully revert side effects.
-  ctx.effect(() => () => {
-    cache.clear()
-    failCache.clear()
-    closeBrowser().catch(() => {})
-  })
+  ctx.effect(() => () => {
+    cache.clear()
+    failCache.clear()
+    return closeBrowser()
+  })
 
   console.log('[dsh-read-url] plugin loaded; tools read_url, read_url_batch, read_url_links, read_url_site registered')
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.8.0",
+  "version": "1.8.1",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",

package/spa.js

@@ -4,6 +4,11 @@
 // When absent, reads fall back to static extraction with a clear hint.
 
 let browserPromise = null
+const closingBrowsers = new Set()
+
+function renderError(error) {
+  return `Render failed: ${String(error?.message || error).split('\n', 1)[0].slice(0, 500)}`
+}
 
 // Heuristic: a page whose HTML carries many <script> tags is likely a
 // client-rendered SPA (Vue/React) whose body lives only after JS execution.
@@ -33,12 +38,19 @@ async function getBrowser() {
   if (!browserPromise) {
     // A failed launch (playwright later installed / chromium download finished)
     // must not stay cached as a rejected promise — reset so the next call retries.
-    browserPromise = import('playwright')
+    const pending = import('playwright')
       .then(({ chromium }) => chromium.launch({ headless: true }))
+      .then((browser) => {
+        browser.once('disconnected', () => {
+          if (browserPromise === pending) browserPromise = null
+        })
+        return browser
+      })
       .catch((e) => {
-        browserPromise = null
+        if (browserPromise === pending) browserPromise = null
         throw e
       })
+    browserPromise = pending
   }
   return browserPromise
 }
@@ -49,12 +61,23 @@ export async function renderPage(url, externalSignal) {
   let browser
   try {
     browser = await getBrowser()
-  } catch {
-    return { error: 'SPA 渲染需 playwright（npm i playwright && npx playwright install chromium）' }
+  } catch (error) {
+    if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+    return { error: error?.code === 'ERR_MODULE_NOT_FOUND'
+      ? 'SPA 渲染需 playwright（npm i playwright && npx playwright install chromium）'
+      : renderError(error) }
   }
+  if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   let page
+  let pageClosing
+  const closePage = () => {
+    if (page && !pageClosing) pageClosing = page.close().catch(() => {})
+    return pageClosing
+  }
   try {
     page = await browser.newPage()
+    externalSignal?.addEventListener('abort', closePage, { once: true })
+    if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
     // 'domcontentloaded' instead of 'networkidle': heartbeat-polling sites
     // (qq-news, juejin) never go idle and would time out at 30s. Instead wait
     // for the DOM to stabilize (content stops growing) up to 10s — SPA paint
@@ -72,7 +95,8 @@ export async function renderPage(url, externalSignal) {
         .evaluate(() => (document.body ? document.body.innerHTML.length : 0))
         .catch(() => -1)
       if (Date.now() - t0 > 10000) break
-      if (len === -1 && ++evalFails >= 2) break
+      evalFails = len === -1 ? evalFails + 1 : 0
+      if (evalFails >= 2) break
       // stop once two consecutive reads agree — including empty bodies (a
       // blank page should not burn the full 10s poll)
       if (len === prevLen && prevLen >= 0) break
@@ -93,25 +117,33 @@ export async function renderPage(url, externalSignal) {
         if (!looksLikeChallenge(html)) break
       }
     }
+    if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
     const finalUrl = page.url()
     return { html, finalUrl }
   } catch (e) {
     if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
-    return { error: `Render failed: ${e.message}` }
+    return { error: renderError(e) }
   } finally {
-    if (page) await page.close().catch(() => {})
+    externalSignal?.removeEventListener('abort', closePage)
+    await closePage()
   }
 }
 
 // Called on plugin unload (temporal composability): release the browser.
 export async function closeBrowser() {
-  if (!browserPromise) return
   const p = browserPromise
   browserPromise = null
-  try {
-    const b = await p
-    await b.close().catch(() => {})
-  } catch {
-    // launch never completed — nothing to close
+  if (p) {
+    const closing = (async () => {
+      try {
+        const b = await p
+        await b.close()
+      } catch {
+        // A failed launch or an already closed browser needs no further cleanup.
+      }
+    })()
+    closingBrowsers.add(closing)
+    closing.finally(() => closingBrowsers.delete(closing))
   }
+  await Promise.all(closingBrowsers)
 }