dsh-read-url 1.9.2 → 1.9.3

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## [1.9.3] - 2026-10-19
+
+### 修复
+
+- Markdown 按配对块处理嵌套容器与列表，保留段落间距、父级编号和子列表缩进；代码围栏/引用与后文正确分隔，属性与递归深度继续有界。
+- 数值配置经范围限制后向下取整，避免小数超时触发 `AbortSignal.timeout` 错误或小数分页上限多抓一页。
+- 代理回退遵守 `no_proxy`/`NO_PROXY` 的域名、IP、端口和 CIDR 边界，避免绕过代理的地址被直连与 curl 重复请求；curl 禁用 URL glob 展开，原样处理方括号查询和花括号路径。
+- curl 启用受支持的 HTTP 响应解压，避免 gzip 字节进入正文；单独检查解压后字节限额，兼容旧版 curl 的传输大小语义，stdout 超限返回明确错误。
+- 失败诊断不再输出代理配置地址，避免用户名、密码进入工具结果或模型上下文。
+- 提供方明确标记为纯文本时，链接和站点读取不再解析正文中的 HTML 示例或跟随示例跳转；保留原文、输出预算与提供方截断提示。
+- 质询页识别的空白段使用明确长度上限，消除相邻无界空白造成的平方回溯。
+- 双语说明更新 DeepSeek Fetch API 能力边界，并将早期官方工具对比明确标为历史快照。
+
+### 验证
+
+- 413 个零依赖单元断言、15 个真实 Chromium SPA 断言及 70 项对抗探针通过；新增 38 条单元检查、6 条探针，覆盖列表内代码围栏闭合及引用块边界。原始缺陷对照 v1.9.1/v1.9.2 复现。
+- 最终源码在本机桌面 0.2.0-rc.2（Electron 44.0.0 / Node 24.18.1）通过 17 项集成与 15 项 Chromium SPA 检查；官方 DSH 0.2.1-rc.2 通过 17 项服务集成及 9 项实际 DeepSeek Fetch 适配器检查。适配器传输使用受控响应，未覆盖外部 API 实际服务或完整 UI/模型会话。
+- 本机桌面升级为 0.2.1-rc.2（Electron 44.7.0 / Node 24.21.0）后，相同冻结源码在实际内置运行时另通过 17 项集成、3 项工具作用域与 15 项 Chromium SPA 检查。
+- 已审计八个接触包、Cordis/Loader 与四个桌面加载文件；社区升级卡截至 0.2.0-rc.2，新版由官方源码交叉核验。发布版 patch 另通过 4 项加载器及 2 项临时 profile 检查。
+- 2026-10-19 发布前 152 站：105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW，OK 站零噪声。相同 HTML/DOM 对照 v1.9.2 的 text 全文/续读一致，Markdown 结构修复无可见文字丢失或重复；波动站为验证页或旧版同现的响应边界。22 个错误分别为 13 个 HTTP 错误、3 个预期类型拒绝、6 个网络失败。
+- Bilibili 视频页在修复 gzip 解压后由乱码变为正常正文；实际 curl 8.19/8.21 共 12 个正常/超限/重定向/取消场景通过，包含旧版 curl 解压后越界保护。
+- 核心零运行时依赖；四工具 description/schema 保持 1,159/1,876 字符，属性长度与递归上限保留。
+
 ## [1.9.2] - 2026-10-03
 
 ### 修复

package/README.md

@@ -22,10 +22,12 @@ Continue at `charsStart + charsReturned`; the returned `charsStart` may exceed t
 
 ## Why
 
-DSH agents can search (getting links and snippets) but lack the step of "reading a URL into clean body text". The official `tool-web` `web_fetch` does a **whole-page turndown conversion** (nav/ads/sidebars all preserved) with a default cap of 200,000 characters — a token black hole. This plugin returns only what the model actually needs: **cleaned body + essential metadata**, truncated by default.
+This plugin keeps URL-reading output compact: **cleaned body + essential metadata** from HTML, paragraph-aligned truncation, continuation and a session cache. It also preserves plain text or Markdown returned by the configured DSH provider. DSH 0.2.1-rc.2 includes a DeepSeek Fetch API provider, so available source content and metadata depend on the selected provider.
 
 ### Competitor comparison (measured from source/docs, 2026-08-15)
 
+This is a historical snapshot; it does not describe the newer DeepSeek Fetch API provider.
+
 | Capability | Official `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
 |---|---|---|---|---|
 | Body cleaning (container-level) | ❌ whole page | ⚠️ tag-level, nav/footer leak in | ⚠️ custom, noisy | ✅ article/main containers + noise stripping |
@@ -339,6 +341,15 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - The 152-site sweep returned **104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW**, with no noise in OK results. Known NOISE and fluctuating sites matched v1.9.1 on identical HTML/DOM and optional dependencies, including full continuation and 800-character output. The newly failing feed returned HTTP 403 in both versions.
 - JSON-LD metadata/body extraction accepts valid attribute variants and rejects lookalike attributes. Author recursion is bounded; body arrays retain only strings. Core runtime dependencies remain zero, with fixed model input of **1,159 description characters / 1,876 schema characters**.
 
+### v1.9.3 Desktop maintenance validation (2026-10-19)
+
+- **413 zero-dependency unit assertions, 15 real Chromium SPA assertions and 70 adversarial probes** passed. Fixes cover nested Markdown containers/lists, fractional numeric configuration, literal text-provider content, proxy diagnostics, `NO_PROXY`, curl URL globbing/decompression and bounded challenge detection. Reproductions against v1.9.1/v1.9.2 confirm existing defects; regression tests also cover code-fence and quote boundaries within lists.
+- [DSH 0.2.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.1-rc.2), published October 17, was audited against 0.2.0-rc.2 across eight contact packages, Cordis/Loader and four Desktop loading files. The final plugin passed **17 official-service integration checks and 9 DeepSeek Fetch provider checks**. The published v1.9.2 patch also passed 4 loader and 2 temporary-profile checks. The [community upgrade card](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.2.md) currently stops at 0.2.0-rc.2; newer behavior was checked directly against official packages.
+- After upgrading Desktop to **0.2.1-rc.2 / Electron 44.7.0 / Node 24.21.0**, its actual bundled runtime passed **17 integration checks, 3 tool-scope checks and 15 Chromium SPA checks** against the same final plugin. The prior 0.2.0-rc.2 runtime also passed 17 integration and 15 SPA checks. Service responses were controlled; these checks do not cover a full model conversation or UI interaction.
+- The release 152-site sweep returned **105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW**, with no noise in OK results. Errors comprised 13 HTTP refusals/missing pages, 3 expected content-type rejections and 6 network failures. Saved HTML/DOM replays matched v1.9.2 in full text and continuation; Markdown restored structure without losing or duplicating visible text. Dynamic verification pages matched on identical input. The previously garbled Bilibili video page returned readable content after gzip decompression. Real curl 8.19/8.21 passed 12 decompression, size-limit, redirect and cancellation scenarios.
+- The DeepSeek Fetch provider returns Markdown as `body.kind=text`. Single-page reading preserves that text; links/site operations report that HTML navigation was not scanned, rather than interpreting HTML examples as links, redirects or SPA scripts. The provider does not expose the target's final URL or status, so HTML metadata, automatic pagination and site discovery are not equivalent to raw HTML retrieval.
+- Nested Markdown blocks retain paragraph boundaries, list nesting, parent numbering and separate code/quote endings from following text. Attribute scans and the 100-level recursion limit remain bounded. Proxy retrieval decompresses supported HTTP encodings and checks the decoded byte budget even with older curl versions. Core runtime dependencies remain zero; fixed model input remains **1,159 description characters / 1,876 schema characters**.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)

package/README.zh.md

@@ -22,10 +22,12 @@ DeepSeek Harness 的 URL 阅读插件：抓取任意 URL——**网页（HTML）
 
 ## 为什么做它
 
-DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净正文"这一步。官方 `tool-web` 的 `web_fetch` 是**整页 turndown 转换**（导航/广告/侧栏全保留），默认上限 20 万字符——token 黑洞。本插件只返回模型真正需要的：**净化后的正文 + 必要元数据**，并默认截断。
+本插件控制 URL 阅读的输出量：从 HTML 提取**净化后的正文 + 必要元数据**，提供段落截断、续读和会话缓存；也保留 DSH 配置的提供方返回的纯文本或 Markdown。DSH 0.2.1-rc.2 已包含 DeepSeek Fetch API 提供方，可获得的原文和元数据取决于所选提供方。
 
 ### 同类插件对比（2026-08-15 实测源码/文档）
 
+以下是历史快照，不代表新版 DeepSeek Fetch API 提供方的行为。
+
 | 能力 | 官方 `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
 |---|---|---|---|---|
 | 正文净化（容器级提取） | ❌ 整页渲染 | ⚠️ 标签级去噪，nav/footer 仍混入 | ⚠️ 自研，含噪音 | ✅ article/main 容器 + 噪音剥离 |
@@ -339,6 +341,15 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 - 152 站结果为 **104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW**，OK 站零噪声。已知 NOISE 与状态波动站使用相同 HTML/DOM 和可选依赖对照 v1.9.1，全文续读及 800 字输出一致；新增失败的订阅站在新旧版本均返回 HTTP 403。
 - JSON-LD 元数据/正文支持合法属性变体并排除伪属性；作者递归有界，正文数组仅保留字符串。核心仍为零运行时依赖，模型固定输入为 **description 1,159 字符 / schema 1,876 字符**。
 
+### v1.9.3 桌面版维护验证（2026-10-19）
+
+- **413 个零依赖单元断言、15 个真实 Chromium SPA 断言、70 项对抗探针**通过。修复覆盖 Markdown 嵌套容器/列表、小数配置、提供方纯文本、代理诊断、`NO_PROXY`、curl URL 展开/解压及有界质询页识别；v1.9.1/v1.9.2 对照确认原始缺陷既有，回归测试另覆盖列表内代码围栏闭合及引用块边界。
+- 10 月 17 日发布的 [DSH 0.2.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.1-rc.2) 已对照 0.2.0-rc.2 审计八个接触包、Cordis/Loader 及四个桌面加载链文件。最终插件通过 **17 项官方服务集成和 9 项 DeepSeek Fetch 提供方检查**；已发布 v1.9.2 的 patch 另通过 4 项加载器和 2 项临时 profile 检查。[社区升级卡](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.2.md)目前仅到 0.2.0-rc.2，新版行为直接用官方包核验。
+- 桌面升级至 **0.2.1-rc.2 / Electron 44.7.0 / Node 24.21.0** 后，实际内置运行时对相同最终插件通过 **17 项集成、3 项工具作用域和 15 项 Chromium SPA 检查**；升级前的 0.2.0-rc.2 运行时也已通过 17 项集成与 15 项 SPA。服务响应使用受控数据，未覆盖完整模型会话或 UI 交互。
+- 发布前 152 站为 **105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW**，OK 站零噪声；错误包括 13 个 HTTP 拒绝/页面不存在、3 个预期类型拒绝及 6 个网络失败。保存的同一 HTML/DOM 对照 v1.9.2，text 全文与续读一致，Markdown 恢复结构且无可见文字丢失或重复；动态验证页同输入结果一致。此前乱码的 Bilibili 视频页经 gzip 解压后返回正常正文。实际 curl 8.19/8.21 共通过 12 个解压、限额、跳转及取消场景。
+- DeepSeek Fetch 提供方以 `body.kind=text` 返回 Markdown。单页读取保留原文；链接/站点工具明确提示未扫描 HTML 导航，避免把 HTML 示例当作链接、跳转或 SPA 脚本。该提供方不暴露目标最终 URL 和状态，因此 HTML 元数据、自动分页和站点发现不能视为与原始 HTML 抓取等价。
+- Markdown 嵌套块保留段落边界、列表层级、父级编号，代码块/引用与后文正确分隔；属性扫描和 100 层递归上限继续有界。代理读取解压受支持的 HTTP 编码，并对旧版 curl 同样执行解压后字节限额。核心仍为零运行时依赖，模型固定输入仍为 **description 1,159 字符 / schema 1,876 字符**。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）

package/index.js

@@ -100,6 +100,10 @@ function allowedDirectUrl(url, origins) {
 const DIRECT_BOUNDARY_ERROR = 'Direct fetch stopped: target is outside directFetchOrigins or contains credentials'
 const DIRECT_SPA_HINT = '已配置直连仅支持静态读取，未启用浏览器渲染'
 
+function providerTextHint(page) {
+  return '提供方返回纯文本，未扫描 HTML 链接或跳转' + (page.sourceTruncated ? '；提供方已截断原文，当前内容不是完整页面' : '')
+}
+
 function getDecoder(enc) {
   let d = decoders.get(enc)
   if (!d) {
@@ -328,7 +332,7 @@ async function raceFetch(url, externalSignal, cfg, proxy) {
 
 async function fetchPage(url, externalSignal, cfg) {
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
-  const proxy = await detectProxy()
+  const proxy = await detectProxy(url)
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   if (proxy) {
     const winner = await raceFetch(url, externalSignal, cfg, proxy)
@@ -336,7 +340,7 @@ async function fetchPage(url, externalSignal, cfg) {
     const dErr = winner.failures.find((f) => f.label === 'direct')
     const pErr = winner.failures.find((f) => f.label === 'proxy')
     const px = pErr ? `，代理返回 ${pErr.error}` : '，代理未连通'
-    return { error: `Fetch failed: ${dErr ? dErr.error : 'unknown error'}（直连失败；已尝试代理 ${proxy}${px}）` }
+    return { error: `Fetch failed: ${dErr ? dErr.error : 'unknown error'}（直连失败；已尝试代理${px}）` }
   }
   // No proxy configured: plain direct connect (the original behaviour).
   const timeoutSignal = AbortSignal.timeout(cfg.timeoutMs)
@@ -955,18 +959,78 @@ export function inlineMd(html, depth = 0) {
   return img.restore(out.trim())
 }
 
-export function blockMd(html, depth = 0) {
+// Pair structural blocks in one forward pass so nested siblings keep their
+// own boundaries. Raw/code bodies and quoted attribute values are opaque.
+function markdownBlockPairs(html) {
+  const pairs = new Map()
+  const stacks = new Map()
+  const structural = new Set(['div', 'section', 'article', 'main', 'blockquote', 'ul', 'ol', 'li'])
+  const opaque = new Set(['pre', 'code', 'script', 'style', 'textarea', 'template', 'noscript'])
+  const re = /<!--|<(\/?)([a-z][a-z0-9:-]{0,63})(?=[\s/>])/gi
+  let m
+  while ((m = re.exec(html))) {
+    if (!m[2]) {
+      const end = html.indexOf('-->', re.lastIndex)
+      if (end < 0) break
+      re.lastIndex = end + 3
+      continue
+    }
+    let quote = '', end = -1
+    const limit = Math.min(html.length, m.index + 1002)
+    for (let i = re.lastIndex; i < limit; i++) {
+      const ch = html[i]
+      if (quote) {
+        if (ch === quote) quote = ''
+      } else if (ch === '"' || ch === "'") quote = ch
+      else if (ch === '<') break
+      else if (ch === '>') { end = i + 1; break }
+    }
+    if (end < 0) continue
+    const attrs = html.slice(re.lastIndex, end - 1)
+    re.lastIndex = end
+    const tag = m[2].toLowerCase()
+    if (!m[1] && opaque.has(tag)) {
+      const close = new RegExp(`</${tag}\\s{0,1000}>`, 'gi')
+      close.lastIndex = re.lastIndex
+      if (!close.exec(html)) break
+      re.lastIndex = close.lastIndex
+      continue
+    }
+    if (!structural.has(tag)) continue
+    if (!m[1]) {
+      if (!stacks.has(tag)) stacks.set(tag, [])
+      stacks.get(tag).push({ tag, attrs, index: m.index, start: re.lastIndex })
+    } else if (!attrs.trim()) {
+      const opening = stacks.get(tag)?.pop()
+      if (opening && m.index - opening.start <= 3145728) pairs.set(opening.index, { ...opening, close: m.index, end: re.lastIndex })
+    }
+  }
+  return pairs
+}
+
+export function blockMd(html, depth = 0, escapeText = false) {
   if (depth > MD_MAX_DEPTH) {
-    return decodeTextEntities(html.replace(/<[^>]{1,1000}>/g, ' ')).replace(/\s+/g, ' ').trim()
+    const text = decodeTextEntities(html.replace(/<[^>]{1,1000}>/g, ' ')).replace(/\s+/g, ' ').trim()
+    return escapeText ? escInline(text) : text
   }
   const img = imgsToMarkdown(html)
   html = stripUnmatchedOpeners(img.html)
+  const blocks = markdownBlockPairs(html)
   let out = ''
   const re = /<(?:br|hr)\b[^>]{0,1000}>|<([a-zA-Z0-9]{1,100})((?:"[^"]{0,1000}"|'[^']{0,1000}'|[^'">]){0,1000})>([\s\S]{0,3145728}?)<\/\1>|<[^>]{1,1000}>|([^<]{1,3145728})/gi
   let m
   while ((m = re.exec(html))) {
+    const block = blocks.get(m.index)
+    if (block) {
+      m[0] = html.slice(m.index, block.end)
+      m[1] = block.tag
+      m[2] = block.attrs
+      m[3] = html.slice(block.start, block.close)
+      re.lastIndex = block.end
+    }
     if (m[4] !== undefined) {
-      out += decodeTextEntities(m[4])
+      const text = decodeTextEntities(m[4])
+      out += escapeText ? escInline(text) : text
       continue
     }
     if (!m[1]) {
@@ -988,11 +1052,11 @@ export function blockMd(html, depth = 0) {
       const t = inlineMd(inner, depth + 1)
       if (t) out += `\n\n${t}`
     } else if (tag === 'div' || tag === 'section' || tag === 'article' || tag === 'main') {
-      const t = blockMd(inner, depth + 1)
-      if (t) out += `\n\n${t}`
+      const t = blockMd(inner, depth + 1, escapeText)
+      if (t) out += `\n\n${t}\n\n`
     } else if (tag === 'blockquote') {
-      const t = blockMd(inner, depth + 1).trim()
-      out += `\n\n> ${t.replace(/\n/g, '\n> ')}`
+      const t = blockMd(inner, depth + 1, escapeText).trim()
+      out += `\n\n> ${t.replace(/\n/g, '\n> ')}\n\n`
     } else if (tag === 'pre') {
       // Strip wrapper tags FIRST, then decode entities — a literal &lt;div&gt;
       // in the code must become <div> in the fenced block, but a real nested
@@ -1003,22 +1067,31 @@ export function blockMd(html, depth = 0) {
       const lang =
         /<code\b[^>]{0,1000}class=["'][^"']{0,1000}(?:language|lang)-([\w#+.-]{1,80})(?:[\s"'])/i.exec(inner)?.[1] || ''
       const fence = backtickFence(code, 3)
-      out += `\n\n${fence}${lang}\n${code}\n${fence}`
+      out += `\n\n${fence}${lang}\n${code}\n${fence}\n\n`
     } else if (tag === 'code') {
       out += img.protect(inlineCode(inner))
     } else if (tag === 'ul' || tag === 'ol') {
-      // Without a single </li> nothing can pair, so the item scan is skipped.
       const items = []
-      if (/<\/li\b/i.test(inner)) {
-        const liRe = /<li\b[^>]{0,1000}>([\s\S]{0,3145728}?)<\/li>/gi
-        let lm, idx = 1
-        while ((lm = liRe.exec(inner))) {
-          const t = inlineMd(lm[1], depth + 1)
-          items.push(tag === 'ol' ? `${idx}. ${t}` : `- ${t}`)
-          idx++
-        }
+      const itemBlocks = block ? blocks : markdownBlockPairs(inner)
+      const itemOffset = block ? block.start : 0
+      const liRe = /<li(?=[\s>])/gi
+      let lm, idx = 1
+      while ((lm = liRe.exec(inner))) {
+        const item = itemBlocks.get(itemOffset + lm.index)
+        if (!item) continue
+        liRe.lastIndex = item.end - itemOffset
+        const body = inner.slice(item.start - itemOffset, item.close - itemOffset)
+        const t = /<(?:ul|ol|div|section|article|main|p|pre|blockquote|h[1-6]|table)(?=[\s>])/i.test(body)
+          ? blockMd(body, depth + 1, true).replace(/\n{3,}/g, '\n\n').trim()
+          : inlineMd(body, depth + 1)
+        const prefix = tag === 'ol' ? `${idx++}. ` : '- '
+        const indent = ' '.repeat(prefix.length)
+        const lines = t.split('\n')
+        items.push(/^<(?:ul|ol)(?=[\s>])/i.test(body.trimStart())
+          ? prefix.trimEnd() + '\n' + lines.map(line => line ? indent + line : line).join('\n')
+          : prefix + lines.map((line, i) => i && line ? indent + line : line).join('\n'))
       }
-      if (items.length) out += `\n\n${items.join('\n')}`
+      if (items.length) out += `\n\n${items.join('\n')}\n\n`
     } else if (tag === 'li') {
       out += `\n- ${inlineMd(inner, depth + 1)}`
     } else if (tag === 'br') {
@@ -1061,7 +1134,7 @@ export function blockMd(html, depth = 0) {
       const t = inlineMd(m[0], depth + 1)
       if (t) out += t
     } else {
-      const t = blockMd(inner, depth + 1)
+      const t = blockMd(inner, depth + 1, escapeText)
       if (t) out += t
     }
   }
@@ -2197,6 +2270,7 @@ function readLinksTool(ctx, cfg) {
       const page = await fetchResource(ctx, url, signal, cfg, route)
       if (signal && signal.aborted) return { error: 'cancelled' }
       if (page.error) return { error: page.error }
+      if (page.kind === 'text') return { url: page.finalUrl || url, count: 0, links: [], spaHint: providerTextHint(page) }
       let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
       let finalUrl = page.finalUrl || url
       // Same meta-refresh shell following as read_url: a link hop serves a
@@ -2422,6 +2496,14 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         if (depth > 0 && !sameHost(pageUrl, host)) {
           return { url, depth, error: `Redirected outside the site: ${pageUrl}` }
         }
+        if (page.kind === 'text') {
+          if (depth === 0) host = hostOf(pageUrl)
+          return {
+            url: pageUrl, depth, title: '', chars: html.length,
+            text: includeContent ? smartTruncate(html, perMax).text : undefined,
+            links: [], route, spaHint: providerTextHint(page),
+          }
+        }
         // Same shell-following as read_url: a meta-refresh stub would otherwise
         // be crawled AS the page (title-less hop text pollutes results and the
         // real content is never reached). Fails open — fetch problems keep the
@@ -2574,7 +2656,7 @@ export function apply(ctx, config) {
   const cfg = { ...DEFAULTS, ...(config || {}) }
   const clamp = (v, d, min, max) => {
     const n = Number(v)
-    return Number.isFinite(n) ? Math.min(max, Math.max(min, n)) : d
+    return Number.isFinite(n) ? Math.floor(Math.min(max, Math.max(min, n))) : d
   }
   cfg.timeoutMs = clamp(cfg.timeoutMs, DEFAULTS.timeoutMs, 500, 120000)
   cfg.maxBytes = clamp(cfg.maxBytes, DEFAULTS.maxBytes, 65536, 16 * 1024 * 1024)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.9.2",
+  "version": "1.9.3",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",

package/proxy-fallback.js

@@ -1,21 +1,17 @@
-// proxy-fallback.js — retry a failed direct fetch through the user's proxy
+// proxy-fallback.js — optional proxy channel for direct webpage retrieval
 //
-// Design: direct connect first, proxy only as fallback. When the direct
-// fetch fails with a CONNECTION-class error (blocked network, refused,
-// DNS miss, connect timeout), we look up the user's own proxy — first the
-// standard HTTP_PROXY/HTTPS_PROXY env vars, then the Windows system proxy
-// (registry) where proxy apps like Clash actually persist it — and retry
-// once through it via the system `curl` (ships with Windows 10+/macOS/Linux;
-// zero npm deps). The proxy URL is passed explicitly with `-x`, so we don't
-// depend on curl's env-var reading.
+// When the host web provider is absent, configured proxy retrieval races
+// native fetch and the first success wins. Proxy discovery uses standard
+// HTTP_PROXY/HTTPS_PROXY env vars, then the Windows system proxy registry.
+// NO_PROXY exclusions retain only the native fetch. Proxy requests use system
+// curl (Windows 10+/macOS/Linux; zero npm deps) with an explicit `-x` URL.
 //
-// Transparency: a successful proxied read returns exactly like a direct one
-// (the caller never knows); a failed proxied attempt returns null and the
-// caller reports the ORIGINAL direct-connect error with a clear note, so the
-// model/user can act (turn the proxy on, or accept the network boundary).
+// Successful proxy responses use the direct channel's body shape. HTTP and
+// content errors remain diagnostic; unavailable curl or proxies return null.
 import { execFile, execFileSync } from 'node:child_process'
 import { existsSync } from 'node:fs'
 import { win32 } from 'node:path'
+import { BlockList, isIP } from 'node:net'
 
 // Desktop launches can inherit a reduced PATH. Prefer the Windows-shipped
 // utilities when present; other platforms and custom installations use PATH.
@@ -76,8 +72,7 @@ const SYS_PROXY_RETRY_MS = 60 * 1000
 
 // Windows system proxy from the registry (HKCU Internet Settings). PowerShell
 // is used because `reg.exe` is frequently blocked by security policy; the
-// read is synchronous and cached after the first call (it only runs after a
-// direct-connect failure, so the ~1s cost is invisible on the happy path).
+// read is synchronous and cached to avoid repeated discovery during reads.
 // A FAILED read is cached as '' but re-tried after 60s — an environment that
 // transiently blocks the PowerShell child (EDR, policy) should not starve
 // the whole session of the proxy.
@@ -107,8 +102,72 @@ function readWindowsSystemProxy() {
   }
 }
 
+// Match curl's domain/IP exclusions before starting a redundant direct curl
+// request. Port-qualified hosts and wildcard suffixes are also accepted.
+function bypassesProxy(url) {
+  const exclusions = process.env.no_proxy || process.env.NO_PROXY
+  if (!url || !exclusions) return false
+  let target
+  try { target = new URL(url) } catch { return false }
+  const hostname = target.hostname.toLowerCase().replace(/^\[|\]$/g, '').replace(/\.$/, '')
+  const ipVersion = isIP(hostname)
+  const port = target.port || (target.protocol === 'https:' ? '443' : '80')
+  for (let entry of exclusions.split(',')) {
+    entry = entry.trim().toLowerCase()
+    if (!entry) continue
+    if (entry === '*') return true
+    const slash = entry.indexOf('/')
+    if (slash >= 0) {
+      const address = entry.slice(0, slash)
+      const prefix = entry.slice(slash + 1)
+      if (!ipVersion || isIP(address) !== ipVersion || !/^\d{1,3}$/.test(prefix)) continue
+      try {
+        const block = new BlockList()
+        const family = ipVersion === 6 ? 'ipv6' : 'ipv4'
+        block.addSubnet(address, Number(prefix), family)
+        if (block.check(hostname, family)) return true
+      } catch {}
+      continue
+    }
+    let entryPort = ''
+    if (entry.startsWith('[')) {
+      const end = entry.indexOf(']')
+      if (end < 0) continue
+      const tail = entry.slice(end + 1)
+      if (tail && !/^:\d{1,5}$/.test(tail)) continue
+      entryPort = tail.slice(1)
+      entry = entry.slice(1, end)
+    } else if (!isIP(entry) && entry.indexOf(':') >= 0) {
+      const colon = entry.lastIndexOf(':')
+      entryPort = entry.slice(colon + 1)
+      entry = entry.slice(0, colon)
+      if (entry.includes(':') || !/^\d{1,5}$/.test(entryPort)) continue
+    }
+    if (entryPort && Number(entryPort) !== Number(port)) continue
+    if (isIP(entry)) {
+      if (!ipVersion) continue
+      try {
+        const canonical = new URL(isIP(entry) === 6 ? `http://[${entry}]` : `http://${entry}`).hostname
+          .replace(/^\[|\]$/g, '')
+        if (canonical === hostname) return true
+      } catch {}
+      continue
+    }
+    if (ipVersion) continue
+    if (entry.startsWith('*.')) {
+      if (hostname.endsWith(entry.slice(1).replace(/\.$/, ''))) return true
+      continue
+    }
+    else if (entry.startsWith('.')) entry = entry.slice(1)
+    entry = entry.replace(/\.$/, '')
+    if (entry && (hostname === entry || hostname.endsWith(`.${entry}`))) return true
+  }
+  return false
+}
+
 // The user's proxy, in priority order: env vars, then Windows system proxy.
-export async function detectProxy() {
+export async function detectProxy(url) {
+  if (bypassesProxy(url)) return ''
   const env =
     process.env.HTTPS_PROXY ||
     process.env.https_proxy ||
@@ -125,11 +184,14 @@ export async function detectProxy() {
 // `proxyOverride` lets the caller pass an already-detected proxy (the race
 // path probes once up front) instead of re-detecting here.
 export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride) {
-  const proxy = proxyOverride || (await detectProxy())
+  if (bypassesProxy(url)) return null
+  const proxy = proxyOverride || (await detectProxy(url))
   if (!proxy) return null
   const maxSec = Math.max(10, Math.ceil(cfg.timeoutMs / 1000) + 5)
   const args = [
     '-sS',
+    '--globoff',
+    '--compressed', // request and decode supported HTTP content encodings
     '-L', // follow redirects, like the direct fetch path
     '--max-time', String(maxSec),
     '--max-filesize', String(cfg.maxBytes), // unit is BYTES, not KB
@@ -154,6 +216,9 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
     if (nl < 0) return null // no trailing metadata — abnormal response, treat as failure
     const meta = buf.subarray(nl + 1).toString('latin1').trim()
     const body = buf.subarray(0, nl)
+    // Older curl versions bound the compressed transfer, not the decoded
+    // body. Do not let the metadata allowance enlarge the content budget.
+    if (body.length > cfg.maxBytes) return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
     const parts = meta.split(' ')
     const codeStr = parts.shift()
     const finalUrlEff = parts.length > 1 ? parts.pop() : url
@@ -174,7 +239,9 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
   } catch (err) {
     // curl error 63 = exceeded --max-filesize (page too large, not a proxy
     // problem — report it as such instead of "proxy unreachable")
-    if (err && err.code === 63) return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
+    if (err && (err.code === 63 || (err.code === 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER' && err.message.startsWith('stdout')))) {
+      return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
+    }
     // curl missing, proxy down, or cancelled — let the caller report the
     // original direct-connect error (the proxy attempt is best-effort)
     return null

package/spa.js

@@ -28,7 +28,7 @@ export function looksLikeSpa(html) {
 // stabilizes while the challenge sits in "Verification successful. Waiting
 // for ... to respond" — the poll loop returned that transitional page and
 // the caller mistook it for an improvement over static extraction.
-const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s*[|·]?\s*cloudflare|enable javascript and cookies to continue/i
+const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s{0,1000}(?:[|·]\s{0,1000})?cloudflare|enable javascript and cookies to continue/i
 
 export function looksLikeChallenge(html) {
   return typeof html === 'string' && CHALLENGE_RE.test(html)