dsh-read-url 1.9.2 → 1.9.4

package/CHANGELOG.md

@@ -1,5 +1,47 @@
 # Changelog
 
+## [1.9.4] - 2026-11-15
+
+### 修复
+
+- Windows 系统代理映射按请求的 HTTP/HTTPS 协议选择，缓存保留完整映射。环境变量同时配置两种代理时优先选择目标协议，单一代理配置保留兼容回退。
+- 链接扫描和站点读取保留 HTML 提供方的原文截断提示，刷新至完整目标后清除旧提示，刷新失败时保留当前页面提示。
+- 短前言或过早的句子边界不会耗尽整次正文读取，干净正文优先使用窗口后半段的段落或句子边界，并保留准确续读坐标。
+- 含原始 HTML 的窗口保留早期对齐，避免扩展短首段时引入更多超长属性残留，字面代码示例按原文续读。
+- 双语文档更新 DSH 0.2.2 HTTP 提供方的内网访问行为。
+
+### 验证
+
+- 435 个单元断言、15 个真实 Chromium SPA 断言、70 项对抗探针通过，新增 22 条单元检查。分协议代理选择、截断提示及短前言问题在 v1.9.2/v1.9.3 同输入复现。
+- 本机 DSH 桌面 0.2.2-rc.1 的 Electron 44.7.0 / Node 24.21.0 通过 17 项服务和 3 项作用域检查。官方 0.2.2-rc.3 通过 17 项服务、9 项 DeepSeek 提供方与 5 项 HTTP 提供方受控检查，发布版 patch 通过加载器 4 项和临时 profile 2 项检查。
+- 官方 npm latest 为 0.2.2-rc.1、next 为 0.2.2-rc.3。八个接触包、Cordis/Loader 与四个桌面加载文件完成字节差异审计，交叉核对截至 0.2.2-alpha.3 的社区升级卡。新版工具执行器的调整已纳入集成测试，插件消费接口保持兼容。
+- 干净正文随机续读及 emoji 边界检查通过，固定 800 字符窗口的截断耗时不随全文大小增长。原始 HTML 的长属性扫描限制继续保留，部分站点仍有既有残留标签边界。
+- 核心零运行时依赖，四工具 description/schema 为 1,159/1,876 字符。受控服务检查未覆盖真实模型会话或外部 DeepSeek API 服务。
+- 最终 152 站为 104 OK / 21 THIN / 4 EMPTY / 23 ERR / 0 THREW，OK 站零噪声，三个已知 NOISE 站点同输入对照 v1.9.3 一致。错误为 7 个 HTTP 错误、3 个预期类型拒绝、13 个连接/DNS 失败。
+
+## [1.9.3] - 2026-10-19
+
+### 修复
+
+- Markdown 按配对块处理嵌套容器与列表，保留段落间距、父级编号和子列表缩进；代码围栏/引用与后文正确分隔，属性与递归深度继续有界。
+- 数值配置经范围限制后向下取整，避免小数超时触发 `AbortSignal.timeout` 错误或小数分页上限多抓一页。
+- 代理回退遵守 `no_proxy`/`NO_PROXY` 的域名、IP、端口和 CIDR 边界，避免绕过代理的地址被直连与 curl 重复请求；curl 禁用 URL glob 展开，原样处理方括号查询和花括号路径。
+- curl 启用受支持的 HTTP 响应解压，避免 gzip 字节进入正文；单独检查解压后字节限额，兼容旧版 curl 的传输大小语义，stdout 超限返回明确错误。
+- 失败诊断不再输出代理配置地址，避免用户名、密码进入工具结果或模型上下文。
+- 提供方明确标记为纯文本时，链接和站点读取不再解析正文中的 HTML 示例或跟随示例跳转；保留原文、输出预算与提供方截断提示。
+- 质询页识别的空白段使用明确长度上限，消除相邻无界空白造成的平方回溯。
+- 双语说明更新 DeepSeek Fetch API 能力边界，并将早期官方工具对比明确标为历史快照。
+
+### 验证
+
+- 413 个零依赖单元断言、15 个真实 Chromium SPA 断言及 70 项对抗探针通过；新增 38 条单元检查、6 条探针，覆盖列表内代码围栏闭合及引用块边界。原始缺陷对照 v1.9.1/v1.9.2 复现。
+- 最终源码在本机桌面 0.2.0-rc.2（Electron 44.0.0 / Node 24.18.1）通过 17 项集成与 15 项 Chromium SPA 检查；官方 DSH 0.2.1-rc.2 通过 17 项服务集成及 9 项实际 DeepSeek Fetch 适配器检查。适配器传输使用受控响应，未覆盖外部 API 实际服务或完整 UI/模型会话。
+- 本机桌面升级为 0.2.1-rc.2（Electron 44.7.0 / Node 24.21.0）后，相同冻结源码在实际内置运行时另通过 17 项集成、3 项工具作用域与 15 项 Chromium SPA 检查。
+- 已审计八个接触包、Cordis/Loader 与四个桌面加载文件；社区升级卡截至 0.2.0-rc.2，新版由官方源码交叉核验。发布版 patch 另通过 4 项加载器及 2 项临时 profile 检查。
+- 2026-10-19 发布前 152 站：105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW，OK 站零噪声。相同 HTML/DOM 对照 v1.9.2 的 text 全文/续读一致，Markdown 结构修复无可见文字丢失或重复；波动站为验证页或旧版同现的响应边界。22 个错误分别为 13 个 HTTP 错误、3 个预期类型拒绝、6 个网络失败。
+- Bilibili 视频页在修复 gzip 解压后由乱码变为正常正文；实际 curl 8.19/8.21 共 12 个正常/超限/重定向/取消场景通过，包含旧版 curl 解压后越界保护。
+- 核心零运行时依赖；四工具 description/schema 保持 1,159/1,876 字符，属性长度与递归上限保留。
+
 ## [1.9.2] - 2026-10-03
 
 ### 修复

package/README.md

@@ -22,10 +22,12 @@ Continue at `charsStart + charsReturned`; the returned `charsStart` may exceed t
 
 ## Why
 
-DSH agents can search (getting links and snippets) but lack the step of "reading a URL into clean body text". The official `tool-web` `web_fetch` does a **whole-page turndown conversion** (nav/ads/sidebars all preserved) with a default cap of 200,000 characters — a token black hole. This plugin returns only what the model actually needs: **cleaned body + essential metadata**, truncated by default.
+This plugin keeps URL-reading output compact: **cleaned body + essential metadata** from HTML, paragraph-aligned truncation, continuation and a session cache. It also preserves plain text or Markdown returned by the configured DSH provider. DSH 0.2.1-rc.2 includes a DeepSeek Fetch API provider, so available source content and metadata depend on the selected provider.
 
 ### Competitor comparison (measured from source/docs, 2026-08-15)
 
+This is a historical snapshot; it does not describe the newer DeepSeek Fetch API provider.
+
 | Capability | Official `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
 |---|---|---|---|---|
 | Body cleaning (container-level) | ❌ whole page | ⚠️ tag-level, nav/footer leak in | ⚠️ custom, noisy | ✅ article/main containers + noise stripping |
@@ -202,7 +204,7 @@ Values are coerced and clamped to sane ranges at load — quoted numbers in YAML
       - https://raw.githubusercontent.com
 ```
 
-Listed origins use direct fetch immediately, bypassing the host web provider **and its SSRF checks**. Add only origins you trust; this also permits their DNS/hosts mappings to local or private addresses. The option is not a tool argument. Unlisted initial URLs keep the existing provider selection and fallback behavior; a provider refusal does not enable this exception automatically.
+Listed origins use direct fetch immediately, bypassing the host web provider and its access policy. Add only origins you trust, including their DNS/hosts mappings to local or private addresses. DSH 0.2.2's HTTP provider already supports local and private destinations. Other providers may apply different access policies. The option is configured in the profile. Unlisted initial URLs retain provider selection and fallback behavior. A provider refusal does not automatically enable explicit direct retrieval.
 
 Matching is exact after URL normalization: scheme, hostname and port must match; subdomains are separate. At most 32 entries of 2,048 characters each are accepted. Only an HTTP(S) origin with an optional trailing `/` is valid: credentials, paths, queries, fragments, wildcards and ambiguous syntax cause a configuration error. Unlike numeric options, invalid entries do not fall back to defaults.
 
@@ -339,6 +341,23 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - The 152-site sweep returned **104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW**, with no noise in OK results. Known NOISE and fluctuating sites matched v1.9.1 on identical HTML/DOM and optional dependencies, including full continuation and 800-character output. The newly failing feed returned HTTP 403 in both versions.
 - JSON-LD metadata/body extraction accepts valid attribute variants and rejects lookalike attributes. Author recursion is bounded; body arrays retain only strings. Core runtime dependencies remain zero, with fixed model input of **1,159 description characters / 1,876 schema characters**.
 
+### v1.9.3 Desktop maintenance validation (2026-10-19)
+
+- **413 zero-dependency unit assertions, 15 real Chromium SPA assertions and 70 adversarial probes** passed. Fixes cover nested Markdown containers/lists, fractional numeric configuration, literal text-provider content, proxy diagnostics, `NO_PROXY`, curl URL globbing/decompression and bounded challenge detection. Reproductions against v1.9.1/v1.9.2 confirm existing defects; regression tests also cover code-fence and quote boundaries within lists.
+- [DSH 0.2.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.1-rc.2), published October 17, was audited against 0.2.0-rc.2 across eight contact packages, Cordis/Loader and four Desktop loading files. The final plugin passed **17 official-service integration checks and 9 DeepSeek Fetch provider checks**. The published v1.9.2 patch also passed 4 loader and 2 temporary-profile checks. The [community upgrade card](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.2.md) currently stops at 0.2.0-rc.2; newer behavior was checked directly against official packages.
+- After upgrading Desktop to **0.2.1-rc.2 / Electron 44.7.0 / Node 24.21.0**, its actual bundled runtime passed **17 integration checks, 3 tool-scope checks and 15 Chromium SPA checks** against the same final plugin. The prior 0.2.0-rc.2 runtime also passed 17 integration and 15 SPA checks. Service responses were controlled; these checks do not cover a full model conversation or UI interaction.
+- The release 152-site sweep returned **105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW**, with no noise in OK results. Errors comprised 13 HTTP refusals/missing pages, 3 expected content-type rejections and 6 network failures. Saved HTML/DOM replays matched v1.9.2 in full text and continuation; Markdown restored structure without losing or duplicating visible text. Dynamic verification pages matched on identical input. The previously garbled Bilibili video page returned readable content after gzip decompression. Real curl 8.19/8.21 passed 12 decompression, size-limit, redirect and cancellation scenarios.
+- The DeepSeek Fetch provider returns Markdown as `body.kind=text`. Single-page reading preserves that text; links/site operations report that HTML navigation was not scanned, rather than interpreting HTML examples as links, redirects or SPA scripts. The provider does not expose the target's final URL or status, so HTML metadata, automatic pagination and site discovery are not equivalent to raw HTML retrieval.
+- Nested Markdown blocks retain paragraph boundaries, list nesting, parent numbering and separate code/quote endings from following text. Attribute scans and the 100-level recursion limit remain bounded. Proxy retrieval decompresses supported HTTP encodings and checks the decoded byte budget even with older curl versions. Core runtime dependencies remain zero; fixed model input remains **1,159 description characters / 1,876 schema characters**.
+
+### v1.9.4 Maintenance validation (2026-11-15)
+
+- **435 unit assertions, 15 real Chromium SPA assertions and 70 adversarial probes** passed. The 22 new unit checks cover protocol-specific proxy selection, HTML-provider truncation hints and continuation after short prose prefixes. Core runtime dependencies remain zero and fixed tool input remains **1,159 description characters / 1,876 schema characters**.
+- Official npm channels report `latest=0.2.2-rc.1` and `next=0.2.2-rc.3`, both release candidates. The [November 13 release](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.2-rc.3) was audited using byte differences across eight contact packages, Cordis/Loader and four Desktop loading files. Community upgrade cards currently reach 0.2.2-alpha.3. The checked plugin interfaces remain compatible.
+- The installed Desktop is **0.2.2-rc.1 / Electron 44.7.0 / Node 24.21.0**. Its bundled runtime passed **17 service checks and 3 tool-scope checks**, alongside the 15 SPA assertions. Official 0.2.2-rc.3 passed **17 service checks, 9 DeepSeek provider checks and 5 HTTP-provider checks** with controlled responses. The published plugin patch also passed 4 loader and 2 temporary-profile checks. Full model conversations and Desktop UI interactions were outside these checks.
+- Clean prose uses a paragraph or sentence boundary in the latter half of the requested window, avoiding tiny heading-only responses. Windows containing raw HTML retain earlier alignment. Literal examples remain available through exact-offset continuation. The 1,000-character attribute limit remains deliberate, so some pages with long attributes still expose residual markup or return a short initial window.
+- The final 152-site sweep completed with **104 OK / 21 THIN / 4 EMPTY / 23 ERR / 0 THREW**, with zero noise in OK results. The three known NOISE pages matched v1.9.3 on identical input. The 23 errors comprised 7 HTTP errors, 3 expected content-type rejections and 13 connection/DNS failures. Continuation and the five-page crawl completed successfully.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)

package/README.zh.md

@@ -22,10 +22,12 @@ DeepSeek Harness 的 URL 阅读插件：抓取任意 URL——**网页（HTML）
 
 ## 为什么做它
 
-DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净正文"这一步。官方 `tool-web` 的 `web_fetch` 是**整页 turndown 转换**（导航/广告/侧栏全保留），默认上限 20 万字符——token 黑洞。本插件只返回模型真正需要的：**净化后的正文 + 必要元数据**，并默认截断。
+本插件控制 URL 阅读的输出量：从 HTML 提取**净化后的正文 + 必要元数据**，提供段落截断、续读和会话缓存；也保留 DSH 配置的提供方返回的纯文本或 Markdown。DSH 0.2.1-rc.2 已包含 DeepSeek Fetch API 提供方，可获得的原文和元数据取决于所选提供方。
 
 ### 同类插件对比（2026-08-15 实测源码/文档）
 
+以下是历史快照，不代表新版 DeepSeek Fetch API 提供方的行为。
+
 | 能力 | 官方 `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
 |---|---|---|---|---|
 | 正文净化（容器级提取） | ❌ 整页渲染 | ⚠️ 标签级去噪，nav/footer 仍混入 | ⚠️ 自研，含噪音 | ✅ article/main 容器 + 噪音剥离 |
@@ -202,7 +204,7 @@ chars 800+800/12398 · cached
       - https://raw.githubusercontent.com
 ```
 
-命中的来源立即直连，跳过宿主 web provider **及其 SSRF 检查**。仅添加你信任的来源；其 DNS/hosts 映射到本机或私有地址也会被允许。此选项不开放为工具参数。未列出的初始 URL 保留既有 provider 选择与回退行为；provider 拒绝不会自动启用此例外。
+命中的来源立即直连，跳过宿主 web provider 及其访问策略。仅添加你信任的来源，包括其 DNS/hosts 到本机或私有地址的映射。DSH 0.2.2 的 HTTP 提供方已支持本机和内网地址，其他提供方可采用不同访问策略。此选项由 profile 配置。未列出的初始 URL 保留既有 provider 选择与回退行为，provider 拒绝不会自动启用显式直连。
 
 按 URL 规范化后的协议、域名、端口精确匹配，子域名需单独配置。最多 32 项，每项不超过 2,048 字符。仅接受 HTTP(S) origin，可带一个结尾 `/`；凭据、路径、查询、片段、通配符和歧义写法都会触发配置错误。此选项与数字选项不同，非法项不会静默回退默认值。
 
@@ -339,6 +341,23 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 - 152 站结果为 **104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW**，OK 站零噪声。已知 NOISE 与状态波动站使用相同 HTML/DOM 和可选依赖对照 v1.9.1，全文续读及 800 字输出一致；新增失败的订阅站在新旧版本均返回 HTTP 403。
 - JSON-LD 元数据/正文支持合法属性变体并排除伪属性；作者递归有界，正文数组仅保留字符串。核心仍为零运行时依赖，模型固定输入为 **description 1,159 字符 / schema 1,876 字符**。
 
+### v1.9.3 桌面版维护验证（2026-10-19）
+
+- **413 个零依赖单元断言、15 个真实 Chromium SPA 断言、70 项对抗探针**通过。修复覆盖 Markdown 嵌套容器/列表、小数配置、提供方纯文本、代理诊断、`NO_PROXY`、curl URL 展开/解压及有界质询页识别；v1.9.1/v1.9.2 对照确认原始缺陷既有，回归测试另覆盖列表内代码围栏闭合及引用块边界。
+- 10 月 17 日发布的 [DSH 0.2.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.1-rc.2) 已对照 0.2.0-rc.2 审计八个接触包、Cordis/Loader 及四个桌面加载链文件。最终插件通过 **17 项官方服务集成和 9 项 DeepSeek Fetch 提供方检查**；已发布 v1.9.2 的 patch 另通过 4 项加载器和 2 项临时 profile 检查。[社区升级卡](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.2.md)目前仅到 0.2.0-rc.2，新版行为直接用官方包核验。
+- 桌面升级至 **0.2.1-rc.2 / Electron 44.7.0 / Node 24.21.0** 后，实际内置运行时对相同最终插件通过 **17 项集成、3 项工具作用域和 15 项 Chromium SPA 检查**；升级前的 0.2.0-rc.2 运行时也已通过 17 项集成与 15 项 SPA。服务响应使用受控数据，未覆盖完整模型会话或 UI 交互。
+- 发布前 152 站为 **105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW**，OK 站零噪声；错误包括 13 个 HTTP 拒绝/页面不存在、3 个预期类型拒绝及 6 个网络失败。保存的同一 HTML/DOM 对照 v1.9.2，text 全文与续读一致，Markdown 恢复结构且无可见文字丢失或重复；动态验证页同输入结果一致。此前乱码的 Bilibili 视频页经 gzip 解压后返回正常正文。实际 curl 8.19/8.21 共通过 12 个解压、限额、跳转及取消场景。
+- DeepSeek Fetch 提供方以 `body.kind=text` 返回 Markdown。单页读取保留原文；链接/站点工具明确提示未扫描 HTML 导航，避免把 HTML 示例当作链接、跳转或 SPA 脚本。该提供方不暴露目标最终 URL 和状态，因此 HTML 元数据、自动分页和站点发现不能视为与原始 HTML 抓取等价。
+- Markdown 嵌套块保留段落边界、列表层级、父级编号，代码块/引用与后文正确分隔；属性扫描和 100 层递归上限继续有界。代理读取解压受支持的 HTTP 编码，并对旧版 curl 同样执行解压后字节限额。核心仍为零运行时依赖，模型固定输入仍为 **description 1,159 字符 / schema 1,876 字符**。
+
+### v1.9.4 维护验证（2026-11-15）
+
+- **435 个单元断言、15 个真实 Chromium SPA 断言、70 项对抗探针**通过。新增 22 条单元检查覆盖分协议代理选择、HTML 提供方截断提示及短前言后的续读。核心零运行时依赖，工具固定输入仍为 **description 1,159 字符 / schema 1,876 字符**。
+- 官方 npm 频道为 `latest=0.2.2-rc.1`、`next=0.2.2-rc.3`，均为候选版本。[11 月 13 日发布的 rc.3](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.2-rc.3) 已通过八个接触包、Cordis/Loader 及四个桌面加载文件的字节差异审计。社区升级卡目前到 0.2.2-alpha.3，已核查的插件接口保持兼容。
+- 本机桌面为 **0.2.2-rc.1 / Electron 44.7.0 / Node 24.21.0**，内置运行时通过 **17 项服务集成、3 项工具作用域**检查，并通过上述 15 项 SPA 断言。官方 0.2.2-rc.3 通过 **17 项服务、9 项 DeepSeek 提供方和 5 项 HTTP 提供方**受控检查，已发布插件 patch 另通过 4 项加载器和 2 项临时 profile 检查。验证范围未覆盖完整模型会话和桌面 UI 交互。
+- 干净正文在请求窗口的后半段选择段落或句子边界，避免只返回极短标题。含原始 HTML 的窗口保留早期对齐，字面示例可按精确 offset 继续读取。属性长度继续限制为 1,000 字符，部分超长属性页面仍可能残留标签或返回较短的首段。
+- 最终 152 站为 **104 OK / 21 THIN / 4 EMPTY / 23 ERR / 0 THREW**，OK 站零噪声。三个已知 NOISE 页面在同一输入下与 v1.9.3 一致。23 个错误包括 7 个 HTTP 错误、3 个预期类型拒绝及 13 个连接/DNS 失败，续读和五页站点抓取完成。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）

package/index.js

@@ -99,6 +99,11 @@ function allowedDirectUrl(url, origins) {
 
 const DIRECT_BOUNDARY_ERROR = 'Direct fetch stopped: target is outside directFetchOrigins or contains credentials'
 const DIRECT_SPA_HINT = '已配置直连仅支持静态读取，未启用浏览器渲染'
+const SOURCE_TRUNCATED_HINT = '提供方已截断原文，当前内容不是完整页面'
+
+function providerTextHint(page) {
+  return '提供方返回纯文本，未扫描 HTML 链接或跳转' + (page.sourceTruncated ? '；提供方已截断原文，当前内容不是完整页面' : '')
+}
 
 function getDecoder(enc) {
   let d = decoders.get(enc)
@@ -328,7 +333,7 @@ async function raceFetch(url, externalSignal, cfg, proxy) {
 
 async function fetchPage(url, externalSignal, cfg) {
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
-  const proxy = await detectProxy()
+  const proxy = await detectProxy(url)
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   if (proxy) {
     const winner = await raceFetch(url, externalSignal, cfg, proxy)
@@ -336,7 +341,7 @@ async function fetchPage(url, externalSignal, cfg) {
     const dErr = winner.failures.find((f) => f.label === 'direct')
     const pErr = winner.failures.find((f) => f.label === 'proxy')
     const px = pErr ? `，代理返回 ${pErr.error}` : '，代理未连通'
-    return { error: `Fetch failed: ${dErr ? dErr.error : 'unknown error'}（直连失败；已尝试代理 ${proxy}${px}）` }
+    return { error: `Fetch failed: ${dErr ? dErr.error : 'unknown error'}（直连失败；已尝试代理${px}）` }
   }
   // No proxy configured: plain direct connect (the original behaviour).
   const timeoutSignal = AbortSignal.timeout(cfg.timeoutMs)
@@ -955,18 +960,78 @@ export function inlineMd(html, depth = 0) {
   return img.restore(out.trim())
 }
 
-export function blockMd(html, depth = 0) {
+// Pair structural blocks in one forward pass so nested siblings keep their
+// own boundaries. Raw/code bodies and quoted attribute values are opaque.
+function markdownBlockPairs(html) {
+  const pairs = new Map()
+  const stacks = new Map()
+  const structural = new Set(['div', 'section', 'article', 'main', 'blockquote', 'ul', 'ol', 'li'])
+  const opaque = new Set(['pre', 'code', 'script', 'style', 'textarea', 'template', 'noscript'])
+  const re = /<!--|<(\/?)([a-z][a-z0-9:-]{0,63})(?=[\s/>])/gi
+  let m
+  while ((m = re.exec(html))) {
+    if (!m[2]) {
+      const end = html.indexOf('-->', re.lastIndex)
+      if (end < 0) break
+      re.lastIndex = end + 3
+      continue
+    }
+    let quote = '', end = -1
+    const limit = Math.min(html.length, m.index + 1002)
+    for (let i = re.lastIndex; i < limit; i++) {
+      const ch = html[i]
+      if (quote) {
+        if (ch === quote) quote = ''
+      } else if (ch === '"' || ch === "'") quote = ch
+      else if (ch === '<') break
+      else if (ch === '>') { end = i + 1; break }
+    }
+    if (end < 0) continue
+    const attrs = html.slice(re.lastIndex, end - 1)
+    re.lastIndex = end
+    const tag = m[2].toLowerCase()
+    if (!m[1] && opaque.has(tag)) {
+      const close = new RegExp(`</${tag}\\s{0,1000}>`, 'gi')
+      close.lastIndex = re.lastIndex
+      if (!close.exec(html)) break
+      re.lastIndex = close.lastIndex
+      continue
+    }
+    if (!structural.has(tag)) continue
+    if (!m[1]) {
+      if (!stacks.has(tag)) stacks.set(tag, [])
+      stacks.get(tag).push({ tag, attrs, index: m.index, start: re.lastIndex })
+    } else if (!attrs.trim()) {
+      const opening = stacks.get(tag)?.pop()
+      if (opening && m.index - opening.start <= 3145728) pairs.set(opening.index, { ...opening, close: m.index, end: re.lastIndex })
+    }
+  }
+  return pairs
+}
+
+export function blockMd(html, depth = 0, escapeText = false) {
   if (depth > MD_MAX_DEPTH) {
-    return decodeTextEntities(html.replace(/<[^>]{1,1000}>/g, ' ')).replace(/\s+/g, ' ').trim()
+    const text = decodeTextEntities(html.replace(/<[^>]{1,1000}>/g, ' ')).replace(/\s+/g, ' ').trim()
+    return escapeText ? escInline(text) : text
   }
   const img = imgsToMarkdown(html)
   html = stripUnmatchedOpeners(img.html)
+  const blocks = markdownBlockPairs(html)
   let out = ''
   const re = /<(?:br|hr)\b[^>]{0,1000}>|<([a-zA-Z0-9]{1,100})((?:"[^"]{0,1000}"|'[^']{0,1000}'|[^'">]){0,1000})>([\s\S]{0,3145728}?)<\/\1>|<[^>]{1,1000}>|([^<]{1,3145728})/gi
   let m
   while ((m = re.exec(html))) {
+    const block = blocks.get(m.index)
+    if (block) {
+      m[0] = html.slice(m.index, block.end)
+      m[1] = block.tag
+      m[2] = block.attrs
+      m[3] = html.slice(block.start, block.close)
+      re.lastIndex = block.end
+    }
     if (m[4] !== undefined) {
-      out += decodeTextEntities(m[4])
+      const text = decodeTextEntities(m[4])
+      out += escapeText ? escInline(text) : text
       continue
     }
     if (!m[1]) {
@@ -988,11 +1053,11 @@ export function blockMd(html, depth = 0) {
       const t = inlineMd(inner, depth + 1)
       if (t) out += `\n\n${t}`
     } else if (tag === 'div' || tag === 'section' || tag === 'article' || tag === 'main') {
-      const t = blockMd(inner, depth + 1)
-      if (t) out += `\n\n${t}`
+      const t = blockMd(inner, depth + 1, escapeText)
+      if (t) out += `\n\n${t}\n\n`
     } else if (tag === 'blockquote') {
-      const t = blockMd(inner, depth + 1).trim()
-      out += `\n\n> ${t.replace(/\n/g, '\n> ')}`
+      const t = blockMd(inner, depth + 1, escapeText).trim()
+      out += `\n\n> ${t.replace(/\n/g, '\n> ')}\n\n`
     } else if (tag === 'pre') {
       // Strip wrapper tags FIRST, then decode entities — a literal &lt;div&gt;
       // in the code must become <div> in the fenced block, but a real nested
@@ -1003,22 +1068,31 @@ export function blockMd(html, depth = 0) {
       const lang =
         /<code\b[^>]{0,1000}class=["'][^"']{0,1000}(?:language|lang)-([\w#+.-]{1,80})(?:[\s"'])/i.exec(inner)?.[1] || ''
       const fence = backtickFence(code, 3)
-      out += `\n\n${fence}${lang}\n${code}\n${fence}`
+      out += `\n\n${fence}${lang}\n${code}\n${fence}\n\n`
     } else if (tag === 'code') {
       out += img.protect(inlineCode(inner))
     } else if (tag === 'ul' || tag === 'ol') {
-      // Without a single </li> nothing can pair, so the item scan is skipped.
       const items = []
-      if (/<\/li\b/i.test(inner)) {
-        const liRe = /<li\b[^>]{0,1000}>([\s\S]{0,3145728}?)<\/li>/gi
-        let lm, idx = 1
-        while ((lm = liRe.exec(inner))) {
-          const t = inlineMd(lm[1], depth + 1)
-          items.push(tag === 'ol' ? `${idx}. ${t}` : `- ${t}`)
-          idx++
-        }
+      const itemBlocks = block ? blocks : markdownBlockPairs(inner)
+      const itemOffset = block ? block.start : 0
+      const liRe = /<li(?=[\s>])/gi
+      let lm, idx = 1
+      while ((lm = liRe.exec(inner))) {
+        const item = itemBlocks.get(itemOffset + lm.index)
+        if (!item) continue
+        liRe.lastIndex = item.end - itemOffset
+        const body = inner.slice(item.start - itemOffset, item.close - itemOffset)
+        const t = /<(?:ul|ol|div|section|article|main|p|pre|blockquote|h[1-6]|table)(?=[\s>])/i.test(body)
+          ? blockMd(body, depth + 1, true).replace(/\n{3,}/g, '\n\n').trim()
+          : inlineMd(body, depth + 1)
+        const prefix = tag === 'ol' ? `${idx++}. ` : '- '
+        const indent = ' '.repeat(prefix.length)
+        const lines = t.split('\n')
+        items.push(/^<(?:ul|ol)(?=[\s>])/i.test(body.trimStart())
+          ? prefix.trimEnd() + '\n' + lines.map(line => line ? indent + line : line).join('\n')
+          : prefix + lines.map((line, i) => i && line ? indent + line : line).join('\n'))
       }
-      if (items.length) out += `\n\n${items.join('\n')}`
+      if (items.length) out += `\n\n${items.join('\n')}\n\n`
     } else if (tag === 'li') {
       out += `\n- ${inlineMd(inner, depth + 1)}`
     } else if (tag === 'br') {
@@ -1061,7 +1135,7 @@ export function blockMd(html, depth = 0) {
       const t = inlineMd(m[0], depth + 1)
       if (t) out += t
     } else {
-      const t = blockMd(inner, depth + 1)
+      const t = blockMd(inner, depth + 1, escapeText)
       if (t) out += t
     }
   }
@@ -1422,15 +1496,21 @@ export function smartTruncate(text, maxChars, offset = 0) {
   }
   let end = Math.min(total, start + maxChars)
   if (end < total && maxChars > 0) {
+    // Prefer useful prose boundaries; raw HTML permits earlier boundaries
+    // to avoid returning unparsed attributes.
+    const rawTag = /<\/?[a-z][a-z0-9:-]{0,63}(?=[\s/>])/i.test(text.slice(start, end))
+    const minEnd = start + (rawTag ? 1 : Math.ceil(maxChars / 2))
     const window = text.slice(start, end + 2)
     let boundary = window.lastIndexOf('\n\n', end - start)
     if (boundary >= 0) boundary += start
     while (boundary > start && text[boundary - 1] === '\n') boundary--
-    if (boundary > start) end = boundary
+    if (boundary >= minEnd) end = boundary
     else {
       // Prefer the last complete sentence inside the current window.
-      for (let i = end - 1; i >= start; i--) {
-        if ('.!?\n。！？'.includes(text[i])) {
+      for (let i = end - 1; i >= minEnd - 1; i--) {
+        const ch = text[i]
+        // Rejected paragraph separators are not sentence boundaries either.
+        if ('.!?。！？'.includes(ch) || (ch === '\n' && text[i - 1] !== '\n' && text[i + 1] !== '\n')) {
           end = i + 1
           break
         }
@@ -1901,7 +1981,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   if (payload.error) return failWith(payload.error)
   const resultUrl = payload.finalUrl || url
-  const truncatedHint = '提供方已截断原文，当前内容不是完整页面'
+  const truncatedHint = SOURCE_TRUNCATED_HINT
   let sourceHint = payload.sourceTruncated ? truncatedHint : ''
 
   if (payload.kind === 'text') {
@@ -2197,6 +2277,7 @@ function readLinksTool(ctx, cfg) {
       const page = await fetchResource(ctx, url, signal, cfg, route)
       if (signal && signal.aborted) return { error: 'cancelled' }
       if (page.error) return { error: page.error }
+      if (page.kind === 'text') return { url: page.finalUrl || url, count: 0, links: [], spaHint: providerTextHint(page) }
       let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
       let finalUrl = page.finalUrl || url
       // Same meta-refresh shell following as read_url: a link hop serves a
@@ -2205,6 +2286,7 @@ function readLinksTool(ctx, cfg) {
       if (signal && signal.aborted) return { error: 'cancelled' }
       html = followed.html
       finalUrl = followed.finalUrl
+      let sourceTruncated = followed.sourceTruncated ?? page.sourceTruncated
       let links = extractLinks(html, limit, finalUrl)
       // SPA fallback: a JS-only page yields no links from its static HTML;
       // render it and re-extract when the static result looks empty. Any
@@ -2221,10 +2303,11 @@ function readLinksTool(ctx, cfg) {
           if (rl.length > links.length) {
             links = rl
             finalUrl = rr.finalUrl || finalUrl
+            sourceTruncated = false
           }
         }
       }
-      const spaHint = [route.hint, route.restricted && needsRender ? DIRECT_SPA_HINT : ''].filter(Boolean).join('；')
+      const spaHint = [sourceTruncated ? SOURCE_TRUNCATED_HINT : '', route.hint, route.restricted && needsRender ? DIRECT_SPA_HINT : ''].filter(Boolean).join('；')
       return { url: finalUrl, count: links.length, links, ...(spaHint ? { spaHint } : {}) }
     },
   }
@@ -2422,6 +2505,14 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         if (depth > 0 && !sameHost(pageUrl, host)) {
           return { url, depth, error: `Redirected outside the site: ${pageUrl}` }
         }
+        if (page.kind === 'text') {
+          if (depth === 0) host = hostOf(pageUrl)
+          return {
+            url: pageUrl, depth, title: '', chars: html.length,
+            text: includeContent ? smartTruncate(html, perMax).text : undefined,
+            links: [], route, spaHint: providerTextHint(page),
+          }
+        }
         // Same shell-following as read_url: a meta-refresh stub would otherwise
         // be crawled AS the page (title-less hop text pollutes results and the
         // real content is never reached). Fails open — fetch problems keep the
@@ -2430,12 +2521,14 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         if (externalSignal && externalSignal.aborted) return null
         html = followed.html
         pageUrl = followed.finalUrl
+        const sourceTruncated = followed.sourceTruncated ?? page.sourceTruncated
         if (depth === 0) host = hostOf(pageUrl)
         else if (!sameHost(pageUrl, host)) {
           return { url, depth, error: `Redirected outside the site: ${pageUrl}` }
         }
         const ex = extract(html, 'text')
         const links = extractLinks(html, 100, pageUrl)
+        const spaHint = [sourceTruncated ? SOURCE_TRUNCATED_HINT : '', route.hint].filter(Boolean).join('；')
         return {
           url: pageUrl,
           depth,
@@ -2444,7 +2537,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
           text: includeContent ? smartTruncate(ex.text, perMax).text : undefined,
           links,
           route,
-          ...(route.hint ? { spaHint: route.hint } : {}),
+          ...(spaHint ? { spaHint } : {}),
         }
       }),
     )
@@ -2574,7 +2667,7 @@ export function apply(ctx, config) {
   const cfg = { ...DEFAULTS, ...(config || {}) }
   const clamp = (v, d, min, max) => {
     const n = Number(v)
-    return Number.isFinite(n) ? Math.min(max, Math.max(min, n)) : d
+    return Number.isFinite(n) ? Math.floor(Math.min(max, Math.max(min, n))) : d
   }
   cfg.timeoutMs = clamp(cfg.timeoutMs, DEFAULTS.timeoutMs, 500, 120000)
   cfg.maxBytes = clamp(cfg.maxBytes, DEFAULTS.maxBytes, 65536, 16 * 1024 * 1024)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.9.2",
+  "version": "1.9.4",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",

package/proxy-fallback.js

@@ -1,21 +1,17 @@
-// proxy-fallback.js — retry a failed direct fetch through the user's proxy
+// proxy-fallback.js — optional proxy channel for direct webpage retrieval
 //
-// Design: direct connect first, proxy only as fallback. When the direct
-// fetch fails with a CONNECTION-class error (blocked network, refused,
-// DNS miss, connect timeout), we look up the user's own proxy — first the
-// standard HTTP_PROXY/HTTPS_PROXY env vars, then the Windows system proxy
-// (registry) where proxy apps like Clash actually persist it — and retry
-// once through it via the system `curl` (ships with Windows 10+/macOS/Linux;
-// zero npm deps). The proxy URL is passed explicitly with `-x`, so we don't
-// depend on curl's env-var reading.
+// When the host web provider is absent, configured proxy retrieval races
+// native fetch and the first success wins. Proxy discovery uses standard
+// HTTP_PROXY/HTTPS_PROXY env vars, then the Windows system proxy registry.
+// NO_PROXY exclusions retain only the native fetch. Proxy requests use system
+// curl (Windows 10+/macOS/Linux; zero npm deps) with an explicit `-x` URL.
 //
-// Transparency: a successful proxied read returns exactly like a direct one
-// (the caller never knows); a failed proxied attempt returns null and the
-// caller reports the ORIGINAL direct-connect error with a clear note, so the
-// model/user can act (turn the proxy on, or accept the network boundary).
+// Successful proxy responses use the direct channel's body shape. HTTP and
+// content errors remain diagnostic; unavailable curl or proxies return null.
 import { execFile, execFileSync } from 'node:child_process'
 import { existsSync } from 'node:fs'
 import { win32 } from 'node:path'
+import { BlockList, isIP } from 'node:net'
 
 // Desktop launches can inherit a reduced PATH. Prefer the Windows-shipped
 // utilities when present; other platforms and custom installations use PATH.
@@ -76,8 +72,7 @@ const SYS_PROXY_RETRY_MS = 60 * 1000
 
 // Windows system proxy from the registry (HKCU Internet Settings). PowerShell
 // is used because `reg.exe` is frequently blocked by security policy; the
-// read is synchronous and cached after the first call (it only runs after a
-// direct-connect failure, so the ~1s cost is invisible on the happy path).
+// read is synchronous and cached to avoid repeated discovery during reads.
 // A FAILED read is cached as '' but re-tried after 60s — an environment that
 // transiently blocks the PowerShell child (EDR, policy) should not starve
 // the whole session of the proxy.
@@ -97,9 +92,8 @@ function readWindowsSystemProxy() {
     )
     const [enable, server] = out.trim().split(/\r?\n/).map((s) => s.trim())
     if (enable !== '1' || !server) return sysProxyCache
-    // ProxyServer may be "host:port" or "http=...;https=host:port"
-    const httpsPart = server.split(';').map((s) => s.trim()).find((s) => /^https=/i.test(s))
-    sysProxyCache = (httpsPart ? httpsPart.replace(/^https=/i, '') : server) || ''
+    // Cache the complete map so HTTP and HTTPS share discovery, not selection.
+    sysProxyCache = server
     return sysProxyCache
   } catch {
     sysProxyCache = ''
@@ -107,15 +101,90 @@ function readWindowsSystemProxy() {
   }
 }
 
+// Match curl's domain/IP exclusions before starting a redundant direct curl
+// request. Port-qualified hosts and wildcard suffixes are also accepted.
+function bypassesProxy(url) {
+  const exclusions = process.env.no_proxy || process.env.NO_PROXY
+  if (!url || !exclusions) return false
+  let target
+  try { target = new URL(url) } catch { return false }
+  const hostname = target.hostname.toLowerCase().replace(/^\[|\]$/g, '').replace(/\.$/, '')
+  const ipVersion = isIP(hostname)
+  const port = target.port || (target.protocol === 'https:' ? '443' : '80')
+  for (let entry of exclusions.split(',')) {
+    entry = entry.trim().toLowerCase()
+    if (!entry) continue
+    if (entry === '*') return true
+    const slash = entry.indexOf('/')
+    if (slash >= 0) {
+      const address = entry.slice(0, slash)
+      const prefix = entry.slice(slash + 1)
+      if (!ipVersion || isIP(address) !== ipVersion || !/^\d{1,3}$/.test(prefix)) continue
+      try {
+        const block = new BlockList()
+        const family = ipVersion === 6 ? 'ipv6' : 'ipv4'
+        block.addSubnet(address, Number(prefix), family)
+        if (block.check(hostname, family)) return true
+      } catch {}
+      continue
+    }
+    let entryPort = ''
+    if (entry.startsWith('[')) {
+      const end = entry.indexOf(']')
+      if (end < 0) continue
+      const tail = entry.slice(end + 1)
+      if (tail && !/^:\d{1,5}$/.test(tail)) continue
+      entryPort = tail.slice(1)
+      entry = entry.slice(1, end)
+    } else if (!isIP(entry) && entry.indexOf(':') >= 0) {
+      const colon = entry.lastIndexOf(':')
+      entryPort = entry.slice(colon + 1)
+      entry = entry.slice(0, colon)
+      if (entry.includes(':') || !/^\d{1,5}$/.test(entryPort)) continue
+    }
+    if (entryPort && Number(entryPort) !== Number(port)) continue
+    if (isIP(entry)) {
+      if (!ipVersion) continue
+      try {
+        const canonical = new URL(isIP(entry) === 6 ? `http://[${entry}]` : `http://${entry}`).hostname
+          .replace(/^\[|\]$/g, '')
+        if (canonical === hostname) return true
+      } catch {}
+      continue
+    }
+    if (ipVersion) continue
+    if (entry.startsWith('*.')) {
+      if (hostname.endsWith(entry.slice(1).replace(/\.$/, ''))) return true
+      continue
+    }
+    else if (entry.startsWith('.')) entry = entry.slice(1)
+    entry = entry.replace(/\.$/, '')
+    if (entry && (hostname === entry || hostname.endsWith(`.${entry}`))) return true
+  }
+  return false
+}
+
 // The user's proxy, in priority order: env vars, then Windows system proxy.
-export async function detectProxy() {
-  const env =
-    process.env.HTTPS_PROXY ||
-    process.env.https_proxy ||
-    process.env.HTTP_PROXY ||
-    process.env.http_proxy
+export async function detectProxy(url) {
+  if (bypassesProxy(url)) return ''
+  let protocol = ''
+  try { protocol = new URL(url).protocol.slice(0, -1) } catch {}
+  const httpProxy = process.env.HTTP_PROXY || process.env.http_proxy
+  const httpsProxy = process.env.HTTPS_PROXY || process.env.https_proxy
+  // Prefer the requested protocol; retain a sole configured proxy as fallback.
+  const env = protocol === 'http' ? httpProxy || httpsProxy : httpsProxy || httpProxy
   if (env) return env
-  if (process.platform === 'win32') return readWindowsSystemProxy()
+  if (process.platform === 'win32') {
+    const server = readWindowsSystemProxy()
+    const entries = server.split(';').map(entry => entry.trim())
+    if (!entries.some(entry => /^[a-z]{1,16}\s{0,16}=/i.test(entry))) return server
+    const proxies = new Map()
+    for (const entry of entries) {
+      const equal = entry.indexOf('=')
+      if (equal > 0) proxies.set(entry.slice(0, equal).trim().toLowerCase(), entry.slice(equal + 1).trim())
+    }
+    return protocol ? proxies.get(protocol) || '' : proxies.get('https') || proxies.get('http') || ''
+  }
   return ''
 }
 
@@ -125,11 +194,14 @@ export async function detectProxy() {
 // `proxyOverride` lets the caller pass an already-detected proxy (the race
 // path probes once up front) instead of re-detecting here.
 export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride) {
-  const proxy = proxyOverride || (await detectProxy())
+  if (bypassesProxy(url)) return null
+  const proxy = proxyOverride || (await detectProxy(url))
   if (!proxy) return null
   const maxSec = Math.max(10, Math.ceil(cfg.timeoutMs / 1000) + 5)
   const args = [
     '-sS',
+    '--globoff',
+    '--compressed', // request and decode supported HTTP content encodings
     '-L', // follow redirects, like the direct fetch path
     '--max-time', String(maxSec),
     '--max-filesize', String(cfg.maxBytes), // unit is BYTES, not KB
@@ -154,6 +226,9 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
     if (nl < 0) return null // no trailing metadata — abnormal response, treat as failure
     const meta = buf.subarray(nl + 1).toString('latin1').trim()
     const body = buf.subarray(0, nl)
+    // Older curl versions bound the compressed transfer, not the decoded
+    // body. Do not let the metadata allowance enlarge the content budget.
+    if (body.length > cfg.maxBytes) return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
     const parts = meta.split(' ')
     const codeStr = parts.shift()
     const finalUrlEff = parts.length > 1 ? parts.pop() : url
@@ -174,7 +249,9 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
   } catch (err) {
     // curl error 63 = exceeded --max-filesize (page too large, not a proxy
     // problem — report it as such instead of "proxy unreachable")
-    if (err && err.code === 63) return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
+    if (err && (err.code === 63 || (err.code === 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER' && err.message.startsWith('stdout')))) {
+      return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
+    }
     // curl missing, proxy down, or cancelled — let the caller report the
     // original direct-connect error (the proxy attempt is best-effort)
     return null

package/spa.js

@@ -28,7 +28,7 @@ export function looksLikeSpa(html) {
 // stabilizes while the challenge sits in "Verification successful. Waiting
 // for ... to respond" — the poll loop returned that transitional page and
 // the caller mistook it for an improvement over static extraction.
-const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s*[|·]?\s*cloudflare|enable javascript and cookies to continue/i
+const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s{0,1000}(?:[|·]\s{0,1000})?cloudflare|enable javascript and cookies to continue/i
 
 export function looksLikeChallenge(html) {
   return typeof html === 'string' && CHALLENGE_RE.test(html)