dsh-read-url 1.9.1 → 1.9.3

package/CHANGELOG.md

@@ -1,5 +1,45 @@
 # Changelog
 
+## [1.9.3] - 2026-10-19
+
+### 修复
+
+- Markdown 按配对块处理嵌套容器与列表，保留段落间距、父级编号和子列表缩进；代码围栏/引用与后文正确分隔，属性与递归深度继续有界。
+- 数值配置经范围限制后向下取整，避免小数超时触发 `AbortSignal.timeout` 错误或小数分页上限多抓一页。
+- 代理回退遵守 `no_proxy`/`NO_PROXY` 的域名、IP、端口和 CIDR 边界，避免绕过代理的地址被直连与 curl 重复请求；curl 禁用 URL glob 展开，原样处理方括号查询和花括号路径。
+- curl 启用受支持的 HTTP 响应解压，避免 gzip 字节进入正文；单独检查解压后字节限额，兼容旧版 curl 的传输大小语义，stdout 超限返回明确错误。
+- 失败诊断不再输出代理配置地址，避免用户名、密码进入工具结果或模型上下文。
+- 提供方明确标记为纯文本时，链接和站点读取不再解析正文中的 HTML 示例或跟随示例跳转；保留原文、输出预算与提供方截断提示。
+- 质询页识别的空白段使用明确长度上限，消除相邻无界空白造成的平方回溯。
+- 双语说明更新 DeepSeek Fetch API 能力边界，并将早期官方工具对比明确标为历史快照。
+
+### 验证
+
+- 413 个零依赖单元断言、15 个真实 Chromium SPA 断言及 70 项对抗探针通过；新增 38 条单元检查、6 条探针，覆盖列表内代码围栏闭合及引用块边界。原始缺陷对照 v1.9.1/v1.9.2 复现。
+- 最终源码在本机桌面 0.2.0-rc.2（Electron 44.0.0 / Node 24.18.1）通过 17 项集成与 15 项 Chromium SPA 检查；官方 DSH 0.2.1-rc.2 通过 17 项服务集成及 9 项实际 DeepSeek Fetch 适配器检查。适配器传输使用受控响应，未覆盖外部 API 实际服务或完整 UI/模型会话。
+- 本机桌面升级为 0.2.1-rc.2（Electron 44.7.0 / Node 24.21.0）后，相同冻结源码在实际内置运行时另通过 17 项集成、3 项工具作用域与 15 项 Chromium SPA 检查。
+- 已审计八个接触包、Cordis/Loader 与四个桌面加载文件；社区升级卡截至 0.2.0-rc.2，新版由官方源码交叉核验。发布版 patch 另通过 4 项加载器及 2 项临时 profile 检查。
+- 2026-10-19 发布前 152 站：105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW，OK 站零噪声。相同 HTML/DOM 对照 v1.9.2 的 text 全文/续读一致，Markdown 结构修复无可见文字丢失或重复；波动站为验证页或旧版同现的响应边界。22 个错误分别为 13 个 HTTP 错误、3 个预期类型拒绝、6 个网络失败。
+- Bilibili 视频页在修复 gzip 解压后由乱码变为正常正文；实际 curl 8.19/8.21 共 12 个正常/超限/重定向/取消场景通过，包含旧版 curl 解压后越界保护。
+- 核心零运行时依赖；四工具 description/schema 保持 1,159/1,876 字符，属性长度与递归上限保留。
+
+## [1.9.2] - 2026-10-03
+
+### 修复
+
+- Windows 代理回退优先使用系统目录中的 PowerShell/curl，避免桌面进程的精简 PATH 导致代理发现或请求失败；系统工具不可用时保留原有回退与取消行为。
+- JSON-LD 按真实 `type` 属性读取，支持等号空格、无引号值及引号内的 `>`；忽略 `data-type`、注释、普通脚本及 raw-text 元素中的伪 JSON-LD。
+- JSON-LD 作者解析限制递归深度，正文数组仅拼接字符串项，避免深层数组触发栈溢出或非文本值污染正文；保留标签长度、JSON 大小及递归上限。
+- 双语安装说明补充桌面版插件入口、独立 profile、内置运行时及本地链接代码重启加载方式。
+
+### 验证
+
+- 375 个零依赖单元断言、15 个真实 Chromium SPA 断言、64 项对抗探针通过；新增 23 条单元检查和 5 条 JSON-LD 探针。JSON-LD 原始缺陷在 v1.9.0/v1.9.1 同输入均复现；Windows 空 PATH 的真实代理请求对照 v1.9.1 确认修复。
+- DSH 桌面版 0.2.0-rc.2 的实际 Electron 44.0.0 / Node 24.18.1 通过 17 项官方服务集成、3 项工具作用域、2 项本地代理 A/B 及 15 项 Chromium SPA 检查。极简 preset 继承全局插件工具；服务响应使用受控数据，未覆盖完整模型会话。
+- DSH 0.1.7-rc.2 → 0.2.0-rc.2 的 8 个接触包与 4 个桌面加载链文件完成字节差异审计，并交叉核对社区 0.2.0-rc.1 升级卡；已核查接口无需适配修改。0.2.1-alpha.1 未审计。
+- 2026-10-03 的 152 站回归：104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW，OK 站零噪声。已知 NOISE 与状态波动站使用相同 HTML/DOM 和可选依赖对照 v1.9.1，全文续读与 800 字输出一致；新增订阅站 HTTP 403 在新旧版本均复现。
+- 核心仍为零运行时依赖；四工具 description/schema 为 1,159/1,876 字符。
+
 ## [1.9.1] - 2026-09-28
 
 ### 修复

package/README.md

@@ -22,10 +22,12 @@ Continue at `charsStart + charsReturned`; the returned `charsStart` may exceed t
 
 ## Why
 
-DSH agents can search (getting links and snippets) but lack the step of "reading a URL into clean body text". The official `tool-web` `web_fetch` does a **whole-page turndown conversion** (nav/ads/sidebars all preserved) with a default cap of 200,000 characters — a token black hole. This plugin returns only what the model actually needs: **cleaned body + essential metadata**, truncated by default.
+This plugin keeps URL-reading output compact: **cleaned body + essential metadata** from HTML, paragraph-aligned truncation, continuation and a session cache. It also preserves plain text or Markdown returned by the configured DSH provider. DSH 0.2.1-rc.2 includes a DeepSeek Fetch API provider, so available source content and metadata depend on the selected provider.
 
 ### Competitor comparison (measured from source/docs, 2026-08-15)
 
+This is a historical snapshot; it does not describe the newer DeepSeek Fetch API provider.
+
 | Capability | Official `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
 |---|---|---|---|---|
 | Body cleaning (container-level) | ❌ whole page | ⚠️ tag-level, nav/footer leak in | ⚠️ custom, noisy | ✅ article/main containers + noise stripping |
@@ -55,8 +57,16 @@ Implemented per official docs (`docs/capability-seams.md`, `docs/cordis-primer.m
 
 ## Install
 
+### DSH Desktop
+
+Open **Plugins → Add plugin**, enter `dsh-read-url` (npm package), the GitHub repository URL, or the local project directory, and install it. In the plugin details, check the displayed version and that the component is **running**.
+
+Desktop uses its own `desktop` profile and bundled DSH runtime. CLI profile installations and global CLI upgrades are separate from Desktop. When configuring this plugin manually, use the Desktop profile's `cordis.patch.yml`. A local directory installation links to that source directory; restart Desktop after changing its code to load a fresh module.
+
+### CLI / local development
+
 ```bash
-# From GitHub (recommended, easy updates)
+# From GitHub in the web profile
 npx @deepseek-ai/dsh plugin --profile web add github:2672243194/dsh-read-url
 
 # Local development
@@ -323,6 +333,23 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - [DSH 0.1.7-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.7-rc.2) compatibility was checked against official npm package byte differences and community upgrade notes (available through rc.1). Both published v1.9.0 and this maintenance code passed **17 actual official-service integration checks each**; the published patch passed **4 loader checks** covering loading, execution, configuration remount and removal. Web-provider/runtime contracts remain compatible; no adaptation change is needed for the checked interfaces. These checks used controlled provider responses, not a complete host UI/model session.
 - Core runtime dependencies remain zero. Model input stays at **1,159 description characters / 1,876 schema characters**; site body summaries remain opt-in and respect `maxCharsPerPage` without an extra 80-character render cutoff.
 
+### v1.9.2 Desktop maintenance validation (2026-10-03)
+
+- **375 zero-dependency unit assertions, 15 real Chromium SPA assertions and 64 adversarial probes** passed. Coverage includes Windows proxy fallback with a minimal PATH, real JSON-LD attributes, comment/raw-text isolation, bounded scans and deeply nested data.
+- The installed **DSH Desktop 0.2.0-rc.2** runtime (Electron 44.0.0 / Node 24.18.1) passed **17 official-service integration checks, 3 tool-scope checks, 2 local proxy A/B checks and 15 Chromium SPA checks**. The minimal preset inherits the four globally registered plugin tools. Service responses were controlled fixtures; these checks do not cover a full model conversation.
+- [DSH 0.2.0-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.0-rc.2) was audited against 0.1.7-rc.2 using byte differences across eight contact packages and four Desktop loading files, with the [community 0.2.0-rc.1 upgrade card](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.1.md) as a cross-check. The verified interfaces require no adaptation. The newer 0.2.1-alpha.1 was not audited.
+- The 152-site sweep returned **104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW**, with no noise in OK results. Known NOISE and fluctuating sites matched v1.9.1 on identical HTML/DOM and optional dependencies, including full continuation and 800-character output. The newly failing feed returned HTTP 403 in both versions.
+- JSON-LD metadata/body extraction accepts valid attribute variants and rejects lookalike attributes. Author recursion is bounded; body arrays retain only strings. Core runtime dependencies remain zero, with fixed model input of **1,159 description characters / 1,876 schema characters**.
+
+### v1.9.3 Desktop maintenance validation (2026-10-19)
+
+- **413 zero-dependency unit assertions, 15 real Chromium SPA assertions and 70 adversarial probes** passed. Fixes cover nested Markdown containers/lists, fractional numeric configuration, literal text-provider content, proxy diagnostics, `NO_PROXY`, curl URL globbing/decompression and bounded challenge detection. Reproductions against v1.9.1/v1.9.2 confirm existing defects; regression tests also cover code-fence and quote boundaries within lists.
+- [DSH 0.2.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.1-rc.2), published October 17, was audited against 0.2.0-rc.2 across eight contact packages, Cordis/Loader and four Desktop loading files. The final plugin passed **17 official-service integration checks and 9 DeepSeek Fetch provider checks**. The published v1.9.2 patch also passed 4 loader and 2 temporary-profile checks. The [community upgrade card](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.2.md) currently stops at 0.2.0-rc.2; newer behavior was checked directly against official packages.
+- After upgrading Desktop to **0.2.1-rc.2 / Electron 44.7.0 / Node 24.21.0**, its actual bundled runtime passed **17 integration checks, 3 tool-scope checks and 15 Chromium SPA checks** against the same final plugin. The prior 0.2.0-rc.2 runtime also passed 17 integration and 15 SPA checks. Service responses were controlled; these checks do not cover a full model conversation or UI interaction.
+- The release 152-site sweep returned **105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW**, with no noise in OK results. Errors comprised 13 HTTP refusals/missing pages, 3 expected content-type rejections and 6 network failures. Saved HTML/DOM replays matched v1.9.2 in full text and continuation; Markdown restored structure without losing or duplicating visible text. Dynamic verification pages matched on identical input. The previously garbled Bilibili video page returned readable content after gzip decompression. Real curl 8.19/8.21 passed 12 decompression, size-limit, redirect and cancellation scenarios.
+- The DeepSeek Fetch provider returns Markdown as `body.kind=text`. Single-page reading preserves that text; links/site operations report that HTML navigation was not scanned, rather than interpreting HTML examples as links, redirects or SPA scripts. The provider does not expose the target's final URL or status, so HTML metadata, automatic pagination and site discovery are not equivalent to raw HTML retrieval.
+- Nested Markdown blocks retain paragraph boundaries, list nesting, parent numbering and separate code/quote endings from following text. Attribute scans and the 100-level recursion limit remain bounded. Proxy retrieval decompresses supported HTTP encodings and checks the decoded byte budget even with older curl versions. Core runtime dependencies remain zero; fixed model input remains **1,159 description characters / 1,876 schema characters**.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)

package/README.zh.md

@@ -22,10 +22,12 @@ DeepSeek Harness 的 URL 阅读插件：抓取任意 URL——**网页（HTML）
 
 ## 为什么做它
 
-DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净正文"这一步。官方 `tool-web` 的 `web_fetch` 是**整页 turndown 转换**（导航/广告/侧栏全保留），默认上限 20 万字符——token 黑洞。本插件只返回模型真正需要的：**净化后的正文 + 必要元数据**，并默认截断。
+本插件控制 URL 阅读的输出量：从 HTML 提取**净化后的正文 + 必要元数据**，提供段落截断、续读和会话缓存；也保留 DSH 配置的提供方返回的纯文本或 Markdown。DSH 0.2.1-rc.2 已包含 DeepSeek Fetch API 提供方，可获得的原文和元数据取决于所选提供方。
 
 ### 同类插件对比（2026-08-15 实测源码/文档）
 
+以下是历史快照，不代表新版 DeepSeek Fetch API 提供方的行为。
+
 | 能力 | 官方 `tool-web` web_fetch | dsh-webfetch | dsh-scrape-webpage | **dsh-read-url** |
 |---|---|---|---|---|
 | 正文净化（容器级提取） | ❌ 整页渲染 | ⚠️ 标签级去噪，nav/footer 仍混入 | ⚠️ 自研，含噪音 | ✅ article/main 容器 + 噪音剥离 |
@@ -55,8 +57,16 @@ DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净
 
 ## 安装
 
+### DSH 桌面版
+
+打开 **插件 → 添加插件**，输入 `dsh-read-url`（npm 包名）、GitHub 仓库地址或本地项目目录后安装。在插件详情检查版本，以及组件是否显示 **运行中**。
+
+桌面版使用独立的 `desktop` profile 和内置 DSH 运行时。CLI 各 profile 的安装及全局 CLI 升级与桌面版分别管理。手动配置本插件时，应使用桌面 profile 的 `cordis.patch.yml`。本地目录安装会链接源目录，修改代码后应重启桌面版，以加载新的模块。
+
+### CLI / 本地开发
+
 ```bash
-# 从 GitHub（推荐，便于更新）
+# 从 GitHub 安装到 web profile
 npx @deepseek-ai/dsh plugin --profile web add github:2672243194/dsh-read-url
 
 # 本地开发
@@ -323,6 +333,23 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 - [DSH 0.1.7-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.7-rc.2) 已完成官方 npm 接触包字节差异核查，并与社区升级卡交叉验证（卡片目前到 rc.1）。发布版 v1.9.0 与当前维护代码各通过 **17 项真实官方服务集成检查**；发布版 patch 另通过 **4 项加载器检查**，覆盖加载、调用、配置重挂和卸载。Web provider/runtime 契约兼容，已核查接口无需适配修改；测试使用受控 provider 响应，未覆盖完整宿主 UI/模型会话。
 - 核心仍为零运行时依赖，模型固定输入仍为 **description 1,159 字符 / schema 1,876 字符**；站点正文摘要仍默认关闭，启用后遵循 `maxCharsPerPage`，不再额外截成 80 字符。
 
+### v1.9.2 桌面版维护验证（2026-10-03）
+
+- **375 个零依赖单元断言、15 个真实 Chromium SPA 断言、64 项对抗探针**通过。覆盖 Windows 精简 PATH 下的代理回退、JSON-LD 真实属性、注释/raw-text 隔离、有界扫描与深层数据。
+- 已安装的 **DSH 桌面版 0.2.0-rc.2** 运行时（Electron 44.0.0 / Node 24.18.1）通过 **17 项官方服务集成、3 项工具作用域、2 项本地代理 A/B 及 15 项 Chromium SPA 检查**。极简 preset 继承全局注册的四个插件工具；服务响应使用受控数据，未覆盖完整模型会话。
+- [DSH 0.2.0-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.0-rc.2) 已对照 0.1.7-rc.2 完成八个接触包与四个桌面加载链文件的字节差异审计，并交叉核对[社区 0.2.0-rc.1 升级卡](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.1.md)。已核查接口无需适配修改；更新的 0.2.1-alpha.1 未审计。
+- 152 站结果为 **104 OK / 20 THIN / 4 EMPTY / 24 ERR / 0 THREW**，OK 站零噪声。已知 NOISE 与状态波动站使用相同 HTML/DOM 和可选依赖对照 v1.9.1，全文续读及 800 字输出一致；新增失败的订阅站在新旧版本均返回 HTTP 403。
+- JSON-LD 元数据/正文支持合法属性变体并排除伪属性；作者递归有界，正文数组仅保留字符串。核心仍为零运行时依赖，模型固定输入为 **description 1,159 字符 / schema 1,876 字符**。
+
+### v1.9.3 桌面版维护验证（2026-10-19）
+
+- **413 个零依赖单元断言、15 个真实 Chromium SPA 断言、70 项对抗探针**通过。修复覆盖 Markdown 嵌套容器/列表、小数配置、提供方纯文本、代理诊断、`NO_PROXY`、curl URL 展开/解压及有界质询页识别；v1.9.1/v1.9.2 对照确认原始缺陷既有，回归测试另覆盖列表内代码围栏闭合及引用块边界。
+- 10 月 17 日发布的 [DSH 0.2.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.2.1-rc.2) 已对照 0.2.0-rc.2 审计八个接触包、Cordis/Loader 及四个桌面加载链文件。最终插件通过 **17 项官方服务集成和 9 项 DeepSeek Fetch 提供方检查**；已发布 v1.9.2 的 patch 另通过 4 项加载器和 2 项临时 profile 检查。[社区升级卡](https://github.com/oh-my-dsh/dsh-plugin-upgrade-skill/blob/main/skills/plugin-upgrade/references/v0.2.0-rc.2.md)目前仅到 0.2.0-rc.2，新版行为直接用官方包核验。
+- 桌面升级至 **0.2.1-rc.2 / Electron 44.7.0 / Node 24.21.0** 后，实际内置运行时对相同最终插件通过 **17 项集成、3 项工具作用域和 15 项 Chromium SPA 检查**；升级前的 0.2.0-rc.2 运行时也已通过 17 项集成与 15 项 SPA。服务响应使用受控数据，未覆盖完整模型会话或 UI 交互。
+- 发布前 152 站为 **105 OK / 21 THIN / 4 EMPTY / 22 ERR / 0 THREW**，OK 站零噪声；错误包括 13 个 HTTP 拒绝/页面不存在、3 个预期类型拒绝及 6 个网络失败。保存的同一 HTML/DOM 对照 v1.9.2，text 全文与续读一致，Markdown 恢复结构且无可见文字丢失或重复；动态验证页同输入结果一致。此前乱码的 Bilibili 视频页经 gzip 解压后返回正常正文。实际 curl 8.19/8.21 共通过 12 个解压、限额、跳转及取消场景。
+- DeepSeek Fetch 提供方以 `body.kind=text` 返回 Markdown。单页读取保留原文；链接/站点工具明确提示未扫描 HTML 导航，避免把 HTML 示例当作链接、跳转或 SPA 脚本。该提供方不暴露目标最终 URL 和状态，因此 HTML 元数据、自动分页和站点发现不能视为与原始 HTML 抓取等价。
+- Markdown 嵌套块保留段落边界、列表层级、父级编号，代码块/引用与后文正确分隔；属性扫描和 100 层递归上限继续有界。代理读取解压受支持的 HTTP 编码，并对旧版 curl 同样执行解压后字节限额。核心仍为零运行时依赖，模型固定输入仍为 **description 1,159 字符 / schema 1,876 字符**。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）

package/index.js

@@ -100,6 +100,10 @@ function allowedDirectUrl(url, origins) {
 const DIRECT_BOUNDARY_ERROR = 'Direct fetch stopped: target is outside directFetchOrigins or contains credentials'
 const DIRECT_SPA_HINT = '已配置直连仅支持静态读取，未启用浏览器渲染'
 
+function providerTextHint(page) {
+  return '提供方返回纯文本，未扫描 HTML 链接或跳转' + (page.sourceTruncated ? '；提供方已截断原文，当前内容不是完整页面' : '')
+}
+
 function getDecoder(enc) {
   let d = decoders.get(enc)
   if (!d) {
@@ -328,7 +332,7 @@ async function raceFetch(url, externalSignal, cfg, proxy) {
 
 async function fetchPage(url, externalSignal, cfg) {
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
-  const proxy = await detectProxy()
+  const proxy = await detectProxy(url)
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   if (proxy) {
     const winner = await raceFetch(url, externalSignal, cfg, proxy)
@@ -336,7 +340,7 @@ async function fetchPage(url, externalSignal, cfg) {
     const dErr = winner.failures.find((f) => f.label === 'direct')
     const pErr = winner.failures.find((f) => f.label === 'proxy')
     const px = pErr ? `，代理返回 ${pErr.error}` : '，代理未连通'
-    return { error: `Fetch failed: ${dErr ? dErr.error : 'unknown error'}（直连失败；已尝试代理 ${proxy}${px}）` }
+    return { error: `Fetch failed: ${dErr ? dErr.error : 'unknown error'}（直连失败；已尝试代理${px}）` }
   }
   // No proxy configured: plain direct connect (the original behaviour).
   const timeoutSignal = AbortSignal.timeout(cfg.timeoutMs)
@@ -955,18 +959,78 @@ export function inlineMd(html, depth = 0) {
   return img.restore(out.trim())
 }
 
-export function blockMd(html, depth = 0) {
+// Pair structural blocks in one forward pass so nested siblings keep their
+// own boundaries. Raw/code bodies and quoted attribute values are opaque.
+function markdownBlockPairs(html) {
+  const pairs = new Map()
+  const stacks = new Map()
+  const structural = new Set(['div', 'section', 'article', 'main', 'blockquote', 'ul', 'ol', 'li'])
+  const opaque = new Set(['pre', 'code', 'script', 'style', 'textarea', 'template', 'noscript'])
+  const re = /<!--|<(\/?)([a-z][a-z0-9:-]{0,63})(?=[\s/>])/gi
+  let m
+  while ((m = re.exec(html))) {
+    if (!m[2]) {
+      const end = html.indexOf('-->', re.lastIndex)
+      if (end < 0) break
+      re.lastIndex = end + 3
+      continue
+    }
+    let quote = '', end = -1
+    const limit = Math.min(html.length, m.index + 1002)
+    for (let i = re.lastIndex; i < limit; i++) {
+      const ch = html[i]
+      if (quote) {
+        if (ch === quote) quote = ''
+      } else if (ch === '"' || ch === "'") quote = ch
+      else if (ch === '<') break
+      else if (ch === '>') { end = i + 1; break }
+    }
+    if (end < 0) continue
+    const attrs = html.slice(re.lastIndex, end - 1)
+    re.lastIndex = end
+    const tag = m[2].toLowerCase()
+    if (!m[1] && opaque.has(tag)) {
+      const close = new RegExp(`</${tag}\\s{0,1000}>`, 'gi')
+      close.lastIndex = re.lastIndex
+      if (!close.exec(html)) break
+      re.lastIndex = close.lastIndex
+      continue
+    }
+    if (!structural.has(tag)) continue
+    if (!m[1]) {
+      if (!stacks.has(tag)) stacks.set(tag, [])
+      stacks.get(tag).push({ tag, attrs, index: m.index, start: re.lastIndex })
+    } else if (!attrs.trim()) {
+      const opening = stacks.get(tag)?.pop()
+      if (opening && m.index - opening.start <= 3145728) pairs.set(opening.index, { ...opening, close: m.index, end: re.lastIndex })
+    }
+  }
+  return pairs
+}
+
+export function blockMd(html, depth = 0, escapeText = false) {
   if (depth > MD_MAX_DEPTH) {
-    return decodeTextEntities(html.replace(/<[^>]{1,1000}>/g, ' ')).replace(/\s+/g, ' ').trim()
+    const text = decodeTextEntities(html.replace(/<[^>]{1,1000}>/g, ' ')).replace(/\s+/g, ' ').trim()
+    return escapeText ? escInline(text) : text
   }
   const img = imgsToMarkdown(html)
   html = stripUnmatchedOpeners(img.html)
+  const blocks = markdownBlockPairs(html)
   let out = ''
   const re = /<(?:br|hr)\b[^>]{0,1000}>|<([a-zA-Z0-9]{1,100})((?:"[^"]{0,1000}"|'[^']{0,1000}'|[^'">]){0,1000})>([\s\S]{0,3145728}?)<\/\1>|<[^>]{1,1000}>|([^<]{1,3145728})/gi
   let m
   while ((m = re.exec(html))) {
+    const block = blocks.get(m.index)
+    if (block) {
+      m[0] = html.slice(m.index, block.end)
+      m[1] = block.tag
+      m[2] = block.attrs
+      m[3] = html.slice(block.start, block.close)
+      re.lastIndex = block.end
+    }
     if (m[4] !== undefined) {
-      out += decodeTextEntities(m[4])
+      const text = decodeTextEntities(m[4])
+      out += escapeText ? escInline(text) : text
       continue
     }
     if (!m[1]) {
@@ -988,11 +1052,11 @@ export function blockMd(html, depth = 0) {
       const t = inlineMd(inner, depth + 1)
       if (t) out += `\n\n${t}`
     } else if (tag === 'div' || tag === 'section' || tag === 'article' || tag === 'main') {
-      const t = blockMd(inner, depth + 1)
-      if (t) out += `\n\n${t}`
+      const t = blockMd(inner, depth + 1, escapeText)
+      if (t) out += `\n\n${t}\n\n`
     } else if (tag === 'blockquote') {
-      const t = blockMd(inner, depth + 1).trim()
-      out += `\n\n> ${t.replace(/\n/g, '\n> ')}`
+      const t = blockMd(inner, depth + 1, escapeText).trim()
+      out += `\n\n> ${t.replace(/\n/g, '\n> ')}\n\n`
     } else if (tag === 'pre') {
       // Strip wrapper tags FIRST, then decode entities — a literal &lt;div&gt;
       // in the code must become <div> in the fenced block, but a real nested
@@ -1003,22 +1067,31 @@ export function blockMd(html, depth = 0) {
       const lang =
         /<code\b[^>]{0,1000}class=["'][^"']{0,1000}(?:language|lang)-([\w#+.-]{1,80})(?:[\s"'])/i.exec(inner)?.[1] || ''
       const fence = backtickFence(code, 3)
-      out += `\n\n${fence}${lang}\n${code}\n${fence}`
+      out += `\n\n${fence}${lang}\n${code}\n${fence}\n\n`
     } else if (tag === 'code') {
       out += img.protect(inlineCode(inner))
     } else if (tag === 'ul' || tag === 'ol') {
-      // Without a single </li> nothing can pair, so the item scan is skipped.
       const items = []
-      if (/<\/li\b/i.test(inner)) {
-        const liRe = /<li\b[^>]{0,1000}>([\s\S]{0,3145728}?)<\/li>/gi
-        let lm, idx = 1
-        while ((lm = liRe.exec(inner))) {
-          const t = inlineMd(lm[1], depth + 1)
-          items.push(tag === 'ol' ? `${idx}. ${t}` : `- ${t}`)
-          idx++
-        }
+      const itemBlocks = block ? blocks : markdownBlockPairs(inner)
+      const itemOffset = block ? block.start : 0
+      const liRe = /<li(?=[\s>])/gi
+      let lm, idx = 1
+      while ((lm = liRe.exec(inner))) {
+        const item = itemBlocks.get(itemOffset + lm.index)
+        if (!item) continue
+        liRe.lastIndex = item.end - itemOffset
+        const body = inner.slice(item.start - itemOffset, item.close - itemOffset)
+        const t = /<(?:ul|ol|div|section|article|main|p|pre|blockquote|h[1-6]|table)(?=[\s>])/i.test(body)
+          ? blockMd(body, depth + 1, true).replace(/\n{3,}/g, '\n\n').trim()
+          : inlineMd(body, depth + 1)
+        const prefix = tag === 'ol' ? `${idx++}. ` : '- '
+        const indent = ' '.repeat(prefix.length)
+        const lines = t.split('\n')
+        items.push(/^<(?:ul|ol)(?=[\s>])/i.test(body.trimStart())
+          ? prefix.trimEnd() + '\n' + lines.map(line => line ? indent + line : line).join('\n')
+          : prefix + lines.map((line, i) => i && line ? indent + line : line).join('\n'))
       }
-      if (items.length) out += `\n\n${items.join('\n')}`
+      if (items.length) out += `\n\n${items.join('\n')}\n\n`
     } else if (tag === 'li') {
       out += `\n- ${inlineMd(inner, depth + 1)}`
     } else if (tag === 'br') {
@@ -1061,7 +1134,7 @@ export function blockMd(html, depth = 0) {
       const t = inlineMd(m[0], depth + 1)
       if (t) out += t
     } else {
-      const t = blockMd(inner, depth + 1)
+      const t = blockMd(inner, depth + 1, escapeText)
       if (t) out += t
     }
   }
@@ -1116,7 +1189,39 @@ function extractMeta(html) {
 // Bounded block size keeps a hostile giant script tag from dragging the scan.
 // 100k covers real article-list JSON-LD (news sites often embed dozens of
 // items); metadata sits in the first node anyway.
-const LDJSON_RE = /<script[^>]{0,1000}type=["']application\/ld\+json["'][^>]{0,1000}>([\s\S]{0,100000}?)<\/script\s*>/gi
+function* jsonLdBlocks(html) {
+  const open = /<!--|<([a-z][a-z0-9:-]{0,63})(?=[\s>])/gi
+  const rawNames = new Set(['script', 'style', 'textarea', 'title', 'xmp', 'iframe', 'noembed', 'noframes', 'noscript', 'plaintext'])
+  let match
+  while ((match = open.exec(html))) {
+    if (!match[1]) {
+      const end = html.indexOf('-->', open.lastIndex)
+      if (end < 0) break
+      open.lastIndex = end + 3
+      continue
+    }
+    const tag = navigationTags(html.slice(match.index, match.index + 1002), /^<[a-z][a-z0-9:-]{0,63}(?=[\s>])/gi).next().value
+    if (!tag) continue
+    const end = match.index + tag.end
+    open.lastIndex = end
+    const name = match[1].toLowerCase()
+    if (!rawNames.has(name)) continue
+    if (name === 'plaintext') break
+    const close = new RegExp(`</${name}\\s{0,1000}>`, 'gi')
+    close.lastIndex = end
+    const closing = close.exec(html)
+    if (!closing) break
+    // Raw-text bodies and quoted attributes cannot introduce script elements.
+    open.lastIndex = close.lastIndex
+    if (name !== 'script' || tag.attrs.type?.trim().toLowerCase() !== 'application/ld+json' || closing.index - end > 100000) continue
+    try {
+      yield JSON.parse(html.slice(end, closing.index))
+    } catch {
+      // Invalid data blocks do not prevent reading later JSON-LD blocks.
+    }
+  }
+}
+
 function jsonLdMeta(html) {
   // Recursively collect every node: nested structures are common (ItemList →
   // mainEntity → Article, WebPage → mainEntityOfPage), so a flat top-level
@@ -1136,20 +1241,15 @@ function jsonLdMeta(html) {
       if (v && typeof v === 'object') collect(v, out, depth + 1)
     }
   }
-  const authorOf = (a) => {
+  const authorOf = (a, depth = 0) => {
+    if (depth > 20) return ''
     if (typeof a === 'string') return a
-    if (Array.isArray(a)) return authorOf(a[0])
-    if (a && typeof a === 'object') return typeof a.name === 'string' ? a.name : authorOf(a.name)
+    if (Array.isArray(a)) return authorOf(a[0], depth + 1)
+    if (a && typeof a === 'object') return typeof a.name === 'string' ? a.name : authorOf(a.name, depth + 1)
     return ''
   }
   const clean = (s) => (s ? decodeTextEntities(s).slice(0, 40) : '')
-  for (const m of html.matchAll(LDJSON_RE)) {
-    let json
-    try {
-      json = JSON.parse(m[1])
-    } catch {
-      continue
-    }
+  for (const json of jsonLdBlocks(html)) {
     const nodes = []
     collect(json, nodes)
     for (const n of nodes) {
@@ -1165,15 +1265,9 @@ function jsonLdMeta(html) {
 // (articleBody) while the visible HTML is an anti-bot shell or a thin
 // fragment — a fallback body source the DOM extractors cannot reach. Returns
 // the first string articleBody found (array values joined), stripped to
-// paragraphs via textLines. Block scan is the same bounded LDJSON_RE.
+// paragraphs via textLines. Metadata and body share the bounded block scan.
 function jsonLdArticleBody(html) {
-  for (const m of html.matchAll(LDJSON_RE)) {
-    let json
-    try {
-      json = JSON.parse(m[1])
-    } catch {
-      continue
-    }
+  for (const json of jsonLdBlocks(html)) {
     const nodes = []
     const collect = (node, depth = 0) => {
       if (!node || typeof node !== 'object' || depth > 20) return
@@ -1189,7 +1283,7 @@ function jsonLdArticleBody(html) {
     }
     collect(json)
     for (const n of nodes) {
-      const raw = Array.isArray(n.articleBody) ? n.articleBody.join('\n\n') : n.articleBody
+      const raw = Array.isArray(n.articleBody) ? n.articleBody.filter(part => typeof part === 'string').join('\n\n') : n.articleBody
       if (typeof raw === 'string' && raw.trim()) return textLines(raw).slice(0, 200000)
     }
   }
@@ -2176,6 +2270,7 @@ function readLinksTool(ctx, cfg) {
       const page = await fetchResource(ctx, url, signal, cfg, route)
       if (signal && signal.aborted) return { error: 'cancelled' }
       if (page.error) return { error: page.error }
+      if (page.kind === 'text') return { url: page.finalUrl || url, count: 0, links: [], spaHint: providerTextHint(page) }
       let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
       let finalUrl = page.finalUrl || url
       // Same meta-refresh shell following as read_url: a link hop serves a
@@ -2401,6 +2496,14 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         if (depth > 0 && !sameHost(pageUrl, host)) {
           return { url, depth, error: `Redirected outside the site: ${pageUrl}` }
         }
+        if (page.kind === 'text') {
+          if (depth === 0) host = hostOf(pageUrl)
+          return {
+            url: pageUrl, depth, title: '', chars: html.length,
+            text: includeContent ? smartTruncate(html, perMax).text : undefined,
+            links: [], route, spaHint: providerTextHint(page),
+          }
+        }
         // Same shell-following as read_url: a meta-refresh stub would otherwise
         // be crawled AS the page (title-less hop text pollutes results and the
         // real content is never reached). Fails open — fetch problems keep the
@@ -2553,7 +2656,7 @@ export function apply(ctx, config) {
   const cfg = { ...DEFAULTS, ...(config || {}) }
   const clamp = (v, d, min, max) => {
     const n = Number(v)
-    return Number.isFinite(n) ? Math.min(max, Math.max(min, n)) : d
+    return Number.isFinite(n) ? Math.floor(Math.min(max, Math.max(min, n))) : d
   }
   cfg.timeoutMs = clamp(cfg.timeoutMs, DEFAULTS.timeoutMs, 500, 120000)
   cfg.maxBytes = clamp(cfg.maxBytes, DEFAULTS.maxBytes, 65536, 16 * 1024 * 1024)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.9.1",
+  "version": "1.9.3",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",

package/proxy-fallback.js

@@ -1,19 +1,32 @@
-// proxy-fallback.js — retry a failed direct fetch through the user's proxy
+// proxy-fallback.js — optional proxy channel for direct webpage retrieval
 //
-// Design: direct connect first, proxy only as fallback. When the direct
-// fetch fails with a CONNECTION-class error (blocked network, refused,
-// DNS miss, connect timeout), we look up the user's own proxy — first the
-// standard HTTP_PROXY/HTTPS_PROXY env vars, then the Windows system proxy
-// (registry) where proxy apps like Clash actually persist it — and retry
-// once through it via the system `curl` (ships with Windows 10+/macOS/Linux;
-// zero npm deps). The proxy URL is passed explicitly with `-x`, so we don't
-// depend on curl's env-var reading.
+// When the host web provider is absent, configured proxy retrieval races
+// native fetch and the first success wins. Proxy discovery uses standard
+// HTTP_PROXY/HTTPS_PROXY env vars, then the Windows system proxy registry.
+// NO_PROXY exclusions retain only the native fetch. Proxy requests use system
+// curl (Windows 10+/macOS/Linux; zero npm deps) with an explicit `-x` URL.
 //
-// Transparency: a successful proxied read returns exactly like a direct one
-// (the caller never knows); a failed proxied attempt returns null and the
-// caller reports the ORIGINAL direct-connect error with a clear note, so the
-// model/user can act (turn the proxy on, or accept the network boundary).
+// Successful proxy responses use the direct channel's body shape. HTTP and
+// content errors remain diagnostic; unavailable curl or proxies return null.
 import { execFile, execFileSync } from 'node:child_process'
+import { existsSync } from 'node:fs'
+import { win32 } from 'node:path'
+import { BlockList, isIP } from 'node:net'
+
+// Desktop launches can inherit a reduced PATH. Prefer the Windows-shipped
+// utilities when present; other platforms and custom installations use PATH.
+function systemExecutable(name) {
+  if (process.platform !== 'win32') return name
+  const parts = name === 'powershell'
+    ? ['System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe']
+    : ['System32', 'curl.exe']
+  for (const root of [process.env.SystemRoot, process.env.WINDIR]) {
+    if (typeof root !== 'string' || !win32.isAbsolute(root)) continue
+    const executable = win32.join(root, ...parts)
+    if (existsSync(executable)) return executable
+  }
+  return name
+}
 
 // Some servers omit the Content-Type header entirely; the content-type gate
 // then has nothing to test and a binary body (PDF/PNG/zip) would flow into
@@ -59,8 +72,7 @@ const SYS_PROXY_RETRY_MS = 60 * 1000
 
 // Windows system proxy from the registry (HKCU Internet Settings). PowerShell
 // is used because `reg.exe` is frequently blocked by security policy; the
-// read is synchronous and cached after the first call (it only runs after a
-// direct-connect failure, so the ~1s cost is invisible on the happy path).
+// read is synchronous and cached to avoid repeated discovery during reads.
 // A FAILED read is cached as '' but re-tried after 60s — an environment that
 // transiently blocks the PowerShell child (EDR, policy) should not starve
 // the whole session of the proxy.
@@ -71,7 +83,7 @@ function readWindowsSystemProxy() {
   sysProxyCheckedAt = now
   try {
     const out = execFileSync(
-      'powershell',
+      systemExecutable('powershell'),
       [
         '-NoProfile', '-Command',
         "(Get-ItemProperty 'HKCU:\\Software\\Microsoft\\Windows\\CurrentVersion\\Internet Settings').ProxyEnable, (Get-ItemProperty 'HKCU:\\Software\\Microsoft\\Windows\\CurrentVersion\\Internet Settings').ProxyServer",
@@ -90,8 +102,72 @@ function readWindowsSystemProxy() {
   }
 }
 
+// Match curl's domain/IP exclusions before starting a redundant direct curl
+// request. Port-qualified hosts and wildcard suffixes are also accepted.
+function bypassesProxy(url) {
+  const exclusions = process.env.no_proxy || process.env.NO_PROXY
+  if (!url || !exclusions) return false
+  let target
+  try { target = new URL(url) } catch { return false }
+  const hostname = target.hostname.toLowerCase().replace(/^\[|\]$/g, '').replace(/\.$/, '')
+  const ipVersion = isIP(hostname)
+  const port = target.port || (target.protocol === 'https:' ? '443' : '80')
+  for (let entry of exclusions.split(',')) {
+    entry = entry.trim().toLowerCase()
+    if (!entry) continue
+    if (entry === '*') return true
+    const slash = entry.indexOf('/')
+    if (slash >= 0) {
+      const address = entry.slice(0, slash)
+      const prefix = entry.slice(slash + 1)
+      if (!ipVersion || isIP(address) !== ipVersion || !/^\d{1,3}$/.test(prefix)) continue
+      try {
+        const block = new BlockList()
+        const family = ipVersion === 6 ? 'ipv6' : 'ipv4'
+        block.addSubnet(address, Number(prefix), family)
+        if (block.check(hostname, family)) return true
+      } catch {}
+      continue
+    }
+    let entryPort = ''
+    if (entry.startsWith('[')) {
+      const end = entry.indexOf(']')
+      if (end < 0) continue
+      const tail = entry.slice(end + 1)
+      if (tail && !/^:\d{1,5}$/.test(tail)) continue
+      entryPort = tail.slice(1)
+      entry = entry.slice(1, end)
+    } else if (!isIP(entry) && entry.indexOf(':') >= 0) {
+      const colon = entry.lastIndexOf(':')
+      entryPort = entry.slice(colon + 1)
+      entry = entry.slice(0, colon)
+      if (entry.includes(':') || !/^\d{1,5}$/.test(entryPort)) continue
+    }
+    if (entryPort && Number(entryPort) !== Number(port)) continue
+    if (isIP(entry)) {
+      if (!ipVersion) continue
+      try {
+        const canonical = new URL(isIP(entry) === 6 ? `http://[${entry}]` : `http://${entry}`).hostname
+          .replace(/^\[|\]$/g, '')
+        if (canonical === hostname) return true
+      } catch {}
+      continue
+    }
+    if (ipVersion) continue
+    if (entry.startsWith('*.')) {
+      if (hostname.endsWith(entry.slice(1).replace(/\.$/, ''))) return true
+      continue
+    }
+    else if (entry.startsWith('.')) entry = entry.slice(1)
+    entry = entry.replace(/\.$/, '')
+    if (entry && (hostname === entry || hostname.endsWith(`.${entry}`))) return true
+  }
+  return false
+}
+
 // The user's proxy, in priority order: env vars, then Windows system proxy.
-export async function detectProxy() {
+export async function detectProxy(url) {
+  if (bypassesProxy(url)) return ''
   const env =
     process.env.HTTPS_PROXY ||
     process.env.https_proxy ||
@@ -108,11 +184,14 @@ export async function detectProxy() {
 // `proxyOverride` lets the caller pass an already-detected proxy (the race
 // path probes once up front) instead of re-detecting here.
 export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride) {
-  const proxy = proxyOverride || (await detectProxy())
+  if (bypassesProxy(url)) return null
+  const proxy = proxyOverride || (await detectProxy(url))
   if (!proxy) return null
   const maxSec = Math.max(10, Math.ceil(cfg.timeoutMs / 1000) + 5)
   const args = [
     '-sS',
+    '--globoff',
+    '--compressed', // request and decode supported HTTP content encodings
     '-L', // follow redirects, like the direct fetch path
     '--max-time', String(maxSec),
     '--max-filesize', String(cfg.maxBytes), // unit is BYTES, not KB
@@ -127,7 +206,7 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
   ]
   try {
     const buf = await new Promise((resolve, reject) => {
-      execFile('curl', args, { maxBuffer: cfg.maxBytes + 8192, encoding: 'buffer', windowsHide: true, signal: externalSignal || undefined }, (err, stdout) => {
+      execFile(systemExecutable('curl'), args, { maxBuffer: cfg.maxBytes + 8192, encoding: 'buffer', windowsHide: true, signal: externalSignal || undefined }, (err, stdout) => {
         if (err) return reject(err)
         resolve(stdout)
       })
@@ -137,6 +216,9 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
     if (nl < 0) return null // no trailing metadata — abnormal response, treat as failure
     const meta = buf.subarray(nl + 1).toString('latin1').trim()
     const body = buf.subarray(0, nl)
+    // Older curl versions bound the compressed transfer, not the decoded
+    // body. Do not let the metadata allowance enlarge the content budget.
+    if (body.length > cfg.maxBytes) return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
     const parts = meta.split(' ')
     const codeStr = parts.shift()
     const finalUrlEff = parts.length > 1 ? parts.pop() : url
@@ -157,7 +239,9 @@ export async function fetchViaCurlProxy(url, cfg, externalSignal, proxyOverride)
   } catch (err) {
     // curl error 63 = exceeded --max-filesize (page too large, not a proxy
     // problem — report it as such instead of "proxy unreachable")
-    if (err && err.code === 63) return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
+    if (err && (err.code === 63 || (err.code === 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER' && err.message.startsWith('stdout')))) {
+      return { error: `Page exceeds ${cfg.maxBytes} bytes (via proxy)` }
+    }
     // curl missing, proxy down, or cancelled — let the caller report the
     // original direct-connect error (the proxy attempt is best-effort)
     return null

package/spa.js

@@ -28,7 +28,7 @@ export function looksLikeSpa(html) {
 // stabilizes while the challenge sits in "Verification successful. Waiting
 // for ... to respond" — the poll loop returned that transitional page and
 // the caller mistook it for an improvement over static extraction.
-const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s*[|·]?\s*cloudflare|enable javascript and cookies to continue/i
+const CHALLENGE_RE = /just a moment|performing security verification|verifying (you are|your) human|cf-chl|cf_chl|cdn-cgi\/challenge-platform|attention required!\s{0,1000}(?:[|·]\s{0,1000})?cloudflare|enable javascript and cookies to continue/i
 
 export function looksLikeChallenge(html) {
   return typeof html === 'string' && CHALLENGE_RE.test(html)