dsh-read-url 1.8.0 → 1.9.0

package/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## [1.9.0] - 2026-09-19
+
+### 新增
+
+- 新增仅 profile 可配置的 `directFetchOrigins`，默认空列表。按协议、域名和端口精确匹配，命中后直接读取，显式跳过宿主 provider 及其 SSRF 检查；未命中的初始 URL 维持原有行为，不因任意 provider 错误自动直连。
+- 例外链的 HTTP 重定向、meta-refresh、分页及爬取子页面均受完整列表约束；HTTP 跳转逐跳校验，每次尝试最多 5 跳并检测循环。例外不进入 curl 代理竞速或 SPA 渲染，失败边界/静态读取提示保留到工具输出。
+- 成功和失败缓存按规范化后的完整配置隔离；非法配置明确报错，工具参数不允许扩大直连范围。双语 README 增补配置方法及信任边界。
+
+### 验证
+
+- 322 个零依赖单元断言、15 个真实 Chromium SPA 断言、53 项对抗探针通过；新增 22 条直连策略检查与 3 条 SPA 限制检查。
+- 受控 HTTP 服务与 provider 桩覆盖默认拒绝、显式直连、重定向、分页、四工具、缓存隔离、取消、超时及正文上限。
+- 152 站实网回归：105 OK / 23 THIN+EMPTY / 24 ERR / 0 THREW，OK 站零噪声。Vue、React、酷狗三个已知 NOISE 站同输入 v1.8.1 对照的 text/Markdown 全文与 800 字符窗口逐字一致；ERR 均为网络、反爬、无效地址或预期不支持的类型。
+- 四工具 description 共 1,159 字符，参数 schema 共 1,876 字符；零运行时依赖，新增正则有界。
+
+## [1.8.1] - 2026-09-19
+
+### 修复
+
+- SPA 导航取消及时关闭当前页面；启动期间取消不再创建页面，卸载等待浏览器关闭，并发清理共享等待，断开连接后可重新启动；旧启动失败不会清空新浏览器引用。
+- 浏览器启动异常保留简短真实原因，避免把执行权限错误误判为缺少 Playwright；DOM 轮询按连续失败次数判断停止。
+- 有效成功缓存优先于并发失败，成功写入或命中清理同键失败记录。
+- 站点爬取按 URL 路径边界过滤，保留 feedback、admin-guide 及查询参数中包含登录地址的正常页面。
+- 编码声明支持等号空格和无引号属性，排除 data-charset 伪属性；Markdown 链接与图片按真实属性读取，正确处理实体、括号和图片说明文本。
+
+### 验证
+
+- 297 个零依赖单元断言、15 个真实 Chromium SPA 断言、53 项对抗探针通过；新增 21 条单元断言和 3 条 SPA 断言。
+- 152 站实网回归：104 OK / 23 THIN+EMPTY / 25 ERR / 0 THREW，OK 站零噪声；Vue、React、酷狗三站已知 NOISE 在同输入 v1.8.0 A/B 中逐字一致。
+- 修复前 v1.7.0 / v1.8.0 同输入均复现相关缺陷，属于既有问题；核心零运行时依赖，新增或修改的正则保持有界。
+- 四工具 description 共 1,159 字符，参数 schema 共 1,876 字符；预算仍为 1,250 / 2,000。
+
 ## [1.8.0] - 2026-09-07
 
 ### 内容与准确性

package/README.md

@@ -47,7 +47,7 @@ DSH agents can search (getting links and snippets) but lack the step of "reading
 
 Implemented per official docs (`docs/capability-seams.md`, `docs/cordis-primer.md`, `docs/tool-execution-pipeline.md`):
 
-1. **Web access via the `ctx.web` capability seam** — use `ctx.web.fetch({ url }, signal)` and its decoded `body.kind/content` result for reads, redirects and crawl requests; fall back to direct retrieval when the service or provider is unavailable. Provider HTTP errors and policy decisions remain errors;
+1. **Web access via the `ctx.web` capability seam** — use `ctx.web.fetch({ url }, signal)` and its decoded `body.kind/content` result for reads, redirects and crawl requests; fall back to direct retrieval when the service or provider is unavailable. Provider HTTP errors and policy decisions remain errors. The optional `directFetchOrigins` profile setting selects explicit direct retrieval for trusted origins;
 2. **Reversible side effects** — the session cache is registered under `ctx.effect`, auto-cleared on plugin unload (temporal composability);
 3. **Cooperative tool-call timeout** — `ToolDefinition.timeoutMs` declares the budget, `execute(args, exec)` forwards `exec.signal` to fetch; the timeout policy is enforced by the pipeline, never exposed to the model;
 4. **Model-facing simplicity** — render emits compact text (`title:` header + body); the model consumes it directly with no JSON parsing. Defaults are the most token-efficient; structured output is opt-in;
@@ -181,6 +181,27 @@ Override via the profile's `cordis.patch.yml` (defaults in the plugin's own `cor
 
 Values are coerced and clamped to sane ranges at load — quoted numbers in YAML work, garbage falls back to defaults.
 
+#### Explicit direct origins (v1.9.0)
+
+`directFetchOrigins` defaults to `[]`. For a trusted local reverse proxy (for example, a GitHub hostname mapped by your hosts file), explicitly list the origins in the plugin's **profile configuration**, then reload the plugin/profile:
+
+```yaml
+- id: dsh-read-url
+  config:
+    directFetchOrigins:
+      - https://github.com
+      - https://api.github.com
+      - https://raw.githubusercontent.com
+```
+
+Listed origins use direct fetch immediately, bypassing the host web provider **and its SSRF checks**. Add only origins you trust; this also permits their DNS/hosts mappings to local or private addresses. The option is not a tool argument. Unlisted initial URLs keep the existing provider selection and fallback behavior; a provider refusal does not enable this exception automatically.
+
+Matching is exact after URL normalization: scheme, hostname and port must match; subdomains are separate. At most 32 entries of 2,048 characters each are accepted. Only an HTTP(S) origin with an optional trailing `/` is valid: credentials, paths, queries, fragments, wildcards and ambiguous syntax cause a configuration error. Unlike numeric options, invalid entries do not fall back to defaults.
+
+Once a request selects explicit direct retrieval, every subsequent HTTP redirect, meta-refresh, pagination request and descendant crawl stays within the list. Redirects handled internally by the provider remain on the provider path. Direct HTTP requests follow at most 5 redirects per attempt and stop loops early. An outside target is stopped before requesting it; refresh/pagination blocked by this policy retain the readable static result with a hint. Explicit direct reads use neither the plugin's curl proxy race nor SPA rendering, preventing those mechanisms from widening the exception. When `spaRender` is enabled and `read_url` or `read_url_links` would otherwise render a script-only page, they return a static-reading hint. Cancellation, response-size limits and one bounded Retry-After retry still apply. All four tools share the policy; success and failure caches are isolated by the complete normalized list, including after changing or clearing it.
+
+Controlled HTTP servers and provider fixtures cover default refusals, explicit routing, redirects, pagination, all four tools, cache isolation, cancellation, timeouts and response-size limits.
+
 ### Output (compact)
 
 ```json
@@ -274,10 +295,26 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 | Batch + failure isolation | 4-URL mix | ✅ 2/4 ok, failures isolated |
 | Site crawl | Ruan Yifeng blog | ✅ 5/5 pages tree map |
 
+### Released v1.8.0 validation snapshot
+
 - **276 zero-dependency assertions + 12 SPA assertions** pass. `node test.mjs` covers provider contracts, cancellation/retry, extraction/metadata, RSS/Atom, Markdown, continuation, batching and crawl boundaries. `node probe.mjs` checks pathological tags, nesting, long whitespace and bounded scanning.
 - Local maintenance validation (2026-09-07): **102 OK / 27 THIN+EMPTY / 23 ERR / 0 THREW** across 152 sites, with no noise among OK results; 53 adversarial probes pass. ERR results are network/anti-bot failures, invalid addresses or intentionally unsupported binary types. Known NOISE flags were checked against the previous code using identical inputs.
 - Real case: on a Xiaoheihe post, comment like-counts (`up` field) could not be attributed from flattened text — **precise fields should come from the page's underlying data API** (e.g. `/bbs/app/link/tree` JSON). This is a shared boundary of text extractors, not a defect.
 
+### v1.8.1 maintenance validation (2026-09-19)
+
+- **297 zero-dependency assertions + 15 real Chromium SPA assertions + 53 adversarial probes** pass. `test-lifecycle.mjs`, included in `test.mjs`, uses a local browser fixture for cancellation, launch failures, concurrent cleanup and restart checks.
+- The 152-site run returned **104 OK / 23 THIN+EMPTY / 25 ERR / 0 THREW**, with no noise among OK results. The three known NOISE sites have identical decoded content and extracted output against v1.8.0 using the same input. Network errors and intentionally unsupported types remain separate from extraction results.
+- SPA cancellation closes the active page; plugin disposal awaits browser shutdown. Valid success-cache entries take precedence over concurrent failures. Crawl filtering uses path boundaries. Charset and Markdown attributes use bounded parsing.
+- Fixed model input cost remains **1,159 description characters / 1,876 schema characters**, within the 1,250 / 2,000 budgets.
+
+### v1.9.0 validation (2026-09-19)
+
+- **322 zero-dependency unit assertions, 15 real Chromium SPA assertions and 53 adversarial probes** passed. The suite includes 22 direct-origin policy checks and 3 SPA restriction checks.
+- The 152-site sweep returned **105 OK / 23 THIN+EMPTY / 24 ERR / 0 THREW**, with no noise in OK results.
+- Vue, React and Kugou's known NOISE cases matched v1.8.1 exactly using identical HTML and optional dependencies, in both text/Markdown and full/800-character outputs. Errors were network, anti-bot, invalid-address or expected content-type boundaries.
+- All four tools enforce the explicit origin policy, including redirect/crawl boundaries, cache isolation and visible static-reading hints. Model input remains **1,159 description characters / 1,876 schema characters**, within the 1,250 / 2,000 budgets.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)
@@ -306,7 +343,7 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 
 ```bash
 node test.mjs          # zero-dependency self-tests (charset/extract/markdown/truncate)
-node test-spa.mjs      # SPA rendering tests (12 assertions; SKIPs if playwright absent)
+node test-spa.mjs      # SPA rendering tests (15 assertions; SKIPs if playwright absent)
 node multi-site.mjs    # 152-site real-world sweep (needs network, CONC=8 tunable): portals/SPA/login-walls/static/feeds/JSON/anti-bot/net-boundaries
 
 # End-to-end (requires DSH CLI)

package/README.zh.md

@@ -47,7 +47,7 @@ DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净
 
 按官方文档实现（`docs/capability-seams.md`、`docs/cordis-primer.md`、`docs/tool-execution-pipeline.md`）：
 
-1. **网络访问走 `ctx.web` 能力缝**——读取、跳转与爬取使用 `ctx.web.fetch({ url }, signal)` 及解码后的 `body.kind/content`；服务或提供方不可用时回退直连。提供方 HTTP 错误与策略拒绝保留为错误；
+1. **网络访问走 `ctx.web` 能力缝**——读取、跳转与爬取使用 `ctx.web.fetch({ url }, signal)` 及解码后的 `body.kind/content`；服务或提供方不可用时回退直连。提供方 HTTP 错误与策略拒绝保留为错误。可选的 profile 配置 `directFetchOrigins` 为可信来源选择显式直连；
 2. **可逆副作用**——会话缓存注册在 `ctx.effect` 下，插件卸载即自动清理（时间可组合性）；
 3. **协作式工具调用超时**——`ToolDefinition.timeoutMs` 声明预算，`execute(args, exec)` 把 `exec.signal` 转发给 fetch，超时策略由管线强制执行，不把超时暴露给模型；
 4. **模型视角精简**——render 输出紧凑文本（`title:` 头部 + 正文），模型直接消费，无需解析 JSON；默认参数最省 token，结构化能力按需开启；
@@ -181,6 +181,27 @@ chars 800+800/12398 · cached
 
 配置在加载时统一强转数字并钳制到合理范围——YAML 里加引号的数字也能用，非法值回退默认。
 
+#### 显式直连来源（v1.9.0）
+
+`directFetchOrigins` 默认为 `[]`。对于可信的本地反向代理（例如通过 hosts 文件映射 GitHub 域名），在插件的 **profile 配置**中显式列出来源，再重载插件/profile：
+
+```yaml
+- id: dsh-read-url
+  config:
+    directFetchOrigins:
+      - https://github.com
+      - https://api.github.com
+      - https://raw.githubusercontent.com
+```
+
+命中的来源立即直连，跳过宿主 web provider **及其 SSRF 检查**。仅添加你信任的来源；其 DNS/hosts 映射到本机或私有地址也会被允许。此选项不开放为工具参数。未列出的初始 URL 保留既有 provider 选择与回退行为；provider 拒绝不会自动启用此例外。
+
+按 URL 规范化后的协议、域名、端口精确匹配，子域名需单独配置。最多 32 项，每项不超过 2,048 字符。仅接受 HTTP(S) origin，可带一个结尾 `/`；凭据、路径、查询、片段、通配符和歧义写法都会触发配置错误。此选项与数字选项不同，非法项不会静默回退默认值。
+
+请求选用显式直连后，后续 HTTP 重定向、meta-refresh、分页及爬取子页面均限制在列表内；provider 内部处理的跳转仍属于 provider 路径。直连每次尝试最多跟随 5 次 HTTP 重定向，遇循环提前停止。越界目标在发请求前停止；被此策略阻止的刷新跳转或分页保留可读的静态结果并附提示。显式直连不使用插件的 curl 代理竞速或 SPA 渲染，防止它们扩大例外范围；启用 `spaRender` 且 `read_url` 或 `read_url_links` 原本需要渲染脚本空壳时，会提示仅能静态读取。取消、正文大小限制及一次有界 Retry-After 重试继续生效。四个工具统一遵守此策略；成功与失败缓存按完整规范化列表隔离，修改或清空列表后不会误用旧策略的结果。
+
+受控 HTTP 服务与 provider 桩覆盖默认拒绝、显式直连、重定向、分页、四工具、缓存隔离、取消、超时及正文上限。
+
 ### 输出结构（紧凑）
 
 ```json
@@ -274,10 +295,26 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 | **批量 + 失败隔离** | 4 URL 混合 | ✅ 2/4 成功、失败隔离 |
 | **整站爬取** | 阮一峰博客 | ✅ 5/5 页树状站点地图 |
 
+### 已发布 v1.8.0 验证快照
+
 - **276 个零依赖单元断言 + 12 个 SPA 断言**通过；`node test.mjs` 包含网络契约、取消重试、正文/元数据、RSS/Atom、Markdown、续读、批量与站点边界测试。`node probe.mjs` 验证极端标签、嵌套、长空白和有界扫描性能。
 - 本机维护验证（2026-09-07）：152 站为 **102 OK / 27 THIN+EMPTY / 23 ERR / 0 THREW**，OK 站零噪声；53 项对抗探针通过。ERR 为网络、反爬、无效地址或预期不支持的二进制类型；已知 NOISE 标记经同输入新旧 A/B 核查。
 - 一个真实案例：小黑盒帖子的评论点赞数（`up` 字段）无法从扁平文本确定归属——**精确字段应走页面背后的数据 API**（如 `/bbs/app/link/tree` JSON），这是同类文本提取器的共同边界，不是缺陷。
 
+### v1.8.1 维护验证（2026-09-19）
+
+- **297 个零依赖单元断言 + 15 个真实 Chromium SPA 断言 + 53 项对抗探针**通过。`test.mjs` 已包含 `test-lifecycle.mjs`，通过本地浏览器桩验证取消、启动失败、并发清理和重启。
+- 152 站实网结果为 **104 OK / 23 THIN+EMPTY / 25 ERR / 0 THREW**，OK 站零噪声。三个已知 NOISE 站在同输入 v1.8.0 对照中，解码内容与提取结果逐字一致；网络错误与预期不支持的类型单独归类。
+- SPA 取消关闭当前页面，插件卸载等待浏览器关闭；有效成功缓存优先于并发失败；爬取过滤遵循路径边界；编码与 Markdown 属性使用有界解析。
+- 模型固定输入成本仍为 **description 1,159 字符 / schema 1,876 字符**，低于 1,250 / 2,000 预算。
+
+### v1.9.0 验证（2026-09-19）
+
+- **322 个零依赖单元断言、15 个真实 Chromium SPA 断言、53 项对抗探针**通过；包括 22 条直连来源策略检查与 3 条 SPA 限制检查。
+- 152 站实网结果为 **105 OK / 23 THIN+EMPTY / 24 ERR / 0 THREW**，OK 站零噪声。
+- Vue、React、酷狗三个已知 NOISE 站采用同份 HTML 与相同可选依赖对照 v1.8.1，text/Markdown 全文及 800 字符窗口逐字一致；ERR 均为网络、反爬、无效地址或预期不支持的类型。
+- 四个工具统一遵守显式来源策略，覆盖跳转与爬取边界、缓存隔离及可见的静态读取提示。模型固定输入成本为 **description 1,159 字符 / schema 1,876 字符**，低于 1,250 / 2,000 预算。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）
@@ -308,7 +345,7 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 node test.mjs          # 单元自测（转码/提取/Markdown/截断/批量/站点爬取/缓存隔离/配置钳制/控制字符/JSON-LD/base href/meta-refresh）
 
 # SPA 渲染真实测试（需 playwright 已安装，未装自动 SKIP）
-node test-spa.mjs      # 12 断言：JS 正文/渲染后链接/工具不崩溃/缓存隔离/JS 跳转壳
+node test-spa.mjs      # 15 断言：JS 正文/渲染后链接/缓存隔离/导航取消/浏览器重启
 
 # 152 站真实世界验证（需联网，CONC=8 可调并发）
 node multi-site.mjs    # 门户/SPA/登录墙/静态/订阅源/JSON/反爬/网络边界，输出分级结果

package/cordis.patch.yml

@@ -11,4 +11,5 @@
         spaRender: true
         paginate: true
         paginateMax: 3
+        directFetchOrigins: []
         userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'

package/index.js

@@ -2,7 +2,8 @@
 // Zero external dependencies (Node 20+ built-ins only).
 // Aligned with DSH architecture:
 //   - all network access goes through the ctx.web capability seam (official
-//     docs/capability-seams.md), falling back to global fetch when absent
+//     docs/capability-seams.md), except explicit directFetchOrigins; falls
+//     back to global fetch when absent
 //   - cache is registered under ctx.effect, so unload fully reverts it
 //     (temporal composability, docs/cordis-primer.md)
 //   - cooperative tool-call timeout via ToolDefinition.timeoutMs + exec.signal
@@ -33,7 +34,8 @@ const DEFAULTS = {
   cacheMax: 32,
   spaRender: true,
   paginate: true,
-  paginateMax: 3,
+  paginateMax: 3,
+  directFetchOrigins: [],
   userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36',
 }
 
@@ -46,11 +48,57 @@ function cacheFail(key, error) {
   if (failCache.size >= FAIL_CACHE_MAX) failCache.delete(failCache.keys().next().value)
   failCache.set(key, { time: Date.now(), error })
 }
-function cacheStore(key, full, cacheMax) {
-  if (cache.size >= cacheMax) cache.delete(cache.keys().next().value)
-  cache.set(key, { time: Date.now(), full })
+function cacheStore(key, full, cacheMax) {
+  if (cache.size >= cacheMax) cache.delete(cache.keys().next().value)
+  cache.set(key, { time: Date.now(), full })
+  failCache.delete(key)
 }
-const decoders = new Map()
+const decoders = new Map()
+
+// Owner configuration only: tool arguments cannot widen this trust boundary.
+// Preserve scheme/port, reject ambiguous origin syntax, and fail closed on
+// invalid configuration instead of silently accepting a partial allowlist.
+function directOrigins(values = []) {
+  const invalid = () => new Error('Invalid directFetchOrigins: expected at most 32 exact http(s) origins without credentials, paths, queries, fragments or wildcards')
+  if (!Array.isArray(values) || values.length > 32) throw invalid()
+  const origins = new Set()
+  for (const value of values) {
+    if (typeof value !== 'string' || value.length > 2048) throw invalid()
+    const raw = value.trim()
+    let u
+    try { u = new URL(raw) } catch { throw invalid() }
+    if (!raw.toLowerCase().startsWith(`${u.protocol}//`) ||
+        !['http:', 'https:'].includes(u.protocol) || u.username || u.password ||
+        u.pathname !== '/' || raw.includes('?') || raw.includes('#') ||
+        raw.includes('*') || raw.includes('\\') || raw.includes('@') ||
+        [...raw].some(c => c.charCodeAt(0) <= 32 || c.charCodeAt(0) === 127)) throw invalid()
+    // Do not accept URL parser repairs such as /a/.. or percent escapes in
+    // hostnames. Only an authority and one optional trailing slash are valid.
+    const authority = raw.slice(raw.indexOf('://') + 3)
+    if (authority.includes('%') || authority.replace(/\/$/, '').includes('/')) throw invalid()
+    origins.add(u.origin)
+  }
+  return [...origins].sort()
+}
+
+function createRoute(cfg) {
+  const origins = directOrigins(cfg.directFetchOrigins)
+  return { origins: new Set(origins), key: JSON.stringify(origins), restricted: false, hint: '' }
+}
+
+function matchesDirectOrigin(url, origins) {
+  try { return origins.has(new URL(url).origin) } catch { return false }
+}
+
+function allowedDirectUrl(url, origins) {
+  try {
+    const u = new URL(url)
+    return !u.username && !u.password && origins.has(u.origin) && ['http:', 'https:'].includes(u.protocol)
+  } catch { return false }
+}
+
+const DIRECT_BOUNDARY_ERROR = 'Direct fetch stopped: target is outside directFetchOrigins or contains credentials'
+const DIRECT_SPA_HINT = '已配置直连仅支持静态读取，未启用浏览器渲染'
 
 function getDecoder(enc) {
   let d = decoders.get(enc)
@@ -65,21 +113,29 @@ function getDecoder(enc) {
   return d
 }
 
-function sniffCharset(buffer, contentType) {
+function declaredCharset(contentType) {
+  const m = /(?:^|;)[ \t]{0,1000}charset[ \t]{0,1000}=[ \t]{0,1000}(?:"([\w.:-]{1,80})"|'([\w.:-]{1,80})'|([\w.:-]{1,80}))(?=[ \t;]|$)/i.exec(contentType || '')
+  return m ? m[1] || m[2] || m[3] : null
+}
+
+function sniffCharset(buffer, contentType) {
   // BOM first — byte-level evidence beats any declared charset (a UTF-16 body
   // with a latin1-read <meta> probe garbles the meta match anyway).
   if (buffer[0] === 0xef && buffer[1] === 0xbb && buffer[2] === 0xbf) return 'utf-8'
   if (buffer[0] === 0xff && buffer[1] === 0xfe) return 'utf-16le'
   if (buffer[0] === 0xfe && buffer[1] === 0xff) return 'utf-16be'
-  const m = /charset=["']?([\w-]+)/i.exec(contentType || '')
-  if (m) return m[1]
-  const head = buffer.subarray(0, 2048).toString('latin1').toLowerCase()
-  const meta = /<meta[^>]{0,1000}charset=["']?([\w-]+)/i.exec(head)
-  if (meta) return meta[1]
-  // Legacy form: <meta http-equiv="Content-Type" content="text/html; charset=gb2312">
-  // (no charset= attribute anywhere) — common on old GBK pages with no header charset.
-  const metaEquiv = /<meta[^>]{0,1000}(?:http-equiv)=["']content-type["'][^>]{0,1000}content=["'][^"']*charset=([\w-]+)/i.exec(head)
-  if (metaEquiv) return metaEquiv[1]
+  const header = declaredCharset(contentType)
+  if (header) return header
+  const head = buffer.subarray(0, 2048).toString('latin1')
+  for (const meta of head.matchAll(/<meta(?=[\s/>])[^>]{0,1000}>/gi)) {
+    const attrs = htmlAttributes(meta[0])
+    const charset = attrs.charset?.trim()
+    if (charset && /^[\w.:-]{1,80}$/.test(charset)) return charset
+    if (attrs['http-equiv']?.toLowerCase() === 'content-type') {
+      const legacy = declaredCharset(attrs.content)
+      if (legacy) return legacy
+    }
+  }
   return null
 }
 
@@ -115,13 +171,31 @@ function retryAfterMs(res) {
 }
 
 // Single fetch attempt. Never throws; maps abort/timeout/network to { error }.
-async function directFetchOnce(url, signal, cfg) {
-  try {
-    const res = await fetch(url, {
-      redirect: 'follow',
-      signal,
-      headers: { 'user-agent': cfg.userAgent, accept: 'text/html,application/xhtml+xml,*/*;q=0.8' },
-    })
+async function directFetchOnce(url, signal, cfg, origins) {
+  try {
+    let currentUrl = url
+    let res
+    const seen = new Set()
+    for (let hops = 0; ; hops++) {
+      if (signal && signal.aborted) return { error: 'cancelled' }
+      if (origins && !allowedDirectUrl(currentUrl, origins)) return { error: DIRECT_BOUNDARY_ERROR }
+      if (origins) {
+        const normalized = normalizeUrl(currentUrl)
+        if (seen.has(normalized)) return { error: 'Direct fetch redirect loop' }
+        seen.add(normalized)
+      }
+      res = await fetch(currentUrl, {
+        redirect: origins ? 'manual' : 'follow',
+        signal,
+        headers: { 'user-agent': cfg.userAgent, accept: 'text/html,application/xhtml+xml,*/*;q=0.8' },
+      })
+      if (!origins || ![301, 302, 303, 307, 308].includes(res.status)) break
+      if (res.body) res.body.cancel().catch(() => {})
+      const location = res.headers.get('location')
+      if (!location) return { error: `HTTP ${res.status} without Location` }
+      if (hops >= 5) return { error: 'Direct fetch exceeded 5 redirects' }
+      currentUrl = new URL(location, currentUrl).href
+    }
     // Early-return paths must cancel the body stream, or the connection is
     // held open undrained and cannot return to the keep-alive pool.
     const discardBody = () => { if (res.body) res.body.cancel().catch(() => {}) }
@@ -141,7 +215,7 @@ async function directFetchOnce(url, signal, cfg) {
       discardBody()
       return { error: `Unsupported content-type: ${contentType.split(';')[0]}` }
     }
-    if (!res.body) return { buffer: Buffer.alloc(0), contentType, finalUrl: res.url }
+    if (!res.body) return { buffer: Buffer.alloc(0), contentType, finalUrl: res.url || currentUrl }
     const chunks = []
     let size = 0
     for await (const chunk of res.body) {
@@ -154,7 +228,7 @@ async function directFetchOnce(url, signal, cfg) {
     if (!contentType && looksBinary(buffer)) {
       return { error: 'Unsupported content-type (no header, binary body)' }
     }
-    return { buffer, contentType, finalUrl: res.url }
+    return { buffer, contentType, finalUrl: res.url || currentUrl }
   } catch (e) {
     if (e.name === 'AbortError' || e.name === 'TimeoutError') {
       return { error: `Timeout after ${cfg.timeoutMs}ms or cancelled` }
@@ -171,15 +245,15 @@ async function directFetchOnce(url, signal, cfg) {
 // Exported for tests: the double-throttle path must always produce an { error }
 // shape — a second `{ retryAfterMs }` would otherwise leak past the race
 // filter into decodeBuffer as a bufferless "success" (bare TypeError there).
-export async function directFetch(url, signal, cfg) {
-  const first = await directFetchOnce(url, signal, cfg)
+export async function directFetch(url, signal, cfg, origins) {
+  const first = await directFetchOnce(url, signal, cfg, origins)
   if (first && first.retryAfterMs !== undefined && !(signal && signal.aborted)) {
     await new Promise((resolve) => {
       const t = setTimeout(resolve, first.retryAfterMs)
       if (signal) signal.addEventListener('abort', () => { clearTimeout(t); resolve() }, { once: true })
     })
     if (signal && signal.aborted) return { error: 'cancelled' }
-    const second = await directFetchOnce(url, signal, cfg)
+    const second = await directFetchOnce(url, signal, cfg, origins)
     // The retry budget is spent — report the rate limit instead of returning
     // a shapeless { retryAfterMs } (no buffer, no error) to the caller.
     if (second && second.retryAfterMs !== undefined) {
@@ -300,6 +374,27 @@ async function fetchViaWebSeam(ctx, url, externalSignal, cfg) {
     return { error: `Web provider failed: ${String((e && e.message) || e).slice(0, 160)}` }
   }
 }
+
+// Every fetch in a tool call passes here. Once a chain enters an explicit
+// direct origin it stays restricted, including meta refresh and pagination.
+// Never send an exception through curl -L or a browser: both may make further
+// requests outside this origin list. Unlisted initial URLs retain host policy.
+async function fetchResource(ctx, url, externalSignal, cfg, route) {
+  if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+  if (route.restricted || matchesDirectOrigin(url, route.origins)) {
+    route.restricted = true
+    if (!allowedDirectUrl(url, route.origins)) {
+      route.hint = DIRECT_BOUNDARY_ERROR
+      return { error: DIRECT_BOUNDARY_ERROR }
+    }
+    const timeout = AbortSignal.timeout(cfg.timeoutMs)
+    const signal = externalSignal ? AbortSignal.any([externalSignal, timeout]) : timeout
+    const page = await directFetch(url, signal, cfg, route.origins)
+    if (page.error) route.hint = page.error
+    return page
+  }
+  return await fetchViaWebSeam(ctx, url, externalSignal, cfg) || await fetchPage(url, externalSignal, cfg)
+}
 
 // Decode common HTML entities in already-stripped text. Runs AFTER tag
 // stripping so escaped tags stay visible to the stripper and only remaining
@@ -708,9 +803,15 @@ function stripNoise(mainHtml) {
 }
 
 // ---- HTML -> Markdown (lightweight, tag-state machine) ----
-function escInline(s) {
-  return s.replace(/([\\`*_[\]])/g, '\\$1')
-}
+function escInline(s) {
+  return s.replace(/([\\`*_[\]])/g, '\\$1')
+}
+
+function markdownTarget(value) {
+  return decodeUrlAttribute(value).trim()
+    .replace(/\(/g, '%28').replace(/\)/g, '%29')
+    .replace(/[<>\s\\]/g, encodeURIComponent)
+}
 
 // <img> is a void element (never matches the paired-tag regex in either md
 // walker): convert descriptive images to markdown BEFORE the tag walk, via a
@@ -729,25 +830,30 @@ function imgsToMarkdown(html) {
   // in <picture><source srcset>. When the <img> fallback carries no usable
   // src (lazy placeholder), the first source URL is injected into it so the
   // normal img pass below still emits the image.
-  html = html.replace(/<picture\b[^>]*>([\s\S]{0,50000}?)<\/picture\s*>/gi, (block) => {
-    const imgTag = /<img\b[^>]{0,1000}>/i.exec(block)
-    const imgSrc = imgTag && /(?:^|\s)src=["']([^"']+)["']/i.exec(imgTag[0])
-    if (imgTag && imgSrc && !/^data:/i.test(imgSrc[1])) return block
-    const source = /<source\b[^>]{0,1000}\bsrcset=["']([^"'\s>]+)/i.exec(block)
-    if (!source || !imgTag) return block
-    return block.replace(/<img\b[^>]{0,1000}>/i, (img0) => img0.replace(/<img/i, `<img src="${source[1]}"`))
-  })
-  const out = html.replace(/<(?:amp-)?img\b[^>]{0,1000}>/gi, (tag) => {
-    const alt = /alt=["']([^"']*)["']/i.exec(tag)
-    let src = /src=["']([^"']+)["']/i.exec(tag)
-    let srcVal = src && src[1]
+  html = html.replace(/<picture\b[^>]{0,1000}>([\s\S]{0,50000}?)<\/picture\s{0,1000}>/gi, (block) => {
+    const imgTag = /<img\b[^>]{0,1000}>/i.exec(block)
+    const imgSrc = imgTag && htmlAttributes(imgTag[0]).src
+    if (!imgTag || (imgSrc && !/^data:/i.test(imgSrc))) return block
+    let source = ''
+    for (const tag of block.matchAll(/<source\b[^>]{0,1000}>/gi)) {
+      source = htmlAttributes(tag[0]).srcset?.trim().split(/[\s,]{1,1000}/, 1)[0] || ''
+      if (source) break
+    }
+    if (!source) return block
+    const escaped = decodeUrlAttribute(source).replace(/&/g, '&amp;').replace(/"/g, '&quot;')
+    return block.replace(imgTag[0], () => `<img src="${escaped}"${imgTag[0].slice(4)}`)
+  })
+  const out = html.replace(/<(?:amp-)?img\b[^>]{0,1000}>/gi, (tag) => {
+    const attrs = htmlAttributes(tag)
+    const alt = decodeTextEntities(attrs.alt || '').replace(/\s{1,1000}/g, ' ').trim()
+    let srcVal = attrs.src?.trim()
     // Lazy-load wrappers keep the real URL in data-* attributes; a data: URI
     // src is an inline placeholder spacer, not a usable image reference.
     if (!srcVal || /^data:/i.test(srcVal)) {
-      const lazy = /data-(?:src|original|lazy-src)=["']([^"']+)["']/i.exec(tag)
-      if (lazy) srcVal = lazy[1]
-    }
-    return protect(srcVal && alt && alt[1].trim() ? `![${alt[1].trim()}](${srcVal})` : '')
+      const lazy = attrs['data-src'] || attrs['data-original'] || attrs['data-lazy-src']
+      if (lazy) srcVal = lazy.trim()
+    }
+    return protect(srcVal && alt ? `![${escInline(alt)}](${markdownTarget(srcVal)})` : '')
   })
   return {
     html: out,
@@ -826,13 +932,13 @@ export function inlineMd(html, depth = 0) {
       out += img.protect(inlineCode(m[3]))
       continue
     }
-    const inner = inlineMd(m[3], depth + 1)
-    if (tag === 'a') {
-      const href = /href=["']([^"']+)["']/i.exec(m[2])
+    const inner = inlineMd(m[3], depth + 1)
+    if (tag === 'a') {
+      const href = htmlAttributes(`<a${m[2]}>`).href
       // A raw ')' in the target would terminate the markdown link early
       // (common on wiki/framework URLs) — percent-encode parens to keep the
       // link parseable by the model.
-      out += href ? `[${inner}](${href[1].replace(/\)/g, '%29').replace(/\(/g, '%28')})` : inner
+      out += href?.trim() ? `[${inner}](${markdownTarget(href)})` : inner
     } else if (tag === 'strong' || tag === 'b') out += `**${inner}**`
     else if (tag === 'em' || tag === 'i') out += `*${inner}*`
     else out += inner
@@ -1547,7 +1653,7 @@ export function metaRefreshTarget(html, baseUrl) {
 // cache key is the ORIGINAL url, and a shell re-request should re-follow
 // rather than serve a foreign page.
 const MAX_META_REFRESH_HOPS = 3
-async function followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx) {
+async function followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx, route) {
   let cur = html
   let curUrl = finalUrl
   let curCharset = '' // set only when a hop actually happened
@@ -1566,7 +1672,7 @@ async function followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx) {
     const norm = normalizeUrl(target)
     if (!norm || seen.has(norm)) break
     seen.add(norm)
-    const page = await fetchViaWebSeam(ctx, target, hopSignal, cfg) || await fetchPage(target, hopSignal, cfg)
+    const page = await fetchResource(ctx, target, hopSignal, cfg, route)
     if (page.error) break
     const ct = page.kind === 'html' ? 'text/html' : (page.contentType || '').split(';')[0].toLowerCase().trim()
     if (!/html|xhtml/.test(ct)) break
@@ -1583,6 +1689,8 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   const url = String((args && args.url) || '').trim()
   if (!/^https?:\/\//i.test(url)) return { error: 'Only http/https URLs are supported' }
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+  let route
+  try { route = createRoute(cfg) } catch (e) { return { error: e.message } }
   const maxChars = Math.max(500, Math.min(20000, Number(args.maxChars) || cfg.maxChars))
   const mode = args.mode === 'markdown' ? 'markdown' : 'text'
   const offset = Math.max(0, Number(args.offset) || 0)
@@ -1612,8 +1720,20 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   } catch {
     /* keep bare url */
   }
-  const cacheKey = `${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
-  // Failed fetches are cached briefly so the model doesn't re-request a
+  // Include the complete policy: an unlisted entry may reach a listed origin
+  // through refresh/pagination, and must not reuse content after revocation.
+  const cacheKey = `${route.key}|${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
+  const hit = cache.get(cacheKey)
+  if (hit && Date.now() - hit.time < cfg.cacheTtlMs) {
+    // Valid content takes precedence over a concurrent fetch failure.
+    failCache.delete(cacheKey)
+    // LRU touch keeps hot pages available throughout their TTL.
+    cache.delete(cacheKey)
+    cache.set(cacheKey, hit)
+    const sliced = sliceFrom(hit.full, offset, maxChars)
+    return { ...sliced, cached: true }
+  }
+  // Failed fetches are cached briefly so the model doesn't re-request a
   // broken URL in a loop (token + latency saver).
   const fh = failCache.get(cacheKey)
   if (fh) {
@@ -1626,15 +1746,6 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     }
     failCache.delete(cacheKey)
   }
-  const hit = cache.get(cacheKey)
-  if (hit && Date.now() - hit.time < cfg.cacheTtlMs) {
-    // LRU touch: hot pages survive eviction even when cached long ago — the
-    // Map re-insertion moves the entry to the newest slot.
-    cache.delete(cacheKey)
-    cache.set(cacheKey, hit)
-    const sliced = sliceFrom(hit.full, offset, maxChars)
-    return { ...sliced, cached: true }
-  }
 
   const failWith = (error) => {
     if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
@@ -1679,9 +1790,9 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     }
   }
   const dispatch = async () => {
-    const viaSeam = await fetchViaWebSeam(ctx, url, externalSignal, cfg)
-    const page = viaSeam || await fetchPage(url, externalSignal, cfg)
+    const page = await fetchResource(ctx, url, externalSignal, cfg, route)
     if (page.error) return { error: page.error }
+    const viaSeam = typeof page.html === 'string'
     const ct = viaSeam ? (page.kind === 'text' ? 'text/plain' : 'text/html') : (page.contentType || '').split(';')[0].toLowerCase().trim()
     const decoded = viaSeam ? { text: page.html, charset: 'provider-decoded' } : decodeBuffer(page.buffer, page.contentType)
     const meta = { finalUrl: page.finalUrl || url, sourceTruncated: page.sourceTruncated === true }
@@ -1761,7 +1872,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   // follow immediately, BEFORE extraction and SPA rendering — a shell that
   // redirects needs no headless round-trip, and extracting the stub is wasted
   // work. Fails open on any fetch problem; the follow is hop-bounded.
-  const followed = await followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx)
+  const followed = await followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx, route)
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   html = followed.html
   finalUrl = followed.finalUrl
@@ -1805,7 +1916,10 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   let renderedHtml = null
   const needsRender = !extracted.text || extracted.text.length < 200
   const scriptShell = !extracted.text && /<script[\s>]/i.test(html)
-  if (cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
+  if (route.restricted && cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
+    spaHint = [spaHint, DIRECT_SPA_HINT].filter(Boolean).join('；')
+  }
+  if (!route.restricted && cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
     const rr = await renderPage(finalUrl || url, externalSignal)
     if (rr.html) {
       renderedHtml = rr.html
@@ -1861,7 +1975,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     const nu = normalizeUrl(nextUrl)
     if (!nu || seen.has(nu)) break
     seen.add(nu)
-    const page = await fetchViaWebSeam(ctx, nextUrl, externalSignal, cfg) || await fetchPage(nextUrl, externalSignal, cfg)
+    const page = await fetchResource(ctx, nextUrl, externalSignal, cfg, route)
     if (page.error) break
     const pageFinal = normalizeUrl(page.finalUrl || nextUrl)
     if (!pageFinal || !sameHost(pageFinal, startHost) || (pageFinal !== nu && seen.has(pageFinal))) break
@@ -1877,7 +1991,9 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     nextUrl = findNextLink(nextHtml, page.finalUrl || nextUrl)
   }
 
-  // A near-empty body (login wall, anti-bot shell) with no render hint yet
+  if (route.hint) spaHint = [spaHint, route.hint].filter(Boolean).join('；')
+
+  // A near-empty body (login wall, anti-bot shell) with no render hint yet
   // still looks like "the page content" to the model — one constant-cost line
   // tells it not to keep offset-reading a 40-char page.
   if (!spaHint && extracted.text && extracted.text.length < 60) {
@@ -1971,8 +2087,9 @@ function renderLinks(value) {
   if (typeof value === 'string') return value
   const r = value || {}
   if (r.error) return `Error: ${r.error}`
-  if (!Array.isArray(r.links) || r.links.length === 0) return `No links found on ${r.url}`
-  const lines = [`${r.count} link(s) on ${r.url}:`, UNTRUSTED_NOTICE]
+  if (!Array.isArray(r.links) || r.links.length === 0) return `No links found on ${r.url}${r.spaHint ? `\n提示: ${r.spaHint}` : ''}`
+  const lines = [`${r.count} link(s) on ${r.url}:`, UNTRUSTED_NOTICE]
+  if (r.spaHint) lines.push(`提示: ${r.spaHint}`)
   for (const l of r.links) lines.push(`- ${l.title || l.url} — ${l.url}`)
   return lines.join('\n')
 }
@@ -2012,14 +2129,15 @@ function readLinksTool(ctx, cfg) {
     async execute(args, exec) {
       const url = String((args && args.url) || '').trim()
       if (!/^https?:\/\//i.test(url)) return { error: 'Only http/https URLs are supported' }
-      const limit = Math.max(1, Math.min(50, Number(args.limit) || cfg.maxLinks))
-      const page = await fetchViaWebSeam(ctx, url, exec && exec.signal, cfg) || await fetchPage(url, exec && exec.signal, cfg)
+      const limit = Math.max(1, Math.min(50, Number(args.limit) || cfg.maxLinks))
+      const route = createRoute(cfg)
+      const page = await fetchResource(ctx, url, exec && exec.signal, cfg, route)
       if (page.error) return { error: page.error }
       let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
       let finalUrl = page.finalUrl || url
       // Same meta-refresh shell following as read_url: a link hop serves a
       // stub whose real links live at the target. Fails open, hop-bounded.
-      const followed = await followMetaRefresh(html, finalUrl, exec && exec.signal, cfg, ctx)
+      const followed = await followMetaRefresh(html, finalUrl, exec && exec.signal, cfg, ctx, route)
       html = followed.html
       finalUrl = followed.finalUrl
       let links = extractLinks(html, limit, finalUrl)
@@ -2027,7 +2145,8 @@ function readLinksTool(ctx, cfg) {
       // render it and re-extract when the static result looks empty. Any
       // <script> counts as renderable when there are no links at all — the
       // static shell may be a JS-redirect (same rationale as read_url).
-      if (links.length < 3 && cfg.spaRender !== false && (looksLikeSpa(html) || (links.length === 0 && /<script[\s>]/i.test(html)))) {
+      const needsRender = links.length < 3 && cfg.spaRender !== false && (looksLikeSpa(html) || (links.length === 0 && /<script[\s>]/i.test(html)))
+      if (!route.restricted && needsRender) {
         const rr = await renderPage(finalUrl || url, exec && exec.signal)
         // A challenge interstitial's links are the challenge's own scripts —
         // never an improvement over whatever the static HTML offered.
@@ -2039,7 +2158,8 @@ function readLinksTool(ctx, cfg) {
           }
         }
       }
-      return { url: finalUrl, count: links.length, links }
+      const spaHint = [route.hint, route.restricted && needsRender ? DIRECT_SPA_HINT : ''].filter(Boolean).join('；')
+      return { url: finalUrl, count: links.length, links, ...(spaHint ? { spaHint } : {}) }
     },
   }
 }
@@ -2070,7 +2190,8 @@ function renderBatch(value) {
       continue
     }
     const head = p.title || p.url
-    lines.push('', `--- ${head} (${p.chars} 字符${p.cached ? ' · cached' : ''}) ---`, p.text || '(无可读内容)')
+    lines.push('', `--- ${head} (${p.chars} 字符${p.cached ? ' · cached' : ''}) ---`, p.text || '(无可读内容)')
+    if (p.spaHint) lines.push(`提示: ${p.spaHint}`)
     if (Array.isArray(p.links) && p.links.length) {
       const ls = p.links.map((l) => l.title + ' — ' + l.url)
       lines.push(`links: ${ls.slice(0, 6).join(' | ')}${ls.length > 6 ? ` …共${ls.length}个` : ''}`)
@@ -2160,7 +2281,8 @@ function readUrlBatchTool(ctx, cfg) {
             cached: p.cached === true,
             text: p.text,
           }
-          if (Array.isArray(p.links) && p.links.length) out.links = p.links
+          if (Array.isArray(p.links) && p.links.length) out.links = p.links
+          if (p.spaHint) out.spaHint = p.spaHint
           return out
         }),
       }
@@ -2189,18 +2311,24 @@ function sameHost(a, b) {
 }
 
 // URLs not worth crawling: static assets, login/auth paths, feeds, sitemaps.
-const NOISE_EXT = /\.(png|jpe?g|gif|svg|webp|ico|bmp|css|js|json|xml|pdf|zip|gz|tar|7z|mp3|m3u8?|mpd|flv|ts|mp4|avi|mov|webm|woff2?|ttf|eot|map)(\?|#|$)/i
-const NOISE_PATH = /(\/login|\/signin|\/register|\/logout|\/signup|\/api\/|\/admin|\/wp-admin|\/wp-login|\/feed|\/rss|\/sitemap|\/robots\.txt|\/cdn-cgi)/i
-function isNoiseUrl(url) {
-  return NOISE_EXT.test(url) || NOISE_PATH.test(url)
-}
+const NOISE_EXT = /\.(png|jpe?g|gif|svg|webp|ico|bmp|css|js|json|xml|pdf|zip|gz|tar|7z|mp3|m3u8?|mpd|flv|ts|mp4|avi|mov|webm|woff2?|ttf|eot|map)$/i
+const NOISE_PATH = /\/(?:login|signin|register|logout|signup|api|admin|wp-admin|wp-login|feed|rss|sitemap|robots\.txt|cdn-cgi)(?:\/|[.;]|$)/i
+function isNoiseUrl(url) {
+  try {
+    const path = new URL(url).pathname
+    return NOISE_EXT.test(path) || NOISE_PATH.test(path)
+  } catch {
+    return true
+  }
+}
 
 // BFS crawl of a single site. Only same-host http(s) pages are followed;
 // static assets / auth paths are skipped; visited URLs are deduped; a per-page
 // failure is recorded and does not abort the crawl. No SPA rendering here —
 // that is read_url's job; crawling favors speed and breadth.
 async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
-  const { maxPages, maxDepth, includeContent, perMax } = opts
+  const { maxPages, maxDepth, includeContent, perMax } = opts
+  const policy = createRoute(cfg)
   let host = hostOf(entryUrl)
   const visited = new Set()
   const resolved = new Set()
@@ -2214,12 +2342,13 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
     // Never overshoot the page budget within a batch round.
     const batch = queue.splice(0, Math.min(2, maxPages - attempted, queue.length))
     const results = await Promise.all(
-      batch.map(async ({ url, depth }) => {
+      batch.map(async ({ url, depth, restricted = false }) => {
         const norm = normalizeUrl(url)
         if (!norm || visited.has(norm)) return null
         visited.add(norm)
         attempted++
-        const page = await fetchViaWebSeam(ctx, url, externalSignal, cfg) || await fetchPage(url, externalSignal, cfg)
+        const route = { ...policy, restricted }
+        const page = await fetchResource(ctx, url, externalSignal, cfg, route)
         if (page.error) return { url, depth, error: page.error }
         let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
         let pageUrl = page.finalUrl || url
@@ -2230,7 +2359,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         // be crawled AS the page (title-less hop text pollutes results and the
         // real content is never reached). Fails open — fetch problems keep the
         // stub's own HTML.
-        const followed = await followMetaRefresh(html, pageUrl, externalSignal, cfg, ctx)
+        const followed = await followMetaRefresh(html, pageUrl, externalSignal, cfg, ctx, route)
         html = followed.html
         pageUrl = followed.finalUrl
         if (depth === 0) host = hostOf(pageUrl)
@@ -2245,7 +2374,9 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
           title: ex.title || '',
           chars: ex.text.length,
           text: includeContent ? smartTruncate(ex.text, perMax).text : undefined,
-          links,
+          links,
+          route,
+          ...(route.hint ? { spaHint: route.hint } : {}),
         }
       }),
     )
@@ -2259,7 +2390,8 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
       if (!finalNorm || resolved.has(finalNorm)) continue
       resolved.add(finalNorm)
       visited.add(finalNorm)
-      pages.push(r)
+      const { route, ...publicPage } = r
+      pages.push(publicPage)
       if (r.depth + 1 <= maxDepth) {
         // Queue cap: a huge site (millions of links) would otherwise grow the
         // queue unboundedly while maxPages bounds only what we *process*.
@@ -2267,7 +2399,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         for (const l of r.links) {
           if (!sameHost(l.url, host) || isNoiseUrl(l.url)) continue
           const n = normalizeUrl(l.url)
-          if (n && !visited.has(n)) queue.push({ url: l.url, depth: r.depth + 1 })
+          if (n && !visited.has(n)) queue.push({ url: l.url, depth: r.depth + 1, restricted: route.restricted })
         }
       }
     }
@@ -2283,7 +2415,8 @@ function renderSite(value) {
   for (const p of v.pages) {
     const indent = '  '.repeat(p.depth)
     const head = p.title || p.url
-    lines.push(`${indent}[${p.depth}] ${head} (${p.chars} 字符)  ${p.url}`)
+    lines.push(`${indent}[${p.depth}] ${head} (${p.chars} 字符)  ${p.url}`)
+    if (p.spaHint) lines.push(`${indent}提示: ${p.spaHint}`)
     if (p.text) lines.push(`${indent}   ${p.text.slice(0, 80)}`)
   }
   for (const f of v.failures) lines.push(`[失败] ${f.url} — ${f.error}`)
@@ -2350,9 +2483,10 @@ function readUrlSiteTool(ctx, cfg) {
         succeeded: pages.length,
         failed: failures.length,
         pages: pages.map((p) => {
-          const o = { url: p.url, depth: p.depth, title: p.title || '', chars: p.chars }
-          if (p.text) o.text = p.text
-          return o
+          const o = { url: p.url, depth: p.depth, title: p.title || '', chars: p.chars }
+          if (p.text) o.text = p.text
+          if (p.spaHint) o.spaHint = p.spaHint
+          return o
         }),
         failures,
       }
@@ -2379,14 +2513,15 @@ export function apply(ctx, config) {
   cfg.paginateMax = clamp(cfg.paginateMax, DEFAULTS.paginateMax, 1, 10)
   if (typeof cfg.userAgent !== 'string' || !cfg.userAgent.trim()) cfg.userAgent = DEFAULTS.userAgent
   if (typeof cfg.spaRender !== 'boolean') cfg.spaRender = DEFAULTS.spaRender
-  if (typeof cfg.paginate !== 'boolean') cfg.paginate = DEFAULTS.paginate
+  if (typeof cfg.paginate !== 'boolean') cfg.paginate = DEFAULTS.paginate
+  cfg.directFetchOrigins = directOrigins(cfg.directFetchOrigins)
 
   // Temporal composability: unload must fully revert side effects.
-  ctx.effect(() => () => {
-    cache.clear()
-    failCache.clear()
-    closeBrowser().catch(() => {})
-  })
+  ctx.effect(() => () => {
+    cache.clear()
+    failCache.clear()
+    return closeBrowser()
+  })
 
   console.log('[dsh-read-url] plugin loaded; tools read_url, read_url_batch, read_url_links, read_url_site registered')
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",

package/spa.js

@@ -4,6 +4,11 @@
 // When absent, reads fall back to static extraction with a clear hint.
 
 let browserPromise = null
+const closingBrowsers = new Set()
+
+function renderError(error) {
+  return `Render failed: ${String(error?.message || error).split('\n', 1)[0].slice(0, 500)}`
+}
 
 // Heuristic: a page whose HTML carries many <script> tags is likely a
 // client-rendered SPA (Vue/React) whose body lives only after JS execution.
@@ -33,12 +38,19 @@ async function getBrowser() {
   if (!browserPromise) {
     // A failed launch (playwright later installed / chromium download finished)
     // must not stay cached as a rejected promise — reset so the next call retries.
-    browserPromise = import('playwright')
+    const pending = import('playwright')
       .then(({ chromium }) => chromium.launch({ headless: true }))
+      .then((browser) => {
+        browser.once('disconnected', () => {
+          if (browserPromise === pending) browserPromise = null
+        })
+        return browser
+      })
       .catch((e) => {
-        browserPromise = null
+        if (browserPromise === pending) browserPromise = null
         throw e
       })
+    browserPromise = pending
   }
   return browserPromise
 }
@@ -49,12 +61,23 @@ export async function renderPage(url, externalSignal) {
   let browser
   try {
     browser = await getBrowser()
-  } catch {
-    return { error: 'SPA 渲染需 playwright（npm i playwright && npx playwright install chromium）' }
+  } catch (error) {
+    if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+    return { error: error?.code === 'ERR_MODULE_NOT_FOUND'
+      ? 'SPA 渲染需 playwright（npm i playwright && npx playwright install chromium）'
+      : renderError(error) }
   }
+  if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   let page
+  let pageClosing
+  const closePage = () => {
+    if (page && !pageClosing) pageClosing = page.close().catch(() => {})
+    return pageClosing
+  }
   try {
     page = await browser.newPage()
+    externalSignal?.addEventListener('abort', closePage, { once: true })
+    if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
     // 'domcontentloaded' instead of 'networkidle': heartbeat-polling sites
     // (qq-news, juejin) never go idle and would time out at 30s. Instead wait
     // for the DOM to stabilize (content stops growing) up to 10s — SPA paint
@@ -72,7 +95,8 @@ export async function renderPage(url, externalSignal) {
         .evaluate(() => (document.body ? document.body.innerHTML.length : 0))
         .catch(() => -1)
       if (Date.now() - t0 > 10000) break
-      if (len === -1 && ++evalFails >= 2) break
+      evalFails = len === -1 ? evalFails + 1 : 0
+      if (evalFails >= 2) break
       // stop once two consecutive reads agree — including empty bodies (a
       // blank page should not burn the full 10s poll)
       if (len === prevLen && prevLen >= 0) break
@@ -93,25 +117,33 @@ export async function renderPage(url, externalSignal) {
         if (!looksLikeChallenge(html)) break
       }
     }
+    if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
     const finalUrl = page.url()
     return { html, finalUrl }
   } catch (e) {
     if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
-    return { error: `Render failed: ${e.message}` }
+    return { error: renderError(e) }
   } finally {
-    if (page) await page.close().catch(() => {})
+    externalSignal?.removeEventListener('abort', closePage)
+    await closePage()
   }
 }
 
 // Called on plugin unload (temporal composability): release the browser.
 export async function closeBrowser() {
-  if (!browserPromise) return
   const p = browserPromise
   browserPromise = null
-  try {
-    const b = await p
-    await b.close().catch(() => {})
-  } catch {
-    // launch never completed — nothing to close
+  if (p) {
+    const closing = (async () => {
+      try {
+        const b = await p
+        await b.close()
+      } catch {
+        // A failed launch or an already closed browser needs no further cleanup.
+      }
+    })()
+    closingBrowsers.add(closing)
+    closing.finally(() => closingBrowsers.delete(closing))
   }
+  await Promise.all(closingBrowsers)
 }