dsh-read-url 1.8.1 → 1.9.0

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [1.9.0] - 2026-09-19
+
+### 新增
+
+- 新增仅 profile 可配置的 `directFetchOrigins`，默认空列表。按协议、域名和端口精确匹配，命中后直接读取，显式跳过宿主 provider 及其 SSRF 检查；未命中的初始 URL 维持原有行为，不因任意 provider 错误自动直连。
+- 例外链的 HTTP 重定向、meta-refresh、分页及爬取子页面均受完整列表约束；HTTP 跳转逐跳校验，每次尝试最多 5 跳并检测循环。例外不进入 curl 代理竞速或 SPA 渲染，失败边界/静态读取提示保留到工具输出。
+- 成功和失败缓存按规范化后的完整配置隔离；非法配置明确报错，工具参数不允许扩大直连范围。双语 README 增补配置方法及信任边界。
+
+### 验证
+
+- 322 个零依赖单元断言、15 个真实 Chromium SPA 断言、53 项对抗探针通过；新增 22 条直连策略检查与 3 条 SPA 限制检查。
+- 受控 HTTP 服务与 provider 桩覆盖默认拒绝、显式直连、重定向、分页、四工具、缓存隔离、取消、超时及正文上限。
+- 152 站实网回归：105 OK / 23 THIN+EMPTY / 24 ERR / 0 THREW，OK 站零噪声。Vue、React、酷狗三个已知 NOISE 站同输入 v1.8.1 对照的 text/Markdown 全文与 800 字符窗口逐字一致；ERR 均为网络、反爬、无效地址或预期不支持的类型。
+- 四工具 description 共 1,159 字符，参数 schema 共 1,876 字符；零运行时依赖，新增正则有界。
+
 ## [1.8.1] - 2026-09-19
 
 ### 修复

package/README.md

@@ -47,7 +47,7 @@ DSH agents can search (getting links and snippets) but lack the step of "reading
 
 Implemented per official docs (`docs/capability-seams.md`, `docs/cordis-primer.md`, `docs/tool-execution-pipeline.md`):
 
-1. **Web access via the `ctx.web` capability seam** — use `ctx.web.fetch({ url }, signal)` and its decoded `body.kind/content` result for reads, redirects and crawl requests; fall back to direct retrieval when the service or provider is unavailable. Provider HTTP errors and policy decisions remain errors;
+1. **Web access via the `ctx.web` capability seam** — use `ctx.web.fetch({ url }, signal)` and its decoded `body.kind/content` result for reads, redirects and crawl requests; fall back to direct retrieval when the service or provider is unavailable. Provider HTTP errors and policy decisions remain errors. The optional `directFetchOrigins` profile setting selects explicit direct retrieval for trusted origins;
 2. **Reversible side effects** — the session cache is registered under `ctx.effect`, auto-cleared on plugin unload (temporal composability);
 3. **Cooperative tool-call timeout** — `ToolDefinition.timeoutMs` declares the budget, `execute(args, exec)` forwards `exec.signal` to fetch; the timeout policy is enforced by the pipeline, never exposed to the model;
 4. **Model-facing simplicity** — render emits compact text (`title:` header + body); the model consumes it directly with no JSON parsing. Defaults are the most token-efficient; structured output is opt-in;
@@ -181,6 +181,27 @@ Override via the profile's `cordis.patch.yml` (defaults in the plugin's own `cor
 
 Values are coerced and clamped to sane ranges at load — quoted numbers in YAML work, garbage falls back to defaults.
 
+#### Explicit direct origins (v1.9.0)
+
+`directFetchOrigins` defaults to `[]`. For a trusted local reverse proxy (for example, a GitHub hostname mapped by your hosts file), explicitly list the origins in the plugin's **profile configuration**, then reload the plugin/profile:
+
+```yaml
+- id: dsh-read-url
+  config:
+    directFetchOrigins:
+      - https://github.com
+      - https://api.github.com
+      - https://raw.githubusercontent.com
+```
+
+Listed origins use direct fetch immediately, bypassing the host web provider **and its SSRF checks**. Add only origins you trust; this also permits their DNS/hosts mappings to local or private addresses. The option is not a tool argument. Unlisted initial URLs keep the existing provider selection and fallback behavior; a provider refusal does not enable this exception automatically.
+
+Matching is exact after URL normalization: scheme, hostname and port must match; subdomains are separate. At most 32 entries of 2,048 characters each are accepted. Only an HTTP(S) origin with an optional trailing `/` is valid: credentials, paths, queries, fragments, wildcards and ambiguous syntax cause a configuration error. Unlike numeric options, invalid entries do not fall back to defaults.
+
+Once a request selects explicit direct retrieval, every subsequent HTTP redirect, meta-refresh, pagination request and descendant crawl stays within the list. Redirects handled internally by the provider remain on the provider path. Direct HTTP requests follow at most 5 redirects per attempt and stop loops early. An outside target is stopped before requesting it; refresh/pagination blocked by this policy retain the readable static result with a hint. Explicit direct reads use neither the plugin's curl proxy race nor SPA rendering, preventing those mechanisms from widening the exception. When `spaRender` is enabled and `read_url` or `read_url_links` would otherwise render a script-only page, they return a static-reading hint. Cancellation, response-size limits and one bounded Retry-After retry still apply. All four tools share the policy; success and failure caches are isolated by the complete normalized list, including after changing or clearing it.
+
+Controlled HTTP servers and provider fixtures cover default refusals, explicit routing, redirects, pagination, all four tools, cache isolation, cancellation, timeouts and response-size limits.
+
 ### Output (compact)
 
 ```json
@@ -287,6 +308,13 @@ Plus a **15-item DSH acceptance round** (a real agent driving every read_url too
 - SPA cancellation closes the active page; plugin disposal awaits browser shutdown. Valid success-cache entries take precedence over concurrent failures. Crawl filtering uses path boundaries. Charset and Markdown attributes use bounded parsing.
 - Fixed model input cost remains **1,159 description characters / 1,876 schema characters**, within the 1,250 / 2,000 budgets.
 
+### v1.9.0 validation (2026-09-19)
+
+- **322 zero-dependency unit assertions, 15 real Chromium SPA assertions and 53 adversarial probes** passed. The suite includes 22 direct-origin policy checks and 3 SPA restriction checks.
+- The 152-site sweep returned **105 OK / 23 THIN+EMPTY / 24 ERR / 0 THREW**, with no noise in OK results.
+- Vue, React and Kugou's known NOISE cases matched v1.8.1 exactly using identical HTML and optional dependencies, in both text/Markdown and full/800-character outputs. Errors were network, anti-bot, invalid-address or expected content-type boundaries.
+- All four tools enforce the explicit origin policy, including redirect/crawl boundaries, cache isolation and visible static-reading hints. Model input remains **1,159 description characters / 1,876 schema characters**, within the 1,250 / 2,000 budgets.
+
 ## Roadmap
 
 - [x] Single-page continuation (`offset` parameter)

package/README.zh.md

@@ -47,7 +47,7 @@ DSH 的 Agent 能搜索（返回链接和片段），但缺"把 URL 读成干净
 
 按官方文档实现（`docs/capability-seams.md`、`docs/cordis-primer.md`、`docs/tool-execution-pipeline.md`）：
 
-1. **网络访问走 `ctx.web` 能力缝**——读取、跳转与爬取使用 `ctx.web.fetch({ url }, signal)` 及解码后的 `body.kind/content`；服务或提供方不可用时回退直连。提供方 HTTP 错误与策略拒绝保留为错误；
+1. **网络访问走 `ctx.web` 能力缝**——读取、跳转与爬取使用 `ctx.web.fetch({ url }, signal)` 及解码后的 `body.kind/content`；服务或提供方不可用时回退直连。提供方 HTTP 错误与策略拒绝保留为错误。可选的 profile 配置 `directFetchOrigins` 为可信来源选择显式直连；
 2. **可逆副作用**——会话缓存注册在 `ctx.effect` 下，插件卸载即自动清理（时间可组合性）；
 3. **协作式工具调用超时**——`ToolDefinition.timeoutMs` 声明预算，`execute(args, exec)` 把 `exec.signal` 转发给 fetch，超时策略由管线强制执行，不把超时暴露给模型；
 4. **模型视角精简**——render 输出紧凑文本（`title:` 头部 + 正文），模型直接消费，无需解析 JSON；默认参数最省 token，结构化能力按需开启；
@@ -181,6 +181,27 @@ chars 800+800/12398 · cached
 
 配置在加载时统一强转数字并钳制到合理范围——YAML 里加引号的数字也能用，非法值回退默认。
 
+#### 显式直连来源（v1.9.0）
+
+`directFetchOrigins` 默认为 `[]`。对于可信的本地反向代理（例如通过 hosts 文件映射 GitHub 域名），在插件的 **profile 配置**中显式列出来源，再重载插件/profile：
+
+```yaml
+- id: dsh-read-url
+  config:
+    directFetchOrigins:
+      - https://github.com
+      - https://api.github.com
+      - https://raw.githubusercontent.com
+```
+
+命中的来源立即直连，跳过宿主 web provider **及其 SSRF 检查**。仅添加你信任的来源；其 DNS/hosts 映射到本机或私有地址也会被允许。此选项不开放为工具参数。未列出的初始 URL 保留既有 provider 选择与回退行为；provider 拒绝不会自动启用此例外。
+
+按 URL 规范化后的协议、域名、端口精确匹配，子域名需单独配置。最多 32 项，每项不超过 2,048 字符。仅接受 HTTP(S) origin，可带一个结尾 `/`；凭据、路径、查询、片段、通配符和歧义写法都会触发配置错误。此选项与数字选项不同，非法项不会静默回退默认值。
+
+请求选用显式直连后，后续 HTTP 重定向、meta-refresh、分页及爬取子页面均限制在列表内；provider 内部处理的跳转仍属于 provider 路径。直连每次尝试最多跟随 5 次 HTTP 重定向，遇循环提前停止。越界目标在发请求前停止；被此策略阻止的刷新跳转或分页保留可读的静态结果并附提示。显式直连不使用插件的 curl 代理竞速或 SPA 渲染，防止它们扩大例外范围；启用 `spaRender` 且 `read_url` 或 `read_url_links` 原本需要渲染脚本空壳时，会提示仅能静态读取。取消、正文大小限制及一次有界 Retry-After 重试继续生效。四个工具统一遵守此策略；成功与失败缓存按完整规范化列表隔离，修改或清空列表后不会误用旧策略的结果。
+
+受控 HTTP 服务与 provider 桩覆盖默认拒绝、显式直连、重定向、分页、四工具、缓存隔离、取消、超时及正文上限。
+
 ### 输出结构（紧凑）
 
 ```json
@@ -287,6 +308,13 @@ v1.4.0 复验（2026-08-28，代理环境）：python-docs stdtypes.html 锚点
 - SPA 取消关闭当前页面，插件卸载等待浏览器关闭；有效成功缓存优先于并发失败；爬取过滤遵循路径边界；编码与 Markdown 属性使用有界解析。
 - 模型固定输入成本仍为 **description 1,159 字符 / schema 1,876 字符**，低于 1,250 / 2,000 预算。
 
+### v1.9.0 验证（2026-09-19）
+
+- **322 个零依赖单元断言、15 个真实 Chromium SPA 断言、53 项对抗探针**通过；包括 22 条直连来源策略检查与 3 条 SPA 限制检查。
+- 152 站实网结果为 **105 OK / 23 THIN+EMPTY / 24 ERR / 0 THREW**，OK 站零噪声。
+- Vue、React、酷狗三个已知 NOISE 站采用同份 HTML 与相同可选依赖对照 v1.8.1，text/Markdown 全文及 800 字符窗口逐字一致；ERR 均为网络、反爬、无效地址或预期不支持的类型。
+- 四个工具统一遵守显式来源策略，覆盖跳转与爬取边界、缓存隔离及可见的静态读取提示。模型固定输入成本为 **description 1,159 字符 / schema 1,876 字符**，低于 1,250 / 2,000 预算。
+
 ## Roadmap
 
 - [x] 单页多段续读（`offset` 参数）

package/cordis.patch.yml

@@ -11,4 +11,5 @@
         spaRender: true
         paginate: true
         paginateMax: 3
+        directFetchOrigins: []
         userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'

package/index.js

@@ -2,7 +2,8 @@
 // Zero external dependencies (Node 20+ built-ins only).
 // Aligned with DSH architecture:
 //   - all network access goes through the ctx.web capability seam (official
-//     docs/capability-seams.md), falling back to global fetch when absent
+//     docs/capability-seams.md), except explicit directFetchOrigins; falls
+//     back to global fetch when absent
 //   - cache is registered under ctx.effect, so unload fully reverts it
 //     (temporal composability, docs/cordis-primer.md)
 //   - cooperative tool-call timeout via ToolDefinition.timeoutMs + exec.signal
@@ -33,7 +34,8 @@ const DEFAULTS = {
   cacheMax: 32,
   spaRender: true,
   paginate: true,
-  paginateMax: 3,
+  paginateMax: 3,
+  directFetchOrigins: [],
   userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36',
 }
 
@@ -51,7 +53,52 @@ function cacheStore(key, full, cacheMax) {
   cache.set(key, { time: Date.now(), full })
   failCache.delete(key)
 }
-const decoders = new Map()
+const decoders = new Map()
+
+// Owner configuration only: tool arguments cannot widen this trust boundary.
+// Preserve scheme/port, reject ambiguous origin syntax, and fail closed on
+// invalid configuration instead of silently accepting a partial allowlist.
+function directOrigins(values = []) {
+  const invalid = () => new Error('Invalid directFetchOrigins: expected at most 32 exact http(s) origins without credentials, paths, queries, fragments or wildcards')
+  if (!Array.isArray(values) || values.length > 32) throw invalid()
+  const origins = new Set()
+  for (const value of values) {
+    if (typeof value !== 'string' || value.length > 2048) throw invalid()
+    const raw = value.trim()
+    let u
+    try { u = new URL(raw) } catch { throw invalid() }
+    if (!raw.toLowerCase().startsWith(`${u.protocol}//`) ||
+        !['http:', 'https:'].includes(u.protocol) || u.username || u.password ||
+        u.pathname !== '/' || raw.includes('?') || raw.includes('#') ||
+        raw.includes('*') || raw.includes('\\') || raw.includes('@') ||
+        [...raw].some(c => c.charCodeAt(0) <= 32 || c.charCodeAt(0) === 127)) throw invalid()
+    // Do not accept URL parser repairs such as /a/.. or percent escapes in
+    // hostnames. Only an authority and one optional trailing slash are valid.
+    const authority = raw.slice(raw.indexOf('://') + 3)
+    if (authority.includes('%') || authority.replace(/\/$/, '').includes('/')) throw invalid()
+    origins.add(u.origin)
+  }
+  return [...origins].sort()
+}
+
+function createRoute(cfg) {
+  const origins = directOrigins(cfg.directFetchOrigins)
+  return { origins: new Set(origins), key: JSON.stringify(origins), restricted: false, hint: '' }
+}
+
+function matchesDirectOrigin(url, origins) {
+  try { return origins.has(new URL(url).origin) } catch { return false }
+}
+
+function allowedDirectUrl(url, origins) {
+  try {
+    const u = new URL(url)
+    return !u.username && !u.password && origins.has(u.origin) && ['http:', 'https:'].includes(u.protocol)
+  } catch { return false }
+}
+
+const DIRECT_BOUNDARY_ERROR = 'Direct fetch stopped: target is outside directFetchOrigins or contains credentials'
+const DIRECT_SPA_HINT = '已配置直连仅支持静态读取，未启用浏览器渲染'
 
 function getDecoder(enc) {
   let d = decoders.get(enc)
@@ -124,13 +171,31 @@ function retryAfterMs(res) {
 }
 
 // Single fetch attempt. Never throws; maps abort/timeout/network to { error }.
-async function directFetchOnce(url, signal, cfg) {
-  try {
-    const res = await fetch(url, {
-      redirect: 'follow',
-      signal,
-      headers: { 'user-agent': cfg.userAgent, accept: 'text/html,application/xhtml+xml,*/*;q=0.8' },
-    })
+async function directFetchOnce(url, signal, cfg, origins) {
+  try {
+    let currentUrl = url
+    let res
+    const seen = new Set()
+    for (let hops = 0; ; hops++) {
+      if (signal && signal.aborted) return { error: 'cancelled' }
+      if (origins && !allowedDirectUrl(currentUrl, origins)) return { error: DIRECT_BOUNDARY_ERROR }
+      if (origins) {
+        const normalized = normalizeUrl(currentUrl)
+        if (seen.has(normalized)) return { error: 'Direct fetch redirect loop' }
+        seen.add(normalized)
+      }
+      res = await fetch(currentUrl, {
+        redirect: origins ? 'manual' : 'follow',
+        signal,
+        headers: { 'user-agent': cfg.userAgent, accept: 'text/html,application/xhtml+xml,*/*;q=0.8' },
+      })
+      if (!origins || ![301, 302, 303, 307, 308].includes(res.status)) break
+      if (res.body) res.body.cancel().catch(() => {})
+      const location = res.headers.get('location')
+      if (!location) return { error: `HTTP ${res.status} without Location` }
+      if (hops >= 5) return { error: 'Direct fetch exceeded 5 redirects' }
+      currentUrl = new URL(location, currentUrl).href
+    }
     // Early-return paths must cancel the body stream, or the connection is
     // held open undrained and cannot return to the keep-alive pool.
     const discardBody = () => { if (res.body) res.body.cancel().catch(() => {}) }
@@ -150,7 +215,7 @@ async function directFetchOnce(url, signal, cfg) {
       discardBody()
       return { error: `Unsupported content-type: ${contentType.split(';')[0]}` }
     }
-    if (!res.body) return { buffer: Buffer.alloc(0), contentType, finalUrl: res.url }
+    if (!res.body) return { buffer: Buffer.alloc(0), contentType, finalUrl: res.url || currentUrl }
     const chunks = []
     let size = 0
     for await (const chunk of res.body) {
@@ -163,7 +228,7 @@ async function directFetchOnce(url, signal, cfg) {
     if (!contentType && looksBinary(buffer)) {
       return { error: 'Unsupported content-type (no header, binary body)' }
     }
-    return { buffer, contentType, finalUrl: res.url }
+    return { buffer, contentType, finalUrl: res.url || currentUrl }
   } catch (e) {
     if (e.name === 'AbortError' || e.name === 'TimeoutError') {
       return { error: `Timeout after ${cfg.timeoutMs}ms or cancelled` }
@@ -180,15 +245,15 @@ async function directFetchOnce(url, signal, cfg) {
 // Exported for tests: the double-throttle path must always produce an { error }
 // shape — a second `{ retryAfterMs }` would otherwise leak past the race
 // filter into decodeBuffer as a bufferless "success" (bare TypeError there).
-export async function directFetch(url, signal, cfg) {
-  const first = await directFetchOnce(url, signal, cfg)
+export async function directFetch(url, signal, cfg, origins) {
+  const first = await directFetchOnce(url, signal, cfg, origins)
   if (first && first.retryAfterMs !== undefined && !(signal && signal.aborted)) {
     await new Promise((resolve) => {
       const t = setTimeout(resolve, first.retryAfterMs)
       if (signal) signal.addEventListener('abort', () => { clearTimeout(t); resolve() }, { once: true })
     })
     if (signal && signal.aborted) return { error: 'cancelled' }
-    const second = await directFetchOnce(url, signal, cfg)
+    const second = await directFetchOnce(url, signal, cfg, origins)
     // The retry budget is spent — report the rate limit instead of returning
     // a shapeless { retryAfterMs } (no buffer, no error) to the caller.
     if (second && second.retryAfterMs !== undefined) {
@@ -309,6 +374,27 @@ async function fetchViaWebSeam(ctx, url, externalSignal, cfg) {
     return { error: `Web provider failed: ${String((e && e.message) || e).slice(0, 160)}` }
   }
 }
+
+// Every fetch in a tool call passes here. Once a chain enters an explicit
+// direct origin it stays restricted, including meta refresh and pagination.
+// Never send an exception through curl -L or a browser: both may make further
+// requests outside this origin list. Unlisted initial URLs retain host policy.
+async function fetchResource(ctx, url, externalSignal, cfg, route) {
+  if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+  if (route.restricted || matchesDirectOrigin(url, route.origins)) {
+    route.restricted = true
+    if (!allowedDirectUrl(url, route.origins)) {
+      route.hint = DIRECT_BOUNDARY_ERROR
+      return { error: DIRECT_BOUNDARY_ERROR }
+    }
+    const timeout = AbortSignal.timeout(cfg.timeoutMs)
+    const signal = externalSignal ? AbortSignal.any([externalSignal, timeout]) : timeout
+    const page = await directFetch(url, signal, cfg, route.origins)
+    if (page.error) route.hint = page.error
+    return page
+  }
+  return await fetchViaWebSeam(ctx, url, externalSignal, cfg) || await fetchPage(url, externalSignal, cfg)
+}
 
 // Decode common HTML entities in already-stripped text. Runs AFTER tag
 // stripping so escaped tags stay visible to the stripper and only remaining
@@ -1567,7 +1653,7 @@ export function metaRefreshTarget(html, baseUrl) {
 // cache key is the ORIGINAL url, and a shell re-request should re-follow
 // rather than serve a foreign page.
 const MAX_META_REFRESH_HOPS = 3
-async function followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx) {
+async function followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx, route) {
   let cur = html
   let curUrl = finalUrl
   let curCharset = '' // set only when a hop actually happened
@@ -1586,7 +1672,7 @@ async function followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx) {
     const norm = normalizeUrl(target)
     if (!norm || seen.has(norm)) break
     seen.add(norm)
-    const page = await fetchViaWebSeam(ctx, target, hopSignal, cfg) || await fetchPage(target, hopSignal, cfg)
+    const page = await fetchResource(ctx, target, hopSignal, cfg, route)
     if (page.error) break
     const ct = page.kind === 'html' ? 'text/html' : (page.contentType || '').split(';')[0].toLowerCase().trim()
     if (!/html|xhtml/.test(ct)) break
@@ -1603,6 +1689,8 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   const url = String((args && args.url) || '').trim()
   if (!/^https?:\/\//i.test(url)) return { error: 'Only http/https URLs are supported' }
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
+  let route
+  try { route = createRoute(cfg) } catch (e) { return { error: e.message } }
   const maxChars = Math.max(500, Math.min(20000, Number(args.maxChars) || cfg.maxChars))
   const mode = args.mode === 'markdown' ? 'markdown' : 'text'
   const offset = Math.max(0, Number(args.offset) || 0)
@@ -1632,7 +1720,9 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   } catch {
     /* keep bare url */
   }
-  const cacheKey = `${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
+  // Include the complete policy: an unlisted entry may reach a listed origin
+  // through refresh/pagination, and must not reuse content after revocation.
+  const cacheKey = `${route.key}|${cacheUrl}|${mode}|${args.includeLinks === true ? 'links' : 'no-links'}`
   const hit = cache.get(cacheKey)
   if (hit && Date.now() - hit.time < cfg.cacheTtlMs) {
     // Valid content takes precedence over a concurrent fetch failure.
@@ -1700,9 +1790,9 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     }
   }
   const dispatch = async () => {
-    const viaSeam = await fetchViaWebSeam(ctx, url, externalSignal, cfg)
-    const page = viaSeam || await fetchPage(url, externalSignal, cfg)
+    const page = await fetchResource(ctx, url, externalSignal, cfg, route)
     if (page.error) return { error: page.error }
+    const viaSeam = typeof page.html === 'string'
     const ct = viaSeam ? (page.kind === 'text' ? 'text/plain' : 'text/html') : (page.contentType || '').split(';')[0].toLowerCase().trim()
     const decoded = viaSeam ? { text: page.html, charset: 'provider-decoded' } : decodeBuffer(page.buffer, page.contentType)
     const meta = { finalUrl: page.finalUrl || url, sourceTruncated: page.sourceTruncated === true }
@@ -1782,7 +1872,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   // follow immediately, BEFORE extraction and SPA rendering — a shell that
   // redirects needs no headless round-trip, and extracting the stub is wasted
   // work. Fails open on any fetch problem; the follow is hop-bounded.
-  const followed = await followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx)
+  const followed = await followMetaRefresh(html, finalUrl, externalSignal, cfg, ctx, route)
   if (externalSignal && externalSignal.aborted) return { error: 'cancelled' }
   html = followed.html
   finalUrl = followed.finalUrl
@@ -1826,7 +1916,10 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
   let renderedHtml = null
   const needsRender = !extracted.text || extracted.text.length < 200
   const scriptShell = !extracted.text && /<script[\s>]/i.test(html)
-  if (cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
+  if (route.restricted && cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
+    spaHint = [spaHint, DIRECT_SPA_HINT].filter(Boolean).join('；')
+  }
+  if (!route.restricted && cfg.spaRender !== false && needsRender && (looksLikeSpa(html) || scriptShell)) {
     const rr = await renderPage(finalUrl || url, externalSignal)
     if (rr.html) {
       renderedHtml = rr.html
@@ -1882,7 +1975,7 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     const nu = normalizeUrl(nextUrl)
     if (!nu || seen.has(nu)) break
     seen.add(nu)
-    const page = await fetchViaWebSeam(ctx, nextUrl, externalSignal, cfg) || await fetchPage(nextUrl, externalSignal, cfg)
+    const page = await fetchResource(ctx, nextUrl, externalSignal, cfg, route)
     if (page.error) break
     const pageFinal = normalizeUrl(page.finalUrl || nextUrl)
     if (!pageFinal || !sameHost(pageFinal, startHost) || (pageFinal !== nu && seen.has(pageFinal))) break
@@ -1898,7 +1991,9 @@ export async function readUrl(args, ctx, externalSignal, cfg = DEFAULTS) {
     nextUrl = findNextLink(nextHtml, page.finalUrl || nextUrl)
   }
 
-  // A near-empty body (login wall, anti-bot shell) with no render hint yet
+  if (route.hint) spaHint = [spaHint, route.hint].filter(Boolean).join('；')
+
+  // A near-empty body (login wall, anti-bot shell) with no render hint yet
   // still looks like "the page content" to the model — one constant-cost line
   // tells it not to keep offset-reading a 40-char page.
   if (!spaHint && extracted.text && extracted.text.length < 60) {
@@ -1992,8 +2087,9 @@ function renderLinks(value) {
   if (typeof value === 'string') return value
   const r = value || {}
   if (r.error) return `Error: ${r.error}`
-  if (!Array.isArray(r.links) || r.links.length === 0) return `No links found on ${r.url}`
-  const lines = [`${r.count} link(s) on ${r.url}:`, UNTRUSTED_NOTICE]
+  if (!Array.isArray(r.links) || r.links.length === 0) return `No links found on ${r.url}${r.spaHint ? `\n提示: ${r.spaHint}` : ''}`
+  const lines = [`${r.count} link(s) on ${r.url}:`, UNTRUSTED_NOTICE]
+  if (r.spaHint) lines.push(`提示: ${r.spaHint}`)
   for (const l of r.links) lines.push(`- ${l.title || l.url} — ${l.url}`)
   return lines.join('\n')
 }
@@ -2033,14 +2129,15 @@ function readLinksTool(ctx, cfg) {
     async execute(args, exec) {
       const url = String((args && args.url) || '').trim()
       if (!/^https?:\/\//i.test(url)) return { error: 'Only http/https URLs are supported' }
-      const limit = Math.max(1, Math.min(50, Number(args.limit) || cfg.maxLinks))
-      const page = await fetchViaWebSeam(ctx, url, exec && exec.signal, cfg) || await fetchPage(url, exec && exec.signal, cfg)
+      const limit = Math.max(1, Math.min(50, Number(args.limit) || cfg.maxLinks))
+      const route = createRoute(cfg)
+      const page = await fetchResource(ctx, url, exec && exec.signal, cfg, route)
       if (page.error) return { error: page.error }
       let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
       let finalUrl = page.finalUrl || url
       // Same meta-refresh shell following as read_url: a link hop serves a
       // stub whose real links live at the target. Fails open, hop-bounded.
-      const followed = await followMetaRefresh(html, finalUrl, exec && exec.signal, cfg, ctx)
+      const followed = await followMetaRefresh(html, finalUrl, exec && exec.signal, cfg, ctx, route)
       html = followed.html
       finalUrl = followed.finalUrl
       let links = extractLinks(html, limit, finalUrl)
@@ -2048,7 +2145,8 @@ function readLinksTool(ctx, cfg) {
       // render it and re-extract when the static result looks empty. Any
       // <script> counts as renderable when there are no links at all — the
       // static shell may be a JS-redirect (same rationale as read_url).
-      if (links.length < 3 && cfg.spaRender !== false && (looksLikeSpa(html) || (links.length === 0 && /<script[\s>]/i.test(html)))) {
+      const needsRender = links.length < 3 && cfg.spaRender !== false && (looksLikeSpa(html) || (links.length === 0 && /<script[\s>]/i.test(html)))
+      if (!route.restricted && needsRender) {
         const rr = await renderPage(finalUrl || url, exec && exec.signal)
         // A challenge interstitial's links are the challenge's own scripts —
         // never an improvement over whatever the static HTML offered.
@@ -2060,7 +2158,8 @@ function readLinksTool(ctx, cfg) {
           }
         }
       }
-      return { url: finalUrl, count: links.length, links }
+      const spaHint = [route.hint, route.restricted && needsRender ? DIRECT_SPA_HINT : ''].filter(Boolean).join('；')
+      return { url: finalUrl, count: links.length, links, ...(spaHint ? { spaHint } : {}) }
     },
   }
 }
@@ -2091,7 +2190,8 @@ function renderBatch(value) {
       continue
     }
     const head = p.title || p.url
-    lines.push('', `--- ${head} (${p.chars} 字符${p.cached ? ' · cached' : ''}) ---`, p.text || '(无可读内容)')
+    lines.push('', `--- ${head} (${p.chars} 字符${p.cached ? ' · cached' : ''}) ---`, p.text || '(无可读内容)')
+    if (p.spaHint) lines.push(`提示: ${p.spaHint}`)
     if (Array.isArray(p.links) && p.links.length) {
       const ls = p.links.map((l) => l.title + ' — ' + l.url)
       lines.push(`links: ${ls.slice(0, 6).join(' | ')}${ls.length > 6 ? ` …共${ls.length}个` : ''}`)
@@ -2181,7 +2281,8 @@ function readUrlBatchTool(ctx, cfg) {
             cached: p.cached === true,
             text: p.text,
           }
-          if (Array.isArray(p.links) && p.links.length) out.links = p.links
+          if (Array.isArray(p.links) && p.links.length) out.links = p.links
+          if (p.spaHint) out.spaHint = p.spaHint
           return out
         }),
       }
@@ -2226,7 +2327,8 @@ function isNoiseUrl(url) {
 // failure is recorded and does not abort the crawl. No SPA rendering here —
 // that is read_url's job; crawling favors speed and breadth.
 async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
-  const { maxPages, maxDepth, includeContent, perMax } = opts
+  const { maxPages, maxDepth, includeContent, perMax } = opts
+  const policy = createRoute(cfg)
   let host = hostOf(entryUrl)
   const visited = new Set()
   const resolved = new Set()
@@ -2240,12 +2342,13 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
     // Never overshoot the page budget within a batch round.
     const batch = queue.splice(0, Math.min(2, maxPages - attempted, queue.length))
     const results = await Promise.all(
-      batch.map(async ({ url, depth }) => {
+      batch.map(async ({ url, depth, restricted = false }) => {
         const norm = normalizeUrl(url)
         if (!norm || visited.has(norm)) return null
         visited.add(norm)
         attempted++
-        const page = await fetchViaWebSeam(ctx, url, externalSignal, cfg) || await fetchPage(url, externalSignal, cfg)
+        const route = { ...policy, restricted }
+        const page = await fetchResource(ctx, url, externalSignal, cfg, route)
         if (page.error) return { url, depth, error: page.error }
         let html = typeof page.html === 'string' ? page.html : decodeBuffer(page.buffer, page.contentType).text
         let pageUrl = page.finalUrl || url
@@ -2256,7 +2359,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         // be crawled AS the page (title-less hop text pollutes results and the
         // real content is never reached). Fails open — fetch problems keep the
         // stub's own HTML.
-        const followed = await followMetaRefresh(html, pageUrl, externalSignal, cfg, ctx)
+        const followed = await followMetaRefresh(html, pageUrl, externalSignal, cfg, ctx, route)
         html = followed.html
         pageUrl = followed.finalUrl
         if (depth === 0) host = hostOf(pageUrl)
@@ -2271,7 +2374,9 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
           title: ex.title || '',
           chars: ex.text.length,
           text: includeContent ? smartTruncate(ex.text, perMax).text : undefined,
-          links,
+          links,
+          route,
+          ...(route.hint ? { spaHint: route.hint } : {}),
         }
       }),
     )
@@ -2285,7 +2390,8 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
       if (!finalNorm || resolved.has(finalNorm)) continue
       resolved.add(finalNorm)
       visited.add(finalNorm)
-      pages.push(r)
+      const { route, ...publicPage } = r
+      pages.push(publicPage)
       if (r.depth + 1 <= maxDepth) {
         // Queue cap: a huge site (millions of links) would otherwise grow the
         // queue unboundedly while maxPages bounds only what we *process*.
@@ -2293,7 +2399,7 @@ async function crawlSite(entryUrl, cfg, opts, externalSignal, ctx) {
         for (const l of r.links) {
           if (!sameHost(l.url, host) || isNoiseUrl(l.url)) continue
           const n = normalizeUrl(l.url)
-          if (n && !visited.has(n)) queue.push({ url: l.url, depth: r.depth + 1 })
+          if (n && !visited.has(n)) queue.push({ url: l.url, depth: r.depth + 1, restricted: route.restricted })
         }
       }
     }
@@ -2309,7 +2415,8 @@ function renderSite(value) {
   for (const p of v.pages) {
     const indent = '  '.repeat(p.depth)
     const head = p.title || p.url
-    lines.push(`${indent}[${p.depth}] ${head} (${p.chars} 字符)  ${p.url}`)
+    lines.push(`${indent}[${p.depth}] ${head} (${p.chars} 字符)  ${p.url}`)
+    if (p.spaHint) lines.push(`${indent}提示: ${p.spaHint}`)
     if (p.text) lines.push(`${indent}   ${p.text.slice(0, 80)}`)
   }
   for (const f of v.failures) lines.push(`[失败] ${f.url} — ${f.error}`)
@@ -2376,9 +2483,10 @@ function readUrlSiteTool(ctx, cfg) {
         succeeded: pages.length,
         failed: failures.length,
         pages: pages.map((p) => {
-          const o = { url: p.url, depth: p.depth, title: p.title || '', chars: p.chars }
-          if (p.text) o.text = p.text
-          return o
+          const o = { url: p.url, depth: p.depth, title: p.title || '', chars: p.chars }
+          if (p.text) o.text = p.text
+          if (p.spaHint) o.spaHint = p.spaHint
+          return o
         }),
         failures,
       }
@@ -2405,7 +2513,8 @@ export function apply(ctx, config) {
   cfg.paginateMax = clamp(cfg.paginateMax, DEFAULTS.paginateMax, 1, 10)
   if (typeof cfg.userAgent !== 'string' || !cfg.userAgent.trim()) cfg.userAgent = DEFAULTS.userAgent
   if (typeof cfg.spaRender !== 'boolean') cfg.spaRender = DEFAULTS.spaRender
-  if (typeof cfg.paginate !== 'boolean') cfg.paginate = DEFAULTS.paginate
+  if (typeof cfg.paginate !== 'boolean') cfg.paginate = DEFAULTS.paginate
+  cfg.directFetchOrigins = directOrigins(cfg.directFetchOrigins)
 
   // Temporal composability: unload must fully revert side effects.
   ctx.effect(() => () => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-read-url",
-  "version": "1.8.1",
+  "version": "1.9.0",
   "description": "DeepSeek Harness URL reader: fetch any webpage (HTML/JSON/RSS/Atom), auto-detect encoding (UTF-16 BOM/GBK/Big5/Shift-JIS), extract clean main content with auto-pagination, output compact text/Markdown to save tokens. Zero runtime dependencies, no API key, drop-in. Follows meta-refresh shells, reads JSON-LD metadata, honors base href.",
   "type": "module",
   "main": "index.js",