@demigodmode/pi-web-agent 1.11.0 → 1.13.0

package/CHANGELOG.md

@@ -18,6 +18,35 @@ The format is intentionally simple and release-oriented.
 ### Breaking
 - None.
 
+## [1.13.0] - 2026-09-24
+### Added
+- `web_explore` now uses the research question to pick relevant sections from long HTML pages instead of stopping at the first 4,000 characters. It can reach an answer buried later in a page, whether the page came through HTTP, headless browsing, or Firecrawl. Direct `web_fetch` output is unchanged. (#36)
+
+### Changed
+- You.com search and `/web-agent doctor` now use the documented `/v1/search` endpoint. Web and news results appear in the usual search list, with web results first. Result descriptions provide the snippets when present. (#60)
+
+### Fixed
+- Research still excludes recognized bot-check pages when their verification message sits outside the section chosen for the question. That page cannot be used as evidence just because another section looks relevant. (#36)
+- `/web-agent doctor` no longer repeats the default DuckDuckGo and HTTP backends after listing them in the config summary. It still shows checks and warnings for backends that need them. (#64)
+- On pages with several `<article>` sections and no `<main>`, query-based reads now search across the articles. A match in the first article no longer hides an answer in a later one. (#71)
+
+### Breaking
+- None.
+
+## [1.12.0] - 2026-09-17
+### Added
+- A network allow list, under Settings → Backends → Network allow list. web_explore now refuses private and local addresses when the link came from the model or a page it read, so if you genuinely want it to read something on your own network (an internal docs site, a service on localhost), add that range here as a CIDR like `10.0.0.0/24`. Entries are checked when you save, and ones that would allow everything (`0.0.0.0/0`, `::/0`) are rejected, since that would quietly turn the protection off. It is also the fix if every fetch suddenly fails: some proxy apps run in fake-IP mode and make every site look like it lives in `198.18.0.0/15`, and adding that range gets you going again. (#53)
+- A "Trust the upstream proxy to enforce private-address restrictions" setting (`backends.network.trustProxyDns`). With a proxy configured, pi-web-agent normally looks each address up itself and asks the proxy to connect to that exact IP, so the proxy can't send a request somewhere else. Some proxies only accept hostnames, and some networks only resolve names inside the proxy. Turning this on hands the hostname to the proxy instead. It is off by default because it moves the address decision to your proxy, and even when it is on, localhost, private IPs typed straight into a link, and anything your own machine resolves to a private address are still refused. `/web-agent doctor` shows the allow list and this setting. (#53)
+
+### Changed
+- Search and fetch fallback now act on why a backend failed. Rate limits pause that backend for the time the provider asks (capped at 15 minutes), exhausted quotas and bad API keys skip it until settings change, and timeouts or server errors get one retry. An empty result is treated as a real answer instead of a failure, a private-address refusal or an invalid shared proxy setting never falls back to another backend, and answers say when some search backends were unavailable. DuckDuckGo no longer retries on its own. (#55)
+
+### Fixed
+- web_explore no longer fetches private, loopback, or link-local addresses when the link came from the model or a page it read. The check happens where connections are opened, through a local guard proxy, so it also covers redirects, DNS changes, and everything a headless page loads. A web page could previously steer it at cloud metadata endpoints or services on your network. Addresses you configured yourself (search backends, SearXNG, Firecrawl, the proxy) are unaffected. If you do need to reach a private range, add it under Settings → Backends → Network allow list. With an upstream proxy that only accepts hostnames, turn on "Trust the upstream proxy to enforce private-address restrictions". (#53)
+
+### Breaking
+- None.
+
 ## [1.11.0] - 2026-09-16
 ### Added
 - Optional proxy support. Point web_explore at an HTTP/HTTPS proxy and everything outbound goes through it: search, fetch, Firecrawl, the GitHub/PDF/YouTube readers, doctor health checks, and the headless browser. Off unless you set it, and nothing changes if you don't. Set it in Settings → Backends, and keep credentials in `PI_WEB_AGENT_PROXY_USERNAME` / `PI_WEB_AGENT_PROXY_PASSWORD` rather than in the URL. Thanks to @lo-tp for building this. (#50)

package/README.md

@@ -34,6 +34,8 @@ One public tool, `web_explore`, that does bounded web research for Pi: search, f
 - **Six search backends.** DuckDuckGo (keyless default), SearXNG, Brave, You.com, Exa, Tavily.
 - **Optional search fanout.** Query several backends at once, dedupe, and rank pages that more than one provider agreed on to the top. Off by default; flip it to `on` or `auto`.
 - **Honest by default.** Weak, narrow, blocked, or cautionary evidence gets flagged instead of dressed up as confidence.
+- **Safe with untrusted pages.** Links the model picks, or finds on a page it read, can't reach localhost, your LAN, or cloud metadata endpoints. That includes redirects and everything a headless page loads. Addresses you configure yourself aren't affected, and an allow list covers the private ranges you do want.
+- **Fallback that knows why.** Rate-limited or misconfigured backends get skipped for a while, flaky ones get one retry, and answers say when some search backends were unavailable.
 - **Bounded output.** `compact` / `preview` / `verbose` transcript modes.
 - **Zero-config to start.** Runs keyless out of the box (DuckDuckGo search, local browser, the built-in readers). Opt into hosted backends, fallback, search fanout, and per-tool output modes through config when you want more control.
 

package/dist/backends/config.d.ts

@@ -34,17 +34,25 @@ export type FetchBackendConfig = {
 export type HeadlessBackendConfig = {
     provider: 'local-browser';
 };
+export type NetworkConfig = {
+    /** CIDR ranges exempted from the private-address guard (#53). */
+    allowRanges?: string[];
+    /** Trust the upstream proxy to enforce private-address restrictions. */
+    trustProxyDns?: boolean;
+};
 export type BackendConfig = {
     search: SearchBackendConfig;
     fetch: FetchBackendConfig;
     headless: HeadlessBackendConfig;
     proxy?: ProxyConfig;
+    network?: NetworkConfig;
 };
 export type BackendConfigOverride = {
     search?: Partial<SearchBackendConfig>;
     fetch?: Partial<FetchBackendConfig>;
     headless?: Partial<HeadlessBackendConfig>;
     proxy?: ProxyConfig;
+    network?: NetworkConfig;
 };
 export type BackendConfigFile = {
     backends?: {
@@ -70,6 +78,10 @@ export type BackendConfigFile = {
             username?: unknown;
             password?: unknown;
         };
+        network?: {
+            allowRanges?: unknown;
+            trustProxyDns?: unknown;
+        };
     };
 };
 export declare const DEFAULT_BACKEND_CONFIG: BackendConfig;
@@ -88,6 +100,7 @@ export declare function stripProxyCredentials(url: string): string;
  */
 export declare function isValidProxyUrl(url: string): boolean;
 export declare function extractProxyConfig(value: unknown): ProxyConfig | undefined;
+export declare function extractNetworkConfig(value: unknown): NetworkConfig | undefined;
 export declare function usableSearchProviders(search: SearchBackendConfig, env?: NodeJS.ProcessEnv): SearchProviderName[];
 export declare function extractBackendConfigOverride(file: BackendConfigFile | null | undefined): BackendConfigOverride;
 export declare function validateBackendConfig(config: BackendConfig): string[];

package/dist/backends/config.js

@@ -1,3 +1,4 @@
+import { parseCidr } from '../fetch/network-guard.js';
 export const DEFAULT_BACKEND_CONFIG = {
     search: { provider: 'duckduckgo' },
     fetch: { provider: 'http' },
@@ -79,6 +80,19 @@ export function extractProxyConfig(value) {
         config.password = raw.password;
     return config;
 }
+export function extractNetworkConfig(value) {
+    if (!value || typeof value !== 'object')
+        return undefined;
+    const raw = value;
+    const config = {};
+    if (Array.isArray(raw.allowRanges)) {
+        // Coerce non-string entries so validation flags them instead of dropping the list.
+        config.allowRanges = raw.allowRanges.map((entry) => (typeof entry === 'string' ? entry : String(entry)));
+    }
+    if (typeof raw.trustProxyDns === 'boolean')
+        config.trustProxyDns = raw.trustProxyDns;
+    return Object.keys(config).length > 0 ? config : undefined;
+}
 function extractSearxngOptions(value) {
     if (!value || typeof value !== 'object')
         return undefined;
@@ -187,6 +201,10 @@ export function extractBackendConfigOverride(file) {
     if (proxy) {
         override.proxy = proxy;
     }
+    const network = extractNetworkConfig(backends?.network);
+    if (network) {
+        override.network = network;
+    }
     return override;
 }
 export function validateBackendConfig(config) {
@@ -214,6 +232,18 @@ export function validateBackendConfig(config) {
     if (config.fetch.provider === 'firecrawl' && !config.fetch.baseUrl) {
         issues.push('fetch provider firecrawl requires backends.fetch.baseUrl');
     }
+    for (const range of config.network?.allowRanges ?? []) {
+        const cidr = parseCidr(range);
+        if (!cidr) {
+            issues.push(`backends.network.allowRanges entry "${range}" is not a valid CIDR range`);
+        }
+        else if (cidr.prefix === 0) {
+            issues.push(`backends.network.allowRanges entry "${range}" allows every address, which turns the guard off; list specific ranges instead`);
+        }
+    }
+    if (config.network?.trustProxyDns && !config.proxy?.url?.trim()) {
+        issues.push('backends.network.trustProxyDns has no effect without backends.proxy');
+    }
     if (config.search.fallback === 'duckduckgo' && config.search.provider !== 'searxng' && config.search.provider !== 'brave' && config.search.provider !== 'youcom' && config.search.provider !== 'exa' && config.search.provider !== 'tavily') {
         issues.push('search fallback duckduckgo is only supported when search provider is searxng, brave, youcom, exa, or tavily');
     }
@@ -260,6 +290,17 @@ function mergeFetchConfig(current, override) {
     }
     return { ...current, ...override };
 }
+function mergeNetworkConfig(base, layer) {
+    if (!layer)
+        return base;
+    const next = { ...base };
+    // A layer's allow list replaces the lower one outright; trust is set independently.
+    if (layer.allowRanges)
+        next.allowRanges = [...layer.allowRanges];
+    if (layer.trustProxyDns !== undefined)
+        next.trustProxyDns = layer.trustProxyDns;
+    return next;
+}
 export function mergeBackendConfigLayers(...layers) {
     return layers.reduce((merged, layer) => ({
         search: mergeSearchConfig(merged.search, layer?.search),
@@ -269,6 +310,8 @@ export function mergeBackendConfigLayers(...layers) {
             ? layer.proxy.url === ''
                 ? undefined // explicit disable overrides any proxy from lower layers
                 : { ...merged.proxy, ...layer.proxy }
-            : merged.proxy
+            : merged.proxy,
+        // Replace, don't union: a project list is the whole list for that project.
+        network: mergeNetworkConfig(merged.network, layer?.network)
     }), DEFAULT_BACKEND_CONFIG);
 }

package/dist/backends/doctor.js

@@ -1,3 +1,4 @@
+import { normalizeYouComResults, YOUCOM_SEARCH_URL } from '../search/youcom.js';
 function withTimeout(timeoutMs) {
     const controller = new AbortController();
     const timeout = setTimeout(() => controller.abort(), timeoutMs);
@@ -13,7 +14,7 @@ function braveDoctorUrl() {
     return url.toString();
 }
 function youcomDoctorBody() {
-    return JSON.stringify({ query: 'pi-web-agent-doctor', max_results: 1 });
+    return JSON.stringify({ query: 'pi-web-agent-doctor', count: 1 });
 }
 function exaDoctorBody() {
     return JSON.stringify({ query: 'pi-web-agent-doctor', numResults: 1 });
@@ -42,10 +43,7 @@ function firecrawlDoctorBody(options = {}) {
 }
 export async function checkBackendHealth(config, { fetchImpl = fetch, timeoutMs = 3_000 } = {}) {
     const lines = [];
-    if (config.search.provider === 'duckduckgo') {
-        lines.push('search backend: duckduckgo');
-    }
-    else if (config.search.provider === 'brave') {
+    if (config.search.provider === 'brave') {
         const apiKey = process.env.PI_WEB_AGENT_BRAVE_API_KEY;
         if (!apiKey?.trim()) {
             lines.push('search backend: brave warning (missing PI_WEB_AGENT_BRAVE_API_KEY)');
@@ -86,7 +84,7 @@ export async function checkBackendHealth(config, { fetchImpl = fetch, timeoutMs
         else {
             const timeout = withTimeout(timeoutMs);
             try {
-                const response = await fetchImpl('https://api.you.com/v1/agents/search', {
+                const response = await fetchImpl(YOUCOM_SEARCH_URL, {
                     method: 'POST',
                     headers: {
                         Accept: 'application/json',
@@ -100,8 +98,8 @@ export async function checkBackendHealth(config, { fetchImpl = fetch, timeoutMs
                     lines.push(`search backend: youcom warning (HTTP ${response.status})`);
                 }
                 else {
-                    const json = (await response.json());
-                    lines.push(Array.isArray(json.results)
+                    const normalized = normalizeYouComResults(await response.json());
+                    lines.push(normalized && (normalized.rawCount === 0 || normalized.results.length > 0)
                         ? 'search backend: youcom ok'
                         : 'search backend: youcom warning (unexpected response)');
                 }
@@ -186,23 +184,26 @@ export async function checkBackendHealth(config, { fetchImpl = fetch, timeoutMs
             }
         }
     }
-    else if (!config.search.baseUrl) {
-        lines.push('search backend: searxng warning (missing baseUrl)');
-    }
-    else {
-        const timeout = withTimeout(timeoutMs);
-        try {
-            const response = await fetchImpl(searxngDoctorUrl(config.search.baseUrl, config.search.options), { signal: timeout.signal });
-            const json = (await response.json());
-            lines.push(response.ok && Array.isArray(json.results)
-                ? 'search backend: searxng ok'
-                : 'search backend: searxng warning (unexpected response)');
-        }
-        catch (error) {
-            lines.push(`search backend: searxng warning (${message(error)})`);
+    else if (config.search.provider === 'searxng') {
+        const baseUrl = config.search.baseUrl;
+        if (!baseUrl) {
+            lines.push('search backend: searxng warning (missing baseUrl)');
         }
-        finally {
-            timeout.done();
+        else {
+            const timeout = withTimeout(timeoutMs);
+            try {
+                const response = await fetchImpl(searxngDoctorUrl(baseUrl, config.search.options), { signal: timeout.signal });
+                const json = (await response.json());
+                lines.push(response.ok && Array.isArray(json.results)
+                    ? 'search backend: searxng ok'
+                    : 'search backend: searxng warning (unexpected response)');
+            }
+            catch (error) {
+                lines.push(`search backend: searxng warning (${message(error)})`);
+            }
+            finally {
+                timeout.done();
+            }
         }
     }
     if (config.search.fallback) {
@@ -241,13 +242,10 @@ export async function checkBackendHealth(config, { fetchImpl = fetch, timeoutMs
             }
         }
     }
-    if (config.fetch.provider === 'http') {
-        lines.push('fetch backend: http');
-    }
-    else if (!config.fetch.baseUrl) {
+    if (config.fetch.provider !== 'http' && !config.fetch.baseUrl) {
         lines.push('fetch backend: firecrawl warning (missing baseUrl)');
     }
-    else {
+    else if (config.fetch.provider !== 'http') {
         const timeout = withTimeout(timeoutMs);
         try {
             const headers = { 'content-type': 'application/json' };

package/dist/backends/factory.d.ts

@@ -1,24 +1,26 @@
 import { createFirecrawlFetcher } from '../fetch/firecrawl-fetch.js';
+import { type GuardProxy, type GuardProxyOptions } from '../fetch/guard-proxy.js';
+import { type NetworkGuard } from '../fetch/network-guard.js';
 import { createBraveSearchTool } from '../search/brave.js';
 import { createYouComSearchTool } from '../search/youcom.js';
 import { createExaSearchTool } from '../search/exa.js';
 import { createTavilySearchTool } from '../search/tavily.js';
 import { createSearxngSearchTool } from '../search/searxng.js';
+import { type PolicyDeps } from './fallback-policy.js';
+import { type ProviderHealth } from './provider-health.js';
 import { createWebFetchHeadlessTool } from '../tools/web-fetch-headless.js';
 import { createWebFetchTool } from '../tools/web-fetch.js';
 import { createWebSearchTool } from '../tools/web-search.js';
-import type { WebFetchHeadlessResponse, WebFetchResponse, WebSearchResponse } from '../types.js';
+import type { ResearchFetchInput, WebFetchHeadlessResponse, WebFetchResponse, WebSearchResponse } from '../types.js';
 import { type BackendConfig, type ProxyConfig } from './config.js';
 export type BackendSet = {
     search: (input: {
         query: string;
     }) => Promise<WebSearchResponse>;
-    fetchPage: (input: {
-        url: string;
-    }) => Promise<WebFetchResponse>;
-    headlessFetch: (input: {
-        url: string;
-    }) => Promise<WebFetchHeadlessResponse>;
+    fetchPage: (input: ResearchFetchInput) => Promise<WebFetchResponse>;
+    headlessFetch: (input: ResearchFetchInput) => Promise<WebFetchHeadlessResponse>;
+    /** Releases the guard proxy and its agents. Idempotent; never starts the proxy. */
+    close: () => Promise<void>;
 };
 export type BackendFactoryDeps = {
     createDuckDuckGoSearch?: typeof createWebSearchTool;
@@ -31,5 +33,14 @@ export type BackendFactoryDeps = {
     createFirecrawlFetch?: typeof createFirecrawlFetcher;
     createHeadlessFetch?: typeof createWebFetchHeadlessTool;
     createProxyFetch?: (proxy: ProxyConfig) => typeof fetch;
+    networkGuard?: NetworkGuard;
+    /** Test seam: the fetch used for model-chosen URLs, before redirect handling. */
+    createModelFetch?: (guard: NetworkGuard) => typeof fetch;
+    createGuardProxy?: (options: GuardProxyOptions) => Promise<GuardProxy>;
+    providerHealth?: ProviderHealth;
+    /** Test seam for the retry sleep, jitter, and clock. */
+    policy?: Omit<PolicyDeps, 'health'>;
+    /** Test seam for the per-call fanout provider timeout. */
+    fanoutTimeoutMs?: number;
 };
 export declare function createBackendSet(config?: BackendConfig, deps?: BackendFactoryDeps): BackendSet;

package/dist/backends/factory.js

@@ -2,13 +2,19 @@ import { createFirecrawlFetcher } from '../fetch/firecrawl-fetch.js';
 import { createHttpFetcher } from '../fetch/http-fetch.js';
 import { createProxyFetch, resolveProxyCredentials } from '../fetch/proxy-fetch.js';
 import { headlessFetch } from '../fetch/headless-fetch.js';
+import { createGuardedFetch } from '../fetch/guarded-fetch.js';
+import { startGuardProxy } from '../fetch/guard-proxy.js';
+import { createGuardProxyFetch } from '../fetch/guard-proxy-fetch.js';
+import { createNetworkGuard, findGuardError } from '../fetch/network-guard.js';
 import { createBraveSearchTool } from '../search/brave.js';
 import { createYouComSearchTool } from '../search/youcom.js';
 import { fetchDuckDuckGoHtml } from '../search/duckduckgo.js';
 import { createExaSearchTool } from '../search/exa.js';
 import { createTavilySearchTool } from '../search/tavily.js';
 import { createSearxngSearchTool } from '../search/searxng.js';
-import { createFanoutSearch } from '../search/fanout.js';
+import { createFanoutSearch, FANOUT_PROVIDER_TIMEOUT_MS, withCallTimeout } from '../search/fanout.js';
+import { chainSearch, withFetchPolicy, withSearchPolicy } from './fallback-policy.js';
+import { createProviderHealth } from './provider-health.js';
 import { buildFetchPresentation } from '../presentation/fetch-presentation.js';
 import { buildSearchPresentation } from '../presentation/search-presentation.js';
 import { createWebFetchHeadlessTool } from '../tools/web-fetch-headless.js';
@@ -27,7 +33,8 @@ function invalidSearxngSearch() {
             metadata: { backend: 'searxng', cacheHit: false },
             error: {
                 code: 'BACKEND_CONFIG_INVALID',
-                message: 'SearXNG search requires backends.search.baseUrl.'
+                message: 'SearXNG search requires backends.search.baseUrl.',
+                failure: { kind: 'not_configured' }
             }
         };
         return { ...result, presentation: buildSearchPresentation(result) };
@@ -41,47 +48,36 @@ function invalidFirecrawlFetch() {
             metadata: { method: 'firecrawl', cacheHit: false },
             error: {
                 code: 'BACKEND_CONFIG_INVALID',
-                message: 'Firecrawl fetch requires backends.fetch.baseUrl.'
+                message: 'Firecrawl fetch requires backends.fetch.baseUrl.',
+                failure: { kind: 'not_configured' }
             }
         };
         return { ...result, presentation: buildFetchPresentation(result) };
     };
 }
-function withSearchFallback(primary, fallback, fallbackFrom) {
+/**
+ * Checks the target URL before dispatching to the http fetcher, a content
+ * reader, or Firecrawl. Doing it here means a private URL is refused once, in
+ * one place, and never reaches Firecrawl or its http fallback (#53).
+ */
+function withTargetGuard(fetchPage, guard, method) {
     return async (input) => {
-        const first = await primary(input);
-        if (first.status !== 'error')
-            return first;
-        const second = await fallback(input);
-        const result = {
-            ...second,
-            metadata: {
-                ...second.metadata,
-                fallbackFrom,
-                fallbackReason: first.error?.message ?? `${fallbackFrom} search failed.`,
-                // Keep the primary's fanout provenance (which providers were tried/skipped) even though
-                // the answer came from the fallback backend.
-                ...(first.metadata.fanout ? { fanout: first.metadata.fanout } : {})
-            }
-        };
-        return { ...result, presentation: buildSearchPresentation(result) };
-    };
-}
-function withFetchFallback(primary, fallback) {
-    return async (input) => {
-        const first = await primary(input);
-        if (first.status !== 'error' && first.status !== 'needs_headless')
-            return first;
-        const second = await fallback(input);
-        const result = {
-            ...second,
-            metadata: {
-                ...second.metadata,
-                fallbackFrom: 'firecrawl',
-                fallbackReason: first.error?.message ?? 'Firecrawl fetch failed.'
-            }
-        };
-        return { ...result, presentation: buildFetchPresentation(result) };
+        try {
+            await guard.assertUrlAllowed(input.url);
+        }
+        catch (error) {
+            const blocked = findGuardError(error);
+            if (!blocked)
+                throw error;
+            const result = {
+                status: 'error',
+                url: input.url,
+                metadata: { method, cacheHit: false },
+                error: { code: blocked.code, message: blocked.message, failure: { kind: 'guard_refused' } }
+            };
+            return { ...result, presentation: buildFetchPresentation(result) };
+        }
+        return fetchPage(input);
     };
 }
 export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
@@ -109,7 +105,7 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
                     status: 'error',
                     results: [],
                     metadata: { backend: config.search.provider, cacheHit: false },
-                    error: { code: 'BACKEND_CONFIG_INVALID', message }
+                    error: { code: 'BACKEND_CONFIG_INVALID', message, failure: { kind: 'config_global' } }
                 };
                 return { ...result, presentation: buildSearchPresentation(result) };
             },
@@ -118,7 +114,7 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
                     status: 'error',
                     url,
                     metadata: { method: 'http', cacheHit: false },
-                    error: { code: 'BACKEND_CONFIG_INVALID', message }
+                    error: { code: 'BACKEND_CONFIG_INVALID', message, failure: { kind: 'config_global' } }
                 };
                 return { ...result, presentation: buildFetchPresentation(result) };
             },
@@ -127,24 +123,71 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
                     status: 'error',
                     url,
                     metadata: { method: 'headless', cacheHit: false },
-                    error: { code: 'BACKEND_CONFIG_INVALID', message }
+                    error: { code: 'BACKEND_CONFIG_INVALID', message, failure: { kind: 'config_global' } }
                 };
                 return { ...result, presentation: buildFetchPresentation(result) };
-            }
+            },
+            close: async () => undefined
         };
     }
-    // When a proxy is configured, every outbound HTTP request (search, fetch,
-    // readers, and doctor-style checks) goes through it; headless browser
-    // traffic gets the same proxy via Playwright launch options.
+    // When a proxy is configured, every outbound HTTP request goes through it.
+    // User-configured endpoints use fetchImpl directly; model-chosen fetches and
+    // the headless browser reach it through the guard proxy below.
     const fetchImpl = proxy ? makeProxyFetch(proxy) : fetch;
     const proxyCredentials = proxy ? resolveProxyCredentials(proxy) : undefined;
-    const proxyBrowserOptions = proxy
-        ? {
-            server: stripProxyCredentials(proxy.url),
-            ...(proxyCredentials?.username !== undefined ? { username: proxyCredentials.username } : {}),
-            ...(proxyCredentials?.password !== undefined ? { password: proxyCredentials.password } : {})
+    // Model-chosen URLs only. Search APIs and the configured SearXNG/Firecrawl
+    // endpoints keep using fetchImpl: the user typed those (#53).
+    const networkGuard = deps.networkGuard ?? createNetworkGuard({ allowRanges: config.network?.allowRanges ?? [] });
+    // One guard proxy per backend set, started on first use. It is the single
+    // place the address policy is enforced, for Node fetches and the browser,
+    // and it chains to the user's upstream proxy itself.
+    let guardProxy;
+    let isClosed = false;
+    const getGuardProxy = () => {
+        if (isClosed)
+            return Promise.reject(new Error('Backend set is closed.'));
+        if (guardProxy)
+            return guardProxy;
+        const started = (deps.createGuardProxy ?? startGuardProxy)({
+            guard: networkGuard,
+            ...(proxy
+                ? {
+                    upstream: {
+                        url: stripProxyCredentials(proxy.url),
+                        ...(proxyCredentials?.username !== undefined ? { username: proxyCredentials.username } : {}),
+                        ...(proxyCredentials?.password !== undefined ? { password: proxyCredentials.password } : {})
+                    }
+                }
+                : {}),
+            trustProxyDns: config.network?.trustProxyDns === true
+        });
+        guardProxy = started;
+        // A failed start should not stick around for the rest of the backend
+        // set's life: clear it so the next call tries again, unless a newer
+        // attempt has already replaced it.
+        started.catch(() => {
+            if (guardProxy === started)
+                guardProxy = undefined;
+        });
+        return started;
+    };
+    const modelFetch = deps.createModelFetch
+        ? deps.createModelFetch(networkGuard)
+        : createGuardProxyFetch(getGuardProxy);
+    const targetFetch = createGuardedFetch(modelFetch, networkGuard);
+    let closing;
+    const close = () => (closing ??= (async () => {
+        isClosed = true;
+        if ('close' in modelFetch)
+            await modelFetch.close();
+        if (guardProxy) {
+            const started = await guardProxy.catch(() => undefined);
+            await started?.close().catch(() => undefined);
         }
-        : undefined;
+    })());
+    // One health state per backend set: a rebuilt set (config change) starts fresh.
+    const policyDeps = { health: deps.providerHealth ?? createProviderHealth(deps.policy?.now ? { now: deps.policy.now } : {}), ...deps.policy };
+    const guarded = (name, search, healthKey) => withSearchPolicy(name, search, policyDeps, healthKey);
     const createDuckDuckGo = () => createDuckDuckGoSearch({ searchHtml: (query) => fetchDuckDuckGoHtml(query, { fetchImpl }) });
     function buildProviderSearch(name) {
         switch (name) {
@@ -165,34 +208,12 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
                 return createDuckDuckGo();
         }
     }
-    let search = config.search.provider === 'searxng'
-        ? config.search.baseUrl
-            ? createSearxngSearch({ baseUrl: config.search.baseUrl, options: config.search.options, fetchImpl })
-            : invalidSearxngSearch()
-        : config.search.provider === 'brave'
-            ? createBraveSearch({ apiKey: process.env.PI_WEB_AGENT_BRAVE_API_KEY, fetchImpl })
-            : config.search.provider === 'youcom'
-                ? createYouComSearch({ apiKey: process.env.YDC_API_KEY, fetchImpl })
-                : config.search.provider === 'exa'
-                    ? createExaSearch({ apiKey: process.env.EXA_API_KEY, fetchImpl })
-                    : config.search.provider === 'tavily'
-                        ? createTavilySearch({ apiKey: process.env.TAVILY_API_KEY, fetchImpl })
-                        : createDuckDuckGo();
-    if (config.search.provider === 'searxng' && config.search.fallback === 'duckduckgo') {
-        search = withSearchFallback(search, createDuckDuckGo(), 'searxng');
-    }
-    if (config.search.provider === 'brave' && config.search.fallback === 'duckduckgo') {
-        search = withSearchFallback(search, createDuckDuckGo(), 'brave');
-    }
-    if (config.search.provider === 'youcom' && config.search.fallback === 'duckduckgo') {
-        search = withSearchFallback(search, createDuckDuckGo(), 'youcom');
-    }
-    if (config.search.provider === 'exa' && config.search.fallback === 'duckduckgo') {
-        search = withSearchFallback(search, createDuckDuckGo(), 'exa');
-    }
-    if (config.search.provider === 'tavily' && config.search.fallback === 'duckduckgo') {
-        search = withSearchFallback(search, createDuckDuckGo(), 'tavily');
+    const primarySearch = guarded(config.search.provider, buildProviderSearch(config.search.provider));
+    const chain = [primarySearch];
+    if (config.search.provider !== 'duckduckgo' && config.search.fallback === 'duckduckgo') {
+        chain.push(guarded('duckduckgo', createDuckDuckGo()));
     }
+    let search = chainSearch(chain, policyDeps);
     const fanoutConfig = config.search.fanout;
     if (fanoutConfig && fanoutConfig.mode !== 'off') {
         const baseNames = fanoutConfig.providers && fanoutConfig.providers.length > 0
@@ -203,9 +224,12 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
             ? [...baseNames, 'duckduckgo']
             : baseNames;
         const ordered = [config.search.provider, ...providerNames.filter((n) => n !== config.search.provider)].filter((n, i, arr) => arr.indexOf(n) === i);
+        const timeoutMs = deps.fanoutTimeoutMs ?? FANOUT_PROVIDER_TIMEOUT_MS;
         search = createFanoutSearch({
-            providers: ordered.map((name) => ({ name, search: buildProviderSearch(name) })),
-            mode: fanoutConfig.mode
+            // The timeout sits inside the policy so a stalled provider gets its one transient retry.
+            providers: ordered.map((name) => ({ name, search: guarded(name, withCallTimeout(buildProviderSearch(name), timeoutMs, name)) })),
+            mode: fanoutConfig.mode,
+            timeoutMs
         });
     }
     // Keep the keyless Tavily safety net for the no-key DuckDuckGo default, even under fanout —
@@ -214,32 +238,35 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
     const keylessFallbackDisabled = process.env.PI_WEB_AGENT_DISABLE_KEYLESS_FALLBACK === '1';
     const usingDuckDuckGoDefault = config.search.provider === 'duckduckgo' || !config.search.provider;
     if (usingDuckDuckGoDefault && !keylessFallbackDisabled) {
-        search = withSearchFallback(search, createTavilySearch({ keyless: true, fetchImpl }), 'duckduckgo');
+        // chainSearch only falls back on non-terminal failures, so a terminal result or bad_request never reaches keyless Tavily.
+        search = chainSearch([search, guarded('tavily', createTavilySearch({ keyless: true, fetchImpl }), 'tavily-keyless')], policyDeps);
     }
-    const httpFetch = createHttpFetch({ fetchPage: createHttpFetcher({ fetchImpl }) });
-    let fetchPage = config.fetch.provider === 'firecrawl'
-        ? config.fetch.baseUrl
-            ? createHttpFetch({
-                fetchPage: createFirecrawlFetch({
-                    baseUrl: config.fetch.baseUrl,
-                    apiKey: config.fetch.apiKey ?? process.env.PI_WEB_AGENT_FIRECRAWL_API_KEY,
-                    options: config.fetch.options,
-                    fetchImpl
-                })
-            })
-            : createHttpFetch({ fetchPage: invalidFirecrawlFetch() })
+    const httpFetcher = createHttpFetcher({ fetchImpl: targetFetch });
+    const httpFetch = createHttpFetch({ fetchPage: ({ url, query }) => httpFetcher(url, query) });
+    const firecrawlFetcher = config.fetch.baseUrl
+        ? createFirecrawlFetch({
+            baseUrl: config.fetch.baseUrl,
+            apiKey: config.fetch.apiKey ?? process.env.PI_WEB_AGENT_FIRECRAWL_API_KEY,
+            options: config.fetch.options,
+            fetchImpl
+        })
+        : invalidFirecrawlFetch();
+    const fetchPage = config.fetch.provider === 'firecrawl'
+        ? withFetchPolicy(createHttpFetch({ fetchPage: ({ url, query }) => firecrawlFetcher(url, query) }), config.fetch.fallback === 'http' ? httpFetch : undefined, policyDeps)
         : httpFetch;
-    if (config.fetch.provider === 'firecrawl' && config.fetch.fallback === 'http') {
-        fetchPage = withFetchFallback(fetchPage, httpFetch);
-    }
     const fetchPageWithReaders = createSpecialContentResolver({
-        readers: [createGithubReader({ fetchImpl }), createPdfReader({ fetchImpl }), createYoutubeReader({ fetchImpl })],
+        readers: [
+            createGithubReader({ fetchImpl: targetFetch }),
+            createPdfReader({ fetchImpl: targetFetch }),
+            createYoutubeReader({ fetchImpl: targetFetch })
+        ],
         fallback: fetchPage
     });
-    const headlessPage = (url) => proxyBrowserOptions ? headlessFetch(url, { proxy: proxyBrowserOptions }) : headlessFetch(url);
+    const headlessPage = ({ url, query }) => headlessFetch(url, { query, guard: networkGuard, guardProxy: getGuardProxy });
     return {
         search,
-        fetchPage: fetchPageWithReaders,
-        headlessFetch: createHeadlessFetch({ fetchPage: headlessPage })
+        fetchPage: withTargetGuard(fetchPageWithReaders, networkGuard, config.fetch.provider === 'firecrawl' ? 'firecrawl' : 'http'),
+        headlessFetch: createHeadlessFetch({ fetchPage: headlessPage }),
+        close
     };
 }