@demigodmode/pi-web-agent 1.10.0 → 1.12.0

package/CHANGELOG.md

@@ -18,6 +18,33 @@ The format is intentionally simple and release-oriented.
 ### Breaking
 - None.
 
+## [1.12.0] - 2026-09-17
+### Added
+- A network allow list, under Settings → Backends → Network allow list. web_explore now refuses private and local addresses when the link came from the model or a page it read, so if you genuinely want it to read something on your own network (an internal docs site, a service on localhost), add that range here as a CIDR like `10.0.0.0/24`. Entries are checked when you save, and ones that would allow everything (`0.0.0.0/0`, `::/0`) are rejected, since that would quietly turn the protection off. It is also the fix if every fetch suddenly fails: some proxy apps run in fake-IP mode and make every site look like it lives in `198.18.0.0/15`, and adding that range gets you going again. (#53)
+- A "Trust the upstream proxy to enforce private-address restrictions" setting (`backends.network.trustProxyDns`). With a proxy configured, pi-web-agent normally looks each address up itself and asks the proxy to connect to that exact IP, so the proxy can't send a request somewhere else. Some proxies only accept hostnames, and some networks only resolve names inside the proxy. Turning this on hands the hostname to the proxy instead. It is off by default because it moves the address decision to your proxy, and even when it is on, localhost, private IPs typed straight into a link, and anything your own machine resolves to a private address are still refused. `/web-agent doctor` shows the allow list and this setting. (#53)
+
+### Changed
+- Search and fetch fallback now act on why a backend failed. Rate limits pause that backend for the time the provider asks (capped at 15 minutes), exhausted quotas and bad API keys skip it until settings change, and timeouts or server errors get one retry. An empty result is treated as a real answer instead of a failure, a private-address refusal or an invalid shared proxy setting never falls back to another backend, and answers say when some search backends were unavailable. DuckDuckGo no longer retries on its own. (#55)
+
+### Fixed
+- web_explore no longer fetches private, loopback, or link-local addresses when the link came from the model or a page it read. The check happens where connections are opened, through a local guard proxy, so it also covers redirects, DNS changes, and everything a headless page loads. A web page could previously steer it at cloud metadata endpoints or services on your network. Addresses you configured yourself (search backends, SearXNG, Firecrawl, the proxy) are unaffected. If you do need to reach a private range, add it under Settings → Backends → Network allow list. With an upstream proxy that only accepts hostnames, turn on "Trust the upstream proxy to enforce private-address restrictions". (#53)
+
+### Breaking
+- None.
+
+## [1.11.0] - 2026-09-16
+### Added
+- Optional proxy support. Point web_explore at an HTTP/HTTPS proxy and everything outbound goes through it: search, fetch, Firecrawl, the GitHub/PDF/YouTube readers, doctor health checks, and the headless browser. Off unless you set it, and nothing changes if you don't. Set it in Settings → Backends, and keep credentials in `PI_WEB_AGENT_PROXY_USERNAME` / `PI_WEB_AGENT_PROXY_PASSWORD` rather than in the URL. Thanks to @lo-tp for building this. (#50)
+
+### Changed
+- `/web-agent doctor` reports whether the jsdom compat patch is in place, and prints the command to reapply it if it isn't.
+
+### Fixed
+- Pi failing to start with `Cannot find module 'punycode/'` or `Set operation called on non-Set object`. The patch for this was only applied at install time, but every pi extension shares one node_modules tree, so installing or updating anything else quietly reverted it. It now reapplies every time the extension loads. Also fixes it picking the wrong copy of the dependency when the shared tree holds more than one. (#34)
+
+### Breaking
+- None.
+
 ## [1.10.0] - 2026-08-25
 ### Added
 - The zero-config default has a safety net now. When DuckDuckGo blocks a search (which happens fast on VPS and datacenter IPs), web search falls back to Tavily's keyless endpoint, no account or API key needed, so a fresh install still returns results instead of an error. Set `PI_WEB_AGENT_DISABLE_KEYLESS_FALLBACK=1` if you'd rather it just fail. (#42)

package/README.md

@@ -4,9 +4,24 @@
 
 # pi-web-agent
 
-[![CI](https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml)
-[![npm version](https://img.shields.io/npm/v/@demigodmode/pi-web-agent)](https://www.npmjs.com/package/@demigodmode/pi-web-agent)
-[![Docs](https://img.shields.io/badge/docs-github%20pages-blue)](https://demigodmode.github.io/pi-web-agent/)
+<p align="center">
+  <a href="https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/demigodmode/pi-web-agent/ci.yml?branch=main&style=flat-square&logo=github&label=CI" alt="CI"></a>
+  <a href="https://www.npmjs.com/package/@demigodmode/pi-web-agent"><img src="https://img.shields.io/npm/v/@demigodmode/pi-web-agent?style=flat-square&logo=npm&logoColor=white&color=CB3837" alt="npm version"></a>
+  <a href="https://www.npmjs.com/package/@demigodmode/pi-web-agent"><img src="https://img.shields.io/npm/dm/@demigodmode/pi-web-agent?style=flat-square&color=0A7BBB&label=downloads" alt="npm downloads"></a>
+  <img src="https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-475569?style=flat-square" alt="Platform: macOS, Linux, Windows">
+  <a href="https://demigodmode.github.io/pi-web-agent/"><img src="https://img.shields.io/badge/docs-github%20pages-2088FF?style=flat-square&logo=readthedocs&logoColor=white" alt="Docs"></a>
+  <a href="https://github.com/demigodmode/pi-web-agent/blob/main/LICENSE"><img src="https://img.shields.io/npm/l/@demigodmode/pi-web-agent?style=flat-square&color=6F42C1" alt="License"></a>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/search%20backends-475569?style=flat-square" alt="Search backends">
+  <img src="https://img.shields.io/badge/DuckDuckGo-475569?style=flat-square&logo=duckduckgo&logoColor=white" alt="DuckDuckGo">
+  <img src="https://img.shields.io/badge/SearXNG-475569?style=flat-square&logo=searxng&logoColor=white" alt="SearXNG">
+  <img src="https://img.shields.io/badge/Brave-475569?style=flat-square&logo=brave&logoColor=white" alt="Brave Search">
+  <img src="https://img.shields.io/badge/You.com-475569?style=flat-square" alt="You.com">
+  <img src="https://img.shields.io/badge/Exa-475569?style=flat-square" alt="Exa">
+  <img src="https://img.shields.io/badge/Tavily-475569?style=flat-square" alt="Tavily">
+</p>
 
 One public tool, `web_explore`, that does bounded web research for Pi: search, fetch, targeted browser rendering, ranking, and honest caveats, all behind a single call.
 
@@ -26,7 +41,7 @@ One public tool, `web_explore`, that does bounded web research for Pi: search, f
 
 Compared to other web tooling for agents:
 
-- **Hands-off.** No curator, no browser windows to approve, no step that pops you out of your session. Ask `web_explore` once and the answer comes back with caveats. Nothing to babysit.
+- **Hands-off.** No curator, no browser windows to approve, no step that pops you out of your session. Ask `web_explore` once and the answer comes back cleanly. Nothing to babysit.
 - **Keyless by default.** Search, page reads, and the GitHub/PDF/YouTube readers all work with no API keys. Add hosted providers only when you want them.
 - **Bounded and honest.** Compact output by default, and it says when a read was not good enough instead of returning fake confidence.
 

package/dist/backends/config.d.ts

@@ -12,6 +12,11 @@ export type FirecrawlOptions = {
     formats?: string[];
     onlyMainContent?: boolean;
 };
+export type ProxyConfig = {
+    url: string;
+    username?: string;
+    password?: string;
+};
 export type SearchBackendConfig = {
     provider: 'duckduckgo' | 'searxng' | 'brave' | 'youcom' | 'exa' | 'tavily';
     baseUrl?: string;
@@ -29,15 +34,25 @@ export type FetchBackendConfig = {
 export type HeadlessBackendConfig = {
     provider: 'local-browser';
 };
+export type NetworkConfig = {
+    /** CIDR ranges exempted from the private-address guard (#53). */
+    allowRanges?: string[];
+    /** Trust the upstream proxy to enforce private-address restrictions. */
+    trustProxyDns?: boolean;
+};
 export type BackendConfig = {
     search: SearchBackendConfig;
     fetch: FetchBackendConfig;
     headless: HeadlessBackendConfig;
+    proxy?: ProxyConfig;
+    network?: NetworkConfig;
 };
 export type BackendConfigOverride = {
     search?: Partial<SearchBackendConfig>;
     fetch?: Partial<FetchBackendConfig>;
     headless?: Partial<HeadlessBackendConfig>;
+    proxy?: ProxyConfig;
+    network?: NetworkConfig;
 };
 export type BackendConfigFile = {
     backends?: {
@@ -58,9 +73,34 @@ export type BackendConfigFile = {
         headless?: {
             provider?: unknown;
         };
+        proxy?: {
+            url?: unknown;
+            username?: unknown;
+            password?: unknown;
+        };
+        network?: {
+            allowRanges?: unknown;
+            trustProxyDns?: unknown;
+        };
     };
 };
 export declare const DEFAULT_BACKEND_CONFIG: BackendConfig;
+/**
+ * Remove any credentials (user:pass@) embedded in a proxy URL. pi-web-agent
+ * never reads or sends credentials from the URL; proxy auth comes from
+ * backends.proxy.username/password or PI_WEB_AGENT_PROXY_USERNAME /
+ * PI_WEB_AGENT_PROXY_PASSWORD. Stripping them here keeps them out of logs,
+ * doctor output, the settings UI, and the proxy connection itself.
+ */
+export declare function stripProxyCredentials(url: string): string;
+/**
+ * Whether a proxy url passes validation: a blank url is the "disable proxy"
+ * marker and passes; anything else must parse as an http(s) URL. This is a
+ * pure syntax check — no connectivity check is performed.
+ */
+export declare function isValidProxyUrl(url: string): boolean;
+export declare function extractProxyConfig(value: unknown): ProxyConfig | undefined;
+export declare function extractNetworkConfig(value: unknown): NetworkConfig | undefined;
 export declare function usableSearchProviders(search: SearchBackendConfig, env?: NodeJS.ProcessEnv): SearchProviderName[];
 export declare function extractBackendConfigOverride(file: BackendConfigFile | null | undefined): BackendConfigOverride;
 export declare function validateBackendConfig(config: BackendConfig): string[];

package/dist/backends/config.js

@@ -1,3 +1,4 @@
+import { parseCidr } from '../fetch/network-guard.js';
 export const DEFAULT_BACKEND_CONFIG = {
     search: { provider: 'duckduckgo' },
     fetch: { provider: 'http' },
@@ -9,6 +10,89 @@ function extractStringArray(value) {
     const strings = value.filter((item) => typeof item === 'string');
     return strings.length === value.length ? strings : undefined;
 }
+/**
+ * Remove any credentials (user:pass@) embedded in a proxy URL. pi-web-agent
+ * never reads or sends credentials from the URL; proxy auth comes from
+ * backends.proxy.username/password or PI_WEB_AGENT_PROXY_USERNAME /
+ * PI_WEB_AGENT_PROXY_PASSWORD. Stripping them here keeps them out of logs,
+ * doctor output, the settings UI, and the proxy connection itself.
+ */
+export function stripProxyCredentials(url) {
+    let parsed;
+    try {
+        parsed = new URL(url);
+    }
+    catch {
+        return url;
+    }
+    if (!parsed.username && !parsed.password)
+        return url; // already credential-free
+    parsed.username = '';
+    parsed.password = '';
+    return parsed.toString().replace(/\/$/, '');
+}
+/**
+ * Whether a proxy url passes validation: a blank url is the "disable proxy"
+ * marker and passes; anything else must parse as an http(s) URL. This is a
+ * pure syntax check — no connectivity check is performed.
+ */
+export function isValidProxyUrl(url) {
+    if (url.trim() === '')
+        return true;
+    try {
+        const parsed = new URL(url);
+        return parsed.protocol === 'http:' || parsed.protocol === 'https:';
+    }
+    catch {
+        return false;
+    }
+}
+export function extractProxyConfig(value) {
+    if (!value || typeof value !== 'object')
+        return undefined;
+    const raw = value;
+    // An explicitly present but blank url is the "disable proxy" marker: it lets a
+    // higher-priority layer (e.g. a project) clear a proxy set in a lower layer.
+    if (typeof raw.url === 'string' && raw.url.trim() === '') {
+        return { url: '' };
+    }
+    if (typeof raw.url !== 'string' || !raw.url.trim())
+        return undefined;
+    const url = raw.url.trim();
+    let parsed;
+    try {
+        parsed = new URL(url);
+    }
+    catch {
+        parsed = undefined;
+    }
+    // A valid url is normalized. A malformed url is kept as-is (rather than
+    // dropped) so validation can flag it and the backend factory can fail loudly
+    // instead of silently sending traffic direct to the websites.
+    const config = {
+        url: parsed && (parsed.protocol === 'http:' || parsed.protocol === 'https:')
+            ? parsed.toString().replace(/\/$/, '')
+            : url
+    };
+    if (typeof raw.username === 'string' && raw.username.trim())
+        config.username = raw.username;
+    if (typeof raw.password === 'string')
+        config.password = raw.password;
+    return config;
+}
+export function extractNetworkConfig(value) {
+    if (!value || typeof value !== 'object')
+        return undefined;
+    const raw = value;
+    const config = {};
+    if (Array.isArray(raw.allowRanges)) {
+        // Coerce non-string entries so validation flags them instead of dropping the list.
+        config.allowRanges = raw.allowRanges.map((entry) => (typeof entry === 'string' ? entry : String(entry)));
+    }
+    if (typeof raw.trustProxyDns === 'boolean')
+        config.trustProxyDns = raw.trustProxyDns;
+    return Object.keys(config).length > 0 ? config : undefined;
+}
 function extractSearxngOptions(value) {
     if (!value || typeof value !== 'object')
         return undefined;
@@ -113,6 +197,14 @@ export function extractBackendConfigOverride(file) {
     if (backends?.headless?.provider === 'local-browser') {
         override.headless = { provider: 'local-browser' };
     }
+    const proxy = extractProxyConfig(backends?.proxy);
+    if (proxy) {
+        override.proxy = proxy;
+    }
+    const network = extractNetworkConfig(backends?.network);
+    if (network) {
+        override.network = network;
+    }
     return override;
 }
 export function validateBackendConfig(config) {
@@ -120,9 +212,38 @@ export function validateBackendConfig(config) {
     if (config.search.provider === 'searxng' && !config.search.baseUrl) {
         issues.push('search provider searxng requires backends.search.baseUrl');
     }
+    if (config.proxy && config.proxy.url.trim() !== '') {
+        let parsed;
+        try {
+            parsed = new URL(config.proxy.url);
+        }
+        catch {
+            parsed = undefined;
+        }
+        if (!isValidProxyUrl(config.proxy.url)) {
+            issues.push('backends.proxy.url must be an http or https URL');
+        }
+        else if (parsed && (parsed.username || parsed.password)) {
+            // Credentials belong in backends.proxy.username/password or the env vars
+            // below, never in the URL itself.
+            issues.push('backends.proxy.url must not include credentials (user:pass@); set PI_WEB_AGENT_PROXY_USERNAME and PI_WEB_AGENT_PROXY_PASSWORD (or backends.proxy.username / backends.proxy.password) instead');
+        }
+    }
     if (config.fetch.provider === 'firecrawl' && !config.fetch.baseUrl) {
         issues.push('fetch provider firecrawl requires backends.fetch.baseUrl');
     }
+    for (const range of config.network?.allowRanges ?? []) {
+        const cidr = parseCidr(range);
+        if (!cidr) {
+            issues.push(`backends.network.allowRanges entry "${range}" is not a valid CIDR range`);
+        }
+        else if (cidr.prefix === 0) {
+            issues.push(`backends.network.allowRanges entry "${range}" allows every address, which turns the guard off; list specific ranges instead`);
+        }
+    }
+    if (config.network?.trustProxyDns && !config.proxy?.url?.trim()) {
+        issues.push('backends.network.trustProxyDns has no effect without backends.proxy');
+    }
     if (config.search.fallback === 'duckduckgo' && config.search.provider !== 'searxng' && config.search.provider !== 'brave' && config.search.provider !== 'youcom' && config.search.provider !== 'exa' && config.search.provider !== 'tavily') {
         issues.push('search fallback duckduckgo is only supported when search provider is searxng, brave, youcom, exa, or tavily');
     }
@@ -169,10 +290,28 @@ function mergeFetchConfig(current, override) {
     }
     return { ...current, ...override };
 }
+function mergeNetworkConfig(base, layer) {
+    if (!layer)
+        return base;
+    const next = { ...base };
+    // A layer's allow list replaces the lower one outright; trust is set independently.
+    if (layer.allowRanges)
+        next.allowRanges = [...layer.allowRanges];
+    if (layer.trustProxyDns !== undefined)
+        next.trustProxyDns = layer.trustProxyDns;
+    return next;
+}
 export function mergeBackendConfigLayers(...layers) {
     return layers.reduce((merged, layer) => ({
         search: mergeSearchConfig(merged.search, layer?.search),
         fetch: mergeFetchConfig(merged.fetch, layer?.fetch),
-        headless: { ...merged.headless, ...layer?.headless }
+        headless: { ...merged.headless, ...layer?.headless },
+        proxy: layer?.proxy
+            ? layer.proxy.url === ''
+                ? undefined // explicit disable overrides any proxy from lower layers
+                : { ...merged.proxy, ...layer.proxy }
+            : merged.proxy,
+        // Replace, don't union: a project list is the whole list for that project.
+        network: mergeNetworkConfig(merged.network, layer?.network)
     }), DEFAULT_BACKEND_CONFIG);
 }

package/dist/backends/factory.d.ts

@@ -1,14 +1,18 @@
 import { createFirecrawlFetcher } from '../fetch/firecrawl-fetch.js';
+import { type GuardProxy, type GuardProxyOptions } from '../fetch/guard-proxy.js';
+import { type NetworkGuard } from '../fetch/network-guard.js';
 import { createBraveSearchTool } from '../search/brave.js';
 import { createYouComSearchTool } from '../search/youcom.js';
 import { createExaSearchTool } from '../search/exa.js';
 import { createTavilySearchTool } from '../search/tavily.js';
 import { createSearxngSearchTool } from '../search/searxng.js';
+import { type PolicyDeps } from './fallback-policy.js';
+import { type ProviderHealth } from './provider-health.js';
 import { createWebFetchHeadlessTool } from '../tools/web-fetch-headless.js';
 import { createWebFetchTool } from '../tools/web-fetch.js';
 import { createWebSearchTool } from '../tools/web-search.js';
 import type { WebFetchHeadlessResponse, WebFetchResponse, WebSearchResponse } from '../types.js';
-import { type BackendConfig } from './config.js';
+import { type BackendConfig, type ProxyConfig } from './config.js';
 export type BackendSet = {
     search: (input: {
         query: string;
@@ -19,6 +23,8 @@ export type BackendSet = {
     headlessFetch: (input: {
         url: string;
     }) => Promise<WebFetchHeadlessResponse>;
+    /** Releases the guard proxy and its agents. Idempotent; never starts the proxy. */
+    close: () => Promise<void>;
 };
 export type BackendFactoryDeps = {
     createDuckDuckGoSearch?: typeof createWebSearchTool;
@@ -30,5 +36,15 @@ export type BackendFactoryDeps = {
     createHttpFetch?: typeof createWebFetchTool;
     createFirecrawlFetch?: typeof createFirecrawlFetcher;
     createHeadlessFetch?: typeof createWebFetchHeadlessTool;
+    createProxyFetch?: (proxy: ProxyConfig) => typeof fetch;
+    networkGuard?: NetworkGuard;
+    /** Test seam: the fetch used for model-chosen URLs, before redirect handling. */
+    createModelFetch?: (guard: NetworkGuard) => typeof fetch;
+    createGuardProxy?: (options: GuardProxyOptions) => Promise<GuardProxy>;
+    providerHealth?: ProviderHealth;
+    /** Test seam for the retry sleep, jitter, and clock. */
+    policy?: Omit<PolicyDeps, 'health'>;
+    /** Test seam for the per-call fanout provider timeout. */
+    fanoutTimeoutMs?: number;
 };
 export declare function createBackendSet(config?: BackendConfig, deps?: BackendFactoryDeps): BackendSet;