@demigodmode/pi-web-agent 1.9.0 → 1.11.0

package/CHANGELOG.md

@@ -18,6 +18,32 @@ The format is intentionally simple and release-oriented.
 ### Breaking
 - None.
 
+## [1.11.0] - 2026-09-16
+### Added
+- Optional proxy support. Point web_explore at an HTTP/HTTPS proxy and everything outbound goes through it: search, fetch, Firecrawl, the GitHub/PDF/YouTube readers, doctor health checks, and the headless browser. Off unless you set it, and nothing changes if you don't. Set it in Settings → Backends, and keep credentials in `PI_WEB_AGENT_PROXY_USERNAME` / `PI_WEB_AGENT_PROXY_PASSWORD` rather than in the URL. Thanks to @lo-tp for building this. (#50)
+
+### Changed
+- `/web-agent doctor` reports whether the jsdom compat patch is in place, and prints the command to reapply it if it isn't.
+
+### Fixed
+- Pi failing to start with `Cannot find module 'punycode/'` or `Set operation called on non-Set object`. The patch for this was only applied at install time, but every pi extension shares one node_modules tree, so installing or updating anything else quietly reverted it. It now reapplies every time the extension loads. Also fixes it picking the wrong copy of the dependency when the shared tree holds more than one. (#34)
+
+### Breaking
+- None.
+
+## [1.10.0] - 2026-08-25
+### Added
+- The zero-config default has a safety net now. When DuckDuckGo blocks a search (which happens fast on VPS and datacenter IPs), web search falls back to Tavily's keyless endpoint, no account or API key needed, so a fresh install still returns results instead of an error. Set `PI_WEB_AGENT_DISABLE_KEYLESS_FALLBACK=1` if you'd rather it just fail. (#42)
+
+### Changed
+- The keyless DuckDuckGo default is a lot harder to bot-wall. It sends real browser headers and retries once when a page looks blocked, instead of going out as an obvious script and getting bounced on the first request. (#42)
+
+### Fixed
+- Blocked searches are labeled honestly. A bot-wall page comes back as "blocked" (which triggers the fallback) instead of "couldn't parse", including the 200-status pages that mix a no-results message with a bot-wall notice. (#42)
+
+### Breaking
+- None.
+
 ## [1.9.0] - 2026-08-15
 ### Added
 - Search fanout. Ask a hard question and web_explore can now hit several of your configured search backends at once, dedupe the merged results, and rank the pages that more than one provider agreed on to the top. Off by default. Flip it to on or auto in Settings → Backends (auto only fans out when the first provider comes back thin, so easy queries stay cheap). Preview/verbose shows which providers ran and which got skipped. (#35)

package/README.md

@@ -1,131 +1,136 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/demigodmode/pi-web-agent/main/docs/public/pi-web-agent-banner.png" alt="pi-web-agent: bounded web research for Pi" width="100%">
+</p>
+
 # pi-web-agent
 
-[![CI](https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml)
-[![npm version](https://img.shields.io/npm/v/@demigodmode/pi-web-agent)](https://www.npmjs.com/package/@demigodmode/pi-web-agent)
-[![Docs](https://img.shields.io/badge/docs-github%20pages-blue)](https://demigodmode.github.io/pi-web-agent/)
+<p align="center">
+  <a href="https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/demigodmode/pi-web-agent/ci.yml?branch=main&style=flat-square&logo=github&label=CI" alt="CI"></a>
+  <a href="https://www.npmjs.com/package/@demigodmode/pi-web-agent"><img src="https://img.shields.io/npm/v/@demigodmode/pi-web-agent?style=flat-square&logo=npm&logoColor=white&color=CB3837" alt="npm version"></a>
+  <a href="https://www.npmjs.com/package/@demigodmode/pi-web-agent"><img src="https://img.shields.io/npm/dm/@demigodmode/pi-web-agent?style=flat-square&color=0A7BBB&label=downloads" alt="npm downloads"></a>
+  <img src="https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-475569?style=flat-square" alt="Platform: macOS, Linux, Windows">
+  <a href="https://demigodmode.github.io/pi-web-agent/"><img src="https://img.shields.io/badge/docs-github%20pages-2088FF?style=flat-square&logo=readthedocs&logoColor=white" alt="Docs"></a>
+  <a href="https://github.com/demigodmode/pi-web-agent/blob/main/LICENSE"><img src="https://img.shields.io/npm/l/@demigodmode/pi-web-agent?style=flat-square&color=6F42C1" alt="License"></a>
+</p>
 
-`@demigodmode/pi-web-agent` is a Pi package for web access.
+<p align="center">
+  <img src="https://img.shields.io/badge/search%20backends-475569?style=flat-square" alt="Search backends">
+  <img src="https://img.shields.io/badge/DuckDuckGo-475569?style=flat-square&logo=duckduckgo&logoColor=white" alt="DuckDuckGo">
+  <img src="https://img.shields.io/badge/SearXNG-475569?style=flat-square&logo=searxng&logoColor=white" alt="SearXNG">
+  <img src="https://img.shields.io/badge/Brave-475569?style=flat-square&logo=brave&logoColor=white" alt="Brave Search">
+  <img src="https://img.shields.io/badge/You.com-475569?style=flat-square" alt="You.com">
+  <img src="https://img.shields.io/badge/Exa-475569?style=flat-square" alt="Exa">
+  <img src="https://img.shields.io/badge/Tavily-475569?style=flat-square" alt="Tavily">
+</p>
 
-Most agent web tools blur search, fetch, browser rendering, and research into one vague thing. `pi-web-agent` exposes one public research tool, `web_explore`, and keeps search/fetch/headless work inside that bounded workflow.
+One public tool, `web_explore`, that does bounded web research for Pi: search, fetch, targeted browser rendering, ranking, and honest caveats, all behind a single call.
 
-The point is keeping the model-facing boundary simple: ask `web_explore` to research a question, and it handles direct links, discovery, HTTP reads, targeted browser rendering, source ranking, source-quality checks, and caveats internally.
+> Most agent web tooling blurs search, fetch, rendering, and synthesis into one vague thing. `pi-web-agent` keeps that boundary simple, and it is stricter about what it actually did: bot-check pages, narrow source sets, unreadable threads, and conflicting evidence show up as caveats instead of fake confidence.
 
-That sounds obvious, but a lot of agent tooling gets fuzzy right there. This package is meant to be stricter about what it actually did and more willing to say when a read was not good enough to trust. Bot-check pages, narrow source sets, unreadable threads, and cautionary/conflicting evidence should show up as caveats instead of fake confidence.
+## What you get
 
-## Install
+- **One tool.** `web_explore` handles direct links, discovery, HTTP reads, targeted headless rendering, source ranking, source-quality checks, and caveats internally.
+- **Reads the real content behind links.** Paste a GitHub, PDF, or YouTube URL and it pulls the actual thing (GitHub files/issues/PRs from the API, PDF text, YouTube transcripts), keyless. So "summarize this PDF" or "what does this repo do" works off the source, not the page shell.
+- **Six search backends.** DuckDuckGo (keyless default), SearXNG, Brave, You.com, Exa, Tavily.
+- **Optional search fanout.** Query several backends at once, dedupe, and rank pages that more than one provider agreed on to the top. Off by default; flip it to `on` or `auto`.
+- **Honest by default.** Weak, narrow, blocked, or cautionary evidence gets flagged instead of dressed up as confidence.
+- **Bounded output.** `compact` / `preview` / `verbose` transcript modes.
+- **Zero-config to start.** Runs keyless out of the box (DuckDuckGo search, local browser, the built-in readers). Opt into hosted backends, fallback, search fanout, and per-tool output modes through config when you want more control.
 
-Compatibility notice: current `pi-web-agent` requires Pi 0.74+ because Pi packages moved to the `@earendil-works/*` scope. Update Pi before updating this package. If you are on an older Pi version, stay on `@demigodmode/pi-web-agent@0.6.x` until Pi is updated.
+## Why pi-web-agent
 
-```bash
-pi install npm:@demigodmode/pi-web-agent
-```
+Compared to other web tooling for agents:
 
-After installing, reload or restart Pi. Run `/web-agent` for the action menu, or `/web-agent doctor` to check whether the package loaded cleanly and which web backends are configured.
+- **Hands-off.** No curator, no browser windows to approve, no step that pops you out of your session. Ask `web_explore` once and the answer comes back cleanly. Nothing to babysit.
+- **Keyless by default.** Search, page reads, and the GitHub/PDF/YouTube readers all work with no API keys. Add hosted providers only when you want them.
+- **Bounded and honest.** Compact output by default, and it says when a read was not good enough instead of returning fake confidence.
 
-Headless rendering first tries a detectable Chromium-family browser: Chrome, Chromium, Edge, or Brave. If none is found, it falls back to Playwright-managed Chromium and still launches headless. Firefox/Safari-only systems can still use search and plain HTTP reads; browser-rendered fallback uses Chromium.
+## Install
 
-Later on, update installed packages with:
+> `pi-web-agent` requires Pi 0.74+ (Pi packages moved to the `@earendil-works/*` scope). Update Pi before updating this package. On older Pi, stay on `@demigodmode/pi-web-agent@0.6.x`.
 
 ```bash
-pi update --extensions
+pi install npm:@demigodmode/pi-web-agent
 ```
 
-## Docs
+Reload or restart Pi after installing, then:
 
-Docs site:
+```text
+/web-agent doctor   # check it loaded and show configured backends
+/web-agent          # action menu
+```
 
-- https://demigodmode.github.io/pi-web-agent/
+Update later with `pi update --extensions`.
 
-Work on the docs locally:
+**Browser rendering:** headless first tries a detected Chromium-family browser (Chrome, Chromium, Edge, Brave). If none is found, it falls back to Playwright-managed Chromium. Firefox/Safari-only systems still get search and plain HTTP reads.
 
-```bash
-npm run docs:dev
-```
+## Usage
 
-Build the docs:
-
-```bash
-npm run docs:build
-```
+Ask `web_explore` a web question:
 
-## Presentation modes
+> Find current docs and discussions on configuring Vitest coverage with the v8 provider.
 
-`pi-web-agent` renders web tool output in one visible mode at a time:
+Or hand it a link to read:
 
-- `compact` — short summary, default everywhere
-- `preview` — slightly richer bounded view
-- `verbose` — fuller bounded view
+> Summarize this PDF: https://arxiv.org/pdf/1706.03762
 
-See the `v0.3.0` release notes for a before/after of the transcript cleanup:
+If a pass comes back thin, call `web_explore` again with a narrower query.
 
-- https://github.com/demigodmode/pi-web-agent/releases/tag/v0.3.0
+## Backends
 
-## Settings
+Defaults are DuckDuckGo search, plain HTTP fetch, and local-browser headless. Switch providers from `/web-agent settings → Backends`. API keys stay in environment variables, never in config files.
 
-Primary UI:
+| Backend | Role | Enable with |
+| --- | --- | --- |
+| DuckDuckGo | search (default) | nothing, keyless |
+| SearXNG | search (self-hosted) | base URL |
+| Brave | search (hosted) | `PI_WEB_AGENT_BRAVE_API_KEY` |
+| You.com | search (hosted) | `YDC_API_KEY` |
+| Exa | search (hosted) | `EXA_API_KEY` |
+| Tavily | search (hosted) | `TAVILY_API_KEY` |
+| Firecrawl | fetch (self-hosted) | base URL + `PI_WEB_AGENT_FIRECRAWL_API_KEY` |
+| GitHub reader | content | `GITHUB_TOKEN` (optional, raises the rate limit) |
 
-```text
-/web-agent settings
-```
+Full config shape (fallback, SearXNG/Firecrawl options, fanout): see the [self-hosted backends docs](https://demigodmode.github.io/pi-web-agent/self-hosted-backends).
 
-Helper commands:
+## Settings
 
 ```text
-/web-agent doctor
-/web-agent show
+/web-agent settings                    # main UI
+/web-agent doctor                      # health check
+/web-agent show                        # effective config
 /web-agent changelog
-/web-agent reset project
-/web-agent reset global
-/web-agent mode preview
-/web-agent mode web_explore verbose
-/web-agent mode web_explore inherit
+/web-agent mode web_explore verbose    # per-tool presentation mode
+/web-agent reset project | global
 ```
 
-Config files:
+Config is JSON, and project config overrides global:
 
 ```text
 Global:  ~/.pi/agent/extensions/pi-web-agent/config.json
 Project: .pi/extensions/pi-web-agent/config.json
 ```
 
-Precedence:
-
-- built-in defaults
-- global config
-- project config
-
-Project config overrides global config.
-
-Example:
-
 ```json
 {
   "presentation": {
     "defaultMode": "compact",
-    "tools": {
-      "web_explore": { "mode": "verbose" }
-    }
+    "tools": { "web_explore": { "mode": "verbose" } }
   }
 }
 ```
 
-Backend config is also supported. Defaults remain DuckDuckGo search, plain HTTP fetch, and local-browser headless fallback with managed Chromium fallback configured. If you have a Brave Search or You.com API key, either can be selected as a hosted search backend while `web_explore` still handles page reading, ranking, and caveats itself.
+Presentation modes:
 
-Search backends: DuckDuckGo (default), SearXNG (self-hosted), Brave Search (hosted), You.com (hosted).
+- `compact`: short summary, the default everywhere
+- `preview`: slightly richer bounded view
+- `verbose`: fuller bounded view
 
-Backend settings can be changed from:
-
-```text
-/web-agent settings
-```
-
-Choose **Backends** to edit search/fetch providers, fallback behavior, and SearXNG or Firecrawl base URLs interactively. Brave Search uses `PI_WEB_AGENT_BRAVE_API_KEY`, You.com uses `YDC_API_KEY`, Exa uses `EXA_API_KEY`, and Tavily uses `TAVILY_API_KEY`. Firecrawl API keys should also stay in environment variables rather than being written into config files.
-
-For the full backend config shape, including SearXNG, Brave, You.com, Firecrawl, and fallback behavior, see:
+## Docs
 
-- https://demigodmode.github.io/pi-web-agent/self-hosted-backends
+Full docs: <https://demigodmode.github.io/pi-web-agent/>. Work on them locally with `npm run docs:dev`.
 
-## Local development
+## Development
 
 ```bash
 npm install
@@ -134,10 +139,8 @@ npm run lint
 npm run build
 ```
 
-For local Pi work, this repo includes `.pi/extensions/pi-web-agent.ts`.
-
-If Pi is already running, use `/reload` after changes.
+Local Pi work uses `.pi/extensions/pi-web-agent.ts`; run `/reload` after changes.
 
 ## License
 
-AGPL-3.0-only. See `LICENSE`.
+AGPL-3.0-only. See [LICENSE](LICENSE).

package/dist/backends/config.d.ts

@@ -12,6 +12,11 @@ export type FirecrawlOptions = {
     formats?: string[];
     onlyMainContent?: boolean;
 };
+export type ProxyConfig = {
+    url: string;
+    username?: string;
+    password?: string;
+};
 export type SearchBackendConfig = {
     provider: 'duckduckgo' | 'searxng' | 'brave' | 'youcom' | 'exa' | 'tavily';
     baseUrl?: string;
@@ -33,11 +38,13 @@ export type BackendConfig = {
     search: SearchBackendConfig;
     fetch: FetchBackendConfig;
     headless: HeadlessBackendConfig;
+    proxy?: ProxyConfig;
 };
 export type BackendConfigOverride = {
     search?: Partial<SearchBackendConfig>;
     fetch?: Partial<FetchBackendConfig>;
     headless?: Partial<HeadlessBackendConfig>;
+    proxy?: ProxyConfig;
 };
 export type BackendConfigFile = {
     backends?: {
@@ -58,9 +65,29 @@ export type BackendConfigFile = {
         headless?: {
             provider?: unknown;
         };
+        proxy?: {
+            url?: unknown;
+            username?: unknown;
+            password?: unknown;
+        };
     };
 };
 export declare const DEFAULT_BACKEND_CONFIG: BackendConfig;
+/**
+ * Remove any credentials (user:pass@) embedded in a proxy URL. pi-web-agent
+ * never reads or sends credentials from the URL; proxy auth comes from
+ * backends.proxy.username/password or PI_WEB_AGENT_PROXY_USERNAME /
+ * PI_WEB_AGENT_PROXY_PASSWORD. Stripping them here keeps them out of logs,
+ * doctor output, the settings UI, and the proxy connection itself.
+ */
+export declare function stripProxyCredentials(url: string): string;
+/**
+ * Whether a proxy url passes validation: a blank url is the "disable proxy"
+ * marker and passes; anything else must parse as an http(s) URL. This is a
+ * pure syntax check — no connectivity check is performed.
+ */
+export declare function isValidProxyUrl(url: string): boolean;
+export declare function extractProxyConfig(value: unknown): ProxyConfig | undefined;
 export declare function usableSearchProviders(search: SearchBackendConfig, env?: NodeJS.ProcessEnv): SearchProviderName[];
 export declare function extractBackendConfigOverride(file: BackendConfigFile | null | undefined): BackendConfigOverride;
 export declare function validateBackendConfig(config: BackendConfig): string[];

package/dist/backends/config.js

@@ -9,6 +9,76 @@ function extractStringArray(value) {
     const strings = value.filter((item) => typeof item === 'string');
     return strings.length === value.length ? strings : undefined;
 }
+/**
+ * Remove any credentials (user:pass@) embedded in a proxy URL. pi-web-agent
+ * never reads or sends credentials from the URL; proxy auth comes from
+ * backends.proxy.username/password or PI_WEB_AGENT_PROXY_USERNAME /
+ * PI_WEB_AGENT_PROXY_PASSWORD. Stripping them here keeps them out of logs,
+ * doctor output, the settings UI, and the proxy connection itself.
+ */
+export function stripProxyCredentials(url) {
+    let parsed;
+    try {
+        parsed = new URL(url);
+    }
+    catch {
+        return url;
+    }
+    if (!parsed.username && !parsed.password)
+        return url; // already credential-free
+    parsed.username = '';
+    parsed.password = '';
+    return parsed.toString().replace(/\/$/, '');
+}
+/**
+ * Whether a proxy url passes validation: a blank url is the "disable proxy"
+ * marker and passes; anything else must parse as an http(s) URL. This is a
+ * pure syntax check — no connectivity check is performed.
+ */
+export function isValidProxyUrl(url) {
+    if (url.trim() === '')
+        return true;
+    try {
+        const parsed = new URL(url);
+        return parsed.protocol === 'http:' || parsed.protocol === 'https:';
+    }
+    catch {
+        return false;
+    }
+}
+export function extractProxyConfig(value) {
+    if (!value || typeof value !== 'object')
+        return undefined;
+    const raw = value;
+    // An explicitly present but blank url is the "disable proxy" marker: it lets a
+    // higher-priority layer (e.g. a project) clear a proxy set in a lower layer.
+    if (typeof raw.url === 'string' && raw.url.trim() === '') {
+        return { url: '' };
+    }
+    if (typeof raw.url !== 'string' || !raw.url.trim())
+        return undefined;
+    const url = raw.url.trim();
+    let parsed;
+    try {
+        parsed = new URL(url);
+    }
+    catch {
+        parsed = undefined;
+    }
+    // A valid url is normalized. A malformed url is kept as-is (rather than
+    // dropped) so validation can flag it and the backend factory can fail loudly
+    // instead of silently sending traffic direct to the websites.
+    const config = {
+        url: parsed && (parsed.protocol === 'http:' || parsed.protocol === 'https:')
+            ? parsed.toString().replace(/\/$/, '')
+            : url
+    };
+    if (typeof raw.username === 'string' && raw.username.trim())
+        config.username = raw.username;
+    if (typeof raw.password === 'string')
+        config.password = raw.password;
+    return config;
+}
 function extractSearxngOptions(value) {
     if (!value || typeof value !== 'object')
         return undefined;
@@ -113,6 +183,10 @@ export function extractBackendConfigOverride(file) {
     if (backends?.headless?.provider === 'local-browser') {
         override.headless = { provider: 'local-browser' };
     }
+    const proxy = extractProxyConfig(backends?.proxy);
+    if (proxy) {
+        override.proxy = proxy;
+    }
     return override;
 }
 export function validateBackendConfig(config) {
@@ -120,6 +194,23 @@ export function validateBackendConfig(config) {
     if (config.search.provider === 'searxng' && !config.search.baseUrl) {
         issues.push('search provider searxng requires backends.search.baseUrl');
     }
+    if (config.proxy && config.proxy.url.trim() !== '') {
+        let parsed;
+        try {
+            parsed = new URL(config.proxy.url);
+        }
+        catch {
+            parsed = undefined;
+        }
+        if (!isValidProxyUrl(config.proxy.url)) {
+            issues.push('backends.proxy.url must be an http or https URL');
+        }
+        else if (parsed && (parsed.username || parsed.password)) {
+            // Credentials belong in backends.proxy.username/password or the env vars
+            // below, never in the URL itself.
+            issues.push('backends.proxy.url must not include credentials (user:pass@); set PI_WEB_AGENT_PROXY_USERNAME and PI_WEB_AGENT_PROXY_PASSWORD (or backends.proxy.username / backends.proxy.password) instead');
+        }
+    }
     if (config.fetch.provider === 'firecrawl' && !config.fetch.baseUrl) {
         issues.push('fetch provider firecrawl requires backends.fetch.baseUrl');
     }
@@ -173,6 +264,11 @@ export function mergeBackendConfigLayers(...layers) {
     return layers.reduce((merged, layer) => ({
         search: mergeSearchConfig(merged.search, layer?.search),
         fetch: mergeFetchConfig(merged.fetch, layer?.fetch),
-        headless: { ...merged.headless, ...layer?.headless }
+        headless: { ...merged.headless, ...layer?.headless },
+        proxy: layer?.proxy
+            ? layer.proxy.url === ''
+                ? undefined // explicit disable overrides any proxy from lower layers
+                : { ...merged.proxy, ...layer.proxy }
+            : merged.proxy
     }), DEFAULT_BACKEND_CONFIG);
 }

package/dist/backends/factory.d.ts

@@ -8,7 +8,7 @@ import { createWebFetchHeadlessTool } from '../tools/web-fetch-headless.js';
 import { createWebFetchTool } from '../tools/web-fetch.js';
 import { createWebSearchTool } from '../tools/web-search.js';
 import type { WebFetchHeadlessResponse, WebFetchResponse, WebSearchResponse } from '../types.js';
-import { type BackendConfig } from './config.js';
+import { type BackendConfig, type ProxyConfig } from './config.js';
 export type BackendSet = {
     search: (input: {
         query: string;
@@ -30,5 +30,6 @@ export type BackendFactoryDeps = {
     createHttpFetch?: typeof createWebFetchTool;
     createFirecrawlFetch?: typeof createFirecrawlFetcher;
     createHeadlessFetch?: typeof createWebFetchHeadlessTool;
+    createProxyFetch?: (proxy: ProxyConfig) => typeof fetch;
 };
 export declare function createBackendSet(config?: BackendConfig, deps?: BackendFactoryDeps): BackendSet;