@demigodmode/pi-web-agent 1.9.0 → 1.10.0

package/CHANGELOG.md

@@ -18,6 +18,19 @@ The format is intentionally simple and release-oriented.
 ### Breaking
 - None.
 
+## [1.10.0] - 2026-08-25
+### Added
+- The zero-config default has a safety net now. When DuckDuckGo blocks a search (which happens fast on VPS and datacenter IPs), web search falls back to Tavily's keyless endpoint, no account or API key needed, so a fresh install still returns results instead of an error. Set `PI_WEB_AGENT_DISABLE_KEYLESS_FALLBACK=1` if you'd rather it just fail. (#42)
+
+### Changed
+- The keyless DuckDuckGo default is a lot harder to bot-wall. It sends real browser headers and retries once when a page looks blocked, instead of going out as an obvious script and getting bounced on the first request. (#42)
+
+### Fixed
+- Blocked searches are labeled honestly. A bot-wall page comes back as "blocked" (which triggers the fallback) instead of "couldn't parse", including the 200-status pages that mix a no-results message with a bot-wall notice. (#42)
+
+### Breaking
+- None.
+
 ## [1.9.0] - 2026-08-15
 ### Added
 - Search fanout. Ask a hard question and web_explore can now hit several of your configured search backends at once, dedupe the merged results, and rank the pages that more than one provider agreed on to the top. Off by default. Flip it to on or auto in Settings → Backends (auto only fans out when the first provider comes back thin, so easy queries stay cheap). Preview/verbose shows which providers ran and which got skipped. (#35)

package/README.md

@@ -1,131 +1,121 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/demigodmode/pi-web-agent/main/docs/public/pi-web-agent-banner.png" alt="pi-web-agent: bounded web research for Pi" width="100%">
+</p>
+
 # pi-web-agent
 
 [![CI](https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/demigodmode/pi-web-agent/actions/workflows/ci.yml)
 [![npm version](https://img.shields.io/npm/v/@demigodmode/pi-web-agent)](https://www.npmjs.com/package/@demigodmode/pi-web-agent)
 [![Docs](https://img.shields.io/badge/docs-github%20pages-blue)](https://demigodmode.github.io/pi-web-agent/)
 
-`@demigodmode/pi-web-agent` is a Pi package for web access.
+One public tool, `web_explore`, that does bounded web research for Pi: search, fetch, targeted browser rendering, ranking, and honest caveats, all behind a single call.
+
+> Most agent web tooling blurs search, fetch, rendering, and synthesis into one vague thing. `pi-web-agent` keeps that boundary simple, and it is stricter about what it actually did: bot-check pages, narrow source sets, unreadable threads, and conflicting evidence show up as caveats instead of fake confidence.
+
+## What you get
+
+- **One tool.** `web_explore` handles direct links, discovery, HTTP reads, targeted headless rendering, source ranking, source-quality checks, and caveats internally.
+- **Reads the real content behind links.** Paste a GitHub, PDF, or YouTube URL and it pulls the actual thing (GitHub files/issues/PRs from the API, PDF text, YouTube transcripts), keyless. So "summarize this PDF" or "what does this repo do" works off the source, not the page shell.
+- **Six search backends.** DuckDuckGo (keyless default), SearXNG, Brave, You.com, Exa, Tavily.
+- **Optional search fanout.** Query several backends at once, dedupe, and rank pages that more than one provider agreed on to the top. Off by default; flip it to `on` or `auto`.
+- **Honest by default.** Weak, narrow, blocked, or cautionary evidence gets flagged instead of dressed up as confidence.
+- **Bounded output.** `compact` / `preview` / `verbose` transcript modes.
+- **Zero-config to start.** Runs keyless out of the box (DuckDuckGo search, local browser, the built-in readers). Opt into hosted backends, fallback, search fanout, and per-tool output modes through config when you want more control.
 
-Most agent web tools blur search, fetch, browser rendering, and research into one vague thing. `pi-web-agent` exposes one public research tool, `web_explore`, and keeps search/fetch/headless work inside that bounded workflow.
+## Why pi-web-agent
 
-The point is keeping the model-facing boundary simple: ask `web_explore` to research a question, and it handles direct links, discovery, HTTP reads, targeted browser rendering, source ranking, source-quality checks, and caveats internally.
+Compared to other web tooling for agents:
 
-That sounds obvious, but a lot of agent tooling gets fuzzy right there. This package is meant to be stricter about what it actually did and more willing to say when a read was not good enough to trust. Bot-check pages, narrow source sets, unreadable threads, and cautionary/conflicting evidence should show up as caveats instead of fake confidence.
+- **Hands-off.** No curator, no browser windows to approve, no step that pops you out of your session. Ask `web_explore` once and the answer comes back with caveats. Nothing to babysit.
+- **Keyless by default.** Search, page reads, and the GitHub/PDF/YouTube readers all work with no API keys. Add hosted providers only when you want them.
+- **Bounded and honest.** Compact output by default, and it says when a read was not good enough instead of returning fake confidence.
 
 ## Install
 
-Compatibility notice: current `pi-web-agent` requires Pi 0.74+ because Pi packages moved to the `@earendil-works/*` scope. Update Pi before updating this package. If you are on an older Pi version, stay on `@demigodmode/pi-web-agent@0.6.x` until Pi is updated.
+> `pi-web-agent` requires Pi 0.74+ (Pi packages moved to the `@earendil-works/*` scope). Update Pi before updating this package. On older Pi, stay on `@demigodmode/pi-web-agent@0.6.x`.
 
 ```bash
 pi install npm:@demigodmode/pi-web-agent
 ```
 
-After installing, reload or restart Pi. Run `/web-agent` for the action menu, or `/web-agent doctor` to check whether the package loaded cleanly and which web backends are configured.
+Reload or restart Pi after installing, then:
 
-Headless rendering first tries a detectable Chromium-family browser: Chrome, Chromium, Edge, or Brave. If none is found, it falls back to Playwright-managed Chromium and still launches headless. Firefox/Safari-only systems can still use search and plain HTTP reads; browser-rendered fallback uses Chromium.
-
-Later on, update installed packages with:
-
-```bash
-pi update --extensions
+```text
+/web-agent doctor   # check it loaded and show configured backends
+/web-agent          # action menu
 ```
 
-## Docs
+Update later with `pi update --extensions`.
 
-Docs site:
+**Browser rendering:** headless first tries a detected Chromium-family browser (Chrome, Chromium, Edge, Brave). If none is found, it falls back to Playwright-managed Chromium. Firefox/Safari-only systems still get search and plain HTTP reads.
 
-- https://demigodmode.github.io/pi-web-agent/
+## Usage
 
-Work on the docs locally:
+Ask `web_explore` a web question:
 
-```bash
-npm run docs:dev
-```
+> Find current docs and discussions on configuring Vitest coverage with the v8 provider.
 
-Build the docs:
+Or hand it a link to read:
 
-```bash
-npm run docs:build
-```
+> Summarize this PDF: https://arxiv.org/pdf/1706.03762
 
-## Presentation modes
+If a pass comes back thin, call `web_explore` again with a narrower query.
 
-`pi-web-agent` renders web tool output in one visible mode at a time:
+## Backends
 
-- `compact` — short summary, default everywhere
-- `preview` — slightly richer bounded view
-- `verbose` — fuller bounded view
+Defaults are DuckDuckGo search, plain HTTP fetch, and local-browser headless. Switch providers from `/web-agent settings → Backends`. API keys stay in environment variables, never in config files.
 
-See the `v0.3.0` release notes for a before/after of the transcript cleanup:
+| Backend | Role | Enable with |
+| --- | --- | --- |
+| DuckDuckGo | search (default) | nothing, keyless |
+| SearXNG | search (self-hosted) | base URL |
+| Brave | search (hosted) | `PI_WEB_AGENT_BRAVE_API_KEY` |
+| You.com | search (hosted) | `YDC_API_KEY` |
+| Exa | search (hosted) | `EXA_API_KEY` |
+| Tavily | search (hosted) | `TAVILY_API_KEY` |
+| Firecrawl | fetch (self-hosted) | base URL + `PI_WEB_AGENT_FIRECRAWL_API_KEY` |
+| GitHub reader | content | `GITHUB_TOKEN` (optional, raises the rate limit) |
 
-- https://github.com/demigodmode/pi-web-agent/releases/tag/v0.3.0
+Full config shape (fallback, SearXNG/Firecrawl options, fanout): see the [self-hosted backends docs](https://demigodmode.github.io/pi-web-agent/self-hosted-backends).
 
 ## Settings
 
-Primary UI:
-
 ```text
-/web-agent settings
-```
-
-Helper commands:
-
-```text
-/web-agent doctor
-/web-agent show
+/web-agent settings                    # main UI
+/web-agent doctor                      # health check
+/web-agent show                        # effective config
 /web-agent changelog
-/web-agent reset project
-/web-agent reset global
-/web-agent mode preview
-/web-agent mode web_explore verbose
-/web-agent mode web_explore inherit
+/web-agent mode web_explore verbose    # per-tool presentation mode
+/web-agent reset project | global
 ```
 
-Config files:
+Config is JSON, and project config overrides global:
 
 ```text
 Global:  ~/.pi/agent/extensions/pi-web-agent/config.json
 Project: .pi/extensions/pi-web-agent/config.json
 ```
 
-Precedence:
-
-- built-in defaults
-- global config
-- project config
-
-Project config overrides global config.
-
-Example:
-
 ```json
 {
   "presentation": {
     "defaultMode": "compact",
-    "tools": {
-      "web_explore": { "mode": "verbose" }
-    }
+    "tools": { "web_explore": { "mode": "verbose" } }
   }
 }
 ```
 
-Backend config is also supported. Defaults remain DuckDuckGo search, plain HTTP fetch, and local-browser headless fallback with managed Chromium fallback configured. If you have a Brave Search or You.com API key, either can be selected as a hosted search backend while `web_explore` still handles page reading, ranking, and caveats itself.
+Presentation modes:
 
-Search backends: DuckDuckGo (default), SearXNG (self-hosted), Brave Search (hosted), You.com (hosted).
+- `compact`: short summary, the default everywhere
+- `preview`: slightly richer bounded view
+- `verbose`: fuller bounded view
 
-Backend settings can be changed from:
-
-```text
-/web-agent settings
-```
-
-Choose **Backends** to edit search/fetch providers, fallback behavior, and SearXNG or Firecrawl base URLs interactively. Brave Search uses `PI_WEB_AGENT_BRAVE_API_KEY`, You.com uses `YDC_API_KEY`, Exa uses `EXA_API_KEY`, and Tavily uses `TAVILY_API_KEY`. Firecrawl API keys should also stay in environment variables rather than being written into config files.
-
-For the full backend config shape, including SearXNG, Brave, You.com, Firecrawl, and fallback behavior, see:
+## Docs
 
-- https://demigodmode.github.io/pi-web-agent/self-hosted-backends
+Full docs: <https://demigodmode.github.io/pi-web-agent/>. Work on them locally with `npm run docs:dev`.
 
-## Local development
+## Development
 
 ```bash
 npm install
@@ -134,10 +124,8 @@ npm run lint
 npm run build
 ```
 
-For local Pi work, this repo includes `.pi/extensions/pi-web-agent.ts`.
-
-If Pi is already running, use `/reload` after changes.
+Local Pi work uses `.pi/extensions/pi-web-agent.ts`; run `/reload` after changes.
 
 ## License
 
-AGPL-3.0-only. See `LICENSE`.
+AGPL-3.0-only. See [LICENSE](LICENSE).

package/dist/backends/factory.js

@@ -54,7 +54,10 @@ function withSearchFallback(primary, fallback, fallbackFrom) {
             metadata: {
                 ...second.metadata,
                 fallbackFrom,
-                fallbackReason: first.error?.message ?? `${fallbackFrom} search failed.`
+                fallbackReason: first.error?.message ?? `${fallbackFrom} search failed.`,
+                // Keep the primary's fanout provenance (which providers were tried/skipped) even though
+                // the answer came from the fallback backend.
+                ...(first.metadata.fanout ? { fanout: first.metadata.fanout } : {})
             }
         };
         return { ...result, presentation: buildSearchPresentation(result) };
@@ -149,6 +152,14 @@ export function createBackendSet(config = DEFAULT_BACKEND_CONFIG, deps = {}) {
             mode: fanoutConfig.mode
         });
     }
+    // Keep the keyless Tavily safety net for the no-key DuckDuckGo default, even under fanout —
+    // it wraps whatever search ended up being (plain DDG or the fanout set) so a total failure
+    // still has somewhere to go. Opt out with PI_WEB_AGENT_DISABLE_KEYLESS_FALLBACK=1.
+    const keylessFallbackDisabled = process.env.PI_WEB_AGENT_DISABLE_KEYLESS_FALLBACK === '1';
+    const usingDuckDuckGoDefault = config.search.provider === 'duckduckgo' || !config.search.provider;
+    if (usingDuckDuckGoDefault && !keylessFallbackDisabled) {
+        search = withSearchFallback(search, createTavilySearch({ keyless: true }), 'duckduckgo');
+    }
     const httpFetch = createHttpFetch();
     let fetchPage = config.fetch.provider === 'firecrawl'
         ? config.fetch.baseUrl

package/dist/search/duckduckgo.d.ts

@@ -5,5 +5,14 @@ export type ParsedDuckDuckGoResults = {
     hasResultContainers: boolean;
 };
 export declare function buildSearchUrl(query: string): string;
-export declare function fetchDuckDuckGoHtml(query: string): Promise<string>;
+export declare const DUCKDUCKGO_HEADERS: {
+    readonly 'User-Agent': "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0";
+    readonly Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
+    readonly 'Accept-Language': "en-US,en;q=0.9";
+};
+export declare function fetchDuckDuckGoHtml(query: string, { fetchImpl, retries, sleep }?: {
+    fetchImpl?: typeof fetch;
+    retries?: number;
+    sleep?: (ms: number) => Promise<void>;
+}): Promise<string>;
 export declare function parseDuckDuckGoResults(html: string): ParsedDuckDuckGoResults;

package/dist/search/duckduckgo.js

@@ -21,12 +21,30 @@ export function buildSearchUrl(query) {
     const params = new URLSearchParams({ q: query });
     return `https://html.duckduckgo.com/html/?${params.toString()}`;
 }
-export async function fetchDuckDuckGoHtml(query) {
-    const response = await fetch(buildSearchUrl(query));
-    if (!response.ok) {
-        throw new Error(`DuckDuckGo request failed with ${response.status}`);
+export const DUCKDUCKGO_HEADERS = {
+    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0',
+    Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+    'Accept-Language': 'en-US,en;q=0.9'
+};
+const defaultSleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+export async function fetchDuckDuckGoHtml(query, { fetchImpl = fetch, retries = 1, sleep = defaultSleep } = {}) {
+    let lastError;
+    for (let attempt = 0; attempt <= retries; attempt += 1) {
+        try {
+            const response = await fetchImpl(buildSearchUrl(query), { headers: { ...DUCKDUCKGO_HEADERS } });
+            if (!response.ok) {
+                throw new Error(`DuckDuckGo request failed with ${response.status}`);
+            }
+            return response.text();
+        }
+        catch (error) {
+            lastError = error;
+            if (attempt < retries) {
+                await sleep(500);
+            }
+        }
     }
-    return response.text();
+    throw lastError instanceof Error ? lastError : new Error('DuckDuckGo request failed');
 }
 export function parseDuckDuckGoResults(html) {
     const $ = cheerio.load(html);

package/dist/search/tavily.d.ts

@@ -1,6 +1,7 @@
 import type { WebSearchResponse } from '../types.js';
-export declare function createTavilySearchTool({ apiKey, fetchImpl }: {
+export declare function createTavilySearchTool({ apiKey, keyless, fetchImpl }: {
     apiKey?: string;
+    keyless?: boolean;
     fetchImpl?: typeof fetch;
 }): ({ query }: {
     query: string;

package/dist/search/tavily.js

@@ -17,7 +17,7 @@ function normalizeResults(response) {
         ];
     });
 }
-export function createTavilySearchTool({ apiKey, fetchImpl = fetch }) {
+export function createTavilySearchTool({ apiKey, keyless = false, fetchImpl = fetch }) {
     return async function tavilySearch({ query }) {
         const normalizedQuery = query.trim();
         if (!normalizedQuery) {
@@ -28,7 +28,7 @@ export function createTavilySearchTool({ apiKey, fetchImpl = fetch }) {
                 error: { code: 'INVALID_QUERY', message: 'Query must not be empty.' }
             });
         }
-        if (!apiKey?.trim()) {
+        if (!apiKey?.trim() && !keyless) {
             return resultWithPresentation({
                 status: 'error',
                 results: [],
@@ -45,7 +45,9 @@ export function createTavilySearchTool({ apiKey, fetchImpl = fetch }) {
                 headers: {
                     Accept: 'application/json',
                     'Content-Type': 'application/json',
-                    Authorization: `Bearer ${apiKey}`
+                    ...(apiKey?.trim()
+                        ? { Authorization: `Bearer ${apiKey}` }
+                        : { 'X-Tavily-Access-Mode': 'keyless' })
                 },
                 body: JSON.stringify({ query: normalizedQuery, max_results: 10 })
             });

package/dist/tools/web-search.js

@@ -27,7 +27,11 @@ function htmlLooksBlocked(html) {
         normalized.includes('challenge') ||
         normalized.includes('verify you are human') ||
         normalized.includes('are you a robot') ||
-        normalized.includes('unusual traffic'));
+        normalized.includes('unusual traffic') ||
+        normalized.includes('automated requests') ||
+        normalized.includes('automated queries') ||
+        normalized.includes('detected unusual') ||
+        normalized.includes('too many requests'));
 }
 export function createWebSearchTool({ searchHtml = fetchDuckDuckGoHtml, cache = createTtlCache({ ttlMs: 30_000 }) } = {}) {
     return async function webSearch({ query }) {
@@ -57,8 +61,14 @@ export function createWebSearchTool({ searchHtml = fetchDuckDuckGoHtml, cache =
             };
         }
         try {
-            const html = await searchHtml(normalizedQuery);
-            const parsed = parseDuckDuckGoResults(html);
+            let html = await searchHtml(normalizedQuery);
+            let parsed = parseDuckDuckGoResults(html);
+            // A 200-OK bot-wall reads as a successful fetch, so the fetch-layer retry never sees it.
+            // Give a page that looks blocked one more shot here before we classify it.
+            if (parsed.results.length === 0 && htmlLooksBlocked(html)) {
+                html = await searchHtml(normalizedQuery);
+                parsed = parseDuckDuckGoResults(html);
+            }
             if (parsed.results.length > 0) {
                 const result = {
                     status: 'ok',
@@ -71,14 +81,16 @@ export function createWebSearchTool({ searchHtml = fetchDuckDuckGoHtml, cache =
                     presentation: buildSearchPresentation(result)
                 };
             }
-            if (parsed.noResults) {
+            // Check for a bot-wall before "no results": a page can carry both markers, and BLOCKED is
+            // the honest call since it routes to the fallback instead of a dead end.
+            if (htmlLooksBlocked(html)) {
                 const result = {
                     status: 'error',
                     results: [],
                     metadata: { backend: 'duckduckgo', cacheHit: false },
                     error: {
-                        code: 'NO_RESULTS',
-                        message: 'DuckDuckGo returned no usable results for this query.'
+                        code: 'BLOCKED',
+                        message: 'DuckDuckGo search appears to be blocked or rate limited.'
                     }
                 };
                 return {
@@ -86,14 +98,14 @@ export function createWebSearchTool({ searchHtml = fetchDuckDuckGoHtml, cache =
                     presentation: buildSearchPresentation(result)
                 };
             }
-            if (htmlLooksBlocked(html)) {
+            if (parsed.noResults) {
                 const result = {
                     status: 'error',
                     results: [],
                     metadata: { backend: 'duckduckgo', cacheHit: false },
                     error: {
-                        code: 'BLOCKED',
-                        message: 'DuckDuckGo search appears to be blocked or rate limited.'
+                        code: 'NO_RESULTS',
+                        message: 'DuckDuckGo returned no usable results for this query.'
                     }
                 };
                 return {

package/dist/types.d.ts

@@ -20,7 +20,7 @@ export type ToolError = {
 export type SearchMetadata = {
     backend: 'duckduckgo' | 'searxng' | 'brave' | 'youcom' | 'exa' | 'tavily';
     cacheHit: boolean;
-    fallbackFrom?: 'searxng' | 'brave' | 'youcom' | 'exa' | 'tavily';
+    fallbackFrom?: 'searxng' | 'brave' | 'youcom' | 'exa' | 'tavily' | 'duckduckgo';
     fallbackReason?: string;
     fanout?: FanoutMetadata;
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@demigodmode/pi-web-agent",
-  "version": "1.9.0",
+  "version": "1.10.0",
   "description": "Pi package for reliable web access with explicit search, fetch, and headless boundaries.",
   "type": "module",
   "main": "./dist/extension.js",