@demigodmode/pi-web-agent 0.1.0

package/README.md

@@ -0,0 +1,176 @@
+# pi-web-agent
+
+`@demigodmode/pi-web-agent` is a Pi package for reliable web access.
+
+It is built around a simple rule: searching for a page is not the same thing as reading it. This package keeps those steps separate, prefers plain HTTP by default, and is designed to say "I couldn't read this reliably" instead of making something up.
+
+## What it does
+
+The package is built around three tools:
+
+- `web_search` finds relevant pages and returns titles, URLs, and snippets
+- `web_fetch` fetches a specific page over plain HTTP and tries to extract readable content
+- `web_fetch_headless` is the explicit browser-based path for pages that need rendering
+
+The boundary between those tools is intentional.
+
+`web_search` is for discovery. It should not imply that a page was fetched.
+
+`web_fetch` is for reading a page over HTTP. If the result looks weak, incomplete, blocked, or too script-heavy, it should return `needs_headless` instead of bluffing.
+
+`web_fetch_headless` exists for the cases where a browser really is required. It is opt-in only.
+
+## Why this exists
+
+A lot of web tooling in coding agents gets fuzzy in exactly the wrong places. Search results get treated like page reads. Browser fallback happens behind the scenes. Failures get softened into fake confidence.
+
+This package is trying to do the opposite.
+
+The rules are straightforward:
+
+- no hidden browser launch
+- no automatic HTTP-to-headless fallback
+- no claiming a page was read when only snippets were available
+- explicit structured failure when the result is incomplete or blocked
+
+## What makes it different
+
+The main thing is the contract.
+
+`web_search` discovers sources.
+
+`web_fetch` reads over HTTP only.
+
+`web_fetch_headless` is the explicit browser path.
+
+That separation is the whole point. It makes failures easier to reason about and avoids the weird behavior where a tool quietly changes execution mode behind your back.
+
+## Install
+
+Install it through Pi:
+
+```bash
+pi install npm:@demigodmode/pi-web-agent
+```
+
+Update installed packages later with:
+
+```bash
+pi update
+```
+
+If you just want to inspect the package from npm directly, the package name is:
+
+```bash
+npm view @demigodmode/pi-web-agent
+```
+
+## Current status
+
+This repo is in early MVP shape, but it is no longer just a design doc.
+
+Right now it has:
+
+- a TypeScript project scaffold
+- shared result and status contracts
+- a DuckDuckGo HTML parser for `web_search`
+- an HTTP fetch path with readability-based extraction and conservative escalation to `needs_headless`
+- a real browser-backed `web_fetch_headless` implementation with local browser resolution
+- repo-local Pi extension wiring for development
+- a test suite around parser behavior, contracts, extraction, caching, and tool adapters
+- optional smoke coverage for local installed browsers
+
+So the project is real and usable, but still early.
+
+## Example behavior
+
+These are conceptual examples of the contract the package is aiming to expose.
+
+### Search
+
+`web_search("pi coding agent")`
+
+Returns discovery results like:
+
+- title
+- URL
+- snippet
+
+It does not imply the page was fetched.
+
+### HTTP fetch
+
+`web_fetch("https://example.com/article")`
+
+If the page is readable over plain HTTP, it should return extracted content.
+
+If the page looks too script-heavy, too thin, blocked, or otherwise unreliable, it should return `needs_headless` instead of pretending the extraction is good enough.
+
+### Explicit headless fetch
+
+`web_fetch_headless("https://example.com/app")`
+
+This is the browser-based path for pages that really need rendering.
+
+This path now launches a local browser explicitly, waits for the rendered page to settle, and then extracts readable content from the rendered HTML.
+
+## Local development
+
+Install dependencies:
+
+```bash
+npm install
+```
+
+Run tests with coverage:
+
+```bash
+npm test
+```
+
+Run the typecheck used as lint:
+
+```bash
+npm run lint
+```
+
+Build the project:
+
+```bash
+npm run build
+```
+
+To run the optional real-browser smoke test for headless fetch, set `PI_HEADLESS_SMOKE=1` before running Vitest. It stays skipped by default so local browser install differences do not make the normal test suite flaky.
+
+Coverage is now part of the normal `npm test` flow. Vitest prints a text summary in the terminal and writes the full HTML report to `coverage/`.
+
+### Trying it in Pi locally
+
+This repo includes a project-local Pi extension entrypoint at `.pi/extensions/pi-web-agent.ts` for development and hot reload.
+
+For the published npm package, Pi loads the compiled runtime from `dist/extension.js` via the `pi.extensions` entry in `package.json`.
+
+After starting Pi in this project, use `/reload` if you change the extension code and want Pi to pick up the latest version.
+
+## Project layout
+
+The code is split into small modules on purpose.
+
+- `src/extension.ts` - package entry surface
+- `src/tools/` - thin tool adapters
+- `src/search/` - search backend logic
+- `src/fetch/` - HTTP and headless fetch logic
+- `src/extract/` - readable-content extraction
+- `src/cache/` - small cache utilities
+- `src/types.ts` - shared contracts
+- `tests/` - parser, contract, extraction, fetch, and adapter tests
+
+## Near-term next steps
+
+The next chunk of work is pretty clear:
+
+- keep tightening weak-content escalation on tricky HTTP targets
+- improve cleanup of noisy rendered-page extraction on busy sites
+- expand fixtures and end-to-end coverage
+- add alternate search backends behind a first-class provider abstraction
+

package/dist/cache/ttl-cache.d.ts

@@ -0,0 +1,8 @@
+export declare function createCacheKey(parts: Array<string | number | boolean>): string;
+export declare function createTtlCache<T>({ ttlMs, now }: {
+    ttlMs: number;
+    now?: () => number;
+}): {
+    get(key: string): T | undefined;
+    set(key: string, value: T): void;
+};

package/dist/cache/ttl-cache.js

@@ -0,0 +1,21 @@
+export function createCacheKey(parts) {
+    return JSON.stringify(parts);
+}
+export function createTtlCache({ ttlMs, now = () => Date.now() }) {
+    const entries = new Map();
+    return {
+        get(key) {
+            const entry = entries.get(key);
+            if (!entry)
+                return undefined;
+            if (entry.expiresAt <= now()) {
+                entries.delete(key);
+                return undefined;
+            }
+            return entry.value;
+        },
+        set(key, value) {
+            entries.set(key, { value, expiresAt: now() + ttlMs });
+        }
+    };
+}

package/dist/extension.d.ts

@@ -0,0 +1,2 @@
+import type { ExtensionAPI } from '@mariozechner/pi-coding-agent';
+export default function extension(pi: ExtensionAPI): void;

package/dist/extension.js

@@ -0,0 +1,114 @@
+import { Type } from '@sinclair/typebox';
+import { createWebExploreTool } from './tools/web-explore.js';
+import { createWebFetchTool } from './tools/web-fetch.js';
+import { createWebFetchHeadlessTool } from './tools/web-fetch-headless.js';
+import { createWebSearchTool } from './tools/web-search.js';
+const WEB_EXPLORE_REMINDER_TYPE = 'pi-web-agent-web-explore-reminder';
+const WEB_EXPLORE_REMINDER = 'web_explore has already been used for this research task. Only call low-level web tools if there is a specific unresolved gap. Do not keep searching or fetching just for extra confirmation.';
+function hasSuccessfulWebExplore(messages) {
+    return messages.some((message) => message.role === 'toolResult' && message.toolName === 'web_explore' && !message.isError);
+}
+function hasWebExploreReminder(messages) {
+    return messages.some((message) => message.role === 'custom' && message.customType === WEB_EXPLORE_REMINDER_TYPE);
+}
+export default function extension(pi) {
+    const webSearch = createWebSearchTool();
+    const webFetch = createWebFetchTool();
+    const webFetchHeadless = createWebFetchHeadlessTool();
+    const webExplore = createWebExploreTool();
+    pi.on('before_agent_start', async (event) => {
+        return {
+            systemPrompt: `${event.systemPrompt}\n\n` +
+                'For web research questions that require finding and comparing multiple sources, prefer web_explore. ' +
+                'Use web_search, web_fetch, and web_fetch_headless for direct/manual operations like explicit search calls, specific URL reads, or debugging. ' +
+                'After using web_explore, only call low-level web tools if there is a specific unresolved gap. ' +
+                'Do not keep searching or fetching just for extra confirmation.'
+        };
+    });
+    pi.on('context', async (event) => {
+        if (!hasSuccessfulWebExplore(event.messages) || hasWebExploreReminder(event.messages)) {
+            return { messages: event.messages };
+        }
+        return {
+            messages: [
+                ...event.messages,
+                {
+                    role: 'custom',
+                    customType: WEB_EXPLORE_REMINDER_TYPE,
+                    content: WEB_EXPLORE_REMINDER,
+                    display: false,
+                    timestamp: Date.now()
+                }
+            ]
+        };
+    });
+    pi.registerTool({
+        name: 'web_search',
+        label: 'Web Search',
+        description: 'Direct search tool for manual discovery of links and snippets. Use for explicit search requests or when the user wants raw search results. Prefer web_explore for broader research questions.',
+        parameters: Type.Object({
+            query: Type.String({ description: 'Search query.' })
+        }),
+        async execute(_toolCallId, params) {
+            const result = await webSearch({ query: params.query });
+            return {
+                content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
+                details: result,
+                isError: result.status === 'error'
+            };
+        }
+    });
+    pi.registerTool({
+        name: 'web_fetch',
+        label: 'Web Fetch',
+        description: 'Direct HTTP page fetch for a specific URL. Use when the user wants one page read directly. Prefer web_explore for broader research across multiple sources.',
+        parameters: Type.Object({
+            url: Type.String({ description: 'HTTP or HTTPS URL to fetch.' })
+        }),
+        async execute(_toolCallId, params) {
+            const result = await webFetch({ url: params.url });
+            return {
+                content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
+                details: result,
+                isError: result.status === 'error'
+            };
+        }
+    });
+    pi.registerTool({
+        name: 'web_fetch_headless',
+        label: 'Web Fetch Headless',
+        description: 'Direct headless page fetch for a specific URL when browser rendering is explicitly needed. Prefer web_explore for research tasks; it decides headless escalation internally.',
+        parameters: Type.Object({
+            url: Type.String({ description: 'HTTP or HTTPS URL to fetch in headless mode.' })
+        }),
+        async execute(_toolCallId, params) {
+            const result = await webFetchHeadless({ url: params.url });
+            return {
+                content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
+                details: result,
+                isError: result.status === 'error'
+            };
+        }
+    });
+    pi.registerTool({
+        name: 'web_explore',
+        label: 'Web Explore',
+        description: 'Research a web question using bounded search/fetch passes, source ranking, and targeted headless escalation. Prefer this for multi-source web research, current docs/discussion lookups, and recommendation summaries. Use this instead of chaining low-level web tools for the same research task.',
+        parameters: Type.Object({
+            query: Type.String({ description: 'Web research question to explore.' })
+        }),
+        async execute(_toolCallId, params) {
+            const result = await webExplore({ query: params.query });
+            return {
+                content: [
+                    {
+                        type: 'text',
+                        text: result.status === 'ok' ? result.text : JSON.stringify(result, null, 2)
+                    }
+                ],
+                details: result,
+                isError: result.status === 'error'
+            };
+        }
+    });
+}

package/dist/extract/readability.d.ts

@@ -0,0 +1,8 @@
+import type { ExtractedContent } from '../types.js';
+export type ReadableExtractionMode = 'readability' | 'fallback';
+export type SafeReadableExtraction = {
+    mode: ReadableExtractionMode;
+    content: ExtractedContent;
+};
+export declare function extractReadableContent(html: string, maxLength?: number): ExtractedContent;
+export declare function extractReadableContentSafely(html: string, maxLength?: number): SafeReadableExtraction;

package/dist/extract/readability.js

@@ -0,0 +1,93 @@
+import { Readability } from '@mozilla/readability';
+import { JSDOM, VirtualConsole } from 'jsdom';
+export function extractReadableContent(html, maxLength = 4000) {
+    let stylesheetError;
+    const virtualConsole = new VirtualConsole();
+    virtualConsole.on('jsdomError', (error) => {
+        if (!stylesheetError && error.message.includes('Could not parse CSS stylesheet')) {
+            stylesheetError = error;
+        }
+    });
+    const dom = new JSDOM(html, {
+        url: 'https://example.com',
+        virtualConsole
+    });
+    if (stylesheetError) {
+        throw stylesheetError;
+    }
+    const article = new Readability(dom.window.document).parse();
+    const rawText = (article?.textContent ?? dom.window.document.body.textContent ?? '').trim();
+    const text = rawText.slice(0, maxLength);
+    const fallbackTitle = dom.window.document.title || undefined;
+    return {
+        title: article?.title ?? fallbackTitle,
+        byline: article?.byline || undefined,
+        text
+    };
+}
+function decodeHtmlEntities(text) {
+    return text
+        .replace(/ /gi, ' ')
+        .replace(/&/gi, '&')
+        .replace(/&lt;/gi, '<')
+        .replace(/&gt;/gi, '>')
+        .replace(/&quot;/gi, '"')
+        .replace(/&#39;/gi, "'")
+        .replace(/&#x27;/gi, "'")
+        .replace(/&#x2F;/gi, '/')
+        .replace(/&#(\d+);/g, (_, code) => String.fromCharCode(Number(code)))
+        .replace(/&#x([\da-f]+);/gi, (_, code) => String.fromCharCode(parseInt(code, 16)));
+}
+function extractTitle(html) {
+    const match = html.match(/<title[^>]*>([\s\S]*?)<\/title>/i);
+    if (!match)
+        return undefined;
+    return decodeHtmlEntities(match[1].replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim()) || undefined;
+}
+function stripTagContent(html, tagName) {
+    return html.replace(new RegExp(`<${tagName}\\b[^>]*>[\\s\\S]*?<\\/${tagName}>`, 'gi'), ' ');
+}
+function extractPreferredSection(html) {
+    const mainMatch = html.match(/<main\b[^>]*>([\s\S]*?)<\/main>/i);
+    if (mainMatch)
+        return mainMatch[1];
+    const articleMatch = html.match(/<article\b[^>]*>([\s\S]*?)<\/article>/i);
+    if (articleMatch)
+        return articleMatch[1];
+    const bodyMatch = html.match(/<body\b[^>]*>([\s\S]*?)<\/body>/i);
+    if (bodyMatch)
+        return bodyMatch[1];
+    return html;
+}
+function extractFallbackText(html, maxLength) {
+    const title = extractTitle(html);
+    let section = extractPreferredSection(html);
+    section = stripTagContent(section, 'script');
+    section = stripTagContent(section, 'style');
+    section = stripTagContent(section, 'noscript');
+    section = stripTagContent(section, 'svg');
+    section = stripTagContent(section, 'template');
+    const text = decodeHtmlEntities(section)
+        .replace(/<[^>]+>/g, ' ')
+        .replace(/\s+/g, ' ')
+        .trim()
+        .slice(0, maxLength);
+    return {
+        title,
+        text
+    };
+}
+export function extractReadableContentSafely(html, maxLength = 4000) {
+    try {
+        return {
+            mode: 'readability',
+            content: extractReadableContent(html, maxLength)
+        };
+    }
+    catch {
+        return {
+            mode: 'fallback',
+            content: extractFallbackText(html, maxLength)
+        };
+    }
+}

package/dist/fetch/browser-resolution.d.ts

@@ -0,0 +1,15 @@
+export type BrowserResolutionResult = {
+    ok: true;
+    executablePath: string;
+    browser: 'configured' | 'chrome' | 'edge';
+} | {
+    ok: false;
+    error: {
+        code: 'BROWSER_NOT_FOUND' | 'CONFIGURED_BROWSER_NOT_FOUND';
+        message: string;
+    };
+};
+export declare function resolveBrowserExecutable({ configuredPath, fileExists }: {
+    configuredPath?: string;
+    fileExists?: (path: string) => Promise<boolean>;
+}): Promise<BrowserResolutionResult>;

package/dist/fetch/browser-resolution.js

@@ -0,0 +1,55 @@
+const WINDOWS_CANDIDATES = {
+    chrome: [
+        'C:/Program Files/Google/Chrome/Application/chrome.exe',
+        'C:/Program Files (x86)/Google/Chrome/Application/chrome.exe'
+    ],
+    edge: [
+        'C:/Program Files/Microsoft/Edge/Application/msedge.exe',
+        'C:/Program Files (x86)/Microsoft/Edge/Application/msedge.exe'
+    ]
+};
+export async function resolveBrowserExecutable({ configuredPath, fileExists = defaultFileExists }) {
+    if (configuredPath) {
+        if (await fileExists(configuredPath)) {
+            return {
+                ok: true,
+                executablePath: configuredPath,
+                browser: 'configured'
+            };
+        }
+        return {
+            ok: false,
+            error: {
+                code: 'CONFIGURED_BROWSER_NOT_FOUND',
+                message: `Configured browser path was not found: ${configuredPath}`
+            }
+        };
+    }
+    for (const path of WINDOWS_CANDIDATES.chrome) {
+        if (await fileExists(path)) {
+            return { ok: true, executablePath: path, browser: 'chrome' };
+        }
+    }
+    for (const path of WINDOWS_CANDIDATES.edge) {
+        if (await fileExists(path)) {
+            return { ok: true, executablePath: path, browser: 'edge' };
+        }
+    }
+    return {
+        ok: false,
+        error: {
+            code: 'BROWSER_NOT_FOUND',
+            message: 'No compatible local browser was found for headless fetch.'
+        }
+    };
+}
+async function defaultFileExists(path) {
+    try {
+        const { access } = await import('node:fs/promises');
+        await access(path);
+        return true;
+    }
+    catch {
+        return false;
+    }
+}

package/dist/fetch/headless-fetch.d.ts

@@ -0,0 +1,18 @@
+import { type BrowserResolutionResult } from './browser-resolution.js';
+import type { WebFetchHeadlessResponse } from '../types.js';
+export declare function headlessFetch(url: string, { configuredPath, resolveBrowser, launchBrowser, now }?: {
+    configuredPath?: string;
+    resolveBrowser?: (options?: {
+        configuredPath?: string;
+    }) => Promise<BrowserResolutionResult>;
+    launchBrowser?: (options: {
+        executablePath: string;
+    }) => Promise<{
+        newContext: () => Promise<{
+            newPage: () => Promise<any>;
+            close: () => Promise<void>;
+        }>;
+        close: () => Promise<void>;
+    }>;
+    now?: () => number;
+}): Promise<WebFetchHeadlessResponse>;

package/dist/fetch/headless-fetch.js

@@ -0,0 +1,87 @@
+import { chromium } from 'playwright-core';
+import { extractReadableContentSafely } from '../extract/readability.js';
+import { resolveBrowserExecutable } from './browser-resolution.js';
+function cleanupRenderedText(text) {
+    let cleaned = text.replace(/(Show more)(\s+\1){1,}/gi, '$1');
+    cleaned = cleaned.replace(/(Privacy Terms)(\s+\1){1,}/gi, '$1');
+    cleaned = cleaned.replace(/\s+/g, ' ').trim();
+    return cleaned;
+}
+export async function headlessFetch(url, { configuredPath, resolveBrowser = (options) => resolveBrowserExecutable({ configuredPath: options?.configuredPath }), launchBrowser = ({ executablePath }) => chromium.launch({ executablePath, headless: true }), now = () => Date.now() } = {}) {
+    const resolved = await resolveBrowser({ configuredPath });
+    if (!resolved.ok) {
+        return {
+            status: 'error',
+            url,
+            metadata: { method: 'headless', cacheHit: false },
+            error: resolved.error
+        };
+    }
+    let browser;
+    let context;
+    let page;
+    try {
+        browser = await launchBrowser({ executablePath: resolved.executablePath });
+        context = await browser.newContext();
+        page = await context.newPage();
+        const startedAt = now();
+        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 20000 });
+        await page.waitForLoadState('load', { timeout: 10000 });
+        await page.waitForLoadState('networkidle', { timeout: 5000 }).catch(() => undefined);
+        const html = await page.content();
+        const finishedAt = now();
+        const extraction = extractReadableContentSafely(html);
+        const cleanedContent = {
+            ...extraction.content,
+            text: cleanupRenderedText(extraction.content.text)
+        };
+        if (!cleanedContent.text || cleanedContent.text.length < 40) {
+            return {
+                status: 'blocked',
+                url,
+                metadata: {
+                    method: 'headless',
+                    cacheHit: false,
+                    browser: resolved.browser,
+                    navigationMs: finishedAt - startedAt
+                },
+                error: {
+                    code: 'HEADLESS_EXTRACTION_WEAK',
+                    message: 'Rendered page did not produce enough readable content.'
+                }
+            };
+        }
+        return {
+            status: 'ok',
+            url,
+            content: cleanedContent,
+            metadata: {
+                method: 'headless',
+                cacheHit: false,
+                browser: resolved.browser,
+                navigationMs: finishedAt - startedAt,
+                truncated: cleanedContent.text.length >= 4000
+            }
+        };
+    }
+    catch (error) {
+        return {
+            status: 'error',
+            url,
+            metadata: {
+                method: 'headless',
+                cacheHit: false,
+                browser: resolved.browser
+            },
+            error: {
+                code: 'HEADLESS_NAVIGATION_FAILED',
+                message: error instanceof Error ? error.message : 'Unknown headless navigation failure.'
+            }
+        };
+    }
+    finally {
+        await page?.close?.().catch(() => undefined);
+        await context?.close?.().catch(() => undefined);
+        await browser?.close?.().catch(() => undefined);
+    }
+}

package/dist/fetch/http-fetch.d.ts

@@ -0,0 +1,4 @@
+import type { WebFetchResponse } from '../types.js';
+export declare function createHttpFetcher({ fetchImpl }?: {
+    fetchImpl?: typeof fetch;
+}): (url: string) => Promise<WebFetchResponse>;

package/dist/fetch/http-fetch.js

@@ -0,0 +1,50 @@
+import { extractReadableContentSafely } from '../extract/readability.js';
+function looksLikeScriptShell(html) {
+    const lower = html.toLowerCase();
+    return lower.includes('<script') && (lower.includes('id="app"') || lower.includes('id="root"'));
+}
+function isWeakHttpContent(options) {
+    const normalizedText = options.text.replace(/\s+/g, ' ').trim();
+    const normalizedHtml = options.html.replace(/\s+/g, ' ').trim();
+    const textLength = normalizedText.length;
+    const htmlLength = normalizedHtml.length;
+    const hasGenericShellMarker = /enable javascript|javascript required|please turn on javascript/i.test(options.html);
+    const veryShortBody = textLength > 0 && textLength < 120;
+    const lowDensity = htmlLength > 0 && textLength / htmlLength < 0.02;
+    return veryShortBody && (lowDensity || hasGenericShellMarker);
+}
+export function createHttpFetcher({ fetchImpl = fetch } = {}) {
+    return async function httpFetch(url) {
+        const response = await fetchImpl(url);
+        const contentType = response.headers.get('content-type') ?? '';
+        if (!contentType.includes('text/html')) {
+            return {
+                status: 'unsupported',
+                url: response.url,
+                metadata: { method: 'http', cacheHit: false, contentType }
+            };
+        }
+        const html = await response.text();
+        const extraction = extractReadableContentSafely(html);
+        const content = extraction.content;
+        if (looksLikeScriptShell(html) ||
+            content.text.length < 40 ||
+            isWeakHttpContent({ html, title: content.title, text: content.text })) {
+            return {
+                status: 'needs_headless',
+                url: response.url,
+                metadata: { method: 'http', cacheHit: false, contentType },
+                error: {
+                    code: 'WEAK_EXTRACTION',
+                    message: 'HTTP extraction was not reliable enough.'
+                }
+            };
+        }
+        return {
+            status: 'ok',
+            url: response.url,
+            content,
+            metadata: { method: 'http', cacheHit: false, contentType, truncated: content.text.length >= 4000 }
+        };
+    };
+}