@bacnh85/pi-web 0.3.0 → 0.4.3

package/README.md

@@ -1,31 +1,23 @@
-# pi-web
+# pi-web v0.4
 
-Pi extension for web search, readable page extraction, Firecrawl scraping/crawling, and Crawl4AI headless browser crawling.
+Pi extension for **unified web search, content extraction, site crawling, and page capture**.
 
-## Install
+Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, and Crawl4AI — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.
 
-Install the published package from npm:
+## Install
 
 ```bash
 pi install npm:@bacnh85/pi-web
 ```
 
-From this repository checkout, install only this extension package:
+From this repository:
 
 ```bash
-cd extensions/pi-web
-npm install
-cd ../..
-
 pi install ./extensions/pi-web
 # or test directly
 pi -e ./extensions/pi-web
 ```
 
-The package manifest points Pi directly at `./index.ts`, so published npm installs and local installs load the same extension entrypoint.
-
-If you manually copy this directory instead of using `pi install`, run `npm install --omit=dev` in the copied `pi-web` directory so readable-content dependencies such as `@mozilla/readability` are present.
-
 ## Configuration
 
 Environment lookup order:
@@ -33,79 +25,198 @@ Environment lookup order:
 1. Process environment
 2. Current working directory `.env.local`
 3. Current working directory `.env`
-4. Pi global config `.env.local` (`$PI_CODING_AGENT_DIR/.env.local` when set; otherwise `~/.pi/agent/.env.local`)
-5. Pi global config `.env` (`$PI_CODING_AGENT_DIR/.env` when set; otherwise `~/.pi/agent/.env`)
+4. Pi global config `~/.pi/agent/.env.local`
+5. Pi global config `~/.pi/agent/.env`
 
 Variables:
 
-- `BRAVE_API_KEY` — required for Brave Search.
-- `SEARXNG_BASE_URL` — optional; defaults to `http://172.30.55.22:8888`.
-- `FIRECRAWL_API_URL` — optional; defaults to `https://api.firecrawl.dev/v2`.
-- `FIRECRAWL_API_KEY` — required for hosted Firecrawl, optional for self-hosted instances without auth.
-- `FIRECRAWL_TIMEOUT_MS` — optional request timeout, default `60000`.
-- `CRAWL4AI_BASE_URL` — optional; defaults to `http://172.30.55.22:11235`.
-- `CRAWL4AI_API_TOKEN` — required if Crawl4AI server has auth enabled (v0.9+ default).
-- `CRAWL4AI_TIMEOUT_MS` — optional request timeout, default `60000`.
+| Variable | Required | Default | Notes |
+|---|---|---|---|
+| `BRAVE_API_KEY` | No (1) | — | Brave Search API key |
+| `SEARXNG_BASE_URL` | No | `http://172.30.55.22:8888` | Self-hosted SearXNG |
+| `FIRECRAWL_API_URL` | No | `https://api.firecrawl.dev/v2` | Self-hosted or hosted |
+| `FIRECRAWL_API_KEY` | No (2) | — | Required for hosted Firecrawl |
+| `CRAWL4AI_API_URL` | No | `http://172.30.55.22:11235` | Self-hosted Crawl4AI |
+| `CRAWL4AI_API_TOKEN` | No (3) | — | Required if Crawl4AI auth enabled |
+
+> (1) At least one search backend (SearXNG, Brave, or Firecrawl) must be configured for `web_search`.
+> (2) Required for hosted Firecrawl; optional for self-hosted instances without auth.
+> (3) Required for Crawl4AI v0.9+ default config.
 
-Secrets are never printed; status reports only show presence/source.
+Secrets are never printed; `web_status` reports only presence/source.
 
 ## Tools
 
-Brave:
+### `web_search` — Unified search
 
-- `brave_search` — search web results, optionally fetch readable result content.
+Searches the web. Auto-selects backends adaptively: SearXNG for broad self-hosted discovery, Brave for precision-sensitive queries and `include_content`, Firecrawl as last resort.
 
-Web Content:
+```
+web_search query="ansible podman quadlet" count=5
+web_search query="ansible documentation" backend=brave count=10
+web_search query="latest python release" engines="google,github"
+web_search query="riven media" include_content=true
+```
+
+Parameters:
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `query` | string | — | Search query |
+| `count` | number | 5 | Number of results (max 20) |
+| `freshness` | string | — | Time filter: `pw`, `pm`, `py`, or `YYYY-MM-DDtoYYYY-MM-DD` |
+| `country` | string | `US` | Two-letter country code |
+| `backend` | string | `auto` | Force backend: `auto`, `searxng`, `brave`, `firecrawl` |
+| `engines` | string | — | SearXNG engine override, e.g. `google,github` |
+| `include_content` | boolean | false | Fetch page content alongside results |
+| `content_chars` | number | 5000 | Max content chars per result |
 
-- `web_content` — fetch a URL and extract readable markdown (formerly `brave_content`; renamed in v0.2.0 because it does not use the Brave API).
+**Auto-selection behavior:**
 
-SearXNG:
+1. **SearXNG** — first for broad/general discovery, especially when `engines` is supplied.
+2. **Brave** — first for precision-sensitive queries (`site:`, quoted phrases, docs/API/source lookups, short proper-name queries) and whenever `include_content` is true. Requires `BRAVE_API_KEY`.
+3. **Firecrawl Search** — last resort. ⚠️ Poor semantic accuracy on domain-specific/ambiguous queries (e.g., "Riven" returns League of Legends results). Prefer SearXNG or Brave for precision.
 
-- `searxng_search` — search web results through a configured self-hosted SearXNG metasearch instance.
+Tool output includes search diagnostics showing attempted backends and the selected backend.
 
-Firecrawl:
+Use `backend` parameter to force a specific backend when needed.
 
-- `firecrawl_search` — search web/news/images, optionally scrape result markdown.
-- `firecrawl_scrape` — scrape a URL as markdown/html/links/summary/json.
-- `firecrawl_map` — discover URLs for a site.
-- `firecrawl_crawl` — start a conservative crawl, optionally polling for results.
+### `web_extract` — Unified content extraction
 
-Crawl4AI (requires running Docker server at `CRAWL4AI_BASE_URL`):
+Extracts readable content from a URL. Auto-selects backend: static (JSDOM) → dynamic (Firecrawl) → full (Crawl4AI), with extraction diagnostics showing fallback attempts.
 
-- `crawl4ai_scrape` — extract clean LLM-ready markdown with content filtering (fit/raw/bm25/llm).
-- `crawl4ai_crawl` — batch crawl 1-100 URLs with full result data (HTML, markdown, links, media).
-- `crawl4ai_stream` — streaming crawl via SSE-style newline-delimited JSON.
-- `crawl4ai_screenshot` — full-page PNG screenshot capture.
-- `crawl4ai_pdf` — PDF document generation.
-- `crawl4ai_status` — check Crawl4AI server health, version, and auth status.
+```
+web_extract url="https://docs.ansible.com/..."
+web_extract url="https://riven.tv/" mode=static
+web_extract url="https://example.com" mode=dynamic prompt="Extract pricing plans"
+```
 
-Utility:
+Parameters:
 
-- `web_status` — show all provider configuration status without secrets.
-- `/web-status` — command version of the status check.
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `url` | string | — | URL to extract |
+| `mode` | string | `auto` | `auto`, `static`, `dynamic`, or `full` |
+| `prompt` | string | — | Prompt for JSON extraction (dynamic mode) |
+| `schema` | any | — | JSON schema for structured extraction (dynamic mode) |
+| `content_chars` | number | 20000 | Max content chars |
+| `wait_for` | number | — | Milliseconds to wait for Firecrawl dynamic rendering. Crawl4AI `/md` full mode may ignore this. |
+| `mobile` | boolean | false | Emulate mobile viewport (dynamic mode) |
 
-## Library structure
+**Mode behavior:**
 
-The extension is split into `lib/` modules:
+| Mode | Backend | Best for | API key needed |
+|---|---|---|---|
+| `static` | JSDOM+Readability | Simple static pages, blog posts, docs | No |
+| `dynamic` | Firecrawl Scrape | JS-rendered pages, dynamic content | Maybe |
+| `full` | Crawl4AI | JS-heavy SPA, complex rendering | Maybe |
+| `auto` (default) | static → dynamic → full | Unknown page type | Maybe |
+
+In `auto` mode, fallbacks are noted in the output (e.g., `[Extraction fell back to Firecrawl Scrape (dynamic mode)]`). If `static` extraction fails, the tool gracefully escalates to heavier backends.
+
+> ⚠️ **Note on Firecrawl Scrape**: Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls back to `full` mode (Crawl4AI) in `auto` mode.
+
+### `web_map` — Site URL discovery
+
+Discovers URLs from a site using Firecrawl Map. Best on base domains; may return fewer results on sub-paths.
+
+```
+web_map url="https://riven.tv"
+web_map url="https://docs.example.com" sitemap=only
+```
+
+Parameters: `url`, `limit` (default 100), `include_subdomains`, `search`, `sitemap`, `use_index`, `ignore_cache`.
+
+### `web_crawl` — Site crawl
+
+Crawls pages from a site. Two modes:
+
+- **`light`** (default): Firecrawl Crawl — conservative, docs-focused, single URL.
+- **`full`**: Crawl4AI Crawl — headless browser, rendered data, media, links, up to 100 URLs.
+
+```
+web_crawl url="https://docs.example.com" limit=10          # Firecrawl light mode
+web_crawl urls=["https://a.com","https://b.com"] mode=full  # Crawl4AI full mode
+web_crawl url="https://example.com" mode=light poll=true    # Poll for completion
+```
+
+### `web_screenshot` — Page screenshot
+
+Captures a full-page PNG screenshot using Crawl4AI. Returns base64-encoded PNG.
+
+```
+web_screenshot url="https://example.com"
+web_screenshot url="https://example.com" wait_for=5 wait_for_images=true
+```
+
+### `web_pdf` — Page PDF
+
+Generates a PDF document using Crawl4AI. Returns base64-encoded PDF.
+
+```
+web_pdf url="https://example.com/article"
+```
+
+### `web_status` — Provider status
+
+Shows all provider configuration status and Crawl4AI server health.
+
+```
+web_status
+```
+
+Typical output:
+
+```json
+{
+  "brave": { "apiKeyFound": true, "apiKeySource": "process.env" },
+  "searxng": { "baseUrl": "http://172.30.55.22:8888", ... },
+  "firecrawl": { "baseUrl": "http://172.30.55.22:3002/v2", ... },
+  "crawl4ai": {
+    "baseUrl": "http://172.30.55.22:11235",
+    ...
+    "health": { "status": "healthy", "version": "0.5.0", ... }
+  }
+}
+```
+
+## Library structure
 
 | Module | Contents |
 |--------|----------|
 | `lib/config.ts` | Environment loading, config helpers for all providers |
-| `lib/format.ts` | Text sanitization, truncation, search/scrape result formatting |
+| `lib/format.ts` | Text sanitization, truncation, crawl/scrape result formatting |
 | `lib/content.ts` | Readable content extraction (JSDOM + Readability + Turndown) |
 | `lib/retry.ts` | Retry with exponential backoff for transient HTTP failures |
-| `lib/brave.ts` | Brave Search API fetch client |
-| `lib/searxng.ts` | SearXNG metasearch fetch client |
-| `lib/firecrawl.ts` | Firecrawl API fetch client with v2→v1 fallback |
-| `lib/crawl4ai.ts` | Crawl4AI Docker API fetch client (scrape, crawl, stream, screenshot, PDF, health) |
-
-## Retry behavior
-
-All HTTP-backed tools (`brave_search`, `searxng_search`, `firecrawl_*`, `crawl4ai_*`) use a `withRetry` wrapper with exponential backoff (up to 2 retries, ~1s initial delay). Non-transient errors (auth, validation, not found) are never retried.
-
-## Migration from 0.1.x
-
-- `brave_content` has been renamed to `web_content`. Update any agent instructions or skills that reference `brave_content` to use `web_content` instead. The tool API and behavior are unchanged.
+| `lib/brave.ts` | Brave Search API fetch client (internal) |
+| `lib/searxng.ts` | SearXNG metasearch fetch client (internal) |
+| `lib/firecrawl.ts` | Firecrawl API fetch client with v2→v1 fallback (internal) |
+| `lib/crawl4ai.ts` | Crawl4AI Docker API fetch client (internal) |
+| `lib/search.ts` | Unified search orchestrator — probes backends, fallback chain |
+| `lib/extract.ts` | Unified extraction orchestrator — mode-based backend selection |
+
+## Migration from 0.3.x
+
+v0.4 replaces the 14 individual backend-specific tools with 7 unified tools:
+
+| v0.3 tool | v0.4 replacement |
+|---|---|
+| `brave_search` | `web_search` with `backend: "brave"` |
+| `searxng_search` | `web_search` with `backend: "searxng"` |
+| `firecrawl_search` | `web_search` with `backend: "firecrawl"` |
+| `web_content` | `web_extract` with `mode: "static"` |
+| `firecrawl_scrape` | `web_extract` with `mode: "dynamic"` |
+| `crawl4ai_scrape` | `web_extract` with `mode: "full"` |
+| `firecrawl_map` | `web_map` (same behavior) |
+| `firecrawl_crawl` | `web_crawl` with `mode: "light"` |
+| `crawl4ai_crawl` | `web_crawl` with `mode: "full"` |
+| `crawl4ai_stream` | (removed — use `web_crawl` with `mode: "full"`) |
+| `crawl4ai_screenshot` | `web_screenshot` (same behavior) |
+| `crawl4ai_pdf` | `web_pdf` (same behavior) |
+| `crawl4ai_status` | Merged into `web_status` |
+| `web_status` | `web_status` (enhanced with Crawl4AI health) |
+
+All v0.3 tool names were removed in v0.4. Update any agent instructions or skills that reference the old names.
 
 ## Development
 
@@ -116,20 +227,3 @@ npm test
 # Run only unit tests
 npm run test:unit
 ```
-
-## Practical Guidance
-
-- Use `searxng_search` first for general web search, docs lookup, current facts, and source discovery; it is fast, self-hosted, and avoids hosted API costs/rate limits.
-- Use `brave_search` as a fast hosted fallback when SearXNG results are weak/unavailable, or when an independent search index is useful. Use `include_content` sparingly.
-- Use `web_content` for fast known-URL article/docs extraction when simple readable markdown is enough.
-- Use `firecrawl_search` when SearXNG/Brave are unavailable, when Firecrawl search is preferred, or when search results should include scraped markdown.
-- Use `firecrawl_scrape` for known URLs when extraction quality matters, pages are dynamic, links are needed, or structured JSON extraction is requested.
-- Use `firecrawl_map` before crawling to discover candidate URLs and keep crawls targeted.
-- Use `firecrawl_crawl` only when multiple pages are required; keep limits conservative and use include/exclude paths.
-- Use `crawl4ai_scrape` for JavaScript-rendered pages and when you need high-quality LLM-ready markdown.
-- Use `crawl4ai_crawl` when you need full crawl data (HTML, links, media) from multiple URLs.
-- Use `crawl4ai_screenshot` or `crawl4ai_pdf` for visual/printable snapshots of rendered pages.
-- Prefer `web_content` or `firecrawl_scrape` over Crawl4AI for simple static pages — Crawl4AI launches a full headless browser and is more resource-intensive.
-- When answering from web content, cite source URLs from result links or metadata.
-- Use `web_status` when provider configuration is uncertain or a web tool fails due to credentials/config.
-- Use `crawl4ai_status` when Crawl4AI-specific connection issues occur.

package/extensions/.mocharc.yml

@@ -0,0 +1,5 @@
+require:
+  - tsx
+spec:
+  - test/**/*.test.ts
+timeout: 10000