@pi-unipi/web-api 0.1.15 → 0.1.16

package/README.md

@@ -1,48 +1,73 @@
 # @pi-unipi/web-api
 
-Web search, read, and summarize tools with provider-based backend selection for Pi coding agent.
+Web search, page reading, and content summarization for the agent. The read path uses a local smart-fetch engine by default — free, no API key, browser-grade TLS fingerprinting that bypasses Cloudflare.
 
-## Overview
+Paid providers (SerpAPI, Tavily, Firecrawl, Perplexity) are available as fallbacks. DuckDuckGo and Jina work out of the box for search.
 
-`@pi-unipi/web-api` provides agent tools for web access:
+## Commands
 
-- **web_search** — Search the web using various providers
-- **multi_web_content_read** — Extract content from URLs using the local smart-fetch engine (default) or provider fallbacks
-- **web_llm_summarize** — Summarize web content using LLM
+| Command | Description |
+|---------|-------------|
+| `/unipi:web-settings` | Configure providers, API keys, and smart-fetch defaults |
+| `/unipi:web-cache-clear` | Clear all cached web content |
 
-The read path uses a **smart-fetch engine** by default — free, local, no API key required.
+## Special Triggers
 
-## Features
+Workflow skills detect web-api and inject web tools for research-type commands:
 
-- **Smart-Fetch Engine** — Local content extraction with browser-grade TLS fingerprinting
-- **Provider-based architecture** — Multiple search/read providers with unified interface
-- **Smart selection** — Auto-select cheapest available provider
-- **Batch reading** — Fetch multiple URLs concurrently with progress
-- **API key management** — Interactive TUI for key configuration
-- **Caching** — Web content cached with configurable TTL
-- **Subagent integration** — Tools automatically available to spawned subagents
+| Skill | What Changes |
+|-------|--------------|
+| `research` | Full web search, read, summarize |
+| `gather-context` | External documentation lookup |
+| `consultant` | Industry best practices research |
+| `subagents` (explore) | Web research in parallel |
 
-## Installation
+The footer and info-screen don't display web-api data — it's a tool package, not a state package.
 
-```bash
-npm install @pi-unipi/web-api
+## Agent Tools
+
+| Tool | Description |
+|------|-------------|
+| `web_search` | Search the web via provider |
+| `multi_web_content_read` | Extract content from URLs (smart-fetch or provider) |
+| `web_llm_summarize` | Summarize web content via LLM |
+
+### web_search
+
+```
+# Auto-select cheapest provider
+web_search(query: "TypeScript generics")
+
+# Use specific provider
+web_search(query: "latest AI research", source: 4)  # Tavily
 ```
 
-Add to your pi configuration:
+### multi_web_content_read
 
-```json
-{
-  "pi": {
-    "extensions": [
-      "node_modules/@pi-unipi/web-api/src/index.ts"
-    ]
-  }
-}
+```
+# Single URL (smart-fetch engine by default)
+multi_web_content_read(url: "https://example.com/article")
+
+# Batch URLs
+multi_web_content_read(url: ["https://example.com/a", "https://example.com/b"])
+
+# Provider fallback (Jina Reader)
+multi_web_content_read(url: "https://example.com/article", source: 1)
+
+# Custom options
+multi_web_content_read(url: "https://example.com/article", format: "json", maxChars: 10000)
+```
+
+### web_llm_summarize
+
+```
+web_llm_summarize(url: "https://example.com/long-article")
+web_llm_summarize(url: "https://example.com/research", prompt: "Extract key findings")
 ```
 
 ## Smart-Fetch Engine
 
-The smart-fetch engine is a local content extraction pipeline:
+Local content extraction pipeline — no API key required:
 
 | Component | Purpose |
 |-----------|---------|
@@ -50,17 +75,11 @@ The smart-fetch engine is a local content extraction pipeline:
 | **defuddle** | Intelligent content extraction from HTML |
 | **linkedom** | Server-side DOM parsing |
 
-**Features:**
-- No API key required
-- Browser-level anti-bot bypass
-- Clean markdown output with metadata (title, author, site, word count)
-- Batch concurrent fetching with progress
-- Client-side meta redirect following
-- Multiple output formats (markdown, HTML, text, JSON)
+Outputs clean markdown with metadata (title, author, site, word count). Supports batch concurrent fetching with progress.
 
 ## Providers
 
-### Search Providers
+### Search
 
 | Provider | Rank | Cost | API Key |
 |----------|------|------|---------|
@@ -70,35 +89,27 @@ The smart-fetch engine is a local content extraction pipeline:
 | Tavily | 4 | Paid | Required |
 | Perplexity | 5 | Paid | Required |
 
-### Read Providers
+### Read
 
 | Provider | Rank | Cost | API Key |
 |----------|------|------|---------|
-| **Smart-Fetch Engine** | 0 | Free | No |
+| Smart-Fetch Engine | 0 | Free | No |
 | Jina AI Reader | 1 | Freemium | Optional |
 | Firecrawl | 2 | Paid | Required |
 | Perplexity | 3 | Paid | Required |
 
-**Note:** Rank 0 is the smart-fetch engine (default). Provider fallbacks are available via `source` parameter.
-
-### Summarize Providers
+### Summarize
 
 | Provider | Rank | Cost | API Key |
 |----------|------|------|---------|
 | Perplexity | 1 | Paid | Required |
 | LLM Summarize | 2 | LLM tokens | No |
 
-## Configuration
+## Configurables
 
 ### API Keys
 
-Configure API keys via the interactive settings command:
-
-```
-/unipi:web-settings
-```
-
-Or set environment variables:
+Configure via `/unipi:web-settings` (interactive TUI) or environment variables:
 
 ```bash
 export SERPAPI_KEY="your-key"
@@ -108,135 +119,30 @@ export PERPLEXITY_API_KEY="your-key"
 export JINA_API_KEY="your-key"
 ```
 
-### Smart-Fetch Defaults
+Providers auto-enable when you add a valid API key.
 
-Configure default browser profile, OS, max chars, timeout, etc. via:
+### Smart-Fetch Defaults
 
-```
-/unipi:web-settings → "Smart Fetch Defaults"
-```
+Configure browser profile, OS, max chars, timeout via `/unipi:web-settings → "Smart Fetch Defaults"`.
 
 ### Settings Files
 
 - **Auth:** `~/.unipi/config/web-api/auth.json` (API keys, gitignored)
 - **Config:** `~/.unipi/config/web-api/config.json` (provider settings, smart-fetch defaults)
 
-## Usage
-
-### Web Search
-
-```
-# Auto-select cheapest provider
-web_search(query: "TypeScript generics")
-
-# Use specific provider
-web_search(query: "latest AI research", source: 4)  # Tavily
-```
-
-### Multi Web Content Read
-
-```
-# Single URL (uses smart-fetch engine by default)
-multi_web_content_read(url: "https://example.com/article")
-
-# Batch URLs
-multi_web_content_read(url: ["https://example.com/a", "https://example.com/b"])
-
-# Use provider fallback (Jina Reader)
-multi_web_content_read(url: "https://example.com/article", source: 1)
-
-# Custom options
-multi_web_content_read(url: "https://example.com/article", format: "json", maxChars: 10000)
-
-# Advanced: custom browser profile
-multi_web_content_read(url: "https://example.com/article", browser: "chrome_145", os: "windows")
-```
-
-### Web Summarize
-
-```
-# Auto-summarize
-web_llm_summarize(url: "https://example.com/long-article")
-
-# Custom prompt
-web_llm_summarize(url: "https://example.com/research", prompt: "Extract key findings")
-```
-
-## Commands
-
-### /unipi:web-settings
-
-Interactive settings dialog for managing providers, API keys, and smart-fetch defaults.
-
-- **Auto-enable on key input** — provider is automatically enabled when you add a valid API key
-- **Smart-fetch configuration** — set default browser, OS, timeout, etc.
-- **Cursor memory** — last configured provider moves to the top of the list
-
-### /unipi:web-cache-clear
-
-Clear all cached web content.
-
-## Cache
+### Cache
 
 - Default TTL: 1 hour
 - Cache location: `~/.unipi/config/web-api/cache/`
-- Smart-fetch cache keys include URL + browser + format + maxChars
 - Automatic for all read operations
 
-## Dependencies
-
-| Package | Version | Purpose |
-|---------|---------|---------|
-| wreq-js | ^2.3.0 | TLS fingerprinting |
-| defuddle | ^0.18.1 | Content extraction |
-| linkedom | ^0.18.12 | Server-side DOM |
-| lodash | ^4.17.21 | Filename sanitization |
-| mime-types | ^2.1.35 | MIME type mapping |
-
 ## Troubleshooting
 
-### No provider available
-
-If you see "No search provider available":
-
-1. Run `/unipi:web-settings`
-2. Add API keys for paid providers (they auto-enable on key input)
-3. Or manually enable a free provider
-
-### Smart-fetch fails
+**No provider available:** Run `/unipi:web-settings` and add API keys or enable a free provider.
 
-If smart-fetch fails to extract content:
+**Smart-fetch fails:** Try a different browser profile (`browser: "chrome_133"`) or a provider fallback (`source: 1`).
 
-1. Try a different browser profile: `browser: "chrome_133"`
-2. Try a provider fallback: `source: 1` (Jina Reader)
-3. Check if the site requires JavaScript execution (not supported)
-
-### API key invalid
-
-If API key validation fails:
-
-1. Check the key is correct
-2. Verify the key has sufficient permissions
-3. Check provider status at their website
-
-### Rate limiting
-
-If you hit rate limits:
-
-1. Add an API key for higher limits
-2. Use the smart-fetch engine (default, no limits)
-3. Use a different provider
-4. Wait and retry
-
-## Development
-
-```bash
-# Type check
-npx tsc --noEmit
-
-# Build
-npm run build
-```
+**Rate limiting:** Add an API key for higher limits, use smart-fetch (no limits), or try a different provider.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pi-unipi/web-api",
-  "version": "0.1.15",
+  "version": "0.1.16",
   "description": "Web search, read, and summarize tools with provider-based backend selection for Pi coding agent",
   "type": "module",
   "license": "MIT",