@bacnh85/pi-web 0.1.1 → 0.2.0

package/README.md

@@ -51,7 +51,10 @@ Secrets are never printed; status reports only show presence/source.
 Brave:
 
 - `brave_search` — search web results, optionally fetch readable result content.
-- `brave_content` — fetch a URL and extract readable markdown.
+
+Web Content:
+
+- `web_content` — fetch a URL and extract readable markdown (formerly `brave_content`; renamed in v0.2.0 because it does not use the Brave API).
 
 SearXNG:
 
@@ -69,11 +72,43 @@ Utility:
 - `web_status` — show configured provider status without secrets.
 - `/web-status` — command version of the status check.
 
+## Library structure
+
+The extension is split into `lib/` modules:
+
+| Module | Contents |
+|--------|----------|
+| `lib/config.ts` | Environment loading, config helpers for all providers |
+| `lib/format.ts` | Text sanitization, truncation, search/scrape result formatting |
+| `lib/content.ts` | Readable content extraction (JSDOM + Readability + Turndown) |
+| `lib/retry.ts` | Retry with exponential backoff for transient HTTP failures |
+| `lib/brave.ts` | Brave Search API fetch client |
+| `lib/searxng.ts` | SearXNG metasearch fetch client |
+| `lib/firecrawl.ts` | Firecrawl API fetch client with v2→v1 fallback |
+
+## Retry behavior
+
+All HTTP-backed tools (`brave_search`, `searxng_search`, `firecrawl_*`) use a `withRetry` wrapper with exponential backoff (up to 2 retries, ~1s initial delay). Non-transient errors (auth, validation, not found) are never retried.
+
+## Migration from 0.1.x
+
+- `brave_content` has been renamed to `web_content`. Update any agent instructions or skills that reference `brave_content` to use `web_content` instead. The tool API and behavior are unchanged.
+
+## Development
+
+```bash
+# Run all tests
+npm test
+
+# Run only unit tests
+npm run test:unit
+```
+
 ## Practical Guidance
 
 - Use `searxng_search` first for general web search, docs lookup, current facts, and source discovery; it is fast, self-hosted, and avoids hosted API costs/rate limits.
 - Use `brave_search` as a fast hosted fallback when SearXNG results are weak/unavailable, or when an independent search index is useful. Use `include_content` sparingly.
-- Use `brave_content` for fast known-URL article/docs extraction when simple readable markdown is enough.
+- Use `web_content` for fast known-URL article/docs extraction when simple readable markdown is enough.
 - Use `firecrawl_search` when SearXNG/Brave are unavailable, when Firecrawl search is preferred, or when search results should include scraped markdown.
 - Use `firecrawl_scrape` for known URLs when extraction quality matters, pages are dynamic, links are needed, or structured JSON extraction is requested.
 - Use `firecrawl_map` before crawling to discover candidate URLs and keep crawls targeted.