pi-web-toolkit 0.2.1 → 0.3.0

package/CHANGELOG.md

@@ -0,0 +1,145 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] - 2026-06-23
+
+### Added
+
+- **Firecrawl Keyless fallback** — `web_search`, `web_fetch`, and `web_browse` now automatically retry through [Firecrawl Keyless](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) (1,000 free credits/month, **no API key, no signup**) when their local backend errors out, or when `web_search` returns zero results. The fallback is keyless-only, never the primary path, and degrades gracefully to the original local-tool error if the `firecrawl-cli` is absent, the IP is flagged, the quota is exhausted, or the fallback is disabled.
+- Three explicit escape-hatch tools for capabilities the local backends lack: `firecrawl_search` (sources, `github`/`research`/`pdf` categories, domain filters), `firecrawl_scrape` (anti-bot bypass, JS rendering, PDF parsing), and `firecrawl_interact` (natural-language page interaction).
+- `extensions/utils/firecrawl.ts` — a deep Firecrawl CLI wrapper (scrape/search/interact argument builders, output parsers, graceful-skip failure classifier, keyless-eligibility check, and fallback-decision predicates).
+- Optional external CLI dependency: `npm install -g firecrawl-cli`.
+- Environment toggle `PI_WEB_FIRECRAWL_FALLBACK` (default on) to disable all Firecrawl usage.
+- `test/firecrawl/test.ts` — pure-function regression tests for the firecrawl wrapper boundary (wired into `npm test` as `test:firecrawl`).
+- ADR 0001 and `CONTEXT.md` glossary entries (`Firecrawl keyless`, `cloud fallback`, `free credits`, `graceful skip`) documenting the local-first → optional keyless cloud fallback architectural decision.
+
+### Changed
+
+- **Default network/privacy behavior.** When a local web tool fails, it now makes a cloud request to Firecrawl (sending the URL/query and page content) before giving up. The fallback is **keyless-only** — it never reads, stores, or sends an API key, and spawns the CLI under an isolated temporary `HOME` with the key env stripped. To enforce a strict local-only / no-cloud-egress policy, set `PI_WEB_FIRECRAWL_FALLBACK=0`.
+- `web_search` falls back on a SearXNG error **or** zero results; `web_fetch` falls back on a scrapling failure (incl. its HTTP-GET fallback); `web_browse` falls back only on runtime failures (missing/broken `agent-browser`), never on caller validation errors. `web_batch_fetch` has no fallback (Firecrawl batch scrape is not keyless).
+- Firecrawl results report `creditsUsed` where the source provides it (search, interact); scrape responses do not surface it.
+- README tagline and hero now describe the toolkit as local-first with an optional keyless cloud fallback; features table, install prompt, configuration, project structure, tool reference, and usage guide updated accordingly.
+- `cli-runner` gained an optional `env` passthrough so the firecrawl CLI can be spawned keyless-only.
+
+## [0.2.2] - 2026-06-11
+
+### Added
+
+- Self-contained README prompt that Pi users can copy to install and verify the package, SearXNG, Scrapling, and agent-browser.
+- `CHANGELOG.md` to the files included in the published npm package.
+
+### Changed
+
+- Corrected README, tool reference, usage guide, agent guidance, project context, and test documentation to match current repository behavior.
+- Clarified when to use each tool, Scrapling fallback behavior, external dependency requirements, and SearXNG JSON API setup.
+- Test scripts now use the locally installed `tsx` development dependency.
+- User-visible tool descriptions now distinguish pages that need interaction from those that do not.
+
+### Fixed
+
+- Corrected historical changelog dates and inaccurate claims about robots.txt enforcement, tool limits, runtime configuration, and test coverage.
+- Corrected GitHub issue-reading commands in agent guidance.
+
+## [0.2.1] - 2026-06-10
+
+### Added
+
+- README tools preview grid with screenshots for `web_search`, `web_fetch`, `web_batch_fetch`, and `web_browse`.
+- Agent-browser parser regression tests covering array, wrapped, single-item, and invalid JSON output shapes.
+
+### Fixed
+
+- `web_browse` now accepts multiple agent-browser batch JSON output shapes instead of assuming a top-level array.
+
+### Changed
+
+- `npm test` now also runs the agent-browser parser regression suite.
+
+## [0.2.0] - 2026-06-09
+
+### Added
+
+- `extensions/utils/cli-runner.ts` — centralized CLI process spawning with timeout and AbortSignal support.
+- `extensions/utils/content-preview.ts` — intelligent content extraction from scraped pages.
+- `extensions/utils/output-sink.ts` — truncation and temp-file fallback, replacing `truncateHead` + manual `writeFile`/`mkdtemp` in every tool.
+- `extensions/utils/render-helpers.ts` — URL abbreviations, text normalization, and error formatting for TUI.
+- `extensions/utils/tool-factory.ts` — common tool registration patterns.
+- `CLAUDE.md` — symlink to `AGENTS.md` for IDE/agent integration.
+- `CONTEXT.md` — project domain summary for pi runtime context.
+- `test/` directory — automated test suite under `test/content-preview/` with fixtures, baselines, snapshots, and summary report.
+
+### Changed
+
+- All 4 tools (`web_search`, `web_fetch`, `web_browse`, `web_batch_fetch`) refactored to use new shared utils, eliminating ~200 lines of duplicated truncate/output logic per tool.
+- `scrapling.ts` and `agent-browser.ts` now use `cli-runner`, eliminating duplicate `spawn` logic.
+- `web_search` — `language` default changed from `"auto"` to `""` (omits param when unset to use SearXNG default).
+- `web_search` — `promptGuidelines` now recommends `web_batch_fetch` for parallel reading of 2–5 results.
+- `web_batch_fetch` — added live progress tracking with per-URL status (fetching / done / error).
+- `web_browse` — added step formatting and tracking (`formatBrowseStep` + `steps` in details).
+- Unified TUI redesign across all 4 tools:
+  - Consistent `isError` rendering with `✗` status, error text, and context details.
+  - Enhanced `isPartial` rendering with domain/URL context and live progress indicators.
+  - `fullOutputPath` rendered in accent color.
+  - `renderCall` tags: `[stealthy]`, `[selector=...]`, `[headed]`, `concurrency`.
+- `web_fetch` — content preview (500-char extract) shown in collapsed and expanded views.
+- `web_browse` — expanded view shows complete step list + preview.
+- `web_batch_fetch` — collapsed shows top 3 successes with previews; expanded shows full success list + failure list.
+
+### Meta
+
+- Stop tracking `package-lock.json` (library project; reproducible by downstream consumers).
+- Add `typecheck`, `test`, and `test:approve` scripts to `package.json`.
+
+## [0.1.2] - 2026-06-08
+
+### Added
+
+- `utils/agent-browser.ts` — extracted from `web_browse.ts` to encapsulate all agent-browser CLI interaction (command building, process spawning, JSON parsing, session cleanup).
+- `tsconfig.json` — TypeScript project configuration for CI type-checking.
+- GitHub Actions CI workflow (`ci.yml`) — runs `tsc --noEmit` on every push and PR.
+
+### Changed
+
+- `web_search` — `SEARXNG_URL` is now read at execute time instead of module load time, so in-process environment changes take effect without reloading the extension.
+- `utils/scrapling.ts` — introduced `runScraplingWithFallback()` with configurable `noGetFallback` option, eliminating duplicate fallback logic in `web_fetch` and `web_batch_fetch`.
+- `web_browse.ts` — reduced from ~400 lines to ~194 lines by moving CLI logic to `utils/agent-browser.ts`.
+- README — added `## Configuration` section, `## Contributing` section, CI badge, and updated project structure with design principles.
+
+### Fixed
+
+- Preserved GET fallback for `web_batch_fetch` when `stealthy: true` fails, maintaining backward compatibility with the previous batch implementation.
+
+## [0.1.1] - 2026-06-04
+
+### Added
+
+- `web_batch_fetch` — parallel multi-page fetching via scrapling.
+- Built-in output truncation with temp-file fallback for all tools.
+- TUI renderers for tool calls and results.
+
+### Changed
+
+- Unified extension entry point at `extensions/index.ts`.
+
+## [0.1.0] - 2026-06-03
+
+### Added
+
+- `web_search` — SearXNG web search.
+- `web_fetch` — single-page extraction via scrapling.
+- `web_browse` — interactive browser automation via agent-browser.
+- LLM-optimized `promptGuidelines` and `promptSnippet` for every tool.
+
+[Unreleased]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.2...HEAD
+[0.2.2]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.1...v0.2.2
+[0.2.1]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.0...v0.2.1
+[0.2.0]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.2...v0.2.0
+[0.1.2]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.1...v0.1.2
+[0.1.1]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/Wade11s/pi-web-toolkit/releases/tag/v0.1.0

package/README.md

@@ -1,22 +1,28 @@
 # pi-web-toolkit
 
 [![npm version](https://badge.fury.io/js/pi-web-toolkit.svg)](https://www.npmjs.com/package/pi-web-toolkit)
+[![Pi package](https://img.shields.io/badge/Pi-package-111111.svg)](https://pi.dev/packages/pi-web-toolkit)
 [![CI](https://github.com/Wade11s/pi-web-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/Wade11s/pi-web-toolkit/actions)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 ![Node.js](https://img.shields.io/badge/node-%3E%3D22-339933)
 
-**100% open-source. Zero API keys. Zero fees.**
+**Local-first & 100% open-source. No required API keys or paid services.**
 
-Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch static pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All self-hosted, all local, all free — with built-in truncation safety and LLM-optimized prompt guidelines.
+Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All primary backends run locally or are self-hosted, with an **optional Firecrawl Keyless cloud fallback** (no API key, no signup) so the local tools keep working when a backend is missing or fails. Built-in truncation safety and LLM-optimized prompt guidelines throughout.
 
 ## Features
 
 | Tool | Backend | Purpose | Current Limit |
 |------|---------|---------|---------------|
-| **`web_search`** | [SearXNG](https://github.com/searxng/searxng) | Search the web with scored, ranked results from multiple engines — always the first step in web research | 20 results (max 60, auto-pages up to 3 pages) |
-| **`web_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch a single static page as clean markdown | — |
-| **`web_batch_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch 2–15 pages in parallel for research synthesis | 3 concurrent (max 5) |
+| **`web_search`** | [SearXNG](https://github.com/searxng/searxng) | Discover scored, ranked results from multiple engines | 20 results (max 60, auto-pages up to 3 pages) |
+| **`web_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch a single page as clean markdown | — |
+| **`web_batch_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch 1–15 pages in parallel for research synthesis (2–5 recommended) | 3 concurrent (max 5) |
 | **`web_browse`** | [agent-browser](https://github.com/vercel-labs/agent-browser) | Interact with a page (click, scroll, fill) then extract content | 25 actions |
+| **`firecrawl_search`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud search with sources/categories/domain filters | — |
+| **`firecrawl_scrape`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud single-page fetch (anti-bot / JS / PDF) | — |
+| **`firecrawl_interact`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud natural-language page interaction | — |
+
+> **Firecrawl fallback.** `web_search`, `web_fetch`, and `web_browse` automatically retry through Firecrawl Keyless (1,000 free credits/month, no API key) when their local backend errors out or search returns nothing. The three `firecrawl_*` tools are explicit escape hatches. Disable it with `PI_WEB_FIRECRAWL_FALLBACK=0`. Install the optional CLI: `npm install -g firecrawl-cli`.
 
 ## Tools Preview
 
@@ -40,14 +46,101 @@ A quick look at how pi renders toolkit calls while an agent searches, fetches, b
   </tr>
 </table>
 
+## Install with Pi Agent
+
+Copy and send the prompt below to Pi. It will install this package and its external dependencies for you.
+
+```text
+Install pi-web-toolkit and its external dependencies. Complete and verify every
+step yourself; do not rely on web browsing or external documentation. Inspect
+the machine first and reuse working installations. Ask before using sudo,
+changing shell profiles, overwriting configuration, or modifying existing
+services or containers.
+
+1. Ensure Node.js 22+, npm, Docker, OpenSSL, curl, uv, and Pi are installed, and
+   that Docker is running. Install only missing or incompatible prerequisites.
+2. Configure SearXNG:
+   - Test SEARXNG_URL when set, then http://localhost:8080.
+   - Verify /search?q=test&format=json returns JSON with a results array.
+   - If neither endpoint works, first ensure no existing container or config
+     would be overwritten, then create a local-only instance by running:
+
+mkdir -p "$HOME/.config/searxng"
+cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
+use_default_settings: true
+
+search:
+  formats:
+    - html
+    - json
+YAML
+
+docker run -d \
+  --name searxng \
+  --restart unless-stopped \
+  -p 127.0.0.1:8080:8080 \
+  -e FORCE_OWNERSHIP=false \
+  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
+  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
+  docker.io/searxng/searxng:latest
+
+   - Verify the selected endpoint by running:
+
+SEARXNG_ENDPOINT="${SEARXNG_URL:-http://localhost:8080}"
+curl -fsS --get "${SEARXNG_ENDPOINT%/}/search" \
+  --data-urlencode "q=test" \
+  --data "format=json" |
+  grep -q '"results"' && echo "SearXNG JSON API ready"
+
+   - Pi uses http://localhost:8080 by default. Set SEARXNG_URL before starting
+     Pi only when using another endpoint.
+3. Install and verify Scrapling:
+   uv tool install "scrapling[all]"
+   scrapling install
+   scrapling --help
+4. Install and verify agent-browser:
+   npm install -g agent-browser
+   agent-browser install
+   agent-browser doctor
+   On Linux, use agent-browser install --with-deps if required.
+5. Optionally install firecrawl-cli for the keyless cloud fallback (no API key
+   needed; the fallback degrades gracefully if it is absent):
+   npm install -g firecrawl-cli
+6. After all dependencies pass verification, install the package:
+   pi install npm:pi-web-toolkit
+
+Report what was installed or reused, all verification results, the SearXNG
+endpoint Pi will use, and whether Pi must be restarted. Do not report success
+until every check passes.
+```
+
 ## Quick Start
 
 ### 1. Install external dependencies
 
+The commands below assume a POSIX shell with Docker, OpenSSL, curl, uv, and Node.js 22+ with npm.
+
 ```bash
-# SearXNG (for search)
-docker run -d --name searxng -p 8080:8080 -v searxng:/etc/searxng searxng/searxng
-export SEARXNG_URL="http://localhost:8080"
+# SearXNG (for search; local-only instance with the required JSON API)
+mkdir -p "$HOME/.config/searxng"
+cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
+use_default_settings: true
+
+search:
+  formats:
+    - html
+    - json
+YAML
+
+docker run -d \
+  --name searxng \
+  --restart unless-stopped \
+  -p 127.0.0.1:8080:8080 \
+  -e FORCE_OWNERSHIP=false \
+  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
+  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
+  docker.io/searxng/searxng:latest
+export SEARXNG_URL="http://127.0.0.1:8080"
 
 # scrapling (for fetch & batch fetch)
 uv tool install "scrapling[all]"
@@ -55,12 +148,19 @@ scrapling install
 
 # agent-browser (for browse)
 npm i -g agent-browser && agent-browser install
+# On Linux hosts missing browser system libraries: agent-browser install --with-deps
+
+# firecrawl-cli (OPTIONAL — enables the keyless cloud fallback; no API key needed)
+npm i -g firecrawl-cli
 ```
 
 **Verify dependencies:**
 ```bash
 # SearXNG
-curl -s "$SEARXNG_URL" | head
+curl -fsS --get "$SEARXNG_URL/search" \
+  --data-urlencode "q=searxng" \
+  --data "format=json" |
+  grep -q '"results"' && echo "SearXNG JSON API ready"
 
 # scrapling
 scrapling --help
@@ -81,44 +181,69 @@ pi install git:github.com/Wade11s/pi-web-toolkit
 
 ## Configuration
 
-All tools are configured via **environment variables** at runtime — no rebuild or restart required.
+`web_search` reads its SearXNG endpoint from an environment variable. Set it before starting pi; no build step is required.
 
 | Variable | Default | Used By | Description |
 |----------|---------|---------|-------------|
 | `SEARXNG_URL` | `http://localhost:8080` | `web_search` | Your SearXNG instance endpoint |
+| `PI_WEB_FIRECRAWL_FALLBACK` | `1` (on) | all tools | Set to `0`/`false`/`no`/`off` to disable the optional Firecrawl keyless cloud fallback for a strict local-only policy. |
 
 Set before starting pi:
 
 ```bash
 export SEARXNG_URL="https://searxng.example.com"
+# Optional: disable the Firecrawl cloud fallback entirely
+export PI_WEB_FIRECRAWL_FALLBACK=0
+```
+
+### Optional: Firecrawl keyless fallback
+
+When a local backend (`web_search`/`web_fetch`/`web_browse`) fails or returns nothing, the tools automatically retry through [Firecrawl Keyless](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) — 1,000 free credits/month, **no API key, no signup**. The `firecrawl_*` tools are explicit escape hatches for capabilities the local backends lack (search categories, cloud rendering, natural-language interaction).
+
+Install the optional CLI (the fallback degrades gracefully if it is absent):
+
+```bash
+npm install -g firecrawl-cli
 ```
 
+The fallback is **keyless-only**: it never reads or stores an API key, and spawns the CLI under an isolated temporary `HOME` with the key env stripped. **Privacy:** when the fallback runs, the URL and page content are sent to Firecrawl's cloud.
+
 ## Project Structure
 
 ```
 pi-web-toolkit/
 ├── extensions/
-│   ├── index.ts              # Unified entry point — registers all 4 tools
+│   ├── index.ts              # Unified entry point — registers all 7 tools (4 local + 3 Firecrawl keyless)
 │   ├── utils/
-│   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal
+│   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal/env
 │   │   ├── content-preview.ts # Intelligent content extraction from scraped pages
 │   │   ├── output-sink.ts    # Truncation + temp-file fallback
 │   │   ├── render-helpers.ts # URL abbreviations, text normalization, error formatting for TUI
 │   │   ├── scrapling.ts      # Reusable scrapling CLI wrapper (shared by fetch + batch)
 │   │   ├── tool-factory.ts   # Common tool registration patterns
-│   │   └── agent-browser.ts  # agent-browser CLI wrapper (shared by web_browse)
-│   ├── web_search.ts         # SearXNG search tool
-│   ├── web_fetch.ts          # Single-page scrapling fetcher
+│   │   ├── agent-browser.ts  # agent-browser CLI wrapper (shared by web_browse)
+│   │   └── firecrawl.ts      # Firecrawl keyless CLI wrapper + fallback decisions (shared by firecrawl_* tools + fallbacks)
+│   ├── web_search.ts         # SearXNG search tool (+ Firecrawl fallback)
+│   ├── web_fetch.ts          # Single-page scrapling fetcher (+ Firecrawl fallback)
 │   ├── web_batch_fetch.ts    # Parallel scrapling fetcher
-│   └── web_browse.ts         # Interactive browser automation (agent-browser)
+│   ├── web_browse.ts         # Interactive browser automation (agent-browser + Firecrawl fallback)
+│   ├── firecrawl_search.ts   # Firecrawl keyless search (escape hatch)
+│   ├── firecrawl_scrape.ts   # Firecrawl keyless single-page fetch (escape hatch)
+│   └── firecrawl_interact.ts # Firecrawl keyless natural-language interaction (escape hatch)
 ├── test/
-│   └── content-preview/      # Automated test suite with fixtures & snapshots
+│   ├── agent-browser/        # agent-browser output parser regression tests
+│   ├── content-preview/      # Content preview fixtures, baselines & snapshots
+│   └── README.md             # Test suite structure and conventions
 ├── docs/
 │   ├── tools.md              # Full parameter specs
-│   └── guide.md              # Decision tree & tool comparison
+│   ├── guide.md              # Decision tree & tool comparison
+│   └── agents/               # Issue tracker, triage and domain guidance
+├── AGENTS.md
+├── CONTEXT.md
 ├── CHANGELOG.md
 ├── package.json
 ├── README.md
+├── tsconfig.json
 └── LICENSE
 ```
 

package/docs/adr/0001-firecrawl-keyless-cloud-fallback.md

@@ -0,0 +1,5 @@
+# Firecrawl Keyless as an optional cloud fallback
+
+pi-web-toolkit was local-first and self-hosted by design: SearXNG, scrapling, and agent-browser all run on the user's machine, and the README guaranteed "100% open-source. No required API keys or paid services." We decided to add **Firecrawl Keyless** as a strictly optional, fallback-only cloud layer: when a local backend errors out (or `web_search` returns nothing), the same tool transparently retries through the official `firecrawl-cli` in keyless mode, and three explicit `firecrawl_*` tools are exposed for capabilities the local backends lack.
+
+This is hard to reverse once users and the agent come to rely on the fallback, surprising to a reader who assumes a local-only toolkit, and the result of a real trade-off (zero-config reliability vs. cloud egress, a privacy surface, and a third-party dependency). The fallback defaults **on**, is **keyless-only** (no API key, no signup, no stored credentials — the CLI is spawned under an isolated temp `HOME` with the key env stripped), and is **opt-out-able** via `PI_WEB_FIRECRAWL_FALLBACK=0`. We drive `firecrawl-cli` (an official Firecrawl client) rather than hand-rolling REST because Firecrawl only grants the free keyless tier to official clients, and we restrict it to the keyless endpoints (`/search`, `/scrape`, `/interact`); API-key mode, self-hosted URLs, OAuth, and non-keyless endpoints (`/map`, `/crawl`, `/batch/scrape`, etc.) are deliberately out of scope. The decision and the graceful-skip behavior (never leave the user worse off than the local tool already did) are encoded in the Firecrawl CLI wrapper module.

package/docs/agents/issue-tracker.md

@@ -5,7 +5,7 @@ Issues and PRDs for this repo live as GitHub issues. Use the `gh` CLI for all op
 ## Conventions
 
 - **Create an issue**: `gh issue create --title "..." --body "..."`. Use a heredoc for multi-line bodies.
-- **Read an issue**: `gh issue view <number> --comments`, filtering comments by `jq` and also fetching labels.
+- **Read an issue**: `gh issue view <number> --json number,title,body,labels,comments --jq '{number, title, body, labels: [.labels[].name], comments: [.comments[].body]}'`.
 - **List issues**: `gh issue list --state open --json number,title,body,labels,comments --jq '[.[] | {number, title, body, labels: [.labels[].name], comments: [.comments[].body]}]'` with appropriate `--label` and `--state` filters.
 - **Comment on an issue**: `gh issue comment <number> --body "..."`
 - **Apply / remove labels**: `gh issue edit <number> --add-label "..."` / `--remove-label "..."`
@@ -19,4 +19,4 @@ Create a GitHub issue.
 
 ## When a skill says "fetch the relevant ticket"
 
-Run `gh issue view <number> --comments`.
+Run `gh issue view <number> --json number,title,body,labels,comments --jq '{number, title, body, labels: [.labels[].name], comments: [.comments[].body]}'`.

package/docs/guide.md

@@ -8,19 +8,19 @@ User asks about something external / current
 ├─→ web_search("...")
 │   │
 │   ├─→ 1 relevant result?
-│   │   └─→ web_fetch(url)                     ← static page
+│   │   └─→ web_fetch(url)                     ← no interaction needed
 │   │   OR
 │   │   └─→ web_browse(url, actions)           ← needs interaction
 │   │
 │   └─→ 2–5 relevant results?
-│       ├─→ All static pages?
+│       ├─→ All need no interaction?
 │       │   └─→ web_batch_fetch(urls[])        ← parallel fetch
 │       └─→ Some need interaction?
-│           └─→ web_fetch (static ones)
+│           └─→ web_fetch (no-interaction ones)
 │               web_browse (interactive ones)  ← sequential
 │
 └─→ User provides a URL directly
-    ├─→ Static / loads on first request?
+    ├─→ No interaction needed / loads on first request?
     │   └─→ web_fetch(url)
     └─→ Needs clicking / scrolling / waiting?
         └─→ web_browse(url, actions)
@@ -32,10 +32,35 @@ User asks about something external / current
 
 | | `web_fetch` | `web_browse` | `web_batch_fetch` |
 |--|-------------|--------------|-------------------|
-| **Pages** | 1 | 1 | 2–15 |
-| **Browser** | Yes (scrapling) | Yes (agent-browser) | Yes (scrapling) |
+| **Pages** | 1 | 1 | 1–15 (2–5 recommended) |
+| **Browser** | Yes (Scrapling) | Yes (agent-browser) | Yes (Scrapling) |
 | **Interaction** | ❌ No | ✅ Click, fill, scroll, wait | ❌ No |
 | **Selector** | ✅ Per-URL | ✅ Final state | ✅ Applied to all |
-| **Stealthy** | ✅ Yes | ❌ No (planned) | ✅ Yes |
+| **Stealthy** | ✅ Yes | ❌ No | ✅ Yes |
 | **Speed** | Fast | Slower (browser ops) | Medium (parallel) |
 | **Best for** | Articles, docs, blogs | SPAs, forms, pagination | Research synthesis |
+
+`web_fetch` falls back to HTTP GET after a normal browser fetch fails, but not in stealthy mode. `web_batch_fetch` falls back to GET after failed browser fetches in all modes.
+
+---
+
+## Firecrawl Keyless fallback
+
+When a local backend cannot do the job, the tools automatically retry through **Firecrawl Keyless** (1,000 free credits/month, no API key, no signup) before giving up. It is **fallback-only** — never the primary path — and is **opt-out-able** with `PI_WEB_FIRECRAWL_FALLBACK=0`. Requires the optional `firecrawl-cli` (`npm install -g firecrawl-cli`); if it is absent the tools simply surface the original local error.
+
+| Tool | Falls back to Firecrawl when… |
+|------|-------------------------------|
+| `web_search` | SearXNG errors out **or** returns zero results |
+| `web_fetch` | scrapling (incl. its HTTP-GET fallback) fails — anti-bot, heavy JS, PDFs |
+| `web_browse` | agent-browser is missing or its batch fails (not on caller validation errors) |
+| `web_batch_fetch` | (no fallback — Firecrawl batch scrape is not keyless) |
+
+The three `firecrawl_*` tools are the explicit escape hatches for capabilities the local backends lack (`github`/`research`/`pdf` search categories, cloud rendering, natural-language interaction).
+
+**Graceful skip.** If the fallback itself cannot help — the CLI is missing, the IP is flagged as suspicious, the keyless quota is exhausted, or the fallback is disabled — the tool falls through to the original local-tool error so the user is never left worse off.
+
+**Credit budgeting.** Search ≈ 2 credits / 10 results, scrape ≈ 1 credit / page, interact ≈ 2 credits/min (code-only) or ≈ 7 credits/min (AI prompt). Results report `creditsUsed` where the source provides it. The fallback stays conservative (small limits) against the 1,000 credits/month allowance.
+
+**Privacy.** Firecrawl is a cloud service: when the fallback runs, the URL/query and page content leave the machine. Set `PI_WEB_FIRECRAWL_FALLBACK=0` to enforce a strict local-only, no-cloud-egress policy. The fallback is **keyless-only** — it never reads, stores, or sends an API key, and spawns the CLI under an isolated temporary `HOME`.
+
+---

package/docs/tools.md

@@ -7,22 +7,24 @@ Search the web via SearXNG. Returns ranked results with title, URL, and snippet.
 ```typescript
 {
   query: string,           // Search query
-  language?: string,       // Language code (en, de, fr...). Default: "auto"
+  language?: string,       // Language code (en, en-US, de...). Omit to use the SearXNG default.
   results?: number,        // Max results (1–60). Default: 20. Automatically pages through SearXNG (up to 3 pages) if needed.
 }
 ```
 
-**When to use:** The user asks about current events, facts, or anything requiring up-to-date information. This is always the **first step** of web research.
+**When to use:** The user asks about current events, facts, or anything requiring up-to-date information and has not already provided the source URLs.
 
-**Empty results behavior:** When no results are found, `web_search` returns a list of **suggestions** — alternative queries that SearXNG believes may yield better results. The agent can use these suggestions to automatically refine and retry the search.
+**Empty results behavior:** When no results are found, `web_search` includes any query **suggestions** provided by SearXNG. The agent can use them to refine and retry the search.
 
 **Pagination:** `web_search` automatically fetches up to 3 pages from SearXNG and deduplicates by URL. You do not need to call it multiple times for deeper results.
 
+**Full output:** For non-empty searches, the formatted result output is always written to a temporary file. Returned text is also truncated to pi's default line/byte limits when necessary.
+
 ---
 
 ## `web_fetch`
 
-Fetch a single page and convert it to clean markdown. Uses scrapling's browser automation for JS-heavy sites.
+Fetch a single page and convert it to clean markdown. Uses Scrapling's browser-based `fetch` command first, then falls back to an HTTP GET when allowed. Stealthy mode uses `stealthy-fetch` and does not fall back to GET.
 
 ```typescript
 {
@@ -39,9 +41,9 @@ Fetch a single page and convert it to clean markdown. Uses scrapling's browser a
 
 **Example flow:**
 ```
-User: "What's the latest Rust release?"
-→ web_search("latest Rust programming language release")
-→ web_fetch("https://blog.rust-lang.org/2026/06/02/maintainers-fund/")
+User: "How do I install Rust?"
+→ web_search("official Rust installation guide")
+→ web_fetch("https://www.rust-lang.org/tools/install")
 → Agent answers with full context
 ```
 
@@ -51,7 +53,7 @@ User: "What's the latest Rust release?"
 
 Open a real browser, perform a chain of actions (click, fill, scroll, wait), then extract content.
 
-Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI for native browser automation via Chrome CDP.
+Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI with batched JSON commands.
 
 ```typescript
 {
@@ -65,12 +67,15 @@ Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI for n
     | { type: "wait_selector", selector: string, state?: "attached" | "visible" | "hidden" }
     | { type: "scroll", direction: "down" | "up" | "bottom" | "top", amount?: number }
   >,
+  // Maximum: 25 actions
   selector?: string,       // Extract content from final page state
   headless?: boolean,      // Default: true
   timeout?: number,        // Overall browser batch timeout (ms). Default: 30000
 }
 ```
 
+When `selector` is omitted, the tool returns agent-browser's interactive accessibility snapshot rather than full page text.
+
 **When to use:**
 - The page requires **clicking** before showing target content (e.g. "Load more", pagination, tab switching)
 - The page requires **filling a form** (e.g. search box, login)
@@ -123,7 +128,7 @@ Fetch multiple pages in parallel and return aggregated content.
 
 ```typescript
 {
-  urls: string[],          // 1–10 URLs
+  urls: string[],          // 1–15 URLs; 2–5 recommended
   selector?: string,       // CSS selector applied to ALL pages
   stealthy?: boolean,      // Default: false
   max_concurrency?: number // Parallel fetches (1–5). Default: 3
@@ -136,7 +141,9 @@ Fetch multiple pages in parallel and return aggregated content.
 - Comparing implementations across different docs/pages
 - Research synthesis requiring multiple sources
 
-**NOT for:** Single pages (use `web_fetch` — simpler and supports per-URL stealthy mode).
+**NOT recommended for:** Single pages. The schema accepts one URL, but `web_fetch` is simpler and provides single-page behavior.
+
+Each page starts with the selected Scrapling browser fetcher. Failed attempts fall back to HTTP GET, including when batch stealthy mode is enabled.
 
 **Example flow:**
 ```
@@ -153,3 +160,65 @@ User: "Compare Python asyncio, Trio, and curio"
   })
 → Agent synthesizes comparison from all 3 sources
 ```
+
+---
+
+## Firecrawl keyless tools (optional cloud escape hatches)
+
+These three tools talk to [Firecrawl](https://www.firecrawl.dev) in **keyless** mode: 1,000 free credits/month, **no API key and no signup**. They require the optional `firecrawl-cli` (`npm install -g firecrawl-cli`). **Privacy:** the URL/query/page content is sent to Firecrawl's cloud.
+
+They double as the implementation of the automatic fallback: `web_search`/`web_fetch`/`web_browse` retry through Firecrawl keyless when their local backend fails (or search returns nothing). Disable all Firecrawl usage with `PI_WEB_FIRECRAWL_FALLBACK=0`.
+
+### `firecrawl_search`
+
+Cloud web search via Firecrawl keyless, with capabilities the local SearXNG tool lacks.
+
+```typescript
+{
+  query: string,
+  limit?: number,                       // 1–100. Default 10
+  sources?: Array<"web"|"images"|"news">,
+  categories?: Array<"github"|"research"|"pdf">,
+  country?: string,                     // ISO code, e.g. "US", "DE"
+  tbs?: string,                         // qdr:h|d|w|m|y
+  location?: string,
+  includeDomains?: string[],            // hostnames; folded into the query as site: operators
+  excludeDomains?: string[],
+}
+```
+
+**When to use:** `web_search` failed or returned nothing; or you need `github`/`research`/`pdf` categories, images/news sources, or domain scoping that SearXNG does not provide.
+
+### `firecrawl_scrape`
+
+Cloud single-page fetch via Firecrawl keyless (anti-bot bypass, JS rendering, PDF parsing).
+
+```typescript
+{
+  url: string,
+  waitFor?: number,           // ms to wait for JS rendering
+  includeTags?: string[],     // Firecrawl tag filter (not a CSS selector)
+  excludeTags?: string[],
+  onlyMainContent?: boolean,  // Default: true
+}
+```
+
+**When to use:** `web_fetch` failed on an anti-bot-protected, JavaScript-heavy, or PDF page.
+
+### `firecrawl_interact`
+
+Open a URL in a live Firecrawl browser session and drive it with a natural-language prompt (or code).
+
+```typescript
+{
+  url: string,
+  prompt?: string,                            // natural-language task (required unless code is set)
+  code?: string,                              // code to run in the browser sandbox
+  language?: "node"|"python"|"bash",
+  timeout?: number,                           // seconds (1–300)
+}
+```
+
+**When to use:** `web_browse` cannot run (agent-browser missing / OS deps missing), or you want natural-language page interaction without hand-written CSS selectors. Write each prompt as a single, focused task.
+
+---