pi-web-toolkit 0.3.1 → 0.3.3

package/CHANGELOG.md

@@ -7,6 +7,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.3] - 2026-06-28
+
+### Added
+
+- `install.sh` bootstrap installer for one-command pi-web-toolkit setup, including dependency verification, SearXNG endpoint selection, toolkit config writing, optional Firecrawl setup, local development install mode, and `--doctor` diagnostics.
+- Toolkit config support at `${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json`, with environment variables taking precedence for SearXNG endpoints, Firecrawl fallback enablement, Firecrawl runner selection, and external CLI command paths.
+- Public SearXNG endpoint discovery from `searx.space` with JSON API verification, plus an explicit isolated local Docker SearXNG option.
+- Explicit Firecrawl runner selection through `firecrawlRunner` / `PI_WEB_FIRECRAWL_RUNNER`, supporting `installed`, `npx`, and `bunx`.
+- Regression tests for toolkit config precedence and installer behavior, including public endpoint and local Docker flows.
+
+### Changed
+
+- README and guide now present the bootstrap installer as the primary installation path while keeping manual setup as an advanced option.
+- External CLI wrappers can use configured absolute command paths, reducing reliance on shell profile/PATH changes after installer runs.
+- `web_search` fallback behavior is covered by regression tests so missing optional Firecrawl runners do not appear as the primary search backend failure.
+
+## [0.3.2] - 2026-06-25
+
+### Fixed
+
+- Kept the agent's web-tool selection local-first: ordinary URL reads now prefer `web_fetch`, discovery prefers `web_search`, and interaction prefers `web_browse`; `firecrawl_*` tools are documented and prompted as fallback-only unless explicitly requested.
+- Fixed `firecrawl_scrape` and `firecrawl_interact` partial-result rendering type-check errors caused by reading `details` before declaration.
+
+### Changed
+
+- Reduced web-tool prompt metadata overhead by consolidating shared routing rules and shortening per-tool `promptSnippet`/`promptGuidelines` text.
+- Added a tool-routing prompt regression test and included it in `npm test`.
+
 ## [0.3.1] - 2026-06-23
 
 ### Changed
@@ -145,7 +173,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `web_browse` — interactive browser automation via agent-browser.
 - LLM-optimized `promptGuidelines` and `promptSnippet` for every tool.
 
-[Unreleased]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.2...HEAD
+[Unreleased]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.3.3...HEAD
+[0.3.3]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.3.2...v0.3.3
+[0.3.2]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.3.1...v0.3.2
+[0.3.1]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.3.0...v0.3.1
+[0.3.0]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.2...v0.3.0
 [0.2.2]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.1...v0.2.2
 [0.2.1]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.2...v0.2.0

package/README.md

@@ -22,7 +22,7 @@ Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch
 | **`firecrawl_scrape`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud single-page fetch (anti-bot / JS / PDF) | — |
 | **`firecrawl_interact`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud natural-language page interaction | — |
 
-> **Firecrawl fallback.** `web_search`, `web_fetch`, and `web_browse` automatically retry through Firecrawl Keyless (1,000 free credits/month, no API key) when their local backend errors out or search returns nothing. The three `firecrawl_*` tools are explicit escape hatches. Disable it with `PI_WEB_FIRECRAWL_FALLBACK=0`. Install the optional CLI: `npm install -g firecrawl-cli`.
+> **Firecrawl fallback.** `web_search`, `web_fetch`, and `web_browse` are the local-first primary tools and automatically retry through Firecrawl Keyless (1,000 free credits/month, no API key) only when their local backend errors out or search returns nothing. The three `firecrawl_*` tools are fallback-only escape hatches; agents are instructed not to call them first unless you explicitly ask for Firecrawl/cloud behavior or a local-first tool already failed. Disable fallback use with `PI_WEB_FIRECRAWL_FALLBACK=0` or toolkit config `"firecrawlFallback": false`. Install the optional CLI: `npm install -g firecrawl-cli`.
 
 ## Tools Preview
 
@@ -46,159 +46,154 @@ A quick look at how pi renders toolkit calls while an agent searches, fetches, b
   </tr>
 </table>
 
-## Install with Pi Agent
+## Quick Start
 
-Copy and send the prompt below to Pi. It will install this package and its external dependencies for you.
+### Install
 
-```text
-Install pi-web-toolkit and its external dependencies. Complete and verify every
-step yourself; do not rely on web browsing or external documentation. Inspect
-the machine first and reuse working installations. Ask before using sudo,
-changing shell profiles, overwriting configuration, or modifying existing
-services or containers.
-
-1. Ensure Node.js 22+, npm, Docker, OpenSSL, curl, uv, and Pi are installed, and
-   that Docker is running. Install only missing or incompatible prerequisites.
-2. Configure SearXNG:
-   - Test SEARXNG_URL when set, then http://localhost:8080.
-   - Verify /search?q=test&format=json returns JSON with a results array.
-   - If neither endpoint works, first ensure no existing container or config
-     would be overwritten, then create a local-only instance by running:
-
-mkdir -p "$HOME/.config/searxng"
-cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
-use_default_settings: true
-
-search:
-  formats:
-    - html
-    - json
-YAML
-
-docker run -d \
-  --name searxng \
-  --restart unless-stopped \
-  -p 127.0.0.1:8080:8080 \
-  -e FORCE_OWNERSHIP=false \
-  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
-  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
-  docker.io/searxng/searxng:latest
-
-   - Verify the selected endpoint by running:
-
-SEARXNG_ENDPOINT="${SEARXNG_URL:-http://localhost:8080}"
-curl -fsS --get "${SEARXNG_ENDPOINT%/}/search" \
-  --data-urlencode "q=test" \
-  --data "format=json" |
-  grep -q '"results"' && echo "SearXNG JSON API ready"
-
-   - Pi uses http://localhost:8080 by default. Set SEARXNG_URL before starting
-     Pi only when using another endpoint.
-3. Install and verify Scrapling:
-   uv tool install "scrapling[all]"
-   scrapling install
-   scrapling --help
-4. Install and verify agent-browser:
-   npm install -g agent-browser
-   agent-browser install
-   agent-browser doctor
-   On Linux, use agent-browser install --with-deps if required.
-5. Optionally install firecrawl-cli for the keyless cloud fallback (no API key
-   needed; the fallback degrades gracefully if it is absent):
-   npm install -g firecrawl-cli
-6. After all dependencies pass verification, install the package:
-   pi install npm:pi-web-toolkit
-
-Report what was installed or reused, all verification results, the SearXNG
-endpoint Pi will use, and whether Pi must be restarted. Do not report success
-until every check passes.
+Run the bootstrap installer:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/Wade11s/pi-web-toolkit/main/install.sh | bash
 ```
 
-## Quick Start
+This is the normal install path. It installs the pi package, configures external runtime dependencies, verifies everything, and writes persistent runtime options to `${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json`.
+
+When it finishes, **restart pi** so the package is loaded. If pi-web-toolkit was already loaded and only toolkit config changed, `/reload` may also work.
+
+### What the installer does
+
+The installer:
 
-### 1. Install external dependencies
+- Checks Node.js 22+, npm, Pi, curl, OpenSSL, and uv.
+- Installs or reuses Scrapling and agent-browser.
+- Configures a JSON-capable SearXNG endpoint for `web_search`.
+- Optionally installs `firecrawl-cli` for the Firecrawl Keyless fallback.
+- Writes toolkit config with the selected endpoint and discovered CLI paths.
+- Installs the pi package with `pi install npm:pi-web-toolkit`.
+- Runs final verification before reporting success.
 
-The commands below assume a POSIX shell with Docker, OpenSSL, curl, uv, and Node.js 22+ with npm.
+The installer is conservative. It does **not** silently install Docker, Node.js, Pi, Homebrew, OS packages, use sudo, change shell profiles, or overwrite user-managed SearXNG resources.
+
+### Common installer options
+
+When piping from curl, pass flags after `bash -s --`:
 
 ```bash
-# SearXNG (for search; local-only instance with the required JSON API)
-mkdir -p "$HOME/.config/searxng"
-cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
-use_default_settings: true
-
-search:
-  formats:
-    - html
-    - json
-YAML
-
-docker run -d \
-  --name searxng \
-  --restart unless-stopped \
-  -p 127.0.0.1:8080:8080 \
-  -e FORCE_OWNERSHIP=false \
-  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
-  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
-  docker.io/searxng/searxng:latest
-export SEARXNG_URL="http://127.0.0.1:8080"
+curl -fsSL https://raw.githubusercontent.com/Wade11s/pi-web-toolkit/main/install.sh | bash -s -- --yes --searxng-url https://searxng.example.com --no-firecrawl
+```
+
+If you have cloned the repo, run the same flags directly:
+
+| Goal | Command |
+|------|---------|
+| Use an existing/self-hosted SearXNG endpoint | `./install.sh --searxng-url https://searxng.example.com` |
+| Non-interactive install with a known endpoint | `./install.sh --yes --searxng-url https://searxng.example.com --no-firecrawl` |
+| Explicitly auto-select a verified public SearXNG endpoint | `./install.sh --yes --auto-searxng public --no-firecrawl` |
+| Start/reuse isolated local Docker SearXNG | `./install.sh --yes --auto-searxng local-docker --searxng-port 8080 --no-firecrawl` |
+| Install optional Firecrawl Keyless fallback with global CLI | `./install.sh --with-firecrawl --firecrawl-runner installed` |
+| Enable Firecrawl fallback through opt-in `npx` runner | `./install.sh --with-firecrawl --firecrawl-runner npx` |
+| Enable Firecrawl fallback through opt-in `bunx` runner | `./install.sh --with-firecrawl --firecrawl-runner bunx` |
+| Verify readiness without changing anything | `./install.sh --doctor` |
+| Install from the current checkout | `./install.sh --local` |
+
+### SearXNG endpoint choices
+
+`web_search` needs a SearXNG endpoint that supports JSON search responses:
+
+```bash
+curl -fsS --get "https://searxng.example.com/search" \
+  --data-urlencode "q=searxng" \
+  --data "format=json" | grep -q '"results"'
+```
+
+The installer can use:
+
+- An existing/self-hosted endpoint passed with `--searxng-url`.
+- A working local endpoint such as `http://localhost:8080`.
+- A public endpoint discovered from `searx.space`, ranked by health signals, then verified with `format=json`.
+- An isolated local Docker endpoint using container `pi-web-toolkit-searxng` and config under the toolkit config directory.
+
+Public endpoints are not silently selected by default because search queries leave your machine. Use `--auto-searxng public` only when that trade-off is acceptable.
+
+### Manual install (advanced)
+
+If you prefer to install dependencies yourself:
+
+```bash
+# SearXNG endpoint: provide an existing JSON-capable endpoint, or run your own
+export SEARXNG_URL="https://searxng.example.com"
 
 # scrapling (for fetch & batch fetch)
 uv tool install "scrapling[all]"
 scrapling install
 
 # agent-browser (for browse)
-npm i -g agent-browser && agent-browser install
-# On Linux hosts missing browser system libraries: agent-browser install --with-deps
+npm i -g agent-browser
+agent-browser install
+agent-browser doctor
 
-# firecrawl-cli (OPTIONAL — enables the keyless cloud fallback; no API key needed)
+# firecrawl-cli (optional cloud fallback; no API key needed)
 npm i -g firecrawl-cli
+
+# pi package
+pi install npm:pi-web-toolkit
 ```
 
-**Verify dependencies:**
+A SearXNG endpoint must support `format=json`:
+
 ```bash
-# SearXNG
 curl -fsS --get "$SEARXNG_URL/search" \
   --data-urlencode "q=searxng" \
-  --data "format=json" |
-  grep -q '"results"' && echo "SearXNG JSON API ready"
+  --data "format=json" | grep -q '"results"'
+```
 
-# scrapling
-scrapling --help
+## Configuration
 
-# agent-browser
-agent-browser doctor
-```
+Runtime configuration is resolved in this order: environment variables first, then the toolkit config file written by the installer, then built-in defaults. No build step is required.
 
-### 2. Install the extension
-#### From npm
-```bash
-pi install npm:pi-web-toolkit
-```
-#### From GitHub
-```bash
-pi install git:github.com/Wade11s/pi-web-toolkit
-```
+Default toolkit config path:
 
-## Configuration
+```text
+${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json
+```
 
-`web_search` reads its SearXNG endpoint from an environment variable. Set it before starting pi; no build step is required.
+Example:
+
+```json
+{
+  "searxngUrl": "https://searxng.example.com",
+  "firecrawlFallback": false,
+  "firecrawlRunner": "installed",
+  "commands": {
+    "scrapling": "/Users/alice/.local/bin/scrapling",
+    "agentBrowser": "/Users/alice/.npm-global/bin/agent-browser",
+    "firecrawl": "/Users/alice/.npm-global/bin/firecrawl"
+  }
+}
+```
 
-| Variable | Default | Used By | Description |
-|----------|---------|---------|-------------|
-| `SEARXNG_URL` | `http://localhost:8080` | `web_search` | Your SearXNG instance endpoint |
-| `PI_WEB_FIRECRAWL_FALLBACK` | `1` (on) | all tools | Set to `0`/`false`/`no`/`off` to disable the optional Firecrawl keyless cloud fallback for a strict local-only policy. |
+| Variable | Toolkit config key | Default | Used By | Description |
+|----------|--------------------|---------|---------|-------------|
+| `SEARXNG_URL` | `searxngUrl` | `http://localhost:8080` | `web_search` | SearXNG endpoint. Must support `/search?q=...&format=json`. |
+| `PI_WEB_FIRECRAWL_FALLBACK` | `firecrawlFallback` | `true` | all Firecrawl fallback paths | Set env to `0`/`false`/`no`/`off`, or config to `false`, to disable cloud fallback. |
+| `PI_WEB_FIRECRAWL_RUNNER` | `firecrawlRunner` | `installed` | all Firecrawl fallback paths | Firecrawl runner: `installed`, `npx`, or `bunx`. `npx`/`bunx` are opt-in because they may run or download packages at fallback time. |
+| `SCRAPLING_BIN` | `commands.scrapling` | `scrapling` | `web_fetch`, `web_batch_fetch` | Scrapling executable path. |
+| `AGENT_BROWSER_BIN` | `commands.agentBrowser` | `agent-browser` | `web_browse` | agent-browser executable path. |
+| `FIRECRAWL_BIN` | `commands.firecrawl` | `firecrawl` | `firecrawl_*`, fallback paths | Firecrawl CLI executable path. |
+| `PI_WEB_TOOLKIT_CONFIG` | — | `${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json` | all tools | Override the toolkit config file location. |
 
-Set before starting pi:
+Set env vars before starting pi when you need a temporary override:
 
 ```bash
 export SEARXNG_URL="https://searxng.example.com"
-# Optional: disable the Firecrawl cloud fallback entirely
+export SCRAPLING_BIN="$HOME/.local/bin/scrapling"
 export PI_WEB_FIRECRAWL_FALLBACK=0
+export PI_WEB_FIRECRAWL_RUNNER=npx
 ```
 
 ### Optional: Firecrawl keyless fallback
 
-When a local backend (`web_search`/`web_fetch`/`web_browse`) fails or returns nothing, the tools automatically retry through [Firecrawl Keyless](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) — 1,000 free credits/month, **no API key, no signup**. The `firecrawl_*` tools are explicit escape hatches for capabilities the local backends lack (search categories, cloud rendering, natural-language interaction).
+When a local backend (`web_search`/`web_fetch`/`web_browse`) fails or returns nothing, the tools automatically retry through [Firecrawl Keyless](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) — 1,000 free credits/month, **no API key, no signup**. The `firecrawl_*` tools are fallback-only explicit escape hatches for capabilities the local backends lack (search categories, cloud rendering, natural-language interaction). Agents should use `web_fetch`/`web_search`/`web_browse` first unless you explicitly request Firecrawl/cloud behavior.
 
 Install the optional CLI (the fallback degrades gracefully if it is absent):
 
@@ -206,8 +201,40 @@ Install the optional CLI (the fallback degrades gracefully if it is absent):
 npm install -g firecrawl-cli
 ```
 
+Alternatively, opt into a runner that executes the official CLI on demand:
+
+```json
+{ "firecrawlRunner": "npx" }
+```
+
+Allowed runners are `installed`, `npx`, and `bunx`. The default is `installed`; `npx` and `bunx` are never selected automatically because they may run or download packages at fallback time.
+
 The fallback is **keyless-only**: it never reads or stores an API key, and spawns the CLI under an isolated temporary `HOME` with the key env stripped. **Privacy:** when the fallback runs, the URL and page content are sent to Firecrawl's cloud.
 
+## Troubleshooting
+
+Run doctor mode when an install fails, when filing an issue, or when you want to verify an existing setup. It is verify-only: it does not install dependencies, write config, start containers, or run `pi install`.
+
+```bash
+./install.sh --doctor
+# or, without cloning:
+curl -fsSL https://raw.githubusercontent.com/Wade11s/pi-web-toolkit/main/install.sh | bash -s -- --doctor
+```
+
+Common failures:
+
+| Symptom | Fix |
+|---------|-----|
+| Node.js is too old | Install Node.js 22+ and retry. |
+| `uv` is missing | Install uv, then rerun the installer. |
+| SearXNG returns HTML/403 instead of JSON | Use another endpoint or enable `search.formats: json` on your SearXNG instance. |
+| Docker local SearXNG fails | Start Docker first, or use `--searxng-url` / `--auto-searxng public`. |
+| `agent-browser doctor` fails on Linux | Rerun with `--agent-browser-with-deps` or install the missing browser system libraries manually. |
+| Firecrawl fallback says runner missing | Install `firecrawl-cli`, choose `--firecrawl-runner npx`, choose `--firecrawl-runner bunx`, or disable fallback with `--no-firecrawl`. |
+| pi does not show the tools after install | Restart pi. |
+
+To remove the pi package, run `pi remove npm:pi-web-toolkit`. To remove the toolkit config, delete `${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json`. If the installer created local SearXNG, remove container `pi-web-toolkit-searxng` and the toolkit SearXNG config directory manually.
+
 ## Project Structure
 
 ```
@@ -216,6 +243,7 @@ pi-web-toolkit/
 │   ├── index.ts              # Unified entry point — registers all 7 tools (4 local + 3 Firecrawl keyless)
 │   ├── utils/
 │   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal/env
+│   │   ├── config.ts         # Toolkit config and external CLI path resolution
 │   │   ├── content-preview.ts # Intelligent content extraction from scraped pages
 │   │   ├── output-sink.ts    # Truncation + temp-file fallback
 │   │   ├── render-helpers.ts # URL abbreviations, text normalization, error formatting for TUI
@@ -232,7 +260,10 @@ pi-web-toolkit/
 │   └── firecrawl_interact.ts # Firecrawl keyless natural-language interaction (escape hatch)
 ├── test/
 │   ├── agent-browser/        # agent-browser output parser regression tests
+│   ├── config/               # Toolkit config precedence tests
 │   ├── content-preview/      # Content preview fixtures, baselines & snapshots
+│   ├── installer/            # Bootstrap installer behavior tests
+│   ├── web-search/           # SearXNG-first fallback behavior tests
 │   └── README.md             # Test suite structure and conventions
 ├── docs/
 │   ├── tools.md              # Full parameter specs
@@ -241,6 +272,7 @@ pi-web-toolkit/
 ├── AGENTS.md
 ├── CONTEXT.md
 ├── CHANGELOG.md
+├── install.sh
 ├── package.json
 ├── README.md
 ├── tsconfig.json
@@ -251,7 +283,7 @@ pi-web-toolkit/
 - **Unified registration** — `index.ts` is the single source of truth for what pi loads.
 - **Shared utilities** — `utils/` modules encapsulate CLI spawning, content extraction, output truncation, TUI formatting, and common registration patterns; tool files import only from `utils/`, never from each other.
 - **Per-tool isolation** — each tool owns its own schema, execute logic, and TUI renderer; no cross-imports except via `utils/`.
-- **Runtime config** — environment variables are read at execute time, not build time.
+- **Runtime config** — environment variables and toolkit config are read at execute time, not build time.
 
 ## Reference
 

package/docs/adr/0001-firecrawl-keyless-cloud-fallback.md

@@ -2,4 +2,4 @@
 
 pi-web-toolkit was local-first and self-hosted by design: SearXNG, scrapling, and agent-browser all run on the user's machine, and the README guaranteed "100% open-source. No required API keys or paid services." We decided to add **Firecrawl Keyless** as a strictly optional, fallback-only cloud layer: when a local backend errors out (or `web_search` returns nothing), the same tool transparently retries through the official `firecrawl-cli` in keyless mode, and three explicit `firecrawl_*` tools are exposed for capabilities the local backends lack.
 
-This is hard to reverse once users and the agent come to rely on the fallback, surprising to a reader who assumes a local-only toolkit, and the result of a real trade-off (zero-config reliability vs. cloud egress, a privacy surface, and a third-party dependency). The fallback defaults **on**, is **keyless-only** (no API key, no signup, no stored credentials — the CLI is spawned under an isolated temp `HOME` with the key env stripped), and is **opt-out-able** via `PI_WEB_FIRECRAWL_FALLBACK=0`. We drive `firecrawl-cli` (an official Firecrawl client) rather than hand-rolling REST because Firecrawl only grants the free keyless tier to official clients, and we restrict it to the keyless endpoints (`/search`, `/scrape`, `/interact`); API-key mode, self-hosted URLs, OAuth, and non-keyless endpoints (`/map`, `/crawl`, `/batch/scrape`, etc.) are deliberately out of scope. The decision and the graceful-skip behavior (never leave the user worse off than the local tool already did) are encoded in the Firecrawl CLI wrapper module.
+This is hard to reverse once users and the agent come to rely on the fallback, surprising to a reader who assumes a local-only toolkit, and the result of a real trade-off (zero-config reliability vs. cloud egress, a privacy surface, and a third-party dependency). The fallback defaults **on**, is **keyless-only** (no API key, no signup, no stored credentials — the CLI is spawned under an isolated temp `HOME` with the key env stripped), and is **opt-out-able** via `PI_WEB_FIRECRAWL_FALLBACK=0`. We drive `firecrawl-cli` (an official Firecrawl client) rather than hand-rolling REST because Firecrawl only grants the free keyless tier to official clients, and we restrict it to the keyless endpoints (`/search`, `/scrape`, `/interact`); API-key mode, self-hosted URLs, OAuth, and non-keyless endpoints (`/map`, `/crawl`, `/batch/scrape`, etc.) are deliberately out of scope. The default Firecrawl runner is an installed `firecrawl` executable; `npx` and `bunx` runners are explicit opt-ins because they may run or download packages at fallback time. The decision and the graceful-skip behavior (never leave the user worse off than the local tool already did) are encoded in the Firecrawl CLI wrapper module.

package/docs/adr/0002-toolkit-config-for-installer-selections.md

@@ -0,0 +1,3 @@
+# Toolkit config for installer selections
+
+The installer writes selected pi-web-toolkit runtime options, especially the SearXNG endpoint, Firecrawl fallback policy, Firecrawl runner, and discovered external CLI paths, to `${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json` instead of modifying shell profiles or relying only on transient environment variables. Environment variables keep highest precedence, but the toolkit config gives installer choices persistent effect after restarting pi without changing the user's shell startup files.

package/docs/adr/0003-conservative-installer-prerequisites.md

@@ -0,0 +1,3 @@
+# Conservative installer prerequisites
+
+The bootstrap installer automates user-level dependency setup and pi-web-toolkit configuration, but it does not silently install or alter system-level prerequisites such as Node.js, Pi, Docker, Homebrew, or OS package-manager state. It asks before optional setup steps such as a local Docker SearXNG container and otherwise reports precise remediation commands, preserving user control over system-wide changes while still making the common path easier.

package/docs/adr/0004-searxng-endpoint-discovery.md

@@ -0,0 +1,3 @@
+# Interactive SearXNG endpoint discovery
+
+The bootstrap installer discovers public SearXNG candidates from `searx.space`, ranks them by health signals, and verifies the JSON search API before presenting them to the user. It does not silently choose a public endpoint unless explicitly requested, because using a remote SearXNG service changes the user's privacy and reliability profile; local or custom endpoints remain first-class choices.

package/docs/guide.md

@@ -44,9 +44,25 @@ User asks about something external / current
 
 ---
 
+## Installation and endpoint selection
+
+Use the root `install.sh` bootstrap installer for ordinary installs. It verifies prerequisites, installs or reuses Scrapling and agent-browser, configures a JSON-capable SearXNG endpoint, writes toolkit config, installs the pi package, and performs final verification.
+
+SearXNG endpoint selection is endpoint-first:
+
+1. `--searxng-url` or `SEARXNG_URL`
+2. existing toolkit config
+3. working localhost endpoints
+4. explicit public discovery with `--auto-searxng public`
+5. explicit isolated local Docker setup with `--auto-searxng local-docker`
+
+Public SearXNG endpoints are discovered from `searx.space`, ranked by health signals, then verified with `/search?q=...&format=json` before use. The installer does not silently choose a public endpoint unless explicitly requested. Use `./install.sh --doctor` for verify-only diagnostics; normal installation already performs final verification.
+
+---
+
 ## Firecrawl Keyless fallback
 
-When a local backend cannot do the job, the tools automatically retry through **Firecrawl Keyless** (1,000 free credits/month, no API key, no signup) before giving up. It is **fallback-only** — never the primary path — and is **opt-out-able** with `PI_WEB_FIRECRAWL_FALLBACK=0`. Requires the optional `firecrawl-cli` (`npm install -g firecrawl-cli`); if it is absent the tools simply surface the original local error.
+When a local backend cannot do the job, the tools automatically retry through **Firecrawl Keyless** (1,000 free credits/month, no API key, no signup) before giving up. It is **fallback-only** — never the primary path — and is **opt-out-able** with `PI_WEB_FIRECRAWL_FALLBACK=0` or toolkit config `"firecrawlFallback": false`. Requires the optional `firecrawl-cli` (`npm install -g firecrawl-cli`) or an explicit `firecrawlRunner` of `npx`/`bunx`; if no runner is available the tools simply surface the original local error. Agents should call `web_search`/`web_fetch`/`web_browse` first and call `firecrawl_*` directly only after the corresponding local-first tool failed, or when the user explicitly asks for Firecrawl/cloud behavior.
 
 | Tool | Falls back to Firecrawl when… |
 |------|-------------------------------|
@@ -55,12 +71,12 @@ When a local backend cannot do the job, the tools automatically retry through **
 | `web_browse` | agent-browser is missing or its batch fails (not on caller validation errors) |
 | `web_batch_fetch` | (no fallback — Firecrawl batch scrape is not keyless) |
 
-The three `firecrawl_*` tools are the explicit escape hatches for capabilities the local backends lack (`github`/`research`/`pdf` search categories, cloud rendering, natural-language interaction).
+The three `firecrawl_*` tools are fallback-only explicit escape hatches for capabilities the local backends lack (`github`/`research`/`pdf` search categories, cloud rendering, natural-language interaction). They are not the first step for ordinary URL reading; `web_fetch` already performs Firecrawl fallback internally when local fetching fails.
 
-**Graceful skip.** If the fallback itself cannot help — the CLI is missing, the IP is flagged as suspicious, the keyless quota is exhausted, or the fallback is disabled — the tool falls through to the original local-tool error so the user is never left worse off.
+**Graceful skip.** If the fallback itself cannot help — the runner is missing, the IP is flagged as suspicious, the keyless quota is exhausted, or the fallback is disabled — the tool falls through to the original local-tool error so the user is never left worse off. `firecrawlRunner` defaults to `installed`; `npx` and `bunx` are opt-in because they may run or download packages at fallback time.
 
 **Credit budgeting.** Search ≈ 2 credits / 10 results, scrape ≈ 1 credit / page, interact ≈ 2 credits/min (code-only) or ≈ 7 credits/min (AI prompt). Results report `creditsUsed` where the source provides it. The fallback stays conservative (small limits) against the 1,000 credits/month allowance.
 
-**Privacy.** Firecrawl is a cloud service: when the fallback runs, the URL/query and page content leave the machine. Set `PI_WEB_FIRECRAWL_FALLBACK=0` to enforce a strict local-only, no-cloud-egress policy. The fallback is **keyless-only** — it never reads, stores, or sends an API key, and spawns the CLI under an isolated temporary `HOME`.
+**Privacy.** Firecrawl is a cloud service: when the fallback runs, the URL/query and page content leave the machine. Set `PI_WEB_FIRECRAWL_FALLBACK=0` or toolkit config `"firecrawlFallback": false` to enforce a strict local-only, no-cloud-egress policy. The fallback is **keyless-only** — it never reads, stores, or sends an API key, and spawns the CLI under an isolated temporary `HOME`.
 
 ---

package/docs/tools.md

@@ -1,5 +1,20 @@
 # Tool Reference
 
+## Runtime configuration
+
+Tools resolve runtime configuration in this order: environment variables, toolkit config, then built-in defaults. The installer writes toolkit config to `${XDG_CONFIG_HOME:-~/.config}/pi-web-toolkit/config.json`; override that path with `PI_WEB_TOOLKIT_CONFIG`.
+
+| Variable | Toolkit config key | Default | Used By |
+|----------|--------------------|---------|---------|
+| `SEARXNG_URL` | `searxngUrl` | `http://localhost:8080` | `web_search` |
+| `PI_WEB_FIRECRAWL_FALLBACK` | `firecrawlFallback` | `true` | Firecrawl fallback paths |
+| `PI_WEB_FIRECRAWL_RUNNER` | `firecrawlRunner` | `installed` | Firecrawl fallback paths |
+| `SCRAPLING_BIN` | `commands.scrapling` | `scrapling` | `web_fetch`, `web_batch_fetch` |
+| `AGENT_BROWSER_BIN` | `commands.agentBrowser` | `agent-browser` | `web_browse` |
+| `FIRECRAWL_BIN` | `commands.firecrawl` | `firecrawl` | `firecrawl_*` and fallback paths |
+
+If toolkit config exists but is malformed, tools fail with a clear config error instead of silently ignoring the file. `firecrawlRunner` accepts `installed`, `npx`, or `bunx`; `npx` and `bunx` are opt-in because they may run or download packages at fallback time.
+
 ## `web_search`
 
 Search the web via SearXNG. Returns ranked results with title, URL, and snippet. Automatically aggregates up to 3 pages of SearXNG results when more than ~20 are needed.
@@ -12,7 +27,7 @@ Search the web via SearXNG. Returns ranked results with title, URL, and snippet.
 }
 ```
 
-**When to use:** The user asks about current events, facts, or anything requiring up-to-date information and has not already provided the source URLs.
+**When to use:** The user asks about current events, facts, or anything requiring up-to-date information and has not already provided the source URLs. Use `web_search` before `firecrawl_search`; `web_search` already performs Firecrawl fallback internally when SearXNG fails or returns nothing.
 
 **Empty results behavior:** When no results are found, `web_search` includes any query **suggestions** provided by SearXNG. The agent can use them to refine and retry the search.
 
@@ -35,9 +50,10 @@ Fetch a single page and convert it to clean markdown. Uses Scrapling's browser-b
 ```
 
 **When to use:**
-- After `web_search` finds a relevant result
+- As the first attempt for a user-provided URL or after `web_search` finds a relevant result
 - The page is static or loads its content on first request
 - You need to read **one** article, doc, or blog post
+- Before `firecrawl_scrape`; `web_fetch` already performs Firecrawl fallback internally when the local fetcher fails
 
 **Example flow:**
 ```
@@ -77,10 +93,12 @@ Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI with
 When `selector` is omitted, the tool returns agent-browser's interactive accessibility snapshot rather than full page text.
 
 **When to use:**
+- As the first attempt when the page requires interaction
 - The page requires **clicking** before showing target content (e.g. "Load more", pagination, tab switching)
 - The page requires **filling a form** (e.g. search box, login)
 - The page requires **scrolling** to load lazy content (infinite scroll)
 - The page requires **waiting** for JS to render content (SPA)
+- Before `firecrawl_interact`; `web_browse` already performs Firecrawl fallback internally when local browser automation fails
 
 **Example flows:**
 
@@ -163,11 +181,11 @@ User: "Compare Python asyncio, Trio, and curio"
 
 ---
 
-## Firecrawl keyless tools (optional cloud escape hatches)
+## Firecrawl keyless tools (optional fallback-only cloud escape hatches)
 
 These three tools talk to [Firecrawl](https://www.firecrawl.dev) in **keyless** mode: 1,000 free credits/month, **no API key and no signup**. They require the optional `firecrawl-cli` (`npm install -g firecrawl-cli`). **Privacy:** the URL/query/page content is sent to Firecrawl's cloud.
 
-They double as the implementation of the automatic fallback: `web_search`/`web_fetch`/`web_browse` retry through Firecrawl keyless when their local backend fails (or search returns nothing). Disable all Firecrawl usage with `PI_WEB_FIRECRAWL_FALLBACK=0`.
+They double as the implementation of the automatic fallback: `web_search`/`web_fetch`/`web_browse` retry through Firecrawl keyless when their local backend fails (or search returns nothing). Do not use `firecrawl_*` as the first attempt for ordinary search, URL reading, or page interaction; use the corresponding local-first tool first unless the user explicitly asks for Firecrawl/cloud behavior. Disable all Firecrawl usage with `PI_WEB_FIRECRAWL_FALLBACK=0` or toolkit config `"firecrawlFallback": false`.
 
 ### `firecrawl_search`
 
@@ -187,7 +205,7 @@ Cloud web search via Firecrawl keyless, with capabilities the local SearXNG tool
 }
 ```
 
-**When to use:** `web_search` failed or returned nothing; or you need `github`/`research`/`pdf` categories, images/news sources, or domain scoping that SearXNG does not provide.
+**When to use:** `web_search` failed or returned nothing; you need `github`/`research`/`pdf` categories, images/news sources, or domain scoping that SearXNG does not provide; or the user explicitly asked for Firecrawl/cloud search. Do not use it before `web_search` for ordinary discovery.
 
 ### `firecrawl_scrape`
 
@@ -203,7 +221,7 @@ Cloud single-page fetch via Firecrawl keyless (anti-bot bypass, JS rendering, PD
 }
 ```
 
-**When to use:** `web_fetch` failed on an anti-bot-protected, JavaScript-heavy, or PDF page.
+**When to use:** `web_fetch` failed on an anti-bot-protected, JavaScript-heavy, or PDF page, or the user explicitly asked for Firecrawl/cloud scraping. Do not use it before `web_fetch` for ordinary URL reading.
 
 ### `firecrawl_interact`
 
@@ -219,6 +237,6 @@ Open a URL in a live Firecrawl browser session and drive it with a natural-langu
 }
 ```
 
-**When to use:** `web_browse` cannot run (agent-browser missing / OS deps missing), or you want natural-language page interaction without hand-written CSS selectors. Write each prompt as a single, focused task.
+**When to use:** `web_browse` cannot run (agent-browser missing / OS deps missing), you need natural-language page interaction without hand-written CSS selectors, or the user explicitly asked for Firecrawl/cloud interaction. Do not use it before `web_browse` for ordinary page interaction. Write each prompt as a single, focused task.
 
 ---

package/extensions/firecrawl_interact.ts

@@ -39,18 +39,18 @@ const firecrawlInteractTool = defineTool({
   name: "firecrawl_interact",
   label: "Firecrawl Interact",
   description: [
+    "Fallback-only cloud browser interaction via Firecrawl keyless.",
+    "Do not use firecrawl_interact as the first attempt for ordinary page interaction; use web_browse first.",
     "Open a URL in a live Firecrawl browser session and drive it with a natural-language",
-    "prompt (or code), returning the result. Keyless — no API key, no signup.",
-    "Use firecrawl_interact when the local web_browse cannot run, or when you want",
-    "natural-language page interaction without CSS selectors.",
+    "prompt (or code), returning the result. Use only when web_browse cannot run,",
+    "when the user explicitly asks for Firecrawl/cloud interaction, or when you need natural-language page interaction without CSS selectors.",
     "Privacy: the URL, page content, and prompt are sent to Firecrawl's cloud.",
     `Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; if truncated, full output is saved to a temp file.`,
   ].join(" "),
-  promptSnippet: "Drive a page via Firecrawl keyless (natural-language interaction)",
+  promptSnippet: "Fallback-only Firecrawl interaction",
   promptGuidelines: [
-    "Prefer web_browse first; reach for firecrawl_interact when web_browse can't run or you want NL interaction.",
-    "Write each prompt as a single, focused task; the session can be reused across calls.",
-    "Always pass the full URL including https://.",
+    "Use firecrawl_interact only after web_browse fails, for needed NL interaction, or explicit cloud interaction.",
+    "Keep firecrawl_interact prompt/code focused.",
   ],
   parameters: FirecrawlInteractParamsSchema,
 
@@ -95,13 +95,6 @@ const firecrawlInteractTool = defineTool({
 
   renderResult(result, { expanded, isPartial }, theme, context) {
     const isError = context?.isError ?? false;
-
-    if (isPartial) {
-      const domain = details?.url ? getDomain(details.url) : "";
-      const label = domain ? `Interacting with ${domain} via Firecrawl...` : "Interacting via Firecrawl...";
-      return new Text(theme.fg("warning", label), 0, 0);
-    }
-
     const details = result.details as {
       url?: string;
       output?: string;
@@ -111,6 +104,12 @@ const firecrawlInteractTool = defineTool({
       creditsUsed?: number;
     } | undefined;
 
+    if (isPartial) {
+      const domain = details?.url ? getDomain(details.url) : "";
+      const label = domain ? `Interacting with ${domain} via Firecrawl...` : "Interacting via Firecrawl...";
+      return new Text(theme.fg("warning", label), 0, 0);
+    }
+
     if (isError) {
       const errText = getErrorText(result);
       let text = theme.fg("error", "✗ Firecrawl interact failed");