pi-web-toolkit 0.2.0 → 0.2.2

package/CHANGELOG.md

@@ -0,0 +1,125 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.2] - 2026-06-11
+
+### Added
+
+- Self-contained README prompt that Pi users can copy to install and verify the package, SearXNG, Scrapling, and agent-browser.
+- `CHANGELOG.md` to the files included in the published npm package.
+
+### Changed
+
+- Corrected README, tool reference, usage guide, agent guidance, project context, and test documentation to match current repository behavior.
+- Clarified when to use each tool, Scrapling fallback behavior, external dependency requirements, and SearXNG JSON API setup.
+- Test scripts now use the locally installed `tsx` development dependency.
+- User-visible tool descriptions now distinguish pages that need interaction from those that do not.
+
+### Fixed
+
+- Corrected historical changelog dates and inaccurate claims about robots.txt enforcement, tool limits, runtime configuration, and test coverage.
+- Corrected GitHub issue-reading commands in agent guidance.
+
+## [0.2.1] - 2026-06-10
+
+### Added
+
+- README tools preview grid with screenshots for `web_search`, `web_fetch`, `web_batch_fetch`, and `web_browse`.
+- Agent-browser parser regression tests covering array, wrapped, single-item, and invalid JSON output shapes.
+
+### Fixed
+
+- `web_browse` now accepts multiple agent-browser batch JSON output shapes instead of assuming a top-level array.
+
+### Changed
+
+- `npm test` now also runs the agent-browser parser regression suite.
+
+## [0.2.0] - 2026-06-09
+
+### Added
+
+- `extensions/utils/cli-runner.ts` — centralized CLI process spawning with timeout and AbortSignal support.
+- `extensions/utils/content-preview.ts` — intelligent content extraction from scraped pages.
+- `extensions/utils/output-sink.ts` — truncation and temp-file fallback, replacing `truncateHead` + manual `writeFile`/`mkdtemp` in every tool.
+- `extensions/utils/render-helpers.ts` — URL abbreviations, text normalization, and error formatting for TUI.
+- `extensions/utils/tool-factory.ts` — common tool registration patterns.
+- `CLAUDE.md` — symlink to `AGENTS.md` for IDE/agent integration.
+- `CONTEXT.md` — project domain summary for pi runtime context.
+- `test/` directory — automated test suite under `test/content-preview/` with fixtures, baselines, snapshots, and summary report.
+
+### Changed
+
+- All 4 tools (`web_search`, `web_fetch`, `web_browse`, `web_batch_fetch`) refactored to use new shared utils, eliminating ~200 lines of duplicated truncate/output logic per tool.
+- `scrapling.ts` and `agent-browser.ts` now use `cli-runner`, eliminating duplicate `spawn` logic.
+- `web_search` — `language` default changed from `"auto"` to `""` (omits param when unset to use SearXNG default).
+- `web_search` — `promptGuidelines` now recommends `web_batch_fetch` for parallel reading of 2–5 results.
+- `web_batch_fetch` — added live progress tracking with per-URL status (fetching / done / error).
+- `web_browse` — added step formatting and tracking (`formatBrowseStep` + `steps` in details).
+- Unified TUI redesign across all 4 tools:
+  - Consistent `isError` rendering with `✗` status, error text, and context details.
+  - Enhanced `isPartial` rendering with domain/URL context and live progress indicators.
+  - `fullOutputPath` rendered in accent color.
+  - `renderCall` tags: `[stealthy]`, `[selector=...]`, `[headed]`, `concurrency`.
+- `web_fetch` — content preview (500-char extract) shown in collapsed and expanded views.
+- `web_browse` — expanded view shows complete step list + preview.
+- `web_batch_fetch` — collapsed shows top 3 successes with previews; expanded shows full success list + failure list.
+
+### Meta
+
+- Stop tracking `package-lock.json` (library project; reproducible by downstream consumers).
+- Add `typecheck`, `test`, and `test:approve` scripts to `package.json`.
+
+## [0.1.2] - 2026-06-08
+
+### Added
+
+- `utils/agent-browser.ts` — extracted from `web_browse.ts` to encapsulate all agent-browser CLI interaction (command building, process spawning, JSON parsing, session cleanup).
+- `tsconfig.json` — TypeScript project configuration for CI type-checking.
+- GitHub Actions CI workflow (`ci.yml`) — runs `tsc --noEmit` on every push and PR.
+
+### Changed
+
+- `web_search` — `SEARXNG_URL` is now read at execute time instead of module load time, so in-process environment changes take effect without reloading the extension.
+- `utils/scrapling.ts` — introduced `runScraplingWithFallback()` with configurable `noGetFallback` option, eliminating duplicate fallback logic in `web_fetch` and `web_batch_fetch`.
+- `web_browse.ts` — reduced from ~400 lines to ~194 lines by moving CLI logic to `utils/agent-browser.ts`.
+- README — added `## Configuration` section, `## Contributing` section, CI badge, and updated project structure with design principles.
+
+### Fixed
+
+- Preserved GET fallback for `web_batch_fetch` when `stealthy: true` fails, maintaining backward compatibility with the previous batch implementation.
+
+## [0.1.1] - 2026-06-04
+
+### Added
+
+- `web_batch_fetch` — parallel multi-page fetching via scrapling.
+- Built-in output truncation with temp-file fallback for all tools.
+- TUI renderers for tool calls and results.
+
+### Changed
+
+- Unified extension entry point at `extensions/index.ts`.
+
+## [0.1.0] - 2026-06-03
+
+### Added
+
+- `web_search` — SearXNG web search.
+- `web_fetch` — single-page extraction via scrapling.
+- `web_browse` — interactive browser automation via agent-browser.
+- LLM-optimized `promptGuidelines` and `promptSnippet` for every tool.
+
+[Unreleased]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.2...HEAD
+[0.2.2]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.1...v0.2.2
+[0.2.1]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.2.0...v0.2.1
+[0.2.0]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.2...v0.2.0
+[0.1.2]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.1...v0.1.2
+[0.1.1]: https://github.com/Wade11s/pi-web-toolkit/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/Wade11s/pi-web-toolkit/releases/tag/v0.1.0

package/README.md

@@ -1,31 +1,138 @@
 # pi-web-toolkit
 
 [![npm version](https://badge.fury.io/js/pi-web-toolkit.svg)](https://www.npmjs.com/package/pi-web-toolkit)
+[![Pi package](https://img.shields.io/badge/Pi-package-111111.svg)](https://pi.dev/packages/pi-web-toolkit)
 [![CI](https://github.com/Wade11s/pi-web-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/Wade11s/pi-web-toolkit/actions)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 ![Node.js](https://img.shields.io/badge/node-%3E%3D22-339933)
 
-**100% open-source. Zero API keys. Zero fees.**
+**100% open-source. No required API keys or paid services.**
 
-Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch static pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All self-hosted, all local, all free — with built-in truncation safety and LLM-optimized prompt guidelines.
+Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All backends run locally or are self-hosted, with built-in truncation safety and LLM-optimized prompt guidelines.
 
 ## Features
 
 | Tool | Backend | Purpose | Current Limit |
 |------|---------|---------|---------------|
-| **`web_search`** | [SearXNG](https://github.com/searxng/searxng) | Search the web with scored, ranked results from multiple engines — always the first step in web research | 20 results (max 60, auto-pages up to 3 pages) |
-| **`web_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch a single static page as clean markdown | — |
-| **`web_batch_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch 2–15 pages in parallel for research synthesis | 3 concurrent (max 5) |
+| **`web_search`** | [SearXNG](https://github.com/searxng/searxng) | Discover scored, ranked results from multiple engines | 20 results (max 60, auto-pages up to 3 pages) |
+| **`web_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch a single page as clean markdown | — |
+| **`web_batch_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch 1–15 pages in parallel for research synthesis (2–5 recommended) | 3 concurrent (max 5) |
 | **`web_browse`** | [agent-browser](https://github.com/vercel-labs/agent-browser) | Interact with a page (click, scroll, fill) then extract content | 25 actions |
 
+## Tools Preview
+
+A quick look at how pi renders toolkit calls while an agent searches, fetches, batches, and browses the web.
+
+<table>
+  <tr>
+    <td width="50%"><strong>Multi-tool research flow</strong><br><img src="docs/assets/screenshots/tools-workflow-preview.png" alt="pi-web-toolkit multi-tool research preview"></td>
+    <td width="50%"><strong><code>web_search</code> expanded results</strong><br><img src="docs/assets/screenshots/web-search-results-expanded.png" alt="web_search expanded results"></td>
+  </tr>
+  <tr>
+    <td width="50%"><strong><code>web_batch_fetch</code> progress</strong><br><img src="docs/assets/screenshots/web-batch-fetch-progress.png" alt="web_batch_fetch progress"></td>
+    <td width="50%"><strong><code>web_batch_fetch</code> results</strong><br><img src="docs/assets/screenshots/web-batch-fetch-results.png" alt="web_batch_fetch results"></td>
+  </tr>
+  <tr>
+    <td width="50%"><strong><code>web_fetch</code> result preview</strong><br><img src="docs/assets/screenshots/web-fetch-summary.png" alt="web_fetch result preview"></td>
+    <td width="50%"><strong><code>web_browse</code> headless browser flow</strong><br><img src="docs/assets/screenshots/web-browse-headless.png" alt="web_browse headless browser flow"></td>
+  </tr>
+  <tr>
+    <td colspan="2"><strong>End-to-end research summary</strong><br><img src="docs/assets/screenshots/web-research-workflow.png" alt="end-to-end web research workflow"></td>
+  </tr>
+</table>
+
+## Install with Pi Agent
+
+Copy and send the prompt below to Pi. It will install this package and its external dependencies for you.
+
+```text
+Install pi-web-toolkit and its external dependencies. Complete and verify every
+step yourself; do not rely on web browsing or external documentation. Inspect
+the machine first and reuse working installations. Ask before using sudo,
+changing shell profiles, overwriting configuration, or modifying existing
+services or containers.
+
+1. Ensure Node.js 22+, npm, Docker, OpenSSL, curl, uv, and Pi are installed, and
+   that Docker is running. Install only missing or incompatible prerequisites.
+2. Configure SearXNG:
+   - Test SEARXNG_URL when set, then http://localhost:8080.
+   - Verify /search?q=test&format=json returns JSON with a results array.
+   - If neither endpoint works, first ensure no existing container or config
+     would be overwritten, then create a local-only instance by running:
+
+mkdir -p "$HOME/.config/searxng"
+cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
+use_default_settings: true
+
+search:
+  formats:
+    - html
+    - json
+YAML
+
+docker run -d \
+  --name searxng \
+  --restart unless-stopped \
+  -p 127.0.0.1:8080:8080 \
+  -e FORCE_OWNERSHIP=false \
+  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
+  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
+  docker.io/searxng/searxng:latest
+
+   - Verify the selected endpoint by running:
+
+SEARXNG_ENDPOINT="${SEARXNG_URL:-http://localhost:8080}"
+curl -fsS --get "${SEARXNG_ENDPOINT%/}/search" \
+  --data-urlencode "q=test" \
+  --data "format=json" |
+  grep -q '"results"' && echo "SearXNG JSON API ready"
+
+   - Pi uses http://localhost:8080 by default. Set SEARXNG_URL before starting
+     Pi only when using another endpoint.
+3. Install and verify Scrapling:
+   uv tool install "scrapling[all]"
+   scrapling install
+   scrapling --help
+4. Install and verify agent-browser:
+   npm install -g agent-browser
+   agent-browser install
+   agent-browser doctor
+   On Linux, use agent-browser install --with-deps if required.
+5. After all dependencies pass verification, install the package:
+   pi install npm:pi-web-toolkit
+
+Report what was installed or reused, all verification results, the SearXNG
+endpoint Pi will use, and whether Pi must be restarted. Do not report success
+until every check passes.
+```
+
 ## Quick Start
 
 ### 1. Install external dependencies
 
+The commands below assume a POSIX shell with Docker, OpenSSL, curl, uv, and Node.js 22+ with npm.
+
 ```bash
-# SearXNG (for search)
-docker run -d --name searxng -p 8080:8080 -v searxng:/etc/searxng searxng/searxng
-export SEARXNG_URL="http://localhost:8080"
+# SearXNG (for search; local-only instance with the required JSON API)
+mkdir -p "$HOME/.config/searxng"
+cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
+use_default_settings: true
+
+search:
+  formats:
+    - html
+    - json
+YAML
+
+docker run -d \
+  --name searxng \
+  --restart unless-stopped \
+  -p 127.0.0.1:8080:8080 \
+  -e FORCE_OWNERSHIP=false \
+  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
+  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
+  docker.io/searxng/searxng:latest
+export SEARXNG_URL="http://127.0.0.1:8080"
 
 # scrapling (for fetch & batch fetch)
 uv tool install "scrapling[all]"
@@ -33,12 +140,16 @@ scrapling install
 
 # agent-browser (for browse)
 npm i -g agent-browser && agent-browser install
+# On Linux hosts missing browser system libraries: agent-browser install --with-deps
 ```
 
 **Verify dependencies:**
 ```bash
 # SearXNG
-curl -s "$SEARXNG_URL" | head
+curl -fsS --get "$SEARXNG_URL/search" \
+  --data-urlencode "q=searxng" \
+  --data "format=json" |
+  grep -q '"results"' && echo "SearXNG JSON API ready"
 
 # scrapling
 scrapling --help
@@ -59,7 +170,7 @@ pi install git:github.com/Wade11s/pi-web-toolkit
 
 ## Configuration
 
-All tools are configured via **environment variables** at runtime — no rebuild or restart required.
+`web_search` reads its SearXNG endpoint from an environment variable. Set it before starting pi; no build step is required.
 
 | Variable | Default | Used By | Description |
 |----------|---------|---------|-------------|
@@ -78,24 +189,37 @@ pi-web-toolkit/
 ├── extensions/
 │   ├── index.ts              # Unified entry point — registers all 4 tools
 │   ├── utils/
+│   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal
+│   │   ├── content-preview.ts # Intelligent content extraction from scraped pages
+│   │   ├── output-sink.ts    # Truncation + temp-file fallback
+│   │   ├── render-helpers.ts # URL abbreviations, text normalization, error formatting for TUI
 │   │   ├── scrapling.ts      # Reusable scrapling CLI wrapper (shared by fetch + batch)
+│   │   ├── tool-factory.ts   # Common tool registration patterns
 │   │   └── agent-browser.ts  # agent-browser CLI wrapper (shared by web_browse)
 │   ├── web_search.ts         # SearXNG search tool
 │   ├── web_fetch.ts          # Single-page scrapling fetcher
 │   ├── web_batch_fetch.ts    # Parallel scrapling fetcher
 │   └── web_browse.ts         # Interactive browser automation (agent-browser)
+├── test/
+│   ├── agent-browser/        # agent-browser output parser regression tests
+│   ├── content-preview/      # Content preview fixtures, baselines & snapshots
+│   └── README.md             # Test suite structure and conventions
 ├── docs/
 │   ├── tools.md              # Full parameter specs
-│   └── guide.md              # Decision tree & tool comparison
+│   ├── guide.md              # Decision tree & tool comparison
+│   └── agents/               # Issue tracker, triage and domain guidance
+├── AGENTS.md
+├── CONTEXT.md
 ├── CHANGELOG.md
 ├── package.json
 ├── README.md
+├── tsconfig.json
 └── LICENSE
 ```
 
 **Design principles:**
 - **Unified registration** — `index.ts` is the single source of truth for what pi loads.
-- **Shared utilities** — `utils/scrapling.ts` and `utils/agent-browser.ts` encapsulate the CLI wrappers and fallback logic; tool files import only from `utils/`, never from each other.
+- **Shared utilities** — `utils/` modules encapsulate CLI spawning, content extraction, output truncation, TUI formatting, and common registration patterns; tool files import only from `utils/`, never from each other.
 - **Per-tool isolation** — each tool owns its own schema, execute logic, and TUI renderer; no cross-imports except via `utils/`.
 - **Runtime config** — environment variables are read at execute time, not build time.
 
@@ -112,7 +236,10 @@ pi-web-toolkit/
 pi install ./
 
 # Type-check (no build step; pi loads TypeScript directly)
-npx tsc --noEmit
+npm run typecheck
+
+# Run tests
+npm run test
 
 # Verify external CLI dependencies
 scrapling --help

package/docs/agents/issue-tracker.md

@@ -5,7 +5,7 @@ Issues and PRDs for this repo live as GitHub issues. Use the `gh` CLI for all op
 ## Conventions
 
 - **Create an issue**: `gh issue create --title "..." --body "..."`. Use a heredoc for multi-line bodies.
-- **Read an issue**: `gh issue view <number> --comments`, filtering comments by `jq` and also fetching labels.
+- **Read an issue**: `gh issue view <number> --json number,title,body,labels,comments --jq '{number, title, body, labels: [.labels[].name], comments: [.comments[].body]}'`.
 - **List issues**: `gh issue list --state open --json number,title,body,labels,comments --jq '[.[] | {number, title, body, labels: [.labels[].name], comments: [.comments[].body]}]'` with appropriate `--label` and `--state` filters.
 - **Comment on an issue**: `gh issue comment <number> --body "..."`
 - **Apply / remove labels**: `gh issue edit <number> --add-label "..."` / `--remove-label "..."`
@@ -19,4 +19,4 @@ Create a GitHub issue.
 
 ## When a skill says "fetch the relevant ticket"
 
-Run `gh issue view <number> --comments`.
+Run `gh issue view <number> --json number,title,body,labels,comments --jq '{number, title, body, labels: [.labels[].name], comments: [.comments[].body]}'`.

package/docs/guide.md

@@ -8,19 +8,19 @@ User asks about something external / current
 ├─→ web_search("...")
 │   │
 │   ├─→ 1 relevant result?
-│   │   └─→ web_fetch(url)                     ← static page
+│   │   └─→ web_fetch(url)                     ← no interaction needed
 │   │   OR
 │   │   └─→ web_browse(url, actions)           ← needs interaction
 │   │
 │   └─→ 2–5 relevant results?
-│       ├─→ All static pages?
+│       ├─→ All need no interaction?
 │       │   └─→ web_batch_fetch(urls[])        ← parallel fetch
 │       └─→ Some need interaction?
-│           └─→ web_fetch (static ones)
+│           └─→ web_fetch (no-interaction ones)
 │               web_browse (interactive ones)  ← sequential
 │
 └─→ User provides a URL directly
-    ├─→ Static / loads on first request?
+    ├─→ No interaction needed / loads on first request?
     │   └─→ web_fetch(url)
     └─→ Needs clicking / scrolling / waiting?
         └─→ web_browse(url, actions)
@@ -32,10 +32,12 @@ User asks about something external / current
 
 | | `web_fetch` | `web_browse` | `web_batch_fetch` |
 |--|-------------|--------------|-------------------|
-| **Pages** | 1 | 1 | 2–15 |
-| **Browser** | Yes (scrapling) | Yes (agent-browser) | Yes (scrapling) |
+| **Pages** | 1 | 1 | 1–15 (2–5 recommended) |
+| **Browser** | Yes (Scrapling) | Yes (agent-browser) | Yes (Scrapling) |
 | **Interaction** | ❌ No | ✅ Click, fill, scroll, wait | ❌ No |
 | **Selector** | ✅ Per-URL | ✅ Final state | ✅ Applied to all |
-| **Stealthy** | ✅ Yes | ❌ No (planned) | ✅ Yes |
+| **Stealthy** | ✅ Yes | ❌ No | ✅ Yes |
 | **Speed** | Fast | Slower (browser ops) | Medium (parallel) |
 | **Best for** | Articles, docs, blogs | SPAs, forms, pagination | Research synthesis |
+
+`web_fetch` falls back to HTTP GET after a normal browser fetch fails, but not in stealthy mode. `web_batch_fetch` falls back to GET after failed browser fetches in all modes.

package/docs/tools.md

@@ -7,22 +7,24 @@ Search the web via SearXNG. Returns ranked results with title, URL, and snippet.
 ```typescript
 {
   query: string,           // Search query
-  language?: string,       // Language code (en, de, fr...). Default: "auto"
+  language?: string,       // Language code (en, en-US, de...). Omit to use the SearXNG default.
   results?: number,        // Max results (1–60). Default: 20. Automatically pages through SearXNG (up to 3 pages) if needed.
 }
 ```
 
-**When to use:** The user asks about current events, facts, or anything requiring up-to-date information. This is always the **first step** of web research.
+**When to use:** The user asks about current events, facts, or anything requiring up-to-date information and has not already provided the source URLs.
 
-**Empty results behavior:** When no results are found, `web_search` returns a list of **suggestions** — alternative queries that SearXNG believes may yield better results. The agent can use these suggestions to automatically refine and retry the search.
+**Empty results behavior:** When no results are found, `web_search` includes any query **suggestions** provided by SearXNG. The agent can use them to refine and retry the search.
 
 **Pagination:** `web_search` automatically fetches up to 3 pages from SearXNG and deduplicates by URL. You do not need to call it multiple times for deeper results.
 
+**Full output:** For non-empty searches, the formatted result output is always written to a temporary file. Returned text is also truncated to pi's default line/byte limits when necessary.
+
 ---
 
 ## `web_fetch`
 
-Fetch a single page and convert it to clean markdown. Uses scrapling's browser automation for JS-heavy sites.
+Fetch a single page and convert it to clean markdown. Uses Scrapling's browser-based `fetch` command first, then falls back to an HTTP GET when allowed. Stealthy mode uses `stealthy-fetch` and does not fall back to GET.
 
 ```typescript
 {
@@ -39,9 +41,9 @@ Fetch a single page and convert it to clean markdown. Uses scrapling's browser a
 
 **Example flow:**
 ```
-User: "What's the latest Rust release?"
-→ web_search("latest Rust programming language release")
-→ web_fetch("https://blog.rust-lang.org/2026/06/02/maintainers-fund/")
+User: "How do I install Rust?"
+→ web_search("official Rust installation guide")
+→ web_fetch("https://www.rust-lang.org/tools/install")
 → Agent answers with full context
 ```
 
@@ -51,7 +53,7 @@ User: "What's the latest Rust release?"
 
 Open a real browser, perform a chain of actions (click, fill, scroll, wait), then extract content.
 
-Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI for native browser automation via Chrome CDP.
+Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI with batched JSON commands.
 
 ```typescript
 {
@@ -65,12 +67,15 @@ Uses the [agent-browser](https://github.com/vercel-labs/agent-browser) CLI for n
     | { type: "wait_selector", selector: string, state?: "attached" | "visible" | "hidden" }
     | { type: "scroll", direction: "down" | "up" | "bottom" | "top", amount?: number }
   >,
+  // Maximum: 25 actions
   selector?: string,       // Extract content from final page state
   headless?: boolean,      // Default: true
   timeout?: number,        // Overall browser batch timeout (ms). Default: 30000
 }
 ```
 
+When `selector` is omitted, the tool returns agent-browser's interactive accessibility snapshot rather than full page text.
+
 **When to use:**
 - The page requires **clicking** before showing target content (e.g. "Load more", pagination, tab switching)
 - The page requires **filling a form** (e.g. search box, login)
@@ -123,7 +128,7 @@ Fetch multiple pages in parallel and return aggregated content.
 
 ```typescript
 {
-  urls: string[],          // 1–10 URLs
+  urls: string[],          // 1–15 URLs; 2–5 recommended
   selector?: string,       // CSS selector applied to ALL pages
   stealthy?: boolean,      // Default: false
   max_concurrency?: number // Parallel fetches (1–5). Default: 3
@@ -136,7 +141,9 @@ Fetch multiple pages in parallel and return aggregated content.
 - Comparing implementations across different docs/pages
 - Research synthesis requiring multiple sources
 
-**NOT for:** Single pages (use `web_fetch` — simpler and supports per-URL stealthy mode).
+**NOT recommended for:** Single pages. The schema accepts one URL, but `web_fetch` is simpler and provides single-page behavior.
+
+Each page starts with the selected Scrapling browser fetcher. Failed attempts fall back to HTTP GET, including when batch stealthy mode is enabled.
 
 **Example flow:**
 ```

package/extensions/index.ts

@@ -3,7 +3,7 @@
  *
  * Registers all web research tools as a single extension:
  *   - web_search: Search via SearXNG
- *   - web_fetch: Fetch static pages with scrapling
+ *   - web_fetch: Fetch a single page with scrapling
  *   - web_browse: Interactive browser automation via agent-browser
  *   - web_batch_fetch: Concurrent multi-page fetching
  */

package/extensions/utils/agent-browser.ts

@@ -25,6 +25,48 @@ export interface AgentBrowserBatchItem {
   error?: string | null;
 }
 
+function isRecord(value: unknown): value is Record<string, unknown> {
+  return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+
+function isBatchItem(value: unknown): value is AgentBrowserBatchItem {
+  return isRecord(value)
+    && typeof value.success === "boolean"
+    && Array.isArray(value.command)
+    && value.command.every((part) => typeof part === "string");
+}
+
+function describeBatchOutput(value: unknown): string {
+  if (Array.isArray(value)) return `array with ${value.length} item(s)`;
+  if (isRecord(value)) return `object with keys: ${Object.keys(value).join(", ") || "(none)"}`;
+  return typeof value;
+}
+
+export function parseAgentBrowserBatchOutput(stdout: string): AgentBrowserBatchItem[] {
+  const parsed = JSON.parse(stdout) as unknown;
+
+  if (Array.isArray(parsed)) {
+    if (parsed.every(isBatchItem)) return parsed;
+    throw new Error(`Expected every batch result item to contain { success, command }; got ${describeBatchOutput(parsed)}`);
+  }
+
+  if (isBatchItem(parsed)) {
+    return [parsed];
+  }
+
+  if (isRecord(parsed)) {
+    for (const key of ["results", "items", "data", "commands"]) {
+      const candidate = parsed[key];
+      if (Array.isArray(candidate)) {
+        if (candidate.every(isBatchItem)) return candidate;
+        throw new Error(`Expected ${key} to contain batch result items; got ${describeBatchOutput(candidate)}`);
+      }
+    }
+  }
+
+  throw new Error(`Expected JSON array of batch results; got ${describeBatchOutput(parsed)}`);
+}
+
 function requireString(action: BrowseAction, field: "selector" | "value" | "key"): string {
   const value = action[field] as string | undefined;
   if (typeof value !== "string" || value.length === 0) {
@@ -150,7 +192,7 @@ export async function runAgentBrowserBatch(
     }
 
     try {
-      return JSON.parse(result.stdout) as AgentBrowserBatchItem[];
+      return parseAgentBrowserBatchOutput(result.stdout);
     } catch (err: any) {
       throw new Error(
         `Failed to parse agent-browser output: ${err.message}\nstdout: ${result.stdout}\nstderr: ${result.stderr}`

package/extensions/utils/cli-runner.ts

@@ -3,7 +3,7 @@
  *
  * Provides a single interface for running external CLI commands
  * with consistent signal handling, timeout support, and stdout/stderr
- * collection. Enables testability by allowing the runner to be swapped.
+ * collection.
  */
 
 import { spawn, type ChildProcess } from "node:child_process";

package/extensions/web_batch_fetch.ts

@@ -109,8 +109,7 @@ const webBatchFetchTool = defineTool({
   label: "Web Batch Fetch",
   description: [
     "Fetch multiple web pages in parallel and return their content aggregated.",
-    "Use web_batch_fetch AFTER web_search when there are 2–5 relevant results",
-    "that the agent wants to read simultaneously for comparison or synthesis.",
+    "Use web_batch_fetch for 2–5 relevant URLs, whether discovered by search or provided by the user.",
     "For a single page, use web_fetch instead.",
     `Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; if truncated, full output is saved to a temp file.`,
   ].join(" "),

package/extensions/web_browse.ts

@@ -8,7 +8,7 @@
  * filling forms, waiting for dynamic content) BEFORE its target content
  * becomes available.
  *
- * For static pages that need no interaction, use `web_fetch` instead.
+ * For pages that need no interaction, use `web_fetch` instead.
  */
 
 import {
@@ -88,10 +88,10 @@ const webBrowseTool = defineTool({
   description: [
     "Interact with a web page through a browser: navigate, click, fill forms, scroll,",
     "wait for content, and then extract text.",
-    "Uses the agent-browser CLI for fast, native browser automation via Chrome CDP.",
+    "Uses the agent-browser CLI with batched JSON commands.",
     "Use web_browse when the target content requires interaction (clicking buttons,",
     "scrolling, filling search boxes, waiting for JS to load) before it becomes available.",
-    "For static pages that need no interaction, use web_fetch instead.",
+    "For pages that need no interaction, use web_fetch instead.",
     `Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; if truncated, full output is saved to a temp file.`,
   ].join(" "),
   promptSnippet: "Interact with a web page (click, scroll, fill) and extract content",
@@ -100,7 +100,7 @@ const webBrowseTool = defineTool({
     "Use web_browse for SPAs, pagination (click 'Load more'), search forms, tab switching, and modal dialogs.",
     "For static articles, docs, or blogs that load everything on first request, prefer web_fetch.",
     "After web_search returns results, prefer web_fetch for reading individual articles.",
-    "Only use web_browse if web_fetch fails to get the needed content.",
+    "Use web_browse directly when interaction is required; otherwise try web_fetch first.",
     "Always provide a selector to extract only the relevant content area — avoid dumping full page text.",
   ],
   parameters: WebBrowseParamsSchema,

package/extensions/web_fetch.ts

@@ -41,13 +41,13 @@ const webFetchTool = defineTool({
   description: [
     "Fetch and extract readable content from a web page URL.",
     "Uses scrapling to download the page and convert it to clean markdown.",
-    "Use web_fetch AFTER web_search to read the full content of a result page.",
-    "Respects robots.txt and site ToS.",
+    "Use web_fetch to read the full content of a specific result or user-provided URL.",
+    "Callers remain responsible for robots.txt and site terms; Scrapling extract commands do not enforce them automatically.",
     `Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; if truncated, full output is saved to a temp file.`,
   ].join(" "),
   promptSnippet: "Fetch full page content from a URL as markdown",
   promptGuidelines: [
-    "Use web_fetch to read a single static page (article, doc, or blog) when given a specific URL.",
+    "Use web_fetch to read a single page (article, doc, or blog) that needs no interaction.",
     "For a single URL, always use web_fetch instead of web_batch_fetch.",
     "If the page is dynamic/JavaScript-heavy, the tool automatically uses browser automation.",
     "When reading multiple (2–5) pages at once (e.g., after web_search), prefer web_batch_fetch over repeated web_fetch calls.",

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "pi-web-toolkit",
-  "version": "0.2.0",
-  "description": "Web research toolkit for the pi coding agent. Search via SearXNG, fetch static pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel.",
+  "version": "0.2.2",
+  "description": "Web research toolkit for the pi coding agent. Search via SearXNG, fetch pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel.",
   "author": "Wade Huang <fastwade11@gmail.com>",
   "license": "MIT",
   "repository": {
@@ -13,16 +13,18 @@
   },
   "homepage": "https://github.com/Wade11s/pi-web-toolkit#readme",
   "keywords": ["pi-package", "pi-extension", "web-search", "scrapling", "agent-browser"],
-  "files": ["extensions", "docs", "README.md", "package.json", "LICENSE"],
+  "files": ["extensions", "docs", "README.md", "CHANGELOG.md", "package.json", "LICENSE"],
   "engines": {
     "node": ">=22.0.0"
   },
   "scripts": {
     "typecheck": "tsc --noEmit",
-    "test": "npx tsx test/content-preview/test.ts",
-    "test:approve": "npx tsx test/content-preview/test.ts --approve"
+    "test": "tsx test/content-preview/test.ts && tsx test/agent-browser/test.ts",
+    "test:agent-browser": "tsx test/agent-browser/test.ts",
+    "test:approve": "tsx test/content-preview/test.ts --approve"
   },
   "devDependencies": {
+    "tsx": "^4.22.4",
     "typescript": "^5.7.0"
   },
   "peerDependencies": {