pi-web-toolkit 0.2.2 → 0.3.0

package/CHANGELOG.md

@@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-06-23
+
+### Added
+
+- **Firecrawl Keyless fallback** — `web_search`, `web_fetch`, and `web_browse` now automatically retry through [Firecrawl Keyless](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) (1,000 free credits/month, **no API key, no signup**) when their local backend errors out, or when `web_search` returns zero results. The fallback is keyless-only, never the primary path, and degrades gracefully to the original local-tool error if the `firecrawl-cli` is absent, the IP is flagged, the quota is exhausted, or the fallback is disabled.
+- Three explicit escape-hatch tools for capabilities the local backends lack: `firecrawl_search` (sources, `github`/`research`/`pdf` categories, domain filters), `firecrawl_scrape` (anti-bot bypass, JS rendering, PDF parsing), and `firecrawl_interact` (natural-language page interaction).
+- `extensions/utils/firecrawl.ts` — a deep Firecrawl CLI wrapper (scrape/search/interact argument builders, output parsers, graceful-skip failure classifier, keyless-eligibility check, and fallback-decision predicates).
+- Optional external CLI dependency: `npm install -g firecrawl-cli`.
+- Environment toggle `PI_WEB_FIRECRAWL_FALLBACK` (default on) to disable all Firecrawl usage.
+- `test/firecrawl/test.ts` — pure-function regression tests for the firecrawl wrapper boundary (wired into `npm test` as `test:firecrawl`).
+- ADR 0001 and `CONTEXT.md` glossary entries (`Firecrawl keyless`, `cloud fallback`, `free credits`, `graceful skip`) documenting the local-first → optional keyless cloud fallback architectural decision.
+
+### Changed
+
+- **Default network/privacy behavior.** When a local web tool fails, it now makes a cloud request to Firecrawl (sending the URL/query and page content) before giving up. The fallback is **keyless-only** — it never reads, stores, or sends an API key, and spawns the CLI under an isolated temporary `HOME` with the key env stripped. To enforce a strict local-only / no-cloud-egress policy, set `PI_WEB_FIRECRAWL_FALLBACK=0`.
+- `web_search` falls back on a SearXNG error **or** zero results; `web_fetch` falls back on a scrapling failure (incl. its HTTP-GET fallback); `web_browse` falls back only on runtime failures (missing/broken `agent-browser`), never on caller validation errors. `web_batch_fetch` has no fallback (Firecrawl batch scrape is not keyless).
+- Firecrawl results report `creditsUsed` where the source provides it (search, interact); scrape responses do not surface it.
+- README tagline and hero now describe the toolkit as local-first with an optional keyless cloud fallback; features table, install prompt, configuration, project structure, tool reference, and usage guide updated accordingly.
+- `cli-runner` gained an optional `env` passthrough so the firecrawl CLI can be spawned keyless-only.
+
 ## [0.2.2] - 2026-06-11
 
 ### Added

package/README.md

@@ -6,9 +6,9 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 ![Node.js](https://img.shields.io/badge/node-%3E%3D22-339933)
 
-**100% open-source. No required API keys or paid services.**
+**Local-first & 100% open-source. No required API keys or paid services.**
 
-Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All backends run locally or are self-hosted, with built-in truncation safety and LLM-optimized prompt guidelines.
+Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All primary backends run locally or are self-hosted, with an **optional Firecrawl Keyless cloud fallback** (no API key, no signup) so the local tools keep working when a backend is missing or fails. Built-in truncation safety and LLM-optimized prompt guidelines throughout.
 
 ## Features
 
@@ -18,6 +18,11 @@ Web research toolkit for [pi](https://pi.dev) agents. Search via SearXNG, fetch
 | **`web_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch a single page as clean markdown | — |
 | **`web_batch_fetch`** | [scrapling](https://github.com/D4Vinci/Scrapling) | Fetch 1–15 pages in parallel for research synthesis (2–5 recommended) | 3 concurrent (max 5) |
 | **`web_browse`** | [agent-browser](https://github.com/vercel-labs/agent-browser) | Interact with a page (click, scroll, fill) then extract content | 25 actions |
+| **`firecrawl_search`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud search with sources/categories/domain filters | — |
+| **`firecrawl_scrape`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud single-page fetch (anti-bot / JS / PDF) | — |
+| **`firecrawl_interact`** | [firecrawl-cli](https://github.com/firecrawl/cli) (keyless) | Cloud natural-language page interaction | — |
+
+> **Firecrawl fallback.** `web_search`, `web_fetch`, and `web_browse` automatically retry through Firecrawl Keyless (1,000 free credits/month, no API key) when their local backend errors out or search returns nothing. The three `firecrawl_*` tools are explicit escape hatches. Disable it with `PI_WEB_FIRECRAWL_FALLBACK=0`. Install the optional CLI: `npm install -g firecrawl-cli`.
 
 ## Tools Preview
 
@@ -98,7 +103,10 @@ curl -fsS --get "${SEARXNG_ENDPOINT%/}/search" \
    agent-browser install
    agent-browser doctor
    On Linux, use agent-browser install --with-deps if required.
-5. After all dependencies pass verification, install the package:
+5. Optionally install firecrawl-cli for the keyless cloud fallback (no API key
+   needed; the fallback degrades gracefully if it is absent):
+   npm install -g firecrawl-cli
+6. After all dependencies pass verification, install the package:
    pi install npm:pi-web-toolkit
 
 Report what was installed or reused, all verification results, the SearXNG
@@ -141,6 +149,9 @@ scrapling install
 # agent-browser (for browse)
 npm i -g agent-browser && agent-browser install
 # On Linux hosts missing browser system libraries: agent-browser install --with-deps
+
+# firecrawl-cli (OPTIONAL — enables the keyless cloud fallback; no API key needed)
+npm i -g firecrawl-cli
 ```
 
 **Verify dependencies:**
@@ -175,31 +186,50 @@ pi install git:github.com/Wade11s/pi-web-toolkit
 | Variable | Default | Used By | Description |
 |----------|---------|---------|-------------|
 | `SEARXNG_URL` | `http://localhost:8080` | `web_search` | Your SearXNG instance endpoint |
+| `PI_WEB_FIRECRAWL_FALLBACK` | `1` (on) | all tools | Set to `0`/`false`/`no`/`off` to disable the optional Firecrawl keyless cloud fallback for a strict local-only policy. |
 
 Set before starting pi:
 
 ```bash
 export SEARXNG_URL="https://searxng.example.com"
+# Optional: disable the Firecrawl cloud fallback entirely
+export PI_WEB_FIRECRAWL_FALLBACK=0
 ```
 
+### Optional: Firecrawl keyless fallback
+
+When a local backend (`web_search`/`web_fetch`/`web_browse`) fails or returns nothing, the tools automatically retry through [Firecrawl Keyless](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) — 1,000 free credits/month, **no API key, no signup**. The `firecrawl_*` tools are explicit escape hatches for capabilities the local backends lack (search categories, cloud rendering, natural-language interaction).
+
+Install the optional CLI (the fallback degrades gracefully if it is absent):
+
+```bash
+npm install -g firecrawl-cli
+```
+
+The fallback is **keyless-only**: it never reads or stores an API key, and spawns the CLI under an isolated temporary `HOME` with the key env stripped. **Privacy:** when the fallback runs, the URL and page content are sent to Firecrawl's cloud.
+
 ## Project Structure
 
 ```
 pi-web-toolkit/
 ├── extensions/
-│   ├── index.ts              # Unified entry point — registers all 4 tools
+│   ├── index.ts              # Unified entry point — registers all 7 tools (4 local + 3 Firecrawl keyless)
 │   ├── utils/
-│   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal
+│   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal/env
 │   │   ├── content-preview.ts # Intelligent content extraction from scraped pages
 │   │   ├── output-sink.ts    # Truncation + temp-file fallback
 │   │   ├── render-helpers.ts # URL abbreviations, text normalization, error formatting for TUI
 │   │   ├── scrapling.ts      # Reusable scrapling CLI wrapper (shared by fetch + batch)
 │   │   ├── tool-factory.ts   # Common tool registration patterns
-│   │   └── agent-browser.ts  # agent-browser CLI wrapper (shared by web_browse)
-│   ├── web_search.ts         # SearXNG search tool
-│   ├── web_fetch.ts          # Single-page scrapling fetcher
+│   │   ├── agent-browser.ts  # agent-browser CLI wrapper (shared by web_browse)
+│   │   └── firecrawl.ts      # Firecrawl keyless CLI wrapper + fallback decisions (shared by firecrawl_* tools + fallbacks)
+│   ├── web_search.ts         # SearXNG search tool (+ Firecrawl fallback)
+│   ├── web_fetch.ts          # Single-page scrapling fetcher (+ Firecrawl fallback)
 │   ├── web_batch_fetch.ts    # Parallel scrapling fetcher
-│   └── web_browse.ts         # Interactive browser automation (agent-browser)
+│   ├── web_browse.ts         # Interactive browser automation (agent-browser + Firecrawl fallback)
+│   ├── firecrawl_search.ts   # Firecrawl keyless search (escape hatch)
+│   ├── firecrawl_scrape.ts   # Firecrawl keyless single-page fetch (escape hatch)
+│   └── firecrawl_interact.ts # Firecrawl keyless natural-language interaction (escape hatch)
 ├── test/
 │   ├── agent-browser/        # agent-browser output parser regression tests
 │   ├── content-preview/      # Content preview fixtures, baselines & snapshots

package/docs/adr/0001-firecrawl-keyless-cloud-fallback.md

@@ -0,0 +1,5 @@
+# Firecrawl Keyless as an optional cloud fallback
+
+pi-web-toolkit was local-first and self-hosted by design: SearXNG, scrapling, and agent-browser all run on the user's machine, and the README guaranteed "100% open-source. No required API keys or paid services." We decided to add **Firecrawl Keyless** as a strictly optional, fallback-only cloud layer: when a local backend errors out (or `web_search` returns nothing), the same tool transparently retries through the official `firecrawl-cli` in keyless mode, and three explicit `firecrawl_*` tools are exposed for capabilities the local backends lack.
+
+This is hard to reverse once users and the agent come to rely on the fallback, surprising to a reader who assumes a local-only toolkit, and the result of a real trade-off (zero-config reliability vs. cloud egress, a privacy surface, and a third-party dependency). The fallback defaults **on**, is **keyless-only** (no API key, no signup, no stored credentials — the CLI is spawned under an isolated temp `HOME` with the key env stripped), and is **opt-out-able** via `PI_WEB_FIRECRAWL_FALLBACK=0`. We drive `firecrawl-cli` (an official Firecrawl client) rather than hand-rolling REST because Firecrawl only grants the free keyless tier to official clients, and we restrict it to the keyless endpoints (`/search`, `/scrape`, `/interact`); API-key mode, self-hosted URLs, OAuth, and non-keyless endpoints (`/map`, `/crawl`, `/batch/scrape`, etc.) are deliberately out of scope. The decision and the graceful-skip behavior (never leave the user worse off than the local tool already did) are encoded in the Firecrawl CLI wrapper module.

package/docs/guide.md

@@ -41,3 +41,26 @@ User asks about something external / current
 | **Best for** | Articles, docs, blogs | SPAs, forms, pagination | Research synthesis |
 
 `web_fetch` falls back to HTTP GET after a normal browser fetch fails, but not in stealthy mode. `web_batch_fetch` falls back to GET after failed browser fetches in all modes.
+
+---
+
+## Firecrawl Keyless fallback
+
+When a local backend cannot do the job, the tools automatically retry through **Firecrawl Keyless** (1,000 free credits/month, no API key, no signup) before giving up. It is **fallback-only** — never the primary path — and is **opt-out-able** with `PI_WEB_FIRECRAWL_FALLBACK=0`. Requires the optional `firecrawl-cli` (`npm install -g firecrawl-cli`); if it is absent the tools simply surface the original local error.
+
+| Tool | Falls back to Firecrawl when… |
+|------|-------------------------------|
+| `web_search` | SearXNG errors out **or** returns zero results |
+| `web_fetch` | scrapling (incl. its HTTP-GET fallback) fails — anti-bot, heavy JS, PDFs |
+| `web_browse` | agent-browser is missing or its batch fails (not on caller validation errors) |
+| `web_batch_fetch` | (no fallback — Firecrawl batch scrape is not keyless) |
+
+The three `firecrawl_*` tools are the explicit escape hatches for capabilities the local backends lack (`github`/`research`/`pdf` search categories, cloud rendering, natural-language interaction).
+
+**Graceful skip.** If the fallback itself cannot help — the CLI is missing, the IP is flagged as suspicious, the keyless quota is exhausted, or the fallback is disabled — the tool falls through to the original local-tool error so the user is never left worse off.
+
+**Credit budgeting.** Search ≈ 2 credits / 10 results, scrape ≈ 1 credit / page, interact ≈ 2 credits/min (code-only) or ≈ 7 credits/min (AI prompt). Results report `creditsUsed` where the source provides it. The fallback stays conservative (small limits) against the 1,000 credits/month allowance.
+
+**Privacy.** Firecrawl is a cloud service: when the fallback runs, the URL/query and page content leave the machine. Set `PI_WEB_FIRECRAWL_FALLBACK=0` to enforce a strict local-only, no-cloud-egress policy. The fallback is **keyless-only** — it never reads, stores, or sends an API key, and spawns the CLI under an isolated temporary `HOME`.
+
+---

package/docs/tools.md

@@ -160,3 +160,65 @@ User: "Compare Python asyncio, Trio, and curio"
   })
 → Agent synthesizes comparison from all 3 sources
 ```
+
+---
+
+## Firecrawl keyless tools (optional cloud escape hatches)
+
+These three tools talk to [Firecrawl](https://www.firecrawl.dev) in **keyless** mode: 1,000 free credits/month, **no API key and no signup**. They require the optional `firecrawl-cli` (`npm install -g firecrawl-cli`). **Privacy:** the URL/query/page content is sent to Firecrawl's cloud.
+
+They double as the implementation of the automatic fallback: `web_search`/`web_fetch`/`web_browse` retry through Firecrawl keyless when their local backend fails (or search returns nothing). Disable all Firecrawl usage with `PI_WEB_FIRECRAWL_FALLBACK=0`.
+
+### `firecrawl_search`
+
+Cloud web search via Firecrawl keyless, with capabilities the local SearXNG tool lacks.
+
+```typescript
+{
+  query: string,
+  limit?: number,                       // 1–100. Default 10
+  sources?: Array<"web"|"images"|"news">,
+  categories?: Array<"github"|"research"|"pdf">,
+  country?: string,                     // ISO code, e.g. "US", "DE"
+  tbs?: string,                         // qdr:h|d|w|m|y
+  location?: string,
+  includeDomains?: string[],            // hostnames; folded into the query as site: operators
+  excludeDomains?: string[],
+}
+```
+
+**When to use:** `web_search` failed or returned nothing; or you need `github`/`research`/`pdf` categories, images/news sources, or domain scoping that SearXNG does not provide.
+
+### `firecrawl_scrape`
+
+Cloud single-page fetch via Firecrawl keyless (anti-bot bypass, JS rendering, PDF parsing).
+
+```typescript
+{
+  url: string,
+  waitFor?: number,           // ms to wait for JS rendering
+  includeTags?: string[],     // Firecrawl tag filter (not a CSS selector)
+  excludeTags?: string[],
+  onlyMainContent?: boolean,  // Default: true
+}
+```
+
+**When to use:** `web_fetch` failed on an anti-bot-protected, JavaScript-heavy, or PDF page.
+
+### `firecrawl_interact`
+
+Open a URL in a live Firecrawl browser session and drive it with a natural-language prompt (or code).
+
+```typescript
+{
+  url: string,
+  prompt?: string,                            // natural-language task (required unless code is set)
+  code?: string,                              // code to run in the browser sandbox
+  language?: "node"|"python"|"bash",
+  timeout?: number,                           // seconds (1–300)
+}
+```
+
+**When to use:** `web_browse` cannot run (agent-browser missing / OS deps missing), or you want natural-language page interaction without hand-written CSS selectors. Write each prompt as a single, focused task.
+
+---

package/extensions/firecrawl_interact.ts

@@ -0,0 +1,147 @@
+/**
+ * Firecrawl Interact Extension — natural-language browser interaction (keyless)
+ *
+ * Provides a `firecrawl_interact` tool that scrapes a URL to start a live
+ * Firecrawl browser session, then drives the page with a natural-language
+ * prompt (or code) and returns the result. It is an escape hatch for
+ * interactive pages the local agent-browser tool cannot run (missing CLI,
+ * missing OS browser deps), and underpins the automatic `web_browse` fallback.
+ *
+ * Requires: `npm install -g firecrawl-cli` (optional; degrades gracefully).
+ * Privacy: the URL, page content, and prompt are sent to Firecrawl's cloud.
+ */
+
+import {
+  defineTool,
+  type ExtensionAPI,
+  formatSize,
+  DEFAULT_MAX_BYTES,
+  DEFAULT_MAX_LINES,
+} from "@earendil-works/pi-coding-agent";
+import { Text } from "@earendil-works/pi-tui";
+import { Type, type Static } from "typebox";
+import { StringEnum } from "@earendil-works/pi-ai";
+import { interactKeyless } from "./utils/firecrawl";
+import { writeWithFallback } from "./utils/output-sink";
+import { abbreviateUrl, getErrorText, normalizeWhitespace } from "./utils/render-helpers";
+
+export const FirecrawlInteractParamsSchema = Type.Object({
+  url: Type.String({ description: "Full URL to open and interact with" }),
+  prompt: Type.Optional(Type.String({ description: "Natural-language task for the AI agent (e.g. 'Click the pricing tab and return the price')" })),
+  code: Type.Optional(Type.String({ description: "Code to execute in the browser sandbox instead of a prompt" })),
+  language: Type.Optional(StringEnum(["node", "python", "bash"] as const)),
+  timeout: Type.Optional(Type.Integer({ description: "Timeout in seconds (1-300). Default: 30", minimum: 1, maximum: 300 })),
+});
+
+export type FirecrawlInteractInput = Static<typeof FirecrawlInteractParamsSchema>;
+
+const firecrawlInteractTool = defineTool({
+  name: "firecrawl_interact",
+  label: "Firecrawl Interact",
+  description: [
+    "Open a URL in a live Firecrawl browser session and drive it with a natural-language",
+    "prompt (or code), returning the result. Keyless — no API key, no signup.",
+    "Use firecrawl_interact when the local web_browse cannot run, or when you want",
+    "natural-language page interaction without CSS selectors.",
+    "Privacy: the URL, page content, and prompt are sent to Firecrawl's cloud.",
+    `Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; if truncated, full output is saved to a temp file.`,
+  ].join(" "),
+  promptSnippet: "Drive a page via Firecrawl keyless (natural-language interaction)",
+  promptGuidelines: [
+    "Prefer web_browse first; reach for firecrawl_interact when web_browse can't run or you want NL interaction.",
+    "Write each prompt as a single, focused task; the session can be reused across calls.",
+    "Always pass the full URL including https://.",
+  ],
+  parameters: FirecrawlInteractParamsSchema,
+
+  async execute(_toolCallId, params, signal) {
+    if (!params.prompt && !params.code) {
+      throw new Error("firecrawl_interact requires either a prompt or code.");
+    }
+    const out = await interactKeyless(
+      params.url,
+      { prompt: params.prompt, code: params.code, language: params.language, timeout: params.timeout },
+      signal,
+    );
+
+    if (!out.ok) {
+      const reason = out.failure?.reason ?? "unknown error";
+      throw new Error(`Firecrawl interact failed (${out.failure?.kind}): ${reason}`);
+    }
+
+    const rawText = `Interacted: ${params.url}\n(via Firecrawl keyless${out.creditsUsed !== undefined ? `, ${out.creditsUsed} credits` : ""})\n${out.liveViewUrl ? `Live view: ${out.liveViewUrl}\n` : ""}\n---\n\n${out.output || "(no output)"}`;
+    const sink = await writeWithFallback(rawText, { tmpPrefix: "pi-firecrawl-interact-" });
+    const preview = (out.output || "").replace(/\s+/g, " ").trim().slice(0, 500);
+
+    return {
+      content: [{ type: "text", text: sink.text }],
+      details: {
+        url: params.url,
+        output: out.output,
+        preview,
+        fullOutputPath: sink.fullOutputPath,
+        liveViewUrl: out.liveViewUrl,
+        creditsUsed: out.creditsUsed,
+        viaFirecrawl: true,
+      },
+    };
+  },
+
+  renderCall(args, theme) {
+    let text = theme.fg("toolTitle", theme.bold("firecrawl_interact "));
+    text += theme.fg("muted", args.url);
+    if (args.prompt) text += theme.fg("dim", ` — ${args.prompt.slice(0, 60)}`);
+    return new Text(text, 0, 0);
+  },
+
+  renderResult(result, { expanded, isPartial }, theme, context) {
+    const isError = context?.isError ?? false;
+
+    if (isPartial) {
+      return new Text(theme.fg("warning", "Interacting via Firecrawl..."), 0, 0);
+    }
+
+    const details = result.details as {
+      url?: string;
+      output?: string;
+      preview?: string;
+      fullOutputPath?: string;
+      liveViewUrl?: string;
+      creditsUsed?: number;
+    } | undefined;
+
+    if (isError) {
+      const errText = getErrorText(result);
+      let text = theme.fg("error", "✗ Firecrawl interact failed");
+      if (details?.url) text += `  ${theme.fg("dim", abbreviateUrl(details.url))}`;
+      text += `\n\n  ${theme.fg("toolOutput", errText)}`;
+      return new Text(text, 0, 0);
+    }
+
+    let text = theme.fg("success", "✓ Interacted");
+    text += theme.fg("accent", " [Firecrawl keyless]");
+    if (details?.url) text += `  ${theme.fg("dim", abbreviateUrl(details.url))}`;
+    if (details?.creditsUsed !== undefined) text += theme.fg("muted", ` ${details.creditsUsed} credits`);
+
+    if (!expanded && details?.preview) {
+      const snippet = normalizeWhitespace(details.preview);
+      const short = snippet.length > 160 ? snippet.slice(0, 160).replace(/\s+\S*$/, "") + "..." : snippet;
+      text += `\n\n  ${theme.fg("muted", short)}`;
+    }
+
+    if (expanded) {
+      if (details?.output) {
+        text += `\n\n  ${theme.fg("muted", normalizeWhitespace(details.output))}`;
+      }
+      if (details?.fullOutputPath) {
+        text += `\n\n${theme.fg("accent", `Full output: ${details.fullOutputPath}`)}`;
+      }
+    }
+
+    return new Text(text, 0, 0);
+  },
+});
+
+export default function (pi: ExtensionAPI) {
+  pi.registerTool(firecrawlInteractTool);
+}

package/extensions/firecrawl_scrape.ts

@@ -0,0 +1,154 @@
+/**
+ * Firecrawl Scrape Extension — single-page fetch via firecrawl-cli (keyless)
+ *
+ * Provides a `firecrawl_scrape` tool that fetches a single URL as clean
+ * markdown through the official Firecrawl CLI in keyless mode (no API key,
+ * no signup). It is an explicit escape hatch for hard targets the local
+ * scrapling fetcher cannot handle (anti-bot, heavy JS, PDFs), and also
+ * underpins the automatic `web_fetch` fallback.
+ *
+ * Requires: `npm install -g firecrawl-cli` (optional; the tool degrades
+ * gracefully and reports when the CLI is unavailable).
+ *
+ * Privacy: the URL and page content are sent to Firecrawl's cloud.
+ */
+
+import {
+  defineTool,
+  type ExtensionAPI,
+  formatSize,
+  DEFAULT_MAX_BYTES,
+  DEFAULT_MAX_LINES,
+} from "@earendil-works/pi-coding-agent";
+import { Text } from "@earendil-works/pi-tui";
+import { Type, type Static } from "typebox";
+import { scrapeKeyless } from "./utils/firecrawl";
+import { extractPreview } from "./utils/content-preview";
+import { writeWithFallback } from "./utils/output-sink";
+import { abbreviateUrl, getErrorText, normalizeWhitespace, formatExtraction } from "./utils/render-helpers";
+
+export const FirecrawlScrapeParamsSchema = Type.Object({
+  url: Type.String({ description: "Full URL to fetch (e.g. https://example.com/article)" }),
+  waitFor: Type.Optional(Type.Integer({ description: "Wait (ms) before scraping for JS-rendered content", minimum: 0 })),
+  includeTags: Type.Optional(Type.Array(Type.String(), { description: "HTML tags to include (Firecrawl tag filter, not a CSS selector)" })),
+  excludeTags: Type.Optional(Type.Array(Type.String(), { description: "HTML tags to exclude" })),
+  onlyMainContent: Type.Optional(Type.Boolean({ description: "Extract only main content (drop nav/footer). Default: true", default: true })),
+});
+
+export type FirecrawlScrapeInput = Static<typeof FirecrawlScrapeParamsSchema>;
+
+const firecrawlScrapeTool = defineTool({
+  name: "firecrawl_scrape",
+  label: "Firecrawl Scrape",
+  description: [
+    "Fetch a single page as clean markdown via Firecrawl (keyless — no API key, no signup).",
+    "Use firecrawl_scrape when the local web_fetch fails on a hard target (anti-bot,",
+    "JavaScript-heavy pages, PDFs) or when you need Firecrawl's cloud rendering directly.",
+    "Privacy: the URL and page content are sent to Firecrawl's cloud.",
+    `Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; if truncated, full output is saved to a temp file.`,
+  ].join(" "),
+  promptSnippet: "Fetch a single page via Firecrawl keyless (anti-bot / JS / PDF fallback)",
+  promptGuidelines: [
+    "Prefer web_fetch first; reach for firecrawl_scrape when web_fetch fails or you need cloud rendering.",
+    "firecrawl_scrape handles anti-bot protection, JS-heavy SPAs, and PDFs that scrapling may miss.",
+    "Always pass the full URL including https://.",
+  ],
+  parameters: FirecrawlScrapeParamsSchema,
+
+  async execute(_toolCallId, params, signal) {
+    const out = await scrapeKeyless(params.url, {
+      waitFor: params.waitFor,
+      includeTags: params.includeTags,
+      excludeTags: params.excludeTags,
+      onlyMainContent: params.onlyMainContent,
+    }, signal);
+
+    if (!out.ok) {
+      const reason = out.failure?.reason ?? "unknown error";
+      throw new Error(`Firecrawl scrape failed (${out.failure?.kind}): ${reason}`);
+    }
+
+    const preview = extractPreview(out.content, 500);
+    const rawText = `Fetched: ${params.url}\n(via Firecrawl keyless${out.creditsUsed !== undefined ? `, ${out.creditsUsed} credits` : ""})\nSize: ${out.bytes} bytes\n\n---\n\n${out.content}`;
+    const sink = await writeWithFallback(rawText, { tmpPrefix: "pi-firecrawl-scrape-full-" });
+
+    return {
+      content: [{ type: "text", text: sink.text }],
+      details: {
+        url: params.url,
+        bytes: out.bytes,
+        fullOutputPath: sink.fullOutputPath,
+        preview,
+        title: out.title,
+        creditsUsed: out.creditsUsed,
+        viaFirecrawl: true,
+      },
+    };
+  },
+
+  renderCall(args, theme) {
+    let text = theme.fg("toolTitle", theme.bold("firecrawl_scrape "));
+    text += theme.fg("muted", args.url);
+    if (args.waitFor) {
+      text += theme.fg("dim", ` [wait=${args.waitFor}]`);
+    }
+    return new Text(text, 0, 0);
+  },
+
+  renderResult(result, { expanded, isPartial }, theme, context) {
+    const isError = context?.isError ?? false;
+
+    if (isPartial) {
+      return new Text(theme.fg("warning", "Scraping via Firecrawl..."), 0, 0);
+    }
+
+    const details = result.details as {
+      url?: string;
+      bytes?: number;
+      fullOutputPath?: string;
+      preview?: string;
+      title?: string;
+      creditsUsed?: number;
+    } | undefined;
+
+    if (isError) {
+      const errText = getErrorText(result);
+      let text = theme.fg("error", "✗ Firecrawl scrape failed");
+      if (details?.url) text += `  ${theme.fg("dim", abbreviateUrl(details.url))}`;
+      text += `\n\n  ${theme.fg("toolOutput", errText)}`;
+      return new Text(text, 0, 0);
+    }
+
+    let text = theme.fg("success", "✓ Fetched");
+    text += theme.fg("accent", " [Firecrawl keyless]");
+    if (details?.title) {
+      text += `  ${theme.fg("toolTitle", details.title)}`;
+    } else if (details?.url) {
+      text += `  ${theme.fg("dim", abbreviateUrl(details.url))}`;
+    }
+    if (details?.bytes && details?.preview) {
+      text += `  ${theme.fg("muted", formatExtraction(details.bytes, details.preview.length))}`;
+    }
+
+    if (!expanded && details?.preview) {
+      const snippet = normalizeWhitespace(details.preview);
+      const short = snippet.length > 160 ? snippet.slice(0, 160).replace(/\s+\S*$/, "") + "..." : snippet;
+      text += `\n\n  ${theme.fg("muted", short)}`;
+    }
+
+    if (expanded) {
+      if (details?.preview) {
+        text += `\n\n  ${theme.fg("muted", normalizeWhitespace(details.preview))}`;
+      }
+      if (details?.fullOutputPath) {
+        text += `\n\n${theme.fg("accent", `Full output: ${details.fullOutputPath}`)}`;
+      }
+    }
+
+    return new Text(text, 0, 0);
+  },
+});
+
+export default function (pi: ExtensionAPI) {
+  pi.registerTool(firecrawlScrapeTool);
+}