@bacnh85/pi-web 0.5.7 → 0.6.0

package/CHANGELOG.md

@@ -1,13 +1,20 @@
 # Changelog
 
+## 0.6.0 (2026-08-07)
+
+### Features
+
+- **agy extraction backend:** `web_extract` gains a new `agy` mode that uses the Antigravity CLI (Gemini/Claude) native `read_url` web tool to fetch bot-protected and anti-AI-scraping pages that block Firecrawl/Crawl4AI. `auto` mode now falls back static → dynamic → full → agy; explicit `mode: "agy"` forces it. Structured extraction (`prompt`/`schema`) is supported. `web_status` reports `agy.installed`.
+- agy is optional and self-contained: if the CLI is not installed, `auto` mode skips it silently and existing flows are unchanged. Install: `curl -fsSL https://antigravity.google/cli/install.sh | bash`, then authenticate once with `agy`.
+
+All notable changes to `pi-web` will be documented in this file.
+
 ## 0.5.7 (2026-08-05)
 
 ### Improvements
 
 - Patch version bump for release sync and package documentation update.
 
-All notable changes to `pi-web` will be documented in this file.
-
 ## 0.5.6 (2026-08-01)
 
 ### Features

package/README.md

@@ -2,7 +2,7 @@
 
 Pi extension for **unified web search, content extraction, site crawling, and page capture**.
 
-Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, and Crawl4AI — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.
+Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, Crawl4AI, and agy (Gemini/Claude, when installed) — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.
 
 ## Install
 
@@ -79,12 +79,13 @@ Use `backend` parameter to force a specific backend when needed.
 
 ### `web_extract` — Unified content extraction
 
-Extracts readable content from a URL. Auto-selects backend: static (JSDOM) → dynamic (Firecrawl) → full (Crawl4AI), with extraction diagnostics showing fallback attempts.
+Extracts readable content from a URL. Auto-selects backend: static (JSDOM) → dynamic (Firecrawl) → full (Crawl4AI) → agy (model-backed), with extraction diagnostics showing fallback attempts.
 
 ```
 web_extract url="https://docs.ansible.com/..."
 web_extract url="https://riven.tv/" mode=static
 web_extract url="https://example.com" mode=dynamic prompt="Extract pricing plans"
+web_extract url="https://blocked.example.com" mode=agy
 ```
 
 Parameters:
@@ -92,9 +93,9 @@ Parameters:
 | Parameter | Type | Default | Description |
 |---|---|---|---|
 | `url` | string | — | URL to extract |
-| `mode` | string | `auto` | `auto`, `static`, `dynamic`, or `full` |
-| `prompt` | string | — | Prompt for JSON extraction (dynamic mode) |
-| `schema` | any | — | JSON schema for structured extraction (dynamic mode) |
+| `mode` | string | `auto` | `auto`, `static`, `dynamic`, `full`, or `agy` |
+| `prompt` | string | — | Prompt for JSON extraction (dynamic/agy modes) |
+| `schema` | any | — | JSON schema for structured extraction (dynamic/agy modes) |
 | `content_chars` | number | 20000 | Max content chars |
 | `wait_for` | number | — | Milliseconds to wait for Firecrawl dynamic rendering. Crawl4AI `/md` full mode may ignore this. |
 | `mobile` | boolean | false | Emulate mobile viewport (dynamic mode) |
@@ -106,11 +107,14 @@ Parameters:
 | `static` | JSDOM+Readability | Simple static pages, blog posts, docs | No |
 | `dynamic` | Firecrawl Scrape | JS-rendered pages, dynamic content | Maybe |
 | `full` | Crawl4AI | JS-heavy SPA, complex rendering | Maybe |
-| `auto` (default) | static → dynamic → full | Unknown page type | Maybe |
+| `agy` | agy (Gemini/Claude) | Bot-protected / anti-AI-scraping pages | agy CLI installed |
+| `auto` (default) | static → dynamic → full → agy | Unknown page type | Maybe |
 
 In `auto` mode, fallbacks are noted in the output (e.g., `[Extraction fell back to Firecrawl Scrape (dynamic mode)]`). If `static` extraction fails, the tool gracefully escalates to heavier backends.
 
-> ⚠️ **Note on Firecrawl Scrape**: Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls back to `full` mode (Crawl4AI) in `auto` mode.
+> ⚠️ **Note on Firecrawl Scrape**: Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls back to `full` mode (Crawl4AI) in `auto` mode, and to `agy` mode as a last resort.
+
+> **`agy` mode (optional)**: Uses the [Antigravity CLI](https://antigravity.google/) with Gemini/Claude — its native `read_url` browser tool can fetch pages that block Firecrawl/Crawl4AI. Install with `curl -fsSL https://antigravity.google/cli/install.sh | bash`, authenticate once with `agy`, then `auto` mode falls back to it automatically. If agy is not installed, `auto` mode skips it silently; `web_status` reports `agy.installed`.
 
 ### `web_map` — Site URL discovery
 
@@ -172,7 +176,8 @@ Typical output:
     "baseUrl": "http://172.30.55.22:11235",
     ...
     "health": { "status": "healthy", "version": "0.5.0", ... }
-  }
+  },
+  "agy": { "installed": true }
 }
 ```
 
@@ -188,6 +193,7 @@ Typical output:
 | `lib/searxng.ts` | SearXNG metasearch fetch client (internal) |
 | `lib/firecrawl.ts` | Firecrawl API fetch client with v2→v1 fallback (internal) |
 | `lib/crawl4ai.ts` | Crawl4AI Docker API fetch client (internal) |
+| `lib/agy.ts` | agy (Antigravity CLI) spawn helper — `read_url` extraction via Gemini/Claude |
 | `lib/search.ts` | Unified search orchestrator — probes backends, fallback chain |
 | `lib/extract.ts` | Unified extraction orchestrator — mode-based backend selection |
 

package/extensions/index.ts

@@ -62,7 +62,8 @@ The pi-web extension provides 7 unified tools that auto-select the best backend:
 - **\`web_search\`** — Search the web (auto: SearXNG → Brave → Firecrawl). Use
   \`backend\` for explicit control, \`engines\` for SearXNG tuning.
 - **\`web_extract\`** — Extract readable content from a URL (auto: static JSDOM
-  → dynamic Firecrawl → full Crawl4AI). Use \`mode\` for explicit control.
+  → dynamic Firecrawl → full Crawl4AI → agy model-backed). Use \`mode\` for
+  explicit control.
 - **\`web_map\`** — Discover URLs from a site (Firecrawl Map).
 - **\`web_crawl\`** — Crawl multiple pages. \`mode: "light"\` (Firecrawl) or
   \`mode: "full"\` (Crawl4AI).
@@ -74,7 +75,8 @@ Backend selection rules:
 - Firecrawl Search has poor semantic accuracy on domain-specific queries; prefer
   SearXNG or Brave for precision (force via \`backend\`).
 - Firecrawl Scrape fails on bot-protected sites (e.g. Ansible docs); Crawl4AI
-  handles those (force via \`mode: "full"\`).
+  handles those (force via \`mode: "full"\`), and agy (Gemini/Claude read_url)
+  handles the rest as a last-resort fallback (force via \`mode: "agy"\`).
 - Always cite source URLs when web results materially support an answer.`;
 
 
@@ -136,23 +138,24 @@ export default function piWebExtension(pi: ExtensionAPI) {
     name: "web_extract",
     label: "Web Content Extraction",
     description:
-      "Extract readable content from a URL. Auto mode: static\u2192dynamic\u2192full.",
+      "Extract readable content from a URL. Auto mode: static\u2192dynamic\u2192full\u2192agy.",
     promptSnippet: "Extract readable webpage content as markdown",
     promptGuidelines: [
       "Clean markdown from a known URL.",
-      "mode: 'static' (no API key, JSDOM), 'dynamic' (Firecrawl JS), 'full' (Crawl4AI).",
-      "'auto' tries static\u2192dynamic\u2192full; see diagnostics for fallback chain.",
-      "Use prompt+schema for structured JSON extraction (dynamic mode only).",
+      "mode: 'static' (no API key, JSDOM), 'dynamic' (Firecrawl JS), 'full' (Crawl4AI), 'agy' (Gemini/Claude via agy).",
+      "'auto' tries static\u2192dynamic\u2192full\u2192agy; see diagnostics for fallback chain.",
+      "mode: 'agy' uses agy's native read_url for bot-protected/JS-heavy pages \u2014 last-resort fallback in auto.",
+      "Use prompt+schema for structured JSON extraction (dynamic/agy modes).",
       "Cite the source URL.",
     ],
     parameters: Type.Object({
       url: Type.String(),
       mode: Type.Optional(Type.Union(
-        [Type.Literal("auto"), Type.Literal("static"), Type.Literal("dynamic"), Type.Literal("full")],
-        { default: "auto", description: "auto, static, dynamic, full." },
+        [Type.Literal("auto"), Type.Literal("static"), Type.Literal("dynamic"), Type.Literal("full"), Type.Literal("agy")],
+        { default: "auto", description: "auto, static, dynamic, full, agy." },
       )),
-      prompt: Type.Optional(Type.String({ description: "Prompt for structured JSON extraction (dynamic mode only)." })),
-      schema: Type.Optional(Type.Any({ description: "JSON schema for structured extraction (dynamic mode only)." })),
+      prompt: Type.Optional(Type.String({ description: "Prompt for structured JSON extraction (dynamic/agy modes)." })),
+      schema: Type.Optional(Type.Any({ description: "JSON schema for structured extraction (dynamic/agy modes)." })),
       content_chars: Type.Optional(Type.Number({ default: 20000 })),
       wait_for: Type.Optional(Type.Number({ description: "Ms to wait for Firecrawl render before extraction." })),
       mobile: Type.Optional(Type.Boolean({ default: false, description: "Mobile viewport (dynamic mode only)." })),
@@ -407,6 +410,8 @@ export default function piWebExtension(pi: ExtensionAPI) {
       const c4aiUrl = findEnvValue("CRAWL4AI_API_URL", cwd, trusted);
       const c4aiToken = findEnvValue("CRAWL4AI_API_TOKEN", cwd, trusted);
 
+      const { isAgyInstalled } = await import("./lib/agy");
+
       const fcBaseUrl = normalizeFirecrawlBaseUrl(fireUrl.value);
       const fcHosted = !fireUrl.value || fcBaseUrl.startsWith(HOSTED_FIRECRAWL_BASE_URL);
 
@@ -427,6 +432,7 @@ export default function piWebExtension(pi: ExtensionAPI) {
           apiTokenFound: Boolean(c4aiToken.value),
           apiTokenSource: c4aiToken.value ? c4aiToken.source : "not set",
         },
+        agy: { installed: isAgyInstalled() },
       };
 
       // Crawl4AI health check

package/extensions/lib/agy.ts

@@ -0,0 +1,201 @@
+// agy-based web extraction — uses agy's native read_url tool via Gemini/Claude.
+// Self-contained: does NOT import from pi-agy (different concern, no coupling).
+//
+// Runs agy in --mode plan (read-only: no file writes). Verified (agy 1.1.11):
+// plan mode auto-approves read_url in headless -p WITHOUT
+// --dangerously-skip-permissions, so no broad permission bypass is needed.
+
+import { createRequire } from "node:module";
+
+import { sanitizeError } from "./format";
+
+// ponytail: require (not ESM import) so tests can patch cp.spawn/spawnSync
+// without mocking the module graph — same pattern as pi-agy.
+const _require = createRequire(import.meta.url);
+const cp = _require("node:child_process") as typeof import("node:child_process");
+
+const AGY_FETCH_TIMEOUT_MS = 90_000; // agy needs time for model call + web fetch
+const AGY_PROBE_TIMEOUT_MS = 5_000;
+const AGY_MAX_OUTPUT_BYTES = 200_000; // bound output to protect Pi context
+export const AGY_MODEL = "gemini-3.6-flash-medium"; // ponytail: fixed default; users needing model control use agy_execute
+
+// Cache install status with a TTL — spawnSync blocks the event loop up to
+// AGY_PROBE_TIMEOUT_MS, and web_status/extract can call this repeatedly.
+let agyInstalledCache: { ok: boolean; at: number } | null = null;
+const AGY_INSTALL_CACHE_TTL_MS = 60_000;
+
+export function isAgyInstalled(): boolean {
+  if (agyInstalledCache && Date.now() - agyInstalledCache.at < AGY_INSTALL_CACHE_TTL_MS) {
+    return agyInstalledCache.ok;
+  }
+  // ponytail: spawnSync is the simplest reliable probe; result is cached so the
+  // event-loop block happens at most once per 60s.
+  try {
+    const r = cp.spawnSync("agy", ["--version"], { timeout: AGY_PROBE_TIMEOUT_MS, stdio: "ignore" });
+    agyInstalledCache = { ok: r.status === 0, at: Date.now() };
+    return agyInstalledCache.ok;
+  } catch {
+    agyInstalledCache = { ok: false, at: Date.now() };
+    return false;
+  }
+}
+
+// Test-only: clear the cached install status.
+export function resetAgyInstalledCache(): void {
+  agyInstalledCache = null;
+}
+
+// Validate + sanitize a URL before it is interpolated into the agy model prompt.
+// Prompt-injection guard: reject non-http(s) schemes and strip control chars/newlines
+// so a crafted URL cannot break out of the "fetch this URL" framing.
+export function sanitizeAgyUrl(url: string): string {
+  const trimmed = url.trim();
+  let parsed: URL;
+  try {
+    parsed = new URL(trimmed);
+  } catch {
+    throw new Error(`Invalid URL for agy extraction: ${trimmed.slice(0, 200)}`);
+  }
+  if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
+    throw new Error(`Unsupported URL scheme for agy extraction: ${parsed.protocol}`);
+  }
+  // Neutralize any control characters/newlines so the URL stays on one line
+  // inside the prompt and cannot smuggle instructions.
+  return trimmed.replace(/[\x00-\x1f\x7f]/g, "");
+}
+
+// Build the agy CLI args for a web-fetch task. Pure — tested without mocking.
+export function buildAgyFetchArgs(url: string, prompt?: string, schema?: unknown): string[] {
+  const safeUrl = sanitizeAgyUrl(url);
+  const structured = prompt || schema !== undefined;
+  const jsonInstruction = schema !== undefined
+    ? `\nReturn as JSON matching this schema: ${JSON.stringify(schema)}`
+    : "\nReturn the result as JSON.";
+  const fetchInstruction = structured
+    ? (prompt ? `Then extract this information: ${prompt}` : "Then extract the requested fields.") + jsonInstruction
+    : "Return the full page content as clean markdown.";
+
+  const agyPrompt = `Use your read_url web tool to fetch this URL: ${safeUrl}\n\n${fetchInstruction}\n\nReturn ONLY the result, no commentary.`;
+
+  return [
+    "--model",
+    AGY_MODEL,
+    "--mode",
+    "plan", // read-only: no file writes
+    "--print-timeout",
+    "90s",
+    // Verified (agy 1.1.11): plan mode auto-approves read_url in headless -p
+    // WITHOUT --dangerously-skip-permissions — pi-agy's flag is only needed for
+    // write modes (accept-edits/sandbox). Omitting it keeps the auto-approval
+    // surface at read-only web tools only.
+    "--output-format",
+    "json", // structured response for clean parsing
+    "-p",
+    agyPrompt,
+  ];
+}
+
+export async function extractViaAgy(params: {
+  url: string;
+  prompt?: string; // structured extraction prompt
+  schema?: unknown; // JSON schema for structured extraction
+  contentChars?: number;
+  signal?: AbortSignal;
+}): Promise<string> {
+  const { url, prompt, schema, contentChars, signal } = params;
+  const output = await spawnAgyRaw(buildAgyFetchArgs(url, prompt, schema), signal);
+  return parseAgyResponse(output, contentChars ?? 20000);
+}
+
+// Parse structured JSON out of agy's model output — used when a prompt/schema was
+// requested so the structured result is first-class, matching dynamic mode's
+// `structured` field. Handles both fenced (```json ... ```) and bare JSON, which
+// the model produces nondeterministically. Pure — tested without mocking.
+export function parseAgyStructured(markdown: string): unknown {
+  const trimmed = markdown.trim();
+  if (!trimmed) return undefined;
+  // Fenced block first, then bare JSON object/array.
+  const m = trimmed.match(/^```(?:json)?\s*([\s\S]*?)```/);
+  const candidate = m ? m[1] : trimmed.startsWith("{") || trimmed.startsWith("[") ? trimmed : undefined;
+  if (candidate === undefined) return undefined;
+  try {
+    return JSON.parse(candidate);
+  } catch {
+    return undefined;
+  }
+}
+
+// Spawn agy with bounded output collection + proper error handling.
+async function spawnAgyRaw(args: string[], signal?: AbortSignal): Promise<string> {
+  // ponytail: agy is a Go binary in PATH (official installer); no shell wrapper.
+  const child = cp.spawn("agy", args, {
+    stdio: ["ignore", "pipe", "pipe"],
+    signal,
+    timeout: AGY_FETCH_TIMEOUT_MS + 5_000, // let agy report its own print timeout first
+  });
+
+  const stdout: Buffer[] = [];
+  const stderr: Buffer[] = [];
+  let stdoutBytes = 0;
+  let stderrBytes = 0;
+
+  child.stdout.on("data", (d: Buffer) => {
+    // Capture all bytes up to the cap, including the partial final chunk
+    // (mirrors pi-agy's appendBounded behavior).
+    if (stdoutBytes < AGY_MAX_OUTPUT_BYTES) {
+      stdout.push(d.subarray(0, AGY_MAX_OUTPUT_BYTES - stdoutBytes));
+    }
+    stdoutBytes += d.length;
+  });
+  child.stderr.on("data", (d: Buffer) => {
+    stderrBytes += d.length;
+    if (stderrBytes <= 16 * 1024) stderr.push(d);
+  });
+
+  return new Promise<string>((resolve, reject) => {
+    let settled = false;
+    const done = (fn: () => void) => {
+      if (settled) return;
+      settled = true;
+      fn();
+    };
+
+    child.on("error", (err: Error) => {
+      done(() => {
+        if (signal?.aborted) reject(new Error("agy was cancelled"));
+        else if ((err as NodeJS.ErrnoException).code === "ENOENT") {
+          reject(new Error("Antigravity CLI (agy) not found in PATH. Install: curl -fsSL https://antigravity.google/cli/install.sh | bash"));
+        } else reject(new Error(`agy spawn failed: ${sanitizeError(err)}`));
+      });
+    });
+
+    child.on("close", (code: number | null, sig: string | null) => {
+      done(() => {
+        const out = Buffer.concat(stdout).toString("utf8");
+        const err = Buffer.concat(stderr).toString("utf8");
+
+        if (sig === "SIGTERM" || sig === "SIGKILL" || code === null) {
+          reject(new Error(`agy was cancelled (${sig || "timeout"})`));
+        } else if (code !== 0) {
+          reject(new Error(`agy exited with code ${code}: ${sanitizeError((err || out).slice(0, 1000).trim() || "(no output)")}`));
+        } else {
+          // stdout only — stderr may carry warnings that would corrupt JSON envelope parsing
+          resolve(out);
+        }
+      });
+    });
+  });
+}
+
+// Extract .response from agy's JSON envelope, fall back to raw text. Pure — tested without mocking.
+export function parseAgyResponse(raw: string, maxChars: number): string {
+  let text = raw;
+  try {
+    const parsed = JSON.parse(raw);
+    if (typeof parsed.response === "string") text = parsed.response;
+    else if (parsed.response !== undefined) text = JSON.stringify(parsed.response);
+  } catch {
+    // not JSON — use raw
+  }
+  return text.slice(0, maxChars);
+}

package/extensions/lib/extract.ts

@@ -3,16 +3,18 @@
 //   "static"  → JSDOM+Readability (no external API)
 //   "dynamic" → Firecrawl Scrape (JS rendering/structured JSON)
 //   "full"    → Crawl4AI markdown endpoint
-//   "auto"    → static → dynamic → full
+//   "agy"     → agy (Gemini/Claude) native read_url — bot-protected/JS-heavy pages
+//   "auto"    → static → dynamic → full → agy
 
 import { cwdFromContext, includeProjectEnv } from "./config";
 import { loadFirecrawlConfig, loadCrawl4aiConfig, type FirecrawlConfig, type Crawl4aiConfig } from "./config";
 import { fetchReadableContent } from "./content";
 import { firecrawlRequest, type FirecrawlResult } from "./firecrawl";
 import { fetchCrawl4aiMarkdown } from "./crawl4ai";
+import { isAgyInstalled, extractViaAgy, parseAgyStructured } from "./agy";
 import { formatFirecrawlScrape, sanitizeError } from "./format";
 
-export type ExtractMode = "auto" | "static" | "dynamic" | "full";
+export type ExtractMode = "auto" | "static" | "dynamic" | "full" | "agy";
 export type ExtractAttemptStatus = "success" | "empty" | "error";
 
 export interface ExtractParams {
@@ -142,13 +144,49 @@ async function extractFull(params: ExtractParams, ctx?: Record<string, unknown>)
 function modesFor(params: ExtractParams): Array<Exclude<ExtractMode, "auto">> {
   const mode = params.mode ?? "auto";
   if (mode !== "auto") return [mode];
-  return ["static", "dynamic", "full"];
+  return ["static", "dynamic", "full", "agy"];
 }
 
 async function runExtractor(mode: Exclude<ExtractMode, "auto">, params: ExtractParams, ctx?: Record<string, unknown>): Promise<ExtractResult | null> {
   if (mode === "static") return extractStatic(params);
   if (mode === "dynamic") return extractDynamic(params, ctx);
-  return extractFull(params, ctx);
+  if (mode === "full") return extractFull(params, ctx);
+  return extractAgy(params);
+}
+
+async function extractAgy(params: ExtractParams): Promise<ExtractResult | null> {
+  if (!isAgyInstalled()) return null; // graceful skip — recorded as skipped attempt
+  const run = () => extractViaAgy({
+    url: params.url,
+    prompt: params.prompt,
+    schema: params.schema,
+    contentChars: params.content_chars,
+    signal: params.signal,
+  });
+  // ponytail: the model occasionally returns empty output (transient); one retry
+  // materially improves reliability at the cost of one extra spawn.
+  let markdown = await run();
+  if (!markdown.trim()) markdown = await run();
+  let structured: unknown;
+  // Match dynamic mode: prompt OR schema requests structured output.
+  if (params.prompt || params.schema !== undefined) {
+    structured = parseAgyStructured(markdown);
+    if (structured !== undefined) {
+      // Strip the JSON (fenced or bare) so the result is clean markdown; the
+      // structured payload is surfaced separately (rendered like dynamic mode).
+      const fenced = /```(?:json)?\s*[\s\S]*?```/.test(markdown);
+      let clean = fenced ? markdown.replace(/```(?:json)?\s*[\s\S]*?```/, "").trim() : "";
+      if (!fenced && markdown.trim().length > 0) {
+        // Bare JSON (no fence) — the whole output is JSON, so nothing remains.
+        clean = "";
+      }
+      // When the model returned only JSON, don't duplicate it as plain text body
+      // (it is already surfaced via structuredSection below).
+      const body = clean || "(Structured extraction only — see JSON below)";
+      return { title: "", markdown: body + structuredSection(structured), backend: "agy", structured };
+    }
+  }
+  return { title: "", markdown, backend: "agy" };
 }
 
 export async function extractWithDiagnostics(params: ExtractParams): Promise<ExtractDiagnostics> {
@@ -162,7 +200,8 @@ export async function extractWithDiagnostics(params: ExtractParams): Promise<Ext
       const result = await runExtractor(mode, params, ctx);
       if (isUseful(result, mode, explicit)) {
         if (!explicit && attempts.length > 0) {
-          result.markdown = `[Extraction fell back to ${mode === "dynamic" ? "Firecrawl Scrape (dynamic mode)" : "Crawl4AI (full browser mode)"}]\n\n${result.markdown}`;
+          const backendLabel = mode === "dynamic" ? "Firecrawl Scrape (dynamic mode)" : mode === "full" ? "Crawl4AI (full browser mode)" : "agy (model-backed browser)";
+          result.markdown = `[Extraction fell back to ${backendLabel}]\n\n${result.markdown}`;
         }
         attempts.push({ mode, backend: result.backend, status: "success", message: `Selected ${mode}`, contentLength: result.markdown.length });
         return { result, attempts, selectedMode: mode, fallbackUsed: attempts.length > 1 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bacnh85/pi-web",
-  "version": "0.5.7",
+  "version": "0.6.0",
   "description": "Pi extension for web search, page extraction, Firecrawl scraping/crawling, and Crawl4AI headless browser crawling.",
   "type": "module",
   "license": "MIT",

package/skills/pi-web/SKILL.md

@@ -5,14 +5,14 @@ description: Web search, content extraction, site crawling, and page capture via
 
 # pi-web — Unified Web Tools
 
-Use the **7 unified tools** from the `pi-web` extension for all web-related tasks. These tools automatically select the best backend from SearXNG, Brave Search, Firecrawl, and Crawl4AI — you don't need to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive queries and inline content prefer Brave.
+Use the **7 unified tools** from the `pi-web` extension for all web-related tasks. These tools automatically select the best backend from SearXNG, Brave Search, Firecrawl, Crawl4AI, and agy (when installed) — you don't need to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive queries and inline content prefer Brave.
 
 ## Quick Reference
 
 | Tool | Purpose | Auto-selection |
 |---|---|---|
 | `web_search` | Search the web for sources, docs, facts | SearXNG → Brave → Firecrawl |
-| `web_extract` | Extract readable content from a URL | Static (JSDOM) → Dynamic (Firecrawl) → Full (Crawl4AI) |
+| `web_extract` | Extract readable content from a URL | Static (JSDOM) → Dynamic (Firecrawl) → Full (Crawl4AI) → agy (model-backed) |
 | `web_map` | Discover URLs from a site | Firecrawl Map (only option) |
 | `web_crawl` | Crawl multiple pages from a site | Light (Firecrawl) or Full (Crawl4AI) |
 | `web_screenshot` | Capture page screenshot as PNG | Crawl4AI (only option) |
@@ -35,7 +35,8 @@ What do you need?
 │     ├─ static page (blog, docs): mode=static (fastest, no API key)
 │     ├─ dynamic page (JS-rendered): mode=dynamic
 │     ├─ JS-heavy SPA: mode=full
-│     └─ auto (default): tries static > dynamic > full
+│     ├─ bot-protected / blocked to scrapers: mode=agy (Gemini/Claude via agy)
+│     └─ auto (default): tries static > dynamic > full > agy
 │
 ├── Site URL discovery (find pages on a site)
 │   → web_map
@@ -78,6 +79,9 @@ What do you need?
    - ❌ Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls through Crawl4AI in `auto` mode.
    - ✅ Supported: prompt-based JSON extraction, schema-based structured extraction.
 3. **full** (Crawl4AI headless browser) — handles all content types. **Resource-intensive** (launches a full headless browser). Use only when static and dynamic modes fail, or when explicitly needed.
+4. **agy** (Gemini/Claude via agy CLI) — last-resort fallback. Uses agy's native `read_url` web tool with a model-driven browser, so it can fetch pages that block Firecrawl/Crawl4AI (bot protection, anti-AI scraping). Requires the `agy` CLI installed and authenticated. Also supports prompt/schema-based structured extraction.
+
+> **Requirement for agy mode**: install and authenticate the Antigravity CLI: `curl -fsSL https://antigravity.google/cli/install.sh | bash` then run `agy` once interactively. If agy is not installed, `auto` mode skips it silently; explicit `mode: "agy"` reports the install hint.
 
 ## Fallback Strategy
 
@@ -85,8 +89,9 @@ If one tool fails, try the next option in the chain:
 
 - **Search issues**: `web_search` auto-fallbacks and reports backend diagnostics. If all backends fail, configure at least one via env vars (check `web_status`).
 - **Extraction issues**: `web_extract` auto-fallbacks in `auto` mode. If all modes fail:
-  1. Try `web_screenshot` for a visual snapshot — may work when extraction is blocked.
-  2. The page may require interactive login, CAPTCHA, or be a non-HTML resource.
+  1. Try `mode: "agy"` explicitly — agy's model-backed browser often gets pages that block Firecrawl/Crawl4AI.
+  2. Try `web_screenshot` for a visual snapshot — may work when extraction is blocked.
+  3. The page may require interactive login, CAPTCHA, or be a non-HTML resource.
 - **Tool not found**: Ensure `pi-web` extension is installed (`pi install ./extensions/pi-web`).
 
 ## Cross-tool Decision Guide
@@ -95,6 +100,7 @@ If one tool fails, try the next option in the chain:
 |---|---|---|
 | A few specific pages from a site | `web_map` + `web_extract` on each URL | `web_crawl` (heavier than needed) |
 | Content from a JS-heavy page that fails in `auto` mode | `web_extract` with `mode: "full"` | Retrying `auto` mode repeatedly |
+| Content from a bot-protected page that blocks Firecrawl/Crawl4AI | `web_extract` with `mode: "agy"` | Retrying `web_extract` with all modes |
 | A visual of a bot-protected page | `web_screenshot` | Retrying `web_extract` with all modes |
 | Content alongside search results | `web_search` with `include_content: true` (auto prefers Brave) or `backend: "brave"` | Search snippets alone |
 | Printable/archivable page | `web_pdf` | Taking a screenshot and converting |
@@ -105,4 +111,5 @@ If one tool fails, try the next option in the chain:
 - **Always cite source URLs** when using web content in answers.
 - `web_status` shows which backends are configured without printing secrets. For Firecrawl, `apiKeyFound: false` is normal for self-hosted instances without auth — check the `ready` field to see if Firecrawl is actually usable.
 - The `backend` and `mode` parameters give explicit control when auto-selection is not desired.
+- `mode: "agy"` requires the `agy` CLI (Antigravity) installed and authenticated. `web_status` reports `agy.installed` so you can check availability without guessing.
 - Backend-specific config (API keys, URLs) comes from environment variables, not tool parameters. Use `web_status` to verify configuration.