@bacnh85/pi-web 0.5.7 → 0.6.1

package/CHANGELOG.md

@@ -1,13 +1,39 @@
 # Changelog
 
+## 0.6.2 (2026-08-30)
+
+### Changed
+
+- Trimmed static prompt overhead ~385 tokens/turn: compressed the injected
+  `Web Tool Routing` guidance block (1,247 -> 281 chars, same routing table
+  + backend rules), cut all 7 tools' promptGuidelines to <=2 unique lines,
+  and shortened web_crawl/web_screenshot schema descriptions. One
+  hook.test.ts assertion updated to the compressed phrasing. No tool,
+  parameter, or default changed.
+
+## 0.6.1 (2026-08-19)
+
+### Changed
+
+- `web_extract` agy backend default model updated to `gemini-3.7-flash-medium`
+  — the current Flash generation in agy 1.1.x (3.6 is still served, this just
+  follows the latest).
+
+## 0.6.0 (2026-08-07)
+
+### Features
+
+- **agy extraction backend:** `web_extract` gains a new `agy` mode that uses the Antigravity CLI (Gemini/Claude) native `read_url` web tool to fetch bot-protected and anti-AI-scraping pages that block Firecrawl/Crawl4AI. `auto` mode now falls back static → dynamic → full → agy; explicit `mode: "agy"` forces it. Structured extraction (`prompt`/`schema`) is supported. `web_status` reports `agy.installed`.
+- agy is optional and self-contained: if the CLI is not installed, `auto` mode skips it silently and existing flows are unchanged. Install: `curl -fsSL https://antigravity.google/cli/install.sh | bash`, then authenticate once with `agy`.
+
+All notable changes to `pi-web` will be documented in this file.
+
 ## 0.5.7 (2026-08-05)
 
 ### Improvements
 
 - Patch version bump for release sync and package documentation update.
 
-All notable changes to `pi-web` will be documented in this file.
-
 ## 0.5.6 (2026-08-01)
 
 ### Features

package/README.md

@@ -2,7 +2,7 @@
 
 Pi extension for **unified web search, content extraction, site crawling, and page capture**.
 
-Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, and Crawl4AI — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.
+Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, Crawl4AI, and agy (Gemini/Claude, when installed) — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.
 
 ## Install
 
@@ -79,12 +79,13 @@ Use `backend` parameter to force a specific backend when needed.
 
 ### `web_extract` — Unified content extraction
 
-Extracts readable content from a URL. Auto-selects backend: static (JSDOM) → dynamic (Firecrawl) → full (Crawl4AI), with extraction diagnostics showing fallback attempts.
+Extracts readable content from a URL. Auto-selects backend: static (JSDOM) → dynamic (Firecrawl) → full (Crawl4AI) → agy (model-backed), with extraction diagnostics showing fallback attempts.
 
 ```
 web_extract url="https://docs.ansible.com/..."
 web_extract url="https://riven.tv/" mode=static
 web_extract url="https://example.com" mode=dynamic prompt="Extract pricing plans"
+web_extract url="https://blocked.example.com" mode=agy
 ```
 
 Parameters:
@@ -92,9 +93,9 @@ Parameters:
 | Parameter | Type | Default | Description |
 |---|---|---|---|
 | `url` | string | — | URL to extract |
-| `mode` | string | `auto` | `auto`, `static`, `dynamic`, or `full` |
-| `prompt` | string | — | Prompt for JSON extraction (dynamic mode) |
-| `schema` | any | — | JSON schema for structured extraction (dynamic mode) |
+| `mode` | string | `auto` | `auto`, `static`, `dynamic`, `full`, or `agy` |
+| `prompt` | string | — | Prompt for JSON extraction (dynamic/agy modes) |
+| `schema` | any | — | JSON schema for structured extraction (dynamic/agy modes) |
 | `content_chars` | number | 20000 | Max content chars |
 | `wait_for` | number | — | Milliseconds to wait for Firecrawl dynamic rendering. Crawl4AI `/md` full mode may ignore this. |
 | `mobile` | boolean | false | Emulate mobile viewport (dynamic mode) |
@@ -106,11 +107,14 @@ Parameters:
 | `static` | JSDOM+Readability | Simple static pages, blog posts, docs | No |
 | `dynamic` | Firecrawl Scrape | JS-rendered pages, dynamic content | Maybe |
 | `full` | Crawl4AI | JS-heavy SPA, complex rendering | Maybe |
-| `auto` (default) | static → dynamic → full | Unknown page type | Maybe |
+| `agy` | agy (Gemini/Claude) | Bot-protected / anti-AI-scraping pages | agy CLI installed |
+| `auto` (default) | static → dynamic → full → agy | Unknown page type | Maybe |
 
 In `auto` mode, fallbacks are noted in the output (e.g., `[Extraction fell back to Firecrawl Scrape (dynamic mode)]`). If `static` extraction fails, the tool gracefully escalates to heavier backends.
 
-> ⚠️ **Note on Firecrawl Scrape**: Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls back to `full` mode (Crawl4AI) in `auto` mode.
+> ⚠️ **Note on Firecrawl Scrape**: Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls back to `full` mode (Crawl4AI) in `auto` mode, and to `agy` mode as a last resort.
+
+> **`agy` mode (optional)**: Uses the [Antigravity CLI](https://antigravity.google/) with Gemini/Claude — its native `read_url` browser tool can fetch pages that block Firecrawl/Crawl4AI. Install with `curl -fsSL https://antigravity.google/cli/install.sh | bash`, authenticate once with `agy`, then `auto` mode falls back to it automatically. If agy is not installed, `auto` mode skips it silently; `web_status` reports `agy.installed`.
 
 ### `web_map` — Site URL discovery
 
@@ -172,7 +176,8 @@ Typical output:
     "baseUrl": "http://172.30.55.22:11235",
     ...
     "health": { "status": "healthy", "version": "0.5.0", ... }
-  }
+  },
+  "agy": { "installed": true }
 }
 ```
 
@@ -188,6 +193,7 @@ Typical output:
 | `lib/searxng.ts` | SearXNG metasearch fetch client (internal) |
 | `lib/firecrawl.ts` | Firecrawl API fetch client with v2→v1 fallback (internal) |
 | `lib/crawl4ai.ts` | Crawl4AI Docker API fetch client (internal) |
+| `lib/agy.ts` | agy (Antigravity CLI) spawn helper — `read_url` extraction via Gemini/Claude |
 | `lib/search.ts` | Unified search orchestrator — probes backends, fallback chain |
 | `lib/extract.ts` | Unified extraction orchestrator — mode-based backend selection |
 

package/extensions/index.ts

@@ -57,25 +57,14 @@ const crawl4aiControlSchema = {
 // the guidance travels with the package and disappears when pi-web is absent.
 const WEB_ROUTING_GUIDANCE = `## Web Tool Routing (pi-web)
 
-The pi-web extension provides 7 unified tools that auto-select the best backend:
+- **web_search** — web search (auto: SearXNG → Brave → Firecrawl; force via \`backend\`, tune via \`engines\`).
+- **web_extract** — URL → markdown (auto: static JSDOM → dynamic Firecrawl → full Crawl4AI → agy; force via \`mode\`; prompt+schema for JSON extraction).
+- **web_map** — discover site URLs (Firecrawl Map).
+- **web_crawl** — multi-page crawl: \`mode: "light"\` (Firecrawl, url) or \`mode: "full"\` (Crawl4AI, urls[]).
+- **web_screenshot** / **web_pdf** — page capture (Crawl4AI).
+- **web_status** — provider config + health.
 
-- **\`web_search\`** — Search the web (auto: SearXNG → Brave → Firecrawl). Use
-  \`backend\` for explicit control, \`engines\` for SearXNG tuning.
-- **\`web_extract\`** — Extract readable content from a URL (auto: static JSDOM
-  → dynamic Firecrawl → full Crawl4AI). Use \`mode\` for explicit control.
-- **\`web_map\`** — Discover URLs from a site (Firecrawl Map).
-- **\`web_crawl\`** — Crawl multiple pages. \`mode: "light"\` (Firecrawl) or
-  \`mode: "full"\` (Crawl4AI).
-- **\`web_screenshot\`** / **\`web_pdf\`** — Visual/page capture (Crawl4AI).
-- **\`web_status\`** — Check provider configuration and server health.
-
-Backend selection rules:
-
-- Firecrawl Search has poor semantic accuracy on domain-specific queries; prefer
-  SearXNG or Brave for precision (force via \`backend\`).
-- Firecrawl Scrape fails on bot-protected sites (e.g. Ansible docs); Crawl4AI
-  handles those (force via \`mode: "full"\`).
-- Always cite source URLs when web results materially support an answer.`;
+Rules: Firecrawl Search is weak on domain-specific queries — prefer SearXNG/Brave; Firecrawl Scrape fails on bot-protected sites — use Crawl4AI (\`mode: "full"\`) then agy (\`mode: "agy"\`); cite source URLs.`;
 
 
 // ---------------------------------------------------------------------------
@@ -90,13 +79,7 @@ export default function piWebExtension(pi: ExtensionAPI) {
     description:
       "Search the web. Auto-selects backends: SearXNG, Brave, Firecrawl.",
     promptSnippet: "Search current web results",
-    promptGuidelines: [
-      "Source discovery, docs, facts, and general search.",
-      "Broad discovery uses SearXNG; precision/site/docs use Brave; Firecrawl is fallback.",
-      "Force backend via backend:'brave'|'searxng' for poor auto results.",
-      "Use engines='google,github' for SearXNG tuning.",
-      "Cite source URLs.",
-    ],
+    promptGuidelines: ["Source discovery, docs, facts. Precision/site/docs → Brave via backend:'brave'; tune SearXNG via engines.", "Cite source URLs."],
     parameters: Type.Object({
       query: Type.String(),
       count: Type.Optional(Type.Number({ default: 5 })),
@@ -136,23 +119,17 @@ export default function piWebExtension(pi: ExtensionAPI) {
     name: "web_extract",
     label: "Web Content Extraction",
     description:
-      "Extract readable content from a URL. Auto mode: static\u2192dynamic\u2192full.",
+      "Extract readable content from a URL. Auto mode: static\u2192dynamic\u2192full\u2192agy.",
     promptSnippet: "Extract readable webpage content as markdown",
-    promptGuidelines: [
-      "Clean markdown from a known URL.",
-      "mode: 'static' (no API key, JSDOM), 'dynamic' (Firecrawl JS), 'full' (Crawl4AI).",
-      "'auto' tries static\u2192dynamic\u2192full; see diagnostics for fallback chain.",
-      "Use prompt+schema for structured JSON extraction (dynamic mode only).",
-      "Cite the source URL.",
-    ],
+    promptGuidelines: ["Markdown from a known URL; prompt+schema for structured JSON extraction.", "Cite the source URL."],
     parameters: Type.Object({
       url: Type.String(),
       mode: Type.Optional(Type.Union(
-        [Type.Literal("auto"), Type.Literal("static"), Type.Literal("dynamic"), Type.Literal("full")],
-        { default: "auto", description: "auto, static, dynamic, full." },
+        [Type.Literal("auto"), Type.Literal("static"), Type.Literal("dynamic"), Type.Literal("full"), Type.Literal("agy")],
+        { default: "auto", description: "auto, static, dynamic, full, agy." },
       )),
-      prompt: Type.Optional(Type.String({ description: "Prompt for structured JSON extraction (dynamic mode only)." })),
-      schema: Type.Optional(Type.Any({ description: "JSON schema for structured extraction (dynamic mode only)." })),
+      prompt: Type.Optional(Type.String({ description: "Prompt for structured JSON extraction (dynamic/agy modes)." })),
+      schema: Type.Optional(Type.Any({ description: "JSON schema for structured extraction (dynamic/agy modes)." })),
       content_chars: Type.Optional(Type.Number({ default: 20000 })),
       wait_for: Type.Optional(Type.Number({ description: "Ms to wait for Firecrawl render before extraction." })),
       mobile: Type.Optional(Type.Boolean({ default: false, description: "Mobile viewport (dynamic mode only)." })),
@@ -188,11 +165,7 @@ export default function piWebExtension(pi: ExtensionAPI) {
     description:
       "Discover site URLs via Firecrawl Map.",
     promptSnippet: "Map site URLs",
-    promptGuidelines: [
-      "Discover site URLs before crawling. Prefer web_extract for small jobs.",
-      "Best on base domains. sitemap:'only' for sub-path discovery.",
-      "Keep limits small unless broad discovery is requested.",
-    ],
+    promptGuidelines: ["URL discovery before crawling; prefer web_extract for small jobs."],
     parameters: Type.Object({
       url: Type.String(),
       limit: Type.Optional(Type.Number({ default: 100 })),
@@ -231,21 +204,17 @@ export default function piWebExtension(pi: ExtensionAPI) {
     description:
       "Crawl pages. Firecrawl 'light' or Crawl4AI 'full' headless mode.",
     promptSnippet: "Crawl a small site section",
-    promptGuidelines: [
-      "Prefer web_map + web_extract over crawl for small jobs.",
-      "'light'=Firecrawl (url param), 'full'=Crawl4AI (urls[] param).",
-      "Keep limit low (default 10).",
-    ],
+    promptGuidelines: ["'light'=Firecrawl (url), 'full'=Crawl4AI (urls[]). Prefer web_map + web_extract for small jobs."],
     parameters: Type.Object({
-      url: Type.Optional(Type.String({ description: "URL for Firecrawl-style crawl (mode:'light')." })),
-      urls: Type.Optional(Type.Array(Type.String(), { description: "URLs for Crawl4AI-style crawl (mode:'full'), up to 100." })),
+      url: Type.Optional(Type.String({ description: "URL for mode:'light' (Firecrawl)." })),
+      urls: Type.Optional(Type.Array(Type.String(), { description: "URLs for mode:'full' (Crawl4AI), up to 100." })),
       mode: Type.Optional(Type.Union([Type.Literal("light"), Type.Literal("full")], { default: "light", description: "'light'(Firecrawl) or 'full'(Crawl4AI)." })),
       limit: Type.Optional(Type.Number({ default: 10 })),
-      include_paths: Type.Optional(Type.String({ description: "Comma-separated paths to include (Firecrawl mode)." })),
-      exclude_paths: Type.Optional(Type.String({ description: "Comma-separated paths to exclude (Firecrawl mode)." })),
-      poll: Type.Optional(Type.Boolean({ default: false, description: "Poll for completion (Firecrawl mode)." })),
-      browser_config: Type.Optional(Type.Any({ description: "BrowserConfig JSON (full mode only)." })),
-      crawler_config: Type.Optional(Type.Any({ description: "CrawlerRunConfig JSON (full mode only)." })),
+      include_paths: Type.Optional(Type.String({ description: "Comma-separated include paths (light mode)." })),
+      exclude_paths: Type.Optional(Type.String({ description: "Comma-separated exclude paths (light mode)." })),
+      poll: Type.Optional(Type.Boolean({ default: false, description: "Poll until completion (light mode)." })),
+      browser_config: Type.Optional(Type.Any({ description: "BrowserConfig JSON (full mode)." })),
+      crawler_config: Type.Optional(Type.Any({ description: "CrawlerRunConfig JSON (full mode)." })),
       content_chars: Type.Optional(Type.Number({ default: 20000 })),
       ...firecrawlControlSchema,
       ...crawl4aiControlSchema,
@@ -319,14 +288,11 @@ export default function piWebExtension(pi: ExtensionAPI) {
     description:
       "Full-page PNG screenshot via Crawl4AI.",
     promptSnippet: "Screenshot a webpage",
-    promptGuidelines: [
-      "Use when web_extract fails on JS-heavy or bot-protected pages.",
-      "wait_for (default 2s) delays capture for dynamic content.",
-    ],
+    promptGuidelines: ["Full-page PNG; use when web_extract fails on JS-heavy pages."],
     parameters: Type.Object({
       url: Type.String(),
       wait_for: Type.Optional(Type.Number({ default: 2, description: "Seconds to wait before capture." })),
-      wait_for_images: Type.Optional(Type.Boolean({ default: false, description: "Wait for images before capture." })),
+      wait_for_images: Type.Optional(Type.Boolean({ default: false })),
       ...crawl4aiControlSchema,
       ...sharedControlSchema,
     }),
@@ -359,10 +325,7 @@ export default function piWebExtension(pi: ExtensionAPI) {
     description:
       "PDF document via Crawl4AI.",
     promptSnippet: "PDF a webpage",
-    promptGuidelines: [
-      "Printable or archivable page snapshot.",
-      "Returns base64 PDF string.",
-    ],
+    promptGuidelines: ["Printable/archivable page snapshot; returns base64 PDF."],
     parameters: Type.Object({
       url: Type.String(),
       ...crawl4aiControlSchema,
@@ -389,11 +352,7 @@ export default function piWebExtension(pi: ExtensionAPI) {
     description:
       "Show web provider config status without printing secrets.",
     promptSnippet: "Check web provider config and server status",
-    promptGuidelines: [
-      "Check which backends are configured and their server status.",
-      "Never prints secrets — reports only presence and source.",
-      "apiKeyFound:false is normal for self-hosted Firecrawl; check ready field.",
-    ],
+    promptGuidelines: ["Reports backend presence/health; never prints secrets."],
     parameters: Type.Object({}),
     async execute(_id: string, _params: Record<string, unknown>, signal: AbortSignal, _onUpdate: unknown, ctx: any) {
       const cwd = cwdFromContext(ctx);
@@ -407,6 +366,8 @@ export default function piWebExtension(pi: ExtensionAPI) {
       const c4aiUrl = findEnvValue("CRAWL4AI_API_URL", cwd, trusted);
       const c4aiToken = findEnvValue("CRAWL4AI_API_TOKEN", cwd, trusted);
 
+      const { isAgyInstalled } = await import("./lib/agy");
+
       const fcBaseUrl = normalizeFirecrawlBaseUrl(fireUrl.value);
       const fcHosted = !fireUrl.value || fcBaseUrl.startsWith(HOSTED_FIRECRAWL_BASE_URL);
 
@@ -427,6 +388,7 @@ export default function piWebExtension(pi: ExtensionAPI) {
           apiTokenFound: Boolean(c4aiToken.value),
           apiTokenSource: c4aiToken.value ? c4aiToken.source : "not set",
         },
+        agy: { installed: isAgyInstalled() },
       };
 
       // Crawl4AI health check

package/extensions/lib/agy.ts

@@ -0,0 +1,201 @@
+// agy-based web extraction — uses agy's native read_url tool via Gemini/Claude.
+// Self-contained: does NOT import from pi-agy (different concern, no coupling).
+//
+// Runs agy in --mode plan (read-only: no file writes). Verified (agy 1.1.11):
+// plan mode auto-approves read_url in headless -p WITHOUT
+// --dangerously-skip-permissions, so no broad permission bypass is needed.
+
+import { createRequire } from "node:module";
+
+import { sanitizeError } from "./format";
+
+// ponytail: require (not ESM import) so tests can patch cp.spawn/spawnSync
+// without mocking the module graph — same pattern as pi-agy.
+const _require = createRequire(import.meta.url);
+const cp = _require("node:child_process") as typeof import("node:child_process");
+
+const AGY_FETCH_TIMEOUT_MS = 90_000; // agy needs time for model call + web fetch
+const AGY_PROBE_TIMEOUT_MS = 5_000;
+const AGY_MAX_OUTPUT_BYTES = 200_000; // bound output to protect Pi context
+export const AGY_MODEL = "gemini-3.7-flash-medium"; // ponytail: fixed default; users needing model control use agy_execute
+
+// Cache install status with a TTL — spawnSync blocks the event loop up to
+// AGY_PROBE_TIMEOUT_MS, and web_status/extract can call this repeatedly.
+let agyInstalledCache: { ok: boolean; at: number } | null = null;
+const AGY_INSTALL_CACHE_TTL_MS = 60_000;
+
+export function isAgyInstalled(): boolean {
+  if (agyInstalledCache && Date.now() - agyInstalledCache.at < AGY_INSTALL_CACHE_TTL_MS) {
+    return agyInstalledCache.ok;
+  }
+  // ponytail: spawnSync is the simplest reliable probe; result is cached so the
+  // event-loop block happens at most once per 60s.
+  try {
+    const r = cp.spawnSync("agy", ["--version"], { timeout: AGY_PROBE_TIMEOUT_MS, stdio: "ignore" });
+    agyInstalledCache = { ok: r.status === 0, at: Date.now() };
+    return agyInstalledCache.ok;
+  } catch {
+    agyInstalledCache = { ok: false, at: Date.now() };
+    return false;
+  }
+}
+
+// Test-only: clear the cached install status.
+export function resetAgyInstalledCache(): void {
+  agyInstalledCache = null;
+}
+
+// Validate + sanitize a URL before it is interpolated into the agy model prompt.
+// Prompt-injection guard: reject non-http(s) schemes and strip control chars/newlines
+// so a crafted URL cannot break out of the "fetch this URL" framing.
+export function sanitizeAgyUrl(url: string): string {
+  const trimmed = url.trim();
+  let parsed: URL;
+  try {
+    parsed = new URL(trimmed);
+  } catch {
+    throw new Error(`Invalid URL for agy extraction: ${trimmed.slice(0, 200)}`);
+  }
+  if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
+    throw new Error(`Unsupported URL scheme for agy extraction: ${parsed.protocol}`);
+  }
+  // Neutralize any control characters/newlines so the URL stays on one line
+  // inside the prompt and cannot smuggle instructions.
+  return trimmed.replace(/[\x00-\x1f\x7f]/g, "");
+}
+
+// Build the agy CLI args for a web-fetch task. Pure — tested without mocking.
+export function buildAgyFetchArgs(url: string, prompt?: string, schema?: unknown): string[] {
+  const safeUrl = sanitizeAgyUrl(url);
+  const structured = prompt || schema !== undefined;
+  const jsonInstruction = schema !== undefined
+    ? `\nReturn as JSON matching this schema: ${JSON.stringify(schema)}`
+    : "\nReturn the result as JSON.";
+  const fetchInstruction = structured
+    ? (prompt ? `Then extract this information: ${prompt}` : "Then extract the requested fields.") + jsonInstruction
+    : "Return the full page content as clean markdown.";
+
+  const agyPrompt = `Use your read_url web tool to fetch this URL: ${safeUrl}\n\n${fetchInstruction}\n\nReturn ONLY the result, no commentary.`;
+
+  return [
+    "--model",
+    AGY_MODEL,
+    "--mode",
+    "plan", // read-only: no file writes
+    "--print-timeout",
+    "90s",
+    // Verified (agy 1.1.11): plan mode auto-approves read_url in headless -p
+    // WITHOUT --dangerously-skip-permissions — pi-agy's flag is only needed for
+    // write modes (accept-edits/sandbox). Omitting it keeps the auto-approval
+    // surface at read-only web tools only.
+    "--output-format",
+    "json", // structured response for clean parsing
+    "-p",
+    agyPrompt,
+  ];
+}
+
+export async function extractViaAgy(params: {
+  url: string;
+  prompt?: string; // structured extraction prompt
+  schema?: unknown; // JSON schema for structured extraction
+  contentChars?: number;
+  signal?: AbortSignal;
+}): Promise<string> {
+  const { url, prompt, schema, contentChars, signal } = params;
+  const output = await spawnAgyRaw(buildAgyFetchArgs(url, prompt, schema), signal);
+  return parseAgyResponse(output, contentChars ?? 20000);
+}
+
+// Parse structured JSON out of agy's model output — used when a prompt/schema was
+// requested so the structured result is first-class, matching dynamic mode's
+// `structured` field. Handles both fenced (```json ... ```) and bare JSON, which
+// the model produces nondeterministically. Pure — tested without mocking.
+export function parseAgyStructured(markdown: string): unknown {
+  const trimmed = markdown.trim();
+  if (!trimmed) return undefined;
+  // Fenced block first, then bare JSON object/array.
+  const m = trimmed.match(/^```(?:json)?\s*([\s\S]*?)```/);
+  const candidate = m ? m[1] : trimmed.startsWith("{") || trimmed.startsWith("[") ? trimmed : undefined;
+  if (candidate === undefined) return undefined;
+  try {
+    return JSON.parse(candidate);
+  } catch {
+    return undefined;
+  }
+}
+
+// Spawn agy with bounded output collection + proper error handling.
+async function spawnAgyRaw(args: string[], signal?: AbortSignal): Promise<string> {
+  // ponytail: agy is a Go binary in PATH (official installer); no shell wrapper.
+  const child = cp.spawn("agy", args, {
+    stdio: ["ignore", "pipe", "pipe"],
+    signal,
+    timeout: AGY_FETCH_TIMEOUT_MS + 5_000, // let agy report its own print timeout first
+  });
+
+  const stdout: Buffer[] = [];
+  const stderr: Buffer[] = [];
+  let stdoutBytes = 0;
+  let stderrBytes = 0;
+
+  child.stdout.on("data", (d: Buffer) => {
+    // Capture all bytes up to the cap, including the partial final chunk
+    // (mirrors pi-agy's appendBounded behavior).
+    if (stdoutBytes < AGY_MAX_OUTPUT_BYTES) {
+      stdout.push(d.subarray(0, AGY_MAX_OUTPUT_BYTES - stdoutBytes));
+    }
+    stdoutBytes += d.length;
+  });
+  child.stderr.on("data", (d: Buffer) => {
+    stderrBytes += d.length;
+    if (stderrBytes <= 16 * 1024) stderr.push(d);
+  });
+
+  return new Promise<string>((resolve, reject) => {
+    let settled = false;
+    const done = (fn: () => void) => {
+      if (settled) return;
+      settled = true;
+      fn();
+    };
+
+    child.on("error", (err: Error) => {
+      done(() => {
+        if (signal?.aborted) reject(new Error("agy was cancelled"));
+        else if ((err as NodeJS.ErrnoException).code === "ENOENT") {
+          reject(new Error("Antigravity CLI (agy) not found in PATH. Install: curl -fsSL https://antigravity.google/cli/install.sh | bash"));
+        } else reject(new Error(`agy spawn failed: ${sanitizeError(err)}`));
+      });
+    });
+
+    child.on("close", (code: number | null, sig: string | null) => {
+      done(() => {
+        const out = Buffer.concat(stdout).toString("utf8");
+        const err = Buffer.concat(stderr).toString("utf8");
+
+        if (sig === "SIGTERM" || sig === "SIGKILL" || code === null) {
+          reject(new Error(`agy was cancelled (${sig || "timeout"})`));
+        } else if (code !== 0) {
+          reject(new Error(`agy exited with code ${code}: ${sanitizeError((err || out).slice(0, 1000).trim() || "(no output)")}`));
+        } else {
+          // stdout only — stderr may carry warnings that would corrupt JSON envelope parsing
+          resolve(out);
+        }
+      });
+    });
+  });
+}
+
+// Extract .response from agy's JSON envelope, fall back to raw text. Pure — tested without mocking.
+export function parseAgyResponse(raw: string, maxChars: number): string {
+  let text = raw;
+  try {
+    const parsed = JSON.parse(raw);
+    if (typeof parsed.response === "string") text = parsed.response;
+    else if (parsed.response !== undefined) text = JSON.stringify(parsed.response);
+  } catch {
+    // not JSON — use raw
+  }
+  return text.slice(0, maxChars);
+}

package/extensions/lib/extract.ts

@@ -3,16 +3,18 @@
 //   "static"  → JSDOM+Readability (no external API)
 //   "dynamic" → Firecrawl Scrape (JS rendering/structured JSON)
 //   "full"    → Crawl4AI markdown endpoint
-//   "auto"    → static → dynamic → full
+//   "agy"     → agy (Gemini/Claude) native read_url — bot-protected/JS-heavy pages
+//   "auto"    → static → dynamic → full → agy
 
 import { cwdFromContext, includeProjectEnv } from "./config";
 import { loadFirecrawlConfig, loadCrawl4aiConfig, type FirecrawlConfig, type Crawl4aiConfig } from "./config";
 import { fetchReadableContent } from "./content";
 import { firecrawlRequest, type FirecrawlResult } from "./firecrawl";
 import { fetchCrawl4aiMarkdown } from "./crawl4ai";
+import { isAgyInstalled, extractViaAgy, parseAgyStructured } from "./agy";
 import { formatFirecrawlScrape, sanitizeError } from "./format";
 
-export type ExtractMode = "auto" | "static" | "dynamic" | "full";
+export type ExtractMode = "auto" | "static" | "dynamic" | "full" | "agy";
 export type ExtractAttemptStatus = "success" | "empty" | "error";
 
 export interface ExtractParams {
@@ -142,13 +144,49 @@ async function extractFull(params: ExtractParams, ctx?: Record<string, unknown>)
 function modesFor(params: ExtractParams): Array<Exclude<ExtractMode, "auto">> {
   const mode = params.mode ?? "auto";
   if (mode !== "auto") return [mode];
-  return ["static", "dynamic", "full"];
+  return ["static", "dynamic", "full", "agy"];
 }
 
 async function runExtractor(mode: Exclude<ExtractMode, "auto">, params: ExtractParams, ctx?: Record<string, unknown>): Promise<ExtractResult | null> {
   if (mode === "static") return extractStatic(params);
   if (mode === "dynamic") return extractDynamic(params, ctx);
-  return extractFull(params, ctx);
+  if (mode === "full") return extractFull(params, ctx);
+  return extractAgy(params);
+}
+
+async function extractAgy(params: ExtractParams): Promise<ExtractResult | null> {
+  if (!isAgyInstalled()) return null; // graceful skip — recorded as skipped attempt
+  const run = () => extractViaAgy({
+    url: params.url,
+    prompt: params.prompt,
+    schema: params.schema,
+    contentChars: params.content_chars,
+    signal: params.signal,
+  });
+  // ponytail: the model occasionally returns empty output (transient); one retry
+  // materially improves reliability at the cost of one extra spawn.
+  let markdown = await run();
+  if (!markdown.trim()) markdown = await run();
+  let structured: unknown;
+  // Match dynamic mode: prompt OR schema requests structured output.
+  if (params.prompt || params.schema !== undefined) {
+    structured = parseAgyStructured(markdown);
+    if (structured !== undefined) {
+      // Strip the JSON (fenced or bare) so the result is clean markdown; the
+      // structured payload is surfaced separately (rendered like dynamic mode).
+      const fenced = /```(?:json)?\s*[\s\S]*?```/.test(markdown);
+      let clean = fenced ? markdown.replace(/```(?:json)?\s*[\s\S]*?```/, "").trim() : "";
+      if (!fenced && markdown.trim().length > 0) {
+        // Bare JSON (no fence) — the whole output is JSON, so nothing remains.
+        clean = "";
+      }
+      // When the model returned only JSON, don't duplicate it as plain text body
+      // (it is already surfaced via structuredSection below).
+      const body = clean || "(Structured extraction only — see JSON below)";
+      return { title: "", markdown: body + structuredSection(structured), backend: "agy", structured };
+    }
+  }
+  return { title: "", markdown, backend: "agy" };
 }
 
 export async function extractWithDiagnostics(params: ExtractParams): Promise<ExtractDiagnostics> {
@@ -162,7 +200,8 @@ export async function extractWithDiagnostics(params: ExtractParams): Promise<Ext
       const result = await runExtractor(mode, params, ctx);
       if (isUseful(result, mode, explicit)) {
         if (!explicit && attempts.length > 0) {
-          result.markdown = `[Extraction fell back to ${mode === "dynamic" ? "Firecrawl Scrape (dynamic mode)" : "Crawl4AI (full browser mode)"}]\n\n${result.markdown}`;
+          const backendLabel = mode === "dynamic" ? "Firecrawl Scrape (dynamic mode)" : mode === "full" ? "Crawl4AI (full browser mode)" : "agy (model-backed browser)";
+          result.markdown = `[Extraction fell back to ${backendLabel}]\n\n${result.markdown}`;
         }
         attempts.push({ mode, backend: result.backend, status: "success", message: `Selected ${mode}`, contentLength: result.markdown.length });
         return { result, attempts, selectedMode: mode, fallbackUsed: attempts.length > 1 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bacnh85/pi-web",
-  "version": "0.5.7",
+  "version": "0.6.1",
   "description": "Pi extension for web search, page extraction, Firecrawl scraping/crawling, and Crawl4AI headless browser crawling.",
   "type": "module",
   "license": "MIT",
@@ -59,7 +59,13 @@
     "@types/node": "^26.0.1",
     "@types/turndown": "^5.0.6",
     "chai": "^6.2.2",
-    "mocha": "^11.7.6",
+    "mocha": "^11.8.0",
     "tsx": "^4.22.4"
+  },
+  "overrides": {
+    "serialize-javascript@>=5.0.0 <7.0.5": "^7.0.5",
+    "js-yaml@>=4.0.0 <4.3.1": "^4.3.1",
+    "brace-expansion@>=2.0.0 <2.1.4": "^2.1.4",
+    "diff": "^8.0.3"
   }
 }

package/skills/pi-web/SKILL.md

@@ -5,14 +5,14 @@ description: Web search, content extraction, site crawling, and page capture via
 
 # pi-web — Unified Web Tools
 
-Use the **7 unified tools** from the `pi-web` extension for all web-related tasks. These tools automatically select the best backend from SearXNG, Brave Search, Firecrawl, and Crawl4AI — you don't need to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive queries and inline content prefer Brave.
+Use the **7 unified tools** from the `pi-web` extension for all web-related tasks. These tools automatically select the best backend from SearXNG, Brave Search, Firecrawl, Crawl4AI, and agy (when installed) — you don't need to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive queries and inline content prefer Brave.
 
 ## Quick Reference
 
 | Tool | Purpose | Auto-selection |
 |---|---|---|
 | `web_search` | Search the web for sources, docs, facts | SearXNG → Brave → Firecrawl |
-| `web_extract` | Extract readable content from a URL | Static (JSDOM) → Dynamic (Firecrawl) → Full (Crawl4AI) |
+| `web_extract` | Extract readable content from a URL | Static (JSDOM) → Dynamic (Firecrawl) → Full (Crawl4AI) → agy (model-backed) |
 | `web_map` | Discover URLs from a site | Firecrawl Map (only option) |
 | `web_crawl` | Crawl multiple pages from a site | Light (Firecrawl) or Full (Crawl4AI) |
 | `web_screenshot` | Capture page screenshot as PNG | Crawl4AI (only option) |
@@ -35,7 +35,8 @@ What do you need?
 │     ├─ static page (blog, docs): mode=static (fastest, no API key)
 │     ├─ dynamic page (JS-rendered): mode=dynamic
 │     ├─ JS-heavy SPA: mode=full
-│     └─ auto (default): tries static > dynamic > full
+│     ├─ bot-protected / blocked to scrapers: mode=agy (Gemini/Claude via agy)
+│     └─ auto (default): tries static > dynamic > full > agy
 │
 ├── Site URL discovery (find pages on a site)
 │   → web_map
@@ -78,6 +79,9 @@ What do you need?
    - ❌ Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls through Crawl4AI in `auto` mode.
    - ✅ Supported: prompt-based JSON extraction, schema-based structured extraction.
 3. **full** (Crawl4AI headless browser) — handles all content types. **Resource-intensive** (launches a full headless browser). Use only when static and dynamic modes fail, or when explicitly needed.
+4. **agy** (Gemini/Claude via agy CLI) — last-resort fallback. Uses agy's native `read_url` web tool with a model-driven browser, so it can fetch pages that block Firecrawl/Crawl4AI (bot protection, anti-AI scraping). Requires the `agy` CLI installed and authenticated. Also supports prompt/schema-based structured extraction.
+
+> **Requirement for agy mode**: install and authenticate the Antigravity CLI: `curl -fsSL https://antigravity.google/cli/install.sh | bash` then run `agy` once interactively. If agy is not installed, `auto` mode skips it silently; explicit `mode: "agy"` reports the install hint.
 
 ## Fallback Strategy
 
@@ -85,8 +89,9 @@ If one tool fails, try the next option in the chain:
 
 - **Search issues**: `web_search` auto-fallbacks and reports backend diagnostics. If all backends fail, configure at least one via env vars (check `web_status`).
 - **Extraction issues**: `web_extract` auto-fallbacks in `auto` mode. If all modes fail:
-  1. Try `web_screenshot` for a visual snapshot — may work when extraction is blocked.
-  2. The page may require interactive login, CAPTCHA, or be a non-HTML resource.
+  1. Try `mode: "agy"` explicitly — agy's model-backed browser often gets pages that block Firecrawl/Crawl4AI.
+  2. Try `web_screenshot` for a visual snapshot — may work when extraction is blocked.
+  3. The page may require interactive login, CAPTCHA, or be a non-HTML resource.
 - **Tool not found**: Ensure `pi-web` extension is installed (`pi install ./extensions/pi-web`).
 
 ## Cross-tool Decision Guide
@@ -95,6 +100,7 @@ If one tool fails, try the next option in the chain:
 |---|---|---|
 | A few specific pages from a site | `web_map` + `web_extract` on each URL | `web_crawl` (heavier than needed) |
 | Content from a JS-heavy page that fails in `auto` mode | `web_extract` with `mode: "full"` | Retrying `auto` mode repeatedly |
+| Content from a bot-protected page that blocks Firecrawl/Crawl4AI | `web_extract` with `mode: "agy"` | Retrying `web_extract` with all modes |
 | A visual of a bot-protected page | `web_screenshot` | Retrying `web_extract` with all modes |
 | Content alongside search results | `web_search` with `include_content: true` (auto prefers Brave) or `backend: "brave"` | Search snippets alone |
 | Printable/archivable page | `web_pdf` | Taking a screenshot and converting |
@@ -105,4 +111,5 @@ If one tool fails, try the next option in the chain:
 - **Always cite source URLs** when using web content in answers.
 - `web_status` shows which backends are configured without printing secrets. For Firecrawl, `apiKeyFound: false` is normal for self-hosted instances without auth — check the `ready` field to see if Firecrawl is actually usable.
 - The `backend` and `mode` parameters give explicit control when auto-selection is not desired.
+- `mode: "agy"` requires the `agy` CLI (Antigravity) installed and authenticated. `web_status` reports `agy.installed` so you can check availability without guessing.
 - Backend-specific config (API keys, URLs) comes from environment variables, not tool parameters. Use `web_status` to verify configuration.