@ogulcancelik/pi-web-browse 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Can Celik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,73 @@
+# pi-web-browse
+
+Web search and content extraction skill for [pi](https://github.com/badlogic/pi-mono). Search the web and fetch pages via a real headless browser (CDP).
+
+**Works on Linux, macOS, and Windows.**
+
+## Features
+
+- 🔍 **Web Search** - Search via Google (falls back to DuckDuckGo if blocked)
+- 🌐 **Page Fetching** - Extract readable content from any URL
+- 🤖 **Bot Protection Bypass** - Handles JS challenges, Cloudflare, etc.
+- 🚀 **Persistent Daemon** - Warm browser session for fast subsequent requests
+- 🖥️ **Cross-Platform** - Auto-detects Brave, Chrome, Edge, Chromium
+
+## Install
+
+```bash
+pi install npm:@ogulcancelik/pi-web-browse
+```
+
+Or via git:
+
+```bash
+pi install github.com/ogulcancelik/pi-web-browse
+```
+
+(Optional, try without installing):
+
+```bash
+pi -e npm:@ogulcancelik/pi-web-browse
+```
+
+After first use, the agent will guide you through setup.
+
+## Usage
+
+The agent will automatically use this skill when you ask it to search the web or fetch page content.
+
+You can also invoke it directly:
+
+```bash
+/skill:web-browse "rust async runtime"
+```
+
+## Configuration
+
+Environment variables (all optional):
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `WEB_BROWSE_BROWSER_BIN` | Browser binary path | Auto-detected |
+| `WEB_BROWSE_USER_AGENT` | User-Agent string | Chrome on Windows |
+| `WEB_BROWSE_DAEMON_PORT` | Daemon HTTP port | 9377 |
+| `WEB_BROWSE_CDP_PORT` | Chrome DevTools port | 9223 |
+| `WEB_BROWSE_DEBUG_DUMP` | Save debug files on failure | off |
+
+## Browser Detection
+
+The skill auto-detects browsers in common locations:
+
+- **Linux:** brave, brave-browser, google-chrome, chromium (from PATH)
+- **macOS:** Brave Browser, Google Chrome, Chromium, Edge (in /Applications)
+- **Windows:** Brave, Chrome, Edge, Chromium (Program Files, LocalAppData)
+
+## How It Works
+
+1. **Search** - Uses Google via headless browser (falls back to DuckDuckGo if blocked)
+2. **Fetch** - Opens URL in headless Chromium, waits for JS, extracts readable content
+3. **Daemon** - Keeps a warm browser session for speed + bot-protection bypass
+
+## License
+
+MIT

package/SKILL.md

@@ -0,0 +1,101 @@
+---
+name: web-browse
+description: "Search the web and fetch/read pages via a real headless browser (CDP). Use this instead of curl when sites are JS-heavy or bot-protected. Works on Linux, macOS, and Windows."
+---
+
+# Web Browse
+
+Search the web, then open/fetch pages in a **real browser session** (headless Chromium via CDP) and extract readable text.
+Use this instead of `curl` when sites are JS-heavy or bot-protected.
+
+## Setup
+
+Run once before first use:
+
+```bash
+cd {baseDir}
+npm install
+```
+
+The skill auto-detects browsers already installed on your system (Brave, Chrome, Edge, Chromium).
+On Windows, Edge is pre-installed and works out of the box.
+
+**No browser installed?** (rare) Run: `npx playwright install chromium`
+
+## Configuration (optional)
+
+Environment variables:
+
+| Variable | Description |
+|----------|-------------|
+| `WEB_BROWSE_BROWSER_BIN` | Path to browser binary (auto-detected if not set) |
+| `WEB_BROWSE_USER_AGENT` | Override User-Agent string |
+| `WEB_BROWSE_DAEMON_PORT` | Daemon port (default: 9377) |
+| `WEB_BROWSE_CDP_PORT` | CDP port (default: 9223) |
+| `WEB_BROWSE_DEBUG_DUMP` | Set to `1` to save screenshots/HTML on failures |
+
+You can also pass `--browser-bin <path>` as a CLI argument.
+
+## Usage
+
+```bash
+# Search (results are cached for ~10 minutes)
+{baseDir}/web-browse.js "your query"
+{baseDir}/web-browse.js "your query" -n 10
+
+# Fetch specific cached results by index
+{baseDir}/web-browse.js --fetch 1,3,5
+
+# Fetch a specific URL
+{baseDir}/web-browse.js --url <url>          # truncated (~2000 chars)
+{baseDir}/web-browse.js --url <url> --full   # full content
+```
+
+**Windows note:** Use `node {baseDir}/web-browse.js` instead of `{baseDir}/web-browse.js`
+
+## Default behavior: persistent daemon (auto)
+
+Direct calls automatically start/use a local daemon that keeps a **persistent headless browser+CDP session**.
+This avoids browser startup overhead and helps with bot-protection pages that auto-clear (e.g. Anubis PoW).
+
+### Daemon controls (optional, for debugging)
+
+```bash
+{baseDir}/web-browse.js --daemon status
+{baseDir}/web-browse.js --daemon start
+{baseDir}/web-browse.js --daemon stop
+{baseDir}/web-browse.js --daemon restart
+```
+
+### Bypass daemon (one-shot)
+
+```bash
+{baseDir}/web-browse.js --no-daemon --url https://example.com
+{baseDir}/web-browse.js --no-daemon "your query"
+```
+
+## Workflow
+
+1) **Search** → see snippets → decide what to read
+2) **Fetch by index** → `--fetch 1,3` opens those results and extracts content
+
+```bash
+{baseDir}/web-browse.js "rust async runtime"  # shows results
+{baseDir}/web-browse.js --fetch 1,3           # fetches result #1 and #3
+```
+
+## Browser Support
+
+The skill auto-detects installed browsers in this order:
+
+**Linux:** brave, brave-browser, google-chrome, chromium
+**macOS:** Brave Browser, Google Chrome, Chromium, Microsoft Edge (in /Applications)
+**Windows:** Brave, Chrome, Edge, Chromium (common install paths)
+
+To use a specific browser, set `WEB_BROWSE_BROWSER_BIN` or pass `--browser-bin <path>`.
+
+## Notes
+
+- Content is truncated by default to save tokens; use `--full` for complete output.
+- The daemon keeps a warm browser session for faster subsequent requests.
+- CDP profile is stored in `~/.config/web-browse-cdp-profile/` (configurable via `--cdp-profile`).

package/lib/bot-protection.js

@@ -0,0 +1,92 @@
+const DEFAULT_MARKERS = [
+  "making sure you're not a bot",
+  "protected by anubis",
+  "anubis uses a proof-of-work",
+  "checking your browser",
+  "just a moment",
+  "cf-browser-verification",
+  "enable javascript and cookies to continue",
+  "attention required",
+  "verify you are human",
+  "unusual traffic",
+];
+
+export function isLikelyBotProtectionText(title, text, markers = DEFAULT_MARKERS) {
+  const t = String(title || "").toLowerCase();
+  const body = String(text || "").slice(0, 6000).toLowerCase();
+  const haystack = `${t}\n${body}`;
+  return markers.some((m) => haystack.includes(m));
+}
+
+export async function isLikelyBotProtectionPage(page, markers = DEFAULT_MARKERS) {
+  for (let attempt = 0; attempt < 3; attempt += 1) {
+    try {
+      return await page.evaluate((markersArg) => {
+        const title = (document.title || "").toLowerCase();
+        const text = (document.body?.innerText || "").slice(0, 6000).toLowerCase();
+        const haystack = `${title}\n${text}`;
+        return markersArg.some((marker) => haystack.includes(marker));
+      }, markers);
+    } catch (err) {
+      const message = err instanceof Error ? err.message : String(err);
+      if (message.includes("Execution context was destroyed") || message.includes("Cannot find context")) {
+        await page.waitForTimeout(250).catch(() => {});
+        continue;
+      }
+
+      return false;
+    }
+  }
+
+  return false;
+}
+
+export async function waitForBotProtectionToClear(
+  page,
+  url,
+  {
+    markers = DEFAULT_MARKERS,
+    timeoutMs = 30000,
+    log = (msg) => console.error(msg),
+  } = {},
+) {
+  // Fast-path: for normal pages, do NOT wait for networkidle.
+  await page.waitForTimeout(150 + Math.floor(Math.random() * 150));
+
+  let detected = false;
+  for (let i = 0; i < 3; i += 1) {
+    detected = await isLikelyBotProtectionPage(page, markers);
+    if (detected) break;
+    await page.waitForTimeout(200);
+  }
+
+  if (!detected) return { detected: false, cleared: true, waitedMs: 0 };
+
+  const start = Date.now();
+  log(`Bot protection detected for ${url}. Waiting for it to clear...`);
+
+  await page
+    .waitForFunction(
+      (markersArg) => {
+        const title = (document.title || "").toLowerCase();
+        const text = (document.body?.innerText || "").slice(0, 6000).toLowerCase();
+        const haystack = `${title}\n${text}`;
+        return !markersArg.some((marker) => haystack.includes(marker));
+      },
+      markers,
+      { timeout: timeoutMs },
+    )
+    .catch(() => {});
+
+  await page.waitForLoadState("domcontentloaded", { timeout: 10000 }).catch(() => {});
+  await page.waitForLoadState("networkidle", { timeout: 5000 }).catch(() => {});
+  await page.waitForTimeout(150);
+
+  const stillBlocked = await isLikelyBotProtectionPage(page, markers);
+  if (stillBlocked) {
+    const title = await page.title().catch(() => "");
+    throw new Error(`Bot protection challenge did not clear (title="${title}")`);
+  }
+
+  return { detected: true, cleared: true, waitedMs: Date.now() - start };
+}

package/lib/browser-bin.js

@@ -0,0 +1,133 @@
+import { accessSync, constants, existsSync } from "node:fs";
+import { join, delimiter } from "node:path";
+import { platform } from "node:os";
+
+const PLATFORM = platform();
+const IS_MACOS = PLATFORM === "darwin";
+const IS_WINDOWS = PLATFORM === "win32";
+
+// macOS .app bundle paths (checked as absolute paths)
+const MACOS_BROWSER_PATHS = [
+  "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser",
+  "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
+  "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
+  "/Applications/Chromium.app/Contents/MacOS/Chromium",
+  "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
+];
+
+// Linux binary names (searched on PATH)
+const LINUX_BROWSER_NAMES = [
+  "brave",
+  "brave-browser",
+  "google-chrome",
+  "google-chrome-stable",
+  "chromium",
+  "chromium-browser",
+];
+
+// Windows browser paths (common install locations)
+const WINDOWS_BROWSER_PATHS = [
+  // Brave
+  join(process.env.LOCALAPPDATA || "", "BraveSoftware", "Brave-Browser", "Application", "brave.exe"),
+  join(process.env.PROGRAMFILES || "", "BraveSoftware", "Brave-Browser", "Application", "brave.exe"),
+  join(process.env["PROGRAMFILES(X86)"] || "", "BraveSoftware", "Brave-Browser", "Application", "brave.exe"),
+  // Chrome
+  join(process.env.LOCALAPPDATA || "", "Google", "Chrome", "Application", "chrome.exe"),
+  join(process.env.PROGRAMFILES || "", "Google", "Chrome", "Application", "chrome.exe"),
+  join(process.env["PROGRAMFILES(X86)"] || "", "Google", "Chrome", "Application", "chrome.exe"),
+  // Edge (comes with Windows 10/11)
+  join(process.env.PROGRAMFILES || "", "Microsoft", "Edge", "Application", "msedge.exe"),
+  join(process.env["PROGRAMFILES(X86)"] || "", "Microsoft", "Edge", "Application", "msedge.exe"),
+  // Chromium
+  join(process.env.LOCALAPPDATA || "", "Chromium", "Application", "chrome.exe"),
+];
+
+function isExecutableFile(filePath) {
+  try {
+    // On Windows, just check if file exists (no X_OK bit)
+    if (IS_WINDOWS) {
+      return existsSync(filePath);
+    }
+    accessSync(filePath, constants.X_OK);
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+function findExecutableOnPath(name, env = process.env) {
+  if (!name) return null;
+
+  // Absolute/relative path - check directly
+  // Handle both Unix "/" and Windows "\" or "C:\"
+  if (name.includes("/") || (IS_WINDOWS && (name.includes("\\") || /^[A-Za-z]:/.test(name)))) {
+    return isExecutableFile(name) ? name : null;
+  }
+
+  const pathEnv = env.PATH || env.Path || "";
+  const dirs = pathEnv.split(delimiter).filter(Boolean);
+
+  for (const dir of dirs) {
+    const candidate = join(dir, name);
+    if (isExecutableFile(candidate)) return candidate;
+    // On Windows, try with .exe extension if not provided
+    if (IS_WINDOWS && !name.toLowerCase().endsWith(".exe")) {
+      const candidateExe = join(dir, name + ".exe");
+      if (isExecutableFile(candidateExe)) return candidateExe;
+    }
+  }
+
+  return null;
+}
+
+/**
+ * Resolve a browser binary for CDP automation.
+ *
+ * Precedence:
+ *  - preferredBin (CLI)
+ *  - WEB_BROWSE_BROWSER_BIN
+ *  - BRAVE_BIN (backwards compat)
+ *  - OS-specific defaults (macOS .app bundles, Windows paths, or Linux PATH names)
+ */
+export function resolveBrowserBin(preferredBin = null, env = process.env) {
+  // Priority overrides (env vars, CLI arg)
+  const overrides = [
+    preferredBin,
+    env.WEB_BROWSE_BROWSER_BIN,
+    env.BRAVE_BIN,
+  ].filter(Boolean);
+
+  // Check overrides first
+  for (const cand of overrides) {
+    const resolved = findExecutableOnPath(cand, env);
+    if (resolved) return resolved;
+  }
+
+  // OS-specific browser paths
+  let osCandidates;
+  let osName;
+  if (IS_MACOS) {
+    osCandidates = MACOS_BROWSER_PATHS;
+    osName = "macOS";
+  } else if (IS_WINDOWS) {
+    osCandidates = WINDOWS_BROWSER_PATHS;
+    osName = "Windows";
+  } else {
+    osCandidates = LINUX_BROWSER_NAMES;
+    osName = "Linux";
+  }
+
+  for (const cand of osCandidates) {
+    // Skip empty paths (from undefined env vars on Windows)
+    if (!cand || cand.startsWith(join("", ""))) continue;
+    const resolved = IS_WINDOWS ? (isExecutableFile(cand) ? cand : null) : findExecutableOnPath(cand, env);
+    if (resolved) return resolved;
+  }
+
+  const allTried = [...overrides, ...osCandidates.filter(Boolean)];
+  throw new Error(
+    `No supported browser binary found on ${osName}. ` +
+      "Set WEB_BROWSE_BROWSER_BIN or BRAVE_BIN, or pass --browser-bin <path>. " +
+      `Tried: ${allTried.join(", ")}`,
+  );
+}

package/lib/cdp.js

@@ -0,0 +1,218 @@
+import { spawn } from "node:child_process";
+import { createServer as createNetServer } from "node:net";
+import { platform } from "node:os";
+
+import { resolveBrowserBin } from "./browser-bin.js";
+
+const PLATFORM = platform();
+const IS_MACOS = PLATFORM === "darwin";
+const IS_WINDOWS = PLATFORM === "win32";
+
+export async function waitForCdpVersion(port, timeoutMs = 10000) {
+  const start = Date.now();
+
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const response = await fetch(`http://127.0.0.1:${port}/json/version`, { method: "GET" });
+      if (response.ok) {
+        const payload = await response.json().catch(() => null);
+        if (
+          payload &&
+          typeof payload === "object" &&
+          typeof payload.webSocketDebuggerUrl === "string" &&
+          payload.webSocketDebuggerUrl.startsWith("ws")
+        ) {
+          return payload;
+        }
+      }
+    } catch {
+      // ignore
+    }
+
+    await new Promise((resolve) => setTimeout(resolve, 300));
+  }
+
+  return null;
+}
+
+export async function waitForCdp(port, timeoutMs = 10000) {
+  return Boolean(await waitForCdpVersion(port, timeoutMs));
+}
+
+async function getEphemeralPort() {
+  return new Promise((resolve, reject) => {
+    const server = createNetServer();
+    server.unref();
+
+    server.once("error", reject);
+
+    server.listen(0, "127.0.0.1", () => {
+      const address = server.address();
+      const port = typeof address === "object" && address ? address.port : undefined;
+
+      server.close((err) => {
+        if (err) return reject(err);
+        if (!port) return reject(new Error("Could not determine ephemeral port"));
+        resolve(port);
+      });
+    });
+  });
+}
+
+async function isPortAvailable(port) {
+  if (!Number.isInteger(port) || port <= 0 || port > 65535) return false;
+
+  return new Promise((resolve) => {
+    const server = createNetServer();
+    server.unref();
+
+    server.once("error", () => resolve(false));
+
+    server.listen(port, "127.0.0.1", () => {
+      server.close(() => resolve(true));
+    });
+  });
+}
+
+async function chooseAvailablePort(preferredPort) {
+  if (await isPortAvailable(preferredPort)) return preferredPort;
+
+  for (let offset = 1; offset <= 25; offset += 1) {
+    const candidate = preferredPort + offset;
+    if (await isPortAvailable(candidate)) return candidate;
+  }
+
+  return await getEphemeralPort();
+}
+
+export async function startBrowserForCdp(preferredPort, profileDir, browserBin = null, spawnedProcessGroupPids = null) {
+  const bin = resolveBrowserBin(browserBin);
+  const port = await chooseAvailablePort(preferredPort);
+
+  // OS-specific headless flags
+  let headlessArgs;
+  if (IS_MACOS || IS_WINDOWS) {
+    // macOS and Windows: use standard headless mode
+    headlessArgs = [
+      "--headless=new",
+      "--window-size=1280,720",
+    ];
+  } else {
+    // Linux: use ozone headless platform (Wayland/X11 independent)
+    headlessArgs = [
+      "--ozone-platform=headless",
+      "--ozone-override-screen-size=1280,720",
+    ];
+  }
+
+  const args = [
+    ...headlessArgs,
+    "--no-sandbox",
+    "--disable-setuid-sandbox",
+    "--no-first-run",
+    "--no-default-browser-check",
+
+    // Reduce background throttling so JS challenges (e.g., Anubis PoW) run at normal speed.
+    "--disable-background-timer-throttling",
+    "--disable-backgrounding-occluded-windows",
+    "--disable-renderer-backgrounding",
+
+    `--remote-debugging-port=${port}`,
+    "--remote-debugging-address=127.0.0.1",
+    `--user-data-dir=${profileDir}`,
+    "about:blank",
+  ];
+
+  const env = { ...process.env };
+  // Prevent any UI from connecting to the current Wayland/X11 session (Linux only).
+  if (!IS_MACOS && !IS_WINDOWS) {
+    delete env.WAYLAND_DISPLAY;
+    delete env.DISPLAY;
+  }
+
+  // On Windows, don't use detached mode (process groups work differently)
+  const spawnOpts = IS_WINDOWS
+    ? { stdio: "ignore", env }
+    : { stdio: "ignore", detached: true, env };
+
+  const proc = spawn(bin, args, spawnOpts);
+  if (!IS_WINDOWS) proc.unref();
+
+  if (spawnedProcessGroupPids && proc.pid) {
+    spawnedProcessGroupPids.add(proc.pid);
+  }
+
+  const ready = await waitForCdp(port, 15000);
+  if (!ready) {
+    if (spawnedProcessGroupPids && proc.pid) spawnedProcessGroupPids.delete(proc.pid);
+
+    killBrowserProcess(proc);
+
+    throw new Error(`Failed to start browser with CDP on port ${port} (bin=${bin})`);
+  }
+
+  return { proc, port, bin };
+}
+
+/**
+ * Kill a browser process (cross-platform)
+ */
+export function killBrowserProcess(proc) {
+  if (!proc || !proc.pid) return;
+  
+  try {
+    if (IS_WINDOWS) {
+      // Windows: use taskkill to kill process tree
+      spawn("taskkill", ["/pid", proc.pid.toString(), "/T", "/F"], { stdio: "ignore" });
+    } else {
+      // Unix: kill process group (negative PID)
+      process.kill(-proc.pid);
+    }
+  } catch {
+    // Fallback: try killing just the process
+    try {
+      proc.kill();
+    } catch {
+      // ignore
+    }
+  }
+}
+
+export function isLikelyUsableBrowserCdp(versionPayload) {
+  if (!versionPayload || typeof versionPayload !== "object") return false;
+
+  const userAgent = typeof versionPayload["User-Agent"] === "string" ? versionPayload["User-Agent"] : "";
+  if (userAgent.toLowerCase().includes("electron/")) return false;
+
+  return true;
+}
+
+export async function resolveCdpOptions({ useCdpFlag, cdpStartFlag, cdpPortValue }) {
+  let effectiveUseCdp = useCdpFlag || cdpStartFlag;
+  let effectiveCdpStart = cdpStartFlag;
+  let effectiveCdpPort = cdpPortValue;
+
+  if (!effectiveUseCdp && !effectiveCdpStart) {
+    const cdp9223 = await waitForCdpVersion(9223, 1000);
+    if (isLikelyUsableBrowserCdp(cdp9223)) {
+      effectiveUseCdp = true;
+      effectiveCdpPort = 9223;
+    } else {
+      const cdp9222 = await waitForCdpVersion(9222, 1000);
+      if (isLikelyUsableBrowserCdp(cdp9222)) {
+        effectiveUseCdp = true;
+        effectiveCdpPort = 9222;
+      } else {
+        effectiveUseCdp = true;
+        effectiveCdpStart = true;
+        effectiveCdpPort = 9223;
+      }
+    }
+  }
+
+  return {
+    useCdp: effectiveUseCdp,
+    cdpStart: effectiveCdpStart,
+    cdpPort: effectiveCdpPort,
+  };
+}