pi-unsloth-webtools 0.6.0 → 0.7.0

package/README.md

@@ -1,8 +1,14 @@
 # pi-unsloth-webtools
 
-A [pi](https://github.com/earendil-works/pi-coding-agent) extension providing `web_search` and
-`web_fetch` tools, ported from the Unsloth Studio codebase
-([`unslothai/unsloth`](https://github.com/unslothai/unsloth), `studio/backend/core/inference/`).
+A [pi](https://github.com/earendil-works/pi-coding-agent) extension providing `web_search`,
+`web_fetch`, and `web_render` tools. It began as a port of the Unsloth Studio codebase
+([`unslothai/unsloth`](https://github.com/unslothai/unsloth), `studio/backend/core/inference/`);
+the engine, extraction, and PDF layers are still derived from it, but the package is no longer
+behavior-identical to Studio — it enables local file and private-address fetching by default
+and adds a fetch cache, Wayback fallbacks, page metadata, a third-party rendering tool
+(`web_render`), and other behavior Studio does not have. See
+[Known differences from Studio](#known-differences-from-studio). The `unsloth` in the name marks
+provenance, not affiliation.
 
 ## Install
 
@@ -20,6 +26,8 @@ pi install /path/to/pi-unsloth-webtools
 
 ## What it does
 
+All three tools display their target in the TUI tool row: `web_search "query"`, `web_search <url>` in url mode, `web_fetch <url>`, and `web_render <url>`.
+
 ### web_search
 
 Mirrors Unsloth Studio's `web_search` tool:
@@ -112,8 +120,34 @@ Port of Studio's `_fetch_page_text` / `_fetch_url_raw` pipeline:
   `application-name`) lines are added when declared, so the model can judge recency and
   provenance.
 
+### web_render
+
+Renders a public page to Markdown through the third-party Jina Reader (`r.jina.ai`) for pages
+that need JavaScript to render:
+
+- Every target is validated and resolved locally first: http/https only, and any private,
+  loopback, link-local, or otherwise non-public address is refused. Local files are refused on
+  this path regardless of `webFetch.allowPrivateAddresses` / `webFetch.allowLocalFiles`, because
+  the URL is sent to Jina.
+- `unslothWebTools.jinaApiKey` (or `webRender.jinaApiKey`, or the `JINA_API_KEY` environment
+  variable) raises the Reader's rate limits; without a key it still works at Jina's free limits.
+- Output is Markdown prefixed with `Title:` / `URL:` lines and a `Rendered via the Jina Reader`
+  provenance line. An optional `maxChars` truncates, like `web_fetch`.
+- Keyless Reader requests are rate-limited per outgoing IP; see
+  [Companion: rotating exit IPs](#companion-rotating-exit-ips).
+- Enabled by default. Disable it with `/webtools-config` (`webRenderEnabled` in
+  `~/.config/pi-unsloth-webtools/config.json`), which deactivates the tool for the session.
+
 ## Known differences from Studio
 
+- Local access: Studio validates every resolved address against non-public ranges and fetches
+  public web content only. This port permits private/loopback/link-local targets and local files
+  (`file://` URLs and absolute, `~/`, `./` paths) by default; opt out with
+  `webFetch.allowPrivateAddresses: false` and `webFetch.allowLocalFiles: false` to restore
+  Studio's behavior.
+- Third-party rendering: the extra `web_render` tool asks the Jina Reader (`r.jina.ai`) to fetch
+  the page, so the target URL leaves the machine. Studio has no third-party rendering path. This
+  path always refuses local files and non-public addresses, regardless of the local-access settings.
 - PDF styling: MuPDF.js exposes one font per line, so mixed-style lines style the
   whole line instead of per-span; superscript, subscript, underline, strikeout, and
   highlight markers are not emitted. Tables use a conservative text-grid detector:
@@ -133,8 +167,10 @@ Port of Studio's `_fetch_page_text` / `_fetch_url_raw` pipeline:
   generic engine failures. The timeout budget bounds the entire sweep: per-engine
   timeouts shrink as the budget is consumed, so the reported timeout matches the
   worst-case wall time.
-- Proxies: Studio routes through environment proxies; this port always connects
-  directly with DNS pinning (deliberately out of scope).
+- Proxies: Studio routes through environment proxies; this port's direct fetch always connects
+  directly with DNS pinning (deliberately out of scope). The search and `web_render` paths use the
+  process-wide `fetch`, so an agent-level proxy dispatcher does apply to them — see
+  [Companion: rotating exit IPs](#companion-rotating-exit-ips).
 - Dedup and titles: the aggregator keys on canonicalized hrefs (`utm_*`/tracking parameters
   and fragments stripped, then the URL re-serialized); fetched HTML pages are prefixed with
   the document `<title>`. Studio keys on raw hrefs and returns the converted body alone.
@@ -145,14 +181,16 @@ Port of Studio's `_fetch_page_text` / `_fetch_url_raw` pipeline:
 
 ## When to use alternatives
 
-This package is intentionally a faithful, zero-dependency port of Studio's pipeline. Use it
-when you need deterministic, offline-friendly behavior with strong SSRF guarantees and test
-parity with `unsloth/studio`. For other tradeoffs, prefer:
+This package keeps Studio's deterministic, zero-dependency pipeline and test parity with
+`unsloth/studio`, then layers local access, third-party rendering, and other Studio-independent
+behavior on top. The SSRF guard is real and thoroughly tested, but it is opt-in:
+`allowPrivateAddresses` defaults to `true`, and local files are readable unless `allowLocalFiles`
+is `false`. For other tradeoffs, prefer:
 
 | Need | Use |
 |---|---|
 | Browser-like TLS/HTTP fingerprinting to unblock bot-defended pages | `pi-smart-fetch` (`wreq-js` `chrome_145`) |
-| Headless Chrome for JS-rendered SPAs/YouTube/Reddit threads | `georgebashi/pi-web-fetch` (puppeteer + trafilatura) |
+| Headless Chrome for JS-rendered SPAs/YouTube/Reddit threads | Built-in `web_render` (Jina Reader) first; `georgebashi/pi-web-fetch` (puppeteer + trafilatura) when the Reader falls short |
 | Hosted search with semantic ranking and no scraping | `Brave Search API` / `Tavily` / `Exa` via `pi-ollama-web-search` |
 | Prompt-focused page distillation to save context | `pi-web-fetch` `prompt` -> sub-agent or Claude Code `WebFetch(url,prompt)` |
 | Batch fetching many URLs concurrently | `pi-smart-fetch` `batch_web_fetch` or call `web_fetch` in parallel |
@@ -160,6 +198,25 @@ parity with `unsloth/studio`. For other tradeoffs, prefer:
 Mixing is supported: `pi install npm:pi-unsloth-webtools npm:pi-smart-fetch` lets the model
 choose the best tool per URL. No need to fork this package to add those features.
 
+## Companion: rotating exit IPs
+
+[`pi-tor-proxy`](https://github.com/YuGiMob/pi-tor-proxy) routes pi's in-process `fetch` traffic
+through Tor (it downloads and manages its own Tor binary) and gives each pi instance its own
+circuit and exit IP. The search sweep and `web_render` both use `fetch`, so they leave through
+the current Tor exit, and Jina rate-limits keyless Reader requests per outgoing IP —
+`/tor-cycle` swaps the exit those limits are counted against, while `/tor-country` and
+`/tor-exclude` constrain which exits are used.
+
+```sh
+pi install npm:pi-unsloth-webtools npm:pi-tor-proxy
+```
+
+`web_fetch` is not routed: it connects directly through `node:http`/`node:https` with a pinned,
+validated IP and ignores the proxy variables (see the proxy note under Known differences).
+Caveats: Tor mode supports Linux and macOS only, adds latency, and many search engines and
+Cloudflare-fronted services challenge or block Tor exits, so cycling helps with per-IP limits but
+is not a guarantee.
+
 ## Configuration
 
 Optional settings in `~/.pi/agent/settings.json` or `.pi/settings.json` (project overrides global):
@@ -187,10 +244,30 @@ Optional settings in `~/.pi/agent/settings.json` or `.pi/settings.json` (project
 | `websitePolicy` | none | Not read from settings. Tools run unrestricted by default; `websitePolicy` is a programmatic option the host passes to `webSearch` / `fetchPageText` |
 | `unslothWebTools.allowPrivateAddresses` / `webFetch.allowPrivateAddresses` | `true` | Opt out to restore the resolved-IP SSRF guard: private/loopback/link-local hosts (localhost, LAN IPs) are refused again. Non-canonical numeric IP encodings stay blocked either way |
 | `unslothWebTools.allowLocalFiles` / `webFetch.allowLocalFiles` | `true` | Opt out to refuse local files in `web_fetch` and `web_search` url mode (`file://` URLs, absolute, `~/`, or `./` paths); when enabled, PDFs are extracted and HTML converted |
+| `unslothWebTools.jinaApiKey` / `webRender.jinaApiKey` | none (`JINA_API_KEY` fallback) | API key for `web_render`'s Jina Reader; raises its rate limits. Settings keys win over the environment variable |
+
+### Settings window
+
+`/webtools-config` opens an interactive settings window (↑↓ navigate, space toggle, q close). Settings
+persist across sessions in `~/.config/pi-unsloth-webtools/config.json`, created when a setting is
+first changed:
+
+```json
+{
+  "webRenderEnabled": true
+}
+```
+
+| Key | Default | Description |
+|---|---|---|
+| `webRenderEnabled` | `true` | When `false`, the `web_render` tool is deactivated for the session; `web_search` and `web_fetch` are unaffected |
+
+On non-Windows platforms the directory honors `XDG_CONFIG_HOME` when set (falling back to
+`~/.config`); on Windows it always uses `~/.config`.
 
 Tool params always win over file defaults. Search dedup also strips default ports, so `https://example.com:443/a` and `https://example.com/a` collapse.
 
-Environment overrides: `PI_UNSLOTH_CACHE_DIR` changes the fetch cache directory, `PI_UNSLOTH_WEBTOOLS_STATS` opts into append-only sweep stats JSONL, `PI_CODING_AGENT_DIR` / `PI_AGENT_DIR` change the global settings directory. Cache entries live 1 hour and stale copies are served only after a network failure.
+Environment overrides: `PI_UNSLOTH_CACHE_DIR` changes the fetch cache directory, `PI_UNSLOTH_WEBTOOLS_STATS` opts into append-only sweep stats JSONL, `PI_CODING_AGENT_DIR` / `PI_AGENT_DIR` change the global settings directory, and `JINA_API_KEY` supplies the `web_render` key when no settings key is set. Cache entries live 1 hour and stale copies are served only after a network failure.
 
 ## Troubleshooting
 
@@ -212,7 +289,10 @@ Match on the exact prefix. Do not retry blocked hosts with spelling tricks.
 | PDF without text | `(PDF contains no extractable text)` / `(PDF content could not be read as text...)` | Scanned or encrypted PDF. |
 | Download cap hit | `... (page truncated at the download limit)` | Raw fetch hit 512 KiB (10 MiB for PDFs). |
 | maxChars cut | `... (truncated, N chars total)` | Raise `maxChars` for the full text. |
-| Empty page | `(page returned no readable text)` | Page had no extractable text; JS-rendered pages need a browser tool. |
+| Empty page | `(page returned no readable text)` | Page had no extractable text; try `web_render`, which renders JavaScript pages. |
+| Render blocked (local file) | `Blocked: web_render cannot fetch local files.` | Use `web_fetch`, which reads local files by default. |
+| Render blocked (private host) | `Blocked: refusing to fetch the non-public address ...` | `web_render` only reaches public hosts; use `web_fetch` for localhost/LAN. |
+| Render failure | `Failed to render URL: ...` | Jina rejected or failed the request (rate limit, bad key, unreachable page). Retry, or check `unslothWebTools.jinaApiKey` / `JINA_API_KEY`. |
 | GitHub rewrite | `README of ... (fetched via the GitHub README API):` | Expected repo-root rewrite, not the HTML chrome. |
 | Cache fallback | `Served from cache` / `STALE cache from YYYY-MM-DD` | Network failed; output is the cached copy with its date. |
 | Wayback fallback | `Fetched from Wayback Machine snapshot (YYYY-MM-DD) for ...` | Original 404'd; output is the archived copy with its date. |
@@ -253,6 +333,8 @@ The suite ports Unsloth Studio's own tests for these tools:
 - `test/smoke.test.ts`: live network checks against real hosts, including a per-engine
   result-health sweep (at least two engines must return well-formed results; engines
   that block or reset connections from datacenter IPs count as unhealthy, not failures)
+- `test/web-render.test.ts`: Jina Reader rendering with a stubbed fetch and DNS, the public-only
+  guard, policy enforcement, error mapping, truncation, cancellation, and API key precedence
 
 The seams (`seams.resolve` / `seams.request` / `rawFetch`) replace the network stack
 with fakes, mirroring how the Studio suite monkeypatches `_validate_and_resolve_host`

package/config-ui.ts

@@ -0,0 +1,105 @@
+import type { Theme } from "@earendil-works/pi-coding-agent";
+import { Key, matchesKey, visibleWidth } from "@earendil-works/pi-tui";
+import { readConfig, type Config } from "./config.ts";
+
+export type ConfigToggleKey = "webRenderEnabled";
+
+export interface ConfigRow {
+  key: ConfigToggleKey;
+  label: string;
+  hint: string;
+  enabled: boolean;
+}
+
+export function configRows(config: Config): ConfigRow[] {
+  return [
+    {
+      key: "webRenderEnabled",
+      label: "Web render",
+      hint: "web_render tool (Jina Reader)",
+      enabled: config.webRenderEnabled !== false,
+    },
+  ];
+}
+
+function padRow(theme: Theme, innerWidth: number, content: string): string {
+  const padded = content + " ".repeat(Math.max(0, innerWidth - visibleWidth(content)));
+  return theme.fg("border", "│") + padded + theme.fg("border", "│");
+}
+
+export class WebToolsConfigOverlay {
+  private rows: ConfigRow[] = [];
+  private selected = 0;
+
+  constructor(
+    private readonly opts: {
+      tui: { requestRender(force?: boolean): void };
+      theme: Theme;
+      done: () => void;
+      onToggle: (key: ConfigToggleKey) => Promise<void>;
+    },
+  ) {}
+
+  async load(): Promise<void> {
+    this.rows = configRows(await readConfig());
+  }
+
+  private runToggle(row: ConfigRow): void {
+    this.opts.tui.requestRender(true);
+    void this.opts
+      .onToggle(row.key)
+      .then(async () => {
+        this.rows = configRows(await readConfig());
+        this.opts.tui.requestRender(true);
+      })
+      .catch((error: unknown) => {
+        console.error("Failed to toggle web tools setting:", error);
+      });
+  }
+
+  private toggleSelected(): void {
+    const row = this.rows[this.selected];
+    if (!row) return;
+    row.enabled = !row.enabled;
+    this.runToggle(row);
+  }
+
+  handleInput(data: string): void {
+    if (matchesKey(data, Key.up) || data === "k") {
+      this.selected = (this.selected + this.rows.length - 1) % this.rows.length;
+      return;
+    }
+    if (matchesKey(data, Key.down) || data === "j") {
+      this.selected = (this.selected + 1) % this.rows.length;
+      return;
+    }
+    if (matchesKey(data, Key.space) || matchesKey(data, Key.enter) || data === " " || data === "\r" || data === "\n") {
+      this.toggleSelected();
+      return;
+    }
+    if (matchesKey(data, Key.escape) || data === "q") {
+      this.opts.done();
+    }
+  }
+
+  invalidate(): void {}
+
+  render(width: number): string[] {
+    const theme = this.opts.theme;
+    const innerWidth = width - 2;
+    const lines: string[] = [];
+    lines.push(theme.fg("border", `╭${"─".repeat(innerWidth)}╮`));
+    lines.push(padRow(theme, innerWidth, ` ${theme.fg("accent", theme.bold("Web Tools Config"))}`));
+    lines.push(theme.fg("border", `├${"─".repeat(innerWidth)}┤`));
+    this.rows.forEach((row, index) => {
+      const cursor = index === this.selected ? theme.fg("accent", "> ") : "  ";
+      const box = row.enabled ? theme.fg("success", "[x]") : theme.fg("dim", "[ ]");
+      const label = index === this.selected ? theme.fg("accent", theme.bold(row.label)) : row.label;
+      lines.push(padRow(theme, innerWidth, `${cursor}${box} ${label} ${theme.fg("dim", `— ${row.hint}`)}`));
+    });
+    lines.push(theme.fg("border", `├${"─".repeat(innerWidth)}┤`));
+    lines.push(padRow(theme, innerWidth, theme.fg("dim", " ↑↓ navigate · space toggle · q close")));
+    lines.push(theme.fg("border", `╰${"─".repeat(innerWidth)}╯`));
+    return lines;
+  }
+}

package/config.ts

@@ -0,0 +1,207 @@
+import { randomUUID } from "node:crypto";
+import { mkdir, open, readFile, rename, rm, stat, writeFile } from "node:fs/promises";
+import { homedir } from "node:os";
+import { dirname, join } from "node:path";
+
+const CONFIG_DIR_NAME = "pi-unsloth-webtools";
+const TEMP_PREFIX = ".tmp-";
+const CONFIG_LOCK_DELAY_MS = 25;
+const CONFIG_LOCK_STALE_MS = 5000;
+const CONFIG_LOCK_RETRIES = Math.ceil(CONFIG_LOCK_STALE_MS / CONFIG_LOCK_DELAY_MS) * 2;
+
+export interface Config {
+  webRenderEnabled: boolean;
+}
+
+const DEFAULT_CONFIG: Config = {
+  webRenderEnabled: true,
+};
+
+function homeBase(): string {
+  const envHome = process.env.HOME;
+  return envHome && envHome.length > 0 ? envHome : homedir();
+}
+
+function configBase(): string {
+  if (process.platform !== "win32") {
+    const xdg = process.env.XDG_CONFIG_HOME;
+    if (xdg && xdg.length > 0) return xdg;
+  }
+  return join(homeBase(), ".config");
+}
+
+export function configDir(): string {
+  return join(configBase(), CONFIG_DIR_NAME);
+}
+
+export function configPath(): string {
+  return join(configDir(), "config.json");
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+  return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+
+function errCode(error: unknown): string | undefined {
+  if (typeof error !== "object" || error === null) return undefined;
+  const code = (error as { code?: unknown }).code;
+  return typeof code === "string" ? code : undefined;
+}
+
+function parseConfig(content: string): Config {
+  const parsed = JSON.parse(content) as unknown;
+  if (!isRecord(parsed) || (parsed.webRenderEnabled !== undefined && typeof parsed.webRenderEnabled !== "boolean")) {
+    throw new Error("config.json must be an object with a boolean webRenderEnabled field");
+  }
+  return {
+    webRenderEnabled:
+      typeof parsed.webRenderEnabled === "boolean" ? parsed.webRenderEnabled : DEFAULT_CONFIG.webRenderEnabled,
+  };
+}
+
+async function loadConfigFile(): Promise<{ config: Config; corrupted: boolean }> {
+  let content: string;
+  try {
+    content = await readFile(configPath(), "utf-8");
+  } catch (error: unknown) {
+    if (errCode(error) === "ENOENT") return { config: { ...DEFAULT_CONFIG }, corrupted: false };
+    console.error("Config file unreadable, using defaults:", error);
+    return { config: { ...DEFAULT_CONFIG }, corrupted: false };
+  }
+  try {
+    return { config: parseConfig(content), corrupted: false };
+  } catch (error: unknown) {
+    try {
+      const badPath = configPath();
+      await rename(badPath, `${badPath}.corrupt-${Date.now()}-${process.pid}-${Math.random().toString(36).slice(2)}`);
+    } catch {}
+    console.error("Config file corrupted, quarantined, using defaults:", error);
+    return { config: { ...DEFAULT_CONFIG }, corrupted: true };
+  }
+}
+
+export async function readConfig(): Promise<Config> {
+  return (await loadConfigFile()).config;
+}
+
+export async function readConfigWithStatus(): Promise<{ config: Config; corrupted: boolean }> {
+  return loadConfigFile();
+}
+
+async function syncDir(dir: string): Promise<void> {
+  if (process.platform === "win32") return;
+  try {
+    const handle = await open(dir, "r");
+    try {
+      await handle.sync();
+    } finally {
+      await handle.close();
+    }
+  } catch {}
+}
+
+export async function writeConfig(config: Config): Promise<void> {
+  const path = configPath();
+  const dir = dirname(path);
+  await mkdir(dir, { recursive: true, mode: 0o700 });
+  const tempPath = join(dir, `${TEMP_PREFIX}${randomUUID()}`);
+  const content = JSON.stringify(config, null, 2);
+  await writeFile(tempPath, content, { mode: 0o600 });
+  try {
+    await rename(tempPath, path);
+    await syncDir(dir);
+  } catch (error: unknown) {
+    if (process.platform === "win32" && errCode(error) === "EPERM") {
+      try {
+        await writeFile(path, content, "utf-8");
+        return;
+      } finally {
+        try {
+          await rm(tempPath, { force: true });
+        } catch {}
+      }
+    }
+    try {
+      await rm(tempPath, { force: true });
+    } catch {}
+    throw error;
+  }
+}
+
+interface ConfigLock {
+  path: string;
+  dev: number;
+  ino: number;
+}
+
+async function acquireConfigLock(lockPath: string): Promise<ConfigLock> {
+  try {
+    await mkdir(dirname(lockPath), { recursive: true, mode: 0o700 });
+  } catch {}
+  for (let attempt = 0; attempt < CONFIG_LOCK_RETRIES; attempt++) {
+    try {
+      await mkdir(lockPath, { mode: 0o700 });
+    } catch (error: unknown) {
+      if (errCode(error) === "ENOENT") {
+        try {
+          await mkdir(dirname(lockPath), { recursive: true, mode: 0o700 });
+        } catch {}
+        continue;
+      }
+      if (errCode(error) !== "EEXIST") throw error;
+      try {
+        const lockStats = await stat(lockPath);
+        if (Date.now() - lockStats.mtimeMs > CONFIG_LOCK_STALE_MS) {
+          await rm(lockPath, { recursive: true, force: true });
+          continue;
+        }
+      } catch {}
+      await new Promise<void>((resolve) => setTimeout(resolve, CONFIG_LOCK_DELAY_MS));
+      continue;
+    }
+    try {
+      const lockStats = await stat(lockPath);
+      return { path: lockPath, dev: lockStats.dev, ino: lockStats.ino };
+    } catch (error: unknown) {
+      if (errCode(error) !== "ENOENT") {
+        try {
+          await rm(lockPath, { recursive: true, force: true });
+        } catch {}
+      }
+      continue;
+    }
+  }
+  throw new Error(`[E_ACCESS] Could not acquire config lock: ${lockPath}`);
+}
+
+async function releaseConfigLock(lock: ConfigLock): Promise<void> {
+  try {
+    const lockStats = await stat(lock.path);
+    if (lockStats.dev !== lock.dev || lockStats.ino !== lock.ino) return;
+  } catch {
+    return;
+  }
+  try {
+    await rm(lock.path, { recursive: true, force: true });
+  } catch {}
+}
+
+export async function updateConfig(mut: (config: Config) => void): Promise<Config> {
+  const path = configPath();
+  const lock = await acquireConfigLock(`${path}.lock`);
+  try {
+    const config = await readConfig();
+    mut(config);
+    await writeConfig(config);
+    return config;
+  } finally {
+    await releaseConfigLock(lock);
+  }
+}
+
+export async function toggleWebRender(): Promise<boolean> {
+  const config = await updateConfig((current) => {
+    current.webRenderEnabled = !(current.webRenderEnabled === true);
+  });
+  return config.webRenderEnabled === true;
+}

package/index.ts

@@ -1,8 +1,21 @@
-import { defineTool, type ExtensionAPI, type ExtensionContext } from "@earendil-works/pi-coding-agent";
+import { defineTool, type ExtensionAPI, type ExtensionContext, type Theme } from "@earendil-works/pi-coding-agent";
 import { Type } from "typebox";
+import { collapseWhitespace } from "./html-to-md.ts";
 import { SEARCH_TIMEOUT_MS, webSearch as defaultWebSearch } from "./web-search.ts";
 import { DEFAULT_FETCH_TIMEOUT_MS, fetchPageText as defaultFetchPageText } from "./web-fetch.ts";
-import { loadDefaultFetchSettings, loadDefaultFetchTimeoutMs } from "./settings.ts";
+import { renderPageText as defaultRenderPageText } from "./web-render.ts";
+import { loadDefaultFetchSettings, loadDefaultFetchTimeoutMs, loadJinaApiKey } from "./settings.ts";
+import { readConfigWithStatus, toggleWebRender } from "./config.ts";
+import { WebToolsConfigOverlay } from "./config-ui.ts";
+
+function toolCallLine(theme: Theme, name: string, detail: string) {
+  const line = theme.fg("toolTitle", theme.bold(name)) + (detail ? ` ${theme.fg("accent", detail)}` : "");
+  return { render: () => [line], invalidate: () => {} };
+}
+
+function collapsedArg(value: unknown): string {
+  return collapseWhitespace(typeof value === "string" ? value : "");
+}
 
 function positiveNumber(value: unknown): number | undefined {
   if (typeof value !== "number" || !Number.isFinite(value)) return undefined;
@@ -67,14 +80,31 @@ const WebFetchParams = Type.Object({
   ),
 });
 
+const WebRenderParams = Type.Object({
+  url: Type.String({ description: "Public URL of the page to render" }),
+  maxChars: Type.Optional(
+    Type.Number({
+      description: "Truncate the returned content to this many characters (default: no limit)",
+    }),
+  ),
+  timeoutMs: Type.Optional(
+    Type.Number({
+      minimum: 1000,
+      description: "Overall timeout in milliseconds (default: 60000)",
+    }),
+  ),
+});
+
 export interface WebToolsDeps {
   fetchPageText?: typeof defaultFetchPageText;
   webSearch?: typeof defaultWebSearch;
+  renderPageText?: typeof defaultRenderPageText;
 }
 
 export function createWebTools(deps: WebToolsDeps = {}) {
   const fetchPageText = deps.fetchPageText ?? defaultFetchPageText;
   const webSearch = deps.webSearch ?? defaultWebSearch;
+  const renderPageText = deps.renderPageText ?? defaultRenderPageText;
   return {
     webSearchTool: defineTool({
       name: "web_search",
@@ -87,6 +117,11 @@ export function createWebTools(deps: WebToolsDeps = {}) {
         'Use web_search with the url parameter (e.g. {"url": "<URL>"}) to read the full text of a page found in search results.',
       ],
       parameters: WebSearchParams,
+      renderCall(args, theme) {
+        const url = collapsedArg(args.url);
+        const query = collapsedArg(args.query);
+        return toolCallLine(theme, "web_search", url || (query ? `"${query}"` : ""));
+      },
       async execute(_toolCallId, params, signal, onUpdate, _ctx) {
         if (params.url?.trim()) {
           const url = params.url.trim();
@@ -135,6 +170,9 @@ export function createWebTools(deps: WebToolsDeps = {}) {
         "webFetch.allowLocalFiles: false in settings. The download size is capped.",
       promptSnippet: "Fetch a web page and return readable text content",
       parameters: WebFetchParams,
+      renderCall(args, theme) {
+        return toolCallLine(theme, "web_fetch", collapsedArg(args.url));
+      },
       async execute(_toolCallId, params, signal, onUpdate, _ctx) {
         onUpdate?.({ content: [{ type: "text", text: `Fetching ${params.url}...` }], details: {} });
         const cwd = (_ctx as ExtensionContext | undefined)?.cwd;
@@ -149,11 +187,91 @@ export function createWebTools(deps: WebToolsDeps = {}) {
         return { content: [{ type: "text", text }], details: {} };
       },
     }),
+    webRenderTool: defineTool({
+      name: "web_render",
+      label: "Web Render",
+      description:
+        "Render a public web page to Markdown through the third-party Jina Reader (r.jina.ai). " +
+        "Use it for JavaScript-rendered pages that web_fetch cannot read. " +
+        "The target URL is sent to Jina; local files and non-public addresses are refused. " +
+        "An optional JINA_API_KEY or unslothWebTools.jinaApiKey raises the Reader's rate limits.",
+      promptSnippet: "Render a JavaScript-rendered page to Markdown via the Jina Reader",
+      promptGuidelines: [
+        'Use web_render when web_fetch reports "(page returned no readable text)" or the page only fills in through JavaScript.',
+      ],
+      parameters: WebRenderParams,
+      renderCall(args, theme) {
+        return toolCallLine(theme, "web_render", collapsedArg(args.url));
+      },
+      async execute(_toolCallId, params, signal, onUpdate, _ctx) {
+        onUpdate?.({ content: [{ type: "text", text: `Rendering ${params.url}...` }], details: {} });
+        const cwd = (_ctx as ExtensionContext | undefined)?.cwd;
+        const { timeoutMs, maxChars } = await fetchDefaults(cwd, params);
+        const apiKey = await loadJinaApiKey(cwd);
+        const text = await renderPageText(params.url, {
+          timeoutMs,
+          maxChars,
+          signal: signal ?? undefined,
+          apiKey,
+        });
+        return { content: [{ type: "text", text }], details: {} };
+      },
+    }),
   };
 }
 
 export default function (pi: ExtensionAPI) {
-  const { webSearchTool, webFetchTool } = createWebTools();
+  const { webSearchTool, webFetchTool, webRenderTool } = createWebTools();
   pi.registerTool(webSearchTool);
   pi.registerTool(webFetchTool);
+  pi.registerTool(webRenderTool);
+
+  pi.on("session_start", async (_event, ctx) => {
+    try {
+      const { config, corrupted } = await readConfigWithStatus();
+      if (corrupted && ctx.hasUI) {
+        ctx.ui.notify("Web tools config was corrupt and was reset to defaults", "warning");
+      }
+      if (!config.webRenderEnabled) {
+        pi.setActiveTools(pi.getActiveTools().filter((name) => name !== "web_render"));
+      }
+    } catch (error) {
+      console.error("Failed to load web tools config:", error);
+    }
+  });
+
+  pi.registerCommand("webtools-config", {
+    description: "Open the web tools settings window (web_render on/off)",
+    handler: async (_args, ctx) => {
+      if (!ctx.hasUI) {
+        ctx.ui.notify("/webtools-config requires interactive mode", "error");
+        return;
+      }
+      await ctx.ui.custom<void>(
+        async (tui, theme, _keybindings, done) => {
+          const overlay = new WebToolsConfigOverlay({
+            tui,
+            theme,
+            done,
+            onToggle: async (key) => {
+              if (key !== "webRenderEnabled") return;
+              const enabled = await toggleWebRender();
+              const active = pi.getActiveTools();
+              pi.setActiveTools(
+                enabled
+                  ? [...new Set([...active, "web_render"])]
+                  : active.filter((name) => name !== "web_render"),
+              );
+            },
+          });
+          await overlay.load();
+          return overlay;
+        },
+        {
+          overlay: true,
+          overlayOptions: { anchor: "center", width: "90%", minWidth: 60, maxHeight: "90%" },
+        },
+      );
+    },
+  });
 }

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "pi-unsloth-webtools",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "type": "module",
-  "description": "Pi extension: web_search and web_fetch tools ported from the Unsloth Studio codebase (DuckDuckGo search, SSRF-safe direct fetching, HTML-to-Markdown extraction)",
+  "description": "Pi extension: web_search and web_fetch tools that began as a port of the Unsloth Studio codebase and now diverge from it (multi-engine search, opt-in SSRF guard, HTML-to-Markdown extraction)",
   "main": "index.ts",
   "repository": {
     "type": "git",
@@ -23,8 +23,11 @@
   "license": "AGPL-3.0",
   "files": [
     "index.ts",
+    "config.ts",
+    "config-ui.ts",
     "web-search.ts",
     "web-fetch.ts",
+    "web-render.ts",
     "web-access.ts",
     "html-to-md.ts",
     "engines.ts",
@@ -44,6 +47,7 @@
   },
   "peerDependencies": {
     "@earendil-works/pi-coding-agent": "*",
+    "@earendil-works/pi-tui": "*",
     "typebox": "*"
   },
   "engines": {

package/settings.ts

@@ -49,6 +49,21 @@ function pickBoolean(data: Record<string, unknown>, paths: string[][]): boolean
   return undefined;
 }
 
+function pickString(data: Record<string, unknown>, paths: string[][]): string | undefined {
+  for (const path of paths) {
+    let cur: unknown = data;
+    for (const key of path) {
+      if (cur && typeof cur === "object" && !Array.isArray(cur)) cur = (cur as Record<string, unknown>)[key];
+      else {
+        cur = undefined;
+        break;
+      }
+    }
+    if (typeof cur === "string" && cur.trim()) return cur.trim();
+  }
+  return undefined;
+}
+
 const MAX_RESULTS = 5;
 
 const MAX_RESULTS_PATHS: string[][] = [
@@ -79,6 +94,11 @@ const ALLOW_LOCAL_FILES_PATHS: string[][] = [
   ["webFetch", "allowLocalFiles"],
 ];
 
+const JINA_API_KEY_PATHS: string[][] = [
+  ["unslothWebTools", "jinaApiKey"],
+  ["webRender", "jinaApiKey"],
+];
+
 function clampMaxResults(value: number): number {
   return Math.min(20, Math.max(1, value));
 }
@@ -148,3 +168,14 @@ export async function loadDefaultFetchSettings(cwd?: string): Promise<{
   }
   return { maxChars, timeoutMs, allowPrivateAddresses, allowLocalFiles };
 }
+
+export async function loadJinaApiKey(cwd?: string): Promise<string | undefined> {
+  let result: string | undefined;
+  for (const data of await settingsEntries(cwd)) {
+    const candidate = pickString(data, JINA_API_KEY_PATHS);
+    if (candidate !== undefined) result = candidate;
+  }
+  if (result !== undefined) return result;
+  const env = process.env.JINA_API_KEY?.trim();
+  return env ? env : undefined;
+}

package/web-fetch.ts

@@ -557,7 +557,7 @@ function sleepAbortable(ms: number, signal?: AbortSignal): Promise<void> {
   });
 }
 
-async function resolveAndValidate(
+export async function resolveAndValidateHost(
   hostname: string,
   signal?: AbortSignal,
   allowPrivateAddresses = false,
@@ -824,7 +824,7 @@ export async function fetchUrlRaw(
   const maxBytes = options.maxBytes ?? MAX_FETCH_BYTES;
   const maxPdfBytes = options.maxPdfBytes ?? MAX_PDF_FETCH_BYTES;
   const seams = options.seams ?? {};
-  const resolveHost = seams.resolve ?? resolveAndValidate;
+  const resolveHost = seams.resolve ?? resolveAndValidateHost;
   const allowPrivateAddresses = options.allowPrivateAddresses ?? true;
   const performRequest = seams.request ?? requestHop;
   const resolveWithBudget = async (hostname: string): Promise<ResolvedHost> => {
@@ -1256,7 +1256,7 @@ async function fetchWaybackSnapshot(
 }
 const WINDOWS_PATH_RE = /^[a-zA-Z]:[\\/]/;
 
-function parseLocalPath(url: string): string | null {
+export function parseLocalPath(url: string): string | null {
   const trimmed = url.trim();
   if (/^file:/i.test(trimmed)) {
     try {

package/web-render.ts

@@ -0,0 +1,100 @@
+import { collapseWhitespace } from "./html-to-md.ts";
+import { checkUrlAccess, MAX_SIGNAL_TIMEOUT_MS, normalizeUrlScheme, type WebsitePolicy } from "./web-access.ts";
+import { parseLocalPath, resolveAndValidateHost, truncatePageText } from "./web-fetch.ts";
+
+const DEFAULT_RENDER_TIMEOUT_MS = 60_000;
+const JINA_READER_URL = "https://r.jina.ai/";
+const CANCELLED_MESSAGE = "Failed to render URL: cancelled.";
+const TIMED_OUT_MESSAGE = "Failed to render URL: timed out.";
+const LOCAL_FILE_MESSAGE = "Blocked: web_render cannot fetch local files.";
+const EMPTY_READER_MESSAGE = "Failed to render URL: the Jina Reader returned no content.";
+
+export interface RenderPageOptions {
+  timeoutMs?: number;
+  maxChars?: number;
+  signal?: AbortSignal;
+  websitePolicy?: WebsitePolicy | null;
+  apiKey?: string;
+}
+
+interface JinaReaderData {
+  title?: string;
+  url?: string;
+  content?: string;
+}
+
+interface JinaReaderPayload {
+  data?: JinaReaderData;
+}
+
+function clampTimeout(value: number | undefined): number {
+  if (value === undefined || !Number.isFinite(value)) return DEFAULT_RENDER_TIMEOUT_MS;
+  return Math.min(MAX_SIGNAL_TIMEOUT_MS, Math.max(1, Math.floor(value)));
+}
+
+function budgetMessage(caller: AbortSignal | undefined, signal: AbortSignal): string | null {
+  if (caller?.aborted) return CANCELLED_MESSAGE;
+  if (signal.aborted) return TIMED_OUT_MESSAGE;
+  return null;
+}
+
+async function readerErrorDetail(response: Response): Promise<string> {
+  try {
+    const body = (await response.json()) as { readableMessage?: unknown; message?: unknown; detail?: unknown };
+    const detail = [body.readableMessage, body.message, body.detail].find(
+      (value) => typeof value === "string" && value.trim().length > 0,
+    );
+    if (detail) return `${response.status} ${detail}`;
+  } catch {}
+  return `${response.status} ${response.statusText}`.trim();
+}
+
+function formatRenderedPage(data: JinaReaderData, fallbackUrl: string, maxChars: number | undefined): string {
+  const title = collapseWhitespace(String(data.title ?? ""));
+  const finalUrl = collapseWhitespace(String(data.url ?? "")) || fallbackUrl;
+  const content = String(data.content ?? "").trim();
+  if (!content) return truncatePageText("", maxChars);
+  const header: string[] = [];
+  if (title) header.push(`Title: ${title}`);
+  header.push(`URL: ${finalUrl}`);
+  header.push("Rendered via the Jina Reader (r.jina.ai).");
+  return truncatePageText(`${header.join("\n")}\n\n${content}`, maxChars);
+}
+
+export async function renderPageText(url: string, options: RenderPageOptions = {}): Promise<string> {
+  const target = typeof url === "string" ? url.trim() : "";
+  const policy = options.websitePolicy ?? null;
+  if (!target) return checkUrlAccess("", policy)[1];
+  if (parseLocalPath(target) !== null) return LOCAL_FILE_MESSAGE;
+  const normalized = normalizeUrlScheme(target);
+  const [allowed, reason, hostname] = checkUrlAccess(normalized, policy);
+  if (!allowed) return reason;
+  const timeoutSignal = AbortSignal.timeout(clampTimeout(options.timeoutMs));
+  const signal = options.signal ? AbortSignal.any([options.signal, timeoutSignal]) : timeoutSignal;
+  const resolved = await resolveAndValidateHost(hostname, signal, false);
+  const resolutionBudget = budgetMessage(options.signal, signal);
+  if (resolutionBudget !== null) return resolutionBudget;
+  if (!resolved.ok) return resolved.reason;
+  const headers: Record<string, string> = { Accept: "application/json" };
+  const apiKey = options.apiKey?.trim();
+  if (apiKey) headers["Authorization"] = `Bearer ${apiKey}`;
+  let response: Response;
+  try {
+    response = await fetch(JINA_READER_URL + encodeURIComponent(normalized), { headers, signal });
+  } catch (err) {
+    const budget = budgetMessage(options.signal, signal);
+    if (budget !== null) return budget;
+    return `Failed to render URL: ${err instanceof Error ? err.message : String(err)}`;
+  }
+  if (!response.ok) return `Failed to render URL: ${await readerErrorDetail(response)}`;
+  let payload: JinaReaderPayload;
+  try {
+    payload = (await response.json()) as JinaReaderPayload;
+  } catch {
+    const budget = budgetMessage(options.signal, signal);
+    if (budget !== null) return budget;
+    return `Failed to render URL: the Jina Reader returned a non-JSON response (HTTP ${response.status}).`;
+  }
+  if (!payload || !payload.data) return EMPTY_READER_MESSAGE;
+  return formatRenderedPage(payload.data, normalized, options.maxChars);
+}