@mrclrchtr/supi-web 11.1.0 → 11.3.0

package/CHANGELOG.md

@@ -0,0 +1,6 @@
+# Changelog
+
+## [11.3.0]
+
+- Provides its tools to SuPi Agent Runs and Reviewer Sessions when those packages request them.
+- Added package-specific release notes for web fetching, search, and library documentation tools.

package/README.md

@@ -14,14 +14,14 @@ Adds web fetch, public web search, and Context7 documentation tools to the [Pi c
 
 After installation, keep asking Pi normal questions. The agent can:
 
-- **Read public web pages as clean Markdown** — extract the main article instead of returning navigation, scripts, styles, and raw HTML.
+- **Read public web pages as best-effort Markdown** — prefer main content, but results may include page clutter or omit content.
 - **Prefer source Markdown when available** — detect Markdown responses and common Markdown siblings before converting HTML.
 - **Handle plain-text and source files** — wrap them in fenced code blocks with a language hint when the URL provides one.
 - **Look up current library docs** — search Context7 for the right project and version, then retrieve documentation focused on the task at hand.
 - **Search public web sources** — return source titles, URLs, and source excerpts without generating an answer.
 - **Protect the context window** — return small results inline, move long pages to temporary files, and preserve full output when model-visible text must be truncated.
 
-Relative page links and images become absolute by default, so the agent can follow them without reconstructing URLs.
+When converting HTML, relative links and image sources become absolute by default. Native Markdown is returned unchanged.
 
 ## Example requests
 
@@ -52,14 +52,14 @@ All four tools declare PI-native `outputSchema` contracts. Direct calls still re
 
 | Tool | Structured value |
 |---|---|
-| `web_fetch_md` | `mode: "inline"` with complete `markdown`, or `mode: "file"` with `filePath`; both include resolved `url`, `chars`, and `lines` |
+| `web_fetch_md` | `mode: "inline"` with the full returned `markdown`, or `mode: "file"` with `filePath`; both include source `url`, `chars`, and `lines` |
 | `web_search` | Complete `sources` (`title`, `url`, `excerpts`, optional `sourceDate`), `sourceCount`, and `excerptCount` |
 | `web_docs_search` | All `results` with IDs, names, complete descriptions/versions, scores, and snippet counts; `count` is the number returned by Context7 |
 | `web_docs_fetch` | `libraryId`, `chars`, `lines`, and either `format: "markdown"` with complete `markdown`, or `format: "snippets"` with SDK-normalized `snippets` (`title`, `content`, `source`) for `raw: true` |
 
 Inline/data results include `modelOutput.truncated` and, when text is truncated, `modelOutput.fullOutputPath`. These describe the **direct-call text**, not missing structured data. Library search also reports `modelOutput.omittedResults`: matches excluded by the table's top-10 row limit before line/byte truncation. Structured descriptions and version lists are not shortened. File-mode page fetches retain the existing disk delivery instead of duplicating the body in structured data.
 
-Results describe the evidence returned by the provider, not exhaustive coverage of the web or a library. Empty search arrays and empty snippet arrays are successful outcomes. Authentication, network, malformed web-search data, and cancellation failures still throw; codemode calls reject rather than returning a successful empty result. No tool currently returns a schema-defined failure or paginated result.
+Results describe the evidence returned by the provider, not exhaustive coverage of a page, the web, or a library. Empty search arrays and empty snippet arrays are successful outcomes. Authentication, network, malformed web-search data, and cancellation failures still throw; codemode calls reject rather than returning a successful empty result. No tool currently returns a schema-defined failure or paginated result.
 
 ```javascript
 const libraries = await tools.web_docs_search({
@@ -79,6 +79,21 @@ return [];
 
 Structured values can be larger than their text previews. Select the fields you need before printing them. The package's `./api` export provides `WebFetchMdOutput`, `WebSearchOutput`, `WebDocsSearchOutput`, and `WebDocsFetchOutput` types. Active Pi sessions need `/reload` or a restart to load extension changes.
 
+HTTP fetch failures include the requested URL and response status in the error message. For independent batch fetches, catch each call so a missing page does not stop the remaining work:
+
+```javascript
+for (const url of urls) {
+  try {
+    const page = await tools.web_fetch_md({ url, output_mode: "file" });
+    text({ url, filePath: page.filePath });
+  } catch (error) {
+    text({ url, error: String(error) });
+  }
+}
+```
+
+Uncaught failures stop the codemode script; `store()` writes are committed only when the script succeeds. The fetch tool does not discover relocated pages automatically.
+
 ## Web Search
 
 `web_search` uses the installed `bx` executable. The tool is enabled by default, but it is available only when `bx` is on `PATH`. Search freshness accepts `pd`, `pw`, `pm`, `py`, or an ordered `YYYY-MM-DDtoYYYY-MM-DD` range.
@@ -87,20 +102,57 @@ Structured values can be larger than their text previews. Select the fields you
 
 Install `bx` and configure it with the [Brave Search CLI instructions](https://github.com/brave/brave-search-cli). Start Pi from the shell where `command -v bx` finds the executable, then verify it with `bx --version`. `bx` handles its own credentials and configuration. Do not put credentials in Pi settings, prompts, or repository files.
 
-To enable or disable the tool, open `/supi-settings`, choose **Web Search**, and change **Web Search tool**. Run `/reload` after the change. The setting applies only when the extension starts or reloads. If `bx` was not available, install or configure it, make sure it is on `PATH`, and run `/reload`.
+To enable or disable the tool, open `/supi-settings`, choose **Web**, and change **Web Search tool**. Run `/reload` after the change. The setting applies only when the extension starts or reloads. If `bx` was not available, install or configure it, make sure it is on `PATH`, and run `/reload`.
 
 Search results are provider-reported source material. A source date can be a publication or modification date and is not a publication-date guarantee.
 
 ## How page fetching works
 
-`web_fetch_md` chooses the cleanest representation it can find:
+`web_fetch_md` tries these representations in order:
 
 1. Use Markdown returned directly by the server.
 2. Detect Markdown or plain text from the response content.
 3. Check common `.md`, `.markdown`, `index.md`, and `README.md` alternatives.
 4. For HTML, extract the primary readable content and convert it to GitHub-flavored Markdown.
 
-It follows redirects, removes scripts and styles, preserves useful structures such as headings, links, lists, code blocks, and tables, and makes relative links absolute by default.
+It follows redirects and, for HTML, removes scripts and styles while preserving headings, links, lists, code blocks, and tables.
+
+Native Markdown is returned unchanged, including relative links and code examples. For HTML conversion, `abs_links` defaults to `true`; set it to `false` to preserve original link and image destinations. Unsafe URL schemes are removed in either HTML mode.
+
+### Optional Firecrawl retrieval
+
+Direct fetching remains the default and needs no API key. To allow Firecrawl, open `/supi-settings`, choose **Web**, and enable **Firecrawl** in project or global scope. It is off by default. Changes apply to subsequent fetch calls without `/reload`; enabling it permits agent-selected retrieval, not automatic URL sharing or fallback.
+
+Export `FIRECRAWL_API_KEY` before starting Pi. The key stays in the environment, not settings. Enabling without a key saves the preference and shows a warning; direct fetching remains available.
+
+The agent can select one `fetch_mode`:
+
+| Mode | Behavior |
+|---|---|
+| `direct` (default) | Use the local negotiation and conversion pipeline; never contact Firecrawl |
+| `firecrawl_fallback` | Fetch directly, then use Firecrawl only after terminal HTTP 403 |
+| `firecrawl` | Skip direct fetching and use Firecrawl immediately, useful for JS-rendered pages or poor local extraction even after HTTP 200 |
+
+Both Firecrawl modes require the setting to be enabled. While disabled, explicit Firecrawl calls fail before DNS or HTTP requests.
+
+For a public page blocked with HTTP 403:
+
+```javascript
+const page = await tools.web_fetch_md({
+  url: "https://example.com/article",
+  fetch_mode: "firecrawl_fallback",
+  output_mode: "file",
+});
+return page;
+```
+
+For JS-rendered content, select `fetch_mode: "firecrawl"` instead. The key is read only when Firecrawl is actually needed and is not returned in results. Firecrawl receives the requested URL and may consume credits, including for cached results. The tool requests Markdown only, with no structured-extraction add-ons or client-side retries. Firecrawl manages its own retrieval pipeline.
+
+`firecrawl_fallback` runs only after the direct pipeline ends with HTTP 403—not for authentication errors (401), missing pages, rate limits, network failures, or cancellation. Neither mode retries Firecrawl failures through another provider. Both modes reject credential-bearing URLs, local/reserved hosts, and hosts resolving to non-public addresses before sharing them. Do not select Firecrawl for signed URLs or login-protected, private, or paywalled pages. These checks are not a sandbox or proof that a page is publicly accessible.
+
+Firecrawl results retain the requested source URL and include `retrieval.provider: "firecrawl"`, plus `cacheState` and `cachedAt` when supplied. Both inline/file responses disclose retrieval provenance. [Firecrawl caching](https://docs.firecrawl.dev/features/scrape#caching-and-freshness) allows cached content up to two days old by default; missing cache metadata does not establish freshness. Empty, unsuccessful, warning-bearing, and recognizable access-block responses are failures, not usable article content. A null or absent warning is accepted. Firecrawl can still return incomplete or noisy content, so inspect it before relying on it.
+
+`timeout_ms` applies to each direct HTTP request and to the complete Firecrawl attempt, including DNS validation and response reading; it is not a whole-tool deadline. The provider timeout is clamped to its supported 1–300 second range, while the local deadline still enforces shorter budgets. Cancellation is forwarded to Firecrawl. Native/Firecrawl Markdown remains unchanged, so `abs_links` applies only to local HTML conversion.
 
 ### Large pages
 
@@ -131,7 +183,7 @@ export CONTEXT7_API_KEY="ctx7sk-..."
 pi
 ```
 
-`web_fetch_md` does not use Context7 and needs no API key. Credentials are resolved when a documentation tool is called, not while the extension loads.
+`web_fetch_md` does not use Context7. Direct fetching needs no API key; optional Firecrawl retrieval uses `FIRECRAWL_API_KEY`. Credentials are resolved when needed, not while the extension loads.
 
 `raw: true` requests JSON and normalizes both provider code snippets and information snippets into `{ title, content, source }` objects. It does not expose the untouched REST envelope or provider ranking metadata. Default mode retains Context7's formatted text.
 
@@ -139,7 +191,8 @@ Requests make one attempt, forward cancellation, and add no SDK timeout. The exp
 
 ## Boundaries
 
-- `web_fetch_md` is for public `http://` and `https://` sources. It does not authenticate to login-protected, private, or paywalled pages.
-- It fetches HTTP responses and does not run a browser, page JavaScript, or interactive flows; client-rendered pages may expose little useful content.
+- `web_fetch_md` is intended for public `http://` and `https://` sources. It does not authenticate to login-protected, private, or paywalled pages.
+- Direct fetching is not network-isolated and does not enforce a public-address restriction. The private-target guard runs only before forwarding a URL to Firecrawl, in either Firecrawl mode.
+- Direct fetching does not run a browser, page JavaScript, or interactive flows; client-rendered pages may expose little useful content. Explicit Firecrawl selection delegates retrieval to that external service.
 - When the `gh` CLI is available, Pi is guided to use it instead for GitHub URLs.
 - Fetched pages and search excerpts are external, untrusted content. Treat them as source material, not as repository instructions.

package/node_modules/@mrclrchtr/supi-core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mrclrchtr/supi-core",
-  "version": "11.1.0",
+  "version": "11.3.0",
   "description": "Shared settings, configuration, reporting, and session infrastructure",
   "license": "MIT",
   "repository": {

package/node_modules/@mrclrchtr/supi-core/src/child-web-tools.ts

@@ -0,0 +1,110 @@
+// Session-provided Web tools for SuPi child sessions (Agent Runs and Reviewer Sessions).
+//
+// The loaded supi-web extension answers a synchronous request on the parent event bus.
+// Consumers never depend on supi-web; when it is absent or provides no tools, the
+// request returns undefined and the child simply lacks Web tools.
+
+import type { ExtensionAPI, ExtensionFactory } from "@earendil-works/pi-coding-agent";
+
+/** Web tools that child sessions may receive when the containing session provides them. */
+export const CHILD_WEB_TOOL_NAMES = [
+  "web_fetch_md",
+  "web_search",
+  "web_docs_search",
+  "web_docs_fetch",
+] as const;
+
+/** One session-provided Web tool name. */
+export type ChildWebToolName = (typeof CHILD_WEB_TOOL_NAMES)[number];
+
+/**
+ * Web tools currently callable in the containing session plus a child-only extension
+ * factory that registers them. The factory must not capture the parent session; it may
+ * capture plain data such as the containing session's configuration directory.
+ */
+export interface ChildWebToolProvision {
+  readonly toolNames: readonly ChildWebToolName[];
+  /** Register the requested subset of `toolNames` in a child session. */
+  readonly createFactory: (toolNames: readonly ChildWebToolName[]) => ExtensionFactory;
+}
+
+/** Plain containing-session data supplied with a child Web tool request. */
+export interface ChildWebToolsRequestContext {
+  /**
+   * Working directory whose SuPi configuration governs Web settings such as Firecrawl.
+   * Children with a different cwd (for example a frozen Review Workspace) keep this policy.
+   */
+  readonly configCwd: string;
+}
+
+type EventBusLike = Pick<ExtensionAPI["events"], "on" | "emit">;
+
+const REQUEST_CHANNEL = "supi:child-web-tools:request";
+const RESPONSE_CHANNEL = "supi:child-web-tools:response";
+
+interface ChildWebToolsRequest extends ChildWebToolsRequestContext {
+  readonly requestId: string;
+}
+
+interface ChildWebToolsResponse {
+  readonly requestId: string;
+  readonly provision: ChildWebToolProvision;
+}
+
+let nextRequestId = 0;
+
+/**
+ * Answer child Web tool requests on the parent event bus. The callback must be synchronous
+ * because Pi's event bus does not await listeners. Returns an unsubscribe function.
+ */
+export function provideChildWebTools(
+  events: EventBusLike,
+  resolve: (context: ChildWebToolsRequestContext) => ChildWebToolProvision | undefined,
+): () => void {
+  return events.on(REQUEST_CHANNEL, (data) => {
+    const request = data as Partial<ChildWebToolsRequest> | undefined;
+    if (typeof request?.requestId !== "string" || typeof request.configCwd !== "string") return;
+    const { requestId, configCwd } = request;
+    const provision = resolve({ configCwd });
+    if (!provision || provision.toolNames.length === 0) return;
+    events.emit(RESPONSE_CHANNEL, { requestId, provision } satisfies ChildWebToolsResponse);
+  });
+}
+
+/** Capture the Web tools the containing session currently provides, if any. */
+export function requestChildWebTools(
+  events: EventBusLike,
+  context: ChildWebToolsRequestContext,
+): ChildWebToolProvision | undefined {
+  const requestId = `child-web-${process.pid}-${++nextRequestId}`;
+  let provision: ChildWebToolProvision | undefined;
+  const unsubscribe = events.on(RESPONSE_CHANNEL, (data) => {
+    const response = data as Partial<ChildWebToolsResponse> | undefined;
+    if (response?.requestId === requestId && response.provision) provision = response.provision;
+  });
+  try {
+    events.emit(REQUEST_CHANNEL, { requestId, ...context } satisfies ChildWebToolsRequest);
+  } finally {
+    unsubscribe();
+  }
+  return provision;
+}
+
+/**
+ * Return the Web tools callable in a session under Pi's exposure rules: active `direct`
+ * tools, plus registered `codemode` and `deferred` tools. `model-only` and `hidden`
+ * tools are not callable.
+ */
+export function callableChildWebToolNames(
+  pi: Pick<ExtensionAPI, "getAllTools" | "getActiveTools">,
+): ChildWebToolName[] {
+  const active = new Set(pi.getActiveTools());
+  const exposures = new Map(
+    pi.getAllTools().map((tool) => [tool.name, tool.exposure ?? "direct"] as const),
+  );
+  return CHILD_WEB_TOOL_NAMES.filter((name) => {
+    const exposure = exposures.get(name);
+    if (exposure === "direct") return active.has(name);
+    return exposure === "codemode" || exposure === "deferred";
+  });
+}

package/node_modules/@mrclrchtr/supi-core/src/session.ts

@@ -1,5 +1,16 @@
 // supi-core session domain — session utilities and registries.
 
+export type {
+  ChildWebToolName,
+  ChildWebToolProvision,
+  ChildWebToolsRequestContext,
+} from "./child-web-tools.ts";
+export {
+  CHILD_WEB_TOOL_NAMES,
+  callableChildWebToolNames,
+  provideChildWebTools,
+  requestChildWebTools,
+} from "./child-web-tools.ts";
 export { createRegistry, createSessionStateRegistry } from "./registry-utils.ts";
 export type {
   SessionCapabilitySkill,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mrclrchtr/supi-web",
-  "version": "11.1.0",
+  "version": "11.3.0",
   "description": "Public web pages, web search sources, and focused Context7 docs",
   "license": "MIT",
   "repository": {
@@ -29,6 +29,7 @@
   ],
   "type": "module",
   "files": [
+    "CHANGELOG.md",
     "src/**/*.ts",
     "README.md"
   ],
@@ -38,7 +39,7 @@
     "@mozilla/readability": "^0.6.0",
     "turndown": "^7.2.0",
     "turndown-plugin-gfm": "^1.0.2",
-    "@mrclrchtr/supi-core": "11.1.0"
+    "@mrclrchtr/supi-core": "11.3.0"
   },
   "bundledDependencies": [
     "@mrclrchtr/supi-core"

package/src/child.ts

@@ -0,0 +1,48 @@
+/** Child-session provision of Web tools for SuPi Agent Runs and Reviewer Sessions. */
+
+import type { ExtensionAPI, ExtensionFactory } from "@earendil-works/pi-coding-agent";
+import {
+  type ChildWebToolName,
+  callableChildWebToolNames,
+  provideChildWebTools,
+} from "@mrclrchtr/supi-core/session";
+import { registerWebDocsFetchTool } from "./tool/web_docs_fetch/register.ts";
+import { WEB_DOCS_FETCH_TOOL_NAME } from "./tool/web_docs_fetch/spec.ts";
+import { registerWebDocsSearchTool } from "./tool/web_docs_search/register.ts";
+import { WEB_DOCS_SEARCH_TOOL_NAME } from "./tool/web_docs_search/spec.ts";
+import { registerWebFetchMdTool } from "./tool/web_fetch_md/register.ts";
+import { WEB_FETCH_MD_TOOL_NAME } from "./tool/web_fetch_md/spec.ts";
+import { registerWebSearchTool } from "./tool/web_search/register.ts";
+import { WEB_SEARCH_TOOL_NAME } from "./tool/web_search/spec.ts";
+
+type Registrar = (pi: ExtensionAPI, configCwd: string) => void;
+
+const registrars: Record<ChildWebToolName, Registrar> = {
+  [WEB_FETCH_MD_TOOL_NAME]: registerWebFetchMdTool,
+  [WEB_SEARCH_TOOL_NAME]: (pi) => registerWebSearchTool(pi),
+  [WEB_DOCS_SEARCH_TOOL_NAME]: (pi) => registerWebDocsSearchTool(pi),
+  [WEB_DOCS_FETCH_TOOL_NAME]: (pi) => registerWebDocsFetchTool(pi),
+};
+
+/**
+ * Build a child-only factory that registers the selected Web tools without settings,
+ * availability checks, or warnings. Availability was decided in the containing session,
+ * and Web settings keep resolving from `configCwd`, the containing session's directory.
+ */
+export function createChildWebToolsFactory(
+  toolNames: readonly ChildWebToolName[],
+  configCwd: string,
+): ExtensionFactory {
+  const selected = [...new Set(toolNames)];
+  return (pi) => {
+    for (const name of selected) registrars[name](pi, configCwd);
+  };
+}
+
+/** Answer child Web tool requests with the tools currently callable in this session. */
+export function registerChildWebToolProvider(pi: ExtensionAPI): void {
+  provideChildWebTools(pi.events, ({ configCwd }) => ({
+    toolNames: callableChildWebToolNames(pi),
+    createFactory: (toolNames) => createChildWebToolsFactory(toolNames, configCwd),
+  }));
+}

package/src/config.ts

@@ -6,15 +6,21 @@ export const WEB_CONFIG_SECTION = "web";
 /** Persisted key for the Web Search tool. */
 export const WEB_SEARCH_ENABLED_KEY = "webSearchEnabled";
 
+/** Persisted opt-in for external Firecrawl retrieval. */
+export const FIRECRAWL_ENABLED_KEY = "firecrawlEnabled";
+
 /** Web extension configuration. */
 export interface WebConfig extends Record<string, unknown> {
   /** Enable the web_search tool. */
   webSearchEnabled: boolean;
+  /** Allow explicitly selected Firecrawl fetches; ordinary calls still fetch directly. */
+  firecrawlEnabled: boolean;
 }
 
 /** Default Web extension configuration. */
 export const WEB_DEFAULTS: WebConfig = {
   webSearchEnabled: true,
+  firecrawlEnabled: false,
 };
 
 /** Load the merged Web extension configuration. */
@@ -25,5 +31,9 @@ export function loadWebConfig(cwd: string, homeDir?: string): WebConfig {
       typeof raw.webSearchEnabled === "boolean"
         ? raw.webSearchEnabled
         : WEB_DEFAULTS.webSearchEnabled,
+    firecrawlEnabled:
+      typeof raw.firecrawlEnabled === "boolean"
+        ? raw.firecrawlEnabled
+        : WEB_DEFAULTS.firecrawlEnabled,
   };
 }

package/src/convert.ts

@@ -7,12 +7,15 @@ import { JSDOM, VirtualConsole } from "jsdom";
 import Turndown from "turndown";
 import { guessLanguage } from "./fetch.ts";
 
+const ORIGINAL_URL_ATTRIBUTE = "data-supi-original-url";
+
 /**
  * Convert HTML to clean Markdown.
  *
  * @param html - Raw HTML string.
  * @param baseUrl - Base URL for resolving relative links.
- * @param options.absLinks - Whether to absolutize links and image sources.
+ * @param options.absLinks - Whether to absolutize HTML links and image sources. False preserves
+ * original destinations; unsafe URL schemes are removed in either mode.
  * @returns Clean Markdown string.
  */
 export async function htmlToMarkdown(
@@ -28,6 +31,7 @@ export async function htmlToMarkdown(
   }
 
   const doc = createDocument(html, baseUrl);
+  if (!absLinks) rememberOriginalUrls(doc);
 
   // Remove script/style/noscript
   for (const tag of ["script", "style", "noscript"]) {
@@ -49,9 +53,7 @@ export async function htmlToMarkdown(
   const contentDoc = createDocument(`<html><body>${contentHtml}</body></html>`, baseUrl);
   const body = contentDoc.body;
 
-  if (absLinks) {
-    absolutizeLinks(body, baseUrl);
-  }
+  normalizeLinks(body, baseUrl, absLinks);
 
   const turndown = await createTurndown();
   let markdown = turndown.turndown(body);
@@ -91,23 +93,37 @@ function createDocument(html: string, url: string): Document {
   return new JSDOM(html, { url, virtualConsole }).window.document;
 }
 
-function absolutizeLinks(root: Element, baseUrl: string): void {
-  for (const a of root.querySelectorAll("a[href]")) {
-    const resolved = resolveUrl(a.getAttribute("href") || "", baseUrl);
-    if (resolved) {
-      a.setAttribute("href", resolved);
-    } else {
-      a.removeAttribute("href");
-    }
+/**
+ * Readability rewrites URLs during extraction. DOM attributes survive its cloning, so
+ * remember original destinations there, discarding any input-supplied markers first.
+ */
+function rememberOriginalUrls(doc: Document): void {
+  for (const element of doc.querySelectorAll(`[${ORIGINAL_URL_ATTRIBUTE}]`)) {
+    element.removeAttribute(ORIGINAL_URL_ATTRIBUTE);
+  }
+  for (const element of doc.querySelectorAll("a[href], img[src]")) {
+    const attribute = element.localName === "a" ? "href" : "src";
+    element.setAttribute(ORIGINAL_URL_ATTRIBUTE, element.getAttribute(attribute) ?? "");
   }
-  for (const img of root.querySelectorAll("img[src]")) {
-    const resolved = resolveUrl(img.getAttribute("src") || "", baseUrl);
-    if (resolved) {
-      img.setAttribute("src", resolved);
+}
+
+/** Restore original destinations when requested and filter unsafe URLs in both modes. */
+function normalizeLinks(root: Element, baseUrl: string, absLinks: boolean): void {
+  for (const element of root.querySelectorAll("a, img")) {
+    const attribute = element.localName === "a" ? "href" : "src";
+    const original = absLinks ? null : element.getAttribute(ORIGINAL_URL_ATTRIBUTE);
+    const value = original ?? element.getAttribute(attribute);
+    if (value === null) continue;
+    const normalized = normalizeUrl(value, baseUrl, absLinks);
+    if (normalized) {
+      element.setAttribute(attribute, normalized);
     } else {
-      img.removeAttribute("src");
+      element.removeAttribute(attribute);
     }
   }
+  for (const element of root.querySelectorAll(`[${ORIGINAL_URL_ATTRIBUTE}]`)) {
+    element.removeAttribute(ORIGINAL_URL_ATTRIBUTE);
+  }
 }
 
 /** Dangerous URI schemes that must never be used in href/src attributes. */
@@ -121,7 +137,7 @@ function hasDangerousScheme(value: string): boolean {
   return DANGEROUS_SCHEMES.includes(scheme);
 }
 
-function resolveUrl(href: string, baseUrl: string): string {
+function normalizeUrl(href: string, baseUrl: string, absLinks: boolean): string {
   const trimmed = String(href || "").trim();
   if (
     !trimmed ||
@@ -140,7 +156,7 @@ function resolveUrl(href: string, baseUrl: string): string {
     if (resolved.protocol !== "http:" && resolved.protocol !== "https:") {
       return "";
     }
-    return resolved.toString();
+    return absLinks ? resolved.toString() : trimmed;
   } catch {
     return trimmed;
   }

package/src/extension.ts

@@ -1,7 +1,9 @@
+import { registerChildWebToolProvider } from "./child.ts";
 import docsExtension from "./docs.ts";
 import webExtension from "./web.ts";
 
 export default function extension(pi: Parameters<typeof webExtension>[0]): void {
   webExtension(pi);
   docsExtension(pi);
+  registerChildWebToolProvider(pi);
 }

package/src/fetch.ts

@@ -93,7 +93,7 @@ async function tryHeadNegotiation(
       signal,
     );
     if (!getRes.ok)
-      throw new FetchError(`Fetch failed: ${getRes.status} ${getRes.statusText}`, {
+      throw new FetchError(`Fetch failed for ${url}: ${getRes.status} ${getRes.statusText}`, {
         status: getRes.status,
       });
     return {
@@ -227,9 +227,12 @@ async function fetchFullTextResult(options: FetchFullTextOptions): Promise<Fetch
     options.signal,
   );
   if (!fullRes.ok)
-    throw new FetchError(`Fetch failed: ${fullRes.status} ${fullRes.statusText}`, {
-      status: fullRes.status,
-    });
+    throw new FetchError(
+      `Fetch failed for ${options.url}: ${fullRes.status} ${fullRes.statusText}`,
+      {
+        status: fullRes.status,
+      },
+    );
   return buildFetchResult(fullRes, options.url, options.contentType, options.kind);
 }
 
@@ -280,7 +283,9 @@ async function fetchAsHtml(
     signal,
   );
   if (!res.ok)
-    throw new FetchError(`Fetch failed: ${res.status} ${res.statusText}`, { status: res.status });
+    throw new FetchError(`Fetch failed for ${url}: ${res.status} ${res.statusText}`, {
+      status: res.status,
+    });
   return {
     url: res.url || url,
     text: await res.text(),

package/src/settings-registration.ts

@@ -5,6 +5,7 @@ import {
   type SettingsModule,
 } from "@mrclrchtr/supi-core/settings";
 import {
+  FIRECRAWL_ENABLED_KEY,
   loadWebConfig,
   WEB_CONFIG_SECTION,
   WEB_DEFAULTS,
@@ -14,11 +15,11 @@ import { isBxAvailable } from "./tool/web_search/bx.ts";
 
 const RELOAD_NOTICE = "Web Search setting saved. Run /reload to apply this change.";
 
-/** Register the Web Search setting without changing active tools. */
+/** Contribute Web settings; Firecrawl applies per call, Web Search requires reload. */
 export function registerWebSettings(pi: ExtensionAPI, homeDir?: string): void {
   const fixedSettings = defineConfigSettings({
     id: WEB_CONFIG_SECTION,
-    label: "Web Search",
+    label: "Web",
     section: WEB_CONFIG_SECTION,
     defaults: WEB_DEFAULTS,
     fields: [
@@ -28,6 +29,13 @@ export function registerWebSettings(pi: ExtensionAPI, homeDir?: string): void {
         label: "Web Search tool",
         description: "Enable the web_search tool when bx is available on PATH",
       },
+      {
+        kind: "boolean" as const,
+        key: FIRECRAWL_ENABLED_KEY,
+        label: "Firecrawl",
+        description:
+          "Allow agent-selected Firecrawl retrieval using FIRECRAWL_API_KEY; shares public URLs and may consume credits. Ordinary fetches stay direct.",
+      },
     ],
     ...(homeDir ? { homeDir } : {}),
   });
@@ -36,6 +44,20 @@ export function registerWebSettings(pi: ExtensionAPI, homeDir?: string): void {
     ...fixedSettings,
     apply: async (request) => {
       const result = await fixedSettings.apply(request);
+      if (request.fieldKey === FIRECRAWL_ENABLED_KEY) {
+        const missingKey =
+          loadWebConfig(request.cwd, homeDir).firecrawlEnabled &&
+          !process.env.FIRECRAWL_API_KEY?.trim();
+        return {
+          ...result,
+          notice: {
+            message: missingKey
+              ? "Firecrawl preference saved, but FIRECRAWL_API_KEY is missing. Export it before starting Pi to use Firecrawl."
+              : "Firecrawl setting saved. Applies to subsequent fetch calls; ordinary fetches stay direct.",
+            level: missingKey ? "warning" : "info",
+          },
+        };
+      }
       if (
         request.fieldKey === WEB_SEARCH_ENABLED_KEY &&
         loadWebConfig(request.cwd, homeDir).webSearchEnabled &&

package/src/tool/web_fetch_md/execute.ts

@@ -1,8 +1,9 @@
 import type { AgentToolUpdateCallback, ExtensionContext } from "@earendil-works/pi-coding-agent";
+import { loadWebConfig } from "../../config.ts";
 import { htmlToMarkdown, wrapAsCodeBlock } from "../../convert.ts";
-import { fetchWithNegotiation, isValidHttpUrl } from "../../fetch.ts";
+import { isValidHttpUrl } from "../../fetch.ts";
 import { writeTempFile } from "../../temp-file.ts";
-import { limitModelVisibleOutput } from "../result.ts";
+import { fetchPage } from "./fetch.ts";
 import {
   WEB_FETCH_INLINE_MAX_CHARS,
   type WebFetchMdInput,
@@ -10,13 +11,29 @@ import {
 } from "./input.ts";
 import { buildFileResult, buildInlineResult, type WebFetchResult } from "./result.ts";
 
-// biome-ignore lint/complexity/useMaxParams: pi ToolDefinition.execute signature
-export async function runWebFetch(
-  _toolCallId: string,
+/**
+ * Build web_fetch_md execution. `configCwd` fixes where Web settings (Firecrawl) are read;
+ * omitted, settings come from the calling session's cwd.
+ */
+export function createWebFetchExecute(configCwd?: string) {
+  // biome-ignore lint/complexity/useMaxParams: pi ToolDefinition.execute signature
+  return (
+    _toolCallId: string,
+    params: unknown,
+    signal: AbortSignal | undefined,
+    onUpdate: AgentToolUpdateCallback<Record<string, unknown>> | undefined,
+    ctx: ExtensionContext,
+  ): Promise<WebFetchResult> => fetchAsMarkdown(params, signal, onUpdate, configCwd ?? ctx.cwd);
+}
+
+/** Default web_fetch_md execution reading Web settings from the calling session's cwd. */
+export const runWebFetch = createWebFetchExecute();
+
+async function fetchAsMarkdown(
   params: unknown,
   signal: AbortSignal | undefined,
   onUpdate: AgentToolUpdateCallback<Record<string, unknown>> | undefined,
-  _ctx: ExtensionContext,
+  configCwd: string,
 ): Promise<WebFetchResult> {
   const input = (params ?? {}) as WebFetchMdInput;
   const url = String(input.url || "").trim();
@@ -33,22 +50,29 @@ export async function runWebFetch(
     details: { url, outputMode },
   });
 
-  const result = await fetchWithNegotiation(url, { timeoutMs, signal });
+  const result = await fetchPage(url, {
+    timeoutMs,
+    signal,
+    fetchMode: input.fetch_mode,
+    firecrawlEnabled: loadWebConfig(configCwd).firecrawlEnabled,
+  });
   const markdown = await resolveMarkdown(result, absLinks);
   const lines = markdown.split("\n").length;
   const chars = markdown.length;
-  const base = { chars, lines, url: result.url, outputMode };
+  const base = {
+    chars,
+    lines,
+    url: result.url,
+    outputMode,
+    ...(result.retrieval ? { retrieval: result.retrieval } : {}),
+  };
 
   if (shouldReturnFile(outputMode, chars)) {
     const filePath = await writeTempFile(markdown, "web-fetch-md", ".md");
     return buildFileResult(base, filePath);
   }
 
-  const output = await limitModelVisibleOutput(markdown, {
-    tempPrefix: "web-fetch-md",
-    suffix: ".md",
-  });
-  return buildInlineResult(base, markdown, output);
+  return buildInlineResult(base, markdown);
 }
 
 function shouldReturnFile(outputMode: WebFetchOutputMode, chars: number): boolean {

package/src/tool/web_fetch_md/fetch.ts

@@ -0,0 +1,253 @@
+import { lookup } from "node:dns/promises";
+import { BlockList, isIP } from "node:net";
+import { Type } from "typebox";
+import { Value } from "typebox/value";
+import {
+  FetchError,
+  type FetchOptions,
+  type FetchResult,
+  fetchWithNegotiation,
+} from "../../fetch.ts";
+import { WEB_FETCH_MODES, type WebFetchMode } from "./input.ts";
+import { FIRECRAWL_METADATA_MAX_CHARS, type FirecrawlRetrieval } from "./result-schema.ts";
+
+const FIRECRAWL_URL = "https://api.firecrawl.dev/v2/scrape";
+const providerPageSchema = Type.Object({
+  success: Type.Literal(true),
+  data: Type.Object({
+    markdown: Type.String(),
+    warning: Type.Optional(Type.Union([Type.String(), Type.Null()])),
+    metadata: Type.Optional(
+      Type.Object({
+        title: Type.Optional(Type.Union([Type.String(), Type.Array(Type.String())])),
+        statusCode: Type.Optional(Type.Integer()),
+        cacheState: Type.Optional(Type.String({ maxLength: FIRECRAWL_METADATA_MAX_CHARS })),
+        cachedAt: Type.Optional(Type.String({ maxLength: FIRECRAWL_METADATA_MAX_CHARS })),
+      }),
+    ),
+  }),
+});
+
+const nonPublicIPv4 = new BlockList();
+for (const [address, prefix] of [
+  ["0.0.0.0", 8],
+  ["10.0.0.0", 8],
+  ["100.64.0.0", 10],
+  ["127.0.0.0", 8],
+  ["169.254.0.0", 16],
+  ["172.16.0.0", 12],
+  ["192.0.0.0", 24],
+  ["192.0.2.0", 24],
+  ["192.88.99.0", 24],
+  ["192.168.0.0", 16],
+  ["198.18.0.0", 15],
+  ["198.51.100.0", 24],
+  ["203.0.113.0", 24],
+  ["224.0.0.0", 4],
+  ["240.0.0.0", 4],
+] as const)
+  nonPublicIPv4.addSubnet(address, prefix, "ipv4");
+
+// Conservatively permit global-unicast IPv6 only, excluding special-purpose ranges.
+// This also rejects loopback, link-local, unique-local, and IPv4-mapped literals.
+const globalIPv6 = new BlockList();
+globalIPv6.addSubnet("2000::", 3, "ipv6");
+const specialIPv6 = new BlockList();
+specialIPv6.addSubnet("2001::", 23, "ipv6");
+specialIPv6.addSubnet("2001:db8::", 32, "ipv6");
+specialIPv6.addSubnet("2002::", 16, "ipv6");
+specialIPv6.addSubnet("3fff::", 20, "ipv6");
+
+/** A page body with provenance when an external provider supplied it. */
+export interface PageFetchResult extends FetchResult {
+  retrieval?: FirecrawlRetrieval;
+}
+
+/** Honor the explicit route; Firecrawl requires a user opt-in even for fallback. */
+export async function fetchPage(
+  url: string,
+  options: FetchOptions & { fetchMode?: WebFetchMode; firecrawlEnabled: boolean },
+): Promise<PageFetchResult> {
+  const mode = options.fetchMode ?? "direct";
+  if (!WEB_FETCH_MODES.includes(mode)) {
+    throw new FetchError(
+      'Unsupported fetch_mode; select "direct", "firecrawl_fallback", or "firecrawl"',
+    );
+  }
+  if (mode !== "direct" && !options.firecrawlEnabled) {
+    throw new FetchError("Firecrawl is disabled. Enable Web > Firecrawl in /supi-settings.");
+  }
+  if (mode === "firecrawl") return fetchWithFirecrawl(url, options);
+
+  try {
+    return await fetchWithNegotiation(url, options);
+  } catch (error) {
+    options.signal?.throwIfAborted();
+    if (!(error instanceof FetchError) || error.status !== 403) throw error;
+    if (mode === "direct") {
+      const setupHint = options.firecrawlEnabled
+        ? ""
+        : "requires Firecrawl enabled in /supi-settings and FIRECRAWL_API_KEY; ";
+      throw new FetchError(
+        `${error.message}. For a public page, retry with fetch_mode: "firecrawl_fallback" (${setupHint}shares the URL and may consume credits).`,
+        { status: 403, cause: error },
+      );
+    }
+  }
+  return fetchWithFirecrawl(url, options);
+}
+
+async function fetchWithFirecrawl(url: string, options: FetchOptions): Promise<PageFetchResult> {
+  const key = process.env.FIRECRAWL_API_KEY?.trim();
+  if (!key) throw new FetchError("Firecrawl requires FIRECRAWL_API_KEY");
+  const timeoutMs = options.timeoutMs ?? 30_000;
+  const timeout = AbortSignal.timeout(timeoutMs);
+  const signal = options.signal ? AbortSignal.any([options.signal, timeout]) : timeout;
+  try {
+    return await abortable(requestFirecrawl(url, { key, timeoutMs, signal }), signal);
+  } catch (error) {
+    options.signal?.throwIfAborted();
+    if (timeout.aborted) {
+      // biome-ignore lint/style/useErrorCause: retain the timeout reason, not possibly credential-bearing provider errors
+      throw new FetchError(`Firecrawl timed out after ${timeoutMs}ms`, {
+        cause: timeout.reason,
+      });
+    }
+    if (error instanceof FetchError) throw error;
+    // biome-ignore lint/style/useErrorCause: transport/parse errors may reflect Authorization; never retain them as causes
+    throw new FetchError("Firecrawl request failed");
+  }
+}
+
+async function requestFirecrawl(
+  url: string,
+  options: { key: string; timeoutMs: number; signal: AbortSignal },
+): Promise<PageFetchResult> {
+  options.signal.throwIfAborted();
+  const target = await publicTarget(url);
+  options.signal.throwIfAborted();
+  const response = await fetch(FIRECRAWL_URL, {
+    method: "POST",
+    redirect: "error",
+    signal: options.signal,
+    headers: { Authorization: `Bearer ${options.key}`, "Content-Type": "application/json" },
+    body: JSON.stringify({
+      url: target.href,
+      formats: ["markdown"],
+      onlyMainContent: true,
+      // The provider requires >= 1s; the local signal still enforces a shorter budget.
+      timeout: Math.max(1_000, Math.min(options.timeoutMs, 300_000)),
+    }),
+  });
+  if (!response.ok) {
+    throw new FetchError(`Firecrawl failed: HTTP ${response.status}`, {
+      status: response.status,
+    });
+  }
+  const body: unknown = await response.json();
+  if (!Value.Check(providerPageSchema, body)) {
+    throw new FetchError("Firecrawl returned an unsuccessful or invalid response");
+  }
+  const { markdown, metadata, warning } = body.data;
+  if (
+    [markdown, metadata?.cacheState, metadata?.cachedAt].some((value) =>
+      value?.includes(options.key),
+    )
+  ) {
+    throw new FetchError("Firecrawl returned credential-bearing content");
+  }
+  if (!markdown.trim() || warning?.trim()) {
+    throw new FetchError(
+      "Firecrawl returned empty content or a warning; content may be incomplete",
+    );
+  }
+  const status = metadata?.statusCode;
+  if (
+    (status !== undefined && status !== 304 && (status < 200 || status >= 300)) ||
+    isAccessBlock(metadata?.title, markdown)
+  ) {
+    throw new FetchError("Firecrawl returned an error or access-block page, not source content");
+  }
+  // Keep only allowlisted provenance, never the raw provider envelope or credentials.
+  return {
+    url,
+    text: markdown,
+    contentType: "text/markdown",
+    isMarkdown: true,
+    isPlainText: false,
+    retrieval: {
+      provider: "firecrawl",
+      ...(metadata?.cacheState ? { cacheState: metadata.cacheState } : {}),
+      ...(metadata?.cachedAt ? { cachedAt: metadata.cachedAt } : {}),
+    },
+  };
+}
+
+/** Validate before disclosing a URL. DNS checks are conservative, not a sandbox for the reader. */
+async function publicTarget(url: string): Promise<URL> {
+  const target = new URL(url);
+  const hostname = target.hostname.replace(/^\[|\]$/g, "").replace(/\.$/, "");
+  if (target.username || target.password) {
+    throw new FetchError("Firecrawl does not accept URLs containing credentials");
+  }
+  if (isIP(hostname)) {
+    if (!isPublicAddress(hostname)) throw new FetchError("Firecrawl requires a public URL");
+  } else {
+    if (
+      !hostname.includes(".") ||
+      /(?:^|\.)(?:localhost|local|localdomain|internal|test|invalid|example|onion|lan|home|corp)$|\.home\.arpa$/i.test(
+        hostname,
+      )
+    ) {
+      throw new FetchError("Firecrawl requires a public hostname");
+    }
+    const addresses = await lookup(hostname, { all: true, verbatim: true });
+    if (!addresses.length || addresses.some(({ address }) => !isPublicAddress(address))) {
+      throw new FetchError("Firecrawl requires exclusively public DNS addresses");
+    }
+  }
+  // Fragments are not part of HTTP requests and may contain client-only information.
+  target.hash = "";
+  return target;
+}
+
+function isPublicAddress(address: string): boolean {
+  if (isIP(address) === 4) return !nonPublicIPv4.check(address, "ipv4");
+  return (
+    isIP(address) === 6 && globalIPv6.check(address, "ipv6") && !specialIPv6.check(address, "ipv6")
+  );
+}
+
+function isAccessBlock(title: string | string[] | undefined, markdown: string): boolean {
+  const titles = Array.isArray(title) ? title : [title];
+  return (
+    titles.some((value) =>
+      /^(?:attention required!?|just a moment[.!…]*|access denied|403 forbidden|verify (?:you are|you're) human|checking your browser)(?:\s*[-|].*)?$/i.test(
+        value?.trim() ?? "",
+      ),
+    ) ||
+    // Only a page-leading heading signals a block; documentation may discuss these errors.
+    /^#{1,2}\s+(?:Sorry, you have been blocked|Access denied|403 Forbidden|Just a moment\.{0,3})\s*$/i.test(
+      markdown.trimStart().split("\n", 1)[0] ?? "",
+    )
+  );
+}
+
+/** Bound DNS, HTTP, and response-body work; an outstanding lookup cannot trigger a later request. */
+function abortable<T>(pending: Promise<T>, signal: AbortSignal): Promise<T> {
+  return new Promise((resolve, reject) => {
+    const abort = () => reject(signal.reason);
+    signal.addEventListener("abort", abort, { once: true });
+    pending.then(
+      (value) => {
+        signal.removeEventListener("abort", abort);
+        resolve(value);
+      },
+      (error) => {
+        signal.removeEventListener("abort", abort);
+        reject(error);
+      },
+    );
+    if (signal.aborted) abort();
+  });
+}

package/src/tool/web_fetch_md/guidance.ts

@@ -1,7 +1,7 @@
 import type { WebToolPromptSurface } from "../tool-specs.ts";
 import { WEB_FETCH_INLINE_MAX_CHARS } from "./spec.ts";
 
-export const toolDescription = `Fetch a public http(s) URL as Markdown. Not for login or private pages. Use gh for GitHub URLs when available. output_mode auto returns up to ${WEB_FETCH_INLINE_MAX_CHARS.toLocaleString()} characters inline and otherwise returns a temporary-file path; file always returns a temporary-file path. Links are absolute by default.`;
+export const toolDescription = `Fetch public http(s) pages as best-effort Markdown, directly or via opt-in Firecrawl for JS-rendered pages. Not for login, private, or paywalled pages. Direct fetching is not network-isolated. Use gh for GitHub URLs when available. Auto returns up to ${WEB_FETCH_INLINE_MAX_CHARS.toLocaleString()} chars inline, otherwise a temp file. Native Markdown is unchanged; HTML links default to absolute.`;
 
 export const promptSnippet = "fetch a public URL as Markdown";
 

package/src/tool/web_fetch_md/input.ts

@@ -5,7 +5,6 @@
 // execute binding, and a spec <-> execute cycle would risk undefined
 // bindings depending on module evaluation order.
 
-import { StringEnum } from "@earendil-works/pi-ai";
 import type { Static } from "typebox";
 import { Type } from "typebox";
 import { FETCH_TIMEOUT_MAX_MS } from "../../fetch.ts";
@@ -14,7 +13,12 @@ export const WEB_FETCH_INLINE_MAX_CHARS = 15_000;
 export const WEB_FETCH_OUTPUT_MODES = ["auto", "inline", "file"] as const;
 export type WebFetchOutputMode = (typeof WEB_FETCH_OUTPUT_MODES)[number];
 
-const OutputModeEnum = StringEnum(WEB_FETCH_OUTPUT_MODES, {
+/** Explicit retrieval routes; enabling Firecrawl never changes the default route. */
+export const WEB_FETCH_MODES = ["direct", "firecrawl_fallback", "firecrawl"] as const;
+export type WebFetchMode = (typeof WEB_FETCH_MODES)[number];
+
+const OutputModeEnum = Type.Enum(WEB_FETCH_OUTPUT_MODES, {
+  type: "string",
   default: "auto",
   description: "auto, inline, or file output",
 });
@@ -23,10 +27,23 @@ export const webFetchMdParameters = Type.Object(
   {
     url: Type.String({ description: "Public http(s) URL" }),
     output_mode: Type.Optional(OutputModeEnum),
-    abs_links: Type.Optional(Type.Boolean({ description: "Absolute links/images", default: true })),
+    fetch_mode: Type.Optional(
+      Type.Enum(WEB_FETCH_MODES, {
+        type: "string",
+        default: "direct",
+        description:
+          "direct fetches locally; firecrawl_fallback uses Firecrawl only after HTTP 403; firecrawl skips direct fetching for JS-heavy or poorly extracted pages. Firecrawl requires opt-in via /supi-settings and FIRECRAWL_API_KEY, shares the URL and may consume credits. Never use it for signed, login-protected, private, or paywalled URLs.",
+      }),
+    ),
+    abs_links: Type.Optional(
+      Type.Boolean({
+        description: "Absolutize HTML links/images; false preserves original destinations.",
+        default: true,
+      }),
+    ),
     timeout_ms: Type.Optional(
       Type.Integer({
-        description: "Fetch timeout (ms)",
+        description: "Timeout per direct request and for the complete Firecrawl attempt (ms)",
         default: 30_000,
         minimum: 0,
         maximum: FETCH_TIMEOUT_MAX_MS,

package/src/tool/web_fetch_md/register.ts

@@ -1,13 +1,18 @@
 import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import { createWebFetchExecute } from "./execute.ts";
 import { getWebFetchPromptSurface } from "./guidance.ts";
 import { renderWebFetchCall, renderWebFetchResult } from "./render.ts";
 import { webFetchMdSpec } from "./spec.ts";
 
-/** Register the web_fetch_md tool. */
-export function registerWebFetchMdTool(pi: ExtensionAPI): void {
+/**
+ * Register the web_fetch_md tool. `configCwd` fixes where Web settings are read, so a child
+ * session keeps its containing session's Firecrawl policy.
+ */
+export function registerWebFetchMdTool(pi: ExtensionAPI, configCwd?: string): void {
   const surface = getWebFetchPromptSurface();
   pi.registerTool({
     ...webFetchMdSpec,
+    ...(configCwd ? { execute: createWebFetchExecute(configCwd) } : {}),
     description: surface.description,
     promptSnippet: surface.promptSnippet,
     promptGuidelines: surface.promptGuidelines,

package/src/tool/web_fetch_md/render.ts

@@ -8,7 +8,8 @@ export function renderWebFetchCall(args: unknown, theme: Theme) {
   const input = (args ?? {}) as WebFetchMdInput;
   const url = typeof input.url === "string" ? input.url : "";
   const outputMode = typeof input.output_mode === "string" ? input.output_mode : undefined;
-  return renderToolCall(WEB_FETCH_MD_TOOL_NAME, url, theme, outputMode);
+  const annotation = [outputMode, input.fetch_mode].filter(Boolean).join(", ");
+  return renderToolCall(WEB_FETCH_MD_TOOL_NAME, url, theme, annotation || undefined);
 }
 
 /** Transcript renderer for web_fetch_md tool results. */
@@ -55,6 +56,7 @@ function buildWebFetchSummary(
         "dim",
         ` (${details.chars.toLocaleString()} chars, ${details.lines.toLocaleString()} lines)`,
       ),
+      theme.fg("dim", retrievalSummary(details)),
     ].join("");
   }
 
@@ -66,9 +68,17 @@ function buildWebFetchSummary(
     ),
   ].join("");
 
+  summary += theme.fg("dim", retrievalSummary(details));
+
   if (details.truncation?.truncated) {
     summary += theme.fg("warning", " [truncated]");
   }
 
   return summary;
 }
+
+function retrievalSummary(details: WebFetchDetails): string {
+  if (!details.retrieval) return "";
+  const cache = details.retrieval.cacheState ? `; cache: ${details.retrieval.cacheState}` : "";
+  return ` via Firecrawl${cache}`;
+}

package/src/tool/web_fetch_md/result-schema.ts

@@ -1,8 +1,21 @@
 import { type Static, Type } from "typebox";
 import { modelOutputSchema } from "../result-schema.ts";
 
+/** Bound external provenance used in model-visible file summaries. */
+export const FIRECRAWL_METADATA_MAX_CHARS = 128;
+
+const firecrawlRetrievalSchema = Type.Object({
+  provider: Type.Literal("firecrawl"),
+  cacheState: Type.Optional(Type.String({ maxLength: FIRECRAWL_METADATA_MAX_CHARS })),
+  cachedAt: Type.Optional(Type.String({ maxLength: FIRECRAWL_METADATA_MAX_CHARS })),
+});
+
+/** External reader provenance; missing cache metadata does not imply a fresh fetch. */
+export type FirecrawlRetrieval = Static<typeof firecrawlRetrievalSchema>;
+
 const metrics = {
-  url: Type.String({ description: "Resolved source URL." }),
+  url: Type.String({ description: "Source URL; the requested URL when using Firecrawl." }),
+  retrieval: Type.Optional(firecrawlRetrievalSchema),
   chars: Type.Integer({ minimum: 0 }),
   lines: Type.Integer({ minimum: 0 }),
 };

package/src/tool/web_fetch_md/result.ts

@@ -1,7 +1,7 @@
 import type { AgentToolResult, TruncationResult } from "@earendil-works/pi-coding-agent";
-import { type ModelVisibleOutput, modelOutputMetadata } from "../result.ts";
+import { limitModelVisibleOutput, modelOutputMetadata } from "../result.ts";
 import type { WebFetchOutputMode } from "./input.ts";
-import type { WebFetchMdOutput } from "./result-schema.ts";
+import type { FirecrawlRetrieval, WebFetchMdOutput } from "./result-schema.ts";
 
 /** Final fetch result requires both native representations. */
 export type WebFetchResult = AgentToolResult<WebFetchDetails> & {
@@ -13,6 +13,7 @@ export interface WebFetchDetails extends Record<string, unknown> {
   lines: number;
   url: string;
   outputMode: WebFetchOutputMode;
+  retrieval?: FirecrawlRetrieval;
   filePath?: string;
   truncation?: TruncationResult;
   fullOutputPath?: string;
@@ -24,6 +25,7 @@ export interface WebFetchResultBase {
   lines: number;
   url: string;
   outputMode: WebFetchOutputMode;
+  retrieval?: FirecrawlRetrieval;
 }
 
 /** Assemble the file-mode result for one fetch. */
@@ -32,7 +34,7 @@ export function buildFileResult(base: WebFetchResultBase, filePath: string): Web
     content: [
       {
         type: "text",
-        text: `Content written to ${filePath} (${base.chars.toLocaleString()} chars, ${base.lines.toLocaleString()} lines). Use the read tool to access it.`,
+        text: `${retrievalNotice(base)}Content written to ${filePath} (${base.chars.toLocaleString()} chars, ${base.lines.toLocaleString()} lines). Use the read tool to access it.`,
       },
     ],
     structuredContent: {
@@ -41,17 +43,21 @@ export function buildFileResult(base: WebFetchResultBase, filePath: string): Web
       chars: base.chars,
       lines: base.lines,
       filePath,
+      ...(base.retrieval ? { retrieval: base.retrieval } : {}),
     },
     details: { ...base, filePath },
   };
 }
 
 /** Assemble the inline-mode result for one fetch. */
-export function buildInlineResult(
+export async function buildInlineResult(
   base: WebFetchResultBase,
   markdown: string,
-  output: ModelVisibleOutput,
-): WebFetchResult {
+): Promise<WebFetchResult> {
+  const output = await limitModelVisibleOutput(retrievalNotice(base) + markdown, {
+    tempPrefix: "web-fetch-md",
+    suffix: ".md",
+  });
   return {
     content: [{ type: "text", text: output.text }],
     structuredContent: {
@@ -61,6 +67,7 @@ export function buildInlineResult(
       lines: base.lines,
       markdown,
       modelOutput: modelOutputMetadata(output),
+      ...(base.retrieval ? { retrieval: base.retrieval } : {}),
     },
     details: {
       ...base,
@@ -69,3 +76,14 @@ export function buildInlineResult(
     },
   };
 }
+
+function retrievalNotice(base: WebFetchResultBase): string {
+  if (!base.retrieval) return "";
+  const { cacheState, cachedAt } = base.retrieval;
+  const cache = cacheState ? `cache: ${cacheState}` : "cache status unknown";
+  const timestamp = cachedAt ? `; cached at ${cachedAt}` : "";
+  // Provider fields are schema-bounded; shorten URL display without losing the full structured URL.
+  const source =
+    base.url.length > 1024 ? `${base.url.slice(0, 1024)}… [source URL shortened]` : base.url;
+  return `Retrieved via Firecrawl (${cache}${timestamp}). Source: ${source}\n\n`;
+}