@zeldrisho/pi-web-fetch 0.7.1 → 0.8.1

package/CHANGELOG.md

@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.8.1] - 2026-09-07
+
+### Security
+
+- redact URL user information and common credential-bearing query parameters from progress messages, rendered calls, content source labels, and result URL metadata
+
+## [0.8.0] - 2026-09-01
+
+### Added
+
+- Add optional deterministic query-focused extraction that selects matching source sections from the complete cached document, preserves source order, supports continuation over the focused view, and reports honest selection evidence in `details.focus`
+- Add bounded, fence-aware document outlines to result details, including total words, section word counts, omitted-heading counts, and conservative fallback headings for structure lost during extraction
+- Add an opt-in live extraction-quality corpus that checks expected markers and content size while reporting extractor choice and latency
+- Revalidate stale cache entries with `ETag` and `Last-Modified`, reporting cache hits, HTTP 304 revalidations, and misses separately
+- Add explicit bounded diagnostics for JavaScript requirements, bot walls, consent interstitials, and sparse extraction
+- Expose bounded normalized internal and external links with anchor text and omitted counts without fetching linked pages
+- Coordinate requests per origin and retry `429`/`503` responses with bounded `Retry-After` handling or jittered backoff
+- Add focused-ranking baselines for heading, phrase, and stemming experiments
+
+### Fixed
+
+- Resolve relative Open Graph, Twitter, and canonical page URLs before Defuddle metadata extraction so successful GitHub release fetches no longer leak `ERR_INVALID_URL` warnings into Pi's TUI
+- Remove leaked style elements, standalone CSS rules, and block at-rules from extracted Markdown while preserving fenced stylesheet examples
+- Expand extraction regressions for GFM tables, malformed issue markup, and content-rich app shells
+
 ## [0.7.1] - 2026-08-28
 
 ### Fixed
@@ -110,7 +135,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Add bounded public page fetching ([b979496](https://github.com/zeldrisho/pi-packages/commit/b979496b32de1cead172ba570307e4a4a7b3421d))
 
-[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.7.1...HEAD
+[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.8.1...HEAD
+[0.8.1]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.8.0...pi-web-fetch-v0.8.1
+[0.8.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.7.1...pi-web-fetch-v0.8.0
 [0.7.1]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.7.0...pi-web-fetch-v0.7.1
 [0.7.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.6.1...pi-web-fetch-v0.7.0
 [0.6.1]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.6.0...pi-web-fetch-v0.6.1

package/README.md

@@ -20,6 +20,8 @@ The `web_fetch` tool accepts public HTTP and HTTPS URLs. It supports textual con
 
 For safety, the tool blocks URLs containing credentials, local hostnames, private or reserved network targets, unsafe redirects, raw responses larger than 5 MiB, and unsupported content types. The `maxCharacters` parameter controls returned Markdown length; it does not change the raw download limit.
 
+For long pages, the optional `query` parameter returns a deterministic focused view containing source sections that match the query. Filtering is performed locally after the complete page has been fetched and cached; it does not use an LLM or alter the cached document. Matching sections retain source order, and `details.focus` reports matched, total, and omitted section counts plus original and focused character counts. Continuation offsets apply to the focused view when `query` is present. Omit `query` to read the complete document, especially when verifying context or when no sections match.
+
 In Pi's interactive UI, fetched content uses Pi's standard collapsed preview; use the configured tool-expansion shortcut (`Ctrl+O` by default) to show all visible tool output. Output sent to the agent remains bounded: each call returns at most `maxCharacters` characters of extracted Markdown (default 6,000) and is additionally capped by Pi's 2,000-line / 50 KiB tool-output limit, so fetching cannot bloat the conversation context. The `offset` parameter is a character offset into extracted content, not a byte range into the remote response. When a result is truncated, call the tool again with the returned `nextOffset` as `offset` to continue reading. Fetched and extracted pages are cached in byte-bounded memory for a limited time so continuation requests can reuse the same content. Concurrent requests for the same URL share one fetch; cancelling one caller does not cancel work still needed by another.
 
 Every result includes `details.truncation`. Complete output reports `{ truncated: false, strategy: "none" }`. Truncated output reports `strategy: "continuation"` and a valid `nextOffset`. The existing top-level `details.truncated` and `details.nextOffset` fields remain available.
@@ -54,9 +56,17 @@ Directory and tree listings (`https://github.com/<owner>/<repo>/tree/...`) are a
 
 ### Caching and evidence
 
-Fetched and extracted pages are cached in byte-bounded memory and also persisted to a private, cross-session disk cache (24h TTL, files created `0700`/`0600`) so identical requests reuse the same content across Pi sessions. Concurrent requests for the same URL share one fetch; cancelling one caller does not cancel work still needed by another.
+Fetched and extracted pages are cached in byte-bounded memory and also persisted to a private, cross-session disk cache (files created `0700`/`0600`). Entries are fresh for 24 hours and retained stale for bounded conditional revalidation. Stale entries use their `ETag` and `Last-Modified` validators through the same URL validation, DNS pinning, redirect, timeout, and response-limit boundary as an initial fetch. `details.cacheStatus` distinguishes `hit`, `revalidated` (HTTP 304), and `miss`; the compatibility `cached` flag is true for hits and revalidations.
+
+Requests are coordinated per origin. Responses with status `429` or `503` are retried at most twice, honoring a bounded `Retry-After` value or using bounded jittered exponential backoff. Each caller retains independent cancellation while queued or backing off.
+
+Each result includes honest-evidence `details`: `requestedUrl` and `finalUrl` (after any rewrite or redirect), `contentKind` (a coarse classification such as `article`, `code-file`, `repository-readme`, `raw-text`, or `markup-shell`), and `confidence` (`high`/`medium`/`low`). For HTML, `details.extractionDiagnostics` separately reports JavaScript requirements, bot walls, consent interstitials, and sparse extraction; `shellSuspected` remains only as a deprecated compatibility summary.
+
+`details.links` exposes at most 16 normalized internal and 16 external HTTP(S) links with bounded anchor text and omitted counts. Unsafe schemes and credential-bearing URLs are rejected. Links are evidence from the fetched document only; `web_fetch` never fetches linked pages implicitly.
+
+`details.outline` provides bounded document-shape metadata: total words, heading count, the first 12 headings with section word counts, and the number of omitted headings. It ignores headings inside fenced code and conservatively infers short title lines only when extraction produced no Markdown headings. Heading text is untrusted remote content and is capped before being exposed.
 
-Each result includes honest-evidence `details`: `requestedUrl` and `finalUrl` (after any rewrite or redirect), `contentKind` (a coarse classification such as `article`, `code-file`, `repository-readme`, `raw-text`, or `markup-shell`), `shellSuspected` (true when the page looks like an app shell, bot wall, or consent page), and `confidence` (`high`/`medium`/`low`) derived from the extractor, content length, and `shellSuspected`.
+Extraction regressions cover tables, malformed issue-like markup, and app-shell false positives. Before changing focused-section ranking, run `vp exec jiti scripts/benchmark-focus.ts` to record heading, phrase, and stemming baselines.
 
 ## Uninstall
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zeldrisho/pi-web-fetch",
-  "version": "0.7.1",
+  "version": "0.8.1",
   "description": "Pi extension for secure, bounded public web page fetching and Markdown extraction",
   "keywords": [
     "pi",

package/src/cache.ts

@@ -40,6 +40,12 @@ export function resolveCachePath(directory: string, keyPath: string): string {
   return full;
 }
 
+/**
+ * Encodes an expiration timestamp as an 8-byte big-endian buffer.
+ *
+ * @param expiresAt - Expiration timestamp in milliseconds
+ * @returns 8-byte buffer containing the encoded timestamp
+ */
 function encodeExpiresAt(expiresAt: number): Uint8Array {
   const out = new Uint8Array(8);
   let value = BigInt(Math.round(expiresAt));
@@ -50,6 +56,12 @@ function encodeExpiresAt(expiresAt: number): Uint8Array {
   return out;
 }
 
+/**
+ * Decodes an 8-byte big-endian buffer to an expiration timestamp.
+ *
+ * @param bytes - 8-byte buffer containing the encoded timestamp
+ * @returns Expiration timestamp in milliseconds
+ */
 function decodeExpiresAt(bytes: Uint8Array): number {
   let value = 0n;
   for (let index = 0; index < 8; index += 1) value = (value << 8n) | BigInt(bytes[index]);
@@ -69,14 +81,30 @@ export class ExpiringLruCache<K, V> {
     readonly persistence?: CachePersistence<K, V>,
   ) {}
 
+  /**
+   * Gets the total byte size of all cached values.
+   *
+   * @returns Current aggregate byte size
+   */
   get byteSize(): number {
     return this.#byteSize;
   }
 
+  /**
+   * Gets the number of cached entries.
+   *
+   * @returns Current entry count
+   */
   get size(): number {
     return this.#entries.size;
   }
 
+  /**
+   * Retrieves a value from the cache, loading from disk if needed.
+   *
+   * @param key - Cache key
+   * @returns Cached value or undefined if not found or expired
+   */
   get(key: K): V | undefined {
     const entry = this.#entries.get(key);
     if (entry) {
@@ -95,6 +123,14 @@ export class ExpiringLruCache<K, V> {
     return undefined;
   }
 
+  /**
+   * Stores a value in the cache with an expiration time.
+   *
+   * @param key - Cache key
+   * @param value - Value to store
+   * @param expiresAt - Expiration timestamp in milliseconds
+   * @returns True if the value was stored, false if it was too large
+   */
   set(key: K, value: V, expiresAt: number): boolean {
     this.#delete(key);
     const size = this.sizeOf(value);

package/src/content.ts

@@ -1,4 +1,5 @@
 import { DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES } from "@earendil-works/pi-coding-agent";
+import type { ExtractedLinks, ExtractionDiagnostics } from "./evidence";
 
 const CONTENT_LINE_BUDGET = Math.max(1, DEFAULT_MAX_LINES - 10);
 const CONTENT_BYTE_BUDGET = Math.max(1_024, DEFAULT_MAX_BYTES - 2_048);
@@ -16,8 +17,16 @@ export interface CompleteDocument {
   markdown: string;
   title?: string;
   extractor: "defuddle" | "basic" | "raw";
-  /** True when the page appears to be an app shell, bot wall, or consent page. */
+  /** @deprecated Use extractionDiagnostics for the individual quality signals. */
   shellSuspected: boolean;
+  /** Explicit bounded extraction-quality evidence for HTML documents. */
+  extractionDiagnostics?: ExtractionDiagnostics;
+  /** Bounded normalized links extracted from this document; linked pages were not fetched. */
+  links?: ExtractedLinks;
+  /** HTTP validators retained for stale-cache revalidation. */
+  validators?: { etag?: string; lastModified?: string };
+  /** Time at which this representation was fetched or successfully revalidated. */
+  cachedAt?: number;
   /** True when this document is the site's /llms.txt served instead of a low-quality page. */
   llmsTxtFallback?: boolean;
   /** Set when the site publishes a usable /llms.txt index and this document is not that index. */

package/src/evidence.ts

@@ -0,0 +1,118 @@
+import { parseHTML } from "linkedom";
+
+const MAX_LINKS = 16;
+const MAX_ANCHOR_CHARACTERS = 120;
+
+/** Bounded, machine-readable reasons that an HTML extraction may be incomplete. */
+export interface ExtractionDiagnostics {
+  javascriptRequired: boolean;
+  botWall: boolean;
+  consentInterstitial: boolean;
+  sparseExtraction: boolean;
+  rawCharacters: number;
+  extractedCharacters: number;
+}
+
+/** One normalized link found in the fetched document. */
+export interface ExtractedLink {
+  url: string;
+  anchorText: string;
+}
+
+/** Bounded links from the fetched document. Linked pages are never fetched implicitly. */
+export interface ExtractedLinks {
+  internal: ExtractedLink[];
+  external: ExtractedLink[];
+  omittedInternal: number;
+  omittedExternal: number;
+}
+
+const JAVASCRIPT_MARKERS = [/please\s+enable\s+javascript/i, /javascript\s+(?:is\s+)?required/i];
+const BOT_MARKERS = [
+  /are\s+you\s+a\s+robot/i,
+  /verify\s+you\s+are\s+human/i,
+  /checking\s+your\s+browser/i,
+  /<title>\s*just\s+a\s+moment/i,
+];
+const CONSENT_MARKERS = [
+  /manage\s+(?:your\s+)?consent/i,
+  /your\s+(?:privacy\s+)?consent/i,
+  /consent\s+to\s+(?:our\s+use\s+of\s+cookies|cookies)/i,
+  /accept\s+(?:all\s+)?cookies/i,
+  /we\s+use\s+cookies/i,
+];
+
+/** Produces explicit, bounded extraction-quality signals instead of one opaque shell flag. */
+export function diagnoseExtraction(raw: string, markdown: string): ExtractionDiagnostics {
+  return {
+    javascriptRequired: JAVASCRIPT_MARKERS.some((marker) => marker.test(raw)),
+    botWall: BOT_MARKERS.some((marker) => marker.test(raw)),
+    consentInterstitial: CONSENT_MARKERS.some((marker) => marker.test(raw)),
+    sparseExtraction:
+      raw.length > 4_000 && markdown.length < 1_024 && markdown.length < raw.length * 0.008,
+    rawCharacters: raw.length,
+    extractedCharacters: markdown.length,
+  };
+}
+
+/**
+ * Checks if extraction diagnostics indicate any warning conditions.
+ *
+ * @param diagnostics - Extraction diagnostics to check
+ * @returns True if any warning condition is present, false otherwise
+ */
+export function hasExtractionWarning(diagnostics: ExtractionDiagnostics): boolean {
+  return (
+    diagnostics.javascriptRequired ||
+    diagnostics.botWall ||
+    diagnostics.consentInterstitial ||
+    diagnostics.sparseExtraction
+  );
+}
+
+/**
+ * Normalizes and truncates anchor text for link extraction.
+ *
+ * @param value - Raw anchor text
+ * @returns Normalized text, truncated to MAX_ANCHOR_CHARACTERS if needed
+ */
+function normalizedAnchor(value: string): string {
+  const text = value.replace(/\s+/g, " ").trim();
+  return text.length > MAX_ANCHOR_CHARACTERS
+    ? `${text.slice(0, MAX_ANCHOR_CHARACTERS - 1)}…`
+    : text;
+}
+
+/** Extracts a bounded set of safe HTTP(S) links without making any network requests. */
+export function extractDocumentLinks(raw: string, pageUrl: URL): ExtractedLinks {
+  const { document } = parseHTML(raw);
+  const internal: ExtractedLink[] = [];
+  const external: ExtractedLink[] = [];
+  let omittedInternal = 0;
+  let omittedExternal = 0;
+  const seen = new Set<string>();
+
+  for (const element of document.querySelectorAll<HTMLAnchorElement>("a[href]")) {
+    const href = element.getAttribute("href");
+    if (!href) continue;
+    let target: URL;
+    try {
+      target = new URL(href, pageUrl);
+    } catch {
+      continue;
+    }
+    if (!["http:", "https:"].includes(target.protocol) || target.username || target.password)
+      continue;
+    target.hash = "";
+    const anchorText = normalizedAnchor(element.textContent ?? "");
+    const key = `${target.href}\0${anchorText}`;
+    if (seen.has(key)) continue;
+    seen.add(key);
+    const isInternal = target.origin === pageUrl.origin;
+    const bucket = isInternal ? internal : external;
+    if (bucket.length < MAX_LINKS) bucket.push({ url: target.href, anchorText });
+    else if (isInternal) omittedInternal += 1;
+    else omittedExternal += 1;
+  }
+  return { internal, external, omittedInternal, omittedExternal };
+}

package/src/extract.ts

@@ -85,6 +85,31 @@ export function assertAbsoluteHttpUrl(value: string | URL): URL {
   return url;
 }
 
+/** Resolves page-URL metadata that Defuddle otherwise parses without a base URL. */
+function resolveDocumentRelativeMetadataUrls(document: Document, pageUrl: URL): void {
+  for (const meta of document.querySelectorAll<HTMLMetaElement>("meta[content]")) {
+    const key = (meta.getAttribute("property") ?? meta.getAttribute("name"))?.toLowerCase();
+    if (key !== "og:url" && key !== "twitter:url") continue;
+    const content = meta.getAttribute("content");
+    if (!content) continue;
+    try {
+      meta.setAttribute("content", new URL(content, pageUrl).href);
+    } catch {
+      meta.removeAttribute("content");
+    }
+  }
+
+  for (const link of document.querySelectorAll<HTMLLinkElement>('link[rel="canonical"][href]')) {
+    const href = link.getAttribute("href");
+    if (!href) continue;
+    try {
+      link.setAttribute("href", new URL(href, pageUrl).href);
+    } catch {
+      link.removeAttribute("href");
+    }
+  }
+}
+
 /**
  * Type guard that checks if an unknown value is a string.
  *
@@ -148,6 +173,104 @@ function deduplicateAdjacentLinks(markdown: string): string {
   return out.join("\n").replace(/(\[[^\n]+?\([^)]+\)[^\n]*\n)\1/g, "$1");
 }
 
+const CSS_BLOCK_AT_RULE = /^\s*@(container|font-face|keyframes|layer|media|page|supports)\b/i;
+const STANDALONE_CSS_RULE =
+  /^\s*(?:[.#][-_a-zA-Z]|\*\s*[.#:[>+~]|(?:html|body|main|article|nav|header|footer|aside)(?:\b|[.#:[>+~]))[^{}]*\{[^{}]*\}\s*$/i;
+const COMPLETE_STYLE_ELEMENT = /<style\b[^>]*>[\s\S]*?<\/style\s*>/gi;
+const CLOSING_STYLE_ELEMENT = /<\/style\s*>/i;
+const OPENING_STYLE_ELEMENT = /<style\b[^>]*>/i;
+
+/**
+ * Iteratively removes complete style elements from a string to prevent nested fragment recreation.
+ *
+ * @param value - The string to strip style elements from
+ * @returns The string with all complete style elements removed
+ */
+function stripCompleteStyleElements(value: string): string {
+  let cleaned = value;
+  while (true) {
+    const next = cleaned.replace(COMPLETE_STYLE_ELEMENT, "");
+    if (next === cleaned) return cleaned;
+    cleaned = next;
+  }
+}
+
+/**
+ * Calculates the net brace depth change in a string (opening braces minus closing braces).
+ *
+ * @param value - The string to analyze
+ * @returns The difference between opening and closing brace counts
+ */
+function braceDelta(value: string): number {
+  return (value.match(/{/g)?.length ?? 0) - (value.match(/}/g)?.length ?? 0);
+}
+
+/**
+ * Removes leaked stylesheet fragments from extracted Markdown while preserving fenced examples.
+ *
+ * The matcher is deliberately conservative: it removes style elements, recognized block at-rules,
+ * and complete standalone selector/declaration lines. Prose that merely discusses CSS and fenced
+ * CSS/SCSS/Less examples remain untouched.
+ */
+export function stripExtractedCssCruft(markdown: string): string {
+  const output: string[] = [];
+  let fence: { character: string; length: number } | undefined;
+  let styleElement = false;
+  let cssBlockDepth = 0;
+
+  for (const originalLine of markdown.split("\n")) {
+    const fenceMatch = /^[ \t]{0,3}(`{3,}|~{3,})/.exec(originalLine);
+    if (!fence && fenceMatch) {
+      fence = { character: fenceMatch[1][0], length: fenceMatch[1].length };
+      output.push(originalLine);
+      continue;
+    }
+    if (fence) {
+      output.push(originalLine);
+      if (
+        fenceMatch &&
+        fenceMatch[1][0] === fence.character &&
+        fenceMatch[1].length >= fence.length &&
+        !originalLine.slice(fenceMatch[0].length).trim()
+      ) {
+        fence = undefined;
+      }
+      continue;
+    }
+
+    let line = originalLine;
+    if (styleElement) {
+      const close = CLOSING_STYLE_ELEMENT.exec(line);
+      if (!close) continue;
+      line = line.slice(close.index + close[0].length);
+      styleElement = false;
+    }
+    line = stripCompleteStyleElements(line);
+    const open = OPENING_STYLE_ELEMENT.exec(line);
+    if (open) {
+      line = line.slice(0, open.index);
+      styleElement = true;
+    }
+
+    if (cssBlockDepth > 0) {
+      cssBlockDepth += braceDelta(line);
+      if (cssBlockDepth < 0) cssBlockDepth = 0;
+      continue;
+    }
+    if (CSS_BLOCK_AT_RULE.test(line)) {
+      cssBlockDepth = Math.max(0, braceDelta(line));
+      continue;
+    }
+    if (STANDALONE_CSS_RULE.test(line)) continue;
+    output.push(line);
+  }
+
+  return output
+    .join("\n")
+    .replace(/\n{3,}/g, "\n\n")
+    .trim();
+}
+
 /**
  * Runs Defuddle over an already-normalized document with Markdown extraction enabled.
  *
@@ -273,6 +396,10 @@ export async function extractHtmlToMarkdown(
   try {
     assertAbsoluteHttpUrl(baseUrl);
     const { document } = parseHTML(html);
+    // Defuddle parses URL metadata without using its explicit page URL as the
+    // base. Resolve relative values first so it does not log ERR_INVALID_URL
+    // through Pi's TUI even though extraction succeeds.
+    resolveDocumentRelativeMetadataUrls(document, baseUrl);
     removeMalformedSchemaOrgData(document);
     normalizeSelectorUnsafeIds(document);
     stripChromeWrappers(document, baseUrl);
@@ -281,7 +408,9 @@ export async function extractHtmlToMarkdown(
     // `/owner/repo/releases`) and metadata against the real origin instead of
     // dropping the scheme and host and constructing `new URL(pathname)`.
     const result = await runDefuddle(document, baseUrl.href);
-    const markdown = deduplicateAdjacentLinks(result?.content?.trim() ?? "");
+    const markdown = stripExtractedCssCruft(
+      deduplicateAdjacentLinks(result?.content?.trim() ?? ""),
+    );
     const trimmedTitle = result?.title?.trim();
     if (markdown) {
       return {
@@ -294,5 +423,9 @@ export async function extractHtmlToMarkdown(
   } catch {
     // Fall through to the basic converter for malformed or unsupported pages.
   }
-  return { markdown: htmlToMarkdownFallback(html), extractor: "basic", ...advertised };
+  return {
+    markdown: stripExtractedCssCruft(htmlToMarkdownFallback(html)),
+    extractor: "basic",
+    ...advertised,
+  };
 }

package/src/fetch.ts

@@ -1,9 +1,11 @@
 import type { IncomingMessage } from "node:http";
 import { awaitWithAbort } from "./abort";
 import { sliceCompleteDocument, type CompleteDocument, type FetchResult } from "./content";
+import { diagnoseExtraction, extractDocumentLinks, hasExtractionWarning } from "./evidence";
 import { extractHtmlToMarkdown } from "./extract";
 import { requestFollowingRedirects, type RedirectDependencies } from "./network-redirects";
 import type { ValidatedTarget } from "./network-policy";
+import { redactUrlForDisplay } from "./redact";
 import {
   decodeResponse,
   FETCH_MAX_BYTES,
@@ -56,37 +58,9 @@ export function normalizeGitHubRawUrl(rawUrl: string): string {
   }
 }
 
-const APP_SHELL_MARKERS = [
-  /please\s+enable\s+javascript/i,
-  /enable\s+javascript/i,
-  // Consent is only treated as an interstitial signal when it appears in a
-  // cookie/consent-banner phrase. A bare "consent" matches ordinary prose
-  // (e.g. privacy articles) and must not flag readable content as a shell.
-  /manage\s+(your\s+)?consent/i,
-  /your\s+(privacy\s+)?consent/i,
-  /consent\s+to\s+(our\s+use\s+of\s+cookies|cookies)/i,
-  /accept\s+(all\s+)?cookies/i,
-  /we\s+use\s+cookies/i,
-  /are\s+you\s+a\s+robot/i,
-  /verify\s+you\s+are\s+human/i,
-  /checking\s+your\s+browser/i,
-  /<title>\s*just\s+a\s+moment/i,
-];
-
-/**
- * Detects pages that are likely app shells, bot walls, or consent interstitials rather than
- * readable content.
- *
- * @param raw - The raw response body
- * @param markdown - The extracted Markdown
- * @returns True when the extracted text is suspiciously sparse relative to the raw page
- */
+/** @deprecated Prefer diagnoseExtraction for explicit extraction-quality signals. */
 export function detectAppShell(raw: string, markdown: string): boolean {
-  if (APP_SHELL_MARKERS.some((marker) => marker.test(raw))) return true;
-  // Require the extracted text to be both absolutely tiny and a very small
-  // fraction of the raw payload, so content-rich pages (e.g. React/Next.js SPAs
-  // whose raw HTML is dominated by inline scripts) are not mistaken for shells.
-  return raw.length > 4000 && markdown.length < 1024 && markdown.length < raw.length * 0.008;
+  return hasExtractionWarning(diagnoseExtraction(raw, markdown));
 }
 
 const REQUEST_TIMEOUT_MS = 20_000;
@@ -238,10 +212,13 @@ async function documentFromResponse(
     }
   } else markdown = raw.trim();
 
-  const shellSuspected =
-    contentType === "text/html" || contentType === "application/xhtml+xml"
-      ? detectAppShell(raw, markdown)
-      : false;
+  const isHtml = contentType === "text/html" || contentType === "application/xhtml+xml";
+  const extractionDiagnostics = isHtml ? diagnoseExtraction(raw, markdown) : undefined;
+  const shellSuspected = extractionDiagnostics
+    ? hasExtractionWarning(extractionDiagnostics)
+    : false;
+  const etag = responseHeader(response, "etag");
+  const lastModified = responseHeader(response, "last-modified");
 
   return {
     url: target.url.toString(),
@@ -250,6 +227,10 @@ async function documentFromResponse(
     title,
     extractor,
     shellSuspected,
+    extractionDiagnostics,
+    links: isHtml ? extractDocumentLinks(raw, target.url) : undefined,
+    validators: etag || lastModified ? { etag, lastModified } : undefined,
+    cachedAt: Date.now(),
     llmsTxtDescribedBy: describedBy,
     markdownAlternateUrl,
   };
@@ -267,10 +248,10 @@ function assertAbsoluteHttpUrlForFetch(value: string): URL {
   try {
     url = new URL(value);
   } catch {
-    throw new Error(`Invalid URL: ${value}`);
+    throw new Error("web_fetch received an invalid URL.");
   }
   if (url.protocol !== "http:" && url.protocol !== "https:") {
-    throw new Error(`web_fetch only supports http(s) URLs: ${value}`);
+    throw new Error(`web_fetch only supports http(s) URLs: ${redactUrlForDisplay(url)}`);
   }
   return url;
 }
@@ -284,11 +265,12 @@ function assertAbsoluteHttpUrlForFetch(value: string): URL {
  * @returns The complete document with extracted markdown content
  * @throws If the request times out, is cancelled, or encounters an error
  */
-export async function fetchCompleteDocument(
+async function fetchDocument(
   rawUrl: string,
   signal: AbortSignal | undefined,
   dependencies: FetchRemoteDependencies,
-): Promise<CompleteDocument> {
+  cached?: CompleteDocument,
+): Promise<{ document: CompleteDocument; revalidated: boolean }> {
   assertAbsoluteHttpUrlForFetch(normalizeGitHubRawUrl(rawUrl));
   const controller = new AbortController();
   const timeoutMs = dependencies.timeoutMs ?? REQUEST_TIMEOUT_MS;
@@ -300,17 +282,36 @@ export async function fetchCompleteDocument(
   }, timeoutMs);
   const cancel = () => controller.abort();
   signal?.addEventListener("abort", cancel, { once: true });
+  const conditionalHeaders: Record<string, string> = {};
+  if (cached?.validators?.etag) conditionalHeaders["If-None-Match"] = cached.validators.etag;
+  if (cached?.validators?.lastModified)
+    conditionalHeaders["If-Modified-Since"] = cached.validators.lastModified;
 
   try {
     const { target, response } = await requestFollowingRedirects(
       normalizeGitHubRawUrl(rawUrl),
       controller.signal,
-      {
-        validateUrl: dependencies.validateUrl,
-        request: dependencies.request,
-      },
+      dependencies,
+      conditionalHeaders,
     );
-    return await documentFromResponse(target, response, controller.signal, extractHtml);
+    if (response.statusCode === 304 && cached) {
+      response.resume();
+      const etag = responseHeader(response, "etag") ?? cached.validators?.etag;
+      const lastModified =
+        responseHeader(response, "last-modified") ?? cached.validators?.lastModified;
+      return {
+        document: {
+          ...cached,
+          validators: etag || lastModified ? { etag, lastModified } : undefined,
+          cachedAt: Date.now(),
+        },
+        revalidated: true,
+      };
+    }
+    return {
+      document: await documentFromResponse(target, response, controller.signal, extractHtml),
+      revalidated: false,
+    };
   } catch (error) {
     if (timedOut) throw new Error(`web_fetch timed out after ${timeoutMs / 1000} seconds.`);
     if (signal?.aborted) throw new Error("web_fetch was cancelled.");
@@ -321,6 +322,24 @@ export async function fetchCompleteDocument(
   }
 }
 
+export async function fetchCompleteDocument(
+  rawUrl: string,
+  signal: AbortSignal | undefined,
+  dependencies: FetchRemoteDependencies,
+): Promise<CompleteDocument> {
+  return (await fetchDocument(rawUrl, signal, dependencies)).document;
+}
+
+/** Revalidates a stale representation with its ETag and Last-Modified validators. */
+export async function revalidateCompleteDocument(
+  rawUrl: string,
+  cached: CompleteDocument,
+  signal: AbortSignal | undefined,
+  dependencies: FetchRemoteDependencies,
+): Promise<{ document: CompleteDocument; revalidated: boolean }> {
+  return fetchDocument(rawUrl, signal, dependencies, cached);
+}
+
 /**
  * Fetches a URL and returns a slice of the content starting at the given offset.
  *