@zeldrisho/pi-web-fetch 0.14.1 → 0.16.0

package/CHANGELOG.md

@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.16.0] - 2026-10-28
+
+### Added
+
+- Automatically resolve supported AWS WAF challenges when fetching pages.
+
+## [0.15.0] - 2026-10-28
+
+### Added
+
+- Return minified JavaScript, JSON, and YAML responses as indented, fenced code blocks.
+- Identify OpenAPI and Swagger JSON and YAML documents and JavaScript bundles containing literal specs as `openapi-spec`.
+
+### Changed
+
+- Focus large fenced source documents using line-aligned excerpts.
+- Shorten documentation index notices.
+- Probe one more ancestor directory for `/llms.txt`, so locale-prefixed documentation sites resolve to their section index.
+
+### Fixed
+
+- Report AWS WAF challenge responses as blocks instead of empty pages.
+- Fetch Markdown documentation that static hosts serve as `application/octet-stream`.
+- Fall back to a discovered documentation index when a bot challenge withholds the requested page.
+
 ## [0.14.1] - 2026-10-25
 
 ### Changed
@@ -270,7 +295,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Add bounded public page fetching ([b979496](https://github.com/zeldrisho/pi-packages/commit/b979496b32de1cead172ba570307e4a4a7b3421d))
 
-[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.1...HEAD
+[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.16.0...HEAD
+[0.16.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.15.0...pi-web-fetch-v0.16.0
+[0.15.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.1...pi-web-fetch-v0.15.0
 [0.14.1]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.0...pi-web-fetch-v0.14.1
 [0.14.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.13.0...pi-web-fetch-v0.14.0
 [0.13.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.12.0...pi-web-fetch-v0.13.0

package/README.md

@@ -15,7 +15,7 @@ Use `pi config` to enable or disable the extension, and `pi list` to see configu
 
 ## Usage
 
-The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markdown, plain text, JSON, and XML. HTML is converted to Markdown; images are omitted. URLs with credentials, local or private network targets, unsafe redirects, downloads over 5 MiB, and unsupported content types are rejected.
+The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markdown, plain text, JSON, and XML. HTML is converted to Markdown; images are omitted. Minified JavaScript, JSON, and YAML are returned as indented code blocks so long single-line payloads stay readable. URLs with credentials, local or private network targets, unsafe redirects, downloads over 5 MiB, and unsupported content types are rejected.
 
 | Parameter | Description                                                                                               |
 | --------- | --------------------------------------------------------------------------------------------------------- |
@@ -23,15 +23,13 @@ The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markd
 | `query`   | Optional; selects matching sections from a long page. Sections remain in source order.                    |
 | `offset`  | Optional character offset for continuing through extracted content; takes precedence over a URL fragment. |
 
-Output is bounded to 6,000 characters by default. For long pages, use the returned `nextOffset` to continue. A query selects a focused view of the fetched page; omit it to read the complete document. Pi's preview renders Markdown with the active theme and is collapsed by default (`Ctrl+O` expands it). The call preview shows the URL; the focus query and character offset are not repeated there.
+Output is bounded to 6,000 characters by default. For long pages, use the returned `nextOffset` to continue. Use `query` to focus on matching sections, or omit it to read the complete document. Pi's preview is collapsed by default; press `Ctrl+O` to expand it.
 
 For example, a URL such as `https://github.com/shadcn-ui/lint#get-started` starts at the matching heading. GitHub `blob` links and bare gists are converted to raw source URLs; repository roots use their README. GitHub directory listings are not supported.
 
 ### Page alternatives and bot challenges
 
-The tool can discover advertised Markdown versions and `/llms.txt` indexes. When an index is discovered, its URL is shown in the result so you can follow it if the fetched page is insufficient; when the index replaces the requested page, the result explains that fallback. Anubis proof-of-work challenges are supported; unsupported challenges and JavaScript CAPTCHAs return a blocked result. There is no browser fallback. Challenge cookies are not reused between fetches.
-
-Pages with HTML nested beyond 512 elements are rejected.
+The tool can discover advertised Markdown versions and `/llms.txt` indexes. An index URL appears in the result when found; follow it if the page is insufficient. The tool can solve Anubis and supported AWS WAF proof-of-work challenges. If an AWS WAF or Cloudflare challenge remains blocked, it may fall back to the site's documentation index. Unsupported challenges and JavaScript CAPTCHAs return a blocked result; there is no browser fallback.
 
 ### Caching
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zeldrisho/pi-web-fetch",
-  "version": "0.14.1",
+  "version": "0.16.0",
   "description": "Pi extension for secure, bounded public web page fetching and Markdown extraction",
   "keywords": [
     "pi",
@@ -30,8 +30,11 @@
     "access": "public"
   },
   "dependencies": {
+    "acorn": "^8.19.0",
+    "acorn-walk": "^8.3.5",
     "defuddle": "^0.19.5",
-    "linkedom": "^0.18.13"
+    "linkedom": "^0.18.13",
+    "yaml": "^2.9.1"
   },
   "devDependencies": {
     "@earendil-works/pi-coding-agent": "^1.4.0",

package/src/acquisition/fetch.ts

@@ -14,6 +14,13 @@ import {
   parseAnubisChallenge,
   solveAnubisChallenge,
 } from "../challenges/anubis.ts";
+import {
+  AWS_WAF_TOKEN_COOKIE,
+  isAwsWafChallengeResponse,
+  type AwsWafResolver,
+  type AwsWafSolution,
+} from "../challenges/aws-waf.ts";
+import { nativeAwsWafResolver } from "../challenges/aws-waf-resolver.ts";
 import {
   sliceCompleteDocument,
   type CompleteDocument,
@@ -27,6 +34,8 @@ import {
 } from "../document/evidence.ts";
 import { extractHtmlToMarkdown } from "../document/extract.ts";
 import { assertConversionDepth } from "../document/complexity.ts";
+import { formatMinifiedSource } from "../document/source-format.ts";
+import { isOpenapiSource } from "../document/openapi.ts";
 import {
   requestFollowingRedirects,
   type RedirectDependencies,
@@ -87,6 +96,10 @@ export interface FetchRemoteDependencies extends RedirectDependencies {
   timeoutMs?: number;
   /** When false, an Anubis interstitial is reported as blocked instead of solved. Defaults to true. */
   solveAnubisChallenges?: boolean;
+  /** When false, an AWS WAF challenge is reported as blocked instead of solved. Defaults to true. */
+  solveAwsWafChallenges?: boolean;
+  /** Resolver used for AWS WAF challenges; defaults to the in-process resolver. */
+  awsWafResolver?: AwsWafResolver;
 }
 
 /**
@@ -154,17 +167,47 @@ function isHtmlContentType(
   return value === "text/html" || value === "application/xhtml+xml";
 }
 
+/**
+ * Path suffixes whose content is read as plain text when a server labels it
+ * `application/octet-stream`. Static hosts commonly serve Markdown documentation
+ * with that generic binary type; restricting the exception to these suffixes keeps
+ * real binary downloads out of extraction.
+ */
+const TEXT_DOCUMENT_PATH_PATTERN = /\.(?:md|markdown|mdx|txt)$/i;
+
 /** Narrows a media type to those web_fetch is willing to read. */
-function isAllowedContentType(value: string | undefined): value is string {
-  return (
-    value !== undefined && (value.startsWith("text/") || ALLOWED_CONTENT_TYPES.includes(value))
-  );
+function isAllowedContentType(value: string | undefined, pathname: string): value is string {
+  if (value === undefined) return false;
+
+  if (value.startsWith("text/") || ALLOWED_CONTENT_TYPES.includes(value)) return true;
+
+  return value === "application/octet-stream" && TEXT_DOCUMENT_PATH_PATTERN.test(pathname);
+}
+
+/**
+ * Identifies a page a bot-protection layer refused to serve.
+ *
+ * The response carries no page content, so callers may substitute a site index the
+ * same page would otherwise have been extracted from.
+ */
+export class ChallengeBlockedError extends Error {
+  constructor(message: string) {
+    super(message);
+    this.name = "ChallengeBlockedError";
+  }
 }
 
 /** Throws the established error for a response whose status is not successful. */
 function assertSuccessfulResponse(response: IncomingMessage): void {
   const status = response.statusCode ?? 0;
 
+  if (isAwsWafChallengeResponse(response)) {
+    response.resume();
+    throw new ChallengeBlockedError(
+      `web_fetch was blocked by an AWS WAF anti-bot challenge (HTTP ${status}).`,
+    );
+  }
+
   if (status >= 200 && status < 300) return;
 
   const cloudflareChallenge =
@@ -173,7 +216,9 @@ function assertSuccessfulResponse(response: IncomingMessage): void {
   response.resume();
 
   if (cloudflareChallenge) {
-    throw new Error(`web_fetch was blocked by Cloudflare's anti-bot challenge (HTTP ${status}).`);
+    throw new ChallengeBlockedError(
+      `web_fetch was blocked by Cloudflare's anti-bot challenge (HTTP ${status}).`,
+    );
   }
 
   const authenticationHint = [401, 403, 404].includes(status)
@@ -243,6 +288,8 @@ async function documentFromDecoded(
   let extractor: CompleteDocument["extractor"] = "raw";
   let describedBy: string | undefined = linkHints.describedBy;
   let markdownAlternateUrl: string | undefined = linkHints.markdownAlternate;
+  let sourceLanguage: string | undefined;
+  const openapiSpec = isOpenapiSource(raw, contentType);
 
   if (isHtmlContentType(contentType)) {
     const extracted = await awaitWithAbort(extractHtml(raw, target.url), signal);
@@ -254,12 +301,23 @@ async function documentFromDecoded(
     markdownAlternateUrl = markdownAlternateUrl ?? extracted.markdownAlternateLink;
   } else if (contentType === "application/json") {
     try {
-      markdown = `\`\`\`json\n${JSON.stringify(JSON.parse(raw), null, 2)}\n\`\`\``;
+      const parsed: unknown = JSON.parse(raw);
+
+      markdown = `\`\`\`json\n${JSON.stringify(parsed, null, 2)}\n\`\`\``;
+      sourceLanguage = "json";
     } catch {
-      markdown = raw;
+      // Minified or JSON-like bodies are still worth structuring even when they do
+      // not parse as strict JSON.
+      const formatted = formatMinifiedSource(raw, contentType);
+
+      markdown = formatted?.markdown ?? raw;
+      sourceLanguage = formatted?.language;
     }
   } else {
-    markdown = raw;
+    const formatted = formatMinifiedSource(raw, contentType);
+
+    markdown = formatted?.markdown ?? raw;
+    sourceLanguage = formatted?.language;
 
     const markdownContent = /\bmarkdown\b|\.md(?:$|[?#])/i.test(
       contentTypeHeader + target.url.pathname,
@@ -306,6 +364,8 @@ async function documentFromDecoded(
     cachedAt: Date.now(),
     llmsTxtDescribedBy: describedBy,
     markdownAlternateUrl,
+    sourceLanguage,
+    openapiSpec,
   };
 }
 
@@ -334,7 +394,7 @@ async function documentFromResponse(
   const contentTypeHeader = responseContentTypeHeader(response);
   const contentType = normalizeContentType(contentTypeHeader);
 
-  if (!isAllowedContentType(contentType)) {
+  if (!isAllowedContentType(contentType, target.url.pathname)) {
     response.destroy();
     throw new Error(
       `web_fetch does not support ${contentType ?? (contentTypeHeader || "this content type")}.`,
@@ -368,6 +428,7 @@ async function documentFromAcquisition(
   dependencies: FetchRemoteDependencies,
   allowSolve: boolean,
 ): Promise<CompleteDocument> {
+  assertSuccessfulResponse(response);
   const status = response.statusCode ?? 0;
   const contentTypeHeader = responseContentTypeHeader(response);
   const contentType = normalizeContentType(contentTypeHeader);
@@ -443,6 +504,37 @@ function assertAbsoluteHttpUrlForFetch(value: string): URL {
   return url;
 }
 
+/**
+ * Attempts one WAF token resolution while preserving caller cancellation.
+ * Solver failures are intentionally converted into the normal blocked path.
+ */
+async function resolveAwsWafToken(
+  resolver: AwsWafResolver,
+  target: ValidatedTarget,
+  signal: AbortSignal,
+  network: RedirectDependencies,
+  budgetMs: number,
+): Promise<AwsWafSolution | undefined> {
+  if (budgetMs <= 0) return undefined;
+
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), budgetMs);
+  const resolverSignal = AbortSignal.any([signal, controller.signal]);
+
+  try {
+    return await awaitWithAbort(
+      resolver.resolve({ url: target.url, signal: resolverSignal, network }),
+      resolverSignal,
+    );
+  } catch (error) {
+    if (signal.aborted) throw error;
+
+    return undefined;
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
 /**
  * Fetches a URL and returns the complete document with extracted content.
  *
@@ -458,10 +550,12 @@ async function fetchDocument(
   dependencies: FetchRemoteDependencies,
   cached?: CompleteDocument,
   allowEmptyContent = false,
+  retryRepresentationMiss = true,
 ): Promise<{ document: CompleteDocument; revalidated: boolean }> {
   assertAbsoluteHttpUrlForFetch(normalizeGitHubRawUrl(rawUrl));
   const controller = new AbortController();
   const timeoutMs = dependencies.timeoutMs ?? REQUEST_TIMEOUT_MS;
+  const deadline = Date.now() + timeoutMs;
   const extractHtml = dependencies.extractHtml ?? extractHtmlToMarkdown;
   let timedOut = false;
 
@@ -499,8 +593,10 @@ async function fetchDocument(
     // Some servers route Markdown negotiation to a missing representation even
     // though their HTML page exists (for example, Redis documentation URLs).
     // Retry only these representation-related statuses, through the same
-    // validated redirect/network boundary, with HTML preferred.
-    if ([404, 406].includes(response.statusCode ?? 0)) {
+    // validated redirect/network boundary, with HTML preferred. Callers that probe
+    // for one exact representation opt out: HTML from that URL is never the
+    // representation they are looking for.
+    if (retryRepresentationMiss && [404, 406].includes(response.statusCode ?? 0)) {
       response.resume();
       ({ target, response } = await requestFollowingRedirects(
         target.url,
@@ -526,6 +622,48 @@ async function fetchDocument(
       }
     }
 
+    if (
+      isAwsWafChallengeResponse(response) &&
+      requestDependencies.solveAwsWafChallenges !== false
+    ) {
+      const resolver = requestDependencies.awsWafResolver ?? nativeAwsWafResolver;
+
+      if (resolver) {
+        response.resume();
+
+        const solution = await resolveAwsWafToken(
+          resolver,
+          target,
+          controller.signal,
+          requestDependencies,
+          Math.floor(Math.max(0, deadline - Date.now()) / 2),
+        );
+
+        if (solution) {
+          const stored = requestDependencies.cookies?.set(
+            AWS_WAF_TOKEN_COOKIE,
+            solution.token,
+            target.url,
+          );
+
+          if (stored) {
+            // Retry once inside the same validated redirect and cookie boundary.
+            // Keep the browser identity that earned the token, and report a
+            // second challenge as blocked instead of looping.
+            const retryDependencies = solution.userAgent
+              ? { ...requestDependencies, userAgent: solution.userAgent }
+              : requestDependencies;
+
+            ({ target, response } = await requestFollowingRedirects(
+              target.url,
+              controller.signal,
+              retryDependencies,
+            ));
+          }
+        }
+      }
+    }
+
     if (response.statusCode === 304 && cached) {
       response.resume();
       const etag = responseHeader(response, "etag") ?? cached.validators?.etag;
@@ -566,13 +704,30 @@ async function fetchDocument(
   }
 }
 
+/**
+ * Fetches a complete document for one URL.
+ *
+ * @param allowEmptyContent - Whether an empty body is acceptable
+ * @param retryRepresentationMiss - Whether a 404/406 may be retried as HTML. Probe
+ *   callers that want one exact representation pass false.
+ */
 export async function fetchCompleteDocument(
   rawUrl: string,
   signal: AbortSignal | undefined,
   dependencies: FetchRemoteDependencies,
   allowEmptyContent = false,
+  retryRepresentationMiss = true,
 ): Promise<CompleteDocument> {
-  return (await fetchDocument(rawUrl, signal, dependencies, undefined, allowEmptyContent)).document;
+  return (
+    await fetchDocument(
+      rawUrl,
+      signal,
+      dependencies,
+      undefined,
+      allowEmptyContent,
+      retryRepresentationMiss,
+    )
+  ).document;
 }
 
 /** Revalidates a stale representation with its ETag and Last-Modified validators. */

package/src/acquisition/llms.ts

@@ -1,6 +1,10 @@
 import { ExpiringLruCache } from "../cache";
 import type { CompleteDocument } from "../document/content.ts";
-import { fetchCompleteDocument, type FetchRemoteDependencies } from "./fetch.ts";
+import {
+  ChallengeBlockedError,
+  fetchCompleteDocument,
+  type FetchRemoteDependencies,
+} from "./fetch.ts";
 import { InflightCoalescer } from "../inflight";
 
 const CACHE_TTL_MS = 24 * 60 * 60 * 1_000;
@@ -97,8 +101,13 @@ interface LlmsTxtProbe {
 
 const MAX_LLMS_TXT_PROBE_ENTRIES = 512;
 
-/** Deepest ancestor-directory `/llms.txt` probed, e.g. `/r2/x/y` probes `/r2/llms.txt`. */
-const MAX_LLMS_TXT_DIRECTORY_DEPTH = 1;
+/**
+ * Deepest ancestor-directory `/llms.txt` probed, e.g. `/a/b/c` also probes
+ * `/a/b/llms.txt`. Locale-prefixed documentation sites publish their section index
+ * one level below the locale root, so two levels are needed to reach
+ * `/en/docs/llms.txt` from `/en/docs/catalog/...`.
+ */
+const MAX_LLMS_TXT_DIRECTORY_DEPTH = 2;
 
 const llmsTxtProbes = new ExpiringLruCache<string, LlmsTxtProbe>(
   MAX_LLMS_TXT_PROBE_ENTRIES,
@@ -179,10 +188,16 @@ async function probeUsableRawText(
       if (existing) return existing;
 
       try {
-        const document = await fetchCompleteDocument(candidate.toString(), sharedSignal, {
-          ...dependencies,
-          solveAnubisChallenges: false,
-        });
+        const document = await fetchCompleteDocument(
+          candidate.toString(),
+          sharedSignal,
+          { ...dependencies, solveAnubisChallenges: false, solveAwsWafChallenges: false },
+          // An index probe never accepts an empty body.
+          false,
+          // A probe wants exactly this representation; retrying as HTML only burns
+          // a request on a URL that is already known to be missing.
+          false,
+        );
 
         const available = isUsableLlmsTxtIndex(document) ? document : undefined;
         const result = { document: available, expires: Date.now() + CACHE_TTL_MS };
@@ -226,8 +241,8 @@ function isSafeToProbe(candidate: URL | undefined, primaryUrl: string): boolean
  *
  * - Probes the site's `/llms.txt` index once per origin per TTL window, in parallel with
  *   the primary fetch, so later calls can advertise it.
- * - Serves the index instead of the page when the page looks like an app shell or carries
- *   too little readable text.
+ * - Serves the index instead of the page when the page looks like an app shell, carries
+ *   too little readable text, or was withheld by a bot-protection challenge.
  * - Otherwise annotates the page with the index URL so agents can discover sibling pages.
  *
  * A missing or useless index can never degrade the primary fetch.
@@ -247,16 +262,32 @@ export async function fetchDocumentWithLlmsTxtSupport(
 
   const candidates = buildLlmsTxtCandidateUrls(rawUrl);
 
-  const [primary, blindIndex] = await Promise.all([
+  // A withheld page must not discard an index that is still usable, so the pair is
+  // settled instead of rejected together.
+  const [primarySettled, indexSettled] = await Promise.allSettled([
     fetchCompleteDocument(rawUrl, signal, dependencies, true),
     candidates.length > 0
       ? ensureLlmsTxtIndex(candidates, signal, dependencies)
       : Promise.resolve(undefined),
   ]);
 
+  // An index probe only rejects when the caller cancels, which must still surface.
+  if (indexSettled.status === "rejected") throw indexSettled.reason;
+
+  if (primarySettled.status === "rejected") {
+    // A bot-protection layer replaced the page with a challenge that carries no
+    // content; the discovered index is the only remaining pointer to that content.
+    if (indexSettled.value && primarySettled.reason instanceof ChallengeBlockedError)
+      return { ...indexSettled.value.document, llmsTxtFallback: true };
+
+    throw primarySettled.reason;
+  }
+
+  const primary = primarySettled.value;
+
   // A `describedby` advertisement names the covering index authoritatively per
   // llmstxt.org v2, so it outranks the blind root/section probes whenever usable.
-  let index = blindIndex;
+  let index = indexSettled.value;
 
   if (primary.llmsTxtDescribedBy) {
     const describedBy = toAbsoluteUrl(primary.llmsTxtDescribedBy);