@zeldrisho/pi-web-fetch 0.14.1 → 0.15.0

package/CHANGELOG.md

@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.15.0] - 2026-10-28
+
+### Added
+
+- Return minified JavaScript, JSON, and YAML responses as indented, fenced code blocks.
+- Identify OpenAPI and Swagger JSON and YAML documents and JavaScript bundles containing literal specs as `openapi-spec`.
+
+### Changed
+
+- Focus large fenced source documents using line-aligned excerpts.
+- Shorten documentation index notices.
+- Probe one more ancestor directory for `/llms.txt`, so locale-prefixed documentation sites resolve to their section index.
+
+### Fixed
+
+- Report AWS WAF challenge responses as blocks instead of empty pages.
+- Fetch Markdown documentation that static hosts serve as `application/octet-stream`.
+- Fall back to a discovered documentation index when a bot challenge withholds the requested page.
+
 ## [0.14.1] - 2026-10-25
 
 ### Changed
@@ -270,7 +289,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Add bounded public page fetching ([b979496](https://github.com/zeldrisho/pi-packages/commit/b979496b32de1cead172ba570307e4a4a7b3421d))
 
-[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.1...HEAD
+[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.15.0...HEAD
+[0.15.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.1...pi-web-fetch-v0.15.0
 [0.14.1]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.0...pi-web-fetch-v0.14.1
 [0.14.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.13.0...pi-web-fetch-v0.14.0
 [0.13.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.12.0...pi-web-fetch-v0.13.0

package/README.md

@@ -15,7 +15,7 @@ Use `pi config` to enable or disable the extension, and `pi list` to see configu
 
 ## Usage
 
-The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markdown, plain text, JSON, and XML. HTML is converted to Markdown; images are omitted. URLs with credentials, local or private network targets, unsafe redirects, downloads over 5 MiB, and unsupported content types are rejected.
+The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markdown, plain text, JSON, and XML. HTML is converted to Markdown; images are omitted. Minified JavaScript, JSON, and YAML are returned as indented code blocks so long single-line payloads stay readable. URLs with credentials, local or private network targets, unsafe redirects, downloads over 5 MiB, and unsupported content types are rejected.
 
 | Parameter | Description                                                                                               |
 | --------- | --------------------------------------------------------------------------------------------------------- |
@@ -23,13 +23,13 @@ The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markd
 | `query`   | Optional; selects matching sections from a long page. Sections remain in source order.                    |
 | `offset`  | Optional character offset for continuing through extracted content; takes precedence over a URL fragment. |
 
-Output is bounded to 6,000 characters by default. For long pages, use the returned `nextOffset` to continue. A query selects a focused view of the fetched page; omit it to read the complete document. Pi's preview renders Markdown with the active theme and is collapsed by default (`Ctrl+O` expands it). The call preview shows the URL; the focus query and character offset are not repeated there.
+Output is bounded to 6,000 characters by default. For long pages, use the returned `nextOffset` to continue. A query selects a focused view of the fetched page; omit it to read the complete document. Large fenced source files can be focused in line-aligned excerpts; excerpts are not necessarily valid standalone source. OpenAPI and Swagger JSON or YAML documents and JavaScript bundles containing literal specs can be reported as `openapi-spec`; detection is best-effort and limited to source bodies up to 512 KiB. Pi's preview renders Markdown with the active theme and is collapsed by default (`Ctrl+O` expands it). The call preview shows the URL; the focus query and character offset are not repeated there.
 
 For example, a URL such as `https://github.com/shadcn-ui/lint#get-started` starts at the matching heading. GitHub `blob` links and bare gists are converted to raw source URLs; repository roots use their README. GitHub directory listings are not supported.
 
 ### Page alternatives and bot challenges
 
-The tool can discover advertised Markdown versions and `/llms.txt` indexes. When an index is discovered, its URL is shown in the result so you can follow it if the fetched page is insufficient; when the index replaces the requested page, the result explains that fallback. Anubis proof-of-work challenges are supported; unsupported challenges and JavaScript CAPTCHAs return a blocked result. There is no browser fallback. Challenge cookies are not reused between fetches.
+The tool can discover advertised Markdown versions and `/llms.txt` indexes. When an index is discovered, its URL is shown in the result so you can follow it if the fetched page is insufficient; when the index replaces the requested page, the result explains that fallback. Markdown files served with a generic binary content type are read as text. Anubis proof-of-work challenges are supported, and a page withheld by an AWS WAF or Cloudflare bot challenge falls back to the site's documentation index when one exists. Other unsupported challenges and JavaScript CAPTCHAs return a blocked result. There is no browser fallback. Challenge cookies are not reused between fetches.
 
 Pages with HTML nested beyond 512 elements are rejected.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zeldrisho/pi-web-fetch",
-  "version": "0.14.1",
+  "version": "0.15.0",
   "description": "Pi extension for secure, bounded public web page fetching and Markdown extraction",
   "keywords": [
     "pi",
@@ -30,8 +30,11 @@
     "access": "public"
   },
   "dependencies": {
+    "acorn": "^8.19.0",
+    "acorn-walk": "^8.3.5",
     "defuddle": "^0.19.5",
-    "linkedom": "^0.18.13"
+    "linkedom": "^0.18.13",
+    "yaml": "^2.9.1"
   },
   "devDependencies": {
     "@earendil-works/pi-coding-agent": "^1.4.0",

package/src/acquisition/fetch.ts

@@ -27,6 +27,8 @@ import {
 } from "../document/evidence.ts";
 import { extractHtmlToMarkdown } from "../document/extract.ts";
 import { assertConversionDepth } from "../document/complexity.ts";
+import { formatMinifiedSource } from "../document/source-format.ts";
+import { isOpenapiSource } from "../document/openapi.ts";
 import {
   requestFollowingRedirects,
   type RedirectDependencies,
@@ -154,17 +156,47 @@ function isHtmlContentType(
   return value === "text/html" || value === "application/xhtml+xml";
 }
 
+/**
+ * Path suffixes whose content is read as plain text when a server labels it
+ * `application/octet-stream`. Static hosts commonly serve Markdown documentation
+ * with that generic binary type; restricting the exception to these suffixes keeps
+ * real binary downloads out of extraction.
+ */
+const TEXT_DOCUMENT_PATH_PATTERN = /\.(?:md|markdown|mdx|txt)$/i;
+
 /** Narrows a media type to those web_fetch is willing to read. */
-function isAllowedContentType(value: string | undefined): value is string {
-  return (
-    value !== undefined && (value.startsWith("text/") || ALLOWED_CONTENT_TYPES.includes(value))
-  );
+function isAllowedContentType(value: string | undefined, pathname: string): value is string {
+  if (value === undefined) return false;
+
+  if (value.startsWith("text/") || ALLOWED_CONTENT_TYPES.includes(value)) return true;
+
+  return value === "application/octet-stream" && TEXT_DOCUMENT_PATH_PATTERN.test(pathname);
+}
+
+/**
+ * Identifies a page a bot-protection layer refused to serve.
+ *
+ * The response carries no page content, so callers may substitute a site index the
+ * same page would otherwise have been extracted from.
+ */
+export class ChallengeBlockedError extends Error {
+  constructor(message: string) {
+    super(message);
+    this.name = "ChallengeBlockedError";
+  }
 }
 
 /** Throws the established error for a response whose status is not successful. */
 function assertSuccessfulResponse(response: IncomingMessage): void {
   const status = response.statusCode ?? 0;
 
+  if (responseHeader(response, "x-amzn-waf-action")?.trim().toLowerCase() === "challenge") {
+    response.resume();
+    throw new ChallengeBlockedError(
+      `web_fetch was blocked by an AWS WAF anti-bot challenge (HTTP ${status}).`,
+    );
+  }
+
   if (status >= 200 && status < 300) return;
 
   const cloudflareChallenge =
@@ -173,7 +205,9 @@ function assertSuccessfulResponse(response: IncomingMessage): void {
   response.resume();
 
   if (cloudflareChallenge) {
-    throw new Error(`web_fetch was blocked by Cloudflare's anti-bot challenge (HTTP ${status}).`);
+    throw new ChallengeBlockedError(
+      `web_fetch was blocked by Cloudflare's anti-bot challenge (HTTP ${status}).`,
+    );
   }
 
   const authenticationHint = [401, 403, 404].includes(status)
@@ -243,6 +277,8 @@ async function documentFromDecoded(
   let extractor: CompleteDocument["extractor"] = "raw";
   let describedBy: string | undefined = linkHints.describedBy;
   let markdownAlternateUrl: string | undefined = linkHints.markdownAlternate;
+  let sourceLanguage: string | undefined;
+  const openapiSpec = isOpenapiSource(raw, contentType);
 
   if (isHtmlContentType(contentType)) {
     const extracted = await awaitWithAbort(extractHtml(raw, target.url), signal);
@@ -254,12 +290,23 @@ async function documentFromDecoded(
     markdownAlternateUrl = markdownAlternateUrl ?? extracted.markdownAlternateLink;
   } else if (contentType === "application/json") {
     try {
-      markdown = `\`\`\`json\n${JSON.stringify(JSON.parse(raw), null, 2)}\n\`\`\``;
+      const parsed: unknown = JSON.parse(raw);
+
+      markdown = `\`\`\`json\n${JSON.stringify(parsed, null, 2)}\n\`\`\``;
+      sourceLanguage = "json";
     } catch {
-      markdown = raw;
+      // Minified or JSON-like bodies are still worth structuring even when they do
+      // not parse as strict JSON.
+      const formatted = formatMinifiedSource(raw, contentType);
+
+      markdown = formatted?.markdown ?? raw;
+      sourceLanguage = formatted?.language;
     }
   } else {
-    markdown = raw;
+    const formatted = formatMinifiedSource(raw, contentType);
+
+    markdown = formatted?.markdown ?? raw;
+    sourceLanguage = formatted?.language;
 
     const markdownContent = /\bmarkdown\b|\.md(?:$|[?#])/i.test(
       contentTypeHeader + target.url.pathname,
@@ -306,6 +353,8 @@ async function documentFromDecoded(
     cachedAt: Date.now(),
     llmsTxtDescribedBy: describedBy,
     markdownAlternateUrl,
+    sourceLanguage,
+    openapiSpec,
   };
 }
 
@@ -334,7 +383,7 @@ async function documentFromResponse(
   const contentTypeHeader = responseContentTypeHeader(response);
   const contentType = normalizeContentType(contentTypeHeader);
 
-  if (!isAllowedContentType(contentType)) {
+  if (!isAllowedContentType(contentType, target.url.pathname)) {
     response.destroy();
     throw new Error(
       `web_fetch does not support ${contentType ?? (contentTypeHeader || "this content type")}.`,
@@ -368,6 +417,7 @@ async function documentFromAcquisition(
   dependencies: FetchRemoteDependencies,
   allowSolve: boolean,
 ): Promise<CompleteDocument> {
+  assertSuccessfulResponse(response);
   const status = response.statusCode ?? 0;
   const contentTypeHeader = responseContentTypeHeader(response);
   const contentType = normalizeContentType(contentTypeHeader);
@@ -458,6 +508,7 @@ async function fetchDocument(
   dependencies: FetchRemoteDependencies,
   cached?: CompleteDocument,
   allowEmptyContent = false,
+  retryRepresentationMiss = true,
 ): Promise<{ document: CompleteDocument; revalidated: boolean }> {
   assertAbsoluteHttpUrlForFetch(normalizeGitHubRawUrl(rawUrl));
   const controller = new AbortController();
@@ -499,8 +550,10 @@ async function fetchDocument(
     // Some servers route Markdown negotiation to a missing representation even
     // though their HTML page exists (for example, Redis documentation URLs).
     // Retry only these representation-related statuses, through the same
-    // validated redirect/network boundary, with HTML preferred.
-    if ([404, 406].includes(response.statusCode ?? 0)) {
+    // validated redirect/network boundary, with HTML preferred. Callers that probe
+    // for one exact representation opt out: HTML from that URL is never the
+    // representation they are looking for.
+    if (retryRepresentationMiss && [404, 406].includes(response.statusCode ?? 0)) {
       response.resume();
       ({ target, response } = await requestFollowingRedirects(
         target.url,
@@ -566,13 +619,30 @@ async function fetchDocument(
   }
 }
 
+/**
+ * Fetches a complete document for one URL.
+ *
+ * @param allowEmptyContent - Whether an empty body is acceptable
+ * @param retryRepresentationMiss - Whether a 404/406 may be retried as HTML. Probe
+ *   callers that want one exact representation pass false.
+ */
 export async function fetchCompleteDocument(
   rawUrl: string,
   signal: AbortSignal | undefined,
   dependencies: FetchRemoteDependencies,
   allowEmptyContent = false,
+  retryRepresentationMiss = true,
 ): Promise<CompleteDocument> {
-  return (await fetchDocument(rawUrl, signal, dependencies, undefined, allowEmptyContent)).document;
+  return (
+    await fetchDocument(
+      rawUrl,
+      signal,
+      dependencies,
+      undefined,
+      allowEmptyContent,
+      retryRepresentationMiss,
+    )
+  ).document;
 }
 
 /** Revalidates a stale representation with its ETag and Last-Modified validators. */

package/src/acquisition/llms.ts

@@ -1,6 +1,10 @@
 import { ExpiringLruCache } from "../cache";
 import type { CompleteDocument } from "../document/content.ts";
-import { fetchCompleteDocument, type FetchRemoteDependencies } from "./fetch.ts";
+import {
+  ChallengeBlockedError,
+  fetchCompleteDocument,
+  type FetchRemoteDependencies,
+} from "./fetch.ts";
 import { InflightCoalescer } from "../inflight";
 
 const CACHE_TTL_MS = 24 * 60 * 60 * 1_000;
@@ -97,8 +101,13 @@ interface LlmsTxtProbe {
 
 const MAX_LLMS_TXT_PROBE_ENTRIES = 512;
 
-/** Deepest ancestor-directory `/llms.txt` probed, e.g. `/r2/x/y` probes `/r2/llms.txt`. */
-const MAX_LLMS_TXT_DIRECTORY_DEPTH = 1;
+/**
+ * Deepest ancestor-directory `/llms.txt` probed, e.g. `/a/b/c` also probes
+ * `/a/b/llms.txt`. Locale-prefixed documentation sites publish their section index
+ * one level below the locale root, so two levels are needed to reach
+ * `/en/docs/llms.txt` from `/en/docs/catalog/...`.
+ */
+const MAX_LLMS_TXT_DIRECTORY_DEPTH = 2;
 
 const llmsTxtProbes = new ExpiringLruCache<string, LlmsTxtProbe>(
   MAX_LLMS_TXT_PROBE_ENTRIES,
@@ -179,10 +188,16 @@ async function probeUsableRawText(
       if (existing) return existing;
 
       try {
-        const document = await fetchCompleteDocument(candidate.toString(), sharedSignal, {
-          ...dependencies,
-          solveAnubisChallenges: false,
-        });
+        const document = await fetchCompleteDocument(
+          candidate.toString(),
+          sharedSignal,
+          { ...dependencies, solveAnubisChallenges: false },
+          // An index probe never accepts an empty body.
+          false,
+          // A probe wants exactly this representation; retrying as HTML only burns
+          // a request on a URL that is already known to be missing.
+          false,
+        );
 
         const available = isUsableLlmsTxtIndex(document) ? document : undefined;
         const result = { document: available, expires: Date.now() + CACHE_TTL_MS };
@@ -226,8 +241,8 @@ function isSafeToProbe(candidate: URL | undefined, primaryUrl: string): boolean
  *
  * - Probes the site's `/llms.txt` index once per origin per TTL window, in parallel with
  *   the primary fetch, so later calls can advertise it.
- * - Serves the index instead of the page when the page looks like an app shell or carries
- *   too little readable text.
+ * - Serves the index instead of the page when the page looks like an app shell, carries
+ *   too little readable text, or was withheld by a bot-protection challenge.
  * - Otherwise annotates the page with the index URL so agents can discover sibling pages.
  *
  * A missing or useless index can never degrade the primary fetch.
@@ -247,16 +262,32 @@ export async function fetchDocumentWithLlmsTxtSupport(
 
   const candidates = buildLlmsTxtCandidateUrls(rawUrl);
 
-  const [primary, blindIndex] = await Promise.all([
+  // A withheld page must not discard an index that is still usable, so the pair is
+  // settled instead of rejected together.
+  const [primarySettled, indexSettled] = await Promise.allSettled([
     fetchCompleteDocument(rawUrl, signal, dependencies, true),
     candidates.length > 0
       ? ensureLlmsTxtIndex(candidates, signal, dependencies)
       : Promise.resolve(undefined),
   ]);
 
+  // An index probe only rejects when the caller cancels, which must still surface.
+  if (indexSettled.status === "rejected") throw indexSettled.reason;
+
+  if (primarySettled.status === "rejected") {
+    // A bot-protection layer replaced the page with a challenge that carries no
+    // content; the discovered index is the only remaining pointer to that content.
+    if (indexSettled.value && primarySettled.reason instanceof ChallengeBlockedError)
+      return { ...indexSettled.value.document, llmsTxtFallback: true };
+
+    throw primarySettled.reason;
+  }
+
+  const primary = primarySettled.value;
+
   // A `describedby` advertisement names the covering index authoritatively per
   // llmstxt.org v2, so it outranks the blind root/section probes whenever usable.
-  let index = blindIndex;
+  let index = indexSettled.value;
 
   if (primary.llmsTxtDescribedBy) {
     const describedBy = toAbsoluteUrl(primary.llmsTxtDescribedBy);

package/src/document/content.ts

@@ -33,6 +33,10 @@ export interface CompleteDocument {
   cachedAt?: number;
   /** True when this document is the site's /llms.txt served instead of a low-quality page. */
   llmsTxtFallback?: boolean;
+  /** Code-fence language when the body was returned as reformatted source. */
+  sourceLanguage?: string;
+  /** Whether the response is a standalone OpenAPI or Swagger document. */
+  openapiSpec?: boolean;
   /** Set when the site publishes a usable /llms.txt index and this document is not that index. */
   llmsTxtIndexUrl?: string;
   /** Absolute URL advertised via `rel="describedby"` (header or HTML link) for this page. */

package/src/document/focus.ts

@@ -22,27 +22,130 @@ interface Section {
   tokens: string[];
 }
 
+interface SplitSections {
+  sections: Section[];
+  /** Separator that rejoins selected sections; windowed source stays contiguous. */
+  separator: "\n" | "\n\n";
+}
+
+/** Target size of one section when a block has no headings and no blank lines. */
+const WINDOW_TARGET_CHARACTERS = 8_000;
+
+/** Fewest lines a headingless block needs before it is split into windows. */
+const WINDOW_MIN_LINES = 80;
+
+/** Fewest characters a headingless block needs before it is split into windows. */
+const WINDOW_MIN_CHARACTERS = 12_000;
+
 function tokens(value: string): string[] {
   return (value.toLowerCase().match(WORD_PATTERN) ?? []).filter((token) => token.length > 1);
 }
 
+/**
+ * Splits one headingless, blank-line-free block into line-aligned windows.
+ *
+ * Formatted source and log files have neither headings nor blank lines, so treating the
+ * whole body as a single section would make focus select everything. Windows are bounded
+ * by characters but never cut a line, and short blocks keep their single section.
+ */
+function splitIntoWindows(block: string): string[] {
+  const lines = block.split(/\r?\n/);
+
+  if (lines.length < WINDOW_MIN_LINES && block.length < WINDOW_MIN_CHARACTERS) return [block];
+
+  const windows: string[] = [];
+  let current: string[] = [];
+  let size = 0;
+
+  for (const line of lines) {
+    if (size > 0 && size + line.length + 1 > WINDOW_TARGET_CHARACTERS) {
+      windows.push(current.join("\n"));
+      current = [];
+      size = 0;
+    }
+
+    current.push(line);
+    size += line.length + 1;
+  }
+
+  const last = current.join("\n");
+
+  if (last) windows.push(last);
+
+  return windows.filter(Boolean);
+}
+
 /**
  * Splits Markdown into sections. Headings stay attached to their following content; a
- * headingless document falls back to paragraphs so focused extraction remains useful for
- * plain-text and basic-extractor responses.
+ * headingless document falls back to paragraphs, and a headingless block without blank
+ * lines falls back to line windows so focused extraction remains useful for plain-text,
+ * log, and reformatted-source responses.
  */
-function splitSections(markdown: string): Section[] {
+function splitSections(markdown: string): SplitSections {
+  // Window a standalone fenced source body without leaving partial Markdown fences.
+  const fenced = /^( {0,3})(`{3,}|~{3,})([^\n\r]*)\r?\n([\s\S]*?)\r?\n\1\2\s*$/.exec(markdown);
+
+  const hasInteriorClose = fenced?.[4].split(/\r?\n/).some((line) => {
+    const close = /^[ \t]{0,3}(`{3,}|~{3,})[ \t]*$/.exec(line);
+
+    return close && close[1][0] === fenced[2][0] && close[1].length >= fenced[2].length;
+  });
+
+  if (fenced && !hasInteriorClose) {
+    const windows = splitIntoWindows(fenced[4]);
+
+    if (windows.length > 1) {
+      return {
+        sections: windows.map((window) => ({
+          markdown: `${fenced[1]}${fenced[2]}${fenced[3]}\n${window}\n${fenced[1]}${fenced[2]}`,
+          tokens: tokens(window),
+        })),
+        separator: "\n\n",
+      };
+    }
+
+    return { sections: [{ markdown, tokens: tokens(fenced[4]) }], separator: "\n\n" };
+  }
+
   const blocks = markdown
     .split(/\n{2,}/)
     .map((block) => block.trim())
     .filter(Boolean);
 
-  if (blocks.length === 0) return [];
+  if (blocks.length === 0) return { sections: [], separator: "\n\n" };
 
   const lines = markdown.split(/\r?\n/);
-  const hasHeadings = lines.some((line) => HEADING_PATTERN.test(line));
+  let fence: { marker: string; length: number } | undefined;
+
+  const headingLines = lines.map((line) => {
+    const match = /^[ \t]{0,3}(`{3,}|~{3,})(.*)$/.exec(line);
+
+    if (match) {
+      if (!fence) fence = { marker: match[1][0], length: match[1].length };
+      else if (match[1][0] === fence.marker && match[1].length >= fence.length && !match[2].trim())
+        fence = undefined;
+
+      return false;
+    }
+
+    return !fence && HEADING_PATTERN.test(line);
+  });
+
+  const hasHeadings = headingLines.some(Boolean);
+
+  if (!hasHeadings) {
+    // Keep mixed fenced Markdown intact rather than splitting a fence at blank lines.
+    if (lines.some((line) => /^[ \t]{0,3}(`{3,}|~{3,})/.test(line))) {
+      return { sections: [{ markdown, tokens: tokens(markdown) }], separator: "\n\n" };
+    }
 
-  if (!hasHeadings) return blocks.map((block) => ({ markdown: block, tokens: tokens(block) }));
+    const windowed = blocks.flatMap(splitIntoWindows);
+
+    return {
+      sections: windowed.map((block) => ({ markdown: block, tokens: tokens(block) })),
+      separator: blocks.length === 1 && windowed.length > 1 ? "\n" : "\n\n",
+    };
+  }
 
   const sections: string[] = [];
   let current: string[] = [];
@@ -54,14 +157,17 @@ function splitSections(markdown: string): Section[] {
     current = [];
   };
 
-  for (const line of lines) {
-    if (HEADING_PATTERN.test(line)) flush();
+  for (const [index, line] of lines.entries()) {
+    if (headingLines[index]) flush();
     current.push(line);
   }
 
   flush();
 
-  return sections.map((section) => ({ markdown: section, tokens: tokens(section) }));
+  return {
+    sections: sections.map((section) => ({ markdown: section, tokens: tokens(section) })),
+    separator: "\n\n",
+  };
 }
 
 /**
@@ -72,7 +178,7 @@ function splitSections(markdown: string): Section[] {
 export function focusMarkdown(markdown: string, query: string): FocusResult {
   const normalizedQuery = query.trim();
   const queryTerms = [...new Set(tokens(normalizedQuery))];
-  const sections = splitSections(markdown);
+  const { sections, separator } = splitSections(markdown);
   const originalCharacters = markdown.length;
 
   if (queryTerms.length === 0 || sections.length === 0) {
@@ -126,7 +232,7 @@ export function focusMarkdown(markdown: string, query: string): FocusResult {
     return score > 0;
   });
 
-  const focused = selected.map((section) => section.markdown).join("\n\n");
+  const focused = selected.map((section) => section.markdown).join(separator);
 
   return {
     markdown: focused,

package/src/document/openapi.ts

@@ -0,0 +1,80 @@
+import { parse } from "acorn";
+import { simple } from "acorn-walk";
+import { parseDocument } from "yaml";
+import { Type } from "typebox";
+import { Check } from "typebox/value";
+
+const schema = Type.Union([
+  Type.Object({
+    openapi: Type.String({ pattern: "^3\\.\\d+\\.\\d+$" }),
+    info: Type.Object({}),
+    paths: Type.Object({}),
+  }),
+  Type.Object({ swagger: Type.Literal("2.0"), info: Type.Object({}), paths: Type.Object({}) }),
+]);
+
+// Parsing is synchronous and untrusted. Larger inputs remain readable without classification.
+const MAX_CLASSIFICATION_CHARACTERS = 512 * 1024;
+
+/** Conservatively recognizes a document or a literal object embedded in JavaScript. Never executes source. */
+export function isOpenapiSource(raw: string, contentType: string): boolean {
+  if (raw.length > MAX_CLASSIFICATION_CHARACTERS) return false;
+
+  try {
+    if (contentType === "application/json") return Check(schema, JSON.parse(raw));
+
+    if (contentType === "text/yaml") {
+      const document = parseDocument(raw, { uniqueKeys: true });
+
+      return document.errors.length === 0 && Check(schema, document.toJS({ maxAliasCount: 0 }));
+    }
+
+    if (contentType !== "text/javascript") return false;
+    const ast = parse(raw, { ecmaVersion: "latest", sourceType: "module" });
+    let found = false;
+    simple(ast, {
+      ObjectExpression(node) {
+        let version = false;
+        let info = false;
+        let paths = false;
+
+        for (const property of node.properties) {
+          if (property.type !== "Property" || property.computed || property.kind !== "init")
+            continue;
+
+          const key =
+            property.key.type === "Identifier"
+              ? property.key.name
+              : property.key.type === "Literal"
+                ? property.key.value
+                : undefined;
+
+          if ((key === "openapi" || key === "swagger") && property.value.type === "Literal") {
+            const value = property.value.value;
+            version ||=
+              key === "swagger"
+                ? value === "2.0"
+                : Check(Type.String({ pattern: "^3\\.\\d+\\.\\d+$" }), value);
+          }
+
+          if (key === "info" && property.value.type === "ObjectExpression") info = true;
+
+          if (key === "paths" && property.value.type === "ObjectExpression") paths = true;
+        }
+
+        if (version && info && paths) found = true;
+      },
+    });
+
+    return found;
+  } catch {
+    return false;
+  }
+}
+
+/** Reconstructs detection from older cached canonical representations, without a network request. */
+export function isCachedOpenapi(markdown: string, contentType: string): boolean {
+  const match = /^```(?:json|js|yaml)\r?\n([\s\S]*)\r?\n```\s*$/.exec(markdown);
+
+  return isOpenapiSource(match ? match[1] : markdown, contentType);
+}

package/src/document/source-format.ts

@@ -0,0 +1,180 @@
+/**
+ * Source text that is read as code rather than prose, mapped to the code-fence
+ * language used when one of these bodies is reformatted.
+ */
+const SOURCE_LANGUAGE_BY_CONTENT_TYPE = {
+  "application/json": "json",
+  "text/javascript": "js",
+  "text/yaml": "yaml",
+} as const;
+
+type SourceContentType = keyof typeof SOURCE_LANGUAGE_BY_CONTENT_TYPE;
+
+/** Narrows a media type to the source types that are reformatted. */
+function isSourceContentType(value: string): value is SourceContentType {
+  return Object.hasOwn(SOURCE_LANGUAGE_BY_CONTENT_TYPE, value);
+}
+
+/** Shortest body reformatted; below this a page is returned verbatim. */
+const MINIFIED_MIN_CHARACTERS = 2_000;
+
+/**
+ * Largest body reformatted. Reformatting triples transient memory for a document too
+ * large to cache, so bigger bodies keep their original single-line layout.
+ */
+const MINIFIED_MAX_CHARACTERS = 2 * 1_024 * 1_024;
+
+/**
+ * Shortest average line that still counts as machine-generated. Bundled source is
+ * packed into very long lines even when it keeps a few real line breaks inside block
+ * scalars, while hand-written source keeps short lines.
+ */
+const MINIFIED_MIN_AVERAGE_LINE_CHARACTERS = 300;
+
+/** Deepest nesting that still receives extra indentation. */
+const MAX_FORMAT_DEPTH = 24;
+
+const INDENT = "  ";
+
+const QUOTE_CHARACTERS = new Set(['"', "'", "`"]);
+
+const OPENING_CHARACTERS = new Set(["{", "["]);
+
+const CLOSING_CHARACTERS = new Set(["}", "]"]);
+
+const LINE_ENDING_CHARACTERS = new Set([",", ";"]);
+
+/**
+ * Returns whether a body is machine-generated source rather than hand-written code.
+ *
+ * Only minified bodies are reformatted: a hand-written file already carries the
+ * author's line structure, which is more useful than any generated layout.
+ */
+function isMinified(value: string): boolean {
+  if (value.length < MINIFIED_MIN_CHARACTERS || value.length > MINIFIED_MAX_CHARACTERS)
+    return false;
+
+  let lineBreaks = 0;
+
+  for (let index = 0; index < value.length; index += 1) {
+    if (value[index] === "\n") lineBreaks += 1;
+  }
+
+  return value.length / (lineBreaks + 1) >= MINIFIED_MIN_AVERAGE_LINE_CHARACTERS;
+}
+
+/** Returns the fence language for a content type, or undefined when it is not source. */
+export function sourceLanguageFor(contentType: string): string | undefined {
+  return isSourceContentType(contentType)
+    ? SOURCE_LANGUAGE_BY_CONTENT_TYPE[contentType]
+    : undefined;
+}
+
+/**
+ * Adds line breaks and indentation to minified source without interpreting it.
+ *
+ * Layout whitespace is normalized and every other character is preserved in order.
+ * String literals are tracked only far enough to keep their punctuation from changing
+ * the layout, so no value is parsed, evaluated, or re-serialized.
+ *
+ * @param value - The minified source
+ * @returns Indented source, or undefined when the input was not minified
+ */
+export function indentMinifiedSource(value: string): string | undefined {
+  if (!isMinified(value)) return undefined;
+
+  const lines: string[] = [];
+  let current = "";
+  let depth = 0;
+  let quote: string | undefined;
+  let escaped = false;
+
+  const indent = (): string => INDENT.repeat(Math.min(depth, MAX_FORMAT_DEPTH));
+
+  const flush = (): void => {
+    const line = current.trimEnd();
+
+    if (line) lines.push(line);
+    current = "";
+  };
+
+  for (const character of value) {
+    if (quote !== undefined) {
+      current += character;
+
+      if (escaped) escaped = false;
+      else if (character === "\\") escaped = true;
+      else if (character === quote) quote = undefined;
+
+      continue;
+    }
+
+    if (QUOTE_CHARACTERS.has(character)) {
+      quote = character;
+      current += character;
+
+      continue;
+    }
+
+    if (OPENING_CHARACTERS.has(character)) {
+      current += character;
+      flush();
+      depth += 1;
+      current = indent();
+
+      continue;
+    }
+
+    if (CLOSING_CHARACTERS.has(character)) {
+      flush();
+      depth = Math.max(0, depth - 1);
+      current = indent() + character;
+
+      continue;
+    }
+
+    if (LINE_ENDING_CHARACTERS.has(character)) {
+      current += character;
+      flush();
+      current = indent();
+
+      continue;
+    }
+
+    if (character === "\n") {
+      flush();
+      current = indent();
+
+      continue;
+    }
+
+    if (character !== "\r") current += character;
+  }
+
+  flush();
+
+  return lines.join("\n");
+}
+
+/**
+ * Reformats minified JavaScript, JSON, or YAML as a fenced code block.
+ *
+ * @param raw - The decoded response body
+ * @param contentType - The normalized response media type
+ * @returns A fenced, indented Markdown block, or undefined when the body is not
+ *   source that benefits from reformatting
+ */
+export function formatMinifiedSource(
+  raw: string,
+  contentType: string,
+): { markdown: string; language: string } | undefined {
+  const language = sourceLanguageFor(contentType);
+
+  if (language === undefined) return undefined;
+
+  const indented = indentMinifiedSource(raw);
+
+  if (indented === undefined) return undefined;
+
+  return { language, markdown: `\`\`\`${language}\n${indented}\n\`\`\`` };
+}

package/src/service.ts

@@ -13,6 +13,7 @@ import { sliceCompleteDocument, type CompleteDocument } from "./document/content
 import { revalidateCompleteDocument, type FetchRemoteDependencies } from "./acquisition/fetch.ts";
 import { fetchDocumentWithLlmsTxtSupport } from "./acquisition/llms.ts";
 import { focusMarkdown, type FocusDetails } from "./document/focus.ts";
+import { isCachedOpenapi } from "./document/openapi.ts";
 import { resolveFragmentOffset, urlWithoutFragment } from "./document/fragments.ts";
 import { InflightCoalescer } from "./inflight";
 import { createDocumentOutline, type DocumentOutline } from "./document/outline.ts";
@@ -43,6 +44,7 @@ const encoder = new TextEncoder();
 export type ContentKind =
   | "repository-readme"
   | "code-file"
+  | "openapi-spec"
   | "directory-listing"
   | "llms-index"
   | "article"
@@ -73,14 +75,24 @@ function safeUrl(value: string): URL | undefined {
  * @param url - The fetched URL
  * @param extractor - Extraction method used
  * @param shellSuspected - Whether the page appears to be an app shell
+ * @param sourceLanguage - Code-fence language when the body was reformatted as source
  * @returns Content classification
  */
 export function classifyContentKind(
   url: string,
   extractor: CompleteDocument["extractor"],
   shellSuspected: boolean,
+  sourceLanguage?: string,
+  openapiSpec = false,
 ): ContentKind {
   if (shellSuspected) return "markup-shell";
+
+  if (openapiSpec) return "openapi-spec";
+
+  // A reformatted body is code whatever host served it: the response was read as
+  // source, not as page prose.
+  if (sourceLanguage !== undefined) return "code-file";
+
   const parsed = safeUrl(url);
   const host = parsed?.hostname ?? "";
   const path = parsed?.pathname ?? "";
@@ -423,6 +435,7 @@ export async function executeWebFetch(
 
   const fresh = Boolean(
     document &&
+    !document.llmsTxtFallback &&
     !document.extractionDiagnostics?.botWall &&
     (document.cachedAt === undefined || now - document.cachedAt < CACHE_TTL_MS),
   );
@@ -576,7 +589,15 @@ export async function executeWebFetch(
   const rawFinalUrl = result.url;
   const finalUrl = redactUrlForDisplay(rawFinalUrl);
   const shellSuspected = result.shellSuspected;
-  const contentKind = classifyContentKind(rawFinalUrl, result.extractor, shellSuspected);
+
+  const contentKind = classifyContentKind(
+    rawFinalUrl,
+    result.extractor,
+    shellSuspected,
+    result.sourceLanguage,
+    document.openapiSpec ?? isCachedOpenapi(document.markdown, document.contentType),
+  );
+
   const confidence = classifyConfidence(result.extractor, shellSuspected, result.markdown.length);
 
   const requestedPath = safeUrl(params.url)?.pathname.replace(/\/+$/u, "").toLowerCase();
@@ -603,10 +624,7 @@ export async function executeWebFetch(
             ]
           : []),
         ...(showLlmsNotice && result.llmsTxtIndexUrl
-          ? [
-              `Documentation index available: ${redactUrlForDisplay(result.llmsTxtIndexUrl)}. Use it if this page does not cover the topic.`,
-              "",
-            ]
+          ? [`Index: ${redactUrlForDisplay(result.llmsTxtIndexUrl)}`, ""]
           : []),
         ...(resolvedFragment.fragment !== undefined && !fragmentMatched
           ? [