@zeldrisho/pi-web-fetch 0.16.1 → 0.16.3

package/CHANGELOG.md

@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.16.3] - 2026-11-01
+
+### Fixed
+
+- Improve focus queries for identifiers, common words, word variants, HTML headings, and oversized content.
+- Keep fragment-focused continuation offsets on the same view.
+- Shorten focus, error, fallback, and end-of-results notices.
+
+## [0.16.2] - 2026-10-30
+
+### Changed
+
+- Remove the package's Node.js engine restriction; Node.js 24 remains the tested runtime.
+- Clarify the model-facing tool description to reflect support for public URLs beyond webpages.
+
 ## [0.16.1] - 2026-10-28
 
 ### Changed
@@ -301,7 +316,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Add bounded public page fetching ([b979496](https://github.com/zeldrisho/pi-packages/commit/b979496b32de1cead172ba570307e4a4a7b3421d))
 
-[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.16.1...HEAD
+[Unreleased]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.16.3...HEAD
+[0.16.3]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.16.2...pi-web-fetch-v0.16.3
+[0.16.2]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.16.1...pi-web-fetch-v0.16.2
 [0.16.1]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.16.0...pi-web-fetch-v0.16.1
 [0.16.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.15.0...pi-web-fetch-v0.16.0
 [0.15.0]: https://github.com/zeldrisho/pi-packages/compare/pi-web-fetch-v0.14.1...pi-web-fetch-v0.15.0

package/README.md

@@ -1,6 +1,6 @@
 # @zeldrisho/pi-web-fetch
 
-Fetch public HTTP(S) pages and source files as bounded Markdown with Pi. No API key is required.
+Fetch bounded content from public HTTP(S) pages and source files with Pi. No API key is required.
 
 ## Install
 
@@ -17,13 +17,13 @@ Use `pi config` to enable or disable the extension, and `pi list` to see configu
 
 The `web_fetch` tool accepts public HTTP and HTTPS URLs and supports HTML, Markdown, plain text, JSON, and XML. HTML is converted to Markdown; images are omitted. Minified JavaScript, JSON, and YAML are returned as indented code blocks so long single-line payloads stay readable. URLs with credentials, local or private network targets, unsafe redirects, downloads over 5 MiB, and unsupported content types are rejected.
 
-| Parameter | Description                                                                                               |
-| --------- | --------------------------------------------------------------------------------------------------------- |
-| `url`     | Required page URL. URL fragments can start output at a matching heading or anchor.                        |
-| `query`   | Optional; selects matching sections from a long page. Sections remain in source order.                    |
-| `offset`  | Optional character offset for continuing through extracted content; takes precedence over a URL fragment. |
+| Parameter | Description                                                                                                                                                                                         |
+| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `url`     | Required page URL. URL fragments can start output at a matching heading or anchor.                                                                                                                  |
+| `query`   | Optional; selects matching sections from a long page. Sections remain in source order.                                                                                                              |
+| `offset`  | Optional character offset for continuing through extracted content. With a query and fragment, offsets apply to the fragment-focused view; otherwise, an offset takes precedence over the fragment. |
 
-Output is bounded to 6,000 characters by default. For long pages, use the returned `nextOffset` to continue. Use `query` to focus on matching sections, or omit it to read the complete document. Pi's preview is collapsed by default; press `Ctrl+O` to expand it.
+Output is bounded to 6,000 characters by default. For long pages, use the returned `nextOffset` to continue with the same URL and query. Use `query` to focus on matching sections, or omit it to read the complete document. Queries use approximate word matching and relevance selection; dotted or hyphenated identifiers and flags are matched literally. Focus can omit relevant content, so omit the query when completeness matters. Pi's preview is collapsed by default; press `Ctrl+O` to expand it.
 
 For example, a URL such as `https://github.com/zeldrisho/pi-packages/blob/main/packages/pi-web-fetch/README.md#usage` starts at the matching heading. GitHub `blob` links and bare gists are converted to raw source URLs; repository roots use their README. GitHub directory listings are not supported.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zeldrisho/pi-web-fetch",
-  "version": "0.16.1",
+  "version": "0.16.3",
   "description": "Pi extension for secure, bounded public web page fetching and Markdown extraction",
   "keywords": [
     "pi",
@@ -41,17 +41,14 @@
     "@earendil-works/pi-tui": "^1.4.0",
     "typebox": "^1.4.2",
     "typescript": "^7.0.2",
-    "vite": "npm:@voidzero-dev/vite-plus-core@1.3.0",
-    "vite-plus": "1.3.0"
+    "vite": "npm:@voidzero-dev/vite-plus-core@1.4.0",
+    "vite-plus": "1.4.0"
   },
   "peerDependencies": {
     "@earendil-works/pi-coding-agent": "*",
     "@earendil-works/pi-tui": "*",
     "typebox": "*"
   },
-  "engines": {
-    "node": ">=24"
-  },
   "pi": {
     "extensions": [
       "./src/index.ts"

package/src/acquisition/fetch.ts

@@ -221,11 +221,14 @@ function assertSuccessfulResponse(response: IncomingMessage): void {
     );
   }
 
-  const authenticationHint = [401, 403, 404].includes(status)
-    ? " The page may be missing, private, or require authentication."
-    : "";
-
-  throw new Error(`web_fetch returned HTTP ${status}.${authenticationHint}`);
+  const hint =
+    status === 404
+      ? " Page not found or inaccessible."
+      : status === 401 || status === 403
+        ? " Access denied or authentication required."
+        : "";
+
+  throw new Error(`HTTP ${status}.${hint}`);
 }
 
 /**

package/src/document/content.ts

@@ -119,7 +119,7 @@ export function sliceCompleteDocument(
   if (truncated) {
     markdown += `\n\n[Content truncated. Continue with offset=${end} to read the next chunk.]`;
   } else if (offset > 0) {
-    markdown += "\n\n[End of page content.]";
+    markdown += "\n\n[End of results.]";
   }
 
   return {

package/src/document/focus.ts

@@ -1,6 +1,6 @@
 const WORD_PATTERN = /[\p{L}\p{N}_+#]+/gu;
 
-const HEADING_PATTERN = /^#{1,6}\s+\S/;
+const HEADING_PATTERN = /^(?:#{1,6}\s+\S| {0,3}<h[1-6](?:\s[^>]*|)>)/i;
 
 /** Metadata describing deterministic query-focused extraction. */
 export interface FocusDetails {
@@ -35,18 +35,95 @@ const WINDOW_TARGET_CHARACTERS = 8_000;
 const WINDOW_MIN_LINES = 80;
 
 /** Fewest characters a headingless block needs before it is split into windows. */
-const WINDOW_MIN_CHARACTERS = 12_000;
+const WINDOW_MIN_CHARACTERS = 6_000;
+
+const STOP_WORDS = new Set([
+  "a",
+  "an",
+  "and",
+  "are",
+  "for",
+  "how",
+  "in",
+  "is",
+  "of",
+  "on",
+  "the",
+  "to",
+  "with",
+]);
 
 function tokens(value: string): string[] {
-  return (value.toLowerCase().match(WORD_PATTERN) ?? []).filter((token) => token.length > 1);
+  return (value.toLowerCase().match(WORD_PATTERN) ?? []).map((token) => {
+    let word = token;
+
+    if (word.length > 5 && /(?:ss|ch|sh|x|z)es$/.test(word)) word = word.slice(0, -2);
+    else if (word.length > 4 && word.endsWith("s") && !word.endsWith("ss"))
+      word = word.slice(0, -1);
+
+    if (word.length > 6 && word.endsWith("ion")) return word.slice(0, -3);
+
+    if (word.length > 4 && /(?:ing|ed)$/.test(word)) return word.replace(/(?:ing|ed)$/, "");
+
+    if (word.length > 4 && /(?:ch|sh|v)e$/.test(word)) return word.slice(0, -1);
+
+    return word;
+  });
+}
+
+function containsIdentifier(markdown: string, identifier: string): boolean {
+  const text = markdown.toLowerCase();
+  let index = text.indexOf(identifier);
+
+  while (index !== -1) {
+    const before = text[index - 1] ?? "";
+    const after = text[index + identifier.length] ?? "";
+
+    const beforeWord =
+      /[\p{L}\p{N}_+#-]/u.test(before) ||
+      (before === "." && /[\p{L}\p{N}]/u.test(text[index - 2] ?? ""));
+
+    const afterWord =
+      /[\p{L}\p{N}_+#-]/u.test(after) ||
+      (after === "." && /[\p{L}\p{N}]/u.test(text[index + identifier.length + 1] ?? ""));
+
+    if (!beforeWord && !afterWord) return true;
+    index = text.indexOf(identifier, index + 1);
+  }
+
+  return false;
+}
+
+/** Split long lines at whitespace without breaking identifiers or surrogate pairs. */
+function splitLongLine(line: string): string[] {
+  const parts: string[] = [];
+  let start = 0;
+  const target = WINDOW_TARGET_CHARACTERS / 2;
+
+  while (line.length - start > target) {
+    let end = start + target;
+
+    while (end > start && !/\s/u.test(line[end])) end -= 1;
+
+    if (end === start) {
+      end = start + target;
+
+      while (end < line.length && !/\s/u.test(line[end])) end += 1;
+    }
+
+    if (end >= line.length) break;
+    parts.push(line.slice(start, end));
+    start = end;
+  }
+
+  if (start < line.length) parts.push(line.slice(start));
+
+  return parts;
 }
 
 /**
- * Splits one headingless, blank-line-free block into line-aligned windows.
- *
- * Formatted source and log files have neither headings nor blank lines, so treating the
- * whole body as a single section would make focus select everything. Windows are bounded
- * by characters but never cut a line, and short blocks keep their single section.
+ * Splits headingless blocks into line-aligned windows, splitting long lines at
+ * whitespace when possible. Unbroken tokens stay intact even if oversized.
  */
 function splitIntoWindows(block: string): string[] {
   const lines = block.split(/\r?\n/);
@@ -58,6 +135,15 @@ function splitIntoWindows(block: string): string[] {
   let size = 0;
 
   for (const line of lines) {
+    if (line.length > WINDOW_TARGET_CHARACTERS / 2) {
+      if (current.length) windows.push(current.join("\n"));
+      current = [];
+      size = 0;
+
+      windows.push(...splitLongLine(line));
+      continue;
+    }
+
     if (size > 0 && size + line.length + 1 > WINDOW_TARGET_CHARACTERS) {
       windows.push(current.join("\n"));
       current = [];
@@ -134,9 +220,41 @@ function splitSections(markdown: string): SplitSections {
   const hasHeadings = headingLines.some(Boolean);
 
   if (!hasHeadings) {
-    // Keep mixed fenced Markdown intact rather than splitting a fence at blank lines.
+    // Split mixed prose and fenced blocks only at blank lines outside fences.
     if (lines.some((line) => /^[ \t]{0,3}(`{3,}|~{3,})/.test(line))) {
-      return { sections: [{ markdown, tokens: tokens(markdown) }], separator: "\n\n" };
+      const parts: string[] = [];
+      let current: string[] = [];
+      let fence: { marker: string; length: number } | undefined;
+
+      for (const line of lines) {
+        const match = /^[ \t]{0,3}(`{3,}|~{3,})(.*)$/.exec(line);
+
+        if (match) {
+          if (!fence) fence = { marker: match[1][0], length: match[1].length };
+          else if (
+            match[1][0] === fence.marker &&
+            match[1].length >= fence.length &&
+            !match[2].trim()
+          )
+            fence = undefined;
+        }
+
+        if (!fence && !line.trim()) {
+          const part = current.join("\n").trim();
+
+          if (part) parts.push(part);
+          current = [];
+        } else current.push(line);
+      }
+
+      const part = current.join("\n").trim();
+
+      if (part) parts.push(part);
+
+      return {
+        sections: parts.map((part) => ({ markdown: part, tokens: tokens(part) })),
+        separator: "\n\n",
+      };
     }
 
     const windowed = blocks.flatMap(splitIntoWindows);
@@ -164,8 +282,55 @@ function splitSections(markdown: string): SplitSections {
 
   flush();
 
+  // Split oversized sections at blank lines outside fences, so matches do not pull
+  // in an entire chapter. Keep the heading with its first content block.
+  const bounded = sections.flatMap((section) => {
+    if (section.length <= WINDOW_MIN_CHARACTERS) return [section];
+    const chunks: string[] = [];
+    let current: string[] = [];
+    let size = 0;
+    let inFence: { marker: string; length: number } | undefined;
+
+    for (const line of section.split("\n")) {
+      const match = /^[ \t]{0,3}(`{3,}|~{3,})(.*)$/.exec(line);
+
+      if (match) {
+        if (!inFence) inFence = { marker: match[1][0], length: match[1].length };
+        else if (
+          match[1][0] === inFence.marker &&
+          match[1].length >= inFence.length &&
+          !match[2].trim()
+        )
+          inFence = undefined;
+      }
+
+      if (!inFence && line.length > WINDOW_TARGET_CHARACTERS / 2) {
+        if (current.length) chunks.push(current.join("\n").trim());
+        current = [];
+        size = 0;
+
+        chunks.push(...splitLongLine(line));
+      } else if (!inFence && !line.trim() && size >= WINDOW_TARGET_CHARACTERS / 2) {
+        const chunk = current.join("\n").trim();
+
+        if (chunk) chunks.push(chunk);
+        current = [];
+        size = 0;
+      } else {
+        current.push(line);
+        size += line.length + 1;
+      }
+    }
+
+    const chunk = current.join("\n").trim();
+
+    if (chunk) chunks.push(chunk);
+
+    return chunks;
+  });
+
   return {
-    sections: sections.map((section) => ({ markdown: section, tokens: tokens(section) })),
+    sections: bounded.map((section) => ({ markdown: section, tokens: tokens(section) })),
     separator: "\n\n",
   };
 }
@@ -177,11 +342,13 @@ function splitSections(markdown: string): SplitSections {
  */
 export function focusMarkdown(markdown: string, query: string): FocusResult {
   const normalizedQuery = query.trim();
-  const queryTerms = [...new Set(tokens(normalizedQuery))];
+  const allTerms = [...new Set(tokens(normalizedQuery))];
+  const meaningfulTerms = allTerms.filter((term) => !STOP_WORDS.has(term));
+  const queryTerms = meaningfulTerms.length ? meaningfulTerms : allTerms;
   const { sections, separator } = splitSections(markdown);
   const originalCharacters = markdown.length;
 
-  if (queryTerms.length === 0 || sections.length === 0) {
+  if (!normalizedQuery || sections.length === 0) {
     return {
       markdown: "",
       details: {
@@ -207,7 +374,14 @@ export function focusMarkdown(markdown: string, query: string): FocusResult {
     );
   }
 
-  const selected = sections.filter((section) => {
+  // Punctuated identifiers and flags are literal, never broad component searches.
+  const literalIdentifier =
+    !/\s/u.test(normalizedQuery) && /[.\-/]/u.test(normalizedQuery)
+      ? normalizedQuery.toLowerCase()
+      : undefined;
+
+  const scores = sections.map((section) => {
+    if (literalIdentifier) return containsIdentifier(section.markdown, literalIdentifier) ? 1 : 0;
     const frequencies = new Map<string, number>();
 
     for (const token of section.tokens) frequencies.set(token, (frequencies.get(token) ?? 0) + 1);
@@ -229,9 +403,15 @@ export function focusMarkdown(markdown: string, query: string): FocusResult {
         ((frequency * (1.2 + 1)) / (frequency + 1.2 * lengthNormalization));
     }
 
-    return score > 0;
+    return score;
   });
 
+  const highestScore = scores.reduce((highest, score) => Math.max(highest, score), 0);
+
+  const selected = sections.filter(
+    (_, index) => scores[index] > 0 && scores[index] >= highestScore * 0.5,
+  );
+
   const focused = selected.map((section) => section.markdown).join(separator);
 
   return {

package/src/index.ts

@@ -100,7 +100,7 @@ export default function (pi: ExtensionAPI) {
   pi.registerTool<typeof webFetchParameters, Partial<WebFetchDetails>>({
     name: "web_fetch",
     label: "Web Fetch",
-    description: "Read a public webpage as Markdown.",
+    description: "Read content from a public URL.",
     parameters: webFetchParameters,
     outputSchema: webFetchOutputSchema,
     annotations: { readOnlyHint: true, destructiveHint: false, openWorldHint: true },
@@ -115,7 +115,21 @@ export default function (pi: ExtensionAPI) {
     },
 
     async execute(_toolCallId, params, signal, onUpdate) {
-      return executeWebFetch(params, signal, onUpdate);
+      try {
+        return await executeWebFetch(params, signal, onUpdate);
+      } catch (error) {
+        if (error instanceof Error && /^web_fetch(?:: | )/.test(error.message)) {
+          const message = error.message.replace(/^web_fetch(?:: | )/, "");
+          throw new Error(
+            message === "was cancelled."
+              ? "Cancelled."
+              : message.charAt(0).toUpperCase() + message.slice(1),
+            { cause: error },
+          );
+        }
+
+        throw error;
+      }
     },
 
     /**

package/src/render.ts

@@ -37,7 +37,7 @@ export function renderCollapsibleOutput(
     );
   }
 
-  if (warning) {
+  if (warning && !visible.includes(warning)) {
     component.addChild(new Spacer(1));
     component.addChild(new Text(theme.fg("warning", warning), 0, 0));
   }

package/src/service.ts

@@ -374,7 +374,8 @@ const inflightFetches = new InflightCoalescer<string, FetchAcquisition>(MAX_INFL
  * Executes a validated web fetch, including caching, focusing, and bounded pagination.
  *
  * A URL fragment selects the starting heading, anchor, or source range unless
- * an explicit offset is supplied. A query selects sections before pagination.
+ * an explicit offset is supplied without a query. A query selects sections before pagination;
+ * its continuation offsets retain the fragment-selected view.
  * Missing fragments may suggest up to three links to other pages with that
  * fragment. Discovered llms.txt index URLs appear on every unblocked result,
  * including cache hits. Detected bot-wall documents return a blocked result
@@ -539,13 +540,13 @@ export async function executeWebFetch(
 
   const startingOffset = explicitOffset ?? fragmentOffset ?? FETCH_DEFAULT_OFFSET;
 
-  const hasFragmentRange =
-    explicitOffset === undefined &&
-    fragmentOffset !== undefined &&
-    resolvedFragment.endOffset !== undefined;
+  const useFragment =
+    fragmentOffset !== undefined && (explicitOffset === undefined || params.query !== undefined);
+
+  const hasFragmentRange = useFragment && resolvedFragment.endOffset !== undefined;
 
   const fragmentDocument =
-    explicitOffset === undefined && fragmentOffset !== undefined
+    useFragment && fragmentOffset !== undefined
       ? {
           ...document,
           markdown: hasFragmentRange
@@ -614,13 +615,13 @@ export async function executeWebFetch(
     : [
         ...(result.markdownAlternateFallback
           ? [
-              "[The requested page looked like an app shell or had little readable text, so this is the Markdown version advertised by the site instead.]",
+              "[Page content was sparse; showing the site's advertised Markdown version instead.]",
               "",
             ]
           : []),
         ...(result.llmsTxtFallback
           ? [
-              "[Showing the site's documentation index instead of the requested page. Follow a relevant link to read the documentation.]",
+              "[Showing the site index instead of the requested page. Follow a link for details.]",
               "",
             ]
           : []),
@@ -629,7 +630,7 @@ export async function executeWebFetch(
           : []),
         ...(resolvedFragment.fragment !== undefined && !fragmentMatched
           ? [
-              "[The requested URL fragment was not found; showing the page from the beginning.]",
+              "[Fragment not found; showing the page from the beginning.]",
               ...(fragmentSuggestions.length
                 ? [`[This page links to the requested fragment: ${fragmentSuggestions.join(", ")}]`]
                 : []),
@@ -640,8 +641,8 @@ export async function executeWebFetch(
         ...(focused
           ? [
               focused.details.matchedSections > 0
-                ? `[Showing ${focused.details.matchedSections} of ${focused.details.totalSections} source sections selected by the focus query. Offsets apply to this focused view; omit query to read the complete document.]`
-                : "[No source sections matched the focus query. Omit query to read the complete document.]",
+                ? `[Query matched ${focused.details.matchedSections}/${focused.details.totalSections} sections. Offsets apply to this view; omit query for the full page.]`
+                : "[No sections matched. Omit query for the full page.]",
               "",
             ]
           : []),