pi-unsloth-webtools 0.1.0

package/README.md

@@ -0,0 +1,101 @@
+# pi-unsloth-webtools
+
+A [pi](https://github.com/earendil-works/pi-coding-agent) extension providing `web_search` and
+`web_fetch` tools, ported from the Unsloth Studio codebase
+([`unslothai/unsloth`](https://github.com/unslothai/unsloth), `studio/backend/core/inference/`).
+
+## What it does
+
+### web_search
+
+Mirrors Unsloth Studio's `web_search` tool:
+
+- Searches exactly like Studio's pinned `ddgs==9.14.4` `DDGS.text()`: the same seven engines
+  (duckduckgo, brave, google, mojeek, yahoo, yandex, wikipedia; bing is disabled upstream),
+  the same provider-deduplication, href-dedupe aggregator with frequency ordering, and the
+  same `SimpleFilterRanker` re-ranking. Formats results identically: `Title:` / `URL:` /
+  `Snippet:` blocks separated by `---`, ending with the hint to pass `{"url": "<URL>"}` to
+  read a full page.
+- Accepts an optional `url` parameter; when given, fetches that page's text instead of searching.
+- Rate-limit, timeout, and empty-result messages mirror Studio's `_search_failure_message`.
+
+### web_fetch
+
+Port of Studio's `_fetch_page_text` / `_fetch_url_raw` pipeline:
+
+- URL scheme normalization (bare hosts like `google.com` become `https://google.com`).
+- URL validation: http/https only, no credentials or encoded hostnames, hostname/port checks.
+- DNS resolution with SSRF protection: every resolved address is validated against
+  private/loopback/link-local/CGNAT/documentation/multicast/reserved ranges, then the validated IP
+  is pinned for the connection (custom `lookup` + SNI `servername`), so DNS cannot rebind between
+  validation and fetch.
+- GitHub repo root pages are rewritten to the unauthenticated README API
+  (`Accept: application/vnd.github.raw+json`), falling back to the HTML page on failure.
+- Up to 5 redirect hops, each re-validated and re-resolved against the same rules.
+- 512 KiB download cap (10 MiB for PDFs), overall deadline + per-hop socket timeouts, abort-aware
+  (`signal` cancels mid-flight).
+- PDF text extraction via the official MuPDF.js engine (the same C library pymupdf wraps):
+  object streams, all filters, ToUnicode fonts, encryption detection, and a
+  pymupdf4llm-style markdown layer (headings, bold/italic, code fences, links, tables)
+  with Studio's corrupted/incomplete fallback to plain text.
+- Content sniffing: MIME allow/deny, binary magic signatures, PDF magic + text extraction
+  (built-in zlib-based extractor, not pymupdf-grade), charset decoding (declared charset, BOM
+  sniffing for UTF-8/16/32, cp1252 rescue for mislabeled single-byte pages).
+- HTML → Markdown conversion ported from Studio's dependency-free `_html_to_md.py`: headings,
+  links, emphasis, lists, tables, blockquotes, code fences, entity decoding; hidden-element
+  stripping (`hidden`, `aria-hidden`, inline styles); `<article>`/`<main>` main-content scoping
+  with link-density header stripping; boilerplate-line removal.
+- No page-size budget: fetched pages and PDFs are returned in full (Studio's window-aware
+  cap is deliberately dropped; the optional `maxChars` parameter still truncates when given).
+  The 512 KiB / 10 MiB download caps still bound the raw fetch.
+- HTML entity decoding replicates CPython's `html.unescape` (full 2,231-entry HTML5 table,
+  longest-prefix rule, Windows-1252 numeric mappings), matching Studio byte-for-byte.
+
+See `ROADMAP.md` for the remaining gaps (PDF extraction, TLS fingerprinting, proxies).
+
+## Install
+
+```sh
+pi install /path/to/pi-unsloth-webtools
+```
+
+or add `npm:pi-unsloth-webtools` (once published) to the `packages` array in
+`~/.pi/agent/settings.json`.
+
+## Development
+
+```sh
+npm install
+npm run typecheck
+npm test
+```
+
+## Tests
+
+The suite ports Unsloth Studio's own tests for these tools:
+
+- `test/html-to-md.test.ts` — hidden-element stripping and main-content scoping (from
+  `test_web_fetch_extraction.py`)
+- `test/header-strip.test.ts` — the header link-density suite plus article-vs-main selection
+  and boilerplate cases (from `test_web_fetch_extraction.py`)
+- `test/binary-guard.test.ts` — the MIME/magic/charset/PDF matrix (from
+  `test_web_fetch_binary_guard.py`)
+- `test/web-search-policy.test.ts` — policy filtering, overfetch, and failure messages
+  (from `test_web_access_policy.py`)
+- `test/fetch-flow.test.ts` — GitHub README rewrite, deadline/cancellation, HTML sniffing
+  (from `test_web_fetch_extraction.py`; the fetch client is injected via seams)
+- `test/engines.test.ts` — the ddgs engine port: normalizers, the XPath subset, the
+  aggregator, the ranker, and the Wikipedia engine with a stubbed fetch
+- `test/pdf-parity.test.ts` — MuPDF engine capabilities: PDF 1.5 object streams,
+  ASCII85Decode, font `/Differences` encodings, pymupdf4llm-style headings/links/tables
+- `test/smoke.test.ts` — live network checks against real hosts
+
+The seams (`seams.resolve` / `seams.request` / `rawFetch`) replace the network stack
+with fakes, mirroring how the Studio suite monkeypatches `_validate_and_resolve_host`
+and `build_opener`.
+
+## License
+
+The ported logic derives from Unsloth Studio
+([AGPL-3.0-only](https://github.com/unslothai/unsloth/blob/main/studio/LICENSE.AGPL-3.0)), so this
+package is released under the same **AGPL-3.0-only** license.

package/ROADMAP.md

@@ -0,0 +1,77 @@
+# Roadmap
+
+Tracking the remaining gaps between this port and Unsloth Studio's web tools, and the
+planned work to close them.
+
+## Implemented
+
+### PDF text extraction (MuPDF engine)
+
+`pdf.ts` uses the official **MuPDF.js** (`mupdf` npm package) — the same C engine that
+PyMuPDF wraps — replacing the earlier minimal built-in extractor:
+
+- Full xref handling: tables, cross-reference streams, and PDF 1.5+ object streams
+- All standard stream filters (FlateDecode, ASCII85Decode, LZW, RunLength, DCT, JPX, ...)
+- Font encodings and ToUnicode mapping (non-Latin text extracts correctly)
+- Encryption detection via `needsPassword()` (reported as unreadable, matching pymupdf's
+  no-password behavior)
+- The markdown layer replicates pymupdf4llm's algorithm: `IdentifyHeaders` font-size
+  heading detection, `get_raw_lines` line reconstruction (tolerance 3, 10% span-join
+  delta), `write_text` styling (bold/italic/mono, code fences, bullets, link
+  resolution with `%0x`-escaped URIs), and Studio's corrupted/incomplete fallback to
+  plain MuPDF text with the exact thresholds from `backend/core/rag/parsers.py`
+- Table detection and pipe-markdown rendering matching pymupdf's `Table.to_markdown`
+  shape (`|header|`, `|---|`, detail rows, `Col{i}` fill for empty headers)
+
+Known deltas vs pymupdf4llm:
+
+- Span-level styling: MuPDF.js's structured-text JSON exposes one font per line, so
+  mixed-style lines (one bold word inside a body line) style the whole line instead of
+  per-span. Line-level styling matches for homogeneous lines.
+- Superscript/subscript/underline/strikeout/highlight markers are not emitted (the
+  JSON does not expose char-level flags).
+- Table detection is a conservative text-grid detector (column-start clustering with
+  a 5 pt tolerance, contiguous multi-row bands) instead of PyMuPDF's
+  `find_tables()` vector-graphics analysis. Aligned text tables are detected; tables
+  defined only by drawn rules without aligned text are not.
+
+The minimal extractor remains as an automatic fallback when the `mupdf` package cannot
+be loaded (for example a stripped install).
+
+## Not planned (explicit decisions)
+
+### Proxy support
+
+Studio routes requests through urllib's environment proxies and honors
+`UNSLOTH_STUDIO_DISABLE_DNS_PINNING` for enterprise proxies. This port always connects
+directly with DNS pinning. Deliberately out of scope.
+
+### Page size budgets
+
+Studio's `_page_char_budget()` sizes fetched pages to the serving model's context window.
+This port deliberately has **no character budget at all**: fetched pages and PDFs are
+returned in full (the PDF page-count cap of 50 pages remains, matching Studio). The raw
+download caps (512 KiB text / 10 MiB PDF) still bound what is fetched. A `maxChars`
+parameter remains available on the tool for callers that want to truncate.
+
+## Known behavioral differences
+
+### Search engine set
+
+The port implements ddgs 9.14.4's `DDGS.text()` exactly: the same seven engines
+(duckduckgo, brave, google, mojeek, yahoo, yandex, wikipedia — bing is `disabled` in
+ddgs upstream), the same provider-deduplication, href-dedupe aggregator with
+frequency ordering, and the same `SimpleFilterRanker` re-ranking. Remaining deltas:
+
+- **TLS fingerprinting**: ddgs uses `primp` with browser TLS impersonation. Node's
+  `fetch` has a different fingerprint, so Google/Brave/Yahoo/Yandex may block or serve
+  consent pages more aggressively. When an engine is blocked it simply contributes no
+  results, exactly as when ddgs is blocked.
+- **User agents**: ddgs uses `fake_useragent`'s database; the port uses a fixed set of
+  browser UAs plus ddgs's own Android Google UA generator.
+
+### Entity decoding
+
+`decodeHtmlEntities` replicates CPython's `html.unescape` exactly (full HTML5 table,
+longest-prefix rule, Windows-1252 numeric mappings, invalid-codepoint handling), so the
+HTML-to-Markdown converter and search-result normalization match Studio's output.