pi-unsloth-webtools 0.9.0 → 0.10.0

package/README.md

@@ -38,13 +38,15 @@ Both tools display their target in the TUI tool row: `web_search "query"` and `w
 
 Mirrors Unsloth Studio's `web_search` tool:
 
-- Searches exactly like Studio's pinned `ddgs==9.14.4` `DDGS.text()`: the same seven engines
-  (duckduckgo, brave, google, mojeek, yahoo, yandex, wikipedia; bing is disabled upstream),
-  the same provider deduplication, href-dedupe aggregator with frequency ordering (hrefs are
+- Searches like Studio's pinned `ddgs==9.14.4` `DDGS.text()`: the Studio engines that still work
+  here (duckduckgo, yandex; the others are behind bot walls, see
+  [Known differences from Studio](#known-differences-from-studio)) plus startpage,
+  the same provider deduplication, and an href-dedupe aggregator (hrefs are
   canonicalized first — `utm_*`/tracking parameters and fragments are dropped and the URL is
   re-serialized, collapsing host-case, default-port, and trailing-slash variants — so the
-  same page found via different tracking links collapses), and the same `SimpleFilterRanker`
-  re-ranking. Formats results identically: `Title:` / `URL:` /
+  same page found via different tracking links collapses). Results are fused with weighted
+  reciprocal-rank fusion: each engine's own ordering, not a keyword guess, decides relevance,
+  and the fused list is capped per registrable domain. Formats results identically:
   `Snippet:` blocks separated by `---`, ending with the hint to call `web_fetch` to
   read a full page.
 - Rate-limit, timeout, and empty-result messages mirror Studio's `_search_failure_message`.
@@ -53,6 +55,11 @@ Mirrors Unsloth Studio's `web_search` tool:
   skipped); timeouts and cancellations are never retried.
 - Sweeps stop as soon as enough results are gathered: engines still in flight are aborted
   instead of being allowed to run to their timeout.
+- Engine requests go through the browser-fingerprint transport first (`webFetch.transport`,
+  default `tls-first`) and fall back to the plain Node transport; a refusal on one transport is
+  retried on the other. Redirects are followed manually, so every hop is re-checked against the
+  website policy and private-address literals are refused. When a SOCKS5 proxy is configured the
+  sweep stays on the plain transport, keeping agent and Tor routing intact.
 
 ### web_fetch
 
@@ -172,9 +179,10 @@ third-party rendering service.
   still gets its chance.
 - Lightpanda identifies itself honestly and refuses to impersonate a browser user agent, so hard
   anti-bot walls are returned as failures (the direct error, or the incomplete-content note).
-- Binary resolution: `lightpanda` on `PATH`, then `PI_LIGHTPANDA_BIN`, then
-  `webRender.lightpandaPath`. Prebuilt binaries exist for Linux (glibc; musl needs a source
-  build) and macOS, plus Docker images; Windows needs WSL2.
+- Binary resolution: `webRender.lightpandaPath`, then `PI_LIGHTPANDA_BIN`, then the launcher
+  installed by `scripts/install-lightpanda.sh`, then `lightpanda` on `PATH`. Prebuilt binaries
+  exist for Linux (glibc; musl needs a source build) and macOS, plus Docker images; Windows
+  needs WSL2.
 - Version matters: 1.0.0 renders JavaScript-heavy pages that the 0.2.x line cannot — measured on
   the same machine, IMDb went from a 76-byte empty document to 21k characters, dribbble from 32
   characters to 18k, and a Medium article from a challenge page to real text. Linux builds from
@@ -186,7 +194,9 @@ third-party rendering service.
 - `bash scripts/install-lightpanda.sh` installs the newest stable release. When the system glibc
   predates what the binary needs, it downloads Debian's `libc6` for the current stable suite,
   extracts it into the install directory, and writes a launcher shim — so 1.0.0 runs on a
-  glibc 2.36 host without touching the system libraries. The script prints the path to use.
+  glibc 2.36 host without touching the system libraries. It also writes
+  `webRender.lightpandaPath` into the global settings, so the extension picks the binary up with
+  no further setup; `--no-configure` skips that and `--print-path` prints the launcher path for CI.
 - Disable the tier with `webRender.lightpandaEnabled: false`; `web_fetch` then stops after the
   network attempts and reports the original error.
 
@@ -215,11 +225,31 @@ third-party rendering service.
   any numeric-only line at a fixed edge position where page numbers appear on at least
   half the pages (so a one-off number sharing that position is dropped too, while fused
   labels like `Page 3 of 12` survive). Studio and pymupdf4llm return them verbatim.
-- Search engines: Node's `fetch` TLS fingerprint differs from ddgs's `primp`
-  impersonation, so Google/Brave/Yahoo/Yandex may block or serve consent pages more
-  aggressively (a blocked engine simply contributes no results). User agents are a
-  fixed browser set plus ddgs's Android Google UA generator, not `fake_useragent`'s
-  database.
+- Search engines: the sweep speaks a browser's TLS/HTTP2 shape through `wreq-js` first (the same
+  transport `web_fetch` uses), so engine fingerprints match Chrome rather than Node's `fetch`;
+  the plain Node transport is the fallback. User agents on that fallback are a fixed browser set,
+  not `fake_useragent`'s database.
+- Startpage: an extra engine beyond Studio's set, matching newer ddgs (its Google-backed index
+  means it fills the `google` provider slot). It answers a plain `GET /sp/search?query=` and
+  serves an Anubis proof-of-work challenge to clients that do not look like a browser, so it works
+  through the browser-fingerprint transport and the fallback's browser headers, but not a bare
+  Node `fetch`. Startpage's POST endpoint and its safesearch parameter are challenge-gated and are
+  not used.
+- Unused engines: mojeek, yahoo, google, brave and wikipedia are not part of the sweep. A
+  non-JavaScript client cannot get past mojeek (JavaScript challenge) or yahoo (`_bv` bot beacon),
+  google and brave answer rate limits and bot interstitials instead of results (HTTP 429 in
+  testing), and wikipedia is an API for a source the other engines already return — so the sweep
+  no longer spends a slot on it, and `wikipedia.org` results are ranked like any other instead of
+  being forced to the top. The remaining three engines can be narrowed further per machine with
+  `webSearch.engines` (see Configuration).
+- Ranking: Studio's `SimpleFilterRanker` (keyword buckets with `wikipedia.org` pinned first) is
+  replaced by weighted reciprocal-rank fusion over each engine's own ordering. On the
+  `npm run engine:eval` query set that lifts mean precision@5 from 0.28 (keyword buckets) to 0.35 —
+  the fusion keeps the ranking signal the engines already computed instead of guessing from query
+  substrings. Weights default to uniform because the three engines overlap so little (mean Jaccard
+  0.07–0.22) that weighting mostly decides which engine dominates the list rather than which result
+  is better. A per-domain cap exists (`webSearch.maxPerHost`) but is off by default: it measurably
+  costs precision, and the uncapped fused top-5 already spans 4.3 of 5 distinct domains.
 - Empty sweeps: ddgs 9.14.4 raises the last engine exception; this port reports a
   timeout whenever any engine timed out, so the timeout message is not masked by later
   generic engine failures. The timeout budget bounds the entire sweep: per-engine
@@ -228,16 +258,16 @@ third-party rendering service.
 - Proxies: Studio routes through environment proxies; this port resolves and pins the target IP and
   tunnels that connection through `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` when the proxy is a
   SOCKS5 proxy (`NO_PROXY` exclusions respected; DNS stays local for the guard). Other proxy
-  schemes fall back to a direct connection. The search path uses the process-wide `fetch`, so an
-  agent-level proxy dispatcher applies there too — see
-  [Companion: rotating exit IPs](#companion-rotating-exit-ips).
+  schemes fall back to a direct connection. The search sweep stays on the process-wide `fetch`
+  whenever a SOCKS5 proxy is configured, so an agent-level proxy dispatcher still applies there —
+  see [Companion: rotating exit IPs](#companion-rotating-exit-ips).
 - Dedup and titles: the aggregator keys on canonicalized hrefs (`utm_*`/tracking parameters
   and fragments stripped, then the URL re-serialized); fetched HTML pages are prefixed with
   the document `<title>`. Studio keys on raw hrefs and returns the converted body alone.
 - Upstream drift: current ddgs ships ten backends (adding bing, startpage, grokipedia),
   requires a `vqd` token for DuckDuckGo, and exposes an `extract()` mode. This port
-  deliberately pins the Studio snapshot — seven engines, bing disabled upstream, no vqd,
-  no pagination — so engine behavior matches Studio rather than ddgs head.
+  uses duckduckgo and yandex from the Studio snapshot plus startpage — bing stays
+  disabled, no vqd, no pagination — so engine behavior matches Studio rather than ddgs head.
 
 ## When to use alternatives
 
@@ -263,8 +293,9 @@ choose the best tool per URL. No need to fork this package to add those features
 
 [`pi-tor-proxy`](https://github.com/YuGiMob/pi-tor-proxy) routes pi's in-process `fetch` traffic
 through Tor (it downloads and manages its own Tor binary) and gives each pi instance its own
-circuit and exit IP. The search sweep uses the process-wide `fetch`, so it leaves through the
-current Tor exit, and many search engines rate-limit or challenge per outgoing IP —
+circuit and exit IP. With a SOCKS5 proxy configured, the search sweep stays on the process-wide
+`fetch`, so it leaves through the current Tor exit, and many search engines rate-limit or
+challenge per outgoing IP —
 `/tor-cycle` swaps the exit those limits are counted against, while `/tor-country` and
 `/tor-exclude` constrain which exits are used.
 
@@ -307,9 +338,12 @@ Optional settings in `~/.pi/agent/settings.json` or `.pi/settings.json` (project
 | `websitePolicy` | none | Not read from settings. Tools run unrestricted by default; `websitePolicy` is a programmatic option the host passes to `webSearch` / `fetchPageText` |
 | `unslothWebTools.allowPrivateAddresses` / `webFetch.allowPrivateAddresses` | `true` | Opt out to restore the resolved-IP SSRF guard: private/loopback/link-local hosts (localhost, LAN IPs) are refused again. Non-canonical numeric IP encodings stay blocked either way |
 | `unslothWebTools.allowLocalFiles` / `webFetch.allowLocalFiles` | `true` | Opt out to refuse local files in `web_fetch` (`file://` URLs, absolute, `~/`, or `./` paths); when enabled, PDFs are extracted and HTML converted |
-| `webFetch.transport` / `unslothWebTools.transport` | `tls-first` | Fetch transport order: `tls-first` (default), `direct-first`, or `off` to disable the browser-fingerprint transport entirely |
+| `webFetch.transport` / `unslothWebTools.transport` | `tls-first` | Transport order for `web_fetch` and `web_search` engine requests: `tls-first` (default), `direct-first`, or `off` to disable the browser-fingerprint transport entirely |
+| `webSearch.engines` / `unslothWebTools.engines` | all three (duckduckgo, yandex, startpage) | Restrict `web_search` to a subset of engine names, e.g. `["duckduckgo", "yandex"]`; unknown names are ignored, and a list that matches nothing falls back to every engine |
+| `webSearch.engineWeights` / `unslothWebTools.engineWeights` | all `1` | Per-engine fusion weight, e.g. `{"startpage": 2, "yandex": 0.5}`; only positive numbers are read |
+| `webSearch.maxPerHost` / `unslothWebTools.maxPerHost` | `0` (no cap) | Most results one registrable domain may contribute; `0` disables the cap |
 | `webRender.lightpandaEnabled` / `unslothWebTools.lightpandaEnabled` | `true` | Opt out to disable local Lightpanda rendering |
-| `webRender.lightpandaPath` / `unslothWebTools.lightpandaPath` | `lightpanda` on `PATH` (`PI_LIGHTPANDA_BIN` fallback) | Path to the Lightpanda binary used for local rendering |
+| `webRender.lightpandaPath` / `unslothWebTools.lightpandaPath` | launcher installed by `scripts/install-lightpanda.sh`, else `lightpanda` on `PATH` | Path to the Lightpanda binary used for local rendering |
 | `webRender.lightpandaCommand` / `unslothWebTools.lightpandaCommand` | none | Command prefix that launches the renderer, for WSL (`["wsl.exe","-e","<path>"]`) or containers; overrides `lightpandaPath`. The fetch flags are appended to it |
 
 Environment overrides: `PI_UNSLOTH_CACHE_DIR` changes the fetch cache directory, `PI_UNSLOTH_WEBTOOLS_STATS` opts into append-only sweep stats JSONL, `PI_CODING_AGENT_DIR` / `PI_AGENT_DIR` change the global settings directory, and `PI_LIGHTPANDA_BIN` points at the local renderer binary. Cache entries live 1 hour and stale copies are served only after a network failure. SOCKS5 proxies named by `HTTPS_PROXY`, `HTTP_PROXY`, or `ALL_PROXY` are honored on every fetch (`NO_PROXY` exclusions apply).
@@ -354,36 +388,21 @@ npm test
 npm run test:unit
 npm run test:smoke
 bash scripts/install-lightpanda.sh
-npm run compare:fetch
-npm run compare:browsers
 npm run camoufox:warmup
-npm run stealth:matrix
-/tmp/pyenv/bin/python scripts/stealth-python.py
+npm run engine:eval
 ```
 
-`npm run compare:fetch` runs a live head-to-head of the direct fetch, the TLS-impersonation retry,
-local Lightpanda rendering over a target list. It accepts URLs as arguments and `--no-lightpanda`
-to drop the render tier. `npm run compare:browsers` adds a Camoufox column;
-install it separately (`npm i camoufox-js playwright-core && npx camoufox-js fetch`, plus GTK3
-libraries on Linux) and use `--seconds=N` to bound how long it waits out a JS challenge.
-`npm run stealth:matrix` compares stealth-browser options against one walled page (plus
-`bot.sannysoft.com` detection rows and a plain-page sanity check): raw CDP to a system Chromium
-(`PI_CHROMIUM_BIN` to point at it), Playwright with its bundled Chromium, Patchright, and Camoufox.
-Every browser dependency is loaded through a guarded dynamic import, so nothing is added to
-`package.json`; install whichever rows you want to measure. `--attempts=N` and `--seconds=N` bound
-the walled-page attempts, and passing row names runs a subset (`raw-cdp`, `playwright`, `patchright`,
-`camoufox`).
-
-`scripts/stealth-python.py` is the same idea for the Python-side options (nodriver, CloakBrowser,
-DrissionPage, cloudscraper, curl_cffi) against the same walled page; it needs a venv with those
-packages installed and is not wired into any npm script. Measured findings are in its module docstring.
-
 `npm run camoufox:warmup` measures what a warm Camoufox costs and buys: launch time, idle CPU and
 RSS, per-fetch latency with the browser already running, and whether a persistent profile
 (`user_data_dir`, pinned fingerprint) lets a Cloudflare clearance survive a restart. Flags:
 `--virtual` for a virtual display, `--pin=0` to let Camoufox rotate fingerprints, `--seconds=N`,
 `--idle=N`.
 
+`npm run engine:eval` measures each search engine against a small hand-labelled developer query
+set: yield, latency and failures, pairwise URL overlap, unique contribution, and offline fusion
+simulations with candidate weight vectors and per-host caps. `--cache=FILE` reuses a previous run
+without touching the network; `--delay`, `--timeout`, `--max` and `--transport` tune the sweep.
+
 ## Tests
 
 `npm test` runs the full suite. `npm run test:unit` skips the live-network smoke tests,
@@ -404,7 +423,7 @@ The suite ports Unsloth Studio's own tests for these tools:
 - `test/fetch-flow.test.ts`: GitHub README rewrite, deadline/cancellation, HTML sniffing
   (from `test_web_fetch_extraction.py`; the fetch client is injected via seams)
 - `test/engines.test.ts`: the ddgs engine port, normalizers, the XPath subset, the
-  aggregator, the ranker, and the Wikipedia engine with a stubbed fetch
+  reciprocal-rank aggregator, the per-host cap, and the Startpage engine with a stubbed fetch
 - `test/pdf-parity.test.ts`: MuPDF engine capabilities, PDF 1.5 object streams,
   ASCII85Decode, font `/Differences` encodings, pymupdf4llm-style headings/links/tables
 - `test/entities.test.ts`: `decodeHtmlEntities` parity with CPython `html.unescape`,