pi-wsearch 0.2.0 → 0.2.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-wsearch",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Pi extension: web.search + web.fetch",
   "license": "MIT",
   "type": "module",
@@ -66,9 +66,9 @@
     "src/",
     "README.md",
     "LICENSE",
-    "CHANGELOG.md"
+    "!CHANGELOG.md"
   ],
   "publishConfig": {
     "access": "public"
   }
-}
+}

package/CHANGELOG.md

@@ -1,62 +0,0 @@
-# Changelog
-
-All notable changes to this project are documented in this file.
-
-## [Unreleased]
-
-### Added
-
-- Over-cap sections split into addressable `(cont. N)` outline entries; no section tail is ever unreachable.
-- Promo/nag blocks (premium upsells, newsletter signups) filtered from outline and section bodies.
-- Adjacent verbatim-duplicate paragraphs dropped.
-- Headingless pages get a synthesized `(lead)` outline entry.
-- Heading-only stub sections (page chrome like "## Latest commit", "### Stars" with no body) dropped — a real section carries prose under its heading.
-- GitHub repo pages serve the raw README under the original URL (repo pages are ~90% UI chrome); filename candidates tried in order, graceful fall-through to the normal page path on failure; non-root paths (blob/tree/issues) unaffected.
-- Local extraction parity with paid readers: HTML tables → markdown pipe tables (structure preserved instead of flattened to word soup); MathML formulas → embedded TeX when annotated; KaTeX span soup (client-rendered, no TeX source in HTML) dropped instead of linearizing into scrambled fragments.
-- News-page hard-sample cleanup: per-line junk filtering (photo credits like "Chip Somodevilla / Getty Images file", orphan "38d ago" card fragments, IE-browser nags, "Skip to Content" links, "FOR SUBSCRIBERS" nags), ALL-CAPS card kickers, empty `##` heading debris, bullet leader-line collapsing, and `<title>`-echo blocks dropped — NBC/CBS-class hub pages now open directly on their story list.
-- Broader hard-sample sweep (forums, docs SPAs, helpdesks, recipes, e-commerce): newsletter signup modules ("#### Email Newsletter" + email-field trust lines) dropped; access-notice pages (People Inc-style HTTP 402 with an HTML notice) treated as bot-walls — UA retry + chain fallback, and their notice text can never pass as content.
-- Keyless jina reader tier: runs without a key when jina is explicitly named in `WSEARCH_FETCH_CHAIN` (opt-in, rate-limited ~20 req/min); a configured key also enables it.
-- Reader backends for kagi (Universal Summarizer — last in chain: lossy summaries never pre-empt full extraction; covers PDFs/office docs/YouTube), TinyFish (Fetch API, free at any balance), serper (Webpage Scrape) and You.com (Contents API) — key-gated, docs-verified contracts.
-- Junk list covers social-syndication mirrors (twiscan, twstalker, unrollnow, threadreaderapp, nitter, upd.dev); any numeric `status|thread/<id>` path on any host is junk (uniform rule, no favored-host allowlist).
-- `engines_errored` structured field plus `⚠ engines:` diagnostics line on search output; engines with a stable 401/402/403 error are skipped for the rest of the process (memo cleared on `/websearch login|logout`).
-- Reader-chain exhaustion now reports every backend's error, not just the last one.
-- Intrinsic BM25-flavored lexical rescoring in the fuse stage (title hits 2x, IDF-lite rarity via token length, bounded 0.25 multiplier): single-engine hits with strong query overlap reorder near-ties; multi-engine agreement still dominates. Deterministic, model-free.
-- `/websearch` onboarding lists jina's free tier among the cheapest paths.
-- Observe-only fetch-through ledger (`wsearch/fetchtrace.jsonl`): search records each result's rank/engine-agreement; a successful first fetch of a search-surfaced URL appends the tuple (query, rank, engines, url). Pure measurement — zero behavior change — the ground-truth layer for judging future ranking changes (top-3 fetch-decision precision).
-- URL canonicalization depth-up: paths case-folded and duplicate slashes collapsed (server-equivalent spellings no longer split RRF agreement across "different" URLs).
-- Mojibake detection and repair in the byte-decode path: UTF-8-read-as-CP1252 double-encoding (declared charset lies, upstream proxy mangling) is detected by signature density and re-decoded; clean and declared-latin1 pages pass through untouched.
-- Stale-if-error cache reserve: expired entries survive as a reserve instead of being purged; when a previously-readable page's live fetch fails (wall tightened, host down), the expired copy is served with a `[STALE — ~Nh old]` title marker rather than an error. `get()` never serves expired copies; the reserve is replaced on overwrite and capped by the LRU cap.
-- FineWeb/Gopher repetition gates on extracted text: >30% duplicate lines or >20% repeated unique 5-grams marks the output low-quality (catches link-farms and menu-soup that survive every pattern filter; published thresholds, not invented ones).
-- Conditional revalidation (ETag/If-None-Match, Last-Modified/If-Modified-Since): cached copies store their validators; an expired copy revalidates before re-download — a 304 re-marks it fresh for free (local path only; reader-chain results carry no validators).
-- Fetched pages now carry `publishedAt` (JSON-LD datePublished → article:published_time meta → <time datetime>, first match wins) and a `paywalled: true` flag when soft-paywall overlay signatures (fc-ab-root, metering-modal, isAccessibleForFree:false) are present — detection only, never bypass.
-- GitHub blob URLs (single source files) serve the raw file content under the original URL, same contract as the README preference — the repo-page path buried files under navigation UI.
-- Cookie-consent banner boilerplate ("We use cookies to improve…", "By continuing to browse…", "Accept all", "Manage preferences") filtered.
-- Page-wide duplicate-paragraph removal: paragraphs repeated verbatim anywhere on a page are dropped from their later occurrences (adjacent-dedupe caught only back-to-back copies; hub pages re-serve card text across sections — pure token waste).
-- Accept header prefers `text/markdown` over HTML: sites with server-side content negotiation (agent-readability ecosystem) now hand over their clean markdown rendition; non-negotiating servers ignore the header entirely.
-- Zero-result recovery: when a search returns nothing, one automatic retry runs with operators stripped (`site:`, quotes, `-term`) or the rarest term dropped — the agent no longer burns a round-trip to do it manually; the retry is noted in the output.
-- `site:` operator normalization: routed to native domain-filter params where supported (tavily `include_domains`, exa `includeDomains`), passed through verbatim for engines with native operator support (serper, brave, kagi) — no more diluting domain-restricted queries into full-web text search.
-- Search result titles strip the site-name suffix ("Page Title | Site Name") — display noise and title-token pollution.
-- Search results display a normalized date (ISO, from engine date fields incl. relative forms like "3 days ago") when the engine provides one.
-- Archive fallback for dead pages: a 404 (dead host, removed docs) tries archive.org's Wayback availability API and serves the snapshot through the normal local path, title-marked `[ARCHIVED copy <date> — live page is gone]`. Doctrine line: 404/dead-host only, never 403-walls; best-effort (lookup failure → plain error as before).
-
-### Changed
-
-- Fetch cache key bumped to `v3` (extraction pipeline shape changed).
-- `OUTLINE_CAP` raised from 14 to 20.
-- Reader chain default order: jina (free tier) → tinyfish (free at any balance) → firecrawl → tavily → exa → serper → you → kagi (lossy summarizer last).
-- Permanent local-failure guard narrowed to 404/oversized/empty: 403 bot-walls now advance to the chain (renderers exist to read them); `r.jina.ai`/`s.jina.ai` proxy URLs never enter the chain.
-- Search engines pin `WSEARCH_ENGINES` and readers pin `WSEARCH_FETCH_CHAIN` at runtime as before; chain status display mirrors the new default order.
-
-### Fixed
-
-- `splitSection` infinite loop on space-free runs (base64 blobs, packed table cells) — cut enforces a hard progress floor.
-- Dropped tiny preambles (e.g. leaked `<title>` text) no longer shift heading names off by one.
-- Reader chain was spent on permanent local failures (404/oversized/empty URLs) — now returns the real status instantly instead of burning paid backends.
-- Jina reader gate contradicted its keyless-capable read implementation (blocked keyless reads despite tolerant code) — gate aligned to the explicit opt-in.
-- Dead engines (e.g. firecrawl with exhausted credits) dragged every search fan-out ~7s — skipped after the first stable failure.
-- `/websearch off|on` withdrew/restored tools at registration only, leaving them declared to the model until a manual reload — the active tool set is now synced via `pi.setActiveTools()` so the change applies on the next model request.
-- Fetching `r.jina.ai/<target>` sent paid renderers at a proxy URL (a renderer cannot render a proxy) — now fails fast with the proxy's own status.
-
-### Removed
-
-- `CAP_BREAK_GRACE` boundary heuristic (subsumed by the progress floor).