@staticn0va/wigolo 0.1.0 → 0.1.2

package/LICENSE

@@ -19,7 +19,7 @@ Additional Use Grant: You may make production use of the Licensed Work,
 
 Change Date:          2029-04-12
 
-Change License:       MIT License
+Change License:       GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later)
 
 For information about alternative licensing arrangements, contact:
 ktowhid20@gmail.com

package/README.md

@@ -1,272 +1,191 @@
-<div align="center">
-
 # wigolo
 
-**Local-first web search MCP server for AI coding agents.**
-
-Search, fetch, crawl, cache, and extract — zero API keys, zero cloud, zero cost.
-
-[![License: BSL 1.1](https://img.shields.io/badge/License-BSL_1.1-blue.svg)](LICENSE)
-[![Node.js](https://img.shields.io/badge/node-%3E%3D20-brightgreen)](https://nodejs.org)
-[![TypeScript](https://img.shields.io/badge/TypeScript-5.x-blue)](https://www.typescriptlang.org/)
-
-[Quick Start](#quick-start) · [Features](#features) · [Why wigolo?](#why-wigolo) · [Roadmap](#roadmap)
+Local-first web intelligence MCP server. 8 tools: `search`, `fetch`, `crawl`, `cache`, `extract`, `find_similar`, `research`, `agent`. Runs on Node 20+. No API keys required for the core path.
 
-</div>
+> **Status:** `v0.1.1` — early release. The v1 retrieval engine has shipped but is still **opt-in** (`WIGOLO_SEARCH=v1`); default uses the legacy SearXNG backend. Full v1.0 will land after the cross-tool benchmark + default flip.
 
-```
-$ claude mcp add wigolo -- npx wigolo
-Added MCP server wigolo
+## Install
 
-$ # That's it. Your agent now has web search.
+```bash
+npx @staticn0va/wigolo init
 ```
 
-## What is this?
-
-wigolo gives AI coding agents (Claude Code, Cursor, Gemini CLI, Codex, Windsurf) web search, page fetching, site crawling, content extraction, and a local knowledge cache. It runs entirely on your machine. No API keys, no cloud, no cost — works out of the box with `npx`.
-
-## Quick Start
+The init flow runs a system check, downloads the embedding + reranker models, bootstraps SearXNG, detects installed AI coding agents (Claude Code, Cursor, Gemini CLI, Codex, Windsurf, Zed, OpenCode), and writes MCP config + skill docs for each one.
 
-**Claude Code:**
-```bash
-claude mcp add wigolo -- npx wigolo
-```
+Or wire it yourself in any MCP client:
 
-**Cursor / VS Code / any MCP client:**
 ```json
 {
   "mcpServers": {
     "wigolo": {
       "command": "npx",
-      "args": ["wigolo"]
+      "args": ["-y", "@staticn0va/wigolo"]
     }
   }
 }
 ```
 
-**Optional warmup (improves quality on first use):**
-```bash
-npx wigolo warmup          # Downloads Playwright + SearXNG
-npx wigolo warmup --all    # + ML reranking + Trafilatura extraction
-```
-
-## Prerequisites
-
-- **Node.js 20+** — [Download](https://nodejs.org/) or `brew install node` (macOS) / `winget install OpenJS.NodeJS` (Windows) / `sudo apt install nodejs` (Ubuntu/Debian)
-- **Python 3.8+** *(recommended)* — [Download](https://python.org/) or `brew install python3` (macOS) / `winget install Python.Python.3` (Windows) / `sudo apt install python3` (Ubuntu/Debian)
-- **Docker** *(optional)* — Alternative to Python for running SearXNG.
-
-Everything else (Playwright, SearXNG) is downloaded automatically on first use or via `npx wigolo warmup`.
-
-### What works without Python?
-
-Everything except embedded SearXNG. Without Python, search falls back to direct scraping of Bing, DuckDuckGo, and Startpage — functional but less reliable. All other tools (fetch, crawl, cache, extract) work fully with just Node.js.
-
-## Features
-
-### search
-
-Search the web and get full markdown content in one call — not snippets.
-
-```
-search("React Server Components best practices", { max_results: 5 })
-→ titles, URLs, relevance scores, and full extracted markdown per result
-```
-
-- Domain filtering: `include_domains: ["react.dev"]`, `exclude_domains: ["medium.com"]`
-- Date filtering: `from_date: "2024-01-01"`, `to_date: "2025-01-01"`
-- Category search: `general`, `news`, `code`, `docs`, `papers`
-- ML reranking with FlashRank when installed
-- Falls back to direct engine scraping when SearXNG is unavailable
-
-### fetch
-
-Fetch any URL and get clean markdown. The page-fetching engine behind `search`.
-
-```
-fetch("https://docs.react.dev/reference/react/useState")
-→ clean markdown, links, images, metadata, cached for future use
-```
-
-- Smart routing: HTTP first, Playwright fallback for JS-rendered pages (auto-detected)
-- Section targeting: `section: "Parameters"` extracts content under that heading
-- Authenticated browsing: `use_auth: true` with stored session or Chrome profile
-- PDF support: text extraction via pdf-parse
-
-### crawl
-
-Crawl a site from a seed URL — documentation sites, wikis, anything.
-
-```
-crawl("https://docs.example.com", { strategy: "sitemap", max_pages: 50 })
-→ array of pages with titles, markdown, depth
-```
-
-- Strategies: `bfs`, `dfs`, `sitemap`, `map` (URL discovery only — no content, faster)
-- URL filtering with include/exclude patterns (regex)
-- robots.txt compliance
-- Cross-page content deduplication (strips repeated nav/header/footer)
-- Total character budget to prevent context overflow
+Global install for repeated CLI use:
 
-### cache
-
-Query previously fetched content without hitting the network.
+```bash
+npm i -g @staticn0va/wigolo
+wigolo --help
+```
+
+## CLI
+
+| Command | What it does |
+|---|---|
+| `wigolo` (no args) | Boot MCP server on stdio (used by MCP clients) |
+| `wigolo init` | Interactive onboarding (browser pick, agent detect, MCP config) |
+| `wigolo warmup [--all] [--embeddings] [--reranker]` | Pre-fetch models + bootstrap SearXNG |
+| `wigolo doctor` | Diagnostic: Python, browsers, models, SearXNG, RSS feeds, telemetry |
+| `wigolo health` | Quick OK/degraded exit code |
+| `wigolo serve [--port N]` | Run as HTTP daemon |
+| `wigolo shell` | Interactive REPL against the 8 tools |
+| `wigolo backfill [--dry-run] [--limit N]` | Embed cached pages missing vectors |
+| `wigolo setup mcp` | Wire MCP config into installed agents |
+| `wigolo status` | Show running daemon status |
+| `wigolo plugin <subcommand>` | Manage plugins |
+| `wigolo uninstall` | Remove wigolo install |
+| `wigolo --help` / `wigolo --version` | Help + version |
+
+## The 8 MCP tools
+
+| Tool | Use when |
+|---|---|
+| `search` | Need info on a topic, no URL yet. Pass query string or array of 3-5 keyword variants for breadth. |
+| `fetch` | Have a specific URL. Returns clean markdown + metadata. JS rendering auto-detected. |
+| `crawl` | Need many pages from one site. Strategies: `bfs`, `dfs`, `sitemap`, `map`. |
+| `cache` | Check the local store before going to the network. FTS5 + optional vec hybrid. |
+| `extract` | Specific data points (tables, metadata, schema-shaped fields). Modes: `selector`, `tables`, `metadata`, `schema`, `structured`. |
+| `find_similar` | "More like this" given a URL or concept. Hybrid FTS + embeddings + web expansion. |
+| `research` | Multi-step investigation: decomposition → parallel search → synthesis with citations. |
+| `agent` | Natural-language data gathering across multiple sources with optional JSON schema. |
+
+Each tool surfaces a per-session instruction block (~2 KB) plus a `wigolo://docs/usage` resource with the full routing guide.
+
+## Engine selection
+
+Two retrieval paths today; toggled by env var:
 
-```
-cache({ query: "React hooks", url_pattern: "*react.dev*" })
-→ matching cached pages with full markdown
+```bash
+WIGOLO_SEARCH=v1       # new path: 11 direct engines, intent-routed verticals, RRF, RSS, recency boost
+WIGOLO_SEARCH=searxng  # current default: SearXNG aggregator (legacy)
 ```
 
-- SQLite FTS5 full-text search over all cached content
-- Combined filters: text query + URL pattern + date range
-- Cache stats and selective clearing
+The v1 engine ships:
+- Direct engines per vertical — general (HN Algolia, lobste.rs, DuckDuckGo, Bing, Startpage), news (HN Algolia, lobste.rs, Bing News), code (GitHub Code, StackOverflow), docs (MDN, DevDocs), papers (arXiv, Semantic Scholar)
+- Intent router + weighted RRF orchestrator
+- Date-range intent classifier + recency boost
+- Opt-in RSS feed engine (`WIGOLO_RSS_FEEDS=url1,url2`)
+- `agent_context.recent_urls` dedup with case-insensitive path matching for IIS / archive.org / Microsoft docs
 
-### extract
+Default stays on SearXNG until the cross-tool benchmark gate clears (Phase 16). Flip with `WIGOLO_SEARCH=v1` to try the new path now.
 
-Structured data extraction from any URL or HTML.
+## Local stack
 
-```
-extract("https://example.com/product", { mode: "schema", schema: { price: "string", name: "string" } })
-→ { price: "$29.99", name: "Widget Pro" }
-```
-
-Modes:
-- `selector` — CSS selector → text content
-- `tables` — HTML tables → structured row objects
-- `metadata` — title, description, author, date, JSON-LD
-- `schema` — JSON Schema → heuristic field matching from page content
-
-## Why wigolo?
+- **Search aggregator (legacy path):** SearXNG — bootstrapped to `~/.wigolo/searxng/` on first run (native venv preferred, Docker fallback)
+- **Browser:** Playwright Chromium / Firefox / WebKit; Lightpanda available as a fast JS-renderer alternative
+- **Content extraction:** Defuddle with content-type routing for news / recipe / product / paper / event JSON-LD, plus a Mozilla Readability fallback
+- **Embeddings:** `fastembed` running ONNX `BGE-small-en-v1.5` (384-dim) — cached under `~/.wigolo/fastembed/`
+- **Reranker:** `@huggingface/transformers` cross-encoder `Xenova/ms-marco-MiniLM-L-6-v2` — cached under `~/.wigolo/transformers/`
+- **Cache:** SQLite WAL + FTS5; optional vector hybrid via `sqlite-vec` when the extension is loadable on your platform
+- **Process model:** stdio MCP server by default; HTTP daemon (`wigolo serve`) and REPL (`wigolo shell`) also available
 
-| | wigolo | Tavily | Firecrawl | Exa |
-|---|---|---|---|---|
-| Cost | Free | $30–500/mo | $16–500/mo | $7/1K queries |
-| API key required | None | Yes | Yes | Yes |
-| Authenticated browsing | Yes | No | No | No |
-| Localhost access | Yes | No | No | No |
-| Local cache + FTS | Yes | No | No | No |
-| Search + extract unified | Yes | Yes | Partial | Partial |
-| ML reranking | Local | Proprietary | No | Neural index |
-| Rate limits | None | Tiered | Tiered | Tiered |
+## LLM extraction fallback (optional)
 
-## Configuration
-
-wigolo works with zero configuration. For advanced use:
+`extract` with `mode: "schema"` falls back to an LLM when heuristics miss. Set one of:
 
 ```bash
-# Use an existing SearXNG instance instead of the embedded one
-SEARXNG_URL=http://localhost:8888
-
-# Authenticated browsing — export session state via Playwright
-WIGOLO_AUTH_STATE_PATH=~/.wigolo/auth.json
-
-# Or use your Chrome profile directly (close Chrome first)
-WIGOLO_CHROME_PROFILE_PATH=~/.config/google-chrome/Default
-
-# ML reranking (install with: npx wigolo warmup --reranker)
-WIGOLO_RERANKER=flashrank
-
-# Tune extraction — auto/always/never
-WIGOLO_TRAFILATURA=auto
-
-# Logging
-LOG_LEVEL=info          # debug, info, warn, error
-LOG_FORMAT=json         # json, text
-```
-
-Full list of env vars:
-
-| Variable | Default | Description |
-|---|---|---|
-| `SEARXNG_URL` | *(auto)* | External SearXNG URL |
-| `SEARXNG_MODE` | `native` | `native` or `docker` |
-| `SEARXNG_PORT` | `8888` | Port for embedded SearXNG |
-| `WIGOLO_DATA_DIR` | `~/.wigolo` | Data + cache directory |
-| `WIGOLO_AUTH_STATE_PATH` | — | Playwright storage state JSON |
-| `WIGOLO_CHROME_PROFILE_PATH` | — | Chrome user data directory |
-| `WIGOLO_RERANKER` | `none` | `flashrank` or `none` |
-| `WIGOLO_TRAFILATURA` | `auto` | `auto`, `always`, or `never` |
-| `MAX_BROWSERS` | `3` | Concurrent Playwright contexts |
-| `FETCH_TIMEOUT_MS` | `10000` | HTTP fetch timeout |
-| `CRAWL_CONCURRENCY` | `2` | Concurrent crawl requests |
-| `RESPECT_ROBOTS_TXT` | `true` | Honor robots.txt |
-
-## How it works
-
-```
-search query
-    → SearXNG (70+ engines) or direct scraping (Bing/DDG/Startpage)
-    → deduplicate by URL
-    → domain/date/category filters
-    → ML reranking (FlashRank, optional)
-    → link validation
-    → fetch + extract top N results in parallel
-    → return markdown
-
-Each step degrades gracefully:
-  SearXNG down?        → direct scraping fallback
-  Page needs JS?       → auto-detected, Playwright used transparently
-  Extractor fails?     → ensemble: site-specific → Defuddle → Trafilatura → Readability → Turndown
-  Already fetched?     → served from SQLite cache with FTS5
+export ANTHROPIC_API_KEY=...
+export OPENAI_API_KEY=...
+export GOOGLE_API_KEY=...
+export GROQ_API_KEY=...
+# optional: pin which provider
+export WIGOLO_LLM_PROVIDER=anthropic|openai|gemini|groq
 ```
 
-**Extraction ensemble** — every page runs through multiple extractors in order, falling back if content is below threshold:
-1. Site-specific extractors (GitHub, Stack Overflow, MDN, docs frameworks)
-2. Defuddle — markdown-aware, site-adaptive
-3. Trafilatura — high-precision article extraction (Python, optional)
-4. Readability.js — battle-tested Mozilla algorithm
-5. Raw Turndown — last resort HTML-to-markdown
+If no key is set the fallback is skipped — `extract` still works through the heuristic path. Calls are cached (default 7 days) and rate-limited per request.
 
-## Roadmap
+## Local LLM fallback (research synthesis)
 
-### v2.1 — Next
-- [ ] Daemon mode — persistent HTTP server, zero startup latency
-- [ ] Browser interaction — click, type, scroll before extraction
-- [ ] Content change detection — diff monitoring for cached pages
-- [ ] CDP session discovery — attach to running Chrome for seamless auth
-- [ ] Plugin system — community extractors and search engines
+Set `WIGOLO_LLM_PROVIDER=openai-compatible` plus `WIGOLO_LLM_ENDPOINT=http://localhost:11434/v1` to let `research` use a local model (e.g. Ollama) when the host MCP client doesn't support sampling.
 
-### v2.2
-- [ ] Multi-browser pool — Chromium + Firefox for fingerprint diversity
-- [ ] Interactive REPL (`wigolo shell`)
-- [ ] Agent skill distribution — MCP registry listings, `SKILL.md`
+## Config flags worth knowing
 
-### v3 — The Knowledge Engine
-- [ ] Answer synthesis — search + LLM = direct answers with citations (bring your own key)
-- [ ] Semantic search — local vector embeddings over cached content (`findSimilar`)
-- [ ] Agent endpoint — describe what you need, no URLs required
-- [ ] Streaming answers — real-time generation as results come in
-- [ ] Knowledge graph — entity and relationship extraction from crawled content
-- [ ] Auto re-crawl scheduler — keep documentation fresh automatically
-- [ ] Lightpanda browser — optional ultra-lightweight headless browser (11x less RAM than Chrome)
-- [ ] Cloud sync — share cache across machines via rclone (S3, Drive, Dropbox)
-- [ ] Team knowledge base — shared indexed content across team members
-
-## Troubleshooting
-
-**SearXNG won't start**
-Make sure `python3` is on your PATH and version 3.8+. Check with `python3 --version`. Alternatively, set `SEARXNG_MODE=docker` if Docker is available.
-
-**Playwright browser not found**
-Run `npx wigolo warmup` to download Chromium. This is done automatically on first use but can fail behind corporate proxies.
-
-**Search returns no results**
-If SearXNG and all fallback engines fail, check your network connection. Behind a proxy? Set `PROXY_URL=http://your-proxy:port`.
-
-**Permission errors on `~/.wigolo/`**
-wigolo stores its cache and SearXNG installation in `~/.wigolo/`. Ensure your user has write access. Override with `WIGOLO_DATA_DIR=/your/path`.
-
-## Contributing
-
-PRs welcome. Open an issue first to discuss what you'd like to change.
+| Env var | Default | What it does |
+|---|---|---|
+| `WIGOLO_SEARCH` | `searxng` | `v1` to use the new direct-engine path |
+| `WIGOLO_RSS_FEEDS` | unset | Comma-separated feed URLs; v1 news vertical picks them up |
+| `WIGOLO_DEDUP_CASE_INSENSITIVE_HOSTS` | unset | Comma-separated hostnames where `/A` == `/a` for dedup |
+| `WIGOLO_CRAWL_INDEX` | `0` | `1` to fire-and-forget upsert crawled pages into the vector store |
+| `WIGOLO_EAGER_WARMUP` | `0` | `1` warms embed + rerank at MCP server start (non-blocking) |
+| `WIGOLO_TELEMETRY` | `0` | `1` enables opt-in NDJSON telemetry; off by default |
+| `WIGOLO_DATA_DIR` | `~/.wigolo` | Override data dir for cache, models, SearXNG state |
+| `WIGOLO_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
+| `WIGOLO_LOG_FORMAT` | `json` | `json` or `text`; both write to stderr |
+
+## What's known to work
+
+- 8 MCP tools, full test suite passing (3500+ unit + integration tests on macOS arm64)
+- `init` flow on macOS for Claude Code, Cursor, Gemini CLI, Codex, Windsurf, Zed, OpenCode
+- `WIGOLO_SEARCH=v1` runs end-to-end: intent routing, direct engines, RRF, recency boost, agent_context
+- SQLite + sqlite-vec hybrid on macOS arm64; FTS5-only graceful degradation on alpine/musl (sqlite-vec extension absent)
+- Defuddle extraction with content-type routing (news/recipe/product/paper/event JSON-LD)
+- Conditional GET with true 304 short-circuit through SmartRouter (saves bandwidth on revisits)
+- Doctor + warmup + backfill CLIs
+- Opt-in eager warmup, telemetry, RSS feeds
+
+## What's still gated / not done
+
+- **Phase 16** — 5-way blind bench (wigolo-v1 vs wigolo-legacy vs Tavily vs Exa vs Firecrawl) not yet captured at the current SHA. Smoke `--subset 2` on this build: `wigolo-v1=10, wigolo-legacy=9.5, exa=10, firecrawl=10, tavily=8`. Full run needed before flipping the default.
+- **Phase 17 default flip** — `WIGOLO_SEARCH` will stay `searxng` until Phase 16 passes. Set `WIGOLO_SEARCH=v1` to opt in now.
+- **Phase 18 v1.0 release** — pinned to bench-gated default flip. This `0.1.1` build is the engine-overhaul snapshot.
+- **Onboarding for engine/RSS/LLM in Ink TUI** — plain `init` asks the new prompts; the Ink TUI phase machine still needs a matching screen.
+- **Bench numbers** — Phase 6/7/8/11/12/13/15 perf + extraction benches are scaffolded but their numbers haven't been captured to `benchmarks/*/output/`.
+
+## Architecture in one glance
+
+```
+src/
+  index.ts          CLI router
+  server.ts         MCP server (8 tools + 1 resource)
+  config.ts         52+ env vars
+  cli/              warmup, doctor, health, auth, plugin, shell, init, status, backfill, setup-mcp
+  tools/            thin MCP handlers (one per tool, delegate to domain)
+  fetch/            SmartRouter (HTTP-first → Playwright), browser pool, auth, Lightpanda
+  extraction/       Defuddle + content-type routing + named schemas + LLM fallback
+  search/           SearXNG client + direct engines + dedup + rerank + RRF + multi-query + answer synth
+  search/v1/        v1 engine: intent router + verticals + orchestrator + RSS + recency + context-rank
+  crawl/            BFS/DFS/sitemap/map + robots.txt + ETag-incremental
+  cache/            SQLite FTS5 + sqlite-vec hybrid + migrations + backfill
+  embedding/        fastembed (BGE-small-en-v1.5)
+  research/         decomposition → parallel search → synthesis + citation graph
+  agent/            plan → execute → synthesize
+  searxng/          process + Docker management + bootstrap retry
+  providers/        embed, rerank, extract, vector-store, search interfaces
+```
+
+## Common pitfalls
+
+- **First run is slow** — model downloads (~250 MB combined) + SearXNG bootstrap (~30 s). `wigolo warmup --all` upfront avoids it during first MCP request.
+- **`wigolo doctor` shows `ML reranker: not installed`** — run `wigolo warmup --reranker` to fetch the cross-encoder model (~22 MB).
+- **`category: 'images'` rejected on `WIGOLO_SEARCH=v1`** — v1 has no images vertical (yet). Use the legacy path (`WIGOLO_SEARCH=searxng`) or omit `category`.
+- **`sqlite-vec extension failed to load`** — your platform (alpine/musl) doesn't have prebuilt binaries. The cache still works via FTS5; vector search is disabled.
+
+## Development
 
 ```bash
 git clone https://github.com/KnockOutEZ/wigolo
 cd wigolo
 npm install
-npm test
+npm run build         # tsup → dist/, then tsc → dist/*.d.ts
+npm test              # full vitest suite
+npm run lint          # tsc --noEmit
+npm run dev           # tsx src/index.ts
 ```
 
 ## License
 
-[BSL 1.1](LICENSE) — free for individuals, small teams (under $1M revenue), education, and open source. Converts to MIT on 2029-04-12.
+BUSL-1.1 — see `LICENSE`.