@staticn0va/wigolo 0.1.0 → 0.1.1

package/LICENSE

@@ -19,7 +19,7 @@ Additional Use Grant: You may make production use of the Licensed Work,
 
 Change Date:          2029-04-12
 
-Change License:       MIT License
+Change License:       GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later)
 
 For information about alternative licensing arrangements, contact:
 ktowhid20@gmail.com

package/README.md

@@ -2,34 +2,80 @@
 
 # wigolo
 
-**Local-first web search MCP server for AI coding agents.**
+**Local-first web intelligence for AI coding agents.**
 
-Search, fetch, crawl, cache, and extract — zero API keys, zero cloud, zero cost.
+Search, fetch, crawl, cache, and extract — ML reranking, semantic embeddings, persistent local cache. Zero API keys, zero cloud, zero cost.
 
 [![License: BSL 1.1](https://img.shields.io/badge/License-BSL_1.1-blue.svg)](LICENSE)
 [![Node.js](https://img.shields.io/badge/node-%3E%3D20-brightgreen)](https://nodejs.org)
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.x-blue)](https://www.typescriptlang.org/)
 
-[Quick Start](#quick-start) · [Features](#features) · [Why wigolo?](#why-wigolo) · [Roadmap](#roadmap)
+[Quick Start](#quick-start) · [Features](#features) · [Why wigolo?](#why-wigolo)
 
 </div>
 
 ```
-$ claude mcp add wigolo -- npx wigolo
-Added MCP server wigolo
-
-$ # That's it. Your agent now has web search.
+$ npx @staticn0va/wigolo init
 ```
 
+One command. Interactive TUI walks you through everything: system check, browser selection, dependency installation, verification, agent detection, MCP configuration, and skill installation. Done in under two minutes.
+
+</div>
+
 ## What is this?
 
-wigolo gives AI coding agents (Claude Code, Cursor, Gemini CLI, Codex, Windsurf) web search, page fetching, site crawling, content extraction, and a local knowledge cache. It runs entirely on your machine. No API keys, no cloud, no cost — works out of the box with `npx`.
+wigolo gives AI coding agents (Claude Code, Cursor, Gemini CLI, Codex, Windsurf, Zed, OpenCode) web search, page fetching, site crawling, content extraction, and a local knowledge cache. It runs entirely on your machine. No API keys, no cloud, no cost — works out of the box with `npx`.
 
 ## Quick Start
 
+### Option A: Interactive setup (recommended)
+
+```bash
+npx @staticn0va/wigolo init
+```
+
+The TUI handles everything:
+1. **System check** — verifies Node.js, Python, Docker, disk space
+2. **Browser selection** — Lightpanda (fast headless), Chromium, or Firefox
+3. **Install** — search engine, browser, content extractor, ML reranker, embeddings
+4. **Verify** — starts search engine, checks all components
+5. **Agent config** — detects and configures MCP for your AI tools
+6. **Skill install** — writes tool documentation to each agent's instruction system
+
+For ongoing use, install globally:
+```bash
+npm i -g @staticn0va/wigolo
+wigolo init      # re-run setup
+wigolo doctor    # system diagnostics
+wigolo status    # quick health check
+wigolo shell     # interactive REPL
+```
+
+### Option B: Manual setup
+
+**1. Warm up:**
+
+```bash
+npx @staticn0va/wigolo warmup --all
+```
+
+Flag menu:
+
+```bash
+npx @staticn0va/wigolo warmup                # browser engine + search engine only
+npx @staticn0va/wigolo warmup --all          # + reranker + trafilatura + embeddings + lightpanda + verify
+npx @staticn0va/wigolo warmup --reranker     # Install ML reranker
+npx @staticn0va/wigolo warmup --trafilatura  # Install content extractor
+npx @staticn0va/wigolo warmup --embeddings   # Install semantic embeddings
+npx @staticn0va/wigolo warmup --verify       # Start search engine, test all components
+npx @staticn0va/wigolo warmup --force        # Wipe search engine state/install/locks and re-bootstrap
+```
+
+**2. Connect your agent:**
+
 **Claude Code:**
 ```bash
-claude mcp add wigolo -- npx wigolo
+claude mcp add wigolo -- npx @staticn0va/wigolo
 ```
 
 **Cursor / VS Code / any MCP client:**
@@ -38,29 +84,73 @@ claude mcp add wigolo -- npx wigolo
   "mcpServers": {
     "wigolo": {
       "command": "npx",
-      "args": ["wigolo"]
+      "args": ["@staticn0va/wigolo"]
     }
   }
 }
 ```
 
-**Optional warmup (improves quality on first use):**
+> Skipping setup still works — wigolo bootstraps in the background on first tool call — but early searches will be lower quality until the install finishes.
+
+## Diagnostics
+
+```bash
+wigolo doctor    # full component health check
+wigolo status    # quick overview
+```
+
+Or via npx: `npx @staticn0va/wigolo doctor`. Reports the state of every component. Exits 0 when healthy, 1 when degraded. Usable in scripts: `wigolo doctor && my-agent`.
+
+## Daemon Mode
+
+Run wigolo as a persistent HTTP server for lower latency and shared infrastructure:
+
+### Start the daemon
+
+```bash
+npx @staticn0va/wigolo serve
+npx @staticn0va/wigolo serve --port 4444 --host 0.0.0.0
+```
+
+The daemon exposes:
+- `POST /mcp` -- StreamableHTTP MCP transport (preferred)
+- `GET /sse` -- SSE MCP transport (legacy compatibility)
+- `GET /health` -- Health check endpoint
+
+### Check health
+
 ```bash
-npx wigolo warmup          # Downloads Playwright + SearXNG
-npx wigolo warmup --all    # + ML reranking + Trafilatura extraction
+npx @staticn0va/wigolo health
+# or
+curl http://127.0.0.1:3333/health
 ```
 
+Returns:
+```json
+{
+  "status": "healthy",
+  "searxng": "active",
+  "browsers": "ready",
+  "cache": "active",
+  "uptime_seconds": 3600
+}
+```
+
+### Auto-connect
+
+When starting in stdio mode, wigolo checks if a daemon is already running on `WIGOLO_DAEMON_PORT`. If detected, a notice is printed to stderr. Full stdio-to-daemon proxy is planned for v2.1.
+
 ## Prerequisites
 
 - **Node.js 20+** — [Download](https://nodejs.org/) or `brew install node` (macOS) / `winget install OpenJS.NodeJS` (Windows) / `sudo apt install nodejs` (Ubuntu/Debian)
 - **Python 3.8+** *(recommended)* — [Download](https://python.org/) or `brew install python3` (macOS) / `winget install Python.Python.3` (Windows) / `sudo apt install python3` (Ubuntu/Debian)
-- **Docker** *(optional)* — Alternative to Python for running SearXNG.
+- **Docker** *(optional)* — Alternative for running the search engine container.
 
-Everything else (Playwright, SearXNG) is downloaded automatically on first use or via `npx wigolo warmup`.
+Everything else (browser, search engine) is downloaded automatically on first use or via `npx @staticn0va/wigolo warmup`.
 
 ### What works without Python?
 
-Everything except embedded SearXNG. Without Python, search falls back to direct scraping of Bing, DuckDuckGo, and Startpage — functional but less reliable. All other tools (fetch, crawl, cache, extract) work fully with just Node.js.
+Everything except the embedded search engine. Without Python, search falls back to direct scraping of Bing, DuckDuckGo, and Startpage — functional but less reliable. All other tools (fetch, crawl, cache, extract) work fully with just Node.js.
 
 ## Features
 
@@ -76,8 +166,8 @@ search("React Server Components best practices", { max_results: 5 })
 - Domain filtering: `include_domains: ["react.dev"]`, `exclude_domains: ["medium.com"]`
 - Date filtering: `from_date: "2024-01-01"`, `to_date: "2025-01-01"`
 - Category search: `general`, `news`, `code`, `docs`, `papers`
-- ML reranking with FlashRank when installed
-- Falls back to direct engine scraping when SearXNG is unavailable
+- ML reranking when installed
+- Falls back to direct engine scraping when search engine is unavailable
 
 ### fetch
 
@@ -88,7 +178,7 @@ fetch("https://docs.react.dev/reference/react/useState")
 → clean markdown, links, images, metadata, cached for future use
 ```
 
-- Smart routing: HTTP first, Playwright fallback for JS-rendered pages (auto-detected)
+- Smart routing: HTTP first, browser engine fallback for JS-rendered pages (auto-detected)
 - Section targeting: `section: "Parameters"` extracts content under that heading
 - Authenticated browsing: `use_auth: true` with stored session or Chrome profile
 - PDF support: text extraction via pdf-parse
@@ -117,7 +207,7 @@ cache({ query: "React hooks", url_pattern: "*react.dev*" })
 → matching cached pages with full markdown
 ```
 
-- SQLite FTS5 full-text search over all cached content
+- Full-text search over all cached content
 - Combined filters: text query + URL pattern + date range
 - Cache stats and selective clearing
 
@@ -154,17 +244,17 @@ Modes:
 wigolo works with zero configuration. For advanced use:
 
 ```bash
-# Use an existing SearXNG instance instead of the embedded one
+# Use an existing search engine instance instead of the embedded one
 SEARXNG_URL=http://localhost:8888
 
-# Authenticated browsing — export session state via Playwright
+# Authenticated browsing — export browser session state
 WIGOLO_AUTH_STATE_PATH=~/.wigolo/auth.json
 
 # Or use your Chrome profile directly (close Chrome first)
 WIGOLO_CHROME_PROFILE_PATH=~/.config/google-chrome/Default
 
-# ML reranking (install with: npx wigolo warmup --reranker)
-WIGOLO_RERANKER=flashrank
+# ML reranking (install with: npx @staticn0va/wigolo warmup --reranker)
+WIGOLO_RERANKER=onnx
 
 # Tune extraction — auto/always/never
 WIGOLO_TRAFILATURA=auto
@@ -178,83 +268,96 @@ Full list of env vars:
 
 | Variable | Default | Description |
 |---|---|---|
-| `SEARXNG_URL` | *(auto)* | External SearXNG URL |
+| `SEARXNG_URL` | *(auto)* | External search engine URL |
 | `SEARXNG_MODE` | `native` | `native` or `docker` |
-| `SEARXNG_PORT` | `8888` | Port for embedded SearXNG |
+| `SEARXNG_PORT` | `8888` | Port for embedded search engine |
 | `WIGOLO_DATA_DIR` | `~/.wigolo` | Data + cache directory |
-| `WIGOLO_AUTH_STATE_PATH` | — | Playwright storage state JSON |
+| `WIGOLO_AUTH_STATE_PATH` | — | Browser session state JSON |
 | `WIGOLO_CHROME_PROFILE_PATH` | — | Chrome user data directory |
-| `WIGOLO_RERANKER` | `none` | `flashrank` or `none` |
-| `WIGOLO_TRAFILATURA` | `auto` | `auto`, `always`, or `never` |
-| `MAX_BROWSERS` | `3` | Concurrent Playwright contexts |
+| `WIGOLO_RERANKER` | `onnx` | ML reranker: `onnx` or `none` (`flashrank` accepted as legacy alias) |
+| `WIGOLO_TRAFILATURA` | `auto` | Content extractor: `auto`, `always`, or `never` |
+| `MAX_BROWSERS` | `3` | Concurrent browser contexts |
 | `FETCH_TIMEOUT_MS` | `10000` | HTTP fetch timeout |
 | `CRAWL_CONCURRENCY` | `2` | Concurrent crawl requests |
 | `RESPECT_ROBOTS_TXT` | `true` | Honor robots.txt |
+| `WIGOLO_BOOTSTRAP_MAX_ATTEMPTS` | `3` | Cap on search engine bootstrap auto-retries |
+| `WIGOLO_BOOTSTRAP_BACKOFF_SECONDS` | `30,3600,86400` | Backoff seconds for retry attempts 1, 2, 3 |
+| `WIGOLO_HEALTH_PROBE_INTERVAL_MS` | `30000` | Interval between search engine health probes |
+| `WIGOLO_DAEMON_PORT` | `3333` | HTTP server port for daemon mode |
+| `WIGOLO_DAEMON_HOST` | `127.0.0.1` | HTTP server bind address for daemon mode |
 
 ## How it works
 
 ```
 search query
-    → SearXNG (70+ engines) or direct scraping (Bing/DDG/Startpage)
+    → search engine (70+ engines) or fallback engines (Bing/DDG/Startpage)
     → deduplicate by URL
     → domain/date/category filters
-    → ML reranking (FlashRank, optional)
+    → ML reranking (optional)
     → link validation
     → fetch + extract top N results in parallel
     → return markdown
 
 Each step degrades gracefully:
-  SearXNG down?        → direct scraping fallback
-  Page needs JS?       → auto-detected, Playwright used transparently
-  Extractor fails?     → ensemble: site-specific → Defuddle → Trafilatura → Readability → Turndown
-  Already fetched?     → served from SQLite cache with FTS5
+  Search engine down?  → fallback engine scraping
+  Page needs JS?       → auto-detected, browser rendering used transparently
+  Extractor fails?     → ensemble pipeline (site-specific → primary → content → fallback → converter)
+  Already fetched?     → served from local cache
 ```
 
-**Extraction ensemble** — every page runs through multiple extractors in order, falling back if content is below threshold:
+Search engine bootstrap failures are self-healing: wigolo retries after 30 seconds, 1 hour, and 24 hours on successive server restarts. Once attempts are exhausted, fallback scraping stays active until the user runs `warmup --force`. Tool responses include a one-time fallback warning so agents can surface the recovery command. See `doctor` for the full state.
+
+**Extraction pipeline** — every page runs through multiple extractors in order, falling back if content is below threshold:
 1. Site-specific extractors (GitHub, Stack Overflow, MDN, docs frameworks)
-2. Defuddle — markdown-aware, site-adaptive
-3. Trafilatura — high-precision article extraction (Python, optional)
-4. Readability.js — battle-tested Mozilla algorithm
-5. Raw Turndown — last resort HTML-to-markdown
-
-## Roadmap
-
-### v2.1 — Next
-- [ ] Daemon mode — persistent HTTP server, zero startup latency
-- [ ] Browser interaction — click, type, scroll before extraction
-- [ ] Content change detection — diff monitoring for cached pages
-- [ ] CDP session discovery — attach to running Chrome for seamless auth
-- [ ] Plugin system — community extractors and search engines
-
-### v2.2
-- [ ] Multi-browser pool — Chromium + Firefox for fingerprint diversity
-- [ ] Interactive REPL (`wigolo shell`)
-- [ ] Agent skill distribution — MCP registry listings, `SKILL.md`
-
-### v3 — The Knowledge Engine
-- [ ] Answer synthesis — search + LLM = direct answers with citations (bring your own key)
-- [ ] Semantic search — local vector embeddings over cached content (`findSimilar`)
-- [ ] Agent endpoint — describe what you need, no URLs required
-- [ ] Streaming answers — real-time generation as results come in
-- [ ] Knowledge graph — entity and relationship extraction from crawled content
-- [ ] Auto re-crawl scheduler — keep documentation fresh automatically
-- [ ] Lightpanda browser — optional ultra-lightweight headless browser (11x less RAM than Chrome)
-- [ ] Cloud sync — share cache across machines via rclone (S3, Drive, Dropbox)
-- [ ] Team knowledge base — shared indexed content across team members
+2. Primary extractor — markdown-aware, site-adaptive
+3. Content extraction engine — high-precision article extraction (optional, Python)
+4. Fallback extractor — battle-tested browser-compat algorithm
+5. HTML-to-markdown converter — last resort
+
+**ML reranker** — the optional cross-encoder reranker runs as a long-lived Python subprocess (mirroring the embeddings subprocess pattern), keeping the heavy model resident across requests with no per-call cold start.
+
+## Discovery
+
+wigolo is listed on MCP server registries for agent discovery:
+
+- **SKILL.md** — machine-readable tool description at repo root, auto-installed to each agent's instruction system by `wigolo init`
+- **npm** — `npm info @staticn0va/wigolo` or search for `mcp-server` keyword
+
+The `init` TUI automatically configures MCP and installs SKILL.md for all selected agents. Manual setup:
+```bash
+claude mcp add wigolo -- npx @staticn0va/wigolo
+```
 
 ## Troubleshooting
 
-**SearXNG won't start**
-Make sure `python3` is on your PATH and version 3.8+. Check with `python3 --version`. Alternatively, set `SEARXNG_MODE=docker` if Docker is available.
+Start with `npx @staticn0va/wigolo doctor` — it reports the state of every component and is the fastest way to find the cause.
+
+**First search is slow or returns odd results**
+Search engine is still bootstrapping in the background. Either wait a minute, or (recommended) run `npx @staticn0va/wigolo warmup --all` before connecting your agent.
+
+**ML reranker / content extractor / embeddings "not installed"**
+These are optional Python extras. Install them with `npx @staticn0va/wigolo warmup --all` (or per-component: `--reranker`, `--trafilatura`, `--embeddings`). wigolo uses a private venv under `~/.wigolo/searxng/venv` so your system Python stays untouched.
+
+**Search engine won't start**
+Make sure `python3` is on your PATH and version 3.8+. Check with `python3 --version`. If bootstrap got interrupted, `npx @staticn0va/wigolo warmup --force` wipes the state and reinstalls. Alternatively, set `SEARXNG_MODE=docker` if Docker is available.
 
-**Playwright browser not found**
-Run `npx wigolo warmup` to download Chromium. This is done automatically on first use but can fail behind corporate proxies.
+**Doctor reports search engine "not running"**
+That's expected when you haven't made a search yet — the process starts on-demand when the MCP server needs it. Doctor only marks it degraded if the install is broken.
+
+**Browser engine not found**
+Run `npx @staticn0va/wigolo warmup` to download Chromium. This is done automatically on first use but can fail behind corporate proxies.
 
 **Search returns no results**
-If SearXNG and all fallback engines fail, check your network connection. Behind a proxy? Set `PROXY_URL=http://your-proxy:port`.
+If all search engines fail, check your network connection. Behind a proxy? Set `PROXY_URL=http://your-proxy:port`.
 
 **Permission errors on `~/.wigolo/`**
-wigolo stores its cache and SearXNG installation in `~/.wigolo/`. Ensure your user has write access. Override with `WIGOLO_DATA_DIR=/your/path`.
+wigolo stores its cache and search engine state in `~/.wigolo/`. Ensure your user has write access. Override with `WIGOLO_DATA_DIR=/your/path`.
+
+**Start fresh**
+```bash
+rm -rf ~/.wigolo
+npx @staticn0va/wigolo init    # or: warmup --all
+```
 
 ## Contributing
 
@@ -267,6 +370,25 @@ npm install
 npm test
 ```
 
+## Releasing
+
+Releases are triggered by pushing a version tag. CI handles the rest.
+
+```bash
+# on main, all changes committed and pushed
+make release-patch   # or: release-minor / release-major
+```
+
+Run `make help` for all targets, or `make release-dry-run` to preview the npm tarball.
+
+The `release` workflow will:
+1. Build a clean `dist/`
+2. Verify the tag matches `package.json` version
+3. Publish to npm with provenance
+4. Create a GitHub Release with auto-generated notes
+
+Requires the `NPM_TOKEN` repository secret (npm automation token with publish scope).
+
 ## License
 
-[BSL 1.1](LICENSE) — free for individuals, small teams (under $1M revenue), education, and open source. Converts to MIT on 2029-04-12.
+[BSL 1.1](LICENSE) — free for individuals, small teams (under $1M revenue), education, and open source. Converts to AGPL-3.0 on 2029-04-12.