pi-webveil 0.1.0 → 0.1.1

package/README.md

@@ -13,33 +13,258 @@ works perfectly well non-anonymously (direct egress).
 webveil is a pnpm workspace monorepo. The **core** (`search()` / `fetch()`) is plain,
 framework-agnostic. Two thin frontends wrap that same core:
 
-- **[`webveil`](packages/webveil)** — an [incur](https://github.com/wevm/incur)-based
+- **[`webveil`](packages/webveil)**, an [incur](https://github.com/wevm/incur)-based
   **CLI + MCP server** (`--mcp`, skills, `--llms`, TOON output). Pi-agnostic; usable by any
   agent (pi via pi-mcp-adapter, Claude Code, Cursor, Codex, bash). Has a `webveil` bin.
-- **[`pi-webveil`](packages/pi-webveil)** — a **pi extension** registering `web_search` and
+- **[`pi-webveil`](packages/pi-webveil)**, a **pi extension** registering `web_search` and
   `web_fetch` tools that call the core in-process. A drop-in replacement for Ollama's tools
   (same names), which is the original motivation. Depends on `webveil` via `workspace:*`.
 
+## Quick start
+
+webveil needs a **backend** to get results from. The zero-config default is a local
+**SearXNG** at `http://127.0.0.1:8080` on `direct` egress (non-anonymous). There is
+**no** zero-setup + anonymous + real-web-results option in the ecosystem, see
+[`work/notes/ideas/default-backend-policy-account-vs-origin.md`](work/notes/ideas/default-backend-policy-account-vs-origin.md);
+SearXNG (you run it) is the closest, `tavily-compat` (needs an account/key) is the other.
+
+### Run SearXNG (matches the default with no config)
+
+```sh
+# Docker: the container binds 8080 internally; map host 8080 -> container 8080
+# so it matches webveil's default baseUrl exactly.
+docker run -d --name searxng -p 8080:8080 searxng/searxng
+```
+
+Then `webveil search "…"` / `web_fetch` work with no config.
+
+> **Port gotcha (you WILL hit this):** SearXNG's default port depends on how you install
+> it. A bare-metal / pip / source install defaults to **8888** (`settings.yml`
+> `server.port: 8888`). The Docker image binds **8080** internally regardless (its
+> entrypoint forces `0.0.0.0:8080`). SearXNG's own docs suggest `docker run … -p 8888:8080`
+> (host 8888 → container 8080). webveil's default expects **8080**. If your instance is on
+> any other port, point webveil at it:
+>
+> ```sh
+> export WEBVEIL_BASE_URL=http://127.0.0.1:8888   # or wherever your instance listens
+> ```
+>
+> or set `baseUrl` in `.pi/webveil.json` (see config seam below).
+
+### Other SearXNG install options
+
+webveil needs something to point `baseUrl` at: an **HTTP `host:port`**, or (script install)
+the **Unix socket** itself. How you get one:
+
+- **Docker (above)**, binds a real TCP port directly; simplest if you only need webveil.
+- **Install script as a background service** (`sudo -H ./utils/searxng.sh install all`,
+  see <https://docs.searxng.org/admin/installation-scripts.html>), sets SearXNG up as a
+  systemd/uWSGI service. **Gotcha:** by default this listens on a **Unix socket**
+  (`socket = /usr/local/searxng/run/socket`), NOT a TCP port. And, crucially, that default
+  socket speaks the **native uwsgi protocol, NOT HTTP** (`socket = …`, not `http-socket =
+  …`), so even a `curl --unix-socket … http://localhost/` returns HTTP 000. webveil's
+  `unix:` baseUrl speaks **HTTP over a unix socket** via undici, so it CANNOT reach that
+  default uwsgi socket directly. Three ways to reach the install-script instance:
+  - **Point webveil straight at an HTTP unix socket** (no proxy, no extra process), once the
+    socket actually speaks HTTP. The install-script default does NOT, so first make uWSGI
+    serve HTTP on the socket: in the generated `.ini`, replace
+    `socket = /usr/local/searxng/run/socket` with
+    `http-socket = /usr/local/searxng/run/socket` (HTTP over the socket instead of the
+    uwsgi protocol). THEN point webveil at it with a `unix:` URL naming the socket file:
+    ```sh
+    export WEBVEIL_BASE_URL=unix:/usr/local/searxng/run/socket
+    ```
+    webveil dials the socket directly over undici (`Agent({connect:{socketPath}})`, no
+    extra dependency) and issues its normal `/search?...&format=json` request. The grammar
+    is `unix:<socketPath>[:<httpPath>]`: the socket file path, then an OPTIONAL `:` +
+    base path (mount point) the SearXNG app lives under (defaults to `/`, so the example
+    above requests `/search`; a non-root mount is `unix:/usr/local/searxng/run/socket:/searxng`).
+    (`unix:` works against ANY HTTP-on-a-unix-socket server, e.g. a Caddy/nginx upstream
+    bound to a socket; the uwsgi-vs-`http-socket` distinction above is the SearXNG-specific
+    catch.)
+    **Egress must be `direct`** for this: a Unix socket is inherently local, so combining a
+    `unix:` baseUrl with `egress=http`/`socks5` fails loud (proxying a local hop is fake
+    anonymity, see "Where does anonymity live?" below; proxy SearXNG's `outgoing.proxies`
+    instead and keep webveil `direct`).
+  - **Front it with a reverse proxy** (this is what the SearXNG docs' nginx/apache step is
+    for, it bridges HTTP-on-a-port to the uWSGI socket, serving BOTH the browser UI and
+    webveil). **Any HTTP server works**, the docs say so explicitly; **Caddy is fine** and
+    a good pick if you already run it. Plain Caddy `reverse_proxy` speaks **HTTP** to its
+    upstream, so point it at an `http-socket` (see below) or a TCP `http-socket`:
+    ```caddy
+    searxng.example.com {
+        reverse_proxy unix//usr/local/searxng/run/socket   # plain reverse_proxy = HTTP, so the socket must be http-socket = (not the uwsgi socket =)
+    }
+    ```
+    Then point webveil at the Caddy address. (Set SearXNG's `server.base_url` in
+    `settings.yml` to match, and keep the limiter in mind, see below.) If you want a Caddy
+    frontend AND webveil-direct, the simplest path is ONE `http-socket` that both consume
+    (Caddy's HTTP `reverse_proxy` and webveil's `unix:` both speak HTTP to it); you only
+    need the uwsgi `socket = ` form if Caddy uses an explicit uwsgi transport.
+  - **Or make uWSGI listen on a TCP port** instead of the socket: in the generated
+    `.ini`, replace `socket = …/run/socket` with `http-socket = 127.0.0.1:8888`, then point
+    webveil at `http://127.0.0.1:8888`. Good when you want ONLY webveil (no public web UI /
+    TLS).
+
+> **You will also need to enable the JSON API and (for a local instance) disable the
+> limiter.** A fresh script install ships with `server.limiter: true` and often no `json`
+> output format, so webveil gets `429 TOO MANY REQUESTS` or an HTML page. In SearXNG's
+> `settings.yml` set `server.limiter: false` + `server.public_instance: false` (safe for a
+> LOCAL, socket-only instance, NOT internet-exposed) and add `json` under `search.formats:`
+> (`[html, json]`), then restart uWSGI. This applies to EVERY option above, it is a
+> SearXNG-side requirement, not a webveil one.
+
+Full SearXNG install options (Docker, Compose, script, bare-metal): the official docs at
+<https://docs.searxng.org/admin/installation.html>. Install topology + the
+uwsgi-vs-`http-socket`, limiter, and reverse-proxy details captured in
+[`work/notes/findings/searxng-install-topology.md`](work/notes/findings/searxng-install-topology.md)
+and
+[`work/notes/findings/searxng-script-socket-is-uwsgi-not-http.md`](work/notes/findings/searxng-script-socket-is-uwsgi-not-http.md).
+
+### Where does anonymity live? (read before turning on egress)
+
+**webveil's egress only anonymizes webveil's OWN outbound hop** (webveil → backend, and
+`web_fetch` → the target URL). It does NOT anonymize what a backend does next. This has a
+load-bearing consequence for SearXNG:
+
+- A **local** SearXNG makes its actual search-engine requests (→ Google/Bing/…) from
+  **its own process, on your machine, with your real IP**. That hop is OUTSIDE webveil's
+  egress. So setting `WEBVEIL_EGRESS=socks5` while `baseUrl` is `127.0.0.1` does **NOT**
+  make your searches anonymous, webveil would just be proxying a pointless localhost call,
+  while SearXNG crawls the web from your real IP. That is **false confidence**, the worst
+  outcome.
+- **webveil refuses this combo (fail-loud):** a non-`direct` egress (`http`/`socks5`) with
+  a **loopback `baseUrl`** is rejected with an error, rather than silently giving you fake
+  anonymity. (A *remote* SearXNG over SOCKS is legitimate and allowed, the guard keys on
+  loopback specifically.)
+
+So the correct setups:
+
+| Goal | webveil egress | backend | Who anonymizes the web hop |
+| --- | --- | --- | --- |
+| Local SearXNG, anonymous searches | `direct` | local SearXNG | **SearXNG itself**, set its `outgoing.proxies` (Tor/SOCKS) in `settings.yml` |
+| Remote SearXNG, hide your IP from it | `socks5` | the **remote** SearXNG url | webveil's hop (Mullvad/Tor) |
+| Anonymous `web_fetch` of arbitrary URLs | `socks5` | (any) | webveil's hop |
+| Non-anonymous everyday use | `direct` | local SearXNG | nobody (honest) |
+
+Rule of thumb: **proxy the hop that actually reaches the public internet.** For a
+self-hosted SearXNG that hop is SearXNG's, so the proxy goes on SearXNG
+(`outgoing.proxies`), and webveil stays `direct`. webveil's `socks5` mode is for *remote*
+backends and for `web_fetch`. See
+[`work/notes/findings/webveil-anonymity-boundary.md`](work/notes/findings/webveil-anonymity-boundary.md).
+
 ## How it works (seams)
 
-- **core** — the framework-agnostic `search(query, opts)` and `fetch(url, opts)` functions.
+- **core**, the framework-agnostic `search(query, opts)` and `fetch(url, opts)` functions.
   Both frontends call the same core.
-- **backend seam** — where results/content come from: `searxng` (keyless self-hosted
+- **backend seam**, where results/content come from: `searxng` (keyless self-hosted
   metasearch), `tavily-compat` (a generic Tavily-shaped `/search` + `/extract`), and
   `custom` (a local command via a JSON stdin/stdout contract). The backend is handed a
   proxied `http` helper so it cannot bypass egress.
-- **egress seam** — how outbound HTTP leaves the machine: `direct`, `http` (undici
+- **egress seam**, how outbound HTTP leaves the machine: `direct`, `http` (undici
   `ProxyAgent`), or `socks5` (Tor `127.0.0.1:9050`, Mullvad `10.64.0.1:1080`). SOCKS5 is
   the mode that matters for anonymity. Fail-loud if a configured proxy cannot be built.
-- **config seam** — per-folder resolution: env > nearest `.pi/webveil.json` walking up from
+  **Egress is per-request and scoped to webveil ONLY**, it is NOT a system-wide proxy. It
+  governs webveil's own search/fetch traffic (and the `fetch` it injects into distilly),
+  and nothing else: your shell, `git push`, the browser, and the OS are untouched. So
+  webveil on `socks5` does NOT route your `git push` through the proxy. See
+  [Anonymous egress](#anonymous-egress-mullvad--tor) and
+  [`work/notes/findings/mullvad-socks5-egress-mechanics.md`](work/notes/findings/mullvad-socks5-egress-mechanics.md).
+- **config seam**, per-folder resolution: env > nearest `.pi/webveil.json` walking up from
   cwd > global `~/.pi/agent/webveil.json` > defaults. Per folder = per account/egress.
-- **extractor seam** — `urlToMarkdown` via `distilly/fetch` by default, injected with
+- **extractor seam**, `urlToMarkdown` via `distilly/fetch` by default, injected with
   webveil's egress-bound `fetch`; a backend's own `/extract` (Tavily-compat) may override
   it. Owns the context-friendly markdown + size presets (`s`/`m`/`l`/`f`). See
   [`docs/adr/0001`](docs/adr/0001-extractor-uses-distilly-fetch-with-injected-egress.md).
-- **security** — an SSRF guard lives in the egress fetch, so it covers distilly's
+- **security**, an SSRF guard lives in the egress fetch, so it covers distilly's
   rule-rewritten requests too.
 
+## Anonymous egress (Mullvad / Tor)
+
+By default webveil uses `direct` egress (your real IP, non-anonymous). Anonymity is
+**opt-in**: it is enabled ONLY when you set it in config/env. webveil never auto-enables a
+proxy (silent anonymity would be a footgun in the other direction).
+
+Enable SOCKS5 egress for webveil:
+
+```sh
+export WEBVEIL_EGRESS=socks5
+export WEBVEIL_EGRESS_URL=socks5://10.64.0.1:1080     # Mullvad
+# or socks5://127.0.0.1:9050                          # Tor
+```
+
+or per folder in `.pi/webveil.json`:
+
+```json
+{ "egress": { "mode": "socks5", "url": "socks5://10.64.0.1:1080" } }
+```
+
+### Two layers keep your `git push` (and everything else) off the proxy
+
+A common worry: "if I route through Mullvad, will my `git push` to GitHub leak under the
+VPN exit IP?" With webveil, **no**, for two independent reasons:
+
+1. **webveil's egress is per-request and webveil-only.** It applies the SOCKS5 dispatcher
+   inside its own search/fetch code; it does not install a system proxy. `git`, your shell,
+   and the OS are never touched. webveil on `socks5` proxies webveil's traffic and nothing
+   else.
+2. **You configure split routing** (below) so that even at the OS level, only the proxy IP
+   goes through the tunnel.
+
+### Mullvad: use the SOCKS5 proxy WITHOUT tunnelling all your traffic
+
+Mullvad's SOCKS5 proxy at `10.64.0.1:1080` **only exists while a Mullvad WireGuard tunnel
+is up** (it is reachable only through the tunnel). The trick is to keep the tunnel up but
+tell WireGuard NOT to route your normal traffic through it, only the proxy IP. Add this to
+your Mullvad WireGuard `.conf` (`[Interface]` section):
+
+```ini
+Table = off
+PostUp  = ip -4 route add 10.64.0.1/32 dev %i; ip -4 route add 10.124.0.0/22 dev %i
+PreDown = ip -4 route delete 10.64.0.1/32 dev %i; ip -4 route delete 10.124.0.0/22 dev %i
+```
+
+`Table = off` stops WireGuard from grabbing the default route; the manual routes send ONLY
+Mullvad's SOCKS5 proxy IPs through the tunnel (`10.124.0.0/22` is the multihop range).
+Result: webveil's SOCKS5 requests exit via Mullvad; all other traffic (git, browser, OS)
+uses your normal ISP connection. (Simpler alternative: leave WireGuard's routing alone and
+rely on layer 1, but split routing is the belt-and-braces version.)
+
+Verify the proxy works: `curl https://ipv4.am.i.mullvad.net --socks5-hostname 10.64.0.1`
+should return a Mullvad exit IP; a plain `curl https://am.i.mullvad.net` should return your
+real IP (proving only the proxy is tunnelled).
+
+### "Different exit identity for webveil than for the rest of the machine"
+
+If you want webveil to exit somewhere different from your system, you have options, but be
+clear on what is and isn't possible (see
+[`work/notes/findings/mullvad-socks5-egress-mechanics.md`](work/notes/findings/mullvad-socks5-egress-mechanics.md)):
+
+- **Different exit LOCATION, same account (easy).** Point webveil at a specific multihop
+  SOCKS5 host so it exits elsewhere than your tunnel's entry:
+  `WEBVEIL_EGRESS_URL=socks5://us-nyc-wg-socks5-001.relays.mullvad.net:1080`. Your tunnel
+  enters where your Mullvad app is connected; webveil's traffic exits in NYC. Same Mullvad
+  account, unlinkable-by-location.
+- **Two DIFFERENT Mullvad ACCOUNTS at once (hard, not a webveil feature).** Mullvad's
+  SOCKS5 proxy is a property of the ONE active WireGuard tunnel, which is tied to ONE
+  account's key. SOCKS5 multihop changes exit location, NOT account. To run account A
+  system-wide AND account B for webveil simultaneously, you must isolate them at the OS
+  level: run webveil inside its own network namespace / VM / container that has its own
+  WireGuard tunnel on account B, while the host runs account A. That is infrastructure work
+  outside webveil. For most people, "don't link my searches to my git" is already solved by
+  split routing above (searches exit via Mullvad, git stays on your real IP, not correlated
+  by exit IP), without needing a second account.
+
+### Tor
+
+`WEBVEIL_EGRESS_URL=socks5://127.0.0.1:9050` with the Tor daemon running. Same per-request,
+webveil-only scoping applies.
+
+> **Caveat:** webveil's `socks5` mode is NOT a whole-machine VPN. Do not assume enabling it
+> anonymizes anything other than webveil. Conversely, a system-wide full-tunnel VPN under
+> your logged-in identity is the thing that CAN deanonymize a `git push`; webveil's scoped
+> egress deliberately avoids that.
+
 ## License
 
 AGPL-3.0-or-later. webveil depends on `distilly` (MIT, the local HTML-to-markdown

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-webveil",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "Pi extension: web_search and web_fetch tools backed by webveil. A drop-in, anonymity-capable replacement for Ollama's web_search/web_fetch.",
   "license": "AGPL-3.0-or-later",
   "keywords": [
@@ -41,7 +41,7 @@
     ]
   },
   "dependencies": {
-    "webveil": "0.1.0"
+    "webveil": "0.1.1"
   },
   "devDependencies": {
     "@types/node": "^25.2.0",