webveil 0.0.0 → 0.1.1

package/README.md

@@ -0,0 +1,326 @@
+# webveil
+
+**Anonymous-capable, self-hosted, account-free** web **search + fetch** for AI agents.
+
+webveil replaces account-bound tools (notably Ollama's `web_search` / `web_fetch`, which
+proxy a hosted service and sign every request with your account identity) with a
+self-hosted path that has **no account, no API key**, and an **egress you control**
+(direct, HTTP proxy, or SOCKS5/Tor) so searches and fetches can be anonymous. It also
+works perfectly well non-anonymously (direct egress).
+
+## Packages
+
+webveil is a pnpm workspace monorepo. The **core** (`search()` / `fetch()`) is plain,
+framework-agnostic. Two thin frontends wrap that same core:
+
+- **[`webveil`](packages/webveil)**, an [incur](https://github.com/wevm/incur)-based
+  **CLI + MCP server** (`--mcp`, skills, `--llms`, TOON output). Pi-agnostic; usable by any
+  agent (pi via pi-mcp-adapter, Claude Code, Cursor, Codex, bash). Has a `webveil` bin.
+- **[`pi-webveil`](packages/pi-webveil)**, a **pi extension** registering `web_search` and
+  `web_fetch` tools that call the core in-process. A drop-in replacement for Ollama's tools
+  (same names), which is the original motivation. Depends on `webveil` via `workspace:*`.
+
+## Quick start
+
+webveil needs a **backend** to get results from. The zero-config default is a local
+**SearXNG** at `http://127.0.0.1:8080` on `direct` egress (non-anonymous). There is
+**no** zero-setup + anonymous + real-web-results option in the ecosystem, see
+[`work/notes/ideas/default-backend-policy-account-vs-origin.md`](work/notes/ideas/default-backend-policy-account-vs-origin.md);
+SearXNG (you run it) is the closest, `tavily-compat` (needs an account/key) is the other.
+
+### Run SearXNG (matches the default with no config)
+
+```sh
+# Docker: the container binds 8080 internally; map host 8080 -> container 8080
+# so it matches webveil's default baseUrl exactly.
+docker run -d --name searxng -p 8080:8080 searxng/searxng
+```
+
+Then `webveil search "…"` / `web_fetch` work with no config.
+
+> **Port gotcha (you WILL hit this):** SearXNG's default port depends on how you install
+> it. A bare-metal / pip / source install defaults to **8888** (`settings.yml`
+> `server.port: 8888`). The Docker image binds **8080** internally regardless (its
+> entrypoint forces `0.0.0.0:8080`). SearXNG's own docs suggest `docker run … -p 8888:8080`
+> (host 8888 → container 8080). webveil's default expects **8080**. If your instance is on
+> any other port, point webveil at it:
+>
+> ```sh
+> export WEBVEIL_BASE_URL=http://127.0.0.1:8888   # or wherever your instance listens
+> ```
+>
+> or set `baseUrl` in `.pi/webveil.json` (see config seam below).
+
+### Other SearXNG install options
+
+webveil needs something to point `baseUrl` at: an **HTTP `host:port`**, or (script install)
+the **Unix socket** itself. How you get one:
+
+- **Docker (above)**, binds a real TCP port directly; simplest if you only need webveil.
+- **Install script as a background service** (`sudo -H ./utils/searxng.sh install all`,
+  see <https://docs.searxng.org/admin/installation-scripts.html>), sets SearXNG up as a
+  systemd/uWSGI service. **Gotcha:** by default this listens on a **Unix socket**
+  (`socket = /usr/local/searxng/run/socket`), NOT a TCP port. And, crucially, that default
+  socket speaks the **native uwsgi protocol, NOT HTTP** (`socket = …`, not `http-socket =
+  …`), so even a `curl --unix-socket … http://localhost/` returns HTTP 000. webveil's
+  `unix:` baseUrl speaks **HTTP over a unix socket** via undici, so it CANNOT reach that
+  default uwsgi socket directly. Three ways to reach the install-script instance:
+  - **Point webveil straight at an HTTP unix socket** (no proxy, no extra process), once the
+    socket actually speaks HTTP. The install-script default does NOT, so first make uWSGI
+    serve HTTP on the socket: in the generated `.ini`, replace
+    `socket = /usr/local/searxng/run/socket` with
+    `http-socket = /usr/local/searxng/run/socket` (HTTP over the socket instead of the
+    uwsgi protocol). THEN point webveil at it with a `unix:` URL naming the socket file:
+    ```sh
+    export WEBVEIL_BASE_URL=unix:/usr/local/searxng/run/socket
+    ```
+    webveil dials the socket directly over undici (`Agent({connect:{socketPath}})`, no
+    extra dependency) and issues its normal `/search?...&format=json` request. The grammar
+    is `unix:<socketPath>[:<httpPath>]`: the socket file path, then an OPTIONAL `:` +
+    base path (mount point) the SearXNG app lives under (defaults to `/`, so the example
+    above requests `/search`; a non-root mount is `unix:/usr/local/searxng/run/socket:/searxng`).
+    (`unix:` works against ANY HTTP-on-a-unix-socket server, e.g. a Caddy/nginx upstream
+    bound to a socket; the uwsgi-vs-`http-socket` distinction above is the SearXNG-specific
+    catch.)
+    **Egress must be `direct`** for this: a Unix socket is inherently local, so combining a
+    `unix:` baseUrl with `egress=http`/`socks5` fails loud (proxying a local hop is fake
+    anonymity, see "Where does anonymity live?" below; proxy SearXNG's `outgoing.proxies`
+    instead and keep webveil `direct`).
+  - **Front it with a reverse proxy** (this is what the SearXNG docs' nginx/apache step is
+    for, it bridges HTTP-on-a-port to the uWSGI socket, serving BOTH the browser UI and
+    webveil). **Any HTTP server works**, the docs say so explicitly; **Caddy is fine** and
+    a good pick if you already run it. Plain Caddy `reverse_proxy` speaks **HTTP** to its
+    upstream, so point it at an `http-socket` (see below) or a TCP `http-socket`:
+    ```caddy
+    searxng.example.com {
+        reverse_proxy unix//usr/local/searxng/run/socket   # plain reverse_proxy = HTTP, so the socket must be http-socket = (not the uwsgi socket =)
+    }
+    ```
+    Then point webveil at the Caddy address. (Set SearXNG's `server.base_url` in
+    `settings.yml` to match, and keep the limiter in mind, see below.) If you want a Caddy
+    frontend AND webveil-direct, the simplest path is ONE `http-socket` that both consume
+    (Caddy's HTTP `reverse_proxy` and webveil's `unix:` both speak HTTP to it); you only
+    need the uwsgi `socket = ` form if Caddy uses an explicit uwsgi transport.
+  - **Or make uWSGI listen on a TCP port** instead of the socket: in the generated
+    `.ini`, replace `socket = …/run/socket` with `http-socket = 127.0.0.1:8888`, then point
+    webveil at `http://127.0.0.1:8888`. Good when you want ONLY webveil (no public web UI /
+    TLS).
+
+> **You will also need to enable the JSON API and (for a local instance) disable the
+> limiter.** A fresh script install ships with `server.limiter: true` and often no `json`
+> output format, so webveil gets `429 TOO MANY REQUESTS` or an HTML page. In SearXNG's
+> `settings.yml` set `server.limiter: false` + `server.public_instance: false` (safe for a
+> LOCAL, socket-only instance, NOT internet-exposed) and add `json` under `search.formats:`
+> (`[html, json]`), then restart uWSGI. This applies to EVERY option above, it is a
+> SearXNG-side requirement, not a webveil one.
+
+Full SearXNG install options (Docker, Compose, script, bare-metal): the official docs at
+<https://docs.searxng.org/admin/installation.html>. Install topology + the
+uwsgi-vs-`http-socket`, limiter, and reverse-proxy details captured in
+[`work/notes/findings/searxng-install-topology.md`](work/notes/findings/searxng-install-topology.md)
+and
+[`work/notes/findings/searxng-script-socket-is-uwsgi-not-http.md`](work/notes/findings/searxng-script-socket-is-uwsgi-not-http.md).
+
+### Where does anonymity live? (read before turning on egress)
+
+**webveil's egress only anonymizes webveil's OWN outbound hop** (webveil → backend, and
+`web_fetch` → the target URL). It does NOT anonymize what a backend does next. This has a
+load-bearing consequence for SearXNG:
+
+- A **local** SearXNG makes its actual search-engine requests (→ Google/Bing/…) from
+  **its own process, on your machine, with your real IP**. That hop is OUTSIDE webveil's
+  egress. So setting `WEBVEIL_EGRESS=socks5` while `baseUrl` is `127.0.0.1` does **NOT**
+  make your searches anonymous, webveil would just be proxying a pointless localhost call,
+  while SearXNG crawls the web from your real IP. That is **false confidence**, the worst
+  outcome.
+- **webveil refuses this combo (fail-loud):** a non-`direct` egress (`http`/`socks5`) with
+  a **loopback `baseUrl`** is rejected with an error, rather than silently giving you fake
+  anonymity. (A *remote* SearXNG over SOCKS is legitimate and allowed, the guard keys on
+  loopback specifically.)
+
+So the correct setups:
+
+| Goal | webveil egress | backend | Who anonymizes the web hop |
+| --- | --- | --- | --- |
+| Local SearXNG, anonymous searches | `direct` | local SearXNG | **SearXNG itself**, set its `outgoing.proxies` (Tor/SOCKS) in `settings.yml` |
+| Remote SearXNG, hide your IP from it | `socks5` | the **remote** SearXNG url | webveil's hop (Mullvad/Tor) |
+| Anonymous `web_fetch` of arbitrary URLs | `socks5` | (any) | webveil's hop |
+| Non-anonymous everyday use | `direct` | local SearXNG | nobody (honest) |
+
+Rule of thumb: **proxy the hop that actually reaches the public internet.** For a
+self-hosted SearXNG that hop is SearXNG's, so the proxy goes on SearXNG
+(`outgoing.proxies`), and webveil stays `direct`. webveil's `socks5` mode is for *remote*
+backends and for `web_fetch`. See
+[`work/notes/findings/webveil-anonymity-boundary.md`](work/notes/findings/webveil-anonymity-boundary.md).
+
+## How it works (seams)
+
+- **core**, the framework-agnostic `search(query, opts)` and `fetch(url, opts)` functions.
+  Both frontends call the same core.
+- **backend seam**, where results/content come from: `searxng` (keyless self-hosted
+  metasearch), `tavily-compat` (a generic Tavily-shaped `/search` + `/extract`), and
+  `custom` (a local command via a JSON stdin/stdout contract). The backend is handed a
+  proxied `http` helper so it cannot bypass egress.
+- **egress seam**, how outbound HTTP leaves the machine: `direct`, `http` (undici
+  `ProxyAgent`), or `socks5` (Tor `127.0.0.1:9050`, Mullvad `10.64.0.1:1080`). SOCKS5 is
+  the mode that matters for anonymity. Fail-loud if a configured proxy cannot be built.
+  **Egress is per-request and scoped to webveil ONLY**, it is NOT a system-wide proxy. It
+  governs webveil's own search/fetch traffic (and the `fetch` it injects into distilly),
+  and nothing else: your shell, `git push`, the browser, and the OS are untouched. So
+  webveil on `socks5` does NOT route your `git push` through the proxy. See
+  [Anonymous egress](#anonymous-egress-mullvad--tor) and
+  [`work/notes/findings/mullvad-socks5-egress-mechanics.md`](work/notes/findings/mullvad-socks5-egress-mechanics.md).
+- **config seam**, per-folder resolution: env > nearest `.pi/webveil.json` walking up from
+  cwd > global `~/.pi/agent/webveil.json` > defaults. Per folder = per account/egress.
+- **extractor seam**, `urlToMarkdown` via `distilly/fetch` by default, injected with
+  webveil's egress-bound `fetch`; a backend's own `/extract` (Tavily-compat) may override
+  it. Owns the context-friendly markdown + size presets (`s`/`m`/`l`/`f`). See
+  [`docs/adr/0001`](docs/adr/0001-extractor-uses-distilly-fetch-with-injected-egress.md).
+- **security**, an SSRF guard lives in the egress fetch, so it covers distilly's
+  rule-rewritten requests too.
+
+## Anonymous egress (Mullvad / Tor)
+
+By default webveil uses `direct` egress (your real IP, non-anonymous). Anonymity is
+**opt-in**: it is enabled ONLY when you set it in config/env. webveil never auto-enables a
+proxy (silent anonymity would be a footgun in the other direction).
+
+Enable SOCKS5 egress for webveil:
+
+```sh
+export WEBVEIL_EGRESS=socks5
+export WEBVEIL_EGRESS_URL=socks5://10.64.0.1:1080     # Mullvad
+# or socks5://127.0.0.1:9050                          # Tor
+```
+
+or per folder in `.pi/webveil.json`:
+
+```json
+{ "egress": { "mode": "socks5", "url": "socks5://10.64.0.1:1080" } }
+```
+
+### Two layers keep your `git push` (and everything else) off the proxy
+
+A common worry: "if I route through Mullvad, will my `git push` to GitHub leak under the
+VPN exit IP?" With webveil, **no**, for two independent reasons:
+
+1. **webveil's egress is per-request and webveil-only.** It applies the SOCKS5 dispatcher
+   inside its own search/fetch code; it does not install a system proxy. `git`, your shell,
+   and the OS are never touched. webveil on `socks5` proxies webveil's traffic and nothing
+   else.
+2. **You configure split routing** (below) so that even at the OS level, only the proxy IP
+   goes through the tunnel.
+
+### Mullvad: use the SOCKS5 proxy WITHOUT tunnelling all your traffic
+
+Mullvad's SOCKS5 proxy at `10.64.0.1:1080` **only exists while a Mullvad WireGuard tunnel
+is up** (it is reachable only through the tunnel). The trick is to keep the tunnel up but
+tell WireGuard NOT to route your normal traffic through it, only the proxy IP. Add this to
+your Mullvad WireGuard `.conf` (`[Interface]` section):
+
+```ini
+Table = off
+PostUp  = ip -4 route add 10.64.0.1/32 dev %i; ip -4 route add 10.124.0.0/22 dev %i
+PreDown = ip -4 route delete 10.64.0.1/32 dev %i; ip -4 route delete 10.124.0.0/22 dev %i
+```
+
+`Table = off` stops WireGuard from grabbing the default route; the manual routes send ONLY
+Mullvad's SOCKS5 proxy IPs through the tunnel (`10.124.0.0/22` is the multihop range).
+Result: webveil's SOCKS5 requests exit via Mullvad; all other traffic (git, browser, OS)
+uses your normal ISP connection. (Simpler alternative: leave WireGuard's routing alone and
+rely on layer 1, but split routing is the belt-and-braces version.)
+
+Verify the proxy works: `curl https://ipv4.am.i.mullvad.net --socks5-hostname 10.64.0.1`
+should return a Mullvad exit IP; a plain `curl https://am.i.mullvad.net` should return your
+real IP (proving only the proxy is tunnelled).
+
+### "Different exit identity for webveil than for the rest of the machine"
+
+If you want webveil to exit somewhere different from your system, you have options, but be
+clear on what is and isn't possible (see
+[`work/notes/findings/mullvad-socks5-egress-mechanics.md`](work/notes/findings/mullvad-socks5-egress-mechanics.md)):
+
+- **Different exit LOCATION, same account (easy).** Point webveil at a specific multihop
+  SOCKS5 host so it exits elsewhere than your tunnel's entry:
+  `WEBVEIL_EGRESS_URL=socks5://us-nyc-wg-socks5-001.relays.mullvad.net:1080`. Your tunnel
+  enters where your Mullvad app is connected; webveil's traffic exits in NYC. Same Mullvad
+  account, unlinkable-by-location.
+- **Two DIFFERENT Mullvad ACCOUNTS at once (hard, not a webveil feature).** Mullvad's
+  SOCKS5 proxy is a property of the ONE active WireGuard tunnel, which is tied to ONE
+  account's key. SOCKS5 multihop changes exit location, NOT account. To run account A
+  system-wide AND account B for webveil simultaneously, you must isolate them at the OS
+  level: run webveil inside its own network namespace / VM / container that has its own
+  WireGuard tunnel on account B, while the host runs account A. That is infrastructure work
+  outside webveil. For most people, "don't link my searches to my git" is already solved by
+  split routing above (searches exit via Mullvad, git stays on your real IP, not correlated
+  by exit IP), without needing a second account.
+
+### Tor
+
+`WEBVEIL_EGRESS_URL=socks5://127.0.0.1:9050` with the Tor daemon running. Same per-request,
+webveil-only scoping applies.
+
+> **Caveat:** webveil's `socks5` mode is NOT a whole-machine VPN. Do not assume enabling it
+> anonymizes anything other than webveil. Conversely, a system-wide full-tunnel VPN under
+> your logged-in identity is the thing that CAN deanonymize a `git push`; webveil's scoped
+> egress deliberately avoids that.
+
+## License
+
+AGPL-3.0-or-later. webveil depends on `distilly` (MIT, the local HTML-to-markdown
+extractor; webveil uses its networked `distilly/fetch` entrypoint with an injected egress
+fetch) and `incur` (MIT). MIT code may be used by AGPL software; `distilly` stays
+GPL/AGPL-free so it remains cleanly reusable under MIT. See [`LICENSE`](LICENSE) and
+[`COPYRIGHT`](COPYRIGHT).
+
+## Size discipline (per-module LOC)
+
+Every module stays small with one responsibility. Per-module LOC is tracked here as a
+first-class quality signal. `target` is the rough ceiling from `CONTEXT.md` (a ceiling, not
+a promise); `LOC` is the actual line count of the built file.
+
+### `packages/webveil` (core + CLI/MCP frontend)
+
+| module                             |  LOC | target |
+| ---------------------------------- | ---: | -----: |
+| src/index.ts (barrel)              |   82 |      - |
+| src/cli.ts (incur frontend)        |  106 |    ~80 |
+| src/core/search.ts                 |  104 |    ~90 |
+| src/core/fetch.ts                  |  132 |    ~90 |
+| src/core/config.ts                 |  106 |    ~80 |
+| src/core/egress.ts                 |  106 |    ~70 |
+| src/core/http.ts                   |   62 |    ~60 |
+| src/core/extract.ts                |   82 |    ~60 |
+| src/core/security.ts (SSRF guard)  |  141 |      - |
+| src/core/backends/types.ts         |   61 |    ~40 |
+| src/core/backends/registry.ts      |   41 |    ~60 |
+| src/core/backends/searxng.ts       |   70 |    ~90 |
+| src/core/backends/tavily-compat.ts |  156 |    ~90 |
+| src/core/backends/custom.ts        |  159 |    ~70 |
+| **subtotal**                       | 1408 |        |
+
+### `packages/pi-webveil` (pi extension frontend)
+
+| module       | LOC | target |
+| ------------ | --: | -----: |
+| src/index.ts | 168 |    ~90 |
+
+**Total own source: 1576 LOC** (excluding deps).
+
+> Reality vs. target: several modules currently exceed their `CONTEXT.md` ceilings (notably
+> `tavily-compat.ts`, `custom.ts`, `pi-webveil/src/index.ts`), and two built modules
+> (`index.ts` barrel and `security.ts` SSRF guard) were not in the original target list. The
+> table above reflects the modules as actually built. For calibration, comparable pi
+> web-search extensions: `pi-searxng-search` 350 LOC (1 backend, no egress, no fetch),
+> `leing2021/pi-search` 1714, `pi-search-hub` 9047, `pi-web-providers` 18961. webveil
+> delivers a 3-backend + egress + fetch + per-folder-config tool by leaning on `incur`
+> (CLI/MCP/skills) and `distilly` (extraction).
+
+## Develop
+
+```sh
+pnpm install
+pnpm build
+pnpm test
+pnpm format:check
+```

package/dist/cli.d.ts

@@ -0,0 +1,58 @@
+#!/usr/bin/env node
+import { Cli } from 'incur';
+import { search as coreSearch } from './core/search.js';
+import { fetch as coreFetch } from './core/fetch.js';
+/**
+ * The two core functions the frontend wraps, seamed so tests can inject fakes.
+ * Defaults are the real core; a test passes spies to assert the wiring.
+ */
+export interface CliDeps {
+    search?: typeof coreSearch;
+    fetch?: typeof coreFetch;
+}
+/**
+ * Build the webveil CLI. Returns the incur `Cli` so a caller (the bin below, or
+ * a test) decides how to serve it. The `search`/`fetch` commands forward to the
+ * injected core, normalizing nothing themselves — the core already deduped,
+ * clamped, and size-bounded.
+ */
+export declare function createCli(deps?: CliDeps): Cli.Cli<{
+    search: {
+        args: {
+            query: string;
+        };
+        options: {
+            maxResults?: number | undefined;
+        };
+    };
+} & {
+    fetch: {
+        args: {
+            url: string;
+        };
+        options: {
+            size?: "s" | "m" | "l" | "f" | undefined;
+        };
+    };
+}, undefined, undefined, undefined>;
+declare const cli: Cli.Cli<{
+    search: {
+        args: {
+            query: string;
+        };
+        options: {
+            maxResults?: number | undefined;
+        };
+    };
+} & {
+    fetch: {
+        args: {
+            url: string;
+        };
+        options: {
+            size?: "s" | "m" | "l" | "f" | undefined;
+        };
+    };
+}, undefined, undefined, undefined>;
+export default cli;
+//# sourceMappingURL=cli.d.ts.map

package/dist/cli.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAmBA,OAAO,EAAC,GAAG,EAAI,MAAM,OAAO,CAAC;AAC7B,OAAO,EAAC,MAAM,IAAI,UAAU,EAAC,MAAM,kBAAkB,CAAC;AACtD,OAAO,EAAC,KAAK,IAAI,SAAS,EAAC,MAAM,iBAAiB,CAAC;AAEnD;;;GAGG;AACH,MAAM,WAAW,OAAO;IACvB,MAAM,CAAC,EAAE,OAAO,UAAU,CAAC;IAC3B,KAAK,CAAC,EAAE,OAAO,SAAS,CAAC;CACzB;AAKD;;;;;GAKG;AACH,wBAAgB,SAAS,CAAC,IAAI,GAAE,OAAY;;;;;;;;;;;;;;;;;;oCA4C3C;AAKD,QAAA,MAAM,GAAG;;;;;;;;;;;;;;;;;;mCAAc,CAAC;AAexB,eAAe,GAAG,CAAC"}

package/dist/cli.js

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+// webveil — the incur-based CLI + MCP frontend. ONE `Cli.create()` definition
+// yields the CLI, an MCP server (`--mcp`), skills (`skills add`), a `--llms`
+// manifest, TOON output, and token pagination for free (incur). Pi-agnostic:
+// any agent (pi via pi-mcp-adapter, Claude Code, Cursor, Codex, bash) consumes
+// it the same way. The `webveil` bin points at the built `dist/cli.js`.
+//
+// This is the THIN frontend: each command only parses argv/options and calls
+// the SAME framework-agnostic core (`search()` / `fetch()`) the pi extension
+// calls. The core owns config/egress/backend/extraction; this file owns no
+// network logic of its own.
+//
+// Testability: `createCli(deps)` takes the core functions as injectable deps so
+// a test wires fakes and asserts the commands call the core (via `cli.serve`
+// with custom argv/stdout) WITHOUT touching the network. The bottom of the file
+// builds the real CLI and serves it when run as the bin.
+import { argv } from 'node:process';
+import { fileURLToPath } from 'node:url';
+import { Cli, z } from 'incur';
+import { search as coreSearch } from './core/search.js';
+import { fetch as coreFetch } from './core/fetch.js';
+/** The size presets `fetch` accepts, mirroring the core's `FetchSize`. */
+const SIZES = ['s', 'm', 'l', 'f'];
+/**
+ * Build the webveil CLI. Returns the incur `Cli` so a caller (the bin below, or
+ * a test) decides how to serve it. The `search`/`fetch` commands forward to the
+ * injected core, normalizing nothing themselves — the core already deduped,
+ * clamped, and size-bounded.
+ */
+export function createCli(deps = {}) {
+    const search = deps.search ?? coreSearch;
+    const fetch = deps.fetch ?? coreFetch;
+    return Cli.create('webveil', {
+        description: 'Anonymous-capable, self-hosted, account-free web search + fetch for agents.',
+    })
+        .command('search', {
+        description: 'Search the web via the configured backend and egress.',
+        args: z.object({
+            query: z.string().describe('The search query'),
+        }),
+        options: z.object({
+            maxResults: z.coerce
+                .number()
+                .optional()
+                .describe('Maximum number of results to return'),
+        }),
+        alias: { maxResults: 'n' },
+        async run(c) {
+            const results = await search(c.args.query, {
+                maxResults: c.options.maxResults,
+            });
+            return { results };
+        },
+    })
+        .command('fetch', {
+        description: 'Fetch a URL as clean, size-bounded markdown via the configured egress.',
+        args: z.object({
+            url: z.string().describe('The URL to fetch'),
+        }),
+        options: z.object({
+            size: z
+                .enum(SIZES)
+                .optional()
+                .describe('Page-size budget preset: s | m | l | f'),
+        }),
+        alias: { size: 's' },
+        async run(c) {
+            return fetch(c.args.url, { size: c.options.size });
+        },
+    });
+}
+// The real CLI (also `export default` so `incur gen` can import it for typed
+// CTAs). Serving is GUARDED to the bin entry below, so importing this module in
+// a test never consumes `process.argv` or exits the process.
+const cli = createCli();
+/** True when this module is the process entry (the `webveil` bin), not imported. */
+function isMain() {
+    const entry = argv[1];
+    if (!entry)
+        return false;
+    try {
+        return fileURLToPath(import.meta.url) === entry;
+    }
+    catch {
+        return false;
+    }
+}
+if (isMain())
+    cli.serve();
+export default cli;
+//# sourceMappingURL=cli.js.map

package/dist/cli.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA,8EAA8E;AAC9E,6EAA6E;AAC7E,6EAA6E;AAC7E,+EAA+E;AAC/E,wEAAwE;AACxE,EAAE;AACF,6EAA6E;AAC7E,6EAA6E;AAC7E,2EAA2E;AAC3E,4BAA4B;AAC5B,EAAE;AACF,gFAAgF;AAChF,6EAA6E;AAC7E,gFAAgF;AAChF,yDAAyD;AAEzD,OAAO,EAAC,IAAI,EAAC,MAAM,cAAc,CAAC;AAClC,OAAO,EAAC,aAAa,EAAC,MAAM,UAAU,CAAC;AACvC,OAAO,EAAC,GAAG,EAAE,CAAC,EAAC,MAAM,OAAO,CAAC;AAC7B,OAAO,EAAC,MAAM,IAAI,UAAU,EAAC,MAAM,kBAAkB,CAAC;AACtD,OAAO,EAAC,KAAK,IAAI,SAAS,EAAC,MAAM,iBAAiB,CAAC;AAWnD,0EAA0E;AAC1E,MAAM,KAAK,GAAG,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,CAAU,CAAC;AAE5C;;;;;GAKG;AACH,MAAM,UAAU,SAAS,CAAC,OAAgB,EAAE;IAC3C,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,IAAI,UAAU,CAAC;IACzC,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,IAAI,SAAS,CAAC;IAEtC,OAAO,GAAG,CAAC,MAAM,CAAC,SAAS,EAAE;QAC5B,WAAW,EACV,6EAA6E;KAC9E,CAAC;SACA,OAAO,CAAC,QAAQ,EAAE;QAClB,WAAW,EAAE,uDAAuD;QACpE,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC;YACd,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,kBAAkB,CAAC;SAC9C,CAAC;QACF,OAAO,EAAE,CAAC,CAAC,MAAM,CAAC;YACjB,UAAU,EAAE,CAAC,CAAC,MAAM;iBAClB,MAAM,EAAE;iBACR,QAAQ,EAAE;iBACV,QAAQ,CAAC,qCAAqC,CAAC;SACjD,CAAC;QACF,KAAK,EAAE,EAAC,UAAU,EAAE,GAAG,EAAC;QACxB,KAAK,CAAC,GAAG,CAAC,CAAC;YACV,MAAM,OAAO,GAAG,MAAM,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,EAAE;gBAC1C,UAAU,EAAE,CAAC,CAAC,OAAO,CAAC,UAAU;aAChC,CAAC,CAAC;YACH,OAAO,EAAC,OAAO,EAAC,CAAC;QAClB,CAAC;KACD,CAAC;SACD,OAAO,CAAC,OAAO,EAAE;QACjB,WAAW,EACV,wEAAwE;QACzE,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC;YACd,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,kBAAkB,CAAC;SAC5C,CAAC;QACF,OAAO,EAAE,CAAC,CAAC,MAAM,CAAC;YACjB,IAAI,EAAE,CAAC;iBACL,IAAI,CAAC,KAAK,CAAC;iBACX,QAAQ,EAAE;iBACV,QAAQ,CAAC,wCAAwC,CAAC;SACpD,CAAC;QACF,KAAK,EAAE,EAAC,IAAI,EAAE,GAAG,EAAC;QAClB,KAAK,CAAC,GAAG,CAAC,CAAC;YACV,OAAO,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,EAAC,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,IAAI,EAAC,CAAC,CAAC;QAClD,CAAC;KACD,CAAC,CAAC;AACL,CAAC;AAED,6EAA6E;AAC7E,gFAAgF;AAChF,6DAA6D;AAC7D,MAAM,GAAG,GAAG,SAAS,EAAE,CAAC;AAExB,oFAAoF;AACpF,SAAS,MAAM;IACd,MAAM,KAAK,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;IACtB,IAAI,CAAC,KAAK;QAAE,OAAO,KAAK,CAAC;IACzB,IAAI,CAAC;QACJ,OAAO,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,KAAK,CAAC;IACjD,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,KAAK,CAAC;IACd,CAAC;AACF,CAAC;AAED,IAAI,MAAM,EAAE;IAAE,GAAG,CAAC,KAAK,EAAE,CAAC;AAE1B,eAAe,GAAG,CAAC"}

package/dist/core/backends/custom.d.ts

@@ -0,0 +1,15 @@
+import { spawn as defaultSpawn } from 'node:child_process';
+import type { Config } from '../config.js';
+import type { Backend } from './types.js';
+/**
+ * Minimal `spawn` shape this backend needs, seamed so a test can inject a fake
+ * without a real subprocess. Defaults to `node:child_process` `spawn`.
+ */
+export type SpawnFn = typeof defaultSpawn;
+/**
+ * Build a custom backend bound to the configured command. The command owns its
+ * own I/O; webveil hands it the request as JSON on stdin and parses
+ * SearchResult[] from stdout, failing clearly on malformed output.
+ */
+export declare function createCustomBackend(config: Config, spawn?: SpawnFn): Backend;
+//# sourceMappingURL=custom.d.ts.map

package/dist/core/backends/custom.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"custom.d.ts","sourceRoot":"","sources":["../../../src/core/backends/custom.ts"],"names":[],"mappings":"AAoBA,OAAO,EAAC,KAAK,IAAI,YAAY,EAAC,MAAM,oBAAoB,CAAC;AACzD,OAAO,KAAK,EAAC,MAAM,EAAC,MAAM,cAAc,CAAC;AACzC,OAAO,KAAK,EAAC,OAAO,EAAoC,MAAM,YAAY,CAAC;AAe3E;;;GAGG;AACH,MAAM,MAAM,OAAO,GAAG,OAAO,YAAY,CAAC;AAsF1C;;;;GAIG;AACH,wBAAgB,mBAAmB,CAClC,MAAM,EAAE,MAAM,EACd,KAAK,GAAE,OAAsB,GAC3B,OAAO,CAuBT"}

package/dist/core/backends/custom.js

@@ -0,0 +1,106 @@
+// custom backend — the local-command escape hatch (contract lifted from
+// pi-web-providers' custom-wrapper). Instead of an HTTP source, it spawns a
+// configured local command, writes the request as JSON to its stdin, and parses
+// `SearchResult[]` from its stdout. This lets any local script be a backend.
+//
+// Egress note: this backend owns its own I/O (the spawned command does whatever
+// it wants), so the handed `http` helper is unused here — there is no outbound
+// HTTP for webveil to proxy. It still returns the normalized SearchResult shape.
+//
+// Command source: the configured `baseUrl` carries the command line, parsed as a
+// whitespace-separated argv (first token = executable, rest = args), matching how
+// the other backends read `baseUrl` as "where results come from". (Recorded
+// decision; see the task's Decisions block.)
+//
+// Contract:
+//   stdin  <- JSON: {"query": string, "maxResults"?: number}
+//   stdout -> JSON: SearchResult[]  (each {title, url, snippet?})
+// Malformed stdout (non-JSON, not an array, or entries missing url/title) FAILS
+// CLEARLY — it never silently returns an empty list.
+import { spawn as defaultSpawn } from 'node:child_process';
+function str(value) {
+    return typeof value === 'string' && value.length > 0 ? value : undefined;
+}
+/** Parse the configured command line into [executable, ...args]. */
+function parseCommand(baseUrl) {
+    const parts = baseUrl.trim().split(/\s+/).filter(Boolean);
+    if (parts.length === 0)
+        throw new Error('custom: no command configured (set baseUrl to the command to run)');
+    return [parts[0], parts.slice(1)];
+}
+/**
+ * Normalize one stdout entry into a SearchResult, FAILING CLEARLY on a malformed
+ * entry rather than dropping it — the custom contract is explicit, so a missing
+ * url/title is a contract violation the user should see, not a silent skip.
+ */
+function toResult(entry, index) {
+    if (typeof entry !== 'object' || entry === null)
+        throw new Error(`custom: malformed output — result[${index}] is not an object`);
+    const hit = entry;
+    const url = str(hit.url);
+    const title = str(hit.title);
+    if (!url || !title)
+        throw new Error(`custom: malformed output — result[${index}] is missing a url or title`);
+    const snippet = str(hit.snippet);
+    return snippet ? { title, url, snippet } : { title, url };
+}
+/** Parse the command's stdout into SearchResult[], failing clearly on garbage. */
+function parseOutput(stdout) {
+    const trimmed = stdout.trim();
+    if (trimmed.length === 0)
+        throw new Error('custom: command produced no output');
+    let parsed;
+    try {
+        parsed = JSON.parse(trimmed);
+    }
+    catch (cause) {
+        throw new Error(`custom: malformed output — stdout is not valid JSON: ${cause.message}`);
+    }
+    if (!Array.isArray(parsed))
+        throw new Error('custom: malformed output — expected a JSON array of results');
+    return parsed.map(toResult);
+}
+/** Spawn the command, write the request to stdin, and collect stdout/stderr. */
+function runCommand(spawn, exe, args, request, signal) {
+    return new Promise((resolve, reject) => {
+        const child = spawn(exe, args, {
+            stdio: ['pipe', 'pipe', 'pipe'],
+            signal,
+        });
+        let stdout = '';
+        let stderr = '';
+        child.stdout?.on('data', (chunk) => (stdout += String(chunk)));
+        child.stderr?.on('data', (chunk) => (stderr += String(chunk)));
+        child.on('error', (err) => reject(new Error(`custom: failed to spawn '${exe}': ${err.message}`)));
+        child.on('close', (code) => resolve({ stdout, stderr, code }));
+        child.stdin?.on('error', () => {
+            // A command that exits before reading stdin closes the pipe; ignore the
+            // EPIPE here and let the close handler report via exit code/stderr.
+        });
+        child.stdin?.end(JSON.stringify(request));
+    });
+}
+/**
+ * Build a custom backend bound to the configured command. The command owns its
+ * own I/O; webveil hands it the request as JSON on stdin and parses
+ * SearchResult[] from stdout, failing clearly on malformed output.
+ */
+export function createCustomBackend(config, spawn = defaultSpawn) {
+    const [exe, args] = parseCommand(config.baseUrl);
+    return {
+        async search(query, _http, options = {}) {
+            const request = { query };
+            if (options.maxResults !== undefined)
+                request.maxResults = options.maxResults;
+            const run = await runCommand(spawn, exe, args, request, options.signal);
+            if (run.code !== 0)
+                throw new Error(`custom: command '${exe}' exited with code ${run.code}` +
+                    (run.stderr.trim() ? `: ${run.stderr.trim()}` : ''));
+            const results = parseOutput(run.stdout);
+            return options.maxResults !== undefined
+                ? results.slice(0, options.maxResults)
+                : results;
+        },
+    };
+}
+//# sourceMappingURL=custom.js.map

package/dist/core/backends/custom.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"custom.js","sourceRoot":"","sources":["../../../src/core/backends/custom.ts"],"names":[],"mappings":"AAAA,wEAAwE;AACxE,4EAA4E;AAC5E,gFAAgF;AAChF,6EAA6E;AAC7E,EAAE;AACF,gFAAgF;AAChF,+EAA+E;AAC/E,iFAAiF;AACjF,EAAE;AACF,iFAAiF;AACjF,kFAAkF;AAClF,4EAA4E;AAC5E,6CAA6C;AAC7C,EAAE;AACF,YAAY;AACZ,6DAA6D;AAC7D,kEAAkE;AAClE,gFAAgF;AAChF,qDAAqD;AAErD,OAAO,EAAC,KAAK,IAAI,YAAY,EAAC,MAAM,oBAAoB,CAAC;AAuBzD,SAAS,GAAG,CAAC,KAAc;IAC1B,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;AAC1E,CAAC;AAED,oEAAoE;AACpE,SAAS,YAAY,CAAC,OAAe;IACpC,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IAC1D,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QACrB,MAAM,IAAI,KAAK,CACd,mEAAmE,CACnE,CAAC;IACH,OAAO,CAAC,KAAK,CAAC,CAAC,CAAE,EAAE,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;AACpC,CAAC;AAED;;;;GAIG;AACH,SAAS,QAAQ,CAAC,KAAc,EAAE,KAAa;IAC9C,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI;QAC9C,MAAM,IAAI,KAAK,CACd,qCAAqC,KAAK,oBAAoB,CAC9D,CAAC;IACH,MAAM,GAAG,GAAG,KAAgC,CAAC;IAC7C,MAAM,GAAG,GAAG,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IACzB,MAAM,KAAK,GAAG,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;IAC7B,IAAI,CAAC,GAAG,IAAI,CAAC,KAAK;QACjB,MAAM,IAAI,KAAK,CACd,qCAAqC,KAAK,6BAA6B,CACvE,CAAC;IACH,MAAM,OAAO,GAAG,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACjC,OAAO,OAAO,CAAC,CAAC,CAAC,EAAC,KAAK,EAAE,GAAG,EAAE,OAAO,EAAC,CAAC,CAAC,CAAC,EAAC,KAAK,EAAE,GAAG,EAAC,CAAC;AACvD,CAAC;AAED,kFAAkF;AAClF,SAAS,WAAW,CAAC,MAAc;IAClC,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,EAAE,CAAC;IAC9B,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QACvB,MAAM,IAAI,KAAK,CAAC,oCAAoC,CAAC,CAAC;IACvD,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACJ,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;IAC9B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QAChB,MAAM,IAAI,KAAK,CACd,wDAAyD,KAAe,CAAC,OAAO,EAAE,CAClF,CAAC;IACH,CAAC;IACD,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC;QACzB,MAAM,IAAI,KAAK,CACd,6DAA6D,CAC7D,CAAC;IACH,OAAO,MAAM,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;AAC7B,CAAC;AAED,gFAAgF;AAChF,SAAS,UAAU,CAClB,KAAc,EACd,GAAW,EACX,IAAc,EACd,OAAsB,EACtB,MAAoB;IAEpB,OAAO,IAAI,OAAO,CAAa,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QAClD,MAAM,KAAK,GAAG,KAAK,CAAC,GAAG,EAAE,IAAI,EAAE;YAC9B,KAAK,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC;YAC/B,MAAM;SACN,CAAC,CAAC;QACH,IAAI,MAAM,GAAG,EAAE,CAAC;QAChB,IAAI,MAAM,GAAG,EAAE,CAAC;QAChB,KAAK,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,MAAM,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;QAC/D,KAAK,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,MAAM,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;QAC/D,KAAK,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,GAAG,EAAE,EAAE,CACzB,MAAM,CAAC,IAAI,KAAK,CAAC,4BAA4B,GAAG,MAAM,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC,CACrE,CAAC;QACF,KAAK,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,CAAC,EAAC,MAAM,EAAE,MAAM,EAAE,IAAI,EAAC,CAAC,CAAC,CAAC;QAC7D,KAAK,CAAC,KAAK,EAAE,EAAE,CAAC,OAAO,EAAE,GAAG,EAAE;YAC7B,wEAAwE;YACxE,oEAAoE;QACrE,CAAC,CAAC,CAAC;QACH,KAAK,CAAC,KAAK,EAAE,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC,CAAC;IAC3C,CAAC,CAAC,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,mBAAmB,CAClC,MAAc,EACd,QAAiB,YAAY;IAE7B,MAAM,CAAC,GAAG,EAAE,IAAI,CAAC,GAAG,YAAY,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IACjD,OAAO;QACN,KAAK,CAAC,MAAM,CACX,KAAa,EACb,KAAW,EACX,UAAyB,EAAE;YAE3B,MAAM,OAAO,GAAkB,EAAC,KAAK,EAAC,CAAC;YACvC,IAAI,OAAO,CAAC,UAAU,KAAK,SAAS;gBACnC,OAAO,CAAC,UAAU,GAAG,OAAO,CAAC,UAAU,CAAC;YACzC,MAAM,GAAG,GAAG,MAAM,UAAU,CAAC,KAAK,EAAE,GAAG,EAAE,IAAI,EAAE,OAAO,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC;YACxE,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC;gBACjB,MAAM,IAAI,KAAK,CACd,oBAAoB,GAAG,sBAAsB,GAAG,CAAC,IAAI,EAAE;oBACtD,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CACpD,CAAC;YACH,MAAM,OAAO,GAAG,WAAW,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;YACxC,OAAO,OAAO,CAAC,UAAU,KAAK,SAAS;gBACtC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,OAAO,CAAC,UAAU,CAAC;gBACtC,CAAC,CAAC,OAAO,CAAC;QACZ,CAAC;KACD,CAAC;AACH,CAAC"}

package/dist/core/backends/registry.d.ts

@@ -0,0 +1,13 @@
+import type { Config } from '../config.js';
+import type { Backend } from './types.js';
+/** Builds a Backend from the resolved config (knows its baseUrl / apiKey). */
+export type BackendFactory = (config: Config) => Backend;
+/** The backend names the registry can resolve. */
+export declare function backendNames(): string[];
+/**
+ * Resolve a backend name to a constructed Backend. Throws clearly on an unknown
+ * name (listing the known ones) so a misconfigured `backend` fails loud, never
+ * silently no-ops.
+ */
+export declare function getBackend(name: string, config: Config): Backend;
+//# sourceMappingURL=registry.d.ts.map

package/dist/core/backends/registry.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"registry.d.ts","sourceRoot":"","sources":["../../../src/core/backends/registry.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAC,MAAM,EAAC,MAAM,cAAc,CAAC;AACzC,OAAO,KAAK,EAAC,OAAO,EAAC,MAAM,YAAY,CAAC;AAKxC,8EAA8E;AAC9E,MAAM,MAAM,cAAc,GAAG,CAAC,MAAM,EAAE,MAAM,KAAK,OAAO,CAAC;AASzD,kDAAkD;AAClD,wBAAgB,YAAY,IAAI,MAAM,EAAE,CAEvC;AAED;;;;GAIG;AACH,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAOhE"}