@lincoln504/pi-research 0.1.12 → 1.0.0

package/docs/CONFIGURATION.md

@@ -0,0 +1,249 @@
+## Configuration
+
+Every front-end (the pi extension, the standalone CLI / agent skill — the surface
+OpenClaw and other skills-aware hosts run — and the SDK) shares one configuration
+model. This document covers the
+settings exposed in the `/research-config` TUI first, then the complete
+environment-variable reference, and finally how the configuration layers resolve.
+
+![The /research-config settings TUI](media/04-config.gif)
+
+### Settings in the TUI
+
+Run `/research-config` in the pi extension to open an interactive menu. Selecting a
+setting and pressing `Enter` / `Space` cycles its value; the change is saved
+immediately. A setting is written to one of two scopes:
+
+- `[project]` — saved per working directory in the central project registry, so
+  a given repo can carry its own value without changing your global default.
+- user — saved to the shared base file (`config.env`), applying to every
+  directory and front-end unless a higher layer overrides it.
+
+| Setting | Scope | Values | Env var |
+|---------|-------|--------|---------|
+| `/research` depth | project | normal · deep · ultra | `PI_RESEARCH_DEFAULT_RESEARCH_DEPTH` |
+| Knowledge Mode | project | none · project · global | `PI_RESEARCH_KNOWLEDGE_STORE_MODE` |
+| Researcher Timeout | user | 3 · 5 · 10 · 15 · 20 · 30 (minutes) | `PI_RESEARCH_TIMEOUT_MS` |
+| Max Concurrency | user | 1 – 5 | `PI_RESEARCH_MAX_RESEARCHERS` |
+| Scrape Batches | user | unlimited · 1 · 2 · 3 · 5 · 10 · 15 | `PI_RESEARCH_MAX_SCRAPE_BATCHES` |
+| Auto-export Report | user | true · false | `PI_RESEARCH_REPORT_EXPORT_ENABLED` |
+| Embedding Model | user | one of the supported models | `PI_RESEARCH_EMBEDDING_MODEL` |
+| Embedding Device | user | GPU · CPU | `PI_RESEARCH_EMBEDDING_DEVICE` |
+| Cache Retention | user | 7 · 14 · 30 · 60 · 90 · 180 · 365 (days) | `PI_RESEARCH_CACHE_TTL_DAYS` |
+| Debug Logging | user | true · false | `PI_RESEARCH_DEBUG` |
+
+Embedding Device offers two choices in the menu. GPU maps to the
+auto-detect path: pi-research probes whether WebGPU actually runs on this machine
+and falls back to CPU if it does not. CPU forces CPU-only
+inference. Raw forced-GPU (no probe) is reachable only through the
+`PI_RESEARCH_EMBEDDING_DEVICE=webgpu` environment variable, for benchmarking — see
+the [knowledge store doc](KNOWLEDGE-STORE.md).
+
+The Embedding Model, Embedding Device, and Cache Retention rows appear
+only when Knowledge Mode is not `none`.
+
+The menu also offers actions that are not settings: Run Health Check, Store
+Status, Clear Project / User Store, Session Metrics, Clear Debug Logs, and
+Install / Remove in External Agents (the coding-agent
+skill installer). Browser worker count is deliberately not in the menu — it is
+CPU/RAM-sensitive and set only via `PI_RESEARCH_WORKER_THREADS`.
+
+### Environment variables
+
+Every setting is also an environment variable. The repo's
+[`.env.example`](https://github.com/Lincoln504/pi-research/blob/main/.env.example)
+is the canonical, exhaustive list with inline notes; this section groups the same
+variables with their defaults and valid ranges.
+Out-of-range numeric values are clamped (with a warning); an invalid enum value
+falls back to the default (with a warning).
+
+The TUI-exposed variables are marked `(TUI)`. The `[project]` mark indicates a
+project-scoped key (saved per directory in the registry); all others are
+user-scoped.
+
+Research
+
+| Variable | Default | Range | Description |
+|----------|---------|-------|-------------|
+| `PI_RESEARCH_TIMEOUT_MS` (TUI) | `300000` | 180000–1800000 | Per-researcher timeout (3–30 min). |
+| `PI_RESEARCH_MAX_RESEARCHERS` (TUI) | `3` | 1–5 | Parallel researchers. |
+| `PI_RESEARCH_DEFAULT_RESEARCH_DEPTH` (TUI) `[project]` | `1` | 1–3 | Depth for `/research` and the CLI when `--depth` is omitted (1=normal, 2=deep, 3=ultra). |
+| `PI_RESEARCH_MAX_SCRAPE_BATCHES` (TUI) | `2` | 0–99 | Scrape batches per researcher (0 = unlimited). |
+| `PI_RESEARCH_MAX_GATHERING_CALLS` | `12` | 1–100 | Shared web-gathering calls per researcher (`search` + `security_search` + `stackexchange` + `youtube_transcript`). Local `grep` has its own separate budget. |
+| `PI_RESEARCH_MAX_CONCURRENT_SCRAPES` | `3` | 1–20 | Concurrent URLs fetched per scrape batch. |
+| `PI_RESEARCH_MAX_RETRIES` | `2` | 0–5 | Retries per researcher request. |
+| `PI_RESEARCH_RETRY_DELAY_MS` | `2000` | 100–10000 | Base delay between retries. |
+| `PI_RESEARCH_WORKER_THREADS` | `4` | 1–10 | Browser worker processes. Higher = more throughput, more CPU/RAM. |
+| `PI_RESEARCH_WORKER_CONCURRENCY` | `2` | 1–10 | Tasks per worker process. |
+| `PI_RESEARCH_MODEL` | _(session model)_ | — | Model override for researcher sub-agents (deep and quick) and knowledge synthesis. The coordinator and evaluator keep using the session model. Accepts `provider/id` or a bare model id. |
+| `PI_RESEARCH_DISABLED_TOOLS` | _(none)_ | — | Comma-separated research tools to disable for a run (`search`, `scrape`, `security_search`, `stackexchange`, `youtube_transcript`, `grep`). Removed from every researcher's toolset and named in the coordinator/evaluator prompt. |
+| `PI_RESEARCH_REPORT_EXPORT_ENABLED` (TUI) | `false` | — | Front-ends write a Markdown report to disk and surface its path. |
+| `PI_RESEARCH_REPORT_EXPORT_DIR` | _(smart cwd)_ | — | Pin exported reports to a fixed directory, bypassing the cwd-relative resolution. Useful for the agent skill, which runs from the host agent's arbitrary directory. |
+| `PI_RESEARCH_MAX_SCRAPE_TOKEN_FRACTION_FOR_SCRAPING` | `0.15` | 0.05–1.0 | Max fraction of the context window used for initial scrape context. |
+| `PI_RESEARCH_AVG_TOKENS_PER_SCRAPE` | `2500` | 500–10000 | Estimated tokens per scrape result, used for planning. |
+
+YouTube transcripts
+
+| Variable | Default | Range | Description |
+|----------|---------|-------|-------------|
+| `PI_RESEARCH_YOUTUBE_TRANSCRIPT_MAX_VIDEOS` | `3` | 1–5 | Videos transcribed per `youtube_transcript` call. |
+| `PI_RESEARCH_YOUTUBE_TRANSCRIPT_TIMEOUT_MS` | `20000` | 5000–120000 | Per-video transcript timeout. |
+| `PI_RESEARCH_YOUTUBE_TRANSCRIPT_LANG` | `en` | — | Preferred caption language (BCP-47 prefix). |
+| `PI_RESEARCH_YOUTUBE_QUERY_EVERY_N` | `5` | 1–100 | Append `youtube` to roughly one-in-N search queries (1 = every query). |
+| `PI_RESEARCH_YOUTUBE_POTOKEN_REQUEST_KEY` | _(built-in)_ | — | Advanced: override the BotGuard PoToken web request key (only if YouTube rotates the public key and transcripts start failing). |
+
+Timeouts
+
+| Variable | Default | Range | Description |
+|----------|---------|-------|-------------|
+| `PI_RESEARCH_LLM_TIMEOUT_MS` | `300000` | 60000–600000 | Coordinator / evaluator / repair / knowledge LLM call timeout. |
+| `PI_RESEARCH_SCRAPE_TIMEOUT_MS` | `15000` | 5000–120000 | Per-page scrape (page-load) timeout. |
+| `PI_RESEARCH_SEARCH_TIMEOUT_MS` | `45000` | 5000–120000 | Browser search page timeout. |
+| `PI_RESEARCH_BROWSER_TASK_TIMEOUT_MS` | `10000` | 2000–120000 | Queue-wait / overhead margin added to each browser op's own timeout (a search task ceiling is `SEARCH_TIMEOUT_MS` + this; a scrape is `SCRAPE_TIMEOUT_MS` + this). |
+| `PI_RESEARCH_HEALTH_CHECK_TIMEOUT_MS` | `10000` | 2000–120000 | Pre-flight health-check timeout. |
+
+LLM output & reasoning
+
+These are advanced, env-only knobs (not in the TUI).
+
+| Variable | Default | Range | Description |
+|----------|---------|-------|-------------|
+| `PI_RESEARCH_LLM_THINKING_LEVEL` | `off` | off · minimal · low · medium · high | Chain-of-thought level for all engine LLM work (coordinator, evaluator, synthesis, JSON-repair, knowledge extraction, and researcher sub-agents). Off by default — these calls emit structured JSON / cited reports, so a thinking block only consumes the output budget and can truncate the answer. Clamped per model by pi. |
+| `PI_RESEARCH_PLANNING_MAX_TOKENS` | `16384` | 1024–131072 | Max output tokens for the plan + mid-round evaluator decision. Clamped to the model's real ceiling. |
+| `PI_RESEARCH_SYNTHESIS_MAX_TOKENS` | `32768` | 1024–131072 | Max output tokens for the final synthesized report. Clamped to the model's real ceiling. |
+
+Knowledge store
+
+See the [knowledge store doc](KNOWLEDGE-STORE.md) for what each value does.
+
+| Variable | Default | Range | Description |
+|----------|---------|-------|-------------|
+| `PI_RESEARCH_KNOWLEDGE_STORE_MODE` (TUI) `[project]` | `global` | none · project · global | Store scope: one shared store across every directory (`global`), scoped to the current directory (`project`), or disabled (`none`). |
+| `PI_RESEARCH_EMBEDDING_MODEL` (TUI) | `onnx-community/granite-embedding-small-english-r2-ONNX` | — | Embedding model. Changing it clears the store and starts fresh. |
+| `PI_RESEARCH_EMBEDDING_DEVICE` (TUI) | `auto` | auto · webgpu · cpu | Inference backend. `auto` probes WebGPU viability out-of-process and falls back to CPU; `cpu` forces CPU; `webgpu` forces the GPU path with no probe (advanced — can hard-crash on a software GPU). The TUI exposes only `auto` (as "GPU") and `cpu`. |
+| `PI_RESEARCH_CACHE_TTL_DAYS` (TUI) | `30` | 1–365 | How long cached findings are kept before eviction. |
+| `PI_RESEARCH_KNOWLEDGE_STORE_MAX_SERVE_AGE_DAYS` | `0` | 0–3650 | Max age a cached scrape may be *served* at read time before it's treated as a miss and re-scraped fresh. `0` = disabled (serve at any age up to the TTL). Cache age is always surfaced to the model regardless. |
+| `PI_RESEARCH_MIGRATION_STRATEGY` | `backup` | drop · backup · re-embed | What to do with stored data when the embedding model changes. |
+| `PI_RESEARCH_KNOWLEDGE_DIR` | _(auto)_ | — | Override the knowledge-store database directory. Default: `~/.pi/research/knowledge_db`. |
+| `PI_RESEARCH_EMBEDDING_MODEL_INIT_TIMEOUT_MS` | `300000` | 10000–600000 | Embedding-model initialization timeout (first-time download can be slow). |
+| `PI_RESEARCH_WEBGPU_REPROBE` | _(unset)_ | — | Set `1` to discard the cached WebGPU-viability verdict and probe again on next use. |
+
+API keys (all optional)
+
+| Variable | Description |
+|----------|-------------|
+| `PI_RESEARCH_API_KEY` / `PI_RESEARCH_PROVIDER` | Explicit LLM credentials for SDK / CLI mode (not needed when using pi's own auth). Provider is required alongside the key. |
+| `STACKEXCHANGE_API_KEY` | Raises the Stack Exchange tool's limit from 300/day to 10,000/day. Obtain at <https://stackapps.com/apps/oauth>. |
+| `GITHUB_TOKEN` | Raises the security tool's GitHub Advisory limit from 60/hr to 5000/hr (any default-scope token). |
+| `NVD_API_KEY` | Raises the security tool's NVD limit ~10× and tightens request spacing. Request at <https://nvd.nist.gov/developers/request-an-api-key>. Recommended when using severity-filtered security searches: those issue a second (CVSS v2) NVD query to catch v2-only CVEs, which roughly doubles request time against the 6 s/request unauthenticated throttle. |
+
+Diagnostics & platform
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `PI_RESEARCH_DEBUG` (TUI) | `false` | Verbose INFO+DEBUG logging to the log file. |
+| `PI_RESEARCH_CONSOLE_LOG` | `false` | Mirror logs to stdout/stderr (useful in CI / headless). |
+| `PI_RESEARCH_LOG_PATH` | _(OS temp)_ | Override the verbose log file path. Browser workers inherit it automatically. |
+| `PI_RESEARCH_LOG_FILE` | _(unset)_ | Send browser worker-thread logs to a separate file. If unset, workers log to `PI_RESEARCH_LOG_PATH`. |
+| `PI_RESEARCH_TMP_DIR` | `~/.cache/pi-research/profiles` | Transient per-worker browser-profile directory. Disk-backed by default (kept off a RAM-backed `/tmp` so per-worker profiles don't add memory pressure). Point under the system temp dir to opt into tmpfs/RAM. |
+| `PI_RESEARCH_STATE_DIR` | `~/.pi/research/state` | Override the state directory (active sessions, browser status, project registry). |
+| `PI_RESEARCH_TUI_REFRESH_DEBOUNCE_MS` | `100` | TUI refresh debounce (0–1000 ms). |
+| `PI_RESEARCH_SKIP_HEALTHCHECK` | _(unset)_ | Set `1`/`true` to skip the pre-flight browser/embedding health check and rely on per-task timeouts. |
+| `PI_RESEARCH_USE_XVFB` | _(unset)_ | Linux only. Bare-TTY runs are true-headless and need no X server; set `true` to opt into a virtual framebuffer (`sudo apt install xvfb`). |
+| `PI_RESEARCH_SKILL_DIR` | _(auto)_ | Override the bundled research-skill source directory used by the skill installer. |
+| `PI_RESEARCH_PURGE_BROWSERS` | _(unset)_ | Set `1` during `npm uninstall` to also delete the shared camoufox browser cache (kept by default because other installs may use it). |
+| `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD` | _(unset)_ | Set `1` during `npm install` to skip the camoufox browser download (fetched lazily on first use instead; standard Playwright convention). |
+| `PLAYWRIGHT_INSTALL_DEPS` | _(unset)_ | Linux only. Set `true` during `npm install` to also install system libraries via `npx playwright install-deps` (same as `npm run install:system-deps`). |
+| `PI_RESEARCH_CONFIG_DIR_NAME` | `.pi` | Override the host config-directory name under your home dir (advanced; e.g. set to share another harness's config root). |
+
+Testing only — never enable in production
+
+| Variable | Description |
+|----------|-------------|
+| `PI_RESEARCH_MOCK_SEARCH` | Return fabricated search results instead of real web data. |
+| `PI_RESEARCH_MOCK_SCRAPE` | Return fabricated scrape results instead of real page content. |
+| `PI_RESEARCH_FORCE_READY` | Bypass readiness checks and run even when critical services failed to initialize. |
+| `PI_RESEARCH_ALLOW_LOOPBACK_SCRAPE` | Permit scraping loopback/private/internal addresses, bypassing SSRF protection. |
+
+### How configuration is layered
+
+Configuration resolves from the following layers, lowest to highest precedence
+(later wins):
+
+```
+built-in defaults
+  < ~/.pi/research/config.env                       (base, shared; edited by /research-config)
+  < ~/.pi/research/{pi,cli}.env                      (optional per-front-end overlay)
+  < legacy .pi-research.env in the cwd               (deprecated; auto-migrated to the registry)
+  < project registry                                 (~/.pi/research/state/project-settings.json, per directory)
+  < process.env                                      (real shell env always wins)
+```
+
+Base file. `config.env` holds your shared, user-scoped settings. The
+`/research-config` TUI edits only this file (and the project registry) — never the
+overlays or the merged view — so overlay values are never baked back into the base.
+
+Per-front-end overlays. Each front-end reads only its own optional overlay,
+layered over the shared base, so they can be configured independently. Exactly two
+exist:
+
+- `~/.pi/research/pi.env` — the pi extension
+- `~/.pi/research/cli.env` — the standalone CLI / agent skill (the surface OpenClaw
+  and other skills-aware hosts run)
+
+The overlay files do not exist by default; create the one you need by hand. There is
+intentionally no `sdk.env`: the SDK is a library configured from code (see
+[SDK.md](SDK.md)), not from a global file.
+
+Example — give the standalone CLI / agent skill its own model and depth without
+touching the pi extension:
+
+```sh
+# ~/.pi/research/config.env   (shared baseline)
+PI_RESEARCH_KNOWLEDGE_STORE_MODE=project
+
+# ~/.pi/research/cli.env       (standalone CLI / agent skill only)
+PI_RESEARCH_MODEL=openrouter/anthropic/claude-sonnet-4-6
+PI_RESEARCH_DEFAULT_RESEARCH_DEPTH=2
+```
+
+Project registry. Project-scoped settings (research depth and knowledge-store
+mode) are stored per directory in `project-settings.json`, keyed by normalized
+working-directory path. They override the base and overlays for that directory only.
+In the pi extension the `/research-config` TUI writes them. On the standalone CLI you can
+set the knowledge-store mode per directory directly:
+
+```sh
+pi-research knowledge-config                       # show the mode here and where it comes from
+pi-research knowledge-config set none|project|global
+```
+
+Under the agent skill you don't have to run this yourself — you ask the agent (e.g. "disable the
+knowledge store here" / "make it project-scoped") and it runs the same command on your
+behalf. Either way it lands in the registry (above `config.env` in precedence), so a
+per-directory value overrides a machine-wide `config.env` default; a real environment
+variable still outranks both.
+
+process.env. A real environment variable always wins. For a one-off override,
+just export the variable for that process.
+
+> The base file is not loaded automatically by your shell. Either use the
+> `/research-config` TUI (which writes it), export variables in your shell, or use a
+> loader such as direnv. `.env.example` is a reference, not an active config file.
+
+### Where files live
+
+All pi-research state lives under its own namespace, `~/.pi/research/`:
+
+| Path | Contents |
+|------|----------|
+| `~/.pi/research/config.env` | Shared base configuration (user-scoped settings). |
+| `~/.pi/research/{pi,cli}.env` | Optional per-front-end overlays. |
+| `~/.pi/research/state/project-settings.json` | Project registry (per-directory settings). |
+| `~/.pi/research/state/` | Active sessions, browser status, locks. |
+| `~/.pi/research/knowledge_db/` | The knowledge store (LanceDB), unless `PI_RESEARCH_KNOWLEDGE_DIR` is set. |
+| `~/.cache/pi-research/profiles/` | Transient browser profiles, unless `PI_RESEARCH_TMP_DIR` is set. |
+| `~/.cache/pi-research/webgpu-viability.json` | Cached WebGPU-viability verdict (see the knowledge store doc). |
+
+Paths can be relocated with `PI_RESEARCH_STATE_DIR`, `PI_RESEARCH_KNOWLEDGE_DIR`,
+and `PI_RESEARCH_TMP_DIR`.

package/docs/KNOWLEDGE-STORE.md

@@ -0,0 +1,213 @@
+## Knowledge Store
+
+The knowledge store is a local vector database of past research findings. It is used
+in two distinct ways, and it is an optional cache — research works without it:
+
+- **Knowledge-first answering (advisory).** The `research_knowledge_search` tool
+  answers a repeat or overlapping question directly from stored results. The agent is
+  asked to try it *before* the live `research` tool, but this is guidance the model
+  follows — it is not enforced, and `research_knowledge_search` is a separate tool, not
+  a gate in front of `research`.
+- **Seeding a live run (automatic).** A live `research` run never answers from the
+  store; instead the orchestrator hands each researcher the previously useful URLs for
+  its goal as starting points to re-scrape live.
+
+![Knowledge-store hit — a cached answer returned without a live run](media/03-knowledge-store.gif)
+
+Together these make repeat work faster and cheaper.
+
+### What it stores
+
+The store is a [LanceDB](https://lancedb.com) table on disk. After each research
+round, the cited URLs from the researchers' reports are enqueued and written in the
+background: each source's summary (and, when available, its full scraped Markdown) is
+split into chunks, each chunk is embedded into a vector, and the rows are stored.
+
+Each row carries the embedding vector, the source URL (normalized for
+deduplication), the summary text and full content, a timestamp, and scope flags. A
+content hash dedupes re-ingested URLs: an unchanged page is skipped; a changed page
+replaces the old rows. The full page Markdown is cached once per document so a stored
+finding can be rehydrated later without re-scraping.
+
+Writes never block a research run — they go through an asynchronous writer queue that
+is drained at the end of the round and on shutdown.
+
+### Scopes: none, project, global
+
+The store's reach is set by Knowledge Mode (`PI_RESEARCH_KNOWLEDGE_STORE_MODE`),
+a project-scoped setting you can change per directory:
+
+| Mode | Behavior |
+|------|----------|
+| `global` (default) | One store shared across every directory. A finding cached in one project is retrievable from any other. |
+| `project` | Findings are scoped to the working directory they were created in; only that directory retrieves them. |
+| `none` | The store is disabled — no data is read or written. The `research_knowledge_search` tool stays registered (so re-enabling the mode later needs no pi restart) but is not advertised to the agent: its prompt guidance is stripped, and any direct call returns a "knowledge store is disabled" result. The `/knowledge-store` command is unavailable. |
+
+Change the mode for the current directory with the `/research-config` TUI (Knowledge Mode)
+in the pi extension, or with `pi-research knowledge-config set none|project|global` on the
+standalone CLI (`pi-research knowledge-config` shows the current mode and its source). Under
+the agent skill you don't have to run anything — you ask the agent to change it and it runs that
+command for you. All persist to the per-directory project registry; a per-directory value
+overrides a machine-wide `PI_RESEARCH_KNOWLEDGE_STORE_MODE` in `config.env`, and a real
+environment variable overrides everything. The change applies on the next run — no restart.
+
+All scopes share one physical LanceDB directory; project vs. global rows are
+distinguished by columns (a normalized workspace path and a global flag) and filtered
+at query time, not by separate folders. The default database directory is
+`~/.pi/research/knowledge_db/` (override with `PI_RESEARCH_KNOWLEDGE_DIR`).
+
+The embedding model is lazy — it only downloads and initializes the first time
+the store is actually written or searched, so the `global` default adds no startup
+cost until a run caches its first page.
+
+### How a run uses the store
+
+The store is driven by the orchestrator, not called ad hoc by researcher agents,
+which keeps its use deterministic:
+
+1. Before each researcher starts, the orchestrator searches the store for the
+   researcher's goal and injects any matching historical URLs — each with its prior
+   summary — into that researcher's prompt as suggested starting points to re-scrape.
+2. After the round, the cited URLs and their descriptions are enqueued into the
+   writer queue for the next session.
+
+Separately, the `research_knowledge_search` tool (and the SDK's `searchKnowledge()`)
+lets the model query the store directly: it rehydrates the most relevant stored
+documents, asks a background LLM whether they answer the question, and returns a
+synthesized answer with citations — or reports that live research is needed.
+
+### Embeddings and the model
+
+Embeddings are computed locally with
+[`@huggingface/transformers`](https://github.com/huggingface/transformers.js) over
+ONNX. The default model is
+`onnx-community/granite-embedding-small-english-r2-ONNX` (English; a 512-token
+chunk window). Each supported model defines its own chunk size, pooling strategy,
+and prefixes, and produces a fixed vector dimension that the table's schema is built
+around.
+
+Supported models (`PI_RESEARCH_EMBEDDING_MODEL`):
+
+| Model | Languages |
+|-------|-----------|
+| `onnx-community/granite-embedding-small-english-r2-ONNX` (default) | English |
+| `Xenova/multilingual-e5-small` | Multilingual |
+| `Xenova/multilingual-e5-base` | Multilingual |
+| `Xenova/bge-m3` | Multilingual |
+| `onnx-community/embeddinggemma-300m-ONNX` | Multilingual |
+| `onnx-community/Qwen3-Embedding-0.6B-ONNX` | Multilingual |
+| `Xenova/all-MiniLM-L6-v2` | English |
+| `Xenova/bge-small-en-v1.5` | English |
+| `Xenova/all-mpnet-base-v2` | English |
+
+Changing the model invalidates existing vectors (they have a different dimension and
+meaning), so the store is migrated (see Changing the model, below).
+The model is downloaded from Hugging Face on first use and cached; the first download
+can take a few minutes (raise `PI_RESEARCH_EMBEDDING_MODEL_INIT_TIMEOUT_MS` on a slow
+connection).
+
+### Device selection
+
+Embeddings run on either the GPU (WebGPU, via the runtime's bundled Dawn backend) or
+the CPU. The backend is chosen by `PI_RESEARCH_EMBEDDING_DEVICE`:
+
+- `auto` (default; shown as GPU in the TUI) — pi-research probes WebGPU
+  viability in a disposable child process: it loads the model and runs one real
+  embedding there. If that succeeds, the GPU is used; if it fails, the CPU is used.
+  The verdict is cached, so the probe runs at most once per machine + model.
+- `cpu` (shown as CPU in the TUI) — forces CPU inference, no probe.
+- `webgpu` — forces the GPU path with no probe. Advanced / env-only; see below.
+
+Why the probe exists. Some hosts — VMs, containers, CI runners, headless machines
+with a software Vulkan driver — expose a GPU the native backend cannot run compute
+on. That failure is a native segfault, not a catchable error, so it terminates the
+process. The `auto` probe tests viability in a child process (whose crash cannot
+affect the main process) and falls back to CPU. Forcing `webgpu` skips this check and
+can crash on such a host, so the `/research-config` menu offers only GPU (= `auto`)
+and CPU. Raw `webgpu` stays available through the environment variable for
+benchmarking on a host with a known-good GPU.
+
+The cached verdict lives at `~/.cache/pi-research/webgpu-viability.json`, keyed by
+platform, architecture, Node major version, and model. Set
+`PI_RESEARCH_WEBGPU_REPROBE=1` to discard it and probe again (for example after a
+driver upgrade).
+
+### Platform support (no Intel Mac)
+
+The store depends on two native components — the ONNX runtime for embeddings and
+LanceDB for vector storage — that ship prebuilt binaries only for certain
+platform/architecture pairs:
+
+| Platform | Architecture | Knowledge store |
+|----------|--------------|-----------------|
+| macOS | Apple Silicon (arm64) | Supported |
+| macOS | Intel (x64) | Not available |
+| Linux | x64 / arm64 | Supported |
+| Windows | x64 / arm64 | Supported |
+
+Intel Macs (`darwin-x64`) have no prebuilt binary for either component, so the
+knowledge store cannot run there. The degradation is automatic:
+
+- Research still works. Search, scraping, YouTube transcripts, the security databases,
+  Stack Exchange, planning, and synthesis are unaffected — only the store is missing.
+- The health check reports the store as disabled, not unhealthy: its health component
+  returns disabled (native embedding/vector stack unavailable on this platform) with a
+  healthy status, so the missing component does not drag overall health to "unhealthy"
+  or block a quick (depth-0) run.
+- `research_knowledge_search` does no caching or retrieval; every run goes to the live web.
+
+No configuration is needed on an Intel Mac.
+
+### Retention and eviction
+
+Cached findings are kept for `PI_RESEARCH_CACHE_TTL_DAYS` (default 30; range 1–365).
+Eviction is checked when the store opens and removes only rows older than the cutoff
+within the current scope. Lower the value for fresher data and less disk; raise it to
+keep history longer.
+
+### Changing the model: migration
+
+When the configured embedding model differs from the one the stored vectors were
+built with, the store is migrated according to `PI_RESEARCH_MIGRATION_STRATEGY`:
+
+| Strategy | What happens |
+|----------|--------------|
+| `backup` (default) | The old table is renamed aside (`knowledge_backup_<timestamp>.lance`) and a fresh table is created for the new model. Old data is preserved on disk but not searched. |
+| `drop` | The old table is discarded and a fresh one created. Fast; no backup. |
+| `re-embed` | Every stored document is re-embedded with the new model into a new table, preserving history. Slowest. |
+
+If the chosen strategy fails, pi-research falls back to `backup`, then to `drop`,
+rather than leaving the store in a broken state. Changing the model from the
+`/research-config` menu always clears the current store and starts fresh.
+
+### Managing the store
+
+From `/research-config`:
+
+- Store Status — entry counts (project and user), the active embedding model
+  and device, and the on-disk path.
+- Clear Project Store / Clear User Store — permanently delete the
+  project-scoped or global rows (shown according to the current mode).
+- Run Health Check — exercises the browser pool, GPU/embedding, and knowledge
+  store connectivity, and reports the store's health state.
+
+The store grows copy-on-write (each run appends a version), so it is compacted
+automatically after any run that changed the stored data — stale versions and
+indices are pruned to keep it bounded. There is no manual maintenance command.
+
+### Settings
+
+| Setting | Variable | Default |
+|---------|----------|---------|
+| Knowledge Mode (project-scoped) | `PI_RESEARCH_KNOWLEDGE_STORE_MODE` | `global` |
+| Embedding model | `PI_RESEARCH_EMBEDDING_MODEL` | `onnx-community/granite-embedding-small-english-r2-ONNX` |
+| Embedding device | `PI_RESEARCH_EMBEDDING_DEVICE` | `auto` |
+| Cache retention (days) | `PI_RESEARCH_CACHE_TTL_DAYS` | `30` |
+| Migration strategy | `PI_RESEARCH_MIGRATION_STRATEGY` | `backup` |
+| Database directory | `PI_RESEARCH_KNOWLEDGE_DIR` | `~/.pi/research/knowledge_db` |
+| Model init timeout (ms) | `PI_RESEARCH_EMBEDDING_MODEL_INIT_TIMEOUT_MS` | `300000` |
+| Re-probe WebGPU | `PI_RESEARCH_WEBGPU_REPROBE` | _(unset)_ |
+
+See [CONFIGURATION.md](CONFIGURATION.md) for the full configuration model, and
+[ARCHITECTURE.md](ARCHITECTURE.md) for how the store fits into the
+engine.

package/docs/PI-EXTENSION.md

@@ -0,0 +1,91 @@
+## Pi Extension
+
+pi-research integrates as a [pi](https://github.com/earendil-works/pi)
+extension (`src/index.ts`) — a multi-agent web research engine with a real-time
+TUI, registered directly in the pi process.
+
+### Usage
+
+The `research` tool is auto-registered, so the model invokes it
+from natural language, and the tool understands the depth needed (1–3) from the query.
+
+```bash
+pi -p "research the latest developments in WebAssembly"
+pi -p "do a thorough deep-dive on the AI inference hardware landscape"
+```
+
+Three slash commands are also registered:
+
+| Command | Description |
+|---------|-------------|
+| `/research <query>` | Invokes the `research` tool directly at the configured default depth (`PI_RESEARCH_DEFAULT_RESEARCH_DEPTH`, 1 by default) — a plain live run with no LLM turn. It does not parse an inline depth and does **not** consult the knowledge store; use `/knowledge-store <query>` for a store-only lookup. |
+| `/research-config` | Opens the interactive TUI settings dashboard. |
+| `/knowledge-store <query>` | Searches the local knowledge store for a query and returns a synthesised answer from previously researched findings. Unavailable when Knowledge Mode is `none`. The store auto-manages its own compaction, so there is no maintenance subcommand. |
+
+![Running a live investigation with the /research slash command](media/01-slash-research.gif)
+
+### Tools
+
+The extension registers three tools:
+
+| Tool | Registered |
+|------|-----------|
+| `research` | always |
+| `health` | always |
+| `research_knowledge_search` | always (see note) |
+
+`research_knowledge_search` is registered unconditionally so a Knowledge Mode
+change takes effect without restarting pi (pi has no unregister API). When
+`PI_RESEARCH_KNOWLEDGE_STORE_MODE` is `none` the tool is not advertised to the
+agent — its prompt guidance is stripped, and any call returns a "store disabled"
+result; the `/knowledge-store` command is likewise unavailable. Advertisement and
+the store's read/write paths are gated on the live mode, not on this registration.
+
+Tool exclusion — the `research` tool honors an `excludeTools` list taken from
+the pi session context when the host forwards one.
+
+### TUI
+
+During a run pi-research renders a live progress panel:
+
+- Researcher slices — one per agent: status, URLs scraped, actions taken.
+- Wave animation — active-crawl indicator.
+- Token usage — model tokens + estimated cost (non-decreasing guard).
+- Status flashes — green on success, red on failure.
+- Steering messages — queued and active mid-run user guidance.
+
+| Key | Action |
+|-----|--------|
+| `Escape` | Cancel active research |
+| Arrow keys | Navigate the `/research-config` menu |
+| `Enter` / `Space` | Cycle a setting's value |
+
+### Configuration
+
+Manage settings through `/research-config`, which edits two layers:
+
+- Global — base `~/.pi/research/config.env` (applies to all front-ends).
+- Project — the centralized registry (`~/.pi/research/state/project-settings.json`),
+  scoped per working directory. Only depth and knowledge-store mode are
+  project-scoped, so a given repo can carry its own research depth without
+  changing your global default.
+
+To configure the pi extension independently of the other front-ends, add an
+optional overlay at `~/.pi/research/pi.env` (it layers over `config.env` for the
+pi extension only). The full configuration model, precedence, and the complete
+environment-variable list live in [CONFIGURATION.md](CONFIGURATION.md).
+
+### Coding-agent skill installer
+
+The `/research-config` menu can install the `pi-research` skill into your other
+coding agents (Claude, Codex, OpenClaw) so they can run web research through the
+CLI, and remove it again — with exact, manifest-tracked cleanup. See
+[AGENT-SKILL.md](AGENT-SKILL.md) for the full installation flow.
+
+### Lifecycle
+
+- `activate` — registers commands, tools, the TUI controller, and initializes services.
+- `deactivate` — drains the writer queue, closes LanceDB, terminates the browser pool, disposes the embedding model.
+- `session_shutdown` — branches on `event.reason`: a `quit` triggers process-exit cleanup; reload / new / resume / fork clean up without exiting.
+
+Extension state is isolated per pi session, so `/reload` is safe.