@lincoln504/pi-research 1.3.9 → 1.3.11

package/docs/CHANGELOG.md

@@ -5,6 +5,70 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.11] - 2026-08-14
+
+Alongside the revert below, a four-way independent audit (docs-vs-code
+consistency, orchestration/TUI/citations, the knowledge store, and CLI/config
+precedence) swept the areas least recently re-checked, and an independent
+fresh-eyes review of the resulting delta caught two follow-on defects (fixed
+below). Regression tests cover the highest-severity fixes — the `--config`
+credential drop (mutation-verified), the revalidation touch, the tool-name and
+initial-links validation, the exit-78 message split, and the citation marker —
+plus live end-to-end verification of the `--config` and validation behavior;
+the remaining fixes are covered by the existing suites they run under.
+
+### Fixed
+
+- **`--config` silently dropped credentials stored in the base `config.env`.** With an explicit `--config <path>`, the env bridge layered only `cli.env` + the named file — the global base was never read. Typed config keys survived (getConfig reads the base independently), but the env-only keys the bridge exists for — `PI_RESEARCH_API_KEY`, `PI_RESEARCH_PROVIDER`, and the security-API keys — vanished, so `research "q" --config ci.env` exited 78 ("no model or API key") on a correctly configured machine, contradicting the flag's own help text. The global base now always layers underneath (base → cli.env → named file). Reproduced live before the fix; regression test fails against the pre-fix code.
+- **Two sessions configured with different embedding models could permanently destroy knowledge-store data.** Every session opens the same physical table, and a model mismatch migrates it aside as a backup — so two sessions with different `PI_RESEARCH_EMBEDDING_MODEL` values alternate migrations, and the backup pruner's keep-only-newest rule deleted the only surviving copy of the first model's data after just two alternations, silently. Superseded backups are now retained until they age past `CACHE_TTL_DAYS` (a backup older than the TTL holds only rows eviction would have deleted anyway), so alternation can no longer destroy data inside the retention window.
+- **A stable page became permanently "stale" once `KNOWLEDGE_STORE_MAX_SERVE_AGE_DAYS` was set.** The over-age read is treated as a miss and re-scraped, but the re-ingest's content-hash dedup skipped the write without refreshing the timestamp — so the entry could never satisfy the serve-age gate again (a live scrape on every run, for exactly the content that never changes) and TTL eviction eventually deleted an entry re-verified byte-identical possibly hours earlier. The unchanged-content skip now touches the current generation's timestamp — guarded by a just-in-time newest-generation re-check so a concurrent process's newer ingest can't be outranked by the touch.
+- **An interrupted `knowledge` search could report success.** `cmdKnowledge` wired the abort signal but never consulted the cancellation latch on the resolve path — its own comment named "resolve normally and report ok:true/exit 0 for an interrupted run" as the failure being prevented, and only the throwing half was handled. It now mirrors `cmdResearch`: exit 128+N and a `cancelled: true` JSON field.
+- **Cancelled `--json` research output nondeterministically lost its diagnostics.** The signal handler's fire-and-forget shutdown resets the SDK globals that `researchers`/`trackedErrors` are read from; reading them at JSON-emit time raced it. They are now snapshotted as soon as the run resolves, which shrinks the race window to near zero (a shutdown that completes before the run's continuation is even scheduled can still win — strictly narrower than before).
+- **Fresh-store init could burn all 5 retries against a never-initialized embedder.** The model-mismatch local fallback embedder is constructed uninitialized, and only the migration branch warmed it — the no-existing-table branch went straight to `createTable()`, which throws "embedder dimension unknown", failing every retry identically. The fresh-table branch now warms the embedder the same way.
+- **Headless pi-extension runs leaked per-run session state for the process lifetime.** The non-TUI cleanup path cleared steering messages but never called `endResearchSession`, so every run left its failure-count entries and AbortController in the module-level session map (the TUI and SDK paths both free them).
+- **A run that failed while queued for a run slot fired no terminal observer event.** Capacity exhaustion or cancel-while-queued rethrew before any orchestrator existed, so an observer that had just been told `run_queued` never received `onError` — violating the "exactly one of onComplete/onError" contract. The terminal callback now fires on both paths.
+- **The rebuilt report carried a stray `##`/`**`/`>` line when the model wrote a decorated CITED LINKS header.** The header detector deliberately tolerates markdown prefixes, but both body-slicing consumers cut at the header *text*, leaving the tolerated marker dangling at the end of the body. Both now slice at the start of the header line.
+- **`--exclude-tools` accepted unknown tool names silently** (a typo excluded nothing for the whole billed run — the repository's own test fixture had exactly this bug, excluding a tool named `security` that does not exist); names are now validated against the canonical set. The fresh-eyes review then caught the validator omitting `read` — excluding researchers' local-file reads is a legitimate hardening move that had always worked and would have started failing on the CLI; `read` is in the valid set. **A bare `--initial-links` with zero URLs** likewise proceeded silently and now fails fast like every other value-taking flag.
+- **The exit-78 message conflated "nothing configured" with "model configured, key missing".** SKILL.md tells agents to relay the message verbatim, so users who had set `PI_RESEARCH_MODEL` were pointed at the wrong knob. The two cases now produce distinct messages.
+- **`PI_RESEARCH_*` boolean/enum env vars silently mis-parsed with surrounding whitespace** (`'true '` read as false); both parsers now trim like the numeric parser always has. **On a model-load timeout, the still-loading ONNX pipeline now gets disposed** when it eventually resolves instead of leaking one native session per retry.
+
+### Documentation
+
+- `.env.example` corrections: `PI_RESEARCH_ALLOW_LOOPBACK_SCRAPE` described a wider hole than the loopback-only scope it actually has (the same overstatement 1.3.9 fixed in CONFIGURATION.md); `PI_RESEARCH_LLM_TIMEOUT_MS` still showed the old 10-minute ceiling (raised to 30 in the schema); the "exhaustive reference" claim is honored again by documenting `PLAYWRIGHT_BROWSERS_PATH`, `XDG_CACHE_HOME`, and the skill launcher's `PI_RESEARCH_BIN`/`PI_RESEARCH_PATH`.
+- ARCHITECTURE.md's "Key design decisions" still said researchers "can read and grep files" — stale since grep became default-excluded, and wrong outright since 1.3.10 made the exclusion unconditional. Rewritten, along with the tool table's "off by default" wording, the extension event list, and the fixed "≤3 videos" phrasing (configurable 1–5). README's "read and search files" corrected to read-only-files. CONFIGURATION.md now states that provider-native key variables are honored only as real environment variables, and the CLI help finally mentions `--version`/`-v` (implemented all along).
+
+### Removed
+
+- **The 1.3.10 researcher `max_tokens` cap (`PI_RESEARCH_RESEARCHER_MAX_TOKENS`) is reverted.** It solved a non-problem and created a real risk. The 402 it "fixed" only fires when an OpenRouter balance is nearly exhausted, and that error already names the remedy — it is the provider's top-up prompt, not a client defect. Meanwhile the cap re-introduced the exact failure class `PLANNING_MAX_TOKENS` was raised to escape in 1.2.x: reasoning tokens count against `max_tokens` on OpenAI-compatible APIs, so with `PI_RESEARCH_LLM_THINKING_LEVEL` raised — or a model that cannot fully disable thinking — a 16k ceiling can be consumed before or during the report and truncate it. It also second-guessed the model catalog: pi sessions everywhere else honor the catalog `maxTokens`, and a user who wants a different ceiling can already set one per model via `modelOverrides` in `~/.pi/agent/models.json`, which covers every session this package creates. Researcher sessions once again inherit the model's catalog ceiling unmodified.
+
+## [1.3.10] - 2026-08-14
+
+The post-release verification round for 1.3.9 kept digging and found that the
+fetch scrape path — the fast primary transport in front of the stealth browser —
+had been dead in production on every supported Node version, hidden completely by
+the browser fallback. Three more researcher-session defects surfaced from the same
+investigation, two of them demonstrated live against a real provider. Each fix
+carries a regression test verified to fail against the pre-fix code (the
+accounting fix pins the deep-researcher site; the quick site shares the same
+helper call), and the fetch fix carries the first test of that layer that uses a
+real socket instead of a mocked `fetch`. An independent fresh-eyes review of the
+whole delta then caught two follow-on defects, fixed below.
+
+### Fixed
+
+- **Every fetch scrape failed instantly; the whole path silently ran on the browser fallback.** `getSsrfSafeDispatcher()` built an `Agent` from the undici 8.x this package depends on and handed it to Node's **global** `fetch()` — which is backed by the undici compiled *into* the runtime (6.x on Node 22, 7.x on Node 25). Cross-major dispatchers are rejected outright (`UND_ERR_INVALID_ARG: invalid onRequestStart method`), so every fetch attempt threw before a single packet left the machine, burned one retry, and fell back to the stealth browser: ~3.4 s per page against the ~0.9 s the fetch path delivers, the browser pool saturated with work it never needed — and the connect-time SSRF DNS pin, the entire reason the dispatcher exists, never engaged once (request-time validation still applied throughout, so this was defence-in-depth loss, not an open hole). Nothing caught it because every test of the layer stubs the global `fetch`, and the fallback kept scrapes succeeding. The replacement, `getSsrfSafeFetcher()`, returns the Agent *paired with undici's own `fetch`*, making the invalid combination unrepresentable, and a new test drives the real fetch path against a real loopback socket with the browser layer disabled — it fails against the pre-fix code with the exact production error. Measured on live runs: 0 fetch / 5 browser successes before, 4 fetch / 0 browser after.
+- **Researchers sent the model's full ~230k catalog ceiling as `max_tokens` — and OpenRouter 402'd every call on a low balance.** The researcher session inherited the model's catalog `maxTokens` as its per-request output cap. Providers that budget-check `max_tokens` up front rejected the call before running it: observed live as `402: You requested up to 231037 tokens, but can only afford 24467` on every researcher, failing entire runs whose real cost was cents. Researcher output per call — tool calls plus a cited report — is nowhere near that. New `PI_RESEARCH_RESEARCHER_MAX_TOKENS` knob (default 16384, range 1024–131072), clamped to the model's real ceiling, applied in the shared session factory so deep and quick sub-agents are both covered. *(Reverted in 1.3.11 — see above.)*
+- **`--exclude-tools` silently granted researchers local-filesystem `grep`.** A non-empty exclusion list *replaced* the default exclusions instead of adding to them, so `--exclude-tools stackexchange` enabled a capability as a side effect of a flag documented as "disable internal tools". The same defect was fixed for `PI_RESEARCH_DISABLED_TOOLS` in 1.3.0 with the rationale "a subtract-only setting must never grant a capability"; the caller-facing surface kept the replacement semantics until now. Both surfaces are strictly additive over the default exclusions. (Behavioral change: there is no longer any way to hand researchers `grep`, which ARCHITECTURE.md previously documented as a quirk of the replacement rule.)
+- **A misconfigured $0 price table was silent on depth-0 runs.** Both researcher `message_end` accounting sites hand-copied the extract→increment→emit block instead of calling `recordLlmUsage`, so `warnIfUnpriced` — added in 1.1.2 precisely because a hand-written `models.json` entry silently discards pi's pricing — never ran for researcher usage. Depth-1+ runs still warned via the coordinator; depth-0 runs have no coordinator and reported `$0.0000` with no diagnostic at all. Metric labels and observer payloads are unchanged (`onTokensConsumed` now fires before `onResearcherProgress` rather than after — no consumer is order-sensitive), and the deep path keeps its `safeObserve` isolation.
+- **A connect-time SSRF block was filed as a generic fetch error.** undici never surfaces the connector's `ESSRFBLOCKED` error directly — `fetch` throws `TypeError: fetch failed` with the real error down the `cause` chain — so the message-based classification at both tracking sites missed it and a DNS-rebinding block would have been invisible in the SSRF metrics. Unreachable before this release (the dispatcher path had never executed); now that it is live, `isSsrfBlockError()` walks the bounded cause chain. Diagnostics-only: the block itself always held.
+- **One unit test silently made live network requests.** The fetch-stub migration for the undici change covered two of the three test files that drive the scrape layer; the missed one (`scraper-pdf-native-module-failure`) kept stubbing the *global* fetch, so every run of that "hermetic" test performed real DNS and an outbound TLS connection to a third-party host — and passed only because that host's certificate mismatch happened to reproduce the stubbed failure. Found by the fresh-eyes review; it now shims `getSsrfSafeFetcher` like its siblings.
+
+### Verified
+
+- Full suites green after the changes: 191 unit files / 2457 tests, integration 53+8 skipped and 103, lint, both type-checks, dependency rules.
+- Live production runs on OpenRouter at depths 0, 1, and 2: reports with citations, fetch-path scrapes at ~0.6–1.5 s/page with zero browser fallbacks and zero undici errors, non-zero cost accounting, knowledge store round-trip (same-day scrapes retrieved and grounded).
+- The stderr FD-2 capture deliberately still skips pipes/sockets: when a parent process is reading stderr (pi host, MCP, CI), stealing FD 2 would hide fatal diagnostics to suppress cosmetic Dawn noise. Reviewed and kept as designed.
+
 ## [1.3.9] - 2026-08-14
 
 A whole-tree audit sweep. Six defects, each found by looking at what the code

package/docs/CONFIGURATION.md

@@ -69,7 +69,7 @@ Research
 | `PI_RESEARCH_MAX_RESEARCHERS` (TUI) | `3` | 1–5 | Parallel researchers. |
 | `PI_RESEARCH_DEFAULT_RESEARCH_DEPTH` (TUI) `[project]` | `1` | 1–3 | Depth for `/research` and the CLI when `--depth` is omitted (1=normal, 2=deep, 3=ultra). |
 | `PI_RESEARCH_MAX_SCRAPE_BATCHES` (TUI) | `2` | 0–99 | Scrape batches per researcher (0 = unlimited). |
-| `PI_RESEARCH_MAX_GATHERING_CALLS` | `12` | 1–100 | Shared web-gathering calls per researcher (`search` + `security_search` + `stackexchange` + `youtube_transcript`). Local `grep` has its own separate budget. |
+| `PI_RESEARCH_MAX_GATHERING_CALLS` | `12` | 1–100 | Shared web-gathering calls per researcher (`search` + `security_search` + `stackexchange` + `youtube_transcript`). |
 | `PI_RESEARCH_MAX_CONCURRENT_SCRAPES` | `3` | 1–20 | Concurrent URLs fetched per scrape batch. |
 | `PI_RESEARCH_MAX_RETRIES` | `2` | 0–5 | Retries per researcher request. |
 | `PI_RESEARCH_RETRY_DELAY_MS` | `2000` | 100–10000 | Base delay between retries. |
@@ -135,7 +135,7 @@ API keys
 
 | Variable | Description |
 |----------|-------------|
-| `PI_RESEARCH_API_KEY` / `PI_RESEARCH_PROVIDER` | Explicit LLM credentials for SDK / CLI mode (not needed when pi's configuration supplies the key). On the CLI / agent skill both can also live in `config.env` / `cli.env`. The provider is required alongside the key, or inferred from a `provider/model-id` `PI_RESEARCH_MODEL`. |
+| `PI_RESEARCH_API_KEY` / `PI_RESEARCH_PROVIDER` | Explicit LLM credentials for SDK / CLI mode (not needed when pi's configuration supplies the key). On the CLI / agent skill both can also live in `config.env` / `cli.env`. The provider is required alongside the key, or inferred from a `provider/model-id` `PI_RESEARCH_MODEL`. Note: provider-native variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …) are honored only as **real environment variables** — placing one in `config.env` / `cli.env` does nothing (the files bridge only `PI_RESEARCH_*` keys plus `STACKEXCHANGE_API_KEY` / `GITHUB_TOKEN` / `NVD_API_KEY`). |
 | `STACKEXCHANGE_API_KEY` | Raises the Stack Exchange tool's limit from 300/day to 10,000/day. Obtain at <https://stackapps.com/apps/oauth>. |
 | `GITHUB_TOKEN` | Raises the security tool's GitHub Advisory limit from 60/hr to 5000/hr (any default-scope token). |
 | `NVD_API_KEY` | Raises the security tool's NVD limit ~10× and tightens request spacing. Request at <https://nvd.nist.gov/developers/request-an-api-key>. Recommended when using severity-filtered security searches: those issue a second (CVSS v2) NVD query to catch v2-only CVEs, which roughly doubles request time against the 6 s/request unauthenticated throttle. |
@@ -158,6 +158,7 @@ Diagnostics & platform
 | `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD` | _(unset)_ | Set `1` during `npm install` to skip the camoufox browser download (fetched lazily on first use instead; standard Playwright convention). |
 | `PLAYWRIGHT_BROWSERS_PATH` | _(user cache)_ | Standard Playwright variable, honoured here too: overrides where the camoufox binary is stored and looked up, and is passed through to browser workers. |
 | `XDG_CACHE_HOME` | `~/.cache` | Standard XDG variable. When set, every `~/.cache/pi-research/...` path below is rooted at `$XDG_CACHE_HOME/pi-research/...` instead. |
+| `PI_RESEARCH_BIN` (alias `PI_RESEARCH_PATH`) | _(auto)_ | Agent-skill launcher only: explicit path to the pi-research engine binary when auto-resolution (PATH → local install → npx) should be bypassed. See [skills/pi-research/references/configuration.md](../skills/pi-research/references/configuration.md). |
 | `PLAYWRIGHT_INSTALL_DEPS` | _(unset)_ | Linux only. Set `true` during `npm install` to also install system libraries via `npx playwright install-deps` (same as `npm run install:system-deps`). |
 | `PI_RESEARCH_STRICT_SETUP` | _(unset)_ | Read by the bundled `scripts/setup.cjs` during `npm install`. Set `1`/`true` to make a failed browser download fail the install instead of deferring it to first use. |
 | `PI_RESEARCH_CONFIG_DIR_NAME` | `.pi` | Override the host config-directory name under your home dir (advanced; e.g. set to share another harness's config root). |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lincoln504/pi-research",
-  "version": "1.3.9",
+  "version": "1.3.11",
   "description": "Free unlimited web search and deep research for the Pi coding agent. Local knowledge store. Also works from Claude Code, Codex, and OpenClaw.",
   "type": "module",
   "private": false,

package/skills/pi-research/SKILL.md

@@ -2,7 +2,7 @@
 name: pi-research
 description: "Multi-agent web research for any coding agent. Use for anything requiring current information from the internet — news, trends, product/service/company facts, statistics, people, places, events, APIs, library behavior, or 'what is X / how does X work' questions. Also use to search a local knowledge store of previously researched findings before doing live research. Powered by the pi-research engine (search, stealth scraping, security databases, Stack Exchange). Not for reading local files, running project commands, or analyzing this repository's own source."
 license: MIT
-metadata: { "author": "Lincoln Deen", "version": "1.3.9", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
+metadata: { "author": "Lincoln Deen", "version": "1.3.11", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
 allowed-tools: Bash(node:*)
 ---
 

package/src/cli.ts

@@ -44,6 +44,7 @@ import { exportResearchReport, appendExportMessage } from './utils/research-expo
 import { getConfig, getGlobalConfigDir, getGlobalEnvFilePath, getInterfaceEnvFilePath, getStateDir, saveConfig, resetConfig, describeKnowledgeStoreMode, isProjectScopedKey, parseDotEnv } from './config.ts';
 import { getAgentDir } from '@earendil-works/pi-coding-agent';
 import { buildModelRegistry, safeGetAvailable } from './core/llm/model-registry-factory.ts';
+import { RESEARCH_TOOL_NAMES } from './constants.ts';
 
 // ---------------------------------------------------------------------------
 // Constants
@@ -234,7 +235,8 @@ export class ConfigFileError extends Error {}
  * the real environment, so real shell exports always win.
  */
 export function bridgeConfigEnv(explicitConfigPath?: string): { path: string; loaded: boolean } {
-  const basePath = explicitConfigPath ?? getGlobalEnvFilePath();
+  const globalBasePath = getGlobalEnvFilePath();
+  const basePath = explicitConfigPath ?? globalBasePath;
   const cliOverlayPath = getInterfaceEnvFilePath('cli');
   let loaded = false;
   // Merge base then overlay into one map FIRST (overlay wins), and only then
@@ -252,13 +254,18 @@ export function bridgeConfigEnv(explicitConfigPath?: string): { path: string; lo
   // the user passed on this exact invocation is the most specific intent, and the
   // ambient cli.env overlay silently overriding it re-created the very failure
   // this bridge exists to fix (a key set in `--config ci.env` simply not applying).
+  // The GLOBAL base always stays underneath: this bridge exists for env-only keys
+  // (PI_RESEARCH_API_KEY & co.) that getConfig never carries, so dropping the base
+  // here silently dropped credentials stored only in config.env — `--config extra.env`
+  // exit-78'd on a correctly configured machine while the help text promised
+  // "base config.env keys it does not set still apply".
   const layers = explicitConfigPath !== undefined
-    ? [cliOverlayPath, basePath]
-    : [basePath, cliOverlayPath];
+    ? [globalBasePath, cliOverlayPath, explicitConfigPath]
+    : [globalBasePath, cliOverlayPath];
   for (const filePath of layers) {
     // Only the EXPLICIT --config file fails loudly (see ConfigFileError above);
     // the ambient layers keep their missing/unreadable tolerance unchanged.
-    const isExplicit = explicitConfigPath !== undefined && filePath === basePath;
+    const isExplicit = explicitConfigPath !== undefined && filePath === explicitConfigPath;
     try {
       if (!fs.existsSync(filePath)) {
         if (isExplicit) throw new ConfigFileError(`--config file not found: ${filePath}`);
@@ -495,11 +502,20 @@ async function detectCredentials(explicitModel?: string): Promise<CredentialDete
     modelFrom,
     piAuthPresent,
     piModelsPresent,
-    problem:
-      'No model or API key is configured for pi-research. Provide credentials via environment ' +
-      'variables (PI_RESEARCH_API_KEY + PI_RESEARCH_PROVIDER + PI_RESEARCH_MODEL) or, if you use ' +
-      "pi, via pi's configuration (a provider entry in ~/.pi/agent/auth.json, or a per-provider " +
-      'apiKey in ~/.pi/agent/models.json).',
+    // A model configured with no usable key is a different (and misleading to
+    // conflate) gap than nothing configured at all: SKILL.md tells agents to relay
+    // this text verbatim, so "No model … is configured" pointed users who HAD set
+    // PI_RESEARCH_MODEL at the wrong knob.
+    problem: model
+      ? `A model is configured (${model}) but no API key is available for ` +
+        `${provider ? `provider "${provider}"` : 'its provider'}. Provide one via the ` +
+        'PI_RESEARCH_API_KEY environment variable (with PI_RESEARCH_PROVIDER), or, if you use pi, ' +
+        "via pi's configuration (a provider entry in ~/.pi/agent/auth.json, or a per-provider " +
+        'apiKey in ~/.pi/agent/models.json).'
+      : 'No model or API key is configured for pi-research. Provide credentials via environment ' +
+        'variables (PI_RESEARCH_API_KEY + PI_RESEARCH_PROVIDER + PI_RESEARCH_MODEL) or, if you use ' +
+        "pi, via pi's configuration (a provider entry in ~/.pi/agent/auth.json, or a per-provider " +
+        'apiKey in ~/.pi/agent/models.json).',
   };
 }
 
@@ -604,6 +620,14 @@ export async function cmdResearch(args: ResearchArgs): Promise<number> {
       ...(args.initialLinks ? { initialLinks: args.initialLinks } : {}),
     }, abortController.signal);
 
+    // Snapshot the run diagnostics NOW, before any further await: on a cancelled
+    // run, onSignal's fire-and-forget safeShutdown runs concurrently with this
+    // tail and resets these SDK globals — reading them at JSON-emit time made
+    // `researchers`/`trackedErrors` silently vanish from a cancelled-but-
+    // successful payload depending on who won the race.
+    const outcome = getLastResearcherOutcome();
+    const errorReport = getLastErrorReport();
+
     // Optionally persist the report to a file (opt-in via
     // PI_RESEARCH_REPORT_EXPORT_ENABLED). The SDK is a pure library and does not
     // write files; the CLI front-end does, mirroring the pi tool path.
@@ -636,8 +660,7 @@ export async function cmdResearch(args: ResearchArgs): Promise<number> {
       // (ISSUES-ANALYSIS-2026-07-09.md, Symptom 4): a caller reading only stdout/JSON
       // had no way to distinguish "thin report, little material out there" from "thin
       // report because most researchers died mid-run" without opening the log file.
-      const outcome = getLastResearcherOutcome();
-      const errorReport = getLastErrorReport();
+      // (outcome/errorReport were snapshotted right after the run resolved — see above.)
       toStdout(pretty({
         ok: true,
         depth,
@@ -719,11 +742,20 @@ export async function cmdKnowledge(queries: string[], json?: boolean): Promise<n
     await safeShutdown();
   }
 
+  // Same latch as cmdResearch's post-run check: the abort signal's checks inside
+  // searchKnowledge are point-in-time, so a Ctrl-C landing after the last one lets
+  // the search resolve normally — without this, an interrupted run raced onSignal's
+  // flushAndExit and could report ok:true / exit 0. The rationale comment above
+  // named exactly this failure; only the throwing half was previously handled.
+  if (cancellationRequested) {
+    exit = cancellationSignal ? exitCodeForSignal(cancellationSignal) : EXIT.CANCELLED;
+  }
+
   if (json) {
     // `ok: true` makes `ok` a reliable success/failure discriminant across every
     // knowledge JSON variant (disabled-store above, reportError's ok:false below,
     // and this success shape) — additive only, every existing field is unchanged.
-    toStdout(pretty({ ok: true, ...result }));
+    toStdout(pretty({ ok: true, ...(cancellationRequested ? { cancelled: true } : {}), ...result }));
   } else {
     toStdout((result.text.endsWith('\n') ? result.text : result.text + '\n'));
   }
@@ -1158,6 +1190,7 @@ COMMANDS
     --dry-run | --json           Plan only; JSON output.
 
   help, --help, -h               Show this help.
+  --version, -v                  Print the pi-research version.
 
 CONFIGURE
   Credentials are resolved in this order (first match wins):
@@ -1372,6 +1405,15 @@ export function parseArgs(argv: string[]): ParsedArgs {
       } else if (a === '--exclude-tools') {
         const v = takeOptionValue(rest, ++i, '--exclude-tools');
         excludeTools = v.split(',').map((s) => s.trim()).filter(Boolean);
+        // Unknown names fail fast: the merge downstream is a blind Set union, so a
+        // typo ("securty_search") silently excluded nothing while the user believed
+        // the tool was off for the whole (billed) run.
+        const unknown = excludeTools.filter((t) => !RESEARCH_TOOL_NAMES.includes(t));
+        if (unknown.length > 0) {
+          throw new UsageError(
+            `--exclude-tools: unknown tool name${unknown.length === 1 ? '' : 's'} ` +
+            `${unknown.map((t) => `"${t}"`).join(', ')}. Valid names: ${RESEARCH_TOOL_NAMES.join(', ')}.`);
+        }
         if (excludeTools.length === 0) excludeTools = undefined;
       } else if (a === '--initial-links') {
         // Consume until the next flag or end. Validate each entry: these are templated
@@ -1392,6 +1434,10 @@ export function parseArgs(argv: string[]): ParsedArgs {
           }
           j++;
         }
+        // Every other value-taking flag fails fast on a missing value; a bare
+        // `--initial-links` (immediately followed by another flag or end of args)
+        // silently proceeding with zero links was the one exception.
+        if (j === i + 1) throw new UsageError('--initial-links requires at least one URL.');
         i = j - 1;
       } else if (a === '--config') {
         configPath = takeOptionValue(rest, ++i, '--config');

package/src/config.ts

@@ -158,7 +158,8 @@ export const ConfigSchema = Type.Object({
   /**
    * Comma-separated list of research tools to DISABLE for every researcher in a run
    * (e.g. "scrape", or "youtube_transcript,stackexchange"). Valid tool names:
-   * search, scrape, security_search, stackexchange, youtube_transcript, grep.
+   * search, scrape, security_search, stackexchange, youtube_transcript, grep, read
+   * (RESEARCH_TOOL_NAMES in constants.ts is the canonical list).
    * Disable-only: the list is merged into the per-run excludeTools stream, so disabled
    * tools are both removed from each researcher's toolset AND named in the
    * coordinator/evaluator "DISABLED TOOLS" prompt section. Env-only
@@ -1214,7 +1215,9 @@ function parseEnvNumber(env: Record<string, string | undefined>, key: string, de
 }
 
 function parseEnvBool(env: Record<string, string | undefined>, key: string, def: boolean): boolean {
-  const v = env[key];
+  // Trim like parseEnvNumber does: file values arrive trimmed via parseDotEnv, but a
+  // real env var `PI_RESEARCH_X='true '` otherwise silently read false.
+  const v = env[key]?.trim();
   if (v === undefined || v === '') return def;
   // Strict by design: only the literal 'true' (any case) is truthy; every other value is false.
   // This is intentional and asserted in config.test.ts ('1'/'yes' → false) — do not loosen it.
@@ -1237,7 +1240,7 @@ function parseEnvEnum<T extends string>(
   allowed: readonly T[],
   def: T,
 ): T {
-  const raw = env[key];
+  const raw = env[key]?.trim();
   if (raw === undefined || raw === '') return def;
   if ((allowed as readonly string[]).includes(raw)) return raw as T;
   logger.warn(`[config] Invalid ${key}="${raw}" (expected one of: ${allowed.join(', ')}). Using default "${def}".`);

package/src/constants.ts

@@ -182,25 +182,33 @@ export const MAX_QUERIES_PER_RESEARCHER_LEVEL_3 = 20;
  */
 export const DEFAULT_EXCLUDED_TOOLS: readonly string[] = ['grep'];
 
+/**
+ * The complete set of excludable research tool names — the vocabulary
+ * `--exclude-tools` / `excludeTools` / PI_RESEARCH_DISABLED_TOOLS accept.
+ * Kept here (next to DEFAULT_EXCLUDED_TOOLS) so validation and the exclusion
+ * machinery cannot drift; the DISABLED_TOOLS doc in config.ts names the same set.
+ * `read` is a researcher tool too (local file reads) — excluding it is a
+ * legitimate hardening move, and omitting it from this list broke exactly that
+ * on the CLI when validation was introduced.
+ */
+export const RESEARCH_TOOL_NAMES: readonly string[] = [
+  'search', 'scrape', 'security_search', 'stackexchange', 'youtube_transcript', 'grep', 'read',
+];
+
 /**
  * Resolve the effective researcher exclusion list.
  *
- * An EMPTY array is "no preference", not "exclude nothing". The two front-ends
- * express no-preference differently — the pi extension always builds an array
- * (`[...new Set([...])]`, often empty) while the CLI omits the key entirely — so a
- * plain truthiness test made the same user input mean opposite things: grep was
- * available in the extension and excluded on the CLI, regardless of depth.
- *
- * `configDisabled` (PI_RESEARCH_DISABLED_TOOLS) is strictly ADDITIVE. Folding it in
- * as a replacement meant disabling any unrelated tool silently ENABLED grep, so a
- * subtract-only setting could grant a capability.
+ * EVERY input is strictly ADDITIVE on top of DEFAULT_EXCLUDED_TOOLS. Both
+ * subtract-only surfaces — `--exclude-tools` / `excludeTools` and
+ * PI_RESEARCH_DISABLED_TOOLS (`configDisabled`) — are documented as "disable
+ * tools", so neither may GRANT a capability as a side effect. Each has held the
+ * replacement semantics at some point, with the same consequence both times:
+ * excluding any unrelated tool silently enabled local-filesystem grep for web
+ * researchers.
  */
 export function resolveExcludedTools(
   callerExclusions: readonly string[] | undefined,
   configDisabled: readonly string[] = [],
 ): string[] {
-  const base = callerExclusions && callerExclusions.length > 0
-    ? callerExclusions
-    : DEFAULT_EXCLUDED_TOOLS;
-  return [...new Set([...base, ...configDisabled])];
+  return [...new Set([...DEFAULT_EXCLUDED_TOOLS, ...(callerExclusions ?? []), ...configDisabled])];
 }

package/src/infrastructure/browser/thread-worker.mjs

@@ -1512,8 +1512,9 @@ __export(scraper_utils_exports, {
   extractDomain: () => extractDomain,
   formatErrorWithCause: () => formatErrorWithCause,
   getRandomUserAgent: () => getRandomUserAgent,
-  getSsrfSafeDispatcher: () => getSsrfSafeDispatcher,
+  getSsrfSafeFetcher: () => getSsrfSafeFetcher,
   isBenignScrapeFailure: () => isBenignScrapeFailure,
+  isSsrfBlockError: () => isSsrfBlockError,
   ssrfSafeLookup: () => ssrfSafeLookup,
   stripImageLinks: () => stripImageLinks,
   validateContent: () => validateContent,
@@ -1633,16 +1634,29 @@ function ssrfSafeLookup(hostname, options, callback, resolveImpl = dnsCb.lookup)
     return callback(null, safe[0].address, safe[0].family);
   });
 }
-async function getSsrfSafeDispatcher() {
-  if (cachedSsrfDispatcher !== void 0) return cachedSsrfDispatcher;
+function isSsrfBlockError(error) {
+  let e = error;
+  for (let depth = 0; depth < 5 && e instanceof Error; depth++) {
+    if (e.code === "ESSRFBLOCKED" || e.message.includes("not allowed")) {
+      return true;
+    }
+    e = e.cause;
+  }
+  return false;
+}
+async function getSsrfSafeFetcher() {
+  if (cachedSsrfFetcher !== void 0) return cachedSsrfFetcher;
   try {
-    const { Agent } = await import("undici");
-    cachedSsrfDispatcher = new Agent({ connect: { lookup: ssrfSafeLookup } });
+    const { Agent, fetch: undiciFetch } = await import("undici");
+    cachedSsrfFetcher = {
+      fetch: undiciFetch,
+      dispatcher: new Agent({ connect: { lookup: ssrfSafeLookup } })
+    };
   } catch (e) {
     logger.warn(`[Scrapers] undici unavailable; connect-time SSRF pinning disabled (request-time validation still active): ${e instanceof Error ? e.message : String(e)}`);
-    cachedSsrfDispatcher = null;
+    cachedSsrfFetcher = null;
   }
-  return cachedSsrfDispatcher;
+  return cachedSsrfFetcher;
 }
 function formatErrorWithCause(err) {
   try {
@@ -1776,7 +1790,7 @@ function validateContent(html, markdown, url) {
     throw error;
   }
 }
-var cachedSsrfDispatcher;
+var cachedSsrfFetcher;
 var init_scraper_utils = __esm({
   "src/web-research/scraper-utils.ts"() {
     "use strict";
@@ -1785,7 +1799,7 @@ var init_scraper_utils = __esm({
     init_logger();
     init_scraper_types();
     init_browser_error_utils();
-    cachedSsrfDispatcher = void 0;
+    cachedSsrfFetcher = void 0;
   }
 });
 
@@ -2566,7 +2580,8 @@ var ConfigSchema = Type.Object({
   /**
    * Comma-separated list of research tools to DISABLE for every researcher in a run
    * (e.g. "scrape", or "youtube_transcript,stackexchange"). Valid tool names:
-   * search, scrape, security_search, stackexchange, youtube_transcript, grep.
+   * search, scrape, security_search, stackexchange, youtube_transcript, grep, read
+   * (RESEARCH_TOOL_NAMES in constants.ts is the canonical list).
    * Disable-only: the list is merged into the per-run excludeTools stream, so disabled
    * tools are both removed from each researcher's toolset AND named in the
    * coordinator/evaluator "DISABLED TOOLS" prompt section. Env-only

package/src/knowledge/embedder-init.ts

@@ -42,7 +42,23 @@ export async function loadPipelineWithTimeout(
         interOpNumThreads: 1,
       },
     });
-    return await withTimeout(pipelinePromise, timeoutMs, errorMessage);
+    // withTimeout races but cannot cancel: on timeout the underlying pipeline()
+    // keeps loading in the background and its eventual ONNX session would leak
+    // (the caller's catch disposes this.pipeline, which was never assigned).
+    // Dispose the late arrival when it eventually resolves — one orphaned native
+    // session per retry adds up on a slow disk.
+    let timedOut = true;
+    try {
+      const p = await withTimeout(pipelinePromise, timeoutMs, errorMessage);
+      timedOut = false;
+      return p;
+    } finally {
+      if (timedOut) {
+        pipelinePromise
+          .then((late) => (late as unknown as { dispose?: () => Promise<void> }).dispose?.())
+          .catch(() => { /* load failed after the timeout — nothing to dispose */ });
+      }
+    }
   });
 
   return { pipeline: loadedPipeline, errorMessage };

package/src/knowledge/store.ts

@@ -290,6 +290,15 @@ export class KnowledgeStore implements IKnowledgeStore {
           logger.warn('[store] Failed to read table schema:', schemaErr);
         }
       } else {
+        // Same warm-up as the migration branch above: createTable() throws
+        // "embedder dimension unknown" on a never-initialized embedder. This
+        // branch (no existing table — fresh dbDir or first-ever run) hit exactly
+        // that when the embedding factory handed back its model-mismatch local
+        // fallback, which is constructed but never initialize()d: every one of
+        // the caller's init retries burned against the same cached instance.
+        if (this.options.embedder.getDimension() === null) {
+          await this.withEmbedderReconnect(embedder => embedder.embedMany([' ']));
+        }
         this.table = await this.createTable();
       }
 
@@ -787,6 +796,49 @@ export class KnowledgeStore implements IKnowledgeStore {
     }
   }
 
+  /**
+   * Refresh the timestamp of one CURRENT generation for url+type — the
+   * revalidation "touch" for the writer-queue's content-unchanged skip. Without
+   * it a stable page kept its original timestamp forever: with
+   * KNOWLEDGE_STORE_MAX_SERVE_AGE_DAYS set, the serve-age gate then treated the
+   * entry as permanently stale (a live re-scrape on every run, for exactly the
+   * content that never changes), and TTL eviction eventually deleted an entry
+   * that had been re-verified byte-identical possibly hours earlier.
+   */
+  async touchByUrlAndType(url: string, ingestionType: string, generationTs: number, newTs: number): Promise<void> {
+    return this.trackOperation('touchByUrlAndType', undefined, async () => {
+      let retryCount = 0;
+      const MAX_RETRIES = 5;
+      const BASE_DELAY = 100;
+      const scopeFilter = this.getScopeFilter();
+      while (retryCount <= MAX_RETRIES) {
+        const table = await this.getFreshTable();
+        try {
+          const escapedUrl = url.replace(/'/g, "''");
+          const escapedType = ingestionType.replace(/'/g, "''");
+          // valuesSql with integer-coerced literals: timestamp is an Int64 column,
+          // and both inputs are integer-coerced so the SQL can never carry
+          // anything but digits.
+          await table.update({
+            where: `url = '${escapedUrl}' AND ingestion_type = '${escapedType}' AND timestamp = ${Math.floor(generationTs)} AND (${scopeFilter})`,
+            valuesSql: { timestamp: String(Math.floor(newTs)) },
+          });
+          logger.log(`[store] Refreshed timestamp for unchanged ${ingestionType} chunks of ${url}`);
+          return;
+        } catch (err) {
+          if (isLanceCommitConflict(err)) {
+            retryCount++;
+            if (retryCount > MAX_RETRIES) throw err;
+            const delay = Math.floor(Math.random() * (BASE_DELAY * Math.pow(2, retryCount - 1)));
+            await new Promise(r => setTimeout(r, delay));
+            continue;
+          }
+          throw err;
+        }
+      }
+    });
+  }
+
   async findDocumentsByUrl(url: string): Promise<StoreDocument[]> {
     // Coordinate with close(): findByUrl runs via the writer-queue in the post-run
     // background window (link-description storage), exactly when a Ctrl+C-driven
@@ -1179,7 +1231,18 @@ export class KnowledgeStore implements IKnowledgeStore {
           backupDirs.map(async name => ({ name, ts: await this.backupDirTimestamp(name) }))
         );
         ranked.sort((a, b) => a.ts - b.ts);
-        for (const { name } of ranked.slice(0, -1)) {
+        // Retain older backups until they age past the store's own TTL, not just
+        // the single newest one. Two sessions configured with DIFFERENT embedding
+        // models alternate migrations against the shared table (each open sees
+        // "wrong model" → backup → fresh table), and keep-only-newest deleted the
+        // only surviving copy of the first model's data after just two
+        // alternations — silent, permanent loss. A backup older than ttlDays holds
+        // only rows eviction would have deleted anyway, so time-based retention
+        // matches the store's own data-retention contract.
+        const ttlDays = this.options.ttlDays ?? 30;
+        const backupCutoff = Math.trunc(Date.now() - (ttlDays * 24 * 60 * 60 * 1000));
+        for (const { name, ts } of ranked.slice(0, -1)) {
+          if (ts >= backupCutoff) continue;
           const fullPath = path.join(this.options.dbDir, name);
           logger.info(`[store] Removing superseded backup directory: ${name}`);
           await fsPromises.rm(fullPath, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 });

package/src/knowledge/writer-queue.ts

@@ -18,6 +18,9 @@ interface WriterQueueOptions {
     findByUrl: (url: string) => Promise<StoreDocument[]>;
     deleteByUrlAndType: (url: string, type: string, olderThan?: number) => Promise<void>;
     addDocuments: (docs: StoreDocument[]) => Promise<void>;
+    /** Revalidation touch for the content-unchanged skip (see that path). Optional
+     *  so lightweight test doubles need not implement it. */
+    touchByUrlAndType?: (url: string, type: string, generationTs: number, newTs: number) => Promise<void>;
   };
   chunker: Chunker | null;
 }
@@ -199,6 +202,31 @@ export class WriterQueue implements IWriterQueue {
             metrics.increment('knowledge_ingest_stale_prune_failed_total', 1);
           });
         }
+        // The content was just re-fetched and verified byte-identical — record
+        // that as freshness. Without this touch, a stable page's timestamp never
+        // moved: the serve-age gate treated it as permanently stale (live scrape
+        // every run) and TTL eviction eventually deleted a revalidated entry.
+        //
+        // Re-check the newest generation first: a CONCURRENT process may have
+        // ingested changed content for this url+type since our dedup read, and
+        // stamping our (now-old) generation with a later timestamp would make the
+        // stale rows outrank the genuinely newer ones. The touch predicate is
+        // pinned to `timestamp = newestTs`, so the residual window is only the
+        // gap between this re-read and the update — self-healing on the next
+        // real scrape either way.
+        if (this.options.store.touchByUrlAndType) {
+          try {
+            const recheck = (await this.options.store.findByUrl(normalizedUrl))
+              .filter(c => c.metadata['ingestionType'] === incomingType);
+            const stillNewest = recheck.length > 0
+              && Math.max(...recheck.map(d => Number(d.timestamp) || 0)) === newestTs;
+            if (stillNewest) {
+              await this.options.store.touchByUrlAndType(normalizedUrl, incomingType, newestTs, Date.now());
+            }
+          } catch (err) {
+            logger.warn(`[writer-queue] Timestamp refresh for unchanged ${normalizedUrl} (${incomingType}) failed:`, err);
+          }
+        }
         logger.log(`[writer-queue] Skipping ${normalizedUrl} (${incomingType}) — content unchanged.`);
         return;
       }

package/src/orchestration/quick-research-orchestrator.ts

@@ -13,7 +13,7 @@ import {
 import { type Model } from '@earendil-works/pi-ai';
 import { injectCurrentDate } from '../core/llm/inject-date.ts';
 import { loadPrompt } from '../core/llm/prompts.ts';
-import { extractUsage } from '../types/llm.ts';
+import { recordLlmUsage } from '../utils/llm-usage.ts';
 import { logger } from '../logger.ts';
 import { getConfig, type Config } from '../config.ts';
 import { createResearcherSession } from './researcher.ts';
@@ -238,13 +238,13 @@ export class QuickResearchOrchestrator {
 
                 const rawUsage = msg['usage'] as any;
                 if (rawUsage) {
-                    const { tokens, cost } = extractUsage(resolvedModel, rawUsage);
-
+                    const { tokens, cost } = recordLlmUsage(resolvedModel, rawUsage, {
+                        component: 'quick_researcher',
+                        complexity: 0,
+                        observer,
+                    });
                     if (tokens > 0 || cost > 0) {
-                        metrics.increment('llm_tokens_total', tokens, { component: 'quick_researcher', complexity: '0' });
-                        metrics.increment('llm_cost_total', cost, { component: 'quick_researcher', complexity: '0' });
                         observer?.onResearcherProgress?.('quick', undefined, tokens, cost);
-                        observer?.onTokensConsumed?.(tokens, cost);
                     }
                 }
             } else if (event.type === 'tool_execution_start') {