@lincoln504/pi-research 1.4.3 → 1.5.0

package/docs/CHANGELOG.md

@@ -5,7 +5,51 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [1.5.0] - 2026-08-20
+
+Whole-project audit (release integrity, CI/test health, SSRF consistency,
+docs-vs-code accuracy, and an adversarial re-check of every fix claimed in
+the 1.4.1 changelog entry) that found several defects in code the 1.4.1
+audit itself had just "fixed" — proof that a changelog claim needs
+independent re-verification, not just a passing regression test for the
+first failure mode it covers. Plus a prompt-caching workstream: a full verification of how
+prompt caching actually behaves across every provider path pi supports,
+restructuring of every LLM prompt in the codebase for cache-prefix
+invariance, and a caching-aware scrape budget.
+
+### Added
+
+- **Prompt-caching-aware scrape budget.** When prompt caching is known to be active for the run's resolved research model — the Anthropic API (caching is native and always on), or an `openai-completions` provider route explicitly configured with Anthropic-style `cacheControlFormat` — the effective per-researcher scrape-batch maximum is the configured `PI_RESEARCH_MAX_SCRAPE_BATCHES` plus one: the cached system-prompt/tool prefix makes the extra batch's context re-send cheap. The detection is deliberately conservative (`isPromptCachingActiveForModel`): providers whose caching is implicit/server-side (OpenAI Responses, Google, Mistral, Bedrock) report false, because a false positive would raise the budget without the discount that pays for it. One arithmetic source of truth (`getMaxScrapeBatches`) feeds tracker enforcement, the researcher prompt text, the scrape tool's own protocol text, and the TUI progress units, so they can never disagree; the increment is capped so a configured 99 (the documented max) never crosses into the >99 "unlimited" display band (`src/core/llm/research-model-resolver.ts`, `src/constants.ts`, `src/utils/tool-usage-tracker.ts`, `src/tools/scrape.ts`, `src/observers/research-observer-impl.ts`).
+- **Prompt-cache metrics surfaced to users.** The per-run summary's token line now reads e.g. `**150,000** tokens (**90,000** from cache)` whenever the provider reported prompt-cache reads, and `ResearchStats` (SDK `getLastRunStats()`) carries `cacheReadTokens`/`cacheWriteTokens`. Previously the `llm_cache_read/write_tokens_total` counters existed but reached no user-facing surface, so there was no way to see whether caching was working (`src/utils/metrics-summary.ts`).
+- **Weak-grounding notice on completed reports.** A run that completes but verifies fewer than 3 sources now appends an explicit `Grounding notice` after its CITED LINKS list telling the reader (human or calling agent) that unattributed claims are unverified. Driven by a real production run: a depth-1 report whose researchers retrieved almost nothing shipped as a fully sourced-looking document with one citation and fabricated most of its specifics, and the calling agent had to detect the degradation itself. The existing zero-source notes already covered the empty case; this covers the sourced-looking-but-barely-grounded one. The notice sits BEFORE the CITED LINKS header — anything after the last citation entry gets absorbed into that entry's description when quick mode re-parses the report for knowledge-store entries. The CLI additionally prints a stderr `WARNING:` line on completion when a report carries the weak-grounding or no-sources notice, so callers that read stderr (the agent skill does) can distinguish a degraded run from a healthy one without parsing the report (`src/orchestration/research-synthesis-service.ts`, `src/cli.ts`).
+
+### Changed
+
+- **Every researcher's system prompt is now byte-identical across researchers, rounds, and retry attempts.** The per-researcher goal, knowledge-store results, initial evidence, sibling-coordination notes, and steering messages moved from the system prompt into the researcher's initial user message. A byte-identical system prompt + tool set is the cacheable prefix providers key on, so on caching providers every researcher after the first now gets prompt-cache reads on that prefix instead of a fresh cache write per researcher (previously each researcher's unique system prompt guaranteed zero cross-researcher reuse — with Anthropic's 25% cache-write premium, strictly worse than no caching). Quick mode got the same restructure, which also removed a long-standing redundancy where the query was sent twice (once templated into the system prompt, once as the user message). Regression tests pin the invariance directly (`src/prompts/researcher.md`, `src/orchestration/researcher-executor.ts`, `src/orchestration/quick-research-orchestrator.ts`).
+- **The knowledge-search extractor and relevance-triage prompts split static rules from per-call data.** Both stateless background calls put the queries, conversation history, and reference documents/candidates inside the system prompt, so on caching providers every call was a distinct cache entry — pure cache-write premium with zero reads. The templates now carry a `USER_TURN` marker: rules above it stay the (cacheable, byte-identical) system prompt; the per-call sections below are delivered as the user message. A missing marker falls back to the legacy all-in-system shape rather than corrupting the prompt (`src/prompts/research-knowledge-search-extractor.md`, `src/prompts/research-knowledge-relevance-triage.md`, `src/tools/research-knowledge-search.ts`).
+- **Removed a vestigial instruction from the researcher prompt.** "Do NOT try to search the store manually" told researchers not to use a tool they have never been given (`research_knowledge_search` is a top-level tool, not part of the researcher toolset); the surviving wording just explains where knowledge-store results come from and how to cite them.
+- `docs/SDK.md` now opens by describing the package as a high-level research SDK.
+
+### Fixed
+
+- **A markdown-fenced JSON synthesis envelope could ship verbatim as the final report.** `salvageReportText` suppresses broken JSON envelopes when parse and repair both fail, but its check was `startsWith('{')` — a ```` ```json ````-fenced envelope starts with backticks and sailed through (observed in production 2026-08-20: a fenced `{"action":"synthesize",…}` blob, internal protocol fields included, published as a user-facing report). The sniff now strips one leading/trailing code fence before the shape check; prose that merely contains a fenced block is still preserved (`src/core/planning-service.ts`).
+- **The release workflow's credentialed `publish` job interpolated an untrusted job output into its shell.** `npm publish "${{ needs.test-install.outputs.tar_file }}"` splices text originating in the one job that runs untrusted dependency lifecycle scripts directly into the script of the only job holding `id-token: write` — a script-injection channel that re-opened the exact credential/untrusted-code separation the 1.4.2 publish restructure exists to enforce (GitHub's hardening guide lists `needs.*.outputs.*` as an injection source). The publish job now derives the tarball filename deterministically from the release tag (env-indirected), verifies the downloaded artifact contains exactly that file, and the `test-install` job no longer exports the output at all (`.github/workflows/release.yml`).
+- **The browser scrape layer classified PDF-vs-HTML against the pre-redirect URL** — the same bug class 1.4.2 fixed in the fetch layer via `currentUrl`, left unfixed one layer down. A download endpoint without `.pdf` that redirects to the real `.pdf` file (with missing/mislabeled content-type) fell into the HTML branch, serializing Firefox's PDF-viewer chrome instead of the document, and the oversized-PDF early abort never armed for it. Both browser-layer checks now use the response's own post-redirect URL (`src/infrastructure/browser/thread-worker-messaging.ts`).
+- **Unit tests polluted the real shared log file.** The test setup's `PI_RESEARCH_LOG_PATH` redirect ran in the setup module's body — but its own static imports (metrics → logger) are hoisted and evaluate first, so the logger singleton had already bound the user's real log path before the redirect executed. Every error-path unit test then appended real-looking WARN/ERROR entries (6 spurious ERRORs and hundreds of WARNs in one observed day) that materially slowed forensic diagnosis of real runs. The env assignments now live in an import-free module imported first (`test/setup/unit-env.ts`).
+- **The missing-config advisory was the single most common line in the log.** `getConfig` warned "No configuration found for workspace" on every call from an unconfigured directory — 362 entries in one observed day. Now warned once per workspace per process (`src/config.ts`).
+- Singular knowledge-search progress line rendered a bare trailing slash ("1 query/") — same rate-vs-count confusion the search-burst line was already fixed for (`src/cli.ts`).
+- `isVerboseFromEnv`'s doc comment claimed config loads sync `config.DEBUG` into the env; only saves do. The comment now says exactly when a `DEBUG=true` line in config.env takes effect (`src/utils/log-utils.ts`).
+- **Four 1.4.2 fixes shipped without the regression tests their release notes implied.** Backfilled and verified: `security_search` must not charge the shared gathering budget on unknown-database/invalid-severity rejects; `redactSecrets` must still mark the scan-cut tail as truncated when redaction shrinks the prefix under the cap; `uninstallSkill([])` must remove nothing (an explicit empty list is not `--all`); and redirect-aware PDF/HTML classification on the fetch path (both directions, verified failing against the pre-fix code).
+- **Knowledge-store migration could still lose data if BOTH the canonical rename and its own revert failed.** The 1.4.1 fix for the manifest/disk desync bug added a revert-the-rename path for when the post-rename manifest save fails, but the revert's own failure fell through into a catch block written for a different case (the first rename never happening at all) — which pointed the manifest at a directory that no longer existed on disk. Now retries the manifest save against the name that actually matches disk, and logs loudly if that also fails, instead of silently stranding the data; a third-order failure (the recovery reopen itself failing) is also handled — and the manifest repair is deliberately not gated behind the reopen, since it needs only file writes, so a reopen-only fault still gets the manifest fixed for the next restart. Three regression tests cover the compound failures directly (`src/knowledge/store.ts`).
+- **`SIGQUIT` was documented as gracefully handled and wasn't.** `docs/SDK.md` and `SKILL.md` list SIGQUIT alongside SIGINT/SIGTERM/SIGHUP as a signal the CLI catches and shuts down cleanly from — but no listener was ever registered for it, so a real SIGQUIT hit Node's default disposition (immediate kill, no abort, no drain) instead. Added the missing handler to both the CLI and SDK; a real-subprocess test now sends an actual SIGQUIT to the built binary and distinguishes "our handler ran" from "the OS killed it" (`src/cli.ts`, `src/sdk.ts`).
+- **Embedding-server shutdown could dispose the pipeline while a second queued embed was starting.** `SerialQueue.waitForIdle()` only checked the currently-executing task; there's a real gap, verified with a controlled reproduction, between one task clearing that state and the next queued task claiming it, and a caller checking exactly in that gap saw "idle" while a second native call was about to run against the about-to-be-disposed pipeline — the same class of native-GPU-concurrency bug this queue exists to prevent, one layer deeper than the 1.4.1 fix covered. Now also tracks whether the queue is still mid-drain (`src/infrastructure/embedding/embedding-server.ts`).
+- **Two sibling headless research runs finishing at nearly the same moment could both skip clearing shared steering messages.** `endResearchSession`'s "last run in the session" gate only checked TUI-only state (`order`, `panels`), so on every headless run it was trivially true — one headless run ending reset steering while a sibling headless run was still consuming it. The gate now also requires the abort registry (populated by TUI and headless runs alike) to be empty, and because this run's own entry is removed in the same synchronous function, the whichever-sibling-finishes-last decision is atomic; the separate, racy guard the research tool used to run before its cleanup was removed in favor of this one (`src/orchestration/session-state.ts`, `src/tools/research-tool-definition.ts`).
+- **A single YouTube transcript could dominate a researcher's context window.** The `youtube_transcript` tool concatenated raw transcripts into its tool response with no per-video cap — a long video (or several in one batch call) could inject hundreds of thousands of characters. Each transcript is now truncated with the same per-document cap and marker the scrape tool uses (`src/youtube/index.ts`).
+- **A legacy `.pi-research.env` file in the current directory could set user-scoped keys.** The tool runs in arbitrary (including untrusted) repositories; a checked-in legacy env file could override user-level configuration such as API endpoints for anyone who ran research in that directory. CWD-local legacy files are now restricted to the same local-scope key allowlist that already governed project-scoped registry storage; user-scoped keys in them are ignored (`src/config.ts`).
+- **The scrape tool's own protocol text could disagree with the enforced limit.** `scrape.ts` computed its batch count from config directly instead of asking the tracker that actually enforces the limit; with the new caching-aware budget those could differ by one. The tool now reads the tracker's limit, falling back to config only when no tracker is wired (`src/tools/scrape.ts`).
+- **Release workflow's credentialed `publish` job ran an unscoped `npm install`.** `npm install -g npm@11` (needed for the OIDC trusted-publishing exchange) executed in the one job holding `id-token: write`, without `--ignore-scripts` — npm itself ships no install-time lifecycle scripts today, but the job's whole design is to never run installable code there on trust alone. Added `--ignore-scripts`; verified the resulting npm binary still publishes correctly (`.github/workflows/release.yml`).
+- **README overstated the knowledge store as always-on.** "Every finding is written to a local LanceDB knowledge store" implied unconditional, automatic behavior; it's actually conditional on the store being enabled. Reworded to "can be saved."
+- **`AGENT-SKILL.md` presented the Codex skill-install path as equally confirmed as Claude/OpenClaw's.** The installer itself tags the Codex path `confidence: 'unverified'` ("path not confirmed by official docs"); the doc now says so.
 
 ## [1.4.3] - 2026-08-19
 

package/docs/CONFIGURATION.md

@@ -68,7 +68,7 @@ Research
 | `PI_RESEARCH_TIMEOUT_MS` (TUI) | `300000` | 180000–1800000 | Per-researcher timeout (3–30 min). |
 | `PI_RESEARCH_MAX_RESEARCHERS` (TUI) | `3` | 1–5 | Parallel researchers. |
 | `PI_RESEARCH_DEFAULT_RESEARCH_DEPTH` (TUI) `[project]` | `1` | 1–3 | Depth for `/research` and the CLI when `--depth` is omitted (1=normal, 2=deep, 3=ultra). |
-| `PI_RESEARCH_MAX_SCRAPE_BATCHES` (TUI) | `2` | 0–99 | Scrape batches per researcher (0 = unlimited). |
+| `PI_RESEARCH_MAX_SCRAPE_BATCHES` (TUI) | `2` | 0–99 | Scrape batches per researcher (0 = unlimited). When prompt caching is known to be active for the resolved research model (Anthropic API models, or a provider route explicitly configured for Anthropic-style cache control), the effective limit is this value plus one — the cached prompt prefix makes the extra batch cheap. |
 | `PI_RESEARCH_MAX_GATHERING_CALLS` | `12` | 1–100 | Shared web-gathering calls per researcher (`search` + `security_search` + `stackexchange` + `youtube_transcript`). |
 | `PI_RESEARCH_MAX_CONCURRENT_SCRAPES` | `3` | 1–20 | Concurrent URLs fetched per scrape batch. |
 | `PI_RESEARCH_MAX_RETRIES` | `2` | 0–5 | Retries per researcher request. |

package/docs/SDK.md

@@ -1,6 +1,6 @@
 ## SDK
 
-The programmatic SDK for scripts, CI, and custom tooling. For configuration (the
+A high-level research SDK for scripts, CI, and custom tooling. For configuration (the
 layering model, the TUI settings, and every environment variable) see
 [CONFIGURATION.md](CONFIGURATION.md).
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lincoln504/pi-research",
-  "version": "1.4.3",
+  "version": "1.5.0",
   "description": "Web research for your agents with smart and safe tooling + knowledge store",
   "type": "module",
   "private": false,

package/skills/pi-research/SKILL.md

@@ -2,7 +2,7 @@
 name: pi-research
 description: "Multi-agent web research for any coding agent. Use for anything requiring current information from the internet — news, trends, product/service/company facts, statistics, people, places, events, APIs, library behavior, or 'what is X / how does X work' questions. Also use to search a local knowledge store of previously researched findings before doing live research. Powered by the pi-research engine (search, stealth scraping, security databases, Stack Exchange). Not for reading local files, running project commands, or analyzing this repository's own source."
 license: MIT
-metadata: { "author": "Lincoln Deen", "version": "1.4.3", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
+metadata: { "author": "Lincoln Deen", "version": "1.5.0", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
 allowed-tools: Bash(node:*)
 ---
 

package/src/cli.ts

@@ -150,7 +150,7 @@ function pretty(obj: unknown): string {
  * lifecycle event to stderr. Keeps a long research run visible without spamming
  * stdout (the report goes to stdout, untouched).
  */
-function makeProgressObserver(): HeadlessObserverOptions {
+export function makeProgressObserver(): HeadlessObserverOptions {
   return {
     enableLogging: false,
     onProgress: (event: string, data?: any) => {
@@ -196,9 +196,24 @@ function makeProgressObserver(): HeadlessObserverOptions {
           }
           break;
         }
-        case 'complete':
+        case 'complete': {
           toStderr(`  • complete (${data?.result?.length ?? 0} chars)\n`);
+          // Degradation signal for callers that read stderr (the agent skill
+          // does): a run that completes but is weakly grounded or retrieved no
+          // sources at all looks identical to a healthy run on exit code and
+          // the "complete" line alone. Two real runs on 2026-08-20 shipped
+          // degraded reports (1 citation / 0 sources) with nothing on stderr to
+          // distinguish them. The markers are the synthesis service's own
+          // notices, appended by ensureCitedLinks.
+          const reportText = typeof data?.result === 'string' ? data.result : '';
+          if (reportText.includes('No web sources were successfully retrieved') ||
+              reportText.includes('None of the links above could be verified')) {
+            toStderr('  • WARNING: no web sources could be retrieved — the report is unverified\n');
+          } else if (reportText.includes('Grounding notice: only')) {
+            toStderr('  • WARNING: weakly grounded report (fewer than 3 verified sources) — treat unattributed claims as unverified\n');
+          }
           break;
+        }
         case 'error':
           toStderr(`  • error: ${data?.message}\n`);
           break;
@@ -738,7 +753,9 @@ export async function cmdKnowledge(queries: string[], json?: boolean): Promise<n
       config: getConfig(process.cwd(), 'cli'),
       ignoreGlobalConfig: true,
     });
-    toStderr(`[pi-research] searching knowledge store (${queries.length} query/${queries.length === 1 ? '' : 's'})…\n`);
+    // "N queries", not "N query/s" — same rationale as the search_start line above:
+    // the slash form reads as a rate and renders "1 query/" in the singular.
+    toStderr(`[pi-research] searching knowledge store (${queries.length} ${queries.length === 1 ? 'query' : 'queries'})…\n`);
     result = await searchKnowledge(queries, abortController.signal);
   } catch (err) {
     return reportError(err, 'knowledge search', json);
@@ -1660,10 +1677,34 @@ async function main(argv: string[]): Promise<number> {
   // left the SDK's handler to tear the container down underneath a live orchestrator
   // — exactly the race the CLI installs its own handlers to avoid.
   process.on('SIGHUP', () => onSignal('SIGHUP'));
+  // SIGQUIT: documented (see exitCodeForSignal's comment and the exit-code
+  // tables in docs/SDK.md and SKILL.md) as handled the same way as SIGHUP,
+  // but no listener was ever registered for it — an unhandled SIGQUIT hits
+  // Node's default disposition (immediate termination, no abort, no drain),
+  // silently breaking that documented guarantee. Has no Windows equivalent;
+  // registering the listener there is a documented no-op, so no guard needed.
+  process.on('SIGQUIT', () => onSignal('SIGQUIT'));
   if (process.platform === 'win32') {
     process.on('SIGBREAK', () => onSignal('SIGBREAK'));
   }
 
+  // Test-only seam: idle here, right after every handler above is registered,
+  // instead of parsing argv/running a command. Lets a subprocess test deliver
+  // a real OS signal and observe THIS handler run (process.exit with the
+  // derived code) rather than Node's default disposition (process killed by
+  // the signal, no code) — the exact distinction that would have caught the
+  // missing SIGQUIT registration this seam was added for. Never set in
+  // production; see the SIGQUIT subprocess test in test/unit/cli.test.ts.
+  if (process.env['PI_RESEARCH_TEST_HANG_FOR_SIGNAL'] === '1') {
+    process.stdout.write('PI_RESEARCH_TEST_READY\n');
+    // A bare unresolved Promise does not keep the event loop alive by
+    // itself (only real libuv handles do — an installed signal listener
+    // isn't one either), so without a ref'd timer the process would exit 0
+    // on its own the instant this synchronous section finishes, before any
+    // signal could ever arrive.
+    await new Promise<never>(() => setInterval(() => {}, 1000));
+  }
+
   let parsed: ParsedArgs;
   try {
     parsed = parseArgs(argv);

package/src/config.ts

@@ -751,16 +751,23 @@ function loadEnvFiles(cwd: string, iface?: ConfigInterface): Record<string, stri
     try {
       logger.warn(`[config] WARNING: .pi-research.env files are deprecated. Settings will be auto-migrated to the centralized registry. Remove ${legacyPath} after migration.`);
       legacyEnv = parseDotEnv(fs.readFileSync(legacyPath, 'utf-8'));
-      Object.assign(merged, legacyEnv);
-      
-      // Auto-migrate legacy settings to the central registry — but only LOCAL_SCOPE_KEYS. The
-      // registry is project-scoped storage by design (saveConfig writes only these keys to it);
-      // persisting user-scoped legacy keys would freeze them per-workspace and silently override
-      // the user's global config.env forever. Non-local legacy keys still apply to THIS resolution
-      // via the Object.assign(merged, legacyEnv) above — they just aren't persisted.
+      // Only LOCAL_SCOPE_KEYS from this run's resolution too — same boundary
+      // step 4 below already enforces for the registry this file migrates
+      // into ("the registry is project-scoped storage... never user-scoped
+      // keys"). This file lives in the CWD, which for this tool is routinely
+      // an arbitrary, potentially untrusted repository a coding agent was
+      // just pointed at — applying every key unconditionally let a dropped-in
+      // .pi-research.env silently control user-scoped settings (RESEARCH_MODEL,
+      // EMBEDDING_DEVICE, DISABLED_TOOLS, export destinations, ...) for that
+      // run with no confirmation, for any setting the user hadn't already set
+      // via a real environment variable (which still wins over all of this).
+      // Non-local keys are ignored here exactly as they already are for the
+      // registry — this predecessor mechanism should not be MORE permissive
+      // than the successor it's migrating into.
       const legacyLocal = Object.fromEntries(
         Object.entries(legacyEnv).filter(([k]) => LOCAL_SCOPE_KEYS.has(k))
       );
+      Object.assign(merged, legacyLocal);
       const existingLocal = Object.fromEntries(
         Object.entries(findRegistryEntry(registry, normalizedCwd) ?? {}).filter(([k]) => LOCAL_SCOPE_KEYS.has(k))
       );
@@ -799,8 +806,14 @@ function loadEnvFiles(cwd: string, iface?: ConfigInterface): Record<string, stri
       }
     }
   } else if (Object.keys(merged).length === 0 && !fs.existsSync(legacyPath) && !fs.existsSync(globalPath)) {
-    // 5. Warning for missing config
-    logger.warn(`[config] No configuration found for workspace: ${cwd}. Using code defaults. Run /research-config to configure.`);
+    // 5. Warning for missing config — once per workspace per process. getConfig
+    // is called on many hot paths (every tool invocation, health checks), and
+    // repeating this advisory each time made it the single most common line in
+    // the shared log (362 entries in one observed day), burying real warnings.
+    if (!warnedMissingConfigWorkspaces.has(normalizedCwd)) {
+      warnedMissingConfigWorkspaces.add(normalizedCwd);
+      logger.warn(`[config] No configuration found for workspace: ${cwd}. Using code defaults. Run /research-config to configure.`);
+    }
   }
 
   warnLegacyEnvVarsOnce(merged);
@@ -825,6 +838,10 @@ const LEGACY_ENV_VARS: Record<string, string> = {
 
 let warnedLegacyEnvVars = false;
 
+// Workspaces already warned about having no configuration (see loadEnvFiles) —
+// per-process dedup so the advisory appears once, not on every getConfig call.
+const warnedMissingConfigWorkspaces = new Set<string>();
+
 /** Warn once per process about set-but-dead 0.1.x env vars (upgrade aid; never throws). */
 function warnLegacyEnvVarsOnce(merged: Record<string, string>): void {
   if (warnedLegacyEnvVars) return;

package/src/constants.ts

@@ -46,11 +46,26 @@ export function getMaxGatheringCalls(config?: Config): number {
 /**
  * Get the maximum scrape batches from config.
  * This function should be used instead of MAX_SCRAPE_CALLS to support dynamic configuration.
+ *
+ * @param cachingActive - Pass `true` when prompt caching is active for this
+ * run's resolved model (see isPromptCachingActiveForModel in
+ * core/llm/research-model-resolver.ts). With caching active, the EFFECTIVE
+ * maximum is the configured value plus one: a cached run's system-prompt/
+ * tool-definition prefix is cheap to re-send, so the same token budget
+ * covers one more scrape batch per researcher. Every consumer of the batch
+ * count (tracker enforcement, prompt text, progress units) must go through
+ * this function with the same cachingActive value so they all agree on the
+ * effective number. The "unlimited" sentinel (0 or >99) is unchanged —
+ * adding one to unlimited is meaningless.
  */
-export function getMaxScrapeBatches(config?: Config): number {
+export function getMaxScrapeBatches(config?: Config, cachingActive = false): number {
   try {
     const batches = (config || getConfig()).MAX_SCRAPE_BATCHES;
-    return batches === 0 || batches > 99 ? 999999 : batches;
+    if (batches === 0 || batches > 99) return 999999;
+    // Cap at 99: a configured 99 (the documented max) plus one would land in the
+    // >99 "unlimited" band that every renderer checks — the prompt would say
+    // "unlimited batches" while enforcement actually blocked at 100.
+    return cachingActive ? Math.min(batches + 1, 99) : batches;
   } catch {
     return 2; // Fallback to default
   }
@@ -63,9 +78,13 @@ export const MAX_SCRAPE_URLS = 6;
  * Get the units per researcher for the progress bar.
  * Units = 1 (for start/search) + number of scrape batches.
  * When batches are unlimited (MAX_SCRAPE_BATCHES=0), caps at 4 for a sane progress estimate.
+ *
+ * Relative to the EFFECTIVE batch count: pass the run's cachingActive (from
+ * isPromptCachingActiveForModel over the resolved model) so progress units
+ * agree with the limit the tracker actually enforces.
  */
-export function getUnitsPerResearcher(config?: Config): number {
-  const batches = getMaxScrapeBatches(config);
+export function getUnitsPerResearcher(config?: Config, cachingActive = false): number {
+  const batches = getMaxScrapeBatches(config, cachingActive);
   return 1 + Math.min(batches, 4);
 }
 

package/src/core/llm/research-model-resolver.ts

@@ -89,3 +89,41 @@ export function resolveResearchModel(options: {
 
   throw new Error(buildNoModelAvailableMessage());
 }
+
+/**
+ * Best-effort, CONSERVATIVE signal for "will pi-ai apply prompt caching for
+ * this resolved model." pi-ai owns all cache-mechanism logic internally (this
+ * codebase never constructs cache_control blocks itself) and exposes no
+ * public API to ask in advance for most providers, so this can only answer
+ * for the two client-visible cases:
+ *
+ * - `api === 'anthropic-messages'`: caching is native and on by default (no
+ *   capability gate at all — see pi-ai's anthropic-messages provider).
+ * - `api === 'openai-completions'` with `compat.cacheControlFormat ===
+ *   'anthropic'`: an explicit opt-in to Anthropic-style cache_control
+ *   emulation (e.g. an OpenRouter/gateway route to an Anthropic model
+ *   configured with this override).
+ *
+ * Every other case returns false even though caching may actually be
+ * active: pi's generated model catalog sets `cacheControlFormat` on zero
+ * models, so plain `openai-completions` traffic (this project's custom
+ * providers — glm-coding, cerebras, zai, OpenRouter, ...) has its cache
+ * compat auto-detected by pi-ai's own private, unexported heuristic INSIDE
+ * the live request — not knowable client-side, before or after model
+ * resolution. Providers with fully implicit/automatic caching
+ * (openai-responses family, google-generative-ai, google-vertex,
+ * mistral-conversations, bedrock-converse-stream) have no "will it be
+ * active" signal at all — it depends on provider-side runtime behavior
+ * (minimum prompt length, server cache state). Treat this as "caching is
+ * DEFINITELY active" — a false negative is expected and fine for providers
+ * outside these two cases, but a false positive would be a real bug.
+ */
+export function isPromptCachingActiveForModel(model: Model<any> | undefined): boolean {
+  if (!model) return false;
+  if (model.api === 'anthropic-messages') return true;
+  if (model.api === 'openai-completions') {
+    const compat = model.compat as { cacheControlFormat?: string } | undefined;
+    return compat?.cacheControlFormat === 'anthropic';
+  }
+  return false;
+}

package/src/core/planning-service.ts

@@ -220,8 +220,19 @@ function isDegradableLlmError(error: unknown, signal?: AbortSignal): boolean {
 export function salvageReportText(raw: string): string {
   const trimmed = raw.trim();
   if (trimmed.length <= 50) return '';
+  // Envelope sniff. Models routinely wrap the JSON envelope in a markdown code
+  // fence (```json … ```); a fenced envelope starts with backticks, not '{', so a
+  // bare startsWith('{') check shipped one verbatim as a user-facing report
+  // (observed in production 2026-08-20: a ```json-fenced {"action":"synthesize",…}
+  // blob, internal protocol fields included, published as the final report).
+  // Strip one leading fence line and a trailing fence before the shape check.
+  let sniff = trimmed;
+  const fence = sniff.match(/^```[a-zA-Z0-9_-]*[ \t]*\r?\n/);
+  if (fence) {
+    sniff = sniff.slice(fence[0].length).replace(/\r?\n```\s*$/, '').trim();
+  }
   // Looks like a (broken) JSON envelope rather than a report — suppress it.
-  if (trimmed.startsWith('{') && /"(?:action|content|researchers)"\s*:/.test(trimmed)) {
+  if (sniff.startsWith('{') && /"(?:action|content|researchers)"\s*:/.test(sniff)) {
     return '';
   }
   return raw;

package/src/index.ts

@@ -556,7 +556,9 @@ export default async function (pi: ExtensionAPI) {
 
     // Researcher sub-agent sessions already carry the authoritative steering copy:
     // the executor (deep) and quick orchestrator inject "ADDITIONAL USER GUIDANCE"/
-    // "ADDITIONAL CONSIDERATIONS" into the researcher system prompt themselves.
+    // "ADDITIONAL CONSIDERATIONS" into the researcher's initial user message themselves
+    // (kept OUT of the system prompt so its cacheable prefix stays byte-identical
+    // across researchers — see researcher-executor.ts).
     // Appending the same messages again here delivered every steering message
     // TWICE to every researcher — return the prompt untouched. (The coordinator and
     // the research lead never pass through this hook at all: they run via

package/src/infrastructure/browser/thread-worker-messaging.ts

@@ -542,7 +542,11 @@ export async function executeScrapeTask(
               // Matches the PDF branch below AND the fetch layer's own detection
               // (web-scraper.ts): a server can mislabel or omit content-type, so a
               // `.pdf` URL extension is an equally-trusted signal, not just the header.
-              const looksLikePdf = ct?.includes('application/pdf') || url.toLowerCase().endsWith('.pdf');
+              // resp.url() — THIS response's URL, not the original task URL: goto()
+              // follows redirects, and a download endpoint without `.pdf` that
+              // redirects to the real `.pdf` file must still arm the early abort
+              // (same redirect-aware fix the fetch layer got via currentUrl).
+              const looksLikePdf = ct?.includes('application/pdf') || String(resp.url()).toLowerCase().endsWith('.pdf');
               if (!looksLikePdf || oversizedPdfError || poisonedError) return;
               const declared = parseInt((await resp.headerValue('content-length').catch(() => null)) || '', 10);
               if (Number.isFinite(declared) && declared > MAX_PDF_SIZE) {

package/src/infrastructure/browser/thread-worker.mjs

@@ -2332,7 +2332,7 @@ async function executeScrapeTask(_context, url, signal) {
         if (resp.request().resourceType() === "document" && resp.frame() === page.mainFrame()) {
           finalMainFrameAddr = resp.serverAddr().catch(() => null);
           const check2 = Promise.resolve(resp.headerValue("content-type")).then(async (ct) => {
-            const looksLikePdf = ct?.includes("application/pdf") || url.toLowerCase().endsWith(".pdf");
+            const looksLikePdf = ct?.includes("application/pdf") || String(resp.url()).toLowerCase().endsWith(".pdf");
             if (!looksLikePdf || oversizedPdfError || poisonedError) return;
             const declared = parseInt(await resp.headerValue("content-length").catch(() => null) || "", 10);
             if (Number.isFinite(declared) && declared > MAX_PDF_SIZE) {

package/src/infrastructure/embedding/embedding-server.ts

@@ -178,19 +178,39 @@ class SerialQueue {
   }
 
   /**
-   * Resolve once no native call is in flight. Used by EmbeddingServer.shutdown()
-   * to wait out a still-running inference before disposing the pipeline —
-   * without this, disposal could run concurrently with a legitimately slow (not
-   * hung) embed/embedMany call on the same re-entrant-locked GPU context, the
-   * exact native-segfault class this queue exists to prevent. No timeout: a
-   * promise that truly never settles is already an accepted, pre-existing
-   * leak-to-process-exit scenario elsewhere in this class (see the poison path's
+   * Resolve once no native call is in flight AND no queued task is still due
+   * to run. Used by EmbeddingServer.shutdown() to wait out a still-running
+   * inference before disposing the pipeline — without this, disposal could
+   * run concurrently with a legitimately slow (not hung) embed/embedMany call
+   * on the same re-entrant-locked GPU context, the exact native-segfault
+   * class this queue exists to prevent. No timeout: a promise that truly
+   * never settles is already an accepted, pre-existing leak-to-process-exit
+   * scenario elsewhere in this class (see the poison path's
    * disposeEmbedder=false handling) — better to leak than risk concurrent
    * native access.
+   *
+   * Checking only `currentWork` is NOT enough with a second task already
+   * queued: a task's `run()` clears `currentWork` to null in its `finally`
+   * (once its native call has settled), and `pump()` only claims it again for
+   * the next task on a LATER microtask tick, once `run()` for the current
+   * task has fully unwound and `await entry.run()` resumes. If a caller's own
+   * check happens to land in that gap, `currentWork` reads as null even
+   * though a second native call is about to start (or a queue in the process
+   * of unwinding to its next task hasn't reached it yet) — exactly the window
+   * disposal must not run in. `running` stays true for the queue's entire
+   * drain, across every task in `tasks`, not just the current one, so it has
+   * no equivalent gap and closes this one.
    */
   async waitForIdle(): Promise<void> {
-    while (this.currentWork) {
-      await this.currentWork.catch(() => {});
+    while (this.currentWork || this.running) {
+      if (this.currentWork) {
+        await this.currentWork.catch(() => {});
+      } else {
+        // Nothing executing right now, but pump() is still mid-drain (about
+        // to claim the next queued task, or unwinding) — yield a microtask
+        // so it can advance instead of reporting idle into this exact gap.
+        await Promise.resolve();
+      }
     }
   }
 

package/src/knowledge/store.ts

@@ -527,10 +527,66 @@ export class KnowledgeStore implements IKnowledgeStore {
           // out of the way and start a fresh empty table over one manifest
           // write hiccup.
           logger.error('[store] Failed to persist canonical table name after rename — reverting to keep manifest and disk in sync.');
-          await fsPromises.rename(canonicalDir, tempDir);
-          this.tableName = tempTableName;
-          this.table = await this.db.openTable(this.tableName);
-          logger.info(`[store] Migrated data remains safe and accessible as ${tempTableName}; canonical rename deferred to a future successful manifest save.`);
+          try {
+            await fsPromises.rename(canonicalDir, tempDir);
+            this.tableName = tempTableName;
+            this.table = await this.db.openTable(this.tableName);
+            logger.info(`[store] Migrated data remains safe and accessible as ${tempTableName}; canonical rename deferred to a future successful manifest save.`);
+          } catch (revertErr) {
+            // The revert itself failed too. fs.rename is atomic on a single
+            // filesystem — a failed rename never leaves data half-moved — so
+            // the data is still fully at canonicalDir, not tempDir. The outer
+            // catch below was written for a DIFFERENT case (the first rename
+            // never happening at all) and would wrongly point the manifest at
+            // tempTableName, whose directory no longer exists on disk. Instead,
+            // try to persist a manifest against the name that actually matches
+            // disk right now, so the mismatch this whole branch exists to avoid
+            // doesn't reappear one level deeper.
+            logger.error(`[store] Revert rename also failed; data remains on disk at ${canonicalDir}: ${revertErr}`);
+            // Everything from here on is wrapped in its OWN try/catch and never
+            // rethrows: a third compounding failure (e.g. openTable() itself
+            // throwing, on the same underlying disk fault that already broke
+            // two renames) must not escape to the outer catch (renameErr)
+            // either — that one assumes data is still at tempDir, which by
+            // this point is doubly false.
+            // The manifest repair is deliberately NOT gated behind the table
+            // reopen: saveManifest depends only on writeFile/rename, not on an
+            // open table, so a reopen-only failure (a LanceDB-level fault on an
+            // otherwise-writable filesystem) must not skip a repair the code can
+            // still make — a skipped repair leaves the persisted manifest
+            // pointing at the nonexistent temp name, which is the next-restart
+            // empty-table hazard this whole branch exists to prevent.
+            this.tableName = canonicalName;
+            let reopened = false;
+            try {
+              this.table = await this.db.openTable(this.tableName);
+              reopened = true;
+            } catch (openErr) {
+              // A third compounding failure: even reopening the table at its
+              // actual on-disk location failed. This process has no working
+              // table handle for it, but the data itself should still be
+              // physically intact at canonicalDir (only the rename and its
+              // revert have run, and neither drops data — they only move a
+              // directory or fail outright). The manifest repair below still
+              // proceeds, so a future restart can find the data even though
+              // this process cannot serve it.
+              logger.error(`[store] CRITICAL: could not reopen the table at '${canonicalName}' after a failed rename revert (${openErr}). Data should still be intact on disk at ${canonicalDir}, but this process has no working table handle for it. Attempting the manifest repair anyway so a restart can find the data; verify ${canonicalDir} exists if problems persist.`);
+            }
+            try {
+              if (await this.saveManifest()) {
+                logger.warn(`[store] Recovered: manifest now points at '${canonicalName}', matching the data's actual on-disk location${reopened ? '' : ' (this process still has no working table handle — restart to serve it)'}.`);
+              } else {
+                // Both the revert and this recovery manifest save failed. The
+                // on-disk manifest (last persisted pointing at tempTableName,
+                // before the old table was dropped) now disagrees with disk.
+                // Fail loud rather than let a future restart silently create
+                // an empty table under the stale name.
+                logger.error(`[store] CRITICAL: could not persist a manifest matching disk after a failed rename revert. Data is intact on disk at ${canonicalDir} (table '${canonicalName}'), but the persisted manifest is stale and points at '${tempTableName}', which no longer exists. If this process exits before a future successful manifest save, the next start will not find the data automatically — manually repair the manifest to point at '${canonicalName}' before restarting.`);
+              }
+            } catch (manifestErr) {
+              logger.error(`[store] CRITICAL: manifest repair threw after a failed rename revert (${manifestErr}). Data is intact on disk at ${canonicalDir} (table '${canonicalName}'), but the persisted manifest is stale and points at '${tempTableName}'. Manually repair the manifest to point at '${canonicalName}' before restarting.`);
+            }
+          }
         }
       } catch (renameErr) {
         // If rename fails (cross-device, permissions, etc.) we keep the temp name

package/src/observers/research-observer-impl.ts

@@ -43,6 +43,14 @@ export interface ObserverContext {
    */
   renderImmediate?: () => void;
   researchComplexity: number;
+  /**
+   * Whether prompt caching is active for the run's resolved model (see
+   * isPromptCachingActiveForModel). With caching active the effective
+   * per-researcher scrape-batch maximum is one higher, and progress units
+   * must be computed relative to that same effective count the tracker
+   * enforces — not the base configured value.
+   */
+  cachingActive?: boolean;
 }
 
 export interface ObserverState {
@@ -94,7 +102,7 @@ export function createResearchObserver(
         activateSlice(panelState, state.quickSliceLabel);
         updateSliceStatus(panelState, state.quickSliceLabel, 'starting...', debouncedRefresh);
 
-        const units = getUnitsPerResearcher();
+        const units = getUnitsPerResearcher(undefined, ctx.cachingActive);
         panelState.progress = { expected: units, made: 0 };
       }
       // Quick research: researcher is running → steering is acceptable
@@ -152,7 +160,7 @@ export function createResearchObserver(
     onPlanningSuccess: (plan) => {
       updateSliceStatus(panelState, 'coord', 'ready', debouncedRefresh);
       completeSlice(panelState, 'coord');
-      const unitsPerResearcher = getUnitsPerResearcher();
+      const unitsPerResearcher = getUnitsPerResearcher(undefined, ctx.cachingActive);
       const count = plan.researchers?.length || 0;
       const units = (count * unitsPerResearcher) + LEAD_EVAL_UNITS;
       panelState.progress = { expected: units, made: 0 };
@@ -319,7 +327,7 @@ export function createResearchObserver(
 
     onResearcherProgress: (id, status, tokens, cost) => {
       const sliceId = id === 'quick' ? state.quickSliceLabel : id;
-      const unitsPerResearcher = getUnitsPerResearcher();
+      const unitsPerResearcher = getUnitsPerResearcher(undefined, ctx.cachingActive);
       
       if (status !== undefined) {
         if (status.startsWith('done:')) {
@@ -350,7 +358,7 @@ export function createResearchObserver(
     onResearcherComplete: (id, _report) => {
       const sliceId = id === 'quick' ? state.quickSliceLabel : id;
       if (panelState.progress) {
-        const unitsPerResearcher = getUnitsPerResearcher();
+        const unitsPerResearcher = getUnitsPerResearcher(undefined, ctx.cachingActive);
         const current = progressCredits.get(id) ?? 0;
         const remaining = unitsPerResearcher - current;
         if (remaining > 0) {
@@ -365,7 +373,7 @@ export function createResearchObserver(
     onResearcherFailure: (id, _error) => {
       const sliceId = id === 'quick' ? state.quickSliceLabel : id;
       if (panelState.progress) {
-        const unitsPerResearcher = getUnitsPerResearcher();
+        const unitsPerResearcher = getUnitsPerResearcher(undefined, ctx.cachingActive);
         const current = progressCredits.get(id) ?? 0;
         const remaining = unitsPerResearcher - current;
         if (remaining > 0) {
@@ -430,7 +438,7 @@ export function createResearchObserver(
       } else {
         // Delegation: prepare for new round's search → researchers → steering acceptable
         if (plan?.researchers && plan.researchers.length > 0 && panelState.progress) {
-          const unitsPerResearcher = getUnitsPerResearcher();
+          const unitsPerResearcher = getUnitsPerResearcher(undefined, ctx.cachingActive);
           panelState.progress.expected += (plan.researchers.length * unitsPerResearcher) + LEAD_EVAL_UNITS;
         }
         panelState.steeringAcceptable = true;