@lincoln504/pi-research 1.2.0 → 1.3.1

package/docs/CHANGELOG.md

@@ -5,6 +5,66 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.1] - 2026-08-06
+
+A dedicated post-release investigation pass over 1.3.0 — three independent
+agents (fresh-eyes code review, packaging/release-artifact audit, isolated
+runtime CLI smoke test) plus a full from-scratch local gate run — to confirm
+the shipped artifact and codebase were correct before considering the
+release closed. The packaging audit and runtime smoke test found nothing;
+the code review found one real, currently-live defect, fixed below.
+
+### Fixed
+
+- **The pi-extension research tool could report a cancelled-but-partial run as an ordinary success.** `DeepResearchOrchestrator.run()` does not throw on cancellation once at least one researcher has produced a report — it resolves normally with a fallback synthesis (the same documented contract 1.3.0's `cmdResearch` fix was built around). `research-tool-definition.ts`'s abort check only lived in its `catch` block, so this resolves-normally path fell straight through to unconditional success handling: metrics recorded `status: 'success'`, and the text handed back to the calling agent carried no indication it was a partial report. This matters more here than at the CLI: `AgentToolResult`'s `details` field is logs/UI-only by its own type contract, so `content[].text` is the *only* channel the calling agent has to learn a report is partial rather than complete — a silent miscategorization here could lead an agent to treat cut-short research as a finished answer. Same bug class as the 1.3.0 CLI fix, at a call site that round's review missed. Fixed by checking the abort signal immediately after `runResearch()` resolves: the metrics status now reflects the real outcome, and a cancellation note is prepended to the report text itself.
+
+Not a minor bump: unlike 1.3.0's `cmdResearch` fix, this does not touch the CLI's exit-code contract — it is scoped entirely to the pi-extension tool call path.
+
+## [1.3.0] - 2026-08-06
+
+Bundles three unreleased fix rounds that accumulated on top of 1.2.0 (each
+already pushed to main but never cut), plus a final fresh-eyes review pass
+done specifically to gate this release: an independent reviewer read every
+line of the combined diff against 1.2.0 with no knowledge of why any of it
+was written, and found several real defects the rounds that produced the
+code had missed — two of them the most severe in this release. Every item
+below was re-verified against the current code (not the original bug
+report) before being counted as fixed; the two highest-severity findings
+were personally reproduced (one as an actual OOM crash) before dispatching
+a fix.
+
+A minor (not patch) version for the same reason 1.2.0 was: the range
+changes the cancellation exit-code contract again, in the same category
+the last release documented as minor-worthy. A cancelled research run that
+had already collected at least one report used to report plain success
+(`ok: true`, exit `0`) because the orchestrator's fallback-synthesis path
+resolves normally on cancellation rather than throwing — `cmdResearch`'s
+success branch never consulted the cancellation latch, only its error
+branch did. It now exits `128 + signal` (or `130` for a programmatic
+abort) and sets `cancelled: true` in `--json`, matching every other
+cancellation-reporting path fixed in 1.2.0.
+
+### Fixed
+
+**Found in this release's own fresh-eyes pass (new, on top of the three prior rounds):**
+- **The chunker's own safety invariant was false — `overlap < targetSize` does not, in fact, prevent infinite loops.** Reproduced directly: `new Chunker({targetSize: 5, overlap: 4}).chunk(...)` on text containing a boundary-straddling emoji crashed the worker process with an out-of-memory error rather than merely running slowly. The UTF-16 surrogate-pair backoff (added earlier in this range) can land on the exact fixed point the forward-progress guard already enforces for `end`, so neither `start` nor `end` advances between iterations and the same chunk is pushed forever. A matching, milder gap let a lone high surrogate ship uncorrected into stored/embedded text whenever backing off would have violated that same guard. Both closed: the surrogate fix now grows past the low surrogate instead of silently doing nothing when shrinking would re-collide, and a new forward-progress invariant on `start` — mirroring the existing one on `end` — guarantees real per-iteration progress independent of any heuristic interaction, not just the ones already traced by hand. A brute-force sweep across small targetSize/overlap/emoji-position combinations backs the fix, not just the one reproducing triple.
+- **Uninstall's "is this directory exclusively ours" check was a bare name match, not an ownership check.** `isPiResearchNamespaced()` treated any path segment literally equal to `pi-research` as sufficient proof a `PI_RESEARCH_STATE_DIR` override was safe to delete wholesale — but naming a directory after the tool pointed at it is a completely ordinary thing to do (e.g. organizing several tools' state under `~/agents-state/<tool-name>/`), and nothing stops a user from keeping unrelated content in that same folder. The one regression test guarding this path only ever planted pi-research-owned entries, never foreign content, so it could never have caught this. Now only the exact default `<dirName>/research` layout — something the runtime actually constructs itself and can therefore prove is exclusively its own — authorizes wholesale removal; every other override, however named, only ever loses the specific entries the runtime is known to create there.
+- **A researcher could still run a full billed session after its run was told to stop.** The three fast-stop checkpoints added earlier in this range (top-of-loop, initial-links loop, post-backoff sleep) left one window uncovered: between `createResearcherSession()` resolving and the session being registered for `abortAllSessions()`'s visibility, a few lines later behind an awaited call. A stop tripping in exactly that gap was invisible to the abort scan, and the caller's bounded ~10s wait for self-stopping researchers elapsed before the session's own LLM call and scrape burst finished — the run returned and released its slot while the researcher kept going in the background. A fourth checkpoint immediately after registration closes it.
+- **A "FIX" comment described a circuit-breaker key format that has never existed in production.** `research-cleanup.ts` cleared session breakers assuming they're keyed `${piSessionId}-<8hex>`, but every real caller keys them by the plain `researchId` (generated independently of the pi session) — the function that would produce the assumed format has no production caller at all. The clear was consequently a no-op, masked on the normal run-completion path by a second, correctly-keyed clear elsewhere; the gap only mattered when this was the sole cleanup path (a throw before the orchestrator ever runs). Fixed to clear by the real key; the now-unused prefix-clearing utility this bug motivated, along with its own comment repeating the same wrong premise, is removed rather than left as bait for the same mistake again.
+- **A follower's healthcheck request couldn't be cancelled the way search/scrape already can.** The abort-signal plumbing added earlier in this range for search/scrape was never extended to healthcheck: the interface type carries no `signal` parameter and the one production caller always calls without one — currently harmless, but silently ready to reintroduce "follower cancellation burns a leader pool slot to timeout" the moment any caller starts passing a real signal, with no compiler error to flag it. Wired through, mirroring search/scrape exactly.
+- **Routine follower-cancellation was logged and counted as a server error.** The same abort-signal work introduced rejections shaped like `"Task search aborted while running"`, which the pool-shutdown error classifier doesn't recognize — so every cancelled in-flight browser task (a normal, frequent event whenever a research run is cancelled) logged at ERROR and inflated `browser_search_errors_total`/`browser_scrape_errors_total` before the layer that correctly demotes it ever saw the error. Each of the three call sites now checks the actual signal the rejection is a consequence of, rather than pattern-matching text — deliberately not widened inside the shared classifier, which a different caller also uses to detect a genuine leader handover.
+- **Severity filtering for a security-database search normalized only one of three backends.** `getSeverityParam()` was applied on the NVD leg only; GitHub and OSV read the raw, unvalidated value directly — reproducing, for two of three backends, the exact "unrecognized value silently filters everything to zero" bug an earlier round in this range fixed for NVD. Masked today because the sole production caller does its own separate, stricter validation first. Now normalized once, ahead of all three legs.
+- A flaky CI failure surfaced during this release's own gate run traced to a test relying on a fixed 50ms sleep to assume a lock had been fully acquired before tearing it down — on a loaded runner, teardown could occasionally win the race and land in a different, already-correctly-handled code path than the one the test meant to exercise. Replaced with a real "critical section has started" signal; no production code was at fault.
+
+**From the first post-1.2.0 audit round** (orchestration, browser, embedding/state, knowledge, CLI/config, utils, tools, SDK/planning — ~40 defects): aborted researchers restarting as untracked zombies (an `=== 'Aborted'` string match missed the actual sentinel shape); PDF content zeroed over cluster IPC and the resulting extraction error cached as a successful scrape; the stale-profile sweep touching arbitrary shared-tmp entries outside its own namespace; cluster workers leaking the research query into `ps` output; `clearEmbeddingServer` gaining compare-and-delete semantics (a mere follower's clean shutdown could deregister a live leader); `getEmbedder` refusing to join an in-flight init for a different model/device; a code-fence-straddling chunk boundary emitting one overlap-sliver per character; an explicit `--config` file's keys previously applying nowhere when shadowed by the ambient overlay; CLI flags silently swallowing their own values (`--model --json` running against the literal model `"--json"`); nested embedder-init stdio capture losing LIFO ordering on a mid-capture cancel; the scrape context gate billing pre-slice batch counts; StackExchange rate-limit/quota state not surviving across tool calls; and NVD severity filtering not normalizing case or the MODERATE→MEDIUM synonym.
+
+**From the second post-1.2.0 audit round** (browser/web-research, orchestration/SDK, knowledge/embedding, CLI/config/install — ~38 defects, including 2 regressions in the first round's own diff caught by that round's fresh-eyes pass): browser scrapes never checking HTTP status, so 4xx/5xx error pages were cached as citable content; the 100MB PDF / 25MB HTML size caps enforced only after the worker had already buffered and IPC'd the full body; search failures always hitting the global circuit breaker instead of the run's own; a follower's cancelled run burning its leader's pool slot to full timeout instead of aborting on socket close; fast-stop not reaching a researcher still acquiring links, sleeping between attempts, or rebuilding its session; the inter-attempt backoff sleep being non-abortable; a poisoned embedding leader staying eligible to re-elect onto GPU in the same process; NaN embedding vectors laundering to 0 across the HTTP hop; the full-text-search skip-guard's version bookkeeping missing rows committed mid-rebuild; a missing/unreadable explicit `--config` file being silently ignored instead of erroring; duplicate SIGINT delivery (process-group plus launcher-forwarded) hitting a consumed one-shot handler and force-exiting mid-cleanup; and the uninstall script's original, pre-this-release ownership gap for `PI_RESEARCH_STATE_DIR` (the one further hardened above).
+
+**From the backlog-clearing round** (re-auditing every item deferred since v1.0.7 plus both rounds above): the cancellation-reporting gap described above; `knowledge --json` success payloads not carrying `ok: true` despite claiming parity with the error shape, and `knowledge-config` having no `--json` error handling at all; the SSRF IPv6 gate missing `fec0::/10` and the NAT64 well-known prefix `64:ff9b::/96` (a real bypass on hosts whose egress traverses a NAT64 gateway, reachable via `64:ff9b::169.254.169.254`); steering messages already consumed by a finished run leaking into the next unrelated run's system-prompt injection; `clearBrowserServer` gaining the same compare-and-delete semantics its embedding-leader sibling already had; a second, independent chunker surrogate-pair gap (raw UTF-16 boundaries with no pair awareness); and steering-extended round budgets being invisible to the round-phase prompt logic (`PlanningService` recomputed `maxRounds` from the base complexity table only). `HealthCheckService` — zero runtime consumers, fully superseded by the live `src/healthcheck/registry.ts` system — was deleted.
+
+### Changed
+- **A cancelled-but-partially-successful research run now reports its true status.** See above — this is the same exit-code contract 1.2.0 established, extended to a success-path case that fix didn't reach.
+
 ## [1.2.0] - 2026-08-05
 
 Clears the medium/low findings deferred out of the 1.1.2 audit, then a second
@@ -256,20 +316,73 @@ and the host-version self-check gained the tested-ceiling warning.
 
 ---
 
-## Deferred Findings (from v1.0.7 audit)
-
-The following issues were identified during the v1.0.7 audit but intentionally deferred to future releases:
-
-1. **KnowledgeStore.close() timeout behavior** - The close() method proceeds even when the pending-write wait times out. A correct fix requires threading cancellation tokens through write paths or documenting the 10s timeout as intended behavior.
-
-2. **Cross-session teardown via shared singletons** - Session shutdown handlers run process-wide cleanup for reload/new/fork events. The correct behavior depends on pi's session model semantics and needs further investigation.
-
-3. **Lower-severity hardening items** - Various low-confidence or low-impact findings noted for future investigation:
-   - SIGINT-adjacent cancellation race
-   - Knowledge-store migration vs. concurrent access
-   - Global activity gate reused for per-session decisions
-   - Zombie queued browser tasks after timeout
-   - Non-atomic `clearBrowserServer`
-   - Inconsistent `knowledge --json` payload shapes
-   - Backward-clock-step tolerance across multiple rate limiters
+## Deferred Findings — cleared 2026-08-06
+
+The v1.0.7-era list below was re-audited against the current tree, three weeks
+and many fix rounds after it was written. Every item resolved one of two ways:
+already fixed as a side effect of unrelated work since, or fixed directly in
+this pass with a regression test. None remain open.
+
+- **KnowledgeStore.close() timeout behavior** — already safe: writes register
+  into the active-write set before the close-in-progress check can be seen,
+  and LanceDB table handles keep working after the owning connection closes,
+  so a timed-out close no longer strands or corrupts an in-flight write.
+- **Cross-session teardown via shared singletons** — not reachable: pi fully
+  serializes a session's teardown before the next one is created, so shared
+  singletons are never disposed out from under a still-live session.
+- **SIGINT-adjacent cancellation race** — real gap, fixed: a cancelled run
+  that had already produced a report returned through the orchestrator's
+  normal (non-throwing) partial-synthesis path, so the CLI's success branch
+  never saw the cancellation latch and reported `ok:true`/exit 0 for a run
+  the user had stopped. The success path now checks the latch and reports
+  the same `cancelled: true` / `128+signal` contract the error path already
+  used.
+- **Knowledge-store migration vs. concurrent access** — already safe: a
+  cross-process file lock serializes migrations, and every read/write
+  re-resolves its table handle, so a migration racing another process's
+  handle fails that operation cleanly instead of serving stale data.
+- **Global activity gate reused for per-session decisions** — real gap,
+  fixed: steering messages already consumed by a finished research run
+  survived in session state and leaked into the next, unrelated run's
+  system-prompt injection and final-report attribution. Only genuinely
+  still-queued messages now survive a run boundary.
+- **Zombie queued browser tasks after timeout** — already fixed (in the
+  first post-1.2.0 audit round): a queued task's client-side timeout now
+  actually removes it from the scheduler's queue instead of leaving it to
+  be dispatched later.
+- **Non-atomic `clearBrowserServer`** — real gap, fixed: the file write
+  itself was already atomic, but the read-decide-clear sequence around it
+  was not — a dead-leader restart path could wipe a legitimately new
+  leader's registration elected in the same window. `clearBrowserServer`
+  now takes an optional expected-owner argument and compares-and-deletes
+  inside the lock, mirroring the embedding leader's existing CAS.
+- **Inconsistent `knowledge --json` payload shapes** — real gap, fixed: the
+  success payload never carried `ok:true` despite a comment claiming parity
+  with the error shape, and `knowledge-config`'s failure path had no
+  try/catch at all and fell through to a plain-text-only handler even under
+  `--json`. Both now emit a consistent, documented shape.
+- **Backward-clock-step tolerance across multiple rate limiters** — already
+  safe: every limiter's timing arithmetic leans toward waiting longer on a
+  negative clock delta, never toward a burst or a permanent wedge.
+
+Two further items surfaced and closed in the same pass, found via later
+audit rounds' own deferred lists rather than the v1.0.7 one:
+- **SSRF: `isPrivateIpv6` missed `fec0::/10`** (deprecated site-local) and
+  **`64:ff9b::/96`** (the NAT64 well-known prefix — actively deployed today,
+  and a real bypass vector on a host whose egress traverses a NAT64
+  gateway). Both ranges are now blocked.
+- **Chunk boundaries could split a UTF-16 surrogate pair**, corrupting one
+  character (renders as U+FFFD) in stored/embedded text at an unlucky
+  boundary — the same class of bug already handled in three other places
+  in the codebase, now handled here too, on both the truncation and the
+  overlap-derived start boundary.
+- **Steering-extended round budgets weren't visible to the round-phase
+  prompt.** When mid-run steering extended the round cap beyond the base
+  per-complexity value, the planning service still computed guidance
+  against the base value, producing an impossible "Round 4 of 3" and,
+  more consequentially, prematurely selecting LATE-phase framing during
+  rounds steering had explicitly unlocked.
+
+One dead-code cleanup: `HealthCheckService` (a superseded, zero-consumer
+predecessor of the live `src/healthcheck/registry.ts` system) was removed.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lincoln504/pi-research",
-  "version": "1.2.0",
+  "version": "1.3.1",
   "description": "A set of web research tools (search, scrape, and others) and research agent system that uses them safely and efficiently.",
   "type": "module",
   "private": false,

package/scripts/cleanup.cjs

@@ -16,7 +16,7 @@
  */
 
 const { spawnSync } = require('child_process');
-const { rmSync, existsSync, lstatSync, readlinkSync, readFileSync, writeFileSync, unlinkSync } = require('fs');
+const { rmSync, existsSync, lstatSync, readdirSync, readlinkSync, readFileSync, writeFileSync, unlinkSync } = require('fs');
 const os = require('os');
 const path = require('path');
 
@@ -181,9 +181,71 @@ const rawDirName = process.env.PI_RESEARCH_CONFIG_DIR_NAME;
 const dirName = rawDirName && !rawDirName.includes('/') && !rawDirName.includes('\\') && rawDirName !== '.' && rawDirName !== '..'
   ? rawDirName
   : '.pi';
+
+/**
+ * Is this path PROVABLY equivalent to the layout the runtime itself would have
+ * constructed by default (`<dirName>/research`)? Wholesale rm of a state-dir
+ * override is only safe when the answer is yes: PI_RESEARCH_STATE_DIR may point
+ * at a SHARED directory, and rm'ing the override itself (unlike the cache sweep,
+ * which is namespaced via join(cacheHome, 'pi-research')) deleted everything the
+ * user kept beside our state.
+ *
+ * Deliberately NOT a bare "does any segment literally say pi-research" check —
+ * that matches naming convention, not ownership. A user is entirely free to
+ * point the override at, say, `~/agents-state/pi-research/` while keeping other
+ * tools' notes alongside it in the same folder (a natural way to organize
+ * several tools' state by name); a substring match would treat that shared
+ * folder as exclusively ours and wholesale-delete it. Only the exact default
+ * layout is something WE constructed and can therefore prove is ours.
+ */
+function isPiResearchNamespaced(p) {
+  const segs = p.split(/[\\/]+/).filter(Boolean);
+  for (let i = 0; i < segs.length - 1; i++) {
+    if (segs[i] === dirName && segs[i + 1] === 'research') return true;
+  }
+  return false;
+}
+
+/**
+ * Targeted state sweep for a non-namespaced override dir: delete ONLY the
+ * entries the runtime actually creates there and leave everything else — and
+ * the directory itself — untouched. The entry list mirrors the writers:
+ *   research-state.json + research-state-<hex>.tmp   (state-manager)
+ *   .locks/, backups/                                 (state-path-configuration)
+ *   project-settings.json + its .lock / .tmp. / .lock.stale- companions (config.ts)
+ *   research-run-slot-<i>.lock                        (research-run-semaphore)
+ * Best-effort; never throws.
+ */
+function removeOwnedStateEntries(dir) {
+  if (!dir || !existsSync(dir)) return;
+  let entries;
+  try { entries = readdirSync(dir); } catch { return; }
+  const owned = (name) =>
+    name === 'research-state.json' ||
+    (name.startsWith('research-state-') && name.endsWith('.tmp')) ||
+    name === '.locks' ||
+    name === 'backups' ||
+    name === 'project-settings.json' ||
+    name.startsWith('project-settings.json.') ||
+    /^research-run-slot-\d+\.lock$/.test(name);
+  for (const name of entries) {
+    if (!owned(name)) continue;
+    try {
+      rmSync(path.join(dir, name), { recursive: true, force: true, maxRetries: 5, retryDelay: 50 });
+    } catch (error) {
+      console.warn(`pi-research: could not remove state entry ${path.join(dir, name)}: ${error instanceof Error ? error.message : String(error)}`);
+    }
+  }
+  console.log(`pi-research: removed pi-research state entries from ${dir} (directory preserved — not a pi-research-namespaced path).`);
+}
+
 const stateDir = stateOverride && path.isAbsolute(stateOverride)
   ? stateOverride
   : path.join(os.homedir(), dirName, 'research', 'state');
-removeOwnedScratchDir('state dir', stateDir);
+if (stateOverride && path.isAbsolute(stateOverride) && !isPiResearchNamespaced(stateDir)) {
+  removeOwnedStateEntries(stateDir);
+} else {
+  removeOwnedScratchDir('state dir', stateDir);
+}
 
 process.exit(0);

package/skills/pi-research/SKILL.md

@@ -2,7 +2,7 @@
 name: pi-research
 description: "Multi-agent web research for any coding agent. Use for anything requiring current information from the internet — news, trends, product/service/company facts, statistics, people, places, events, APIs, library behavior, or 'what is X / how does X work' questions. Also use to search a local knowledge store of previously researched findings before doing live research. Powered by the pi-research engine (search, stealth scraping, security databases, Stack Exchange). Not for reading local files, running project commands, or analyzing this repository's own source."
 license: MIT
-metadata: { "author": "Lincoln Deen", "version": "1.2.0", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
+metadata: { "author": "Lincoln Deen", "version": "1.3.1", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
 allowed-tools: Bash(node:*)
 ---
 

package/src/cleanup/research-cleanup.ts

@@ -82,11 +82,19 @@ export function createCleanupFunction(
     const { resetLogger } = await import('../logger.ts');
     resetLogger(researchId);
     
-    // FIX (#10): Clear session circuit breaker to prevent unbounded map growth.
+    // FIX (#10): Clear the session's browser circuit breaker to prevent unbounded
+    // map growth. Breakers are keyed by the plain researchId — createResearchRunId()
+    // generates it (`run-<8 hex>`) independently of piSessionId in
+    // research-tool-definition.ts, and tools/scrape.ts / tools/search.ts thread that
+    // same bare researchId into getBrowserCircuitBreaker() as BrowserTask.sessionId.
+    // (The `${piSessionId}-<8 hex>` shape this comment used to assume is produced
+    // only by generateSessionId(), which has no production caller.) This duplicates
+    // the clearSessionCircuitBreaker(targetId) call research-orchestration-service.ts
+    // already makes in its own finally block on the normal run-completion path; it
+    // matters here on paths where this cleanup runs alone, e.g. a throw before
+    // orchestrator.run() is ever reached.
     const { clearSessionCircuitBreaker } = await import('../infrastructure/browser/browser-error-utils.ts');
-    if (piSessionId) {
-      clearSessionCircuitBreaker(piSessionId);
-    }
+    clearSessionCircuitBreaker(researchId);
     
     const activePanels = getPiActivePanels(piSessionId);
     if (activePanels.length === 0) {