pi-research 1.1.0 → 1.2.0

package/README.md

@@ -1,22 +1,36 @@
 # pi-research
 
+![pi-research logo](docs/assets/pi-research-logo.png)
+
 [![npm version](https://img.shields.io/npm/v/pi-research?color=blue)](https://www.npmjs.com/package/pi-research)
 [![tests](https://img.shields.io/badge/tests-56%2F56-brightgreen)](https://github.com/endgegnerbert-tech/pi-research)
 [![Pi package](https://img.shields.io/badge/pi-package-blueviolet)](https://pi.ai)
 
-`pi-research` is a Pi extension for fast, local-first web research inside the agent.
+`pi-research` is a Pi extension for grounded web research.
+It searches, ranks, compares, and synthesizes sources inside the agent.
 
-It searches the live web, ranks sources, reads the most relevant pages, and synthesizes a grounded answer with citations.
-It does **not** require an external research API or API key, and it is not a browser automation tool.
+![community packs](docs/assets/pi-research-community.png)
 
 ## Why it exists
 
-Agents usually need two things to answer well:
+When agents answer well, they usually do three things:
+
+1. search the right places
+2. prefer authoritative sources
+3. explain confidence and gaps clearly
 
-1. a way to search the web efficiently
-2. a way to turn sources into a usable answer
+`pi-research` does that without an external research service.
 
-`pi-research` does both inside Pi, so the agent can research topics without relying on a separate hosted research service.
+## Best practices
+
+- use `fast` for short factual lookups
+- use `deep` for comparisons, conflicts, or unclear questions
+- use `code` for docs, repos, README-driven answers, and snippets
+- use `academic` for paper-heavy topics
+- set `options.requireAuthoritative: true` when source quality matters more than recall
+- use `options.format: json` when you need machine-readable output
+- add `options.files` when local docs matter
+- keep questions specific; vague prompts create noisy retrieval
 
 ## What it does
 
@@ -26,7 +40,7 @@ Agents usually need two things to answer well:
 - follows up when the first pass is not enough
 - extracts code blocks for code-focused questions
 - supports local files as additional sources
-- returns a structured result with citations and confidence metadata
+- returns structured results with citations, confidence, conflicts, and gaps
 
 ## What it is not
 
@@ -34,22 +48,6 @@ Agents usually need two things to answer well:
 - not an offline knowledge base
 - not a replacement for page navigation
 
-## Install
-
-### For Pi
-
-```bash
-pi install npm:pi-research
-```
-
-### For npm-based workflows
-
-```bash
-npm install pi-research
-```
-
-GitHub repository: https://github.com/endgegnerbert-tech/pi-research
-
 ## Quick start
 
 ```text
@@ -57,11 +55,11 @@ What are the trade-offs between B-trees and LSM-trees?
 ```
 
 ```text
-Show me the best way to add health checks to Docker Compose.
+Compare React Server Components with traditional SSR.
 ```
 
 ```text
-Compare React Server Components with traditional SSR.
+How do I add retries to a Node.js fetch wrapper?
 ```
 
 ## Modes
@@ -73,6 +71,26 @@ Compare React Server Components with traditional SSR.
 | `code` | docs, READMEs, repositories, and code snippets |
 | `academic` | scholarly sources and paper-heavy topics |
 
+## Output
+
+The tool returns structured data including:
+
+- `answer`
+- `bullets`
+- `sources`
+- `citations`
+- `codeBlocks`
+- `confidence`
+- `confidenceScore`
+- `sufficient`
+- `authoritativeSourcesFound`
+- `openSubQuestions`
+- `missingAspects`
+- `conflictSummary`
+- `unverifiedClaims`
+- `sourceTypes`
+- `meta`
+
 ## Public tool parameters
 
 - `query` — research question to answer
@@ -133,44 +151,10 @@ options:
     - ./docs/spec.md
 ```
 
-## Output
-
-The tool returns structured data including:
-
-- `answer`
-- `bullets`
-- `sources`
-- `citations`
-- `codeBlocks`
-- `confidence`
-- `confidenceScore`
-- `sufficient`
-- `authoritativeSourcesFound`
-- `openSubQuestions`
-- `missingAspects`
-- `conflictSummary`
-- `unverifiedClaims`
-- `sourceTypes`
-- `meta`
-
-## How it works
-
-- **query-isolated caching**: repeated identical research can be skipped when the previous result was already sufficient
-- **source scoring**: official docs, READMEs, papers, and local files are preferred over weak sources
-- **follow-up planning**: unclear or conflicting results trigger another round of research
-- **conflict detection**: opposing claims are surfaced explicitly
-- **fact checking**: unsupported answer sentences are marked as unverified
-- **local source input**: files can be added directly to the research context
-
-## Limits
-
-- it still depends on live web access for web research
-- it does not browse pages like a human user
-- it is not fully offline unless you only use local files
-- it is not a browser interaction tool
-
 ## Domain packs
 
+Built-in packs now steer routing and source selection:
+
 - `web`
 - `github`
 - `security`
@@ -183,9 +167,14 @@ The tool returns structured data including:
 
 ## Community packs
 
-You can add your own domain pack by copying `lib/domains/template.js`, adapting the `run()` function, and registering it in `lib/domains/index.js`.
+You can add your own domain pack without changing the core research engine:
+
+1. copy `lib/domains/template.js`
+2. implement your domain-specific `run(question, options)` logic
+3. register the pack in `lib/domains/index.js`
+4. add eval cases in `eval/cases/<your-domain>/`
 
-Minimal starter example:
+Starter example:
 
 ```js
 export default {
@@ -209,16 +198,92 @@ export default {
 
 Run `npm run eval` to execute the eval harness.
 
-## Package info
+## Install
 
-- Package name: `pi-research`
-- Entry point: `extensions/pi-research.ts`
-- Tool name: `pi-research`
-- License: MIT
+### Pi Coding Agent — extension
+
+Existing Pi users should keep installing the main package:
+
+```bash
+pi install npm:pi-research
+```
+
+This registers the Pi extension and keeps the public tool name `pi-research`.
+
+### MCP-only — any agent
+
+Run the MCP server directly from npm:
+
+```bash
+npx -y pi-research
+```
+
+The MCP server identifies itself as `unblind-mcp`, but the tool it exposes is still named `pi-research`.
+
+### Global MCP install
+
+```bash
+npm install -g pi-research
+unblind-mcp
+```
+
+The global install also provides `pi-research` as a CLI alias for the same MCP server:
+
+```bash
+pi-research
+```
+
+### Local development
+
+```bash
+node ./mcp/server.js
+```
+
+Convenience script:
+
+```bash
+npm run --silent mcp
+```
+
+Example MCP config:
+
+```json
+{
+  "mcpServers": {
+    "unblind-mcp": {
+      "command": "npx",
+      "args": ["-y", "pi-research"]
+    }
+  }
+}
+```
+
+Local path config:
+
+```json
+{
+  "mcpServers": {
+    "unblind-mcp": {
+      "command": "node",
+      "args": ["/path/to/pi-research/mcp/server.js"]
+    }
+  }
+}
+```
+
+Compatibility note: `mcp-server.js` remains as a deprecated root-level shim for older local configs.
+
+### Future `unblind-mcp` package
+
+A separate npm package named `unblind-mcp` can be added later as a tiny wrapper around `pi-research`. It should depend on `pi-research` and start the same MCP server, not duplicate the engine.
 
 ## Release notes
 
-- Pi install: `pi install npm:pi-research`
-- npm install: `npm install pi-research`
+- Package name: `pi-research`
+- Version: `1.1.2`
+- Entry point: `extensions/pi-research.ts`
+- MCP entry point: `mcp/server.js`
+- MCP compatibility shim: `mcp-server.js`
+- License: MIT
+- Third-party notices: `THIRD_PARTY_NOTICES.md`
 - GitHub: `https://github.com/endgegnerbert-tech/pi-research`
-- Community packs: copy the template pack and register it in `lib/domains/index.js`

package/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,17 @@
+# Third-Party Notices
+
+## Scrapling
+
+This project includes ideas and/or adapted implementation details from Scrapling.
+
+Copyright (c) 2024, Karim shoair
+
+BSD 3-Clause License
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

package/index.js

@@ -2,12 +2,13 @@ import { Type } from "typebox";
 
 import { compactResearchPayload, classifyQueryIntent, inferOfficialDocsSite } from "./lib/research.js";
 import { clearResearchMemory, hashResearchQuery, setResearchMemory, shouldSkipResearch } from "./lib/research-memory.js";
+import { logResearchEvent } from "./lib/local-logger.js";
 import { runWebResearch } from "./lib/web-research.js";
 
 const RESEARCH_STATE = new Map();
 
 function buildWebResearchGuidance() {
-  return "Use pi-research for web search and research. Prefer fast mode for simple questions, deep mode for comparisons or ambiguous cases, and code/academic modes when source type matters.";
+  return "Use pi-research for current facts, docs, best practices, comparisons, and citations. Search if unsure.";
 }
 
 function defaultMode(query) {
@@ -61,11 +62,17 @@ export default function webResearchExtension(pi) {
   pi.on("before_agent_start", async (event) => {
     RESEARCH_STATE.clear();
     clearResearchMemory();
+    await logResearchEvent("agent_start", {
+      systemPrompt: event.systemPrompt,
+      guidance: buildWebResearchGuidance(),
+    });
     return { systemPrompt: `${event.systemPrompt}\n\n${buildWebResearchGuidance()}` };
   });
 
   pi.on("tool_call", async (event) => {
     if (event.toolName !== "pi-research") return;
+    event.input ||= {};
+    const originalInput = { ...event.input };
     if (!event.input.mode) event.input.mode = defaultMode(event.input.query || "");
 
     const queryHash = hashResearchQuery(event.input.query || "");
@@ -73,9 +80,26 @@ export default function webResearchExtension(pi) {
     const mode = event.input.mode;
     const isolate = Boolean(event.input.isolate || process.env.RESEARCH_ISOLATE === "1");
     const force = Boolean(event.input.force);
+    let blocked = false;
+    let reason = "";
 
     if (shouldSkipResearch({ queryHash, lastHash: state.lastHash, lastWasSufficient: state.lastSufficient, force, isolate })) {
-      return { block: true, reason: "Recent pi-research result was already sufficient for this exact query." };
+      blocked = true;
+      reason = "Recent pi-research result was already sufficient for this exact query.";
+      await logResearchEvent("tool_call", {
+        originalInput,
+        finalInput: { ...event.input },
+        queryHash,
+        blocked,
+        reason,
+        state: {
+          count: state.count,
+          lastHash: state.lastHash,
+          lastSufficient: state.lastSufficient,
+          fastRecoveryAllowed: state.fastRecoveryAllowed,
+        },
+      });
+      return { block: true, reason };
     }
 
     if (mode === "fast" && state.count === 1 && state.fastRecoveryAllowed && !force && !isolate) {
@@ -85,34 +109,58 @@ export default function webResearchExtension(pi) {
 
     state.count += 1;
     state.lastHash = queryHash;
+    await logResearchEvent("tool_call", {
+      originalInput,
+      finalInput: { ...event.input },
+      queryHash,
+      blocked,
+      state: {
+        count: state.count,
+        lastHash: state.lastHash,
+        lastSufficient: state.lastSufficient,
+        fastRecoveryAllowed: state.fastRecoveryAllowed,
+      },
+    });
   });
 
   pi.on("tool_result", async (event) => {
-    if (event.toolName === "pi-research" && !event.isError && event.details?.ok) {
-      const queryHash = hashResearchQuery(event.input?.query || "");
-      const state = getState(queryHash);
-      state.lastHash = queryHash;
-      state.lastSufficient = Boolean(event.details.sufficient);
-      const query = event.input?.query || "";
-      state.fastRecoveryAllowed = !event.details.sufficient
-        && !event.details.authoritativeSourcesFound
-        && ["best_practice", "temporal", "definition"].includes(classifyQueryIntent(query || ""));
-      setResearchMemory(`last:${queryHash}`, event.details);
+    if (event.toolName === "pi-research") {
+      if (!event.isError && event.details?.ok) {
+        const queryHash = hashResearchQuery(event.input?.query || "");
+        const state = getState(queryHash);
+        state.lastHash = queryHash;
+        state.lastSufficient = Boolean(event.details.sufficient);
+        const query = event.input?.query || "";
+        state.fastRecoveryAllowed = !event.details.sufficient
+          && !event.details.authoritativeSourcesFound
+          && ["best_practice", "temporal", "definition"].includes(classifyQueryIntent(query || ""));
+        setResearchMemory(`last:${queryHash}`, event.details);
+      }
+      await logResearchEvent("tool_result", {
+        toolName: event.toolName,
+        isError: event.isError,
+        input: event.input,
+        details: event.details,
+      });
     }
     return compactWebResearchToolResult(event) || undefined;
   });
 
   pi.registerTool({
     name: "pi-research",
-    label: "Pi Research",
-    description: "Search and research the web.",
-    promptSnippet: "Use this for web research when needed.",
-    promptGuidelines: ["Use pi-research for search, source ranking, and summarization."],
+    label: "Web Research",
+    description: "Live sources, ranking, and cited answers.",
+    promptSnippet: "Use for current or uncertain answers with citations.",
+    promptGuidelines: [
+      "Use for current facts, docs, best practices, comparisons, and verification.",
+      "Search instead of guessing.",
+      "Pick fast, deep, code, or academic mode as needed.",
+    ],
     parameters: Type.Object({
-      query: Type.String({ description: "Research question to answer from the web" }),
-      mode: Type.Optional(Type.Union([Type.Literal("fast"), Type.Literal("deep"), Type.Literal("code"), Type.Literal("academic")], { description: "Research mode", default: "fast" })),
-      force: Type.Optional(Type.Boolean({ description: "Bypass sufficiency gating and cached answers for this call" })),
-      isolate: Type.Optional(Type.Boolean({ description: "Run this query in isolation without session/query cache reuse" })),
+      query: Type.String({ description: "Live web question" }),
+      mode: Type.Optional(Type.Union([Type.Literal("fast"), Type.Literal("deep"), Type.Literal("code"), Type.Literal("academic")], { description: "Mode", default: "fast" })),
+      force: Type.Optional(Type.Boolean({ description: "Ignore cache" })),
+      isolate: Type.Optional(Type.Boolean({ description: "No cache reuse" })),
       options: Type.Optional(Type.Object({
         allowedSources: Type.Optional(Type.Array(Type.String())),
         maxTurns: Type.Optional(Type.Number()),

package/lib/local-logger.js

@@ -0,0 +1,54 @@
+import { appendFile, mkdir } from "node:fs/promises";
+import { dirname, join } from "node:path";
+import { homedir } from "node:os";
+
+const LOG_PATH = process.env.PI_RESEARCH_LOG_PATH || join(homedir(), ".pi", "logs", "pi-research.jsonl");
+let writeChain = Promise.resolve();
+
+function sanitize(value, depth = 0, seen = new WeakSet()) {
+  if (value === null || value === undefined) return value;
+  if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") return value;
+  if (typeof value === "bigint") return value.toString();
+  if (typeof value === "function") return `[Function ${value.name || "anonymous"}]`;
+  if (value instanceof Date) return value.toISOString();
+  if (value instanceof RegExp) return value.toString();
+  if (value instanceof Error) {
+    return { name: value.name, message: value.message, stack: value.stack };
+  }
+  if (Array.isArray(value)) {
+    if (depth >= 6) return "[MaxDepth]";
+    return value.map((item) => sanitize(item, depth + 1, seen));
+  }
+  if (typeof value === "object") {
+    if (seen.has(value)) return "[Circular]";
+    if (depth >= 6) return "[MaxDepth]";
+    seen.add(value);
+    const output = {};
+    for (const [key, item] of Object.entries(value)) output[key] = sanitize(item, depth + 1, seen);
+    seen.delete(value);
+    return output;
+  }
+  return String(value);
+}
+
+export function getResearchLogPath() {
+  return LOG_PATH;
+}
+
+export async function logResearchEvent(type, data = {}) {
+  const record = {
+    ts: new Date().toISOString(),
+    pid: process.pid,
+    cwd: process.cwd(),
+    type,
+    data: sanitize(data),
+  };
+  const line = `${JSON.stringify(record)}\n`;
+  writeChain = writeChain
+    .then(async () => {
+      await mkdir(dirname(LOG_PATH), { recursive: true });
+      await appendFile(LOG_PATH, line, "utf8");
+    })
+    .catch(() => {});
+  return writeChain;
+}

package/lib/page-fetch-adapter.js

@@ -0,0 +1,180 @@
+import { spawn } from "node:child_process";
+import { fileURLToPath } from "node:url";
+import path from "node:path";
+
+const SCRAPLING_ROOT = fileURLToPath(new URL("../Scrapling", import.meta.url));
+const BLOCKED_PATTERNS = [
+  /cloudflare/i,
+  /turnstile/i,
+  /captcha/i,
+  /please enable cookies/i,
+  /bot detection/i,
+  /verify you are human/i,
+  /security check/i,
+];
+const DYNAMIC_PATTERNS = [
+  /__next_data__/i,
+  /__nuxt__/i,
+  /data-reactroot/i,
+  /hydrat/i,
+  /window\.__INITIAL_STATE__/i,
+  /id=["']app["']/i,
+  /id=["']root["']/i,
+];
+
+function stripHtml(value) {
+  return String(value || "")
+    .replace(/<script[\s\S]*?<\/script>/gi, " ")
+    .replace(/<style[\s\S]*?<\/style>/gi, " ")
+    .replace(/<noscript[\s\S]*?<\/noscript>/gi, " ")
+    .replace(/<[^>]+>/g, " ")
+    .replace(/&nbsp;/g, " ")
+    .replace(/\s+/g, " ")
+    .trim();
+}
+
+export function assessPageAttempt({ status = 200, body = "", contentType = "", url = "" } = {}) {
+  const text = String(body || "");
+  const plain = stripHtml(text);
+  const lower = `${text}\n${url}`.toLowerCase();
+  const antiBotSignal = BLOCKED_PATTERNS.some((pattern) => pattern.test(lower));
+  const blocked = status === 403 || status === 429 || (antiBotSignal && plain.length < 1000);
+  const dynamic = !blocked && (DYNAMIC_PATTERNS.some((pattern) => pattern.test(lower)) || (text.includes("<script") && plain.length < 400));
+  const weak = blocked || plain.length < 300 || (!/text\/(html|plain)/i.test(contentType) && plain.length < 500);
+
+  return {
+    blocked,
+    dynamic,
+    weak,
+    mode: blocked ? "stealthy" : dynamic ? "dynamic" : "async",
+    plainLength: plain.length,
+  };
+}
+
+export function chooseScraplingMode(input) {
+  return assessPageAttempt(input).mode;
+}
+
+function pythonScript() {
+  return String.raw`
+import asyncio
+import json
+import os
+import sys
+
+root = sys.argv[1]
+mode = sys.argv[2]
+url = sys.argv[3]
+payload = json.loads(sys.argv[4])
+
+sys.path.insert(0, root)
+
+async def main():
+    from scrapling.fetchers import AsyncFetcher, DynamicFetcher, StealthyFetcher
+
+    timeout = payload.get("timeout")
+    kwargs = {}
+    if timeout:
+        kwargs["timeout"] = timeout
+
+    if mode == "async":
+        response = await AsyncFetcher.get(url, **kwargs)
+    elif mode == "dynamic":
+        response = DynamicFetcher.fetch(url, **kwargs)
+    else:
+        response = StealthyFetcher.fetch(url, **kwargs)
+
+    headers = {}
+    raw_headers = getattr(response, "headers", None)
+    if hasattr(raw_headers, "items"):
+        headers = dict(raw_headers.items())
+    else:
+        try:
+            headers = dict(raw_headers or {})
+        except Exception:
+            headers = {}
+
+    body = getattr(response, "body", None)
+    if body is None:
+        candidate = getattr(response, "text", None)
+        body = candidate() if callable(candidate) else candidate
+
+    if isinstance(body, bytes):
+        body = body.decode("utf-8", "replace")
+    elif not isinstance(body, str):
+        body = str(body or "")
+
+    out = {
+        "ok": True,
+        "url": getattr(response, "url", url),
+        "status": getattr(response, "status", 200),
+        "contentType": headers.get("content-type", ""),
+        "body": body,
+        "headers": headers,
+    }
+    print(json.dumps(out))
+
+try:
+    asyncio.run(main())
+except Exception as exc:
+    print(json.dumps({"ok": False, "error": str(exc), "type": exc.__class__.__name__}))
+    sys.exit(1)
+`;
+}
+
+export async function fetchWithScrapling(url, mode, signal, config = {}) {
+  if (!mode) return null;
+
+  return await new Promise((resolve) => {
+    const child = spawn(process.env.PYTHON || "python3", ["-c", pythonScript(), SCRAPLING_ROOT, mode, url, JSON.stringify({ timeout: config.pageTimeoutMs || 30000 })], {
+      env: {
+        ...process.env,
+        PYTHONPATH: [SCRAPLING_ROOT, process.env.PYTHONPATH].filter(Boolean).join(path.delimiter),
+      },
+      stdio: ["ignore", "pipe", "pipe"],
+    });
+
+    let stdout = "";
+    let stderr = "";
+    child.stdout.on("data", (chunk) => {
+      stdout += chunk;
+    });
+    child.stderr.on("data", (chunk) => {
+      stderr += chunk;
+    });
+
+    const finish = (value) => {
+      if (!signal) return resolve(value);
+      if (signal.aborted) return resolve(null);
+      return resolve(value);
+    };
+
+    child.on("error", () => finish(null));
+    child.on("close", (code) => {
+      if (code !== 0) return finish(null);
+      try {
+        const parsed = JSON.parse(stdout.trim() || "{}");
+        if (!parsed.ok) return finish(null);
+        return finish(parsed);
+      } catch {
+        if (stderr) return finish(null);
+        return finish(null);
+      }
+    });
+
+    if (signal) {
+      const abort = () => {
+        child.kill("SIGKILL");
+        finish(null);
+      };
+      if (signal.aborted) abort();
+      else signal.addEventListener("abort", abort, { once: true });
+    }
+  });
+}
+
+export const pageFetchAdapter = {
+  assessPageAttempt,
+  chooseScraplingMode,
+  fetchWithScrapling,
+};

package/lib/web-research.js

@@ -35,6 +35,7 @@ import {
   scoreSourceEntry,
   selectRelevantChunks,
 } from "./research.js";
+import { pageFetchAdapter } from "./page-fetch-adapter.js";
 import { resolveOutputFormat, shouldRequireAuthoritativeSources } from "./research-output.js";
 import { planResearch } from "./planner.js";
 import {
@@ -45,6 +46,7 @@ import {
   setResearchMemory,
   writeCachedResult,
 } from "./research-memory.js";
+import { logResearchEvent } from "./local-logger.js";
 
 const USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36";
 const MIN_PAGE_TEXT = 300;
@@ -262,7 +264,8 @@ async function searchArxiv(query, signal, config) {
       const published = entry.match(/<published>([^<]+)<\/published>/)?.[1]?.slice(0, 10);
       return sourceFromPaper(title, url, summary, published);
     }).filter((item) => item.url && item.title);
-  } catch {
+  } catch (error) {
+    await logResearchEvent("search_error", { provider: "arxiv", query, error });
     return [];
   }
 }
@@ -272,7 +275,8 @@ async function searchSemanticScholar(query, signal, config) {
     const response = await fetchTextWithRetry(`https://api.semanticscholar.org/graph/v1/paper/search?query=${encodeURIComponent(query)}&limit=${config.resultsPerQuery}&fields=title,abstract,url,year`, signal, 2, {}, config.pageTimeoutMs);
     const data = await response.json();
     return (data?.data || []).map((item) => sourceFromPaper(item.title, item.url || `https://www.semanticscholar.org/search?q=${encodeURIComponent(item.title)}`, item.abstract || "", item.year ? `${item.year}-01-01` : null)).filter((item) => item.title);
-  } catch {
+  } catch (error) {
+    await logResearchEvent("search_error", { provider: "semanticscholar", query, error });
     return [];
   }
 }
@@ -287,7 +291,8 @@ async function searchCrossref(query, signal, config) {
       const publishDate = dateParts.length ? `${String(dateParts[0]).padStart(4, "0")}-${String(dateParts[1] || 1).padStart(2, "0")}-${String(dateParts[2] || 1).padStart(2, "0")}` : null;
       return sourceFromPaper(item.title?.[0] || "", doi, String(item.abstract || "").replace(/<[^>]+>/g, " "), publishDate);
     }).filter((item) => item.url && item.title);
-  } catch {
+  } catch (error) {
+    await logResearchEvent("search_error", { provider: "crossref", query, error });
     return [];
   }
 }
@@ -346,7 +351,11 @@ function shouldSkipUrl(url) {
 }
 
 function shouldUseJinaFirst(url) {
-  return /(^|\.)medium\.com$|(^|\.)dev\.to$|(^|\.)substack\.com$/i.test(new URL(url).hostname);
+  try {
+    return /(^|\.)medium\.com$|(^|\.)dev\.to$|(^|\.)substack\.com$/i.test(new URL(url).hostname);
+  } catch {
+    return false;
+  }
 }
 
 function pageFromText(title, url, text, config, extra = {}) {
@@ -377,7 +386,11 @@ function withinTimeframe(page, config) {
 }
 
 export async function fetchPageSource(url, signal, config = getResearchConfig()) {
-  if (shouldSkipUrl(url)) return null;
+  if (shouldSkipUrl(url)) {
+    await logResearchEvent("fetch_skip", { url, reason: "login_or_account_url" });
+    return null;
+  }
+  const adapter = config.fetchAdapter || pageFetchAdapter;
   const cacheKey = `${normalizeUrl(url)}::${config.pageTextLimit}::${JSON.stringify({
     preferRecent: config.preferRecent || false,
     minYear: config.minYear || "",
@@ -385,10 +398,20 @@ export async function fetchPageSource(url, signal, config = getResearchConfig())
     useJinaFallback: Boolean(config.useJinaFallback),
   })}`;
   const cached = config.isolate ? null : getCacheValue(pageCache, cacheKey);
-  if (cached) return cached;
+  if (cached) {
+    await logResearchEvent("fetch_cache_hit", { url, cacheKey, title: cached.title, textLength: cached.text?.length || 0 });
+    return cached;
+  }
+
+  await logResearchEvent("fetch_start", { url, cacheKey, config: { isolate: config.isolate, useJinaFallback: Boolean(config.useJinaFallback), pageTextLimit: config.pageTextLimit } });
+
   if (shouldUseJinaFirst(url)) {
     const first = await fetchJinaPageSource(url, signal, config);
-    return config.isolate ? first : setCacheValue(pageCache, cacheKey, first, PAGE_CACHE_TTL_MS);
+    if (first && withinTimeframe(first, config)) {
+      const page = config.isolate ? first : setCacheValue(pageCache, cacheKey, first, PAGE_CACHE_TTL_MS);
+      await logResearchEvent("fetch_end", { url, via: "jina_first", success: Boolean(page), page: page ? { title: page.title, sourceType: page.sourceType, publishDate: page.publishDate, textLength: page.text?.length || 0 } : null });
+      return page;
+    }
   }
 
   try {
@@ -398,22 +421,48 @@ export async function fetchPageSource(url, signal, config = getResearchConfig())
     }, config.pageTimeoutMs);
 
     const contentType = response.headers.get("content-type") || "";
-    if (!contentType.includes("text/html") && !contentType.includes("text/plain")) return null;
+    if (!contentType.includes("text/html") && !contentType.includes("text/plain")) {
+      await logResearchEvent("fetch_end", { url, success: false, reason: "unsupported_content_type", contentType });
+      return null;
+    }
 
     const body = await response.text();
     const snapshot = extractPageSnapshot(body, response.url || url);
-    const page = pageFromText(snapshot.title, snapshot.url, snapshot.text, config, {
+    let page = pageFromText(snapshot.title, snapshot.url, snapshot.text, config, {
       publishDate: extractPublishDate(body),
       sourceType: classifySourceType(snapshot.url, snapshot.title),
       codeBlocks: snapshot.codeBlocks,
     });
 
+    const assessment = adapter.assessPageAttempt?.({
+      status: response.status ?? 200,
+      body,
+      contentType,
+      url: response.url || url,
+    });
+
+    if ((!page && assessment?.weak) || assessment?.dynamic || assessment?.blocked) {
+      const scrapling = await adapter.fetchWithScrapling?.(url, assessment.mode, signal, config);
+      if (scrapling?.body) {
+        const scraplingSnapshot = extractPageSnapshot(scrapling.body, scrapling.url || url);
+        page = pageFromText(scraplingSnapshot.title, scraplingSnapshot.url, scraplingSnapshot.text, config, {
+          publishDate: extractPublishDate(scrapling.body),
+          sourceType: classifySourceType(scraplingSnapshot.url, scraplingSnapshot.title),
+          codeBlocks: scraplingSnapshot.codeBlocks,
+        });
+      }
+    }
+
     const resolved = page || await fetchJinaPageSource(url, signal, config);
     const finalPage = resolved && withinTimeframe(resolved, config) ? resolved : null;
-    return config.isolate ? finalPage : setCacheValue(pageCache, cacheKey, finalPage, PAGE_CACHE_TTL_MS);
-  } catch {
+    const stored = config.isolate ? finalPage : setCacheValue(pageCache, cacheKey, finalPage, PAGE_CACHE_TTL_MS);
+    await logResearchEvent("fetch_end", { url, success: Boolean(stored), page: stored ? { title: stored.title, sourceType: stored.sourceType, publishDate: stored.publishDate, textLength: stored.text?.length || 0 } : null });
+    return stored;
+  } catch (error) {
     const fallback = await fetchJinaPageSource(url, signal, config);
-    return config.isolate ? fallback : setCacheValue(pageCache, cacheKey, fallback, PAGE_CACHE_TTL_MS);
+    const stored = config.isolate ? fallback : setCacheValue(pageCache, cacheKey, fallback, PAGE_CACHE_TTL_MS);
+    await logResearchEvent("fetch_error", { url, error, fallback: stored ? { title: stored.title, sourceType: stored.sourceType, publishDate: stored.publishDate, textLength: stored.text?.length || 0 } : null });
+    return stored;
   }
 }
 
@@ -427,9 +476,10 @@ async function readLocalFiles(paths, config) {
         publishDate: null,
         local: true,
       });
+      await logResearchEvent("local_file_read", { path, success: Boolean(page), textLength: text.length, page: page ? { title: page.title, textLength: page.text.length } : null });
       if (page) pages.push(page);
-    } catch {
-      // ignore unreadable files
+    } catch (error) {
+      await logResearchEvent("local_file_error", { path, error });
     }
   }
   return pages;
@@ -455,6 +505,7 @@ function fallbackSynthesis(query, pages) {
 }
 
 export async function synthesizeResearch(query, pages, ctx, signal) {
+  await logResearchEvent("synthesis_start", { query, pages: pages.map((page) => ({ title: page.title, url: page.url, sourceType: page.sourceType, textLength: page.text?.length || 0 })) });
   const prompt = [
     "You are a concise research synthesizer.",
     "Answer only from the provided sources.",
@@ -489,19 +540,23 @@ export async function synthesizeResearch(query, pages, ctx, signal) {
           score: typeof pages[id - 1].score === "number" ? pages[id - 1].score : scoreSourceEntry(pages[id - 1], query).total,
           authoritative: typeof pages[id - 1].authoritative === "boolean" ? pages[id - 1].authoritative : scoreSourceEntry(pages[id - 1], query).authoritative,
         })), query);
-        return {
+        const result = {
           answer: parsed.answer.trim(),
           bullets: parsed.bullets.map((item) => String(item).trim()).filter(Boolean).slice(0, 5),
           sources,
           citations: Array.isArray(parsed.citations) ? parsed.citations.slice(0, 8) : sources.map((source) => ({ text: source.title, sourceIndex: source.number || 0 })),
         };
+        await logResearchEvent("synthesis_end", { query, result });
+        return result;
       }
     }
   } catch {
     // fall through
   }
 
-  return fallbackSynthesis(query, pages);
+  const fallback = fallbackSynthesis(query, pages);
+  await logResearchEvent("synthesis_end", { query, result: fallback, fallback: true });
+  return fallback;
 }
 
 function planSubqueries(rootQuery, currentQuery, config, sufficiency) {
@@ -536,17 +591,28 @@ export async function runWebResearch(query, ctx, signal, onUpdate, mode = "fast"
   const config = getResearchConfig(typeof mode === "object" ? { ...mode, domain } : { mode, domain });
   const cacheKey = modeCacheKey(query, config);
 
+  await logResearchEvent("research_start", { query, mode: config.mode, domain, config });
+
   if (!config.isolate && !config.force) {
     const memoryHit = getResearchMemory(cacheKey);
-    if (memoryHit) return memoryHit;
+    if (memoryHit) {
+      await logResearchEvent("research_cache_hit", { query, cacheKey, source: "memory" });
+      await logResearchEvent("research_end", { ...memoryHit, cacheHit: true, cacheSource: "memory" });
+      return memoryHit;
+    }
     const persistentHit = readCachedResult(cacheKey);
     if (persistentHit) {
       setResearchMemory(cacheKey, persistentHit);
+      await logResearchEvent("research_cache_hit", { query, cacheKey, source: "disk" });
+      await logResearchEvent("research_end", { ...persistentHit, cacheHit: true, cacheSource: "disk" });
       return persistentHit;
     }
   }
 
-  const emit = (stage, text) => onUpdate?.({ content: [{ type: "text", text: `[pipeline:${stage}] ${text}` }] });
+  const emit = (stage, text) => {
+    void logResearchEvent("pipeline_stage", { query, stage, text });
+    return onUpdate?.({ content: [{ type: "text", text: `[pipeline:${stage}] ${text}` }] });
+  };
   const startedAt = Date.now();
   const seenUrls = new Set();
   const seenContentHashes = new Set();
@@ -585,6 +651,11 @@ export async function runWebResearch(query, ctx, signal, onUpdate, mode = "fast"
     emit("search", `Searching ${queriesThisTurn.length} queries...`);
 
     const searchGroups = await Promise.all(queriesThisTurn.map((subquery) => searchDuckDuckGo(subquery, signal, config)));
+    await logResearchEvent("search_results", {
+      query,
+      queries: queriesThisTurn,
+      results: searchGroups.flat().map((result) => ({ title: result.title, url: result.url, snippet: result.snippet, sourceType: result.sourceType, publishDate: result.publishDate })),
+    });
     const results = rankSearchResults(searchGroups.flat(), query, config.maxPages * 2, config)
       .filter((result) => {
         const key = normalizeUrl(result.url);
@@ -596,6 +667,11 @@ export async function runWebResearch(query, ctx, signal, onUpdate, mode = "fast"
 
     emit("fetch", `Reading ${results.length} sources...`);
     const pageCandidates = await Promise.all(results.map((result) => fetchPageSource(result.url, signal, config)));
+    await logResearchEvent("page_fetch_results", {
+      query,
+      urls: results.map((result) => result.url),
+      pages: pageCandidates.filter(Boolean).map((page) => ({ title: page.title, url: page.url, sourceType: page.sourceType, publishDate: page.publishDate, textLength: page.text?.length || 0 })),
+    });
     const rankedPages = rankFetchedPages(pageCandidates.filter(Boolean).map((page) => {
       const scored = scoreSourceEntry(page, query);
       return {
@@ -647,7 +723,7 @@ export async function runWebResearch(query, ctx, signal, onUpdate, mode = "fast"
   }
 
   if (mergedPages.length === 0) {
-    return {
+    const emptyResult = {
       ok: false,
       action: "web_research",
       query,
@@ -657,6 +733,8 @@ export async function runWebResearch(query, ctx, signal, onUpdate, mode = "fast"
       openSubQuestions: buildFallbackQueries(query),
       error: "No readable web sources were retrieved.",
     };
+    await logResearchEvent("research_end", emptyResult);
+    return emptyResult;
   }
 
   emit("synthesis", `Synthesizing ${mergedPages.length} sources...`);
@@ -726,6 +804,7 @@ export async function runWebResearch(query, ctx, signal, onUpdate, mode = "fast"
 
   setResearchMemory(cacheKey, result);
   writeCachedResult(cacheKey, result, config.cacheTtlMs);
+  await logResearchEvent("research_end", result);
   return result;
 }
 

package/mcp/server.js

@@ -0,0 +1,242 @@
+#!/usr/bin/env node
+
+import { realpathSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+
+import pkg from "../package.json" with { type: "json" };
+import { classifyQueryIntent } from "../lib/research.js";
+import { runWebResearch } from "../lib/web-research.js";
+
+const SERVER_NAME = "unblind-mcp";
+const TOOL_NAME = "pi-research";
+
+function buildWebResearchGuidance() {
+  return "Use pi-research for current facts, docs, best practices, comparisons, and citations. Search if unsure.";
+}
+
+function defaultMode(query) {
+  const intent = classifyQueryIntent(query);
+  if (intent === "comparison" || intent === "comparative") return "deep";
+  if (intent === "academic") return "academic";
+  return "fast";
+}
+
+function buildToolDefinition() {
+  return {
+    name: TOOL_NAME,
+    description: "Live sources, ranking, and cited answers.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        query: { type: "string", description: "Live web question" },
+        mode: {
+          type: "string",
+          enum: ["fast", "deep", "code", "academic"],
+          description: "Mode",
+        },
+        force: { type: "boolean", description: "Ignore cache" },
+        isolate: { type: "boolean", description: "No cache reuse" },
+        options: {
+          type: "object",
+          properties: {
+            allowedSources: { type: "array", items: { type: "string" } },
+            maxTurns: { type: "number" },
+            maxSites: { type: "number" },
+            requireAuthoritative: { type: "boolean" },
+            minYear: { type: "number" },
+            maxYear: { type: "number" },
+            preferRecent: { type: "boolean" },
+            files: { type: "array", items: { type: "string" } },
+            format: {
+              type: "string",
+              enum: ["markdown", "json", "table", "latex"],
+            },
+            deepResearchConfig: {
+              type: "object",
+              properties: {
+                depth: { type: "number", enum: [1, 2, 3] },
+                breadth: { type: "number", enum: [2, 3, 4] },
+                concurrency: { type: "number", enum: [1, 2, 3, 4] },
+              },
+              additionalProperties: false,
+            },
+          },
+          additionalProperties: false,
+        },
+      },
+      required: ["query"],
+      additionalProperties: false,
+    },
+  };
+}
+
+function buildInitializeResult(protocolVersion) {
+  return {
+    protocolVersion: protocolVersion || "2025-03-26",
+    capabilities: {
+      tools: {},
+    },
+    serverInfo: {
+      name: SERVER_NAME,
+      version: pkg.version,
+    },
+    instructions: buildWebResearchGuidance(),
+  };
+}
+
+function buildToolResult(payload) {
+  const text = payload?.ok ? (payload.contentText || JSON.stringify(payload, null, 2)) : JSON.stringify(payload, null, 2);
+  return {
+    content: [{ type: "text", text }],
+    structuredContent: payload,
+    isError: !payload?.ok,
+  };
+}
+
+async function runResearchTool(params = {}, run = runWebResearch) {
+  const mode = params.mode ?? defaultMode(params.query || "");
+  const payload = await run(params.query || "", undefined, undefined, undefined, {
+    mode,
+    force: params.force,
+    isolate: params.isolate,
+    ...(params.options || {}),
+  });
+  return buildToolResult(payload);
+}
+
+function jsonRpcError(id, code, message, data) {
+  const error = { code, message };
+  if (data !== undefined) error.data = data;
+  return { jsonrpc: "2.0", id, error };
+}
+
+async function handleMcpRequest(message, deps = {}) {
+  const run = deps.runWebResearchFn || runWebResearch;
+
+  if (!message || typeof message !== "object") {
+    return jsonRpcError(null, -32600, "Invalid Request");
+  }
+
+  if (typeof message.method !== "string") {
+    return jsonRpcError(message.id ?? null, -32600, "Invalid Request");
+  }
+
+  if (message.method === "notifications/initialized") return null;
+
+  try {
+    if (message.method === "initialize") {
+      return {
+        jsonrpc: "2.0",
+        id: message.id,
+        result: buildInitializeResult(message.params?.protocolVersion),
+      };
+    }
+
+    if (message.method === "tools/list") {
+      return {
+        jsonrpc: "2.0",
+        id: message.id,
+        result: { tools: [buildToolDefinition()] },
+      };
+    }
+
+    if (message.method === "tools/call") {
+      const params = message.params || {};
+      if (params.name !== TOOL_NAME) {
+        return jsonRpcError(message.id ?? null, -32602, `Unknown tool: ${String(params.name || "")}`);
+      }
+      const toolResult = await runResearchTool(params.arguments || {}, run);
+      return {
+        jsonrpc: "2.0",
+        id: message.id,
+        result: toolResult,
+      };
+    }
+
+    return jsonRpcError(message.id ?? null, -32601, `Method not found: ${message.method}`);
+  } catch (error) {
+    const text = error instanceof Error ? error.stack || error.message : String(error);
+    return {
+      jsonrpc: "2.0",
+      id: message.id ?? null,
+      result: {
+        content: [{ type: "text", text }],
+        isError: true,
+      },
+    };
+  }
+}
+
+export function startMcpServer({ input = process.stdin, output = process.stdout, errorOutput = process.stderr, runWebResearchFn = runWebResearch } = {}) {
+  function send(message) {
+    const json = JSON.stringify(message);
+    output.write(`Content-Length: ${Buffer.byteLength(json, "utf8")}\r\n\r\n${json}`);
+  }
+
+  let buffer = Buffer.alloc(0);
+
+  function pump() {
+    while (true) {
+      const headerEnd = buffer.indexOf("\r\n\r\n");
+      if (headerEnd === -1) return;
+
+      const headerText = buffer.slice(0, headerEnd).toString("utf8");
+      const match = headerText.match(/content-length:\s*(\d+)/i);
+      if (!match) {
+        buffer = buffer.slice(headerEnd + 4);
+        continue;
+      }
+
+      const length = Number(match[1]);
+      const bodyStart = headerEnd + 4;
+      const bodyEnd = bodyStart + length;
+      if (buffer.length < bodyEnd) return;
+
+      const bodyText = buffer.slice(bodyStart, bodyEnd).toString("utf8");
+      buffer = buffer.slice(bodyEnd);
+      if (!bodyText.trim()) continue;
+
+      let message;
+      try {
+        message = JSON.parse(bodyText);
+      } catch (error) {
+        errorOutput.write(`${String(error)}\n`);
+        continue;
+      }
+
+      void handleMcpRequest(message, { runWebResearchFn }).then((response) => {
+        if (response) send(response);
+      }).catch((error) => {
+        const text = error instanceof Error ? error.stack || error.message : String(error);
+        send({ jsonrpc: "2.0", id: message?.id ?? null, result: { content: [{ type: "text", text }], isError: true } });
+      });
+    }
+  }
+
+  input.on("data", (chunk) => {
+    buffer = Buffer.concat([buffer, chunk]);
+    pump();
+  });
+
+  input.on("end", () => {
+    process.exitCode = 0;
+  });
+}
+
+function isMainModule(metaUrl) {
+  if (!process.argv[1]) return false;
+  return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(metaUrl));
+}
+
+if (isMainModule(import.meta.url)) {
+  startMcpServer();
+}
+
+export {
+  buildInitializeResult,
+  buildToolDefinition,
+  buildToolResult,
+  defaultMode,
+  handleMcpRequest,
+  runResearchTool,
+};

package/mcp-server.js

@@ -0,0 +1,18 @@
+#!/usr/bin/env node
+
+import { realpathSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+
+import { startMcpServer } from "./mcp/server.js";
+
+export * from "./mcp/server.js";
+
+function isMainModule(metaUrl) {
+  if (!process.argv[1]) return false;
+  return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(metaUrl));
+}
+
+if (isMainModule(import.meta.url)) {
+  process.stderr.write("mcp-server.js is deprecated; use mcp/server.js instead.\n");
+  startMcpServer();
+}

package/package.json

@@ -1,16 +1,23 @@
 {
   "name": "pi-research",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "private": false,
   "type": "module",
   "description": "Pi extension for web research.",
   "license": "MIT",
   "main": "./index.js",
+  "bin": {
+    "pi-research": "./mcp/server.js",
+    "unblind-mcp": "./mcp/server.js"
+  },
   "files": [
     "extensions",
     "index.js",
     "lib",
+    "mcp",
+    "mcp-server.js",
     "README.md",
+    "THIRD_PARTY_NOTICES.md",
     "package.json"
   ],
   "repository": {
@@ -26,7 +33,8 @@
   ],
   "scripts": {
     "test": "node --test",
-    "eval": "node --test test/eval-runner.test.js"
+    "eval": "node --test test/eval-runner.test.js",
+    "mcp": "node ./mcp/server.js"
   },
   "dependencies": {
     "@mariozechner/pi-ai": "*",