pi-research 1.1.0 → 1.1.2

package/README.md

@@ -1,22 +1,36 @@
 # pi-research
 
+![pi-research logo](docs/assets/pi-research-logo.png)
+
 [![npm version](https://img.shields.io/npm/v/pi-research?color=blue)](https://www.npmjs.com/package/pi-research)
 [![tests](https://img.shields.io/badge/tests-56%2F56-brightgreen)](https://github.com/endgegnerbert-tech/pi-research)
 [![Pi package](https://img.shields.io/badge/pi-package-blueviolet)](https://pi.ai)
 
-`pi-research` is a Pi extension for fast, local-first web research inside the agent.
+`pi-research` is a Pi extension for grounded web research.
+It searches, ranks, compares, and synthesizes sources inside the agent.
 
-It searches the live web, ranks sources, reads the most relevant pages, and synthesizes a grounded answer with citations.
-It does **not** require an external research API or API key, and it is not a browser automation tool.
+![community packs](docs/assets/pi-research-community.png)
 
 ## Why it exists
 
-Agents usually need two things to answer well:
+When agents answer well, they usually do three things:
+
+1. search the right places
+2. prefer authoritative sources
+3. explain confidence and gaps clearly
+
+`pi-research` does that without an external research service.
 
-1. a way to search the web efficiently
-2. a way to turn sources into a usable answer
+## Best practices
 
-`pi-research` does both inside Pi, so the agent can research topics without relying on a separate hosted research service.
+- use `fast` for short factual lookups
+- use `deep` for comparisons, conflicts, or unclear questions
+- use `code` for docs, repos, README-driven answers, and snippets
+- use `academic` for paper-heavy topics
+- set `options.requireAuthoritative: true` when source quality matters more than recall
+- use `options.format: json` when you need machine-readable output
+- add `options.files` when local docs matter
+- keep questions specific; vague prompts create noisy retrieval
 
 ## What it does
 
@@ -26,7 +40,7 @@ Agents usually need two things to answer well:
 - follows up when the first pass is not enough
 - extracts code blocks for code-focused questions
 - supports local files as additional sources
-- returns a structured result with citations and confidence metadata
+- returns structured results with citations, confidence, conflicts, and gaps
 
 ## What it is not
 
@@ -34,22 +48,6 @@ Agents usually need two things to answer well:
 - not an offline knowledge base
 - not a replacement for page navigation
 
-## Install
-
-### For Pi
-
-```bash
-pi install npm:pi-research
-```
-
-### For npm-based workflows
-
-```bash
-npm install pi-research
-```
-
-GitHub repository: https://github.com/endgegnerbert-tech/pi-research
-
 ## Quick start
 
 ```text
@@ -57,11 +55,11 @@ What are the trade-offs between B-trees and LSM-trees?
 ```
 
 ```text
-Show me the best way to add health checks to Docker Compose.
+Compare React Server Components with traditional SSR.
 ```
 
 ```text
-Compare React Server Components with traditional SSR.
+How do I add retries to a Node.js fetch wrapper?
 ```
 
 ## Modes
@@ -73,6 +71,26 @@ Compare React Server Components with traditional SSR.
 | `code` | docs, READMEs, repositories, and code snippets |
 | `academic` | scholarly sources and paper-heavy topics |
 
+## Output
+
+The tool returns structured data including:
+
+- `answer`
+- `bullets`
+- `sources`
+- `citations`
+- `codeBlocks`
+- `confidence`
+- `confidenceScore`
+- `sufficient`
+- `authoritativeSourcesFound`
+- `openSubQuestions`
+- `missingAspects`
+- `conflictSummary`
+- `unverifiedClaims`
+- `sourceTypes`
+- `meta`
+
 ## Public tool parameters
 
 - `query` — research question to answer
@@ -133,44 +151,10 @@ options:
     - ./docs/spec.md
 ```
 
-## Output
-
-The tool returns structured data including:
-
-- `answer`
-- `bullets`
-- `sources`
-- `citations`
-- `codeBlocks`
-- `confidence`
-- `confidenceScore`
-- `sufficient`
-- `authoritativeSourcesFound`
-- `openSubQuestions`
-- `missingAspects`
-- `conflictSummary`
-- `unverifiedClaims`
-- `sourceTypes`
-- `meta`
-
-## How it works
-
-- **query-isolated caching**: repeated identical research can be skipped when the previous result was already sufficient
-- **source scoring**: official docs, READMEs, papers, and local files are preferred over weak sources
-- **follow-up planning**: unclear or conflicting results trigger another round of research
-- **conflict detection**: opposing claims are surfaced explicitly
-- **fact checking**: unsupported answer sentences are marked as unverified
-- **local source input**: files can be added directly to the research context
-
-## Limits
-
-- it still depends on live web access for web research
-- it does not browse pages like a human user
-- it is not fully offline unless you only use local files
-- it is not a browser interaction tool
-
 ## Domain packs
 
+Built-in packs now steer routing and source selection:
+
 - `web`
 - `github`
 - `security`
@@ -183,9 +167,14 @@ The tool returns structured data including:
 
 ## Community packs
 
-You can add your own domain pack by copying `lib/domains/template.js`, adapting the `run()` function, and registering it in `lib/domains/index.js`.
+You can add your own domain pack without changing the core research engine:
+
+1. copy `lib/domains/template.js`
+2. implement your domain-specific `run(question, options)` logic
+3. register the pack in `lib/domains/index.js`
+4. add eval cases in `eval/cases/<your-domain>/`
 
-Minimal starter example:
+Starter example:
 
 ```js
 export default {
@@ -209,16 +198,25 @@ export default {
 
 Run `npm run eval` to execute the eval harness.
 
-## Package info
+## Install
 
-- Package name: `pi-research`
-- Entry point: `extensions/pi-research.ts`
-- Tool name: `pi-research`
-- License: MIT
+### For Pi
+
+```bash
+pi install npm:pi-research
+```
+
+### For npm-based workflows
+
+```bash
+npm install pi-research
+```
 
 ## Release notes
 
-- Pi install: `pi install npm:pi-research`
-- npm install: `npm install pi-research`
+- Package name: `pi-research`
+- Version: `1.1.1`
+- Entry point: `extensions/pi-research.ts`
+- License: MIT
+- Third-party notices: `THIRD_PARTY_NOTICES.md`
 - GitHub: `https://github.com/endgegnerbert-tech/pi-research`
-- Community packs: copy the template pack and register it in `lib/domains/index.js`

package/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,17 @@
+# Third-Party Notices
+
+## Scrapling
+
+This project includes ideas and/or adapted implementation details from Scrapling.
+
+Copyright (c) 2024, Karim shoair
+
+BSD 3-Clause License
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

package/index.js

@@ -7,7 +7,7 @@ import { runWebResearch } from "./lib/web-research.js";
 const RESEARCH_STATE = new Map();
 
 function buildWebResearchGuidance() {
-  return "Use pi-research for web search and research. Prefer fast mode for simple questions, deep mode for comparisons or ambiguous cases, and code/academic modes when source type matters.";
+  return "Use pi-research for current facts, docs, best practices, comparisons, and citations. Search if unsure.";
 }
 
 function defaultMode(query) {
@@ -104,15 +104,19 @@ export default function webResearchExtension(pi) {
 
   pi.registerTool({
     name: "pi-research",
-    label: "Pi Research",
-    description: "Search and research the web.",
-    promptSnippet: "Use this for web research when needed.",
-    promptGuidelines: ["Use pi-research for search, source ranking, and summarization."],
+    label: "Web Research",
+    description: "Live sources, ranking, and cited answers.",
+    promptSnippet: "Use for current or uncertain answers with citations.",
+    promptGuidelines: [
+      "Use for current facts, docs, best practices, comparisons, and verification.",
+      "Search instead of guessing.",
+      "Pick fast, deep, code, or academic mode as needed.",
+    ],
     parameters: Type.Object({
-      query: Type.String({ description: "Research question to answer from the web" }),
-      mode: Type.Optional(Type.Union([Type.Literal("fast"), Type.Literal("deep"), Type.Literal("code"), Type.Literal("academic")], { description: "Research mode", default: "fast" })),
-      force: Type.Optional(Type.Boolean({ description: "Bypass sufficiency gating and cached answers for this call" })),
-      isolate: Type.Optional(Type.Boolean({ description: "Run this query in isolation without session/query cache reuse" })),
+      query: Type.String({ description: "Live web question" }),
+      mode: Type.Optional(Type.Union([Type.Literal("fast"), Type.Literal("deep"), Type.Literal("code"), Type.Literal("academic")], { description: "Mode", default: "fast" })),
+      force: Type.Optional(Type.Boolean({ description: "Ignore cache" })),
+      isolate: Type.Optional(Type.Boolean({ description: "No cache reuse" })),
       options: Type.Optional(Type.Object({
         allowedSources: Type.Optional(Type.Array(Type.String())),
         maxTurns: Type.Optional(Type.Number()),

package/lib/page-fetch-adapter.js

@@ -0,0 +1,180 @@
+import { spawn } from "node:child_process";
+import { fileURLToPath } from "node:url";
+import path from "node:path";
+
+const SCRAPLING_ROOT = fileURLToPath(new URL("../Scrapling", import.meta.url));
+const BLOCKED_PATTERNS = [
+  /cloudflare/i,
+  /turnstile/i,
+  /captcha/i,
+  /please enable cookies/i,
+  /bot detection/i,
+  /verify you are human/i,
+  /security check/i,
+];
+const DYNAMIC_PATTERNS = [
+  /__next_data__/i,
+  /__nuxt__/i,
+  /data-reactroot/i,
+  /hydrat/i,
+  /window\.__INITIAL_STATE__/i,
+  /id=["']app["']/i,
+  /id=["']root["']/i,
+];
+
+function stripHtml(value) {
+  return String(value || "")
+    .replace(/<script[\s\S]*?<\/script>/gi, " ")
+    .replace(/<style[\s\S]*?<\/style>/gi, " ")
+    .replace(/<noscript[\s\S]*?<\/noscript>/gi, " ")
+    .replace(/<[^>]+>/g, " ")
+    .replace(/&nbsp;/g, " ")
+    .replace(/\s+/g, " ")
+    .trim();
+}
+
+export function assessPageAttempt({ status = 200, body = "", contentType = "", url = "" } = {}) {
+  const text = String(body || "");
+  const plain = stripHtml(text);
+  const lower = `${text}\n${url}`.toLowerCase();
+  const antiBotSignal = BLOCKED_PATTERNS.some((pattern) => pattern.test(lower));
+  const blocked = status === 403 || status === 429 || (antiBotSignal && plain.length < 1000);
+  const dynamic = !blocked && (DYNAMIC_PATTERNS.some((pattern) => pattern.test(lower)) || (text.includes("<script") && plain.length < 400));
+  const weak = blocked || plain.length < 300 || (!/text\/(html|plain)/i.test(contentType) && plain.length < 500);
+
+  return {
+    blocked,
+    dynamic,
+    weak,
+    mode: blocked ? "stealthy" : dynamic ? "dynamic" : "async",
+    plainLength: plain.length,
+  };
+}
+
+export function chooseScraplingMode(input) {
+  return assessPageAttempt(input).mode;
+}
+
+function pythonScript() {
+  return String.raw`
+import asyncio
+import json
+import os
+import sys
+
+root = sys.argv[1]
+mode = sys.argv[2]
+url = sys.argv[3]
+payload = json.loads(sys.argv[4])
+
+sys.path.insert(0, root)
+
+async def main():
+    from scrapling.fetchers import AsyncFetcher, DynamicFetcher, StealthyFetcher
+
+    timeout = payload.get("timeout")
+    kwargs = {}
+    if timeout:
+        kwargs["timeout"] = timeout
+
+    if mode == "async":
+        response = await AsyncFetcher.get(url, **kwargs)
+    elif mode == "dynamic":
+        response = DynamicFetcher.fetch(url, **kwargs)
+    else:
+        response = StealthyFetcher.fetch(url, **kwargs)
+
+    headers = {}
+    raw_headers = getattr(response, "headers", None)
+    if hasattr(raw_headers, "items"):
+        headers = dict(raw_headers.items())
+    else:
+        try:
+            headers = dict(raw_headers or {})
+        except Exception:
+            headers = {}
+
+    body = getattr(response, "body", None)
+    if body is None:
+        candidate = getattr(response, "text", None)
+        body = candidate() if callable(candidate) else candidate
+
+    if isinstance(body, bytes):
+        body = body.decode("utf-8", "replace")
+    elif not isinstance(body, str):
+        body = str(body or "")
+
+    out = {
+        "ok": True,
+        "url": getattr(response, "url", url),
+        "status": getattr(response, "status", 200),
+        "contentType": headers.get("content-type", ""),
+        "body": body,
+        "headers": headers,
+    }
+    print(json.dumps(out))
+
+try:
+    asyncio.run(main())
+except Exception as exc:
+    print(json.dumps({"ok": False, "error": str(exc), "type": exc.__class__.__name__}))
+    sys.exit(1)
+`;
+}
+
+export async function fetchWithScrapling(url, mode, signal, config = {}) {
+  if (!mode) return null;
+
+  return await new Promise((resolve) => {
+    const child = spawn(process.env.PYTHON || "python3", ["-c", pythonScript(), SCRAPLING_ROOT, mode, url, JSON.stringify({ timeout: config.pageTimeoutMs || 30000 })], {
+      env: {
+        ...process.env,
+        PYTHONPATH: [SCRAPLING_ROOT, process.env.PYTHONPATH].filter(Boolean).join(path.delimiter),
+      },
+      stdio: ["ignore", "pipe", "pipe"],
+    });
+
+    let stdout = "";
+    let stderr = "";
+    child.stdout.on("data", (chunk) => {
+      stdout += chunk;
+    });
+    child.stderr.on("data", (chunk) => {
+      stderr += chunk;
+    });
+
+    const finish = (value) => {
+      if (!signal) return resolve(value);
+      if (signal.aborted) return resolve(null);
+      return resolve(value);
+    };
+
+    child.on("error", () => finish(null));
+    child.on("close", (code) => {
+      if (code !== 0) return finish(null);
+      try {
+        const parsed = JSON.parse(stdout.trim() || "{}");
+        if (!parsed.ok) return finish(null);
+        return finish(parsed);
+      } catch {
+        if (stderr) return finish(null);
+        return finish(null);
+      }
+    });
+
+    if (signal) {
+      const abort = () => {
+        child.kill("SIGKILL");
+        finish(null);
+      };
+      if (signal.aborted) abort();
+      else signal.addEventListener("abort", abort, { once: true });
+    }
+  });
+}
+
+export const pageFetchAdapter = {
+  assessPageAttempt,
+  chooseScraplingMode,
+  fetchWithScrapling,
+};

package/lib/web-research.js

@@ -35,6 +35,7 @@ import {
   scoreSourceEntry,
   selectRelevantChunks,
 } from "./research.js";
+import { pageFetchAdapter } from "./page-fetch-adapter.js";
 import { resolveOutputFormat, shouldRequireAuthoritativeSources } from "./research-output.js";
 import { planResearch } from "./planner.js";
 import {
@@ -346,7 +347,11 @@ function shouldSkipUrl(url) {
 }
 
 function shouldUseJinaFirst(url) {
-  return /(^|\.)medium\.com$|(^|\.)dev\.to$|(^|\.)substack\.com$/i.test(new URL(url).hostname);
+  try {
+    return /(^|\.)medium\.com$|(^|\.)dev\.to$|(^|\.)substack\.com$/i.test(new URL(url).hostname);
+  } catch {
+    return false;
+  }
 }
 
 function pageFromText(title, url, text, config, extra = {}) {
@@ -378,6 +383,7 @@ function withinTimeframe(page, config) {
 
 export async function fetchPageSource(url, signal, config = getResearchConfig()) {
   if (shouldSkipUrl(url)) return null;
+  const adapter = config.fetchAdapter || pageFetchAdapter;
   const cacheKey = `${normalizeUrl(url)}::${config.pageTextLimit}::${JSON.stringify({
     preferRecent: config.preferRecent || false,
     minYear: config.minYear || "",
@@ -386,9 +392,12 @@ export async function fetchPageSource(url, signal, config = getResearchConfig())
   })}`;
   const cached = config.isolate ? null : getCacheValue(pageCache, cacheKey);
   if (cached) return cached;
+
   if (shouldUseJinaFirst(url)) {
     const first = await fetchJinaPageSource(url, signal, config);
-    return config.isolate ? first : setCacheValue(pageCache, cacheKey, first, PAGE_CACHE_TTL_MS);
+    if (first && withinTimeframe(first, config)) {
+      return config.isolate ? first : setCacheValue(pageCache, cacheKey, first, PAGE_CACHE_TTL_MS);
+    }
   }
 
   try {
@@ -402,12 +411,31 @@ export async function fetchPageSource(url, signal, config = getResearchConfig())
 
     const body = await response.text();
     const snapshot = extractPageSnapshot(body, response.url || url);
-    const page = pageFromText(snapshot.title, snapshot.url, snapshot.text, config, {
+    let page = pageFromText(snapshot.title, snapshot.url, snapshot.text, config, {
       publishDate: extractPublishDate(body),
       sourceType: classifySourceType(snapshot.url, snapshot.title),
       codeBlocks: snapshot.codeBlocks,
     });
 
+    const assessment = adapter.assessPageAttempt?.({
+      status: response.status ?? 200,
+      body,
+      contentType,
+      url: response.url || url,
+    });
+
+    if ((!page && assessment?.weak) || assessment?.dynamic || assessment?.blocked) {
+      const scrapling = await adapter.fetchWithScrapling?.(url, assessment.mode, signal, config);
+      if (scrapling?.body) {
+        const scraplingSnapshot = extractPageSnapshot(scrapling.body, scrapling.url || url);
+        page = pageFromText(scraplingSnapshot.title, scraplingSnapshot.url, scraplingSnapshot.text, config, {
+          publishDate: extractPublishDate(scrapling.body),
+          sourceType: classifySourceType(scraplingSnapshot.url, scraplingSnapshot.title),
+          codeBlocks: scraplingSnapshot.codeBlocks,
+        });
+      }
+    }
+
     const resolved = page || await fetchJinaPageSource(url, signal, config);
     const finalPage = resolved && withinTimeframe(resolved, config) ? resolved : null;
     return config.isolate ? finalPage : setCacheValue(pageCache, cacheKey, finalPage, PAGE_CACHE_TTL_MS);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-research",
-  "version": "1.1.0",
+  "version": "1.1.2",
   "private": false,
   "type": "module",
   "description": "Pi extension for web research.",
@@ -11,6 +11,7 @@
     "index.js",
     "lib",
     "README.md",
+    "THIRD_PARTY_NOTICES.md",
     "package.json"
   ],
   "repository": {