@apmantza/greedysearch-pi 1.6.5 → 1.7.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## v1.6.5 (2026-04-04)
+
+### Security
+- **Private URL blocking** — Added validation to block requests to localhost, RFC1918 private addresses (10.x, 192.168.x), and .local/.internal domains. Prevents accidental exposure of internal services.
+
+### Features
+- **GitHub URL rewriting** — GitHub blob URLs (`github.com/owner/repo/blob/...`) are automatically rewritten to `raw.githubusercontent.com` for faster, cleaner raw file access.
+- **GitHub repo cloning** — Root and tree URLs now trigger `git clone --depth 1` for complete repo access. Agent can explore files locally instead of parsing rendered HTML. Includes README preview and directory tree listing.
+- **Head+tail content trimming** — Large documents now use smart truncation: keeps 75% from the beginning (introduction) + 25% from the end (conclusions/examples) with `[...content trimmed...]` marker, instead of simple truncation.
+- **Anubis bot detection** — Added detection for the new Anubis proof-of-work anti-bot system (`protected by anubis`, `anubis uses a proof-of-work`).
+
+### Fixes
+- **Perplexity clipboard retry** — Added single retry with 2s delay when clipboard extraction fails, improving reliability.
+
 ## v1.6.4 (2026-04-02)
 
 ### Fixes

package/README.md

@@ -1,14 +1,12 @@
 # GreedySearch for Pi
 
-Pi extension that adds `greedy_search`, `deep_research`, and `coding_task` tools — multi-engine AI search via browser automation. **NO API KEYS needed.**
+Pi extension that adds `greedy_search`, `deep_research`, and `coding_task` tools -- multi-engine AI search via browser automation. **NO API KEYS needed.**
 
-Fans out queries to Perplexity, Bing Copilot, and Google AI simultaneously. Returns AI-synthesized answers with deduped sources. Streams progress as each engine completes.
+Fans out queries to Perplexity, Bing Copilot, and Google AI simultaneously. Returns AI-synthesized answers with fetched source content. Streams progress as each engine completes.
 
-Forked from [GreedySearch-claude](https://github.com/apmantza/GreedySearch-claude).
-
-## Quick Note
+**New in v2.0:** HTTP-first source fetching with Mozilla Readability extraction (~3x faster), smart query-aware source ranking.
 
-**No API keys required** — this tool uses Chrome DevTools Protocol (CDP) to interact with search engines directly through a browser. It launches its own isolated Chrome instance, so it won't interfere with your main browser session.
+Forked from [GreedySearch-claude](https://github.com/apmantza/GreedySearch-claude).
 
 ## Install
 
@@ -24,10 +22,14 @@ pi install git:github.com/apmantza/GreedySearch-pi
 
 ## Quick Start
 
-Once installed, Pi gains a `greedy_search` tool with three depth levels.
+Once installed, Pi gains a `greedy_search` tool with two modes.
 
-```
-greedy_search({ query: "What's new in React 19?", depth: "standard" })
+```javascript
+// Default: multi-engine + source fetch + synthesis
+ Greedy_search({ query: "What's new in React 19?" })
+
+// Fast: single engine, no synthesis
+ greedy_search({ query: "What's new in React 19?", depth: "fast", engine: "perplexity" })
 ```
 
 ## Parameters
@@ -36,22 +38,25 @@ greedy_search({ query: "What's new in React 19?", depth: "standard" })
 |-----------|------|---------|-------------|
 | `query` | string | required | The search question |
 | `engine` | string | `"all"` | `all`, `perplexity`, `bing`, `google`, `gemini` |
-| `depth` | string | `"standard"` | `fast` (1 engine), `standard` (3 engines + synthesis), `deep` (3 + fetch + synthesis + confidence) |
+| `depth` | string | `"standard"` | `fast` (1 engine, no fetch), `standard` (3 engines + fetch + synthesis) |
 | `fullAnswer` | boolean | `false` | Return complete answer (~3000+ chars) vs truncated preview (~300 chars) |
 
 ## Depth Levels
 
 | Depth | Engines | Synthesis | Source Fetch | Time | Best For |
 |-------|---------|-----------|--------------|------|----------|
-| `fast` | 1 | — | — | 15-30s | Quick lookup, single perspective |
-| `standard` | 3 | ✅ | — | 30-90s | Default — balanced speed/quality |
-| `deep` | 3 | ✅ | ✅ (top 5) | 60-180s | Research that matters — architecture decisions |
+| `fast` | 1 | no | no | 10-30s | Quick lookup, single perspective |
+| `standard` | 3 | yes | yes (top 5) | 15-30s | **Default** -- balanced, grounded answers |
 
-## Engines (for fast mode)
+**Standard mode** (default for `engine: "all"`): Queries 3 engines, fetches content from top 5 sources via HTTP (with Readability extraction), synthesizes grounded answer with citations.
+
+**Fast mode**: Single engine, no source fetching or synthesis. Good for quick checks.
+
+## Engines
 
 | Engine | Alias | Best for |
 |--------|-------|----------|
-| `all` | — | All 3 engines — but for fast single-engine, pick one below |
+| `all` | - | **Default** -- all 3 engines with synthesis + source fetch |
 | `perplexity` | `p` | Technical Q&A, code explanations, documentation |
 | `bing` | `b` | Recent news, Microsoft ecosystem |
 | `google` | `g` | Broad coverage, multiple perspectives |
@@ -62,66 +67,87 @@ greedy_search({ query: "What's new in React 19?", depth: "standard" })
 When using `engine: "all"`, the tool streams progress as each engine completes:
 
 ```
-**Searching...** ⏳ perplexity · ⏳ bing · ⏳ google
-**Searching...** ✅ perplexity done · ⏳ bing · ⏳ google
-**Searching...** ✅ perplexity done · ✅ bing done · ⏳ google
-**Searching...** ✅ perplexity done · ✅ bing done · ✅ google done
+**Searching...** pending: perplexity, bing, google
+**Searching...** done: perplexity, pending: bing, google
+**Searching...** done: perplexity, done: bing, pending: google
+**Searching...** done: perplexity, done: bing, done: google
+**Synthesizing...** with Gemini
 ```
 
-## Deep Research Mode
+## Source Fetching (HTTP-First)
 
-For research that matters — architecture decisions, library comparisons — use `depth: "deep"`:
+GreedySearch now uses **HTTP-first source fetching** with Mozilla Readability for content extraction:
 
-```
-greedy_search({ query: "best auth patterns for SaaS in 2026", depth: "deep" })
-```
+- **HTTP**: Fast (~200-800ms), parallel, structured markdown output
+- **Browser fallback**: Only when HTTP fails (bot protection, JS-heavy sites)
+- **Typical success rate**: 90%+ of documentation sites work via HTTP
+- **Speed improvement**: ~3x faster than browser-only fetching (15-30s vs 60-180s)
 
-Deep mode: 3 engines + source fetching (top 5) + synthesis + confidence scores. ~60-180s but returns grounded synthesis with fetched evidence.
+The old regex-based HTML stripping has been replaced with professional-grade content extraction that preserves document structure, code blocks, and headings.
 
-**Standard vs Deep:**
-- `standard` (default): 3 engines + synthesis. Good for most research.
-- `deep`: Same + fetches source content for grounded answers. Use when the answer really matters.
+## Smart Source Ranking
 
-**Legacy:** `deep_research` tool still works — aliases to `greedy_search` with `depth: "deep"`.
+Sources are now ranked using query-aware domain boosting:
 
-## Full vs Short Answers
+- **Query keywords** boost official docs (e.g., "react" → react.dev +10 points)
+- **Consensus**: Sources found by multiple engines rank higher
+- **Source type**: Official docs > repos > blogs > community
+- **URL patterns**: `/docs/`, `/api/`, `/reference/` get extra boost
 
-Default mode returns ~300 char summaries to save tokens. Use `fullAnswer: true` for complete responses:
+40+ tech stacks have preferred domain mappings including React, Node.js, Python, Rust, Go, Prisma, Supabase, and more.
 
-```
-greedy_search({ query: "explain the React compiler", engine: "perplexity", fullAnswer: true })
-```
+## GitHub Content Extraction
+
+GreedySearch handles GitHub URLs intelligently:
+
+- **Blob URLs** (`/blob/`) — Automatically rewritten to `raw.githubusercontent.com` for instant raw file access
+- **Tree/Root URLs** — Clones repo locally with `git clone --depth 1`, returns README preview + file tree for agent exploration
+- **Benefits**: Real file contents (not rendered HTML), accurate line numbers, works with private repos via `gh` CLI auth
+
+## Security
+
+- **Private URL blocking** — Requests to localhost, RFC1918 addresses (10.x, 192.168.x), and .local/.internal domains are automatically blocked
+- **Cross-host redirect detection** — Detects redirects to authentication/login pages and falls back to browser extraction
+- **File protocol blocking** — `file://` URLs are rejected
 
 ## Examples
 
-**Quick lookup (fast):**
+**Default research (multi-engine + sources + synthesis):**
 
-```
-greedy_search({ query: "How to use async await in Python", depth: "fast", engine: "perplexity" })
+```javascript
+greedy_search({ query: "Best practices for monorepo structure" })
 ```
 
-**Compare tools (standard):**
+**Quick lookup (fast):**
 
-```
-greedy_search({ query: "Prisma vs Drizzle in 2026", depth: "standard" })
+```javascript
+greedy_search({ query: "How to use async await in Python", depth: "fast", engine: "perplexity" })
 ```
 
-**Deep research (architecture decision):**
+**Compare tools:**
 
-```
-greedy_search({ query: "Best practices for monorepo structure", depth: "deep" })
+```javascript
+greedy_search({ query: "Prisma vs Drizzle in 2026" })
 ```
 
 **Debug an error:**
 
+```javascript
+greedy_search({ query: "Error: Cannot find module 'react-dom/client' Next.js 15" })
 ```
-greedy_search({ query: "Error: Cannot find module 'react-dom/client' Next.js 15", depth: "standard" })
+
+## Full vs Short Answers
+
+Default mode returns ~300 char summaries to save tokens. Use `fullAnswer: true` for complete responses:
+
+```javascript
+greedy_search({ query: "explain the React compiler", engine: "perplexity", fullAnswer: true })
 ```
 
 ## Requirements
 
-- **Chrome** — must be installed. The extension auto-launches a dedicated Chrome instance on port 9222 with its own isolated profile and DevTools port file, separate from your main browser session.
-- **Node.js 22+** — for built-in `fetch` and WebSocket support.
+- **Chrome** -- must be installed. The extension auto-launches a dedicated Chrome instance on port 9222 with its own isolated profile and DevTools port file, separate from your main browser session.
+- **Node.js 22+** -- for built-in `fetch` and WebSocket support.
 
 ## Setup (first time)
 
@@ -143,22 +169,6 @@ Check status:
 node ~/.pi/agent/git/GreedySearch-pi/launch.mjs --status
 ```
 
-## Testing
-
-Run the test suite to verify everything works:
-
-```bash
-./test.sh           # full suite (~3-4 min)
-./test.sh quick     # skip parallel tests (~1 min)
-./test.sh parallel  # parallel race condition tests only
-```
-
-Tests verify:
-- Single engine mode (perplexity, bing, google)
-- Sequential "all" mode searches
-- Parallel "all" mode (5 concurrent searches) — detects tab race conditions
-- Synthesis mode with Gemini
-
 ## Troubleshooting
 
 ### "Chrome not found"
@@ -180,7 +190,7 @@ node ~/.pi/agent/git/GreedySearch-pi/launch.mjs
 
 ### Google / Bing "verify you're human"
 
-The extension auto-clicks verification buttons and Cloudflare Turnstile challenges using broad keyword matching — resilient to variations like "Verify you are human" or localised button text. For hard CAPTCHAs (image puzzles), solve manually in the Chrome window that opens.
+The extension auto-clicks verification buttons and Cloudflare Turnstile challenges using broad keyword matching -- resilient to variations like "Verify you are human" or localised button text. For hard CAPTCHAs (image puzzles), solve manually in the Chrome window that opens.
 
 ### Parallel searches failing
 
@@ -192,60 +202,60 @@ Chrome may be unresponsive. Restart it with `launch.mjs --kill` then `launch.mjs
 
 ### Sources are empty or junk links
 
-Sources are now extracted by regex-parsing Markdown links (`[title](url)`) from the clipboard text captured after each engine responds — not from DOM selectors that break when the engine's UI updates. If sources are empty, the engine's clipboard copy didn't include formatted links (Bing Copilot currently falls into this category).
+Sources are now extracted by regex-parsing Markdown links (`[title](url)`) from the clipboard text captured after each engine responds -- not from DOM selectors that break when the engine's UI updates. If sources are empty, the engine's clipboard copy didn't include formatted links (Bing Copilot currently falls into this category).
 
 ## How It Works
 
-- `index.ts` — Pi extension, registers `greedy_search` tool with streaming progress
-- `search.mjs` — CLI runner, spawns extractors in parallel, emits `PROGRESS:` events to stderr
-- `launch.mjs` — launches dedicated Chrome on port 9222 with isolated profile
-- `extractors/` — per-engine CDP scrapers (Perplexity, Bing Copilot, Google AI, Gemini)
-- `cdp.mjs` — Chrome DevTools Protocol CLI for browser automation
-- `skills/greedy-search/SKILL.md` — skill file that guides the model on when/how to use greedy_search
+- `index.ts` -- Pi extension, registers `greedy_search` tool with streaming progress
+- `search.mjs` -- CLI runner, spawns extractors in parallel, emits `PROGRESS:` events to stderr
+- `launch.mjs` -- launches dedicated Chrome on port 9222 with isolated profile
+- `extractors/` -- per-engine CDP scrapers (Perplexity, Bing Copilot, Google AI, Gemini)
+- `cdp.mjs` -- Chrome DevTools Protocol CLI for browser automation
+- `skills/greedy-search/SKILL.md` -- skill file that guides the model on when/how to use greedy_search
 
 ## Changelog
 
 ### v1.6.1 (2026-03-31)
-- **Single-engine full answers by default** — `engine: "google"` (or any single engine) now returns complete answers instead of truncated previews. Multi-engine (`all`) still truncates to save tokens during synthesis.
-- **Codebase refactored** — extracted 438 lines from `index.ts` into modular formatters (`src/formatters/`) reducing cognitive complexity from 360 to ~60 and maintainability index from 11.2 to ~40+
-- **Removed codebase search confusion** — clarified that `greedy_search` is WEB SEARCH ONLY (not for searching local code)
+- **Single-engine full answers by default** -- `engine: "google"` (or any single engine) now returns complete answers instead of truncated previews. Multi-engine (`all`) still truncates to save tokens during synthesis.
+- **Codebase refactored** -- extracted 438 lines from `index.ts` into modular formatters (`src/formatters/`) reducing cognitive complexity from 360 to ~60 and maintainability index from 11.2 to ~40+
+- **Removed codebase search confusion** -- clarified that `greedy_search` is WEB SEARCH ONLY (not for searching local code)
 
 ### v1.6.0 (2026-03-29)
-- **Merged deep_research into greedy_search** — new `depth` parameter: `fast` (1 engine), `standard` (3 engines + synthesis), `deep` (3 engines + fetch + synthesis + confidence)
-- **Simpler API** — one tool with clear speed/quality tradeoffs instead of separate tools with overlapping flags
-- **Backward compatible** — `deep_research` still works as alias, `--synthesize` and `--deep-research` flags still function
-- **Updated documentation** — README and skill docs now use `depth` parameter throughout
+- **Merged deep_research into greedy_search** -- new `depth` parameter: `fast` (1 engine), `standard` (3 engines + synthesis), `deep` (3 engines + fetch + synthesis + confidence)
+- **Simpler API** -- one tool with clear speed/quality tradeoffs instead of separate tools with overlapping flags
+- **Backward compatible** -- `deep_research` still works as alias, `--synthesize` and `--deep-research` flags still function
+- **Updated documentation** -- README and skill docs now use `depth` parameter throughout
 
 ### v1.5.1 (2026-03-29)
-- Fixed npm package — added `.pi-lens/` and test files to `.npmignore`
+- Fixed npm package -- added `.pi-lens/` and test files to `.npmignore`
 
 ### v1.5.0 (2026-03-29)
 
-- **Code extraction fixed** — `coding_task` now uses clipboard interception to preserve markdown code blocks (was losing them via DOM scraping)
-- **Chrome targeting hardened** — all tools now consistently target the dedicated GreedySearch Chrome via `CDP_PROFILE_DIR`, preventing fallback to user's main Chrome session
-- **Shared utilities** — extracted ~220 lines of duplicate code from extractors into `common.mjs` (cdp wrapper, tab management, clipboard interception)
-- **Documentation leaner** — skill documentation reduced 61% (180 → 70 lines) while preserving all decision-making info
-- **NO API KEYS** — updated messaging to emphasize this works via browser automation, no API keys needed
+- **Code extraction fixed** -- `coding_task` now uses clipboard interception to preserve markdown code blocks (was losing them via DOM scraping)
+- **Chrome targeting hardened** -- all tools now consistently target the dedicated GreedySearch Chrome via `CDP_PROFILE_DIR`, preventing fallback to user's main Chrome session
+- **Shared utilities** -- extracted ~220 lines of duplicate code from extractors into `common.mjs` (cdp wrapper, tab management, clipboard interception)
+- **Documentation leaner** -- skill documentation reduced 61% (180 -> 70 lines) while preserving all decision-making info
+- **NO API KEYS** -- updated messaging to emphasize this works via browser automation, no API keys needed
 
 ### v1.4.2 (2026-03-25)
 
-- **Fresh isolated tabs** — each search now always creates a new `about:blank` tab via `Target.createTarget` and refreshes the CDP page cache immediately after, preventing SPA navigation failures and stale DOM state from prior queries
-- **Regex-based citation extraction** — all extractors (Perplexity, Bing, Gemini) now parse sources from clipboard Markdown links (`[title](url)`) instead of DOM selectors that break on UI updates
-- **Relaxed verification detection** — `consent.mjs` now uses broad keyword matching (`includes('verify')`, `includes('human')`) instead of anchored regexes, correctly catching button text variants like "Verify you are human" across Cloudflare, Microsoft, and generic modals
+- **Fresh isolated tabs** -- each search now always creates a new `about:blank` tab via `Target.createTarget` and refreshes the CDP page cache immediately after, preventing SPA navigation failures and stale DOM state from prior queries
+- **Regex-based citation extraction** -- all extractors (Perplexity, Bing, Gemini) now parse sources from clipboard Markdown links (`[title](url)`) instead of DOM selectors that break on UI updates
+- **Relaxed verification detection** -- `consent.mjs` now uses broad keyword matching (`includes('verify')`, `includes('human')`) instead of anchored regexes, correctly catching button text variants like "Verify you are human" across Cloudflare, Microsoft, and generic modals
 
 ---
 
 ### v1.4.1
 
-- **Fixed parallel synthesis** — multiple `greedy_search` calls with `synthesize: true` now run safely in parallel. Each search creates a fresh Gemini tab that gets cleaned up after synthesis, preventing tab conflicts and "Uncaught" errors.
+- **Fixed parallel synthesis** -- multiple `greedy_search` calls with `synthesize: true` now run safely in parallel. Each search creates a fresh Gemini tab that gets cleaned up after synthesis, preventing tab conflicts and "Uncaught" errors.
 
 ### v1.4.0
 
-- **Grounded synthesis** — Gemini now receives a normalized source registry with stable source IDs, agreement summaries, caveats, and cited claims
-- **Real deep research** — top sources are fetched before synthesis so deep research answers are grounded in fetched evidence, not just engine summaries
-- **Richer source metadata** — source output now includes canonical URLs, domains, source types, per-engine attribution, and confidence metadata
-- **Cleaner tab lifecycle** — temporary Perplexity, Bing, and Google tabs are closed after each fan-out search, and synthesis finishes on the Gemini tab
-- **Isolated Chrome targeting** — GreedySearch now refuses to fall back to your normal Chrome session, preventing stray remote-debugging prompts
+- **Grounded synthesis** -- Gemini now receives a normalized source registry with stable source IDs, agreement summaries, caveats, and cited claims
+- **Real deep research** -- top sources are fetched before synthesis so deep research answers are grounded in fetched evidence, not just engine summaries
+- **Richer source metadata** -- source output now includes canonical URLs, domains, source types, per-engine attribution, and confidence metadata
+- **Cleaner tab lifecycle** -- temporary Perplexity, Bing, and Google tabs are closed after each fan-out search, and synthesis finishes on the Gemini tab
+- **Isolated Chrome targeting** -- GreedySearch now refuses to fall back to your normal Chrome session, preventing stray remote-debugging prompts
 
 ## License