webpeel 0.11.0 → 0.12.1

package/README.md

@@ -87,12 +87,16 @@ First 25 fetches work instantly, no signup. After that, [sign up free](https://a
 | **Stealth mode** | ✅ v2, all plans | ✅ | ⚠️ Limited | ❌ |
 | **Browser profiles** | ✅ Persistent sessions | ❌ | ❌ | ❌ |
 | **Hotel search** | ✅ Multi-source parallel | ❌ | ❌ | ❌ |
-| **CSS schema extraction** | ✅ 6 bundled + auto-detect | ❌ | ❌ | ❌ |
+| **CSS schema extraction** | ✅ 7 bundled + auto-detect | ❌ | ❌ | ❌ |
 | **LLM extraction** | ✅ BYOK, cost tracking | ⚠️ Cloud only | ❌ | ❌ |
 | **Firecrawl-compatible** | ✅ Drop-in replacement | ✅ Native | ❌ | ❌ |
 | **Self-hosting** | ✅ Docker compose | ⚠️ Complex | ❌ | N/A |
 | **Autonomous agent** | ✅ BYOK any LLM | ⚠️ Locked | ❌ | ❌ |
-| **MCP tools** | ✅ 11 tools | 3 | 0 | 1 |
+| **Deep research** | ✅ Multi-source + BM25 | ⚠️ Cloud only | ❌ | ❌ |
+| **Content pruning** | ✅ 2-pass, 15-33% savings | ❌ | ❌ | ❌ |
+| **BM25 filtering** | ✅ Query-focused | ❌ | ❌ | ❌ |
+| **Python SDK** | ✅ `pip install` | ✅ | ❌ | ❌ |
+| **MCP tools** | ✅ 13 tools | ~6 | 0 | 1 |
 | **License** | ✅ AGPL-3.0 | AGPL-3.0 | Proprietary | MIT |
 | **Pricing** | **Free / $9 / $29** | $0 / $16 / $83 | Custom | Free |
 
@@ -179,7 +183,27 @@ Zero dependencies. Pure Python 3.8+. [Full SDK docs →](python-sdk/README.md)
 
 > **Where to add this config:** Claude Desktop → `~/Library/Application Support/Claude/claude_desktop_config.json` · Cursor → Settings → MCP Servers · VS Code → `~/.vscode/mcp.json` · Windsurf → `~/.codeium/windsurf/mcp_config.json`
 
-### Docker (Self-Hosted)
+### Docker
+
+**MCP Server (stdio — for Claude Desktop, Cursor, Windsurf):**
+
+```bash
+docker run -i webpeel/mcp
+```
+
+**MCP Server (HTTP Streamable transport):**
+
+```bash
+docker run -e MCP_HTTP_MODE=true -p 3100:3100 webpeel/mcp
+```
+
+**API Server (Firecrawl-compatible REST API):**
+
+```bash
+docker run -p 3000:3000 webpeel/api
+```
+
+**Self-Hosted (full stack with database):**
 
 ```bash
 git clone https://github.com/webpeel/webpeel.git
@@ -188,6 +212,19 @@ cd webpeel && docker compose up
 
 Full API at `http://localhost:3000`. AGPL-3.0 licensed. [Commercial licensing available](mailto:support@webpeel.dev).
 
+**MCP config for Docker:**
+
+```json
+{
+  "mcpServers": {
+    "webpeel": {
+      "command": "docker",
+      "args": ["run", "-i", "--rm", "webpeel/mcp"]
+    }
+  }
+}
+```
+
 ## Features
 
 ### 🎯 Smart Escalation
@@ -216,6 +253,36 @@ npx webpeel crawl https://docs.example.com --max-pages 100
 npx webpeel map https://example.com --max-urls 5000
 ```
 
+### 🔬 Deep Research
+
+Multi-source research with BM25 relevance ranking. No API key needed for sources mode.
+
+```bash
+# Get ranked sources with relevance scores
+npx webpeel research "best web scraping tools 2025" --max-sources 5
+
+# Full synthesis with LLM (BYOK)
+npx webpeel research "compare Firecrawl vs Crawl4AI" --llm-key sk-...
+```
+
+### 🧹 Token Efficiency
+
+Save 15-77% on AI tokens automatically.
+
+```bash
+# Content pruning (default ON — strips nav/footer/sidebar)
+npx webpeel https://en.wikipedia.org/wiki/Web_scraping
+
+# Query-focused filtering (BM25)
+npx webpeel https://en.wikipedia.org/wiki/Web_scraping --focus "legal issues"
+
+# Token budget (hard cap)
+npx webpeel https://en.wikipedia.org/wiki/Web_scraping --budget 3000
+
+# Combined: prune → focus → budget = 77% savings
+npx webpeel https://en.wikipedia.org/wiki/Web_scraping --focus "legal" --budget 3000
+```
+
 ### 🤖 Autonomous Agent (BYOK)
 
 Give it a prompt, it researches the web using your own LLM key.
@@ -228,24 +295,30 @@ npx webpeel agent "Compare pricing of Notion vs Coda" --llm-key sk-...
 
 | Feature | CLI | Node.js | Python | API |
 |---------|:---:|:-------:|:------:|:---:|
+| Web scraping | ✅ | ✅ | ✅ | ✅ |
+| Deep research | ✅ | ✅ | ✅ | ✅ |
+| Content pruning | ✅ | ✅ | ✅ | ✅ |
+| BM25 query filtering | ✅ | ✅ | — | ✅ |
 | Structured extraction | ✅ | ✅ | ✅ | ✅ |
 | CSS schema extraction | ✅ | ✅ | — | ✅ |
-| LLM extraction (BYOK) | ✅ | ✅ | — | ✅ |
+| LLM extraction (BYOK) | ✅ | ✅ | ✅ | ✅ |
+| Page actions | ✅ | ✅ | ✅ | ✅ |
 | Browser profiles | ✅ | ✅ | — | — |
+| Screenshots | ✅ | ✅ | ✅ | ✅ |
+| Crawling | ✅ | ✅ | ✅ | ✅ |
+| Batch fetching | ✅ | ✅ | ✅ | ✅ |
 | Hotel search | ✅ | — | — | — |
-| Screenshots | ✅ | ✅ | — | ✅ |
+| Token budget | ✅ | ✅ | ✅ | ✅ |
+| Smart chunking | ✅ | ✅ | — | — |
 | Branding extraction | ✅ | ✅ | — | — |
 | Change tracking | ✅ | ✅ | — | — |
-| Token budget | ✅ | ✅ | ✅ | ✅ |
-| Tag filtering | ✅ | ✅ | ✅ | ✅ |
-| Image extraction | ✅ | ✅ | — | ✅ |
 | AI summarization | ✅ | ✅ | — | ✅ |
 | Batch processing | — | ✅ | — | ✅ |
 | PDF extraction | ✅ | ✅ | — | — |
 
 ## Integrations
 
-Works with **LangChain**, **LlamaIndex**, **CrewAI**, **Dify**, and **n8n**. [Integration docs →](https://webpeel.dev/docs)
+Works with **CrewAI**, **Dify**, and **n8n** via the Firecrawl-compatible API. LangChain & LlamaIndex integrations coming soon. [Integration docs →](https://webpeel.dev/docs)
 
 ## Hosted API
 

package/dist/cli.js

@@ -146,6 +146,11 @@ program
     .option('--include-tags <tags>', 'Comma-separated HTML tags/selectors to include (e.g., "main,article,.content")')
     .option('--exclude-tags <tags>', 'Comma-separated HTML tags/selectors to exclude (e.g., "nav,footer,aside")')
     .option('--only-main-content', 'Shortcut for --include-tags main,article')
+    .option('--full-content', 'Return full page content (disable automatic content density pruning)')
+    .option('--focus <query>', 'Query-focused filtering — only return content relevant to this query (BM25 ranking)')
+    .option('--chunk <size>', 'Split content into N-token chunks for LLM processing (default strategy: semantic)', parseInt)
+    .option('--chunk-overlap <tokens>', 'Overlap tokens between chunks (default: 200)', parseInt)
+    .option('--chunk-strategy <strategy>', 'Chunking strategy: fixed, semantic (default), paragraph')
     .option('-H, --header <header...>', 'Custom headers (e.g., "Authorization: Bearer token")')
     .option('--cookie <cookie...>', 'Cookies to set (e.g., "session=abc123")')
     .option('--cache <ttl>', 'Cache results locally (e.g., "5m", "1h", "1d") — default: 5m')
@@ -169,7 +174,8 @@ program
     .option('--extract-all', 'Auto-detect and extract repeated listing items (e.g., search results)')
     .option('--schema <name>', 'Force a specific extraction schema by name or domain (e.g., "booking.com", "amazon")')
     .option('--list-schemas', 'List all available extraction schemas and their supported domains')
-    .option('--scroll-extract [count]', 'Scroll page N times to load lazy content, then extract (implies --render)', (v) => parseInt(v, 10))
+    .option('--scroll-extract [count]', 'Scroll page N times to load lazy content (bare flag = smart auto-scroll until stable), then extract (implies --render)', (v) => parseInt(v, 10))
+    .option('--scroll-extract-timeout <ms>', 'Total timeout in ms for auto-scroll (default: 30000, only used with bare --scroll-extract)', parseInt)
     .option('--csv', 'Output extraction results as CSV')
     .option('--table', 'Output extraction results as a formatted table')
     .option('--pages <n>', 'Follow pagination "Next" links for N pages (max 10)', (v) => parseInt(v, 10))
@@ -453,11 +459,16 @@ program
         // --stealth auto-enables --render (stealth requires browser)
         // --action auto-enables --render (actions require browser)
         // --scroll-extract implies --render (needs browser)
-        const scrollExtractCount = options.scrollExtract !== undefined
-            ? (typeof options.scrollExtract === 'number' ? options.scrollExtract : 3)
-            : 0;
-        const useRender = options.render || options.stealth || (actions && actions.length > 0) || scrollExtractCount > 0 || false;
-        // Inject scroll actions when --scroll-extract is used
+        //
+        // Bare --scroll-extract (no number) → smart autoScroll (detects stable height)
+        // --scroll-extract N (with number) → legacy fixed N scrolls via actions
+        const scrollExtractRaw = options.scrollExtract;
+        const isAutoScroll = scrollExtractRaw !== undefined && typeof scrollExtractRaw !== 'number';
+        const scrollExtractCount = isAutoScroll
+            ? 0
+            : (scrollExtractRaw !== undefined ? scrollExtractRaw : 0);
+        const useRender = options.render || options.stealth || (actions && actions.length > 0) || scrollExtractCount > 0 || isAutoScroll || false;
+        // Inject scroll actions when --scroll-extract N (fixed count) is used
         if (scrollExtractCount > 0) {
             const scrollActions = [];
             for (let i = 0; i < scrollExtractCount; i++) {
@@ -493,6 +504,11 @@ program
             headed: options.headed || false,
             storageState: resolvedStorageState,
             proxy: options.proxy,
+            fullPage: options.fullContent || false,
+            // Smart auto-scroll (bare --scroll-extract flag)
+            autoScroll: isAutoScroll
+                ? { timeout: options.scrollExtractTimeout }
+                : undefined,
         };
         // Add summary option if requested
         if (options.summary) {
@@ -569,6 +585,41 @@ program
                 result.tokens = estimateTokens(distilled);
             }
         }
+        // --- BM25 Query-Focused Filtering ---
+        if (options.focus && result.content) {
+            const { filterByRelevance } = await import('./core/bm25-filter.js');
+            const focusResult = filterByRelevance(result.content, { query: options.focus });
+            result.content = focusResult.content;
+            result.tokens = estimateTokens(focusResult.content);
+            if (isJson) {
+                result.focusQuery = options.focus;
+                result.focusReduction = focusResult.reductionPercent;
+            }
+        }
+        // --- Smart Chunking ---
+        if (options.chunk && options.chunk > 0 && result.content) {
+            const { chunkContent } = await import('./core/chunking.js');
+            const chunkResult = chunkContent(result.content, {
+                chunkSize: options.chunk,
+                overlap: options.chunkOverlap || 200,
+                strategy: options.chunkStrategy || 'semantic',
+            });
+            // Replace content with chunked output
+            if (isJson) {
+                result.chunks = chunkResult.chunks;
+                result.totalChunks = chunkResult.totalChunks;
+                result.originalTokens = chunkResult.originalTokens;
+                // Keep content as first chunk for non-JSON fallback
+                result.content = chunkResult.chunks[0]?.content || '';
+                result.tokens = chunkResult.chunks[0]?.tokens || 0;
+            }
+            else {
+                // Plain text mode: output chunks separated by markers
+                const chunkOutput = chunkResult.chunks.map((c, i) => `--- Chunk ${i + 1}/${chunkResult.totalChunks} (${c.tokens} tokens) ---\n${c.content}`).join('\n\n');
+                result.content = chunkOutput;
+                result.tokens = chunkResult.totalTokens;
+            }
+        }
         // --- #4: Content quality warning ---
         const isHtmlContent = result.contentType ? result.contentType.toLowerCase().includes('html') : true;
         const isRedirect = false; // peel() follows redirects — final result is always 200
@@ -3118,6 +3169,87 @@ program
         process.exit(1);
     }
 });
+// ============================================================
+// research command — autonomous multi-step web research
+// ============================================================
+program
+    .command('research <query>')
+    .description('Conduct autonomous multi-step web research on a topic and synthesize a report')
+    .option('--max-sources <n>', 'Maximum sources to consult (default: 5)', '5')
+    .option('--max-depth <n>', 'Link-following depth (default: 1)', '1')
+    .option('--format <f>', 'Output format: report (default) or sources', 'report')
+    .option('--llm-key <key>', 'LLM API key for synthesis (or env OPENAI_API_KEY)')
+    .option('--llm-model <model>', 'LLM model for synthesis (default: gpt-4o-mini)')
+    .option('--llm-base-url <url>', 'LLM API base URL (default: https://api.openai.com/v1)')
+    .option('--timeout <ms>', 'Max research time in ms (default: 60000)', '60000')
+    .option('--json', 'Output result as JSON')
+    .option('-s, --silent', 'Suppress progress output')
+    .action(async (query, options) => {
+    const isSilent = !!options.silent;
+    const isJson = !!options.json;
+    const maxSources = parseInt(options.maxSources) || 5;
+    const maxDepth = parseInt(options.maxDepth) || 1;
+    const timeout = parseInt(options.timeout) || 60000;
+    const outputFormat = options.format === 'sources' ? 'sources' : 'report';
+    const apiKey = options.llmKey || process.env.OPENAI_API_KEY;
+    const model = options.llmModel;
+    const baseUrl = options.llmBaseUrl;
+    const phaseIcons = {
+        searching: '🔍',
+        fetching: '📄',
+        extracting: '🧠',
+        following: '🔗',
+        synthesizing: '✍️',
+    };
+    try {
+        const { research } = await import('./core/research.js');
+        const result = await research({
+            query,
+            maxSources,
+            maxDepth,
+            timeout,
+            outputFormat: outputFormat,
+            apiKey,
+            model,
+            baseUrl,
+            onProgress: (step) => {
+                if (!isSilent && !isJson) {
+                    const icon = phaseIcons[step.phase] ?? '⚙️';
+                    const extra = step.sourcesFound !== undefined
+                        ? ` (found ${step.sourcesFound})`
+                        : step.sourcesFetched !== undefined
+                            ? ` (${step.sourcesFetched} fetched)`
+                            : '';
+                    process.stderr.write(`${icon} ${step.message}${extra}...\n`);
+                }
+            },
+        });
+        if (isJson) {
+            await writeStdout(JSON.stringify(result, null, 2) + '\n');
+        }
+        else {
+            await writeStdout(result.report + '\n');
+            if (!isSilent) {
+                const elapsed = (result.elapsed / 1000).toFixed(1);
+                const cost = result.cost !== undefined ? ` | cost: $${result.cost.toFixed(4)}` : '';
+                process.stderr.write(`\n📊 ${result.sourcesConsulted} sources consulted (${result.totalSourcesFound} found) | ${elapsed}s${cost}\n`);
+            }
+        }
+        await cleanup();
+        process.exit(0);
+    }
+    catch (error) {
+        const msg = error instanceof Error ? error.message : 'Unknown error';
+        if (isJson) {
+            await writeStdout(JSON.stringify({ error: msg, code: 'RESEARCH_FAILED' }) + '\n');
+        }
+        else {
+            console.error(`\nError: ${msg}`);
+        }
+        await cleanup();
+        process.exit(1);
+    }
+});
 program.parse();
 // ============================================================
 // Time formatting helper