webpeel 0.16.0 → 0.17.0

package/README.md

@@ -1,447 +1,368 @@
 <p align="center">
   <a href="https://webpeel.dev">
-    <img src=".github/banner.svg" alt="WebPeel — Web fetching for AI agents" width="100%">
+    <img src=".github/banner.svg" alt="WebPeel — Web data API for AI agents" width="100%">
   </a>
 </p>
 
 <p align="center">
-  <a href="https://www.npmjs.com/package/webpeel"><img src="https://img.shields.io/npm/v/webpeel.svg" alt="npm version"></a>
-  <a href="https://pypi.org/project/webpeel/"><img src="https://img.shields.io/pypi/v/webpeel.svg" alt="PyPI version"></a>
-  <a href="https://www.npmjs.com/package/webpeel"><img src="https://img.shields.io/npm/dm/webpeel.svg" alt="downloads"></a>
-  <a href="https://github.com/webpeel/webpeel/stargazers"><img src="https://img.shields.io/github/stars/webpeel/webpeel.svg" alt="GitHub stars"></a>
-  <a href="https://github.com/webpeel/webpeel/actions/workflows/ci.yml"><img src="https://github.com/webpeel/webpeel/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
-  <a href="https://www.gnu.org/licenses/agpl-3.0"><img src="https://img.shields.io/badge/License-AGPL%20v3-blue.svg" alt="AGPL v3"></a>
+  <a href="https://www.npmjs.com/package/webpeel"><img src="https://img.shields.io/npm/v/webpeel.svg?style=flat-square" alt="npm version"></a>
+  <a href="https://pypi.org/project/webpeel/"><img src="https://img.shields.io/pypi/v/webpeel.svg?style=flat-square" alt="PyPI version"></a>
+  <a href="https://www.npmjs.com/package/webpeel"><img src="https://img.shields.io/npm/dm/webpeel.svg?style=flat-square" alt="Monthly downloads"></a>
+  <a href="https://github.com/webpeel/webpeel/stargazers"><img src="https://img.shields.io/github/stars/webpeel/webpeel.svg?style=flat-square" alt="GitHub stars"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-WebPeel%20SDK-blue.svg?style=flat-square" alt="License"></a>
+  <a href="https://status.webpeel.dev"><img src="https://img.shields.io/badge/status-operational-brightgreen.svg?style=flat-square" alt="Status"></a>
 </p>
 
 <p align="center">
-  <strong>Web intelligence for AI agents.</strong><br>
-  Fetch any URL → clean markdown. YouTube transcripts. Reddit threads. Quick answers. No API keys needed.
+  <strong>The web data API for AI agents.</strong><br>
+  Fetch, search, extract, and understand any webpage — with one API call.
 </p>
 
 <p align="center">
-  <a href="https://webpeel.dev">Website</a> ·
   <a href="https://webpeel.dev/docs">Docs</a> ·
-  <a href="https://webpeel.dev/playground">Playground</a> ·
   <a href="https://app.webpeel.dev">Dashboard</a> ·
-  <a href="https://github.com/webpeel/webpeel/discussions">Discussions</a>
+  <a href="https://webpeel.dev/docs/api">API Reference</a> ·
+  <a href="https://discord.gg/webpeel">Discord</a> ·
+  <a href="https://status.webpeel.dev">Status</a>
 </p>
 
 ---
 
-> **WebPeel** gives AI agents reliable web access in one call. It handles JavaScript rendering, bot detection, and content extraction automatically — your agent gets clean, structured data. 18 MCP tools, 1,098 tests, 100% open source.
+## Get Started
 
----
-
-## 🚀 Quick Start
+### Install
 
 ```bash
-npx webpeel "https://example.com"
-```
-
-**More examples:**
+# Node.js / TypeScript
+npm install webpeel
 
-```bash
-# YouTube transcript — no API key!
-npx webpeel "https://youtube.com/watch?v=dQw4w9WgXcQ"
-
-# Ask any page a question — no LLM key!
-npx webpeel "https://openai.com/pricing" -q "how much does GPT-4 cost?"
-
-# Reddit thread — structured JSON
-npx webpeel "https://reddit.com/r/programming/comments/..." --json
+# Python
+pip install webpeel
 
-# Reader mode — strips all noise
-npx webpeel "https://nytimes.com/article" --readable
+# No install — use directly
+npx webpeel "https://example.com"
 ```
 
-No install needed. First 25 fetches work without signup. [Get 500/week free →](https://app.webpeel.dev/signup)
+### Usage
 
-### MCP Server (for Claude, Cursor, VS Code, Windsurf)
+**TypeScript**
+```typescript
+import { WebPeel } from 'webpeel';
 
-```json
-{
-  "mcpServers": {
-    "webpeel": {
-      "command": "npx",
-      "args": ["-y", "webpeel", "mcp"]
-    }
-  }
-}
+const wp = new WebPeel({ apiKey: process.env.WEBPEEL_API_KEY });
+const result = await wp.fetch('https://news.ycombinator.com');
+console.log(result.markdown); // Clean, structured content
 ```
 
-[![Install in Claude Desktop](https://img.shields.io/badge/Install-Claude%20Desktop-5B3FFF?style=for-the-badge&logo=anthropic)](https://mcp.so/install/webpeel?for=claude)
-[![Install in VS Code](https://img.shields.io/badge/Install-VS%20Code-007ACC?style=for-the-badge&logo=visualstudiocode)](https://mcp.so/install/webpeel?for=vscode)
+**Python**
+```python
+from webpeel import WebPeel
 
-### REST API
+wp = WebPeel(api_key=os.environ["WEBPEEL_API_KEY"])
+result = wp.fetch("https://news.ycombinator.com")
+print(result.markdown)  # Clean, structured content
+```
 
+**curl**
 ```bash
 curl "https://api.webpeel.dev/v1/fetch?url=https://example.com" \
-  -H "Authorization: Bearer wp_YOUR_KEY"
+  -H "Authorization: Bearer $WEBPEEL_API_KEY"
 ```
 
----
-
-## ✨ What can it do?
-
-| | Feature | What you get |
-|---|---------|-------------|
-| 🌐 | **Fetch** | Any URL → clean markdown, text, or JSON. Auto-handles JS rendering, bot detection, CAPTCHAs |
-| 🎬 | **YouTube** | Full video transcripts with timestamps. No API key |
-| 🐦 | **Twitter/Reddit/GitHub/HN** | Structured data from social platforms via native APIs |
-| ❓ | **Quick Answer** | Ask a question about any page. BM25 scoring, no LLM key |
-| 📖 | **Reader Mode** | Browser Reader Mode for AI — strips nav, ads, cookies, 25+ noise patterns |
-| 🔍 | **Search** | Web search across 27+ sites. Deep research with multi-hop analysis |
-| 📊 | **Extract** | Pricing pages, products, contacts → structured JSON. CSS/JSON Schema/LLM extraction |
-| 🕵️ | **Stealth** | Bypasses Cloudflare, PerimeterX, DataDome, Akamai. 28 auto-stealth domains |
-| 🏨 | **Hotels** | Kayak + Booking + Google Travel + Expedia in parallel |
-| 🔄 | **Monitor** | Watch URLs for changes, get webhook notifications |
-| 🕷️ | **Crawl** | BFS/DFS site crawling, sitemap discovery, robots.txt compliance |
-| 📸 | **Screenshot** | Full-page or viewport screenshots |
-| 🐍 | **Python SDK** | `pip install webpeel` — sync + async client |
+[Get your free API key →](https://app.webpeel.dev/signup) · No credit card required · 500 requests/week free
 
 ---
 
-## 🏆 How does it compare?
-
-| Feature | WebPeel | Firecrawl | Crawl4AI | Jina Reader |
-|---------|:-------:|:---------:|:--------:|:-----------:|
-| YouTube transcripts | ✅ | ❌ | ❌ | ❌ |
-| LLM-free Q&A | ✅ | ❌ | ❌ | ❌ |
-| Reader mode | ✅ | ❌ | ❌ | ❌ |
-| Domain extractors (Twitter, Reddit, GH, HN) | ✅ | ❌ | ❌ | ❌ |
-| Auto-extract (pricing, products) | ✅ | ❌ | ❌ | ❌ |
-| URL monitoring | ✅ | ❌ | ❌ | ❌ |
-| Stealth / anti-bot | ✅ | ⚡ Hosted only | ✅ | ❌ |
-| MCP server | ✅ 18 tools | ✅ 4 tools | ❌ | ❌ |
-| Deep research | ✅ | ❌ | ❌ | ❌ |
-| Hotel search | ✅ | ❌ | ❌ | ❌ |
-| Self-hostable | ✅ | ✅ | ✅ | ❌ |
-| Free tier | 500/week | 500 credits | Unlimited | Unlimited |
-| Open source | AGPL-3.0 | AGPL-3.0 | Apache-2.0 | N/A |
+## What It Does
 
----
-
-## ⚡ Benchmark
-
-Evaluated on 30 real-world URLs across 6 categories (static, dynamic, SPA, protected, documents, international):
-
-| | WebPeel | Next best |
-|---|:---:|:---:|
-| **Success rate** | **100%** (30/30) | 93.3% |
-| **Content quality** | **92.3%** | 83.2% |
-
-WebPeel is the only tool that extracted content from all 30 test URLs. [Full methodology →](https://webpeel.dev/blog/benchmarks)
+| | Capability | Result |
+|---|---|---|
+| 🌐 | **Fetch** | Any URL → clean markdown or JSON. Handles JavaScript, bot detection, and dynamic content automatically |
+| 🔍 | **Search** | Web search with structured results — titles, URLs, snippets, and optional full-page content |
+| 📊 | **Extract** | Pull structured data using JSON Schema. Products, pricing, contacts, tables — any pattern |
+| 🕷️ | **Crawl** | Map and scrape entire websites with one API call. Follows links, respects robots.txt |
+| 🤖 | **MCP** | 18 tools natively available in Claude, Cursor, VS Code, Windsurf, and any MCP-compatible agent |
+| 📸 | **Screenshot** | Full-page or viewport screenshots in PNG/JPEG |
+| 🎬 | **YouTube** | Video transcripts with timestamps — no YouTube API key required |
+| 👁️ | **Monitor** | Watch pages for changes and receive webhook notifications |
 
 ---
 
-## 🤖 MCP Integration
-
-WebPeel exposes **18 tools** to your AI coding assistant:
-
-| Tool | What it does |
-|------|--------------|
-| `webpeel_fetch` | Fetch any URL → markdown. Smart escalation built in. Supports `readable: true` for reader mode |
-| `webpeel_search` | Web search with structured results across 27+ sources |
-| `webpeel_batch` | Fetch multiple URLs concurrently |
-| `webpeel_crawl` | Crawl a site with depth/page limits |
-| `webpeel_map` | Discover all URLs on a domain |
-| `webpeel_extract` | Structured extraction (CSS, JSON Schema, or LLM) |
-| `webpeel_screenshot` | Screenshot any page (full-page or viewport) |
-| `webpeel_research` | Deep multi-hop research on a topic |
-| `webpeel_summarize` | AI summary of any URL |
-| `webpeel_answer` | Ask a question about a URL's content |
-| `webpeel_change_track` | Detect changes between two fetches |
-| `webpeel_brand` | Extract branding assets from a site |
-| `webpeel_deep_fetch` | Search + batch fetch + merge — comprehensive research, no LLM key |
-| `webpeel_youtube` | Extract YouTube video transcripts — all URL formats, no API key |
-| `webpeel_auto_extract` | Heuristic structured data extraction — auto-detects pricing, products, contacts |
-| `webpeel_quick_answer` | BM25-powered Q&A — ask any question about any page, no LLM key |
-| `webpeel_watch` | Persistent URL change monitoring with webhook notifications |
-| `webpeel_hotels` | Hotel search across Kayak, Booking.com, Google Travel, Expedia in parallel |
-
-<details>
-<summary>Setup for Claude Desktop, Cursor, VS Code, Windsurf, Docker</summary>
-
-**Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
-```json
-{
-  "mcpServers": {
-    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
-  }
-}
-```
+## Anti-Bot Bypass Stack
 
-**Cursor** (Settings → MCP Servers):
-```json
-{
-  "mcpServers": {
-    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
-  }
-}
-```
+WebPeel uses a 4-layer escalation chain to bypass bot protection — all built in-house, no paid proxy services required:
 
-**VS Code** (`~/.vscode/mcp.json`):
-```json
-{
-  "servers": {
-    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
-  }
-}
 ```
-
-**Windsurf** (`~/.codeium/windsurf/mcp_config.json`):
-```json
-{
-  "mcpServers": {
-    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
-  }
-}
+1. PeelTLS      — Chrome TLS fingerprint spoofing (in-process Go binary)  ~85% of sites
+2. CF Worker    — Cloudflare edge network proxy (different IP reputation)  +5%
+3. Google Cache — Cached page copy if available                            +2%
+4. Search       — Extract from search engine snippets (last resort)        last resort
 ```
 
-**Docker (stdio)**:
-```json
-{
-  "mcpServers": {
-    "webpeel": { "command": "docker", "args": ["run", "-i", "--rm", "webpeel/mcp"] }
-  }
-}
-```
+**For e-commerce sites**, WebPeel uses official APIs before attempting HTML scraping:
+- **Best Buy** — Free Products API (50K queries/day). Set `BESTBUY_API_KEY` env var.
+- **Walmart** — Frontend API (may be blocked; falls through gracefully)
+- **Reddit, GitHub, HN, Wikipedia, YouTube, ArXiv** — Official APIs, always fast
 
-**Hosted endpoint** (no local server needed):
-```json
-{
-  "mcpServers": {
-    "webpeel": {
-      "url": "https://api.webpeel.dev/mcp",
-      "headers": { "Authorization": "Bearer YOUR_API_KEY" }
-    }
-  }
-}
+**Self-hosted CF Worker** (100K requests/day free):
+```bash
+cd worker && npx wrangler deploy
+# Then set WEBPEEL_CF_WORKER_URL and WEBPEEL_CF_WORKER_TOKEN env vars
 ```
-</details>
 
 ---
 
-## 🔬 Deep Research
+## Benchmarks
 
-Multi-hop research that thinks like a researcher, not a search engine:
+Independent testing across 500 URLs including e-commerce, news, SaaS, and social platforms.
 
-```bash
-# Sources only — no API key needed
-npx webpeel research "best practices for rate limiting APIs" --max-sources 8
+| Metric | **WebPeel** | Firecrawl | Crawl4AI | Jina Reader |
+|--------|:-----------:|:---------:|:--------:|:-----------:|
+| Success rate (protected sites) | **94%** | 71% | 58% | 49% |
+| Median response time | **380ms** | 890ms | 1,240ms | 520ms |
+| Content quality score¹ | **0.91** | 0.74 | 0.69 | 0.72 |
+| Price per 1,000 requests | **$0.80** | $5.33 | self-host | $1.00 |
 
-# Full synthesis with LLM (BYOK)
-npx webpeel research "compare Firecrawl vs Crawl4AI vs WebPeel" --llm-key sk-...
-```
+¹ Content quality = signal-to-noise ratio (relevant content vs boilerplate), scored 0–1.
 
-Search → fetch top results → extract key passages (BM25) → follow the most relevant links → synthesize. No circular references, no duplicate content.
+> Methodology: Tested Feb 2026. Protected sites = Cloudflare/bot-protected pages. Quality scored by GPT-4o on content relevance and completeness. [Full methodology →](https://webpeel.dev/benchmarks)
 
 ---
 
-## 📦 Extraction
+## Pricing
 
-<details>
-<summary>CSS Schema, JSON Schema, and LLM extraction — click to expand</summary>
+| Plan | Price | Requests | Features |
+|------|-------|----------|----------|
+| **Free** | $0/mo | 500/week | Fetch, search, extract, crawl |
+| **Pro** | $9/mo | 1,250/week | Everything + protected site access |
+| **Max** | $29/mo | 6,250/week | Everything + priority queue |
+| **Enterprise** | Custom | Unlimited | SLA, dedicated infra, custom domains |
 
-### CSS Schema (zero config, auto-detected)
+All plans include: full API access, TypeScript + Python SDKs, MCP server, CLI.  
+[See full pricing →](https://webpeel.dev/pricing)
 
-```bash
-# Auto-detects Amazon and applies the built-in schema
-npx webpeel "https://www.amazon.com/s?k=mechanical+keyboard" --json
-
-# Force a specific schema
-npx webpeel "https://www.booking.com/searchresults.html?city=Paris" --schema booking --json
-
-# List all built-in schemas
-npx webpeel --list-schemas
-```
-
-Built-in schemas: `amazon` · `booking` · `ebay` · `expedia` · `hackernews` · `walmart` · `yelp`
-
-### JSON Schema (type-safe structured extraction)
+---
 
-```bash
-npx webpeel "https://example.com/product" \
-  --extract-schema '{"type":"object","properties":{"title":{"type":"string"},"price":{"type":"number"}}}' \
-  --llm-key sk-...
-```
+## SDK
 
-### LLM Extraction (natural language, BYOK)
+### TypeScript / Node.js
 
-```bash
-npx webpeel "https://hn.algolia.com" \
-  --llm-extract "top 10 posts with title, score, and comment count" \
-  --llm-key $OPENAI_API_KEY \
-  --json
-```
+```typescript
+import { WebPeel } from 'webpeel';
 
-> **Note:** WebPeel is an ESM-only package. Use `import` syntax:
-> ```js
-> import { peel } from 'webpeel';
-> ```
-> CommonJS `require()` is not supported. If your project uses CommonJS, use dynamic import: `const { peel } = await import('webpeel');`
+const wp = new WebPeel({ apiKey: process.env.WEBPEEL_API_KEY });
 
-```typescript
-import { peel } from 'webpeel';
+// Fetch a page
+const page = await wp.fetch('https://stripe.com/pricing', {
+  format: 'markdown',  // 'markdown' | 'html' | 'text' | 'json'
+});
 
-// CSS selector extraction
-const result = await peel('https://news.ycombinator.com', {
-  extract: { selectors: { titles: '.titleline > a', scores: '.score' } }
+// Search the web
+const results = await wp.search('best vector databases 2025', {
+  limit: 5,
+  fetchContent: true,  // Optionally fetch full content for each result
 });
 
-// LLM extraction with JSON Schema
-const product = await peel('https://example.com/product', {
-  llmExtract: 'title, price, rating, availability',
-  llmKey: process.env.OPENAI_API_KEY,
+// Extract structured data
+const pricing = await wp.extract('https://stripe.com/pricing', {
+  schema: {
+    type: 'object',
+    properties: {
+      plans: {
+        type: 'array',
+        items: { type: 'object', properties: {
+          name: { type: 'string' },
+          price: { type: 'string' },
+          features: { type: 'array', items: { type: 'string' } }
+        }}
+      }
+    }
+  }
 });
-```
-</details>
 
----
+// Crawl a site
+const crawl = await wp.crawl('https://docs.example.com', {
+  maxPages: 50,
+  maxDepth: 3,
+  outputFormat: 'markdown',
+});
+for await (const page of crawl) {
+  console.log(page.url, page.markdown);
+}
 
-## 🛡️ Stealth & Anti-Bot
+// Screenshot
+const shot = await wp.screenshot('https://webpeel.dev', { fullPage: true });
+fs.writeFileSync('screenshot.png', shot.image, 'base64');
+```
 
-<details>
-<summary>Supported bot-protection vendors and auto-stealth domains — click to expand</summary>
+[Full TypeScript reference →](https://webpeel.dev/docs/sdk/typescript)
 
-WebPeel detects 7 bot-protection vendors automatically:
+### Python
 
-- **Cloudflare** (JS challenge, Turnstile, Bot Management)
-- **PerimeterX / HUMAN** (behavioral analysis)
-- **DataDome** (ML-based bot detection)
-- **Akamai Bot Manager**
-- **Distil Networks**
-- **reCAPTCHA / hCaptcha**
-- **Generic challenge pages**
+```python
+from webpeel import WebPeel
+import os
+
+wp = WebPeel(api_key=os.environ["WEBPEEL_API_KEY"])
+
+# Fetch a page
+page = wp.fetch("https://stripe.com/pricing", format="markdown")
+print(page.markdown)
+
+# Search
+results = wp.search("best vector databases 2025", limit=5)
+for r in results:
+    print(r.title, r.url)
+
+# Extract structured data
+pricing = wp.extract("https://stripe.com/pricing", schema={
+    "type": "object",
+    "properties": {
+        "plans": {
+            "type": "array",
+            "items": { "type": "object", "properties": {
+                "name": { "type": "string" },
+                "price": { "type": "string" }
+            }}
+        }
+    }
+})
 
-28 high-protection domains (Amazon, LinkedIn, Glassdoor, Zillow, Ticketmaster, and more) automatically route through stealth mode — no flags needed.
+# Async client
+from webpeel import AsyncWebPeel
+import asyncio
 
-```bash
-# Explicitly enable stealth
-npx webpeel "https://glassdoor.com/jobs" --stealth
+async def main():
+    wp = AsyncWebPeel(api_key=os.environ["WEBPEEL_API_KEY"])
+    results = await asyncio.gather(
+        wp.fetch("https://site1.com"),
+        wp.fetch("https://site2.com"),
+        wp.fetch("https://site3.com"),
+    )
 
-# Auto-escalation (stealth triggers automatically on challenge detection)
-npx webpeel "https://amazon.com/dp/ASIN"
+asyncio.run(main())
 ```
-</details>
-
----
 
-## 🏨 Hotel Search
+[Full Python reference →](https://webpeel.dev/docs/sdk/python)
 
-<details>
-<summary>Multi-source hotel search — click to expand</summary>
+### MCP — For AI Agents
 
-Search Kayak, Booking.com, Google Travel, and Expedia in parallel — returns unified results in one call.
+Give Claude, Cursor, or any MCP-compatible agent the ability to browse the web.
 
-```bash
-npx webpeel hotels "Paris" --check-in 2025-06-01 --check-out 2025-06-07 --guests 2 --json
+**Claude Desktop** (`~/.claude/claude_desktop_config.json`):
+```json
+{
+  "mcpServers": {
+    "webpeel": {
+      "command": "npx",
+      "args": ["-y", "webpeel", "mcp"],
+      "env": {
+        "WEBPEEL_API_KEY": "wp_your_key_here"
+      }
+    }
+  }
+}
 ```
 
-Available as `webpeel_hotels` MCP tool and via the REST API.
-</details>
+**Cursor / VS Code** (`.cursor/mcp.json` or `.vscode/mcp.json`):
+```json
+{
+  "mcpServers": {
+    "webpeel": {
+      "command": "npx",
+      "args": ["-y", "webpeel", "mcp"],
+      "env": {
+        "WEBPEEL_API_KEY": "wp_your_key_here"
+      }
+    }
+  }
+}
+```
 
----
+Available MCP tools: `fetch`, `search`, `extract`, `crawl`, `screenshot`, `youtube_transcript`, `monitor_start`, `monitor_stop`, `monitor_list`, `batch_fetch`, `map_site`, `diff`, `summarize`, `qa`, `pdf`, `reddit`, `twitter`, `github` — 18 tools total.
 
-## 💳 Pricing
+[![Install in Claude Desktop](https://img.shields.io/badge/Install-Claude%20Desktop-5B3FFF?style=for-the-badge&logo=anthropic)](https://mcp.so/install/webpeel?for=claude)
+[![Install in VS Code](https://img.shields.io/badge/Install-VS%20Code-007ACC?style=for-the-badge&logo=visualstudiocode)](https://mcp.so/install/webpeel?for=vscode)
 
-| Plan | Price | Weekly Fetches | Burst |
-|------|------:|:--------------:|:-----:|
-| **Free** | $0/mo | 500/wk | 50/hr |
-| **Pro** | $9/mo | 1,250/wk | 100/hr |
-| **Max** | $29/mo | 6,250/wk | 500/hr |
+[MCP setup guide →](https://webpeel.dev/docs/mcp)
 
-All features on all plans. Pro/Max add pay-as-you-go extra usage. Quota resets every Monday.
+### CLI
 
-[Sign up free →](https://app.webpeel.dev/signup) · [Compare with Firecrawl →](https://webpeel.dev/migrate-from-firecrawl)
+```bash
+# Install globally
+npm install -g webpeel
 
----
+# Fetch a page (outputs clean markdown)
+webpeel "https://news.ycombinator.com"
 
-## 🐍 Python SDK
+# Search the web
+webpeel search "typescript orm comparison 2025"
 
-<details>
-<summary>Python SDK usage — click to expand</summary>
+# Extract structured data
+webpeel extract "https://stripe.com/pricing" --schema pricing-schema.json
 
-```bash
-pip install webpeel
-```
+# Crawl a site, save to folder
+webpeel crawl "https://docs.example.com" --output ./docs-dump --max-pages 100
 
-```python
-from webpeel import WebPeel
+# Screenshot
+webpeel screenshot "https://webpeel.dev" --full-page --output screenshot.png
 
-client = WebPeel(api_key="wp_...")  # or WEBPEEL_API_KEY env var
+# YouTube transcript
+webpeel youtube "https://youtube.com/watch?v=dQw4w9WgXcQ"
 
-result = client.scrape("https://example.com")
-print(result.content)    # Clean markdown
-print(result.metadata)   # title, description, author, ...
+# Ask a question about a page
+webpeel qa "https://openai.com/pricing" --question "How much does GPT-4o cost per million tokens?"
 
-results = client.search("latest AI research papers")
-job = client.crawl("https://docs.example.com", limit=100)
-result = client.scrape("https://protected-site.com", render=True, stealth=True)
+# Output as JSON
+webpeel "https://example.com" --json
 ```
 
-Sync and async clients. Pure Python 3.8+, zero dependencies. [Full SDK docs →](python-sdk/README.md)
-</details>
-
 ---
 
-## 🐳 Self-Hosting
+## API Reference
 
-```bash
-git clone https://github.com/webpeel/webpeel.git
-cd webpeel && docker compose up
-```
-
-Full REST API at `http://localhost:3000`. AGPL-3.0 licensed. [Self-hosting guide →](SELF_HOST.md)
+Base URL: `https://api.webpeel.dev/v1`
 
 ```bash
-docker run -i webpeel/mcp          # MCP server only
-docker run -p 3000:3000 webpeel/api  # API server only
-```
+# Fetch
+GET /fetch?url=<url>&format=markdown
 
----
+# Search
+GET /search?q=<query>&limit=10
 
-## 🤝 Contributing
+# Extract
+POST /extract
+{ "url": "...", "schema": { ... } }
 
-```bash
-git clone https://github.com/webpeel/webpeel.git
-cd webpeel
-npm install && npm run build
-npm test
-```
+# Crawl
+POST /crawl
+{ "url": "...", "maxPages": 50, "maxDepth": 3 }
 
-- **Bug reports:** [Open an issue](https://github.com/webpeel/webpeel/issues)
-- **Feature requests:** [Start a discussion](https://github.com/webpeel/webpeel/discussions)
-- **Code:** See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines
+# Screenshot
+GET /screenshot?url=<url>&fullPage=true
 
-The project has 1,098 tests. Please add tests for new features.
-
----
+# YouTube transcript
+GET /youtube?url=<youtube_url>
+```
 
-## Star History
+All endpoints require `Authorization: Bearer wp_YOUR_KEY`.
 
-<a href="https://star-history.com/#webpeel/webpeel&Date">
-  <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=webpeel/webpeel&type=Date&theme=dark" />
-    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=webpeel/webpeel&type=Date" />
-    <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=webpeel/webpeel&type=Date" width="600" />
-  </picture>
-</a>
+[Full API reference →](https://webpeel.dev/docs/api)
 
 ---
 
-## License
-
-[AGPL-3.0](LICENSE) — free to use, modify, and distribute. If you run a modified version as a network service, you must release your source under AGPL-3.0.
+## Links
 
-Need a commercial license? [support@webpeel.dev](mailto:support@webpeel.dev)
-
-> Versions 0.7.1 and earlier were released under MIT and remain MIT-licensed.
+- 📖 [Documentation](https://webpeel.dev/docs) — Guides, references, and examples
+- 🚀 [Dashboard](https://app.webpeel.dev) — Manage your API keys and usage
+- 🔌 [API Reference](https://webpeel.dev/docs/api) — Full endpoint documentation
+- 💬 [Discord](https://discord.gg/webpeel) — Community and support
+- 📊 [Status](https://status.webpeel.dev) — Uptime and incidents
+- 💰 [Pricing](https://webpeel.dev/pricing) — Plans and limits
+- 📈 [Benchmarks](https://webpeel.dev/benchmarks) — How we compare
 
 ---
 
 <p align="center">
-  If WebPeel saves you time, <a href="https://github.com/webpeel/webpeel"><strong>⭐ star the repo</strong></a> — it helps others find it.
+  <a href="https://app.webpeel.dev/signup">Get started free →</a>
 </p>
-
-© [WebPeel](https://github.com/webpeel)