webpeel 0.2.0 → 0.3.1

package/README.md

@@ -2,11 +2,12 @@
 
 [![npm version](https://img.shields.io/npm/v/webpeel.svg)](https://www.npmjs.com/package/webpeel)
 [![npm downloads](https://img.shields.io/npm/dm/webpeel.svg)](https://www.npmjs.com/package/webpeel)
+[![GitHub stars](https://img.shields.io/github/stars/JakeLiuMe/webpeel.svg)](https://github.com/JakeLiuMe/webpeel/stargazers)
 [![CI](https://github.com/JakeLiuMe/webpeel/actions/workflows/ci.yml/badge.svg)](https://github.com/JakeLiuMe/webpeel/actions/workflows/ci.yml)
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.6-blue.svg)](https://www.typescriptlang.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-Turn any web page into clean markdown. Zero config. Free forever.
+Turn any web page into clean markdown. **Stealth mode. Crawl mode. Zero config. Free forever.**
 
 ```bash
 npx webpeel https://news.ycombinator.com
@@ -34,18 +35,40 @@ npx webpeel https://news.ycombinator.com
 
 |  | **WebPeel** | Firecrawl | Jina Reader | MCP Fetch |
 |---|:---:|:---:|:---:|:---:|
-| **Local execution** | ✅ Free forever | ❌ Cloud only | ❌ Cloud only | ✅ Free |
+| **Free tier** | ✅ 125/week | ❌ Cloud only | ❌ Cloud only | ✅ Unlimited |
 | **JS rendering** | ✅ Auto-escalates | ✅ Always | ❌ No | ❌ No |
-| **Anti-bot handling** | ✅ Stealth mode | ✅ Yes | ⚠️ Limited | ❌ No |
+| **Stealth mode** | ✅ Built-in | ✅ Yes | ⚠️ Limited | ❌ No |
+| **Crawl mode** | ✅ Built-in | ✅ Yes | ❌ No | ❌ No |
 | **MCP Server** | ✅ Built-in | ✅ Separate repo | ❌ No | ✅ Yes |
 | **Zero config** | ✅ `npx webpeel` | ❌ API key required | ❌ API key required | ✅ Yes |
-| **Free tier** | ∞ Unlimited local | 500 pages (one-time) | 1000 req/month | ∞ Local only |
-| **Hosted API** | $9/mo (5K pages) | $16/mo (3K pages) | $200/mo (Starter) | N/A |
-| **Credit rollover** | ✅ Up to 1 month | ❌ Expire monthly | ❌ N/A | ❌ N/A |
+| **Free tier** | 125/week | 500 pages (one-time) | 1000 req/month | ∞ Unlimited |
+| **Hosted API** | $9/mo (1,250/wk) | $16/mo (3K/mo) | $200/mo (Starter) | N/A |
+| **Weekly reset** | N/A | ❌ Monthly only | ❌ Monthly only | ❌ N/A |
+| **Extra usage** | N/A | ✅ Pay-as-you-go | ❌ Upgrade only | N/A |
+| **Rollover** | N/A | ✅ 1 week | ❌ Expire monthly | ❌ N/A |
 | **Soft limits** | ✅ Never blocked | ❌ Hard cut-off | ❌ Rate limited | ❌ N/A |
 | **Markdown output** | ✅ Optimized for AI | ✅ Yes | ✅ Yes | ⚠️ Basic |
 
-**WebPeel gives you Firecrawl's power without the price tag.** Run locally for free, or use our hosted API when you need scale.
+**WebPeel gives you Firecrawl's power with a generous free tier.** Like Claude Code — pay only when you need more.
+
+### Usage Model
+
+WebPeel uses a **weekly usage budget** for all users (CLI and API):
+
+- **First 25 fetches**: No account needed — try it instantly
+- **Free tier**: 125 fetches/week (resets every Monday)
+- **Pro tier**: 1,250 fetches/week ($9/mo)
+- **Max tier**: 6,250 fetches/week ($29/mo)
+
+**Credit costs**: Basic fetch = 1 credit, Stealth mode = 5 credits, Search = 1 credit, Crawl = 1 credit/page
+
+**Open source**: The CLI is MIT licensed — you can self-host if needed. But the hosted API requires authentication after 25 fetches.
+
+### Highlights
+
+1. **🎭 Stealth Mode** — Bypass bot detection with playwright-extra stealth plugin. Works on sites that block regular scrapers.
+2. **🕷️ Crawl Mode** — Follow links and extract entire sites. Respects robots.txt and rate limits automatically.
+3. **💰 Generous Free Tier** — Like Claude Code: 125 free fetches every week. First 25 work instantly, no signup. Open source MIT.
 
 ---
 
@@ -54,9 +77,21 @@ npx webpeel https://news.ycombinator.com
 ### CLI (Zero Install)
 
 ```bash
-# Basic usage
+# First 25 fetches work instantly, no signup
 npx webpeel https://example.com
 
+# After 25 fetches, sign up for free (125/week)
+webpeel login
+
+# Check your usage
+webpeel usage
+
+# Stealth mode (bypass bot detection)
+npx webpeel https://protected-site.com --stealth
+
+# Crawl a website (follow links, respect robots.txt)
+npx webpeel crawl https://example.com --max-pages 20 --max-depth 2
+
 # JSON output with metadata
 npx webpeel https://example.com --json
 
@@ -91,9 +126,9 @@ const result = await peel('https://example.com', {
 });
 ```
 
-### MCP Server (Claude Desktop, Cursor, VS Code)
+### MCP Server (Claude Desktop, Cursor, VS Code, Windsurf)
 
-WebPeel provides two MCP tools: `webpeel_fetch` (fetch a URL) and `webpeel_search` (DuckDuckGo search + fetch results).
+WebPeel provides four MCP tools: `webpeel_fetch` (fetch a URL), `webpeel_search` (search the web), `webpeel_batch` (fetch multiple URLs), and `webpeel_crawl` (crawl a site).
 
 #### Claude Desktop
 
@@ -145,6 +180,50 @@ Or install with one click:
 [![Install in Claude Desktop](https://img.shields.io/badge/Install-Claude%20Desktop-5B3FFF?style=for-the-badge&logo=anthropic)](https://mcp.so/install/webpeel?for=claude)
 [![Install in VS Code](https://img.shields.io/badge/Install-VS%20Code-007ACC?style=for-the-badge&logo=visualstudiocode)](https://mcp.so/install/webpeel?for=vscode)
 
+#### Windsurf
+
+Add to `~/.codeium/windsurf/mcp_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "webpeel": {
+      "command": "npx",
+      "args": ["-y", "webpeel", "mcp"]
+    }
+  }
+}
+```
+
+---
+
+## Use with Claude Code
+
+One command to add WebPeel to Claude Code:
+
+```bash
+claude mcp add webpeel -- npx -y webpeel mcp
+```
+
+Or add to your project's `.mcp.json` for team sharing:
+
+```json
+{
+  "mcpServers": {
+    "webpeel": {
+      "command": "npx",
+      "args": ["-y", "webpeel", "mcp"]
+    }
+  }
+}
+```
+
+This gives Claude Code access to:
+- **webpeel_fetch** — Fetch any URL as clean markdown (with stealth mode for protected sites)
+- **webpeel_search** — Search the web via DuckDuckGo
+- **webpeel_batch** — Fetch multiple URLs concurrently
+- **webpeel_crawl** — Crawl websites following links
+
 ---
 
 ## How It Works: Smart Escalation
@@ -156,16 +235,16 @@ WebPeel tries the fastest method first, then escalates only when needed:
 │                    Smart Escalation                          │
 └─────────────────────────────────────────────────────────────┘
 
-Simple HTTP Fetch          Browser Rendering         Stealth Mode
-    ~200ms                      ~2 seconds             ~5 seconds
+Simple HTTP Fetch     →     Browser Rendering    →     Stealth Mode
+    ~200ms                      ~2 seconds               ~5 seconds
        │                            │                       │
        ├─ User-Agent headers        ├─ Full JS execution   ├─ Anti-detect
-       ├─ Cheerio parsing           ├─ Wait for content    ├─ Proxy rotation
-       ├─ Fast & cheap              ├─ Screenshots         └─ Cloudflare bypass
-       │                            │
-       ▼                            ▼
-   Works for 80%              Works for 19%            Works for 1%
-   of websites                (JS-heavy sites)         (heavily protected)
+       ├─ Cheerio parsing           ├─ Wait for content    ├─ Fingerprint mask
+       ├─ Fast & cheap              ├─ Screenshots         ├─ Cloudflare bypass
+       │                            │                       │
+       ▼                            ▼                       ▼
+   Works for 80%              Works for 15%            Works for 5%
+   of websites                (JS-heavy sites)         (bot-protected)
 ```
 
 **Why this matters:**
@@ -244,7 +323,7 @@ await cleanup();  // Close browser instances
 
 ## Hosted API
 
-Live at `https://webpeel-api.onrender.com` — or use the CLI locally for free.
+Live at `https://webpeel-api.onrender.com` — authentication required after first 25 fetches.
 
 ```bash
 # Register and get your API key
@@ -257,29 +336,46 @@ curl "https://webpeel-api.onrender.com/v1/fetch?url=https://example.com" \
   -H "Authorization: Bearer wp_live_your_api_key"
 ```
 
-### Pricing
+### Pricing — Weekly Reset Model
+
+Usage resets every **Monday at 00:00 UTC**, just like Claude Code.
+
+| Plan | Price | Weekly Fetches | Burst Limit | Stealth Mode | Extra Usage |
+|------|------:|---------------:|:-----------:|:------------:|:-----------:|
+| **Free** | $0 | 125/wk (~500/mo) | 25/hr | ❌ | ❌ |
+| **Pro** | $9/mo | 1,250/wk (~5K/mo) | 100/hr | ✅ | ✅ |
+| **Max** | $29/mo | 6,250/wk (~25K/mo) | 500/hr | ✅ | ✅ |
+
+**Three layers of usage control:**
+1. **Burst limit** — Per-hour cap (25/hr free, 100/hr Pro, 500/hr Max) prevents hammering
+2. **Weekly limit** — Main usage gate, resets every Monday
+3. **Extra usage** — When you hit your weekly limit, keep fetching at pay-as-you-go rates
 
-| Plan | Price | Fetches/Month | JS Rendering | Key Features |
-|------|------:|---------------:|:------------:|----------|
-| **Local CLI** | $0 | ∞ Unlimited | ✅ | Full power, your machine |
-| **Cloud Free** | $0 | 500 | ❌ | Soft limits — never blocked |
-| **Cloud Pro** | $9/mo | 5,000 | ✅ | Credit rollover, soft limits |
-| **Cloud Max** | $29/mo | 25,000 | ✅ | Priority queue, credit rollover |
+**Extra usage rates (Pro/Max only):**
+| Fetch Type | Cost |
+|-----------|------|
+| Basic (HTTP) | $0.002 |
+| Stealth (browser) | $0.01 |
+| Search | $0.001 |
 
-### Why WebPeel Pro Beats Firecrawl
+### Why WebPeel Beats Firecrawl
 
-| Feature | WebPeel Local | WebPeel Pro | Firecrawl Hobby |
+| Feature | WebPeel Free | WebPeel Pro | Firecrawl Hobby |
 |---------|:-------------:|:-----------:|:---------------:|
 | **Price** | $0 | $9/mo | $16/mo |
-| **Monthly Fetches** | ∞ | 5,000 | 3,000 |
-| **Credit Rollover** | N/A | ✅ 1 month | ❌ Expire monthly |
-| **Soft Limits** | ✅ Always | ✅ Never locked out | ❌ Hard cut-off |
-| **Self-Host** | ✅ MIT | N/A | ❌ AGPL |
+| **Weekly Fetches** | 125/wk | 1,250/wk | ~750/wk |
+| **Rollover** | ❌ | ✅ 1 week | ❌ Expire monthly |
+| **Soft Limits** | ✅ Degrades | ✅ Never locked out | ❌ Hard cut-off |
+| **Extra Usage** | ❌ | ✅ Pay-as-you-go | ❌ Upgrade only |
+| **Self-Host** | ✅ MIT | ✅ MIT | ❌ AGPL |
 
 **Key differentiators:**
-- **Soft limits on every tier** — When you hit your limit, we degrade to HTTP-only instead of blocking you. Even free users are never locked out.
-- **Credits roll over** — Unused fetches carry forward for 1 month (Firecrawl expires monthly)
-- **CLI is always free** — No vendor lock-in. Run unlimited locally forever.
+- **Like Claude Code** — Generous free tier (125/week), pay when you need more
+- **Weekly resets** — Your usage refreshes every Monday, not once a month
+- **Soft limits on every tier** — At 100%, we degrade gracefully instead of blocking you
+- **Extra usage** — Pro/Max users can toggle on pay-as-you-go with spending caps (no surprise bills)
+- **First 25 free** — Try it instantly, no signup required
+- **Open source** — MIT licensed, self-host if you want full control
 
 See pricing at [webpeel.dev](https://webpeel.dev/#pricing)
 
@@ -388,12 +484,19 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 - [x] CLI with smart escalation
 - [x] TypeScript library
 - [x] MCP server for Claude/Cursor/VS Code
-- [ ] Hosted API with authentication
-- [ ] Rate limiting and caching
-- [ ] Batch processing API
-- [ ] Screenshot capture
+- [x] Hosted API with authentication and usage tracking
+- [x] Rate limiting and caching
+- [x] Batch processing API (`batch <file>`)
+- [x] Screenshot capture (`--screenshot`)
+- [x] CSS selector filtering (`--selector`, `--exclude`)
+- [x] DuckDuckGo search (`search <query>`)
+- [x] Custom headers and cookies
+- [x] Weekly reset usage model with extra usage
+- [x] Stealth mode (playwright-extra + anti-detect)
+- [x] Crawl mode (follow links, respect robots.txt)
 - [ ] PDF extraction
 - [ ] Webhook notifications for monitoring
+- [ ] AI CAPTCHA solving (planned)
 
 Vote on features and roadmap at [GitHub Discussions](https://github.com/JakeLiuMe/webpeel/discussions).
 
@@ -402,7 +505,7 @@ Vote on features and roadmap at [GitHub Discussions](https://github.com/JakeLiuM
 ## FAQ
 
 **Q: How is this different from Firecrawl?**  
-A: WebPeel runs locally for free (Firecrawl is cloud-only). We also have smart escalation to avoid burning resources on simple pages.
+A: WebPeel has a more generous free tier (125/week vs Firecrawl's 500 one-time credits) and uses weekly resets like Claude Code. We also have smart escalation to avoid burning resources on simple pages.
 
 **Q: Can I self-host the API server?**  
 A: Yes! Run `npm run serve` to start the API server. See [docs/self-hosting.md](docs/self-hosting.md) (coming soon).
@@ -411,10 +514,10 @@ A: Yes! Run `npm run serve` to start the API server. See [docs/self-hosting.md](
 A: WebPeel is a tool — how you use it is up to you. Always check a site's ToS before fetching at scale. We recommend respecting `robots.txt` in your own workflows.
 
 **Q: What about CAPTCHA and Cloudflare?**  
-A: WebPeel handles most Cloudflare challenges automatically. For CAPTCHAs, you'll need a solving service (not included).
+A: WebPeel handles most Cloudflare challenges automatically via stealth mode. AI-powered CAPTCHA solving is on our roadmap.
 
 **Q: Can I use this in production?**  
-A: Yes, but be mindful of rate limits. The hosted API (coming soon) is better for high-volume production use.
+A: Yes! The hosted API at `https://webpeel-api.onrender.com` is production-ready with authentication, rate limiting, and usage tracking.
 
 ---
 

package/dist/cli-auth.d.ts

@@ -0,0 +1,84 @@
+/**
+ * CLI Authentication & Usage Tracking
+ *
+ * Handles:
+ * - Anonymous usage (25 free fetches)
+ * - API key authentication
+ * - Usage checking against API
+ * - Config file management (~/.webpeel/config.json)
+ */
+interface CLIConfig {
+    apiKey?: string;
+    anonymousUsage: number;
+    lastReset: string;
+    planTier?: string;
+    planCachedAt?: string;
+}
+interface UsageCheckResult {
+    allowed: boolean;
+    message?: string;
+    isAnonymous?: boolean;
+    usageInfo?: {
+        used: number;
+        limit: number;
+        remaining: number;
+    };
+}
+/**
+ * Load config from ~/.webpeel/config.json
+ */
+export declare function loadConfig(): CLIConfig;
+/**
+ * Save config to ~/.webpeel/config.json
+ */
+export declare function saveConfig(config: CLIConfig): void;
+/**
+ * Delete config file
+ */
+export declare function deleteConfig(): void;
+/**
+ * Check usage quota before making a request
+ */
+export declare function checkUsage(): Promise<UsageCheckResult>;
+export type PremiumFeature = 'stealth' | 'crawl' | 'batch';
+/**
+ * Check if user has access to a premium feature.
+ * Returns { allowed: true } for paid users, or a helpful upgrade message.
+ *
+ * Priority:
+ * 1. No API key → blocked (must sign up)
+ * 2. Has API key + cached plan → check plan tier
+ * 3. Has API key + no cache → check API, then cache
+ * 4. API unreachable + cached plan within 7 days → use cache
+ * 5. API unreachable + stale cache → allow gracefully (trust the user)
+ */
+export declare function checkFeatureAccess(feature: PremiumFeature): Promise<{
+    allowed: boolean;
+    message?: string;
+}>;
+/**
+ * Show usage footer after successful fetch (for free/anonymous users only)
+ */
+export declare function showUsageFooter(usageInfo: {
+    used: number;
+    limit: number;
+    remaining: number;
+} | undefined, isAnonymous: boolean, stealth?: boolean): void;
+/**
+ * Prompt user for API key via stdin
+ */
+export declare function promptForApiKey(): Promise<string>;
+/**
+ * Login command - save API key to config
+ */
+export declare function handleLogin(): Promise<void>;
+/**
+ * Logout command - remove API key from config
+ */
+export declare function handleLogout(): void;
+/**
+ * Usage command - show current quota
+ */
+export declare function handleUsage(): Promise<void>;
+export {};
+//# sourceMappingURL=cli-auth.d.ts.map

package/dist/cli-auth.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli-auth.d.ts","sourceRoot":"","sources":["../src/cli-auth.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAcH,UAAU,SAAS;IACjB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,UAAU,gBAAgB;IACxB,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,SAAS,CAAC,EAAE;QACV,IAAI,EAAE,MAAM,CAAC;QACb,KAAK,EAAE,MAAM,CAAC;QACd,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;CACH;AAsBD;;GAEG;AACH,wBAAgB,UAAU,IAAI,SAAS,CAyBtC;AAED;;GAEG;AACH,wBAAgB,UAAU,CAAC,MAAM,EAAE,SAAS,GAAG,IAAI,CAWlD;AAED;;GAEG;AACH,wBAAgB,YAAY,IAAI,IAAI,CAQnC;AAwBD;;GAEG;AACH,wBAAsB,UAAU,IAAI,OAAO,CAAC,gBAAgB,CAAC,CA+F5D;AAGD,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,OAAO,GAAG,OAAO,CAAC;AAQ3D;;;;;;;;;;GAUG;AACH,wBAAsB,kBAAkB,CAAC,OAAO,EAAE,cAAc,GAAG,OAAO,CAAC;IAAE,OAAO,EAAE,OAAO,CAAC;IAAC,OAAO,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC,CAmEjH;AAED;;GAEG;AACH,wBAAgB,eAAe,CAC7B,SAAS,EAAE;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,SAAS,EAAE,MAAM,CAAA;CAAE,GAAG,SAAS,EACzE,WAAW,EAAE,OAAO,EACpB,OAAO,GAAE,OAAe,GACvB,IAAI,CAaN;AAED;;GAEG;AACH,wBAAsB,eAAe,IAAI,OAAO,CAAC,MAAM,CAAC,CAYvD;AAED;;GAEG;AACH,wBAAsB,WAAW,IAAI,OAAO,CAAC,IAAI,CAAC,CA8BjD;AAED;;GAEG;AACH,wBAAgB,YAAY,IAAI,IAAI,CAUnC;AAED;;GAEG;AACH,wBAAsB,WAAW,IAAI,OAAO,CAAC,IAAI,CAAC,CAkEjD"}