webpeel 0.3.0 → 0.3.2

package/README.md

@@ -7,7 +7,7 @@
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.6-blue.svg)](https://www.typescriptlang.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-Turn any web page into clean markdown. **Stealth mode. Crawl mode. Zero config. Free forever.**
+Turn any web page into clean markdown. **Smart escalation. Stealth mode. Crawl mode. Free to start.**
 
 ```bash
 npx webpeel https://news.ycombinator.com
@@ -35,27 +35,37 @@ npx webpeel https://news.ycombinator.com
 
 |  | **WebPeel** | Firecrawl | Jina Reader | MCP Fetch |
 |---|:---:|:---:|:---:|:---:|
-| **Local execution** | ✅ Free forever | ❌ Cloud only | ❌ Cloud only | ✅ Free |
+| **Free tier** | ✅ 125/week | ❌ Cloud only | ❌ Cloud only | ✅ Unlimited |
 | **JS rendering** | ✅ Auto-escalates | ✅ Always | ❌ No | ❌ No |
-| **Stealth mode** | ✅ Built-in | ✅ Yes | ⚠️ Limited | ❌ No |
-| **Crawl mode** | ✅ Built-in | ✅ Yes | ❌ No | ❌ No |
+| **Stealth mode** | ✅ Pro plan | ✅ Yes | ⚠️ Limited | ❌ No |
+| **Crawl mode** | ✅ Pro plan | ✅ Yes | ❌ No | ❌ No |
 | **MCP Server** | ✅ Built-in | ✅ Separate repo | ❌ No | ✅ Yes |
 | **Zero config** | ✅ `npx webpeel` | ❌ API key required | ❌ API key required | ✅ Yes |
-| **Free tier** | ∞ Unlimited local | 500 pages (one-time) | 1000 req/month | ∞ Local only |
 | **Hosted API** | $9/mo (1,250/wk) | $16/mo (3K/mo) | $200/mo (Starter) | N/A |
-| **Weekly reset** | N/A | ❌ Monthly only | ❌ Monthly only | ❌ N/A |
-| **Extra usage** | N/A | ✅ Pay-as-you-go | ❌ Upgrade only | N/A |
-| **Rollover** | N/A | ✅ 1 week | ❌ Expire monthly | ❌ N/A |
-| **Soft limits** | ✅ Never blocked | ❌ Hard cut-off | ❌ Rate limited | ❌ N/A |
+| **Weekly reset** | ✅ Every Monday | ❌ Monthly only | ❌ Monthly only | N/A |
+| **Soft limits** | ✅ Never blocked | ❌ Hard cut-off | ❌ Rate limited | N/A |
 | **Markdown output** | ✅ Optimized for AI | ✅ Yes | ✅ Yes | ⚠️ Basic |
 
-**WebPeel gives you Firecrawl's power without the price tag.** Run locally for free, or use our hosted API when you need scale.
+**WebPeel gives you Firecrawl's power with a generous free tier.** Like Claude Code — pay only when you need more.
+
+### Usage Model
+
+WebPeel uses a **weekly usage budget** for all users (CLI and API):
+
+- **First 25 fetches**: No account needed — try it instantly
+- **Free tier**: 125 fetches/week (resets every Monday)
+- **Pro tier**: 1,250 fetches/week ($9/mo)
+- **Max tier**: 6,250 fetches/week ($29/mo)
+
+**Credit costs**: Basic fetch = 1 credit, Stealth mode = 5 credits, Search = 1 credit, Crawl = 1 credit/page
+
+**Open source**: The CLI is MIT licensed — you can self-host if needed. But the hosted API requires authentication after 25 fetches.
 
 ### Highlights
 
-1. **🎭 Stealth Mode** — Bypass bot detection with playwright-extra stealth plugin. Works on sites that block regular scrapers.
-2. **🕷️ Crawl Mode** — Follow links and extract entire sites. Respects robots.txt and rate limits automatically.
-3. **💰 Actually Free** — Run unlimited requests locally. No API keys, no credit cards, no surprises. Open source MIT.
+1. **🎭 Stealth Mode** *(Pro)* — Bypass bot detection with playwright-extra stealth plugin. Works on sites that block regular scrapers.
+2. **🕷️ Crawl Mode** *(Pro)* — Follow links and extract entire sites. Respects robots.txt and rate limits automatically.
+3. **💰 Generous Free Tier** — 125 free fetches every week. First 25 work instantly with no signup. Basic fetch + JS rendering included free.
 
 ---
 
@@ -64,9 +74,15 @@ npx webpeel https://news.ycombinator.com
 ### CLI (Zero Install)
 
 ```bash
-# Basic usage
+# First 25 fetches work instantly, no signup
 npx webpeel https://example.com
 
+# After 25 fetches, sign up for free (125/week)
+webpeel login
+
+# Check your usage
+webpeel usage
+
 # Stealth mode (bypass bot detection)
 npx webpeel https://protected-site.com --stealth
 
@@ -304,16 +320,16 @@ await cleanup();  // Close browser instances
 
 ## Hosted API
 
-Live at `https://webpeel-api.onrender.com` — or use the CLI locally for free.
+Live at `https://api.webpeel.dev` — authentication required after first 25 fetches.
 
 ```bash
 # Register and get your API key
-curl -X POST https://webpeel-api.onrender.com/v1/auth/register \
+curl -X POST https://api.webpeel.dev/v1/auth/register \
   -H "Content-Type: application/json" \
   -d '{"email":"you@example.com","password":"your-password"}'
 
 # Fetch a page
-curl "https://webpeel-api.onrender.com/v1/fetch?url=https://example.com" \
+curl "https://api.webpeel.dev/v1/fetch?url=https://example.com" \
   -H "Authorization: Bearer wp_live_your_api_key"
 ```
 
@@ -323,10 +339,9 @@ Usage resets every **Monday at 00:00 UTC**, just like Claude Code.
 
 | Plan | Price | Weekly Fetches | Burst Limit | Stealth Mode | Extra Usage |
 |------|------:|---------------:|:-----------:|:------------:|:-----------:|
-| **Local CLI** | $0 | ∞ Unlimited | N/A | ✅ | N/A |
-| **Cloud Free** | $0 | 125/wk (~500/mo) | 25/hr | ❌ | ❌ |
-| **Cloud Pro** | $9/mo | 1,250/wk (~5K/mo) | 100/hr | ✅ | ✅ |
-| **Cloud Max** | $29/mo | 6,250/wk (~25K/mo) | 500/hr | ✅ | ✅ |
+| **Free** | $0 | 125/wk (~500/mo) | 25/hr | ❌ | ❌ |
+| **Pro** | $9/mo | 1,250/wk (~5K/mo) | 100/hr | ✅ | ✅ |
+| **Max** | $29/mo | 6,250/wk (~25K/mo) | 500/hr | ✅ | ✅ |
 
 **Three layers of usage control:**
 1. **Burst limit** — Per-hour cap (25/hr free, 100/hr Pro, 500/hr Max) prevents hammering
@@ -342,21 +357,22 @@ Usage resets every **Monday at 00:00 UTC**, just like Claude Code.
 
 ### Why WebPeel Beats Firecrawl
 
-| Feature | WebPeel Local | WebPeel Pro | Firecrawl Hobby |
+| Feature | WebPeel Free | WebPeel Pro | Firecrawl Hobby |
 |---------|:-------------:|:-----------:|:---------------:|
 | **Price** | $0 | $9/mo | $16/mo |
-| **Weekly Fetches** | ∞ | 1,250/wk | ~750/wk |
-| **Rollover** | N/A | ✅ 1 week | ❌ Expire monthly |
-| **Soft Limits** | ✅ Always | ✅ Never locked out | ❌ Hard cut-off |
-| **Extra Usage** | N/A | ✅ Pay-as-you-go | ❌ Upgrade only |
-| **Self-Host** | ✅ MIT | N/A | ❌ AGPL |
+| **Weekly Fetches** | 125/wk | 1,250/wk | ~750/wk |
+| **Rollover** | ❌ | ✅ 1 week | ❌ Expire monthly |
+| **Soft Limits** | ✅ Degrades | ✅ Never locked out | ❌ Hard cut-off |
+| **Extra Usage** | ❌ | ✅ Pay-as-you-go | ❌ Upgrade only |
+| **Self-Host** | ✅ MIT | ✅ MIT | ❌ AGPL |
 
 **Key differentiators:**
+- **Like Claude Code** — Generous free tier (125/week), pay when you need more
 - **Weekly resets** — Your usage refreshes every Monday, not once a month
-- **Soft limits on every tier** — At 100%, we degrade to HTTP-only instead of blocking you
+- **Soft limits on every tier** — At 100%, we degrade gracefully instead of blocking you
 - **Extra usage** — Pro/Max users can toggle on pay-as-you-go with spending caps (no surprise bills)
-- **Rollover** — Unused fetches carry forward 1 week
-- **CLI is always free** — No vendor lock-in. Run unlimited locally forever
+- **First 25 free** — Try it instantly, no signup required
+- **Open source** — MIT licensed, self-host if you want full control
 
 See pricing at [webpeel.dev](https://webpeel.dev/#pricing)
 
@@ -477,7 +493,7 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 - [x] Crawl mode (follow links, respect robots.txt)
 - [ ] PDF extraction
 - [ ] Webhook notifications for monitoring
-- [ ] AI CAPTCHA solving (planned)
+- [ ] Structured data extraction (JSON output)
 
 Vote on features and roadmap at [GitHub Discussions](https://github.com/JakeLiuMe/webpeel/discussions).
 
@@ -486,7 +502,7 @@ Vote on features and roadmap at [GitHub Discussions](https://github.com/JakeLiuM
 ## FAQ
 
 **Q: How is this different from Firecrawl?**  
-A: WebPeel runs locally for free (Firecrawl is cloud-only). We also have smart escalation to avoid burning resources on simple pages.
+A: WebPeel has a more generous free tier (125/week vs Firecrawl's 500 one-time credits) and uses weekly resets like Claude Code. We also have smart escalation to avoid burning resources on simple pages.
 
 **Q: Can I self-host the API server?**  
 A: Yes! Run `npm run serve` to start the API server. See [docs/self-hosting.md](docs/self-hosting.md) (coming soon).
@@ -494,11 +510,11 @@ A: Yes! Run `npm run serve` to start the API server. See [docs/self-hosting.md](
 **Q: Does this violate websites' Terms of Service?**  
 A: WebPeel is a tool — how you use it is up to you. Always check a site's ToS before fetching at scale. We recommend respecting `robots.txt` in your own workflows.
 
-**Q: What about CAPTCHA and Cloudflare?**  
-A: WebPeel handles most Cloudflare challenges automatically via stealth mode. AI-powered CAPTCHA solving is on our roadmap.
+**Q: What about Cloudflare and bot protection?**  
+A: WebPeel handles most Cloudflare challenges automatically via stealth mode (Pro plan). For heavily protected sites, stealth mode uses browser fingerprint randomization to bypass detection.
 
 **Q: Can I use this in production?**  
-A: Yes! The hosted API at `https://webpeel-api.onrender.com` is production-ready with authentication, rate limiting, and usage tracking.
+A: Yes! The hosted API at `https://api.webpeel.dev` is production-ready with authentication, rate limiting, and usage tracking.
 
 ---
 

package/dist/cli-auth.d.ts

@@ -0,0 +1,84 @@
+/**
+ * CLI Authentication & Usage Tracking
+ *
+ * Handles:
+ * - Anonymous usage (25 free fetches)
+ * - API key authentication
+ * - Usage checking against API
+ * - Config file management (~/.webpeel/config.json)
+ */
+interface CLIConfig {
+    apiKey?: string;
+    anonymousUsage: number;
+    lastReset: string;
+    planTier?: string;
+    planCachedAt?: string;
+}
+interface UsageCheckResult {
+    allowed: boolean;
+    message?: string;
+    isAnonymous?: boolean;
+    usageInfo?: {
+        used: number;
+        limit: number;
+        remaining: number;
+    };
+}
+/**
+ * Load config from ~/.webpeel/config.json
+ */
+export declare function loadConfig(): CLIConfig;
+/**
+ * Save config to ~/.webpeel/config.json
+ */
+export declare function saveConfig(config: CLIConfig): void;
+/**
+ * Delete config file
+ */
+export declare function deleteConfig(): void;
+/**
+ * Check usage quota before making a request
+ */
+export declare function checkUsage(): Promise<UsageCheckResult>;
+export type PremiumFeature = 'stealth' | 'crawl' | 'batch';
+/**
+ * Check if user has access to a premium feature.
+ * Returns { allowed: true } for paid users, or a helpful upgrade message.
+ *
+ * Priority:
+ * 1. No API key → blocked (must sign up)
+ * 2. Has API key + cached plan → check plan tier
+ * 3. Has API key + no cache → check API, then cache
+ * 4. API unreachable + cached plan within 7 days → use cache
+ * 5. API unreachable + stale cache → allow gracefully (trust the user)
+ */
+export declare function checkFeatureAccess(feature: PremiumFeature): Promise<{
+    allowed: boolean;
+    message?: string;
+}>;
+/**
+ * Show usage footer after successful fetch (for free/anonymous users only)
+ */
+export declare function showUsageFooter(usageInfo: {
+    used: number;
+    limit: number;
+    remaining: number;
+} | undefined, isAnonymous: boolean, stealth?: boolean): void;
+/**
+ * Prompt user for API key via stdin
+ */
+export declare function promptForApiKey(): Promise<string>;
+/**
+ * Login command - save API key to config
+ */
+export declare function handleLogin(): Promise<void>;
+/**
+ * Logout command - remove API key from config
+ */
+export declare function handleLogout(): void;
+/**
+ * Usage command - show current quota
+ */
+export declare function handleUsage(): Promise<void>;
+export {};
+//# sourceMappingURL=cli-auth.d.ts.map

package/dist/cli-auth.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli-auth.d.ts","sourceRoot":"","sources":["../src/cli-auth.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAcH,UAAU,SAAS;IACjB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,UAAU,gBAAgB;IACxB,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,SAAS,CAAC,EAAE;QACV,IAAI,EAAE,MAAM,CAAC;QACb,KAAK,EAAE,MAAM,CAAC;QACd,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;CACH;AAwBD;;GAEG;AACH,wBAAgB,UAAU,IAAI,SAAS,CAyBtC;AAED;;GAEG;AACH,wBAAgB,UAAU,CAAC,MAAM,EAAE,SAAS,GAAG,IAAI,CAWlD;AAED;;GAEG;AACH,wBAAgB,YAAY,IAAI,IAAI,CAQnC;AAwBD;;GAEG;AACH,wBAAsB,UAAU,IAAI,OAAO,CAAC,gBAAgB,CAAC,CA+F5D;AAGD,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,OAAO,GAAG,OAAO,CAAC;AAQ3D;;;;;;;;;;GAUG;AACH,wBAAsB,kBAAkB,CAAC,OAAO,EAAE,cAAc,GAAG,OAAO,CAAC;IAAE,OAAO,EAAE,OAAO,CAAC;IAAC,OAAO,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC,CAmEjH;AAED;;GAEG;AACH,wBAAgB,eAAe,CAC7B,SAAS,EAAE;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,SAAS,EAAE,MAAM,CAAA;CAAE,GAAG,SAAS,EACzE,WAAW,EAAE,OAAO,EACpB,OAAO,GAAE,OAAe,GACvB,IAAI,CAaN;AAED;;GAEG;AACH,wBAAsB,eAAe,IAAI,OAAO,CAAC,MAAM,CAAC,CAYvD;AAED;;GAEG;AACH,wBAAsB,WAAW,IAAI,OAAO,CAAC,IAAI,CAAC,CAgEjD;AAED;;GAEG;AACH,wBAAgB,YAAY,IAAI,IAAI,CAUnC;AAED;;GAEG;AACH,wBAAsB,WAAW,IAAI,OAAO,CAAC,IAAI,CAAC,CAoEjD"}