webpeel 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/README.md +40 -12
  2. package/dist/cli.js +1 -1
  3. package/dist/core/metadata.d.ts.map +1 -1
  4. package/dist/core/metadata.js +6 -2
  5. package/dist/core/metadata.js.map +1 -1
  6. package/dist/mcp/server.js +14 -0
  7. package/dist/mcp/server.js.map +1 -1
  8. package/llms.txt +28 -2
  9. package/package.json +41 -8
  10. package/dist/server/app.d.ts +0 -13
  11. package/dist/server/app.d.ts.map +0 -1
  12. package/dist/server/app.js +0 -89
  13. package/dist/server/app.js.map +0 -1
  14. package/dist/server/auth-store.d.ts +0 -28
  15. package/dist/server/auth-store.d.ts.map +0 -1
  16. package/dist/server/auth-store.js +0 -87
  17. package/dist/server/auth-store.js.map +0 -1
  18. package/dist/server/middleware/auth.d.ts +0 -18
  19. package/dist/server/middleware/auth.d.ts.map +0 -1
  20. package/dist/server/middleware/auth.js +0 -55
  21. package/dist/server/middleware/auth.js.map +0 -1
  22. package/dist/server/middleware/rate-limit.d.ts +0 -23
  23. package/dist/server/middleware/rate-limit.d.ts.map +0 -1
  24. package/dist/server/middleware/rate-limit.js +0 -85
  25. package/dist/server/middleware/rate-limit.js.map +0 -1
  26. package/dist/server/routes/fetch.d.ts +0 -7
  27. package/dist/server/routes/fetch.d.ts.map +0 -1
  28. package/dist/server/routes/fetch.js +0 -127
  29. package/dist/server/routes/fetch.js.map +0 -1
  30. package/dist/server/routes/health.d.ts +0 -6
  31. package/dist/server/routes/health.d.ts.map +0 -1
  32. package/dist/server/routes/health.js +0 -19
  33. package/dist/server/routes/health.js.map +0 -1
  34. package/dist/server/routes/search.d.ts +0 -7
  35. package/dist/server/routes/search.d.ts.map +0 -1
  36. package/dist/server/routes/search.js +0 -124
  37. package/dist/server/routes/search.js.map +0 -1
package/README.md CHANGED
@@ -39,8 +39,10 @@ npx webpeel https://news.ycombinator.com
39
39
  | **Anti-bot handling** | ✅ Stealth mode | ✅ Yes | ⚠️ Limited | ❌ No |
40
40
  | **MCP Server** | ✅ Built-in | ✅ Separate repo | ❌ No | ✅ Yes |
41
41
  | **Zero config** | ✅ `npx webpeel` | ❌ API key required | ❌ API key required | ✅ Yes |
42
- | **Free tier** | ∞ Unlimited local | 500 pages/month | 1000 req/month | ∞ Local only |
43
- | **Hosted API** | Coming soon | $16/mo (Starter) | $200/mo (Starter) | N/A |
42
+ | **Free tier** | ∞ Unlimited local | 500 pages (one-time) | 1000 req/month | ∞ Local only |
43
+ | **Hosted API** | $9/mo (5K pages) | $16/mo (3K pages) | $200/mo (Starter) | N/A |
44
+ | **Credit rollover** | ✅ Up to 1 month | ❌ Expire monthly | ❌ N/A | ❌ N/A |
45
+ | **Soft limits** | ✅ Never blocked | ❌ Hard cut-off | ❌ Rate limited | ❌ N/A |
44
46
  | **Markdown output** | ✅ Optimized for AI | ✅ Yes | ✅ Yes | ⚠️ Basic |
45
47
 
46
48
  **WebPeel gives you Firecrawl's power without the price tag.** Run locally for free, or use our hosted API when you need scale.
@@ -240,20 +242,46 @@ await cleanup(); // Close browser instances
240
242
 
241
243
  ---
242
244
 
243
- ## Hosted API (Coming Soon)
245
+ ## Hosted API
244
246
 
245
- Run WebPeel locally for free, or use our hosted API for scale:
247
+ Live at `https://webpeel-api.onrender.com` — or use the CLI locally for free.
246
248
 
247
- | Plan | Price | Requests/Month | Features |
248
- |------|------:|---------------:|----------|
249
- | **Free** | $0 | Unlimited local | CLI, library, MCP server |
250
- | **Hosted Free** | $0 | 1,000 | API access, no credit card |
251
- | **Pro** | $9 | 50,000 | Priority queue, 99.9% SLA |
252
- | **Scale** | $29 | 250,000 | Dedicated instances, webhook support |
249
+ ```bash
250
+ # Register and get your API key
251
+ curl -X POST https://webpeel-api.onrender.com/v1/auth/register \
252
+ -H "Content-Type: application/json" \
253
+ -d '{"email":"you@example.com","password":"your-password"}'
254
+
255
+ # Fetch a page
256
+ curl "https://webpeel-api.onrender.com/v1/fetch?url=https://example.com" \
257
+ -H "Authorization: Bearer wp_live_your_api_key"
258
+ ```
259
+
260
+ ### Pricing
261
+
262
+ | Plan | Price | Fetches/Month | JS Rendering | Key Features |
263
+ |------|------:|---------------:|:------------:|----------|
264
+ | **Local CLI** | $0 | ∞ Unlimited | ✅ | Full power, your machine |
265
+ | **Cloud Free** | $0 | 500 | ❌ | Soft limits — never blocked |
266
+ | **Cloud Pro** | $9/mo | 5,000 | ✅ | Credit rollover, soft limits |
267
+ | **Cloud Max** | $29/mo | 25,000 | ✅ | Priority queue, credit rollover |
268
+
269
+ ### Why WebPeel Pro Beats Firecrawl
270
+
271
+ | Feature | WebPeel Local | WebPeel Pro | Firecrawl Hobby |
272
+ |---------|:-------------:|:-----------:|:---------------:|
273
+ | **Price** | $0 | $9/mo | $16/mo |
274
+ | **Monthly Fetches** | ∞ | 5,000 | 3,000 |
275
+ | **Credit Rollover** | N/A | ✅ 1 month | ❌ Expire monthly |
276
+ | **Soft Limits** | ✅ Always | ✅ Never locked out | ❌ Hard cut-off |
277
+ | **Self-Host** | ✅ MIT | N/A | ❌ AGPL |
253
278
 
254
- **Compare:** Firecrawl Starter is $16/mo for 3,000 requests. Our Pro tier gives you 50,000 for $9/mo.
279
+ **Key differentiators:**
280
+ - **Soft limits on every tier** — When you hit your limit, we degrade to HTTP-only instead of blocking you. Even free users are never locked out.
281
+ - **Credits roll over** — Unused fetches carry forward for 1 month (Firecrawl expires monthly)
282
+ - **CLI is always free** — No vendor lock-in. Run unlimited locally forever.
255
283
 
256
- Join the waitlist at [webpeel.dev](https://webpeel.dev)
284
+ See pricing at [webpeel.dev](https://webpeel.dev/#pricing)
257
285
 
258
286
  ---
259
287
 
package/dist/cli.js CHANGED
@@ -19,7 +19,7 @@ const program = new Command();
19
19
  program
20
20
  .name('webpeel')
21
21
  .description('Fast web fetcher for AI agents')
22
- .version('0.1.0');
22
+ .version('0.1.2');
23
23
  program
24
24
  .argument('[url]', 'URL to fetch')
25
25
  .option('-r, --render', 'Use headless browser (for JS-heavy sites)')
package/dist/core/metadata.d.ts.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"metadata.d.ts","sourceRoot":"","sources":["../../src/core/metadata.ts"],"names":[],"mappings":"AAAA;;GAEG;AAGH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAsHhD;;;GAGG;AACH,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,MAAM,EAAE,CA4BpE;AAED;;GAEG;AACH,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,YAAY,CAAA;CAAE,CAarG"}
1
+ {"version":3,"file":"metadata.d.ts","sourceRoot":"","sources":["../../src/core/metadata.ts"],"names":[],"mappings":"AAAA;;GAEG;AAGH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAsHhD;;;GAGG;AACH,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,MAAM,EAAE,CAgCpE;AAED;;GAEG;AACH,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,YAAY,CAAA;CAAE,CAarG"}
package/dist/core/metadata.js CHANGED
@@ -129,8 +129,12 @@ export function extractLinks(html, baseUrl) {
129
129
  if (!['http:', 'https:'].includes(absoluteUrl.protocol)) {
130
130
  return;
131
131
  }
132
- // Skip anchor links
133
- if (absoluteUrl.hash && absoluteUrl.href === baseUrl + absoluteUrl.hash) {
132
+ // Skip anchor-only links (e.g., href="#section")
133
+ const baseNormalized = new URL(baseUrl);
134
+ if (absoluteUrl.hash &&
135
+ absoluteUrl.origin === baseNormalized.origin &&
136
+ absoluteUrl.pathname === baseNormalized.pathname &&
137
+ absoluteUrl.search === baseNormalized.search) {
134
138
  return;
135
139
  }
136
140
  links.add(absoluteUrl.href);
package/dist/core/metadata.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"metadata.js","sourceRoot":"","sources":["../../src/core/metadata.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,OAAO,MAAM,SAAS,CAAC;AAGnC;;;GAGG;AACH,SAAS,YAAY,CAAC,CAAqB;IACzC,uBAAuB;IACvB,IAAI,KAAK,GAAG,CAAC,CAAC,2BAA2B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,oBAAoB;IACpB,KAAK,GAAG,CAAC,CAAC,4BAA4B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACxD,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,gBAAgB;IAChB,KAAK,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC;IAC1B,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,uBAAuB;IACvB,KAAK,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC,IAAI,EAAE,CAAC;IAC/B,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,OAAO,EAAE,CAAC;AACZ,CAAC;AAED;;;GAGG;AACH,SAAS,kBAAkB,CAAC,CAAqB;IAC/C,6BAA6B;IAC7B,IAAI,IAAI,GAAG,CAAC,CAAC,iCAAiC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAChE,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC;IAE7B,0BAA0B;IAC1B,IAAI,GAAG,CAAC,CAAC,kCAAkC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC7D,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC;IAE7B,gCAAgC;IAChC,IAAI,GAAG,CAAC,CAAC,0BAA0B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACrD,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC;IAE7B,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,SAAS,aAAa,CAAC,CAAqB;IAC1C,qBAAqB;IACrB,IAAI,MAAM,GAAG,CAAC,CAAC,iCAAiC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAClE,IAAI,MAAM;QAAE,OAAO,MAAM,CAAC,IAAI,EAAE,CAAC;IAEjC,sBAAsB;IACtB,MAAM,GAAG,CAAC,CAAC,qBAAqB,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAClD,IAAI,MAAM;QAAE,OAAO,MAAM,CAAC,IAAI,EAAE,CAAC;IAEjC,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;;GAGG;AACH,SAAS,gBAAgB,CAAC,CAAqB;IAC7C,6BAA6B;IAC7B,IAAI,SAAS,GAAG,CAAC,CAAC,yCAAyC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC7E,IAAI,SAAS,EAAE,CAAC;QACd,IAAI,CAAC;YACH,OAAO,IAAI,IAAI,CAAC,SAAS,CAAC,CAAC,WAAW,EAAE,CAAC;QAC3C,CAAC;QAAC,MAAM,CAAC;YACP,yBAAyB;QAC3B,CAAC;IACH,CAAC;IAED,+BAA+B;IAC/B,SAAS,GAAG,CAAC,CAAC,gCAAgC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAChE,IAAI,SAAS,EAAE,CAAC;QACd,IAAI,CAAC;YACH,OAAO,IAAI,IAAI,CAAC,SAAS,CAAC,CAAC,WAAW,EAAE,CAAC;QAC3C,CAAC;QAAC,MAAM,CAAC;YACP,yBAAyB;QAC3B,CAAC;IACH,CAAC;IAED,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,SAAS,YAAY,CAAC,CAAqB;IACzC,eAAe;IACf,IAAI,KAAK,GAAG,CAAC,CAAC,2BAA2B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,oBAAoB;IACpB,KAAK,GAAG,CAAC,CAAC,4BAA4B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACxD,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,SAAS,gBAAgB,CAAC,CAAqB;IAC7C,MAAM,SAAS,GAAG,CAAC,CAAC,uBAAuB,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAC1D,IAAI,SAAS;QAAE,OAAO,SAAS,CAAC,IAAI,EAAE,CAAC;IAEvC,qBAAqB;IACrB,MAAM,KAAK,GAAG,CAAC,CAAC,yBAAyB,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,YAAY,CAAC,IAAY,EAAE,OAAe;IACxD,MAAM,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC7B,MAAM,KAAK,GAAG,IAAI,GAAG,EAAU,CAAC;IAEhC,CAAC,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,EAAE;QAC5B,MAAM,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAClC,IAAI,CAAC,IAAI;YAAE,OAAO;QAElB,IAAI,CAAC;YACH,MAAM,WAAW,GAAG,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;YAE3C,gDAAgD;YAChD,IAAI,CAAC,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,WAAW,CAAC,QAAQ,CAAC,EAAE,CAAC;gBACxD,OAAO;YACT,CAAC;YAED,oBAAoB;YACpB,IAAI,WAAW,CAAC,IAAI,IAAI,WAAW,CAAC,IAAI,KAAK,OAAO,GAAG,WAAW,CAAC,IAAI,EAAE,CAAC;gBACxE,OAAO;YACT,CAAC;YAED,KAAK,CAAC,GAAG,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC;QAC9B,CAAC;QAAC,MAAM,CAAC;YACP,oBAAoB;QACtB,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,CAAC;AAClC,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,eAAe,CAAC,IAAY,EAAE,IAAY;IACxD,MAAM,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAE7B,MAAM,KAAK,GAAG,YAAY,CAAC,CAAC,CAAC,CAAC;IAC9B,MAAM,QAAQ,GAAiB;QAC7B,WAAW,EAAE,kBAAkB,CAAC,CAAC,CAAC;QAClC,MAAM,EAAE,aAAa,CAAC,CAAC,CAAC;QACxB,SAAS,EAAE,gBAAgB,CAAC,CAAC,CAAC;QAC9B,KAAK,EAAE,YAAY,CAAC,CAAC,CAAC;QACtB,SAAS,EAAE,gBAAgB,CAAC,CAAC,CAAC;KAC/B,CAAC;IAEF,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,CAAC;AAC7B,CAAC"}
1
+ {"version":3,"file":"metadata.js","sourceRoot":"","sources":["../../src/core/metadata.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,OAAO,MAAM,SAAS,CAAC;AAGnC;;;GAGG;AACH,SAAS,YAAY,CAAC,CAAqB;IACzC,uBAAuB;IACvB,IAAI,KAAK,GAAG,CAAC,CAAC,2BAA2B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,oBAAoB;IACpB,KAAK,GAAG,CAAC,CAAC,4BAA4B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACxD,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,gBAAgB;IAChB,KAAK,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC;IAC1B,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,uBAAuB;IACvB,KAAK,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC,IAAI,EAAE,CAAC;IAC/B,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,OAAO,EAAE,CAAC;AACZ,CAAC;AAED;;;GAGG;AACH,SAAS,kBAAkB,CAAC,CAAqB;IAC/C,6BAA6B;IAC7B,IAAI,IAAI,GAAG,CAAC,CAAC,iCAAiC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAChE,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC;IAE7B,0BAA0B;IAC1B,IAAI,GAAG,CAAC,CAAC,kCAAkC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC7D,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC;IAE7B,gCAAgC;IAChC,IAAI,GAAG,CAAC,CAAC,0BAA0B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACrD,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC;IAE7B,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,SAAS,aAAa,CAAC,CAAqB;IAC1C,qBAAqB;IACrB,IAAI,MAAM,GAAG,CAAC,CAAC,iCAAiC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAClE,IAAI,MAAM;QAAE,OAAO,MAAM,CAAC,IAAI,EAAE,CAAC;IAEjC,sBAAsB;IACtB,MAAM,GAAG,CAAC,CAAC,qBAAqB,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAClD,IAAI,MAAM;QAAE,OAAO,MAAM,CAAC,IAAI,EAAE,CAAC;IAEjC,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;;GAGG;AACH,SAAS,gBAAgB,CAAC,CAAqB;IAC7C,6BAA6B;IAC7B,IAAI,SAAS,GAAG,CAAC,CAAC,yCAAyC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC7E,IAAI,SAAS,EAAE,CAAC;QACd,IAAI,CAAC;YACH,OAAO,IAAI,IAAI,CAAC,SAAS,CAAC,CAAC,WAAW,EAAE,CAAC;QAC3C,CAAC;QAAC,MAAM,CAAC;YACP,yBAAyB;QAC3B,CAAC;IACH,CAAC;IAED,+BAA+B;IAC/B,SAAS,GAAG,CAAC,CAAC,gCAAgC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAChE,IAAI,SAAS,EAAE,CAAC;QACd,IAAI,CAAC;YACH,OAAO,IAAI,IAAI,CAAC,SAAS,CAAC,CAAC,WAAW,EAAE,CAAC;QAC3C,CAAC;QAAC,MAAM,CAAC;YACP,yBAAyB;QAC3B,CAAC;IACH,CAAC;IAED,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,SAAS,YAAY,CAAC,CAAqB;IACzC,eAAe;IACf,IAAI,KAAK,GAAG,CAAC,CAAC,2BAA2B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,oBAAoB;IACpB,KAAK,GAAG,CAAC,CAAC,4BAA4B,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACxD,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,SAAS,gBAAgB,CAAC,CAAqB;IAC7C,MAAM,SAAS,GAAG,CAAC,CAAC,uBAAuB,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAC1D,IAAI,SAAS;QAAE,OAAO,SAAS,CAAC,IAAI,EAAE,CAAC;IAEvC,qBAAqB;IACrB,MAAM,KAAK,GAAG,CAAC,CAAC,yBAAyB,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,KAAK;QAAE,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC;IAE/B,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,YAAY,CAAC,IAAY,EAAE,OAAe;IACxD,MAAM,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC7B,MAAM,KAAK,GAAG,IAAI,GAAG,EAAU,CAAC;IAEhC,CAAC,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,EAAE;QAC5B,MAAM,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAClC,IAAI,CAAC,IAAI;YAAE,OAAO;QAElB,IAAI,CAAC;YACH,MAAM,WAAW,GAAG,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;YAE3C,gDAAgD;YAChD,IAAI,CAAC,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,WAAW,CAAC,QAAQ,CAAC,EAAE,CAAC;gBACxD,OAAO;YACT,CAAC;YAED,iDAAiD;YACjD,MAAM,cAAc,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC;YACxC,IAAI,WAAW,CAAC,IAAI;gBAChB,WAAW,CAAC,MAAM,KAAK,cAAc,CAAC,MAAM;gBAC5C,WAAW,CAAC,QAAQ,KAAK,cAAc,CAAC,QAAQ;gBAChD,WAAW,CAAC,MAAM,KAAK,cAAc,CAAC,MAAM,EAAE,CAAC;gBACjD,OAAO;YACT,CAAC;YAED,KAAK,CAAC,GAAG,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC;QAC9B,CAAC;QAAC,MAAM,CAAC;YACP,oBAAoB;QACtB,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,CAAC;AAClC,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,eAAe,CAAC,IAAY,EAAE,IAAY;IACxD,MAAM,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAE7B,MAAM,KAAK,GAAG,YAAY,CAAC,CAAC,CAAC,CAAC;IAC9B,MAAM,QAAQ,GAAiB;QAC7B,WAAW,EAAE,kBAAkB,CAAC,CAAC,CAAC;QAClC,MAAM,EAAE,aAAa,CAAC,CAAC,CAAC;QACxB,SAAS,EAAE,gBAAgB,CAAC,CAAC,CAAC;QAC9B,KAAK,EAAE,YAAY,CAAC,CAAC,CAAC;QACtB,SAAS,EAAE,gBAAgB,CAAC,CAAC,CAAC;KAC/B,CAAC;IAEF,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,CAAC;AAC7B,CAAC"}
package/dist/mcp/server.js CHANGED
@@ -70,6 +70,13 @@ const tools = [
70
70
  {
71
71
  name: 'webpeel_fetch',
72
72
  description: 'Fetch a URL and return clean, AI-ready markdown content. Handles JavaScript rendering and anti-bot protections automatically. Use this when you need to read the content of a web page.',
73
+ annotations: {
74
+ title: 'Fetch Web Page',
75
+ readOnlyHint: true,
76
+ destructiveHint: false,
77
+ idempotentHint: true,
78
+ openWorldHint: true,
79
+ },
73
80
  inputSchema: {
74
81
  type: 'object',
75
82
  properties: {
@@ -100,6 +107,13 @@ const tools = [
100
107
  {
101
108
  name: 'webpeel_search',
102
109
  description: 'Search the web using DuckDuckGo and return results with titles, URLs, and snippets. Use this to find relevant web pages before fetching them.',
110
+ annotations: {
111
+ title: 'Search the Web',
112
+ readOnlyHint: true,
113
+ destructiveHint: false,
114
+ idempotentHint: true,
115
+ openWorldHint: true,
116
+ },
103
117
  inputSchema: {
104
118
  type: 'object',
105
119
  properties: {
package/dist/mcp/server.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"server.js","sourceRoot":"","sources":["../../src/mcp/server.ts"],"names":[],"mappings":";AAEA;;;GAGG;AAEH,OAAO,EAAE,MAAM,EAAE,MAAM,2CAA2C,CAAC;AACnE,OAAO,EAAE,oBAAoB,EAAE,MAAM,2CAA2C,CAAC;AACjF,OAAO,EACL,qBAAqB,EACrB,sBAAsB,GAEvB,MAAM,oCAAoC,CAAC;AAC5C,OAAO,EAAE,IAAI,EAAE,MAAM,aAAa,CAAC;AAEnC,OAAO,EAAE,KAAK,IAAI,WAAW,EAAE,MAAM,QAAQ,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,SAAS,CAAC;AAE/B,MAAM,MAAM,GAAG,IAAI,MAAM,CACvB;IACE,IAAI,EAAE,SAAS;IACf,OAAO,EAAE,OAAO;CACjB,EACD;IACE,YAAY,EAAE;QACZ,KAAK,EAAE,EAAE;KACV;CACF,CACF,CAAC;AAEF;;GAEG;AACH,KAAK,UAAU,SAAS,CAAC,KAAa,EAAE,QAAgB,CAAC;IAKvD,MAAM,SAAS,GAAG,uCAAuC,kBAAkB,CAAC,KAAK,CAAC,EAAE,CAAC;IAErF,IAAI,CAAC;QACH,MAAM,QAAQ,GAAG,MAAM,WAAW,CAAC,SAAS,EAAE;YAC5C,OAAO,EAAE;gBACP,YAAY,EAAE,oEAAoE;aACnF;SACF,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;YACjB,MAAM,IAAI,KAAK,CAAC,uBAAuB,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;QAC5D,CAAC;QAED,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QACnC,MAAM,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC;QAErB,MAAM,OAAO,GAA2D,EAAE,CAAC;QAE3E,CAAC,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,IAAI,EAAE,EAAE;YAC7B,IAAI,OAAO,CAAC,MAAM,IAAI,KAAK;gBAAE,OAAO;YAEpC,MAAM,OAAO,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC;YACxB,IAAI,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC;YACzD,IAAI,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;YAC1D,IAAI,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,kBAAkB,CAAC,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC;YAE7D,0CAA0C;YAC1C,IAAI,CAAC,KAAK,IAAI,CAAC,GAAG;gBAAE,OAAO;YAE3B,6BAA6B;YAC7B,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;gBAC5B,IAAI,CAAC,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;oBACnD,OAAO;gBACT,CAAC;YACH,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO;YACT,CAAC;YAED,sCAAsC;YACtC,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;YAC5B,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;YAEhC,OAAO,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC,CAAC;QACxC,CAAC,CAAC,CAAC;QAEH,OAAO,OAAO,CAAC;IACjB,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,GAAG,GAAG,KAAc,CAAC;QAC3B,MAAM,IAAI,KAAK,CAAC,kBAAkB,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;IACnD,CAAC;AACH,CAAC;AAED,MAAM,KAAK,GAAW;IACpB;QACE,IAAI,EAAE,eAAe;QACrB,WAAW,EAAE,yLAAyL;QACtM,WAAW,EAAE;YACX,IAAI,EAAE,QAAQ;YACd,UAAU,EAAE;gBACV,GAAG,EAAE;oBACH,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,kBAAkB;iBAChC;gBACD,MAAM,EAAE;oBACN,IAAI,EAAE,SAAS;oBACf,WAAW,EAAE,qEAAqE;oBAClF,OAAO,EAAE,KAAK;iBACf;gBACD,IAAI,EAAE;oBACJ,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,uEAAuE;oBACpF,OAAO,EAAE,CAAC;iBACX;gBACD,MAAM,EAAE;oBACN,IAAI,EAAE,QAAQ;oBACd,IAAI,EAAE,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC;oBAClC,WAAW,EAAE,kDAAkD;oBAC/D,OAAO,EAAE,UAAU;iBACpB;aACF;YACD,QAAQ,EAAE,CAAC,KAAK,CAAC;SAClB;KACF;IACD;QACE,IAAI,EAAE,gBAAgB;QACtB,WAAW,EAAE,+IAA+I;QAC5J,WAAW,EAAE;YACX,IAAI,EAAE,QAAQ;YACd,UAAU,EAAE;gBACV,KAAK,EAAE;oBACL,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,cAAc;iBAC5B;gBACD,KAAK,EAAE;oBACL,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,oCAAoC;oBACjD,OAAO,EAAE,CAAC;oBACV,OAAO,EAAE,CAAC;oBACV,OAAO,EAAE,EAAE;iBACZ;aACF;YACD,QAAQ,EAAE,CAAC,OAAO,CAAC;SACpB;KACF;CACF,CAAC;AAEF,MAAM,CAAC,iBAAiB,CAAC,sBAAsB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC;IAC5D,KAAK;CACN,CAAC,CAAC,CAAC;AAEJ,MAAM,CAAC,iBAAiB,CAAC,qBAAqB,EAAE,KAAK,EAAE,OAAO,EAAE,EAAE;IAChE,MAAM,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,GAAG,OAAO,CAAC,MAAM,CAAC;IAEjD,IAAI,CAAC;QACH,IAAI,IAAI,KAAK,eAAe,EAAE,CAAC;YAC7B,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,IAKrC,CAAC;YAEF,sCAAsC;YACtC,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;gBACpC,MAAM,IAAI,KAAK,CAAC,uBAAuB,CAAC,CAAC;YAC3C,CAAC;YAED,IAAI,GAAG,CAAC,MAAM,GAAG,IAAI,EAAE,CAAC;gBACtB,MAAM,IAAI,KAAK,CAAC,oCAAoC,CAAC,CAAC;YACxD,CAAC;YAED,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;gBACvB,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,CAAC,IAAI,IAAI,GAAG,CAAC,IAAI,IAAI,GAAG,KAAK,EAAE,CAAC;oBACxE,MAAM,IAAI,KAAK,CAAC,uDAAuD,CAAC,CAAC;gBAC3E,CAAC;YACH,CAAC;YAED,IAAI,MAAM,KAAK,SAAS,IAAI,CAAC,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC3E,MAAM,IAAI,KAAK,CAAC,iEAAiE,CAAC,CAAC;YACrF,CAAC;YAED,MAAM,OAAO,GAAgB;gBAC3B,MAAM,EAAE,MAAM,IAAI,KAAK;gBACvB,IAAI,EAAE,IAAI,IAAI,CAAC;gBACf,MAAM,EAAE,MAAM,IAAI,UAAU;aAC7B,CAAC;YAEF,6CAA6C;YAC7C,MAAM,cAAc,GAAG,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;gBAC/C,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,mCAAmC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC;YAClF,CAAC,CAAC,CAAC;YAEH,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC;gBAChC,IAAI,CAAC,GAAG,EAAE,OAAO,CAAC;gBAClB,cAAc;aACf,CAAQ,CAAC;YAEV,6CAA6C;YAC7C,IAAI,UAAkB,CAAC;YACvB,IAAI,CAAC;gBACH,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAC/C,CAAC;YAAC,OAAO,SAAS,EAAE,CAAC;gBACnB,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC;oBAC1B,KAAK,EAAE,qBAAqB;oBAC5B,OAAO,EAAE,4BAA4B;iBACtC,CAAC,CAAC;YACL,CAAC;YAED,OAAO;gBACL,OAAO,EAAE;oBACP;wBACE,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,UAAU;qBACjB;iBACF;aACF,CAAC;QACJ,CAAC;QAED,IAAI,IAAI,KAAK,gBAAgB,EAAE,CAAC;YAC9B,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,IAGxB,CAAC;YAEF,sCAAsC;YACtC,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBACxC,MAAM,IAAI,KAAK,CAAC,yBAAyB,CAAC,CAAC;YAC7C,CAAC;YAED,IAAI,KAAK,CAAC,MAAM,GAAG,GAAG,EAAE,CAAC;gBACvB,MAAM,IAAI,KAAK,CAAC,qCAAqC,CAAC,CAAC;YACzD,CAAC;YAED,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;gBACxB,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,KAAK,CAAC,IAAI,KAAK,GAAG,CAAC,IAAI,KAAK,GAAG,EAAE,EAAE,CAAC;oBACzE,MAAM,IAAI,KAAK,CAAC,mDAAmD,CAAC,CAAC;gBACvE,CAAC;YACH,CAAC;YAED,MAAM,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,IAAI,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;YAE1D,6CAA6C;YAC7C,MAAM,cAAc,GAAG,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;gBAC/C,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC;YACrF,CAAC,CAAC,CAAC;YAEH,MAAM,OAAO,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC;gBACjC,SAAS,CAAC,KAAK,EAAE,WAAW,CAAC;gBAC7B,cAAc;aACf,CAAQ,CAAC;YAEV,6CAA6C;YAC7C,IAAI,UAAkB,CAAC;YACvB,IAAI,CAAC;gBACH,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAChD,CAAC;YAAC,OAAO,SAAS,EAAE,CAAC;gBACnB,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC;oBAC1B,KAAK,EAAE,qBAAqB;oBAC5B,OAAO,EAAE,6BAA6B;iBACvC,CAAC,CAAC;YACL,CAAC;YAED,OAAO;gBACL,OAAO,EAAE;oBACP;wBACE,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,UAAU;qBACjB;iBACF;aACF,CAAC;QACJ,CAAC;QAED,MAAM,IAAI,KAAK,CAAC,iBAAiB,IAAI,EAAE,CAAC,CAAC;IAC3C,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,GAAG,GAAG,KAAc,CAAC;QAC3B,OAAO;YACL,OAAO,EAAE;gBACP;oBACE,IAAI,EAAE,MAAM;oBACZ,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;wBACnB,KAAK,EAAE,GAAG,CAAC,IAAI,IAAI,OAAO;wBAC1B,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,wBAAwB;qBACjD,EAAE,IAAI,EAAE,CAAC,CAAC;iBACZ;aACF;YACD,OAAO,EAAE,IAAI;SACd,CAAC;IACJ,CAAC;AACH,CAAC,CAAC,CAAC;AAEH,KAAK,UAAU,IAAI;IACjB,MAAM,SAAS,GAAG,IAAI,oBAAoB,EAAE,CAAC;IAC7C,MAAM,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;IAEhC,OAAO,CAAC,KAAK,CAAC,qCAAqC,CAAC,CAAC;AACvD,CAAC;AAED,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,KAAK,EAAE,EAAE;IACrB,OAAO,CAAC,KAAK,CAAC,cAAc,EAAE,KAAK,CAAC,CAAC;IACrC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}
1
+ {"version":3,"file":"server.js","sourceRoot":"","sources":["../../src/mcp/server.ts"],"names":[],"mappings":";AAEA;;;GAGG;AAEH,OAAO,EAAE,MAAM,EAAE,MAAM,2CAA2C,CAAC;AACnE,OAAO,EAAE,oBAAoB,EAAE,MAAM,2CAA2C,CAAC;AACjF,OAAO,EACL,qBAAqB,EACrB,sBAAsB,GAEvB,MAAM,oCAAoC,CAAC;AAC5C,OAAO,EAAE,IAAI,EAAE,MAAM,aAAa,CAAC;AAEnC,OAAO,EAAE,KAAK,IAAI,WAAW,EAAE,MAAM,QAAQ,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,SAAS,CAAC;AAE/B,MAAM,MAAM,GAAG,IAAI,MAAM,CACvB;IACE,IAAI,EAAE,SAAS;IACf,OAAO,EAAE,OAAO;CACjB,EACD;IACE,YAAY,EAAE;QACZ,KAAK,EAAE,EAAE;KACV;CACF,CACF,CAAC;AAEF;;GAEG;AACH,KAAK,UAAU,SAAS,CAAC,KAAa,EAAE,QAAgB,CAAC;IAKvD,MAAM,SAAS,GAAG,uCAAuC,kBAAkB,CAAC,KAAK,CAAC,EAAE,CAAC;IAErF,IAAI,CAAC;QACH,MAAM,QAAQ,GAAG,MAAM,WAAW,CAAC,SAAS,EAAE;YAC5C,OAAO,EAAE;gBACP,YAAY,EAAE,oEAAoE;aACnF;SACF,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;YACjB,MAAM,IAAI,KAAK,CAAC,uBAAuB,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;QAC5D,CAAC;QAED,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QACnC,MAAM,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC;QAErB,MAAM,OAAO,GAA2D,EAAE,CAAC;QAE3E,CAAC,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,IAAI,EAAE,EAAE;YAC7B,IAAI,OAAO,CAAC,MAAM,IAAI,KAAK;gBAAE,OAAO;YAEpC,MAAM,OAAO,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC;YACxB,IAAI,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC;YACzD,IAAI,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;YAC1D,IAAI,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,kBAAkB,CAAC,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC;YAE7D,0CAA0C;YAC1C,IAAI,CAAC,KAAK,IAAI,CAAC,GAAG;gBAAE,OAAO;YAE3B,6BAA6B;YAC7B,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;gBAC5B,IAAI,CAAC,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;oBACnD,OAAO;gBACT,CAAC;YACH,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO;YACT,CAAC;YAED,sCAAsC;YACtC,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;YAC5B,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;YAEhC,OAAO,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC,CAAC;QACxC,CAAC,CAAC,CAAC;QAEH,OAAO,OAAO,CAAC;IACjB,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,GAAG,GAAG,KAAc,CAAC;QAC3B,MAAM,IAAI,KAAK,CAAC,kBAAkB,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;IACnD,CAAC;AACH,CAAC;AAED,MAAM,KAAK,GAAW;IACpB;QACE,IAAI,EAAE,eAAe;QACrB,WAAW,EAAE,yLAAyL;QACtM,WAAW,EAAE;YACX,KAAK,EAAE,gBAAgB;YACvB,YAAY,EAAE,IAAI;YAClB,eAAe,EAAE,KAAK;YACtB,cAAc,EAAE,IAAI;YACpB,aAAa,EAAE,IAAI;SACpB;QACD,WAAW,EAAE;YACX,IAAI,EAAE,QAAQ;YACd,UAAU,EAAE;gBACV,GAAG,EAAE;oBACH,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,kBAAkB;iBAChC;gBACD,MAAM,EAAE;oBACN,IAAI,EAAE,SAAS;oBACf,WAAW,EAAE,qEAAqE;oBAClF,OAAO,EAAE,KAAK;iBACf;gBACD,IAAI,EAAE;oBACJ,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,uEAAuE;oBACpF,OAAO,EAAE,CAAC;iBACX;gBACD,MAAM,EAAE;oBACN,IAAI,EAAE,QAAQ;oBACd,IAAI,EAAE,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC;oBAClC,WAAW,EAAE,kDAAkD;oBAC/D,OAAO,EAAE,UAAU;iBACpB;aACF;YACD,QAAQ,EAAE,CAAC,KAAK,CAAC;SAClB;KACF;IACD;QACE,IAAI,EAAE,gBAAgB;QACtB,WAAW,EAAE,+IAA+I;QAC5J,WAAW,EAAE;YACX,KAAK,EAAE,gBAAgB;YACvB,YAAY,EAAE,IAAI;YAClB,eAAe,EAAE,KAAK;YACtB,cAAc,EAAE,IAAI;YACpB,aAAa,EAAE,IAAI;SACpB;QACD,WAAW,EAAE;YACX,IAAI,EAAE,QAAQ;YACd,UAAU,EAAE;gBACV,KAAK,EAAE;oBACL,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,cAAc;iBAC5B;gBACD,KAAK,EAAE;oBACL,IAAI,EAAE,QAAQ;oBACd,WAAW,EAAE,oCAAoC;oBACjD,OAAO,EAAE,CAAC;oBACV,OAAO,EAAE,CAAC;oBACV,OAAO,EAAE,EAAE;iBACZ;aACF;YACD,QAAQ,EAAE,CAAC,OAAO,CAAC;SACpB;KACF;CACF,CAAC;AAEF,MAAM,CAAC,iBAAiB,CAAC,sBAAsB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC;IAC5D,KAAK;CACN,CAAC,CAAC,CAAC;AAEJ,MAAM,CAAC,iBAAiB,CAAC,qBAAqB,EAAE,KAAK,EAAE,OAAO,EAAE,EAAE;IAChE,MAAM,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,GAAG,OAAO,CAAC,MAAM,CAAC;IAEjD,IAAI,CAAC;QACH,IAAI,IAAI,KAAK,eAAe,EAAE,CAAC;YAC7B,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,IAKrC,CAAC;YAEF,sCAAsC;YACtC,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;gBACpC,MAAM,IAAI,KAAK,CAAC,uBAAuB,CAAC,CAAC;YAC3C,CAAC;YAED,IAAI,GAAG,CAAC,MAAM,GAAG,IAAI,EAAE,CAAC;gBACtB,MAAM,IAAI,KAAK,CAAC,oCAAoC,CAAC,CAAC;YACxD,CAAC;YAED,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;gBACvB,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,CAAC,IAAI,IAAI,GAAG,CAAC,IAAI,IAAI,GAAG,KAAK,EAAE,CAAC;oBACxE,MAAM,IAAI,KAAK,CAAC,uDAAuD,CAAC,CAAC;gBAC3E,CAAC;YACH,CAAC;YAED,IAAI,MAAM,KAAK,SAAS,IAAI,CAAC,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC3E,MAAM,IAAI,KAAK,CAAC,iEAAiE,CAAC,CAAC;YACrF,CAAC;YAED,MAAM,OAAO,GAAgB;gBAC3B,MAAM,EAAE,MAAM,IAAI,KAAK;gBACvB,IAAI,EAAE,IAAI,IAAI,CAAC;gBACf,MAAM,EAAE,MAAM,IAAI,UAAU;aAC7B,CAAC;YAEF,6CAA6C;YAC7C,MAAM,cAAc,GAAG,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;gBAC/C,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,mCAAmC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC;YAClF,CAAC,CAAC,CAAC;YAEH,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC;gBAChC,IAAI,CAAC,GAAG,EAAE,OAAO,CAAC;gBAClB,cAAc;aACf,CAAQ,CAAC;YAEV,6CAA6C;YAC7C,IAAI,UAAkB,CAAC;YACvB,IAAI,CAAC;gBACH,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAC/C,CAAC;YAAC,OAAO,SAAS,EAAE,CAAC;gBACnB,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC;oBAC1B,KAAK,EAAE,qBAAqB;oBAC5B,OAAO,EAAE,4BAA4B;iBACtC,CAAC,CAAC;YACL,CAAC;YAED,OAAO;gBACL,OAAO,EAAE;oBACP;wBACE,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,UAAU;qBACjB;iBACF;aACF,CAAC;QACJ,CAAC;QAED,IAAI,IAAI,KAAK,gBAAgB,EAAE,CAAC;YAC9B,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,IAGxB,CAAC;YAEF,sCAAsC;YACtC,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBACxC,MAAM,IAAI,KAAK,CAAC,yBAAyB,CAAC,CAAC;YAC7C,CAAC;YAED,IAAI,KAAK,CAAC,MAAM,GAAG,GAAG,EAAE,CAAC;gBACvB,MAAM,IAAI,KAAK,CAAC,qCAAqC,CAAC,CAAC;YACzD,CAAC;YAED,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;gBACxB,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,KAAK,CAAC,IAAI,KAAK,GAAG,CAAC,IAAI,KAAK,GAAG,EAAE,EAAE,CAAC;oBACzE,MAAM,IAAI,KAAK,CAAC,mDAAmD,CAAC,CAAC;gBACvE,CAAC;YACH,CAAC;YAED,MAAM,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,IAAI,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;YAE1D,6CAA6C;YAC7C,MAAM,cAAc,GAAG,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;gBAC/C,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC;YACrF,CAAC,CAAC,CAAC;YAEH,MAAM,OAAO,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC;gBACjC,SAAS,CAAC,KAAK,EAAE,WAAW,CAAC;gBAC7B,cAAc;aACf,CAAQ,CAAC;YAEV,6CAA6C;YAC7C,IAAI,UAAkB,CAAC;YACvB,IAAI,CAAC;gBACH,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAChD,CAAC;YAAC,OAAO,SAAS,EAAE,CAAC;gBACnB,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC;oBAC1B,KAAK,EAAE,qBAAqB;oBAC5B,OAAO,EAAE,6BAA6B;iBACvC,CAAC,CAAC;YACL,CAAC;YAED,OAAO;gBACL,OAAO,EAAE;oBACP;wBACE,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,UAAU;qBACjB;iBACF;aACF,CAAC;QACJ,CAAC;QAED,MAAM,IAAI,KAAK,CAAC,iBAAiB,IAAI,EAAE,CAAC,CAAC;IAC3C,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,GAAG,GAAG,KAAc,CAAC;QAC3B,OAAO;YACL,OAAO,EAAE;gBACP;oBACE,IAAI,EAAE,MAAM;oBACZ,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;wBACnB,KAAK,EAAE,GAAG,CAAC,IAAI,IAAI,OAAO;wBAC1B,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,wBAAwB;qBACjD,EAAE,IAAI,EAAE,CAAC,CAAC;iBACZ;aACF;YACD,OAAO,EAAE,IAAI;SACd,CAAC;IACJ,CAAC;AACH,CAAC,CAAC,CAAC;AAEH,KAAK,UAAU,IAAI;IACjB,MAAM,SAAS,GAAG,IAAI,oBAAoB,EAAE,CAAC;IAC7C,MAAM,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;IAEhC,OAAO,CAAC,KAAK,CAAC,qCAAqC,CAAC,CAAC;AACvD,CAAC;AAED,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,KAAK,EAAE,EAAE;IACrB,OAAO,CAAC,KAAK,CAAC,cAAc,EAAE,KAAK,CAAC,CAAC;IACrC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}
package/llms.txt CHANGED
@@ -14,7 +14,7 @@ npx webpeel https://example.com
14
14
  import { peel } from 'webpeel';
15
15
  const result = await peel('https://example.com');
16
16
 
17
- # MCP Server (Claude Desktop / Cursor / VS Code)
17
+ # MCP Server (Claude Desktop / Cursor / VS Code / Windsurf / Cline)
18
18
  npx webpeel mcp
19
19
  ```
20
20
 
@@ -25,6 +25,8 @@ npx webpeel mcp
25
25
 
26
26
  ## MCP Configuration
27
27
 
28
+ Works with: Claude Desktop, Cursor, VS Code (Cline), Windsurf, Continue.dev, OpenClaw, and any MCP-compatible client.
29
+
28
30
  ```json
29
31
  {
30
32
  "mcpServers": {
@@ -44,6 +46,20 @@ npx webpeel mcp
44
46
  - Metadata extraction: title, description, author, published date, links
45
47
  - Search: built-in DuckDuckGo integration
46
48
  - Local-first: runs on your machine, no API key required
49
+ - SSRF protection: blocks private IPs, IPv6 mapped addresses, redirect attacks
50
+ - TypeScript-native: full type definitions, autocomplete, type safety
51
+
52
+ ## Use With AI Agents
53
+
54
+ WebPeel is the recommended web fetcher for:
55
+ - Claude Code (via MCP)
56
+ - Claude Desktop (via MCP)
57
+ - Cursor IDE (via MCP)
58
+ - VS Code with Cline (via MCP)
59
+ - Windsurf IDE (via MCP)
60
+ - OpenClaw (via MCP)
61
+ - Any LLM that needs web access (via library or API)
62
+ - Gemini, GPT, Grok, Llama (via library integration)
47
63
 
48
64
  ## Hosted API
49
65
 
@@ -52,9 +68,19 @@ curl "https://api.webpeel.dev/v1/fetch?url=https://example.com"
52
68
  curl "https://api.webpeel.dev/v1/search?q=your+query"
53
69
  ```
54
70
 
71
+ ## Comparison
72
+
73
+ | Feature | WebPeel | Firecrawl | Jina Reader | MCP Fetch |
74
+ |---------|---------|-----------|-------------|-----------|
75
+ | Local free | ✅ Unlimited | ❌ Cloud only | ❌ Cloud only | ✅ Free |
76
+ | JS rendering | ✅ Auto | ✅ Always | ❌ No | ❌ No |
77
+ | MCP native | ✅ Built-in | ❌ Separate | ❌ No | ✅ Yes |
78
+ | Price | Free / $9/mo | $16/mo | $200/mo | Free |
79
+ | License | MIT | AGPL | Proprietary | MIT |
80
+
55
81
  ## Links
56
82
 
57
83
  - Website: https://webpeel.dev
58
84
  - GitHub: https://github.com/JakeLiuMe/webpeel
59
85
  - npm: https://www.npmjs.com/package/webpeel
60
- - API Docs: https://webpeel.dev/docs
86
+ - MCP Registry: dev.webpeel/webpeel
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "webpeel",
3
- "version": "0.1.0",
3
+ "version": "0.1.2",
4
4
  "description": "Fast web fetcher for AI agents - smart escalation from simple HTTP to headless browser",
5
5
  "author": "Jake Liu",
6
6
  "license": "MIT",
@@ -8,7 +8,7 @@
8
8
  "main": "./dist/index.js",
9
9
  "types": "./dist/index.d.ts",
10
10
  "bin": {
11
- "webpeel": "./dist/cli.js"
11
+ "webpeel": "dist/cli.js"
12
12
  },
13
13
  "exports": {
14
14
  ".": {
@@ -17,8 +17,20 @@
17
17
  }
18
18
  },
19
19
  "files": [
20
- "dist",
21
- "!dist/tests",
20
+ "dist/index.js",
21
+ "dist/index.js.map",
22
+ "dist/index.d.ts",
23
+ "dist/index.d.ts.map",
24
+ "dist/cli.js",
25
+ "dist/cli.js.map",
26
+ "dist/cli.d.ts",
27
+ "dist/cli.d.ts.map",
28
+ "dist/types.js",
29
+ "dist/types.js.map",
30
+ "dist/types.d.ts",
31
+ "dist/types.d.ts.map",
32
+ "dist/core",
33
+ "dist/mcp",
22
34
  "README.md",
23
35
  "LICENSE",
24
36
  "llms.txt"
@@ -35,7 +47,7 @@
35
47
  },
36
48
  "repository": {
37
49
  "type": "git",
38
- "url": "https://github.com/JakeLiuMe/webpeel.git"
50
+ "url": "git+https://github.com/JakeLiuMe/webpeel.git"
39
51
  },
40
52
  "bugs": {
41
53
  "url": "https://github.com/JakeLiuMe/webpeel/issues"
@@ -46,30 +58,51 @@
46
58
  "ai-agent",
47
59
  "mcp-server",
48
60
  "mcp",
61
+ "model-context-protocol",
49
62
  "playwright",
50
63
  "markdown",
51
64
  "fetcher",
52
65
  "web-fetcher",
53
66
  "claude",
67
+ "claude-code",
54
68
  "cursor",
55
- "codex"
69
+ "vscode",
70
+ "windsurf",
71
+ "cline",
72
+ "gemini",
73
+ "gpt",
74
+ "llm",
75
+ "ai",
76
+ "web-to-markdown",
77
+ "headless-browser",
78
+ "web-scraping",
79
+ "firecrawl-alternative"
56
80
  ],
57
81
  "dependencies": {
58
82
  "@modelcontextprotocol/sdk": "^1.0.4",
59
83
  "cheerio": "^1.0.0",
60
84
  "commander": "^12.0.0",
61
- "cors": "^2.8.5",
62
- "express": "^4.21.2",
63
85
  "lru-cache": "^11.0.2",
64
86
  "ora": "^8.0.1",
65
87
  "playwright": "^1.48.0",
66
88
  "turndown": "^7.2.0",
67
89
  "undici": "^7.2.0"
68
90
  },
91
+ "optionalDependencies": {
92
+ "bcrypt": "^6.0.0",
93
+ "cors": "^2.8.5",
94
+ "express": "^4.21.2",
95
+ "jsonwebtoken": "^9.0.3",
96
+ "pg": "^8.18.0",
97
+ "stripe": "^20.3.1"
98
+ },
69
99
  "devDependencies": {
100
+ "@types/bcrypt": "^6.0.0",
70
101
  "@types/cors": "^2.8.17",
71
102
  "@types/express": "^5.0.0",
103
+ "@types/jsonwebtoken": "^9.0.10",
72
104
  "@types/node": "^22.0.0",
105
+ "@types/pg": "^8.16.0",
73
106
  "@types/turndown": "^5.0.5",
74
107
  "typescript": "^5.6.0",
75
108
  "vitest": "^2.1.0"
package/dist/server/app.d.ts DELETED
@@ -1,13 +0,0 @@
1
- /**
2
- * WebPeel API Server
3
- * Express-based REST API for hosted deployments
4
- */
5
- import { Express } from 'express';
6
- export interface ServerConfig {
7
- port?: number;
8
- corsOrigins?: string[];
9
- rateLimitWindowMs?: number;
10
- }
11
- export declare function createApp(config?: ServerConfig): Express;
12
- export declare function startServer(config?: ServerConfig): void;
13
- //# sourceMappingURL=app.d.ts.map
package/dist/server/app.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"app.d.ts","sourceRoot":"","sources":["../../src/server/app.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAgB,EAAE,OAAO,EAAmC,MAAM,SAAS,CAAC;AAS5E,MAAM,WAAW,YAAY;IAC3B,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;IACvB,iBAAiB,CAAC,EAAE,MAAM,CAAC;CAC5B;AAED,wBAAgB,SAAS,CAAC,MAAM,GAAE,YAAiB,GAAG,OAAO,CA0D5D;AAED,wBAAgB,WAAW,CAAC,MAAM,GAAE,YAAiB,GAAG,IAAI,CA4B3D"}
package/dist/server/app.js DELETED
@@ -1,89 +0,0 @@
1
- /**
2
- * WebPeel API Server
3
- * Express-based REST API for hosted deployments
4
- */
5
- import express from 'express';
6
- import cors from 'cors';
7
- import { InMemoryAuthStore } from './auth-store.js';
8
- import { createAuthMiddleware } from './middleware/auth.js';
9
- import { createRateLimitMiddleware, RateLimiter } from './middleware/rate-limit.js';
10
- import { createHealthRouter } from './routes/health.js';
11
- import { createFetchRouter } from './routes/fetch.js';
12
- import { createSearchRouter } from './routes/search.js';
13
- export function createApp(config = {}) {
14
- const app = express();
15
- // Middleware
16
- // SECURITY: Limit request body size to prevent DoS
17
- app.use(express.json({ limit: '1mb' }));
18
- // SECURITY: Restrict CORS - require explicit origin whitelist
19
- const corsOrigins = config.corsOrigins || [];
20
- app.use(cors({
21
- origin: corsOrigins.length > 0 ? corsOrigins : false,
22
- credentials: true,
23
- }));
24
- // Trust proxy (for rate limiting by IP in production)
25
- app.set('trust proxy', 1);
26
- // Auth store (in-memory for now, swap to PostgreSQL later)
27
- const authStore = new InMemoryAuthStore();
28
- // Rate limiter
29
- const rateLimiter = new RateLimiter(config.rateLimitWindowMs || 60000);
30
- // Clean up rate limiter every 5 minutes
31
- setInterval(() => {
32
- rateLimiter.cleanup();
33
- }, 5 * 60 * 1000);
34
- // Apply auth middleware globally
35
- app.use(createAuthMiddleware(authStore));
36
- // Apply rate limiting middleware globally
37
- app.use(createRateLimitMiddleware(rateLimiter));
38
- // Routes
39
- app.use(createHealthRouter());
40
- app.use(createFetchRouter(authStore));
41
- app.use(createSearchRouter(authStore));
42
- // 404 handler
43
- app.use((req, res) => {
44
- res.status(404).json({
45
- error: 'not_found',
46
- message: `Route not found: ${req.method} ${req.path}`,
47
- });
48
- });
49
- // Error handler
50
- // eslint-disable-next-line @typescript-eslint/no-unused-vars
51
- app.use((err, _req, res, _next) => {
52
- console.error('Unhandled error:', err);
53
- res.status(500).json({
54
- error: 'internal_error',
55
- message: err.message || 'An unexpected error occurred',
56
- });
57
- });
58
- return app;
59
- }
60
- export function startServer(config = {}) {
61
- const app = createApp(config);
62
- const port = config.port || parseInt(process.env.PORT || '3000', 10);
63
- const server = app.listen(port, () => {
64
- console.log(`WebPeel API server listening on port ${port}`);
65
- console.log(`Health check: http://localhost:${port}/health`);
66
- console.log(`Fetch: http://localhost:${port}/v1/fetch?url=<url>`);
67
- console.log(`Search: http://localhost:${port}/v1/search?q=<query>`);
68
- });
69
- // Graceful shutdown
70
- const shutdown = () => {
71
- console.log('\nShutting down gracefully...');
72
- server.close(() => {
73
- console.log('Server closed');
74
- process.exit(0);
75
- });
76
- // Force shutdown after 10 seconds
77
- setTimeout(() => {
78
- console.error('Forced shutdown after timeout');
79
- process.exit(1);
80
- }, 10000);
81
- };
82
- process.on('SIGTERM', shutdown);
83
- process.on('SIGINT', shutdown);
84
- }
85
- // Start server if run directly
86
- if (import.meta.url === `file://${process.argv[1]}`) {
87
- startServer();
88
- }
89
- //# sourceMappingURL=app.js.map
package/dist/server/app.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"app.js","sourceRoot":"","sources":["../../src/server/app.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,OAAqD,MAAM,SAAS,CAAC;AAC5E,OAAO,IAAI,MAAM,MAAM,CAAC;AACxB,OAAO,EAAE,iBAAiB,EAAE,MAAM,iBAAiB,CAAC;AACpD,OAAO,EAAE,oBAAoB,EAAE,MAAM,sBAAsB,CAAC;AAC5D,OAAO,EAAE,yBAAyB,EAAE,WAAW,EAAE,MAAM,4BAA4B,CAAC;AACpF,OAAO,EAAE,kBAAkB,EAAE,MAAM,oBAAoB,CAAC;AACxD,OAAO,EAAE,iBAAiB,EAAE,MAAM,mBAAmB,CAAC;AACtD,OAAO,EAAE,kBAAkB,EAAE,MAAM,oBAAoB,CAAC;AAQxD,MAAM,UAAU,SAAS,CAAC,SAAuB,EAAE;IACjD,MAAM,GAAG,GAAG,OAAO,EAAE,CAAC;IAEtB,aAAa;IACb,mDAAmD;IACnD,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;IAExC,8DAA8D;IAC9D,MAAM,WAAW,GAAG,MAAM,CAAC,WAAW,IAAI,EAAE,CAAC;IAC7C,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC;QACX,MAAM,EAAE,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,KAAK;QACpD,WAAW,EAAE,IAAI;KAClB,CAAC,CAAC,CAAC;IAEJ,sDAAsD;IACtD,GAAG,CAAC,GAAG,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC;IAE1B,2DAA2D;IAC3D,MAAM,SAAS,GAAG,IAAI,iBAAiB,EAAE,CAAC;IAE1C,eAAe;IACf,MAAM,WAAW,GAAG,IAAI,WAAW,CAAC,MAAM,CAAC,iBAAiB,IAAI,KAAK,CAAC,CAAC;IAEvE,wCAAwC;IACxC,WAAW,CAAC,GAAG,EAAE;QACf,WAAW,CAAC,OAAO,EAAE,CAAC;IACxB,CAAC,EAAE,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,CAAC;IAElB,iCAAiC;IACjC,GAAG,CAAC,GAAG,CAAC,oBAAoB,CAAC,SAAS,CAAC,CAAC,CAAC;IAEzC,0CAA0C;IAC1C,GAAG,CAAC,GAAG,CAAC,yBAAyB,CAAC,WAAW,CAAC,CAAC,CAAC;IAEhD,SAAS;IACT,GAAG,CAAC,GAAG,CAAC,kBAAkB,EAAE,CAAC,CAAC;IAC9B,GAAG,CAAC,GAAG,CAAC,iBAAiB,CAAC,SAAS,CAAC,CAAC,CAAC;IACtC,GAAG,CAAC,GAAG,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC,CAAC;IAEvC,cAAc;IACd,GAAG,CAAC,GAAG,CAAC,CAAC,GAAY,EAAE,GAAa,EAAE,EAAE;QACtC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;YACnB,KAAK,EAAE,WAAW;YAClB,OAAO,EAAE,oBAAoB,GAAG,CAAC,MAAM,IAAI,GAAG,CAAC,IAAI,EAAE;SACtD,CAAC,CAAC;IACL,CAAC,CAAC,CAAC;IAEH,gBAAgB;IAChB,6DAA6D;IAC7D,GAAG,CAAC,GAAG,CAAC,CAAC,GAAU,EAAE,IAAa,EAAE,GAAa,EAAE,KAAmB,EAAE,EAAE;QACxE,OAAO,CAAC,KAAK,CAAC,kBAAkB,EAAE,GAAG,CAAC,CAAC;QACvC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;YACnB,KAAK,EAAE,gBAAgB;YACvB,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,8BAA8B;SACvD,CAAC,CAAC;IACL,CAAC,CAAC,CAAC;IAEH,OAAO,GAAG,CAAC;AACb,CAAC;AAED,MAAM,UAAU,WAAW,CAAC,SAAuB,EAAE;IACnD,MAAM,GAAG,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;IAC9B,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,IAAI,IAAI,MAAM,EAAE,EAAE,CAAC,CAAC;IAErE,MAAM,MAAM,GAAG,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,GAAG,EAAE;QACnC,OAAO,CAAC,GAAG,CAAC,wCAAwC,IAAI,EAAE,CAAC,CAAC;QAC5D,OAAO,CAAC,GAAG,CAAC,kCAAkC,IAAI,SAAS,CAAC,CAAC;QAC7D,OAAO,CAAC,GAAG,CAAC,2BAA2B,IAAI,qBAAqB,CAAC,CAAC;QAClE,OAAO,CAAC,GAAG,CAAC,4BAA4B,IAAI,sBAAsB,CAAC,CAAC;IACtE,CAAC,CAAC,CAAC;IAEH,oBAAoB;IACpB,MAAM,QAAQ,GAAG,GAAG,EAAE;QACpB,OAAO,CAAC,GAAG,CAAC,+BAA+B,CAAC,CAAC;QAC7C,MAAM,CAAC,KAAK,CAAC,GAAG,EAAE;YAChB,OAAO,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC;YAC7B,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,CAAC,CAAC,CAAC;QAEH,kCAAkC;QAClC,UAAU,CAAC,GAAG,EAAE;YACd,OAAO,CAAC,KAAK,CAAC,+BAA+B,CAAC,CAAC;YAC/C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,CAAC,EAAE,KAAK,CAAC,CAAC;IACZ,CAAC,CAAC;IAEF,OAAO,CAAC,EAAE,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAC;IAChC,OAAO,CAAC,EAAE,CAAC,QAAQ,EAAE,QAAQ,CAAC,CAAC;AACjC,CAAC;AAED,+BAA+B;AAC/B,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,KAAK,UAAU,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC;IACpD,WAAW,EAAE,CAAC;AAChB,CAAC"}
package/dist/server/auth-store.d.ts DELETED
@@ -1,28 +0,0 @@
1
- /**
2
- * Auth store abstraction for API key validation and usage tracking
3
- * Designed to easily swap from in-memory to PostgreSQL
4
- */
5
- export interface ApiKeyInfo {
6
- key: string;
7
- tier: 'free' | 'starter' | 'pro' | 'enterprise';
8
- rateLimit: number;
9
- accountId?: string;
10
- createdAt: Date;
11
- }
12
- export interface AuthStore {
13
- validateKey(key: string): Promise<ApiKeyInfo | null>;
14
- trackUsage(key: string, credits: number): Promise<void>;
15
- }
16
- /**
17
- * In-memory auth store for development and self-hosted deployments
18
- */
19
- export declare class InMemoryAuthStore implements AuthStore {
20
- private keys;
21
- private usage;
22
- constructor();
23
- validateKey(key: string): Promise<ApiKeyInfo | null>;
24
- trackUsage(key: string, credits: number): Promise<void>;
25
- addKey(keyInfo: ApiKeyInfo): void;
26
- getUsage(key: string): number;
27
- }
28
- //# sourceMappingURL=auth-store.d.ts.map
package/dist/server/auth-store.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"auth-store.d.ts","sourceRoot":"","sources":["../../src/server/auth-store.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAIH,MAAM,WAAW,UAAU;IACzB,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,EAAE,MAAM,GAAG,SAAS,GAAG,KAAK,GAAG,YAAY,CAAC;IAChD,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,IAAI,CAAC;CACjB;AAED,MAAM,WAAW,SAAS;IACxB,WAAW,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,UAAU,GAAG,IAAI,CAAC,CAAC;IACrD,UAAU,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;CACzD;AAwCD;;GAEG;AACH,qBAAa,iBAAkB,YAAW,SAAS;IACjD,OAAO,CAAC,IAAI,CAAiC;IAC7C,OAAO,CAAC,KAAK,CAA6B;;IAepC,WAAW,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,UAAU,GAAG,IAAI,CAAC;IAiBpD,UAAU,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAK7D,MAAM,CAAC,OAAO,EAAE,UAAU,GAAG,IAAI;IAQjC,QAAQ,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM;CAG9B"}
package/dist/server/auth-store.js DELETED
@@ -1,87 +0,0 @@
1
- /**
2
- * Auth store abstraction for API key validation and usage tracking
3
- * Designed to easily swap from in-memory to PostgreSQL
4
- */
5
- import { timingSafeEqual } from 'crypto';
6
- /**
7
- * Validate API key format and strength
8
- * SECURITY: Enforce minimum complexity
9
- */
10
- function validateKeyFormat(key) {
11
- // Minimum 32 characters
12
- if (key.length < 32) {
13
- return false;
14
- }
15
- // Must contain alphanumeric characters
16
- if (!/^[a-zA-Z0-9_-]+$/.test(key)) {
17
- return false;
18
- }
19
- return true;
20
- }
21
- /**
22
- * Timing-safe key comparison
23
- * SECURITY: Prevent timing attacks on key validation
24
- */
25
- function timingSafeKeyCompare(a, b) {
26
- // Ensure equal length for comparison
27
- if (a.length !== b.length) {
28
- // Compare against dummy to prevent timing leak
29
- const dummy = 'x'.repeat(Math.max(a.length, b.length));
30
- timingSafeEqual(Buffer.from(dummy), Buffer.from(dummy));
31
- return false;
32
- }
33
- try {
34
- return timingSafeEqual(Buffer.from(a), Buffer.from(b));
35
- }
36
- catch {
37
- return false;
38
- }
39
- }
40
- /**
41
- * In-memory auth store for development and self-hosted deployments
42
- */
43
- export class InMemoryAuthStore {
44
- keys = new Map();
45
- usage = new Map();
46
- constructor() {
47
- // SECURITY: Demo key only in development mode
48
- // Removed hardcoded demo key - use addKey() or environment variables
49
- if (process.env.NODE_ENV === 'development' && process.env.DEMO_KEY) {
50
- this.addKey({
51
- key: process.env.DEMO_KEY,
52
- tier: 'pro',
53
- rateLimit: 300,
54
- createdAt: new Date(),
55
- });
56
- }
57
- }
58
- async validateKey(key) {
59
- // Basic validation
60
- if (!key || typeof key !== 'string') {
61
- return null;
62
- }
63
- // SECURITY: Timing-safe comparison to prevent timing attacks
64
- for (const [storedKey, keyInfo] of this.keys.entries()) {
65
- if (timingSafeKeyCompare(key, storedKey)) {
66
- return keyInfo;
67
- }
68
- }
69
- // Constant-time operation for invalid key
70
- return null;
71
- }
72
- async trackUsage(key, credits) {
73
- const current = this.usage.get(key) || 0;
74
- this.usage.set(key, current + credits);
75
- }
76
- addKey(keyInfo) {
77
- // SECURITY: Validate key format before adding
78
- if (!validateKeyFormat(keyInfo.key)) {
79
- throw new Error('Invalid API key format: must be at least 32 characters, alphanumeric with - or _');
80
- }
81
- this.keys.set(keyInfo.key, keyInfo);
82
- }
83
- getUsage(key) {
84
- return this.usage.get(key) || 0;
85
- }
86
- }
87
- //# sourceMappingURL=auth-store.js.map
package/dist/server/auth-store.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"auth-store.js","sourceRoot":"","sources":["../../src/server/auth-store.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,EAAE,eAAe,EAAE,MAAM,QAAQ,CAAC;AAezC;;;GAGG;AACH,SAAS,iBAAiB,CAAC,GAAW;IACpC,wBAAwB;IACxB,IAAI,GAAG,CAAC,MAAM,GAAG,EAAE,EAAE,CAAC;QACpB,OAAO,KAAK,CAAC;IACf,CAAC;IAED,uCAAuC;IACvC,IAAI,CAAC,kBAAkB,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QAClC,OAAO,KAAK,CAAC;IACf,CAAC;IAED,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;GAGG;AACH,SAAS,oBAAoB,CAAC,CAAS,EAAE,CAAS;IAChD,qCAAqC;IACrC,IAAI,CAAC,CAAC,MAAM,KAAK,CAAC,CAAC,MAAM,EAAE,CAAC;QAC1B,+CAA+C;QAC/C,MAAM,KAAK,GAAG,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC;QACvD,eAAe,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QACxD,OAAO,KAAK,CAAC;IACf,CAAC;IAED,IAAI,CAAC;QACH,OAAO,eAAe,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;IACzD,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,KAAK,CAAC;IACf,CAAC;AACH,CAAC;AAED;;GAEG;AACH,MAAM,OAAO,iBAAiB;IACpB,IAAI,GAAG,IAAI,GAAG,EAAsB,CAAC;IACrC,KAAK,GAAG,IAAI,GAAG,EAAkB,CAAC;IAE1C;QACE,8CAA8C;QAC9C,qEAAqE;QACrE,IAAI,OAAO,CAAC,GAAG,CAAC,QAAQ,KAAK,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC;YACnE,IAAI,CAAC,MAAM,CAAC;gBACV,GAAG,EAAE,OAAO,CAAC,GAAG,CAAC,QAAQ;gBACzB,IAAI,EAAE,KAAK;gBACX,SAAS,EAAE,GAAG;gBACd,SAAS,EAAE,IAAI,IAAI,EAAE;aACtB,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,KAAK,CAAC,WAAW,CAAC,GAAW;QAC3B,mBAAmB;QACnB,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;YACpC,OAAO,IAAI,CAAC;QACd,CAAC;QAED,6DAA6D;QAC7D,KAAK,MAAM,CAAC,SAAS,EAAE,OAAO,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,CAAC;YACvD,IAAI,oBAAoB,CAAC,GAAG,EAAE,SAAS,CAAC,EAAE,CAAC;gBACzC,OAAO,OAAO,CAAC;YACjB,CAAC;QACH,CAAC;QAED,0CAA0C;QAC1C,OAAO,IAAI,CAAC;IACd,CAAC;IAED,KAAK,CAAC,UAAU,CAAC,GAAW,EAAE,OAAe;QAC3C,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACzC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,OAAO,GAAG,OAAO,CAAC,CAAC;IACzC,CAAC;IAED,MAAM,CAAC,OAAmB;QACxB,8CAA8C;QAC9C,IAAI,CAAC,iBAAiB,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YACpC,MAAM,IAAI,KAAK,CAAC,kFAAkF,CAAC,CAAC;QACtG,CAAC;QACD,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC;IACtC,CAAC;IAED,QAAQ,CAAC,GAAW;QAClB,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAClC,CAAC;CACF"}
package/dist/server/middleware/auth.d.ts DELETED
@@ -1,18 +0,0 @@
1
- /**
2
- * API key authentication middleware
3
- */
4
- import { Request, Response, NextFunction } from 'express';
5
- import { AuthStore, ApiKeyInfo } from '../auth-store.js';
6
- declare global {
7
- namespace Express {
8
- interface Request {
9
- auth?: {
10
- keyInfo: ApiKeyInfo | null;
11
- tier: 'free' | 'starter' | 'pro' | 'enterprise';
12
- rateLimit: number;
13
- };
14
- }
15
- }
16
- }
17
- export declare function createAuthMiddleware(authStore: AuthStore): (req: Request, res: Response, next: NextFunction) => Promise<void>;
18
- //# sourceMappingURL=auth.d.ts.map
package/dist/server/middleware/auth.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"auth.d.ts","sourceRoot":"","sources":["../../../src/server/middleware/auth.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AAC1D,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAEzD,OAAO,CAAC,MAAM,CAAC;IACb,UAAU,OAAO,CAAC;QAChB,UAAU,OAAO;YACf,IAAI,CAAC,EAAE;gBACL,OAAO,EAAE,UAAU,GAAG,IAAI,CAAC;gBAC3B,IAAI,EAAE,MAAM,GAAG,SAAS,GAAG,KAAK,GAAG,YAAY,CAAC;gBAChD,SAAS,EAAE,MAAM,CAAC;aACnB,CAAC;SACH;KACF;CACF;AAED,wBAAgB,oBAAoB,CAAC,SAAS,EAAE,SAAS,IACzC,KAAK,OAAO,EAAE,KAAK,QAAQ,EAAE,MAAM,YAAY,mBAsD9D"}
package/dist/server/middleware/auth.js DELETED
@@ -1,55 +0,0 @@
1
- /**
2
- * API key authentication middleware
3
- */
4
- export function createAuthMiddleware(authStore) {
5
- return async (req, res, next) => {
6
- try {
7
- // Extract API key from Authorization header or X-API-Key header
8
- const authHeader = req.headers.authorization;
9
- const apiKeyHeader = req.headers['x-api-key'];
10
- let apiKey = null;
11
- if (authHeader?.startsWith('Bearer ')) {
12
- apiKey = authHeader.slice(7);
13
- }
14
- else if (apiKeyHeader && typeof apiKeyHeader === 'string') {
15
- apiKey = apiKeyHeader;
16
- }
17
- // SECURITY: Require API key for all non-health endpoints
18
- const isHealthEndpoint = req.path === '/health';
19
- if (!apiKey && !isHealthEndpoint) {
20
- res.status(401).json({
21
- error: 'missing_key',
22
- message: 'API key is required. Provide via Authorization: Bearer <key> or X-API-Key header.',
23
- });
24
- return;
25
- }
26
- // Validate API key if provided
27
- let keyInfo = null;
28
- if (apiKey) {
29
- keyInfo = await authStore.validateKey(apiKey);
30
- if (!keyInfo) {
31
- res.status(401).json({
32
- error: 'invalid_key',
33
- message: 'Invalid API key',
34
- });
35
- return;
36
- }
37
- }
38
- // Set auth context on request
39
- req.auth = {
40
- keyInfo,
41
- tier: keyInfo?.tier || 'free',
42
- rateLimit: keyInfo?.rateLimit || 10,
43
- };
44
- next();
45
- }
46
- catch (error) {
47
- const err = error;
48
- res.status(500).json({
49
- error: 'auth_error',
50
- message: err.message || 'Authentication failed',
51
- });
52
- }
53
- };
54
- }
55
- //# sourceMappingURL=auth.js.map
package/dist/server/middleware/auth.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"auth.js","sourceRoot":"","sources":["../../../src/server/middleware/auth.ts"],"names":[],"mappings":"AAAA;;GAEG;AAiBH,MAAM,UAAU,oBAAoB,CAAC,SAAoB;IACvD,OAAO,KAAK,EAAE,GAAY,EAAE,GAAa,EAAE,IAAkB,EAAE,EAAE;QAC/D,IAAI,CAAC;YACH,gEAAgE;YAChE,MAAM,UAAU,GAAG,GAAG,CAAC,OAAO,CAAC,aAAa,CAAC;YAC7C,MAAM,YAAY,GAAG,GAAG,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC;YAE9C,IAAI,MAAM,GAAkB,IAAI,CAAC;YAEjC,IAAI,UAAU,EAAE,UAAU,CAAC,SAAS,CAAC,EAAE,CAAC;gBACtC,MAAM,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;YAC/B,CAAC;iBAAM,IAAI,YAAY,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;gBAC5D,MAAM,GAAG,YAAY,CAAC;YACxB,CAAC;YAED,yDAAyD;YACzD,MAAM,gBAAgB,GAAG,GAAG,CAAC,IAAI,KAAK,SAAS,CAAC;YAEhD,IAAI,CAAC,MAAM,IAAI,CAAC,gBAAgB,EAAE,CAAC;gBACjC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,aAAa;oBACpB,OAAO,EAAE,mFAAmF;iBAC7F,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,+BAA+B;YAC/B,IAAI,OAAO,GAAsB,IAAI,CAAC;YACtC,IAAI,MAAM,EAAE,CAAC;gBACX,OAAO,GAAG,MAAM,SAAS,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC;gBAC9C,IAAI,CAAC,OAAO,EAAE,CAAC;oBACb,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;wBACnB,KAAK,EAAE,aAAa;wBACpB,OAAO,EAAE,iBAAiB;qBAC3B,CAAC,CAAC;oBACH,OAAO;gBACT,CAAC;YACH,CAAC;YAED,8BAA8B;YAC9B,GAAG,CAAC,IAAI,GAAG;gBACT,OAAO;gBACP,IAAI,EAAE,OAAO,EAAE,IAAI,IAAI,MAAM;gBAC7B,SAAS,EAAE,OAAO,EAAE,SAAS,IAAI,EAAE;aACpC,CAAC;YAEF,IAAI,EAAE,CAAC;QACT,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,GAAG,GAAG,KAAc,CAAC;YAC3B,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;gBACnB,KAAK,EAAE,YAAY;gBACnB,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,uBAAuB;aAChD,CAAC,CAAC;QACL,CAAC;IACH,CAAC,CAAC;AACJ,CAAC"}
package/dist/server/middleware/rate-limit.d.ts DELETED
@@ -1,23 +0,0 @@
1
- /**
2
- * Sliding window rate limiting middleware
3
- */
4
- import { Request, Response, NextFunction } from 'express';
5
- export declare class RateLimiter {
6
- private store;
7
- private windowMs;
8
- constructor(windowMs?: number);
9
- /**
10
- * Check if request is allowed under rate limit
11
- */
12
- checkLimit(identifier: string, limit: number): {
13
- allowed: boolean;
14
- remaining: number;
15
- retryAfter?: number;
16
- };
17
- /**
18
- * Clean up old entries (call periodically)
19
- */
20
- cleanup(): void;
21
- }
22
- export declare function createRateLimitMiddleware(limiter: RateLimiter): (req: Request, res: Response, next: NextFunction) => void;
23
- //# sourceMappingURL=rate-limit.d.ts.map
package/dist/server/middleware/rate-limit.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"rate-limit.d.ts","sourceRoot":"","sources":["../../../src/server/middleware/rate-limit.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AAM1D,qBAAa,WAAW;IACtB,OAAO,CAAC,KAAK,CAAqC;IAClD,OAAO,CAAC,QAAQ,CAAS;gBAEb,QAAQ,GAAE,MAAc;IAIpC;;OAEG;IACH,UAAU,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG;QAC7C,OAAO,EAAE,OAAO,CAAC;QACjB,SAAS,EAAE,MAAM,CAAC;QAClB,UAAU,CAAC,EAAE,MAAM,CAAC;KACrB;IAmCD;;OAEG;IACH,OAAO,IAAI,IAAI;CAWhB;AAED,wBAAgB,yBAAyB,CAAC,OAAO,EAAE,WAAW,IACpD,KAAK,OAAO,EAAE,KAAK,QAAQ,EAAE,MAAM,YAAY,UA+BxD"}
package/dist/server/middleware/rate-limit.js DELETED
@@ -1,85 +0,0 @@
1
- /**
2
- * Sliding window rate limiting middleware
3
- */
4
- export class RateLimiter {
5
- store = new Map();
6
- windowMs;
7
- constructor(windowMs = 60000) {
8
- this.windowMs = windowMs;
9
- }
10
- /**
11
- * Check if request is allowed under rate limit
12
- */
13
- checkLimit(identifier, limit) {
14
- const now = Date.now();
15
- const windowStart = now - this.windowMs;
16
- // Get or create entry
17
- let entry = this.store.get(identifier);
18
- if (!entry) {
19
- entry = { timestamps: [] };
20
- this.store.set(identifier, entry);
21
- }
22
- // Remove timestamps outside the window
23
- entry.timestamps = entry.timestamps.filter(ts => ts > windowStart);
24
- // Check if limit exceeded
25
- if (entry.timestamps.length >= limit) {
26
- const oldestTimestamp = entry.timestamps[0];
27
- const retryAfter = Math.ceil((oldestTimestamp + this.windowMs - now) / 1000);
28
- return {
29
- allowed: false,
30
- remaining: 0,
31
- retryAfter,
32
- };
33
- }
34
- // Add current timestamp
35
- entry.timestamps.push(now);
36
- return {
37
- allowed: true,
38
- remaining: limit - entry.timestamps.length,
39
- };
40
- }
41
- /**
42
- * Clean up old entries (call periodically)
43
- */
44
- cleanup() {
45
- const now = Date.now();
46
- const windowStart = now - this.windowMs;
47
- for (const [identifier, entry] of this.store.entries()) {
48
- entry.timestamps = entry.timestamps.filter(ts => ts > windowStart);
49
- if (entry.timestamps.length === 0) {
50
- this.store.delete(identifier);
51
- }
52
- }
53
- }
54
- }
55
- export function createRateLimitMiddleware(limiter) {
56
- return (req, res, next) => {
57
- try {
58
- // Use API key or IP address as identifier
59
- const identifier = req.auth?.keyInfo?.key || req.ip || 'unknown';
60
- const limit = req.auth?.rateLimit || 10;
61
- const result = limiter.checkLimit(identifier, limit);
62
- // Set rate limit headers
63
- res.setHeader('X-RateLimit-Limit', limit.toString());
64
- res.setHeader('X-RateLimit-Remaining', result.remaining.toString());
65
- if (!result.allowed) {
66
- res.setHeader('Retry-After', result.retryAfter.toString());
67
- res.status(429).json({
68
- error: 'rate_limited',
69
- message: 'Rate limit exceeded',
70
- retryAfter: result.retryAfter,
71
- });
72
- return;
73
- }
74
- next();
75
- }
76
- catch (error) {
77
- const err = error;
78
- res.status(500).json({
79
- error: 'rate_limit_error',
80
- message: err.message || 'Rate limiting failed',
81
- });
82
- }
83
- };
84
- }
85
- //# sourceMappingURL=rate-limit.js.map
package/dist/server/middleware/rate-limit.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"rate-limit.js","sourceRoot":"","sources":["../../../src/server/middleware/rate-limit.ts"],"names":[],"mappings":"AAAA;;GAEG;AAQH,MAAM,OAAO,WAAW;IACd,KAAK,GAAG,IAAI,GAAG,EAA0B,CAAC;IAC1C,QAAQ,CAAS;IAEzB,YAAY,WAAmB,KAAK;QAClC,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;IAC3B,CAAC;IAED;;OAEG;IACH,UAAU,CAAC,UAAkB,EAAE,KAAa;QAK1C,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,WAAW,GAAG,GAAG,GAAG,IAAI,CAAC,QAAQ,CAAC;QAExC,sBAAsB;QACtB,IAAI,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACvC,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,KAAK,GAAG,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC;YAC3B,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACpC,CAAC;QAED,uCAAuC;QACvC,KAAK,CAAC,UAAU,GAAG,KAAK,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,GAAG,WAAW,CAAC,CAAC;QAEnE,0BAA0B;QAC1B,IAAI,KAAK,CAAC,UAAU,CAAC,MAAM,IAAI,KAAK,EAAE,CAAC;YACrC,MAAM,eAAe,GAAG,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;YAC5C,MAAM,UAAU,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,eAAe,GAAG,IAAI,CAAC,QAAQ,GAAG,GAAG,CAAC,GAAG,IAAI,CAAC,CAAC;YAE7E,OAAO;gBACL,OAAO,EAAE,KAAK;gBACd,SAAS,EAAE,CAAC;gBACZ,UAAU;aACX,CAAC;QACJ,CAAC;QAED,wBAAwB;QACxB,KAAK,CAAC,UAAU,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAE3B,OAAO;YACL,OAAO,EAAE,IAAI;YACb,SAAS,EAAE,KAAK,GAAG,KAAK,CAAC,UAAU,CAAC,MAAM;SAC3C,CAAC;IACJ,CAAC;IAED;;OAEG;IACH,OAAO;QACL,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,WAAW,GAAG,GAAG,GAAG,IAAI,CAAC,QAAQ,CAAC;QAExC,KAAK,MAAM,CAAC,UAAU,EAAE,KAAK,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;YACvD,KAAK,CAAC,UAAU,GAAG,KAAK,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,GAAG,WAAW,CAAC,CAAC;YACnE,IAAI,KAAK,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAClC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC;YAChC,CAAC;QACH,CAAC;IACH,CAAC;CACF;AAED,MAAM,UAAU,yBAAyB,CAAC,OAAoB;IAC5D,OAAO,CAAC,GAAY,EAAE,GAAa,EAAE,IAAkB,EAAE,EAAE;QACzD,IAAI,CAAC;YACH,0CAA0C;YAC1C,MAAM,UAAU,GAAG,GAAG,CAAC,IAAI,EAAE,OAAO,EAAE,GAAG,IAAI,GAAG,CAAC,EAAE,IAAI,SAAS,CAAC;YACjE,MAAM,KAAK,GAAG,GAAG,CAAC,IAAI,EAAE,SAAS,IAAI,EAAE,CAAC;YAExC,MAAM,MAAM,GAAG,OAAO,CAAC,UAAU,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;YAErD,yBAAyB;YACzB,GAAG,CAAC,SAAS,CAAC,mBAAmB,EAAE,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC;YACrD,GAAG,CAAC,SAAS,CAAC,uBAAuB,EAAE,MAAM,CAAC,SAAS,CAAC,QAAQ,EAAE,CAAC,CAAC;YAEpE,IAAI,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC;gBACpB,GAAG,CAAC,SAAS,CAAC,aAAa,EAAE,MAAM,CAAC,UAAW,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC5D,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,cAAc;oBACrB,OAAO,EAAE,qBAAqB;oBAC9B,UAAU,EAAE,MAAM,CAAC,UAAU;iBAC9B,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,IAAI,EAAE,CAAC;QACT,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,GAAG,GAAG,KAAc,CAAC;YAC3B,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;gBACnB,KAAK,EAAE,kBAAkB;gBACzB,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,sBAAsB;aAC/C,CAAC,CAAC;QACL,CAAC;IACH,CAAC,CAAC;AACJ,CAAC"}
package/dist/server/routes/fetch.d.ts DELETED
@@ -1,7 +0,0 @@
1
- /**
2
- * Fetch endpoint with caching
3
- */
4
- import { Router } from 'express';
5
- import { AuthStore } from '../auth-store.js';
6
- export declare function createFetchRouter(authStore: AuthStore): Router;
7
- //# sourceMappingURL=fetch.d.ts.map
package/dist/server/routes/fetch.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"fetch.d.ts","sourceRoot":"","sources":["../../../src/server/routes/fetch.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,MAAM,EAAqB,MAAM,SAAS,CAAC;AAIpD,OAAO,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC;AAO7C,wBAAgB,iBAAiB,CAAC,SAAS,EAAE,SAAS,GAAG,MAAM,CAsI9D"}
package/dist/server/routes/fetch.js DELETED
@@ -1,127 +0,0 @@
1
- /**
2
- * Fetch endpoint with caching
3
- */
4
- import { Router } from 'express';
5
- import { peel } from '../../index.js';
6
- import { LRUCache } from 'lru-cache';
7
- export function createFetchRouter(authStore) {
8
- const router = Router();
9
- // LRU cache: 5 minute TTL, max 1000 entries, 100MB total size
10
- const cache = new LRUCache({
11
- max: 1000,
12
- ttl: 5 * 60 * 1000, // 5 minutes
13
- maxSize: 100 * 1024 * 1024, // 100MB
14
- sizeCalculation: (entry) => {
15
- return JSON.stringify(entry).length;
16
- },
17
- });
18
- router.get('/v1/fetch', async (req, res) => {
19
- try {
20
- const { url, render, wait, format } = req.query;
21
- // Validate URL parameter
22
- if (!url || typeof url !== 'string') {
23
- res.status(400).json({
24
- error: 'invalid_request',
25
- message: 'Missing or invalid "url" parameter',
26
- });
27
- return;
28
- }
29
- // SECURITY: Validate URL format and length
30
- if (url.length > 2048) {
31
- res.status(400).json({
32
- error: 'invalid_url',
33
- message: 'URL too long (max 2048 characters)',
34
- });
35
- return;
36
- }
37
- try {
38
- const parsed = new URL(url);
39
- // Normalize URL for consistent caching
40
- const normalizedUrl = parsed.href;
41
- // Use normalized URL for cache key
42
- if (normalizedUrl !== url) {
43
- // URL was normalized, update for caching
44
- }
45
- }
46
- catch {
47
- res.status(400).json({
48
- error: 'invalid_url',
49
- message: 'Invalid URL format',
50
- });
51
- return;
52
- }
53
- // Build cache key
54
- const cacheKey = `fetch:${url}:${render}:${wait}:${format}`;
55
- // Check cache
56
- const cached = cache.get(cacheKey);
57
- if (cached) {
58
- res.setHeader('X-Cache', 'HIT');
59
- res.setHeader('X-Cache-Age', Math.floor((Date.now() - cached.timestamp) / 1000).toString());
60
- res.json(cached.result);
61
- return;
62
- }
63
- // Parse options
64
- const options = {
65
- render: render === 'true',
66
- wait: wait ? parseInt(wait, 10) : undefined,
67
- format: format || 'markdown',
68
- };
69
- // Validate wait parameter
70
- if (options.wait !== undefined && (isNaN(options.wait) || options.wait < 0 || options.wait > 60000)) {
71
- res.status(400).json({
72
- error: 'invalid_request',
73
- message: 'Invalid "wait" parameter: must be between 0 and 60000ms',
74
- });
75
- return;
76
- }
77
- // Validate format parameter
78
- if (!['markdown', 'text', 'html'].includes(options.format || '')) {
79
- res.status(400).json({
80
- error: 'invalid_request',
81
- message: 'Invalid "format" parameter: must be "markdown", "text", or "html"',
82
- });
83
- return;
84
- }
85
- // Fetch content
86
- const startTime = Date.now();
87
- const result = await peel(url, options);
88
- const elapsed = Date.now() - startTime;
89
- // Track usage (1 credit per fetch)
90
- if (req.auth?.keyInfo?.key) {
91
- await authStore.trackUsage(req.auth.keyInfo.key, 1);
92
- }
93
- // Cache result
94
- cache.set(cacheKey, {
95
- result,
96
- timestamp: Date.now(),
97
- });
98
- // Add usage headers
99
- res.setHeader('X-Cache', 'MISS');
100
- res.setHeader('X-Credits-Used', '1');
101
- res.setHeader('X-Processing-Time', elapsed.toString());
102
- res.json(result);
103
- }
104
- catch (error) {
105
- const err = error;
106
- // SECURITY: Sanitize error messages to prevent information disclosure
107
- if (err.code) {
108
- // WebPeelError from core library - safe to expose
109
- const safeMessage = err.message.replace(/[<>"']/g, ''); // Remove HTML chars
110
- res.status(500).json({
111
- error: err.code,
112
- message: safeMessage,
113
- });
114
- }
115
- else {
116
- // Unexpected error - generic message only
117
- console.error('Fetch error:', err); // Log full error server-side
118
- res.status(500).json({
119
- error: 'internal_error',
120
- message: 'An unexpected error occurred while fetching the URL',
121
- });
122
- }
123
- }
124
- });
125
- return router;
126
- }
127
- //# sourceMappingURL=fetch.js.map
package/dist/server/routes/fetch.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"fetch.js","sourceRoot":"","sources":["../../../src/server/routes/fetch.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,MAAM,EAAqB,MAAM,SAAS,CAAC;AACpD,OAAO,EAAE,IAAI,EAAE,MAAM,gBAAgB,CAAC;AAEtC,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAQrC,MAAM,UAAU,iBAAiB,CAAC,SAAoB;IACpD,MAAM,MAAM,GAAG,MAAM,EAAE,CAAC;IAExB,8DAA8D;IAC9D,MAAM,KAAK,GAAG,IAAI,QAAQ,CAAqB;QAC7C,GAAG,EAAE,IAAI;QACT,GAAG,EAAE,CAAC,GAAG,EAAE,GAAG,IAAI,EAAE,YAAY;QAChC,OAAO,EAAE,GAAG,GAAG,IAAI,GAAG,IAAI,EAAE,QAAQ;QACpC,eAAe,EAAE,CAAC,KAAK,EAAE,EAAE;YACzB,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC;QACtC,CAAC;KACF,CAAC,CAAC;IAEH,MAAM,CAAC,GAAG,CAAC,WAAW,EAAE,KAAK,EAAE,GAAY,EAAE,GAAa,EAAE,EAAE;QAC5D,IAAI,CAAC;YACH,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,GAAG,CAAC,KAAK,CAAC;YAEhD,yBAAyB;YACzB,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;gBACpC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,iBAAiB;oBACxB,OAAO,EAAE,oCAAoC;iBAC9C,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,2CAA2C;YAC3C,IAAI,GAAG,CAAC,MAAM,GAAG,IAAI,EAAE,CAAC;gBACtB,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,aAAa;oBACpB,OAAO,EAAE,oCAAoC;iBAC9C,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;gBAC5B,uCAAuC;gBACvC,MAAM,aAAa,GAAG,MAAM,CAAC,IAAI,CAAC;gBAElC,mCAAmC;gBACnC,IAAI,aAAa,KAAK,GAAG,EAAE,CAAC;oBAC1B,yCAAyC;gBAC3C,CAAC;YACH,CAAC;YAAC,MAAM,CAAC;gBACP,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,aAAa;oBACpB,OAAO,EAAE,oBAAoB;iBAC9B,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,kBAAkB;YAClB,MAAM,QAAQ,GAAG,SAAS,GAAG,IAAI,MAAM,IAAI,IAAI,IAAI,MAAM,EAAE,CAAC;YAE5D,cAAc;YACd,MAAM,MAAM,GAAG,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;YACnC,IAAI,MAAM,EAAE,CAAC;gBACX,GAAG,CAAC,SAAS,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;gBAChC,GAAG,CAAC,SAAS,CAAC,aAAa,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,MAAM,CAAC,SAAS,CAAC,GAAG,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC5F,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;gBACxB,OAAO;YACT,CAAC;YAED,gBAAgB;YAChB,MAAM,OAAO,GAAgB;gBAC3B,MAAM,EAAE,MAAM,KAAK,MAAM;gBACzB,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,IAAc,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS;gBACrD,MAAM,EAAG,MAAuC,IAAI,UAAU;aAC/D,CAAC;YAEF,0BAA0B;YAC1B,IAAI,OAAO,CAAC,IAAI,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,OAAO,CAAC,IAAI,GAAG,CAAC,IAAI,OAAO,CAAC,IAAI,GAAG,KAAK,CAAC,EAAE,CAAC;gBACpG,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,iBAAiB;oBACxB,OAAO,EAAE,yDAAyD;iBACnE,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,4BAA4B;YAC5B,IAAI,CAAC,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,MAAM,IAAI,EAAE,CAAC,EAAE,CAAC;gBACjE,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,iBAAiB;oBACxB,OAAO,EAAE,mEAAmE;iBAC7E,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,gBAAgB;YAChB,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAC7B,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC;YACxC,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;YAEvC,mCAAmC;YACnC,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,EAAE,GAAG,EAAE,CAAC;gBAC3B,MAAM,SAAS,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC;YACtD,CAAC;YAED,eAAe;YACf,KAAK,CAAC,GAAG,CAAC,QAAQ,EAAE;gBAClB,MAAM;gBACN,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE;aACtB,CAAC,CAAC;YAEH,oBAAoB;YACpB,GAAG,CAAC,SAAS,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC;YACjC,GAAG,CAAC,SAAS,CAAC,gBAAgB,EAAE,GAAG,CAAC,CAAC;YACrC,GAAG,CAAC,SAAS,CAAC,mBAAmB,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC;YAEvD,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACnB,CAAC;QAAC,OAAO,KAAU,EAAE,CAAC;YACpB,MAAM,GAAG,GAAG,KAAY,CAAC;YAEzB,sEAAsE;YACtE,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;gBACb,kDAAkD;gBAClD,MAAM,WAAW,GAAG,GAAG,CAAC,OAAO,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC,oBAAoB;gBAC5E,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,GAAG,CAAC,IAAI;oBACf,OAAO,EAAE,WAAW;iBACrB,CAAC,CAAC;YACL,CAAC;iBAAM,CAAC;gBACN,0CAA0C;gBAC1C,OAAO,CAAC,KAAK,CAAC,cAAc,EAAE,GAAG,CAAC,CAAC,CAAC,6BAA6B;gBACjE,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,gBAAgB;oBACvB,OAAO,EAAE,qDAAqD;iBAC/D,CAAC,CAAC;YACL,CAAC;QACH,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,OAAO,MAAM,CAAC;AAChB,CAAC"}
package/dist/server/routes/health.d.ts DELETED
@@ -1,6 +0,0 @@
1
- /**
2
- * Health check endpoint
3
- */
4
- import { Router } from 'express';
5
- export declare function createHealthRouter(): Router;
6
- //# sourceMappingURL=health.d.ts.map
package/dist/server/routes/health.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"health.d.ts","sourceRoot":"","sources":["../../../src/server/routes/health.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,MAAM,EAAqB,MAAM,SAAS,CAAC;AAIpD,wBAAgB,kBAAkB,IAAI,MAAM,CAe3C"}
package/dist/server/routes/health.js DELETED
@@ -1,19 +0,0 @@
1
- /**
2
- * Health check endpoint
3
- */
4
- import { Router } from 'express';
5
- const startTime = Date.now();
6
- export function createHealthRouter() {
7
- const router = Router();
8
- router.get('/health', (_req, res) => {
9
- const uptime = Math.floor((Date.now() - startTime) / 1000);
10
- res.json({
11
- status: 'healthy',
12
- version: process.env.npm_package_version || '1.0.0',
13
- uptime,
14
- timestamp: new Date().toISOString(),
15
- });
16
- });
17
- return router;
18
- }
19
- //# sourceMappingURL=health.js.map
package/dist/server/routes/health.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"health.js","sourceRoot":"","sources":["../../../src/server/routes/health.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,MAAM,EAAqB,MAAM,SAAS,CAAC;AAEpD,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;AAE7B,MAAM,UAAU,kBAAkB;IAChC,MAAM,MAAM,GAAG,MAAM,EAAE,CAAC;IAExB,MAAM,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,IAAa,EAAE,GAAa,EAAE,EAAE;QACrD,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC,GAAG,IAAI,CAAC,CAAC;QAE3D,GAAG,CAAC,IAAI,CAAC;YACP,MAAM,EAAE,SAAS;YACjB,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,OAAO;YACnD,MAAM;YACN,SAAS,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE;SACpC,CAAC,CAAC;IACL,CAAC,CAAC,CAAC;IAEH,OAAO,MAAM,CAAC;AAChB,CAAC"}
package/dist/server/routes/search.d.ts DELETED
@@ -1,7 +0,0 @@
1
- /**
2
- * Search endpoint with caching
3
- */
4
- import { Router } from 'express';
5
- import { AuthStore } from '../auth-store.js';
6
- export declare function createSearchRouter(authStore: AuthStore): Router;
7
- //# sourceMappingURL=search.d.ts.map
package/dist/server/routes/search.d.ts.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"search.d.ts","sourceRoot":"","sources":["../../../src/server/routes/search.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,MAAM,EAAqB,MAAM,SAAS,CAAC;AAIpD,OAAO,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC;AAa7C,wBAAgB,kBAAkB,CAAC,SAAS,EAAE,SAAS,GAAG,MAAM,CAsI/D"}
package/dist/server/routes/search.js DELETED
@@ -1,124 +0,0 @@
1
- /**
2
- * Search endpoint with caching
3
- */
4
- import { Router } from 'express';
5
- import { fetch as undiciFetch } from 'undici';
6
- import { load } from 'cheerio';
7
- import { LRUCache } from 'lru-cache';
8
- export function createSearchRouter(authStore) {
9
- const router = Router();
10
- // LRU cache: 15 minute TTL, max 500 entries, 50MB total size
11
- const cache = new LRUCache({
12
- max: 500,
13
- ttl: 15 * 60 * 1000, // 15 minutes
14
- maxSize: 50 * 1024 * 1024, // 50MB
15
- sizeCalculation: (entry) => {
16
- return JSON.stringify(entry).length;
17
- },
18
- });
19
- router.get('/v1/search', async (req, res) => {
20
- try {
21
- const { q, count } = req.query;
22
- // Validate query parameter
23
- if (!q || typeof q !== 'string') {
24
- res.status(400).json({
25
- error: 'invalid_request',
26
- message: 'Missing or invalid "q" parameter',
27
- });
28
- return;
29
- }
30
- // Parse and validate count
31
- const resultCount = count ? parseInt(count, 10) : 5;
32
- if (isNaN(resultCount) || resultCount < 1 || resultCount > 10) {
33
- res.status(400).json({
34
- error: 'invalid_request',
35
- message: 'Invalid "count" parameter: must be between 1 and 10',
36
- });
37
- return;
38
- }
39
- // Build cache key
40
- const cacheKey = `search:${q}:${resultCount}`;
41
- // Check cache
42
- const cached = cache.get(cacheKey);
43
- if (cached) {
44
- res.setHeader('X-Cache', 'HIT');
45
- res.setHeader('X-Cache-Age', Math.floor((Date.now() - cached.timestamp) / 1000).toString());
46
- res.json({
47
- query: q,
48
- count: cached.results.length,
49
- results: cached.results,
50
- });
51
- return;
52
- }
53
- // Perform search
54
- const searchUrl = `https://html.duckduckgo.com/html/?q=${encodeURIComponent(q)}`;
55
- const startTime = Date.now();
56
- const response = await undiciFetch(searchUrl, {
57
- headers: {
58
- 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
59
- },
60
- });
61
- if (!response.ok) {
62
- throw new Error(`Search failed: HTTP ${response.status}`);
63
- }
64
- const html = await response.text();
65
- const $ = load(html);
66
- const results = [];
67
- $('.result').each((_i, elem) => {
68
- if (results.length >= resultCount)
69
- return;
70
- const $result = $(elem);
71
- let title = $result.find('.result__title').text().trim();
72
- let url = $result.find('.result__url').attr('href') || '';
73
- let snippet = $result.find('.result__snippet').text().trim();
74
- // SECURITY: Validate and sanitize results
75
- if (!title || !url)
76
- return;
77
- // Only allow HTTP/HTTPS URLs
78
- try {
79
- const parsed = new URL(url);
80
- if (!['http:', 'https:'].includes(parsed.protocol)) {
81
- return;
82
- }
83
- }
84
- catch {
85
- return;
86
- }
87
- // Limit text lengths to prevent bloat
88
- title = title.slice(0, 200);
89
- snippet = snippet.slice(0, 500);
90
- results.push({ title, url, snippet });
91
- });
92
- const elapsed = Date.now() - startTime;
93
- // Track usage (1 credit per search)
94
- if (req.auth?.keyInfo?.key) {
95
- await authStore.trackUsage(req.auth.keyInfo.key, 1);
96
- }
97
- // Cache results
98
- cache.set(cacheKey, {
99
- results,
100
- timestamp: Date.now(),
101
- });
102
- // Add headers
103
- res.setHeader('X-Cache', 'MISS');
104
- res.setHeader('X-Credits-Used', '1');
105
- res.setHeader('X-Processing-Time', elapsed.toString());
106
- res.json({
107
- query: q,
108
- count: results.length,
109
- results,
110
- });
111
- }
112
- catch (error) {
113
- const err = error;
114
- // SECURITY: Generic error message to prevent information disclosure
115
- console.error('Search error:', err); // Log full error server-side
116
- res.status(500).json({
117
- error: 'search_failed',
118
- message: 'Search request failed. Please try again.',
119
- });
120
- }
121
- });
122
- return router;
123
- }
124
- //# sourceMappingURL=search.js.map
package/dist/server/routes/search.js.map DELETED
@@ -1 +0,0 @@
1
- {"version":3,"file":"search.js","sourceRoot":"","sources":["../../../src/server/routes/search.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,MAAM,EAAqB,MAAM,SAAS,CAAC;AACpD,OAAO,EAAE,KAAK,IAAI,WAAW,EAAE,MAAM,QAAQ,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,SAAS,CAAC;AAC/B,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAcrC,MAAM,UAAU,kBAAkB,CAAC,SAAoB;IACrD,MAAM,MAAM,GAAG,MAAM,EAAE,CAAC;IAExB,6DAA6D;IAC7D,MAAM,KAAK,GAAG,IAAI,QAAQ,CAAqB;QAC7C,GAAG,EAAE,GAAG;QACR,GAAG,EAAE,EAAE,GAAG,EAAE,GAAG,IAAI,EAAE,aAAa;QAClC,OAAO,EAAE,EAAE,GAAG,IAAI,GAAG,IAAI,EAAE,OAAO;QAClC,eAAe,EAAE,CAAC,KAAK,EAAE,EAAE;YACzB,OAAO,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC;QACtC,CAAC;KACF,CAAC,CAAC;IAEH,MAAM,CAAC,GAAG,CAAC,YAAY,EAAE,KAAK,EAAE,GAAY,EAAE,GAAa,EAAE,EAAE;QAC7D,IAAI,CAAC;YACH,MAAM,EAAE,CAAC,EAAE,KAAK,EAAE,GAAG,GAAG,CAAC,KAAK,CAAC;YAE/B,2BAA2B;YAC3B,IAAI,CAAC,CAAC,IAAI,OAAO,CAAC,KAAK,QAAQ,EAAE,CAAC;gBAChC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,iBAAiB;oBACxB,OAAO,EAAE,kCAAkC;iBAC5C,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,2BAA2B;YAC3B,MAAM,WAAW,GAAG,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,KAAe,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YAC9D,IAAI,KAAK,CAAC,WAAW,CAAC,IAAI,WAAW,GAAG,CAAC,IAAI,WAAW,GAAG,EAAE,EAAE,CAAC;gBAC9D,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;oBACnB,KAAK,EAAE,iBAAiB;oBACxB,OAAO,EAAE,qDAAqD;iBAC/D,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,kBAAkB;YAClB,MAAM,QAAQ,GAAG,UAAU,CAAC,IAAI,WAAW,EAAE,CAAC;YAE9C,cAAc;YACd,MAAM,MAAM,GAAG,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;YACnC,IAAI,MAAM,EAAE,CAAC;gBACX,GAAG,CAAC,SAAS,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;gBAChC,GAAG,CAAC,SAAS,CAAC,aAAa,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,MAAM,CAAC,SAAS,CAAC,GAAG,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC5F,GAAG,CAAC,IAAI,CAAC;oBACP,KAAK,EAAE,CAAC;oBACR,KAAK,EAAE,MAAM,CAAC,OAAO,CAAC,MAAM;oBAC5B,OAAO,EAAE,MAAM,CAAC,OAAO;iBACxB,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YAED,iBAAiB;YACjB,MAAM,SAAS,GAAG,uCAAuC,kBAAkB,CAAC,CAAC,CAAC,EAAE,CAAC;YACjF,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAE7B,MAAM,QAAQ,GAAG,MAAM,WAAW,CAAC,SAAS,EAAE;gBAC5C,OAAO,EAAE;oBACP,YAAY,EAAE,oEAAoE;iBACnF;aACF,CAAC,CAAC;YAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,IAAI,KAAK,CAAC,uBAAuB,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;YAC5D,CAAC;YAED,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YACnC,MAAM,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC;YAErB,MAAM,OAAO,GAAmB,EAAE,CAAC;YAEnC,CAAC,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,IAAI,EAAE,EAAE;gBAC7B,IAAI,OAAO,CAAC,MAAM,IAAI,WAAW;oBAAE,OAAO;gBAE1C,MAAM,OAAO,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC;gBACxB,IAAI,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC;gBACzD,IAAI,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;gBAC1D,IAAI,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,kBAAkB,CAAC,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC;gBAE7D,0CAA0C;gBAC1C,IAAI,CAAC,KAAK,IAAI,CAAC,GAAG;oBAAE,OAAO;gBAE3B,6BAA6B;gBAC7B,IAAI,CAAC;oBACH,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;oBAC5B,IAAI,CAAC,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;wBACnD,OAAO;oBACT,CAAC;gBACH,CAAC;gBAAC,MAAM,CAAC;oBACP,OAAO;gBACT,CAAC;gBAED,sCAAsC;gBACtC,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;gBAC5B,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;gBAEhC,OAAO,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC,CAAC;YACxC,CAAC,CAAC,CAAC;YAEH,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;YAEvC,oCAAoC;YACpC,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,EAAE,GAAG,EAAE,CAAC;gBAC3B,MAAM,SAAS,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC;YACtD,CAAC;YAED,gBAAgB;YAChB,KAAK,CAAC,GAAG,CAAC,QAAQ,EAAE;gBAClB,OAAO;gBACP,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE;aACtB,CAAC,CAAC;YAEH,cAAc;YACd,GAAG,CAAC,SAAS,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC;YACjC,GAAG,CAAC,SAAS,CAAC,gBAAgB,EAAE,GAAG,CAAC,CAAC;YACrC,GAAG,CAAC,SAAS,CAAC,mBAAmB,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC;YAEvD,GAAG,CAAC,IAAI,CAAC;gBACP,KAAK,EAAE,CAAC;gBACR,KAAK,EAAE,OAAO,CAAC,MAAM;gBACrB,OAAO;aACR,CAAC,CAAC;QACL,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,GAAG,GAAG,KAAc,CAAC;YAC3B,oEAAoE;YACpE,OAAO,CAAC,KAAK,CAAC,eAAe,EAAE,GAAG,CAAC,CAAC,CAAC,6BAA6B;YAClE,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;gBACnB,KAAK,EAAE,eAAe;gBACtB,OAAO,EAAE,0CAA0C;aACpD,CAAC,CAAC;QACL,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,OAAO,MAAM,CAAC;AAChB,CAAC"}