scrapebadger 0.4.0 → 0.7.0

package/README.md

@@ -12,17 +12,20 @@
   <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
 </p>
 
-The official Node.js/TypeScript client library for the [ScrapeBadger](https://scrapebadger.com) API.
+The official Node.js/TypeScript client library for the [ScrapeBadger](https://scrapebadger.com) API — Twitter, Google, Vinted, and general web scraping.
 
 ## Features
 
 - **Full TypeScript Support** - Complete type definitions for all API endpoints
 - **Modern ESM & CommonJS** - Works with both module systems
 - **Async Iterators** - Automatic pagination with `for await...of` syntax
-- **Smart Rate Limit Handling** - Reads API rate limit headers and automatically throttles pagination to avoid hitting limits
-- **Resilient Retries** - 10 automatic retries with exponential backoff on server errors, with colored console warnings on each retry
-- **Error Handling** - Typed exceptions for different error scenarios
-- **Tree Shakeable** - Import only what you need
+- **Smart Rate Limiting** - Reads API headers and throttles pagination automatically
+- **Resilient Retries** - Exponential backoff with colored console warnings
+- **Typed Exceptions** - Distinct error classes for every failure scenario
+- **37+ Twitter endpoints** - Tweets, users, lists, communities, trends, geo, real-time streams
+- **19 Google product APIs** - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, **Local Pack**, **Shorts**, **Flights**, Products
+- **Vinted scraping** - Search items, item details, user profiles, brands, colors, markets
+- **Web scraping** - Anti-bot bypass, JS rendering, and AI data extraction
 
 ## Installation
 
@@ -43,271 +46,39 @@ pnpm add scrapebadger
 ```typescript
 import { ScrapeBadger } from "scrapebadger";
 
-// Create client with API key
 const client = new ScrapeBadger({ apiKey: "your-api-key" });
 
-// Or use environment variable (SCRAPEBADGER_API_KEY)
-const client = new ScrapeBadger();
-
 // Get a tweet
 const tweet = await client.twitter.tweets.getById("1234567890");
 console.log(`@${tweet.username}: ${tweet.text}`);
 
+// Scrape a website
+const result = await client.web.scrape("https://scrapebadger.com", { format: "markdown" });
+console.log(result.content);
+
 // Get a user profile
 const user = await client.twitter.users.getByUsername("elonmusk");
 console.log(`${user.name} has ${user.followers_count.toLocaleString()} followers`);
 ```
 
-## Usage Examples
-
-### Search Tweets
-
-```typescript
-import { ScrapeBadger } from "scrapebadger";
-
-const client = new ScrapeBadger({ apiKey: "your-api-key" });
-
-// Basic search (returns first page)
-const results = await client.twitter.tweets.search("python programming");
-for (const tweet of results.data) {
-  console.log(`@${tweet.username}: ${tweet.text}`);
-}
-
-// Paginate manually
-if (results.hasMore) {
-  const nextPage = await client.twitter.tweets.search("python programming", {
-    cursor: results.nextCursor,
-  });
-}
-
-// Automatic pagination with async iterators
-for await (const tweet of client.twitter.tweets.searchAll("python", { maxItems: 100 })) {
-  console.log(tweet.text);
-}
-
-// Collect all results into an array
-import { collectAll } from "scrapebadger";
-
-const tweets = await collectAll(
-  client.twitter.tweets.searchAll("python", { maxItems: 100 })
-);
-console.log(`Fetched ${tweets.length} tweets`);
-```
-
-### User Operations
-
-```typescript
-// Get user by username
-const user = await client.twitter.users.getByUsername("elonmusk");
-
-// Get user by ID
-const userById = await client.twitter.users.getById("44196397");
-
-// Get extended profile info
-const about = await client.twitter.users.getAbout("elonmusk");
-console.log(`Account based in: ${about.account_based_in}`);
-console.log(`Username changes: ${about.username_changes}`);
-
-// Get followers
-const followers = await client.twitter.users.getFollowers("elonmusk");
-for (const follower of followers.data) {
-  console.log(`@${follower.username}`);
-}
-
-// Iterate through all followers
-for await (const follower of client.twitter.users.getFollowersAll("elonmusk", {
-  maxItems: 1000,
-})) {
-  console.log(follower.username);
-}
-
-// Search users
-const users = await client.twitter.users.search("python developer");
-```
-
-### Lists
-
-```typescript
-// Get list details
-const list = await client.twitter.lists.getDetail("123456");
-console.log(`${list.name}: ${list.member_count} members`);
-
-// Get list tweets
-const tweets = await client.twitter.lists.getTweets("123456");
-
-// Get list members
-const members = await client.twitter.lists.getMembers("123456");
-
-// Search for lists
-const lists = await client.twitter.lists.search("tech leaders");
-```
-
-### Communities
-
-```typescript
-// Get community details
-const community = await client.twitter.communities.getDetail("123456");
-console.log(`${community.name}: ${community.member_count} members`);
-
-// Get community tweets
-const tweets = await client.twitter.communities.getTweets("123456", {
-  tweetType: "Latest",
-});
-
-// Search communities
-const communities = await client.twitter.communities.search("python developers");
-```
-
-### Trends
-
-```typescript
-// Get trending topics
-const trends = await client.twitter.trends.getTrends();
-for (const trend of trends.data) {
-  console.log(`${trend.name}: ${trend.tweet_count || "N/A"} tweets`);
-}
-
-// Get trends by category
-const newsTrends = await client.twitter.trends.getTrends({
-  category: "news",
-});
-
-// Get trends for a specific location
-const usTrends = await client.twitter.trends.getPlaceTrends(23424977); // US WOEID
-console.log(`Trends in ${usTrends.name}:`);
-for (const trend of usTrends.trends) {
-  console.log(`  - ${trend.name}`);
-}
-
-// Get available locations
-const locations = await client.twitter.trends.getAvailableLocations();
-```
-
-### Geographic Places
-
-```typescript
-// Search for places
-const places = await client.twitter.geo.search({ query: "San Francisco" });
-for (const place of places.data) {
-  console.log(`${place.full_name} (${place.place_type})`);
-}
-
-// Search by coordinates
-const nearby = await client.twitter.geo.search({
-  lat: 37.7749,
-  long: -122.4194,
-  granularity: "city",
-});
-
-// Get place details
-const place = await client.twitter.geo.getDetail("5a110d312052166f");
-```
-
-### Stream Monitoring
-
-Real-time tweet monitoring with WebSocket streaming and webhook delivery.
-
-```typescript
-import { ScrapeBadger, WebSocketStreamError } from "scrapebadger";
-
-const client = new ScrapeBadger({ apiKey: "your-api-key" });
-
-// Create a monitor
-const monitor = await client.twitter.stream.createMonitor({
-  name: "Tech Leaders",
-  usernames: ["elonmusk", "naval", "sama"],
-  pollIntervalSeconds: 10,
-  webhookUrl: "https://example.com/webhook",
-});
-console.log(`Created: ${monitor.id}, tier: ${monitor.pricing_tier}`);
-console.log(`Credits/hr: ${monitor.estimated_credits_per_hour}`);
-
-// List monitors
-const { monitors, total } = await client.twitter.stream.listMonitors({ status: "active" });
-console.log(`${total} active monitors`);
-
-// Pause / resume
-await client.twitter.stream.pauseMonitor(monitor.id);
-await client.twitter.stream.resumeMonitor(monitor.id);
-
-// Delete
-await client.twitter.stream.deleteMonitor(monitor.id);
-```
-
-#### EventEmitter streaming
-
-```typescript
-const stream = client.twitter.stream.connect({
-  reconnect: true,
-  reconnectDelaySeconds: 90,
-});
-
-stream.on("connected", (e) => {
-  console.log("Connected, connection ID:", e.connectionId);
-});
-
-stream.on("tweet", (event) => {
-  console.log(`@${event.authorUsername}: ${event.tweet.text}`);
-  console.log(`  latency: ${event.latencyMs}ms`);
-});
-
-stream.on("error", (err) => {
-  if (err instanceof WebSocketStreamError && err.code === 4001) {
-    console.error("API key rejected");
-  } else {
-    console.error("Stream error:", err.message);
-  }
-});
-
-stream.on("close", () => console.log("Stream closed"));
-
-// Later: graceful disconnect
-stream.close();
-```
-
-#### AsyncIterator streaming
+## Authentication
 
 ```typescript
-import { WebSocketStreamError } from "scrapebadger";
+// Pass API key directly
+const client = new ScrapeBadger({ apiKey: "sb_live_xxxxxxxxxxxxx" });
 
-try {
-  for await (const event of client.twitter.stream.connectIter({
-    reconnect: true,
-    reconnectDelaySeconds: 90,
-  })) {
-    if (event.type === "tweet") {
-      console.log(`@${event.authorUsername}: ${event.latencyMs}ms latency`);
-    }
-  }
-} catch (err) {
-  if (err instanceof WebSocketStreamError) {
-    console.error("Stream failed:", err.message, err.code);
-  }
-}
+// Or use environment variable SCRAPEBADGER_API_KEY
+const client = new ScrapeBadger();
 ```
 
-#### Webhook signature verification
+## Available APIs
 
-```typescript
-import { verifyWebhookSignature } from "scrapebadger/twitter";
-import express from "express";
-
-const app = express();
-
-app.post(
-  "/webhook",
-  express.raw({ type: "application/json" }),
-  (req, res) => {
-    const sig = req.headers["x-scrapebadger-signature"] as string;
-    if (!verifyWebhookSignature("my-webhook-secret", req.body, sig)) {
-      return res.status(401).json({ error: "Invalid signature" });
-    }
-    const event = JSON.parse(req.body.toString());
-    console.log("Received tweet:", event.tweet_id);
-    res.sendStatus(200);
-  }
-);
-```
+| API | Description | Documentation |
+|-----|-------------|---------------|
+| **Web Scraping** | Scrape any website with JS rendering, anti-bot bypass, and AI extraction | [Web Scraping Guide](docs/web-scraping.md) |
+| **Twitter** | 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams | [Twitter Guide](docs/twitter.md) |
+| **Google** | 19 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products | [Google Guide](docs/google.md) |
+| **Vinted** | Search items, get details, user profiles, and reference data across all Vinted markets | [Vinted Guide](docs/vinted.md) |
 
 ## Error Handling
 
@@ -383,44 +154,8 @@ slows down to spread requests across the remaining window — preventing 429 err
 
 This works transparently with all tier levels (Free: 60/min, Basic: 300/min,
 Pro: 1000/min, Enterprise: 5000/min).
-```
-
-## API Reference
-
-### Client
-
-- `ScrapeBadger` - Main client class
-
-### Twitter Module
-
-- `client.twitter.tweets` - Tweet operations
-- `client.twitter.users` - User operations
-- `client.twitter.lists` - List operations
-- `client.twitter.communities` - Community operations
-- `client.twitter.trends` - Trend operations
-- `client.twitter.geo` - Geographic place operations
-- `client.twitter.stream` - Real-time stream monitor management and WebSocket streaming
-
-### Stream Client Methods
-
-- `createMonitor(params)` - Create a stream monitor
-- `listMonitors(options?)` - List monitors with optional status filter
-- `getMonitor(id)` - Get a monitor by ID
-- `updateMonitor(id, params)` - Partially update a monitor
-- `pauseMonitor(id)` - Pause an active monitor
-- `resumeMonitor(id)` - Resume a paused monitor
-- `deleteMonitor(id)` - Delete a monitor (irreversible)
-- `listDeliveryLogs(options?)` - List tweet delivery logs
-- `listBillingLogs(options?)` - List billing activity logs
-- `connect(options?)` - Connect via EventEmitter (`.on("tweet", handler)`)
-- `connectIter(options?)` - Connect via AsyncIterator (`for await`)
-
-### Utilities
-
-- `collectAll(asyncIterator)` - Collect async iterator results into an array
-- `verifyWebhookSignature(secret, body, header)` - Verify incoming webhook HMAC signature
 
-### Exceptions
+## Exceptions
 
 - `ScrapeBadgerError` - Base exception class
 - `AuthenticationError` - Invalid or missing API key
@@ -444,6 +179,7 @@ MIT License - see [LICENSE](LICENSE) for details.
 
 ## Links
 
-- [Documentation](https://scrapebadger.com/docs)
+- [Documentation](https://docs.scrapebadger.com)
+- [Discord](https://discord.com/invite/3WvwTyWVCx)
 - [GitHub](https://github.com/scrape-badger/scrapebadger-node)
 - [npm](https://www.npmjs.com/package/scrapebadger)