@dealcrawl/sdk 2.1.3 → 2.3.0

package/README.md

@@ -8,7 +8,7 @@ Official TypeScript SDK for the DealCrawl web scraping and crawling API.
 
 ## Features
 
-- 🚀 **Full API Coverage** - Access all 41 DealCrawl API endpoints
+- 🚀 **Full API Coverage** - Access all 45 DealCrawl API endpoints
 - 📦 **Zero Dependencies** - Uses native `fetch`, works everywhere
 - 🔒 **Type-Safe** - Complete TypeScript definitions
 - ⚡ **Automatic Retries** - Built-in retry logic with exponential backoff
@@ -98,6 +98,98 @@ const job = await client.scrape.withScreenshot("https://example.com", {
 | `excludeTags` | string[] | - | HTML tags to exclude |
 | `onlyMainContent` | boolean | true | Extract main content only |
 
+### Batch Scrape - Bulk URL Scraping (NEW)
+
+```typescript
+// Scrape multiple URLs in one request (1-100 URLs)
+const batch = await client.scrape.batch({
+  urls: [
+    { url: "https://shop1.com/product1" },
+    { url: "https://shop2.com/deal", extractDeal: true },
+    { url: "https://shop3.com/sale", screenshot: { enabled: true } },
+  ],
+  defaults: {
+    detectSignals: true,
+    timeout: 30000,
+  },
+});
+
+// Get batch status
+const status = await client.scrape.getBatchStatus(batch.batchId);
+
+// Wait for all batch jobs
+const results = await client.waitForAll(batch.jobIds);
+```
+
+**Batch Options:**
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `urls` | array | required | 1-100 URL objects with optional overrides |
+| `defaults` | object | - | Default options applied to all URLs |
+| `priority` | number | 5 | Priority 1-10 (higher = faster) |
+| `delay` | number | 0 | Delay between URLs (0-5000ms) |
+| `webhookUrl` | string | - | Webhook for batch completion |
+| `ref` | string | - | Custom reference ID for tracking |
+
+### Search - Web Search with AI (NEW)
+
+```typescript
+// Basic search
+const job = await client.search.create({
+  query: "laptop deals black friday",
+  maxResults: 20,
+});
+
+// AI-optimized search with deal scoring
+const job = await client.search.create({
+  query: "iPhone discount",
+  useAiOptimization: true,
+  aiProvider: "openai",
+  aiModel: "gpt-4o-mini",
+  useDealScoring: true,
+});
+
+// Search with auto-scraping of results
+const job = await client.search.create({
+  query: "promo codes electronics",
+  autoScrape: true,
+  autoScrapeLimit: 5,
+});
+
+// Filtered search
+const job = await client.search.create({
+  query: "software deals",
+  filters: {
+    location: "fr",
+    language: "fr",
+    dateRange: "month",
+    domains: ["amazon.fr", "cdiscount.com"],
+  },
+});
+
+// Check search API status
+const status = await client.search.getStatus();
+
+// Convenience: search and wait
+const result = await client.searchAndWait({
+  query: "gaming laptop deals",
+  useDealScoring: true,
+});
+```
+
+**Search Options:**
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `query` | string | required | Search query |
+| `maxResults` | number | 10 | Results to return (1-100) |
+| `useAiOptimization` | boolean | false | AI-enhance the query |
+| `aiProvider` | string | "openai" | "openai" or "anthropic" |
+| `aiModel` | string | - | Model ID (gpt-4o-mini, claude-3-5-sonnet, etc.) |
+| `useDealScoring` | boolean | false | Score results for deal relevance |
+| `autoScrape` | boolean | false | Auto-scrape top results |
+| `autoScrapeLimit` | number | 3 | Number of results to scrape |
+| `filters` | object | - | Location, language, date, domains |
+
 ### Crawl - Website Crawling
 
 ```typescript
@@ -295,8 +387,10 @@ const rotated = await client.keys.rotate(keyId, {
   newName: "Production Key v2",
 });
 
-// Revoke a key
-await client.keys.revoke(keyId);
+// Revoke a key with reason
+await client.keys.revoke(keyId, {
+  reason: "Key compromised - rotating for security",
+});
 
 // Get key stats
 const stats = await client.keys.getStats(keyId, { days: 30 });
@@ -304,17 +398,19 @@ const stats = await client.keys.getStats(keyId, { days: 30 });
 
 **Available Scopes:**
 
-| Scope             | Endpoint              | Description            |
-| ----------------- | --------------------- | ---------------------- |
-| `scrape`          | `POST /v1/scrape`     | Create scrape jobs     |
-| `crawl`           | `POST /v1/crawl`      | Create crawl jobs      |
-| `dork`            | `POST /v1/dork`       | Create dork searches   |
-| `extract`         | `POST /v1/extract`    | Create extraction jobs |
-| `status`          | `GET /v1/status/:id`  | Read job status        |
-| `data:read`       | `GET /v1/data/*`      | Read jobs/deals        |
-| `data:export`     | `GET /v1/data/export` | Export data            |
-| `keys:manage`     | `/v1/keys`            | Manage API keys        |
-| `webhooks:manage` | `/v1/webhooks`        | Manage webhooks        |
+| Scope             | Endpoint                | Description              |
+| ----------------- | ----------------------- | ------------------------ |
+| `scrape`          | `POST /v1/scrape`       | Create scrape jobs       |
+| `scrape:batch`    | `POST /v1/scrape/batch` | Create batch scrape jobs |
+| `search`          | `POST /v1/search`       | Create search jobs       |
+| `crawl`           | `POST /v1/crawl`        | Create crawl jobs        |
+| `dork`            | `POST /v1/dork`         | Create dork searches     |
+| `extract`         | `POST /v1/extract`      | Create extraction jobs   |
+| `status`          | `GET /v1/status/:id`    | Read job status          |
+| `data:read`       | `GET /v1/data/*`        | Read jobs/deals          |
+| `data:export`     | `GET /v1/data/export`   | Export data              |
+| `keys:manage`     | `/v1/keys`              | Manage API keys          |
+| `webhooks:manage` | `/v1/webhooks`          | Manage webhooks          |
 
 **Scope Examples:**
 
@@ -406,6 +502,43 @@ const result = await client.crawlAndWait({
 });
 ```
 
+## Field Selection (NEW)
+
+Reduce response payload size by selecting only the fields you need:
+
+```typescript
+// Select specific fields from job status
+const status = await client.status.get(jobId, {
+  fields: ["id", "status", "progress", "result.title", "result.url"],
+});
+
+// Select fields from deals list
+const deals = await client.data.listDeals({
+  minScore: 70,
+  fields: ["id", "title", "price", "discount", "dealScore"],
+});
+
+// Nested field selection
+const jobs = await client.data.listJobs({
+  fields: ["id", "status", "result.deals.title", "result.deals.price"],
+});
+```
+
+**Benefits:**
+
+- 85-90% payload reduction for large responses
+- Faster API responses
+- Lower bandwidth usage
+
+**Supported Endpoints:**
+
+- `GET /v1/status/:jobId`
+- `GET /v1/status/:jobId/deals`
+- `GET /v1/status/:jobId/metrics`
+- `GET /v1/data/jobs`
+- `GET /v1/data/deals`
+- `GET /v1/data/:jobId`
+
 ## Error Handling
 
 ```typescript