@monostate/node-scraper 1.8.0 → 1.8.1

package/BULK_SCRAPING.md

@@ -0,0 +1,626 @@
+# Bulk Scraping Guide
+
+The `@monostate/node-scraper` package provides powerful bulk scraping capabilities with automatic request queueing, progress tracking, and efficient resource management.
+
+## Key Features
+
+- **Automatic Request Queueing**: Never worry about "too many browser instances" errors. Requests are automatically queued when the browser pool is full.
+- **Smart Browser Pooling**: Reuses browser instances for better performance while preventing memory leaks.
+- **Real-time Progress Tracking**: Monitor scraping progress with customizable callbacks.
+- **Streaming Support**: Process results as they complete for memory-efficient handling of large datasets.
+- **Graceful Error Handling**: Continue processing even when some URLs fail, with detailed error reporting.
+
+## Table of Contents
+
+- [Automatic Request Queueing](#automatic-request-queueing)
+- [Basic Usage](#basic-usage)
+- [Streaming Results](#streaming-results)
+- [Configuration Options](#configuration-options)
+- [Error Handling](#error-handling)
+- [Performance Optimization](#performance-optimization)
+- [Real-World Examples](#real-world-examples)
+- [Best Practices](#best-practices)
+
+## Automatic Request Queueing
+
+One of the most powerful features of v1.8.0 is automatic request queueing. When you make multiple concurrent requests:
+
+```javascript
+// Before v1.8.0: This would throw errors when browser pool is full
+// After v1.8.0: Requests are automatically queued!
+
+const urls = Array.from({ length: 100 }, (_, i) => `https://example.com/page${i}`);
+
+// Even with 100 URLs and only 3 browser instances, no errors!
+const results = await bulkScrape(urls, {
+  concurrency: 20,  // Request 20 at a time
+  method: 'puppeteer'  // Force browser usage
+});
+
+// The browser pool (max 3 instances) automatically queues requests
+// No "too many browser instances" errors!
+```
+
+### How It Works
+
+1. **Browser Pool**: Maximum of 3 browser instances by default
+2. **Request Queue**: When all browsers are busy, new requests wait in a queue
+3. **Automatic Processing**: As browsers become available, queued requests are processed
+4. **No Errors**: Instead of throwing errors, requests wait their turn
+
+### Benefits
+
+- **No Manual Retry Logic**: The SDK handles queueing automatically
+- **Memory Efficient**: Only 3 browsers maximum, preventing OOM errors
+- **Optimal Performance**: Browsers are reused for faster processing
+- **Graceful Degradation**: System remains stable under high load
+
+### Works Everywhere
+
+The automatic queueing works with all scraping methods, not just bulk operations:
+
+```javascript
+import { smartScrape } from '@monostate/node-scraper';
+
+// Make 50 parallel requests with Puppeteer
+// Only 3 browsers will be created, rest will queue automatically
+const promises = [];
+for (let i = 0; i < 50; i++) {
+  promises.push(
+    smartScrape(`https://example.com/page${i}`, { method: 'puppeteer' })
+  );
+}
+
+// All 50 requests complete successfully!
+// No "too many browser instances" errors
+const results = await Promise.all(promises);
+console.log(`Completed ${results.length} requests with only 3 browsers!`);
+```
+
+## Basic Usage
+
+### Simple Bulk Scraping
+
+```javascript
+import { bulkScrape } from '@monostate/node-scraper';
+
+const urls = [
+  'https://example1.com',
+  'https://example2.com',
+  'https://example3.com'
+];
+
+const results = await bulkScrape(urls);
+
+// Access results
+results.success.forEach(result => {
+  console.log(`URL: ${result.url}`);
+  console.log(`Method: ${result.method}`);
+  console.log(`Duration: ${result.duration}ms`);
+  console.log(`Content: ${result.content.substring(0, 100)}...`);
+});
+
+// Check failures
+results.failed.forEach(failure => {
+  console.log(`Failed URL: ${failure.url}`);
+  console.log(`Error: ${failure.error}`);
+});
+```
+
+### With Progress Tracking
+
+```javascript
+const results = await bulkScrape(urls, {
+  progressCallback: (progress) => {
+    console.log(`Progress: ${progress.percentage.toFixed(1)}%`);
+    console.log(`Current: ${progress.current}`);
+    console.log(`Processed: ${progress.processed}/${progress.total}`);
+  }
+});
+```
+
+## Streaming Results
+
+For large datasets, streaming allows you to process results as they complete:
+
+### Basic Streaming
+
+```javascript
+import { bulkScrapeStream } from '@monostate/node-scraper';
+
+const stats = await bulkScrapeStream(urls, {
+  onResult: (result) => {
+    console.log(`Success: ${result.url}`);
+    // Process immediately - save to database, write to file, etc.
+  },
+  onError: (error) => {
+    console.log(`Failed: ${error.url} - ${error.error}`);
+  }
+});
+
+console.log(`Total processed: ${stats.processed}`);
+console.log(`Success rate: ${(stats.successful / stats.total * 100).toFixed(1)}%`);
+```
+
+### Stream to File
+
+```javascript
+import { createWriteStream } from 'fs';
+import { bulkScrapeStream } from '@monostate/node-scraper';
+
+const outputStream = createWriteStream('results.jsonl');
+
+await bulkScrapeStream(urls, {
+  onResult: async (result) => {
+    // Write each result as a JSON line
+    outputStream.write(JSON.stringify(result) + '\n');
+  },
+  onError: async (error) => {
+    // Log errors to a separate file
+    outputStream.write(JSON.stringify({ ...error, isError: true }) + '\n');
+  }
+});
+
+outputStream.end();
+```
+
+### Stream to Database
+
+```javascript
+import { bulkScrapeStream } from '@monostate/node-scraper';
+import { db } from './database.js';
+
+await bulkScrapeStream(urls, {
+  concurrency: 10,
+  onResult: async (result) => {
+    await db.scraped_pages.insert({
+      url: result.url,
+      content: result.content,
+      method: result.method,
+      duration_ms: result.duration,
+      scraped_at: new Date(result.timestamp)
+    });
+  },
+  onError: async (error) => {
+    await db.scrape_errors.insert({
+      url: error.url,
+      error_message: error.error,
+      failed_at: new Date(error.timestamp)
+    });
+  }
+});
+```
+
+## Configuration Options
+
+### Concurrency Control
+
+```javascript
+// Low concurrency for rate-limited sites
+const results = await bulkScrape(urls, {
+  concurrency: 2,  // Only 2 parallel requests
+  timeout: 30000   // 30 second timeout per request
+});
+
+// High concurrency for your own servers
+const results = await bulkScrape(internalUrls, {
+  concurrency: 20,  // 20 parallel requests
+  timeout: 5000     // 5 second timeout
+});
+```
+
+### Method Selection
+
+```javascript
+// Force all URLs to use Puppeteer
+const results = await bulkScrape(urls, {
+  method: 'puppeteer',
+  concurrency: 3  // Puppeteer is resource-intensive
+});
+
+// Force direct fetch for known static sites
+const results = await bulkScrape(staticUrls, {
+  method: 'direct',
+  concurrency: 50  // Direct fetch can handle high concurrency
+});
+```
+
+### Continue vs Stop on Error
+
+```javascript
+// Continue processing even if some URLs fail (default)
+const results = await bulkScrape(urls, {
+  continueOnError: true
+});
+
+// Stop immediately on first error
+try {
+  const results = await bulkScrape(urls, {
+    continueOnError: false
+  });
+} catch (error) {
+  console.error('Bulk scraping stopped due to error:', error);
+}
+```
+
+## Error Handling
+
+### Retry Failed URLs
+
+```javascript
+// First pass
+const results = await bulkScrape(urls);
+
+// Retry failures with different method
+if (results.failed.length > 0) {
+  const failedUrls = results.failed.map(f => f.url);
+  const retryResults = await bulkScrape(failedUrls, {
+    method: 'puppeteer',  // Try with full browser
+    timeout: 60000        // Longer timeout
+  });
+}
+```
+
+### Custom Error Handling
+
+```javascript
+await bulkScrapeStream(urls, {
+  onResult: (result) => {
+    // Process successful results
+  },
+  onError: async (error) => {
+    // Categorize and handle different error types
+    if (error.error.includes('timeout')) {
+      await logTimeoutError(error);
+    } else if (error.error.includes('404')) {
+      await handle404(error);
+    } else {
+      await logGeneralError(error);
+    }
+  }
+});
+```
+
+## Performance Optimization
+
+### Dynamic Concurrency
+
+```javascript
+// Start with low concurrency and increase based on success rate
+let concurrency = 5;
+const batchSize = 100;
+
+for (let i = 0; i < urls.length; i += batchSize) {
+  const batch = urls.slice(i, i + batchSize);
+  
+  const results = await bulkScrape(batch, { concurrency });
+  
+  const successRate = results.stats.successful / batch.length;
+  
+  // Adjust concurrency based on success rate
+  if (successRate > 0.95) {
+    concurrency = Math.min(concurrency + 2, 20);
+  } else if (successRate < 0.8) {
+    concurrency = Math.max(concurrency - 2, 2);
+  }
+  
+  console.log(`Batch complete. Success rate: ${(successRate * 100).toFixed(1)}%. New concurrency: ${concurrency}`);
+}
+```
+
+### Memory-Efficient Processing
+
+```javascript
+// Process in chunks to avoid memory issues
+async function processLargeDataset(allUrls) {
+  const chunkSize = 1000;
+  const results = {
+    successful: 0,
+    failed: 0,
+    totalTime: 0
+  };
+  
+  for (let i = 0; i < allUrls.length; i += chunkSize) {
+    const chunk = allUrls.slice(i, i + chunkSize);
+    console.log(`Processing chunk ${i / chunkSize + 1} of ${Math.ceil(allUrls.length / chunkSize)}`);
+    
+    const chunkResults = await bulkScrape(chunk, {
+      concurrency: 10,
+      progressCallback: (p) => {
+        const overallProgress = ((i + p.processed) / allUrls.length * 100).toFixed(1);
+        console.log(`Overall progress: ${overallProgress}%`);
+      }
+    });
+    
+    results.successful += chunkResults.stats.successful;
+    results.failed += chunkResults.stats.failed;
+    results.totalTime += chunkResults.stats.totalTime;
+    
+    // Optional: Process results immediately to free memory
+    await processChunkResults(chunkResults);
+  }
+  
+  return results;
+}
+```
+
+## Real-World Examples
+
+### E-commerce Price Monitoring
+
+```javascript
+import { bulkScrapeStream } from '@monostate/node-scraper';
+
+const productUrls = [
+  'https://shop1.com/product/laptop-123',
+  'https://shop2.com/items/laptop-123',
+  // ... hundreds of product URLs
+];
+
+const priceData = [];
+
+await bulkScrapeStream(productUrls, {
+  concurrency: 5,
+  onResult: async (result) => {
+    // Extract price from scraped content
+    const content = JSON.parse(result.content);
+    const price = extractPrice(content);
+    
+    priceData.push({
+      url: result.url,
+      price: price,
+      timestamp: result.timestamp
+    });
+  },
+  progressCallback: (progress) => {
+    process.stdout.write(`\rChecking prices: ${progress.percentage.toFixed(0)}%`);
+  }
+});
+
+// Analyze price data
+const avgPrice = priceData.reduce((sum, p) => sum + p.price, 0) / priceData.length;
+console.log(`\nAverage price: $${avgPrice.toFixed(2)}`);
+```
+
+### News Aggregation
+
+```javascript
+import { bulkScrape } from '@monostate/node-scraper';
+
+const newsUrls = [
+  'https://news1.com/latest',
+  'https://news2.com/today',
+  'https://news3.com/breaking'
+];
+
+const results = await bulkScrape(newsUrls, {
+  concurrency: 3,
+  timeout: 10000
+});
+
+// Extract and combine articles
+const allArticles = [];
+results.success.forEach(result => {
+  const content = JSON.parse(result.content);
+  const articles = extractArticles(content);
+  allArticles.push(...articles.map(a => ({
+    ...a,
+    source: new URL(result.url).hostname
+  })));
+});
+
+// Sort by date and deduplicate
+const uniqueArticles = deduplicateArticles(allArticles);
+console.log(`Found ${uniqueArticles.length} unique articles`);
+```
+
+### SEO Analysis
+
+```javascript
+import { bulkScrape } from '@monostate/node-scraper';
+
+async function analyzeSEO(urls) {
+  const results = await bulkScrape(urls, {
+    concurrency: 10,
+    method: 'auto'
+  });
+  
+  const seoData = results.success.map(result => {
+    const content = JSON.parse(result.content);
+    return {
+      url: result.url,
+      title: content.title,
+      metaDescription: content.metaDescription,
+      headings: content.headings,
+      loadTime: result.duration,
+      method: result.method,
+      hasStructuredData: !!content.structuredData
+    };
+  });
+  
+  // Generate SEO report
+  const avgLoadTime = seoData.reduce((sum, d) => sum + d.loadTime, 0) / seoData.length;
+  const missingTitles = seoData.filter(d => !d.title).length;
+  const missingDescriptions = seoData.filter(d => !d.metaDescription).length;
+  
+  return {
+    totalAnalyzed: seoData.length,
+    avgLoadTime: Math.round(avgLoadTime),
+    missingTitles,
+    missingDescriptions,
+    details: seoData
+  };
+}
+```
+
+## Best Practices
+
+### 1. Respect Rate Limits
+
+```javascript
+// Add delays between requests for external sites
+async function respectfulBulkScrape(urls, delayMs = 1000) {
+  const results = [];
+  
+  for (const url of urls) {
+    const result = await smartScrape(url);
+    results.push(result);
+    
+    // Wait before next request
+    await new Promise(resolve => setTimeout(resolve, delayMs));
+  }
+  
+  return results;
+}
+```
+
+### 2. Handle Different Content Types
+
+```javascript
+const results = await bulkScrape(mixedUrls, {
+  progressCallback: (progress) => {
+    console.log(`Processing: ${progress.current}`);
+  }
+});
+
+// Separate different content types
+const pdfResults = results.success.filter(r => r.contentType?.includes('pdf'));
+const htmlResults = results.success.filter(r => !r.contentType?.includes('pdf'));
+
+console.log(`Found ${pdfResults.length} PDFs and ${htmlResults.length} web pages`);
+```
+
+### 3. Monitor Resource Usage
+
+```javascript
+import { BNCASmartScraper } from '@monostate/node-scraper';
+import browserPool from '@monostate/node-scraper/browser-pool.js';
+
+const scraper = new BNCASmartScraper({ verbose: true });
+
+// Monitor memory usage during bulk scraping
+const memoryUsage = [];
+const interval = setInterval(() => {
+  memoryUsage.push(process.memoryUsage());
+  
+  // Log browser pool statistics
+  const poolStats = browserPool.getStats();
+  console.log('Browser Pool:', {
+    active: poolStats.busyCount,
+    idle: poolStats.idleCount,
+    queued: poolStats.queueLength,
+    totalCreated: poolStats.created,
+    reused: poolStats.reused
+  });
+}, 1000);
+
+try {
+  const results = await scraper.bulkScrape(urls, {
+    concurrency: 10,
+    progressCallback: (p) => {
+      const mem = process.memoryUsage();
+      console.log(`Progress: ${p.percentage.toFixed(1)}% | Memory: ${(mem.heapUsed / 1024 / 1024).toFixed(1)}MB`);
+    }
+  });
+} finally {
+  clearInterval(interval);
+  await scraper.cleanup();
+}
+```
+
+### 4. Implement Circuit Breaker
+
+```javascript
+class CircuitBreaker {
+  constructor(threshold = 5, timeout = 60000) {
+    this.failureCount = 0;
+    this.threshold = threshold;
+    this.timeout = timeout;
+    this.state = 'CLOSED';
+    this.nextAttempt = Date.now();
+  }
+  
+  async call(fn) {
+    if (this.state === 'OPEN') {
+      if (Date.now() < this.nextAttempt) {
+        throw new Error('Circuit breaker is OPEN');
+      }
+      this.state = 'HALF_OPEN';
+    }
+    
+    try {
+      const result = await fn();
+      this.onSuccess();
+      return result;
+    } catch (error) {
+      this.onFailure();
+      throw error;
+    }
+  }
+  
+  onSuccess() {
+    this.failureCount = 0;
+    this.state = 'CLOSED';
+  }
+  
+  onFailure() {
+    this.failureCount++;
+    if (this.failureCount >= this.threshold) {
+      this.state = 'OPEN';
+      this.nextAttempt = Date.now() + this.timeout;
+    }
+  }
+}
+
+// Use with bulk scraping
+const breaker = new CircuitBreaker();
+const results = [];
+
+for (const url of urls) {
+  try {
+    const result = await breaker.call(() => smartScrape(url));
+    results.push(result);
+  } catch (error) {
+    console.error(`Failed to scrape ${url}: ${error.message}`);
+  }
+}
+```
+
+## Performance Tips
+
+1. **Use appropriate concurrency**: Start with 5-10 concurrent requests and adjust based on performance
+2. **Choose the right method**: Use `direct` for static sites, `puppeteer` for SPAs
+3. **Stream large datasets**: Use `bulkScrapeStream` for datasets over 1000 URLs
+4. **Monitor memory usage**: Process results immediately in streaming mode
+5. **Implement retry logic**: Some failures are temporary
+6. **Cache results**: Avoid re-scraping unchanged content
+7. **Use timeouts**: Prevent hanging requests from blocking progress
+
+## Troubleshooting
+
+### High Memory Usage
+
+If you experience high memory usage:
+
+1. Reduce concurrency
+2. Use streaming mode
+3. Process results immediately
+4. Call `cleanup()` periodically
+
+### Slow Performance
+
+If scraping is slow:
+
+1. Increase concurrency (if server allows)
+2. Use `direct` method when possible
+3. Reduce timeout values
+4. Check network connectivity
+
+### Many Failures
+
+If many URLs fail:
+
+1. Check if sites require authentication
+2. Verify URLs are correct
+3. Use `puppeteer` method for JavaScript-heavy sites
+4. Implement retry logic with backoff

package/README.md

@@ -19,6 +19,8 @@ yarn add @monostate/node-scraper
 pnpm add @monostate/node-scraper
 ```
 
+**Fixed in v1.8.1**: Critical production fix - browser-pool.js now included in npm package.
+
 **New in v1.8.0**: Bulk scraping with automatic request queueing, progress tracking, and streaming results! Process hundreds of URLs efficiently. Plus critical memory leak fix with browser pooling.
 
 **Fixed in v1.7.0**: Critical cross-platform compatibility fix - binaries are now correctly downloaded per platform instead of being bundled.

package/browser-pool.js

@@ -0,0 +1,229 @@
+class BrowserPool {
+    constructor(maxInstances = 3, idleTimeout = 5000) {
+        this.maxInstances = maxInstances;
+        this.idleTimeout = idleTimeout;
+        this.pool = [];
+        this.busyBrowsers = new Set();
+        this.cleanupTimer = null;
+        this.requestQueue = [];
+        this.stats = {
+            created: 0,
+            reused: 0,
+            queued: 0,
+            cleaned: 0
+        };
+    }
+
+    async getBrowser() {
+        // Try to get an idle browser from pool
+        let browser = this.pool.find(b => !this.busyBrowsers.has(b.instance));
+        
+        if (browser) {
+            browser.lastUsed = Date.now();
+            this.busyBrowsers.add(browser.instance);
+            this.stats.reused++;
+            return browser.instance;
+        }
+
+        // Create new browser if under limit
+        if (this.pool.length < this.maxInstances) {
+            browser = await this.createBrowser();
+            this.pool.push(browser);
+            this.busyBrowsers.add(browser.instance);
+            this.stats.created++;
+            return browser.instance;
+        }
+
+        // Queue the request and wait for available browser
+        this.stats.queued++;
+        return this.queueRequest();
+    }
+
+    async createBrowser() {
+        const puppeteer = await this.getPuppeteer();
+        const instance = await puppeteer.launch({
+            headless: 'new',
+            args: [
+                '--no-sandbox',
+                '--disable-setuid-sandbox',
+                '--disable-dev-shm-usage',
+                '--disable-gpu',
+                '--disable-web-security',
+                '--disable-features=VizDisplayCompositor',
+                '--disable-background-timer-throttling',
+                '--disable-backgrounding-occluded-windows',
+                '--disable-renderer-backgrounding',
+                '--disable-extensions',
+                '--disable-default-apps',
+                '--disable-sync',
+                '--metrics-recording-only',
+                '--mute-audio',
+                '--no-first-run'
+            ]
+        });
+
+        const browser = {
+            instance,
+            created: Date.now(),
+            lastUsed: Date.now(),
+            pageCount: 0
+        };
+
+        // Handle browser disconnect
+        instance.on('disconnected', () => {
+            this.removeBrowser(browser);
+            this.processQueue();
+        });
+
+        return browser;
+    }
+
+    async getPuppeteer() {
+        try {
+            const puppeteer = await import('puppeteer');
+            return puppeteer.default || puppeteer;
+        } catch (error) {
+            throw new Error('Puppeteer is not installed. Please install it to use Puppeteer-based scraping.');
+        }
+    }
+
+    async queueRequest() {
+        return new Promise((resolve) => {
+            this.requestQueue.push({ resolve, timestamp: Date.now() });
+        });
+    }
+
+    processQueue() {
+        if (this.requestQueue.length === 0) return;
+
+        // Find available browser
+        const available = this.pool.find(b => !this.busyBrowsers.has(b.instance));
+        if (!available) return;
+
+        // Process oldest request in queue
+        const request = this.requestQueue.shift();
+        if (request) {
+            available.lastUsed = Date.now();
+            this.busyBrowsers.add(available.instance);
+            request.resolve(available.instance);
+        }
+    }
+
+    releaseBrowser(browser) {
+        this.busyBrowsers.delete(browser);
+        
+        // Process any queued requests
+        this.processQueue();
+        
+        // Start cleanup timer if not already running
+        if (!this.cleanupTimer) {
+            this.cleanupTimer = setTimeout(() => this.cleanup(), this.idleTimeout);
+        }
+    }
+
+    removeBrowser(browserObj) {
+        const index = this.pool.findIndex(b => b.instance === browserObj.instance);
+        if (index !== -1) {
+            this.pool.splice(index, 1);
+            this.busyBrowsers.delete(browserObj.instance);
+        }
+    }
+
+    async cleanup() {
+        this.cleanupTimer = null;
+        const now = Date.now();
+        const toRemove = [];
+
+        // Keep at least one browser if there are queued requests
+        const minBrowsers = this.requestQueue.length > 0 ? 1 : 0;
+
+        for (const browser of this.pool) {
+            // Skip if we need to keep minimum browsers
+            if (this.pool.length - toRemove.length <= minBrowsers) break;
+            
+            // Remove idle browsers
+            const isIdle = !this.busyBrowsers.has(browser.instance);
+            const idleTime = now - browser.lastUsed;
+            
+            if (isIdle && idleTime > this.idleTimeout) {
+                toRemove.push(browser);
+            }
+        }
+
+        // Close idle browsers
+        for (const browser of toRemove) {
+            try {
+                // Check if browser is still connected
+                if (browser.instance && browser.instance.isConnected()) {
+                    await browser.instance.close();
+                }
+                this.removeBrowser(browser);
+                this.stats.cleaned++;
+            } catch (error) {
+                // Silently ignore protocol errors and disconnection errors
+                if (!error.message.includes('Protocol error') && 
+                    !error.message.includes('Target closed') &&
+                    !error.message.includes('Connection closed')) {
+                    console.warn('Error closing browser:', error.message);
+                }
+                // Remove browser even if close failed
+                this.removeBrowser(browser);
+            }
+        }
+
+        // Schedule next cleanup if there are still browsers
+        if (this.pool.length > 0) {
+            this.cleanupTimer = setTimeout(() => this.cleanup(), this.idleTimeout);
+        }
+    }
+
+    async closeAll() {
+        if (this.cleanupTimer) {
+            clearTimeout(this.cleanupTimer);
+            this.cleanupTimer = null;
+        }
+
+        // Clear the queue
+        this.requestQueue = [];
+
+        const closePromises = this.pool.map(async (browser) => {
+            try {
+                // Check if browser is still connected
+                if (browser.instance && browser.instance.isConnected()) {
+                    await browser.instance.close();
+                }
+            } catch (error) {
+                // Silently ignore protocol errors and disconnection errors
+                if (!error.message.includes('Protocol error') && 
+                    !error.message.includes('Target closed') &&
+                    !error.message.includes('Connection closed')) {
+                    console.warn('Error closing browser:', error.message);
+                }
+            }
+        });
+
+        await Promise.all(closePromises);
+        this.pool = [];
+        this.busyBrowsers.clear();
+    }
+
+    getStats() {
+        return {
+            ...this.stats,
+            poolSize: this.pool.length,
+            busyCount: this.busyBrowsers.size,
+            idleCount: this.pool.length - this.busyBrowsers.size,
+            queueLength: this.requestQueue.length
+        };
+    }
+}
+
+// Global browser pool instance
+const browserPool = new BrowserPool(3, 5000);
+
+// Graceful shutdown
+process.on('SIGTERM', () => browserPool.closeAll());
+process.on('SIGINT', () => browserPool.closeAll());
+process.on('beforeExit', () => browserPool.closeAll());
+
+export default browserPool;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@monostate/node-scraper",
-  "version": "1.8.0",
+  "version": "1.8.1",
   "description": "Intelligent web scraping with AI Q&A, PDF support and multi-level fallback system - 11x faster than traditional scrapers",
   "type": "module",
   "main": "index.js",
@@ -14,7 +14,9 @@
   "files": [
     "index.js",
     "index.d.ts",
+    "browser-pool.js",
     "README.md",
+    "BULK_SCRAPING.md",
     "package.json",
     "scripts/"
   ],