@monostate/node-scraper 1.2.0 → 1.3.0

package/README.md

@@ -19,7 +19,9 @@ yarn add @monostate/node-scraper
 pnpm add @monostate/node-scraper
 ```
 
-**🎉 New in v1.2.0**: Lightpanda binary is now automatically downloaded and configured during installation! No manual setup required.
+**🎉 New in v1.3.0**: PDF parsing support added! Automatically extracts text, metadata, and page count from PDF documents.
+
+**✨ Also in v1.2.0**: Lightpanda binary is now automatically downloaded and configured during installation! No manual setup required.
 
 ### Zero-Configuration Setup
 
@@ -45,6 +47,10 @@ console.log(screenshot.screenshot); // Base64 encoded image
 // Quick screenshot (optimized for speed)
 const quick = await quickShot('https://example.com');
 console.log(quick.screenshot); // Fast screenshot capture
+
+// PDF parsing (automatic detection)
+const pdfResult = await smartScrape('https://example.com/document.pdf');
+console.log(pdfResult.content); // Extracted text, metadata, page count
 ```
 
 ### Advanced Usage
@@ -66,12 +72,13 @@ await scraper.cleanup(); // Clean up resources
 
 ## 🔧 How It Works
 
-BNCA uses a sophisticated 3-tier fallback system:
+BNCA uses a sophisticated multi-tier system with intelligent detection:
 
 ### 1. 🔄 Direct Fetch (Fastest)
 - Pure HTTP requests with intelligent HTML parsing
 - **Performance**: Sub-second responses
 - **Success rate**: 75% of websites
+- **PDF Detection**: Automatically detects PDFs by URL, content-type, or magic bytes
 
 ### 2. 🐼 Lightpanda Browser (Fast)
 - Lightweight browser engine (2-3x faster than Chromium)
@@ -83,6 +90,12 @@ BNCA uses a sophisticated 3-tier fallback system:
 - **Performance**: Complete JavaScript execution
 - **Fallback triggers**: Complex interactions needed
 
+### 📄 PDF Parser (Specialized)
+- Automatic PDF detection and parsing
+- **Features**: Text extraction, metadata, page count
+- **Smart Detection**: Works even when PDFs are served with wrong content-types
+- **Performance**: Typically 100-500ms for most PDFs
+
 ### 📸 Screenshot Methods
 - **Chrome CLI**: Direct Chrome screenshot capture
 - **Quickshot**: Optimized with retry logic and smart timeouts
@@ -186,6 +199,34 @@ Clean up resources (close browser instances).
 await scraper.cleanup();
 ```
 
+### 📄 PDF Support
+
+BNCA automatically detects and parses PDF documents:
+
+```javascript
+const pdfResult = await smartScrape('https://example.com/document.pdf');
+
+// Parsed content includes:
+const content = JSON.parse(pdfResult.content);
+console.log(content.title);          // PDF title
+console.log(content.author);         // Author name
+console.log(content.pages);          // Number of pages
+console.log(content.text);           // Full extracted text
+console.log(content.creationDate);   // Creation date
+console.log(content.metadata);       // Additional metadata
+```
+
+**PDF Detection Methods:**
+- URL ending with `.pdf`
+- Content-Type header `application/pdf`
+- Binary content starting with `%PDF` (magic bytes)
+- Works with PDFs served as `application/octet-stream` (e.g., GitHub raw files)
+
+**Limitations:**
+- Maximum file size: 20MB
+- Text extraction only (no image OCR)
+- Requires `pdf-parse` dependency (automatically installed)
+
 ## 📱 Next.js Integration
 
 ### API Route Example
@@ -354,7 +395,14 @@ const result: ScrapingResult = await scraper.scrape('https://example.com');
 
 ## 📋 Changelog
 
-### v1.2.0 (Latest)
+### v1.3.0 (Latest)
+- 📄 **PDF Support**: Full PDF parsing with text extraction, metadata, and page count
+- 🔍 **Smart PDF Detection**: Detects PDFs by URL patterns, content-type, or magic bytes
+- 🚀 **Robust Parsing**: Handles PDFs served with incorrect content-types (e.g., GitHub raw files)
+- ⚡ **Fast Performance**: PDF parsing typically completes in 100-500ms
+- 📊 **Comprehensive Extraction**: Title, author, creation date, page count, and full text
+
+### v1.2.0
 - 🎉 **Auto-Installation**: Lightpanda binary is now automatically downloaded during `npm install`
 - 🔧 **Cross-Platform Support**: Automatic detection and installation for macOS, Linux, and Windows/WSL
 - ⚡ **Improved Performance**: Enhanced binary detection and ES6 module compatibility

package/index.js

@@ -5,6 +5,7 @@ import { existsSync, statSync } from 'fs';
 import path from 'path';
 import { fileURLToPath } from 'url';
 import { promises as fsPromises } from 'fs';
+import pdfParse from 'pdf-parse/lib/pdf-parse.js';
 
 let puppeteer = null;
 try {
@@ -42,7 +43,8 @@ export class BNCASmartScraper {
     this.stats = {
       directFetch: { attempts: 0, successes: 0 },
       lightpanda: { attempts: 0, successes: 0 },
-      puppeteer: { attempts: 0, successes: 0 }
+      puppeteer: { attempts: 0, successes: 0 },
+      pdf: { attempts: 0, successes: 0 }
     };
   }
   
@@ -60,6 +62,35 @@ export class BNCASmartScraper {
     let lastError = null;
     
     try {
+      // Check if URL is a PDF (by extension or content-type check)
+      const isPdfUrl = url.toLowerCase().endsWith('.pdf') || 
+                       url.toLowerCase().includes('.pdf?') ||
+                       url.toLowerCase().includes('/pdf/');
+                       
+      if (isPdfUrl) {
+        this.log('  📄 PDF detected, using PDF parser...');
+        result = await this.tryPDFParse(url, config);
+        
+        if (result.success) {
+          method = 'pdf';
+          this.log('  ✅ PDF parsing successful');
+          
+          const totalTime = Date.now() - startTime;
+          return {
+            ...result,
+            method,
+            performance: {
+              totalTime,
+              method
+            },
+            stats: this.getStats()
+          };
+        } else {
+          this.log('  ❌ PDF parsing failed');
+          lastError = result.error;
+        }
+      }
+      
       // Step 1: Try direct fetch first (fastest)
       this.log('  🔄 Attempting direct fetch...');
       result = await this.tryDirectFetch(url, config);
@@ -67,6 +98,29 @@ export class BNCASmartScraper {
       if (result.success && !result.needsBrowser) {
         method = 'direct-fetch';
         this.log('  ✅ Direct fetch successful');
+      } else if (result.isPdf) {
+        // Direct fetch detected a PDF, try PDF parser
+        this.log('  📄 Direct fetch detected PDF content, using PDF parser...');
+        result = await this.tryPDFParse(url, config);
+        
+        if (result.success) {
+          method = 'pdf';
+          this.log('  ✅ PDF parsing successful');
+          
+          const totalTime = Date.now() - startTime;
+          return {
+            ...result,
+            method,
+            performance: {
+              totalTime,
+              method
+            },
+            stats: this.getStats()
+          };
+        } else {
+          this.log('  ❌ PDF parsing failed');
+          lastError = result.error;
+        }
       } else {
         this.log(result.needsBrowser ? '  ⚠️  Browser rendering required' : '  ❌ Direct fetch failed');
         lastError = result.error;
@@ -152,7 +206,32 @@ export class BNCASmartScraper {
         };
       }
       
-      const html = await response.text();
+      // Check if the response is actually a PDF
+      const contentType = response.headers.get('content-type') || '';
+      if (contentType.includes('application/pdf')) {
+        return {
+          success: false,
+          error: 'Content is PDF, should use PDF parser',
+          isPdf: true
+        };
+      }
+      
+      // Get response as array buffer to check magic bytes
+      const buffer = await response.arrayBuffer();
+      const firstBytes = new Uint8Array(buffer.slice(0, 5));
+      const signature = Array.from(firstBytes).map(b => String.fromCharCode(b)).join('');
+      
+      // Check for PDF magic bytes
+      if (signature.startsWith('%PDF')) {
+        return {
+          success: false,
+          error: 'Content is PDF (detected by magic bytes), should use PDF parser',
+          isPdf: true
+        };
+      }
+      
+      // Convert buffer back to text for HTML processing
+      const html = new TextDecoder().decode(buffer);
       
       // Intelligent browser detection
       const needsBrowser = this.detectBrowserRequirement(html, url);
@@ -390,6 +469,95 @@ export class BNCASmartScraper {
     }
   }
   
+  /**
+   * PDF parsing method - handles PDF documents
+   */
+  async tryPDFParse(url, config) {
+    this.stats.pdf.attempts++;
+    
+    try {
+      // Download PDF with timeout
+      const controller = new AbortController();
+      const timeoutId = setTimeout(() => controller.abort(), config.timeout);
+      
+      const response = await fetch(url, {
+        headers: {
+          'User-Agent': config.userAgent,
+          'Accept': 'application/pdf,*/*'
+        },
+        signal: controller.signal
+      });
+      
+      clearTimeout(timeoutId);
+      
+      if (!response.ok) {
+        return {
+          success: false,
+          error: `HTTP ${response.status}: ${response.statusText}`
+        };
+      }
+      
+      // Check content type (be lenient - accept various content types)
+      const contentType = response.headers.get('content-type') || '';
+      const acceptableTypes = ['pdf', 'octet-stream', 'binary', 'download'];
+      const isAcceptableType = acceptableTypes.some(type => contentType.includes(type));
+      
+      if (!isAcceptableType && !url.toLowerCase().includes('.pdf')) {
+        return {
+          success: false,
+          error: `Not a PDF document: ${contentType}`
+        };
+      }
+      
+      // Get PDF buffer
+      const arrayBuffer = await response.arrayBuffer();
+      const buffer = Buffer.from(arrayBuffer);
+      
+      // Check size limit (20MB)
+      if (buffer.length > 20 * 1024 * 1024) {
+        return {
+          success: false,
+          error: 'PDF too large (max 20MB)'
+        };
+      }
+      
+      // Parse PDF
+      const pdfData = await pdfParse(buffer);
+      
+      // Extract structured content
+      const content = {
+        title: pdfData.info?.Title || 'Untitled PDF',
+        author: pdfData.info?.Author || '',
+        subject: pdfData.info?.Subject || '',
+        keywords: pdfData.info?.Keywords || '',
+        creator: pdfData.info?.Creator || '',
+        producer: pdfData.info?.Producer || '',
+        creationDate: pdfData.info?.CreationDate || '',
+        modificationDate: pdfData.info?.ModificationDate || '',
+        pages: pdfData.numpages || 0,
+        text: pdfData.text || '',
+        metadata: pdfData.metadata || null,
+        url: url
+      };
+      
+      this.stats.pdf.successes++;
+      
+      return {
+        success: true,
+        content: JSON.stringify(content, null, 2),
+        size: buffer.length,
+        contentType: 'application/pdf',
+        pages: content.pages
+      };
+      
+    } catch (error) {
+      return {
+        success: false,
+        error: `PDF parsing error: ${error.message}`
+      };
+    }
+  }
+  
   /**
    * Intelligent detection of browser requirement
    */
@@ -633,7 +801,9 @@ export class BNCASmartScraper {
         lightpanda: this.stats.lightpanda.attempts > 0 ? 
           (this.stats.lightpanda.successes / this.stats.lightpanda.attempts * 100).toFixed(1) + '%' : '0%',
         puppeteer: this.stats.puppeteer.attempts > 0 ? 
-          (this.stats.puppeteer.successes / this.stats.puppeteer.attempts * 100).toFixed(1) + '%' : '0%'
+          (this.stats.puppeteer.successes / this.stats.puppeteer.attempts * 100).toFixed(1) + '%' : '0%',
+        pdf: this.stats.pdf.attempts > 0 ? 
+          (this.stats.pdf.successes / this.stats.pdf.attempts * 100).toFixed(1) + '%' : '0%'
       }
     };
   }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@monostate/node-scraper",
-  "version": "1.2.0",
-  "description": "Intelligent web scraping with multi-level fallback system - 11.35x faster than Firecrawl",
+  "version": "1.3.0",
+  "description": "Intelligent web scraping with PDF support and multi-level fallback system - 11.35x faster than Firecrawl",
   "type": "module",
   "main": "index.js",
   "types": "index.d.ts",
@@ -19,6 +19,9 @@
     "scripts/",
     "bin/"
   ],
+  "scripts": {
+    "postinstall": "node scripts/install-lightpanda.js"
+  },
   "keywords": [
     "web-scraping",
     "crawling",
@@ -37,7 +40,8 @@
   "author": "BNCA Team",
   "license": "MIT",
   "dependencies": {
-    "node-fetch": "^3.3.2"
+    "node-fetch": "^3.3.2",
+    "pdf-parse": "^1.1.1"
   },
   "peerDependencies": {
     "puppeteer": ">=20.0.0"
@@ -65,8 +69,5 @@
   },
   "publishConfig": {
     "access": "public"
-  },
-  "scripts": {
-    "postinstall": "node scripts/install-lightpanda.js"
   }
 }

package/LICENSE

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2025 BNCA Team
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.