clean-web-scraper 3.2.3 → 3.2.5

package/README.md

@@ -14,7 +14,6 @@ A powerful Node.js web scraper that extracts clean, readable content from websit
 - 🤖 AI-friendly output formats (JSONL, CSV, clean text)
 - 📊 Rich metadata extraction
 - 📁 Combine results from multiple scrapers into a unified dataset
-- 🎯 Turn any website into an AI training dataset
 
 ## 🛠️ Prerequisites
 
@@ -53,17 +52,10 @@ const scraper = new WebScraper({
   jsonlOutputPath: './example.com/train.jsonl', // Optional: Custom JSONL output path
   textOutputPath: "./example.com/texts",        // Optional: Custom text output path
   csvOutputPath: "./example.com/train.csv",     // Optional: Custom CSV output path
-  maxDepth: 3,                                  // Optional: Maximum depth for recursive crawling
-  includeMetadata: false,                       // Optional: Include metadata in output files
-  metadataFields: ['title', 'description']      // Optional: Specify metadata fields to include
 });
 await scraper.start();
 ```
 
-```bash
-node example-usage.js
-```
-
 ## 💻 Advanced Usage: Multi-Site Scraping
 
 ```js
@@ -72,13 +64,19 @@ const WebScraper = require('clean-web-scraper');
 // Scrape documentation website
 const docsScraper = new WebScraper({
   baseURL: 'https://docs.example.com',
-  scrapResultPath: './datasets/docs'
+  scrapResultPath: './datasets/docs',
+  maxDepth: 3,                               // Optional: Maximum depth for recursive crawling
+  includeMetadata: true,                     // Optional: Include metadata in output files
+  metadataFields: ['title', 'description']   // Optional: Specify metadata fields to include
 });
 
 // Scrape blog website
 const blogScraper = new WebScraper({
   baseURL: 'https://blog.example.com',
-  scrapResultPath: './datasets/blog'
+  scrapResultPath: './datasets/blog',
+  maxDepth: 3,                               // Optional: Maximum depth for recursive crawling
+  includeMetadata: true,                     // Optional: Include metadata in output files
+  metadataFields: ['title', 'description']   // Optional: Specify metadata fields to include
 });
 
 // Start scraping both sites
@@ -89,16 +87,18 @@ await blogScraper.start();
 await WebScraper.combineResults('./combined', [docsScraper, blogScraper]);
 ```
 
+```bash
+node example-usage.js
+```
+
 ## 📤 Output
 
 Your AI-ready content is saved in a clean, structured format:
 
-- 📁 Base folder: ./folderPath/example.com/
+- 📁 Base folder: `./folderPath/example.com/`
 - 📑 Files preserve original URL paths
-- 📝 Pure text format, perfect for LLM training and fine-tuning
-- 🤖 No HTML, no mess - just clean, structured text ready for AI consumption
-- 📊 JSONL output for ML training
-- 📈 CSV output with clean text content
+- 🤖 No HTML, no noise - just clean, structured text (`.txt` files)
+- 📊 `JSONL` and `CSV` outputs, ready for AI consumption and model training
 
 ```bash
 example.com/
@@ -138,10 +138,13 @@ combined/
 
 ### 📝 Text Files (*.txt)
 
+```text
 The actual article content starts here. This is the clean, processed text of the article that was extracted from the webpage
+```
 
 ### 📑 Text Files with Metadata (texts_with_metadata/*.txt)
 
+```text
 title: My Awesome Page
 description: This is a great article about coding
 author: John Doe
@@ -151,6 +154,7 @@ dateScraped: 2024-01-20T10:30:00Z
 \-\-\-
 
 The actual article content starts here. This is the clean, processed text of the article that was extracted from the webpage.
+```
 
 ### 📊 JSONL Files (train.jsonl)
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clean-web-scraper",
-  "version": "3.2.3",
+  "version": "3.2.5",
   "main": "main.js",
   "scripts": {
     "start": "node main.js",