enterprise-ai-recursive-web-scraper 1.0.0 → 1.0.2

package/README.md

@@ -38,30 +38,61 @@
 
 ## 🚀 Quick Start
 
+To install the package, run:
+
 ```bash
-npm i enterprise-ai-recursive-web-scraper
+npm install enterprise-ai-recursive-web-scraper
 ```
 
-```typescript
-import { WebScraper } from "enterprise-ai-recursive-web-scraper";
+### Using the CLI
+
+The `enterprise-ai-recursive-web-scraper` package includes a command-line interface (CLI) that you can use to perform web scraping tasks directly from the terminal.
+
+#### Installation
 
-async function main() {
-    const scraper = new WebScraper({
-        outputDir: "scraping_output",
-        verbose: true
-    });
+Ensure that the package is installed globally to use the CLI:
 
-    const results = await scraper.scrapeWebsite("https://example.com");
-    console.log(results);
-}
+```bash
+npm install -g enterprise-ai-recursive-web-scraper
+```
+
+#### Running the CLI
 
-main().catch(console.error);
+Once installed, you can use the `web-scraper` command to start scraping. Here’s a basic example of how to use it:
+
+```bash
+web-scraper --api-key YOUR_API_KEY --url https://example.com --output ./output
 ```
 
+#### CLI Options
+
+- `-k, --api-key <key>`: **(Required)** Your Google Gemini API key.
+- `-u, --url <url>`: **(Required)** The URL of the website you want to scrape.
+- `-o, --output <directory>`: The directory where the scraped data will be saved. Default is `scraping_output`.
+- `-d, --depth <number>`: Maximum crawl depth. Default is `3`.
+- `-c, --concurrency <number>`: Concurrent scraping limit. Default is `5`.
+- `-t, --timeout <seconds>`: Request timeout in seconds. Default is `30`.
+- `-f, --format <type>`: Output format (`json`, `csv`, `markdown`). Default is `json`.
+- `--screenshot`: Capture screenshots of pages.
+- `--no-headless`: Run the browser in non-headless mode.
+- `--proxy <url>`: Use a proxy server.
+- `-v, --verbose`: Enable verbose logging.
+- `--config <path>`: Path to a configuration file.
+
+#### Example Command
+
+```bash
+web-scraper --api-key YOUR_API_KEY --url https://example.com --output ./output --depth 5 --concurrency 10 --format csv --verbose
+```
+
+This command will scrape the specified URL with a maximum depth of 5, using 10 concurrent requests, and save the output in CSV format in the `./output` directory with verbose logging enabled.
+
 ## 🔧 Advanced Usage
 
 ### Structured Data Extraction
 
+To extract structured data using a JSON schema, you can use the `JsonExtractionStrategy`:
+
 ```typescript
 import { WebScraper, JsonExtractionStrategy } from "enterprise-ai-recursive-web-scraper";
 
@@ -81,6 +112,8 @@ const scraper = new WebScraper({
 
 ### Custom Browser Session
 
+You can customize the browser session with specific configurations:
+
 ```typescript
 import { WebScraper } from "enterprise-ai-recursive-web-scraper";
 
@@ -119,4 +152,4 @@ const scraper = new WebScraper({
 
 MIT © [Mike Odnis](https://github.com/WomB0ComB0)
 
-> 💙 Built with [`create-typescript-app`](https://github.com/JoshuaKGoldberg/create-typescript-app)
+> 💙 Built with [`create-typescript-app`](https://github.com/JoshuaKGoldberg/create-typescript-app)

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "enterprise-ai-recursive-web-scraper",
-	"version": "1.0.0",
+	"version": "1.0.2",
 	"description": "AI powered, recursive, web-scraper utilizing Gemini models, Puppeteer, and Playwright",
 	"repository": {
 		"type": "git",
@@ -13,14 +13,18 @@
 	},
 	"type": "module",
 	"main": "lib/cli.cjs",
-	"files": [
-		"lib/**/*",
-		"package.json",
-		"LICENSE.md",
-		"README.md"
-	],
+  "files": [
+    "lib/*.js",
+    "lib/*.cjs",
+    "lib/*.d.ts",
+    "lib/*.d.cts",
+    "lib/*.map",
+    "package.json",
+    "LICENSE.md",
+    "README.md"
+  ],
 	"scripts": {
-		"build": "bun run build:pre && tsup src/cli.ts src/index.ts --format esm,cjs --dts --outDir lib",
+		"build": "bun run build:pre && tsup",
 		"build:pre": "bunx puppeteer browsers install chrome",
 		"prepublishOnly": "bun run build",
 		"prepack": "bun run build"
@@ -62,5 +66,17 @@
 	],
 	"bin": {
 		"web-scraper": "./lib/cli.cjs"
+	},
+	"exports": {
+		".": {
+			"require": "./lib/index.cjs",
+			"import": "./lib/index.js",
+			"types": "./lib/index.d.ts"
+		},
+		"./cli": {
+			"require": "./lib/cli.cjs",
+			"import": "./lib/cli.js",
+			"types": "./lib/cli.d.ts"
+		}
 	}
 }