@memvid/maw 1.0.0

package/README.md

@@ -0,0 +1,188 @@
+<div align="center">
+
+<img src="s_maw.svg" alt="maw" width="400">
+
+**Crawl any site. Search it forever.**
+
+[![npm](https://img.shields.io/npm/v/@memvid/maw)](https://www.npmjs.com/package/@memvid/maw)
+[![downloads](https://img.shields.io/npm/dm/@memvid/maw)](https://www.npmjs.com/package/@memvid/maw)
+[![license](https://img.shields.io/npm/l/@memvid/maw)](LICENSE)
+
+[Install](#install) · [Commands](#commands) · [Examples](#examples) · [FAQ](#faq)
+
+</div>
+
+---
+
+Feed the maw. It never forgets.
+
+```bash
+npx @memvid/maw https://stripe.com/docs
+```
+
+That's it. The entire Stripe docs are now in a 40MB file you can search and ask questions to. Offline. Forever.
+
+## Why?
+
+Because you shouldn't need to keep 47 browser tabs open or bookmark links you'll never read again. Crawl once, query forever.
+
+```bash
+# later, when you actually need it
+maw ask stripe.mv2 "how do webhooks work?"
+```
+
+## Install
+
+```bash
+npm i -g @memvid/maw
+```
+
+Or just use `npx @memvid/maw` without installing.
+
+## What it does
+
+```
+maw https://react.dev           → react.mv2 (312 pages, 18s)
+maw https://docs.python.org     → python.mv2 (2,847 pages, 4min)
+maw .                           → repo.mv2 (your local git repo)
+maw https://news.ycombinator.com/item?id=12345  → just that page
+```
+
+Smart defaults:
+- **Single page URL?** Fetches just that page
+- **Domain root?** Crawls the whole site
+- **Local path?** Reads your git repo
+- **Protected site?** Auto-switches to stealth browser
+
+## Commands
+
+### Crawl
+
+```bash
+maw <url>                           # → maw.mv2
+maw <url> -o docs.mv2               # custom output
+maw <url> docs.mv2                  # same thing (appends if exists)
+maw <url> --depth 5 --max-pages 500 # go deeper
+```
+
+### Query
+
+```bash
+maw find docs.mv2 "authentication"  # full-text search
+maw ask docs.mv2 "how does X work?" # AI answer (needs OPENAI_API_KEY)
+maw list docs.mv2                   # see what's inside
+```
+
+### Preview
+
+```bash
+maw preview stripe.com              # shows sitemap, estimated page count
+```
+
+### Export
+
+```bash
+maw export docs.mv2 -f markdown -o docs.md
+maw export docs.mv2 -f json -o docs.json
+```
+
+## Embeddings
+
+Want semantic search? Add `--embed`:
+
+```bash
+maw https://docs.whatever.com --embed openai
+```
+
+Uses OpenAI embeddings for semantic search. Costs ~$0.01 per 1000 pages. Without it, you get BM25 keyword search (still good, just different).
+
+## How it works
+
+Most sites work with a simple fetch. When that fails (Cloudflare, JS-heavy SPAs), maw falls back to a real browser. When *that* fails (aggressive anti-bot), it uses stealth mode.
+
+```
+fetch (fast) → playwright (slower) → rebrowser (stealth)
+     ↓              ↓                      ↓
+   works?        blocked?              blocked?
+     ↓              ↓                      ↓
+    done          retry                  done
+```
+
+90% of sites never need the browser. The 10% that do, just work.
+
+## Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `-o, --output <file>` | Output file | `maw.mv2` |
+| `-d, --depth <n>` | Crawl depth | `2` |
+| `-m, --max-pages <n>` | Max pages to crawl | `150` |
+| `-c, --concurrency <n>` | Parallel requests | `10` |
+| `-r, --rate-limit <n>` | Requests per second | `10` |
+| `--include <regex>` | Only crawl matching URLs | - |
+| `--exclude <regex>` | Skip matching URLs | - |
+| `--browser` | Force browser mode | - |
+| `--stealth` | Force stealth mode | - |
+| `--embed [model]` | Enable embeddings | - |
+| `--no-robots` | Ignore robots.txt | - |
+| `--no-sitemap` | Skip sitemap discovery | - |
+
+## Examples
+
+```bash
+# documentation sites
+maw https://react.dev
+maw https://docs.python.org
+maw https://stripe.com/docs
+
+# news/blogs
+maw https://paulgraham.com/articles.html
+maw "https://news.ycombinator.com/item?id=40000000"
+
+# your own repos
+maw . -o my-project.mv2
+maw https://github.com/user/repo
+
+# combine multiple sources
+maw https://react.dev https://nextjs.org -o frontend.mv2
+
+# deep crawl with embeddings
+maw https://kubernetes.io/docs --depth 4 --max-pages 1000 --embed openai
+```
+
+## Limits
+
+Files up to **50MB** work without any API key. That's roughly 500-2000 pages depending on content.
+
+For bigger crawls, get a key at [memvid.com](https://memvid.com).
+
+## FAQ
+
+**Is this legal?**
+
+Respects robots.txt by default. Use `--no-robots` at your own discretion.
+
+**Why .mv2?**
+
+It's a [memvid](https://memvid.com) file — single-file database with full-text search, embeddings, and temporal queries baked in. Think SQLite but for documents.
+
+**Can I use it programmatically?**
+
+```javascript
+import { crawl, query } from 'maw'
+
+await crawl('https://example.com', { output: 'site.mv2' })
+const results = await query('site.mv2', 'search term')
+```
+
+**What about rate limiting?**
+
+Default is 10 req/sec with automatic backoff. Most sites won't notice you. If you're hitting APIs, consider `--rate-limit 2`.
+
+**Does it handle JavaScript-rendered content?**
+
+Yes. If fetch fails, it automatically tries Playwright. For heavily protected sites, use `--stealth`.
+
+---
+
+[MIT License](LICENSE) · Built on [memvid](https://memvid.com)

package/dist/bin/maw.d.ts

@@ -0,0 +1,6 @@
+#!/usr/bin/env node
+/**
+ * maw CLI - Feed the maw. It never forgets.
+ */
+export {};
+//# sourceMappingURL=maw.d.ts.map

package/dist/bin/maw.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"maw.d.ts","sourceRoot":"","sources":["../../bin/maw.ts"],"names":[],"mappings":";AAEA;;GAEG"}

package/dist/bin/maw.js

@@ -0,0 +1,275 @@
+#!/usr/bin/env node
+/**
+ * maw CLI - Feed the maw. It never forgets.
+ */
+import { existsSync } from 'fs';
+import { Command } from 'commander';
+import { maw, find, ask, list, preview, exportDocs } from '../src/index.js';
+import { setLogMode } from '../src/utils/logger.js';
+import * as ui from '../src/utils/ui.js';
+const VERSION = '1.0.0';
+const program = new Command();
+program
+    .name('maw')
+    .description('Feed the maw. It never forgets.')
+    .version(VERSION);
+// Main command: maw <urls...> [file.mv2]
+program
+    .argument('[urls...]', 'URLs/repos to consume, optionally followed by target.mv2 to append')
+    .option('-o, --output <file>', 'Output .mv2 file', 'maw.mv2')
+    .option('-d, --depth <n>', 'Crawl depth (auto: 0 for pages, 2 for domains)')
+    .option('-c, --concurrency <n>', 'Concurrent requests', '10')
+    .option('-m, --max-pages <n>', 'Maximum pages to crawl (default: 150)')
+    .option('-r, --rate-limit <n>', 'Requests per second', '10')
+    .option('-t, --timeout <ms>', 'Request timeout in ms', '10000')
+    .option('--include <pattern>', 'URL pattern to include (regex)')
+    .option('--exclude <pattern>', 'URL pattern to exclude (regex)')
+    .option('--label <label>', 'Label for ingested documents', 'web')
+    .option('--memory <id>', 'Cloud memory ID to bind to (from memvid.com/dashboard)')
+    .option('--sitemap', 'Use sitemap.xml for discovery (default: true)')
+    .option('--no-sitemap', 'Disable sitemap discovery')
+    .option('--no-robots', 'Ignore robots.txt')
+    .option('--browser', 'Force browser mode (for JavaScript-heavy sites)')
+    .option('--stealth', 'Force stealth mode (bypasses anti-bot)')
+    .option('--embed [model]', 'Enable semantic embeddings (models: bge-small, openai, nvidia)')
+    .option('-q, --quiet', 'Minimal output')
+    .option('-v, --verbose', 'Verbose output')
+    .action(async (urls, options) => {
+    if (urls.length === 0) {
+        // Show banner and help
+        console.log(ui.banner());
+        program.help();
+        return;
+    }
+    setLogMode(options.quiet, options.verbose);
+    // Check if any argument is an .mv2 file (use as output target for appending)
+    // e.g., `maw https://example.com knowledge.mv2` or `maw knowledge.mv2 https://example.com`
+    const mv2Files = urls.filter((u) => u.endsWith('.mv2'));
+    const sources = urls.filter((u) => !u.endsWith('.mv2'));
+    // Determine output file: explicit -o flag > .mv2 in args > default
+    let outputFile = options.output;
+    if (mv2Files.length > 0 && options.output === 'maw.mv2') {
+        // Use the .mv2 file from args if no explicit -o was given
+        outputFile = mv2Files[0];
+        if (mv2Files.length > 1) {
+            console.error(ui.errorMessage('Only one .mv2 file can be specified as target'));
+            process.exit(1);
+        }
+    }
+    if (sources.length === 0) {
+        console.error(ui.errorMessage('No URLs or sources provided'));
+        process.exit(1);
+    }
+    // Show header - detect git repos vs URLs
+    if (!options.quiet) {
+        const isGit = sources.some((u) => u.startsWith('https://github.com/') ||
+            u.startsWith('https://gitlab.com/') ||
+            u.includes('.git') ||
+            u.startsWith('.') ||
+            u.startsWith('/'));
+        const label = isGit ? 'maw (git)' : 'maw';
+        const urlDisplay = sources.length === 1 ? sources[0] : `${sources.length} sources`;
+        console.log(ui.header(label, urlDisplay));
+        // Show if appending to existing file
+        if (existsSync(outputFile)) {
+            console.log(ui.theme.info(`  → Adding to ${outputFile}\n`));
+        }
+        // Show embedding mode if enabled
+        if (options.embed) {
+            const model = typeof options.embed === 'string' ? options.embed : 'bge-small';
+            console.log(ui.theme.info(`  Semantic embeddings enabled (${model})`));
+            console.log(ui.theme.dim('  This improves search quality but takes longer.\n'));
+        }
+    }
+    try {
+        const result = await maw(sources, {
+            output: outputFile,
+            depth: options.depth ? parseInt(options.depth, 10) : undefined, // undefined triggers auto-detect
+            concurrency: parseInt(options.concurrency, 10),
+            maxPages: options.maxPages ? parseInt(options.maxPages, 10) : undefined,
+            rateLimit: parseInt(options.rateLimit, 10),
+            timeout: parseInt(options.timeout, 10),
+            includePattern: options.include ? new RegExp(options.include) : undefined,
+            excludePattern: options.exclude ? new RegExp(options.exclude) : undefined,
+            label: options.label,
+            memoryId: options.memory,
+            useSitemap: options.sitemap,
+            respectRobots: options.robots,
+            forceEngine: options.stealth ? 'rebrowser' : options.browser ? 'playwright' : undefined,
+            enableEmbedding: !!options.embed,
+            embeddingModel: typeof options.embed === 'string' ? options.embed : 'bge-small',
+            quiet: options.quiet,
+            verbose: options.verbose,
+        });
+        // Success output
+        console.log(ui.successMessage(result.output, result.size, result.pages, result.duration));
+        // Show dedup stats if any skipped
+        const dedupStats = result.stats.dedup;
+        if (dedupStats && (dedupStats.localeSkipped > 0 || dedupStats.similarSkipped > 0)) {
+            console.log(ui.dedupStats(dedupStats));
+        }
+        // Show engine stats in verbose mode
+        if (options.verbose) {
+            console.log(ui.engineStats(result.stats));
+        }
+        // Warnings and cloud sync status
+        if (result.stoppedAtLimit) {
+            console.log(ui.limitWarning());
+        }
+        else if (result.memoryId) {
+            console.log(ui.cloudSyncMessage(result.memoryId));
+        }
+        else if (!options.quiet) {
+            console.log(ui.theme.dim('  It will never forget.'));
+        }
+        console.log();
+    }
+    catch (error) {
+        console.error(ui.errorMessage(error.message));
+        process.exit(1);
+    }
+});
+// find command: maw find <file> <query>
+program
+    .command('find <file> <query>')
+    .description('Search in an .mv2 file')
+    .option('-k, --top <n>', 'Number of results (default: 10)', '10')
+    .option('--json', 'Output as JSON')
+    .action(async (file, query, options) => {
+    try {
+        const results = await find(file, query, { k: parseInt(options.top, 10) });
+        if (options.json) {
+            console.log(JSON.stringify(results, null, 2));
+            return;
+        }
+        console.log();
+        console.log(ui.searchResults(results.hits || []));
+    }
+    catch (error) {
+        console.error(ui.errorMessage(error.message));
+        process.exit(1);
+    }
+});
+// ask command: maw ask <file> <question>
+program
+    .command('ask <file> <question>')
+    .description('Ask a question using an .mv2 file')
+    .option('--model <model>', 'LLM model to use (default: gpt-4o-mini)', 'gpt-4o-mini')
+    .option('--api-key <key>', 'API key for the model')
+    .option('-k, --context <n>', 'Number of context chunks to retrieve (auto: 15 for overview questions, 8 otherwise)')
+    .option('--json', 'Output as JSON')
+    .action(async (file, question, options) => {
+    try {
+        const result = await ask(file, question, {
+            model: options.model,
+            apiKey: options.apiKey || process.env.OPENAI_API_KEY,
+            k: options.context ? parseInt(options.context, 10) : undefined, // Let ingestor decide default
+        });
+        if (options.json) {
+            console.log(JSON.stringify(result, null, 2));
+            return;
+        }
+        console.log(ui.askResult(result.answer, result.sources));
+    }
+    catch (error) {
+        console.error(ui.errorMessage(error.message));
+        process.exit(1);
+    }
+});
+// list command: maw list <file>
+program
+    .command('list <file>')
+    .description('List documents in an .mv2 file')
+    .option('-l, --limit <n>', 'Number of documents to show (default: 20)', '20')
+    .option('--json', 'Output as JSON')
+    .action(async (file, options) => {
+    try {
+        const results = await list(file, { limit: parseInt(options.limit, 10) });
+        if (options.json) {
+            console.log(JSON.stringify(results, null, 2));
+            return;
+        }
+        const items = results.hits || results.frames || results;
+        if (Array.isArray(items) && items.length > 0) {
+            console.log(ui.listDocuments(items.map((item) => ({
+                title: item.title || item.preview?.slice(0, 60) || `Frame ${item.frame_id}`,
+                url: item.metadata?.url || item.uri,
+                preview: item.preview,
+            }))));
+        }
+        else {
+            console.log(`\n  ${ui.theme.muted('No documents found.')}\n`);
+        }
+    }
+    catch (error) {
+        console.error(ui.errorMessage(error.message));
+        process.exit(1);
+    }
+});
+// preview command: maw preview <url> (or np)
+program
+    .command('preview <url>')
+    .alias('np')
+    .description('Preview available pages on a site (sitemap discovery)')
+    .option('-l, --limit <n>', 'Number of pages to show', '20')
+    .option('--json', 'Output as JSON')
+    .action(async (url, options) => {
+    try {
+        const result = await preview(url, { limit: parseInt(options.limit, 10) });
+        if (options.json) {
+            console.log(JSON.stringify(result, null, 2));
+            return;
+        }
+        console.log(ui.previewResults(result));
+    }
+    catch (error) {
+        console.error(ui.errorMessage(error.message));
+        process.exit(1);
+    }
+});
+// export command: maw export <file>
+program
+    .command('export <file>')
+    .description('Export .mv2 file to other formats')
+    .option('-f, --format <format>', 'Output format: json, markdown, csv', 'json')
+    .option('--out <file>', 'Output file (default: stdout)')
+    .action(async (file, options) => {
+    try {
+        // Get full content for all documents
+        const docs = await exportDocs(file, { limit: 10000 });
+        let output;
+        switch (options.format) {
+            case 'markdown':
+                output = docs.map((doc) => {
+                    return `# ${doc.title}\n\n${doc.content}\n\n---\n`;
+                }).join('\n');
+                break;
+            case 'csv':
+                const headers = ['title', 'uri'];
+                const rows = docs.map((doc) => {
+                    return [
+                        `"${(doc.title || '').replace(/"/g, '""')}"`,
+                        `"${(doc.uri || '').replace(/"/g, '""')}"`,
+                    ].join(',');
+                });
+                output = [headers.join(','), ...rows].join('\n');
+                break;
+            default:
+                output = JSON.stringify(docs, null, 2);
+        }
+        if (options.out) {
+            const { writeFileSync } = await import('fs');
+            writeFileSync(options.out, output);
+            console.log(ui.theme.success(`\n  Exported ${docs.length} documents to ${options.out}\n`));
+        }
+        else {
+            console.log(output);
+        }
+    }
+    catch (error) {
+        console.error(ui.errorMessage(error.message));
+        process.exit(1);
+    }
+});
+program.parse();
+//# sourceMappingURL=maw.js.map

package/dist/bin/maw.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"maw.js","sourceRoot":"","sources":["../../bin/maw.ts"],"names":[],"mappings":";AAEA;;GAEG;AAEH,OAAO,EAAE,UAAU,EAAE,MAAM,IAAI,CAAC;AAChC,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,GAAG,EAAE,IAAI,EAAE,GAAG,EAAE,IAAI,EAAE,OAAO,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AAC5E,OAAO,EAAE,UAAU,EAAE,MAAM,wBAAwB,CAAC;AACpD,OAAO,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAEzC,MAAM,OAAO,GAAG,OAAO,CAAC;AAExB,MAAM,OAAO,GAAG,IAAI,OAAO,EAAE,CAAC;AAE9B,OAAO;KACJ,IAAI,CAAC,KAAK,CAAC;KACX,WAAW,CAAC,iCAAiC,CAAC;KAC9C,OAAO,CAAC,OAAO,CAAC,CAAC;AAEpB,yCAAyC;AACzC,OAAO;KACJ,QAAQ,CAAC,WAAW,EAAE,oEAAoE,CAAC;KAC3F,MAAM,CAAC,qBAAqB,EAAE,kBAAkB,EAAE,SAAS,CAAC;KAC5D,MAAM,CAAC,iBAAiB,EAAE,gDAAgD,CAAC;KAC3E,MAAM,CAAC,uBAAuB,EAAE,qBAAqB,EAAE,IAAI,CAAC;KAC5D,MAAM,CAAC,qBAAqB,EAAE,uCAAuC,CAAC;KACtE,MAAM,CAAC,sBAAsB,EAAE,qBAAqB,EAAE,IAAI,CAAC;KAC3D,MAAM,CAAC,oBAAoB,EAAE,uBAAuB,EAAE,OAAO,CAAC;KAC9D,MAAM,CAAC,qBAAqB,EAAE,gCAAgC,CAAC;KAC/D,MAAM,CAAC,qBAAqB,EAAE,gCAAgC,CAAC;KAC/D,MAAM,CAAC,iBAAiB,EAAE,8BAA8B,EAAE,KAAK,CAAC;KAChE,MAAM,CAAC,eAAe,EAAE,wDAAwD,CAAC;KACjF,MAAM,CAAC,WAAW,EAAE,+CAA+C,CAAC;KACpE,MAAM,CAAC,cAAc,EAAE,2BAA2B,CAAC;KACnD,MAAM,CAAC,aAAa,EAAE,mBAAmB,CAAC;KAC1C,MAAM,CAAC,WAAW,EAAE,iDAAiD,CAAC;KACtE,MAAM,CAAC,WAAW,EAAE,wCAAwC,CAAC;KAC7D,MAAM,CAAC,iBAAiB,EAAE,gEAAgE,CAAC;KAC3F,MAAM,CAAC,aAAa,EAAE,gBAAgB,CAAC;KACvC,MAAM,CAAC,eAAe,EAAE,gBAAgB,CAAC;KACzC,MAAM,CAAC,KAAK,EAAE,IAAI,EAAE,OAAO,EAAE,EAAE;IAC9B,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtB,uBAAuB;QACvB,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,CAAC;QACzB,OAAO,CAAC,IAAI,EAAE,CAAC;QACf,OAAO;IACT,CAAC;IAED,UAAU,CAAC,OAAO,CAAC,KAAK,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC;IAE3C,6EAA6E;IAC7E,2FAA2F;IAC3F,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC;IAChE,MAAM,OAAO,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC;IAEhE,mEAAmE;IACnE,IAAI,UAAU,GAAG,OAAO,CAAC,MAAM,CAAC;IAChC,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QACxD,0DAA0D;QAC1D,UAAU,GAAG,QAAQ,CAAC,CAAC,CAAC,CAAC;QACzB,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACxB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,+CAA+C,CAAC,CAAC,CAAC;YAChF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,CAAC;IACH,CAAC;IAED,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,6BAA6B,CAAC,CAAC,CAAC;QAC9D,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,yCAAyC;IACzC,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC;QACnB,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,CAAS,EAAE,EAAE,CACvC,CAAC,CAAC,UAAU,CAAC,qBAAqB,CAAC;YACnC,CAAC,CAAC,UAAU,CAAC,qBAAqB,CAAC;YACnC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC;YAClB,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC;YACjB,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC,CAClB,CAAC;QACF,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,KAAK,CAAC;QAC1C,MAAM,UAAU,GAAG,OAAO,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,MAAM,UAAU,CAAC;QACnF,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,MAAM,CAAC,KAAK,EAAE,UAAU,CAAC,CAAC,CAAC;QAE1C,qCAAqC;QACrC,IAAI,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;YAC3B,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,iBAAiB,UAAU,IAAI,CAAC,CAAC,CAAC;QAC9D,CAAC;QAED,iCAAiC;QACjC,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;YAClB,MAAM,KAAK,GAAG,OAAO,OAAO,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,WAAW,CAAC;YAC9E,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,kCAAkC,KAAK,GAAG,CAAC,CAAC,CAAC;YACvE,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,oDAAoD,CAAC,CAAC,CAAC;QAClF,CAAC;IACH,CAAC;IAED,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,GAAG,CAAC,OAAO,EAAE;YAChC,MAAM,EAAE,UAAU;YAClB,KAAK,EAAE,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,EAAG,iCAAiC;YAClG,WAAW,EAAE,QAAQ,CAAC,OAAO,CAAC,WAAW,EAAE,EAAE,CAAC;YAC9C,QAAQ,EAAE,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS;YACvE,SAAS,EAAE,QAAQ,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC;YAC1C,OAAO,EAAE,QAAQ,CAAC,OAAO,CAAC,OAAO,EAAE,EAAE,CAAC;YACtC,cAAc,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,SAAS;YACzE,cAAc,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,SAAS;YACzE,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,QAAQ,EAAE,OAAO,CAAC,MAAM;YACxB,UAAU,EAAE,OAAO,CAAC,OAAO;YAC3B,aAAa,EAAE,OAAO,CAAC,MAAM;YAC7B,WAAW,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,SAAS;YACvF,eAAe,EAAE,CAAC,CAAC,OAAO,CAAC,KAAK;YAChC,cAAc,EAAE,OAAO,OAAO,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,WAAW;YAC/E,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,OAAO,EAAE,OAAO,CAAC,OAAO;SACzB,CAAC,CAAC;QAEH,iBAAiB;QACjB,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,cAAc,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC;QAE1F,kCAAkC;QAClC,MAAM,UAAU,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC;QACtC,IAAI,UAAU,IAAI,CAAC,UAAU,CAAC,aAAa,GAAG,CAAC,IAAI,UAAU,CAAC,cAAc,GAAG,CAAC,CAAC,EAAE,CAAC;YAClF,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,UAAU,CAAC,UAAU,CAAC,CAAC,CAAC;QACzC,CAAC;QAED,oCAAoC;QACpC,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC;YACpB,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,WAAW,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;QAC5C,CAAC;QAED,iCAAiC;QACjC,IAAI,MAAM,CAAC,cAAc,EAAE,CAAC;YAC1B,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,YAAY,EAAE,CAAC,CAAC;QACjC,CAAC;aAAM,IAAI,MAAM,CAAC,QAAQ,EAAE,CAAC;YAC3B,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,gBAAgB,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC;QACpD,CAAC;aAAM,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC;YAC1B,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC,CAAC;QACvD,CAAC;QAED,OAAO,CAAC,GAAG,EAAE,CAAC;IAChB,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,wCAAwC;AACxC,OAAO;KACJ,OAAO,CAAC,qBAAqB,CAAC;KAC9B,WAAW,CAAC,wBAAwB,CAAC;KACrC,MAAM,CAAC,eAAe,EAAE,iCAAiC,EAAE,IAAI,CAAC;KAChE,MAAM,CAAC,QAAQ,EAAE,gBAAgB,CAAC;KAClC,MAAM,CAAC,KAAK,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,EAAE;IACrC,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,EAAE,QAAQ,CAAC,OAAO,CAAC,GAAG,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;QAE1E,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;YACjB,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;YAC9C,OAAO;QACT,CAAC;QAED,OAAO,CAAC,GAAG,EAAE,CAAC;QACd,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,aAAa,CAAC,OAAO,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC,CAAC;IACpD,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,yCAAyC;AACzC,OAAO;KACJ,OAAO,CAAC,uBAAuB,CAAC;KAChC,WAAW,CAAC,mCAAmC,CAAC;KAChD,MAAM,CAAC,iBAAiB,EAAE,yCAAyC,EAAE,aAAa,CAAC;KACnF,MAAM,CAAC,iBAAiB,EAAE,uBAAuB,CAAC;KAClD,MAAM,CAAC,mBAAmB,EAAE,qFAAqF,CAAC;KAClH,MAAM,CAAC,QAAQ,EAAE,gBAAgB,CAAC;KAClC,MAAM,CAAC,KAAK,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,EAAE;IACxC,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,QAAQ,EAAE;YACvC,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,OAAO,CAAC,GAAG,CAAC,cAAc;YACpD,CAAC,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,EAAE,8BAA8B;SAC/F,CAAC,CAAC;QAEH,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;YACjB,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;YAC7C,OAAO;QACT,CAAC;QAED,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,SAAS,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC;IAC3D,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,gCAAgC;AAChC,OAAO;KACJ,OAAO,CAAC,aAAa,CAAC;KACtB,WAAW,CAAC,gCAAgC,CAAC;KAC7C,MAAM,CAAC,iBAAiB,EAAE,2CAA2C,EAAE,IAAI,CAAC;KAC5E,MAAM,CAAC,QAAQ,EAAE,gBAAgB,CAAC;KAClC,MAAM,CAAC,KAAK,EAAE,IAAI,EAAE,OAAO,EAAE,EAAE;IAC9B,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,EAAE,KAAK,EAAE,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;QAEzE,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;YACjB,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;YAC9C,OAAO;QACT,CAAC;QAED,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,MAAM,IAAI,OAAO,CAAC;QACxD,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC7C,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,aAAa,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAS,EAAE,EAAE,CAAC,CAAC;gBACrD,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,SAAS,IAAI,CAAC,QAAQ,EAAE;gBAC3E,GAAG,EAAE,IAAI,CAAC,QAAQ,EAAE,GAAG,IAAI,IAAI,CAAC,GAAG;gBACnC,OAAO,EAAE,IAAI,CAAC,OAAO;aACtB,CAAC,CAAC,CAAC,CAAC,CAAC;QACR,CAAC;aAAM,CAAC;YACN,OAAO,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC,KAAK,CAAC,KAAK,CAAC,qBAAqB,CAAC,IAAI,CAAC,CAAC;QAChE,CAAC;IACH,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,6CAA6C;AAC7C,OAAO;KACJ,OAAO,CAAC,eAAe,CAAC;KACxB,KAAK,CAAC,IAAI,CAAC;KACX,WAAW,CAAC,uDAAuD,CAAC;KACpE,MAAM,CAAC,iBAAiB,EAAE,yBAAyB,EAAE,IAAI,CAAC;KAC1D,MAAM,CAAC,QAAQ,EAAE,gBAAgB,CAAC;KAClC,MAAM,CAAC,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE;IAC7B,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,GAAG,EAAE,EAAE,KAAK,EAAE,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;QAE1E,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;YACjB,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;YAC7C,OAAO;QACT,CAAC;QAED,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,cAAc,CAAC,MAAM,CAAC,CAAC,CAAC;IACzC,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,oCAAoC;AACpC,OAAO;KACJ,OAAO,CAAC,eAAe,CAAC;KACxB,WAAW,CAAC,mCAAmC,CAAC;KAChD,MAAM,CAAC,uBAAuB,EAAE,oCAAoC,EAAE,MAAM,CAAC;KAC7E,MAAM,CAAC,cAAc,EAAE,+BAA+B,CAAC;KACvD,MAAM,CAAC,KAAK,EAAE,IAAI,EAAE,OAAO,EAAE,EAAE;IAC9B,IAAI,CAAC;QACH,qCAAqC;QACrC,MAAM,IAAI,GAAG,MAAM,UAAU,CAAC,IAAI,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC;QAEtD,IAAI,MAAc,CAAC;QAEnB,QAAQ,OAAO,CAAC,MAAM,EAAE,CAAC;YACvB,KAAK,UAAU;gBACb,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE;oBACxB,OAAO,KAAK,GAAG,CAAC,KAAK,OAAO,GAAG,CAAC,OAAO,WAAW,CAAC;gBACrD,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gBACd,MAAM;YAER,KAAK,KAAK;gBACR,MAAM,OAAO,GAAG,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;gBACjC,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE;oBAC5B,OAAO;wBACL,IAAI,CAAC,GAAG,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,IAAI,EAAE,IAAI,CAAC,GAAG;wBAC5C,IAAI,CAAC,GAAG,CAAC,GAAG,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,IAAI,EAAE,IAAI,CAAC,GAAG;qBAC3C,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;gBACd,CAAC,CAAC,CAAC;gBACH,MAAM,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,GAAG,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gBACjD,MAAM;YAER;gBACE,MAAM,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;QAC3C,CAAC;QAED,IAAI,OAAO,CAAC,GAAG,EAAE,CAAC;YAChB,MAAM,EAAE,aAAa,EAAE,GAAG,MAAM,MAAM,CAAC,IAAI,CAAC,CAAC;YAC7C,aAAa,CAAC,OAAO,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YACnC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,gBAAgB,IAAI,CAAC,MAAM,iBAAiB,OAAO,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC;QAC7F,CAAC;aAAM,CAAC;YACN,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QACtB,CAAC;IACH,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,CAAC;AAEL,OAAO,CAAC,KAAK,EAAE,CAAC"}

package/dist/src/crawler/index.d.ts

@@ -0,0 +1,71 @@
+/**
+ * Web crawler with rate limiting
+ */
+import { type ExtractResult } from '../extractor/index.js';
+export interface CrawlOptions {
+    depth: number;
+    concurrency: number;
+    maxPages: number;
+    rateLimit: number;
+    timeout: number;
+    respectRobots: boolean;
+    useSitemap: boolean;
+    includePattern?: RegExp;
+    excludePattern?: RegExp;
+    forceEngine?: 'fetch' | 'playwright' | 'rebrowser';
+}
+export interface CrawlResult {
+    url: string;
+    finalUrl: string;
+    extracted: ExtractResult;
+    depth: number;
+    engine: string;
+}
+export declare class Crawler {
+    private options;
+    private engine;
+    private extractor;
+    private robots;
+    private sitemap;
+    private dedup;
+    private visited;
+    private queue;
+    private baseHosts;
+    private results;
+    private pending;
+    constructor(options?: Partial<CrawlOptions>);
+    /**
+     * Crawl URLs and yield results as they complete
+     */
+    crawl(startUrls: string[]): AsyncGenerator<CrawlResult>;
+    private parseSitemaps;
+    private addToQueue;
+    private processUrl;
+    private shouldCrawl;
+    /**
+     * Get engine statistics
+     */
+    getStats(): {
+        visited: number;
+        queued: number;
+        pending: number;
+        dedup: {
+            uniquePaths: number;
+            uniqueContent: number;
+            localeSkipped: number;
+            similarSkipped: number;
+            total: number;
+        };
+        fetch: number;
+        playwright: number;
+        rebrowser: number;
+        blocked: number;
+    };
+    /**
+     * Close all resources
+     */
+    close(): Promise<void>;
+}
+export { RobotsParser } from './robots.js';
+export { SitemapParser } from './sitemap.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/src/crawler/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/crawler/index.ts"],"names":[],"mappings":"AAAA;;GAEG;AAIH,OAAO,EAAa,KAAK,aAAa,EAAE,MAAM,uBAAuB,CAAC;AAOtE,MAAM,WAAW,YAAY;IAC3B,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,aAAa,EAAE,OAAO,CAAC;IACvB,UAAU,EAAE,OAAO,CAAC;IACpB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,WAAW,CAAC,EAAE,OAAO,GAAG,YAAY,GAAG,WAAW,CAAC;CACpD;AAED,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,MAAM,CAAC;IACZ,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,EAAE,aAAa,CAAC;IACzB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;CAChB;AAcD,qBAAa,OAAO;IAClB,OAAO,CAAC,OAAO,CAAe;IAC9B,OAAO,CAAC,MAAM,CAAkB;IAChC,OAAO,CAAC,SAAS,CAAY;IAC7B,OAAO,CAAC,MAAM,CAAe;IAC7B,OAAO,CAAC,OAAO,CAAgB;IAC/B,OAAO,CAAC,KAAK,CAAe;IAC5B,OAAO,CAAC,OAAO,CAA0B;IACzC,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,SAAS,CAA0B;IAC3C,OAAO,CAAC,OAAO,CAAqB;IACpC,OAAO,CAAC,OAAO,CAA0D;gBAE7D,OAAO,GAAE,OAAO,CAAC,YAAY,CAAM;IAgB/C;;OAEG;IACI,KAAK,CAAC,SAAS,EAAE,MAAM,EAAE,GAAG,cAAc,CAAC,WAAW,CAAC;YAkEhD,aAAa;IA2B3B,OAAO,CAAC,UAAU;YAaJ,UAAU;IA+CxB,OAAO,CAAC,WAAW;IAyCnB;;OAEG;IACH,QAAQ;;;;;;;;;;;;;;;;IAUR;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;CAI7B;AAED,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC"}