messi-crawler 1.0.0

package/README.md

@@ -0,0 +1,201 @@
+# Web Crawler
+
+An interactive, CLI-driven web crawler built with Node.js, TypeScript, and a PostgreSQL backend. Specifically engineered to systematically extract and compile programming-related documentation from seed URLs, the crawler offers flexible output strategies — structured database records or compiled PDF eBooks — with politeness constraints, domain filtering, and depth controls.
+
+## System Requirements
+
+- **Node.js** (version 18 or greater)
+- **PostgreSQL** (local or hosted instance like Supabase)
+
+## Features
+
+### Interactive CLI Wizard
+Run `npm run crawl` to launch an interactive setup wizard that guides you through:
+- **Output mode selection** — Database (structured records) or PDF (compiled eBook)
+- **Seed URL source** — Use URLs from `seeds.txt`, config defaults, or enter custom URLs
+- **Performance tuning** — Configure depth, crawl delay, worker count, and page limits
+
+### Flexible Output Strategies
+- **Database mode** — Stores extracted content (URL, title, description, headings, text) as structured records in PostgreSQL with link graph tracking
+- **PDF mode** — Compiles all crawled pages into a formatted PDF eBook with cover page, table of contents, and styled chapters. PDFs are auto-versioned (`documentation.pdf`, `documentation2.pdf`, etc.) to avoid overwrites
+
+### Safety & Politeness
+- **Minimum crawl delay** — Enforces a 500ms floor on `CRAWL_DELAY_MS` to prevent accidental aggressive request rates
+- **Robots.txt compliance** — Respects disallow directives per domain
+- **Domain filtering** — Restricts crawling to seed domains only; child links outside allowed domains are ignored
+- **Concurrency limits** — Configurable worker pool to control concurrent requests
+
+### Session-Scoped Crawling
+- Each run filters the DB queue to only process URLs matching the current session's allowed domains
+- Stale pending URLs from previous runs are automatically cleared at startup
+- Session page counter tracks progress independently of cumulative DB totals
+
+## Configuration
+
+The crawler is configured interactively via the CLI wizard, but you can also pre-set defaults using environment variables or `src/config.ts`.
+
+### Environment Variables (`.env`)
+Create a `.env` file in the root directory with:
+
+```env
+DATABASE_URL=postgresql://user:password@host:5432/dbname
+MAX_DEPTH=3
+CRAWL_DELAY_MS=1000
+WORKER_COUNT=5
+MAX_PAGES=1000
+OUTPUT_MODE=database
+```
+
+### Seed URLs (`seeds.txt`)
+Add target URLs to `seeds.txt`, one per line:
+
+```
+# Programming documentation sources
+https://react.dev
+https://developer.mozilla.org
+https://www.typescriptlang.org/
+```
+
+Lines starting with `#` are ignored. The wizard defaults to using `seeds.txt` if present.
+
+### Configuration Fields
+
+| Field              | Description                                                                 | Default       |
+|--------------------|-----------------------------------------------------------------------------|---------------|
+| `MAX_DEPTH`        | Maximum link hops from seed URLs (0 = seeds only)                          | `3`           |
+| `CRAWL_DELAY_MS`   | Politeness delay per domain (min 500ms enforced)                           | `1000`        |
+| `WORKER_COUNT`     | Number of concurrent workers                                               | `5`           |
+| `MAX_PAGES`        | Page limit per session (0 = unlimited)                                     | `1000`        |
+| `OUTPUT_MODE`      | Output destination: `database` or `pdf`                                    | `database`    |
+| `DATABASE_URL`     | PostgreSQL connection string                                               | (required)    |
+
+## Database Setup
+
+Before running the crawler, initialize the database schema.
+
+### 1. Create the database
+```sql
+CREATE DATABASE web_crawler;
+```
+
+### 2. Apply the schema
+```bash
+psql -U postgres -d web_crawler -f src/db/schema.sql
+```
+
+Or if using a hosted service like Supabase, run the contents of `src/db/schema.sql` in the SQL editor.
+
+### 3. Configure the connection
+Update `DATABASE_URL` in `.env` with your connection string:
+```
+DATABASE_URL=postgresql://postgres:password@db.example.supabase.co:5432/postgres
+```
+
+## Usage
+
+### Start the Crawler
+```bash
+npm run crawl
+```
+This launches the interactive wizard, then starts crawling with your chosen settings.
+
+### Pre-configure (optional)
+```bash
+npm run config
+```
+Runs the standalone configuration wizard, writes settings to `.env` and patches `src/config.ts`.
+
+### Clear the Database
+```bash
+npm run db:clear
+```
+Truncates all tables and resets the crawler state.
+
+### Run Tests
+```bash
+npm test
+```
+
+## Output
+
+### Database Mode
+Crawled data is stored across four tables:
+- **`urls`** — All discovered URLs with status tracking (`PENDING`, `FETCHING`, `DONE`, `FAILED`)
+- **`crawled_pages`** — Extracted content (title, description, headings, text)
+- **`links`** — Link graph edges (from → to relationships)
+- **`domain_stats`** — Per-domain aggregate statistics
+
+Query examples:
+```sql
+-- Get all successfully crawled pages
+SELECT url, title FROM crawled_pages 
+JOIN urls ON crawled_pages.url_id = urls.id;
+
+-- View domain statistics
+SELECT * FROM domain_stats;
+```
+
+### PDF Mode
+Each crawl generates a compiled PDF in `output/`:
+- `documentation.pdf` (first run)
+- `documentation2.pdf` (second run)
+- etc.
+
+PDFs include:
+- Styled cover page with generation timestamp
+- One chapter per crawled page with title, URL, description, headings outline, and body text
+- Footer with page numbers
+
+## Architecture
+
+### Core Components
+
+| Module                     | Purpose                                                              |
+|----------------------------|----------------------------------------------------------------------|
+| `src/index.ts`             | Main entry point; runs CLI wizard and orchestrates crawl session   |
+| `src/setup.ts`             | Standalone configuration wizard (for `npm run config`)             |
+| `src/frontier/scheduler.ts`| Round-robin scheduler with politeness delays and concurrency limits |
+| `src/worker/worker.ts`     | Processes individual URLs: download, extract, persist              |
+| `src/worker/downloader.ts` | HTTP client with redirect handling and timeouts                    |
+| `src/worker/extractor.ts`  | Cheerio-based HTML parser for metadata and content                 |
+| `src/output/`              | Strategy pattern for output destinations (DB or PDF)               |
+| `src/db/queries.ts`        | Database queries for URL state management and link tracking        |
+| `src/frontier/robots.ts`   | Robots.txt parser with per-domain caching                          |
+
+### Design Patterns
+
+- **Strategy Pattern** — Output destinations (`DatabaseStrategy`, `PdfStrategy`) implement a common `OutputStrategy` interface, allowing runtime switching
+- **Round-robin scheduling** — Domains are processed in rotation with per-domain cooldowns to enforce politeness delays
+- **Optimistic locking** — PostgreSQL `FOR UPDATE SKIP LOCKED` prevents workers from claiming the same URL
+
+## Safety & Best Practices
+
+- **Politeness floor** — `CRAWL_DELAY_MS` cannot be set below 500ms; attempts to do so are flagged and auto-corrected
+- **Domain scoping** — Only URLs matching `ALLOWED_DOMAINS` (derived from seeds) are crawled
+- **Robots.txt compliance** — URLs disallowed by `robots.txt` are marked failed without download
+- **Graceful shutdown** — On reaching `MAX_PAGES`, the scheduler waits for in-flight workers to complete before closing the DB pool
+- **Crash recovery** — On startup, any URLs stuck in `FETCHING` state are reset to `PENDING`
+
+## Troubleshooting
+
+### Database connection errors
+- Verify `DATABASE_URL` is correct and the database exists
+- Check that the host/port is reachable (port 5432 is commonly blocked on public networks; use Supabase's connection pooler on port 6543 if needed)
+- Ensure the password is URL-encoded if it contains special characters
+
+### Crawler picks up wrong URLs
+- Run `npm run db:clear` to wipe stale data from previous runs
+- Verify `seeds.txt` contains only the URLs you want
+- Check that `ALLOWED_DOMAINS` in the wizard output matches your intent
+
+### Crawl delay too aggressive
+- The minimum is 500ms. If you set a lower value, it's automatically raised with a warning.
+- Increase `CRAWL_DELAY_MS` if target servers rate-limit or block requests
+
+## License
+
+ISC
+
+## Repository
+
+[github.com/lightning4747/Web-crawler](https://github.com/lightning4747/Web-crawler)

package/dist/cli/renderer.js

@@ -0,0 +1,71 @@
+/**
+ * Terminal rendering utilities — colours, symbols, layout helpers.
+ * Keeps all ANSI logic in one place so the wizard stays readable.
+ */
+export const c = {
+    reset: "\x1b[0m",
+    bold: "\x1b[1m",
+    dim: "\x1b[2m",
+    italic: "\x1b[3m",
+    // foreground
+    white: "\x1b[97m",
+    gray: "\x1b[90m",
+    cyan: "\x1b[96m",
+    green: "\x1b[92m",
+    yellow: "\x1b[93m",
+    red: "\x1b[91m",
+    blue: "\x1b[94m",
+    magenta: "\x1b[95m",
+    orange: "\x1b[38;5;208m",
+};
+export const sym = {
+    dot: "·",
+    bullet: "•",
+    arrow: "›",
+    check: "✓",
+    cross: "✗",
+    warn: "⚠",
+    info: "ℹ",
+    sparkle: "◆",
+    bar: "│",
+    corner: "╰",
+    tee: "├",
+    horiz: "─",
+};
+/** Wraps text in an ANSI style sequence. */
+export function style(text, ...styles) {
+    return styles.join("") + text + c.reset;
+}
+/** Prints a blank line. */
+export function blank() { console.log(); }
+/** dim separator line */
+export function divider(width = 52) {
+    console.log(style(sym.horiz.repeat(width), c.dim, c.gray));
+}
+/** A styled section label, e.g.  ◆ Seeds */
+export function section(label) {
+    console.log(style(`${sym.sparkle} ${label}`, c.bold, c.cyan));
+}
+/** A key/value summary row, e.g.   › depth    3 */
+export function row(key, value, valueColor = c.white) {
+    const pad = 16;
+    const k = style(key.padEnd(pad), c.gray);
+    const v = style(value, valueColor);
+    console.log(`  ${style(sym.arrow, c.dim, c.gray)} ${k}${v}`);
+}
+/** Success line */
+export function ok(msg) {
+    console.log(`  ${style(sym.check, c.green)} ${style(msg, c.white)}`);
+}
+/** Warning line */
+export function warn(msg) {
+    console.log(`  ${style(sym.warn, c.yellow)} ${style(msg, c.yellow)}`);
+}
+/** Error line */
+export function err(msg) {
+    console.log(`  ${style(sym.cross, c.red)} ${style(msg, c.red)}`);
+}
+/** Info line */
+export function info(msg) {
+    console.log(`  ${style(sym.info, c.blue)} ${style(msg, c.gray)}`);
+}

package/dist/config.js

@@ -0,0 +1,18 @@
+import dotenv from "dotenv";
+dotenv.config();
+export const config = {
+    MAX_DEPTH: parseInt(process.env.MAX_DEPTH || "3", 10),
+    CRAWL_DELAY_MS: parseInt(process.env.CRAWL_DELAY_MS || "1000", 10),
+    WORKER_COUNT: parseInt(process.env.WORKER_COUNT || "10", 10),
+    REQUEST_TIMEOUT_MS: parseInt(process.env.REQUEST_TIMEOUT_MS || "10000", 10),
+    MAX_REDIRECTS: parseInt(process.env.MAX_REDIRECTS || "5", 10),
+    MAX_PAGES: parseInt(process.env.MAX_PAGES || "1000", 10),
+    /** Output destination: "database" | "pdf" */
+    OUTPUT_MODE: (process.env.OUTPUT_MODE || "database"),
+    SEED_URLS: [
+        "https://www.akc.org/dog-breeds/",
+    ],
+    ALLOWED_DOMAINS: [
+        "www.akc.org",
+    ],
+};

package/dist/db/clear.js

@@ -0,0 +1,16 @@
+import { pool } from "./client.js";
+async function clearDatabase() {
+    console.log("Clearing database...");
+    try {
+        // Truncate all crawler tables and reset the auto-increment IDs
+        await pool.query("TRUNCATE TABLE links, crawled_pages, urls, domain_stats RESTART IDENTITY CASCADE;");
+        console.log("Database cleared successfully.");
+    }
+    catch (error) {
+        console.error("Error clearing database:", error);
+    }
+    finally {
+        await pool.end();
+    }
+}
+clearDatabase();

package/dist/db/client.js

@@ -0,0 +1,20 @@
+import pg from "pg";
+import dotenv from "dotenv";
+dotenv.config();
+const { Pool } = pg;
+const connectionString = process.env.DATABASE_URL;
+export const pool = connectionString
+    ? new Pool({ connectionString })
+    : new Pool({
+        host: process.env.PGHOST || "localhost",
+        port: parseInt(process.env.PGPORT || "5432", 10),
+        user: process.env.PGUSER || "postgres",
+        password: process.env.PGPASSWORD || "",
+        database: process.env.PGDATABASE || "web_crawler",
+    });
+export async function query(text, params) {
+    return pool.query(text, params);
+}
+export async function closePool() {
+    await pool.end();
+}

package/dist/db/queries.js

@@ -0,0 +1,179 @@
+import { query, pool } from "./client.js";
+/**
+ * Claims the next PENDING URL for a given domain and marks it as FETCHING.
+ * Uses FOR UPDATE SKIP LOCKED to prevent multiple workers from claiming the same URL.
+ */
+export async function claimNextURL(domain) {
+    const res = await query(`UPDATE urls
+     SET status = 'FETCHING', fetched_at = NOW()
+     WHERE id = (
+       SELECT id FROM urls
+       WHERE status = 'PENDING' AND domain = $1
+       ORDER BY depth ASC, discovered_at ASC
+       LIMIT 1
+       FOR UPDATE SKIP LOCKED
+     )
+     RETURNING id, url, domain, status, depth`, [domain]);
+    if (res.rows.length === 0) {
+        return null;
+    }
+    return res.rows[0];
+}
+/**
+ * Atomically updates URL status to DONE and inserts the crawled page content.
+ */
+export async function markDone(urlId, content) {
+    const client = await pool.connect();
+    try {
+        await client.query("BEGIN");
+        await client.query(`INSERT INTO crawled_pages (url_id, title, description, canonical_url, headings, text_content)
+       VALUES ($1, $2, $3, $4, $5, $6)`, [
+            urlId,
+            content.title,
+            content.description,
+            content.canonicalUrl,
+            JSON.stringify(content.headings),
+            content.textContent,
+        ]);
+        await client.query(`UPDATE urls
+       SET status = 'DONE', fetched_at = NOW()
+       WHERE id = $1`, [urlId]);
+        await client.query("COMMIT");
+    }
+    catch (error) {
+        await client.query("ROLLBACK");
+        throw error;
+    }
+    finally {
+        client.release();
+    }
+}
+/**
+ * Marks a URL status as FAILED and stores the error message.
+ */
+export async function markFailed(urlId, errorMessage) {
+    await query(`UPDATE urls
+     SET status = 'FAILED', error_message = $2, fetched_at = NOW()
+     WHERE id = $1`, [urlId, errorMessage]);
+}
+/**
+ * Inserts a URL as PENDING if it doesn't already exist.
+ * Returns the ID of the URL (whether newly inserted or already existing).
+ */
+export async function insertURL(url, domain, depth) {
+    const res = await query(`WITH ins AS (
+       INSERT INTO urls (url, domain, status, depth)
+       VALUES ($1, $2, 'PENDING', $3)
+       ON CONFLICT (url) DO NOTHING
+       RETURNING id
+     )
+     SELECT id FROM ins
+     UNION ALL
+     SELECT id FROM urls WHERE url = $1
+     LIMIT 1`, [url, domain, depth]);
+    return res.rows[0].id;
+}
+/**
+ * Inserts a link relationship between two URLs.
+ */
+export async function insertLink(fromUrlId, toUrlId) {
+    await query(`INSERT INTO links (from_url_id, to_url_id)
+     VALUES ($1, $2)
+     ON CONFLICT (from_url_id, to_url_id) DO NOTHING`, [fromUrlId, toUrlId]);
+}
+/**
+ * Resets all URLs with FETCHING status back to PENDING.
+ * Used for crash recovery on startup to release stale locks.
+ */
+export async function resetStaleLocks() {
+    await query(`UPDATE urls
+     SET status = 'PENDING'
+     WHERE status = 'FETCHING'`);
+}
+/**
+ * Deletes all PENDING URLs whose domain is not in the provided allowed list.
+ * This includes child links discovered during previous crawls, ensuring a new
+ * session scoped to different seeds starts with a clean queue.
+ */
+export async function clearPendingURLs(allowedDomains) {
+    if (allowedDomains.length === 0)
+        return;
+    const result = await query(`DELETE FROM urls
+     WHERE status = 'PENDING'
+       AND domain <> ALL($1::text[])`, [allowedDomains]);
+    const deleted = result.rowCount ?? 0;
+    if (deleted > 0) {
+        console.log(`[setup] Cleared ${deleted} stale PENDING URL(s) outside allowed domains.`);
+    }
+}
+/**
+ * Retrieves aggregate statistics across all URLs.
+ */
+export async function getGlobalStats() {
+    const res = await query(`SELECT status, COUNT(*) as count
+     FROM urls
+     GROUP BY status`);
+    const stats = { pending: 0, fetching: 0, done: 0, failed: 0 };
+    for (const row of res.rows) {
+        const status = row.status.toLowerCase();
+        const count = parseInt(row.count, 10);
+        if (status === "pending")
+            stats.pending = count;
+        else if (status === "fetching")
+            stats.fetching = count;
+        else if (status === "done")
+            stats.done = count;
+        else if (status === "failed")
+            stats.failed = count;
+    }
+    return stats;
+}
+/**
+ * Recomputes and updates domain-level stats in the domain_stats table.
+ */
+export async function refreshDomainStats() {
+    await query(`
+    CREATE TABLE IF NOT EXISTS domain_stats (
+      domain TEXT PRIMARY KEY,
+      pending_count INTEGER NOT NULL DEFAULT 0,
+      fetching_count INTEGER NOT NULL DEFAULT 0,
+      done_count INTEGER NOT NULL DEFAULT 0,
+      failed_count INTEGER NOT NULL DEFAULT 0,
+      last_crawled_at TIMESTAMPTZ
+    )
+  `);
+    await query(`
+    INSERT INTO domain_stats (domain, pending_count, fetching_count, done_count, failed_count, last_crawled_at)
+    SELECT
+      domain,
+      COUNT(*) FILTER (WHERE status = 'PENDING') as pending_count,
+      COUNT(*) FILTER (WHERE status = 'FETCHING') as fetching_count,
+      COUNT(*) FILTER (WHERE status = 'DONE') as done_count,
+      COUNT(*) FILTER (WHERE status = 'FAILED') as failed_count,
+      MAX(fetched_at) as last_crawled_at
+    FROM urls
+    GROUP BY domain
+    ON CONFLICT (domain) DO UPDATE SET
+      pending_count = EXCLUDED.pending_count,
+      fetching_count = EXCLUDED.fetching_count,
+      done_count = EXCLUDED.done_count,
+      failed_count = EXCLUDED.failed_count,
+      last_crawled_at = EXCLUDED.last_crawled_at
+  `);
+}
+/**
+ * Retrieves per-domain statistics.
+ */
+export async function getDomainStats() {
+    const res = await query(`SELECT domain, pending_count, fetching_count, done_count, failed_count, last_crawled_at
+     FROM domain_stats
+     ORDER BY domain ASC`);
+    return res.rows.map((row) => ({
+        domain: row.domain,
+        pending_count: parseInt(row.pending_count, 10),
+        fetching_count: parseInt(row.fetching_count, 10),
+        done_count: parseInt(row.done_count, 10),
+        failed_count: parseInt(row.failed_count, 10),
+        last_crawled_at: row.last_crawled_at ? new Date(row.last_crawled_at) : null,
+    }));
+}

package/dist/frontier/frontier.js

@@ -0,0 +1,44 @@
+import { query } from "../db/client.js";
+import { config } from "../config.js";
+/**
+ * Retrieves the list of unique domains that have at least one pending URL,
+ * restricted to the domains allowed in the current crawl session.
+ */
+export async function getPendingDomains() {
+    const allowed = config.ALLOWED_DOMAINS;
+    // If ALLOWED_DOMAINS is populated, filter at the DB level so the scheduler
+    // never even sees pending rows from outside the current session's scope.
+    if (allowed && allowed.length > 0) {
+        const res = await query(`SELECT DISTINCT domain
+       FROM urls
+       WHERE status = 'PENDING'
+         AND domain = ANY($1::text[])`, [allowed]);
+        return res.rows.map((row) => row.domain);
+    }
+    const res = await query(`SELECT DISTINCT domain
+     FROM urls
+     WHERE status = 'PENDING'`);
+    return res.rows.map((row) => row.domain);
+}
+/**
+ * Retrieves the count of pending URLs bucketed by domain,
+ * restricted to the allowed domains in the current session.
+ */
+export async function getPendingCounts() {
+    const allowed = config.ALLOWED_DOMAINS;
+    const res = allowed && allowed.length > 0
+        ? await query(`SELECT domain, COUNT(*) as count
+         FROM urls
+         WHERE status = 'PENDING'
+           AND domain = ANY($1::text[])
+         GROUP BY domain`, [allowed])
+        : await query(`SELECT domain, COUNT(*) as count
+         FROM urls
+         WHERE status = 'PENDING'
+         GROUP BY domain`);
+    const counts = {};
+    for (const row of res.rows) {
+        counts[row.domain] = parseInt(row.count, 10);
+    }
+    return counts;
+}

package/dist/frontier/logger.js

@@ -0,0 +1,65 @@
+import { getGlobalStats, refreshDomainStats, getDomainStats } from "../db/queries.js";
+let loggerInterval = null;
+let lastDoneAndFailedCount = 0;
+let startTime = 0;
+/**
+ * Starts a background interval to log crawler progress periodically.
+ */
+export async function startProgressLogger(intervalMs = 5000) {
+    if (loggerInterval)
+        return;
+    startTime = Date.now();
+    try {
+        const initialStats = await getGlobalStats();
+        lastDoneAndFailedCount = initialStats.done + initialStats.failed;
+    }
+    catch (err) {
+        lastDoneAndFailedCount = 0;
+    }
+    loggerInterval = setInterval(async () => {
+        try {
+            // 1. Sync statistics to domain_stats table
+            await refreshDomainStats();
+            // 2. Fetch global statistics
+            const globalStats = await getGlobalStats();
+            // 3. Fetch domain-level statistics
+            const domainStats = await getDomainStats();
+            // 4. Calculate crawl rates
+            const currentCompleted = globalStats.done + globalStats.failed;
+            const completedSinceStart = currentCompleted - lastDoneAndFailedCount;
+            const elapsedMinutes = (Date.now() - startTime) / 60000;
+            const crawlRate = elapsedMinutes > 0 ? (completedSinceStart / elapsedMinutes).toFixed(1) : "0.0";
+            // 5. Build and output the formatted log messages
+            console.log(`\n=== Crawler Progress Report ===`);
+            console.log(`Speed: ${crawlRate} pages/min`);
+            console.log(`Global Status Breakdown:`);
+            console.log(`  PENDING : ${globalStats.pending}`);
+            console.log(`  FETCHING: ${globalStats.fetching}`);
+            console.log(`  DONE    : ${globalStats.done}`);
+            console.log(`  FAILED  : ${globalStats.failed}`);
+            if (domainStats.length > 0) {
+                console.log(`Domain Breakdown:`);
+                for (const ds of domainStats) {
+                    const lastCrawledStr = ds.last_crawled_at ? ds.last_crawled_at.toISOString() : "never";
+                    console.log(`  - ${ds.domain}: PENDING: ${ds.pending_count} | FETCHING: ${ds.fetching_count} | DONE: ${ds.done_count} | FAILED: ${ds.failed_count} (Last Crawled: ${lastCrawledStr})`);
+                }
+            }
+            console.log(`================================\n`);
+        }
+        catch (error) {
+            console.error("Error generating crawler progress logs:", error);
+        }
+    }, intervalMs);
+    if (loggerInterval && typeof loggerInterval.unref === "function") {
+        loggerInterval.unref();
+    }
+}
+/**
+ * Stops the progress logger interval.
+ */
+export function stopProgressLogger() {
+    if (loggerInterval) {
+        clearInterval(loggerInterval);
+        loggerInterval = null;
+    }
+}

package/dist/frontier/robots.js

@@ -0,0 +1,46 @@
+import { request } from "undici";
+// @ts-ignore
+import robotsParser from "robots-parser";
+import { getDomain } from "../normalizer.js";
+import { config } from "../config.js";
+const robotsCache = new Map();
+const fetchFailures = new Set();
+/**
+ * Checks if a URL is allowed to be crawled according to the domain's robots.txt rules.
+ * Caches robots.txt rules per domain to avoid duplicate requests.
+ */
+export async function isAllowedByRobots(urlStr) {
+    const domain = getDomain(urlStr);
+    if (!domain)
+        return false;
+    if (fetchFailures.has(domain)) {
+        return true;
+    }
+    let parser = robotsCache.get(domain);
+    if (!parser) {
+        const robotsUrl = `https://${domain}/robots.txt`;
+        try {
+            const res = await request(robotsUrl, {
+                method: "GET",
+                headersTimeout: config.REQUEST_TIMEOUT_MS,
+                bodyTimeout: config.REQUEST_TIMEOUT_MS,
+            });
+            if (res.statusCode === 200) {
+                const content = await res.body.text();
+                const parserCreator = robotsParser;
+                parser = parserCreator(robotsUrl, content);
+                robotsCache.set(domain, parser);
+            }
+            else {
+                fetchFailures.add(domain);
+                return true;
+            }
+        }
+        catch (e) {
+            // On network/request errors, default to allowed but do not permanently cache failure
+            return true;
+        }
+    }
+    const isAllowed = parser.isAllowed(urlStr, "WebCrawler");
+    return isAllowed === undefined ? true : isAllowed;
+}