cockroach-crawler 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ajnas
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,159 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/AjnasNB/cockroach-crawler/main/assets/logo.png" alt="Cockroach Crawler" width="620">
+</p>
+
+# Cockroach Crawler
+
+Cockroach Crawler is an open-source public-web crawler for agent workflows. It turns crawlable public pages into clean JSON or JSONL with titles, metadata, readable text, markdown, links, status codes, content types, and timestamps.
+
+It is designed for documentation indexing, RAG ingestion, QA crawling, content inventory, public research, and agent toolchains that need a local crawler with a simple CLI and JavaScript API.
+
+## What it does
+
+- Crawls public HTTP/HTTPS pages from one or more seed URLs.
+- Respects `robots.txt` by default.
+- Discovers sitemaps from `robots.txt` and `/sitemap.xml`.
+- Extracts readable text and markdown.
+- Outputs JSON or JSONL for agent pipelines.
+- Supports concurrency, per-origin delay, max pages, max depth, include/exclude filters, URL files, and same-origin controls.
+- Uses a clear user agent and supports contact details.
+- Avoids likely private account/admin/cart/login URLs by default.
+- Ships zero hosted dependencies: no account, API key, browser session, or LLM call is required.
+
+## What it does not do
+
+Cockroach Crawler is not a stealth scraper. It does not include bypass tooling for login walls, paywalls, CAPTCHA, anti-bot systems, authorization boundaries, or robots.txt. If a site owner requires permission, get permission and crawl within their terms.
+
+## When to use it
+
+Use Cockroach Crawler when an agent or data pipeline needs a repeatable public-web fetcher that returns compact, structured page records. It is intentionally smaller than full crawler frameworks and intentionally simpler than browser agents.
+
+| Use case | Cockroach Crawler fit |
+| --- | --- |
+| Crawl public docs, blogs, help centers, or marketing pages | Strong |
+| Export crawl results as JSONL or markdown for RAG | Strong |
+| Respect robots.txt and crawl-delay style politeness controls | Strong |
+| Crawl JavaScript-only apps that require clicks, login, or a live browser | Use Browser Use, Playwright, Puppeteer, Crawlee, or another browser stack |
+| Bypass paywalls, CAPTCHA, anti-bot, auth, or owner restrictions | Not supported |
+
+## Comparison
+
+Cockroach Crawler is not a universal replacement for Browser Use, Crawlee, Scrapy, Firecrawl, Crawl4AI, or hosted browser tools. It is better for one narrow job: low-friction public HTML crawling into agent-friendly JSON/JSONL/Markdown from Node.js.
+
+| Tool | Best at | Tradeoff compared with Cockroach Crawler |
+| --- | --- | --- |
+| Browser Use | LLM-driven browser control and interactive workflows | Heavier stack; better for clicking/forms, not a lightweight batch crawler |
+| Crawlee | Production-grade crawling with browser and queue primitives | More powerful, larger API surface, more setup |
+| Scrapy | Mature Python crawling framework | Excellent framework, but Python-first and less direct for Node agent pipelines |
+| Firecrawl | Hosted/API-first page extraction and crawling | Great API product; self-hosting/licensing/deployment is heavier |
+| Crawl4AI | LLM-oriented Python crawling and markdown extraction | Strong AI extraction focus; Python-first |
+| Cockroach Crawler | Local public-web crawl to JSONL/Markdown for agents | Smaller scope; no JS browser rendering, no stealth, no hosted extraction |
+
+The goal is not to beat every crawler at every job. The goal is to be the simplest useful crawler an agent can call locally when it needs public pages converted into clean records.
+
+## Install
+
+```bash
+npm install -g cockroach-crawler
+```
+
+Or run without global install:
+
+```bash
+npx cockroach-crawler https://example.com --max-pages 20 --jsonl
+```
+
+## CLI examples
+
+Crawl one public site:
+
+```bash
+cockroach-crawl https://example.com --max-pages 50 --jsonl --output crawl.jsonl
+```
+
+Use sitemaps and include only docs URLs:
+
+```bash
+cockroach-crawl https://example.com --sitemaps --include "/docs/" --max-pages 200 --output docs.json
+```
+
+Read many seed URLs from a file:
+
+```bash
+cockroach-crawl --url-file urls.txt --max-pages 100 --jsonl
+```
+
+Add a contact-aware user agent:
+
+```bash
+cockroach-crawl https://example.com --contact "mailto:you@example.com"
+```
+
+## CLI options
+
+- `--url-file <file>`: read seed URLs from a text file, one URL per line.
+- `--max-pages <n>`: maximum pages to return. Default: `50`.
+- `--max-depth <n>`: maximum link depth from seeds. Default: `2`.
+- `--concurrency <n>`: concurrent workers. Default: `4`.
+- `--delay <ms>`: minimum delay per origin. Default: `250`.
+- `--timeout <ms>`: request timeout. Default: `15000`.
+- `--sitemaps`: discover URLs from robots.txt sitemaps and `/sitemap.xml`.
+- `--all-origins`: allow crawling across discovered origins.
+- `--include <regex>`: only crawl URLs matching regex. Can be repeated.
+- `--exclude <regex>`: skip URLs matching regex. Can be repeated.
+- `--allow-non-public`: allow likely login/account/admin/cart URLs.
+- `--jsonl`: output JSON Lines instead of a JSON array.
+- `--output <file>`: write output to a file.
+- `--user-agent <ua>`: custom user agent.
+- `--contact <email/url>`: add contact detail to the default user agent.
+
+## Library API
+
+```js
+import { crawl } from "cockroach-crawler";
+
+const pages = await crawl({
+  seeds: ["https://example.com"],
+  maxPages: 25,
+  maxDepth: 2,
+  concurrency: 4,
+  includeSitemaps: true,
+  include: ["/docs/"],
+  exclude: ["/login", "/account"],
+  onPage(page) {
+    console.log(page.url, page.title);
+  }
+});
+
+console.log(pages[0].markdown);
+```
+
+## Output shape
+
+```json
+{
+  "url": "https://example.com/",
+  "canonical": "https://example.com/",
+  "title": "Example",
+  "description": "Example description",
+  "h1": "Example",
+  "text": "Readable text...",
+  "markdown": "# Example\n\nReadable markdown...",
+  "links": ["https://example.com/about"],
+  "fetchedAt": "2026-06-27T00:00:00.000Z",
+  "status": 200,
+  "contentType": "text/html; charset=utf-8"
+}
+```
+
+## Development
+
+```bash
+npm install
+npm test
+npm run bench
+```
+
+## License
+
+MIT

package/bin/cockroach-crawl.js

@@ -0,0 +1,157 @@
+#!/usr/bin/env node
+import { createWriteStream } from "node:fs";
+import { readFile } from "node:fs/promises";
+import { crawl } from "../src/index.js";
+
+function usage() {
+  console.log(`
+Cockroach Crawler
+
+Usage:
+  cockroach-crawl <url> [more urls...] [options]
+
+Options:
+  --url-file <file>       Read seed URLs from a text file, one URL per line
+  --max-pages <n>         Maximum pages to return. Default: 50
+  --max-depth <n>         Maximum link depth from seeds. Default: 2
+  --concurrency <n>       Concurrent workers. Default: 4
+  --delay <ms>            Minimum delay per origin. Default: 250
+  --timeout <ms>          Request timeout. Default: 15000
+  --sitemaps              Discover URLs from robots.txt sitemaps and /sitemap.xml
+  --all-origins           Allow crawling across origins discovered from links
+  --include <regex>       Only crawl URLs matching regex. Can be repeated
+  --exclude <regex>       Skip URLs matching regex. Can be repeated
+  --allow-non-public      Allow likely login/account/admin/cart URLs
+  --jsonl                 Output JSON Lines instead of a JSON array
+  --output <file>         Write output to a file
+  --user-agent <ua>       Custom user agent
+  --contact <email/url>   Add contact detail to the default user agent
+  --help                  Show this help
+
+This crawler respects robots.txt by default and does not bypass access controls.
+`);
+}
+
+async function urlsFromFile(file) {
+  const text = await readFile(file, "utf8");
+  return text
+    .split(/\r?\n/)
+    .map((line) => line.trim())
+    .filter((line) => line && !line.startsWith("#"));
+}
+
+async function readArgs(argv) {
+  const urls = [];
+  const include = [];
+  const exclude = [];
+  const options = {
+    maxPages: 50,
+    maxDepth: 2,
+    concurrency: 4,
+    delayMs: 250,
+    timeoutMs: 15_000,
+    includeSitemaps: false,
+    sameOrigin: true,
+    publicOnly: true,
+    jsonl: false,
+    output: null,
+    userAgent: undefined
+  };
+
+  for (let i = 0; i < argv.length; i += 1) {
+    const arg = argv[i];
+    if (arg === "--help" || arg === "-h") {
+      options.help = true;
+    } else if (arg === "--url-file") {
+      urls.push(...(await urlsFromFile(argv[++i])));
+    } else if (arg === "--max-pages") {
+      options.maxPages = Number(argv[++i]);
+    } else if (arg === "--max-depth") {
+      options.maxDepth = Number(argv[++i]);
+    } else if (arg === "--concurrency") {
+      options.concurrency = Number(argv[++i]);
+    } else if (arg === "--delay") {
+      options.delayMs = Number(argv[++i]);
+    } else if (arg === "--timeout") {
+      options.timeoutMs = Number(argv[++i]);
+    } else if (arg === "--sitemaps") {
+      options.includeSitemaps = true;
+    } else if (arg === "--all-origins") {
+      options.sameOrigin = false;
+    } else if (arg === "--include") {
+      include.push(argv[++i]);
+    } else if (arg === "--exclude") {
+      exclude.push(argv[++i]);
+    } else if (arg === "--allow-non-public") {
+      options.publicOnly = false;
+    } else if (arg === "--jsonl") {
+      options.jsonl = true;
+    } else if (arg === "--output" || arg === "-o") {
+      options.output = argv[++i];
+    } else if (arg === "--user-agent") {
+      options.userAgent = argv[++i];
+    } else if (arg === "--contact") {
+      const contact = argv[++i];
+      options.userAgent = `CockroachCrawler/0.1 (+${contact})`;
+    } else if (arg.startsWith("-")) {
+      throw new Error(`Unknown option: ${arg}`);
+    } else {
+      urls.push(arg);
+    }
+  }
+
+  options.include = include;
+  options.exclude = exclude;
+  return { urls, options };
+}
+
+function writeOutput(pages, options) {
+  const body = options.jsonl
+    ? pages.map((page) => JSON.stringify(page)).join("\n") + "\n"
+    : JSON.stringify({ pages, stats: pages.stats || null }, null, 2) + "\n";
+
+  if (!options.output) {
+    process.stdout.write(body);
+    return;
+  }
+  const stream = createWriteStream(options.output, { encoding: "utf8" });
+  stream.end(body);
+}
+
+async function main() {
+  const { urls, options } = await readArgs(process.argv.slice(2));
+  if (options.help) {
+    usage();
+    return;
+  }
+  if (!urls.length) {
+    usage();
+    process.exitCode = 1;
+    return;
+  }
+
+  const pages = await crawl({
+    seeds: urls,
+    maxPages: options.maxPages,
+    maxDepth: options.maxDepth,
+    concurrency: options.concurrency,
+    delayMs: options.delayMs,
+    timeoutMs: options.timeoutMs,
+    includeSitemaps: options.includeSitemaps,
+    sameOrigin: options.sameOrigin,
+    publicOnly: options.publicOnly,
+    include: options.include,
+    exclude: options.exclude,
+    userAgent: options.userAgent,
+    onError: (failure) => {
+      process.stderr.write(`crawl warning: ${failure.url}: ${failure.error}\n`);
+    }
+  });
+
+  writeOutput(pages, options);
+}
+
+main().catch((error) => {
+  process.stderr.write(`${error.stack || error.message || String(error)}\n`);
+  process.exitCode = 1;
+});

package/package.json

@@ -0,0 +1,62 @@
+{
+  "name": "cockroach-crawler",
+  "version": "0.1.0",
+  "description": "Public-web crawler for agent workflows with robots.txt, sitemaps, depth limits, filters, markdown extraction, and JSONL output.",
+  "license": "MIT",
+  "author": {
+    "name": "Ajnas",
+    "email": "ajnasnb@gmail.com"
+  },
+  "type": "module",
+  "bin": {
+    "cockroach-crawl": "bin/cockroach-crawl.js",
+    "cockroach-crawler": "bin/cockroach-crawl.js"
+  },
+  "main": "src/index.js",
+  "exports": {
+    ".": "./src/index.js"
+  },
+  "files": [
+    "bin/",
+    "src/",
+    "assets/",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "node --test",
+    "bench": "node bench/local-benchmark.mjs",
+    "crawl": "node bin/cockroach-crawl.js"
+  },
+  "keywords": [
+    "crawler",
+    "cockroach-crawler",
+    "agent",
+    "web-crawler",
+    "scraper",
+    "robots.txt",
+    "sitemap",
+    "markdown",
+    "jsonl",
+    "research"
+  ],
+  "dependencies": {
+    "cheerio": "^1.0.0",
+    "robots-parser": "^3.0.1",
+    "turndown": "^7.2.0"
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/AjnasNB/cockroach-crawler.git"
+  },
+  "bugs": {
+    "url": "https://github.com/AjnasNB/cockroach-crawler/issues"
+  },
+  "homepage": "https://github.com/AjnasNB/cockroach-crawler#readme",
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/src/index.js

@@ -0,0 +1,386 @@
+import * as cheerio from "cheerio";
+import robotsParser from "robots-parser";
+import TurndownService from "turndown";
+
+const DEFAULT_USER_AGENT = "CockroachCrawler/0.1 (+https://github.com/AjnasNB/cockroach-crawler)";
+const DEFAULT_MAX_BYTES = 3 * 1024 * 1024;
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+
+function toUrl(value, base) {
+  try {
+    return new URL(value, base).toString();
+  } catch {
+    return null;
+  }
+}
+
+function normalizeUrl(value) {
+  const url = new URL(value);
+  url.hash = "";
+  if ((url.protocol === "http:" && url.port === "80") || (url.protocol === "https:" && url.port === "443")) {
+    url.port = "";
+  }
+  return url.toString();
+}
+
+function isHttpUrl(value) {
+  try {
+    const url = new URL(value);
+    return url.protocol === "http:" || url.protocol === "https:";
+  } catch {
+    return false;
+  }
+}
+
+function sameOrigin(a, b) {
+  return new URL(a).origin === new URL(b).origin;
+}
+
+function compilePatterns(values) {
+  return (values || []).map((value) => new RegExp(value));
+}
+
+function matchesAny(value, patterns) {
+  return patterns.some((pattern) => pattern.test(value));
+}
+
+async function fetchText(url, options) {
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), options.timeoutMs);
+  try {
+    const response = await fetch(url, {
+      headers: {
+        "user-agent": options.userAgent,
+        accept: options.accept || "text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.5"
+      },
+      redirect: "follow",
+      signal: controller.signal
+    });
+
+    const contentType = response.headers.get("content-type") || "";
+    const length = Number(response.headers.get("content-length") || 0);
+    if (length > options.maxBytes) {
+      throw new Error(`Response too large: ${length} bytes`);
+    }
+
+    const reader = response.body?.getReader();
+    if (!reader) {
+      return { response, text: await response.text(), contentType };
+    }
+
+    const chunks = [];
+    let received = 0;
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      received += value.byteLength;
+      if (received > options.maxBytes) {
+        throw new Error(`Response exceeded maxBytes: ${options.maxBytes}`);
+      }
+      chunks.push(value);
+    }
+    const buffer = Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)));
+    return { response, text: buffer.toString("utf8"), contentType };
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+function parseRobotsSitemaps(robotsText) {
+  return robotsText
+    .split(/\r?\n/)
+    .map((line) => line.trim())
+    .filter((line) => /^sitemap:/i.test(line))
+    .map((line) => line.replace(/^sitemap:\s*/i, "").trim())
+    .filter(Boolean);
+}
+
+async function loadRobots(origin, options) {
+  const robotsUrl = new URL("/robots.txt", origin).toString();
+  try {
+    const { response, text } = await fetchText(robotsUrl, {
+      ...options,
+      accept: "text/plain,*/*;q=0.5",
+      maxBytes: Math.min(options.maxBytes, 512 * 1024)
+    });
+    if (!response.ok) {
+      return {
+        parser: robotsParser(robotsUrl, ""),
+        sitemaps: []
+      };
+    }
+    return {
+      parser: robotsParser(robotsUrl, text),
+      sitemaps: parseRobotsSitemaps(text)
+    };
+  } catch {
+    return {
+      parser: robotsParser(robotsUrl, ""),
+      sitemaps: []
+    };
+  }
+}
+
+async function discoverSitemapUrls(sitemapUrl, options) {
+  const discovered = [];
+  try {
+    const { response, text, contentType } = await fetchText(sitemapUrl, {
+      ...options,
+      accept: "application/xml,text/xml,*/*;q=0.5"
+    });
+    if (!response.ok) return discovered;
+    if (!contentType.includes("xml") && !text.trim().startsWith("<")) return discovered;
+
+    const $ = cheerio.load(text, { xmlMode: true });
+    $("url > loc").each((_, el) => {
+      const value = $(el).text().trim();
+      if (isHttpUrl(value)) discovered.push(normalizeUrl(value));
+    });
+    $("sitemap > loc").each((_, el) => {
+      const value = $(el).text().trim();
+      if (isHttpUrl(value)) discovered.push(normalizeUrl(value));
+    });
+  } catch {
+    return discovered;
+  }
+  return discovered;
+}
+
+function extractLinks($, baseUrl) {
+  const links = [];
+  $("a[href]").each((_, el) => {
+    const href = $(el).attr("href");
+    const resolved = toUrl(href, baseUrl);
+    if (resolved && isHttpUrl(resolved)) {
+      links.push(normalizeUrl(resolved));
+    }
+  });
+  return [...new Set(links)];
+}
+
+function cleanForExtraction($) {
+  $("script, style, noscript, template, svg, canvas, iframe").remove();
+  $("[hidden], [aria-hidden='true']").remove();
+}
+
+export function extractPage(html, url) {
+  const $ = cheerio.load(html);
+  cleanForExtraction($);
+
+  const title = ($("title").first().text() || $("h1").first().text() || "").trim().replace(/\s+/g, " ");
+  const description = ($("meta[name='description']").attr("content") || "").trim();
+  const h1 = $("h1").first().text().trim().replace(/\s+/g, " ");
+  const canonical = toUrl($("link[rel='canonical']").attr("href") || url, url);
+  const links = extractLinks($, url);
+
+  const main = $("main").first();
+  const contentRoot = main.length ? main : $("body");
+  const htmlFragment = contentRoot.html() || "";
+  const text = contentRoot.text().replace(/\s+/g, " ").trim();
+
+  const turndown = new TurndownService({
+    headingStyle: "atx",
+    codeBlockStyle: "fenced",
+    bulletListMarker: "-"
+  });
+  const markdown = turndown.turndown(htmlFragment).replace(/\n{3,}/g, "\n\n").trim();
+
+  return {
+    url,
+    canonical,
+    title,
+    description,
+    h1,
+    text,
+    markdown,
+    links,
+    fetchedAt: new Date().toISOString()
+  };
+}
+
+class CrawlQueue {
+  constructor() {
+    this.items = [];
+    this.offset = 0;
+  }
+
+  push(url) {
+    const next = typeof url === "string" ? { url, depth: 0 } : url;
+    this.items.push(next);
+  }
+
+  shift() {
+    if (this.offset >= this.items.length) return null;
+      const value = this.items[this.offset];
+    this.offset += 1;
+    if (this.offset > 1000 && this.offset * 2 > this.items.length) {
+      this.items = this.items.slice(this.offset);
+      this.offset = 0;
+    }
+    return value;
+  }
+
+  get length() {
+    return this.items.length - this.offset;
+  }
+}
+
+export async function crawl(input = {}) {
+  const seeds = (input.seeds || input.urls || [])
+    .map((seed) => (isHttpUrl(seed) ? normalizeUrl(seed) : null))
+    .filter(Boolean);
+
+  if (!seeds.length) {
+    throw new Error("At least one http(s) seed URL is required.");
+  }
+
+  const options = {
+    maxPages: input.maxPages ?? 50,
+    concurrency: input.concurrency ?? 4,
+    sameOrigin: input.sameOrigin ?? true,
+    maxDepth: input.maxDepth ?? 2,
+    include: compilePatterns(input.include),
+    exclude: compilePatterns(input.exclude),
+    publicOnly: input.publicOnly ?? true,
+    includeSitemaps: input.includeSitemaps ?? false,
+    obeyRobots: input.obeyRobots ?? true,
+    userAgent: input.userAgent || DEFAULT_USER_AGENT,
+    delayMs: input.delayMs ?? 250,
+    timeoutMs: input.timeoutMs ?? 15_000,
+    maxBytes: input.maxBytes ?? DEFAULT_MAX_BYTES,
+    onPage: input.onPage || null,
+    onError: input.onError || null
+  };
+
+  const queue = new CrawlQueue();
+  const seen = new Set();
+  const enqueued = new Set();
+  const results = [];
+  const stats = {
+    fetched: 0,
+    skippedRobots: 0,
+    skippedFiltered: 0,
+    skippedNonPublic: 0,
+    errors: 0
+  };
+  const robotsByOrigin = new Map();
+  const lastFetchByOrigin = new Map();
+  const seedOrigins = new Set(seeds.map((seed) => new URL(seed).origin));
+
+  const isAllowedByFilters = (url) => {
+    if (options.include.length && !matchesAny(url, options.include)) return false;
+    if (options.exclude.length && matchesAny(url, options.exclude)) return false;
+    return true;
+  };
+
+  const isPublicUrl = (url) => {
+    if (!options.publicOnly) return true;
+    const parsed = new URL(url);
+    const banned = /(?:login|signin|sign-in|signup|account|admin|dashboard|checkout|cart|billing|private|wp-admin)/i;
+    return !banned.test(parsed.pathname);
+  };
+
+  const enqueue = (url, depth = 0) => {
+    if (!url || enqueued.has(url) || seen.has(url)) return;
+    if (options.sameOrigin && ![...seedOrigins].some((origin) => sameOrigin(url, origin))) return;
+    if (!isAllowedByFilters(url)) {
+      stats.skippedFiltered += 1;
+      return;
+    }
+    if (!isPublicUrl(url)) {
+      stats.skippedNonPublic += 1;
+      return;
+    }
+    if (depth > options.maxDepth) return;
+    enqueued.add(url);
+    queue.push({ url, depth });
+  };
+
+  for (const seed of seeds) enqueue(seed, 0);
+
+  async function getRobots(url) {
+    const origin = new URL(url).origin;
+    if (!robotsByOrigin.has(origin)) {
+      robotsByOrigin.set(origin, await loadRobots(origin, options));
+    }
+    return robotsByOrigin.get(origin);
+  }
+
+  if (options.includeSitemaps) {
+    for (const seed of seeds) {
+      const robots = await getRobots(seed);
+      const sitemapUrls = robots.sitemaps.length
+        ? robots.sitemaps
+        : [new URL("/sitemap.xml", new URL(seed).origin).toString()];
+      for (const sitemapUrl of sitemapUrls) {
+        const urls = await discoverSitemapUrls(sitemapUrl, options);
+        for (const url of urls) enqueue(url, 0);
+      }
+    }
+  }
+
+  async function waitForOrigin(url) {
+    const origin = new URL(url).origin;
+    const last = lastFetchByOrigin.get(origin) || 0;
+    const waitMs = Math.max(0, last + options.delayMs - Date.now());
+    if (waitMs) await sleep(waitMs);
+    lastFetchByOrigin.set(origin, Date.now());
+  }
+
+  async function worker() {
+    while (results.length < options.maxPages) {
+      const item = queue.shift();
+      if (!item) return;
+      const { url, depth } = item;
+      if (seen.has(url)) continue;
+      seen.add(url);
+
+      try {
+        if (options.obeyRobots) {
+          const robots = await getRobots(url);
+          if (robots.parser && !robots.parser.isAllowed(url, options.userAgent)) {
+            stats.skippedRobots += 1;
+            continue;
+          }
+        }
+
+        await waitForOrigin(url);
+        const { response, text, contentType } = await fetchText(url, options);
+        if (!response.ok) {
+          throw new Error(`HTTP ${response.status}`);
+        }
+        stats.fetched += 1;
+        if (!/html|xml|text\//i.test(contentType)) {
+          continue;
+        }
+
+        const page = extractPage(text, response.url || url);
+        page.status = response.status;
+        page.contentType = contentType;
+        results.push(page);
+        if (options.onPage) await options.onPage(page);
+
+        for (const link of page.links) {
+          if (results.length + queue.length >= options.maxPages * 6) break;
+          enqueue(link, depth + 1);
+        }
+      } catch (error) {
+        stats.errors += 1;
+        const failure = { url, error: error.message || String(error) };
+        if (options.onError) await options.onError(failure);
+      }
+    }
+  }
+
+  const workerCount = Math.max(1, Math.min(options.concurrency, options.maxPages));
+  await Promise.all(Array.from({ length: workerCount }, () => worker()));
+  const sliced = results.slice(0, options.maxPages);
+  Object.defineProperty(sliced, "stats", {
+    value: stats,
+    enumerable: false
+  });
+  return sliced;
+}
+
+export { discoverSitemapUrls, normalizeUrl };