@pitlane/crawler 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,25 @@
+# @pitlane/crawler
+
+## 0.1.0
+
+Initial release.
+
+- `crawl(router, options)` — walks an app by dispatching requests into its
+  router's `fetch`, yielding `{ pathname, filepath, response }` per path.
+  Follows `<a href>` and `<link rel="alternate">`, queues the assets a page
+  references, honours `rel="nofollow"` and `<meta name="robots">`, skips
+  cross-origin and non-navigable hrefs, and visits each path once. A redirect
+  yields nothing and reports through `onRedirect`, since there is no document
+  to write and the app still answers the path at runtime; under `spider` the
+  same-origin target is queued instead. Any other non-2xx response aborts the
+  crawl. `paths`, `spider`, `assets`, `concurrency`, `ignorePageNofollow`, and
+  `onRedirect` configure it.
+- `staticPaths(routes)` — the paths a Remix 3 route map can serve with no
+  params, deduplicated and sorted. `GET` and method-agnostic routes whose
+  patterns declare no variables or wildcards.
+- The API comes from [remix-run/remix#11150](https://github.com/remix-run/remix/pull/11150),
+  which was closed with the implementation kept beside the Remix docs site.
+  Two deliberate differences: `assets` is a new option, because a bundler has
+  usually emitted those files already, and the first error wins over the last
+  when several paths fail under concurrency.
+- Tested against `remix@3.0.0-beta.10`.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mark Malstrom
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,101 @@
+# @pitlane/crawler
+
+Spider a [Remix 3](https://remix.run) fetch router in memory.
+
+`crawl(router)` dispatches requests straight into `router.fetch` and yields every response it gets back, following the links each page contains. No socket, no server, no browser, no HTTP: the router is the whole transport, so an app can be walked wherever the app itself runs.
+
+That makes prerendering a `for await` loop.
+
+```ts
+import { crawl } from "@pitlane/crawler";
+import * as fs from "node:fs/promises";
+import * as path from "node:path";
+
+import router from "./app/entry.server.ts";
+
+for await (let { pathname, filepath, response } of crawl(router)) {
+    let outputPath = path.join("dist", filepath);
+    await fs.mkdir(path.dirname(outputPath), { recursive: true });
+    await fs.writeFile(outputPath, new Uint8Array(await response.arrayBuffer()));
+    console.log(`${pathname} -> ${outputPath}`);
+}
+```
+
+## Install
+
+```sh
+npm install @pitlane/crawler
+# or
+vp add @pitlane/crawler
+```
+
+Requires `remix@^3.0.0-beta.10` as a peer.
+
+Using the [`remix()` Vite plugin](https://pitlane.tools/package/dev/)? You do not need this package directly — `remix({ prerender })` runs it for you. See the [prerendering guide](https://pitlane.tools/guides/prerendering).
+
+For everything else a walk is good for — static exports, sitemaps, link checks, render smoke tests — see the [crawling guide](https://pitlane.tools/guides/crawler).
+
+## `crawl(router, options?)`
+
+Returns an async iterator of `{ pathname, filepath, response }`, one per fetched path.
+
+- `pathname` — the path that was requested.
+- `filepath` — where the response belongs on disk. HTML gets `<pathname>/index.html` so a static host serves it back for the original path; everything else keeps its own path.
+- `response` — the router's response, body unread.
+
+Results arrive in completion order, and every path is fetched at most once.
+
+A redirect yields nothing: there is no document to write, and the app still answers the path at runtime. It reports through `onRedirect` instead, and when `spider` is on the same-origin target is queued, so a crawl seeded at a `/` that points elsewhere still finds the site. Any other non-2xx response aborts the crawl with `Crawl failed: <status> <statusText> (<pathname>)`.
+
+| Option               | Type                            | Default | Purpose                                                                                                                                |
+| -------------------- | ------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------- |
+| `paths`              | `string[]`                      | `["/"]` | Where to start.                                                                                                                        |
+| `spider`             | `boolean`                       | `true`  | Follow `<a href>` and `<link rel="alternate">` to find more paths.                                                                     |
+| `assets`             | `boolean`                       | `true`  | Queue the `<link href>`, `<script src>`, and `<img src>` each page references. Turn it off when a bundler already emitted those files. |
+| `concurrency`        | `number`                        | `1`     | How many paths to fetch at once.                                                                                                       |
+| `ignorePageNofollow` | `(pathname: string) => boolean` | —       | Crawl a page's links even though the page asked robots not to follow them.                                                             |
+| `onRedirect`         | `(pathname, location) => void`  | —       | Called for a path that redirected instead of returning a document. `location` is `null` when the redirect named none.                  |
+
+The first argument is anything with a `fetch(request: Request)` method: a `createRouter()` router, a built server bundle's default export, a worker-style `{ fetch }` object.
+
+### What spidering respects
+
+Crawling stops where a crawler should stop, so a run over a real site does not wander:
+
+- `rel="nofollow"` on a link, and `<meta name="robots" content="nofollow">` (or `googlebot`) on a page.
+- Absolute and protocol-relative URLs, which belong to another origin.
+- `#fragment`, `mailto:`, `tel:`, `javascript:`, and `data:` hrefs.
+
+`ignorePageNofollow` is the escape hatch for the case where a page's `nofollow` is aimed at search engines rather than at you — a versioned docs tree that should not be indexed but does need to be built.
+
+## `staticPaths(routes)`
+
+The question that comes before a crawl: which paths can this app serve with no params?
+
+```ts
+import { staticPaths } from "@pitlane/crawler";
+import { get, route } from "remix/routes";
+
+let routes = route({
+    home: "/",
+    blog: get("/blog"),
+    post: get("/blog/:slug"),
+});
+
+staticPaths(routes); // ["/", "/blog"]
+```
+
+A route qualifies when it answers `GET` (or any method) and its pattern declares no variables or wildcards. `/blog/:slug` is left out, because its values live outside the route map. Results are deduplicated and sorted, so a build that renders them lists its output the same way every time.
+
+## Provenance
+
+The `crawl` API comes from [remix-run/remix#11150](https://github.com/remix-run/remix/pull/11150), which proposed it for `fetch-router` and was closed in favour of keeping the implementation next to the Remix docs site. This package brings it back out as something an application can install, with two changes:
+
+- `assets` is new. Upstream always queues a page's assets, which is right for a site with no bundler and wrong for one where Vite already emitted them.
+- The first error wins when several paths fail at once, rather than the last.
+
+`staticPaths` has no upstream counterpart. It is the Remix 3 answer to React Router's `getStaticPaths`: a Remix router exposes no route table, but the route map an app builds it from is an ordinary object, and that is the thing worth reading.
+
+## License
+
+MIT

package/dist/index.d.mts

@@ -0,0 +1,101 @@
+import { RouteMap } from "remix/routes";
+//#region src/crawl.d.ts
+/** One fetched path and the response the router produced for it. */
+interface CrawlResult {
+  /** The path that was requested, exactly as it was queued. */
+  pathname: string;
+  /**
+   * Where the response belongs on disk. HTML lands at `<pathname>/index.html`
+   * so a static host serves it for the original path; everything else keeps
+   * its own path.
+   */
+  filepath: string;
+  /** The router's response. Unconsumed: the body is still readable. */
+  response: Response;
+}
+interface CrawlOptions {
+  /**
+   * Paths to start from.
+   *
+   * @default ["/"]
+   */
+  paths?: string[];
+  /**
+   * Follow `<a href>` and `<link rel="alternate">` to discover more paths.
+   * Turn it off to fetch exactly the paths given.
+   *
+   * @default true
+   */
+  spider?: boolean;
+  /**
+   * Queue the `<link href>`, `<script src>`, and `<img src>` a page
+   * references. Turn it off when something else already emitted those files,
+   * as a bundler does.
+   *
+   * @default true
+   */
+  assets?: boolean;
+  /**
+   * How many paths to fetch at once.
+   *
+   * @default 1
+   */
+  concurrency?: number;
+  /**
+   * Crawl a page's links even though the page asked robots not to follow
+   * them. Receives the page's path.
+   */
+  ignorePageNofollow?: (pathname: string) => boolean;
+  /**
+   * Called for a path that answered with a redirect rather than a document.
+   * Nothing is yielded for it: there is no page to write, and the app still
+   * answers the path at runtime.
+   *
+   * Receives the requested path and the `Location` it pointed at, which is
+   * `null` for a redirect that named none.
+   */
+  onRedirect?: (pathname: string, location: string | null) => void;
+}
+/**
+ * What {@link crawl} needs from a router. A `createRouter()` router satisfies
+ * it, and so does anything else that answers a `Request` — a built server
+ * bundle's default export, a worker-style `{ fetch }` object, a test double.
+ */
+interface CrawlTarget {
+  fetch(request: Request): Response | Promise<Response>;
+}
+/**
+ * Walks an app by dispatching requests straight into its router, yielding each
+ * response as it arrives. No socket, no server, no browser: the router's
+ * `fetch` is the whole transport, so this runs anywhere the app itself runs.
+ *
+ * Yields in completion order. Every path is fetched at most once. A redirect
+ * yields nothing and reports through {@link CrawlOptions.onRedirect}; any
+ * other non-2xx response aborts the crawl.
+ *
+ * @param router The router to crawl.
+ * @param options Crawl options.
+ * @returns An async iterator of results, one per fetched path.
+ */
+declare function crawl(router: CrawlTarget, options?: CrawlOptions): AsyncIterableIterator<CrawlResult>;
+//#endregion
+//#region src/static-paths.d.ts
+/**
+ * Collects every path in a route map that can be requested without params:
+ * the Remix 3 answer to "what pages does this app have", and the input a
+ * prerender pass needs before it knows any dynamic values.
+ *
+ * A route qualifies when it answers `GET` (or any method) and its pattern
+ * declares no variables or wildcards. `/blog` is a static path; `/blog/:slug`
+ * is not, because its values live outside the route map. Routes constrained to
+ * a protocol or hostname are skipped too, since their href is not a path.
+ *
+ * Results are deduplicated and sorted, so a build that prerenders them lists
+ * its output the same way every time.
+ *
+ * @param routes The route map, usually the one the app's router is built from.
+ * @returns The static paths, sorted.
+ */
+declare function staticPaths(routes: RouteMap): string[];
+//#endregion
+export { type CrawlOptions, type CrawlResult, type CrawlTarget, crawl, staticPaths };

package/dist/index.mjs

@@ -0,0 +1,402 @@
+import { getRoutePatternCaptures } from "remix/route-pattern";
+//#region src/html-parser.ts
+const VOID_ELEMENTS = /* @__PURE__ */ new Set([
+	"area",
+	"base",
+	"br",
+	"col",
+	"embed",
+	"hr",
+	"img",
+	"input",
+	"link",
+	"meta",
+	"param",
+	"source",
+	"track",
+	"wbr"
+]);
+const RAW_TEXT_ELEMENTS = /* @__PURE__ */ new Set(["script", "style"]);
+var Element = class {
+	name;
+	#attributes = /* @__PURE__ */ new Map();
+	#children = [];
+	#selfClosing;
+	constructor(name, selfClosing = false) {
+		this.name = name.toLowerCase();
+		this.#selfClosing = selfClosing;
+	}
+	setAttribute(name, value) {
+		this.#attributes.set(name.toLowerCase(), value);
+	}
+	getAttribute(name) {
+		return this.#attributes.get(name.toLowerCase()) ?? null;
+	}
+	get innerHTML() {
+		return serialize(this.#children);
+	}
+	set innerHTML(value) {
+		this.#children = [value];
+	}
+	appendChild(child) {
+		this.#children.push(child);
+	}
+	isSelfClosing() {
+		return this.#selfClosing;
+	}
+	toString() {
+		if (this.name === "#comment") return this.#attributes.get("text") ?? "";
+		let attrs = Array.from(this.#attributes.entries()).map(([key, value]) => value === "" ? key : `${key}="${value.replace(/"/g, """)}"`).join(" ");
+		let attrString = attrs ? ` ${attrs}` : "";
+		if (this.name.startsWith("!")) return `<${this.name}${attrString}>`;
+		if (VOID_ELEMENTS.has(this.name) || this.#selfClosing) return `<${this.name}${attrString} />`;
+		return `<${this.name}${attrString}>${serialize(this.#children)}</${this.name}>`;
+	}
+};
+/** A parsed document: the flat element list, plus the tree for serialization. */
+var Document = class {
+	#elements;
+	#children;
+	constructor(elements, children) {
+		this.#elements = elements;
+		this.#children = children;
+	}
+	/** Every element in the document, in source order. */
+	get elements() {
+		return this.#elements;
+	}
+	toString() {
+		return serialize(this.#children);
+	}
+};
+function serialize(children) {
+	return children.map((child) => typeof child === "string" ? child : child.toString()).join("");
+}
+/**
+* Index of the `>` that closes the tag opening at `start`, skipping any `>`
+* inside a quoted attribute value. Returns -1 when the tag never closes.
+*/
+function findTagEnd(html, start) {
+	let quote = null;
+	for (let i = start; i < html.length; i++) {
+		let char = html[i];
+		if (char === void 0) break;
+		if (quote) {
+			if (char === quote) quote = null;
+			continue;
+		}
+		if (char === "\"" || char === "'") {
+			quote = char;
+			continue;
+		}
+		if (char === ">") return i;
+	}
+	return -1;
+}
+function parseTag(content) {
+	let source = content.trim();
+	let selfClosing = false;
+	if (source.endsWith("/")) {
+		selfClosing = true;
+		source = source.slice(0, -1).trimEnd();
+	}
+	let nameEnd = 0;
+	while (nameEnd < source.length && !/\s/.test(source[nameEnd] ?? "")) nameEnd++;
+	let name = source.slice(0, nameEnd);
+	if (!name) return null;
+	let element = new Element(name, selfClosing);
+	let i = nameEnd;
+	while (i < source.length) {
+		while (i < source.length && /\s/.test(source[i] ?? "")) i++;
+		if (i >= source.length) break;
+		let attrNameStart = i;
+		while (i < source.length && !/[\s=]/.test(source[i] ?? "")) i++;
+		let attrName = source.slice(attrNameStart, i);
+		if (!attrName) break;
+		while (i < source.length && /\s/.test(source[i] ?? "")) i++;
+		if (source[i] !== "=") {
+			element.setAttribute(attrName, "");
+			continue;
+		}
+		i++;
+		while (i < source.length && /\s/.test(source[i] ?? "")) i++;
+		let quote = source[i];
+		if (quote === "\"" || quote === "'") {
+			i++;
+			let valueStart = i;
+			while (i < source.length && source[i] !== quote) i++;
+			element.setAttribute(attrName, source.slice(valueStart, i));
+			if (source[i] === quote) i++;
+			continue;
+		}
+		let valueStart = i;
+		while (i < source.length && !/\s/.test(source[i] ?? "")) i++;
+		element.setAttribute(attrName, source.slice(valueStart, i));
+	}
+	return element;
+}
+/**
+* Parses an HTML document into a flat element list and a serializable tree.
+*
+* @param html The document source.
+* @param options Parse options.
+* @returns The parsed document.
+*/
+function parse(html, options) {
+	let elements = [];
+	let children = [];
+	let stack = [];
+	let i = 0;
+	let appendChild = (child) => {
+		let parent = stack.at(-1);
+		if (parent) parent.appendChild(child);
+		else children.push(child);
+	};
+	let appendElement = (element) => {
+		elements.push(element);
+		appendChild(element);
+	};
+	let closeElement = (name) => {
+		let normalized = name.toLowerCase();
+		for (let index = stack.length - 1; index >= 0; index--) if (stack[index]?.name === normalized) {
+			stack.length = index;
+			return;
+		}
+	};
+	while (i < html.length) {
+		let lt = html.indexOf("<", i);
+		if (lt === -1) {
+			appendChild(html.slice(i));
+			break;
+		}
+		if (lt > i) appendChild(html.slice(i, lt));
+		if (html.startsWith("<!--", lt)) {
+			let end = html.indexOf("-->", lt + 4);
+			if (end === -1) {
+				appendChild(html.slice(lt));
+				break;
+			}
+			if (options?.comment) {
+				let comment = new Element("#comment");
+				comment.setAttribute("text", html.slice(lt, end + 3));
+				appendElement(comment);
+			}
+			i = end + 3;
+			continue;
+		}
+		if (html[lt + 1] === "/") {
+			let end = findTagEnd(html, lt + 2);
+			if (end === -1) {
+				appendChild(html.slice(lt));
+				break;
+			}
+			let tagName = html.slice(lt + 2, end).trim().split(/\s+/, 1)[0];
+			if (tagName) closeElement(tagName);
+			i = end + 1;
+			continue;
+		}
+		let end = findTagEnd(html, lt + 1);
+		if (end === -1) {
+			appendChild(html.slice(lt));
+			break;
+		}
+		let element = parseTag(html.slice(lt + 1, end));
+		if (!element) {
+			appendChild(html.slice(lt, end + 1));
+			i = end + 1;
+			continue;
+		}
+		appendElement(element);
+		i = end + 1;
+		if (element.name.startsWith("!") || VOID_ELEMENTS.has(element.name) || element.isSelfClosing()) continue;
+		if (RAW_TEXT_ELEMENTS.has(element.name)) {
+			let closeStart = html.toLowerCase().indexOf(`</${element.name}`, i);
+			if (closeStart === -1) {
+				element.appendChild(html.slice(i));
+				break;
+			}
+			element.appendChild(html.slice(i, closeStart));
+			let closeEnd = findTagEnd(html, closeStart + 2);
+			if (closeEnd === -1) break;
+			i = closeEnd + 1;
+			continue;
+		}
+		stack.push(element);
+	}
+	return new Document(elements, children);
+}
+//#endregion
+//#region src/crawl.ts
+/**
+* Requests are dispatched straight into the router, so the origin is never
+* used for anything but satisfying the `Request` constructor.
+*/
+const BASE_URL = "http://localhost";
+/**
+* Walks an app by dispatching requests straight into its router, yielding each
+* response as it arrives. No socket, no server, no browser: the router's
+* `fetch` is the whole transport, so this runs anywhere the app itself runs.
+*
+* Yields in completion order. Every path is fetched at most once. A redirect
+* yields nothing and reports through {@link CrawlOptions.onRedirect}; any
+* other non-2xx response aborts the crawl.
+*
+* @param router The router to crawl.
+* @param options Crawl options.
+* @returns An async iterator of results, one per fetched path.
+*/
+async function* crawl(router, options = {}) {
+	let { paths = ["/"], spider = true, assets = true, concurrency = 1, ignorePageNofollow, onRedirect } = options;
+	let queue = [];
+	let visited = /* @__PURE__ */ new Set();
+	let results = [];
+	let active = 0;
+	let error;
+	let notify = () => {};
+	let gate = new Promise((resolve) => notify = resolve);
+	function bump() {
+		let previous = notify;
+		gate = new Promise((resolve) => notify = resolve);
+		previous();
+	}
+	enqueue(paths);
+	while (true) {
+		while (active < concurrency && queue.length > 0) fetchOne(queue.shift());
+		if (error) throw error;
+		if (results.length > 0) {
+			yield results.shift();
+			continue;
+		}
+		if (active === 0 && queue.length === 0) break;
+		await gate;
+	}
+	function enqueue(pathnames) {
+		for (let pathname of pathnames) {
+			if (visited.has(pathname)) continue;
+			visited.add(pathname);
+			queue.push(pathname);
+		}
+	}
+	async function fetchOne(pathname) {
+		active++;
+		try {
+			let response = await router.fetch(new Request(`${BASE_URL}${pathname}`));
+			if (response.status >= 300 && response.status < 400) {
+				let location = response.headers.get("Location");
+				onRedirect?.(pathname, location);
+				await response.body?.cancel();
+				if (spider && location != null) enqueue(resolveAll([location], pathname));
+				return;
+			}
+			if (!response.ok) {
+				let status = [response.status, response.statusText].filter(Boolean).join(" ");
+				throw new Error(`Crawl failed: ${status} (${pathname})`);
+			}
+			if (!response.headers.get("Content-Type")?.includes("text/html")) {
+				results.push({
+					pathname,
+					filepath: pathname,
+					response
+				});
+				return;
+			}
+			let cloned = response.clone();
+			results.push({
+				pathname,
+				filepath: pathname.replace(/\/?$/, "/index.html"),
+				response
+			});
+			let document = parse(await cloned.text());
+			if (assets) enqueue(extractAssetPaths(document.elements, pathname));
+			if (spider && (ignorePageNofollow?.(pathname) || shouldCrawlLinks(document.elements))) enqueue(extractLinkPaths(document.elements, pathname));
+		} catch (thrown) {
+			error ??= thrown;
+		} finally {
+			active--;
+			bump();
+		}
+	}
+}
+function extractAssetPaths(elements, baseUrl) {
+	let hrefs = elements.filter((element) => element.name === "link" && !rel(element).includes("nofollow")).map((element) => element.getAttribute("href"));
+	let sources = elements.filter((element) => element.name === "script" || element.name === "img").map((element) => element.getAttribute("src"));
+	return resolveAll([...hrefs, ...sources], baseUrl);
+}
+function extractLinkPaths(elements, baseUrl) {
+	return resolveAll(elements.filter((element) => !rel(element).includes("nofollow") && (element.name === "a" || element.name === "link" && rel(element).includes("alternate"))).map((element) => element.getAttribute("href")), baseUrl);
+}
+function resolveAll(hrefs, baseUrl) {
+	let paths = [];
+	for (let href of hrefs) {
+		if (href == null || isNonNavigable(href) || !isRelativeUrl(href)) continue;
+		let resolved = resolveHref(href, baseUrl);
+		if (resolved != null) paths.push(resolved);
+	}
+	return paths;
+}
+/** Whether a page opted out of having its links followed. */
+function shouldCrawlLinks(elements) {
+	return !elements.some((element) => {
+		if (element.name !== "meta") return false;
+		let name = element.getAttribute("name")?.toLowerCase();
+		if (name !== "robots" && name !== "googlebot") return false;
+		return (element.getAttribute("content")?.toLowerCase() ?? "").split(/[\s,]+/).includes("nofollow");
+	});
+}
+function rel(element) {
+	return element.getAttribute("rel")?.split(/\s+/) ?? [];
+}
+function isNonNavigable(href) {
+	return href.startsWith("#") || href.startsWith("mailto:") || href.startsWith("tel:") || href.startsWith("javascript:") || href.startsWith("data:");
+}
+function isRelativeUrl(href) {
+	return !href.startsWith("http://") && !href.startsWith("https://") && !href.startsWith("//");
+}
+function resolveHref(href, baseUrl) {
+	if (href.startsWith("/")) return href;
+	try {
+		return new URL(href, `${BASE_URL}${baseUrl}`).pathname;
+	} catch {
+		return null;
+	}
+}
+//#endregion
+//#region src/static-paths.ts
+function isRoute(value) {
+	return typeof value === "object" && value !== null && "method" in value && "pattern" in value && typeof value.href === "function";
+}
+/**
+* Collects every path in a route map that can be requested without params:
+* the Remix 3 answer to "what pages does this app have", and the input a
+* prerender pass needs before it knows any dynamic values.
+*
+* A route qualifies when it answers `GET` (or any method) and its pattern
+* declares no variables or wildcards. `/blog` is a static path; `/blog/:slug`
+* is not, because its values live outside the route map. Routes constrained to
+* a protocol or hostname are skipped too, since their href is not a path.
+*
+* Results are deduplicated and sorted, so a build that prerenders them lists
+* its output the same way every time.
+*
+* @param routes The route map, usually the one the app's router is built from.
+* @returns The static paths, sorted.
+*/
+function staticPaths(routes) {
+	let paths = /* @__PURE__ */ new Set();
+	collect(routes, paths);
+	return [...paths].sort();
+}
+function collect(node, paths) {
+	for (let value of Object.values(node)) {
+		if (isRoute(value)) {
+			if (value.method !== "GET" && value.method !== "ANY") continue;
+			if (getRoutePatternCaptures(value.pattern).length > 0) continue;
+			let href = value.href();
+			if (href.startsWith("/")) paths.add(href);
+			continue;
+		}
+		if (typeof value === "object" && value !== null) collect(value, paths);
+	}
+}
+//#endregion
+export { crawl, staticPaths };

package/package.json

@@ -0,0 +1,51 @@
+{
+    "name": "@pitlane/crawler",
+    "version": "0.1.0",
+    "description": "crawl() — spider a Remix 3 fetch router in memory to prerender it, plus static-path discovery from a route map.",
+    "keywords": [
+        "crawl",
+        "fetch-router",
+        "pitlane",
+        "prerender",
+        "remix",
+        "ssg"
+    ],
+    "homepage": "https://pitlane.tools/package/crawler/",
+    "bugs": {
+        "url": "https://github.com/pitlane-tools/pitlane/issues"
+    },
+    "license": "MIT",
+    "author": "Mark Malstrom <mark@malstrom.me>",
+    "repository": {
+        "type": "git",
+        "url": "git+https://github.com/pitlane-tools/pitlane.git",
+        "directory": "packages/crawler"
+    },
+    "files": [
+        "dist",
+        "CHANGELOG.md"
+    ],
+    "type": "module",
+    "types": "./dist/index.d.mts",
+    "exports": {
+        ".": {
+            "types": "./dist/index.d.mts",
+            "import": "./dist/index.mjs"
+        }
+    },
+    "scripts": {
+        "prepublishOnly": "vp run build"
+    },
+    "devDependencies": {
+        "@types/node": "^25.5.0",
+        "remix": "3.0.0-beta.10",
+        "typescript": "^7.0.2",
+        "vite-plus": "^0.2.6"
+    },
+    "peerDependencies": {
+        "remix": "^3.0.0-beta.10"
+    },
+    "engines": {
+        "node": "^20.19.0 || >=22.12.0"
+    }
+}