prerender-crawler 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ryan Carniato
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,122 @@
+# prerender-crawler
+
+Framework-agnostic build-time prerendering. Point it at anything fetch-shaped — `Request` in, `Response` out — and it crawls the site into static files: seed pages, link discovery, header hints, redirects, retries, throttling, and an integration seam for capturing build-time data alongside the pages. No browser, no subprocess, no framework knowledge.
+
+Ships as an engine (`prerender-crawler`) and a Vite plugin built on it (`prerender-crawler/vite`).
+
+## Vite plugin
+
+```ts
+// vite.config.ts
+import { defineConfig } from "vite";
+import { prerender } from "prerender-crawler/vite";
+
+export default defineConfig({
+  plugins: [
+    framework(), // anything producing a client build and an SSR build
+    prerender({ mode: "static" })
+  ]
+});
+```
+
+The plugin is build-only. It assumes three things about the app:
+
+1. `vite build` produces a client output directory (the `client` environment's `outDir`, default `dist/client`).
+2. Some environment's output includes a module exporting a request handler — `handleRequest`, `fetch`, or `default.fetch`. Default: `server.js` in the `ssr` environment's `outDir`; override with `serverEntry`.
+3. Optionally, a [`filesystem-routing`](https://www.npmjs.com/package/filesystem-routing) route directory names the static pages.
+
+After the other environments build, it imports the server handler and crawls it in-process. Pages and integration-emitted files land in the client output.
+
+### Options
+
+Everything from [`PrerenderOptions`](#engine-options) plus:
+
+| Option        | Default                  |                                                                                                                                                                                                                                                                                                                                                             |
+| ------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `serverEntry` | `<ssr outDir>/server.js` | Built module exporting the handler.                                                                                                                                                                                                                                                                                                                         |
+| `fileRoutes`  | `true`                   | Seed the crawl with the static pages of the project's `filesystem-routing` directory. `true` applies when the package and `src/routes` exist and is skipped silently otherwise; pass `{ dir, extensions }` to mirror a customized `fileRoutes()` (then a missing package is an error); `false` disables. Dynamic routes are still found by following links. |
+
+### `import.meta.env.PRERENDER_MODE`
+
+The plugin defines this constant in every build environment with the run's `mode`. Runtime code can ask "am I a prerendered build, and which kind" — absent (dev, or a build without the plugin) means live. It's how integrations ship posture-aware client behavior without the app wiring anything.
+
+## Modes
+
+The one distinction every downstream policy keys on:
+
+- **`"static"`** — the written files _are_ the deployment. Every rendered page is written; anything the crawl didn't produce doesn't exist at runtime, so integrations treat gaps as errors.
+- **`"hybrid"`** — a live server is deployed alongside. The crawl is a build-time pass (data baking, selected pages). Rendered pages are _not_ written by default, because on most hosts a static HTML file shadows live SSR of the same route; gaps fall back to the server.
+
+## Engine
+
+The plugin is a thin driver. The engine works with any transport:
+
+```ts
+import { runPrerender } from "prerender-crawler";
+
+const result = await runPrerender({
+  transport: { fetch: request => app.handle(request) },
+  outDir: "dist",
+  pages: ["/", "/about", { path: "/404", filename: "404.html" }],
+  mode: "static"
+});
+
+result.pages; // RenderedPage[] — path, referrers, filename, emitted, html
+result.files; // EmittedFile[]  — what integrations emitted
+result.skipped; // SkippedPage[]  — failures left out (failOnError: false)
+```
+
+### Engine options
+
+| Option                   | Default                        |                                                                                                                                                                                         |
+| ------------------------ | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `mode`                   | `"static"`                     | See [Modes](#modes). Decides the `emitPages` default.                                                                                                                                   |
+| `pages`                  | `["/"]`                        | Seeds: strings, `{ path, filename?, emit? }` entries, or a (async) function returning them. Duplicates collapse to one render.                                                          |
+| `crawlLinks`             | `true`                         | Follow same-origin links in rendered HTML. The only way dynamic routes are discovered without explicit seeding.                                                                         |
+| `hintHeader`             | `"x-prerender"`                | Response header naming additional paths (comma-separated) — the route the data lives on announces the routes built from it.                                                             |
+| `filter`                 |                                | `(path) => boolean`; drops a discovered path before it's fetched.                                                                                                                       |
+| `concurrency`            | `8`                            | Pages in flight at once.                                                                                                                                                                |
+| `interval`               | `0`                            | Minimum ms between the starts of consecutive requests across all workers — a throttle for renders hitting rate-limited APIs.                                                            |
+| `retries` / `retryDelay` | `2` / `500`                    | Re-fetch attempts for a failed page, and the wait between them.                                                                                                                         |
+| `failOnError`            | `true`                         | Whether a page that still fails after retries fails the run. Otherwise it's reported in `skipped`, with the pages that linked to it.                                                    |
+| `maxRedirects`           | `5`                            | Internal redirect hops followed for one page.                                                                                                                                           |
+| `emitPages`              | `true` static / `false` hybrid | Whether rendered pages are written: a boolean, or a per-path predicate. Per-entry `emit` overrides. Unemitted pages still render fully — links are still followed, data still captured. |
+| `autoSubfolderIndex`     | `true`                         | `/about` → `about/index.html` (true) or `about.html` (false).                                                                                                                           |
+| `origin`                 | `"http://localhost"`           | Origin requests are minted under.                                                                                                                                                       |
+| `onRendered`             |                                | Observes every rendered page — the seam for sitemaps and post-processing.                                                                                                               |
+| `integrations`           | `[]`                           | See below.                                                                                                                                                                              |
+
+### Integrations
+
+An integration is a hook bundle that participates in the run without owning it:
+
+```ts
+interface PrerenderIntegration {
+  name: string;
+  setup?(context: PrerenderContext): void | Promise<void>; // before the first render
+  teardown?(context: PrerenderContext): void | Promise<void>; // after the last render, before writes
+  client?: string; // module a bundler plugin imports into the client build (reserved)
+}
+
+interface PrerenderContext {
+  mode: PrerenderMode;
+  origin: string;
+  outDir: string;
+  emitFile(file: { filename: string; contents: string | Uint8Array }): void;
+}
+```
+
+`emitFile` is the channel for artifacts produced during the crawl — captured server-function results, extracted payloads, sitemaps. Throwing from `teardown` fails the run: the place to verify the crawl produced everything the runtime half will need. [`@solidjs/prerender`](../solid) is the reference integration.
+
+### Utilities
+
+- `fileRoutePages({ root, dir, extensions })` / `staticRoutePaths(entries)` — the static page paths of a `filesystem-routing` manifest, as a `pages` source.
+- `extractLinks(html)`, `normalizeLink(href, from)`, `normalizePath(path)`, `outputFilename(path, autoSubfolderIndex)` — the crawl's own primitives.
+
+## Requirements
+
+Node 20+. Vite 7 or 8 for the plugin (optional peer). `filesystem-routing` ≥ 0.2 for route seeding (optional peer).
+
+## License
+
+MIT

package/dist/crawl.d.ts

@@ -0,0 +1,12 @@
+import type { PrerenderOptions, PrerenderResult, Transport } from "./types.ts";
+export interface RunOptions extends PrerenderOptions {
+    transport: Transport;
+    outDir: string;
+}
+/**
+ * The run: seed, fetch, follow redirects, write, discover (links + header
+ * hints), repeat until the queue drains. Every mechanism here is
+ * transport-agnostic — in-process dispatch and loopback HTTP drive the
+ * identical loop.
+ */
+export declare function runPrerender(options: RunOptions): Promise<PrerenderResult>;

package/dist/crawl.js

@@ -0,0 +1,222 @@
+import { mkdir, writeFile } from "node:fs/promises";
+import { dirname, join } from "node:path";
+import { extractLinks, normalizeLink, normalizePath } from "./links.js";
+import { outputFilename } from "./output.js";
+const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));
+async function resolveSeeds(pages) {
+    const source = typeof pages === "function" ? await pages() : (pages ?? ["/"]);
+    // Seed sources overlap routinely (an explicit list plus a route-manifest
+    // scan both naming `/`): one render per path, the first spelling wins.
+    const byPath = new Map();
+    for (const entry of source) {
+        const page = typeof entry === "string"
+            ? { path: normalizePath(entry) }
+            : { ...entry, path: normalizePath(entry.path) };
+        if (!byPath.has(page.path))
+            byPath.set(page.path, page);
+    }
+    return [...byPath.values()];
+}
+/**
+ * The run: seed, fetch, follow redirects, write, discover (links + header
+ * hints), repeat until the queue drains. Every mechanism here is
+ * transport-agnostic — in-process dispatch and loopback HTTP drive the
+ * identical loop.
+ */
+export async function runPrerender(options) {
+    const { transport, outDir, mode = "static", crawlLinks = true, hintHeader = "x-prerender", filter, concurrency = 8, interval = 0, retries = 2, retryDelay = 500, failOnError = true, maxRedirects = 5, 
+    // hybrid's crawl bakes data; writing its HTML would shadow live SSR
+    emitPages = mode === "static", autoSubfolderIndex = true, origin = "http://localhost", onRendered, integrations = [] } = options;
+    const originUrl = new URL(origin);
+    const rendered = [];
+    const skipped = [];
+    const emitted = [];
+    const context = {
+        mode,
+        origin,
+        outDir,
+        emitFile: file => void emitted.push(file)
+    };
+    const seeds = await resolveSeeds(options.pages);
+    const seen = new Set(seeds.map(page => page.path));
+    const queue = [...seeds];
+    // Provenance: which pages named each discovered path. Recorded for every
+    // mention (not just the first), so a failure can point at every page
+    // carrying the broken link.
+    const referrers = new Map();
+    const referrersOf = (path) => [...(referrers.get(path) ?? [])];
+    // Discovered paths (crawled links, header hints) pass the filter;
+    // explicitly seeded pages are the caller's statement of intent and skip it.
+    const discovered = (path, from) => {
+        let sources = referrers.get(path);
+        if (!sources)
+            referrers.set(path, (sources = new Set()));
+        sources.add(from);
+        if (seen.has(path) || (filter && !filter(path)))
+            return;
+        seen.add(path);
+        queue.push({ path });
+    };
+    // The throttle: every request start claims the next slot on a shared
+    // timeline, so starts are at least `interval` apart no matter how many
+    // workers are running.
+    let nextSlot = 0;
+    async function pace() {
+        if (interval <= 0)
+            return;
+        const now = Date.now();
+        const slot = Math.max(now, nextSlot);
+        nextSlot = slot + interval;
+        if (slot > now)
+            await wait(slot - now);
+    }
+    async function fetchFollowingRedirects(path) {
+        let url = new URL(path, originUrl);
+        for (let hop = 0;; hop++) {
+            await pace();
+            const response = await transport.fetch(new Request(url, { headers: { accept: "text/html,*/*", [hintHeader]: "1" } }));
+            const location = response.headers.get("location");
+            if (response.status < 300 || response.status >= 400 || !location)
+                return response;
+            if (hop >= maxRedirects) {
+                throw new Error(`Redirect chain from ${path} exceeded ${maxRedirects} hops`);
+            }
+            const target = new URL(location, url);
+            if (target.origin !== originUrl.origin) {
+                // an external redirect terminates the chain; the page becomes a stub
+                return response;
+            }
+            url = target;
+        }
+    }
+    async function renderPage(entry) {
+        let response;
+        let error;
+        for (let attempt = 0; attempt <= retries; attempt++) {
+            if (attempt > 0)
+                await wait(retryDelay);
+            try {
+                response = await fetchFollowingRedirects(entry.path);
+                error = undefined;
+                if (response.status < 500)
+                    break; // retry only what might heal
+            }
+            catch (thrown) {
+                error = thrown;
+            }
+        }
+        if (!response || error !== undefined || response.status >= 400) {
+            // The message carries provenance: a 404 is usually a broken link, and
+            // the fix lives on the pages that carry it, not at the missing route.
+            const from = referrersOf(entry.path);
+            const linked = from.length ? ` (linked from ${from.join(", ")})` : "";
+            const failure = error !== undefined
+                ? new Error(`Prerendering ${entry.path} failed${linked}: ${describe(error)}`, {
+                    cause: error
+                })
+                : new Error(`Prerendering ${entry.path} answered ${response.status}${linked}`);
+            if (failOnError)
+                throw failure;
+            // referrers are completed once the crawl settles: pages still in
+            // flight may yet link here, and the report should name all of them
+            skipped.push({ path: entry.path, error: failure, referrers: [] });
+            return;
+        }
+        const filename = entry.filename ?? outputFilename(entry.path, autoSubfolderIndex);
+        let html;
+        const contentType = response.headers.get("content-type") ?? "";
+        const isHTML = contentType.includes("text/html");
+        const location = response.headers.get("location");
+        if (location && response.status >= 300 && response.status < 400) {
+            // external redirect (internal ones were followed): a meta-refresh stub
+            // keeps the path working on hosts without redirect support
+            html = redirectStub(location);
+        }
+        else {
+            html = await response.text();
+        }
+        // Emission is policy, rendering is not: an unemitted page has still
+        // fully executed (integration capture happened server-side) and its
+        // links still feed the crawl — it just leaves no HTML file behind to
+        // shadow a live server's SSR of the route.
+        const emitted = entry.emit ?? (typeof emitPages === "function" ? emitPages(entry.path) : emitPages);
+        if (emitted)
+            await writeOutput(outDir, filename, html);
+        if (isHTML) {
+            const pageUrl = new URL(entry.path, originUrl);
+            if (crawlLinks) {
+                for (const path of extractLinks(html, pageUrl))
+                    discovered(path, entry.path);
+            }
+        }
+        const hints = response.headers.get(hintHeader);
+        if (hints) {
+            for (const hint of hints.split(",")) {
+                const path = normalizeLink(hint.trim(), originUrl, originUrl.origin);
+                if (path !== undefined)
+                    discovered(path, entry.path);
+            }
+        }
+        const page = {
+            path: entry.path,
+            referrers: referrersOf(entry.path),
+            filename,
+            emitted,
+            response,
+            html
+        };
+        rendered.push(page);
+        if (onRendered)
+            await onRendered(page);
+    }
+    try {
+        for (const integration of integrations)
+            await integration.setup?.(context);
+        // A worker pool over a queue that grows while it drains: discoveries
+        // enqueue, idle workers pick them up, the run resolves when the last
+        // worker goes idle against an empty queue.
+        await new Promise((resolve, reject) => {
+            let active = 0;
+            let failed = false;
+            const pump = () => {
+                if (failed)
+                    return;
+                while (active < concurrency && queue.length) {
+                    const entry = queue.shift();
+                    active++;
+                    renderPage(entry).then(() => {
+                        active--;
+                        pump();
+                    }, failure => {
+                        failed = true;
+                        reject(failure);
+                    });
+                }
+                if (active === 0 && queue.length === 0)
+                    resolve();
+            };
+            pump();
+        });
+        for (const miss of skipped)
+            miss.referrers = referrersOf(miss.path);
+        for (const integration of integrations)
+            await integration.teardown?.(context);
+        for (const file of emitted) {
+            await writeOutput(outDir, file.filename, file.contents);
+        }
+    }
+    finally {
+        await transport.close?.();
+    }
+    return { pages: rendered, files: emitted, skipped };
+}
+const describe = (error) => (error instanceof Error ? error.message : String(error));
+function redirectStub(location) {
+    const target = String(location).replace(/"/g, "&quot;");
+    return `<!DOCTYPE html><html><head><meta http-equiv="refresh" content="0; url=${target}"><link rel="canonical" href="${target}"></head></html>`;
+}
+async function writeOutput(outDir, filename, contents) {
+    const target = join(outDir, filename);
+    await mkdir(dirname(target), { recursive: true });
+    await writeFile(target, contents);
+}

package/dist/file-routes.d.ts

@@ -0,0 +1,35 @@
+export interface FileRoutePagesOptions {
+    /** Project root the route dir resolves against. @default process.cwd() */
+    root?: string;
+    /** Route directory, mirroring `fileRoutes({ dir })`. @default "src/routes" */
+    dir?: string;
+    /** Route file extensions, mirroring `fileRoutes({ extensions })`. @default ["js", "jsx", "ts", "tsx"] */
+    extensions?: string[];
+}
+/** The subset of a `filesystem-routing` manifest entry this module reads. */
+export interface RouteEntryLike {
+    path: string;
+    page?: boolean;
+}
+/**
+ * The statically addressable page paths of a route manifest: pages whose
+ * path has no parameter or catch-all segment, with `(group)` segments
+ * stripped the way emission adapters strip them. Pure — the seam the
+ * plugin and tests share.
+ */
+export declare function staticRoutePaths(entries: readonly RouteEntryLike[]): string[];
+/**
+ * A `pages` source scanning a `filesystem-routing` route directory for its
+ * static pages. The package is resolved from the project root (it is the
+ * APP's dependency), loaded lazily so projects without it pay nothing.
+ *
+ * ```ts
+ * prerender({ pages: fileRoutePages({ dir: "src/pages" }) })
+ * ```
+ *
+ * The `prerender()` plugin applies this automatically (`fileRoutes: true`,
+ * the default) when the package and the route directory exist.
+ */
+export declare function fileRoutePages(options?: FileRoutePagesOptions): () => Promise<string[]>;
+/** Whether `filesystem-routing` resolves from the project (for the plugin's auto mode). */
+export declare function hasFileSystemRouting(root: string): boolean;

package/dist/file-routes.js

@@ -0,0 +1,84 @@
+var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExtension) || function (path, preserveJsx) {
+    if (typeof path === "string" && /^\.\.?\//.test(path)) {
+        return path.replace(/\.(tsx)$|((?:\.d)?)((?:\.[^./]+?)?)\.([cm]?)ts$/i, function (m, tsx, d, ext, cm) {
+            return tsx ? preserveJsx ? ".jsx" : ".js" : d && (!ext || !cm) ? m : (d + ext + "." + cm.toLowerCase() + "js");
+        });
+    }
+    return path;
+};
+// Route-manifest seeding: the static pages a file-system router declares
+// are known before anything renders, so they seed the crawl directly —
+// a page nothing links to still gets built, and the crawl starts wide
+// instead of unwinding from `/`. Dynamic routes (`/posts/:slug`,
+// `/*404`) are left to link discovery: only a render knows their values.
+import { createRequire } from "node:module";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+/**
+ * The statically addressable page paths of a route manifest: pages whose
+ * path has no parameter or catch-all segment, with `(group)` segments
+ * stripped the way emission adapters strip them. Pure — the seam the
+ * plugin and tests share.
+ */
+export function staticRoutePaths(entries) {
+    const paths = new Set();
+    for (const entry of entries) {
+        if (!entry.page)
+            continue;
+        const segments = entry.path.split("/").filter(segment => segment !== "");
+        if (segments.some(segment => segment.startsWith(":") || segment.startsWith("*")))
+            continue;
+        const concrete = segments.filter(segment => !/^\(.*\)$/.test(segment));
+        paths.add("/" + concrete.join("/"));
+    }
+    return [...paths];
+}
+/**
+ * A `pages` source scanning a `filesystem-routing` route directory for its
+ * static pages. The package is resolved from the project root (it is the
+ * APP's dependency), loaded lazily so projects without it pay nothing.
+ *
+ * ```ts
+ * prerender({ pages: fileRoutePages({ dir: "src/pages" }) })
+ * ```
+ *
+ * The `prerender()` plugin applies this automatically (`fileRoutes: true`,
+ * the default) when the package and the route directory exist.
+ */
+export function fileRoutePages(options = {}) {
+    const root = options.root ?? process.cwd();
+    const dir = path.resolve(root, options.dir ?? "src/routes");
+    const extensions = options.extensions ?? ["js", "jsx", "ts", "tsx"];
+    return async () => {
+        const routing = await loadFileSystemRouting(root);
+        const router = new routing.PageFileSystemRouter({ dir, extensions });
+        return staticRoutePaths(await router.getRoutes());
+    };
+}
+/** Whether `filesystem-routing` resolves from the project (for the plugin's auto mode). */
+export function hasFileSystemRouting(root) {
+    try {
+        resolveFileSystemRouting(root);
+        return true;
+    }
+    catch {
+        return false;
+    }
+}
+function resolveFileSystemRouting(root) {
+    // from the project first (the app's copy), then from here (a test or a
+    // setup that installed it alongside the plugin)
+    for (const from of [path.join(root, "package.json"), import.meta.url]) {
+        try {
+            return createRequire(from).resolve("filesystem-routing");
+        }
+        catch {
+            // try the next base
+        }
+    }
+    throw new Error(`fileRoutePages needs the "filesystem-routing" package, which could not be resolved from ${root}.`);
+}
+async function loadFileSystemRouting(root) {
+    const resolved = resolveFileSystemRouting(root);
+    return (await import(__rewriteRelativeImportExtension(pathToFileURL(resolved).href)));
+}

package/dist/index.d.ts

@@ -0,0 +1,5 @@
+export { runPrerender } from "./crawl.ts";
+export type { RunOptions } from "./crawl.ts";
+export { extractLinks, normalizeLink, normalizePath } from "./links.ts";
+export { outputFilename } from "./output.ts";
+export type { EmittedFile, PageEntry, PagesSource, PrerenderContext, PrerenderIntegration, PrerenderMode, PrerenderOptions, PrerenderResult, RenderedPage, SkippedPage, Transport } from "./types.ts";

package/dist/index.js

@@ -0,0 +1,3 @@
+export { runPrerender } from "./crawl.js";
+export { extractLinks, normalizeLink, normalizePath } from "./links.js";
+export { outputFilename } from "./output.js";

package/dist/links.d.ts

@@ -0,0 +1,24 @@
+/**
+ * Link discovery: which hrefs in a rendered page name more pages of this
+ * site. The rules here are correctness fixes other prerenderers earned one
+ * bug report at a time — resolve relative hrefs against the PAGE's URL
+ * (not the origin), honor <base href>, strip queries and fragments before
+ * dedupe, and never leave the origin.
+ */
+/**
+ * Extracts the crawlable same-origin route paths from a page's HTML.
+ * Returned paths are normalized (`/about`, no query, no fragment, no
+ * trailing slash except the root) and deduped.
+ */
+export declare function extractLinks(html: string, pageUrl: URL): string[];
+/**
+ * Resolves one href to a normalized same-origin path, or undefined when it
+ * is not a page of this site (foreign origin, unparseable).
+ */
+export declare function normalizeLink(href: string, base: URL, origin: string): string | undefined;
+/**
+ * One spelling per page: query and fragment never reach here (URL parsing
+ * split them off), the trailing slash is dropped (except the root), and
+ * percent-encoding is left exactly as the URL parser produced it.
+ */
+export declare function normalizePath(pathname: string): string;

package/dist/links.js

@@ -0,0 +1,67 @@
+/**
+ * Link discovery: which hrefs in a rendered page name more pages of this
+ * site. The rules here are correctness fixes other prerenderers earned one
+ * bug report at a time — resolve relative hrefs against the PAGE's URL
+ * (not the origin), honor <base href>, strip queries and fragments before
+ * dedupe, and never leave the origin.
+ */
+const LINK_PATTERN = /<a\s[^>]*?href\s*=\s*(?:"([^"]*)"|'([^']*)')/gis;
+const BASE_PATTERN = /<base\s[^>]*?href\s*=\s*(?:"([^"]*)"|'([^']*)')/is;
+/** Schemes that are never pages. */
+const NON_PAGE_SCHEME = /^(?:mailto|tel|javascript|data|blob|about):/i;
+/**
+ * Extracts the crawlable same-origin route paths from a page's HTML.
+ * Returned paths are normalized (`/about`, no query, no fragment, no
+ * trailing slash except the root) and deduped.
+ */
+export function extractLinks(html, pageUrl) {
+    // <base href> shifts what relative hrefs resolve against, exactly as the
+    // browser would resolve them.
+    const baseMatch = BASE_PATTERN.exec(html);
+    let base = pageUrl;
+    if (baseMatch) {
+        const href = baseMatch[1] ?? baseMatch[2] ?? "";
+        try {
+            base = new URL(href, pageUrl);
+        }
+        catch {
+            // an unparseable <base> is ignored, like a browser ignores it
+        }
+    }
+    const found = new Set();
+    for (const match of html.matchAll(LINK_PATTERN)) {
+        const href = (match[1] ?? match[2] ?? "").trim();
+        if (!href || NON_PAGE_SCHEME.test(href))
+            continue;
+        const path = normalizeLink(href, base, pageUrl.origin);
+        if (path !== undefined)
+            found.add(path);
+    }
+    return [...found];
+}
+/**
+ * Resolves one href to a normalized same-origin path, or undefined when it
+ * is not a page of this site (foreign origin, unparseable).
+ */
+export function normalizeLink(href, base, origin) {
+    let url;
+    try {
+        url = new URL(href, base);
+    }
+    catch {
+        return undefined;
+    }
+    if (url.origin !== origin)
+        return undefined;
+    return normalizePath(url.pathname);
+}
+/**
+ * One spelling per page: query and fragment never reach here (URL parsing
+ * split them off), the trailing slash is dropped (except the root), and
+ * percent-encoding is left exactly as the URL parser produced it.
+ */
+export function normalizePath(pathname) {
+    if (pathname === "" || pathname === "/")
+        return "/";
+    return pathname.endsWith("/") ? pathname.slice(0, -1) : pathname;
+}

package/dist/output.d.ts

@@ -0,0 +1,7 @@
+/**
+ * Where a route's HTML lands on disk. Static hosts resolve `/about` by
+ * probing `about/index.html` (and some `about.html`) — `autoSubfolderIndex`
+ * picks which convention the output follows. A path that already names a
+ * file (`/sitemap.xml`) is written verbatim.
+ */
+export declare function outputFilename(path: string, autoSubfolderIndex: boolean): string;

package/dist/output.js

@@ -0,0 +1,16 @@
+/**
+ * Where a route's HTML lands on disk. Static hosts resolve `/about` by
+ * probing `about/index.html` (and some `about.html`) — `autoSubfolderIndex`
+ * picks which convention the output follows. A path that already names a
+ * file (`/sitemap.xml`) is written verbatim.
+ */
+export function outputFilename(path, autoSubfolderIndex) {
+    if (path === "/")
+        return "index.html";
+    const trimmed = path.replace(/^\/+/, "");
+    // a final segment with an extension is a file, not a route
+    const lastSegment = trimmed.slice(trimmed.lastIndexOf("/") + 1);
+    if (lastSegment.includes("."))
+        return trimmed;
+    return autoSubfolderIndex ? `${trimmed}/index.html` : `${trimmed}.html`;
+}

package/dist/types.d.ts

@@ -0,0 +1,180 @@
+/**
+ * The engine's whole contract with the app is one function: something
+ * fetch-shaped it can throw `Request`s at. In-process dispatch against a
+ * built server entry and loopback HTTP against a preview server are both
+ * just transports; the crawler never knows which one it is driving.
+ */
+export interface Transport {
+    fetch(request: Request): Promise<Response>;
+    /** Released when the run finishes, success or failure. */
+    close?(): void | Promise<void>;
+}
+/** A page to prerender: a route path, optionally with an explicit output filename. */
+export interface PageEntry {
+    path: string;
+    /**
+     * Output file relative to the output directory. Defaults to the path's
+     * natural mapping (`/` -> `index.html`, `/about` -> `about/index.html`
+     * under `autoSubfolderIndex`, `/sitemap.xml` -> `sitemap.xml`). The
+     * escape hatch for host conventions: `{ path: "/404", filename:
+     * "404.html" }`.
+     */
+    filename?: string;
+    /**
+     * Whether this page's HTML is written to disk, overriding the run-level
+     * `emitPages` policy for this entry. Rendering is unaffected either way:
+     * an unemitted page still executes fully (its links are still crawled,
+     * its captured data still baked) — emission only decides whether the
+     * HTML becomes a static file, which on most hosts SHADOWS live SSR for
+     * that route.
+     */
+    emit?: boolean;
+}
+/**
+ * Seed pages: a list, or a function producing one — the seam route
+ * sources (filesystem routing manifests, CMS queries) feed. Anything
+ * shaped `{ path }` is accepted alongside plain strings.
+ */
+export type PagesSource = Array<string | PageEntry> | (() => Array<string | PageEntry> | Promise<Array<string | PageEntry>>);
+/** A page the engine rendered (and, when `emitted`, wrote). */
+export interface RenderedPage {
+    /** The normalized route path (`/about`), origin and query stripped. */
+    path: string;
+    /**
+     * The pages whose links (or hint headers) led here, as known when this
+     * page rendered; empty for seeds. Provenance for diagnostics — "why was
+     * this route crawled" — and for sitemap tooling.
+     */
+    referrers: string[];
+    /** The page's output file (written only when `emitted`), relative to the output directory. */
+    filename: string;
+    /** Whether the HTML was written to disk (see `emitPages` / `PageEntry.emit`). */
+    emitted: boolean;
+    /** The response the transport answered with (body consumed). */
+    response: Response;
+    /** The rendered HTML. */
+    html: string;
+}
+/** An extra artifact an integration ships alongside the rendered pages. */
+export interface EmittedFile {
+    /** Output path relative to the output directory. */
+    filename: string;
+    contents: string | Uint8Array;
+}
+/**
+ * What the output is for. The distinction every policy downstream keys on:
+ *
+ * - `"static"`: the written files ARE the deployment (SSG). Every rendered
+ *   page is written; anything the crawl did not produce does not exist at
+ *   runtime, so integrations treat gaps as errors.
+ * - `"hybrid"`: a live server is deployed alongside. The crawl is a
+ *   build-time pass (data baking, selected pages); rendered pages are not
+ *   written by default because a static HTML file shadows live SSR of the
+ *   same route on most hosts, and gaps fall back to the server.
+ */
+export type PrerenderMode = "static" | "hybrid";
+/**
+ * The context integrations set up against. `emitFile` is the channel for
+ * artifacts produced during the crawl (payload extraction, captured
+ * server-function results): queued during the run, written with the pages.
+ */
+export interface PrerenderContext {
+    mode: PrerenderMode;
+    origin: string;
+    outDir: string;
+    emitFile(file: EmittedFile): void;
+}
+/** A hook bundle participating in the run without owning any of it. */
+export interface PrerenderIntegration {
+    name: string;
+    /** Before the first page renders. */
+    setup?(context: PrerenderContext): void | Promise<void>;
+    /**
+     * After the last page rendered, before any file is written. Throwing
+     * here fails the run — the place for an integration to verify the crawl
+     * produced everything its runtime half will need.
+     */
+    teardown?(context: PrerenderContext): void | Promise<void>;
+    /**
+     * A module specifier the bundler integration imports for side effects
+     * into the CLIENT build when this integration is active — how an
+     * integration ships runtime behavior (a transport interceptor, a
+     * posture switch) without the app wiring it by hand. Not consumed by the
+     * engine itself; reserved for bundler plugins built on it.
+     */
+    client?: string;
+}
+export interface PrerenderOptions {
+    /** See `PrerenderMode`. Decides the `emitPages` default. @default "static" */
+    mode?: PrerenderMode;
+    /** Seed pages. @default ["/"] */
+    pages?: PagesSource;
+    /**
+     * Extract same-origin links from rendered HTML and prerender them too.
+     * The only way dynamic routes are discovered without explicit seeding.
+     * @default true
+     */
+    crawlLinks?: boolean;
+    /**
+     * Response header a rendered page names additional paths on
+     * (comma-separated) — the route the data lives on announces the routes
+     * built from it. @default "x-prerender"
+     */
+    hintHeader?: string;
+    /** Drops a discovered path before it is fetched. */
+    filter?(path: string): boolean;
+    /** Pages in flight at once. @default 8 */
+    concurrency?: number;
+    /**
+     * Minimum milliseconds between the starts of consecutive requests, across
+     * all workers — a throttle for renders that call rate-limited external
+     * APIs. `concurrency` bounds how many are in flight; `interval` bounds how
+     * fast new ones begin. @default 0
+     */
+    interval?: number;
+    /** Re-fetch attempts for a failed page. @default 2 */
+    retries?: number;
+    /** Milliseconds between attempts. @default 500 */
+    retryDelay?: number;
+    /**
+     * Whether a page that still fails after retries fails the run. Pages
+     * skipped by `failOnError: false` are reported in the result.
+     * @default true
+     */
+    failOnError?: boolean;
+    /** Internal redirect hops followed for one page. @default 5 */
+    maxRedirects?: number;
+    /**
+     * Whether rendered pages are written to disk: a blanket policy, or a
+     * per-path predicate; per-entry `emit` flags override it either way. A
+     * page excluded from emission still renders fully — link discovery and
+     * integration capture (build-time data baking) happen regardless — it
+     * just produces no HTML file. Turn this off (or scope it) when a live
+     * server keeps serving the crawled routes: a written HTML file is served
+     * ahead of SSR by most hosts, freezing the route.
+     * @default true in `static` mode, false in `hybrid`
+     */
+    emitPages?: boolean | ((path: string) => boolean);
+    /** `/about` -> `about/index.html` (true) or `about.html` (false). @default true */
+    autoSubfolderIndex?: boolean;
+    /** Origin requests are minted under. @default "http://localhost" */
+    origin?: string;
+    /** Observes every written page — the seam for sitemaps and post-processing. */
+    onRendered?(page: RenderedPage): void | Promise<void>;
+    integrations?: PrerenderIntegration[];
+}
+/** A page that failed after retries and was left out of the output. */
+export interface SkippedPage {
+    path: string;
+    error: unknown;
+    /** The pages that linked here — where to look for the broken link. */
+    referrers: string[];
+}
+/** What a finished run reports. */
+export interface PrerenderResult {
+    pages: RenderedPage[];
+    /** Extra files integrations emitted. */
+    files: EmittedFile[];
+    /** Paths that failed and were skipped (only with `failOnError: false`). */
+    skipped: SkippedPage[];
+}

package/dist/types.js

@@ -0,0 +1 @@
+export {};

package/dist/vite.d.ts

@@ -0,0 +1,42 @@
+import type { Plugin } from "vite";
+import type { FileRoutePagesOptions } from "./file-routes.ts";
+import type { PrerenderOptions } from "./types.ts";
+export { fileRoutePages, staticRoutePaths } from "./file-routes.ts";
+export type { FileRoutePagesOptions, RouteEntryLike } from "./file-routes.ts";
+export type * from "./types.ts";
+/** The `import.meta.env` key the plugin defines with the build's `PrerenderMode`. */
+export declare const PRERENDER_MODE_ENV = "PRERENDER_MODE";
+export interface PrerenderPluginOptions extends PrerenderOptions {
+    /**
+     * Path (relative to the project root) of the built module exporting the
+     * request handler — `handleRequest`, `fetch`, or `default.fetch`.
+     * Defaults to `server.js` inside the `ssr` environment's output directory.
+     */
+    serverEntry?: string;
+    /**
+     * Seed the crawl with the static pages of the project's
+     * `filesystem-routing` route directory, merged with `pages`. Dynamic
+     * routes (`/posts/:slug`) are still discovered by links — only a render
+     * knows their values.
+     *
+     * `true` (default) applies when the package and `src/routes` exist and
+     * is silently skipped otherwise; pass options to mirror a customized
+     * `fileRoutes({ dir, extensions })` (then a missing package is an error);
+     * `false` disables it.
+     */
+    fileRoutes?: boolean | FileRoutePagesOptions;
+}
+/**
+ * Prerenders the app at build time: crawls the built server handler from
+ * `pages` (default `["/"]`, plus the file-routed static pages, plus every
+ * same-origin link discovered along the way) and writes each page's HTML —
+ * and whatever the integrations emit — into the client output.
+ *
+ * ```ts
+ * import { prerender } from "prerender-crawler/vite";
+ * export default defineConfig({
+ *   plugins: [framework(), prerender({ mode: "static", integrations: [...] })]
+ * });
+ * ```
+ */
+export declare function prerender(options?: PrerenderPluginOptions): Plugin;

package/dist/vite.js

@@ -0,0 +1,140 @@
+var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExtension) || function (path, preserveJsx) {
+    if (typeof path === "string" && /^\.\.?\//.test(path)) {
+        return path.replace(/\.(tsx)$|((?:\.d)?)((?:\.[^./]+?)?)\.([cm]?)ts$/i, function (m, tsx, d, ext, cm) {
+            return tsx ? preserveJsx ? ".jsx" : ".js" : d && (!ext || !cm) ? m : (d + ext + "." + cm.toLowerCase() + "js");
+        });
+    }
+    return path;
+};
+// The Vite plugin: `prerender()` from `prerender-crawler/vite`.
+//
+// Framework-agnostic by construction. It knows three things about the app:
+// that `vite build` produces a client output directory, that some
+// environment's output includes a module exporting a fetch-shaped handler
+// (`handleRequest` or `fetch`: Request in, Response out), and — optionally
+// — that a `filesystem-routing` directory names the static pages. Anything
+// framework-specific rides along as an integration (see
+// `PrerenderIntegration`), the same seam the engine exposes to non-Vite
+// drivers.
+//
+// Responsibilities, all build-only:
+//
+// 1. Orchestration: a `buildApp` hook (declaring one claims the app build;
+//    Vite's build-everything fallback stands down) that builds the
+//    remaining environments, imports the built server handler, and drives
+//    the crawl against it in-process — no HTTP server, no subprocess.
+//    Pages and integration-emitted files land in the client output.
+// 2. Posture: `import.meta.env.PRERENDER_MODE` is defined in every build
+//    environment so runtime code can ask "am I a prerendered build, and of
+//    which kind" — the one bit of client-side knowledge integrations need.
+//    Absent (dev, or a build without this plugin) means "live".
+// 3. Seeding: the static pages of a `filesystem-routing` route directory
+//    seed the crawl automatically, so a page nothing links to still builds.
+import { existsSync } from "node:fs";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+import { runPrerender } from "./crawl.js";
+import { fileRoutePages, hasFileSystemRouting } from "./file-routes.js";
+export { fileRoutePages, staticRoutePaths } from "./file-routes.js";
+/** The `import.meta.env` key the plugin defines with the build's `PrerenderMode`. */
+export const PRERENDER_MODE_ENV = "PRERENDER_MODE";
+/**
+ * Prerenders the app at build time: crawls the built server handler from
+ * `pages` (default `["/"]`, plus the file-routed static pages, plus every
+ * same-origin link discovered along the way) and writes each page's HTML —
+ * and whatever the integrations emit — into the client output.
+ *
+ * ```ts
+ * import { prerender } from "prerender-crawler/vite";
+ * export default defineConfig({
+ *   plugins: [framework(), prerender({ mode: "static", integrations: [...] })]
+ * });
+ * ```
+ */
+export function prerender(options = {}) {
+    const mode = options.mode ?? "static";
+    return {
+        name: "prerender",
+        apply: "build",
+        config() {
+            // Vite merges `import.meta.env.*` defines into the whole-object form
+            // too, so `import.meta.env.PRERENDER_MODE` and `import.meta.env`
+            // destructuring both see it. Every environment gets it: the server
+            // build may legitimately ask as well.
+            return { define: { [`import.meta.env.${PRERENDER_MODE_ENV}`]: JSON.stringify(mode) } };
+        },
+        buildApp: {
+            order: "post",
+            async handler(builder) {
+                // Build whatever is not yet built, in definition order. Frameworks
+                // that build the client first at normal order have already done so
+                // here; declaring this hook suppressed the fallbacks that would
+                // otherwise have built the rest.
+                for (const environment of Object.values(builder.environments)) {
+                    if (!environment.isBuilt)
+                        await builder.build(environment);
+                }
+                const root = builder.config.root;
+                const logger = builder.config.logger;
+                const clientOut = path.resolve(root, builder.environments.client?.config.build.outDir ?? "dist/client");
+                const ssrOut = path.resolve(root, builder.environments.ssr?.config.build.outDir ?? "dist/server");
+                const entry = options.serverEntry
+                    ? path.resolve(root, options.serverEntry)
+                    : path.join(ssrOut, "server.js");
+                const handleRequest = await loadHandler(entry);
+                const routeSeeds = await fileRouteSeeds(root, options.fileRoutes);
+                const { serverEntry: _entry, fileRoutes: _fileRoutes, pages, ...crawl } = options;
+                const result = await runPrerender({
+                    ...crawl,
+                    mode,
+                    pages: async () => [
+                        ...(typeof pages === "function" ? await pages() : (pages ?? ["/"])),
+                        ...routeSeeds
+                    ],
+                    transport: { fetch: request => handleRequest(request) },
+                    outDir: clientOut
+                });
+                const written = result.pages.filter(page => page.emitted).length;
+                const seeded = routeSeeds.length ? `, ${routeSeeds.length} seeded from file routes` : "";
+                logger.info(`[prerender] rendered ${result.pages.length} page(s) (${written} written${seeded}), ` +
+                    `${result.files.length} file(s) emitted -> ${path.relative(root, clientOut)}`);
+                for (const miss of result.skipped) {
+                    logger.warn(`[prerender] skipped ${miss.path}: ${describe(miss.error)}`);
+                }
+            }
+        }
+    };
+}
+async function loadHandler(entry) {
+    let serverModule;
+    try {
+        serverModule = await import(__rewriteRelativeImportExtension(pathToFileURL(entry).href));
+    }
+    catch (cause) {
+        throw new Error(`prerender could not import the built server entry at ${entry}. Prerendering renders ` +
+            `pages through the server build — make sure an SSR build runs (the server build is a ` +
+            `build-time tool here; it need not be deployed) or point \`serverEntry\` at a module ` +
+            `exporting handleRequest/fetch.`, { cause });
+    }
+    const handler = serverModule.handleRequest ??
+        serverModule.fetch ??
+        serverModule.default?.fetch;
+    if (typeof handler !== "function") {
+        throw new Error(`The server entry at ${entry} exports none of handleRequest, fetch, or default.fetch — ` +
+            `prerender needs a Request -> Response handler to render pages through.`);
+    }
+    return handler;
+}
+async function fileRouteSeeds(root, option) {
+    if (option === false)
+        return [];
+    const explicit = typeof option === "object" ? option : undefined;
+    if (!explicit) {
+        // auto mode: only when the project actually uses file routing
+        const dir = path.resolve(root, "src/routes");
+        if (!hasFileSystemRouting(root) || !existsSync(dir))
+            return [];
+    }
+    return fileRoutePages({ root, ...explicit })();
+}
+const describe = (error) => (error instanceof Error ? error.message : String(error));

package/package.json

@@ -0,0 +1,67 @@
+{
+  "name": "prerender-crawler",
+  "version": "0.1.0",
+  "description": "Framework-agnostic build-time prerendering. Point it at anything fetch-shaped (Request in, Response out) and it crawls the site into static HTML: seeds, link discovery, header hints, redirects, retries, throttling, and an integration seam for build-time data capture. Ships a Vite plugin at prerender-crawler/vite.",
+  "license": "MIT",
+  "author": "Ryan Carniato",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/solidjs/prerender-crawler",
+    "directory": "packages/crawler"
+  },
+  "homepage": "https://github.com/solidjs/prerender-crawler/tree/main/packages/crawler#readme",
+  "bugs": "https://github.com/solidjs/prerender-crawler/issues",
+  "keywords": [
+    "prerender",
+    "prerendering",
+    "static site generation",
+    "ssg",
+    "crawler",
+    "build-time",
+    "vite",
+    "vite-plugin"
+  ],
+  "type": "module",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "default": "./dist/index.js"
+    },
+    "./vite": {
+      "types": "./dist/vite.d.ts",
+      "default": "./dist/vite.js"
+    }
+  },
+  "files": [
+    "dist",
+    "LICENSE",
+    "README.md"
+  ],
+  "engines": {
+    "node": ">=20"
+  },
+  "peerDependencies": {
+    "filesystem-routing": ">=0.2.0",
+    "vite": "^7.0.0 || ^8.0.0"
+  },
+  "peerDependenciesMeta": {
+    "filesystem-routing": {
+      "optional": true
+    },
+    "vite": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "filesystem-routing": "0.2.1",
+    "typescript": "^5.8.0",
+    "vite": "^8.0.0",
+    "vitest": "^4.0.0"
+  },
+  "scripts": {
+    "build": "rm -rf dist && tsc",
+    "test": "vitest run",
+    "test:watch": "vitest"
+  }
+}