@pitlane/dev 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -1,5 +1,50 @@
 # @pitlane/dev
 
+## 0.5.0
+
+Build-time prerendering.
+
+- `remix({ prerender })` renders paths to static HTML during `vite build` and
+  writes them into the client output, so a host answers those URLs and the
+  server never sees them. The API mirrors React Router's: `true` for every
+  static path in the route map, an array for an explicit list, a function
+  receiving `getStaticPaths()` for a list that mixes static and dynamic paths,
+  or an object adding `concurrency`. `spider` is one option beyond that set —
+  it follows the links each rendered page contains, which suits a site whose
+  pages all reach each other.
+- There is no second rendering path. The build sends a `Request` through the
+  same fetch handler production runs, after both environments are built and the
+  assets manifest is written, so the HTML on disk names real hashed chunks
+  rather than dev URLs.
+- `getStaticPaths()` reads the `routes` named export of the built server entry.
+  A Remix 3 router exposes no route table, but the route map it is built from
+  is an ordinary object, so exporting it is all the app has to do. A build that
+  asks for static paths without that export fails with a message saying so.
+- A path that answers with a redirect is logged and skipped rather than failing
+  the build. `prerender: true` asks for every static path in the route map, and
+  a `/` that points at the real landing path is an ordinary thing to find in
+  there; there is no document to write for one, and the app still answers it at
+  runtime. Under `spider` the redirect is followed instead, because that is
+  what following links means. Any other failing response still stops the build,
+  since a listed path that 404s is a stale list and a spidered one is a dead
+  internal link.
+- Bundles built for another runtime prerender too, with no extra
+  configuration. Node cannot import a Workers bundle, so when the import
+  fails the build starts the project's own preview server and renders through
+  that: `@cloudflare/vite-plugin` boots workerd with the app's real bindings,
+  and any platform plugin contributing a preview server works the same way. On
+  that path the route map is read from the module the server entry gets it
+  from, since the bundle holding the export is the thing that will not load.
+- Prerendered output is written relative to Vite's `base`: an app whose routes
+  live under `/repo/` still writes `blog/index.html`, because the host mounts
+  the client directory at the base.
+- `remix({ server: false, prerender })` throws. Prerendering renders through the
+  server entry, and SPA mode builds no server.
+- The crawler is a new package,
+  [`@pitlane/crawler`](https://pitlane.tools/package/crawler/), installable on
+  its own. It brings back the `crawl()` API from
+  [remix-run/remix#11150](https://github.com/remix-run/remix/pull/11150).
+
 ## 0.4.0
 
 SPA mode.

package/README.md

@@ -115,6 +115,7 @@ Leave it unguarded: in a production build the specifier resolves to a component
 ```ts
 remix({
     server: true, // default — false selects SPA mode
+    prerender: undefined, // default — true, a path array, a function, or a config object
     clientEntry: "app/entry.browser", // default — false disables the client build
     serverEntry: "app/entry.server", // default
     serverEnvironments: ["ssr"], // default
@@ -122,13 +123,68 @@ remix({
 });
 ```
 
-| Option               | Type              | Default               | Purpose                                                                                                                                                           |
-| -------------------- | ----------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `server`             | `boolean`         | `true`                | Whether the app has a server. Pass `false` for [SPA mode](#spa-mode), which ignores every option below.                                                           |
-| `clientEntry`        | `string \| false` | `"app/entry.browser"` | Client entry module. Pass `false` for fully server-rendered apps with no hydration.                                                                               |
-| `serverEntry`        | `string`          | `"app/entry.server"`  | Server entry module, built as `dist/ssr/index.js`.                                                                                                                |
-| `serverEnvironments` | `string[]`        | `["ssr"]`             | Environment names the `clientEntry()` transform treats as "server".                                                                                               |
-| `serverHandler`      | `boolean`         | `true`                | Serve dev requests through your server entry. Set `false` when `@cloudflare/vite-plugin`, `@netlify/vite-plugin`, or `nitro/vite` owns dev-time request handling. |
+| Option               | Type                               | Default               | Purpose                                                                                                                                                           |
+| -------------------- | ---------------------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `server`             | `boolean`                          | `true`                | Whether the app has a server. Pass `false` for [SPA mode](#spa-mode), which ignores every option below.                                                           |
+| `prerender`          | `boolean \| string[] \| fn \| obj` | none                  | Render paths to static HTML at build time. See [Prerendering](#prerendering).                                                                                     |
+| `clientEntry`        | `string \| false`                  | `"app/entry.browser"` | Client entry module. Pass `false` for fully server-rendered apps with no hydration.                                                                               |
+| `serverEntry`        | `string`                           | `"app/entry.server"`  | Server entry module, built as `dist/ssr/index.js`.                                                                                                                |
+| `serverEnvironments` | `string[]`                         | `["ssr"]`             | Environment names the `clientEntry()` transform treats as "server".                                                                                               |
+| `serverHandler`      | `boolean`                          | `true`                | Serve dev requests through your server entry. Set `false` when `@cloudflare/vite-plugin`, `@netlify/vite-plugin`, or `nitro/vite` owns dev-time request handling. |
+
+## Prerendering
+
+`prerender` renders paths to static HTML during `vite build` and writes them
+into the client output, so a CDN answers those URLs and the server never sees
+them. There is no second rendering path: the build sends a `Request` through
+the same fetch handler production runs.
+
+```ts
+remix({ prerender: ["/", "/blog", "/blog/hello-world"] });
+```
+
+`true` prerenders every static path in the app's route map, which the server
+entry exports alongside its handler:
+
+```ts
+// app/entry.server.tsx
+export { routes } from "./routes.ts";
+export default router;
+```
+
+A function computes the list, and gets `getStaticPaths()` for the route-map
+half of it:
+
+```ts
+remix({
+    async prerender({ getStaticPaths }) {
+        let slugs = await getPostSlugsFromCMS();
+        return [...getStaticPaths(), ...slugs.map(slug => `/blog/${slug}`)];
+    },
+});
+```
+
+The object form adds `concurrency`, and `spider` for following the links each
+rendered page contains:
+
+```ts
+remix({ prerender: { paths: ["/"], spider: true, concurrency: 4 } });
+```
+
+Each path lands at `<path>/index.html` under the client output. Rendering runs
+after both builds and the assets manifest, so the HTML names real hashed
+chunks. Unsupported with `server: false`, which builds no server to render
+with.
+
+A bundle Node cannot import renders anyway: `@cloudflare/vite-plugin` and
+friends already contribute a preview server, so when the import fails the
+build starts that server and renders through it, inside the runtime that will
+serve the pages. Nothing extra to configure.
+
+Full details in the [prerendering guide](https://pitlane.tools/guides/prerendering);
+the crawler underneath is [`@pitlane/crawler`](https://pitlane.tools/package/crawler/),
+and the [crawling guide](https://pitlane.tools/guides/crawler) covers using it
+on its own.
 
 ## SPA mode
 

package/dist/index.d.mts

@@ -1,4 +1,54 @@
 import { PluginOption } from "vite";
+//#region src/prerender.d.ts
+/**
+ * What a `prerender` function receives.
+ */
+interface PrerenderContext {
+  /**
+   * Every path the app's route map can serve without params — `/` and
+   * `/blog`, but not `/blog/:slug`, whose values live outside the route map.
+   *
+   * Reads the `routes` named export of the built server entry. Throws when
+   * the server entry does not export one, since there is nothing else to
+   * enumerate.
+   */
+  getStaticPaths(): string[];
+}
+/**
+ * A function that decides which paths to prerender, for path sets that need
+ * async work — a CMS query, a filesystem scan, a database read.
+ */
+type PrerenderPaths = (context: PrerenderContext) => string[] | Promise<string[]> | Iterable<string> | Promise<Iterable<string>>;
+/**
+ * The paths to prerender: `true` for every static path in the route map, an
+ * explicit list, or a function that computes one.
+ */
+type PrerenderPathsOption = boolean | string[] | PrerenderPaths;
+interface PrerenderConfig {
+  /**
+   * The paths to prerender.
+   *
+   * @default true
+   */
+  paths?: PrerenderPathsOption;
+  /**
+   * How many paths to render at once. Rendering is CPU-bound in-process, so
+   * the useful value depends on how much of a render waits on I/O.
+   *
+   * @default 1
+   */
+  concurrency?: number;
+  /**
+   * Also follow the links each rendered page contains, and prerender those
+   * too. Turns the path list into a set of starting points rather than the
+   * complete answer, which suits a site whose pages all link to each other.
+   *
+   * @default false
+   */
+  spider?: boolean;
+}
+type PrerenderOption = PrerenderPathsOption | PrerenderConfig;
+//#endregion
 //#region src/index.d.ts
 interface RemixPluginOptions {
   /**
@@ -51,6 +101,22 @@ interface RemixPluginOptions {
    * @default true
    */
   serverHandler?: boolean;
+  /**
+   * Render paths to static HTML at build time and write them into the client
+   * output, so a host can serve the file and skip the server entirely.
+   *
+   * `true` prerenders every static path in the app's route map, which the
+   * server entry must export as `routes`. An array prerenders exactly those
+   * paths. A function computes them, and receives `getStaticPaths()` for the
+   * route-map half of a list that also has dynamic paths in it. The object
+   * form adds `concurrency` and `spider`.
+   *
+   * Build-time only, and unsupported with `server: false`: prerendering
+   * renders through the server entry, and there is none.
+   *
+   * @default undefined
+   */
+  prerender?: PrerenderOption;
 }
 /**
  * Wires Remix 3 into a Vite or Vite+ project: multi-environment build
@@ -72,4 +138,4 @@ interface RemixPluginOptions {
  */
 declare function remix(options?: RemixPluginOptions): PluginOption;
 //#endregion
-export { RemixPluginOptions, remix };
+export { type PrerenderConfig, type PrerenderContext, type PrerenderOption, type PrerenderPaths, type PrerenderPathsOption, RemixPluginOptions, remix };

package/dist/index.mjs

@@ -2,9 +2,266 @@ import fullstack from "@hiogawa/vite-plugin-fullstack";
 import * as fs from "node:fs";
 import * as path from "node:path";
 import * as url from "node:url";
-import MagicString from "magic-string";
+import { crawl, staticPaths } from "@pitlane/crawler";
+import * as fs$1 from "node:fs/promises";
+import { createServer, preview } from "vite";
 import { parseSync } from "oxc-parser";
+import MagicString from "magic-string";
 import { transformComponentsForBrowser, transformComponentsForServer } from "remix/ui-hmr";
+//#region src/prerender-target.ts
+/**
+* Opens the built application for prerendering.
+*
+* The fast path imports the server bundle and calls its fetch handler in this
+* process: no socket, no second copy of the app, and the route map comes along
+* as a named export. A bundle built for another runtime cannot be imported
+* here — a Workers bundle opens with `import { env } from "cloudflare:workers"`
+* — so the fallback boots the project's own `vite preview` server and
+* dispatches over loopback. The platform's Vite plugin owns that server, so
+* pages still render inside the runtime that will serve them, and nothing in
+* the app has to name the runtime a second time.
+*
+* @param ssrConfig The resolved config of the `ssr` environment.
+*/
+async function openTarget(ssrConfig) {
+	let entry = await importServerEntry(ssrConfig);
+	if (!entry) return openPreviewTarget(ssrConfig);
+	let handler = entry.default;
+	if (!handler || typeof handler.fetch !== "function") throw new Error(`[@pitlane/dev] prerender needs ${serverEntryPath(ssrConfig)} to default-export a fetch handler (an object with fetch(request: Request)), e.g. \`export default router\`.`);
+	return {
+		fetch: (request) => handler.fetch(request),
+		routes: entry.routes,
+		viaPreviewServer: false,
+		close: () => Promise.resolve()
+	};
+}
+function serverEntryPath(ssrConfig) {
+	return path.resolve(ssrConfig.root, ssrConfig.build.outDir, "index.js");
+}
+/**
+* Imports the built server entry, or reports that this process cannot.
+*
+* Dynamic by necessity: the specifier is a runtime-computed path into the
+* app's own build output. The bundle's modification time rides along in the
+* URL, because Node's module registry is keyed on the specifier — without it a
+* second build in the same process (a watch rebuild, or one test after
+* another) would prerender through the first build's handler.
+*/
+async function importServerEntry(ssrConfig) {
+	let entryPath = serverEntryPath(ssrConfig);
+	let stats = await fs$1.stat(entryPath).catch(() => void 0);
+	if (!stats) throw new Error(`[@pitlane/dev] prerender expected the server build at ${entryPath}, and found nothing there.`);
+	try {
+		let specifier = url.pathToFileURL(entryPath);
+		specifier.searchParams.set("t", String(stats.mtimeMs));
+		return await import(
+			/* @vite-ignore */
+			specifier.href
+);
+	} catch {
+		return;
+	}
+}
+/**
+* Boots the project's preview server and dispatches through it.
+*
+* `preview()` re-reads the project's config, so every plugin that contributes
+* a preview server gets to — `@cloudflare/vite-plugin` boots workerd with the
+* app's real bindings, and the same holds for any other platform plugin.
+*/
+async function openPreviewTarget(ssrConfig) {
+	let server = await preview({
+		root: ssrConfig.root,
+		configFile: ssrConfig.configFile,
+		logLevel: "silent",
+		preview: { host: "127.0.0.1" }
+	});
+	let local = server.resolvedUrls?.local[0];
+	if (!local) {
+		await server.close();
+		throw new Error(`[@pitlane/dev] prerender started a preview server to render ${serverEntryPath(ssrConfig)} through, and it is not listening on any URL.`);
+	}
+	let origin = new URL(local).origin;
+	return {
+		fetch(request) {
+			let { pathname, search } = new URL(request.url);
+			return fetch(new URL(pathname + search, origin), { redirect: "manual" });
+		},
+		viaPreviewServer: true,
+		close: () => server.close()
+	};
+}
+//#endregion
+//#region src/route-map.ts
+/**
+* Loads the app's route map from source.
+*
+* Prerendering normally reads the `routes` export straight off the built
+* server bundle, which is the artifact production runs. A bundle built for
+* another runtime cannot be imported here, so this takes the long way round:
+* find the module the server entry gets `routes` from, and load that one. A
+* route map is a plain object of patterns, so it runs anywhere — the platform
+* imports that make the bundle unloadable live in the entry, not in it.
+*
+* @param ssrConfig The resolved config of the `ssr` environment.
+* @param serverEntry The plugin's `serverEntry` option.
+* @returns The route map, or `undefined` when the entry does not export one.
+*/
+async function loadRouteMap(ssrConfig, serverEntry) {
+	let server = await createServer({
+		configFile: false,
+		root: ssrConfig.root,
+		logLevel: "silent",
+		server: {
+			middlewareMode: true,
+			watch: null
+		},
+		optimizeDeps: { noDiscovery: true },
+		resolve: {
+			alias: ssrConfig.resolve.alias,
+			extensions: ssrConfig.resolve.extensions
+		}
+	});
+	try {
+		let entryId = await resolve(server, path.resolve(ssrConfig.root, serverEntry));
+		if (!entryId) return void 0;
+		let specifier = routeMapSpecifier(await fs$1.readFile(entryId, "utf8"), entryId);
+		if (!specifier) return void 0;
+		let routesId = await resolve(server, specifier, entryId);
+		if (!routesId) return void 0;
+		return (await server.ssrLoadModule(routesId)).routes;
+	} finally {
+		await server.close();
+	}
+}
+async function resolve(server, source, importer) {
+	return (await server.environments.ssr.pluginContainer.resolveId(source, importer))?.id;
+}
+/**
+* The module a server entry gets its `routes` binding from, in either of the
+* two shapes that reads naturally:
+*
+* ```ts
+* export { routes } from "./routes.ts";
+* // or
+* import { routes } from "./routes.ts";
+* export { routes };
+* ```
+*
+* A route map declared inline in the entry has no other module to point at,
+* and yields `undefined`.
+*/
+function routeMapSpecifier(code, filename) {
+	let program = parseSync(filename, code).program;
+	let local;
+	for (let node of program.body) {
+		if (node.type !== "ExportNamedDeclaration") continue;
+		for (let specifier of node.specifiers) {
+			if (moduleExportName(specifier.exported) !== "routes") continue;
+			if (node.source) return node.source.value;
+			local = moduleExportName(specifier.local);
+		}
+	}
+	if (!local) return void 0;
+	for (let node of program.body) {
+		if (node.type !== "ImportDeclaration") continue;
+		for (let specifier of node.specifiers ?? []) if (specifier.local.name === local) return node.source.value;
+	}
+}
+/** An exported or imported name, which the grammar allows to be a string. */
+function moduleExportName(node) {
+	return node.type === "Identifier" ? node.name ?? "" : node.value ?? "";
+}
+//#endregion
+//#region src/prerender.ts
+function isPrerenderConfig(option) {
+	return typeof option === "object" && !Array.isArray(option);
+}
+/**
+* Renders paths to static HTML at build time by dispatching requests through
+* the app's own fetch handler, then writing each response into the client
+* output. A host serves those files directly and the runtime server never sees
+* the request.
+*
+* Runs after both environments are built and the assets manifest is written,
+* so the server entry resolves real hashed client asset URLs — the HTML on
+* disk is the same HTML the runtime server would produce.
+*
+* @param builder The Vite builder mid-`buildApp`.
+* @param option The plugin's `prerender` option.
+* @param serverEntry The plugin's `serverEntry` option.
+* @returns The files written and the paths that redirected instead.
+*/
+async function prerender(builder, option, serverEntry) {
+	let { paths = true, concurrency = 1, spider = false } = isPrerenderConfig(option) ? option : { paths: option };
+	let redirected = [];
+	if (paths === false) return {
+		written: [],
+		redirected
+	};
+	let ssrConfig = builder.environments.ssr?.config;
+	let clientConfig = builder.environments.client?.config ?? ssrConfig;
+	if (!ssrConfig || !clientConfig) throw new Error("[@pitlane/dev] prerender needs both an ssr and a client environment.");
+	let target = await openTarget(ssrConfig);
+	let outDir = path.resolve(clientConfig.root, clientConfig.build.outDir);
+	let written = [];
+	try {
+		let resolved = await resolvePaths(paths, target.routes ?? (Array.isArray(paths) ? void 0 : await loadRouteMap(ssrConfig, serverEntry)), ssrConfig);
+		if (resolved.length === 0) return {
+			written,
+			redirected
+		};
+		for await (let result of crawl(target, {
+			paths: resolved,
+			spider,
+			assets: false,
+			concurrency,
+			onRedirect: (pathname, location) => redirected.push({
+				pathname,
+				location
+			})
+		})) written.push(await writeResult(result, outDir, clientConfig.base));
+	} catch (error) {
+		if (!target.viaPreviewServer) throw error;
+		throw new Error(`[@pitlane/dev] prerender rendered through this project's preview server, because ${serverEntryPath(ssrConfig)} is a bundle built for another runtime that Node cannot import. The preview server has to answer the paths being prerendered, which is the platform plugin's job (\`@cloudflare/vite-plugin\` and the like).`, { cause: error });
+	} finally {
+		await target.close();
+	}
+	return {
+		written,
+		redirected
+	};
+}
+async function resolvePaths(paths, routes, ssrConfig) {
+	let getStaticPaths = () => {
+		if (!routes) throw new Error(`[@pitlane/dev] prerender needs ${serverEntryPath(ssrConfig)} to export its route map to enumerate static paths: \`export { routes } from "./routes.ts"\` in your server entry. Pass an explicit path array instead if the app has no route map.`);
+		return staticPaths(routes);
+	};
+	if (paths === true) return getStaticPaths();
+	if (Array.isArray(paths)) return [...paths];
+	return [...await paths({ getStaticPaths })];
+}
+/**
+* Writes one crawl result under the client output directory, returning the
+* path it wrote relative to that directory.
+*
+* `base` is stripped from the output path: a project deployed at `/repo/`
+* routes on `/repo/blog` but the file still belongs at `blog/index.html`,
+* because the host mounts the whole directory at the base.
+*/
+async function writeResult(result, outDir, base) {
+	let relative = stripBase(result.filepath, base).replace(/^\/+/, "");
+	let outputPath = path.join(outDir, relative);
+	await fs$1.mkdir(path.dirname(outputPath), { recursive: true });
+	await fs$1.writeFile(outputPath, new Uint8Array(await result.response.arrayBuffer()));
+	return relative;
+}
+function stripBase(filepath, base) {
+	if (base === "/" || !base) return filepath;
+	let prefix = base.endsWith("/") ? base.slice(0, -1) : base;
+	return filepath.startsWith(`${prefix}/`) ? filepath.slice(prefix.length) : filepath;
+}
+//#endregion
 //#region src/build.ts
 function isFullstackBuilder(builder) {
 	return "writeAssetsManifest" in builder && typeof builder.writeAssetsManifest === "function";
@@ -159,7 +416,7 @@ function ensureAssetsManifest() {
 * client: the client build resolves `?assets=ssr` against the SSR manifest,
 * so the order is load-bearing.
 */
-function build({ clientEntry, serverEntry }) {
+function build({ clientEntry, serverEntry, prerender: paths }) {
 	let hasClientEntry = clientEntry !== false;
 	return {
 		name: "pitlane-remix-build",
@@ -176,6 +433,14 @@ function build({ clientEntry, serverEntry }) {
 			if (hasClientEntry) await builder.build(builder.environments.client);
 			if (isFullstackBuilder(builder)) await builder.writeAssetsManifest();
 			ensureAssetsManifest();
+			if (paths !== void 0) {
+				let { written, redirected } = await prerender(builder, paths, serverEntry);
+				for (let file of written) this.info(`prerendered ${file}`);
+				for (let { pathname, location } of redirected) {
+					let target = location ?? "an unnamed location";
+					this.info(`skipped ${pathname} (redirects to ${target})`);
+				}
+			}
 		},
 		config(userConfig) {
 			let environments = userConfig.environments;
@@ -532,7 +797,7 @@ function isFetchHandler(value) {
 * plugin skips itself so the platform plugin's preview can take over. That
 * failure → skip contract is documented behavior, not an accident.
 */
-function preview() {
+function preview$1() {
 	return {
 		name: "pitlane-remix-preview-server",
 		async configurePreviewServer(server) {
@@ -649,8 +914,11 @@ function findClientEntryCalls(program) {
 * Deno.
 */
 function remix(options = {}) {
-	let { clientEntry = "app/entry.browser", server = true, serverEntry = "app/entry.server", serverEnvironments = ["ssr"], serverHandler = true } = options;
-	if (!server) return spa();
+	let { clientEntry = "app/entry.browser", server = true, serverEntry = "app/entry.server", serverEnvironments = ["ssr"], serverHandler = true, prerender } = options;
+	if (!server) {
+		if (prerender !== void 0) throw new Error("[@pitlane/dev] remix({ server: false, prerender }) is not supported: prerendering renders through the server entry, and `server: false` builds no server. Drop `server: false` to prerender, or drop `prerender` to stay a SPA.");
+		return spa();
+	}
 	let serverEnvironmentSet = new Set(serverEnvironments);
 	return [
 		fullstack({
@@ -660,10 +928,11 @@ function remix(options = {}) {
 		buildCompat(),
 		build({
 			clientEntry,
-			serverEntry
+			serverEntry,
+			prerender
 		}),
 		runtimeInline(),
-		preview(),
+		preview$1(),
 		suppressAbortErrors(),
 		normalizeWriteHead(),
 		componentHmr(serverEnvironmentSet),

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "@pitlane/dev",
-    "version": "0.4.0",
-    "description": "remix() — the Remix 3 Vite plugin: build orchestration, clientEntry() hydration transform, dev server with component and server-data HMR, and preview for any Vite or Vite+ project.",
+    "version": "0.5.0",
+    "description": "remix() — the Remix 3 Vite plugin: build orchestration, clientEntry() hydration transform, dev server with component and server-data HMR, SPA mode, build-time prerendering, and preview for any Vite or Vite+ project.",
     "keywords": [
         "pitlane",
         "remix",
@@ -43,6 +43,7 @@
     },
     "dependencies": {
         "@hiogawa/vite-plugin-fullstack": "0.0.11",
+        "@pitlane/crawler": "workspace:^",
         "magic-string": "^0.30.21",
         "oxc-parser": "^0.141.0"
     },