@africanpilot/next-snapshot 0.1.0 → 0.2.0

package/README.md

@@ -86,7 +86,8 @@ A config is an ES module; relative paths resolve against it.
 |---|---|---|
 | `app.cwd`, `app.start`, `app.build`, `app.port` | — | How to run the app. `{port}` is substituted into `start`. `build` runs when there is no `.next/BUILD_ID`, or with `--build`. Omit `app` and set `url` to capture a server you started yourself. |
 | `url` | `http://localhost:{port}` | Origin to capture. `localhost`, because Next builds redirect URLs on it and a session cookie set on `127.0.0.1` is not sent to `localhost`. |
-| `aliases` | loopback spellings | Other origins that are the same app (a canonical host it redirects to); URLs on them are treated as the app's own. `localhost`, `127.0.0.1` and `[::1]` are always aliases of each other. |
+| `aliases` | loopback spellings | Other origins that are the same app (a canonical host it redirects to); URLs on them are treated as the app's own. `localhost`, `127.0.0.1`, `[::1]` and `0.0.0.0` on the same port are always aliases of each other. |
+| `docker` | — | `{ container, staticPath }` — the app runs in a container, so copy its build output here with `docker cp` before bundling. `staticPath` defaults to `/app/.next/static`. See [Docker](#an-app-running-in-docker). |
 | `out` | `./<name>.html` | Output file. The capture directory, bundle report, verify report and screenshots sit next to it. |
 | `title` | — | Title shown while the file opens. |
 | `start` | `/` | Page to open when the file has no hash. |
@@ -96,7 +97,7 @@ A config is an ES module; relative paths resolve against it.
 | `variants` | one | `[{ id, label, login({context, request, origin}) }]`. One crawl per variant; the file can switch between them. |
 | `defaultVariant` | first | |
 | `explore.selects` | `true` | Try each option of each visible `<select>` once per page path. |
-| `explore.tabs` | `true` | Click each tab-like control once per page path: `[role=tab]`, and "button bars" — an element whose children are two or more buttons and nothing else. A URL the click writes (router push/replace, or a bare `history.replaceState`) is captured. |
+| `explore.tabs` | `true` | Click each tab-like control and capture the URL it writes (router push/replace, or a bare `history.replaceState`). Candidates: `[role=tab]`, and "button bars" — an element whose children are two or more buttons and nothing else. `true` clicks each once per page **path**; `"url"` clicks each once per captured **page**, so a route whose pages differ by query (`?programme=…`) gets every tab for every one of them. `"url"` multiplies that route's pages by the number of tabs — use it when clicking through the app moves between a tab and a query at the same time. |
 | `explore.click` | `[]` | Extra CSS selectors to click the same way. |
 | `explore.denyText` | sign out, delete, approve, submit, save… | Controls whose label matches are never clicked. Writes are blocked at the network anyway; this protects the session and client-side state. |
 | `explore.custom` | — | `async ({page, key, variant, discover}) => {}` for app-specific discovery (clicking tabs that change the URL, etc). |
@@ -105,9 +106,66 @@ A config is an ES module; relative paths resolve against it.
 | `offline.missingLinks` | `"show"` | Links to pages the snapshot does not hold: `"show"` (they open a "not in snapshot" page), `"disable"` (dimmed, not clickable) or `"hide"`. With `include` narrowing the crawl, `"hide"` removes the nav entries for everything left out. |
 | `offline.badge` | `"bottom-right"` | The "Offline snapshot" pill; `false` to hide. |
 | `offline.switcher` | `true` | A variant `<select>` in the badge. |
+| `compress` | `"gzip"` | How pages are packed. `"gzip"` stores each page on its own, decoded by the browser itself. `"zstd"` sorts pages by route, packs them into clusters of `clusterBytes` and compresses each cluster as one stream, inlining an 8 KB decoder — far smaller for an app with many similar pages. See [Size](#size-and-compression). |
+| `clusterBytes` | 4 MB | Raw bytes of pages per cluster, with `compress: "zstd"`. Bigger is smaller, but the first page of each cluster takes longer to open. |
 | `includeStatic` | `true` | Also embed every file under `.next/static`, so lazily-loaded chunks the crawl never triggered are present. |
 | `viewport`, `locale`, `timezoneId`, `browser` | | Passed to Chrome. `browser.executablePath` if Chrome is not installed. |
 
+## Size and compression
+
+Pages of an app repeat each other: the same layout, nav and table shell, over
+and over. By default each page is gzipped on its own, which cannot exploit that
+— gzip looks only 32 KB back, and a page's near-twin is further away than that.
+For a handful of pages this costs nothing worth fixing.
+
+For an app with hundreds of pages, `compress: "zstd"` sorts pages by route,
+packs them into clusters and compresses each cluster as one stream, so the
+repetition is paid for once:
+
+```js
+export default {
+  // …
+  compress: "zstd",
+  clusterBytes: 4 * 1024 * 1024,   // the default
+};
+```
+
+On a 260-page report app, measured: **12.4 MB → about 2 MB**. The cost is an
+8 KB decoder inlined in the file, and tens of milliseconds to open the first
+page of a cluster; pages in an already-decoded cluster are free. Assets stay
+per-page gzip either way, since they are wanted all at once at startup and are
+mostly already-compressed formats.
+
+Why not brotli, which is smaller still: Chrome cannot decompress brotli from
+JavaScript (`DecompressionStream` has no brotli there, and no zstd anywhere), so
+it would mean inlining a 208 KB decoder to save about 10%.
+
+## An app running in Docker
+
+The tool drives Chrome on your machine, so it reaches the container the same way
+your browser does. Start the app with its port published — `docker run -p
+3000:3000 …`, or `ports: ["3000:3000"]` in compose — and point `url` at it:
+
+```js
+export default {
+  name: "my-app",
+  url: "http://localhost:3000",
+  docker: { container: "my-app" },   // `docker ps` shows the name
+  seeds: ["/"],
+};
+```
+
+`docker` copies the build output out of the container with `docker cp`, so
+chunks that only load later — a modal, a menu — are in the file too. Without it
+the snapshot holds only what the crawl happened to load. If your image puts the
+app somewhere other than `/app`, set `staticPath` to match. A failed copy is a
+warning, not an error: the capture continues without it.
+
+Redirects to `0.0.0.0` need nothing extra. Next's images set
+`HOSTNAME=0.0.0.0`, and an app that builds absolute URLs from it redirects
+there; `0.0.0.0` on the same port is already an alias of the origin. For any
+other host it redirects to, add it to `aliases`.
+
 ## What it cannot do
 
 - **Anything not captured is not there.** A URL no link, prefetch, select or tab
@@ -117,7 +175,9 @@ A config is an ES module; relative paths resolve against it.
   remembered, and returning to it re-serves the page it came from at that URL.
 - **Exploration is one option at a time.** Each select option and each tab is
   tried once per page path, from the first URL of that path the crawl reached —
-  not every combination. Seed the combinations you need.
+  not every combination. `explore.tabs: "url"` covers the common case (every
+  tab of every page of a route); for anything else, seed the combinations you
+  need. A URL that was never captured shows a "not in this snapshot" page.
 - **Writes.** Forms and fetches that POST/PUT/DELETE are refused unless an
   `offline.post` handler emulates them. Server Actions fail the same way.
 - **Soft navigation.** Every navigation is a full page boot, so in-memory client

package/lib/bundle.mjs

@@ -6,12 +6,25 @@
 //         real URL and a placeholder for the frame shim go first in <head>.
 //   (The transformations themselves live in rewrite.mjs.)
 //
-// Every body is gzipped and base64'd into its own inert <script type=text/plain>.
-// The runtime (runtime/shell.js) decodes assets once into blob: URLs, replaces
-// tokens with them, and decodes a page only when it is navigated to.
+// Every body is then packed into the file as base64 in an inert
+// <script type="text/plain">, one of two ways:
+//
+//   compress: "gzip"  each body gzipped on its own. The browser decodes it
+//                     natively, and the file needs no decoder of its own.
+//   compress: "zstd"  pages are sorted by route and packed into clusters of
+//                     `clusterBytes`, each compressed as one zstd stream, with
+//                     a small decoder inlined. Pages of an app repeat each
+//                     other heavily and gzip's window is only 32KB, so a page
+//                     cannot see its near-twin; a cluster can. Assets stay
+//                     per-body gzip: they are wanted all at once at startup,
+//                     and most are already-compressed formats.
+//
+// The runtime (runtime/shell.js) decodes an asset once into a blob: URL, and a
+// page only when it is navigated to — a cluster at a time, cached.
 
 import crypto from "node:crypto";
 import fs from "node:fs/promises";
+import { createRequire } from "node:module";
 import path from "node:path";
 import { fileURLToPath } from "node:url";
 import zlib from "node:zlib";
@@ -20,6 +33,7 @@ import { urlKey } from "./key.js";
 import { createRewriter, escHTML, serialisePost, virtualiseLocation } from "./rewrite.mjs";
 
 const HERE = path.dirname(fileURLToPath(import.meta.url));
+const require = createRequire(import.meta.url);
 
 // Nothing in the file may reach the network. Anything that tries is refused by
 // the browser and shows up as a CSP violation, which `verify` reports.
@@ -41,6 +55,12 @@ export async function bundle(cfg, log) {
   const origin = M.origin;
   const readBody = (sha) => fs.readFile(path.join(cfg.captureDir, "bodies", sha));
   const t0 = Date.now();
+  const zstd = cfg.compress === "zstd";
+  if (zstd && typeof zlib.zstdCompressSync !== "function") {
+    throw new Error(
+      `compress: "zstd" needs a Node built with zstd (zlib.zstdCompressSync); this is ${process.version}. Upgrade Node, or use compress: "gzip".`,
+    );
+  }
 
   // --- asset table ---------------------------------------------------------
   const assets = [];
@@ -74,18 +94,17 @@ export async function bundle(cfg, log) {
   });
 
   // --- bodies ---------------------------------------------------------------
-  const out = []; // id -> base64(gzip)
-  const outIndex = new Map(); // content hash -> id
-  const sizes = []; // {what, raw, packed}
-  function emit(buf, what) {
+  // Collected first, compressed at the end: how they are packed depends on all
+  // of them. Identical content is stored once, whatever refers to it.
+  const bodies = []; // id -> { buf, what, page }
+  const bodyIndex = new Map(); // content hash -> id
+  function emit(buf, what, page = false) {
     const h = crypto.createHash("sha256").update(buf).digest("hex");
-    let id = outIndex.get(h);
+    let id = bodyIndex.get(h);
     if (id == null) {
-      const packed = zlib.gzipSync(buf, { level: 9 }).toString("base64");
-      id = out.length;
-      out.push(packed);
-      outIndex.set(h, id);
-      sizes.push({ what, raw: buf.length, packed: packed.length });
+      id = bodies.length;
+      bodies.push({ buf, what, page });
+      bodyIndex.set(h, id);
     }
     return id;
   }
@@ -127,12 +146,58 @@ export async function bundle(cfg, log) {
         if (i >= 0) pagesOut[v][k] = { a: i };
       } else {
         const html = (await readBody(e.body)).toString("utf8");
-        pagesOut[v][k] = { b: emit(Buffer.from(rewriteHTML(html, k, v)), `page ${v} ${k}`), s: e.status };
+        pagesOut[v][k] = { b: emit(Buffer.from(rewriteHTML(html, k, v)), `page ${v} ${k}`, true), s: e.status };
         pageCount++;
       }
     }
   }
 
+  // --- packing ------------------------------------------------------------------
+  const gzipB64 = (buf) => zlib.gzipSync(buf, { level: 9 }).toString("base64");
+  const blocks = new Map(); // body id -> base64, for bodies stored on their own
+  const clusters = []; // base64 of each cluster
+  const locs = {}; // body id -> [cluster, offset, length], for clustered pages
+  let clusterRawMax = 0;
+
+  if (!zstd) {
+    for (let i = 0; i < bodies.length; i++) blocks.set(i, gzipB64(bodies[i].buf));
+  } else {
+    for (let i = 0; i < bodies.length; i++) if (!bodies[i].page) blocks.set(i, gzipB64(bodies[i].buf));
+    // Sorted by route and query, so a cluster holds pages that resemble each
+    // other — which is the whole reason a cluster is smaller than its parts.
+    const pageIds = bodies.map((b, i) => i).filter((i) => bodies[i].page).sort((x, y) => (bodies[x].what < bodies[y].what ? -1 : 1));
+    // A window at least as large as a cluster: a page must be able to match
+    // against any earlier page in the same cluster.
+    const windowLog = Math.min(27, Math.max(20, Math.ceil(Math.log2(Math.max(cfg.clusterBytes, 1)))));
+    let cur = [];
+    let size = 0;
+    const flush = () => {
+      if (!cur.length) return;
+      let offset = 0;
+      for (const i of cur) {
+        locs[i] = [clusters.length, offset, bodies[i].buf.length];
+        offset += bodies[i].buf.length;
+      }
+      const raw = Buffer.concat(cur.map((i) => bodies[i].buf));
+      clusterRawMax = Math.max(clusterRawMax, raw.length);
+      clusters.push(
+        zlib
+          .zstdCompressSync(raw, {
+            params: { [zlib.constants.ZSTD_c_compressionLevel]: 19, [zlib.constants.ZSTD_c_windowLog]: windowLog },
+          })
+          .toString("base64"),
+      );
+      cur = [];
+      size = 0;
+    };
+    for (const i of pageIds) {
+      if (cur.length && size + bodies[i].buf.length > cfg.clusterBytes) flush();
+      cur.push(i);
+      size += bodies[i].buf.length;
+    }
+    flush();
+  }
+
   // --- runtime + output ---------------------------------------------------------
   const frameSrc = await fs.readFile(path.join(HERE, "runtime", "frame.js"), "utf8");
   const shellSrc = (await fs.readFile(path.join(HERE, "runtime", "shell.js"), "utf8"))
@@ -152,6 +217,8 @@ export async function bundle(cfg, log) {
     badge: cfg.offline.badge,
     switcher: cfg.offline.switcher,
     missingLinks: cfg.offline.missingLinks,
+    codec: zstd ? "zstd" : "gzip",
+    locs: zstd ? locs : undefined,
     assets: assets.map((a) => [a.k, a.v, a.b, a.type, a.status ?? 200]),
     pages: pagesOut,
   };
@@ -168,7 +235,9 @@ export async function bundle(cfg, log) {
     `<div id="no-loading">Opening ${title}…</div>`,
     `<script type="application/json" id="no-manifest">${JSON.stringify(manifestOut).replace(/</g, "\\u003c")}</script>`,
   ];
-  for (let i = 0; i < out.length; i++) parts.push(`<script type="text/plain" id="no-b${i}">${out[i]}</script>`);
+  for (const [id, b64] of blocks) parts.push(`<script type="text/plain" id="no-b${id}">${b64}</script>`);
+  for (let i = 0; i < clusters.length; i++) parts.push(`<script type="text/plain" id="no-c${i}">${clusters[i]}</script>`);
+  if (zstd) parts.push(`<script>${await inlineZstdDecoder()}</script>`);
   parts.push(`<script>${shellSrc.replace(/<\/script/gi, "<\\/script").replace(/<!--/g, "<\\!--")}</script>`);
   parts.push(`</body></html>\n`);
 
@@ -178,25 +247,57 @@ export async function bundle(cfg, log) {
 
   // --- report ----------------------------------------------------------------------
   const mb = (n) => (n / 1024 / 1024).toFixed(2) + " MB";
-  const pageBytes = sizes.filter((s) => s.what.startsWith("page")).reduce((n, s) => n + s.packed, 0);
-  const assetBytes = sizes.filter((s) => s.what.startsWith("asset")).reduce((n, s) => n + s.packed, 0);
+  const assetBytes = assets.reduce((n, a) => n + (blocks.get(a.b)?.length ?? 0), 0);
+  const pageBytes = zstd
+    ? clusters.reduce((n, c) => n + c.length, 0)
+    : bodies.reduce((n, b, i) => n + (b.page ? blocks.get(i).length : 0), 0);
   log("");
   log(`bundle finished in ${((Date.now() - t0) / 1000).toFixed(1)}s -> ${path.relative(process.cwd(), cfg.out)}`);
   log(`  file size        ${mb(html.length)}   (pages ${mb(pageBytes)}, assets ${mb(assetBytes)})`);
-  log(`  pages            ${pageCount} captured, ${out.length} unique bodies after dedupe`);
+  log(`  pages            ${pageCount} captured, ${bodies.filter((b) => b.page).length} unique bodies after dedupe`);
   log(`  assets           ${assets.length} (${jsRewritten} scripts with location virtualised)`);
-  const biggest = [...sizes].sort((a, b) => b.packed - a.packed).slice(0, 6);
-  log(`  largest bodies   ${biggest.map((s) => `${s.what.slice(0, 70)} ${mb(s.packed)}`).join("\n                   ")}`);
+  if (zstd) {
+    log(`  clusters         ${clusters.length} zstd, up to ${mb(clusterRawMax)} of pages each, decoded on demand`);
+  } else {
+    log(`  compression      gzip per page — try compress: "zstd" if this file is large`);
+  }
   if (missing.size) {
     const top = [...missing.entries()].sort((a, b) => b[1] - a[1]).slice(0, 10);
     log(`  NOT CAPTURED     ${missing.size} referenced URL(s) will fail offline:`);
     for (const [k, n] of top) log(`                   ${k}  (${n}x)`);
   }
-  const report = { file: cfg.out, bytes: html.length, pageCount, bodies: out.length, missing: Object.fromEntries(missing), sizes };
+  const report = {
+    file: cfg.out,
+    bytes: html.length,
+    pageCount,
+    codec: manifestOut.codec,
+    clusters: clusters.length,
+    pageBytes,
+    assetBytes,
+    missing: Object.fromEntries(missing),
+  };
   await fs.writeFile(cfg.out.replace(/\.html?$/, "") + ".bundle.json", JSON.stringify(report, null, 1));
   return report;
 }
 
+/**
+ * fzstd's UMD build, which assigns a `fzstd` global when run as a plain script.
+ * Read by path rather than resolved: the package's exports map does not expose
+ * this build, and it is the only one that works inside the snapshot.
+ */
+async function inlineZstdDecoder() {
+  // Resolved through the package's main entry, then up to its directory: the
+  // exports map does not expose "./package.json" or "./umd/index.js", so
+  // neither can be resolved by subpath.
+  const pkg = path.dirname(path.dirname(require.resolve("fzstd")));
+  const umd = path.join(pkg, "umd", "index.js");
+  try {
+    return await fs.readFile(umd, "utf8");
+  } catch (e) {
+    throw new Error(`compress: "zstd" needs fzstd's UMD build at ${umd}, which is missing (${e.code}). Reinstall dependencies, or use compress: "gzip".`);
+  }
+}
+
 const SHELL_CSS = `
 html,body{margin:0;height:100%;overflow:hidden;background:#fff}
 .no-frame{position:fixed;inset:0;width:100%;height:100%;border:0;display:block;background:#fff}

package/lib/capture.mjs

@@ -9,6 +9,7 @@
 // or OPTIONS is aborted before it leaves the browser. Only a variant's `login`
 // hook, which runs before that guard is installed, can write.
 
+import { spawnSync } from "node:child_process";
 import crypto from "node:crypto";
 import fss from "node:fs";
 import fs from "node:fs/promises";
@@ -146,9 +147,13 @@ export async function capture(cfg, log) {
     await browser.close();
   }
 
-  if (cfg.includeStatic && cfg.staticDir && fss.existsSync(cfg.staticDir)) {
-    const n = await addDir(cfg.staticDir, cfg.staticPrefix, Infinity);
-    log(`static: +${n} build files from ${path.relative(process.cwd(), cfg.staticDir)}`);
+  // The build output, so chunks the crawl never triggered are in the file too.
+  // In Docker it lives inside the container; copy it out first.
+  let staticDir = cfg.staticDir;
+  if (cfg.docker?.container) staticDir = copyFromContainer(cfg, log) ?? staticDir;
+  if (cfg.includeStatic && staticDir && fss.existsSync(staticDir)) {
+    const n = await addDir(staticDir, cfg.staticPrefix, Infinity);
+    log(`static: +${n} build files from ${path.relative(process.cwd(), staticDir)}`);
   }
   if (cfg.publicDir && fss.existsSync(cfg.publicDir)) {
     const n = await addDir(cfg.publicDir, "/", cfg.maxPublicFileBytes);
@@ -181,7 +186,20 @@ export async function capture(cfg, log) {
       return route.abort("blockedbyclient");
     });
     const page = await context.newPage();
-    const s = { v, context, page, visited: new Set(), rsc: new Set(), files: new Set(), explored: new Set(), current: null };
+    const s = {
+      v,
+      context,
+      page,
+      visited: new Set(),
+      rsc: new Set(),
+      files: new Set(),
+      explored: new Set(),
+      // Keys a tab click produced. In "url" mode they are not explored again:
+      // their tab strip leads back to pages already captured, and re-clicking
+      // it on every one of them is what turns this from linear into expensive.
+      fromTabs: new Set(),
+      current: null,
+    };
     context.on("response", (res) => track(onResponse(s, res)));
     // A click that opens a window must not leave a second crawler behind.
     context.on("page", (p) => {
@@ -364,12 +382,19 @@ export async function capture(cfg, log) {
   // a tab that never asks the server — expose views that no link names.
   // Candidates: [role=tab], the configured `explore.click` selectors, and
   // "button bars" (an element whose children are two or more buttons and
-  // nothing else), which is how most tab strips are built without ARIA. Each
-  // (page path, label) is clicked once per variant.
+  // nothing else), which is how most tab strips are built without ARIA.
+  //
+  // `explore.tabs: true` clicks each label once per page *path*: cheap, but a
+  // route whose pages differ by query (?programme=…) then holds tab views for
+  // only the first of them. `"url"` clicks each label once per captured page,
+  // so every programme gets every tab — pages, and file size, multiply by the
+  // number of tabs.
   async function exploreClicks(s, key) {
     const { page } = s;
     const found = new Set();
-    const pathOnly = key.split("?")[0];
+    const perUrl = cfg.explore.tabs === "url";
+    if (perUrl && s.fromTabs.has(key)) return found;
+    const scope = perUrl ? key : key.split("?")[0];
     const tag = () =>
       page
         .evaluate(
@@ -404,7 +429,7 @@ export async function capture(cfg, log) {
     const labels = (await tag()).slice(0, cfg.explore.maxClicks);
     let dirty = false;
     for (const label of labels) {
-      const sig = `${pathOnly}::click:${label}`;
+      const sig = `${scope}::click:${label}`;
       if (s.explored.has(sig)) continue;
       s.explored.add(sig);
       if (dirty) {
@@ -420,6 +445,7 @@ export async function capture(cfg, log) {
         const k = keyOf(page.url());
         if (k && k !== key) {
           found.add(k);
+          s.fromTabs.add(k);
           dirty = true;
         }
       } catch {
@@ -496,6 +522,31 @@ export async function capture(cfg, log) {
   }
 }
 
+/**
+ * `docker cp <container>:<staticPath>` into the capture directory. Returns the
+ * local path, or null with a warning — a missing build is worth saying out
+ * loud, but it does not stop a capture that is otherwise fine.
+ */
+function copyFromContainer(cfg, log) {
+  const { container, staticPath } = cfg.docker;
+  const dest = path.join(cfg.captureDir, "docker-static");
+  // Ours, written by the previous run: `docker cp` nests into a directory that
+  // already exists, which would bury the files a level deeper each time.
+  fss.rmSync(dest, { recursive: true, force: true });
+  const r = spawnSync("docker", ["cp", `${container}:${staticPath}`, dest], { encoding: "utf8" });
+  if (r.error?.code === "ENOENT") {
+    log(`  warn: docker is not installed, so ${container}:${staticPath} could not be copied`);
+    return null;
+  }
+  if (r.status !== 0) {
+    log(`  warn: docker cp ${container}:${staticPath} failed — ${(r.stderr || "").trim().split("\n")[0]}`);
+    log(`  warn: continuing without the build output; lazily-loaded chunks may be missing offline`);
+    return null;
+  }
+  log(`docker: copied ${container}:${staticPath}`);
+  return dest;
+}
+
 async function walk(dir) {
   const out = [];
   for (const e of await fs.readdir(dir, { withFileTypes: true })) {

package/lib/config.mjs

@@ -22,7 +22,10 @@ export async function loadConfig(file) {
   // other — a sign-in that redirects across them loses its session.
   const origin = new URL(raw.url ?? `http://localhost:${port}`).origin;
   const o = new URL(origin);
-  const loopback = ["localhost", "127.0.0.1", "[::1]"];
+  // 0.0.0.0 is here because of Docker: Next's own images set HOSTNAME=0.0.0.0,
+  // and an app that builds absolute URLs from that redirects to 0.0.0.0:PORT.
+  // Without the alias those pages look like a different site and are dropped.
+  const loopback = ["localhost", "127.0.0.1", "[::1]", "0.0.0.0"];
   const aliases = [
     ...(loopback.includes(o.hostname) ? loopback.map((h) => `${o.protocol}//${h}${o.port ? ":" + o.port : ""}`) : []),
     ...(raw.aliases ?? []),
@@ -35,6 +38,9 @@ export async function loadConfig(file) {
     label: v.id,
     ...v,
   }));
+  const compress = raw.compress ?? "gzip";
+  if (compress !== "gzip" && compress !== "zstd") throw new Error(`compress must be "gzip" or "zstd", not ${JSON.stringify(compress)}`);
+
   const ids = new Set();
   for (const v of variants) {
     if (!/^[A-Za-z0-9_.-]+$/.test(v.id)) throw new Error(`Variant id "${v.id}" must be [A-Za-z0-9_.-]+`);
@@ -70,7 +76,10 @@ export async function loadConfig(file) {
     explore: {
       selects: true,
       maxOptions: 40,
-      // Click tab-like controls and record any URL they write.
+      // Click tab-like controls and record any URL they write. `true` clicks
+      // each once per page path; "url" clicks each once per captured page, so
+      // a route whose pages differ by query gets every tab for every one of
+      // them — at the cost of multiplying pages by the number of tabs.
       tabs: true,
       // Extra CSS selectors to click the same way.
       click: [],
@@ -83,6 +92,21 @@ export async function loadConfig(file) {
     },
     variants,
     defaultVariant: raw.defaultVariant ?? variants[0].id,
+    // The app runs in a container: its build output is not on this disk, so
+    // `docker cp` it out before bundling. `staticPath` is where the build lives
+    // inside the image (Next's own Dockerfile puts it under /app).
+    docker: raw.docker ? { staticPath: "/app/.next/static", ...raw.docker } : null,
+    // How page bodies are packed into the file.
+    //
+    //   "gzip"  each page gzipped on its own, decoded by the browser itself.
+    //   "zstd"  pages of a route packed together into clusters and compressed
+    //           as one stream, with a small decoder inlined. Pages of an app
+    //           mostly repeat each other, and gzip's 32KB window cannot see
+    //           past one page — clustering is what makes a big app small.
+    compress,
+    // Raw bytes of pages per cluster, when clustering. Bigger is smaller but
+    // slower to open the first page of each cluster.
+    clusterBytes: raw.clusterBytes ?? 4 * 1024 * 1024,
     includeStatic: raw.includeStatic ?? true,
     staticDir: r(raw.staticDir) ?? defaultStatic,
     staticPrefix: raw.staticPrefix ?? "/_next/static/",

package/lib/runtime/shell.js

@@ -41,15 +41,39 @@
     for (var i = 0; i < n; i++) a[i] = bin.charCodeAt(i);
     return a;
   }
+  // Pages may be packed in clusters: many of them compressed as one stream,
+  // because pages of an app repeat each other and gzip cannot see past one
+  // page. A cluster is decoded on first use and the last few are kept.
+  var LOCS = M.locs || null;
+  var clusterCache = new Map();
+  function cluster(i) {
+    var hit = clusterCache.get(i);
+    if (hit) return hit;
+    var p = (async function () {
+      var el = document.getElementById("no-c" + i);
+      if (!el) throw new Error("snapshot cluster " + i + " is missing");
+      return fzstd.decompress(b64(el.textContent.trim()));
+    })();
+    clusterCache.set(i, p);
+    // Three decoded clusters is a few MB; older ones cost nothing to redo.
+    if (clusterCache.size > 3) clusterCache.delete(clusterCache.keys().next().value);
+    return p;
+  }
+
   var kept = {};
   function body(id, keep) {
     if (kept[id]) return kept[id];
-    var p = (async function () {
-      var el = document.getElementById("no-b" + id);
-      if (!el) throw new Error("snapshot body " + id + " is missing");
-      var stream = new Blob([b64(el.textContent.trim())]).stream().pipeThrough(new DecompressionStream("gzip"));
-      return new Uint8Array(await new Response(stream).arrayBuffer());
-    })();
+    var loc = LOCS && LOCS[id];
+    var p = loc
+      ? cluster(loc[0]).then(function (bytes) {
+          return bytes.subarray(loc[1], loc[1] + loc[2]);
+        })
+      : (async function () {
+          var el = document.getElementById("no-b" + id);
+          if (!el) throw new Error("snapshot body " + id + " is missing");
+          var stream = new Blob([b64(el.textContent.trim())]).stream().pipeThrough(new DecompressionStream("gzip"));
+          return new Uint8Array(await new Response(stream).arrayBuffer());
+        })();
     if (keep) kept[id] = p;
     return p;
   }

package/lib/server.mjs

@@ -8,7 +8,10 @@ import path from "node:path";
 export async function withServer(cfg, opts, log, fn) {
   if (!cfg.app?.start) {
     if (!(await isUp(cfg.origin))) {
-      throw new Error(`Nothing is answering at ${cfg.origin}. Start the app, or give the config an app.start command.`);
+      throw new Error(
+        `Nothing is answering at ${cfg.origin}. Start the app, or give the config an app.start command.\n` +
+          `If it runs in Docker, publish the port to this machine (docker run -p 3000:3000 …) and set url to that port.`,
+      );
     }
     log(`using the server already running at ${cfg.origin}`);
     return fn();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@africanpilot/next-snapshot",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Capture a running Next.js app and bundle it into one self-contained, offline HTML file.",
   "keywords": [
     "nextjs",
@@ -45,6 +45,7 @@
   },
   "dependencies": {
     "esbuild": "^0.28.2",
+    "fzstd": "^0.1.1",
     "playwright-core": "~1.63.0"
   },
   "publishConfig": {