@vanhanit/pre-compress 1.0.1 → 1.1.0

package/README.md

@@ -16,9 +16,9 @@
 
 pre-compress writes a gzip (`.gz`) and a brotli (`.br`) version next to each file matched by the glob patterns, using
 the highest compression levels. A web server can then send the compressed file from disk instead of compressing it on
-every request, for example with `gzip_static` and `brotli_static` in nginx.
+every request, see [Serve the compressed files](#serve-the-compressed-files).
 
-- Files that are already compressed (`.gz` and `.br`) are never compressed again.
+- Files that are already compressed (`.gz`, `.br` and `.zst`) are never compressed again.
 - Compressed files are written atomically, so a server never sends a partially written file.
 - No dependencies, only Node.js built-ins.
 - Free to use, also in companies, under the [PolyForm Shield License](#license).
@@ -64,11 +64,12 @@ Usage: pre-compress [options]
 Compress static files to be served by a web server.
 
 Options:
-  -f, --files <files...>  the files to compress, supports glob patterns to match multiple files
-  -d, --delete-larger     delete compressed files that are not smaller than the source
-  -s, --skip-existing     don't compress if the compressed version is newer than the source
-  -V, --version           output the version number
-  -h, --help              display help for command
+  -f, --files <files...>          the files to compress, supports glob patterns to match multiple files
+  -e, --encodings <encodings...>  the encodings to write: gzip, brotli, zstd (default: gzip,brotli)
+  -d, --delete-larger             delete compressed files that are not smaller than the source
+  -s, --skip-existing             don't compress if the compressed version is newer than the source
+  -V, --version                   output the version number
+  -h, --help                      display help for command
 ```
 
 Quote the patterns so the shell passes them on unexpanded:
@@ -77,11 +78,80 @@ Quote the patterns so the shell passes them on unexpanded:
 pre-compress --delete-larger --skip-existing --files "dist/**/*.{html,css,js,svg}"
 ```
 
+- `--encodings` selects the encodings to write. Add `zstd` to also write a zstandard (`.zst`) version, which recent
+  browsers support. The zstd API of Node.js is still experimental.
 - `--delete-larger` removes compressed versions that would not save any bytes, so the server sends the source instead.
 - `--skip-existing` only compresses files that changed since they were last compressed, which speeds up repeated runs.
 
 The command exits with code 1 when the arguments are invalid or a file cannot be compressed.
 
+## Serve the compressed files
+
+The server must send the compressed file only to browsers that accept its encoding, with the type of the original file,
+a `Content-Encoding` header and `Vary: Accept-Encoding` so caches keep one copy per encoding.
+
+### nginx
+
+`gzip_static` is part of nginx and included in the official packages and Docker images. Brotli and zstd need the
+third-party modules [ngx_brotli](https://github.com/google/ngx_brotli) and
+[zstd-nginx-module](https://github.com/tokers/zstd-nginx-module), which many distributions package, for example as
+`libnginx-mod-http-brotli-static` on Debian and Ubuntu.
+
+```nginx
+location / {
+    gzip_static on;
+    brotli_static on;  # ngx_brotli
+    zstd_static on;    # zstd-nginx-module
+    gzip_vary on;      # also adds Vary: Accept-Encoding for brotli and zstd
+    gzip_proxied any;  # gzip_static skips requests from proxies and CDNs (with a Via header) by default
+}
+```
+
+Leave out the directives of modules you don't have. nginx doesn't weigh the client's preferences between the modules:
+when a browser accepts several encodings, the first module that finds a file wins. Brotli files are usually the
+smallest, so with nginx, the default gzip and brotli files are enough.
+
+### Caddy
+
+Caddy supports all three encodings without plugins and picks the one the browser prefers, using the listed order when it
+has no preference:
+
+```caddyfile
+example.com {
+    root * /var/www
+    file_server {
+        precompressed br zstd gzip
+    }
+}
+```
+
+## Benchmark
+
+`npm run bench` serves files with a Node.js server, compressed on every request at the default levels servers use, or
+pre-compressed by pre-compress, and measures them under load. These are the results for the minified JavaScript in
+`node_modules` (5 files, 231 KiB) with 16 connections, on 4 cores of an Intel Xeon at 2.10 GHz and Node.js 22:
+
+| Encoding | Compressed     | Level | Size    | Requests/s | p50 ms | p99 ms | Server CPU ms/request |
+| -------- | -------------- | ----- | ------- | ---------- | ------ | ------ | --------------------- |
+| identity | no             | -     | 231 KiB | 11908      | 1.07   | 4.80   | 0.051                 |
+| gzip     | on request     | 6     | 61 KiB  | 2797       | 5.25   | 12.19  | 1.168                 |
+| gzip     | pre-compressed | 9     | 61 KiB  | 18058      | 0.76   | 2.90   | 0.033                 |
+| br       | on request     | 6     | 57 KiB  | 1793       | 8.10   | 19.90  | 1.868                 |
+| br       | pre-compressed | 11    | 53 KiB  | 17892      | 0.76   | 2.93   | 0.035                 |
+| zstd     | on request     | 3     | 63 KiB  | 3351       | 3.98   | 25.87  | 0.865                 |
+| zstd     | pre-compressed | 19    | 56 KiB  | 16565      | 0.80   | 3.02   | 0.036                 |
+
+Pre-compressed brotli files are 77% smaller than the originals and 7% smaller than brotli compressed on request, because
+the highest quality is too slow to use on every request. The server answers 10 times more requests than when it
+compresses with brotli, with 50 times less CPU per request, and even more than without compression, since it sends
+fewer bytes. Compressing the files once took 0.2 seconds.
+
+Pass options and your own files to measure them:
+
+```shell
+npm run bench -- --duration 10 --connections 64 "dist/**/*.{html,css,js}"
+```
+
 ## Development
 
 ```shell

package/dist/args.js

@@ -1,24 +1,33 @@
 import { parseArgs } from "node:util";
+import { ENCODING_NAMES } from "./compress.js";
 /**
  * Invalid command line arguments.
  */
 export class ArgumentError extends Error {
     name = "ArgumentError";
 }
+const DEFAULT_ENCODINGS = ["gzip", "brotli"];
 const OPTIONS = {
     files: { type: "string", short: "f", multiple: true },
+    encodings: { type: "string", short: "e", multiple: true },
     "delete-larger": { type: "boolean", short: "d" },
     "skip-existing": { type: "boolean", short: "s" },
     version: { type: "boolean", short: "V" },
     help: { type: "boolean", short: "h" }
 };
-const HELP = [
-    ["-f, --files <files...>", "the files to compress, supports glob patterns to match multiple files"],
-    ["-d, --delete-larger", "delete compressed files that are not smaller than the source"],
-    ["-s, --skip-existing", "don't compress if the compressed version is newer than the source"],
-    ["-V, --version", "output the version number"],
-    ["-h, --help", "display help for command"]
-];
+const DESCRIPTIONS = {
+    files: "the files to compress, supports glob patterns to match multiple files",
+    encodings: `the encodings to write: ${ENCODING_NAMES.join(", ")} (default: ${DEFAULT_ENCODINGS.join(",")})`,
+    "delete-larger": "delete compressed files that are not smaller than the source",
+    "skip-existing": "don't compress if the compressed version is newer than the source",
+    version: "output the version number",
+    help: "display help for command"
+};
+function flags(name) {
+    const option = OPTIONS[name];
+    const value = option.type === "string" ? ` <${name}...>` : "";
+    return `-${option.short}, --${name}${value}`;
+}
 function parse(args) {
     try {
         return parseArgs({ args: [...args], options: OPTIONS, allowPositionals: true });
@@ -27,6 +36,15 @@ function parse(args) {
         throw new ArgumentError(error.message);
     }
 }
+function parseEncodings(values) {
+    const names = new Set(values.flatMap((value) => value.split(",")));
+    for (const name of names) {
+        if (!ENCODING_NAMES.includes(name)) {
+            throw new ArgumentError(`unknown encoding '${name}', choose from ${ENCODING_NAMES.join(", ")}`);
+        }
+    }
+    return [...names];
+}
 /**
  * Parses the command line arguments, without the node and script paths. Patterns may follow `-f` or be repeated.
  */
@@ -40,20 +58,24 @@ export function parseArguments(args) {
     }
     const patterns = [...(values.files ?? []), ...positionals];
     if (patterns.length === 0) {
-        throw new ArgumentError("required option '-f, --files <files...>' not specified");
+        throw new ArgumentError(`required option '${flags("files")}' not specified`);
     }
     return {
         name: "compress",
         patterns,
-        deleteLarger: values["delete-larger"] ?? false,
-        skipExisting: values["skip-existing"] ?? false
+        options: {
+            encodings: values.encodings ? parseEncodings(values.encodings) : DEFAULT_ENCODINGS,
+            deleteLarger: values["delete-larger"] ?? false,
+            skipExisting: values["skip-existing"] ?? false
+        }
     };
 }
 /**
  * Returns the help text.
  */
 export function usage(name, description) {
-    const width = Math.max(...HELP.map(([flags]) => flags.length));
-    const options = HELP.map(([flags, text]) => `  ${flags.padEnd(width)}  ${text}`);
+    const names = Object.keys(OPTIONS);
+    const width = Math.max(...names.map((option) => flags(option).length));
+    const options = names.map((option) => `  ${flags(option).padEnd(width)}  ${DESCRIPTIONS[option]}`);
     return [`Usage: ${name} [options]`, "", description, "", "Options:", ...options].join("\n");
 }

package/dist/cli.js

@@ -5,41 +5,29 @@ import { ArgumentError, parseArguments, usage } from "./args.js";
 import { compressFile, isCompressed } from "./compress.js";
 import { mapConcurrently } from "./concurrency.js";
 import { findFiles } from "./files.js";
-import { formatDuration, pluralize } from "./format.js";
 import { readPackageInfo } from "./package-info.js";
-function report({ file, size, outputs }, io) {
-    for (const output of outputs) {
-        if (output.status === "written") {
-            io.log(`Compressed file ${styleText("yellow", file)} to ${styleText("green", output.file)}`);
-        }
-        else if (output.status === "skipped") {
-            io.log(`Skipped file ${styleText("yellow", output.file)} because it is up to date`);
-        }
-        else {
-            io.log(`Deleted file ${styleText("red", output.file)} because ${output.size} byte(s) is not smaller than ${size} byte(s)`);
-        }
-    }
-}
-async function compress(command, io) {
+import { describeResult, summarize } from "./report.js";
+const consoleIo = {
+    log: (message) => console.log(message),
+    error: (message) => console.error(message),
+    style: (color, text) => styleText(color, text)
+};
+async function compress({ patterns, options }, io) {
     const startTime = performance.now();
-    const files = (await findFiles(command.patterns)).filter((file) => !isCompressed(file));
+    const files = (await findFiles(patterns)).filter((file) => !isCompressed(file));
     const results = await mapConcurrently(files, availableParallelism(), async (file) => {
-        const result = await compressFile(file, command);
-        report(result, io);
+        const result = await compressFile(file, options);
+        for (const line of describeResult(result, io.style)) {
+            io.log(line);
+        }
         return result;
     });
-    const timeTakenMs = performance.now() - startTime;
-    const outputs = results.flatMap((result) => result.outputs);
-    const count = (status) => outputs.filter((output) => output.status === status).length;
-    io.log(`Processed ${pluralize(files.length, "file")} in ${styleText("green", formatDuration(timeTakenMs))}: ` +
-        `${styleText("green", String(count("written")))} written, ` +
-        `${styleText("yellowBright", String(count("skipped")))} skipped, ` +
-        `${styleText("red", String(count("deleted")))} deleted`);
+    io.log(summarize(results, performance.now() - startTime, io.style));
 }
 /**
  * Runs the command line interface with the arguments, without the node and script paths, and returns the exit code.
  */
-export async function run(args, io = console) {
+export async function run(args, io = consoleIo) {
     const { command: name, version, description } = readPackageInfo();
     try {
         const command = parseArguments(args);

package/dist/compress.js

@@ -1,16 +1,17 @@
 import { readFile, rm, stat } from "node:fs/promises";
 import { extname } from "node:path";
 import { promisify } from "node:util";
-import { brotliCompress, constants, gzip } from "node:zlib";
+import { brotliCompress, constants, gzip, zstdCompress } from "node:zlib";
 import { replaceFile, statFile } from "./files.js";
 const gzipAsync = promisify(gzip);
 const brotliCompressAsync = promisify(brotliCompress);
-const ENCODINGS = [
-    {
+const zstdCompressAsync = promisify(zstdCompress);
+const ENCODINGS = {
+    gzip: {
         extension: "gz",
         compress: (data) => gzipAsync(data, { level: constants.Z_BEST_COMPRESSION })
     },
-    {
+    brotli: {
         extension: "br",
         compress: (data) => brotliCompressAsync(data, {
             params: {
@@ -19,26 +20,37 @@ const ENCODINGS = [
                 [constants.BROTLI_PARAM_SIZE_HINT]: data.length
             }
         })
+    },
+    zstd: {
+        extension: "zst",
+        // Level 19 keeps the window at 8 MB or less, the most browsers must support for zstd (RFC 9659).
+        compress: (data) => zstdCompressAsync(data, {
+            params: { [constants.ZSTD_c_compressionLevel]: 19 },
+            pledgedSrcSize: data.length
+        })
     }
-];
+};
+/** The names of the supported encodings. */
+export const ENCODING_NAMES = Object.keys(ENCODINGS);
 /**
  * Whether the file has the extension of a compressed file this tool writes.
  */
 export function isCompressed(file) {
     const extension = extname(file).slice(1).toLowerCase();
-    return ENCODINGS.some((encoding) => encoding.extension === extension);
+    return Object.values(ENCODINGS).some((encoding) => encoding.extension === extension);
 }
 async function isUpToDate(target, source) {
     const stats = await statFile(target);
     return stats !== undefined && stats.mtimeMs >= source.mtimeMs;
 }
 /**
- * Writes a compressed file next to the source for every encoding.
+ * Writes a compressed file next to the source for every selected encoding.
  */
 export async function compressFile(file, options) {
     const source = await stat(file);
     let data;
-    const outputs = await Promise.all(ENCODINGS.map(async (encoding) => {
+    const outputs = await Promise.all(options.encodings.map(async (name) => {
+        const encoding = ENCODINGS[name];
         const target = `${file}.${encoding.extension}`;
         if (options.skipExisting && (await isUpToDate(target, source))) {
             return { file: target, status: "skipped" };

package/dist/main.js

@@ -1,3 +1,6 @@
 #!/usr/bin/env node
-import { run } from "./cli.js";
+import { configureThreadPool } from "./thread-pool.js";
+// The CLI is loaded afterwards, so nothing can use the thread pool before it is sized.
+configureThreadPool();
+const { run } = await import("./cli.js");
 process.exitCode = await run(process.argv.slice(2));

package/dist/report.js

@@ -0,0 +1,30 @@
+import { formatDuration, pluralize } from "./format.js";
+function describeOutput(file, size, output, style) {
+    switch (output.status) {
+        case "written":
+            return `Compressed file ${style("yellow", file)} to ${style("green", output.file)}`;
+        case "skipped":
+            return `Skipped file ${style("yellow", output.file)} because it is up to date`;
+        case "deleted":
+            return `Deleted file ${style("red", output.file)} because ${output.size} byte(s) is not smaller than ${size} byte(s)`;
+    }
+}
+/**
+ * Returns a line for every compressed file written, skipped or deleted for a source file.
+ */
+export function describeResult({ file, size, outputs }, style) {
+    return outputs.map((output) => describeOutput(file, size, output, style));
+}
+/**
+ * Returns a line counting what was done to the compressed files of all processed source files.
+ */
+export function summarize(results, milliseconds, style) {
+    const counts = { written: 0, skipped: 0, deleted: 0 };
+    for (const output of results.flatMap((result) => result.outputs)) {
+        counts[output.status]++;
+    }
+    return (`Processed ${pluralize(results.length, "file")} in ${style("green", formatDuration(milliseconds))}: ` +
+        `${style("green", String(counts.written))} written, ` +
+        `${style("yellowBright", String(counts.skipped))} skipped, ` +
+        `${style("red", String(counts.deleted))} deleted`);
+}

package/dist/thread-pool.js

@@ -0,0 +1,9 @@
+import { availableParallelism } from "node:os";
+const DEFAULT_SIZE = 4;
+/**
+ * Sizes libuv's thread pool, which runs the compression, to the available cores. Its default of 4 threads leaves the
+ * other cores idle. Takes effect only before the pool is first used, and keeps a size that is already set.
+ */
+export function configureThreadPool(env = process.env, cores = availableParallelism()) {
+    env.UV_THREADPOOL_SIZE ??= String(Math.max(DEFAULT_SIZE, cores));
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vanhanit/pre-compress",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "description": "Compress static files to be served by a web server.",
   "author": "Thomas Vanhaniemi",
   "type": "module",
@@ -58,6 +58,7 @@
     "lint": "eslint .",
     "format": "prettier --write .",
     "format:check": "prettier --check .",
+    "bench": "node bench/bench.ts",
     "check": "npm run lint && npm run format:check && npm run typecheck && npm test"
   }
 }