squoosh-batch 1.0.0

package/README.md

@@ -0,0 +1,127 @@
+# Squoosh batch CLI
+
+Compress a folder of images without opening the web app. It runs the same
+WebAssembly codecs as batch mode, with the same default settings, so the same
+image and options give you the same file.
+
+```sh
+npm install -g squoosh-batch
+squoosh-batch photos/
+```
+
+No dependencies — the codecs are WebAssembly and ship inside the package.
+Node 18.3 or newer.
+
+From a checkout of this repo you can also run it without installing:
+
+```sh
+node cli/index.js photos/
+```
+
+## Usage
+
+```
+squoosh-batch [options] <files or directories...>
+
+  -f, --format <name>    Output format, repeatable or comma-separated.
+                         mozJPEG, webP, avif, jxl, oxiPNG, wp2, qoi
+                         (default: mozJPEG)
+  -o, --out-dir <dir>    Where to write results (default: ./squooshed)
+  -q, --quality <n>      Quality for lossy formats, overriding the default
+      --max-width <n>    Shrink images to fit this width
+      --max-height <n>   Shrink images to fit this height
+  -c, --concurrency <n>  Images to work on at once (default: CPUs - 1, max 4)
+      --overwrite        Replace existing output files
+  -h, --help             Show this
+```
+
+Examples:
+
+```sh
+# Everything in a folder, to JPEG
+node cli/index.js photos/
+
+# Two formats at once, at quality 60
+node cli/index.js -f webP -f avif -q 60 -o out photos/*.jpg
+
+# Shrink oversized images on the way through
+node cli/index.js -f webP --max-width 2000 hero.png
+```
+
+Directories are scanned one level deep for images. Formats are named exactly as
+the web app names them, and matched case-insensitively, so `-f webp` and
+`-f webP` both work. With more than one format, results go into a folder per
+format. Existing files are left alone unless you pass `--overwrite`.
+
+Images are decoded and encoded across a pool of worker threads, one image per
+worker, which is the same arrangement batch mode uses in the browser.
+
+## Does it match the web app?
+
+For most formats, exactly — same bytes, same checksum. Verified by compressing
+the same images both ways and comparing SHA-256:
+
+| Format  | Matches the web app |                                             |
+| ------- | ------------------- | ------------------------------------------- |
+| MozJPEG | Yes                 | byte-identical                              |
+| WebP    | Yes                 | byte-identical                              |
+| OxiPNG  | Yes                 | byte-identical                              |
+| WebP v2 | Yes                 | byte-identical                              |
+| QOI     | Yes                 | byte-identical                              |
+| AVIF    | No                  | a few tenths of a percent different in size |
+| JPEG XL | No                  | a few tenths of a percent different in size |
+
+AVIF and JPEG XL differ because the browser runs **multi-threaded** builds of
+those two codecs, and only single-threaded builds can load under Node. The
+number of threads changes how those encoders divide the work, which changes the
+bytes they emit. Quality is equivalent; the files just aren't identical.
+
+Two formats are missing here: **Browser JPEG** and **Browser PNG**. Those encode
+through a `<canvas>`, which only exists in a browser. Use MozJPEG and OxiPNG,
+which are better anyway.
+
+### Reading images
+
+The web app asks the browser to decode JPEG and PNG, because browsers already
+know how. Without a browser this uses the same wasm decoders Squoosh ships for
+its other formats. The two agree byte-for-byte on JPEG and on PNGs without
+transparency.
+
+PNGs with **semi-transparent** pixels are the one exception. A browser stores
+transparent images with premultiplied alpha and loses a little precision
+converting back, so a pixel the wasm decoder reads as `[136,128,119]` comes back
+from a canvas as `[136,127,119]`. That difference is invisible, but it's enough
+to change the compressed bytes. The CLI's reading is the more accurate of the
+two.
+
+## Publishing a release
+
+The package is `cli/`, and it carries its own copy of the codecs so an
+installed copy doesn't need the rest of the repo. `prepack` copies them in, so
+`npm pack` and `npm publish` both pick them up — there's no separate build step
+to remember.
+
+```sh
+cd cli
+npm version patch      # or minor / major
+npm publish
+```
+
+The first publish needs `npm login` and, since the name is unscoped, an account
+that can claim `squoosh-batch`. Check what you're about to ship first:
+
+```sh
+npm pack --dry-run
+```
+
+That should list `index.js`, `lib/`, `README.md` and about 30 files under
+`vendor/codecs` — roughly 3 MB compressed.
+
+`cli/vendor` is generated and git-ignored; don't commit it.
+
+## Keeping the defaults in step
+
+`lib/defaults.js` copies the default options out of the web app's encoder
+metadata (`src/features/encoders/*/shared/meta.ts`), and each block names the
+file it came from. If you change a default in the web app, change it here too,
+or the two will quietly stop producing the same files.

package/index.js

@@ -0,0 +1,273 @@
+#!/usr/bin/env node
+/**
+ * Squoosh batch compression from the command line.
+ *
+ * Runs the same wasm codecs as the web app's batch mode, with the same default
+ * options, so the same image and settings give you the same file.
+ */
+import { readFile, writeFile, mkdir, stat, readdir } from 'fs/promises';
+import { basename, extname, join, resolve } from 'path';
+import { parseArgs } from 'util';
+
+import { encoders } from './lib/codecs.js';
+import { WorkerPool, defaultConcurrency } from './lib/pool.js';
+
+const formatNames = Object.keys(encoders);
+
+const usage = `
+Usage: squoosh-batch [options] <files or directories...>
+
+Options:
+  -f, --format <name>    Output format, repeatable or comma-separated.
+                         ${formatNames.join(', ')}
+                         (default: mozJPEG)
+  -o, --out-dir <dir>    Where to write results (default: ./squooshed)
+  -q, --quality <n>      Quality for lossy formats, overriding the default
+      --max-width <n>    Shrink images to fit this width
+      --max-height <n>   Shrink images to fit this height
+  -c, --concurrency <n>  Images to work on at once (default: CPUs - 1, max 4)
+      --overwrite        Replace existing output files
+  -h, --help             Show this
+
+Examples:
+  squoosh-batch photos/
+  squoosh-batch -f webP -f avif -q 60 -o out photos/*.jpg
+  squoosh-batch -f webP --max-width 2000 hero.png
+`.trim();
+
+const imageExtensions = new Set([
+  '.jpg',
+  '.jpeg',
+  '.png',
+  '.webp',
+  '.avif',
+  '.jxl',
+  '.wp2',
+  '.qoi',
+  '.gif',
+  '.bmp',
+]);
+
+/** Expand directories into the image files inside them. */
+async function collectFiles(inputs) {
+  const files = [];
+
+  for (const input of inputs) {
+    const path = resolve(input);
+    let info;
+
+    try {
+      info = await stat(path);
+    } catch {
+      throw Error(`No such file or directory: ${input}`);
+    }
+
+    if (info.isDirectory()) {
+      const entries = await readdir(path, { withFileTypes: true });
+      for (const entry of entries) {
+        if (!entry.isFile()) continue;
+        if (!imageExtensions.has(extname(entry.name).toLowerCase())) continue;
+        files.push(join(path, entry.name));
+      }
+      continue;
+    }
+
+    files.push(path);
+  }
+
+  return files;
+}
+
+function parseFormats(values) {
+  const names = (values ?? ['mozJPEG'])
+    .flatMap((value) => value.split(','))
+    .map((value) => value.trim())
+    .filter(Boolean);
+
+  // Accept any capitalisation, since "mozjpeg" is the obvious thing to type.
+  return [...new Set(names)].map((name) => {
+    const match = formatNames.find(
+      (known) => known.toLowerCase() === name.toLowerCase(),
+    );
+    if (!match) {
+      throw Error(
+        `Unknown format "${name}". Available: ${formatNames.join(', ')}`,
+      );
+    }
+    return match;
+  });
+}
+
+function formatSize(bytes) {
+  if (bytes < 1000) return `${bytes} B`;
+  if (bytes < 1000 * 1000) return `${(bytes / 1000).toFixed(1)} kB`;
+  return `${(bytes / 1000 / 1000).toFixed(2)} MB`;
+}
+
+function formatDelta(from, to) {
+  if (from === 0) return '';
+  const change = Math.round(((to - from) / from) * 100);
+  return `${change > 0 ? '+' : ''}${change}%`;
+}
+
+async function main() {
+  let args;
+
+  try {
+    args = parseArgs({
+      allowPositionals: true,
+      options: {
+        format: { type: 'string', short: 'f', multiple: true },
+        'out-dir': { type: 'string', short: 'o' },
+        quality: { type: 'string', short: 'q' },
+        'max-width': { type: 'string' },
+        'max-height': { type: 'string' },
+        concurrency: { type: 'string', short: 'c' },
+        overwrite: { type: 'boolean' },
+        help: { type: 'boolean', short: 'h' },
+      },
+    });
+  } catch (err) {
+    console.error(`${err.message}\n\n${usage}`);
+    process.exit(1);
+  }
+
+  if (args.values.help || args.positionals.length === 0) {
+    console.log(usage);
+    process.exit(args.values.help ? 0 : 1);
+  }
+
+  const number = (value, name) => {
+    if (value === undefined) return undefined;
+    const parsed = Number(value);
+    if (!Number.isFinite(parsed) || parsed <= 0) {
+      throw Error(`--${name} must be a positive number`);
+    }
+    return parsed;
+  };
+
+  const formats = parseFormats(args.values.format);
+  const quality = number(args.values.quality, 'quality');
+  const maxWidth = number(args.values['max-width'], 'max-width');
+  const maxHeight = number(args.values['max-height'], 'max-height');
+  const outDir = resolve(args.values['out-dir'] ?? 'squooshed');
+  const files = await collectFiles(args.positionals);
+
+  if (files.length === 0) throw Error('No images found');
+
+  await mkdir(outDir, { recursive: true });
+  for (const format of formats) {
+    if (formats.length > 1) {
+      await mkdir(join(outDir, encoders[format].extension), {
+        recursive: true,
+      });
+    }
+  }
+
+  const concurrency = Math.min(
+    number(args.values.concurrency, 'concurrency') ?? defaultConcurrency(),
+    files.length,
+  );
+
+  console.log(
+    `Compressing ${files.length} image${files.length === 1 ? '' : 's'} ` +
+      `to ${formats.map((f) => encoders[f].label).join(', ')}` +
+      `${concurrency > 1 ? ` (${concurrency} at a time)` : ''}\n`,
+  );
+
+  let sourceTotal = 0;
+  const outputTotals = new Map(formats.map((format) => [format, 0]));
+  let failures = 0;
+  let failedFiles = 0;
+  let done = 0;
+
+  const pool = new WorkerPool(concurrency);
+
+  // Hand every image to the pool at once and let it meter them out. Results
+  // print as they land, so one slow image doesn't hold up the rest.
+  const tasks = files.map(async (path) => {
+    const name = basename(path);
+
+    let result;
+    try {
+      result = await pool.run({ path, formats, quality, maxWidth, maxHeight });
+    } catch (err) {
+      failures++;
+      failedFiles++;
+      done++;
+      console.log(
+        `[${done}/${files.length}] ${name}\n    error  ${err.message}`,
+      );
+      return;
+    }
+
+    done++;
+    const counter = `[${done}/${files.length}]`;
+
+    sourceTotal += result.sourceSize;
+    console.log(
+      `${counter} ${name}  ${formatSize(result.sourceSize)} · ` +
+        `${result.width}×${result.height}`,
+    );
+
+    for (const { format, data, error } of result.results) {
+      const { extension, label } = encoders[format];
+
+      if (error) {
+        failures++;
+        console.log(`    ${label.padEnd(20)} error  ${error}`);
+        continue;
+      }
+
+      const outName = name.replace(/\.[^.]*$/, '') + '.' + extension;
+      const outPath = join(
+        outDir,
+        formats.length > 1 ? extension : '',
+        outName,
+      );
+
+      if (!args.values.overwrite) {
+        try {
+          await stat(outPath);
+          console.log(`    ${label.padEnd(20)} skipped, file exists`);
+          continue;
+        } catch {
+          // Doesn't exist, which is what we want.
+        }
+      }
+
+      await writeFile(outPath, data);
+      outputTotals.set(format, outputTotals.get(format) + data.length);
+      console.log(
+        `    ${label.padEnd(20)} ${formatSize(data.length).padStart(9)}  ` +
+          `${formatDelta(result.sourceSize, data.length)}`,
+      );
+    }
+  });
+
+  await Promise.all(tasks);
+  await pool.close();
+
+  const compressed = files.length - failedFiles;
+  console.log(
+    `\n${compressed} image${compressed === 1 ? '' : 's'} · ` +
+      `${formatSize(sourceTotal)} original` +
+      (failedFiles > 0 ? `  (${failedFiles} failed)` : ''),
+  );
+  for (const format of formats) {
+    const total = outputTotals.get(format);
+    console.log(
+      `  ${encoders[format].label.padEnd(20)} ${formatSize(total).padStart(
+        9,
+      )}  ` + `${formatDelta(sourceTotal, total)}`,
+    );
+  }
+  console.log(`\nWritten to ${outDir}`);
+
+  if (failures > 0) process.exitCode = 1;
+}
+
+main().catch((err) => {
+  console.error(`\n${err.message}`);
+  process.exit(1);
+});

package/lib/codec-paths.js

@@ -0,0 +1,62 @@
+/**
+ * Where each codec lives, and how it's built.
+ *
+ * This is the single list of what the CLI needs from `codecs/`. The publish
+ * step reads it to copy exactly those files into the package, so adding a
+ * codec here is all it takes for a release to carry it.
+ *
+ * `kind` picks the loader: `emscripten` builds sit next to a `.wasm` of the
+ * same name, `bindgen` ones name their wasm separately.
+ */
+export const decoders = {
+  'image/jpeg': { kind: 'emscripten', path: 'mozjpeg/dec/mozjpeg_node_dec.js' },
+  'image/png': {
+    kind: 'bindgen',
+    path: 'png/pkg/squoosh_png.js',
+    wasm: 'squoosh_png_bg.wasm',
+  },
+  'image/webp': { kind: 'emscripten', path: 'webp/dec/webp_node_dec.js' },
+  'image/avif': { kind: 'emscripten', path: 'avif/dec/avif_node_dec.js' },
+  'image/jxl': { kind: 'emscripten', path: 'jxl/dec/jxl_node_dec.js' },
+  'image/webp2': { kind: 'emscripten', path: 'wp2/dec/wp2_node_dec.js' },
+  'image/qoi': { kind: 'emscripten', path: 'qoi/dec/qoi_dec.js' },
+};
+
+export const encoderBuilds = {
+  mozJPEG: { kind: 'emscripten', path: 'mozjpeg/enc/mozjpeg_node_enc.js' },
+  // The browser prefers webp_enc_simd; there's no SIMD build for Node, and
+  // both produce byte-identical output.
+  webP: { kind: 'emscripten', path: 'webp/enc/webp_node_enc.js' },
+  // The Node builds are a newer revision taking different options, so this
+  // uses the browser's own single-threaded build. The browser reaches for
+  // avif_enc_mt when threads are available, which differs slightly.
+  avif: { kind: 'emscripten', path: 'avif/enc/avif_enc.js' },
+  // The browser prefers jxl_enc_mt_simd; Node only has the plain build.
+  jxl: { kind: 'emscripten', path: 'jxl/enc/jxl_node_enc.js' },
+  // The browser uses the parallel build; optimisation is deterministic, so
+  // the single-threaded one produces the same file.
+  oxiPNG: {
+    kind: 'bindgen',
+    path: 'oxipng/pkg/squoosh_oxipng.js',
+    wasm: 'squoosh_oxipng_bg.wasm',
+  },
+  // The browser prefers wp2_enc_mt_simd; Node only has the plain build.
+  wp2: { kind: 'emscripten', path: 'wp2/enc/wp2_node_enc.js' },
+  // Only a browser build exists, and it runs here unchanged.
+  qoi: { kind: 'emscripten', path: 'qoi/enc/qoi_enc.js' },
+};
+
+export const resizeBuild = {
+  kind: 'bindgen',
+  path: 'resize/pkg/squoosh_resize.js',
+  wasm: 'squoosh_resize_bg.wasm',
+};
+
+/** Every build the CLI can reach for. */
+export function allBuilds() {
+  return [
+    ...Object.values(decoders),
+    ...Object.values(encoderBuilds),
+    resizeBuild,
+  ];
+}

package/lib/codecs.js

@@ -0,0 +1,183 @@
+/**
+ * Decoders, encoders and resizing, mirroring what the web app's workers do so
+ * the same input and options produce the same bytes.
+ *
+ * Each entry points at the same wasm the browser runs. Where a codec ships
+ * several builds (SIMD, multi-threaded), the note on that entry says which one
+ * the browser picks and which one is used here.
+ */
+import { existsSync } from 'fs';
+import { fileURLToPath } from 'url';
+import { dirname, join } from 'path';
+import { loadBindgen, loadEmscripten, ImageData } from './wasm.js';
+import {
+  decoders as decoderBuilds,
+  encoderBuilds,
+  resizeBuild,
+} from './codec-paths.js';
+
+const here = dirname(fileURLToPath(import.meta.url));
+
+/**
+ * Published packages carry their own copy of the codecs; a repo checkout uses
+ * the ones already there.
+ */
+const vendored = join(here, '..', 'vendor', 'codecs');
+const codecRoot = existsSync(vendored)
+  ? vendored
+  : join(here, '..', '..', 'codecs');
+
+/** Load a build described in codec-paths. */
+function load(build) {
+  const path = join(codecRoot, build.path);
+  return build.kind === 'bindgen'
+    ? loadBindgen(path, build.wasm)
+    : loadEmscripten(path);
+}
+
+/** Mirrors sniffMimeType in the web app's client utils. */
+const magicNumbers = [
+  [/^\x89PNG\x0D\x0A\x1A\x0A/, 'image/png'],
+  [/^\xFF\xD8\xFF/, 'image/jpeg'],
+  [/^RIFF....WEBPVP8[LX ]/s, 'image/webp'],
+  [/^\x00\x00\x00 ftypavif\x00\x00\x00\x00/, 'image/avif'],
+  [/^\xff\x0a/, 'image/jxl'],
+  [/^\x00\x00\x00\x0cJXL \x0d\x0a\x87\x0a/, 'image/jxl'],
+  [/^\xF4\xFF\x6F/, 'image/webp2'],
+  [/^qoif/, 'image/qoi'],
+  [/^GIF87a/, 'image/gif'],
+  [/^GIF89a/, 'image/gif'],
+  [/^BM/, 'image/bmp'],
+];
+
+export function sniffMimeType(buffer) {
+  const head = Array.from(buffer.subarray(0, 16))
+    .map((v) => String.fromCodePoint(v))
+    .join('');
+
+  for (const [pattern, mimeType] of magicNumbers) {
+    if (pattern.test(head)) return mimeType;
+  }
+  return '';
+}
+
+const decoders = Object.fromEntries(
+  Object.entries(decoderBuilds).map(([mimeType, build]) => [
+    mimeType,
+    async (data) => (await load(build)).decode(data),
+  ]),
+);
+
+export function canDecode(mimeType) {
+  return mimeType in decoders;
+}
+
+export async function decode(data) {
+  const mimeType = sniffMimeType(data);
+  const decoder = decoders[mimeType];
+
+  if (!decoder) {
+    throw Error(
+      mimeType
+        ? `Can't decode ${mimeType}`
+        : "Doesn't look like an image this tool can read",
+    );
+  }
+
+  const result = await decoder(data);
+  if (!result) throw Error(`Couldn't decode ${mimeType}`);
+  return result;
+}
+
+/**
+ * Encoders, keyed by the same names the web app uses, so `--format webP`
+ * refers to exactly the entry the UI calls "WebP".
+ *
+ * browserJPEG and browserPNG are deliberately absent: they encode through a
+ * canvas, which only exists in a browser.
+ */
+const encoderMeta = {
+  mozJPEG: { label: 'MozJPEG', extension: 'jpg' },
+  webP: { label: 'WebP', extension: 'webp' },
+  avif: { label: 'AVIF', extension: 'avif' },
+  jxl: { label: 'JPEG XL (beta)', extension: 'jxl' },
+  oxiPNG: { label: 'OxiPNG', extension: 'png' },
+  wp2: { label: 'WebP v2 (unstable)', extension: 'wp2' },
+  qoi: { label: 'QOI', extension: 'qoi' },
+};
+
+/**
+ * Encoders, keyed by the same names the web app uses, so `--format webP`
+ * refers to exactly the entry the UI calls "WebP".
+ *
+ * browserJPEG and browserPNG are deliberately absent: they encode through a
+ * canvas, which only exists in a browser.
+ */
+export const encoders = Object.fromEntries(
+  Object.entries(encoderMeta).map(([name, meta]) => [
+    name,
+    {
+      ...meta,
+      async encode(image, options) {
+        const module = await load(encoderBuilds[name]);
+
+        if (name === 'oxiPNG') {
+          return module.optimise(
+            image.data,
+            image.width,
+            image.height,
+            options.level,
+            options.interlace,
+          );
+        }
+
+        const result = module.encode(
+          image.data,
+          image.width,
+          image.height,
+          options,
+        );
+        if (!result) throw Error('Encoding error');
+        return result;
+      },
+    },
+  ]),
+);
+
+/** Resize methods by index, as the resize worker orders them. */
+const resizeMethods = ['triangle', 'catrom', 'mitchell', 'lanczos3'];
+
+/**
+ * Scale an image, matching what batch mode asks the resize worker for:
+ * lanczos3, stretch, premultiplied, in linear RGB.
+ */
+export async function resize(image, width, height) {
+  const module = await load(resizeBuild);
+
+  const result = module.resize(
+    new Uint8Array(
+      image.data.buffer,
+      image.data.byteOffset,
+      image.data.byteLength,
+    ),
+    image.width,
+    image.height,
+    width,
+    height,
+    resizeMethods.indexOf('lanczos3'),
+    true, // premultiply
+    true, // linearRGB
+  );
+
+  return new ImageData(new Uint8ClampedArray(result.buffer), width, height);
+}
+
+/** Scale down to fit a box, preserving aspect ratio. Mirrors fitWithin. */
+export function fitWithin(width, height, maxWidth, maxHeight) {
+  const scale = Math.min(maxWidth / width, maxHeight / height);
+  if (scale >= 1) return undefined;
+  return {
+    width: Math.max(1, Math.round(width * scale)),
+    height: Math.max(1, Math.round(height * scale)),
+  };
+}