image-guard 5.5.0 → 5.6.0

package/README.md

@@ -8,11 +8,15 @@ Image Guard takes care of near-lossless compression of your images, to help you
 
 It’s convenient because setup is simple. Run it right away—done. Or install, run, add hook—done.
 
-It’s as safe as possible because compression happens losslessly (near-lossless for JPG and GIF images). That allows you to stop worrying about forgetting to compress images, but also about sacrificing too much quality. (You can take care of additional optimizations manually or through other tooling.)
+It’s as safe as possible because compression happens losslessly (near-lossless for PNG, JPG, and GIF images). That allows you to stop worrying about forgetting to compress images, but also about sacrificing too much quality. (You can take care of additional optimizations manually or through other tooling.)
 
 ## Installation and Use
 
-(Note available parameters below.)
+On Windows, Image Guard needs Git Bash, which comes with [Git for Windows](https://gitforwindows.org/). Git hooks use it already. For npx and npm scripts (from any shell, including PowerShell), have npm use Git Bash, too:
+
+```shell
+npm config set script-shell "C:\Program Files\Git\bin\bash.exe"
+```
 
 ### Ways to Use Image Guard
 
@@ -103,7 +107,7 @@ npm pkg set scripts.postprepare="grep -qxF './node_modules/.bin/image-guard --st
 
 If you give the hook options like `--ignore`, add them to the `postprepare` script as well—the hook command is the only place to configure Image Guard, and the next install rewrites that line.
 
-**Important:** When you commit images that have not yet been compressed, the automated compression process (triggered by the pre-commit hook) will modify those image files to reduce their size. As a result, after your initial commit attempt, you will see these images appear as changed files in Git. To include the optimized images in your repository, you need to stage and commit them again. In rare cases, if further compression is possible, you may need to repeat this process until no further changes are detected. This workflow is intentional and ensures that only optimally compressed images are committed. Many editors can display diffs for images, helping you review these changes.
+**Important:** When you commit images that have not yet been compressed, the pre-commit hook compresses them and stages the result, so the commit includes the compressed images. Images with changes you haven’t staged are the exception: Image Guard compresses them, too, but leaves them unstaged (and says so), so as not to commit changes you held back. Many editors can display diffs for images, helping you review these changes.
 
 ### Parameters
 
@@ -119,7 +123,7 @@ Image Guard takes an optional path to the directory to process—`npx image-guar
 
 * `--dry`/`-d` allows you to run Image Guard in “dry mode.” All changes are shown in the terminal (unless combined with `--quiet`, which leaves just the summary).
 
-* `--staged` (recommended with automated use) triggers a mode that watches PNG, JPG, GIF, WebP, and AVIF files (and HEIC/HEIF, when `--heic-to-avif` is used) from staged changes (`git diff --cached`) and only processes those files—that approach makes Image Guard more efficient in operation. Because the file set comes from Git, `--staged` can’t be combined with a path.
+* `--staged` (recommended with automated use) triggers a mode that watches PNG, JPG, GIF, WebP, and AVIF files (and HEIC/HEIF, when `--heic-to-avif` is used) from staged changes (`git diff --cached`) and only processes those files—that approach makes Image Guard more efficient in operation. It then stages the compressed (and converted) images. Because the file set comes from Git, `--staged` can’t be combined with a path.
 
 * `--no-cache` processes all images, including those recorded as already processed (see [“Record of Processed Images”](#record-of-processed-images)).
 
@@ -146,6 +150,12 @@ Image Guard is a Node script that uses [sharp](https://www.npmjs.com/package/sha
 
 Automated compression works by monitoring whether a given [change list](https://webglossary.info/terms/change-list/) includes any PNGs, JPGs, GIFs, WebPs, or AVIFs (and HEIC/HEIF when opted in). It’s initiated by a Git hook. Only those images are compressed where there is an improvement. HEIC conversion uses [heic-decode](https://www.npmjs.com/package/heic-decode) for patent-free decoding and sharp for lossy AVIF encoding (quality 80). The processed images can then be committed to the underlying repository.
 
+Compression removes metadata, except for what affects how an image looks: its orientation, its color profile (unless that’s sRGB, which browsers assume anyway), CMYK color, and the bit depth of AVIF images.
+
+PNG images with more than 256 colors are reduced to 256 colors (with dithering) where that makes them smaller and a comparison with the original shows no visible change. Other PNG images, like most photos, grayscale and 16-bit images, images with smooth gradients or small elements in rare colors, and images over 40 megapixels, are compressed losslessly, keeping their bit depth.
+
+Animated PNGs (APNG) are skipped, as sharp would keep only their first frame.
+
 Through this approach, though glossed over here, Image Guard makes up for what’s missing or complicated in other packages, namely easy, near-riskless, automatable, resource-friendly in-repo optimization.
 
 ### Record of Processed Images
@@ -158,7 +168,7 @@ The record is kept in node_modules/.cache/image-guard of the project being proce
 
 You use Image Guard when you need a simple, automatable, robust solution to compress images in a way that limits unnecessary image payload right from the start, in your repositories, and that reduces the risk that entirely uncompressed images go into production.
 
-As Image Guard compresses near-losslessly, there’s little risk of quality issues from compression. (Lossless compression is not possible for every image format, however, so there’s a risk when excessively iterating over the same images. Doing so may eventually degrade quality. The [record of processed images](#record-of-processed-images) prevents this, unless it’s bypassed or cleared.)
+As Image Guard compresses near-losslessly, there’s little risk of quality issues from compression. Repeat runs don’t degrade quality further, even when the [record of processed images](#record-of-processed-images) is bypassed or cleared.
 
 ## What Does Image Guard _Not_ Do?
 

package/bin/image-guard.sh

@@ -12,4 +12,13 @@ if ! command -v node > /dev/null 2>&1; then
   echo "Image Guard: Node not found. Install Node.js via nvm, Volta, asdf, mise, or Homebrew." >&2
   exit 1
 fi
-exec node "$(dirname "$0")/../image-guard/bin/image-guard.js" "$@"
+# Follows symlinks (as from node_modules/.bin or a global bin folder) to this script’s own folder
+self="$0"
+while [ -L "$self" ]; do
+  target="$(readlink "$self")"
+  case "$target" in
+    /*) self="$target" ;;
+    *) self="$(dirname "$self")/$target" ;;
+  esac
+done
+exec node "$(dirname "$self")/image-guard.js" "$@"

package/package.json

@@ -7,7 +7,7 @@
   },
   "dependencies": {
     "heic-decode": "^2.1.0",
-    "ignore": "^7.0.12",
+    "ignore": "^7.0.14",
     "sharp": "^0.35.5",
     "simple-git": "^4.0.2",
     "tinyglobby": "^0.2.17"
@@ -15,7 +15,7 @@
   "description": "Near-lossless image compressor that’s easy to use and easy to automate",
   "devDependencies": {
     "@eslint/js": "^10.0.1",
-    "eslint": "^10.12.0",
+    "eslint": "^10.13.0",
     "globals": "^17.13.0"
   },
   "engines": {
@@ -60,5 +60,5 @@
     "test": "node --test bin/image-guard.test.js benchmark/benchmark.test.js"
   },
   "type": "module",
-  "version": "5.5.0"
+  "version": "5.6.0"
 }

package/src/index.js

@@ -175,18 +175,25 @@ Options:
   }
   const limit = createLimiter(desiredFileConcurrency)
 
-  const processResults = (results) => {
+  // Paths written or removed, by the image they came from
+  const changesBySource = new Map()
+
+  const processResults = (results, files) => {
     let hadFailures = false
     let addedKB = 0
-    for (const r of results) {
+    results.forEach((r, i) => {
       if (r.status === 'fulfilled') {
-        addedKB += r.value
+        addedKB += r.value.saved
+        if (r.value.changed.length) changesBySource.set(files[i], r.value.changed)
       } else {
         hadFailures = true
-        const reason = r.reason && r.reason.message ? r.reason.message : String(r.reason)
-        console.error(styleStderr('red', 'Task failed:'), reason)
+        if (r.reason?.cause) {
+          console.error(styleStderr('red', `${r.reason.message}:`), r.reason.cause)
+        } else {
+          console.error(styleStderr('red', 'Task failed:'), r.reason?.message ?? String(r.reason))
+        }
       }
-    }
+    })
     return { hadFailures, addedKB }
   }
 
@@ -196,7 +203,7 @@ Options:
     const tasks = files.map(file => limit(() => utils.compression(file, dry, argv.quiet, cache)))
     const results = await Promise.allSettled(tasks)
 
-    const { hadFailures, addedKB } = processResults(results)
+    const { hadFailures, addedKB } = processResults(results, files)
     savedKB += addedKB
     return hadFailures
   }
@@ -207,7 +214,7 @@ Options:
     const tasks = files.map(file => limit(() => utils.conversion(file, dry, keepOriginal, argv.quiet)))
     const results = await Promise.allSettled(tasks)
 
-    const { hadFailures, addedKB } = processResults(results)
+    const { hadFailures, addedKB } = processResults(results, files)
     savedKB += addedKB
     return hadFailures
   }
@@ -275,10 +282,14 @@ Options:
   let totalFiles = 0
 
   if (argv.staged) {
-    const git = simpleGit()
     try {
-      const diffOutput = await git.raw(['diff', '--name-only', '--cached', '--diff-filter=ACMRT'])
-      const stagedFiles = diffOutput.split('\n').map(s => s.trim()).filter(Boolean)
+      // Git reports paths relative to the repository root
+      process.chdir((await simpleGit().revparse(['--show-toplevel'])).trim())
+      const git = simpleGit()
+      // `-z` keeps Git from quoting names with non-ASCII characters
+      const listFiles = async (args) => (await git.raw(['diff', '--name-only', '-z', ...args])).split('\0').filter(Boolean)
+      const stagedFiles = await listFiles(['--cached', '--diff-filter=ACMRT'])
+      const unstagedFiles = new Set(await listFiles([]))
 
       // Collect compression and conversion candidates in parallel
       const [compressionFiles, conversionFiles] = await Promise.all([
@@ -293,6 +304,21 @@ Options:
         convert(conversionFiles, argv.dry, argv['keep-heic'])
       ])
       hadFailures = compFailed || convFailed
+
+      // Stages the results so the commit includes them, except where that would
+      // also stage changes the user left unstaged
+      const pathsToStage = []
+      for (const [file, changed] of changesBySource) {
+        if (unstagedFiles.has(file)) {
+          console.warn(styleStderr('yellow', `Didn’t stage ${changed.join(', ')}, as ${file} has unstaged changes`))
+        } else {
+          pathsToStage.push(...changed)
+        }
+      }
+      // Batched to stay within command-line length limits
+      for (let i = 0; i < pathsToStage.length; i += 100) {
+        await git.raw(['add', '--all', '--', ...pathsToStage.slice(i, i + 100).map(p => `:(literal)${p}`)])
+      }
     } catch (err) {
       console.error(err)
       hadFailures = true

package/src/utils.js

@@ -8,6 +8,8 @@ import { styleText } from 'node:util'
 import decode from 'heic-decode'
 
 const MAX_FILE_SIZE = 100 * 1024 * 1024 // 100 MB
+// Limits palette reduction, as checking it holds two decoded copies of the image (320 MB at the limit)
+const MAX_PIXELS_QUANTIZED = 40 * 1000 * 1000 // 40 megapixels
 
 // Compression configuration for each format (as sharp names them)
 const formatConfigs = {
@@ -16,8 +18,13 @@ const formatConfigs = {
     settings: { quality: 100 }
   },
   png: {
-    options: { pages: -1 },
-    settings: { animated: true, compressionLevel: 9, quality: 100 } // Still waiting for APNG support though (`animated` doesn’t seem to have an effect), https://github.com/lovell/sharp/issues/2375
+    options: {},
+    settings: { compressionLevel: 9 },
+    // Lossless for images that already use a palette
+    settingsPalette: { palette: true, quality: 100, dither: 0, effort: 10 },
+    // Reduces other images to 256 colors, kept only where that doesn’t visibly change them
+    // (effort 7 being as small and as close to the original as 10, in two-thirds of the time)
+    settingsQuantized: { palette: true, quality: 100, effort: 7 }
   },
   gif: {
     options: { pages: -1 },
@@ -40,10 +47,114 @@ const formatConfigs = {
   }
 }
 
+// Checks the profile description, as sRGB profiles only add bytes (sRGB being
+// what images without a profile are taken to be)
+const isSrgbProfile = (icc) => {
+  try {
+    const count = icc.readUInt32BE(128)
+    for (let i = 0; i < count; i++) {
+      const entry = 132 + i * 12
+      if (icc.toString('latin1', entry, entry + 4) !== 'desc') continue
+      const offset = icc.readUInt32BE(entry + 4)
+      // Removing null bytes covers both ASCII and UTF-16 descriptions
+      return icc.toString('latin1', offset, offset + icc.readUInt32BE(entry + 8)).replaceAll('\0', '').includes('sRGB')
+    }
+  } catch {
+    // Malformed profiles are kept
+  }
+  return false
+}
+
+// Keeps what affects how the image looks—orientation, color profile, CMYK, grayscale, and
+// high bit depth—while dropping all other metadata
+const createPipeline = (input, format, metadata, quantize = false) => {
+  const { options, settings, settingsPalette, settingsQuantized } = formatConfigs[format]
+  const settingsImage = quantize ? settingsQuantized : metadata.isPalette && settingsPalette ? settingsPalette : settings
+  let pipeline = sharp(input, options)
+  if (metadata.icc && !isSrgbProfile(metadata.icc)) pipeline = pipeline.keepIccProfile()
+  if (metadata.orientation > 1) pipeline = pipeline.withExif({ IFD0: { Orientation: String(metadata.orientation) } })
+  if (metadata.space === 'cmyk') pipeline = pipeline.pipelineColourspace('cmyk').toColourspace('cmyk')
+  if (format === 'png' && !quantize && (metadata.bitsPerSample > 8 || metadata.space === 'b-w')) {
+    return pipeline.toColourspace(metadata.space).toFormat(format, settingsImage)
+  }
+  if (format === 'avif' && metadata.bitsPerSample > 8) {
+    return pipeline.toColourspace(metadata.space).toFormat(format, { ...settingsImage, bitdepth: metadata.bitsPerSample })
+  }
+  return pipeline.toFormat(format, settingsImage)
+}
+
+// Compares lightly blurred pixels, so that dithering, which the eye blends, counts less than
+// shifted colors and blotches—overall, and in every 16×16 area, so that small changes, like
+// icons losing their color, count, too; the limits follow a visual review, in which established
+// metrics (SSIMULACRA2, DSSIM, SSIM) predicted visible changes less well
+const isVisiblySame = async (input, output) => {
+  const read = (data) => sharp(data).toColourspace('srgb').ensureAlpha().blur(1).raw().toBuffer({ resolveWithObject: true })
+  const [{ data: pixelsInput, info: { width, height } }, { data: pixelsOutput }] = await Promise.all([read(input), read(output)])
+  // Counts differences by value, as sorting them would hold up other images for longer
+  const histogram = new Uint32Array(256)
+  // Sums differences in 8×8 cells, of which 2×2 make up an area
+  const widthCells = Math.ceil(width / 8)
+  const heightCells = Math.ceil(height / 8)
+  const cells = new Float64Array(widthCells * heightCells)
+  let sum = 0
+  for (let y = 0; y < height; y++) {
+    for (let x = 0; x < width; x++) {
+      const i = (y * width + x) * 4
+      // Weighs colors by opacity, as colors of transparent pixels don’t show
+      const alphaInput = pixelsInput[i + 3]
+      const alphaOutput = pixelsOutput[i + 3]
+      let diff = Math.abs(alphaInput - alphaOutput) * 255
+      for (let c = i; c < i + 3; c++) {
+        const diffColor = Math.abs(pixelsInput[c] * alphaInput - pixelsOutput[c] * alphaOutput)
+        if (diffColor > diff) diff = diffColor
+      }
+      histogram[(diff / 255) | 0]++
+      sum += diff / 255
+      cells[(y >> 3) * widthCells + (x >> 3)] += diff / 255
+    }
+  }
+  const count = width * height
+  // The difference that 99.9% of pixels stay within
+  let seen = 0
+  let diffTop = 0
+  while ((seen += histogram[diffTop]) <= Math.floor(count * 0.999)) diffTop++
+  // The largest average difference of an area, with areas overlapping by half
+  let diffArea = 0
+  for (let y = 0; y < heightCells - 1; y++) {
+    for (let x = 0; x < widthCells - 1; x++) {
+      const k = y * widthCells + x
+      const diffAverage = (cells[k] + cells[k + 1] + cells[k + widthCells] + cells[k + widthCells + 1]) / 256
+      if (diffAverage > diffArea) diffArea = diffAverage
+    }
+  }
+  return sum / count <= 3 && diffTop <= 20 && diffArea <= 5.5
+}
+
+// Reduces PNG images to 256 colors where that doesn’t visibly change them—once reduced, they
+// use a palette, which later runs keep losslessly—and compresses all other images as configured
+const encode = async (input, format) => {
+  const metadata = await sharp(input).metadata()
+  // Leaves out grayscale images, as palettes don’t make 8-bit ones smaller, and 16-bit images,
+  // whose precision the comparison can’t see
+  if (format === 'png' && !metadata.isPalette && metadata.channels > 1 && !(metadata.bitsPerSample > 8) && metadata.width * metadata.height <= MAX_PIXELS_QUANTIZED) {
+    const output = await createPipeline(input, format, metadata, true).toBuffer()
+    // Gives way to lossless compression where dithering makes images larger, as in some screenshots
+    if (output.length < input.length && await isVisiblySame(input, output)) return output
+  }
+  return createPipeline(input, format, metadata).toBuffer()
+}
+
+// Detects APNG images, of which libvips would keep only the first frame
+const isAnimatedPng = (data) => {
+  const actl = data.indexOf('acTL')
+  const idat = data.indexOf('IDAT')
+  return actl !== -1 && (idat === -1 || actl < idat)
+}
+
 export const hash = (data) => createHash('sha256').update(data).digest('hex')
 
 // Identifies what compression results depend on, to tell when recorded results no longer apply
-const fingerprint = hash(JSON.stringify({ formatConfigs, versions: sharp.versions })).slice(0, 16)
+const fingerprint = hash(JSON.stringify({ formatConfigs, MAX_PIXELS_QUANTIZED, pipeline: [encode, createPipeline, isSrgbProfile, isVisiblySame].map(String), versions: sharp.versions })).slice(0, 16)
 
 // `styleText` detects color on `process.stdout` by default, so anything bound
 // for STDERR has to name that stream—otherwise a redirected STDERR collects
@@ -71,6 +182,9 @@ const retryFileOperation = async (operation, maxRetries = 5, delayMs = 100) => {
   }
 }
 
+// Bytes saved, and the paths written or removed (for staging)
+const noChange = () => ({ saved: 0, changed: [] })
+
 const compression = async (filename, dry, quiet = false, cache) => {
   const filenameBackup = `${filename}.bak`
   const fileSizeBefore = await size(filename)
@@ -79,12 +193,12 @@ const compression = async (filename, dry, quiet = false, cache) => {
 
   if (fileSizeBefore === 0) {
     logMessage(`Skipped ${filename} (${sizeReadable(fileSizeBefore)})`, dry, 'yellow', quiet)
-    return 0
+    return noChange()
   }
 
   if (fileSizeBefore > MAX_FILE_SIZE) {
     logMessage(`Skipped ${filename} (file too large: ${sizeReadable(fileSizeBefore)})`, dry, 'yellow', quiet)
-    return 0
+    return noChange()
   }
 
   // Place temp file next to the original to maximize same-device atomic rename
@@ -103,8 +217,7 @@ const compression = async (filename, dry, quiet = false, cache) => {
     }
 
     const outputFormat = ext === 'jpg' ? 'jpeg' : ext // sharp uses “jpeg” instead of “jpg”
-    const config = formatConfigs[outputFormat]
-    if (!config) {
+    if (!formatConfigs[outputFormat]) {
       throw new Error(`Unsupported file type for ${filename}`)
     }
 
@@ -115,12 +228,15 @@ const compression = async (filename, dry, quiet = false, cache) => {
     const entryInput = cache ? toEntry(input) : undefined
     if (cache?.has(entryInput)) {
       logMessage(`Skipped ${filename} (already processed)`, dry, 'white', quiet)
-      return 0
+      return noChange()
     }
 
-    const output = await sharp(input, config.options)
-      .toFormat(outputFormat, config.settings)
-      .toBuffer()
+    if (outputFormat === 'png' && isAnimatedPng(input)) {
+      logMessage(`Skipped ${filename} (animated PNG)`, dry, 'yellow', quiet)
+      return noChange()
+    }
+
+    const output = await encode(input, outputFormat)
     const fileSizeAfter = output.length
     if (fileSizeAfter === 0) {
       throw new Error('Compressed file size is 0')
@@ -166,11 +282,11 @@ const compression = async (filename, dry, quiet = false, cache) => {
 
     logMessage(`${status} ${filename} (${details})`, dry, color, quiet)
 
-    if (dry) {
-      return 0
+    if (dry || !replacementSucceeded) {
+      return noChange()
     }
 
-    return fileSizeAfter < fileSizeBefore ? fileSizeBefore - fileSizeAfter : 0
+    return { saved: fileSizeBefore - fileSizeAfter, changed: [filename] }
 
   } catch (err) {
 
@@ -179,6 +295,7 @@ const compression = async (filename, dry, quiet = false, cache) => {
     if (msg.includes('corrupt header') ||
       msg.includes('unexpected end of') ||
       msg.includes('invalid image') ||
+      msg.includes('unsupported image format') ||
       msg.includes('gifload:') ||
       msg.includes('pngload:') ||
       msg.includes('jpegload:') ||
@@ -186,10 +303,9 @@ const compression = async (filename, dry, quiet = false, cache) => {
       msg.includes('avifload:')
     ) {
       logMessage(`Skipped ${filename} (corrupt file)`, dry, 'yellow', quiet)
-    } else {
-      console.error(styleStderr('red', `Error compressing ${filename}:`), err)
+      return noChange()
     }
-    return 0
+    throw new Error(`Error compressing ${filename}`, { cause: err })
 
   } finally {
 
@@ -229,17 +345,17 @@ const conversion = async (filename, dry, keepOriginal, quiet = false) => {
 
   if (fileSizeBefore === 0) {
     logMessage(`Skipped ${filename} (${sizeReadable(fileSizeBefore)})`, dry, 'yellow', quiet)
-    return 0
+    return noChange()
   }
 
   if (fileSizeBefore > MAX_FILE_SIZE) {
     logMessage(`Skipped ${filename} (file too large: ${sizeReadable(fileSizeBefore)})`, dry, 'yellow', quiet)
-    return 0
+    return noChange()
   }
 
   if (!/\.hei[cf]$/i.test(filename)) {
     logMessage(`Skipped ${filename} (not a HEIC/HEIF file)`, dry, 'yellow', quiet)
-    return 0
+    return noChange()
   }
 
   const avifPath = filename.replace(/\.hei[cf]$/i, '.avif')
@@ -248,7 +364,7 @@ const conversion = async (filename, dry, keepOriginal, quiet = false) => {
   try {
     await fs.access(avifPath)
     logMessage(`Skipped ${filename} (${path.basename(avifPath)} already exists)`, dry, 'yellow', quiet)
-    return 0
+    return noChange()
   } catch {
     // File does not exist—proceed with conversion
   }
@@ -270,7 +386,7 @@ const conversion = async (filename, dry, keepOriginal, quiet = false) => {
 
     if (dry) {
       await retryFileOperation(() => fs.unlink(avifPath))
-      return 0
+      return noChange()
     }
 
     // Delete original HEIC/HEIF file unless `--keep-heic` is set
@@ -278,7 +394,10 @@ const conversion = async (filename, dry, keepOriginal, quiet = false) => {
       await retryFileOperation(() => fs.unlink(filename))
     }
 
-    return fileSizeAfter < fileSizeBefore ? fileSizeBefore - fileSizeAfter : 0
+    return {
+      saved: fileSizeAfter < fileSizeBefore ? fileSizeBefore - fileSizeAfter : 0,
+      changed: keepOriginal ? [avifPath] : [avifPath, filename]
+    }
 
   } catch (err) {
     // Clean up partial AVIF if it was created
@@ -295,10 +414,9 @@ const conversion = async (filename, dry, keepOriginal, quiet = false) => {
       err.message.includes('HEIF image not found')
     )) {
       logMessage(`Skipped ${filename} (corrupt or unsupported HEIC/HEIF file)`, dry, 'yellow', quiet)
-    } else {
-      console.error(styleStderr('red', `Error converting ${filename}:`), err)
+      return noChange()
     }
-    return 0
+    throw new Error(`Error converting ${filename}`, { cause: err })
   }
 }