image-guard 5.4.1 → 5.5.0

package/README.md

@@ -121,6 +121,8 @@ Image Guard takes an optional path to the directory to process—`npx image-guar
 
 * `--staged` (recommended with automated use) triggers a mode that watches PNG, JPG, GIF, WebP, and AVIF files (and HEIC/HEIF, when `--heic-to-avif` is used) from staged changes (`git diff --cached`) and only processes those files—that approach makes Image Guard more efficient in operation. Because the file set comes from Git, `--staged` can’t be combined with a path.
 
+* `--no-cache` processes all images, including those recorded as already processed (see [“Record of Processed Images”](#record-of-processed-images)).
+
 * `--help`/`-h` lists the arguments and options.
 
 * `--version`/`-V` prints the version number.
@@ -146,11 +148,17 @@ Automated compression works by monitoring whether a given [change list](https://
 
 Through this approach, though glossed over here, Image Guard makes up for what’s missing or complicated in other packages, namely easy, near-riskless, automatable, resource-friendly in-repo optimization.
 
+### Record of Processed Images
+
+To keep repeat runs fast, Image Guard remembers which images it has already processed (compressed, or found not to shrink any further) by their content, and skips them in later runs. This also protects JPG and GIF images from being re-encoded over and over. An image is processed again as soon as its content changes, and all images are when the compression settings or the underlying image libraries change (as through an update).
+
+The record is kept in node_modules/.cache/image-guard of the project being processed, or—where there is no node_modules folder, as is common with `npx`—in the per-user cache folder (~/Library/Caches/image-guard on macOS, %LOCALAPPDATA%\image-guard\Cache on Windows, and ~/.cache/image-guard elsewhere). The `IMAGE_GUARD_CACHE_DIR` environment variable sets a different folder—for example, to persist the record in CI. Deleting the folder, or using `--no-cache`, makes Image Guard process all images again.
+
 ## Why Use Image Guard?
 
 You use Image Guard when you need a simple, automatable, robust solution to compress images in a way that limits unnecessary image payload right from the start, in your repositories, and that reduces the risk that entirely uncompressed images go into production.
 
-As Image Guard compresses near-losslessly, there’s little risk of quality issues from compression. (Lossless compression is not possible for every image format, however, so there’s a risk when excessively iterating over the same images. Doing so may eventually degrade quality.)
+As Image Guard compresses near-losslessly, there’s little risk of quality issues from compression. (Lossless compression is not possible for every image format, however, so there’s a risk when excessively iterating over the same images. Doing so may eventually degrade quality. The [record of processed images](#record-of-processed-images) prevents this, unless it’s bypassed or cleared.)
 
 ## What Does Image Guard _Not_ Do?
 
@@ -158,11 +166,24 @@ Image Guard is no substitute for image fine-tuning and micro-optimization. That
 
 That is, micro-optimization still needs to be taken care of through other means, whether manually or through tools. Image Guard just solves the problem that images are checked in or go live that are not compressed _at all_.
 
-## What’s Next?
+## Working on Image Guard
+
+### Benchmarking
+
+To check how a change affects speed, memory, and compression, run the benchmark on any local folder with images:
+
+```shell
+npm run benchmark -- ~/Projects
+```
+
+The benchmark copies a deterministic sample of the images (honoring .gitignore) to a temporary folder, runs Image Guard on fresh copies of it several times (each without a record of processed images), and reports the median time, peak memory, and bytes saved. Your images are not modified.
 
-There are a few ideas, like adding light SVG support, or ensuring compatibility with projects in which the project’s .git folder is not at the same level as its package.json (currently, automatic mode doesn’t work in these cases).
+* `--runs <n>` sets the number of runs (default: 3).
+* `--sample <n>` sets the approximate sample size (default: 300). Each format is sampled in proportion to its share, with at least 10 files per format where available.
+* `--save` saves the result as the baseline (benchmark/baseline.json, ignored by Git), which later runs on the same sample compare against.
+* `--profile` additionally compresses each image on its own, in dry mode, and reports the time per format and the slowest files.
 
-Feedback is appreciated: Please [file an issue](https://github.com/j9t/image-guard/issues) or send a pull request. Thank you!
+To compare a branch with `main`, run the benchmark with `--save` on `main`, then without it on the branch. Differences of around 10% can be noise; use more runs, and repeat a run on `main`, before relying on smaller ones.
 
 ## License
 

package/package.json

@@ -6,16 +6,20 @@
     "imagemin-merlin": "bin/image-guard.sh"
   },
   "dependencies": {
-    "globby": "^16.2.4",
     "heic-decode": "^2.1.0",
-    "sharp": "^0.35.4",
-    "simple-git": "^3.36.0"
+    "ignore": "^7.0.12",
+    "sharp": "^0.35.5",
+    "simple-git": "^4.0.2",
+    "tinyglobby": "^0.2.17"
   },
   "description": "Near-lossless image compressor that’s easy to use and easy to automate",
   "devDependencies": {
     "@eslint/js": "^10.0.1",
-    "eslint": "^10.11.0",
-    "globals": "^17.12.0"
+    "eslint": "^10.12.0",
+    "globals": "^17.13.0"
+  },
+  "engines": {
+    "node": ">=22.13"
   },
   "files": [
     "bin/image-guard.js",
@@ -49,11 +53,12 @@
     "url": "git+https://github.com/j9t/image-guard.git"
   },
   "scripts": {
+    "benchmark": "node benchmark/benchmark.js",
     "lint": "eslint .",
     "lint:fix": "eslint . --fix",
     "prepare": "git config core.hooksPath .githooks || true",
-    "test": "node --test bin/image-guard.test.js"
+    "test": "node --test bin/image-guard.test.js benchmark/benchmark.test.js"
   },
   "type": "module",
-  "version": "5.4.1"
+  "version": "5.5.0"
 }

package/src/cache.js

@@ -0,0 +1,97 @@
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
+import { styleStderr } from './utils.js'
+
+// Records made with other settings or encoder versions are deleted once unused for this long
+const MAX_AGE_STALE = 30 * 24 * 60 * 60 * 1000
+
+// Matches record files only, as the folder may be shared (see `IMAGE_GUARD_CACHE_DIR`)
+const PATTERN_RECORD = /^[0-9a-f]{16}\.txt$/
+
+// The nearest project’s node_modules/.cache, else the per-user cache folder
+export function findDirCache(dirStart) {
+  if (process.env.IMAGE_GUARD_CACHE_DIR) return path.resolve(process.env.IMAGE_GUARD_CACHE_DIR)
+  for (let current = path.resolve(dirStart); ; current = path.dirname(current)) {
+    if (fs.existsSync(path.join(current, 'package.json'))) {
+      const dirModules = path.join(current, 'node_modules')
+      if (fs.existsSync(dirModules)) return path.join(dirModules, '.cache', 'image-guard')
+      break
+    }
+    if (path.dirname(current) === current) break
+  }
+  if (process.platform === 'darwin') return path.join(os.homedir(), 'Library', 'Caches', 'image-guard')
+  if (process.platform === 'win32') return path.join(process.env.LOCALAPPDATA || path.join(os.homedir(), 'AppData', 'Local'), 'image-guard', 'Cache')
+  return path.join(process.env.XDG_CACHE_HOME || path.join(os.homedir(), '.cache'), 'image-guard')
+}
+
+// Remembers images that compression can’t shrink any further (as format and
+// content hash); the fingerprint keeps records apart per compression settings
+// and encoder version, so changing either processes all images again
+export function createCache(dirStart, fingerprint) {
+  const dirCache = findDirCache(dirStart)
+  const fileRecord = path.join(dirCache, `${fingerprint}.txt`)
+  const added = []
+  let entries = new Set()
+  let hits = 0
+
+  try {
+    entries = new Set(fs.readFileSync(fileRecord, 'utf8').split('\n').filter(Boolean))
+  } catch (err) {
+    if (err.code !== 'ENOENT') console.warn(styleStderr('yellow', `Could not read the record of processed images (${err.message})`))
+  }
+
+  return {
+    has(entry) {
+      const found = entries.has(entry)
+      if (found) hits++
+      return found
+    },
+    add(entry) {
+      if (entries.has(entry)) return
+      entries.add(entry)
+      added.push(entry)
+    },
+    get hits() {
+      return hits
+    },
+    save() {
+      if (added.length === 0) {
+        // Keeps runs with other settings from deleting a record that’s still in use
+        if (hits > 0) {
+          try {
+            const now = new Date()
+            fs.utimesSync(fileRecord, now, now)
+          } catch {
+            // Best-effort—at worst, the images are processed once more
+          }
+        }
+        return
+      }
+      try {
+        fs.mkdirSync(dirCache, { recursive: true })
+        // Appending (rather than rewriting) keeps what concurrent runs add
+        fs.appendFileSync(fileRecord, added.map(entry => `${entry}\n`).join(''))
+      } catch (err) {
+        console.warn(styleStderr('yellow', `Could not save the record of processed images (${err.message})`))
+        return
+      }
+      let names = []
+      try {
+        names = fs.readdirSync(dirCache)
+      } catch {
+        // Nothing to clean up if the folder can’t be read
+      }
+      for (const name of names) {
+        const file = path.join(dirCache, name)
+        if (file === fileRecord || !PATTERN_RECORD.test(name)) continue
+        try {
+          if (Date.now() - fs.statSync(file).mtimeMs > MAX_AGE_STALE) fs.rmSync(file)
+        } catch (err) {
+          // Another run may have deleted it already
+          if (err.code !== 'ENOENT') console.warn(styleStderr('yellow', `Could not delete outdated record ${file} (${err.message})`))
+        }
+      }
+    }
+  }
+}

package/src/gitignore.js

@@ -0,0 +1,50 @@
+import fs from 'node:fs'
+import path from 'node:path'
+import ignore from 'ignore'
+
+function findGitRoot(dir) {
+  for (let current = dir; ; current = path.dirname(current)) {
+    if (fs.existsSync(path.join(current, '.git'))) return current
+    if (path.dirname(current) === current) return undefined
+  }
+}
+
+// Returns a predicate for absolute paths under `cwd` that applies .gitignore
+// files from the repository root (or `cwd` outside a repository) down to each
+// path, as Git does: Deeper rules take precedence, and nothing inside an
+// ignored directory can be re-included
+export function createGitignoreFilter(cwd) {
+  const base = path.resolve(cwd)
+  const top = findGitRoot(base) ?? base
+  const rulesByDir = new Map()
+
+  const getRules = (dir) => {
+    if (!rulesByDir.has(dir)) {
+      let rules
+      try {
+        rules = ignore().add(fs.readFileSync(path.join(dir, '.gitignore'), 'utf8'))
+      } catch (err) {
+        if (err.code !== 'ENOENT' && err.code !== 'ENOTDIR') throw err
+      }
+      rulesByDir.set(dir, rules)
+    }
+    return rulesByDir.get(dir)
+  }
+
+  return (file) => {
+    const segments = path.relative(top, file).split(path.sep)
+    for (let i = 1; i <= segments.length; i++) {
+      const suffix = i < segments.length ? '/' : ''
+      let ignored = false
+      for (let j = 0; j < i; j++) {
+        const rules = getRules(path.join(top, ...segments.slice(0, j)))
+        if (!rules) continue
+        const result = rules.test(segments.slice(j, i).join('/') + suffix)
+        if (result.ignored) ignored = true
+        else if (result.unignored) ignored = false
+      }
+      if (ignored) return true
+    }
+    return false
+  }
+}

package/src/index.js

@@ -1,12 +1,14 @@
 // This file, which had been forked from imagemin-merlin, was modified for image-guard: https://github.com/sumcumo/imagemin-merlin/compare/master...j9t:master
 
-import { globby, convertPathToPattern } from 'globby'
-import simpleGit from 'simple-git'
+import { glob, convertPathToPattern } from 'tinyglobby'
+import { simpleGit } from 'simple-git'
 import { parseArgs, styleText } from 'node:util'
 import fsSync from 'node:fs'
 import os from 'node:os'
 import path from 'node:path'
 import sharp from 'sharp'
+import { createCache } from './cache.js'
+import { createGitignoreFilter } from './gitignore.js'
 import { styleStderr, utils } from './utils.js'
 
 // Files to be compressed
@@ -30,6 +32,7 @@ export async function runImageGuard() {
     quiet: { type: 'boolean', short: 'q', default: false },
     dry: { type: 'boolean', short: 'd', default: false },
     staged: { type: 'boolean', default: false },
+    'no-cache': { type: 'boolean', default: false },
     help: { type: 'boolean', short: 'h', default: false },
     version: { type: 'boolean', short: 'V', default: false }
   }
@@ -58,6 +61,7 @@ Options:
   -q, --quiet           Print only the final summary
   -d, --dry             Show what would change without writing any files
       --staged          Process only images staged in Git (not combinable with a directory)
+      --no-cache        Process images even if recorded as already processed
   -h, --help            Show this help
   -V, --version         Show the version number`)
     return
@@ -88,11 +92,20 @@ Options:
     throw setupError(`Not a directory: ${dir}`)
   }
 
+  // Remembers images that can’t be compressed further, so later runs can skip them
+  const cache = argv['no-cache'] ? undefined : createCache(argv.staged ? '.' : dir, utils.fingerprint)
+
+  const noteCached = () => {
+    const hits = cache?.hits ?? 0
+    if (hits === 0) return ''
+    return hits === 1 ? ' 1 image was skipped as already processed.' : ` ${hits} images were skipped as already processed.`
+  }
+
   // Share status
   const summary = (run, includesConversion = false) => {
     if (run) {
       const action = includesConversion ? 'compression and conversion' : 'compression'
-      console.info(styleText(['bold'], `\nDefensive base ${action} completed. You saved ${utils.sizeReadable(savedKB)}.`))
+      console.info(styleText(['bold'], `\nDefensive base ${action} completed. You saved ${utils.sizeReadable(savedKB)}.${noteCached()}`))
     } else {
       const what = includesConversion ? 'images to compress or convert' : 'images to compress'
       console.info(styleText(['bold'], `There were no ${what}.`))
@@ -147,8 +160,14 @@ Options:
     })
   }
 
-  const desiredFileConcurrency = Math.min(os.cpus().length, 4)
-  const perTaskThreads = Math.max(1, Math.floor(os.cpus().length / Math.max(1, desiredFileConcurrency)))
+  // Capped to bound memory, as each file in flight holds its decoded pixels
+  const desiredFileConcurrency = Math.min(os.availableParallelism(), 8)
+  const perTaskThreads = Math.max(1, Math.floor(os.availableParallelism() / desiredFileConcurrency))
+
+  // sharp runs on libuv’s thread pool (4 threads by default), which caps files
+  // in flight regardless of the limiter; this must happen before the first
+  // asynchronous I/O and respects a user setting
+  process.env.UV_THREADPOOL_SIZE ??= String(desiredFileConcurrency)
   try {
     sharp.concurrency(perTaskThreads)
   } catch {
@@ -174,7 +193,7 @@ Options:
   const compress = async (files, dry) => {
     if (files.length === 0) return false
 
-    const tasks = files.map(file => limit(() => utils.compression(file, dry, argv.quiet)))
+    const tasks = files.map(file => limit(() => utils.compression(file, dry, argv.quiet, cache)))
     const results = await Promise.allSettled(tasks)
 
     const { hadFailures, addedKB } = processResults(results)
@@ -194,7 +213,7 @@ Options:
   }
 
   // Plain patterns also get a “/**” variant so directories are excluded with
-  // their contents: “dir/” resolves only where globby can stat it, and a bare
+  // their contents: “dir/” resolves only where tinyglobby can stat it, and a bare
   // “dir” only covers what the walk descends into
   const getIgnorePatterns = (ignore) => {
     return (ignore || '')
@@ -223,14 +242,14 @@ Options:
   // lookup are relative to the searched directory, not to the shell’s—and the
   // results are rejoined for display and file access
   const findFiles = async (patterns, options = {}) => {
-    const files = await globby(patterns, {
+    const files = await glob(patterns, {
       cwd: dir,
-      gitignore: true,
       onlyFiles: true,
       caseSensitiveMatch: false,
       ...options
     })
-    return files.map(file => path.join(dir, file))
+    const isGitignored = createGitignoreFilter(dir)
+    return files.map(file => path.join(dir, file)).filter(file => !isGitignored(path.resolve(file)))
   }
 
   const filterStagedFiles = async (stagedFiles, types) => {
@@ -240,12 +259,13 @@ Options:
 
     if (ignoreList.length > 0) {
       const escapedPaths = byExt.map(p => convertPathToPattern(p))
-      return globby([...escapedPaths, ...ignoreList], {
-        gitignore: true,
+      const files = await glob([...escapedPaths, ...ignoreList], {
         expandDirectories: false,
         onlyFiles: true,
         caseSensitiveMatch: false
       })
+      const isGitignored = createGitignoreFilter('.')
+      return files.filter(file => !isGitignored(path.resolve(file)))
     }
     return byExt
   }
@@ -301,11 +321,13 @@ Options:
     }
   }
 
+  if (!argv.dry) cache?.save()
+
   if (hadFailures) {
     process.exitCode = 1
     if (totalFiles > 0) {
       const action = doConversion ? 'compression and conversion' : 'compression'
-      const savings = savedKB > 0 ? ` You saved ${utils.sizeReadable(savedKB)}.` : ''
+      const savings = `${savedKB > 0 ? ` You saved ${utils.sizeReadable(savedKB)}.` : ''}${noteCached()}`
       console.info(styleText(['bold'], `\nDefensive base ${action} partially completed (some tasks failed).${savings}`))
     } else {
       summary(false, doConversion)

package/src/utils.js

@@ -1,5 +1,6 @@
 // This file, which had been forked from imagemin-merlin, was modified for image-guard: https://github.com/sumcumo/imagemin-merlin/compare/master...j9t:master
 
+import { createHash } from 'node:crypto'
 import { promises as fs } from 'node:fs'
 import path from 'node:path'
 import sharp from 'sharp'
@@ -8,6 +9,42 @@ import decode from 'heic-decode'
 
 const MAX_FILE_SIZE = 100 * 1024 * 1024 // 100 MB
 
+// Compression configuration for each format (as sharp names them)
+const formatConfigs = {
+  jpeg: {
+    options: {},
+    settings: { quality: 100 }
+  },
+  png: {
+    options: { pages: -1 },
+    settings: { animated: true, compressionLevel: 9, quality: 100 } // Still waiting for APNG support though (`animated` doesn’t seem to have an effect), https://github.com/lovell/sharp/issues/2375
+  },
+  gif: {
+    options: { pages: -1 },
+    settings: {
+      reuse: true,               // Preserve original palette for lossless quality (default)
+      effort: 10,                // Maximum compression effort
+      dither: 0,                 // No dithering = lossless quality
+      interFrameMaxError: 0,     // No transparency errors = lossless (default)
+      interPaletteMaxError: 0,   // Perfect palette match = lossless
+      colors: 256                // Full palette available (default)
+    }
+  },
+  webp: {
+    options: { pages: -1 },
+    settings: { animated: true, lossless: true }
+  },
+  avif: {
+    options: {},
+    settings: { effort: 5, lossless: true } // Temporarily specifying effort, too, as per https://github.com/lovell/sharp/issues/4370#issuecomment-2798848572
+  }
+}
+
+export const hash = (data) => createHash('sha256').update(data).digest('hex')
+
+// Identifies what compression results depend on, to tell when recorded results no longer apply
+const fingerprint = hash(JSON.stringify({ formatConfigs, versions: sharp.versions })).slice(0, 16)
+
 // `styleText` detects color on `process.stdout` by default, so anything bound
 // for STDERR has to name that stream—otherwise a redirected STDERR collects
 // escape codes whenever STDOUT happens to be a terminal
@@ -34,7 +71,7 @@ const retryFileOperation = async (operation, maxRetries = 5, delayMs = 100) => {
   }
 }
 
-const compression = async (filename, dry, quiet = false) => {
+const compression = async (filename, dry, quiet = false, cache) => {
   const filenameBackup = `${filename}.bak`
   const fileSizeBefore = await size(filename)
   // Track whether original file was successfully replaced
@@ -66,48 +103,28 @@ const compression = async (filename, dry, quiet = false) => {
     }
 
     const outputFormat = ext === 'jpg' ? 'jpeg' : ext // sharp uses “jpeg” instead of “jpg”
-
-    // Compression configuration for each format
-    const formatConfigs = {
-      png: {
-        options: { pages: -1 },
-        settings: { animated: true, compressionLevel: 9, quality: 100 } // Still waiting for APNG support though (`animated` doesn’t seem to have an effect), https://github.com/lovell/sharp/issues/2375
-      },
-      gif: {
-        options: { pages: -1 },
-        settings: {
-          reuse: true,               // Preserve original palette for lossless quality (default)
-          effort: 10,                // Maximum compression effort
-          dither: 0,                 // No dithering = lossless quality
-          interFrameMaxError: 0,     // No transparency errors = lossless (default)
-          interPaletteMaxError: 0,   // Perfect palette match = lossless
-          colors: 256                // Full palette available (default)
-        }
-      },
-      webp: {
-        options: { pages: -1 },
-        settings: { animated: true, lossless: true }
-      },
-      avif: {
-        options: {},
-        settings: { effort: 5, lossless: true } // Temporarily specifying effort, too, as per https://github.com/lovell/sharp/issues/4370#issuecomment-2798848572
-      }
+    const config = formatConfigs[outputFormat]
+    if (!config) {
+      throw new Error(`Unsupported file type for ${filename}`)
     }
 
-    // Apply format-specific compression or use default
-    const config = formatConfigs[outputFormat]
-    if (config) {
-      await sharp(filename, config.options)
-        .toFormat(outputFormat, config.settings)
-        .toFile(tempFilePath)
-    } else {
-      // Fallback for any other supported formats (like JPG)
-      await sharp(filename)
-        .toFormat(outputFormat, { quality: 100 })
-        .toFile(tempFilePath)
+    // The format follows the extension
+    const toEntry = (data) => `${outputFormat}:${hash(data)}`
+
+    const input = await fs.readFile(filename)
+    const entryInput = cache ? toEntry(input) : undefined
+    if (cache?.has(entryInput)) {
+      logMessage(`Skipped ${filename} (already processed)`, dry, 'white', quiet)
+      return 0
     }
 
-    const fileSizeAfter = await size(tempFilePath)
+    const output = await sharp(input, config.options)
+      .toFormat(outputFormat, config.settings)
+      .toBuffer()
+    const fileSizeAfter = output.length
+    if (fileSizeAfter === 0) {
+      throw new Error('Compressed file size is 0')
+    }
 
     let color = 'white'
     let status = 'Skipped'
@@ -120,6 +137,7 @@ const compression = async (filename, dry, quiet = false) => {
       if (!dry) {
         // Only now create a backup and replace the original
         await retryFileOperation(() => fs.copyFile(filename, filenameBackup))
+        await fs.writeFile(tempFilePath, output)
         // Prefer atomic rename when possible
         try {
           await retryFileOperation(() => fs.rename(tempFilePath, filename))
@@ -141,14 +159,15 @@ const compression = async (filename, dry, quiet = false) => {
       details = 'already compressed more effectively'
     }
 
+    // Records the file as it now is, so compressed results aren’t re-encoded again either
+    if (!dry) {
+      cache?.add(fileSizeAfter < fileSizeBefore ? toEntry(output) : entryInput)
+    }
+
     logMessage(`${status} ${filename} (${details})`, dry, color, quiet)
 
     if (dry) {
-      return 0 // Temp file cleaned up in finally
-    }
-
-    if (fileSizeAfter === 0) {
-      console.error(styleStderr('red', `Error compressing ${filename}: Compressed file size is 0`))
+      return 0
     }
 
     return fileSizeAfter < fileSizeBefore ? fileSizeBefore - fileSizeAfter : 0
@@ -174,7 +193,7 @@ const compression = async (filename, dry, quiet = false) => {
 
   } finally {
 
-    // Clean up temp file if it wasn’t consumed (covers dry-run, error, and no-improvement paths)
+    // Clean up temp file if it wasn’t consumed (covers error paths)
     if (!tempConsumed) {
       try {
         await retryFileOperation(() => fs.unlink(tempFilePath))
@@ -290,4 +309,4 @@ const size = async (file) => {
 
 const sizeReadable = (size) => `${(size / 1024).toFixed(2)} KB` // eslint-disable-line no-irregular-whitespace
 
-export const utils = { compression, conversion, sizeReadable }
+export const utils = { compression, conversion, sizeReadable, fingerprint }