zerobucket 0.5.0 → 0.6.0

package/README.md

@@ -10,15 +10,16 @@ This is the Node.js/TypeScript sibling of the [Python package](https://pypi.org/
 **It uses the exact same table**, so a Node service and a Python service can share one database
 and read each other's images.
 
-> **Status: early (0.5.x).** PostgreSQL **and SQLite**, each in classic and dedup modes, with S3-compatible
-> object-storage tiering and a `zerobucket` command line tool. MySQL and HEIC are on the roadmap
-> (see [What's not here yet](#whats-not-here-yet)).
+> **Status: early (0.6.x).** PostgreSQL **and SQLite**, each in classic and dedup modes, with S3-compatible
+> object-storage tiering, HEIC (iPhone photo) support and a `zerobucket` command line tool. MySQL is on the
+> roadmap (see [What's not here yet](#whats-not-here-yet)).
 
 ## Install
 
 ```bash
 npm install zerobucket pg sharp          # PostgreSQL
 npm install zerobucket better-sqlite3 sharp   # or SQLite (see below): no database server at all
+npm install libheif-js                         # optional: accept HEIC (iPhone) photos, see below
 ```
 
 `pg` (the Postgres driver), `better-sqlite3` (SQLite) and `sharp` (image decoding/optimizing) are peer dependencies, so you
@@ -177,6 +178,38 @@ await images.put(pdfBuf, { validator: new PDFValidator() }); // any ContentValid
 Optimize strips metadata, applies EXIF orientation first, never upscales, and re-validates its own
 output. Implement `ContentValidator` (`validate(data, { maxBytes })`) to store other content types.
 
+## HEIC (iPhone photos)
+
+iPhones upload HEIC by default, and most validators reject it. With the optional `libheif-js` installed, HEIC/HEIF
+is accepted like any other format: validated by really decoding it, stored **as uploaded** (`image/heic`), and
+convertible with `optimize`.
+
+```bash
+npm install libheif-js
+```
+
+```ts
+const id = await images.put(iphonePhoto); // stored unchanged, mimeType "image/heic"
+const id2 = await images.put(iphonePhoto, {
+  optimize: { format: "webp", maxWidth: 1600 },
+}); // convert on the way in
+```
+
+- **Convert for the web.** HEIC is only displayable in Safari. Serve a web format by uploading with
+  `optimize: { format: "webp" | "jpeg" | "png" }`, as above. HEIC orientation is applied during conversion.
+- **Decode only, by design.** There is no HEVC _encoder_ for Node, so `optimize` can convert **from** HEIC but not **to** it:
+  `optimize: { format: "heic" }` throws a clear error (Python's `pillow-heif` can encode), and `optimize: true` on a HEIC
+  needs an explicit target format instead of silently choosing one.
+- **Never blocks your server.** Decoding is ~2 s of CPU for a 12-megapixel photo. It runs in a dedicated worker thread, so
+  other requests keep being served (tested), and the worker never keeps your process from exiting.
+- **Safe on hostile input.** Pixel bombs are refused from the file header _before_ any decoding (`maxPixels`), truncated or
+  corrupt files are rejected with the decoder's own diagnosis, and a missing `libheif-js` is reported as "install it", never as "corrupt image".
+- Recognised by content (`ftyp` brands `heic heix hevc heim heis mif1 msf1`), never by file extension: same rules as the Python package, and both
+  accept and reject the same files (checked by the conformance harness against `pillow-heif`).
+- **License:** `libheif-js` is LGPL-3.0. It is an optional peer dependency that **you** install; it is not bundled with or redistributed by
+  this MIT-licensed package. If you use it, review its license for your distribution model.
+- If a file contains several images, the first top-level image is used.
+
 ## Command line
 
 The package installs a `zerobucket` command (`npx zerobucket ...`, or install globally with `npm i -g zerobucket`).
@@ -333,28 +366,29 @@ same corpus of valid and corrupt files identically.
 
 ## Differences from the Python package
 
-|                                                | Python                                                                     | npm                                                                                                            |
-| ---------------------------------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
-| API                                            | sync + async clients                                                       | one async client                                                                                               |
-| Malformed id (`"abc"`)                         | `StorageError` from Postgres                                               | treated as not found (`ImageNotFoundError`, no DB round trip), friendlier for `/images/:id` routes             |
-| `optimize`                                     | Pillow; EXIF stripped, not applied                                         | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                       |
-| `putMany` inserts                              | pipelined                                                                  | one transaction, statements run sequentially                                                                   |
-| Formats                                        | JPEG, PNG, WebP, HEIC (extra)                                              | JPEG, PNG, WebP (HEIC not yet)                                                                                 |
-| Event duration                                 | `duration_seconds`                                                         | `durationMs`                                                                                                   |
-| `migrateClassicToDedup`                        | loads the whole table into memory; fails if re-run or if any row is tiered | batched, safe to re-run, refuses tiered rows with a clear message                                              |
-| Dedup `putMany` / `deleteMany` locking         | arbitrary order (overlapping batches can deadlock)                         | sorted checksum order (cannot deadlock)                                                                        |
-| SQLite API                                     | synchronous `SQLiteBackend` + separate async backend                       | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                |
-| SQLite `onOperation` / busy retry              | none                                                                       | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                           |
-| SQLite stream deleted mid-read                 | sync adapter keeps reading a WAL snapshot (async adapter raises)           | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                         |
-| CLI                                            | PostgreSQL, classic mode only; `verify` crashes on tiered rows             | PostgreSQL **and** SQLite, classic **and** dedup; `info`/`verify` understand tiered rows; `migrate --to-dedup` |
-| `connection=` without an open transaction      | multi-statement ops rely on the driver's implicit transaction              | always atomic (own transaction or savepoint)                                                                   |
-| `delete(id, connection=...)` on a tiered image | deletes the S3 object immediately                                          | leaves the object until you delete it after commit (a rolled-back transaction can never lose data)             |
-| S3 delete failure after `delete()`             | raises after the row is gone                                               | never fails the delete; reported via `onOperation`                                                             |
-| Options                                        | snake_case kwargs                                                          | camelCase options object                                                                                       |
+|                                                | Python                                                                     | npm                                                                                                              |
+| ---------------------------------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| API                                            | sync + async clients                                                       | one async client                                                                                                 |
+| Malformed id (`"abc"`)                         | `StorageError` from Postgres                                               | treated as not found (`ImageNotFoundError`, no DB round trip), friendlier for `/images/:id` routes               |
+| `optimize`                                     | Pillow; EXIF stripped, not applied                                         | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                         |
+| `putMany` inserts                              | pipelined                                                                  | one transaction, statements run sequentially                                                                     |
+| Formats                                        | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                     | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                    |
+| Convert TO HEIC with `optimize`                | yes (pillow-heif)                                                          | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format` |
+| Event duration                                 | `duration_seconds`                                                         | `durationMs`                                                                                                     |
+| `migrateClassicToDedup`                        | loads the whole table into memory; fails if re-run or if any row is tiered | batched, safe to re-run, refuses tiered rows with a clear message                                                |
+| Dedup `putMany` / `deleteMany` locking         | arbitrary order (overlapping batches can deadlock)                         | sorted checksum order (cannot deadlock)                                                                          |
+| SQLite API                                     | synchronous `SQLiteBackend` + separate async backend                       | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                  |
+| SQLite `onOperation` / busy retry              | none                                                                       | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                             |
+| SQLite stream deleted mid-read                 | sync adapter keeps reading a WAL snapshot (async adapter raises)           | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                           |
+| CLI                                            | PostgreSQL, classic mode only; `verify` crashes on tiered rows             | PostgreSQL **and** SQLite, classic **and** dedup; `info`/`verify` understand tiered rows; `migrate --to-dedup`   |
+| `connection=` without an open transaction      | multi-statement ops rely on the driver's implicit transaction              | always atomic (own transaction or savepoint)                                                                     |
+| `delete(id, connection=...)` on a tiered image | deletes the S3 object immediately                                          | leaves the object until you delete it after commit (a rolled-back transaction can never lose data)               |
+| S3 delete failure after `delete()`             | raises after the row is gone                                               | never fails the delete; reported via `onOperation`                                                               |
+| Options                                        | snake_case kwargs                                                          | camelCase options object                                                                                         |
 
 ## What's not here yet
 
-Planned, in order: **MySQL/MariaDB** and **HEIC**.
+Planned: **MySQL/MariaDB**.
 Tiering and dedup cannot be combined (same rule as the Python package).
 
 ## Runtime notes

package/dist/{chunk-W2C37XYV.js → chunk-SCPC3QDR.js}

@@ -65,4 +65,4 @@ export {
   StorageError,
   AccessDeniedError
 };
-//# sourceMappingURL=chunk-W2C37XYV.js.map
+//# sourceMappingURL=chunk-SCPC3QDR.js.map

package/dist/http.js

@@ -1,7 +1,7 @@
 import {
   AccessDeniedError,
   ImageNotFoundError
-} from "./chunk-W2C37XYV.js";
+} from "./chunk-SCPC3QDR.js";
 
 // src/http.ts
 import { Readable } from "stream";

package/dist/index.cjs

@@ -54,10 +54,14 @@ __export(src_exports, {
 });
 module.exports = __toCommonJS(src_exports);
 
+// node_modules/tsup/assets/cjs_shims.js
+var getImportMetaUrl = () => typeof document === "undefined" ? new URL(`file:${__filename}`).href : document.currentScript && document.currentScript.tagName.toUpperCase() === "SCRIPT" ? document.currentScript.src : new URL("main.js", document.baseURI).href;
+var importMetaUrl = /* @__PURE__ */ getImportMetaUrl();
+
 // src/client.ts
 var import_node_crypto2 = require("crypto");
 var import_promises = require("fs/promises");
-var import_node_path = require("path");
+var import_node_path2 = require("path");
 
 // src/errors.ts
 var ZeroBucketError = class extends Error {
@@ -1980,11 +1984,191 @@ var SQLiteBackend = class {
   }
 };
 
+// src/heic.ts
+var import_node_module = require("module");
+var import_node_path = require("path");
+var import_node_worker_threads = require("worker_threads");
+var HEIC_BRANDS = /* @__PURE__ */ new Set([
+  "heic",
+  "heix",
+  "hevc",
+  "heim",
+  "heis",
+  "mif1",
+  "msf1"
+]);
+function looksLikeHeic(data) {
+  if (data.length < 12) return false;
+  if (String.fromCharCode(data[4], data[5], data[6], data[7]) !== "ftyp")
+    return false;
+  return HEIC_BRANDS.has(
+    String.fromCharCode(data[8], data[9], data[10], data[11])
+  );
+}
+var HEIC_INSTALL_HINT = "npm install libheif-js";
+var WORKER_SOURCE = `
+const { parentPort, workerData } = require("node:worker_threads");
+const libheif = require(workerData.libPath);
+// libheif-js reports decode problems by dumping a raw error object to the console. Capture that text instead:
+// it becomes part of the error we return (the useful diagnosis), and nothing is printed to the host's stderr.
+let diag = "";
+for (const level of ["log", "info", "warn", "error"]) {
+  console[level] = (...args) => {
+    diag = args.map((a) => (a && a.message) || (typeof a === "string" ? a : "")).filter(Boolean).join(" ").trim();
+  };
+}
+const describe = (e) => {
+  const base = String((e && e.message) || e);
+  return diag && !base.includes(diag) ? base + ": " + diag : base;
+};
+function open(bytes) {
+  const images = new libheif.HeifDecoder().decode(bytes);
+  if (!images || images.length === 0) throw new Error("no images found in HEIF container");
+  return images[0];
+}
+parentPort.on("message", async ({ id, op, data }) => {
+  diag = "";
+  try {
+    const img = open(new Uint8Array(data));
+    const width = img.get_width();
+    const height = img.get_height();
+    if (op === "info") return parentPort.postMessage({ id, ok: true, width, height });
+    const out = await new Promise((resolve, reject) =>
+      img.display({ data: new Uint8ClampedArray(width * height * 4), width, height }, (d) =>
+        d ? resolve(d) : reject(new Error("HEIF processing error")),
+      ),
+    );
+    if (op === "validate") return parentPort.postMessage({ id, ok: true, width, height });
+    const rgba = out.data.buffer;
+    parentPort.postMessage({ id, ok: true, width, height, rgba }, [rgba]);
+  } catch (e) {
+    parentPort.postMessage({ id, ok: false, error: describe(e) });
+  }
+});
+`;
+var HeicWorkerError = class extends StorageError {
+};
+var worker;
+var nextId = 1;
+var pending = /* @__PURE__ */ new Map();
+var stats = { info: 0, validate: 0, decode: 0, workersStarted: 0 };
+var resolveLib = () => {
+  const attempts = [
+    () => (0, import_node_module.createRequire)(importMetaUrl).resolve("libheif-js/wasm-bundle"),
+    // next to / above this package
+    () => (0, import_node_module.createRequire)((0, import_node_path.join)(process.cwd(), "noop.js")).resolve(
+      "libheif-js/wasm-bundle"
+    )
+    // the app's own node_modules
+  ];
+  let last;
+  for (const attempt of attempts) {
+    try {
+      return attempt();
+    } catch (e) {
+      last = e;
+    }
+  }
+  throw last;
+};
+function heicAvailable() {
+  try {
+    resolveLib();
+    return true;
+  } catch {
+    return false;
+  }
+}
+function releaseIfIdle(w) {
+  for (const p of pending.values()) if (p.owner === w) return;
+  w.unref();
+}
+function startWorker() {
+  let libPath;
+  try {
+    libPath = resolveLib();
+  } catch (cause) {
+    throw new StorageError(
+      `HEIC support requires the optional dependency 'libheif-js'. Install it with: ${HEIC_INSTALL_HINT}`,
+      { cause }
+    );
+  }
+  const w = new import_node_worker_threads.Worker(WORKER_SOURCE, { eval: true, workerData: { libPath } });
+  stats.workersStarted += 1;
+  w.on(
+    "message",
+    (m) => {
+      const p = pending.get(m.id);
+      if (!p) return;
+      pending.delete(m.id);
+      releaseIfIdle(w);
+      if (m.ok) p.resolve(m);
+      else p.reject(new Error(m.error));
+    }
+  );
+  const fail = (err) => {
+    if (worker === w) worker = void 0;
+    for (const [id, p] of pending) {
+      if (p.owner !== w) continue;
+      pending.delete(id);
+      p.reject(err);
+    }
+  };
+  w.on(
+    "error",
+    (e) => fail(
+      new HeicWorkerError(
+        `HEIC decoder worker crashed: ${e instanceof Error ? e.message : String(e)}`,
+        { cause: e }
+      )
+    )
+  );
+  w.on(
+    "exit",
+    (code) => fail(
+      new HeicWorkerError(
+        `HEIC decoder worker exited unexpectedly (code ${code})`
+      )
+    )
+  );
+  w.unref();
+  return w;
+}
+function call(op, data) {
+  stats[op] += 1;
+  const w = worker ??= startWorker();
+  const id = nextId++;
+  const copy = new Uint8Array(data).buffer;
+  return new Promise((resolve, reject) => {
+    pending.set(id, { owner: w, resolve, reject });
+    w.ref();
+    w.postMessage({ id, op, data: copy }, [copy]);
+  });
+}
+async function heicInfo(data) {
+  const r = await call("info", data);
+  return { width: r.width, height: r.height };
+}
+async function heicValidate(data) {
+  const r = await call("validate", data);
+  return { width: r.width, height: r.height };
+}
+async function decodeHeic(data) {
+  const r = await call("decode", data);
+  return {
+    width: r.width,
+    height: r.height,
+    rgba: Buffer.from(r.rgba)
+  };
+}
+
 // src/validation.ts
 var FORMAT_TO_MIME = {
   jpeg: "image/jpeg",
   png: "image/png",
-  webp: "image/webp"
+  webp: "image/webp",
+  heic: "image/heic"
+  // optional: needs `npm install libheif-js` (see heic.ts)
 };
 var SUPPORTED_FORMATS = Object.keys(FORMAT_TO_MIME);
 var DEFAULT_MAX_PIXELS = 89e6;
@@ -2012,6 +2196,8 @@ async function validateImage(data, opts) {
   const sizeBytes = data.length;
   if (sizeBytes > maxBytes) throw new ImageTooLargeError(sizeBytes, maxBytes);
   if (sizeBytes === 0) throw new CorruptedImageError("Image data is empty");
+  if (looksLikeHeic(data))
+    return validateHeic(data, { maxBytes, maxPixels, allowedFormats });
   const sharp = await loadSharp();
   const make = () => sharp(data, { limitInputPixels: maxPixels, failOn: "error" });
   let format;
@@ -2047,6 +2233,45 @@ async function validateImage(data, opts) {
   }
   return { mimeType: FORMAT_TO_MIME[format], width, height, sizeBytes };
 }
+async function validateHeic(data, {
+  maxBytes,
+  maxPixels,
+  allowedFormats
+}) {
+  const sizeBytes = data.length;
+  if (!heicAvailable()) {
+    throw new ImageValidationError(
+      `This looks like a HEIC/HEIF image, which requires an optional dependency. Install it with: ${HEIC_INSTALL_HINT}`
+    );
+  }
+  if (!allowedFormats.includes("heic"))
+    throw new UnsupportedFormatError("heic", allowedFormats);
+  let width;
+  let height;
+  try {
+    ({ width, height } = await heicInfo(data));
+  } catch (exc) {
+    if (exc instanceof StorageError) throw exc;
+    throw new CorruptedImageError(
+      `Could not decode image: ${errMessage(exc)}`,
+      { cause: exc }
+    );
+  }
+  if (!width || !height)
+    throw new CorruptedImageError("Image has no readable dimensions");
+  if (width * height > maxPixels)
+    throw new ImageTooLargeError(sizeBytes, maxBytes);
+  try {
+    await heicValidate(data);
+  } catch (exc) {
+    if (exc instanceof StorageError) throw exc;
+    throw new CorruptedImageError(
+      `Image data is truncated or corrupted: ${errMessage(exc)}`,
+      { cause: exc }
+    );
+  }
+  return { mimeType: FORMAT_TO_MIME.heic, width, height, sizeBytes };
+}
 function isPixelLimit(exc) {
   return /pixel limit/i.test(errMessage(exc));
 }
@@ -2055,28 +2280,53 @@ function errMessage(exc) {
 }
 
 // src/optimize.ts
+var ENCODABLE_FORMATS = ["jpeg", "png", "webp"];
 var DEFAULT_JPEG_QUALITY = 90;
 var DEFAULT_WEBP_QUALITY = 88;
-var FORMAT_ALIASES = { jpg: "jpeg" };
 async function optimizeImage(data, opts) {
   const { maxWidth, quality, maxBytes, maxPixels = DEFAULT_MAX_PIXELS } = opts;
   const sharp = await loadSharp();
   const originalSizeBytes = data.length;
-  const base = sharp(data, { limitInputPixels: maxPixels, failOn: "error" });
-  const meta = await base.metadata();
-  const requested = (opts.format ?? meta.format ?? "jpeg").toLowerCase();
-  const target = FORMAT_ALIASES[requested] ?? requested;
-  if (!SUPPORTED_FORMATS.includes(target)) {
+  const requestedRaw = opts.format?.toLowerCase();
+  const requested = requestedRaw === "jpg" ? "jpeg" : requestedRaw;
+  if (requested === "heic" || requested === "heif") {
     throw new ImageValidationError(
-      `Cannot re-encode to unsupported format ${JSON.stringify(target)}. Supported: ${JSON.stringify(
-        [...SUPPORTED_FORMATS].sort()
-      )}`
+      "Encoding to HEIC is not available in the Node package (there is no HEVC encoder for Node). Optimize to jpeg, webp or png instead, or store the original HEIC without `optimize`."
     );
   }
-  let pipeline = sharp(data, {
+  let input = data;
+  let inputOptions = {
     limitInputPixels: maxPixels,
     failOn: "error"
-  }).rotate();
+  };
+  let sourceFormat;
+  if (looksLikeHeic(data)) {
+    if (!heicAvailable()) {
+      throw new ImageValidationError(
+        `This looks like a HEIC/HEIF image, which requires an optional dependency. Install it with: ${HEIC_INSTALL_HINT}`
+      );
+    }
+    if (!requested) {
+      throw new ImageValidationError(
+        "Optimizing a HEIC image needs an explicit output format, because re-encoding to HEIC is not available in the Node package. Pass optimize: { format: 'jpeg' | 'webp' | 'png' } (for example { format: 'webp', maxWidth: 1600 })."
+      );
+    }
+    const decoded = await decodeHeic(data);
+    input = decoded.rgba;
+    inputOptions = {
+      raw: { width: decoded.width, height: decoded.height, channels: 4 }
+    };
+    sourceFormat = "heic";
+  } else {
+    sourceFormat = (await sharp(data, inputOptions).metadata()).format;
+  }
+  const target = requested ?? sourceFormat ?? "jpeg";
+  if (!ENCODABLE_FORMATS.includes(target)) {
+    throw new ImageValidationError(
+      `Cannot re-encode to unsupported format ${JSON.stringify(target)}. Supported: ${JSON.stringify([...ENCODABLE_FORMATS].sort())}`
+    );
+  }
+  let pipeline = sharp(input, inputOptions).rotate();
   if (maxWidth !== void 0) {
     pipeline = pipeline.resize({
       width: maxWidth,
@@ -2519,7 +2769,7 @@ async function readInput(image, filename, maxBytes) {
     if (size > maxBytes) throw new ImageTooLargeError(size, maxBytes);
     return {
       data: await (0, import_promises.readFile)(image),
-      filename: filename ?? (0, import_node_path.basename)(image)
+      filename: filename ?? (0, import_node_path2.basename)(image)
     };
   }
   if (Buffer.isBuffer(image)) return { data: image, filename };
@@ -2554,7 +2804,7 @@ async function readInput(image, filename, maxBytes) {
     const p = image.path;
     return {
       data: Buffer.concat(chunks, total),
-      filename: filename ?? (typeof p === "string" ? (0, import_node_path.basename)(p) : void 0)
+      filename: filename ?? (typeof p === "string" ? (0, import_node_path2.basename)(p) : void 0)
     };
   }
   throw new TypeError(