zerobucket 0.8.2 → 0.9.1

package/README.md

@@ -123,11 +123,13 @@ await images.get(id, { context: { userId } }); // throws AccessDeniedError if th
 **A hook that throws fails closed**: the original exception propagates; it is never treated as
 "allow". Denied calls never reach the database. `exists()` is deliberately not gated.
 
-**Your hook receives the id exactly as the caller supplied it**, before ZeroBucket validates it, so it can be a
-malformed string such as one taken straight from a URL. Check the format before using it in SQL (for example against a
-UUID pattern, returning `true` for a malformed id so ZeroBucket answers "not found"); otherwise a request for
-`/images/garbage` becomes a database error and a `500`. The [example app](https://github.com/KedarGhadyalji/ZeroBucket/tree/main/examples/node-integration)
-shows a working hook, plus transactional uploads and serving.
+**Your hook only ever sees a well-formed UUID.** A malformed id (such as one taken straight from a URL) can never
+exist, so it is "not found" (`ImageNotFoundError`, or `error: "not found"` for that item in `getMany`) and your hook is
+never called with it. You can use the id in SQL (`WHERE id = $1` on a `uuid` column) without validating it first. _(Before
+0.9.0 the hook received the raw string, so on 0.8.x guard against a malformed id yourself: a request for
+`/images/garbage` otherwise became a database error and a `500`.)_ The
+[example app](https://github.com/KedarGhadyalji/ZeroBucket/tree/main/examples/node-integration) shows a working hook, plus
+transactional uploads and serving.
 
 ## Streaming and ranges
 
@@ -451,33 +453,55 @@ same corpus of valid and corrupt files identically.
 
 ## Differences from the Python package
 
-Python column = zerobucket on PyPI **0.25.1 or later** (several of its behaviours below were fixed in 0.25.1; older versions
-differ).
-
-|                                              | Python                                                                | npm                                                                                                                                |
-| -------------------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
-| API                                          | sync + async clients                                                  | one async client                                                                                                                   |
-| Malformed id (`"abc"`)                       | `StorageError` from Postgres                                          | treated as not found (`ImageNotFoundError`, no DB round trip), friendlier for `/images/:id` routes                                 |
-| `optimize`                                   | Pillow; EXIF stripped, not applied                                    | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                                           |
-| `putMany` inserts                            | pipelined                                                             | one transaction, statements run sequentially                                                                                       |
-| Formats                                      | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                                      |
-| Convert TO HEIC with `optimize`              | yes (pillow-heif)                                                     | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format`                   |
-| Event duration                               | `duration_seconds`                                                    | `durationMs`                                                                                                                       |
-| `migrateClassicToDedup`, tiered classic rows | always **skips** them and reports `tiered_rows_skipped`               | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python                       |
-| `migrateClassicToDedup`, other behaviour     | batched, safe to re-run (server-side cursor)                          | batched, safe to re-run (keyset paging, advisory lock so concurrent runs cannot race)                                              |
-| SQLite API                                   | synchronous `SQLiteBackend` + separate async backend                  | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                                    |
-| SQLite `onOperation` / busy retry            | none                                                                  | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                                               |
-| SQLite stream deleted mid-read               | sync adapter keeps reading a WAL snapshot (async adapter raises)      | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                                             |
-| MySQL `onOperation` / retry                  | sync `MySQLBackend` yes; `AsyncMySQLBackend` none                     | yes (same client for everything)                                                                                                   |
-| CLI                                          | PostgreSQL, classic mode only; `verify` skips tiered rows with a note | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` also checks tiered images; `migrate --to-dedup` |
-| `connection=` without an open transaction    | multi-statement ops rely on the driver's implicit transaction         | always atomic (own transaction or savepoint)                                                                                       |
-| S3 delete failure after `delete()`           | raises after the row is gone                                          | never fails the delete; reported via `onOperation`                                                                                 |
-| Options                                      | snake_case kwargs                                                     | camelCase options object                                                                                                           |
-
-Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
+Python column = zerobucket on PyPI **0.26.0 or later** (several of its behaviours below were fixed in 0.25.1 and 0.26.0; older
+versions differ).
+
+|                                              | Python                                                                | npm                                                                                                                                 |
+| -------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| API                                          | sync + async clients                                                  | one async client                                                                                                                    |
+| `optimize`                                   | Pillow; EXIF stripped, not applied                                    | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                                            |
+| JPEG encoder in `optimize`                   | Pillow, progressive + optimized                                       | sharp with `mozjpeg`, progressive: on smooth photos roughly **half the size** at about the same SSIM (measured); WebP and PNG match |
+| `putMany` inserts                            | pipelined                                                             | one transaction, statements run sequentially                                                                                        |
+| Formats                                      | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                                       |
+| Convert TO HEIC with `optimize`              | yes (pillow-heif)                                                     | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format`                    |
+| Event duration                               | `duration_seconds`                                                    | `durationMs`                                                                                                                        |
+| `migrateClassicToDedup`, tiered classic rows | always **skips** them and reports `tiered_rows_skipped`               | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python                        |
+| `migrateClassicToDedup`, other behaviour     | batched, safe to re-run (server-side cursor)                          | batched, safe to re-run (keyset paging, advisory lock so concurrent runs cannot race)                                               |
+| SQLite API                                   | synchronous `SQLiteBackend` + separate async backend                  | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                                     |
+| SQLite `onOperation` / busy retry            | none                                                                  | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                                                |
+| SQLite stream deleted mid-read               | sync adapter keeps reading a WAL snapshot (async adapter raises)      | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                                              |
+| MySQL `onOperation` / retry                  | sync `MySQLBackend` yes; `AsyncMySQLBackend` none                     | yes (same client for everything)                                                                                                    |
+| CLI                                          | PostgreSQL, classic mode only; `verify` skips tiered rows with a note | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` also checks tiered images; `migrate --to-dedup`  |
+| `connection=` without an open transaction    | multi-statement ops rely on the driver's implicit transaction         | always atomic (own transaction or savepoint)                                                                                        |
+| S3 delete failure after `delete()`           | raises after the row is gone                                          | never fails the delete; reported via `onOperation`                                                                                  |
+| Options                                      | snake_case kwargs                                                     | camelCase options object                                                                                                            |
+
+Both packages treat a malformed id as "not found" on every backend (Python 0.26.0+, npm 0.9.0+) and never pass one to a
+`before_get`/`beforeGet` hook. Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
 never lose data), take dedup batch locks in sorted checksum order (overlapping batches cannot deadlock), and skip the
 MySQL `ALTER TABLE` at startup when the schema is current.
 
+## Performance
+
+Measured on one single-core container, so treat it as directional; the full tables (PostgreSQL, SQLite, MariaDB, per size,
+memory, event-loop impact, throughput) are in the
+[benchmark results](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/benchmarks/node/RESULTS.md).
+
+| Image size   | `put()` (PostgreSQL) | `get()`     | Notes                                                               |
+| ------------ | -------------------- | ----------- | ------------------------------------------------------------------- |
+| up to ~500KB | under 60ms           | under 5ms   | Feels like any other write.                                         |
+| 0.5-1.5MB    | about 60-160ms       | under 10ms  | Fine for uploads.                                                   |
+| about 5MB    | about 0.6s           | about 45ms  | About 58MB of transient memory per upload.                          |
+| about 10MB   | about 1.2-1.4s       | about 120ms | About 130MB of transient memory per upload. Default limit is 8 MiB. |
+
+- `put()` time is mostly image validation (a full decode), so it follows the image's pixel count, not the database you pick.
+- The decode runs off the main thread: the event loop stalled at most about 16ms during a 10MB `put()`.
+- Memory is the decoded image, roughly 11x the file size for a 45-megapixel JPEG. Cap `maxBytes` and `maxPixels` on public endpoints.
+- Stored size equals the file size: JPEG, PNG and WebP are already compressed, so the database cannot shrink them further.
+- Compression quality of `optimize` is measured in the
+  [compression results](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/benchmarks/node/COMPRESSION_RESULTS.md).
+  Do not convert flat graphics (screenshots, logos) to JPEG; they grow.
+
 ## Limitations (read before using in production)
 
 - **Images live in your database.** That is the point (one backup, one transaction, no extra service), but it means

package/dist/{client-l4jPIFJI.d.cts → client-7ZNu3xMJ.d.cts}

@@ -392,7 +392,9 @@ interface ZeroBucketOptions<C = unknown> {
      * Authorization hook for get / getMany (per id) / getStream / streamTo / metadata.
      * Return false to deny. A hook that THROWS fails closed: the exception
      * propagates (or is captured per item in getMany) and is never treated as allow.
-     * NOT called for exists().
+     * NOT called for exists(). Only ever called with a well-formed UUID: a malformed id is
+     * "not found" (ImageNotFoundError, or `error: "not found"` per item in getMany) and never
+     * reaches the hook.
      */
     beforeGet?: BeforeGetHook<C>;
     /** Authorization hook for put / putMany (evaluated once per call). Same fail-closed rule. */

package/dist/{client-l4jPIFJI.d.ts → client-7ZNu3xMJ.d.ts}

@@ -392,7 +392,9 @@ interface ZeroBucketOptions<C = unknown> {
      * Authorization hook for get / getMany (per id) / getStream / streamTo / metadata.
      * Return false to deny. A hook that THROWS fails closed: the exception
      * propagates (or is captured per item in getMany) and is never treated as allow.
-     * NOT called for exists().
+     * NOT called for exists(). Only ever called with a well-formed UUID: a malformed id is
+     * "not found" (ImageNotFoundError, or `error: "not found"` per item in getMany) and never
+     * reaches the hook.
      */
     beforeGet?: BeforeGetHook<C>;
     /** Authorization hook for put / putMany (evaluated once per call). Same fail-closed rule. */

package/dist/http.d.cts

@@ -1,5 +1,5 @@
 import { IncomingMessage, ServerResponse } from 'node:http';
-import { Z as ZeroBucket } from './client-l4jPIFJI.cjs';
+import { Z as ZeroBucket } from './client-7ZNu3xMJ.cjs';
 import 'pg';
 
 /**

package/dist/http.d.ts

@@ -1,5 +1,5 @@
 import { IncomingMessage, ServerResponse } from 'node:http';
-import { Z as ZeroBucket } from './client-l4jPIFJI.js';
+import { Z as ZeroBucket } from './client-7ZNu3xMJ.js';
 import 'pg';
 
 /**

package/dist/index.cjs

@@ -3407,6 +3407,7 @@ var ZeroBucket = class _ZeroBucket {
   }
   // ---- hooks -----------------------------------------------------------
   async checkBeforeGet(imageId, context) {
+    if (!isValidId(imageId)) throw new ImageNotFoundError(imageId);
     if (!this.beforeGet) return;
     if (!await this.beforeGet(imageId, context))
       throw new AccessDeniedError("get", imageId);
@@ -3570,6 +3571,10 @@ var ZeroBucket = class _ZeroBucket {
     const allowedIdx = [];
     for (let i = 0; i < imageIds.length; i++) {
       const imageId = imageIds[i];
+      if (!isValidId(imageId)) {
+        results[i] = { imageId, image: null, error: "not found", success: false };
+        continue;
+      }
       if (this.beforeGet) {
         try {
           if (!await this.beforeGet(imageId, options.context)) {