zerobucket 0.7.0 → 0.8.1

package/README.md

@@ -10,8 +10,7 @@ This is the Node.js/TypeScript sibling of the [Python package](https://pypi.org/
 **It uses the exact same table**, so a Node service and a Python service can share one database
 and read each other's images.
 
-> **Status: early (0.7.x).** PostgreSQL **and SQLite**, each in classic and dedup modes, plus **MySQL/MariaDB** (classic
-> mode; dedup and the CLI are not available for MySQL yet, see [What's not here yet](#whats-not-here-yet)), with
+> **Status: early (0.8.x).** PostgreSQL, SQLite **and MySQL/MariaDB**, each in classic and dedup modes, with
 > S3-compatible object-storage tiering, HEIC (iPhone photo) support and a `zerobucket` command line tool.
 
 ## Install
@@ -220,10 +219,13 @@ const id2 = await images.put(iphonePhoto, {
 
 The package installs a `zerobucket` command (`npx zerobucket ...`, or install globally with `npm i -g zerobucket`).
 It has the same commands, flags, messages and exit codes as the Python package's CLI, and additionally works with
-**SQLite** and **dedup mode**, and understands images that were tiered to object storage. It does not know MySQL/MariaDB yet.
+**SQLite**, **MySQL/MariaDB** (a `mysql://` or `mariadb://` `--database-url`, or `ZEROBUCKET_DATABASE_URL`) and **dedup
+mode**, and understands images that were tiered to object storage. On MySQL the on-disk size is read from
+`information_schema` and is approximate (MySQL 8 caches those statistics for up to 24 hours).
 
 ```bash
 export ZEROBUCKET_DATABASE_URL=postgresql://user:pass@localhost/mydb     # or: --database-url URL
+# MySQL/MariaDB:   export ZEROBUCKET_DATABASE_URL=mysql://user:pass@localhost:3306/mydb   # mariadb:// works too
 # SQLite instead:  export ZEROBUCKET_SQLITE_PATH=./images.db              # or: --sqlite PATH
 
 zerobucket init                 # create the schema (add --dedup for the dedup tables)
@@ -234,6 +236,7 @@ zerobucket tier IMAGE_ID --bucket my-images                         # move one i
 zerobucket tier --older-than 90 --min-size 500000 --bucket my-images --dry-run   # preview a bulk move
 zerobucket tier --all --limit 1000 --bucket my-images --endpoint-url https://<acct>.r2.cloudflarestorage.com
 zerobucket migrate --to-dedup   # PostgreSQL: copy classic images into the dedup tables (non-destructive, re-runnable)
+zerobucket migrate --to-dedup --skip-tiered   # ...skipping images tiered to object storage instead of refusing
 ```
 
 - **Exit codes:** `0` success, `1` problems found or an operation failed, `2` usage or connection error. That makes
@@ -308,12 +311,16 @@ const images = new ZeroBucket({
 
 Same API as everywhere else: `put`, `get`, `getMany`, `putMany`, `delete`, `deleteMany`, `exists`, `metadata`, streaming
 and ranges, access-control hooks, the optimize pipeline, custom validators, HEIC, `onOperation` metrics, retry with
-backoff, pooling (`poolMaxSize`, `poolTimeoutMs`), **object-storage tiering** and `connection` (your own transaction).
-It uses the Python package's table, byte for byte (`SHOW CREATE TABLE` is identical), so a Node app and a Python app can
-share one MySQL/MariaDB database (the conformance harness verifies both directions, including tiered rows).
+backoff, pooling (`poolMaxSize`, `poolTimeoutMs`), **dedup mode** (`dedup: true`), **object-storage tiering** and
+`connection` (your own transaction). The [`zerobucket` CLI](#command-line) works on it too: pass a `mysql://` or
+`mariadb://` `--database-url`.
 
-**Not available on MySQL yet:** dedup mode (`dedup: true` throws, rather than being ignored) and the `zerobucket` command
-line tool (it knows PostgreSQL and SQLite). The Python package has both for MySQL.
+It uses the Python package's tables, byte for byte (`SHOW CREATE TABLE` is identical for the classic table and for both
+dedup tables), so a Node app and a Python app can share one MySQL/MariaDB database. The conformance harness verifies both
+directions in both modes, including tiered rows and shared dedup blobs.
+
+As everywhere, dedup and tiering cannot be combined (`dedup: true` with `objectStorage` throws). Copying an existing
+classic table into the dedup tables (`migrate --to-dedup`) is PostgreSQL-only, as in the Python package.
 
 What is different from Postgres, and why (each verified against a real MariaDB 10.11):
 
@@ -331,6 +338,18 @@ What is different from Postgres, and why (each verified against a real MariaDB 1
 - **Migration takes a server-side lock** (`GET_LOCK`), so many processes starting at once do not race on the DDL, and it
   issues **no DDL at all** when the schema is current. (Even a no-op `ALTER TABLE` needs an exclusive metadata lock that
   waits behind every open transaction, and everything else on that table then queues behind _it_.)
+- **Dedup** stores each distinct image once (`zerobucket_blobs`, with a `ref_count`) and one row per image
+  (`zerobucket_image_refs`). Without `RETURNING`, a delete decrements with `UPDATE ... ref_count = ref_count - n` (which
+  takes the row lock) and then removes the blob once the count reaches zero; the foreign key makes deleting a blob that is
+  still referenced impossible. A batch groups identical images, touches each distinct blob **once, in sorted checksum
+  order**, and decrements in the same order, so overlapping batches cannot deadlock.
+- **Batch deletes use one `SELECT ... FOR UPDATE` and then point deletes, never `DELETE ... WHERE id IN (...)`.** On
+  MariaDB 10.11, with a small table, the `IN` delete is planned as a full scan (`EXPLAIN` says `type=ALL`), and a scan
+  under REPEATABLE READ locks every row it passes, including rows another batch already holds. Two overlapping
+  `deleteMany` batches deadlocked in 126 of 300 transactions with an `IN` delete and in 0 with point deletes (also 0 with
+  a concurrent writer on the same blobs); `READ COMMITTED` and `FORCE INDEX` alone did not fix it. MySQL 8.0.46 plans the
+  same statement as a range over just the listed ids and did **not** deadlock in the same test. Point deletes do not
+  depend on the plan, so they are used on both.
 - **Large images and `max_allowed_packet`.** An image bigger than the server's limit fails with an error that says so and
   how to raise it (`SET GLOBAL max_allowed_packet = ...`). Images are sent as raw bytes, so the limit applies to the image
   size, not twice that.
@@ -343,8 +362,9 @@ What is different from Postgres, and why (each verified against a real MariaDB 1
   skipped on your connection (it cannot know whether you will commit).
 - **Streaming** uses ranged `SUBSTRING()` reads, so a delete mid-stream raises `StorageError` (like Postgres). Tiered images
   stream with real S3 `Range` requests.
-- **Tested on MariaDB 10.11.** Real MySQL 8.x is expected to behave the same (the SQL avoids MariaDB-only features) but has
-  not been run here. Use MariaDB 10.2.1+ or MySQL 8.0.16+: older servers accept the `CHECK` constraint syntax but ignore it.
+- **Tested on MariaDB 10.11 and MySQL 8.0.46** (the MySQL, dedup and CLI test suites, the Node/Python conformance harness
+  and the schema comparison all pass on both). Other versions are expected to behave the same but have not been run. Use
+  MariaDB 10.2.1+ or MySQL 8.0.16+: older servers accept the `CHECK` constraint syntax but ignore it.
 
 ## Deduplication
 
@@ -371,7 +391,11 @@ await images.delete(b); // last reference gone: blob removed
 
 **Moving existing data in:** `await images.migrateClassicToDedup()` copies every classic image into the dedup tables,
 keeping each id, and never modifies the classic table. It runs in small batches (bounded memory), is safe to re-run
-(already-migrated ids are skipped and counted), and refuses up front if any classic image is tiered to object storage.
+(already-migrated ids are skipped and counted), and refuses up front if any classic image is tiered to object storage
+(their bytes are not in the database to copy, and dedup mode has no tiering), changing nothing. If you would rather
+migrate everything else, pass `{ skipTiered: true }` (CLI: `--skip-tiered`): tiered images are left untouched in the
+classic table, counted in `tieredSkipped`, and **will not exist in the dedup tables**, so only use it if you can live
+without them there.
 
 ## Object-storage tiering
 
@@ -421,30 +445,52 @@ same corpus of valid and corrupt files identically.
 
 ## Differences from the Python package
 
-|                                                | Python                                                                     | npm                                                                                                              |
-| ---------------------------------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
-| API                                            | sync + async clients                                                       | one async client                                                                                                 |
-| Malformed id (`"abc"`)                         | `StorageError` from Postgres                                               | treated as not found (`ImageNotFoundError`, no DB round trip), friendlier for `/images/:id` routes               |
-| `optimize`                                     | Pillow; EXIF stripped, not applied                                         | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                         |
-| `putMany` inserts                              | pipelined                                                                  | one transaction, statements run sequentially                                                                     |
-| Formats                                        | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                     | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                    |
-| Convert TO HEIC with `optimize`                | yes (pillow-heif)                                                          | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format` |
-| Event duration                                 | `duration_seconds`                                                         | `durationMs`                                                                                                     |
-| `migrateClassicToDedup`                        | loads the whole table into memory; fails if re-run or if any row is tiered | batched, safe to re-run, refuses tiered rows with a clear message                                                |
-| Dedup `putMany` / `deleteMany` locking         | arbitrary order (overlapping batches can deadlock)                         | sorted checksum order (cannot deadlock)                                                                          |
-| SQLite API                                     | synchronous `SQLiteBackend` + separate async backend                       | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                  |
-| SQLite `onOperation` / busy retry              | none                                                                       | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                             |
-| SQLite stream deleted mid-read                 | sync adapter keeps reading a WAL snapshot (async adapter raises)           | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                           |
-| CLI                                            | PostgreSQL, classic mode only; `verify` crashes on tiered rows             | PostgreSQL **and** SQLite, classic **and** dedup; `info`/`verify` understand tiered rows; `migrate --to-dedup`   |
-| `connection=` without an open transaction      | multi-statement ops rely on the driver's implicit transaction              | always atomic (own transaction or savepoint)                                                                     |
-| `delete(id, connection=...)` on a tiered image | deletes the S3 object immediately                                          | leaves the object until you delete it after commit (a rolled-back transaction can never lose data)               |
-| S3 delete failure after `delete()`             | raises after the row is gone                                               | never fails the delete; reported via `onOperation`                                                               |
-| Options                                        | snake_case kwargs                                                          | camelCase options object                                                                                         |
-
-## What's not here yet
-
-Planned for MySQL/MariaDB: **dedup mode**, the **`zerobucket` CLI**, and a MySQL-specific operations guide.
-Tiering and dedup cannot be combined (same rule as the Python package).
+Python column = zerobucket on PyPI **0.25.1 or later** (several of its behaviours below were fixed in 0.25.1; older versions
+differ).
+
+|                                              | Python                                                                | npm                                                                                                                                |
+| -------------------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
+| API                                          | sync + async clients                                                  | one async client                                                                                                                   |
+| Malformed id (`"abc"`)                       | `StorageError` from Postgres                                          | treated as not found (`ImageNotFoundError`, no DB round trip), friendlier for `/images/:id` routes                                 |
+| `optimize`                                   | Pillow; EXIF stripped, not applied                                    | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                                           |
+| `putMany` inserts                            | pipelined                                                             | one transaction, statements run sequentially                                                                                       |
+| Formats                                      | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                                      |
+| Convert TO HEIC with `optimize`              | yes (pillow-heif)                                                     | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format`                   |
+| Event duration                               | `duration_seconds`                                                    | `durationMs`                                                                                                                       |
+| `migrateClassicToDedup`, tiered classic rows | always **skips** them and reports `tiered_rows_skipped`               | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python                       |
+| `migrateClassicToDedup`, other behaviour     | batched, safe to re-run (server-side cursor)                          | batched, safe to re-run (keyset paging, advisory lock so concurrent runs cannot race)                                              |
+| SQLite API                                   | synchronous `SQLiteBackend` + separate async backend                  | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                                    |
+| SQLite `onOperation` / busy retry            | none                                                                  | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                                               |
+| SQLite stream deleted mid-read               | sync adapter keeps reading a WAL snapshot (async adapter raises)      | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                                             |
+| MySQL `onOperation` / retry                  | sync `MySQLBackend` yes; `AsyncMySQLBackend` none                     | yes (same client for everything)                                                                                                   |
+| CLI                                          | PostgreSQL, classic mode only; `verify` skips tiered rows with a note | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` also checks tiered images; `migrate --to-dedup` |
+| `connection=` without an open transaction    | multi-statement ops rely on the driver's implicit transaction         | always atomic (own transaction or savepoint)                                                                                       |
+| S3 delete failure after `delete()`           | raises after the row is gone                                          | never fails the delete; reported via `onOperation`                                                                                 |
+| Options                                      | snake_case kwargs                                                     | camelCase options object                                                                                                           |
+
+Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
+never lose data), take dedup batch locks in sorted checksum order (overlapping batches cannot deadlock), and skip the
+MySQL `ALTER TABLE` at startup when the schema is current.
+
+## Limitations (read before using in production)
+
+- **Images live in your database.** That is the point (one backup, one transaction, no extra service), but it means
+  this is not built for very large files or high-volume media: every image goes through the database and counts toward
+  its size, its backups and its replication. The default size limit is 8 MiB (`maxBytes`). For large video or
+  millions of big files, use object storage directly.
+- **Backups and space need attention.** A whole-database dump now includes every image, and deleting images does not
+  give disk space back on its own (PostgreSQL needs autovacuum, MySQL `OPTIMIZE TABLE`, SQLite `VACUUM`). SQLite
+  files must be backed up with `.backup`, not copied. See the
+  [Operations Guide](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/docs/OPERATIONS.md).
+- **Dedup and tiering cannot be combined** (a shared blob has many owners): `dedup: true` with `objectStorage` throws.
+- **`migrate --to-dedup` is PostgreSQL-only**, as in the Python package. Tiered classic images cannot be migrated (see
+  Deduplication for `skipTiered`).
+- **HEIC can be read and converted from, but not written.** There is no HEVC encoder for Node, so `optimize` cannot
+  produce HEIC.
+- **SQLite allows one writer at a time**; lock waits are async and never block the event loop, but write-heavy
+  workloads belong on PostgreSQL or MySQL/MariaDB.
+- **Tested versions:** CI runs PostgreSQL 16 and MariaDB 10.11; MySQL 8.0.46 was also run by hand. Other versions are
+  expected to behave the same but have not been run.
 
 ## Runtime notes