zerobucket 0.10.1 → 0.11.0

package/README.md

@@ -243,7 +243,7 @@ zerobucket verify --sample 200  # ...or just a random sample (good for a nightly
 zerobucket tier IMAGE_ID --bucket my-images                         # move one image to S3
 zerobucket tier --older-than 90 --min-size 500000 --bucket my-images --dry-run   # preview a bulk move
 zerobucket tier --all --limit 1000 --bucket my-images --endpoint-url https://<acct>.r2.cloudflarestorage.com
-zerobucket migrate --to-dedup   # PostgreSQL: copy classic images into the dedup tables (non-destructive, re-runnable)
+zerobucket migrate --to-dedup   # copy classic images into the dedup tables (PostgreSQL, SQLite and MySQL; non-destructive, re-runnable)
 zerobucket migrate --to-dedup --skip-tiered   # ...skipping images tiered to object storage instead of refusing
 ```
 
@@ -306,7 +306,8 @@ How it behaves in Node, and why:
 - **Node 22+ for SQLite.** `better-sqlite3` 13 requires Node 22 or newer and crashes on older versions. It ships its Windows,
   macOS and Linux binaries inside the npm package, so no compiler or Visual Studio is needed. Postgres users can stay on Node 20.
   (`better-sqlite3` 12 declares Node 20 support, but it is untested here.)
-- Not available on SQLite: `migrateClassicToDedup()` (Python has it for Postgres only, too).
+- `migrateClassicToDedup()` works on SQLite (since 0.11.0): one `BEGIN IMMEDIATE` transaction, so it is all-or-nothing, and it
+  holds SQLite's single write lock for the whole copy, so other writers to the file wait until it finishes (readers do not).
 
 ## MySQL / MariaDB
 
@@ -328,7 +329,9 @@ dedup tables), so a Node app and a Python app can share one MySQL/MariaDB databa
 directions in both modes, including tiered rows and shared dedup blobs.
 
 As everywhere, dedup and tiering cannot be combined (`dedup: true` with `objectStorage` throws). Copying an existing
-classic table into the dedup tables (`migrate --to-dedup`) is PostgreSQL-only, as in the Python package.
+classic table into the dedup tables (`migrate --to-dedup`) works on MySQL/MariaDB since 0.11.0: one transaction (all-or-nothing)
+and a named lock (`GET_LOCK`) so two migrations started at once cannot race. Timestamps are copied as text, never as JS dates,
+because `DATETIME(6)` keeps microseconds and a JS date does not.
 
 What is different from Postgres, and why (each verified against a real MariaDB 10.11):
 
@@ -400,7 +403,8 @@ await images.delete(b); // last reference gone: blob removed
 **Moving existing data in:** `await images.migrateClassicToDedup()` copies every classic image into the dedup tables,
 keeping each id, and never modifies the classic table. It runs in small batches (bounded memory), is safe to re-run
 (already-migrated ids are skipped and counted), and refuses up front if any classic image is tiered to object storage
-(their bytes are not in the database to copy, and dedup mode has no tiering), changing nothing. If you would rather
+(their bytes are not in the database to copy, and dedup mode has no tiering), changing nothing. Works on PostgreSQL, SQLite
+and MySQL/MariaDB (SQLite and MySQL since 0.11.0). If you would rather
 migrate everything else, pass `{ skipTiered: true }` (CLI: `--skip-tiered`): tiered images are left untouched in the
 classic table, counted in `tieredSkipped`, and **will not exist in the dedup tables**, so only use it if you can live
 without them there.
@@ -453,29 +457,31 @@ same corpus of valid and corrupt files identically.
 
 ## Differences from the Python package
 
-Python column = zerobucket on PyPI **0.26.0 or later** (several of its behaviours below were fixed in 0.25.1 and 0.26.0; older
-versions differ).
-
-|                                                                | Python                                                                | npm                                                                                                                                 |
-| -------------------------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
-| API                                                            | sync + async clients                                                  | one async client                                                                                                                    |
-| `optimize`                                                     | Pillow; EXIF stripped, not applied                                    | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                                            |
-| JPEG encoder in `optimize`                                     | Pillow, progressive + optimized                                       | sharp with `mozjpeg`, progressive: on smooth photos roughly **half the size** at about the same SSIM (measured); WebP and PNG match |
-| JPEG missing only its last 1-2 bytes (the end-of-image marker) | accepted (Pillow reads every pixel)                                   | **rejected** (libvips treats a premature end as an error); truncated interlaced PNGs are rejected by both since npm 0.10.0          |
-| `putMany` inserts                                              | pipelined                                                             | one transaction, statements run sequentially                                                                                        |
-| Formats                                                        | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                                       |
-| Convert TO HEIC with `optimize`                                | yes (pillow-heif)                                                     | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format`                    |
-| Event duration                                                 | `duration_seconds`                                                    | `durationMs`                                                                                                                        |
-| `migrateClassicToDedup`, tiered classic rows                   | always **skips** them and reports `tiered_rows_skipped`               | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python                        |
-| `migrateClassicToDedup`, other behaviour                       | batched, safe to re-run (server-side cursor)                          | batched, safe to re-run (keyset paging, advisory lock so concurrent runs cannot race)                                               |
-| SQLite API                                                     | synchronous `SQLiteBackend` + separate async backend                  | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                                     |
-| SQLite `onOperation` / busy retry                              | none                                                                  | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                                                |
-| SQLite stream deleted mid-read                                 | sync adapter keeps reading a WAL snapshot (async adapter raises)      | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                                              |
-| MySQL `onOperation` / retry                                    | sync `MySQLBackend` yes; `AsyncMySQLBackend` none                     | yes (same client for everything)                                                                                                    |
-| CLI                                                            | PostgreSQL, classic mode only; `verify` skips tiered rows with a note | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` also checks tiered images; `migrate --to-dedup`  |
-| `connection=` without an open transaction                      | multi-statement ops rely on the driver's implicit transaction         | always atomic (own transaction or savepoint)                                                                                        |
-| S3 delete failure after `delete()`                             | raises after the row is gone                                          | never fails the delete; reported via `onOperation`                                                                                  |
-| Options                                                        | snake_case kwargs                                                     | camelCase options object                                                                                                            |
+Python column = zerobucket on PyPI **0.30.0 or later** (several of its behaviours below were fixed or added in 0.25.1, 0.26.0 and
+0.27.0 to 0.30.0; older versions differ).
+
+|                                                                | Python                                                                                                                                                          | npm                                                                                                                                             |
+| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| API                                                            | sync + async clients                                                                                                                                            | one async client                                                                                                                                |
+| Async clients: dedup and tiering                               | every async adapter (since 0.30.0)                                                                                                                              | one async client, always                                                                                                                        |
+| `optimize`                                                     | Pillow; EXIF stripped, not applied                                                                                                                              | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG                                                                        |
+| JPEG encoder in `optimize`                                     | Pillow, progressive + optimized                                                                                                                                 | sharp with `mozjpeg`, progressive: on smooth photos roughly **half the size** at about the same SSIM (measured); WebP and PNG match             |
+| JPEG missing only its last 1-2 bytes (the end-of-image marker) | accepted (Pillow reads every pixel)                                                                                                                             | **rejected** (libvips treats a premature end as an error); truncated interlaced PNGs are rejected by both since npm 0.10.0                      |
+| `putMany` inserts                                              | pipelined                                                                                                                                                       | one transaction, statements run sequentially                                                                                                    |
+| Formats                                                        | JPEG, PNG, WebP, HEIC (`[heic]` extra)                                                                                                                          | JPEG, PNG, WebP, HEIC (optional `libheif-js`)                                                                                                   |
+| Convert TO HEIC with `optimize`                                | yes (pillow-heif)                                                                                                                                               | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format`                                |
+| Event duration                                                 | `duration_seconds`                                                                                                                                              | `durationMs`                                                                                                                                    |
+| `migrateClassicToDedup`, tiered classic rows                   | always **skips** them and reports `tiered_rows_skipped`                                                                                                         | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python                                    |
+| `migrateClassicToDedup`, other behaviour                       | batched, safe to re-run (server-side cursor)                                                                                                                    | batched, safe to re-run (keyset paging); concurrent runs cannot race (PostgreSQL advisory lock, MySQL named lock, SQLite write lock)            |
+| `migrateClassicToDedup`, databases                             | PostgreSQL only                                                                                                                                                 | PostgreSQL, SQLite **and** MySQL/MariaDB (SQLite and MySQL since 0.11.0)                                                                        |
+| SQLite API                                                     | synchronous `SQLiteBackend` + separate async backend                                                                                                            | one async client; the sync driver runs on the event loop, lock waits are async (never block it)                                                 |
+| SQLite `onOperation` / busy retry                              | `on_operation` events and busy retry, sync and async (since 0.27.0)                                                                                             | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`)                                                                            |
+| SQLite stream deleted mid-read                                 | sync adapter keeps reading a WAL snapshot (async adapter raises)                                                                                                | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot                                                          |
+| MySQL `onOperation` / retry                                    | yes, sync and async (since 0.27.0)                                                                                                                              | yes (same client for everything)                                                                                                                |
+| CLI                                                            | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` checks tiered images (since 0.28.0); `migrate --to-dedup` on PostgreSQL only | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` also checks tiered images; `migrate --to-dedup` on all three |
+| `connection=` without an open transaction                      | multi-statement ops rely on the driver's implicit transaction                                                                                                   | always atomic (own transaction or savepoint)                                                                                                    |
+| S3 delete failure after `delete()`                             | never fails the delete; reported as an `object_storage_delete` event (since 0.29.0)                                                                             | never fails the delete; reported via `onOperation`                                                                                              |
+| Options                                                        | snake_case kwargs                                                                                                                                               | camelCase options object                                                                                                                        |
 
 Both packages treat a malformed id as "not found" on every backend (Python 0.26.0+, npm 0.9.0+) and never pass one to a
 `before_get`/`beforeGet` hook. Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
@@ -521,8 +527,8 @@ memory, event-loop impact, throughput, and the before and after of each release)
   files must be backed up with `.backup`, not copied. See the
   [Operations Guide](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/docs/OPERATIONS.md).
 - **Dedup and tiering cannot be combined** (a shared blob has many owners): `dedup: true` with `objectStorage` throws.
-- **`migrate --to-dedup` is PostgreSQL-only**, as in the Python package. Tiered classic images cannot be migrated (see
-  Deduplication for `skipTiered`).
+- **Tiered classic images cannot be migrated** by `migrate --to-dedup` (see Deduplication for `skipTiered`). The migration itself
+  works on PostgreSQL, SQLite and MySQL/MariaDB since 0.11.0; the Python package still has it for PostgreSQL only.
 - **HEIC can be read and converted from, but not written.** There is no HEVC encoder for Node, so `optimize` cannot
   produce HEIC.
 - **SQLite allows one writer at a time**; lock waits are async and never block the event loop, but write-heavy

package/dist/bin.js

@@ -1695,6 +1695,125 @@ var SQLiteBackend = class {
       throw wrap("Failed to check image existence", exc);
     }
   }
+  // ---- migration: classic -> dedup --------------------------------------
+  /**
+   * One-time, NON-DESTRUCTIVE copy of every row in the classic `zerobucket_images` table into this
+   * dedup instance's tables, preserving every id (external references keep working) and
+   * deduplicating identical content on the way. The classic table is never modified. Same contract
+   * as `PostgresBackend.migrateClassicToDedup()`.
+   *
+   * Rows are copied in small batches (bounded memory instead of loading the whole table);
+   * re-running is safe (already-migrated ids are skipped and reported, not an error). Classic rows
+   * already tiered to object storage are refused up front with a clear message (their bytes are
+   * not in the database to copy) unless `skipTiered: true`, which skips and counts them instead.
+   * All-or-nothing: ONE `BEGIN IMMEDIATE` transaction, so a failure part-way leaves the dedup
+   * tables exactly as they were. That transaction holds SQLite's single write lock for the whole
+   * copy, so other writers to this database file wait until it finishes (readers are not blocked).
+   */
+  async migrateClassicToDedup(opts = {}) {
+    if (!this.dedup) {
+      throw new StorageError(
+        "migrateClassicToDedup() must be called on an instance created with dedup: true."
+      );
+    }
+    try {
+      return await this.execute(
+        "migrate_to_dedup",
+        void 0,
+        "write",
+        (db) => {
+          if (!db.prepare(
+            "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'zerobucket_images'"
+          ).get()) {
+            throw new StorageError(
+              "No classic zerobucket_images table found in this database -- nothing to migrate."
+            );
+          }
+          let tieredSkipped = 0;
+          const hasTierCol = db.prepare("PRAGMA table_info(zerobucket_images)").all().some((c) => c.name === "storage_backend");
+          if (hasTierCol) {
+            const tiered = db.prepare(
+              "SELECT count(*) AS n FROM zerobucket_images WHERE storage_backend <> 'sqlite'"
+            ).get().n;
+            if (tiered > 0 && !opts.skipTiered) {
+              throw new StorageError(
+                `${tiered} classic image(s) are tiered to object storage, so their bytes are not in the database to migrate. Dedup mode does not support tiering. Migrate only an un-tiered table, bring those images back first, or pass skipTiered: true (CLI: --skip-tiered) to skip them and migrate the rest.`
+              );
+            }
+            tieredSkipped = tiered;
+          }
+          const onlyInDb = hasTierCol && tieredSkipped > 0;
+          const seen = /* @__PURE__ */ new Set();
+          let migrated = 0;
+          let already = 0;
+          let cursor = null;
+          const BATCH = 20;
+          const columns = "id, data, mime_type, original_filename, size_bytes, width, height, checksum_sha256, created_at, updated_at";
+          for (; ; ) {
+            const page = cursor ? db.prepare(
+              `SELECT ${columns} FROM zerobucket_images WHERE (created_at, id) > (?, ?)${onlyInDb ? " AND storage_backend = 'sqlite'" : ""} ORDER BY created_at, id LIMIT ${BATCH}`
+            ).all(cursor.createdAt, cursor.id) : db.prepare(
+              `SELECT ${columns} FROM zerobucket_images${onlyInDb ? " WHERE storage_backend = 'sqlite'" : ""} ORDER BY created_at, id LIMIT ${BATCH}`
+            ).all();
+            if (page.length === 0) break;
+            const last = page[page.length - 1];
+            if (cursor && cursor.createdAt === last.created_at && cursor.id === last.id) {
+              throw new StorageError(
+                "Migration made no progress (keyset cursor did not advance); aborting instead of looping."
+              );
+            }
+            cursor = { createdAt: last.created_at, id: last.id };
+            const have = new Set(
+              db.prepare(
+                `SELECT id FROM zerobucket_image_refs WHERE id IN (${placeholders(page.length)})`
+              ).all(...page.map((r) => r.id)).map((r) => r.id)
+            );
+            const fresh = page.filter((r) => !have.has(r.id));
+            already += page.length - fresh.length;
+            const groups = /* @__PURE__ */ new Map();
+            for (const r of fresh) {
+              const g = groups.get(r.checksum_sha256);
+              if (g) g.n += 1;
+              else groups.set(r.checksum_sha256, { row: r, n: 1 });
+            }
+            for (const checksum of [...groups.keys()].sort()) {
+              const { row, n } = groups.get(checksum);
+              db.prepare(D_UPSERT).run(
+                checksum,
+                row.data,
+                row.mime_type,
+                row.size_bytes,
+                row.width,
+                row.height,
+                n,
+                row.created_at
+              );
+            }
+            for (const r of fresh) {
+              db.prepare(D_INSERT_REF2).run(
+                r.id,
+                r.checksum_sha256,
+                r.original_filename,
+                r.created_at,
+                r.updated_at
+              );
+              seen.add(r.checksum_sha256);
+              migrated += 1;
+            }
+          }
+          return {
+            imagesMigrated: migrated,
+            distinctBlobsCreated: seen.size,
+            duplicateReferencesFound: migrated - seen.size,
+            alreadyMigrated: already,
+            tieredSkipped
+          };
+        }
+      );
+    } catch (exc) {
+      throw wrap("Migration to dedup failed", exc);
+    }
+  }
   // ---- maintenance (CLI) -----------------------------------------------
   async getInfo() {
     try {
@@ -1905,6 +2024,7 @@ CREATE TABLE IF NOT EXISTS zerobucket_images (
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`;
 var TIERING_CHECK = "chk_zerobucket_storage";
 var MIGRATE_LOCK = "zerobucket_migrate";
+var MIGRATE_DEDUP_LOCK = "zerobucket_migrate_dedup";
 var COLS2 = `id, data, mime_type, original_filename, size_bytes, width, height, checksum_sha256, storage_backend, object_storage_bucket, object_storage_key`;
 var INSERT3 = `INSERT INTO zerobucket_images (id, data, mime_type, original_filename, size_bytes, width, height, checksum_sha256, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`;
 var SELECT_FULL3 = `SELECT ${COLS2} FROM zerobucket_images WHERE id = ?`;
@@ -2264,15 +2384,31 @@ var MySQLBackend = class {
   /**
    * Run `work` on a pooled connection. `tx` wraps it in a transaction. Transient failures retry the
    * WHOLE closure on a fresh connection after a full rollback.
+   *
+   * `lock` serializes this work against other callers holding the same named lock (`GET_LOCK`). The
+   * ORDER matters and is the whole point: the lock is taken BEFORE the transaction begins (so the
+   * transaction's snapshot is taken after any earlier holder has finished) and released AFTER the
+   * transaction commits or rolls back (so a later holder can never start while this one's rows are
+   * still uncommitted and invisible to it). Releasing inside `work`, before the commit, lets a second
+   * holder read a stale snapshot and collide on primary keys.
    */
-  async runPooled(operation, work, tx) {
+  async runPooled(operation, work, tx, lock) {
     const start = performance.now();
     let attempt = 0;
     for (; ; ) {
       let conn;
       let destroy = false;
+      let held = false;
       try {
         conn = await this.acquire();
+        if (lock) {
+          const got = (await conn.query("SELECT GET_LOCK(?, ?) AS got", [
+            lock.name,
+            lock.timeoutSec
+          ]))[0][0]?.got;
+          if (got !== 1) throw new StorageError(lock.timeoutMessage);
+          held = true;
+        }
         if (tx) await conn.beginTransaction();
         const result = await work(conn);
         if (tx) await conn.commit();
@@ -2294,6 +2430,13 @@ var MySQLBackend = class {
         }
       } finally {
         if (conn) {
+          if (held && lock && !destroy) {
+            try {
+              await conn.query("SELECT RELEASE_LOCK(?)", [lock.name]);
+            } catch {
+              destroy = true;
+            }
+          }
           if (destroy) conn.destroy();
           else conn.release();
         }
@@ -2307,7 +2450,10 @@ var MySQLBackend = class {
    * connection given  -> run once on it; no retry, and no commit/rollback of the caller's transaction.
    * connection absent -> pooled, retried (see runPooled).
    */
-  async run(operation, connection, work, { tx = false } = {}) {
+  async run(operation, connection, work, {
+    tx = false,
+    lock
+  } = {}) {
     if (connection) {
       if (!isMysqlConnection(connection)) {
         throw new TypeError(
@@ -2325,7 +2471,7 @@ var MySQLBackend = class {
       }
     }
     await this.ready();
-    return this.runPooled(operation, work, tx);
+    return this.runPooled(operation, work, tx, lock);
   }
   /**
    * Make multi-statement work all-or-nothing on a CALLER-supplied connection.
@@ -2599,6 +2745,139 @@ var MySQLBackend = class {
       throw wrap("Failed to check image existence", exc);
     }
   }
+  // ---- migration: classic -> dedup --------------------------------------
+  /**
+   * One-time, NON-DESTRUCTIVE copy of every row in the classic `zerobucket_images` table into this
+   * dedup instance's tables, preserving every id (external references keep working) and
+   * deduplicating identical content on the way. The classic table is never modified. Same contract
+   * as `PostgresBackend.migrateClassicToDedup()`.
+   *
+   * Rows are copied in small batches (bounded memory instead of loading the whole table);
+   * re-running is safe (already-migrated ids are skipped and reported, not an error). Classic rows
+   * already tiered to object storage are refused up front with a clear message (their bytes are
+   * not in the database to copy) unless `skipTiered: true`, which skips and counts them instead.
+   * All-or-nothing: ONE transaction, so a failure part-way leaves the dedup tables exactly as they
+   * were. A named lock (`GET_LOCK`) serializes two migrations started at once; it is taken before the
+   * transaction and released only after it commits (see `runPooled`), so a second migration always sees
+   * the first one's rows.
+   */
+  async migrateClassicToDedup(opts = {}) {
+    if (!this.dedup) {
+      throw new StorageError(
+        "migrateClassicToDedup() must be called on an instance created with dedup: true."
+      );
+    }
+    try {
+      return await this.run(
+        "migrate_to_dedup",
+        void 0,
+        (q) => this.copyClassicToDedup(q, opts),
+        {
+          tx: true,
+          lock: {
+            name: MIGRATE_DEDUP_LOCK,
+            timeoutSec: 30,
+            timeoutMessage: "Timed out waiting for another migration to dedup to finish."
+          }
+        }
+      );
+    } catch (exc) {
+      throw wrap("Migration to dedup failed", exc);
+    }
+  }
+  async copyClassicToDedup(q, opts) {
+    const rowsOf = async (sql, params) => (await q.query(sql, params))[0];
+    if ((await rowsOf(
+      "SELECT 1 FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'zerobucket_images'"
+    )).length === 0) {
+      throw new StorageError(
+        "No classic zerobucket_images table found in this database -- nothing to migrate."
+      );
+    }
+    let tieredSkipped = 0;
+    const hasTierCol = (await rowsOf(
+      "SELECT 1 FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'zerobucket_images' AND COLUMN_NAME = 'storage_backend'"
+    )).length > 0;
+    if (hasTierCol) {
+      const tiered = Number(
+        (await rowsOf(
+          `SELECT COUNT(*) AS n FROM zerobucket_images WHERE storage_backend <> '${LOCAL}'`
+        ))[0].n
+      );
+      if (tiered > 0 && !opts.skipTiered) {
+        throw new StorageError(
+          `${tiered} classic image(s) are tiered to object storage, so their bytes are not in the database to migrate. Dedup mode does not support tiering. Migrate only an un-tiered table, bring those images back first, or pass skipTiered: true (CLI: --skip-tiered) to skip them and migrate the rest.`
+        );
+      }
+      tieredSkipped = tiered;
+    }
+    const onlyInDb = hasTierCol && tieredSkipped > 0;
+    const seen = /* @__PURE__ */ new Set();
+    let migrated = 0;
+    let already = 0;
+    let cursor = null;
+    const BATCH = 20;
+    const columns = "id, data, mime_type, original_filename, size_bytes, width, height, checksum_sha256, CAST(created_at AS CHAR) AS created_at_txt, CAST(updated_at AS CHAR) AS updated_at_txt";
+    for (; ; ) {
+      const page = cursor ? await rowsOf(
+        `SELECT ${columns} FROM zerobucket_images WHERE (created_at, id) > (?, ?)${onlyInDb ? ` AND storage_backend = '${LOCAL}'` : ""} ORDER BY created_at, id LIMIT ${BATCH}`,
+        [cursor.createdAt, cursor.id]
+      ) : await rowsOf(
+        `SELECT ${columns} FROM zerobucket_images${onlyInDb ? ` WHERE storage_backend = '${LOCAL}'` : ""} ORDER BY created_at, id LIMIT ${BATCH}`
+      );
+      if (page.length === 0) break;
+      const last = page[page.length - 1];
+      if (cursor && cursor.createdAt === last.created_at_txt && cursor.id === last.id) {
+        throw new StorageError(
+          "Migration made no progress (keyset cursor did not advance); aborting instead of looping."
+        );
+      }
+      cursor = { createdAt: last.created_at_txt, id: last.id };
+      const have = new Set(
+        (await rowsOf("SELECT id FROM zerobucket_image_refs WHERE id IN (?)", [
+          page.map((r) => r.id)
+        ])).map((r) => String(r.id))
+      );
+      const fresh = page.filter((r) => !have.has(String(r.id)));
+      already += page.length - fresh.length;
+      const groups = /* @__PURE__ */ new Map();
+      for (const r of fresh) {
+        const g = groups.get(r.checksum_sha256);
+        if (g) g.n += 1;
+        else groups.set(r.checksum_sha256, { row: r, n: 1 });
+      }
+      for (const checksum of [...groups.keys()].sort()) {
+        const { row, n } = groups.get(checksum);
+        await q.query(upsertBlobSql(n), [
+          checksum,
+          row.data,
+          row.mime_type,
+          row.size_bytes,
+          row.width,
+          row.height,
+          row.created_at_txt
+        ]);
+      }
+      for (const r of fresh) {
+        await q.query(D_INSERT_REF3, [
+          r.id,
+          r.checksum_sha256,
+          r.original_filename,
+          r.created_at_txt,
+          r.updated_at_txt
+        ]);
+        seen.add(r.checksum_sha256);
+        migrated += 1;
+      }
+    }
+    return {
+      imagesMigrated: migrated,
+      distinctBlobsCreated: seen.size,
+      duplicateReferencesFound: migrated - seen.size,
+      alreadyMigrated: already,
+      tieredSkipped
+    };
+  }
   // ---- maintenance (CLI) -----------------------------------------------
   /** Storage statistics, or null if the tables do not exist yet (nothing is created). */
   async getInfo() {
@@ -3000,7 +3279,7 @@ Usage: zerobucket <command> [options]
 Commands:
   init       Create the schema (tables and indexes) if missing.
   migrate    Apply schema migrations (currently: same as init).
-             --to-dedup  copy every classic image into the dedup tables (PostgreSQL only; non-destructive)
+             --to-dedup  copy every classic image into the dedup tables (PostgreSQL, SQLite and MySQL; non-destructive)
              --skip-tiered  with --to-dedup: skip images tiered to object storage instead of refusing
   info       Show storage stats: count, size, breakdown by format.
   verify     Re-checksum stored images to detect corruption.
@@ -3167,8 +3446,7 @@ async function cmdMigrate(v, env, io) {
     throw usageError("Error: --skip-tiered only applies with --to-dedup.");
   if (v["to-dedup"]) {
     const target = resolveTarget(v, env);
-    if (target.kind !== "postgres")
-      throw usageError("Error: --to-dedup is only supported for PostgreSQL.");
+    requireSqliteFile(target, "Nothing to migrate.");
     const backend = openBackend(target, { dedup: true, autoMigrate: true });
     await connect(backend, "connect");
     try {