zerobucket 0.11.0 → 0.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -32
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -339,7 +339,9 @@ What is different from Postgres, and why (each verified against a real MariaDB 1
|
|
|
339
339
|
`deleteMany()` run `SELECT ... FOR UPDATE` and then `DELETE` in one transaction. That is also how a tiered row's object
|
|
340
340
|
gets cleaned up.
|
|
341
341
|
- **Timestamps are written as UTC strings** (`DATETIME(6)`, naive UTC, like the Python package), so your server's or your
|
|
342
|
-
process's timezone setting cannot change what is stored.
|
|
342
|
+
process's timezone setting cannot change what is stored. The age filter behind `olderThanDays` and `tier --older-than`
|
|
343
|
+
compares against `UTC_TIMESTAMP()`, never `NOW()`; since 0.11.1 a test runs it with the session time zone at `+13:00` and
|
|
344
|
+
`-12:00` (the old suites only ever ran on UTC servers, where the two agree).
|
|
343
345
|
- **Pooling.** `mysql2` opens connections on demand, so `poolMinSize` is accepted for parity but there is no minimum to hold.
|
|
344
346
|
`mysql2` has no acquire timeout, so `poolTimeoutMs` is implemented here: waiting longer than that for a free connection
|
|
345
347
|
throws a clear `StorageError`, which is **not** retried (retrying would only wait again).
|
|
@@ -373,9 +375,9 @@ What is different from Postgres, and why (each verified against a real MariaDB 1
|
|
|
373
375
|
skipped on your connection (it cannot know whether you will commit).
|
|
374
376
|
- **Streaming** uses ranged `SUBSTRING()` reads, so a delete mid-stream raises `StorageError` (like Postgres). Tiered images
|
|
375
377
|
stream with real S3 `Range` requests.
|
|
376
|
-
- **Tested on MariaDB 10.11 and MySQL 8.0
|
|
377
|
-
|
|
378
|
-
MariaDB 10.2.1+ or MySQL 8.0.16+: older servers accept the `CHECK` constraint syntax but ignore it.
|
|
378
|
+
- **Tested on MariaDB 10.11 and MySQL 8.0** (CI runs the MySQL, dedup and CLI test suites and the Node/Python conformance
|
|
379
|
+
harness on both; the schema comparison was also run by hand on 8.0.46). Other versions are expected to behave the same but have not
|
|
380
|
+
been run. Use MariaDB 10.2.1+ or MySQL 8.0.16+: older servers accept the `CHECK` constraint syntax but ignore it.
|
|
379
381
|
|
|
380
382
|
## Deduplication
|
|
381
383
|
|
|
@@ -457,31 +459,29 @@ same corpus of valid and corrupt files identically.
|
|
|
457
459
|
|
|
458
460
|
## Differences from the Python package
|
|
459
461
|
|
|
460
|
-
Python column = zerobucket on PyPI **0.
|
|
461
|
-
0.27.0 to 0.
|
|
462
|
-
|
|
463
|
-
| | Python
|
|
464
|
-
| -------------------------------------------------------------- |
|
|
465
|
-
| API | sync + async clients
|
|
466
|
-
| Async
|
|
467
|
-
| `optimize` | Pillow; EXIF stripped, not applied
|
|
468
|
-
| JPEG encoder in `optimize` | Pillow, progressive + optimized
|
|
469
|
-
| JPEG missing only its last 1-2 bytes (the end-of-image marker) | accepted (Pillow reads every pixel)
|
|
470
|
-
| `putMany` inserts | pipelined
|
|
471
|
-
| Formats | JPEG, PNG, WebP, HEIC (`[heic]` extra)
|
|
472
|
-
| Convert TO HEIC with `optimize` | yes (pillow-heif)
|
|
473
|
-
| Event duration | `duration_seconds`
|
|
474
|
-
| `migrateClassicToDedup`, tiered classic rows | always **skips** them and reports `tiered_rows_skipped`
|
|
475
|
-
| `migrateClassicToDedup`, other behaviour | batched, safe to re-run
|
|
476
|
-
|
|
|
477
|
-
| SQLite
|
|
478
|
-
| SQLite
|
|
479
|
-
|
|
|
480
|
-
|
|
|
481
|
-
|
|
|
482
|
-
|
|
|
483
|
-
| S3 delete failure after `delete()` | never fails the delete; reported as an `object_storage_delete` event (since 0.29.0) | never fails the delete; reported via `onOperation` |
|
|
484
|
-
| Options | snake_case kwargs | camelCase options object |
|
|
462
|
+
Python column = zerobucket on PyPI **0.33.0 or later** (several of its behaviours below were fixed or added in 0.25.1, 0.26.0 and
|
|
463
|
+
0.27.0 to 0.33.0; older versions differ).
|
|
464
|
+
|
|
465
|
+
| | Python | npm |
|
|
466
|
+
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
|
|
467
|
+
| API | sync + async clients | one async client |
|
|
468
|
+
| Async client feature set | everything the sync client has: dedup and tiering (since 0.30.0), hooks, `optimize` and `validator` (since 0.32.0), `connection=` (since 0.33.0); a hook may be a plain or an `async def` function | one async client, always, with all of them |
|
|
469
|
+
| `optimize` | Pillow; EXIF stripped, not applied | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG |
|
|
470
|
+
| JPEG encoder in `optimize` | Pillow, progressive + optimized | sharp with `mozjpeg`, progressive: on smooth photos roughly **half the size** at about the same SSIM (measured); WebP and PNG match |
|
|
471
|
+
| JPEG missing only its last 1-2 bytes (the end-of-image marker) | accepted (Pillow reads every pixel) | **rejected** (libvips treats a premature end as an error); truncated interlaced PNGs are rejected by both since npm 0.10.0 |
|
|
472
|
+
| `putMany` inserts | pipelined | one transaction, statements run sequentially |
|
|
473
|
+
| Formats | JPEG, PNG, WebP, HEIC (`[heic]` extra) | JPEG, PNG, WebP, HEIC (optional `libheif-js`) |
|
|
474
|
+
| Convert TO HEIC with `optimize` | yes (pillow-heif) | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format` |
|
|
475
|
+
| Event duration | `duration_seconds` | `durationMs` |
|
|
476
|
+
| `migrateClassicToDedup`, tiered classic rows | always **skips** them and reports `tiered_rows_skipped` | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python |
|
|
477
|
+
| `migrateClassicToDedup`, other behaviour | batched, safe to re-run. PostgreSQL: server-side cursor and **no lock**, so two runs at once make the second fail with a unique-key error and roll back (nothing is double-counted). SQLite: write lock. MySQL/MariaDB: named lock (the same one npm uses), so concurrent runs cannot race there | batched, safe to re-run (keyset paging); concurrent runs cannot race (PostgreSQL advisory lock, MySQL named lock, SQLite write lock) |
|
|
478
|
+
| SQLite API | synchronous `SQLiteBackend` + separate async backend | one async client; the sync driver runs on the event loop, lock waits are async (never block it) |
|
|
479
|
+
| SQLite `onOperation` / busy retry | `on_operation` events and busy retry, sync and async (since 0.27.0) | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`) |
|
|
480
|
+
| SQLite stream deleted mid-read | sync adapter keeps reading a WAL snapshot (async adapter raises) | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot |
|
|
481
|
+
| MySQL `onOperation` / retry | yes, sync and async (since 0.27.0) | yes (same client for everything) |
|
|
482
|
+
| `connection=` without an open transaction | multi-statement ops rely on the driver's implicit transaction | always atomic (own transaction or savepoint) |
|
|
483
|
+
| S3 delete failure after `delete()` | never fails the delete; reported as an `object_storage_delete` event (since 0.29.0) | never fails the delete; reported via `onOperation` |
|
|
484
|
+
| Options | snake_case kwargs | camelCase options object |
|
|
485
485
|
|
|
486
486
|
Both packages treat a malformed id as "not found" on every backend (Python 0.26.0+, npm 0.9.0+) and never pass one to a
|
|
487
487
|
`before_get`/`beforeGet` hook. Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
|
|
@@ -528,13 +528,14 @@ memory, event-loop impact, throughput, and the before and after of each release)
|
|
|
528
528
|
[Operations Guide](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/docs/OPERATIONS.md).
|
|
529
529
|
- **Dedup and tiering cannot be combined** (a shared blob has many owners): `dedup: true` with `objectStorage` throws.
|
|
530
530
|
- **Tiered classic images cannot be migrated** by `migrate --to-dedup` (see Deduplication for `skipTiered`). The migration itself
|
|
531
|
-
works on PostgreSQL, SQLite and MySQL/MariaDB since 0.11.0
|
|
531
|
+
works on PostgreSQL, SQLite and MySQL/MariaDB (npm since 0.11.0, the Python package since 0.31.0).
|
|
532
532
|
- **HEIC can be read and converted from, but not written.** There is no HEVC encoder for Node, so `optimize` cannot
|
|
533
533
|
produce HEIC.
|
|
534
534
|
- **SQLite allows one writer at a time**; lock waits are async and never block the event loop, but write-heavy
|
|
535
535
|
workloads belong on PostgreSQL or MySQL/MariaDB.
|
|
536
|
-
- **Tested versions:** CI runs PostgreSQL 16
|
|
537
|
-
|
|
536
|
+
- **Tested versions:** CI runs PostgreSQL 16, MariaDB 10.11 and MySQL 8.0 on Node 20, 22 and 24, and the SQLite and
|
|
537
|
+
filesystem tests on Windows (GitHub's Windows runners cannot run database servers, so PostgreSQL, MySQL and S3 are not
|
|
538
|
+
tested there). Other versions are expected to behave the same but have not been run.
|
|
538
539
|
|
|
539
540
|
## Runtime notes
|
|
540
541
|
|