zerobucket 0.9.0 → 0.9.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +46 -25
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -453,34 +453,55 @@ same corpus of valid and corrupt files identically.
|
|
|
453
453
|
|
|
454
454
|
## Differences from the Python package
|
|
455
455
|
|
|
456
|
-
Python column = zerobucket on PyPI **0.
|
|
457
|
-
differ).
|
|
458
|
-
|
|
459
|
-
| | Python | npm
|
|
460
|
-
| -------------------------------------------- | --------------------------------------------------------------------- |
|
|
461
|
-
| API | sync + async clients | one async client
|
|
462
|
-
|
|
|
463
|
-
|
|
|
464
|
-
| `
|
|
465
|
-
| `
|
|
466
|
-
|
|
|
467
|
-
|
|
|
468
|
-
|
|
|
469
|
-
| `migrateClassicToDedup`,
|
|
470
|
-
| `
|
|
471
|
-
| SQLite
|
|
472
|
-
| SQLite
|
|
473
|
-
|
|
|
474
|
-
|
|
|
475
|
-
|
|
|
476
|
-
|
|
|
477
|
-
|
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
|
|
456
|
+
Python column = zerobucket on PyPI **0.26.0 or later** (several of its behaviours below were fixed in 0.25.1 and 0.26.0; older
|
|
457
|
+
versions differ).
|
|
458
|
+
|
|
459
|
+
| | Python | npm |
|
|
460
|
+
| -------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
461
|
+
| API | sync + async clients | one async client |
|
|
462
|
+
| `optimize` | Pillow; EXIF stripped, not applied | sharp; EXIF **applied then stripped**; transparency → **white** for JPEG |
|
|
463
|
+
| JPEG encoder in `optimize` | Pillow, progressive + optimized | sharp with `mozjpeg`, progressive: on smooth photos roughly **half the size** at about the same SSIM (measured); WebP and PNG match |
|
|
464
|
+
| `putMany` inserts | pipelined | one transaction, statements run sequentially |
|
|
465
|
+
| Formats | JPEG, PNG, WebP, HEIC (`[heic]` extra) | JPEG, PNG, WebP, HEIC (optional `libheif-js`) |
|
|
466
|
+
| Convert TO HEIC with `optimize` | yes (pillow-heif) | no (no HEVC encoder for Node); converting FROM HEIC works; `optimize: true` on a HEIC needs an explicit `format` |
|
|
467
|
+
| Event duration | `duration_seconds` | `durationMs` |
|
|
468
|
+
| `migrateClassicToDedup`, tiered classic rows | always **skips** them and reports `tiered_rows_skipped` | **refuses** by default (changes nothing); `skipTiered: true` / `--skip-tiered` skips and reports like Python |
|
|
469
|
+
| `migrateClassicToDedup`, other behaviour | batched, safe to re-run (server-side cursor) | batched, safe to re-run (keyset paging, advisory lock so concurrent runs cannot race) |
|
|
470
|
+
| SQLite API | synchronous `SQLiteBackend` + separate async backend | one async client; the sync driver runs on the event loop, lock waits are async (never block it) |
|
|
471
|
+
| SQLite `onOperation` / busy retry | none | `onOperation` events and async `SQLITE_BUSY` retry (`busyTimeoutMs`) |
|
|
472
|
+
| SQLite stream deleted mid-read | sync adapter keeps reading a WAL snapshot (async adapter raises) | raises `StorageError`, like Postgres; use `connection` in a transaction for a snapshot |
|
|
473
|
+
| MySQL `onOperation` / retry | sync `MySQLBackend` yes; `AsyncMySQLBackend` none | yes (same client for everything) |
|
|
474
|
+
| CLI | PostgreSQL, classic mode only; `verify` skips tiered rows with a note | PostgreSQL, SQLite **and** MySQL/MariaDB, classic **and** dedup; `verify --bucket` also checks tiered images; `migrate --to-dedup` |
|
|
475
|
+
| `connection=` without an open transaction | multi-statement ops rely on the driver's implicit transaction | always atomic (own transaction or savepoint) |
|
|
476
|
+
| S3 delete failure after `delete()` | raises after the row is gone | never fails the delete; reported via `onOperation` |
|
|
477
|
+
| Options | snake_case kwargs | camelCase options object |
|
|
478
|
+
|
|
479
|
+
Both packages treat a malformed id as "not found" on every backend (Python 0.26.0+, npm 0.9.0+) and never pass one to a
|
|
480
|
+
`before_get`/`beforeGet` hook. Both packages now leave a tiered image's object in place on `delete(id, connection=...)` (a rolled-back transaction can
|
|
481
481
|
never lose data), take dedup batch locks in sorted checksum order (overlapping batches cannot deadlock), and skip the
|
|
482
482
|
MySQL `ALTER TABLE` at startup when the schema is current.
|
|
483
483
|
|
|
484
|
+
## Performance
|
|
485
|
+
|
|
486
|
+
Measured on one single-core container, so treat it as directional; the full tables (PostgreSQL, SQLite, MariaDB, per size,
|
|
487
|
+
memory, event-loop impact, throughput) are in the
|
|
488
|
+
[benchmark results](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/benchmarks/node/RESULTS.md).
|
|
489
|
+
|
|
490
|
+
| Image size | `put()` (PostgreSQL) | `get()` | Notes |
|
|
491
|
+
| ------------ | -------------------- | ----------- | ------------------------------------------------------------------- |
|
|
492
|
+
| up to ~500KB | under 60ms | under 5ms | Feels like any other write. |
|
|
493
|
+
| 0.5-1.5MB | about 60-160ms | under 10ms | Fine for uploads. |
|
|
494
|
+
| about 5MB | about 0.6s | about 45ms | About 58MB of transient memory per upload. |
|
|
495
|
+
| about 10MB | about 1.2-1.4s | about 120ms | About 130MB of transient memory per upload. Default limit is 8 MiB. |
|
|
496
|
+
|
|
497
|
+
- `put()` time is mostly image validation (a full decode), so it follows the image's pixel count, not the database you pick.
|
|
498
|
+
- The decode runs off the main thread: the event loop stalled at most about 16ms during a 10MB `put()`.
|
|
499
|
+
- Memory is the decoded image, roughly 11x the file size for a 45-megapixel JPEG. Cap `maxBytes` and `maxPixels` on public endpoints.
|
|
500
|
+
- Stored size equals the file size: JPEG, PNG and WebP are already compressed, so the database cannot shrink them further.
|
|
501
|
+
- Compression quality of `optimize` is measured in the
|
|
502
|
+
[compression results](https://github.com/KedarGhadyalji/ZeroBucket/blob/main/benchmarks/node/COMPRESSION_RESULTS.md).
|
|
503
|
+
Do not convert flat graphics (screenshots, logos) to JPEG; they grow.
|
|
504
|
+
|
|
484
505
|
## Limitations (read before using in production)
|
|
485
506
|
|
|
486
507
|
- **Images live in your database.** That is the point (one backup, one transaction, no extra service), but it means
|