@harperfast/rocksdb-js 2.7.1 → 2.8.0

package/README.md

@@ -11,6 +11,8 @@ A Node.js binding for the RocksDB library.
 - Custom stores provide ability to override default database interactions
 - Efficient binary key and value encoding
 - Configurable block/blob compression (LZ4, Zstd, Zlib, and more)
+- Observable background errors via the `'error'` event or `db.getLastError()`, with in-process
+  recovery (`db.resume()`)
 - Access to internal RocksDB statistics
 - Designed for Node.js and Bun on Linux, macOS, and Windows
 
@@ -53,6 +55,16 @@ Creates a new database instance.
     when linked, else no compression). See [Compression](#compression). Throws if the algorithm is
     not compiled into the native build — check [`supportedCompression`](#supportedcompression) for
     the available list.
+  - `compressionForAllColumnFamilies: boolean` When `true`, applies `compression` to every column
+    family opened for the database rather than only the column family specified by `name`. Requires an explicit
+    `compression`. Defaults to `false`. See [Compression](#compression).
+  - `dbWriteBufferSize: number` The total memtable memory budget in bytes shared across all of the
+    database's column families. When the combined size of all memtables reaches this value, RocksDB
+    flushes the largest one. `0` (the default) disables this global trigger, so per-column-family
+    `writeBufferSize` alone drives flushing. This is distinct from the process-wide
+    [`writeBufferManagerSize`](#dbconfigoptions) config option. Database-wide, so it binds when the
+    path is first opened in this process: a later open of the same path — including from another
+    worker thread — keeps the first opener's value rather than overriding or rejecting it.
   - `disableWAL: boolean` Whether to disable the RocksDB write ahead log. Defaults to `false`.
   - `enableStats: boolean` When `true` and the database is open, RocksDB will captures stats that
     are retrieved by calling `db.getStats()`. Enabling statistics imposes 5-10% in overhead.
@@ -76,6 +88,17 @@ Creates a new database instance.
     compaction falls behind under sustained ingest); a positive `int32` is an explicit cap. Reads
     only pay a reopen cost when the number of live table files exceeds the budget, so raise the
     process fd limit (and with it the derived budget) for very large databases.
+  - `maxWriteBufferNumber: number` The maximum number of memtables that can be queued per column
+    family before writes stall. Higher values absorb write bursts while flushes catch up, at the
+    cost of memory (roughly `maxWriteBufferNumber * writeBufferSize` per column family). Defaults to
+    `16`.
+  - `maxWriteBufferSizeToMaintain: number` The number of bytes of recent memtable history to keep in
+    memory for transaction conflict checking. `-1` (the default) derives the value from
+    `maxWriteBufferNumber * writeBufferSize` (the RocksDB-recommended default for optimistic
+    transactions) — except when a stalling [`writeBufferManager`](#dbconfigoptions) is configured
+    (`writeBufferManagerSize > 0` with `writeBufferManagerAllowStall`), in which case it resolves to
+    `0` to avoid retaining memtable history the manager will never release. An explicit non-negative
+    value is always honored as-is.
   - `name: string` The column family name. Defaults to `"default"`.
   - `noBlockCache: boolean` When `true`, disables the block cache. Block caching is enabled by
     default and the cache is shared across all database instances.
@@ -85,7 +108,10 @@ Creates a new database instance.
     until commit. Defaults to `false`.
   - `readOnly: boolean` When `true`, the database is opened in read-only mode. Read operations are
     permitted. Write operations will throw an error with code `ERR_DATABASE_READONLY`. Transactions
-    are a no-op in read-only mode.
+    are a no-op in read-only mode. [`flush()`](#dbflushoptions-promisevoid),
+    [`flushSync()`](#dbflushsyncoptions-void), [`compact()`](#dbcompactoptions-promisevoid) and
+    [`compactSync()`](#dbcompactsyncoptions-void) are the exception: they have nothing to do when
+    there are no writes, so they succeed as no-ops rather than throwing.
   - `statsLevel: StatsLevel` Controls which type of statistics to skip and reduce statistic
     overhead. Defaults to `StatsLevel.ExceptDetailedTimers`.
   - `store: Store` A custom store that handles all interaction between the `RocksDatabase` or
@@ -96,10 +122,9 @@ Creates a new database instance.
     modified time to be older than the retention period before it is rotated to the next sequence
     number. Value must be between `0.0` and `1.0`. A threshold of `0.0` means ignore age check.
     Defaults to `0.75`.
-  - `transactionLogMaxSize: number` The maximum size of a transaction log file. If a log file is
-    empty, the first log entry will always be added regardless if it's larger than the max size. If
-    a log file is not empty and the entry is larger than the space available, the log file is
-    rotated to the next sequence number. Defaults to 16 MB.
+  - `transactionLogMaxSize: number` The target maximum size of a transaction log file. Transactions
+    are never split across files: if the complete transaction does not fit, the log rotates before
+    writing it. A transaction written to an empty file may exceed the target. Defaults to 16 MB.
   - `transactionLogRetention: string | number` The number of minutes to retain transaction logs
     before purging. Defaults to `'3d'` (3 days).
   - `transactionLogsPath: string` The path to store transaction logs. Defaults to
@@ -109,6 +134,9 @@ Creates a new database instance.
     the verification slot for each written key. Enable this only for column families whose records
     are cached (e.g. the primary column family of a table). Defaults to `false`. Requires
     `verificationTableEntries` to be configured before the first database is opened.
+  - `writeBufferSize: number` The per-column-family memtable size in bytes at which the memtable is
+    sealed and flushed to an SST file. Smaller values produce more frequent, faster flushes; larger
+    values batch more writes per SST file at the cost of memory. Defaults to `16777216` (16 MB).
 
 ### `db.close()`
 
@@ -146,18 +174,6 @@ const db = RocksDatabase.open('path/to/db', {
 console.log(db.compression); // { algorithm: 'zstd', level: 3 }
 ```
 
-### `db.logOptions: { maxLogFileSize: number, infoLogLevel: number }`
-
-Returns the informational-log settings currently in effect for this database, read live from
-RocksDB. These are database-wide settings (not per-column-family). `maxLogFileSize` is the
-per-file size cap for informational log files (`LOG` / `LOG.old.*`); `infoLogLevel` is the logging
-verbosity. The database must be open.
-
-```typescript
-const db = RocksDatabase.open('path/to/db', { maxLogFileSize: 4 * 1024 * 1024 });
-console.log(db.logOptions); // { maxLogFileSize: 4194304, infoLogLevel: 1 }
-```
-
 ### `db.config(options)`
 
 Sets global database settings.
@@ -203,6 +219,18 @@ Returns `true` if the database is open, otherwise false.
 console.log(db.isOpen()); // true or false
 ```
 
+### `db.logOptions: { maxLogFileSize: number, infoLogLevel: number }`
+
+Returns the informational-log settings currently in effect for this database, read live from
+RocksDB. These are database-wide settings (not per-column-family). `maxLogFileSize` is the
+per-file size cap for informational log files (`LOG` / `LOG.old.*`); `infoLogLevel` is the logging
+verbosity. The database must be open.
+
+```typescript
+const db = RocksDatabase.open('path/to/db', { maxLogFileSize: 4 * 1024 * 1024 });
+console.log(db.logOptions); // { maxLogFileSize: 4194304, infoLogLevel: 1 }
+```
+
 ### `db.name: string`
 
 Returns the database column family's name.
@@ -302,9 +330,13 @@ await db.compact({ start: 'a', end: 'z' });
 await db.compact({ bottommost: true });
 ```
 
+On a [read-only](#new-rocksdatabasepath-options) database this is a no-op: arguments are still
+validated, but the returned promise resolves without compacting rather than rejecting.
+
 ### `db.compactSync(options?): void`
 
-Synchronous version of `compact()`.
+Synchronous version of `compact()`. On a [read-only](#new-rocksdatabasepath-options) database it
+validates its arguments and then returns without compacting, rather than throwing.
 
 ```typescript
 db.compactSync();
@@ -345,23 +377,63 @@ db.dropSync();
 db.close();
 ```
 
-### `db.flush(): Promise<void>`
+### `db.flush(options?): Promise<void>`
 
 Flushes all in-memory data to disk asynchronously.
 
+- `options: object`
+  - `allowWriteStall?: boolean` Whether the flush may proceed even though it will stall writes for
+    its duration. Defaults to `false`, which — despite how that reads — means the flush **waits**
+    until it can run without causing a stall, with no timeout: on a database stuck in a stall
+    condition (immutable-memtable backlog, L0 stop trigger, an exhausted `WriteBufferManager`
+    budget), the returned promise never settles, and since `flush()` runs on the libuv threadpool
+    (default size 4), a handful of concurrently stalled flushes can exhaust the pool and stall
+    every unrelated `fs`/`dns`/`crypto` call and cold-cache `get()` in the process, not only this
+    database's. Pass `true` up front, before issuing a flush you expect might stall, when it's a
+    durability gate you'd rather have stall writers than wait indefinitely — there's no way to
+    cancel an in-flight `flush()`, so this isn't a rescue for one already hung, and forcing the
+    memtable switch can itself prolong an existing L0 stop-trigger condition rather than clear it.
+    That cost is database-wide, covering every column family on the (process-global,
+    `worker_threads`-shared) database handle, not just the caller's — and it relocates the hang
+    rather than removing it: a stalled write blocks the database's single commit thread, which
+    dispatches every `Transaction.commit()` in order, so every commit behind it queues up too,
+    including ones from callers that never touched flush. This is a different knob from
+    `writeBufferManagerAllowStall` (see [`new RocksDatabase()`](#new-rocksdatabasepath-options)
+    options), with nearly opposite polarity: that one governs whether the `WriteBufferManager` may
+    stall writers at all, this one governs whether one manual flush is willing to cause a stall
+    rather than wait one out.
+
 ```typescript
 await db.flush();
+
+// Chosen up front, as a durability gate willing to pay the stall cost
+await db.flush({ allowWriteStall: true });
 ```
 
-### `db.flushSync(): void`
+On a [read-only](#new-rocksdatabasepath-options) database this is a no-op: the options bag is still
+validated, but the returned promise resolves without flushing rather than rejecting.
+
+### `db.flushSync(options?): void`
 
 Flushes all in-memory data to disk synchronously. Note that this can be an expensive operation, so
 it is recommended to use `flush()` if you want to keep the event loop free.
 
+- `options: object` Same as [`flush()`](#dbflushoptions-promisevoid). The default
+  `allowWriteStall: false` wait is taken on the calling thread, which here is the JS thread — so
+  the hazard described there is strictly larger on this entry point, not smaller: instead of
+  parking one libuv pool worker it freezes the event loop outright, and it holds the in-flight
+  operation claim that `close()` waits on, so the database cannot be closed out from under it
+  either. Prefer `flush()` if there is any chance the database is in a stall condition.
+
 ```typescript
 db.flushSync();
+
+db.flushSync({ allowWriteStall: true });
 ```
 
+On a [read-only](#new-rocksdatabasepath-options) database it validates its options and then returns
+without flushing, rather than throwing.
+
 ### `db.get(key: Key, options?: GetOptions): MaybePromise<any>`
 
 Retreives the value for a given key. If the key does not exist, it will resolve `undefined`.
@@ -408,13 +480,75 @@ the `expectedVersion` option is used.
 ### `db.getEstimatedKeyCount(): number`
 
 Retrieves the estimated number of keys in the database. This is an alias for
-`db.getDBIntProperty('rocksdb.estimate-num-keys')`.
+`db.getDBIntProperty('rocksdb.estimate-num-keys')`; use `estimateCount()` for range support and a
+confidence indicator.
 
 ```typescript
 const estimated = db.getEstimatedKeyCount();
 console.log(estimated);
 ```
 
+### `db.estimateCount(options?: CountEstimateOptions): CountEstimate`
+
+Estimates the number of keys in the database, or within a key range, returning
+`{ count, confidence }`. Unlike `getKeysCount()`, this never iterates: the estimate is derived
+from RocksDB statistics (memtable stats plus approximate SST sizes converted through the entry
+density of the SSTs overlapping the range), so its cost scales with the number of SSTs overlapping
+the range rather than the number of keys. Reading cold table properties can do I/O through the
+table cache, so narrow ranges are preferable. A start-only range is computed as the
+whole-database estimate minus the complement, so it does the work of the range _below_ `start`.
+Accuracy improves with range size. Resolution is bounded by SST data-block granularity, so a range
+narrower than a block is unreliable in either direction: it may over-report or report 0 for present
+keys, and its low `confidence` is the signal. Recently deleted or overwritten entries may be counted
+until compaction.
+Estimates always reflect committed state; writes pending in a transaction are not included. Set
+`reverse: true` to use `getRange()`'s reverse convention (`start` is the upper bound and `end` is
+the lower bound). An inverted range (`start` ≥ `end`) returns
+`{ count: 0, confidence: 1 }`.
+
+`confidence` is a heuristic 0–1 indicator of how trustworthy `count` is — exactly 1 only when the
+count is exact. It is derived from the estimate's resolution (data-block/memtable-sampling
+granularity relative to the count), the tombstone fraction of the overlapping SSTs, and — for
+start-only ranges — the error compounded by complement subtraction. Treat it as an ordering
+signal (e.g. when to trust an estimate for query planning vs fall back to a heuristic), not a
+statistical bound.
+
+```typescript
+const { count, confidence } = db.estimateCount({ start: 'a', end: 'z' });
+```
+
+### `db.createCountEstimator(options?: CountEstimatorOptions): CountEstimator`
+
+Creates an estimator that progressively refines a range count estimate while the range is being
+iterated — useful for reporting a total alongside a page of results without scanning the full
+range. Before any traversal, `estimate()` returns the pure statistical estimate (same as
+`estimateCount(range)`). As the caller reports progress with `advance(lastKey, count)` (e.g. once
+per page), `estimate()` returns the exact traversed count plus a statistical estimate of the
+remainder, calibrated by the observed ratio of actual-to-estimated entries over the portion
+already traversed — so the count converges toward the exact total. `confidence` is the
+exactness-weighted blend of the traversed portion and the remainder's confidence, so it approaches 1
+as the exact portion grows, although a checkpoint may decrease when calibration makes a large
+correction.
+Each checkpoint reads committed state, so a traversal performed against a transaction snapshot may
+be calibrated against data committed after that snapshot.
+When traversal completes, call `finish()` and `estimate()` returns the exact count with
+confidence 1. Reverse ranges follow `getRange`: set `start` to the upper bound and `end` to the
+lower bound, then set `reverse: true`. The caller owns the progress contract: cursors must move
+monotonically through the range and each entry must be reported exactly once.
+
+```typescript
+const range = { start: 'a', end: 'z' };
+const estimator = db.createCountEstimator(range);
+let lastKey;
+let pageSize = 0;
+for (const { key } of db.getRange({ ...range, limit: 25 })) {
+	lastKey = key;
+	pageSize++;
+}
+estimator.advance(lastKey, pageSize);
+const { count, confidence } = estimator.estimate();
+```
+
 ### `db.getKeys(options?: IteratorOptions): ExtendedIterable`
 
 Retrieves all keys within a range.
@@ -692,6 +826,15 @@ not committed after the configured number of coordinated retries, the transactio
 an `ERR_TRANSACTION_ABANDONED` error. Coordinated retry requires the column family to be opened with
 `verificationTable: true`.
 
+That wait is bounded so a conflicting transaction that is never committed or aborted cannot block
+the commit forever: if the write intent has not been released after `ROCKSDB_JS_PARK_TIMEOUT_MS`
+(default `5000`), the commit resolves anyway and consumes a retry attempt exactly as a real release
+would. A conflicting transaction held for longer than roughly `maxRetries` times that timeout
+therefore ends in `ERR_TRANSACTION_ABANDONED` rather than waiting indefinitely. Deployments where
+waiting is preferable to failing should raise the timeout; there is no way to disable the bound.
+In particular `0` does not disable it: `0`, negative, and unparseable values all fall back to the
+`5000` default, and a value between `1` and `49` is clamped up to `50`.
+
 ### Class: `Transaction`
 
 The transaction callback is passed in a `Transaction` instance which contains all of the same data
@@ -768,6 +911,73 @@ await db.transaction(async (txn) => {
 });
 ```
 
+## Error Handling
+
+### `db.getLastError(): BackgroundError | null`
+
+Returns the most recent [`BackgroundError`](#event-error) observed on this database, or `null` when
+none has occurred. This is the **pull** counterpart to the `'error'` [event](#event-error): use it
+for an on-demand check (e.g. a health probe) or to catch an error that fired before a listener was
+attached. The value is historical — it is **not** cleared by a successful
+[`db.resume()`](#dbresume-void); reset it explicitly with
+[`db.setLastError(null)`](#dbsetlasterrorerror-void). When the returned error's `writesDisabled` is
+`true`, writes are stopped until recovery.
+
+```typescript
+const err = db.getLastError();
+if (err?.writesDisabled) {
+	// ...free disk space, then...
+	db.resume();
+}
+```
+
+### `db.setLastError(error?): void`
+
+Sets or clears the last background error, mirroring the Win32 `SetLastError` / `GetLastError` pair.
+
+- **Clear** — pass `null` (or no argument). [`db.getLastError()`](#dbgetlasterror-backgrounderror--null)
+  then returns `null` and no event fires. This is how you reset the error after handling or
+  recovering it (e.g. after [`db.resume()`](#dbresume-void)) — the equivalent of
+  `SetLastError(ERROR_SUCCESS)`.
+- **Set** — pass an object (`{ message, severity?, severityName?, writesDisabled?, reason?, reasonName? }`;
+  `type` defaults to `'background'`). It is stored and the `'error'` [event](#event-error) fires with
+  the reconstructed [`BackgroundError`](#event-error). Useful for surfacing an application-level
+  "this database is unusable" state, and for deterministically exercising the error path in tests.
+
+```typescript
+// reset after recovery
+db.resume();
+db.setLastError(null);
+
+// inject (e.g. in a test)
+db.on('error', (err) => console.error(err.message));
+db.setLastError({
+	message: 'disk quota exceeded',
+	severity: 2,
+	severityName: 'hard',
+	writesDisabled: true,
+});
+```
+
+### `db.resume(): void`
+
+Attempts to recover the database from a background error (see the `'error'` [event](#event-error))
+by calling RocksDB's `DB::Resume()`. Call this after the underlying condition has cleared — e.g.
+once disk space has been freed. On success writes are accepted again (and RocksDB can resume the
+obsolete-file cleanup it was blocking); on failure — the condition has not actually cleared — it
+throws with the RocksDB error and the database stays read-only. A no-op on a healthy database.
+
+Runs synchronously and may briefly block, since recovery can re-flush memtables.
+
+```typescript
+db.on('error', (err) => {
+	if (err.writesDisabled) {
+		// ...free disk space, then...
+		db.resume();
+	}
+});
+```
+
 ## Events
 
 ### Event: `'aftercommit'`
@@ -794,6 +1004,57 @@ The `'committed'` event is emitted after the transaction has been written. When
 emitted, the transaction is still cleaning up. If you need to know when the transaction is fully
 complete, use the `'aftercommit'` event.
 
+### Event: `'error'`
+
+When a write fails at the filesystem level — a full disk, an exhausted quota — RocksDB records a
+background error and, for a hard-or-worse severity, stops accepting writes. The database emits an
+`'error'` event carrying a `BackgroundError` so a consumer can observe it and recover in-process
+instead of restarting. When `writesDisabled` is `true`, call
+[`db.resume()`](#dbresume-void) after the underlying condition clears. Register the listener with
+`db.on('error', ...)` before the database is put under load — or use the pull-based
+[`db.getLastError()`](#dbgetlasterror-backgrounderror--null) to catch an error that fired earlier.
+
+`BackgroundError` is a real `Error` subclass, exported from the package, so both
+`err instanceof Error` and `err instanceof BackgroundError` hold. It extends `Error` with:
+
+- `message: string` — the RocksDB error string.
+- `name: string` — always `'BackgroundError'`.
+- `type: string` — the error-class discriminator; always `'background'`.
+- `severity: number` — the RocksDB `Status::Severity`: `1` soft, `2` hard, `3` fatal,
+  `4` unrecoverable.
+- `severityName: string` — `'soft'`, `'hard'`, `'fatal'`, or `'unrecoverable'`.
+- `writesDisabled: boolean` — whether RocksDB has disabled writes on the database (`severity >= 2`).
+  Only then is `resume()` warranted; a soft error is auto-recoverable and leaves writes enabled.
+  Distinct from opening the database in read-only mode.
+- `reason?: number` / `reasonName?: string` — the originating `BackgroundErrorReason`
+  (e.g. `'flush'`, `'compaction'`), present when the error came from a reason-bearing callback.
+
+```typescript
+import { BackgroundError } from '@harperfast/rocksdb-js';
+
+db.on('error', (err) => {
+	if (err instanceof BackgroundError && err.writesDisabled) {
+		console.error(`writes disabled (${err.severityName}): ${err.message}`);
+	}
+});
+```
+
+```typescript
+db.on('error', (err) => {
+	if (err.writesDisabled) {
+		// Hard-or-worse: RocksDB has disabled writes on this database. They stay
+		// disabled until you clear the underlying condition and call db.resume().
+		console.error(`database writes disabled (${err.severityName}): ${err.message}`);
+	} else {
+		// Soft error: writes are NOT disabled and RocksDB will auto-recover. This
+		// is typically a transient background hiccup (e.g. a retryable I/O error
+		// during flush or compaction), surfaced for visibility only — no action
+		// is required.
+		console.warn(`recoverable background error (${err.severityName}): ${err.message}`);
+	}
+});
+```
+
 ## Event API
 
 `rocksdb-js` provides a EventEmitter-like API that lets you asynchronously notify events to one or
@@ -1705,7 +1966,7 @@ snapshots of a database. A backup covers the **entire database** — every colum
 manifest, and (by default) the write-ahead log — so it is not scoped to an individual `Store`.
 
 A backup can be written to a local **directory** (incremental, with a management API) or streamed to
-a **`WritableStream`** as a tar archive with no intermediate copy on disk. See
+a `WritableStream` as a tar archive with no intermediate copy on disk. See
 [docs/backups.md](docs/backups.md) for a full guide covering both modes, restore, checkpoints, and
 caveats.
 
@@ -1781,10 +2042,10 @@ required bytes — before any files are copied, so a full destination fails fast
 through (a mid-copy failure can leave zero usable backups). The required size is estimated
 conservatively as a full copy of the database's live files, plus the transaction-log snapshot when
 `transactionLogs` is set; since a backup only ever copies files, this can never under-estimate.
-Incremental backups to a right-sized volume may therefore be over-rejected — set `checkDiskSpace:
-false` to skip the check. It also self-disables when free space can't be determined (e.g. some
-network filesystems report `0`), so it never blocks a backup on an untrustworthy number. Only
-directory-target backups are checked; the streaming backup path is unaffected.
+Incremental backups to a right-sized volume may therefore be over-rejected — set
+`checkDiskSpace: false` to skip the check. It also self-disables when free space can't be determined
+(e.g. some network filesystems report `0`), so it never blocks a backup on an untrustworthy number.
+Only directory-target backups are checked; the streaming backup path is unaffected.
 
 ### `db.backup(stream: WritableStream<Uint8Array>, options?: BackupStreamOptions): Promise<void>`