@harperfast/rocksdb-js 2.11.0 → 2.13.0

package/README.md

@@ -46,6 +46,19 @@ Creates a new database instance.
 - `path: string` The path to write the database files to. This path does not need to exist, but the
   parent directories do.
 - `options: object` [optional]
+  - `arenaBlockSize: number` The size in bytes of the blocks a column family's memtable allocates
+    from. Every column family that has been written to holds at least one block, and with a
+    `writeBufferManagerSize` set, those blocks count against that shared budget even when the
+    column family is idle — so with many column families, large blocks can use up most of a small
+    budget and cause constant tiny flushes. Defaults to `0`, which picks a size automatically: with
+    a write buffer manager, `writeBufferManagerSize / 2048`, but at least 64 KiB and at most
+    `writeBufferSize / 8` or 1 MiB, whichever is smaller (a 64 MiB budget gives 64 KiB, 512 MiB
+    gives 256 KiB, 2 GiB or more gives 1 MiB); without one, RocksDB's own default of
+    `writeBufferSize / 8` capped at 1 MiB (1 MiB at the default `writeBufferSize`). The automatic
+    size uses the manager's budget at the time the column family is opened. An explicit value is
+    used as given, except that RocksDB raises one below 4 KiB to 4 KiB. Like `writeBufferSize`,
+    this applies to every column family when a database is first opened in the process and to
+    column families created later; a column family that is already open keeps its size.
   - `compression: string | { algorithm: string, level?: number }` The block/blob compression
     algorithm for this column family. Pass an algorithm name — one of `'none'`, `'snappy'`,
     `'zlib'`, `'bzip2'`, `'lz4'`, `'lz4hc'`, or `'zstd'` — or an object with an
@@ -119,9 +132,10 @@ Creates a new database instance.
   - `readOnly: boolean` When `true`, the database is opened in read-only mode. Read operations are
     permitted. Write operations will throw an error with code `ERR_DATABASE_READONLY`. Transactions
     are a no-op in read-only mode. [`flush()`](#dbflushoptions-promisevoid),
-    [`flushSync()`](#dbflushsyncoptions-void), [`compact()`](#dbcompactoptions-promisevoid) and
-    [`compactSync()`](#dbcompactsyncoptions-void) are the exception: they have nothing to do when
-    there are no writes, so they succeed as no-ops rather than throwing.
+    [`flushSync()`](#dbflushsyncoptions-void), [`compact()`](#dbcompactoptions-promisevoid),
+    [`compactSync()`](#dbcompactsyncoptions-void) and [`syncWal()`](#dbsyncwal-promisevoid) are the
+    exception: they have nothing to do when there are no writes, so they succeed as no-ops rather
+    than throwing.
 
     A read-only open is a **point-in-time snapshot**: it never sees later writes. It is safe
     against a quiescent database, but **unsafe against a live writer**: the open holds no reference
@@ -276,6 +290,17 @@ Sets global database settings.
     callers could already produce it. With `1`, a database's async commits run one at a time in
     the order `commit()` was called. Databases already open keep their limit.
   - `compactOnClose: boolean` When `true`, compacts the database on close. Defaults to `false`.
+  - `transactionLogAbortMarkers: boolean` When `true`, a transaction that wrote its
+    transaction-log batch and then ends without committing (an `abort()`, a commit error it gives
+    up on, or a worker torn down with it pending) gets that batch marked aborted in place, and every
+    reader skips it; see [Abort markers](#abort-markers). Defaults to `false`, which behaves as
+    before. Read when a transaction starts and when it is reset for a retry.
+  - `transactionLogWriteIntents: boolean` When `true` with `transactionLogAbortMarkers` on, a
+    conflicting attempt is refused before it writes its batch when an older transaction holds a
+    write intent on one of its keys; see [Abort markers](#abort-markers). This shortens the worst
+    waits of transactions that overlap heavily, but adds retries and costs throughput when
+    transactions are large or contention is low. Defaults to `false`. Needs `verificationTable`.
+    Read with `transactionLogAbortMarkers`.
   - `occValidation: 'parallel' | 'serial'` How writable optimistic databases opened after the call
     validate commits for conflicts. Defaults to `'parallel'` (RocksDB's default). Databases already
     open keep their policy. Pessimistic, read-only and secondary opens are unaffected.
@@ -328,6 +353,19 @@ Sets global database settings.
     [`db.isWriteStalled()`](#dbiswritestalled-boolean) — see
     [`getWriteBufferManagerStats()`](#getwritebuffermanagerstats-writebuffermanagerstats),
     which also describes the watchdog that logs a sustained stall.
+  - `writeBufferManagerFlushPolicy: 'oldest' | 'largest' | 'largestAcrossDBs'` What the write
+    buffer manager flushes when it is over budget. `'oldest'` flushes the oldest memtable of the
+    database that is being written; `'largest'` its largest memtable (databases here are opened with
+    atomic flush, so this currently behaves like `'oldest'`); `'largestAcrossDBs'` has a background
+    thread flush whichever database holds the most memtable memory, whether or not it is being
+    written. When not set, the policy is `'largestAcrossDBs'` while
+    `writeBufferManagerAllowStall` is `true` and `'oldest'` otherwise, and it follows later changes
+    to `writeBufferManagerAllowStall`. With stalls allowed, `'oldest'` can stop writes permanently:
+    a write to one database waits for memory held by other databases that nothing is writing to,
+    so nothing flushes them. Without stalls, `'largestAcrossDBs'` has measured slower. Can be
+    changed while databases are open. Once set, the value stays in effect for the life of the
+    process; it is reported as `flushPolicy` by
+    [`getWriteBufferManagerStats()`](#getwritebuffermanagerstats-writebuffermanagerstats).
   - `writeBufferManagerCostToCache: boolean` When `true`, memtable memory is "charged" against the
     shared block cache so the block cache and write buffers draw from a single pool. During write
     bursts the cache shrinks to make room for memtables; once memtables flush, the cache can grow
@@ -370,6 +408,8 @@ where the call was made.
 - `mutableMemoryUsage: number` The share of `memoryUsage` held by active (mutable) memtables; the
   rest is memtables awaiting flush plus retained write history.
 - `allowStall: boolean`, `costToCache: boolean` The manager's configuration.
+- `flushPolicy: 'oldest' | 'largest' | 'largestAcrossDBs'` The flush policy in effect (see
+  `writeBufferManagerFlushPolicy`); before a manager exists, the policy one would be created with.
 - `stallActive: boolean` Whether the manager is currently stalling writes.
 - `stallActiveMs: number` How long the current stall has been active; `0` when not stalled.
   Sampled once a second by the watchdog, so `0` for a stall's first second and whenever
@@ -388,7 +428,7 @@ where the call was made.
 
 `bufferSize`, `memoryUsage`, `mutableMemoryUsage`, `stallActive` and `stallActiveMs` are also in
 [`db.getStats()`](#dbgetstatsall-boolean-rocksdbstats) and `db.getStat()` under the same
-`writeBufferManager.` prefix, for scraping; `enabled`, `allowStall`, `costToCache`,
+`writeBufferManager.` prefix, for scraping; `enabled`, `allowStall`, `costToCache`, `flushPolicy`,
 `watchdogRunning`, and the column-family inventory are only here, because collecting them either
 walks the database registry or isn't scrape-shaped.
 
@@ -806,6 +846,22 @@ db.flushSync({ allowWriteStall: true });
 On a [read-only](#new-rocksdatabasepath-options) database it validates its options and then returns
 without flushing, rather than throwing.
 
+### `db.syncWal(): Promise<void>`
+
+Syncs the write-ahead log to stable storage (`DB::SyncWAL`), so every WAL-backed write committed
+before the call survives a power loss afterwards. Unlike [`flush()`](#dbflushoptions-promisevoid) it
+writes no SST file and never stalls writers. Writes that skipped the WAL (`disableWAL`) and
+transaction-log files, which keep their own durability, are not covered. RocksDB serializes
+concurrent syncs of one WAL, so keep one in flight per database rather than queueing them on the
+libuv threadpool.
+
+```typescript
+await db.syncWal();
+```
+
+On a [read-only](#new-rocksdatabasepath-options) database the returned promise resolves without
+syncing.
+
 ### `db.get(key: Key, options?: GetOptions): MaybePromise<any>`
 
 Retreives the value for a given key. If the key does not exist, it will resolve `undefined`.
@@ -1250,7 +1306,9 @@ The `attempt` parameter is the number of times the transaction has been retried.
 - `disableSnapshot?: boolean` Whether to disable snapshots. Defaults to `false`.
 - `maxRetries?: number` The maximum number of times to retry the transaction. Defaults to `3`.
 - `retryOnBusy?: boolean` Whether to retry the transaction if the commit fails with `IsBusy`.
-  Defaults to `true` when the transaction is bound to a transaction log, otherwise `false`.
+  Defaults to `true` when the transaction wrote a transaction-log batch or was refused before
+  writing one (`refusedBeforeLog`), otherwise `false`. With `false`, `db.transaction()` also
+  commits without `coordinatedRetry`, so a conflict fails at once instead of waiting first.
 
 ### Transaction Retry Logic
 
@@ -1263,7 +1321,45 @@ a bad state and the transaction will need to be retried. If the max retries is r
 transaction is not retried, a `ERR_TRANSACTION_ABANDONED` error will be thrown.
 
 Users should use the `attempt` transaction callback parameter to ensure duplicate transaction log
-entries are not added.
+entries are not added. `txn.logWritten()` reports whether the batch is already in the log: a commit
+that conflicts after writing it keeps it for the retry, while an attempt refused before writing it
+(`err.refusedBeforeLog`, with `transactionLogAbortMarkers`) leaves nothing and must stage it again.
+
+### Abort markers
+
+With `RocksDatabase.config({ transactionLogAbortMarkers: true })`, a transaction's log batch is
+never left live when the transaction does not commit:
+
+- A transaction that wrote its batch and then ends without committing has the batch marked aborted:
+  bit `0x02` (`TRANSACTION_LOG_ENTRY_ABORTED_FLAG`) is set on its first entry, and the change is
+  synced to disk before `abort()` returns or the batch stops holding back committed readers. Every
+  reader skips the batch. `abort()` then no longer throws `ERR_TRANSACTION_ABANDONED`.
+- If the marker cannot be written or synced, `abort()` throws `ERR_TRANSACTION_ABORT_UNRESOLVED`
+  (again on a second call), committed readers stay behind the batch, and the log refuses new batches
+  until a retry on the next batch, database flush or close marks it.
+- Under `occValidation: 'parallel'` (the default), a retried optimistic commit that RocksDB's own
+  conflict check shows would fail is refused before it writes its batch too, with
+  `refusedBeforeLog` set (needs the RocksDB build this package pins). A first attempt skips the
+  check, so commits that do not conflict pay nothing for it, and so does a retry whose batch an
+  earlier attempt already wrote.
+- With `transactionLogWriteIntents: true` as well, a commit that finds an older transaction holding
+  a write intent on one of its keys (with `verificationTable: true`) is refused before writing its
+  batch, with `refusedBeforeLog` set on the
+  error, or, if its batch is already written from an earlier attempt, before committing. With
+  `coordinatedRetry` it waits for that transaction first. `commitSync()` never waits: it throws, and
+  `transactionSync()` retries after a short backoff.
+- `txn.setParkDeadline(epochMs)` bounds those waits by the caller's own deadline.
+- `log.getStats().totals` reports `abortsMarked`, `abortMarkFailures`, and `watermarkPinnedUs` /
+  `pinnedTransactions`: how long logged batches held committed readers back after their first
+  conflict, until they committed or ended.
+- A backup waits for every batch it captures that has neither committed nor been aborted, and is
+  refused if one is still unresolved after `ROCKSDB_JS_BACKUP_UNRESOLVED_WAIT_MS` (default
+  `30000`).
+
+A reader older than the release that added aborted-batch support shows an aborted batch as live, so
+turn this on only once every reader of the logs (replicas, restores, repair tools) has it. A batch
+whose transaction was still unresolved when the process crashed is not marked, and is replayed as
+before.
 
 When `coordinatedRetry: true`, the retry behavior changes for `IsBusy` conflicts: instead of
 rejecting (or retrying immediately and potentially conflicting again), the commit waits until the
@@ -1297,6 +1393,12 @@ operations methods as the `RocksDatabase` instance plus:
 - `txn.commitSync()` Synchronously commits and closes the transaction.
 - `txn.getTimestamp(): number` Retrieves the transaction timestamp in milliseconds as a decimal. It
   defaults to a process-wide monotonic value assigned when the transaction was created.
+- `txn.lastConflictKind(): number` Why the most recent attempt was refused or retried, one of
+  `ConflictKind`: `none` (0), `intentBeforeLog` (1), `intentAfterLog` (2), `atCommit` (3),
+  `checkBeforeLog` (4).
+- `txn.logWritten(): boolean` Whether the transaction's batch is in the transaction log.
+- `txn.setParkDeadline(epochMs: number): void` Past this time a coordinated-retry wait resolves at
+  once. `0` clears it.
 - `txn.id: number` The read-only transaction ID. Transaction IDs are unique to the RocksDB database
   path, regardless the database name/column family.
 - `txn.setTimestamp(ts?: number): void` Overrides the transaction timestamp in milliseconds. If
@@ -1937,6 +2039,10 @@ Attempts to acquire a lock for a given key. If the lock is available, the functi
 and the optional `onUnlocked` callback is never called. If the lock is not available, the function
 returns `false` and the `onUnlocked` callback is queued until the lock is released.
 
+A queued `onUnlocked` callback keeps the thread's event loop alive until it is called, even when the
+lock is held by another thread. A thread with a callback queued on a lock that is never released
+will not exit on its own; closing the database that queued the callback does not cancel it.
+
 When a database is closed, all locks associated to it will be unlocked.
 
 ```typescript
@@ -1977,7 +2083,9 @@ db.unlock('foo'); // false, already unlocked
 
 ### `db.withLock(key: Key, callback: () => void | Promise<void>): Promise<void>`
 
-Runs a function with guaranteed exclusive access across all threads.
+Runs a function with guaranteed exclusive access across all threads. The returned promise rejects
+with the error if the callback throws or its promise rejects; the lock is released either way. A
+pending `withLock()` keeps the thread's event loop alive until its callback has run.
 
 ```typescript
 await db.withLock('key', async () => {
@@ -2005,9 +2113,7 @@ await Promise.all([
 ]);
 ```
 
-Note: If the `callback` throws an error, Node.js suppress the error. Node.js 18.3.0 introduced a
-`--force-node-api-uncaught-exceptions-policy` flag which will cause errors to emit the
-`'uncaughtException'` event. Future Node.js releases will enable this flag by default.
+Note: Errors thrown by a `tryLock()` `onUnlocked` callback are discarded.
 
 ## File Locking
 
@@ -2199,20 +2305,21 @@ const names = db.listLogs();
 
 ### `db.purgeLogs({ includeEntryCounts: true, ...options }): { path: string; entries: number }[]`
 
-Deletes transaction log files older than the `transactionLogRetention` (defaults to 3 days).
-Ordinary retention keeps the sequence file named by `txn.state` and every newer file as the live
-store's retention floor, or the highest sequence file when there is no persisted flush position. It
-removes only an eligible contiguous prefix below that floor. An idle store can therefore retain one
-file past the cutoff until a later write rotates and flushes it; that extra file is bounded by the
-store's `transactionLogMaxSize` (except when a single transaction exceeds the target). That bound
-covers only the floor file: when the flush position lags — `txn.state` stuck at an old sequence
-because RocksDB flushing is behind — every file above the floor is retained too, which
-`transactionLogMaxSize` does not bound. `purge.retainedUnflushedFiles` reports that case. Use
-`destroy: true` only to remove the store itself.
+Deletes transaction log files older than the `transactionLogRetention` (defaults to 3 days). A file
+is deleted only when it lies entirely at or before the flushed position recorded in `txn.state`, so
+every entry in it is already in RocksDB; that includes the file `txn.state` names and the file
+currently being written, so an idle, fully flushed store is left with no log files. Retention removes
+a contiguous prefix: the first file that is too new or holds an unflushed entry stops the run. When
+the flush position lags — `txn.state` stuck at an old sequence because RocksDB flushing is behind —
+every file after it is retained regardless of age; `purge.retainedUnflushedFiles` reports that case.
+While a backup with `transactionLogs` is copying a store's files, purges of that store are skipped
+(`purge.retentionPins` reports it). On a filesystem that rejects a directory fsync, the store's last
+file is kept, since `txn.state` could not be made durable before deleting it.
+Use `destroy: true` only to remove the store itself.
 
 - `options: object`
   - `before?: number` Remove transaction log files older than the specified timestamp, subject to
-    the retention floor and contiguous-prefix rules above.
+    the flushed-position and contiguous-prefix rules above.
   - `destroy?: boolean` When `true`, deletes transaction log stores including all log sequence files
     on disk.
   - `includeEntryCounts?: boolean` When `true`, counts the entries in each deleted log file and
@@ -2341,6 +2448,22 @@ The iterator produces an object with the log entry timestamp and data.
   - `timestamp: number` The entry timestamp used to collate entries by transaction.
   - `endTxn: boolean` This is `true` when the entry is the last entry in a transaction.
 
+`next()` can throw two errors that leave the iterator usable, so calling `next()` again continues
+past the problem. Other errors, such as a failure to open a segment, do not move the iterator, so
+retrying them without a fix fails the same way.
+
+- `TransactionLogGapError`: the iterator reached segments it had not read yet and could not read
+  them, usually because retention purged them while the reader lagged. Their entries are not in
+  this log anymore, so a consumer that replicates the log should treat its copy as discontinuous
+  (for example, by resyncing from a full copy) rather than continuing. Properties: `logId` (the
+  last segment read), `resumeLogId` (the segment the next `next()` call reads), and
+  `unavailableLogCount`. A new query positioned within the retained segments does not report the
+  segments purged before its starting point.
+- `CorruptFrameError` (a `RangeError`): an entry's framing is broken. `resyncPosition` is where
+  valid framing resumes, or `undefined` when nothing follows the break.
+
+Both classes are exported from the package.
+
 ```typescript
 const log = db.useLog('foo');
 const iter = log.query({});
@@ -2386,9 +2509,8 @@ stats.totals.transactionsWritten; // lifetime count of transactions written
 
 The `purge.retainedUnflushedFiles` gauge is useful for diagnosing why logs are not being cleaned
 up: a file can be older than the retention period but still retained because its transactions have
-not yet been flushed to RocksDB (purging it would be unsafe for crash recovery). The retention
-floor — the sequence named by `txn.state`, or the highest sequence when there is no persisted flush
-position — is not counted as purgeable even when it is old and fully flushed.
+not yet been flushed to RocksDB (purging it would be unsafe for crash recovery). An old, fully
+flushed current file counts as purgeable.
 
 ### Transaction Log Initialization
 
@@ -2495,7 +2617,7 @@ Contract:
   `QueryPerformanceCounter` on Windows).
 - **Origin and lifetime**: the origin is unspecified and fixed for the life of the process. Every
   sample taken in the process is in one domain — the main thread and every `worker_threads` worker,
-  including workers started or restarted at any later time, under Node, Bun and Deno alike. Samples
+  including workers started or restarted at any later time, under Node and Bun alike. Samples
   are not meaningful across processes or restarts, and are unrelated to the Unix epoch: never compare
   them with `Date.now()`, `db.getMonotonicTimestamp()` or transaction timestamps.
 - **Monotonic, not unique**: a sample taken after another (in real time, on any thread) is `>=` it.
@@ -2789,7 +2911,7 @@ backup references them, so this is not equivalent to deleting files manually.
 
 Deletes all but the newest `keepCount` backups.
 
-### `backups.verify(backupDir: string, backupId: number, options?: { verifyWithChecksum?: boolean; verifyTransactionLogs?: boolean }): Promise<void>`
+### `backups.verify(backupDir: string, backupId: number, options?: { verifyWithChecksum?: boolean; verifyTables?: boolean; verifyTransactionLogs?: boolean }): Promise<void>`
 
 Verifies a backup's file sizes, and optionally their checksums (which requires reading all
 backed-up data). Resolves if the backup is intact and rejects otherwise.
@@ -2800,8 +2922,17 @@ When the backup was created with `transactionLogs: true`, the backup's transacti
 log file's header and entry framing must be intact (snapshots are copied on committed entry
 boundaries, so even a torn tail fails verification). Set `verifyTransactionLogs: false` to skip.
 
+Set `verifyTables: true` (default `false`) to also prove this build can read the backup's table
+files before you restore it. Each SST is opened in turn, one at a time, so file-descriptor use does
+not grow with the number of tables. Verification rejects, naming the file, when this build cannot
+decode a table's footer (for example a `format_version` written by a newer RocksDB) or its
+properties, or when the table records a compression algorithm this build does not support (see
+[`supportedCompression`](#supportedcompression)). Checksums cannot catch either case: the file is
+intact, just unreadable here, and RocksDB would otherwise fail on the first read after a restore.
+Blob files are not checked.
+
 ```typescript
-await backups.verify('/path/to/backups', 1, { verifyWithChecksum: true });
+await backups.verify('/path/to/backups', 1, { verifyWithChecksum: true, verifyTables: true });
 ```
 
 ## Custom Store