@harperfast/rocksdb-js 2.1.0 → 2.3.0

package/README.md

@@ -74,6 +74,11 @@ Creates a new database instance.
     before purging. Defaults to `'3d'` (3 days).
   - `transactionLogsPath: string` The path to store transaction logs. Defaults to
     `"${db.path}/transaction_logs"`.
+  - `verificationTable: boolean` When `true`, this column family participates in the process-global
+    [Verification Table](#verification-table): transaction writes to this column family invalidate
+    the verification slot for each written key. Enable this only for column families whose records
+    are cached (e.g. the primary column family of a table). Defaults to `false`. Requires
+    `verificationTableEntries` to be configured before the first database is opened.
 
 ### `db.close()`
 
@@ -106,6 +111,11 @@ Sets global database settings.
     Defaults to 32MB. Set to `0` (zero) disables block cache for future opened databases. Existing
     block cache for any opened databases is resized immediately. Negative values throw an error.
   - `compactOnClose: boolean` When `true`, compacts the database on close. Defaults to `false`.
+  - `verificationTableEntries: number` The number of slots in the process-global
+    [Verification Table](#verification-table). Each slot is 8 bytes, so the default of `131072`
+    (128K) slots is 1 MB. Set to `0` to disable the verification table. This must be configured
+    before the first database is opened; once the table is materialized, attempts to change this
+    value throw.
   - `writeBufferManagerAllowStall: boolean` When `true`, writes are stalled once the manager's
     `buffer_size` is exceeded, providing a hard cap on memtable memory. When `false`, memtables are
     allowed to grow past the limit and flushes are simply scheduled more aggressively. Off by
@@ -305,9 +315,28 @@ assert.equal(result, 'foo');
 
 Note that all errors are returned as rejected promises.
 
+See [`GetOptions`](#getoptions) for the available options.
+
+When the `expectedVersion` option is set and the [Verification Table](#verification-table) records
+a matching version for the key, `get()` returns the `FRESH_VERSION_FLAG` sentinel
+(`constants.FRESH_VERSION_FLAG`) instead of reading the value — signalling that any value the
+caller has already cached for this key is still fresh and no read was performed. Be sure to check
+for this sentinel before treating the result as a value:
+
+```typescript
+import { constants } from '@harperfast/rocksdb-js';
+
+const result = db.get(key, { expectedVersion: cachedEntry.version });
+if (result === constants.FRESH_VERSION_FLAG) {
+	// the cached value is still valid; no read occurred
+	return cachedEntry.value;
+}
+```
+
 ### `db.getSync(key: Key, options?: GetOptions): any`
 
-Synchronous version of `get()`.
+Synchronous version of `get()`. Like `get()`, this can return the `FRESH_VERSION_FLAG` sentinel when
+the `expectedVersion` option is used.
 
 ### `db.getEstimatedKeyCount(): number`
 
@@ -564,6 +593,12 @@ The `attempt` parameter is the number of times the transaction has been retried.
 
 ### `TransactionOptions`
 
+- `coordinatedRetry?: boolean` When `true`, an `IsBusy` conflict at commit time is retried
+  automatically instead of being rejected. Rather than retrying immediately and racing the
+  conflicting transaction again, the retry waits until the conflicting transaction has committed and
+  released its write intent, then re-runs the transaction body right away with no backoff delay.
+  Requires the column family to be opened with `verificationTable: true`. See
+  [Verification Table](#verification-table). Defaults to `false`.
 - `disableSnapshot?: boolean` Whether to disable snapshots. Defaults to `false`.
 - `maxRetries?: number` The maximum number of times to retry the transaction. Defaults to `3`.
 - `retryOnBusy?: boolean` Whether to retry the transaction if the commit fails with `IsBusy`.
@@ -582,6 +617,14 @@ transaction is not retried, a `ERR_TRANSACTION_ABANDONED` error will be thrown.
 Users should use the `attempt` transaction callback parameter to ensure duplicate transaction log
 entries are not added.
 
+When `coordinatedRetry: true`, the retry behavior changes for `IsBusy` conflicts: instead of
+rejecting (or retrying immediately and potentially conflicting again), the commit waits until the
+conflicting transaction has committed and released its write intent, then re-runs the transaction
+body immediately with no backoff. This is still bounded by `maxRetries` — if the transaction has
+not committed after the configured number of coordinated retries, the transaction is abandoned with
+an `ERR_TRANSACTION_ABANDONED` error. Coordinated retry requires the column family to be opened with
+`verificationTable: true`.
+
 ### Class: `Transaction`
 
 The transaction callback is passed in a `Transaction` instance which contains all of the same data
@@ -953,6 +996,74 @@ Note: If the `callback` throws an error, Node.js suppress the error. Node.js 18.
 `--force-node-api-uncaught-exceptions-policy` flag which will cause errors to emit the
 `'uncaughtException'` event. Future Node.js releases will enable this flag by default.
 
+## Verification Table
+
+The verification table is a process-global, fixed-size structure that lets an application cheaply
+check whether a value it has already cached is still fresh, without performing a full read. It is
+intended for read-heavy workloads where records carry a monotonically increasing numeric version.
+
+Each record's version is the numeric value stored in the first 8 bytes of its value (interpreted as
+a big-endian float64). The table maps `(database, column family, key)` to a single 8-byte slot that
+holds the last-known version for that key. Because slots are addressed by a hash, distinct keys may
+share a slot; a collision only ever causes a conservative miss (a real read), never a stale value to
+be treated as fresh.
+
+The freshness check works as follows:
+
+1. Pass `{ expectedVersion }` to `get()` / `getSync()`. If the slot currently records that version,
+   the read is skipped and the `FRESH_VERSION_FLAG` sentinel is returned.
+2. On a cold read, pass `{ populateVersion: true }` (or call `db.populateVersion()` afterward) to
+   seed the slot with the version extracted from the value, so subsequent freshness checks hit.
+
+Transaction writes to a column family opened with `verificationTable: true` invalidate the slot for
+each written key at write time, so a stale version can never survive a write. This is also what
+enables [`coordinatedRetry`](#transactionoptions): a conflicting transaction parks on the slot and
+retries once the write intent is released.
+
+The table is **process-global** and backed by a single shared structure, so versions populated on
+the main thread are visible to `worker_threads` workers and vice versa.
+
+### Enabling the verification table
+
+The table must be sized before the first database is opened, via the
+[`verificationTableEntries`](#dbconfigoptions) config option (default `131072` slots = 1 MB; set to
+`0` to disable). Then opt-in per column family with the `verificationTable: true` open option:
+
+```typescript
+import { RocksDatabase } from '@harperfast/rocksdb-js';
+
+RocksDatabase.config({ verificationTableEntries: 128 * 1024 });
+
+const db = RocksDatabase.open('path/to/db', { verificationTable: true });
+```
+
+Enable `verificationTable` only for column families whose records are cached (e.g. the primary
+column family of a table); enabling it adds per-write slot invalidation overhead.
+
+### `db.verifyVersion(key: Key, version: number): boolean`
+
+Returns `true` when the verification table currently records `version` for `key` (in this database
+and column family), indicating a cached value with that version is still fresh. Returns `false`
+otherwise — including when the table is disabled. This is a cheap, synchronous check that performs
+no database read.
+
+```typescript
+if (db.verifyVersion(key, cachedEntry.version)) {
+	return cachedEntry.value;
+}
+const value = db.getSync(key);
+db.populateVersion(key, extractVersion(value));
+```
+
+### `db.populateVersion(key: Key, version: number): void`
+
+Seeds the verification-table slot for `key` with `version`. This is typically called after a full
+read where the caller already knows the version. It has no effect if the slot is currently
+lock-tagged (a transaction is mid-write on that key) or if the verification table is disabled.
+
+Passing `{ populateVersion: true }` to `get()` / `getSync()` performs the equivalent seeding
+automatically after a cold read, avoiding a separate call.
+
 ## Transaction Log
 
 A user controlled API for logging transactions. This API is designed to be generic so that you can
@@ -1011,19 +1122,34 @@ const names = db.listLogs();
 
 ### `db.purgeLogs(options?): string[]`
 
+### `db.purgeLogs({ includeEntryCounts: true, ...options }): { path: string; entries: number }[]`
+
 Deletes transaction log files older than the `transactionLogRetention` (defaults to 3 days).
 
 - `options: object`
   - `before?: number` Remove all transaction log files older than the specified timestamp.
   - `destroy?: boolean` When `true`, deletes transaction log stores including all log sequence files
     on disk.
+  - `includeEntryCounts?: boolean` When `true`, counts the entries in each deleted log file and
+    returns an array of `{ path, entries }` objects instead of an array of file paths. Counting reads
+    each file before it is removed, so it is only performed when this option is enabled.
   - `name?: string` The name of a store to limit the purging to.
 
-Returns an array with the full path of each log file deleted.
+The method is overloaded so the return type follows `includeEntryCounts`: by default (omitted or
+`false`) it returns `string[]` — the full path of each log file deleted; when `includeEntryCounts` is
+`true` it returns `{ path: string; entries: number }[]`, each entry being the `path` of the deleted
+log file and the number of `entries` it held. Because of the overloads, the object-array form is
+returned directly when `includeEntryCounts: true` is passed as a literal, with no casting required.
 
 ```typescript
 const removed = db.purgeLogs();
 console.log(`Removed ${removed.length} log files`);
+
+// Include the entry count for each deleted log file:
+const purged = db.purgeLogs({ includeEntryCounts: true });
+for (const { path, entries } of purged) {
+	console.log(`Removed ${path} (${entries} entries)`);
+}
 ```
 
 ### `db.useLog(name): TransactionLog`
@@ -1259,6 +1385,137 @@ import { versions } from '@harperfast/rocksdb-js';
 console.log(versions); // { "rocksdb": "10.10.1", "rocksdb-js": "0.1.2" }
 ```
 
+## Checkpoints
+
+### `db.createCheckpoint(targetPath: string): Promise<void>`
+
+Creates a [checkpoint](https://github.com/facebook/rocksdb/wiki/Checkpoints) — a point-in-time,
+fully independent copy of the entire database (all column families) at `targetPath` — and resolves
+once written. Unlike a backup, a checkpoint is a normal, writable database: open it as a new
+`RocksDatabase` and it diverges independently from the source.
+
+SST and blob files are **hard-linked** when `targetPath` is on the **same filesystem** as the
+database, and **copied** otherwise; other files (such as the `MANIFEST`) are always copied. As a
+result the operation is near-instant on the same filesystem and as costly as a full copy across
+filesystems. The memtable is flushed so the checkpoint includes the latest writes even when the
+WAL is disabled.
+
+Parent directories are created as needed. `targetPath` itself must not already exist — RocksDB
+creates the checkpoint directory — and the call rejects with `Create checkpoint failed: target
+path exists` if it does (other failures, such as a full disk, surface the RocksDB status message).
+The caller is responsible for opening the checkpoint and for eventually deleting the directory.
+
+```typescript
+const db = RocksDatabase.open('/path/to/database');
+await db.createCheckpoint('/path/to/checkpoint');
+
+// The checkpoint is a normal, writable database.
+const branch = RocksDatabase.open('/path/to/checkpoint');
+```
+
+## Backups
+
+Backups use RocksDB's `BackupEngine` to capture consistent, incremental, checksum-verified
+snapshots of a database. A backup covers the **entire database** — every column family, the
+manifest, and (by default) the write-ahead log — so it is not scoped to an individual `Store`.
+
+Creating a backup is an instance method (`db.backup()`) because it needs a live database. The
+remaining operations act on a backup directory and do not require an open database, so they are
+grouped under the `backups` namespace export.
+
+```typescript
+import { RocksDatabase, backups } from '@harperfast/rocksdb-js';
+
+const db = RocksDatabase.open('/path/to/database');
+await db.put('foo', 'bar');
+
+// Create a backup, then restore it into a fresh directory.
+const id = await db.backup('/path/to/backups');
+await backups.restore('/path/to/backups', '/path/to/restored-db');
+```
+
+### `db.backup(backupDir: string, options?: BackupOptions): Promise<number>`
+
+Creates a new backup of the entire database into `backupDir`, creating parent directories as
+needed, and resolves with the new backup id (a monotonically increasing integer). Subsequent
+backups into the same directory are incremental — unchanged immutable files are shared rather than
+re-copied.
+
+When the database was opened with `disableWAL`, the memtable is flushed before the backup by
+default so unflushed data is not lost; otherwise flushing follows `options.flushBeforeBackup`.
+
+```typescript
+const id = await db.backup('/path/to/backups', { metadata: 'nightly-2026-06-04' });
+```
+
+`BackupOptions`:
+
+| Option                    | Type      | Default                | Description                                                           |
+| ------------------------- | --------- | ---------------------- | --------------------------------------------------------------------- |
+| `flushBeforeBackup`       | `boolean` | `true` if WAL disabled | Flush the memtable before backing up.                                 |
+| `metadata`                | `string`  | `''`                   | Application metadata stored with the backup, returned by `list()`.    |
+| `shareTableFiles`         | `boolean` | `true`                 | Share files between backups to enable incremental backups.            |
+| `shareFilesWithChecksum`  | `boolean` | `true`                 | Distinguish shared files by checksum to avoid cross-database clashes. |
+| `backupLogFiles`          | `boolean` | `true`                 | Include write-ahead log files in the backup.                          |
+| `sync`                    | `boolean` | `true`                 | `fsync` backup files for crash consistency.                           |
+| `maxBackgroundOperations` | `number`  | `1`                    | Number of background threads used to copy files.                      |
+
+### `backups.restore(backupDir: string, dbDir: string, options?: RestoreOptions): Promise<void>`
+
+Restores a backup from `backupDir` into `dbDir` (creating parent directories as needed). The
+database must **not** be open at `dbDir`, and the default restore mode is **destructive** — it
+purges `dbDir` before restoring. Restoring into the backup directory itself is rejected.
+
+```typescript
+// Restore the latest backup.
+await backups.restore('/path/to/backups', '/path/to/restored-db');
+
+// Restore a specific backup id without purging matching existing files.
+await backups.restore('/path/to/backups', '/path/to/restored-db', {
+	backupId: 1,
+	mode: 'keepLatestDbSessionIdFiles',
+});
+```
+
+`RestoreOptions`:
+
+| Option         | Type                                                                  | Default           | Description                                                  |
+| -------------- | --------------------------------------------------------------------- | ----------------- | ------------------------------------------------------------ |
+| `backupId`     | `number`                                                              | latest backup     | The backup id to restore.                                    |
+| `walDir`       | `string`                                                              | `dbDir`           | Directory to restore write-ahead log files into.             |
+| `keepLogFiles` | `boolean`                                                             | `false`           | Keep existing log files in `walDir` rather than overwriting. |
+| `mode`         | `'purgeAllFiles' \| 'keepLatestDbSessionIdFiles' \| 'verifyChecksum'` | `'purgeAllFiles'` | The restore strategy (default purges the destination).       |
+
+### `backups.list(backupDir: string): Promise<BackupInfo[]>`
+
+Lists the non-corrupt backups in `backupDir`, ordered by id.
+
+```typescript
+const list = await backups.list('/path/to/backups');
+// [{ backupId: 1, timestamp: 1749000000, size: 4096, numberFiles: 3, appMetadata: '' }, ...]
+```
+
+Each `BackupInfo` contains `backupId`, `timestamp` (seconds since the epoch), `size` (bytes),
+`numberFiles`, and `appMetadata`.
+
+### `backups.delete(backupDir: string, backupId: number): Promise<void>`
+
+Deletes a specific backup. Shared files are reference-counted and only removed once no remaining
+backup references them, so this is not equivalent to deleting files manually.
+
+### `backups.purge(backupDir: string, keepCount: number): Promise<void>`
+
+Deletes all but the newest `keepCount` backups.
+
+### `backups.verify(backupDir: string, backupId: number, options?: { verifyWithChecksum?: boolean }): Promise<void>`
+
+Verifies a backup's file sizes, and optionally their checksums (which requires reading all
+backed-up data). Resolves if the backup is intact and rejects otherwise.
+
+```typescript
+await backups.verify('/path/to/backups', 1, { verifyWithChecksum: true });
+```
+
 ## Custom Store
 
 The store is a class that sits between the `RocksDatabase` or `Transaction` instance and the native
@@ -1275,7 +1532,7 @@ The default `Store` contains the following methods which can be overridden:
 - `decodeValue(value)`
 - `encodeKey(key)`
 - `encodeValue(value)`
-- `get(context, key, resolve, reject, txnId?)`
+- `get(context, key, alwaysCreateNewBuffer?, options?)`
 - `getCount(context, options?, txnId?)`
 - `getKeys(context, options?)`
 - `getKeysCount(context, options?)`
@@ -1286,11 +1543,13 @@ The default `Store` contains the following methods which can be overridden:
 - `isOpen()`
 - `listLogs()`
 - `open()`
+- `populateVersion(key, version)`
 - `putSync(context, key, value, options?)`
 - `removeSync(context, key, options?)`
 - `tryLock(key, onUnlocked?)`
 - `unlock(key)`
 - `useLog(context, name)`
+- `verifyVersion(key, version)`
 - `withLock(key, callback?)`
 
 To use it, extend the default `Store` and pass in an instance of your store into the `RocksDatabase`
@@ -1300,15 +1559,15 @@ constructor.
 import { RocksDatabase, Store } from '@harperfast/rocksdb-js';
 
 class MyStore extends Store {
-  get(context, key, resolve, reject, txnId) {
-    console.log('Getting:' key);
-    return super.get(context, key, resolve, reject, txnId);
-  }
-
-  putSync(context, key, value, options) {
-    console.log('Putting:', key);
-    return super.putSync(context, key, value, options);
-  }
+	get(context, key, alwaysCreateNewBuffer, options) {
+		console.log('Getting:', key);
+		return super.get(context, key, alwaysCreateNewBuffer, options);
+	}
+
+	putSync(context, key, value, options) {
+		console.log('Putting:', key);
+		return super.putSync(context, key, value, options);
+	}
 }
 
 const myStore = new MyStore('path/to/db');
@@ -1328,6 +1587,22 @@ console.log(await db.get('foo'));
 
 ## Interfaces
 
+### `GetOptions`
+
+Options for `get()`, `getSync()`, and the `getBinary*` methods.
+
+- `options: object`
+  - `expectedVersion: number` When set, the [Verification Table](#verification-table) is checked
+    before reading. If the slot holds this version, the read is skipped and the
+    `FRESH_VERSION_FLAG` sentinel is returned. After a database read, the slot is seeded with the
+    version extracted from the value. Requires the column family to be opened with
+    `verificationTable: true`.
+  - `populateVersion: boolean` When `true`, after a database read the verification-table slot is
+    seeded with the version extracted from the value, eliminating the need for a separate
+    `db.populateVersion()` call on cold reads. Defaults to `false`.
+  - `skipDecode: boolean` When `true`, the value is returned without being decoded. Defaults to
+    `false`.
+
 ### `RocksDBOptions`
 
 - `options: object`
@@ -1393,6 +1668,11 @@ Options:
 
 Available commands:
 
+- `backups <dir> [subcommand]` Manage database backups; with no subcommand or `ls`/`list`, lists the backups in
+  `<dir>`. Subcommands: `backup` (create a backup of the open database), `restore <backup-id>`
+  (restore into the open database after confirmation; unavailable in read-only mode),
+  `verify <backup-id>` (checksum verification), `delete <backup-id>`, and `purge <keep-count>`
+  (delete all but the newest backups). `delete` and `purge` report the recovered disk space.
 - `clear` Clear all data in the current column family
 - `columns` List column families
 - `compact` Compact the current column family