@harperfast/rocksdb-js 2.8.0 → 2.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +337 -19
- package/dist/index.cjs +271 -36
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +317 -34
- package/dist/index.d.mts +317 -34
- package/dist/index.mjs +270 -36
- package/dist/index.mjs.map +1 -1
- package/docs/stats.md +46 -1
- package/docs/transaction-log.md +18 -5
- package/docs/transaction-timestamp-integrity-design.md +123 -0
- package/package.json +19 -19
package/README.md
CHANGED
|
@@ -112,6 +112,40 @@ Creates a new database instance.
|
|
|
112
112
|
[`flushSync()`](#dbflushsyncoptions-void), [`compact()`](#dbcompactoptions-promisevoid) and
|
|
113
113
|
[`compactSync()`](#dbcompactsyncoptions-void) are the exception: they have nothing to do when
|
|
114
114
|
there are no writes, so they succeed as no-ops rather than throwing.
|
|
115
|
+
|
|
116
|
+
A read-only open is a **point-in-time snapshot**: it never sees later writes. It is safe
|
|
117
|
+
against a quiescent database, but **unsafe against a live writer**: the open holds no reference
|
|
118
|
+
on the files it is about to read, so a concurrent compaction, blob GC, or flush in the writing
|
|
119
|
+
process can delete one mid-open and the open throws with code `ERR_CONCURRENT_COMPACTION` — the
|
|
120
|
+
database is not corrupt; the reader lost a race, and retrying can succeed. Even a successful
|
|
121
|
+
open can fail _later reads_ the same way, because files the snapshot references are opened
|
|
122
|
+
lazily while the writer keeps deleting obsolete ones. To follow a database another process is
|
|
123
|
+
actively writing, open a secondary instead (`secondaryPath` below) — that is the supported mode
|
|
124
|
+
for a live follower.
|
|
125
|
+
|
|
126
|
+
- `secondaryPath: string` Opens the database as a **secondary instance**: a read-only follower of
|
|
127
|
+
a live primary that tolerates the primary deleting files and sees new writes on each
|
|
128
|
+
[`catchUpWithPrimary()`](#dbcatchupwithprimary-promisevoid) call. This is the supported way to
|
|
129
|
+
read a database another process is actively writing. The value is the secondary instance's own
|
|
130
|
+
workspace directory (created if missing) where RocksDB keeps the secondary's private state — it
|
|
131
|
+
must be outside `path` (enforced) and exclusive to one secondary instance. Exclusivity is
|
|
132
|
+
enforced: reusing a workspace in-process for a different database is rejected at open (the same
|
|
133
|
+
database + workspace share the one follower instance), and a kernel advisory lock on
|
|
134
|
+
`<secondaryPath>/.secondary.lock` (same discipline and caveats as the
|
|
135
|
+
[backup directory lock](#backups)) excludes other processes — with the same limits: on
|
|
136
|
+
filesystems without advisory locking (`flock` unsupported — e.g. the FUSE/9p mounts behind
|
|
137
|
+
Docker Desktop bind mounts) the lock degrades to a no-op, and on network filesystems with
|
|
138
|
+
node-local `flock` (NFS `local_lock`, CIFS, 9p) it does not exclude across hosts, so workspace
|
|
139
|
+
exclusivity is the caller's job there. Implies
|
|
140
|
+
`readOnly: true` (an explicit `readOnly: false` throws), so all read-only behavior above
|
|
141
|
+
applies. Forces `maxOpenFiles: -1`: every table and blob file is opened and held for the life
|
|
142
|
+
of each version, which is what makes the primary's deletions safe — budget file descriptors
|
|
143
|
+
accordingly on large databases (an explicit `maxOpenFiles` other than `-1` is rejected). Column
|
|
144
|
+
families created by the primary after the secondary opens are invisible until the secondary
|
|
145
|
+
reopens. Note: RocksDB upstream documents secondary instances as unsupported in combination
|
|
146
|
+
with integrated BlobDB (which this library enables for values ≥ 2KB); the pinned RocksDB build
|
|
147
|
+
handles blob files in secondary mode — covered by native regression tests — but upstream does
|
|
148
|
+
not guarantee the combination.
|
|
115
149
|
- `statsLevel: StatsLevel` Controls which type of statistics to skip and reduce statistic
|
|
116
150
|
overhead. Defaults to `StatsLevel.ExceptDetailedTimers`.
|
|
117
151
|
- `store: Store` A custom store that handles all interaction between the `RocksDatabase` or
|
|
@@ -156,7 +190,7 @@ Returns the list of column families in the RocksDB database.
|
|
|
156
190
|
const db = RocksDatabase.open('path/to/db');
|
|
157
191
|
console.log(db.columns); // ['default']
|
|
158
192
|
|
|
159
|
-
|
|
193
|
+
db.use('users');
|
|
160
194
|
console.log(db.columns); // ['default', 'users']
|
|
161
195
|
```
|
|
162
196
|
|
|
@@ -191,7 +225,11 @@ Sets global database settings.
|
|
|
191
225
|
- `writeBufferManagerAllowStall: boolean` When `true`, writes are stalled once the manager's
|
|
192
226
|
`buffer_size` is exceeded, providing a hard cap on memtable memory. When `false`, memtables are
|
|
193
227
|
allowed to grow past the limit and flushes are simply scheduled more aggressively. Off by
|
|
194
|
-
default to favor write throughput over hard memory bounding. Defaults to `false`.
|
|
228
|
+
default to favor write throughput over hard memory bounding. Defaults to `false`. A stall here
|
|
229
|
+
is invisible to RocksDB's own stall counters and to
|
|
230
|
+
[`db.isWriteStalled()`](#dbiswritestalled-boolean) — see
|
|
231
|
+
[`getWriteBufferManagerStats()`](#getwritebuffermanagerstats-writebuffermanagerstats),
|
|
232
|
+
which also describes the watchdog that logs a sustained stall.
|
|
195
233
|
- `writeBufferManagerCostToCache: boolean` When `true`, memtable memory is "charged" against the
|
|
196
234
|
shared block cache so the block cache and write buffers draw from a single pool. During write
|
|
197
235
|
bursts the cache shrinks to make room for memtables; once memtables flush, the cache can grow
|
|
@@ -211,6 +249,80 @@ RocksDatabase.config({
|
|
|
211
249
|
});
|
|
212
250
|
```
|
|
213
251
|
|
|
252
|
+
### `getWriteBufferManagerStats(): WriteBufferManagerStats`
|
|
253
|
+
|
|
254
|
+
Reads the live state of the `WriteBufferManager`.
|
|
255
|
+
|
|
256
|
+
**Process-wide, not per-database.** The manager is a singleton shared by every database opened in
|
|
257
|
+
this process, `worker_threads` included, so these values describe the whole process regardless of
|
|
258
|
+
where the call was made.
|
|
259
|
+
|
|
260
|
+
- `enabled: boolean` Whether a manager has been created. `bufferSize`, `memoryUsage`,
|
|
261
|
+
`mutableMemoryUsage`, `stallActive`, `stallActiveMs`, `watchdogRunning` and `columnFamilies` are
|
|
262
|
+
`0`/`false` when not; `allowStall` and `costToCache` still reflect the configured setting (a
|
|
263
|
+
manager is only created once `writeBufferManagerSize` is also set), and `inventoryAvailable` is
|
|
264
|
+
`true` (there is nothing to fail to collect).
|
|
265
|
+
- `bufferSize: number` The budget in bytes (`writeBufferManagerSize`, read live).
|
|
266
|
+
- `memoryUsage: number` Total memtable memory in bytes charged against the manager.
|
|
267
|
+
- `mutableMemoryUsage: number` The share of `memoryUsage` held by active (mutable) memtables; the
|
|
268
|
+
rest is memtables awaiting flush plus retained write history.
|
|
269
|
+
- `allowStall: boolean`, `costToCache: boolean` The manager's configuration.
|
|
270
|
+
- `stallActive: boolean` Whether the manager is currently stalling writes.
|
|
271
|
+
- `stallActiveMs: number` How long the current stall has been active; `0` when not stalled.
|
|
272
|
+
Sampled once a second by the watchdog, so `0` for a stall's first second and whenever
|
|
273
|
+
`watchdogRunning` is `false`.
|
|
274
|
+
- `watchdogRunning: boolean` Whether the stall watchdog thread is running.
|
|
275
|
+
- `columnFamilies: number` Live column families across every database attached to this manager. A
|
|
276
|
+
dropped column family keeps charging the manager until its last handle closes, so it is counted
|
|
277
|
+
until then.
|
|
278
|
+
- `inventoryAvailable: boolean` `false` when the inventory could not be collected because the
|
|
279
|
+
database registry or a column-family inventory was locked. The two inventory fields are then
|
|
280
|
+
empty and everything else is still live; the call never blocks on database work that may itself
|
|
281
|
+
be stalled.
|
|
282
|
+
- `maxWriteBufferSizeToMaintain: Record<string, number>` Effective per-column-family retained-history
|
|
283
|
+
target (as a decimal string) to how many of those column families carry it. Effective, not
|
|
284
|
+
requested: RocksDB rewrites a requested `0` for a transaction database.
|
|
285
|
+
|
|
286
|
+
`bufferSize`, `memoryUsage`, `mutableMemoryUsage`, `stallActive` and `stallActiveMs` are also in
|
|
287
|
+
[`db.getStats()`](#dbgetstatsall-boolean-rocksdbstats) and `db.getStat()` under the same
|
|
288
|
+
`writeBufferManager.` prefix, for scraping; `enabled`, `allowStall`, `costToCache`,
|
|
289
|
+
`watchdogRunning`, and the column-family inventory are only here, because collecting them either
|
|
290
|
+
walks the database registry or isn't scrape-shaped.
|
|
291
|
+
|
|
292
|
+
`stallActive` is the signal that distinguishes a stalled process from an idle one.
|
|
293
|
+
[`db.isWriteStalled()`](#dbiswritestalled-boolean), the [`'writeStall'` event](#event-writestall)
|
|
294
|
+
and `rocksdb.stall.micros` all track RocksDB's `WriteController`, which a `WriteBufferManager`
|
|
295
|
+
stall never goes through.
|
|
296
|
+
|
|
297
|
+
#### The stall watchdog
|
|
298
|
+
|
|
299
|
+
While a manager exists with `writeBufferManagerAllowStall`, one low-frequency thread samples the
|
|
300
|
+
manager once a second. When a stall has been continuously active past
|
|
301
|
+
`ROCKSDB_JS_WBM_STALL_WARN_MS` (default `5000`; `0` disables the watchdog) it writes **one** line
|
|
302
|
+
to `stderr` and emits it as a process-wide `'log.warn'` event — once per stall episode, not per
|
|
303
|
+
blocked writer and not per sample:
|
|
304
|
+
|
|
305
|
+
```
|
|
306
|
+
[rocksdb-js] WriteBufferManager write stall active for 5.0s - no write can complete until memtable
|
|
307
|
+
memory drops below the budget. budget=661.2MB usage=662.2MB (100.2%) mutable=12.4MB (1.9%)
|
|
308
|
+
allowStall=true costToCache=true columnFamilies=28 maxWriteBufferSizeToMaintain={268435456:28}
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
It needs its own thread because every other candidate is blocked by the very condition it reports:
|
|
312
|
+
a writer is parked inside RocksDB, RocksDB's stall callbacks are `WriteController`-only, and a JS
|
|
313
|
+
timer cannot fire on a thread parked in a synchronous write. The line goes to `stderr` as well as
|
|
314
|
+
the event so it is not lost when nothing has registered a `'log.warn'` listener; route one and
|
|
315
|
+
ignore `stderr` if you would rather have it in your own log.
|
|
316
|
+
|
|
317
|
+
```typescript
|
|
318
|
+
RocksDatabase.on('log.warn', (message) => logger.warn(message));
|
|
319
|
+
|
|
320
|
+
const wbm = getWriteBufferManagerStats();
|
|
321
|
+
if (wbm.stallActive) {
|
|
322
|
+
logger.warn(`writes stalled for ${wbm.stallActiveMs}ms: ${wbm.memoryUsage}/${wbm.bufferSize}`);
|
|
323
|
+
}
|
|
324
|
+
```
|
|
325
|
+
|
|
214
326
|
### `db.isOpen(): boolean`
|
|
215
327
|
|
|
216
328
|
Returns `true` if the database is open, otherwise false.
|
|
@@ -260,6 +372,20 @@ There's also a static `open()` method for convenience that performs the same thi
|
|
|
260
372
|
const db = RocksDatabase.open('path/to/db');
|
|
261
373
|
```
|
|
262
374
|
|
|
375
|
+
### `db.secondaryPath: string | undefined`
|
|
376
|
+
|
|
377
|
+
The secondary instance's workspace directory when the database was opened as a
|
|
378
|
+
[secondary](#new-rocksdatabasepath-options) (a read-only follower of a live primary), or
|
|
379
|
+
`undefined` for a regular or plain read-only open.
|
|
380
|
+
|
|
381
|
+
```typescript
|
|
382
|
+
const follower = RocksDatabase.open('/path/to/database', {
|
|
383
|
+
secondaryPath: '/path/to/follower-workspace',
|
|
384
|
+
});
|
|
385
|
+
console.log(follower.secondaryPath); // '/path/to/follower-workspace'
|
|
386
|
+
console.log(follower.readOnly); // true
|
|
387
|
+
```
|
|
388
|
+
|
|
263
389
|
### `db.status: 'opened' | 'closed'`
|
|
264
390
|
|
|
265
391
|
Returns a string `'opened'` or `'closed'` indicating if the database is opened or closed.
|
|
@@ -268,8 +394,95 @@ Returns a string `'opened'` or `'closed'` indicating if the database is opened o
|
|
|
268
394
|
console.log(db.status);
|
|
269
395
|
```
|
|
270
396
|
|
|
397
|
+
### `db.use(name, options?): RocksDatabase`
|
|
398
|
+
|
|
399
|
+
Returns a `RocksDatabase` bound to the `name` column family of this same database, opening — and
|
|
400
|
+
creating, if it does not exist — the column family on first use. This is a factory (like
|
|
401
|
+
[`useLog`](#dbuselogname-transactionlog)), not a stateful switch: the returned view is an
|
|
402
|
+
independent instance whose own reads and writes target its column family, while sharing the same
|
|
403
|
+
underlying database. Because a single RocksDB database backs every column family, a transaction,
|
|
404
|
+
backup, or checkpoint still spans all of them.
|
|
405
|
+
|
|
406
|
+
- `name: string` The column family name.
|
|
407
|
+
- `options?: object` Options for the column family (same shape as the constructor's, minus `name`),
|
|
408
|
+
overriding the options inherited from this database. Options only take effect when the view is
|
|
409
|
+
(re)opened.
|
|
410
|
+
|
|
411
|
+
`use()` is get-or-create, backed by a weak cache: `db.use('events') === db.use('events')` while the
|
|
412
|
+
view is still referenced **and open**, and calling with this database's own column-family name
|
|
413
|
+
returns `this`. A view that has been closed (or garbage-collected — the cache does not pin it) is
|
|
414
|
+
transparently recreated on the next `use()`. Views are independent handles: closing this database
|
|
415
|
+
does not close them (and vice versa); the underlying database stays open until every handle is closed
|
|
416
|
+
or collected.
|
|
417
|
+
|
|
418
|
+
The view's store is derived by `Store#createColumnFamilyStore(name, options)`, which builds an
|
|
419
|
+
independent store of the same class with **its own codec state** — views never share a mutable
|
|
420
|
+
encoder/decoder. Consequently a database configured with a _pre-constructed_ encoder/decoder instance
|
|
421
|
+
cannot derive views (its `name`/`structures` would be shared and corrupt both column families); use an
|
|
422
|
+
encoder factory (`{ Encoder }`) or a named `encoding` instead. A custom `Store` whose constructor
|
|
423
|
+
can't be recreated from `(path, options)` (e.g. it takes injected dependencies) should override
|
|
424
|
+
`createColumnFamilyStore` to build its views.
|
|
425
|
+
|
|
426
|
+
```typescript
|
|
427
|
+
const db = RocksDatabase.open('path/to/db');
|
|
428
|
+
|
|
429
|
+
const events = db.use('events');
|
|
430
|
+
await events.put('e1', payload);
|
|
431
|
+
|
|
432
|
+
await db.put('k', 'v'); // default column family, unaffected
|
|
433
|
+
console.log(events.get('e1')); // payload
|
|
434
|
+
console.log(db.get('e1')); // undefined — different column family
|
|
435
|
+
|
|
436
|
+
events.close();
|
|
437
|
+
db.close();
|
|
438
|
+
```
|
|
439
|
+
|
|
271
440
|
## Data Operations
|
|
272
441
|
|
|
442
|
+
### `db.catchUpWithPrimary(): Promise<void>`
|
|
443
|
+
|
|
444
|
+
Advances a [secondary instance](#new-rocksdatabasepath-options) to the primary's current state by
|
|
445
|
+
tailing and replaying the primary's MANIFEST and WAL. A secondary does not see the primary's
|
|
446
|
+
writes — flushed or not — until it catches up. Reads through the handle remain safe while the
|
|
447
|
+
catch-up runs; catch-ups on the same database are serialized internally.
|
|
448
|
+
|
|
449
|
+
Throws with code `ERR_NOT_SECONDARY` on a database that was not opened with `secondaryPath`.
|
|
450
|
+
Column families the primary created after the secondary opened stay invisible until the secondary
|
|
451
|
+
reopens; ones the primary dropped remain readable until then. Catch-up advances the **database**
|
|
452
|
+
view only: transaction-log reads through a secondary serve the stores discovered at open, and a
|
|
453
|
+
store the primary creates afterward becomes visible only on reopen. Whether entries appended to an
|
|
454
|
+
already-open store are visible depends on where the writer is: a cross-process primary's appends
|
|
455
|
+
are not (the reader's view of the file extent is fixed at open), while a writer in the _same_
|
|
456
|
+
process shares the store object, so its appends are. Either way a log entry can describe data the
|
|
457
|
+
database view does not have yet — the log write completes before the RocksDB commit for every
|
|
458
|
+
writer, so log-leads-database is the normal direction and a consumer has to tolerate it. Serialize
|
|
459
|
+
catch-up calls per database. Each async call holds a libuv worker for the whole replay, and
|
|
460
|
+
concurrent calls queue on an internal per-database mutex while holding theirs, so a handful of
|
|
461
|
+
overlapping catch-ups on a backlogged follower can exhaust the default four-thread pool and stall
|
|
462
|
+
unrelated `fs`/`dns`/`crypto` work in the process. Await one before starting the next rather than
|
|
463
|
+
firing one per timer tick.
|
|
464
|
+
|
|
465
|
+
```typescript
|
|
466
|
+
const primary = RocksDatabase.open('/path/to/database');
|
|
467
|
+
const follower = RocksDatabase.open('/path/to/database', {
|
|
468
|
+
secondaryPath: '/path/to/follower-workspace',
|
|
469
|
+
});
|
|
470
|
+
|
|
471
|
+
primary.putSync('foo', 'bar');
|
|
472
|
+
console.log(follower.getSync('foo')); // undefined — not caught up yet
|
|
473
|
+
await follower.catchUpWithPrimary();
|
|
474
|
+
console.log(follower.getSync('foo')); // 'bar'
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
### `db.catchUpWithPrimarySync(): void`
|
|
478
|
+
|
|
479
|
+
Synchronous version of `catchUpWithPrimary()`. The replay runs on the JS thread, so prefer the
|
|
480
|
+
async form unless the caller is already blocking.
|
|
481
|
+
|
|
482
|
+
```typescript
|
|
483
|
+
follower.catchUpWithPrimarySync();
|
|
484
|
+
```
|
|
485
|
+
|
|
273
486
|
### `db.clear(options?): Promise<number>`
|
|
274
487
|
|
|
275
488
|
Asychronously removes all data in the current database.
|
|
@@ -571,7 +784,7 @@ const range = db.getKeysCount({ start: 'a', end: 'z' }); // exact number of keys
|
|
|
571
784
|
### `db.getMonotonicTimestamp(): number`
|
|
572
785
|
|
|
573
786
|
Returns the current timestamp as a monotonically increasing timestamp in milliseconds represented as
|
|
574
|
-
a decimal number.
|
|
787
|
+
a decimal number. This process-wide clock also supplies each transaction's initial timestamp.
|
|
575
788
|
|
|
576
789
|
```typescript
|
|
577
790
|
const ts = db.getMonotonicTimestamp();
|
|
@@ -631,6 +844,25 @@ const blobFiles = db.getDBIntProperty('rocksdb.num-blob-files');
|
|
|
631
844
|
const numKeys = db.getDBIntProperty('rocksdb.estimate-num-keys');
|
|
632
845
|
```
|
|
633
846
|
|
|
847
|
+
### `db.isWriteStalled(): boolean`
|
|
848
|
+
|
|
849
|
+
Whether RocksDB is currently applying write backpressure to this database —
|
|
850
|
+
delaying (rate-limiting) or fully stopping writes. Reads the live
|
|
851
|
+
`rocksdb.is-write-stopped` and `rocksdb.actual-delayed-write-rate` properties.
|
|
852
|
+
|
|
853
|
+
This is the authoritative, live pull counterpart to the
|
|
854
|
+
[`'writeStall'` event](#event-writestall): the event pushes per-column-family
|
|
855
|
+
entries into a stall (rate-limited), while this reports the current state on
|
|
856
|
+
demand and can never be stale. It is **database-wide** — the write controller is
|
|
857
|
+
shared across every column family, so it answers "are writes stalled anywhere"
|
|
858
|
+
rather than for one column family.
|
|
859
|
+
|
|
860
|
+
```typescript
|
|
861
|
+
if (db.isWriteStalled()) {
|
|
862
|
+
console.warn('writes are currently throttled or blocked');
|
|
863
|
+
}
|
|
864
|
+
```
|
|
865
|
+
|
|
634
866
|
### `db.getRange(options?: IteratorOptions): ExtendedIterable`
|
|
635
867
|
|
|
636
868
|
Retrieves a range of keys and their values. Supports both synchronous and asynchronous iteration.
|
|
@@ -652,6 +884,23 @@ for (const { key, value } of db.getRange({ start: 'a', end: 'z' })) {
|
|
|
652
884
|
}
|
|
653
885
|
```
|
|
654
886
|
|
|
887
|
+
Pass `transaction: txn` to iterate this column family through that transaction, exactly as
|
|
888
|
+
`get()` does with the same option: the iterator sees the transaction's staged writes and reads on its
|
|
889
|
+
snapshot, and range bounds apply to staged keys too. `getKeys()` and `getKeysCount()` accept it as
|
|
890
|
+
well. On a transaction (`txn.getRange()`) the transaction itself is the context and takes precedence
|
|
891
|
+
over a `transaction` option, as it does for `txn.get()`. Such an iterator is closed when the
|
|
892
|
+
transaction commits or aborts: a later `next()` throws, while `return()` stays a no-op. Opening a
|
|
893
|
+
range or counting through a transaction that has already started committing throws as well.
|
|
894
|
+
|
|
895
|
+
```typescript
|
|
896
|
+
await db.transaction(async (txn) => {
|
|
897
|
+
await txn.put('c', 'staged');
|
|
898
|
+
for (const { key, value } of db.getRange({ start: 'a', end: 'z', transaction: txn })) {
|
|
899
|
+
console.log({ key, value }); // includes { key: "c", value: "staged" }
|
|
900
|
+
}
|
|
901
|
+
});
|
|
902
|
+
```
|
|
903
|
+
|
|
655
904
|
### `db.getUserSharedBuffer(key: Key, defaultBuffer: ArrayBuffer, options?)`
|
|
656
905
|
|
|
657
906
|
Creates a new buffer with the contents of `defaultBuffer` that can be accessed across threads. This
|
|
@@ -848,13 +1097,14 @@ operations methods as the `RocksDatabase` instance plus:
|
|
|
848
1097
|
no further transaction operations are permitted. Calling this method multiple times has no effect.
|
|
849
1098
|
- `txn.commit(): Promise<void>` Asynchronously commits the transaction and closes the transaction.
|
|
850
1099
|
- `txn.commitSync()` Synchronously commits and closes the transaction.
|
|
851
|
-
- `txn.getTimestamp(): number` Retrieves the transaction
|
|
852
|
-
defaults to
|
|
1100
|
+
- `txn.getTimestamp(): number` Retrieves the transaction timestamp in milliseconds as a decimal. It
|
|
1101
|
+
defaults to a process-wide monotonic value assigned when the transaction was created.
|
|
853
1102
|
- `txn.id: number` The read-only transaction ID. Transaction IDs are unique to the RocksDB database
|
|
854
1103
|
path, regardless the database name/column family.
|
|
855
|
-
- `txn.setTimestamp(ts?: number): void` Overrides the transaction
|
|
856
|
-
a timestamp, it
|
|
857
|
-
|
|
1104
|
+
- `txn.setTimestamp(ts?: number): void` Overrides the transaction timestamp in milliseconds. If
|
|
1105
|
+
called without a timestamp, it claims a fresh monotonic value. It must be called before staging
|
|
1106
|
+
any write or transaction-log entry, and a supplied value must be finite, positive, and below
|
|
1107
|
+
`8.64e15`.
|
|
858
1108
|
|
|
859
1109
|
#### `txn.abandonWrites(): void`
|
|
860
1110
|
|
|
@@ -890,8 +1140,9 @@ Once called, no further transaction operations are permitted.
|
|
|
890
1140
|
|
|
891
1141
|
#### `txn.getTimestamp(): number`
|
|
892
1142
|
|
|
893
|
-
Retrieves the transaction
|
|
894
|
-
the transaction was created.
|
|
1143
|
+
Retrieves the transaction timestamp in milliseconds since the Unix epoch as a decimal. It defaults
|
|
1144
|
+
to a process-wide monotonic value assigned when the transaction was created. The transaction log
|
|
1145
|
+
uses this timestamp as the batch key; producers may also encode it into their own record format.
|
|
895
1146
|
|
|
896
1147
|
#### `txn.id`
|
|
897
1148
|
|
|
@@ -900,14 +1151,24 @@ Type: `number`
|
|
|
900
1151
|
The transaction ID represented as a 32-bit unsigned integer. Transaction IDs are unique to the
|
|
901
1152
|
RocksDB database path, regardless the database name/column family.
|
|
902
1153
|
|
|
903
|
-
#### `txn.setTimestamp(ts
|
|
1154
|
+
#### `txn.setTimestamp(ts?: number): void`
|
|
1155
|
+
|
|
1156
|
+
Overrides the transaction timestamp in milliseconds since the Unix epoch. Replication receivers and
|
|
1157
|
+
crash replay can use this to adopt an origin transaction's log key. If called without a timestamp,
|
|
1158
|
+
it claims a fresh monotonic value.
|
|
1159
|
+
|
|
1160
|
+
The override must happen while the transaction is pending and before any database write or
|
|
1161
|
+
transaction-log entry is staged. A log batch that has already been written keeps its timestamp
|
|
1162
|
+
across a coordinated retry; reapplying that same timestamp remains idempotent while pending. A
|
|
1163
|
+
supplied value must be finite, positive, and below `8.64e15`.
|
|
904
1164
|
|
|
905
|
-
|
|
906
|
-
|
|
1165
|
+
rocksdb-js does not define a record's value layout or version metadata. A producer that copies the
|
|
1166
|
+
transaction timestamp into record bytes is responsible for calling `setTimestamp()` before reading
|
|
1167
|
+
and encoding it.
|
|
907
1168
|
|
|
908
1169
|
```typescript
|
|
909
1170
|
await db.transaction(async (txn) => {
|
|
910
|
-
txn.setTimestamp(Date.now()
|
|
1171
|
+
txn.setTimestamp(Date.now());
|
|
911
1172
|
});
|
|
912
1173
|
```
|
|
913
1174
|
|
|
@@ -1055,6 +1316,38 @@ db.on('error', (err) => {
|
|
|
1055
1316
|
});
|
|
1056
1317
|
```
|
|
1057
1318
|
|
|
1319
|
+
### Event: `'writeStall'`
|
|
1320
|
+
|
|
1321
|
+
Emitted when a column family **enters** a RocksDB write-stall — the push signal that writes are being
|
|
1322
|
+
throttled or blocked (for example when `dbWriteBufferSize` is too small for the number of column
|
|
1323
|
+
families and RocksDB thrashes on premature flushes). Listeners receive three string arguments:
|
|
1324
|
+
|
|
1325
|
+
- `columnFamily: string` — the column family that entered the stall (`'default'` for the primary).
|
|
1326
|
+
- `previousCondition: string` — `'normal' | 'delayed' | 'stopped'`.
|
|
1327
|
+
- `currentCondition: string` — `'delayed'` (writes rate-limited) or `'stopped'` (writes blocked until
|
|
1328
|
+
a flush frees a memtable/L0 slot).
|
|
1329
|
+
|
|
1330
|
+
The event is **rising-edge only and rate-limited** per column family. RocksDB oscillates a CF's stall
|
|
1331
|
+
condition many times per second during a sustained stall, so the event fires when a CF _enters_ a
|
|
1332
|
+
stall and then at most once per `ROCKSDB_JS_WRITE_STALL_DEBOUNCE_MS` (default 1000 ms) while it stays
|
|
1333
|
+
stalling — never a flood. **Recovery is not pushed**: reliably distinguishing a brief dip from a real
|
|
1334
|
+
recovery isn't possible without a timer, so for current state use the pull below rather than waiting
|
|
1335
|
+
for a `'normal'` event.
|
|
1336
|
+
|
|
1337
|
+
For "are writes stalled right now?", call [`db.isWriteStalled()`](#dbiswritestalled-boolean) — it
|
|
1338
|
+
reads the live write-controller state and is authoritative. Note it is **database-wide** (the write
|
|
1339
|
+
controller is shared across all column families), so it answers "is anything stalling writes" while
|
|
1340
|
+
the event names _which_ CF; a listener attached mid-stall should consult it rather than expect a
|
|
1341
|
+
missed rising edge.
|
|
1342
|
+
|
|
1343
|
+
```typescript
|
|
1344
|
+
db.on('writeStall', (cf, prev, cur) => {
|
|
1345
|
+
console.warn(`RocksDB write stall entered on ${cf}: ${prev} -> ${cur}`);
|
|
1346
|
+
});
|
|
1347
|
+
// ...and to check the current state on demand:
|
|
1348
|
+
if (db.isWriteStalled()) console.warn('writes are currently throttled or blocked');
|
|
1349
|
+
```
|
|
1350
|
+
|
|
1058
1351
|
## Event API
|
|
1059
1352
|
|
|
1060
1353
|
`rocksdb-js` provides a EventEmitter-like API that lets you asynchronously notify events to one or
|
|
@@ -1195,8 +1488,11 @@ Column family and ticker stat values are 64-bit unsigned integers and histogram
|
|
|
1195
1488
|
`StatsHistogramData` objects.
|
|
1196
1489
|
|
|
1197
1490
|
The result also always includes a summarized, aggregate set of `txnlog.*` keys covering all of
|
|
1198
|
-
the database's transaction logs
|
|
1199
|
-
|
|
1491
|
+
the database's transaction logs, and a set of `writeBufferManager.*` keys describing the
|
|
1492
|
+
process-wide `WriteBufferManager`
|
|
1493
|
+
([`getWriteBufferManagerStats()`](#getwritebuffermanagerstats-writebuffermanagerstats)
|
|
1494
|
+
carries the same values plus its column-family inventory). Both sets are present regardless of
|
|
1495
|
+
whether statistics are enabled and can be fetched individually with `db.getStat()`. For detailed, per-log statistics —
|
|
1200
1496
|
including memory-map usage — use [`log.getStats()`](#loggetstats-transactionlogstats). All stat
|
|
1201
1497
|
names are documented in [docs/stats.md](docs/stats.md).
|
|
1202
1498
|
|
|
@@ -1503,6 +1799,14 @@ holds the last-known version for that key. Because slots are addressed by a hash
|
|
|
1503
1799
|
share a slot; a collision only ever causes a conservative miss (a real read), never a stale value to
|
|
1504
1800
|
be treated as fresh.
|
|
1505
1801
|
|
|
1802
|
+
This first word is the only version the table derives on its own: it is what a read with
|
|
1803
|
+
`populateVersion: true` publishes and what a transaction write invalidates against. A producer whose
|
|
1804
|
+
record format also carries a separately assigned version elsewhere in the value must keep using the
|
|
1805
|
+
first word on both sides of the check — as `expectedVersion`, and as the argument to any explicit
|
|
1806
|
+
[`db.populateVersion()`](#dbpopulateversionkey-key-version-number-void) call, which publishes
|
|
1807
|
+
whatever it is given. Comparing against the other value would miss the fast path, and publishing it
|
|
1808
|
+
would make the slot disagree with what the next write invalidates.
|
|
1809
|
+
|
|
1506
1810
|
The freshness check works as follows:
|
|
1507
1811
|
|
|
1508
1812
|
1. Pass `{ expectedVersion }` to `get()` / `getSync()`. If the slot currently records that version,
|
|
@@ -1620,9 +1924,19 @@ const names = db.listLogs();
|
|
|
1620
1924
|
### `db.purgeLogs({ includeEntryCounts: true, ...options }): { path: string; entries: number }[]`
|
|
1621
1925
|
|
|
1622
1926
|
Deletes transaction log files older than the `transactionLogRetention` (defaults to 3 days).
|
|
1927
|
+
Ordinary retention keeps the sequence file named by `txn.state` and every newer file as the live
|
|
1928
|
+
store's retention floor, or the highest sequence file when there is no persisted flush position. It
|
|
1929
|
+
removes only an eligible contiguous prefix below that floor. An idle store can therefore retain one
|
|
1930
|
+
file past the cutoff until a later write rotates and flushes it; that extra file is bounded by the
|
|
1931
|
+
store's `transactionLogMaxSize` (except when a single transaction exceeds the target). That bound
|
|
1932
|
+
covers only the floor file: when the flush position lags — `txn.state` stuck at an old sequence
|
|
1933
|
+
because RocksDB flushing is behind — every file above the floor is retained too, which
|
|
1934
|
+
`transactionLogMaxSize` does not bound. `purge.retainedUnflushedFiles` reports that case. Use
|
|
1935
|
+
`destroy: true` only to remove the store itself.
|
|
1623
1936
|
|
|
1624
1937
|
- `options: object`
|
|
1625
|
-
- `before?: number` Remove
|
|
1938
|
+
- `before?: number` Remove transaction log files older than the specified timestamp, subject to
|
|
1939
|
+
the retention floor and contiguous-prefix rules above.
|
|
1626
1940
|
- `destroy?: boolean` When `true`, deletes transaction log stores including all log sequence files
|
|
1627
1941
|
on disk.
|
|
1628
1942
|
- `includeEntryCounts?: boolean` When `true`, counts the entries in each deleted log file and
|
|
@@ -1796,7 +2110,9 @@ stats.totals.transactionsWritten; // lifetime count of transactions written
|
|
|
1796
2110
|
|
|
1797
2111
|
The `purge.retainedUnflushedFiles` gauge is useful for diagnosing why logs are not being cleaned
|
|
1798
2112
|
up: a file can be older than the retention period but still retained because its transactions have
|
|
1799
|
-
not yet been flushed to RocksDB (purging it would be unsafe for crash recovery).
|
|
2113
|
+
not yet been flushed to RocksDB (purging it would be unsafe for crash recovery). The retention
|
|
2114
|
+
floor — the sequence named by `txn.state`, or the highest sequence when there is no persisted flush
|
|
2115
|
+
position — is not counted as purgeable even when it is old and fully flushed.
|
|
1800
2116
|
|
|
1801
2117
|
### Transaction Log Initialization
|
|
1802
2118
|
|
|
@@ -2275,7 +2591,9 @@ Options for `get()`, `getSync()`, and the `getBinary*` methods.
|
|
|
2275
2591
|
- `tailing: boolean` When `true`, creates a "tailing iterator" which is a special iterator that
|
|
2276
2592
|
has a view of the complete database including newly added data and is optimized for sequential
|
|
2277
2593
|
reads. This will return records that were inserted into the database after the creation of the
|
|
2278
|
-
iterator. Defaults to `false`.
|
|
2594
|
+
iterator. Defaults to `false`. A tailing iterator ignores a transaction's snapshot: through a
|
|
2595
|
+
transaction it still reads the latest committed state, merged with that transaction's staged
|
|
2596
|
+
writes.
|
|
2279
2597
|
|
|
2280
2598
|
### `RangeOptions`
|
|
2281
2599
|
|