@harperfast/rocksdb-js 2.12.0 → 2.14.0

package/docs/performance/mutex-contention.md

@@ -171,11 +171,12 @@ A separate native reproduction confirmed [#860](https://github.com/HarperFast/ro
 `tryClose()` releases its written-position lock before latching admission closed.
 A transaction can bind, finish `writeBatch()` and reduce the pending count back to zero
 between those checks. A temporary scheduling hook before phase 3 reproduced
-`pending=0`, one real uncommitted position, and `tryClose() == true`. This is a
+`pending=0`, one real uncommitted position, and `tryClose() == true`. This was a
 pre-existing close-protocol defect, not fixed by the completion-lock change. Its
 production trigger is a concurrent destructive log purge; no customer incident or
-JavaScript-level data loss was reproduced. Keep the binding lock until a separately
-reviewed admission/drain fix resolves that handoff.
+JavaScript-level data loss was reproduced. It is now fixed without changing the binding
+lock: every bind advances a generation that `tryClose()` rechecks before latching (see
+"Conditional close" in `src/binding/transaction_log/DESIGN.md`).
 
 The investigation also found [#859](https://github.com/HarperFast/rocksdb-js/issues/859):
 the DB-instance stress test awaits an empty promise array in its worker-close phase.

package/docs/performance/transaction-state-2026-11-09.csv

@@ -0,0 +1,28 @@
+lifecycle,trial,plain_ns,atomic_ns
+commit,1,0.490,2.389
+commit,2,0.416,2.180
+commit,3,0.409,2.009
+commit,4,0.368,1.901
+commit,5,0.368,1.854
+commit,6,0.353,1.729
+commit,7,0.359,1.714
+commit,8,0.361,1.697
+commit,9,0.365,1.698
+retry_then_commit,1,0.609,3.140
+retry_then_commit,2,0.602,3.137
+retry_then_commit,3,0.608,3.147
+retry_then_commit,4,0.602,3.144
+retry_then_commit,5,0.609,3.142
+retry_then_commit,6,0.602,3.138
+retry_then_commit,7,0.603,3.137
+retry_then_commit,8,0.602,3.137
+retry_then_commit,9,0.609,3.147
+abort,1,0.294,1.058
+abort,2,0.477,1.089
+abort,3,0.260,1.066
+abort,4,0.479,1.062
+abort,5,0.288,1.060
+abort,6,0.510,1.062
+abort,7,0.285,1.118
+abort,8,0.507,1.117
+abort,9,0.507,1.120

package/docs/performance/transaction-state.md

@@ -0,0 +1,67 @@
+# Transaction state CAS cost (#1013)
+
+The state cell prevents a failed commit's completion from reviving a handle that a
+foreign `close()` has aborted. Its lifecycle transitions add atomic read-modify-write
+work on transaction paths, including the read-only transaction abort path.
+
+## Work per lifecycle
+
+Counts below are for the normal uncontended paths, excluding existing atomics in
+admission, async-work tracking, and teardown. A CAS uses acquire-release ordering on
+success and relaxed ordering on failure.
+
+| Path                                               | State-cell work replacing plain state accesses                                                                                                 |
+| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| Successful `commit()` or `commitSync()`            | Two strong CAS operations: admission and successful completion. The subsequent `close()` loads `Committed` and performs no state CAS.          |
+| Failed attempt returned to the caller for retry    | Two strong CAS operations: admission and revert to `Pending`. Each subsequent attempt pays its own admission and completion/revert.            |
+| Explicit abort of a pending transaction            | One successful weak CAS to `Aborted`; the subsequent `close()` loads `Aborted` without another state CAS.                                      |
+| Closing a pending or committing transaction        | One successful weak CAS in the uncontended case. A racing revert or spurious failure can repeat the loop; observing a terminal state stops it. |
+| Per-operation state gates (`get`, `put`, `remove`) | Relaxed atomic loads, with no CAS. These remain advisory and do not protect `txn` from concurrent teardown.                                    |
+
+Setup-error cleanup also attempts `Committing` → `Pending`; a concurrent abort makes
+that CAS fail without reviving the handle. No mutex, allocation, or process-global
+counter was added by the state cell. It is per transaction, but async execution can
+still transfer its cache line between the owning JS thread and a commit worker.
+
+## Local measurement
+
+[The standalone benchmark](../../benchmark/transaction-state.cc) compares the actual
+`TransactionStateCell` with a single-threaded plain-field reference: unconditional
+admission/success stores, a checked revert, and checked abort stores. Both use the
+same byte-sized enum to isolate atomic access cost (the old production enum was
+int-sized). Compiler barriers keep accesses between phases from being eliminated;
+they emit no hardware fences. This measures only the state operations, initialization,
+and loop overhead, not RocksDB, N-API, rollback, log I/O, or scheduling.
+
+Run from the repository root with Clang or GCC:
+
+```sh
+clang++ -std=c++20 -O3 -DNDEBUG -Isrc/binding benchmark/transaction-state.cc -o /tmp/transaction-state-bench
+/tmp/transaction-state-bench
+```
+
+Measured against `b05ca26f05f6fee7255e427b850ccb0b57e97336` on 2026-11-09,
+macOS 27.0.1 arm64, Apple Clang 21.0.0. Each variant gets a 100,000-iteration warmup,
+then nine paired trials of 5,000,000 lifecycles, alternating variant order. No builds
+or tests ran concurrently with the measurement. The shared host was not isolated
+or CPU-pinned; nearby load averages were 1.52 / 1.62 / 1.70, so small timings are
+host-sensitive. Generated assembly retained `casalb` for the atomic transitions
+and ordinary byte loads/stores for the reference.
+
+[Raw trials](transaction-state-2026-11-09.csv), nanoseconds per lifecycle:
+
+| Lifecycle                                           | Plain median (range) | Atomic median (range) | Difference of medians |
+| --------------------------------------------------- | -------------------: | --------------------: | --------------------: |
+| Successful commit and close                         |  0.368 (0.353–0.490) |   1.854 (1.697–2.389) |             +1.486 ns |
+| One failed attempt, then successful retry and close |  0.603 (0.602–0.609) |   3.140 (3.137–3.147) |             +2.537 ns |
+| Explicit abort and close                            |  0.477 (0.260–0.510) |   1.066 (1.058–1.120) |             +0.589 ns |
+
+**Performance:** Successful commits add two state CAS operations in place of plain
+stores; failed attempts also pay an admission and a revert CAS, and explicit aborts
+add one successful CAS. The local uncontended state-only comparison measured
++1.486 ns per successful commit/close and +0.589 ns per explicit abort/close at the
+median, buying an indivisible transition that cannot overwrite a concurrent abort.
+The per-record gates use relaxed loads, not RMW operations. These measurements do
+not establish an end-to-end throughput regression or bound cross-core cache-line
+transfer, contention, or costs on x86/other hosts. In particular, the single-threaded
+benchmark does not measure the JS-thread-to-commit-worker handoff.

package/docs/stats.md

@@ -189,6 +189,10 @@ which includes all of the stats above plus the following:
 | `rocksdb.blobdb.gc.num.new.files`                                  | Number of new blob files generated by garbage collection (legacy stacked BlobDB).                                                             | ticker    |
 | `rocksdb.blobdb.get.micros`                                        | Distribution of `Get()` latencies (microseconds) (legacy stacked BlobDB).                                                                     | histogram |
 | `rocksdb.blobdb.key.size`                                          | Distribution of key sizes for BlobDB operations.                                                                                              | histogram |
+| `rocksdb.blobdb.lazy.partial.bytes.saved`                          | Bytes not read from blob files thanks to partial (byte-range) lazy reads: the value size minus the bytes actually read.                       | ticker    |
+| `rocksdb.blobdb.lazy.partial.read.count`                           | Number of lazy blob reads that read only a byte range of an uncompressed blob value.                                                          | ticker    |
+| `rocksdb.blobdb.lazy.read.bytes`                                   | Bytes read from storage while resolving lazy wide-column results (`GetEntityLazy`/`MultiGetEntityLazy`).                                      | ticker    |
+| `rocksdb.blobdb.lazy.read.count`                                   | Number of storage reads issued while resolving lazy wide-column results (`GetEntityLazy`/`MultiGetEntityLazy`).                               | ticker    |
 | `rocksdb.blobdb.multiget.micros`                                   | Distribution of `MultiGet()` latencies (microseconds) (legacy stacked BlobDB).                                                                | histogram |
 | `rocksdb.blobdb.next.micros`                                       | Distribution of iterator `Next()` latencies (microseconds) (legacy stacked BlobDB).                                                           | histogram |
 | `rocksdb.blobdb.num.get`                                           | Number of `Get()` calls (legacy stacked BlobDB).                                                                                              | ticker    |

package/docs/steady-clock-design.md

@@ -87,7 +87,7 @@ separate properties. No promise is made that runtime timers use the same suspend
 ## Verification
 
 The end-to-end route is the real N-API export through `src/index.ts`, parent/worker bracketing on
-Node, Bun and Deno, staggered/restarted and concurrent workers (including `workerFirst >= parentBefore + stagger - tolerance`), elapsed
+Node and Bun (Deno: local `pnpm test:deno` only, #947), staggered/restarted and concurrent workers (including `workerFirst >= parentBefore + stagger - tolerance`), elapsed
 progression, and built ESM/CJS
 exports. Temporarily substituting a worker-local clock must make those brackets fail, proving
 they reject the motivating defect.

package/docs/transaction-log.md

@@ -58,7 +58,7 @@ The token is used to validate that the file is indeed a transaction log.
 
 #### `version`
 
-The transaction log file format version. Currently, version `1` is the latest.
+The transaction log file format version. Currently, version `1` is the latest. Opening a database whose newest transaction-log segments include one with a higher version (written by a newer release) fails with an error naming the segment, rather than skipping it, and this release never appends after such a segment. A malformed or unreadable segment is still skipped.
 
 #### `timestamp`
 
@@ -84,9 +84,10 @@ The size of the data entry directly following the transaction header.
 
 Transaction entry related flags.
 
-| Flag                              | Value  | Description                                |
-| --------------------------------- | ------ | ------------------------------------------ |
-| `TRANSACTION_LOG_ENTRY_LAST_FLAG` | `0x01` | Indicates the last entry for a transaction |
+| Flag                                 | Value  | Description                                                                                         |
+| ------------------------------------ | ------ | --------------------------------------------------------------------------------------------------- |
+| `TRANSACTION_LOG_ENTRY_LAST_FLAG`    | `0x01` | Indicates the last entry for a transaction                                                          |
+| `TRANSACTION_LOG_ENTRY_ABORTED_FLAG` | `0x02` | On a transaction's first entry: the transaction did not commit, and readers skip all of its entries |
 
 ## Encoding Details
 
@@ -210,11 +211,39 @@ await db.transaction((txn) => {
   bytes can name a later position than was flushed, and retention would then treat unflushed
   entries below it as flushed.
 - A reader that starts from the last flushed position, or one already inside a purged file,
-  continues at the lowest retained sequence (or at the next write when nothing is retained).
+  continues at the next retained sequence after that position (or at the next write when nothing is
+  retained).
 - Downgrading to a release without this rule after a store's last file was purged is unsafe: older
   releases restart an empty store at sequence 1, below the flushed position, so a replay from
   `txn.state` skips the new writes and retention treats them as flushed. Remove the store's
-  `txn.state` before downgrading.
+  `txn.state` and `txn.purged` before downgrading.
+
+### Purge Watermark
+
+Each store keeps `txn.purged`, the record behind `log.getPurgeWatermark()`: an upper bound on every
+entry key in every file the store deleted. Keys are not ordered within or across files (see below),
+so neither file age nor the keys still in the log bound what a purge removed; only the deleted files
+themselves did, and they are gone.
+
+- Every purge (`db.purgeLogs()` and the retention that runs when a database opens) raises the
+  record to the largest key of each file it is about to delete and syncs it before the first file is
+  deleted. A crash in between leaves the record above the files that survived, never below a file
+  that was deleted. A record that cannot be written stops the purge and warns through `log.warn`.
+  Closing a database and rotating a file delete nothing.
+- The largest key of a file this process wrote is tracked as entries are appended. A file written
+  by an earlier process is read once, before deletion, by the same scan that
+  [the timestamp floor](#the-timestamp-floor-at-open) uses; one purge run spends at most 2 seconds
+  on these reads (`ROCKSDB_JS_PURGE_SCAN_MS`, read once per process), and files it could not read wait
+  for the next run. A deployment whose files take longer than that to read raises the budget. A file whose keys cannot be
+  proven because it is corrupt is still deleted, and the watermark becomes unknown.
+- The record is reported as known only when every file deletion in the store's history was recorded
+  this way. A store created before this release has no record and reports unknown permanently, as
+  does a store whose files an older release deleted after the record was written (detected when a
+  sequence after the record's is missing), or whose record cannot be read. Files deleted outside
+  rocksdb-js cannot be detected in general.
+- `purgeLogs({ destroy: true })` marks the record unknown before it deletes anything, and leaves the
+  store open, with its files, if it cannot. A log recreated under the same name starts a new history.
+- Backups capture the record with the files they copy, so a restored store keeps its watermark.
 
 ### Error Handling
 
@@ -268,7 +297,10 @@ segment's creation, so there is no shortcut: every segment is read.
 Retention can purge every segment of the named log. The walk then finds no keys and leaves the clock
 alone, so a backward step after restart can reissue a key that was only in purged segments. With
 age-based retention alone that needs a backward step at least as large as the retention period; a
-`purgeLogs({ before })` cutoff closer to now shrinks it to the time since that cutoff.
+`purgeLogs({ before })` cutoff closer to now shrinks it to the time since that cutoff. A caller that
+needs the guarantee past retention persists its own floor — the largest key it has issued, or any
+bound it has advertised — and restores it with `raiseTimestampFloor()` before the process starts a
+transaction. That call only raises the in-memory clock; persisting the value is the caller's job.
 
 Name only a log this process originates. A log a replication receiver writes under an adopted origin
 timestamp is keyed by another node's clock, and seeding from it would ratchet this process's clock

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@harperfast/rocksdb-js",
-  "version": "2.12.0",
+  "version": "2.14.0",
   "description": "RocksDB binding for Node.js",
   "homepage": "https://github.com/HarperFast/rocksdb-js",
   "bugs": [
@@ -38,39 +38,39 @@
   },
   "devDependencies": {
     "@rollup/plugin-replace": "6.0.3",
-    "@types/node": "26.6.5",
+    "@types/node": "26.6.6",
     "@vitest/coverage-v8": "4.1.11",
     "cross-env": "10.1.0",
-    "dotenv": "18.0.7",
-    "lefthook": "2.2.1",
-    "lmdb": "3.5.7",
+    "dotenv": "18.0.9",
+    "lefthook": "2.2.2",
+    "lmdb": "3.5.9",
     "node-gyp": "13.1.0",
-    "oxfmt": "0.72.0",
-    "oxlint": "1.87.0",
+    "oxfmt": "0.77.0",
+    "oxlint": "1.92.0",
     "prebuildify": "6.0.1",
-    "semver": "7.8.5",
+    "semver": "7.8.6",
     "tar": "7.5.22",
-    "tsdown": "0.23.1",
-    "tslib": "2.8.1",
+    "tsdown": "0.23.7",
+    "tslib": "2.9.1",
     "typescript": "7.0.2",
     "vitest": "4.1.11"
   },
   "optionalDependencies": {
-    "@harperfast/rocksdb-js-darwin-arm64": "2.12.0",
-    "@harperfast/rocksdb-js-darwin-x64": "2.12.0",
-    "@harperfast/rocksdb-js-linux-arm64-glibc": "2.12.0",
-    "@harperfast/rocksdb-js-linux-arm64-musl": "2.12.0",
-    "@harperfast/rocksdb-js-linux-x64-glibc": "2.12.0",
-    "@harperfast/rocksdb-js-linux-x64-musl": "2.12.0",
-    "@harperfast/rocksdb-js-win32-arm64": "2.12.0",
-    "@harperfast/rocksdb-js-win32-x64": "2.12.0"
+    "@harperfast/rocksdb-js-darwin-arm64": "2.14.0",
+    "@harperfast/rocksdb-js-darwin-x64": "2.14.0",
+    "@harperfast/rocksdb-js-linux-arm64-glibc": "2.14.0",
+    "@harperfast/rocksdb-js-linux-arm64-musl": "2.14.0",
+    "@harperfast/rocksdb-js-linux-x64-glibc": "2.14.0",
+    "@harperfast/rocksdb-js-linux-x64-musl": "2.14.0",
+    "@harperfast/rocksdb-js-win32-arm64": "2.14.0",
+    "@harperfast/rocksdb-js-win32-x64": "2.14.0"
   },
   "engines": {
     "node": "^22.18.0 || >=24.0.0"
   },
   "gypfile": true,
   "rocksdb": {
-    "version": "11.8.1"
+    "version": "11.13.2"
   },
   "scripts": {
     "bench": "cross-env CI=1 node --expose-gc ./node_modules/vitest/vitest.mjs bench --passWithNoTests --outputJson benchmark-results.json",