@harperfast/rocksdb-js 2.7.1 → 2.9.0

package/docs/transaction-log.md

@@ -198,7 +198,9 @@ await db.transaction((txn) => {
 - Log files are automatically rotated when either the index or data file reaches their configured
   maximum sizes
 - Rotation happens on the next write after the size limit is exceeded
-- Old log files can be automatically purged based on retention policy
+- Old log files can be automatically purged based on retention policy. The sequence file named by
+  `txn.state` and every newer file form the live store's retention floor, so an idle store can keep
+  one bounded file past the cutoff until a later write rotates and flushes it.
 
 ### Error Handling
 
@@ -218,10 +220,21 @@ The CLI exposes the same check as `verify-logs [name]`.
 
 ## Reading The Transaction Log
 
-Log entries are not guaranteed to be in order, but are guaranteed to have a monotonic timestamp.
-When reading the transaction log file, each transaction entry header must be read, then sorted and
-indexed. Using this index, queries can find all entries within a time range using a binary search
-and seek to get the associated entry data.
+Log entries are not in timestamp order. A transaction claims its timestamp from the process-wide
+monotonic clock when it is constructed but is appended when it commits, so under concurrency a later
+entry can carry a smaller timestamp; an entry that adopted an origin timestamp with
+`txn.setTimestamp()` carries that origin's clock instead. Timestamps are not unique: the monotonic
+clock never issues the same value twice within a process, but a restart after the wall clock moved
+backwards can reissue one, and `txn.setTimestamp()` can assign any value — including one already
+in the log — to as many transactions as the caller likes. Deduplicating on the timestamp alone is
+therefore never safe; a consumer that needs identity has to supply it (Harper pairs the timestamp
+with the originating node).
+
+When reading the transaction log file, each transaction entry header must be read and indexed. The
+index records only the entries whose timestamp is greater than every earlier one in the file — a
+running maxima — and a query seeks to the lower bound of that index, which is guaranteed to sit at
+or before every entry in the requested range. Reading forward from there and filtering is what makes
+range queries correct on an unordered file.
 
 ### Sequential Read
 

package/docs/transaction-timestamp-integrity-design.md

@@ -0,0 +1,123 @@
+# Transaction timestamp integrity
+
+## Intent
+
+Harper's source-record version and a RocksDB transaction's write identity are separate values.
+Harper's `RecordEncoder` and `PrimaryRocksDatabase` already own the record value layout and decode
+its metadata; rocksdb-js owns the transaction timestamp and uses it as the transaction-log batch
+key. This change only makes overriding that timestamp safe for Harper's two legitimate adopters,
+replication apply and crash replay.
+
+## Invariant
+
+Within one transaction attempt, the timestamp cannot change after the transaction leaves `Pending`
+or stages a database write or transaction-log entry. A retry may adopt again only when the failed
+attempt did not durably write its log batch; once a batch is durable, its timestamp remains frozen
+across retry resets. Encoding a transaction timestamp into record bytes remains the producer's
+obligation because native code cannot observe when a producer reads and copies the timestamp.
+
+## Requirements traced to Harper
+
+- HarperFast/harper#2065 intentionally allowed a source record's version to differ from the
+  transaction timestamp.
+- HarperFast/harper#2409 fixed the immediate subscription regression by carrying both values in
+  Harper's audit representation.
+- HarperFast/harper#2411 and HarperFast/harper-pro#790 require replay and replication respectively
+  to adopt the origin transaction timestamp through `setTimestamp()` before applying writes.
+- HarperFast/harper#2412 owns the optional second version word in `RecordEncoder`, including its
+  encoding, decoding, compatibility gates, and call-site audit.
+
+Checked against Harper's source rather than the issue text: `resources/RecordEncoder.ts` decodes the
+value header itself (producing `localTime`, `version` and the metadata flags) and owns every flag bit
+in that word, and `resources/PrimaryRocksDatabase.ts#getEntry()` overrides the base read entirely —
+it calls `super.getSync()`/`super.get()` and reads the encoder's output. So nothing in Harper would
+call a rocksdb-js decoder for the second word. Harper's three `setTimestamp()` call sites
+(`resources/DatabaseTransaction.ts` at the read-transaction, save and retry-replay paths) each set it
+on a freshly constructed transaction before any write, which is what this change makes enforceable.
+
+One consequence for #2412 that is not in its call-site audit: `PrimaryRocksDatabase#getEntry()` passes
+the cached entry's `version` as `expectedVersion` to the verification table, and the table keys on the
+value's **first** word. Once `version` becomes the record version, that argument has to become
+`localTime`, or cached reads stop hitting the fast path — and a record version that happened to equal
+the key's current first word would report a stale copy as fresh.
+
+## Approaches considered
+
+### Different layer
+
+Teach rocksdb-js to define and decode Harper's second version word. Rejected because Harper's
+`PrimaryRocksDatabase#getEntry()` already overrides the base read path and receives the structured
+entry produced by Harper's `RecordEncoder`; a second decoder in rocksdb-js would duplicate the
+layout and cannot supply Harper's other metadata fields.
+
+Reserving Harper's proposed `0x20000` bit in rocksdb-js without decoding it was also considered.
+Rejected because rocksdb-js does not allocate the producer's metadata namespace: Harper already
+defines the other optional-field flags in that word, while rocksdb-js knows `0x10000` only because
+the native verification table gives that specific bit behavior. A bit with no native behavior or
+public rocksdb-js contract belongs beside Harper's encoder and downgrade gate.
+
+### Deeper cause
+
+Remove `setTimestamp()` and redesign replay and replication so rocksdb-js always issues a new local
+timestamp. Rejected because Harper's record-to-log lookup and per-origin resume cursor require the
+origin transaction timestamp to remain the transaction-log key on every receiver.
+
+A construction-time timestamp option was considered as a safer replacement for the setter. It
+would add another public API and migrate the replay and replication callers, but could not remove
+the existing setter compatibly; therefore it would not eliminate the state it is meant to prevent.
+
+### Do less
+
+Make no rocksdb-js change and rely on Harper always calling `setTimestamp()` before staging writes.
+Rejected because the current public API permits a later call, after the producer has encoded the old
+timestamp or the log batch has captured it, which can split one transaction across two identities.
+
+For the separate clock-rollback concern, seeding the process clock from the largest durable key
+across the database's transaction logs was considered and rejected on Harper's own log layout.
+`resources/RocksTransactionLogStore.ts` opens one log per origin node — `useLog('local')` for locally
+originated writes and `logById(nodeId)` for each peer — and a receiver deliberately keeps the origin's
+clock as the key in the peer's log (HarperFast/harper-pro#790). A "largest key in the database" seed
+therefore reads adopted peer keys as if they were locally issued: one peer running ahead ratchets this
+node's clock on its next restart, and the node then originates those keys to its own peers. Reading
+the headers only to warn has the same ambiguity — normal peer skew would be reported as a local clock
+rollback. A correct floor has to be told which log is locally originated (a per-log opt-in, or an
+explicit "raise the floor to this value" call Harper makes from its own `local` log), which is a
+different API than the one the issue sketched and is not a prerequisite for the dual-version record
+work.
+
+### Chosen
+
+Keep `setTimestamp()` as the adoption API, reject invalid timestamp values, and reject changes once
+the transaction is no longer pending or has staged a write or log entry. Preserve the existing
+retry behavior: a reset with no durable batch clears staged writes and may adopt again, while
+`committedPosition` freezes a timestamp whose batch already landed. Remove the second-word read API,
+flag, and clock-floor-at-open implementation from this PR. The first two belong to Harper's record
+codec; the clock floor addresses a pre-existing restart/clock-rollback concern and, as specified,
+would seed from adopted origin keys as readily as from locally issued ones. The existing
+`latestTimestamp` behavior used to populate new transaction-log segment headers remains unchanged.
+
+Document what is true about log keys today, which the issue also asked for: entries are appended in
+commit order rather than timestamp order, an adopted key carries another node's clock, keys are
+unique only within a process, and `findPositionByTimestamp` is a running-maxima seek rather than a
+sorted binary search.
+
+## Verification route
+
+The existing transaction tests cover all transaction modes, invalid inputs, post-write and
+post-dispatch rejection, log-entry staging, and a coordinated retry whose log batch is already
+durable. README and AGENTS documentation will state the enforceable transaction-layer rule and the
+producer-owned record boundary. The repository build, native tests, full Vitest suite, and checks
+will run before push.
+
+## Alternatives rejected
+
+Freezing the timestamp on the first `getTimestamp()` call would break existing documented and tested
+read-then-override behavior. A getter is observational for callers that do not encode its result,
+and making every read a state transition still could not prove that a producer's record bytes match
+the transaction timestamp. The enforced boundary therefore remains the first native staging event;
+producer ordering is documented explicitly.
+
+Retaining the removed clock-floor branch's far-future header bound would change the append path using
+a ten-year policy introduced only for that floor. This change instead restores `latestTimestamp` and
+segment-header behavior exactly to `origin/main`; any provenance-aware clock rollback design belongs
+in separate work.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@harperfast/rocksdb-js",
-  "version": "2.7.1",
+  "version": "2.9.0",
   "description": "RocksDB binding for Node.js",
   "homepage": "https://github.com/HarperFast/rocksdb-js",
   "bugs": [
@@ -32,60 +32,59 @@
     }
   },
   "dependencies": {
-    "@harperfast/extended-iterable": "1.0.3",
-    "msgpackr": "2.0.5",
-    "ordered-binary": "1.6.1"
+    "@harperfast/extended-iterable": "^1.0.3",
+    "msgpackr": "^2.0.6",
+    "ordered-binary": "^1.6.1"
   },
   "devDependencies": {
     "@rollup/plugin-replace": "6.0.3",
-    "@types/node": "26.1.2",
-    "@vitest/coverage-v8": "4.1.10",
+    "@types/node": "26.4.1",
+    "@vitest/coverage-v8": "4.1.11",
     "cross-env": "10.1.0",
     "dotenv": "17.4.2",
-    "lefthook": "2.1.10",
+    "lefthook": "2.1.12",
     "lmdb": "3.5.6",
-    "node-gyp": "13.0.1",
-    "oxfmt": "0.62.0",
-    "oxlint": "1.77.0",
+    "node-gyp": "13.0.2",
+    "oxfmt": "0.66.0",
+    "oxlint": "1.81.0",
     "prebuildify": "6.0.1",
     "semver": "7.8.5",
     "tar": "7.5.22",
-    "tsdown": "0.22.14",
+    "tsdown": "0.23.0",
     "tslib": "2.8.1",
-    "tsx": "4.23.5",
     "typescript": "7.0.2",
-    "vitest": "4.1.10"
+    "vitest": "4.1.11"
   },
   "optionalDependencies": {
-    "@harperfast/rocksdb-js-darwin-arm64": "2.7.1",
-    "@harperfast/rocksdb-js-darwin-x64": "2.7.1",
-    "@harperfast/rocksdb-js-linux-arm64-glibc": "2.7.1",
-    "@harperfast/rocksdb-js-linux-arm64-musl": "2.7.1",
-    "@harperfast/rocksdb-js-linux-x64-glibc": "2.7.1",
-    "@harperfast/rocksdb-js-linux-x64-musl": "2.7.1",
-    "@harperfast/rocksdb-js-win32-arm64": "2.7.1",
-    "@harperfast/rocksdb-js-win32-x64": "2.7.1"
+    "@harperfast/rocksdb-js-darwin-arm64": "2.9.0",
+    "@harperfast/rocksdb-js-darwin-x64": "2.9.0",
+    "@harperfast/rocksdb-js-linux-arm64-glibc": "2.9.0",
+    "@harperfast/rocksdb-js-linux-arm64-musl": "2.9.0",
+    "@harperfast/rocksdb-js-linux-x64-glibc": "2.9.0",
+    "@harperfast/rocksdb-js-linux-x64-musl": "2.9.0",
+    "@harperfast/rocksdb-js-win32-arm64": "2.9.0",
+    "@harperfast/rocksdb-js-win32-x64": "2.9.0"
   },
   "engines": {
     "node": "^22.18.0 || >=24.0.0"
   },
   "gypfile": true,
   "rocksdb": {
-    "version": "11.1.2"
+    "version": "11.8.1"
   },
   "scripts": {
     "bench": "cross-env CI=1 node --expose-gc ./node_modules/vitest/vitest.mjs bench --passWithNoTests --outputJson benchmark-results.json",
     "bench:bun": "cross-env CI=1 bun --bun bench",
     "bench:deno": "cross-env CI=1 deno run --allow-all --sloppy-imports ./node_modules/vitest/vitest.mjs bench",
-    "build": "pnpm build:bundle && pnpm rebuild",
+    "build": "pnpm build:bundle && pnpm run rebuild",
     "build:binding": "node-gyp build",
     "build:binding:debug": "node-gyp build --coverage --debug --verbose",
     "build:bundle": "node scripts/clean-dist.mjs && tsdown -c tsdown.config.ts",
     "build:bundle:minify": "cross-env MINIFY=1 pnpm build:bundle",
     "check": "pnpm type-check && pnpm lint && pnpm fmt:check",
     "clean": "node-gyp clean",
-    "coverage": "tsx scripts/coverage/main.ts",
-    "coverage:native": "tsx scripts/native-test/coverage.mjs",
+    "coverage": "node scripts/coverage/main.ts",
+    "coverage:native": "node scripts/native-test/coverage.mjs",
     "fmt": "oxfmt",
     "fmt:check": "oxfmt --check",
     "lint": "oxlint",
@@ -95,7 +94,7 @@
     "test:bun": "cross-env CI=1 bun --bun --config=bunfig.toml ./node_modules/vitest/vitest.mjs --reporter=verbose",
     "test:deno": "cross-env CI=1 deno run --allow-all --sloppy-imports --v8-flags=--expose-gc ./node_modules/vitest/vitest.mjs",
     "test:stress": "cross-env CI=1 node --expose-gc ./node_modules/vitest/vitest.mjs --reporter=verbose --config=vitest-stress.config.ts",
-    "test:native": "tsx scripts/native-test/run.mjs",
+    "test:native": "node scripts/native-test/run.mjs",
     "type-check": "tsc --noEmit"
   }
 }