@push.rocks/smartdb 5.3.2 → 5.5.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartdb",
-  "version": "5.3.2",
+  "version": "5.5.0",
   "private": false,
   "description": "A MongoDB-compatible embedded database server with wire protocol support, backed by a high-performance Rust engine.",
   "exports": {
@@ -15,10 +15,10 @@
     "@api.global/typedserver": "^8.4.6",
     "@design.estate/dees-element": "^2.2.4",
     "@git.zone/tsbuild": "^4.4.2",
-    "@git.zone/tsbundle": "^2.11.4",
-    "@git.zone/tsrust": "^1.10.3",
+    "@git.zone/tsbundle": "^2.13.0",
+    "@git.zone/tsrust": "^1.10.4",
     "@git.zone/tstest": "^4.0.0",
-    "@types/node": "26.2.0",
+    "@types/node": "26.4.1",
     "mongodb": "^7.5.0"
   },
   "dependencies": {

package/readme.hints.md

@@ -1,5 +1,23 @@
 # smartdb hints
 
+## Single-Node Concern Profile, Namespace Clock, and Deadlines (5.5.0)
+
+- `rustdb-commands/src/concerns.rs` is the sole owner of `readConcern`, `writeConcern`, and `maxTimeMS` validation. It runs in the router before authentication, sessions, permits, or gates. The profile is stated for one voting node: fsynced acknowledgement satisfies `w: 1`, `w: "majority"`, and `j: true`; `w: N > 1` is `UnsatisfiableWriteConcern` (100), tag sets are `UnknownReplWriteConcern` (79), cluster-time read concerns and misplaced fields are `InvalidOptions` (72), malformed deadlines are `BadValue` (2). `j: true` on `MemoryStorageAdapter` is `BadValue` through `StorageAdapter::journaled_writes()`. The driver sends a transaction's read concern on whichever statement starts the transaction, so a transaction-starting insert may carry one. Never accept a concern by widening an allowlist; every accepted value must map to a guarantee the engine actually provides.
+- `rustdb-commands/src/clock.rs` (`CommandContext::namespace_clock`) is the publication clock. Every in-process publication that changes visible namespace contents records `publish_namespace` while it still holds the namespace write lock (point writes, transaction commit completion, `create`, `drop`, `renameCollection`, `$out`, `$merge`, oplog revert) or `publish_database` (`dropDatabase`, in-process durable replacement through `invalidate_database_runtime`, and cross-process publication observed through `invalidate_database_runtime_for_epoch`). `$out` publishes right after it drops the old contents and again after the refill, so a failed refill still invalidates pinned snapshots. Index DDL does not change documents and does not publish. Adding a new mutation path without a publication is a snapshot-consistency bug.
+- Transactions pin `TransactionState::snapshot_sequence` when they materialize their first namespace. `transactions::load_transaction_docs_locked` requires the caller to hold the namespace write lock (insert/update/delete already do); `load_transaction_docs` acquires it for readers (`find`, `count`, `distinct`). A namespace whose `last_publication` exceeds the pinned sequence fails with `SnapshotUnavailable` (246); the router labels it `TransientTransactionError` so `withTransaction` restarts. This turns lazy per-namespace materialization into one database-wide snapshot without a second data copy; Storage VNext MVCC read timestamps replace the materialized copy itself.
+- Sessions retain `last_outcome` (`TransactionOutcome::Committed`/`Aborted` per transaction number) through `SessionEngine::finish_transaction_if_bound`. `commitTransaction`/`abortTransaction` without an active binding replay that outcome: repeated commit is `ok`, abort-after-commit is `TransactionCommitted` (256), commit-after-abort is `NoSuchTransaction` (251), and an older `txnNumber` on start is `TransactionTooOld` (225). Outcomes are process-local; after a crash the outcome of an in-flight commit is discoverable only from the data, which is why callers must journal their own attempts inside the transaction.
+- `rustdb-commands/src/deadline.rs` holds the `maxTimeMS` deadline in a tokio task-local set by `CommandRouter::route_with_response_policy`. Checks are cooperative: admission, `deadline::wait_bounded` inside `acquire_write_lock`/`acquire_maintenance_write_gate`, `wait_until_deadline` around the maintenance read lease, `check()` after materialization in readers, `check_every` in insert/delete loops, and one final `check()` before `apply_write_batch_with_completion` in commit. Handler futures are not cancellation-safe, so never wrap a handler in `tokio::time::timeout`; add a check at the next safe point instead. `SMARTDB_TEST_OPERATION_DELAY=<command>:<ms>` (test-support builds only) injects a pre-dispatch delay for deterministic expiry tests.
+- Error labels are derived from the response code in `attach_error_labels` so pre-dispatch rejections and handler errors label identically: transaction statements with 112/246/251 carry `TransientTransactionError`; `commitTransaction` with 50 or 64 carries `UnknownTransactionCommitResult`.
+
+## Storage VNext A0 Committed Log
+
+- `rustdb-kernel` is a fixture-only, non-serving A0 durability slice. It is not linked into the released `rustdb` binary and provides no BSON, CRUD, MVCC, concern, deadline, migration, replication, or availability behavior.
+- The immutable `kernel-manifest.sdb` stores format, group, timeline, and fixed segment identity only. It never stores a mutable commit frontier. Complete prepared/commit-marker pairs in `commit-0000000000000001.sdb` are the sole A0 commit authority.
+- Each new command validates and prepares before append, receives the next contiguous nonzero LSN, and appends one `KPR1` prepared frame plus one `KCM1` marker binding the exact prepared bytes. Exact retries return the original LSN/response without appending; fingerprint conflicts, invalid commands, and segment exhaustion consume no LSN or state.
+- Any append or sync error after write admission is ambiguous and permanently fences that handle, including reads and exact retries. Reopen is the only reconciliation path. Reopen syncs every complete recovered segment before exposing state or receipts.
+- Recovery accepts only the declared timeline, contiguous LSNs, valid CRC frames, a matching marker position/hash, valid commands, and one durable record per receipt identity. It may truncate only an incomplete final frame of the expected kind or a complete valid prepared record without a complete marker. Complete corruption, wrong-kind tails, invalid prepared records, holes, duplicates, and marker mismatches fail without changing the segment.
+- The descriptor-safe root/framing primitives are shared by `rustdb-kernel` and `rustdb-replication`. Their common parent-slot/in-root lock namespace is intentional: kernel and Raft roots must be exclusive and are never interchangeable.
+
 ## Authoritative File Recovery
 
 - `data.rdb` is the sole runtime source for rebuilding KeyDir. The v1 `keydir.hint` format cannot prove completeness or data-generation identity, so existing regular hint files are tolerated for compatibility and diagnostics but their contents are never loaded as runtime state. New shutdowns, compactions, and v0-to-v1 migrations do not publish hints.

package/readme.md

@@ -354,7 +354,7 @@ Legacy v0 JSON collections are converted into hidden sibling staging directories
 
 Persisted users also carry a random principal identity and a monotonic generation. SmartDB reloads and resolves that identity for every authenticated command, so password or role changes take effect immediately and stale sockets are rejected. Deleting and recreating the same username creates a different principal; an old connection cannot inherit the replacement user's authority. Cross-process user updates are serialized through the persisted users-file lock.
 
-Single-node transactions are supported through official MongoDB driver sessions. `find`, `count`, `distinct`, `insert`, `update`, `findAndModify`, and `delete` use the transaction snapshot and buffered write set; `commitTransaction` applies that write set with conflict checks, and `abortTransaction` discards it. After applicable authentication, authorization, and allocation-policy checks, collection, database, and index DDL, user-management mutations, and aggregate pipelines ending in `$out` or `$merge` reject transaction envelopes with `OperationNotSupportedInTransaction` (code 263) before session or transaction creation, maintenance gates, or mutation. Live logical sessions remain resumable across socket disconnects. Bounded background cleanup aborts expired transactions, removes expired sessions, and releases publication leases; explicit `endSessions` and `killSessions` do the same for their active transactions.
+Single-node transactions are supported through official MongoDB driver sessions. `find`, `count`, `distinct`, `insert`, `update`, `findAndModify`, and `delete` use the transaction snapshot and buffered write set; `commitTransaction` applies that write set with conflict checks, and `abortTransaction` discards it. The snapshot is database-wide: the first namespace a transaction materializes pins a publication sequence on the namespace clock, and any later namespace that was published after that sequence fails closed with `SnapshotUnavailable` (code 246) and the `TransientTransactionError` label so the driver restarts the transaction on fresh state; write-write conflicts surface at commit as `WriteConflict` (code 112) with the same label. Each session remembers the outcome of its last transaction number: a repeated `commitTransaction` replays `ok`, `abortTransaction` after a commit reports `TransactionCommitted` (code 256), `commitTransaction` after an abort reports `NoSuchTransaction` (code 251), and starting a transaction with an older number reports `TransactionTooOld` (code 225). After applicable authentication, authorization, and allocation-policy checks, collection, database, and index DDL, user-management mutations, and aggregate pipelines ending in `$out` or `$merge` reject transaction envelopes with `OperationNotSupportedInTransaction` (code 263) before session or transaction creation, maintenance gates, or mutation. Live logical sessions remain resumable across socket disconnects. Bounded background cleanup aborts expired transactions, removes expired sessions, and releases publication leases; explicit `endSessions` and `killSessions` do the same for their active transactions.
 
 ### Durable Database Publication Holds
 
@@ -497,6 +497,7 @@ await client.db('admin').command({ usersInfo: 'reader' });
 | `port` | `number` | Actual bound port while running; otherwise the configured port (TCP mode) |
 | `host` | `string` | Configured host (TCP mode) |
 | `socketPath` | `string \| undefined` | Socket path (socket mode) |
+| `processId` | `number \| undefined` | Operating-system pid of the spawned Rust engine while it runs as a child process (for crash and kill tests) |
 | `getMetrics()` | `Promise<ISmartDbMetrics>` | Server metrics (db/collection counts, sessions, transactions, auth, uptime) |
 | `getOpLog(params?)` | `Promise<IOpLogResult>` | Query oplog entries with optional filters |
 | `getOpLogStats()` | `Promise<IOpLogStats>` | Aggregate oplog statistics |
@@ -999,22 +1000,56 @@ than a claim about exact allocator heap usage.
 
 ### Read and Write Concerns and Wire Deadlines
 
-SmartDB 5.x does not implement MongoDB `readConcern` or `writeConcern`
-semantics. A command containing either top-level field is rejected with
-`InvalidOptions` (code 72) before authentication, session creation, transaction
-preparation, publication-gate acquisition, or database mutation. Callers must not
-interpret a successful SmartDB 5.x response as a majority or replica-backed
-acknowledgement. The exact `writeConcern: { w: 0 }` envelope used by the official
-driver for `endSessions`, and the equivalent envelope on `killSessions`, are
-accepted only to preserve best-effort session cleanup; neither is a database-write
-acknowledgement.
-
-SmartDB 5.x also does not enforce MongoDB wire `maxTimeMS` or `maxCommitTimeMS`
-deadlines. A command containing either top-level field is rejected at the same
-early boundary with `InvalidOptions` (code 72). The official driver transmits a
-transaction's `maxCommitTimeMS` as `maxTimeMS` on `commitTransaction`, which is
-therefore rejected rather than silently ignored. These wire fields are distinct
-from the fail-stop `timeoutMs` option on SmartDB's TypeScript management methods.
+SmartDB serves every database from exactly one voting node and states its
+concern profile in those terms instead of ignoring or blanket-rejecting the
+fields:
+
+- **Write concern.** Every acknowledged write is fsynced to the write-ahead log
+  and the data file before the response is sent, so `w: 1`, `w: "majority"`
+  (the majority of one voter is that voter), and `j: true` are all satisfied by
+  the same durable commit. `w: 0` is acknowledged the same way whenever a
+  response is expected; the exact `writeConcern: { w: 0 }` `endSessions` and
+  `killSessions` envelopes still run without a transport response. `wtimeout`
+  bounds a replication wait that does not exist on one node and has no
+  additional effect. `w: N` for `N > 1` fails with `UnsatisfiableWriteConcern`
+  (code 100), tag-set modes fail with `UnknownReplWriteConcern` (code 79), and
+  `j: true` fails with `BadValue` (code 2) on the in-memory storage backend,
+  which does not journal. Write concern is accepted only on commands that
+  write (including DDL, user management, `commitTransaction`, and
+  `abortTransaction`) and is rejected with `InvalidOptions` (code 72) inside a
+  multi-statement transaction, where it belongs on the commit.
+- **Read concern.** `local`, `available`, `majority`, and `linearizable` all
+  observe the durably committed state, because nothing is acknowledged before
+  it is durable and reads never observe uncommitted transaction buffers.
+  `snapshot` is accepted only on the first statement of a multi-document
+  transaction and is served by the database-wide transaction snapshot
+  described below. Read concern is accepted on `find`, `count`, `distinct`,
+  and `aggregate` only, and `afterClusterTime`, `atClusterTime`, and
+  `afterOpTime` are rejected with `InvalidOptions` because a single node issues
+  no cluster time.
+- **Deadlines.** `maxTimeMS` attaches a cooperative deadline to the command.
+  It is checked at admission, during namespace-lock and maintenance-gate
+  waits, before `find`, `count`, and `distinct` materialize documents, between
+  documents in `insert` and `delete` batches, and immediately before a
+  transaction commit is handed to storage; expiry fails with `MaxTimeMSExpired`
+  (code 50). Database-permit waits keep their fixed internal timeout, and a
+  single storage step or an in-progress `update` batch runs to completion
+  before the next check, so a write that expires after its first documents
+  were published reports the expiry while those documents remain durable,
+  exactly like an interrupted MongoDB batch. The official driver transmits a
+  transaction's `maxCommitTimeMS` as `maxTimeMS` on `commitTransaction`; a
+  commit that expires before its batch is handed to storage publishes nothing,
+  records the transaction as aborted, and carries the
+  `UnknownTransactionCommitResult` label; the driver's commit retry then sees
+  `NoSuchTransaction` with `TransientTransactionError` and restarts the whole
+  transaction. `getMore` rejects `maxTimeMS` with `BadValue` because SmartDB
+  has no awaitData cursors, and a literal `maxCommitTimeMS` field is rejected
+  with `InvalidOptions`.
+
+Callers may therefore rely on a successful SmartDB response as a durable,
+single-node acknowledgement. They must not read it as a replica-backed
+acknowledgement: majority semantics across several voters arrive with
+replication.
 
 ### Unsupported Administrative Commands
 
@@ -1034,7 +1069,7 @@ available and reports SmartDB's fixed disabled state.
 
 ## Rust Crate Architecture 🦀
 
-The Rust engine is organized as a Cargo workspace with 11 focused crates:
+The Rust engine is organized as a Cargo workspace with 12 focused crates:
 
 | Crate | Purpose |
 |---|---|
@@ -1043,6 +1078,7 @@ The Rust engine is organized as a Cargo workspace with 11 focused crates:
 | `rustdb-wire` | Wire protocol parser/encoder (OP_MSG, OP_QUERY, OP_REPLY) |
 | `rustdb-query` | Query matcher, update engine, aggregation, sort, projection |
 | `rustdb-state` | Internal deterministic Storage VNext commands, stable retry identities, and durable receipts |
+| `rustdb-kernel` | Internal non-serving Storage VNext committed-log kernel with stable timelines, contiguous LSNs, strict recovery, and exact receipt replay |
 | `rustdb-replication` | Internal non-serving OpenRaft 0.9.25 one-voter fresh-root foundation; not linked into the released `rustdb` server |
 | `rustdb-storage` | Storage backends (memory, file), OpLog with point-in-time replay |
 | `rustdb-index` | B-tree/hash indexes, query planner (IXSCAN/COLLSCAN) |
@@ -1050,7 +1086,7 @@ The Rust engine is organized as a Cargo workspace with 11 focused crates:
 | `rustdb-auth` | SCRAM-SHA-256 credential handling, user metadata persistence, RBAC checks |
 | `rustdb-commands` | 40+ command handlers wiring everything together |
 
-Storage VNext currently opens a fresh root only on Linux, using descriptor-relative access, checks for a supported local filesystem, and dual parent-slot/in-root leases. Stable retry identities resolve to durable receipts, including capacity-race rejection, with crash/reopen and post-quiescence snapshot/log proof. This internal foundation has no wire concerns, migration, serving listener, multi-voter transport, or HA claim. macOS and other non-Linux VNext root opens fail closed for now; existing released legacy binaries remain cross-compiled for the targets below.
+Storage VNext currently opens a fresh root only on Linux, using descriptor-relative access, checks for a supported local filesystem, and dual parent-slot/in-root leases. Its fixture-only A0 kernel appends checksummed prepared/commit-marker pairs to one bounded SmartDB-owned segment, assigns one stable timeline and contiguous LSNs, returns exact retry receipts without another append, and rebuilds materialized state before exposing a reopened root. Ambiguous append results fence the live handle; recovery syncs complete pairs before exposing them, repairs only a valid incomplete final pair, and rejects corruption or foreign roots unchanged. The separate one-voter OpenRaft foundation retains its capacity-race, crash/reopen, and snapshot/log proof, but is not yet integrated with the A0 commit kernel. These internal foundations have no BSON document substrate, MVCC, checkpoints, segment rollover, wire concerns, migration, serving listener, multi-voter transport, or HA claim. macOS and other non-Linux VNext root opens fail closed for now; existing released legacy binaries remain cross-compiled for the targets below.
 
 Cross-compiled for `linux_amd64`, `linux_arm64`, `macos_amd64`, and `macos_arm64` via [@git.zone/tsrust](https://www.npmjs.com/package/@git.zone/tsrust).
 

package/readme.plan.md

@@ -3,7 +3,7 @@
 Status: canonical direction, approved 2026-08-17. Reread before starting engine or
 managed-service work.
 
-Baseline: `@push.rocks/smartdb` 5.3.1. SmartDB is a Rust database engine with a
+Baseline: `@push.rocks/smartdb` 5.5.0. SmartDB is a Rust database engine with a
 TypeScript lifecycle facade, a MongoDB wire-protocol command surface, file and
 memory storage, authentication, transactions, resource fencing, and official
 MongoDB Node.js driver integration.
@@ -16,9 +16,24 @@ Current execution status:
   legacy hint contents, streams CRC validation, and repairs only an incomplete
   final record-header prefix
 - strict latest-per-key runtime WAL recovery and post-publication point-write
-  fencing are implemented in the current Pending 5.x maintenance slice; durable
-  retry receipts, crash-safe ordinary DDL, debug-server security, and general
-  crash/soak qualification remain separate required slices
+  fencing were released in 5.3.2; crash-safe ordinary DDL, debug-server security,
+  and general crash/soak qualification remain separate required 5.x slices
+- 5.4.0 released the non-serving, fixture-only A0 committed-log kernel with one
+  static timeline, contiguous LSNs, exact retry receipts, bounded
+  change-proportional appends, strict pair recovery, and ambiguity fencing; the
+  follow-up A1/MVCC/rollover/checkpoint work is parked unreleased on the
+  `storage-vnext-a1-wip` branch
+- A0 has no BSON document substrate, MVCC, checkpoint, segment rollover, OpenRaft
+  integration, wire serving, migration, or availability claim; Phase 3B and the
+  earlier Phase 0/1 gates have not passed
+- 5.5.0 serves the truthful single-node concern profile on the legacy engine:
+  fsync-backed `w: 1`/`majority`/`j` acknowledgement, committed-state reads for
+  `local`/`available`/`majority`/`linearizable`, database-wide transaction
+  snapshots pinned by the namespace publication clock with fail-closed
+  `SnapshotUnavailable` retries, cooperative `maxTimeMS` deadlines, and
+  per-session idempotent commit/abort outcomes; unsatisfiable `w: N > 1`, tag
+  sets, cluster-time read concerns, and journaled writes on the memory backend
+  remain rejected
 
 ## Product objective
 
@@ -131,7 +146,10 @@ and reproducible baseline measurements.
 ## Phase 1: truthful single-node behavior
 
 - stop over-advertising MongoDB version, wire behavior, and batch limits
-- reject ignored concerns, options, commands, and successful administrative stubs
+- serve the single-node concern profile truthfully and reject every concern,
+  option, command, or administrative stub whose guarantee one node cannot
+  provide (completed 2026-09-02: concerns, deadlines, database-wide transaction
+  snapshots, idempotent transaction outcomes)
 - completed slice (2026-08-25): validate OP_MSG required flags, CRC-32C
   checksums, bounded framing, and exact no-response admission and execution
 - preserve complete BSON type and value identity for document and index keys
@@ -214,10 +232,12 @@ Gates:
 - add configurable service classes and fair scheduling
 - add only features selected by the certified compatibility profile
 
-Semantic work may proceed after Phase 2. Retryable outcomes, transaction publication,
-and durable index publication depend on Storage VNext. Unsupported local concerns
-remain rejected until Storage VNext implements them; majority concerns remain
-rejected until replication implements them.
+Semantic work may proceed after Phase 2. Crash-durable retryable outcomes,
+transaction publication, and durable index publication move onto Storage VNext.
+The single-node concern profile is served today: every acknowledged write is
+durable before the response, reads observe committed state, and transactions
+observe one database-wide snapshot. Multi-voter majority acknowledgement and
+cluster-time read concerns remain rejected until replication implements them.
 
 Gate: bounded-memory scans at maximum declared scale, published p99 latency and
 capacity curves, proven tenant isolation, and zero certified-profile mismatches.
@@ -251,7 +271,8 @@ logged state.
 - replicate the Storage VNext log across three replicas in separate failure domains
 - add terms, quorum commit, snapshot installation, log catch-up, and membership
   changes
-- implement truthful local and majority concerns
+- extend the single-node concern profile to multi-voter majority acknowledgement
+  and cluster-time read concerns
 - publish accurate `hello` topology and election metadata
 - add causal times, retryable outcomes, idempotent transaction commits, and
   failover-safe session behavior

package/third-party-notices.md

@@ -3,7 +3,7 @@
 The Rust executables distributed with this package include
 third-party components. This document covers the complete third-party package
 set recorded by `rust/Cargo.lock`, including target-inactive and compile-time
-components. The eleven `rustdb*` workspace crates are covered by the package's
+components. The twelve `rustdb*` workspace crates are covered by the package's
 own MIT license and are not repeated below.
 
 ## Component Inventory

package/ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdb',
-  version: '5.3.2',
+  version: '5.5.0',
   description: 'A MongoDB-compatible embedded database server with wire protocol support, backed by a high-performance Rust engine.'
 }

package/ts/ts_smartdb/rust-db-bridge.ts

@@ -389,6 +389,18 @@ export function getRustDbAllocatorEnv(
  */
 export class RustDbBridge extends EventEmitter {
   private bridge: plugins.smartrust.RustBridge<TSmartDbCommands>;
+
+  /**
+   * Operating-system process id of the spawned Rust engine while it runs as a
+   * child process of this Node.js process. Undefined before spawn, after
+   * termination, or when the bridge is connected to an external process.
+   */
+  public get processId(): number | undefined {
+    const transport = (this.bridge as unknown as {
+      transport?: { childProcess?: { pid?: number } | null };
+    }).transport;
+    return transport?.childProcess?.pid;
+  }
   private terminationConfirmed = true;
   private terminationPromise: Promise<void> | undefined;
   private terminationFailure: unknown;

package/ts/ts_smartdb/server/SmartdbServer.ts

@@ -387,6 +387,11 @@ export class SmartdbServer {
   /**
    * Check if the server is running
    */
+  /** Process id of the spawned Rust engine, when it runs as a child process. */
+  get processId(): number | undefined {
+    return this.bridge.processId;
+  }
+
   get running(): boolean {
     return this.isRunning;
   }