@lossless.org/nosqldb 8.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.smartconfig.json +107 -0
- package/dist_rust/rustdb_linux_amd64 +0 -0
- package/dist_rust/rustdb_linux_amd64.tsrust-build.json +14 -0
- package/dist_rust/rustdb_linux_arm64 +0 -0
- package/dist_rust/rustdb_linux_arm64.tsrust-build.json +14 -0
- package/dist_rust/rustdb_macos_amd64 +0 -0
- package/dist_rust/rustdb_macos_amd64.tsrust-build.json +14 -0
- package/dist_rust/rustdb_macos_arm64 +0 -0
- package/dist_rust/rustdb_macos_arm64.tsrust-build.json +14 -0
- package/dist_ts/00_commitinfo_data.d.ts +8 -0
- package/dist_ts/00_commitinfo_data.js +9 -0
- package/dist_ts/index.d.ts +8 -0
- package/dist_ts/index.js +10 -0
- package/dist_ts/ts_local/classes.localnosqldb.d.ts +119 -0
- package/dist_ts/ts_local/classes.localnosqldb.js +386 -0
- package/dist_ts/ts_local/index.d.ts +5 -0
- package/dist_ts/ts_local/index.js +3 -0
- package/dist_ts/ts_local/plugins.d.ts +1 -0
- package/dist_ts/ts_local/plugins.js +3 -0
- package/dist_ts/ts_nosqldb/classes.storagerehearsal.d.ts +19 -0
- package/dist_ts/ts_nosqldb/classes.storagerehearsal.js +122 -0
- package/dist_ts/ts_nosqldb/file-storage-options.d.ts +13 -0
- package/dist_ts/ts_nosqldb/file-storage-options.js +24 -0
- package/dist_ts/ts_nosqldb/file-storage-startup.d.ts +3 -0
- package/dist_ts/ts_nosqldb/file-storage-startup.js +6 -0
- package/dist_ts/ts_nosqldb/index.d.ts +14 -0
- package/dist_ts/ts_nosqldb/index.js +11 -0
- package/dist_ts/ts_nosqldb/offline-inspection.d.ts +72 -0
- package/dist_ts/ts_nosqldb/offline-inspection.js +282 -0
- package/dist_ts/ts_nosqldb/plugins.d.ts +7 -0
- package/dist_ts/ts_nosqldb/plugins.js +10 -0
- package/dist_ts/ts_nosqldb/resource-fencing.d.ts +39 -0
- package/dist_ts/ts_nosqldb/resource-fencing.js +271 -0
- package/dist_ts/ts_nosqldb/rust-db-bridge.d.ts +213 -0
- package/dist_ts/ts_nosqldb/rust-db-bridge.js +439 -0
- package/dist_ts/ts_nosqldb/server/classes.nosqldbserver.d.ts +204 -0
- package/dist_ts/ts_nosqldb/server/classes.nosqldbserver.js +456 -0
- package/dist_ts/ts_nosqldb/server/index.d.ts +3 -0
- package/dist_ts/ts_nosqldb/server/index.js +3 -0
- package/dist_ts/ts_nosqldb/service-types.d.ts +341 -0
- package/dist_ts/ts_nosqldb/service-types.js +62 -0
- package/dist_ts/ts_nosqldb/storage-rehearsal.d.ts +103 -0
- package/dist_ts/ts_nosqldb/storage-rehearsal.js +131 -0
- package/dist_ts/ts_nosqldb/storage-root-relocation.d.ts +27 -0
- package/dist_ts/ts_nosqldb/storage-root-relocation.js +211 -0
- package/dist_ts_debugserver/bundled.d.ts +4 -0
- package/dist_ts_debugserver/bundled.js +16 -0
- package/dist_ts_debugserver/classes.debugserver.d.ts +36 -0
- package/dist_ts_debugserver/classes.debugserver.js +95 -0
- package/dist_ts_debugserver/index.d.ts +2 -0
- package/dist_ts_debugserver/index.js +2 -0
- package/dist_ts_debugserver/plugins.d.ts +2 -0
- package/dist_ts_debugserver/plugins.js +3 -0
- package/dist_ts_debugui/index.d.ts +2 -0
- package/dist_ts_debugui/index.js +2 -0
- package/dist_ts_debugui/nosqldb-debugui.d.ts +62 -0
- package/dist_ts_debugui/nosqldb-debugui.js +1132 -0
- package/dist_ts_debugui/plugins.d.ts +1 -0
- package/dist_ts_debugui/plugins.js +2 -0
- package/license +21 -0
- package/package.json +82 -0
- package/readme.md +1733 -0
- package/third-party-notices.md +925 -0
- package/ts/00_commitinfo_data.ts +8 -0
- package/ts/index.ts +85 -0
- package/ts/ts_local/classes.localnosqldb.ts +500 -0
- package/ts/ts_local/index.ts +19 -0
- package/ts/ts_local/plugins.ts +1 -0
- package/ts/ts_nosqldb/classes.storagerehearsal.ts +109 -0
- package/ts/ts_nosqldb/file-storage-options.ts +33 -0
- package/ts/ts_nosqldb/file-storage-startup.ts +9 -0
- package/ts/ts_nosqldb/index.ts +104 -0
- package/ts/ts_nosqldb/offline-inspection.ts +375 -0
- package/ts/ts_nosqldb/plugins.ts +12 -0
- package/ts/ts_nosqldb/resource-fencing.ts +368 -0
- package/ts/ts_nosqldb/rust-db-bridge.ts +964 -0
- package/ts/ts_nosqldb/server/classes.nosqldbserver.ts +696 -0
- package/ts/ts_nosqldb/server/index.ts +4 -0
- package/ts/ts_nosqldb/service-types.ts +484 -0
- package/ts/ts_nosqldb/storage-rehearsal.ts +215 -0
- package/ts/ts_nosqldb/storage-root-relocation.ts +277 -0
package/readme.md
ADDED
|
@@ -0,0 +1,1733 @@
|
|
|
1
|
+
# @lossless.org/nosqldb
|
|
2
|
+
|
|
3
|
+
A MongoDB-wire-compatible embedded database server powered by a native Rust MVCC engine. It supports the documented command surface through the official `mongodb` driver without an external MongoDB server. Memory and file storage use the same paged engine, including the operation log, point-in-time revert and debug dashboard. Matching executables are bundled with the package.
|
|
4
|
+
|
|
5
|
+
## Issue Reporting and Security
|
|
6
|
+
|
|
7
|
+
For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
|
|
8
|
+
|
|
9
|
+
## Install
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pnpm add @lossless.org/nosqldb
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Moving from @push.rocks/smartdb
|
|
16
|
+
|
|
17
|
+
The module has moved to `@lossless.org/nosqldb`. Install the new package and update imports and product-specific names together:
|
|
18
|
+
|
|
19
|
+
| Previous name | New name |
|
|
20
|
+
| --- | --- |
|
|
21
|
+
| `@push.rocks/smartdb` | `@lossless.org/nosqldb` |
|
|
22
|
+
| `SmartdbServer` | `NoSqlDbServer` |
|
|
23
|
+
| `LocalSmartDb` | `LocalNoSqlDb` |
|
|
24
|
+
| `SmartdbStorageRehearsal` | `NoSqlDbStorageRehearsal` |
|
|
25
|
+
| `SmartdbDebugServer` / `SmartdbDebugUi` | `NoSqlDbDebugServer` / `NoSqlDbDebugUi` |
|
|
26
|
+
| `ISmartdb*`, `ISmartDb*`, `TSmartdb*`, `TSmartDb*` | `INoSqlDb*`, `TNoSqlDb*` |
|
|
27
|
+
| `createSmartDb*`, `normalizeSmartDb*`, `smartDb*` | `createNoSqlDb*`, `normalizeNoSqlDb*`, `noSqlDb*` |
|
|
28
|
+
| `SMARTDB_*` environment variables | `NOSQLDB_*` environment variables |
|
|
29
|
+
| `<smartdb-debugui>` / `/api/smartdb/*` | `<nosqldb-debugui>` / `/api/nosqldb/*` |
|
|
30
|
+
| Benchmark `--engine smartdb` | Benchmark `--engine nosqldb` |
|
|
31
|
+
|
|
32
|
+
The `./debugui` and `./debugserver` package subpaths stay the same. The new module exports the new API names; it does not export aliases for the previous names. Update explicit binary overrides to `NOSQLDB_RUST_BINARY`, and rename the release-builder environment variables shown below.
|
|
33
|
+
|
|
34
|
+
This is a package and API rename, not a storage-format conversion. Existing current-format v7 databases, authentication records, allocation identities, export formats, and operation receipts retain their original byte formats. Versioned `smartdb.*` format identifiers, the `smartdb` allocation provider, hash domains, and private locking/workspace names deliberately remain unchanged. Do not edit those values in stored data or rewrite existing receipts. The native `rustdb` executable, `rustdb_*` packaged artifact names, and `.__rustdb_*` storage directories also keep their existing identities. Earlier incompatible storage formats remain subject to the admission restrictions documented below.
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## What It Does
|
|
39
|
+
|
|
40
|
+
`@lossless.org/nosqldb` is a **real database server** that speaks the wire protocol used by MongoDB drivers. The core engine is written in Rust for high performance, with a thin TypeScript orchestration layer. Connect with the standard `mongodb` Node.js driver — no mocks, no stubs, no external binaries required.
|
|
41
|
+
|
|
42
|
+
### Why NoSQLDB?
|
|
43
|
+
|
|
44
|
+
| | NoSQLDB | External DB Server |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| **Startup** | Managed by the application | Managed separately |
|
|
47
|
+
| **Binary distribution** | Matching executables bundled in the package | Separate installation |
|
|
48
|
+
| **Install** | `pnpm add` | System package / Docker |
|
|
49
|
+
| **Persistence** | Memory or file-based | Full disk engine |
|
|
50
|
+
| **Debug UI** | Built in | External tooling |
|
|
51
|
+
| **Point-in-time revert** | Built in | Depends on the database |
|
|
52
|
+
| **Perfect for** | Unit tests, CI/CD, prototyping, local dev, embedded | Production at scale |
|
|
53
|
+
|
|
54
|
+
### Three Ways to Use It
|
|
55
|
+
|
|
56
|
+
- 🎯 **`LocalNoSqlDb`** — Zero-config convenience. Give it a folder path, get a persistent database over a Unix socket. Done.
|
|
57
|
+
- 🏗️ **`NoSqlDbServer`** — Full control. Configure port, host, storage backend, Unix sockets. Great for test fixtures or custom setups.
|
|
58
|
+
- 🖥️ **`NoSqlDbDebugServer`** — Launch a web dashboard to visually browse collections, inspect the operation log, and revert to any point in time.
|
|
59
|
+
|
|
60
|
+
### Architecture: TypeScript + Rust 🦀
|
|
61
|
+
|
|
62
|
+
NoSQLDB uses a **sidecar binary** pattern — TypeScript handles lifecycle, Rust handles all database operations:
|
|
63
|
+
|
|
64
|
+
```
|
|
65
|
+
┌──────────────────────────────────────────────────────────────┐
|
|
66
|
+
│ Your Application │
|
|
67
|
+
│ (TypeScript / Node.js) │
|
|
68
|
+
│ ┌──────────────────┐ ┌───────────────────────────┐ │
|
|
69
|
+
│ │ NoSqlDbServer │─────▶│ RustDbBridge (IPC) │ │
|
|
70
|
+
│ │ or LocalNoSqlDb │ │ @push.rocks/smartrust │ │
|
|
71
|
+
│ └──────────────────┘ └───────────┬───────────────┘ │
|
|
72
|
+
└────────────────────────────────────────┼─────────────────────┘
|
|
73
|
+
│ spawn + JSON IPC
|
|
74
|
+
▼
|
|
75
|
+
┌──────────────────────────────────────────────────────────────┐
|
|
76
|
+
│ rustdb binary │
|
|
77
|
+
│ │
|
|
78
|
+
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────┐ │
|
|
79
|
+
│ │ Wire Protocol│→ │Command Router│→ │ Handlers │ │
|
|
80
|
+
│ │ (OP_MSG) │ │ (40+ cmds) │ │ Find,Insert.. │ │
|
|
81
|
+
│ └──────────────┘ └──────────────┘ └───────┬───────┘ │
|
|
82
|
+
│ │ │
|
|
83
|
+
│ ┌─────────┐ ┌────────┐ ┌───────────┐ ┌──────┴──────┐ │
|
|
84
|
+
│ │ Query │ │ Update │ │Aggregation│ │ Index │ │
|
|
85
|
+
│ │ Matcher │ │ Engine │ │ Engine │ │ Engine │ │
|
|
86
|
+
│ └─────────┘ └────────┘ └───────────┘ └─────────────┘ │
|
|
87
|
+
│ │
|
|
88
|
+
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────┐ │
|
|
89
|
+
│ │ Native MVCC pages│ │ Memory/file I/O │ │ OpLog │ │
|
|
90
|
+
│ └──────────────────┘ └──────────────────┘ └──────────┘ │
|
|
91
|
+
└──────────────────────────────────────────────────────────────┘
|
|
92
|
+
▲
|
|
93
|
+
│ TCP / Unix Socket (wire protocol)
|
|
94
|
+
│
|
|
95
|
+
┌─────────────┴────────────────────────────────────────────────┐
|
|
96
|
+
│ MongoClient (mongodb npm driver) │
|
|
97
|
+
│ Connects directly to Rust binary │
|
|
98
|
+
└──────────────────────────────────────────────────────────────┘
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
The TypeScript layer handles **lifecycle only** (start/stop/configure via IPC). All database operations flow directly from the `MongoClient` to the Rust binary over TCP or Unix sockets — **zero per-query IPC overhead**.
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## Quick Start
|
|
106
|
+
|
|
107
|
+
### Option 1: LocalNoSqlDb (Zero Config) 🎯
|
|
108
|
+
|
|
109
|
+
The fastest way to get a persistent local database:
|
|
110
|
+
|
|
111
|
+
```typescript
|
|
112
|
+
import { LocalNoSqlDb } from '@lossless.org/nosqldb';
|
|
113
|
+
import { MongoClient } from 'mongodb';
|
|
114
|
+
|
|
115
|
+
// Point it at a folder — that's it
|
|
116
|
+
const db = new LocalNoSqlDb({ folderPath: './my-data' });
|
|
117
|
+
const { connectionUri } = await db.start();
|
|
118
|
+
|
|
119
|
+
// Connect with the standard driver
|
|
120
|
+
const client = new MongoClient(connectionUri, { directConnection: true });
|
|
121
|
+
await client.connect();
|
|
122
|
+
|
|
123
|
+
// Use it like any wire-protocol-compatible database
|
|
124
|
+
const users = client.db('myapp').collection('users');
|
|
125
|
+
await users.insertOne({ name: 'Alice', email: 'alice@example.com' });
|
|
126
|
+
const user = await users.findOne({ name: 'Alice' });
|
|
127
|
+
console.log(user); // { _id: ObjectId(...), name: 'Alice', email: 'alice@example.com' }
|
|
128
|
+
|
|
129
|
+
// Data persists to disk automatically — survives restarts!
|
|
130
|
+
await client.close();
|
|
131
|
+
await db.stop();
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### Option 2: NoSqlDbServer (Full Control) 🏗️
|
|
135
|
+
|
|
136
|
+
```typescript
|
|
137
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
138
|
+
import { MongoClient } from 'mongodb';
|
|
139
|
+
|
|
140
|
+
// TCP mode
|
|
141
|
+
const server = new NoSqlDbServer({ port: 27017 });
|
|
142
|
+
await server.start();
|
|
143
|
+
|
|
144
|
+
const client = new MongoClient('mongodb://127.0.0.1:27017');
|
|
145
|
+
await client.connect();
|
|
146
|
+
|
|
147
|
+
const db = client.db('myapp');
|
|
148
|
+
await db.collection('users').insertOne({ name: 'Alice', age: 30 });
|
|
149
|
+
const user = await db.collection('users').findOne({ name: 'Alice' });
|
|
150
|
+
|
|
151
|
+
await client.close();
|
|
152
|
+
await server.stop();
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
### Option 3: Debug Server (Visual Dashboard) 🖥️
|
|
156
|
+
|
|
157
|
+
Launch a web-based dashboard to inspect your database in real time:
|
|
158
|
+
|
|
159
|
+
`debugserver` and `debugui` are optional subpath exports. Install their
|
|
160
|
+
debug dependencies only when you use them:
|
|
161
|
+
|
|
162
|
+
```bash
|
|
163
|
+
pnpm add '@api.global/typedserver@^8' '@design.estate/dees-element@^2'
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
```typescript
|
|
167
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
168
|
+
import { NoSqlDbDebugServer } from '@lossless.org/nosqldb/debugserver';
|
|
169
|
+
|
|
170
|
+
const server = new NoSqlDbServer({ storage: 'memory' });
|
|
171
|
+
await server.start();
|
|
172
|
+
|
|
173
|
+
const debugServer = new NoSqlDbDebugServer(server, { port: 4000 });
|
|
174
|
+
await debugServer.start();
|
|
175
|
+
// Open http://localhost:4000 in your browser 🚀
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
The debug dashboard gives you:
|
|
179
|
+
- 📊 **Dashboard** — server status, uptime, database/collection counts, operation breakdown
|
|
180
|
+
- 📁 **Collection Browser** — browse databases, collections, and documents interactively
|
|
181
|
+
- 📝 **OpLog Timeline** — every insert, update, and delete with expandable field-level diffs
|
|
182
|
+
- ⏪ **Point-in-Time Revert** — select any oplog sequence, preview what will be undone, and execute
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
|
|
186
|
+
## 📝 Operation Log & Point-in-Time Revert
|
|
187
|
+
|
|
188
|
+
The native engine provides these APIs for both memory and file storage. Revert
|
|
189
|
+
publishes one atomic indexed data change and preserves authentication, fencing,
|
|
190
|
+
publication holds and durable retry history.
|
|
191
|
+
|
|
192
|
+
Every write operation (insert, update, delete) is automatically recorded in an in-memory **operation log (OpLog)** with full before/after document snapshots. The OpLog lives in RAM and resets on restart — it covers the current session only, and retention is bounded (default: 10,000 entries or 64 MiB, whichever is hit first; configurable via the `oplog` server option). Once a limit is exceeded, the oldest entries are evicted. This enables:
|
|
193
|
+
|
|
194
|
+
- **Change tracking** — see exactly what changed, when, and in which collection
|
|
195
|
+
- **Field-level diffs** — compare previous and new document states
|
|
196
|
+
- **Point-in-time revert** — undo operations back to any retained sequence number
|
|
197
|
+
- **Dry-run preview** — see what would be reverted before executing
|
|
198
|
+
|
|
199
|
+
### Programmatic OpLog API
|
|
200
|
+
|
|
201
|
+
```typescript
|
|
202
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
203
|
+
|
|
204
|
+
const server = new NoSqlDbServer({ port: 27017 });
|
|
205
|
+
await server.start();
|
|
206
|
+
|
|
207
|
+
// ... perform some CRUD operations via MongoClient ...
|
|
208
|
+
|
|
209
|
+
// Get oplog entries
|
|
210
|
+
const oplog = await server.getOpLog({ limit: 50 });
|
|
211
|
+
console.log(oplog.entries);
|
|
212
|
+
// [{ seq: 1, op: 'insert', db: 'myapp', collection: 'users', document: {...}, previousDocument: null }, ...]
|
|
213
|
+
|
|
214
|
+
// Get aggregate stats
|
|
215
|
+
const stats = await server.getOpLogStats();
|
|
216
|
+
console.log(stats);
|
|
217
|
+
// { currentSeq: 42, totalEntries: 42, oldestSeq: 1, approxBytes: 18342, entriesByOp: { insert: 20, update: 15, delete: 7 } }
|
|
218
|
+
|
|
219
|
+
// Preview a revert (dry run)
|
|
220
|
+
const preview = await server.revertToSeq(30, true);
|
|
221
|
+
console.log(`Would undo ${preview.reverted} operations`);
|
|
222
|
+
|
|
223
|
+
// Execute the revert — undoes all operations after seq 30
|
|
224
|
+
const result = await server.revertToSeq(30, false);
|
|
225
|
+
console.log(`Reverted ${result.reverted} operations`);
|
|
226
|
+
|
|
227
|
+
// Reverts can only reach back as far as retained oplog history —
|
|
228
|
+
// revertToSeq returns an error if the target sequence is older than the
|
|
229
|
+
// oldest retained entry (evicted by the retention limits).
|
|
230
|
+
|
|
231
|
+
// Browse collections programmatically
|
|
232
|
+
const collections = await server.getCollections();
|
|
233
|
+
const docs = await server.getDocuments('myapp', 'users', 50, 0);
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
### OpLog Entry Structure
|
|
237
|
+
|
|
238
|
+
Each entry contains:
|
|
239
|
+
|
|
240
|
+
| Field | Type | Description |
|
|
241
|
+
|---|---|---|
|
|
242
|
+
| `seq` | `number` | Monotonically increasing sequence number |
|
|
243
|
+
| `timestampMs` | `number` | Unix timestamp in milliseconds |
|
|
244
|
+
| `op` | `'insert' \| 'update' \| 'delete'` | Operation type |
|
|
245
|
+
| `db` | `string` | Database name |
|
|
246
|
+
| `collection` | `string` | Collection name |
|
|
247
|
+
| `documentId` | `string` | Document `_id` as hex string |
|
|
248
|
+
| `document` | `object \| null` | New document state (null for deletes) |
|
|
249
|
+
| `previousDocument` | `object \| null` | Previous document state (null for inserts) |
|
|
250
|
+
|
|
251
|
+
---
|
|
252
|
+
|
|
253
|
+
## ✋ No-Op Write Detection
|
|
254
|
+
|
|
255
|
+
The engine detects document rewrites that change nothing and skips them entirely — no storage write, no WAL append, no index update, and no oplog entry. A skipped rewrite still counts as matched (`matchedCount: 1`) but reports `modifiedCount: 0`.
|
|
256
|
+
|
|
257
|
+
Two classes are skipped:
|
|
258
|
+
|
|
259
|
+
- **Identical** — the post-image equals the stored document byte for byte.
|
|
260
|
+
- **Volatile-only** — the post-image differs only in top-level volatile metadata fields (currently `_updatedAt`, the field ODM layers such as `@push.rocks/smartdata` restamp on every save). The stored document is kept as-is, including its existing `_updatedAt`, so the timestamp means "last real change" rather than "last save call".
|
|
261
|
+
|
|
262
|
+
This makes periodic reconcile loops that re-save unchanged documents cost nothing at the engine level. A caller that needs a document to actually change must change a non-volatile field.
|
|
263
|
+
|
|
264
|
+
Counters are exposed through `serverStatus`:
|
|
265
|
+
|
|
266
|
+
```javascript
|
|
267
|
+
const status = await db.command({ serverStatus: 1 });
|
|
268
|
+
console.log(status.writes);
|
|
269
|
+
// {
|
|
270
|
+
// updatesWritten: 42, // rewrites that reached storage, index, and oplog
|
|
271
|
+
// noopSkipped: {
|
|
272
|
+
// identical: 1337, // post-image equal to the stored document
|
|
273
|
+
// volatileOnly: 271, // only volatile metadata differed
|
|
274
|
+
// },
|
|
275
|
+
// }
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
---
|
|
279
|
+
|
|
280
|
+
## API Reference
|
|
281
|
+
|
|
282
|
+
### NoSqlDbServer
|
|
283
|
+
|
|
284
|
+
The core server class. Manages the Rust database engine and exposes connection details.
|
|
285
|
+
|
|
286
|
+
#### Constructor Options (`INoSqlDbServerOptions`)
|
|
287
|
+
|
|
288
|
+
`binaryPath?: string` selects one exact engine executable for this server. When
|
|
289
|
+
supplied, Smartrust validates a regular executable file and never searches
|
|
290
|
+
`NOSQLDB_RUST_BINARY`, npm packages, development directories or `PATH`, even if
|
|
291
|
+
the selected path is unusable. Missing, empty, malformed, non-file and
|
|
292
|
+
non-executable paths reject startup with `RustBinaryLocatorError`, code
|
|
293
|
+
`ERR_RUST_BINARY_EXPLICIT_PATH_INVALID`. Validation never changes the target's
|
|
294
|
+
permissions. Executable launch or readiness failures also fail startup without
|
|
295
|
+
selecting another engine. Omitting the option preserves normal discovery.
|
|
296
|
+
|
|
297
|
+
Use an absolute path to the caller-installed, verified artifact. Smartrust
|
|
298
|
+
anchors relative paths when the underlying bridge is constructed, without
|
|
299
|
+
normalizing away filesystem symlink traversal. A bare name is a relative file
|
|
300
|
+
path, not a `PATH` lookup. Options are instance-local; no environment variables
|
|
301
|
+
are changed. Installer verification and protected installation paths remain the
|
|
302
|
+
caller's responsibility.
|
|
303
|
+
|
|
304
|
+
```typescript
|
|
305
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
306
|
+
|
|
307
|
+
const server = new NoSqlDbServer({
|
|
308
|
+
binaryPath: '/opt/spark/engines/nosqldb/rustdb_linux_amd64',
|
|
309
|
+
socketPath: '/run/spark/private/nosqldb.sock',
|
|
310
|
+
storage: 'file',
|
|
311
|
+
storagePath: '/var/lib/spark/nosqldb',
|
|
312
|
+
});
|
|
313
|
+
await server.start({ timeoutMs: 30_000 });
|
|
314
|
+
try {
|
|
315
|
+
const connectionUri = server.getConnectionUri();
|
|
316
|
+
// Pass connectionUri to SmartData for application persistence.
|
|
317
|
+
} finally {
|
|
318
|
+
await server.stop(); // Resolves after the owned engine has terminated.
|
|
319
|
+
}
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
The socket parent directory must already exist with the caller's desired access
|
|
323
|
+
permissions. `RustDbBridge` also accepts `{ binaryPath }` for direct lifecycle use.
|
|
324
|
+
|
|
325
|
+
#### Native memory and file storage
|
|
326
|
+
|
|
327
|
+
The native MVCC engine is the sole serving engine on Linux and macOS.
|
|
328
|
+
`storage: 'memory'` (the default) uses ephemeral native storage;
|
|
329
|
+
`storage: 'file'` persists the same record model under `storagePath`.
|
|
330
|
+
`binaryPath` selects the executable; it is independent of storage selection.
|
|
331
|
+
|
|
332
|
+
`fileStorageEngine`, `persistPath`, `persistIntervalMs` and
|
|
333
|
+
`auth.usersPath` have been removed. Supplying them rejects at construction,
|
|
334
|
+
before filesystem or process side effects. Authentication belongs to the native
|
|
335
|
+
database and persists with file storage. Memory storage is ephemeral.
|
|
336
|
+
|
|
337
|
+
```typescript
|
|
338
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
339
|
+
|
|
340
|
+
const server = new NoSqlDbServer({
|
|
341
|
+
binaryPath: '/opt/spark/engines/nosqldb/rustdb_linux_amd64',
|
|
342
|
+
socketPath: '/run/spark/private/nosqldb.sock',
|
|
343
|
+
storage: 'file',
|
|
344
|
+
storagePath: '/var/lib/spark/nosqldb',
|
|
345
|
+
auth: {
|
|
346
|
+
enabled: true,
|
|
347
|
+
scramIterations: 15000,
|
|
348
|
+
users: [{ database: 'app', username: 'app-owner',
|
|
349
|
+
password: bootstrapPassword, roles: ['readWrite'] }],
|
|
350
|
+
},
|
|
351
|
+
});
|
|
352
|
+
await server.start({ timeoutMs: 30_000 });
|
|
353
|
+
try {
|
|
354
|
+
const connectionUri = server.getConnectionUri();
|
|
355
|
+
// Connect SmartData with the app credentials after native ownership is acquired.
|
|
356
|
+
} finally {
|
|
357
|
+
await server.stop();
|
|
358
|
+
}
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
Supply `bootstrapPassword` from protected caller configuration. The work factor
|
|
362
|
+
above is an example, not a universal default. Reopen with the stored SCRAM work
|
|
363
|
+
factor; bootstrap users may be omitted after initialization. `LocalNoSqlDb`
|
|
364
|
+
accepts `folderPath`, `socketPath`, `binaryPath` and the same `auth` options.
|
|
365
|
+
|
|
366
|
+
#### Current-format-only file startup
|
|
367
|
+
|
|
368
|
+
`INoSqlDbServerOptions.fileStorageStartup` and
|
|
369
|
+
`ILocalNoSqlDbOptions.fileStorageStartup` accept only
|
|
370
|
+
`'current-format-only'`, which is also the default for file storage.
|
|
371
|
+
The exported type is `TNoSqlDbFileStorageStartup`. Invalid values, including
|
|
372
|
+
`'migrate'`, reject at construction. Selecting the policy with memory storage
|
|
373
|
+
also rejects.
|
|
374
|
+
|
|
375
|
+
Startup accepts a fresh root or a current native root. Foreign, old, mixed and
|
|
376
|
+
orphaned migration staging layouts are rejected without conversion. TypeScript
|
|
377
|
+
performs no storage writes before starting the engine. Native format admission
|
|
378
|
+
uses the owning detector; the exclusive native file-root lease is acquired before
|
|
379
|
+
initialization, recovery, authentication or index preparation.
|
|
380
|
+
|
|
381
|
+
Only after `await server.start()` succeeds should consumers prepare SmartData
|
|
382
|
+
models or run application migrations. Start cancellation and deadlines cover the
|
|
383
|
+
native operation; failed-start cleanup confirms child exit, and `stop()` remains
|
|
384
|
+
retryable if termination cannot yet be confirmed. A second owner of the same
|
|
385
|
+
root is rejected without disturbing the winner. Callers must also exclude
|
|
386
|
+
manual edits and lifecycle changes outside NoSQLDB's ownership protocol.
|
|
387
|
+
|
|
388
|
+
No automatic format or auth-permission migration runs during startup.
|
|
389
|
+
Persisted-format conversions belong exclusively in the top-level
|
|
390
|
+
`ts_migration/` folder and require their own explicit qualification. This
|
|
391
|
+
release contains no old-engine or authentication-schema conversion.
|
|
392
|
+
|
|
393
|
+
```typescript
|
|
394
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
395
|
+
|
|
396
|
+
// TCP mode (default)
|
|
397
|
+
const server = new NoSqlDbServer({
|
|
398
|
+
port: 0, // Default: 27017; 0 requests an OS-assigned port
|
|
399
|
+
host: '127.0.0.1', // Default: 127.0.0.1
|
|
400
|
+
storage: 'memory', // 'memory' or 'file' (default: 'memory')
|
|
401
|
+
storagePath: './data', // Required when storage is 'file'
|
|
402
|
+
});
|
|
403
|
+
const startupController = new AbortController();
|
|
404
|
+
await server.start({
|
|
405
|
+
signal: startupController.signal,
|
|
406
|
+
timeoutMs: 30_000,
|
|
407
|
+
});
|
|
408
|
+
console.log(server.port); // Actual bound port while running
|
|
409
|
+
console.log(server.getConnectionUri()); // mongodb://127.0.0.1:<actual-port>
|
|
410
|
+
|
|
411
|
+
// Unix socket mode — no port conflicts!
|
|
412
|
+
const server = new NoSqlDbServer({
|
|
413
|
+
socketPath: '/tmp/nosqldb.sock',
|
|
414
|
+
storage: 'file',
|
|
415
|
+
storagePath: './data',
|
|
416
|
+
});
|
|
417
|
+
|
|
418
|
+
// Bounded in-memory oplog retention
|
|
419
|
+
const server = new NoSqlDbServer({
|
|
420
|
+
port: 27017,
|
|
421
|
+
oplog: {
|
|
422
|
+
maxEntries: 50000, // Default: 10000
|
|
423
|
+
maxBytes: 128 * 1024 * 1024, // Default: 64 MiB
|
|
424
|
+
},
|
|
425
|
+
});
|
|
426
|
+
|
|
427
|
+
// TLS transport for TCP mode
|
|
428
|
+
const tlsServer = new NoSqlDbServer({
|
|
429
|
+
port: 27017,
|
|
430
|
+
tls: {
|
|
431
|
+
enabled: true,
|
|
432
|
+
certPath: './certs/server.pem',
|
|
433
|
+
keyPath: './certs/server.key',
|
|
434
|
+
// caPath: './certs/client-ca.pem',
|
|
435
|
+
// requireClientCert: true, // Enables mTLS client certificate checks
|
|
436
|
+
},
|
|
437
|
+
});
|
|
438
|
+
|
|
439
|
+
// SCRAM-SHA-256 authentication
|
|
440
|
+
const secureServer = new NoSqlDbServer({
|
|
441
|
+
port: 27017,
|
|
442
|
+
auth: {
|
|
443
|
+
enabled: true,
|
|
444
|
+
users: [
|
|
445
|
+
{
|
|
446
|
+
username: 'root',
|
|
447
|
+
password: 'change-me',
|
|
448
|
+
database: 'admin',
|
|
449
|
+
roles: ['root'],
|
|
450
|
+
},
|
|
451
|
+
],
|
|
452
|
+
},
|
|
453
|
+
});
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
When `auth.enabled` is true, protected commands require successful SCRAM-SHA-256 authentication through the official MongoDB driver:
|
|
457
|
+
|
|
458
|
+
```typescript
|
|
459
|
+
const client = new MongoClient('mongodb://root:change-me@127.0.0.1:27017/admin?authSource=admin', {
|
|
460
|
+
directConnection: true,
|
|
461
|
+
});
|
|
462
|
+
await client.connect();
|
|
463
|
+
```
|
|
464
|
+
|
|
465
|
+
TLS is available for TCP listeners. `getConnectionUri()` includes `?tls=true` when TLS is enabled; pass the trusted CA to the MongoDB driver with `tlsCAFile`, `ca`, or `secureContext`.
|
|
466
|
+
|
|
467
|
+
Authentication verifies SCRAM credentials, denies unauthenticated commands, and enforces command-level built-in roles for supported operations. `connectionStatus` reports the authenticated users and roles for the current socket.
|
|
468
|
+
|
|
469
|
+
Supported built-in role names are `root`, `read`, `readWrite`, `dbAdmin`,
|
|
470
|
+
`userAdmin`, `clusterMonitor`, `readAnyDatabase`, `readWriteAnyDatabase`,
|
|
471
|
+
`dbAdminAnyDatabase` and `userAdminAnyDatabase`. Native auth records contain
|
|
472
|
+
derived SCRAM credentials, protected decoy material, the immutable work factor,
|
|
473
|
+
principal identities, generations and bounded grant replay state. Plaintext
|
|
474
|
+
passwords are not persisted. `auth.scramIterations` must match the stored
|
|
475
|
+
profile on every reopen. Historical or malformed auth records fail closed.
|
|
476
|
+
|
|
477
|
+
Every authenticated command resolves the current principal and generation.
|
|
478
|
+
Password and role changes invalidate old sockets. Deleting and recreating a
|
|
479
|
+
username creates a different principal; an old connection cannot inherit its
|
|
480
|
+
authority. Auth changes and the data/control records they protect share native
|
|
481
|
+
atomic publications.
|
|
482
|
+
|
|
483
|
+
Single-node transactions pin one committed root across namespaces and retain
|
|
484
|
+
changed documents, rather than copying a database. Supported reads and writes
|
|
485
|
+
see the transaction snapshot plus its staged changes. Commit validates conflicts
|
|
486
|
+
and publishes atomically; abort discards staged changes. Write conflicts report
|
|
487
|
+
`WriteConflict` (112) with `TransientTransactionError`.
|
|
488
|
+
|
|
489
|
+
Commit/abort outcomes and successful retryable statement results survive file
|
|
490
|
+
storage restart. Duplicate commits replay their outcome; abort after commit
|
|
491
|
+
reports `TransactionCommitted` (256), commit after abort reports
|
|
492
|
+
`NoSuchTransaction` (251), and older transaction numbers report
|
|
493
|
+
`TransactionTooOld` (225). Unsupported transactional DDL, auth mutations and
|
|
494
|
+
`$out`/`$merge` reject before transaction admission or mutation.
|
|
495
|
+
|
|
496
|
+
Logical sessions survive socket disconnects. Expiry, `endSessions` and
|
|
497
|
+
`killSessions` drain active work and release cursors and namespace gates.
|
|
498
|
+
Rejected new transaction envelopes do not consume session capacity or reset
|
|
499
|
+
existing sessions. See the native resource budgets below.
|
|
500
|
+
|
|
501
|
+
### Durable Database Publication Holds
|
|
502
|
+
|
|
503
|
+
On supported Linux and macOS filesystems, file-backed NoSQLDB can keep a replaced or deleted database closed after the local mutation is durable and until a downstream coordinator confirms its own fsync. Check the explicit health contract before using this flow:
|
|
504
|
+
|
|
505
|
+
```typescript
|
|
506
|
+
const health = await server.getHealth();
|
|
507
|
+
if (
|
|
508
|
+
health.publicationHoldVersion !== 1 ||
|
|
509
|
+
!health.publicationHoldSupported ||
|
|
510
|
+
health.publicationHoldRequiresExternalDrain
|
|
511
|
+
) {
|
|
512
|
+
throw new Error('NoSQLDB publication holds are unavailable');
|
|
513
|
+
}
|
|
514
|
+
```
|
|
515
|
+
|
|
516
|
+
Set `holdPublication: true` on an exact fenced `importDatabase()` or `deleteDatabaseTenant()` operation. The result contains a held `resourceFence` with a one-time `publicationCapability`. Treat that capability as a secret: do not log it or persist it outside protected control-plane state. After downstream state is durable, submit the exact receipt to `commitDatabasePublication()`:
|
|
517
|
+
|
|
518
|
+
```typescript
|
|
519
|
+
import type { INoSqlDbHeldPublicationReceipt } from '@lossless.org/nosqldb';
|
|
520
|
+
|
|
521
|
+
const held = await server.importDatabase({
|
|
522
|
+
databaseName: 'tenant_a',
|
|
523
|
+
source: snapshot,
|
|
524
|
+
username: 'tenant_a_user',
|
|
525
|
+
fence: {
|
|
526
|
+
version: 1,
|
|
527
|
+
scopeId: 'corestore-node-1.tenant-a',
|
|
528
|
+
token: 42,
|
|
529
|
+
mutationId: 'restore-42',
|
|
530
|
+
payloadSha256: controlPlanePayloadSha256,
|
|
531
|
+
},
|
|
532
|
+
holdPublication: true,
|
|
533
|
+
});
|
|
534
|
+
|
|
535
|
+
const receipt = held.resourceFence as INoSqlDbHeldPublicationReceipt;
|
|
536
|
+
await persistAndFsyncDownstreamState(receipt);
|
|
537
|
+
const released = await server.commitDatabasePublication({
|
|
538
|
+
databaseName: 'tenant_a',
|
|
539
|
+
resourceFence: receipt,
|
|
540
|
+
});
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
The held barrier survives restart and blocks wire commands, transactions and index restoration for that database while unrelated databases remain available. Native physical compaction preserves the held records. Commit is exact and idempotent. A successful commit removes the raw capability from durable state and retains only protected verification material. Provider/root identity and bounded validation of native fencing records make copied, incomplete or corrupt allocation state fail closed. A higher fencing token compacts resolved older receipts, so long-lived coordinators do not exhaust receipt capacity.
|
|
544
|
+
|
|
545
|
+
Durable database publication holds and allocation fencing use native file storage on the supported Linux and macOS filesystems.
|
|
546
|
+
|
|
547
|
+
Coordinators that do not yet have their own durable record can call `getDatabaseResourceFenceState({ databaseName })` before creating it. File-backed NoSQLDB returns the durable scope, highest token, current publication phase, and allocation lifecycle/identity when the name is allocation-managed, or `null` for an unfenced name. Startup first validates both directions of every allocation's fencing/auth binding. The inspection never returns mutation IDs, mutation payload digests, publication receipts, or publication capabilities. Treat any active phase, identity mismatch, corrupt state, unsupported storage, or unsafe token as a hard stop rather than creating coordinator state.
|
|
548
|
+
|
|
549
|
+
### Authoritative Database Allocation
|
|
550
|
+
|
|
551
|
+
`allocateDatabaseTenant()` is the create-only control-plane API for permanently allocation-managed database names. It is available only when `allocationFencingVersion === 1`, `allocationFencingSupported === true`, and `allocationFencingRequiresDrain === true`. The request requires literal `expectedAbsent: true`, an `INoSqlDbResourceFence`, and the tenant database, username, and password. `roles` is optional and defaults to `['readWrite', 'dbAdmin']`. Under native ownership and database/control admission, NoSQLDB proves catalog and principal absence, then commits the catalog, principal, complete auth allocation binding and fencing receipt at one position. It returns an `INoSqlDbAllocateDatabaseTenantResult`; its `.allocation` field is the durable `INoSqlDbDatabaseAllocationIdentity`. Exact fence replay returns the same allocation ID, generation, principal ID, and receipt digests.
|
|
552
|
+
|
|
553
|
+
Pass the returned `allocation` identity to every later `ensureDatabaseTenant()`, `importDatabase()`, and `deleteDatabaseTenant()` request for that name. Managed import and delete requests also require the exact allocation `username`; the identity alone is insufficient. Each distinct managed mutation requires a strictly newer fence token; the current token is accepted only for exact receipt replay or continuation, and older receipts become explicitly stale after advancement. Held publication receipts carry the allocation into `commitDatabasePublication()`. Managed requests without it and unmanaged requests with it fail closed. Deprovision drains affected runtime authority, then atomically retires the catalog, exact principal, grants and auth binding with a permanent `deprovisioned` fencing tombstone. A later allocation requires a newer fence token, increments generation, and receives a fresh allocation and principal identity. Wire data access is available only while active and only to the current allocation principal; database/user ownership-changing wire commands are rejected.
|
|
554
|
+
|
|
555
|
+
Allocation bindings are native records, excluded from logical database exports and never accepted from import payloads. Startup and stopped-storage inspection require the complete matching fence and auth binding even when no read grant was ever issued. Missing bindings are rejected without reconstruction. This includes allocated roots produced by experimental native versions that did not create the complete binding. This release provides no automatic conversion or downgrade path; never remove or edit private metadata to force admission.
|
|
556
|
+
|
|
557
|
+
### Read-Only Tenant Attestation
|
|
558
|
+
|
|
559
|
+
`attestDatabaseTenant()` verifies that an existing database is exclusively owned by the exact username, role set, and candidate password. It returns only `{ matches: boolean }`; it never returns a URI, credential, principal identity, or mismatch detail. Missing databases or users, conflicting ownership, role differences, allocation-principal drift, and password mismatch all return `false` through the same constant-work credential path.
|
|
560
|
+
|
|
561
|
+
Attestation pins one native snapshot under the database's read-only admission. Its auth, allocation and data observations share that committed root. Attestation does not refresh process caches or call tenant ensure, create, rotate, import, delete, fence repair, publication-epoch observation, or cache invalidation. Corrupt, held, deprovisioned, disabled, or busy provider state fails as an operational error instead of returning a match result.
|
|
562
|
+
|
|
563
|
+
```typescript
|
|
564
|
+
const result = await server.attestDatabaseTenant({
|
|
565
|
+
databaseName: 'tenant_a',
|
|
566
|
+
username: 'tenant_a_user',
|
|
567
|
+
roles: ['readWrite'],
|
|
568
|
+
password: candidatePassword,
|
|
569
|
+
});
|
|
570
|
+
|
|
571
|
+
if (!result.matches) {
|
|
572
|
+
throw new Error('Existing tenant authority does not match');
|
|
573
|
+
}
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
### Allocation-Bound Read Access Grants
|
|
577
|
+
|
|
578
|
+
Scratch diagnostics can use `issueDatabaseAllocationReadAccessGrant()` to create one auxiliary read principal for an exact active allocation. The capability is advertised only when `databaseAllocationReadAccessGrantVersion === 1` and `databaseAllocationReadAccessGrantSupported === true`; this requires file storage, enabled authentication and ready allocation fencing. Native MVCC commits grant principals, authentication and allocation bindings atomically in the storage root. The caller supplies `databaseName`, a bounded `grantId`, a bounded unique `username`, `password`, and the complete `INoSqlDbDatabaseAllocationIdentity`. Roles and expiry are not caller-controlled. NoSQLDB always assigns exactly `['read']` and a fixed, non-renewing 12-minute expiry.
|
|
579
|
+
|
|
580
|
+
```typescript
|
|
581
|
+
const grant = await server.issueDatabaseAllocationReadAccessGrant({
|
|
582
|
+
databaseName: allocation.databaseName,
|
|
583
|
+
grantId: 'scratch-diagnostic-2026-08-08',
|
|
584
|
+
username: 'scratch_diagnostic_reader',
|
|
585
|
+
password: generatedSecret,
|
|
586
|
+
allocation,
|
|
587
|
+
});
|
|
588
|
+
|
|
589
|
+
const diagnosticClient = new MongoClient(grant.mongodbUri!, {
|
|
590
|
+
directConnection: true,
|
|
591
|
+
});
|
|
592
|
+
await diagnosticClient.connect();
|
|
593
|
+
```
|
|
594
|
+
|
|
595
|
+
Exact issue replay validates the candidate password against the persisted SCRAM credential and returns the original principal generation, `issuedAt`, and `expiresAt` only while that grant remains active and unexpired; it never renews the grant. Expired or revoked `grantId` values are rejected. NoSQLDB permits at most one active read grant per allocation. Revoked and expired `grantId` values remain in a per-allocation anti-replay set capped at 1,024 entries, with no eviction while the allocation exists. Capacity exhaustion fails closed. Allocation deprovision uses a two-phase fail-closed sequence: any active grant is first durably retired and tombstoned, then the database is durably deleted, and finally the exact owner plus that allocation's replay state are removed in one auth rewrite. Interruption before completion leaves the grant retired rather than usable.
|
|
596
|
+
|
|
597
|
+
Native deprovision also durably retires an active grant before draining its
|
|
598
|
+
transaction, then publishes database deletion, exact owner cleanup and its
|
|
599
|
+
fencing receipt together. Bounded periodic cleanup retires expired grant
|
|
600
|
+
principals and cancels their sessions and cursors.
|
|
601
|
+
|
|
602
|
+
Grant-specific rejections from the issue and revoke methods are normalized automatically as `NoSqlDbAllocationReadAccessGrantError` with stable `EGRANT_INVALID_REQUEST`, `EGRANT_CREDENTIAL_MISMATCH`, `EGRANT_CONFLICT`, `EGRANT_REPLAY_CAPACITY`, or `EGRANT_STATE_UNAVAILABLE` codes. The companion exports `TNoSqlDbAllocationReadAccessGrantErrorCode`, `noSqlDbAllocationReadAccessGrantErrorCodes`, and `normalizeNoSqlDbAllocationReadAccessGrantError(error, code)` support type-safe handling and explicit normalization of structured bridge errors. Allocation and resource-fence failures retain their existing `NoSqlDbResourceFenceError` codes. Callers should classify errors by these exported classes and codes rather than by message text.
|
|
603
|
+
|
|
604
|
+
Active allocation read grants never survive a NoSQLDB engine restart or a new attachment to the durable auth store. Startup durably retires and tombstones them before accepting work; diagnostics must issue a new `grantId`, username, and credential after restart.
|
|
605
|
+
|
|
606
|
+
Grant principals can read only their exact allocation database. Reads, collection/index metadata, and read-only transactions are supported. Writes, DDL, user/admin operations, writing aggregates (`$out` and `$merge`), cross-database targets, and allocation ownership changes are denied. Revocation, expiry cleanup, and allocation deprovision abort matching logical sessions and transactions, release retained database permits, and remove matching cursors. Stale sockets fail their next command.
|
|
607
|
+
|
|
608
|
+
`revokeDatabaseAllocationReadAccessGrant()` takes `databaseName`, `grantId`, and the complete allocation identity, returns `{ revoked: true }` without a secret, and is idempotent. Revoke-before-issue records anti-replay state only while a read permit proves that exact allocation is current. A delayed revoke after deprovision returns success without recreating auth metadata.
|
|
609
|
+
|
|
610
|
+
Native `exportDatabase()` pins a committed database root while ordinary writes
|
|
611
|
+
continue. Catalog, auth and publication checks use the same root; DDL remains
|
|
612
|
+
excluded by the database gate. Callers can lower `maxEncodedBytes`,
|
|
613
|
+
`maxCollections`, `maxDocuments` and `maxIndexes`; the server enforces these
|
|
614
|
+
bounds while scanning. Results use canonical MongoDB Extended JSON, preserving
|
|
615
|
+
every BSON type and numeric width.
|
|
616
|
+
|
|
617
|
+
`importDatabase()` validates canonical or relaxed Extended JSON, builds a private
|
|
618
|
+
native catalog in bounded commits, then atomically publishes the complete catalog
|
|
619
|
+
with its fencing receipt. Incomplete staging is never visible to data readers.
|
|
620
|
+
Cancellation or restart retires unpublished staging through the owning native
|
|
621
|
+
logic. Index aliases, empty namespaces and validated `system.views` definitions
|
|
622
|
+
are preserved; view IDs are rebound when the database name changes. Keep exported
|
|
623
|
+
objects as JSON values rather than coercing BSON numeric wrappers into JavaScript
|
|
624
|
+
numbers. These APIs transfer database contents; they do not convert storage files.
|
|
625
|
+
|
|
626
|
+
`getDatabaseContentDigest()` computes a bounded, database-name-independent SHA-256 over collection names, exact BSON document bytes, and persisted index specifications. Collection, document, and index enumeration order is normalized; BSON field order and compound-index key order remain significant. The physical `system.views` catalog is included, with only its already-validated deterministic database prefix removed from `_id` for hashing so a cross-name import retains the same digest. `INoSqlDbGetDatabaseContentDigestInput` accepts `databaseName` and optional `INoSqlDbDatabaseContentDigestLimits` fields `maxScannedBsonBytes`, `maxCollections`, `maxDocuments`, and `maxIndexes`. Callers may lower the server ceilings of 96 MiB scanned BSON, 10,000 collections, 1,000,000 documents, and 100,000 indexes. The `INoSqlDbDatabaseContentDigest` result reports format `smartdb.database.content-digest.v1`, algorithm `sha256`, the lowercase digest, and exact scan counters.
|
|
627
|
+
|
|
628
|
+
`NoSqlDbServer.start()` and the lifecycle-critical health, fence-state, digest, export, import, publication-commit, tenant-allocation, tenant-attestation, tenant-ensure, tenant-delete, read-grant issue, and read-grant revoke methods accept optional `INoSqlDbManagementOperationOptions` with `signal?: AbortSignal` and `timeoutMs?: number`. `timeoutMs`, when provided, must be a positive safe integer no greater than `2,147,483,647`. Startup applies both cancellation and one deadline across sidecar spawn, native admission, recovery and database readiness. Cancellation and deadlines are fail-stop once the sidecar is owned: NoSQLDB attempts to terminate the Rust engine before rejecting. Once termination is confirmed, the operation cannot continue after the caller releases ownership. If termination itself fails, the rejection retains bridge ownership and the service owner must retry `stop()` until it succeeds. After cancellation or deadline termination, restart NoSQLDB before accepting more traffic.
|
|
629
|
+
|
|
630
|
+
```typescript
|
|
631
|
+
const controller = new AbortController();
|
|
632
|
+
const exported = await server.exportDatabase(
|
|
633
|
+
{ databaseName: 'myapp' },
|
|
634
|
+
{ signal: controller.signal, timeoutMs: 5 * 60_000 },
|
|
635
|
+
);
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
Basic user management commands are available for authenticated users with `root` or `userAdmin` privileges:
|
|
639
|
+
|
|
640
|
+
```typescript
|
|
641
|
+
await client.db('admin').command({
|
|
642
|
+
createUser: 'reader',
|
|
643
|
+
pwd: 'readpass',
|
|
644
|
+
roles: [{ role: 'read', db: 'myapp' }],
|
|
645
|
+
});
|
|
646
|
+
|
|
647
|
+
await client.db('admin').command({ usersInfo: 'reader' });
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
#### Methods & Properties
|
|
651
|
+
|
|
652
|
+
| Method / Property | Type | Description |
|
|
653
|
+
|---|---|---|
|
|
654
|
+
| `start(options?)` | `Promise<void>` | Start under one optional cancellation/deadline budget; join partial sidecars before rejection or retain ownership for `stop()` retry if termination fails |
|
|
655
|
+
| `stop()` | `Promise<void>` | Wait for active startup work, then stop the server and confirm Rust bridge cleanup, including partial startup cleanup |
|
|
656
|
+
| `getConnectionUri()` | `string` | Get the active `mongodb://` URI; before start and after stop, port `0` remains unresolved |
|
|
657
|
+
| `running` | `boolean` | Whether the server is currently running |
|
|
658
|
+
| `port` | `number` | Actual bound port while running; otherwise the configured port (TCP mode) |
|
|
659
|
+
| `host` | `string` | Configured host (TCP mode) |
|
|
660
|
+
| `socketPath` | `string \| undefined` | Socket path (socket mode) |
|
|
661
|
+
| `processId` | `number \| undefined` | Operating-system pid of the spawned Rust engine while it runs as a child process (for crash and kill tests) |
|
|
662
|
+
| `getMetrics()` | `Promise<INoSqlDbMetrics>` | Server metrics (db/collection counts, sessions, transactions, auth, uptime) |
|
|
663
|
+
| `getOpLog(params?)` | `Promise<IOpLogResult>` | Query oplog entries with optional filters |
|
|
664
|
+
| `getOpLogStats()` | `Promise<IOpLogStats>` | Aggregate oplog statistics |
|
|
665
|
+
| `revertToSeq(seq, dryRun?)` | `Promise<IRevertResult>` | Revert to a specific oplog sequence (must be within retained oplog history) |
|
|
666
|
+
| `getCollections(db?)` | `Promise<ICollectionInfo[]>` | List all collections with counts |
|
|
667
|
+
| `getDocuments(db, coll, limit?, skip?)` | `Promise<IDocumentsResult>` | Browse documents with pagination |
|
|
668
|
+
| `getHealth(options?)` | `Promise<INoSqlDbHealth>` | Read readiness and explicit publication-hold capability fields with optional fail-stop cancellation/deadline ownership |
|
|
669
|
+
| `allocateDatabaseTenant(params, options?)` | `Promise<INoSqlDbAllocateDatabaseTenantResult>` | Authoritatively allocate an absent file-backed tenant and return its durable allocation identity; supports fail-stop cancellation and deadlines |
|
|
670
|
+
| `attestDatabaseTenant(params, options?)` | `Promise<INoSqlDbAttestDatabaseTenantResult>` | Verify exact existing tenant ownership, roles, and candidate credential without mutating provider, auth, fence, or runtime cache state |
|
|
671
|
+
| `issueDatabaseAllocationReadAccessGrant(params, options?)` | `Promise<INoSqlDbIssueDatabaseAllocationReadAccessGrantResult>` | Issue or exactly replay one fixed 12-minute read grant bound to the complete active allocation identity |
|
|
672
|
+
| `revokeDatabaseAllocationReadAccessGrant(params, options?)` | `Promise<INoSqlDbRevokeDatabaseAllocationReadAccessGrantResult>` | Idempotently revoke/tombstone one exact allocation read grant and invalidate its live runtime state |
|
|
673
|
+
| `ensureDatabaseTenant(params, options?)` | `Promise<INoSqlDbEnsureDatabaseTenantResult>` | Idempotently ensure an exact fenced tenant; supports fail-stop cancellation and deadlines |
|
|
674
|
+
| `getDatabaseResourceFenceState(params, options?)` | `Promise<INoSqlDbDatabaseResourceFenceState \| null>` | Safely inspect a file-backed database fence high-water mark and publication phase without exposing receipts or capability material; supports fail-stop cancellation and deadlines |
|
|
675
|
+
| `getDatabaseContentDigest(params, options?)` | `Promise<INoSqlDbDatabaseContentDigest>` | Compute a bounded, database-name-independent digest over exact BSON documents and persisted index specifications; supports fail-stop cancellation and deadlines |
|
|
676
|
+
| `exportDatabase(params, options?)` | `Promise<INoSqlDbDatabaseExport>` | Export one database as lossless canonical MongoDB Extended JSON; supports fail-stop cancellation and deadlines |
|
|
677
|
+
| `importDatabase(params, options?)` | `Promise<INoSqlDbImportDatabaseResult>` | Durably replace one database from canonical or relaxed MongoDB Extended JSON, optionally leaving publication held; supports fail-stop cancellation and deadlines |
|
|
678
|
+
| `deleteDatabaseTenant(params, options?)` | `Promise<INoSqlDbDeleteDatabaseTenantResult>` | Durably delete an exact tenant database/user, optionally leaving publication held; supports fail-stop cancellation and deadlines |
|
|
679
|
+
| `commitDatabasePublication(params, options?)` | `Promise<TNoSqlDbCommitDatabasePublicationResult>` | Idempotently release an exact held publication receipt; supports fail-stop cancellation and deadlines |
|
|
680
|
+
|
|
681
|
+
### NoSqlDbStorageRehearsal
|
|
682
|
+
|
|
683
|
+
`NoSqlDbStorageRehearsal.createDestination(input, options?)` creates an isolated,
|
|
684
|
+
authenticated copy of an initialized, stopped file-storage root. It verifies
|
|
685
|
+
the source's existing native owner, provider and authentication bindings while
|
|
686
|
+
holding its existing outer and native file-root leases continuously.
|
|
687
|
+
The source is read-only throughout: no initialization, recovery, migration,
|
|
688
|
+
permission repair, journal retirement or identity editing occurs there. Successful
|
|
689
|
+
creation leaves it independently restartable at its original path.
|
|
690
|
+
|
|
691
|
+
```typescript
|
|
692
|
+
import { NoSqlDbServer, NoSqlDbStorageRehearsal } from '@lossless.org/nosqldb';
|
|
693
|
+
|
|
694
|
+
const binaryPath = '/opt/corestore/engines/nosqldb/rustdb_linux_amd64';
|
|
695
|
+
const operation = new NoSqlDbStorageRehearsal({ binaryPath });
|
|
696
|
+
const request = {
|
|
697
|
+
sourceFolderPath: '/var/lib/corestore/authoritative',
|
|
698
|
+
destinationFolderPath: '/var/lib/corestore/rehearsals/upgrade-1',
|
|
699
|
+
operationId: 'corestore-upgrade-1',
|
|
700
|
+
};
|
|
701
|
+
const controller = new AbortController();
|
|
702
|
+
try {
|
|
703
|
+
// Corestore has stopped the source and retains its maintenance exclusion.
|
|
704
|
+
const receipt = await operation.createDestination(request, {
|
|
705
|
+
timeoutMs: 30_000,
|
|
706
|
+
signal: controller.signal,
|
|
707
|
+
});
|
|
708
|
+
const candidate = new NoSqlDbServer({
|
|
709
|
+
binaryPath,
|
|
710
|
+
storage: 'file',
|
|
711
|
+
storagePath: receipt.destinationFolderPath,
|
|
712
|
+
fileStorageStartup: 'current-format-only',
|
|
713
|
+
storageRehearsalId: receipt.operationId,
|
|
714
|
+
socketPath: '/run/corestore/private/rehearsal.sock',
|
|
715
|
+
auth: {
|
|
716
|
+
enabled: true,
|
|
717
|
+
scramIterations: receipt.scramIterations,
|
|
718
|
+
},
|
|
719
|
+
});
|
|
720
|
+
try {
|
|
721
|
+
await candidate.start({ timeoutMs: 30_000 });
|
|
722
|
+
const uri = candidate.getConnectionUri();
|
|
723
|
+
// Connect SmartData with the preserved credentials. Prepare its models,
|
|
724
|
+
// then run Corestore's versioned ts_migration modules against this URI.
|
|
725
|
+
// Keep external services and application side effects isolated in Corestore.
|
|
726
|
+
} finally {
|
|
727
|
+
await candidate.stop();
|
|
728
|
+
}
|
|
729
|
+
// Retain the typed receipt through Corestore's normal persistence API.
|
|
730
|
+
} finally {
|
|
731
|
+
await operation.stop();
|
|
732
|
+
}
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
The destination's parent and socket parent must already exist, be owned by the
|
|
736
|
+
effective user and have mode 0700. The socket must be absent and outside the
|
|
737
|
+
storage root. Source and destination paths must be absolute, canonical, distinct,
|
|
738
|
+
non-nested paths on the same supported local filesystem/device. The destination
|
|
739
|
+
must be absent on the first attempt. Neither request nor receipt contains a
|
|
740
|
+
password. `binaryPath` selects exactly the caller-verified executable through
|
|
741
|
+
Smartrust; omission retains normal engine discovery without environment changes.
|
|
742
|
+
|
|
743
|
+
The public types are `INoSqlDbStorageRehearsalOptions`,
|
|
744
|
+
`INoSqlDbStorageRehearsalInput`, `INoSqlDbStorageRehearsalReceipt` and
|
|
745
|
+
`INoSqlDbStorageRehearsalPreservation`. An optional
|
|
746
|
+
`expectedSourceProviderRootId` pins a previously attested source provider ID;
|
|
747
|
+
NoSQLDB always verifies the source's stored bindings even when this is omitted.
|
|
748
|
+
Consumers do not inspect private identity files. Optional
|
|
749
|
+
`limits: { maximumEntries, maximumBytes }` can lower the source scan limits.
|
|
750
|
+
|
|
751
|
+
The supported storage profile is `native-mvcc-authenticated-v1`; creation
|
|
752
|
+
evidence uses `smartdb.storage-rehearsal.receipt.v2`. Rehearsal accepts current
|
|
753
|
+
native authentication records only. The removed `authMigration` option and
|
|
754
|
+
schema-2 worker conversion are not accepted.
|
|
755
|
+
|
|
756
|
+
| State | Rehearsal contract |
|
|
757
|
+
|---|---|
|
|
758
|
+
| Contents and catalogs | Preserve exact BSON documents, indexes and aliases, empty namespaces, and internal databases including `admin`, `local` and `config`. |
|
|
759
|
+
| Authentication | Preserve usernames, roles, salts, password verifiers, work factor, principal identities, generations and retained grant replay history. |
|
|
760
|
+
| Fencing and holds | Preserve allocations, retired history, counters and exact receipt records. Held databases remain held; a rehearsal cannot release their production capabilities. |
|
|
761
|
+
| Sessions and outcomes | Preserve durable retry and transaction outcomes, including retained and retiring session records. |
|
|
762
|
+
| Native history | Preserve the source group/timeline and committed records. The destination adds one committed provider-binding record with isolated purpose; it never resets authority counters. |
|
|
763
|
+
|
|
764
|
+
Active source owners, rehearsal-as-source, old or mixed layouts, unknown entries,
|
|
765
|
+
unfinished import or fencing operations, active read grants, unfinished or
|
|
766
|
+
quarantined indexes, invalid catalogs, corrupt auth/outcomes and storage requiring
|
|
767
|
+
recovery are rejected before publication. Source symlinks, hardlinks, nested
|
|
768
|
+
mounts, special objects, unsafe owners/modes and identity changes are rejected.
|
|
769
|
+
Creation supports the qualified Linux local-filesystem set and writable local
|
|
770
|
+
macOS APFS with native ownership and atomic publication. Other filesystems and
|
|
771
|
+
operating systems are rejected before destination initialization.
|
|
772
|
+
|
|
773
|
+
Defaults and hard limits are 100,000 filesystem entries, 1 TiB of file bytes,
|
|
774
|
+
depth 16, 4095-byte paths and 64 MiB of retained inventory metadata. The owning
|
|
775
|
+
native catalog, index, auth, outcome and fencing validators also enforce their
|
|
776
|
+
record and semantic work budgets. Exhausted bounds reject without truncation or
|
|
777
|
+
reset. LSNs and fencing high-water marks in the receipt are decimal strings,
|
|
778
|
+
preserving their complete unsigned 64-bit range. These are operation bounds,
|
|
779
|
+
not total serving document limits.
|
|
780
|
+
|
|
781
|
+
Copying and hashing stream 64 KiB chunks. Complete source/destination validation
|
|
782
|
+
compares the preserved record digest as well as physical identity and contents;
|
|
783
|
+
this is not a constant-time snapshot or reflink.
|
|
784
|
+
|
|
785
|
+
`storageRehearsalId?: string` is supported by both `INoSqlDbServerOptions` and
|
|
786
|
+
`ILocalNoSqlDbOptions`. It requires the exact recorded operation ID, file storage,
|
|
787
|
+
`fileStorageStartup: 'current-format-only'`, the recorded auth work factor,
|
|
788
|
+
enabled authentication, no bootstrap users, and a private explicit Unix socket.
|
|
789
|
+
`LocalNoSqlDb` also forwards `auth?: INoSqlDbAuthOptions`. Ordinary startup rejects
|
|
790
|
+
a rehearsal root. Native startup verifies the completion seal and new physical
|
|
791
|
+
provider binding and preserved native auth profile before recovery or auth initialization. Health
|
|
792
|
+
reports `storagePurpose: 'rehearsal'` and `storageRehearsalId`; production fencing,
|
|
793
|
+
allocation and publication capabilities are disabled. Authenticated ordinary
|
|
794
|
+
SmartData reads, writes and transactions remain available on unheld databases.
|
|
795
|
+
Corestore owns application/outbound-service isolation. There is no promotion
|
|
796
|
+
API: changing startup flags cannot turn this destination into production authority.
|
|
797
|
+
|
|
798
|
+
One instance admits one operation at a time; independent instances can select
|
|
799
|
+
different executables and sources. Native leases exclude active source owners
|
|
800
|
+
and conflicting copies. `timeoutMs` defaults to one hour and is capped at 24 hours;
|
|
801
|
+
the budget includes engine lookup/spawn and the native operation. Abort, deadline
|
|
802
|
+
and `stop()` terminate the owned sidecar; methods settle after confirmed exit.
|
|
803
|
+
Cleanup can outlast the deadline. If termination cannot be confirmed, the instance
|
|
804
|
+
retains its bridge and `stop()` remains retryable. Native work also checks parent
|
|
805
|
+
process death. Source stability relies on the native lease plus the caller's
|
|
806
|
+
maintenance exclusion from manual edits and restarts between attempts.
|
|
807
|
+
|
|
808
|
+
Publication uses a private sibling operation workspace, a bounded durable journal,
|
|
809
|
+
a distinct native destination lease and an atomic exclusive rename. The renamed
|
|
810
|
+
root cannot start until its completion seal is durable. After interruption, retry
|
|
811
|
+
the **identical request**, including paths, operation ID, expected provider ID and
|
|
812
|
+
limits, using the same NoSQLDB release or a compatible newer release. A retry may
|
|
813
|
+
use a different deadline. Keep the source stopped and unchanged until an unfinished
|
|
814
|
+
copy completes; do not edit, remove or move either root or the operation workspace.
|
|
815
|
+
Completed retries return the same creation receipt even after legitimate rehearsal
|
|
816
|
+
writes or source restart. They verify the destination's identity and seal and
|
|
817
|
+
re-establish publication durability; they do not attest to its later contents.
|
|
818
|
+
|
|
819
|
+
`NoSqlDbStorageRehearsalError` has fixed, secret-free messages and no raw nested
|
|
820
|
+
cause. Codes are `INVALID_REQUEST`, `UNSUPPORTED`, `BUSY`, `NOT_FOUND`,
|
|
821
|
+
`DESTINATION_CONFLICT`, `IDENTITY_MISMATCH`, `STATE_CORRUPT`, `SOURCE_CHANGED`,
|
|
822
|
+
`CAPACITY`, `CANCELLED`, `IO`, `RECOVERY_REQUIRED` and `BINARY_UNAVAILABLE`.
|
|
823
|
+
Uncertain command completion or cleanup reports `RECOVERY_REQUIRED`; an error
|
|
824
|
+
does not imply an absent destination. Exact retry reconciles interrupted copies,
|
|
825
|
+
renames and receipt writes without replacing unrelated paths. A changed source
|
|
826
|
+
or tampered destination fails closed.
|
|
827
|
+
|
|
828
|
+
The receipt identifies both paths/providers, the destination device/inode, request
|
|
829
|
+
and receipt hashes, storage profile, public auth work factor, preservation counts
|
|
830
|
+
and explicit `productionAuthority: false` / `promotionSupported: false`. It exposes
|
|
831
|
+
no usernames, principal IDs, per-file auth hashes, verifiers or publication secrets.
|
|
832
|
+
It is integrity-checked creation evidence, not an installer signature. Source
|
|
833
|
+
verification compares complete bytes, object identities, owners, link counts,
|
|
834
|
+
modes, modification times and change times; ordinary read access times may change.
|
|
835
|
+
Matching Linux and macOS amd64/arm64 engines and their checksum/provenance files ship in the
|
|
836
|
+
same published NoSQLDB package's `dist_rust/` directory; see the engine artifact
|
|
837
|
+
section below.
|
|
838
|
+
|
|
839
|
+
### LocalNoSqlDb
|
|
840
|
+
|
|
841
|
+
Zero-config wrapper around NoSqlDbServer. Uses Unix sockets and file-based persistence.
|
|
842
|
+
|
|
843
|
+
#### Constructor Options (`ILocalNoSqlDbOptions`)
|
|
844
|
+
|
|
845
|
+
```typescript
|
|
846
|
+
import { LocalNoSqlDb } from '@lossless.org/nosqldb';
|
|
847
|
+
|
|
848
|
+
const db = new LocalNoSqlDb({
|
|
849
|
+
folderPath: './data', // Required: data storage directory
|
|
850
|
+
socketPath: '/tmp/custom.sock', // Optional: custom socket (default: auto-generated)
|
|
851
|
+
binaryPath: '/opt/spark/engines/nosqldb/rustdb_linux_amd64', // Optional: exact engine
|
|
852
|
+
fileStorageStartup: 'current-format-only', // Optional; also the default
|
|
853
|
+
});
|
|
854
|
+
```
|
|
855
|
+
|
|
856
|
+
`binaryPath` has the same strict selection contract as `NoSqlDbServer` and is
|
|
857
|
+
passed through unchanged on every start. Supply an absolute path because the
|
|
858
|
+
underlying server and bridge are constructed by `LocalNoSqlDb.start()`.
|
|
859
|
+
|
|
860
|
+
#### Methods & Properties
|
|
861
|
+
|
|
862
|
+
| Method / Property | Type | Description |
|
|
863
|
+
|---|---|---|
|
|
864
|
+
| `start()` | `Promise<ILocalNoSqlDbConnectionInfo>` | Start and return connection info |
|
|
865
|
+
| `stop()` | `Promise<void>` | Stop the server |
|
|
866
|
+
| `getConnectionInfo()` | `ILocalNoSqlDbConnectionInfo` | Get current connection info |
|
|
867
|
+
| `getConnectionUri()` | `string` | Get the connection URI |
|
|
868
|
+
| `getServer()` | `NoSqlDbServer` | Access the underlying server |
|
|
869
|
+
| `running` | `boolean` | Whether the server is running |
|
|
870
|
+
| `LocalNoSqlDb.relocateStoppedStorageRoot(input, options?)` | `Promise<ILocalNoSqlDbStoppedStorageRootRelocationReceipt>` | Atomically relocate one eligible stopped Linux or macOS file-storage root and retain a source receipt |
|
|
871
|
+
| `LocalNoSqlDb.inspectOfflineStringValue(input, options?)` | `Promise<TLocalNoSqlDbOfflineStringValueInspectionResult>` | Read one exact top-level string value from stopped file storage without starting or mutating the engine |
|
|
872
|
+
| `LocalNoSqlDb.inspectOfflinePhysicalNamespaces(input, options?)` | `Promise<ILocalNoSqlDbOfflinePhysicalNamespaceInspectionResult>` | Validate stopped physical file storage and return only deterministic database and collection names |
|
|
873
|
+
|
|
874
|
+
#### Stopped Storage-Root Relocation
|
|
875
|
+
|
|
876
|
+
`relocateStoppedStorageRoot()` moves an eligible stopped file-storage root without starting `RustDb`, binding a listener, running a TypeScript migration, initializing storage or auth, or performing WAL recovery. It supports Linux on the existing qualified local-filesystem set and macOS on writable local APFS volumes with ownership, persistent object identities, `flock`, and atomic exclusive/swap rename support. The source root, source parent, and destination parent must be trusted descriptor-safe objects on the same filesystem. The supplied source and destination must be absolute and resolve to distinct, non-root, non-nested physical paths, and the destination must be absent. macOS also rejects case-folded, Unicode-normalized, and ancestry aliases before creating relocation state. The trusted parents may differ.
|
|
877
|
+
|
|
878
|
+
```typescript
|
|
879
|
+
import { LocalNoSqlDb } from '@lossless.org/nosqldb';
|
|
880
|
+
|
|
881
|
+
const receipt = await LocalNoSqlDb.relocateStoppedStorageRoot({
|
|
882
|
+
sourceFolderPath: '/var/lib/myapp/harness-controller/nosqldb',
|
|
883
|
+
destinationFolderPath: '/var/lib/myapp/hcon/nosqldb',
|
|
884
|
+
relocationId: 'move-2026-08-17-1',
|
|
885
|
+
}, {
|
|
886
|
+
timeoutMs: 30_000,
|
|
887
|
+
});
|
|
888
|
+
|
|
889
|
+
// Start only the destination. The source is now the retained receipt file.
|
|
890
|
+
const db = new LocalNoSqlDb({ folderPath: receipt.destinationFolderPath });
|
|
891
|
+
await db.start();
|
|
892
|
+
```
|
|
893
|
+
|
|
894
|
+
The input must be a plain object with exactly the own enumerable data fields `sourceFolderPath`, `destinationFolderPath`, and `relocationId`. Each supplied path must already be absolute, is normalized before sidecar creation, is limited to 1 through 4095 UTF-8 bytes, and cannot contain control characters. Longer destination paths are supported within that bound. `relocationId` is a 1 through 256-byte ASCII identifier beginning with an alphanumeric character and otherwise using only alphanumerics, `.`, `_`, `:`, `@`, or `-`. Management `timeoutMs` and `signal` use the same validation and one-deadline behavior as other NoSQLDB management operations. The native operation also checks its monotonic deadline and parent-process lifetime during admission and publication.
|
|
895
|
+
|
|
896
|
+
The returned exact receipt has this shape:
|
|
897
|
+
|
|
898
|
+
```typescript
|
|
899
|
+
interface ILocalNoSqlDbStoppedStorageRootRelocationReceipt {
|
|
900
|
+
format: 'smartdb.storage-root-relocation.receipt.v1';
|
|
901
|
+
version: 1;
|
|
902
|
+
relocationId: string;
|
|
903
|
+
sourceFolderPath: string;
|
|
904
|
+
destinationFolderPath: string;
|
|
905
|
+
providerRootId: string;
|
|
906
|
+
storageRootDevice: string; // canonical decimal u64
|
|
907
|
+
storageRootInode: string; // canonical decimal u64
|
|
908
|
+
receiptSha256: string;
|
|
909
|
+
sourceReceiptRetained: true;
|
|
910
|
+
}
|
|
911
|
+
```
|
|
912
|
+
|
|
913
|
+
On success, the destination directory is the original storage-root inode and the source path is an exact mode-0600 regular-file receipt. NoSQLDB does not remove that source receipt. The operation appends one native commit that changes only the fencing profile's physical path binding. All other record bytes and versions are preserved, including documents, indexes, empty namespaces, auth verifiers, principal identities, transaction outcomes, fencing history, holds and counters. The native group and timeline, provider identity, and original fencing receipt domain remain unchanged. The native commit position advances exactly once, including after retries.
|
|
914
|
+
|
|
915
|
+
Relocation supports stopped current native production roots with or without authentication. It never initializes missing ownership or storage metadata. Rehearsal destinations, legacy or mixed layouts, active read-access grants, unfinished fencing or allocation operations, building or quarantined indexes, symlinks, unknown entries, unsafe modes/ownership/link counts, exceeded inspection bounds, active owners and ambiguous identities are rejected explicitly. Admission uses the same owning catalog, auth, index, outcome and fencing validators as other stopped native operations. This operation moves the existing authority; it does not promote a rehearsal copy or duplicate production authority.
|
|
916
|
+
|
|
917
|
+
The operation durably records one immutable native append plan and reserves the receipt before changing the path binding. It retains exclusive ownership through the append, atomic directory/receipt exchange and confirmed journal cleanup. If a call returns `RECOVERY_REQUIRED`, stop startup attempts and retry the exact normalized source path, destination path and `relocationId` with the same native-capable NoSQLDB version. Startup rejects any relocation-workspace occupancy and never mutates or recovers it. Do not delete, rename, edit or replace either path between attempts. Native relocation does not accept unfinished journals from the removed Bitcask engine.
|
|
918
|
+
|
|
919
|
+
An exact completed replay returns the same receipt only while the destination remains the current root. A later successful relocation leaves another receipt at that destination path and supersedes the earlier replay topology; retry the later operation instead. Manual copy or rename remains unsupported and continues to fail provider path/inode fencing.
|
|
920
|
+
|
|
921
|
+
Failures are exposed as `LocalNoSqlDbStorageRootRelocationError` with a code-only union: `INVALID_REQUEST`, `UNSUPPORTED`, `BUSY`, `NOT_FOUND`, `DESTINATION_CONFLICT`, `IDENTITY_MISMATCH`, `STATE_CORRUPT`, `SOURCE_CHANGED`, `CAPACITY`, `CANCELLED`, `IO`, or `RECOVERY_REQUIRED`. Errors and receipts contain no credential or document material. Transport ambiguity after dispatch is `RECOVERY_REQUIRED`; exact retry is the recovery action. An explicit native cancellation or I/O error can also leave private recovery state, which the identical request can finish after the sidecar has confirmed exit.
|
|
922
|
+
|
|
923
|
+
#### Offline String-Value Inspection
|
|
924
|
+
|
|
925
|
+
`LocalNoSqlDb.inspectOfflineStringValue(input, options?)` reads one string from a
|
|
926
|
+
stopped current native file root on supported Linux filesystems or macOS APFS.
|
|
927
|
+
It starts only a disposable management sidecar, acquires the existing native
|
|
928
|
+
leases read-only, and pins the committed root. It never starts a listener,
|
|
929
|
+
initializes storage, repairs tails, replays writes or compacts the source.
|
|
930
|
+
|
|
931
|
+
```typescript
|
|
932
|
+
import { LocalNoSqlDb } from '@lossless.org/nosqldb';
|
|
933
|
+
|
|
934
|
+
const result = await LocalNoSqlDb.inspectOfflineStringValue({
|
|
935
|
+
folderPath: '/var/lib/myapp/nosqldb',
|
|
936
|
+
databaseName: 'myapp',
|
|
937
|
+
collectionName: 'MetadataDoc',
|
|
938
|
+
match: { field: 'key', value: 'schemaVersion' },
|
|
939
|
+
valueField: 'value',
|
|
940
|
+
limits: {
|
|
941
|
+
maximumEntries: 100_000,
|
|
942
|
+
maximumTotalFileBytes: 1024 ** 4,
|
|
943
|
+
maximumRecords: 1_000_000,
|
|
944
|
+
maximumRecordBytes: 16 * 1024 * 1024,
|
|
945
|
+
maximumResultBytes: 1024 * 1024,
|
|
946
|
+
},
|
|
947
|
+
}, { timeoutMs: 5000 });
|
|
948
|
+
|
|
949
|
+
if (result.status === 'found') console.log(result.value);
|
|
950
|
+
```
|
|
951
|
+
|
|
952
|
+
The result is exactly `{ status: 'not-found' }` or
|
|
953
|
+
`{ status: 'found', value: string }`. No other document fields cross IPC.
|
|
954
|
+
Multiple matches, non-string results, corrupted storage, changed source state,
|
|
955
|
+
unsafe objects, missing owner metadata and exhausted limits reject the operation.
|
|
956
|
+
|
|
957
|
+
#### Offline Physical Namespace Inspection
|
|
958
|
+
|
|
959
|
+
`LocalNoSqlDb.inspectOfflinePhysicalNamespaces(input, options?)` validates and
|
|
960
|
+
lists the stopped native database/collection catalog. Empty namespaces are
|
|
961
|
+
retained. It validates native document/count/index relationships and compares
|
|
962
|
+
complete bounded filesystem observations before returning; no partial result is
|
|
963
|
+
published. Documents, index definitions, IDs and values stay inside the sidecar.
|
|
964
|
+
|
|
965
|
+
```typescript
|
|
966
|
+
const namespaces = await LocalNoSqlDb.inspectOfflinePhysicalNamespaces({
|
|
967
|
+
folderPath: '/var/lib/myapp/nosqldb',
|
|
968
|
+
limits: {
|
|
969
|
+
maximumEntries: 100_000,
|
|
970
|
+
maximumDatabases: 1_000_000,
|
|
971
|
+
maximumCollections: 1_000_000,
|
|
972
|
+
maximumIndexes: 1_000_000,
|
|
973
|
+
maximumTotalFileBytes: 1024 ** 4,
|
|
974
|
+
maximumRecordsPerCollection: 1_000_000_000,
|
|
975
|
+
maximumRecordBytes: 16 * 1024 * 1024,
|
|
976
|
+
maximumResultBytes: 16 * 1024 * 1024,
|
|
977
|
+
},
|
|
978
|
+
}, { timeoutMs: 30_000 });
|
|
979
|
+
// { schemaVersion: 1, databases: [{ name: 'myapp', collections: ['MetadataDoc'] }] }
|
|
980
|
+
```
|
|
981
|
+
|
|
982
|
+
These required limits are validated before sidecar creation:
|
|
983
|
+
|
|
984
|
+
| Limit | Supported range |
|
|
985
|
+
|---|---|
|
|
986
|
+
| `maximumEntries` | 1–100,000 filesystem entries |
|
|
987
|
+
| `maximumTotalFileBytes` | 1 byte–1 TiB |
|
|
988
|
+
| `maximumRecords` / `maximumRecordsPerCollection` | 1–1,000,000,000 |
|
|
989
|
+
| `maximumRecordBytes` | 5 bytes–16 MiB |
|
|
990
|
+
| `maximumResultBytes` | 1 byte–16 MiB for a value; 32 bytes–16 MiB for namespaces |
|
|
991
|
+
| `maximumDatabases`, `maximumCollections`, `maximumIndexes` | 1–1,000,000 each for namespace inspection |
|
|
992
|
+
| Complete encoded request | At most 2 MiB |
|
|
993
|
+
| `timeoutMs` | 1–2,147,483,647 milliseconds |
|
|
994
|
+
|
|
995
|
+
All numeric values must be safe integers. Paths are limited to 4095 UTF-8 bytes
|
|
996
|
+
before and after absolute resolution; namespace and field names to 255 bytes.
|
|
997
|
+
The string match is at most 1 MiB and must fit `maximumRecordBytes`.
|
|
998
|
+
Removed per-Bitcask-file limits are rejected.
|
|
999
|
+
|
|
1000
|
+
One optional deadline and AbortSignal cover lookup/spawn, native inspection and
|
|
1001
|
+
delivery. Cancellation kills and reaps the disposable sidecar before rejection.
|
|
1002
|
+
An active database owner blocks inspection. The caller must retain continuous
|
|
1003
|
+
lifecycle exclusion through acceptance or handoff of the result: the native
|
|
1004
|
+
lease covers inspection, and cannot prevent a restart after the sidecar exits.
|
|
1005
|
+
|
|
1006
|
+
### NoSqlDbDebugServer
|
|
1007
|
+
|
|
1008
|
+
Web-based debug dashboard served via `@api.global/typedserver`. Import from the `debugserver` subpath:
|
|
1009
|
+
|
|
1010
|
+
```typescript
|
|
1011
|
+
import { NoSqlDbDebugServer } from '@lossless.org/nosqldb/debugserver';
|
|
1012
|
+
|
|
1013
|
+
const debugServer = new NoSqlDbDebugServer(server, { port: 4000 });
|
|
1014
|
+
await debugServer.start();
|
|
1015
|
+
// Dashboard at http://localhost:4000
|
|
1016
|
+
|
|
1017
|
+
await debugServer.stop();
|
|
1018
|
+
```
|
|
1019
|
+
|
|
1020
|
+
The UI is bundled as base64-encoded content (via `@git.zone/tsbundle`) and served from memory — no static file directory needed.
|
|
1021
|
+
|
|
1022
|
+
### NoSqlDbDebugUi (Web Component)
|
|
1023
|
+
|
|
1024
|
+
For embedding the debug UI directly into your own web application, import the `<nosqldb-debugui>` web component:
|
|
1025
|
+
|
|
1026
|
+
```typescript
|
|
1027
|
+
import { NoSqlDbDebugUi } from '@lossless.org/nosqldb/debugui';
|
|
1028
|
+
|
|
1029
|
+
// In your HTML/lit template:
|
|
1030
|
+
// <nosqldb-debugui .server=${myNoSqlDbServer}></nosqldb-debugui>
|
|
1031
|
+
//
|
|
1032
|
+
// Or in HTTP mode (when served by NoSqlDbDebugServer):
|
|
1033
|
+
// <nosqldb-debugui apiBaseUrl=""></nosqldb-debugui>
|
|
1034
|
+
```
|
|
1035
|
+
|
|
1036
|
+
---
|
|
1037
|
+
|
|
1038
|
+
## Supported Operations
|
|
1039
|
+
|
|
1040
|
+
NoSQLDB supports the core operations through the wire protocol. Use the standard `mongodb` driver — these all work:
|
|
1041
|
+
|
|
1042
|
+
### CRUD
|
|
1043
|
+
|
|
1044
|
+
```typescript
|
|
1045
|
+
// Insert
|
|
1046
|
+
await collection.insertOne({ name: 'Bob' });
|
|
1047
|
+
await collection.insertMany([{ a: 1 }, { a: 2 }]);
|
|
1048
|
+
|
|
1049
|
+
// Find
|
|
1050
|
+
const doc = await collection.findOne({ name: 'Bob' });
|
|
1051
|
+
const docs = await collection.find({ age: { $gte: 18 } }).toArray();
|
|
1052
|
+
|
|
1053
|
+
// Update
|
|
1054
|
+
await collection.updateOne({ name: 'Bob' }, { $set: { age: 25 } });
|
|
1055
|
+
await collection.updateMany({ active: false }, { $set: { archived: true } });
|
|
1056
|
+
|
|
1057
|
+
// Delete
|
|
1058
|
+
await collection.deleteOne({ name: 'Bob' });
|
|
1059
|
+
await collection.deleteMany({ archived: true });
|
|
1060
|
+
|
|
1061
|
+
// Replace
|
|
1062
|
+
await collection.replaceOne({ _id: id }, { name: 'New Bob', age: 30 });
|
|
1063
|
+
|
|
1064
|
+
// Find and Modify
|
|
1065
|
+
await collection.findOneAndUpdate({ name: 'Bob' }, { $inc: { visits: 1 } }, { returnDocument: 'after' });
|
|
1066
|
+
await collection.findOneAndDelete({ expired: true });
|
|
1067
|
+
await collection.findOneAndReplace({ _id: id }, { name: 'Replaced' }, { returnDocument: 'after' });
|
|
1068
|
+
```
|
|
1069
|
+
|
|
1070
|
+
### Query Operators
|
|
1071
|
+
|
|
1072
|
+
```typescript
|
|
1073
|
+
// Comparison
|
|
1074
|
+
{ age: { $eq: 25 } } { age: { $ne: 25 } }
|
|
1075
|
+
{ age: { $gt: 18 } } { age: { $lt: 65 } }
|
|
1076
|
+
{ age: { $gte: 18 } } { age: { $lte: 65 } }
|
|
1077
|
+
{ status: { $in: ['active', 'pending'] } }
|
|
1078
|
+
{ status: { $nin: ['deleted'] } }
|
|
1079
|
+
|
|
1080
|
+
// Logical
|
|
1081
|
+
{ $and: [{ age: { $gte: 18 } }, { active: true }] }
|
|
1082
|
+
{ $or: [{ status: 'active' }, { admin: true }] }
|
|
1083
|
+
{ $not: { status: 'deleted' } }
|
|
1084
|
+
|
|
1085
|
+
// Element
|
|
1086
|
+
{ email: { $exists: true } }
|
|
1087
|
+
{ type: { $type: 'string' } }
|
|
1088
|
+
|
|
1089
|
+
// Array
|
|
1090
|
+
{ tags: { $all: ['mongodb', 'database'] } }
|
|
1091
|
+
{ scores: { $elemMatch: { $gte: 80, $lt: 90 } } }
|
|
1092
|
+
{ tags: { $size: 3 } }
|
|
1093
|
+
|
|
1094
|
+
// Regex
|
|
1095
|
+
{ name: { $regex: /^Al/i } }
|
|
1096
|
+
```
|
|
1097
|
+
|
|
1098
|
+
### Update Operators
|
|
1099
|
+
|
|
1100
|
+
```typescript
|
|
1101
|
+
{ $set: { name: 'New Name' } }
|
|
1102
|
+
{ $unset: { tempField: '' } }
|
|
1103
|
+
{ $inc: { count: 1 } }
|
|
1104
|
+
{ $mul: { price: 1.1 } }
|
|
1105
|
+
{ $min: { low: 50 } } { $max: { high: 100 } }
|
|
1106
|
+
{ $push: { tags: 'new' } } { $pull: { tags: 'old' } }
|
|
1107
|
+
{ $addToSet: { tags: 'unique' } }
|
|
1108
|
+
{ $pop: { queue: 1 } } // Remove last
|
|
1109
|
+
{ $pop: { queue: -1 } } // Remove first
|
|
1110
|
+
{ $rename: { old: 'new' } }
|
|
1111
|
+
{ $currentDate: { lastModified: true } }
|
|
1112
|
+
```
|
|
1113
|
+
|
|
1114
|
+
### Aggregation Pipeline
|
|
1115
|
+
|
|
1116
|
+
```typescript
|
|
1117
|
+
const results = await collection.aggregate([
|
|
1118
|
+
{ $match: { status: 'active' } },
|
|
1119
|
+
{ $group: { _id: '$category', total: { $sum: '$amount' } } },
|
|
1120
|
+
{ $sort: { total: -1 } },
|
|
1121
|
+
{ $limit: 10 },
|
|
1122
|
+
{ $project: { category: '$_id', total: 1, _id: 0 } },
|
|
1123
|
+
]).toArray();
|
|
1124
|
+
```
|
|
1125
|
+
|
|
1126
|
+
**Supported stages:** `$match`, `$project`, `$group`, `$sort`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$count`, `$facet`, `$replaceRoot`, `$set`, `$unionWith`, `$out`, `$merge`
|
|
1127
|
+
|
|
1128
|
+
`$lookup` supports the equality form and a bounded correlated pipeline form with
|
|
1129
|
+
`let`, an optional `$match` using `$expr`/`$and`/`$eq`, and an optional following
|
|
1130
|
+
`$limit`. Correlated string equality predicates can use a full-key or subset
|
|
1131
|
+
foreign equality index to reduce candidates when available. Candidates are
|
|
1132
|
+
paged, the complete expression remains authoritative, and `$limit` counts only
|
|
1133
|
+
exact expression matches. Other BSON value shapes use the bounded scan path.
|
|
1134
|
+
Mixed `localField`/`foreignField` plus `pipeline` lookups and other inner stages
|
|
1135
|
+
are rejected explicitly. Complete filter trees are validated before data is
|
|
1136
|
+
read, and query nesting plus aggregation work are bounded.
|
|
1137
|
+
|
|
1138
|
+
A leading `$match` narrows the pipeline's source load instead of filtering
|
|
1139
|
+
after it: only matching documents are materialized, using an equality index when
|
|
1140
|
+
one serves the filter and a bounded scan otherwise. Filter shapes the query
|
|
1141
|
+
matcher cannot pre-compile fall back to the unnarrowed load, so results are
|
|
1142
|
+
unchanged. The materialization limit therefore applies to the matched set, which
|
|
1143
|
+
is what lets `countDocuments(filter)` run against collections larger than it.
|
|
1144
|
+
|
|
1145
|
+
**Group accumulators:** `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`, `$push`, `$addToSet`, `$count`
|
|
1146
|
+
|
|
1147
|
+
### Indexes
|
|
1148
|
+
|
|
1149
|
+
```typescript
|
|
1150
|
+
await collection.createIndex({ email: 1 }, { unique: true });
|
|
1151
|
+
await collection.createIndex({ name: 1, age: -1 }); // compound
|
|
1152
|
+
await collection.createIndex({ field: 1 }, { sparse: true });
|
|
1153
|
+
const indexes = await collection.listIndexes().toArray();
|
|
1154
|
+
await collection.dropIndex('email_1');
|
|
1155
|
+
await collection.dropIndexes(); // drop all except _id
|
|
1156
|
+
```
|
|
1157
|
+
|
|
1158
|
+
> 🛡️ **Unique indexes are enforced at the engine level.** Duplicate values are rejected with a `DuplicateKey` error (code 11000) *before* the document is written to disk — on `insertOne`, `updateOne`, `findAndModify`, and upserts. Index definitions and entries are committed with native storage and survive restart.
|
|
1159
|
+
|
|
1160
|
+
NoSQLDB supports ascending and descending index keys with `name`, `unique`,
|
|
1161
|
+
`sparse`, and `expireAfterSeconds` options. It accepts `background` as a
|
|
1162
|
+
validated no-op and index version `v: 2`. Unsupported options, including
|
|
1163
|
+
`partialFilterExpression`, `collation`, and `hidden`, are rejected before any
|
|
1164
|
+
index in the request is created. Native maintenance expires eligible BSON dates
|
|
1165
|
+
and date arrays in bounded batches, preserving publication holds and active
|
|
1166
|
+
snapshots.
|
|
1167
|
+
|
|
1168
|
+
`find()` and collection `aggregate()` accept `hint` as an index name (including
|
|
1169
|
+
`_id_`) or an exact ascending/descending key pattern. Key-pattern order matters;
|
|
1170
|
+
if several indexes share a key pattern, use a name. The selected active scalar
|
|
1171
|
+
index supplies every candidate, using safe full-key or leading-prefix equality
|
|
1172
|
+
bounds, or a full scan of that index. The complete predicate still runs, and explicit sorting,
|
|
1173
|
+
projection, skip, and limit retain their normal semantics. Index traversal uses
|
|
1174
|
+
bounded pages from a pinned committed root across client requests. Unsorted results have no guaranteed BSON value order.
|
|
1175
|
+
|
|
1176
|
+
```typescript
|
|
1177
|
+
await collection.createIndex({ organizationId: 1, userId: 1 }, { name: 'organization_user' });
|
|
1178
|
+
const page = await collection.find({ organizationId: 'org-1' }, {
|
|
1179
|
+
hint: 'organization_user', sort: { userId: 1, _id: 1 }, skip: 20, limit: 10,
|
|
1180
|
+
}).toArray();
|
|
1181
|
+
const count = await collection.countDocuments({ organizationId: 'org-1' }, {
|
|
1182
|
+
hint: 'organization_user',
|
|
1183
|
+
});
|
|
1184
|
+
```
|
|
1185
|
+
|
|
1186
|
+
Unknown or quarantined indexes, malformed/ambiguous patterns, `$natural`, and
|
|
1187
|
+
hints on missing collections or views fail explicitly. Forced multikey traversal
|
|
1188
|
+
is unsupported and rejects; unhinted array queries keep their existing scan
|
|
1189
|
+
behavior. Sparse hints intentionally exclude documents absent from that index,
|
|
1190
|
+
including for empty filters and counts. A hint is never silently replaced by a
|
|
1191
|
+
collection scan or another index.
|
|
1192
|
+
|
|
1193
|
+
Hinted transaction reads construct the selected index from the existing bounded
|
|
1194
|
+
transaction snapshot plus buffered writes; they never fetch candidate documents
|
|
1195
|
+
from live storage. Read-only aggregate source pipelines also use that transaction
|
|
1196
|
+
view. Foreign-collection stages (`$lookup`, `$unionWith`) and writing stages
|
|
1197
|
+
(`$out`, `$merge`), including stages inside `$facet`, reject within transactions.
|
|
1198
|
+
Nontransaction execution uses the streaming and bounded-resource paths below.
|
|
1199
|
+
|
|
1200
|
+
### Scalable query execution
|
|
1201
|
+
|
|
1202
|
+
Collections have no fixed total-document ceiling on the nontransaction `find`,
|
|
1203
|
+
`count`, `distinct`, `update`, `delete`, and `findAndModify` paths. Result cursors
|
|
1204
|
+
read bounded pages instead of retaining the whole result. Multi-document writes
|
|
1205
|
+
visit each source document once, including when updates move indexed values.
|
|
1206
|
+
Index publication stages changed keys instead of copying the complete collection
|
|
1207
|
+
index engine for each write.
|
|
1208
|
+
|
|
1209
|
+
Nontransaction aggregation pipelines composed of `$match`, `$project`, `$unset`,
|
|
1210
|
+
`$set`, `$addFields`, `$replaceRoot`, `$replaceWith`, `$skip`, `$limit`, `$sort`,
|
|
1211
|
+
`$group`, `$count`, `$unwind`, `$lookup`, `$unionWith`, and `$facet` also operate
|
|
1212
|
+
beyond 10,000 input or output documents.
|
|
1213
|
+
Counts use exact collection metadata where applicable, otherwise stream matching
|
|
1214
|
+
records. A limit bounds the work needed to find the requested number of matches.
|
|
1215
|
+
Without a limit, exact counts continue to EOF. Hints, deadlines and empty-result
|
|
1216
|
+
shapes remain supported; signed BSON count overflow and resource exhaustion
|
|
1217
|
+
reject explicitly. Exact counts never substitute estimates, and transaction
|
|
1218
|
+
counts retain the snapshot admission limits described below.
|
|
1219
|
+
Grouping accumulates one group at a time after an external sort; constant-key
|
|
1220
|
+
groups stream directly. Group accumulator semantics follow the documented local
|
|
1221
|
+
expression surface; this does not add the complete MongoDB aggregation language.
|
|
1222
|
+
Unwind retains one input while expanding arrays across batches. Lookup scans
|
|
1223
|
+
bounded foreign pages or indexed candidates, then assembles one byte-bounded
|
|
1224
|
+
result document. Unindexed joins can still require repeated foreign scans.
|
|
1225
|
+
Facet branches replay a shared input buffer or temporary spool; their combined
|
|
1226
|
+
result must fit the BSON document budget.
|
|
1227
|
+
|
|
1228
|
+
Scalar ordered indexes can satisfy complete sort keys after equality fields,
|
|
1229
|
+
including reverse traversal and mixed compound directions. This path currently
|
|
1230
|
+
requires distinct full index keys to preserve stable equal-key ordering. Arrays,
|
|
1231
|
+
unsupported BSON index types, and equal-key data use the general sort executor.
|
|
1232
|
+
Numeric comparison and scalar equality indexes share exact Int32, Int64, Double,
|
|
1233
|
+
and Decimal128 ordering, including large integers, signed zero, and NaNs.
|
|
1234
|
+
|
|
1235
|
+
Small sorted limits retain top-k candidates. Larger sorts and groups use bounded
|
|
1236
|
+
runs in unnamed temporary files, with automatic cleanup on exhaustion, cursor
|
|
1237
|
+
closure, cancellation, errors, and process exit. Configure the shared resources
|
|
1238
|
+
through `NoSqlDbServer`:
|
|
1239
|
+
|
|
1240
|
+
```typescript
|
|
1241
|
+
const server = new NoSqlDbServer({
|
|
1242
|
+
storage: 'file',
|
|
1243
|
+
storagePath: '/path/to/database',
|
|
1244
|
+
query: {
|
|
1245
|
+
sortBufferBytes: 4 * 1024 * 1024,
|
|
1246
|
+
maxSortSpillBytes: 4 * 1024 * 1024 * 1024,
|
|
1247
|
+
maxConcurrentSorts: 4,
|
|
1248
|
+
},
|
|
1249
|
+
});
|
|
1250
|
+
```
|
|
1251
|
+
|
|
1252
|
+
`sortBufferBytes` is an encoded-input target, not an exact allocator heap limit;
|
|
1253
|
+
a single BSON document may exceed the target. The spill budget includes retained
|
|
1254
|
+
cursor files, facet input spools, and simultaneous merge input/output. Exhausting resource admission
|
|
1255
|
+
returns an explicit error instead of truncating results. Cursor continuations
|
|
1256
|
+
share a 256 MiB accounting budget. One continuation admits at most 160 MiB plus
|
|
1257
|
+
4 KiB, including pinned query state, changed-document overlays and bounded
|
|
1258
|
+
large-index-key candidates. Wire batches and pending document pages retain their
|
|
1259
|
+
16 MiB limits. `distinct`
|
|
1260
|
+
results and individual group documents must fit their BSON response/document
|
|
1261
|
+
byte budgets. The 10,000-operation wire batch limit still applies to each request;
|
|
1262
|
+
the driver can split larger input batches.
|
|
1263
|
+
|
|
1264
|
+
Native streaming reads pin a committed root across cursor pages. Later writes do
|
|
1265
|
+
not change that snapshot. Current authorization and publication ownership are
|
|
1266
|
+
revalidated before a continuation is served.
|
|
1267
|
+
Ownership checks, namespace gates, byte budgets, and `maxTimeMS` remain active
|
|
1268
|
+
through `getMore` and cleanup.
|
|
1269
|
+
|
|
1270
|
+
Transactions use pinned roots and changed-document overlays; the total database
|
|
1271
|
+
size is not a transaction snapshot limit. Complete write sets remain bounded by
|
|
1272
|
+
transaction and native commit byte admission. `$out` and `$merge` have no
|
|
1273
|
+
10,000-document output ceiling; their atomic output is subject to the same write
|
|
1274
|
+
budgets and rejects before publication if those budgets are exceeded. Native
|
|
1275
|
+
database import uses private staging for larger supported exports, then publishes
|
|
1276
|
+
the catalog and receipt atomically. Existing old-engine storage is rejected
|
|
1277
|
+
without conversion.
|
|
1278
|
+
|
|
1279
|
+
### Comparing performance with MongoDB
|
|
1280
|
+
|
|
1281
|
+
The repository includes [benchmark/run.ts](benchmark/run.ts), which starts fresh,
|
|
1282
|
+
loopback-only NoSQLDB and original MongoDB processes with isolated temporary
|
|
1283
|
+
storage. Supply an original `mongod` executable; the harness does not download
|
|
1284
|
+
one or use a MongoDB-compatible replacement as the reference server.
|
|
1285
|
+
|
|
1286
|
+
```bash
|
|
1287
|
+
NOSQLDB_RUST_BINARY="$PWD/rust/target/release/rustdb" \
|
|
1288
|
+
MONGODB_BINARY=/path/to/original/mongod \
|
|
1289
|
+
tsx benchmark/run.ts --engine both --sizes 1000,10001,100000 \
|
|
1290
|
+
--concurrency 1,16 --repetitions 3 --samples 250 --warmup 50 \
|
|
1291
|
+
> .nogit/scaling-comparison.jsonl
|
|
1292
|
+
```
|
|
1293
|
+
|
|
1294
|
+
Run on Linux after building the current binary and stopping competing benchmark
|
|
1295
|
+
or build work. Both engines use the same driver and data, `w:1`, `j:true`, local
|
|
1296
|
+
reads, snapshot transactions, disabled retryable writes, and one connection per
|
|
1297
|
+
worker. MongoDB runs as a single-voter replica set. Repetitions alternate engine
|
|
1298
|
+
order. `--id-type string` covers string identifiers; `--payload-bytes` changes
|
|
1299
|
+
record size. Full scans and transactions use the separate `--scan-samples` count
|
|
1300
|
+
(default 3) because they are substantially more expensive.
|
|
1301
|
+
|
|
1302
|
+
JSON-lines output records binary hashes, host/version metadata, correctness
|
|
1303
|
+
failures, seed/index time, latency percentiles, throughput, process RSS and peak
|
|
1304
|
+
RSS, CPU ticks, and physical read/write bytes. Temporary database directories are
|
|
1305
|
+
retained for inspection and their paths are recorded. Failed operations remain
|
|
1306
|
+
visible; they must not be compared as successful fast queries. Small sample
|
|
1307
|
+
counts and a shared development host are diagnostic evidence, not a general
|
|
1308
|
+
claim of superiority over MongoDB. The [recorded comparison](benchmark/readme.md)
|
|
1309
|
+
includes both measured improvements and remaining gaps.
|
|
1310
|
+
|
|
1311
|
+
### Database & Admin
|
|
1312
|
+
|
|
1313
|
+
```typescript
|
|
1314
|
+
await db.listCollections().toArray();
|
|
1315
|
+
await db.createCollection('new');
|
|
1316
|
+
await db.dropCollection('old');
|
|
1317
|
+
await db.dropDatabase();
|
|
1318
|
+
await db.stats();
|
|
1319
|
+
|
|
1320
|
+
const admin = client.db().admin();
|
|
1321
|
+
await admin.listDatabases();
|
|
1322
|
+
await admin.ping();
|
|
1323
|
+
await admin.serverStatus();
|
|
1324
|
+
```
|
|
1325
|
+
|
|
1326
|
+
NoSQLDB creates ordinary collections and one deliberately restricted view form.
|
|
1327
|
+
Capped collections, validators, collation, time-series collections, clustered
|
|
1328
|
+
indexes, encrypted fields, and all general view pipelines remain unsupported and
|
|
1329
|
+
are rejected before the database or namespace is created.
|
|
1330
|
+
|
|
1331
|
+
### Restricted Deny-All Views
|
|
1332
|
+
|
|
1333
|
+
The only accepted view definition is the exact deny-all pipeline below. This is
|
|
1334
|
+
intended for compatibility barriers such as renaming a legacy source and leaving
|
|
1335
|
+
its old namespace present but permanently empty:
|
|
1336
|
+
|
|
1337
|
+
```typescript
|
|
1338
|
+
await db.collection('secrets').rename('secrets_v2');
|
|
1339
|
+
await db.createCollection('secrets', {
|
|
1340
|
+
viewOn: 'secrets_v2',
|
|
1341
|
+
pipeline: [{ $match: { $expr: { $eq: [1, 0] } } }],
|
|
1342
|
+
});
|
|
1343
|
+
```
|
|
1344
|
+
|
|
1345
|
+
Both `viewOn` and `pipeline` are required. The pipeline must contain exactly one
|
|
1346
|
+
`$match` stage with exactly `$expr: { $eq: [1, 0] }`; empty pipelines, numeric or
|
|
1347
|
+
structural variants, additional stages, and additional collection-definition
|
|
1348
|
+
options are rejected with `InvalidOptions` (code 72) before mutation. Command
|
|
1349
|
+
metadata such as `$db`, `lsid`, `comment`, and API-version fields remains
|
|
1350
|
+
accepted. `viewOn` must identify an existing same-database collection or
|
|
1351
|
+
restricted view at creation time. General MongoDB views are not supported.
|
|
1352
|
+
|
|
1353
|
+
The logical view has no physical collection. `find`, `count`, and `distinct`
|
|
1354
|
+
return empty results. `aggregate` starts with an empty source and still applies
|
|
1355
|
+
the client pipeline; `$lookup` and `$unionWith` resolve the view as empty.
|
|
1356
|
+
Transactions first reading a current view record an explicit empty snapshot; a
|
|
1357
|
+
transaction that captured a populated collection before it is renamed and
|
|
1358
|
+
replaced by a view keeps that original snapshot for `find`, `count`, and
|
|
1359
|
+
`distinct` until commit conflict handling. Inserts, updates, deletes,
|
|
1360
|
+
`findAndModify`, indexes, `$out`, `$merge`, rename, and other data mutations fail
|
|
1361
|
+
with `CommandNotSupportedOnView` (code 166). `dropCollection` is the supported
|
|
1362
|
+
DDL exception for removing a logical restricted view.
|
|
1363
|
+
|
|
1364
|
+
Definitions are persisted in the physical `system.views` catalog. As in MongoDB,
|
|
1365
|
+
`listCollections` lists `system.views` as a physical storage collection and the
|
|
1366
|
+
logical namespace as type `view` with `{ viewOn, pipeline }`, read-only metadata,
|
|
1367
|
+
and no `idIndex`. `system.views` is reserved from direct wire reads,
|
|
1368
|
+
writes, index operations, create, drop, rename, and aggregation output. Whole
|
|
1369
|
+
database drop and trusted database export/import/digest operations handle it as
|
|
1370
|
+
physical catalog data. Export never materializes a view; cross-database import
|
|
1371
|
+
remaps the exact catalog `_id` database prefix. Startup validates databases whose
|
|
1372
|
+
publication lease can be acquired before listener bind; a publication-held
|
|
1373
|
+
database is validated on its first permit after release. Import validates before
|
|
1374
|
+
mutation. Malformed, oversized, noncanonical, duplicate, reserved, or colliding
|
|
1375
|
+
catalog state fails closed.
|
|
1376
|
+
|
|
1377
|
+
`system.views` was not interpreted by earlier SmartDB releases. This reservation
|
|
1378
|
+
is a major-version compatibility boundary: before upgrading, operators must use
|
|
1379
|
+
the older release to rename or remove any unrelated physical `system.views`
|
|
1380
|
+
collection. Once upgraded, direct catalog rename or drop is intentionally
|
|
1381
|
+
rejected with code 166.
|
|
1382
|
+
|
|
1383
|
+
### Bulk Operations
|
|
1384
|
+
|
|
1385
|
+
```typescript
|
|
1386
|
+
const result = await collection.bulkWrite([
|
|
1387
|
+
{ insertOne: { document: { name: 'Bulk1' } } },
|
|
1388
|
+
{ updateOne: { filter: { name: 'X' }, update: { $set: { bulk: true } } } },
|
|
1389
|
+
{ deleteOne: { filter: { name: 'Expired' } } },
|
|
1390
|
+
]);
|
|
1391
|
+
```
|
|
1392
|
+
|
|
1393
|
+
### Count & Distinct
|
|
1394
|
+
|
|
1395
|
+
```typescript
|
|
1396
|
+
const count = await collection.countDocuments({ status: 'active' });
|
|
1397
|
+
const estimated = await collection.estimatedDocumentCount();
|
|
1398
|
+
const names = await collection.distinct('name');
|
|
1399
|
+
```
|
|
1400
|
+
|
|
1401
|
+
---
|
|
1402
|
+
|
|
1403
|
+
## Wire Protocol Commands
|
|
1404
|
+
|
|
1405
|
+
| Category | Commands |
|
|
1406
|
+
|---|---|
|
|
1407
|
+
| **Handshake** | `hello`, `isMaster`, `ismaster` |
|
|
1408
|
+
| **CRUD** | `find`, `insert`, `update`, `delete`, `findAndModify`, `getMore`, `killCursors` |
|
|
1409
|
+
| **Aggregation** | `aggregate`, `count`, `distinct` |
|
|
1410
|
+
| **Indexes** | `createIndexes`, `dropIndexes`, `listIndexes` |
|
|
1411
|
+
| **Sessions** | `startSession`, `endSessions`, `killSessions` |
|
|
1412
|
+
| **Transactions** | `startTransaction`, `commitTransaction`, `abortTransaction` through driver sessions |
|
|
1413
|
+
| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `dbStats`, `collStats`, `connectionStatus`, `renameCollection` |
|
|
1414
|
+
|
|
1415
|
+
Advertises MongoDB wire protocol versions 0–21. The documented command surface is tested with the official `mongodb` Node.js driver version 7.5.x.
|
|
1416
|
+
|
|
1417
|
+
OP_MSG parsing rejects unknown required flags, invalid CRC-32C checksums,
|
|
1418
|
+
duplicate BSON keys, noncanonical array indexes, malformed section boundaries,
|
|
1419
|
+
multiple body sections, duplicate or colliding document-sequence identifiers,
|
|
1420
|
+
and missing or non-string `$db` fields. Early command-envelope validation then
|
|
1421
|
+
accepts document sequences only as `insert.documents`, `update.updates`, or
|
|
1422
|
+
`delete.deletes`; empty, oversized, extra, mismatched, and body-colliding forms
|
|
1423
|
+
reject before authentication or command resources are acquired. All three accepted
|
|
1424
|
+
forms are consumed as the corresponding bulk command argument. Unknown optional
|
|
1425
|
+
flags are ignored. A message is limited to 48,000,000 bytes, each BSON document to
|
|
1426
|
+
16,777,216 bytes, and BSON nesting to 100 levels. One message may contain at most
|
|
1427
|
+
128 sections, 10,000 document-sequence documents, 100,000 BSON elements, and
|
|
1428
|
+
100,000 BSON document or array containers. The advertised `maxWriteBatchSize` is
|
|
1429
|
+
therefore 10,000.
|
|
1430
|
+
|
|
1431
|
+
The `moreToCome` request flag is accepted only for the exact `endSessions` or
|
|
1432
|
+
`killSessions` cleanup envelope: an `admin` command body containing only the
|
|
1433
|
+
session array, `writeConcern: { w: 0 }`, and `$db: "admin"`, with no document
|
|
1434
|
+
sequence. It executes without a transport response and leaves the connection
|
|
1435
|
+
reusable. Every other `moreToCome` envelope closes before command routing or
|
|
1436
|
+
response-owned resource creation.
|
|
1437
|
+
|
|
1438
|
+
Raw receive buffers and parsed requests are charged separately against a
|
|
1439
|
+
process-wide 256 MiB encoded-byte admission budget. A partial frame is charged by
|
|
1440
|
+
its incrementally reserved receive capacity, not by the length in its prefix.
|
|
1441
|
+
Receive leases retain their high-water charge until the buffer is dropped or
|
|
1442
|
+
successfully compacted, and every allocation growth is reconciled before another
|
|
1443
|
+
socket read. Large unread pipelines remain charged and are compacted only at
|
|
1444
|
+
geometric utilization thresholds; the budget is an encoded-input bound rather
|
|
1445
|
+
than a claim about exact allocator heap usage.
|
|
1446
|
+
|
|
1447
|
+
### Read and Write Concerns and Wire Deadlines
|
|
1448
|
+
|
|
1449
|
+
NoSQLDB serves every database from exactly one voting node and states its
|
|
1450
|
+
concern profile in those terms instead of ignoring or blanket-rejecting the
|
|
1451
|
+
fields:
|
|
1452
|
+
|
|
1453
|
+
- **Write concern.** Every acknowledged write is fsynced to the write-ahead log
|
|
1454
|
+
and the data file before the response is sent, so `w: 1`, `w: "majority"`
|
|
1455
|
+
(the majority of one voter is that voter), and `j: true` are all satisfied by
|
|
1456
|
+
the same durable commit. `w: 0` is acknowledged the same way whenever a
|
|
1457
|
+
response is expected; the exact `writeConcern: { w: 0 }` `endSessions` and
|
|
1458
|
+
`killSessions` envelopes still run without a transport response. `wtimeout`
|
|
1459
|
+
bounds a replication wait that does not exist on one node and has no
|
|
1460
|
+
additional effect. `w: N` for `N > 1` fails with `UnsatisfiableWriteConcern`
|
|
1461
|
+
(code 100), tag-set modes fail with `UnknownReplWriteConcern` (code 79), and
|
|
1462
|
+
`j: true` fails with `BadValue` (code 2) on the in-memory storage backend,
|
|
1463
|
+
which does not journal. Write concern is accepted only on commands that
|
|
1464
|
+
write (including DDL, user management, `commitTransaction`, and
|
|
1465
|
+
`abortTransaction`) and is rejected with `InvalidOptions` (code 72) inside a
|
|
1466
|
+
multi-statement transaction, where it belongs on the commit.
|
|
1467
|
+
- **Read concern.** `local`, `available`, `majority`, and `linearizable` all
|
|
1468
|
+
observe the durably committed state, because nothing is acknowledged before
|
|
1469
|
+
it is durable and reads never observe uncommitted transaction buffers.
|
|
1470
|
+
`snapshot` is accepted only on the first statement of a multi-document
|
|
1471
|
+
transaction and is served by the database-wide transaction snapshot
|
|
1472
|
+
described below. Read concern is accepted on `find`, `count`, `distinct`,
|
|
1473
|
+
and `aggregate` only, and `afterClusterTime`, `atClusterTime`, and
|
|
1474
|
+
`afterOpTime` are rejected with `InvalidOptions` because a single node issues
|
|
1475
|
+
no cluster time.
|
|
1476
|
+
- **Deadlines.** `maxTimeMS` attaches a cooperative deadline to the command.
|
|
1477
|
+
It is checked at admission, during namespace-lock and maintenance-gate
|
|
1478
|
+
waits, before `find`, `count`, and `distinct` materialize documents, between
|
|
1479
|
+
documents in `insert` and `delete` batches, and immediately before a
|
|
1480
|
+
transaction commit is handed to storage; expiry fails with `MaxTimeMSExpired`
|
|
1481
|
+
(code 50). Database-permit waits keep their fixed internal timeout, and a
|
|
1482
|
+
single storage step or an in-progress `update` batch runs to completion
|
|
1483
|
+
before the next check, so a write that expires after its first documents
|
|
1484
|
+
were published reports the expiry while those documents remain durable,
|
|
1485
|
+
exactly like an interrupted MongoDB batch. The official driver transmits a
|
|
1486
|
+
transaction's `maxCommitTimeMS` as `maxTimeMS` on `commitTransaction`; a
|
|
1487
|
+
commit that expires before its batch is handed to storage publishes nothing,
|
|
1488
|
+
records the transaction as aborted, and carries the
|
|
1489
|
+
`UnknownTransactionCommitResult` label; the driver's commit retry then sees
|
|
1490
|
+
`NoSuchTransaction` with `TransientTransactionError` and restarts the whole
|
|
1491
|
+
transaction. `getMore` rejects `maxTimeMS` with `BadValue` because NoSQLDB
|
|
1492
|
+
has no awaitData cursors, and a literal `maxCommitTimeMS` field is rejected
|
|
1493
|
+
with `InvalidOptions`.
|
|
1494
|
+
|
|
1495
|
+
Callers may therefore rely on a successful NoSQLDB response as a durable,
|
|
1496
|
+
single-node acknowledgement. They must not read it as a replica-backed
|
|
1497
|
+
acknowledgement: majority semantics across several voters arrive with
|
|
1498
|
+
replication.
|
|
1499
|
+
|
|
1500
|
+
### Unsupported Administrative Commands
|
|
1501
|
+
|
|
1502
|
+
NoSQLDB returns `CommandNotFound` (code 59) for the wire commands `validate`,
|
|
1503
|
+
`explain`, legacy `authenticate`, `buildInfo`/`buildinfo`, `hostInfo`,
|
|
1504
|
+
`whatsmyuri`, `getLog`, `getCmdLineOpts`, `getParameter`, `setFreeMonitoring`,
|
|
1505
|
+
`currentOp`, `killOp`, `top`, `profile`, `compact`, `reIndex`, `fsync`, and
|
|
1506
|
+
`connPoolSync`. These commands previously had placeholder success responses but
|
|
1507
|
+
do not yet implement their advertised MongoDB semantics. Authentication and
|
|
1508
|
+
authorization errors retain precedence; otherwise rejection occurs before
|
|
1509
|
+
database publication permits, session or transaction creation, and command
|
|
1510
|
+
dispatch. This restriction applies to the wire `validate` command, not the
|
|
1511
|
+
supported offline `--validate-data` tool. `getFreeMonitoringStatus` remains
|
|
1512
|
+
available and reports NoSQLDB's fixed disabled state.
|
|
1513
|
+
|
|
1514
|
+
---
|
|
1515
|
+
|
|
1516
|
+
## Rust Crate Architecture 🦀
|
|
1517
|
+
|
|
1518
|
+
The Rust engine is organized as a Cargo workspace with 12 focused crates:
|
|
1519
|
+
|
|
1520
|
+
| Crate | Purpose |
|
|
1521
|
+
|---|---|
|
|
1522
|
+
| `rustdb` | Binary entry point: TCP/Unix listener, management IPC, CLI |
|
|
1523
|
+
| `rustdb-config` | Server configuration types (serde, camelCase JSON) |
|
|
1524
|
+
| `rustdb-wire` | Wire protocol parser/encoder (OP_MSG, OP_QUERY, OP_REPLY) |
|
|
1525
|
+
| `rustdb-query` | Query matcher, update engine, aggregation, sort, projection |
|
|
1526
|
+
| `rustdb-state` | Internal deterministic storage commands, stable retry identities, and durable receipts |
|
|
1527
|
+
| `rustdb-kernel` | Native paged committed-log kernel with stable timelines, contiguous LSNs, strict recovery and exact receipt replay |
|
|
1528
|
+
| `rustdb-replication` | Internal non-serving OpenRaft 0.9.25 one-voter fresh-root foundation; not linked into the released `rustdb` server |
|
|
1529
|
+
| `rustdb-storage` | Native catalogs, documents, exact counts, staged imports, reclamation and bounded OpLog |
|
|
1530
|
+
| `rustdb-index` | Native index records, query plans, uniqueness, bounded builds and TTL |
|
|
1531
|
+
| `rustdb-txn` | Transaction + session management with snapshot isolation |
|
|
1532
|
+
| `rustdb-auth` | SCRAM-SHA-256 credential handling, user metadata persistence, RBAC checks |
|
|
1533
|
+
| `rustdb-commands` | 40+ command handlers wiring everything together |
|
|
1534
|
+
|
|
1535
|
+
|
|
1536
|
+
The native MVCC engine serves both memory and file storage.
|
|
1537
|
+
Its Rust APIs use `mvcc` modules and `Mvcc` type names independently of npm
|
|
1538
|
+
versions and persisted format versions. The K2 kernel stores immutable paged
|
|
1539
|
+
roots, typed BSON records and checksummed commit markers in one append stream.
|
|
1540
|
+
Snapshots pin roots; normal writes append changed pages. Descriptor-relative
|
|
1541
|
+
access and the enclosing native file-root lease protect the entire lifecycle.
|
|
1542
|
+
Ambiguous publication fences the owner until recovery. Confirmed close drains
|
|
1543
|
+
requests, sessions, cursors and retained workers before releasing ownership.
|
|
1544
|
+
|
|
1545
|
+
The bounded page cache retains validated key prefixes for point lookups when
|
|
1546
|
+
large separators cannot fit as complete nodes. Prefix coverage grows with the
|
|
1547
|
+
probe length and preserves exact ordering; scans and writes use complete keys.
|
|
1548
|
+
This keeps unrelated small-key operations from repeatedly loading wide index
|
|
1549
|
+
separators without increasing the cache budget or changing the stored format.
|
|
1550
|
+
|
|
1551
|
+
Document, catalog, exact-count, natural-order, index-entry, unique-owner,
|
|
1552
|
+
authentication, fencing and retry-receipt records share atomic publications.
|
|
1553
|
+
Transactions retain changed documents and immutable read views with byte and
|
|
1554
|
+
concurrency admission; they do not copy a complete database snapshot or impose a
|
|
1555
|
+
10,000-document base-snapshot ceiling. Reads, `getMore`, indexes and streaming
|
|
1556
|
+
aggregation use pinned bounded pages. Transaction sorts use the shared external
|
|
1557
|
+
sort executor. Successful retryable statements retain exact results, including
|
|
1558
|
+
generated IDs and zero-match outcomes, across restart. Failed statements do not
|
|
1559
|
+
receive success receipts. Session termination drains active work and retires its
|
|
1560
|
+
cursors.
|
|
1561
|
+
|
|
1562
|
+
Custom indexes build in bounded batches under a writer reservation and become
|
|
1563
|
+
Ready together. Startup resumes interrupted builds under native ownership before
|
|
1564
|
+
serving. If a resumed build confirms duplicate keys or exceeds its work budget,
|
|
1565
|
+
its Building state remains intact and a secret-free warning identifies the need
|
|
1566
|
+
for explicit index repair. Reads avoid that index; indexed writes fail closed.
|
|
1567
|
+
Use the supported index drop/recreate operation to resolve it. Cancellation,
|
|
1568
|
+
corruption and uncertain publication fail startup. Held databases remain closed
|
|
1569
|
+
and are skipped during recovery; releasing a hold does not run extra mutations.
|
|
1570
|
+
Their interrupted indexes can recover on a later restart.
|
|
1571
|
+
|
|
1572
|
+
Index names and aliases, sparse and unique constraints, empty collections,
|
|
1573
|
+
restricted-view catalogs and exact BSON survive supported database export/import.
|
|
1574
|
+
Native import replaces the entire data catalog and its publication receipt in
|
|
1575
|
+
one commit, preserving managed principal and allocation identities. Old pinned
|
|
1576
|
+
reads retain the previous catalog. Exact fenced retries append nothing; held
|
|
1577
|
+
replacements remain inaccessible until their exact receipt is released.
|
|
1578
|
+
|
|
1579
|
+
The serving profile is:
|
|
1580
|
+
|
|
1581
|
+
| Surface | Native support |
|
|
1582
|
+
|---|---|
|
|
1583
|
+
| Runtime | Linux and macOS; memory or current native file storage |
|
|
1584
|
+
| Wire data | `find`, `count`, `distinct`, `aggregate`, `getMore`, `killCursors`, `insert`, `update`, `delete`, `findAndModify`; supported `$out`/`$merge` |
|
|
1585
|
+
| Catalog/admin | Collection/database/index DDL, rename, listings, supported statistics and diagnostics, user/role administration |
|
|
1586
|
+
| Auth/sessions | SCRAM-SHA-256, logical sessions, snapshot transactions, commit/abort and durable retryable writes |
|
|
1587
|
+
| Management | Health, tenants, grants, attestation, fencing, publication holds, export/import and content digest |
|
|
1588
|
+
| Debug | Bounded current-session OpLog, atomic data revert, document/collection pages and dashboard |
|
|
1589
|
+
| Maintenance | Bounded TTL expiry, retired record reclamation, orphan import retirement and generation compaction |
|
|
1590
|
+
| Stopped storage | Native inspection and relocation; isolated authenticated rehearsal on its qualified platform profile |
|
|
1591
|
+
| Import/control admission | 100 MiB encoded import; bounded private staging and a small final catalog/receipt commit. Other control requests: 1 MiB |
|
|
1592
|
+
| External cluster | Replication and HA remain unsupported; the one-voter OpenRaft foundation is not linked into serving |
|
|
1593
|
+
|
|
1594
|
+
These are operation and resource budgets, not total database/document limits.
|
|
1595
|
+
Unsupported options and commands reject explicitly. Imports that cannot fit one
|
|
1596
|
+
derived document or index entry within the native commit budget reject before
|
|
1597
|
+
publishing the replacement. Reclamation preserves pinned roots and retires parent
|
|
1598
|
+
identities only after their children have been handled. Automatic compaction
|
|
1599
|
+
requires at least 64 MiB of growth and 50% growth over its prior baseline.
|
|
1600
|
+
|
|
1601
|
+
There is one serving engine and no Bitcask fallback. No historical storage or
|
|
1602
|
+
authentication conversion runs during startup or rehearsal. Conversion code,
|
|
1603
|
+
when explicitly commissioned, belongs exclusively in top-level `ts_migration/`.
|
|
1604
|
+
|
|
1605
|
+
NoSQLDB makes no general MongoDB performance or feature-parity claim.
|
|
1606
|
+
Versioned packages contain Linux amd64/arm64 and macOS amd64/arm64 native
|
|
1607
|
+
executables through [@git.zone/tsrust](https://www.npmjs.com/package/@git.zone/tsrust).
|
|
1608
|
+
|
|
1609
|
+
### Multi-host release artifacts
|
|
1610
|
+
|
|
1611
|
+
Consumers obtain the matching engines from the same versioned
|
|
1612
|
+
`@lossless.org/nosqldb` npm tarball on npmjs or Verdaccio. It includes
|
|
1613
|
+
`dist_rust/rustdb_linux_amd64`, `dist_rust/rustdb_linux_arm64`,
|
|
1614
|
+
`dist_rust/rustdb_macos_amd64` and `dist_rust/rustdb_macos_arm64`, each accompanied
|
|
1615
|
+
by `<binary>.tsrust-build.json`. These provenance files contain `binarySha256`,
|
|
1616
|
+
the package version, Git commit and target. Verify the tarball's registry
|
|
1617
|
+
integrity and each executable's SHA-256 against that matching provenance file
|
|
1618
|
+
before installation. Both Linux artifacts are statically linked. Spark can
|
|
1619
|
+
extract and install these files independently of its compiled JavaScript bundle;
|
|
1620
|
+
NoSQLDB does not download engines at startup.
|
|
1621
|
+
|
|
1622
|
+
Normal `tsrust` builds select only the current host's partition from `targetsByHost`. Releases use `tsrust matrix` to coordinate both partitions after the version commit exists, then publish only after native verification passes and every exact-commit binary has executed and passed strict provenance assembly.
|
|
1623
|
+
|
|
1624
|
+
The release coordinator needs Git, Node.js, pnpm and SSH access to both configured builders. The Linux amd64 builder needs `aarch64-linux-gnu-gcc` and a registered Linux arm64 `binfmt` interpreter. Linux file storage requires kernel 5.6 or newer with `openat2` enabled. Cross-architecture file qualification requires QEMU linux-user 9.2 or newer; earlier versions cannot execute the secure path-resolution operations even when a simple executable probe passes. The Apple Silicon builder needs Rosetta. The matrix qualifies macOS first and then Linux; both partitions must pass before artifact assembly. Set these environment variables to authorized builders and their dedicated temporary roots:
|
|
1625
|
+
|
|
1626
|
+
```bash
|
|
1627
|
+
export NOSQLDB_RELEASE_MACOS_SSH="release-user@mac-builder.example"
|
|
1628
|
+
export NOSQLDB_RELEASE_MACOS_TEMP_ROOT="/absolute/dedicated/nosqldb-release"
|
|
1629
|
+
export NOSQLDB_RELEASE_LINUX_SSH="release-user@linux-builder.example"
|
|
1630
|
+
export NOSQLDB_RELEASE_LINUX_TEMP_ROOT="/absolute/dedicated/nosqldb-release"
|
|
1631
|
+
```
|
|
1632
|
+
|
|
1633
|
+
`gitzone release` first runs `pnpm run release:check-builders`, before creating a version commit or tag. It verifies the project-local tsrust version, compiles and executes the required architecture probes, and rejects missing cross-toolchains or emulation, an unsuitable macOS host, missing tools, or an unsafe remote temporary root. The release build then:
|
|
1634
|
+
|
|
1635
|
+
1. Builds the TypeScript package and debug UI bundle from the version commit.
|
|
1636
|
+
2. Streams an exact Git bundle into isolated Linux and macOS SSH worker workspaces and installs each frozen lockfile.
|
|
1637
|
+
3. Runs the configured `release:verify-native` command on each worker: the full Rust workspace, TypeScript test checks, public API suite, and explicit-engine authentication/ownership/rehearsal tests through the worker's other packaged architecture. Tests use short, private disposable filesystem roots under `/var/tmp` on Linux and canonical `/tmp` on macOS. Linux builders must provide `/var/tmp` on a native-supported durable filesystem; tmpfs is rejected. Failures retain diagnostics and prevent assembly.
|
|
1638
|
+
4. Builds and executes the Linux amd64, Linux arm64, macOS arm64, and macOS amd64 binaries on their assigned workers.
|
|
1639
|
+
5. Retrieves bounded artifact sets and transactionally publishes the complete provenance-checked matrix into `dist_rust/`.
|
|
1640
|
+
|
|
1641
|
+
No host address or path is stored in the repository. A failed run retains its unique local diagnostics and owner-marked remote diagnostics, then prints their paths. Remote cleanup is an owner-validated publication gate, and an incomplete matrix never replaces the prior `dist_rust/`.
|
|
1642
|
+
|
|
1643
|
+
### Storage Engine Reliability 🔒
|
|
1644
|
+
|
|
1645
|
+
- Atomic native publications include data, indexes, authentication, fences and
|
|
1646
|
+
durable outcomes. File writes are acknowledged after durable commit.
|
|
1647
|
+
- Immutable roots keep reads stable through concurrent writes and compaction.
|
|
1648
|
+
- Checksums, frame identities and contiguous positions validate committed
|
|
1649
|
+
history. Only a valid incomplete final append is repairable during owned
|
|
1650
|
+
startup; complete or interior corruption is rejected.
|
|
1651
|
+
- Uncertain publication fences the engine until confirmed recovery.
|
|
1652
|
+
- Bounded maintenance reclaims unreachable records and obsolete generations
|
|
1653
|
+
after their readers release them.
|
|
1654
|
+
- Unix listener startup preserves active sockets and non-socket targets.
|
|
1655
|
+
Verified stale sockets are recoverable; cleanup checks the owned inode.
|
|
1656
|
+
|
|
1657
|
+
### Data Integrity CLI 🔍
|
|
1658
|
+
|
|
1659
|
+
```bash
|
|
1660
|
+
./dist_rust/rustdb_linux_amd64 --validate-data /path/to/data
|
|
1661
|
+
# {"schemaVersion":1,"databases":[{"name":"mydb","collections":["users"]}]}
|
|
1662
|
+
```
|
|
1663
|
+
|
|
1664
|
+
The CLI uses the same stopped native namespace inspector. It validates storage,
|
|
1665
|
+
catalogs and indexes while retaining the existing native leases, without starting
|
|
1666
|
+
a listener or repairing the source. It returns JSON on success and a nonzero exit
|
|
1667
|
+
on failure. No configuration file is required. The operation has a one-hour
|
|
1668
|
+
deadline, 100,000 filesystem-entry and 1 TiB byte limits, and bounded native
|
|
1669
|
+
record/result admission. This is storage integrity evidence; it does not replace
|
|
1670
|
+
the full authenticated rehearsal or production ownership protocol.
|
|
1671
|
+
|
|
1672
|
+
---
|
|
1673
|
+
|
|
1674
|
+
## Testing Example
|
|
1675
|
+
|
|
1676
|
+
```typescript
|
|
1677
|
+
import { expect, tap } from '@git.zone/tstest/tapbundle';
|
|
1678
|
+
import { NoSqlDbServer } from '@lossless.org/nosqldb';
|
|
1679
|
+
import { MongoClient } from 'mongodb';
|
|
1680
|
+
|
|
1681
|
+
let server: NoSqlDbServer;
|
|
1682
|
+
let client: MongoClient;
|
|
1683
|
+
|
|
1684
|
+
tap.test('setup', async () => {
|
|
1685
|
+
server = new NoSqlDbServer({ port: 27117 });
|
|
1686
|
+
await server.start();
|
|
1687
|
+
client = new MongoClient('mongodb://127.0.0.1:27117', { directConnection: true });
|
|
1688
|
+
await client.connect();
|
|
1689
|
+
});
|
|
1690
|
+
|
|
1691
|
+
tap.test('should insert and find', async () => {
|
|
1692
|
+
const col = client.db('test').collection('items');
|
|
1693
|
+
await col.insertOne({ name: 'Widget', price: 9.99 });
|
|
1694
|
+
const item = await col.findOne({ name: 'Widget' });
|
|
1695
|
+
expect(item?.price).toEqual(9.99);
|
|
1696
|
+
});
|
|
1697
|
+
|
|
1698
|
+
tap.test('should track changes in oplog', async () => {
|
|
1699
|
+
const oplog = await server.getOpLog();
|
|
1700
|
+
expect(oplog.entries.length).toBeGreaterThan(0);
|
|
1701
|
+
expect(oplog.entries[0].op).toEqual('insert');
|
|
1702
|
+
});
|
|
1703
|
+
|
|
1704
|
+
tap.test('teardown', async () => {
|
|
1705
|
+
await client.close();
|
|
1706
|
+
await server.stop();
|
|
1707
|
+
});
|
|
1708
|
+
|
|
1709
|
+
export default tap.start();
|
|
1710
|
+
```
|
|
1711
|
+
|
|
1712
|
+
---
|
|
1713
|
+
|
|
1714
|
+
## License and Legal Information
|
|
1715
|
+
|
|
1716
|
+
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.
|
|
1717
|
+
|
|
1718
|
+
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
|
|
1719
|
+
|
|
1720
|
+
### Trademarks
|
|
1721
|
+
|
|
1722
|
+
This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.
|
|
1723
|
+
|
|
1724
|
+
Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.
|
|
1725
|
+
|
|
1726
|
+
### Company Information
|
|
1727
|
+
|
|
1728
|
+
Task Venture Capital GmbH
|
|
1729
|
+
Registered at District Court Bremen HRB 35230 HB, Germany
|
|
1730
|
+
|
|
1731
|
+
For any legal inquiries or further information, please contact us via email at hello@task.vc.
|
|
1732
|
+
|
|
1733
|
+
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
|