rumongo 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/BENCHMARKS.md +448 -0
- package/LICENSE +21 -0
- package/MIGRATION.md +69 -0
- package/README.md +255 -0
- package/dist/index.d.ts +42 -0
- package/dist/index.js +112 -0
- package/dist/model.d.ts +27 -0
- package/dist/model.js +50 -0
- package/dist/shadow.d.ts +14 -0
- package/dist/shadow.js +53 -0
- package/index.d.ts +51 -0
- package/index.js +317 -0
- package/package.json +50 -0
- package/rumongo.linux-x64-gnu.node +0 -0
- package/worker/pool-worker.js +47 -0
- package/worker/pool.d.ts +42 -0
- package/worker/pool.js +100 -0
package/README.md
ADDED
|
@@ -0,0 +1,255 @@
|
|
|
1
|
+
# rumongo
|
|
2
|
+
|
|
3
|
+
**Rust + Mongo.** A Rust-native MongoDB **read** driver for Node.js (napi-rs).
|
|
4
|
+
Faster reads than the official Node driver and Mongoose by doing BSON parsing in
|
|
5
|
+
Rust — pipelined fetch, off-thread parallel parse, and optional lazy zero-copy
|
|
6
|
+
field access.
|
|
7
|
+
|
|
8
|
+
> Read path only. Writes / hooks / virtuals / populate / validation are **not**
|
|
9
|
+
> covered — keep the official driver or Mongoose for those. See
|
|
10
|
+
> [MIGRATION.md](MIGRATION.md).
|
|
11
|
+
|
|
12
|
+
## Performance
|
|
13
|
+
|
|
14
|
+
- **1.6–3.7× faster reads** than the official Node driver (by projection width).
|
|
15
|
+
- **~2× vs Mongoose `.lean()`, ~5× vs hydrated Mongoose.**
|
|
16
|
+
- **~10× lower event-loop jitter** on a single query; **near-zero** with lazy or
|
|
17
|
+
worker-thread offload.
|
|
18
|
+
- **7.3× vs the raw driver** when reading a few fields of a wide doc (lazy).
|
|
19
|
+
|
|
20
|
+
### Final benchmark results
|
|
21
|
+
|
|
22
|
+
Consolidated preset suite (`bench/suite.js`), 30k docs over a 45-field document,
|
|
23
|
+
warmup + 6 iters, mean ± sd. Local MongoDB 8.0, Node v20.4, 12 cores.
|
|
24
|
+
[BENCHMARKS.md](BENCHMARKS.md) keeps the **full progressive log** (every phase);
|
|
25
|
+
the tables below are the **final snapshot**.
|
|
26
|
+
|
|
27
|
+
**A) Driver `find` — official Node driver vs rumongo (eager)**
|
|
28
|
+
|
|
29
|
+
| preset | fields | official (ms) | rumongo (ms) | speedup |
|
|
30
|
+
|---|---|---|---|---|
|
|
31
|
+
| few | 4 | 649 ± 95 | 178 ± 17 | **3.65×** |
|
|
32
|
+
| small | 9 | 792 ± 62 | 304 ± 16 | 2.61× |
|
|
33
|
+
| medium | 15 | 687 ± 49 | 418 ± 54 | 1.64× |
|
|
34
|
+
| large | 35 | 1532 ± 132 | 841 ± 61 | 1.82× |
|
|
35
|
+
| full | 45 | 2032 ± 135 | 1031 ± 59 | 1.97× |
|
|
36
|
+
|
|
37
|
+
**B) ODM — Mongoose `.lean()` vs rumongo Model**
|
|
38
|
+
|
|
39
|
+
| preset | fields | mongoose (ms) | Model (ms) | speedup |
|
|
40
|
+
|---|---|---|---|---|
|
|
41
|
+
| few | 4 | 477 ± 53 | 177 ± 18 | 2.69× |
|
|
42
|
+
| small | 9 | 559 ± 35 | 284 ± 31 | 1.97× |
|
|
43
|
+
| medium | 15 | 680 ± 68 | 405 ± 35 | 1.68× |
|
|
44
|
+
| large | 35 | 1455 ± 24 | 850 ± 65 | 1.71× |
|
|
45
|
+
| full | 45 | 2041 ± 167 | 1031 ± 98 | 1.98× |
|
|
46
|
+
|
|
47
|
+
**C) Event-loop jitter** (full preset, single query): official `149.2ms` ·
|
|
48
|
+
rumongo `13.8ms` (~10× lower).
|
|
49
|
+
|
|
50
|
+
**D) Lazy / wide-doc, few-field read** (20k docs × 33 fields, read 2 fields, 10
|
|
51
|
+
concurrent): rumongo lazy **7.3× vs the official driver**, 2.2× vs rumongo eager.
|
|
52
|
+
|
|
53
|
+
> ⚠️ All numbers are **localhost** (≈0 network latency) — a lower bound for
|
|
54
|
+
> pipeline/concurrency wins, upper bound for CPU-bound wins. The headline 15–20×
|
|
55
|
+
> shows up only under lazy/narrow-read or worker-offload patterns, not eager
|
|
56
|
+
> full-document reads.
|
|
57
|
+
|
|
58
|
+
## How it works (in plain terms)
|
|
59
|
+
|
|
60
|
+
MongoDB sends every document over the wire as **BSON** — a compact binary blob.
|
|
61
|
+
Before your code can use it, something has to **decode** that binary into objects
|
|
62
|
+
you can read (`doc.name`, `doc.age`, …). The whole speed difference comes down to
|
|
63
|
+
two questions: **who** does the decoding (the single JavaScript thread, or many
|
|
64
|
+
Rust CPU cores) and **how much** it decodes (every field, or only the ones you touch).
|
|
65
|
+
|
|
66
|
+
### 1. A normal `find` — who decodes the data?
|
|
67
|
+
|
|
68
|
+
The official driver does all the binary→object work on the **one** thread that
|
|
69
|
+
also runs your entire app. rumongo hands that work to Rust, spread across CPU
|
|
70
|
+
cores, **off** the main thread — so your app stays responsive.
|
|
71
|
+
|
|
72
|
+
```mermaid
|
|
73
|
+
flowchart TB
|
|
74
|
+
subgraph OFF["🔵 Official Node driver"]
|
|
75
|
+
direction TB
|
|
76
|
+
O1["MongoDB sends BSON (binary)"]
|
|
77
|
+
O2["JS main thread decodes<br/>every field of every doc"]
|
|
78
|
+
O3["Builds thousands of JS objects<br/>(heavy garbage collection)"]
|
|
79
|
+
O4["Your code gets the docs"]
|
|
80
|
+
O1 --> O2 --> O3 --> O4
|
|
81
|
+
OX(["⚠️ All on the ONE thread that<br/>runs your app → event loop frozen"])
|
|
82
|
+
O2 -.-> OX
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
subgraph RU["🟢 rumongo (Rust)"]
|
|
86
|
+
direction TB
|
|
87
|
+
R1["MongoDB sends BSON (binary)"]
|
|
88
|
+
R2["Rust fetches batches in a pipeline<br/>(network + parsing overlap)"]
|
|
89
|
+
R3["Many CPU cores decode<br/>batches in parallel"]
|
|
90
|
+
R4["Hand finished result back to JS"]
|
|
91
|
+
R5["Your code gets the docs"]
|
|
92
|
+
R1 --> R2 --> R3 --> R4 --> R5
|
|
93
|
+
RX(["✅ Done in Rust, OFF the JS thread<br/>→ main event loop stays free"])
|
|
94
|
+
R3 -.-> RX
|
|
95
|
+
end
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### 2. `findLazy` — how much does it decode?
|
|
99
|
+
|
|
100
|
+
Each document may have 25 fields, but your loop might read only 2. The official
|
|
101
|
+
driver decodes **all** of them up front. rumongo keeps the doc as raw bytes and
|
|
102
|
+
decodes a field **only when you touch it**.
|
|
103
|
+
|
|
104
|
+
```mermaid
|
|
105
|
+
flowchart LR
|
|
106
|
+
subgraph OFFL["🔵 Official"]
|
|
107
|
+
A1["Get a doc"] --> A2["Decode ALL 25 fields<br/>up front"] --> A3["You read 2 →<br/>23 fields of work wasted"]
|
|
108
|
+
end
|
|
109
|
+
subgraph RUL["🟢 rumongo findLazy"]
|
|
110
|
+
B1["Get a doc,<br/>keep it as raw bytes"] --> B2["Decode a field ONLY<br/>when accessed (doc.name)"] --> B3["You read 2 →<br/>decode 2, skip 23"]
|
|
111
|
+
end
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### 3. Concurrent load — why "jitter" stays low
|
|
115
|
+
|
|
116
|
+
"Jitter" = how long the event loop froze, i.e. how unresponsive your app got to
|
|
117
|
+
*other* users while a query was being decoded. Under many concurrent queries the
|
|
118
|
+
official driver piles every decode onto the single JS thread; rumongo keeps that
|
|
119
|
+
work in Rust, off-thread.
|
|
120
|
+
|
|
121
|
+
```mermaid
|
|
122
|
+
sequenceDiagram
|
|
123
|
+
participant App as Your app (event loop)
|
|
124
|
+
participant JS as Official driver
|
|
125
|
+
participant Rust as rumongo (Rust threads)
|
|
126
|
+
|
|
127
|
+
Note over App,JS: Official — decode runs ON the event loop
|
|
128
|
+
App->>JS: 10 queries at once
|
|
129
|
+
JS-->>JS: decode all BSON on the JS thread
|
|
130
|
+
Note over App: ⛔ loop frozen — other users wait (high jitter)
|
|
131
|
+
JS-->>App: results
|
|
132
|
+
|
|
133
|
+
Note over App,Rust: rumongo — decode runs OFF the event loop
|
|
134
|
+
App->>Rust: 10 queries at once
|
|
135
|
+
Rust-->>Rust: decode BSON on Rust CPU cores
|
|
136
|
+
Note over App: ✅ loop free — app keeps answering (low jitter)
|
|
137
|
+
Rust-->>App: results
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
**One line:** the official driver does *all* the binary→object work on the single
|
|
141
|
+
thread that also runs your whole app; rumongo pushes that work into Rust on
|
|
142
|
+
multiple cores, off the main thread, and — in lazy mode — only does the part you
|
|
143
|
+
actually use.
|
|
144
|
+
|
|
145
|
+
## Install / build
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
npm install # deps
|
|
149
|
+
npm run build # compile the Rust addon (release) -> rumongo.<platform>.node
|
|
150
|
+
npm run build:ts # compile the TypeScript layer -> dist/
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Requires a Rust toolchain and a reachable MongoDB. Target: linux-x64 (current).
|
|
154
|
+
|
|
155
|
+
## Usage
|
|
156
|
+
|
|
157
|
+
```js
|
|
158
|
+
import { MongoClient } from 'rumongo'
|
|
159
|
+
|
|
160
|
+
const client = await MongoClient.connect('mongodb://localhost:27017', {
|
|
161
|
+
maxPoolSize: 20,
|
|
162
|
+
serverSelectionTimeoutMs: 5000,
|
|
163
|
+
})
|
|
164
|
+
const users = client.collection('app', 'users')
|
|
165
|
+
|
|
166
|
+
// eager: plain JS objects
|
|
167
|
+
const all = await users.find({ active: true }, { sort: { age: -1 }, limit: 100 })
|
|
168
|
+
|
|
169
|
+
await client.close() // IMPORTANT: stops background monitors so the process exits
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
### Three read APIs (pick by shape)
|
|
173
|
+
|
|
174
|
+
| API | use when |
|
|
175
|
+
|---|---|
|
|
176
|
+
| `collection.find(filter, opts)` | small/medium results; returns plain objects |
|
|
177
|
+
| `collection.findLazy(filter, opts)` | wide docs, few fields read — fields parse on access |
|
|
178
|
+
| `collection.findCursor(filter, opts)` → `nextBatch()` | large/streaming results; bounded memory |
|
|
179
|
+
|
|
180
|
+
```js
|
|
181
|
+
// lazy: only the fields you touch are parsed
|
|
182
|
+
const docs = await users.findLazy({}, { limit: 1000 })
|
|
183
|
+
for (const d of docs) console.log(d.name) // parses just `name`
|
|
184
|
+
|
|
185
|
+
// streaming cursor: process a batch at a time (bounded memory)
|
|
186
|
+
const cur = await users.findCursor({}, { batchSize: 1000 })
|
|
187
|
+
let batch
|
|
188
|
+
while ((batch = await cur.nextBatch()) !== null) {
|
|
189
|
+
for (const d of batch) process(d)
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
### Mongoose-style Model (projection pushdown)
|
|
194
|
+
|
|
195
|
+
```js
|
|
196
|
+
import { Model } from 'rumongo'
|
|
197
|
+
const User = Model.define(users, { name: 1, age: 1 }) // schema fields = projection
|
|
198
|
+
await User.find({ age: { $gte: 18 } }, { sort: { age: 1 } })
|
|
199
|
+
await User.findOne({ name: 'Ann' })
|
|
200
|
+
await User.findById('507f1f77bcf86cd799439011') // 24-char hex string
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
### Filters with BSON types (Extended JSON)
|
|
204
|
+
|
|
205
|
+
Filters cross the JS↔Rust boundary as JSON, so use Extended JSON for BSON types:
|
|
206
|
+
|
|
207
|
+
```js
|
|
208
|
+
await users.find({ _id: { $oid: '507f1f77bcf86cd799439011' } })
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
### Worker pool (opt-in) — keep the main loop free under heavy load
|
|
212
|
+
|
|
213
|
+
Run aggregation/transform work inside worker threads; only the result returns to
|
|
214
|
+
main, so the event loop stays responsive. Best for "do the work, return a small
|
|
215
|
+
result" (counts, sums, transforms, exports).
|
|
216
|
+
|
|
217
|
+
```js
|
|
218
|
+
import { WorkerPool } from 'rumongo'
|
|
219
|
+
const pool = await WorkerPool.create({ uri, size: 6 })
|
|
220
|
+
const { acc } = await pool.reduce('app', 'users', { active: true }, {}, (a, d) => a + d.age, 0)
|
|
221
|
+
await pool.close()
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
### Shadow mode — validate before cutover
|
|
225
|
+
|
|
226
|
+
Run rumongo alongside the official driver, compare, log divergences, but always
|
|
227
|
+
return the official result.
|
|
228
|
+
|
|
229
|
+
```js
|
|
230
|
+
import { shadow } from 'rumongo'
|
|
231
|
+
const res = await shadow(
|
|
232
|
+
() => rumongoColl.find(q),
|
|
233
|
+
() => officialColl.find(q).toArray(),
|
|
234
|
+
(diff) => logger.warn('divergence', diff),
|
|
235
|
+
)
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
## Debugging
|
|
239
|
+
|
|
240
|
+
Set `RUMONGO_DEBUG=1` to log each query's collection, doc count, and elapsed time
|
|
241
|
+
to stderr.
|
|
242
|
+
|
|
243
|
+
## Tests
|
|
244
|
+
|
|
245
|
+
```bash
|
|
246
|
+
node --test __tests__/parity/ # 23 parity tests vs official driver
|
|
247
|
+
node --test __tests__/model/ # 9 Model tests vs Mongoose
|
|
248
|
+
node --test __tests__/lazy/ # lazy field-access tests
|
|
249
|
+
node --test __tests__/integration/ # connectivity, pipeline, leak
|
|
250
|
+
node bench/suite.js # benchmark suite
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
## License
|
|
254
|
+
|
|
255
|
+
MIT
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
import { MongoClient as NativeClient, FindCursor as NativeCursor } from '../index';
|
|
2
|
+
export { WorkerPool } from '../worker/pool';
|
|
3
|
+
export type { WorkerPoolOptions } from '../worker/pool';
|
|
4
|
+
export { Model, Schema } from './model';
|
|
5
|
+
export type { SchemaDefinition, QueryOptions } from './model';
|
|
6
|
+
export { shadow, deepEqual } from './shadow';
|
|
7
|
+
export type { Divergence } from './shadow';
|
|
8
|
+
export interface ConnectOptions {
|
|
9
|
+
maxPoolSize?: number;
|
|
10
|
+
minPoolSize?: number;
|
|
11
|
+
connectTimeoutMs?: number;
|
|
12
|
+
serverSelectionTimeoutMs?: number;
|
|
13
|
+
appName?: string;
|
|
14
|
+
}
|
|
15
|
+
export declare class MongoClient {
|
|
16
|
+
private readonly native;
|
|
17
|
+
private constructor();
|
|
18
|
+
static connect(uri: string, options?: ConnectOptions): Promise<MongoClient>;
|
|
19
|
+
collection(db: string, name: string): Collection;
|
|
20
|
+
close(): Promise<void>;
|
|
21
|
+
}
|
|
22
|
+
export declare class Collection {
|
|
23
|
+
private readonly native;
|
|
24
|
+
private readonly db;
|
|
25
|
+
private readonly name;
|
|
26
|
+
constructor(native: NativeClient, db: string, name: string);
|
|
27
|
+
/** Eager: fully parsed plain JS objects. */
|
|
28
|
+
find<T = Record<string, unknown>>(filter?: object, options?: object): Promise<T[]>;
|
|
29
|
+
/** Lazy: fields parse only when read. Returns Proxy-wrapped documents. */
|
|
30
|
+
findLazy(filter?: object, options?: object): Promise<Record<string, unknown>[]>;
|
|
31
|
+
/** Streaming: pull one batch of (Proxy-wrapped) docs at a time. Bounded memory. */
|
|
32
|
+
findCursor(filter?: object, options?: object): Promise<LazyCursor>;
|
|
33
|
+
}
|
|
34
|
+
/** Wraps the native cursor; each batch's docs are Proxy-wrapped for dot access. */
|
|
35
|
+
export declare class LazyCursor {
|
|
36
|
+
private readonly native;
|
|
37
|
+
constructor(native: NativeCursor);
|
|
38
|
+
/** Next batch of docs, or null when exhausted. */
|
|
39
|
+
nextBatch(): Promise<Record<string, unknown>[] | null>;
|
|
40
|
+
/** Async iterator over individual docs across all batches. */
|
|
41
|
+
[Symbol.asyncIterator](): AsyncGenerator<Record<string, unknown>>;
|
|
42
|
+
}
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
// Public TypeScript API for rumongo.
|
|
3
|
+
// - find() eager: returns plain JS objects (parsed from JSON strings)
|
|
4
|
+
// - findLazy() Phase 4: returns Proxy-wrapped RawDoc — fields parse on access
|
|
5
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
+
exports.LazyCursor = exports.Collection = exports.MongoClient = exports.deepEqual = exports.shadow = exports.Schema = exports.Model = exports.WorkerPool = void 0;
|
|
7
|
+
const index_1 = require("../index");
|
|
8
|
+
// Opt-in worker pool (see worker/pool.js). Re-exported as part of the public API.
|
|
9
|
+
var pool_1 = require("../worker/pool");
|
|
10
|
+
Object.defineProperty(exports, "WorkerPool", { enumerable: true, get: function () { return pool_1.WorkerPool; } });
|
|
11
|
+
// Mongoose-style Model + projection pushdown.
|
|
12
|
+
var model_1 = require("./model");
|
|
13
|
+
Object.defineProperty(exports, "Model", { enumerable: true, get: function () { return model_1.Model; } });
|
|
14
|
+
Object.defineProperty(exports, "Schema", { enumerable: true, get: function () { return model_1.Schema; } });
|
|
15
|
+
// Shadow mode (validate rumongo vs official in production).
|
|
16
|
+
var shadow_1 = require("./shadow");
|
|
17
|
+
Object.defineProperty(exports, "shadow", { enumerable: true, get: function () { return shadow_1.shadow; } });
|
|
18
|
+
Object.defineProperty(exports, "deepEqual", { enumerable: true, get: function () { return shadow_1.deepEqual; } });
|
|
19
|
+
// Wrap a RawDoc so `doc.field` parses just that field on demand, while spread
|
|
20
|
+
// (`{...doc}`) and JSON.stringify still see all fields (via ownKeys + descriptors).
|
|
21
|
+
function wrapRawDoc(doc) {
|
|
22
|
+
return new Proxy(doc, {
|
|
23
|
+
get(target, prop) {
|
|
24
|
+
if (typeof prop !== 'string')
|
|
25
|
+
return undefined;
|
|
26
|
+
if (prop === 'toObject' || prop === 'toJSON')
|
|
27
|
+
return () => target.toObject();
|
|
28
|
+
if (prop === 'toString')
|
|
29
|
+
return () => JSON.stringify(target.toObject());
|
|
30
|
+
if (prop === '__isRawDoc')
|
|
31
|
+
return true;
|
|
32
|
+
if (prop === 'then')
|
|
33
|
+
return undefined; // never look thenable
|
|
34
|
+
return target.getField(prop);
|
|
35
|
+
},
|
|
36
|
+
has(target, prop) {
|
|
37
|
+
return typeof prop === 'string' && target.keys().includes(prop);
|
|
38
|
+
},
|
|
39
|
+
ownKeys(target) {
|
|
40
|
+
return target.keys();
|
|
41
|
+
},
|
|
42
|
+
getOwnPropertyDescriptor(target, prop) {
|
|
43
|
+
if (typeof prop === 'string' && target.keys().includes(prop)) {
|
|
44
|
+
return {
|
|
45
|
+
enumerable: true,
|
|
46
|
+
configurable: true,
|
|
47
|
+
value: target.getField(prop),
|
|
48
|
+
};
|
|
49
|
+
}
|
|
50
|
+
return undefined;
|
|
51
|
+
},
|
|
52
|
+
});
|
|
53
|
+
}
|
|
54
|
+
class MongoClient {
|
|
55
|
+
constructor(native) {
|
|
56
|
+
this.native = native;
|
|
57
|
+
}
|
|
58
|
+
static async connect(uri, options) {
|
|
59
|
+
const opts = options ? JSON.stringify(options) : undefined;
|
|
60
|
+
return new MongoClient(await index_1.MongoClient.connect(uri, opts));
|
|
61
|
+
}
|
|
62
|
+
collection(db, name) {
|
|
63
|
+
return new Collection(this.native, db, name);
|
|
64
|
+
}
|
|
65
|
+
async close() {
|
|
66
|
+
await this.native.close();
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
exports.MongoClient = MongoClient;
|
|
70
|
+
class Collection {
|
|
71
|
+
constructor(native, db, name) {
|
|
72
|
+
this.native = native;
|
|
73
|
+
this.db = db;
|
|
74
|
+
this.name = name;
|
|
75
|
+
}
|
|
76
|
+
/** Eager: fully parsed plain JS objects. */
|
|
77
|
+
async find(filter = {}, options = {}) {
|
|
78
|
+
const rows = await this.native.find(this.db, this.name, JSON.stringify(filter), JSON.stringify(options));
|
|
79
|
+
return rows.map((r) => JSON.parse(r));
|
|
80
|
+
}
|
|
81
|
+
/** Lazy: fields parse only when read. Returns Proxy-wrapped documents. */
|
|
82
|
+
async findLazy(filter = {}, options = {}) {
|
|
83
|
+
const docs = await this.native.findLazy(this.db, this.name, JSON.stringify(filter), JSON.stringify(options));
|
|
84
|
+
return docs.map(wrapRawDoc);
|
|
85
|
+
}
|
|
86
|
+
/** Streaming: pull one batch of (Proxy-wrapped) docs at a time. Bounded memory. */
|
|
87
|
+
async findCursor(filter = {}, options = {}) {
|
|
88
|
+
const cur = await this.native.findCursor(this.db, this.name, JSON.stringify(filter), JSON.stringify(options));
|
|
89
|
+
return new LazyCursor(cur);
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
exports.Collection = Collection;
|
|
93
|
+
/** Wraps the native cursor; each batch's docs are Proxy-wrapped for dot access. */
|
|
94
|
+
class LazyCursor {
|
|
95
|
+
constructor(native) {
|
|
96
|
+
this.native = native;
|
|
97
|
+
}
|
|
98
|
+
/** Next batch of docs, or null when exhausted. */
|
|
99
|
+
async nextBatch() {
|
|
100
|
+
const batch = await this.native.nextBatch();
|
|
101
|
+
return batch === null ? null : batch.map(wrapRawDoc);
|
|
102
|
+
}
|
|
103
|
+
/** Async iterator over individual docs across all batches. */
|
|
104
|
+
async *[Symbol.asyncIterator]() {
|
|
105
|
+
let batch;
|
|
106
|
+
while ((batch = await this.nextBatch()) !== null) {
|
|
107
|
+
for (const d of batch)
|
|
108
|
+
yield d;
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
exports.LazyCursor = LazyCursor;
|
package/dist/model.d.ts
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
import { Collection } from './index';
|
|
2
|
+
export type SchemaDefinition = Record<string, unknown>;
|
|
3
|
+
export declare class Schema {
|
|
4
|
+
readonly fields: string[];
|
|
5
|
+
readonly projection: Record<string, 1>;
|
|
6
|
+
constructor(definition: SchemaDefinition);
|
|
7
|
+
}
|
|
8
|
+
export interface QueryOptions {
|
|
9
|
+
sort?: Record<string, 1 | -1>;
|
|
10
|
+
limit?: number;
|
|
11
|
+
skip?: number;
|
|
12
|
+
}
|
|
13
|
+
export declare class Model<T = Record<string, unknown>> {
|
|
14
|
+
private readonly collection;
|
|
15
|
+
private readonly schema;
|
|
16
|
+
private constructor();
|
|
17
|
+
/** Define a model over a collection from a schema definition. */
|
|
18
|
+
static define<U = Record<string, unknown>>(collection: Collection, definition: SchemaDefinition): Model<U>;
|
|
19
|
+
/** The cached projection MongoDB receives (schema fields only). */
|
|
20
|
+
getProjection(): Record<string, 1>;
|
|
21
|
+
/** find with the schema projection pushed down. */
|
|
22
|
+
find(filter?: object, options?: QueryOptions): Promise<T[]>;
|
|
23
|
+
/** First match, or null. */
|
|
24
|
+
findOne(filter?: object, options?: QueryOptions): Promise<T | null>;
|
|
25
|
+
/** Find by _id. Accepts a 24-char hex ObjectId string (Mongoose-style). */
|
|
26
|
+
findById(id: string, options?: QueryOptions): Promise<T | null>;
|
|
27
|
+
}
|
package/dist/model.js
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
// Phase 5 — Mongoose-style Model with projection pushdown.
|
|
3
|
+
//
|
|
4
|
+
// A Model wraps a Collection + a schema (field list). Every query automatically
|
|
5
|
+
// applies a projection of the schema fields, so MongoDB only sends the fields
|
|
6
|
+
// you defined — less wire data, less to parse. The projection is built once and
|
|
7
|
+
// cached.
|
|
8
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
9
|
+
exports.Model = exports.Schema = void 0;
|
|
10
|
+
class Schema {
|
|
11
|
+
constructor(definition) {
|
|
12
|
+
this.fields = Object.keys(definition);
|
|
13
|
+
this.projection = {};
|
|
14
|
+
for (const f of this.fields)
|
|
15
|
+
this.projection[f] = 1;
|
|
16
|
+
}
|
|
17
|
+
}
|
|
18
|
+
exports.Schema = Schema;
|
|
19
|
+
class Model {
|
|
20
|
+
constructor(collection, schema) {
|
|
21
|
+
this.collection = collection;
|
|
22
|
+
this.schema = schema;
|
|
23
|
+
}
|
|
24
|
+
/** Define a model over a collection from a schema definition. */
|
|
25
|
+
static define(collection, definition) {
|
|
26
|
+
return new Model(collection, new Schema(definition));
|
|
27
|
+
}
|
|
28
|
+
/** The cached projection MongoDB receives (schema fields only). */
|
|
29
|
+
getProjection() {
|
|
30
|
+
return this.schema.projection;
|
|
31
|
+
}
|
|
32
|
+
/** find with the schema projection pushed down. */
|
|
33
|
+
async find(filter = {}, options = {}) {
|
|
34
|
+
return this.collection.find(filter, {
|
|
35
|
+
...options,
|
|
36
|
+
projection: this.schema.projection,
|
|
37
|
+
});
|
|
38
|
+
}
|
|
39
|
+
/** First match, or null. */
|
|
40
|
+
async findOne(filter = {}, options = {}) {
|
|
41
|
+
const rows = await this.find(filter, { ...options, limit: 1 });
|
|
42
|
+
return rows[0] ?? null;
|
|
43
|
+
}
|
|
44
|
+
/** Find by _id. Accepts a 24-char hex ObjectId string (Mongoose-style). */
|
|
45
|
+
async findById(id, options = {}) {
|
|
46
|
+
// Encode as Extended JSON so the Rust layer casts it to an ObjectId.
|
|
47
|
+
return this.findOne({ _id: { $oid: id } }, options);
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
exports.Model = Model;
|
package/dist/shadow.d.ts
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
/** Structural deep-equality via canonical JSON (order-insensitive for objects). */
|
|
2
|
+
export declare function deepEqual(a: unknown, b: unknown): boolean;
|
|
3
|
+
export interface Divergence<T> {
|
|
4
|
+
rust: T[];
|
|
5
|
+
official: T[];
|
|
6
|
+
}
|
|
7
|
+
/**
|
|
8
|
+
* Run both implementations concurrently. If results diverge, call `onDivergence`
|
|
9
|
+
* (e.g. log/metric) but still return the official result. If rumongo throws, the
|
|
10
|
+
* divergence callback gets the error and the official result is returned.
|
|
11
|
+
*/
|
|
12
|
+
export declare function shadow<T>(rustFn: () => Promise<T[]>, officialFn: () => Promise<T[]>, onDivergence: (info: Divergence<T> & {
|
|
13
|
+
error?: unknown;
|
|
14
|
+
}) => void): Promise<T[]>;
|
package/dist/shadow.js
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
// Shadow mode: run rumongo alongside the official driver, compare results, log
|
|
3
|
+
// divergences — but always return the official result. Lets you validate rumongo
|
|
4
|
+
// in production before cutover without risking responses.
|
|
5
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
+
exports.deepEqual = deepEqual;
|
|
7
|
+
exports.shadow = shadow;
|
|
8
|
+
/** Structural deep-equality via canonical JSON (order-insensitive for objects). */
|
|
9
|
+
function deepEqual(a, b) {
|
|
10
|
+
return canon(a) === canon(b);
|
|
11
|
+
}
|
|
12
|
+
function canon(v) {
|
|
13
|
+
return JSON.stringify(sortKeys(v));
|
|
14
|
+
}
|
|
15
|
+
// Normalize the official driver's rich BSON types to the Extended-JSON shapes
|
|
16
|
+
// rumongo emits, so equal DATA compares equal regardless of JS class:
|
|
17
|
+
// ObjectId -> {$oid: hex}, Date -> {$date:{$numberLong: ms}}
|
|
18
|
+
function sortKeys(v) {
|
|
19
|
+
if (v === null || typeof v !== 'object')
|
|
20
|
+
return v;
|
|
21
|
+
if (v instanceof Date)
|
|
22
|
+
return { $date: { $numberLong: String(v.getTime()) } };
|
|
23
|
+
const oid = v;
|
|
24
|
+
if (typeof oid.toHexString === 'function' && oid._bsontype === 'ObjectId') {
|
|
25
|
+
return { $oid: oid.toHexString() };
|
|
26
|
+
}
|
|
27
|
+
if (Array.isArray(v))
|
|
28
|
+
return v.map(sortKeys);
|
|
29
|
+
const o = v;
|
|
30
|
+
const out = {};
|
|
31
|
+
for (const k of Object.keys(o).sort())
|
|
32
|
+
out[k] = sortKeys(o[k]);
|
|
33
|
+
return out;
|
|
34
|
+
}
|
|
35
|
+
/**
|
|
36
|
+
* Run both implementations concurrently. If results diverge, call `onDivergence`
|
|
37
|
+
* (e.g. log/metric) but still return the official result. If rumongo throws, the
|
|
38
|
+
* divergence callback gets the error and the official result is returned.
|
|
39
|
+
*/
|
|
40
|
+
async function shadow(rustFn, officialFn, onDivergence) {
|
|
41
|
+
const [rustSettled, official] = await Promise.all([
|
|
42
|
+
rustFn().then((r) => ({ ok: true, r }), (error) => ({ ok: false, error })),
|
|
43
|
+
officialFn(),
|
|
44
|
+
]);
|
|
45
|
+
if (!rustSettled.ok) {
|
|
46
|
+
onDivergence({ rust: [], official, error: rustSettled.error });
|
|
47
|
+
return official;
|
|
48
|
+
}
|
|
49
|
+
if (!deepEqual(rustSettled.r, official)) {
|
|
50
|
+
onDivergence({ rust: rustSettled.r, official });
|
|
51
|
+
}
|
|
52
|
+
return official;
|
|
53
|
+
}
|
package/index.d.ts
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
/* tslint:disable */
|
|
2
|
+
/* eslint-disable */
|
|
3
|
+
|
|
4
|
+
/* auto-generated by NAPI-RS */
|
|
5
|
+
|
|
6
|
+
export declare class MongoClient {
|
|
7
|
+
/**
|
|
8
|
+
* Connect to MongoDB. `options` is an optional JSON string with pool/timeout
|
|
9
|
+
* config: `{ maxPoolSize, minPoolSize, connectTimeoutMs,
|
|
10
|
+
* serverSelectionTimeoutMs, appName }`. The driver already pools connections;
|
|
11
|
+
* these expose the knobs.
|
|
12
|
+
*/
|
|
13
|
+
static connect(uri: string, options?: string | undefined | null): Promise<MongoClient>
|
|
14
|
+
/**
|
|
15
|
+
* Terminate background workers and close connections. Call before process
|
|
16
|
+
* exit: otherwise the driver's topology-monitor tasks keep the runtime
|
|
17
|
+
* alive and Node will not exit. `immediate(true)` does not wait for live
|
|
18
|
+
* cursor handles (acceptable at teardown).
|
|
19
|
+
*/
|
|
20
|
+
close(): Promise<void>
|
|
21
|
+
/** Eager find: returns one JSON string per document (Phase 3 path). */
|
|
22
|
+
find(db: string, coll: string, filterJson: string, optsJson: string): Promise<Array<string>>
|
|
23
|
+
/**
|
|
24
|
+
* Lazy find (Phase 4): returns `RawDoc` handles holding raw bytes. No values
|
|
25
|
+
* are parsed until JS reads a field, so the event loop is not blocked by a
|
|
26
|
+
* deserialize burst on return.
|
|
27
|
+
*/
|
|
28
|
+
findLazy(db: string, coll: string, filterJson: string, optsJson: string): Promise<Array<RawDoc>>
|
|
29
|
+
/**
|
|
30
|
+
* Streaming lazy find (Phase 4): returns a `FindCursor`. Pull batches with
|
|
31
|
+
* `next_batch()` and process+drop each before the next, so peak live memory
|
|
32
|
+
* is one batch and GC jitter under concurrency stays low.
|
|
33
|
+
*/
|
|
34
|
+
findCursor(db: string, coll: string, filterJson: string, optsJson: string): Promise<FindCursor>
|
|
35
|
+
}
|
|
36
|
+
export declare class FindCursor {
|
|
37
|
+
/**
|
|
38
|
+
* Next batch of lazy `RawDoc` handles, or `null` when exhausted.
|
|
39
|
+
* Awaiting this is a real loop yield point, so timers/IO run between batches.
|
|
40
|
+
*/
|
|
41
|
+
nextBatch(): Promise<Array<RawDoc> | null>
|
|
42
|
+
}
|
|
43
|
+
/** A lazily-parsed MongoDB document backed by raw BSON bytes. */
|
|
44
|
+
export declare class RawDoc {
|
|
45
|
+
/** Parse and return a single field. `undefined` if absent. */
|
|
46
|
+
getField(name: string): unknown
|
|
47
|
+
/** Full parse into a plain JS object (escape hatch for spread/JSON.stringify). */
|
|
48
|
+
toObject(): object
|
|
49
|
+
/** Field names without parsing any values. */
|
|
50
|
+
keys(): Array<string>
|
|
51
|
+
}
|