nedb-engine 7.2.0 → 8.8.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +551 -440
- package/index.d.ts +64 -64
- package/index.js +84 -84
- package/native.d.ts +24 -0
- package/nedb.darwin-arm64.node +0 -0
- package/nedb.darwin-x64.node +0 -0
- package/nedb.linux-arm64-gnu.node +0 -0
- package/nedb.linux-arm64-musl.node +0 -0
- package/nedb.linux-x64-gnu.node +0 -0
- package/nedb.linux-x64-musl.node +0 -0
- package/nedb.win32-x64-msvc.node +0 -0
- package/nedbd-v2-darwin-arm64 +0 -0
- package/nedbd-v2-darwin-x64 +0 -0
- package/nedbd-v2-linux-arm64 +0 -0
- package/nedbd-v2-linux-arm64-musl +0 -0
- package/nedbd-v2-linux-x64 +0 -0
- package/nedbd-v2-linux-x64-musl +0 -0
- package/nedbd-v2-win-x64.exe +0 -0
- package/nesql.js +90 -0
- package/package.json +6 -3
package/README.md
CHANGED
|
@@ -5,7 +5,8 @@
|
|
|
5
5
|
**Content-addressed Merkle DAG · Hash-chained · Time-traveling · Bi-temporal · Causally-provable embedded database.**
|
|
6
6
|
|
|
7
7
|
Replay-protected · idempotent · relational · filterable · sortable · searchable · concurrent.
|
|
8
|
-
One Rust core → ships to **PyPI** and **
|
|
8
|
+
One Rust core → ships to **PyPI**, **npm** and **crates.io** from a single source,
|
|
9
|
+
at the same version on the same tag.
|
|
9
10
|
|
|
10
11
|
[](https://pypi.org/project/nedb-engine/)
|
|
11
12
|
[](https://crates.io/crates/nedb-engine)
|
|
@@ -15,7 +16,7 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
|
|
|
15
16
|
[](https://www.npmjs.com/package/nedb-engine-client)
|
|
16
17
|
[](https://github.com/Eth-Interchained/nedb/blob/master/LICENSE) [](https://github.com/Eth-Interchained/nedb/blob/master/LICENSE)
|
|
17
18
|
|
|
18
|
-
**[Studio → studio.interchained.org](https://studio.interchained.org)** · **[nedb.aiassist.net](https://nedb.aiassist.net)**
|
|
19
|
+
**[Studio → studio.interchained.org](https://studio.interchained.org)** · **[nedb.aiassist.net](https://nedb.aiassist.net)** · **[neSQL → the language](https://github.com/Eth-Interchained/neSQL)**
|
|
19
20
|
|
|
20
21
|
> ## 🟢 Free in production under $1M revenue
|
|
21
22
|
> NEDB is licensed under the **Business Source License 1.1** (since 4.0.0). If your organisation's annual
|
|
@@ -31,6 +32,101 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
|
|
|
31
32
|
|
|
32
33
|
---
|
|
33
34
|
|
|
35
|
+
## Provenance — the reason NEDB exists
|
|
36
|
+
|
|
37
|
+
Every database stores *what*. NEDB stores *what*, *when*, *when it was true*, and *why* — all sealed
|
|
38
|
+
in a cryptographic hash chain that proves none of it was tampered with.
|
|
39
|
+
|
|
40
|
+
Two time axes, one causal graph, one hash chain:
|
|
41
|
+
|
|
42
|
+
| Question | Clause / field | Answers |
|
|
43
|
+
|---|---|---|
|
|
44
|
+
| *What* is the value? | the row itself | current state |
|
|
45
|
+
| *When was it written?* | `AS OF <seq>` — transaction time | "set to 999 at seq 41" |
|
|
46
|
+
| *When was it true in the world?* | `VALID AS OF "<date>"` — valid time | "the 2024 rate" |
|
|
47
|
+
| *Why did it happen?* | `caused_by` → `TRACE caused_by` | the exact parent writes |
|
|
48
|
+
| *What did it cause?* | `TRACE caused_by REVERSE` | forward consequence |
|
|
49
|
+
| *Can I prove any of this to a third party?* | `verify()` · Merkle head · `proof()` | yes — locally, without trusting the server |
|
|
50
|
+
|
|
51
|
+
```python
|
|
52
|
+
from nedb import NEDB
|
|
53
|
+
|
|
54
|
+
db = NEDB("./mydata")
|
|
55
|
+
|
|
56
|
+
# Transaction time — the value at any past sequence, forever (no garbage collection)
|
|
57
|
+
snap = db.seq
|
|
58
|
+
db.put("users", "alice", {"age": 32})
|
|
59
|
+
db.get("users", "alice", as_of=snap) # → the prior version
|
|
60
|
+
|
|
61
|
+
# Valid time — what the world believed on a date, regardless of when written
|
|
62
|
+
db.put("policy", "rate_2024", {"pct": 5.0}, valid_from="2024-01-01", valid_to="2024-12-31")
|
|
63
|
+
db.query('FROM policy VALID AS OF "2024-06-15"') # → rate 5.0
|
|
64
|
+
|
|
65
|
+
# Both axes at once — what did the system KNOW at seq 200 about 2024-02-15?
|
|
66
|
+
db.query('FROM policy AS OF 200 VALID AS OF "2024-02-15"')
|
|
67
|
+
|
|
68
|
+
# Causal provenance — an edge, not a comment
|
|
69
|
+
db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
|
|
70
|
+
seq_msg = db.seq
|
|
71
|
+
db.put("beliefs", "dark_mode", {"value": True},
|
|
72
|
+
caused_by=[seq_msg], evidence="user_message", confidence=0.95)
|
|
73
|
+
|
|
74
|
+
db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by') # → msg_1
|
|
75
|
+
db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode
|
|
76
|
+
|
|
77
|
+
# Tamper evidence — a 64-char BLAKE2b Merkle head on every write, verifiable offline
|
|
78
|
+
assert db.verify()
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Provenance travels with the row, and it is **selectable like any other column** — `_id`, `_hash`,
|
|
82
|
+
`_seq`, `_coll` on every document, `_caused_by` / `_valid_from` / `_valid_to` where set:
|
|
83
|
+
|
|
84
|
+
```sql
|
|
85
|
+
SELECT _id, _hash, _seq FROM audit ORDER BY _seq;
|
|
86
|
+
SELECT _id FROM audit TRACE caused_by; -- works in NQL and in SQL
|
|
87
|
+
INSERT INTO audit (_id, _caused_by, kind) VALUES ('leaf', '<parent-hash>', 'reprice');
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
One hash that commits to **what a database currently says** — independent of the route it took to
|
|
91
|
+
get there — is a separate primitive, and it is specified, versioned, and test-locked across both
|
|
92
|
+
engines: see [`docs/state-root-v1.md`](docs/state-root-v1.md) and
|
|
93
|
+
[`vectors/state_root_v1.json`](vectors/state_root_v1.json). The running Merkle head commits to
|
|
94
|
+
*history*; the state root commits to *now*; replica agreement and drift detection compare roots.
|
|
95
|
+
|
|
96
|
+
**The chain is not a side feature — it is the write path.** A SQL `UPDATE` is a new version, a SQL
|
|
97
|
+
`DELETE` is a tombstone, and `verify()` still passes afterwards, because writes through any surface
|
|
98
|
+
are ordinary engine writes. There is no configuration that silently drops history.
|
|
99
|
+
|
|
100
|
+
---
|
|
101
|
+
|
|
102
|
+
## The four-axis model
|
|
103
|
+
|
|
104
|
+
| Axis | Mechanism | Other databases |
|
|
105
|
+
|---|---|:---:|
|
|
106
|
+
| **What** — current state | content-addressed objects, BLAKE2b-verified on every read | the only axis they have |
|
|
107
|
+
| **When** — transaction time | every version retained, `AS OF <seq>` | ❌ |
|
|
108
|
+
| **True-when** — valid time | bi-temporal fields, `VALID AS OF "<date>"` | ❌ |
|
|
109
|
+
| **Why** — causal provenance | `caused_by` edges in the DAG, `TRACE` both directions | ❌ |
|
|
110
|
+
|
|
111
|
+
SQLite, Redis and MongoDB store *what*. Add triggers and you have written a side table no query
|
|
112
|
+
optimizer knows about, with no chain and no proof. NEDB's provenance is native, indexed,
|
|
113
|
+
traversable — and the same hash chain that keeps it honest also protects it.
|
|
114
|
+
|
|
115
|
+
### The comparison table
|
|
116
|
+
|
|
117
|
+
| Capability | NEDB | SQLite | Redis | MongoDB |
|
|
118
|
+
|---|:---:|:---:|:---:|:---:|
|
|
119
|
+
| Hash-chained tamper evidence | ✅ | ❌ | ❌ | ❌ |
|
|
120
|
+
| Time-travel reads (`AS OF seq`) | ✅ | ❌ | ❌ | ❌ |
|
|
121
|
+
| Bi-temporal (`VALID AS OF date`) | ✅ | ❌ | ❌ | ❌ |
|
|
122
|
+
| Causal Write Provenance | ✅ | ❌ | ❌ | ❌ |
|
|
123
|
+
| Replay-protected idempotent writes | ✅ | ❌ | ❌ | ❌ |
|
|
124
|
+
| SQL + Redis + MongoDB adapters | ✅ | — | — | — |
|
|
125
|
+
| Concurrent group-commit daemon | ✅ | ❌ | ✅ | ✅ |
|
|
126
|
+
| At-rest AES-256-GCM encryption | ✅ | ❌ | ❌ | — |
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
34
130
|
## [neSQL](https://github.com/Eth-Interchained/neSQL) — the language this engine speaks
|
|
35
131
|
|
|
36
132
|
Nobody should have to learn a query language to use a database. That sentence cost
|
|
@@ -40,15 +136,31 @@ NEDB's PostgreSQL endpoint answers `psql`, SQLAlchemy Core **and** ORM, asyncpg
|
|
|
40
136
|
node-postgres against a live store. It used to get there by *translating* SQL into
|
|
41
137
|
NQL, and a translation can only reach as far as the target language's shape.
|
|
42
138
|
|
|
43
|
-
**neSQL is the name for what replaced that
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
139
|
+
**neSQL is the name for what replaced that**, and it is exactly as much of an
|
|
140
|
+
addition as it sounds like:
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
neSQL = PostgreSQL SQL · inherited whole, not reimplemented
|
|
144
|
+
+ NEDB SQL · what a permanent, hash-chained store can answer
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
**We inherit, then we gain.** The left-hand side is PostgreSQL's real grammar —
|
|
148
|
+
`gram.y`, 19,513 lines and 492 keywords, vendored from 17.4 at
|
|
149
|
+
[`vendor/postgresql/`](vendor/postgresql/) with its licence intact. Not a subset,
|
|
150
|
+
not a lookalike: the definition every other tool in the world was built against.
|
|
151
|
+
If it is valid PostgreSQL and the evaluator can parse it, it runs.
|
|
152
|
+
|
|
153
|
+
The right-hand side is what NEDB adds because it can — `AS OF SYSTEM TIME`,
|
|
154
|
+
`VALID AS OF`, `SEARCH`, `TRACE`, `TRAVERSE`. These are clauses PostgreSQL has no
|
|
155
|
+
spelling for, because a store that overwrites has nothing to point them at. They
|
|
156
|
+
are additions **to** the vendored grammar, never deviations **from** it.
|
|
48
157
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
158
|
+
So neSQL is not a dialect of SQL that you have to learn around. It is PostgreSQL
|
|
159
|
+
plus the questions a database with permanent memory can be asked. Anything you
|
|
160
|
+
already write keeps working; the new clauses are there when you need them.
|
|
161
|
+
|
|
162
|
+
Which half a statement is read as is decided **structurally**, never guessed:
|
|
163
|
+
NQL's own form begins `FROM`, PostgreSQL has no statement form that begins with
|
|
52
164
|
`FROM`, so the leading keyword partitions the two vocabularies rather than hinting
|
|
53
165
|
at them. A first word in neither is refused *naming both*.
|
|
54
166
|
|
|
@@ -59,7 +171,7 @@ evaluator with no flag to set, the `nesql` CLI likewise.
|
|
|
59
171
|
[](https://crates.io/crates/nesql)
|
|
60
172
|
[](https://www.npmjs.com/package/nesql-engine)
|
|
61
173
|
|
|
62
|
-
Those three badges read **0.0.1** next to an engine at
|
|
174
|
+
Those three badges read **0.0.1** next to an engine at 8.0.0, and that is
|
|
63
175
|
deliberate rather than neglected. They are **reserved names**: each package loads,
|
|
64
176
|
reports the vendored PostgreSQL release, and answers `is_release() == false`,
|
|
65
177
|
because a package that imports cleanly and then lies is worse than one that is not
|
|
@@ -67,10 +179,10 @@ published. The engine you actually install is `nedb-engine`. The
|
|
|
67
179
|
[neSQL repository](https://github.com/Eth-Interchained/neSQL) holds the language —
|
|
68
180
|
both halves of the grammar and the CLI's source, side by side.
|
|
69
181
|
|
|
70
|
-
### `nesql` — the CLI, and it speaks
|
|
182
|
+
### `nesql` — the CLI, and it speaks neSQL
|
|
71
183
|
|
|
72
|
-
Ships
|
|
73
|
-
|
|
184
|
+
Ships inside `pip install nedb-engine` — no daemon, no port — and answers both
|
|
185
|
+
halves of the language through **one** `query` command:
|
|
74
186
|
|
|
75
187
|
```console
|
|
76
188
|
$ nesql --db ./store query "SELECT who, total FROM orders ORDER BY total DESC"
|
|
@@ -202,7 +314,7 @@ the bar for changing that written down.
|
|
|
202
314
|
|
|
203
315
|
---
|
|
204
316
|
|
|
205
|
-
##
|
|
317
|
+
## The query language grew up · *landed in 3.3.0*
|
|
206
318
|
|
|
207
319
|
`WHERE` was six operators wide (`= != > < >= <=`) joined by an implicit `AND`.
|
|
208
320
|
It now takes a full boolean expression, in **both** engines, and the clauses
|
|
@@ -342,17 +454,21 @@ a seq is exact where a time would be approximate.
|
|
|
342
454
|
compacted store answers "not available at that sequence" rather than
|
|
343
455
|
returning a stale value, and `verify()` stays clean.
|
|
344
456
|
|
|
345
|
-
|
|
457
|
+
**NEDB speaks SQL. That sentence used to carry a caveat, and no longer does.**
|
|
346
458
|
|
|
347
|
-
|
|
348
|
-
SELECT
|
|
349
|
-
|
|
459
|
+
For most of this project's life it was true that the endpoint served a
|
|
460
|
+
documented *subset* of `SELECT` **translated** into NQL — and every refusal in
|
|
461
|
+
the table below traced to that one cause: a rewrite can only reach as far as the
|
|
462
|
+
target language's shape, and NQL's shape is single-collection with no projection.
|
|
350
463
|
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
464
|
+
That translator no longer answers `SELECT`. The evaluator does, for every
|
|
465
|
+
statement it can parse, with nothing to enable. It is kept for writes and for
|
|
466
|
+
anything outside the `SELECT` grammar, which is why a statement it cannot parse
|
|
467
|
+
still gets an answer rather than an error.
|
|
468
|
+
|
|
469
|
+
The table is preserved below as history, because the distinction between "the
|
|
470
|
+
engine could never do this" and "the translator could not reach it" is the whole
|
|
471
|
+
story of how neSQL happened — and only one of those was ever true.
|
|
356
472
|
|
|
357
473
|
| Expressible in NQL | Not expressible there, and why | the evaluator |
|
|
358
474
|
| --- | --- | --- |
|
|
@@ -372,42 +488,6 @@ NEDB is append-only *so that history cannot be discarded* — that is the produc
|
|
|
372
488
|
not a gap — and DDL is refused because collections are created by the first write
|
|
373
489
|
to them. Those answers do not change.
|
|
374
490
|
|
|
375
|
-
### Every other row on that table was a translation artefact — and one flag removes them
|
|
376
|
-
|
|
377
|
-
> ### 🆕 [**neSQL**](https://github.com/Eth-Interchained/neSQL) — PostgreSQL's grammar, NEDB's memory
|
|
378
|
-
>
|
|
379
|
-
> Those refusals were never the engine's limits. `sqlselect.rs` has had
|
|
380
|
-
> nested-loop and hash joins, subqueries, `EXISTS`, quantified comparisons, set
|
|
381
|
-
> operations, `array_agg(x ORDER BY y)` and derived tables for some time — they
|
|
382
|
-
> were simply unreachable *through a translator*, because the translator's
|
|
383
|
-
> target was NQL. They are reachable now, with nothing to set.
|
|
384
|
-
>
|
|
385
|
-
> neSQL vendors PostgreSQL's **real grammar** — `gram.y`, 19,513 lines and 492
|
|
386
|
-
> keywords, from 17.4, licence intact — and extends it with the clauses NEDB
|
|
387
|
-
> needs, rather than rewriting SQL into a language that cannot express it.
|
|
388
|
-
>
|
|
389
|
-
> It is worth knowing *why* the temporal clauses were never free: `SYSTEM_TIME`,
|
|
390
|
-
> `PERIOD` and `PORTION` appear **zero** times in PostgreSQL's grammar. Postgres
|
|
391
|
-
> has no temporal SQL at all, and `AS OF SYSTEM TIME` is a CockroachDB
|
|
392
|
-
> extension — so NEDB's temporal clauses are additions *to* the vendored grammar
|
|
393
|
-
> rather than deviations *from* it. The same road CockroachDB, Materialize and
|
|
394
|
-
> RisingWave took. What the grammar does hand over free: `WITH RECURSIVE`,
|
|
395
|
-
> window functions, `GROUPING SETS` and `MERGE`.
|
|
396
|
-
>
|
|
397
|
-
> ```bash
|
|
398
|
-
> pip install nesql · cargo add nesql · npm install nesql-engine
|
|
399
|
-
> ```
|
|
400
|
-
>
|
|
401
|
-
> Those three are **reserved names**, not a product: each loads and answers
|
|
402
|
-
> `is_release() == false`, because a package that imports cleanly and then lies
|
|
403
|
-
> is worse than one that isn't published. The engine is what ships today, and
|
|
404
|
-
> neSQL will be this same engine under its own name.
|
|
405
|
-
>
|
|
406
|
-
> **NQL is not being deleted, and it is not being wrapped.** Its verbs are SQL
|
|
407
|
-
> clauses now — `AS OF SYSTEM TIME`, `VALID AS OF`, `SEARCH` — parsed by the SQL
|
|
408
|
-
> side and executed by the NQL engine. One implementation, two front-ends,
|
|
409
|
-
> neither one a second-class guest.
|
|
410
|
-
|
|
411
491
|
Every refusal names the boundary instead of saying "syntax error", and a
|
|
412
492
|
grouped query that projects a column SQL would reject gets Postgres's own
|
|
413
493
|
message rather than a silent `NULL`.
|
|
@@ -521,7 +601,7 @@ SQL `UPDATE`, the prior value is still readable at its original sequence.
|
|
|
521
601
|
|
|
522
602
|
---
|
|
523
603
|
|
|
524
|
-
##
|
|
604
|
+
## Wrap the databases you already run · *landed in 3.2.0*
|
|
525
605
|
|
|
526
606
|
NEDB adds **tamper-evident causal provenance to a database you already have**, in one line, without
|
|
527
607
|
rip-and-replace. Five adapters, one surface:
|
|
@@ -616,7 +696,7 @@ permissive, and the two Python runtime dependencies are BSD and Apache.
|
|
|
616
696
|
**Versions 3.0.0 – 3.3.1 stay MIT, irrevocably.** If you already have NEDB at 3.3.1 or earlier, your
|
|
617
697
|
rights in that copy are untouched. This applies to 4.0.0 and later only.
|
|
618
698
|
|
|
619
|
-
### Also in 3.2.0
|
|
699
|
+
### Also landed in 3.2.0
|
|
620
700
|
|
|
621
701
|
- **A durability defect that pinned every embedded database.** The background flush ticker held a
|
|
622
702
|
strong `Arc<Db>` in an unconditional loop, so the handle was never dropped: the exclusive data-dir
|
|
@@ -636,7 +716,7 @@ rights in that copy are untouched. This applies to 4.0.0 and later only.
|
|
|
636
716
|
|
|
637
717
|
---
|
|
638
718
|
|
|
639
|
-
##
|
|
719
|
+
## Durability & recovery · *landed in 2.8.6*
|
|
640
720
|
|
|
641
721
|
Three defects found by killing a real engine at every persistence boundary and by filling a real
|
|
642
722
|
filesystem to zero free blocks. **If you are on 2.8.5 or earlier, upgrade.**
|
|
@@ -681,11 +761,11 @@ value. Ten writes drain as nine records. Changing the convention would break exi
|
|
|
681
761
|
|
|
682
762
|
---
|
|
683
763
|
|
|
684
|
-
##
|
|
764
|
+
## Distribution — three aligned distributions, one tag
|
|
685
765
|
|
|
686
|
-
|
|
766
|
+
NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with native addons for **macOS (arm64 + x86_64), Linux (x86_64 + aarch64, glibc + musl) and Windows x86_64** (see [**Releasing**](#releasing) below). All native wheels (Linux + Windows on GitHub Actions; macOS on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
|
|
687
767
|
|
|
688
|
-
###
|
|
768
|
+
### Cast — the database understands English · *landed in 2.8.0*
|
|
689
769
|
|
|
690
770
|
`POST /v1/databases/<name>/cast` turns a short English prompt into NQL, using a **3.33M-parameter model that runs locally on CPU**. No API key, no network call, no per-token bill.
|
|
691
771
|
|
|
@@ -728,9 +808,9 @@ nedbd --dag --data ./data
|
|
|
728
808
|
NEDBD_DAG=1 NEDB_TMK=<32-byte-hex> nedbd --data ./data
|
|
729
809
|
|
|
730
810
|
curl http://127.0.0.1:7070/health
|
|
731
|
-
# {"ok":true,"version":"
|
|
811
|
+
# {"ok":true,"version":"7.2.0","service":"nedbd","engine":"dag","startup_ready":true,"encrypted":true}
|
|
732
812
|
|
|
733
|
-
# Tail the live event stream (
|
|
813
|
+
# Tail the live event stream (since 2.2.31)
|
|
734
814
|
curl http://127.0.0.1:7070/events
|
|
735
815
|
# event: scan data: {"objects":730000,"of":1310703,"rate":21043,"eta_s":28}
|
|
736
816
|
# event: ready data: {"seq":1310703,"head":"b2:9c14e07a…"}
|
|
@@ -754,307 +834,159 @@ curl http://127.0.0.1:7070/events
|
|
|
754
834
|
|
|
755
835
|
**v1 AOF engine is still shipped and unchanged** — `nedbd` (no flag) runs v1.
|
|
756
836
|
|
|
757
|
-
**Production status:** [vision.interchained.org](https://vision.interchained.org) is live
|
|
837
|
+
**Production status:** [vision.interchained.org](https://vision.interchained.org) is live — verified reachable 15 Sep 2026.
|
|
838
|
+
|
|
839
|
+
The deployment figures below are a **dated snapshot**, not a live readout: **1,310,703 sequences** indexed, AES-256-GCM encrypted at rest, block height **620,989**, measured on engine **v2.2.31**. The engine version a deployment runs is not exposed on its public surface, so treat the version here as the one those numbers were taken on rather than as what is running today.
|
|
758
840
|
|
|
759
841
|
---
|
|
760
842
|
|
|
761
|
-
##
|
|
843
|
+
## Performance — every number dated and reproducible
|
|
762
844
|
|
|
763
|
-
|
|
845
|
+
NEDB is benchmarked honestly: the machine is named, the command is in the repo, and a number you cannot reproduce is a number we do not print. Absolute figures are **not** cross-machine comparable — read the ratios.
|
|
764
846
|
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
| Hash-chained tamper evidence | ✅ | ❌ | ❌ | ❌ |
|
|
768
|
-
| Time-travel reads (`AS OF seq`) | ✅ | ❌ | ❌ | ❌ |
|
|
769
|
-
| Bi-temporal (`VALID AS OF date`) | ✅ | ❌ | ❌ | ❌ |
|
|
770
|
-
| Causal Write Provenance | ✅ | ❌ | ❌ | ❌ |
|
|
771
|
-
| Replay-protected idempotent writes | ✅ | ❌ | ❌ | ❌ |
|
|
772
|
-
| SQL + Redis + MongoDB adapters | ✅ | — | — | — |
|
|
773
|
-
| Concurrent group-commit daemon | ✅ | ❌ | ✅ | ✅ |
|
|
774
|
-
| At-rest AES-256-GCM encryption | ✅ | ❌ | ❌ | — |
|
|
847
|
+
**Embedded core (Rust engine, in-process) — `bench/RESULTS.md`, 2026-06-14, Linux x86-64, Python 3.9.25.
|
|
848
|
+
Reproduce: `python3 bench/benchmarks.py --save`**
|
|
775
849
|
|
|
776
|
-
|
|
850
|
+
| Operation | Throughput | Latency (avg) |
|
|
851
|
+
|-----------|-----------|---------------|
|
|
852
|
+
| PUT (replace, no index) | **63.5K/s** | 15.74 µs |
|
|
853
|
+
| GET (point read, HEAD) | **1.33M/s** | 0.75 µs |
|
|
854
|
+
| GET (`AS OF` — time-travel read) | **942.9K/s** | 1.06 µs |
|
|
855
|
+
| QUERY: eq filter, no index (scan) | 514.1K/s | 1.95 µs |
|
|
856
|
+
| QUERY: eq filter, eq index | **1.45M/s** | 0.69 µs |
|
|
857
|
+
| QUERY: ORDER BY + ordered index, LIMIT 20 | 454.7K/s | 2.20 µs |
|
|
858
|
+
| QUERY: SEARCH, inverted index | 492.3K/s | 2.03 µs |
|
|
859
|
+
| PUT durable (AOF + fsync) | 7.3K/s | 137.59 µs |
|
|
777
860
|
|
|
778
|
-
|
|
861
|
+
Time travel is not a tax: an `AS OF` read runs at **~70% of the speed of a current-state read**.
|
|
779
862
|
|
|
780
|
-
|
|
781
|
-
|
|
782
|
-
npm install nedb-engine # Node ≥ 16 — napi-rs prebuilt binaries
|
|
783
|
-
```
|
|
863
|
+
**The daemon over HTTP (v2 DAG server v2.2.31, Intel iMac — 10k writes / 100k reads / 30k objects, AES-256-GCM on).
|
|
864
|
+
Reproduce: `NEDBD_DAG=1 nedbd --data /tmp/perf &` then `python3 tests/test_dag_perf.py --n 10000 --reads 100000`**
|
|
784
865
|
|
|
785
|
-
|
|
866
|
+
| Operation | Throughput | p50 | p99 |
|
|
867
|
+
|---|---|---|---|
|
|
868
|
+
| Sequential writes | **418 ops/s** | 2.3 ms | 3.3 ms |
|
|
869
|
+
| Point-lookup reads | **478 ops/s** | 2.0 ms | 3.0 ms |
|
|
870
|
+
| ORDER BY queries | **489 ops/s** | 1.8 ms | 4.3 ms |
|
|
871
|
+
| Batch writes (500 ops/req) | **1,104 ops/s** | 0.9 ms | 1.2 ms |
|
|
872
|
+
| Tamper-verify (30k objects) | ~21,000 BLAKE2b/sec | — | 1.38 s total |
|
|
786
873
|
|
|
787
|
-
|
|
874
|
+
p99 latencies hold because of `TCP_NODELAY` on the axum listener — without it macOS loopback adds the Nagle algorithm's 40–200 ms delay on small writes.
|
|
788
875
|
|
|
789
|
-
|
|
790
|
-
|
|
791
|
-
| Linux x86_64 | glibc | ✅ manylinux | ✅ |
|
|
792
|
-
| Linux x86_64 | musl (Alpine) | ✅ musllinux | ✅ |
|
|
793
|
-
| Linux aarch64 (Graviton, Ampere, Apple-Silicon containers) | glibc | ✅ manylinux | ✅ |
|
|
794
|
-
| Linux aarch64 | musl (Alpine) | ✅ musllinux | ✅ |
|
|
795
|
-
| macOS arm64 + x86_64 | — | ✅ | ✅ |
|
|
796
|
-
| Windows x86_64 | MSVC | ✅ | ✅ |
|
|
876
|
+
**SQL joins inside the evaluator (nested loop vs hash, fixed-seed workload, engine 4.0.0, Linux x86-64) —
|
|
877
|
+
`docs/BENCH-sqlselect.md`. Reproduce: `cargo run --release --example sqlbench`**
|
|
797
878
|
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
(
|
|
801
|
-
|
|
879
|
+
| workload | nested (ms) | hash (ms) | speedup |
|
|
880
|
+
|---|---:|---:|---:|
|
|
881
|
+
| equality join (1,000 × 500 rows) | 366.62 | 4.05 | **90.6×** |
|
|
882
|
+
| left join | 373.27 | 4.03 | **92.5×** |
|
|
883
|
+
| join + sort | 403.51 | 6.35 | **63.6×** |
|
|
884
|
+
| join + broad pred | 336.34 | 4.68 | **71.9×** |
|
|
885
|
+
| non-equality join | 344.69 | 346.33 | — *(same path, no hash key)* |
|
|
802
886
|
|
|
803
|
-
|
|
887
|
+
**Indexed range scans (20,000 rows, two identical stores, one indexed) — `scripts/bench_index_range.py`**
|
|
804
888
|
|
|
805
|
-
|
|
889
|
+
| Query | Scan | Indexed | Speedup |
|
|
890
|
+
| --- | --- | --- | --- |
|
|
891
|
+
| `WHERE fee = 10000` | 137 ms | 0.01 ms | **17,000×** |
|
|
892
|
+
| `WHERE fee IN (a, b, c)` | 185 ms | 0.02 ms | 9,700× |
|
|
893
|
+
| `WHERE fee BETWEEN …` (1% of rows) | 186 ms | 1.1 ms | 170× |
|
|
894
|
+
| `WHERE fee BETWEEN …` (10% of rows) | 188 ms | 13 ms | 14× |
|
|
895
|
+
| unindexed field (control) | 188 ms | 187 ms | 1.0× |
|
|
806
896
|
|
|
807
|
-
|
|
808
|
-
from nedb import NEDB
|
|
897
|
+
**v1 Python server (baseline — single-threaded AOF):**
|
|
809
898
|
|
|
810
|
-
|
|
811
|
-
|
|
899
|
+
| Operation | Throughput | p99 latency |
|
|
900
|
+
|---|---|---|
|
|
901
|
+
| Sequential PUT | ~23/s | 44 ms |
|
|
902
|
+
| Concurrent PUT (16 workers) | ~92/s | 48 ms |
|
|
903
|
+
| Batch PUT (500 ops/request) | ~520 ops/s | 1.9 ms/op |
|
|
904
|
+
| Point-lookup read (NQL) | ~23/s | 44 ms |
|
|
905
|
+
| Rust napi PUT (FFI) | ~70K/s | — |
|
|
906
|
+
| Rust napi GET (FFI) | ~330K/s | — |
|
|
812
907
|
|
|
813
|
-
|
|
814
|
-
db.create_index("users", "bio", "search")
|
|
908
|
+
And the number from the v3 section that justifies the whole storage line: a real itcd chainstate flush of 2,549 coins went from **minutes** on the v2 loose store to **1.71 s** on `--dag-v3` — measured on the real chain, not a fixture.
|
|
815
909
|
|
|
816
|
-
|
|
817
|
-
db.put("users", "bob", {"name": "Bob", "age": 24, "status": "active", "bio": "python dev"})
|
|
910
|
+
---
|
|
818
911
|
|
|
819
|
-
|
|
820
|
-
db.query('FROM users WHERE status = "active" ORDER BY age ASC')
|
|
821
|
-
db.query('FROM users SEARCH "rust"')
|
|
822
|
-
db.query('FROM users GROUP BY status COUNT')
|
|
912
|
+
## nedbd — the concurrent server daemon
|
|
823
913
|
|
|
824
|
-
|
|
825
|
-
db.query('FROM users WHERE status IN ("active", "trialing")')
|
|
826
|
-
db.query('FROM users WHERE age BETWEEN 25 AND 40')
|
|
827
|
-
db.query('FROM users WHERE bio LIKE "%rust%" AND NOT (status = "retired")')
|
|
828
|
-
db.query('FROM users WHERE (age < 25 OR age > 60) AND bio IS NOT NULL')
|
|
914
|
+
nedbd runs NEDB as a long-lived process with an HTTP/JSON API and an optional RESP2 wire protocol. Built on a **single-writer group-commit sequencer** — parallel reads, batched durable writes, one hash-chain per database, zero write-write races.
|
|
829
915
|
|
|
830
|
-
|
|
831
|
-
|
|
832
|
-
|
|
833
|
-
|
|
916
|
+
```bash
|
|
917
|
+
nedbd # :7070, data ./nedb-data (v1 AOF engine)
|
|
918
|
+
nedbd --dag --data ./data # v2 DAG engine (or NEDBD_DAG=1)
|
|
919
|
+
NEDBD_RESP2_PORT=6380 nedbd # also speak RESP2 (redis-cli compatible)
|
|
920
|
+
nedbd --log-level 2 # 0=errors 1=requests 2=deploy 3=verbose
|
|
834
921
|
|
|
835
|
-
#
|
|
836
|
-
|
|
837
|
-
|
|
838
|
-
db.query('FROM policy VALID AS OF "2024-06-15"') # → rate 5.0
|
|
922
|
+
# Live event stream (since 2.2.31) — SSE: scan progress, ready, per-write head
|
|
923
|
+
curl http://127.0.0.1:7070/events
|
|
924
|
+
```
|
|
839
925
|
|
|
840
|
-
|
|
841
|
-
db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
|
|
842
|
-
seq_msg = db.seq
|
|
843
|
-
db.put("beliefs", "dark_mode", {"value": True},
|
|
844
|
-
caused_by=[seq_msg], evidence="user_message", confidence=0.95)
|
|
845
|
-
db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by') # → msg_1
|
|
846
|
-
db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode
|
|
926
|
+
### Companion CLIs
|
|
847
927
|
|
|
848
|
-
|
|
849
|
-
db.link("users:alice", "follows", "users:bob")
|
|
850
|
-
db.query('FROM users WHERE _id = "alice" TRAVERSE follows')
|
|
928
|
+
Alongside the daemon, `cargo install nedb-engine` ships **`nedb-cli`** — operate on a store directory offline (`head`/`status`/`verify`/`get`/`scan`/`flush`/`repair`/`export`) — and **`nedb-inspector`**, a deterministic checker that warns when a durable open lacks flush-on-exit wiring. Full reference: [**docs/CLI.md**](docs/CLI.md).
|
|
851
929
|
|
|
852
|
-
|
|
853
|
-
assert db.verify() # cryptographic proof — no tampering
|
|
930
|
+
### Startup modes
|
|
854
931
|
|
|
855
|
-
|
|
856
|
-
from
|
|
857
|
-
sql_exec(db, "SELECT * FROM users WHERE status = 'active' ORDER BY age DESC")
|
|
858
|
-
r = RedisCompat(db); r.execute("HSET", "user:1", "name", "Alice")
|
|
859
|
-
MongoClient(db)["users"].find({"status": "active"}).sort("age", -1).to_list()
|
|
860
|
-
```
|
|
932
|
+
- **Warm start** — every restart after the first open reads the `MANIFEST` file and restores `seq` + Merkle `head` in **O(1)**. No scan, no replay, independent of dataset size. Boots in milliseconds.
|
|
933
|
+
- **Cold start** — first open of an existing dataset spawns the integrity scan in a background thread *and accepts connections immediately*. Reads serve instantly from the content-addressed DAG; writes return `HTTP 503 startup in progress` until the `startup_ready` gate flips. Progress (objects, rate, ETA) streams over `GET /events`.
|
|
861
934
|
|
|
862
|
-
|
|
935
|
+
### Environment variables
|
|
863
936
|
|
|
864
|
-
|
|
937
|
+
| Variable | Default | Description |
|
|
938
|
+
|---|---|---|
|
|
939
|
+
| `NEDBD_DAG` | `0` | Set `1` to launch the v2 DAG engine (`nedbd-v2`). Same as `--dag`. |
|
|
940
|
+
| `NEDBD_HOST` | `127.0.0.1` | Bind address. **v2.2.31** defaults to loopback (was `0.0.0.0`) — security hardening fix. Set explicitly to `0.0.0.0` to expose. |
|
|
941
|
+
| `NEDBD_PORT` | `7070` | HTTP bind port. |
|
|
942
|
+
| `NEDBD_TOKEN` | unset | Optional bearer token; required on every `/v1/*` request when set. |
|
|
943
|
+
| `NEDB_TMK` | unset | 32-byte hex AES-256-GCM at-rest encryption key. |
|
|
944
|
+
| `NEDBD_DATA` | `./nedb-data` | Root directory. v2 creates `dag/`, IdIndex sharded across **256 subdirectories**, and a small `MANIFEST` file. |
|
|
945
|
+
| `NEDBD_CAST` | `0` | Set `1` to enable the `/cast` natural-language planner. Same as `--cast`. Requires a build with `--features cast`. See [**Cast**](#cast--natural-language-into-nql). |
|
|
946
|
+
| `NEDBD_CAST_MODEL` | unset | Explicit path to a `model.cast` container. Otherwise searched in the data dir, `$CAST_HOME`, and `~/.cache/nedb-cast-slm/`. |
|
|
865
947
|
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
|
|
871
|
-
|
|
948
|
+
```bash
|
|
949
|
+
# Create a database with seed data and relations
|
|
950
|
+
curl -X POST :7070/v1/databases -d '{
|
|
951
|
+
"name": "shop",
|
|
952
|
+
"init": {
|
|
953
|
+
"indexes": [["users","status","eq"]],
|
|
954
|
+
"seed": {"users": [{"_id":"u1","name":"Alice","status":"active"}]},
|
|
955
|
+
"links": [["users:u1","buys","orders:o1"]]
|
|
956
|
+
}}'
|
|
872
957
|
|
|
873
|
-
|
|
874
|
-
|
|
958
|
+
# Query — the endpoint speaks neSQL: SQL *or* NQL, routed on the first keyword
|
|
959
|
+
curl -X POST :7070/v1/databases/shop/query \
|
|
960
|
+
-d '{"nql":"SELECT name FROM users WHERE status = '"'"'active'"'"' ORDER BY name"}'
|
|
961
|
+
# → {"rows":[{"name":"Alice"}],"count":1,"dialect":"sql", ...}
|
|
875
962
|
|
|
876
|
-
|
|
877
|
-
|
|
963
|
+
curl -X POST :7070/v1/databases/shop/query \
|
|
964
|
+
-d '{"nql":"FROM users WHERE status = \"active\" ORDER BY name ASC"}'
|
|
965
|
+
# → {"rows":[...],"count":1,"dialect":"nql", ...}
|
|
878
966
|
|
|
879
|
-
c.put("users", "u1", {"id": "u1", "email": "a@b.c"}, idem="signup-u1")
|
|
880
|
-
c.query('FROM users WHERE email = "a@b.c"') # full NQL rides through
|
|
881
|
-
c.query("FROM users AS OF 41") # time-travel included
|
|
882
967
|
|
|
883
|
-
|
|
884
|
-
|
|
885
|
-
|
|
886
|
-
|
|
968
|
+
**The field is still called `nql`, and its contents no longer have to be.** This
|
|
969
|
+
endpoint accepts **neSQL** — NQL *or* PostgreSQL SQL — and answers with the
|
|
970
|
+
`dialect` it chose. The name is unchanged because every existing HTTP client
|
|
971
|
+
sends it; renaming would break them to gain nothing. Old NQL clients are
|
|
972
|
+
unaffected.
|
|
887
973
|
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
|
|
892
|
-
if_seq=d.get("_seq", -1))])
|
|
893
|
-
c.cas_retry(bump)
|
|
974
|
+
Routing is **structural, not guessed**. NQL statements begin `FROM`; PostgreSQL
|
|
975
|
+
has no statement form that begins with `FROM`, so the leading keyword partitions
|
|
976
|
+
the two vocabularies rather than hinting at them. A first word in neither is
|
|
977
|
+
refused *naming both* — never handed to whichever parser seems likelier.
|
|
894
978
|
|
|
895
|
-
|
|
896
|
-
|
|
897
|
-
|
|
979
|
+
```bash
|
|
980
|
+
curl -X POST :7070/v1/databases/shop/query -d '{"nql":"GRANT ALL ON users"}'
|
|
981
|
+
# → 400 "GRANT" does not begin a statement in either half of neSQL
|
|
982
|
+
# NQL statements begin with: FROM
|
|
983
|
+
# SQL statements begin with: SELECT, INSERT, UPDATE, ...
|
|
898
984
|
```
|
|
899
985
|
|
|
900
|
-
|
|
901
|
-
|
|
902
|
-
|
|
903
|
-
|
|
904
|
-
`NedbConflict`, `CasExhausted`.
|
|
905
|
-
|
|
906
|
-
---
|
|
907
|
-
|
|
908
|
-
## Redis layer-2 — wrap_redis()
|
|
909
|
-
|
|
910
|
-
Already running on Redis? Wrap your connection in one line and gain NEDB features *alongside* your existing Redis app — no migration required.
|
|
911
|
-
|
|
912
|
-
```python
|
|
913
|
-
import redis, json
|
|
914
|
-
from nedb import wrap_redis
|
|
915
|
-
|
|
916
|
-
r = wrap_redis(redis.Redis("localhost", 6379), db_name="rideshare")
|
|
917
|
-
|
|
918
|
-
# Step 1 — register: map Redis key globs to NEDB collections (chainable)
|
|
919
|
-
(r.nedb
|
|
920
|
-
.register("driver:*", collection="driver", value_parser=json.loads)
|
|
921
|
-
.register("trip:*", collection="trip", value_type="hash")
|
|
922
|
-
)
|
|
923
|
-
|
|
924
|
-
# Step 2 — backfill: import all existing Redis data into NEDB in one pass
|
|
925
|
-
imported = r.nedb.backfill() # → int (keys imported)
|
|
926
|
-
|
|
927
|
-
# Step 3 — shadow: all future r.set/hset/... auto-chain into NEDB
|
|
928
|
-
r.nedb.shadow_writes = True
|
|
929
|
-
|
|
930
|
-
# ─── Alice's app keeps running — zero changes ───────────────────────────
|
|
931
|
-
r.set("driver:d1", json.dumps({"name": "Bob", "status": "active"})) # ← shadowed
|
|
932
|
-
r.hset("trip:t1", mapping={"status": "en_route", "driver_id": "d1"}) # ← shadowed
|
|
933
|
-
|
|
934
|
-
# ─── New features available on the same connection ──────────────────────
|
|
935
|
-
r.nedb.query('FROM driver WHERE status = "active" ORDER BY lat ASC')
|
|
936
|
-
r.nedb.verify() # → True (every write chain-verified)
|
|
937
|
-
r.nedb.head() # → 64-char BLAKE2b commitment hash
|
|
938
|
-
```
|
|
939
|
-
|
|
940
|
-
**Isolation guarantee:** NEDB never writes to Alice's namespace. It owns only:
|
|
941
|
-
|
|
942
|
-
| Key | Type | Purpose |
|
|
943
|
-
|-----|------|---------|
|
|
944
|
-
| `nedb:{db_name}:oplog` | Redis Stream | append-only op log |
|
|
945
|
-
| `nedb:{db_name}:snapshot` | Redis Hash | checkpoint |
|
|
946
|
-
| `nedb:{db_name}:meta` | Redis Hash | index config |
|
|
947
|
-
|
|
948
|
-
See [`examples/fakeredis_demo.py`](examples/fakeredis_demo.py) for a full local demo (no Redis server needed).
|
|
949
|
-
|
|
950
|
-
---
|
|
951
|
-
|
|
952
|
-
## Node.js
|
|
953
|
-
|
|
954
|
-
```javascript
|
|
955
|
-
import { NedbCore } from "nedb-engine";
|
|
956
|
-
|
|
957
|
-
const db = new NedbCore(); // in-memory
|
|
958
|
-
// const db = NedbCore.open("./data"); // durable
|
|
959
|
-
|
|
960
|
-
db.createIndex("users", "status", "eq");
|
|
961
|
-
db.put("users", "alice", JSON.stringify({ name: "Alice", age: 31, status: "active" }));
|
|
962
|
-
|
|
963
|
-
// Time-travel
|
|
964
|
-
const snap = db.seq(); // BigInt
|
|
965
|
-
db.put("users", "alice", JSON.stringify({ name: "Alice", age: 32, status: "retired" }));
|
|
966
|
-
JSON.parse(db.getAsOf("users", "alice", snap)).age; // → 31
|
|
967
|
-
|
|
968
|
-
// Full NQL
|
|
969
|
-
const rows = db.query('FROM users WHERE status = "active" ORDER BY age ASC');
|
|
970
|
-
rows.map(r => JSON.parse(r));
|
|
971
|
-
|
|
972
|
-
// Tamper evidence
|
|
973
|
-
db.verify(); // → true
|
|
974
|
-
db.head(); // → 64-char BLAKE2b commitment hash
|
|
975
|
-
db.seq(); // → BigInt
|
|
976
|
-
```
|
|
977
|
-
|
|
978
|
-
---
|
|
979
|
-
|
|
980
|
-
## nedbd — the concurrent server daemon
|
|
981
|
-
|
|
982
|
-
nedbd runs NEDB as a long-lived process with an HTTP/JSON API and an optional RESP2 wire protocol. Built on a **single-writer group-commit sequencer** — parallel reads, batched durable writes, one hash-chain per database, zero write-write races.
|
|
983
|
-
|
|
984
|
-
```bash
|
|
985
|
-
nedbd # :7070, data ./nedb-data (v1 AOF engine)
|
|
986
|
-
nedbd --dag --data ./data # v2 DAG engine (or NEDBD_DAG=1)
|
|
987
|
-
NEDBD_RESP2_PORT=6380 nedbd # also speak RESP2 (redis-cli compatible)
|
|
988
|
-
nedbd --log-level 2 # 0=errors 1=requests 2=deploy 3=verbose
|
|
989
|
-
|
|
990
|
-
# Live event stream (new in v2.2.31) — SSE: scan progress, ready, per-write head
|
|
991
|
-
curl http://127.0.0.1:7070/events
|
|
992
|
-
```
|
|
993
|
-
|
|
994
|
-
### Companion CLIs
|
|
995
|
-
|
|
996
|
-
Alongside the daemon, `cargo install nedb-engine` ships **`nedb-cli`** — operate on a store directory offline (`head`/`status`/`verify`/`get`/`scan`/`flush`/`repair`/`export`) — and **`nedb-inspector`**, a deterministic checker that warns when a durable open lacks flush-on-exit wiring. Full reference: [**docs/CLI.md**](docs/CLI.md).
|
|
997
|
-
|
|
998
|
-
### Startup modes (v2.2.31)
|
|
999
|
-
|
|
1000
|
-
- **Warm start** — every restart after the first open reads the `MANIFEST` file and restores `seq` + Merkle `head` in **O(1)**. No scan, no replay, independent of dataset size. Boots in milliseconds.
|
|
1001
|
-
- **Cold start** — first open of an existing dataset spawns the integrity scan in a background thread *and accepts connections immediately*. Reads serve instantly from the content-addressed DAG; writes return `HTTP 503 startup in progress` until the `startup_ready` gate flips. Progress (objects, rate, ETA) streams over `GET /events`.
|
|
1002
|
-
|
|
1003
|
-
### Environment variables
|
|
1004
|
-
|
|
1005
|
-
| Variable | Default | Description |
|
|
1006
|
-
|---|---|---|
|
|
1007
|
-
| `NEDBD_DAG` | `0` | Set `1` to launch the v2 DAG engine (`nedbd-v2`). Same as `--dag`. |
|
|
1008
|
-
| `NEDBD_HOST` | `127.0.0.1` | Bind address. **v2.2.31** defaults to loopback (was `0.0.0.0`) — security hardening fix. Set explicitly to `0.0.0.0` to expose. |
|
|
1009
|
-
| `NEDBD_PORT` | `7070` | HTTP bind port. |
|
|
1010
|
-
| `NEDBD_TOKEN` | unset | Optional bearer token; required on every `/v1/*` request when set. |
|
|
1011
|
-
| `NEDB_TMK` | unset | 32-byte hex AES-256-GCM at-rest encryption key. |
|
|
1012
|
-
| `NEDBD_DATA` | `./nedb-data` | Root directory. v2 creates `dag/`, IdIndex sharded across **256 subdirectories**, and a small `MANIFEST` file. |
|
|
1013
|
-
| `NEDBD_CAST` | `0` | Set `1` to enable the `/cast` natural-language planner. Same as `--cast`. Requires a build with `--features cast`. See [**Cast**](#cast--natural-language-into-nql). |
|
|
1014
|
-
| `NEDBD_CAST_MODEL` | unset | Explicit path to a `model.cast` container. Otherwise searched in the data dir, `$CAST_HOME`, and `~/.cache/nedb-cast-slm/`. |
|
|
1015
|
-
|
|
1016
|
-
```bash
|
|
1017
|
-
# Create a database with seed data and relations
|
|
1018
|
-
curl -X POST :7070/v1/databases -d '{
|
|
1019
|
-
"name": "shop",
|
|
1020
|
-
"init": {
|
|
1021
|
-
"indexes": [["users","status","eq"]],
|
|
1022
|
-
"seed": {"users": [{"_id":"u1","name":"Alice","status":"active"}]},
|
|
1023
|
-
"links": [["users:u1","buys","orders:o1"]]
|
|
1024
|
-
}}'
|
|
1025
|
-
|
|
1026
|
-
# Query — the endpoint speaks neQL: SQL *or* NQL, routed on the first keyword
|
|
1027
|
-
curl -X POST :7070/v1/databases/shop/query \
|
|
1028
|
-
-d '{"nql":"SELECT name FROM users WHERE status = '"'"'active'"'"' ORDER BY name"}'
|
|
1029
|
-
# → {"rows":[{"name":"Alice"}],"count":1,"dialect":"sql", ...}
|
|
1030
|
-
|
|
1031
|
-
curl -X POST :7070/v1/databases/shop/query \
|
|
1032
|
-
-d '{"nql":"FROM users WHERE status = \"active\" ORDER BY name ASC"}'
|
|
1033
|
-
# → {"rows":[...],"count":1,"dialect":"nql", ...}
|
|
1034
|
-
|
|
1035
|
-
|
|
1036
|
-
**The field is still called `nql`, and its contents no longer have to be.** This
|
|
1037
|
-
endpoint accepts **neQL** — NQL *or* PostgreSQL SQL — and answers with the
|
|
1038
|
-
`dialect` it chose. The name is unchanged because every existing HTTP client
|
|
1039
|
-
sends it; renaming would break them to gain nothing. Old NQL clients are
|
|
1040
|
-
unaffected.
|
|
1041
|
-
|
|
1042
|
-
Routing is **structural, not guessed**. NQL statements begin `FROM`; PostgreSQL
|
|
1043
|
-
has no statement form that begins with `FROM`, so the leading keyword partitions
|
|
1044
|
-
the two vocabularies rather than hinting at them. A first word in neither is
|
|
1045
|
-
refused *naming both* — never handed to whichever parser seems likelier.
|
|
1046
|
-
|
|
1047
|
-
```bash
|
|
1048
|
-
curl -X POST :7070/v1/databases/shop/query -d '{"nql":"GRANT ALL ON users"}'
|
|
1049
|
-
# → 400 "GRANT" does not begin a statement in either half of neQL
|
|
1050
|
-
# NQL statements begin with: FROM
|
|
1051
|
-
# SQL statements begin with: SELECT, INSERT, UPDATE, ...
|
|
1052
|
-
```
|
|
1053
|
-
|
|
1054
|
-
It is the **same router** `nesql query` uses — `nedb_engine::neql::route`, which
|
|
1055
|
-
the CLI re-exports rather than copies. Two implementations of that decision
|
|
1056
|
-
would let the daemon and the CLI disagree about what a statement *means*, which
|
|
1057
|
-
is worse than disagreeing about a result: nothing looks broken when it happens.
|
|
986
|
+
It is the **same router** `nesql query` uses — `nedb_engine::neql::route`, which
|
|
987
|
+
the CLI re-exports rather than copies. Two implementations of that decision
|
|
988
|
+
would let the daemon and the CLI disagree about what a statement *means*, which
|
|
989
|
+
is worse than disagreeing about a result: nothing looks broken when it happens.
|
|
1058
990
|
|
|
1059
991
|
# Verify the hash chain
|
|
1060
992
|
curl :7070/v1/databases/shop/verify
|
|
@@ -1153,14 +1085,13 @@ db.query("FROM blocks WHERE height BETWEEN 600000 AND 600100")
|
|
|
1153
1085
|
```
|
|
1154
1086
|
|
|
1155
1087
|
Measured on 20,000 rows with `scripts/bench_index_range.py` — two identical
|
|
1156
|
-
databases, one indexed, one not
|
|
1088
|
+
databases, one indexed, one not (the full table lives in
|
|
1089
|
+
[**Performance**](#performance--every-number-dated-and-reproducible)):
|
|
1157
1090
|
|
|
1158
1091
|
| Query | Scan | Indexed | Speedup |
|
|
1159
1092
|
| --- | --- | --- | --- |
|
|
1160
1093
|
| `WHERE fee = 10000` | 137 ms | 0.01 ms | 17,000× |
|
|
1161
|
-
| `WHERE fee IN (a, b, c)` | 185 ms | 0.02 ms | 9,700× |
|
|
1162
1094
|
| `WHERE fee BETWEEN …` (1% of rows) | 186 ms | 1.1 ms | 170× |
|
|
1163
|
-
| `WHERE fee BETWEEN …` (10% of rows) | 188 ms | 13 ms | 14× |
|
|
1164
1095
|
| unindexed field (control) | 188 ms | 187 ms | 1.0× |
|
|
1165
1096
|
|
|
1166
1097
|
The planner asks the index how many rows each candidate range covers and takes
|
|
@@ -1481,108 +1412,237 @@ Two habits that avoid most misses:
|
|
|
1481
1412
|
|
|
1482
1413
|
---
|
|
1483
1414
|
|
|
1484
|
-
##
|
|
1415
|
+
## Architecture
|
|
1485
1416
|
|
|
1486
|
-
|
|
1417
|
+
```
|
|
1418
|
+
┌──────────────────────────────────────────────────────────┐
|
|
1419
|
+
put/del → │ OpLog (BLAKE2b hash chain · per-client nonce · │ ← single source of truth
|
|
1420
|
+
link │ idempotency keys · causal provenance fields) │
|
|
1421
|
+
└───────────────┬──────────────────────────────────────────┘
|
|
1422
|
+
deterministic fold │ (state = pure function of the log)
|
|
1423
|
+
┌──────────────┬──────────┴──────┬───────────────┬────────────────┐
|
|
1424
|
+
▼ ▼ ▼ ▼ ▼
|
|
1425
|
+
MVCC store Relations Indexes CauseMap BlobStore
|
|
1426
|
+
(time-travel) (graph+AS OF) eq/ord/search (reverse index) (Cascade CDC)
|
|
1487
1427
|
|
|
1488
|
-
|
|
1489
|
-
|
|
1490
|
-
|
|
1491
|
-
|
|
1492
|
-
|
|
1493
|
-
| Batch writes (500 ops/req) | **1,104 ops/s** | 0.9 ms | 1.2 ms |
|
|
1494
|
-
| Tamper-verify (30k objects) | ~21,000 BLAKE2b/sec | — | 1.38 s total |
|
|
1428
|
+
┌─────────────────────────────────┐
|
|
1429
|
+
Thread-safe → │ Sequencer (group-commit) │ ← single writer, parallel readers
|
|
1430
|
+
│ — one committer thread/db │
|
|
1431
|
+
│ — batch fsync │
|
|
1432
|
+
└─────────────────────────────────┘
|
|
1495
1433
|
|
|
1496
|
-
|
|
1434
|
+
Compatibility adapters: SQL · Redis · MongoDB
|
|
1435
|
+
Wire protocols: HTTP/JSON · RESP2 · PostgreSQL wire
|
|
1436
|
+
Encryption: AES-256-GCM at-rest (TMK/DEK double-envelope)
|
|
1437
|
+
```
|
|
1497
1438
|
|
|
1498
|
-
|
|
1439
|
+
The provenance machinery is load-bearing infrastructure, not metadata stapled on: `CauseMap` is a
|
|
1440
|
+
reverse index over causal edges (that is why `TRACE … REVERSE` is as fast as forward), and the
|
|
1441
|
+
`Relations` layer backs both `TRAVERSE` and the `AS OF` joins.
|
|
1499
1442
|
|
|
1500
|
-
|
|
1501
|
-
|---|---|---|
|
|
1502
|
-
| Sequential PUT | ~23/s | 44 ms |
|
|
1503
|
-
| Concurrent PUT (16 workers) | ~92/s | 48 ms |
|
|
1504
|
-
| Batch PUT (500 ops/request) | ~520 ops/s | 1.9 ms/op |
|
|
1505
|
-
| Point-lookup read (NQL) | ~23/s | 44 ms |
|
|
1506
|
-
| Rust napi PUT (FFI) | ~70K/s | — |
|
|
1507
|
-
| Rust napi GET (FFI) | ~330K/s | — |
|
|
1443
|
+
---
|
|
1508
1444
|
|
|
1509
|
-
|
|
1445
|
+
## Install
|
|
1510
1446
|
|
|
1511
1447
|
```bash
|
|
1512
|
-
|
|
1513
|
-
|
|
1448
|
+
pip install nedb-engine # Python ≥ 3.8 — pure-Python + optional Rust native wheel
|
|
1449
|
+
npm install nedb-engine # Node ≥ 16 — napi-rs prebuilt binaries
|
|
1514
1450
|
```
|
|
1515
1451
|
|
|
1452
|
+
### Prebuilt platforms
|
|
1453
|
+
|
|
1454
|
+
Both registries ship prebuilt binaries for:
|
|
1455
|
+
|
|
1456
|
+
| Platform | libc | Python wheel | Node addon |
|
|
1457
|
+
|---|---|---|---|
|
|
1458
|
+
| Linux x86_64 | glibc | ✅ manylinux | ✅ |
|
|
1459
|
+
| Linux x86_64 | musl (Alpine) | ✅ musllinux | ✅ |
|
|
1460
|
+
| Linux aarch64 (Graviton, Ampere, Apple-Silicon containers) | glibc | ✅ manylinux | ✅ |
|
|
1461
|
+
| Linux aarch64 | musl (Alpine) | ✅ musllinux | ✅ |
|
|
1462
|
+
| macOS arm64 + x86_64 | — | ✅ | ✅ |
|
|
1463
|
+
| Windows x86_64 | MSVC | ✅ | ✅ |
|
|
1464
|
+
|
|
1465
|
+
On Python, any platform without a prebuilt wheel still installs: pip falls back
|
|
1466
|
+
to the universal `py3-none-any` wheel and you get the pure-Python v1 AOF engine
|
|
1467
|
+
(correct, slower, no embedded DAG). On Node there is no such fallback — an
|
|
1468
|
+
unlisted platform has no addon.
|
|
1469
|
+
|
|
1516
1470
|
---
|
|
1517
1471
|
|
|
1518
|
-
##
|
|
1472
|
+
## Python — 5-minute tour
|
|
1519
1473
|
|
|
1520
|
-
|
|
1474
|
+
Every example in this tour was executed against the released **8.0.0** package.
|
|
1521
1475
|
|
|
1522
|
-
|
|
1476
|
+
```python
|
|
1477
|
+
from nedb import NEDB
|
|
1523
1478
|
|
|
1524
|
-
|
|
1479
|
+
db = NEDB("./mydata") # durable: every op is AOF-logged, fsync'd, and hash-chained
|
|
1480
|
+
# db = NEDB() # or in-memory
|
|
1525
1481
|
|
|
1526
|
-
|
|
1482
|
+
db.create_index("users", "status", "eq")
|
|
1483
|
+
db.create_index("users", "bio", "search")
|
|
1527
1484
|
|
|
1528
|
-
|
|
1485
|
+
db.put("users", "alice", {"name": "Alice", "age": 31, "status": "active", "bio": "rust hacker"})
|
|
1486
|
+
db.put("users", "bob", {"name": "Bob", "age": 24, "status": "active", "bio": "python dev"})
|
|
1529
1487
|
|
|
1530
|
-
|
|
1531
|
-
|
|
1532
|
-
|
|
1533
|
-
|
|
1488
|
+
# NQL: WHERE + ORDER BY + LIMIT + SEARCH + TRAVERSE + GROUP BY
|
|
1489
|
+
db.query('FROM users WHERE status = "active" ORDER BY age ASC')
|
|
1490
|
+
db.query('FROM users SEARCH "rust"')
|
|
1491
|
+
db.query('FROM users GROUP BY status COUNT')
|
|
1534
1492
|
|
|
1535
|
-
|
|
1493
|
+
# Full boolean predicates — IN, BETWEEN, LIKE, IS NULL, OR, NOT, parentheses
|
|
1494
|
+
db.query('FROM users WHERE status IN ("active", "trialing")')
|
|
1495
|
+
db.query('FROM users WHERE age BETWEEN 25 AND 40')
|
|
1496
|
+
db.query('FROM users WHERE bio LIKE "%rust%" AND NOT (status = "retired")')
|
|
1497
|
+
db.query('FROM users WHERE (age < 25 OR age > 60) AND bio IS NOT NULL')
|
|
1536
1498
|
|
|
1537
|
-
|
|
1538
|
-
|
|
1539
|
-
|
|
1499
|
+
# Time-travel — AS OF any past sequence
|
|
1500
|
+
snap = db.seq
|
|
1501
|
+
db.put("users", "alice", {"name": "Alice", "age": 32, "status": "retired"})
|
|
1502
|
+
db.get("users", "alice", as_of=snap) # → age 31, status active
|
|
1540
1503
|
|
|
1541
|
-
#
|
|
1542
|
-
|
|
1504
|
+
# Bi-temporal — VALID AS OF any past date
|
|
1505
|
+
db.put("policy", "rate_2024", {"pct": 5.0}, valid_from="2024-01-01", valid_to="2024-12-31")
|
|
1506
|
+
db.put("policy", "rate_2025", {"pct": 6.0}, valid_from="2025-01-01")
|
|
1507
|
+
db.query('FROM policy VALID AS OF "2024-06-15"') # → rate 5.0
|
|
1508
|
+
|
|
1509
|
+
# Causal Write Provenance — why did this write happen?
|
|
1510
|
+
db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
|
|
1511
|
+
seq_msg = db.seq
|
|
1512
|
+
db.put("beliefs", "dark_mode", {"value": True},
|
|
1513
|
+
caused_by=[seq_msg], evidence="user_message", confidence=0.95)
|
|
1514
|
+
db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by') # → msg_1
|
|
1515
|
+
db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode
|
|
1516
|
+
|
|
1517
|
+
# Relations + graph traversal
|
|
1518
|
+
db.link("users:alice", "follows", "users:bob")
|
|
1519
|
+
db.query('FROM users WHERE _id = "alice" TRAVERSE follows')
|
|
1520
|
+
|
|
1521
|
+
# Hash-chain integrity — verify() is a call, head is the commitment
|
|
1522
|
+
assert db.verify() # cryptographic proof — no tampering
|
|
1523
|
+
db.head # → 64-char BLAKE2b Merkle head (a property, not a call)
|
|
1524
|
+
|
|
1525
|
+
# SQL, Redis, MongoDB compatibility adapters
|
|
1526
|
+
from nedb import sql_exec, RedisCompat, MongoClient
|
|
1527
|
+
sql_exec(db, "SELECT * FROM users WHERE status = 'active' ORDER BY age DESC")
|
|
1528
|
+
r = RedisCompat(db); r.execute("HSET", "user:1", "name", "Alice")
|
|
1529
|
+
MongoClient(db)["users"].find({"status": "active"}).sort("age", -1).to_list()
|
|
1543
1530
|
```
|
|
1544
1531
|
|
|
1545
|
-
|
|
1532
|
+
---
|
|
1546
1533
|
|
|
1547
|
-
|
|
1534
|
+
## Official Python client — talk to nedbd over HTTP
|
|
1548
1535
|
|
|
1549
|
-
|
|
1536
|
+
Running the daemon? `nedb.client.NedbClient` is the official client for its
|
|
1537
|
+
HTTP API — extracted from the battle-tested clients that ran a production
|
|
1538
|
+
Redis→NEDB mainnet migration, speaking the full route surface: queries,
|
|
1539
|
+
atomic CAS transactions, TTL, indexes, relations, Merkle proofs, and the
|
|
1540
|
+
Mongo-compat endpoint. Env-var defaults (`NEDBD_URL`, `NEDBD_TOKEN`,
|
|
1541
|
+
`NEDB_DB`) mirror the daemon's own.
|
|
1550
1542
|
|
|
1551
|
-
|
|
1552
|
-
|
|
1553
|
-
| 2,002 coins / 275 kB | **1.93 s** | *minutes* |
|
|
1554
|
-
| 2,549 coins / 366 kB | **1.71 s** | *minutes* |
|
|
1543
|
+
```python
|
|
1544
|
+
from nedb import NedbClient, PreconditionFailed, op_put
|
|
1555
1545
|
|
|
1556
|
-
|
|
1546
|
+
c = NedbClient("http://127.0.0.1:7070", db="app", token="s3cret")
|
|
1547
|
+
c.ensure_database()
|
|
1557
1548
|
|
|
1558
|
-
|
|
1549
|
+
c.put("users", "u1", {"id": "u1", "email": "a@b.c"}, idem="signup-u1")
|
|
1550
|
+
c.query('FROM users WHERE email = "a@b.c"') # full NQL rides through
|
|
1551
|
+
c.query("FROM users AS OF 41") # time-travel included
|
|
1559
1552
|
|
|
1560
|
-
|
|
1553
|
+
# Atomic all-or-nothing transaction with engine-checked preconditions —
|
|
1554
|
+
# the primitive that replaces Redis Lua scripts (if_seq: N = CAS, -1 = create-once)
|
|
1555
|
+
doc = c.get_doc("users", "u1") # docs carry _seq
|
|
1556
|
+
c.tx([op_put("users", "u1", {**doc, "plan": "pro"}, if_seq=doc["_seq"])])
|
|
1557
|
+
|
|
1558
|
+
# Contested writes: retry ONLY on PreconditionFailed, capped backoff
|
|
1559
|
+
def bump():
|
|
1560
|
+
d = c.get_doc("counters", "hits") or {"n": 0}
|
|
1561
|
+
return c.tx([op_put("counters", "hits", {"n": d.get("n", 0) + 1},
|
|
1562
|
+
if_seq=d.get("_seq", -1))])
|
|
1563
|
+
c.cas_retry(bump)
|
|
1564
|
+
|
|
1565
|
+
# Integrity, verifiable WITHOUT trusting the server
|
|
1566
|
+
proof = c.proof(c.log(limit=1)[0]["hash"])
|
|
1567
|
+
from nedb import verify_proof; verify_proof(proof) # -> True, locally
|
|
1568
|
+
```
|
|
1569
|
+
|
|
1570
|
+
A CAS miss raises the **same `PreconditionFailed`** (with the same
|
|
1571
|
+
`.failures` shape) the embedded engine raises — code written against
|
|
1572
|
+
`NEDB.tx` ports to the HTTP client without changing its except-clauses.
|
|
1573
|
+
Typed errors throughout: `NedbAuthError`, `NedbNotFound`, `NedbBadRequest`,
|
|
1574
|
+
`NedbConflict`, `CasExhausted`.
|
|
1561
1575
|
|
|
1562
1576
|
---
|
|
1563
1577
|
|
|
1564
|
-
##
|
|
1578
|
+
## Redis layer-2 — wrap_redis()
|
|
1579
|
+
|
|
1580
|
+
Already running on Redis? Wrap your connection in one line and gain NEDB features *alongside* your existing Redis app — no migration required.
|
|
1565
1581
|
|
|
1582
|
+
```python
|
|
1583
|
+
import redis, json
|
|
1584
|
+
from nedb import wrap_redis
|
|
1585
|
+
|
|
1586
|
+
r = wrap_redis(redis.Redis("localhost", 6379), db_name="rideshare")
|
|
1587
|
+
|
|
1588
|
+
# Step 1 — register: map Redis key globs to NEDB collections (chainable)
|
|
1589
|
+
(r.nedb
|
|
1590
|
+
.register("driver:*", collection="driver", value_parser=json.loads)
|
|
1591
|
+
.register("trip:*", collection="trip", value_type="hash")
|
|
1592
|
+
)
|
|
1593
|
+
|
|
1594
|
+
# Step 2 — backfill: import all existing Redis data into NEDB in one pass
|
|
1595
|
+
imported = r.nedb.backfill() # → int (keys imported)
|
|
1596
|
+
|
|
1597
|
+
# Step 3 — shadow: all future r.set/hset/... auto-chain into NEDB
|
|
1598
|
+
r.nedb.shadow_writes = True
|
|
1599
|
+
|
|
1600
|
+
# ─── Alice's app keeps running — zero changes ───────────────────────────
|
|
1601
|
+
r.set("driver:d1", json.dumps({"name": "Bob", "status": "active"})) # ← shadowed
|
|
1602
|
+
r.hset("trip:t1", mapping={"status": "en_route", "driver_id": "d1"}) # ← shadowed
|
|
1603
|
+
|
|
1604
|
+
# ─── New features available on the same connection ──────────────────────
|
|
1605
|
+
r.nedb.query('FROM driver WHERE status = "active" ORDER BY lat ASC')
|
|
1606
|
+
r.nedb.verify() # → True (every write chain-verified)
|
|
1607
|
+
r.nedb.head() # → 64-char BLAKE2b commitment hash
|
|
1566
1608
|
```
|
|
1567
|
-
┌──────────────────────────────────────────────────────────┐
|
|
1568
|
-
put/del → │ OpLog (BLAKE2b hash chain · per-client nonce · │ ← single source of truth
|
|
1569
|
-
link │ idempotency keys · causal provenance fields) │
|
|
1570
|
-
└───────────────┬──────────────────────────────────────────┘
|
|
1571
|
-
deterministic fold │ (state = pure function of the log)
|
|
1572
|
-
┌──────────────┬──────────┴──────┬───────────────┬────────────────┐
|
|
1573
|
-
▼ ▼ ▼ ▼ ▼
|
|
1574
|
-
MVCC store Relations Indexes CauseMap BlobStore
|
|
1575
|
-
(time-travel) (graph+AS OF) eq/ord/search (reverse index) (Cascade CDC)
|
|
1576
1609
|
|
|
1577
|
-
|
|
1578
|
-
Thread-safe → │ Sequencer (group-commit) │ ← single writer, parallel readers
|
|
1579
|
-
│ — one committer thread/db │
|
|
1580
|
-
│ — batch fsync │
|
|
1581
|
-
└─────────────────────────────────┘
|
|
1610
|
+
**Isolation guarantee:** NEDB never writes to Alice's namespace. It owns only:
|
|
1582
1611
|
|
|
1583
|
-
|
|
1584
|
-
|
|
1585
|
-
|
|
1612
|
+
| Key | Type | Purpose |
|
|
1613
|
+
|-----|------|---------|
|
|
1614
|
+
| `nedb:{db_name}:oplog` | Redis Stream | append-only op log |
|
|
1615
|
+
| `nedb:{db_name}:snapshot` | Redis Hash | checkpoint |
|
|
1616
|
+
| `nedb:{db_name}:meta` | Redis Hash | index config |
|
|
1617
|
+
|
|
1618
|
+
See [`examples/fakeredis_demo.py`](examples/fakeredis_demo.py) for a full local demo (no Redis server needed).
|
|
1619
|
+
|
|
1620
|
+
---
|
|
1621
|
+
|
|
1622
|
+
## Node.js
|
|
1623
|
+
|
|
1624
|
+
```javascript
|
|
1625
|
+
import { NedbCore } from "nedb-engine";
|
|
1626
|
+
|
|
1627
|
+
const db = new NedbCore(); // in-memory
|
|
1628
|
+
// const db = NedbCore.open("./data"); // durable
|
|
1629
|
+
|
|
1630
|
+
db.createIndex("users", "status", "eq");
|
|
1631
|
+
db.put("users", "alice", JSON.stringify({ name: "Alice", age: 31, status: "active" }));
|
|
1632
|
+
|
|
1633
|
+
// Time-travel
|
|
1634
|
+
const snap = db.seq(); // BigInt
|
|
1635
|
+
db.put("users", "alice", JSON.stringify({ name: "Alice", age: 32, status: "retired" }));
|
|
1636
|
+
JSON.parse(db.getAsOf("users", "alice", snap)).age; # → 31
|
|
1637
|
+
|
|
1638
|
+
// Full NQL
|
|
1639
|
+
const rows = db.query('FROM users WHERE status = "active" ORDER BY age ASC');
|
|
1640
|
+
rows.map(r => JSON.parse(r));
|
|
1641
|
+
|
|
1642
|
+
// Tamper evidence
|
|
1643
|
+
db.verify(); // → true
|
|
1644
|
+
db.head(); // → 64-char BLAKE2b commitment hash
|
|
1645
|
+
db.seq(); // → BigInt
|
|
1586
1646
|
```
|
|
1587
1647
|
|
|
1588
1648
|
---
|
|
@@ -1615,53 +1675,98 @@ const rows = await db.query("FROM blocks LIMIT 10");
|
|
|
1615
1675
|
|
|
1616
1676
|
---
|
|
1617
1677
|
|
|
1678
|
+
## NEDB v3 — Segment / Pack Object Store
|
|
1679
|
+
|
|
1680
|
+
**v3 is an opt-in storage substrate that replaces the loose one-file-per-object layout with append-only *segment packs* — the difference between a chainstate flush that takes *minutes* and one that takes *under two seconds*.** It is **off by default** (byte-for-byte v2), enabled with one flag, and **transparent** to everything above the storage layer: NQL, `AS OF`, `VALID AS OF`, `TRACE`, the BLAKE2b Merkle head, and causal provenance all behave identically.
|
|
1681
|
+
|
|
1682
|
+
### Why it exists
|
|
1683
|
+
|
|
1684
|
+
v2 stores every document version as its own content-addressed file at `objects/{hash[:2]}/{hash[2:]}`. That makes writes trivially atomic (write `.tmp` → `rename`) and corruption-proof — but each write costs a file create + `fsync` + rename **plus** a directory B-tree update. At scale that filesystem-metadata churn dominates: on a busy disk it caps sustained writes around **~185/s**, and a batch flush of a few thousand objects degrades into minutes. The bottleneck is the *number of files touched*, not the bytes written.
|
|
1685
|
+
|
|
1686
|
+
### What it does
|
|
1687
|
+
|
|
1688
|
+
v3 batches objects into append-only **segment packs** — `objects/segments/seg-NNNNNN.dat` — where each record is `[content_len: u32-LE][content]`. A write appends to the active segment and updates an in-memory `hash → (segment_id, offset, len)` map; a batch commits with a **single `fsync`**. Thousands of per-file syscalls collapse into one sequential append plus one durability point, so **flush cost scales with bytes (sequential I/O), not object-count × syscall overhead.**
|
|
1689
|
+
|
|
1690
|
+
- **Compaction / pruning** — `compact()` keeps the *live set* (the current version of every document, resolved from the id-index), rewrites those records into fresh segments, and reclaims the superseded/dead versions.
|
|
1691
|
+
- **`.idx` sidecars** — each segment carries a sidecar (`NIX1` magic + entry count + fixed 44-byte entries + a BLAKE2b-256 checksum) so reopen rebuilds the in-memory index by reading the sidecar instead of scanning the whole pack. A missing or corrupt sidecar falls back to a full scan-and-heal — slower, never fatal.
|
|
1692
|
+
- **Dual-read migration** — opening an existing v2 store in v3 mode is **non-destructive**: old loose objects stay fully readable, and only *new* writes go to segments. No migration step, no downtime, no rewrite.
|
|
1693
|
+
- **Durable flush-on-close** — `flush_all()` (and `Db`'s `Drop`) fsync the active segment, matching the flush-on-close contract of sled / RocksDB.
|
|
1694
|
+
|
|
1695
|
+
### How to enable
|
|
1696
|
+
|
|
1697
|
+
```bash
|
|
1698
|
+
# Engine / nedbd-v2 (the native daemon from npm / the native wheel)
|
|
1699
|
+
nedbd-v2 --dag-v3 --data /var/lib/nedb # real flag as of v2.4.3 — or set NEDB_DAG_V3=1
|
|
1700
|
+
|
|
1701
|
+
# itcd — Bitcoin-fork node embedding NEDB via nedb-ffi
|
|
1702
|
+
interchainedd -dagv3 # puts chainstate AND block index on segments
|
|
1703
|
+
```
|
|
1704
|
+
|
|
1705
|
+
The switch is read once, when each database's object store is constructed at open time. Default off → v2 loose objects.
|
|
1706
|
+
|
|
1707
|
+
### Real-world result
|
|
1708
|
+
|
|
1709
|
+
itcd (a Bitcoin Core 0.21 fork that replaces LevelDB chainstate with NEDB) syncing on `-dagv3`, measured `FlushStateToDisk` on real chainstate:
|
|
1710
|
+
|
|
1711
|
+
| Flush (coins → disk) | v3 segment store | v2 loose store |
|
|
1712
|
+
|---|---|---|
|
|
1713
|
+
| 2,002 coins / 275 kB | **1.93 s** | *minutes* |
|
|
1714
|
+
| 2,549 coins / 366 kB | **1.71 s** | *minutes* |
|
|
1715
|
+
|
|
1716
|
+
Note the *larger* batch finishing *faster* — v3's cost is dominated by the single per-batch `fsync`, not per-coin work, so effective throughput (~1,000–1,500 coins/s here) climbs as batches grow, against the loose store's ~185 writes/s metadata ceiling. The gap only widens as the UTXO set grows: sequential-append cost tracks data volume, while per-file cost compounds with object count.
|
|
1717
|
+
|
|
1718
|
+
### When to use it
|
|
1719
|
+
|
|
1720
|
+
Reach for v3 on high-write, large-object-count workloads — blockchain chainstate / block index, event sourcing, high-frequency agent memory. For small or read-mostly stores the loose layout is perfectly fine, which is exactly why v3 stays opt-in.
|
|
1721
|
+
|
|
1722
|
+
---
|
|
1723
|
+
|
|
1618
1724
|
## Repo layout
|
|
1619
1725
|
|
|
1620
1726
|
```
|
|
1621
1727
|
python/nedb/ reference engine (pure Python — always-works baseline)
|
|
1622
1728
|
rust/
|
|
1729
|
+
nedb-v2/ v2 DAG engine (tokio + axum + BLAKE2b DAG) — the core everything binds
|
|
1730
|
+
nesql-cli/ the nesql CLI (query / root / inspect / diff / tag / branch / merge)
|
|
1623
1731
|
nedb-core/ v1 production Rust engine (shared by both runtimes)
|
|
1624
1732
|
nedb-py/ maturin PyO3 binding → PyPI native wheels
|
|
1625
1733
|
nedb-node/ napi-rs binding → npm native addons
|
|
1626
|
-
nedb-
|
|
1627
|
-
|
|
1628
|
-
|
|
1629
|
-
|
|
1734
|
+
nedb-wrap/ the wrap_* surface at native speed
|
|
1735
|
+
vendor/postgresql/ PostgreSQL 17.4 grammar, vendored (gram.y, kwlist.h, system_views.sql, information_schema.sql)
|
|
1736
|
+
distributions/ crypto-database + aof-db submodules (the tri-distribution release)
|
|
1737
|
+
bench/ benchmarks.py + RESULTS.md — the dated embedded-core numbers
|
|
1630
1738
|
tests/ engine + concurrent + causal + bitemporal + deploy + perf benchmarks
|
|
1631
|
-
|
|
1632
|
-
docs/
|
|
1739
|
+
vectors/ state_root_v1.json — cross-engine state-root test vectors
|
|
1740
|
+
docs/ SPEC.md · CLI.md · DURABILITY.md · REPLICATION.md · BENCH-sqlselect.md
|
|
1741
|
+
client/ nedb-engine-client — async Python + TypeScript HTTP clients
|
|
1742
|
+
examples/ agent-loop · mini-chain · resp2 demos · fakeredis demo
|
|
1743
|
+
scripts/ release.py · test-cast.sh · bench_index_range.py · seed-shop.sh
|
|
1633
1744
|
```
|
|
1634
1745
|
|
|
1635
1746
|
---
|
|
1636
1747
|
|
|
1637
|
-
##
|
|
1638
|
-
|
|
1639
|
-
|
|
1640
|
-
|
|
1641
|
-
|
|
1642
|
-
-
|
|
1643
|
-
|
|
1644
|
-
-
|
|
1645
|
-
|
|
1646
|
-
-
|
|
1647
|
-
-
|
|
1648
|
-
|
|
1649
|
-
-
|
|
1650
|
-
-
|
|
1651
|
-
|
|
1652
|
-
-
|
|
1653
|
-
-
|
|
1654
|
-
-
|
|
1655
|
-
-
|
|
1656
|
-
-
|
|
1657
|
-
-
|
|
1658
|
-
-
|
|
1659
|
-
- [ ] PyO3 + napi-rs bindings updated to v2 DAG API
|
|
1660
|
-
- [ ] NEDB Studio DAG mode toggle
|
|
1661
|
-
- [ ] Merkle inclusion proofs — prove a document existed at a specific time to a third party
|
|
1662
|
-
- [ ] Git-style branching — fork database state, experiment, merge or discard
|
|
1663
|
-
- [ ] Agent Memory SDK — `Memory.remember()` / `Memory.recall()` / `Memory.trace()`
|
|
1664
|
-
- [ ] Live query subscriptions (SSE) — push diffs when query results change
|
|
1748
|
+
## Ship log — the highlights, in order
|
|
1749
|
+
|
|
1750
|
+
NEDB's release history outgrew any changelog, and the README was where versions went to pile up.
|
|
1751
|
+
This is the map; the commits are the story.
|
|
1752
|
+
|
|
1753
|
+
- **8.x — neSQL ships.** PostgreSQL's real grammar vendored and extended; one evaluator answers every
|
|
1754
|
+
`SELECT` with nothing to enable; NQL's verbs are SQL clauses; the `nesql` CLI ships in the wheel.
|
|
1755
|
+
- **7.x — the neSQL endpoint surface.** The HTTP `/query` route speaks both dialects through one
|
|
1756
|
+
router; the CLI's publish path lands.
|
|
1757
|
+
- **6.x — hardening, discipline, the road to one name.**
|
|
1758
|
+
- **5.x — neSQL named, registered, vendored.** The grammar lands at `vendor/postgresql/`;
|
|
1759
|
+
`nesql` reserved on all three registries.
|
|
1760
|
+
- **4.0 — BUSL-1.1** (free under $1M revenue; Apache 2.0 on the 2030-09-11 Change Date).
|
|
1761
|
+
- **3.3 — the query language grows up.** Full boolean predicates in both engines; nine silent
|
|
1762
|
+
defects fixed; cross-engine parity gated.
|
|
1763
|
+
- **3.2 — the wrap family.** Redis/SQLite/Postgres/MySQL/Mongo adapters; durability fixes; PR CI.
|
|
1764
|
+
- **3.0–3.3.1 — the MIT era** (irrevocably MIT).
|
|
1765
|
+
- **2.8 — Cast.** A 3.33M-parameter model trained inside the engine's own parser; drift detection.
|
|
1766
|
+
- **2.5 — flush-on-exit, nedb-cli, inspector, replication contract.**
|
|
1767
|
+
- **2.4 — v3 segment store** (`--dag-v3`): itcd chainstate flush minutes → ~1.3 s.
|
|
1768
|
+
- **2.2 — v2 DAG engine.** Content-addressed Merkle DAG, O(1) warm start, SSE `/events`.
|
|
1769
|
+
- **1.x — v1 AOF engine.** Hash-chained log, MVCC time-travel, bi-temporal, `TRACE caused_by`.
|
|
1665
1770
|
|
|
1666
1771
|
---
|
|
1667
1772
|
|
|
@@ -1673,14 +1778,16 @@ Prompt-to-database scaffolding GUI with schema graph, NQL console, time-travel s
|
|
|
1673
1778
|
|
|
1674
1779
|
---
|
|
1675
1780
|
|
|
1676
|
-
## Repos
|
|
1781
|
+
## Repos & packages
|
|
1677
1782
|
|
|
1678
|
-
|
|
|
1783
|
+
| Where | What |
|
|
1679
1784
|
|---|---|
|
|
1680
|
-
| [
|
|
1785
|
+
| [Eth-Interchained/nedb](https://github.com/Eth-Interchained/nedb) | **canonical source** — engine, Rust core, CI, this README |
|
|
1786
|
+
| [aiassistsecure/nedb](https://github.com/aiassistsecure/nedb) | mirror + [GitHub Pages site](https://nedb.aiassist.net) |
|
|
1787
|
+
| [Eth-Interchained/neSQL](https://github.com/Eth-Interchained/neSQL) | the language — both halves of the grammar, CLI source |
|
|
1681
1788
|
| [aiassistsecure/nedb-studio](https://github.com/aiassistsecure/nedb-studio) | Studio UI (GPLv3) |
|
|
1682
1789
|
|
|
1683
|
-
**Packages:** [PyPI nedb-engine](https://pypi.org/project/nedb-engine/) · [npm nedb-engine](https://www.npmjs.com/package/nedb-engine)
|
|
1790
|
+
**Packages:** [PyPI nedb-engine](https://pypi.org/project/nedb-engine/) · [npm nedb-engine](https://www.npmjs.com/package/nedb-engine) · [crates.io nedb-engine](https://crates.io/crates/nedb-engine) — plus the aligned `crypto-database` and `aof-db` distributions on all three registries.
|
|
1684
1791
|
|
|
1685
1792
|
---
|
|
1686
1793
|
|
|
@@ -1708,7 +1815,11 @@ Requires `GITHUB_TOKEN` (`repo` + `workflow` scope) in the environment. It never
|
|
|
1708
1815
|
|
|
1709
1816
|
## License
|
|
1710
1817
|
|
|
1711
|
-
**
|
|
1818
|
+
**Business Source License 1.1** — free for any organisation under USD $1M annual revenue, including
|
|
1819
|
+
commercial and production use. Converts to **Apache 2.0** on **2030-09-11**, automatically and
|
|
1820
|
+
permanently. Versions 3.0.0–3.3.1 remain MIT, irrevocably. See [`LICENSE`](LICENSE) and
|
|
1821
|
+
[`COPYING-APACHE-2.0.txt`](COPYING-APACHE-2.0.txt).
|
|
1822
|
+
|
|
1712
1823
|
© 2026 INTERCHAINED LLC — [interchained.org](https://interchained.org)
|
|
1713
1824
|
|
|
1714
1825
|
---
|