nedb-engine 7.2.0 → 8.8.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,7 +5,8 @@
5
5
  **Content-addressed Merkle DAG · Hash-chained · Time-traveling · Bi-temporal · Causally-provable embedded database.**
6
6
 
7
7
  Replay-protected · idempotent · relational · filterable · sortable · searchable · concurrent.
8
- One Rust core → ships to **PyPI** and **npm** from a single source.
8
+ One Rust core → ships to **PyPI**, **npm** and **crates.io** from a single source,
9
+ at the same version on the same tag.
9
10
 
10
11
  [![PyPI](https://img.shields.io/pypi/v/nedb-engine?label=PyPI&color=6366f1)](https://pypi.org/project/nedb-engine/)
11
12
  [![crates.io](https://img.shields.io/crates/v/nedb-engine?label=crates.io&color=f97316)](https://crates.io/crates/nedb-engine)
@@ -15,7 +16,7 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
15
16
  [![nedb-engine-client npm](https://img.shields.io/npm/v/nedb-engine-client?label=nedb-engine-client&color=34d399)](https://www.npmjs.com/package/nedb-engine-client)
16
17
  [![License: BUSL-1.1](https://img.shields.io/badge/license-BUSL--1.1-f59e0b)](https://github.com/Eth-Interchained/nedb/blob/master/LICENSE) [![Free under $1M revenue](https://img.shields.io/badge/free%20under%20%241M%20revenue-22c55e)](https://github.com/Eth-Interchained/nedb/blob/master/LICENSE)
17
18
 
18
- **[Studio → studio.interchained.org](https://studio.interchained.org)** · **[nedb.aiassist.net](https://nedb.aiassist.net)**
19
+ **[Studio → studio.interchained.org](https://studio.interchained.org)** · **[nedb.aiassist.net](https://nedb.aiassist.net)** · **[neSQL → the language](https://github.com/Eth-Interchained/neSQL)**
19
20
 
20
21
  > ## 🟢 Free in production under $1M revenue
21
22
  > NEDB is licensed under the **Business Source License 1.1** (since 4.0.0). If your organisation's annual
@@ -31,6 +32,101 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
31
32
 
32
33
  ---
33
34
 
35
+ ## Provenance — the reason NEDB exists
36
+
37
+ Every database stores *what*. NEDB stores *what*, *when*, *when it was true*, and *why* — all sealed
38
+ in a cryptographic hash chain that proves none of it was tampered with.
39
+
40
+ Two time axes, one causal graph, one hash chain:
41
+
42
+ | Question | Clause / field | Answers |
43
+ |---|---|---|
44
+ | *What* is the value? | the row itself | current state |
45
+ | *When was it written?* | `AS OF <seq>` — transaction time | "set to 999 at seq 41" |
46
+ | *When was it true in the world?* | `VALID AS OF "<date>"` — valid time | "the 2024 rate" |
47
+ | *Why did it happen?* | `caused_by` → `TRACE caused_by` | the exact parent writes |
48
+ | *What did it cause?* | `TRACE caused_by REVERSE` | forward consequence |
49
+ | *Can I prove any of this to a third party?* | `verify()` · Merkle head · `proof()` | yes — locally, without trusting the server |
50
+
51
+ ```python
52
+ from nedb import NEDB
53
+
54
+ db = NEDB("./mydata")
55
+
56
+ # Transaction time — the value at any past sequence, forever (no garbage collection)
57
+ snap = db.seq
58
+ db.put("users", "alice", {"age": 32})
59
+ db.get("users", "alice", as_of=snap) # → the prior version
60
+
61
+ # Valid time — what the world believed on a date, regardless of when written
62
+ db.put("policy", "rate_2024", {"pct": 5.0}, valid_from="2024-01-01", valid_to="2024-12-31")
63
+ db.query('FROM policy VALID AS OF "2024-06-15"') # → rate 5.0
64
+
65
+ # Both axes at once — what did the system KNOW at seq 200 about 2024-02-15?
66
+ db.query('FROM policy AS OF 200 VALID AS OF "2024-02-15"')
67
+
68
+ # Causal provenance — an edge, not a comment
69
+ db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
70
+ seq_msg = db.seq
71
+ db.put("beliefs", "dark_mode", {"value": True},
72
+ caused_by=[seq_msg], evidence="user_message", confidence=0.95)
73
+
74
+ db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by') # → msg_1
75
+ db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode
76
+
77
+ # Tamper evidence — a 64-char BLAKE2b Merkle head on every write, verifiable offline
78
+ assert db.verify()
79
+ ```
80
+
81
+ Provenance travels with the row, and it is **selectable like any other column** — `_id`, `_hash`,
82
+ `_seq`, `_coll` on every document, `_caused_by` / `_valid_from` / `_valid_to` where set:
83
+
84
+ ```sql
85
+ SELECT _id, _hash, _seq FROM audit ORDER BY _seq;
86
+ SELECT _id FROM audit TRACE caused_by; -- works in NQL and in SQL
87
+ INSERT INTO audit (_id, _caused_by, kind) VALUES ('leaf', '<parent-hash>', 'reprice');
88
+ ```
89
+
90
+ One hash that commits to **what a database currently says** — independent of the route it took to
91
+ get there — is a separate primitive, and it is specified, versioned, and test-locked across both
92
+ engines: see [`docs/state-root-v1.md`](docs/state-root-v1.md) and
93
+ [`vectors/state_root_v1.json`](vectors/state_root_v1.json). The running Merkle head commits to
94
+ *history*; the state root commits to *now*; replica agreement and drift detection compare roots.
95
+
96
+ **The chain is not a side feature — it is the write path.** A SQL `UPDATE` is a new version, a SQL
97
+ `DELETE` is a tombstone, and `verify()` still passes afterwards, because writes through any surface
98
+ are ordinary engine writes. There is no configuration that silently drops history.
99
+
100
+ ---
101
+
102
+ ## The four-axis model
103
+
104
+ | Axis | Mechanism | Other databases |
105
+ |---|---|:---:|
106
+ | **What** — current state | content-addressed objects, BLAKE2b-verified on every read | the only axis they have |
107
+ | **When** — transaction time | every version retained, `AS OF <seq>` | ❌ |
108
+ | **True-when** — valid time | bi-temporal fields, `VALID AS OF "<date>"` | ❌ |
109
+ | **Why** — causal provenance | `caused_by` edges in the DAG, `TRACE` both directions | ❌ |
110
+
111
+ SQLite, Redis and MongoDB store *what*. Add triggers and you have written a side table no query
112
+ optimizer knows about, with no chain and no proof. NEDB's provenance is native, indexed,
113
+ traversable — and the same hash chain that keeps it honest also protects it.
114
+
115
+ ### The comparison table
116
+
117
+ | Capability | NEDB | SQLite | Redis | MongoDB |
118
+ |---|:---:|:---:|:---:|:---:|
119
+ | Hash-chained tamper evidence | ✅ | ❌ | ❌ | ❌ |
120
+ | Time-travel reads (`AS OF seq`) | ✅ | ❌ | ❌ | ❌ |
121
+ | Bi-temporal (`VALID AS OF date`) | ✅ | ❌ | ❌ | ❌ |
122
+ | Causal Write Provenance | ✅ | ❌ | ❌ | ❌ |
123
+ | Replay-protected idempotent writes | ✅ | ❌ | ❌ | ❌ |
124
+ | SQL + Redis + MongoDB adapters | ✅ | — | — | — |
125
+ | Concurrent group-commit daemon | ✅ | ❌ | ✅ | ✅ |
126
+ | At-rest AES-256-GCM encryption | ✅ | ❌ | ❌ | — |
127
+
128
+ ---
129
+
34
130
  ## [neSQL](https://github.com/Eth-Interchained/neSQL) — the language this engine speaks
35
131
 
36
132
  Nobody should have to learn a query language to use a database. That sentence cost
@@ -40,15 +136,31 @@ NEDB's PostgreSQL endpoint answers `psql`, SQLAlchemy Core **and** ORM, asyncpg
40
136
  node-postgres against a live store. It used to get there by *translating* SQL into
41
137
  NQL, and a translation can only reach as far as the target language's shape.
42
138
 
43
- **neSQL is the name for what replaced that.** PostgreSQL's real grammar (`gram.y`,
44
- 19,513 lines, 492 keywords, vendored from 17.4 at
45
- [`vendor/postgresql/`](vendor/postgresql/) with its licence intact), extended with
46
- NEDB's temporal and causal clauses. **Two front-ends, one plan. NQL folded in, not
47
- deleted.**
139
+ **neSQL is the name for what replaced that**, and it is exactly as much of an
140
+ addition as it sounds like:
141
+
142
+ ```
143
+ neSQL = PostgreSQL SQL · inherited whole, not reimplemented
144
+ + NEDB SQL · what a permanent, hash-chained store can answer
145
+ ```
146
+
147
+ **We inherit, then we gain.** The left-hand side is PostgreSQL's real grammar —
148
+ `gram.y`, 19,513 lines and 492 keywords, vendored from 17.4 at
149
+ [`vendor/postgresql/`](vendor/postgresql/) with its licence intact. Not a subset,
150
+ not a lookalike: the definition every other tool in the world was built against.
151
+ If it is valid PostgreSQL and the evaluator can parse it, it runs.
152
+
153
+ The right-hand side is what NEDB adds because it can — `AS OF SYSTEM TIME`,
154
+ `VALID AS OF`, `SEARCH`, `TRACE`, `TRAVERSE`. These are clauses PostgreSQL has no
155
+ spelling for, because a store that overwrites has nothing to point them at. They
156
+ are additions **to** the vendored grammar, never deviations **from** it.
48
157
 
49
- **neQL** is the name for the pair NQL *and* PostgreSQL SQL, one language with two
50
- halves. Which half a statement is read as is decided **structurally**, not guessed:
51
- NQL statements begin `FROM`, and PostgreSQL has no statement form that begins with
158
+ So neSQL is not a dialect of SQL that you have to learn around. It is PostgreSQL
159
+ plus the questions a database with permanent memory can be asked. Anything you
160
+ already write keeps working; the new clauses are there when you need them.
161
+
162
+ Which half a statement is read as is decided **structurally**, never guessed:
163
+ NQL's own form begins `FROM`, PostgreSQL has no statement form that begins with
52
164
  `FROM`, so the leading keyword partitions the two vocabularies rather than hinting
53
165
  at them. A first word in neither is refused *naming both*.
54
166
 
@@ -59,7 +171,7 @@ evaluator with no flag to set, the `nesql` CLI likewise.
59
171
  [![neSQL on crates.io](https://img.shields.io/crates/v/nesql?label=nesql%20·%20crates.io&color=a855f7)](https://crates.io/crates/nesql)
60
172
  [![neSQL on npm](https://img.shields.io/npm/v/nesql-engine?label=nesql-engine%20·%20npm&color=a855f7)](https://www.npmjs.com/package/nesql-engine)
61
173
 
62
- Those three badges read **0.0.1** next to an engine at 6.1.0, and that is
174
+ Those three badges read **0.0.1** next to an engine at 8.0.0, and that is
63
175
  deliberate rather than neglected. They are **reserved names**: each package loads,
64
176
  reports the vendored PostgreSQL release, and answers `is_release() == false`,
65
177
  because a package that imports cleanly and then lies is worse than one that is not
@@ -67,10 +179,10 @@ published. The engine you actually install is `nedb-engine`. The
67
179
  [neSQL repository](https://github.com/Eth-Interchained/neSQL) holds the language —
68
180
  both halves of the grammar and the CLI's source, side by side.
69
181
 
70
- ### `nesql` — the CLI, and it speaks neQL
182
+ ### `nesql` — the CLI, and it speaks neSQL
71
183
 
72
- Ships in this release, no flag. `nesql` opens a store directly no daemon, no
73
- port — and answers both halves of the language through **one** `query` command:
184
+ Ships inside `pip install nedb-engine` no daemon, no portand answers both
185
+ halves of the language through **one** `query` command:
74
186
 
75
187
  ```console
76
188
  $ nesql --db ./store query "SELECT who, total FROM orders ORDER BY total DESC"
@@ -202,7 +314,7 @@ the bar for changing that written down.
202
314
 
203
315
  ---
204
316
 
205
- ## New in 3.3.0 — the query language grew up
317
+ ## The query language grew up · *landed in 3.3.0*
206
318
 
207
319
  `WHERE` was six operators wide (`= != > < >= <=`) joined by an implicit `AND`.
208
320
  It now takes a full boolean expression, in **both** engines, and the clauses
@@ -342,17 +454,21 @@ a seq is exact where a time would be approximate.
342
454
  compacted store answers "not available at that sequence" rather than
343
455
  returning a stale value, and `verify()` stays clean.
344
456
 
345
- Provenance is selectable like any other column:
457
+ **NEDB speaks SQL. That sentence used to carry a caveat, and no longer does.**
346
458
 
347
- ```sql
348
- SELECT _id, _hash, _seq FROM audit ORDER BY _seq;
349
- ```
459
+ For most of this project's life it was true that the endpoint served a
460
+ documented *subset* of `SELECT` **translated** into NQL and every refusal in
461
+ the table below traced to that one cause: a rewrite can only reach as far as the
462
+ target language's shape, and NQL's shape is single-collection with no projection.
350
463
 
351
- **This is not "NEDB speaks SQL", and the endpoint is careful to say so.** It is
352
- a documented subset of `SELECT` **translated** to NQL and that word is doing
353
- all the work in this sentence. Every refusal below traces to the same cause:
354
- NQL is the engine's native language, so SQL has to be rewritten into it, and a
355
- rewrite can only ever reach as far as the target language's shape.
464
+ That translator no longer answers `SELECT`. The evaluator does, for every
465
+ statement it can parse, with nothing to enable. It is kept for writes and for
466
+ anything outside the `SELECT` grammar, which is why a statement it cannot parse
467
+ still gets an answer rather than an error.
468
+
469
+ The table is preserved below as history, because the distinction between "the
470
+ engine could never do this" and "the translator could not reach it" is the whole
471
+ story of how neSQL happened — and only one of those was ever true.
356
472
 
357
473
  | Expressible in NQL | Not expressible there, and why | the evaluator |
358
474
  | --- | --- | --- |
@@ -372,42 +488,6 @@ NEDB is append-only *so that history cannot be discarded* — that is the produc
372
488
  not a gap — and DDL is refused because collections are created by the first write
373
489
  to them. Those answers do not change.
374
490
 
375
- ### Every other row on that table was a translation artefact — and one flag removes them
376
-
377
- > ### 🆕 [**neSQL**](https://github.com/Eth-Interchained/neSQL) — PostgreSQL's grammar, NEDB's memory
378
- >
379
- > Those refusals were never the engine's limits. `sqlselect.rs` has had
380
- > nested-loop and hash joins, subqueries, `EXISTS`, quantified comparisons, set
381
- > operations, `array_agg(x ORDER BY y)` and derived tables for some time — they
382
- > were simply unreachable *through a translator*, because the translator's
383
- > target was NQL. They are reachable now, with nothing to set.
384
- >
385
- > neSQL vendors PostgreSQL's **real grammar** — `gram.y`, 19,513 lines and 492
386
- > keywords, from 17.4, licence intact — and extends it with the clauses NEDB
387
- > needs, rather than rewriting SQL into a language that cannot express it.
388
- >
389
- > It is worth knowing *why* the temporal clauses were never free: `SYSTEM_TIME`,
390
- > `PERIOD` and `PORTION` appear **zero** times in PostgreSQL's grammar. Postgres
391
- > has no temporal SQL at all, and `AS OF SYSTEM TIME` is a CockroachDB
392
- > extension — so NEDB's temporal clauses are additions *to* the vendored grammar
393
- > rather than deviations *from* it. The same road CockroachDB, Materialize and
394
- > RisingWave took. What the grammar does hand over free: `WITH RECURSIVE`,
395
- > window functions, `GROUPING SETS` and `MERGE`.
396
- >
397
- > ```bash
398
- > pip install nesql · cargo add nesql · npm install nesql-engine
399
- > ```
400
- >
401
- > Those three are **reserved names**, not a product: each loads and answers
402
- > `is_release() == false`, because a package that imports cleanly and then lies
403
- > is worse than one that isn't published. The engine is what ships today, and
404
- > neSQL will be this same engine under its own name.
405
- >
406
- > **NQL is not being deleted, and it is not being wrapped.** Its verbs are SQL
407
- > clauses now — `AS OF SYSTEM TIME`, `VALID AS OF`, `SEARCH` — parsed by the SQL
408
- > side and executed by the NQL engine. One implementation, two front-ends,
409
- > neither one a second-class guest.
410
-
411
491
  Every refusal names the boundary instead of saying "syntax error", and a
412
492
  grouped query that projects a column SQL would reject gets Postgres's own
413
493
  message rather than a silent `NULL`.
@@ -521,7 +601,7 @@ SQL `UPDATE`, the prior value is still readable at its original sequence.
521
601
 
522
602
  ---
523
603
 
524
- ## New in 3.2.0 — wrap the databases you already run
604
+ ## Wrap the databases you already run · *landed in 3.2.0*
525
605
 
526
606
  NEDB adds **tamper-evident causal provenance to a database you already have**, in one line, without
527
607
  rip-and-replace. Five adapters, one surface:
@@ -616,7 +696,7 @@ permissive, and the two Python runtime dependencies are BSD and Apache.
616
696
  **Versions 3.0.0 – 3.3.1 stay MIT, irrevocably.** If you already have NEDB at 3.3.1 or earlier, your
617
697
  rights in that copy are untouched. This applies to 4.0.0 and later only.
618
698
 
619
- ### Also in 3.2.0
699
+ ### Also landed in 3.2.0
620
700
 
621
701
  - **A durability defect that pinned every embedded database.** The background flush ticker held a
622
702
  strong `Arc<Db>` in an unconditional loop, so the handle was never dropped: the exclusive data-dir
@@ -636,7 +716,7 @@ rights in that copy are untouched. This applies to 4.0.0 and later only.
636
716
 
637
717
  ---
638
718
 
639
- ## Earlier 2.8.6 durability & recovery
719
+ ## Durability & recovery · *landed in 2.8.6*
640
720
 
641
721
  Three defects found by killing a real engine at every persistence boundary and by filling a real
642
722
  filesystem to zero free blocks. **If you are on 2.8.5 or earlier, upgrade.**
@@ -681,11 +761,11 @@ value. Ten writes drain as nine records. Changing the convention would break exi
681
761
 
682
762
  ---
683
763
 
684
- ## NEDB v3.2.0 Production Stable
764
+ ## Distributionthree aligned distributions, one tag
685
765
 
686
- **Current stable: 3.2.0** — NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with native addons for **macOS (arm64 + x86_64), Linux (x86_64 + aarch64, glibc + musl) and Windows x86_64** (see [**Releasing**](#releasing) below). All native wheels (Linux + Windows on GitHub Actions; macOS on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
766
+ NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with native addons for **macOS (arm64 + x86_64), Linux (x86_64 + aarch64, glibc + musl) and Windows x86_64** (see [**Releasing**](#releasing) below). All native wheels (Linux + Windows on GitHub Actions; macOS on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
687
767
 
688
- ### New in 2.8.0 Cast: the database understands English
768
+ ### Cast — the database understands English · *landed in 2.8.0*
689
769
 
690
770
  `POST /v1/databases/<name>/cast` turns a short English prompt into NQL, using a **3.33M-parameter model that runs locally on CPU**. No API key, no network call, no per-token bill.
691
771
 
@@ -728,9 +808,9 @@ nedbd --dag --data ./data
728
808
  NEDBD_DAG=1 NEDB_TMK=<32-byte-hex> nedbd --data ./data
729
809
 
730
810
  curl http://127.0.0.1:7070/health
731
- # {"ok":true,"version":"3.2.0","service":"nedbd","engine":"dag","startup_ready":true,"encrypted":true}
811
+ # {"ok":true,"version":"7.2.0","service":"nedbd","engine":"dag","startup_ready":true,"encrypted":true}
732
812
 
733
- # Tail the live event stream (new in v2.2.31)
813
+ # Tail the live event stream (since 2.2.31)
734
814
  curl http://127.0.0.1:7070/events
735
815
  # event: scan data: {"objects":730000,"of":1310703,"rate":21043,"eta_s":28}
736
816
  # event: ready data: {"seq":1310703,"head":"b2:9c14e07a…"}
@@ -754,307 +834,159 @@ curl http://127.0.0.1:7070/events
754
834
 
755
835
  **v1 AOF engine is still shipped and unchanged** — `nedbd` (no flag) runs v1.
756
836
 
757
- **Production status:** [vision.interchained.org](https://vision.interchained.org) is live on v2.2.31 **1,310,703 sequences** indexed in the Vision database, AES-256-GCM encrypted at rest, at block height **620,989**.
837
+ **Production status:** [vision.interchained.org](https://vision.interchained.org) is live — verified reachable 15 Sep 2026.
838
+
839
+ The deployment figures below are a **dated snapshot**, not a live readout: **1,310,703 sequences** indexed, AES-256-GCM encrypted at rest, block height **620,989**, measured on engine **v2.2.31**. The engine version a deployment runs is not exposed on its public surface, so treat the version here as the one those numbers were taken on rather than as what is running today.
758
840
 
759
841
  ---
760
842
 
761
- ## What makes NEDB different
843
+ ## Performance every number dated and reproducible
762
844
 
763
- Every database stores *what*. NEDB stores *what*, *when*, *when it was true*, and *why* all sealed in a cryptographic hash chain that proves none of it was tampered with.
845
+ NEDB is benchmarked honestly: the machine is named, the command is in the repo, and a number you cannot reproduce is a number we do not print. Absolute figures are **not** cross-machine comparable — read the ratios.
764
846
 
765
- | Capability | NEDB | SQLite | Redis | MongoDB |
766
- |---|:---:|:---:|:---:|:---:|
767
- | Hash-chained tamper evidence | ✅ | ❌ | ❌ | ❌ |
768
- | Time-travel reads (`AS OF seq`) | ✅ | ❌ | ❌ | ❌ |
769
- | Bi-temporal (`VALID AS OF date`) | ✅ | ❌ | ❌ | ❌ |
770
- | Causal Write Provenance | ✅ | ❌ | ❌ | ❌ |
771
- | Replay-protected idempotent writes | ✅ | ❌ | ❌ | ❌ |
772
- | SQL + Redis + MongoDB adapters | ✅ | — | — | — |
773
- | Concurrent group-commit daemon | ✅ | ❌ | ✅ | ✅ |
774
- | At-rest AES-256-GCM encryption | ✅ | ❌ | ❌ | — |
847
+ **Embedded core (Rust engine, in-process) `bench/RESULTS.md`, 2026-06-14, Linux x86-64, Python 3.9.25.
848
+ Reproduce: `python3 bench/benchmarks.py --save`**
775
849
 
776
- ---
850
+ | Operation | Throughput | Latency (avg) |
851
+ |-----------|-----------|---------------|
852
+ | PUT (replace, no index) | **63.5K/s** | 15.74 µs |
853
+ | GET (point read, HEAD) | **1.33M/s** | 0.75 µs |
854
+ | GET (`AS OF` — time-travel read) | **942.9K/s** | 1.06 µs |
855
+ | QUERY: eq filter, no index (scan) | 514.1K/s | 1.95 µs |
856
+ | QUERY: eq filter, eq index | **1.45M/s** | 0.69 µs |
857
+ | QUERY: ORDER BY + ordered index, LIMIT 20 | 454.7K/s | 2.20 µs |
858
+ | QUERY: SEARCH, inverted index | 492.3K/s | 2.03 µs |
859
+ | PUT durable (AOF + fsync) | 7.3K/s | 137.59 µs |
777
860
 
778
- ## Install
861
+ Time travel is not a tax: an `AS OF` read runs at **~70% of the speed of a current-state read**.
779
862
 
780
- ```bash
781
- pip install nedb-engine # Python 3.8 pure-Python + optional Rust native wheel
782
- npm install nedb-engine # Node ≥ 16 — napi-rs prebuilt binaries
783
- ```
863
+ **The daemon over HTTP (v2 DAG server v2.2.31, Intel iMac — 10k writes / 100k reads / 30k objects, AES-256-GCM on).
864
+ Reproduce: `NEDBD_DAG=1 nedbd --data /tmp/perf &` then `python3 tests/test_dag_perf.py --n 10000 --reads 100000`**
784
865
 
785
- ### Prebuilt platforms
866
+ | Operation | Throughput | p50 | p99 |
867
+ |---|---|---|---|
868
+ | Sequential writes | **418 ops/s** | 2.3 ms | 3.3 ms |
869
+ | Point-lookup reads | **478 ops/s** | 2.0 ms | 3.0 ms |
870
+ | ORDER BY queries | **489 ops/s** | 1.8 ms | 4.3 ms |
871
+ | Batch writes (500 ops/req) | **1,104 ops/s** | 0.9 ms | 1.2 ms |
872
+ | Tamper-verify (30k objects) | ~21,000 BLAKE2b/sec | — | 1.38 s total |
786
873
 
787
- Both registries ship prebuilt binaries for:
874
+ p99 latencies hold because of `TCP_NODELAY` on the axum listener — without it macOS loopback adds the Nagle algorithm's 40–200 ms delay on small writes.
788
875
 
789
- | Platform | libc | Python wheel | Node addon |
790
- |---|---|---|---|
791
- | Linux x86_64 | glibc | ✅ manylinux | ✅ |
792
- | Linux x86_64 | musl (Alpine) | ✅ musllinux | ✅ |
793
- | Linux aarch64 (Graviton, Ampere, Apple-Silicon containers) | glibc | ✅ manylinux | ✅ |
794
- | Linux aarch64 | musl (Alpine) | ✅ musllinux | ✅ |
795
- | macOS arm64 + x86_64 | — | ✅ | ✅ |
796
- | Windows x86_64 | MSVC | ✅ | ✅ |
876
+ **SQL joins inside the evaluator (nested loop vs hash, fixed-seed workload, engine 4.0.0, Linux x86-64) —
877
+ `docs/BENCH-sqlselect.md`. Reproduce: `cargo run --release --example sqlbench`**
797
878
 
798
- On Python, any platform without a prebuilt wheel still installs: pip falls back
799
- to the universal `py3-none-any` wheel and you get the pure-Python v1 AOF engine
800
- (correct, slower, no embedded DAG). On Node there is no such fallback — an
801
- unlisted platform has no addon.
879
+ | workload | nested (ms) | hash (ms) | speedup |
880
+ |---|---:|---:|---:|
881
+ | equality join (1,000 × 500 rows) | 366.62 | 4.05 | **90.6×** |
882
+ | left join | 373.27 | 4.03 | **92.5×** |
883
+ | join + sort | 403.51 | 6.35 | **63.6×** |
884
+ | join + broad pred | 336.34 | 4.68 | **71.9×** |
885
+ | non-equality join | 344.69 | 346.33 | — *(same path, no hash key)* |
802
886
 
803
- ---
887
+ **Indexed range scans (20,000 rows, two identical stores, one indexed) — `scripts/bench_index_range.py`**
804
888
 
805
- ## Python 5-minute tour
889
+ | Query | Scan | Indexed | Speedup |
890
+ | --- | --- | --- | --- |
891
+ | `WHERE fee = 10000` | 137 ms | 0.01 ms | **17,000×** |
892
+ | `WHERE fee IN (a, b, c)` | 185 ms | 0.02 ms | 9,700× |
893
+ | `WHERE fee BETWEEN …` (1% of rows) | 186 ms | 1.1 ms | 170× |
894
+ | `WHERE fee BETWEEN …` (10% of rows) | 188 ms | 13 ms | 14× |
895
+ | unindexed field (control) | 188 ms | 187 ms | 1.0× |
806
896
 
807
- ```python
808
- from nedb import NEDB
897
+ **v1 Python server (baseline — single-threaded AOF):**
809
898
 
810
- db = NEDB("./mydata") # durable: every op is AOF-logged, fsync'd, and hash-chained
811
- # db = NEDB() # or in-memory
899
+ | Operation | Throughput | p99 latency |
900
+ |---|---|---|
901
+ | Sequential PUT | ~23/s | 44 ms |
902
+ | Concurrent PUT (16 workers) | ~92/s | 48 ms |
903
+ | Batch PUT (500 ops/request) | ~520 ops/s | 1.9 ms/op |
904
+ | Point-lookup read (NQL) | ~23/s | 44 ms |
905
+ | Rust napi PUT (FFI) | ~70K/s | — |
906
+ | Rust napi GET (FFI) | ~330K/s | — |
812
907
 
813
- db.create_index("users", "status", "eq")
814
- db.create_index("users", "bio", "search")
908
+ And the number from the v3 section that justifies the whole storage line: a real itcd chainstate flush of 2,549 coins went from **minutes** on the v2 loose store to **1.71 s** on `--dag-v3` — measured on the real chain, not a fixture.
815
909
 
816
- db.put("users", "alice", {"name": "Alice", "age": 31, "status": "active", "bio": "rust hacker"})
817
- db.put("users", "bob", {"name": "Bob", "age": 24, "status": "active", "bio": "python dev"})
910
+ ---
818
911
 
819
- # NQL: WHERE + ORDER BY + LIMIT + SEARCH + TRAVERSE + GROUP BY
820
- db.query('FROM users WHERE status = "active" ORDER BY age ASC')
821
- db.query('FROM users SEARCH "rust"')
822
- db.query('FROM users GROUP BY status COUNT')
912
+ ## nedbd the concurrent server daemon
823
913
 
824
- # Full boolean predicatesIN, BETWEEN, LIKE, IS NULL, OR, NOT, parentheses
825
- db.query('FROM users WHERE status IN ("active", "trialing")')
826
- db.query('FROM users WHERE age BETWEEN 25 AND 40')
827
- db.query('FROM users WHERE bio LIKE "%rust%" AND NOT (status = "retired")')
828
- db.query('FROM users WHERE (age < 25 OR age > 60) AND bio IS NOT NULL')
914
+ nedbd runs NEDB as a long-lived process with an HTTP/JSON API and an optional RESP2 wire protocol. Built on a **single-writer group-commit sequencer** parallel reads, batched durable writes, one hash-chain per database, zero write-write races.
829
915
 
830
- # Time-travel — AS OF any past sequence
831
- snap = db.seq
832
- db.put("users", "alice", {"name": "Alice", "age": 32, "status": "retired"})
833
- db.get("users", "alice", as_of=snap) # age 31, status active
916
+ ```bash
917
+ nedbd # :7070, data ./nedb-data (v1 AOF engine)
918
+ nedbd --dag --data ./data # v2 DAG engine (or NEDBD_DAG=1)
919
+ NEDBD_RESP2_PORT=6380 nedbd # also speak RESP2 (redis-cli compatible)
920
+ nedbd --log-level 2 # 0=errors 1=requests 2=deploy 3=verbose
834
921
 
835
- # Bi-temporalVALID AS OF any past date
836
- db.put("policy", "rate_2024", {"pct": 5.0}, valid_from="2024-01-01", valid_to="2024-12-31")
837
- db.put("policy", "rate_2025", {"pct": 6.0}, valid_from="2025-01-01")
838
- db.query('FROM policy VALID AS OF "2024-06-15"') # → rate 5.0
922
+ # Live event stream (since 2.2.31) SSE: scan progress, ready, per-write head
923
+ curl http://127.0.0.1:7070/events
924
+ ```
839
925
 
840
- # Causal Write Provenance — why did this write happen?
841
- db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
842
- seq_msg = db.seq
843
- db.put("beliefs", "dark_mode", {"value": True},
844
- caused_by=[seq_msg], evidence="user_message", confidence=0.95)
845
- db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by') # → msg_1
846
- db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode
926
+ ### Companion CLIs
847
927
 
848
- # Relations + graph traversal
849
- db.link("users:alice", "follows", "users:bob")
850
- db.query('FROM users WHERE _id = "alice" TRAVERSE follows')
928
+ Alongside the daemon, `cargo install nedb-engine` ships **`nedb-cli`** — operate on a store directory offline (`head`/`status`/`verify`/`get`/`scan`/`flush`/`repair`/`export`) — and **`nedb-inspector`**, a deterministic checker that warns when a durable open lacks flush-on-exit wiring. Full reference: [**docs/CLI.md**](docs/CLI.md).
851
929
 
852
- # Hash-chain integrity
853
- assert db.verify() # cryptographic proof — no tampering
930
+ ### Startup modes
854
931
 
855
- # SQL, Redis, MongoDB compatibility adapters
856
- from nedb import sql_exec, RedisCompat, MongoClient
857
- sql_exec(db, "SELECT * FROM users WHERE status = 'active' ORDER BY age DESC")
858
- r = RedisCompat(db); r.execute("HSET", "user:1", "name", "Alice")
859
- MongoClient(db)["users"].find({"status": "active"}).sort("age", -1).to_list()
860
- ```
932
+ - **Warm start** — every restart after the first open reads the `MANIFEST` file and restores `seq` + Merkle `head` in **O(1)**. No scan, no replay, independent of dataset size. Boots in milliseconds.
933
+ - **Cold start** — first open of an existing dataset spawns the integrity scan in a background thread *and accepts connections immediately*. Reads serve instantly from the content-addressed DAG; writes return `HTTP 503 startup in progress` until the `startup_ready` gate flips. Progress (objects, rate, ETA) streams over `GET /events`.
861
934
 
862
- ---
935
+ ### Environment variables
863
936
 
864
- ## Official Python client talk to nedbd over HTTP
937
+ | Variable | Default | Description |
938
+ |---|---|---|
939
+ | `NEDBD_DAG` | `0` | Set `1` to launch the v2 DAG engine (`nedbd-v2`). Same as `--dag`. |
940
+ | `NEDBD_HOST` | `127.0.0.1` | Bind address. **v2.2.31** defaults to loopback (was `0.0.0.0`) — security hardening fix. Set explicitly to `0.0.0.0` to expose. |
941
+ | `NEDBD_PORT` | `7070` | HTTP bind port. |
942
+ | `NEDBD_TOKEN` | unset | Optional bearer token; required on every `/v1/*` request when set. |
943
+ | `NEDB_TMK` | unset | 32-byte hex AES-256-GCM at-rest encryption key. |
944
+ | `NEDBD_DATA` | `./nedb-data` | Root directory. v2 creates `dag/`, IdIndex sharded across **256 subdirectories**, and a small `MANIFEST` file. |
945
+ | `NEDBD_CAST` | `0` | Set `1` to enable the `/cast` natural-language planner. Same as `--cast`. Requires a build with `--features cast`. See [**Cast**](#cast--natural-language-into-nql). |
946
+ | `NEDBD_CAST_MODEL` | unset | Explicit path to a `model.cast` container. Otherwise searched in the data dir, `$CAST_HOME`, and `~/.cache/nedb-cast-slm/`. |
865
947
 
866
- Running the daemon? `nedb.client.NedbClient` is the official client for its
867
- HTTP API extracted from the battle-tested clients that ran a production
868
- Redis→NEDB mainnet migration, speaking the full route surface: queries,
869
- atomic CAS transactions, TTL, indexes, relations, Merkle proofs, and the
870
- Mongo-compat endpoint. Env-var defaults (`NEDBD_URL`, `NEDBD_TOKEN`,
871
- `NEDB_DB`) mirror the daemon's own.
948
+ ```bash
949
+ # Create a database with seed data and relations
950
+ curl -X POST :7070/v1/databases -d '{
951
+ "name": "shop",
952
+ "init": {
953
+ "indexes": [["users","status","eq"]],
954
+ "seed": {"users": [{"_id":"u1","name":"Alice","status":"active"}]},
955
+ "links": [["users:u1","buys","orders:o1"]]
956
+ }}'
872
957
 
873
- ```python
874
- from nedb import NedbClient, PreconditionFailed, op_put
958
+ # Query — the endpoint speaks neSQL: SQL *or* NQL, routed on the first keyword
959
+ curl -X POST :7070/v1/databases/shop/query \
960
+ -d '{"nql":"SELECT name FROM users WHERE status = '"'"'active'"'"' ORDER BY name"}'
961
+ # → {"rows":[{"name":"Alice"}],"count":1,"dialect":"sql", ...}
875
962
 
876
- c = NedbClient("http://127.0.0.1:7070", db="app", token="s3cret")
877
- c.ensure_database()
963
+ curl -X POST :7070/v1/databases/shop/query \
964
+ -d '{"nql":"FROM users WHERE status = \"active\" ORDER BY name ASC"}'
965
+ # → {"rows":[...],"count":1,"dialect":"nql", ...}
878
966
 
879
- c.put("users", "u1", {"id": "u1", "email": "a@b.c"}, idem="signup-u1")
880
- c.query('FROM users WHERE email = "a@b.c"') # full NQL rides through
881
- c.query("FROM users AS OF 41") # time-travel included
882
967
 
883
- # Atomic all-or-nothing transaction with engine-checked preconditions
884
- # the primitive that replaces Redis Lua scripts (if_seq: N = CAS, -1 = create-once)
885
- doc = c.get_doc("users", "u1") # docs carry _seq
886
- c.tx([op_put("users", "u1", {**doc, "plan": "pro"}, if_seq=doc["_seq"])])
968
+ **The field is still called `nql`, and its contents no longer have to be.** This
969
+ endpoint accepts **neSQL** NQL *or* PostgreSQL SQL and answers with the
970
+ `dialect` it chose. The name is unchanged because every existing HTTP client
971
+ sends it; renaming would break them to gain nothing. Old NQL clients are
972
+ unaffected.
887
973
 
888
- # Contested writes: retry ONLY on PreconditionFailed, capped backoff
889
- def bump():
890
- d = c.get_doc("counters", "hits") or {"n": 0}
891
- return c.tx([op_put("counters", "hits", {"n": d.get("n", 0) + 1},
892
- if_seq=d.get("_seq", -1))])
893
- c.cas_retry(bump)
974
+ Routing is **structural, not guessed**. NQL statements begin `FROM`; PostgreSQL
975
+ has no statement form that begins with `FROM`, so the leading keyword partitions
976
+ the two vocabularies rather than hinting at them. A first word in neither is
977
+ refused *naming both* never handed to whichever parser seems likelier.
894
978
 
895
- # Integrity, verifiable WITHOUT trusting the server
896
- proof = c.proof(c.log(limit=1)[0]["hash"])
897
- from nedb import verify_proof; verify_proof(proof) # -> True, locally
979
+ ```bash
980
+ curl -X POST :7070/v1/databases/shop/query -d '{"nql":"GRANT ALL ON users"}'
981
+ # 400 "GRANT" does not begin a statement in either half of neSQL
982
+ # NQL statements begin with: FROM
983
+ # SQL statements begin with: SELECT, INSERT, UPDATE, ...
898
984
  ```
899
985
 
900
- A CAS miss raises the **same `PreconditionFailed`** (with the same
901
- `.failures` shape) the embedded engine raises code written against
902
- `NEDB.tx` ports to the HTTP client without changing its except-clauses.
903
- Typed errors throughout: `NedbAuthError`, `NedbNotFound`, `NedbBadRequest`,
904
- `NedbConflict`, `CasExhausted`.
905
-
906
- ---
907
-
908
- ## Redis layer-2 — wrap_redis()
909
-
910
- Already running on Redis? Wrap your connection in one line and gain NEDB features *alongside* your existing Redis app — no migration required.
911
-
912
- ```python
913
- import redis, json
914
- from nedb import wrap_redis
915
-
916
- r = wrap_redis(redis.Redis("localhost", 6379), db_name="rideshare")
917
-
918
- # Step 1 — register: map Redis key globs to NEDB collections (chainable)
919
- (r.nedb
920
- .register("driver:*", collection="driver", value_parser=json.loads)
921
- .register("trip:*", collection="trip", value_type="hash")
922
- )
923
-
924
- # Step 2 — backfill: import all existing Redis data into NEDB in one pass
925
- imported = r.nedb.backfill() # → int (keys imported)
926
-
927
- # Step 3 — shadow: all future r.set/hset/... auto-chain into NEDB
928
- r.nedb.shadow_writes = True
929
-
930
- # ─── Alice's app keeps running — zero changes ───────────────────────────
931
- r.set("driver:d1", json.dumps({"name": "Bob", "status": "active"})) # ← shadowed
932
- r.hset("trip:t1", mapping={"status": "en_route", "driver_id": "d1"}) # ← shadowed
933
-
934
- # ─── New features available on the same connection ──────────────────────
935
- r.nedb.query('FROM driver WHERE status = "active" ORDER BY lat ASC')
936
- r.nedb.verify() # → True (every write chain-verified)
937
- r.nedb.head() # → 64-char BLAKE2b commitment hash
938
- ```
939
-
940
- **Isolation guarantee:** NEDB never writes to Alice's namespace. It owns only:
941
-
942
- | Key | Type | Purpose |
943
- |-----|------|---------|
944
- | `nedb:{db_name}:oplog` | Redis Stream | append-only op log |
945
- | `nedb:{db_name}:snapshot` | Redis Hash | checkpoint |
946
- | `nedb:{db_name}:meta` | Redis Hash | index config |
947
-
948
- See [`examples/fakeredis_demo.py`](examples/fakeredis_demo.py) for a full local demo (no Redis server needed).
949
-
950
- ---
951
-
952
- ## Node.js
953
-
954
- ```javascript
955
- import { NedbCore } from "nedb-engine";
956
-
957
- const db = new NedbCore(); // in-memory
958
- // const db = NedbCore.open("./data"); // durable
959
-
960
- db.createIndex("users", "status", "eq");
961
- db.put("users", "alice", JSON.stringify({ name: "Alice", age: 31, status: "active" }));
962
-
963
- // Time-travel
964
- const snap = db.seq(); // BigInt
965
- db.put("users", "alice", JSON.stringify({ name: "Alice", age: 32, status: "retired" }));
966
- JSON.parse(db.getAsOf("users", "alice", snap)).age; // → 31
967
-
968
- // Full NQL
969
- const rows = db.query('FROM users WHERE status = "active" ORDER BY age ASC');
970
- rows.map(r => JSON.parse(r));
971
-
972
- // Tamper evidence
973
- db.verify(); // → true
974
- db.head(); // → 64-char BLAKE2b commitment hash
975
- db.seq(); // → BigInt
976
- ```
977
-
978
- ---
979
-
980
- ## nedbd — the concurrent server daemon
981
-
982
- nedbd runs NEDB as a long-lived process with an HTTP/JSON API and an optional RESP2 wire protocol. Built on a **single-writer group-commit sequencer** — parallel reads, batched durable writes, one hash-chain per database, zero write-write races.
983
-
984
- ```bash
985
- nedbd # :7070, data ./nedb-data (v1 AOF engine)
986
- nedbd --dag --data ./data # v2 DAG engine (or NEDBD_DAG=1)
987
- NEDBD_RESP2_PORT=6380 nedbd # also speak RESP2 (redis-cli compatible)
988
- nedbd --log-level 2 # 0=errors 1=requests 2=deploy 3=verbose
989
-
990
- # Live event stream (new in v2.2.31) — SSE: scan progress, ready, per-write head
991
- curl http://127.0.0.1:7070/events
992
- ```
993
-
994
- ### Companion CLIs
995
-
996
- Alongside the daemon, `cargo install nedb-engine` ships **`nedb-cli`** — operate on a store directory offline (`head`/`status`/`verify`/`get`/`scan`/`flush`/`repair`/`export`) — and **`nedb-inspector`**, a deterministic checker that warns when a durable open lacks flush-on-exit wiring. Full reference: [**docs/CLI.md**](docs/CLI.md).
997
-
998
- ### Startup modes (v2.2.31)
999
-
1000
- - **Warm start** — every restart after the first open reads the `MANIFEST` file and restores `seq` + Merkle `head` in **O(1)**. No scan, no replay, independent of dataset size. Boots in milliseconds.
1001
- - **Cold start** — first open of an existing dataset spawns the integrity scan in a background thread *and accepts connections immediately*. Reads serve instantly from the content-addressed DAG; writes return `HTTP 503 startup in progress` until the `startup_ready` gate flips. Progress (objects, rate, ETA) streams over `GET /events`.
1002
-
1003
- ### Environment variables
1004
-
1005
- | Variable | Default | Description |
1006
- |---|---|---|
1007
- | `NEDBD_DAG` | `0` | Set `1` to launch the v2 DAG engine (`nedbd-v2`). Same as `--dag`. |
1008
- | `NEDBD_HOST` | `127.0.0.1` | Bind address. **v2.2.31** defaults to loopback (was `0.0.0.0`) — security hardening fix. Set explicitly to `0.0.0.0` to expose. |
1009
- | `NEDBD_PORT` | `7070` | HTTP bind port. |
1010
- | `NEDBD_TOKEN` | unset | Optional bearer token; required on every `/v1/*` request when set. |
1011
- | `NEDB_TMK` | unset | 32-byte hex AES-256-GCM at-rest encryption key. |
1012
- | `NEDBD_DATA` | `./nedb-data` | Root directory. v2 creates `dag/`, IdIndex sharded across **256 subdirectories**, and a small `MANIFEST` file. |
1013
- | `NEDBD_CAST` | `0` | Set `1` to enable the `/cast` natural-language planner. Same as `--cast`. Requires a build with `--features cast`. See [**Cast**](#cast--natural-language-into-nql). |
1014
- | `NEDBD_CAST_MODEL` | unset | Explicit path to a `model.cast` container. Otherwise searched in the data dir, `$CAST_HOME`, and `~/.cache/nedb-cast-slm/`. |
1015
-
1016
- ```bash
1017
- # Create a database with seed data and relations
1018
- curl -X POST :7070/v1/databases -d '{
1019
- "name": "shop",
1020
- "init": {
1021
- "indexes": [["users","status","eq"]],
1022
- "seed": {"users": [{"_id":"u1","name":"Alice","status":"active"}]},
1023
- "links": [["users:u1","buys","orders:o1"]]
1024
- }}'
1025
-
1026
- # Query — the endpoint speaks neQL: SQL *or* NQL, routed on the first keyword
1027
- curl -X POST :7070/v1/databases/shop/query \
1028
- -d '{"nql":"SELECT name FROM users WHERE status = '"'"'active'"'"' ORDER BY name"}'
1029
- # → {"rows":[{"name":"Alice"}],"count":1,"dialect":"sql", ...}
1030
-
1031
- curl -X POST :7070/v1/databases/shop/query \
1032
- -d '{"nql":"FROM users WHERE status = \"active\" ORDER BY name ASC"}'
1033
- # → {"rows":[...],"count":1,"dialect":"nql", ...}
1034
-
1035
-
1036
- **The field is still called `nql`, and its contents no longer have to be.** This
1037
- endpoint accepts **neQL** — NQL *or* PostgreSQL SQL — and answers with the
1038
- `dialect` it chose. The name is unchanged because every existing HTTP client
1039
- sends it; renaming would break them to gain nothing. Old NQL clients are
1040
- unaffected.
1041
-
1042
- Routing is **structural, not guessed**. NQL statements begin `FROM`; PostgreSQL
1043
- has no statement form that begins with `FROM`, so the leading keyword partitions
1044
- the two vocabularies rather than hinting at them. A first word in neither is
1045
- refused *naming both* — never handed to whichever parser seems likelier.
1046
-
1047
- ```bash
1048
- curl -X POST :7070/v1/databases/shop/query -d '{"nql":"GRANT ALL ON users"}'
1049
- # → 400 "GRANT" does not begin a statement in either half of neQL
1050
- # NQL statements begin with: FROM
1051
- # SQL statements begin with: SELECT, INSERT, UPDATE, ...
1052
- ```
1053
-
1054
- It is the **same router** `nesql query` uses — `nedb_engine::neql::route`, which
1055
- the CLI re-exports rather than copies. Two implementations of that decision
1056
- would let the daemon and the CLI disagree about what a statement *means*, which
1057
- is worse than disagreeing about a result: nothing looks broken when it happens.
986
+ It is the **same router** `nesql query` uses — `nedb_engine::neql::route`, which
987
+ the CLI re-exports rather than copies. Two implementations of that decision
988
+ would let the daemon and the CLI disagree about what a statement *means*, which
989
+ is worse than disagreeing about a result: nothing looks broken when it happens.
1058
990
 
1059
991
  # Verify the hash chain
1060
992
  curl :7070/v1/databases/shop/verify
@@ -1153,14 +1085,13 @@ db.query("FROM blocks WHERE height BETWEEN 600000 AND 600100")
1153
1085
  ```
1154
1086
 
1155
1087
  Measured on 20,000 rows with `scripts/bench_index_range.py` — two identical
1156
- databases, one indexed, one not:
1088
+ databases, one indexed, one not (the full table lives in
1089
+ [**Performance**](#performance--every-number-dated-and-reproducible)):
1157
1090
 
1158
1091
  | Query | Scan | Indexed | Speedup |
1159
1092
  | --- | --- | --- | --- |
1160
1093
  | `WHERE fee = 10000` | 137 ms | 0.01 ms | 17,000× |
1161
- | `WHERE fee IN (a, b, c)` | 185 ms | 0.02 ms | 9,700× |
1162
1094
  | `WHERE fee BETWEEN …` (1% of rows) | 186 ms | 1.1 ms | 170× |
1163
- | `WHERE fee BETWEEN …` (10% of rows) | 188 ms | 13 ms | 14× |
1164
1095
  | unindexed field (control) | 188 ms | 187 ms | 1.0× |
1165
1096
 
1166
1097
  The planner asks the index how many rows each candidate range covers and takes
@@ -1481,108 +1412,237 @@ Two habits that avoid most misses:
1481
1412
 
1482
1413
  ---
1483
1414
 
1484
- ## Performance
1415
+ ## Architecture
1485
1416
 
1486
- **v2 DAG Rust server (v2.2.31, Intel iMac — 10k writes / 100k reads / 30k objects, AES-256-GCM on):**
1417
+ ```
1418
+ ┌──────────────────────────────────────────────────────────┐
1419
+ put/del → │ OpLog (BLAKE2b hash chain · per-client nonce · │ ← single source of truth
1420
+ link │ idempotency keys · causal provenance fields) │
1421
+ └───────────────┬──────────────────────────────────────────┘
1422
+ deterministic fold │ (state = pure function of the log)
1423
+ ┌──────────────┬──────────┴──────┬───────────────┬────────────────┐
1424
+ ▼ ▼ ▼ ▼ ▼
1425
+ MVCC store Relations Indexes CauseMap BlobStore
1426
+ (time-travel) (graph+AS OF) eq/ord/search (reverse index) (Cascade CDC)
1487
1427
 
1488
- | Operation | Throughput | p50 | p99 |
1489
- |---|---|---|---|
1490
- | Sequential writes | **418 ops/s** | 2.3 ms | 3.3 ms |
1491
- | Point-lookup reads | **478 ops/s** | 2.0 ms | 3.0 ms |
1492
- | ORDER BY queries | **489 ops/s** | 1.8 ms | 4.3 ms |
1493
- | Batch writes (500 ops/req) | **1,104 ops/s** | 0.9 ms | 1.2 ms |
1494
- | Tamper-verify (30k objects) | ~21,000 BLAKE2b/sec | — | 1.38 s total |
1428
+ ┌─────────────────────────────────┐
1429
+ Thread-safe → │ Sequencer (group-commit) │ ← single writer, parallel readers
1430
+ │ — one committer thread/db │
1431
+ │ — batch fsync │
1432
+ └─────────────────────────────────┘
1495
1433
 
1496
- p99 latencies hold because of `TCP_NODELAY` on the axum listener — without it macOS loopback adds the Nagle algorithm's 40–200 ms delay on small writes.
1434
+ Compatibility adapters: SQL · Redis · MongoDB
1435
+ Wire protocols: HTTP/JSON · RESP2 · PostgreSQL wire
1436
+ Encryption: AES-256-GCM at-rest (TMK/DEK double-envelope)
1437
+ ```
1497
1438
 
1498
- **v1 Python server (baseline — single-threaded AOF):**
1439
+ The provenance machinery is load-bearing infrastructure, not metadata stapled on: `CauseMap` is a
1440
+ reverse index over causal edges (that is why `TRACE … REVERSE` is as fast as forward), and the
1441
+ `Relations` layer backs both `TRAVERSE` and the `AS OF` joins.
1499
1442
 
1500
- | Operation | Throughput | p99 latency |
1501
- |---|---|---|
1502
- | Sequential PUT | ~23/s | 44 ms |
1503
- | Concurrent PUT (16 workers) | ~92/s | 48 ms |
1504
- | Batch PUT (500 ops/request) | ~520 ops/s | 1.9 ms/op |
1505
- | Point-lookup read (NQL) | ~23/s | 44 ms |
1506
- | Rust napi PUT (FFI) | ~70K/s | — |
1507
- | Rust napi GET (FFI) | ~330K/s | — |
1443
+ ---
1508
1444
 
1509
- Reproduce with the included benchmark:
1445
+ ## Install
1510
1446
 
1511
1447
  ```bash
1512
- NEDBD_DAG=1 nedbd --data /tmp/perf &
1513
- python3 tests/test_dag_perf.py --n 10000 --reads 100000
1448
+ pip install nedb-engine # Python ≥ 3.8 — pure-Python + optional Rust native wheel
1449
+ npm install nedb-engine # Node 16 — napi-rs prebuilt binaries
1514
1450
  ```
1515
1451
 
1452
+ ### Prebuilt platforms
1453
+
1454
+ Both registries ship prebuilt binaries for:
1455
+
1456
+ | Platform | libc | Python wheel | Node addon |
1457
+ |---|---|---|---|
1458
+ | Linux x86_64 | glibc | ✅ manylinux | ✅ |
1459
+ | Linux x86_64 | musl (Alpine) | ✅ musllinux | ✅ |
1460
+ | Linux aarch64 (Graviton, Ampere, Apple-Silicon containers) | glibc | ✅ manylinux | ✅ |
1461
+ | Linux aarch64 | musl (Alpine) | ✅ musllinux | ✅ |
1462
+ | macOS arm64 + x86_64 | — | ✅ | ✅ |
1463
+ | Windows x86_64 | MSVC | ✅ | ✅ |
1464
+
1465
+ On Python, any platform without a prebuilt wheel still installs: pip falls back
1466
+ to the universal `py3-none-any` wheel and you get the pure-Python v1 AOF engine
1467
+ (correct, slower, no embedded DAG). On Node there is no such fallback — an
1468
+ unlisted platform has no addon.
1469
+
1516
1470
  ---
1517
1471
 
1518
- ## NEDB v3 Segment / Pack Object Store
1472
+ ## Python5-minute tour
1519
1473
 
1520
- **v3 is an opt-in storage substrate that replaces the loose one-file-per-object layout with append-only *segment packs* — the difference between a chainstate flush that takes *minutes* and one that takes *under two seconds*.** It is **off by default** (byte-for-byte v2), enabled with one flag, and **transparent** to everything above the storage layer: NQL, `AS OF`, `VALID AS OF`, `TRACE`, the BLAKE2b Merkle head, and causal provenance all behave identically.
1474
+ Every example in this tour was executed against the released **8.0.0** package.
1521
1475
 
1522
- ### Why it exists
1476
+ ```python
1477
+ from nedb import NEDB
1523
1478
 
1524
- v2 stores every document version as its own content-addressed file at `objects/{hash[:2]}/{hash[2:]}`. That makes writes trivially atomic (write `.tmp` → `rename`) and corruption-proof — but each write costs a file create + `fsync` + rename **plus** a directory B-tree update. At scale that filesystem-metadata churn dominates: on a busy disk it caps sustained writes around **~185/s**, and a batch flush of a few thousand objects degrades into minutes. The bottleneck is the *number of files touched*, not the bytes written.
1479
+ db = NEDB("./mydata") # durable: every op is AOF-logged, fsync'd, and hash-chained
1480
+ # db = NEDB() # or in-memory
1525
1481
 
1526
- ### What it does
1482
+ db.create_index("users", "status", "eq")
1483
+ db.create_index("users", "bio", "search")
1527
1484
 
1528
- v3 batches objects into append-only **segment packs** — `objects/segments/seg-NNNNNN.dat` — where each record is `[content_len: u32-LE][content]`. A write appends to the active segment and updates an in-memory `hash → (segment_id, offset, len)` map; a batch commits with a **single `fsync`**. Thousands of per-file syscalls collapse into one sequential append plus one durability point, so **flush cost scales with bytes (sequential I/O), not object-count × syscall overhead.**
1485
+ db.put("users", "alice", {"name": "Alice", "age": 31, "status": "active", "bio": "rust hacker"})
1486
+ db.put("users", "bob", {"name": "Bob", "age": 24, "status": "active", "bio": "python dev"})
1529
1487
 
1530
- - **Compaction / pruning** `compact()` keeps the *live set* (the current version of every document, resolved from the id-index), rewrites those records into fresh segments, and reclaims the superseded/dead versions.
1531
- - **`.idx` sidecars** — each segment carries a sidecar (`NIX1` magic + entry count + fixed 44-byte entries + a BLAKE2b-256 checksum) so reopen rebuilds the in-memory index by reading the sidecar instead of scanning the whole pack. A missing or corrupt sidecar falls back to a full scan-and-heal — slower, never fatal.
1532
- - **Dual-read migration** — opening an existing v2 store in v3 mode is **non-destructive**: old loose objects stay fully readable, and only *new* writes go to segments. No migration step, no downtime, no rewrite.
1533
- - **Durable flush-on-close** `flush_all()` (and `Db`'s `Drop`) fsync the active segment, matching the flush-on-close contract of sled / RocksDB.
1488
+ # NQL: WHERE + ORDER BY + LIMIT + SEARCH + TRAVERSE + GROUP BY
1489
+ db.query('FROM users WHERE status = "active" ORDER BY age ASC')
1490
+ db.query('FROM users SEARCH "rust"')
1491
+ db.query('FROM users GROUP BY status COUNT')
1534
1492
 
1535
- ### How to enable
1493
+ # Full boolean predicates — IN, BETWEEN, LIKE, IS NULL, OR, NOT, parentheses
1494
+ db.query('FROM users WHERE status IN ("active", "trialing")')
1495
+ db.query('FROM users WHERE age BETWEEN 25 AND 40')
1496
+ db.query('FROM users WHERE bio LIKE "%rust%" AND NOT (status = "retired")')
1497
+ db.query('FROM users WHERE (age < 25 OR age > 60) AND bio IS NOT NULL')
1536
1498
 
1537
- ```bash
1538
- # Engine / nedbd-v2 (the native daemon from npm / the native wheel)
1539
- nedbd-v2 --dag-v3 --data /var/lib/nedb # real flag as of v2.4.3 — or set NEDB_DAG_V3=1
1499
+ # Time-travel — AS OF any past sequence
1500
+ snap = db.seq
1501
+ db.put("users", "alice", {"name": "Alice", "age": 32, "status": "retired"})
1502
+ db.get("users", "alice", as_of=snap) # → age 31, status active
1540
1503
 
1541
- # itcdBitcoin-fork node embedding NEDB via nedb-ffi
1542
- interchainedd -dagv3 # puts chainstate AND block index on segments
1504
+ # Bi-temporalVALID AS OF any past date
1505
+ db.put("policy", "rate_2024", {"pct": 5.0}, valid_from="2024-01-01", valid_to="2024-12-31")
1506
+ db.put("policy", "rate_2025", {"pct": 6.0}, valid_from="2025-01-01")
1507
+ db.query('FROM policy VALID AS OF "2024-06-15"') # → rate 5.0
1508
+
1509
+ # Causal Write Provenance — why did this write happen?
1510
+ db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
1511
+ seq_msg = db.seq
1512
+ db.put("beliefs", "dark_mode", {"value": True},
1513
+ caused_by=[seq_msg], evidence="user_message", confidence=0.95)
1514
+ db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by') # → msg_1
1515
+ db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode
1516
+
1517
+ # Relations + graph traversal
1518
+ db.link("users:alice", "follows", "users:bob")
1519
+ db.query('FROM users WHERE _id = "alice" TRAVERSE follows')
1520
+
1521
+ # Hash-chain integrity — verify() is a call, head is the commitment
1522
+ assert db.verify() # cryptographic proof — no tampering
1523
+ db.head # → 64-char BLAKE2b Merkle head (a property, not a call)
1524
+
1525
+ # SQL, Redis, MongoDB compatibility adapters
1526
+ from nedb import sql_exec, RedisCompat, MongoClient
1527
+ sql_exec(db, "SELECT * FROM users WHERE status = 'active' ORDER BY age DESC")
1528
+ r = RedisCompat(db); r.execute("HSET", "user:1", "name", "Alice")
1529
+ MongoClient(db)["users"].find({"status": "active"}).sort("age", -1).to_list()
1543
1530
  ```
1544
1531
 
1545
- The switch is read once, when each database's object store is constructed at open time. Default off → v2 loose objects.
1532
+ ---
1546
1533
 
1547
- ### Real-world result
1534
+ ## Official Python client — talk to nedbd over HTTP
1548
1535
 
1549
- itcd (a Bitcoin Core 0.21 fork that replaces LevelDB chainstate with NEDB) syncing on `-dagv3`, measured `FlushStateToDisk` on real chainstate:
1536
+ Running the daemon? `nedb.client.NedbClient` is the official client for its
1537
+ HTTP API — extracted from the battle-tested clients that ran a production
1538
+ Redis→NEDB mainnet migration, speaking the full route surface: queries,
1539
+ atomic CAS transactions, TTL, indexes, relations, Merkle proofs, and the
1540
+ Mongo-compat endpoint. Env-var defaults (`NEDBD_URL`, `NEDBD_TOKEN`,
1541
+ `NEDB_DB`) mirror the daemon's own.
1550
1542
 
1551
- | Flush (coins → disk) | v3 segment store | v2 loose store |
1552
- |---|---|---|
1553
- | 2,002 coins / 275 kB | **1.93 s** | *minutes* |
1554
- | 2,549 coins / 366 kB | **1.71 s** | *minutes* |
1543
+ ```python
1544
+ from nedb import NedbClient, PreconditionFailed, op_put
1555
1545
 
1556
- Note the *larger* batch finishing *faster* — v3's cost is dominated by the single per-batch `fsync`, not per-coin work, so effective throughput (~1,000–1,500 coins/s here) climbs as batches grow, against the loose store's ~185 writes/s metadata ceiling. The gap only widens as the UTXO set grows: sequential-append cost tracks data volume, while per-file cost compounds with object count.
1546
+ c = NedbClient("http://127.0.0.1:7070", db="app", token="s3cret")
1547
+ c.ensure_database()
1557
1548
 
1558
- ### When to use it
1549
+ c.put("users", "u1", {"id": "u1", "email": "a@b.c"}, idem="signup-u1")
1550
+ c.query('FROM users WHERE email = "a@b.c"') # full NQL rides through
1551
+ c.query("FROM users AS OF 41") # time-travel included
1559
1552
 
1560
- Reach for v3 on high-write, large-object-count workloads blockchain chainstate / block index, event sourcing, high-frequency agent memory. For small or read-mostly stores the loose layout is perfectly fine, which is exactly why v3 stays opt-in.
1553
+ # Atomic all-or-nothing transaction with engine-checked preconditions
1554
+ # the primitive that replaces Redis Lua scripts (if_seq: N = CAS, -1 = create-once)
1555
+ doc = c.get_doc("users", "u1") # docs carry _seq
1556
+ c.tx([op_put("users", "u1", {**doc, "plan": "pro"}, if_seq=doc["_seq"])])
1557
+
1558
+ # Contested writes: retry ONLY on PreconditionFailed, capped backoff
1559
+ def bump():
1560
+ d = c.get_doc("counters", "hits") or {"n": 0}
1561
+ return c.tx([op_put("counters", "hits", {"n": d.get("n", 0) + 1},
1562
+ if_seq=d.get("_seq", -1))])
1563
+ c.cas_retry(bump)
1564
+
1565
+ # Integrity, verifiable WITHOUT trusting the server
1566
+ proof = c.proof(c.log(limit=1)[0]["hash"])
1567
+ from nedb import verify_proof; verify_proof(proof) # -> True, locally
1568
+ ```
1569
+
1570
+ A CAS miss raises the **same `PreconditionFailed`** (with the same
1571
+ `.failures` shape) the embedded engine raises — code written against
1572
+ `NEDB.tx` ports to the HTTP client without changing its except-clauses.
1573
+ Typed errors throughout: `NedbAuthError`, `NedbNotFound`, `NedbBadRequest`,
1574
+ `NedbConflict`, `CasExhausted`.
1561
1575
 
1562
1576
  ---
1563
1577
 
1564
- ## Architecture
1578
+ ## Redis layer-2 — wrap_redis()
1579
+
1580
+ Already running on Redis? Wrap your connection in one line and gain NEDB features *alongside* your existing Redis app — no migration required.
1565
1581
 
1582
+ ```python
1583
+ import redis, json
1584
+ from nedb import wrap_redis
1585
+
1586
+ r = wrap_redis(redis.Redis("localhost", 6379), db_name="rideshare")
1587
+
1588
+ # Step 1 — register: map Redis key globs to NEDB collections (chainable)
1589
+ (r.nedb
1590
+ .register("driver:*", collection="driver", value_parser=json.loads)
1591
+ .register("trip:*", collection="trip", value_type="hash")
1592
+ )
1593
+
1594
+ # Step 2 — backfill: import all existing Redis data into NEDB in one pass
1595
+ imported = r.nedb.backfill() # → int (keys imported)
1596
+
1597
+ # Step 3 — shadow: all future r.set/hset/... auto-chain into NEDB
1598
+ r.nedb.shadow_writes = True
1599
+
1600
+ # ─── Alice's app keeps running — zero changes ───────────────────────────
1601
+ r.set("driver:d1", json.dumps({"name": "Bob", "status": "active"})) # ← shadowed
1602
+ r.hset("trip:t1", mapping={"status": "en_route", "driver_id": "d1"}) # ← shadowed
1603
+
1604
+ # ─── New features available on the same connection ──────────────────────
1605
+ r.nedb.query('FROM driver WHERE status = "active" ORDER BY lat ASC')
1606
+ r.nedb.verify() # → True (every write chain-verified)
1607
+ r.nedb.head() # → 64-char BLAKE2b commitment hash
1566
1608
  ```
1567
- ┌──────────────────────────────────────────────────────────┐
1568
- put/del → │ OpLog (BLAKE2b hash chain · per-client nonce · │ ← single source of truth
1569
- link │ idempotency keys · causal provenance fields) │
1570
- └───────────────┬──────────────────────────────────────────┘
1571
- deterministic fold │ (state = pure function of the log)
1572
- ┌──────────────┬──────────┴──────┬───────────────┬────────────────┐
1573
- ▼ ▼ ▼ ▼ ▼
1574
- MVCC store Relations Indexes CauseMap BlobStore
1575
- (time-travel) (graph+AS OF) eq/ord/search (reverse index) (Cascade CDC)
1576
1609
 
1577
- ┌─────────────────────────────────┐
1578
- Thread-safe → │ Sequencer (group-commit) │ ← single writer, parallel readers
1579
- │ — one committer thread/db │
1580
- │ — batch fsync │
1581
- └─────────────────────────────────┘
1610
+ **Isolation guarantee:** NEDB never writes to Alice's namespace. It owns only:
1582
1611
 
1583
- Compatibility adapters: SQL · Redis · MongoDB
1584
- Wire protocols: HTTP/JSON · RESP2
1585
- Encryption: AES-256-GCM at-rest (TMK/DEK double-envelope)
1612
+ | Key | Type | Purpose |
1613
+ |-----|------|---------|
1614
+ | `nedb:{db_name}:oplog` | Redis Stream | append-only op log |
1615
+ | `nedb:{db_name}:snapshot` | Redis Hash | checkpoint |
1616
+ | `nedb:{db_name}:meta` | Redis Hash | index config |
1617
+
1618
+ See [`examples/fakeredis_demo.py`](examples/fakeredis_demo.py) for a full local demo (no Redis server needed).
1619
+
1620
+ ---
1621
+
1622
+ ## Node.js
1623
+
1624
+ ```javascript
1625
+ import { NedbCore } from "nedb-engine";
1626
+
1627
+ const db = new NedbCore(); // in-memory
1628
+ // const db = NedbCore.open("./data"); // durable
1629
+
1630
+ db.createIndex("users", "status", "eq");
1631
+ db.put("users", "alice", JSON.stringify({ name: "Alice", age: 31, status: "active" }));
1632
+
1633
+ // Time-travel
1634
+ const snap = db.seq(); // BigInt
1635
+ db.put("users", "alice", JSON.stringify({ name: "Alice", age: 32, status: "retired" }));
1636
+ JSON.parse(db.getAsOf("users", "alice", snap)).age; # → 31
1637
+
1638
+ // Full NQL
1639
+ const rows = db.query('FROM users WHERE status = "active" ORDER BY age ASC');
1640
+ rows.map(r => JSON.parse(r));
1641
+
1642
+ // Tamper evidence
1643
+ db.verify(); // → true
1644
+ db.head(); // → 64-char BLAKE2b commitment hash
1645
+ db.seq(); // → BigInt
1586
1646
  ```
1587
1647
 
1588
1648
  ---
@@ -1615,53 +1675,98 @@ const rows = await db.query("FROM blocks LIMIT 10");
1615
1675
 
1616
1676
  ---
1617
1677
 
1678
+ ## NEDB v3 — Segment / Pack Object Store
1679
+
1680
+ **v3 is an opt-in storage substrate that replaces the loose one-file-per-object layout with append-only *segment packs* — the difference between a chainstate flush that takes *minutes* and one that takes *under two seconds*.** It is **off by default** (byte-for-byte v2), enabled with one flag, and **transparent** to everything above the storage layer: NQL, `AS OF`, `VALID AS OF`, `TRACE`, the BLAKE2b Merkle head, and causal provenance all behave identically.
1681
+
1682
+ ### Why it exists
1683
+
1684
+ v2 stores every document version as its own content-addressed file at `objects/{hash[:2]}/{hash[2:]}`. That makes writes trivially atomic (write `.tmp` → `rename`) and corruption-proof — but each write costs a file create + `fsync` + rename **plus** a directory B-tree update. At scale that filesystem-metadata churn dominates: on a busy disk it caps sustained writes around **~185/s**, and a batch flush of a few thousand objects degrades into minutes. The bottleneck is the *number of files touched*, not the bytes written.
1685
+
1686
+ ### What it does
1687
+
1688
+ v3 batches objects into append-only **segment packs** — `objects/segments/seg-NNNNNN.dat` — where each record is `[content_len: u32-LE][content]`. A write appends to the active segment and updates an in-memory `hash → (segment_id, offset, len)` map; a batch commits with a **single `fsync`**. Thousands of per-file syscalls collapse into one sequential append plus one durability point, so **flush cost scales with bytes (sequential I/O), not object-count × syscall overhead.**
1689
+
1690
+ - **Compaction / pruning** — `compact()` keeps the *live set* (the current version of every document, resolved from the id-index), rewrites those records into fresh segments, and reclaims the superseded/dead versions.
1691
+ - **`.idx` sidecars** — each segment carries a sidecar (`NIX1` magic + entry count + fixed 44-byte entries + a BLAKE2b-256 checksum) so reopen rebuilds the in-memory index by reading the sidecar instead of scanning the whole pack. A missing or corrupt sidecar falls back to a full scan-and-heal — slower, never fatal.
1692
+ - **Dual-read migration** — opening an existing v2 store in v3 mode is **non-destructive**: old loose objects stay fully readable, and only *new* writes go to segments. No migration step, no downtime, no rewrite.
1693
+ - **Durable flush-on-close** — `flush_all()` (and `Db`'s `Drop`) fsync the active segment, matching the flush-on-close contract of sled / RocksDB.
1694
+
1695
+ ### How to enable
1696
+
1697
+ ```bash
1698
+ # Engine / nedbd-v2 (the native daemon from npm / the native wheel)
1699
+ nedbd-v2 --dag-v3 --data /var/lib/nedb # real flag as of v2.4.3 — or set NEDB_DAG_V3=1
1700
+
1701
+ # itcd — Bitcoin-fork node embedding NEDB via nedb-ffi
1702
+ interchainedd -dagv3 # puts chainstate AND block index on segments
1703
+ ```
1704
+
1705
+ The switch is read once, when each database's object store is constructed at open time. Default off → v2 loose objects.
1706
+
1707
+ ### Real-world result
1708
+
1709
+ itcd (a Bitcoin Core 0.21 fork that replaces LevelDB chainstate with NEDB) syncing on `-dagv3`, measured `FlushStateToDisk` on real chainstate:
1710
+
1711
+ | Flush (coins → disk) | v3 segment store | v2 loose store |
1712
+ |---|---|---|
1713
+ | 2,002 coins / 275 kB | **1.93 s** | *minutes* |
1714
+ | 2,549 coins / 366 kB | **1.71 s** | *minutes* |
1715
+
1716
+ Note the *larger* batch finishing *faster* — v3's cost is dominated by the single per-batch `fsync`, not per-coin work, so effective throughput (~1,000–1,500 coins/s here) climbs as batches grow, against the loose store's ~185 writes/s metadata ceiling. The gap only widens as the UTXO set grows: sequential-append cost tracks data volume, while per-file cost compounds with object count.
1717
+
1718
+ ### When to use it
1719
+
1720
+ Reach for v3 on high-write, large-object-count workloads — blockchain chainstate / block index, event sourcing, high-frequency agent memory. For small or read-mostly stores the loose layout is perfectly fine, which is exactly why v3 stays opt-in.
1721
+
1722
+ ---
1723
+
1618
1724
  ## Repo layout
1619
1725
 
1620
1726
  ```
1621
1727
  python/nedb/ reference engine (pure Python — always-works baseline)
1622
1728
  rust/
1729
+ nedb-v2/ v2 DAG engine (tokio + axum + BLAKE2b DAG) — the core everything binds
1730
+ nesql-cli/ the nesql CLI (query / root / inspect / diff / tag / branch / merge)
1623
1731
  nedb-core/ v1 production Rust engine (shared by both runtimes)
1624
1732
  nedb-py/ maturin PyO3 binding → PyPI native wheels
1625
1733
  nedb-node/ napi-rs binding → npm native addons
1626
- nedb-v2/ v2 DAG engine (tokio + axum + BLAKE2b DAG)
1627
- client/
1628
- python/ nedb-client async Python HTTP client (pip install nedb-engine-client)
1629
- node/ nedb-client TypeScript HTTP client (npm install nedb-client)
1734
+ nedb-wrap/ the wrap_* surface at native speed
1735
+ vendor/postgresql/ PostgreSQL 17.4 grammar, vendored (gram.y, kwlist.h, system_views.sql, information_schema.sql)
1736
+ distributions/ crypto-database + aof-db submodules (the tri-distribution release)
1737
+ bench/ benchmarks.py + RESULTS.md the dated embedded-core numbers
1630
1738
  tests/ engine + concurrent + causal + bitemporal + deploy + perf benchmarks
1631
- examples/ resp2_python.py resp2_demo.sh
1632
- docs/ index.html reference.html SPEC.md
1739
+ vectors/ state_root_v1.json — cross-engine state-root test vectors
1740
+ docs/ SPEC.md · CLI.md · DURABILITY.md · REPLICATION.md · BENCH-sqlselect.md
1741
+ client/ nedb-engine-client — async Python + TypeScript HTTP clients
1742
+ examples/ agent-loop · mini-chain · resp2 demos · fakeredis demo
1743
+ scripts/ release.py · test-cast.sh · bench_index_range.py · seed-shop.sh
1633
1744
  ```
1634
1745
 
1635
1746
  ---
1636
1747
 
1637
- ## Roadmap
1638
-
1639
- - [x] Hash-chained append-only log tamper evidence, replay protection, idempotency
1640
- - [x] MVCC time-travel `AS OF seq`
1641
- - [x] Bi-temporal — `VALID AS OF "date"` (transaction time + valid time)
1642
- - [x] Causal Write Provenance `caused_by`, `evidence`, `confidence`, `TRACE`
1643
- - [x] Durable AOF persistence + snapshot checkpoints
1644
- - [x] Concurrent group-commit sequencer (nedbd, 15K writes/s under load)
1645
- - [x] AES-256-GCM at-rest encryption (TMK/DEK double-envelope)
1646
- - [x] SQL / Redis / MongoDB compatibility adapters
1647
- - [x] RESP2 wire protocol (redis-cli / redis-benchmark compatible)
1648
- - [x] Rust native core — napi-rs (npm) + maturin PyO3 (PyPI)
1649
- - [x] Self-healing AOF auto-truncates corrupt tail on startup, never hangs
1650
- - [x] **v2 DAG engine** content-addressed Merkle DAG, atomic writes, instant cold start
1651
- - [x] **`nedbd --dag`** — one flag switches to v2 Rust engine; v1 untouched
1652
- - [x] **BLAKE2b Merkle head** tamper-evident root on every response
1653
- - [x] **Tombstone deletes** history preserved in DAG, live id removed from index
1654
- - [x] **Auto-migration**v1 AOF v2 DAG on first `--dag` startup
1655
- - [x] **nedb-client**async Python + TypeScript HTTP client (`pip/npm install nedb-client`)
1656
- - [x] **Intel Mac support** native wheels for `aarch64` + `x86_64` Apple Darwin
1657
- - [x] **v3 segment/pack object store** opt-in `--dag-v3`: append-only packs, one fsync per batch, compaction + `.idx` sidecars, non-destructive dual-read (minutes → <2s chainstate flush on itcd)
1658
- - [ ] In-memory DAG mode `Db::in_memory()` for zero-disk ephemeral sessions
1659
- - [ ] PyO3 + napi-rs bindings updated to v2 DAG API
1660
- - [ ] NEDB Studio DAG mode toggle
1661
- - [ ] Merkle inclusion proofs — prove a document existed at a specific time to a third party
1662
- - [ ] Git-style branching — fork database state, experiment, merge or discard
1663
- - [ ] Agent Memory SDK — `Memory.remember()` / `Memory.recall()` / `Memory.trace()`
1664
- - [ ] Live query subscriptions (SSE) — push diffs when query results change
1748
+ ## Ship log — the highlights, in order
1749
+
1750
+ NEDB's release history outgrew any changelog, and the README was where versions went to pile up.
1751
+ This is the map; the commits are the story.
1752
+
1753
+ - **8.x neSQL ships.** PostgreSQL's real grammar vendored and extended; one evaluator answers every
1754
+ `SELECT` with nothing to enable; NQL's verbs are SQL clauses; the `nesql` CLI ships in the wheel.
1755
+ - **7.x the neSQL endpoint surface.** The HTTP `/query` route speaks both dialects through one
1756
+ router; the CLI's publish path lands.
1757
+ - **6.x hardening, discipline, the road to one name.**
1758
+ - **5.x neSQL named, registered, vendored.** The grammar lands at `vendor/postgresql/`;
1759
+ `nesql` reserved on all three registries.
1760
+ - **4.0 — BUSL-1.1** (free under $1M revenue; Apache 2.0 on the 2030-09-11 Change Date).
1761
+ - **3.3 the query language grows up.** Full boolean predicates in both engines; nine silent
1762
+ defects fixed; cross-engine parity gated.
1763
+ - **3.2 the wrap family.** Redis/SQLite/Postgres/MySQL/Mongo adapters; durability fixes; PR CI.
1764
+ - **3.0–3.3.1the MIT era** (irrevocably MIT).
1765
+ - **2.8Cast.** A 3.33M-parameter model trained inside the engine's own parser; drift detection.
1766
+ - **2.5flush-on-exit, nedb-cli, inspector, replication contract.**
1767
+ - **2.4 v3 segment store** (`--dag-v3`): itcd chainstate flush minutes ~1.3 s.
1768
+ - **2.2 v2 DAG engine.** Content-addressed Merkle DAG, O(1) warm start, SSE `/events`.
1769
+ - **1.x v1 AOF engine.** Hash-chained log, MVCC time-travel, bi-temporal, `TRACE caused_by`.
1665
1770
 
1666
1771
  ---
1667
1772
 
@@ -1673,14 +1778,16 @@ Prompt-to-database scaffolding GUI with schema graph, NQL console, time-travel s
1673
1778
 
1674
1779
  ---
1675
1780
 
1676
- ## Repos
1781
+ ## Repos & packages
1677
1782
 
1678
- | Repo | Description |
1783
+ | Where | What |
1679
1784
  |---|---|
1680
- | [aiassistsecure/nedb](https://github.com/aiassistsecure/nedb) | Source — engine, Rust core, CI |
1785
+ | [Eth-Interchained/nedb](https://github.com/Eth-Interchained/nedb) | **canonical source** — engine, Rust core, CI, this README |
1786
+ | [aiassistsecure/nedb](https://github.com/aiassistsecure/nedb) | mirror + [GitHub Pages site](https://nedb.aiassist.net) |
1787
+ | [Eth-Interchained/neSQL](https://github.com/Eth-Interchained/neSQL) | the language — both halves of the grammar, CLI source |
1681
1788
  | [aiassistsecure/nedb-studio](https://github.com/aiassistsecure/nedb-studio) | Studio UI (GPLv3) |
1682
1789
 
1683
- **Packages:** [PyPI nedb-engine](https://pypi.org/project/nedb-engine/) · [npm nedb-engine](https://www.npmjs.com/package/nedb-engine)
1790
+ **Packages:** [PyPI nedb-engine](https://pypi.org/project/nedb-engine/) · [npm nedb-engine](https://www.npmjs.com/package/nedb-engine) · [crates.io nedb-engine](https://crates.io/crates/nedb-engine) — plus the aligned `crypto-database` and `aof-db` distributions on all three registries.
1684
1791
 
1685
1792
  ---
1686
1793
 
@@ -1708,7 +1815,11 @@ Requires `GITHUB_TOKEN` (`repo` + `workflow` scope) in the environment. It never
1708
1815
 
1709
1816
  ## License
1710
1817
 
1711
- **MIT License** — free for any use, including commercial and production. See [`LICENSE`](LICENSE).
1818
+ **Business Source License 1.1** — free for any organisation under USD $1M annual revenue, including
1819
+ commercial and production use. Converts to **Apache 2.0** on **2030-09-11**, automatically and
1820
+ permanently. Versions 3.0.0–3.3.1 remain MIT, irrevocably. See [`LICENSE`](LICENSE) and
1821
+ [`COPYING-APACHE-2.0.txt`](COPYING-APACHE-2.0.txt).
1822
+
1712
1823
  © 2026 INTERCHAINED LLC — [interchained.org](https://interchained.org)
1713
1824
 
1714
1825
  ---