nedb-engine 2.0.5 → 2.0.31

package/README.md

@@ -2,7 +2,7 @@
 
 # NEDB
 
-**Hash-chained · time-traveling · bi-temporal · causally-provable embedded database.**
+**Content-addressed Merkle DAG · Hash-chained · Time-traveling · Bi-temporal · Causally-provable embedded database.**
 
 Replay-protected · idempotent · relational · filterable · sortable · searchable · concurrent.
 One Rust core → ships to **PyPI** and **npm** from a single source.
@@ -10,6 +10,8 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 [![PyPI](https://img.shields.io/pypi/v/nedb-engine?label=PyPI&color=6366f1)](https://pypi.org/project/nedb-engine/)
 [![npm](https://img.shields.io/npm/v/nedb-engine?label=npm&color=00d4ff)](https://www.npmjs.com/package/nedb-engine)
 [![Tests](https://img.shields.io/badge/tests-266%20passing-34d399)](https://github.com/Eth-Interchained/nedb/actions)
+[![nedb-engine-client PyPI](https://img.shields.io/pypi/v/nedb-engine-client?label=nedb-engine-client&color=34d399)](https://pypi.org/project/nedb-engine-client/)
+[![nedb-engine-client npm](https://img.shields.io/npm/v/nedb-engine-client?label=nedb-engine-client&color=34d399)](https://www.npmjs.com/package/nedb-engine-client)
 
 **[Studio → studio.interchained.org](https://studio.interchained.org)**  ·  **[nedb.aiassist.net](https://nedb.aiassist.net)**
 
@@ -17,6 +19,47 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 
 ---
 
+## v2 — The DAG Engine (current stable: 2.0.27)
+
+NEDB v2 replaces the append-only log (AOF) with a **content-addressed Merkle DAG**. Every document version is an immutable, BLAKE2b-verified object. Nothing is ever overwritten. As of **v2.0.27**, restarts after the first open are **O(1) warm starts** (driven by a `MANIFEST` of `seq` + Merkle head), the **cold scan is deferred** so the daemon accepts connections immediately, and a new **`GET /events` SSE endpoint** streams scan progress + per-write events live.
+
+```bash
+# Run the v2 DAG engine — ships inside pip install nedb-engine
+nedbd --dag --data ./data
+# or
+NEDBD_DAG=1 NEDB_TMK=<32-byte-hex> nedbd --data ./data
+
+curl http://127.0.0.1:7070/health
+# {"ok":true,"version":"2.0.27","service":"nedbd","engine":"dag","startup_ready":true,"encrypted":true}
+
+# Tail the live event stream (new in v2.0.27)
+curl http://127.0.0.1:7070/events
+# event: scan   data: {"objects":730000,"of":1310703,"rate":21043,"eta_s":28}
+# event: ready  data: {"seq":1310703,"head":"b2:9c14e07a…"}
+# event: write  data: {"seq":1310704,"coll":"beliefs","head":"b2:7af3c11e…"}
+```
+
+| Property | v2 DAG | v1 AOF |
+|---|:---:|:---:|
+| Uncorruptable (atomic writes, hash-verified reads) | ✅ | ⚠️ |
+| O(1) warm start via MANIFEST (no scan, no replay) | ✅ | ❌ |
+| Deferred cold scan (socket open immediately) | ✅ | ❌ |
+| O(1) incremental Merkle head (never recomputed) | ✅ | ❌ |
+| Parallel writes (no global lock) | ✅ | ❌ |
+| BLAKE2b Merkle head on every response | ✅ | ❌ |
+| IdIndex sharded across 256 subdirectories | ✅ | ❌ |
+| TCP_NODELAY (no 40–200 ms loopback Nagle delay) | ✅ | ❌ |
+| `GET /events` SSE log stream | ✅ | ❌ |
+| Tombstone deletes (history preserved) | ✅ | ✅ |
+| Auto-migrates v1 AOF → v2 DAG on startup | ✅ | — |
+| Same HTTP API — Vision, Studio, all clients unchanged | ✅ | ✅ |
+
+**v1 AOF engine is still shipped and unchanged** — `nedbd` (no flag) runs v1.
+
+**Production status:** [vision.interchained.org](https://vision.interchained.org) is live on v2.0.27 — **1,310,703 sequences** indexed in the Vision database, AES-256-GCM encrypted at rest, at block height **620,989**.
+
+---
+
 ## What makes NEDB different
 
 Every database stores *what*. NEDB stores *what*, *when*, *when it was true*, and *why* — all sealed in a cryptographic hash chain that proves none of it was tampered with.
@@ -173,11 +216,31 @@ db.seq();      // → BigInt
 nedbd runs NEDB as a long-lived process with an HTTP/JSON API and an optional RESP2 wire protocol. Built on a **single-writer group-commit sequencer** — parallel reads, batched durable writes, one hash-chain per database, zero write-write races.
 
 ```bash
-nedbd                                     # :7070, data ./nedb-data
+nedbd                                     # :7070, data ./nedb-data (v1 AOF engine)
+nedbd --dag --data ./data                 # v2 DAG engine (or NEDBD_DAG=1)
 NEDBD_RESP2_PORT=6380 nedbd               # also speak RESP2 (redis-cli compatible)
 nedbd --log-level 2                       # 0=errors 1=requests 2=deploy 3=verbose
+
+# Live event stream (new in v2.0.27) — SSE: scan progress, ready, per-write head
+curl http://127.0.0.1:7070/events
 ```
 
+### Startup modes (v2.0.27)
+
+- **Warm start** — every restart after the first open reads the `MANIFEST` file and restores `seq` + Merkle `head` in **O(1)**. No scan, no replay, independent of dataset size. Boots in milliseconds.
+- **Cold start** — first open of an existing dataset spawns the integrity scan in a background thread *and accepts connections immediately*. Reads serve instantly from the content-addressed DAG; writes return `HTTP 503 startup in progress` until the `startup_ready` gate flips. Progress (objects, rate, ETA) streams over `GET /events`.
+
+### Environment variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `NEDBD_DAG` | `0` | Set `1` to launch the v2 DAG engine (`nedbd-v2`). Same as `--dag`. |
+| `NEDBD_HOST` | `127.0.0.1` | Bind address. **v2.0.27** defaults to loopback (was `0.0.0.0`) — security hardening fix. Set explicitly to `0.0.0.0` to expose. |
+| `NEDBD_PORT` | `7070` | HTTP bind port. |
+| `NEDBD_TOKEN` | unset | Optional bearer token; required on every `/v1/*` request when set. |
+| `NEDB_TMK` | unset | 32-byte hex AES-256-GCM at-rest encryption key. |
+| `NEDBD_DATA` | `./nedb-data` | Root directory. v2 creates `dag/`, IdIndex sharded across **256 subdirectories**, and a small `MANIFEST` file. |
+
 ```bash
 # Create a database with seed data and relations
 curl -X POST :7070/v1/databases -d '{
@@ -233,15 +296,37 @@ db.query('FROM policy AS OF 200 VALID AS OF "2024-02-15"')
 
 ---
 
-## Performance (v1.0.x · Rust native · Linux x86_64 VPS)
+## Performance
+
+**v2 DAG Rust server (v2.0.27, Intel iMac — 10k writes / 100k reads / 30k objects, AES-256-GCM on):**
+
+| Operation | Throughput | p50 | p99 |
+|---|---|---|---|
+| Sequential writes | **418 ops/s** | 2.3 ms | 3.3 ms |
+| Point-lookup reads | **478 ops/s** | 2.0 ms | 3.0 ms |
+| ORDER BY queries | **489 ops/s** | 1.8 ms | 4.3 ms |
+| Batch writes (500 ops/req) | **1,104 ops/s** | 0.9 ms | 1.2 ms |
+| Tamper-verify (30k objects) | ~21,000 BLAKE2b/sec | — | 1.38 s total |
+
+p99 latencies hold because of `TCP_NODELAY` on the axum listener — without it macOS loopback adds the Nagle algorithm's 40–200 ms delay on small writes.
+
+**v1 Python server (baseline — single-threaded AOF):**
 
-| Operation | Throughput | Notes |
+| Operation | Throughput | p99 latency |
 |---|---|---|
-| PUT (Rust napi, per-op FFI) | ~70K/s | FFI-bound; batch path: ~15K writes/s group-commit |
-| GET (Rust napi, per-op FFI) | ~330K/s | FFI-bound |
-| NQL query (Rust engine) | ~23 µs | 5× faster than pure-Python (~120 µs) |
-| Python PUT (AOF + fsync) | ~7K/s | Durable, per-op |
-| Python GET (in-process) | ~1.3M/s | Zero socket hop |
+| Sequential PUT | ~23/s | 44 ms |
+| Concurrent PUT (16 workers) | ~92/s | 48 ms |
+| Batch PUT (500 ops/request) | ~520 ops/s | 1.9 ms/op |
+| Point-lookup read (NQL) | ~23/s | 44 ms |
+| Rust napi PUT (FFI) | ~70K/s | — |
+| Rust napi GET (FFI) | ~330K/s | — |
+
+Reproduce with the included benchmark:
+
+```bash
+NEDBD_DAG=1 nedbd --data /tmp/perf &
+python3 tests/test_dag_perf.py --n 10000 --reads 100000
+```
 
 ---
 
@@ -271,16 +356,49 @@ Encryption:              AES-256-GCM at-rest (TMK/DEK double-envelope)
 
 ---
 
+## nedb-client — lightweight HTTP client
+
+Connect to any running nedbd instance from Python or TypeScript without embedding the engine:
+
+```bash
+pip install nedb-engine-client          # async Python
+npm install nedb-engine-client   # TypeScript / Node.js 18+
+```
+
+```python
+from nedb_client import NedbClient
+
+async with NedbClient("http://127.0.0.1:7070", db="mydb") as db:
+    await db.put("blocks", "618000", {"height": 618000})
+    rows = await db.query("FROM blocks ORDER BY height DESC LIMIT 10")
+    head = await db.head()    # BLAKE2b Merkle root — changes on every write
+    ok   = await db.verify()  # tamper-evidence check across all objects
+```
+
+```typescript
+import { NedbClient } from "nedb-engine-client";
+const db = new NedbClient({ url: "http://127.0.0.1:7070", db: "mydb" });
+await db.put("blocks", "618000", { height: 618000 });
+const rows = await db.query("FROM blocks LIMIT 10");
+```
+
+---
+
 ## Repo layout
 
 ```
 python/nedb/        reference engine (pure Python — always-works baseline)
 rust/
-  nedb-core/        production Rust engine (shared by both runtimes)
+  nedb-core/        v1 production Rust engine (shared by both runtimes)
   nedb-py/          maturin PyO3 binding → PyPI native wheels
   nedb-node/        napi-rs binding → npm native addons
-tests/              engine + concurrent + causal + bitemporal + deploy tests
+  nedb-v2/          v2 DAG engine (tokio + axum + BLAKE2b DAG)
+client/
+  python/           nedb-client — async Python HTTP client (pip install nedb-engine-client)
+  node/             nedb-client — TypeScript HTTP client  (npm install nedb-client)
+tests/              engine + concurrent + causal + bitemporal + deploy + perf benchmarks
 examples/           resp2_python.py  resp2_demo.sh
+docs/               index.html  reference.html  SPEC.md
 ```
 
 ---
@@ -297,7 +415,17 @@ examples/           resp2_python.py  resp2_demo.sh
 - [x] SQL / Redis / MongoDB compatibility adapters
 - [x] RESP2 wire protocol (redis-cli / redis-benchmark compatible)
 - [x] Rust native core — napi-rs (npm) + maturin PyO3 (PyPI)
-- [x] Self-healing chains (auto-repair structural gaps, detect real tampering)
+- [x] Self-healing AOF — auto-truncates corrupt tail on startup, never hangs
+- [x] **v2 DAG engine** — content-addressed Merkle DAG, atomic writes, instant cold start
+- [x] **`nedbd --dag`** — one flag switches to v2 Rust engine; v1 untouched
+- [x] **BLAKE2b Merkle head** — tamper-evident root on every response
+- [x] **Tombstone deletes** — history preserved in DAG, live id removed from index
+- [x] **Auto-migration** — v1 AOF → v2 DAG on first `--dag` startup
+- [x] **nedb-client** — async Python + TypeScript HTTP client (`pip/npm install nedb-client`)
+- [x] **Intel Mac support** — native wheels for `aarch64` + `x86_64` Apple Darwin
+- [ ] In-memory DAG mode — `Db::in_memory()` for zero-disk ephemeral sessions
+- [ ] PyO3 + napi-rs bindings updated to v2 DAG API
+- [ ] NEDB Studio DAG mode toggle
 - [ ] Merkle inclusion proofs — prove a document existed at a specific time to a third party
 - [ ] Git-style branching — fork database state, experiment, merge or discard
 - [ ] Agent Memory SDK — `Memory.remember()` / `Memory.recall()` / `Memory.trace()`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "nedb-engine",
-  "version": "2.0.5",
+  "version": "2.0.31",
   "description": "NEDB — hash-chained, time-traveling, bi-temporal embedded database with Rust native core. SQL, Redis, MongoDB adapters. Causal Write Provenance. RESP2 wire protocol.",
   "main": "index.js",
   "types": "index.d.ts",