nedb-engine 2.7.1 → 2.8.2

package/README.md

@@ -20,9 +20,29 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 
 ---
 
-## NEDB v2.5.34 — Production Stable
+## NEDB v2.8.0 — Production Stable
 
-**Current stable: 2.5.34** — NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with full mac + linux + windows native addons (see [**Releasing**](#releasing) below). On the engine side it remains a polish release on the complete cross-platform line. The `nedbd-v2` daemon now does **real CLI parsing** — `--dag-v3`, `--data`, `--fast-fsync`, `--help`, `--version` are recognized flags instead of being silently swallowed as the positional data dir — and `npm test` ships a **cinematic native smoke test** that tours v1→v2 migration, the v2 DAG, the v3 segment store, and a causal-provenance audit. All native wheels (Linux + Windows on GitHub Actions; macOS arm64 + x86_64 on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
+**Current stable: 2.8.0** — NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with full mac + linux + windows native addons (see [**Releasing**](#releasing) below). All native wheels (Linux + Windows on GitHub Actions; macOS arm64 + x86_64 on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
+
+### New in 2.8.0 — Cast: the database understands English
+
+`POST /v1/databases/<name>/cast` turns a short English prompt into NQL, using a **3.33M-parameter model that runs locally on CPU**. No API key, no network call, no per-token bill.
+
+```bash
+nedbd --dag --cast ./data     # requires: cargo install nedb-engine --features cast
+
+curl -X POST localhost:7070/v1/databases/shop/cast \
+  -d '{"prompt":"orders over 100"}'
+# → {"nql":"FROM orders WHERE total > 100","valid":true,"collection_known":true,"executed":false}
+```
+
+The model ([**nedb-cast-slm**](https://github.com/aiassistsecure/nedb-cast-slm)) was trained on NQL using NEDB's own parser as generator, grader, and gate — then shipped to PyPI, crates.io, and npm, all three loading the identical weights.
+
+**Why it lives in the engine and not in a client:** the hard part of natural-language querying is knowing the schema, and the engine already holds the live collection list. A plan naming a collection that doesn't exist returns **422 with the reason**, never a silently empty result set. See [**Cast**](#cast--natural-language-into-nql) below.
+
+Off by default — feature-gated at compile time, flag-gated at runtime, and `execute` defaults to `false` so you review the plan before it runs.
+
+**Also in 2.8.0 — an engine bug the feature exposed.** `IdIndex::collections()` did a bare `read_dir` while every other read path overlaid the WAL write buffer, so a brand-new collection was invisible until the 1s flush ticker fired. Unreachable by hand (the ticker fires between keystrokes) but reliable from a script, and latent in `Db::compact()` too, where a missed collection's live objects would be reclaimed as garbage. Fixed, with regression tests that seed *without* flushing.
 
 **New in 2.5.x:**
 
@@ -304,6 +324,8 @@ Alongside the daemon, `cargo install nedb-engine` ships **`nedb-cli`** — opera
 | `NEDBD_TOKEN` | unset | Optional bearer token; required on every `/v1/*` request when set. |
 | `NEDB_TMK` | unset | 32-byte hex AES-256-GCM at-rest encryption key. |
 | `NEDBD_DATA` | `./nedb-data` | Root directory. v2 creates `dag/`, IdIndex sharded across **256 subdirectories**, and a small `MANIFEST` file. |
+| `NEDBD_CAST` | `0` | Set `1` to enable the `/cast` natural-language planner. Same as `--cast`. Requires a build with `--features cast`. See [**Cast**](#cast--natural-language-into-nql). |
+| `NEDBD_CAST_MODEL` | unset | Explicit path to a `model.cast` container. Otherwise searched in the data dir, `$CAST_HOME`, and `~/.cache/nedb-cast-slm/`. |
 
 ```bash
 # Create a database with seed data and relations
@@ -360,6 +382,218 @@ db.query('FROM policy AS OF 200 VALID AS OF "2024-02-15"')
 
 ---
 
+## Cast — natural language into NQL
+
+*New in 2.8.0. Optional, feature-gated, off by default.*
+
+Ten clauses and six operators. That's the whole grammar above — small enough that a **3.33M-parameter** model can learn it completely, and small enough that shipping every query to a frontier model is an absurd amount of machinery.
+
+So we trained one. It runs on CPU, in-process, in milliseconds.
+
+```bash
+curl -X POST localhost:7070/v1/databases/shop/cast \
+  -H 'Content-Type: application/json' \
+  -d '{"prompt":"orders over 100"}'
+```
+
+```json
+{
+  "prompt": "orders over 100",
+  "nql": "FROM orders WHERE total > 100",
+  "valid": true,
+  "collection": "orders",
+  "collection_known": true,
+  "collections": ["orders"],
+  "executed": false,
+  "seq": 3,
+  "head": "262fd9…"
+}
+```
+
+### NEDB is on both ends of this
+
+The model is [**nedb-cast-slm**](https://github.com/aiassistsecure/nedb-cast-slm), and NEDB built it as much as it consumes it.
+
+**NEDB's parser was the training pipeline.** It generated the corpus (sample a random plan → render NQL → render a human paraphrase; 200,000 pairs in 16.5 seconds, perfect labels, zero annotation cost). It was the grader — scoring *parsed plan* equality, not string equality, so `FROM orders WHERE total > 99` and `from orders where total>99` both earn full credit. And it was the gate: no example entered the corpus unless it round-tripped through the real parser to a canonically identical plan.
+
+> Most text-to-DSL projects hand-write a verifier and hope it's right. We didn't write one — it already shipped, and it's the same code the database runs in production.
+
+**Training lineage lives in NEDB too**, chained by `caused_by`:
+
+```
+datasets ──▶ training_runs ──▶ checkpoints ──▶ evals
+```
+
+```python
+db.query("FROM evals TRACE caused_by")   # the exact data behind any score
+```
+
+### Why the planner lives in the engine
+
+The hard part of natural-language querying is not the model. It's the **schema** — and a client has to fetch the collection list and pass it in, where it's stale on arrival. The engine already holds the live list.
+
+So the plan is checked against collections that actually exist, at the moment of the call:
+
+```json
+{ "prompt": "show me all stylists",
+  "nql": "FROM stylists",
+  "valid": true,
+  "collection_known": false,
+  "error": "collection \"stylists\" does not exist in \"shop\"" }
+```
+
+HTTP **422**. Not zero rows — zero rows reads as *"no matching data"*, which would be a lie. The query was perfectly well-formed; the collection was imagined. That's the model's known failure mode on an unfamiliar schema, and the engine is the one component positioned to catch it.
+
+Every `nedbd` client — Python, Node, Studio, `curl` — inherits this without writing a line.
+
+### Three safety properties
+
+| | |
+|---|---|
+| **The model never executes** | It emits text. The text goes to the same `nql::query` path a hand-typed query uses. No second executor exists to audit. |
+| **Validation is parsing** | `nql::parse` and `nql::execute` share one code path, so they cannot disagree about what is well-formed. Invalid output returns 422 *with the offending text*. |
+| **`execute` defaults to false** | You get a plan for review. Running a guess silently is worse than admitting uncertainty. |
+
+That last default earns its keep. A real miss, from a real run:
+
+```
+prompt   "paid orders over 100"
+nql      FROM orders WHERE status = "paid" LIMIT 100      ← wrong
+correct  FROM orders WHERE status = "paid" AND total > 100
+```
+
+It read *"over 100"* as `LIMIT 100` and dropped the predicate. The count still came back **2** — because both paid orders happened to exceed 100. A count-only assertion would have scored it a pass. A human reading `LIMIT 100` catches it in a heartbeat; an auto-executing client does not.
+
+Multi-predicate `WHERE` is the model's weakest clause: **85.1%** exact-plan match on eval, 61.2% on adversarial holdout. The [model card](https://github.com/aiassistsecure/nedb-cast-slm#what-it-gets-wrong) documents every failure mode with examples.
+
+To run it anyway, ask:
+
+```bash
+curl -X POST localhost:7070/v1/databases/shop/cast \
+  -d '{"prompt":"orders over 100","execute":true}'
+# → { …, "executed": true, "count": 2, "rows": [ … ] }
+```
+
+### The failure `valid` cannot catch
+
+A literal the model **invented** rather than copied:
+
+```
+"memories about pricing"  ->  FROM memories SEARCH "handoff"
+```
+
+That query parses. It names a real collection. It returns real rows. Both
+`valid` and `collection_known` are `true` — and it answers a question nobody
+asked. Measured on the released checkpoint:
+
+| terms | in vocabulary | copied correctly |
+|---|---|---|
+| `release flow` · `guardrail` · `handoff` | yes | **3/3** |
+| `pricing` · `deadlines` · `kubernetes` | no | **0/3** — all became `"handoff"` |
+
+So the response carries a `drift` field when a quoted literal is absent from the
+prompt:
+
+```json
+{ "nql": "FROM memories SEARCH \"handoff\"",
+  "valid": true,
+  "collection_known": true,
+  "drift": "generated the literal \"handoff\", which does not appear in the prompt — likely outside the model's vocabulary and substituted. Verify before trusting these results." }
+```
+
+It is **advisory, never fatal** — the plan may still be what you wanted, and
+discarding a valid query would be its own kind of lie. But an unattended caller
+should treat it as a third gate:
+
+```python
+if plan["valid"] and plan["collection_known"] and not plan.get("drift"):
+    rows = await db.query(plan["nql"])
+```
+
+Same root cause as truncated digits (`height 400000` → `4000`): no copy
+mechanism over prompt tokens. Verified at 24/24 on real model output — 3 true
+positives, 21 true negatives, zero false alarms, including the case that matters
+most (correctly inferred enum values like *"refunded orders"* → `status =
+"refunded"` stay silent).
+
+### Enabling it
+
+Two gates, because most deployments want neither the model dependency nor the weights:
+
+```bash
+# compile-time
+cargo install nedb-engine --features cast
+
+# or from a source checkout — builds the engine only, not the language bindings
+cd rust && cargo build --release --features cast
+
+# weights (~13 MB) — GitHub release asset, checksum-verified on load
+curl -L -o ./data/model.cast \
+  https://github.com/aiassistsecure/nedb-cast-slm/releases/download/v10.30.90/model.cast
+
+# runtime
+nedbd --dag --cast ./data
+#   cast     enabled — 3.33M params, vocab 581, ./data/model.cast
+```
+
+Search order: `$NEDBD_CAST_MODEL` → `<data_dir>/model.cast` → `$CAST_HOME/model.cast` → `~/.cache/nedb-cast-slm/v10.30.90/model.cast` (the Python/npm cache location, so a machine that has run either package is already ready).
+
+Built without the feature, the route returns **501** rather than 404 — clients can detect the capability instead of guessing. Built with it but missing weights, the daemon logs loudly and serves everything else normally.
+
+**Verify the whole path:**
+
+```bash
+./scripts/test-cast.sh --boot     # boots a daemon, seeds, casts, executes, checks failure modes
+```
+
+### Casting from a shell
+
+```bash
+./scripts/seed-shop.sh       # a shop database the model already understands
+. ./scripts/nedb.sh          # bash / zsh / Git Bash
+
+nedb-dbs                     # which databases exist
+nedb-use shop                # pick one
+cast "orders over 100"       # plan only — nothing runs
+cast -x "orders over 100"    # plan AND execute
+```
+
+**Seed the names it was trained on.** The model learned six synthetic domains, and `shop` is one of them — `orders(total, status, quantity, customer, placed_at, discounted)`, `products(price, stock, category, rating, title)`, `customers(age, city, tier, lifetime_value, name)`, plus the relations `purchased` / `reviewed` / `belongs_to`. Those names live in its 581-token vocabulary.
+
+Call your collection `purchases` with a `cost` field and it will still emit `FROM orders WHERE total > …`, because that is what it knows. It is a 3.3M-parameter model, not a schema reader. On an unfamiliar schema you get `collection_known: false` — caught, not silently wrong, but caught.
+
+```
+  nql         FROM orders WHERE total > 100
+  valid       yes    collection orders known: yes
+  executed    no  (add -x to run it)
+```
+
+The summary leads with the NQL because **reading it is the job**. `valid: yes` means it parses, not that it's what you meant — `LIMIT 100` parses perfectly.
+
+`NEDB=http://host:7070` points at a remote daemon. Prompts are JSON-escaped, so apostrophes and quotes are safe.
+
+### Prompts it handles well
+
+Accuracy varies by clause, so phrasing matters more than length:
+
+| you want | say | eval |
+|---|---|---|
+| `TRACE caused_by` | *what caused these checkpoints* | 96.5% |
+| `TRAVERSE` | *orders traverse placed_by* | 93.3% |
+| one `WHERE` | *orders over 100* · *active drivers* | 91.2% |
+| `LIMIT` | *top 5 orders* | 91.1% |
+| `SEARCH` | *search orders for refund* | 90.5% |
+| `ORDER BY` | *orders sorted by total descending* | 87.7% |
+| two+ `WHERE` | *paid orders with total over 100* | 85.1% |
+| `GROUP BY` + agg | *orders grouped by status with sum of total* | 77.0% |
+
+Two habits that avoid most misses:
+
+- **Name the field** when a number could be a limit. *"orders with total over 100"* beats *"orders over 100"* — bare *"over N"* is what produced the `LIMIT 100` miss above.
+- **Check numbers over four digits.** Digits are tokenized one at a time, so `height 400000` can come back `4000`.
+
+---
+
 ## Performance
 
 **v2 DAG Rust server (v2.2.31, Intel iMac — 10k writes / 100k reads / 30k objects, AES-256-GCM on):**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "nedb-engine",
-  "version": "2.7.1",
+  "version": "2.8.2",
   "description": "NEDB — hash-chained, time-traveling, bi-temporal embedded database with Rust native core. SQL, Redis, MongoDB adapters. Causal Write Provenance. RESP2 wire protocol.",
   "main": "index.js",
   "types": "index.d.ts",

package/test/smoke.mjs

@@ -15,6 +15,7 @@
 import os from 'node:os';
 import fs from 'node:fs';
 import path from 'node:path';
+import { execFileSync } from 'node:child_process';
 
 // ── tiny presentation toolkit ───────────────────────────────────────────────
 const COLOR = process.stdout.isTTY && !process.env.NO_COLOR;
@@ -53,11 +54,17 @@ const rm = (d) => { try { fs.rmSync(d, { recursive: true, force: true }); } catc
 
 // ── resolve the native addon (installed package, or in-repo after a build) ───
 let NedbCore;
+// Remember WHICH specifier resolved, so the ACT IV reopen subprocess loads the
+// exact same addon this process did — not a different one that happens to be
+// installed. Getting that wrong would make the reopen test silently meaningless.
+let addonSpecifier;
 try {
   ({ NedbCore } = await import('nedb-engine'));
+  addonSpecifier = 'nedb-engine';
 } catch {
   try {
-    ({ NedbCore } = await import(new URL('../index.js', import.meta.url)));
+    addonSpecifier = new URL('../index.js', import.meta.url).href;
+    ({ NedbCore } = await import(addonSpecifier));
   } catch (err) {
     log(red(bold('\n  nedb-engine native addon not found.')));
     note('This smoke test drives the prebuilt NedbCore binding.');
@@ -163,11 +170,31 @@ try {
   kv('objects/ layout', `${bold(looseObjs)} loose object files  ${dim('(content-addressed, one per object)')}`);
 
   // v3 substrate (env set BEFORE open — the engine reads it per-open).
+  //
+  // The WRITER runs in a child process on purpose. A durable open takes an
+  // exclusive flock on the data dir and the binding exposes no close(), so the
+  // lock lives as long as the handle — and flock is per open-file-description,
+  // not per process tree: a child is blocked by its parent's lock just like any
+  // stranger would be. Letting the writer be a separate process that EXITS is
+  // the only way to release the directory and then genuinely reopen it below.
+  //
+  // It also makes this the real crash-and-restart path — process dies, files
+  // remain, next process picks them up — instead of a same-process handle
+  // shuffle that proves nothing about durability.
   process.env.NEDB_DAG_V3 = '1';
   const segDir = tmp('v3seg'); cleanup.push(segDir);
-  const segDb = NedbCore.open(segDir);
-  for (let i = 0; i < docs; i++) segDb.put('utxo', String(i), sample(i));
-  segDb.flush();
+  const writer = `
+    const { NedbCore } = await import(${JSON.stringify(addonSpecifier)});
+    const db = NedbCore.open(${JSON.stringify(segDir)});
+    for (let i = 0; i < ${docs}; i++) {
+      db.put('utxo', String(i), JSON.stringify({ i, payload: 'coin-' + i, ts: 1719400000 + i }));
+    }
+    db.flush();
+  `;
+  execFileSync(process.execPath, ['--input-type=module', '-e', writer], {
+    env: { ...process.env, NEDB_DAG_V3: '1' },
+    encoding: 'utf8',
+  });
   const segPath = path.join(segDir, 'objects', 'segments');
   const segFiles = fs.existsSync(segPath) ? fs.readdirSync(segPath).filter((f) => f.endsWith('.dat')) : [];
   step(`v3 segment substrate — ${bold(docs)} writes  ${dim('(NEDB_DAG_V3=1)')}`);
@@ -180,9 +207,16 @@ try {
     note('segment file not present yet (objects buffered) — engine still serving from memory+WAL.');
   }
   step('v3 round-trips and verifies after reopen');
+  // The writer process has exited, so the flock is gone and this open is a true
+  // reopen of files written by a process that no longer exists.
+  //
+  // Two wrong ways to make this pass, for the record:
+  //   * NEDB_SHARED_OPEN=1 — silences the split-brain guard instead of testing
+  //   * dropping the assertion — stops covering what it was written for
   const segReopen = NedbCore.open(segDir);  // env still set → reopen as v3
   const rt = JSON.parse(segReopen.get('utxo', '7'));
   kv('reopen → utxo/7', JSON.stringify({ i: rt.i, payload: rt.payload }));
+  note('written by a process that has since exited — this is the restart path');
   tick(`verify() = ${green(String(segReopen.verify()))}  ${dim('— segment store + dual-read of any v2 loose objects')}`);
   expect(rt.i === 7 && segReopen.verify() === true, 'v3 persists and verifies across reopen');
   delete process.env.NEDB_DAG_V3;   // leave the environment as we found it