nedb-engine 4.3.2__tar.gz → 6.0.0__tar.gz

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nedb-engine
-Version: 4.3.2
+Version: 6.0.0
 Summary: NEDB — a versioned, self-compressing, time-traveling embedded database (replay-protected, idempotent, relational, searchable) with durable AOF persistence and a server daemon (nedbd).
 Project-URL: Homepage, https://github.com/aiassistsecure/nedb
 Project-URL: Repository, https://github.com/aiassistsecure/nedb
@@ -39,7 +39,7 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 **[Studio → studio.interchained.org](https://studio.interchained.org)**  ·  **[nedb.aiassist.net](https://nedb.aiassist.net)**
 
 > ## 🟢 Free in production under $1M revenue
-> NEDB 4.0.0 is licensed under the **Business Source License 1.1**. If your organisation's annual
+> NEDB is licensed under the **Business Source License 1.1** (since 4.0.0). If your organisation's annual
 > revenue is **under USD $1,000,000**, you may use it in production — commercially, embedded, in
 > closed-source software — with **no permission needed and no royalty**. At **$1M or more**, you
 > need an additional use grant from Interchained LLC: **licensing@interchained.org**.
@@ -59,14 +59,74 @@ endpoint already answers `psql`, SQLAlchemy Core **and** ORM, asyncpg and
 node-postgres against a live store — but it gets there by *translating* SQL into
 NQL, and a translation can only reach as far as the target language's shape.
 
-**[neSQL](https://github.com/Eth-Interchained/neSQL)** removes the translation:
-PostgreSQL's real grammar, vendored with its licence intact, extended with NEDB's
-temporal and causal clauses. Two front-ends, one plan. NQL folded in, not deleted.
+**[neSQL](https://github.com/Eth-Interchained/neSQL)** removes the translation.
+PostgreSQL's real grammar (`gram.y`, 19,513 lines, 492 keywords, vendored from
+17.4 at [`vendor/postgresql/`](vendor/postgresql/) with its licence intact),
+extended with NEDB's temporal and causal clauses. **Two front-ends, one plan.
+NQL folded in, not deleted.**
 
 [![neSQL on PyPI](https://img.shields.io/pypi/v/nesql?label=nesql%20·%20PyPI&color=a855f7)](https://pypi.org/project/nesql/)
 [![neSQL on crates.io](https://img.shields.io/crates/v/nesql?label=nesql%20·%20crates.io&color=a855f7)](https://crates.io/crates/nesql)
 [![neSQL on npm](https://img.shields.io/npm/v/nesql-engine?label=nesql-engine%20·%20npm&color=a855f7)](https://www.npmjs.com/package/nesql-engine)
 
+### Available now, opt-in: `NEDBD_SQL_ENGINE=1`
+
+The SQL engine is in this release and it is **off by default**. Turn it on and a
+user collection is answered by a real SQL evaluator instead of a translation —
+every one of these works, and every one is refused *by name* without it:
+
+```sql
+SELECT o._id, d.name FROM orders o JOIN drivers d ON o.driver = d._id;
+SELECT status, sum(total), avg(total) FROM orders GROUP BY status;
+SELECT _id FROM orders WHERE driver IN (SELECT _id FROM drivers);
+SELECT DISTINCT status FROM orders;
+SELECT _id FROM orders UNION SELECT _id FROM drivers;
+SELECT status, array_agg(_id ORDER BY total DESC) FROM orders GROUP BY status;
+```
+
+`sum(total), avg(total)` in one grouped row is the one worth pointing at. An NQL
+grouped row carries the group key, `count`, and **one** named aggregate — so that
+query was never slow, it was *unrepresentable*. No translator can fix a row
+model, which is the whole reason neSQL exists.
+
+**And NQL's own verbs are now SQL clauses**, so they compose with all of the
+above rather than living on a separate path:
+
+```sql
+-- full-text search from NQL, a join from SQL, one statement
+SELECT o._id, d.name
+  FROM orders SEARCH 'acme' o
+  JOIN drivers d ON o.driver = d._id;
+
+-- one relation in the past, joined against another at the tip
+SELECT h.total, n.total
+  FROM orders AS OF SYSTEM TIME 412 h
+  JOIN audit n ON h._id = n._id;
+
+SELECT _id FROM orders VALID AS OF '2026-01-01';
+```
+
+There is **one implementation** of each verb — the SQL side parses them and the
+NQL engine still executes them — so neither language is a reimplementation of
+the other. `AS OF SYSTEM TIME`, `VALID AS OF` and `SEARCH` are **unreserved
+keywords**: a collection aliased `search`, or a column named `valid`, keeps
+working exactly as before.
+
+**Why it is opt-in rather than the default**, stated plainly because the reason
+is the interesting part. A parity harness runs the same corpus through both
+engines and asserts identical answers — 44 checks, in CI, and it is what earns
+the flag being flipped rather than a benchmark. It already found two real
+divergences: `SELECT *` returned its columns in a different order on each
+engine, and the SQL evaluator built its column list from the **first row alone**,
+so a field only later documents carried silently did not appear at all.
+
+Both are fixed. But 44 checks over six documents proves agreement on the shapes
+we thought to test, and the evaluator still materialises each relation — the
+`WHERE` is pushed into the scan, which narrows *what* is read but not *whether*.
+Flipping the default changes the read path of every existing deployment, and
+"correct on six rows" is not "safe on six million". So it ships as a flag, with
+the bar for changing that written down.
+
 ---
 
 ## New in 3.3.0 — the query language grew up
@@ -221,46 +281,59 @@ all the work in this sentence. Every refusal below traces to the same cause:
 NQL is the engine's native language, so SQL has to be rewritten into it, and a
 rewrite can only ever reach as far as the target language's shape.
 
-| Supported today | Refused, with the reason | neSQL |
+| Supported on the default path | Refused there, with the reason | `NEDBD_SQL_ENGINE=1` |
 | --- | --- | --- |
-| `*`, a column list, `COUNT(*)`, `SUM`/`AVG`/`MIN`/`MAX(col)` | `JOIN` — NQL is single-collection | ✅ joins already exist in the executor |
-| `WHERE` — the whole NQL predicate surface | subqueries, `UNION`, window functions | ✅ subqueries + set ops exist; windows arrive with the grammar |
-| `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`, `OFFSET` | expressions in the select list | ✅ |
-| one named aggregate per grouped row | `sum(x), avg(x)` in one query — NQL's grouped row holds *one* | ✅ the row-model cap goes away |
-| `AS OF SYSTEM TIME <seq>` | DDL, `TRUNCATE`, `GRANT`/`REVOKE` | ⛔️ still refused, and always will be |
-| `INSERT` / `UPDATE` / `DELETE`, all with `RETURNING` | an `INSERT` with no column list | ✅ |
-| `_caused_by` / `_valid_from` / `_valid_to` as INSERT columns | values that are expressions, not literals | ✅ |
+| `*`, a column list, `COUNT(*)`, `SUM`/`AVG`/`MIN`/`MAX(col)` | `JOIN` — NQL is single-collection | ✅ **works** (nested-loop + hash) |
+| `WHERE` — the whole NQL predicate surface | subqueries, `UNION`, window functions | ✅ **subqueries, `EXISTS`, `UNION`/`INTERSECT`/`EXCEPT` work**; window functions arrive with the grammar |
+| `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`, `OFFSET` | expressions in the select list | ✅ **works** |
+| one named aggregate per grouped row | `sum(x), avg(x)` in one query — NQL's grouped row holds *one* | ✅ **works** — the row-model cap is gone |
+| — | `DISTINCT`, `array_agg(x ORDER BY y)`, derived tables | ✅ **works** |
+| `AS OF SYSTEM TIME <seq>`, `VALID AS OF`, `SEARCH` | — | ✅ **also works**, and composes with joins and aggregates |
+| `TRACE`, `TRAVERSE`, `LINK` | — | ↩︎ still answered by the NQL path; SQL has no spelling for them yet |
+| `INSERT` / `UPDATE` / `DELETE`, all with `RETURNING` | an `INSERT` with no column list | writes always take the NQL path |
+| `_caused_by` / `_valid_from` / `_valid_to` as INSERT columns | values that are expressions, not literals | as above |
+| — | DDL, `TRUNCATE`, `GRANT`/`REVOKE` | ⛔️ **refused on both**, and always will be |
 
 The `⛔️` row is the one that is not a limitation. `TRUNCATE` is refused because
 NEDB is append-only *so that history cannot be discarded* — that is the product,
 not a gap — and DDL is refused because collections are created by the first write
 to them. Those answers do not change.
 
-### Every other row on that table is a translation artefact, and it is going away
+### Every other row on that table was a translation artefact — and one flag removes them
 
 > ### 🆕 [**neSQL**](https://github.com/Eth-Interchained/neSQL) — PostgreSQL's grammar, NEDB's memory
 >
-> We stopped translating. neSQL vendors PostgreSQL's **real grammar** — `gram.y`,
-> 19,513 lines and 492 keywords, from PostgreSQL 17.4, licence intact — and extends
-> it with the clauses NEDB needs, rather than rewriting SQL into a language that
-> cannot express it.
+> Those refusals were never the engine's limits. `sqlselect.rs` has had
+> nested-loop and hash joins, subqueries, `EXISTS`, quantified comparisons, set
+> operations, `array_agg(x ORDER BY y)` and derived tables for some time — they
+> were simply unreachable *through a translator*, because the translator's
+> target was NQL. Set `NEDBD_SQL_ENGINE=1` and they are reachable.
+>
+> neSQL vendors PostgreSQL's **real grammar** — `gram.y`, 19,513 lines and 492
+> keywords, from 17.4, licence intact — and extends it with the clauses NEDB
+> needs, rather than rewriting SQL into a language that cannot express it.
 >
-> It is worth knowing *why* this was never free: `SYSTEM_TIME`, `PERIOD` and
-> `PORTION` appear **zero** times in PostgreSQL's grammar. Postgres has no temporal
-> SQL at all. `AS OF SYSTEM TIME` is a CockroachDB extension, which means NEDB's
-> temporal clauses are additions to the vendored grammar rather than deviations
-> from it — the same road CockroachDB, Materialize and RisingWave took.
+> It is worth knowing *why* the temporal clauses were never free: `SYSTEM_TIME`,
+> `PERIOD` and `PORTION` appear **zero** times in PostgreSQL's grammar. Postgres
+> has no temporal SQL at all, and `AS OF SYSTEM TIME` is a CockroachDB
+> extension — so NEDB's temporal clauses are additions *to* the vendored grammar
+> rather than deviations *from* it. The same road CockroachDB, Materialize and
+> RisingWave took. What the grammar does hand over free: `WITH RECURSIVE`,
+> window functions, `GROUPING SETS` and `MERGE`.
 >
 > ```bash
 > pip install nesql   ·   cargo add nesql   ·   npm install nesql-engine
 > ```
 >
-> Names reserved, grammar vendored, executor foundations already shipping inside
-> this engine. Each package loads and answers `is_release() == false`, because a
-> package that imports cleanly and then lies is worse than one that isn't published.
+> Those three are **reserved names**, not a product: each loads and answers
+> `is_release() == false`, because a package that imports cleanly and then lies
+> is worse than one that isn't published. The engine is what ships today, and
+> neSQL will be this same engine under its own name.
 >
-> **NQL is not being deleted.** It gets folded in: two front-ends compiling to one
-> plan, so nothing translates and neither language is a second-class guest.
+> **NQL is not being deleted, and it is not being wrapped.** Its verbs are SQL
+> clauses now — `AS OF SYSTEM TIME`, `VALID AS OF`, `SEARCH` — parsed by the SQL
+> side and executed by the NQL engine. One implementation, two front-ends,
+> neither one a second-class guest.
 
 Every refusal names the boundary instead of saying "syntax error", and a
 grouped query that projects a column SQL would reject gets Postgres's own

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/README.md

@@ -18,7 +18,7 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 **[Studio → studio.interchained.org](https://studio.interchained.org)**  ·  **[nedb.aiassist.net](https://nedb.aiassist.net)**
 
 > ## 🟢 Free in production under $1M revenue
-> NEDB 4.0.0 is licensed under the **Business Source License 1.1**. If your organisation's annual
+> NEDB is licensed under the **Business Source License 1.1** (since 4.0.0). If your organisation's annual
 > revenue is **under USD $1,000,000**, you may use it in production — commercially, embedded, in
 > closed-source software — with **no permission needed and no royalty**. At **$1M or more**, you
 > need an additional use grant from Interchained LLC: **licensing@interchained.org**.
@@ -38,14 +38,74 @@ endpoint already answers `psql`, SQLAlchemy Core **and** ORM, asyncpg and
 node-postgres against a live store — but it gets there by *translating* SQL into
 NQL, and a translation can only reach as far as the target language's shape.
 
-**[neSQL](https://github.com/Eth-Interchained/neSQL)** removes the translation:
-PostgreSQL's real grammar, vendored with its licence intact, extended with NEDB's
-temporal and causal clauses. Two front-ends, one plan. NQL folded in, not deleted.
+**[neSQL](https://github.com/Eth-Interchained/neSQL)** removes the translation.
+PostgreSQL's real grammar (`gram.y`, 19,513 lines, 492 keywords, vendored from
+17.4 at [`vendor/postgresql/`](vendor/postgresql/) with its licence intact),
+extended with NEDB's temporal and causal clauses. **Two front-ends, one plan.
+NQL folded in, not deleted.**
 
 [![neSQL on PyPI](https://img.shields.io/pypi/v/nesql?label=nesql%20·%20PyPI&color=a855f7)](https://pypi.org/project/nesql/)
 [![neSQL on crates.io](https://img.shields.io/crates/v/nesql?label=nesql%20·%20crates.io&color=a855f7)](https://crates.io/crates/nesql)
 [![neSQL on npm](https://img.shields.io/npm/v/nesql-engine?label=nesql-engine%20·%20npm&color=a855f7)](https://www.npmjs.com/package/nesql-engine)
 
+### Available now, opt-in: `NEDBD_SQL_ENGINE=1`
+
+The SQL engine is in this release and it is **off by default**. Turn it on and a
+user collection is answered by a real SQL evaluator instead of a translation —
+every one of these works, and every one is refused *by name* without it:
+
+```sql
+SELECT o._id, d.name FROM orders o JOIN drivers d ON o.driver = d._id;
+SELECT status, sum(total), avg(total) FROM orders GROUP BY status;
+SELECT _id FROM orders WHERE driver IN (SELECT _id FROM drivers);
+SELECT DISTINCT status FROM orders;
+SELECT _id FROM orders UNION SELECT _id FROM drivers;
+SELECT status, array_agg(_id ORDER BY total DESC) FROM orders GROUP BY status;
+```
+
+`sum(total), avg(total)` in one grouped row is the one worth pointing at. An NQL
+grouped row carries the group key, `count`, and **one** named aggregate — so that
+query was never slow, it was *unrepresentable*. No translator can fix a row
+model, which is the whole reason neSQL exists.
+
+**And NQL's own verbs are now SQL clauses**, so they compose with all of the
+above rather than living on a separate path:
+
+```sql
+-- full-text search from NQL, a join from SQL, one statement
+SELECT o._id, d.name
+  FROM orders SEARCH 'acme' o
+  JOIN drivers d ON o.driver = d._id;
+
+-- one relation in the past, joined against another at the tip
+SELECT h.total, n.total
+  FROM orders AS OF SYSTEM TIME 412 h
+  JOIN audit n ON h._id = n._id;
+
+SELECT _id FROM orders VALID AS OF '2026-01-01';
+```
+
+There is **one implementation** of each verb — the SQL side parses them and the
+NQL engine still executes them — so neither language is a reimplementation of
+the other. `AS OF SYSTEM TIME`, `VALID AS OF` and `SEARCH` are **unreserved
+keywords**: a collection aliased `search`, or a column named `valid`, keeps
+working exactly as before.
+
+**Why it is opt-in rather than the default**, stated plainly because the reason
+is the interesting part. A parity harness runs the same corpus through both
+engines and asserts identical answers — 44 checks, in CI, and it is what earns
+the flag being flipped rather than a benchmark. It already found two real
+divergences: `SELECT *` returned its columns in a different order on each
+engine, and the SQL evaluator built its column list from the **first row alone**,
+so a field only later documents carried silently did not appear at all.
+
+Both are fixed. But 44 checks over six documents proves agreement on the shapes
+we thought to test, and the evaluator still materialises each relation — the
+`WHERE` is pushed into the scan, which narrows *what* is read but not *whether*.
+Flipping the default changes the read path of every existing deployment, and
+"correct on six rows" is not "safe on six million". So it ships as a flag, with
+the bar for changing that written down.
+
 ---
 
 ## New in 3.3.0 — the query language grew up
@@ -200,46 +260,59 @@ all the work in this sentence. Every refusal below traces to the same cause:
 NQL is the engine's native language, so SQL has to be rewritten into it, and a
 rewrite can only ever reach as far as the target language's shape.
 
-| Supported today | Refused, with the reason | neSQL |
+| Supported on the default path | Refused there, with the reason | `NEDBD_SQL_ENGINE=1` |
 | --- | --- | --- |
-| `*`, a column list, `COUNT(*)`, `SUM`/`AVG`/`MIN`/`MAX(col)` | `JOIN` — NQL is single-collection | ✅ joins already exist in the executor |
-| `WHERE` — the whole NQL predicate surface | subqueries, `UNION`, window functions | ✅ subqueries + set ops exist; windows arrive with the grammar |
-| `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`, `OFFSET` | expressions in the select list | ✅ |
-| one named aggregate per grouped row | `sum(x), avg(x)` in one query — NQL's grouped row holds *one* | ✅ the row-model cap goes away |
-| `AS OF SYSTEM TIME <seq>` | DDL, `TRUNCATE`, `GRANT`/`REVOKE` | ⛔️ still refused, and always will be |
-| `INSERT` / `UPDATE` / `DELETE`, all with `RETURNING` | an `INSERT` with no column list | ✅ |
-| `_caused_by` / `_valid_from` / `_valid_to` as INSERT columns | values that are expressions, not literals | ✅ |
+| `*`, a column list, `COUNT(*)`, `SUM`/`AVG`/`MIN`/`MAX(col)` | `JOIN` — NQL is single-collection | ✅ **works** (nested-loop + hash) |
+| `WHERE` — the whole NQL predicate surface | subqueries, `UNION`, window functions | ✅ **subqueries, `EXISTS`, `UNION`/`INTERSECT`/`EXCEPT` work**; window functions arrive with the grammar |
+| `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`, `OFFSET` | expressions in the select list | ✅ **works** |
+| one named aggregate per grouped row | `sum(x), avg(x)` in one query — NQL's grouped row holds *one* | ✅ **works** — the row-model cap is gone |
+| — | `DISTINCT`, `array_agg(x ORDER BY y)`, derived tables | ✅ **works** |
+| `AS OF SYSTEM TIME <seq>`, `VALID AS OF`, `SEARCH` | — | ✅ **also works**, and composes with joins and aggregates |
+| `TRACE`, `TRAVERSE`, `LINK` | — | ↩︎ still answered by the NQL path; SQL has no spelling for them yet |
+| `INSERT` / `UPDATE` / `DELETE`, all with `RETURNING` | an `INSERT` with no column list | writes always take the NQL path |
+| `_caused_by` / `_valid_from` / `_valid_to` as INSERT columns | values that are expressions, not literals | as above |
+| — | DDL, `TRUNCATE`, `GRANT`/`REVOKE` | ⛔️ **refused on both**, and always will be |
 
 The `⛔️` row is the one that is not a limitation. `TRUNCATE` is refused because
 NEDB is append-only *so that history cannot be discarded* — that is the product,
 not a gap — and DDL is refused because collections are created by the first write
 to them. Those answers do not change.
 
-### Every other row on that table is a translation artefact, and it is going away
+### Every other row on that table was a translation artefact — and one flag removes them
 
 > ### 🆕 [**neSQL**](https://github.com/Eth-Interchained/neSQL) — PostgreSQL's grammar, NEDB's memory
 >
-> We stopped translating. neSQL vendors PostgreSQL's **real grammar** — `gram.y`,
-> 19,513 lines and 492 keywords, from PostgreSQL 17.4, licence intact — and extends
-> it with the clauses NEDB needs, rather than rewriting SQL into a language that
-> cannot express it.
+> Those refusals were never the engine's limits. `sqlselect.rs` has had
+> nested-loop and hash joins, subqueries, `EXISTS`, quantified comparisons, set
+> operations, `array_agg(x ORDER BY y)` and derived tables for some time — they
+> were simply unreachable *through a translator*, because the translator's
+> target was NQL. Set `NEDBD_SQL_ENGINE=1` and they are reachable.
+>
+> neSQL vendors PostgreSQL's **real grammar** — `gram.y`, 19,513 lines and 492
+> keywords, from 17.4, licence intact — and extends it with the clauses NEDB
+> needs, rather than rewriting SQL into a language that cannot express it.
 >
-> It is worth knowing *why* this was never free: `SYSTEM_TIME`, `PERIOD` and
-> `PORTION` appear **zero** times in PostgreSQL's grammar. Postgres has no temporal
-> SQL at all. `AS OF SYSTEM TIME` is a CockroachDB extension, which means NEDB's
-> temporal clauses are additions to the vendored grammar rather than deviations
-> from it — the same road CockroachDB, Materialize and RisingWave took.
+> It is worth knowing *why* the temporal clauses were never free: `SYSTEM_TIME`,
+> `PERIOD` and `PORTION` appear **zero** times in PostgreSQL's grammar. Postgres
+> has no temporal SQL at all, and `AS OF SYSTEM TIME` is a CockroachDB
+> extension — so NEDB's temporal clauses are additions *to* the vendored grammar
+> rather than deviations *from* it. The same road CockroachDB, Materialize and
+> RisingWave took. What the grammar does hand over free: `WITH RECURSIVE`,
+> window functions, `GROUPING SETS` and `MERGE`.
 >
 > ```bash
 > pip install nesql   ·   cargo add nesql   ·   npm install nesql-engine
 > ```
 >
-> Names reserved, grammar vendored, executor foundations already shipping inside
-> this engine. Each package loads and answers `is_release() == false`, because a
-> package that imports cleanly and then lies is worse than one that isn't published.
+> Those three are **reserved names**, not a product: each loads and answers
+> `is_release() == false`, because a package that imports cleanly and then lies
+> is worse than one that isn't published. The engine is what ships today, and
+> neSQL will be this same engine under its own name.
 >
-> **NQL is not being deleted.** It gets folded in: two front-ends compiling to one
-> plan, so nothing translates and neither language is a second-class guest.
+> **NQL is not being deleted, and it is not being wrapped.** Its verbs are SQL
+> clauses now — `AS OF SYSTEM TIME`, `VALID AS OF`, `SEARCH` — parsed by the SQL
+> side and executed by the NQL engine. One implementation, two front-ends,
+> neither one a second-class guest.
 
 Every refusal names the boundary instead of saying "syntax error", and a
 grouped query that projects a column SQL would reject gets Postgres's own

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/pyproject.toml

@@ -10,7 +10,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "nedb-engine"
-version = "4.3.2"
+version = "6.0.0"
 description = "NEDB — a versioned, self-compressing, time-traveling embedded database (replay-protected, idempotent, relational, searchable) with durable AOF persistence and a server daemon (nedbd)."
 readme = "README.md"
 requires-python = ">=3.8"

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/python/nedb/__init__.py

@@ -106,4 +106,4 @@ __all__ = [
     "verify_proof", "fold_head",
     "_native", "__has_native__",
 ]
-__version__ = "4.3.2"
+__version__ = "6.0.0"

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/python/nedb/engine.py

@@ -18,6 +18,7 @@ from typing import Any, Dict, List, Optional
 
 from .cascade import BlobStore
 from . import snapshot as _snap
+from . import namespace as _ns
 from . import crypto as _crypto
 from .index import Indexes, tokenize
 from .log import Op, OpLog, ReplayError, GENESIS, blake, canon  # noqa: F401  (re-exported)
@@ -117,6 +118,10 @@ class NEDB:
         # caller (the concurrent Sequencer) issues ONE fsync per batch via flush()
         # — group commit. Default False keeps embedded/direct use durable per-op.
         self._defer_sync = False
+        # Collections already known to be registered. A CACHE, never the
+        # answer: `collections()` reads the registry. A stale cache can only
+        # cause a redundant registry check, never a wrong namespace.
+        self._known_collections: set = set()
         # Encryption: resolve TMK (arg > env) → load/create DEK → None if no TMK
         self._dek: Optional[bytes] = None
         resolved_tmk = _crypto.resolve_tmk(tmk)
@@ -472,6 +477,106 @@ class NEDB:
             return False         # already expired
         return True
 
+    # --- collection identity ------------------------------------------------
+    #
+    # Mirrors rust/nedb-v2/src/db.rs. See python/nedb/namespace.py for why the
+    # reference engine carries this too, and for the parity failure that made
+    # it necessary.
+
+    def _ensure_collection(self, coll: str) -> None:
+        """Record that a collection exists, if that is not already recorded.
+
+        Idempotent, and cheap after a collection's first write: a set hit. On a
+        miss it consults the registry itself before writing, so reopening a
+        database does not re-register everything in it.
+
+        ORDER MATTERS. The registry record is appended BEFORE the user's write,
+        exactly as `Db::put` does it in Rust — `validate` then `ensure` then the
+        write. The two engines assign the same sequence numbers to the same
+        writes only if they spend them in the same order, and the cross-engine
+        parity corpus compares absolute sequences.
+        """
+        if coll in self._known_collections:
+            return
+        existing = self.store.get(f"{_ns.COLLECTIONS}:{coll}")
+        if existing is not None:
+            # Registered already. Revive it if it was dropped and is being
+            # written to again — a write is an unambiguous assertion that the
+            # caller means for this collection to exist.
+            if existing.get("dropped"):
+                self._write_collection_record(coll, False)
+            self._known_collections.add(coll)
+            return
+        self._write_collection_record(coll, False)
+        self._known_collections.add(coll)
+
+    def _write_collection_record(self, coll: str, dropped: bool) -> None:
+        """Append a registry record.
+
+        Creation and drop are the same shape because they are the same kind of
+        event: an assertion, at a sequence, about whether a name is live. Goes
+        through the ordinary log so replication, AS OF and verification cover
+        it without being taught about it.
+        """
+        key = f"{_ns.COLLECTIONS}:{coll}"
+        doc = {"_id": coll, "name": coll, "dropped": dropped, "at_seq": self.seq}
+        nonce = self._next("engine")
+        op, created = self._log_append(
+            "engine", nonce, "put",
+            {"key": key, "coll": _ns.COLLECTIONS, "id": coll, "doc": doc},
+        )
+        if created:
+            apply_op(self.store, self.relations, self.indexes, op, self.cause_map)
+
+    def collections(self) -> List[str]:
+        """Every collection that currently exists.
+
+        THE authoritative answer. An emptied-but-created collection is present
+        here; that is the whole point. Derived from recorded events rather than
+        from which document keys happen to be live, so a collection does not
+        cease to exist because its last row was deleted.
+        """
+        live = []
+        prefix = f"{_ns.COLLECTIONS}:"
+        for key in self.store.keys(prefix):
+            rec = self.store.get(key)
+            if rec is None or rec.get("dropped"):
+                continue
+            name = rec.get("name")
+            if name is not None and not _ns.is_reserved(name):
+                live.append(name)
+        return sorted(live)
+
+    def collections_as_of(self, as_of: int) -> List[str]:
+        """Which collections existed as of a sequence. Versioned for free,
+        because the registry is ordinary documents in the log."""
+        live = []
+        prefix = f"{_ns.COLLECTIONS}:"
+        for key in self.store.keys(prefix):
+            rec = self.store.get(key, as_of)
+            if rec is None or rec.get("dropped"):
+                continue
+            name = rec.get("name")
+            if name is not None and not _ns.is_reserved(name):
+                live.append(name)
+        return sorted(live)
+
+    def drop_collection(self, coll: str) -> bool:
+        """Drop a collection: record that the name is no longer live.
+
+        A TOMBSTONE, not an erasure — the same contract `delete` has for
+        documents. Documents are left where they are; quietly destroying
+        history behind a namespace operation is exactly the behaviour the
+        engine refuses to have.
+        """
+        _ns.validate_writable(coll)
+        rec = self.store.get(f"{_ns.COLLECTIONS}:{coll}")
+        if rec is None or rec.get("dropped"):
+            return False
+        self._write_collection_record(coll, True)
+        self._known_collections.discard(coll)
+        return True
+
     # --- mutations ----------------------------------------------------------
     def put(self, coll: str, id: str, doc: dict, client: str = "local",
             nonce: Optional[int] = None, idem: Optional[str] = None,
@@ -481,6 +586,8 @@ class NEDB:
             confidence: Optional[float] = None,
             valid_from: Optional[str] = None,
             valid_to: Optional[str] = None) -> dict:
+        _ns.validate_writable(coll)
+        self._ensure_collection(coll)
         key = f"{coll}:{id}"
         doc = dict(doc)
         doc.setdefault("_id", id)
@@ -504,6 +611,9 @@ class NEDB:
 
     def delete(self, coll: str, id: str, client: str = "local",
                nonce: Optional[int] = None, idem: Optional[str] = None) -> None:
+        # Validated but NOT registered — deleting from a collection that does
+        # not exist should not bring it into existence. Matches `Db::delete`.
+        _ns.validate_writable(coll)
         key = f"{coll}:{id}"
         nonce = self._next(client) if nonce is None else nonce
         op, created = self._log_append(client, nonce, "delete",

nedb_engine-6.0.0/python/nedb/namespace.py

@@ -0,0 +1,126 @@
+# SPDX-FileCopyrightText: 2026 INTERCHAINED LLC
+# SPDX-License-Identifier: BUSL-1.1
+# NEDB · © 2026 INTERCHAINED LLC × Eth-Interchained × Vex (Claude Opus 5)
+
+"""Collection identity for the Python reference engine.
+
+This mirrors `rust/nedb-v2/src/namespace.rs` rule for rule, and it exists for
+the same reason the Rust one does: a collection has to exist because a record
+says so, not because storage happens to have something lying around.
+
+# Why the reference engine needs this too
+
+NEDB ships two independent NQL implementations. When the Rust engine started
+registering collections, the two stopped agreeing about sequence numbers — the
+same writes into a fresh database landed one position apart, because Rust spent
+a sequence on the registry record and Python did not. The cross-engine parity
+corpus caught it immediately:
+
+    parity after a delete: FROM t AS OF 1
+        python [(('_id','a'), ('t','66'))]
+        rust   [(('_id','a'), ('t','55'))]
+
+Rust at `AS OF 1` was Python at `AS OF 0`. The correct response was not to move
+the test's goalposts. If collection existence is a logical fact in the engine,
+it is a logical fact in the reference, and the two agree again because both are
+right rather than because one stopped looking.
+
+The Python engine had the same underlying defect in a different shape, too:
+collections were derived by splitting live document keys, so a collection whose
+last document was deleted simply ceased to exist — indistinguishable from one
+that had never been created.
+"""
+
+RESERVED_PREFIX = "_nedb"
+
+#: The collection registry. Ids are collection names; the newest version of
+#: each says whether that collection is currently live.
+COLLECTIONS = "_nedb.collections"
+
+#: Engine metadata that is neither a collection record nor a root.
+META = "_nedb.meta"
+
+#: Persisted state roots.
+ROOTS = "_nedb.roots"
+
+MAX_NAME_BYTES = 255
+
+
+class ReservedCollection(ValueError):
+    """A write was aimed at the engine's own namespace."""
+
+
+class InvalidCollectionName(ValueError):
+    """A name a collection cannot durably have."""
+
+
+def is_reserved(coll: str) -> bool:
+    """Is this name part of the engine's own namespace?"""
+    return coll == RESERVED_PREFIX or coll.startswith(RESERVED_PREFIX + ".")
+
+
+def refuse_reserved(coll: str) -> None:
+    """Refuse a write the caller is not allowed to make."""
+    if is_reserved(coll):
+        raise ReservedCollection(
+            f"collection {coll!r} is reserved: everything under "
+            f"{RESERVED_PREFIX!r} is engine-owned, and letting a client write "
+            f"there would let it forge the namespace the state root commits to"
+        )
+
+
+def validate_name(coll: str) -> None:
+    """Is this a name a collection can durably HAVE?
+
+    Refuses rather than sanitises. A silently rewritten name is a different
+    collection than the one the caller asked for, and they would never be told.
+
+    The Python engine keys documents as ``coll:id``, so a colon in a collection
+    name would make ``a:b`` and ``a`` + id ``b`` the same key — an aliasing bug
+    the Rust engine cannot have, since it keys by a pair. Refused here for the
+    same reason a path separator is refused there: the name has to survive its
+    own storage layer intact.
+    """
+    if not isinstance(coll, str):
+        raise InvalidCollectionName(
+            f"collection name must be a string, got {type(coll).__name__}"
+        )
+    if coll == "":
+        raise InvalidCollectionName("collection name is empty")
+    n = len(coll.encode("utf-8"))
+    if n > MAX_NAME_BYTES:
+        raise InvalidCollectionName(
+            f"collection name is {n} bytes; the limit is {MAX_NAME_BYTES}"
+        )
+    if coll in (".", ".."):
+        raise InvalidCollectionName(
+            f"collection name {coll!r} is a filesystem path component, not a name"
+        )
+    if "/" in coll or "\\" in coll:
+        raise InvalidCollectionName(
+            f"collection name {coll!r} contains a path separator — on disk a "
+            f"collection name is a directory name, so this does not name a "
+            f"collection, it names a location"
+        )
+    if ":" in coll:
+        raise InvalidCollectionName(
+            f"collection name {coll!r} contains a colon, which is the key "
+            f"separator — 'a:b' and collection 'a' id 'b' would be one key"
+        )
+    if "\0" in coll:
+        raise InvalidCollectionName("collection name contains a NUL byte")
+    if coll != coll.strip():
+        raise InvalidCollectionName(
+            f"collection name {coll!r} has leading or trailing whitespace"
+        )
+
+
+def validate_writable(coll: str) -> None:
+    """A name that may be written to: valid AND not engine-owned."""
+    validate_name(coll)
+    refuse_reserved(coll)
+
+
+def seq_id(seq: int) -> str:
+    """Zero-padded so lexicographic ordering is numeric ordering."""
+    return f"{seq:020d}"

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/python/nedb/server.py

@@ -61,6 +61,7 @@ from typing import Any, Dict, List, Optional
 from urllib.parse import urlparse, parse_qs, unquote
 
 from . import __version__
+from . import namespace as _ns
 from .engine import NEDB
 from .concurrent import Sequencer
 from .log import ReplayError
@@ -180,9 +181,27 @@ class Manager:
 
     @staticmethod
     def collection_counts(db: NEDB) -> Dict[str, int]:
-        counts: Dict[str, int] = {}
+        """Live row counts per collection, for the public surface.
+
+        Two things this must get right, and the second one is why it is not a
+        one-line key split any more.
+
+        Engine-owned collections are EXCLUDED. `_nedb.*` is bookkeeping; it
+        leaked into `/v1/databases/<name>` as a collection with rows, which
+        both named an internal detail publicly and inflated the row count. A
+        deploy test comparing seeded rows against reported rows is what caught
+        it.
+
+        And the collection LIST comes from the registry rather than from which
+        document keys happen to be live, so a collection that was created and
+        then emptied still reports — with zero rows. It exists; it is empty.
+        Those are different facts and the API should not merge them.
+        """
+        counts: Dict[str, int] = {c: 0 for c in db.collections()}
         for key in db.store.keys(""):
             coll = key.split(":", 1)[0]
+            if _ns.is_reserved(coll):
+                continue
             counts[coll] = counts.get(coll, 0) + 1
         return counts
 

nedb_engine-6.0.0/python/nedb/state_root.py

@@ -0,0 +1,225 @@
+# SPDX-FileCopyrightText: 2026 INTERCHAINED LLC
+# SPDX-License-Identifier: BUSL-1.1
+# NEDB · © 2026 INTERCHAINED LLC × Eth-Interchained × Vex (Claude Opus 5)
+
+"""state_root_v1 — the Python reference implementation.
+
+This is deliberately a SECOND implementation, written from the format rules
+rather than translated from the Rust. Its whole job is to disagree if the
+format has a hole in it: a spec that only one program implements is not a
+spec, it is that program's behaviour written down.
+
+The rules it implements, in full:
+
+  hash            BLAKE2b-512, first 32 bytes
+  domains         every hash input starts with a distinct ASCII tag
+  lengths         every variable-length field is preceded by u64 little-endian
+  optionals       one presence byte, then the value if present
+  ordering        leaves sorted by key BYTES (collection, then id)
+  unicode         none applied; names commit as the exact UTF-8 they arrived as
+  odd leaf        promoted unchanged, never duplicated (CVE-2012-2459)
+  leaf count      committed in the subtree root alongside the fold
+  empty           H(tag), a constant, never zero
+  field order     documents commit in their own key order, not sorted
+
+Run `python3 -m nedb.state_root` to check against the committed vectors.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import struct
+from typing import Any, Iterable, Mapping, Optional, Sequence
+
+TAG_EMPTY = b"nedb:state_root_v1:empty"
+TAG_NODE = b"nedb:state_root_v1:node"
+TAG_NS_LEAF = b"nedb:state_root_v1:namespace_leaf"
+TAG_NS_ROOT = b"nedb:state_root_v1:namespace_root"
+TAG_REC_LEAF = b"nedb:state_root_v1:record_leaf"
+TAG_REC_ROOT = b"nedb:state_root_v1:records_root"
+TAG_STATE_ROOT = b"nedb:state_root_v1:state_root"
+
+V_NULL, V_FALSE, V_TRUE, V_I64, V_U64, V_F64, V_STR, V_ARR, V_OBJ = range(9)
+
+I64_MIN, I64_MAX = -(2**63), 2**63 - 1
+U64_MAX = 2**64 - 1
+
+
+def _h(*parts: bytes) -> bytes:
+    d = hashlib.blake2b()
+    for p in parts:
+        d.update(p)
+    return d.digest()[:32]
+
+
+def _lp(b: bytearray, raw: bytes) -> None:
+    b += struct.pack("<Q", len(raw))
+    b += raw
+
+
+def _lp_opt(b: bytearray, s: Optional[str]) -> None:
+    if s is None:
+        b.append(0)
+    else:
+        b.append(1)
+        _lp(b, s.encode("utf-8"))
+
+
+def encode_value(b: bytearray, v: Any) -> None:
+    """Canonical encoding of a JSON value.
+
+    Numbers commit by REPRESENTATION, not by mathematical value, so 1 and 1.0
+    differ. Python makes that distinction naturally (int vs float); the Rust
+    side gets it from serde_json's parsed number type. That the two agree is
+    not automatic, which is exactly why it is in the vectors.
+    """
+    if v is None:
+        b.append(V_NULL)
+    elif v is True:
+        b.append(V_TRUE)
+    elif v is False:
+        b.append(V_FALSE)
+    elif isinstance(v, int):
+        # bool is a subclass of int, handled above.
+        if I64_MIN <= v <= I64_MAX:
+            b.append(V_I64)
+            b += struct.pack("<q", v)
+        elif 0 <= v <= U64_MAX:
+            b.append(V_U64)
+            b += struct.pack("<Q", v)
+        else:
+            raise ValueError(f"integer out of 64-bit range: {v}")
+    elif isinstance(v, float):
+        if v != v:
+            raise ValueError("NaN cannot be committed to a state root")
+        b.append(V_F64)
+        # -0.0 == 0.0, so they must commit identically.
+        b += struct.pack("<d", 0.0 if v == 0.0 else v)
+    elif isinstance(v, str):
+        b.append(V_STR)
+        _lp(b, v.encode("utf-8"))
+    elif isinstance(v, (list, tuple)):
+        b.append(V_ARR)
+        b += struct.pack("<Q", len(v))
+        for it in v:
+            encode_value(b, it)
+    elif isinstance(v, Mapping):
+        b.append(V_OBJ)
+        b += struct.pack("<Q", len(v))
+        # Insertion order, NOT sorted. Python dicts preserve it; the vectors
+        # are loaded with object_pairs_hook=dict to keep the file's order.
+        for k, val in v.items():
+            _lp(b, k.encode("utf-8"))
+            encode_value(b, val)
+    else:
+        raise TypeError(f"not a JSON value: {type(v).__name__}")
+
+
+def namespace_leaf(name: str) -> bytes:
+    b = bytearray()
+    _lp(b, name.encode("utf-8"))
+    return _h(TAG_NS_LEAF, bytes(b))
+
+
+def record_leaf(
+    coll: str,
+    id_: str,
+    data: Any,
+    valid_from: Optional[str] = None,
+    valid_to: Optional[str] = None,
+) -> bytes:
+    b = bytearray()
+    _lp(b, coll.encode("utf-8"))
+    _lp(b, id_.encode("utf-8"))
+    _lp_opt(b, valid_from)
+    _lp_opt(b, valid_to)
+    encode_value(b, data)
+    return _h(TAG_REC_LEAF, bytes(b))
+
+
+def _fold(level: list[bytes]) -> bytes:
+    if not level:
+        return _h(TAG_EMPTY)
+    while len(level) > 1:
+        nxt = []
+        for i in range(0, len(level) - 1, 2):
+            nxt.append(_h(TAG_NODE, level[i], level[i + 1]))
+        if len(level) % 2:
+            nxt.append(level[-1])  # promote; duplicating is CVE-2012-2459
+        level = nxt
+    return level[0]
+
+
+def _subtree(tag: bytes, leaves: list[bytes]) -> bytes:
+    return _h(tag, struct.pack("<Q", len(leaves)), _fold(leaves))
+
+
+def namespace_root(collections: Iterable[str]) -> bytes:
+    names = sorted(set(collections), key=lambda s: s.encode("utf-8"))
+    return _subtree(TAG_NS_ROOT, [namespace_leaf(n) for n in names])
+
+
+def records_root(records: Sequence[Mapping[str, Any]]) -> bytes:
+    ordered = sorted(
+        records,
+        key=lambda r: (r["coll"].encode("utf-8"), r["id"].encode("utf-8")),
+    )
+    leaves = [
+        record_leaf(
+            r["coll"], r["id"], r["data"],
+            r.get("valid_from"), r.get("valid_to"),
+        )
+        for r in ordered
+    ]
+    return _subtree(TAG_REC_ROOT, leaves)
+
+
+def compute(collections: Iterable[str], records: Sequence[Mapping[str, Any]]) -> dict:
+    colls = list(collections)
+    ns = namespace_root(colls)
+    rec = records_root(records)
+    return {
+        "version": "state_root_v1",
+        "namespace_root": ns.hex(),
+        "records_root": rec.hex(),
+        "state_root": _h(TAG_STATE_ROOT, ns, rec).hex(),
+        "collection_count": len(set(colls)),
+        "record_count": len(records),
+    }
+
+
+# ── Vector check ──────────────────────────────────────────────────────────
+
+def check_vectors(path: str) -> tuple[int, list[str]]:
+    """Returns (cases checked, failures). Never raises on a mismatch — the
+    caller decides what a mismatch means, and a mismatch is data, not a crash.
+    """
+    with open(path, encoding="utf-8") as fh:
+        doc = json.load(fh)
+    failures = []
+    for case in doc["cases"]:
+        got = compute(case["collections"], case["records"])
+        want = case["expect"]
+        for field in ("namespace_root", "records_root", "state_root",
+                      "collection_count", "record_count"):
+            if got[field] != want[field]:
+                failures.append(
+                    f"{case['name']}.{field}: python={got[field]!r} "
+                    f"rust={want[field]!r}"
+                )
+    return len(doc["cases"]), failures
+
+
+if __name__ == "__main__":
+    import os
+    import sys
+
+    here = os.path.dirname(os.path.abspath(__file__))
+    default = os.path.join(here, "..", "..", "vectors", "state_root_v1.json")
+    target = sys.argv[1] if len(sys.argv) > 1 else default
+    n, bad = check_vectors(target)
+    for line in bad:
+        print("MISMATCH", line)
+    print(f"{n - len({b.split('.')[0] for b in bad})}/{n} cases agree with Rust")
+    sys.exit(1 if bad else 0)

{nedb_engine-4.3.2 → nedb_engine-6.0.0}/python/nedb/wrap_redis.py

@@ -319,6 +319,11 @@ class NEDBSurface:
         # raise AttributeError on every install lacking the native wheel,
         # i.e. exactly the universal py3-none-any path.
         self._nedbd_mode: bool = False
+        # How many log ops the backend has already been handed. Set before any
+        # branch can call _reload(), because the early-return path (an empty
+        # stream) never reaches the assignment there — and an unset mark would
+        # be an AttributeError on the first write of a brand-new database.
+        self._persisted_ops: int = 0
 
         if nedbd_url:
             # Route all NEDB operations to a running nedbd server
@@ -383,16 +388,40 @@ class NEDBSurface:
                     apply_op(self._db.store, self._db.relations,
                              self._db.indexes, op, self._db.cause_map)
             self._db._nonce = dict(self._db.log._last_nonce)
+        # Whatever was just restored is already in the stream. Without this the
+        # next write would re-append the entire replayed history.
+        self._persisted_ops = len(self._db.log.ops) if hasattr(self._db, "log") else 0
 
     def _persist_last_op(self) -> None:
+        """Persist every op appended since the last call.
+
+        It used to persist only `ops[-1]`, on the assumption that one API
+        call appends exactly one op. That assumption broke the moment a
+        single `put` could append two — a collection's first write now emits
+        its registry record and then the document — and the failure was
+        silent in the worst way: the registry op was never written to the
+        stream, so a replay rebuilt a log whose `prev` links pointed at an op
+        that was not there, and `verify()` returned False on a database that
+        had never been touched by anything but NEDB itself.
+
+        A high-water mark instead of a count, so this stays correct no matter
+        how many ops any future engine operation appends. The mark only ever
+        advances over ops actually handed to the backend, so a failure part
+        way through leaves the rest to be retried rather than skipped.
+        """
         if self._nedbd_mode:
             return  # nedbd persists atomically on each HTTP call
         if not hasattr(self._db, "log"):
             return  # DAG backend persists itself (WAL + MANIFEST, atexit flush)
-        if self._db.log.ops:
-            last = self._db.log.ops[-1]
-            self._backend.append(json.dumps(last.to_dict()))
-            self._backend.publish_ops([json.dumps(last.to_dict())])
+        total = len(self._db.log.ops)
+        if total <= self._persisted_ops:
+            return
+        pending = self._db.log.ops[self._persisted_ops:total]
+        encoded = [json.dumps(op.to_dict()) for op in pending]
+        for line in encoded:
+            self._backend.append(line)
+        self._backend.publish_ops(encoded)
+        self._persisted_ops = total
 
     # ── Collection registration ───────────────────────────────────────────────