nedb-engine 3.2.1 → 3.3.0

package/README.md

@@ -26,6 +26,78 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 
 ---
 
+## New in 3.3.0 — the query language grew up
+
+`WHERE` was six operators wide (`= != > < >= <=`) joined by an implicit `AND`.
+It now takes a full boolean expression, in **both** engines, and the clauses
+around it run in SQL's order.
+
+```sql
+FROM jobs
+WHERE (status IN ("open", "pending") OR fee > 100)
+  AND miner IS NOT NULL
+  AND NOT (region LIKE "eu-%")
+GROUP BY region SUM fee
+HAVING sum_fee > 10000
+ORDER BY sum_fee DESC
+LIMIT 20 OFFSET 40
+```
+
+| added | |
+| --- | --- |
+| `IN (…)` / `NOT IN (…)` | set membership |
+| `BETWEEN a AND b` / `NOT BETWEEN` | inclusive both ends, as in SQL |
+| `LIKE` / `NOT LIKE` / `ILIKE` | `%` any run, `_` any one char |
+| `IS NULL` / `IS NOT NULL` | matches absent **and** explicitly-null |
+| `AND` / `OR` / `NOT` / `(…)` | `AND` binds tighter; parens nest to any depth |
+| `OFFSET n` | pagination; pairs with `LIMIT` |
+| `ORDER BY a, b DESC` | multi-key, per-key direction |
+| `HAVING <predicate>` | filters the aggregated rows |
+| `COUNT` / `SUM f` / `AVG f` / `MIN f` / `MAX f` | whole-result aggregate, one row |
+
+**Indexed range scans.** `=`, `IN`, `BETWEEN` and the inequalities are served
+from a sorted index when one covers the field — a point lookup on 20,000 rows
+goes from 137 ms to 0.01 ms, a 1%-selective `BETWEEN` from 186 ms to 1.1 ms.
+See [**Indexes**](#indexes) for the measured table and the three cases that
+deliberately decline the index.
+
+**Nine silent defects fixed.** None of them crashed; they all returned a
+confident wrong answer with HTTP 200. The worst:
+
+- `GROUP BY status MAX fee` aggregated the *group* field, not the target —
+  answering `1.0` where the real maxima were 40 and 30.
+- `LIMIT 2 GROUP BY status COUNT` truncated the aggregate's **input**, so
+  twelve rows across three statuses reported counts summing to 2.
+- `ORDER BY count DESC` on grouped rows sorted the raw documents on a field
+  that only exists after grouping — silently inert.
+- `FROM jobs OFFSET 2` and a misspelled `ORDRE BY fee` were silently **dropped**
+  and a different query answered. Unknown clauses are now a parse error.
+- `SUM` ran through `f64`, losing integer precision above 2^53 — a real problem
+  for satoshi amounts and block heights. Integer inputs now stay in `i64`.
+- An ordering comparison against a missing field was true in the Rust engine
+  (`WHERE fee < 5` returned rows with no `fee` at all) and false in the Python
+  reference. Now false in both, matching SQL.
+- A field named like a keyword (`count`, `min`, `value`, `status`) was
+  unaddressable, because both lexers canonicalised case at field positions.
+
+**Compatibility.** Verified by building a `nedbd` from the released v3.2.2 tag
+and diffing every answer: **40 of 45 legacy queries byte-identical, zero
+regressions.** The differences are the three bug fixes above, each documented
+in `tests/test_backcompat.py`. `scripts/compare_engine_answers.py` reproduces
+the comparison against any released binary.
+
+**The DAG is untouched.** `tests/test_dag_preserved.py` runs the entire new
+query surface against a live chain and asserts head, seq and `verify()` are
+unchanged afterwards — and that `verify()` still returns *false* when the log
+is tampered with. Reads are reads.
+
+**Cross-engine parity is now gated.** Nothing previously checked that the
+Python reference and the Rust core agreed, which is how they had drifted apart
+in five places. Two suites now run the same battery through both and assert
+identical answers.
+
+---
+
 ## New in 3.2.0 — wrap the databases you already run
 
 NEDB adds **tamper-evident causal provenance to a database you already have**, in one line, without
@@ -271,6 +343,12 @@ db.query('FROM users WHERE status = "active" ORDER BY age ASC')
 db.query('FROM users SEARCH "rust"')
 db.query('FROM users GROUP BY status COUNT')
 
+# Full boolean predicates — IN, BETWEEN, LIKE, IS NULL, OR, NOT, parentheses
+db.query('FROM users WHERE status IN ("active", "trialing")')
+db.query('FROM users WHERE age BETWEEN 25 AND 40')
+db.query('FROM users WHERE bio LIKE "%rust%" AND NOT (status = "retired")')
+db.query('FROM users WHERE (age < 25 OR age > 60) AND bio IS NOT NULL')
+
 # Time-travel — AS OF any past sequence
 snap = db.seq
 db.put("users", "alice", {"name": "Alice", "age": 32, "status": "retired"})
@@ -495,13 +573,185 @@ redis-cli -p 6380 SELECT shop EVAL 'FROM beliefs TRACE caused_by' 0
 FROM <collection>
   [ AS OF <seq> ]                            transaction time (when was it written?)
   [ VALID AS OF "<date>" ]                   valid time (when was it true in the world?)
-  [ WHERE <field> <op> <value> (AND ...) ]   op: = != < <= > >=
+  [ WHERE <predicate> ]                      full boolean predicate, see below
   [ SEARCH "<text>" ]                        full-text search
-  [ ORDER BY <field> [ASC|DESC] ]
   [ TRAVERSE <relation> ]                    graph traversal
   [ TRACE caused_by [REVERSE] ]              causal provenance (why? / what did this cause?)
-  [ LIMIT <n> ]
   [ GROUP BY <field> [COUNT|SUM f|AVG f|MIN f|MAX f] ]
+  [ COUNT | SUM f | AVG f | MIN f | MAX f ]  whole-result aggregate, one row
+  [ HAVING <predicate> ]                     filters the AGGREGATED rows
+  [ ORDER BY <field> [ASC|DESC] (, ...) ]
+  [ LIMIT <n> ] [ OFFSET <n> ]
+```
+
+Clauses are evaluated in SQL's order, whatever order you write them in:
+
+```
+FROM → WHERE → GROUP BY → HAVING → ORDER BY → OFFSET → LIMIT
+```
+
+That matters, and before 3.3.0 it was wrong. `LIMIT` truncated the *input* to
+an aggregate rather than the result, so `LIMIT 5 GROUP BY status COUNT` over
+twelve rows reported counts summing to 5 — it said only five rows existed when
+twelve did. `ORDER BY` ran before grouping, so sorting on `count` or `sum_fee`
+silently did nothing. `VALID AS OF` was applied after `LIMIT` in the Python
+engine, so a limited bi-temporal query returned fewer valid rows than exist.
+
+### Predicates
+
+`WHERE` takes a full boolean expression. `AND` binds tighter than `OR`;
+parentheses override, and nest to any depth.
+
+```
+<predicate> := <or>
+<or>        := <and> [OR <and>]*
+<and>       := <not> [AND <not>]*
+<not>       := [NOT] <primary>
+<primary>   := "(" <predicate> ")" | <comparison>
+```
+
+| Comparison | Example |
+| --- | --- |
+| `= != < <= > >=` | `WHERE height > 600000` |
+| `IN (…)` / `NOT IN (…)` | `WHERE status IN ("open", "pending")` |
+| `BETWEEN a AND b` | `WHERE height BETWEEN 100 AND 200` — inclusive, as in SQL |
+| `NOT BETWEEN a AND b` | `WHERE fee NOT BETWEEN 10 AND 20` |
+| `LIKE` / `NOT LIKE` | `WHERE miner LIKE "Acme%"` — `%` any run, `_` any one char |
+| `ILIKE` | `WHERE miner ILIKE "acme%"` — case-insensitive |
+| `IS NULL` / `IS NOT NULL` | `WHERE miner IS NULL` — matches absent **and** explicitly-null |
+
+```python
+db.query('''FROM jobs
+            WHERE (status IN ("open", "pending") OR fee > 100)
+              AND miner IS NOT NULL
+              AND NOT (region LIKE "eu-%")
+            ORDER BY fee DESC LIMIT 20''')
+```
+
+Filterable metadata fields: `_id`, `_coll`, `_hash`, `_seq`.
+
+`_id = "x"` is an O(1) index lookup rather than a scan — but only when it is a
+genuine conjunct. Under an `OR` it cannot constrain the result set, so the
+planner correctly declines the fast path there.
+
+### Indexes
+
+`=`, `IN (...)`, `BETWEEN` and the one-sided inequalities are served from a
+sorted index when one covers the field, turning a full collection scan into a
+bounded range walk:
+
+```python
+db.create_index("blocks", "height", "sorted")
+db.query("FROM blocks WHERE height BETWEEN 600000 AND 600100")
+```
+
+Measured on 20,000 rows with `scripts/bench_index_range.py` — two identical
+databases, one indexed, one not:
+
+| Query | Scan | Indexed | Speedup |
+| --- | --- | --- | --- |
+| `WHERE fee = 10000` | 137 ms | 0.01 ms | 17,000× |
+| `WHERE fee IN (a, b, c)` | 185 ms | 0.02 ms | 9,700× |
+| `WHERE fee BETWEEN …` (1% of rows) | 186 ms | 1.1 ms | 170× |
+| `WHERE fee BETWEEN …` (10% of rows) | 188 ms | 13 ms | 14× |
+| unindexed field (control) | 188 ms | 187 ms | 1.0× |
+
+The planner asks the index how many rows each candidate range covers and takes
+the narrowest, so `WHERE region = "eu" AND height BETWEEN 600000 AND 600001`
+uses the height index rather than whichever field it saw first. Same-field
+bounds are merged, so `height > 100 AND height < 200` is one walk.
+
+An index is only ever used to NARROW candidates — the full predicate is
+re-evaluated on whatever comes back, so the answer never depends on whether an
+index exists. Three cases deliberately decline it:
+
+- **`IS NULL`.** A document whose field is absent is not in that field's
+  index, so an index scan would return the exact complement of the answer.
+- **Anything under `OR` or `NOT`.** A disjunct does not constrain the result
+  set; narrowing on one arm would silently drop the rows the other arm matched.
+- **`AS OF`.** The sorted index holds current versions only (a superseded hash
+  is dropped on overwrite), so it cannot answer a historical query.
+
+> **Predicates over NULL follow SQL's three-valued logic.** An ordering
+> comparison (`<` `<=` `>` `>=`, and therefore `BETWEEN`) against a missing or
+> null field is never true — `WHERE fee < 5` will not return a row that has no
+> `fee` at all. `LIKE` is false in *both* polarities, so a null row appears in
+> neither `LIKE` nor `NOT LIKE`. `=` and `!=` do operate on null, so
+> `WHERE fee = NULL` selects rows where the field is absent or null, and
+> `WHERE fee != 5` includes them. Use `IS NULL` / `IS NOT NULL` to test
+> presence explicitly.
+
+### Unknown clauses are errors
+
+A query containing a clause the engine does not implement is **rejected**, not
+silently reinterpreted. Before 3.3.0 the parser skipped tokens it did not
+recognise, so `FROM jobs OFFSET 2` returned un-offset rows with HTTP 200 and a
+misspelled `ORDRE BY fee` returned unsorted rows — the engine answered a
+*different query* than the one asked, and said nothing. Both now return
+HTTP 400 with the offending token.
+
+### Aggregates
+
+`SUM`/`AVG`/`MIN`/`MAX` take the field to aggregate; `COUNT` (the default when
+no aggregate is given) takes none.
+
+```python
+db.query('FROM items GROUP BY cat MAX price')
+# → [{"cat": "x", "count": 3, "max_price": 10.0, "value": 10.0}, …]
+```
+
+`count` is the group size. The aggregate only considers rows whose target field
+is numeric, so a group of 5 where 2 carry a numeric `price` reports `count: 5`
+and averages over 2. An aggregate with no numeric input is `null`, never `0`.
+The `value` key is a back-compatible alias for the aggregate result.
+
+Integer inputs give integer results — `SUM`/`MIN`/`MAX` stay in 64-bit
+integers rather than passing through a float, so a sum over satoshi amounts or
+block heights above 2^53 is exact. `AVG` is always fractional.
+
+Groups come back sorted by key unless you say otherwise, so results are stable
+run to run and identical across engines.
+
+Drop the `GROUP BY` for a whole-result aggregate, which returns exactly one row:
+
+```python
+db.query('FROM orders COUNT')                    # → [{"count": 1049, "value": 1049}]
+db.query('FROM orders WHERE status = "paid" COUNT')
+db.query('FROM orders SUM total')                # → [{"count": 1049, "sum_total": 88123, …}]
+```
+
+`COUNT` of an empty result is one row holding `0` — a caller asking "how many?"
+always gets a number. `SUM` of an empty result is `null`.
+
+### HAVING
+
+`WHERE` filters rows before they are grouped; `HAVING` filters the groups.
+
+```python
+db.query('FROM orders GROUP BY region SUM total HAVING sum_total > 10000')
+db.query('FROM orders GROUP BY region COUNT HAVING count BETWEEN 5 AND 50')
+```
+
+`HAVING` runs through the same evaluator as `WHERE`, so it gets the whole
+predicate surface — `IN`, `BETWEEN`, `LIKE`, `OR`, `NOT`, parentheses — rather
+than a poorer second copy.
+
+### Sorting and paging
+
+```python
+db.query('FROM orders ORDER BY region, total DESC')   # ties broken by the next key
+db.query('FROM orders ORDER BY total DESC LIMIT 20 OFFSET 40')
+```
+
+`OFFSET` skips rows of the result and pairs with `LIMIT` for pagination. An
+offset past the end is an empty page, not an error.
+
+A field whose name collides with a reserved word is still addressable — a
+document may legitimately have a `count`, `min`, `value` or `status` field, and
+`WHERE count > 3` reads that field rather than the aggregate:
+
+```python
+db.query('FROM metrics WHERE count > 3 ORDER BY count DESC')
 ```
 
 Combine both time axes:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "nedb-engine",
-  "version": "3.2.1",
+  "version": "3.3.0",
   "description": "NEDB \u2014 hash-chained, time-traveling, bi-temporal embedded database with Rust native core. SQL, Redis, MongoDB adapters. Causal Write Provenance. RESP2 wire protocol.",
   "main": "index.js",
   "exports": {