nedb-engine 2.8.4__tar.gz → 2.8.6__tar.gz

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/.gitignore

@@ -22,3 +22,7 @@ npm/
 
 # Ephemeral maturin staging copy (CI stages python/nedb here before build)
 rust/crates/nedb-py/python/
+
+# generated by `napi build --js native.js --dts native.d.ts` (index.js is the hand-written wrapper)
+native.js
+native.d.ts

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nedb-engine
-Version: 2.8.4
+Version: 2.8.6
 Summary: NEDB — a versioned, self-compressing, time-traveling embedded database (replay-protected, idempotent, relational, searchable) with durable AOF persistence and a server daemon (nedbd).
 Project-URL: Homepage, https://github.com/aiassistsecure/nedb
 Project-URL: Repository, https://github.com/aiassistsecure/nedb
@@ -41,6 +41,83 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 
 ---
 
+## ⚠️ New in 2.8.6 — Durability & Recovery (read this if you store anything you care about)
+
+Three defects found by killing a real engine at every persistence boundary and by filling a real
+filesystem to zero free blocks. All three are fixed. **If you are on 2.8.5 or earlier, upgrade.**
+
+### 1. A failed flush silently discarded acknowledged writes
+
+`IdIndex::flush_write_buf` cleared every buffered entry regardless of whether its disk write
+succeeded. So a flush that hit `ENOSPC` threw the entry away, and no later flush retried it.
+
+Reproduced on a full 22 MiB filesystem: **30 rows acknowledged by `put() -> Ok`, then `list()`
+returned 0 after reopen — while `verify()` reported all 30 objects healthy.** The content-addressed
+objects were durable; the id-index entries that make them findable were gone.
+
+```
+before:  try_flush_all() -> (no return value)      reopen -> 0 rows, verify() = 30 ok
+after:   try_flush_all() -> Err("id-index leaf rows/buf_25: No space left on device (os error 28)")
+         ...free space, retry -> Ok                reopen -> 30 rows
+```
+
+**Fixed:** an entry leaves the WAL only when its write actually landed. Failures stay buffered and
+retry on the next flush.
+
+### 2. Flush errors were unobservable — new `try_flush_all()`
+
+`flush_all()` returns `()` and logged fsync failures to stderr, so a caller could not tell a durable
+flush from a failed one. Anything that takes a destructive or externally-visible action on the
+strength of a persisted record needs to know.
+
+```rust
+// Use this when the outcome matters:
+db.try_flush_all()?;      // Result<()> — id-index WAL + segment sync + MANIFEST
+
+// Still available, still logs, nowhere to propagate (ticker / Drop):
+db.flush_all();
+```
+
+Also new: `Db::try_flush_manifest()` and `IdIndex::try_flush_write_buf()`.
+
+### 3. `repair` could not repair, and `since()` claimed "caught up" while behind
+
+The cold scan rebuilt `seq_index`, per-collection tips, the Merkle head and `MANIFEST` — but **never
+the id index**. A database whose WAL never reached disk came back with every object verifying and
+`list()` empty, and `nedb-cli repair` printed success without fixing it, because
+`start_cold_scan()` is a deliberate no-op on a warm store.
+
+```bash
+nedb-cli repair ./data
+# repaired: 203 id-index entr(ies) rebuilt, 203 node(s) verified, flushed
+```
+
+```rust
+let restored = db.repair()?;   // rebuild id index from objects; highest seq wins
+```
+
+Every object carries its own `coll`, `id` and `seq`, so the id index is fully derivable — a lost WAL
+is recoverable and nothing is invented. `repair()` also recomputes head and tips, so a repaired
+database reopens **warm** instead of coming back up cold with an empty head.
+
+Separately, `since()` set `has_more = hit_limit` alone. On a warm boot the seq index is empty **by
+design** (that is why warm start is O(1)), so every lookup missed and `since()` returned zero nodes
+with `has_more = false` — indistinguishable from genuinely up to date. A consumer following the
+documented drain loop stopped one call in, with every record unread.
+
+**Fixed:** `has_more` is true whenever the cursor is behind the log head. `ScanStatus` gains
+**`seq_index_ready`** — replication consumers should gate on that, not on `scan_complete`, which is
+true on a warm boot precisely because the scan was skipped.
+
+### Known sharp edge (documented, not changed)
+
+`since()`'s cursor is **exclusive** and seqs start at 0, so `since(0, _)` returns `(0, head]` and the
+very first write in a database (seq 0) is unreachable through any cursor value. Ten writes drain as
+nine records. Changing the convention would break existing consumers; a replica seeded from
+`since()` alone starts one record short.
+
+---
+
 ## NEDB v2.8.0 — Production Stable
 
 **Current stable: 2.8.0** — NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with full mac + linux + windows native addons (see [**Releasing**](#releasing) below). All native wheels (Linux + Windows on GitHub Actions; macOS arm64 + x86_64 on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
@@ -68,6 +145,7 @@ Off by default — feature-gated at compile time, flag-gated at runtime, and `ex
 **New in 2.5.x:**
 
 - **Durable-mode auto-flush-on-exit** — a durable store flushes buffered writes on `Ctrl+C` / `SIGTERM`, not just on a clean `Drop`. Automatic in the Node and Python bindings; `Db::install_exit_flush(Arc<Db>)` for standalone Rust binaries. See [**docs/DURABILITY.md**](docs/DURABILITY.md).
+- **2.8.5 — embedded bindings flush on a cadence.** `NedbCore.open()` (Node + Python) now runs the 1 s manifest ticker exactly as `nedbd` does, so a `SIGKILL` / OOM / power cut loses at most one tick of acknowledged writes instead of everything since open. `NEDB_FLUSH_MS` tunes or disables it. See [docs/DURABILITY.md](docs/DURABILITY.md).
 - **`nedb-cli`** — operate on a store directory offline (`head` · `status` · `verify` · `get` · `scan` · `flush` · `repair` · `export`), and **`nedb-inspector`** — a deterministic (no-regex, no-LLM) checker that warns when a durable open lacks flush-on-exit wiring. See [**docs/CLI.md**](docs/CLI.md).
 - **Replication contract** — `tip()` (the latest write), a bounded `since()` changefeed, and a `scan_status()` readiness gate. See [**docs/REPLICATION.md**](docs/REPLICATION.md).
 

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/README.md

@@ -20,6 +20,83 @@ One Rust core → ships to **PyPI** and **npm** from a single source.
 
 ---
 
+## ⚠️ New in 2.8.6 — Durability & Recovery (read this if you store anything you care about)
+
+Three defects found by killing a real engine at every persistence boundary and by filling a real
+filesystem to zero free blocks. All three are fixed. **If you are on 2.8.5 or earlier, upgrade.**
+
+### 1. A failed flush silently discarded acknowledged writes
+
+`IdIndex::flush_write_buf` cleared every buffered entry regardless of whether its disk write
+succeeded. So a flush that hit `ENOSPC` threw the entry away, and no later flush retried it.
+
+Reproduced on a full 22 MiB filesystem: **30 rows acknowledged by `put() -> Ok`, then `list()`
+returned 0 after reopen — while `verify()` reported all 30 objects healthy.** The content-addressed
+objects were durable; the id-index entries that make them findable were gone.
+
+```
+before:  try_flush_all() -> (no return value)      reopen -> 0 rows, verify() = 30 ok
+after:   try_flush_all() -> Err("id-index leaf rows/buf_25: No space left on device (os error 28)")
+         ...free space, retry -> Ok                reopen -> 30 rows
+```
+
+**Fixed:** an entry leaves the WAL only when its write actually landed. Failures stay buffered and
+retry on the next flush.
+
+### 2. Flush errors were unobservable — new `try_flush_all()`
+
+`flush_all()` returns `()` and logged fsync failures to stderr, so a caller could not tell a durable
+flush from a failed one. Anything that takes a destructive or externally-visible action on the
+strength of a persisted record needs to know.
+
+```rust
+// Use this when the outcome matters:
+db.try_flush_all()?;      // Result<()> — id-index WAL + segment sync + MANIFEST
+
+// Still available, still logs, nowhere to propagate (ticker / Drop):
+db.flush_all();
+```
+
+Also new: `Db::try_flush_manifest()` and `IdIndex::try_flush_write_buf()`.
+
+### 3. `repair` could not repair, and `since()` claimed "caught up" while behind
+
+The cold scan rebuilt `seq_index`, per-collection tips, the Merkle head and `MANIFEST` — but **never
+the id index**. A database whose WAL never reached disk came back with every object verifying and
+`list()` empty, and `nedb-cli repair` printed success without fixing it, because
+`start_cold_scan()` is a deliberate no-op on a warm store.
+
+```bash
+nedb-cli repair ./data
+# repaired: 203 id-index entr(ies) rebuilt, 203 node(s) verified, flushed
+```
+
+```rust
+let restored = db.repair()?;   // rebuild id index from objects; highest seq wins
+```
+
+Every object carries its own `coll`, `id` and `seq`, so the id index is fully derivable — a lost WAL
+is recoverable and nothing is invented. `repair()` also recomputes head and tips, so a repaired
+database reopens **warm** instead of coming back up cold with an empty head.
+
+Separately, `since()` set `has_more = hit_limit` alone. On a warm boot the seq index is empty **by
+design** (that is why warm start is O(1)), so every lookup missed and `since()` returned zero nodes
+with `has_more = false` — indistinguishable from genuinely up to date. A consumer following the
+documented drain loop stopped one call in, with every record unread.
+
+**Fixed:** `has_more` is true whenever the cursor is behind the log head. `ScanStatus` gains
+**`seq_index_ready`** — replication consumers should gate on that, not on `scan_complete`, which is
+true on a warm boot precisely because the scan was skipped.
+
+### Known sharp edge (documented, not changed)
+
+`since()`'s cursor is **exclusive** and seqs start at 0, so `since(0, _)` returns `(0, head]` and the
+very first write in a database (seq 0) is unreachable through any cursor value. Ten writes drain as
+nine records. Changing the convention would break existing consumers; a replica seeded from
+`since()` alone starts one record short.
+
+---
+
 ## NEDB v2.8.0 — Production Stable
 
 **Current stable: 2.8.0** — NEDB ships as **three version-aligned distributions** on one tag — `nedb-engine` (flagship), `crypto-database` (verifiable v2/v3 DAG), and `aof-db` (fast append-only) — across npm / PyPI / crates.io with full mac + linux + windows native addons (see [**Releasing**](#releasing) below). All native wheels (Linux + Windows on GitHub Actions; macOS arm64 + x86_64 on Codemagic M2 Mac Minis) **plus** the universal pure-Python wheel ship from a single `v*` tag, with the `nedbd-v2` binary bundled inside `pip install nedb-engine`.
@@ -47,6 +124,7 @@ Off by default — feature-gated at compile time, flag-gated at runtime, and `ex
 **New in 2.5.x:**
 
 - **Durable-mode auto-flush-on-exit** — a durable store flushes buffered writes on `Ctrl+C` / `SIGTERM`, not just on a clean `Drop`. Automatic in the Node and Python bindings; `Db::install_exit_flush(Arc<Db>)` for standalone Rust binaries. See [**docs/DURABILITY.md**](docs/DURABILITY.md).
+- **2.8.5 — embedded bindings flush on a cadence.** `NedbCore.open()` (Node + Python) now runs the 1 s manifest ticker exactly as `nedbd` does, so a `SIGKILL` / OOM / power cut loses at most one tick of acknowledged writes instead of everything since open. `NEDB_FLUSH_MS` tunes or disables it. See [docs/DURABILITY.md](docs/DURABILITY.md).
 - **`nedb-cli`** — operate on a store directory offline (`head` · `status` · `verify` · `get` · `scan` · `flush` · `repair` · `export`), and **`nedb-inspector`** — a deterministic (no-regex, no-LLM) checker that warns when a durable open lacks flush-on-exit wiring. See [**docs/CLI.md**](docs/CLI.md).
 - **Replication contract** — `tip()` (the latest write), a bounded `since()` changefeed, and a `scan_status()` readiness gate. See [**docs/REPLICATION.md**](docs/REPLICATION.md).
 

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/client/node/README.md

@@ -9,6 +9,26 @@ Connect to any running `nedbd` instance from Node.js or the browser. No engine e
 
 ---
 
+---
+
+## ⚠️ Upgrade your server to nedbd 2.8.6
+
+This client talks to `nedbd`, so its durability comes from the server. **2.8.6 fixes three defects in
+2.8.5 and earlier** — a flush that hit a full disk silently discarded acknowledged writes, flush
+failures were unobservable, and `nedb-cli repair` could not actually rebuild a damaged id index.
+
+If a database ever returned rows and now returns none while `/verify` reports every object healthy,
+that is the lost-id-index symptom and it is recoverable:
+
+```bash
+nedb-cli repair ./data
+```
+
+Replication consumers paging `/since`: gate historical catch-up on **`seq_index_ready`** from
+`/status`, not on `scan_complete`. `scan_complete` is true on a warm boot precisely because the scan
+was skipped, and before 2.8.6 `since` reported `has_more: false` in that state — which reads as
+"caught up" on a database with every record unread.
+
 ## Install
 
 ```bash

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/client/python/README.md

@@ -10,6 +10,26 @@ Connect to any running `nedbd` instance — local or remote — with a clean asy
 
 ---
 
+---
+
+## ⚠️ Upgrade your server to nedbd 2.8.6
+
+This client talks to `nedbd`, so its durability comes from the server. **2.8.6 fixes three defects in
+2.8.5 and earlier** — a flush that hit a full disk silently discarded acknowledged writes, flush
+failures were unobservable, and `nedb-cli repair` could not actually rebuild a damaged id index.
+
+If a database ever returned rows and now returns none while `/verify` reports every object healthy,
+that is the lost-id-index symptom and it is recoverable:
+
+```bash
+nedb-cli repair ./data
+```
+
+Replication consumers paging `/since`: gate historical catch-up on **`seq_index_ready`** from
+`/status`, not on `scan_complete`. `scan_complete` is true on a warm boot precisely because the scan
+was skipped, and before 2.8.6 `since` reported `has_more: false` in that state — which reads as
+"caught up" on a database with every record unread.
+
 ## Install
 
 ```bash

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/pyproject.toml

@@ -10,7 +10,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "nedb-engine"
-version = "2.8.4"
+version = "2.8.6"
 description = "NEDB — a versioned, self-compressing, time-traveling embedded database (replay-protected, idempotent, relational, searchable) with durable AOF persistence and a server daemon (nedbd)."
 readme = "README.md"
 requires-python = ">=3.8"

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/python/nedb/__init__.py

@@ -94,4 +94,4 @@ __all__ = [
     "verify_proof", "fold_head",
     "_native", "__has_native__",
 ]
-__version__ = "2.8.4"
+__version__ = "2.8.6"

{nedb_engine-2.8.4 → nedb_engine-2.8.6}/rust/nedb-v2/README.md

@@ -16,6 +16,83 @@ nedbd --dag ./data        # DAG engine (v2, content-addressed, recommended)
 
 ---
 
+## ⚠️ New in 2.8.6 — Durability & Recovery (read this if you store anything you care about)
+
+Three defects found by killing a real engine at every persistence boundary and by filling a real
+filesystem to zero free blocks. All three are fixed. **If you are on 2.8.5 or earlier, upgrade.**
+
+### 1. A failed flush silently discarded acknowledged writes
+
+`IdIndex::flush_write_buf` cleared every buffered entry regardless of whether its disk write
+succeeded. So a flush that hit `ENOSPC` threw the entry away, and no later flush retried it.
+
+Reproduced on a full 22 MiB filesystem: **30 rows acknowledged by `put() -> Ok`, then `list()`
+returned 0 after reopen — while `verify()` reported all 30 objects healthy.** The content-addressed
+objects were durable; the id-index entries that make them findable were gone.
+
+```
+before:  try_flush_all() -> (no return value)      reopen -> 0 rows, verify() = 30 ok
+after:   try_flush_all() -> Err("id-index leaf rows/buf_25: No space left on device (os error 28)")
+         ...free space, retry -> Ok                reopen -> 30 rows
+```
+
+**Fixed:** an entry leaves the WAL only when its write actually landed. Failures stay buffered and
+retry on the next flush.
+
+### 2. Flush errors were unobservable — new `try_flush_all()`
+
+`flush_all()` returns `()` and logged fsync failures to stderr, so a caller could not tell a durable
+flush from a failed one. Anything that takes a destructive or externally-visible action on the
+strength of a persisted record needs to know.
+
+```rust
+// Use this when the outcome matters:
+db.try_flush_all()?;      // Result<()> — id-index WAL + segment sync + MANIFEST
+
+// Still available, still logs, nowhere to propagate (ticker / Drop):
+db.flush_all();
+```
+
+Also new: `Db::try_flush_manifest()` and `IdIndex::try_flush_write_buf()`.
+
+### 3. `repair` could not repair, and `since()` claimed "caught up" while behind
+
+The cold scan rebuilt `seq_index`, per-collection tips, the Merkle head and `MANIFEST` — but **never
+the id index**. A database whose WAL never reached disk came back with every object verifying and
+`list()` empty, and `nedb-cli repair` printed success without fixing it, because
+`start_cold_scan()` is a deliberate no-op on a warm store.
+
+```bash
+nedb-cli repair ./data
+# repaired: 203 id-index entr(ies) rebuilt, 203 node(s) verified, flushed
+```
+
+```rust
+let restored = db.repair()?;   // rebuild id index from objects; highest seq wins
+```
+
+Every object carries its own `coll`, `id` and `seq`, so the id index is fully derivable — a lost WAL
+is recoverable and nothing is invented. `repair()` also recomputes head and tips, so a repaired
+database reopens **warm** instead of coming back up cold with an empty head.
+
+Separately, `since()` set `has_more = hit_limit` alone. On a warm boot the seq index is empty **by
+design** (that is why warm start is O(1)), so every lookup missed and `since()` returned zero nodes
+with `has_more = false` — indistinguishable from genuinely up to date. A consumer following the
+documented drain loop stopped one call in, with every record unread.
+
+**Fixed:** `has_more` is true whenever the cursor is behind the log head. `ScanStatus` gains
+**`seq_index_ready`** — replication consumers should gate on that, not on `scan_complete`, which is
+true on a warm boot precisely because the scan was skipped.
+
+### Known sharp edge (documented, not changed)
+
+`since()`'s cursor is **exclusive** and seqs start at 0, so `since(0, _)` returns `(0, head]` and the
+very first write in a database (seq 0) is unreachable through any cursor value. Ten writes drain as
+nine records. Changing the convention would break existing consumers; a replica seeded from
+`since()` alone starts one record short.
+
+---
+
 ## What is NEDB v2?
 
 NEDB v2 replaces the append-only log (AOF) with a **content-addressed Merkle DAG**: