@push.rocks/smartdb 2.13.2 → 2.14.0

package/dist_ts/00_commitinfo_data.js

@@ -3,7 +3,7 @@
  */
 export const commitinfo = {
     name: '@push.rocks/smartdb',
-    version: '2.13.2',
+    version: '2.14.0',
     description: 'A MongoDB-compatible embedded database server with wire protocol support, backed by a high-performance Rust engine.'
 };
 //# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoiMDBfY29tbWl0aW5mb19kYXRhLmpzIiwic291cmNlUm9vdCI6IiIsInNvdXJjZXMiOlsiLi4vdHMvMDBfY29tbWl0aW5mb19kYXRhLnRzIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiJBQUFBOztHQUVHO0FBQ0gsTUFBTSxDQUFDLE1BQU0sVUFBVSxHQUFHO0lBQ3hCLElBQUksRUFBRSxxQkFBcUI7SUFDM0IsT0FBTyxFQUFFLFFBQVE7SUFDakIsV0FBVyxFQUFFLHFIQUFxSDtDQUNuSSxDQUFBIn0=

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartdb",
-  "version": "2.13.2",
+  "version": "2.14.0",
   "private": false,
   "description": "A MongoDB-compatible embedded database server with wire protocol support, backed by a high-performance Rust engine.",
   "exports": {

package/readme.hints.md

@@ -1,5 +1,12 @@
 # smartdb hints
 
+## No-Op Write Detection (2.14.0)
+
+- `update`/`findAndModify` post-images that equal the stored document skip storage, WAL, index staging, and oplog entirely; volatile-only rewrites (top-level `_updatedAt` differs, everything else equal) are skipped the same way and keep the stored `_updatedAt`. Both report `matched=1, modified=0`.
+- Classification lives in `rustdb-commands/src/handlers/update_handler.rs` (`classify_document_rewrite`); the volatile field list is the `VOLATILE_METADATA_FIELDS` const. The comparison is key-order-insensitive (replacement paths re-append `_id`) and bails to "changed" on duplicate keys.
+- Counters: `serverStatus.writes.{updatesWritten, noopSkipped.identical, noopSkipped.volatileOnly}` backed by `WriteMetrics` on `CommandContext`. Wire-level coverage in `test/test.noop-writes.ts`.
+- Consequence for consumers: a save that changes nothing no longer bumps `_updatedAt` and emits no oplog event. To force a touch, change a non-volatile field.
+
 ## Static Rust Binaries
 
 - The Rust binaries in `dist_rust/` are statically linked (static-pie) via `"static": true` in the `@git.zone/tsrust` block of `.smartconfig.json` (tsrust >= 1.4.1). They run on both glibc (Debian/Ubuntu) and musl (Alpine) systems.

package/readme.md

@@ -227,6 +227,33 @@ Each entry contains:
 
 ---
 
+## ✋ No-Op Write Detection
+
+The engine detects document rewrites that change nothing and skips them entirely — no storage write, no WAL append, no index update, and no oplog entry. A skipped rewrite still counts as matched (`matchedCount: 1`) but reports `modifiedCount: 0`.
+
+Two classes are skipped:
+
+- **Identical** — the post-image equals the stored document byte for byte.
+- **Volatile-only** — the post-image differs only in top-level volatile metadata fields (currently `_updatedAt`, the field ODM layers such as `@push.rocks/smartdata` restamp on every save). The stored document is kept as-is, including its existing `_updatedAt`, so the timestamp means "last real change" rather than "last save call".
+
+This makes periodic reconcile loops that re-save unchanged documents cost nothing at the engine level. A caller that needs a document to actually change must change a non-volatile field.
+
+Counters are exposed through `serverStatus`:
+
+```javascript
+const status = await db.command({ serverStatus: 1 });
+console.log(status.writes);
+// {
+//   updatesWritten: 42,        // rewrites that reached storage, index, and oplog
+//   noopSkipped: {
+//     identical: 1337,         // post-image equal to the stored document
+//     volatileOnly: 271,       // only volatile metadata differed
+//   },
+// }
+```
+
+---
+
 ## API Reference
 
 ### SmartdbServer

package/readme.plan.md

@@ -0,0 +1,99 @@
+# smartdb advancement plan
+
+Status: canonical direction, approved 2026-07-21. Reread before starting any engine work.
+Baseline: @push.rocks/smartdb 2.13.2 (TypeScript API + rustdb Rust engine, ~36k Rust LOC, 255 test fns).
+
+## North star
+
+Make smartdb a boring database:
+
+- **provably crash-consistent** — recovery behavior is a written contract, enforced by a harness, not an aspiration
+- **cost proportional to change** — a write that changes nothing costs nothing; a small change costs a small amount of WAL, oplog, index, and CPU
+- **bounded** — memory, cursors, sessions, scans, and on-disk growth all have enforced ceilings
+- **self-describing** — the engine can explain its own resource usage and slow operations without external tooling
+
+## Evidence base (incident ledger)
+
+Every track below traces to a production incident on the dcrouter hub:
+
+1. **Immortal cursors / unreaped sessions** — cursors and transaction sessions accumulated without bound. Fixed in 2.13.x (`CursorOwner`, per-cursor and global byte caps in `rustdb-commands/src/context.rs`, `remove_connection_cursors`, `take_expired_sessions`). The class stays closed only if bounded-lifecycle remains a tested invariant.
+2. **Query-vs-storage divergence** — an empty `_id_` query index was created at restart while the storage KeyDir still held documents. Queries said "not found"; inserts hit the storage-layer duplicate check (`rustdb-storage/src/memory.rs` / `file.rs` `AlreadyExists`) and threw "already exists: document '<sha>'". Mitigated in 2.13.x (index rebuild at startup, quarantine of invalid legacy unique indexes). The invariant — query layer and storage layer must never disagree after recovery — is not yet machine-checked.
+3. **Write amplification on unchanged saves** — callers re-saving identical documents caused full document rewrites plus full before/after images in the oplog (10k entries / 64MB), driving rustdb to 45–82% CPU. Currently mitigated caller-side only (change-gated persistence in the main consumer). The engine still trusts callers not to churn.
+4. **Bulk-operation ceilings** — deleteMany batch limits and command-size caps made a large startup migration exceed its deadline and forced a rollback. Bulk work is client-paced, non-resumable, and dies with its connection.
+5. **Memory disproportion** — steady-state PSS reached 40–100× live data bytes at points (dual tree+hash indexes per field, allocator retention, oplog images). No ceiling is asserted anywhere.
+
+## Track 1 — Provable durability (foundation, non-negotiable)
+
+- Write the **recovery contract**: for a process kill at any byte offset — during WAL append, compaction, or hint write — state exactly what is durable, what may be lost, and what is rebuilt.
+- Build a **crash-fuzz harness**: property-based runner that executes randomized workloads, kills the engine at random points, restarts, and asserts the invariants:
+  1. every acknowledged write is present
+  2. no phantom documents
+  3. query indexes agree exactly with the storage KeyDir (incident 2 becomes a permanent regression class)
+  4. unique constraints hold
+  5. cursor/session tables are empty after recovery
+- Make the harness a **CI gate**: no release ships without it green.
+
+Provable property: "kill -9 at any moment loses at most unacknowledged writes, and recovery always converges to a consistent index/storage view."
+
+## Track 2 — Write-path economics: cost proportional to change
+
+- **Engine-side no-op write detection** — SHIPPED in 2.14.0: update/findAndModify post-images that equal the stored document (or differ only in the volatile `_updatedAt` metadata field) skip storage, WAL, index, and oplog entirely and report `matched=1, modified=0`. Counters in `serverStatus.writes`. Caller-side gating remains defense in depth, not the architecture.
+- **Oplog diet**: replace full before/after images with `{ns, id, opType, generation, optional patch}`. Consumers that need images read storage at the generation.
+- **Background compaction** (bitcask merge) with cooldown and rate limit, bounding `data.rdb` growth under update churn.
+- **Group commit**: coalesce WAL fsyncs across concurrent writers.
+
+Provable property: "N identical saves cost O(1) WAL bytes and O(1) oplog bytes after the first."
+
+## Track 3 — Read-path economics
+
+- **Index-assisted top-k**: extend the bounded sorted-scan work (2.13.2) so the planner walks an index in sort order instead of scanning and heap-selecting.
+- **Maintained aggregates**: per-collection counts and sizes served from maintained state, never full scans.
+- **Reads never schedule writes**: engine principle — no read path may enqueue hint rewrites, stat persistence, or any other write work.
+
+Provable property: "a monitoring read loop at any frequency produces zero engine writes."
+
+## Track 4 — Server-side bulk and migration primitives
+
+- **Uncapped bulk operations**: deleteMany/updateMany execute engine-side with internal checkpointing (by generation / seek offset), not client-paced batches under command-size caps.
+- **Streaming IPC frames** for large results, retiring the command-size failure class.
+- **Re-attachable maintenance sessions**: a migration survives client disconnect; the client re-attaches by operation id and polls progress. The incident-4 rollback class becomes impossible.
+
+Provable property: "a 10M-row migration survives a client restart and completes exactly once."
+
+## Track 5 — Memory model: bounded and measured
+
+- **k-factor target**: steady-state PSS ≤ 3× live data bytes (observed today: 40–100× under some workloads).
+- **Single index representation** per field where one suffices; lazy-build the second form on demand.
+- KeyDir compaction / shrink-to-fit after mass deletes.
+- **Soak-test CI gate**: 24h synthetic churn workload asserting PSS plateaus within the k-factor.
+- Verify mimalloc purge/decommit settings under the soak test.
+
+Provable property: "PSS is a function of live data, not of history."
+
+## Track 6 — Operability and compatibility
+
+- **serverStatus counters**: no-op writes skipped, WAL bytes written, compaction stats, live cursors/sessions, per-collection sizes — the next incident gets diagnosed from the database, not from strace.
+- **Slow-op log** with thresholds.
+- **Offline toolbox**: `rustdb verify | dump | compact | repair-doc | backup` against a data directory.
+- **On-disk format manifest** with versioning and an N-1 rollback contract: any release can downgrade one release; format changes require a migration note.
+
+Provable property: "any resource question an operator asks during an incident is answerable from serverStatus or the toolbox."
+
+## Fork in the road (decision rule, written now)
+
+Keep hardening the owned engine **while the Track 1 harness stays green**. If, after Tracks 1–2 land, the harness keeps finding structural durability holes, evaluate rebasing the storage layer on a proven embedded core (redb or fjall) behind the unchanged command surface. Revisit this rule once Track 1 is merged — not before, not on gut feeling.
+
+## Sequencing
+
+- **Done**: Track 2 no-op write detection + its serverStatus counters (2.14.0).
+- **Now** (after the current consumer release wave is verified in production): Track 1 harness + recovery contract.
+- **Next**: Track 2 oplog diet + compaction; Track 4 bulk primitives.
+- **Mid-term**: Track 3 read economics; Track 5 k-factor + soak gate; Track 6 toolbox + format manifest.
+
+Format-affecting changes (oplog diet, manifest) target a major version; everything else lands in 2.14+.
+
+## Non-goals
+
+- clustering or replication beyond the current oplog consumers
+- MongoDB API parity
+- multi-process writers

package/ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdb',
-  version: '2.13.2',
+  version: '2.14.0',
   description: 'A MongoDB-compatible embedded database server with wire protocol support, backed by a high-performance Rust engine.'
 }