rust-kgdb 0.1.11 → 0.2.0

package/README.md

@@ -5,6 +5,33 @@
 
 **Production-ready RDF/hypergraph database with 100% W3C SPARQL 1.1 + RDF 1.2 compliance, worst-case optimal joins (WCOJ), and pluggable storage backends.**
 
+> **This npm package provides the high-performance in-memory database.**
+> For **distributed cluster deployment** (1B+ triples, horizontal scaling), contact: **gonnect.uk@gmail.com**
+
+---
+
+## Deployment Modes
+
+rust-kgdb supports three deployment modes:
+
+| Mode | Use Case | Scalability | This Package |
+|------|----------|-------------|--------------|
+| **In-Memory** | Development, embedded apps, testing | Single node, volatile | ✅ **Included** |
+| **Single Node (RocksDB/LMDB)** | Production, persistence needed | Single node, persistent | Via Rust crate |
+| **Distributed Cluster** | Enterprise, 1B+ triples | Horizontal scaling, 9+ partitions | Contact us |
+
+### Need Distributed Cluster?
+
+For enterprise deployments requiring:
+- **Subject-Anchored Partitioning**: All triples for a subject guaranteed on same partition for locality
+- Horizontal scaling across multiple nodes (1B+ triples)
+- HDRF (High-Degree Replicated First) with power-law load balancing
+- **OLAP Query Path**: SQL-based analytical execution for aggregations
+- Subject-Hash Filter for accurate COUNT deduplication across replicas
+- Kubernetes-native deployment with StatefulSet executors
+
+**Request a demo: gonnect.uk@gmail.com**
+
 ---
 
 ## Why rust-kgdb?
@@ -108,7 +135,7 @@ rust-kgdb uses a pluggable storage architecture. **Default is in-memory** (zero
 |---------|--------------|----------|--------|
 | **InMemory** | `default` | Development, testing, embedded | ✅ **Production Ready** |
 | **RocksDB** | `rocksdb-backend` | Production, large datasets | ✅ **61 tests passing** |
-| **LMDB** | `lmdb-backend` | Read-heavy workloads | ⏳ Planned v0.2.0 |
+| **LMDB** | `lmdb-backend` | Read-heavy workloads | ✅ **31 tests passing** |
 
 ### InMemory (Default)
 
@@ -176,6 +203,58 @@ store.flush()?;
 - Unicode & binary data (4 tests)
 - Large key/value handling (8 tests)
 
+### LMDB (Memory-Mapped Persistent)
+
+B+tree based storage with memory-mapped I/O (via `heed` crate). Optimized for **read-heavy workloads** with MVCC (Multi-Version Concurrency Control). Tested with **31 comprehensive tests**.
+
+```toml
+# Cargo.toml - Enable LMDB backend
+[dependencies]
+storage = { version = "0.1.12", features = ["lmdb-backend"] }
+```
+
+```rust
+use storage::{QuadStore, LmdbBackend};
+
+// Create persistent database (default 10GB map size)
+let backend = LmdbBackend::new("/path/to/data")?;
+let store = QuadStore::new(backend);
+
+// Or with custom map size (1GB)
+let backend = LmdbBackend::with_map_size("/path/to/data", 1024 * 1024 * 1024)?;
+
+// Features:
+// - Memory-mapped I/O (zero-copy reads)
+// - MVCC for concurrent readers
+// - Crash-safe ACID transactions
+// - Range & prefix scanning
+// - Excellent for read-heavy workloads
+
+// Sync to disk
+store.flush()?;
+```
+
+**When to use LMDB vs RocksDB:**
+
+| Characteristic | LMDB | RocksDB |
+|----------------|------|---------|
+| **Read Performance** | ✅ Faster (memory-mapped) | Good |
+| **Write Performance** | Good | ✅ Faster (LSM-tree) |
+| **Concurrent Readers** | ✅ Unlimited | Limited by locks |
+| **Write Amplification** | Low | Higher (compaction) |
+| **Memory Usage** | Higher (map size) | Lower (cache-based) |
+| **Best For** | Read-heavy, OLAP | Write-heavy, OLTP |
+
+**LMDB Test Coverage:**
+- Basic CRUD operations (8 tests)
+- Range scanning (4 tests)
+- Prefix scanning (3 tests)
+- Batch operations (3 tests)
+- Large key/value handling (4 tests)
+- Concurrent access (4 tests)
+- Statistics & flush (3 tests)
+- Edge cases (2 tests)
+
 ### TypeScript SDK
 
 The npm package uses the in-memory backend—ideal for:
@@ -507,8 +586,52 @@ Total: ~120 bytes/triple including indexes
 
 ---
 
+## Performance Benchmarks
+
+### By Deployment Mode
+
+| Mode | Lookup | Insert | Memory | Dataset Size |
+|------|--------|--------|--------|--------------|
+| **In-Memory (npm)** | 2.78 µs | 146K/sec | 24 bytes/triple | <10M triples |
+| **Single Node (RocksDB)** | 5-10 µs | 100K/sec | On-disk | <100M triples |
+| **Distributed Cluster** | 10-50 µs | 500K+/sec* | Distributed | **1B+ triples** |
+
+*Aggregate throughput across all executors with HDRF partitioning
+
+### SIMD + PGO Query Performance (LUBM Benchmark)
+
+| Query | Pattern | Time | Improvement |
+|-------|---------|------|-------------|
+| Q5 | 2-hop chain | 53ms | **77% faster** |
+| Q3 | 3-way star | 62ms | **65% faster** |
+| Q4 | 3-hop chain | 101ms | **60% faster** |
+| Q8 | Triangle | 193ms | **53% faster** |
+| Q7 | Hierarchy | 198ms | **42% faster** |
+
+**Average: 44.5% speedup** with zero code changes (compiler optimizations only).
+
+---
+
 ## Version History
 
+### v0.2.0 (2025-12-08) - Distributed Cluster Support
+
+- **NEW: Distributed cluster architecture** with HDRF partitioning
+- **Subject-Hash Filter** for accurate COUNT deduplication across replicas
+- **DataFusion-powered OLAP** with Arrow-native vectorized execution
+- Coordinator-Executor pattern with gRPC communication
+- 9-partition default for optimal data distribution
+- **Contact for cluster deployment**: gonnect.uk@gmail.com
+- **Coming soon**: Embedding support for semantic search (v0.3.0)
+
+### v0.1.12 (2025-12-01) - LMDB Backend Release
+
+- **LMDB storage backend** fully implemented (31 tests passing)
+- Memory-mapped I/O for optimal read performance
+- MVCC concurrency for unlimited concurrent readers
+- Complete LMDB vs RocksDB comparison documentation
+- Sample application with 87 triples demonstrating all features
+
 ### v0.1.9 (2025-12-01) - SIMD + PGO Release
 
 - **44.5% average speedup** via SIMD + PGO compiler optimizations

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "rust-kgdb",
-  "version": "0.1.11",
-  "description": "High-performance RDF/SPARQL database with 100% W3C compliance and WCOJ execution",
+  "version": "0.2.0",
+  "description": "High-performance RDF/SPARQL database with 100% W3C compliance, WCOJ execution, and distributed cluster support",
   "main": "index.js",
   "types": "index.d.ts",
   "napi": {
@@ -21,7 +21,7 @@
     "build:debug": "napi build --platform native/rust-kgdb-napi",
     "prepublishOnly": "napi prepublish -t npm",
     "test": "jest",
-    "version": "0.1.11"
+    "version": "0.2.0"
   },
   "keywords": [
     "rdf",
@@ -56,10 +56,10 @@
     "*.node"
   ],
   "optionalDependencies": {
-    "rust-kgdb-win32-x64-msvc": "0.1.11",
-    "rust-kgdb-darwin-x64": "0.1.11",
-    "rust-kgdb-linux-x64-gnu": "0.1.11",
-    "rust-kgdb-darwin-arm64": "0.1.11",
-    "rust-kgdb-linux-arm64-gnu": "0.1.11"
+    "rust-kgdb-win32-x64-msvc": "0.2.0",
+    "rust-kgdb-darwin-x64": "0.2.0",
+    "rust-kgdb-linux-x64-gnu": "0.2.0",
+    "rust-kgdb-darwin-arm64": "0.2.0",
+    "rust-kgdb-linux-arm64-gnu": "0.2.0"
   }
 }