lodedb 0.2.1__tar.gz → 0.3.0__tar.gz

{lodedb-0.2.1 → lodedb-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lodedb
-Version: 0.2.1
+Version: 0.3.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3 :: Only
@@ -36,23 +36,34 @@ Project-URL: Homepage, https://github.com/Egoist-Machines/LodeDB
 Project-URL: Issues, https://github.com/Egoist-Machines/LodeDB/issues
 Project-URL: Repository, https://github.com/Egoist-Machines/LodeDB
 
-# LodeDB
+<h1 align="center">LodeDB</h1>
+<p align="center">🔥 the <b>fastest</b> and <i>most compact</i> embedded vector database in the world 🌍</p>
 
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](pyproject.toml)
 
-**A fast, exact embedded vector database for local RAG: in-process, on-disk, no server.**
-
 *Built by [Egoist Machines, Inc.](https://egoistmachines.com) - efficient full-stack infrastructure
 for reliable AI systems.*
 
-Most embedded vector databases stop at the CPU. LodeDB runs the same on-disk index on the
-GPU when you have one: batched search hits **24k queries/sec on an A10 and 50k qps on an L40S**,
+LodeDB is great for local RAG; it's _extremely fast_, exact, in-process, and on-disk. We're the **best drop-in** durable memory backend for **LangChain, LlamaIndex, and mem0** across all metrics that matter: the most
+compact on disk, GPU-accelerated, and sub-millisecond on durable writes. Point any of them at
+LodeDB instead of its default store. Over 17.5k documents, per framework default:
+
+| vs the framework's default store | LangChain `InMemoryVectorStore` | LlamaIndex `SimpleVectorStore` | mem0 Qdrant |
+|---|---|---|---|
+| On-disk footprint | **7.2× smaller** (28 vs 199 MB) | **5.3× smaller** (28 vs 145 MB) | **4.6× smaller** (15 vs 70 MB) |
+| Single-query p50 (CPU) | **~410× faster** (0.8 vs 345 ms) | **~500× faster** (0.9 vs 427 ms) | **~25× faster** (0.9 vs 23 ms) |
+| Batched retrieval, 64 (GPU) | **~2,000×** (6,280 vs ~3 qps) | **~2,800×** (5,744 vs ~2 qps) | **~62×** (2,686 vs 43 qps) |
+| Durable add of one memory | **~10,000× faster** (0.8 ms vs 8.6 s) | **~22,000× faster** (0.9 ms vs 19.3 s) | 0.9 vs 0.5 ms (both sub-ms) |
+
+[Full benchmark, all backends
+(FAISS, Chroma, Qdrant, LanceDB, sqlite-vec, pgvector), and method.](benchmarks/memory_integrations)
+
+Most embedded vector databases stop at the CPU. LodeDB runs the same on-disk index **on the
+GPU** when you have one: batched search hits *24k queries/sec on an A10 and 50k qps on an L40S*,
 2.8× to 4.8× the all-CPU ceiling, with recall unchanged. It also persists changed rows
 incrementally, so a commit stays **sub-millisecond even at 1M vectors**.
 
-Fast on a laptop. Faster on a GPU. Exact every time. Never phones home.
-
 - **GPU-resident batch search**: an fp16 copy of the index lives on the GPU, scored with a
   tiled GEMM plus a streaming top-k (`[gpu]`, Linux/CUDA). [How it works](#gpu-resident-index).
 - **O(changed) persistence**: commits only the rows that changed, 173× to 1,308× faster
@@ -196,7 +207,8 @@ db.search("E1234", k=5, mode="hybrid")  # BM25 + RRF: surfaces it in the top-k
 db.search("E1234", k=5, mode="lexical") # BM25 ranking alone, no vector scan
 ```
 
-### Prerequisites
+<details>
+<summary><b>Prerequisites</b></summary>
 
 `mode="hybrid"` and `mode="lexical"` build a BM25 index over your text, so they need a text
 source enabled when you open the database. `mode="vector"` (the default) needs nothing.
@@ -210,8 +222,10 @@ source enabled when you open the database. `mode="vector"` (the default) needs n
 Either source is enough and you can enable both. `store_text=True` is the default, so hybrid
 works out of the box. With neither source enabled, a hybrid or lexical query raises a clear,
 actionable error rather than silently degrading.
+</details>
 
-### How it works
+<details>
+<summary><b>How it works</b></summary>
 
 A `filter` constrains both rankers, so `mode="hybrid"` with a filter returns the true top-k of
 the matching subset. The vector half of a hybrid query runs on the same scan as `mode="vector"`,
@@ -219,8 +233,10 @@ including the GPU-resident batch scan that serves `search_many`; only the BM25 r
 fusion run on the CPU, and the vector kernel and on-disk format are untouched. The serving BM25
 index lives in memory and is maintained incrementally: a small mutation folds just the changed
 chunks into the existing index, so a single `add` never forces a full re-tokenization.
+</details>
 
-### Durable lexical index (`index_text=True`)
+<details>
+<summary><b>Durable lexical index (`index_text=True`)</b></summary>
 
 By default the BM25 index is rebuilt from the retained raw text, so it needs `store_text=True`
 and is re-tokenized on the first hybrid query after opening. Pass `index_text=True` to persist
@@ -241,6 +257,7 @@ db.close()
 reopened = LodeDB(path="./data", index_text=True, store_text=False)
 reopened.search("E1234", k=5, mode="hybrid")  # works after reopen, rebuilt from persisted terms
 ```
+</details>
 
 ## GPU-resident index
 
@@ -339,6 +356,9 @@ your host at `lodedb mcp`:
 
 ```bash
 pip install "lodedb[mcp]"
+
+# for coding assistant:
+lodedb mcp install --client claude-code  # or: claude-desktop | cursor | lm-studio | codex | all
 ```
 
 It exposes `lodedb_add`, `lodedb_search`, `lodedb_remove`, and `lodedb_stats`, plus
@@ -351,7 +371,7 @@ the server with `--exclude-text` to return metrics only (this also withdraws `lo
 `--no-store-text` to keep no text on disk at all. `lodedb_stats` is always metrics-only and raw
 query text never leaves the process.
 
-### One command
+### One command install
 
 `lodedb mcp install` writes the correct entry to a client's config for you, so you do not have to
 find the file or hand-write the JSON/TOML:
@@ -423,6 +443,17 @@ See [`examples/mcp_config.json`](examples/mcp_config.json) for a copy-paste star
 - **Durability is `fast` by default.** Commits are *atomic* but not fsync'd. Pass
   `durability="fsync"` (or `--durability fsync` / `LODEDB_DURABILITY=fsync`) to fsync each
   file and its directory on commit for power-loss durability, at some commit-throughput cost.
+- **WAL commit by default for low-latency durable writes.** Each `add`/`remove` appends one
+  framed record to a `<key>.wal` log and a full generation is checkpointed periodically, so a
+  durable single add costs roughly an order of magnitude less than publishing a whole generation
+  per write, into the sqlite-vec/qdrant range (see the comparison up top). The WAL is replayed
+  crash-atomically on reopen (a half-written trailing record is discarded), every writable open
+  folds it into a clean committed generation, and `close()`/`persist()` checkpoint it. WAL is
+  single-writer: a concurrent `open_readonly` reader still loads a consistent committed
+  generation, but the last *checkpointed* one, not the writer's in-flight WAL. Pass
+  `commit_mode="generation"` (or `LODEDB_COMMIT_MODE=generation`) for the classic path that
+  publishes a crash-atomic, MVCC-readable generation on every write; pick it when many
+  out-of-process readers must see each write the instant it commits.
 - **Local filesystems only.** The OS advisory lock is unreliable on NFS/SMB.
 
 ## Limitations

{lodedb-0.2.1 → lodedb-0.3.0}/README.md

@@ -1,20 +1,31 @@
-# LodeDB
+<h1 align="center">LodeDB</h1>
+<p align="center">🔥 the <b>fastest</b> and <i>most compact</i> embedded vector database in the world 🌍</p>
 
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](pyproject.toml)
 
-**A fast, exact embedded vector database for local RAG: in-process, on-disk, no server.**
-
 *Built by [Egoist Machines, Inc.](https://egoistmachines.com) - efficient full-stack infrastructure
 for reliable AI systems.*
 
-Most embedded vector databases stop at the CPU. LodeDB runs the same on-disk index on the
-GPU when you have one: batched search hits **24k queries/sec on an A10 and 50k qps on an L40S**,
+LodeDB is great for local RAG; it's _extremely fast_, exact, in-process, and on-disk. We're the **best drop-in** durable memory backend for **LangChain, LlamaIndex, and mem0** across all metrics that matter: the most
+compact on disk, GPU-accelerated, and sub-millisecond on durable writes. Point any of them at
+LodeDB instead of its default store. Over 17.5k documents, per framework default:
+
+| vs the framework's default store | LangChain `InMemoryVectorStore` | LlamaIndex `SimpleVectorStore` | mem0 Qdrant |
+|---|---|---|---|
+| On-disk footprint | **7.2× smaller** (28 vs 199 MB) | **5.3× smaller** (28 vs 145 MB) | **4.6× smaller** (15 vs 70 MB) |
+| Single-query p50 (CPU) | **~410× faster** (0.8 vs 345 ms) | **~500× faster** (0.9 vs 427 ms) | **~25× faster** (0.9 vs 23 ms) |
+| Batched retrieval, 64 (GPU) | **~2,000×** (6,280 vs ~3 qps) | **~2,800×** (5,744 vs ~2 qps) | **~62×** (2,686 vs 43 qps) |
+| Durable add of one memory | **~10,000× faster** (0.8 ms vs 8.6 s) | **~22,000× faster** (0.9 ms vs 19.3 s) | 0.9 vs 0.5 ms (both sub-ms) |
+
+[Full benchmark, all backends
+(FAISS, Chroma, Qdrant, LanceDB, sqlite-vec, pgvector), and method.](benchmarks/memory_integrations)
+
+Most embedded vector databases stop at the CPU. LodeDB runs the same on-disk index **on the
+GPU** when you have one: batched search hits *24k queries/sec on an A10 and 50k qps on an L40S*,
 2.8× to 4.8× the all-CPU ceiling, with recall unchanged. It also persists changed rows
 incrementally, so a commit stays **sub-millisecond even at 1M vectors**.
 
-Fast on a laptop. Faster on a GPU. Exact every time. Never phones home.
-
 - **GPU-resident batch search**: an fp16 copy of the index lives on the GPU, scored with a
   tiled GEMM plus a streaming top-k (`[gpu]`, Linux/CUDA). [How it works](#gpu-resident-index).
 - **O(changed) persistence**: commits only the rows that changed, 173× to 1,308× faster
@@ -158,7 +169,8 @@ db.search("E1234", k=5, mode="hybrid")  # BM25 + RRF: surfaces it in the top-k
 db.search("E1234", k=5, mode="lexical") # BM25 ranking alone, no vector scan
 ```
 
-### Prerequisites
+<details>
+<summary><b>Prerequisites</b></summary>
 
 `mode="hybrid"` and `mode="lexical"` build a BM25 index over your text, so they need a text
 source enabled when you open the database. `mode="vector"` (the default) needs nothing.
@@ -172,8 +184,10 @@ source enabled when you open the database. `mode="vector"` (the default) needs n
 Either source is enough and you can enable both. `store_text=True` is the default, so hybrid
 works out of the box. With neither source enabled, a hybrid or lexical query raises a clear,
 actionable error rather than silently degrading.
+</details>
 
-### How it works
+<details>
+<summary><b>How it works</b></summary>
 
 A `filter` constrains both rankers, so `mode="hybrid"` with a filter returns the true top-k of
 the matching subset. The vector half of a hybrid query runs on the same scan as `mode="vector"`,
@@ -181,8 +195,10 @@ including the GPU-resident batch scan that serves `search_many`; only the BM25 r
 fusion run on the CPU, and the vector kernel and on-disk format are untouched. The serving BM25
 index lives in memory and is maintained incrementally: a small mutation folds just the changed
 chunks into the existing index, so a single `add` never forces a full re-tokenization.
+</details>
 
-### Durable lexical index (`index_text=True`)
+<details>
+<summary><b>Durable lexical index (`index_text=True`)</b></summary>
 
 By default the BM25 index is rebuilt from the retained raw text, so it needs `store_text=True`
 and is re-tokenized on the first hybrid query after opening. Pass `index_text=True` to persist
@@ -203,6 +219,7 @@ db.close()
 reopened = LodeDB(path="./data", index_text=True, store_text=False)
 reopened.search("E1234", k=5, mode="hybrid")  # works after reopen, rebuilt from persisted terms
 ```
+</details>
 
 ## GPU-resident index
 
@@ -301,6 +318,9 @@ your host at `lodedb mcp`:
 
 ```bash
 pip install "lodedb[mcp]"
+
+# for coding assistant:
+lodedb mcp install --client claude-code  # or: claude-desktop | cursor | lm-studio | codex | all
 ```
 
 It exposes `lodedb_add`, `lodedb_search`, `lodedb_remove`, and `lodedb_stats`, plus
@@ -313,7 +333,7 @@ the server with `--exclude-text` to return metrics only (this also withdraws `lo
 `--no-store-text` to keep no text on disk at all. `lodedb_stats` is always metrics-only and raw
 query text never leaves the process.
 
-### One command
+### One command install
 
 `lodedb mcp install` writes the correct entry to a client's config for you, so you do not have to
 find the file or hand-write the JSON/TOML:
@@ -385,6 +405,17 @@ See [`examples/mcp_config.json`](examples/mcp_config.json) for a copy-paste star
 - **Durability is `fast` by default.** Commits are *atomic* but not fsync'd. Pass
   `durability="fsync"` (or `--durability fsync` / `LODEDB_DURABILITY=fsync`) to fsync each
   file and its directory on commit for power-loss durability, at some commit-throughput cost.
+- **WAL commit by default for low-latency durable writes.** Each `add`/`remove` appends one
+  framed record to a `<key>.wal` log and a full generation is checkpointed periodically, so a
+  durable single add costs roughly an order of magnitude less than publishing a whole generation
+  per write, into the sqlite-vec/qdrant range (see the comparison up top). The WAL is replayed
+  crash-atomically on reopen (a half-written trailing record is discarded), every writable open
+  folds it into a clean committed generation, and `close()`/`persist()` checkpoint it. WAL is
+  single-writer: a concurrent `open_readonly` reader still loads a consistent committed
+  generation, but the last *checkpointed* one, not the writer's in-flight WAL. Pass
+  `commit_mode="generation"` (or `LODEDB_COMMIT_MODE=generation`) for the classic path that
+  publishes a crash-atomic, MVCC-readable generation on every write; pick it when many
+  out-of-process readers must see each write the instant it commits.
 - **Local filesystems only.** The OS advisory lock is unreliable on NFS/SMB.
 
 ## Limitations

{lodedb-0.2.1 → lodedb-0.3.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "maturin"
 [project]
 name = "lodedb"
 # Static (maturin reads it here); keep in sync with src/lodedb/__init__.py __version__.
-version = "0.2.1"
+version = "0.3.0"
 description = "Local-first, privacy-by-default embedded vector database. Your data never leaves your machine."
 readme = "README.md"
 requires-python = ">=3.11"

{lodedb-0.2.1 → lodedb-0.3.0}/src/lodedb/__init__.py

@@ -19,7 +19,7 @@ from lodedb.local import (
 from lodedb.local.cli import app, main
 
 # Keep in sync with `version` in pyproject.toml (the release workflow asserts they match).
-__version__ = "0.2.1"
+__version__ = "0.3.0"
 
 __all__ = [
     "LOCAL_MODEL_PRESETS",