lodedb 0.1.2__tar.gz → 0.2.0__tar.gz

{lodedb-0.1.2 → lodedb-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lodedb
-Version: 0.1.2
+Version: 0.2.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3 :: Only
@@ -121,6 +121,9 @@ for hits in db.search_many(["fox", "dog"], k=5):   # batched; the GPU can serve
 db.search("fox", k=5, filter={"topic": "animals"})                      # bare scalar = exact
 db.search("fox", k=5, filter={"$or": [{"topic": "animals"}, {"year": {"$gte": 2020}}]})
 
+# hybrid search: vector recall plus exact lexical matches the embedding misses
+db.search("E1234", k=5, mode="hybrid")     # surfaces error codes, serials, dates in the body
+
 db.get(fox)     # -> "the quick brown fox jumps"  (text retained by default)
 db.persist()    # durable .tvim/.tvd/.jsd snapshot; replays on reopen
 ```
@@ -139,6 +142,68 @@ reader.search("fox", k=5)                 # reads a committed snapshot
 reader.add("nope")                        # raises ReadOnlyError
 ```
 
+## Hybrid search
+
+Vector search alone misses exact tokens the embedding does not capture: error codes
+(`E1234`), serial numbers (`ABC-123`), dates (`2024-01-15`). Pass `mode="hybrid"` to run a
+lexical BM25 ranker alongside the vector scan and fuse the two ranked lists with Reciprocal
+Rank Fusion. The lexical ranker matches those tokens exactly, so a document whose body carries
+the code is recovered even when the embedding ranks it nowhere near the top.
+
+```python
+db.add("the turbine tripped and reported fault code E1234 overnight", metadata={"unit": "t3"})
+
+db.search("E1234", k=5)                 # mode="vector" (default): may miss the exact code
+db.search("E1234", k=5, mode="hybrid")  # BM25 + RRF: surfaces it in the top-k
+db.search("E1234", k=5, mode="lexical") # BM25 ranking alone, no vector scan
+```
+
+### Prerequisites
+
+`mode="hybrid"` and `mode="lexical"` build a BM25 index over your text, so they need a text
+source enabled when you open the database. `mode="vector"` (the default) needs nothing.
+
+| Mode | Enable | Source of the BM25 index |
+| --- | --- | --- |
+| `"vector"` (default) | nothing | not used |
+| `"hybrid"`, `"lexical"` | `store_text=True` (on by default) | rebuilt in memory from the retained raw text |
+| `"hybrid"`, `"lexical"` | `index_text=True` | a durable on-disk postings store, no raw text required |
+
+Either source is enough and you can enable both. `store_text=True` is the default, so hybrid
+works out of the box. With neither source enabled, a hybrid or lexical query raises a clear,
+actionable error rather than silently degrading.
+
+### How it works
+
+A `filter` constrains both rankers, so `mode="hybrid"` with a filter returns the true top-k of
+the matching subset. The vector half of a hybrid query runs on the same scan as `mode="vector"`,
+including the GPU-resident batch scan that serves `search_many`; only the BM25 ranking and the
+fusion run on the CPU, and the vector kernel and on-disk format are untouched. The serving BM25
+index lives in memory and is maintained incrementally: a small mutation folds just the changed
+chunks into the existing index, so a single `add` never forces a full re-tokenization.
+
+### Durable lexical index (`index_text=True`)
+
+By default the BM25 index is rebuilt from the retained raw text, so it needs `store_text=True`
+and is re-tokenized on the first hybrid query after opening. Pass `index_text=True` to persist
+it instead: each document's per-chunk terms are captured at `add` time into a dedicated `.tvlex`
+sidecar (a base plus a `.lxd` delta journal, committed O(changed) per write), so hybrid and
+lexical search survive a reopen rebuilt straight from the persisted terms, with no
+re-tokenization and without retaining the raw text at all. The `.tvlex` sidecar holds
+payload-derived terms only and, like the raw-text sidecar, never reaches the redacted artifacts
+or telemetry. The tokenizer lowercases and splits on punctuation but keeps code-like tokens
+whole, so `ABC-123` and `2024-01-15` stay findable as single tokens. Reopen with the same
+`index_text` value you wrote with.
+
+```python
+db = LodeDB(path="./data", index_text=True, store_text=False)  # durable lexical index, no raw text
+db.add("the turbine tripped and reported fault code E1234 overnight")
+db.close()
+
+reopened = LodeDB(path="./data", index_text=True, store_text=False)
+reopened.search("E1234", k=5, mode="hybrid")  # works after reopen, rebuilt from persisted terms
+```
+
 ## GPU-resident index
 
 With the `[gpu]` extra on a CUDA host, LodeDB reconstructs the compact index into an fp16
@@ -255,6 +320,10 @@ lodedb benchmark   # local, metrics-only benchmark
   faster on the measured M1, so the MPS scan stays off by default until newer Apple GPUs are
   re-measured.
 - **Single queries run on the CPU**; the GPU serves batched `search_many`.
+- **Hybrid search needs a lexical source and serves from memory.** `mode="hybrid"`/`"lexical"`
+  need either `store_text=True` (the index built from raw text) or `index_text=True` (a durable
+  `.tvlex` postings store that survives reopens without raw text). The serving index is held in
+  memory and maintained incrementally across mutations.
 - **Single writer per path.** One writer at a time (many concurrent readers), with no live
   cross-process refresh, on local filesystems only. See
   [Concurrency & durability](#concurrency--durability).

{lodedb-0.1.2 → lodedb-0.2.0}/README.md

@@ -87,6 +87,9 @@ for hits in db.search_many(["fox", "dog"], k=5):   # batched; the GPU can serve
 db.search("fox", k=5, filter={"topic": "animals"})                      # bare scalar = exact
 db.search("fox", k=5, filter={"$or": [{"topic": "animals"}, {"year": {"$gte": 2020}}]})
 
+# hybrid search: vector recall plus exact lexical matches the embedding misses
+db.search("E1234", k=5, mode="hybrid")     # surfaces error codes, serials, dates in the body
+
 db.get(fox)     # -> "the quick brown fox jumps"  (text retained by default)
 db.persist()    # durable .tvim/.tvd/.jsd snapshot; replays on reopen
 ```
@@ -105,6 +108,68 @@ reader.search("fox", k=5)                 # reads a committed snapshot
 reader.add("nope")                        # raises ReadOnlyError
 ```
 
+## Hybrid search
+
+Vector search alone misses exact tokens the embedding does not capture: error codes
+(`E1234`), serial numbers (`ABC-123`), dates (`2024-01-15`). Pass `mode="hybrid"` to run a
+lexical BM25 ranker alongside the vector scan and fuse the two ranked lists with Reciprocal
+Rank Fusion. The lexical ranker matches those tokens exactly, so a document whose body carries
+the code is recovered even when the embedding ranks it nowhere near the top.
+
+```python
+db.add("the turbine tripped and reported fault code E1234 overnight", metadata={"unit": "t3"})
+
+db.search("E1234", k=5)                 # mode="vector" (default): may miss the exact code
+db.search("E1234", k=5, mode="hybrid")  # BM25 + RRF: surfaces it in the top-k
+db.search("E1234", k=5, mode="lexical") # BM25 ranking alone, no vector scan
+```
+
+### Prerequisites
+
+`mode="hybrid"` and `mode="lexical"` build a BM25 index over your text, so they need a text
+source enabled when you open the database. `mode="vector"` (the default) needs nothing.
+
+| Mode | Enable | Source of the BM25 index |
+| --- | --- | --- |
+| `"vector"` (default) | nothing | not used |
+| `"hybrid"`, `"lexical"` | `store_text=True` (on by default) | rebuilt in memory from the retained raw text |
+| `"hybrid"`, `"lexical"` | `index_text=True` | a durable on-disk postings store, no raw text required |
+
+Either source is enough and you can enable both. `store_text=True` is the default, so hybrid
+works out of the box. With neither source enabled, a hybrid or lexical query raises a clear,
+actionable error rather than silently degrading.
+
+### How it works
+
+A `filter` constrains both rankers, so `mode="hybrid"` with a filter returns the true top-k of
+the matching subset. The vector half of a hybrid query runs on the same scan as `mode="vector"`,
+including the GPU-resident batch scan that serves `search_many`; only the BM25 ranking and the
+fusion run on the CPU, and the vector kernel and on-disk format are untouched. The serving BM25
+index lives in memory and is maintained incrementally: a small mutation folds just the changed
+chunks into the existing index, so a single `add` never forces a full re-tokenization.
+
+### Durable lexical index (`index_text=True`)
+
+By default the BM25 index is rebuilt from the retained raw text, so it needs `store_text=True`
+and is re-tokenized on the first hybrid query after opening. Pass `index_text=True` to persist
+it instead: each document's per-chunk terms are captured at `add` time into a dedicated `.tvlex`
+sidecar (a base plus a `.lxd` delta journal, committed O(changed) per write), so hybrid and
+lexical search survive a reopen rebuilt straight from the persisted terms, with no
+re-tokenization and without retaining the raw text at all. The `.tvlex` sidecar holds
+payload-derived terms only and, like the raw-text sidecar, never reaches the redacted artifacts
+or telemetry. The tokenizer lowercases and splits on punctuation but keeps code-like tokens
+whole, so `ABC-123` and `2024-01-15` stay findable as single tokens. Reopen with the same
+`index_text` value you wrote with.
+
+```python
+db = LodeDB(path="./data", index_text=True, store_text=False)  # durable lexical index, no raw text
+db.add("the turbine tripped and reported fault code E1234 overnight")
+db.close()
+
+reopened = LodeDB(path="./data", index_text=True, store_text=False)
+reopened.search("E1234", k=5, mode="hybrid")  # works after reopen, rebuilt from persisted terms
+```
+
 ## GPU-resident index
 
 With the `[gpu]` extra on a CUDA host, LodeDB reconstructs the compact index into an fp16
@@ -221,6 +286,10 @@ lodedb benchmark   # local, metrics-only benchmark
   faster on the measured M1, so the MPS scan stays off by default until newer Apple GPUs are
   re-measured.
 - **Single queries run on the CPU**; the GPU serves batched `search_many`.
+- **Hybrid search needs a lexical source and serves from memory.** `mode="hybrid"`/`"lexical"`
+  need either `store_text=True` (the index built from raw text) or `index_text=True` (a durable
+  `.tvlex` postings store that survives reopens without raw text). The serving index is held in
+  memory and maintained incrementally across mutations.
 - **Single writer per path.** One writer at a time (many concurrent readers), with no live
   cross-process refresh, on local filesystems only. See
   [Concurrency & durability](#concurrency--durability).

{lodedb-0.1.2 → lodedb-0.2.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "maturin"
 [project]
 name = "lodedb"
 # Static (maturin reads it here); keep in sync with src/lodedb/__init__.py __version__.
-version = "0.1.2"
+version = "0.2.0"
 description = "Local-first, privacy-by-default embedded vector database. Your data never leaves your machine."
 readme = "README.md"
 requires-python = ">=3.11"

{lodedb-0.1.2 → lodedb-0.2.0}/src/lodedb/__init__.py

@@ -19,7 +19,7 @@ from lodedb.local import (
 from lodedb.local.cli import app, main
 
 # Keep in sync with `version` in pyproject.toml (the release workflow asserts they match).
-__version__ = "0.1.2"
+__version__ = "0.2.0"
 
 __all__ = [
     "LOCAL_MODEL_PRESETS",

{lodedb-0.1.2 → lodedb-0.2.0}/src/lodedb/engine/_commit_manifest.py

@@ -17,6 +17,8 @@ at which a full base was last written):
     <key>.gen/g<epoch>.tvim.tvim-delta/  its ``.tvd`` vector journal
     <key>.gen/g<epoch>.tvtext          opt-in raw-text base (full id->text map)
     <key>.gen/g<epoch>.tvtext.tvtext-delta/  its ``.txd`` raw-text journal
+    <key>.gen/g<epoch>.tvlex           opt-in lexical-index base (id->token lists)
+    <key>.gen/g<epoch>.tvlex.tvlex-delta/  its ``.lxd`` lexical-index journal
 
 and a single top-level pointer:
 
@@ -89,6 +91,12 @@ def base_tvtext_path(persistence_dir: str | Path, index_key: str, epoch: int) ->
     return generation_dir(persistence_dir, index_key) / f"g{int(epoch)}.tvtext"
 
 
+def base_tvlex_path(persistence_dir: str | Path, index_key: str, epoch: int) -> Path:
+    """Returns the lexical-index base path for one index key and base epoch."""
+
+    return generation_dir(persistence_dir, index_key) / f"g{int(epoch)}.tvlex"
+
+
 def is_commit_manifest_name(name: str) -> bool:
     """Returns whether a top-level file name is a root commit manifest."""
 
@@ -173,13 +181,19 @@ def build_commit_body(
     tvim_manifest: dict[str, Any] | None,
     tvim_present: bool,
     tvtext_manifest: dict[str, Any] | None,
+    tvlex_manifest: dict[str, Any] | None = None,
 ) -> dict[str, Any]:
     """Assembles a root-manifest body capturing one consistent committed generation.
 
     ``tvtext_manifest`` is the raw-text journal manifest (base + delta segments)
     for this committed generation, or ``None`` when raw-text storage is disabled
-    or the index holds no text. It is pinned by the root exactly like the JSON
-    and TurboVec manifests, so raw text commits and rolls back atomically.
+    or the index holds no text. ``tvlex_manifest`` is the analogous lexical-index
+    (BM25 postings) journal manifest, or ``None`` when lexical-index persistence
+    is disabled or the index holds no tokens. Both are pinned by the root exactly
+    like the JSON and TurboVec manifests, so the text sidecar and the lexical
+    sidecar commit and roll back atomically with the generation. ``tvlex`` is an
+    optional key read via ``.get`` (like ``tvtext``), so an older reader that
+    does not know it ignores it.
     """
 
     return {
@@ -192,6 +206,7 @@ def build_commit_body(
         "tvim": tvim_manifest,
         "tvim_present": bool(tvim_present),
         "tvtext": tvtext_manifest,
+        "tvlex": tvlex_manifest,
     }
 
 

lodedb-0.2.0/src/lodedb/engine/_filter_plan.py

@@ -0,0 +1,205 @@
+"""Set-based resolution of a validated metadata filter to a document-id set.
+
+Given a generation's in-memory per-field value indexes, this resolves a
+*validated* metadata filter to the matching document-id set using set operations
+and bisect, instead of the O(corpus) per-document compiled-matcher scan that
+``LodeEngine._scan_filter_allowlist`` runs. Comparison semantics mirror
+``_predicate`` exactly: this module reuses its number parsing and ordered
+comparators, so an index-resolved set is identical to the compiled-matcher
+result. (That identity is asserted in tests against the reference scan, which is
+retained as the oracle.)
+
+Stdlib-only; it must not import ``core``. ``_MetadataPostingIndex`` uses it for
+both filtered search (the document set is expanded to chunk ids) and filtered
+enumeration / count (the document set is used directly).
+"""
+
+from __future__ import annotations
+
+import bisect
+from collections.abc import Mapping
+from typing import Any
+
+from lodedb.engine._predicate import _ORDERED_CMP, _as_number
+
+
+class FieldIndex:
+    """Per-key value index for one metadata field within a generation.
+
+    Holds the docs carrying the key, a value -> docs map, and the field's numeric
+    values sorted for bisect (with the non-numeric values kept aside so a numeric
+    range query does not have to scan every distinct value).
+    """
+
+    __slots__ = ("docs", "value_docs", "_num_keys", "_num_values", "_nonnumeric_values")
+
+    def __init__(self) -> None:
+        self.docs: set[str] = set()
+        self.value_docs: dict[str, set[str]] = {}
+        self._num_keys: list[float] = []
+        self._num_values: list[str] = []
+        self._nonnumeric_values: list[str] = []
+
+    def finalize(self) -> None:
+        """Partitions distinct values into a sorted numeric array + the rest."""
+
+        numeric: list[tuple[float, str]] = []
+        nonnumeric: list[str] = []
+        for value in self.value_docs:
+            number = _as_number(value)
+            if number is None:
+                nonnumeric.append(value)
+            else:
+                numeric.append((number, value))
+        numeric.sort()
+        self._num_keys = [number for number, _ in numeric]
+        self._num_values = [value for _, value in numeric]
+        self._nonnumeric_values = nonnumeric
+
+    def numeric_values_satisfying(self, op: str, operand_number: float) -> list[str]:
+        """Returns value strings whose numeric value satisfies ``op operand`` (bisect)."""
+
+        keys = self._num_keys
+        if op == "$gt":
+            return self._num_values[bisect.bisect_right(keys, operand_number):]
+        if op == "$gte":
+            return self._num_values[bisect.bisect_left(keys, operand_number):]
+        if op == "$lt":
+            return self._num_values[: bisect.bisect_left(keys, operand_number)]
+        return self._num_values[: bisect.bisect_right(keys, operand_number)]  # $lte
+
+
+_EMPTY_FIELD = FieldIndex()
+
+
+def build_field_indexes(
+    document_metadata: Mapping[str, Mapping[str, str]],
+) -> tuple[dict[str, FieldIndex], set[str]]:
+    """Builds per-field value indexes and the full document-id set for a generation."""
+
+    fields: dict[str, FieldIndex] = {}
+    all_docs: set[str] = set()
+    for document_id, metadata in document_metadata.items():
+        doc = str(document_id)
+        all_docs.add(doc)
+        for key, value in metadata.items():
+            field = fields.get(str(key))
+            if field is None:
+                field = FieldIndex()
+                fields[str(key)] = field
+            field.docs.add(doc)
+            field.value_docs.setdefault(str(value), set()).add(doc)
+    for field in fields.values():
+        field.finalize()
+    return fields, all_docs
+
+
+def resolve(
+    metadata_filter: Mapping[str, Any],
+    fields: Mapping[str, FieldIndex],
+    all_docs: set[str],
+) -> set[str]:
+    """Resolves a validated metadata filter to the matching document-id set.
+
+    Entries of a node are AND-ed (Mongo semantics, matching ``_predicate``);
+    ``$and``/``$or``/``$not`` recurse. Returns a fresh set the caller may mutate.
+    """
+
+    result: set[str] | None = None
+    for key, spec in metadata_filter.items():
+        if key == "$and":
+            docs = _resolve_and(spec, fields, all_docs)
+        elif key == "$or":
+            docs = set()
+            for sub in spec:
+                docs |= resolve(sub, fields, all_docs)
+        elif key == "$not":
+            docs = all_docs - resolve(spec, fields, all_docs)
+        else:
+            docs = _resolve_field(key, spec, fields, all_docs)
+        result = docs if result is None else (result & docs)
+        if not result:
+            return set()
+    return set(all_docs) if result is None else result
+
+
+def _resolve_and(
+    subs: list[Mapping[str, Any]],
+    fields: Mapping[str, FieldIndex],
+    all_docs: set[str],
+) -> set[str]:
+    result: set[str] | None = None
+    for sub in subs:
+        docs = resolve(sub, fields, all_docs)
+        result = docs if result is None else (result & docs)
+        if not result:
+            return set()
+    return set(all_docs) if result is None else result
+
+
+def _resolve_field(
+    field: str,
+    spec: Any,
+    fields: Mapping[str, FieldIndex],
+    all_docs: set[str],
+) -> set[str]:
+    index = fields.get(field, _EMPTY_FIELD)
+    if not isinstance(spec, Mapping):
+        # Bare scalar is exact-match sugar for $eq.
+        return set(index.value_docs.get(str(spec), ()))
+    result: set[str] | None = None
+    for op, operand in spec.items():
+        docs = _resolve_operator(op, operand, index, all_docs)
+        result = docs if result is None else (result & docs)
+        if not result:
+            return set()
+    return set(all_docs) if result is None else result
+
+
+def _resolve_operator(
+    op: str,
+    operand: Any,
+    index: FieldIndex,
+    all_docs: set[str],
+) -> set[str]:
+    if op == "$eq":
+        return set(index.value_docs.get(str(operand), ()))
+    if op == "$ne":
+        # A missing key satisfies $ne (its absent value is never equal to operand).
+        return all_docs - index.value_docs.get(str(operand), set())
+    if op == "$in":
+        docs: set[str] = set()
+        for value in operand:
+            docs |= index.value_docs.get(str(value), set())
+        return docs
+    if op == "$nin":
+        excluded: set[str] = set()
+        for value in operand:
+            excluded |= index.value_docs.get(str(value), set())
+        return all_docs - excluded
+    if op == "$exists":
+        return set(index.docs) if operand else (all_docs - index.docs)
+    # Ordered ($gt/$gte/$lt/$lte): a missing key never satisfies an ordered op.
+    return _resolve_ordered(op, str(operand), index)
+
+
+def _resolve_ordered(op: str, operand: str, index: FieldIndex) -> set[str]:
+    compare = _ORDERED_CMP[op]
+    operand_number = _as_number(operand)
+    docs: set[str] = set()
+    if operand_number is None:
+        # Non-numeric operand: every comparison is lexicographic (matching
+        # _predicate), so compare each stored value string to the operand.
+        for value, value_docs in index.value_docs.items():
+            if compare(value, operand):
+                docs |= value_docs
+        return docs
+    # Numeric operand: numeric-parseable stored values compare numerically
+    # (resolved by bisect); non-numeric stored values compare lexicographically
+    # against the operand string (kept to the small non-numeric set).
+    for value in index.numeric_values_satisfying(op, operand_number):
+        docs |= index.value_docs[value]
+    for value in index._nonnumeric_values:
+        if compare(value, operand):
+            docs |= index.value_docs[value]
+    return docs