omnidb-server 0.1.0__tar.gz

omnidb_server-0.1.0/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.4
+Name: omnidb-server
+Version: 0.1.0
+Summary: MCP-native agent memory & retrieval server over SQLite (WAL + FTS5) with an hnswlib vector sidecar.
+License: Apache-2.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: mcp<2,>=1.9
+Requires-Dist: numpy>=1.26
+Requires-Dist: httpx>=0.27
+Provides-Extra: semantic
+Requires-Dist: fastembed>=0.3; extra == "semantic"
+Provides-Extra: hnsw
+Requires-Dist: hnswlib>=0.8; extra == "hnsw"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov>=5; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: hypothesis>=6.100; extra == "dev"
+
+# OmniDB — MCP-native agent memory & retrieval server
+
+A local-first, single-binary-feel **agent memory and retrieval server over SQLite**, exposed to AI
+agents through the Model Context Protocol. This is the revised v1 that came out of the August 2026
+research + adversarial review (see `../OmniDB Vault/`): an MCP memory/retrieval server, not a database.
+
+- **Substrate**: SQLite in WAL mode (`synchronous=NORMAL`, `busy_timeout=5000`) — one directory on disk,
+  copyable as a unit (`omnidb.db` + WAL sidecars + vector index file). The single-file mandate was dropped.
+- **Retrieval**: BM25 (FTS5, porter tokenization) fused with dense vector search via Reciprocal Rank Fusion,
+  then a gentle recency/frequency rerank. Hybrid retrieval runs in-process; SQLite table-valued functions
+  (`VECTOR_SIM` / `MEMORY_RECALL`) need a native extension and are the documented upgrade path for the
+  future Go/Rust port.
+- **Vectors**: hnswlib (M=16, efC=200) inner-product on L2-normalized vectors == cosine, persisted as a
+  sidecar file; exact brute-force fallback always available. Deletions are mark-deleted.
+- **Memory model**: plain relational tables (`episodic | semantic | procedural`) — no bespoke LSM, no fixed
+  decay religion (weights are config). Consolidation distills episodic → semantic via a configurable LLM,
+  with an offline stub when none is set.
+
+## Quickstart
+
+```bash
+pip install -e ".[hnsw,semantic,dev]"   # hnsw/semantic extras optional; falls back gracefully
+pytest
+omnidb-server serve                     # MCP stdio server on $OMNIDB_HOME (default ~/.omnidb)
+python -m omnidb_server serve           # equivalent
+python -m omnidb_server stats
+```
+
+### Claude Desktop / any MCP client
+
+See [`examples/claude-desktop-config.example.json`](examples/claude-desktop-config.example.json)
+— includes the env vars for real (fastembed) semantics and a working `HF_HOME`
+override. Minimal version:
+
+```json
+{
+  "mcpServers": {
+    "omnidb": {
+      "command": "python",
+      "args": ["-m", "omnidb_server", "serve"]
+    }
+  }
+}
+```
+
+## Tools
+
+| Tool | Purpose |
+|---|---|
+| `remember(content, kind, metadata, links)` | persist a memory (episodic/semantic/procedural), optional graph links |
+| `recall(query, k, kind, k_fetch)` | hybrid BM25+dense recall with recency/frequency rerank; pool floors at `k` |
+| `update_memory(memory_id, content, metadata)` | edit content (re-embeds + reindexes via FTS trigger) and/or merge metadata |
+| `search(sql)` | read-only SELECT/WITH escape hatch against SQLite |
+| `related(memory_id, max_hops, label)` | undirected graph walk (recursive CTE, cycle-safe, hops clamped 1-8) |
+| `forget(memory_id)` / `forget_many(ids)` | soft delete one / many |
+| `consolidate(hours)` | episodic → semantic distillation |
+| `stats()` | counts, edges, embedder/index info, dead-ratio, all file sizes |
+
+Error messages carry `OMN-*` codes from the original spec's taxonomy (`[OMN-020]` read-only violation, `[OMN-021]` not found, `[OMN-006]` dimension mismatch, `[OMN-011]` timeout, ...).
+
+## Configuration
+
+Precedence: **CLI flags > environment > `omnidb.toml` > defaults**. The TOML file is looked up in
+the CWD and the data directory (field names mirror the config dataclass):
+
+```toml
+half_life_hours = 168
+rrf_k = 60
+k_fetch = 50
+persist_every_ops = 64      # vector-sidecar fsync debouncing
+dead_compact_ratio = 0.2    # hnsw tombstone ratio triggering rebuild-on-open
+```
+
+## Operations
+
+```bash
+omnidb-server compact --home D:/path/to/data   # force full vector-index rebuild
+python -m omnidb_server stats                  # includes pending_compaction, reconciled_at, WAL size
+omnidb-server check                            # FTS integrity, index drift, orphan edges, WAL size
+omnidb-server purge [--vacuum]                 # hard-delete soft-deleted rows; --vacuum reclaims space
+omnidb-server export D:/backup/memories.jsonl  # dump all memories + edges to JSONL
+omnidb-server import D:/backup/memories.jsonl  # restore from a JSONL dump
+```
+
+## Configuration (environment)
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `OMNIDB_HOME` | `~/.omnidb` | data directory |
+| `OMNIDB_EMBEDDER` | `auto` | `auto`/`hashing` are offline-safe (never download); `fastembed` opts into real semantics (`[semantic]` extra, ~100MB model on first run). Embedder identity is pinned per database. |
+| `OMNIDB_INDEX` | `auto` | `auto` \| `hnsw` \| `brute` |
+| `OMNIDB_HALF_LIFE_HOURS` | `168` | recency half-life for rerank |
+| `OMNIDB_RECENCY_WEIGHT` / `OMNIDB_FREQ_WEIGHT` | `0.15` / `0.05` | rerank weights (RRF base is weight-free) |
+| `OMNIDB_K_FETCH` | `50` | candidates fetched per leg before fusion |
+| `OMNIDB_RRF_K` | `60` | RRF smoothing constant |
+| `OMNIDB_MAX_SEARCH_ROWS` | `500` | row cap for the `search` tool |
+| `OMNIDB_LLM_BASE_URL` / `_MODEL` / `_API_KEY` | unset | OpenAI-compatible endpoint used by `consolidate`; stub when unset. With one configured, a failed call consumes nothing (episodes stay retryable) |
+
+## Benchmarking
+
+```bash
+python bench/baseline.py   # writes bench/results.md
+```
+
+Measures hnswlib (M=16, efC=200) recall@10 vs exact brute force across an ef sweep at
+384/768 dims on clustered synthetic data (1k/10k/100k), plus the SQ8 experiment.
+Findings so far: at 768-dim/10k, SQ8+rerank (0.947) beats raw HNSW at ef=128 (0.837);
+at 100k the clustered-synthetic distribution is strongly pessimistic for graph ANN
+(tight clusters create hub structure), where quantized scoring + f32 rerank stays
+robust (0.90) but at full-scan cost (~120ms). Treat as a floor, not a forecast —
+re-run against your real corpus before drawing conclusions.
+
+## Encryption at rest
+
+OmniDB is local-first and writes only to `OMNIDB_HOME`; it phones home to no one. For
+data-at-rest protection we recommend OS-level disk encryption (BitLocker / FileVault /
+LUKS) — that covers the stolen-disk threat model without adding a dependency or a key
+you must manage.
+
+A SQLCipher swap is documented but deliberately not bundled: encrypting the SQLite file
+would require `pysqlcipher3`, whose Windows wheels for Python 3.14 are effectively
+unobtainable, so shipping it as an extra would break installs rather than protect them.
+If your threat model needs per-file encryption beyond full-disk, compile SQLCipher and
+point this package's connection at it — the store layer is plain `sqlite3` DB-API and
+needs no code changes.
+
+## Limitations (v1, deliberate)
+
+- Single-writer process per home directory (SQLite WAL semantics). The vector sidecar is
+  last-writer-wins: run one server per home; startup reconciliation after a crash between the DB and
+  the index file is deferred.
+- The Python `sqlite3` module cannot register virtual tables, so SQL++-style TVFs arrive with the
+  native port; until then hybrid queries go through the tools.
+- Hashing embedder matches surface forms, not paraphrases — install `fastembed` for real semantics.
+  Embedder identity is pinned per database; switching requires a fresh home.
+- No QPS targets: this serves one agent session locally. Gates are interactive latency, cold start,
+  and RAM at scale (re-baselined empirically at true embedding dimensions, not SIFT-128 folklore).

omnidb_server-0.1.0/README.md

@@ -0,0 +1,137 @@
+# OmniDB — MCP-native agent memory & retrieval server
+
+A local-first, single-binary-feel **agent memory and retrieval server over SQLite**, exposed to AI
+agents through the Model Context Protocol. This is the revised v1 that came out of the August 2026
+research + adversarial review (see `../OmniDB Vault/`): an MCP memory/retrieval server, not a database.
+
+- **Substrate**: SQLite in WAL mode (`synchronous=NORMAL`, `busy_timeout=5000`) — one directory on disk,
+  copyable as a unit (`omnidb.db` + WAL sidecars + vector index file). The single-file mandate was dropped.
+- **Retrieval**: BM25 (FTS5, porter tokenization) fused with dense vector search via Reciprocal Rank Fusion,
+  then a gentle recency/frequency rerank. Hybrid retrieval runs in-process; SQLite table-valued functions
+  (`VECTOR_SIM` / `MEMORY_RECALL`) need a native extension and are the documented upgrade path for the
+  future Go/Rust port.
+- **Vectors**: hnswlib (M=16, efC=200) inner-product on L2-normalized vectors == cosine, persisted as a
+  sidecar file; exact brute-force fallback always available. Deletions are mark-deleted.
+- **Memory model**: plain relational tables (`episodic | semantic | procedural`) — no bespoke LSM, no fixed
+  decay religion (weights are config). Consolidation distills episodic → semantic via a configurable LLM,
+  with an offline stub when none is set.
+
+## Quickstart
+
+```bash
+pip install -e ".[hnsw,semantic,dev]"   # hnsw/semantic extras optional; falls back gracefully
+pytest
+omnidb-server serve                     # MCP stdio server on $OMNIDB_HOME (default ~/.omnidb)
+python -m omnidb_server serve           # equivalent
+python -m omnidb_server stats
+```
+
+### Claude Desktop / any MCP client
+
+See [`examples/claude-desktop-config.example.json`](examples/claude-desktop-config.example.json)
+— includes the env vars for real (fastembed) semantics and a working `HF_HOME`
+override. Minimal version:
+
+```json
+{
+  "mcpServers": {
+    "omnidb": {
+      "command": "python",
+      "args": ["-m", "omnidb_server", "serve"]
+    }
+  }
+}
+```
+
+## Tools
+
+| Tool | Purpose |
+|---|---|
+| `remember(content, kind, metadata, links)` | persist a memory (episodic/semantic/procedural), optional graph links |
+| `recall(query, k, kind, k_fetch)` | hybrid BM25+dense recall with recency/frequency rerank; pool floors at `k` |
+| `update_memory(memory_id, content, metadata)` | edit content (re-embeds + reindexes via FTS trigger) and/or merge metadata |
+| `search(sql)` | read-only SELECT/WITH escape hatch against SQLite |
+| `related(memory_id, max_hops, label)` | undirected graph walk (recursive CTE, cycle-safe, hops clamped 1-8) |
+| `forget(memory_id)` / `forget_many(ids)` | soft delete one / many |
+| `consolidate(hours)` | episodic → semantic distillation |
+| `stats()` | counts, edges, embedder/index info, dead-ratio, all file sizes |
+
+Error messages carry `OMN-*` codes from the original spec's taxonomy (`[OMN-020]` read-only violation, `[OMN-021]` not found, `[OMN-006]` dimension mismatch, `[OMN-011]` timeout, ...).
+
+## Configuration
+
+Precedence: **CLI flags > environment > `omnidb.toml` > defaults**. The TOML file is looked up in
+the CWD and the data directory (field names mirror the config dataclass):
+
+```toml
+half_life_hours = 168
+rrf_k = 60
+k_fetch = 50
+persist_every_ops = 64      # vector-sidecar fsync debouncing
+dead_compact_ratio = 0.2    # hnsw tombstone ratio triggering rebuild-on-open
+```
+
+## Operations
+
+```bash
+omnidb-server compact --home D:/path/to/data   # force full vector-index rebuild
+python -m omnidb_server stats                  # includes pending_compaction, reconciled_at, WAL size
+omnidb-server check                            # FTS integrity, index drift, orphan edges, WAL size
+omnidb-server purge [--vacuum]                 # hard-delete soft-deleted rows; --vacuum reclaims space
+omnidb-server export D:/backup/memories.jsonl  # dump all memories + edges to JSONL
+omnidb-server import D:/backup/memories.jsonl  # restore from a JSONL dump
+```
+
+## Configuration (environment)
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `OMNIDB_HOME` | `~/.omnidb` | data directory |
+| `OMNIDB_EMBEDDER` | `auto` | `auto`/`hashing` are offline-safe (never download); `fastembed` opts into real semantics (`[semantic]` extra, ~100MB model on first run). Embedder identity is pinned per database. |
+| `OMNIDB_INDEX` | `auto` | `auto` \| `hnsw` \| `brute` |
+| `OMNIDB_HALF_LIFE_HOURS` | `168` | recency half-life for rerank |
+| `OMNIDB_RECENCY_WEIGHT` / `OMNIDB_FREQ_WEIGHT` | `0.15` / `0.05` | rerank weights (RRF base is weight-free) |
+| `OMNIDB_K_FETCH` | `50` | candidates fetched per leg before fusion |
+| `OMNIDB_RRF_K` | `60` | RRF smoothing constant |
+| `OMNIDB_MAX_SEARCH_ROWS` | `500` | row cap for the `search` tool |
+| `OMNIDB_LLM_BASE_URL` / `_MODEL` / `_API_KEY` | unset | OpenAI-compatible endpoint used by `consolidate`; stub when unset. With one configured, a failed call consumes nothing (episodes stay retryable) |
+
+## Benchmarking
+
+```bash
+python bench/baseline.py   # writes bench/results.md
+```
+
+Measures hnswlib (M=16, efC=200) recall@10 vs exact brute force across an ef sweep at
+384/768 dims on clustered synthetic data (1k/10k/100k), plus the SQ8 experiment.
+Findings so far: at 768-dim/10k, SQ8+rerank (0.947) beats raw HNSW at ef=128 (0.837);
+at 100k the clustered-synthetic distribution is strongly pessimistic for graph ANN
+(tight clusters create hub structure), where quantized scoring + f32 rerank stays
+robust (0.90) but at full-scan cost (~120ms). Treat as a floor, not a forecast —
+re-run against your real corpus before drawing conclusions.
+
+## Encryption at rest
+
+OmniDB is local-first and writes only to `OMNIDB_HOME`; it phones home to no one. For
+data-at-rest protection we recommend OS-level disk encryption (BitLocker / FileVault /
+LUKS) — that covers the stolen-disk threat model without adding a dependency or a key
+you must manage.
+
+A SQLCipher swap is documented but deliberately not bundled: encrypting the SQLite file
+would require `pysqlcipher3`, whose Windows wheels for Python 3.14 are effectively
+unobtainable, so shipping it as an extra would break installs rather than protect them.
+If your threat model needs per-file encryption beyond full-disk, compile SQLCipher and
+point this package's connection at it — the store layer is plain `sqlite3` DB-API and
+needs no code changes.
+
+## Limitations (v1, deliberate)
+
+- Single-writer process per home directory (SQLite WAL semantics). The vector sidecar is
+  last-writer-wins: run one server per home; startup reconciliation after a crash between the DB and
+  the index file is deferred.
+- The Python `sqlite3` module cannot register virtual tables, so SQL++-style TVFs arrive with the
+  native port; until then hybrid queries go through the tools.
+- Hashing embedder matches surface forms, not paraphrases — install `fastembed` for real semantics.
+  Embedder identity is pinned per database; switching requires a fresh home.
+- No QPS targets: this serves one agent session locally. Gates are interactive latency, cold start,
+  and RAM at scale (re-baselined empirically at true embedding dimensions, not SIFT-128 folklore).

omnidb_server-0.1.0/omnidb_server/__init__.py

@@ -0,0 +1,8 @@
+"""OmniDB: an MCP-native agent memory & retrieval server over SQLite."""
+
+from .config import Config
+from .store import MemoryStore, StoreError
+
+__version__ = "0.1.0"
+
+__all__ = ["Config", "MemoryStore", "StoreError", "__version__"]

omnidb_server-0.1.0/omnidb_server/__main__.py

@@ -0,0 +1,6 @@
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

omnidb_server-0.1.0/omnidb_server/cli.py

@@ -0,0 +1,88 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+from .config import Config
+from .consolidate import run_consolidation
+from .server import build_stack
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="omnidb-server",
+                                     description="MCP-native agent memory & retrieval server over SQLite")
+    parser.add_argument("--home", help="data directory (default $OMNIDB_HOME or ~/.omnidb)")
+    # accept --home after the subcommand too; SUPPRESS keeps the subparser
+    # from overwriting a top-level value with its own default
+    common = argparse.ArgumentParser(add_help=False)
+    common.add_argument("--home", default=argparse.SUPPRESS)
+    sub = parser.add_subparsers(dest="command", required=True)
+    sub.add_parser("serve", parents=[common], help="run the MCP stdio server")
+    sub.add_parser("stats", parents=[common], help="print database statistics as JSON")
+    sub.add_parser("compact", parents=[common],
+                   help="force a full vector-index rebuild (compaction / repair)")
+    p_consol = sub.add_parser("consolidate", parents=[common],
+                              help="distill episodic memories into semantic ones")
+    p_consol.add_argument("--hours", type=float, default=24.0)
+    sub.add_parser("check", parents=[common],
+                   help="run integrity checks (FTS, index drift, orphan edges)")
+    p_purge = sub.add_parser("purge", parents=[common],
+                             help="hard-delete soft-deleted memories")
+    p_purge.add_argument("--vacuum", action="store_true",
+                         help="reclaim disk space with VACUUM afterwards")
+    p_export = sub.add_parser("export", parents=[common],
+                              help="dump live memories and edges to a JSONL file")
+    p_export.add_argument("path")
+    p_import = sub.add_parser("import", parents=[common],
+                              help="restore a JSONL export into an empty home")
+    p_import.add_argument("path")
+
+    args = parser.parse_args(argv)
+    home = getattr(args, "home", None)
+
+    try:
+        if args.command == "serve":
+            from .server import serve
+            serve(home)
+            return 0
+
+        cfg = Config.load(home)
+        store = build_stack(cfg)
+        try:
+            if args.command == "stats":
+                print(json.dumps(store.stats(), indent=2))
+            elif args.command == "consolidate":
+                print(json.dumps(run_consolidation(store, args.hours), indent=2))
+            elif args.command == "compact":
+                print(json.dumps(store.rebuild_index(reason="manual"), indent=2))
+            elif args.command == "check":
+                print(json.dumps(store.check(), indent=2))
+            elif args.command == "purge":
+                print(json.dumps(store.purge(do_vacuum=args.vacuum), indent=2))
+            elif args.command == "export":
+                from .portability import export_jsonl
+
+                print(json.dumps(export_jsonl(store, args.path), indent=2))
+            elif args.command == "import":
+                from .portability import import_jsonl
+
+                print(json.dumps(import_jsonl(store, args.path), indent=2))
+        finally:
+            store.close()
+    except Exception as exc:
+        from .config import ConfigError
+        from .db import DbError
+        from .embeddings import EmbeddingError
+        from .store import StoreError
+        from .vector_index import VectorIndexError
+        if isinstance(exc, (StoreError, DbError, EmbeddingError,
+                            VectorIndexError, ConfigError)):
+            print(f"omnidb-server: error: {exc}", file=sys.stderr)
+            return 2
+        raise
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

omnidb_server-0.1.0/omnidb_server/config.py

@@ -0,0 +1,105 @@
+from __future__ import annotations
+
+import os
+import tomllib
+from dataclasses import MISSING, dataclass, fields
+from pathlib import Path
+
+_TOML_NAME = "omnidb.toml"
+
+
+class ConfigError(RuntimeError):
+    pass
+
+
+def _env(name: str) -> str | None:
+    value = os.environ.get(name)
+    return value if value not in (None, "") else None
+
+
+def _coerce(raw, template):
+    """Coerce a TOML/env scalar to the field's declared type."""
+    if template is None or isinstance(template, str):
+        return raw if isinstance(raw, str) else str(raw)
+    if isinstance(template, bool):
+        return str(raw).strip().lower() in ("1", "true", "yes", "on")
+    if isinstance(template, int):
+        return int(float(raw))
+    if isinstance(template, float):
+        return float(raw)
+    return raw
+
+
+def _default_of(field):
+    return None if field.default is MISSING else field.default
+
+
+def _read_toml(path: Path) -> dict:
+    with path.open("rb") as handle:
+        return tomllib.load(handle)
+
+
+@dataclass
+class Config:
+    home: Path
+    embedder: str = "auto"  # auto | hashing | fastembed
+    vector_index: str = "auto"  # auto | hnsw | brute
+    half_life_hours: float = 168.0
+    recency_weight: float = 0.15
+    freq_weight: float = 0.05
+    rrf_k: int = 60
+    k_fetch: int = 50
+    max_search_rows: int = 500
+    persist_every_ops: int = 64
+    persist_every_s: float = 0.25
+    dead_compact_ratio: float = 0.2
+
+    llm_base_url: str | None = None
+    llm_model: str | None = None
+    llm_api_key: str | None = None
+
+    @staticmethod
+    def load(home: str | Path | None = None) -> "Config":
+        """Precedence: CLI arg > environment > omnidb.toml > built-in defaults.
+
+        omnidb.toml is looked up in the CWD and in the resolved home directory
+        (the home copy wins). Field names in the file mirror the dataclass.
+        """
+        arg_home = Path(home) if home is not None else None
+        env_home_value = _env("OMNIDB_HOME")
+        env_home = Path(env_home_value) if env_home_value else None
+        resolved_home = arg_home or env_home or Path.home() / ".omnidb"
+
+        merged: dict = {}
+        cwd_toml = Path.cwd() / _TOML_NAME
+        if cwd_toml.is_file():
+            merged.update(_read_toml(cwd_toml))
+        # a bare "home" key in the CWD file can redirect the home directory
+        # when neither a CLI arg nor the environment pinned one
+        if not arg_home and not env_home and isinstance(merged.get("home"), str):
+            resolved_home = Path(merged["home"])
+        home_toml = resolved_home / _TOML_NAME
+        if home_toml != cwd_toml and home_toml.is_file():
+            merged.update(_read_toml(home_toml))
+
+        known = {f.name for f in fields(Config)}
+        unknown = set(merged) - known
+        if unknown:
+            raise ConfigError(f"unknown keys in {_TOML_NAME}: {sorted(unknown)}")
+
+        values: dict = {}
+        for field in fields(Config):
+            if field.name == "home":
+                continue
+            default = _default_of(field)
+            if field.name in merged:
+                values[field.name] = _coerce(merged[field.name], default)
+            env_value = _env("OMNIDB_" + field.name.upper())
+            if env_value is not None:
+                values[field.name] = _coerce(env_value, default)
+
+        values["home"] = arg_home or env_home or resolved_home
+        return Config(**values)
+
+    # backward-compatible alias used across tests/CLI
+    from_env = load