markdown-memory-vec 0.1.0__py3-none-any.whl

markdown_memory_vec-0.1.0.dist-info/METADATA

@@ -0,0 +1,219 @@
+Metadata-Version: 2.4
+Name: markdown-memory-vec
+Version: 0.1.0
+Summary: Lightweight vector search for Markdown-based memory systems. Chunk, embed, index, and hybrid-search your .md knowledge base with sqlite-vec.
+Project-URL: Homepage, https://github.com/aigente/markdown-memory-vec
+Project-URL: Repository, https://github.com/aigente/markdown-memory-vec
+Author-email: Aigente <dev@aigente.io>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: RAG,embeddings,markdown,memory,semantic-search,sqlite-vec,vector-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10.0; extra == 'dev'
+Requires-Dist: pyright>=1.1.380; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Provides-Extra: vector
+Requires-Dist: sentence-transformers>=3.0; extra == 'vector'
+Requires-Dist: sqlite-vec>=0.1.6; extra == 'vector'
+Description-Content-Type: text/markdown
+
+# markdown-memory-vec
+
+Lightweight vector search for Markdown-based memory systems. Chunk, embed, index, and hybrid-search your `.md` knowledge base with sqlite-vec.
+
+<!-- Badges -->
+<!-- [![PyPI version](https://badge.fury.io/py/markdown-memory-vec.svg)](https://pypi.org/project/markdown-memory-vec/) -->
+<!-- [![Python](https://img.shields.io/pypi/pyversions/markdown-memory-vec.svg)](https://pypi.org/project/markdown-memory-vec/) -->
+<!-- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -->
+
+## Features
+
+- **Markdown-native**: YAML frontmatter parsing for metadata (importance, type, tags)
+- **Smart chunking**: Paragraph-aware splitting with configurable overlap (~400 tokens, 80 overlap)
+- **SHA-256 dedup**: Never re-embed unchanged content — incremental indexing is fast
+- **Hybrid search**: Combines semantic similarity (α), importance weighting (β), and temporal decay (γ)
+- **Zero-copy storage**: sqlite-vec KNN search with cosine distance in a single `.db` file
+
+## Quick Start
+
+### Installation
+
+```bash
+# Core only (YAML parsing, chunking, interfaces)
+pip install markdown-memory-vec
+
+# With vector search support (sqlite-vec + sentence-transformers)
+pip install 'markdown-memory-vec[vector]'
+```
+
+### Python API
+
+```python
+from memory_vec import MemoryVectorService
+
+svc = MemoryVectorService("/path/to/project")
+svc.rebuild_index()                        # Full index build
+results = svc.search("how to deploy")      # Hybrid search
+svc.close()
+```
+
+### Low-level API
+
+```python
+from memory_vec import (
+    SqliteVecStore,
+    SentenceTransformerEmbedder,
+    MemoryIndexer,
+    HybridSearchService,
+)
+
+store = SqliteVecStore("memory.db")
+store.ensure_tables()
+
+embedder = SentenceTransformerEmbedder()
+indexer = MemoryIndexer(store, embedder, memory_root="/path/to/memory")
+indexer.index_directory("/path/to/memory")
+
+search = HybridSearchService(vec_store=store, embedder=embedder)
+results = search.search("how to deploy")
+for r in results:
+    print(f"{r.file_path} (score={r.hybrid_score:.3f}): {r.chunk_text[:80]}...")
+```
+
+## CLI Usage
+
+```bash
+# Full rebuild
+memory-vec /path/to/project --rebuild
+
+# Incremental update (only changed files)
+memory-vec /path/to/project --incremental
+
+# Search
+memory-vec /path/to/project --search "how to deploy" --top-k 5
+
+# Statistics
+memory-vec /path/to/project --stats
+
+# Custom memory subdirectory
+memory-vec /path/to/project --rebuild --memory-subdir "docs/memory"
+
+# Verbose logging
+memory-vec /path/to/project --rebuild -v
+```
+
+## API Reference
+
+### High-level
+
+| Class | Description |
+|-------|-------------|
+| `MemoryVectorService` | All-in-one service: rebuild, incremental index, search, stats |
+
+### Components
+
+| Class | Description |
+|-------|-------------|
+| `SqliteVecStore` | sqlite-vec backed vector store with KNN search |
+| `SentenceTransformerEmbedder` | Lazy-loading sentence-transformers embedder |
+| `MemoryIndexer` | Markdown → chunks → embeddings → store pipeline |
+| `HybridSearchService` | Hybrid scoring: `α×semantic + β×importance + γ×temporal` |
+
+### Interfaces
+
+| Interface | Description |
+|-----------|-------------|
+| `IEmbedder` | Abstract embedder (`embed`, `embed_batch`, `dimension`) |
+| `ISqliteVecStore` | Abstract vector store (`add`, `search`, `delete`, `clear`, `count`) |
+
+### Data Types
+
+| Type | Description |
+|------|-------------|
+| `VectorRecord` | Record for insertion (id, embedding, metadata) |
+| `VectorSearchResult` | Raw KNN result (id, distance, metadata) |
+| `SearchResult` | Hybrid search result with all score components |
+| `MemoryVecMeta` | Metadata dataclass for stored embeddings |
+
+### Utilities
+
+| Function | Description |
+|----------|-------------|
+| `chunk_text(text, chunk_size, overlap_size)` | Split text into overlapping chunks |
+| `parse_frontmatter(text)` | Extract YAML frontmatter from Markdown |
+| `content_hash(text)` | SHA-256 hex digest |
+| `is_sqlite_vec_available()` | Check sqlite-vec availability |
+| `is_sentence_transformers_available()` | Check sentence-transformers availability |
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────┐
+│                MemoryVectorService                  │
+│          (high-level orchestration layer)            │
+├──────────┬──────────┬──────────────┬────────────────┤
+│          │          │              │                 │
+│  Indexer  │  Search  │   Embedder   │     Store      │
+│          │          │              │                 │
+│ .md file │  hybrid  │  sentence-   │  sqlite-vec    │
+│ → chunks │  scoring │  transformers│  KNN + meta    │
+│ → embed  │  α+β+γ   │  (lazy load) │  (cosine)      │
+│ → store  │          │              │                 │
+└──────────┴──────────┴──────────────┴────────────────┘
+     ▲                                      │
+     │         YAML frontmatter             │
+     │         importance/type/tags          ▼
+   ┌─┴─────────────────────┐     ┌─────────────────────┐
+   │    Markdown Files     │     │   vector_index.db    │
+   │   (.claude/memory/)   │     │   (single file)      │
+   └───────────────────────┘     └─────────────────────┘
+```
+
+## Configuration
+
+### HuggingFace Model
+
+By default, uses `paraphrase-multilingual-MiniLM-L12-v2` (384-dim, 50+ languages).
+
+For users in China or regions with slow HuggingFace access:
+
+```bash
+# Use a mirror
+export HF_ENDPOINT=https://hf-mirror.com
+
+# Or use offline mode (model must be pre-cached)
+export HF_HUB_OFFLINE=1
+```
+
+### Hybrid Search Weights
+
+Default: `α=0.6, β=0.2, γ=0.2, λ=0.05`
+
+```python
+search = HybridSearchService(
+    vec_store=store,
+    embedder=embedder,
+    alpha=0.8,    # Semantic weight
+    beta=0.1,     # Importance weight
+    gamma=0.1,    # Temporal decay weight
+    decay_lambda=0.03,  # Slower decay
+)
+```
+
+## License
+
+MIT

markdown_memory_vec-0.1.0.dist-info/RECORD

@@ -0,0 +1,13 @@
+memory_vec/__init__.py,sha256=TYFh7VgOBBw2A1GjM2uTuxr_wIoVASjcjaI9eiddvLw,2356
+memory_vec/__main__.py,sha256=_HGhIzBVmsnb1XKqV_VOGMdIVlPrOWFDNhB3PLFNvG0,3978
+memory_vec/embedder.py,sha256=dPn0DGnNSQ8Uj_gbDDq2FlC8hbRTvOQUBHxWOtEeN3w,5260
+memory_vec/indexer.py,sha256=ZBE7KtY-2x77x5p89SpiZ3oKKqLs2LJGE9SkAu1vTgI,10701
+memory_vec/interfaces.py,sha256=A_vyOIlMiBO3UkW4AMCtpq58pGbSASiIHhAiSvfm16c,2810
+memory_vec/search.py,sha256=aEkfHjJeeUtaLUksfzAW2UfJsEPOoHcnBnVfzXgoQ1w,8624
+memory_vec/service.py,sha256=AylPef106ZWLB9NiEr2Uee2LB0IZjrYiEs8HYn8TtNU,11533
+memory_vec/store.py,sha256=G2GJYtcqYA0Op4oOgjIRZLDzTmpI8HANoUqEMrOwO88,17319
+markdown_memory_vec-0.1.0.dist-info/METADATA,sha256=Xob56dfeNQX5M81IALmxX8uEQioScFS3Y19m-JZCJWE,7956
+markdown_memory_vec-0.1.0.dist-info/WHEEL,sha256=QccIxa26bgl1E6uMy58deGWi-0aeIkkangHcxk2kWfw,87
+markdown_memory_vec-0.1.0.dist-info/entry_points.txt,sha256=TGn2LA56FfN5WnjyVjbe-Q7GqW7NMU2VvbEMXeEHYN8,56
+markdown_memory_vec-0.1.0.dist-info/licenses/LICENSE,sha256=GeX_PyX3s1Hao9OpOj1gon8E4w_75fcZMqvhdA5bd5c,1064
+markdown_memory_vec-0.1.0.dist-info/RECORD,,

markdown_memory_vec-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.29.0
+Root-Is-Purelib: true
+Tag: py3-none-any

markdown_memory_vec-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+memory-vec = memory_vec.__main__:main

markdown_memory_vec-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Aigente
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

memory_vec/__init__.py

@@ -0,0 +1,73 @@
+"""
+Vector search infrastructure for Markdown-based memory systems.
+
+This package provides:
+
+- **Interfaces** (``interfaces.py``): ``IEmbedder``, ``ISqliteVecStore``
+- **Storage** (``store.py``): ``SqliteVecStore`` — concrete sqlite-vec
+  backed vector store implementing ``ISqliteVecStore``
+- **Embedder** (``embedder.py``): ``SentenceTransformerEmbedder`` — concrete
+  embedder using sentence-transformers, implementing ``IEmbedder``
+- **Indexer** (``indexer.py``): ``MemoryIndexer`` — Markdown-to-vector indexing
+  pipeline with chunking, SHA-256 dedup, and YAML frontmatter parsing
+- **Search** (``search.py``): ``HybridSearchService`` — hybrid retrieval
+  combining semantic similarity, importance, and temporal decay
+
+All heavy dependencies (``sqlite-vec``, ``sentence-transformers``) are optional.
+Use the ``is_*_available()`` helpers to check at runtime.
+
+Quick start::
+
+    from memory_vec import (
+        SqliteVecStore,
+        SentenceTransformerEmbedder,
+        MemoryIndexer,
+        HybridSearchService,
+    )
+
+    store = SqliteVecStore("memory.db")
+    store.ensure_tables()
+
+    embedder = SentenceTransformerEmbedder()
+    indexer = MemoryIndexer(store, embedder)
+    indexer.index_directory("path/to/markdown/files")
+
+    search = HybridSearchService(vec_store=store, embedder=embedder)
+    results = search.search("how to deploy")
+"""
+
+# Interfaces
+# Concrete implementations
+from .embedder import SentenceTransformerEmbedder, is_sentence_transformers_available
+from .indexer import MemoryIndexer, chunk_text, parse_frontmatter
+from .interfaces import IEmbedder, ISqliteVecStore, VectorRecord, VectorSearchResult
+from .search import HybridSearchService, SearchResult
+from .service import MemoryVectorService
+from .store import MemoryVecMeta, SqliteVecStore, content_hash, is_sqlite_vec_available
+
+__all__ = [
+    # Interfaces
+    "IEmbedder",
+    "ISqliteVecStore",
+    "VectorRecord",
+    "VectorSearchResult",
+    # Store
+    "SqliteVecStore",
+    "MemoryVecMeta",
+    "content_hash",
+    "is_sqlite_vec_available",
+    # Embedder
+    "SentenceTransformerEmbedder",
+    "is_sentence_transformers_available",
+    # Indexer
+    "MemoryIndexer",
+    "chunk_text",
+    "parse_frontmatter",
+    # Search
+    "HybridSearchService",
+    "SearchResult",
+    # High-level service
+    "MemoryVectorService",
+]
+
+__version__ = "0.1.0"

memory_vec/__main__.py

@@ -0,0 +1,109 @@
+"""
+CLI entry point: ``python -m memory_vec`` or ``memory-vec``.
+
+Provides commands for building, maintaining, and querying the vector index.
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+
+from .service import MemoryVectorService
+
+
+def main() -> None:
+    """Command-line interface for memory vector maintenance."""
+    parser = argparse.ArgumentParser(
+        prog="memory-vec",
+        description="Markdown memory vector index — build, maintain, and search",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=(
+            "Examples:\n"
+            "  memory-vec /path/to/project --rebuild\n"
+            "  memory-vec /path/to/project --incremental\n"
+            "  memory-vec /path/to/project --search 'how to deploy'\n"
+            "  memory-vec /path/to/project --stats\n"
+            "\n"
+            "Environment variables:\n"
+            "  HF_ENDPOINT      HuggingFace mirror URL (e.g. https://hf-mirror.com)\n"
+            "  HF_HUB_OFFLINE   Set to '1' to disable network access\n"
+        ),
+    )
+    parser.add_argument("workspace", help="Project root directory")
+    parser.add_argument("--rebuild", action="store_true", help="Full rebuild of vector index")
+    parser.add_argument("--incremental", action="store_true", help="Incremental re-index (only changed files)")
+    parser.add_argument("--search", type=str, help="Search memories by query")
+    parser.add_argument("--stats", action="store_true", help="Show index statistics")
+    parser.add_argument("--top-k", type=int, default=5, help="Number of search results (default: 5)")
+    parser.add_argument(
+        "--model",
+        type=str,
+        default="paraphrase-multilingual-MiniLM-L12-v2",
+        help="Embedding model name",
+    )
+    parser.add_argument(
+        "--memory-subdir",
+        type=str,
+        default=".claude/memory",
+        help="Memory subdirectory relative to workspace (default: .claude/memory)",
+    )
+    parser.add_argument("--verbose", "-v", action="store_true", help="Enable verbose logging")
+
+    args = parser.parse_args()
+
+    logging.basicConfig(
+        level=logging.DEBUG if args.verbose else logging.INFO,
+        format="%(asctime)s %(levelname)s %(name)s: %(message)s",
+    )
+
+    svc = MemoryVectorService(
+        args.workspace,
+        model_name=args.model,
+        memory_subdir=args.memory_subdir,
+    )
+
+    if not svc.is_available:
+        print("ERROR: Vector dependencies not installed.", file=sys.stderr)  # noqa: T201
+        print("Install with: pip install 'markdown-memory-vec[vector]'", file=sys.stderr)  # noqa: T201
+        sys.exit(1)
+
+    try:
+        if args.rebuild:
+            total = svc.rebuild_index()
+            print(f"Full rebuild complete: {total} chunks indexed")  # noqa: T201
+
+        elif args.incremental:
+            total = svc.incremental_index()
+            print(f"Incremental index: {total} chunks updated")  # noqa: T201
+
+        elif args.search:
+            results = svc.search(args.search, top_k=args.top_k)
+            if not results:
+                print("No results found.")  # noqa: T201
+            else:
+                for i, r in enumerate(results, 1):
+                    print(f"\n--- Result {i} (score: {r['hybrid_score']}) ---")  # noqa: T201
+                    print(f"File: {r['file_path']}")  # noqa: T201
+                    print(f"Type: {r['memory_type']}  Importance: {r['importance']}")  # noqa: T201
+                    text = str(r["chunk_text"])
+                    if len(text) > 200:
+                        text = text[:200] + "..."
+                    print(f"Text: {text}")  # noqa: T201
+
+        elif args.stats:
+            s = svc.stats()
+            print("Vector Index Statistics:")  # noqa: T201
+            for k, v in s.items():
+                print(f"  {k}: {v}")  # noqa: T201
+
+        else:
+            parser.print_help()
+
+    finally:
+        svc.close()
+
+
+if __name__ == "__main__":
+    main()

memory_vec/embedder.py

@@ -0,0 +1,137 @@
+"""
+Embedding model abstraction layer.
+
+Provides a concrete implementation :class:`SentenceTransformerEmbedder` that
+wraps the ``sentence-transformers`` library and conforms to the
+:class:`IEmbedder` interface defined in ``interfaces.py``.
+
+The heavy ML import is deferred until the first call so that importing this
+module is always cheap.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+from pathlib import Path
+from typing import Any, List
+
+from .interfaces import IEmbedder
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Availability check
+# ---------------------------------------------------------------------------
+_sentence_transformers_available = False
+try:
+    import sentence_transformers  # noqa: F401  # type: ignore[import-untyped]
+
+    _sentence_transformers_available = True
+except ImportError:
+    pass
+
+
+def is_sentence_transformers_available() -> bool:
+    """Return ``True`` if the sentence-transformers package is importable."""
+    return _sentence_transformers_available
+
+
+# ---------------------------------------------------------------------------
+# Sentence-transformers implementation
+# ---------------------------------------------------------------------------
+_DEFAULT_MODEL_NAME = "paraphrase-multilingual-MiniLM-L12-v2"
+_DEFAULT_DIMENSION = 384
+
+
+def _ensure_hf_env(model_name: str) -> None:
+    """Set HuggingFace environment variables for reliable model loading.
+
+    Strategy:
+    - If model is already cached locally → set ``HF_HUB_OFFLINE=1`` (no network).
+    - Otherwise respect user-set ``HF_ENDPOINT`` (do **not** override it).
+    """
+    if os.environ.get("HF_HUB_OFFLINE") == "1":
+        return  # User explicitly requested offline mode
+
+    # Check HuggingFace hub cache for the model
+    cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
+    model_dir = cache_dir / f"models--sentence-transformers--{model_name}"
+    if model_dir.exists():
+        os.environ["HF_HUB_OFFLINE"] = "1"
+        logger.info("Model '%s' found in local cache, using offline mode", model_name)
+        return
+
+    # Model not cached — log a hint if no endpoint is set
+    if not os.environ.get("HF_ENDPOINT"):
+        logger.info(
+            "HF_ENDPOINT not set. If downloading is slow, set HF_ENDPOINT to a mirror "
+            "(e.g. https://hf-mirror.com for users in China)."
+        )
+
+
+class SentenceTransformerEmbedder(IEmbedder):
+    """Embedder backed by the ``sentence-transformers`` library.
+
+    The underlying ``SentenceTransformer`` model is loaded lazily on the first
+    call to :meth:`embed` or :meth:`embed_batch`, and is then cached as a
+    class-level variable so it is shared across all instances using the same
+    model name.
+
+    Parameters
+    ----------
+    model_name:
+        HuggingFace model identifier.  Defaults to
+        ``"paraphrase-multilingual-MiniLM-L12-v2"`` (384-dim, 50+ languages).
+    """
+
+    # Class-level cache: model_name -> SentenceTransformer instance
+    _model_cache: dict[str, Any] = {}
+
+    def __init__(self, model_name: str = _DEFAULT_MODEL_NAME) -> None:
+        if not _sentence_transformers_available:
+            raise RuntimeError(
+                "sentence-transformers is not installed. " "Install it with: pip install 'markdown-memory-vec[vector]'"
+            )
+        self._model_name = model_name
+        self._dimension_override: int | None = None
+
+    # -- lazy model loading --------------------------------------------------
+
+    @property
+    def _model(self) -> Any:
+        """Return the cached ``SentenceTransformer`` instance."""
+        if self._model_name not in self._model_cache:
+            from sentence_transformers import SentenceTransformer  # type: ignore[import-untyped]
+
+            _ensure_hf_env(self._model_name)
+            logger.info("Loading sentence-transformer model: %s", self._model_name)
+            model = SentenceTransformer(self._model_name)
+            self._model_cache[self._model_name] = model
+            # Infer dimension from the model
+            dim: int = model.get_sentence_embedding_dimension()  # type: ignore[assignment]
+            self._dimension_override = dim
+        return self._model_cache[self._model_name]
+
+    # -- IEmbedder interface implementation ----------------------------------
+
+    def embed(self, text: str) -> List[float]:
+        """Embed a single text string into a vector."""
+        result = self._model.encode([text], show_progress_bar=False)
+        return result[0].tolist()  # type: ignore[union-attr, no-any-return]
+
+    def embed_batch(self, texts: List[str]) -> List[List[float]]:
+        """Embed multiple texts into vectors."""
+        if not texts:
+            return []
+        embeddings = self._model.encode(texts, show_progress_bar=False)
+        return [vec.tolist() for vec in embeddings]  # type: ignore[union-attr]
+
+    @property
+    def dimension(self) -> int:
+        """Return the vector dimension (triggers model load if unknown)."""
+        if self._dimension_override is not None:
+            return self._dimension_override
+        # Trigger model load to discover dimension
+        _ = self._model
+        return self._dimension_override or _DEFAULT_DIMENSION