microvec 0.1.0__tar.gz

microvec-0.1.0/.claude/settings.local.json

@@ -0,0 +1,22 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(git init:*)",
+      "Bash(git remote:*)",
+      "Bash(git fetch:*)",
+      "Bash(git checkout:*)",
+      "Bash(python3 -m py_compile /Users/waltergonzalez/Documents/projects/microvec/microvector.py)",
+      "Bash(pip install:*)",
+      "Bash(pip3 install:*)",
+      "Bash(python3:*)",
+      "Bash(git add:*)",
+      "Bash(git rm:*)",
+      "Bash(git commit:*)",
+      "Bash(git push:*)",
+      "Bash(curl -s https://pypi.org/pypi/microvector/json)",
+      "Bash(curl -s -o /dev/null -w \"microvectordb: %{http_code}\\\\n\" https://pypi.org/pypi/microvectordb/json)",
+      "Bash(curl -s -o /dev/null -w \"microvec: %{http_code}\\\\n\" https://pypi.org/pypi/microvec/json)",
+      "Bash(curl -s -o /dev/null -w \"micro-vector-db: %{http_code}\\\\n\" https://pypi.org/pypi/micro-vector-db/json)"
+    ]
+  }
+}

microvec-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,61 @@
+name: CI
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dev dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Check formatting (black)
+        run: black --check microvector/ tests/
+
+      - name: Lint (ruff)
+        run: ruff check microvector/ tests/
+
+      - name: Type check (mypy)
+        run: mypy microvector/
+
+  test:
+    name: Test Python ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    needs: lint
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Run tests with coverage
+        run: pytest --cov=microvector --cov-report=xml --cov-fail-under=90
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v4
+        if: matrix.python-version == '3.12'
+        with:
+          files: ./coverage.xml
+          fail_ci_if_error: false

microvec-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+.Python
+*.egg-info/
+dist/
+build/
+.eggs/
+.coverage
+coverage.xml
+htmlcov/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.mvdb/

microvec-0.1.0/CHANGELOG.md

@@ -0,0 +1,47 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [0.1.0] — 2026-03-24
+
+Complete rewrite from prototype to production-grade library.
+
+### Fixed
+
+- **Critical: `IndentationError` in `add_node`** — the original module could not be imported at all due to a syntax error in the method definition.
+- **Euclidean distance returned negative values** — the original returned `-np.linalg.norm(...)`, making far vectors rank as *more* similar than near ones. Fixed to `1 / (1 + distance)`, mapping to `(0, 1]` where higher always means more similar.
+- **Cosine similarity crashed on zero-norm vectors** — division by zero when either vector was all zeros. Now returns `0.0` (no direction, no similarity).
+- **Vector dimension never validated** — `self._dimension` was stored in `__init__` but never checked. Any vector of any size was silently accepted. Now raises `DimensionMismatchError` with clear expected/got values.
+- **Search results dropped the document text** — `search_top_k` returned `(index, score)` tuples, forcing callers to do a separate lookup. Now returns `SearchResult` objects with `index`, `document`, `score`, and `metadata`.
+- **Index counter could diverge after removal** — internal index tracking was done via a list with no gap management. Replaced with `dict[int, Node]` keyed by index; `_next_index` increments monotonically and retired indexes are never reused.
+- **Quantization was untested and had edge cases** — constant vectors (min == max) would cause `linspace` issues. Added input validation, edge-case handling, and a full test suite to verify correctness.
+
+### Added
+
+- **Package structure** — reorganized from a single `microvector.py` file into a proper `microvector/` package with `core.py`, `similarity.py`, `quantization.py`, `models.py`, `exceptions.py`, and `__init__.py`.
+- **`SearchResult` dataclass** — immutable result type with `index`, `document`, `score`, `metadata` fields. Supports sorting.
+- **`Node` dataclass** — replaces internal dict representation.
+- **Custom exceptions** — `MicroVectorError` (base), `DimensionMismatchError`, `EmptyDatabaseError`, `NodeNotFoundError`. All carry structured attributes for programmatic inspection.
+- **`add_nodes()`** — batch insert with fail-fast validation (all inputs validated before any insertion).
+- **`get_node()`** — retrieve a node by index.
+- **`update_node()`** — update vector, document, and/or metadata in place.
+- **`search_by_threshold()`** — return all nodes with score >= a minimum threshold.
+- **`filter_fn` parameter on `search_top_k`** — pre-filter nodes before scoring with any Python predicate.
+- **`dot` distance metric** — raw dot product, best for pre-normalized embeddings.
+- **`__len__`, `__contains__`, `__repr__`** — standard Python dunder methods.
+- **`stats()`** — returns count, dimension, next_index, index_gaps, indices.
+- **`metadata` support** — arbitrary key-value dicts stored per node, returned in search results.
+- **Safe persistence** — replaced `pickle` with JSON manifest + numpy `.npy` file in a `.mvdb/` directory. Safe against arbitrary code execution, portable, human-inspectable.
+- **`pyproject.toml`** — proper packaging with hatchling, PyPI classifiers, optional dev dependencies.
+- **Full test suite** — 107 tests, 98.5% coverage across all modules.
+- **CI/CD** — GitHub Actions workflow: lint (black + ruff + mypy) then test across Python 3.9, 3.10, 3.11, 3.12.
+
+### Changed
+
+- `save_to_disk` / `read_from_disk` → `save` / `MicroVectorDB.load` (classmethod). New format is a `.mvdb/` directory instead of a single pickle file.
+- `search_top_k` now accepts `metric=` keyword (was `distance_metric=`) and returns `list[SearchResult]` instead of `list[tuple]`.

microvec-0.1.0/PKG-INFO

@@ -0,0 +1,290 @@
+Metadata-Version: 2.4
+Name: microvec
+Version: 0.1.0
+Summary: A lightweight, production-grade in-memory vector database
+Project-URL: Homepage, https://github.com/huolter/microVector
+Project-URL: Repository, https://github.com/huolter/microVector
+Project-URL: Bug Tracker, https://github.com/huolter/microVector/issues
+Project-URL: Changelog, https://github.com/huolter/microVector/blob/main/CHANGELOG.md
+Author: huolter
+License: MIT
+Keywords: database,embeddings,rag,search,similarity,vector
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.21.0
+Provides-Extra: dev
+Requires-Dist: black>=23.0; extra == 'dev'
+Requires-Dist: mypy>=1.5; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1; extra == 'dev'
+Requires-Dist: pytest>=7.4; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# MicroVector
+
+A lightweight, production-grade in-memory vector database for Python.
+
+```
+pip install microvector
+```
+
+No external services. No complex setup. Just numpy and your embeddings.
+
+---
+
+## Features
+
+- **Fast similarity search** — cosine, euclidean, and dot product metrics
+- **Rich results** — every search result includes the document text and metadata, not just an index
+- **Batch operations** — insert thousands of vectors in one call
+- **Metadata filtering** — filter candidates before scoring with any Python predicate
+- **Safe persistence** — JSON + numpy format (no pickle, no security risk)
+- **Full type hints** — works great with mypy and IDEs
+- **Zero required dependencies** beyond numpy
+
+---
+
+## Quick Start
+
+```python
+import numpy as np
+from microvector import MicroVectorDB
+
+# Create a database for 768-dimensional embeddings (e.g. OpenAI ada-002)
+db = MicroVectorDB(dimension=768)
+
+# Add documents with their embeddings
+db.add_node(embed("Paris is the capital of France"), "Paris is the capital of France")
+db.add_node(embed("The Eiffel Tower is in Paris"),   "The Eiffel Tower is in Paris")
+db.add_node(embed("Python is a programming language"), "Python is a programming language")
+
+# Search — results include document text, not just indexes
+results = db.search_top_k(embed("What city is the Eiffel Tower in?"), k=2)
+for r in results:
+    print(f"{r.score:.3f}  {r.document}")
+# 0.921  The Eiffel Tower is in Paris
+# 0.887  Paris is the capital of France
+
+# Save and reload
+db.save("my_knowledge_base")
+db = MicroVectorDB.load("my_knowledge_base")
+```
+
+---
+
+## Installation
+
+**Requires Python 3.9+ and numpy >= 1.21.**
+
+```bash
+pip install microvector
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/huolter/microVector
+cd microVector
+pip install -e ".[dev]"
+```
+
+---
+
+## API Reference
+
+### `MicroVectorDB(dimension)`
+
+Create a new database. All vectors must have this exact dimension.
+
+```python
+db = MicroVectorDB(dimension=512)
+```
+
+---
+
+### `add_node(vector, document, metadata=None, num_bits=None) → int`
+
+Add a single vector. Returns the assigned index.
+
+```python
+idx = db.add_node(
+    vector=np.array([...]),
+    document="The text this vector represents",
+    metadata={"source": "wikipedia", "date": "2024-01"},
+    num_bits=8,   # optional: apply 8-bit scalar quantization
+)
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `vector` | `np.ndarray` | 1-D array matching the database dimension |
+| `document` | `str` | Text or identifier associated with this vector |
+| `metadata` | `dict` | Optional key-value pairs stored alongside the vector |
+| `num_bits` | `int` | Optional quantization (1–16 bits). Reduces memory at the cost of precision. |
+
+---
+
+### `add_nodes(vectors, documents, metadata=None) → list[int]`
+
+Batch insert. All inputs are validated before any insertion occurs.
+
+```python
+import numpy as np
+
+vectors = np.random.rand(1000, 512)
+docs = [f"Document {i}" for i in range(1000)]
+indices = db.add_nodes(vectors, docs)
+```
+
+---
+
+### `search_top_k(query_vector, k, metric='cosine', filter_fn=None) → list[SearchResult]`
+
+Find the top-k most similar vectors. Returns results sorted by score descending.
+
+```python
+results = db.search_top_k(query, k=5)
+results = db.search_top_k(query, k=5, metric="euclidean")
+
+# Filter before scoring — only consider documents tagged "news"
+results = db.search_top_k(
+    query, k=5,
+    filter_fn=lambda node: node.metadata.get("type") == "news"
+)
+```
+
+**Metrics:**
+
+| Metric | Range | Notes |
+|--------|-------|-------|
+| `cosine` | [-1, 1] | Best for text embeddings. Direction-based, scale-invariant. |
+| `euclidean` | (0, 1] | `1 / (1 + dist)`. Identical vectors = 1.0. |
+| `dot` | (-∞, +∞) | Fastest. Best for pre-normalized unit vectors. |
+
+**`SearchResult` fields:**
+
+```python
+result.index     # int   — node's assigned index
+result.document  # str   — the document text
+result.score     # float — similarity score (higher = more similar)
+result.metadata  # dict  — metadata stored with this node
+```
+
+---
+
+### `search_by_threshold(query_vector, min_score, metric='cosine') → list[SearchResult]`
+
+Return all nodes with score >= `min_score`, sorted descending.
+
+```python
+results = db.search_by_threshold(query, min_score=0.8)
+```
+
+---
+
+### `get_node(index) → Node`
+
+Retrieve a node by index.
+
+```python
+node = db.get_node(42)
+print(node.document, node.metadata)
+```
+
+---
+
+### `update_node(index, vector=None, document=None, metadata=None)`
+
+Update fields of an existing node. Omit fields to leave them unchanged.
+
+```python
+db.update_node(42, document="Updated text")
+db.update_node(42, metadata={"status": "reviewed"})
+```
+
+---
+
+### `remove_node(index)`
+
+Remove a node. Its index is permanently retired (never reused).
+
+```python
+db.remove_node(42)
+```
+
+---
+
+### `save(path)` / `MicroVectorDB.load(path)`
+
+Persist to and restore from a `.mvdb/` directory. Safe format: JSON + numpy binary.
+
+```python
+db.save("my_db")              # creates my_db.mvdb/
+db = MicroVectorDB.load("my_db")
+```
+
+The `.mvdb/` directory contains:
+- `index.json` — node metadata and documents (human-readable)
+- `vectors.npy` — all vectors as a 2-D numpy array
+
+---
+
+### Utility methods
+
+```python
+len(db)           # number of nodes
+42 in db          # check if index exists
+repr(db)          # MicroVectorDB(dimension=512, nodes=1000, next_index=1000)
+db.stats()        # {'count': 1000, 'dimension': 512, 'next_index': 1000, ...}
+```
+
+---
+
+## Exceptions
+
+All exceptions inherit from `MicroVectorError`.
+
+```python
+from microvector import (
+    MicroVectorError,
+    DimensionMismatchError,  # wrong vector dimension
+    EmptyDatabaseError,      # search on empty database
+    NodeNotFoundError,       # get/update/remove non-existent index
+)
+```
+
+---
+
+## Design Notes
+
+**Why in-memory?** MicroVector is designed for small-to-medium datasets (up to ~100k vectors) where the simplicity of pure-Python outweighs the need for a dedicated service. It starts instantly, needs no configuration, and is trivially embeddable in any Python application.
+
+**Why not pickle?** Pickle can execute arbitrary code when loading untrusted files. MicroVector uses JSON + numpy binary format which is safe, portable, and inspectable with any text editor.
+
+**Index monotonicity.** Deleted indexes are never reused. This prevents stale external references from silently pointing to new data.
+
+---
+
+## Roadmap
+
+- [ ] Approximate nearest neighbor (HNSW)
+- [ ] Voronoi cell indexing
+- [ ] FAISS optional backend for large datasets
+- [ ] Async search support
+- [ ] Benchmarks
+
+---
+
+## License
+
+MIT

microvec-0.1.0/README.md

@@ -0,0 +1,258 @@
+# MicroVector
+
+A lightweight, production-grade in-memory vector database for Python.
+
+```
+pip install microvector
+```
+
+No external services. No complex setup. Just numpy and your embeddings.
+
+---
+
+## Features
+
+- **Fast similarity search** — cosine, euclidean, and dot product metrics
+- **Rich results** — every search result includes the document text and metadata, not just an index
+- **Batch operations** — insert thousands of vectors in one call
+- **Metadata filtering** — filter candidates before scoring with any Python predicate
+- **Safe persistence** — JSON + numpy format (no pickle, no security risk)
+- **Full type hints** — works great with mypy and IDEs
+- **Zero required dependencies** beyond numpy
+
+---
+
+## Quick Start
+
+```python
+import numpy as np
+from microvector import MicroVectorDB
+
+# Create a database for 768-dimensional embeddings (e.g. OpenAI ada-002)
+db = MicroVectorDB(dimension=768)
+
+# Add documents with their embeddings
+db.add_node(embed("Paris is the capital of France"), "Paris is the capital of France")
+db.add_node(embed("The Eiffel Tower is in Paris"),   "The Eiffel Tower is in Paris")
+db.add_node(embed("Python is a programming language"), "Python is a programming language")
+
+# Search — results include document text, not just indexes
+results = db.search_top_k(embed("What city is the Eiffel Tower in?"), k=2)
+for r in results:
+    print(f"{r.score:.3f}  {r.document}")
+# 0.921  The Eiffel Tower is in Paris
+# 0.887  Paris is the capital of France
+
+# Save and reload
+db.save("my_knowledge_base")
+db = MicroVectorDB.load("my_knowledge_base")
+```
+
+---
+
+## Installation
+
+**Requires Python 3.9+ and numpy >= 1.21.**
+
+```bash
+pip install microvector
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/huolter/microVector
+cd microVector
+pip install -e ".[dev]"
+```
+
+---
+
+## API Reference
+
+### `MicroVectorDB(dimension)`
+
+Create a new database. All vectors must have this exact dimension.
+
+```python
+db = MicroVectorDB(dimension=512)
+```
+
+---
+
+### `add_node(vector, document, metadata=None, num_bits=None) → int`
+
+Add a single vector. Returns the assigned index.
+
+```python
+idx = db.add_node(
+    vector=np.array([...]),
+    document="The text this vector represents",
+    metadata={"source": "wikipedia", "date": "2024-01"},
+    num_bits=8,   # optional: apply 8-bit scalar quantization
+)
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `vector` | `np.ndarray` | 1-D array matching the database dimension |
+| `document` | `str` | Text or identifier associated with this vector |
+| `metadata` | `dict` | Optional key-value pairs stored alongside the vector |
+| `num_bits` | `int` | Optional quantization (1–16 bits). Reduces memory at the cost of precision. |
+
+---
+
+### `add_nodes(vectors, documents, metadata=None) → list[int]`
+
+Batch insert. All inputs are validated before any insertion occurs.
+
+```python
+import numpy as np
+
+vectors = np.random.rand(1000, 512)
+docs = [f"Document {i}" for i in range(1000)]
+indices = db.add_nodes(vectors, docs)
+```
+
+---
+
+### `search_top_k(query_vector, k, metric='cosine', filter_fn=None) → list[SearchResult]`
+
+Find the top-k most similar vectors. Returns results sorted by score descending.
+
+```python
+results = db.search_top_k(query, k=5)
+results = db.search_top_k(query, k=5, metric="euclidean")
+
+# Filter before scoring — only consider documents tagged "news"
+results = db.search_top_k(
+    query, k=5,
+    filter_fn=lambda node: node.metadata.get("type") == "news"
+)
+```
+
+**Metrics:**
+
+| Metric | Range | Notes |
+|--------|-------|-------|
+| `cosine` | [-1, 1] | Best for text embeddings. Direction-based, scale-invariant. |
+| `euclidean` | (0, 1] | `1 / (1 + dist)`. Identical vectors = 1.0. |
+| `dot` | (-∞, +∞) | Fastest. Best for pre-normalized unit vectors. |
+
+**`SearchResult` fields:**
+
+```python
+result.index     # int   — node's assigned index
+result.document  # str   — the document text
+result.score     # float — similarity score (higher = more similar)
+result.metadata  # dict  — metadata stored with this node
+```
+
+---
+
+### `search_by_threshold(query_vector, min_score, metric='cosine') → list[SearchResult]`
+
+Return all nodes with score >= `min_score`, sorted descending.
+
+```python
+results = db.search_by_threshold(query, min_score=0.8)
+```
+
+---
+
+### `get_node(index) → Node`
+
+Retrieve a node by index.
+
+```python
+node = db.get_node(42)
+print(node.document, node.metadata)
+```
+
+---
+
+### `update_node(index, vector=None, document=None, metadata=None)`
+
+Update fields of an existing node. Omit fields to leave them unchanged.
+
+```python
+db.update_node(42, document="Updated text")
+db.update_node(42, metadata={"status": "reviewed"})
+```
+
+---
+
+### `remove_node(index)`
+
+Remove a node. Its index is permanently retired (never reused).
+
+```python
+db.remove_node(42)
+```
+
+---
+
+### `save(path)` / `MicroVectorDB.load(path)`
+
+Persist to and restore from a `.mvdb/` directory. Safe format: JSON + numpy binary.
+
+```python
+db.save("my_db")              # creates my_db.mvdb/
+db = MicroVectorDB.load("my_db")
+```
+
+The `.mvdb/` directory contains:
+- `index.json` — node metadata and documents (human-readable)
+- `vectors.npy` — all vectors as a 2-D numpy array
+
+---
+
+### Utility methods
+
+```python
+len(db)           # number of nodes
+42 in db          # check if index exists
+repr(db)          # MicroVectorDB(dimension=512, nodes=1000, next_index=1000)
+db.stats()        # {'count': 1000, 'dimension': 512, 'next_index': 1000, ...}
+```
+
+---
+
+## Exceptions
+
+All exceptions inherit from `MicroVectorError`.
+
+```python
+from microvector import (
+    MicroVectorError,
+    DimensionMismatchError,  # wrong vector dimension
+    EmptyDatabaseError,      # search on empty database
+    NodeNotFoundError,       # get/update/remove non-existent index
+)
+```
+
+---
+
+## Design Notes
+
+**Why in-memory?** MicroVector is designed for small-to-medium datasets (up to ~100k vectors) where the simplicity of pure-Python outweighs the need for a dedicated service. It starts instantly, needs no configuration, and is trivially embeddable in any Python application.
+
+**Why not pickle?** Pickle can execute arbitrary code when loading untrusted files. MicroVector uses JSON + numpy binary format which is safe, portable, and inspectable with any text editor.
+
+**Index monotonicity.** Deleted indexes are never reused. This prevents stale external references from silently pointing to new data.
+
+---
+
+## Roadmap
+
+- [ ] Approximate nearest neighbor (HNSW)
+- [ ] Voronoi cell indexing
+- [ ] FAISS optional backend for large datasets
+- [ ] Async search support
+- [ ] Benchmarks
+
+---
+
+## License
+
+MIT