zvec-db 0.3.0__tar.gz

zvec_db-0.3.0/PKG-INFO

@@ -0,0 +1,720 @@
+Metadata-Version: 2.4
+Name: zvec-db
+Version: 0.3.0
+Summary: Suite d'utilitaires pour la vectorisation sparse et le re-ranking de documents
+Author: Charles Condevaux
+License: MIT
+Project-URL: Homepage, https://github.com/ccdv-ai/zvec-db
+Project-URL: Repository, https://github.com/ccdv-ai/zvec-db.git
+Project-URL: Issues, https://github.com/ccdv-ai/zvec-db/issues
+Keywords: search,ranking,BM25,TF-IDF,sparse,vectors,embeddings
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: zvec
+Requires-Dist: scikit-learn
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: httpx
+Requires-Dist: requests
+Requires-Dist: sentence_transformers
+Requires-Dist: openai
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: cloudpickle
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
+Requires-Dist: nltk>=3.8.0; extra == "test"
+Provides-Extra: preprocessing
+Requires-Dist: nltk>=3.8.0; extra == "preprocessing"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.0.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == "docs"
+Requires-Dist: sphinx-math-dollar>=1.2.0; extra == "docs"
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: black>=25.9.0; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Provides-Extra: build
+Requires-Dist: build; extra == "build"
+Requires-Dist: twine; extra == "build"
+Requires-Dist: wheel; extra == "build"
+
+# zvec-db
+
+[![Version](https://img.shields.io/pypi/v/zvec-db)](https://pypi.org/project/zvec-db/)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/ccdv-ai/zvec-db/blob/main/LICENSE)
+
+Utility suite for sparse/dense vectorization and document re-ranking, designed to work with [zvec](https://github.com/ccdv-ai/zvec).
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Sparse Embedders](#sparse-embedders)
+- [Dense Embedders](#dense-embedders)
+- [Re-ranking](#re-ranking)
+- [Preprocessing](#preprocessing)
+- [Model Persistence](#model-persistence)
+- [Evaluation](#evaluation)
+- [Complete Example: Hybrid Search Pipeline](#complete-example-hybrid-search-pipeline)
+- [License](#license)
+
+---
+
+## Installation
+
+```bash
+pip install zvec-db
+```
+
+**Optional dependencies:**
+
+```bash
+# For preprocessing (stemming, stopwords)
+pip install "zvec-db[preprocessing]"
+
+# For development
+pip install "zvec-db[dev,test,docs]"
+```
+
+---
+
+## Quick Start
+
+### Hybrid search with zvec (recommended)
+
+```python
+import zvec
+from zvec_db.embedders import BM25Embedder, OpenAIEmbedder
+from zvec_db.rerankers import NormalizedWeightedReRanker
+
+# 1. Create embedders
+bm25 = BM25Embedder(max_features=4096)
+bm25.fit(documents)
+
+dense = OpenAIEmbedder(base_url="http://localhost:9300/v1", model="embedding")
+
+# 2. Create collection
+schema = zvec.CollectionSchema(
+    name="docs",
+    vectors=[
+        zvec.VectorSchema("sparse", zvec.DataType.SPARSE_FP32, dimension=4096),
+        zvec.VectorSchema("dense", zvec.DataType.VECTOR_FP32, dimension=1024),
+    ]
+)
+collection = zvec.create_and_open("./my_db", schema)
+
+# 3. Insert
+for i, doc in enumerate(documents):
+    collection.insert(zvec.Doc(
+        id=str(i),
+        fields={"text": doc},
+        vectors={
+            "sparse": bm25.embed(doc),
+            "dense": dense.embed(doc),
+        }
+    ))
+
+# 4. Search with weighted fusion
+# Note: metrics=None because we mix BM25 (arbitrary scores) and dense (COSINE distances)
+results = collection.query(
+    vectors=[
+        zvec.VectorQuery(field_name="sparse", vector=bm25.embed(query)),
+        zvec.VectorQuery(field_name="dense", vector=dense.embed(query)),
+    ],
+    topk=10,
+    reranker=NormalizedWeightedReRanker(
+        metrics=None,  # No automatic conversion (mixed metrics)
+        weights={"sparse": 0.4, "dense": 0.6},
+        normalizer_configs={"sparse": {"method": "bayes"}},
+    ),
+)
+```
+
+---
+
+## Sparse Embedders
+
+All sparse embedders return dictionaries `{index: score, ...}` compatible with zvec's `SPARSE_FP32` format.
+
+### BM25Embedder (recommended)
+
+Standard BM25 scoring - best for general use cases.
+
+```python
+from zvec_db.embedders import BM25Embedder
+from zvec_db.preprocessing import NormalizationConfig
+
+# With automatic preprocessing
+config = NormalizationConfig.aggressive(language="french")
+bm25 = BM25Embedder(
+    max_features=4096,
+    k1=1.2,       # Term frequency saturation (default: 1.2)
+    b=0.75,       # Length normalization (default: 0.75)
+    preprocessing_config=config
+)
+bm25.fit(documents)
+
+vector = bm25.embed("search query")  # {index: score, ...}
+```
+
+### Other sparse embedders
+
+| Embedder | Use case |
+|----------|----------|
+| `TfidfEmbedder` | TF-IDF weighting with sublinear TF option |
+| `CountEmbedder` | Simple term counts (binary option available) |
+| `BM25LEmbedder` | Documents with variable lengths |
+| `BM25PlusEmbedder` | Avoid zero scores with delta smoothing |
+| `DisMaxEmbedder` | Multi-field search (takes maximum score) |
+
+```python
+from zvec_db.embedders import TfidfEmbedder, CountEmbedder, DisMaxEmbedder
+
+tfidf = TfidfEmbedder(max_features=4096, sublinear_tf=True)
+count = CountEmbedder(max_features=4096, binary=True)
+dismax = DisMaxEmbedder(tie_breaker=0.1)
+```
+
+---
+
+## Dense Embedders
+
+### OpenAIEmbedder (API / vLLM)
+
+Works with OpenAI API or compatible endpoints (vLLM, local servers).
+
+```python
+from zvec_db.embedders import OpenAIEmbedder
+
+# OpenAI API
+embedder = OpenAIEmbedder(model="text-embedding-3-small", api_key="sk-...")
+
+# Local vLLM
+embedder = OpenAIEmbedder(
+    base_url="http://localhost:9300/v1",
+    model="embedding",
+    max_batch_size=32,
+)
+vector = embedder.embed("search query")
+```
+
+### SentenceTransformersEmbedder (local)
+
+Run embedding models locally using sentence-transformers.
+
+```python
+from zvec_db.embedders import SentenceTransformersEmbedder
+
+embedder = SentenceTransformersEmbedder(
+    model_name="all-MiniLM-L6-v2",  # 384 dims, fast
+    device="cpu",
+    normalize=True,
+)
+vector = embedder.embed("search query")
+```
+
+---
+
+## Re-ranking
+
+### Understanding distance/similarity metrics
+
+**Problem**: Vector databases store **distances** (smaller = more similar), but fusion algorithms assume **similarities** (larger = more relevant).
+
+The `metrics` parameter handles conversion:
+
+| Metric | Type | Range | Conversion | Usage |
+|--------|------|-------|------------|-------|
+| `COSINE` | Distance | [0, 2] | `1.0 - score/2.0` | Normalized embeddings (Qdrant, zvec) |
+| `L2` | Distance | [0, ∞) | `1 - 2*atan(s)/π` | Euclidean distance |
+| `IP` | Similarity | (-∞, ∞) | None | Inner product (already similarity) |
+| `None` | - | - | None | BM25 scores or already normalized [0, 1] |
+
+**Default**: `metrics=MetricType.COSINE` (main use case with zvec/Qdrant).
+
+```python
+from zvec_db.rerankers import NormalizedWeightedReRanker, MetricType
+
+# COSINE distances from zvec/Qdrant (default)
+reranker = NormalizedWeightedReRanker(topn=10)
+
+# BM25 scores (not distances!)
+reranker = NormalizedWeightedReRanker(topn=10, metrics=None)
+
+# Hybrid: BM25 + dense with per-source normalization
+reranker = NormalizedWeightedReRanker(
+    metrics=None,  # No global conversion
+    weights={"sparse": 0.4, "dense": 0.6},
+    normalizer_configs={
+        "sparse": "bayes",  # BM25: handles outliers well
+        "dense": True,      # Dense: standard normalization
+    },
+)
+```
+
+### Fusion rerankers
+
+#### Normalizer configuration
+
+The `normalizer_configs` parameter controls how scores are normalized per source:
+
+| Value | Effect |
+|-------|--------|
+| `True` | Standard normalization (scales scores to [0, 1]) |
+| `"bayes"`, `"bayesian"`, `"bb25"` | Bayesian sigmoid calibration (robust to outliers). These are aliases for the same method. |
+| `{"method": "bayes", "alpha": 1.0}` | Dict with custom parameters (`alpha`, `beta`) |
+| `None` | Skip normalization (use raw scores after metric conversion) |
+
+**Example:**
+
+```python
+normalizer_configs={
+    "sparse": "bayes",  # Bayesian: handles BM25 outliers well
+    "dense": None,      # Optional: Cosine already scales between in [0, 1]
+}
+```
+
+#### NormalizedWeightedReRanker (weighted fusion)
+
+```python
+from zvec_db.rerankers import NormalizedWeightedReRanker
+
+reranker = NormalizedWeightedReRanker(
+    topn=10,
+    weights={"source1": 0.7, "source2": 0.3},
+    normalizer_configs={"source1": "bayes", "source2": True},
+)
+
+results = collection.query(vectors=[...], topk=20, reranker=reranker)
+```
+
+#### Using `schema` parameter (auto-detect metrics from collection)
+
+When working with zvec collections, you can use the `schema` parameter to automatically infer the correct metrics for each vector field:
+
+```python
+import zvec
+from zvec_db.rerankers import NormalizedWeightedReRanker
+
+# Open existing collection
+collection = zvec.open("./my_collection")
+
+# Reranker auto-infers metrics from schema
+# - SPARSE_FP32 fields -> metrics=None (BM25 scores)
+# - VECTOR_FP32 fields with COSINE -> metrics=MetricType.COSINE
+reranker = NormalizedWeightedReRanker(
+    topn=10,
+    metrics=None,  # Will infer from schema
+    schema=collection.schema,
+    weights={"sparse": 0.4, "dense": 0.6},
+)
+
+# No need to manually specify metrics per source!
+results = collection.query(
+    vectors=[
+        zvec.VectorQuery(field_name="sparse", vector=bm25.embed(query)),
+        zvec.VectorQuery(field_name="dense", vector=dense.embed(query)),
+    ],
+    topk=20,
+    reranker=reranker,
+)
+```
+
+**Manual per-source metrics (alternative):**
+
+```python
+from zvec_db.rerankers import NormalizedWeightedReRanker, MetricType
+
+# Explicit per-source metrics
+reranker = NormalizedWeightedReRanker(
+    topn=10,
+    metrics={
+        "sparse": None,              # BM25 scores (not distances)
+        "dense": MetricType.COSINE,  # Convert COSINE distance [0,2] -> similarity
+    },
+    weights={"sparse": 0.4, "dense": 0.6},
+)
+```
+
+#### NormalizedRrfReRanker (Reciprocal Rank Fusion)
+
+```python
+from zvec_db.rerankers import NormalizedRrfReRanker
+
+reranker = NormalizedRrfReRanker(topn=10, rank_constant=60)
+results = reranker.rerank({"bm25": bm25_docs, "dense": dense_docs})
+```
+
+#### WeightedReRanker (scores already normalized)
+
+Use when scores are already in [0, 1] with "higher=better" orientation.
+
+```python
+from zvec_db.rerankers import WeightedReRanker
+
+reranker = WeightedReRanker(
+    topn=10,
+    weights={"source1": 0.7, "source2": 0.3},
+)
+```
+
+### Default rerankers (ready-to-use)
+
+```python
+from zvec_db.rerankers.defaults import (
+    DefaultWeightedReranker,
+    DefaultHybridReranker,
+    DefaultRrfReranker,
+)
+
+# Weighted fusion with Bayesian normalization
+reranker = DefaultWeightedReranker()
+
+# Optimized hybrid: dense (60%) + BM25 (40%)
+reranker = DefaultHybridReranker()
+
+# RRF with standard parameters
+reranker = DefaultRrfReranker()
+
+results = reranker.rerank({"bm25": bm25_docs, "dense": dense_docs})
+```
+
+### Cross-Encoder rerankers
+
+All cross-encoders require a `query` parameter at initialization.
+
+#### SentenceTransformerReranker (local, binary)
+
+```python
+from zvec_db.rerankers import SentenceTransformerReranker
+
+reranker = SentenceTransformerReranker(
+    query="machine learning",
+    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
+    topn=10,
+)
+results = reranker.rerank({"bm25": docs})
+```
+
+#### ClassificationReranker (local, multi-class)
+
+```python
+from zvec_db.rerankers import ClassificationReranker
+
+reranker = ClassificationReranker(
+    query="machine learning",
+    model_name="your-multi-class-model",
+    num_classes=5,  # Auto-inferred if not specified
+    topn=10,
+)
+results = reranker.rerank({"bm25": docs})
+```
+
+#### OpenAIReranker (API)
+
+```python
+from zvec_db.rerankers import OpenAIReranker
+
+reranker = OpenAIReranker(
+    query="machine learning",
+    base_url="http://localhost:9400/v1",
+    model="BAAI/bge-reranker-v2-m3",
+    endpoint="rerank",  # or "score"
+    topn=10,
+)
+results = reranker.rerank({"bm25": docs})
+```
+
+### Diversification
+
+#### SubmodularReranker (MMR)
+
+Maximize relevance while diversifying results.
+
+```python
+from zvec_db.rerankers import SubmodularReranker
+
+reranker = SubmodularReranker(
+    topn=10,
+    lambda_param=0.7,  # 70% relevance, 30% diversity
+    vector_field="embedding",
+)
+results = reranker.rerank({"source": docs_with_vectors})
+```
+
+---
+
+## Preprocessing
+
+Preprocessing improves sparse embedding quality.
+
+### Automatic (recommended)
+
+```python
+from zvec_db.embedders import BM25Embedder
+from zvec_db.preprocessing import NormalizationConfig
+
+config = NormalizationConfig.aggressive(language="french")
+bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
+bm25.fit(documents)
+# Preprocessing is automatically applied and saved with the model
+```
+
+### Utility functions
+
+```python
+from zvec_db.preprocessing import normalize_text, stem_word, remove_stopwords
+
+# Full pipeline
+normalize_text("  CHAT MANGEAIT  ", lowercase=True, remove_accents=True, stem=True)  # "chat mang"
+
+# Individual functions
+stem_word("mangeaient", language="french")  # "mang"
+remove_stopwords("le chat mange", language="french")  # "chat mange"
+```
+
+**nltk installation:**
+
+```bash
+pip install "zvec-db[preprocessing]"
+```
+
+---
+
+## Model Persistence
+
+```python
+from zvec_db.embedders import BM25Embedder
+
+# Save
+bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
+bm25.fit(documents)
+bm25.save("models/bm25_model.joblib")
+
+# Load
+bm25_loaded = BM25Embedder()
+bm25_loaded.load("models/bm25_model.joblib")
+
+# Embeddings are identical (preprocessing included)
+assert bm25.embed("query") == bm25_loaded.embed("query")
+```
+
+---
+
+## Evaluation
+
+```python
+from zvec_db.evaluation import evaluate_ranking
+
+# Evaluate ranking quality
+metrics = evaluate_ranking(
+    ground_truth=[["doc1", "doc2"], ["doc3"]],
+    predictions=[["doc2", "doc1"], ["doc3", "doc4"]],
+    metrics=["ndcg", "map", "mrr", "recall"],
+)
+```
+
+---
+
+## Development
+
+```bash
+# Clone
+git clone https://github.com/ccdv-ai/zvec-db.git
+cd zvec-db
+
+# Install with all dependencies
+make install
+
+# Run tests
+make test
+
+# Lint
+make lint
+
+# Build docs
+make docs
+```
+
+---
+
+## License
+
+MIT License
+
+---
+
+## Complete Example: Hybrid Search Pipeline
+
+This section demonstrates a complete hybrid search pipeline with BM25 + dense embeddings and re-ranking.
+
+### Setup
+
+```python
+import zvec
+from zvec.model.doc import Doc
+from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder
+from zvec_db.rerankers import NormalizedWeightedReRanker, DefaultHybridReranker
+
+# Sample documents
+documents = [
+    "Machine learning is a subset of artificial intelligence",
+    "Deep learning uses neural networks with many layers",
+    "Natural language processing enables computers to understand text",
+    "Computer vision allows machines to interpret images",
+    "Reinforcement learning trains agents through rewards",
+]
+
+# Initialize embedders
+bm25 = BM25Embedder(max_features=4096, k1=1.2, b=0.75)
+bm25.fit(documents)
+
+dense = SentenceTransformersEmbedder(
+    model_name="all-MiniLM-L6-v2",
+    device="cpu",
+    normalize=True,
+)
+```
+
+### Create and populate collection
+
+```python
+# Create zvec collection
+schema = zvec.CollectionSchema(
+    name="docs",
+    vectors=[
+        zvec.VectorSchema("sparse", zvec.DataType.SPARSE_FP32, dimension=4096),
+        zvec.VectorSchema("dense", zvec.DataType.VECTOR_FP32, dimension=384),
+    ]
+)
+collection = zvec.create_and_open("./my_db", schema)
+
+# Index documents
+for i, doc in enumerate(documents):
+    collection.insert(zvec.Doc(
+        id=str(i),
+        fields={"text": doc},
+        vectors={
+            "sparse": bm25.embed(doc),
+            "dense": dense.embed(doc),
+        }
+    ))
+```
+
+### Hybrid search with re-ranking
+
+```python
+query = "neural networks and deep learning"
+
+# Method 1: Using collection.query with built-in reranker
+results = collection.query(
+    vectors=[
+        zvec.VectorQuery(field_name="sparse", vector=bm25.embed(query)),
+        zvec.VectorQuery(field_name="dense", vector=dense.embed(query)),
+    ],
+    topk=20,
+    reranker=DefaultHybridReranker(
+        weights={"sparse": 0.4, "dense": 0.6},
+    ),
+)
+
+print("Top results:")
+for i, doc in enumerate(results[:5]):
+    print(f"  {i+1}. {doc.fields['text']} (score: {doc.score:.4f})")
+```
+
+### Manual hybrid search (more control)
+
+```python
+from zvec.model.doc import Doc
+
+# 1. Separate searches
+sparse_results = collection.search(
+    vector_name="sparse",
+    vector=bm25.embed(query),
+    topk=20,
+)
+
+dense_results = collection.search(
+    vector_name="dense",
+    vector=dense.embed(query),
+    topk=20,
+)
+
+# 2. Re-rank with schema-based auto-detection
+reranker = NormalizedWeightedReRanker(
+    topn=10,
+    metrics=None,  # Infer from schema
+    schema=collection.schema,
+    weights={"sparse": 0.4, "dense": 0.6},
+    normalizer_configs={
+        "sparse": "bayes",  # Robust to BM25 outliers
+        "dense": None,      # Optional: COSINE is already in [0, 1]
+    },
+)
+
+# 3. Combine and re-rank
+final_results = reranker.rerank({
+    "sparse": sparse_results,
+    "dense": dense_results,
+})
+
+print("\nFinal re-ranked results:")
+for i, doc in enumerate(final_results[:5]):
+    print(f"  {i+1}. {doc.fields['text']} (score: {doc.score:.4f})")
+```
+
+### Standalone re-ranking (no zvec collection)
+
+```python
+# If you're not using zvec, you can still use the rerankers standalone
+
+# Mock search results from different sources
+bm25_results = [
+    Doc(id="doc1", score=15.5, fields={"text": "Machine learning..."}),
+    Doc(id="doc2", score=12.3, fields={"text": "Deep neural..."}),
+    Doc(id="doc3", score=8.7, fields={"text": "AI systems..."}),
+]
+
+dense_results = [
+    Doc(id="doc2", score=0.92, fields={"text": "Deep neural..."}),
+    Doc(id="doc1", score=0.75, fields={"text": "Machine learning..."}),
+    Doc(id="doc4", score=0.68, fields={"text": "Data science..."}),
+]
+
+# Re-rank with explicit metrics
+reranker = NormalizedWeightedReRanker(
+    topn=10,
+    metrics={
+        "bm25": None,              # BM25 scores
+        "dense": MetricType.COSINE,  # COSINE distances [0, 2]
+    },
+    weights={"bm25": 0.4, "dense": 0.6},
+)
+
+final_results = reranker.rerank({
+    "bm25": bm25_results,
+    "dense": dense_results,
+})
+```
+
+---
+
+## Resources
+
+- [Full Documentation](https://zvec-db.readthedocs.io/)
+- [Examples](examples/)
+- [Issue Tracker](https://github.com/ccdv-ai/zvec-db/issues)