embenx 1.3.0__tar.gz

embenx-1.3.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+.DS_Store
+*.db
+*.lance
+docs/*.html
+!docs/index.html
+docs/

embenx-1.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aditya Karnam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

embenx-1.3.0/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: embenx
+Version: 1.3.0
+Summary: Universal embedding retrieval toolkit & benchmark. Search, filter, and rerank across 15+ vector backends with a unified Python API and CLI.
+Author-email: adityak74 <adityakarnam@gmail.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: annoy>=1.17.0
+Requires-Dist: chromadb>=0.4.0
+Requires-Dist: datasets>=2.14.0
+Requires-Dist: duckdb>=1.0.0
+Requires-Dist: faiss-cpu>=1.7.4
+Requires-Dist: flashrank>=0.2.10
+Requires-Dist: hnswlib>=0.8.0
+Requires-Dist: lancedb>=0.2.0
+Requires-Dist: litellm>=1.0.0
+Requires-Dist: mcp>=1.26.0
+Requires-Dist: networkx>=3.0
+Requires-Dist: pillow>=10.0.0
+Requires-Dist: plotly>=5.18.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: psycopg2-binary>=2.9.0
+Requires-Dist: pyarrow>=12.0.0
+Requires-Dist: pymilvus>=2.3.0
+Requires-Dist: qdrant-client>=1.5.0
+Requires-Dist: rank-bm25>=0.2.2
+Requires-Dist: requests>=2.31.0
+Requires-Dist: rerankers>=0.10.0
+Requires-Dist: rich>=10.11.0
+Requires-Dist: safetensors>=0.4.0
+Requires-Dist: scikit-learn>=1.3.0
+Requires-Dist: streamlit>=1.32.0
+Requires-Dist: typer>=0.9.0
+Requires-Dist: usearch>=2.9.0
+Requires-Dist: weaviate-client>=4.5.4
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+<h1>Embenx 🚀</h1>
+
+<p>
+  <strong>Universal embedding retrieval toolkit & benchmark.</strong><br/>
+  Search, filter, and rerank across 15+ vector backends (FAISS, ScaNN, pgvector, etc.) with a unified Python API and CLI.
+</p>
+
+<p>
+  <a href="https://github.com/adityak74/embenx/stargazers"><img src="https://img.shields.io/github/stars/adityak74/embenx?style=flat-square&color=yellow" alt="Stars"/></a>
+  <a href="https://github.com/adityak74/embenx/issues"><img src="https://img.shields.io/github/issues/adityak74/embenx?style=flat-square" alt="Issues"/></a>
+  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-green.svg?style=flat-square" alt="MIT License"/></a>
+  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10+-blue.svg?style=flat-square" alt="Python 3.10+"/></a>
+  <a href="https://adityak74.github.io/embenx/"><img src="https://img.shields.io/badge/docs-live-brightgreen?style=flat-square" alt="Docs"/></a>
+  <a href="https://github.com/astral-sh/uv"><img src="https://img.shields.io/badge/uv-ready-purple.svg?style=flat-square" alt="uv ready"/></a>
+</p>
+
+<p>
+  <a href="https://adityak74.github.io/embenx/">Documentation</a> ·
+  <a href="https://github.com/adityak74/embenx/issues">Report Bug</a> ·
+  <a href="https://github.com/adityak74/embenx/issues">Request Feature</a>
+</p>
+
+</div>
+
+---
+
+## What is Embenx?
+
+Embenx is a Python-native retrieval library that sits between raw vector indices and full-blown vector databases. It provides a high-level `Collection` API for managing embeddings and metadata, supporting advanced features like **filtering**, **reranking**, and **quantization** across 15+ backends.
+
+## Library Usage
+
+```python
+from embenx import Collection
+
+# 1. Initialize a collection
+col = Collection(dimension=768, indexer_type="faiss-hnsw")
+
+# 2. Add data
+col.add(
+    vectors=[[0.1, 0.2, ...], [0.3, 0.4, ...]],
+    metadata=[{"category": "AI", "id": 1, "text": "The quick brown fox"}]
+)
+
+# 3. Search with filtering
+results = col.search(
+    query=[0.1, 0.2, ...],
+    top_k=5,
+    where={"category": "AI"}
+)
+
+# 4. Export to production
+col.export_to_production(backend="qdrant", connection_url="http://localhost:6333")
+```
+
+## Agentic Memory (MCP)
+
+Embenx ships with a built-in **Model Context Protocol (MCP)** server. This allows AI agents (like Claude Desktop) to use Embenx collections as their own long-term memory.
+
+### 1. Start the server
+```bash
+embenx mcp-start
+```
+
+## Visual Explorer
+
+Embenx provides a built-in web UI to visualize your vector collections, including an interactive **HNSW Graph Visualizer** and a **RAG Playground**.
+
+```bash
+embenx explorer
+```
+
+## Features
+
+- **Multimodal Support** — Native support for image embeddings (CLIP).
+- **RAG Playground** — Test retrieval quality with an integrated LLM chat loop.
+- **HNSW Graph Visualizer** — Interactive 3D visualization of navigation layers.
+- **Export to Production** — One-click migration to Qdrant or Milvus clusters.
+- **Unified Collection API** — Table-like interface for vectors and metadata.
+- **Retrieval Zoo** — Instant access to pre-indexed collections (SQuAD, MS-MARCO, etc.).
+- **Agentic Memory (MCP)** — Native Model Context Protocol support for AI agents.
+- **Self-Healing Retrieval** — Integrated feedback loops to automatically improve ranking accuracy.
+- **Temporal Memory (Echo)** — Recency-biased retrieval and time-window filtering (arXiv:2502.16090).
+- **Spatial Memory (ESWM)** — Neuroscience-inspired spatial cognitive maps for navigation (ICLR 2026).
+- **TurboQuant Compression** — 1-bit sign-based quantization for activation tensors (arXiv:2504.19874).
+- **ClusterKV Optimization** — Semantic clustering for high-throughput retrieval (arXiv:2412.03213).
+- **Hybrid Search** — Combine dense vectors with sparse BM25 retrieval using RRF.
+- **KV Cache Offloading (RA-KVC)** — Store and retrieve high-dimensional LLM activations using `safetensors`.
+- **SSM State Hydration** — Persist and prime hidden states ($h_0$) for State Space Models (Mamba-2).
+- **Trajectory Retrieval** — Search for similar state/action sequences for World Models.
+- **Visual Explorer** — Built-in web UI to visualize vector clusters and metadata.
+- **Universal model support** — Integrated LiteLLM for any embedding provider.
+- **Portable Formats** — Native support for Parquet, NumPy (.npy/.npz), and FAISS (.index).
+
+## Supported Indexers
+
+| Indexer | Family | Best For |
+| :--- | :--- | :--- |
+| `faiss` | HNSW, IVF, Flat | Production-grade local search |
+| `scann` | Tree-AH | State-of-the-art speed/recall (Linux) |
+| `usearch` | HNSW | High-performance C++, low latency |
+| `pgvector` | Postgres | Embeddings next to relational data |
+| `lancedb` | Columnar | Large disk-based datasets |
+| `simple` | NumPy | Exact search baseline |
+
+## Installation
+
+```bash
+pip install embenx
+```
+
+## Roadmap
+
+See [ROADMAP.md](ROADMAP.md) for our journey towards production-grade agentic retrieval.
+
+## License
+
+Distributed under the **MIT License**.
+
+---
+
+<div align="center">
+  Built with ❤️ for the AI engineering community by <a href="https://github.com/adityak74">adityak74</a>
+</div>

embenx-1.3.0/README.md

@@ -0,0 +1,126 @@
+<div align="center">
+
+<h1>Embenx 🚀</h1>
+
+<p>
+  <strong>Universal embedding retrieval toolkit & benchmark.</strong><br/>
+  Search, filter, and rerank across 15+ vector backends (FAISS, ScaNN, pgvector, etc.) with a unified Python API and CLI.
+</p>
+
+<p>
+  <a href="https://github.com/adityak74/embenx/stargazers"><img src="https://img.shields.io/github/stars/adityak74/embenx?style=flat-square&color=yellow" alt="Stars"/></a>
+  <a href="https://github.com/adityak74/embenx/issues"><img src="https://img.shields.io/github/issues/adityak74/embenx?style=flat-square" alt="Issues"/></a>
+  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-green.svg?style=flat-square" alt="MIT License"/></a>
+  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10+-blue.svg?style=flat-square" alt="Python 3.10+"/></a>
+  <a href="https://adityak74.github.io/embenx/"><img src="https://img.shields.io/badge/docs-live-brightgreen?style=flat-square" alt="Docs"/></a>
+  <a href="https://github.com/astral-sh/uv"><img src="https://img.shields.io/badge/uv-ready-purple.svg?style=flat-square" alt="uv ready"/></a>
+</p>
+
+<p>
+  <a href="https://adityak74.github.io/embenx/">Documentation</a> ·
+  <a href="https://github.com/adityak74/embenx/issues">Report Bug</a> ·
+  <a href="https://github.com/adityak74/embenx/issues">Request Feature</a>
+</p>
+
+</div>
+
+---
+
+## What is Embenx?
+
+Embenx is a Python-native retrieval library that sits between raw vector indices and full-blown vector databases. It provides a high-level `Collection` API for managing embeddings and metadata, supporting advanced features like **filtering**, **reranking**, and **quantization** across 15+ backends.
+
+## Library Usage
+
+```python
+from embenx import Collection
+
+# 1. Initialize a collection
+col = Collection(dimension=768, indexer_type="faiss-hnsw")
+
+# 2. Add data
+col.add(
+    vectors=[[0.1, 0.2, ...], [0.3, 0.4, ...]],
+    metadata=[{"category": "AI", "id": 1, "text": "The quick brown fox"}]
+)
+
+# 3. Search with filtering
+results = col.search(
+    query=[0.1, 0.2, ...],
+    top_k=5,
+    where={"category": "AI"}
+)
+
+# 4. Export to production
+col.export_to_production(backend="qdrant", connection_url="http://localhost:6333")
+```
+
+## Agentic Memory (MCP)
+
+Embenx ships with a built-in **Model Context Protocol (MCP)** server. This allows AI agents (like Claude Desktop) to use Embenx collections as their own long-term memory.
+
+### 1. Start the server
+```bash
+embenx mcp-start
+```
+
+## Visual Explorer
+
+Embenx provides a built-in web UI to visualize your vector collections, including an interactive **HNSW Graph Visualizer** and a **RAG Playground**.
+
+```bash
+embenx explorer
+```
+
+## Features
+
+- **Multimodal Support** — Native support for image embeddings (CLIP).
+- **RAG Playground** — Test retrieval quality with an integrated LLM chat loop.
+- **HNSW Graph Visualizer** — Interactive 3D visualization of navigation layers.
+- **Export to Production** — One-click migration to Qdrant or Milvus clusters.
+- **Unified Collection API** — Table-like interface for vectors and metadata.
+- **Retrieval Zoo** — Instant access to pre-indexed collections (SQuAD, MS-MARCO, etc.).
+- **Agentic Memory (MCP)** — Native Model Context Protocol support for AI agents.
+- **Self-Healing Retrieval** — Integrated feedback loops to automatically improve ranking accuracy.
+- **Temporal Memory (Echo)** — Recency-biased retrieval and time-window filtering (arXiv:2502.16090).
+- **Spatial Memory (ESWM)** — Neuroscience-inspired spatial cognitive maps for navigation (ICLR 2026).
+- **TurboQuant Compression** — 1-bit sign-based quantization for activation tensors (arXiv:2504.19874).
+- **ClusterKV Optimization** — Semantic clustering for high-throughput retrieval (arXiv:2412.03213).
+- **Hybrid Search** — Combine dense vectors with sparse BM25 retrieval using RRF.
+- **KV Cache Offloading (RA-KVC)** — Store and retrieve high-dimensional LLM activations using `safetensors`.
+- **SSM State Hydration** — Persist and prime hidden states ($h_0$) for State Space Models (Mamba-2).
+- **Trajectory Retrieval** — Search for similar state/action sequences for World Models.
+- **Visual Explorer** — Built-in web UI to visualize vector clusters and metadata.
+- **Universal model support** — Integrated LiteLLM for any embedding provider.
+- **Portable Formats** — Native support for Parquet, NumPy (.npy/.npz), and FAISS (.index).
+
+## Supported Indexers
+
+| Indexer | Family | Best For |
+| :--- | :--- | :--- |
+| `faiss` | HNSW, IVF, Flat | Production-grade local search |
+| `scann` | Tree-AH | State-of-the-art speed/recall (Linux) |
+| `usearch` | HNSW | High-performance C++, low latency |
+| `pgvector` | Postgres | Embeddings next to relational data |
+| `lancedb` | Columnar | Large disk-based datasets |
+| `simple` | NumPy | Exact search baseline |
+
+## Installation
+
+```bash
+pip install embenx
+```
+
+## Roadmap
+
+See [ROADMAP.md](ROADMAP.md) for our journey towards production-grade agentic retrieval.
+
+## License
+
+Distributed under the **MIT License**.
+
+---
+
+<div align="center">
+  Built with ❤️ for the AI engineering community by <a href="https://github.com/adityak74">adityak74</a>
+</div>

embenx-1.3.0/SKILL.md

@@ -0,0 +1,4 @@
+# Embenx Skill 🚀
+Optimized for high-performance embedding retrieval and benchmarking.
+Use `Collection` for high-level operations.
+    

embenx-1.3.0/benchmark.py

@@ -0,0 +1,239 @@
+import importlib.util
+import inspect
+import os
+import time
+from typing import List, Dict, Any, Optional
+
+import psutil
+from rich.console import Console
+from rich.table import Table
+
+from data import load_documents
+from indexers import BaseIndexer, get_indexer_map
+from llm import Embedder
+
+
+def get_memory_usage():
+    process = psutil.Process(os.getpid())
+    return process.memory_info().rss / 1024 / 1024  # MB
+
+
+def load_custom_indexer(script_path: str, console: Console):
+    """
+    Dynamically load a class inheriting from BaseIndexer from a given script.
+    """
+    try:
+        spec = importlib.util.spec_from_file_location("custom_indexer", script_path)
+        if spec is None or spec.loader is None:
+            console.print(f"[red]Could not load spec for {script_path}[/red]")
+            return None, None
+
+        module = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(module)
+
+        for name, obj in inspect.getmembers(module):
+            if inspect.isclass(obj) and issubclass(obj, BaseIndexer) and obj is not BaseIndexer:
+                return name, obj
+
+        console.print(f"[red]No class inheriting from BaseIndexer found in {script_path}[/red]")
+        return None, None
+    except Exception as e:
+        console.print(f"[red]Error loading custom indexer from {script_path}: {e}[/red]")
+        return None, None
+
+
+def benchmark_single_indexer(name, indexer_cls, dimension, embeddings, metadata, console, cleanup=True):
+    console.print(f"\n[bold cyan]--- Benchmarking {name.upper()} ---[/bold cyan]")
+    indexer = indexer_cls(dimension=dimension)
+
+    # Build Index
+    mem_before = get_memory_usage()
+    t0 = time.perf_counter()
+    try:
+        indexer.build_index(embeddings, metadata)
+        build_time = time.perf_counter() - t0
+    except Exception as e:
+        console.print(f"[red]Failed to build index for {name}: {e}[/red]")
+        return None
+
+    mem_after = get_memory_usage()
+    mem_diff = mem_after - mem_before
+    index_size = indexer.get_size()
+
+    # Query Benchmarking
+    query_embeddings = embeddings[: min(10, len(embeddings))]
+    query_time = 0
+    if query_embeddings:
+        t0 = time.perf_counter()
+        for q_emb in query_embeddings:
+            indexer.search(q_emb, top_k=5)
+        query_time = (time.perf_counter() - t0) / len(query_embeddings) * 1000  # ms per query
+
+    result = {
+        "Indexer": name.upper(),
+        "Build Time (s)": f"{build_time:.4f}",
+        "Query Time (ms)": f"{query_time:.2f}",
+        "Index Size (KB)": f"{index_size / 1024:.2f}",
+        "Memory Diff (MB)": f"{mem_diff:.2f}",
+    }
+
+    if cleanup:
+        indexer.cleanup()
+
+    console.print(f"Done {name.upper()}.")
+    return result
+
+
+def run_benchmark(
+    dataset_name: str,
+    split: str,
+    text_column: str,
+    max_docs: int,
+    indexer_names: List[str],
+    model_name: str,
+    console: Console,
+    data_files: str = None,
+    cleanup: bool = True,
+    custom_indexer_script: str = None,
+    subset: str = "default", # Added as optional
+):
+    """
+    Run Embenx benchmarks. Matches original signature for test compatibility.
+    """
+    # Load Data
+    console.print(f"\n[bold]Loading up to {max_docs} documents from {dataset_name}...[/bold]")
+    
+    # Check if dataset_name is actually a path (Parquet benchmark use case)
+    if os.path.exists(dataset_name) and dataset_name.endswith(".parquet"):
+        from core import Collection
+        col = Collection.from_parquet(dataset_name)
+        docs = col._metadata
+        embeddings = col._vectors.tolist()
+        dimension = col.dimension
+    else:
+        # Standard HF/Zoo load
+        docs = load_documents(dataset_name, subset, split, max_docs)
+
+        if not docs:
+            console.print("[red]No documents loaded. Exiting.[/red]")
+            return
+        console.print(f"Loaded {len(docs)} documents.")
+
+        # Embed Data
+        console.print(f"\n[bold]Generating embeddings using LiteLLM ({model_name})...[/bold]")
+        embedder = Embedder(model_name)
+        
+        text_field = text_column
+        if text_field not in docs[0] and "text" in docs[0]: text_field = "text"
+        elif text_field not in docs[0] and "content" in docs[0]: text_field = "content"
+        
+        texts = [d.get(text_field, str(d)) for d in docs]
+        
+        t0 = time.perf_counter()
+        embeddings = embedder.embed_texts(texts)
+        emb_time = time.perf_counter() - t0
+
+        if not embeddings:
+            console.print("[red]Failed to generate embeddings.[/red]")
+            return
+
+        dimension = len(embeddings[0])
+        console.print(f"Generated {len(embeddings)} embeddings of dimension {dimension} in {emb_time:.2f}s.")
+
+    # Initialize Indexers
+    indexers_map = get_indexer_map()
+    
+    if custom_indexer_script:
+        custom_name, custom_cls = load_custom_indexer(custom_indexer_script, console)
+        if custom_cls:
+            c_name_lower = custom_name.lower()
+            indexers_map[c_name_lower] = custom_cls
+            console.print(f"[green]✓[/green] Successfully loaded custom indexer: [bold]{custom_name}[/bold]")
+            if c_name_lower not in [x.lower() for x in indexer_names]:
+                indexer_names.append(c_name_lower)
+
+    results = []
+    for name in indexer_names:
+        name_lower = name.lower()
+        if name_lower not in indexers_map:
+            console.print(f"[yellow]Warning: Indexer '{name}' not found. Skipping.[/yellow]")
+            continue
+
+        res = benchmark_single_indexer(
+            name, indexers_map[name_lower], dimension, embeddings, docs, console, cleanup
+        )
+        if res:
+            results.append(res)
+
+    # Report
+    if results:
+        display_results(results, console)
+        return results
+    return []
+
+
+def display_results(results, console):
+    console.print("\n[bold green]Benchmark Results[/bold green]")
+    table = Table(show_header=True, header_style="bold magenta")
+    table.add_column("Indexer", style="cyan")
+    table.add_column("Build Time (s)", justify="right")
+    table.add_column("Query Time (ms/query)", justify="right")
+    table.add_column("Index Size (KB)", justify="right")
+    table.add_column("Memory Added (MB)", justify="right")
+
+    for r in results:
+        table.add_row(
+            r["Indexer"],
+            r["Build Time (s)"],
+            r["Query Time (ms)"],
+            r["Index Size (KB)"],
+            r["Memory Diff (MB)"],
+        )
+    console.print(table)
+
+def generate_report(results: List[Dict[str, Any]], dataset_name: str, output_path: str = "benchmark_report.md"):
+    """
+    Generate a formatted Markdown technical report from benchmark results.
+    """
+    import datetime
+    
+    report = []
+    report.append(f"# Embenx Retrieval Benchmark Report 🚀")
+    report.append(f"Generated on: {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
+    report.append(f"Dataset: **{dataset_name}**")
+    report.append("\n## Executive Summary")
+    
+    if not results:
+        report.append("No results to report.")
+    else:
+        # Find winners
+        query_times = [float(r["Query Time (ms)"]) for r in results]
+        fastest_idx = query_times.index(min(query_times))
+        fastest = results[fastest_idx]["Indexer"]
+        
+        sizes = [float(r["Index Size (KB)"]) for r in results]
+        smallest_idx = sizes.index(min(sizes))
+        smallest = results[smallest_idx]["Indexer"]
+        
+        report.append(f"- **Fastest Indexer**: {fastest} ({min(query_times):.2f} ms/query)")
+        report.append(f"- **Most Memory Efficient**: {smallest} ({min(sizes):.2f} KB)")
+        
+        report.append("\n## Results Table")
+        report.append("| Indexer | Build Time (s) | Query Time (ms) | Index Size (KB) | Memory Diff (MB) |")
+        report.append("| :--- | :--- | :--- | :--- | :--- |")
+        
+        for r in results:
+            report.append(f"| {r['Indexer']} | {r['Build Time (s)']} | {r['Query Time (ms)']} | {r['Index Size (KB)']} | {r['Memory Diff (MB)']} |")
+            
+        report.append("\n## Analysis & Recommendations")
+        report.append("Based on the data above, we recommend:")
+        if "FAISS-HNSW" in [r["Indexer"] for r in results]:
+            report.append("- Use **FAISS-HNSW** for production-grade local search balancing speed and memory.")
+        if "SCANN" in [r["Indexer"] for r in results]:
+            report.append("- Use **ScaNN** for state-of-the-art speed/recall if on supported hardware.")
+        report.append("- For ultra-low latency requirements, prioritize indexers with sub-1ms query times.")
+    
+    with open(output_path, "w") as f:
+        f.write("\n".join(report))
+    
+    return output_path