ragmint 0.1.0__tar.gz

ragmint-0.1.0/LICENSE

@@ -0,0 +1,19 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+Copyright 2025 André Oliveira
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

ragmint-0.1.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.4
+Name: ragmint
+Version: 0.1.0
+Summary: A modular framework for evaluating and optimizing RAG pipelines.
+Author-email: Andre Oliveira <oandreoliveira@outlook.com>
+License: Apache License 2.0
+Project-URL: Homepage, https://github.com/andyolivers/ragmint
+Project-URL: Documentation, https://andyolivers.com
+Project-URL: Issues, https://github.com/andyolivers/ragmint/issues
+Keywords: RAG,LLM,retrieval,optimization,AI,evaluation
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=2.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: openai>=1.0
+Requires-Dist: tqdm
+Requires-Dist: pyyaml
+Requires-Dist: chromadb>=0.4
+Requires-Dist: faiss-cpu; sys_platform != "darwin"
+Requires-Dist: optuna>=3.0
+Requires-Dist: pytest
+Requires-Dist: colorama
+Dynamic: license-file
+
+# Ragmint
+
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+![License](https://img.shields.io/badge/license-Apache%202.0-green)
+![Tests](https://github.com/andyolivers/ragmint/actions/workflows/tests.yml/badge.svg)
+![Optuna](https://img.shields.io/badge/Optuna-Integrated-orange)
+![Status](https://img.shields.io/badge/Status-Active-success)
+
+![](/assets/images/ragmint-banner.png)
+
+**Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
+
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**.
+
+---
+
+## ✨ Features
+
+- ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
+- ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
+- 🧩 **Embeddings** — OpenAI, HuggingFace  
+- 🧠 **Rerankers** — MMR, CrossEncoder (extensible via plugin interface)  
+- 💾 **Caching, experiment tracking, and reproducibility** out of the box  
+- 🧰 **Clean modular structure** for easy integration in research and production setups  
+
+---
+
+## 🚀 Quick Start
+
+### 1️⃣ Installation
+
+```bash
+git clone https://github.com/andyolivers/ragmint.git
+cd ragmint
+pip install -e .
+```
+
+> The `-e` flag installs Ragmint in editable (development) mode.  
+> Requires **Python ≥ 3.9**.
+
+---
+
+### 2️⃣ Run a RAG Optimization Experiment
+
+```bash
+python ragmint/main.py --config configs/default.yaml --search bayesian
+```
+
+Example `configs/default.yaml`:
+```yaml
+retriever: faiss
+embedding_model: text-embedding-3-small
+reranker:
+  mode: mmr
+  lambda_param: 0.5
+optimization:
+  search_method: bayesian
+  n_trials: 20
+```
+
+---
+
+### 3️⃣ Manual Pipeline Usage
+
+```python
+from ragmint.core.pipeline import RAGPipeline
+
+pipeline = RAGPipeline({
+    "embedding_model": "text-embedding-3-small",
+    "retriever": "faiss",
+})
+
+result = pipeline.run("What is retrieval-augmented generation?")
+print(result)
+```
+
+---
+
+## 🧩 Folder Structure
+
+```
+ragmint/
+├── core/
+│   ├── pipeline.py         # RAGPipeline implementation
+│   ├── retriever.py        # Retriever logic (FAISS, Chroma)
+│   ├── reranker.py         # MMR + CrossEncoder rerankers
+│   └── embedding.py        # Embedding backends
+├── tuner.py                # Grid, Random, Bayesian optimization (Optuna)
+├── utils/                  # Metrics, logging, caching helpers
+├── configs/                # Default experiment configs
+├── experiments/            # Saved experiment results
+├── tests/                  # Unit tests for all components
+├── main.py                 # CLI entrypoint for tuning
+└── pyproject.toml          # Project dependencies & build metadata
+```
+
+---
+
+## 🧪 Running Tests
+
+To verify your setup:
+
+```bash
+pytest -v
+```
+
+Or to test a specific component (e.g., reranker):
+
+```bash
+pytest tests/test_reranker.py -v
+```
+
+All tests are designed for **Pytest** and run with lightweight mock data.
+
+---
+
+## ⚙️ Configuration via `pyproject.toml`
+
+Your `pyproject.toml` automatically includes:
+
+```toml
+[project]
+name = "ragmint"
+version = "0.1.0"
+dependencies = [
+    "numpy",
+    "optuna",
+    "scikit-learn",
+    "faiss-cpu",
+    "chromadb",
+    "pytest",
+    "openai",
+    "tqdm",
+]
+```
+
+---
+
+## 📊 Example Experiment Workflow
+
+1. Define your retriever and reranker configuration in YAML  
+2. Launch an optimization search (Grid, Random, or Bayesian)  
+3. Ragmint evaluates combinations automatically and reports top results  
+4. Export best parameters for production pipelines  
+
+---
+
+## 🧬 Architecture Overview
+
+```mermaid
+flowchart TD
+    A[Query] --> B[Embedder]
+    B --> C[Retriever]
+    C --> D[Reranker]
+    D --> E[Generator]
+    E --> F[Evaluation]
+    F --> G[Optuna Tuner]
+    G -->|Best Params| B
+```
+
+---
+
+## 📘 Example Output
+
+```
+[INFO] Starting Bayesian optimization with Optuna
+[INFO] Trial 7 finished: recall=0.83, latency=0.42s
+[INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+```
+
+---
+
+## 🧠 Why Ragmint?
+
+- Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
+- Works with **LangChain**, **LlamaIndex**, or standalone RAG setups  
+- Designed for **extensibility** — plug in your own models, retrievers, or metrics  
+
+---
+
+## ⚖️ License
+
+Licensed under the **Apache License 2.0** — free for personal, research, and commercial use.
+
+---
+
+## 👤 Author
+
+**André Oliveira**  
+[andyolivers.com](https://andyolivers.com)  
+Data Scientist | AI Engineer

ragmint-0.1.0/README.md

@@ -0,0 +1,192 @@
+# Ragmint
+
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+![License](https://img.shields.io/badge/license-Apache%202.0-green)
+![Tests](https://github.com/andyolivers/ragmint/actions/workflows/tests.yml/badge.svg)
+![Optuna](https://img.shields.io/badge/Optuna-Integrated-orange)
+![Status](https://img.shields.io/badge/Status-Active-success)
+
+![](/assets/images/ragmint-banner.png)
+
+**Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
+
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**.
+
+---
+
+## ✨ Features
+
+- ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
+- ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
+- 🧩 **Embeddings** — OpenAI, HuggingFace  
+- 🧠 **Rerankers** — MMR, CrossEncoder (extensible via plugin interface)  
+- 💾 **Caching, experiment tracking, and reproducibility** out of the box  
+- 🧰 **Clean modular structure** for easy integration in research and production setups  
+
+---
+
+## 🚀 Quick Start
+
+### 1️⃣ Installation
+
+```bash
+git clone https://github.com/andyolivers/ragmint.git
+cd ragmint
+pip install -e .
+```
+
+> The `-e` flag installs Ragmint in editable (development) mode.  
+> Requires **Python ≥ 3.9**.
+
+---
+
+### 2️⃣ Run a RAG Optimization Experiment
+
+```bash
+python ragmint/main.py --config configs/default.yaml --search bayesian
+```
+
+Example `configs/default.yaml`:
+```yaml
+retriever: faiss
+embedding_model: text-embedding-3-small
+reranker:
+  mode: mmr
+  lambda_param: 0.5
+optimization:
+  search_method: bayesian
+  n_trials: 20
+```
+
+---
+
+### 3️⃣ Manual Pipeline Usage
+
+```python
+from ragmint.core.pipeline import RAGPipeline
+
+pipeline = RAGPipeline({
+    "embedding_model": "text-embedding-3-small",
+    "retriever": "faiss",
+})
+
+result = pipeline.run("What is retrieval-augmented generation?")
+print(result)
+```
+
+---
+
+## 🧩 Folder Structure
+
+```
+ragmint/
+├── core/
+│   ├── pipeline.py         # RAGPipeline implementation
+│   ├── retriever.py        # Retriever logic (FAISS, Chroma)
+│   ├── reranker.py         # MMR + CrossEncoder rerankers
+│   └── embedding.py        # Embedding backends
+├── tuner.py                # Grid, Random, Bayesian optimization (Optuna)
+├── utils/                  # Metrics, logging, caching helpers
+├── configs/                # Default experiment configs
+├── experiments/            # Saved experiment results
+├── tests/                  # Unit tests for all components
+├── main.py                 # CLI entrypoint for tuning
+└── pyproject.toml          # Project dependencies & build metadata
+```
+
+---
+
+## 🧪 Running Tests
+
+To verify your setup:
+
+```bash
+pytest -v
+```
+
+Or to test a specific component (e.g., reranker):
+
+```bash
+pytest tests/test_reranker.py -v
+```
+
+All tests are designed for **Pytest** and run with lightweight mock data.
+
+---
+
+## ⚙️ Configuration via `pyproject.toml`
+
+Your `pyproject.toml` automatically includes:
+
+```toml
+[project]
+name = "ragmint"
+version = "0.1.0"
+dependencies = [
+    "numpy",
+    "optuna",
+    "scikit-learn",
+    "faiss-cpu",
+    "chromadb",
+    "pytest",
+    "openai",
+    "tqdm",
+]
+```
+
+---
+
+## 📊 Example Experiment Workflow
+
+1. Define your retriever and reranker configuration in YAML  
+2. Launch an optimization search (Grid, Random, or Bayesian)  
+3. Ragmint evaluates combinations automatically and reports top results  
+4. Export best parameters for production pipelines  
+
+---
+
+## 🧬 Architecture Overview
+
+```mermaid
+flowchart TD
+    A[Query] --> B[Embedder]
+    B --> C[Retriever]
+    C --> D[Reranker]
+    D --> E[Generator]
+    E --> F[Evaluation]
+    F --> G[Optuna Tuner]
+    G -->|Best Params| B
+```
+
+---
+
+## 📘 Example Output
+
+```
+[INFO] Starting Bayesian optimization with Optuna
+[INFO] Trial 7 finished: recall=0.83, latency=0.42s
+[INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+```
+
+---
+
+## 🧠 Why Ragmint?
+
+- Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
+- Works with **LangChain**, **LlamaIndex**, or standalone RAG setups  
+- Designed for **extensibility** — plug in your own models, retrievers, or metrics  
+
+---
+
+## ⚖️ License
+
+Licensed under the **Apache License 2.0** — free for personal, research, and commercial use.
+
+---
+
+## 👤 Author
+
+**André Oliveira**  
+[andyolivers.com](https://andyolivers.com)  
+Data Scientist | AI Engineer

ragmint-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ragmint"
+version = "0.1.0"
+description = "A modular framework for evaluating and optimizing RAG pipelines."
+readme = "README.md"
+license = { text = "Apache License 2.0" }
+authors = [
+  { name = "Andre Oliveira", email = "oandreoliveira@outlook.com" }
+]
+keywords = ["RAG", "LLM", "retrieval", "optimization", "AI", "evaluation"]
+requires-python = ">=3.9"
+dependencies = [
+  "numpy>=1.23",
+  "pandas>=2.0",
+  "scikit-learn>=1.3",
+  "openai>=1.0",
+  "tqdm",
+  "pyyaml",
+  "chromadb>=0.4",
+  "faiss-cpu; sys_platform != 'darwin'",
+  "optuna>=3.0",
+  "pytest",
+  "colorama"
+]
+
+[project.urls]
+Homepage = "https://github.com/andyolivers/ragmint"
+Documentation = "https://andyolivers.com"
+Issues = "https://github.com/andyolivers/ragmint/issues"
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-v"

ragmint-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ragmint-0.1.0/src/ragmint/__main__.py

@@ -0,0 +1,28 @@
+from pathlib import Path
+from ragmint.tuner import RAGMint
+
+def main():
+    # Dynamically resolve the path to the installed ragmint package
+    base_dir = Path(__file__).resolve().parent
+
+    docs_path = base_dir / "experiments" / "corpus"
+    validation_file = base_dir / "experiments" / "validation_qa.json"
+
+    rag = RAGMint(
+        docs_path=str(docs_path),
+        retrievers=["faiss"],
+        embeddings=["openai/text-embedding-3-small"],
+        rerankers=["mmr"],
+    )
+
+    best, results = rag.optimize(
+        validation_set=str(validation_file),
+        metric="faithfulness",
+        search_type="bayesian",
+        trials=10,
+    )
+
+    print("Best config found:\n", best)
+
+if __name__ == "__main__":
+    main()

ragmint-0.1.0/src/ragmint/core/chunking.py

@@ -0,0 +1,22 @@
+from typing import List
+
+
+class Chunker:
+    """
+    Handles text chunking and splitting strategies:
+    - Fixed size chunks
+    - Overlapping windows
+    """
+
+    def __init__(self, chunk_size: int = 500, overlap: int = 100):
+        self.chunk_size = chunk_size
+        self.overlap = overlap
+
+    def chunk_text(self, text: str) -> List[str]:
+        chunks = []
+        start = 0
+        while start < len(text):
+            end = start + self.chunk_size
+            chunks.append(text[start:end])
+            start += self.chunk_size - self.overlap
+        return chunks

ragmint-0.1.0/src/ragmint/core/embeddings.py

@@ -0,0 +1,19 @@
+import numpy as np
+
+
+class EmbeddingModel:
+    """
+    Wrapper for embedding backends (OpenAI, HuggingFace, etc.)
+    """
+
+    def __init__(self, backend: str = "dummy"):
+        self.backend = backend
+
+    def encode(self, texts):
+        if self.backend == "openai":
+            # Example placeholder — integrate with actual OpenAI API
+            return [np.random.rand(768) for _ in texts]
+        elif self.backend == "huggingface":
+            return [np.random.rand(768) for _ in texts]
+        else:
+            return [np.random.rand(768) for _ in texts]

ragmint-0.1.0/src/ragmint/core/evaluation.py

@@ -0,0 +1,27 @@
+import time
+from typing import Dict, Any
+from difflib import SequenceMatcher
+
+
+class Evaluator:
+    """
+    Simple evaluation of generated answers:
+      - Faithfulness (similarity between answer and context)
+      - Latency
+    """
+
+    def __init__(self):
+        pass
+
+    def evaluate(self, query: str, answer: str, context: str) -> Dict[str, Any]:
+        start = time.time()
+        faithfulness = self._similarity(answer, context)
+        latency = time.time() - start
+
+        return {
+            "faithfulness": faithfulness,
+            "latency": latency,
+        }
+
+    def _similarity(self, a: str, b: str) -> float:
+        return SequenceMatcher(None, a, b).ratio()

ragmint-0.1.0/src/ragmint/core/pipeline.py

@@ -0,0 +1,38 @@
+from typing import Any, Dict, List
+from .retriever import Retriever
+from .reranker import Reranker
+from .evaluation import Evaluator
+
+
+class RAGPipeline:
+    """
+    Core Retrieval-Augmented Generation pipeline.
+    Simplified (no generator). It retrieves, reranks, and evaluates.
+    """
+
+    def __init__(self, retriever: Retriever, reranker: Reranker, evaluator: Evaluator):
+        self.retriever = retriever
+        self.reranker = reranker
+        self.evaluator = evaluator
+
+    def run(self, query: str, top_k: int = 5) -> Dict[str, Any]:
+        # Retrieve documents
+        retrieved_docs = self.retriever.retrieve(query, top_k=top_k)
+        # Rerank
+        reranked_docs = self.reranker.rerank(query, retrieved_docs)
+
+        # Use top document as pseudo-answer
+        if reranked_docs:
+            answer = reranked_docs[0]["text"]
+        else:
+            answer = ""
+
+        context = "\n".join([d["text"] for d in reranked_docs])
+        metrics = self.evaluator.evaluate(query, answer, context)
+
+        return {
+            "query": query,
+            "answer": answer,
+            "docs": reranked_docs,
+            "metrics": metrics,
+        }

ragmint-0.1.0/src/ragmint/core/reranker.py

@@ -0,0 +1,62 @@
+from typing import List, Dict, Any
+import numpy as np
+
+
+class Reranker:
+    """
+    Supports:
+      - MMR (Maximal Marginal Relevance)
+      - Dummy CrossEncoder (for demonstration)
+    """
+
+    def __init__(self, mode: str = "mmr", lambda_param: float = 0.5, seed: int = 42):
+        self.mode = mode
+        self.lambda_param = lambda_param
+        np.random.seed(seed)
+
+    def rerank(self, query: str, docs: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        if not docs:
+            return []
+
+        if self.mode == "crossencoder":
+            return self._crossencoder_rerank(query, docs)
+        return self._mmr_rerank(query, docs)
+
+    def _mmr_rerank(self, query: str, docs: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        """Perform MMR reranking using dummy similarity scores."""
+        selected = []
+        remaining = docs.copy()
+
+        while remaining and len(selected) < len(docs):
+            if not selected:
+                # pick doc with highest base score
+                best = max(remaining, key=lambda d: d["score"])
+            else:
+                # MMR balancing between relevance and diversity
+                mmr_scores = []
+                for d in remaining:
+                    max_div = max(
+                        [self._similarity(d["text"], s["text"]) for s in selected],
+                        default=0,
+                    )
+                    mmr_score = (
+                        self.lambda_param * d["score"]
+                        - (1 - self.lambda_param) * max_div
+                    )
+                    mmr_scores.append(mmr_score)
+                best = remaining[int(np.argmax(mmr_scores))]
+            selected.append(best)
+            remaining.remove(best)
+
+        return selected
+
+    def _crossencoder_rerank(self, query: str, docs: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        """Adds a small random perturbation to simulate crossencoder reranking."""
+        for d in docs:
+            d["score"] += np.random.uniform(0, 0.1)
+        return sorted(docs, key=lambda d: d["score"], reverse=True)
+
+    def _similarity(self, a: str, b: str) -> float:
+        """Dummy similarity function between two strings."""
+        # Deterministic pseudo-similarity based on hash
+        return abs(hash(a + b)) % 100 / 100.0