ragmint 0.1.0__tar.gz → 0.2.0__tar.gz

{ragmint-0.1.0/src/ragmint.egg-info → ragmint-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragmint
-Version: 0.1.0
+Version: 0.2.0
 Summary: A modular framework for evaluating and optimizing RAG pipelines.
 Author-email: Andre Oliveira <oandreoliveira@outlook.com>
 License: Apache License 2.0
@@ -22,6 +22,8 @@ Requires-Dist: faiss-cpu; sys_platform != "darwin"
 Requires-Dist: optuna>=3.0
 Requires-Dist: pytest
 Requires-Dist: colorama
+Requires-Dist: google-generativeai>=0.8.0
+Requires-Dist: supabase>=2.4.0
 Dynamic: license-file
 
 # Ragmint
@@ -36,17 +38,19 @@ Dynamic: license-file
 
 **Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
 
-It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**.
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
 
 ---
 
 ## ✨ Features
 
 - ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
 - 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
 - ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
 - 🧩 **Embeddings** — OpenAI, HuggingFace  
-- 🧠 **Rerankers** — MMR, CrossEncoder (extensible via plugin interface)  
 - 💾 **Caching, experiment tracking, and reproducibility** out of the box  
 - 🧰 **Clean modular structure** for easy integration in research and production setups  
 
@@ -103,47 +107,133 @@ print(result)
 
 ---
 
+## 🧪 Dataset Options
+
+Ragmint can automatically load evaluation datasets for your RAG pipeline:
+
+| Mode | Example | Description |
+|------|----------|-------------|
+| 🧱 **Default** | `validation_set=None` | Uses built-in `experiments/validation_qa.json` |
+| 📁 **Custom File** | `validation_set="data/my_eval.json"` | Load your own QA dataset (JSON or CSV) |
+| 🌐 **Hugging Face Dataset** | `validation_set="squad"` | Automatically downloads benchmark datasets (requires `pip install datasets`) |
+
+### Example
+
+```python
+from ragmint.tuner import RAGMint
+
+ragmint = RAGMint(
+    docs_path="data/docs/",
+    retrievers=["faiss", "chroma"],
+    embeddings=["text-embedding-3-small"],
+    rerankers=["mmr"],
+)
+
+# Use built-in default
+ragmint.optimize(validation_set=None)
+
+# Use Hugging Face benchmark
+ragmint.optimize(validation_set="squad")
+
+# Use your own dataset
+ragmint.optimize(validation_set="data/custom_qa.json")
+```
+
+---
+
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
 ## 🧩 Folder Structure
 
 ```
 ragmint/
 ├── core/
-│   ├── pipeline.py         # RAGPipeline implementation
-│   ├── retriever.py        # Retriever logic (FAISS, Chroma)
-│   ├── reranker.py         # MMR + CrossEncoder rerankers
-│   └── embedding.py        # Embedding backends
-├── tuner.py                # Grid, Random, Bayesian optimization (Optuna)
-├── utils/                  # Metrics, logging, caching helpers
-├── configs/                # Default experiment configs
-├── experiments/            # Saved experiment results
-├── tests/                  # Unit tests for all components
-├── main.py                 # CLI entrypoint for tuning
-└── pyproject.toml          # Project dependencies & build metadata
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
 ```
 
 ---
 
 ## 🧪 Running Tests
 
-To verify your setup:
-
 ```bash
 pytest -v
 ```
 
-Or to test a specific component (e.g., reranker):
-
+To include integration tests with Gemini or Claude APIs:
 ```bash
-pytest tests/test_reranker.py -v
+pytest -m integration
 ```
 
-All tests are designed for **Pytest** and run with lightweight mock data.
-
 ---
 
 ## ⚙️ Configuration via `pyproject.toml`
 
-Your `pyproject.toml` automatically includes:
+Your `pyproject.toml` includes all required dependencies:
 
 ```toml
 [project]
@@ -158,6 +248,8 @@ dependencies = [
     "pytest",
     "openai",
     "tqdm",
+    "google-generativeai",
+    "google-genai",
 ]
 ```
 
@@ -165,10 +257,10 @@ dependencies = [
 
 ## 📊 Example Experiment Workflow
 
-1. Define your retriever and reranker configuration in YAML  
-2. Launch an optimization search (Grid, Random, or Bayesian)  
-3. Ragmint evaluates combinations automatically and reports top results  
-4. Export best parameters for production pipelines  
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
 
 ---
 
@@ -181,7 +273,7 @@ flowchart TD
     C --> D[Reranker]
     D --> E[Generator]
     E --> F[Evaluation]
-    F --> G[Optuna Tuner]
+    F --> G[Optuna / AutoRAGTuner]
     G -->|Best Params| B
 ```
 
@@ -191,8 +283,9 @@ flowchart TD
 
 ```
 [INFO] Starting Bayesian optimization with Optuna
-[INFO] Trial 7 finished: recall=0.83, latency=0.42s
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
 [INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
 ```
 
 ---
@@ -200,8 +293,9 @@ flowchart TD
 ## 🧠 Why Ragmint?
 
 - Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
-- Works with **LangChain**, **LlamaIndex**, or standalone RAG setups  
-- Designed for **extensibility** — plug in your own models, retrievers, or metrics  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
 
 ---
 

ragmint-0.2.0/README.md

@@ -0,0 +1,284 @@
+# Ragmint
+
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+![License](https://img.shields.io/badge/license-Apache%202.0-green)
+![Tests](https://github.com/andyolivers/ragmint/actions/workflows/tests.yml/badge.svg)
+![Optuna](https://img.shields.io/badge/Optuna-Integrated-orange)
+![Status](https://img.shields.io/badge/Status-Active-success)
+
+![](/assets/images/ragmint-banner.png)
+
+**Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
+
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
+
+---
+
+## ✨ Features
+
+- ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
+- 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
+- ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
+- 🧩 **Embeddings** — OpenAI, HuggingFace  
+- 💾 **Caching, experiment tracking, and reproducibility** out of the box  
+- 🧰 **Clean modular structure** for easy integration in research and production setups  
+
+---
+
+## 🚀 Quick Start
+
+### 1️⃣ Installation
+
+```bash
+git clone https://github.com/andyolivers/ragmint.git
+cd ragmint
+pip install -e .
+```
+
+> The `-e` flag installs Ragmint in editable (development) mode.  
+> Requires **Python ≥ 3.9**.
+
+---
+
+### 2️⃣ Run a RAG Optimization Experiment
+
+```bash
+python ragmint/main.py --config configs/default.yaml --search bayesian
+```
+
+Example `configs/default.yaml`:
+```yaml
+retriever: faiss
+embedding_model: text-embedding-3-small
+reranker:
+  mode: mmr
+  lambda_param: 0.5
+optimization:
+  search_method: bayesian
+  n_trials: 20
+```
+
+---
+
+### 3️⃣ Manual Pipeline Usage
+
+```python
+from ragmint.core.pipeline import RAGPipeline
+
+pipeline = RAGPipeline({
+    "embedding_model": "text-embedding-3-small",
+    "retriever": "faiss",
+})
+
+result = pipeline.run("What is retrieval-augmented generation?")
+print(result)
+```
+
+---
+
+## 🧪 Dataset Options
+
+Ragmint can automatically load evaluation datasets for your RAG pipeline:
+
+| Mode | Example | Description |
+|------|----------|-------------|
+| 🧱 **Default** | `validation_set=None` | Uses built-in `experiments/validation_qa.json` |
+| 📁 **Custom File** | `validation_set="data/my_eval.json"` | Load your own QA dataset (JSON or CSV) |
+| 🌐 **Hugging Face Dataset** | `validation_set="squad"` | Automatically downloads benchmark datasets (requires `pip install datasets`) |
+
+### Example
+
+```python
+from ragmint.tuner import RAGMint
+
+ragmint = RAGMint(
+    docs_path="data/docs/",
+    retrievers=["faiss", "chroma"],
+    embeddings=["text-embedding-3-small"],
+    rerankers=["mmr"],
+)
+
+# Use built-in default
+ragmint.optimize(validation_set=None)
+
+# Use Hugging Face benchmark
+ragmint.optimize(validation_set="squad")
+
+# Use your own dataset
+ragmint.optimize(validation_set="data/custom_qa.json")
+```
+
+---
+
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
+## 🧩 Folder Structure
+
+```
+ragmint/
+├── core/
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
+```
+
+---
+
+## 🧪 Running Tests
+
+```bash
+pytest -v
+```
+
+To include integration tests with Gemini or Claude APIs:
+```bash
+pytest -m integration
+```
+
+---
+
+## ⚙️ Configuration via `pyproject.toml`
+
+Your `pyproject.toml` includes all required dependencies:
+
+```toml
+[project]
+name = "ragmint"
+version = "0.1.0"
+dependencies = [
+    "numpy",
+    "optuna",
+    "scikit-learn",
+    "faiss-cpu",
+    "chromadb",
+    "pytest",
+    "openai",
+    "tqdm",
+    "google-generativeai",
+    "google-genai",
+]
+```
+
+---
+
+## 📊 Example Experiment Workflow
+
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
+
+---
+
+## 🧬 Architecture Overview
+
+```mermaid
+flowchart TD
+    A[Query] --> B[Embedder]
+    B --> C[Retriever]
+    C --> D[Reranker]
+    D --> E[Generator]
+    E --> F[Evaluation]
+    F --> G[Optuna / AutoRAGTuner]
+    G -->|Best Params| B
+```
+
+---
+
+## 📘 Example Output
+
+```
+[INFO] Starting Bayesian optimization with Optuna
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
+[INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
+```
+
+---
+
+## 🧠 Why Ragmint?
+
+- Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
+
+---
+
+## ⚖️ License
+
+Licensed under the **Apache License 2.0** — free for personal, research, and commercial use.
+
+---
+
+## 👤 Author
+
+**André Oliveira**  
+[andyolivers.com](https://andyolivers.com)  
+Data Scientist | AI Engineer

{ragmint-0.1.0 → ragmint-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ragmint"
-version = "0.1.0"
+version = "0.2.0"
 description = "A modular framework for evaluating and optimizing RAG pipelines."
 readme = "README.md"
 license = { text = "Apache License 2.0" }
@@ -24,7 +24,9 @@ dependencies = [
   "faiss-cpu; sys_platform != 'darwin'",
   "optuna>=3.0",
   "pytest",
-  "colorama"
+  "colorama",
+  "google-generativeai>=0.8.0",
+  "supabase>=2.4.0"
 ]
 
 [project.urls]

ragmint-0.2.0/src/ragmint/autotuner.py

@@ -0,0 +1,33 @@
+"""
+Auto-RAG Tuner
+--------------
+Recommends retriever–embedding pairs dynamically based on corpus size
+and dataset characteristics. Integrates seamlessly with RAGMint evaluator.
+"""
+
+from .core.evaluation import evaluate_config
+
+
+class AutoRAGTuner:
+    def __init__(self, corpus_stats: dict):
+        """
+        corpus_stats: dict
+            Example: {'size': 12000, 'avg_len': 240}
+        """
+        self.corpus_stats = corpus_stats
+
+    def recommend(self):
+        size = self.corpus_stats.get("size", 0)
+        avg_len = self.corpus_stats.get("avg_len", 0)
+
+        if size < 1000:
+            return {"retriever": "BM25", "embedding_model": "OpenAI"}
+        elif size < 10000:
+            return {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+        else:
+            return {"retriever": "FAISS", "embedding_model": "InstructorXL"}
+
+    def auto_tune(self, validation_data):
+        config = self.recommend()
+        results = evaluate_config(config, validation_data)
+        return {"recommended": config, "results": results}

{ragmint-0.1.0 → ragmint-0.2.0}/src/ragmint/core/evaluation.py

@@ -25,3 +25,14 @@ class Evaluator:
 
     def _similarity(self, a: str, b: str) -> float:
         return SequenceMatcher(None, a, b).ratio()
+
+def evaluate_config(config, validation_data):
+    evaluator = Evaluator()
+    results = []
+    for sample in validation_data:
+        query = sample.get("query", "")
+        answer = sample.get("answer", "")
+        context = sample.get("context", "")
+        results.append(evaluator.evaluate(query, answer, context))
+    return results
+

ragmint-0.2.0/src/ragmint/explainer.py

@@ -0,0 +1,61 @@
+"""
+Interpretability Layer
+----------------------
+Uses Gemini or Anthropic Claude to explain why one RAG configuration
+outperforms another. Falls back gracefully if no API key is provided.
+"""
+
+import os
+import json
+
+
+def explain_results(results_a: dict, results_b: dict, model: str = "gemini-1.5-pro") -> str:
+    """
+    Generate a natural-language explanation comparing two RAG experiment results.
+    Priority:
+      1. Anthropic Claude (if ANTHROPIC_API_KEY is set)
+      2. Google Gemini (if GOOGLE_API_KEY is set)
+      3. Fallback text message
+    """
+    prompt = f"""
+    You are an AI evaluation expert.
+    Compare these two RAG experiment results and explain why one performs better.
+    Metrics A: {json.dumps(results_a, indent=2)}
+    Metrics B: {json.dumps(results_b, indent=2)}
+    Provide a concise, human-friendly explanation and practical improvement tips.
+    """
+
+    anthropic_key = os.getenv("ANTHROPIC_API_KEY")
+    google_key = os.getenv("GEMINI_API_KEY")
+
+
+    # 1️⃣ Try Anthropic Claude first
+    if anthropic_key:
+        try:
+            from anthropic import Anthropic
+            client = Anthropic(api_key=anthropic_key)
+            response = client.messages.create(
+                model="claude-3-opus-20240229",
+                max_tokens=300,
+                messages=[{"role": "user", "content": prompt}],
+            )
+            return response.content[0].text
+        except Exception as e:
+            return f"[Claude unavailable] {e}"
+
+    # 2️⃣ Fallback to Google Gemini
+    elif google_key:
+        try:
+            import google.generativeai as genai
+            genai.configure(api_key=google_key)
+            response = genai.GenerativeModel(model).generate_content(prompt)
+            return response.text
+        except Exception as e:
+            return f"[Gemini unavailable] {e}"
+
+    # 3️⃣ Fallback if neither key is available
+    else:
+        return (
+            "[No LLM available] Please set ANTHROPIC_API_KEY or GOOGLE_API_KEY "
+            "to enable interpretability via Claude or Gemini."
+        )