ragmint 0.1.1__tar.gz → 0.2.0__tar.gz

{ragmint-0.1.1/src/ragmint.egg-info → ragmint-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragmint
-Version: 0.1.1
+Version: 0.2.0
 Summary: A modular framework for evaluating and optimizing RAG pipelines.
 Author-email: Andre Oliveira <oandreoliveira@outlook.com>
 License: Apache License 2.0
@@ -22,6 +22,8 @@ Requires-Dist: faiss-cpu; sys_platform != "darwin"
 Requires-Dist: optuna>=3.0
 Requires-Dist: pytest
 Requires-Dist: colorama
+Requires-Dist: google-generativeai>=0.8.0
+Requires-Dist: supabase>=2.4.0
 Dynamic: license-file
 
 # Ragmint
@@ -36,17 +38,19 @@ Dynamic: license-file
 
 **Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
 
-It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**.
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
 
 ---
 
 ## ✨ Features
 
 - ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
 - 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
 - ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
 - 🧩 **Embeddings** — OpenAI, HuggingFace  
-- 🧠 **Rerankers** — MMR, CrossEncoder (extensible via plugin interface)  
 - 💾 **Caching, experiment tracking, and reproducibility** out of the box  
 - 🧰 **Clean modular structure** for easy integration in research and production setups  
 
@@ -102,6 +106,7 @@ print(result)
 ```
 
 ---
+
 ## 🧪 Dataset Options
 
 Ragmint can automatically load evaluation datasets for your RAG pipeline:
@@ -136,47 +141,99 @@ ragmint.optimize(validation_set="data/custom_qa.json")
 
 ---
 
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
 ## 🧩 Folder Structure
 
 ```
 ragmint/
 ├── core/
-│   ├── pipeline.py         # RAGPipeline implementation
-│   ├── retriever.py        # Retriever logic (FAISS, Chroma)
-│   ├── reranker.py         # MMR + CrossEncoder rerankers
-│   └── embedding.py        # Embedding backends
-├── tuner.py                # Grid, Random, Bayesian optimization (Optuna)
-├── utils/                  # Metrics, logging, caching helpers
-├── configs/                # Default experiment configs
-├── experiments/            # Saved experiment results
-├── tests/                  # Unit tests for all components
-├── main.py                 # CLI entrypoint for tuning
-└── pyproject.toml          # Project dependencies & build metadata
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
 ```
 
 ---
 
 ## 🧪 Running Tests
 
-To verify your setup:
-
 ```bash
 pytest -v
 ```
 
-Or to test a specific component (e.g., reranker):
-
+To include integration tests with Gemini or Claude APIs:
 ```bash
-pytest tests/test_reranker.py -v
+pytest -m integration
 ```
 
-All tests are designed for **Pytest** and run with lightweight mock data.
-
 ---
 
 ## ⚙️ Configuration via `pyproject.toml`
 
-Your `pyproject.toml` automatically includes:
+Your `pyproject.toml` includes all required dependencies:
 
 ```toml
 [project]
@@ -191,6 +248,8 @@ dependencies = [
     "pytest",
     "openai",
     "tqdm",
+    "google-generativeai",
+    "google-genai",
 ]
 ```
 
@@ -198,10 +257,10 @@ dependencies = [
 
 ## 📊 Example Experiment Workflow
 
-1. Define your retriever and reranker configuration in YAML  
-2. Launch an optimization search (Grid, Random, or Bayesian)  
-3. Ragmint evaluates combinations automatically and reports top results  
-4. Export best parameters for production pipelines  
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
 
 ---
 
@@ -214,7 +273,7 @@ flowchart TD
     C --> D[Reranker]
     D --> E[Generator]
     E --> F[Evaluation]
-    F --> G[Optuna Tuner]
+    F --> G[Optuna / AutoRAGTuner]
     G -->|Best Params| B
 ```
 
@@ -224,8 +283,9 @@ flowchart TD
 
 ```
 [INFO] Starting Bayesian optimization with Optuna
-[INFO] Trial 7 finished: recall=0.83, latency=0.42s
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
 [INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
 ```
 
 ---
@@ -233,8 +293,9 @@ flowchart TD
 ## 🧠 Why Ragmint?
 
 - Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
-- Works with **LangChain**, **LlamaIndex**, or standalone RAG setups  
-- Designed for **extensibility** — plug in your own models, retrievers, or metrics  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
 
 ---
 

{ragmint-0.1.1 → ragmint-0.2.0}/README.md

@@ -10,17 +10,19 @@
 
 **Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
 
-It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**.
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
 
 ---
 
 ## ✨ Features
 
 - ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
 - 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
 - ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
 - 🧩 **Embeddings** — OpenAI, HuggingFace  
-- 🧠 **Rerankers** — MMR, CrossEncoder (extensible via plugin interface)  
 - 💾 **Caching, experiment tracking, and reproducibility** out of the box  
 - 🧰 **Clean modular structure** for easy integration in research and production setups  
 
@@ -76,6 +78,7 @@ print(result)
 ```
 
 ---
+
 ## 🧪 Dataset Options
 
 Ragmint can automatically load evaluation datasets for your RAG pipeline:
@@ -110,47 +113,99 @@ ragmint.optimize(validation_set="data/custom_qa.json")
 
 ---
 
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
 ## 🧩 Folder Structure
 
 ```
 ragmint/
 ├── core/
-│   ├── pipeline.py         # RAGPipeline implementation
-│   ├── retriever.py        # Retriever logic (FAISS, Chroma)
-│   ├── reranker.py         # MMR + CrossEncoder rerankers
-│   └── embedding.py        # Embedding backends
-├── tuner.py                # Grid, Random, Bayesian optimization (Optuna)
-├── utils/                  # Metrics, logging, caching helpers
-├── configs/                # Default experiment configs
-├── experiments/            # Saved experiment results
-├── tests/                  # Unit tests for all components
-├── main.py                 # CLI entrypoint for tuning
-└── pyproject.toml          # Project dependencies & build metadata
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
 ```
 
 ---
 
 ## 🧪 Running Tests
 
-To verify your setup:
-
 ```bash
 pytest -v
 ```
 
-Or to test a specific component (e.g., reranker):
-
+To include integration tests with Gemini or Claude APIs:
 ```bash
-pytest tests/test_reranker.py -v
+pytest -m integration
 ```
 
-All tests are designed for **Pytest** and run with lightweight mock data.
-
 ---
 
 ## ⚙️ Configuration via `pyproject.toml`
 
-Your `pyproject.toml` automatically includes:
+Your `pyproject.toml` includes all required dependencies:
 
 ```toml
 [project]
@@ -165,6 +220,8 @@ dependencies = [
     "pytest",
     "openai",
     "tqdm",
+    "google-generativeai",
+    "google-genai",
 ]
 ```
 
@@ -172,10 +229,10 @@ dependencies = [
 
 ## 📊 Example Experiment Workflow
 
-1. Define your retriever and reranker configuration in YAML  
-2. Launch an optimization search (Grid, Random, or Bayesian)  
-3. Ragmint evaluates combinations automatically and reports top results  
-4. Export best parameters for production pipelines  
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
 
 ---
 
@@ -188,7 +245,7 @@ flowchart TD
     C --> D[Reranker]
     D --> E[Generator]
     E --> F[Evaluation]
-    F --> G[Optuna Tuner]
+    F --> G[Optuna / AutoRAGTuner]
     G -->|Best Params| B
 ```
 
@@ -198,8 +255,9 @@ flowchart TD
 
 ```
 [INFO] Starting Bayesian optimization with Optuna
-[INFO] Trial 7 finished: recall=0.83, latency=0.42s
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
 [INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
 ```
 
 ---
@@ -207,8 +265,9 @@ flowchart TD
 ## 🧠 Why Ragmint?
 
 - Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
-- Works with **LangChain**, **LlamaIndex**, or standalone RAG setups  
-- Designed for **extensibility** — plug in your own models, retrievers, or metrics  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
 
 ---
 

{ragmint-0.1.1 → ragmint-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ragmint"
-version = "0.1.1"
+version = "0.2.0"
 description = "A modular framework for evaluating and optimizing RAG pipelines."
 readme = "README.md"
 license = { text = "Apache License 2.0" }
@@ -24,7 +24,9 @@ dependencies = [
   "faiss-cpu; sys_platform != 'darwin'",
   "optuna>=3.0",
   "pytest",
-  "colorama"
+  "colorama",
+  "google-generativeai>=0.8.0",
+  "supabase>=2.4.0"
 ]
 
 [project.urls]

ragmint-0.2.0/src/ragmint/autotuner.py

@@ -0,0 +1,33 @@
+"""
+Auto-RAG Tuner
+--------------
+Recommends retriever–embedding pairs dynamically based on corpus size
+and dataset characteristics. Integrates seamlessly with RAGMint evaluator.
+"""
+
+from .core.evaluation import evaluate_config
+
+
+class AutoRAGTuner:
+    def __init__(self, corpus_stats: dict):
+        """
+        corpus_stats: dict
+            Example: {'size': 12000, 'avg_len': 240}
+        """
+        self.corpus_stats = corpus_stats
+
+    def recommend(self):
+        size = self.corpus_stats.get("size", 0)
+        avg_len = self.corpus_stats.get("avg_len", 0)
+
+        if size < 1000:
+            return {"retriever": "BM25", "embedding_model": "OpenAI"}
+        elif size < 10000:
+            return {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+        else:
+            return {"retriever": "FAISS", "embedding_model": "InstructorXL"}
+
+    def auto_tune(self, validation_data):
+        config = self.recommend()
+        results = evaluate_config(config, validation_data)
+        return {"recommended": config, "results": results}

{ragmint-0.1.1 → ragmint-0.2.0}/src/ragmint/core/evaluation.py

@@ -25,3 +25,14 @@ class Evaluator:
 
     def _similarity(self, a: str, b: str) -> float:
         return SequenceMatcher(None, a, b).ratio()
+
+def evaluate_config(config, validation_data):
+    evaluator = Evaluator()
+    results = []
+    for sample in validation_data:
+        query = sample.get("query", "")
+        answer = sample.get("answer", "")
+        context = sample.get("context", "")
+        results.append(evaluator.evaluate(query, answer, context))
+    return results
+

ragmint-0.2.0/src/ragmint/explainer.py

@@ -0,0 +1,61 @@
+"""
+Interpretability Layer
+----------------------
+Uses Gemini or Anthropic Claude to explain why one RAG configuration
+outperforms another. Falls back gracefully if no API key is provided.
+"""
+
+import os
+import json
+
+
+def explain_results(results_a: dict, results_b: dict, model: str = "gemini-1.5-pro") -> str:
+    """
+    Generate a natural-language explanation comparing two RAG experiment results.
+    Priority:
+      1. Anthropic Claude (if ANTHROPIC_API_KEY is set)
+      2. Google Gemini (if GOOGLE_API_KEY is set)
+      3. Fallback text message
+    """
+    prompt = f"""
+    You are an AI evaluation expert.
+    Compare these two RAG experiment results and explain why one performs better.
+    Metrics A: {json.dumps(results_a, indent=2)}
+    Metrics B: {json.dumps(results_b, indent=2)}
+    Provide a concise, human-friendly explanation and practical improvement tips.
+    """
+
+    anthropic_key = os.getenv("ANTHROPIC_API_KEY")
+    google_key = os.getenv("GEMINI_API_KEY")
+
+
+    # 1️⃣ Try Anthropic Claude first
+    if anthropic_key:
+        try:
+            from anthropic import Anthropic
+            client = Anthropic(api_key=anthropic_key)
+            response = client.messages.create(
+                model="claude-3-opus-20240229",
+                max_tokens=300,
+                messages=[{"role": "user", "content": prompt}],
+            )
+            return response.content[0].text
+        except Exception as e:
+            return f"[Claude unavailable] {e}"
+
+    # 2️⃣ Fallback to Google Gemini
+    elif google_key:
+        try:
+            import google.generativeai as genai
+            genai.configure(api_key=google_key)
+            response = genai.GenerativeModel(model).generate_content(prompt)
+            return response.text
+        except Exception as e:
+            return f"[Gemini unavailable] {e}"
+
+    # 3️⃣ Fallback if neither key is available
+    else:
+        return (
+            "[No LLM available] Please set ANTHROPIC_API_KEY or GOOGLE_API_KEY "
+            "to enable interpretability via Claude or Gemini."
+        )

ragmint-0.2.0/src/ragmint/leaderboard.py

@@ -0,0 +1,45 @@
+import os
+import json
+from datetime import datetime
+from typing import Dict, Any, Optional
+from supabase import create_client
+
+class Leaderboard:
+    def __init__(self, storage_path: Optional[str] = None):
+        self.storage_path = storage_path
+        url = os.getenv("SUPABASE_URL")
+        key = os.getenv("SUPABASE_KEY")
+        self.client = None
+        if url and key:
+            self.client = create_client(url, key)
+        elif not storage_path:
+            raise EnvironmentError("Set SUPABASE_URL/SUPABASE_KEY or pass storage_path")
+
+    def upload(self, run_id: str, config: Dict[str, Any], score: float):
+        data = {
+            "run_id": run_id,
+            "config": config,
+            "score": score,
+            "timestamp": datetime.utcnow().isoformat(),
+        }
+        if self.client:
+            return self.client.table("experiments").insert(data).execute()
+        else:
+            os.makedirs(os.path.dirname(self.storage_path), exist_ok=True)
+            with open(self.storage_path, "a", encoding="utf-8") as f:
+                f.write(json.dumps(data) + "\n")
+            return data
+
+    def top_results(self, limit: int = 10):
+        if self.client:
+            return (
+                self.client.table("experiments")
+                .select("*")
+                .order("score", desc=True)
+                .limit(limit)
+                .execute()
+            )
+        else:
+            with open(self.storage_path, "r", encoding="utf-8") as f:
+                lines = [json.loads(line) for line in f]
+            return sorted(lines, key=lambda x: x["score"], reverse=True)[:limit]

ragmint-0.2.0/src/ragmint/tests/conftest.py

@@ -0,0 +1,16 @@
+# src/ragmint/tests/conftest.py
+import os
+from dotenv import load_dotenv
+import pytest
+
+# Load .env from project root
+load_dotenv(dotenv_path=os.path.join(os.path.dirname(__file__), "../../../.env"))
+
+def pytest_configure(config):
+    """Print which keys are loaded (debug)."""
+    google = os.getenv("GEMINI_API_KEY")
+    anthropic = os.getenv("ANTHROPIC_API_KEY")
+    if google:
+        print("✅ GOOGLE_API_KEY loaded")
+    if anthropic:
+        print("✅ ANTHROPIC_API_KEY loaded")

ragmint-0.2.0/src/ragmint/tests/test_autotuner.py

@@ -0,0 +1,42 @@
+import pytest
+from ragmint.autotuner import AutoRAGTuner
+
+
+def test_autorag_recommend_small():
+    """Small corpus should trigger BM25 + OpenAI."""
+    tuner = AutoRAGTuner({"size": 500, "avg_len": 150})
+    rec = tuner.recommend()
+    assert rec["retriever"] == "BM25"
+    assert rec["embedding_model"] == "OpenAI"
+
+
+def test_autorag_recommend_medium():
+    """Medium corpus should trigger Chroma + SentenceTransformers."""
+    tuner = AutoRAGTuner({"size": 5000, "avg_len": 200})
+    rec = tuner.recommend()
+    assert rec["retriever"] == "Chroma"
+    assert rec["embedding_model"] == "SentenceTransformers"
+
+
+def test_autorag_recommend_large():
+    """Large corpus should trigger FAISS + InstructorXL."""
+    tuner = AutoRAGTuner({"size": 50000, "avg_len": 300})
+    rec = tuner.recommend()
+    assert rec["retriever"] == "FAISS"
+    assert rec["embedding_model"] == "InstructorXL"
+
+
+def test_autorag_auto_tune(monkeypatch):
+    """Test auto_tune with a mock validation dataset."""
+    tuner = AutoRAGTuner({"size": 12000, "avg_len": 250})
+
+    # Monkeypatch evaluate_config inside autotuner
+    import ragmint.autotuner as autotuner
+    def mock_eval(config, data):
+        return {"faithfulness": 0.9, "latency": 0.01}
+    monkeypatch.setattr(autotuner, "evaluate_config", mock_eval)
+
+    result = tuner.auto_tune([{"question": "What is AI?", "answer": "Artificial Intelligence"}])
+    assert "recommended" in result
+    assert "results" in result
+    assert isinstance(result["results"], dict)

ragmint-0.2.0/src/ragmint/tests/test_explainer.py

@@ -0,0 +1,20 @@
+import pytest
+from ragmint.explainer import explain_results
+
+
+def test_explain_results_gemini():
+    """Gemini explanation should contain model-specific phrasing."""
+    config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+    config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+    result = explain_results(config_a, config_b, model="gemini")
+    assert isinstance(result, str)
+    assert "Gemini" in result or "gemini" in result
+
+
+def test_explain_results_claude():
+    """Claude explanation should contain model-specific phrasing."""
+    config_a = {"retriever": "FAISS"}
+    config_b = {"retriever": "Chroma"}
+    result = explain_results(config_a, config_b, model="claude")
+    assert isinstance(result, str)
+    assert "Claude" in result or "claude" in result

ragmint-0.2.0/src/ragmint/tests/test_explainer_integration.py

@@ -0,0 +1,18 @@
+import os
+import pytest
+from ragmint.explainer import explain_results
+
+
+@pytest.mark.integration
+def test_real_gemini_explanation():
+    """Run real Gemini call if GOOGLE_API_KEY is set."""
+    if not os.getenv("GEMINI_API_KEY"):
+        pytest.skip("GOOGLE_API_KEY not set")
+
+    config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+    config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+    result = explain_results(config_a, config_b, model="gemini-1.5-pro")
+    assert isinstance(result, str)
+    assert len(result) > 0
+    print("\n[Gemini explanation]:", result[:200], "...")

ragmint-0.2.0/src/ragmint/tests/test_integration_autotuner_ragmint.py

@@ -0,0 +1,60 @@
+import pytest
+from ragmint.tuner import RAGMint
+from ragmint.autotuner import AutoRAGTuner
+
+
+def test_integration_ragmint_autotune(monkeypatch, tmp_path):
+    """
+    Smoke test for integration between AutoRAGTuner and RAGMint.
+    Ensures end-to-end flow runs without real retrievers or embeddings.
+    """
+
+    # --- Mock corpus and validation data ---
+    corpus = tmp_path / "docs"
+    corpus.mkdir()
+    (corpus / "doc1.txt").write_text("This is an AI document.")
+    validation_data = [{"question": "What is AI?", "answer": "Artificial Intelligence"}]
+
+    # --- Mock RAGMint.optimize() to avoid real model work ---
+    def mock_optimize(self, validation_set=None, metric="faithfulness", trials=2):
+        return (
+            {"retriever": "FAISS", "embedding_model": "OpenAI", "score": 0.88},
+            [{"trial": 1, "score": 0.88}],
+        )
+
+    monkeypatch.setattr(RAGMint, "optimize", mock_optimize)
+
+    # --- Mock evaluation used by AutoRAGTuner ---
+    def mock_evaluate_config(config, data):
+        return {"faithfulness": 0.9, "latency": 0.01}
+
+    import ragmint.autotuner as autotuner
+    monkeypatch.setattr(autotuner, "evaluate_config", mock_evaluate_config)
+
+    # --- Create AutoRAGTuner and RAGMint instances ---
+    ragmint = RAGMint(
+        docs_path=str(corpus),
+        retrievers=["faiss", "chroma"],
+        embeddings=["text-embedding-3-small"],
+        rerankers=["mmr"],
+    )
+
+    tuner = AutoRAGTuner({"size": 2000, "avg_len": 150})
+
+    # --- Run Auto-Tune and RAG Optimization ---
+    recommendation = tuner.recommend()
+    assert "retriever" in recommendation
+    assert "embedding_model" in recommendation
+
+    tuning_results = tuner.auto_tune(validation_data)
+    assert "results" in tuning_results
+    assert isinstance(tuning_results["results"], dict)
+
+    # --- Run RAGMint optimization flow (mocked) ---
+    best_config, results = ragmint.optimize(validation_set=validation_data, trials=2)
+    assert isinstance(best_config, dict)
+    assert "score" in best_config
+    assert isinstance(results, list)
+
+    # --- Integration Success ---
+    print(f"Integration OK: AutoRAG recommended {recommendation}, RAGMint best {best_config}")

ragmint-0.2.0/src/ragmint/tests/test_leaderboard.py

@@ -0,0 +1,39 @@
+import json
+import tempfile
+from pathlib import Path
+from ragmint.leaderboard import Leaderboard
+
+
+def test_leaderboard_add_and_top(tmp_path):
+    """Ensure local leaderboard persistence works without Supabase."""
+    file_path = tmp_path / "leaderboard.jsonl"
+    lb = Leaderboard(storage_path=str(file_path))
+
+    # Add two runs
+    lb.upload("run1", {"retriever": "FAISS"}, 0.91)
+    lb.upload("run2", {"retriever": "Chroma"}, 0.85)
+
+    # Verify file content
+    assert file_path.exists()
+    with open(file_path, "r", encoding="utf-8") as f:
+        lines = [json.loads(line) for line in f]
+    assert len(lines) == 2
+
+    # Get top results
+    top = lb.top_results(limit=1)
+    assert isinstance(top, list)
+    assert len(top) == 1
+    assert "score" in top[0]
+
+
+def test_leaderboard_append_existing(tmp_path):
+    """Ensure multiple uploads append properly."""
+    file_path = tmp_path / "leaderboard.jsonl"
+    lb = Leaderboard(storage_path=str(file_path))
+
+    for i in range(3):
+        lb.upload(f"run{i}", {"retriever": "BM25"}, 0.8 + i * 0.05)
+
+    top = lb.top_results(limit=2)
+    assert len(top) == 2
+    assert top[0]["score"] >= top[1]["score"]

{ragmint-0.1.1 → ragmint-0.2.0/src/ragmint.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragmint
-Version: 0.1.1
+Version: 0.2.0
 Summary: A modular framework for evaluating and optimizing RAG pipelines.
 Author-email: Andre Oliveira <oandreoliveira@outlook.com>
 License: Apache License 2.0
@@ -22,6 +22,8 @@ Requires-Dist: faiss-cpu; sys_platform != "darwin"
 Requires-Dist: optuna>=3.0
 Requires-Dist: pytest
 Requires-Dist: colorama
+Requires-Dist: google-generativeai>=0.8.0
+Requires-Dist: supabase>=2.4.0
 Dynamic: license-file
 
 # Ragmint
@@ -36,17 +38,19 @@ Dynamic: license-file
 
 **Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
 
-It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**.
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
 
 ---
 
 ## ✨ Features
 
 - ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
 - 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
 - ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
 - 🧩 **Embeddings** — OpenAI, HuggingFace  
-- 🧠 **Rerankers** — MMR, CrossEncoder (extensible via plugin interface)  
 - 💾 **Caching, experiment tracking, and reproducibility** out of the box  
 - 🧰 **Clean modular structure** for easy integration in research and production setups  
 
@@ -102,6 +106,7 @@ print(result)
 ```
 
 ---
+
 ## 🧪 Dataset Options
 
 Ragmint can automatically load evaluation datasets for your RAG pipeline:
@@ -136,47 +141,99 @@ ragmint.optimize(validation_set="data/custom_qa.json")
 
 ---
 
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
 ## 🧩 Folder Structure
 
 ```
 ragmint/
 ├── core/
-│   ├── pipeline.py         # RAGPipeline implementation
-│   ├── retriever.py        # Retriever logic (FAISS, Chroma)
-│   ├── reranker.py         # MMR + CrossEncoder rerankers
-│   └── embedding.py        # Embedding backends
-├── tuner.py                # Grid, Random, Bayesian optimization (Optuna)
-├── utils/                  # Metrics, logging, caching helpers
-├── configs/                # Default experiment configs
-├── experiments/            # Saved experiment results
-├── tests/                  # Unit tests for all components
-├── main.py                 # CLI entrypoint for tuning
-└── pyproject.toml          # Project dependencies & build metadata
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
 ```
 
 ---
 
 ## 🧪 Running Tests
 
-To verify your setup:
-
 ```bash
 pytest -v
 ```
 
-Or to test a specific component (e.g., reranker):
-
+To include integration tests with Gemini or Claude APIs:
 ```bash
-pytest tests/test_reranker.py -v
+pytest -m integration
 ```
 
-All tests are designed for **Pytest** and run with lightweight mock data.
-
 ---
 
 ## ⚙️ Configuration via `pyproject.toml`
 
-Your `pyproject.toml` automatically includes:
+Your `pyproject.toml` includes all required dependencies:
 
 ```toml
 [project]
@@ -191,6 +248,8 @@ dependencies = [
     "pytest",
     "openai",
     "tqdm",
+    "google-generativeai",
+    "google-genai",
 ]
 ```
 
@@ -198,10 +257,10 @@ dependencies = [
 
 ## 📊 Example Experiment Workflow
 
-1. Define your retriever and reranker configuration in YAML  
-2. Launch an optimization search (Grid, Random, or Bayesian)  
-3. Ragmint evaluates combinations automatically and reports top results  
-4. Export best parameters for production pipelines  
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
 
 ---
 
@@ -214,7 +273,7 @@ flowchart TD
     C --> D[Reranker]
     D --> E[Generator]
     E --> F[Evaluation]
-    F --> G[Optuna Tuner]
+    F --> G[Optuna / AutoRAGTuner]
     G -->|Best Params| B
 ```
 
@@ -224,8 +283,9 @@ flowchart TD
 
 ```
 [INFO] Starting Bayesian optimization with Optuna
-[INFO] Trial 7 finished: recall=0.83, latency=0.42s
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
 [INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
 ```
 
 ---
@@ -233,8 +293,9 @@ flowchart TD
 ## 🧠 Why Ragmint?
 
 - Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
-- Works with **LangChain**, **LlamaIndex**, or standalone RAG setups  
-- Designed for **extensibility** — plug in your own models, retrievers, or metrics  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
 
 ---
 

{ragmint-0.1.1 → ragmint-0.2.0}/src/ragmint.egg-info/SOURCES.txt

@@ -3,6 +3,9 @@ README.md
 pyproject.toml
 src/ragmint/__init__.py
 src/ragmint/__main__.py
+src/ragmint/autotuner.py
+src/ragmint/explainer.py
+src/ragmint/leaderboard.py
 src/ragmint/tuner.py
 src/ragmint.egg-info/PKG-INFO
 src/ragmint.egg-info/SOURCES.txt
@@ -20,6 +23,12 @@ src/ragmint/experiments/__init__.py
 src/ragmint/optimization/__init__.py
 src/ragmint/optimization/search.py
 src/ragmint/tests/__init__.py
+src/ragmint/tests/conftest.py
+src/ragmint/tests/test_autotuner.py
+src/ragmint/tests/test_explainer.py
+src/ragmint/tests/test_explainer_integration.py
+src/ragmint/tests/test_integration_autotuner_ragmint.py
+src/ragmint/tests/test_leaderboard.py
 src/ragmint/tests/test_pipeline.py
 src/ragmint/tests/test_retriever.py
 src/ragmint/tests/test_search.py

{ragmint-0.1.1 → ragmint-0.2.0}/src/ragmint.egg-info/requires.txt

@@ -8,6 +8,8 @@ chromadb>=0.4
 optuna>=3.0
 pytest
 colorama
+google-generativeai>=0.8.0
+supabase>=2.4.0
 
 [:sys_platform != "darwin"]
 faiss-cpu