ragmint 0.2.1__py3-none-any.whl → 0.2.3__py3-none-any.whl

ragmint-0.2.3.data/data/README.md

@@ -0,0 +1,284 @@
+# Ragmint
+
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+![License](https://img.shields.io/badge/license-Apache%202.0-green)
+![Tests](https://github.com/andyolivers/ragmint/actions/workflows/tests.yml/badge.svg)
+![Optuna](https://img.shields.io/badge/Optuna-Integrated-orange)
+![Status](https://img.shields.io/badge/Status-Active-success)
+
+![](/assets/images/ragmint-banner.png)
+
+**Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
+
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
+
+---
+
+## ✨ Features
+
+- ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
+- 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
+- ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
+- 🧩 **Embeddings** — OpenAI, HuggingFace  
+- 💾 **Caching, experiment tracking, and reproducibility** out of the box  
+- 🧰 **Clean modular structure** for easy integration in research and production setups  
+
+---
+
+## 🚀 Quick Start
+
+### 1️⃣ Installation
+
+```bash
+git clone https://github.com/andyolivers/ragmint.git
+cd ragmint
+pip install -e .
+```
+
+> The `-e` flag installs Ragmint in editable (development) mode.  
+> Requires **Python ≥ 3.9**.
+
+---
+
+### 2️⃣ Run a RAG Optimization Experiment
+
+```bash
+python ragmint/main.py --config configs/default.yaml --search bayesian
+```
+
+Example `configs/default.yaml`:
+```yaml
+retriever: faiss
+embedding_model: text-embedding-3-small
+reranker:
+  mode: mmr
+  lambda_param: 0.5
+optimization:
+  search_method: bayesian
+  n_trials: 20
+```
+
+---
+
+### 3️⃣ Manual Pipeline Usage
+
+```python
+from ragmint.core.pipeline import RAGPipeline
+
+pipeline = RAGPipeline({
+    "embedding_model": "text-embedding-3-small",
+    "retriever": "faiss",
+})
+
+result = pipeline.run("What is retrieval-augmented generation?")
+print(result)
+```
+
+---
+
+## 🧪 Dataset Options
+
+Ragmint can automatically load evaluation datasets for your RAG pipeline:
+
+| Mode | Example | Description |
+|------|----------|-------------|
+| 🧱 **Default** | `validation_set=None` | Uses built-in `experiments/validation_qa.json` |
+| 📁 **Custom File** | `validation_set="data/my_eval.json"` | Load your own QA dataset (JSON or CSV) |
+| 🌐 **Hugging Face Dataset** | `validation_set="squad"` | Automatically downloads benchmark datasets (requires `pip install datasets`) |
+
+### Example
+
+```python
+from ragmint.tuner import RAGMint
+
+ragmint = RAGMint(
+    docs_path="data/docs/",
+    retrievers=["faiss", "chroma"],
+    embeddings=["text-embedding-3-small"],
+    rerankers=["mmr"],
+)
+
+# Use built-in default
+ragmint.optimize(validation_set=None)
+
+# Use Hugging Face benchmark
+ragmint.optimize(validation_set="squad")
+
+# Use your own dataset
+ragmint.optimize(validation_set="data/custom_qa.json")
+```
+
+---
+
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
+## 🧩 Folder Structure
+
+```
+ragmint/
+├── core/
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
+```
+
+---
+
+## 🧪 Running Tests
+
+```bash
+pytest -v
+```
+
+To include integration tests with Gemini or Claude APIs:
+```bash
+pytest -m integration
+```
+
+---
+
+## ⚙️ Configuration via `pyproject.toml`
+
+Your `pyproject.toml` includes all required dependencies:
+
+```toml
+[project]
+name = "ragmint"
+version = "0.1.0"
+dependencies = [
+    "numpy",
+    "optuna",
+    "scikit-learn",
+    "faiss-cpu",
+    "chromadb",
+    "pytest",
+    "openai",
+    "tqdm",
+    "google-generativeai",
+    "google-genai",
+]
+```
+
+---
+
+## 📊 Example Experiment Workflow
+
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
+
+---
+
+## 🧬 Architecture Overview
+
+```mermaid
+flowchart TD
+    A[Query] --> B[Embedder]
+    B --> C[Retriever]
+    C --> D[Reranker]
+    D --> E[Generator]
+    E --> F[Evaluation]
+    F --> G[Optuna / AutoRAGTuner]
+    G -->|Best Params| B
+```
+
+---
+
+## 📘 Example Output
+
+```
+[INFO] Starting Bayesian optimization with Optuna
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
+[INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
+```
+
+---
+
+## 🧠 Why Ragmint?
+
+- Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
+
+---
+
+## ⚖️ License
+
+Licensed under the **Apache License 2.0** — free for personal, research, and commercial use.
+
+---
+
+## 👤 Author
+
+**André Oliveira**  
+[andyolivers.com](https://andyolivers.com)  
+Data Scientist | AI Engineer

ragmint-0.2.3.dist-info/METADATA

@@ -0,0 +1,312 @@
+Metadata-Version: 2.4
+Name: ragmint
+Version: 0.2.3
+Summary: A modular framework for evaluating and optimizing RAG pipelines.
+Author-email: Andre Oliveira <oandreoliveira@outlook.com>
+License: Apache License 2.0
+Project-URL: Homepage, https://github.com/andyolivers/ragmint
+Project-URL: Documentation, https://andyolivers.com
+Project-URL: Issues, https://github.com/andyolivers/ragmint/issues
+Keywords: RAG,LLM,retrieval,optimization,AI,evaluation
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=2.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: openai>=1.0
+Requires-Dist: tqdm
+Requires-Dist: pyyaml
+Requires-Dist: chromadb>=0.4
+Requires-Dist: faiss-cpu; sys_platform != "darwin"
+Requires-Dist: optuna>=3.0
+Requires-Dist: pytest
+Requires-Dist: colorama
+Requires-Dist: google-generativeai>=0.8.0
+Requires-Dist: supabase>=2.4.0
+Dynamic: license-file
+
+# Ragmint
+
+![Python](https://img.shields.io/badge/python-3.9%2B-blue)
+![License](https://img.shields.io/badge/license-Apache%202.0-green)
+![Tests](https://github.com/andyolivers/ragmint/actions/workflows/tests.yml/badge.svg)
+![Optuna](https://img.shields.io/badge/Optuna-Integrated-orange)
+![Status](https://img.shields.io/badge/Status-Active-success)
+
+![](/assets/images/ragmint-banner.png)
+
+**Ragmint** (Retrieval-Augmented Generation Model Inspection & Tuning) is a modular, developer-friendly Python library for **evaluating, optimizing, and tuning RAG (Retrieval-Augmented Generation) pipelines**.
+
+It provides a complete toolkit for **retriever selection**, **embedding model tuning**, and **automated RAG evaluation** with support for **Optuna-based Bayesian optimization**, **Auto-RAG tuning**, and **explainability** through Gemini or Claude.
+
+---
+
+## ✨ Features
+
+- ✅ **Automated hyperparameter optimization** (Grid, Random, Bayesian via Optuna)  
+- 🤖 **Auto-RAG Tuner** — dynamically recommends retriever–embedding pairs based on corpus size  
+- 🧠 **Explainability Layer** — interprets RAG performance via Gemini or Claude APIs  
+- 🏆 **Leaderboard Tracking** — stores and ranks experiment runs via JSON or external DB  
+- 🔍 **Built-in RAG evaluation metrics** — faithfulness, recall, BLEU, ROUGE, latency  
+- ⚙️ **Retrievers** — FAISS, Chroma, ElasticSearch  
+- 🧩 **Embeddings** — OpenAI, HuggingFace  
+- 💾 **Caching, experiment tracking, and reproducibility** out of the box  
+- 🧰 **Clean modular structure** for easy integration in research and production setups  
+
+---
+
+## 🚀 Quick Start
+
+### 1️⃣ Installation
+
+```bash
+git clone https://github.com/andyolivers/ragmint.git
+cd ragmint
+pip install -e .
+```
+
+> The `-e` flag installs Ragmint in editable (development) mode.  
+> Requires **Python ≥ 3.9**.
+
+---
+
+### 2️⃣ Run a RAG Optimization Experiment
+
+```bash
+python ragmint/main.py --config configs/default.yaml --search bayesian
+```
+
+Example `configs/default.yaml`:
+```yaml
+retriever: faiss
+embedding_model: text-embedding-3-small
+reranker:
+  mode: mmr
+  lambda_param: 0.5
+optimization:
+  search_method: bayesian
+  n_trials: 20
+```
+
+---
+
+### 3️⃣ Manual Pipeline Usage
+
+```python
+from ragmint.core.pipeline import RAGPipeline
+
+pipeline = RAGPipeline({
+    "embedding_model": "text-embedding-3-small",
+    "retriever": "faiss",
+})
+
+result = pipeline.run("What is retrieval-augmented generation?")
+print(result)
+```
+
+---
+
+## 🧪 Dataset Options
+
+Ragmint can automatically load evaluation datasets for your RAG pipeline:
+
+| Mode | Example | Description |
+|------|----------|-------------|
+| 🧱 **Default** | `validation_set=None` | Uses built-in `experiments/validation_qa.json` |
+| 📁 **Custom File** | `validation_set="data/my_eval.json"` | Load your own QA dataset (JSON or CSV) |
+| 🌐 **Hugging Face Dataset** | `validation_set="squad"` | Automatically downloads benchmark datasets (requires `pip install datasets`) |
+
+### Example
+
+```python
+from ragmint.tuner import RAGMint
+
+ragmint = RAGMint(
+    docs_path="data/docs/",
+    retrievers=["faiss", "chroma"],
+    embeddings=["text-embedding-3-small"],
+    rerankers=["mmr"],
+)
+
+# Use built-in default
+ragmint.optimize(validation_set=None)
+
+# Use Hugging Face benchmark
+ragmint.optimize(validation_set="squad")
+
+# Use your own dataset
+ragmint.optimize(validation_set="data/custom_qa.json")
+```
+
+---
+
+## 🧠 Auto-RAG Tuner
+
+The **AutoRAGTuner** automatically recommends retriever–embedding combinations
+based on corpus size and average document length.
+
+```python
+from ragmint.autotuner import AutoRAGTuner
+
+corpus_stats = {"size": 5000, "avg_len": 250}
+tuner = AutoRAGTuner(corpus_stats)
+recommendation = tuner.recommend()
+print(recommendation)
+# Example output: {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+```
+
+---
+
+## 🏆 Leaderboard Tracking
+
+Track and visualize your best experiments across runs.
+
+```python
+from ragmint.leaderboard import Leaderboard
+
+lb = Leaderboard("experiments/leaderboard.json")
+lb.add_entry({"trial": 1, "faithfulness": 0.87, "latency": 0.12})
+lb.show_top(3)
+```
+
+---
+
+## 🧠 Explainability with Gemini / Claude
+
+Compare two RAG configurations and receive natural language insights
+on **why** one performs better.
+
+```python
+from ragmint.explainer import explain_results
+
+config_a = {"retriever": "FAISS", "embedding_model": "OpenAI"}
+config_b = {"retriever": "Chroma", "embedding_model": "SentenceTransformers"}
+
+explanation = explain_results(config_a, config_b, model="gemini")
+print(explanation)
+```
+
+> Set your API keys in a `.env` file or via environment variables:
+> ```
+> export GOOGLE_API_KEY="your_gemini_key"
+> export ANTHROPIC_API_KEY="your_claude_key"
+> ```
+
+---
+
+## 🧩 Folder Structure
+
+```
+ragmint/
+├── core/
+│   ├── pipeline.py
+│   ├── retriever.py
+│   ├── reranker.py
+│   ├── embedding.py
+│   └── evaluation.py
+├── autotuner.py
+├── explainer.py
+├── leaderboard.py
+├── tuner.py
+├── utils/
+├── configs/
+├── experiments/
+├── tests/
+└── main.py
+```
+
+---
+
+## 🧪 Running Tests
+
+```bash
+pytest -v
+```
+
+To include integration tests with Gemini or Claude APIs:
+```bash
+pytest -m integration
+```
+
+---
+
+## ⚙️ Configuration via `pyproject.toml`
+
+Your `pyproject.toml` includes all required dependencies:
+
+```toml
+[project]
+name = "ragmint"
+version = "0.1.0"
+dependencies = [
+    "numpy",
+    "optuna",
+    "scikit-learn",
+    "faiss-cpu",
+    "chromadb",
+    "pytest",
+    "openai",
+    "tqdm",
+    "google-generativeai",
+    "google-genai",
+]
+```
+
+---
+
+## 📊 Example Experiment Workflow
+
+1. Define your retriever, embedding, and reranker setup  
+2. Launch optimization (Grid, Random, Bayesian) or AutoTune  
+3. Compare performance with explainability  
+4. Persist results to leaderboard for later inspection  
+
+---
+
+## 🧬 Architecture Overview
+
+```mermaid
+flowchart TD
+    A[Query] --> B[Embedder]
+    B --> C[Retriever]
+    C --> D[Reranker]
+    D --> E[Generator]
+    E --> F[Evaluation]
+    F --> G[Optuna / AutoRAGTuner]
+    G -->|Best Params| B
+```
+
+---
+
+## 📘 Example Output
+
+```
+[INFO] Starting Bayesian optimization with Optuna
+[INFO] Trial 7 finished: faithfulness=0.83, latency=0.42s
+[INFO] Best parameters: {'lambda_param': 0.6, 'retriever': 'faiss'}
+[INFO] AutoRAGTuner: Suggested retriever=Chroma for medium corpus
+```
+
+---
+
+## 🧠 Why Ragmint?
+
+- Built for **RAG researchers**, **AI engineers**, and **LLM ops**  
+- Works with **LangChain**, **LlamaIndex**, or standalone setups  
+- Designed for **extensibility** — plug in your own retrievers, models, or metrics  
+- Integrated **explainability and leaderboard** modules for research and production  
+
+---
+
+## ⚖️ License
+
+Licensed under the **Apache License 2.0** — free for personal, research, and commercial use.
+
+---
+
+## 👤 Author
+
+**André Oliveira**  
+[andyolivers.com](https://andyolivers.com)  
+Data Scientist | AI Engineer

{ragmint-0.2.1.dist-info → ragmint-0.2.3.dist-info}/RECORD

@@ -31,8 +31,10 @@ ragmint/utils/caching.py,sha256=LPE2JorOQ90BgVf6NUiS0-bdt-FGpNxDy7FnuwEHzy0,1060
 ragmint/utils/data_loader.py,sha256=GXU9Nc3o0UWxtBeRwiskD1aCjSiNNuRoAokIUODn7q8,2024
 ragmint/utils/logger.py,sha256=X7hTNb3st3fUeQIzSghuoV5B8FWXzm_O3DRkSfJvhmI,1033
 ragmint/utils/metrics.py,sha256=DR8mrdumHtQerK0VrugwYKIG1oNptEcsFqodXq3i2kY,717
-ragmint-0.2.1.dist-info/licenses/LICENSE,sha256=ahkhYfFLI8tGrdxdO2_GaT6OJW2eNwyFT3kYi85QQhc,692
-ragmint-0.2.1.dist-info/METADATA,sha256=sR31_9qGCaarWLNqTc2uNjO4aklKq9nxaGEjtcy5ipU,936
-ragmint-0.2.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-ragmint-0.2.1.dist-info/top_level.txt,sha256=K2ulzMHuvFm6xayvvJdGABeRJAvKDBn6M3EI-3SbYLw,8
-ragmint-0.2.1.dist-info/RECORD,,
+ragmint-0.2.3.data/data/LICENSE,sha256=ahkhYfFLI8tGrdxdO2_GaT6OJW2eNwyFT3kYi85QQhc,692
+ragmint-0.2.3.data/data/README.md,sha256=4rubaB58cFz_LhuWk1JUVBAyQAYbTBwXX8eAZFBocoQ,7010
+ragmint-0.2.3.dist-info/licenses/LICENSE,sha256=ahkhYfFLI8tGrdxdO2_GaT6OJW2eNwyFT3kYi85QQhc,692
+ragmint-0.2.3.dist-info/METADATA,sha256=g1ES-pCYCZJUNY01_1jhEkzpW8hqrZ3gwF9kKGPtgec,7948
+ragmint-0.2.3.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+ragmint-0.2.3.dist-info/top_level.txt,sha256=K2ulzMHuvFm6xayvvJdGABeRJAvKDBn6M3EI-3SbYLw,8
+ragmint-0.2.3.dist-info/RECORD,,

ragmint-0.2.3.dist-info/licenses/LICENSE

@@ -0,0 +1,19 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+Copyright 2025 André Oliveira
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

ragmint-0.2.1.dist-info/METADATA

@@ -1,27 +0,0 @@
-Metadata-Version: 2.4
-Name: ragmint
-Version: 0.2.1
-Summary: A modular framework for evaluating and optimizing RAG pipelines.
-Author-email: Andre Oliveira <oandreoliveira@outlook.com>
-License: Apache License 2.0
-Project-URL: Homepage, https://github.com/andyolivers/ragmint
-Project-URL: Documentation, https://andyolivers.com
-Project-URL: Issues, https://github.com/andyolivers/ragmint/issues
-Keywords: RAG,LLM,retrieval,optimization,AI,evaluation
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy>=1.23
-Requires-Dist: pandas>=2.0
-Requires-Dist: scikit-learn>=1.3
-Requires-Dist: openai>=1.0
-Requires-Dist: tqdm
-Requires-Dist: pyyaml
-Requires-Dist: chromadb>=0.4
-Requires-Dist: faiss-cpu; sys_platform != "darwin"
-Requires-Dist: optuna>=3.0
-Requires-Dist: pytest
-Requires-Dist: colorama
-Requires-Dist: google-generativeai>=0.8.0
-Requires-Dist: supabase>=2.4.0
-Dynamic: license-file