rag-sentinel 0.1.0__tar.gz

rag_sentinel-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 RAGSentinel Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

rag_sentinel-0.1.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md
+include LICENSE
+recursive-include src/rag_sentinel/templates *
+

rag_sentinel-0.1.0/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: rag-sentinel
+Version: 0.1.0
+Summary: RAG Evaluation Framework using Ragas metrics and MLflow tracking
+Author: RAGSentinel Team
+License: MIT
+Project-URL: Homepage, https://github.com/yourusername/rag-sentinel
+Project-URL: Repository, https://github.com/yourusername/rag-sentinel
+Keywords: rag,evaluation,ragas,mlflow,llm,ai
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ragas>=0.2.0
+Requires-Dist: mlflow>=2.9.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: langchain-openai>=0.0.5
+Requires-Dist: langchain-ollama>=0.0.1
+Requires-Dist: langchain-core>=0.1.0
+Requires-Dist: datasets>=2.14.0
+Dynamic: license-file
+
+# RAGSentinel
+
+RAG Evaluation Framework using Ragas metrics and MLflow tracking.
+
+## Installation
+
+```bash
+pip install rag-sentinel
+```
+
+## Quick Start
+
+### 1. Initialize Project
+
+```bash
+mkdir my-rag-eval
+cd my-rag-eval
+rag-sentinel init
+```
+
+This creates:
+- `.env` - LLM/Embeddings API keys
+- `config.ini` - App settings and authentication
+- `rag_eval_config.yaml` - Master configuration
+
+### 2. Configure
+
+Edit `.env`:
+```bash
+LLM_PROVIDER=azure
+AZURE_LLM_API_KEY=your-api-key
+AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com/
+```
+
+Edit `config.ini`:
+```ini
+[app]
+app_url = https://your-rag-app.com/backend
+
+[auth]
+type = cookie
+cookie_name = session
+cookie_value = your-session-cookie
+```
+
+### 3. Create Test Dataset
+
+Create `test_dataset.csv`:
+```csv
+query,ground_truth,chat_id
+Hello,Hello! How can I help you?,1
+```
+
+### 4. Run Evaluation
+
+```bash
+rag-sentinel run
+```
+
+View results at: http://127.0.0.1:5001
+
+## CLI Commands
+
+```bash
+rag-sentinel init              # Initialize project
+rag-sentinel run               # Run evaluation (auto-starts MLflow)
+rag-sentinel run --no-server   # Run without starting MLflow
+rag-sentinel validate          # Validate configuration
+```
+
+## Metrics
+
+- **Faithfulness** - Factual consistency of answer with context
+- **Answer Relevancy** - How relevant the answer is to the question
+- **Context Precision** - Quality of retrieved context
+- **Answer Correctness** - Comparison against ground truth
+
+## License
+
+MIT
+

rag_sentinel-0.1.0/README.md

@@ -0,0 +1,81 @@
+# RAGSentinel
+
+RAG Evaluation Framework using Ragas metrics and MLflow tracking.
+
+## Installation
+
+```bash
+pip install rag-sentinel
+```
+
+## Quick Start
+
+### 1. Initialize Project
+
+```bash
+mkdir my-rag-eval
+cd my-rag-eval
+rag-sentinel init
+```
+
+This creates:
+- `.env` - LLM/Embeddings API keys
+- `config.ini` - App settings and authentication
+- `rag_eval_config.yaml` - Master configuration
+
+### 2. Configure
+
+Edit `.env`:
+```bash
+LLM_PROVIDER=azure
+AZURE_LLM_API_KEY=your-api-key
+AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com/
+```
+
+Edit `config.ini`:
+```ini
+[app]
+app_url = https://your-rag-app.com/backend
+
+[auth]
+type = cookie
+cookie_name = session
+cookie_value = your-session-cookie
+```
+
+### 3. Create Test Dataset
+
+Create `test_dataset.csv`:
+```csv
+query,ground_truth,chat_id
+Hello,Hello! How can I help you?,1
+```
+
+### 4. Run Evaluation
+
+```bash
+rag-sentinel run
+```
+
+View results at: http://127.0.0.1:5001
+
+## CLI Commands
+
+```bash
+rag-sentinel init              # Initialize project
+rag-sentinel run               # Run evaluation (auto-starts MLflow)
+rag-sentinel run --no-server   # Run without starting MLflow
+rag-sentinel validate          # Validate configuration
+```
+
+## Metrics
+
+- **Faithfulness** - Factual consistency of answer with context
+- **Answer Relevancy** - How relevant the answer is to the question
+- **Context Precision** - Quality of retrieved context
+- **Answer Correctness** - Comparison against ground truth
+
+## License
+
+MIT
+

rag_sentinel-0.1.0/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rag-sentinel"
+version = "0.1.0"
+description = "RAG Evaluation Framework using Ragas metrics and MLflow tracking"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "RAGSentinel Team"}
+]
+keywords = ["rag", "evaluation", "ragas", "mlflow", "llm", "ai"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "ragas>=0.2.0",
+    "mlflow>=2.9.0",
+    "pandas>=2.0.0",
+    "pyyaml>=6.0",
+    "requests>=2.31.0",
+    "python-dotenv>=1.0.0",
+    "langchain-openai>=0.0.5",
+    "langchain-ollama>=0.0.1",
+    "langchain-core>=0.1.0",
+    "datasets>=2.14.0",
+]
+
+[project.scripts]
+rag-sentinel = "rag_sentinel.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/yourusername/rag-sentinel"
+Repository = "https://github.com/yourusername/rag-sentinel"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+rag_sentinel = ["templates/*"]
+

rag_sentinel-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

rag_sentinel-0.1.0/src/rag_sentinel/__init__.py

@@ -0,0 +1,6 @@
+"""
+RAGSentinel - RAG Evaluation Framework
+"""
+
+__version__ = "0.1.0"
+

rag_sentinel-0.1.0/src/rag_sentinel/cli.py

@@ -0,0 +1,174 @@
+"""
+RAGSentinel CLI - Command Line Interface
+"""
+
+import os
+import sys
+import shutil
+import socket
+import subprocess
+import time
+import argparse
+from pathlib import Path
+
+# Get the templates directory path
+TEMPLATES_DIR = Path(__file__).parent / "templates"
+
+
+def is_port_in_use(host, port):
+    """Check if a port is in use."""
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        return s.connect_ex((host, port)) == 0
+
+
+def start_mlflow_server(host="127.0.0.1", port=5001):
+    """Start MLflow server as a background process."""
+    if is_port_in_use(host, port):
+        print(f"✓ MLflow server already running at http://{host}:{port}")
+        return None
+    
+    print(f"🚀 Starting MLflow server at http://{host}:{port}...")
+    
+    process = subprocess.Popen(
+        [sys.executable, "-m", "mlflow", "server", "--host", host, "--port", str(port)],
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+        creationflags=subprocess.CREATE_NEW_PROCESS_GROUP if sys.platform == "win32" else 0
+    )
+    
+    # Wait for server to start
+    for _ in range(10):
+        time.sleep(1)
+        if is_port_in_use(host, port):
+            print(f"✓ MLflow server started at http://{host}:{port}")
+            return process
+    
+    print("⚠ MLflow server may not have started properly")
+    return process
+
+
+def cmd_init(args):
+    """Initialize project with config templates."""
+    print("=" * 50)
+    print("RAGSentinel - Project Initialization")
+    print("=" * 50)
+    
+    files_to_copy = [
+        (".env.template", ".env"),
+        ("config.ini.template", "config.ini"),
+        ("rag_eval_config.yaml", "rag_eval_config.yaml"),
+    ]
+    
+    for src_name, dest_name in files_to_copy:
+        src_path = TEMPLATES_DIR / src_name
+        dest_path = Path.cwd() / dest_name
+        
+        if dest_path.exists() and not args.force:
+            print(f"⚠ {dest_name} already exists (use --force to overwrite)")
+        else:
+            if src_path.exists():
+                shutil.copy(src_path, dest_path)
+                print(f"✓ Created {dest_name}")
+            else:
+                print(f"✗ Template not found: {src_name}")
+    
+    print("\n" + "=" * 50)
+    print("Next steps:")
+    print("  1. Edit .env with your LLM/Embeddings API keys")
+    print("  2. Edit config.ini with your app settings")
+    print("  3. Create test_dataset.csv with your test data")
+    print("  4. Run: rag-sentinel run")
+    print("=" * 50)
+
+
+def cmd_run(args):
+    """Run the RAG evaluation."""
+    # Check if config files exist
+    required_files = [".env", "config.ini", "rag_eval_config.yaml"]
+    missing_files = [f for f in required_files if not Path(f).exists()]
+    
+    if missing_files:
+        print("❌ Missing configuration files:")
+        for f in missing_files:
+            print(f"   - {f}")
+        print("\nRun 'rag-sentinel init' first to create config files.")
+        sys.exit(1)
+    
+    # Start MLflow server if not running
+    if not args.no_server:
+        start_mlflow_server()
+    
+    # Import and run evaluation
+    from rag_sentinel.evaluator import run_evaluation
+    run_evaluation()
+
+
+def cmd_validate(args):
+    """Validate configuration files."""
+    print("=" * 50)
+    print("RAGSentinel - Configuration Validation")
+    print("=" * 50)
+    
+    # Check files exist
+    files_to_check = [".env", "config.ini", "rag_eval_config.yaml"]
+    all_exist = True
+    
+    for f in files_to_check:
+        if Path(f).exists():
+            print(f"✓ {f} exists")
+        else:
+            print(f"✗ {f} missing")
+            all_exist = False
+    
+    if not all_exist:
+        print("\n❌ Some files are missing. Run 'rag-sentinel init' first.")
+        return
+    
+    # Try to load config
+    try:
+        from rag_sentinel.evaluator import load_config
+        config = load_config()
+        print("\n✓ Configuration loaded successfully")
+        print(f"  - LLM Provider: {config['ragas']['llm']['provider']}")
+        print(f"  - Embeddings Provider: {config['ragas']['embeddings']['provider']}")
+        print(f"  - Backend URL: {config['backend']['base_url']}")
+        print(f"  - Dataset: {config['dataset']['path']}")
+    except Exception as e:
+        print(f"\n❌ Configuration error: {e}")
+
+
+def main():
+    """Main entry point for CLI."""
+    parser = argparse.ArgumentParser(
+        prog="rag-sentinel",
+        description="RAGSentinel - RAG Evaluation Framework using Ragas and MLflow"
+    )
+    
+    subparsers = parser.add_subparsers(dest="command", help="Available commands")
+    
+    # init command
+    init_parser = subparsers.add_parser("init", help="Initialize project with config templates")
+    init_parser.add_argument("--force", "-f", action="store_true", help="Overwrite existing files")
+    
+    # run command
+    run_parser = subparsers.add_parser("run", help="Run RAG evaluation")
+    run_parser.add_argument("--no-server", action="store_true", help="Don't start MLflow server")
+    
+    # validate command
+    validate_parser = subparsers.add_parser("validate", help="Validate configuration files")
+    
+    args = parser.parse_args()
+    
+    if args.command == "init":
+        cmd_init(args)
+    elif args.command == "run":
+        cmd_run(args)
+    elif args.command == "validate":
+        cmd_validate(args)
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()
+

rag_sentinel-0.1.0/src/rag_sentinel/evaluator.py

@@ -0,0 +1,341 @@
+"""
+RAGSentinel Evaluator - Core evaluation logic
+"""
+
+import os
+import re
+import yaml
+import configparser
+import requests
+import pandas as pd
+import mlflow
+from dotenv import load_dotenv
+from datasets import Dataset
+from ragas import evaluate, RunConfig
+from ragas.metrics import faithfulness, answer_relevancy, context_precision, answer_correctness
+
+
+def resolve_placeholder(value, env_vars, ini_config):
+    """Resolve ${ENV:...} and ${INI:...} placeholders."""
+    if not isinstance(value, str):
+        return value
+
+    # Resolve ${ENV:VAR_NAME}
+    env_pattern = r'\$\{ENV:([^}]+)\}'
+    def env_replacer(match):
+        var_name = match.group(1)
+        return env_vars.get(var_name, '')
+    value = re.sub(env_pattern, env_replacer, value)
+
+    # Resolve ${INI:section.key}
+    ini_pattern = r'\$\{INI:([^}]+)\}'
+    def ini_replacer(match):
+        path = match.group(1)
+        parts = path.split('.')
+        if len(parts) == 2:
+            section, key = parts
+            if ini_config.has_option(section, key):
+                return ini_config.get(section, key)
+        return ''
+    value = re.sub(ini_pattern, ini_replacer, value)
+
+    return value
+
+
+def resolve_config(obj, env_vars, ini_config):
+    """Recursively resolve all placeholders in config."""
+    if isinstance(obj, dict):
+        return {k: resolve_config(v, env_vars, ini_config) for k, v in obj.items()}
+    elif isinstance(obj, list):
+        return [resolve_config(item, env_vars, ini_config) for item in obj]
+    elif isinstance(obj, str):
+        return resolve_placeholder(obj, env_vars, ini_config)
+    return obj
+
+
+def load_config():
+    """Load configuration from .env, config.ini, and rag_eval_config.yaml."""
+    load_dotenv('.env')
+    env_vars = dict(os.environ)
+
+    ini_config = configparser.ConfigParser()
+    ini_config.read('config.ini')
+
+    with open('rag_eval_config.yaml', 'r') as f:
+        yaml_config = yaml.safe_load(f)
+
+    return resolve_config(yaml_config, env_vars, ini_config)
+
+
+def get_llm(config):
+    """Initialize LLM based on provider."""
+    llm_config = config['ragas']['llm']
+    provider = llm_config['provider'].lower()
+
+    if provider == 'azure':
+        from langchain_openai import AzureChatOpenAI
+        return AzureChatOpenAI(
+            azure_endpoint=llm_config['azure_endpoint'],
+            api_key=llm_config['api_key'],
+            api_version=llm_config.get('api_version', '2024-02-15-preview'),
+            deployment_name=llm_config['model'],
+            temperature=float(llm_config.get('temperature', 0.0))
+        )
+    elif provider == 'openai':
+        from langchain_openai import ChatOpenAI
+        return ChatOpenAI(
+            api_key=llm_config['api_key'],
+            model=llm_config['model'],
+            temperature=float(llm_config.get('temperature', 0.0))
+        )
+    elif provider == 'ollama':
+        from langchain_ollama import ChatOllama
+        return ChatOllama(
+            base_url=llm_config.get('base_url', 'http://localhost:11434'),
+            model=llm_config['model'],
+            temperature=float(llm_config.get('temperature', 0.0))
+        )
+    else:
+        raise ValueError(f"Unknown LLM provider: {provider}")
+
+
+def get_embeddings(config):
+    """Initialize embeddings based on provider."""
+    emb_config = config['ragas']['embeddings']
+    provider = emb_config['provider'].lower()
+
+    if provider == 'azure':
+        from langchain_openai import AzureOpenAIEmbeddings
+        return AzureOpenAIEmbeddings(
+            azure_endpoint=emb_config['azure_endpoint'],
+            api_key=emb_config['api_key'],
+            api_version=emb_config.get('api_version', '2024-02-15-preview'),
+            deployment=emb_config['model']
+        )
+    elif provider == 'openai':
+        from langchain_openai import OpenAIEmbeddings
+        return OpenAIEmbeddings(
+            api_key=emb_config['api_key'],
+            model=emb_config['model']
+        )
+    elif provider == 'ollama':
+        from langchain_ollama import OllamaEmbeddings
+        return OllamaEmbeddings(
+            base_url=emb_config.get('base_url', 'http://localhost:11434'),
+            model=emb_config['model']
+        )
+    else:
+        raise ValueError(f"Unknown embeddings provider: {provider}")
+
+
+def get_metrics(config):
+    """Get list of Ragas metrics."""
+    metric_map = {
+        'faithfulness': faithfulness,
+        'answer_relevancy': answer_relevancy,
+        'context_precision': context_precision,
+        'answer_correctness': answer_correctness
+    }
+    return [metric_map[m] for m in config['ragas']['metrics'] if m in metric_map]
+
+
+def get_auth_headers_and_cookies(config):
+    """Get authentication headers and cookies."""
+    auth_config = config.get('auth', {})
+    auth_type = auth_config.get('type', 'none').lower()
+    headers = {}
+    cookies = {}
+
+    if auth_type == 'cookie':
+        cookie_name = auth_config.get('cookie_name', 'session')
+        cookie_value = auth_config.get('cookie_value', '')
+        if cookie_value:
+            cookies[cookie_name] = cookie_value
+    elif auth_type == 'bearer':
+        token = auth_config.get('bearer_token', '')
+        if token:
+            headers['Authorization'] = f'Bearer {token}'
+    elif auth_type == 'header':
+        header_name = auth_config.get('header_name', '')
+        header_value = auth_config.get('header_value', '')
+        if header_name and header_value:
+            headers[header_name] = header_value
+
+    return headers, cookies
+
+
+def extract_response_data(response, endpoint_config):
+    """Extract data from API response."""
+    data = response.json()
+    response_path = endpoint_config.get('response_path', '')
+
+    if response_path:
+        for key in response_path.split('.'):
+            if isinstance(data, dict) and key in data:
+                data = data[key]
+            elif isinstance(data, list) and key.isdigit():
+                data = data[int(key)]
+            else:
+                return data
+    return data
+
+
+def make_api_request(base_url, endpoint_config, query, chat_id, auth_headers, auth_cookies, verify_ssl=True):
+    """Make API request to backend."""
+    url = base_url.rstrip('/') + endpoint_config['path']
+    method = endpoint_config.get('method', 'POST').upper()
+
+    body = endpoint_config.get('body', {}).copy()
+    body['query'] = query
+    body['chat_id'] = chat_id
+
+    headers = {'Content-Type': 'application/json'}
+    headers.update(auth_headers)
+
+    if method == 'POST':
+        response = requests.post(url, json=body, headers=headers, cookies=auth_cookies, verify=verify_ssl)
+    else:
+        response = requests.get(url, params=body, headers=headers, cookies=auth_cookies, verify=verify_ssl)
+
+    response.raise_for_status()
+    return response
+
+
+def get_context(config, query, chat_id, auth_headers, auth_cookies):
+    """Get context from backend API."""
+    base_url = config['backend']['base_url']
+    endpoint_config = config['backend']['endpoints']['context']
+    verify_ssl = config['backend'].get('verify_ssl', True)
+
+    response = make_api_request(base_url, endpoint_config, query, chat_id, auth_headers, auth_cookies, verify_ssl)
+    context = extract_response_data(response, endpoint_config)
+
+    if isinstance(context, list):
+        return [str(c) for c in context]
+    return [str(context)]
+
+
+def get_answer(config, query, chat_id, auth_headers, auth_cookies):
+    """Get answer from backend API."""
+    base_url = config['backend']['base_url']
+    endpoint_config = config['backend']['endpoints']['answer']
+    verify_ssl = config['backend'].get('verify_ssl', True)
+
+    response = make_api_request(base_url, endpoint_config, query, chat_id, auth_headers, auth_cookies, verify_ssl)
+    answer = extract_response_data(response, endpoint_config)
+
+    return str(answer)
+
+
+
+def run_evaluation():
+    """Main evaluation function."""
+    print("=" * 60)
+    print("RAGSentinel - RAG Evaluation Framework")
+    print("=" * 60)
+
+    print("\n📁 Loading configuration...")
+    config = load_config()
+
+    dataset_path = config['dataset']['path']
+    print(f"📊 Loading dataset from {dataset_path}...")
+    dataset = pd.read_csv(dataset_path)
+
+    auth_headers, auth_cookies = get_auth_headers_and_cookies(config)
+
+    results = []
+    print(f"\n🔗 Collecting responses from {config['backend']['base_url']}...")
+
+    for idx, row in dataset.iterrows():
+        chat_id = str(row['chat_id'])
+        query = row['query']
+        ground_truth = row['ground_truth']
+
+        try:
+            context = get_context(config, query, chat_id, auth_headers, auth_cookies)
+            answer = get_answer(config, query, chat_id, auth_headers, auth_cookies)
+
+            results.append({
+                'question': query,
+                'contexts': context,
+                'answer': answer,
+                'ground_truth': ground_truth
+            })
+            print(f"  ✓ Processed query {idx + 1}/{len(dataset)}: {query[:50]}...")
+        except Exception as e:
+            print(f"  ✗ Error processing query {idx + 1}: {e}")
+            continue
+
+    if not results:
+        print("\n❌ No results collected. Exiting.")
+        return
+
+    eval_df = pd.DataFrame(results)
+    print(f"\n✓ Collected {len(eval_df)} responses")
+
+    print("\n🤖 Initializing LLM and embeddings...")
+    llm = get_llm(config)
+    embeddings = get_embeddings(config)
+
+    metrics = get_metrics(config)
+    print(f"   Metrics: {', '.join(config['ragas']['metrics'])}")
+
+    print("\n📈 Preparing data for RAGAS evaluation...")
+    ragas_data = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
+
+    for _, row in eval_df.iterrows():
+        contexts = row.get("contexts", [])
+        if not isinstance(contexts, list):
+            contexts = [str(contexts)]
+        contexts = [str(c) for c in contexts if c and str(c).strip()]
+        if not contexts:
+            contexts = ["No context available."]
+
+        ragas_data["question"].append(str(row["question"]))
+        ragas_data["answer"].append(str(row["answer"]))
+        ragas_data["contexts"].append(contexts)
+        ragas_data["ground_truth"].append(str(row["ground_truth"]))
+
+    dataset = Dataset.from_dict(ragas_data)
+
+    print("\n⏳ Evaluating with Ragas metrics (this may take a while)...")
+
+    run_config = RunConfig(timeout=300, max_retries=3, max_wait=600)
+
+    ragas_result = evaluate(
+        dataset,
+        metrics=metrics,
+        llm=llm,
+        embeddings=embeddings,
+        batch_size=2,
+        run_config=run_config,
+        raise_exceptions=False
+    )
+
+    print("\n📊 Processing results...")
+    scores_df = ragas_result.to_pandas()
+    numeric_columns = scores_df.select_dtypes(include=['float64', 'float32', 'int64', 'int32']).columns
+    mean_scores = scores_df[numeric_columns].mean().to_dict()
+
+    mlflow_config = config['mlflow']
+    mlflow.set_tracking_uri(mlflow_config['tracking_uri'])
+    mlflow.set_experiment(mlflow_config['experiment_name'])
+
+    print("\n📤 Logging results to MLflow...")
+    run_name = mlflow_config.get('run_name', 'RAG Evaluation')
+    with mlflow.start_run(run_name=run_name):
+        print("\n" + "=" * 40)
+        print("📊 EVALUATION RESULTS")
+        print("=" * 40)
+        for metric_name, value in mean_scores.items():
+            mlflow.log_metric(metric_name, value)
+            print(f"   {metric_name}: {value:.4f}")
+
+        mlflow.log_param("dataset_path", dataset_path)
+        mlflow.log_param("num_samples", len(eval_df))
+        mlflow.log_table(data=scores_df, artifact_file="ragas_detailed_results.json")
+
+    print("\n" + "=" * 60)
+    print("✅ Evaluation complete!")
+    print(f"🔗 View results at: {mlflow_config['tracking_uri']}")
+    print("=" * 60)

rag_sentinel-0.1.0/src/rag_sentinel/templates/.env.template

@@ -0,0 +1,52 @@
+# =============================================================================
+# RAGSentinel Environment Variables
+# =============================================================================
+# This file contains sensitive LLM and Embeddings configuration.
+# Copy this file to .env and fill in your values.
+# DO NOT commit .env to version control!
+# =============================================================================
+
+# =============================================================================
+# LLM Configuration
+# =============================================================================
+# Provider options: azure, openai, ollama
+LLM_PROVIDER=azure
+
+# Azure OpenAI LLM Configuration
+AZURE_LLM_API_KEY=
+AZURE_LLM_ENDPOINT=
+AZURE_LLM_DEPLOYMENT_NAME=
+AZURE_LLM_MODEL=gpt-4
+AZURE_LLM_TEMPERATURE=0.0
+AZURE_LLM_API_VERSION=2024-02-15-preview
+
+# OpenAI LLM Configuration
+OPENAI_LLM_API_KEY=
+OPENAI_LLM_MODEL=gpt-4
+OPENAI_LLM_TEMPERATURE=0.0
+
+# Ollama LLM Configuration
+OLLAMA_LLM_BASE_URL=http://localhost:11434
+OLLAMA_LLM_MODEL=llama3
+OLLAMA_LLM_TEMPERATURE=0.0
+
+# =============================================================================
+# Embeddings Configuration
+# =============================================================================
+# Provider options: azure, openai, ollama
+EMBEDDINGS_PROVIDER=azure
+
+# Azure OpenAI Embeddings Configuration
+AZURE_EMBEDDINGS_API_KEY=
+AZURE_EMBEDDINGS_ENDPOINT=
+AZURE_EMBEDDINGS_DEPLOYMENT_NAME=
+AZURE_EMBEDDINGS_API_VERSION=2024-02-15-preview
+
+# OpenAI Embeddings Configuration
+OPENAI_EMBEDDINGS_API_KEY=
+OPENAI_EMBEDDINGS_MODEL=text-embedding-3-small
+
+# Ollama Embeddings Configuration
+OLLAMA_EMBEDDINGS_BASE_URL=http://localhost:11434
+OLLAMA_EMBEDDINGS_MODEL=nomic-embed-text
+

rag_sentinel-0.1.0/src/rag_sentinel/templates/config.ini.template

@@ -0,0 +1,39 @@
+# =============================================================================
+# RAGSentinel Configuration File
+# =============================================================================
+# This file contains non-sensitive configuration settings.
+# Copy this file to config.ini and fill in your values.
+# =============================================================================
+
+[mlflow]
+# MLflow tracking server URI
+tracking_uri = http://127.0.0.1:5001
+
+[app]
+# Backend RAG application URL (e.g., https://your-app.com/backend)
+app_url =
+
+[endpoints]
+# API endpoint paths for context retrieval and answer generation
+context_path = /api/retrieve_context
+answer_path = /api/respond
+
+[dataset]
+# Path to test dataset CSV file (columns: query, ground_truth, chat_id)
+path = test_dataset.csv
+
+[auth]
+# =============================================================================
+# Authentication Configuration
+# =============================================================================
+# type options: cookie, bearer, header, or none
+# For cookie auth: set type=cookie, cookie_name, and cookie_value
+# For bearer auth: set type=bearer and bearer_token
+# For header auth: set type=header, header_name, and header_value
+# =============================================================================
+type = cookie
+cookie_name = session
+cookie_value =
+bearer_token =
+header_name =
+header_value =

rag_sentinel-0.1.0/src/rag_sentinel/templates/rag_eval_config.yaml

@@ -0,0 +1,99 @@
+# =============================================================================
+# RAG Evaluation Configuration
+# =============================================================================
+# This file contains the master configuration for RAG evaluation.
+#
+# Placeholders:
+#   ${ENV:VARIABLE_NAME} - Gets value from .env file (LLM & Embeddings only)
+#   ${INI:section.key}   - Gets value from config.ini file (all other settings)
+#   Direct values        - Hardcoded defaults (rarely need to change)
+# =============================================================================
+
+# MLflow Configuration
+mlflow:
+  tracking_uri: "${INI:mlflow.tracking_uri}"
+  experiment_name: "RAG Evaluation"
+  run_name: "RAG Evaluation Run"
+
+# Backend API Configuration
+backend:
+  base_url: "${INI:app.app_url}"
+  verify_ssl: false
+
+  auth:
+    type: "${INI:auth.type}"
+    cookie_name: "${INI:auth.cookie_name}"
+    cookie_value: "${INI:auth.cookie_value}"
+    bearer_token: "${INI:auth.bearer_token}"
+    header_name: "${INI:auth.header_name}"
+    header_value: "${INI:auth.header_value}"
+
+  endpoints:
+    context:
+      path: "${INI:endpoints.context_path}"
+      method: "POST"
+      headers:
+        Content-Type: "application/json"
+      body:
+        query: "{query}"
+        chat_id: "{chat_id}"
+        selected_option: "QPR Review"
+      response_key: "retrieved_contexts"
+
+    answer:
+      path: "${INI:endpoints.answer_path}"
+      method: "POST"
+      headers:
+        Content-Type: "application/json"
+      body:
+        prompt: "{query}"
+        chat_id: "{chat_id}"
+        selected_option: "QPR Review"
+      stream: true
+      response_key: "response"
+
+# Dataset Configuration
+dataset:
+  path: "${INI:dataset.path}"
+
+# Ragas Metrics Configuration
+ragas:
+  metrics:
+    - "Faithfulness"
+    - "AnswerRelevancy"
+    - "ContextPrecision"
+    - "AnswerCorrectness"
+
+  # LLM Configuration (from .env)
+  llm:
+    provider: "${ENV:LLM_PROVIDER}"
+    azure:
+      api_key: "${ENV:AZURE_LLM_API_KEY}"
+      endpoint: "${ENV:AZURE_LLM_ENDPOINT}"
+      deployment_name: "${ENV:AZURE_LLM_DEPLOYMENT_NAME}"
+      model: "${ENV:AZURE_LLM_MODEL}"
+      temperature: "${ENV:AZURE_LLM_TEMPERATURE}"
+      api_version: "${ENV:AZURE_LLM_API_VERSION}"
+    openai:
+      api_key: "${ENV:OPENAI_LLM_API_KEY}"
+      model: "${ENV:OPENAI_LLM_MODEL}"
+      temperature: "${ENV:OPENAI_LLM_TEMPERATURE}"
+    ollama:
+      base_url: "${ENV:OLLAMA_LLM_BASE_URL}"
+      model: "${ENV:OLLAMA_LLM_MODEL}"
+      temperature: "${ENV:OLLAMA_LLM_TEMPERATURE}"
+
+  # Embeddings Configuration (from .env)
+  embeddings:
+    provider: "${ENV:EMBEDDINGS_PROVIDER}"
+    azure:
+      api_key: "${ENV:AZURE_EMBEDDINGS_API_KEY}"
+      endpoint: "${ENV:AZURE_EMBEDDINGS_ENDPOINT}"
+      deployment_name: "${ENV:AZURE_EMBEDDINGS_DEPLOYMENT_NAME}"
+      api_version: "${ENV:AZURE_EMBEDDINGS_API_VERSION}"
+    openai:
+      api_key: "${ENV:OPENAI_EMBEDDINGS_API_KEY}"
+      model: "${ENV:OPENAI_EMBEDDINGS_MODEL}"
+    ollama:
+      base_url: "${ENV:OLLAMA_EMBEDDINGS_BASE_URL}"
+      model: "${ENV:OLLAMA_EMBEDDINGS_MODEL}"

rag_sentinel-0.1.0/src/rag_sentinel.egg-info/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: rag-sentinel
+Version: 0.1.0
+Summary: RAG Evaluation Framework using Ragas metrics and MLflow tracking
+Author: RAGSentinel Team
+License: MIT
+Project-URL: Homepage, https://github.com/yourusername/rag-sentinel
+Project-URL: Repository, https://github.com/yourusername/rag-sentinel
+Keywords: rag,evaluation,ragas,mlflow,llm,ai
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ragas>=0.2.0
+Requires-Dist: mlflow>=2.9.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: langchain-openai>=0.0.5
+Requires-Dist: langchain-ollama>=0.0.1
+Requires-Dist: langchain-core>=0.1.0
+Requires-Dist: datasets>=2.14.0
+Dynamic: license-file
+
+# RAGSentinel
+
+RAG Evaluation Framework using Ragas metrics and MLflow tracking.
+
+## Installation
+
+```bash
+pip install rag-sentinel
+```
+
+## Quick Start
+
+### 1. Initialize Project
+
+```bash
+mkdir my-rag-eval
+cd my-rag-eval
+rag-sentinel init
+```
+
+This creates:
+- `.env` - LLM/Embeddings API keys
+- `config.ini` - App settings and authentication
+- `rag_eval_config.yaml` - Master configuration
+
+### 2. Configure
+
+Edit `.env`:
+```bash
+LLM_PROVIDER=azure
+AZURE_LLM_API_KEY=your-api-key
+AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com/
+```
+
+Edit `config.ini`:
+```ini
+[app]
+app_url = https://your-rag-app.com/backend
+
+[auth]
+type = cookie
+cookie_name = session
+cookie_value = your-session-cookie
+```
+
+### 3. Create Test Dataset
+
+Create `test_dataset.csv`:
+```csv
+query,ground_truth,chat_id
+Hello,Hello! How can I help you?,1
+```
+
+### 4. Run Evaluation
+
+```bash
+rag-sentinel run
+```
+
+View results at: http://127.0.0.1:5001
+
+## CLI Commands
+
+```bash
+rag-sentinel init              # Initialize project
+rag-sentinel run               # Run evaluation (auto-starts MLflow)
+rag-sentinel run --no-server   # Run without starting MLflow
+rag-sentinel validate          # Validate configuration
+```
+
+## Metrics
+
+- **Faithfulness** - Factual consistency of answer with context
+- **Answer Relevancy** - How relevant the answer is to the question
+- **Context Precision** - Quality of retrieved context
+- **Answer Correctness** - Comparison against ground truth
+
+## License
+
+MIT
+

rag_sentinel-0.1.0/src/rag_sentinel.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+src/rag_sentinel/__init__.py
+src/rag_sentinel/cli.py
+src/rag_sentinel/evaluator.py
+src/rag_sentinel.egg-info/PKG-INFO
+src/rag_sentinel.egg-info/SOURCES.txt
+src/rag_sentinel.egg-info/dependency_links.txt
+src/rag_sentinel.egg-info/entry_points.txt
+src/rag_sentinel.egg-info/requires.txt
+src/rag_sentinel.egg-info/top_level.txt
+src/rag_sentinel/templates/.env.template
+src/rag_sentinel/templates/config.ini.template
+src/rag_sentinel/templates/rag_eval_config.yaml

rag_sentinel-0.1.0/src/rag_sentinel.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

rag_sentinel-0.1.0/src/rag_sentinel.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+rag-sentinel = rag_sentinel.cli:main

rag_sentinel-0.1.0/src/rag_sentinel.egg-info/requires.txt

@@ -0,0 +1,10 @@
+ragas>=0.2.0
+mlflow>=2.9.0
+pandas>=2.0.0
+pyyaml>=6.0
+requests>=2.31.0
+python-dotenv>=1.0.0
+langchain-openai>=0.0.5
+langchain-ollama>=0.0.1
+langchain-core>=0.1.0
+datasets>=2.14.0

rag_sentinel-0.1.0/src/rag_sentinel.egg-info/top_level.txt

@@ -0,0 +1 @@
+rag_sentinel