eval-ai-library 0.3.2__py3-none-any.whl → 0.3.10__py3-none-any.whl

{eval_ai_library-0.3.2.dist-info → eval_ai_library-0.3.10.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-ai-library
-Version: 0.3.2
+Version: 0.3.10
 Summary: Comprehensive AI Model Evaluation Framework with support for multiple LLM providers
 Author-email: Aleksandr Meshkov <alekslynx90@gmail.com>
 License: MIT
@@ -45,6 +45,7 @@ Requires-Dist: html2text>=2020.1.16
 Requires-Dist: markdown>=3.4.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: striprtf>=0.0.26
+Requires-Dist: flask>=3.0.0
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0.0; extra == "dev"
 Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
@@ -807,6 +808,383 @@ response, cost = await chat_complete(
 )
 ```
 
+## Dashboard
+
+The library includes an interactive web dashboard for visualizing evaluation results. All evaluation results are automatically saved to cache and can be viewed in a beautiful web interface.
+
+### Features
+
+- 📊 **Interactive Charts**: Visual representation of metrics with Chart.js
+- 📈 **Metrics Summary**: Aggregate statistics across all evaluations
+- 🔍 **Detailed View**: Drill down into individual test cases and metric results
+- 💾 **Session History**: Access past evaluation runs
+- 🎨 **Beautiful UI**: Modern, responsive interface with color-coded results
+- 🔄 **Real-time Updates**: Refresh to see new evaluation results
+
+### Starting the Dashboard
+
+The dashboard runs as a separate server that you start once and keep running:
+```bash
+# Start dashboard server (from your project directory)
+eval-lib dashboard
+
+# Custom port if 14500 is busy
+eval-lib dashboard --port 8080
+
+# Custom cache directory
+eval-lib dashboard --cache-dir /path/to/cache
+```
+
+Once started, the dashboard will be available at `http://localhost:14500`
+
+### Saving Results to Dashboard
+
+Enable dashboard cache saving in your evaluation:
+```python
+import asyncio
+from eval_lib import (
+    evaluate,
+    EvalTestCase,
+    AnswerRelevancyMetric,
+    FaithfulnessMetric
+)
+
+async def evaluate_with_dashboard():
+    test_cases = [
+        EvalTestCase(
+            input="What is the capital of France?",
+            actual_output="Paris is the capital.",
+            expected_output="Paris",
+            retrieval_context=["Paris is the capital of France."]
+        )
+    ]
+    
+    metrics = [
+        AnswerRelevancyMetric(model="gpt-4o-mini", threshold=0.7),
+        FaithfulnessMetric(model="gpt-4o-mini", threshold=0.8)
+    ]
+    
+    # Results are saved to .eval_cache/ for dashboard viewing
+    results = await evaluate(
+        test_cases=test_cases,
+        metrics=metrics,
+        show_dashboard=True,  # ← Enable dashboard cache
+        session_name="My First Evaluation"  # Optional session name
+    )
+    
+    return results
+
+asyncio.run(evaluate_with_dashboard())
+```
+
+### Typical Workflow
+
+**Terminal 1 - Start Dashboard (once):**
+```bash
+cd ~/my_project
+eval-lib dashboard
+# Leave this terminal open - dashboard stays running
+```
+
+**Terminal 2 - Run Evaluations (multiple times):**
+```python
+# Run evaluation 1
+results1 = await evaluate(
+    test_cases=test_cases1,
+    metrics=metrics,
+    show_dashboard=True,
+    session_name="Evaluation 1"
+)
+
+# Run evaluation 2
+results2 = await evaluate(
+    test_cases=test_cases2,
+    metrics=metrics,
+    show_dashboard=True,
+    session_name="Evaluation 2"
+)
+
+# All results are cached and viewable in dashboard
+```
+
+**Browser:**
+- Open `http://localhost:14500`
+- Refresh page (F5) to see new evaluation results
+- Switch between different evaluation sessions using the dropdown
+
+### Dashboard Features
+
+**Summary Cards:**
+- Total test cases evaluated
+- Total cost across all evaluations
+- Number of metrics used
+
+**Metrics Overview:**
+- Average scores per metric
+- Pass/fail counts
+- Success rates
+- Model used for evaluation
+- Total cost per metric
+
+**Detailed Results Table:**
+- Test case inputs and outputs
+- Individual metric scores
+- Pass/fail status
+- Click "View Details" for full information including:
+  - Complete input/output/expected output
+  - Full retrieval context
+  - Detailed evaluation reasoning
+  - Complete evaluation logs
+
+**Charts:**
+- Bar chart: Average scores by metric
+- Doughnut chart: Success rate distribution
+
+### Cache Management
+
+Results are stored in `.eval_cache/results.json` in your project directory:
+```bash
+# View cache contents
+cat .eval_cache/results.json
+
+# Clear cache via dashboard
+# Click "Clear Cache" button in dashboard UI
+
+# Or manually delete cache
+rm -rf .eval_cache/
+```
+
+### CLI Commands
+```bash
+# Start dashboard with defaults
+eval-lib dashboard
+
+# Custom port
+eval-lib dashboard --port 8080
+
+# Custom cache directory
+eval-lib dashboard --cache-dir /path/to/project/.eval_cache
+
+# Check library version
+eval-lib version
+
+# Help
+eval-lib help
+```
+
+## Custom LLM Providers
+
+The library supports custom LLM providers through the `CustomLLMClient` abstract base class. This allows you to integrate any LLM provider, including internal corporate models, locally-hosted models, or custom endpoints.
+
+### Creating a Custom Provider
+
+Implement the `CustomLLMClient` interface:
+```python
+from eval_lib import CustomLLMClient
+from typing import Optional
+from openai import AsyncOpenAI
+
+class InternalLLMClient(CustomLLMClient):
+    """Client for internal corporate LLM or custom endpoint"""
+    
+    def __init__(
+        self,
+        endpoint: str,
+        model: str,
+        api_key: Optional[str] = None,
+        temperature: float = 0.0
+    ):
+        """
+        Args:
+            endpoint: Your internal LLM endpoint URL (e.g., "https://internal-llm.company.com/v1")
+            model: Model name to use
+            api_key: API key if required (optional for local models)
+            temperature: Default temperature
+        """
+        self.endpoint = endpoint
+        self.model = model
+        self.api_key = api_key or "not-needed"  # Some endpoints don't need auth
+        
+        self.client = AsyncOpenAI(
+            api_key=self.api_key,
+            base_url=self.endpoint
+        )
+    
+    async def chat_complete(
+        self,
+        messages: list[dict[str, str]],
+        temperature: float
+    ) -> tuple[str, Optional[float]]:
+        """Generate response from internal LLM"""
+        response = await self.client.chat.completions.create(
+            model=self.model,
+            messages=messages,
+            temperature=temperature,
+        )
+        text = response.choices[0].message.content.strip()
+        cost = None  # Internal models typically don't have API costs
+        return text, cost
+    
+    def get_model_name(self) -> str:
+        """Return model name for logging"""
+        return f"internal:{self.model}"
+```
+
+### Using Custom Providers
+
+Use your custom provider in any metric:
+```python
+import asyncio
+from eval_lib import (
+    evaluate,
+    EvalTestCase,
+    AnswerRelevancyMetric,
+    FaithfulnessMetric
+)
+
+# Create custom internal LLM client
+internal_llm = InternalLLMClient(
+    endpoint="https://internal-llm.company.com/v1",
+    model="company-gpt-v2",
+    api_key="your-internal-key"  # Optional
+)
+
+# Use in metrics
+test_cases = [
+    EvalTestCase(
+        input="What is the capital of France?",
+        actual_output="Paris is the capital.",
+        expected_output="Paris",
+        retrieval_context=["Paris is the capital of France."]
+    )
+]
+
+metrics = [
+    AnswerRelevancyMetric(
+        model=internal_llm,  # ← Your custom LLM
+        threshold=0.7
+    ),
+    FaithfulnessMetric(
+        model=internal_llm,  # ← Same custom client
+        threshold=0.8
+    )
+]
+
+async def run_evaluation():
+    results = await evaluate(
+        test_cases=test_cases,
+        metrics=metrics,
+        verbose=True
+    )
+    return results
+
+asyncio.run(run_evaluation())
+```
+
+### Mixing Standard and Custom Providers
+
+You can mix standard and custom providers in the same evaluation:
+```python
+# Create custom provider
+internal_llm = InternalLLMClient(
+    endpoint="https://internal-llm.company.com/v1",
+    model="company-model"
+)
+
+# Mix standard OpenAI and custom internal LLM
+metrics = [
+    AnswerRelevancyMetric(
+        model="gpt-4o-mini",  # ← Standard OpenAI
+        threshold=0.7
+    ),
+    FaithfulnessMetric(
+        model=internal_llm,  # ← Custom internal LLM
+        threshold=0.8
+    ),
+    ContextualRelevancyMetric(
+        model="anthropic:claude-sonnet-4-0",  # ← Standard Anthropic
+        threshold=0.7
+    )
+]
+
+results = await evaluate(test_cases=test_cases, metrics=metrics)
+```
+
+### Custom Provider Use Cases
+
+**When to use custom providers:**
+
+1. **Internal Corporate LLMs**: Connect to your company's proprietary models
+2. **Local Models**: Integrate locally-hosted models (vLLM, TGI, LM Studio, Ollama with custom setup)
+3. **Fine-tuned Models**: Use your own fine-tuned models hosted anywhere
+4. **Research Models**: Connect to experimental or research models
+5. **Custom Endpoints**: Any LLM accessible via HTTP endpoint
+
+**Example: Local Model with vLLM**
+```python
+# vLLM server running on localhost:8000
+local_model = InternalLLMClient(
+    endpoint="http://localhost:8000/v1",
+    model="meta-llama/Llama-2-7b-chat",
+    api_key=None  # Local models don't need auth
+)
+
+# Use in evaluation
+metric = AnswerRelevancyMetric(model=local_model, threshold=0.7)
+```
+
+**Example: Corporate Internal Model**
+```python
+# Company's internal LLM with authentication
+company_model = InternalLLMClient(
+    endpoint="https://ai-platform.company.internal/api/v1",
+    model="company-gpt-enterprise",
+    api_key="internal-api-key-here"
+)
+
+# Use in evaluation
+metrics = [
+    AnswerRelevancyMetric(model=company_model, threshold=0.7),
+    FaithfulnessMetric(model=company_model, threshold=0.8)
+]
+```
+
+**Key Requirements:**
+
+1. **`async def chat_complete()`** - Must be async and return `(str, Optional[float])`
+2. **`def get_model_name()`** - Return string identifier for logging
+3. **Error Handling** - Handle connection and API errors appropriately
+4. **Cost** - Return `None` for cost if not applicable (e.g., internal/local models)
+
+### Advanced: Custom Authentication
+
+For custom authentication schemes:
+```python
+class CustomAuthLLMClient(CustomLLMClient):
+    """Client with custom authentication"""
+    
+    def __init__(self, endpoint: str, auth_token: str):
+        self.endpoint = endpoint
+        self.headers = {
+            "Authorization": f"Bearer {auth_token}",
+            "X-Custom-Header": "value"
+        }
+        # Use aiohttp or httpx for custom auth
+        import aiohttp
+        self.session = aiohttp.ClientSession(headers=self.headers)
+    
+    async def chat_complete(self, messages, temperature):
+        async with self.session.post(
+            f"{self.endpoint}/chat",
+            json={"messages": messages, "temperature": temperature}
+        ) as response:
+            data = await response.json()
+            return data["content"], None
+    
+    def get_model_name(self):
+        return "custom-auth-model"
+```
+
 ## Test Data Generation
 
 The library includes a powerful test data generator that can create realistic test cases either from scratch or based on your documents.

{eval_ai_library-0.3.2.dist-info → eval_ai_library-0.3.10.dist-info}/RECORD

@@ -1,8 +1,11 @@
-eval_ai_library-0.3.2.dist-info/licenses/LICENSE,sha256=rK9uLDgWNrCHNdp-Zma_XghDE7Fs0u0kDi3WMcmYx6w,1074
-eval_lib/__init__.py,sha256=IeDW5pLarPmHCBJu-6vFX71g9VxZTS46UkHjDkrU_Gw,3043
-eval_lib/evaluate.py,sha256=GjlXZb5dnl44LCaJwdkyGCYcC50zoNZn3NrofzNAVJ0,11490
+eval_ai_library-0.3.10.dist-info/licenses/LICENSE,sha256=rK9uLDgWNrCHNdp-Zma_XghDE7Fs0u0kDi3WMcmYx6w,1074
+eval_lib/__init__.py,sha256=OMrncAoUbbrJXfaYf8k2wJEGw1e2r9k-s1uXkerZ9mE,3204
+eval_lib/cli.py,sha256=Fvnj6HgCQ3lhx28skweALgHSm3FMEpavQCB3o_sQhtE,4731
+eval_lib/dashboard_server.py,sha256=6ND7ujtzN0PdMyVmJFnKDWrIf4kaodnetLZRPUhYHas,6751
+eval_lib/evaluate.py,sha256=LEjwPsuuPGpdwes-xXesCKtKlBFFMF5X1CpIGJIrZ20,12630
 eval_lib/evaluation_schema.py,sha256=7IDd_uozqewhh7k0p1hKut_20udvRxxkV6thclxKUg0,1904
-eval_lib/llm_client.py,sha256=3eMcarKLkDLDVh4AOxgWbaIzXlzpqsmEfJXNTBonNic,13633
+eval_lib/html.py,sha256=_tBTtwxZpjIwc3TVOyLGDw2VFD77aAeA47JdovoZ0CI,24094
+eval_lib/llm_client.py,sha256=eeTVhCLR1uYbhqOEOSBt3wWPKuzgzA9v8m0F9f-4Gqg,14910
 eval_lib/metric_pattern.py,sha256=wULgMNDeAqJC_Qjglo7bYzY2eGhA_PmY_hA_qGfg0sI,11730
 eval_lib/price.py,sha256=jbmkkUTxPuXrkSHuaJYPl7jSzfDIzQ9p_swWWs26UJ0,1986
 eval_lib/py.typed,sha256=8PjyZ1aVoQpRVvt71muvuq5qE-jTFZkK-GLHkhdebmc,26
@@ -28,7 +31,8 @@ eval_lib/metrics/faithfulness_metric/faithfulness.py,sha256=OqamlhTOps7d-NOStSIK
 eval_lib/metrics/geval/geval.py,sha256=mNciHXnqU2drOJsWlYmbwftGiKM89-Ykw2f6XneIGBM,10629
 eval_lib/metrics/restricted_refusal_metric/restricted_refusal.py,sha256=4QqYgGMcp6W9Lw-v4s0AlUhMSOKvBOEgnLvhqVXaT9I,4286
 eval_lib/metrics/toxicity_metric/toxicity.py,sha256=rBE1_fvpbCRdBpBep1y1LTIhofKR8GD4Eh76EOYzxL0,4076
-eval_ai_library-0.3.2.dist-info/METADATA,sha256=7kIgiDelzM1wbCdKdy4PXuFudIZ3UCUTPXhlVccFe9k,37706
-eval_ai_library-0.3.2.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-eval_ai_library-0.3.2.dist-info/top_level.txt,sha256=uQHpEd2XI0oZgq1eCww9zMvVgDJgwXMWkCD45fYUzEg,9
-eval_ai_library-0.3.2.dist-info/RECORD,,
+eval_ai_library-0.3.10.dist-info/METADATA,sha256=pevxrimXqbreKbRwHZ0GBu_VXsfGhles6OMN2SBOJHo,47969
+eval_ai_library-0.3.10.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+eval_ai_library-0.3.10.dist-info/entry_points.txt,sha256=VTDuJiTezDkBLQw1NWcRoOOuZPHqYgOCcVIoYno-L00,47
+eval_ai_library-0.3.10.dist-info/top_level.txt,sha256=uQHpEd2XI0oZgq1eCww9zMvVgDJgwXMWkCD45fYUzEg,9
+eval_ai_library-0.3.10.dist-info/RECORD,,

eval_ai_library-0.3.10.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+eval-lib = eval_lib.cli:main

eval_lib/__init__.py

@@ -7,7 +7,7 @@ A powerful library for evaluating AI models with support for multiple LLM provid
 and a wide range of evaluation metrics for RAG systems and AI agents.
 """
 
-__version__ = "0.3.2"
+__version__ = "0.3.10"
 __author__ = "Aleksandr Meshkov"
 
 # Core evaluation functions
@@ -39,6 +39,7 @@ from eval_lib.llm_client import (
     chat_complete,
     get_embeddings,
     LLMDescriptor,
+    CustomLLMClient,
     Provider
 )
 
@@ -65,6 +66,10 @@ from eval_lib.agent_metrics import (
     KnowledgeRetentionMetric
 )
 
+from .dashboard_server import (
+    DashboardCache
+)
+
 
 def __getattr__(name):
     """
@@ -106,6 +111,7 @@ __all__ = [
     "chat_complete",
     "get_embeddings",
     "LLMDescriptor",
+    "CustomLLMClient",
     "Provider",
 
     # RAG Metrics
@@ -134,4 +140,8 @@ __all__ = [
     # Utils
     "score_agg",
     "extract_json_block",
+
+    # Dashboard
+    'start_dashboard',
+    'DashboardCache',
 ]

eval_lib/cli.py

@@ -0,0 +1,166 @@
+# eval_lib/cli.py
+"""
+Command-line interface for Eval AI Library
+"""
+
+import argparse
+import sys
+from pathlib import Path
+
+
+def run_dashboard():
+    """Run dashboard server from CLI"""
+    parser = argparse.ArgumentParser(
+        description='Eval AI Library Dashboard Server',
+        prog='eval-lib dashboard'
+    )
+    parser.add_argument(
+        '--port',
+        type=int,
+        default=14500,
+        help='Port to run dashboard on (default: 14500)'
+    )
+    parser.add_argument(
+        '--host',
+        type=str,
+        default='0.0.0.0',
+        help='Host to bind to (default: 0.0.0.0)'
+    )
+    parser.add_argument(
+        '--cache-dir',
+        type=str,
+        default='.eval_cache',
+        help='Path to cache directory (default: .eval_cache)'
+    )
+
+    args = parser.parse_args(sys.argv[2:])  # Skip 'eval-lib' and 'dashboard'
+
+    # Import here to avoid loading everything for --help
+    from eval_lib.dashboard_server import DashboardCache
+    from eval_lib.html import HTML_TEMPLATE
+    from flask import Flask, render_template_string, jsonify
+
+    # Create cache with custom directory
+    def get_fresh_cache():
+        """Reload cache from disk"""
+        return DashboardCache(cache_dir=args.cache_dir)
+
+    cache = get_fresh_cache()
+
+    print("="*70)
+    print("📊 Eval AI Library - Dashboard Server")
+    print("="*70)
+
+    # Check cache
+    latest = cache.get_latest()
+    if latest:
+        print(f"\n✅ Found cached results:")
+        print(f"   Latest session: {latest['session_id']}")
+        print(f"   Timestamp: {latest['timestamp']}")
+        print(f"   Total sessions: {len(cache.get_all())}")
+    else:
+        print("\n⚠️  No cached results found")
+        print("   Run an evaluation with show_dashboard=True to populate cache")
+
+    print(f"\n🚀 Starting server...")
+    print(f"   URL: http://localhost:{args.port}")
+    print(f"   Host: {args.host}")
+    print(f"   Cache: {Path(args.cache_dir).absolute()}")
+    print(f"\n💡 Keep this terminal open to keep the server running")
+    print(f"   Press Ctrl+C to stop\n")
+    print("="*70 + "\n")
+
+    app = Flask(__name__)
+    app.config['WTF_CSRF_ENABLED'] = False
+
+    @app.route('/')
+    def index():
+        return render_template_string(HTML_TEMPLATE)
+
+    @app.route('/favicon.ico')
+    def favicon():
+        return '', 204
+
+    @app.after_request
+    def after_request(response):
+        response.headers['Access-Control-Allow-Origin'] = '*'
+        response.headers['Access-Control-Allow-Methods'] = 'GET, POST, OPTIONS'
+        response.headers['Access-Control-Allow-Headers'] = 'Content-Type'
+        return response
+
+    @app.route('/api/latest')
+    def api_latest():
+        cache = get_fresh_cache()
+        latest = cache.get_latest()
+        if latest:
+            return jsonify(latest)
+        return jsonify({'error': 'No results available'}), 404
+
+    @app.route('/api/sessions')
+    def api_sessions():
+        cache = get_fresh_cache()
+        sessions = [
+            {
+                'session_id': s['session_id'],
+                'timestamp': s['timestamp'],
+                'total_tests': s['data']['total_tests']
+            }
+            for s in cache.get_all()
+        ]
+        return jsonify(sessions)
+
+    @app.route('/api/session/<session_id>')
+    def api_session(session_id):
+        cache = get_fresh_cache()
+        session = cache.get_by_session(session_id)
+        if session:
+            return jsonify(session)
+        return jsonify({'error': 'Session not found'}), 404
+
+    @app.route('/api/clear')
+    def api_clear():
+        cache = get_fresh_cache()
+        cache.clear()
+        return jsonify({'message': 'Cache cleared'})
+
+    try:
+        app.run(
+            host=args.host,
+            port=args.port,
+            debug=False,
+            use_reloader=False,
+            threaded=True
+        )
+    except KeyboardInterrupt:
+        print("\n\n🛑 Dashboard server stopped")
+
+
+def main():
+    """Main CLI entry point"""
+    parser = argparse.ArgumentParser(
+        description='Eval AI Library CLI',
+        usage='eval-lib <command> [options]'
+    )
+    parser.add_argument(
+        'command',
+        help='Command to run (dashboard, version, help)'
+    )
+
+    # Parse only the command
+    args = parser.parse_args(sys.argv[1:2])
+
+    if args.command == 'dashboard':
+        run_dashboard()
+    elif args.command == 'version':
+        from eval_lib import __version__
+        print(f"Eval AI Library v{__version__}")
+    elif args.command == 'help':
+        parser.print_help()
+    else:
+        print(f"Unknown command: {args.command}")
+        print("Available commands: dashboard, version, help")
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()