eval-ai-library 0.2.2__py3-none-any.whl → 0.3.0__py3-none-any.whl

eval_ai_library-0.3.0.dist-info/METADATA

@@ -0,0 +1,1042 @@
+Metadata-Version: 2.4
+Name: eval-ai-library
+Version: 0.3.0
+Summary: Comprehensive AI Model Evaluation Framework with support for multiple LLM providers
+Author-email: Aleksandr Meshkov <alekslynx90@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/meshkovQA/Eval-ai-library
+Project-URL: Documentation, https://github.com/meshkovQA/Eval-ai-library#readme
+Project-URL: Repository, https://github.com/meshkovQA/Eval-ai-library
+Project-URL: Bug Tracker, https://github.com/meshkovQA/Eval-ai-library/issues
+Keywords: ai,evaluation,llm,rag,metrics,testing,quality-assurance
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: anthropic>=0.18.0
+Requires-Dist: google-genai>=0.2.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: langchain>=0.1.0
+Requires-Dist: langchain-community>=0.0.10
+Requires-Dist: langchain-core>=0.1.0
+Requires-Dist: langchain-text-splitters>=0.2.0
+Requires-Dist: pypdf>=3.0.0
+Requires-Dist: python-docx>=0.8.11
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: pillow>=10.0.0
+Requires-Dist: pytesseract>=0.3.10
+Requires-Dist: python-pptx>=0.6.21
+Requires-Dist: PyMuPDF>=1.23.0
+Requires-Dist: mammoth>=1.6.0
+Requires-Dist: PyYAML>=6.0.0
+Requires-Dist: html2text>=2020.1.16
+Requires-Dist: markdown>=3.4.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: striprtf>=0.0.26
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: isort>=5.12.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=6.0.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=1.2.0; extra == "docs"
+Dynamic: license-file
+
+# Eval AI Library
+
+[![Python Version](https://img.shields.io/badge/python-3.9%2B-blue)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Comprehensive AI Model Evaluation Framework with advanced techniques including **Probability-Weighted Scoring** and **Auto Chain-of-Thought**. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.
+
+## Features
+
+- 🎯 **15+ Evaluation Metrics**: RAG metrics and agent-specific evaluations
+- 🧠 **G-Eval Implementation**: State-of-the-art evaluation with probability-weighted scoring
+- 🔗 **Chain-of-Thought**: Automatic generation of evaluation steps from criteria
+- 🤖 **Multi-Provider Support**: OpenAI, Azure OpenAI, Google Gemini, Anthropic Claude, Ollama
+- 📊 **RAG Metrics**: Answer relevancy, faithfulness, contextual precision/recall, and more
+- 🔧 **Agent Metrics**: Tool correctness, task success rate, role adherence, knowledge retention
+- 🎨 **Custom Metrics**: Advanced custom evaluation with CoT and probability weighting
+- 📦 **Data Generation**: Built-in test case generator from documents
+- ⚡ **Async Support**: Full async/await support for efficient evaluation
+- 💰 **Cost Tracking**: Automatic cost calculation for LLM API calls
+- 📝 **Detailed Logging**: Comprehensive evaluation logs for transparency
+
+## Installation
+```bash
+pip install eval-ai-library
+```
+
+### Development Installation
+```bash
+git clone https://github.com/yourusername/eval-ai-library.git
+cd eval-ai-library
+pip install -e ".[dev]"
+```
+
+## Quick Start
+
+### Basic Batch Evaluation
+```python
+import asyncio
+from eval_lib import (
+    evaluate,
+    EvalTestCase,
+    AnswerRelevancyMetric,
+    FaithfulnessMetric,
+    BiasMetric
+)
+
+async def test_batch_standard_metrics():
+    """Test batch evaluation with multiple test cases and standard metrics"""
+
+    # Create test cases
+    test_cases = [
+        EvalTestCase(
+            input="What is the capital of France?",
+            actual_output="The capital of France is Paris.",
+            expected_output="Paris",
+            retrieval_context=["Paris is the capital of France."]
+        ),
+        EvalTestCase(
+            input="What is photosynthesis?",
+            actual_output="The weather today is sunny.",
+            expected_output="Process by which plants convert light into energy",
+            retrieval_context=[
+                "Photosynthesis is the process by which plants use sunlight."]
+        )
+    ]
+
+    # Define metrics
+    metrics = [
+        AnswerRelevancyMetric(
+            model="gpt-4o-mini",
+            threshold=0.7,
+            temperature=0.5,
+        ),
+        FaithfulnessMetric(
+            model="gpt-4o-mini",
+            threshold=0.8,
+            temperature=0.5,
+        ),
+        BiasMetric(
+            model="gpt-4o-mini",
+            threshold=0.8,
+        ),
+    ]
+
+    # Run batch evaluation
+    results = await evaluate(
+        test_cases=test_cases,
+        metrics=metrics,
+        verbose=True
+    )
+
+    return results
+
+
+if __name__ == "__main__":
+    asyncio.run(test_batch_standard_metrics())
+```
+
+### G-Eval with Probability-Weighted Scoring (single evaluation)
+
+G-Eval implements the state-of-the-art evaluation method from the paper ["G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment"](https://arxiv.org/abs/2303.16634). It uses **probability-weighted scoring** (score = Σ p(si) × si) for fine-grained, continuous evaluation scores.
+```python
+from eval_lib import GEval, EvalTestCase
+
+async def evaluate_with_geval():
+    test_case = EvalTestCase(
+        input="Explain quantum computing to a 10-year-old",
+        actual_output="Quantum computers are like super-powerful regular computers that use special tiny particles to solve really hard problems much faster.",
+    )
+    
+    # G-Eval with auto chain-of-thought
+    metric = GEval(
+        model="gpt-4o",  # Works best with GPT-4
+        threshold=0.7,  # Score range: 0.0-1.0
+        name="Clarity & Simplicity",
+        criteria="Evaluate how clear and age-appropriate the explanation is for a 10-year-old child",
+
+        # Evaluation_steps is auto-generated from criteria if not provided
+        evaluation_steps=[
+        "Step 1: Check if the language is appropriate for a 10-year-old. Avoid complex technical terms, jargon, or abstract concepts that children cannot relate to. The vocabulary should be simple and conversational.",
+        
+        "Step 2: Evaluate the use of analogies and examples. Look for comparisons to everyday objects, activities, or experiences familiar to children (toys, games, school, animals, family activities). Good analogies make abstract concepts concrete.",
+        
+        "Step 3: Assess the structure and flow. The explanation should have a clear beginning, middle, and end. Ideas should build logically, starting with familiar concepts before introducing new ones. Sentences should be short and easy to follow.",
+        
+        "Step 4: Check for engagement elements. Look for questions, storytelling, humor, or interactive elements that capture a child's attention. The tone should be friendly and encouraging, not boring or too formal.",
+        
+        "Step 5: Verify completeness without overwhelming. The explanation should cover the main idea adequately but not overload with too many details. It should answer the question without confusing the child with unnecessary complexity.",
+        
+        "Step 6: Assign a score from 0.0 to 1.0, where 0.0 means completely inappropriate or unclear for a child, and 1.0 means perfectly clear, engaging, and age-appropriate."
+        ],
+        n_samples=20,  # Number of samples for probability estimation (default: 20)
+        sampling_temperature=2.0  # High temperature for diverse sampling (default: 2.0)
+    )
+    
+    result = await metric.evaluate(test_case)
+    
+
+asyncio.run(evaluate_with_geval())
+```
+
+### Custom Evaluation with Verdict-Based Scoring (single evaluation)
+
+CustomEvalMetric uses **verdict-based evaluation** with automatic criteria generation for transparent and detailed scoring:
+```python
+from eval_lib import CustomEvalMetric
+
+async def custom_evaluation():
+    test_case = EvalTestCase(
+        input="Explain photosynthesis",
+        actual_output="Photosynthesis is the process where plants use sunlight, water, and carbon dioxide to create oxygen and energy in the form of sugar.",
+    )
+    
+    # Verdict-based custom evaluation
+    metric = CustomEvalMetric(
+        model="gpt-4o-mini",
+        threshold=0.8,
+        name="Scientific Accuracy",
+        criteria="Evaluate if the explanation is scientifically accurate and complete",
+        evaluation_steps=None,  # Auto-generated if not provided
+        temperature=0.8,  # Controls verdict aggregation
+        verbose=True
+    )
+    
+    result = await metric.evaluate(test_case)
+
+asyncio.run(custom_evaluation())
+```
+
+### Agent Evaluation
+```python
+from eval_lib import (
+    evaluate,
+    EvalTestCase,
+    ToolCorrectnessMetric,
+    TaskSuccessRateMetric
+)
+
+async def evaluate_agent():
+    test_cases = EvalTestCase(
+        input="Book a flight to New York for tomorrow",
+        actual_output="I've found available flights and booked your trip to New York for tomorrow.",
+        tools_called=["search_flights", "book_flight"],
+        expected_tools=["search_flights", "book_flight"]
+    )
+    
+    metrics = [
+        ToolCorrectnessMetric(model="gpt-4o-mini", threshold=0.8),
+        TaskSuccessRateMetric(
+            model="gpt-4o-mini",
+            threshold=0.7,
+            temperature=1.0 
+        )
+    ]
+    
+    results = await evaluate(
+        test_cases=[test_cases],
+        metrics=metrics,
+        verbose=True
+    )
+    return results
+
+asyncio.run(evaluate_agent())
+```
+
+### Conversational Evaluation
+```python
+from eval_lib import (
+    evaluate_conversations,
+    ConversationalEvalTestCase,
+    EvalTestCase,
+    RoleAdherenceMetric,
+    KnowledgeRetentionMetric
+)
+
+async def evaluate_conversation():
+    # Create conversations
+    conversations = [
+        ConversationalEvalTestCase(
+            chatbot_role="You are a professional customer support assistant.",
+            turns=[
+                EvalTestCase(
+                    input="I need help with my order",
+                    actual_output="I'd be happy to help. Could you provide your order number?"
+                ),
+                EvalTestCase(
+                    input="It's #12345",
+                    actual_output="Thank you! Let me look up order #12345 for you."
+                ),
+                EvalTestCase(
+                    input="When will it arrive?",
+                    actual_output="Your order will be delivered on October 27, 2025."
+                ),
+            ]
+        ),
+        ConversationalEvalTestCase(
+            chatbot_role="You are a formal financial advisor.",
+            turns=[
+                EvalTestCase(
+                    input="Should I invest in stocks?",
+                    actual_output="Yo dude! Just YOLO into stocks!"
+                ),
+                EvalTestCase(
+                    input="What about bonds?",
+                    actual_output="Bonds are boring, bro!"
+                ),
+            ]
+        ),
+        ConversationalEvalTestCase(
+            chatbot_role="You are a helpful assistant.",
+            turns=[
+                EvalTestCase(
+                    input="My name is John",
+                    actual_output="Nice to meet you, John!"
+                ),
+                EvalTestCase(
+                    input="What's my name?",
+                    actual_output="Your name is John."
+                ),
+                EvalTestCase(
+                    input="Where do I live?",
+                    actual_output="I don't have that information."
+                ),
+            ]
+        ),
+    ]
+    
+    # Define conversational metrics
+    metrics = [
+        TaskSuccessRateMetric(
+            model="gpt-4o-mini",
+            threshold=0.7,
+            temperature=0.9,
+        ),
+        RoleAdherenceMetric(
+            model="gpt-4o-mini",
+            threshold=0.8,
+            temperature=0.5,
+        ),
+        KnowledgeRetentionMetric(
+            model="gpt-4o-mini",
+            threshold=0.7,
+            temperature=0.5,
+        ),
+    ]
+    
+    # Run batch evaluation
+    results = await evaluate_conversations(
+        conv_cases=conversations,
+        metrics=metrics,
+        verbose=True
+    )
+    
+    return results
+
+asyncio.run(evaluate_conversation())
+```
+
+## Available Metrics
+
+### RAG Metrics
+
+#### AnswerRelevancyMetric
+Measures how relevant the answer is to the question using multi-step evaluation:
+1. Infers user intent
+2. Extracts atomic statements from answer
+3. Generates verdicts (fully/mostly/partial/minor/none) for each statement
+4. Aggregates using softmax
+```python
+metric = AnswerRelevancyMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    temperature=0.5  # Controls aggregation strictness
+)
+```
+
+#### FaithfulnessMetric
+Checks if the answer is faithful to the provided context:
+1. Extracts factual claims from answer
+2. Verifies each claim against context (fully/mostly/partial/minor/none)
+3. Aggregates faithfulness score
+```python
+metric = FaithfulnessMetric(
+    model="gpt-4o-mini",
+    threshold=0.8,
+    temperature=0.5
+)
+```
+
+#### ContextualRelevancyMetric
+Evaluates relevance of retrieved context to the question.
+```python
+metric = ContextualRelevancyMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    temperature=0.5
+)
+```
+
+#### ContextualPrecisionMetric
+Measures precision of context retrieval - are the retrieved chunks relevant?
+```python
+metric = ContextualPrecisionMetric(
+    model="gpt-4o-mini",
+    threshold=0.7
+)
+```
+
+#### ContextualRecallMetric
+Measures recall of relevant context - was all relevant information retrieved?
+```python
+metric = ContextualRecallMetric(
+    model="gpt-4o-mini",
+    threshold=0.7
+)
+```
+
+#### BiasMetric
+Detects bias and prejudice in AI-generated output. Score range: 0 (strong bias) and 100 (no bias).
+```python
+metric = BiasMetric(
+    model="gpt-4o-mini",
+    threshold=1.0  # Score range: 0 or 100
+)
+```
+
+#### ToxicityMetric
+Identifies toxic content in responses. Score range: 0 (highly toxic) and 100 (no toxicity).
+```python
+metric = ToxicityMetric(
+    model="gpt-4o-mini",
+    threshold=1.0  # Score range: 0 or 100
+)
+```
+
+#### RestrictedRefusalMetric
+Checks if the AI appropriately refuses harmful or out-of-scope requests.
+```python
+metric = RestrictedRefusalMetric(
+    model="gpt-4o-mini",
+    threshold=0.7
+)
+```
+
+### Agent Metrics
+
+#### ToolCorrectnessMetric
+Validates that the agent calls the correct tools in the right sequence.
+```python
+metric = ToolCorrectnessMetric(
+    model="gpt-4o-mini",
+    threshold=0.8
+)
+```
+
+#### TaskSuccessRateMetric
+````
+**Note:** The metric automatically detects if the conversation contains links/URLs and adds "The user got the link to the requested resource" as an evaluation criterion only when links are present in the dialogue.
+````
+Measures task completion success across conversation:
+1. Infers user's goal
+2. Generates success criteria
+3. Evaluates each criterion (fully/mostly/partial/minor/none)
+4. Aggregates into final score
+```python
+metric = TaskSuccessRateMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    temperature=1.0  # Higher = more lenient aggregation
+)
+```
+
+#### RoleAdherenceMetric
+Evaluates how well the agent maintains its assigned role:
+1. Compares each response against role description
+2. Generates adherence verdicts (fully/mostly/partial/minor/none)
+3. Aggregates across all turns
+```python
+metric = RoleAdherenceMetric(
+    model="gpt-4o-mini",
+    threshold=0.8,
+    temperature=0.5,
+    chatbot_role="You are helpfull assistant" # Set role here directly
+)
+
+```
+
+#### KnowledgeRetentionMetric
+Checks if the agent remembers and recalls information from earlier in the conversation:
+1. Analyzes conversation for retention quality
+2. Generates retention verdicts (fully/mostly/partial/minor/none)
+3. Aggregates into retention score
+```python
+metric = KnowledgeRetentionMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    temperature=0.5
+)
+```
+
+### Custom & Advanced Metrics
+
+#### GEval
+State-of-the-art evaluation using probability-weighted scoring from the [G-Eval paper](https://arxiv.org/abs/2303.16634):
+- **Auto Chain-of-Thought**: Automatically generates evaluation steps from criteria
+- **Probability-Weighted Scoring**: score = Σ p(si) × si using 20 samples
+- **Fine-Grained Scores**: Continuous scores (e.g., 73.45) instead of integers
+```python
+metric = GEval(
+    model="gpt-4o",  # Best with GPT-4 for probability estimation
+    threshold=0.7,
+    name="Coherence",
+    criteria="Evaluate logical flow and structure of the response",
+    evaluation_steps=None,  # Auto-generated if not provided
+    n_samples=20,  # Number of samples for probability estimation
+    sampling_temperature=2.0  # High temperature for diverse sampling
+)
+```
+
+#### CustomEvalMetric
+Verdict-based custom evaluation with automatic criteria generation.
+Automatically:
+- Generates 3-5 specific sub-criteria from main criteria (1 LLM call)
+- Evaluates each criterion with verdicts (fully/mostly/partial/minor/none)
+- Aggregates using softmax (temperature-controlled)
+Total: 1-2 LLM calls
+
+Usage:
+```python
+metric = CustomEvalMetric(
+    model="gpt-4o-mini",
+    threshold=0.8,
+    name="Code Quality",
+    criteria="Evaluate code readability, efficiency, and best practices",
+    evaluation_steps=None,  # Auto-generated if not provided
+    temperature=0.8,  # Controls verdict aggregation (0.1=strict, 1.0=lenient)
+    verbose=True
+)
+
+```
+
+**Example with manual criteria:**
+```python
+metric = CustomEvalMetric(
+    model="gpt-4o-mini",
+    threshold=0.8,
+    name="Child-Friendly Explanation",
+    criteria="Evaluate if explanation is appropriate for a 10-year-old",
+    evaluation_steps=[  # Manual criteria for precise control
+        "Uses simple vocabulary appropriate for 10-year-olds",
+        "Includes relatable analogies or comparisons",
+        "Avoids complex technical jargon",
+        "Explanation is engaging and interesting",
+        "Concept is broken down into understandable parts"
+    ],
+    temperature=0.8,
+    verbose=True
+)
+
+result = await metric.evaluate(test_case)
+```
+
+## Understanding Evaluation Results
+
+### Score Ranges
+
+All metrics use a normalized score range of **0.0 to 1.0**:
+- **0.0**: Complete failure / Does not meet criteria
+- **0.5**: Partial satisfaction / Mixed results
+- **1.0**: Perfect / Fully meets criteria
+
+**Score Interpretation:**
+- **0.8 - 1.0**: Excellent
+- **0.7 - 0.8**: Good (typical threshold)
+- **0.5 - 0.7**: Acceptable with issues
+- **0.0 - 0.5**: Poor / Needs improvement
+
+## Verbose Mode
+
+All metrics support a `verbose` parameter that controls output formatting:
+
+### verbose=False (Default) - JSON Output
+Returns simple dictionary with results:
+```python
+metric = AnswerRelevancyMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    verbose=False  # Default
+)
+
+result = await metric.evaluate(test_case)
+print(result)
+# Output: Simple dictionary
+# {
+#   'name': 'answerRelevancyMetric',
+#   'score': 0.85,
+#   'success': True,
+#   'reason': 'Answer is highly relevant...',
+#   'evaluation_cost': 0.000234,
+#   'evaluation_log': {...}
+# }
+```
+
+### verbose=True - Beautiful Console Output
+Displays formatted results with colors, progress bars, and detailed logs:
+```python
+metric = CustomEvalMetric(
+    model="gpt-4o-mini",
+    threshold=0.9,
+    name="Factual Accuracy",
+    criteria="Evaluate the factual accuracy of the response",
+    verbose=True  # Enable beautiful output
+)
+
+result = await metric.evaluate(test_case)
+# Output: Beautiful formatted display (see image below)
+```
+
+**Console Output with verbose=True:**
+
+**Console Output with verbose=True:**
+```
+╔════════════════════════════════════════════════════════════════╗
+║                    📊answerRelevancyMetric                     ║
+╚════════════════════════════════════════════════════════════════╝
+
+Status: ✅ PASSED
+Score:  0.91 [███████████████████████████░░░] 91%
+Cost:   💰 $0.000178
+Reason:
+  The answer correctly identifies Paris as the capital of France, demonstrating a clear understanding of the
+  user's request. However, it fails to provide a direct and explicit response, which diminishes its overall
+  effectiveness.
+
+Evaluation Log:
+╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ {                                                                                                             │
+│   "input_question": "What is the capital of France?",                                                         │
+│   "answer": "The capital of France is Paris and it is a beautiful city and known for its art and culture.",   │
+│   "user_intent": "The user is seeking information about the capital city of France.",                         │
+│   "comment_user_intent": "Inferred goal of the question.",                                                    │
+│   "statements": [                                                                                             │
+│     "The capital of France is Paris.",                                                                        │
+│     "Paris is a beautiful city.",                                                                             │
+│     "Paris is known for its art and culture."                                                                 │
+│   ],                                                                                                          │
+│   "comment_statements": "Atomic facts extracted from the answer.",                                            │
+│   "verdicts": [                                                                                               │
+│     {                                                                                                         │
+│       "verdict": "fully",                                                                                     │
+│       "reason": "The statement explicitly answers the user's question about the capital of France."           │
+│     },                                                                                                        │
+│     {                                                                                                         │
+│       "verdict": "minor",                                                                                     │
+│       "reason": "While it mentions Paris, it does not directly answer the user's question."                   │
+│     },                                                                                                        │
+│     {                                                                                                         │
+│       "verdict": "minor",                                                                                     │
+│       "reason": "This statement is related to Paris but does not address the user's question about the        │
+│ capital."                                                                                                     │
+│     }                                                                                                         │
+│   ],                                                                                                          │
+│   "comment_verdicts": "Each verdict explains whether a statement is relevant to the question.",               │
+│   "verdict_score": 0.9142,                                                                                    │
+│   "comment_verdict_score": "Proportion of relevant statements in the answer.",                                │
+│   "final_score": 0.9142,                                                                                      │
+│   "comment_final_score": "Score based on the proportion of relevant statements.",                             │
+│   "threshold": 0.7,                                                                                           │
+│   "success": true,                                                                                            │
+│   "comment_success": "Whether the score exceeds the pass threshold.",                                         │
+│   "final_reason": "The answer correctly identifies Paris as the capital of France, demonstrating a clear      │
+│ understanding of the user's request. However, it fails to provide a direct and explicit response, which       │
+│ diminishes its overall effectiveness.",                                                                       │
+│   "comment_reasoning": "Compressed explanation of the key verdict rationales."                                │
+│ }                                                                                                             │
+╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+Features:
+- ✅ Color-coded status (✅ PASSED / ❌ FAILED)
+- 📊 Visual progress bar for scores
+- 💰 Cost tracking display
+- 📝 Formatted reason with word wrapping
+- 📋 Pretty-printed evaluation log in bordered box
+
+**When to use verbose=True:**
+- Interactive development and testing
+- Debugging evaluation issues
+- Presentations and demonstrations
+- Manual review of results
+
+**When to use verbose=False:**
+- Production environments
+- Batch processing
+- Automated testing
+- When storing results in databases
+
+---
+
+## Working with Results
+
+Results are returned as simple dictionaries. Access fields directly:
+```python
+# Run evaluation
+result = await metric.evaluate(test_case)
+
+# Access result fields
+score = result['score']              # 0.0-1.0
+success = result['success']          # True/False
+reason = result['reason']            # String explanation
+cost = result['evaluation_cost']     # USD amount
+log = result['evaluation_log']       # Detailed breakdown
+
+# Example: Check success and print score
+if result['success']:
+    print(f"✅ Passed with score: {result['score']:.2f}")
+else:
+    print(f"❌ Failed: {result['reason']}")
+    
+# Access detailed verdicts (for verdict-based metrics)
+if 'verdicts' in result['evaluation_log']:
+    for verdict in result['evaluation_log']['verdicts']:
+        print(f"- {verdict['verdict']}: {verdict['reason']}")
+```
+
+## Temperature Parameter
+
+Many metrics use a **temperature** parameter for score aggregation (via temperature-weighted scoring):
+
+- **Lower (0.1-0.3)**: **STRICT** - All scores matter equally, low scores heavily penalize the final result. Best for critical applications where even one bad verdict should fail the metric.
+- **Medium (0.4-0.6)**: **BALANCED** - Moderate weighting between high and low scores. Default behavior for most use cases (default: 0.5).
+- **Higher (0.7-1.0)**: **LENIENT** - High scores (fully/mostly) dominate, effectively ignoring partial/minor/none verdicts. Best for exploratory evaluation or when you want to focus on positive signals.
+
+**How it works:** Temperature controls exponential weighting of scores. Higher temperature exponentially boosts high scores (1.0, 0.9), making low scores (0.7, 0.3, 0.0) matter less. Lower temperature treats all scores more equally.
+
+**Example:**
+```python
+# Verdicts: [fully, mostly, partial, minor, none] = [1.0, 0.9, 0.7, 0.3, 0.0]
+
+# STRICT: All verdicts count
+metric = FaithfulnessMetric(temperature=0.1)  
+# Result: ~0.52 (heavily penalized by "minor" and "none")
+
+# BALANCED: Moderate weighting
+metric = AnswerRelevancyMetric(temperature=0.5)  
+# Result: ~0.73 (balanced consideration)
+
+# LENIENT: Only "fully" and "mostly" matter
+metric = TaskSuccessRateMetric(temperature=1.0)  
+# Result: ~0.95 (ignores "partial", "minor", "none")
+```
+
+## LLM Provider Configuration
+
+### OpenAI
+```python
+import os
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+
+from eval_lib import chat_complete
+
+response, cost = await chat_complete(
+    "gpt-4o-mini",  # or "openai:gpt-4o-mini"
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+```
+
+### Azure OpenAI
+```python
+os.environ["AZURE_OPENAI_API_KEY"] = "your-api-key"
+os.environ["AZURE_OPENAI_ENDPOINT"] = "https://your-endpoint.openai.azure.com/"
+os.environ["AZURE_OPENAI_DEPLOYMENT"] = "your-deployment-name"
+
+response, cost = await chat_complete(
+    "azure:gpt-4o",
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+```
+
+### Google Gemini
+```python
+os.environ["GOOGLE_API_KEY"] = "your-api-key"
+
+response, cost = await chat_complete(
+    "google:gemini-2.0-flash",
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+```
+
+### Anthropic Claude
+```python
+os.environ["ANTHROPIC_API_KEY"] = "your-api-key"
+
+response, cost = await chat_complete(
+    "anthropic:claude-sonnet-4-0",
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+```
+
+### Ollama (Local)
+```python
+os.environ["OLLAMA_API_KEY"] = "ollama"  # Can be any value
+os.environ["OLLAMA_API_BASE_URL"] = "http://localhost:11434/v1"
+
+response, cost = await chat_complete(
+    "ollama:llama2",
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+```
+
+## Test Data Generation
+
+The library includes a powerful test data generator that can create realistic test cases either from scratch or based on your documents.
+
+### Supported Document Formats
+
+- **Documents**: PDF, DOCX, DOC, TXT, RTF, ODT
+- **Structured Data**: CSV, TSV, XLSX, JSON, YAML, XML
+- **Web**: HTML, Markdown
+- **Presentations**: PPTX
+- **Images**: PNG, JPG, JPEG (with OCR support)
+
+### Generate from Scratch
+```python
+from eval_lib.datagenerator.datagenerator import DatasetGenerator
+
+generator = DatasetGenerator(
+    model="gpt-4o-mini",
+    agent_description="A customer support chatbot",
+    input_format="User question or request",
+    expected_output_format="Helpful response",
+    test_types=["functionality", "edge_cases"],
+    max_rows=20,
+    question_length="mixed",  # "short", "long", or "mixed"
+    question_openness="mixed",  # "open", "closed", or "mixed"
+    trap_density=0.1,  # 10% trap questions
+    language="en",
+    verbose=True  # Displays beautiful formatted progress, statistics and full dataset preview
+)
+
+dataset = await generator.generate_from_scratch()
+```
+
+### Generate from Documents
+```python
+generator = DatasetGenerator(
+    model="gpt-4o-mini",
+    agent_description="Technical support agent",
+    input_format="Technical question",
+    expected_output_format="Detailed answer with references",
+    test_types=["retrieval", "accuracy"],
+    max_rows=50,
+    chunk_size=1024,
+    chunk_overlap=100,
+    max_chunks=30,
+    verbose=True
+)
+
+file_paths = ["docs/user_guide.pdf", "docs/faq.md"]
+dataset = await generator.generate_from_documents(file_paths)
+
+# Convert to test cases
+from eval_lib import EvalTestCase
+test_cases = [
+    EvalTestCase(
+        input=item["input"],
+        expected_output=item["expected_output"],
+        retrieval_context=[item.get("context", "")]
+    )
+    for item in dataset
+]
+```
+
+## Best Practices
+
+### 1. Choose the Right Model
+
+- **G-Eval**: Use GPT-4 for best results with probability-weighted scoring
+- **Other Metrics**: GPT-4o-mini is cost-effective and sufficient
+- **Custom Eval**: Use GPT-4 for complex criteria, GPT-4o-mini for simple ones
+
+### 2. Set Appropriate Thresholds
+```python
+# Safety metrics - high bar
+BiasMetric(threshold=0.8)
+ToxicityMetric(threshold=0.85)
+
+# Quality metrics - moderate bar
+AnswerRelevancyMetric(threshold=0.7)
+FaithfulnessMetric(threshold=0.75)
+
+# Agent metrics - context-dependent
+TaskSuccessRateMetric(threshold=0.7)  # Most tasks
+RoleAdherenceMetric(threshold=0.9)  # Strict role requirements
+```
+
+### 3. Use Temperature Wisely
+```python
+# STRICT evaluation - critical applications where all verdicts matter
+# Use when: You need high accuracy and can't tolerate bad verdicts
+metric = FaithfulnessMetric(temperature=0.1)
+
+# BALANCED - general use (default)
+# Use when: Standard evaluation with moderate requirements
+metric = AnswerRelevancyMetric(temperature=0.5)
+
+# LENIENT - exploratory evaluation or focusing on positive signals
+# Use when: You want to reward good answers and ignore occasional mistakes
+metric = TaskSuccessRateMetric(temperature=1.0)
+```
+
+**Real-world examples:**
+```python
+# Production RAG system - must be accurate
+faithfulness = FaithfulnessMetric(
+    model="gpt-4o-mini",
+    threshold=0.8,
+    temperature=0.2  # STRICT: verdicts "none", "minor", "partially" significantly impact score
+)
+
+# Customer support chatbot - moderate standards
+role_adherence = RoleAdherenceMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    temperature=0.5  # BALANCED: Standard evaluation
+)
+
+# Experimental feature testing - focus on successes
+task_success = TaskSuccessRateMetric(
+    model="gpt-4o-mini",
+    threshold=0.6,
+    temperature=1.0  # LENIENT: Focuses on "fully" and "mostly" completions
+)
+```
+
+### 4. Leverage Evaluation Logs
+```python
+# Enable verbose mode for automatic detailed display
+metric = AnswerRelevancyMetric(
+    model="gpt-4o-mini",
+    threshold=0.7,
+    verbose=True  # Automatic formatted output with full logs
+)
+
+# Or access logs programmatically
+result = await metric.evaluate(test_case)
+log = result['evaluation_log']
+
+# Debugging failures
+if not result['success']:
+    # All details available in log
+    reason = result['reason']
+    verdicts = log.get('verdicts', [])
+    steps = log.get('evaluation_steps', [])
+```
+
+### 5. Batch Evaluation for Efficiency
+```python
+# Evaluate multiple test cases at once
+results = await evaluate(
+    test_cases=[test_case1, test_case2, test_case3],
+    metrics=[metric1, metric2, metric3]
+)
+
+# Calculate aggregate statistics
+total_cost = sum(
+    metric.evaluation_cost or 0
+    for _, test_results in results
+    for result in test_results
+    for metric in result.metrics_data
+)
+
+success_rate = sum(
+    1 for _, test_results in results
+    for result in test_results
+    if result.success
+) / len(results)
+
+print(f"Total cost: ${total_cost:.4f}")
+print(f"Success rate: {success_rate:.2%}")
+```
+
+
+## Environment Variables
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `OPENAI_API_KEY` | OpenAI API key | For OpenAI |
+| `AZURE_OPENAI_API_KEY` | Azure OpenAI API key | For Azure |
+| `AZURE_OPENAI_ENDPOINT` | Azure OpenAI endpoint URL | For Azure |
+| `AZURE_OPENAI_DEPLOYMENT` | Azure deployment name | For Azure |
+| `GOOGLE_API_KEY` | Google API key | For Google |
+| `ANTHROPIC_API_KEY` | Anthropic API key | For Anthropic |
+| `OLLAMA_API_KEY` | Ollama API key | For Ollama |
+| `OLLAMA_API_BASE_URL` | Ollama base URL | For Ollama |
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Citation
+
+If you use this library in your research, please cite:
+```bibtex
+@software{eval_ai_library,
+  author = {Meshkov, Aleksandr},
+  title = {Eval AI Library: Comprehensive AI Model Evaluation Framework},
+  year = {2025},
+  url = {https://github.com/meshkovQA/Eval-ai-library.git}
+}
+```
+
+### References
+
+This library implements techniques from:
+```bibtex
+@inproceedings{liu2023geval,
+  title={G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
+  author={Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang},
+  booktitle={Proceedings of EMNLP},
+  year={2023}
+}
+```
+
+## Support
+
+- 📧 Email: alekslynx90@gmail.com
+- 🐛 Issues: [GitHub Issues](https://github.com/meshkovQA/Eval-ai-library.git/issues)
+- 📖 Documentation: [Full Documentation](https://github.com/meshkovQA/Eval-ai-library.git#readme)
+
+## Acknowledgments
+
+This library was developed to provide a comprehensive solution for evaluating AI models across different use cases and providers, with state-of-the-art techniques including G-Eval's probability-weighted scoring and automatic chain-of-thought generation.