eval-protocol 0.2.26__tar.gz → 0.2.27__tar.gz

{eval_protocol-0.2.26/eval_protocol.egg-info → eval_protocol-0.2.27}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-protocol
-Version: 0.2.26
+Version: 0.2.27
 Summary: The official Python SDK for Eval Protocol (EP.) EP is an open protocol that standardizes how developers author evals for large language model (LLM) applications.
 Author-email: Fireworks AI <info@fireworks.ai>
 License-Expression: MIT
@@ -119,133 +119,114 @@ Dynamic: license-file
 # Eval Protocol (EP)
 
 [![PyPI - Version](https://img.shields.io/pypi/v/eval-protocol)](https://pypi.org/project/eval-protocol/)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/eval-protocol/python-sdk)
 
-**The open-source toolkit for building your internal model leaderboard.**
+**Stop guessing which AI model to use. Build a data-driven model leaderboard.**
 
-When you have multiple AI models to choose from—different versions, providers,
-or configurations—how do you know which one is best for your use case?
+With hundreds of models and configs, you need objective data to choose the right one for your use case. EP helps you evaluate real traces, compare models, and visualize results locally.
 
 ## 🚀 Features
 
-- **Custom Evaluations**: Write evaluations tailored to your specific business needs
-- **Auto-Evaluation**: Stack-rank models using LLMs as judges with just model traces using out-of-the-box evaluators
-- **RL Environments via MCP**: Build reinforcement learning environments using the Model Control Protocol (MCP) to simulate user interactions and advanced evaluation scenarios
-- **Consistent Testing**: Test across various models and configurations with a unified framework
-- **Resilient Runtime**: Automatic retries for unstable LLM APIs and concurrent execution for long-running evaluations
-- **Rich Visualizations**: Built-in pivot tables and visualizations for result analysis
-- **Data-Driven Decisions**: Make informed model deployment decisions based on comprehensive evaluation results
+- **Pytest authoring**: `@evaluation_test` decorator to configure evaluations
+- **Robust rollouts**: Handles flaky LLM APIs and parallel execution
+- **Integrations**: Works with Langfuse, LangSmith, Braintrust, Responses API
+- **Agent support**: LangGraph and Pydantic AI
+- **MCP RL envs**: Build reinforcement learning environments with MCP
+- **Built-in benchmarks**: AIME, tau-bench
+- **LLM judge**: Stack-rank models using pairwise Arena-Hard-Auto
+- **Local UI**: Pivot/table views for real-time analysis
 
-## Quick Examples
+## ⚡ Quickstart (no labels needed)
 
-### Basic Model Comparison
+Install with your tracing platform extras and set API keys:
 
-Compare models on a simple formatting task:
-
-```python test_bold_format.py
-from eval_protocol.models import EvaluateResult, EvaluationRow, Message
-from eval_protocol.pytest import default_single_turn_rollout_processor, evaluation_test
+```bash
+pip install 'eval-protocol[langfuse]'
 
-@evaluation_test(
-    input_messages=[
-        [
-            Message(role="system", content="Use bold text to highlight important information."),
-            Message(role="user", content="Explain why evaluations matter for AI agents. Make it dramatic!"),
-        ],
-    ],
-    completion_params=[
-        {"model": "fireworks/accounts/fireworks/models/llama-v3p1-8b-instruct"},
-        {"model": "openai/gpt-4"},
-        {"model": "anthropic/claude-3-sonnet"}
-    ],
-    rollout_processor=default_single_turn_rollout_processor,
-    mode="pointwise",
-)
-def test_bold_format(row: EvaluationRow) -> EvaluationRow:
-    """Check if the model's response contains bold text."""
-    assistant_response = row.messages[-1].content
+# Model API keys (set what you need)
+export OPENAI_API_KEY=...
+export FIREWORKS_API_KEY=...
+export GEMINI_API_KEY=...
 
-    if assistant_response is None:
-        row.evaluation_result = EvaluateResult(score=0.0, reason="No response")
-        return row
+# Platform keys
+export LANGFUSE_PUBLIC_KEY=...
+export LANGFUSE_SECRET_KEY=...
+export LANGFUSE_HOST=https://your-deployment.com  # optional
+```
 
-    has_bold = "**" in str(assistant_response)
-    score = 1.0 if has_bold else 0.0
-    reason = "Contains bold text" if has_bold else "No bold text found"
+Minimal evaluation using the built-in AHA judge:
 
-    row.evaluation_result = EvaluateResult(score=score, reason=reason)
-    return row
-```
+```python
+from datetime import datetime
+import pytest
+
+from eval_protocol import (
+    evaluation_test,
+    aha_judge,
+    EvaluationRow,
+    SingleTurnRolloutProcessor,
+    DynamicDataLoader,
+    create_langfuse_adapter,
+)
 
-### Using Datasets
 
-Evaluate models on existing datasets:
+def langfuse_data_generator() -> list[EvaluationRow]:
+    adapter = create_langfuse_adapter()
+    return adapter.get_evaluation_rows(
+        to_timestamp=datetime.utcnow(),
+        limit=20,
+        sample_size=5,
+    )
 
-```python
-from eval_protocol.pytest import evaluation_test
-from eval_protocol.adapters.huggingface import create_gsm8k_adapter
 
-@evaluation_test(
-    input_dataset=["development/gsm8k_sample.jsonl"],  # Local JSONL file
-    dataset_adapter=create_gsm8k_adapter(),  # Adapter to convert data
-    completion_params=[
-        {"model": "openai/gpt-4"},
-        {"model": "anthropic/claude-3-sonnet"}
+@pytest.mark.parametrize(
+    "completion_params",
+    [
+        {"model": "openai/gpt-4.1"},
+        {"model": "fireworks_ai/accounts/fireworks/models/gpt-oss-120b"},
     ],
-    mode="pointwise"
 )
-def test_math_reasoning(row: EvaluationRow) -> EvaluationRow:
-    # Your evaluation logic here
-    return row
+@evaluation_test(
+    data_loaders=DynamicDataLoader(generators=[langfuse_data_generator]),
+    rollout_processor=SingleTurnRolloutProcessor(),
+)
+async def test_llm_judge(row: EvaluationRow) -> EvaluationRow:
+    return await aha_judge(row)
 ```
 
+Run it:
 
-## 📚 Resources
+```bash
+pytest -q -s
+```
 
-- **[Documentation](https://evalprotocol.io)** - Complete guides and API reference
-- **[Discord](https://discord.com/channels/1137072072808472616/1400975572405850155)** - Community discussions
-- **[GitHub](https://github.com/eval-protocol/python-sdk)** - Source code and examples
+The pytest output includes local links for a leaderboard and row-level traces (pivot/table) at `http://localhost:8000`.
 
 ## Installation
 
-**This library requires Python >= 3.10.**
+This library requires Python >= 3.10.
 
-### Basic Installation
-
-Install with pip:
+### pip
 
 ```bash
 pip install eval-protocol
 ```
 
-### Recommended Installation with uv
-
-For better dependency management and faster installs, we recommend using [uv](https://docs.astral.sh/uv/):
+### uv (recommended)
 
 ```bash
-# Install uv if you haven't already
+# Install uv (if needed)
 curl -LsSf https://astral.sh/uv/install.sh | sh
 
-# Install eval-protocol
+# Add to your project
 uv add eval-protocol
 ```
 
-### Optional Dependencies
-
-Install with additional features:
-
-```bash
-# For Langfuse integration
-pip install 'eval-protocol[langfuse]'
-
-# For HuggingFace datasets
-pip install 'eval-protocol[huggingface]'
-
-# For all adapters
-pip install 'eval-protocol[adapters]'
+## 📚 Resources
 
-# For development
-pip install 'eval-protocol[dev]'
-```
+- **[Documentation](https://evalprotocol.io)** – Guides and API reference
+- **[Discord](https://discord.com/channels/1137072072808472616/1400975572405850155)** – Community
+- **[GitHub](https://github.com/eval-protocol/python-sdk)** – Source and examples
 
 ## License
 

eval_protocol-0.2.27/README.md

@@ -0,0 +1,115 @@
+# Eval Protocol (EP)
+
+[![PyPI - Version](https://img.shields.io/pypi/v/eval-protocol)](https://pypi.org/project/eval-protocol/)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/eval-protocol/python-sdk)
+
+**Stop guessing which AI model to use. Build a data-driven model leaderboard.**
+
+With hundreds of models and configs, you need objective data to choose the right one for your use case. EP helps you evaluate real traces, compare models, and visualize results locally.
+
+## 🚀 Features
+
+- **Pytest authoring**: `@evaluation_test` decorator to configure evaluations
+- **Robust rollouts**: Handles flaky LLM APIs and parallel execution
+- **Integrations**: Works with Langfuse, LangSmith, Braintrust, Responses API
+- **Agent support**: LangGraph and Pydantic AI
+- **MCP RL envs**: Build reinforcement learning environments with MCP
+- **Built-in benchmarks**: AIME, tau-bench
+- **LLM judge**: Stack-rank models using pairwise Arena-Hard-Auto
+- **Local UI**: Pivot/table views for real-time analysis
+
+## ⚡ Quickstart (no labels needed)
+
+Install with your tracing platform extras and set API keys:
+
+```bash
+pip install 'eval-protocol[langfuse]'
+
+# Model API keys (set what you need)
+export OPENAI_API_KEY=...
+export FIREWORKS_API_KEY=...
+export GEMINI_API_KEY=...
+
+# Platform keys
+export LANGFUSE_PUBLIC_KEY=...
+export LANGFUSE_SECRET_KEY=...
+export LANGFUSE_HOST=https://your-deployment.com  # optional
+```
+
+Minimal evaluation using the built-in AHA judge:
+
+```python
+from datetime import datetime
+import pytest
+
+from eval_protocol import (
+    evaluation_test,
+    aha_judge,
+    EvaluationRow,
+    SingleTurnRolloutProcessor,
+    DynamicDataLoader,
+    create_langfuse_adapter,
+)
+
+
+def langfuse_data_generator() -> list[EvaluationRow]:
+    adapter = create_langfuse_adapter()
+    return adapter.get_evaluation_rows(
+        to_timestamp=datetime.utcnow(),
+        limit=20,
+        sample_size=5,
+    )
+
+
+@pytest.mark.parametrize(
+    "completion_params",
+    [
+        {"model": "openai/gpt-4.1"},
+        {"model": "fireworks_ai/accounts/fireworks/models/gpt-oss-120b"},
+    ],
+)
+@evaluation_test(
+    data_loaders=DynamicDataLoader(generators=[langfuse_data_generator]),
+    rollout_processor=SingleTurnRolloutProcessor(),
+)
+async def test_llm_judge(row: EvaluationRow) -> EvaluationRow:
+    return await aha_judge(row)
+```
+
+Run it:
+
+```bash
+pytest -q -s
+```
+
+The pytest output includes local links for a leaderboard and row-level traces (pivot/table) at `http://localhost:8000`.
+
+## Installation
+
+This library requires Python >= 3.10.
+
+### pip
+
+```bash
+pip install eval-protocol
+```
+
+### uv (recommended)
+
+```bash
+# Install uv (if needed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Add to your project
+uv add eval-protocol
+```
+
+## 📚 Resources
+
+- **[Documentation](https://evalprotocol.io)** – Guides and API reference
+- **[Discord](https://discord.com/channels/1137072072808472616/1400975572405850155)** – Community
+- **[GitHub](https://github.com/eval-protocol/python-sdk)** – Source and examples
+
+## License
+
+[MIT](LICENSE)

{eval_protocol-0.2.26 → eval_protocol-0.2.27}/eval_protocol/__init__.py

@@ -22,6 +22,7 @@ from .mcp_env import (
     rollout,
     test_mcp,
 )
+from .data_loader import DynamicDataLoader, InlineDataLoader
 
 # Try to import FireworksPolicy if available
 try:
@@ -66,6 +67,8 @@ warnings.filterwarnings("default", category=DeprecationWarning, module="eval_pro
 
 __all__ = [
     "DefaultParameterIdGenerator",
+    "DynamicDataLoader",
+    "InlineDataLoader",
     "aha_judge",
     "multi_turn_assistant_to_ground_truth",
     "assistant_to_ground_truth",

{eval_protocol-0.2.26 → eval_protocol-0.2.27}/eval_protocol/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2025-09-23T16:59:02-0700",
+ "date": "2025-09-23T23:32:05-0700",
  "dirty": false,
  "error": null,
- "full-revisionid": "bcc2d7c22085ccddf46d952e3481012250245d90",
- "version": "0.2.26"
+ "full-revisionid": "aadaf82ad89ca3a8941a348ff818684554f5bf0c",
+ "version": "0.2.27"
 }
 '''  # END VERSION_JSON
 

eval_protocol-0.2.27/eval_protocol/data_loader/__init__.py

@@ -0,0 +1,4 @@
+from .dynamic_data_loader import DynamicDataLoader
+from .inline_data_loader import InlineDataLoader
+
+__all__ = ["DynamicDataLoader", "InlineDataLoader"]

eval_protocol-0.2.27/eval_protocol/data_loader/dynamic_data_loader.py

@@ -0,0 +1,38 @@
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+
+from eval_protocol.data_loader.models import (
+    DataLoaderResult,
+    DataLoaderVariant,
+    EvaluationDataLoader,
+)
+from eval_protocol.models import EvaluationRow
+
+
+@dataclass(kw_only=True)
+class DynamicDataLoader(EvaluationDataLoader):
+    """Data loader for dynamic data generation."""
+
+    generators: Sequence[Callable[[], list[EvaluationRow]]]
+    """Dynamic data generation functions. These callables are invoked each time data
+    needs to be loaded, allowing for dynamic data generation, lazy loading, or data that
+    changes between evaluation runs. Each function should return a list of EvaluationRow
+    objects. This is useful for scenarios like generating test data on-the-fly, loading
+    data from external sources, or creating data with randomized elements for robust testing."""
+
+    def variants(self) -> Sequence[DataLoaderVariant]:
+        variants: Sequence[DataLoaderVariant] = []
+        for generator in self.generators:
+
+            def _load() -> DataLoaderResult:
+                resolved_rows = generator()
+                return DataLoaderResult(
+                    rows=resolved_rows,
+                    type=self.__class__.__name__,
+                    variant_id=generator.__name__,
+                    variant_description=generator.__doc__,
+                )
+
+            variants.append(_load)
+
+        return variants

eval_protocol-0.2.27/eval_protocol/data_loader/factory_data_loader.py

@@ -0,0 +1,38 @@
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+
+from eval_protocol.data_loader.models import (
+    DataLoaderResult,
+    DataLoaderVariant,
+    EvaluationDataLoader,
+)
+from eval_protocol.models import EvaluationRow
+
+
+@dataclass(kw_only=True)
+class DynamicDataLoader(EvaluationDataLoader):
+    """Data loader for dynamic data generation."""
+
+    factory: Sequence[Callable[[], list[EvaluationRow]]]
+    """Dynamic data generation functions. These callables are invoked each time data
+    needs to be loaded, allowing for dynamic data generation, lazy loading, or data that
+    changes between evaluation runs. Each function should return a list of EvaluationRow
+    objects. This is useful for scenarios like generating test data on-the-fly, loading
+    data from external sources, or creating data with randomized elements for robust testing."""
+
+    def variants(self) -> Sequence[DataLoaderVariant]:
+        variants: Sequence[DataLoaderVariant] = []
+        for factory in self.factory:
+
+            def _load() -> DataLoaderResult:
+                resolved_rows = factory()
+                return DataLoaderResult(
+                    rows=resolved_rows,
+                    type=self.__class__.__name__,
+                    variant_id=factory.__name__,
+                    variant_description=factory.__doc__,
+                )
+
+            variants.append(_load)
+
+        return variants

eval_protocol-0.2.27/eval_protocol/data_loader/inline_data_loader.py

@@ -0,0 +1,68 @@
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+from eval_protocol.data_loader.models import (
+    DataLoaderResult,
+    DataLoaderVariant,
+    EvaluationDataLoader,
+)
+from eval_protocol.models import EvaluationRow, Message
+from eval_protocol.pytest.types import InputMessagesParam
+
+
+DEFAULT_VARIANT_ID: str = "inline"
+
+
+@dataclass(kw_only=True)
+class InlineDataLoader(EvaluationDataLoader):
+    """Data loader for inline ``EvaluationRow`` or message payloads."""
+
+    rows: list[EvaluationRow] | None = None
+    """Pre-defined evaluation rows with tools and metadata. Use this when you have complete
+    EvaluationRow objects that include tools, input_metadata, and other structured data.
+    This is the preferred option when working with tool-calling scenarios or when you need
+    to provide additional metadata like row_id, dataset information, or custom fields."""
+
+    messages: Sequence[InputMessagesParam] | None = None
+    """Raw chat completion message history. Use this when you only have simple
+    conversation history without tools or additional metadata. The messages will be
+    automatically converted to EvaluationRow objects. InputMessagesParam is a list of
+    Message objects representing the conversation flow (user, assistant, system messages)."""
+
+    id: str = DEFAULT_VARIANT_ID
+    """Unique identifier for this data loader variant. Used to label and distinguish
+    different input data sources, versions, or configurations. This helps with tracking
+    and organizing evaluation results from different data sources."""
+
+    description: str | None = None
+    """Optional human-readable description of this data loader. Provides additional
+    context about the data source, purpose, or any special characteristics. Used for
+    documentation and debugging purposes. If not provided, the variant_id will be used instead."""
+
+    def __post_init__(self) -> None:
+        if self.rows is None and self.messages is None:
+            raise ValueError("InlineDataLoader requires rows or messages to be provided")
+
+    def variants(self) -> Sequence[DataLoaderVariant]:
+        def _load() -> DataLoaderResult:
+            resolved_rows: list[EvaluationRow] = []
+            if self.rows is not None:
+                resolved_rows = [row.model_copy(deep=True) for row in self.rows]
+            if self.messages is not None:
+                for dataset_messages in self.messages:
+                    row_messages: list[Message] = []
+                    for msg in dataset_messages:
+                        if isinstance(msg, Message):
+                            row_messages.append(msg.model_copy(deep=True))
+                        else:
+                            row_messages.append(Message.model_validate(msg))
+                    resolved_rows.append(EvaluationRow(messages=row_messages))
+
+            return DataLoaderResult(
+                rows=resolved_rows,
+                variant_id=self.id,
+                variant_description=self.description,
+                type=self.__class__.__name__,
+            )
+
+        return [_load]

eval_protocol-0.2.27/eval_protocol/data_loader/models.py

@@ -0,0 +1,128 @@
+"""Data loader abstractions"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from typing import Callable
+from typing_extensions import Protocol
+from abc import ABC, abstractmethod
+
+from pydantic import BaseModel, Field, field_validator
+
+from eval_protocol.models import EvaluationRow
+
+
+class DataLoaderResult(BaseModel):
+    """Rows and metadata returned by a loader variant."""
+
+    rows: list[EvaluationRow] = Field(
+        description="List of evaluation rows loaded from the data source. These are the "
+        "processed and ready-to-use evaluation data that will be fed into the evaluation pipeline."
+    )
+
+    type: str = Field(
+        ...,
+        description="Type of the data loader that produced this result. Used for identification "
+        "and debugging purposes (e.g., 'InlineDataLoader', 'DynamicDataLoader').",
+    )
+
+    variant_id: str = Field(
+        ...,
+        description="Unique identifier for the data loader variant that produced this result. "
+        "Used for tracking and organizing evaluation results from different data sources.",
+    )
+
+    variant_description: str | None = Field(
+        default=None,
+        description="Human-readable description of the data loader variant that produced this result. "
+        "Provides context about what this variant represents, its purpose, or any special characteristics that distinguish "
+        "it from other variants.",
+    )
+
+    preprocessed: bool = Field(
+        default=False,
+        description="Whether the data has been preprocessed. This flag indicates if any "
+        "preprocessing functions have been applied to the data, helping to avoid duplicate "
+        "processing and track data transformation state.",
+    )
+
+    @field_validator("type")
+    @classmethod
+    def validate_type(cls, v: str) -> str:
+        if not v or not v.strip():
+            raise ValueError("type must be non-empty")
+        return v
+
+    @field_validator("variant_id")
+    @classmethod
+    def validate_variant_id(cls, v: str) -> str:
+        if not v or not v.strip():
+            raise ValueError("variant_id must be non-empty")
+        return v
+
+
+class DataLoaderVariant(Protocol):
+    """Single parameterizable variant from a data loader."""
+
+    def __call__(self) -> DataLoaderResult:
+        """Load a dataset for this variant using the provided context."""
+        ...
+
+
+@dataclass(kw_only=True)
+class EvaluationDataLoader(ABC):
+    """Abstract base class for data loaders that can be consumed by ``evaluation_test``."""
+
+    preprocess_fn: Callable[[list[EvaluationRow]], list[EvaluationRow]] | None = None
+    """Optional preprocessing function for evaluation rows. This function is applied
+    to the loaded data before it's returned, allowing for data cleaning, transformation,
+    filtering, or other modifications. The function receives a list of EvaluationRow objects
+    and should return a modified list of EvaluationRow objects."""
+
+    @abstractmethod
+    def variants(self) -> Sequence[DataLoaderVariant]:
+        """Return parameterizable variants emitted by this loader."""
+        ...
+
+    def load(self) -> list[DataLoaderResult]:
+        """Loads all variants of this data loader and return a list of DataLoaderResult."""
+        results = []
+        for variant in self.variants():
+            result = variant()
+            result = self._process_variant(result)
+            results.append(result)
+        return results
+
+    def _process_variant(self, result: DataLoaderResult) -> DataLoaderResult:
+        """Process a single variant: preprocess data and apply metadata."""
+        # Preprocess data
+        original_count = len(result.rows)
+        if self.preprocess_fn:
+            result.rows = self.preprocess_fn(result.rows)
+            result.preprocessed = True
+            processed_count = len(result.rows)
+        else:
+            processed_count = original_count
+
+        # Apply metadata to rows
+        self._apply_metadata(result, original_count, processed_count)
+        return result
+
+    def _apply_metadata(self, result: DataLoaderResult, original_count: int, processed_count: int) -> None:
+        """Apply metadata to all rows in the result."""
+        for row in result.rows:
+            if row.input_metadata.dataset_info is None:
+                row.input_metadata.dataset_info = {}
+
+            # Apply result attributes as metadata
+            for attr_name, attr_value in vars(result).items():
+                """
+                Exclude rows and private attributes from metadata.
+                """
+                if attr_name != "rows" and not attr_name.startswith("_"):
+                    row.input_metadata.dataset_info[f"data_loader_{attr_name}"] = attr_value
+
+            # Apply row counts
+            row.input_metadata.dataset_info["data_loader_num_rows"] = original_count
+            row.input_metadata.dataset_info["data_loader_num_rows_after_preprocessing"] = processed_count