fasteval-langfuse 1.0.0a1__tar.gz

fasteval_langfuse-1.0.0a1/PKG-INFO

@@ -0,0 +1,181 @@
+Metadata-Version: 2.3
+Name: fasteval-langfuse
+Version: 1.0.0a1
+Summary: Langfuse integration for fasteval - evaluate production traces with fasteval metrics
+Author: Intuit
+License: Apache-2.0
+Requires-Dist: fasteval-core>=1.0.0a1
+Requires-Dist: langfuse>=2.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Python: >=3.10, <4.0
+Description-Content-Type: text/markdown
+
+# fasteval-langfuse
+
+Langfuse integration for [fasteval](https://github.com/intuit/fasteval) - evaluate production traces with fasteval's research-backed metrics.
+
+## Installation
+
+```bash
+pip install fasteval-core fasteval-langfuse
+```
+
+## Quick Start
+
+### Evaluate Production Traces
+
+Fetch traces from Langfuse and evaluate them with fasteval metrics:
+
+```python
+from fasteval_langfuse import langfuse_traces
+from fasteval_langfuse.sampling import RandomSamplingStrategy
+import fasteval as fe
+
+@fe.correctness(threshold=0.8)
+@fe.hallucination(threshold=0.9)
+@langfuse_traces(
+    project="production",
+    filter_tags=["customer-support"],
+    time_range="last_24h",
+    sampling=RandomSamplingStrategy(sample_size=200)
+)
+def test_production_traces(trace_id, input, output, context, metadata):
+    # Evaluate the trace
+    fe.score(output, input=input)
+
+# Run with pytest - scores automatically pushed to Langfuse
+# pytest test_production.py -v
+```
+
+### Sampling Strategies
+
+Reduce evaluation costs with intelligent sampling:
+
+```python
+from fasteval_langfuse.sampling import (
+    RandomSamplingStrategy,
+    StratifiedSamplingStrategy,
+    ScoreBasedSamplingStrategy,
+)
+
+# Random sampling - 200 random traces
+@langfuse_traces(
+    project="prod",
+    sampling=RandomSamplingStrategy(sample_size=200, seed=42)
+)
+def test_random_sample(trace_id, input, output, context, metadata):
+    fe.score(output, input=input)
+
+# Stratified sampling - even distribution across user types
+@langfuse_traces(
+    project="prod",
+    sampling=StratifiedSamplingStrategy(
+        strata_key="metadata.user_type",
+        samples_per_stratum=30
+    )
+)
+def test_across_segments(trace_id, input, output, context, metadata):
+    fe.score(output, input=input)
+
+# Score-based sampling - focus on failures
+@langfuse_traces(
+    project="prod",
+    sampling=ScoreBasedSamplingStrategy(
+        score_name="user_rating",
+        low_score_threshold=3.0,
+        low_score_rate=1.0,      # 100% of low ratings
+        high_score_rate=0.05     # 5% of high ratings
+    )
+)
+def test_failures(trace_id, input, output, context, metadata):
+    fe.score(output, input=input)
+```
+
+## Built-in Sampling Strategies
+
+- **NoSamplingStrategy**: Evaluate all matching traces (default)
+- **RandomSamplingStrategy**: Unbiased random sampling
+- **StratifiedSamplingStrategy**: Even distribution across groups
+- **ScoreBasedSamplingStrategy**: Oversample low-scoring traces
+- **RecentFirstSamplingStrategy**: Prioritize recent traces
+
+## Dataset Integration
+
+Evaluate against Langfuse datasets. All dataset columns are passed as parameters - declare what you need:
+
+```python
+from fasteval_langfuse import langfuse_dataset
+
+# Basic usage
+@fe.correctness(threshold=0.8)
+@langfuse_dataset(name="qa-golden-set", version="v2")
+def test_qa_dataset(input, expected_output):
+    response = my_agent(input)
+    fe.score(response, expected_output, input=input)
+
+# Using custom metadata fields
+@fe.correctness(threshold=0.8)
+@langfuse_dataset(name="qa-golden-set", version="v2")
+def test_with_metadata(input, expected_output, difficulty, category):
+    # difficulty and category come from item.metadata
+    response = my_agent(input)
+    fe.score(response, expected_output, input=input)
+
+# Only what you need
+@fe.correctness(threshold=0.8)
+@langfuse_dataset(name="inputs-only")
+def test_minimal(input):
+    # Only declare input, ignore other fields
+    response = my_agent(input)
+    fe.score(response, input=input)
+```
+
+## Configuration
+
+```python
+from fasteval_langfuse import configure_langfuse, LangfuseConfig
+
+configure_langfuse(LangfuseConfig(
+    public_key="pk-...",                # Or from LANGFUSE_PUBLIC_KEY env
+    secret_key="sk-...",                # Or from LANGFUSE_SECRET_KEY env
+    host="https://cloud.langfuse.com",  # Or self-hosted
+    default_project="production",
+    auto_push_scores=True,              # Push scores back automatically
+    score_name_prefix="fasteval_",      # Prefix for score names
+))
+```
+
+## RAG Evaluation with Context
+
+The decorator automatically extracts context from trace metadata:
+
+```python
+@fe.faithfulness(threshold=0.8)
+@fe.contextual_precision(threshold=0.7)
+@langfuse_traces(
+    project="prod",
+    filter_tags=["rag"]
+)
+def test_rag_quality(trace_id, input, output, context, metadata):
+    # context is auto-extracted from metadata keys:
+    # - "context", "retrieved_docs", "documents", "retrieval_context"
+    
+    # Or manually extract if needed:
+    if not context:
+        context = metadata.get("custom_docs_key")
+    
+    fe.score(output, context=context, input=input)
+```
+
+## Benefits
+
+- 💰 **Cost Reduction**: Reduce LLM evaluation costs by 90%+ with sampling
+- ⚡ **Faster Feedback**: Evaluate in minutes vs hours
+- 📊 **Research-Backed Metrics**: Use fasteval's validated evaluation metrics
+- 🎯 **Focus on Issues**: Oversample failures with ScoreBasedSamplingStrategy
+- ✅ **Zero Instrumentation**: Evaluate existing traces without code changes
+- 🔄 **Automatic Scoring**: Evaluation results automatically sync to Langfuse
+
+## License
+
+MIT

fasteval_langfuse-1.0.0a1/README.md

@@ -0,0 +1,169 @@
+# fasteval-langfuse
+
+Langfuse integration for [fasteval](https://github.com/intuit/fasteval) - evaluate production traces with fasteval's research-backed metrics.
+
+## Installation
+
+```bash
+pip install fasteval-core fasteval-langfuse
+```
+
+## Quick Start
+
+### Evaluate Production Traces
+
+Fetch traces from Langfuse and evaluate them with fasteval metrics:
+
+```python
+from fasteval_langfuse import langfuse_traces
+from fasteval_langfuse.sampling import RandomSamplingStrategy
+import fasteval as fe
+
+@fe.correctness(threshold=0.8)
+@fe.hallucination(threshold=0.9)
+@langfuse_traces(
+    project="production",
+    filter_tags=["customer-support"],
+    time_range="last_24h",
+    sampling=RandomSamplingStrategy(sample_size=200)
+)
+def test_production_traces(trace_id, input, output, context, metadata):
+    # Evaluate the trace
+    fe.score(output, input=input)
+
+# Run with pytest - scores automatically pushed to Langfuse
+# pytest test_production.py -v
+```
+
+### Sampling Strategies
+
+Reduce evaluation costs with intelligent sampling:
+
+```python
+from fasteval_langfuse.sampling import (
+    RandomSamplingStrategy,
+    StratifiedSamplingStrategy,
+    ScoreBasedSamplingStrategy,
+)
+
+# Random sampling - 200 random traces
+@langfuse_traces(
+    project="prod",
+    sampling=RandomSamplingStrategy(sample_size=200, seed=42)
+)
+def test_random_sample(trace_id, input, output, context, metadata):
+    fe.score(output, input=input)
+
+# Stratified sampling - even distribution across user types
+@langfuse_traces(
+    project="prod",
+    sampling=StratifiedSamplingStrategy(
+        strata_key="metadata.user_type",
+        samples_per_stratum=30
+    )
+)
+def test_across_segments(trace_id, input, output, context, metadata):
+    fe.score(output, input=input)
+
+# Score-based sampling - focus on failures
+@langfuse_traces(
+    project="prod",
+    sampling=ScoreBasedSamplingStrategy(
+        score_name="user_rating",
+        low_score_threshold=3.0,
+        low_score_rate=1.0,      # 100% of low ratings
+        high_score_rate=0.05     # 5% of high ratings
+    )
+)
+def test_failures(trace_id, input, output, context, metadata):
+    fe.score(output, input=input)
+```
+
+## Built-in Sampling Strategies
+
+- **NoSamplingStrategy**: Evaluate all matching traces (default)
+- **RandomSamplingStrategy**: Unbiased random sampling
+- **StratifiedSamplingStrategy**: Even distribution across groups
+- **ScoreBasedSamplingStrategy**: Oversample low-scoring traces
+- **RecentFirstSamplingStrategy**: Prioritize recent traces
+
+## Dataset Integration
+
+Evaluate against Langfuse datasets. All dataset columns are passed as parameters - declare what you need:
+
+```python
+from fasteval_langfuse import langfuse_dataset
+
+# Basic usage
+@fe.correctness(threshold=0.8)
+@langfuse_dataset(name="qa-golden-set", version="v2")
+def test_qa_dataset(input, expected_output):
+    response = my_agent(input)
+    fe.score(response, expected_output, input=input)
+
+# Using custom metadata fields
+@fe.correctness(threshold=0.8)
+@langfuse_dataset(name="qa-golden-set", version="v2")
+def test_with_metadata(input, expected_output, difficulty, category):
+    # difficulty and category come from item.metadata
+    response = my_agent(input)
+    fe.score(response, expected_output, input=input)
+
+# Only what you need
+@fe.correctness(threshold=0.8)
+@langfuse_dataset(name="inputs-only")
+def test_minimal(input):
+    # Only declare input, ignore other fields
+    response = my_agent(input)
+    fe.score(response, input=input)
+```
+
+## Configuration
+
+```python
+from fasteval_langfuse import configure_langfuse, LangfuseConfig
+
+configure_langfuse(LangfuseConfig(
+    public_key="pk-...",                # Or from LANGFUSE_PUBLIC_KEY env
+    secret_key="sk-...",                # Or from LANGFUSE_SECRET_KEY env
+    host="https://cloud.langfuse.com",  # Or self-hosted
+    default_project="production",
+    auto_push_scores=True,              # Push scores back automatically
+    score_name_prefix="fasteval_",      # Prefix for score names
+))
+```
+
+## RAG Evaluation with Context
+
+The decorator automatically extracts context from trace metadata:
+
+```python
+@fe.faithfulness(threshold=0.8)
+@fe.contextual_precision(threshold=0.7)
+@langfuse_traces(
+    project="prod",
+    filter_tags=["rag"]
+)
+def test_rag_quality(trace_id, input, output, context, metadata):
+    # context is auto-extracted from metadata keys:
+    # - "context", "retrieved_docs", "documents", "retrieval_context"
+    
+    # Or manually extract if needed:
+    if not context:
+        context = metadata.get("custom_docs_key")
+    
+    fe.score(output, context=context, input=input)
+```
+
+## Benefits
+
+- 💰 **Cost Reduction**: Reduce LLM evaluation costs by 90%+ with sampling
+- ⚡ **Faster Feedback**: Evaluate in minutes vs hours
+- 📊 **Research-Backed Metrics**: Use fasteval's validated evaluation metrics
+- 🎯 **Focus on Issues**: Oversample failures with ScoreBasedSamplingStrategy
+- ✅ **Zero Instrumentation**: Evaluate existing traces without code changes
+- 🔄 **Automatic Scoring**: Evaluation results automatically sync to Langfuse
+
+## License
+
+MIT

fasteval_langfuse-1.0.0a1/fasteval_langfuse/__init__.py

@@ -0,0 +1,58 @@
+"""
+fasteval-langfuse: Langfuse integration for fasteval
+
+Evaluate production traces from Langfuse with fasteval metrics.
+
+Install: pip install fasteval-langfuse
+
+Example:
+    from fasteval_langfuse import langfuse_traces
+    from fasteval_langfuse.sampling import RandomSamplingStrategy
+    import fasteval as fe
+
+    @fe.correctness(threshold=0.8)
+    @langfuse_traces(
+        project="production",
+        filter_tags=["customer-support"],
+        time_range="last_24h",
+        sampling=RandomSamplingStrategy(sample_size=200)
+    )
+    def test_production_traces(trace_id, input, output, context, metadata):
+        fe.score(output, input=input)
+
+    # Run with pytest:
+    # pytest test_production.py -v
+"""
+
+from fasteval_langfuse.client import LangfuseClient
+from fasteval_langfuse.config import LangfuseConfig, configure_langfuse, get_config
+from fasteval_langfuse.decorators import langfuse_dataset, langfuse_traces
+from fasteval_langfuse.sampling import (
+    BaseSamplingStrategy,
+    NoSamplingStrategy,
+    RandomSamplingStrategy,
+    RecentFirstSamplingStrategy,
+    ScoreBasedSamplingStrategy,
+    StratifiedSamplingStrategy,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # Decorators
+    "langfuse_traces",
+    "langfuse_dataset",
+    # Configuration
+    "LangfuseConfig",
+    "configure_langfuse",
+    "get_config",
+    # Client
+    "LangfuseClient",
+    # Sampling strategies
+    "BaseSamplingStrategy",
+    "NoSamplingStrategy",
+    "RandomSamplingStrategy",
+    "StratifiedSamplingStrategy",
+    "ScoreBasedSamplingStrategy",
+    "RecentFirstSamplingStrategy",
+]

fasteval_langfuse-1.0.0a1/fasteval_langfuse/client.py

@@ -0,0 +1,180 @@
+"""Langfuse SDK wrapper for fasteval integration."""
+
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
+
+from fasteval_langfuse.config import get_config
+
+if TYPE_CHECKING:
+    from langfuse import Langfuse
+
+
+class LangfuseClient:
+    """
+    Wrapper around Langfuse SDK for trace fetching and score reporting.
+
+    Handles authentication, API calls, and error handling.
+    """
+
+    def __init__(
+        self,
+        public_key: Optional[str] = None,
+        secret_key: Optional[str] = None,
+        host: Optional[str] = None,
+    ):
+        """
+        Initialize Langfuse client.
+
+        Args:
+            public_key: Override config public key
+            secret_key: Override config secret key
+            host: Override config host
+
+        Raises:
+            ValueError: If credentials are missing
+        """
+        config = get_config()
+
+        self.public_key = public_key or config.public_key
+        self.secret_key = secret_key or config.secret_key
+        self.host = host or config.host
+
+        if not self.public_key or not self.secret_key:
+            raise ValueError(
+                "Langfuse credentials required. Set via config or environment variables "
+                "(LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY)"
+            )
+
+        # Lazy import to avoid dependency issues in tests
+        try:
+            from langfuse import Langfuse
+        except ImportError as e:
+            raise ImportError(
+                "Langfuse SDK not installed. Install with: pip install langfuse>=2.0.0"
+            ) from e
+
+        self._client = Langfuse(
+            public_key=self.public_key,
+            secret_key=self.secret_key,
+            host=self.host,
+        )
+
+    def fetch_traces(
+        self,
+        project: Optional[str] = None,
+        tags: Optional[List[str]] = None,
+        from_timestamp: Optional[str] = None,
+        to_timestamp: Optional[str] = None,
+        user_id: Optional[str] = None,
+        session_id: Optional[str] = None,
+        limit: Optional[int] = None,
+    ) -> List[Dict[str, Any]]:
+        """
+        Fetch traces from Langfuse.
+
+        Args:
+            project: Project name to filter by
+            tags: List of tags to filter by
+            from_timestamp: Start timestamp (ISO 8601)
+            to_timestamp: End timestamp (ISO 8601)
+            user_id: Filter by user ID
+            session_id: Filter by session ID
+            limit: Maximum number of traces to fetch
+
+        Returns:
+            List of trace dictionaries
+        """
+        # Build filter parameters
+        filters = {}
+        if tags:
+            filters["tags"] = tags
+        if user_id:
+            filters["user_id"] = user_id
+        if session_id:
+            filters["session_id"] = session_id
+
+        # Fetch traces using Langfuse SDK
+        traces = self._client.fetch_traces(
+            name=project,
+            from_timestamp=from_timestamp,
+            to_timestamp=to_timestamp,
+            limit=limit,
+            **filters,
+        )
+
+        # Convert to dict format
+        return [self._trace_to_dict(trace) for trace in traces.data]
+
+    def push_score(
+        self,
+        trace_id: str,
+        name: str,
+        value: float,
+        comment: Optional[str] = None,
+    ) -> None:
+        """
+        Push an evaluation score to a Langfuse trace.
+
+        Args:
+            trace_id: Langfuse trace ID
+            name: Score name
+            value: Score value (0.0-1.0)
+            comment: Optional score comment/reasoning
+        """
+        self._client.score(
+            trace_id=trace_id,
+            name=name,
+            value=value,
+            comment=comment,
+        )
+
+    def fetch_dataset(
+        self,
+        name: str,
+        version: Optional[str] = None,
+    ) -> List[Dict[str, Any]]:
+        """
+        Fetch dataset items from Langfuse.
+
+        Args:
+            name: Dataset name
+            version: Optional dataset version
+
+        Returns:
+            List of dataset item dictionaries
+        """
+        dataset = self._client.get_dataset(name)
+
+        # Filter by version if specified
+        items = dataset.items
+        if version:
+            items = [item for item in items if item.version == version]
+
+        return [self._dataset_item_to_dict(item) for item in items]
+
+    def _trace_to_dict(self, trace: Any) -> Dict[str, Any]:
+        """Convert Langfuse trace object to dictionary."""
+        return {
+            "id": trace.id,
+            "timestamp": trace.timestamp,
+            "name": trace.name,
+            "user_id": getattr(trace, "user_id", None),
+            "session_id": getattr(trace, "session_id", None),
+            "tags": getattr(trace, "tags", []),
+            "metadata": getattr(trace, "metadata", {}),
+            "input": getattr(trace, "input", None),
+            "output": getattr(trace, "output", None),
+            "scores": getattr(trace, "scores", []),
+        }
+
+    def _dataset_item_to_dict(self, item: Any) -> Dict[str, Any]:
+        """Convert Langfuse dataset item to dictionary."""
+        return {
+            "id": item.id,
+            "input": item.input,
+            "expected_output": item.expected_output,
+            "metadata": getattr(item, "metadata", {}),
+        }
+
+    def flush(self) -> None:
+        """Flush pending scores to Langfuse."""
+        self._client.flush()

fasteval_langfuse-1.0.0a1/fasteval_langfuse/config.py

@@ -0,0 +1,80 @@
+"""Configuration for fasteval-langfuse."""
+
+import os
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+
+class LangfuseConfig(BaseModel):
+    """
+    Configuration for Langfuse integration.
+
+    Attributes:
+        public_key: Langfuse public API key (or from LANGFUSE_PUBLIC_KEY env)
+        secret_key: Langfuse secret API key (or from LANGFUSE_SECRET_KEY env)
+        host: Langfuse host URL (or from LANGFUSE_HOST env)
+        default_project: Default project name for traces
+        auto_push_scores: Automatically push evaluation scores back to Langfuse
+        batch_size: Batch size for fetching traces
+        max_parallel_evals: Maximum parallel evaluations
+        retry_on_failure: Retry failed score pushes
+        score_name_prefix: Prefix for score names in Langfuse
+    """
+
+    public_key: Optional[str] = Field(
+        default_factory=lambda: os.getenv("LANGFUSE_PUBLIC_KEY")
+    )
+    secret_key: Optional[str] = Field(
+        default_factory=lambda: os.getenv("LANGFUSE_SECRET_KEY")
+    )
+    host: str = Field(
+        default_factory=lambda: os.getenv("LANGFUSE_HOST", "https://cloud.langfuse.com")
+    )
+    default_project: Optional[str] = None
+    auto_push_scores: bool = True
+    batch_size: int = 50
+    max_parallel_evals: int = 5
+    retry_on_failure: bool = True
+    score_name_prefix: str = "fasteval_"
+
+
+# Global configuration instance
+_config: Optional[LangfuseConfig] = None
+
+
+def configure_langfuse(config: LangfuseConfig) -> None:
+    """
+    Configure the Langfuse integration.
+
+    Args:
+        config: LangfuseConfig instance
+
+    Example:
+        from fasteval_langfuse import configure_langfuse, LangfuseConfig
+
+        configure_langfuse(LangfuseConfig(
+            public_key="pk-...",
+            secret_key="sk-...",
+            default_project="production"
+        ))
+    """
+    global _config
+    _config = config
+
+
+def get_config() -> LangfuseConfig:
+    """
+    Get the current Langfuse configuration.
+
+    Returns:
+        Current LangfuseConfig, or default if not configured
+
+    Example:
+        config = get_config()
+        print(config.default_project)
+    """
+    global _config
+    if _config is None:
+        _config = LangfuseConfig()
+    return _config