ragbits-evaluate 0.15.0__tar.gz → 1.4.0.dev202601130240__tar.gz

{ragbits_evaluate-0.15.0 → ragbits_evaluate-1.4.0.dev202601130240}/.gitignore

@@ -8,6 +8,10 @@ venv/
 .venv/
 __pycache__/
 **.egg-info/
+.deepeval/
+
+# Local cursor rules
+.cursor/rules/local/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -101,3 +105,12 @@ qdrant/
 .aider*
 
 .DS_Store
+node_modules/
+
+lazygit
+
+lazygit.tar.gz
+
+# chat conversation logs
+duet_conversation.log
+worktrees/

ragbits_evaluate-1.4.0.dev202601130240/CHANGELOG.md

@@ -0,0 +1,240 @@
+# CHANGELOG
+
+## Unreleased
+
+- Feat: introduce agent evaluation pipelines and metrics (HotpotQA, HumanEval, GAIA) (#829)
+
+- Feat: introduce agent simulation module with utilities for agent-to-agent conversation and evaluation scenarios (#857)
+
+- Feat: add structured results to agent simulation with `SimulationResult`, `TurnResult`, `TaskResult`, and `ConversationMetrics` models (#885)
+
+- Feat: add generic `DomainContext` and `DataSnapshot` for flexible agent simulation context (#884)
+
+- Feat: add metrics collection system for agent simulation (`MetricCollector` protocol, `LatencyMetricCollector`, `TokenUsageMetricCollector`, `ToolUsageMetricCollector`) (#882)
+
+## 1.3.0 (2025-09-11)
+
+### Changed
+
+- ragbits-core updated to version v1.3.0
+
+- Optional parallel batches execution in ragbits.evaluate.Evaluator (#769)
+
+## 1.2.2 (2025-08-08)
+
+### Changed
+
+- ragbits-core updated to version v1.2.2
+
+## 1.2.1 (2025-08-04)
+
+### Changed
+
+- ragbits-core updated to version v1.2.1
+
+## 1.2.0 (2025-08-01)
+
+### Changed
+
+- ragbits-core updated to version v1.2.0
+
+## 1.1.0 (2025-07-09)
+
+### Changed
+
+- ragbits-core updated to version v1.1.0
+
+- Update qa data loader docstring (#565)
+- Fix deadlock on qa metrics compute (#609)
+- Upgrade distilabel version to 1.5.0 (#682)
+
+## 1.0.0 (2025-06-04)
+
+### Changed
+
+- ragbits-core updated to version v1.0.0
+
+## 0.20.1 (2025-06-04)
+
+### Changed
+
+- ragbits-core updated to version v0.20.1
+
+## 0.20.0 (2025-06-03)
+
+### Changed
+
+- ragbits-core updated to version v0.20.0
+
+## 0.19.1 (2025-05-27)
+
+### Changed
+
+- ragbits-core updated to version v0.19.1
+
+## 0.19.0 (2025-05-27)
+
+### Changed
+
+- ragbits-core updated to version v0.19.0
+
+- Add evals for question answering (#577)
+- Add support for slicing dataset (#576)
+- Separate load and map ops in data loaders (#576)
+
+## 0.18.0 (2025-05-22)
+
+### Changed
+
+- ragbits-core updated to version v0.18.0
+
+- Add support for custom column names in evaluation dataset (#566)
+- Add support for reference document ids and page numbers in evaluation dataset (#566)
+- BREAKING CHANGE: Adjust eval pipline interface to batch processing (#555)
+- Rename DocumentMeta create_text_document_from_literal to from_literal (#561)
+- Adjust typing for DocumentSearch (#554)
+
+## 0.17.1 (2025-05-09)
+
+### Changed
+
+- ragbits-core updated to version v0.17.1
+
+## 0.17.0 (2025-05-06)
+
+### Changed
+
+- ragbits-core updated to version v0.17.0
+
+- Add tests for ragbits-evaluate package (#390)
+- Integrate sources with dataloaders (#529)
+
+## 0.16.0 (2025-04-29)
+
+### Changed
+
+- ragbits-core updated to version v0.16.0
+
+## 0.15.0 (2025-04-28)
+
+### Changed
+
+- ragbits-core updated to version v0.15.0
+
+## 0.14.0 (2025-04-22)
+
+### Changed
+
+- ragbits-core updated to version v0.14.0
+
+- move sources from ragbits-document-search to ragbits-core (#496)
+
+## 0.13.0 (2025-04-02)
+
+### Changed
+
+- ragbits-core updated to version v0.13.0
+
+## 0.12.0 (2025-03-25)
+
+### Changed
+
+- ragbits-core updated to version v0.12.0
+
+## 0.11.0 (2025-03-25)
+
+### Changed
+
+- ragbits-core updated to version v0.11.0
+
+## 0.10.2 (2025-03-21)
+
+### Changed
+
+- ragbits-core updated to version v0.10.2
+
+## 0.10.1 (2025-03-19)
+
+### Changed
+
+- ragbits-core updated to version v0.10.1
+
+## 0.10.0 (2025-03-17)
+
+### Changed
+
+- ragbits-core updated to version v0.10.0
+
+- Compability with the new Vector Store interface from ragbits-core (#288)
+- chore: fix typo in README.
+- fix typos in doc strings
+
+## 0.9.0 (2025-02-25)
+
+### Changed
+
+- ragbits-core updated to version v0.9.0
+- Add cli for document search evaluation added (#356)
+- Add local data loader (#334).
+
+## 0.8.0 (2025-01-29)
+
+### Changed
+
+- ragbits-core updated to version v0.8.0
+
+## 0.7.0 (2025-01-21)
+
+### Added
+
+- Simplified interface to document-search evaluation (#258).
+
+### Changed
+
+- ragbits-core updated to version v0.7.0
+
+## 0.6.0 (2024-12-27)
+
+### Changed
+
+- ragbits-core updated to version v0.6.0
+
+## 0.5.1 (2024-12-09)
+
+### Changed
+
+- ragbits-core updated to version v0.5.1
+- document search evaluation now returns all Element types, rather than only TextElements (#241).
+
+## 0.5.0 (2024-12-05)
+
+### Changed
+
+- ragbits-core updated to version v0.5.0
+
+## 0.4.0 (2024-11-27)
+
+### Added
+
+- Introduced optimization with optuna (#177).
+- Add synthetic data generation pipeline (#165).
+
+### Changed
+
+- ragbits-core updated to version v0.4.0
+
+## 0.3.0 (2024-11-06)
+
+### Changed
+
+- ragbits-core updated to version v0.3.0
+
+## 0.2.0 (2024-10-23)
+
+- Initial release of the package.
+- Evaluation pipeline framework with capability to define evaluators & metrics.
+- Evaluation pipeline for `ragbits-document-search`.
+
+### Changed
+
+- ragbits-core updated to version v0.2.0

{ragbits_evaluate-0.15.0 → ragbits_evaluate-1.4.0.dev202601130240}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragbits-evaluate
-Version: 0.15.0
+Version: 1.4.0.dev202601130240
 Summary: Evaluation module for Ragbits components
 Project-URL: Homepage, https://github.com/deepsense-ai/ragbits
 Project-URL: Bug Reports, https://github.com/deepsense-ai/ragbits/issues
@@ -22,11 +22,13 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.10
-Requires-Dist: distilabel==1.4.1
-Requires-Dist: hydra-core~=1.3.2
-Requires-Dist: neptune[optuna]~=1.12.0
-Requires-Dist: optuna==4.0.0
-Requires-Dist: ragbits-core==0.15.0
+Requires-Dist: datasets<4.0.0,>=3.0.1
+Requires-Dist: deepeval<3.0.0,>=2.0.0
+Requires-Dist: distilabel<2.0.0,>=1.5.0
+Requires-Dist: hydra-core<2.0.0,>=1.3.2
+Requires-Dist: neptune[optuna]<2.0.0,>=1.12.0
+Requires-Dist: optuna<5.0.0,>=4.0.0
+Requires-Dist: ragbits-core==1.4.0.dev202601130240
 Provides-Extra: relari
 Requires-Dist: continuous-eval<1.0.0,>=0.3.12; extra == 'relari'
 Description-Content-Type: text/markdown

{ragbits_evaluate-0.15.0 → ragbits_evaluate-1.4.0.dev202601130240}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ragbits-evaluate"
-version = "0.15.0"
+version = "1.4.0.dev202601130240"
 description = "Evaluation module for Ragbits components"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -32,7 +32,7 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
     "Topic :: Software Development :: Libraries :: Python Modules",
 ]
-dependencies = ["hydra-core~=1.3.2", "neptune[optuna]~=1.12.0", "optuna==4.0.0", "distilabel==1.4.1", "ragbits-core==0.15.0"]
+dependencies = ["hydra-core>=1.3.2,<2.0.0", "neptune[optuna]>=1.12.0,<2.0.0", "optuna>=4.0.0,<5.0.0", "distilabel>=1.5.0,<2.0.0", "datasets>=3.0.1,<4.0.0", "ragbits-core==1.4.0.dev202601130240", "deepeval>=2.0.0,<3.0.0"]
 
 [project.urls]
 "Homepage" = "https://github.com/deepsense-ai/ragbits"

ragbits_evaluate-1.4.0.dev202601130240/src/ragbits/evaluate/agent_simulation/__init__.py

@@ -0,0 +1,87 @@
+"""Agent simulation utilities for evaluation scenarios.
+
+This module uses lazy imports for components that require optional dependencies
+(ragbits-agents, ragbits-chat) to allow importing result models independently.
+"""
+
+from typing import TYPE_CHECKING
+
+# Import context, metrics, and result models eagerly - they have no external dependencies
+from ragbits.evaluate.agent_simulation.context import DataSnapshot, DomainContext
+from ragbits.evaluate.agent_simulation.metrics import (
+    CompositeMetricCollector,
+    LatencyMetricCollector,
+    MetricCollector,
+    TokenUsageMetricCollector,
+    ToolUsageMetricCollector,
+)
+from ragbits.evaluate.agent_simulation.results import (
+    ConversationMetrics,
+    SimulationResult,
+    SimulationStatus,
+    TaskResult,
+    TurnResult,
+)
+
+if TYPE_CHECKING:
+    from ragbits.evaluate.agent_simulation.conversation import run_simulation
+    from ragbits.evaluate.agent_simulation.deepeval_evaluator import DeepEvalEvaluator
+    from ragbits.evaluate.agent_simulation.logger import ConversationLogger
+    from ragbits.evaluate.agent_simulation.models import Personality, Scenario, Task, Turn
+    from ragbits.evaluate.agent_simulation.scenarios import load_personalities, load_scenarios
+    from ragbits.evaluate.agent_simulation.simulation import GoalChecker, SimulatedUser
+
+__all__ = [
+    "CompositeMetricCollector",
+    "ConversationLogger",
+    "ConversationMetrics",
+    "DataSnapshot",
+    "DeepEvalEvaluator",
+    "DomainContext",
+    "GoalChecker",
+    "LatencyMetricCollector",
+    "MetricCollector",
+    "Personality",
+    "Scenario",
+    "SimulatedUser",
+    "SimulationResult",
+    "SimulationStatus",
+    "Task",
+    "TaskResult",
+    "TokenUsageMetricCollector",
+    "ToolUsageMetricCollector",
+    "Turn",
+    "TurnResult",
+    "load_personalities",
+    "load_scenarios",
+    "run_simulation",
+]
+
+
+def __getattr__(name: str) -> object:
+    """Lazy import for components with optional dependencies."""
+    if name == "run_simulation":
+        from ragbits.evaluate.agent_simulation.conversation import run_simulation
+
+        return run_simulation
+    if name == "DeepEvalEvaluator":
+        from ragbits.evaluate.agent_simulation.deepeval_evaluator import DeepEvalEvaluator
+
+        return DeepEvalEvaluator
+    if name == "ConversationLogger":
+        from ragbits.evaluate.agent_simulation.logger import ConversationLogger
+
+        return ConversationLogger
+    if name in ("Personality", "Scenario", "Task", "Turn"):
+        from ragbits.evaluate.agent_simulation import models
+
+        return getattr(models, name)
+    if name in ("load_personalities", "load_scenarios"):
+        from ragbits.evaluate.agent_simulation import scenarios
+
+        return getattr(scenarios, name)
+    if name in ("GoalChecker", "SimulatedUser"):
+        from ragbits.evaluate.agent_simulation import simulation
+
+        return getattr(simulation, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

ragbits_evaluate-1.4.0.dev202601130240/src/ragbits/evaluate/agent_simulation/context.py

@@ -0,0 +1,118 @@
+"""Context models for agent simulation scenarios."""
+
+from dataclasses import dataclass, field
+from typing import Any
+
+DEFAULT_MAX_ITEMS_IN_PROMPT = 15
+
+
+@dataclass
+class DomainContext:
+    """Domain-specific context for goal checking and simulation.
+
+    Provides additional context to the GoalChecker to avoid false negatives
+    from value interpretation differences or missing domain knowledge.
+
+    The context is intentionally generic - use the `metadata` field for any
+    domain-specific information that doesn't fit the standard fields.
+
+    Example:
+        >>> context = DomainContext(
+        ...     domain_type="customer_support",
+        ...     locale="en_US",
+        ...     metadata={"ticket_statuses": ["open", "pending", "resolved"]},
+        ... )
+        >>> result = await goal_checker.is_task_achieved(task, history, context=context)
+    """
+
+    domain_type: str
+    """Type of domain (e.g., "customer_support", "booking", "search", "qa")."""
+
+    locale: str = "en_US"
+    """Locale for language and formatting (e.g., "en_US", "de_DE")."""
+
+    metadata: dict[str, Any] = field(default_factory=dict)
+    """Arbitrary domain-specific metadata for goal checking context."""
+
+    def format_for_prompt(self) -> str:
+        """Format context for inclusion in LLM prompts.
+
+        Returns:
+            Formatted string suitable for prompt injection.
+        """
+        parts = [
+            f"Domain: {self.domain_type}",
+            f"Locale: {self.locale}",
+        ]
+
+        if self.metadata:
+            parts.append("Additional context:")
+            for key, value in self.metadata.items():
+                if isinstance(value, list) and len(value) > DEFAULT_MAX_ITEMS_IN_PROMPT:
+                    truncated = value[:DEFAULT_MAX_ITEMS_IN_PROMPT]
+                    parts.append(f"  {key}: {truncated} ... and {len(value) - DEFAULT_MAX_ITEMS_IN_PROMPT} more")
+                else:
+                    parts.append(f"  {key}: {value}")
+
+        return "\n".join(parts)
+
+
+@dataclass
+class DataSnapshot:
+    """Sample of available data to ground simulated user requests.
+
+    Provides the simulated user with knowledge of what data actually exists,
+    preventing unrealistic requests for non-existent entities.
+
+    The snapshot is intentionally generic - store any domain-specific data
+    in the `entities` dict with descriptive keys.
+
+    Example:
+        >>> snapshot = DataSnapshot(
+        ...     entities={
+        ...         "available_topics": ["billing", "technical", "returns"],
+        ...         "sample_users": [{"id": "u1", "name": "John"}],
+        ...     },
+        ...     description="Customer support knowledge base",
+        ... )
+        >>> # SimulatedUser will only reference items from this data
+    """
+
+    entities: dict[str, list[Any]] = field(default_factory=dict)
+    """Named collections of available entities (e.g., {"users": [...], "documents": [...]})."""
+
+    description: str = ""
+    """Optional description of the data snapshot for context."""
+
+    def format_for_prompt(self, max_items: int = DEFAULT_MAX_ITEMS_IN_PROMPT) -> str:
+        """Format data snapshot for inclusion in LLM prompts.
+
+        Args:
+            max_items: Maximum number of items to include per entity type.
+
+        Returns:
+            Formatted string suitable for prompt injection.
+        """
+        parts = []
+
+        if self.description:
+            parts.append(f"Context: {self.description}")
+
+        for entity_name, entity_list in self.entities.items():
+            if not entity_list:
+                continue
+
+            truncated = entity_list[:max_items]
+            # Format items - if dicts with 'name', use that; otherwise str()
+            formatted_items = []
+            for item in truncated:
+                if isinstance(item, dict) and "name" in item:
+                    formatted_items.append(item["name"])
+                else:
+                    formatted_items.append(str(item))
+
+            parts.append(f"{entity_name}: {', '.join(formatted_items)}")
+            if len(entity_list) > max_items:
+                parts.append(f"  ... and {len(entity_list) - max_items} more")
+
+        return "\n".join(parts)