agentbyte 0.28.0__tar.gz → 0.30.0__tar.gz

{agentbyte-0.28.0 → agentbyte-0.30.0}/CHANGELOG.md

@@ -4,6 +4,45 @@ All notable changes to Agentbyte are documented in this file.
 
 The format follows Keep a Changelog principles and semantic versioning.
 
+## [0.30.0] - 2026-09-30
+
+### Added
+
+- Workflow streaming parity (spec 0057): `WorkflowRunner.run_stream(..., stream_tokens=...)` forwards live child agent events as `StepEvent` envelopes (workflow, execution, invocation, step, attempt and sequence attribution) through a bounded queue, including across `workflow.as_agent()` and `SubWorkflowStep` boundaries. `WorkflowAgent.run_stream` honors `stream_tokens`. Closing the stream cancels and awaits outstanding steps.
+- `AgentStep` streams its agent and turns tool approval into workflow suspension; resuming validates the exact pending decisions and never reruns completed tools.
+- Checkpoint-backed serving: the WebUI/FastAPI run and SSE endpoints accept `workflow_responses` and `workflow_checkpoint_id` to resume a suspended workflow in the same session. `WorkflowExecution.checkpoint_id` exposes the suspended checkpoint, and `WorkflowRunner.validate_resume_responses` / `BaseStep.validate_resume_response` validate responses without consuming it. Stale, unknown or wrong-owner checkpoints and concurrent runs of the same workflow or session are rejected with HTTP 409.
+- Judge score integrity (spec 0055): `EvalScore.scoring_status` (`scored` | `failed` | `cancelled`) and `failure_reason`, `JudgeScoringError` with safe reason codes, `mean_scored()`, `OptimizationEvidenceError`, and `PairwiseResult.comparison_status`.
+
+### Changed
+
+- **Breaking:** Judges never invent scores. `LLMEvalJudge` and `LLMTrajectoryJudge` require every requested criterion exactly once (matched case- and whitespace-insensitively, reported with the requested spelling) with a finite 0–10 value, and raise `JudgeScoringError` otherwise instead of filling missing criteria with 5.0, clamping, or returning a neutral 5.0 after a failure. `overall` is always the mean of the validated dimensions; a model-supplied overall is ignored. Cancellation raises `asyncio.CancelledError`.
+- **Breaking:** `EvalScore.overall` is `float | None`. `EvalRunner` records judge failures, target exceptions and cancellation as `failed`/`cancelled` outcomes with `overall=None`, empty dimensions and the trajectory kept, instead of a fabricated 0.0. Failure metadata keeps the exception type, never its text. A scored `EvalScore` must have a finite `overall` and dimensions in 0–10.
+- **Breaking:** `PairwiseJudge.compare()` raises instead of returning a tie on failure; unknown winners and out-of-range margins are rejected, not coerced. `PairwiseRunner` records failed comparisons with `winner=None`. `compare_pairwise()` adds `compared`/`failed`/`cancelled` counts and computes win rates over valid comparisons only (`None` when nothing was compared).
+- **Breaking:** `EvalReport.avg_score` and `compare_configurations()` averages cover scored outcomes only and are `None` when nothing was scored; an item with a failed or cancelled score fails the suite. `CompositeJudge` fails when any sub-judge fails rather than renormalizing the remaining weights.
+- **Breaking:** Optimizers only rank candidates whose every task was scored. `Candidate.avg` is `None` for ineligible candidates, the minibatch gate rejects incomplete batches, a seed with an unscored task raises `OptimizationEvidenceError`, and the GEPA adapter raises rather than passing a number for an unscored example. Optimization trace `eval` events report `avg` over scored tasks (`None` if none) plus `n_scored`.
+
+## [0.29.0] - 2026-09-30
+
+### Added
+
+- Unified trajectory evaluation (spec 0046): every eval target now records a runtime-neutral execution trace (`trajectory.trace`) with spans for agents, workflow steps, tools and model calls, plus normalized terminal status and usage. New `WorkflowEvalTarget` and `WorkflowEvalTask` evaluate workflows directly with typed input; workflows wrapped via `workflow.as_agent()` capture the same evidence.
+- Process evaluation: `LLMTrajectoryJudge` scores `tool_use_correctness`, `coordination` and `execution_efficiency` with cited span/event evidence and reports failures as `metadata["judge_failed"]` instead of silently truncating. New deterministic checks `terminal_status`, `steps_completed`, `edge_activated` and `usage_limits`, and `select_span` to score one span in isolation.
+- Orchestrator finalization (spec 0048): `FinalResultPolicy` gains `final_agent` and `finalize_on_exhaustion` (default `True`). When a final agent is configured (`final_agent`, or `prefer_agent` as fallback) and `max_iterations` is about to run out, the last iteration runs that agent with a finalization instruction, bypassing the pattern's selection, context and state hooks; the run stops with `stop_message.source == "MaxIterationsFinalized"`. An unknown `final_agent` raises `ValueError`.
+- `OrchestrationResponse.final_structured_result` exposes the `structured_content` of the message selected as `final_result`, so a typed final agent's answer is available without parsing JSON.
+- `HandoffOrchestrator` accepts `final_result_policy`.
+
+### Changed
+
+- **Breaking:** The agent workflow step module moved from `agentbyte.workflow.steps.agentbyte_agent` to `agentbyte.workflow.steps.agent`, and `AgentbyteAgentInput` / `AgentbyteAgentOutput` are renamed to `AgentStepInput` / `AgentStepOutput` (exported from `agentbyte.workflow` and `agentbyte.workflow.steps`). No compatibility aliases; update imports. `AgentStep` is unchanged.
+- **Breaking:** Agents reject duplicate tool names across `tools` and context-provider tools (spec 0047). The run fails with `finish_reason="error"` and an `AgentConfigurationError` before any model call. Previously, attaching several `SkillsProvider`s silently made every provider but the first unreachable; attach one `SkillsProvider` with several paths (`from_paths([...])` or `AggregatingSkillsSource`) instead.
+- Orchestrators configured with `FinalResultPolicy(prefer_agent=...)` now reserve the last iteration for that agent when the budget is exhausted. Opt out with `finalize_on_exhaustion=False`.
+- `stream_tokens=True` is disabled with a warning when the agent has an `output_format`, because streamed chunks are not parsed into structured output. Typed turns now always carry `structured_content`.
+
+### Fixed
+
+- The agent finalization turn drops tool calls the model returns despite having no tools, so they are never executed or left unanswered in history, and the run reports `max_iterations_finalized` (spec 0042 hardening).
+- Workflow examples that imported the removed `AgentbyteAgentStep` and `Context` names now use `AgentStep` and `WorkflowContext`.
+
 ## [0.28.0] - 2026-09-15
 
 ### Changed

{agentbyte-0.28.0 → agentbyte-0.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.28.0
+Version: 0.30.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -86,7 +86,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.28.0**
+Current release: **0.30.0**
 
 ## Building an Agent
 

{agentbyte-0.28.0 → agentbyte-0.30.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.28.0**
+Current release: **0.30.0**
 
 ## Building an Agent
 

agentbyte-0.30.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.30.0"
+VERSION = __version__

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/agents/agent.py

@@ -12,6 +12,7 @@ from typing import List, Optional, Union
 
 from agentbyte.cancellation_token import CancellationToken
 from agentbyte.context import AgentContext
+from agentbyte.execution_trace import trace_stream
 from agentbyte.llm.types import ChatCompletionChunk
 from agentbyte.messages import (
     AssistantMessage,
@@ -338,6 +339,7 @@ class Agent(BaseAgent):
 
         yield message
 
+    @trace_stream("agent")
     async def run_stream(
         self,
         task: Optional[Union[str, UserMessage, List[Message]]] = None,
@@ -414,6 +416,15 @@ class Agent(BaseAgent):
                 stacklevel=2,
             )
             effective_stream_tokens = False
+        elif stream_tokens and self.output_format is not None:
+            # Streamed chunks are not schema-constrained or parsed, so typed
+            # answers would lose structured_content.
+            warnings.warn(
+                "stream_tokens=True is disabled when output_format is set; "
+                "falling back to non-token execution to keep structured output.",
+                stacklevel=2,
+            )
+            effective_stream_tokens = False
 
         try:
             task_messages = self._convert_task_to_messages(task) if task else []
@@ -434,6 +445,7 @@ class Agent(BaseAgent):
                 if instr:
                     provider_instructions.append(instr)
                 provider_tools.extend(cp_tools)
+            self._check_unique_tool_names(extra_tools=provider_tools)
             extra_instructions = "\n\n".join(provider_instructions) if provider_instructions else None
 
             iteration = 0
@@ -474,7 +486,8 @@ class Agent(BaseAgent):
                 tools = self._get_tools_for_llm(extra_tools=provider_tools) if (self.tools or provider_tools) else None
 
                 # Reserve the final iteration for a tool-free answer so an exhausted
-                # budget yields a complete response instead of nothing.
+                # budget yields a complete response instead of nothing. Needs
+                # max_iterations > 1, otherwise the only turn would lose its tools.
                 is_final_iteration = (
                     self.finalize_on_exhaustion
                     and self.max_iterations > 1
@@ -625,10 +638,14 @@ class Agent(BaseAgent):
                 if completion_result.usage.cost_estimate is not None:
                     cost_estimate += completion_result.usage.cost_estimate
 
+                # The finalization turn offers no tools; drop any tool calls the
+                # model returns anyway so they never execute or dangle in history.
                 assistant_message = AssistantMessage(
                     content=completion_result.message.content,
                     source=self.name,
-                    tool_calls=completion_result.message.tool_calls,
+                    tool_calls=None
+                    if is_final_iteration
+                    else completion_result.message.tool_calls,
                     structured_content=completion_result.structured_output,
                     usage=completion_result.usage,
                 )
@@ -638,7 +655,7 @@ class Agent(BaseAgent):
                         source=self.name,
                         model=completion_result.model,
                         response=completion_result.message.content,
-                        has_tool_calls=bool(completion_result.message.tool_calls),
+                        has_tool_calls=bool(assistant_message.tool_calls),
                     )
 
                 working_context.add_message(assistant_message)

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/agents/base.py

@@ -61,6 +61,15 @@ class BaseAgent(ABC):
         context_providers: Optional[List[ContextProvider]] = None,
         **kwargs: Any,
     ) -> None:
+        """Initialize the agent.
+
+        When ``finalize_on_exhaustion`` is enabled and the agent has tools, the
+        last of ``max_iterations`` is reserved for a tool-free final answer
+        (``finish_reason="max_iterations_finalized"``). This requires
+        ``max_iterations >= 2``: with ``max_iterations=1`` the only turn keeps
+        its tools, so a run that calls a tool ends with
+        ``finish_reason="max_iterations"`` and no final answer.
+        """
         self.name = name
         self.description = description
         self.instructions = instructions
@@ -124,6 +133,30 @@ class BaseAgent(ABC):
                     return tool
         return None
 
+    def _check_unique_tool_names(
+        self,
+        extra_tools: Optional[List[BaseTool]] = None,
+    ) -> None:
+        """Raise if agent tools and context-provider tools share a name.
+
+        Duplicate names are ambiguous for the model and ``_find_tool`` always
+        resolves to the first match, silently shadowing later tools.
+        """
+        seen: set[str] = set()
+        duplicates: list[str] = []
+        for tool in [*self.tools, *(extra_tools or [])]:
+            if tool.name in seen and tool.name not in duplicates:
+                duplicates.append(tool.name)
+            seen.add(tool.name)
+        if duplicates:
+            names = ", ".join(repr(name) for name in duplicates)
+            raise AgentConfigurationError(
+                f"Duplicate tool name(s) {names} across agent tools and context "
+                "providers; tool names must be unique. To use several skill "
+                "directories, attach one SkillsProvider with all paths, e.g. "
+                "SkillsProvider.from_paths([...]) or AggregatingSkillsSource."
+            )
+
     def _get_tools_for_llm(
         self,
         extra_tools: Optional[List[BaseTool]] = None,

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/__init__.py

@@ -1,6 +1,9 @@
 """Evaluation framework for AgentByte."""
 
-from .base import BaseEvalJudge, BaseEvalRunner, BaseEvalTarget
+from .base import BaseEvalJudge, BaseEvalRunner, BaseEvalTarget, JudgeScoringError
+from .checks.process import select_span, terminal_status, steps_completed, edge_activated, usage_limits
+from .judges.trajectory import LLMTrajectoryJudge
+from .targets.workflow import WorkflowEvalTarget
 from .checks import (
     CheckResult,
     EvalCheck,
@@ -16,7 +19,7 @@ from .checks import (
     tool_calls_present,
     tool_calls_present_check,
 )
-from .comparison import compare_configurations
+from .comparison import compare_configurations, mean_scored
 from .eval_dataset import (
     EvalDatasetRecord,
     save_eval_dataset,
@@ -49,11 +52,15 @@ from .types import (
     EvalTask,
     EvalTrajectory,
     ExpectedToolCall,
+    ScoringStatus,
     MultiTurnEvalTask,
+    WorkflowEvalTask,
     PairwiseResult,
 )
 
 __all__ = [
+    "WorkflowEvalTask", "WorkflowEvalTarget", "LLMTrajectoryJudge",
+    "select_span", "terminal_status", "steps_completed", "edge_activated", "usage_limits",
     "AnswerStrategy",
     "ExpectedToolCall",
     "EvalTask",
@@ -64,6 +71,8 @@ __all__ = [
     "BaseEvalTarget",
     "BaseEvalJudge",
     "BaseEvalRunner",
+    "JudgeScoringError",
+    "ScoringStatus",
     "ExactMatchJudge",
     "ContainsJudge",
     "FuzzyMatchJudge",
@@ -82,6 +91,7 @@ __all__ = [
     "EvalSplitConfig",
     "EvalSplitStrategy",
     "compare_configurations",
+    "mean_scored",
     "EvalDatasetRecord",
     "save_eval_dataset",
     "tasks_from_eval_dataset",

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/base.py

@@ -17,6 +17,19 @@ _VALID_ANSWER_STRATEGIES: tuple[AnswerStrategy, ...] = (
 )
 
 
+class JudgeScoringError(Exception):
+    """A judge could not produce a valid judgment.
+
+    ``reason`` is a safe, stable code (for example ``provider_error`` or
+    ``missing_criterion``) suitable for persistence; the message may carry more
+    detail and is not persisted by default.
+    """
+
+    def __init__(self, reason: str, message: str | None = None) -> None:
+        super().__init__(message or reason)
+        self.reason = reason
+
+
 class BaseEvalTarget(ABC):
     """Anything that can execute an evaluation task into a trajectory."""
 
@@ -96,7 +109,12 @@ class BaseEvalJudge(ABC):
         criteria: list[str] | None = None,
         cancellation_token: CancellationToken | None = None,
     ) -> EvalScore:
-        """Score an evaluation trajectory."""
+        """Score an evaluation trajectory.
+
+        Returns a scored ``EvalScore``. Raises ``JudgeScoringError`` when no
+        valid judgment is available and ``asyncio.CancelledError`` when
+        cancelled; judges never substitute a placeholder score.
+        """
 
 
 class BaseEvalRunner(ABC):
@@ -116,4 +134,4 @@ class BaseEvalRunner(ABC):
         """Evaluate a target on multiple tasks."""
 
 
-__all__ = ["BaseEvalTarget", "BaseEvalJudge", "BaseEvalRunner"]
+__all__ = ["BaseEvalTarget", "BaseEvalJudge", "BaseEvalRunner", "JudgeScoringError"]

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/checks/__init__.py

@@ -4,6 +4,7 @@ from agentbyte.eval.report import EvalItemReport, EvalNotPassedError, EvalReport
 
 from .decorator import evaluator
 from .keyword import keyword_check
+from .process import select_span, terminal_status, steps_completed, edge_activated, usage_limits
 from .local import CheckEvaluator, threshold_gate
 from .tool import (
     tool_call_args_match,
@@ -14,6 +15,7 @@ from .tool import (
 from .types import CheckResult, EvalCheck, ExpectedToolCall
 
 __all__ = [
+    "select_span", "terminal_status", "steps_completed", "edge_activated", "usage_limits",
     "ExpectedToolCall",
     "CheckResult",
     "EvalCheck",

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/checks/local.py

@@ -93,6 +93,12 @@ def threshold_gate(
 
     async def _check(trajectory: EvalTrajectory) -> CheckResult:
         score = await judge.score(trajectory)
+        if not score.is_scored or score.overall is None:
+            return CheckResult(
+                passed=False,
+                reason=f"no valid score ({score.scoring_status}: {score.failure_reason})",
+                check_name=f"threshold_gate[{judge.name}]",
+            )
         passed = score.overall >= threshold
         return CheckResult(
             passed=passed,

agentbyte-0.30.0/src/agentbyte/eval/checks/process.py

@@ -0,0 +1,123 @@
+"""Deterministic checks over execution evidence."""
+
+from __future__ import annotations
+
+from agentbyte.eval.types import EvalTrajectory
+from .types import CheckResult, EvalCheck
+
+
+def select_span(trajectory: EvalTrajectory, span_id: str) -> EvalTrajectory:
+    """Project one invocation for existing judges and checks without changing its task."""
+    if trajectory.trace is None:
+        raise ValueError("Trajectory has no execution trace")
+    trace = trajectory.trace.select(span_id)
+    root = trace.spans[0]
+    return trajectory.model_copy(
+        update={
+            "trace": trace,
+            "messages": trace.evidence_messages(),
+            "usage": trace.aggregate_usage(),
+            "success": root.status == "completed",
+            "error": root.error,
+            "metadata": {
+                **trajectory.metadata,
+                "execution_status": root.status,
+                "span_id": span_id,
+            },
+        }
+    )
+
+
+def terminal_status(expected: str = "completed") -> EvalCheck:
+    """Require an explicit normalized terminal status."""
+
+    async def check(trajectory: EvalTrajectory) -> CheckResult:
+        actual = trajectory.metadata.get("execution_status")
+        return CheckResult(
+            passed=actual == expected,
+            reason=f"Expected {expected}; observed {actual}",
+            check_name="terminal_status",
+        )
+
+    return check
+
+
+def steps_completed(*step_ids: str) -> EvalCheck:
+    """Require at least one completed attempt of each step; select a span to scope nesting."""
+
+    async def check(trajectory: EvalTrajectory) -> CheckResult:
+        actual = (
+            {s.step_id for s in trajectory.trace.spans if s.status == "completed"}
+            if trajectory.trace
+            else set()
+        )
+        missing = set(step_ids) - actual
+        return CheckResult(
+            passed=trajectory.trace is not None and not missing,
+            reason=f"Missing completed steps: {sorted(missing)}",
+            check_name="steps_completed",
+        )
+
+    return check
+
+
+def edge_activated(from_step: str, to_step: str) -> EvalCheck:
+    """Require an observed routing edge, not merely adjacent message order."""
+
+    async def check(trajectory: EvalTrajectory) -> CheckResult:
+        found = trajectory.trace is not None and any(
+            e.kind == "EdgeActivatedEvent"
+            and e.data.get("from_step") == from_step
+            and e.data.get("to_step") == to_step
+            for e in trajectory.trace.events
+        )
+        return CheckResult(
+            passed=found,
+            reason=f"Edge {from_step} -> {to_step} observed: {found}",
+            check_name="edge_activated",
+        )
+
+    return check
+
+
+def usage_limits(**limits: float) -> EvalCheck:
+    """Require usage fields to remain within bounds; unknown cost/evidence fails the gate."""
+    valid = {
+        "duration_ms",
+        "llm_calls",
+        "tokens_input",
+        "tokens_output",
+        "tokens_cached",
+        "total_tokens",
+        "tool_calls",
+        "memory_operations",
+        "cost_estimate",
+    }
+    if not limits or set(limits) - valid or any(v < 0 for v in limits.values()):
+        raise ValueError("Supply non-negative limits for recognized Usage fields")
+
+    async def check(trajectory: EvalTrajectory) -> CheckResult:
+        known = trajectory.usage is not None and (
+            trajectory.trace is None or trajectory.trace.usage_complete
+        )
+        passed = known and all(
+            getattr(trajectory.usage, k) is not None
+            and getattr(trajectory.usage, k) <= v
+            for k, v in limits.items()
+        )
+        return CheckResult(
+            passed=passed,
+            reason=f"Usage limits {limits}; complete evidence: {known}",
+            check_name="usage_limits",
+        )
+
+    return check
+
+
+__all__ = [
+    "select_span",
+    "terminal_status",
+    "steps_completed",
+    "edge_activated",
+    "usage_limits",
+]

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/checks/tool.py

@@ -21,7 +21,7 @@ async def tool_calls_present(trajectory: EvalTrajectory) -> CheckResult:
 
     actual_names = {
         tool_call.tool_name
-        for message in trajectory.messages
+        for message in (trajectory.trace.evidence_messages() if trajectory.trace else trajectory.messages)
         if isinstance(message, AssistantMessage) and message.tool_calls
         for tool_call in message.tool_calls
     }
@@ -69,7 +69,7 @@ async def tool_call_args_match(trajectory: EvalTrajectory) -> CheckResult:
         )
 
     actual_calls: list[tuple[str, dict[str, object]]] = []
-    for message in trajectory.messages:
+    for message in (trajectory.trace.evidence_messages() if trajectory.trace else trajectory.messages):
         if not isinstance(message, AssistantMessage) or not message.tool_calls:
             continue
         for tool_call in message.tool_calls:

agentbyte-0.30.0/src/agentbyte/eval/comparison.py

@@ -0,0 +1,58 @@
+"""Helpers for comparing evaluated configurations."""
+
+from __future__ import annotations
+
+from agentbyte.eval.types import EvalScore
+
+
+def compare_configurations(
+    scores_a: list[EvalScore],
+    scores_b: list[EvalScore],
+    label_a: str = "baseline",
+    label_b: str = "candidate",
+) -> dict[str, float | str | int | None]:
+    """Compare average score between two evaluated configurations.
+
+    Averages cover scored outcomes only; failed and cancelled outcomes are
+    counted per side and never enter a mean. ``avg_*`` is None when a side has
+    no scored outcome, and ``improvement_pct`` is None unless both sides have a
+    nonzero baseline to compare against.
+    """
+    avg_a = mean_scored(scores_a)
+    avg_b = mean_scored(scores_b)
+
+    improvement_pct: float | None = None
+    if avg_a is not None and avg_b is not None:
+        if avg_a == 0:
+            improvement_pct = 0.0 if avg_b == 0 else 100.0
+        else:
+            improvement_pct = ((avg_b - avg_a) / avg_a) * 100.0
+
+    return {
+        "label_a": label_a,
+        "label_b": label_b,
+        "avg_a": avg_a,
+        "avg_b": avg_b,
+        "scored_a": _count(scores_a, "scored"),
+        "scored_b": _count(scores_b, "scored"),
+        "failed_a": _count(scores_a, "failed"),
+        "failed_b": _count(scores_b, "failed"),
+        "cancelled_a": _count(scores_a, "cancelled"),
+        "cancelled_b": _count(scores_b, "cancelled"),
+        "improvement_pct": improvement_pct,
+    }
+
+
+def mean_scored(scores: list[EvalScore]) -> float | None:
+    """Mean ``overall`` across scored outcomes, or None when none were scored."""
+    values = [score.overall for score in scores if score.is_scored and score.overall is not None]
+    if not values:
+        return None
+    return sum(values) / len(values)
+
+
+def _count(scores: list[EvalScore], status: str) -> int:
+    return sum(1 for score in scores if score.scoring_status == status)
+
+
+__all__ = ["compare_configurations", "mean_scored"]

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/judges/__init__.py

@@ -4,10 +4,12 @@ from agentbyte.eval.base import BaseEvalJudge
 
 from .composite import CompositeJudge
 from .llm import LLMEvalJudge
+from .trajectory import LLMTrajectoryJudge, ProcessJudgeResponse, ProcessCriterionScore
 from .pairwise import PairwiseJudge
 from .reference import ContainsJudge, ExactMatchJudge, FuzzyMatchJudge
 
 __all__ = [
+    "LLMTrajectoryJudge", "ProcessJudgeResponse", "ProcessCriterionScore",
     "BaseEvalJudge",
     "ExactMatchJudge",
     "ContainsJudge",

{agentbyte-0.28.0 → agentbyte-0.30.0}/src/agentbyte/eval/judges/composite.py

@@ -7,7 +7,7 @@ from collections.abc import Sequence
 
 from agentbyte.cancellation_token import CancellationToken
 
-from agentbyte.eval.base import BaseEvalJudge
+from agentbyte.eval.base import BaseEvalJudge, JudgeScoringError
 from agentbyte.eval.types import EvalScore, EvalTrajectory
 
 
@@ -45,12 +45,30 @@ class CompositeJudge(BaseEvalJudge):
         criteria: list[str] | None = None,
         cancellation_token: CancellationToken | None = None,
     ) -> EvalScore:
-        results = await asyncio.gather(
-            *[
-                judge.score(trajectory, criteria, cancellation_token)
-                for judge, _ in self.judges
-            ]
-        )
+        """Combine sub-judge scores; any sub-judge failure fails the composite.
+
+        Every sub-judge is required. A failed or cancelled component is never
+        dropped and the remaining weights are never renormalized into success.
+        """
+        try:
+            async with asyncio.TaskGroup() as group:
+                tasks = [
+                    group.create_task(judge.score(trajectory, criteria, cancellation_token))
+                    for judge, _ in self.judges
+                ]
+        except* JudgeScoringError as failures:
+            first = failures.exceptions[0]
+            raise JudgeScoringError(
+                "component_failed", f"Sub-judge failed: {first.reason}"
+            ) from first
+        results = [task.result() for task in tasks]
+        overalls: list[float] = []
+        for (judge, _), result in zip(self.judges, results):
+            if not result.is_scored or result.overall is None:
+                raise JudgeScoringError(
+                    "component_failed", f"Sub-judge {judge.name!r} returned no valid score"
+                )
+            overalls.append(result.overall)
 
         dimensions: dict[str, float] = {}
         dimension_weight_totals: dict[str, float] = {}
@@ -58,11 +76,9 @@ class CompositeJudge(BaseEvalJudge):
         metadata_sub_judges: list[dict[str, float | str]] = []
         weighted_overall = 0.0
 
-        for (judge, weight), result in zip(self.judges, results):
-            metadata_sub_judges.append(
-                {"name": judge.name, "weight": weight, "score": result.overall}
-            )
-            weighted_overall += result.overall * weight
+        for (judge, weight), result, overall in zip(self.judges, results, overalls):
+            metadata_sub_judges.append({"name": judge.name, "weight": weight, "score": overall})
+            weighted_overall += overall * weight
             for dimension, score in result.dimensions.items():
                 dimensions[dimension] = dimensions.get(dimension, 0.0) + (score * weight)
                 dimension_weight_totals[dimension] = (