agentbyte 0.29.0__tar.gz → 0.30.0__tar.gz

{agentbyte-0.29.0 → agentbyte-0.30.0}/CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to Agentbyte are documented in this file.
 
 The format follows Keep a Changelog principles and semantic versioning.
 
+## [0.30.0] - 2026-09-30
+
+### Added
+
+- Workflow streaming parity (spec 0057): `WorkflowRunner.run_stream(..., stream_tokens=...)` forwards live child agent events as `StepEvent` envelopes (workflow, execution, invocation, step, attempt and sequence attribution) through a bounded queue, including across `workflow.as_agent()` and `SubWorkflowStep` boundaries. `WorkflowAgent.run_stream` honors `stream_tokens`. Closing the stream cancels and awaits outstanding steps.
+- `AgentStep` streams its agent and turns tool approval into workflow suspension; resuming validates the exact pending decisions and never reruns completed tools.
+- Checkpoint-backed serving: the WebUI/FastAPI run and SSE endpoints accept `workflow_responses` and `workflow_checkpoint_id` to resume a suspended workflow in the same session. `WorkflowExecution.checkpoint_id` exposes the suspended checkpoint, and `WorkflowRunner.validate_resume_responses` / `BaseStep.validate_resume_response` validate responses without consuming it. Stale, unknown or wrong-owner checkpoints and concurrent runs of the same workflow or session are rejected with HTTP 409.
+- Judge score integrity (spec 0055): `EvalScore.scoring_status` (`scored` | `failed` | `cancelled`) and `failure_reason`, `JudgeScoringError` with safe reason codes, `mean_scored()`, `OptimizationEvidenceError`, and `PairwiseResult.comparison_status`.
+
+### Changed
+
+- **Breaking:** Judges never invent scores. `LLMEvalJudge` and `LLMTrajectoryJudge` require every requested criterion exactly once (matched case- and whitespace-insensitively, reported with the requested spelling) with a finite 0–10 value, and raise `JudgeScoringError` otherwise instead of filling missing criteria with 5.0, clamping, or returning a neutral 5.0 after a failure. `overall` is always the mean of the validated dimensions; a model-supplied overall is ignored. Cancellation raises `asyncio.CancelledError`.
+- **Breaking:** `EvalScore.overall` is `float | None`. `EvalRunner` records judge failures, target exceptions and cancellation as `failed`/`cancelled` outcomes with `overall=None`, empty dimensions and the trajectory kept, instead of a fabricated 0.0. Failure metadata keeps the exception type, never its text. A scored `EvalScore` must have a finite `overall` and dimensions in 0–10.
+- **Breaking:** `PairwiseJudge.compare()` raises instead of returning a tie on failure; unknown winners and out-of-range margins are rejected, not coerced. `PairwiseRunner` records failed comparisons with `winner=None`. `compare_pairwise()` adds `compared`/`failed`/`cancelled` counts and computes win rates over valid comparisons only (`None` when nothing was compared).
+- **Breaking:** `EvalReport.avg_score` and `compare_configurations()` averages cover scored outcomes only and are `None` when nothing was scored; an item with a failed or cancelled score fails the suite. `CompositeJudge` fails when any sub-judge fails rather than renormalizing the remaining weights.
+- **Breaking:** Optimizers only rank candidates whose every task was scored. `Candidate.avg` is `None` for ineligible candidates, the minibatch gate rejects incomplete batches, a seed with an unscored task raises `OptimizationEvidenceError`, and the GEPA adapter raises rather than passing a number for an unscored example. Optimization trace `eval` events report `avg` over scored tasks (`None` if none) plus `n_scored`.
+
 ## [0.29.0] - 2026-09-30
 
 ### Added

{agentbyte-0.29.0 → agentbyte-0.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.29.0
+Version: 0.30.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -86,7 +86,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.29.0**
+Current release: **0.30.0**
 
 ## Building an Agent
 

{agentbyte-0.29.0 → agentbyte-0.30.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.29.0**
+Current release: **0.30.0**
 
 ## Building an Agent
 

agentbyte-0.30.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.30.0"
+VERSION = __version__

{agentbyte-0.29.0 → agentbyte-0.30.0}/src/agentbyte/eval/__init__.py

@@ -1,6 +1,6 @@
 """Evaluation framework for AgentByte."""
 
-from .base import BaseEvalJudge, BaseEvalRunner, BaseEvalTarget
+from .base import BaseEvalJudge, BaseEvalRunner, BaseEvalTarget, JudgeScoringError
 from .checks.process import select_span, terminal_status, steps_completed, edge_activated, usage_limits
 from .judges.trajectory import LLMTrajectoryJudge
 from .targets.workflow import WorkflowEvalTarget
@@ -19,7 +19,7 @@ from .checks import (
     tool_calls_present,
     tool_calls_present_check,
 )
-from .comparison import compare_configurations
+from .comparison import compare_configurations, mean_scored
 from .eval_dataset import (
     EvalDatasetRecord,
     save_eval_dataset,
@@ -52,6 +52,7 @@ from .types import (
     EvalTask,
     EvalTrajectory,
     ExpectedToolCall,
+    ScoringStatus,
     MultiTurnEvalTask,
     WorkflowEvalTask,
     PairwiseResult,
@@ -70,6 +71,8 @@ __all__ = [
     "BaseEvalTarget",
     "BaseEvalJudge",
     "BaseEvalRunner",
+    "JudgeScoringError",
+    "ScoringStatus",
     "ExactMatchJudge",
     "ContainsJudge",
     "FuzzyMatchJudge",
@@ -88,6 +91,7 @@ __all__ = [
     "EvalSplitConfig",
     "EvalSplitStrategy",
     "compare_configurations",
+    "mean_scored",
     "EvalDatasetRecord",
     "save_eval_dataset",
     "tasks_from_eval_dataset",

{agentbyte-0.29.0 → agentbyte-0.30.0}/src/agentbyte/eval/base.py

@@ -17,6 +17,19 @@ _VALID_ANSWER_STRATEGIES: tuple[AnswerStrategy, ...] = (
 )
 
 
+class JudgeScoringError(Exception):
+    """A judge could not produce a valid judgment.
+
+    ``reason`` is a safe, stable code (for example ``provider_error`` or
+    ``missing_criterion``) suitable for persistence; the message may carry more
+    detail and is not persisted by default.
+    """
+
+    def __init__(self, reason: str, message: str | None = None) -> None:
+        super().__init__(message or reason)
+        self.reason = reason
+
+
 class BaseEvalTarget(ABC):
     """Anything that can execute an evaluation task into a trajectory."""
 
@@ -96,7 +109,12 @@ class BaseEvalJudge(ABC):
         criteria: list[str] | None = None,
         cancellation_token: CancellationToken | None = None,
     ) -> EvalScore:
-        """Score an evaluation trajectory."""
+        """Score an evaluation trajectory.
+
+        Returns a scored ``EvalScore``. Raises ``JudgeScoringError`` when no
+        valid judgment is available and ``asyncio.CancelledError`` when
+        cancelled; judges never substitute a placeholder score.
+        """
 
 
 class BaseEvalRunner(ABC):
@@ -116,4 +134,4 @@ class BaseEvalRunner(ABC):
         """Evaluate a target on multiple tasks."""
 
 
-__all__ = ["BaseEvalTarget", "BaseEvalJudge", "BaseEvalRunner"]
+__all__ = ["BaseEvalTarget", "BaseEvalJudge", "BaseEvalRunner", "JudgeScoringError"]

{agentbyte-0.29.0 → agentbyte-0.30.0}/src/agentbyte/eval/checks/local.py

@@ -93,6 +93,12 @@ def threshold_gate(
 
     async def _check(trajectory: EvalTrajectory) -> CheckResult:
         score = await judge.score(trajectory)
+        if not score.is_scored or score.overall is None:
+            return CheckResult(
+                passed=False,
+                reason=f"no valid score ({score.scoring_status}: {score.failure_reason})",
+                check_name=f"threshold_gate[{judge.name}]",
+            )
         passed = score.overall >= threshold
         return CheckResult(
             passed=passed,

agentbyte-0.30.0/src/agentbyte/eval/comparison.py

@@ -0,0 +1,58 @@
+"""Helpers for comparing evaluated configurations."""
+
+from __future__ import annotations
+
+from agentbyte.eval.types import EvalScore
+
+
+def compare_configurations(
+    scores_a: list[EvalScore],
+    scores_b: list[EvalScore],
+    label_a: str = "baseline",
+    label_b: str = "candidate",
+) -> dict[str, float | str | int | None]:
+    """Compare average score between two evaluated configurations.
+
+    Averages cover scored outcomes only; failed and cancelled outcomes are
+    counted per side and never enter a mean. ``avg_*`` is None when a side has
+    no scored outcome, and ``improvement_pct`` is None unless both sides have a
+    nonzero baseline to compare against.
+    """
+    avg_a = mean_scored(scores_a)
+    avg_b = mean_scored(scores_b)
+
+    improvement_pct: float | None = None
+    if avg_a is not None and avg_b is not None:
+        if avg_a == 0:
+            improvement_pct = 0.0 if avg_b == 0 else 100.0
+        else:
+            improvement_pct = ((avg_b - avg_a) / avg_a) * 100.0
+
+    return {
+        "label_a": label_a,
+        "label_b": label_b,
+        "avg_a": avg_a,
+        "avg_b": avg_b,
+        "scored_a": _count(scores_a, "scored"),
+        "scored_b": _count(scores_b, "scored"),
+        "failed_a": _count(scores_a, "failed"),
+        "failed_b": _count(scores_b, "failed"),
+        "cancelled_a": _count(scores_a, "cancelled"),
+        "cancelled_b": _count(scores_b, "cancelled"),
+        "improvement_pct": improvement_pct,
+    }
+
+
+def mean_scored(scores: list[EvalScore]) -> float | None:
+    """Mean ``overall`` across scored outcomes, or None when none were scored."""
+    values = [score.overall for score in scores if score.is_scored and score.overall is not None]
+    if not values:
+        return None
+    return sum(values) / len(values)
+
+
+def _count(scores: list[EvalScore], status: str) -> int:
+    return sum(1 for score in scores if score.scoring_status == status)
+
+
+__all__ = ["compare_configurations", "mean_scored"]

{agentbyte-0.29.0 → agentbyte-0.30.0}/src/agentbyte/eval/judges/composite.py

@@ -7,7 +7,7 @@ from collections.abc import Sequence
 
 from agentbyte.cancellation_token import CancellationToken
 
-from agentbyte.eval.base import BaseEvalJudge
+from agentbyte.eval.base import BaseEvalJudge, JudgeScoringError
 from agentbyte.eval.types import EvalScore, EvalTrajectory
 
 
@@ -45,12 +45,30 @@ class CompositeJudge(BaseEvalJudge):
         criteria: list[str] | None = None,
         cancellation_token: CancellationToken | None = None,
     ) -> EvalScore:
-        results = await asyncio.gather(
-            *[
-                judge.score(trajectory, criteria, cancellation_token)
-                for judge, _ in self.judges
-            ]
-        )
+        """Combine sub-judge scores; any sub-judge failure fails the composite.
+
+        Every sub-judge is required. A failed or cancelled component is never
+        dropped and the remaining weights are never renormalized into success.
+        """
+        try:
+            async with asyncio.TaskGroup() as group:
+                tasks = [
+                    group.create_task(judge.score(trajectory, criteria, cancellation_token))
+                    for judge, _ in self.judges
+                ]
+        except* JudgeScoringError as failures:
+            first = failures.exceptions[0]
+            raise JudgeScoringError(
+                "component_failed", f"Sub-judge failed: {first.reason}"
+            ) from first
+        results = [task.result() for task in tasks]
+        overalls: list[float] = []
+        for (judge, _), result in zip(self.judges, results):
+            if not result.is_scored or result.overall is None:
+                raise JudgeScoringError(
+                    "component_failed", f"Sub-judge {judge.name!r} returned no valid score"
+                )
+            overalls.append(result.overall)
 
         dimensions: dict[str, float] = {}
         dimension_weight_totals: dict[str, float] = {}
@@ -58,11 +76,9 @@ class CompositeJudge(BaseEvalJudge):
         metadata_sub_judges: list[dict[str, float | str]] = []
         weighted_overall = 0.0
 
-        for (judge, weight), result in zip(self.judges, results):
-            metadata_sub_judges.append(
-                {"name": judge.name, "weight": weight, "score": result.overall}
-            )
-            weighted_overall += result.overall * weight
+        for (judge, weight), result, overall in zip(self.judges, results, overalls):
+            metadata_sub_judges.append({"name": judge.name, "weight": weight, "score": overall})
+            weighted_overall += overall * weight
             for dimension, score in result.dimensions.items():
                 dimensions[dimension] = dimensions.get(dimension, 0.0) + (score * weight)
                 dimension_weight_totals[dimension] = (

agentbyte-0.30.0/src/agentbyte/eval/judges/llm.py

@@ -0,0 +1,220 @@
+"""LLM-powered evaluation judge."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field, StrictFloat, ValidationError
+
+from agentbyte.cancellation_token import CancellationToken
+from agentbyte.llm.base import BaseChatCompletionClient
+from agentbyte.messages import SystemMessage, UserMessage
+
+from agentbyte.eval.base import BaseEvalJudge, JudgeScoringError
+from agentbyte.eval.judges.validation import (
+    mean_score,
+    normalize_criterion,
+    validate_criterion_scores,
+    validate_requested_criteria,
+)
+from agentbyte.eval.types import AnswerStrategy, EvalScore, EvalTrajectory
+
+
+class _JudgeCriterionScore(BaseModel):
+    criterion: str
+    score: StrictFloat = Field(ge=0.0, le=10.0)
+    reasoning: str
+
+
+class _JudgeStructuredResponse(BaseModel):
+    scores: list[_JudgeCriterionScore]
+
+
+# Kept for the use_structured_output=False path. Values stay untyped so the
+# shared validator (not Pydantic coercion) decides what counts as a score.
+class _JudgeResponse(BaseModel):
+    model_config = ConfigDict(extra="allow")
+
+    dimensions: dict[str, Any] = Field(default_factory=dict)
+    reasoning: dict[str, Any] = Field(default_factory=dict)
+
+
+class LLMEvalJudge(BaseEvalJudge):
+    """Use an LLM to judge a trajectory on one or more criteria."""
+
+    def __init__(
+        self,
+        client: BaseChatCompletionClient,
+        *,
+        name: str | None = None,
+        answer_strategy: AnswerStrategy = "last_non_empty",
+        source_filter: str | None = None,
+        default_criteria: list[str] | None = None,
+        custom_instructions: str | None = None,
+        use_structured_output: bool = True,
+    ) -> None:
+        super().__init__(
+            name=name or f"LLM-{client.model}",
+            answer_strategy=answer_strategy,
+            source_filter=source_filter,
+        )
+        self.client = client
+        self.default_criteria = default_criteria or [
+            "accuracy",
+            "completeness",
+            "helpfulness",
+        ]
+        self.custom_instructions = custom_instructions
+        self.use_structured_output = use_structured_output
+
+    async def score(
+        self,
+        trajectory: EvalTrajectory,
+        criteria: list[str] | None = None,
+        cancellation_token: CancellationToken | None = None,
+    ) -> EvalScore:
+        """Score every requested criterion or raise ``JudgeScoringError``.
+
+        ``overall`` is the mean of the validated dimensions. Cancellation raises
+        ``asyncio.CancelledError``; no placeholder score is ever returned.
+        """
+        # resolve: call-level arg > task metadata > judge default
+        if criteria is None:
+            raw = trajectory.task.metadata.get("_criteria")
+            if isinstance(raw, list):
+                criteria = raw
+            elif isinstance(raw, str):
+                criteria = [raw]
+        eval_criteria = validate_requested_criteria(criteria or self.default_criteria)
+        if cancellation_token and cancellation_token.is_cancelled():
+            raise asyncio.CancelledError()
+
+        answer = self.extract_answer(trajectory)
+        user_payload = {
+            "task_name": trajectory.task.name,
+            "task_input": trajectory.task.input,
+            "expected_output": trajectory.task.expected_output,
+            "actual_output": answer,
+            "success": trajectory.success,
+            "error": trajectory.error,
+            "criteria": eval_criteria,
+            "usage": trajectory.usage.model_dump(mode="json")
+            if trajectory.usage is not None
+            else None,
+            "metadata": trajectory.metadata,
+        }
+        user_message = UserMessage(
+            content=json.dumps(user_payload, ensure_ascii=True, indent=2),
+            source="user",
+        )
+
+        try:
+            if self.use_structured_output:
+                result = await self.client.create(
+                    messages=[
+                        SystemMessage(
+                            content=self._build_structured_prompt(eval_criteria),
+                            source="system",
+                        ),
+                        user_message,
+                    ],
+                    output_format=_JudgeStructuredResponse,
+                )
+            else:
+                result = await self.client.create(
+                    messages=[
+                        SystemMessage(content=self._build_raw_prompt(), source="system"),
+                        user_message,
+                    ],
+                )
+        except Exception as exc:
+            raise JudgeScoringError("provider_error", "Judge model call failed") from exc
+
+        if self.use_structured_output:
+            dimensions, reasoning = self._validate_structured(
+                result.structured_output, eval_criteria
+            )
+        else:
+            dimensions, reasoning = self._validate_raw(result.message.content, eval_criteria)
+
+        return EvalScore(
+            overall=mean_score(dimensions),
+            dimensions=dimensions,
+            reasoning=reasoning,
+            trajectory=trajectory,
+            metadata={
+                "judge": self.name,
+                "model": result.model,
+                "finish_reason": result.finish_reason,
+            },
+        )
+
+    def _build_structured_prompt(self, criteria: list[str]) -> str:
+        criteria_list = ", ".join(f'"{c}"' for c in criteria)
+        prompt = (
+            "You are an evaluation judge. "
+            f"Score the following criteria: [{criteria_list}]. "
+            "Return exactly one scores[] entry per criterion, using the names as given. "
+            "Score each 0–10."
+        )
+        if self.custom_instructions:
+            prompt += f"\n\nAdditional instructions:\n{self.custom_instructions.strip()}"
+        return prompt
+
+    def _build_raw_prompt(self) -> str:
+        prompt = (
+            "You are an evaluation judge. "
+            "Return ONLY a strict JSON object with exactly these keys:\n"
+            '  "dimensions": {"<criterion>": <number 0-10>, ...},\n'
+            '  "reasoning": {"<criterion>": "<explanation string>", ...}\n'
+            "Score every requested criterion exactly once. "
+            "The `reasoning` value MUST be an object (dict) — one key per criterion, "
+            "never a plain string. No extra keys, no markdown, no prose outside the JSON."
+        )
+        if self.custom_instructions:
+            prompt += f"\n\nAdditional instructions:\n{self.custom_instructions.strip()}"
+        return prompt
+
+    @staticmethod
+    def _validate_structured(
+        structured: Any,
+        criteria: list[str],
+    ) -> tuple[dict[str, float], dict[str, str]]:
+        if not isinstance(structured, _JudgeStructuredResponse):
+            raise JudgeScoringError(
+                "invalid_response",
+                f"Expected _JudgeStructuredResponse, got {type(structured).__name__}",
+            )
+        return validate_criterion_scores(
+            criteria,
+            ((item.criterion, item.score, item.reasoning) for item in structured.scores),
+        )
+
+    @staticmethod
+    def _validate_raw(
+        raw_content: str,
+        criteria: list[str],
+    ) -> tuple[dict[str, float], dict[str, str]]:
+        try:
+            parsed = _JudgeResponse.model_validate(json.loads(raw_content))
+        except (json.JSONDecodeError, ValidationError, TypeError) as exc:
+            raise JudgeScoringError("invalid_response", "Judge returned malformed JSON") from exc
+        dimensions, _ = validate_criterion_scores(
+            criteria,
+            ((name, value, None) for name, value in parsed.dimensions.items()),
+        )
+        reasoning_lookup = {
+            normalize_criterion(str(name)): text
+            for name, text in parsed.reasoning.items()
+            if isinstance(text, str)
+        }
+        reasoning = {
+            criterion: reasoning_lookup.get(normalize_criterion(criterion), "")
+            for criterion in criteria
+        }
+        return dimensions, reasoning
+
+
+__all__ = ["LLMEvalJudge"]

{agentbyte-0.29.0 → agentbyte-0.30.0}/src/agentbyte/eval/judges/pairwise.py

@@ -2,25 +2,27 @@
 
 from __future__ import annotations
 
+import asyncio
 import json
+import math
 from typing import Any
 
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, StrictFloat, ValidationError
 
 from agentbyte.cancellation_token import CancellationToken
 from agentbyte.llm.base import BaseChatCompletionClient
 from agentbyte.messages import SystemMessage, UserMessage
 
-from agentbyte.eval.base import BaseEvalJudge
+from agentbyte.eval.base import BaseEvalJudge, JudgeScoringError
 from agentbyte.eval.types import AnswerStrategy, EvalScore, EvalTrajectory, PairwiseResult
 
 
 class _PairwiseResponse(BaseModel):
     model_config = ConfigDict(extra="allow")
 
-    winner: str = Field(default="tie")
-    margin: float = Field(default=0.0)
-    reasoning: str = Field(default="No reasoning provided")
+    winner: str
+    margin: StrictFloat
+    reasoning: str = Field(default="")
 
 
 class PairwiseJudge(BaseEvalJudge):
@@ -63,13 +65,14 @@ class PairwiseJudge(BaseEvalJudge):
         trajectory_b: EvalTrajectory,
         cancellation_token: CancellationToken | None = None,
     ) -> PairwiseResult:
-        """Compare two trajectories and return a pairwise decision.
+        """Compare two trajectories and return a measured pairwise decision.
 
-        Returns a tie with ``margin=0.0`` on cancellation or LLM parse failure.
-        The failure reason is preserved in ``metadata["fallback_reason"]``.
+        Raises ``JudgeScoringError`` when no valid decision is available and
+        ``asyncio.CancelledError`` when cancelled. A failure is never reported
+        as a tie.
         """
         if cancellation_token and cancellation_token.is_cancelled():
-            return self._tie(trajectory_a, trajectory_b, fallback_reason="Cancelled")
+            raise asyncio.CancelledError()
 
         answer_a = self.extract_answer(trajectory_a)
         answer_b = self.extract_answer(trajectory_b)
@@ -106,22 +109,27 @@ class PairwiseJudge(BaseEvalJudge):
                 ],
                 output_format=_PairwiseResponse,
             )
-            parsed = self._parse_response(result.structured_output, result.message.content)
-            winner = parsed.winner if parsed.winner in ("a", "b", "tie") else "tie"
-            margin = max(0.0, min(1.0, float(parsed.margin)))
-            if winner == "tie":
-                margin = 0.0
-            return PairwiseResult(
-                task=trajectory_a.task,
-                winner=winner,  # type: ignore[arg-type]
-                margin=margin,
-                reasoning=parsed.reasoning,
-                trajectory_a=trajectory_a,
-                trajectory_b=trajectory_b,
-                metadata={"judge": self.name, "model": result.model},
-            )
         except Exception as exc:
-            return self._tie(trajectory_a, trajectory_b, fallback_reason=str(exc))
+            raise JudgeScoringError("provider_error", "Pairwise judge model call failed") from exc
+        try:
+            parsed = self._parse_response(result.structured_output, result.message.content)
+        except ValidationError as exc:
+            raise JudgeScoringError("invalid_response", "Malformed pairwise response") from exc
+        winner = parsed.winner.strip().lower()
+        if winner not in ("a", "b", "tie"):
+            raise JudgeScoringError("invalid_response", f"Unknown winner: {parsed.winner!r}")
+        margin = float(parsed.margin)
+        if not math.isfinite(margin) or not 0.0 <= margin <= 1.0:
+            raise JudgeScoringError("invalid_score", f"Margin outside 0-1: {parsed.margin!r}")
+        return PairwiseResult(
+            task=trajectory_a.task,
+            winner=winner,  # type: ignore[arg-type]
+            margin=0.0 if winner == "tie" else margin,
+            reasoning=parsed.reasoning,
+            trajectory_a=trajectory_a,
+            trajectory_b=trajectory_b,
+            metadata={"judge": self.name, "model": result.model},
+        )
 
     async def score(self, *args: Any, **kwargs: Any) -> EvalScore:
         """Not supported on PairwiseJudge — use compare() instead."""
@@ -142,22 +150,5 @@ class PairwiseJudge(BaseEvalJudge):
             return _PairwiseResponse.model_validate(structured_output)
         return _PairwiseResponse.model_validate_json(raw_content)
 
-    def _tie(
-        self,
-        trajectory_a: EvalTrajectory,
-        trajectory_b: EvalTrajectory,
-        *,
-        fallback_reason: str,
-    ) -> PairwiseResult:
-        return PairwiseResult(
-            task=trajectory_a.task,
-            winner="tie",
-            margin=0.0,
-            reasoning=fallback_reason,
-            trajectory_a=trajectory_a,
-            trajectory_b=trajectory_b,
-            metadata={"judge": self.name, "fallback": True, "fallback_reason": fallback_reason},
-        )
-
 
 __all__ = ["PairwiseJudge"]