agentbyte 0.34.0__tar.gz → 0.38.0__tar.gz

{agentbyte-0.34.0 → agentbyte-0.38.0}/CHANGELOG.md

@@ -6,6 +6,95 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.38.0] - 2026-10-02
+
+### Added
+
+- Orchestrator optimization (spec 0062). `agentbyte.optim` can now optimize a
+  multi-agent orchestrator as a whole, not only a single agent:
+  - A pluggable `build_target` seam on `BaseOptimizer` maps a candidate config to a
+    `BaseEvalTarget`. Omitting it preserves the single-agent path byte-for-byte
+    (`config.to_agent` + `AgentEvalTarget`).
+  - `agentbyte.optim.OrchestratorConfig` — a frozen, serializable composite candidate
+    (`{member_name: AgentConfig}`). The controller/termination is *not* snapshotted; it
+    is reattached by a user-supplied `OrchestratorAssembler`.
+  - `orchestrator_build_target(model_client, tool_catalog, assemble)` builds an
+    `OrchestratorEvalTarget` in factory form (a fresh orchestrator per eval case).
+  - `OptimizationSpec.for_members({...})` (→ `MemberOptimizationSpec`) exposes each
+    member's components under a namespaced name (`support.instructions`); edits route
+    back to the owning member, and names outside the composite spec become no-ops.
+  - Per-member credit assignment reuses `source_filter` (spec 0060/0061): the judge
+    scores only one member's messages in the multi-source orchestrated trajectory.
+  - New exports: `OrchestratorConfig`, `OptimizableConfig`, `OrchestratorAssembler`,
+    `TargetBuilder`, `orchestrator_build_target`, `MemberOptimizationSpec`.
+
+### Notes
+
+- Single-agent optimization is unchanged (default `build_target`, `AgentConfig` seed).
+- Workflow-level optimization remains a non-goal: optimize each `AgentStep`'s agent
+  standalone and compose. GEPA stays single-agent (it runs the external `gepa` loop).
+
+## [0.37.0] - 2026-10-02
+
+### Changed
+
+- `agentbyte.eval.history.CaptureMode` is now an `enum.StrEnum` (`SUMMARY`, `FULL`) instead of a
+  `Literal["summary", "full"]` alias (spec 0063). The valid values are importable and
+  autocomplete, and the enum is the single source of truth: the duplicated `("summary", "full")`
+  checks in `EvalRunRecorder` and `EvalSuiteCoordinator._check_capture`, and the re-declared
+  `Literal` on the WebUI request model, are gone. Because `StrEnum` members are `str` subclasses
+  with value-equality (`CaptureMode.FULL == "full"`), callers passing the raw strings keep
+  working, the WebUI JSON contract (`"capture": "full"`) is unchanged, and stored runs reload
+  identically.
+
+### Notes
+
+- Breaking only for code that compared `CaptureMode` with `is` against a string literal
+  (identity); equality (`==`) is the supported contract and is unchanged.
+
+## [0.36.0] - 2026-10-02
+
+### Added
+
+- `agentbyte.eval.ConversationScope` — a `StrEnum` (`FULL`, `LAST_TURN`) that selects which
+  part of a multi-turn conversation a judge evaluates (spec 0061). It is applied before
+  `source_filter` and `AnswerStrategy`, so the three selection axes compose: scope narrows the
+  trajectory to a turn window, `source_filter` keeps matching sources, then `AnswerStrategy`
+  picks the message(s). A turn boundary is the last `UserMessage`; `LAST_TURN` scopes from
+  there to the end of the trajectory.
+- `conversation_scope` keyword (default `ConversationScope.FULL`) on `BaseEvalJudge` and the
+  judges/checks that already accept `answer_strategy` (`LLMEvalJudge`, reference judges,
+  `PairwiseJudge`, `LLMTrajectoryJudge`, `keyword_check`, the `evaluator` decorator).
+- `agentbyte.eval.base.scope_messages(messages, scope)` — the pure helper implementing the
+  turn-window selection.
+
+### Notes
+
+- Additive and non-breaking: the default `FULL` reproduces today's whole-trajectory behaviour
+  for every existing single- and multi-turn caller. `LAST_TURN` is strictly opt-in.
+
+## [0.35.0] - 2026-10-02
+
+Upgrade notes (breaking):
+
+- `agentbyte.eval.AnswerStrategy` is now an `enum.StrEnum` instead of a `Literal[...]` type
+  alias (spec 0060). Prefer the members `AnswerStrategy.LAST_NON_EMPTY`,
+  `AnswerStrategy.LAST_ASSISTANT`, `AnswerStrategy.LAST_CONTENT` and
+  `AnswerStrategy.ALL_ASSISTANT`. Because `StrEnum` members are `str` subclasses with
+  value-equality, passing the raw strings (`"last_assistant"`, etc.) keeps working and
+  serialization is unchanged; only code that compared the value with `is` against a string
+  literal (identity) is affected.
+
+### Changed
+
+- `AnswerStrategy` is a single source of truth: the duplicated `_VALID_ANSWER_STRATEGIES`
+  tuple in `agentbyte.eval.base` is removed and unknown values are rejected via
+  `AnswerStrategy(value)` with a `ValueError` naming the valid members. All judges and checks
+  (`LLMEvalJudge`, reference judges, `PairwiseJudge`, `keyword_check`, the `evaluator`
+  decorator, composite/trajectory judges) default to the enum members.
+- Eval end-to-end notebook (`notebooks/usecases/eval/06-eval-end-to-end.ipynb`) and the
+  example catalog (`examples/eval/acme_support.py`) use the typed `AnswerStrategy` enum.
+
 ## [0.34.0] - 2026-10-01
 
 ### Added

{agentbyte-0.34.0 → agentbyte-0.38.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.34.0
+Version: 0.38.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -72,7 +72,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.34.0**
+Current release: **0.38.0**
 
 ## Building an Agent
 
@@ -368,7 +368,7 @@ uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # htt
 - To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
 Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
-notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/study/05-eval_topics/`.
+notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/design-guide/05-eval_topics/`.
 
 ### Option 3: Scan a project directory
 

{agentbyte-0.34.0 → agentbyte-0.38.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.34.0**
+Current release: **0.38.0**
 
 ## Building an Agent
 
@@ -298,7 +298,7 @@ uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # htt
 - To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
 Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
-notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/study/05-eval_topics/`.
+notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/design-guide/05-eval_topics/`.
 
 ### Option 3: Scan a project directory
 

agentbyte-0.38.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.38.0"
+VERSION = __version__

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/__init__.py

@@ -48,6 +48,7 @@ from .splitting import (
 from .targets import AgentEvalTarget, ModelEvalTarget, MultiTurnAgentEvalTarget, OrchestratorEvalTarget
 from .types import (
     AnswerStrategy,
+    ConversationScope,
     EvalResult,
     EvalScore,
     EvalTask,
@@ -63,6 +64,7 @@ __all__ = [
     "WorkflowEvalTask", "WorkflowEvalTarget", "LLMTrajectoryJudge",
     "select_span", "terminal_status", "steps_completed", "edge_activated", "usage_limits",
     "AnswerStrategy",
+    "ConversationScope",
     "ExpectedToolCall",
     "EvalTask",
     "MultiTurnEvalTask",

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/base.py

@@ -5,16 +5,29 @@ from __future__ import annotations
 from abc import ABC, abstractmethod
 
 from agentbyte.cancellation_token import CancellationToken
-from agentbyte.messages import AssistantMessage
+from agentbyte.messages import AssistantMessage, Message, UserMessage
 
-from .types import AnswerStrategy, EvalScore, EvalTask, EvalTrajectory
+from .types import AnswerStrategy, ConversationScope, EvalScore, EvalTask, EvalTrajectory
 
-_VALID_ANSWER_STRATEGIES: tuple[AnswerStrategy, ...] = (
-    "last_non_empty",
-    "last_assistant",
-    "last_content",
-    "all_assistant",
-)
+
+def scope_messages(
+    messages: list[Message],
+    scope: ConversationScope,
+) -> list[Message]:
+    """Narrow a message list to the chosen conversation scope.
+
+    ``FULL`` returns the list unchanged. ``LAST_TURN`` returns the messages from
+    the last ``UserMessage`` (a turn boundary) to the end, inclusive of that
+    user message; if there is no ``UserMessage`` the whole list is returned.
+    This runs before ``source_filter`` and ``AnswerStrategy`` so the three
+    selection axes compose.
+    """
+    if scope == ConversationScope.LAST_TURN:
+        for index in range(len(messages) - 1, -1, -1):
+            if isinstance(messages[index], UserMessage):
+                return list(messages[index:])
+        return list(messages)
+    return list(messages)
 
 
 class JudgeScoringError(Exception):
@@ -51,17 +64,28 @@ class BaseEvalJudge(ABC):
     def __init__(
         self,
         name: str,
-        answer_strategy: AnswerStrategy = "last_non_empty",
+        answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
         source_filter: str | None = None,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
     ) -> None:
-        if answer_strategy not in _VALID_ANSWER_STRATEGIES:
+        try:
+            answer_strategy = AnswerStrategy(answer_strategy)
+        except ValueError as exc:
             raise ValueError(
                 f"Unsupported answer_strategy: {answer_strategy!r}. "
-                f"Expected one of {list(_VALID_ANSWER_STRATEGIES)}."
-            )
+                f"Expected one of {[s.value for s in AnswerStrategy]}."
+            ) from exc
+        try:
+            conversation_scope = ConversationScope(conversation_scope)
+        except ValueError as exc:
+            raise ValueError(
+                f"Unsupported conversation_scope: {conversation_scope!r}. "
+                f"Expected one of {[s.value for s in ConversationScope]}."
+            ) from exc
         self.name = name
         self.answer_strategy = answer_strategy
         self.source_filter = source_filter
+        self.conversation_scope = conversation_scope
 
     def extract_answer(self, trajectory: EvalTrajectory) -> str:
         """Extract the answer text from a trajectory using the configured strategy.
@@ -71,23 +95,27 @@ class BaseEvalJudge(ABC):
         lets callers target a specific agent's output in multi-agent trajectories
         (e.g. extract the Writer's last message from a Researcher→Writer→Reviewer
         orchestrator run).
+
+        Selection runs in a fixed order: ``conversation_scope`` narrows the
+        trajectory to a turn window, then ``source_filter`` keeps only matching
+        sources, then ``answer_strategy`` picks the message(s) that remain.
         """
-        messages = list(trajectory.messages)
+        messages = scope_messages(trajectory.messages, self.conversation_scope)
         if self.source_filter:
             messages = [m for m in messages if m.source == self.source_filter]
         if not messages:
             return ""
 
-        if self.answer_strategy == "last_content":
+        if self.answer_strategy == AnswerStrategy.LAST_CONTENT:
             return messages[-1].content
 
-        if self.answer_strategy == "last_non_empty":
+        if self.answer_strategy == AnswerStrategy.LAST_NON_EMPTY:
             for message in reversed(messages):
                 if message.content and message.content.strip():
                     return message.content
             return ""
 
-        if self.answer_strategy == "last_assistant":
+        if self.answer_strategy == AnswerStrategy.LAST_ASSISTANT:
             for message in reversed(messages):
                 if isinstance(message, AssistantMessage):
                     return message.content
@@ -134,4 +162,10 @@ class BaseEvalRunner(ABC):
         """Evaluate a target on multiple tasks."""
 
 
-__all__ = ["BaseEvalTarget", "BaseEvalJudge", "BaseEvalRunner", "JudgeScoringError"]
+__all__ = [
+    "BaseEvalTarget",
+    "BaseEvalJudge",
+    "BaseEvalRunner",
+    "JudgeScoringError",
+    "scope_messages",
+]

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/checks/decorator.py

@@ -7,7 +7,7 @@ from collections.abc import Callable
 from typing import Any, overload
 
 from agentbyte.eval.base import BaseEvalJudge
-from agentbyte.eval.types import AnswerStrategy, EvalTrajectory
+from agentbyte.eval.types import AnswerStrategy, ConversationScope, EvalTrajectory
 
 from .types import CheckResult, EvalCheck
 
@@ -95,18 +95,20 @@ def evaluator(fn: Callable[..., Any], /) -> EvalCheck: ...
 @overload
 def evaluator(
     *,
-    answer_strategy: AnswerStrategy = "last_non_empty",
+    answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
     name: str | None = None,
     source_filter: str | None = None,
+    conversation_scope: ConversationScope = ConversationScope.FULL,
 ) -> Callable[[Callable[..., Any]], EvalCheck]: ...
 
 
 def evaluator(
     fn: Callable[..., Any] | None = None,
     *,
-    answer_strategy: AnswerStrategy = "last_non_empty",
+    answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
     name: str | None = None,
     source_filter: str | None = None,
+    conversation_scope: ConversationScope = ConversationScope.FULL,
 ) -> EvalCheck | Callable[[Callable[..., Any]], EvalCheck]:
     """Wrap a plain function as an EvalCheck with parameter injection.
 
@@ -139,6 +141,7 @@ def evaluator(
                 name="extractor",
                 answer_strategy=answer_strategy,
                 source_filter=source_filter,
+                conversation_scope=conversation_scope,
             )
             kwargs = _resolve_args(
                 func,

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/checks/keyword.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from agentbyte.eval.judges.reference import ExactMatchJudge
-from agentbyte.eval.types import AnswerStrategy, EvalTrajectory
+from agentbyte.eval.types import AnswerStrategy, ConversationScope, EvalTrajectory
 
 from .types import CheckResult, EvalCheck
 
@@ -12,7 +12,8 @@ def keyword_check(
     *keywords: str,
     case_sensitive: bool = False,
     source_filter: str | None = None,
-    answer_strategy: AnswerStrategy = "last_assistant",
+    answer_strategy: AnswerStrategy = AnswerStrategy.LAST_ASSISTANT,
+    conversation_scope: ConversationScope = ConversationScope.FULL,
 ) -> EvalCheck:
     """Return a check that verifies all keywords appear in the extracted response.
 
@@ -22,7 +23,11 @@ def keyword_check(
     Pass ``source_filter`` to target a specific agent in multi-agent trajectories
     (e.g. ``source_filter="writer"`` in a Researcher→Writer→Reviewer pipeline).
     """
-    extractor = ExactMatchJudge(answer_strategy=answer_strategy, source_filter=source_filter)
+    extractor = ExactMatchJudge(
+        answer_strategy=answer_strategy,
+        source_filter=source_filter,
+        conversation_scope=conversation_scope,
+    )
 
     async def _check(trajectory: EvalTrajectory) -> CheckResult:
         response = extractor.extract_answer(trajectory)

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/history/models.py

@@ -12,6 +12,7 @@ import math
 import re
 from collections.abc import Callable
 from datetime import date, datetime, timezone
+from enum import StrEnum
 from typing import Any, Literal
 from uuid import uuid4
 
@@ -30,7 +31,20 @@ EVIDENCE_VERSION = 1
 
 RunStatus = Literal["queued", "running", "completed", "failed", "cancelled", "interrupted"]
 ItemState = Literal["pending", "running", "finished", "cancelled", "not_started"]
-CaptureMode = Literal["summary", "full"]
+
+
+class CaptureMode(StrEnum):
+    """How much evidence an evaluation run captures.
+
+    A ``StrEnum``: members compare and serialize as their plain string value
+    (``CaptureMode.FULL == "full"``), so callers may pass either the member or
+    the raw string.
+    """
+
+    SUMMARY = "summary"
+    FULL = "full"
+
+
 EvidenceStatus = Literal["not_captured", "captured", "redacted", "unavailable"]
 TaskType = Literal["task", "workflow", "multi_turn"]
 
@@ -71,7 +85,7 @@ class EvalRunRecord(BaseModel):
     schema_version: int = SCHEMA_VERSION
     status: RunStatus = "queued"
     revision: int = 0
-    capture: CaptureMode = "summary"
+    capture: CaptureMode = CaptureMode.SUMMARY
     dataset_id: str
     dataset_revision: str | None = None
     target_id: str

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/history/recorder.py

@@ -59,17 +59,21 @@ class EvalRunRecorder:
     def __init__(
         self,
         store: EvalRunStore,
-        capture: CaptureMode = "summary",
+        capture: CaptureMode = CaptureMode.SUMMARY,
         *,
         concurrency: int = 4,
         redact: Redactor | None = None,
         worker_id: str | None = None,
     ) -> None:
-        if capture not in ("summary", "full"):
-            raise ValueError("capture must be 'summary' or 'full'")
+        try:
+            capture = CaptureMode(capture)
+        except ValueError as exc:
+            raise ValueError(
+                f"capture must be one of {[m.value for m in CaptureMode]}, got {capture!r}"
+            ) from exc
         if concurrency < 1:
             raise ValueError("concurrency must be at least 1")
-        if redact is not None and capture != "full":
+        if redact is not None and capture != CaptureMode.FULL:
             raise ValueError("redact applies only to capture='full'")
         self.store = store
         self.capture = capture
@@ -315,7 +319,7 @@ class EvalRunRecorder:
     def _outcome(self, run_id: str, index: int, task: EvalTask, score: EvalScore) -> EvalItemRecord:
         evidence: EvalScore | None = None
         evidence_status: EvidenceStatus = "not_captured"
-        if self.capture == "full":
+        if self.capture == CaptureMode.FULL:
             evidence, evidence_status = score, "captured"
             if self.redact is not None:
                 try:

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/judges/composite.py

@@ -8,7 +8,7 @@ from collections.abc import Sequence
 from agentbyte.cancellation_token import CancellationToken
 
 from agentbyte.eval.base import BaseEvalJudge, JudgeScoringError
-from agentbyte.eval.types import EvalScore, EvalTrajectory
+from agentbyte.eval.types import AnswerStrategy, EvalScore, EvalTrajectory
 
 
 class CompositeJudge(BaseEvalJudge):
@@ -36,7 +36,7 @@ class CompositeJudge(BaseEvalJudge):
                 )
             normalized = list(judges)
 
-        super().__init__(name=name, answer_strategy="last_non_empty")
+        super().__init__(name=name, answer_strategy=AnswerStrategy.LAST_NON_EMPTY)
         self.judges = normalized
 
     async def score(

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/judges/llm.py

@@ -21,7 +21,7 @@ from agentbyte.eval.judges.validation import (
     resolve_criteria,
     validate_criterion_scores,
 )
-from agentbyte.eval.types import AnswerStrategy, EvalScore, EvalTrajectory
+from agentbyte.eval.types import AnswerStrategy, ConversationScope, EvalScore, EvalTrajectory
 
 
 class _JudgeCriterionScore(BaseModel):
@@ -51,8 +51,9 @@ class LLMEvalJudge(BaseEvalJudge):
         client: BaseChatCompletionClient,
         *,
         name: str | None = None,
-        answer_strategy: AnswerStrategy = "last_non_empty",
+        answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
         source_filter: str | None = None,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
         default_criteria: list[str] | None = None,
         custom_instructions: str | None = None,
         use_structured_output: bool = True,
@@ -61,6 +62,7 @@ class LLMEvalJudge(BaseEvalJudge):
             name=name or f"LLM-{client.model}",
             answer_strategy=answer_strategy,
             source_filter=source_filter,
+            conversation_scope=conversation_scope,
         )
         require_non_empty_defaults(default_criteria, "default_criteria")
         self.client = client

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/judges/pairwise.py

@@ -15,7 +15,13 @@ from agentbyte.messages import SystemMessage, UserMessage
 
 from agentbyte.eval.base import BaseEvalJudge, JudgeScoringError
 from agentbyte.eval.judges.validation import loads_judge_json, require_non_empty_defaults
-from agentbyte.eval.types import AnswerStrategy, EvalScore, EvalTrajectory, PairwiseResult
+from agentbyte.eval.types import (
+    AnswerStrategy,
+    ConversationScope,
+    EvalScore,
+    EvalTrajectory,
+    PairwiseResult,
+)
 
 
 class _PairwiseResponse(BaseModel):
@@ -47,14 +53,16 @@ class PairwiseJudge(BaseEvalJudge):
         *,
         name: str | None = None,
         criteria: list[str] | None = None,
-        answer_strategy: AnswerStrategy = "last_assistant",
+        answer_strategy: AnswerStrategy = AnswerStrategy.LAST_ASSISTANT,
         source_filter: str | None = None,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
         custom_instructions: str | None = None,
     ) -> None:
         super().__init__(
             name=name or f"Pairwise-{client.model}",
             answer_strategy=answer_strategy,
             source_filter=source_filter,
+            conversation_scope=conversation_scope,
         )
         require_non_empty_defaults(criteria, "criteria")
         self.client = client

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/judges/reference.py

@@ -7,7 +7,7 @@ from difflib import SequenceMatcher
 from agentbyte.cancellation_token import CancellationToken
 
 from agentbyte.eval.base import BaseEvalJudge
-from agentbyte.eval.types import AnswerStrategy, EvalScore, EvalTrajectory
+from agentbyte.eval.types import AnswerStrategy, ConversationScope, EvalScore, EvalTrajectory
 
 
 def _failed_score(judge_name: str, trajectory: EvalTrajectory) -> EvalScore:
@@ -28,12 +28,18 @@ class ExactMatchJudge(BaseEvalJudge):
         self,
         *,
         name: str = "ExactMatch",
-        answer_strategy: AnswerStrategy = "last_non_empty",
+        answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
         source_filter: str | None = None,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
         case_sensitive: bool = False,
         strip_whitespace: bool = True,
     ) -> None:
-        super().__init__(name=name, answer_strategy=answer_strategy, source_filter=source_filter)
+        super().__init__(
+            name=name,
+            answer_strategy=answer_strategy,
+            source_filter=source_filter,
+            conversation_scope=conversation_scope,
+        )
         self.case_sensitive = case_sensitive
         self.strip_whitespace = strip_whitespace
 
@@ -76,11 +82,17 @@ class ContainsJudge(BaseEvalJudge):
         self,
         *,
         name: str = "Contains",
-        answer_strategy: AnswerStrategy = "last_non_empty",
+        answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
         source_filter: str | None = None,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
         case_sensitive: bool = False,
     ) -> None:
-        super().__init__(name=name, answer_strategy=answer_strategy, source_filter=source_filter)
+        super().__init__(
+            name=name,
+            answer_strategy=answer_strategy,
+            source_filter=source_filter,
+            conversation_scope=conversation_scope,
+        )
         self.case_sensitive = case_sensitive
 
     async def score(
@@ -123,14 +135,20 @@ class FuzzyMatchJudge(BaseEvalJudge):
         self,
         *,
         name: str = "FuzzyMatch",
-        answer_strategy: AnswerStrategy = "last_non_empty",
+        answer_strategy: AnswerStrategy = AnswerStrategy.LAST_NON_EMPTY,
         source_filter: str | None = None,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
         case_sensitive: bool = False,
         threshold: float = 0.8,
     ) -> None:
         if not 0 <= threshold <= 1:
             raise ValueError("threshold must be within [0, 1]")
-        super().__init__(name=name, answer_strategy=answer_strategy, source_filter=source_filter)
+        super().__init__(
+            name=name,
+            answer_strategy=answer_strategy,
+            source_filter=source_filter,
+            conversation_scope=conversation_scope,
+        )
         self.case_sensitive = case_sensitive
         self.threshold = threshold
 

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/judges/trajectory.py

@@ -18,7 +18,7 @@ from agentbyte.eval.judges.validation import (
     resolve_criteria,
     validate_criterion_scores,
 )
-from agentbyte.eval.types import EvalScore, EvalTrajectory
+from agentbyte.eval.types import AnswerStrategy, ConversationScope, EvalScore, EvalTrajectory
 
 
 class ProcessCriterionScore(BaseModel):
@@ -49,8 +49,13 @@ class LLMTrajectoryJudge(BaseEvalJudge):
         default_criteria: list[str] | None = None,
         custom_instructions: str | None = None,
         max_payload_chars: int = 100_000,
+        conversation_scope: ConversationScope = ConversationScope.FULL,
     ) -> None:
-        super().__init__(name, answer_strategy="last_assistant")
+        super().__init__(
+            name,
+            answer_strategy=AnswerStrategy.LAST_ASSISTANT,
+            conversation_scope=conversation_scope,
+        )
         if max_payload_chars <= 0:
             raise ValueError("max_payload_chars must be positive")
         require_non_empty_defaults(default_criteria, "default_criteria")

{agentbyte-0.34.0 → agentbyte-0.38.0}/src/agentbyte/eval/types.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import math
+from enum import StrEnum
 from typing import Any, Literal
 
 from pydantic import (
@@ -17,12 +18,31 @@ from agentbyte.execution_trace import ExecutionTrace
 
 from agentbyte.messages import Message, Usage
 
-AnswerStrategy = Literal[
-    "last_non_empty",
-    "last_assistant",
-    "last_content",
-    "all_assistant",
-]
+class AnswerStrategy(StrEnum):
+    """Which message in a trajectory counts as the answer to be judged.
+
+    A ``StrEnum``: members compare and serialize as their plain string value
+    (``AnswerStrategy.LAST_ASSISTANT == "last_assistant"``), so callers may pass
+    either the member or the raw string.
+    """
+
+    LAST_NON_EMPTY = "last_non_empty"
+    LAST_ASSISTANT = "last_assistant"
+    LAST_CONTENT = "last_content"
+    ALL_ASSISTANT = "all_assistant"
+
+
+class ConversationScope(StrEnum):
+    """Which part of a multi-turn conversation a judge evaluates.
+
+    Applied before ``source_filter`` and ``AnswerStrategy``, this narrows the
+    trajectory to a turn window first. A ``StrEnum``: members compare and
+    serialize as their plain string value (``ConversationScope.LAST_TURN ==
+    "last_turn"``), so callers may pass either the member or the raw string.
+    """
+
+    FULL = "full"
+    LAST_TURN = "last_turn"
 
 
 class ExpectedToolCall(BaseModel):
@@ -287,6 +307,7 @@ class PairwiseResult(BaseModel):
 __all__ = [
     "WorkflowEvalTask",
     "AnswerStrategy",
+    "ConversationScope",
     "ScoringStatus",
     "ExpectedToolCall",
     "EvalTask",