agentbyte 0.47.0__tar.gz → 0.48.0__tar.gz

{agentbyte-0.47.0 → agentbyte-0.48.0}/CHANGELOG.md

@@ -6,6 +6,33 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.48.0] - 2026-10-06
+
+### Added
+
+- Evaluation suite workspace and score-aware optimization outcome (spec 0069): `suite` is a
+  first-class, indexed column on eval runs (`list_runs(suite=)`, `GET /api/eval/runs?suite=`), and
+  `OptimizationReport`/`OptimizationRunRecord` expose a shared `OptimizationOutcome`
+  (`improved` / `seed_kept` / `best_candidate_changed`) via `classify_outcome`.
+- Agent as a first-class history dimension (spec 0070): `agent_id`/`agent_name` on `EvalRunRecord`
+  and `OptimizationRunRecord` (indexed `agent_id`, defaulted from `target_id`), `list_runs(agent=)`
+  / `list_optimizations(agent=)`, `GET /api/eval/agents` and `?agent=` filters. `EvalSuite` declares
+  `agent_id`/`agent_name`/`agent_description`; `EvalSuiteRegistry` builds agent cards and raises
+  `DuplicateAgentError` when one `agent_id` maps to two names. The WebUI Evaluations tab is now an
+  agent-card gallery with per-agent pages (suites + Run, grouped history, server-side pagination).
+- `SqlEvalRunStore.from_sqlite_path` / `from_url` convenience constructors that own the engine
+  (spec 0071).
+
+### Changed
+
+- One storage backend for evaluation history (spec 0071): `SqlEvalRunStore` (SQLAlchemy Core) now
+  serves both SQLite and PostgreSQL. The hand-written raw-SQL SQLite store was retired;
+  `SQLiteEvalRunStore` is kept as a thin backward-compatible alias. SQLite gets a single-writer file
+  lock, `PRAGMA synchronous=FULL` and schema auto-migration; the row format is unchanged, so existing
+  `.sqlite` files open with no data migration. History store schema migrates forward to layout 4.
+- The `sqlite` extra now also provides `sqlalchemy[asyncio]` (needed by the unified store); `import
+  agentbyte` with core dependencies only is unchanged.
+
 ## [0.47.0] - 2026-10-06
 
 ### Added

{agentbyte-0.47.0 → agentbyte-0.48.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.47.0
+Version: 0.48.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -70,6 +70,7 @@ Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'sql'
 Requires-Dist: sqlmodel>=0.0.38; extra == 'sql'
 Provides-Extra: sqlite
 Requires-Dist: aiosqlite>=0.22.1; extra == 'sqlite'
+Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'sqlite'
 Provides-Extra: tiktoken
 Requires-Dist: tiktoken>=0.9.0; extra == 'tiktoken'
 Provides-Extra: viz
@@ -83,7 +84,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.47.0**
+Current release: **0.48.0**
 
 ## Building an Agent
 
@@ -317,8 +318,8 @@ uv add "agentbyte[openai,sqlite]"
 | `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
 | `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
 | `gunicorn` | `webui` + `gunicorn` | Production single-worker serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`) |
-| `sqlite` | `aiosqlite` | SQLite datasets, workflow checkpoints, evaluation history |
-| `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, `SqlEvalRunStore` |
+| `sqlite` | `aiosqlite`, `sqlalchemy[asyncio]` | SQLite datasets, workflow checkpoints, evaluation history (`SqlEvalRunStore` over `sqlite+aiosqlite`) |
+| `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, PostgreSQL `SqlEvalRunStore` |
 | `sql` | `sqlite` + `postgres` | Both SQL backends |
 | `tiktoken` | `tiktoken` | `TiktokenCounter` for exact compaction token counts |
 | `optim` | `gepa` | `optimize_with_gepa()` |

{agentbyte-0.47.0 → agentbyte-0.48.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.47.0**
+Current release: **0.48.0**
 
 ## Building an Agent
 
@@ -236,8 +236,8 @@ uv add "agentbyte[openai,sqlite]"
 | `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
 | `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
 | `gunicorn` | `webui` + `gunicorn` | Production single-worker serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`) |
-| `sqlite` | `aiosqlite` | SQLite datasets, workflow checkpoints, evaluation history |
-| `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, `SqlEvalRunStore` |
+| `sqlite` | `aiosqlite`, `sqlalchemy[asyncio]` | SQLite datasets, workflow checkpoints, evaluation history (`SqlEvalRunStore` over `sqlite+aiosqlite`) |
+| `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, PostgreSQL `SqlEvalRunStore` |
 | `sql` | `sqlite` + `postgres` | Both SQL backends |
 | `tiktoken` | `tiktoken` | `TiktokenCounter` for exact compaction token counts |
 | `optim` | `gepa` | `optimize_with_gepa()` |

{agentbyte-0.47.0 → agentbyte-0.48.0}/pyproject.toml

@@ -42,6 +42,7 @@ gunicorn = [
 ]
 sqlite = [
     "aiosqlite>=0.22.1",
+    "sqlalchemy[asyncio]>=2.0.43",
 ]
 postgres = [
     "sqlalchemy[asyncio]>=2.0.43",

agentbyte-0.48.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.48.0"
+VERSION = __version__

{agentbyte-0.47.0 → agentbyte-0.48.0}/src/agentbyte/eval/base.py

@@ -46,8 +46,12 @@ class JudgeScoringError(Exception):
 class BaseEvalTarget(ABC):
     """Anything that can execute an evaluation task into a trajectory."""
 
-    def __init__(self, name: str):
+    def __init__(self, name: str, *, agent_id: str | None = None, agent_name: str | None = None):
         self.name = name
+        # Optional stable agent identity (spec 0070). ``agent_name`` defaults to the target name;
+        # ``agent_id`` falls back to the name when not supplied by the caller or suite.
+        self.agent_id = agent_id
+        self.agent_name = agent_name or name
 
     @abstractmethod
     async def run(

agentbyte-0.48.0/src/agentbyte/eval/history/_writer_lock.py

@@ -0,0 +1,46 @@
+"""Exclusive single-writer file lock for file-backed history databases (SQLite).
+
+Shared by the SQLite path of ``SqlEvalRunStore`` (spec 0071). PostgreSQL uses session-level
+advisory locks instead; SQLite has no server, so a ``<path>.writer.lock`` file holds exclusive
+writer ownership for a store's lifetime and the OS releases it on process exit.
+"""
+
+from __future__ import annotations
+
+import os
+
+from .errors import EvalStoreOwnershipError
+
+
+class WriterLock:
+    """Exclusive per-database writer ownership, released by the OS on process exit."""
+
+    def __init__(self, path: str) -> None:
+        self.path = f"{path}.writer.lock"
+        self._fd: int | None = None
+
+    def acquire(self) -> None:
+        fd = os.open(self.path, os.O_RDWR | os.O_CREAT, 0o600)
+        try:
+            try:
+                import fcntl
+
+                fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+            except ImportError:  # pragma: no cover - Windows
+                import msvcrt
+
+                msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
+        except OSError as exc:
+            os.close(fd)
+            raise EvalStoreOwnershipError(
+                f"Another writer owns {self.path.removesuffix('.writer.lock')}"
+            ) from exc
+        self._fd = fd
+
+    def release(self) -> None:
+        fd, self._fd = self._fd, None
+        if fd is not None:
+            os.close(fd)
+
+
+__all__ = ["WriterLock"]

{agentbyte-0.47.0 → agentbyte-0.48.0}/src/agentbyte/eval/history/conformance.py

@@ -282,6 +282,47 @@ async def check_queries_paginate_filter_and_delete(store: EvalRunStore) -> None:
     assert await store.delete_run(runs[0].id) is False
 
 
+async def check_suite_is_derived_from_labels_and_filterable(store: EvalRunStore) -> None:
+    from .models import SUITE_LABEL_KEY
+
+    alpha = _run(labels={SUITE_LABEL_KEY: "alpha"})
+    beta = _run(labels={SUITE_LABEL_KEY: "beta"})
+    unlabelled = _run()
+    assert (alpha.suite, beta.suite, unlabelled.suite) == ("alpha", "beta", None)
+    explicit = _run(suite="gamma", labels={SUITE_LABEL_KEY: "alpha"})
+    assert explicit.suite == "gamma"  # an explicit suite wins over the label
+
+    for run in (alpha, beta, unlabelled):
+        await store.create_run(run, _items(run, 1))
+
+    assert [r.id for r in await store.list_runs(suite="alpha")] == [alpha.id]
+    assert [r.id for r in await store.list_runs(suite="beta")] == [beta.id]
+    assert await store.list_runs(suite="absent") == []
+    assert {r.id for r in await store.list_runs()} == {alpha.id, beta.id, unlabelled.id}
+
+
+async def check_agent_is_recorded_and_filterable(store: EvalRunStore) -> None:
+    # spec 0070: agent_id defaults from target_id and is filterable, orthogonally to suite.
+    from .models import SUITE_LABEL_KEY
+
+    geo = _run(agent_id="geo-support", agent_name="Geo Support", labels={SUITE_LABEL_KEY: "capitals"})
+    acme = _run(agent_id="acme-support", labels={SUITE_LABEL_KEY: "capitals"})
+    defaulted = _run()  # no agent_id -> falls back to target_id ("t")
+    assert (geo.agent_id, geo.agent_name) == ("geo-support", "Geo Support")
+    assert (acme.agent_id, acme.agent_name) == ("acme-support", "acme-support")  # name defaults to id
+    assert defaulted.agent_id == "t"  # the target_id default
+
+    for run in (geo, acme, defaulted):
+        await store.create_run(run, _items(run, 1))
+
+    assert [r.id for r in await store.list_runs(agent="geo-support")] == [geo.id]
+    assert [r.id for r in await store.list_runs(agent="t")] == [defaulted.id]
+    assert await store.list_runs(agent="absent") == []
+    # agent and suite filter independently
+    assert [r.id for r in await store.list_runs(agent="geo-support", suite="capitals")] == [geo.id]
+    assert await store.list_runs(agent="geo-support", suite="other") == []
+
+
 async def check_unknown_run_is_reported(store: EvalRunStore) -> None:
     assert await store.get_run("missing") is None
     await _raises(EvalRunNotFoundError, store.start_run("missing"))

{agentbyte-0.47.0 → agentbyte-0.48.0}/src/agentbyte/eval/history/models.py

@@ -16,7 +16,7 @@ from enum import StrEnum
 from typing import Any
 from uuid import uuid4
 
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, computed_field, model_validator
 
 from agentbyte.eval.types import EvalScore, EvalTask, MultiTurnEvalTask, ScoringStatus, WorkflowEvalTask
 from agentbyte.execution_trace.models import ExecutionStatus
@@ -27,8 +27,14 @@ from .errors import EvalSchemaVersionError, EvalSerializationError
 SCHEMA_VERSION = 1
 """Version of the stored run/item record layout."""
 
-STORE_SCHEMA_VERSION = 2
-"""Version of the history database layout. 2 adds the ``optimization_runs`` table (spec 0068)."""
+STORE_SCHEMA_VERSION = 4
+"""Version of the history database layout. 2 adds the ``optimization_runs`` table (spec 0068);
+3 adds an indexed ``suite`` column on eval runs for suite-scoped listing (spec 0069); 4 adds an
+indexed ``agent_id`` column on eval runs and optimization runs for agent-scoped listing (spec 0070)."""
+
+SUITE_LABEL_KEY = "agentbyte.suite"
+"""Run-label key carrying the suite name. Workbench's ``SUITE_LABEL`` re-exports this so the
+history layer can derive ``EvalRunRecord.suite`` without importing the workbench package."""
 
 EVIDENCE_VERSION = 1
 """Version of the serialized full-capture evidence envelope."""
@@ -134,10 +140,40 @@ class EvalRunRecord(BaseModel):
     judge_type: str
     criteria: list[str] | None = None
     labels: dict[str, str] = Field(default_factory=dict)
+    suite: str | None = None
+    agent_id: str | None = None
+    agent_name: str | None = None
     owner: str | None = None
     worker_id: str | None = None
     idempotency_key: str | None = None
     previous_run_id: str | None = None
+
+    @model_validator(mode="before")
+    @classmethod
+    def _derive_suite_from_labels(cls, data: Any) -> Any:
+        """Default ``suite`` from the suite label so suite-scoped queries work for every run."""
+        if isinstance(data, dict) and not data.get("suite"):
+            labels = data.get("labels") or {}
+            if isinstance(labels, dict) and labels.get(SUITE_LABEL_KEY):
+                data = {**data, "suite": labels[SUITE_LABEL_KEY]}
+        return data
+
+    @model_validator(mode="before")
+    @classmethod
+    def _default_agent_from_target(cls, data: Any) -> Any:
+        """Default agent identity from ``target_id`` so agent-scoped queries work for every run.
+
+        ``agent_id`` is the machine-readable identity; when a caller does not supply one it falls
+        back to ``target_id`` (the agent's name). ``agent_name`` is a human label defaulting to the
+        resolved ``agent_id`` (spec 0070).
+        """
+        if isinstance(data, dict):
+            agent_id = data.get("agent_id") or data.get("target_id")
+            if agent_id and not data.get("agent_id"):
+                data = {**data, "agent_id": agent_id}
+            if agent_id and not data.get("agent_name"):
+                data = {**data, "agent_name": agent_id}
+        return data
     created_at: datetime = Field(default_factory=utc_now)
     started_at: datetime | None = None
     finished_at: datetime | None = None
@@ -315,6 +351,8 @@ class OptimizationRunRecord(BaseModel):
 
     id: str = Field(default_factory=lambda: str(uuid4()))
     suite: str
+    agent_id: str | None = None
+    agent_name: str | None = None
     created_at: datetime = Field(default_factory=utc_now)
     title: str = "Optimization run"
     optimizer: str | None = None
@@ -328,23 +366,50 @@ class OptimizationRunRecord(BaseModel):
     pass_score: float = 10.0
     trace: dict[str, Any] | None = None
 
+    @computed_field  # serialized so the suite listing shows the outcome without the full trace
+    @property
+    def outcome(self) -> str:
+        """Score-aware outcome string, from the same shared classifier the report uses (spec 0069)."""
+        from agentbyte.optim.report import classify_outcome
+
+        return str(classify_outcome(self.seed, self.best, self.seed_avg, self.best_avg))
+
     @classmethod
     def from_result(
-        cls, result: Any, *, suite: str, pass_score: float = 10.0, title: str | None = None
+        cls,
+        result: Any,
+        *,
+        suite: str,
+        pass_score: float = 10.0,
+        title: str | None = None,
+        agent_id: str | None = None,
+        agent_name: str | None = None,
     ) -> OptimizationRunRecord:
-        """Record an ``OptimizationResult`` under ``suite``."""
-        return cls.from_trace(result.trace, suite=suite, pass_score=pass_score, title=title)
+        """Record an ``OptimizationResult`` under ``suite`` (and optionally an agent, spec 0070)."""
+        return cls.from_trace(
+            result.trace, suite=suite, pass_score=pass_score, title=title,
+            agent_id=agent_id, agent_name=agent_name,
+        )
 
     @classmethod
     def from_trace(
-        cls, trace: Any, *, suite: str, pass_score: float = 10.0, title: str | None = None
+        cls,
+        trace: Any,
+        *,
+        suite: str,
+        pass_score: float = 10.0,
+        title: str | None = None,
+        agent_id: str | None = None,
+        agent_name: str | None = None,
     ) -> OptimizationRunRecord:
-        """Record an ``OptimizationTrace`` (live or loaded) under ``suite``."""
+        """Record an ``OptimizationTrace`` (live or loaded) under ``suite`` (and optional agent)."""
         from agentbyte.optim.report import build_report
 
         report = build_report(trace, title=title or suite, pass_score=pass_score)
         return cls(
             suite=suite,
+            agent_id=agent_id,
+            agent_name=agent_name or agent_id,
             title=report.title,
             optimizer=report.optimizer,
             seed=report.seed,

{agentbyte-0.47.0 → agentbyte-0.48.0}/src/agentbyte/eval/history/recorder.py

@@ -92,6 +92,8 @@ class EvalRunRecorder:
         dataset_revision: str | None = None,
         criteria: list[str] | None = None,
         target_fingerprint: str | None = None,
+        agent_id: str | None = None,
+        agent_name: str | None = None,
         labels: dict[str, str] | None = None,
         owner: str | None = None,
         previous_run_id: str | None = None,
@@ -106,8 +108,8 @@ class EvalRunRecorder:
         run, _ = await self.admit(
             runner, tasks, dataset_id=dataset_id, target_id=target_id,
             dataset_revision=dataset_revision, criteria=criteria,
-            target_fingerprint=target_fingerprint, labels=labels, owner=owner,
-            previous_run_id=previous_run_id,
+            target_fingerprint=target_fingerprint, agent_id=agent_id, agent_name=agent_name,
+            labels=labels, owner=owner, previous_run_id=previous_run_id,
         )
         async with contextlib.aclosing(
             self.execute_stream(
@@ -132,6 +134,8 @@ class EvalRunRecorder:
         dataset_revision: str | None = None,
         criteria: list[str] | None = None,
         target_fingerprint: str | None = None,
+        agent_id: str | None = None,
+        agent_name: str | None = None,
         labels: dict[str, str] | None = None,
         owner: str | None = None,
         previous_run_id: str | None = None,
@@ -148,8 +152,8 @@ class EvalRunRecorder:
         run, _ = await self.admit(
             runner, tasks, dataset_id=dataset_id, target_id=target_id,
             dataset_revision=dataset_revision, criteria=criteria,
-            target_fingerprint=target_fingerprint, labels=labels, owner=owner,
-            previous_run_id=previous_run_id,
+            target_fingerprint=target_fingerprint, agent_id=agent_id, agent_name=agent_name,
+            labels=labels, owner=owner, previous_run_id=previous_run_id,
         )
         async with contextlib.aclosing(
             self.execute_stream(
@@ -169,6 +173,8 @@ class EvalRunRecorder:
         dataset_revision: str | None = None,
         criteria: list[str] | None = None,
         target_fingerprint: str | None = None,
+        agent_id: str | None = None,
+        agent_name: str | None = None,
         labels: dict[str, str] | None = None,
         owner: str | None = None,
         previous_run_id: str | None = None,
@@ -189,6 +195,8 @@ class EvalRunRecorder:
             dataset_revision=dataset_revision,
             target_id=target_id,
             target_fingerprint=target_fingerprint,
+            agent_id=agent_id,
+            agent_name=agent_name,
             judge_name=runner.judge.name,
             judge_type=type(runner.judge).__qualname__,
             criteria=list(criteria) if criteria is not None else None,