agentbyte 0.31.0__tar.gz → 0.32.0__tar.gz

{agentbyte-0.31.0 → agentbyte-0.32.0}/CHANGELOG.md

@@ -6,6 +6,19 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.32.0] - 2026-09-30
+
+Upgrade notes (breaking):
+
+- Optional extras are grouped by database driver. `dataset`, `workflow-sqlite` and `eval-history` are removed; install `agentbyte[sqlite]` instead. pip only warns about an unknown extra, so update install lines before upgrading.
+- New `postgres` extra (`sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg`) for `SqlUsageBackend` and `SqlEvalRunStore`; `sql` is now `sqlite` + `postgres` and installs the same packages as before, plus `greenlet` via `sqlalchemy[asyncio]`.
+
+### Added
+
+- Evaluation history (spec 0051): new `agentbyte.eval.history` with `EvalRunRecorder` (bounded, index-ordered scheduling; each outcome committed before it is yielded; cancellation and stream close account for every planned item), `EvalRunStore`, `MemoryEvalRunStore` and native-async `SQLiteEvalRunStore` (new `sqlite` extra). Summary capture by default (no transcripts, reasoning or error text); opt-in full capture with a redaction hook and data-only reload of structured output (`StructuredPayload`). Transactional item/counter/revision commits, idempotent replays, explicit `recover_interrupted()`, exclusive single-writer ownership, versioned schema and evidence, atomic owner-scoped admission. Errors: `EvalPersistenceError`, `EvalSerializationError`, `EvalSchemaVersionError`, `EvalStoreOwnershipError`, `EvalQueueFullError`, `EvalHistoryConflictError`.
+- Swappable evaluation history backends: `SqlEvalRunStore` for PostgreSQL through a caller-owned SQLAlchemy session factory (`postgres` extra; row locks, advisory-lock single writer, optional schema, `history_metadata()` for Alembic), public `agentbyte.eval.history.transitions`, and the reusable `agentbyte.eval.history.conformance` suite for custom stores.
+- `EvalRunner.evaluate_task()`: public single-task scoring used by the recorder (previously private `_evaluate_single_task`).
+
 ## [0.31.0] - 2026-09-30
 
 Upgrade notes (breaking):

{agentbyte-0.31.0 → agentbyte-0.32.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.31.0
+Version: 0.32.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -25,7 +25,7 @@ Requires-Dist: openai>=1.107.1; extra == 'all'
 Requires-Dist: opentelemetry-api>=1.39.1; extra == 'all'
 Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.39.1; extra == 'all'
 Requires-Dist: opentelemetry-sdk>=1.39.1; extra == 'all'
-Requires-Dist: sqlalchemy>=2.0.43; extra == 'all'
+Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'all'
 Requires-Dist: sqlmodel>=0.0.38; extra == 'all'
 Requires-Dist: uvicorn>=0.42.0; extra == 'all'
 Requires-Dist: youtube-transcript-api>=0.6; extra == 'all'
@@ -35,8 +35,6 @@ Provides-Extra: azureopenai
 Requires-Dist: azure-identity>=1.25.1; extra == 'azureopenai'
 Requires-Dist: openai>=1.107.1; extra == 'azureopenai'
 Provides-Extra: coding
-Provides-Extra: dataset
-Requires-Dist: aiosqlite>=0.22.1; extra == 'dataset'
 Provides-Extra: dev
 Requires-Dist: ipykernel>=7.1.0; extra == 'dev'
 Requires-Dist: nbclient>=0.10.4; extra == 'dev'
@@ -53,6 +51,10 @@ Provides-Extra: otel
 Requires-Dist: opentelemetry-api>=1.39.1; extra == 'otel'
 Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.39.1; extra == 'otel'
 Requires-Dist: opentelemetry-sdk>=1.39.1; extra == 'otel'
+Provides-Extra: postgres
+Requires-Dist: asyncpg>=0.31.0; extra == 'postgres'
+Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'postgres'
+Requires-Dist: sqlmodel>=0.0.38; extra == 'postgres'
 Provides-Extra: research
 Requires-Dist: arxiv>=2.1; extra == 'research'
 Requires-Dist: beautifulsoup4>=4.12; extra == 'research'
@@ -61,8 +63,10 @@ Requires-Dist: youtube-transcript-api>=0.6; extra == 'research'
 Provides-Extra: sql
 Requires-Dist: aiosqlite>=0.22.1; extra == 'sql'
 Requires-Dist: asyncpg>=0.31.0; extra == 'sql'
-Requires-Dist: sqlalchemy>=2.0.43; extra == 'sql'
+Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'sql'
 Requires-Dist: sqlmodel>=0.0.38; extra == 'sql'
+Provides-Extra: sqlite
+Requires-Dist: aiosqlite>=0.22.1; extra == 'sqlite'
 Provides-Extra: test
 Requires-Dist: aiosqlite>=0.22.1; extra == 'test'
 Requires-Dist: arxiv>=2.1; extra == 'test'
@@ -80,15 +84,13 @@ Requires-Dist: graphviz>=0.21; extra == 'viz'
 Provides-Extra: webui
 Requires-Dist: fastapi>=0.135.2; extra == 'webui'
 Requires-Dist: uvicorn[standard]>=0.44.0; extra == 'webui'
-Provides-Extra: workflow-sqlite
-Requires-Dist: aiosqlite>=0.22.1; extra == 'workflow-sqlite'
 Description-Content-Type: text/markdown
 
 # Agentbyte
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.31.0**
+Current release: **0.32.0**
 
 ## Building an Agent
 
@@ -307,10 +309,17 @@ uv sync --all-groups
 Optional extras:
 
 ```bash
-uv sync --extra openai
-uv sync --extra azureopenai
-uv sync --extra otel
-uv sync --extra webui
+uv sync --extra openai        # OpenAI client
+uv sync --extra azureopenai   # Azure OpenAI client + azure-identity
+uv sync --extra aws           # S3 datasets (aioboto3)
+uv sync --extra otel          # OpenTelemetry export
+uv sync --extra webui         # WebUI playground (FastAPI)
+uv sync --extra sqlite        # SQLite stores: datasets, workflow checkpoints, eval history
+uv sync --extra postgres      # PostgreSQL stores: usage logging, eval history
+uv sync --extra sql           # sqlite + postgres
+uv sync --extra optim         # prompt optimization (GEPA)
+uv sync --extra viz           # workflow diagrams (graphviz)
+uv sync --extra research      # research tools
 ```
 
 ### Install in another project (pip / uv add)

{agentbyte-0.31.0 → agentbyte-0.32.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.31.0**
+Current release: **0.32.0**
 
 ## Building an Agent
 
@@ -221,10 +221,17 @@ uv sync --all-groups
 Optional extras:
 
 ```bash
-uv sync --extra openai
-uv sync --extra azureopenai
-uv sync --extra otel
-uv sync --extra webui
+uv sync --extra openai        # OpenAI client
+uv sync --extra azureopenai   # Azure OpenAI client + azure-identity
+uv sync --extra aws           # S3 datasets (aioboto3)
+uv sync --extra otel          # OpenTelemetry export
+uv sync --extra webui         # WebUI playground (FastAPI)
+uv sync --extra sqlite        # SQLite stores: datasets, workflow checkpoints, eval history
+uv sync --extra postgres      # PostgreSQL stores: usage logging, eval history
+uv sync --extra sql           # sqlite + postgres
+uv sync --extra optim         # prompt optimization (GEPA)
+uv sync --extra viz           # workflow diagrams (graphviz)
+uv sync --extra research      # research tools
 ```
 
 ### Install in another project (pip / uv add)

{agentbyte-0.31.0 → agentbyte-0.32.0}/pyproject.toml

@@ -38,17 +38,16 @@ webui = [
     "fastapi>=0.135.2",
     "uvicorn[standard]>=0.44.0",
 ]
-dataset = [
-    "aiosqlite>=0.22.1",
-]
-workflow-sqlite = [
+sqlite = [
     "aiosqlite>=0.22.1",
 ]
-sql = [
-    "sqlalchemy>=2.0.43",
+postgres = [
+    "sqlalchemy[asyncio]>=2.0.43",
     "sqlmodel>=0.0.38",
     "asyncpg>=0.31.0",
-    "aiosqlite>=0.22.1",
+]
+sql = [
+    "agentbyte[sqlite,postgres]",
 ]
 optim = [
     "gepa>=0.1",
@@ -89,7 +88,7 @@ all = [
     "openai>=1.107.1",
     "azure-identity>=1.25.1",
     "aioboto3>=15.5.0",
-    "sqlalchemy>=2.0.43",
+    "sqlalchemy[asyncio]>=2.0.43",
     "sqlmodel>=0.0.38",
     "asyncpg>=0.31.0",
     "aiosqlite>=0.22.1",
@@ -197,7 +196,7 @@ all = [
     "openai>=1.107.1",
     "azure-identity>=1.25.1",
     "aioboto3>=15.5.0",
-    "sqlalchemy>=2.0.43",
+    "sqlalchemy[asyncio]>=2.0.43",
     "sqlmodel>=0.0.38",
     "asyncpg>=0.31.0",
     "aiosqlite>=0.22.1",

agentbyte-0.32.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.32.0"
+VERSION = __version__

agentbyte-0.32.0/src/agentbyte/eval/history/__init__.py

@@ -0,0 +1,75 @@
+"""Durable evaluation history (spec 0051).
+
+``SQLiteEvalRunStore`` needs the ``sqlite`` extra (``aiosqlite``); the driver
+is imported only when a store is opened. ``SqlEvalRunStore`` (PostgreSQL through a
+caller-owned SQLAlchemy session factory) needs the ``postgres`` extra and is loaded on
+first access.
+"""
+
+from typing import Any
+
+from .errors import (
+    EvalHistoryConflictError,
+    EvalHistoryError,
+    EvalPersistenceError,
+    EvalQueueFullError,
+    EvalRedactionError,
+    EvalRunNotFoundError,
+    EvalSchemaVersionError,
+    EvalSerializationError,
+    EvalStoreOwnershipError,
+)
+from .models import (
+    EVIDENCE_VERSION,
+    SCHEMA_VERSION,
+    CaptureMode,
+    EvalItemRecord,
+    EvalRunRecord,
+    EvidenceStatus,
+    ItemState,
+    Redactor,
+    RunStatus,
+    StructuredPayload,
+)
+from .recorder import EvalRunRecorder
+from .sqlite import SQLiteEvalRunStore
+from .store import EvalRunStore, MemoryEvalRunStore
+
+_SQL_EXPORTS = {"SqlEvalRunStore", "history_metadata"}
+
+
+def __getattr__(name: str) -> Any:
+    if name in _SQL_EXPORTS:
+        from . import sql
+
+        return getattr(sql, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+__all__ = [
+    "EVIDENCE_VERSION",
+    "SCHEMA_VERSION",
+    "CaptureMode",
+    "EvalHistoryConflictError",
+    "EvalHistoryError",
+    "EvalItemRecord",
+    "EvalPersistenceError",
+    "EvalQueueFullError",
+    "EvalRedactionError",
+    "EvalRunNotFoundError",
+    "EvalRunRecord",
+    "EvalRunRecorder",
+    "EvalRunStore",
+    "EvalSchemaVersionError",
+    "EvalSerializationError",
+    "EvalStoreOwnershipError",
+    "EvidenceStatus",
+    "ItemState",
+    "MemoryEvalRunStore",
+    "Redactor",
+    "RunStatus",
+    "SQLiteEvalRunStore",
+    "SqlEvalRunStore",
+    "StructuredPayload",
+    "history_metadata",
+]

agentbyte-0.32.0/src/agentbyte/eval/history/conformance.py

@@ -0,0 +1,341 @@
+"""Behavioral checks every ``EvalRunStore`` backend must pass.
+
+Framework-free: each check is an async function that receives a fresh, empty,
+open store and raises ``AssertionError`` on a contract violation. Use
+``run_conformance`` or parametrize your own test runner over ``CHECKS``::
+
+    from agentbyte.eval.history.conformance import CHECKS
+
+    @pytest.mark.parametrize("check", CHECKS.values(), ids=CHECKS.keys())
+    async def test_my_store(check):
+        async with open_my_empty_store() as store:
+            await check(store)
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Awaitable, Callable
+from contextlib import AbstractAsyncContextManager
+
+from pydantic import BaseModel
+
+from agentbyte.eval.base import BaseEvalJudge, BaseEvalTarget, JudgeScoringError
+from agentbyte.eval.runner import EvalRunner
+from agentbyte.eval.types import EvalScore, EvalTask, EvalTrajectory, WorkflowEvalTask
+from agentbyte.execution_trace.models import ExecutionEvent, ExecutionSpan, ExecutionTrace
+from agentbyte.messages import AssistantMessage, UserMessage, Usage
+
+from .errors import (
+    EvalHistoryConflictError,
+    EvalQueueFullError,
+    EvalRunNotFoundError,
+    EvalSerializationError,
+)
+from .models import EvalItemRecord, EvalRunRecord, StructuredPayload
+from .recorder import EvalRunRecorder
+from .store import EvalRunStore
+
+Check = Callable[[EvalRunStore], Awaitable[None]]
+
+
+# -- fixtures -----------------------------------------------------------------------
+
+
+class _Answer(BaseModel):
+    city: str
+
+
+class _Target(BaseEvalTarget):
+    def __init__(self) -> None:
+        super().__init__("conformance")
+        self.active = 0
+        self.max_active = 0
+
+    async def run(self, task, cancellation_token=None):
+        self.active += 1
+        self.max_active = max(self.max_active, self.active)
+        try:
+            await asyncio.sleep(task.metadata.get("delay", 0))
+            if task.metadata.get("target_fails"):
+                raise RuntimeError("target failed")
+            span = ExecutionSpan(kind="agent", name="conformance", status="completed")
+            trace = ExecutionTrace(spans=[span], events=[ExecutionEvent(span_id=span.id, kind="step")])
+            return EvalTrajectory(
+                task=task,
+                messages=[
+                    UserMessage(content=task.input, source="user"),
+                    AssistantMessage(content="answer", source="a", structured_content=_Answer(city="Rome")),
+                ],
+                success=True,
+                usage=Usage(tokens_input=3, tokens_output=2),
+                trace=trace,
+                metadata={"execution_status": "completed"},
+            )
+        finally:
+            self.active -= 1
+
+
+class _Judge(BaseEvalJudge):
+    def __init__(self) -> None:
+        super().__init__("conformance")
+
+    async def score(self, trajectory, criteria=None, cancellation_token=None):
+        if trajectory.task.metadata.get("judge_fails"):
+            raise JudgeScoringError("provider_error")
+        return EvalScore(
+            overall=float(trajectory.task.metadata.get("score", 7.0)),
+            dimensions={"quality": 7.0},
+            trajectory=trajectory,
+        )
+
+
+def _run(**kwargs) -> EvalRunRecord:
+    return EvalRunRecord(dataset_id="d", target_id="t", judge_name="j", judge_type="J", **kwargs)
+
+
+def _items(run: EvalRunRecord, n: int) -> list[EvalItemRecord]:
+    return [EvalItemRecord(run_id=run.id, item_index=i, task_name=f"t{i}") for i in range(n)]
+
+
+def _outcome(run: EvalRunRecord, index: int, overall: float = 5.0, **kwargs) -> EvalItemRecord:
+    return EvalItemRecord(
+        run_id=run.id, item_index=index, task_name=f"t{index}", state="finished",
+        scoring_status="scored", overall=overall, **kwargs,
+    )
+
+
+async def _started(store: EvalRunStore, n: int, **kwargs) -> EvalRunRecord:
+    run = _run(**kwargs)
+    await store.create_run(run, _items(run, n))
+    return await store.start_run(run.id, worker_id=kwargs.get("worker_id"))
+
+
+async def _raises(exc: type[BaseException], awaitable: Awaitable[object]) -> None:
+    try:
+        await awaitable
+    except exc:
+        return
+    raise AssertionError(f"expected {exc.__name__}")
+
+
+# -- checks -------------------------------------------------------------------------
+
+
+async def check_admission_is_idempotent_and_bounded(store: EvalRunStore) -> None:
+    first = _run(owner="alice", idempotency_key="k1")
+    admitted = await store.create_run(first, _items(first, 1))
+    assert (admitted.status, admitted.revision, admitted.planned_count) == ("queued", 1, 1)
+    again = _run(owner="alice", idempotency_key="k1")
+    assert (await store.create_run(again, _items(again, 1))).id == admitted.id
+    other = _run(owner="bob", idempotency_key="k1")
+    assert (await store.create_run(other, _items(other, 1))).id == other.id
+    different = _run(owner="alice", idempotency_key="k1")
+    await _raises(EvalHistoryConflictError, store.create_run(different, _items(different, 2)))
+    same_id = admitted.model_copy(update={"idempotency_key": None})
+    await _raises(EvalHistoryConflictError, store.create_run(same_id, _items(same_id, 1)))
+    full = _run()
+    await _raises(EvalQueueFullError, store.create_run(full, _items(full, 1), max_active_runs=2))
+    assert len(await store.list_runs()) == 2
+
+
+async def check_lifecycle_updates_counters_and_revisions(store: EvalRunStore) -> None:
+    run = await _started(store, 3, worker_id="w1")
+    assert (run.status, run.revision, run.worker_id) == ("running", 2, "w1")
+    await _raises(EvalHistoryConflictError, store.start_run(run.id))
+
+    running = await store.mark_item_running(run.id, 0)
+    assert (running.state, running.revision) == ("running", 3) and running.started_at is not None
+    await _raises(EvalHistoryConflictError, store.mark_item_running(run.id, 0))
+
+    saved = await store.save_item(_outcome(run, 0, 6.0))
+    assert (saved.state, saved.revision, saved.overall) == ("finished", 4, 6.0)
+    assert saved.started_at == running.started_at and saved.finished_at is not None
+    failed = await store.save_item(_outcome(run, 1).model_copy(update={
+        "scoring_status": "failed", "overall": None, "failure_reason": "target_error",
+    }))
+    assert failed.revision == 5
+
+    current = await store.get_run(run.id)
+    assert (current.revision, current.finished_count, current.scored_count, current.failed_count) == (5, 2, 1, 1)
+    closed = await store.finish_run(run.id, "completed")
+    assert (closed.status, closed.revision, closed.not_started_count) == ("completed", 6, 1)
+    assert closed.finished_at is not None and closed.incomplete
+    await _raises(EvalHistoryConflictError, store.save_item(_outcome(run, 2)))
+
+
+async def check_replays_are_idempotent_and_conflicts_explicit(store: EvalRunStore) -> None:
+    run = await _started(store, 1)
+    outcome = _outcome(run, 0)
+    first = await store.save_item(outcome)
+    assert await store.save_item(outcome) == first
+    assert (await store.get_run(run.id)).finished_count == 1
+    await _raises(EvalHistoryConflictError, store.save_item(outcome.model_copy(update={"overall": 1.0})))
+    assert (await store.get_item(run.id, 0)).overall == 5.0
+
+    closed = await store.finish_run(run.id, "completed")
+    assert await store.finish_run(run.id, "completed") == closed
+    await _raises(EvalHistoryConflictError, store.finish_run(run.id, "cancelled"))
+
+
+async def check_evidence_conflict_is_explicit(store: EvalRunStore) -> None:
+    run = await _started(store, 1)
+    score = EvalScore(overall=5.0)
+    await store.save_item(_outcome(run, 0, evidence=score, evidence_status="captured"))
+    other = EvalScore(overall=5.0, reasoning={"q": "different"})
+    await _raises(
+        EvalHistoryConflictError,
+        store.save_item(_outcome(run, 0, evidence=other, evidence_status="captured")),
+    )
+
+
+async def check_finish_accounts_for_every_item(store: EvalRunStore) -> None:
+    run = await _started(store, 3)
+    await store.mark_item_running(run.id, 0)
+    await store.save_item(_outcome(run, 0))
+    await store.mark_item_running(run.id, 1)
+    closed = await store.finish_run(run.id, "cancelled")
+    items = await store.list_items(run.id)
+    assert [i.state for i in items] == ["finished", "cancelled", "not_started"]
+    assert items[1].failure_reason == "cancelled" and items[1].overall is None
+    assert (closed.finished_count, closed.cancelled_count, closed.not_started_count) == (1, 1, 1)
+    await _raises(ValueError, store.finish_run(run.id, "running"))
+
+
+async def check_recovery_is_explicit_and_scoped(store: EvalRunStore) -> None:
+    mine = await _started(store, 2, worker_id="w1")
+    theirs = await _started(store, 1, worker_id="w2")
+    await store.mark_item_running(mine.id, 0)
+    assert (await store.get_run(mine.id)).status == "running"
+
+    (recovered,) = await store.recover_interrupted(worker_id="w1")
+    assert (recovered.id, recovered.status, recovered.error) == (mine.id, "interrupted", "interrupted")
+    assert [i.state for i in await store.list_items(mine.id)] == ["cancelled", "not_started"]
+    assert (await store.get_run(theirs.id)).status == "running"
+    assert [r.id for r in await store.recover_interrupted()] == [theirs.id]
+
+
+async def check_full_evidence_round_trips_as_data(store: EvalRunStore) -> None:
+    turn = [UserMessage(content="same", source="user"), AssistantMessage(content="same", source="a")]
+    task = WorkflowEvalTask(name="wf", input="judge", workflow_input={"order": {"id": 7}})
+    trajectory = EvalTrajectory(
+        task=task,
+        messages=[*turn, *turn, AssistantMessage(content="", source="a", structured_content=_Answer(city="Rome"))],
+        success=True,
+        trace=ExecutionTrace(spans=[ExecutionSpan(kind="agent", name="a")]),
+    )
+    score = EvalScore(overall=8.0, reasoning={"q": "why"}, trajectory=trajectory)
+    run = await _started(store, 1)
+    await store.save_item(_outcome(run, 0, 8.0, evidence=score, evidence_status="captured"))
+
+    listed = (await store.list_items(run.id))[0]
+    assert listed.evidence is None
+    loaded = (await store.get_item(run.id, 0)).evidence
+    assert isinstance(loaded.trajectory.task, WorkflowEvalTask)
+    assert loaded.trajectory.task.workflow_input == {"order": {"id": 7}}
+    assert [(m.role, m.content) for m in loaded.trajectory.messages[:4]] == [
+        ("user", "same"), ("assistant", "same"), ("user", "same"), ("assistant", "same"),
+    ]
+    structured = loaded.trajectory.messages[4].structured_content
+    assert isinstance(structured, StructuredPayload) and structured.model_dump() == {"city": "Rome"}
+    assert loaded.trajectory.trace.id == trajectory.trace.id and loaded.reasoning == {"q": "why"}
+    assert (await store.list_items(run.id, include_evidence=True))[0].evidence == loaded
+
+
+async def check_unserializable_evidence_changes_nothing(store: EvalRunStore) -> None:
+    run = await _started(store, 1)
+    await store.mark_item_running(run.id, 0)
+    before = await store.get_run(run.id)
+    bad = EvalScore(overall=5.0, metadata={"handle": object()})
+    try:
+        await store.save_item(_outcome(run, 0, evidence=bad, evidence_status="captured"))
+    except EvalSerializationError as exc:
+        assert exc.path == "evidence.metadata.handle"
+    else:
+        raise AssertionError("expected EvalSerializationError")
+    assert await store.get_run(run.id) == before
+    assert (await store.get_item(run.id, 0)).state == "running"
+
+
+async def check_queries_paginate_filter_and_delete(store: EvalRunStore) -> None:
+    runs = []
+    for index in range(3):
+        run = _run(owner="alice" if index else "bob")
+        run = run.model_copy(update={"dataset_id": f"d{index}"})
+        await store.create_run(run, _items(run, 5))
+        await store.start_run(run.id)
+        runs.append(await store.finish_run(run.id, "cancelled"))
+    assert [r.id for r in await store.list_runs(limit=2, offset=1)] == [runs[1].id, runs[0].id]
+    assert [r.id for r in await store.list_runs(dataset_id="d2")] == [runs[2].id]
+    assert [r.id for r in await store.list_runs(owner="bob")] == [runs[0].id]
+    assert [r.id for r in await store.list_runs(status="cancelled", limit=1)] == [runs[2].id]
+    assert [i.item_index for i in await store.list_items(runs[0].id, limit=2, offset=3)] == [3, 4]
+
+    active = _run()
+    await store.create_run(active, _items(active, 1))
+    await _raises(EvalHistoryConflictError, store.delete_run(active.id))
+    assert await store.delete_run(runs[0].id) is True
+    assert await store.get_run(runs[0].id) is None
+    assert await store.list_items(runs[0].id) == []
+    assert await store.get_item(runs[0].id, 0) is None
+    assert await store.delete_run(runs[0].id) is False
+
+
+async def check_unknown_run_is_reported(store: EvalRunStore) -> None:
+    assert await store.get_run("missing") is None
+    await _raises(EvalRunNotFoundError, store.start_run("missing"))
+    await _raises(EvalRunNotFoundError, store.finish_run("missing", "failed"))
+    run = await _started(store, 1)
+    await _raises(EvalRunNotFoundError, store.mark_item_running(run.id, 5))
+
+
+async def check_recorder_keeps_duplicates_and_failures(store: EvalRunStore) -> None:
+    tasks = [
+        EvalTask(name="case-1", input="a"),
+        EvalTask(name="case-1", input="b"),
+        EvalTask(name="target", input="c", metadata={"target_fails": True}),
+        EvalTask(name="judge", input="d", metadata={"judge_fails": True}),
+    ]
+    run = await EvalRunRecorder(store).evaluate(
+        EvalRunner(_Judge()), _Target(), tasks, dataset_id="d", target_id="t",
+    )
+    items = await store.list_items(run.id)
+    assert run.status == "completed"
+    assert (run.planned_count, run.finished_count, run.scored_count, run.failed_count) == (4, 4, 2, 2)
+    assert [(i.item_index, i.task_name) for i in items[:2]] == [(0, "case-1"), (1, "case-1")]
+    assert (items[2].failure_reason, items[2].error_type) == ("target_error", "RuntimeError")
+    assert items[3].failure_reason == "provider_error"
+
+
+async def check_recorder_bounds_concurrency_without_lost_writes(store: EvalRunStore) -> None:
+    tasks = [
+        EvalTask(name=f"c{i}", input="x", metadata={"delay": (i % 4) * 0.003, "score": i % 10})
+        for i in range(16)
+    ]
+    target = _Target()
+    run = await EvalRunRecorder(store, concurrency=4).evaluate(
+        EvalRunner(_Judge()), target, tasks, dataset_id="d", target_id="t",
+    )
+    items = await store.list_items(run.id, limit=50)
+    assert target.max_active <= 4
+    assert (run.finished_count, run.scored_count, run.revision) == (16, 16, 1 + 1 + 16 * 2 + 1)
+    assert [i.overall for i in items] == [float(i % 10) for i in range(16)]
+
+
+CHECKS: dict[str, Check] = {
+    name.removeprefix("check_"): check
+    for name, check in dict(globals()).items()
+    if name.startswith("check_")
+}
+
+
+async def run_conformance(
+    open_store: Callable[[], AbstractAsyncContextManager[EvalRunStore]],
+) -> None:
+    """Run every check, each against a fresh empty store from ``open_store``."""
+    for check in CHECKS.values():
+        async with open_store() as store:
+            await check(store)
+
+
+__all__ = ["CHECKS", "Check", "run_conformance"]

agentbyte-0.32.0/src/agentbyte/eval/history/errors.py

@@ -0,0 +1,63 @@
+"""Errors raised by evaluation history stores and the recorder."""
+
+from __future__ import annotations
+
+
+class EvalHistoryError(Exception):
+    """Base class for evaluation history failures."""
+
+
+class EvalPersistenceError(EvalHistoryError):
+    """A history write or read failed; durable success must not be assumed.
+
+    ``run_id`` names the affected run when one is known.
+    """
+
+    def __init__(self, message: str, *, run_id: str | None = None) -> None:
+        super().__init__(f"{message} (run {run_id})" if run_id else message)
+        self.run_id = run_id
+
+
+class EvalRedactionError(EvalPersistenceError):
+    """A supplied redaction hook failed; nothing was persisted for the item."""
+
+
+class EvalHistoryConflictError(EvalHistoryError):
+    """A write contradicts stored state, such as a different terminal outcome."""
+
+
+class EvalSchemaVersionError(EvalHistoryError):
+    """Stored data uses a schema or evidence version this code does not support."""
+
+
+class EvalSerializationError(EvalHistoryError):
+    """A value cannot be stored as JSON; ``path`` names the offending field."""
+
+    def __init__(self, path: str, reason: str) -> None:
+        super().__init__(f"{path}: {reason}")
+        self.path = path
+
+
+class EvalRunNotFoundError(EvalHistoryError, KeyError):
+    """The run or item does not exist in this store."""
+
+
+class EvalQueueFullError(EvalHistoryError):
+    """Admission refused because the active-run capacity is reached."""
+
+
+class EvalStoreOwnershipError(EvalHistoryError):
+    """Another process already owns this history database as its writer."""
+
+
+__all__ = [
+    "EvalHistoryConflictError",
+    "EvalHistoryError",
+    "EvalPersistenceError",
+    "EvalQueueFullError",
+    "EvalRedactionError",
+    "EvalRunNotFoundError",
+    "EvalSchemaVersionError",
+    "EvalSerializationError",
+    "EvalStoreOwnershipError",
+]