agentbyte 0.38.0__tar.gz → 0.40.0__tar.gz

{agentbyte-0.38.0 → agentbyte-0.40.0}/CHANGELOG.md

@@ -6,6 +6,51 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.40.0] - 2026-10-02
+
+### Added
+
+- Evaluation job backend now logs its lifecycle through the `agentbyte.eval.workbench.jobs`
+  logger: INFO on manager start, run admitted/started/finished/cancelled, reused idempotency key
+  and shutdown; WARNING on a full queue and on recovering interrupted runs at start-up; ERROR
+  (with traceback) on a failed run. Previously only failures were logged.
+- New `gunicorn` optional extra (`pip install "agentbyte[gunicorn]"`, composed from `webui`) for
+  single-worker production serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`); added to
+  the `all` extra.
+- Self-contained, uv-based deployment example `examples/webui/eval_app/` (`app.py`,
+  `pyproject.toml`, `gunicorn.conf.py`, `Dockerfile`, `docker-compose.yml`, `README.md`) that
+  serves one evaluation suite (Evaluations only) and builds as an OCI image under Docker or Podman.
+- Deployment design-guide topic `docs/design-guide/04-web_topics/03_topic_deployment.md`
+  documenting the single-writer / one-worker model, the correct container shape (one worker, DB on
+  a volume, graceful SIGTERM) and the Postgres + lease scaling path.
+
+### Changed
+
+- `examples/webui/eval_webui.py` now loads `.env` defensively, so the suite imports as a catalog
+  (and runs in a container) even when `python-dotenv` is not installed.
+
+## [0.39.0] - 2026-10-02
+
+### Added
+
+- WebUI `serve()` and `WebUIServer` now take a `playground: bool = True` flag, so a
+  single entrypoint selects what one server exposes:
+  - `serve(entities=[agent])` — Playground only.
+  - `serve(entities=[agent], eval_registry=r, eval_history=h)` — Playground + Evaluations.
+  - `serve(eval_registry=r, eval_history=h, playground=False)` — Evaluations only (no
+    Playground tab). The flag is threaded through to `AgentbyteWebUIServer(playground=...)`,
+    which already gated the tabs; `agentbyte eval ui` uses the same eval-only mode.
+  - `playground=False` requires an eval registry + history and rejects `entities`/
+    `entities_dir` with a `ValueError` (there is no Playground to register them into).
+
+### Changed
+
+- `examples/webui/eval_webui.py` now serves Evaluations only via
+  `serve(..., playground=False)` and defaults its history to `acme-eval.sqlite`.
+- Documented the Playground/Evaluations/both serving matrix (Python and CLI) in
+  `docs/guides/evaluation-end-to-end.md`, the Playground design topic, and notebook
+  `06-eval-end-to-end`.
+
 ## [0.38.0] - 2026-10-02
 
 ### Added

{agentbyte-0.38.0 → agentbyte-0.40.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.38.0
+Version: 0.40.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -20,6 +20,7 @@ Requires-Dist: beautifulsoup4>=4.12; extra == 'all'
 Requires-Dist: fastapi>=0.135.2; extra == 'all'
 Requires-Dist: gepa>=0.1; extra == 'all'
 Requires-Dist: graphviz>=0.21; extra == 'all'
+Requires-Dist: gunicorn>=23.0.0; extra == 'all'
 Requires-Dist: html2text>=2024.2; extra == 'all'
 Requires-Dist: openai>=1.107.1; extra == 'all'
 Requires-Dist: opentelemetry-api>=1.39.1; extra == 'all'
@@ -35,6 +36,10 @@ Requires-Dist: aioboto3>=15.5.0; extra == 'aws'
 Provides-Extra: azureopenai
 Requires-Dist: azure-identity>=1.25.1; extra == 'azureopenai'
 Requires-Dist: openai>=1.107.1; extra == 'azureopenai'
+Provides-Extra: gunicorn
+Requires-Dist: fastapi>=0.135.2; extra == 'gunicorn'
+Requires-Dist: gunicorn>=23.0.0; extra == 'gunicorn'
+Requires-Dist: uvicorn[standard]>=0.44.0; extra == 'gunicorn'
 Provides-Extra: openai
 Requires-Dist: openai>=1.107.1; extra == 'openai'
 Provides-Extra: optim
@@ -72,7 +77,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.38.0**
+Current release: **0.40.0**
 
 ## Building an Agent
 
@@ -305,6 +310,7 @@ uv add "agentbyte[openai,sqlite]"
 | `aws` | `aioboto3` | S3 dataset sources, writers and publishers |
 | `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
 | `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
+| `gunicorn` | `webui` + `gunicorn` | Production single-worker serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`) |
 | `sqlite` | `aiosqlite` | SQLite datasets, workflow checkpoints, evaluation history |
 | `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, `SqlEvalRunStore` |
 | `sql` | `sqlite` + `postgres` | Both SQL backends |

{agentbyte-0.38.0 → agentbyte-0.40.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.38.0**
+Current release: **0.40.0**
 
 ## Building an Agent
 
@@ -235,6 +235,7 @@ uv add "agentbyte[openai,sqlite]"
 | `aws` | `aioboto3` | S3 dataset sources, writers and publishers |
 | `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
 | `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
+| `gunicorn` | `webui` + `gunicorn` | Production single-worker serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`) |
 | `sqlite` | `aiosqlite` | SQLite datasets, workflow checkpoints, evaluation history |
 | `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, `SqlEvalRunStore` |
 | `sql` | `sqlite` + `postgres` | Both SQL backends |

{agentbyte-0.38.0 → agentbyte-0.40.0}/pyproject.toml

@@ -36,6 +36,10 @@ webui = [
     "fastapi>=0.135.2",
     "uvicorn[standard]>=0.44.0",
 ]
+gunicorn = [
+    "agentbyte[webui]",
+    "gunicorn>=23.0.0",
+]
 sqlite = [
     "aiosqlite>=0.22.1",
 ]
@@ -63,7 +67,7 @@ research = [
     "youtube-transcript-api>=0.6",
 ]
 all = [
-    "agentbyte[openai,azureopenai,aws,otel,webui,sql,tiktoken,optim,viz,research]",
+    "agentbyte[openai,azureopenai,aws,otel,webui,gunicorn,sql,tiktoken,optim,viz,research]",
 ]
 
 

agentbyte-0.40.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.40.0"
+VERSION = __version__

{agentbyte-0.38.0 → agentbyte-0.40.0}/src/agentbyte/eval/workbench/jobs.py

@@ -14,7 +14,7 @@ from collections.abc import AsyncIterator
 from dataclasses import dataclass, field
 
 from agentbyte.cancellation_token import CancellationToken
-from agentbyte.eval.history import CaptureMode, EvalRunRecord
+from agentbyte.eval.history import CaptureMode, EvalQueueFullError, EvalRunRecord
 from agentbyte.eval.history.models import ACTIVE_RUN_STATUSES
 from agentbyte.eval.history.store import _uninterrupted
 
@@ -112,6 +112,17 @@ class EvalJobManager:
             recovered = await self.coordinator.store.recover_interrupted()
         self._workers = [asyncio.create_task(self._work()) for _ in range(self.max_active)]
         self._accepting = True
+        if recovered:
+            logger.warning(
+                "Recovered %d interrupted evaluation run(s) left active by a previous process: %s",
+                len(recovered),
+                ", ".join(run.id for run in recovered),
+            )
+        logger.info(
+            "Evaluation job manager started (max_active=%d, max_queued=%d)",
+            self.max_active,
+            self.max_queued,
+        )
         return recovered
 
     async def close(self) -> None:
@@ -119,6 +130,11 @@ class EvalJobManager:
         self._accepting = False
         self._closed = True
         jobs = list(self._jobs.values())
+        if jobs:
+            logger.info(
+                "Evaluation job manager shutting down; cancelling %d active/queued run(s)",
+                len(jobs),
+            )
         for job in jobs:
             job.cancel_requested = True
             job.token.cancel()
@@ -167,13 +183,26 @@ class EvalJobManager:
         if not idempotency_key:
             raise ValueError("idempotency_key must be a non-empty string")
         self._check_accepting()
-        admitted = await self.coordinator.admit(
-            suite_name,
-            capture=capture,
-            idempotency_key=idempotency_key,
-            max_active_runs=self.capacity,
-        )
+        try:
+            admitted = await self.coordinator.admit(
+                suite_name,
+                capture=capture,
+                idempotency_key=idempotency_key,
+                max_active_runs=self.capacity,
+            )
+        except EvalQueueFullError:
+            logger.warning(
+                "Evaluation queue is full (capacity=%d); rejected suite %r",
+                self.capacity,
+                suite_name,
+            )
+            raise
         if not admitted.created:
+            logger.info(
+                "Reused evaluation run %s for suite %r (idempotency key matched an earlier submission)",
+                admitted.run.id,
+                suite_name,
+            )
             return SubmittedRun(run=admitted.run, created=False)
         job = _Job(admitted)
         if not self._accepting:
@@ -183,6 +212,12 @@ class EvalJobManager:
         self._jobs[job.run_id] = job
         self._queue.put_nowait(job.run_id)
         self._notify(job.run_id)
+        logger.info(
+            "Admitted evaluation run %s for suite %r (capture=%s); queued",
+            job.run_id,
+            suite_name,
+            admitted.run.capture,
+        )
         return SubmittedRun(run=admitted.run, created=True)
 
     async def cancel(self, run_id: str) -> EvalRunRecord:
@@ -194,6 +229,7 @@ class EvalJobManager:
         job = self._jobs.get(run.id)
         if job is None:
             return run
+        logger.info("Cancelling evaluation run %s", run.id)
         job.cancel_requested = True
         job.token.cancel()
         if job.task is not None:
@@ -271,6 +307,8 @@ class EvalJobManager:
             await asyncio.wait([job.task])
 
     async def _execute(self, job: _Job) -> None:
+        suite_name = job.admitted.suite.name
+        logger.info("Starting evaluation run %s for suite %r", job.run_id, suite_name)
         try:
             async for _ in self.coordinator.execute_stream(
                 job.admitted,
@@ -278,8 +316,9 @@ class EvalJobManager:
                 on_start=lambda run: self._notify(run.id),
             ):
                 self._notify(job.run_id)
+            logger.info("Evaluation run %s finished", job.run_id)
         except asyncio.CancelledError:
-            pass
+            logger.info("Evaluation run %s cancelled", job.run_id)
         except Exception:
             # The recorder has already closed the run as failed; keep the worker alive.
             logger.exception("Evaluation run %s failed", job.run_id)

{agentbyte-0.38.0 → agentbyte-0.40.0}/src/agentbyte/webui/server.py

@@ -458,6 +458,7 @@ class WebUIServer:
         eval_registry: EvalSuiteRegistry | None = None,
         eval_store: EvalRunStore | None = None,
         eval_history: str | Path | None = None,
+        playground: bool = True,
     ) -> None:
         self._server = AgentbyteWebUIServer(
             entities_dir=entities_dir,
@@ -467,6 +468,7 @@ class WebUIServer:
             eval_registry=eval_registry,
             eval_store=eval_store,
             eval_history=eval_history,
+            playground=playground,
         )
         self.port = port
         self.host = host
@@ -572,8 +574,24 @@ def serve(
     session_store: SessionStore | None = None,
     eval_registry: EvalSuiteRegistry | None = None,
     eval_history: str | Path | None = None,
+    playground: bool = True,
 ) -> None:
-    """Serve in-memory and/or discovered entities in the WebUI."""
+    """Serve in-memory and/or discovered entities in the WebUI.
+
+    One flag selects the mode:
+
+    - Playground only: ``serve(entities=[agent])``
+    - Playground + Evaluations: ``serve(entities=[agent], eval_registry=r, eval_history=h)``
+    - Evaluations only: ``serve(eval_registry=r, eval_history=h, playground=False)``
+
+    ``playground=False`` serves the Evaluations pages only (no Playground tab); it needs an
+    ``eval_registry`` with ``eval_history`` and rejects ``entities``/``entities_dir``, since
+    there is no Playground to register them into.
+    """
+    if not playground and (entities or entities_dir is not None):
+        raise ValueError(
+            "playground=False serves evaluations only; do not pass entities or entities_dir"
+        )
     server = WebUIServer(
         entities_dir=entities_dir,
         port=port,
@@ -581,6 +599,7 @@ def serve(
         session_store=session_store,
         eval_registry=eval_registry,
         eval_history=eval_history,
+        playground=playground,
     )
     for index, entity in enumerate(entities or []):
         entity_id = _infer_entity_id(entity, index)

{agentbyte-0.38.0 → agentbyte-0.40.0}/tests/eval/test_eval_jobs.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import asyncio
 import contextlib
+import logging
 
 import pytest
 
@@ -389,3 +390,67 @@ async def test_admitted_duplicate_cannot_be_executed_twice() -> None:
         async for _ in coordinator.execute_stream(duplicate):
             pass
 
+
+# -- logging --------------------------------------------------------------------------
+
+_LOGGER = "agentbyte.eval.workbench.jobs"
+
+
+@pytest.mark.asyncio
+async def test_run_lifecycle_is_logged_from_admission_to_finish(caplog) -> None:
+    with caplog.at_level(logging.INFO, logger=_LOGGER):
+        async with jobs(registry=smoke_registry) as manager:
+            submitted = await manager.submit("smoke", idempotency_key="k1")
+            await final_summary(manager, submitted.run.id)
+    messages = [record.getMessage() for record in caplog.records if record.name == _LOGGER]
+    joined = "\n".join(messages)
+    assert "Evaluation job manager started" in joined
+    assert f"Admitted evaluation run {submitted.run.id}" in joined
+    assert f"Starting evaluation run {submitted.run.id}" in joined
+    assert f"Evaluation run {submitted.run.id} finished" in joined
+
+
+@pytest.mark.asyncio
+async def test_reused_idempotency_key_is_logged(caplog) -> None:
+    with caplog.at_level(logging.INFO, logger=_LOGGER):
+        async with jobs() as manager:
+            first = await manager.submit("quick", idempotency_key="dup")
+            await final_summary(manager, first.run.id)
+            caplog.clear()
+            reused = await manager.submit("quick", idempotency_key="dup")
+    assert not reused.created
+    assert any(
+        f"Reused evaluation run {reused.run.id}" in record.getMessage()
+        for record in caplog.records
+        if record.name == _LOGGER
+    )
+
+
+@pytest.mark.asyncio
+async def test_full_queue_is_logged_as_a_warning(caplog) -> None:
+    with caplog.at_level(logging.WARNING, logger=_LOGGER):
+        async with jobs(max_active=1, max_queued=0) as manager:
+            await manager.submit("gated", idempotency_key="busy")
+            with pytest.raises(EvalQueueFullError):
+                await manager.submit("quick", idempotency_key="overflow")
+            GatedTarget.gate.set()
+    assert any(
+        record.levelno == logging.WARNING and "queue is full" in record.getMessage()
+        for record in caplog.records
+        if record.name == _LOGGER
+    )
+
+
+@pytest.mark.asyncio
+async def test_cancellation_is_logged(caplog) -> None:
+    with caplog.at_level(logging.INFO, logger=_LOGGER):
+        async with jobs() as manager:
+            submitted = await manager.submit("gated", idempotency_key="c")
+            await wait_until(lambda: _is(manager, submitted.run.id, "running"))
+            await manager.cancel(submitted.run.id)
+            GatedTarget.gate.set()
+    messages = "\n".join(
+        record.getMessage() for record in caplog.records if record.name == _LOGGER
+    )
+    assert f"Cancelling evaluation run {submitted.run.id}" in messages
+

{agentbyte-0.38.0 → agentbyte-0.40.0}/tests/webui/test_server.py

@@ -13,13 +13,44 @@ except ImportError:  # pragma: no cover - optional dependency path
 
 from agentbyte.session_store import FileSessionStore
 from agentbyte.workflow import FunctionStep, StepMetadata, Workflow, WorkflowConfig
-from agentbyte.webui.server import AgentbyteWebUIServer, _infer_entity_id
+from agentbyte.webui.server import AgentbyteWebUIServer, WebUIServer, _infer_entity_id, serve
 
 from .helpers import DummyAgent, DummyOrchestrator
 
 pytestmark = pytest.mark.skipif(TestClient is None, reason="fastapi not installed")
 
 
+# -- playground / eval-only mode selection (serve + WebUIServer) ---------------
+
+
+def test_webui_server_forwards_playground_false_for_eval_only(tmp_path) -> None:
+    """WebUIServer(playground=False) threads the flag to the backing server: the
+    health endpoint reports Playground disabled while evaluations stay enabled."""
+    server = WebUIServer(playground=False, eval_history=str(tmp_path / "eval.sqlite"))
+    client = TestClient(server.get_app())
+
+    health = client.get("/api/health").json()
+    assert health["playground_enabled"] is False
+    assert health["eval_enabled"] is True
+
+
+def test_webui_server_defaults_to_playground_enabled() -> None:
+    server = WebUIServer()
+    assert server._server.playground is True
+    assert TestClient(server.get_app()).get("/api/health").json()["playground_enabled"] is True
+
+
+def test_serve_rejects_entities_in_eval_only_mode() -> None:
+    """playground=False has no Playground to register in-memory entities into."""
+    with pytest.raises(ValueError, match="evaluations only"):
+        serve(entities=[DummyAgent()], playground=False, eval_history="eval.sqlite")
+
+
+def test_serve_rejects_entities_dir_in_eval_only_mode() -> None:
+    with pytest.raises(ValueError, match="evaluations only"):
+        serve(entities_dir="some/dir", playground=False, eval_history="eval.sqlite")
+
+
 def test_health_endpoint() -> None:
     server = AgentbyteWebUIServer()
     client = TestClient(server.create_app())

agentbyte-0.38.0/src/agentbyte/__about__.py

@@ -1,2 +0,0 @@
-__version__ = "0.38.0"
-VERSION = __version__