agentbyte 0.32.2__tar.gz → 0.34.0__tar.gz

{agentbyte-0.32.2 → agentbyte-0.34.0}/.gitignore

@@ -64,3 +64,10 @@ data/flatfiles/*
 
 # lean-ctx context runtime cache
 .lean-ctx/
+
+# Local SQLite stores (evaluation history, checkpoints) and their writer locks
+*.sqlite
+*.sqlite-journal
+*.sqlite-wal
+*.sqlite-shm
+*.writer.lock

{agentbyte-0.32.2 → agentbyte-0.34.0}/CHANGELOG.md

@@ -6,6 +6,53 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.34.0] - 2026-10-01
+
+### Added
+
+- Evaluation workbench, step 3 (spec 0052): WebUI Evaluations section next to the Playground — registered suites with Run (summary or full evidence), run history with pass/fail verdicts and filters, a live run page (durable snapshots, outcome counts, Cancel) and an item page (transcript, judge scores and reasoning, execution facts, labelled evidence state). Pages have shareable `#/evals/...` links. A server with evaluations but no agents or workflows opens on Evaluations.
+- `/api/health` reports `eval_enabled`; `GET /api/eval/runs` entries include a `verdict` with pass/fail and outcome counts.
+- Simpler eval commands: `--catalog` / `--eval-catalog` (and `registry check`) accept a `.py` file path as well as a module, and `:name` is optional when the module defines exactly one matching object. `agentbyte eval run|results` and `agentbyte webui --eval-catalog` store runs in `./eval.sqlite` unless a history path is given. Example: `agentbyte webui --eval-catalog examples/eval/workbench_catalog.py`.
+- `agentbyte eval ui [--catalog CATALOG] [--history PATH] [--port 8090]`: the Evaluations pages on their own, without the Playground (`AgentbyteWebUIServer(playground=False)`, `serve`/`webui(playground=False)`); `/api/health` reports `playground_enabled`. It refuses to start, with exit code 2, when another process holds the history file.
+- `agentbyte.eval.tasks_from_eval_records(records)`: the tasks a saved eval dataset would produce, built synchronously from in-memory `EvalDatasetRecord`s (for suite catalogs). Saving and loading now share one record/row mapping.
+- Evaluation guide `docs/guides/evaluation-end-to-end.md` (how datasets, suites, runners, history, CLI, HTTP and the UI connect; on-the-fly vs stored runs; CI), the real-model example catalog `examples/eval/acme_support.py` with its fake FAQ dataset (`data/datasets/examples/acme-support`), and the notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb`.
+- The sample WebUI launchers (`examples/webui/in_memory.py`, `presets_webui.py`) accept `--eval-history`, `--eval-catalog`, `--port` and `--no-open`, serving the demo entities and the Evaluations pages together.
+
+### Fixed
+
+- The packaged WebUI called `http://localhost:8080` regardless of the port it was served from; it now calls its own origin, so `agentbyte webui --port N` works.
+- README "Run The WebUI": removed the `--dir examples` suggestion (discovery finds no entities there) and documented the sample launchers, Evaluations and the discovery convention.
+
+## [0.33.0] - 2026-10-01
+
+Upgrade notes (breaking):
+
+- Removed the `test`, `dev` and `coding` extras; they only held contributor tooling. Contributors use `uv sync --all-extras --all-groups`.
+- S3 datasets use native async `aioboto3` (`aws` extra) instead of synchronous `boto3`. `DatasetWriter.write_files`/`dataset_exists`, `DatasetPublisher.publish_dataset` and `agentbyte.dataset.publish_dataset` are now coroutines: `await` them, and make custom writers/publishers `async def`. A `client=` passed to `S3DatasetSource`, `S3DatasetWriter` or `S3DatasetPublisher` must be an async S3 client; it is used as-is and never closed.
+- A missing optional dependency now raises `agentbyte.MissingExtraError` (a `ModuleNotFoundError`) everywhere. The WebUI and `WorkflowVisualizer.to_dot()` previously raised `RuntimeError`.
+
+### Added
+
+- Evaluation workbench, step 1 (spec 0052): `agentbyte.eval.workbench` with `EvalSuite`, `EvalSuiteRegistry`, `EvalSuiteCoordinator` and pass/fail verdicts (`SuiteRunSummary`); an item passes only when execution completed, the judge scored it and `overall >= min_score`. New CLI `agentbyte eval list|run|results` (exit codes 0 passed, 1 failed/incomplete, 2 configuration/storage error, 130 interrupted) and a packaged, provider-free `smoke` suite. `EvalRunRecorder.evaluate`/`evaluate_stream` accept `on_start`. `python -m agentbyte.cli` works.
+- Evaluation workbench, step 2 (spec 0052): HTTP job lifecycle. `EvalJobManager` runs suites in the background (one active, ten queued); admission is idempotent per owner and key and atomic in the store, a full queue is rejected without a record, cancel is idempotent, shutdown commits owned jobs as `cancelled` and startup marks leftovers `interrupted`. WebUI routes under `/api/eval` (suites, runs, items, SSE `snapshot` events keyed by run revision, cancel), enabled by `AgentbyteWebUIServer(eval_store=... | eval_history=..., eval_registry=...)` or `agentbyte webui --eval-history PATH [--eval-catalog module:registry]`. `EvalRunRecorder.admit()`/`execute_stream()` split admission from execution; `SuiteRunSummary.revision`.
+- Evaluation workbench docs: study guide `docs/study/05-eval_topics/05_topic_eval_workbench.md`, offline example catalog `examples/eval/workbench_catalog.py` and notebook `notebooks/usecases/eval/05-eval-workbench.ipynb`.
+- `tiktoken` extra for `TiktokenCounter`.
+- `agentbyte.MissingExtraError`: one error type and message format ("<feature> needs the '<extra>' extra: pip install 'agentbyte[<extra>]'") for every optional dependency, via `agentbyte._optional.require`.
+- `agentbyte.cli.loading.import_object(reference, expected_type)`: one loader for `module.path:object` CLI arguments.
+
+### Changed
+
+- CLI `--catalog` / `registry check` references are imported like `python -m`: modules in the current working directory are found without installing them.
+- Evaluation history idempotency: reusing a key now also requires the same capture, criteria, labels, judge and target fingerprint; otherwise `EvalHistoryConflictError`.
+- README "Installation" is now one table of extras, what they install and which features need them.
+- The S3 dataset importer no longer wraps writer calls in `asyncio.to_thread`.
+- `all` (and `sql`) are composed from the other extras by reference, so they can no longer drift. Dependency groups now hold only local tooling (`test`, `dev`, `experiment`); groups that mirrored extras and the unused `extlib` group (Microsoft Agent Framework packages) were removed.
+
+### Fixed
+
+- SQLite datasets (`sqlite`, `sqlite_db` engines) no longer keep a script from exiting when they are not closed: their cached aiosqlite connection runs on a daemon thread. `aclose()` / `async with` remain the way to release them promptly.
+- Use-case notebooks (eval 01–04, workflow 01–02) updated for the current async dataset, eval-dataset and judge-integrity APIs; the repository's `contracting/metadata26` eval dataset was re-saved with the current schema (`metadata` column). OTel install hints now say `uv sync --extra otel`.
+
 ## [0.32.2] - 2026-10-01
 
 ### Fixed

{agentbyte-0.32.2 → agentbyte-0.34.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.32.2
+Version: 0.34.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -27,22 +27,14 @@ Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.39.1; extra == 'all'
 Requires-Dist: opentelemetry-sdk>=1.39.1; extra == 'all'
 Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'all'
 Requires-Dist: sqlmodel>=0.0.38; extra == 'all'
-Requires-Dist: uvicorn>=0.42.0; extra == 'all'
+Requires-Dist: tiktoken>=0.9.0; extra == 'all'
+Requires-Dist: uvicorn[standard]>=0.44.0; extra == 'all'
 Requires-Dist: youtube-transcript-api>=0.6; extra == 'all'
 Provides-Extra: aws
 Requires-Dist: aioboto3>=15.5.0; extra == 'aws'
 Provides-Extra: azureopenai
 Requires-Dist: azure-identity>=1.25.1; extra == 'azureopenai'
 Requires-Dist: openai>=1.107.1; extra == 'azureopenai'
-Provides-Extra: coding
-Provides-Extra: dev
-Requires-Dist: ipykernel>=7.1.0; extra == 'dev'
-Requires-Dist: nbclient>=0.10.4; extra == 'dev'
-Requires-Dist: nbformat>=5.10.4; extra == 'dev'
-Requires-Dist: picoagents>=0.4.0; extra == 'dev'
-Requires-Dist: pydantic-settings>=2.13.0; extra == 'dev'
-Requires-Dist: pymupdf-layout>=1.26.6; extra == 'dev'
-Requires-Dist: pymupdf4llm>=0.2.9; extra == 'dev'
 Provides-Extra: openai
 Requires-Dist: openai>=1.107.1; extra == 'openai'
 Provides-Extra: optim
@@ -67,18 +59,8 @@ Requires-Dist: sqlalchemy[asyncio]>=2.0.43; extra == 'sql'
 Requires-Dist: sqlmodel>=0.0.38; extra == 'sql'
 Provides-Extra: sqlite
 Requires-Dist: aiosqlite>=0.22.1; extra == 'sqlite'
-Provides-Extra: test
-Requires-Dist: aiosqlite>=0.22.1; extra == 'test'
-Requires-Dist: arxiv>=2.1; extra == 'test'
-Requires-Dist: beautifulsoup4>=4.12; extra == 'test'
-Requires-Dist: greenlet>=3.0.0; extra == 'test'
-Requires-Dist: html2text>=2024.2; extra == 'test'
-Requires-Dist: pytest-asyncio>=1.3.0; extra == 'test'
-Requires-Dist: pytest-cov>=7.0.0; extra == 'test'
-Requires-Dist: pytest>=9.0.1; extra == 'test'
-Requires-Dist: ruff>=0.15.0; extra == 'test'
-Requires-Dist: sqlmodel>=0.0.38; extra == 'test'
-Requires-Dist: youtube-transcript-api>=0.6; extra == 'test'
+Provides-Extra: tiktoken
+Requires-Dist: tiktoken>=0.9.0; extra == 'tiktoken'
 Provides-Extra: viz
 Requires-Dist: graphviz>=0.21; extra == 'viz'
 Provides-Extra: webui
@@ -90,12 +72,16 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.32.2**
+Current release: **0.34.0**
 
 ## Building an Agent
 
 Every example below builds on the same domain: a customer **support agent**. Start with a plain agent and a model client — no tools, no middleware:
 
+```bash
+pip install "agentbyte[openai]"   # the model client lives in an extra; see Install below
+```
+
 ```python
 from agentbyte import Agent
 from agentbyte.llm import OpenAIChatCompletionClient
@@ -302,160 +288,122 @@ Reading a trace: `chat gpt-4.1-mini` spans show **per-call** usage/cost/finish r
 
 Python requirement: **3.11+**
 
-```bash
-uv sync --all-groups
-```
-
-Optional extras:
+The base package (`pip install agentbyte`) has no model provider or storage driver. Add the
+extras for the features you use. Using a feature without its extra raises
+`MissingExtraError` (an `ImportError`) that names the extra to install.
 
 ```bash
-uv sync --extra openai        # OpenAI client
-uv sync --extra azureopenai   # Azure OpenAI client + azure-identity
-uv sync --extra aws           # S3 datasets (aioboto3)
-uv sync --extra otel          # OpenTelemetry export
-uv sync --extra webui         # WebUI playground (FastAPI)
-uv sync --extra sqlite        # SQLite stores: datasets, workflow checkpoints, eval history
-uv sync --extra postgres      # PostgreSQL stores: usage logging, eval history
-uv sync --extra sql           # sqlite + postgres
-uv sync --extra optim         # prompt optimization (GEPA)
-uv sync --extra viz           # workflow diagrams (graphviz)
-uv sync --extra research      # research tools
-```
-
-### Install in another project (pip / uv add)
-
-Use extras to enable provider + telemetry support:
-
-```bash
-pip install "agentbyte[azureopenai,otel]"
-```
-
-```bash
-uv add "agentbyte[azureopenai,otel]"
-```
-
-For the browser WebUI:
-
-```bash
-pip install "agentbyte[webui]"
+pip install "agentbyte[openai,sqlite]"
 # or
-uv add "agentbyte[webui]"
+uv add "agentbyte[openai,sqlite]"
 ```
 
-Install all optional features:
+| Extra | Installs | Needed for |
+| --- | --- | --- |
+| `openai` | `openai` | `OpenAIChatCompletionClient`, `OpenAIEmbeddingClient` |
+| `azureopenai` | `openai`, `azure-identity` | Azure OpenAI clients and Entra ID credentials |
+| `aws` | `aioboto3` | S3 dataset sources, writers and publishers |
+| `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
+| `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
+| `sqlite` | `aiosqlite` | SQLite datasets, workflow checkpoints, evaluation history |
+| `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, `SqlEvalRunStore` |
+| `sql` | `sqlite` + `postgres` | Both SQL backends |
+| `tiktoken` | `tiktoken` | `TiktokenCounter` for exact compaction token counts |
+| `optim` | `gepa` | `optimize_with_gepa()` |
+| `viz` | `graphviz` | `WorkflowVisualizer.to_dot()` |
+| `research` | `beautifulsoup4`, `html2text`, `arxiv`, `youtube-transcript-api` | `agentbyte.tools.research_tools` |
+| `all` | every extra above | Everything |
+
+Contributors working in this repository install every extra plus the local tooling groups
+(`test`, `dev`, `experiment`):
 
 ```bash
-pip install "agentbyte[all]"
-# or
-uv add "agentbyte[all]"
+uv sync --all-extras --all-groups
 ```
 
-Note: the Azure extra is `azureopenai`.
-
 ## Run The WebUI
 
-### Option 1: Run the preset-backed app
-
-This is the easiest way to see the WebUI working end to end with real preset entities:
-- preset agents
-- preset orchestrators
-- preset workflow
-
-Step 1. Install the WebUI extra:
+The WebUI has two sections: the **Playground** (chat with agents, orchestrators and workflows) and
+**Evaluations** (run registered evaluation suites and inspect the results). Install the extra first:
 
 ```bash
-uv sync --extra webui
+uv sync --extra webui            # add --extra sqlite for Evaluations
 ```
 
-Step 2. Start the preset-backed app:
-
-```bash
-uv run python examples/webui/presets_webui.py
-```
+### Option 1: Sample apps
 
-Step 3. Open the browser:
-
-```text
-http://127.0.0.1:8080
-```
-
-If auto-open is enabled in your environment, the browser may open automatically.
-
-### Option 2: Run the WebUI against your current project directory
-
-Use this when you want Agentbyte to scan a directory for exported `agent`, `workflow`, or `orchestrator` objects.
-
-Important: discovery is convention-based. The scanned directory must contain Python modules that expose top-level variables literally named `agent`, `workflow`, or `orchestrator`. If you point `--dir` at a folder that does not export those names, the UI will load but show `No entities found`.
-
-Step 1. Install the WebUI extra:
+No API key needed: three demo agents, a team and a workflow on a fake model client.
 
 ```bash
-uv sync --extra webui
+uv run python examples/webui/in_memory.py                          # http://127.0.0.1:8070
+uv run python examples/webui/in_memory.py --eval-history eval.sqlite   # plus Evaluations
 ```
 
-Step 2. Launch the WebUI and scan the current directory:
+With your OpenAI or Azure OpenAI credentials, the preset agents, teams and workflow:
 
 ```bash
-uv run agentbyte webui --dir .
+uv sync --extra webui --extra openai        # or --extra azureopenai
+uv run python examples/webui/presets_webui.py                      # http://127.0.0.1:8080
+uv run python examples/webui/presets_webui.py --eval-history eval.sqlite
 ```
 
-Step 3. Open the browser:
+Both launchers accept `--port`, `--no-open`, `--eval-history PATH` and `--eval-catalog CATALOG`
+(default: the example suites in `examples/eval/workbench_catalog.py`).
 
-```text
-http://127.0.0.1:8080
-```
+### Option 2: Evaluations for your suites
 
-Useful variants:
+`agentbyte eval ui` opens the Evaluations pages on their own (no Playground). Point it at the
+Python file that defines your `EvalSuiteRegistry`:
 
 ```bash
-uv run agentbyte webui --dir . --port 8080 --host 127.0.0.1 --no-open
-uv run agentbyte webui --dir examples --port 8090
+uv sync --extra webui --extra sqlite
+uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # http://127.0.0.1:8090
 ```
 
-For this repository, the most reliable first-run path is the preset-backed launcher:
+- `--catalog` takes a `.py` file or a module path (`myapp.evaluations`); add `:name` only when the
+  file defines more than one registry. Without `--catalog` it serves the built-in smoke suite.
+- Runs are stored in `./eval.sqlite` unless you pass `--history PATH`. Only one process can write
+  a history file: stop the UI before `agentbyte eval run` on the same file, or use another file.
+- The same catalog works in the terminal: `uv run agentbyte eval run capitals --catalog examples/eval/workbench_catalog.py`.
+- To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
-```bash
-uv run python examples/webui/presets_webui.py
-```
+Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
+notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/study/05-eval_topics/`.
+
+### Option 3: Scan a project directory
 
-Use `agentbyte webui --dir ...` when you have a directory of exportable demo modules, for example:
+`agentbyte webui --dir PATH` loads entities from Python files. Discovery is convention-based: each
+module (or package) must define a top-level variable named `agent`, `orchestrator` or `workflow`
+holding an Agentbyte object. Folders without those names show "No agents, orchestrators, or
+workflows found"; this is the case for `examples/`, which uses the launchers above instead.
 
 ```python
-# my_entities.py
-agent = ...
-workflow = ...
-orchestrator = ...
+# my_entities/support.py
+agent = Agent(name="support", ...)
 ```
 
-### Option 3: Run it programmatically
+```bash
+uv run agentbyte webui --dir my_entities
+uv run agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py   # both sections
+```
 
-Use this when you want to serve in-memory entities directly from Python.
+### Option 4: From Python
 
 ```python
 from agentbyte.webui import serve
 
-serve(entities=[agent], port=8080, auto_open=True)
+serve(entities=[agent], port=8080)
+serve(entities=[agent], eval_registry=registry, eval_history="eval.sqlite")   # with Evaluations
 ```
 
 ### Quick Troubleshooting
 
-If the app does not start:
-
-```bash
-uv sync --extra webui
-```
-
-If port `8080` is already in use:
-
-```bash
-uv run agentbyte webui --dir . --port 8090
-```
-
-If you do not want the browser to open automatically:
-
-```bash
-uv run agentbyte webui --dir . --no-open
-```
+- Port in use: add `--port 8090`. The UI works on any port.
+- No browser wanted: add `--no-open`.
+- "No agents, orchestrators, or workflows found": the `--dir` folder has no module with a top-level
+  `agent`, `orchestrator` or `workflow`; use a launcher from Option 1 to see the samples.
+- No Evaluations switch in the header: start `agentbyte webui` with `--eval-catalog`, or use `agentbyte eval ui`.
+- After upgrading, hard-reload the page (Cmd/Ctrl-Shift-R) to load the new UI.
 
 ## Development
 

{agentbyte-0.32.2 → agentbyte-0.34.0}/README.md

@@ -2,12 +2,16 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.32.2**
+Current release: **0.34.0**
 
 ## Building an Agent
 
 Every example below builds on the same domain: a customer **support agent**. Start with a plain agent and a model client — no tools, no middleware:
 
+```bash
+pip install "agentbyte[openai]"   # the model client lives in an extra; see Install below
+```
+
 ```python
 from agentbyte import Agent
 from agentbyte.llm import OpenAIChatCompletionClient
@@ -214,160 +218,122 @@ Reading a trace: `chat gpt-4.1-mini` spans show **per-call** usage/cost/finish r
 
 Python requirement: **3.11+**
 
-```bash
-uv sync --all-groups
-```
-
-Optional extras:
+The base package (`pip install agentbyte`) has no model provider or storage driver. Add the
+extras for the features you use. Using a feature without its extra raises
+`MissingExtraError` (an `ImportError`) that names the extra to install.
 
 ```bash
-uv sync --extra openai        # OpenAI client
-uv sync --extra azureopenai   # Azure OpenAI client + azure-identity
-uv sync --extra aws           # S3 datasets (aioboto3)
-uv sync --extra otel          # OpenTelemetry export
-uv sync --extra webui         # WebUI playground (FastAPI)
-uv sync --extra sqlite        # SQLite stores: datasets, workflow checkpoints, eval history
-uv sync --extra postgres      # PostgreSQL stores: usage logging, eval history
-uv sync --extra sql           # sqlite + postgres
-uv sync --extra optim         # prompt optimization (GEPA)
-uv sync --extra viz           # workflow diagrams (graphviz)
-uv sync --extra research      # research tools
-```
-
-### Install in another project (pip / uv add)
-
-Use extras to enable provider + telemetry support:
-
-```bash
-pip install "agentbyte[azureopenai,otel]"
-```
-
-```bash
-uv add "agentbyte[azureopenai,otel]"
-```
-
-For the browser WebUI:
-
-```bash
-pip install "agentbyte[webui]"
+pip install "agentbyte[openai,sqlite]"
 # or
-uv add "agentbyte[webui]"
+uv add "agentbyte[openai,sqlite]"
 ```
 
-Install all optional features:
+| Extra | Installs | Needed for |
+| --- | --- | --- |
+| `openai` | `openai` | `OpenAIChatCompletionClient`, `OpenAIEmbeddingClient` |
+| `azureopenai` | `openai`, `azure-identity` | Azure OpenAI clients and Entra ID credentials |
+| `aws` | `aioboto3` | S3 dataset sources, writers and publishers |
+| `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
+| `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
+| `sqlite` | `aiosqlite` | SQLite datasets, workflow checkpoints, evaluation history |
+| `postgres` | `sqlalchemy[asyncio]`, `sqlmodel`, `asyncpg` | `SqlUsageBackend`, `SqlEvalRunStore` |
+| `sql` | `sqlite` + `postgres` | Both SQL backends |
+| `tiktoken` | `tiktoken` | `TiktokenCounter` for exact compaction token counts |
+| `optim` | `gepa` | `optimize_with_gepa()` |
+| `viz` | `graphviz` | `WorkflowVisualizer.to_dot()` |
+| `research` | `beautifulsoup4`, `html2text`, `arxiv`, `youtube-transcript-api` | `agentbyte.tools.research_tools` |
+| `all` | every extra above | Everything |
+
+Contributors working in this repository install every extra plus the local tooling groups
+(`test`, `dev`, `experiment`):
 
 ```bash
-pip install "agentbyte[all]"
-# or
-uv add "agentbyte[all]"
+uv sync --all-extras --all-groups
 ```
 
-Note: the Azure extra is `azureopenai`.
-
 ## Run The WebUI
 
-### Option 1: Run the preset-backed app
-
-This is the easiest way to see the WebUI working end to end with real preset entities:
-- preset agents
-- preset orchestrators
-- preset workflow
-
-Step 1. Install the WebUI extra:
+The WebUI has two sections: the **Playground** (chat with agents, orchestrators and workflows) and
+**Evaluations** (run registered evaluation suites and inspect the results). Install the extra first:
 
 ```bash
-uv sync --extra webui
+uv sync --extra webui            # add --extra sqlite for Evaluations
 ```
 
-Step 2. Start the preset-backed app:
-
-```bash
-uv run python examples/webui/presets_webui.py
-```
+### Option 1: Sample apps
 
-Step 3. Open the browser:
-
-```text
-http://127.0.0.1:8080
-```
-
-If auto-open is enabled in your environment, the browser may open automatically.
-
-### Option 2: Run the WebUI against your current project directory
-
-Use this when you want Agentbyte to scan a directory for exported `agent`, `workflow`, or `orchestrator` objects.
-
-Important: discovery is convention-based. The scanned directory must contain Python modules that expose top-level variables literally named `agent`, `workflow`, or `orchestrator`. If you point `--dir` at a folder that does not export those names, the UI will load but show `No entities found`.
-
-Step 1. Install the WebUI extra:
+No API key needed: three demo agents, a team and a workflow on a fake model client.
 
 ```bash
-uv sync --extra webui
+uv run python examples/webui/in_memory.py                          # http://127.0.0.1:8070
+uv run python examples/webui/in_memory.py --eval-history eval.sqlite   # plus Evaluations
 ```
 
-Step 2. Launch the WebUI and scan the current directory:
+With your OpenAI or Azure OpenAI credentials, the preset agents, teams and workflow:
 
 ```bash
-uv run agentbyte webui --dir .
+uv sync --extra webui --extra openai        # or --extra azureopenai
+uv run python examples/webui/presets_webui.py                      # http://127.0.0.1:8080
+uv run python examples/webui/presets_webui.py --eval-history eval.sqlite
 ```
 
-Step 3. Open the browser:
+Both launchers accept `--port`, `--no-open`, `--eval-history PATH` and `--eval-catalog CATALOG`
+(default: the example suites in `examples/eval/workbench_catalog.py`).
 
-```text
-http://127.0.0.1:8080
-```
+### Option 2: Evaluations for your suites
 
-Useful variants:
+`agentbyte eval ui` opens the Evaluations pages on their own (no Playground). Point it at the
+Python file that defines your `EvalSuiteRegistry`:
 
 ```bash
-uv run agentbyte webui --dir . --port 8080 --host 127.0.0.1 --no-open
-uv run agentbyte webui --dir examples --port 8090
+uv sync --extra webui --extra sqlite
+uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # http://127.0.0.1:8090
 ```
 
-For this repository, the most reliable first-run path is the preset-backed launcher:
+- `--catalog` takes a `.py` file or a module path (`myapp.evaluations`); add `:name` only when the
+  file defines more than one registry. Without `--catalog` it serves the built-in smoke suite.
+- Runs are stored in `./eval.sqlite` unless you pass `--history PATH`. Only one process can write
+  a history file: stop the UI before `agentbyte eval run` on the same file, or use another file.
+- The same catalog works in the terminal: `uv run agentbyte eval run capitals --catalog examples/eval/workbench_catalog.py`.
+- To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
-```bash
-uv run python examples/webui/presets_webui.py
-```
+Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
+notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/study/05-eval_topics/`.
+
+### Option 3: Scan a project directory
 
-Use `agentbyte webui --dir ...` when you have a directory of exportable demo modules, for example:
+`agentbyte webui --dir PATH` loads entities from Python files. Discovery is convention-based: each
+module (or package) must define a top-level variable named `agent`, `orchestrator` or `workflow`
+holding an Agentbyte object. Folders without those names show "No agents, orchestrators, or
+workflows found"; this is the case for `examples/`, which uses the launchers above instead.
 
 ```python
-# my_entities.py
-agent = ...
-workflow = ...
-orchestrator = ...
+# my_entities/support.py
+agent = Agent(name="support", ...)
 ```
 
-### Option 3: Run it programmatically
+```bash
+uv run agentbyte webui --dir my_entities
+uv run agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py   # both sections
+```
 
-Use this when you want to serve in-memory entities directly from Python.
+### Option 4: From Python
 
 ```python
 from agentbyte.webui import serve
 
-serve(entities=[agent], port=8080, auto_open=True)
+serve(entities=[agent], port=8080)
+serve(entities=[agent], eval_registry=registry, eval_history="eval.sqlite")   # with Evaluations
 ```
 
 ### Quick Troubleshooting
 
-If the app does not start:
-
-```bash
-uv sync --extra webui
-```
-
-If port `8080` is already in use:
-
-```bash
-uv run agentbyte webui --dir . --port 8090
-```
-
-If you do not want the browser to open automatically:
-
-```bash
-uv run agentbyte webui --dir . --no-open
-```
+- Port in use: add `--port 8090`. The UI works on any port.
+- No browser wanted: add `--no-open`.
+- "No agents, orchestrators, or workflows found": the `--dir` folder has no module with a top-level
+  `agent`, `orchestrator` or `workflow`; use a launcher from Option 1 to see the samples.
+- No Evaluations switch in the header: start `agentbyte webui` with `--eval-catalog`, or use `agentbyte eval ui`.
+- After upgrading, hard-reload the page (Cmd/Ctrl-Shift-R) to load the new UI.
 
 ## Development