agentbyte 0.48.0__tar.gz → 0.49.0__tar.gz

{agentbyte-0.48.0 → agentbyte-0.49.0}/.gitignore

@@ -62,6 +62,10 @@ AGENTS.LOCAL.md
 data/flatfiles/*
 !data/flatfiles/.gitkeep
 
+# Comprehensive-eval run reports (per-run run<N>-<uuid> dirs): keep the directory, ignore contents
+data/reports/comprehensive-eval/*
+!data/reports/comprehensive-eval/.gitkeep
+
 # lean-ctx context runtime cache
 .lean-ctx/
 

{agentbyte-0.48.0 → agentbyte-0.49.0}/CHANGELOG.md

@@ -6,6 +6,83 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.49.0] - 2026-10-08
+
+### Added
+
+- Environment-selected evaluation history: `EvalHistorySettings`, `open_history`, `describe_history`
+  and `history_exists` in `agentbyte.eval.history`. `AGENTBYTE_EXPERIMENT_HISTORY_BACKEND`
+  (`sqlite` or `postgres`), `AGENTBYTE_EXPERIMENT_HISTORY_SCHEMA`, `AGENTBYTE_EXPERIMENT_HISTORY_URL`
+  and `PG_*` (also read from a project `.env`) now choose the history for the `agentbyte` CLI and the
+  WebUI alike.
+- `OptimizationJob` (`agentbyte.optim`) and `agentbyte optim run <job.py>`: run an optimization from
+  the CLI and record it in the selected history; the run fails fast if the history is locked or
+  unreachable, before any model call.
+- `TaskSet.default`: a task set can declare itself preselected in the benchmark page. With no default
+  nothing is checked and Run benchmark stays disabled until a set is chosen. `GET /api/eval/task-sets`
+  returns `default`.
+- Every history table gets server-side `created_at` and `updated_at` columns.
+
+- Run-to-run comparison for recorded evaluation runs (spec 0072): `compare_runs(summaries)` in
+  `agentbyte.eval.workbench` aligns two or more `SuiteRunSummary` objects by task name (first is the
+  baseline) and reports per-task score deltas, per-run means/improvement/usage deltas, and
+  `RegressionFlag`s. Surfaced as `GET /api/eval/compare?runs=…`, the `agentbyte eval compare` CLI, and
+  a UI compare page.
+- Eval benchmarks and comparison reports (spec 0073): `TaskSet` (named, tiered, combinable task
+  bundles), `BenchmarkVariant`, and `EvalBenchmark` in `agentbyte.eval.workbench`. One benchmark
+  execution runs every variant over the chosen task set(s) and records **one** `ComparisonReport`
+  (`ComparisonRunRecord` + `comparison_runs` table), with `to_markdown()` and per-task grouping. The
+  `agentbyte.presets` benchmark preset ships the comprehensive multi-agent benchmark (four variants,
+  `mini`/`medium`/`hard` task sets).
+- Benchmark HTTP + CLI surface: `GET /api/eval/{task-sets,benchmarks,comparisons}`,
+  `GET /api/eval/comparisons/{id}` (`{run, report, markdown}`), a non-blocking
+  `POST /api/eval/benchmarks/{name}/runs` (returns a QUEUED comparison id; variants run in the
+  background), and `agentbyte eval benchmark list|run` / `eval comparisons` / `eval comparison <id>
+  [--markdown]`.
+
+### Changed
+
+- **Breaking:** the history database layout is now version 6. The store creates tables but never alters
+  them, so an existing history database (SQLite file or PostgreSQL schema) at an older layout is
+  refused; drop and recreate it. Documentation of older-layout migrations was removed.
+- The Evaluations landing shows only benchmark and eval-suite cards, and the benchmark page is two
+  columns (run panel on the left, comparison reports on the right).
+- `examples/webui/eval_optimization_webui.py` replaces `comprehensive_webui.py` and serves every suite
+  and the benchmark, each card switchable by flag. The `contracting_agent_eval.py` example now uses
+  `CheckEvaluator` (formerly `LocalEvaluator`) and unused example imports were removed.
+- Tests no longer cover files under `examples/`, and the suite pins the history backend to SQLite so a
+  developer's `.env` never selects PostgreSQL.
+
+- The Evaluations landing is now a gallery of **two eval-card types** — benchmarks (compare variants
+  → one report) and per-suite **eval-suite** cards (one target over time) — each opening the same
+  run-a-new-execution + history workspace. New suite, benchmark, and comparison detail pages; a
+  benchmark launch is observable (submit + poll) and resilient (a failed variant is listed; the
+  surviving variants still compare). The agent-gallery page was removed from the UI
+  (`GET /api/eval/agents` remains for the `?agent=` run filter).
+- A running benchmark now shows **live per-variant progress**: launching jumps into the comparison
+  page, which lists each variant with its own task-level progress bar, status, and `starting…` state,
+  so you can see which variant is still running and how far along it is. `GET /api/eval/comparisons/{id}`
+  returns a `progress` summary and the active `variant_runs` while a comparison runs (backed by a new
+  `list_runs(comparison=…)` filter and `ComparisonRunRecord.is_active`); a comparison is marked
+  RUNNING when its variants start, and any comparison left active by a previous process is failed on
+  startup. The landing's "Recent suite runs" is scoped to the running catalog's suites, so it stays
+  consistent with the suite cards instead of showing runs recorded by other catalogs.
+- The comprehensive multi-agent example moved to the benchmark model: `comprehensive_workbench.py`
+  takes `--task-set` (combinable) and records one comparison report; `eval_optimization_webui.py`
+  serves it. The standalone CSV `comprehensive_eval.py` and its marimo notebook are unchanged. Root-level
+  example scripts were regrouped under `examples/{llm,agents,tools,optim}/`.
+
+## [0.48.1] - 2026-10-06
+
+### Changed
+
+- The WebUI agent page is now two-column: the suites to run on the left, the agent's grouped
+  history (All / Eval runs / Optimization reports) on the right (stacked on mobile).
+- Examples (`examples/eval/run_eval.py`, `examples/optim/run_optim.py`, `examples/webui/run_webui.py`)
+  open evaluation history through the unified `SqlEvalRunStore` with a caller-owned `AppSettings`
+  (SQLite or PostgreSQL + schema, chosen by `AGENTBYTE_EXPERIMENT_HISTORY_BACKEND`), and declare
+  their suite's agent so it appears in the agent gallery.
+
 ## [0.48.0] - 2026-10-06
 
 ### Added
@@ -684,8 +761,8 @@ Upgrade notes (breaking):
 
 ### Added
 
-- Added `examples/prompt_caching_cost_smoke_test.py`, a live smoke test that verifies prompt-caching cost accounting across `gpt-4.1-mini`, `gpt-4.1`, and `gpt-5.4-mini` through the raw chat completion client.
-- Added `examples/agent_prompt_caching_cost_smoke_test.py`, the same smoke test driven through the `Agent` layer, to verify caching accounting survives the agent execution loop.
+- Added `examples/llm/prompt_caching_cost_smoke_test.py`, a live smoke test that verifies prompt-caching cost accounting across `gpt-4.1-mini`, `gpt-4.1`, and `gpt-5.4-mini` through the raw chat completion client.
+- Added `examples/llm/agent_prompt_caching_cost_smoke_test.py`, the same smoke test driven through the `Agent` layer, to verify caching accounting survives the agent execution loop.
 
 ## [0.24.0] - 2026-08-27
 

{agentbyte-0.48.0 → agentbyte-0.49.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.48.0
+Version: 0.49.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -84,7 +84,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.48.0**
+Current release: **0.49.0**
 
 ## Building an Agent
 
@@ -296,6 +296,25 @@ Per-call span attributes emitted by `OTelMiddleware`:
 
 Reading a trace: `chat gpt-4.1-mini` spans show **per-call** usage/cost/finish reason; the wrapping `agent <name>` span shows the **final accumulated** usage and task outcome for the whole `run()`/`run_stream()` call.
 
+## Documentation
+
+Choose the documentation layer that matches what you are trying to do:
+
+| Source | Start here when you need to |
+|---|---|
+| [Chapters](docs/chapters/) | Learn the concepts and book context behind agent systems |
+| [Design guide](docs/design-guide/) | Learn and apply AgentByte features, then understand their architecture and constraints |
+| [Specifications](docs/specs/) | Review feature decisions, contracts, and delivery history |
+| [Examples](examples/) | Run focused programs built on the public API |
+
+For distributed tool integration, start with [Use MCP tools with AgentByte](docs/design-guide/01-agent_topics/topic_mcp_usage.md)
+and its [runnable filesystem example](examples/mcp/basic_mcp_agent.py).
+
+Examples are grouped by capability. Start with the folder indexes for
+[agents](examples/agents/), [model clients](examples/llm/), [tools](examples/tools/),
+[optimization](examples/optim/), [evaluation](examples/eval/), or
+[orchestration](examples/orchestration/).
+
 ## Installation
 
 Python requirement: **3.11+**
@@ -387,7 +406,9 @@ uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # htt
 - The same catalog works in the terminal: `uv run agentbyte eval run capitals --catalog examples/eval/workbench_catalog.py`.
 - To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
-Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
+Start with the end-to-end design topic,
+`docs/design-guide/05-eval_topics/07_topic_evaluation_end_to_end.md`
+(dataset → suite → history → UI), and the
 notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/design-guide/05-eval_topics/`.
 
 ### Option 3: Scan a project directory

{agentbyte-0.48.0 → agentbyte-0.49.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.48.0**
+Current release: **0.49.0**
 
 ## Building an Agent
 
@@ -214,6 +214,25 @@ Per-call span attributes emitted by `OTelMiddleware`:
 
 Reading a trace: `chat gpt-4.1-mini` spans show **per-call** usage/cost/finish reason; the wrapping `agent <name>` span shows the **final accumulated** usage and task outcome for the whole `run()`/`run_stream()` call.
 
+## Documentation
+
+Choose the documentation layer that matches what you are trying to do:
+
+| Source | Start here when you need to |
+|---|---|
+| [Chapters](docs/chapters/) | Learn the concepts and book context behind agent systems |
+| [Design guide](docs/design-guide/) | Learn and apply AgentByte features, then understand their architecture and constraints |
+| [Specifications](docs/specs/) | Review feature decisions, contracts, and delivery history |
+| [Examples](examples/) | Run focused programs built on the public API |
+
+For distributed tool integration, start with [Use MCP tools with AgentByte](docs/design-guide/01-agent_topics/topic_mcp_usage.md)
+and its [runnable filesystem example](examples/mcp/basic_mcp_agent.py).
+
+Examples are grouped by capability. Start with the folder indexes for
+[agents](examples/agents/), [model clients](examples/llm/), [tools](examples/tools/),
+[optimization](examples/optim/), [evaluation](examples/eval/), or
+[orchestration](examples/orchestration/).
+
 ## Installation
 
 Python requirement: **3.11+**
@@ -305,7 +324,9 @@ uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # htt
 - The same catalog works in the terminal: `uv run agentbyte eval run capitals --catalog examples/eval/workbench_catalog.py`.
 - To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
-Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
+Start with the end-to-end design topic,
+`docs/design-guide/05-eval_topics/07_topic_evaluation_end_to_end.md`
+(dataset → suite → history → UI), and the
 notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/design-guide/05-eval_topics/`.
 
 ### Option 3: Scan a project directory

agentbyte-0.49.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.49.0"
+VERSION = __version__