agentbyte 0.48.1__tar.gz → 0.50.0__tar.gz

{agentbyte-0.48.1 → agentbyte-0.50.0}/.gitignore

@@ -62,6 +62,10 @@ AGENTS.LOCAL.md
 data/flatfiles/*
 !data/flatfiles/.gitkeep
 
+# Comprehensive-eval run reports (per-run run<N>-<uuid> dirs): keep the directory, ignore contents
+data/reports/comprehensive-eval/*
+!data/reports/comprehensive-eval/.gitkeep
+
 # lean-ctx context runtime cache
 .lean-ctx/
 

{agentbyte-0.48.1 → agentbyte-0.50.0}/CHANGELOG.md

@@ -6,6 +6,128 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.50.0] - 2026-10-09
+
+### Changed (breaking)
+
+- `OptimizationJob` now takes `agent_factory`, a `suite` object and a `reflector` client instead of an
+  `optimizer` instance, a hand-written `AgentConfig` seed, a suite name and a pass mark. The seed is
+  derived from the agent the factory builds; tasks, judge, criteria, pass mark and agent identity come
+  from the suite. The factory runs when the job runs and is validated (fresh agents, no shared memory,
+  middleware or providers, faithful rebuild) before the first paid evaluation. Pick another optimizer
+  with `optimizer=` and `optimizer_options=`. Edits are applied to a fresh agent per evaluated task, so
+  memory, middleware and other runtime settings stay as the factory built them.
+
+### Added
+
+- Tuning of tools and of inline and file skills through the factory path; file skills keep their
+  resources and scripts, and class skills stay fixed and raise a clear error if a candidate replaces
+  them. `SkillsProvider.load_skills()` and `transform_skills()`, `TransformingSkillsSource`,
+  `OptimizationJob.prepare()` and `AgentBinding`.
+- `--out DIR` and `--format {text,json,markdown}` on `eval run`, `eval results`, `eval benchmark run`
+  and `eval comparison`, plus `SuiteRunSummary.to_markdown()`: write one Markdown or JSON report per
+  run into `DIR/<run-id>/`, so runs never overwrite each other.
+- `OptimizationJob.agent_id` and `agent_name`: `agentbyte optim run` now stores the run under the
+  suite's agent, as the Python path already could.
+- `--trace` flag on `optim run` and `optim report` (writes `trace.json` next to the report), and
+  `OptimizationReport.save_trace()`.
+- `--concurrency` on `eval benchmark run`: tasks evaluated at once per variant (default 4, at least 1),
+  as `eval run` already had. All variants still run together, so up to variants x concurrency tasks are
+  in flight.
+- `optim run --capture {summary,full}`: `full` keeps agent transcripts in the stored optimization trace
+  (the same word `eval run --capture` uses). `optim report --format text` prints a short summary.
+
+### Removed
+
+- `eval comparison --markdown`; use `--format markdown`.
+
+### Changed
+
+- `examples/eval/comprehensive_workbench.py` is now a plain catalog (a module-level `registry`) run with
+  `agentbyte eval benchmark run comprehensive --task-set mini --catalog examples/eval/comprehensive_workbench.py`.
+  It no longer has its own script arguments or history setup, and a benchmark run exits 0 once its
+  report is recorded rather than failing on low scores. The old documented `--catalog ...:build_registry`
+  form never loaded, because `build_registry` is a function.
+- `optim run` and `optim report` write one report per run, `DIR/<run-id>/report.md` (or `report.json`
+  with `--format json`), instead of `report.md`, `report.json` and `trace.json` directly in `DIR`; the
+  trace is written only with `--trace`. `--format` defaults to the screen format without `--out` and to
+  Markdown with it. `OptimizationReport.save(directory)` is unchanged.
+- The eval_app example reads the history backend from the environment and declares its agent identity;
+  `run_optim.py` and `run_webui.py` default to the same SQLite path as `run_eval.py`.
+- WebUI Evaluations page lists Eval suites first and Benchmarks below them; the WebUI eval docs and the
+  optimization guide cover the environment-driven credentials and history setup.
+
+### Fixed
+
+- WebUI item page: expanded Tool Call and Tool Result details were unreadable (light text on a white
+  box); they now use a dark text colour.
+
+## [0.49.0] - 2026-10-08
+
+### Added
+
+- Environment-selected evaluation history: `EvalHistorySettings`, `open_history`, `describe_history`
+  and `history_exists` in `agentbyte.eval.history`. `AGENTBYTE_EXPERIMENT_HISTORY_BACKEND`
+  (`sqlite` or `postgres`), `AGENTBYTE_EXPERIMENT_HISTORY_SCHEMA`, `AGENTBYTE_EXPERIMENT_HISTORY_URL`
+  and `PG_*` (also read from a project `.env`) now choose the history for the `agentbyte` CLI and the
+  WebUI alike.
+- `OptimizationJob` (`agentbyte.optim`) and `agentbyte optim run <job.py>`: run an optimization from
+  the CLI and record it in the selected history; the run fails fast if the history is locked or
+  unreachable, before any model call.
+- `TaskSet.default`: a task set can declare itself preselected in the benchmark page. With no default
+  nothing is checked and Run benchmark stays disabled until a set is chosen. `GET /api/eval/task-sets`
+  returns `default`.
+- Every history table gets server-side `created_at` and `updated_at` columns.
+
+- Run-to-run comparison for recorded evaluation runs (spec 0072): `compare_runs(summaries)` in
+  `agentbyte.eval.workbench` aligns two or more `SuiteRunSummary` objects by task name (first is the
+  baseline) and reports per-task score deltas, per-run means/improvement/usage deltas, and
+  `RegressionFlag`s. Surfaced as `GET /api/eval/compare?runs=…`, the `agentbyte eval compare` CLI, and
+  a UI compare page.
+- Eval benchmarks and comparison reports (spec 0073): `TaskSet` (named, tiered, combinable task
+  bundles), `BenchmarkVariant`, and `EvalBenchmark` in `agentbyte.eval.workbench`. One benchmark
+  execution runs every variant over the chosen task set(s) and records **one** `ComparisonReport`
+  (`ComparisonRunRecord` + `comparison_runs` table), with `to_markdown()` and per-task grouping. The
+  `agentbyte.presets` benchmark preset ships the comprehensive multi-agent benchmark (four variants,
+  `mini`/`medium`/`hard` task sets).
+- Benchmark HTTP + CLI surface: `GET /api/eval/{task-sets,benchmarks,comparisons}`,
+  `GET /api/eval/comparisons/{id}` (`{run, report, markdown}`), a non-blocking
+  `POST /api/eval/benchmarks/{name}/runs` (returns a QUEUED comparison id; variants run in the
+  background), and `agentbyte eval benchmark list|run` / `eval comparisons` / `eval comparison <id>
+  [--markdown]`.
+
+### Changed
+
+- **Breaking:** the history database layout is now version 6. The store creates tables but never alters
+  them, so an existing history database (SQLite file or PostgreSQL schema) at an older layout is
+  refused; drop and recreate it. Documentation of older-layout migrations was removed.
+- The Evaluations landing shows only benchmark and eval-suite cards, and the benchmark page is two
+  columns (run panel on the left, comparison reports on the right).
+- `examples/webui/eval_optimization_webui.py` replaces `comprehensive_webui.py` and serves every suite
+  and the benchmark, each card switchable by flag. The `contracting_agent_eval.py` example now uses
+  `CheckEvaluator` (formerly `LocalEvaluator`) and unused example imports were removed.
+- Tests no longer cover files under `examples/`, and the suite pins the history backend to SQLite so a
+  developer's `.env` never selects PostgreSQL.
+
+- The Evaluations landing is now a gallery of **two eval-card types** — benchmarks (compare variants
+  → one report) and per-suite **eval-suite** cards (one target over time) — each opening the same
+  run-a-new-execution + history workspace. New suite, benchmark, and comparison detail pages; a
+  benchmark launch is observable (submit + poll) and resilient (a failed variant is listed; the
+  surviving variants still compare). The agent-gallery page was removed from the UI
+  (`GET /api/eval/agents` remains for the `?agent=` run filter).
+- A running benchmark now shows **live per-variant progress**: launching jumps into the comparison
+  page, which lists each variant with its own task-level progress bar, status, and `starting…` state,
+  so you can see which variant is still running and how far along it is. `GET /api/eval/comparisons/{id}`
+  returns a `progress` summary and the active `variant_runs` while a comparison runs (backed by a new
+  `list_runs(comparison=…)` filter and `ComparisonRunRecord.is_active`); a comparison is marked
+  RUNNING when its variants start, and any comparison left active by a previous process is failed on
+  startup. The landing's "Recent suite runs" is scoped to the running catalog's suites, so it stays
+  consistent with the suite cards instead of showing runs recorded by other catalogs.
+- The comprehensive multi-agent example moved to the benchmark model: `comprehensive_workbench.py`
+  takes `--task-set` (combinable) and records one comparison report; `eval_optimization_webui.py`
+  serves it. The standalone CSV `comprehensive_eval.py` and its marimo notebook are unchanged. Root-level
+  example scripts were regrouped under `examples/{llm,agents,tools,optim}/`.
+
 ## [0.48.1] - 2026-10-06
 
 ### Changed
@@ -695,8 +817,8 @@ Upgrade notes (breaking):
 
 ### Added
 
-- Added `examples/prompt_caching_cost_smoke_test.py`, a live smoke test that verifies prompt-caching cost accounting across `gpt-4.1-mini`, `gpt-4.1`, and `gpt-5.4-mini` through the raw chat completion client.
-- Added `examples/agent_prompt_caching_cost_smoke_test.py`, the same smoke test driven through the `Agent` layer, to verify caching accounting survives the agent execution loop.
+- Added `examples/llm/prompt_caching_cost_smoke_test.py`, a live smoke test that verifies prompt-caching cost accounting across `gpt-4.1-mini`, `gpt-4.1`, and `gpt-5.4-mini` through the raw chat completion client.
+- Added `examples/llm/agent_prompt_caching_cost_smoke_test.py`, the same smoke test driven through the `Agent` layer, to verify caching accounting survives the agent execution loop.
 
 ## [0.24.0] - 2026-08-27
 

{agentbyte-0.48.1 → agentbyte-0.50.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.48.1
+Version: 0.50.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -84,7 +84,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.48.1**
+Current release: **0.50.0**
 
 ## Building an Agent
 
@@ -296,6 +296,25 @@ Per-call span attributes emitted by `OTelMiddleware`:
 
 Reading a trace: `chat gpt-4.1-mini` spans show **per-call** usage/cost/finish reason; the wrapping `agent <name>` span shows the **final accumulated** usage and task outcome for the whole `run()`/`run_stream()` call.
 
+## Documentation
+
+Choose the documentation layer that matches what you are trying to do:
+
+| Source | Start here when you need to |
+|---|---|
+| [Chapters](docs/chapters/) | Learn the concepts and book context behind agent systems |
+| [Design guide](docs/design-guide/) | Learn and apply AgentByte features, then understand their architecture and constraints |
+| [Specifications](docs/specs/) | Review feature decisions, contracts, and delivery history |
+| [Examples](examples/) | Run focused programs built on the public API |
+
+For distributed tool integration, start with [Use MCP tools with AgentByte](docs/design-guide/01-agent_topics/topic_mcp_usage.md)
+and its [runnable filesystem example](examples/mcp/basic_mcp_agent.py).
+
+Examples are grouped by capability. Start with the folder indexes for
+[agents](examples/agents/), [model clients](examples/llm/), [tools](examples/tools/),
+[optimization](examples/optim/), [evaluation](examples/eval/), or
+[orchestration](examples/orchestration/).
+
 ## Installation
 
 Python requirement: **3.11+**
@@ -387,7 +406,9 @@ uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # htt
 - The same catalog works in the terminal: `uv run agentbyte eval run capitals --catalog examples/eval/workbench_catalog.py`.
 - To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
-Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
+Start with the end-to-end design topic,
+`docs/design-guide/05-eval_topics/07_topic_evaluation_end_to_end.md`
+(dataset → suite → history → UI), and the
 notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/design-guide/05-eval_topics/`.
 
 ### Option 3: Scan a project directory

{agentbyte-0.48.1 → agentbyte-0.50.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.48.1**
+Current release: **0.50.0**
 
 ## Building an Agent
 
@@ -214,6 +214,25 @@ Per-call span attributes emitted by `OTelMiddleware`:
 
 Reading a trace: `chat gpt-4.1-mini` spans show **per-call** usage/cost/finish reason; the wrapping `agent <name>` span shows the **final accumulated** usage and task outcome for the whole `run()`/`run_stream()` call.
 
+## Documentation
+
+Choose the documentation layer that matches what you are trying to do:
+
+| Source | Start here when you need to |
+|---|---|
+| [Chapters](docs/chapters/) | Learn the concepts and book context behind agent systems |
+| [Design guide](docs/design-guide/) | Learn and apply AgentByte features, then understand their architecture and constraints |
+| [Specifications](docs/specs/) | Review feature decisions, contracts, and delivery history |
+| [Examples](examples/) | Run focused programs built on the public API |
+
+For distributed tool integration, start with [Use MCP tools with AgentByte](docs/design-guide/01-agent_topics/topic_mcp_usage.md)
+and its [runnable filesystem example](examples/mcp/basic_mcp_agent.py).
+
+Examples are grouped by capability. Start with the folder indexes for
+[agents](examples/agents/), [model clients](examples/llm/), [tools](examples/tools/),
+[optimization](examples/optim/), [evaluation](examples/eval/), or
+[orchestration](examples/orchestration/).
+
 ## Installation
 
 Python requirement: **3.11+**
@@ -305,7 +324,9 @@ uv run agentbyte eval ui --catalog examples/eval/workbench_catalog.py      # htt
 - The same catalog works in the terminal: `uv run agentbyte eval run capitals --catalog examples/eval/workbench_catalog.py`.
 - To serve agents and Evaluations together: `agentbyte webui --dir my_entities --eval-catalog myapp/evaluations.py`.
 
-Start with the end-to-end guide, `docs/guides/evaluation-end-to-end.md` (dataset → suite → history → UI), and the
+Start with the end-to-end design topic,
+`docs/design-guide/05-eval_topics/07_topic_evaluation_end_to_end.md`
+(dataset → suite → history → UI), and the
 notebook `notebooks/usecases/eval/06-eval-end-to-end.ipynb` (real OpenAI/Azure model). Details: `docs/design-guide/05-eval_topics/`.
 
 ### Option 3: Scan a project directory

agentbyte-0.50.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.50.0"
+VERSION = __version__