agentbyte 0.49.0__tar.gz → 0.50.0__tar.gz

{agentbyte-0.49.0 → agentbyte-0.50.0}/CHANGELOG.md

@@ -6,6 +6,62 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.50.0] - 2026-10-09
+
+### Changed (breaking)
+
+- `OptimizationJob` now takes `agent_factory`, a `suite` object and a `reflector` client instead of an
+  `optimizer` instance, a hand-written `AgentConfig` seed, a suite name and a pass mark. The seed is
+  derived from the agent the factory builds; tasks, judge, criteria, pass mark and agent identity come
+  from the suite. The factory runs when the job runs and is validated (fresh agents, no shared memory,
+  middleware or providers, faithful rebuild) before the first paid evaluation. Pick another optimizer
+  with `optimizer=` and `optimizer_options=`. Edits are applied to a fresh agent per evaluated task, so
+  memory, middleware and other runtime settings stay as the factory built them.
+
+### Added
+
+- Tuning of tools and of inline and file skills through the factory path; file skills keep their
+  resources and scripts, and class skills stay fixed and raise a clear error if a candidate replaces
+  them. `SkillsProvider.load_skills()` and `transform_skills()`, `TransformingSkillsSource`,
+  `OptimizationJob.prepare()` and `AgentBinding`.
+- `--out DIR` and `--format {text,json,markdown}` on `eval run`, `eval results`, `eval benchmark run`
+  and `eval comparison`, plus `SuiteRunSummary.to_markdown()`: write one Markdown or JSON report per
+  run into `DIR/<run-id>/`, so runs never overwrite each other.
+- `OptimizationJob.agent_id` and `agent_name`: `agentbyte optim run` now stores the run under the
+  suite's agent, as the Python path already could.
+- `--trace` flag on `optim run` and `optim report` (writes `trace.json` next to the report), and
+  `OptimizationReport.save_trace()`.
+- `--concurrency` on `eval benchmark run`: tasks evaluated at once per variant (default 4, at least 1),
+  as `eval run` already had. All variants still run together, so up to variants x concurrency tasks are
+  in flight.
+- `optim run --capture {summary,full}`: `full` keeps agent transcripts in the stored optimization trace
+  (the same word `eval run --capture` uses). `optim report --format text` prints a short summary.
+
+### Removed
+
+- `eval comparison --markdown`; use `--format markdown`.
+
+### Changed
+
+- `examples/eval/comprehensive_workbench.py` is now a plain catalog (a module-level `registry`) run with
+  `agentbyte eval benchmark run comprehensive --task-set mini --catalog examples/eval/comprehensive_workbench.py`.
+  It no longer has its own script arguments or history setup, and a benchmark run exits 0 once its
+  report is recorded rather than failing on low scores. The old documented `--catalog ...:build_registry`
+  form never loaded, because `build_registry` is a function.
+- `optim run` and `optim report` write one report per run, `DIR/<run-id>/report.md` (or `report.json`
+  with `--format json`), instead of `report.md`, `report.json` and `trace.json` directly in `DIR`; the
+  trace is written only with `--trace`. `--format` defaults to the screen format without `--out` and to
+  Markdown with it. `OptimizationReport.save(directory)` is unchanged.
+- The eval_app example reads the history backend from the environment and declares its agent identity;
+  `run_optim.py` and `run_webui.py` default to the same SQLite path as `run_eval.py`.
+- WebUI Evaluations page lists Eval suites first and Benchmarks below them; the WebUI eval docs and the
+  optimization guide cover the environment-driven credentials and history setup.
+
+### Fixed
+
+- WebUI item page: expanded Tool Call and Tool Result details were unreadable (light text on a white
+  box); they now use a dark text colour.
+
 ## [0.49.0] - 2026-10-08
 
 ### Added

{agentbyte-0.49.0 → agentbyte-0.50.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.49.0
+Version: 0.50.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -84,7 +84,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.49.0**
+Current release: **0.50.0**
 
 ## Building an Agent
 

{agentbyte-0.49.0 → agentbyte-0.50.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.49.0**
+Current release: **0.50.0**
 
 ## Building an Agent
 

agentbyte-0.50.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.50.0"
+VERSION = __version__

{agentbyte-0.49.0 → agentbyte-0.50.0}/src/agentbyte/cli/eval_commands.py

@@ -13,6 +13,9 @@ import json
 import sys
 from typing import Any
 
+from agentbyte.cli.history_args import HistoryPathAction, history_is_explicit
+from agentbyte.cli.report_output import deliver
+
 EXIT_PASSED = 0
 EXIT_FAILED = 1
 EXIT_ERROR = 2
@@ -21,12 +24,28 @@ EXIT_INTERRUPTED = 130
 DEFAULT_CATALOG = "agentbyte.eval.workbench.smoke"
 DEFAULT_HISTORY = ".agentbyte/eval/default/eval.sqlite"
 DEFAULT_UI_PORT = 8090
+CONCURRENCY_HELP = "Tasks evaluated at once"
 
 
 class _CliError(Exception):
     """A configuration or storage problem reported with exit code 2."""
 
 
+def _add_report_flags(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--format",
+        choices=["text", "json", "markdown"],
+        default=None,
+        help="Output format (default: text on screen, Markdown in the --out file)",
+    )
+    parser.add_argument(
+        "--out",
+        metavar="DIR",
+        default=None,
+        help="Write the report to DIR/<run-id>/report.md (report.json with --format json)",
+    )
+
+
 def add_eval_parser(subparsers: Any) -> None:
     """Register ``agentbyte eval`` and its sub-commands."""
     eval_parser = subparsers.add_parser("eval", help="Run registered evaluation suites")
@@ -38,8 +57,8 @@ def add_eval_parser(subparsers: Any) -> None:
         f"(default: the built-in smoke suite, {DEFAULT_CATALOG})"
     )
     history_help = (
-        f"SQLite evaluation history file (default: ./{DEFAULT_HISTORY}); ignored when "
-        "AGENTBYTE_EXPERIMENT_HISTORY_BACKEND=postgres"
+        f"SQLite evaluation history file (default: ./{DEFAULT_HISTORY}). An explicit value "
+        "takes precedence over AGENTBYTE_EXPERIMENT_HISTORY_BACKEND=postgres."
     )
 
     list_parser = eval_sub.add_parser("list", help="List the suites in a catalog")
@@ -49,20 +68,20 @@ def add_eval_parser(subparsers: Any) -> None:
     run_parser = eval_sub.add_parser("run", help="Run one suite and record it")
     run_parser.add_argument("suite", help="Registered suite name")
     run_parser.add_argument("--catalog", default=DEFAULT_CATALOG, help=catalog_help)
-    run_parser.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
+    run_parser.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
     run_parser.add_argument(
         "--capture",
         choices=["summary", "full"],
         default="summary",
         help="What to store per item (default: summary; full also stores trajectories)",
     )
-    run_parser.add_argument("--concurrency", type=int, default=4, help="Tasks evaluated at once")
-    run_parser.add_argument("--format", choices=["text", "json"], default="text")
+    run_parser.add_argument("--concurrency", type=int, default=4, help=CONCURRENCY_HELP)
+    _add_report_flags(run_parser)
 
     results_parser = eval_sub.add_parser("results", help="Show the verdict of a recorded run")
     results_parser.add_argument("run_id", help="Run ID printed by 'agentbyte eval run'")
-    results_parser.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
-    results_parser.add_argument("--format", choices=["text", "json"], default="text")
+    results_parser.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
+    _add_report_flags(results_parser)
 
     compare_parser = eval_sub.add_parser(
         "compare", help="Compare two or more recorded runs task by task"
@@ -70,7 +89,7 @@ def add_eval_parser(subparsers: Any) -> None:
     compare_parser.add_argument(
         "run_ids", nargs="+", help="Run IDs to compare; the first is the baseline"
     )
-    compare_parser.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
+    compare_parser.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
     compare_parser.add_argument("--format", choices=["text", "json"], default="text")
     compare_parser.add_argument(
         "--regression-threshold",
@@ -100,19 +119,26 @@ def add_eval_parser(subparsers: Any) -> None:
         help="A task set to run (repeat to combine, e.g. --task-set mini --task-set hard)",
     )
     bench_run.add_argument("--catalog", default=DEFAULT_CATALOG, help=catalog_help)
-    bench_run.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
+    bench_run.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
     bench_run.add_argument(
         "--capture",
         choices=["summary", "full"],
         default="summary",
         help="What to store per item (default: summary; full also stores trajectories)",
     )
-    bench_run.add_argument("--format", choices=["text", "json", "markdown"], default="text")
+    bench_run.add_argument(
+        "--concurrency",
+        type=int,
+        default=4,
+        help="Tasks evaluated at once per variant; all variants run together, so up to "
+        "(variants x this) tasks are in flight",
+    )
+    _add_report_flags(bench_run)
 
     comparisons_parser = eval_sub.add_parser(
         "comparisons", help="List recorded benchmark comparison reports"
     )
-    comparisons_parser.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
+    comparisons_parser.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
     comparisons_parser.add_argument("--benchmark", help="Only comparisons of this benchmark")
     comparisons_parser.add_argument("--format", choices=["text", "json"], default="text")
 
@@ -120,17 +146,14 @@ def add_eval_parser(subparsers: Any) -> None:
         "comparison", help="Show one recorded benchmark comparison report"
     )
     comparison_parser.add_argument("comparison_id", help="Comparison ID from 'eval comparisons'")
-    comparison_parser.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
-    comparison_parser.add_argument("--format", choices=["text", "json", "markdown"], default="text")
-    comparison_parser.add_argument(
-        "--markdown", action="store_true", help="Print the report as Markdown (same as --format markdown)"
-    )
+    comparison_parser.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
+    _add_report_flags(comparison_parser)
 
     ui_parser = eval_sub.add_parser(
         "ui", help="Open the evaluation pages in the browser (Evaluations only, no Playground)"
     )
     ui_parser.add_argument("--catalog", default=DEFAULT_CATALOG, help=catalog_help)
-    ui_parser.add_argument("--history", default=DEFAULT_HISTORY, help=history_help)
+    ui_parser.add_argument("--history", default=DEFAULT_HISTORY, action=HistoryPathAction, help=history_help)
     ui_parser.add_argument("--port", "-p", type=int, default=DEFAULT_UI_PORT,
                            help=f"Port to serve on (default: {DEFAULT_UI_PORT})")
     ui_parser.add_argument("--host", default="127.0.0.1", help="Host to bind (default: 127.0.0.1)")
@@ -177,9 +200,9 @@ def _cmd_ui(args: argparse.Namespace) -> int:
         from agentbyte.webui import launch
     except (MissingExtraError, ImportError) as exc:
         raise _CliError(str(exc)) from exc
-    asyncio.run(_check_history_is_free(args.history))
+    asyncio.run(_check_history_is_free(args.history, force_sqlite=history_is_explicit(args)))
     print(
-        f"Evaluations: http://{args.host}:{args.port}  (history {_history_label(args.history)}; Ctrl-C to stop)",
+        f"Evaluations: http://{args.host}:{args.port}  (history {_history_label(args.history, force_sqlite=history_is_explicit(args))}; Ctrl-C to stop)",
         file=sys.stderr,
     )
     try:
@@ -199,12 +222,12 @@ def _cmd_ui(args: argparse.Namespace) -> int:
     return EXIT_PASSED
 
 
-async def _check_history_is_free(path: str) -> None:
+async def _check_history_is_free(path: str, *, force_sqlite: bool = False) -> None:
     """Fail fast when another process (a CLI run, another server) holds the history file."""
     from agentbyte.eval.history import EvalHistoryError
 
     try:
-        async with _open_store(path, writer=True):
+        async with _open_store(path, writer=True, force_sqlite=force_sqlite):
             pass
     except EvalHistoryError as exc:
         raise _CliError(f"{exc} (only one process can write {path!r} at a time)") from exc
@@ -248,9 +271,13 @@ def _cmd_list(args: argparse.Namespace) -> int:
 # -- run --------------------------------------------------------------------------
 
 
-def _cmd_run(args: argparse.Namespace) -> int:
+def _require_concurrency(args: argparse.Namespace) -> None:
     if args.concurrency < 1:
         raise _CliError("--concurrency must be at least 1")
+
+
+def _cmd_run(args: argparse.Namespace) -> int:
+    _require_concurrency(args)
     registry = load_registry(args.catalog)
     if args.suite not in registry:
         raise _CliError(f"Unknown evaluation suite {args.suite!r}")
@@ -269,7 +296,7 @@ async def _run(args: argparse.Namespace, registry: Any, started: list[str]) -> i
     from agentbyte.eval.history import EvalHistoryError
     from agentbyte.eval.workbench import EvalSuiteCoordinator, EvalSuiteError
 
-    text = args.format == "text"
+    text = args.format in (None, "text")
 
     def on_start(run: Any) -> None:
         started.append(run.id)
@@ -277,7 +304,7 @@ async def _run(args: argparse.Namespace, registry: Any, started: list[str]) -> i
         print(f"run {run.id} started (suite {args.suite})", file=sys.stderr, flush=True)
 
     try:
-        async with _open_store(args.history, writer=True) as store:
+        async with _open_store(args.history, writer=True, force_sqlite=history_is_explicit(args)) as store:
             await store.recover_interrupted()
             coordinator = EvalSuiteCoordinator(
                 store, registry, capture=args.capture, concurrency=args.concurrency
@@ -290,7 +317,7 @@ async def _run(args: argparse.Namespace, registry: Any, started: list[str]) -> i
         if started:
             print(f"run {started[0]} did not complete.", file=sys.stderr)
         raise _CliError(str(exc)) from exc
-    _print_summary(summary, args.format)
+    _print_summary(summary, args.format, args.out)
     return EXIT_PASSED if summary.passed else EXIT_FAILED
 
 
@@ -305,13 +332,13 @@ async def _results(args: argparse.Namespace) -> int:
     from agentbyte.eval.history import EvalHistoryError
     from agentbyte.eval.workbench import EvalSuiteCoordinator, EvalSuiteError
 
-    _require_history(args.history)
+    _require_history(args.history, force_sqlite=history_is_explicit(args))
     try:
-        async with _open_store(args.history, writer=False) as store:
+        async with _open_store(args.history, writer=False, force_sqlite=history_is_explicit(args)) as store:
             summary = await EvalSuiteCoordinator(store).summary(args.run_id)
     except (EvalSuiteError, EvalHistoryError) as exc:
         raise _CliError(str(exc)) from exc
-    _print_summary(summary, args.format)
+    _print_summary(summary, args.format, args.out)
     return EXIT_PASSED if summary.passed else EXIT_FAILED
 
 
@@ -328,9 +355,9 @@ async def _compare(args: argparse.Namespace) -> int:
 
     if len(args.run_ids) < 2:
         raise _CliError("compare needs at least two run IDs (the first is the baseline)")
-    _require_history(args.history)
+    _require_history(args.history, force_sqlite=history_is_explicit(args))
     try:
-        async with _open_store(args.history, writer=False) as store:
+        async with _open_store(args.history, writer=False, force_sqlite=history_is_explicit(args)) as store:
             coordinator = EvalSuiteCoordinator(store)
             summaries = [await coordinator.summary(run_id) for run_id in args.run_ids]
     except (EvalSuiteError, EvalHistoryError) as exc:
@@ -471,6 +498,7 @@ def _cmd_benchmark_list(args: argparse.Namespace) -> int:
 
 
 def _cmd_benchmark_run(args: argparse.Namespace) -> int:
+    _require_concurrency(args)
     registry = load_registry(args.catalog)
     task_sets = args.task_sets or []
     if not task_sets:
@@ -487,9 +515,11 @@ async def _benchmark_run(args: argparse.Namespace, registry: Any, task_sets: lis
     from agentbyte.eval.workbench import EvalSuiteCoordinator, EvalSuiteError
 
     try:
-        async with _open_store(args.history, writer=True) as store:
+        async with _open_store(args.history, writer=True, force_sqlite=history_is_explicit(args)) as store:
             await store.recover_interrupted()
-            coordinator = EvalSuiteCoordinator(store, registry, capture=args.capture)
+            coordinator = EvalSuiteCoordinator(
+                store, registry, capture=args.capture, concurrency=args.concurrency
+            )
             print(
                 f"running benchmark {args.name!r} over task sets: {', '.join(task_sets)}",
                 file=sys.stderr,
@@ -498,7 +528,7 @@ async def _benchmark_run(args: argparse.Namespace, registry: Any, task_sets: lis
             record = await coordinator.run_benchmark(args.name, task_sets, capture=args.capture)
     except (EvalSuiteError, EvalHistoryError) as exc:
         raise _CliError(str(exc)) from exc
-    _print_comparison_record(record, args.format)
+    _print_comparison_record(record, args.format, args.out)
     return EXIT_PASSED if record.status == "completed" else EXIT_FAILED
 
 
@@ -510,9 +540,9 @@ async def _comparisons(args: argparse.Namespace) -> int:
     from agentbyte.eval.history import EvalHistoryError
     from agentbyte.eval.workbench import EvalSuiteCoordinator, EvalSuiteError
 
-    _require_history(args.history)
+    _require_history(args.history, force_sqlite=history_is_explicit(args))
     try:
-        async with _open_store(args.history, writer=False) as store:
+        async with _open_store(args.history, writer=False, force_sqlite=history_is_explicit(args)) as store:
             coordinator = EvalSuiteCoordinator(store)
             records = await store.list_comparisons(
                 benchmark=args.benchmark, owner=coordinator.owner, limit=200
@@ -541,30 +571,31 @@ async def _comparison(args: argparse.Namespace) -> int:
     from agentbyte.eval.history import EvalHistoryError
     from agentbyte.eval.workbench import EvalSuiteCoordinator, EvalSuiteError
 
-    _require_history(args.history)
+    _require_history(args.history, force_sqlite=history_is_explicit(args))
     try:
-        async with _open_store(args.history, writer=False) as store:
+        async with _open_store(args.history, writer=False, force_sqlite=history_is_explicit(args)) as store:
             coordinator = EvalSuiteCoordinator(store)
             record = await store.get_comparison(args.comparison_id)
     except (EvalSuiteError, EvalHistoryError) as exc:
         raise _CliError(str(exc)) from exc
     if record is None or record.owner != coordinator.owner:
         raise _CliError(f"Unknown benchmark comparison {args.comparison_id!r}")
-    _print_comparison_record(record, "markdown" if args.markdown else args.format)
+    _print_comparison_record(record, args.format, args.out)
     return EXIT_PASSED if record.status == "completed" else EXIT_FAILED
 
 
-def _print_comparison_record(record: Any, output_format: str) -> None:
-    if output_format == "json":
-        print(json.dumps(record.model_dump(mode="json"), indent=2))
-        return
-    if output_format == "markdown":
+def _print_comparison_record(record: Any, output_format: str | None, out: str | None = None) -> None:
+    def render(kind: str) -> str:
+        if kind == "json":
+            return json.dumps(record.model_dump(mode="json"), indent=2)
         if record.report_data is None:
-            print(f"Comparison {record.id} has no report (status {record.status}).")
-            return
-        print(record.report().to_markdown())
-        return
+            return f"Comparison {record.id} has no report (status {record.status})."
+        return record.report().to_markdown()
+
+    deliver(output_format, out, render, lambda: _print_comparison_text(record), run_id=record.id)
 
+
+def _print_comparison_text(record: Any) -> None:
     print(f"benchmark {record.benchmark}  comparison {record.id}  status {record.status}")
     print(f"  task sets: {', '.join(record.task_sets)}")
     if record.error:
@@ -599,29 +630,29 @@ def load_registry(import_path: str) -> Any:
         raise _CliError(str(exc)) from exc
 
 
-def _open_store(path: str, *, writer: bool) -> Any:
+def _open_store(path: str, *, writer: bool, force_sqlite: bool = False) -> Any:
     from agentbyte._optional import MissingExtraError, require
     from agentbyte.eval.history import EvalHistoryError, EvalHistorySettings, open_history
 
     try:
         settings = EvalHistorySettings()
-        if not settings.is_postgres:
+        if force_sqlite or not settings.is_postgres:
             require("aiosqlite", extra="sqlite", feature="Evaluation history")
-        return open_history(path, writer=writer, settings=settings)
+        return open_history(path, writer=writer, settings=settings, force_sqlite=force_sqlite)
     except (MissingExtraError, EvalHistoryError) as exc:
         raise _CliError(str(exc)) from exc
 
 
-def _history_label(path: str) -> str:
+def _history_label(path: str, *, force_sqlite: bool = False) -> str:
     from agentbyte.eval.history import describe_history
 
-    return describe_history(path)
+    return describe_history(path, force_sqlite=force_sqlite)
 
 
-def _require_history(path: str) -> None:
+def _require_history(path: str, *, force_sqlite: bool = False) -> None:
     from agentbyte.eval.history import history_exists
 
-    if not history_exists(path):
+    if not history_exists(path, force_sqlite=force_sqlite):
         raise _CliError(f"History file {path!r} does not exist")
 
 
@@ -632,12 +663,18 @@ def _item_line(item: Any) -> str:
     return f"  #{item.item_index} {item.task_name}: {status} {score}{reason}"
 
 
-def _print_summary(summary: Any, output_format: str) -> None:
-    if output_format == "json":
-        data = summary.model_dump(mode="json")
-        data["failed_count"] = summary.failed_count
-        print(json.dumps(data, indent=2))
-        return
+def _print_summary(summary: Any, output_format: str | None, out: str | None = None) -> None:
+    def render(kind: str) -> str:
+        if kind == "json":
+            data = summary.model_dump(mode="json")
+            data["failed_count"] = summary.failed_count
+            return json.dumps(data, indent=2)
+        return summary.to_markdown()
+
+    deliver(output_format, out, render, lambda: _print_summary_text(summary), run_id=summary.run_id)
+
+
+def _print_summary_text(summary: Any) -> None:
     verdict = "PASSED" if summary.passed else "FAILED"
     print(f"suite {summary.suite}  run {summary.run_id}  status {summary.status}  {verdict}")
     print(

agentbyte-0.50.0/src/agentbyte/cli/history_args.py

@@ -0,0 +1,28 @@
+"""Argument parsing helpers for evaluation-history selection."""
+
+from __future__ import annotations
+
+import argparse
+from typing import Any
+
+
+class HistoryPathAction(argparse.Action):
+    """Record that ``--history`` was explicitly supplied as well as its path."""
+
+    def __call__(
+        self,
+        parser: argparse.ArgumentParser,
+        namespace: argparse.Namespace,
+        values: str | None,
+        option_string: str | None = None,
+    ) -> None:
+        setattr(namespace, self.dest, values)
+        setattr(namespace, "history_explicit", True)
+
+
+def history_is_explicit(args: Any) -> bool:
+    """Whether a CLI command supplied ``--history`` rather than accepting its default."""
+    return bool(getattr(args, "history_explicit", False))
+
+
+__all__ = ["HistoryPathAction", "history_is_explicit"]