agentbyte 0.50.0__tar.gz → 0.52.0__tar.gz

{agentbyte-0.50.0 → agentbyte-0.52.0}/CHANGELOG.md

@@ -6,6 +6,52 @@ The format follows Keep a Changelog principles and semantic versioning.
 
 ## [Unreleased]
 
+## [0.52.0] - 2026-10-10
+
+### Added
+
+- `AnthropicChatCompletionClient`: direct Anthropic Claude chat client (`agentbyte[anthropic]`),
+  matching the existing `OpenAIChatCompletionClient` construction/DI/retry/pricing/error-mapping
+  contract (`from_api_key()`, `_to_config`/`_from_config` without persisting secrets, tool-use,
+  streaming).
+- `AnthropicBedrockChatCompletionClient`: the same Claude Messages API client transported through
+  AWS Bedrock (`agentbyte[bedrock]`) via `AsyncAnthropicBedrock` (SigV4, boto3 credential chain),
+  constructed with `from_region()`. Shares conversion/parsing logic with the direct client through
+  `AnthropicConversionMixin`.
+- `CohereBedrockEmbeddingClient`: Cohere Embed v3 (English/Multilingual) and v4 embeddings via AWS
+  Bedrock Runtime (`agentbyte[aws]`, `aioboto3`), with `CohereInputType`/`CohereTruncate` options
+  and the same injected-or-fresh-client pattern as `dataset/s3.py`'s `s3_client()`.
+- `build_chat_client` auto-detects `anthropic` (`ANTHROPIC_API_KEY`) and `anthropic-bedrock`
+  (`AWS_REGION` + `ANTHROPIC_BEDROCK_MODEL`) alongside the existing OpenAI/Azure detection.
+- Pricing for Claude 5.5 (`claude-opus-5-5`, `claude-haiku-5-5`), Cohere Embed v3 (English/
+  Multilingual, estimate), and Cohere Embed v4 (verified against AWS Bedrock's own pricing page).
+
+### Fixed
+
+- `AnthropicBedrockChatCompletionClient` was built on `AsyncAnthropicBedrockMantle`, a Bedrock
+  access path authenticated by a bearer token (`AWS_BEARER_TOKEN_BEDROCK`), not AWS IAM
+  credentials — every call 404'd with a misleading "model does not exist" error against a real
+  account. Caught by live smoke-testing against real Bedrock, invisible to the mocked unit tests.
+  Swapped to `AsyncAnthropicBedrock`, the real SigV4/boto3-credential-chain transport.
+- The Bedrock Claude pricing normalizer didn't strip cross-region inference-profile prefixes
+  (`eu.`/`global.`, required for on-demand invoke on most accounts), silently falling back to
+  default pricing for real Bedrock calls.
+- `CohereBedrockEmbeddingClient` read billed tokens from a response-body field that doesn't exist
+  on real Bedrock responses, so `tokens_input`/`cost_estimate` were silently always zero despite
+  successful calls. Now reads the real `x-amzn-bedrock-input-token-count` response header.
+
+## [0.51.0] - 2026-10-09
+
+### Added
+
+- Per-dimension scores in the WebUI and reports: `SuiteItemVerdict.dimensions` and
+  `SuiteRunSummary.dimension_means`, `TaskCell.dimensions` and `RunColumn.mean_dimensions` (over the
+  shared tasks). The run page gets a Dimensions table, run comparisons and benchmark reports get
+  per-dimension means with the change against the baseline plus a per-task breakdown, and the run and
+  benchmark Markdown reports list the dimension means. Runs without dimensions are unchanged.
+- Benchmark report page: a "Show per-task" selector switches the task-by-variant matrix between the
+  overall score and any judged dimension, with the change against the baseline variant.
+
 ## [0.50.0] - 2026-10-09
 
 ### Changed (breaking)

{agentbyte-0.50.0 → agentbyte-0.52.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentbyte
-Version: 0.50.0
+Version: 0.52.0
 Summary: A toolkit for designing multiagent systems
 Author-email: MrDataPsycho <mr.data.psycho@gmail.com>
 License-Expression: LicenseRef-Proprietary
@@ -17,6 +17,8 @@ Requires-Dist: uvicorn[standard]>=0.44.0; extra == 'a2a'
 Provides-Extra: all
 Requires-Dist: aioboto3>=15.5.0; extra == 'all'
 Requires-Dist: aiosqlite>=0.22.1; extra == 'all'
+Requires-Dist: anthropic>=1.13.0; extra == 'all'
+Requires-Dist: anthropic[bedrock]>=1.13.0; extra == 'all'
 Requires-Dist: arxiv>=2.1; extra == 'all'
 Requires-Dist: asyncpg>=0.31.0; extra == 'all'
 Requires-Dist: azure-identity>=1.25.1; extra == 'all'
@@ -35,11 +37,15 @@ Requires-Dist: sqlmodel>=0.0.38; extra == 'all'
 Requires-Dist: tiktoken>=0.9.0; extra == 'all'
 Requires-Dist: uvicorn[standard]>=0.44.0; extra == 'all'
 Requires-Dist: youtube-transcript-api>=0.6; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=1.13.0; extra == 'anthropic'
 Provides-Extra: aws
 Requires-Dist: aioboto3>=15.5.0; extra == 'aws'
 Provides-Extra: azureopenai
 Requires-Dist: azure-identity>=1.25.1; extra == 'azureopenai'
 Requires-Dist: openai>=1.107.1; extra == 'azureopenai'
+Provides-Extra: bedrock
+Requires-Dist: anthropic[bedrock]>=1.13.0; extra == 'bedrock'
 Provides-Extra: gunicorn
 Requires-Dist: fastapi>=0.135.2; extra == 'gunicorn'
 Requires-Dist: gunicorn>=23.0.0; extra == 'gunicorn'
@@ -84,7 +90,7 @@ Description-Content-Type: text/markdown
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.50.0**
+Current release: **0.51.0**
 
 ## Building an Agent
 
@@ -333,7 +339,9 @@ uv add "agentbyte[openai,sqlite]"
 | --- | --- | --- |
 | `openai` | `openai` | `OpenAIChatCompletionClient`, `OpenAIEmbeddingClient` |
 | `azureopenai` | `openai`, `azure-identity` | Azure OpenAI clients and Entra ID credentials |
-| `aws` | `aioboto3` | S3 dataset sources, writers and publishers |
+| `anthropic` | `anthropic` | Direct Anthropic Claude chat client |
+| `bedrock` | `anthropic[bedrock]` | Anthropic Claude chat through AWS Bedrock |
+| `aws` | `aioboto3` | S3 dataset I/O and Cohere Bedrock embedding client |
 | `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
 | `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
 | `gunicorn` | `webui` + `gunicorn` | Production single-worker serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`) |

{agentbyte-0.50.0 → agentbyte-0.52.0}/README.md

@@ -2,7 +2,7 @@
 
 Agentbyte is an observability-first agentic AI framework for building and studying multiagent systems with a learning-first, implementation-oriented workflow.
 
-Current release: **0.50.0**
+Current release: **0.51.0**
 
 ## Building an Agent
 
@@ -251,7 +251,9 @@ uv add "agentbyte[openai,sqlite]"
 | --- | --- | --- |
 | `openai` | `openai` | `OpenAIChatCompletionClient`, `OpenAIEmbeddingClient` |
 | `azureopenai` | `openai`, `azure-identity` | Azure OpenAI clients and Entra ID credentials |
-| `aws` | `aioboto3` | S3 dataset sources, writers and publishers |
+| `anthropic` | `anthropic` | Direct Anthropic Claude chat client |
+| `bedrock` | `anthropic[bedrock]` | Anthropic Claude chat through AWS Bedrock |
+| `aws` | `aioboto3` | S3 dataset I/O and Cohere Bedrock embedding client |
 | `otel` | OpenTelemetry API, SDK, OTLP exporter | OpenTelemetry tracing and metrics |
 | `webui` | `fastapi`, `uvicorn` | `agentbyte webui` |
 | `gunicorn` | `webui` + `gunicorn` | Production single-worker serving (`gunicorn -k uvicorn.workers.UvicornWorker -w 1`) |

{agentbyte-0.50.0 → agentbyte-0.52.0}/pyproject.toml

@@ -24,6 +24,16 @@ azureopenai = [
     "openai>=1.107.1",
     "azure-identity>=1.25.1",
 ]
+anthropic = [
+    "anthropic>=1.13.0",
+]
+# anthropic[bedrock] does its own AWS SigV4 signing via AnthropicBedrock*; deliberately
+# not named "aws" (that extra is aioboto3, which this one does not need).
+bedrock = [
+    "anthropic[bedrock]>=1.13.0",
+]
+# aioboto3 for AWS I/O: S3 dataset access (dataset/s3.py) and Bedrock Runtime
+# (llm/bedrock/, e.g. CohereBedrockEmbeddingClient).
 aws = [
     "aioboto3>=15.5.0",
 ]
@@ -76,7 +86,7 @@ a2a = [
     "agentbyte[webui]",
 ]
 all = [
-    "agentbyte[openai,azureopenai,aws,otel,webui,gunicorn,sql,tiktoken,optim,viz,research]",
+    "agentbyte[openai,azureopenai,anthropic,bedrock,aws,otel,webui,gunicorn,sql,tiktoken,optim,viz,research]",
 ]
 
 

agentbyte-0.52.0/src/agentbyte/__about__.py

@@ -0,0 +1,2 @@
+__version__ = "0.52.0"
+VERSION = __version__

{agentbyte-0.50.0 → agentbyte-0.52.0}/src/agentbyte/eval/workbench/compare.py

@@ -17,14 +17,13 @@ from collections.abc import Sequence
 from enum import StrEnum
 from typing import Any
 
-from pydantic import BaseModel, ConfigDict
-
+from pydantic import BaseModel, ConfigDict, Field
 from agentbyte.eval.comparison import improvement_pct
 from agentbyte.messages import Usage
 
 from .benchmark import EvalBenchmark, TaskSet, combine_task_sets
 from .suite import EvalSuiteConfigError
-from .verdict import ItemOutcome, SuiteItemVerdict, SuiteRunSummary
+from .verdict import ItemOutcome, SuiteItemVerdict, SuiteRunSummary, mean_dimensions
 
 
 class RegressionFlag(StrEnum):
@@ -44,6 +43,7 @@ class TaskCell(BaseModel):
     run_id: str
     present: bool
     overall: float | None = None
+    dimensions: dict[str, float] = Field(default_factory=dict)
     outcome: ItemOutcome | None = None
     passed: bool | None = None
     delta_vs_baseline: float | None = None
@@ -83,6 +83,7 @@ class RunColumn(BaseModel):
     min_score: float
     is_baseline: bool
     mean_overall: float | None = None
+    mean_dimensions: dict[str, float] = Field(default_factory=dict)
     passed_count: int = 0
     scored_count: int = 0
     improvement_pct: float | None = None
@@ -163,6 +164,19 @@ class ComparisonReport(BaseModel):
             )
         lines.append("")
 
+        dimension_names = list(
+            dict.fromkeys(name for variant in self.variants for name in variant.run.mean_dimensions)
+        )
+        if dimension_names:
+            lines.append("## Dimension means")
+            dimension_header = ["Dimension", *(variant.variant_name for variant in self.variants)]
+            lines.append("| " + " | ".join(dimension_header) + " |")
+            lines.append("| " + " | ".join(["---"] * len(dimension_header)) + " |")
+            for name in dimension_names:
+                means = " | ".join(_fmt(variant.run.mean_dimensions.get(name)) for variant in self.variants)
+                lines.append(f"| {name} | {means} |")
+            lines.append("")
+
         if self.failed_variants:
             lines.append("## Failed variants")
             for failed in self.failed_variants:
@@ -375,6 +389,7 @@ def _cell(
         run_id=run_id,
         present=True,
         overall=verdict.overall,
+        dimensions=dict(verdict.dimensions),
         outcome=verdict.outcome,
         passed=verdict.passed,
         delta_vs_baseline=delta,
@@ -404,6 +419,9 @@ def _column(
         min_score=summary.min_score,
         is_baseline=is_baseline,
         mean_overall=mean,
+        mean_dimensions=mean_dimensions(
+            [v for name in shared_names if (v := run.get(name)) is not None]
+        ),
         passed_count=passed,
         scored_count=scored,
         improvement_pct=None if is_baseline else improvement_pct(baseline_mean, mean),

{agentbyte-0.50.0 → agentbyte-0.52.0}/src/agentbyte/eval/workbench/verdict.py

@@ -6,8 +6,7 @@ import math
 from datetime import datetime
 from enum import StrEnum
 
-from pydantic import BaseModel, ConfigDict
-
+from pydantic import BaseModel, ConfigDict, Field
 from agentbyte.eval.history import (
     CaptureMode,
     EvalItemRecord,
@@ -44,6 +43,7 @@ class SuiteItemVerdict(BaseModel):
     outcome: ItemOutcome
     passed: bool
     overall: float | None = None
+    dimensions: dict[str, float] = Field(default_factory=dict)
     scoring_status: ScoringStatus | None = None
     failure_reason: str | None = None
     execution_status: ExecutionStatus | None = None
@@ -82,6 +82,7 @@ class SuiteRunSummary(BaseModel):
     started_at: datetime | None
     finished_at: datetime | None
     items: list[SuiteItemVerdict]
+    dimension_means: dict[str, float] = Field(default_factory=dict)
 
     @property
     def failed_count(self) -> int:
@@ -120,6 +121,9 @@ class SuiteRunSummary(BaseModel):
             rows.append(("Error", _cell(self.error)))
         lines = [f"# Eval run: {self.suite or self.run_id}", "", "| | |", "|---|---|"]
         lines += [f"| {key} | {value} |" for key, value in rows]
+        if self.dimension_means:
+            lines += ["", "## Dimensions (mean over scored items)", "", "| Dimension | Mean |", "|---|---|"]
+            lines += [f"| {_cell(name)} | {mean:.2f} |" for name, mean in self.dimension_means.items()]
         lines += ["", "## Items", "", "| # | Task | Outcome | Score | Reason |", "|---|---|---|---|---|"]
         for item in self.items:
             score = f"{item.overall:g}" if item.overall is not None else "-"
@@ -148,6 +152,7 @@ def item_verdict(item: EvalItemRecord, min_score: float) -> SuiteItemVerdict:
         outcome=outcome,
         passed=outcome == "passed",
         overall=item.overall if item.scoring_status == "scored" else None,
+        dimensions=dict(item.dimensions) if item.scoring_status == "scored" else {},
         scoring_status=item.scoring_status,
         failure_reason=item.failure_reason,
         execution_status=item.execution_status,
@@ -216,9 +221,20 @@ def summarize(
         started_at=run.started_at,
         finished_at=run.finished_at,
         items=verdicts,
+        dimension_means=mean_dimensions(verdicts),
     )
 
 
+def mean_dimensions(verdicts: list[SuiteItemVerdict]) -> dict[str, float]:
+    """Mean score per dimension over the items that scored it, in first-seen order."""
+    values: dict[str, list[float]] = {}
+    for verdict in verdicts:
+        for name, value in verdict.dimensions.items():
+            if math.isfinite(value):
+                values.setdefault(name, []).append(value)
+    return {name: sum(scores) / len(scores) for name, scores in values.items()}
+
+
 def _total_usage(items: list[EvalItemRecord]) -> Usage | None:
     usages = [item.usage for item in items if item.usage is not None]
     if not usages:

{agentbyte-0.50.0 → agentbyte-0.52.0}/src/agentbyte/llm/__init__.py

@@ -69,6 +69,8 @@ __all__ = [
     "AzureOpenAIChatCompletionClientConfig",
     "AzureOpenAIEmbeddingClient",
     "AzureOpenAIEmbeddingClientConfig",
+    "AnthropicChatCompletionClient",
+    "AnthropicChatCompletionClientConfig",
     # Authentication utilities
     "get_default_token_provider",
     "get_certificate_token_provider",
@@ -77,6 +79,12 @@ __all__ = [
     "OpenAISettings",
     "AzureOpenAISettings",
     "AzureServicePrincipalSettings",
+    "AnthropicSettings",
+    "AnthropicBedrockChatCompletionClient",
+    "AnthropicBedrockChatCompletionClientConfig",
+    "CohereBedrockEmbeddingClient",
+    "CohereBedrockEmbeddingClientConfig",
+    "BedrockSettings",
     # Retry policy
     "RetryPolicy",
     # Exception types
@@ -123,9 +131,22 @@ if TYPE_CHECKING:  # editors and type checkers see the provider classes eagerly
         OpenAIEmbeddingClientConfig,
         OpenAISettings,
     )
+    from .anthropic import (
+        AnthropicChatCompletionClient,
+        AnthropicChatCompletionClientConfig,
+        AnthropicSettings,
+    )
+    from .bedrock import (
+        AnthropicBedrockChatCompletionClient,
+        AnthropicBedrockChatCompletionClientConfig,
+        BedrockSettings,
+        CohereBedrockEmbeddingClient,
+        CohereBedrockEmbeddingClientConfig,
+    )
 
-# Provider clients need optional extras (``openai``, ``azureopenai``); they are
-# imported on first access so ``import agentbyte`` works with core dependencies only.
+# Provider clients need optional extras (``openai``, ``azureopenai``, ``anthropic``);
+# they are imported on first access so ``import agentbyte`` works with core
+# dependencies only.
 _PROVIDER_EXPORTS = {
     "OpenAIChatCompletionClient": ".openai",
     "OpenAIChatCompletionClientConfig": ".openai",
@@ -141,6 +162,25 @@ _PROVIDER_EXPORTS = {
     "create_token_provider_from_string": ".azure",
     "AzureOpenAISettings": ".azure",
     "AzureServicePrincipalSettings": ".azure",
+    "AnthropicChatCompletionClient": ".anthropic",
+    "AnthropicChatCompletionClientConfig": ".anthropic",
+    "AnthropicSettings": ".anthropic",
+    "AnthropicBedrockChatCompletionClient": ".bedrock",
+    "AnthropicBedrockChatCompletionClientConfig": ".bedrock",
+    "CohereBedrockEmbeddingClient": ".bedrock",
+    "CohereBedrockEmbeddingClientConfig": ".bedrock",
+    "BedrockSettings": ".bedrock",
+}
+
+_PROVIDER_EXTRAS = {
+    ".openai": ("openai", "openai"),
+    ".azure": ("openai", "azureopenai"),
+    ".anthropic": ("anthropic", "anthropic"),
+    # `.bedrock` is intentionally absent: its two clients need different extras
+    # (`anthropic[bedrock]` vs. `aioboto3`/`aws`), so `llm/bedrock/__init__.py`
+    # does its own fine-grained, per-symbol `require()` check instead -- a
+    # single module-level check here would wrongly demand both extras for
+    # either client.
 }
 
 
@@ -150,9 +190,12 @@ def __getattr__(name: str):
         raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
     from importlib import import_module
 
-    from agentbyte._optional import require
+    extra_spec = _PROVIDER_EXTRAS.get(module_name)
+    if extra_spec is not None:
+        from agentbyte._optional import require
 
-    require("openai", extra="openai" if module_name == ".openai" else "azureopenai", feature=name)
+        required_module, extra = extra_spec
+        require(required_module, extra=extra, feature=name)
     value = getattr(import_module(module_name, __name__), name)
     globals()[name] = value
     return value

agentbyte-0.52.0/src/agentbyte/llm/anthropic/__init__.py

@@ -0,0 +1,10 @@
+"""Anthropic provider package."""
+
+from .chat import AnthropicChatCompletionClient, AnthropicChatCompletionClientConfig
+from .settings import AnthropicSettings
+
+__all__ = [
+    "AnthropicChatCompletionClient",
+    "AnthropicChatCompletionClientConfig",
+    "AnthropicSettings",
+]