agentevals-cli 0.8.2__tar.gz → 0.8.4__tar.gz

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentevals-cli
-Version: 0.8.2
+Version: 0.8.4
 Summary: Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces
 License-File: LICENSE
 Requires-Python: >=3.11
@@ -425,6 +425,18 @@ Yes. A custom evaluator is any program that reads JSON from stdin and writes a s
 
 Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
+**Can I use agentevals to evaluate Claude Code, Codex, or OpenCode?**
+
+Not today. agentevals scores agent behavior from OpenTelemetry GenAI traces (spans for model calls, tool calls, agent invocations following the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)). The major coding agents do not currently emit telemetry in that shape:
+
+- **Claude Code** ships OTel telemetry as logs, not GenAI spans. A prior proof of concept on a feature branch made it work by stitching hook events into synthetic traces. Reviving that path is on the backlog, not a near-term commitment.
+- **Codex** exposes OTel, but in a different shape we have not yet validated against the GenAI semconv.
+- **OpenCode** did not have OTel support merged the last time we checked.
+
+Retrofitting agentevals to ingest each harness's bespoke telemetry is multiple thousands of lines of glue code per agent, for a use case where the dominant signal is "did the final output feel right," not "did the agent call the right tool with the right arguments in the right order." That kind of vibes evaluation is interesting work for harness and coding-agent vendors themselves, but it is not what agentevals is optimized for.
+
+agentevals is built for the opposite end of the spectrum: smaller, purpose-built, properly instrumented agents (kagent, agentregistry, custom Strands / ADK / LangChain / OpenAI Agents SDK flows) running in cloud native environments, where success is measurable through tool trajectories, response matching, and deterministic pass/fail gates. If that is your use case, we are a good fit. If you are evaluating long-running coding sessions end to end, you probably want a tool built specifically for that shape.
+
 **How does this compare to ADK's evaluations?**
 
 Unlike ADK's eval method, which couples agent execution with evaluation, agentevals only handles scoring: it takes pre-recorded traces and compares them against expected behavior using metrics like tool trajectory matching, response quality, and LLM-based judgments.
@@ -448,3 +460,7 @@ Langfuse is a full observability platform (requires Postgres, ClickHouse, Redis,
 **How does this compare to Opik?**
 
 Opik's primary evaluation path re-runs your application code against a dataset, incurring additional LLM costs per eval run. It also supports online evaluation rules that auto-score production traces. While Opik supports OpenTelemetry ingestion alongside its own SDK, its evaluation workflow still centers on re-execution against datasets. agentevals evaluates pre-recorded OTel traces from any framework without re-execution, and runs entirely locally with no cloud dependency.
+
+## Acknowledgements
+
+agentevals is built on top of [Google's Agent Development Kit](https://github.com/google/adk-python). ADK provides the evaluator protocol and the canonical eval data model (`Invocation`, `EvalSet`, `Evaluator`, prebuilt metrics) that this project extends. `google-adk` is licensed under [Apache 2.0](https://github.com/google/adk-python/blob/main/LICENSE), the same license as agentevals. Thanks to the ADK team and contributors.

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/README.md

@@ -397,6 +397,18 @@ Yes. A custom evaluator is any program that reads JSON from stdin and writes a s
 
 Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
+**Can I use agentevals to evaluate Claude Code, Codex, or OpenCode?**
+
+Not today. agentevals scores agent behavior from OpenTelemetry GenAI traces (spans for model calls, tool calls, agent invocations following the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)). The major coding agents do not currently emit telemetry in that shape:
+
+- **Claude Code** ships OTel telemetry as logs, not GenAI spans. A prior proof of concept on a feature branch made it work by stitching hook events into synthetic traces. Reviving that path is on the backlog, not a near-term commitment.
+- **Codex** exposes OTel, but in a different shape we have not yet validated against the GenAI semconv.
+- **OpenCode** did not have OTel support merged the last time we checked.
+
+Retrofitting agentevals to ingest each harness's bespoke telemetry is multiple thousands of lines of glue code per agent, for a use case where the dominant signal is "did the final output feel right," not "did the agent call the right tool with the right arguments in the right order." That kind of vibes evaluation is interesting work for harness and coding-agent vendors themselves, but it is not what agentevals is optimized for.
+
+agentevals is built for the opposite end of the spectrum: smaller, purpose-built, properly instrumented agents (kagent, agentregistry, custom Strands / ADK / LangChain / OpenAI Agents SDK flows) running in cloud native environments, where success is measurable through tool trajectories, response matching, and deterministic pass/fail gates. If that is your use case, we are a good fit. If you are evaluating long-running coding sessions end to end, you probably want a tool built specifically for that shape.
+
 **How does this compare to ADK's evaluations?**
 
 Unlike ADK's eval method, which couples agent execution with evaluation, agentevals only handles scoring: it takes pre-recorded traces and compares them against expected behavior using metrics like tool trajectory matching, response quality, and LLM-based judgments.
@@ -420,3 +432,7 @@ Langfuse is a full observability platform (requires Postgres, ClickHouse, Redis,
 **How does this compare to Opik?**
 
 Opik's primary evaluation path re-runs your application code against a dataset, incurring additional LLM costs per eval run. It also supports online evaluation rules that auto-score production traces. While Opik supports OpenTelemetry ingestion alongside its own SDK, its evaluation workflow still centers on re-execution against datasets. agentevals evaluates pre-recorded OTel traces from any framework without re-execution, and runs entirely locally with no cloud dependency.
+
+## Acknowledgements
+
+agentevals is built on top of [Google's Agent Development Kit](https://github.com/google/adk-python). ADK provides the evaluator protocol and the canonical eval data model (`Invocation`, `EvalSet`, `Evaluator`, prebuilt metrics) that this project extends. `google-adk` is licensed under [Apache 2.0](https://github.com/google/adk-python/blob/main/LICENSE), the same license as agentevals. Thanks to the ADK team and contributors.

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/charts/agentevals/templates/deployment.yaml

@@ -29,8 +29,9 @@ spec:
       securityContext:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
       serviceAccountName: {{ include "agentevals.serviceAccountName" . }}
-      {{- if .Values.ephemeralVolume.enabled }}
+      {{- if or .Values.ephemeralVolume.enabled .Values.extraVolumes }}
       volumes:
+        {{- if .Values.ephemeralVolume.enabled }}
         - name: agentevals-tmp
           {{- if or .Values.ephemeralVolume.sizeLimit (eq .Values.ephemeralVolume.medium "Memory") }}
           emptyDir:
@@ -43,6 +44,10 @@ spec:
           {{- else }}
           emptyDir: {}
           {{- end }}
+        {{- end }}
+        {{- with .Values.extraVolumes }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
       {{- end }}
       containers:
         - name: agentevals
@@ -70,6 +75,10 @@ spec:
               value: "postgres"
             - name: AGENTEVALS_DATABASE_SCHEMA
               value: {{ .Values.database.postgres.schema | quote }}
+            - name: AGENTEVALS_AUTO_MIGRATE
+              value: {{ .Values.database.postgres.autoMigrate | quote }}
+            - name: AGENTEVALS_DB_CONNECT_TIMEOUT_S
+              value: {{ .Values.database.postgres.connectTimeoutSeconds | quote }}
             {{- if .Values.database.postgres.urlFile }}
             - name: AGENTEVALS_DATABASE_URL_FILE
               value: {{ .Values.database.postgres.urlFile | quote }}
@@ -135,10 +144,15 @@ spec:
               port: http
             initialDelaySeconds: 15
             periodSeconds: 20
-          {{- if .Values.ephemeralVolume.enabled }}
+          {{- if or .Values.ephemeralVolume.enabled .Values.extraVolumeMounts }}
           volumeMounts:
+            {{- if .Values.ephemeralVolume.enabled }}
             - name: agentevals-tmp
               mountPath: /tmp
+            {{- end }}
+            {{- with .Values.extraVolumeMounts }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
           {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/charts/agentevals/values.yaml

@@ -159,6 +159,16 @@ env: []
 # -- Extra envFrom sources (ConfigMapRef, SecretRef)
 envFrom: []
 
+# -- Extra volumes appended to the pod spec. Use this to mount additional
+# config files or secrets (e.g. result-sink credentials) into the pod.
+extraVolumes: []
+
+# -- Extra volumeMounts appended to the main container. Pair with
+# extraVolumes by name. securityContext.readOnlyRootFilesystem is true by
+# default; that only makes the root filesystem read-only, mounted paths
+# themselves are unaffected, so a writable extraVolumes entry works fine.
+extraVolumeMounts: []
+
 # ==============================================================================
 # STORAGE  (preview feature)
 #
@@ -195,6 +205,18 @@ database:
     urlFile: ""
     # -- Postgres schema to use for agentevals tables.
     schema: agentevals
+    # -- Apply pending database migrations during server startup before the
+    # HTTP listener opens. The Postgres advisory lock serialises concurrent
+    # replica starts so this is safe with replicaCount > 1. When set to
+    # false the server refuses to start if the schema is behind or dirty;
+    # run "agentevals migrate up" manually in that case.
+    autoMigrate: true
+    # -- Seconds the startup will spend retrying the initial Postgres
+    # connection before the pod aborts. Default 600s matches the chart's
+    # hard-coded startupProbe budget (failureThreshold 60 x periodSeconds
+    # 10). Going above 600s requires overriding the probe in your own
+    # downstream template.
+    connectTimeoutSeconds: 600
     # -- Bundled Postgres instance for development and evaluation only.
     # Not suitable for production. Deployed when enabled is true and url /
     # urlFile are not set.

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentevals-cli"
-version = "0.8.2"
+version = "0.8.4"
 description = "Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces"
 readme = "README.md"
 requires-python = ">=3.11"

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/src/agentevals/api/app.py

@@ -20,7 +20,7 @@ from ..run.service import RunService
 from ..run.sinks import log_registered_sinks
 from ..run.worker import AsyncRunWorker
 from ..storage import StorageSettings, build_repos
-from ..storage.postgres.migrator import Migrator
+from ..storage.postgres.migrator import Migrator, discover_migrations
 from ..utils.log_buffer import log_buffer
 from .debug_routes import debug_router
 from .routes import router
@@ -31,6 +31,22 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
+_TRUE_VALUES = {"true", "1", "yes", "on"}
+_FALSE_VALUES = {"false", "0", "no", "off"}
+
+
+def _env_bool(name: str, *, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None or raw == "":
+        return default
+    val = raw.strip().lower()
+    if val in _TRUE_VALUES:
+        return True
+    if val in _FALSE_VALUES:
+        return False
+    raise ValueError(f"{name} must be one of true/false/1/0/yes/no/on/off (got: {raw!r})")
+
+
 try:
     from dotenv import load_dotenv
 
@@ -68,13 +84,34 @@ def _build_lifespan():
             logger.error("Storage configuration invalid; /api/runs will not be available: %s", exc)
 
         if storage_settings is not None and storage_settings.backend == "postgres":
-            logger.info("Applying any pending migrations to schema '%s'", storage_settings.schema_name)
             migrator = Migrator(
                 dsn=storage_settings.database_url or "",
                 schema=storage_settings.schema_name,
                 lock_timeout_s=storage_settings.migrate_lock_timeout_s,
             )
-            await migrator.up()
+            if _env_bool("AGENTEVALS_AUTO_MIGRATE", default=True):
+                logger.info("Applying any pending migrations to schema '%s'", storage_settings.schema_name)
+                await migrator.up()
+            else:
+                logger.info(
+                    "AGENTEVALS_AUTO_MIGRATE is disabled; verifying schema '%s' is up to date",
+                    storage_settings.schema_name,
+                )
+                status = await migrator.status()
+                if status.dirty:
+                    raise RuntimeError(
+                        f"schema_migrations is dirty at version {status.version}. "
+                        "Resolve manually and run 'agentevals migrate force <version>', "
+                        "or set AGENTEVALS_AUTO_MIGRATE=true to retry on startup."
+                    )
+                current = status.version
+                pending = [m.version for m in discover_migrations() if current is None or m.version > current]
+                if pending:
+                    raise RuntimeError(
+                        f"Database schema is behind: pending migrations {pending}. "
+                        "Run 'agentevals migrate up' to apply them, "
+                        "or set AGENTEVALS_AUTO_MIGRATE=true to apply on startup."
+                    )
 
             repos = await build_repos(storage_settings)
             app.state.storage_settings = storage_settings

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/src/agentevals/storage/postgres/migrator.py

@@ -13,6 +13,8 @@ from __future__ import annotations
 
 import asyncio
 import logging
+import math
+import os
 import re
 from dataclasses import dataclass
 from importlib.resources import files
@@ -252,23 +254,58 @@ def discover_migrations() -> list[Migration]:
     return _discover_migrations()
 
 
-CONNECT_RETRY_DEADLINE_S = 60.0
-"""Total wall-clock budget for the initial Postgres connection. Bundled PG
-in Kubernetes typically takes 5-15s to be ready (PVC bind, initdb, listener
-bind), so the agentevals lifespan can race the database on a fresh deploy.
-Retrying tolerates that gap rather than failing pod startup and relying on
-CrashLoopBackOff timing to eventually line up."""
+CONNECT_RETRY_DEADLINE_S = 600.0
+"""Default total wall-clock budget for the initial Postgres connection.
+Sized to span Kubernetes bring-up of a freshly provisioned database (PVC
+bind, initdb, listener bind, network policy propagation). Override at
+runtime by setting ``AGENTEVALS_DB_CONNECT_TIMEOUT_S`` to a positive
+number of seconds; an invalid value logs a warning and falls back to this
+default."""
+
+
+def connect_deadline_seconds() -> float:
+    """Resolve the connect-retry budget. Reads ``AGENTEVALS_DB_CONNECT_TIMEOUT_S``
+    and falls back to :data:`CONNECT_RETRY_DEADLINE_S` if the env var is
+    unset, empty, non-numeric, non-finite, or non-positive."""
+    raw = os.getenv("AGENTEVALS_DB_CONNECT_TIMEOUT_S")
+    if raw is None or raw == "":
+        return CONNECT_RETRY_DEADLINE_S
+    try:
+        val = float(raw)
+    except ValueError:
+        logger.warning(
+            "Invalid AGENTEVALS_DB_CONNECT_TIMEOUT_S=%r (not a number); using default %.0fs",
+            raw,
+            CONNECT_RETRY_DEADLINE_S,
+        )
+        return CONNECT_RETRY_DEADLINE_S
+    if not math.isfinite(val):
+        logger.warning(
+            "Invalid AGENTEVALS_DB_CONNECT_TIMEOUT_S=%r (must be finite); using default %.0fs",
+            raw,
+            CONNECT_RETRY_DEADLINE_S,
+        )
+        return CONNECT_RETRY_DEADLINE_S
+    if val <= 0:
+        logger.warning(
+            "Invalid AGENTEVALS_DB_CONNECT_TIMEOUT_S=%r (must be positive); using default %.0fs",
+            raw,
+            CONNECT_RETRY_DEADLINE_S,
+        )
+        return CONNECT_RETRY_DEADLINE_S
+    return val
 
 
 async def connect_with_retry(dsn: str, asyncpg_module) -> "asyncpg.Connection":
     """Open a single asyncpg connection, retrying on connection-refused or
-    server-not-ready errors for up to ``CONNECT_RETRY_DEADLINE_S`` seconds.
+    server-not-ready errors for up to :func:`connect_deadline_seconds`
+    seconds.
 
     Connection-time errors are tolerated; once a connection has been
     established and a query returned, all subsequent failures propagate
     normally.
     """
-    deadline = asyncio.get_event_loop().time() + CONNECT_RETRY_DEADLINE_S
+    deadline = asyncio.get_event_loop().time() + connect_deadline_seconds()
     delay = 0.5
     while True:
         try:

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/src/agentevals/storage/postgres/pool.py

@@ -50,9 +50,9 @@ async def create_pool(settings: StorageSettings) -> "asyncpg.Pool":
         settings.schema_name,
     )
 
-    from .migrator import CONNECT_RETRY_DEADLINE_S
+    from .migrator import connect_deadline_seconds
 
-    deadline = asyncio.get_event_loop().time() + CONNECT_RETRY_DEADLINE_S
+    deadline = asyncio.get_event_loop().time() + connect_deadline_seconds()
     delay = 0.5
     while True:
         try:

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/tests/storage/test_migrator.py

@@ -7,6 +7,7 @@ otherwise those tests skip so the suite stays runnable in pure-Python sandboxes.
 
 from __future__ import annotations
 
+import logging
 import os
 import re
 
@@ -14,10 +15,12 @@ import pytest
 
 from agentevals.storage.postgres.migrator import (
     ADVISORY_LOCK_KEY,
+    CONNECT_RETRY_DEADLINE_S,
     Migration,
     Migrator,
     _apply_schema,
     _discover_migrations,
+    connect_deadline_seconds,
     discover_migrations,
 )
 
@@ -71,6 +74,61 @@ class TestAdvisoryLockKey:
         assert ADVISORY_LOCK_KEY == 7259820376655812345
 
 
+class TestConnectDeadlineSeconds:
+    """``connect_deadline_seconds`` resolves AGENTEVALS_DB_CONNECT_TIMEOUT_S
+    to a float, falling back to CONNECT_RETRY_DEADLINE_S on any input the
+    retry loop cannot consume. Each failure mode logs at WARNING so the
+    cause is diagnosable from pod logs."""
+
+    @pytest.fixture(autouse=True)
+    def _clean_env(self, monkeypatch):
+        monkeypatch.delenv("AGENTEVALS_DB_CONNECT_TIMEOUT_S", raising=False)
+
+    def test_unset_returns_default(self):
+        assert connect_deadline_seconds() == CONNECT_RETRY_DEADLINE_S
+
+    def test_empty_returns_default(self, monkeypatch):
+        monkeypatch.setenv("AGENTEVALS_DB_CONNECT_TIMEOUT_S", "")
+        assert connect_deadline_seconds() == CONNECT_RETRY_DEADLINE_S
+
+    @pytest.mark.parametrize(
+        ("raw", "expected"),
+        [
+            ("42", 42.0),
+            ("120.5", 120.5),
+            ("0.1", 0.1),
+            ("3600", 3600.0),
+        ],
+    )
+    def test_parses_valid_positive_values(self, monkeypatch, raw, expected):
+        monkeypatch.setenv("AGENTEVALS_DB_CONNECT_TIMEOUT_S", raw)
+        assert connect_deadline_seconds() == expected
+
+    @pytest.mark.parametrize(
+        ("raw", "reason_substring"),
+        [
+            ("foo", "not a number"),
+            ("nan", "must be finite"),
+            ("inf", "must be finite"),
+            ("-inf", "must be finite"),
+            ("0", "must be positive"),
+            ("-5", "must be positive"),
+        ],
+    )
+    def test_invalid_values_fall_back_with_warning(self, monkeypatch, caplog, raw, reason_substring):
+        """Bad inputs return the default and log exactly one warning that
+        names the specific validation branch. The cardinality check
+        guards against a refactor that double-logs (e.g. emits both a
+        generic and a specific message)."""
+        monkeypatch.setenv("AGENTEVALS_DB_CONNECT_TIMEOUT_S", raw)
+        with caplog.at_level(logging.WARNING, logger="agentevals.storage.postgres.migrator"):
+            result = connect_deadline_seconds()
+        assert result == CONNECT_RETRY_DEADLINE_S
+        warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"]
+        assert len(warnings) == 1, f"expected one warning, got {warnings}"
+        assert reason_substring in warnings[0]
+
+
 class TestMigrationFilePattern:
     def test_filename_format(self):
         migrations = _discover_migrations()

{agentevals_cli-0.8.2 → agentevals_cli-0.8.4}/uv.lock

@@ -30,7 +30,7 @@ wheels = [
 
 [[package]]
 name = "agentevals-cli"
-version = "0.8.2"
+version = "0.8.4"
 source = { editable = "." }
 dependencies = [
     { name = "click" },