agentevals-cli 0.8.1__tar.gz → 0.8.3__tar.gz

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentevals-cli
-Version: 0.8.1
+Version: 0.8.3
 Summary: Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces
 License-File: LICENSE
 Requires-Python: >=3.11
@@ -425,6 +425,18 @@ Yes. A custom evaluator is any program that reads JSON from stdin and writes a s
 
 Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
+**Can I use agentevals to evaluate Claude Code, Codex, or OpenCode?**
+
+Not today. agentevals scores agent behavior from OpenTelemetry GenAI traces (spans for model calls, tool calls, agent invocations following the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)). The major coding agents do not currently emit telemetry in that shape:
+
+- **Claude Code** ships OTel telemetry as logs, not GenAI spans. A prior proof of concept on a feature branch made it work by stitching hook events into synthetic traces. Reviving that path is on the backlog, not a near-term commitment.
+- **Codex** exposes OTel, but in a different shape we have not yet validated against the GenAI semconv.
+- **OpenCode** did not have OTel support merged the last time we checked.
+
+Retrofitting agentevals to ingest each harness's bespoke telemetry is multiple thousands of lines of glue code per agent, for a use case where the dominant signal is "did the final output feel right," not "did the agent call the right tool with the right arguments in the right order." That kind of vibes evaluation is interesting work for harness and coding-agent vendors themselves, but it is not what agentevals is optimized for.
+
+agentevals is built for the opposite end of the spectrum: smaller, purpose-built, properly instrumented agents (kagent, agentregistry, custom Strands / ADK / LangChain / OpenAI Agents SDK flows) running in cloud native environments, where success is measurable through tool trajectories, response matching, and deterministic pass/fail gates. If that is your use case, we are a good fit. If you are evaluating long-running coding sessions end to end, you probably want a tool built specifically for that shape.
+
 **How does this compare to ADK's evaluations?**
 
 Unlike ADK's eval method, which couples agent execution with evaluation, agentevals only handles scoring: it takes pre-recorded traces and compares them against expected behavior using metrics like tool trajectory matching, response quality, and LLM-based judgments.
@@ -448,3 +460,7 @@ Langfuse is a full observability platform (requires Postgres, ClickHouse, Redis,
 **How does this compare to Opik?**
 
 Opik's primary evaluation path re-runs your application code against a dataset, incurring additional LLM costs per eval run. It also supports online evaluation rules that auto-score production traces. While Opik supports OpenTelemetry ingestion alongside its own SDK, its evaluation workflow still centers on re-execution against datasets. agentevals evaluates pre-recorded OTel traces from any framework without re-execution, and runs entirely locally with no cloud dependency.
+
+## Acknowledgements
+
+agentevals is built on top of [Google's Agent Development Kit](https://github.com/google/adk-python). ADK provides the evaluator protocol and the canonical eval data model (`Invocation`, `EvalSet`, `Evaluator`, prebuilt metrics) that this project extends. `google-adk` is licensed under [Apache 2.0](https://github.com/google/adk-python/blob/main/LICENSE), the same license as agentevals. Thanks to the ADK team and contributors.

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/README.md

@@ -397,6 +397,18 @@ Yes. A custom evaluator is any program that reads JSON from stdin and writes a s
 
 Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
+**Can I use agentevals to evaluate Claude Code, Codex, or OpenCode?**
+
+Not today. agentevals scores agent behavior from OpenTelemetry GenAI traces (spans for model calls, tool calls, agent invocations following the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)). The major coding agents do not currently emit telemetry in that shape:
+
+- **Claude Code** ships OTel telemetry as logs, not GenAI spans. A prior proof of concept on a feature branch made it work by stitching hook events into synthetic traces. Reviving that path is on the backlog, not a near-term commitment.
+- **Codex** exposes OTel, but in a different shape we have not yet validated against the GenAI semconv.
+- **OpenCode** did not have OTel support merged the last time we checked.
+
+Retrofitting agentevals to ingest each harness's bespoke telemetry is multiple thousands of lines of glue code per agent, for a use case where the dominant signal is "did the final output feel right," not "did the agent call the right tool with the right arguments in the right order." That kind of vibes evaluation is interesting work for harness and coding-agent vendors themselves, but it is not what agentevals is optimized for.
+
+agentevals is built for the opposite end of the spectrum: smaller, purpose-built, properly instrumented agents (kagent, agentregistry, custom Strands / ADK / LangChain / OpenAI Agents SDK flows) running in cloud native environments, where success is measurable through tool trajectories, response matching, and deterministic pass/fail gates. If that is your use case, we are a good fit. If you are evaluating long-running coding sessions end to end, you probably want a tool built specifically for that shape.
+
 **How does this compare to ADK's evaluations?**
 
 Unlike ADK's eval method, which couples agent execution with evaluation, agentevals only handles scoring: it takes pre-recorded traces and compares them against expected behavior using metrics like tool trajectory matching, response quality, and LLM-based judgments.
@@ -420,3 +432,7 @@ Langfuse is a full observability platform (requires Postgres, ClickHouse, Redis,
 **How does this compare to Opik?**
 
 Opik's primary evaluation path re-runs your application code against a dataset, incurring additional LLM costs per eval run. It also supports online evaluation rules that auto-score production traces. While Opik supports OpenTelemetry ingestion alongside its own SDK, its evaluation workflow still centers on re-execution against datasets. agentevals evaluates pre-recorded OTel traces from any framework without re-execution, and runs entirely locally with no cloud dependency.
+
+## Acknowledgements
+
+agentevals is built on top of [Google's Agent Development Kit](https://github.com/google/adk-python). ADK provides the evaluator protocol and the canonical eval data model (`Invocation`, `EvalSet`, `Evaluator`, prebuilt metrics) that this project extends. `google-adk` is licensed under [Apache 2.0](https://github.com/google/adk-python/blob/main/LICENSE), the same license as agentevals. Thanks to the ADK team and contributors.

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/charts/agentevals/templates/deployment.yaml

@@ -29,8 +29,9 @@ spec:
       securityContext:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
       serviceAccountName: {{ include "agentevals.serviceAccountName" . }}
-      {{- if .Values.ephemeralVolume.enabled }}
+      {{- if or .Values.ephemeralVolume.enabled .Values.extraVolumes }}
       volumes:
+        {{- if .Values.ephemeralVolume.enabled }}
         - name: agentevals-tmp
           {{- if or .Values.ephemeralVolume.sizeLimit (eq .Values.ephemeralVolume.medium "Memory") }}
           emptyDir:
@@ -43,6 +44,10 @@ spec:
           {{- else }}
           emptyDir: {}
           {{- end }}
+        {{- end }}
+        {{- with .Values.extraVolumes }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
       {{- end }}
       containers:
         - name: agentevals
@@ -70,6 +75,8 @@ spec:
               value: "postgres"
             - name: AGENTEVALS_DATABASE_SCHEMA
               value: {{ .Values.database.postgres.schema | quote }}
+            - name: AGENTEVALS_AUTO_MIGRATE
+              value: {{ .Values.database.postgres.autoMigrate | quote }}
             {{- if .Values.database.postgres.urlFile }}
             - name: AGENTEVALS_DATABASE_URL_FILE
               value: {{ .Values.database.postgres.urlFile | quote }}
@@ -135,10 +142,15 @@ spec:
               port: http
             initialDelaySeconds: 15
             periodSeconds: 20
-          {{- if .Values.ephemeralVolume.enabled }}
+          {{- if or .Values.ephemeralVolume.enabled .Values.extraVolumeMounts }}
           volumeMounts:
+            {{- if .Values.ephemeralVolume.enabled }}
             - name: agentevals-tmp
               mountPath: /tmp
+            {{- end }}
+            {{- with .Values.extraVolumeMounts }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
           {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/charts/agentevals/values.yaml

@@ -159,6 +159,16 @@ env: []
 # -- Extra envFrom sources (ConfigMapRef, SecretRef)
 envFrom: []
 
+# -- Extra volumes appended to the pod spec. Use this to mount additional
+# config files or secrets (e.g. result-sink credentials) into the pod.
+extraVolumes: []
+
+# -- Extra volumeMounts appended to the main container. Pair with
+# extraVolumes by name. securityContext.readOnlyRootFilesystem is true by
+# default; that only makes the root filesystem read-only, mounted paths
+# themselves are unaffected, so a writable extraVolumes entry works fine.
+extraVolumeMounts: []
+
 # ==============================================================================
 # STORAGE  (preview feature)
 #
@@ -195,6 +205,12 @@ database:
     urlFile: ""
     # -- Postgres schema to use for agentevals tables.
     schema: agentevals
+    # -- Apply pending database migrations during server startup before the
+    # HTTP listener opens. The Postgres advisory lock serialises concurrent
+    # replica starts so this is safe with replicaCount > 1. When set to
+    # false the server refuses to start if the schema is behind or dirty;
+    # run "agentevals migrate up" manually in that case.
+    autoMigrate: true
     # -- Bundled Postgres instance for development and evaluation only.
     # Not suitable for production. Deployed when enabled is true and url /
     # urlFile are not set.

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/examples/README.md

@@ -119,6 +119,12 @@ The zero-code and SDK examples implement the same toy agent (dice rolling + prim
 |---------|-------------|
 | [kubernetes/](./kubernetes/) | Deploy agentevals with kagent on Kubernetes using native OTLP gRPC ingestion (or optionally an OTel Collector). Includes a walkthrough for comparing two kagent agents (different models) and evaluating them with tool trajectory and response match scores. |
 
+## Custom result sinks
+
+Plugins can deliver run results (partial metrics, final summary, errors) to arbitrary backends alongside the database. Install a package that declares `[project.entry-points."agentevals.sinks"]`, restart agentevals, then reference the plugin’s `kind` in `spec.sinks` on `POST /api/runs`.
+
+See [custom_sink/README.md](./custom_sink/README.md) for a minimal setuptools plugin and configuration examples.
+
 ## Advanced: GenAI Semantic Convention Patterns
 
 > [!TIP]

agentevals_cli-0.8.3/examples/custom_sink/README.md

@@ -0,0 +1,80 @@
+# Custom result sink plugin
+
+This folder is a tiny installable Python package that registers a result **sink** with agentevals via setuptools **entry points**. The worker fans out partial/final/error events to every configured sink in addition to the database.
+
+## What gets implemented
+
+- **`DemoNdjsonSink`** — subclasses `ResultSink` from `agentevals.run.sinks` and appends one JSON object per line to `path` from the run spec (same pattern as the built-in `file` sink, with a `"demo": true` marker on each line).
+- **`create_demo_sink(spec)`** — factory callable; must accept the full sink dict from the run spec and return a `ResultSink` (see return type in code).
+
+The entry point **name** (`demo_ndjson` in `pyproject.toml`) is the **`kind`** string clients put under `spec.sinks`.
+
+## Install (local dev)
+
+From the agentevals repo root, install the framework first, then this example:
+
+```bash
+uv pip install -e .
+uv pip install -e examples/custom_sink
+```
+
+Restart the agentevals process so `importlib.metadata` picks up the new distribution.
+
+PyPI-style usage is the same: depend on `agentevals-example-custom-sink` next to `agentevals-cli`, install both into the server environment, restart.
+
+## Configure runs
+
+Async runs are submitted with **`POST /api/runs`**. Put your sink in **`spec.sinks`** (requires Postgres storage — see main docs).
+
+Example body (use **absolute** `path` on the host where the agentevals process runs when possible). **`path` must be a file path** (e.g. `/tmp/demo.ndjson`). If `path` is an **existing directory** (including `"."` for the process working directory), output goes to `<path>/agentevals-demo-sink.ndjson`, or `<path>/<filename>` if you add an optional `"filename"` field next to `path` in the sink dict.
+
+The `inline` object must contain real trace data (Jaeger JSON or OTLP), not an empty object.
+
+```json
+{
+  "spec": {
+    "approach": "trace_replay",
+    "target": {
+      "kind": "inline",
+      "traceFormat": "jaeger-json",
+      "inline": {
+        "data": [
+          {
+            "traceID": "61646461646164646164616461646164",
+            "spans": [
+              {
+                "traceID": "61646461646164646164616461646164",
+                "spanID": "6164616461646164",
+                "operationName": "demo-op",
+                "startTime": 1000000,
+                "duration": 100000,
+                "tags": [],
+                "logs": [],
+                "references": [],
+                "processID": "p1"
+              }
+            ],
+            "processes": { "p1": { "serviceName": "demo" } }
+          }
+        ]
+      }
+    },
+    "sinks": [{ "kind": "demo_ndjson", "path": "/tmp/agentevals-demo.ndjson" }]
+  }
+}
+```
+
+You can list several sinks; they run in parallel. Built-in kinds are `stdout`, `file`, and `http_webhook`.
+
+## Publishing your own sink
+
+1. Implement `ResultSink` from `agentevals.run.sinks` (subclass the protocol, or provide the three async methods).
+2. Expose a factory `def create_*(spec: dict) -> ResultSink`.
+3. Add the following to your `pyproject.toml`:
+
+```toml
+[project.entry-points."agentevals.sinks"]
+your_kind = "your_package.module:your_factory"
+```
+
+4. Install the package into the **same environment** as `agentevals serve`, restart, and reference `"kind": "your_kind"` in `spec.sinks`.

agentevals_cli-0.8.3/examples/custom_sink/agentevals_example_custom_sink/__init__.py

@@ -0,0 +1 @@
+"""Example result sink plugin for agentevals (see README)."""

agentevals_cli-0.8.3/examples/custom_sink/agentevals_example_custom_sink/sink.py

@@ -0,0 +1,71 @@
+"""Minimal NDJSON sink registered via setuptools entry points."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+from typing import Any
+from uuid import UUID
+
+from agentevals.run.sinks import ResultSink
+from agentevals.storage.models import Result
+
+
+def _result_payload(r: Result) -> dict:
+    return r.model_dump(mode="json", by_alias=True)
+
+
+_DEFAULT_FILENAME = "agentevals-demo-sink.ndjson"
+
+
+def _resolve_output_file(spec: dict[str, Any]) -> Path:
+    """If ``path`` is an existing directory (including ``.``), write NDJSON inside it."""
+    p = Path(spec["path"]).expanduser()
+    if p.exists() and p.is_dir():
+        name = spec.get("filename") or _DEFAULT_FILENAME
+        return p / name
+    return p
+
+
+class DemoNdjsonSink(ResultSink):
+    """Concrete :class:`~agentevals.run.sinks.ResultSink`; append-only JSON lines with a ``demo`` marker."""
+
+    def __init__(self, path: Path) -> None:
+        self._path = path
+        self._lock = asyncio.Lock()
+
+    async def _write(self, payload: dict) -> None:
+        async with self._lock:
+            self._path.parent.mkdir(parents=True, exist_ok=True)
+            with self._path.open("a") as f:  # noqa: ASYNC230
+                f.write(json.dumps(payload) + "\n")
+
+    async def emit_partial(self, run_id: UUID, results: list[Result], attempt: int) -> None:
+        for r in results:
+            await self._write(
+                {
+                    "phase": "partial",
+                    "run_id": str(run_id),
+                    "attempt": attempt,
+                    "demo": True,
+                    "result": _result_payload(r),
+                }
+            )
+
+    async def emit_final(self, run_id: UUID, summary: dict, attempt: int) -> None:
+        await self._write(
+            {"phase": "final", "run_id": str(run_id), "attempt": attempt, "demo": True, "summary": summary}
+        )
+
+    async def emit_error(self, run_id: UUID, error: str, attempt: int) -> None:
+        await self._write({"phase": "error", "run_id": str(run_id), "attempt": attempt, "demo": True, "error": error})
+
+
+def create_demo_sink(spec: dict[str, Any]) -> ResultSink:
+    """Entry-point factory: returns a :class:`ResultSink`; ``kind`` must be ``demo_ndjson`` (see pyproject).
+
+    ``path`` should normally be a **file** path. If it points at an existing directory (e.g. ``.`` or ``/tmp``),
+    lines are appended to ``<path>/agentevals-demo-sink.ndjson``, or ``<path>/<filename>`` if ``filename`` is set.
+    """
+    return DemoNdjsonSink(_resolve_output_file(spec))

agentevals_cli-0.8.3/examples/custom_sink/pyproject.toml

@@ -0,0 +1,19 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "agentevals-example-custom-sink"
+version = "0.1.0"
+description = "Example setuptools plugin that registers an agentevals result sink"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "agentevals-cli>=0.7.0",
+]
+
+[project.entry-points."agentevals.sinks"]
+demo_ndjson = "agentevals_example_custom_sink.sink:create_demo_sink"
+
+[tool.hatch.build.targets.wheel]
+packages = ["agentevals_example_custom_sink"]

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentevals-cli"
-version = "0.8.1"
+version = "0.8.3"
 description = "Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces"
 readme = "README.md"
 requires-python = ">=3.11"

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/src/agentevals/api/app.py

@@ -17,9 +17,10 @@ from fastapi.responses import StreamingResponse
 from agentevals import __version__
 
 from ..run.service import RunService
+from ..run.sinks import log_registered_sinks
 from ..run.worker import AsyncRunWorker
 from ..storage import StorageSettings, build_repos
-from ..storage.postgres.migrator import Migrator
+from ..storage.postgres.migrator import Migrator, discover_migrations
 from ..utils.log_buffer import log_buffer
 from .debug_routes import debug_router
 from .routes import router
@@ -30,6 +31,22 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
+_TRUE_VALUES = {"true", "1", "yes", "on"}
+_FALSE_VALUES = {"false", "0", "no", "off"}
+
+
+def _env_bool(name: str, *, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None or raw == "":
+        return default
+    val = raw.strip().lower()
+    if val in _TRUE_VALUES:
+        return True
+    if val in _FALSE_VALUES:
+        return False
+    raise ValueError(f"{name} must be one of true/false/1/0/yes/no/on/off (got: {raw!r})")
+
+
 try:
     from dotenv import load_dotenv
 
@@ -67,13 +84,34 @@ def _build_lifespan():
             logger.error("Storage configuration invalid; /api/runs will not be available: %s", exc)
 
         if storage_settings is not None and storage_settings.backend == "postgres":
-            logger.info("Applying any pending migrations to schema '%s'", storage_settings.schema_name)
             migrator = Migrator(
                 dsn=storage_settings.database_url or "",
                 schema=storage_settings.schema_name,
                 lock_timeout_s=storage_settings.migrate_lock_timeout_s,
             )
-            await migrator.up()
+            if _env_bool("AGENTEVALS_AUTO_MIGRATE", default=True):
+                logger.info("Applying any pending migrations to schema '%s'", storage_settings.schema_name)
+                await migrator.up()
+            else:
+                logger.info(
+                    "AGENTEVALS_AUTO_MIGRATE is disabled; verifying schema '%s' is up to date",
+                    storage_settings.schema_name,
+                )
+                status = await migrator.status()
+                if status.dirty:
+                    raise RuntimeError(
+                        f"schema_migrations is dirty at version {status.version}. "
+                        "Resolve manually and run 'agentevals migrate force <version>', "
+                        "or set AGENTEVALS_AUTO_MIGRATE=true to retry on startup."
+                    )
+                current = status.version
+                pending = [m.version for m in discover_migrations() if current is None or m.version > current]
+                if pending:
+                    raise RuntimeError(
+                        f"Database schema is behind: pending migrations {pending}. "
+                        "Run 'agentevals migrate up' to apply them, "
+                        "or set AGENTEVALS_AUTO_MIGRATE=true to apply on startup."
+                    )
 
             repos = await build_repos(storage_settings)
             app.state.storage_settings = storage_settings
@@ -83,6 +121,7 @@ def _build_lifespan():
             worker = AsyncRunWorker(runs=repos.runs, results=repos.results, settings=storage_settings)
             await worker.start()
             app.state.run_worker = worker
+            log_registered_sinks()
 
         yield
 

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/src/agentevals/run/__init__.py

@@ -2,7 +2,7 @@
 
 Contents:
 - :mod:`fetcher` resolves a run spec's ``target`` into a list of traces.
-- :mod:`sinks` fan-out result delivery (stdout, file, http_webhook).
+- :mod:`sinks` fan-out result delivery (built-ins plus setuptools plugins / :func:`~agentevals.run.sinks.register_sink_factory`).
 - :mod:`service` is the synchronous control surface used by HTTP handlers.
 - :mod:`worker` is the in-process loop that claims runs and drives the
   existing :func:`agentevals.runner.run_evaluation_from_traces` pipeline.

{agentevals_cli-0.8.1 → agentevals_cli-0.8.3}/src/agentevals/run/sinks.py

@@ -3,6 +3,14 @@
 The :class:`agentevals.storage.repos.ResultRepository` is always written;
 sinks are an additional delivery channel. Sink failures are logged with
 ``run_id`` / ``result_id`` but do not fail the run.
+
+**Plugins:** third-party packages declare setuptools entry points in group
+``agentevals.sinks`` (entry **name** = ``kind`` string; **value** = ``module:factory``
+callable ``factory(spec: dict) -> ResultSink``). Built-in kinds
+(``stdout``, ``file``, ``http_webhook``) are not overridden by entry points;
+hosts may replace any kind via :func:`register_sink_factory` (highest precedence).
+
+Tests may call :func:`clear_sink_plugin_registry` to drop programmatic registrations.
 """
 
 from __future__ import annotations
@@ -12,8 +20,10 @@ import json
 import logging
 import os
 import sys
+from collections.abc import Callable
+from importlib.metadata import entry_points
 from pathlib import Path
-from typing import Any, Protocol
+from typing import Any, Protocol, cast
 from uuid import UUID
 
 import httpx
@@ -22,6 +32,8 @@ from ..storage.models import Result
 
 logger = logging.getLogger(__name__)
 
+SINK_ENTRY_POINT_GROUP = "agentevals.sinks"
+
 
 class ResultSink(Protocol):
     async def emit_partial(self, run_id: UUID, results: list[Result], attempt: int) -> None: ...
@@ -29,6 +41,11 @@ class ResultSink(Protocol):
     async def emit_error(self, run_id: UUID, error: str, attempt: int) -> None: ...
 
 
+SinkFactory = Callable[[dict[str, Any]], ResultSink]
+
+_PLUGIN_FACTORIES: dict[str, SinkFactory] = {}
+
+
 def _result_payload(r: Result) -> dict:
     return r.model_dump(mode="json", by_alias=True)
 
@@ -187,33 +204,18 @@ class SinkFanout:
             logger.exception("sink delivery failed in phase=%s", phase)
 
 
-def build_sinks(specs: list[dict]) -> SinkFanout:
-    """Construct a fan-out from the run spec's ``sinks`` array.
+def register_sink_factory(kind: str, factory: SinkFactory) -> None:
+    """Register or replace the factory for ``kind`` (overrides built-ins and entry points).
 
-    Each spec is a dict with ``kind`` plus kind-specific args. Unknown kinds
-    are skipped with a warning so a future kind added by a host doesn't
-    break older agentevals replicas mid-rollout.
+    Call during process startup before run workers consume specs. The factory receives
+    the full sink spec dict (including ``kind``) and returns a :class:`ResultSink`.
     """
-    sinks: list[ResultSink] = []
-    for spec in specs:
-        kind = spec.get("kind")
-        if kind == "stdout":
-            sinks.append(StdoutSink())
-        elif kind == "file":
-            sinks.append(FileSink(spec["path"]))
-        elif kind == "http_webhook":
-            sinks.append(
-                HttpWebhookSink(
-                    url=spec["url"],
-                    headers=spec.get("headers"),
-                    headers_from_env=spec.get("headers_from_env") or _extract_env_headers(spec.get("auth")),
-                    timeout_s=float(spec.get("timeout_s", 10.0)),
-                    max_attempts=int(spec.get("max_attempts", 5)),
-                )
-            )
-        else:
-            logger.warning("unknown sink kind '%s'; skipping", kind)
-    return SinkFanout(sinks)
+    _PLUGIN_FACTORIES[kind] = factory
+
+
+def clear_sink_plugin_registry() -> None:
+    """Drop all registrations from :func:`register_sink_factory` (for tests)."""
+    _PLUGIN_FACTORIES.clear()
 
 
 def _extract_env_headers(auth: Any) -> dict[str, str]:
@@ -228,3 +230,85 @@ def _extract_env_headers(auth: Any) -> dict[str, str]:
         if isinstance(value, dict) and "from_env" in value:
             result[header_name] = value["from_env"]
     return result
+
+
+def _http_webhook_from_spec(spec: dict[str, Any]) -> HttpWebhookSink:
+    return HttpWebhookSink(
+        url=spec["url"],
+        headers=spec.get("headers"),
+        headers_from_env=spec.get("headers_from_env") or _extract_env_headers(spec.get("auth")),
+        timeout_s=float(spec.get("timeout_s", 10.0)),
+        max_attempts=int(spec.get("max_attempts", 5)),
+    )
+
+
+def _builtin_factories() -> dict[str, SinkFactory]:
+    return {
+        "stdout": lambda _spec: StdoutSink(),
+        "file": lambda spec: FileSink(spec["path"]),
+        "http_webhook": _http_webhook_from_spec,
+    }
+
+
+def _merge_sink_factories() -> dict[str, SinkFactory]:
+    """Built-ins, then entry points (no built-in shadowing), then programmatic overrides."""
+    merged: dict[str, SinkFactory] = dict(_builtin_factories())
+    eps = entry_points(group=SINK_ENTRY_POINT_GROUP)
+    for ep in eps:
+        if ep.name in merged:
+            logger.debug("skipping sink entry point %r; built-in kind takes precedence", ep.name)
+            continue
+        try:
+            loaded = ep.load()
+            if not callable(loaded):
+                logger.warning("sink entry point %r is not callable; skipping", ep.name)
+                continue
+            merged[ep.name] = cast(SinkFactory, loaded)
+        except Exception:
+            logger.exception("failed to load sink entry point %r", ep.name)
+    merged.update(_PLUGIN_FACTORIES)
+    return merged
+
+
+def registered_sink_kinds() -> tuple[str, ...]:
+    """Sorted sink ``kind`` strings that would resolve if :func:`build_sinks` ran now.
+
+    Includes built-ins, successfully loaded setuptools entry points for group
+    :data:`SINK_ENTRY_POINT_GROUP`, and registrations from
+    :func:`register_sink_factory`. The tuple reflects current process state and
+    can change if the programmatic registry is mutated after startup.
+    """
+    return tuple(sorted(_merge_sink_factories().keys()))
+
+
+def log_registered_sinks() -> None:
+    """Emit one INFO line listing available sink kinds (for operator diagnostics)."""
+    kinds = registered_sink_kinds()
+    logger.info("Result sinks available (%d kinds): %s", len(kinds), ", ".join(kinds))
+
+
+def build_sinks(specs: list[dict]) -> SinkFanout:
+    """Construct a fan-out from the run spec's ``sinks`` array.
+
+    Each spec is a dict with ``kind`` plus kind-specific args. Unknown kinds
+    are skipped with a warning so a future kind added by a host doesn't
+    break older agentevals replicas mid-rollout.
+
+    Factory lookup starts from built-ins, adds setuptools entry points (group
+    ``agentevals.sinks``) for ``kind`` names not already built-in, then applies
+    :func:`register_sink_factory` registrations, which override any prior factory
+    for the same ``kind``. See :func:`_merge_sink_factories`.
+    """
+    factories = _merge_sink_factories()
+    sinks: list[ResultSink] = []
+    for spec in specs:
+        kind = spec.get("kind")
+        factory = factories.get(kind) if kind is not None else None
+        if factory is None:
+            logger.warning("unknown sink kind '%s'; skipping", kind)
+            continue
+        try:
+            sinks.append(factory(spec))
+        except Exception:
+            logger.exception("sink factory failed for kind=%s", kind)
+    return SinkFanout(sinks)