agentevals-cli 0.6.4__tar.gz → 0.7.1__tar.gz

{agentevals_cli-0.6.4 → agentevals_cli-0.7.1}/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.4
 Name: agentevals-cli
-Version: 0.6.4
+Version: 0.7.1
 Summary: Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces
 License-File: LICENSE
 Requires-Python: >=3.11
 Requires-Dist: click>=8.0
 Requires-Dist: fastapi>=0.115.0
-Requires-Dist: google-adk[eval]>=1.25.0
+Requires-Dist: google-adk[eval]>=1.30.0
 Requires-Dist: httpx>=0.27.0
 Requires-Dist: opentelemetry-proto>=1.36.0
 Requires-Dist: python-dotenv>=1.0.0
@@ -59,34 +59,16 @@ agentevals scores performance and inference quality from OpenTelemetry traces. N
 
 ## What is agentevals?
 
-agentevals is a framework-agnostic evaluation solution that scores AI agent behavior directly from [OpenTelemetry](https://opentelemetry.io/) traces. Record your agent's actions once, then evaluate as many times as you want. No re-runs, no guesswork.
+agentevals is a framework-agnostic evaluation solution that scores AI agent behavior directly from [OpenTelemetry](https://opentelemetry.io/) traces. Record your agent's actions once, then evaluate as many times as you want without re-executing or burning extra tokens.
 
 It works with any OTel-instrumented framework (LangChain, Strands, Google ADK, OpenAI Agents SDK, and others), supports Jaeger JSON and native OTLP trace formats, and ships with built-in evaluators, custom evaluator support, and LLM-based judges.
 
-- **CLI** for scripting and CI pipelines
-- **Web UI** for visual inspection and local developer experience
-- **Kubernetes and OTel support** so you can deploy right next to your agents; works natively in your OpenTelemetry pipeline
-- **MCP server** so MCP clients can run evaluations from a conversation
-
-## Why agentevals?
-
-Most evaluation tools require you to **re-execute your agent** for every test, burning tokens, time, and money on duplicate LLM calls. agentevals takes a different approach:
-
 - **No re-execution**: score agents from existing traces without replaying expensive LLM calls
-- **Framework-agnostic**: works with any agent framework that emits OpenTelemetry spans
 - **Golden eval sets**: compare actual behavior against defined expected behaviors for deterministic pass/fail gating
 - **Custom evaluators**: write scoring logic in Python, JavaScript, or any language, or offload scoring to OpenAI Eval API
 - **CI/CD ready**: gate deployments on quality thresholds directly in your pipeline
 - **Local-first**: no cloud dependency required; everything runs on your machine
-
-## How It Works
-
-agentevals follows three simple steps:
-
-1. **Collect traces**: Instrument your agent with OpenTelemetry (or export traces from your tracing backend). Point the OTLP exporter at the agentevals receiver, or load trace files directly.
-2. **Define eval sets**: Create golden evaluation sets that describe expected agent behavior: which tools should be called, in what order, and what the output should look like.
-3. **Run evaluations**: Use the CLI, Web UI, or MCP server to score traces against your eval sets. Get per-metric scores, pass/fail results, and detailed span-level breakdowns.
-
+- **Multiple interfaces**: CLI for scripting and CI, Web UI for visual inspection, MCP server for conversational evaluation, Helm chart for Kubernetes environments
 
 > [!IMPORTANT]
 > This project is under active development. Expect breaking changes.
@@ -95,7 +77,7 @@ agentevals follows three simple steps:
 
 - [Installation](#installation)
 - [Quick Start](#quick-start)
-- [Integration](#integration)
+- [Use-cases and Integrations](#use-cases-and-integrations)
 - [CLI](#cli)
 - [Custom Evaluators](#custom-evaluators)
 - [Web UI](#web-ui)
@@ -194,14 +176,14 @@ agentevals serve
 # opens http://localhost:8001
 ```
 
-You can also point any OTel-instrumented agent directly at the built-in receiver (`OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`). The UI streams tool calls, inputs, and outputs live as your agent runs. For production setups, the same receiver slots into a Kubernetes OTel Collector pipeline as an exporter destination. See [Integration](#integration) and the [Kubernetes example](examples/kubernetes/README.md) for walkthroughs.
+You can also point any OTel-instrumented agent directly at the built-in receiver (`OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`). The UI streams tool calls, inputs, and outputs live as your agent runs. For production setups, the same receiver slots into a Kubernetes OTel Collector pipeline as an exporter destination. See [Use-cases and Integrations](#use-cases-and-integrations) and the [Kubernetes example](examples/kubernetes/README.md) for walkthroughs.
 
 **Next steps:**
 
 - `agentevals evaluator list` to see all built-in and community evaluators
 - [Custom Evaluators](#custom-evaluators) to write your own scoring logic
 
-## Use-cases and integrations
+## Use-cases and Integrations
 
 ### Zero-Code (Recommended)
 
@@ -243,7 +225,7 @@ with app.session(eval_set_id="my-eval"):
 
 Requires `pip install "agentevals-cli[streaming]"`. See [examples/sdk_example/](examples/sdk_example/) for framework-specific patterns.
 
-## CLI for local testing, and CI pipelines
+## CLI
 
 ```bash
 # Multiple traces, JSON output
@@ -306,12 +288,13 @@ A `Dockerfile` is included at the project root. The image bundles the API, web U
 
 ```bash
 docker build -t agentevals .
-docker run -p 8001:8001 -p 4318:4318 agentevals
+docker run -p 8001:8001 -p 4317:4317 -p 4318:4318 agentevals
 ```
 
 | Port | Purpose |
 |------|---------|
 | 8001 | Web UI and REST API |
+| 4317 | OTLP gRPC receiver (traces and logs) |
 | 4318 | OTLP HTTP receiver (traces and logs) |
 | 8080 | MCP (Streamable HTTP) |
 
@@ -389,7 +372,7 @@ See [DEVELOPMENT.md](DEVELOPMENT.md) for build tiers, Makefile targets, and Nix
 
 **Do I need a database or any infrastructure to run agentevals?**
 
-No. agentevals is a single `pip install` with no database, no message queue, and no external services. The CLI evaluates trace files directly from disk. The web UI and live streaming use in-memory session state. You can go from zero to scored traces in under a minute.
+No. agentevals is a single `pip install` with no database, no message queue, and no external services. The CLI evaluates trace files directly from disk. The web UI and live streaming use in-memory session state.
 
 **Does the CLI require a running server?**
 
@@ -397,23 +380,19 @@ No. `agentevals run` evaluates trace files entirely offline. The server (`agente
 
 **Can I use agentevals in CI/CD?**
 
-Yes. The CLI is designed for pipeline use: pass trace files and an eval set, set a threshold, and let the exit code gate your deployment. Combine it with `--output json` for machine-readable results. No server process needed.
+Yes. Pass trace files and an eval set, set a threshold, and let the exit code gate your deployment. Combine with `--output json` for machine-readable results. No server process needed.
 
 **What if I switch agent frameworks?**
 
-Because agentevals uses OpenTelemetry as its universal interface, switching frameworks (e.g., from LangChain to Strands, or from ADK to OpenAI Agents) does not require changing your evaluation setup. As long as your new framework emits OTel spans, the same eval sets and metrics work as before.
+Because agentevals uses OpenTelemetry as its universal interface, switching frameworks does not require changing your evaluation setup. As long as your new framework emits OTel spans, the same eval sets and metrics work as before.
 
 **Can I write evaluators in my own language?**
 
-Yes. A custom evaluator is any program that reads JSON from stdin and writes a score to stdout. Python and JavaScript have first-class scaffolding support (`agentevals evaluator init`), but any language works. If your evaluator has a `requirements.txt`, agentevals manages a cached virtual environment automatically.
+Yes. A custom evaluator is any program that reads JSON from stdin and writes a score to stdout. Python and JavaScript have first-class scaffolding support (`agentevals evaluator init`), but any language works.
 
 **Can I plug agentevals into an existing OTel pipeline?**
 
-Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this exact pattern.
-
-**Can I deploy agentevals on Kubernetes?**
-
-Yes. A Dockerfile and a [Helm chart](charts/agentevals/) are included. A single pod exposes the web UI (8001), OTLP receiver (4318), and MCP server (8080). See the [Kubernetes example](examples/kubernetes/README.md) for a full walkthrough deploying agentevals alongside kagent and an OTel Collector.
+Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
 **How does this compare to ADK's evaluations?**
 
@@ -425,7 +404,7 @@ However, if you're iterating on your agents locally, you can point your agents t
 
 AgentCore's evaluation integration (via `strands-agents-evals`) also couples agent execution with evaluation. It re-invokes the agent for each test case, converts the resulting OTel spans to AWS's ADOT format, and scores them against 4 built-in evaluators (Helpfulness, Accuracy, Harmfulness, Relevance) via a cloud API call. This means you need an AWS account, valid credentials, and network access for every evaluation.
 
-agentevals takes a different approach: it scores pre-recorded traces locally without re-running anything. It works with standard Jaeger JSON and OTLP formats from any framework, supports open-ended metrics (tool trajectory matching, LLM-based judges, custom scorers), and ships with a CLI, web UI, and MCP server. No cloud dependency required, though we do include all ADK's GCP-based evals as of now.
+agentevals scores pre-recorded traces locally without re-running anything. It works with standard Jaeger JSON and OTLP formats from any framework, supports open-ended metrics (tool trajectory matching, LLM-based judges, custom scorers), and ships with a CLI, web UI, and MCP server. No cloud dependency required, though we do include all ADK's GCP-based evals as of now.
 
 **How does this compare to LangSmith?**
 

{agentevals_cli-0.6.4 → agentevals_cli-0.7.1}/README.md

@@ -33,34 +33,16 @@ agentevals scores performance and inference quality from OpenTelemetry traces. N
 
 ## What is agentevals?
 
-agentevals is a framework-agnostic evaluation solution that scores AI agent behavior directly from [OpenTelemetry](https://opentelemetry.io/) traces. Record your agent's actions once, then evaluate as many times as you want. No re-runs, no guesswork.
+agentevals is a framework-agnostic evaluation solution that scores AI agent behavior directly from [OpenTelemetry](https://opentelemetry.io/) traces. Record your agent's actions once, then evaluate as many times as you want without re-executing or burning extra tokens.
 
 It works with any OTel-instrumented framework (LangChain, Strands, Google ADK, OpenAI Agents SDK, and others), supports Jaeger JSON and native OTLP trace formats, and ships with built-in evaluators, custom evaluator support, and LLM-based judges.
 
-- **CLI** for scripting and CI pipelines
-- **Web UI** for visual inspection and local developer experience
-- **Kubernetes and OTel support** so you can deploy right next to your agents; works natively in your OpenTelemetry pipeline
-- **MCP server** so MCP clients can run evaluations from a conversation
-
-## Why agentevals?
-
-Most evaluation tools require you to **re-execute your agent** for every test, burning tokens, time, and money on duplicate LLM calls. agentevals takes a different approach:
-
 - **No re-execution**: score agents from existing traces without replaying expensive LLM calls
-- **Framework-agnostic**: works with any agent framework that emits OpenTelemetry spans
 - **Golden eval sets**: compare actual behavior against defined expected behaviors for deterministic pass/fail gating
 - **Custom evaluators**: write scoring logic in Python, JavaScript, or any language, or offload scoring to OpenAI Eval API
 - **CI/CD ready**: gate deployments on quality thresholds directly in your pipeline
 - **Local-first**: no cloud dependency required; everything runs on your machine
-
-## How It Works
-
-agentevals follows three simple steps:
-
-1. **Collect traces**: Instrument your agent with OpenTelemetry (or export traces from your tracing backend). Point the OTLP exporter at the agentevals receiver, or load trace files directly.
-2. **Define eval sets**: Create golden evaluation sets that describe expected agent behavior: which tools should be called, in what order, and what the output should look like.
-3. **Run evaluations**: Use the CLI, Web UI, or MCP server to score traces against your eval sets. Get per-metric scores, pass/fail results, and detailed span-level breakdowns.
-
+- **Multiple interfaces**: CLI for scripting and CI, Web UI for visual inspection, MCP server for conversational evaluation, Helm chart for Kubernetes environments
 
 > [!IMPORTANT]
 > This project is under active development. Expect breaking changes.
@@ -69,7 +51,7 @@ agentevals follows three simple steps:
 
 - [Installation](#installation)
 - [Quick Start](#quick-start)
-- [Integration](#integration)
+- [Use-cases and Integrations](#use-cases-and-integrations)
 - [CLI](#cli)
 - [Custom Evaluators](#custom-evaluators)
 - [Web UI](#web-ui)
@@ -168,14 +150,14 @@ agentevals serve
 # opens http://localhost:8001
 ```
 
-You can also point any OTel-instrumented agent directly at the built-in receiver (`OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`). The UI streams tool calls, inputs, and outputs live as your agent runs. For production setups, the same receiver slots into a Kubernetes OTel Collector pipeline as an exporter destination. See [Integration](#integration) and the [Kubernetes example](examples/kubernetes/README.md) for walkthroughs.
+You can also point any OTel-instrumented agent directly at the built-in receiver (`OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`). The UI streams tool calls, inputs, and outputs live as your agent runs. For production setups, the same receiver slots into a Kubernetes OTel Collector pipeline as an exporter destination. See [Use-cases and Integrations](#use-cases-and-integrations) and the [Kubernetes example](examples/kubernetes/README.md) for walkthroughs.
 
 **Next steps:**
 
 - `agentevals evaluator list` to see all built-in and community evaluators
 - [Custom Evaluators](#custom-evaluators) to write your own scoring logic
 
-## Use-cases and integrations
+## Use-cases and Integrations
 
 ### Zero-Code (Recommended)
 
@@ -217,7 +199,7 @@ with app.session(eval_set_id="my-eval"):
 
 Requires `pip install "agentevals-cli[streaming]"`. See [examples/sdk_example/](examples/sdk_example/) for framework-specific patterns.
 
-## CLI for local testing, and CI pipelines
+## CLI
 
 ```bash
 # Multiple traces, JSON output
@@ -280,12 +262,13 @@ A `Dockerfile` is included at the project root. The image bundles the API, web U
 
 ```bash
 docker build -t agentevals .
-docker run -p 8001:8001 -p 4318:4318 agentevals
+docker run -p 8001:8001 -p 4317:4317 -p 4318:4318 agentevals
 ```
 
 | Port | Purpose |
 |------|---------|
 | 8001 | Web UI and REST API |
+| 4317 | OTLP gRPC receiver (traces and logs) |
 | 4318 | OTLP HTTP receiver (traces and logs) |
 | 8080 | MCP (Streamable HTTP) |
 
@@ -363,7 +346,7 @@ See [DEVELOPMENT.md](DEVELOPMENT.md) for build tiers, Makefile targets, and Nix
 
 **Do I need a database or any infrastructure to run agentevals?**
 
-No. agentevals is a single `pip install` with no database, no message queue, and no external services. The CLI evaluates trace files directly from disk. The web UI and live streaming use in-memory session state. You can go from zero to scored traces in under a minute.
+No. agentevals is a single `pip install` with no database, no message queue, and no external services. The CLI evaluates trace files directly from disk. The web UI and live streaming use in-memory session state.
 
 **Does the CLI require a running server?**
 
@@ -371,23 +354,19 @@ No. `agentevals run` evaluates trace files entirely offline. The server (`agente
 
 **Can I use agentevals in CI/CD?**
 
-Yes. The CLI is designed for pipeline use: pass trace files and an eval set, set a threshold, and let the exit code gate your deployment. Combine it with `--output json` for machine-readable results. No server process needed.
+Yes. Pass trace files and an eval set, set a threshold, and let the exit code gate your deployment. Combine with `--output json` for machine-readable results. No server process needed.
 
 **What if I switch agent frameworks?**
 
-Because agentevals uses OpenTelemetry as its universal interface, switching frameworks (e.g., from LangChain to Strands, or from ADK to OpenAI Agents) does not require changing your evaluation setup. As long as your new framework emits OTel spans, the same eval sets and metrics work as before.
+Because agentevals uses OpenTelemetry as its universal interface, switching frameworks does not require changing your evaluation setup. As long as your new framework emits OTel spans, the same eval sets and metrics work as before.
 
 **Can I write evaluators in my own language?**
 
-Yes. A custom evaluator is any program that reads JSON from stdin and writes a score to stdout. Python and JavaScript have first-class scaffolding support (`agentevals evaluator init`), but any language works. If your evaluator has a `requirements.txt`, agentevals manages a cached virtual environment automatically.
+Yes. A custom evaluator is any program that reads JSON from stdin and writes a score to stdout. Python and JavaScript have first-class scaffolding support (`agentevals evaluator init`), but any language works.
 
 **Can I plug agentevals into an existing OTel pipeline?**
 
-Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this exact pattern.
-
-**Can I deploy agentevals on Kubernetes?**
-
-Yes. A Dockerfile and a [Helm chart](charts/agentevals/) are included. A single pod exposes the web UI (8001), OTLP receiver (4318), and MCP server (8080). See the [Kubernetes example](examples/kubernetes/README.md) for a full walkthrough deploying agentevals alongside kagent and an OTel Collector.
+Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
 **How does this compare to ADK's evaluations?**
 
@@ -399,7 +378,7 @@ However, if you're iterating on your agents locally, you can point your agents t
 
 AgentCore's evaluation integration (via `strands-agents-evals`) also couples agent execution with evaluation. It re-invokes the agent for each test case, converts the resulting OTel spans to AWS's ADOT format, and scores them against 4 built-in evaluators (Helpfulness, Accuracy, Harmfulness, Relevance) via a cloud API call. This means you need an AWS account, valid credentials, and network access for every evaluation.
 
-agentevals takes a different approach: it scores pre-recorded traces locally without re-running anything. It works with standard Jaeger JSON and OTLP formats from any framework, supports open-ended metrics (tool trajectory matching, LLM-based judges, custom scorers), and ships with a CLI, web UI, and MCP server. No cloud dependency required, though we do include all ADK's GCP-based evals as of now.
+agentevals scores pre-recorded traces locally without re-running anything. It works with standard Jaeger JSON and OTLP formats from any framework, supports open-ended metrics (tool trajectory matching, LLM-based judges, custom scorers), and ships with a CLI, web UI, and MCP server. No cloud dependency required, though we do include all ADK's GCP-based evals as of now.
 
 **How does this compare to LangSmith?**
 

{agentevals_cli-0.6.4 → agentevals_cli-0.7.1}/docs/otel-compatibility.md

@@ -8,7 +8,9 @@ agentevals consumes OpenTelemetry traces to evaluate AI agents. This document co
 
 The [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) define standard span attributes for LLM interactions. agentevals auto-detects this format when spans contain `gen_ai.request.model` or `gen_ai.input.messages`.
 
-Supported attributes:
+This format works with LangChain, Strands, OpenAI instrumentation, Anthropic instrumentation, and any framework that follows the GenAI semantic conventions.
+
+#### Core attributes
 
 | Attribute | Description |
 |-----------|-------------|
@@ -18,9 +20,51 @@ Supported attributes:
 | `gen_ai.response.finish_reasons` | Why the model stopped generating |
 | `gen_ai.usage.input_tokens` | Input token count |
 | `gen_ai.usage.output_tokens` | Output token count |
-| `gen_ai.system` | AI system identifier (e.g. `openai`, `anthropic`) |
 
-This format works with LangChain, Strands, OpenAI instrumentation, Anthropic instrumentation, and any framework that follows the GenAI semantic conventions.
+#### Provider and response metadata (v1.37.0+)
+
+| Attribute | Description |
+|-----------|-------------|
+| `gen_ai.provider.name` | LLM provider (e.g. `openai`, `anthropic`). Replaces the deprecated `gen_ai.system`. |
+| `gen_ai.response.model` | Model name returned in the response |
+| `gen_ai.response.id` | Unique response identifier |
+
+#### Request parameters (v1.40.0)
+
+| Attribute | Description |
+|-----------|-------------|
+| `gen_ai.request.temperature` | Temperature sampling parameter |
+| `gen_ai.request.max_tokens` | Maximum output tokens limit |
+| `gen_ai.request.top_p` | Top-P (nucleus) sampling parameter |
+| `gen_ai.request.top_k` | Top-K sampling parameter |
+
+#### Cache token usage
+
+| Attribute | Description |
+|-----------|-------------|
+| `gen_ai.usage.cache_creation.input_tokens` | Tokens spent creating a prompt cache entry |
+| `gen_ai.usage.cache_read.input_tokens` | Tokens served from an existing cache entry |
+
+These are relevant for providers that support prompt caching (Anthropic, OpenAI). agentevals aggregates these across LLM spans and displays them in the performance summary.
+
+#### Agent and tool metadata (v1.31.0+)
+
+| Attribute | Description |
+|-----------|-------------|
+| `gen_ai.agent.id` | Unique agent identifier |
+| `gen_ai.agent.description` | Agent description |
+| `gen_ai.tool.description` | Tool description |
+| `gen_ai.tool.type` | Tool type classification |
+
+#### Opt-in attributes (v1.37.0+)
+
+These may contain large payloads and are typically gated behind instrumentation flags:
+
+| Attribute | Description |
+|-----------|-------------|
+| `gen_ai.system_instructions` | System prompt text |
+| `gen_ai.tool.definitions` | Tool schema definitions (JSON) |
+| `gen_ai.output.type` | Classification of output content |
 
 ### Google ADK (framework-native)
 
@@ -30,9 +74,33 @@ Google ADK emits spans under the `gcp.vertex.agent` OTel scope with proprietary
 
 Format detection is automatic. When a trace contains both ADK and GenAI attributes, ADK takes priority because it provides richer structured data. The detection logic lives in `src/agentevals/converter.py` (`get_extractor()`).
 
+## Message Formats
+
+GenAI message content (`gen_ai.input.messages`, `gen_ai.output.messages`) can use two JSON schemas. agentevals supports both and normalizes them internally.
+
+### Content-based format
+
+Used by OpenAI and LangChain instrumentors (v2):
+
+```json
+{"role": "user", "content": "Hello"}
+{"role": "assistant", "content": "...", "tool_calls": [{"type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"NYC\"}"}}]}
+```
+
+### Parts-based format (v1.36.0+)
+
+Used by newer instrumentors that follow the GenAI semconv parts schema:
+
+```json
+{"role": "user", "parts": [{"type": "text", "content": "Hello"}]}
+{"role": "assistant", "parts": [{"type": "tool_call", "name": "get_weather", "arguments": {"city": "NYC"}}]}
+```
+
+Both formats are auto-detected per message. Tool calls are normalized to `{name, id, arguments}` regardless of source format.
+
 ## Message Content Delivery
 
-GenAI message content (`gen_ai.input.messages`, `gen_ai.output.messages`) can arrive through three mechanisms. agentevals supports all of them:
+GenAI message content can arrive through three mechanisms. agentevals supports all of them:
 
 ### 1. Span attributes (simplest)
 
@@ -80,18 +148,44 @@ If you maintain an OTel-instrumented agent framework and want to align with the
 
 ## OTLP Receiver
 
-agentevals runs:
+agentevals runs two OTLP receivers:
+
+- **gRPC** on port 4317 (standard OTLP gRPC port, configurable via `--otlp-grpc-port`)
+- **HTTP** on port 4318 (standard OTLP HTTP port)
 
-- OTLP HTTP receiver on port 4318 (standard OTLP HTTP port)
-- OTLP gRPC receiver on port 4317 (standard OTLP gRPC port).
+Both accept traces and logs and feed into the same session manager.
 
-OTLP HTTP accepts:
+### OTLP HTTP
 
 | Endpoint | Content Types |
 |----------|--------------|
 | `/v1/traces` | `application/json`, `application/x-protobuf` |
 | `/v1/logs` | `application/json`, `application/x-protobuf` |
 
-Point OTLP/HTTP exporters at `http://localhost:4318`.
-Point OTLP/gRPC exporters at `localhost:4317` with `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`.
+### OTLP gRPC
+
+Implements the standard `TraceService/Export` and `LogsService/Export` RPCs. Configuration:
+
+| Setting | Default |
+|---------|---------|
+| Max message size | 8 MB |
+| Max concurrent RPCs | 32 |
+| Compression | gzip |
+| TLS | off (insecure) |
+
+### Client configuration
+
+For HTTP exporters:
+
+```bash
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+```
+
+For gRPC exporters:
+
+```bash
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+```
+
 Traces and logs stream into agentevals automatically. See [examples/README.md](../examples/README.md) for zero-code setup instructions.

{agentevals_cli-0.6.4 → agentevals_cli-0.7.1}/examples/README.md

@@ -29,6 +29,7 @@ agentevals accepts OTLP/HTTP on port 4318 (`http/protobuf` and `http/json`) and
 | [zero-code-examples/ollama/](./zero-code-examples/ollama/) | LangChain | Ollama |
 | [zero-code-examples/strands/](./zero-code-examples/strands/) | Strands | OpenAI |
 | [zero-code-examples/adk/](./zero-code-examples/adk/) | Google ADK | Gemini |
+| [zero-code-examples/pydantic-ai/](./zero-code-examples/pydantic-ai/) | Pydantic AI | OpenAI |
 
 This approach works with any framework that has OTel instrumentation: LangChain, Strands, Google ADK, etc. If your framework already emits OTel spans, you only need to add `OTLPSpanExporter` (and `OTLPLogExporter` if it uses GenAI log-based content delivery).
 
@@ -103,6 +104,7 @@ Detection checks for `gen_ai.request.model` / `gen_ai.input.messages` (GenAI sem
 | [zero-code-examples/ollama/](./zero-code-examples/ollama/) | LangChain | Ollama | GenAI semconv (logs) | Standard OTLP export |
 | [zero-code-examples/strands/](./zero-code-examples/strands/) | Strands | OpenAI | GenAI semconv (events*) | Standard OTLP export |
 | [zero-code-examples/adk/](./zero-code-examples/adk/) | Google ADK | Gemini | ADK built-in | Standard OTLP export |
+| [zero-code-examples/pydantic-ai/](./zero-code-examples/pydantic-ai/) | Pydantic AI | OpenAI | GenAI semconv (span attrs) | Standard OTLP export |
 | [langchain_agent](./langchain_agent/) | LangChain | OpenAI | GenAI semconv (logs) | SDK WebSocket |
 | [strands_agent](./strands_agent/) | Strands | OpenAI | GenAI semconv (events*) | SDK WebSocket |
 | [dice_agent](./dice_agent/) | Google ADK | Gemini | ADK built-in | SDK WebSocket |
@@ -217,6 +219,7 @@ python examples/zero-code-examples/langchain/run.py
 python examples/zero-code-examples/ollama/run.py
 python examples/zero-code-examples/strands/run.py
 python examples/zero-code-examples/adk/run.py
+python examples/zero-code-examples/pydantic-ai/run.py
 
 # SDK examples:
 python examples/sdk_example/context_manager_example.py
@@ -232,7 +235,7 @@ python examples/strands_agent/main.py
 Traces stream to the dev server in real-time. Evaluation runs automatically when the session completes.
 
 See each example's README for prerequisites and detailed instructions:
-- [zero-code-examples/](./zero-code-examples/) (LangChain + Strands, standard OTLP)
+- [zero-code-examples/](./zero-code-examples/) (LangChain, Strands, ADK, OpenAI Agents, Pydantic AI — standard OTLP)
 - [dice_agent/README.md](./dice_agent/README.md) (Google ADK + Gemini)
 - [langchain_agent/README.md](./langchain_agent/README.md) (LangChain + OpenAI, SDK)
 - [strands_agent/](./strands_agent/) (Strands + OpenAI, SDK)

agentevals_cli-0.7.1/examples/zero-code-examples/pydantic-ai/requirements.txt

@@ -0,0 +1,5 @@
+pydantic-ai>=1.81.0
+
+opentelemetry-sdk>=1.36.0
+opentelemetry-exporter-otlp-proto-http>=1.36.0
+python-dotenv>=1.0.0

agentevals_cli-0.7.1/examples/zero-code-examples/pydantic-ai/run.py

@@ -0,0 +1,105 @@
+"""Run a dice-rolling Pydantic AI agent with OTLP export — no agentevals SDK.
+
+Demonstrates zero-code integration: any OTel-instrumented agent streams
+traces to agentevals by pointing the OTLP exporter at the receiver.
+
+Pydantic AI has built-in OTel support via Agent.instrument_all(). By default
+it uses version 2 of the GenAI semconv format, storing message content in span
+attributes — only a TracerProvider is needed.
+No separate instrumentation library is needed.
+
+Prerequisites:
+    1. pip install -r requirements.txt
+    2. agentevals serve --dev
+    3. export OPENAI_API_KEY="your-key-here"
+
+Usage:
+    python examples/zero-code-examples/pydantic-ai/run.py
+"""
+
+import os
+import random
+
+from dotenv import load_dotenv
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from pydantic_ai import Agent
+
+load_dotenv(override=True)
+
+
+def roll_die(sides: int) -> int:
+    """Roll a die with the given number of sides and return the result."""
+    return random.randint(1, sides)
+
+
+def check_prime(number: int) -> bool:
+    """Return True if the number is prime, False otherwise."""
+    if number < 2:
+        return False
+    for i in range(2, int(number**0.5) + 1):
+        if number % i == 0:
+            return False
+    return True
+
+
+def main():
+    if not os.getenv("OPENAI_API_KEY"):
+        print("OPENAI_API_KEY not set.")
+        return
+
+    endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318")
+    print(f"OTLP endpoint: {endpoint}")
+
+    os.environ.setdefault(
+        "OTEL_RESOURCE_ATTRIBUTES",
+        "agentevals.eval_set_id=pydantic_ai_eval,agentevals.session_name=pydantic-ai-zero-code",
+    )
+
+    resource = Resource.create()
+
+    tracer_provider = TracerProvider(resource=resource)
+    tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(), schedule_delay_millis=1000))
+    trace.set_tracer_provider(tracer_provider)
+
+    # Enable Pydantic AI's built-in OTel instrumentation. This one call
+    # wires up all agents globally — no framework-specific instrumentor
+    # library (like opentelemetry-instrumentation-openai-v2) is needed.
+    Agent.instrument_all()
+
+    agent = Agent(
+        "openai:gpt-4o-mini",
+        instructions="You are a helpful assistant. You can roll dice and check if numbers are prime.",
+    )
+    agent.tool_plain(roll_die)
+    agent.tool_plain(check_prime)
+
+    test_queries = [
+        "Hi! Can you help me?",
+        "Roll a 20-sided die for me",
+        "Is the number you rolled prime?",
+    ]
+
+    message_history = []
+
+    try:
+        for i, query in enumerate(test_queries, 1):
+            print(f"\n[{i}/{len(test_queries)}] User: {query}")
+
+            result = agent.run_sync(query, message_history=message_history)
+
+            print(f"     Agent: {result.output}")
+
+            # Pass the full message history forward for multi-turn conversation.
+            message_history = result.all_messages()
+    finally:
+        print()
+        tracer_provider.force_flush()
+        print("All traces flushed to OTLP receiver.")
+
+
+if __name__ == "__main__":
+    main()

{agentevals_cli-0.6.4 → agentevals_cli-0.7.1}/flake.lock

@@ -1,25 +1,5 @@
 {
   "nodes": {
-    "devshell": {
-      "inputs": {
-        "nixpkgs": [
-          "nixpkgs"
-        ]
-      },
-      "locked": {
-        "lastModified": 1768818222,
-        "narHash": "sha256-460jc0+CZfyaO8+w8JNtlClB2n4ui1RbHfPTLkpwhU8=",
-        "owner": "numtide",
-        "repo": "devshell",
-        "rev": "255a2b1725a20d060f566e4755dbf571bbbb5f76",
-        "type": "github"
-      },
-      "original": {
-        "owner": "numtide",
-        "repo": "devshell",
-        "type": "github"
-      }
-    },
     "flake-utils": {
       "inputs": {
         "systems": "systems"
@@ -102,7 +82,6 @@
     },
     "root": {
       "inputs": {
-        "devshell": "devshell",
         "flake-utils": "flake-utils",
         "nixpkgs": "nixpkgs",
         "pyproject-build-systems": "pyproject-build-systems",