agentevals-cli 0.9.11__tar.gz → 0.10.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/dependabot.yml +6 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/workflows/ci.yml +28 -5
- agentevals_cli-0.10.0/.github/workflows/kagent-contract-policy.yml +26 -0
- agentevals_cli-0.10.0/.github/workflows/kagent-contract.yml +143 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/workflows/publish-evaluator-sdk.yml +1 -1
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/workflows/release.yml +4 -4
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/CONTRIBUTING.md +17 -25
- agentevals_cli-0.10.0/PKG-INFO +309 -0
- agentevals_cli-0.10.0/README.md +267 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/rbac.yaml +3 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/values.yaml +15 -7
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/docs/custom-evaluators.md +37 -99
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/docs/eval-set-format.md +24 -0
- agentevals_cli-0.10.0/docs/opentelemetry-pipeline.md +174 -0
- agentevals_cli-0.10.0/docs/otel-compatibility.md +122 -0
- agentevals_cli-0.10.0/examples/README.md +138 -0
- agentevals_cli-0.10.0/examples/dice_agent/main.py +113 -0
- agentevals_cli-0.10.0/examples/dice_agent/test_streaming.py +44 -0
- agentevals_cli-0.10.0/examples/kubernetes/README.md +198 -0
- agentevals_cli-0.10.0/examples/kubernetes/agentevals-values.yaml +8 -0
- agentevals_cli-0.10.0/examples/kubernetes/agents.yaml +54 -0
- agentevals_cli-0.10.0/examples/kubernetes/otel-collector.yaml +140 -0
- agentevals_cli-0.10.0/examples/langchain_agent/README.md +40 -0
- agentevals_cli-0.10.0/examples/langchain_agent/main.py +104 -0
- agentevals_cli-0.10.0/examples/langchain_agent/test_streaming.py +44 -0
- agentevals_cli-0.10.0/examples/strands_agent/main.py +88 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/adk/run.py +4 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/langchain/run.py +4 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/ollama/run.py +4 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/openai-agents/requirements.txt +1 -1
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/openai-agents/run.py +4 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/pydantic-ai/run.py +4 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/strands/run.py +4 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/packages/evaluator-sdk-py/src/agentevals_evaluator_sdk/types.py +7 -2
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/pyproject.toml +8 -8
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/_protocol.py +7 -2
- agentevals_cli-0.10.0/src/agentevals/_static/assets/index-DLtCJiNw.js +348 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/_static/index.html +1 -1
- agentevals_cli-0.10.0/src/agentevals/adk_bridge.py +144 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/app.py +19 -12
- agentevals_cli-0.10.0/src/agentevals/api/debug_routes.py +270 -0
- agentevals_cli-0.10.0/src/agentevals/api/dependencies.py +28 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/models.py +10 -47
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/otlp_app.py +6 -2
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/otlp_grpc.py +38 -18
- agentevals_cli-0.10.0/src/agentevals/api/otlp_http.py +348 -0
- agentevals_cli-0.10.0/src/agentevals/api/otlp_processing.py +156 -0
- agentevals_cli-0.10.0/src/agentevals/api/otlp_routes.py +59 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/routes.py +105 -78
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/streaming_routes.py +47 -138
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/builtin_metrics.py +57 -49
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/cli.py +64 -6
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/config.py +67 -45
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/custom_evaluators.py +126 -230
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/eval_config_loader.py +14 -2
- agentevals_cli-0.10.0/src/agentevals/evaluation_events.py +352 -0
- agentevals_cli-0.10.0/src/agentevals/evaluator/resolver.py +138 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/evaluator/sources.py +20 -7
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/evaluator/templates.py +5 -2
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/evaluator/venv.py +51 -3
- agentevals_cli-0.10.0/src/agentevals/genai/__init__.py +1 -0
- agentevals_cli-0.10.0/src/agentevals/genai/extract.py +766 -0
- agentevals_cli-0.10.0/src/agentevals/genai/grouping.py +70 -0
- agentevals_cli-0.10.0/src/agentevals/genai/matching.py +106 -0
- agentevals_cli-0.10.0/src/agentevals/genai/messages.py +264 -0
- agentevals_cli-0.10.0/src/agentevals/genai/model.py +117 -0
- agentevals_cli-0.10.0/src/agentevals/genai/overlay.py +407 -0
- agentevals_cli-0.10.0/src/agentevals/genai/routing.py +62 -0
- agentevals_cli-0.10.0/src/agentevals/genai/semconv.py +105 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/loader/__init__.py +10 -1
- agentevals_cli-0.10.0/src/agentevals/loader/auto.py +147 -0
- agentevals_cli-0.10.0/src/agentevals/loader/jaeger.py +175 -0
- agentevals_cli-0.10.0/src/agentevals/loader/otlp.py +89 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/mcp_server.py +27 -39
- agentevals_cli-0.10.0/src/agentevals/otel/__init__.py +1 -0
- agentevals_cli-0.10.0/src/agentevals/otel/decode.py +678 -0
- agentevals_cli-0.10.0/src/agentevals/otel/encode.py +190 -0
- agentevals_cli-0.10.0/src/agentevals/otel/identity.py +30 -0
- agentevals_cli-0.10.0/src/agentevals/otel/model.py +350 -0
- agentevals_cli-0.10.0/src/agentevals/otel/sdk_export.py +352 -0
- agentevals_cli-0.10.0/src/agentevals/otel/store.py +948 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/run/fetcher.py +5 -24
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/run/result_builder.py +10 -6
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/run/worker.py +5 -3
- agentevals_cli-0.10.0/src/agentevals/runner.py +349 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/sdk.py +92 -143
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/models.py +1 -1
- agentevals_cli-0.10.0/src/agentevals/storage/postgres/migrations/000002_drop_openai_eval.down.sql +6 -0
- agentevals_cli-0.10.0/src/agentevals/storage/postgres/migrations/000002_drop_openai_eval.up.sql +9 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/postgres/migrator.py +7 -1
- agentevals_cli-0.10.0/src/agentevals/streaming/__init__.py +67 -0
- agentevals_cli-0.10.0/src/agentevals/streaming/manager.py +582 -0
- agentevals_cli-0.10.0/src/agentevals/trace_metrics.py +164 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/api/test_evaluate_persistence.py +1 -1
- agentevals_cli-0.10.0/tests/api/test_live_routes.py +234 -0
- agentevals_cli-0.10.0/tests/contract/kagent/README.md +27 -0
- agentevals_cli-0.10.0/tests/contract/kagent/check.sh +59 -0
- agentevals_cli-0.10.0/tests/contract/kagent/policy/agentevals.rego +123 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/main-2cc3ffae/a2a.yaml +64 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/main-2cc3ffae/imports.yaml +22 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/main-2cc3ffae/kagent.yaml +114 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/main-2cc3ffae/manifest.yaml +12 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/main-2cc3ffae/signals.yaml +247 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/pr3121-412ef54e/a2a.yaml +64 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/pr3121-412ef54e/imports.yaml +22 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/pr3121-412ef54e/kagent.yaml +136 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/pr3121-412ef54e/manifest.yaml +12 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/pr3121-412ef54e/signals.yaml +363 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/v1.0.0-alpha9/a2a.yaml +64 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/v1.0.0-alpha9/kagent.yaml +114 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/v1.0.0-alpha9/manifest.yaml +11 -0
- agentevals_cli-0.10.0/tests/contract/kagent/registries/v1.0.0-alpha9/signals.yaml +196 -0
- agentevals_cli-0.10.0/tests/contract/kagent/test.sh +37 -0
- agentevals_cli-0.10.0/tests/contract/kagent/testdata/agentevals_contract_gone/manifest.yaml +8 -0
- agentevals_cli-0.10.0/tests/contract/kagent/testdata/agentevals_contract_gone/model.yaml +11 -0
- agentevals_cli-0.10.0/tests/contract/kagent/testdata/agentevals_contract_missing_attribute/manifest.yaml +5 -0
- agentevals_cli-0.10.0/tests/contract/kagent/testdata/agentevals_contract_missing_attribute/model.yaml +25 -0
- agentevals_cli-0.10.0/tests/contract/kagent/testdata/agentevals_contract_requirement_floor/manifest.yaml +8 -0
- agentevals_cli-0.10.0/tests/contract/kagent/testdata/agentevals_contract_requirement_floor/model.yaml +14 -0
- agentevals_cli-0.10.0/tests/contract/kagent/weaver.sh +15 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/MANIFEST.md +69 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S1.json +398 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S2.json +1220 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S3.json +1648 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S4a.json +1257 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S4b.json +1081 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S5.json +928 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S6a.json +612 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S6b.json +791 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S1.json +465 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S1.truth.json +29 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S2.json +1299 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S2.truth.json +57 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S3.json +1733 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S3.truth.json +69 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S4a.json +1336 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S4b.json +1160 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S5.json +1001 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S5.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S6a.json +679 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S6a.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S6b.json +864 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_default_contrib/S6b.truth.json +41 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S1.json +560 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S2.json +2693 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S3.json +3499 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S4a.json +2628 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S4b.json +1837 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S5.json +1828 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S6a.json +807 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S6b.json +1385 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S1.json +621 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S1.truth.json +29 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S2.json +2718 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S2.truth.json +57 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S3.json +3488 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S3.truth.json +69 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S4a.json +2653 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S4b.json +1886 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S5.json +1877 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S5.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S6a.json +868 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S6a.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S6b.json +1434 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_event_stable_contrib/S6b.truth.json +41 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S1.json +398 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S1.truth.json +29 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S2.json +1220 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S2.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S3.json +1648 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S3.truth.json +63 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S4a.json +1257 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S4a.truth.json +46 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S4b.json +1081 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S4b.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S5.json +928 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S5.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S6a.json +612 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S6a.truth.json +38 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S6b.json +791 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v1_nolegacy/S6b.truth.json +37 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S1.json +418 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S2.json +1240 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S3.json +1708 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S4a.json +1277 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S4b.json +1127 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S5.json +948 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S6a.json +632 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S6b.json +811 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_default/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S1.json +517 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S2.json +2392 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S3.json +2749 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S4a.json +2300 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S4b.json +1673 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S5.json +1554 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S6a.json +848 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S6b.json +1282 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S1.json +517 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S2.json +2392 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S3.json +2749 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S4a.json +2300 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S4b.json +1673 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S5.json +1554 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S6a.json +848 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S6b.json +1282 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_event_only_nolegacy/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S1.json +376 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S2.json +1159 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S3.json +1384 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S4a.json +1169 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S4b.json +1031 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S5.json +894 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S6a.json +632 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S6b.json +757 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_no_content/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S1.json +541 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S2.json +2446 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S3.json +2821 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S4a.json +2354 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S4b.json +1733 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S5.json +1590 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S6a.json +866 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S6b.json +1318 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_and_event/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S1.json +400 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S2.json +1213 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S3.json +1456 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S4a.json +1223 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S4b.json +1091 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S5.json +930 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S6a.json +650 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S6b.json +793 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S1.json +461 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S1.truth.json +29 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S2.json +1292 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S2.truth.json +57 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S3.json +1541 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S3.truth.json +69 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S4a.json +1302 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S4b.json +1164 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S5.json +1003 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S5.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S6a.json +717 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S6a.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S6b.json +866 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_contrib/S6b.truth.json +41 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S1.json +400 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S2.json +1213 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S3.json +1456 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S4a.json +1223 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S4b.json +1091 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S5.json +930 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S6a.json +650 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S6b.json +793 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2_span_only_nolegacy/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S1.json +438 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S2.json +2107 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S3.json +2409 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S4a.json +2015 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S4b.json +1412 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S5.json +1348 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S6a.json +745 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S6b.json +1100 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_event_only/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S1.json +297 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S1.truth.json +29 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S2.json +874 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S2.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S3.json +1044 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S3.truth.json +63 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S4a.json +884 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S4a.truth.json +46 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S4b.json +770 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S4b.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S5.json +688 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S5.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S6a.json +529 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S6a.truth.json +38 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S6b.json +575 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_no_content/S6b.truth.json +37 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S1.json +321 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S1.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S2.json +952 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S2.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S3.json +1128 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S3.truth.json +68 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S4a.json +962 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S4b.json +842 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S5.json +748 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S5.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S6a.json +553 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S6a.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S6b.json +623 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only/S6b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S1.json +382 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S1.truth.json +29 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S2.json +1031 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S2.truth.json +57 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S3.json +1213 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S3.truth.json +69 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S4a.json +1041 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S4a.truth.json +52 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S4b.json +915 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S4b.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S5.json +821 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S5.truth.json +56 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S6a.json +620 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S6a.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S6b.json +696 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/adk_py/v2proj_span_only_contrib/S6b.truth.json +41 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/adk_cycle_poisons_batch.json +162 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/adk_cycle_poisons_batch.truth.json +30 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/adk_go_main.json +289 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/adk_go_main.truth.json +43 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/content_capture_off.json +188 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/content_capture_off.truth.json +36 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/deprecated_aliases.json +95 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/deprecated_aliases.truth.json +26 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/details_event_log_debug.json +163 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/details_event_log_debug.truth.json +30 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/duplicate_span_ids.json +186 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/duplicate_span_ids.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_eventName.json +162 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_eventName.truth.json +36 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_event_name_attr.json +167 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_event_name_attr.truth.json +36 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_otel_event_name.json +167 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_otel_event_name.truth.json +36 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_span_event.json +132 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/ext_eval_span_event.truth.json +36 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/failed_inference_then_retry.json +151 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/failed_inference_then_retry.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/mcp_tool_call.json +315 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/mcp_tool_call.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_agents_handoff.json +308 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_agents_handoff.truth.json +50 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_v2_logs_no_span_id_ambiguous.json +569 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_v2_logs_no_span_id_ambiguous.truth.json +26 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_v2_logs_no_span_id_single.json +244 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_v2_logs_no_span_id_single.truth.json +26 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_v2_logs_span_id.json +537 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/openai_v2_logs_span_id.truth.json +40 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/remote_parent_root.json +108 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/remote_parent_root.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/self_parent.json +159 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/self_parent.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/strands_span_events.json +274 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/strands_span_events.truth.json +42 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/three_trace_conversation.json +292 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/three_trace_conversation.truth.json +55 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/tool_error_no_error_type.json +192 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/tool_error_no_error_type.truth.json +38 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/two_span_cycle.json +212 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/two_span_cycle.truth.json +39 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/upstream_new_parts.json +113 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/upstream_new_parts.truth.json +27 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/util_genai_openai_tool_loop.json +214 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/hand/util_genai_openai_tool_loop.truth.json +41 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/README.md +24 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/adk_go_1turn.json +1 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/adk_go_1turn.truth.json +60 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/adk_go_2turn.json +1 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/adk_go_2turn.truth.json +90 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/claude_native_1turn_subagent.json +1 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/claude_native_1turn_subagent.truth.json +30 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/claude_native_2turn.json +1 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/claude_native_2turn.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/claude_recipe_2turn.json +1 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/kagent/v1.0.0-alpha9/claude_recipe_2turn.truth.json +116 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/helm.json +5333 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/helm.truth.json +44 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/helm_2.json +3708 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/helm_2.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/helm_3.json +7182 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/helm_3.truth.json +47 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/k8s.json +3908 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/k8s.truth.json +28 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/tempo_export_with_batches.json +1 -0
- agentevals_cli-0.10.0/tests/fixtures/otlp/samples/tempo_export_with_batches.truth.json +44 -0
- agentevals_cli-0.10.0/tests/genai/test_continuations.py +119 -0
- agentevals_cli-0.10.0/tests/genai/test_corpus.py +305 -0
- agentevals_cli-0.10.0/tests/genai/test_failed_turns.py +73 -0
- agentevals_cli-0.10.0/tests/genai/test_grouping.py +24 -0
- agentevals_cli-0.10.0/tests/genai/test_legacy_chat.py +48 -0
- agentevals_cli-0.10.0/tests/genai/test_tools.py +137 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/integration/conftest.py +48 -26
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/integration/test_live_agents.py +16 -17
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/integration/test_otlp_grpc_receiver.py +4 -5
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/integration/test_session_grouping.py +49 -44
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/integration/test_timing_stress.py +19 -18
- agentevals_cli-0.10.0/tests/otel/builders.py +185 -0
- agentevals_cli-0.10.0/tests/otel/test_emit.py +316 -0
- agentevals_cli-0.10.0/tests/otel/test_store.py +385 -0
- agentevals_cli-0.10.0/tests/run/__init__.py +0 -0
- agentevals_cli-0.10.0/tests/storage/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/storage/test_migrator.py +3 -3
- agentevals_cli-0.10.0/tests/streaming/__init__.py +0 -0
- agentevals_cli-0.10.0/tests/streaming/test_live.py +224 -0
- agentevals_cli-0.10.0/tests/streaming/test_session_export.py +69 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_api.py +119 -101
- agentevals_cli-0.10.0/tests/test_builtin_metrics.py +295 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_cli.py +1 -1
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_jaeger_loader.py +14 -45
- agentevals_cli-0.10.0/tests/test_layering.py +37 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_loader_auto.py +20 -3
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_mcp_server.py +26 -0
- agentevals_cli-0.10.0/tests/test_otlp_http_protocol.py +807 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_otlp_loader.py +74 -73
- agentevals_cli-0.10.0/tests/test_otlp_receiver.py +985 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_runner.py +92 -119
- agentevals_cli-0.10.0/tests/test_sdk.py +460 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_trace_metrics.py +64 -70
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/package-lock.json +288 -186
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/package.json +12 -12
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/api/client.ts +13 -2
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/InspectorView.tsx +4 -27
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/streaming/LiveConversationPanel.tsx +3 -1
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/streaming/LiveMessage.tsx +18 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/streaming/LiveStreamingView.tsx +75 -16
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/streaming/SessionCard.tsx +70 -23
- agentevals_cli-0.10.0/ui/src/components/streaming/turnStatus.ts +31 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/upload/TraceEditorDrawer.tsx +1 -1
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/config.ts +0 -7
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/context/TraceProvider.tsx +7 -2
- agentevals_cli-0.10.0/ui/src/lib/trace-helpers.ts +2 -0
- agentevals_cli-0.10.0/ui/src/lib/trace-patcher.ts +352 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/types.ts +40 -10
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/utils.ts +13 -19
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/uv.lock +88 -82
- agentevals_cli-0.9.11/PKG-INFO +0 -483
- agentevals_cli-0.9.11/README.md +0 -442
- agentevals_cli-0.9.11/docs/otel-compatibility.md +0 -191
- agentevals_cli-0.9.11/docs/streaming.md +0 -322
- agentevals_cli-0.9.11/examples/README.md +0 -251
- agentevals_cli-0.9.11/examples/custom_evaluators/eval_config_openai_eval.yaml +0 -18
- agentevals_cli-0.9.11/examples/dice_agent/main.py +0 -139
- agentevals_cli-0.9.11/examples/dice_agent/test_streaming.py +0 -63
- agentevals_cli-0.9.11/examples/kubernetes/README.md +0 -256
- agentevals_cli-0.9.11/examples/langchain_agent/README.md +0 -232
- agentevals_cli-0.9.11/examples/langchain_agent/main.py +0 -180
- agentevals_cli-0.9.11/examples/langchain_agent/test_streaming.py +0 -79
- agentevals_cli-0.9.11/examples/strands_agent/main.py +0 -138
- agentevals_cli-0.9.11/src/agentevals/_static/assets/index-G7gMkrDR.js +0 -349
- agentevals_cli-0.9.11/src/agentevals/api/debug_routes.py +0 -262
- agentevals_cli-0.9.11/src/agentevals/api/dependencies.py +0 -28
- agentevals_cli-0.9.11/src/agentevals/api/otlp_processing.py +0 -335
- agentevals_cli-0.9.11/src/agentevals/api/otlp_routes.py +0 -69
- agentevals_cli-0.9.11/src/agentevals/converter.py +0 -319
- agentevals_cli-0.9.11/src/agentevals/evaluator/resolver.py +0 -70
- agentevals_cli-0.9.11/src/agentevals/extraction.py +0 -689
- agentevals_cli-0.9.11/src/agentevals/genai_converter.py +0 -477
- agentevals_cli-0.9.11/src/agentevals/loader/auto.py +0 -108
- agentevals_cli-0.9.11/src/agentevals/loader/base.py +0 -53
- agentevals_cli-0.9.11/src/agentevals/loader/jaeger.py +0 -112
- agentevals_cli-0.9.11/src/agentevals/loader/otlp.py +0 -252
- agentevals_cli-0.9.11/src/agentevals/openai_eval_backend.py +0 -267
- agentevals_cli-0.9.11/src/agentevals/otlp_anyvalue.py +0 -118
- agentevals_cli-0.9.11/src/agentevals/runner.py +0 -357
- agentevals_cli-0.9.11/src/agentevals/streaming/__init__.py +0 -120
- agentevals_cli-0.9.11/src/agentevals/streaming/incremental_processor.py +0 -341
- agentevals_cli-0.9.11/src/agentevals/streaming/processor.py +0 -288
- agentevals_cli-0.9.11/src/agentevals/streaming/ws_server.py +0 -926
- agentevals_cli-0.9.11/src/agentevals/trace_attrs.py +0 -116
- agentevals_cli-0.9.11/src/agentevals/trace_metrics.py +0 -245
- agentevals_cli-0.9.11/src/agentevals/utils/genai_messages.py +0 -142
- agentevals_cli-0.9.11/src/agentevals/utils/log_enrichment.py +0 -192
- agentevals_cli-0.9.11/tests/test_converter.py +0 -567
- agentevals_cli-0.9.11/tests/test_extraction.py +0 -1119
- agentevals_cli-0.9.11/tests/test_genai_converter.py +0 -992
- agentevals_cli-0.9.11/tests/test_incremental_processor.py +0 -26
- agentevals_cli-0.9.11/tests/test_log_enrichment.py +0 -214
- agentevals_cli-0.9.11/tests/test_openai_eval_backend.py +0 -117
- agentevals_cli-0.9.11/tests/test_otlp_receiver.py +0 -1826
- agentevals_cli-0.9.11/tests/test_sdk.py +0 -472
- agentevals_cli-0.9.11/ui/src/lib/trace-helpers.ts +0 -73
- agentevals_cli-0.9.11/ui/src/lib/trace-patcher.ts +0 -328
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.claude/skills/eval/SKILL.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.claude/skills/eval/evals/evals.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.claude/skills/inspect/SKILL.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.claude/skills/inspect/evals/evals.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.dockerignore +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.gitignore +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/.mcp.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/DEVELOPMENT.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/Dockerfile +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/LICENSE +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/Makefile +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/Chart.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/NOTES.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/_helpers.tpl +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/deployment.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/postgresql-secret.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/postgresql.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/service.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/charts/agentevals/templates/serviceaccount.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/docs/assets/logo-color-on-transparent.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/docs/assets/logo-color.png +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/docs/assets/logo-dark-on-transparent.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/docs/run-history.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_evaluators/eval_config.yaml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_evaluators/response_quality.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_evaluators/tool_call_checker.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_sink/README.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_sink/agentevals_example_custom_sink/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_sink/agentevals_example_custom_sink/sink.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/custom_sink/pyproject.toml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/dice_agent/README.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/dice_agent/agent.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/dice_agent/eval_set.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/langchain_agent/agent.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/langchain_agent/eval_set.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/langchain_agent/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/sdk_example/async_example.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/sdk_example/context_manager_example.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/sdk_example/decorator_example.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/sdk_example/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/strands_agent/agent.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/strands_agent/eval_set.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/strands_agent/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/adk/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/langchain/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/ollama/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/pydantic-ai/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/examples/zero-code-examples/strands/requirements.txt +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/flake.lock +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/flake.nix +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/packages/evaluator-sdk-py/README.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/packages/evaluator-sdk-py/pyproject.toml +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/packages/evaluator-sdk-py/src/agentevals_evaluator_sdk/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/packages/evaluator-sdk-py/src/agentevals_evaluator_sdk/decorator.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/eval_set_helm.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/evalset_helm_3_2026-02-23.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/evalset_k8s_2026-02-20.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/helm.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/helm_2.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/helm_3.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/k8s.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/samples/tempo_export_with_batches.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/_static/assets/index-BqibLiHO.css +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/_static/logo.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/_static/vite.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/api/runs_routes.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/evaluator/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/output.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/resolvers/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/resolvers/kubernetes.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/run/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/run/service.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/run/sinks.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/config.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/postgres/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/postgres/migrations/000001_init.down.sql +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/postgres/migrations/000001_init.up.sql +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/postgres/pool.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/repos/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/repos/memory.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/storage/repos/postgres.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/streaming/exports.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/streaming/session.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/utils/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/src/agentevals/utils/log_buffer.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/api/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/api/test_runs_routes.py +0 -0
- {agentevals_cli-0.9.11/tests/integration → agentevals_cli-0.10.0/tests/genai}/__init__.py +0 -0
- {agentevals_cli-0.9.11/tests/resolvers → agentevals_cli-0.10.0/tests/integration}/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/integration/test_evaluation_pipeline.py +0 -0
- {agentevals_cli-0.9.11/tests/run → agentevals_cli-0.10.0/tests/otel}/__init__.py +0 -0
- {agentevals_cli-0.9.11/tests/storage → agentevals_cli-0.10.0/tests/resolvers}/__init__.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/resolvers/test_kubernetes.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/resolvers/test_registry.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/run/test_fetcher.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/run/test_result_builder.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/run/test_service.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/run/test_sinks.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/storage/test_config.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/storage/test_memory_repos.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/storage/test_models.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_credential_injection.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_eval_config_loader.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_output.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/tests/test_protocol.py +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/.gitignore +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/README.md +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/eslint.config.js +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/index.html +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/public/logo.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/public/vite.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/App.css +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/App.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/assets/react.svg +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/annotation-queue/AnnotationDetailPanel.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/annotation-queue/AnnotationQueueView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/annotation-queue/AnnotationTable.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/bug-report/BugReportModal.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/BuilderHeader.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/BuilderView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/EvalCaseCard.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/EvalCasesList.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/InvocationEditor.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/JsonPreview.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/MetadataEditor.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/TraceUploadZone.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/builder/index.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/DashboardView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/MetricScoreCard.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/PerformanceCard.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/PerformanceCharts.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/SummaryStats.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/TraceCard.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/dashboard/TraceTable.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/ComparisonPanel.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/DataSection.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/InspectorHeader.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/InspectorLayout.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/InvocationCard.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/InvocationSummaryPanel.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/MetricResultsSection.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/MetricsComparisonSection.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/PerformanceSection.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/ToolCallList.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/inspector/TrajectoryComparisonDetails.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/runs/PassRateTrendChart.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/runs/PerMetricTrendChart.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/runs/RunDetailView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/runs/RunsHistoryTable.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/runs/RunsView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/runs/runHistory.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/sidebar/Sidebar.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/streaming/SessionMetadata.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/upload/EvalSetEditorDrawer.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/upload/FileDropZone.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/upload/MetricSelector.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/upload/RawJsonPreview.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/upload/UploadView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/components/welcome/WelcomeView.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/context/TraceContext.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/index.css +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/console-capture.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/eval-config.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/evalset-builder.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/network-capture.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/trace-loader.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/lib/trace-metadata.ts +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/src/main.tsx +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/tsconfig.app.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/tsconfig.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/tsconfig.node.json +0 -0
- {agentevals_cli-0.9.11 → agentevals_cli-0.10.0}/ui/vite.config.ts +0 -0
|
@@ -33,6 +33,12 @@ updates:
|
|
|
33
33
|
cooldown:
|
|
34
34
|
default-days: 5
|
|
35
35
|
semver-major-days: 14
|
|
36
|
+
ignore:
|
|
37
|
+
# mcp 2.x renamed FastMCP to MCPServer and dropped host/port from the
|
|
38
|
+
# constructor, which breaks mcp_server.py and the shipped image. Bump by
|
|
39
|
+
# hand alongside the migration.
|
|
40
|
+
- dependency-name: "mcp"
|
|
41
|
+
update-types: ["version-update:semver-major"]
|
|
36
42
|
groups:
|
|
37
43
|
python-google-ai:
|
|
38
44
|
applies-to: version-updates
|
|
@@ -25,7 +25,7 @@ jobs:
|
|
|
25
25
|
with:
|
|
26
26
|
fetch-depth: 0
|
|
27
27
|
|
|
28
|
-
- uses: astral-sh/setup-uv@
|
|
28
|
+
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
|
|
29
29
|
with:
|
|
30
30
|
enable-cache: true
|
|
31
31
|
|
|
@@ -39,7 +39,7 @@ jobs:
|
|
|
39
39
|
uv sync --dev
|
|
40
40
|
|
|
41
41
|
- name: Ruff check
|
|
42
|
-
run: uv run ruff check .
|
|
42
|
+
run: uv run ruff check --no-fix .
|
|
43
43
|
|
|
44
44
|
- name: Ruff format check
|
|
45
45
|
run: uv run ruff format --check .
|
|
@@ -69,7 +69,7 @@ jobs:
|
|
|
69
69
|
steps:
|
|
70
70
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
71
71
|
|
|
72
|
-
- uses: docker/setup-buildx-action@
|
|
72
|
+
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
73
73
|
|
|
74
74
|
# The release workflow is the only other place this Dockerfile gets built, and it
|
|
75
75
|
# runs after the GitHub release is already published. Build it here so a break
|
|
@@ -77,7 +77,7 @@ jobs:
|
|
|
77
77
|
# Single-arch keeps it native; the multi-arch build stays release-only because
|
|
78
78
|
# emulating arm64 through the node and uv layers is an order of magnitude slower.
|
|
79
79
|
- name: Build image
|
|
80
|
-
uses: docker/build-push-action@
|
|
80
|
+
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
|
81
81
|
with:
|
|
82
82
|
context: .
|
|
83
83
|
platforms: linux/amd64
|
|
@@ -122,7 +122,7 @@ jobs:
|
|
|
122
122
|
with:
|
|
123
123
|
fetch-depth: 0
|
|
124
124
|
|
|
125
|
-
- uses: astral-sh/setup-uv@
|
|
125
|
+
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
|
|
126
126
|
with:
|
|
127
127
|
enable-cache: true
|
|
128
128
|
|
|
@@ -137,3 +137,26 @@ jobs:
|
|
|
137
137
|
|
|
138
138
|
- name: Run tests
|
|
139
139
|
run: uv run pytest -m "not integration and not e2e" --tb=short -q
|
|
140
|
+
|
|
141
|
+
integration:
|
|
142
|
+
runs-on: ubuntu-latest
|
|
143
|
+
steps:
|
|
144
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
145
|
+
with:
|
|
146
|
+
fetch-depth: 0
|
|
147
|
+
|
|
148
|
+
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
|
|
149
|
+
with:
|
|
150
|
+
enable-cache: true
|
|
151
|
+
|
|
152
|
+
- name: Install Python 3.13
|
|
153
|
+
run: uv python install 3.13
|
|
154
|
+
|
|
155
|
+
- name: Install dependencies
|
|
156
|
+
run: |
|
|
157
|
+
uv venv --python 3.13
|
|
158
|
+
uv pip install setuptools
|
|
159
|
+
uv sync --dev --all-extras --frozen --python 3.13
|
|
160
|
+
|
|
161
|
+
- name: Run integration tests
|
|
162
|
+
run: uv run pytest tests/integration/ -m "integration and not e2e" --tb=short -q
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
name: kagent contract policy
|
|
2
|
+
|
|
3
|
+
# Tests the kagent contract policy against its local registries and negative
|
|
4
|
+
# cases when the policy or its tooling changes. It never fetches kagent itself.
|
|
5
|
+
on:
|
|
6
|
+
pull_request:
|
|
7
|
+
branches: [main]
|
|
8
|
+
paths:
|
|
9
|
+
- "tests/contract/kagent/**"
|
|
10
|
+
- ".github/workflows/kagent-contract.yml"
|
|
11
|
+
- ".github/workflows/kagent-contract-policy.yml"
|
|
12
|
+
|
|
13
|
+
permissions:
|
|
14
|
+
contents: read
|
|
15
|
+
|
|
16
|
+
jobs:
|
|
17
|
+
selftest:
|
|
18
|
+
runs-on: ubuntu-latest
|
|
19
|
+
timeout-minutes: 20
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
22
|
+
with:
|
|
23
|
+
persist-credentials: false
|
|
24
|
+
|
|
25
|
+
- name: Self test the contract policy
|
|
26
|
+
run: tests/contract/kagent/test.sh
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
name: kagent contract
|
|
2
|
+
|
|
3
|
+
# Checks every day, with Weaver, that kagent's telemetry registry still has what
|
|
4
|
+
# agentevals reads from kagent spans, and keeps one labelled issue up to date
|
|
5
|
+
# while it does not. It never runs on pull requests: kagent changes on its own
|
|
6
|
+
# schedule. See tests/contract/kagent/.
|
|
7
|
+
on:
|
|
8
|
+
schedule:
|
|
9
|
+
- cron: "23 5 * * *"
|
|
10
|
+
workflow_dispatch:
|
|
11
|
+
|
|
12
|
+
permissions: {}
|
|
13
|
+
|
|
14
|
+
concurrency:
|
|
15
|
+
group: kagent-contract
|
|
16
|
+
cancel-in-progress: false
|
|
17
|
+
|
|
18
|
+
defaults:
|
|
19
|
+
run:
|
|
20
|
+
shell: bash
|
|
21
|
+
|
|
22
|
+
env:
|
|
23
|
+
ISSUE_LABEL: kagent-contract
|
|
24
|
+
|
|
25
|
+
jobs:
|
|
26
|
+
check:
|
|
27
|
+
runs-on: ubuntu-latest
|
|
28
|
+
timeout-minutes: 30
|
|
29
|
+
permissions:
|
|
30
|
+
contents: read
|
|
31
|
+
steps:
|
|
32
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
33
|
+
with:
|
|
34
|
+
persist-credentials: false
|
|
35
|
+
|
|
36
|
+
- name: Self test the contract policy
|
|
37
|
+
run: tests/contract/kagent/test.sh
|
|
38
|
+
|
|
39
|
+
# The release tag is fetched and used in one step, and check.sh validates it
|
|
40
|
+
# before use, so nothing fetched passes through GITHUB_ENV or an expression.
|
|
41
|
+
- name: Check kagent main and the newest release
|
|
42
|
+
env:
|
|
43
|
+
GH_TOKEN: ${{ github.token }}
|
|
44
|
+
run: |
|
|
45
|
+
release="$(gh release view --repo kagent-dev/kagent --json tagName --jq .tagName)"
|
|
46
|
+
tests/contract/kagent/check.sh report.md main "$release"
|
|
47
|
+
|
|
48
|
+
- name: Summarize
|
|
49
|
+
if: always()
|
|
50
|
+
continue-on-error: true
|
|
51
|
+
run: |
|
|
52
|
+
{
|
|
53
|
+
if [ -f report.md ]; then cat report.md; fi
|
|
54
|
+
echo
|
|
55
|
+
echo "## kagent main against the committed snapshot"
|
|
56
|
+
echo
|
|
57
|
+
tests/contract/kagent/weaver.sh registry diff --v2 --diff-format markdown \
|
|
58
|
+
-r "https://github.com/kagent-dev/kagent.git@main[telemetry/registry]" \
|
|
59
|
+
--baseline-registry /contract/registries/main-2cc3ffae
|
|
60
|
+
} >> "$GITHUB_STEP_SUMMARY"
|
|
61
|
+
|
|
62
|
+
- name: Upload the report
|
|
63
|
+
if: always()
|
|
64
|
+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
65
|
+
with:
|
|
66
|
+
name: kagent-contract-report
|
|
67
|
+
path: report.md
|
|
68
|
+
if-no-files-found: ignore
|
|
69
|
+
|
|
70
|
+
# The issue body is fixed text plus the report, whose fetched content sits in a
|
|
71
|
+
# code block without backticks. It is always passed as a file, never through the shell.
|
|
72
|
+
report-failure:
|
|
73
|
+
needs: check
|
|
74
|
+
if: failure()
|
|
75
|
+
runs-on: ubuntu-latest
|
|
76
|
+
timeout-minutes: 10
|
|
77
|
+
permissions:
|
|
78
|
+
issues: write
|
|
79
|
+
steps:
|
|
80
|
+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
|
81
|
+
continue-on-error: true
|
|
82
|
+
with:
|
|
83
|
+
name: kagent-contract-report
|
|
84
|
+
|
|
85
|
+
- name: Open or update the issue
|
|
86
|
+
env:
|
|
87
|
+
GH_TOKEN: ${{ github.token }}
|
|
88
|
+
GH_REPO: ${{ github.repository }}
|
|
89
|
+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
|
90
|
+
run: |
|
|
91
|
+
set -euo pipefail
|
|
92
|
+
{
|
|
93
|
+
echo "The daily kagent contract check failed: $RUN_URL"
|
|
94
|
+
echo
|
|
95
|
+
if [ -f report.md ]; then
|
|
96
|
+
cat report.md
|
|
97
|
+
else
|
|
98
|
+
echo "The check failed before it wrote a report (for example in the self test). See the run log."
|
|
99
|
+
echo
|
|
100
|
+
echo "<!-- kagent-contract-report: none -->"
|
|
101
|
+
fi
|
|
102
|
+
} > body.md
|
|
103
|
+
# The report appends its marker last, so the last match is the real one.
|
|
104
|
+
marker="$(grep -o 'kagent-contract-report: [0-9a-f]*' body.md | tail -n 1)"
|
|
105
|
+
|
|
106
|
+
number="$(gh issue list --label "$ISSUE_LABEL" --state open --json number --jq '.[0].number // empty')"
|
|
107
|
+
if [ -z "$number" ]; then
|
|
108
|
+
gh label create "$ISSUE_LABEL" --color B60205 --force \
|
|
109
|
+
--description "The daily check of kagent's telemetry registry failed"
|
|
110
|
+
gh issue create --title "kagent telemetry contract check failed" \
|
|
111
|
+
--label "$ISSUE_LABEL" --body-file body.md
|
|
112
|
+
exit 0
|
|
113
|
+
fi
|
|
114
|
+
|
|
115
|
+
# Only comment when the report changed since the last update. Comments
|
|
116
|
+
# from people are ignored, so nobody can hide a change by quoting a marker.
|
|
117
|
+
latest="$(gh issue view "$number" --json body,comments \
|
|
118
|
+
--jq '([.comments[] | select(.author.login == "github-actions")] | last | .body) // .body')"
|
|
119
|
+
if grep -qF "$marker" <<<"$latest"; then
|
|
120
|
+
echo "Issue #$number already has this report."
|
|
121
|
+
else
|
|
122
|
+
gh issue comment "$number" --body-file body.md
|
|
123
|
+
fi
|
|
124
|
+
|
|
125
|
+
close-resolved:
|
|
126
|
+
needs: check
|
|
127
|
+
if: success()
|
|
128
|
+
runs-on: ubuntu-latest
|
|
129
|
+
timeout-minutes: 10
|
|
130
|
+
permissions:
|
|
131
|
+
issues: write
|
|
132
|
+
steps:
|
|
133
|
+
- name: Close the issue once the check passes again
|
|
134
|
+
env:
|
|
135
|
+
GH_TOKEN: ${{ github.token }}
|
|
136
|
+
GH_REPO: ${{ github.repository }}
|
|
137
|
+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
|
138
|
+
run: |
|
|
139
|
+
set -euo pipefail
|
|
140
|
+
number="$(gh issue list --label "$ISSUE_LABEL" --state open --json number --jq '.[0].number // empty')"
|
|
141
|
+
if [ -n "$number" ]; then
|
|
142
|
+
gh issue close "$number" --comment "The kagent contract check passes again: $RUN_URL"
|
|
143
|
+
fi
|
|
@@ -23,7 +23,7 @@ jobs:
|
|
|
23
23
|
fetch-depth: 0
|
|
24
24
|
|
|
25
25
|
- name: Install uv
|
|
26
|
-
uses: astral-sh/setup-uv@
|
|
26
|
+
uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
|
|
27
27
|
|
|
28
28
|
# Repo root cwd: uv build puts artifacts in ./dist; uv publish looks for dist/* relative to cwd.
|
|
29
29
|
- name: 'Release Python Packages'
|
|
@@ -22,7 +22,7 @@ jobs:
|
|
|
22
22
|
ref: ${{ github.event.inputs.tag || github.ref_name }}
|
|
23
23
|
fetch-depth: 0
|
|
24
24
|
|
|
25
|
-
- uses: astral-sh/setup-uv@
|
|
25
|
+
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
|
|
26
26
|
with:
|
|
27
27
|
enable-cache: true
|
|
28
28
|
|
|
@@ -51,7 +51,7 @@ jobs:
|
|
|
51
51
|
ref: ${{ github.event.inputs.tag || github.ref_name }}
|
|
52
52
|
fetch-depth: 0
|
|
53
53
|
|
|
54
|
-
- uses: astral-sh/setup-uv@
|
|
54
|
+
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
|
|
55
55
|
with:
|
|
56
56
|
enable-cache: true
|
|
57
57
|
|
|
@@ -112,10 +112,10 @@ jobs:
|
|
|
112
112
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
113
113
|
|
|
114
114
|
- name: Set up QEMU
|
|
115
|
-
uses: docker/setup-qemu-action@
|
|
115
|
+
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
|
|
116
116
|
|
|
117
117
|
- name: Set up Docker Buildx
|
|
118
|
-
uses: docker/setup-buildx-action@
|
|
118
|
+
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
119
119
|
|
|
120
120
|
- name: Build and push
|
|
121
121
|
run: |
|
|
@@ -150,34 +150,26 @@ docs/ # Documentation
|
|
|
150
150
|
|
|
151
151
|
## Trace Processing Architecture
|
|
152
152
|
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
### Key Modules
|
|
153
|
+
Telemetry flows through four layers. Each is plain Python with no ADK dependency except the bridge.
|
|
156
154
|
|
|
157
155
|
| Module | What it does |
|
|
158
156
|
|--------|--------------|
|
|
159
|
-
| `
|
|
160
|
-
| `
|
|
161
|
-
| `
|
|
162
|
-
| `
|
|
163
|
-
| `
|
|
164
|
-
| `
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
###
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
1. Add a new `TraceFormatExtractor` implementation in `extraction.py` with `detect()`, `find_invocation_spans()`, `find_llm_spans_in()`, `find_tool_spans_in()`, and `classify_span()`
|
|
177
|
-
2. Register it in the `_EXTRACTORS` list. Order matters here: more specific formats should come first so they get detected before the generic GenAI fallback
|
|
178
|
-
3. If the format introduces new attribute keys, add them to `trace_attrs.py`
|
|
179
|
-
4. If you need conversion logic that the shared extraction functions don't cover, add a dedicated converter module (see `genai_converter.py` for an example)
|
|
180
|
-
5. Add tests to `tests/test_extraction.py` for detection and span classification
|
|
157
|
+
| `otel/model.py`, `otel/decode.py`, `otel/encode.py` | Lossless OTLP envelopes (spans, logs, resources, scopes) decoded from protobuf, JSON or Jaeger files, and encoded back |
|
|
158
|
+
| `genai/overlay.py`, `genai/semconv.py` | Read only view of a span in current GenAI conventions, with fallbacks for deprecated keys, events, ADK and OpenLLMetry attributes |
|
|
159
|
+
| `genai/extract.py`, `genai/model.py` | Turns, logical model calls and tool calls extracted from traces (`extract_conversation`) |
|
|
160
|
+
| `genai/matching.py`, `genai/grouping.py` | Grouping traces into conversations and matching them to eval cases |
|
|
161
|
+
| `adk_bridge.py` | The only place that converts to ADK types (eval sets, built in metrics) |
|
|
162
|
+
| `otel/store.py`, `streaming/manager.py` | Live sessions: routing mechanics, limits, completion, recompute and UI updates |
|
|
163
|
+
| `genai/routing.py` | What the live store routes on: session keys, which traces and logs to keep, agentevals' own records |
|
|
164
|
+
| `evaluation_events.py` | Evaluation results as `gen_ai.evaluation.result` events |
|
|
165
|
+
|
|
166
|
+
Imports point down only: `otel/` imports nothing from `genai/`, and neither imports the runner, the evaluators or `evaluation_events.py`.
|
|
167
|
+
|
|
168
|
+
### Supporting a new producer
|
|
169
|
+
|
|
170
|
+
1. Record a few real traces and add them as fixtures with the turns you expect.
|
|
171
|
+
2. If the producer uses standard GenAI attributes, it should already work. Check with `agentevals run`.
|
|
172
|
+
3. If it uses its own attribute names, add a fallback in `genai/overlay.py`. Never change span classification for one producer; it keys on `gen_ai.operation.name`.
|
|
181
173
|
|
|
182
174
|
### Adding an SDK example
|
|
183
175
|
|
|
@@ -0,0 +1,309 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: agentevals-cli
|
|
3
|
+
Version: 0.10.0
|
|
4
|
+
Summary: Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces
|
|
5
|
+
License-File: LICENSE
|
|
6
|
+
Requires-Python: >=3.11
|
|
7
|
+
Requires-Dist: aiohttp>=3.14.3
|
|
8
|
+
Requires-Dist: authlib>=1.7.2
|
|
9
|
+
Requires-Dist: click>=8.3.3
|
|
10
|
+
Requires-Dist: fastapi>=0.115.0
|
|
11
|
+
Requires-Dist: google-adk[eval]<3,>=2.5.0
|
|
12
|
+
Requires-Dist: googleapis-common-protos>=1.72.0
|
|
13
|
+
Requires-Dist: grpcio>=1.66
|
|
14
|
+
Requires-Dist: httpx>=0.27.0
|
|
15
|
+
Requires-Dist: idna>=3.18
|
|
16
|
+
Requires-Dist: litellm>=1.85.7
|
|
17
|
+
Requires-Dist: nltk>=3.10.2
|
|
18
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.38
|
|
19
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.38
|
|
20
|
+
Requires-Dist: opentelemetry-proto>=1.36.0
|
|
21
|
+
Requires-Dist: opentelemetry-sdk>=1.38
|
|
22
|
+
Requires-Dist: pyasn1>=0.6.4
|
|
23
|
+
Requires-Dist: pygments>=2.20.0
|
|
24
|
+
Requires-Dist: python-dotenv>=1.2.2
|
|
25
|
+
Requires-Dist: python-multipart>=0.0.32
|
|
26
|
+
Requires-Dist: pyyaml>=6.0
|
|
27
|
+
Requires-Dist: requests>=2.34.2
|
|
28
|
+
Requires-Dist: tabulate>=0.9.0
|
|
29
|
+
Requires-Dist: urllib3>=2.7.0
|
|
30
|
+
Requires-Dist: uvicorn[standard]>=0.32.0
|
|
31
|
+
Provides-Extra: kubernetes
|
|
32
|
+
Requires-Dist: kubernetes>=36.0.0; extra == 'kubernetes'
|
|
33
|
+
Provides-Extra: live
|
|
34
|
+
Requires-Dist: httpx>=0.27.0; extra == 'live'
|
|
35
|
+
Requires-Dist: mcp<2,>=1.28.1; extra == 'live'
|
|
36
|
+
Requires-Dist: pydantic-settings>=2.15.0; extra == 'live'
|
|
37
|
+
Requires-Dist: pyjwt>=2.13.0; extra == 'live'
|
|
38
|
+
Provides-Extra: postgres
|
|
39
|
+
Requires-Dist: asyncpg>=0.30.0; extra == 'postgres'
|
|
40
|
+
Provides-Extra: streaming
|
|
41
|
+
Description-Content-Type: text/markdown
|
|
42
|
+
|
|
43
|
+
<p align="center">
|
|
44
|
+
<picture>
|
|
45
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/agentevals-dev/agentevals/main/docs/assets/logo-color-on-transparent.svg">
|
|
46
|
+
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/agentevals-dev/agentevals/main/docs/assets/logo-dark-on-transparent.svg">
|
|
47
|
+
<img src="https://raw.githubusercontent.com/agentevals-dev/agentevals/main/docs/assets/logo-color-on-transparent.svg" alt="agentevals" width="420" />
|
|
48
|
+
</picture>
|
|
49
|
+
</p>
|
|
50
|
+
|
|
51
|
+
<h1 align="center">OpenTelemetry-native agent evaluation</h1>
|
|
52
|
+
|
|
53
|
+
<p align="center">
|
|
54
|
+
Send OTLP traces. Score what your agent did. Get the results back as OpenTelemetry GenAI eval events.<br>
|
|
55
|
+
No SDK, no reruns, no platform to host.
|
|
56
|
+
</p>
|
|
57
|
+
|
|
58
|
+
<p align="center">
|
|
59
|
+
<a href="https://github.com/agentevals-dev/agentevals/stargazers"><img src="https://img.shields.io/github/stars/agentevals-dev/agentevals?style=social" alt="GitHub Stars"></a>
|
|
60
|
+
|
|
61
|
+
<a href="https://discord.gg/cpveEn8Ah2"><img src="https://img.shields.io/discord/1435836734666707190?label=Discord&logo=discord&logoColor=white&color=5865F2" alt="Discord"></a>
|
|
62
|
+
|
|
63
|
+
<a href="https://github.com/agentevals-dev/agentevals/releases"><img src="https://img.shields.io/github/v/release/agentevals-dev/agentevals?label=Release" alt="Release"></a>
|
|
64
|
+
|
|
65
|
+
<a href="https://github.com/agentevals-dev/agentevals/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-green.svg" alt="License"></a>
|
|
66
|
+
|
|
67
|
+
<a href="https://pypi.org/project/agentevals-cli/"><img src="https://img.shields.io/pypi/v/agentevals-cli?label=PyPI&color=blue" alt="PyPI"></a>
|
|
68
|
+
</p>
|
|
69
|
+
|
|
70
|
+
<p align="center">
|
|
71
|
+
<a href="#install">Install</a> · <a href="#quick-start">Quick start</a> · <a href="#docs">Docs</a> · <a href="CONTRIBUTING.md">Contributing</a> · <a href="https://discord.gg/cpveEn8Ah2">Discord</a>
|
|
72
|
+
</p>
|
|
73
|
+
|
|
74
|
+
---
|
|
75
|
+
|
|
76
|
+
## What is agentevals?
|
|
77
|
+
|
|
78
|
+
agentevals scores AI agent behavior from the OpenTelemetry telemetry your agent already produces. It is an OTLP destination like any tracing backend: point an exporter or a Collector at it, and it turns GenAI spans and log events into turns, tool calls, tokens and answers, then scores them against a golden eval set. Results can flow back into your pipeline as `gen_ai.evaluation.result` events.
|
|
79
|
+
|
|
80
|
+
* **Standard telemetry in.** OTLP over HTTP and gRPC, live or from files. Any producer that follows the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) works: Google ADK, Strands, LangChain, OpenAI Agents SDK, Pydantic AI and the official OpenTelemetry GenAI instrumentations.
|
|
81
|
+
* **Standard events out.** One `gen_ai.evaluation.result` log event per metric and turn, parented to the evaluated span, ready for your backend to query, alert on or turn into metrics.
|
|
82
|
+
* **No reruns.** Scoring reads recorded telemetry, so an evaluation costs no agent tokens and can be repeated on the same run.
|
|
83
|
+
* **Gates and judges.** Tool trajectory and response matching for deterministic pass/fail, LLM judges and rubrics for quality, custom evaluators in any language.
|
|
84
|
+
* **Local first.** One `pip install`. CLI for CI, web UI for inspection, MCP server for assistants, Helm chart for Kubernetes.
|
|
85
|
+
|
|
86
|
+
```mermaid
|
|
87
|
+
flowchart LR
|
|
88
|
+
A[Agent with OTel instrumentation] -->|OTLP| C[OpenTelemetry Collector]
|
|
89
|
+
C --> B[Your tracing backend]
|
|
90
|
+
C --> E[agentevals]
|
|
91
|
+
E -->|gen_ai.evaluation.result| C
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The Collector is optional. An agent can export straight to agentevals.
|
|
95
|
+
|
|
96
|
+
> [!IMPORTANT]
|
|
97
|
+
> This project is under active development. Expect breaking changes.
|
|
98
|
+
|
|
99
|
+
## Install
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
pip install agentevals-cli # CLI, REST API, OTLP receivers, embedded web UI
|
|
103
|
+
pip install "agentevals-cli[live]" # adds the MCP server
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
From a clone: `uv sync`, then use `uv run agentevals`. See [DEVELOPMENT.md](DEVELOPMENT.md).
|
|
107
|
+
|
|
108
|
+
## Quick start
|
|
109
|
+
|
|
110
|
+
### Score recorded traces
|
|
111
|
+
|
|
112
|
+
`samples/` holds traces from a Kubernetes Helm agent and eval sets that describe the expected behavior.
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
agentevals run samples/helm.json \
|
|
116
|
+
--eval-set samples/eval_set_helm.json \
|
|
117
|
+
-m tool_trajectory_avg_score
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The eval set expects a call to `helm_list_releases`, and the trace has one:
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
Trace: 3e289017fe03ffd7c4145316d2eb3d0d
|
|
124
|
+
Invocations: 1
|
|
125
|
+
Metric Score Status Per-Invocation Time
|
|
126
|
+
------ ------------------------- ------- -------- ---------------- ------
|
|
127
|
+
[PASS] tool_trajectory_avg_score 1 PASSED 1 5ms
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
A trace from a session that never called the tool fails, and the output says why:
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
agentevals run samples/k8s.json \
|
|
134
|
+
--eval-set samples/eval_set_helm.json \
|
|
135
|
+
-m tool_trajectory_avg_score
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
[FAIL] tool_trajectory_avg_score 0 FAILED 0 1ms
|
|
140
|
+
Invocation 1 trajectory mismatch:
|
|
141
|
+
Expected:
|
|
142
|
+
- helm_list_releases({})
|
|
143
|
+
Actual:
|
|
144
|
+
(none)
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Add `-m response_match_score` to check the final answer as well, `--output json` for machine readable results, and `--emit-otel` to send the results into your pipeline.
|
|
148
|
+
|
|
149
|
+
### Evaluate a live agent
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
# Terminal 1
|
|
153
|
+
agentevals serve --dev
|
|
154
|
+
|
|
155
|
+
# Terminal 2
|
|
156
|
+
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
|
|
157
|
+
export OTEL_RESOURCE_ATTRIBUTES="agentevals.session_name=my-agent,service.instance.id=$(uuidgen)"
|
|
158
|
+
python your_agent.py
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
Sessions appear in the UI at `http://localhost:8001` as the agent runs, with tool calls, inputs and outputs, and can be evaluated there. For gRPC exporters use port 4317 with `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`.
|
|
162
|
+
|
|
163
|
+
Sessions are grouped by `agentevals.session_name`, then `gen_ai.conversation.id`, then `session.id`. `service.instance.id` tells a rerun from the next turn. Set `agentevals.eval_set_id` to associate a session with an eval set.
|
|
164
|
+
|
|
165
|
+
Per producer settings, such as turning on message content capture, are in [OpenTelemetry Compatibility](docs/otel-compatibility.md). Working setups for each framework are in [examples/zero-code-examples/](examples/zero-code-examples/).
|
|
166
|
+
|
|
167
|
+
### Send results back to your pipeline
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
agentevals run trace.json --eval-set eval_set.json -m tool_trajectory_avg_score --emit-otel
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
The exporter is configured by the standard `OTEL_EXPORTER_OTLP_*` variables. `agentevals serve --emit-otel` does the same for evaluations started from the UI and the API. Collector recipes for forwarding telemetry to agentevals and for routing the results are in [OpenTelemetry Pipelines](docs/opentelemetry-pipeline.md).
|
|
174
|
+
|
|
175
|
+
## What agentevals reads
|
|
176
|
+
|
|
177
|
+
Turns come from `invoke_agent` spans, tool calls from `execute_tool` spans, tokens and the model from model call spans, and messages from `gen_ai.input.messages` and `gen_ai.output.messages` as span attributes, span events or log events. Google ADK's `gcp.vertex.agent.*` attributes are read as well. Files can be OTLP JSON, Jaeger JSON or Collector `file` exporter output. Details, including what is ignored and the receiver limits: [OpenTelemetry Compatibility](docs/otel-compatibility.md).
|
|
178
|
+
|
|
179
|
+
## SDK (optional)
|
|
180
|
+
|
|
181
|
+
The Python SDK wraps the OTel boilerplate in a session when you want the session name, eval set and run id set from code:
|
|
182
|
+
|
|
183
|
+
```python
|
|
184
|
+
from agentevals import AgentEvals
|
|
185
|
+
|
|
186
|
+
app = AgentEvals()
|
|
187
|
+
|
|
188
|
+
with app.session(eval_set_id="my-eval"):
|
|
189
|
+
agent.invoke("Roll a 20-sided die for me")
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
Spans and GenAI log events produced inside the session go to agentevals over OTLP. Your own exporters are untouched, and the SDK never reads `OTEL_EXPORTER_OTLP_*`, so credentials for another backend are not sent to agentevals. Endpoint: `endpoint=` or `AGENTEVALS_OTLP_ENDPOINT`, default `http://localhost:4318`. See [examples/sdk_example/](examples/sdk_example/).
|
|
193
|
+
|
|
194
|
+
## Custom evaluators
|
|
195
|
+
|
|
196
|
+
An evaluator is any program that reads JSON from stdin and writes a score to stdout. Scaffold one and reference it next to the built in metrics:
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
agentevals evaluator init my_evaluator
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
```yaml
|
|
203
|
+
evaluators:
|
|
204
|
+
- name: tool_trajectory_avg_score
|
|
205
|
+
type: builtin
|
|
206
|
+
- name: my_evaluator
|
|
207
|
+
type: code
|
|
208
|
+
path: ./evaluators/my_evaluator.py
|
|
209
|
+
threshold: 0.7
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
`type: remote` pulls community evaluators from GitHub. Protocol, SDK helpers and runtimes: [Custom Evaluators](docs/custom-evaluators.md).
|
|
213
|
+
|
|
214
|
+
## Run it
|
|
215
|
+
|
|
216
|
+
**CLI.** `agentevals run --help` lists everything: multiple traces, `--group-by trace|conversation`, `--trajectory-match-type`, judge model selection and eval config files. `agentevals evaluator list` shows built in and community evaluators.
|
|
217
|
+
|
|
218
|
+
**Web UI.** `agentevals serve` on `http://localhost:8001`. Upload traces and eval sets, pick evaluators, inspect span trees, watch live sessions. API docs at `/docs`.
|
|
219
|
+
|
|
220
|
+
**MCP server.** `agentevals mcp` exposes `list_metrics`, `evaluate_traces`, `list_sessions`, `summarize_session` and `evaluate_sessions` to MCP clients. A `.mcp.json` at the repo root lets Claude Code pick it up, and the `/eval` and `/inspect` skills in `.claude/skills/` build on it.
|
|
221
|
+
|
|
222
|
+
**Docker.**
|
|
223
|
+
|
|
224
|
+
```bash
|
|
225
|
+
docker build -t agentevals .
|
|
226
|
+
docker run -p 8001:8001 -p 4317:4317 -p 4318:4318 agentevals
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
| Port | Purpose |
|
|
230
|
+
|------|---------|
|
|
231
|
+
| 8001 | Web UI and REST API |
|
|
232
|
+
| 4317 | OTLP gRPC receiver (traces and logs) |
|
|
233
|
+
| 4318 | OTLP HTTP receiver (traces and logs) |
|
|
234
|
+
| 8080 | MCP (Streamable HTTP) |
|
|
235
|
+
|
|
236
|
+
**Helm.**
|
|
237
|
+
|
|
238
|
+
```bash
|
|
239
|
+
helm install agentevals oci://ghcr.io/agentevals-dev/agentevals/helm/agentevals
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
The chart runs in memory by default. `--set storage.backend=postgres` with a bundled or external Postgres enables the async `/api/runs` pipeline and the Run History view. That storage layer is a preview and its schema may change between minor versions. See [Run History](docs/run-history.md) and the [Kubernetes example](examples/kubernetes/README.md) for a full walkthrough with kagent and an OpenTelemetry Collector.
|
|
243
|
+
|
|
244
|
+
## Examples
|
|
245
|
+
|
|
246
|
+
| Example | What it shows |
|
|
247
|
+
|---------|---------------|
|
|
248
|
+
| [ADK](examples/zero-code-examples/adk/) | Google ADK agent, standard OTLP export |
|
|
249
|
+
| [LangChain](examples/zero-code-examples/langchain/) | LangChain agent, OTLP traces and logs |
|
|
250
|
+
| [Strands](examples/zero-code-examples/strands/) | Strands agent, standard OTLP export |
|
|
251
|
+
| [OpenAI Agents](examples/zero-code-examples/openai-agents/) | OpenAI Agents SDK, standard OTLP export |
|
|
252
|
+
| [Pydantic AI](examples/zero-code-examples/pydantic-ai/) | Pydantic AI agent, standard OTLP export |
|
|
253
|
+
| [Ollama](examples/zero-code-examples/ollama/) | LangChain with a local model |
|
|
254
|
+
| [SDK sessions](examples/README.md#sdk-integration) | LangChain, Strands and ADK agents through `AgentEvals` sessions |
|
|
255
|
+
| [Kubernetes](examples/kubernetes/) | agentevals with kagent and an OpenTelemetry Collector |
|
|
256
|
+
|
|
257
|
+
## Docs
|
|
258
|
+
|
|
259
|
+
| Guide | Description |
|
|
260
|
+
|-------|-------------|
|
|
261
|
+
| [OpenTelemetry Compatibility](docs/otel-compatibility.md) | What to send, per framework setup, sessions, receiver limits |
|
|
262
|
+
| [OpenTelemetry Pipelines](docs/opentelemetry-pipeline.md) | Collector recipes and evaluation results as OpenTelemetry events |
|
|
263
|
+
| [Eval Set Format](docs/eval-set-format.md) | Schema and examples for golden eval sets |
|
|
264
|
+
| [Custom Evaluators](docs/custom-evaluators.md) | Write scoring logic in any language |
|
|
265
|
+
| [Run History](docs/run-history.md) | Persist evaluations to Postgres and trend them in the UI |
|
|
266
|
+
|
|
267
|
+
## Development
|
|
268
|
+
|
|
269
|
+
```bash
|
|
270
|
+
uv run pytest # tests
|
|
271
|
+
uv run agentevals serve --dev # backend
|
|
272
|
+
cd ui && npm run dev # frontend, separate terminal
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
See [DEVELOPMENT.md](DEVELOPMENT.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
276
|
+
|
|
277
|
+
## FAQ
|
|
278
|
+
|
|
279
|
+
**Do I need a database or any infrastructure?**
|
|
280
|
+
|
|
281
|
+
No. The CLI evaluates trace files from disk. The server keeps live sessions in memory. Postgres is optional and only backs run history.
|
|
282
|
+
|
|
283
|
+
**Does the CLI need a running server?**
|
|
284
|
+
|
|
285
|
+
No. `agentevals run` works offline. The server is for the web UI, live OTLP sessions and the MCP tools that read sessions.
|
|
286
|
+
|
|
287
|
+
**Can I use it in CI?**
|
|
288
|
+
|
|
289
|
+
Yes. Pass trace files and an eval set, set a threshold, and let the exit code gate the deployment. `--output json` gives machine readable results.
|
|
290
|
+
|
|
291
|
+
**Can I plug it into an existing OpenTelemetry pipeline?**
|
|
292
|
+
|
|
293
|
+
Yes. agentevals accepts OTLP over HTTP (`http/protobuf` and `http/json`, gzip included) and gRPC, so a default Collector exporter works. With `--emit-otel` it sends evaluation results back as `gen_ai.evaluation.result` events. See [OpenTelemetry Pipelines](docs/opentelemetry-pipeline.md).
|
|
294
|
+
|
|
295
|
+
**Can I evaluate Claude Code, Codex or OpenCode?**
|
|
296
|
+
|
|
297
|
+
Partly. They do not emit GenAI semantic convention spans, so agentevals sees a turn only when something wraps it in one, as kagent does for its Claude Code harness. A Collector recipe can then map Claude Code's own spans to tool and model calls, see the [Kubernetes example](examples/kubernetes/README.md#claude-code-workaround). agentevals is built for instrumented agents whose success is measurable through tool trajectories and responses, not for scoring long coding sessions end to end.
|
|
298
|
+
|
|
299
|
+
**How is this different from LangSmith, Langfuse, Opik or Bedrock AgentCore evaluation?**
|
|
300
|
+
|
|
301
|
+
Those run your agent again against a dataset, or need their own SDK and storage before anything can be scored. agentevals scores the OpenTelemetry telemetry you already have, from any instrumentation, without rerunning the agent and without a platform to host.
|
|
302
|
+
|
|
303
|
+
**How does this compare to ADK's evaluations?**
|
|
304
|
+
|
|
305
|
+
ADK eval runs the agent and scores it in one step. agentevals only scores, from telemetry, and uses ADK's metrics and eval set format to do it. Point an ADK agent at agentevals and you get live inspection and scoring without changing the agent.
|
|
306
|
+
|
|
307
|
+
## Acknowledgements
|
|
308
|
+
|
|
309
|
+
The built in metrics and the eval set format come from [Google's Agent Development Kit](https://github.com/google/adk-python), licensed under [Apache 2.0](https://github.com/google/adk-python/blob/main/LICENSE) like agentevals. Thanks to the ADK team and contributors.
|