gooddata-eval 1.74.1.dev1__tar.gz → 1.74.1.dev3__tar.gz

gooddata_eval-1.74.1.dev3/AGENTS.md

@@ -0,0 +1,114 @@
+# gooddata-eval
+
+`gd-eval` — a CLI and library that drives the GoodData AI agent (a separate service, in
+`gdc-nas`) through a dataset of natural-language questions and scores what comes back,
+including side-by-side comparison across models. Each dataset item is a JSON envelope
+loaded from a local folder or pulled from a Langfuse dataset. Results are aggregated into
+pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to an
+experiment. The newest and most actively developed package in the repo.
+
+## Owns
+
+- The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
+- Dataset loading and the evaluation run loop
+- Per-capability evaluators and their scoring
+- Result reporting, and pushing experiments, scores and trace links to Langfuse
+
+## Does NOT Own
+
+- The agent under evaluation — that lives in `gdc-nas` (gen-ai)
+- Platform access → `gooddata-sdk`
+
+## Architecture
+
+| Path | Role |
+|---|---|
+| `cli/` | argument parsing, and `agentic_runner` — the agentic dispatch and concurrency phases |
+| `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
+| `core/chat/` | SSE client for the agent's streaming chat endpoint |
+| `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
+| `core/dataset/` | dataset format and loading |
+| `core/evaluators/` | single-shot evaluators and their registry |
+| `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
+| `core/reporting/` | console and JSON output rendering |
+| `core/scoring.py`, `core/runner.py` | scoring and orchestration |
+| `core/models.py` | `DatasetItem`, `ChatResult`, `ItemReport` and friends |
+
+**Depends on**: `gooddata-sdk`, `httpx`, `pydantic`, `orjson`, `rich`. The LLM-judge
+evaluator is an optional extra (`llm-judge`, pulling `openai>=1.45,<2.0`); every `openai`
+import site is guarded or deferred so the base install stays usable without it — keep it
+that way.
+
+### Two evaluation paths that share almost nothing
+
+- **Single-shot** kinds send one chat turn and are scored by an `Evaluator` (a Protocol:
+  a `test_kind` attribute plus `evaluate(item, chat_result) -> ItemEvaluation`) looked up
+  from a registry in `core/evaluators/__init__.py`.
+- **Agentic** kinds (`agentic_*`, `vis_agentic`) drive a full multi-turn conversation over
+  the SSE endpoint and are dispatched by an explicit `if`/`elif` chain in
+  `cli/agentic_runner.py`.
+
+Many capabilities exist in **both** forms — visualization, metric skill, alert skill,
+search, general question and guardrail each have a single-turn and a multi-turn
+implementation, sometimes under different `test_kind` strings (`search_tool` vs
+`agentic_search`). These are parallel implementations, not layers.
+
+### Dataset items
+
+`DatasetItem` is the envelope: `id`, `dataset_name`, `test_kind`, `question`, and
+`expected_output: Any`. `expected_output` is deliberately untyped — each evaluator parses
+its own shape. `test_kind` on the item is what labels the result, not the evaluator class,
+which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
+`dashboard_summary` items additionally need `summary_input`.
+
+## Gotchas
+
+**Adding an evaluator is a registry change, not a naming convention.** Single-shot kinds go
+into `_EAGER_EVALUATORS`, or `_LAZY_EVALUATOR_MODULES` plus `_LAZY_EVALUATOR_CLASSES`, in
+`core/evaluators/__init__.py`. Agentic kinds need the string added to `AGENTIC_TEST_KINDS`
+and a new branch in `_dispatch_agentic`. Test file naming follows the capability, but
+naming a test file correctly registers nothing.
+
+**Parallel-safety is a reviewed allowlist, and getting it wrong corrupts results.**
+`WORKSPACE_MUTATING_TEST_KINDS` is computed as `AGENTIC_TEST_KINDS - PARALLEL_SAFE_TEST_KINDS`,
+so a newly added kind defaults to workspace-mutating and runs serially in its own phase.
+That default is correct: agent tool calls create real server-side objects (metrics, alerts).
+Adding a kind to `PARALLEL_SAFE_TEST_KINDS` is a deliberate assertion that it is read-only,
+which nothing in the package can prove for you.
+
+**The SSE client's retry predicate is load-bearing.** `core/chat/sse_client.py` retries
+429/502/503/504 and `httpx.RemoteProtocolError` (a mid-stream disconnect) with exponential
+backoff, and treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The
+`RemoteProtocolError` case was added after it was confirmed live to contaminate a small
+percentage of visualization runs with a hard fail and no retry. Narrowing that predicate
+reintroduces the problem.
+
+**Langfuse trace linking is deliberately off the item critical path.** Polling for trace
+ingestion has no pass/fail signal and inflates or misattributes per-item latency, so
+`BackgroundTraceLinker` defers it and is drained before the report renders
+(`run_trace_link_inline` is the synchronous alternative). Do not "fix" a slow item by
+making trace scoring synchronous again.
+
+**Scoring weights do not sum to 1.** `quality_score` is the fraction of boolean-valued keys
+in `best_detail` that are true, falling back to `pass_at_k` when there are none (text
+evaluators). `value_score` is `0.6 * quality + 0.2 * speed` — the 0.8 total is what the
+code does; treat it as intentional unless you have checked with the owner.
+
+### Fixture shapes
+
+Group-by / attribute expectations in the alert-skill fixtures are written in AAC shape,
+while the tool arguments the agent emits are AFM-shaped. Never deep-compare those two
+directly — convert, or compare field by field. This applies specifically to the
+attribute/group-by fields: `Filters` in the same fixtures is AFM-shaped on both sides and
+is correctly deep-compared as-is. The attribute comparison itself lands with the
+alert group-by work currently on `jt/gdai-2175-eval-alert-attributes`, so on `master` this
+is guidance for the incoming code rather than a description of what is already there.
+
+## Testing
+
+Plain pytest under `tests/`, no cassettes — the agent is stubbed with
+`unittest.mock`. Tests are named per capability (`test_agentic_*.py`), which is the
+convention to follow when adding one.
+
+`ty` is configured here with `allowed-unresolved-imports` for `openai.**` and
+`gooddata_api_client.**`; do not widen that list to paper over a real typing problem.

gooddata_eval-1.74.1.dev3/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.74.1.dev1
+Version: 1.74.1.dev3
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,7 +17,7 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.74.1.dev1
+Requires-Dist: gooddata-sdk~=1.74.1.dev3
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: orjson<4.0.0,>=3.9.15
 Requires-Dist: pydantic<3.0,>=2.6
@@ -141,7 +141,7 @@ gd-eval run \
 | Flag | Description |
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
-| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 | `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
@@ -154,7 +154,8 @@ gd-eval run \
 
 | Flag | Default | Description |
 |---|---|---|
-| `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
+| `--runs K` | `2` | Independent runs per item. |
+| `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
 | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
@@ -180,15 +181,39 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 
-Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+##### Langfuse v4
 
-**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
-because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
-flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
-exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
-fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+One run is one Langfuse **experiment**. Each evaluated item becomes its own trace whose root span carries the
+experiment and dataset-item attributes (`langfuse.experiment.name`, `langfuse.experiment.dataset.id`,
+`langfuse.experiment.item.id`), and the four scores attach to that root observation. gd-eval speaks to Langfuse
+over four REST endpoints and uses no Langfuse SDK, so it runs on every Python version the package supports:
+
+| Endpoint | Used for |
+|---|---|
+| `POST /api/public/otel/v1/traces` | exporting the experiment root span as OTLP/HTTP JSON |
+| `POST /api/public/scores` | one score per write, on a trace or on a single observation inside it |
+| `GET /api/public/v2/observations` | finding the agent's gen-ai trace for a conversation |
+| `GET /api/public/dataset-items` | loading `--langfuse-dataset` items, and resolving an item's dataset id |
+
+Two consequences of v4's immutable observations. gd-eval sets `version` only on its own experiment span, never on
+the agent's gen-ai trace — filter on the gd-eval experiment's `langfuse.version` to compare models. And on the
+agentic kinds the latency in `value_score` is the gen-ai trace's root generation latency, read from the
+observations endpoint; the single-shot `--langfuse` sink keeps using the item's own measured average latency.
+
+Langfuse Cloud drops v3 on **2026-11-16**; a self-hosted Langfuse must be on v4 for any of this to work.
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to turn the whole **agentic** Langfuse write path off — no trace lookup, no
+span export and no scores for `agentic_*` items. The run says so once. It does not reach the `--langfuse` sink,
+which still writes a span and four scores for every single-shot item; drop `--langfuse` to silence that too.
+
+**A local `--dataset` cannot be attached to a Langfuse experiment.** `--langfuse` is refused alongside
+`--dataset` because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not
+depend on that flag — each `evaluate_agentic_*` builds its own client whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported — so a local run still finds its traces and writes its
+scores onto them, and only the per-run grouping fails: the dataset-item lookup 404s and the run reports the item
+as one that does not exist in Langfuse, once. The run also warns about this before it starts. Use
 `--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
 skip linking altogether.
 
@@ -235,10 +260,12 @@ Winner is selected by **pass rate → quality score → latency** (lower latency
 Each item reports **how many of its runs passed**, not only whether one did:
 
 ```json
-"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false, "gate_passed": true
 ```
 
-`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+`pass_at_k` is "did any run pass" — always literal, whatever the gate. `gate_passed` is the verdict the item
+was decided on and is what `passed` counts; the two differ only under `--gate power`, where an item that
+passed 4 of 5 runs is `"pass_at_k": true, "gate_passed": false`. `runs_passed` is the fact that separates a
 reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
 `quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
 run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
@@ -248,6 +275,14 @@ stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on ev
 `runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
 no K and drives its fixture exactly once.
 
+Which of the two decides pass/fail is `--gate`: `any` (default) gates on `pass_at_k`, `power` gates on
+`pass_power_k`. The run records it as a top-level `gate`, and a failure under `power` says so —
+`Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure` — because the message body describes
+the best run, which under pass^K can be a run that passed, and the console repeats the count in `Notes` for
+the same reason. When some runs were ungraded the note says so instead of calling the remainder unstable.
+`agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than
+labelling a report `power` that only part of the dataset was decided under.
+
 Each item additionally carries a per-phase breakdown:
 
 ```json
@@ -408,10 +443,19 @@ Without `[llm-judge]`, those items are **skipped**.
 
 ## Scores (in JSON report and Langfuse)
 
+In Langfuse every score is written to the experiment run's root observation — `traceId` plus `observationId` of
+the item's own root span. On the agentic path each score is mirrored onto the agent's gen-ai trace as well
+(`traceId` only), so a score survives even when one of the two traces is missing.
+
 | Score | Description |
 |---|---|
-| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |
+| `pass_at_k` | 1 if **any** of the K runs passed strict checks, else 0. |
+| `pass_power_k` | 1 only if **every** one of the K runs passed. Agentic kinds only. |
+| `gate_passed` | The verdict that decided the item: `pass_at_k` under `--gate any`, `pass_power_k` under `--gate power`. Agentic kinds only. |
 | `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. |
 | `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (speed = max(0, 1 − latency/60s)). |
 | `latency_s` | Average per-run latency in seconds. |
 | `provider_type` | Model vendor + gateway label (e.g. `ANTHROPIC`, `BEDROCK/ANTHROPIC`, `AZURE/OPENAI`). Stored in Langfuse trace metadata and tags. |
+
+Score names carry no K; K and the gate are on the dataset-run metadata as `eval_k` and `eval_gate`.
+`agentic_visualization` also still writes its historic `pass_at_{K}` / `pass_power_{K}` pair.

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev3}/README.md

@@ -113,7 +113,7 @@ gd-eval run \
 | Flag | Description |
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
-| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 | `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
@@ -126,7 +126,8 @@ gd-eval run \
 
 | Flag | Default | Description |
 |---|---|---|
-| `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
+| `--runs K` | `2` | Independent runs per item. |
+| `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
 | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
@@ -152,15 +153,39 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 
-Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+##### Langfuse v4
 
-**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
-because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
-flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
-exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
-fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+One run is one Langfuse **experiment**. Each evaluated item becomes its own trace whose root span carries the
+experiment and dataset-item attributes (`langfuse.experiment.name`, `langfuse.experiment.dataset.id`,
+`langfuse.experiment.item.id`), and the four scores attach to that root observation. gd-eval speaks to Langfuse
+over four REST endpoints and uses no Langfuse SDK, so it runs on every Python version the package supports:
+
+| Endpoint | Used for |
+|---|---|
+| `POST /api/public/otel/v1/traces` | exporting the experiment root span as OTLP/HTTP JSON |
+| `POST /api/public/scores` | one score per write, on a trace or on a single observation inside it |
+| `GET /api/public/v2/observations` | finding the agent's gen-ai trace for a conversation |
+| `GET /api/public/dataset-items` | loading `--langfuse-dataset` items, and resolving an item's dataset id |
+
+Two consequences of v4's immutable observations. gd-eval sets `version` only on its own experiment span, never on
+the agent's gen-ai trace — filter on the gd-eval experiment's `langfuse.version` to compare models. And on the
+agentic kinds the latency in `value_score` is the gen-ai trace's root generation latency, read from the
+observations endpoint; the single-shot `--langfuse` sink keeps using the item's own measured average latency.
+
+Langfuse Cloud drops v3 on **2026-11-16**; a self-hosted Langfuse must be on v4 for any of this to work.
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to turn the whole **agentic** Langfuse write path off — no trace lookup, no
+span export and no scores for `agentic_*` items. The run says so once. It does not reach the `--langfuse` sink,
+which still writes a span and four scores for every single-shot item; drop `--langfuse` to silence that too.
+
+**A local `--dataset` cannot be attached to a Langfuse experiment.** `--langfuse` is refused alongside
+`--dataset` because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not
+depend on that flag — each `evaluate_agentic_*` builds its own client whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported — so a local run still finds its traces and writes its
+scores onto them, and only the per-run grouping fails: the dataset-item lookup 404s and the run reports the item
+as one that does not exist in Langfuse, once. The run also warns about this before it starts. Use
 `--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
 skip linking altogether.
 
@@ -207,10 +232,12 @@ Winner is selected by **pass rate → quality score → latency** (lower latency
 Each item reports **how many of its runs passed**, not only whether one did:
 
 ```json
-"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false, "gate_passed": true
 ```
 
-`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+`pass_at_k` is "did any run pass" — always literal, whatever the gate. `gate_passed` is the verdict the item
+was decided on and is what `passed` counts; the two differ only under `--gate power`, where an item that
+passed 4 of 5 runs is `"pass_at_k": true, "gate_passed": false`. `runs_passed` is the fact that separates a
 reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
 `quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
 run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
@@ -220,6 +247,14 @@ stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on ev
 `runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
 no K and drives its fixture exactly once.
 
+Which of the two decides pass/fail is `--gate`: `any` (default) gates on `pass_at_k`, `power` gates on
+`pass_power_k`. The run records it as a top-level `gate`, and a failure under `power` says so —
+`Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure` — because the message body describes
+the best run, which under pass^K can be a run that passed, and the console repeats the count in `Notes` for
+the same reason. When some runs were ungraded the note says so instead of calling the remainder unstable.
+`agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than
+labelling a report `power` that only part of the dataset was decided under.
+
 Each item additionally carries a per-phase breakdown:
 
 ```json
@@ -380,10 +415,19 @@ Without `[llm-judge]`, those items are **skipped**.
 
 ## Scores (in JSON report and Langfuse)
 
+In Langfuse every score is written to the experiment run's root observation — `traceId` plus `observationId` of
+the item's own root span. On the agentic path each score is mirrored onto the agent's gen-ai trace as well
+(`traceId` only), so a score survives even when one of the two traces is missing.
+
 | Score | Description |
 |---|---|
-| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |
+| `pass_at_k` | 1 if **any** of the K runs passed strict checks, else 0. |
+| `pass_power_k` | 1 only if **every** one of the K runs passed. Agentic kinds only. |
+| `gate_passed` | The verdict that decided the item: `pass_at_k` under `--gate any`, `pass_power_k` under `--gate power`. Agentic kinds only. |
 | `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. |
 | `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (speed = max(0, 1 − latency/60s)). |
 | `latency_s` | Average per-run latency in seconds. |
 | `provider_type` | Model vendor + gateway label (e.g. `ANTHROPIC`, `BEDROCK/ANTHROPIC`, `AZURE/OPENAI`). Stored in Langfuse trace metadata and tags. |
+
+Score names carry no K; K and the gate are on the dataset-run metadata as `eval_k` and `eval_gate`.
+`agentic_visualization` also still writes its historic `pass_at_{K}` / `pass_power_{K}` pair.

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev3}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.74.1.dev1"
+version = "1.74.1.dev3"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.74.1.dev1",
+    "gooddata-sdk~=1.74.1.dev3",
     "httpx>=0.27,<1.0",
     "orjson>=3.9.15,<4.0.0",
     "pydantic>=2.6,<3.0",
@@ -43,8 +43,8 @@ dev = [
     "pytest>=8.3.5",
 ]
 test = [
-    "pytest~=8.3.4",
-    "pytest-cov~=6.0.0",
+    "pytest~=9.1.1",
+    "pytest-cov~=7.1.0",
     "pytest-json-report==1.5.0",
     "pytest-mock>=3.14.0",
 ]

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev3}/src/gooddata_eval/cli/agentic_runner.py

@@ -8,6 +8,7 @@ import time
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, TypedDict
 
+from gooddata_eval.core.agentic._gate import DEFAULT_GATE, EvalGate, normalize_gate
 from gooddata_eval.core.agentic._langfuse import make_langfuse_client
 from gooddata_eval.core.agentic._trace_linker import BackgroundTraceLinker, SubmitTraceLink, run_trace_link_inline
 from gooddata_eval.core.agentic.alert_skill import evaluate_agentic_alert_skill
@@ -48,6 +49,13 @@ AGENTIC_TEST_KINDS = frozenset(
 )
 
 
+# Agentic kinds that no gate applies to: they drive their fixture exactly once, so there is
+# no K to take pass@K or pass^K over. Named here rather than inline in _dispatch_agentic so
+# the CLI can refuse --gate power for a dataset containing one instead of labelling the whole
+# report `power` when part of it was never gated.
+UNGATED_AGENTIC_TEST_KINDS = frozenset({"agentic_conversation"})
+
+
 # Kinds cleared to run several at a time. An EXPLICIT allowlist, not a subtraction: nothing
 # in this package can prove a kind is read-only, because the mutation happens server-side in
 # whichever tools the agent decides to call. So each entry here is a reviewed judgement, and
@@ -128,9 +136,13 @@ def _dispatch_agentic(
     reasoning_effort: ReasoningEffort | None = None,
     agent_id: str | None = None,
     submit_trace_link: SubmitTraceLink = run_trace_link_inline,
+    gate: EvalGate = DEFAULT_GATE,
 ) -> AgenticEvalOutcome:
     """Call the appropriate evaluate_agentic_* function for the item's test_kind.
 
+    `gate` reaches every kind except those in UNGATED_AGENTIC_TEST_KINDS, which have no K
+    to gate over; the CLI refuses --gate power for a dataset containing one.
+
     Every evaluate_agentic_* function returns an AgenticEvalOutcome (reasoning_steps,
     conversation_id, response_id, detail) on success and attaches the same four attributes
     to its raised *AssertionError on failure -- no kind is exempt.
@@ -155,6 +167,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_outputs=_parse_visualization_expected(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -166,6 +179,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, (dict, list)) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -177,6 +191,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, dict) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -191,6 +206,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_tool_call=expected_args,
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -202,6 +218,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             user_context=item.user_context,
             **lf_kw,
@@ -214,6 +231,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -225,6 +243,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, dict) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -291,6 +310,7 @@ def run_agentic_items(
     on_item_done: Any = None,
     agent_id: str | None = None,
     concurrency: int = 1,
+    gate: EvalGate = DEFAULT_GATE,
 ) -> EvalReport:
     """Run agentic items through evaluate_agentic_* and return an EvalReport.
 
@@ -303,7 +323,7 @@ def run_agentic_items(
     """
     langfuse = make_langfuse_client() if use_langfuse else None
 
-    report = EvalReport(model=model_version)
+    report = EvalReport(model=model_version, gate=normalize_gate(gate))
     total = len(items)
     # Trace linking runs here rather than inside each evaluate_agentic_*, so an item's
     # Langfuse poll overlaps the NEXT item's agent call instead of extending its own
@@ -325,6 +345,9 @@ def run_agentic_items(
             test_kind=item.test_kind,
             question=item.question,
         )
+        # None, not False, for the kinds _dispatch_agentic passes no gate to: ItemReport.passed
+        # then falls back to pass_at_k and gate_passed keeps meaning "a gate ran".
+        gated = item.test_kind not in UNGATED_AGENTIC_TEST_KINDS
         t0 = time.perf_counter()
         try:
             outcome = _dispatch_agentic(
@@ -339,6 +362,7 @@ def run_agentic_items(
                 reasoning_effort,
                 agent_id,
                 submit_trace_link=linker.submit,
+                gate=gate,
             )
             if isinstance(outcome, AgenticEvalOutcome):
                 reasoning_steps = outcome.reasoning_steps
@@ -347,6 +371,8 @@ def run_agentic_items(
                 detail = outcome.detail
             else:
                 reasoning_steps, conversation_id, response_id, detail = outcome, None, None, {}
+            item_report.gate_passed = True if gated else None
+            # Whichever gate decided the item, clearing it means at least one run passed.
             item_report.pass_at_k = True
             item_report.runs = k
             item_report.reasoning_steps = reasoning_steps or []
@@ -356,7 +382,7 @@ def run_agentic_items(
             _apply_timings(item_report, getattr(outcome, "timings", None))
             _apply_run_counts(item_report, outcome)
         except AssertionError as exc:
-            item_report.pass_at_k = False
+            item_report.gate_passed = False if gated else None
             item_report.runs = k
             item_report.reasoning_steps = getattr(exc, "reasoning_steps", None) or []
             item_report.conversation_id = getattr(exc, "conversation_id", None)
@@ -364,6 +390,10 @@ def run_agentic_items(
             item_report.best_detail = getattr(exc, "detail", None) or {}
             _apply_timings(item_report, getattr(exc, "timings", None))
             _apply_run_counts(item_report, exc)
+            # Read off the counts, not off the gate: pass^K fails items where runs did pass,
+            # and reporting those as pass_at_k False would contradict the Langfuse score of
+            # the same name. Kinds that report no count read as 0, i.e. a clean failure.
+            item_report.pass_at_k = item_report.runs_passed > 0
             print(f"[agentic] {item.id} FAIL: {exc}", flush=True)
         except Exception as exc:
             item_report.error = f"{type(exc).__name__}: {exc}"