gooddata-eval 1.74.0__tar.gz → 1.74.1.dev2__tar.gz

gooddata_eval-1.74.1.dev2/AGENTS.md

@@ -0,0 +1,114 @@
+# gooddata-eval
+
+`gd-eval` — a CLI and library that drives the GoodData AI agent (a separate service, in
+`gdc-nas`) through a dataset of natural-language questions and scores what comes back,
+including side-by-side comparison across models. Each dataset item is a JSON envelope
+loaded from a local folder or pulled from a Langfuse dataset. Results are aggregated into
+pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to a
+dataset run. The newest and most actively developed package in the repo.
+
+## Owns
+
+- The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
+- Dataset loading and the evaluation run loop
+- Per-capability evaluators and their scoring
+- Result reporting, and pushing runs, scores and trace links to Langfuse
+
+## Does NOT Own
+
+- The agent under evaluation — that lives in `gdc-nas` (gen-ai)
+- Platform access → `gooddata-sdk`
+
+## Architecture
+
+| Path | Role |
+|---|---|
+| `cli/` | argument parsing, and `agentic_runner` — the agentic dispatch and concurrency phases |
+| `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
+| `core/chat/` | SSE client for the agent's streaming chat endpoint |
+| `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
+| `core/dataset/` | dataset format and loading |
+| `core/evaluators/` | single-shot evaluators and their registry |
+| `core/langfuse/` | `sink.py` only — pushes single-turn scores and dataset-run items |
+| `core/reporting/` | console and JSON output rendering |
+| `core/scoring.py`, `core/runner.py` | scoring and orchestration |
+| `core/models.py` | `DatasetItem`, `ChatResult`, `ItemReport` and friends |
+
+**Depends on**: `gooddata-sdk`, `httpx`, `pydantic`, `orjson`, `rich`. The LLM-judge
+evaluator is an optional extra (`llm-judge`, pulling `openai>=1.45,<2.0`); every `openai`
+import site is guarded or deferred so the base install stays usable without it — keep it
+that way.
+
+### Two evaluation paths that share almost nothing
+
+- **Single-shot** kinds send one chat turn and are scored by an `Evaluator` (a Protocol:
+  a `test_kind` attribute plus `evaluate(item, chat_result) -> ItemEvaluation`) looked up
+  from a registry in `core/evaluators/__init__.py`.
+- **Agentic** kinds (`agentic_*`, `vis_agentic`) drive a full multi-turn conversation over
+  the SSE endpoint and are dispatched by an explicit `if`/`elif` chain in
+  `cli/agentic_runner.py`.
+
+Many capabilities exist in **both** forms — visualization, metric skill, alert skill,
+search, general question and guardrail each have a single-turn and a multi-turn
+implementation, sometimes under different `test_kind` strings (`search_tool` vs
+`agentic_search`). These are parallel implementations, not layers.
+
+### Dataset items
+
+`DatasetItem` is the envelope: `id`, `dataset_name`, `test_kind`, `question`, and
+`expected_output: Any`. `expected_output` is deliberately untyped — each evaluator parses
+its own shape. `test_kind` on the item is what labels the result, not the evaluator class,
+which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
+`dashboard_summary` items additionally need `summary_input`.
+
+## Gotchas
+
+**Adding an evaluator is a registry change, not a naming convention.** Single-shot kinds go
+into `_EAGER_EVALUATORS`, or `_LAZY_EVALUATOR_MODULES` plus `_LAZY_EVALUATOR_CLASSES`, in
+`core/evaluators/__init__.py`. Agentic kinds need the string added to `AGENTIC_TEST_KINDS`
+and a new branch in `_dispatch_agentic`. Test file naming follows the capability, but
+naming a test file correctly registers nothing.
+
+**Parallel-safety is a reviewed allowlist, and getting it wrong corrupts results.**
+`WORKSPACE_MUTATING_TEST_KINDS` is computed as `AGENTIC_TEST_KINDS - PARALLEL_SAFE_TEST_KINDS`,
+so a newly added kind defaults to workspace-mutating and runs serially in its own phase.
+That default is correct: agent tool calls create real server-side objects (metrics, alerts).
+Adding a kind to `PARALLEL_SAFE_TEST_KINDS` is a deliberate assertion that it is read-only,
+which nothing in the package can prove for you.
+
+**The SSE client's retry predicate is load-bearing.** `core/chat/sse_client.py` retries
+429/502/503/504 and `httpx.RemoteProtocolError` (a mid-stream disconnect) with exponential
+backoff, and treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The
+`RemoteProtocolError` case was added after it was confirmed live to contaminate a small
+percentage of visualization runs with a hard fail and no retry. Narrowing that predicate
+reintroduces the problem.
+
+**Langfuse trace linking is deliberately off the item critical path.** Polling for trace
+ingestion has no pass/fail signal and inflates or misattributes per-item latency, so
+`BackgroundTraceLinker` defers it and is drained before the report renders
+(`run_trace_link_inline` is the synchronous alternative). Do not "fix" a slow item by
+making trace scoring synchronous again.
+
+**Scoring weights do not sum to 1.** `quality_score` is the fraction of boolean-valued keys
+in `best_detail` that are true, falling back to `pass_at_k` when there are none (text
+evaluators). `value_score` is `0.6 * quality + 0.2 * speed` — the 0.8 total is what the
+code does; treat it as intentional unless you have checked with the owner.
+
+### Fixture shapes
+
+Group-by / attribute expectations in the alert-skill fixtures are written in AAC shape,
+while the tool arguments the agent emits are AFM-shaped. Never deep-compare those two
+directly — convert, or compare field by field. This applies specifically to the
+attribute/group-by fields: `Filters` in the same fixtures is AFM-shaped on both sides and
+is correctly deep-compared as-is. The attribute comparison itself lands with the
+alert group-by work currently on `jt/gdai-2175-eval-alert-attributes`, so on `master` this
+is guidance for the incoming code rather than a description of what is already there.
+
+## Testing
+
+Plain pytest under `tests/`, no cassettes — the agent is stubbed with
+`unittest.mock`. Tests are named per capability (`test_agentic_*.py`), which is the
+convention to follow when adding one.
+
+`ty` is configured here with `allowed-unresolved-imports` for `openai.**` and
+`gooddata_api_client.**`; do not widen that list to paper over a real typing problem.

gooddata_eval-1.74.1.dev2/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.74.0
+Version: 1.74.1.dev2
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,13 +17,13 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.74.0
+Requires-Dist: gooddata-sdk~=1.74.1.dev2
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: orjson<4.0.0,>=3.9.15
 Requires-Dist: pydantic<3.0,>=2.6
 Requires-Dist: rich<15.0,>=13.0
 Provides-Extra: llm-judge
-Requires-Dist: openai<2.0,>=1.40; extra == 'llm-judge'
+Requires-Dist: openai<2.0,>=1.45; extra == 'llm-judge'
 Description-Content-Type: text/markdown
 
 # gooddata-eval
@@ -142,6 +142,7 @@ gd-eval run \
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
 | `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
 
@@ -154,21 +155,62 @@ gd-eval run \
 | Flag | Default | Description |
 |---|---|---|
 | `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
-| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests. Progress output interleaves when K > 1. |
+| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
 
+**Concurrency and workspace safety.** Agentic kinds that create workspace objects
+(`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
+time whatever `--concurrency` says — a metric or alert created and dropped mid-run would otherwise be visible to
+another item reading the same catalog. **That protection is for the agentic kinds only:** the single-turn
+`metric_skill` and `alert_skill` kinds are still fanned out and the agent performs the same server-side writes on
+that path, so avoid raising `--concurrency` on a dataset of those against a shared workspace. Progress output
+interleaves when K > 1, and per-item latencies rise, so they stop being clean single-request measurements.
+
 #### Output
 
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
+| `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
+| `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
 
 #### Langfuse sink
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Creates one named experiment run per model (`gd-eval-{timestamp}-{model}`, suffixed `-effort-{level}` when `--reasoning-effort` is set so runs differing only by effort stay separate). Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+
+**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
+because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
+flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
+exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
+fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+`--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
+skip linking altogether.
+
+**When trace linking happens.** Finding a gen-ai trace means polling until Langfuse has ingested it, which is
+lag measured in seconds to minutes. That work produces no verdict — the pass/fail is already decided — so it
+does not run inline per item. Every item's Langfuse block is queued and the whole batch runs *after* the agent
+phase, draining before any report is written. Two consequences worth knowing:
+
+- **No item's `latency_s` includes trace linking.** Its cost is reported separately as
+  `latency_breakdown_s.langfuse_s`, and the run prints
+  `[langfuse] trace linking finished in Xs for N item(s); slowest Ys`. If `slowest` approaches the **120s**
+  batched retry budget, links are timing out and scores are being orphaned — look for
+  `[langfuse] WARNING: no trace found for conversation ...`.
+- **The budget depends on who is waiting.** 120s is affordable only because the batch blocks nobody. A direct
+  library caller (`evaluate_agentic_*` without a `submit_trace_link`) polls inline, on its own critical path, and
+  gets **35s** instead — the same cost as before batching existed, so no inline caller pays for a budget raised
+  on the CLI's behalf. Either way a trace that is already ingested costs nothing: the loop looks before it sleeps.
+Scores are always final before the command exits — the run blocks on the batch. Interrupting with Ctrl-C drops
+whatever is still queued rather than making you wait it out: both the queued trace links and, under
+`--concurrency`, the items that have not started. The handful of items already in flight still have to finish —
+worker threads are joined at exit and an in-progress agent call cannot be cancelled — so expect to wait up to one
+`--concurrency`-wide wave, not the rest of the dataset.
 
 ### JSON report shape
 
@@ -190,6 +232,70 @@ The JSON report always uses the nested multi-model shape:
 
 Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
 
+Each item reports **how many of its runs passed**, not only whether one did:
+
+```json
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+```
+
+`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
+`quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
+run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
+inconsistent rather than wrong. The console shows `4/5 runs passed` in `Notes` for a non-unanimous pass and
+stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on every run`.
+
+`runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
+no K and drives its fixture exactly once.
+
+Each item additionally carries a per-phase breakdown:
+
+```json
+"latency_breakdown_s": {
+  "agent_s": 4.02,          // GoodData's own response time — the system under test
+  "judge_s": 1.31,          // LLM-as-judge scoring, post-hoc
+  "simulated_user_s": 0.0,  // our simulated user composing the next turn (multi-turn kinds)
+  "langfuse_s": 5.70        // trace lookup + score writing, off the critical path
+}
+```
+
+Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned
+nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail`, and a
+`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a
+criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0
+would be indistinguishable from the judge genuinely failing the answer, which is the confusion
+`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with
+`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For
+`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass —
+"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows
+the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the
+item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than
+`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`,
+or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes`
+(`1 run(s) ungraded`, `2 criterion(s) ungraded`).
+
+`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add
+up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation
+create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never
+inside it, because trace linking runs outside every item's critical path (see above) — summing all four would
+re-inflate exactly what that design removes.
+
+A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not
+applicable here", not "instant". Today:
+
+| Field | Populated by |
+|---|---|
+| `agent_s` | `agentic_general_question`, `agentic_metric_skill` |
+| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge |
+| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user |
+| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present |
+
+The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each
+`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI
+runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
+linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
+
 ---
 
 ## `gd-eval models`

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev2}/README.md

@@ -114,6 +114,7 @@ gd-eval run \
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
 | `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
 
@@ -126,21 +127,62 @@ gd-eval run \
 | Flag | Default | Description |
 |---|---|---|
 | `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
-| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests. Progress output interleaves when K > 1. |
+| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
 
+**Concurrency and workspace safety.** Agentic kinds that create workspace objects
+(`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
+time whatever `--concurrency` says — a metric or alert created and dropped mid-run would otherwise be visible to
+another item reading the same catalog. **That protection is for the agentic kinds only:** the single-turn
+`metric_skill` and `alert_skill` kinds are still fanned out and the agent performs the same server-side writes on
+that path, so avoid raising `--concurrency` on a dataset of those against a shared workspace. Progress output
+interleaves when K > 1, and per-item latencies rise, so they stop being clean single-request measurements.
+
 #### Output
 
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
+| `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
+| `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
 
 #### Langfuse sink
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Creates one named experiment run per model (`gd-eval-{timestamp}-{model}`, suffixed `-effort-{level}` when `--reasoning-effort` is set so runs differing only by effort stay separate). Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+
+**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
+because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
+flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
+exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
+fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+`--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
+skip linking altogether.
+
+**When trace linking happens.** Finding a gen-ai trace means polling until Langfuse has ingested it, which is
+lag measured in seconds to minutes. That work produces no verdict — the pass/fail is already decided — so it
+does not run inline per item. Every item's Langfuse block is queued and the whole batch runs *after* the agent
+phase, draining before any report is written. Two consequences worth knowing:
+
+- **No item's `latency_s` includes trace linking.** Its cost is reported separately as
+  `latency_breakdown_s.langfuse_s`, and the run prints
+  `[langfuse] trace linking finished in Xs for N item(s); slowest Ys`. If `slowest` approaches the **120s**
+  batched retry budget, links are timing out and scores are being orphaned — look for
+  `[langfuse] WARNING: no trace found for conversation ...`.
+- **The budget depends on who is waiting.** 120s is affordable only because the batch blocks nobody. A direct
+  library caller (`evaluate_agentic_*` without a `submit_trace_link`) polls inline, on its own critical path, and
+  gets **35s** instead — the same cost as before batching existed, so no inline caller pays for a budget raised
+  on the CLI's behalf. Either way a trace that is already ingested costs nothing: the loop looks before it sleeps.
+Scores are always final before the command exits — the run blocks on the batch. Interrupting with Ctrl-C drops
+whatever is still queued rather than making you wait it out: both the queued trace links and, under
+`--concurrency`, the items that have not started. The handful of items already in flight still have to finish —
+worker threads are joined at exit and an in-progress agent call cannot be cancelled — so expect to wait up to one
+`--concurrency`-wide wave, not the rest of the dataset.
 
 ### JSON report shape
 
@@ -162,6 +204,70 @@ The JSON report always uses the nested multi-model shape:
 
 Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
 
+Each item reports **how many of its runs passed**, not only whether one did:
+
+```json
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+```
+
+`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
+`quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
+run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
+inconsistent rather than wrong. The console shows `4/5 runs passed` in `Notes` for a non-unanimous pass and
+stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on every run`.
+
+`runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
+no K and drives its fixture exactly once.
+
+Each item additionally carries a per-phase breakdown:
+
+```json
+"latency_breakdown_s": {
+  "agent_s": 4.02,          // GoodData's own response time — the system under test
+  "judge_s": 1.31,          // LLM-as-judge scoring, post-hoc
+  "simulated_user_s": 0.0,  // our simulated user composing the next turn (multi-turn kinds)
+  "langfuse_s": 5.70        // trace lookup + score writing, off the critical path
+}
+```
+
+Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned
+nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail`, and a
+`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a
+criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0
+would be indistinguishable from the judge genuinely failing the answer, which is the confusion
+`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with
+`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For
+`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass —
+"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows
+the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the
+item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than
+`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`,
+or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes`
+(`1 run(s) ungraded`, `2 criterion(s) ungraded`).
+
+`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add
+up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation
+create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never
+inside it, because trace linking runs outside every item's critical path (see above) — summing all four would
+re-inflate exactly what that design removes.
+
+A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not
+applicable here", not "instant". Today:
+
+| Field | Populated by |
+|---|---|
+| `agent_s` | `agentic_general_question`, `agentic_metric_skill` |
+| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge |
+| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user |
+| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present |
+
+The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each
+`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI
+runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
+linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
+
 ---
 
 ## `gd-eval models`

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev2}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.74.0"
+version = "1.74.1.dev2"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.74.0",
+    "gooddata-sdk~=1.74.1.dev2",
     "httpx>=0.27,<1.0",
     "orjson>=3.9.15,<4.0.0",
     "pydantic>=2.6,<3.0",
@@ -30,7 +30,7 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
-llm-judge = ["openai>=1.40,<2.0"]
+llm-judge = ["openai>=1.45,<2.0"]
 
 [project.scripts]
 gd-eval = "gooddata_eval.cli.main:main"
@@ -43,8 +43,8 @@ dev = [
     "pytest>=8.3.5",
 ]
 test = [
-    "pytest~=8.3.4",
-    "pytest-cov~=6.0.0",
+    "pytest~=9.1.1",
+    "pytest-cov~=7.1.0",
     "pytest-json-report==1.5.0",
     "pytest-mock>=3.14.0",
 ]