gooddata-eval 1.74.0__tar.gz → 1.74.1.dev1__tar.gz

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.74.0
+Version: 1.74.1.dev1
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,13 +17,13 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.74.0
+Requires-Dist: gooddata-sdk~=1.74.1.dev1
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: orjson<4.0.0,>=3.9.15
 Requires-Dist: pydantic<3.0,>=2.6
 Requires-Dist: rich<15.0,>=13.0
 Provides-Extra: llm-judge
-Requires-Dist: openai<2.0,>=1.40; extra == 'llm-judge'
+Requires-Dist: openai<2.0,>=1.45; extra == 'llm-judge'
 Description-Content-Type: text/markdown
 
 # gooddata-eval
@@ -142,6 +142,7 @@ gd-eval run \
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
 | `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
 
@@ -154,21 +155,62 @@ gd-eval run \
 | Flag | Default | Description |
 |---|---|---|
 | `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
-| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests. Progress output interleaves when K > 1. |
+| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
 
+**Concurrency and workspace safety.** Agentic kinds that create workspace objects
+(`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
+time whatever `--concurrency` says — a metric or alert created and dropped mid-run would otherwise be visible to
+another item reading the same catalog. **That protection is for the agentic kinds only:** the single-turn
+`metric_skill` and `alert_skill` kinds are still fanned out and the agent performs the same server-side writes on
+that path, so avoid raising `--concurrency` on a dataset of those against a shared workspace. Progress output
+interleaves when K > 1, and per-item latencies rise, so they stop being clean single-request measurements.
+
 #### Output
 
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
+| `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
+| `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
 
 #### Langfuse sink
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Creates one named experiment run per model (`gd-eval-{timestamp}-{model}`, suffixed `-effort-{level}` when `--reasoning-effort` is set so runs differing only by effort stay separate). Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+
+**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
+because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
+flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
+exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
+fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+`--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
+skip linking altogether.
+
+**When trace linking happens.** Finding a gen-ai trace means polling until Langfuse has ingested it, which is
+lag measured in seconds to minutes. That work produces no verdict — the pass/fail is already decided — so it
+does not run inline per item. Every item's Langfuse block is queued and the whole batch runs *after* the agent
+phase, draining before any report is written. Two consequences worth knowing:
+
+- **No item's `latency_s` includes trace linking.** Its cost is reported separately as
+  `latency_breakdown_s.langfuse_s`, and the run prints
+  `[langfuse] trace linking finished in Xs for N item(s); slowest Ys`. If `slowest` approaches the **120s**
+  batched retry budget, links are timing out and scores are being orphaned — look for
+  `[langfuse] WARNING: no trace found for conversation ...`.
+- **The budget depends on who is waiting.** 120s is affordable only because the batch blocks nobody. A direct
+  library caller (`evaluate_agentic_*` without a `submit_trace_link`) polls inline, on its own critical path, and
+  gets **35s** instead — the same cost as before batching existed, so no inline caller pays for a budget raised
+  on the CLI's behalf. Either way a trace that is already ingested costs nothing: the loop looks before it sleeps.
+Scores are always final before the command exits — the run blocks on the batch. Interrupting with Ctrl-C drops
+whatever is still queued rather than making you wait it out: both the queued trace links and, under
+`--concurrency`, the items that have not started. The handful of items already in flight still have to finish —
+worker threads are joined at exit and an in-progress agent call cannot be cancelled — so expect to wait up to one
+`--concurrency`-wide wave, not the rest of the dataset.
 
 ### JSON report shape
 
@@ -190,6 +232,70 @@ The JSON report always uses the nested multi-model shape:
 
 Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
 
+Each item reports **how many of its runs passed**, not only whether one did:
+
+```json
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+```
+
+`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
+`quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
+run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
+inconsistent rather than wrong. The console shows `4/5 runs passed` in `Notes` for a non-unanimous pass and
+stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on every run`.
+
+`runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
+no K and drives its fixture exactly once.
+
+Each item additionally carries a per-phase breakdown:
+
+```json
+"latency_breakdown_s": {
+  "agent_s": 4.02,          // GoodData's own response time — the system under test
+  "judge_s": 1.31,          // LLM-as-judge scoring, post-hoc
+  "simulated_user_s": 0.0,  // our simulated user composing the next turn (multi-turn kinds)
+  "langfuse_s": 5.70        // trace lookup + score writing, off the critical path
+}
+```
+
+Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned
+nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail`, and a
+`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a
+criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0
+would be indistinguishable from the judge genuinely failing the answer, which is the confusion
+`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with
+`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For
+`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass —
+"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows
+the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the
+item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than
+`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`,
+or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes`
+(`1 run(s) ungraded`, `2 criterion(s) ungraded`).
+
+`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add
+up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation
+create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never
+inside it, because trace linking runs outside every item's critical path (see above) — summing all four would
+re-inflate exactly what that design removes.
+
+A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not
+applicable here", not "instant". Today:
+
+| Field | Populated by |
+|---|---|
+| `agent_s` | `agentic_general_question`, `agentic_metric_skill` |
+| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge |
+| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user |
+| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present |
+
+The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each
+`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI
+runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
+linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
+
 ---
 
 ## `gd-eval models`

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev1}/README.md

@@ -114,6 +114,7 @@ gd-eval run \
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
 | `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
 
@@ -126,21 +127,62 @@ gd-eval run \
 | Flag | Default | Description |
 |---|---|---|
 | `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
-| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests. Progress output interleaves when K > 1. |
+| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
 
+**Concurrency and workspace safety.** Agentic kinds that create workspace objects
+(`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
+time whatever `--concurrency` says — a metric or alert created and dropped mid-run would otherwise be visible to
+another item reading the same catalog. **That protection is for the agentic kinds only:** the single-turn
+`metric_skill` and `alert_skill` kinds are still fanned out and the agent performs the same server-side writes on
+that path, so avoid raising `--concurrency` on a dataset of those against a shared workspace. Progress output
+interleaves when K > 1, and per-item latencies rise, so they stop being clean single-request measurements.
+
 #### Output
 
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
+| `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
+| `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
 
 #### Langfuse sink
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Creates one named experiment run per model (`gd-eval-{timestamp}-{model}`, suffixed `-effort-{level}` when `--reasoning-effort` is set so runs differing only by effort stay separate). Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+
+**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
+because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
+flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
+exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
+fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+`--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
+skip linking altogether.
+
+**When trace linking happens.** Finding a gen-ai trace means polling until Langfuse has ingested it, which is
+lag measured in seconds to minutes. That work produces no verdict — the pass/fail is already decided — so it
+does not run inline per item. Every item's Langfuse block is queued and the whole batch runs *after* the agent
+phase, draining before any report is written. Two consequences worth knowing:
+
+- **No item's `latency_s` includes trace linking.** Its cost is reported separately as
+  `latency_breakdown_s.langfuse_s`, and the run prints
+  `[langfuse] trace linking finished in Xs for N item(s); slowest Ys`. If `slowest` approaches the **120s**
+  batched retry budget, links are timing out and scores are being orphaned — look for
+  `[langfuse] WARNING: no trace found for conversation ...`.
+- **The budget depends on who is waiting.** 120s is affordable only because the batch blocks nobody. A direct
+  library caller (`evaluate_agentic_*` without a `submit_trace_link`) polls inline, on its own critical path, and
+  gets **35s** instead — the same cost as before batching existed, so no inline caller pays for a budget raised
+  on the CLI's behalf. Either way a trace that is already ingested costs nothing: the loop looks before it sleeps.
+Scores are always final before the command exits — the run blocks on the batch. Interrupting with Ctrl-C drops
+whatever is still queued rather than making you wait it out: both the queued trace links and, under
+`--concurrency`, the items that have not started. The handful of items already in flight still have to finish —
+worker threads are joined at exit and an in-progress agent call cannot be cancelled — so expect to wait up to one
+`--concurrency`-wide wave, not the rest of the dataset.
 
 ### JSON report shape
 
@@ -162,6 +204,70 @@ The JSON report always uses the nested multi-model shape:
 
 Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
 
+Each item reports **how many of its runs passed**, not only whether one did:
+
+```json
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+```
+
+`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
+`quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
+run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
+inconsistent rather than wrong. The console shows `4/5 runs passed` in `Notes` for a non-unanimous pass and
+stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on every run`.
+
+`runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
+no K and drives its fixture exactly once.
+
+Each item additionally carries a per-phase breakdown:
+
+```json
+"latency_breakdown_s": {
+  "agent_s": 4.02,          // GoodData's own response time — the system under test
+  "judge_s": 1.31,          // LLM-as-judge scoring, post-hoc
+  "simulated_user_s": 0.0,  // our simulated user composing the next turn (multi-turn kinds)
+  "langfuse_s": 5.70        // trace lookup + score writing, off the critical path
+}
+```
+
+Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned
+nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail`, and a
+`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a
+criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0
+would be indistinguishable from the judge genuinely failing the answer, which is the confusion
+`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with
+`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For
+`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass —
+"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows
+the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the
+item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than
+`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`,
+or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes`
+(`1 run(s) ungraded`, `2 criterion(s) ungraded`).
+
+`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add
+up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation
+create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never
+inside it, because trace linking runs outside every item's critical path (see above) — summing all four would
+re-inflate exactly what that design removes.
+
+A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not
+applicable here", not "instant". Today:
+
+| Field | Populated by |
+|---|---|
+| `agent_s` | `agentic_general_question`, `agentic_metric_skill` |
+| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge |
+| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user |
+| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present |
+
+The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each
+`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI
+runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
+linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
+
 ---
 
 ## `gd-eval models`

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev1}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.74.0"
+version = "1.74.1.dev1"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.74.0",
+    "gooddata-sdk~=1.74.1.dev1",
     "httpx>=0.27,<1.0",
     "orjson>=3.9.15,<4.0.0",
     "pydantic>=2.6,<3.0",
@@ -30,7 +30,7 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
-llm-judge = ["openai>=1.40,<2.0"]
+llm-judge = ["openai>=1.45,<2.0"]
 
 [project.scripts]
 gd-eval = "gooddata_eval.cli.main:main"

{gooddata_eval-1.74.0 → gooddata_eval-1.74.1.dev1}/src/gooddata_eval/cli/agentic_runner.py

@@ -3,10 +3,13 @@
 
 from __future__ import annotations
 
+import sys
 import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, TypedDict
 
 from gooddata_eval.core.agentic._langfuse import make_langfuse_client
+from gooddata_eval.core.agentic._trace_linker import BackgroundTraceLinker, SubmitTraceLink, run_trace_link_inline
 from gooddata_eval.core.agentic.alert_skill import evaluate_agentic_alert_skill
 from gooddata_eval.core.agentic.conversation import ConversationFixture, evaluate_agentic_conversation
 from gooddata_eval.core.agentic.general_question import evaluate_agentic_general_question
@@ -27,6 +30,7 @@ class _LfKw(TypedDict, total=False):
     run_timestamp: str
     model_version_override: str | None
     reasoning_effort: ReasoningEffort | None
+    submit_trace_link: SubmitTraceLink
 
 
 AGENTIC_TEST_KINDS = frozenset(
@@ -44,6 +48,43 @@ AGENTIC_TEST_KINDS = frozenset(
 )
 
 
+# Kinds cleared to run several at a time. An EXPLICIT allowlist, not a subtraction: nothing
+# in this package can prove a kind is read-only, because the mutation happens server-side in
+# whichever tools the agent decides to call. So each entry here is a reviewed judgement, and
+# anything absent -- including a kind added later -- runs serially. Slow is a recoverable
+# mistake; two runs sharing a workspace mid-mutation corrupts eval results silently and
+# reads like a model regression.
+#
+#   agentic_general_question, agentic_guardrail  answer questions only, no tool writes
+#   agentic_search                               search_objects, read-only by definition
+#   vis_agentic, agentic_visualization           visualizations come back as AAC proposals
+#                                                in the chat response; nothing is persisted
+#                                                and neither module has cleanup code
+PARALLEL_SAFE_TEST_KINDS = frozenset(
+    {
+        "agentic_general_question",
+        "agentic_guardrail",
+        "agentic_search",
+        "vis_agentic",
+        "agentic_visualization",
+    }
+)
+
+# Everything else. metric_skill and alert_skill demonstrably create workspace objects (they
+# carry delete_entity_metrics / delete_entity_automations cleanup) and metric_skill._delete_metric
+# records that a leaked metric gets reused by a later test. agentic_conversation drives the
+# metric skill. agentic_kda_skill is here on suspicion rather than proof: it triggers
+# create_key_driver_analysis with no cleanup, and while the evaluator only ever reads that
+# call's ARGUMENTS -- never a created object id -- whether the platform persists anything is
+# unverified. Move it to the allowlist once someone confirms it does not.
+WORKSPACE_MUTATING_TEST_KINDS = frozenset(AGENTIC_TEST_KINDS) - PARALLEL_SAFE_TEST_KINDS
+
+
+def runs_in_parallel(test_kind: str) -> bool:
+    """True only for kinds explicitly cleared for concurrent execution."""
+    return test_kind in PARALLEL_SAFE_TEST_KINDS
+
+
 def _parse_visualization_expected(expected_output: Any) -> list[CreatedVisualization]:
     """Parse expected_output into a list of CreatedVisualization candidates.
 
@@ -86,6 +127,7 @@ def _dispatch_agentic(
     model_version_override: str | None,
     reasoning_effort: ReasoningEffort | None = None,
     agent_id: str | None = None,
+    submit_trace_link: SubmitTraceLink = run_trace_link_inline,
 ) -> AgenticEvalOutcome:
     """Call the appropriate evaluate_agentic_* function for the item's test_kind.
 
@@ -102,6 +144,7 @@ def _dispatch_agentic(
         "run_timestamp": run_ts,
         "model_version_override": model_version_override,
         "reasoning_effort": reasoning_effort,
+        "submit_trace_link": submit_trace_link,
     }
 
     if kind in ("vis_agentic", "agentic_visualization"):
@@ -160,6 +203,7 @@ def _dispatch_agentic(
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
             agent_id=agent_id,
+            user_context=item.user_context,
             **lf_kw,
         )
     elif kind == "agentic_guardrail":
@@ -198,6 +242,40 @@ def _dispatch_agentic(
         raise ValueError(f"Unknown agentic test kind: {kind!r}")
 
 
+def _apply_run_counts(item_report: ItemReport, source: Any) -> None:
+    """Copy how many runs passed, and how many actually ran, onto the item report.
+
+    Kinds that report neither keep the requested K and a 0 count, which reads as "not
+    instrumented" rather than "nothing passed" because ``pass_power_k`` is only consulted
+    for an item that already passed.
+    """
+    runs_passed = getattr(source, "runs_passed", None)
+    if runs_passed is not None:
+        item_report.runs_passed = runs_passed
+    effective = getattr(source, "runs_effective", None)
+    if effective:
+        # Only when the kind knows better than K -- agentic_conversation runs once.
+        item_report.runs_effective = effective
+    # The agentic kinds record their unscored runs in the detail; the report field is the
+    # one place every kind's count is read from.
+    unscored = (getattr(source, "detail", None) or {}).get("unscored_runs")
+    if isinstance(unscored, int):
+        item_report.runs_ungraded = unscored
+
+
+def _apply_timings(item_report: ItemReport, timings: Any) -> None:
+    """Copy an outcome's phase breakdown onto the item report, if the kind recorded one.
+
+    Kinds with no phase instrumentation pass None and keep their 0.0 defaults rather than
+    reporting invented numbers.
+    """
+    if timings is None:
+        return
+    item_report.agent_latency_s = timings.agent_s
+    item_report.judge_latency_s = timings.judge_s
+    item_report.simulated_user_latency_s = timings.simulated_user_s
+
+
 def run_agentic_items(
     items: list[DatasetItem],
     host: str,
@@ -212,14 +290,29 @@ def run_agentic_items(
     on_item_start: Any = None,
     on_item_done: Any = None,
     agent_id: str | None = None,
+    concurrency: int = 1,
 ) -> EvalReport:
-    """Run agentic items through evaluate_agentic_* and return an EvalReport."""
+    """Run agentic items through evaluate_agentic_* and return an EvalReport.
+
+    ``concurrency`` > 1 runs PARALLEL_SAFE_TEST_KINDS items simultaneously.
+    WORKSPACE_MUTATING_TEST_KINDS items always run one at a time, and in a separate phase
+    from the parallel ones -- a metric being created and dropped mid-run would otherwise be
+    visible to a catalog-reading item running alongside it.
+
+    Results are collected in dataset order regardless of completion order.
+    """
     langfuse = make_langfuse_client() if use_langfuse else None
 
     report = EvalReport(model=model_version)
     total = len(items)
+    # Trace linking runs here rather than inside each evaluate_agentic_*, so an item's
+    # Langfuse poll overlaps the NEXT item's agent call instead of extending its own
+    # latency. Drained below before this function returns, so every score is written
+    # before the caller renders a report or decides an exit code.
+    linker = BackgroundTraceLinker()
+    _t0 = time.perf_counter()
 
-    for index, item in enumerate(items, start=1):
+    def _process_item(index: int, item: DatasetItem) -> ItemReport:
         if on_item_start is not None:
             try:
                 on_item_start(index, total, item)
@@ -235,7 +328,17 @@ def run_agentic_items(
         t0 = time.perf_counter()
         try:
             outcome = _dispatch_agentic(
-                item, host, token, workspace_id, k, langfuse, run_ts, model_version, reasoning_effort, agent_id
+                item,
+                host,
+                token,
+                workspace_id,
+                k,
+                langfuse,
+                run_ts,
+                model_version,
+                reasoning_effort,
+                agent_id,
+                submit_trace_link=linker.submit,
             )
             if isinstance(outcome, AgenticEvalOutcome):
                 reasoning_steps = outcome.reasoning_steps
@@ -250,6 +353,8 @@ def run_agentic_items(
             item_report.conversation_id = conversation_id
             item_report.response_id = response_id
             item_report.best_detail = detail or {}
+            _apply_timings(item_report, getattr(outcome, "timings", None))
+            _apply_run_counts(item_report, outcome)
         except AssertionError as exc:
             item_report.pass_at_k = False
             item_report.runs = k
@@ -257,10 +362,17 @@ def run_agentic_items(
             item_report.conversation_id = getattr(exc, "conversation_id", None)
             item_report.response_id = getattr(exc, "response_id", None)
             item_report.best_detail = getattr(exc, "detail", None) or {}
+            _apply_timings(item_report, getattr(exc, "timings", None))
+            _apply_run_counts(item_report, exc)
             print(f"[agentic] {item.id} FAIL: {exc}", flush=True)
         except Exception as exc:
             item_report.error = f"{type(exc).__name__}: {exc}"
             item_report.runs = 0
+            # An item that errored still measured whatever it got through, and those are
+            # the most useful numbers on the report -- an item unevaluable because its
+            # judge broke should not also report the agent as costing 0s. Kinds that
+            # attach no timings to the exception keep their 0.0 defaults.
+            _apply_timings(item_report, getattr(exc, "timings", None))
         finally:
             item_report.latency_s = time.perf_counter() - t0
 
@@ -270,7 +382,79 @@ def run_agentic_items(
             except Exception:
                 pass
 
-        report.items.append(item_report)
+        return item_report
+
+    concurrency = max(1, concurrency)
+    indexed = list(enumerate(items, start=1))
+    if concurrency > 1:
+        parallel = [(i, it) for i, it in indexed if runs_in_parallel(it.test_kind)]
+        serial = [(i, it) for i, it in indexed if not runs_in_parallel(it.test_kind)]
+        if not parallel and serial:
+            blocked = ", ".join(sorted({it.test_kind for _, it in serial}))
+            print(
+                f"warning: --concurrency {concurrency} has no effect here; every item is a "
+                f"workspace-mutating kind ({blocked}) and those always run one at a time.",
+                file=sys.stderr,
+            )
+    else:
+        parallel, serial = [], indexed
+
+    results: dict[int, ItemReport] = {}
+    try:
+        # Two phases, never interleaved: a mutating item creating and dropping a metric
+        # mid-run would otherwise be visible to a catalog-reading item beside it.
+        if parallel:
+            # NOT a `with` block. ThreadPoolExecutor.__exit__ is shutdown(wait=True) with
+            # cancel_futures left False, so an interrupt raised in this thread while it
+            # waits on as_completed runs every QUEUED item to completion before the
+            # KeyboardInterrupt is honoured. cancel_futures drops whatever has not started; the handful already in flight
+            # cannot be cancelled (the interpreter joins those worker threads at exit
+            # regardless), so this bounds the wait at one wave rather than the dataset.
+            pool = ThreadPoolExecutor(max_workers=concurrency, thread_name_prefix="agentic")
+            try:
+                futures = {pool.submit(_process_item, i, it): i for i, it in parallel}
+                for future in as_completed(futures):
+                    results[futures[future]] = future.result()
+            except BaseException:
+                pool.shutdown(wait=False, cancel_futures=True)
+                raise
+            else:
+                pool.shutdown(wait=True)
+        for i, it in serial:
+            results[i] = _process_item(i, it)
+        report.items.extend(results[i] for i in sorted(results))
+    except BaseException:
+        # Ctrl-C, or anything else escaping the loop: drop the queued polls rather than
+        # make the user sit through them (see BackgroundTraceLinker.abandon).
+        linker.abandon()
+        raise
+
+    # Blocks until every deferred trace link has finished: "async" here means the poll
+    # overlaps other items' work, never that the command finishes before scores are final.
+    if linker.pending:
+        # Said before the wait, not after. The batch runs once the last item is done and
+        # can take tens of seconds waiting on Langfuse ingestion; without this the
+        # terminal sits silent right after the final item and looks hung.
+        print(
+            f"[langfuse] linking traces for {linker.pending} item(s); waiting on Langfuse ingestion...",
+            flush=True,
+        )
+    _link_t0 = time.perf_counter()
+    linker.drain()
+    _link_elapsed = time.perf_counter() - _link_t0
+    for finished in report.items:
+        finished.langfuse_latency_s = linker.durations.get(finished.id, 0.0)
+    if linker.durations:
+        # Surfaces whether the retry budget is binding. A slowest close to
+        # _langfuse._LINK_BUDGET_SEC means links are timing out and scores are being
+        # orphaned -- read it together with any "no trace found for conversation" warnings.
+        slowest = max(linker.durations.values())
+        print(
+            f"[langfuse] trace linking finished in {_link_elapsed:.1f}s "
+            f"for {len(linker.durations)} item(s); slowest {slowest:.1f}s",
+            flush=True,
+        )
+    report.wall_clock_s = time.perf_counter() - _t0
 
     if langfuse is not None:
         try: