gooddata-eval 1.74.1.dev2__tar.gz → 1.74.1.dev3__tar.gz

{gooddata_eval-1.74.1.dev2 → gooddata_eval-1.74.1.dev3}/AGENTS.md

@@ -4,15 +4,15 @@
 `gdc-nas`) through a dataset of natural-language questions and scores what comes back,
 including side-by-side comparison across models. Each dataset item is a JSON envelope
 loaded from a local folder or pulled from a Langfuse dataset. Results are aggregated into
-pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to a
-dataset run. The newest and most actively developed package in the repo.
+pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to an
+experiment. The newest and most actively developed package in the repo.
 
 ## Owns
 
 - The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
 - Dataset loading and the evaluation run loop
 - Per-capability evaluators and their scoring
-- Result reporting, and pushing runs, scores and trace links to Langfuse
+- Result reporting, and pushing experiments, scores and trace links to Langfuse
 
 ## Does NOT Own
 
@@ -29,7 +29,7 @@ dataset run. The newest and most actively developed package in the repo.
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
 | `core/dataset/` | dataset format and loading |
 | `core/evaluators/` | single-shot evaluators and their registry |
-| `core/langfuse/` | `sink.py` only — pushes single-turn scores and dataset-run items |
+| `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
 | `core/reporting/` | console and JSON output rendering |
 | `core/scoring.py`, `core/runner.py` | scoring and orchestration |
 | `core/models.py` | `DatasetItem`, `ChatResult`, `ItemReport` and friends |

{gooddata_eval-1.74.1.dev2 → gooddata_eval-1.74.1.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.74.1.dev2
+Version: 1.74.1.dev3
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,7 +17,7 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.74.1.dev2
+Requires-Dist: gooddata-sdk~=1.74.1.dev3
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: orjson<4.0.0,>=3.9.15
 Requires-Dist: pydantic<3.0,>=2.6
@@ -141,7 +141,7 @@ gd-eval run \
 | Flag | Description |
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
-| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 | `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
@@ -154,7 +154,8 @@ gd-eval run \
 
 | Flag | Default | Description |
 |---|---|---|
-| `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
+| `--runs K` | `2` | Independent runs per item. |
+| `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
 | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
@@ -180,15 +181,39 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 
-Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+##### Langfuse v4
 
-**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
-because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
-flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
-exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
-fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+One run is one Langfuse **experiment**. Each evaluated item becomes its own trace whose root span carries the
+experiment and dataset-item attributes (`langfuse.experiment.name`, `langfuse.experiment.dataset.id`,
+`langfuse.experiment.item.id`), and the four scores attach to that root observation. gd-eval speaks to Langfuse
+over four REST endpoints and uses no Langfuse SDK, so it runs on every Python version the package supports:
+
+| Endpoint | Used for |
+|---|---|
+| `POST /api/public/otel/v1/traces` | exporting the experiment root span as OTLP/HTTP JSON |
+| `POST /api/public/scores` | one score per write, on a trace or on a single observation inside it |
+| `GET /api/public/v2/observations` | finding the agent's gen-ai trace for a conversation |
+| `GET /api/public/dataset-items` | loading `--langfuse-dataset` items, and resolving an item's dataset id |
+
+Two consequences of v4's immutable observations. gd-eval sets `version` only on its own experiment span, never on
+the agent's gen-ai trace — filter on the gd-eval experiment's `langfuse.version` to compare models. And on the
+agentic kinds the latency in `value_score` is the gen-ai trace's root generation latency, read from the
+observations endpoint; the single-shot `--langfuse` sink keeps using the item's own measured average latency.
+
+Langfuse Cloud drops v3 on **2026-11-16**; a self-hosted Langfuse must be on v4 for any of this to work.
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to turn the whole **agentic** Langfuse write path off — no trace lookup, no
+span export and no scores for `agentic_*` items. The run says so once. It does not reach the `--langfuse` sink,
+which still writes a span and four scores for every single-shot item; drop `--langfuse` to silence that too.
+
+**A local `--dataset` cannot be attached to a Langfuse experiment.** `--langfuse` is refused alongside
+`--dataset` because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not
+depend on that flag — each `evaluate_agentic_*` builds its own client whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported — so a local run still finds its traces and writes its
+scores onto them, and only the per-run grouping fails: the dataset-item lookup 404s and the run reports the item
+as one that does not exist in Langfuse, once. The run also warns about this before it starts. Use
 `--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
 skip linking altogether.
 
@@ -235,10 +260,12 @@ Winner is selected by **pass rate → quality score → latency** (lower latency
 Each item reports **how many of its runs passed**, not only whether one did:
 
 ```json
-"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false, "gate_passed": true
 ```
 
-`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+`pass_at_k` is "did any run pass" — always literal, whatever the gate. `gate_passed` is the verdict the item
+was decided on and is what `passed` counts; the two differ only under `--gate power`, where an item that
+passed 4 of 5 runs is `"pass_at_k": true, "gate_passed": false`. `runs_passed` is the fact that separates a
 reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
 `quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
 run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
@@ -248,6 +275,14 @@ stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on ev
 `runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
 no K and drives its fixture exactly once.
 
+Which of the two decides pass/fail is `--gate`: `any` (default) gates on `pass_at_k`, `power` gates on
+`pass_power_k`. The run records it as a top-level `gate`, and a failure under `power` says so —
+`Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure` — because the message body describes
+the best run, which under pass^K can be a run that passed, and the console repeats the count in `Notes` for
+the same reason. When some runs were ungraded the note says so instead of calling the remainder unstable.
+`agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than
+labelling a report `power` that only part of the dataset was decided under.
+
 Each item additionally carries a per-phase breakdown:
 
 ```json
@@ -408,10 +443,19 @@ Without `[llm-judge]`, those items are **skipped**.
 
 ## Scores (in JSON report and Langfuse)
 
+In Langfuse every score is written to the experiment run's root observation — `traceId` plus `observationId` of
+the item's own root span. On the agentic path each score is mirrored onto the agent's gen-ai trace as well
+(`traceId` only), so a score survives even when one of the two traces is missing.
+
 | Score | Description |
 |---|---|
-| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |
+| `pass_at_k` | 1 if **any** of the K runs passed strict checks, else 0. |
+| `pass_power_k` | 1 only if **every** one of the K runs passed. Agentic kinds only. |
+| `gate_passed` | The verdict that decided the item: `pass_at_k` under `--gate any`, `pass_power_k` under `--gate power`. Agentic kinds only. |
 | `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. |
 | `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (speed = max(0, 1 − latency/60s)). |
 | `latency_s` | Average per-run latency in seconds. |
 | `provider_type` | Model vendor + gateway label (e.g. `ANTHROPIC`, `BEDROCK/ANTHROPIC`, `AZURE/OPENAI`). Stored in Langfuse trace metadata and tags. |
+
+Score names carry no K; K and the gate are on the dataset-run metadata as `eval_k` and `eval_gate`.
+`agentic_visualization` also still writes its historic `pass_at_{K}` / `pass_power_{K}` pair.

{gooddata_eval-1.74.1.dev2 → gooddata_eval-1.74.1.dev3}/README.md

@@ -113,7 +113,7 @@ gd-eval run \
 | Flag | Description |
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
-| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 | `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
@@ -126,7 +126,8 @@ gd-eval run \
 
 | Flag | Default | Description |
 |---|---|---|
-| `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
+| `--runs K` | `2` | Independent runs per item. |
+| `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
 | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
@@ -152,15 +153,39 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_BASE_URL` (or the legacy `LANGFUSE_HOST`). |
 
-Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+##### Langfuse v4
 
-**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
-because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
-flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
-exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
-fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+One run is one Langfuse **experiment**. Each evaluated item becomes its own trace whose root span carries the
+experiment and dataset-item attributes (`langfuse.experiment.name`, `langfuse.experiment.dataset.id`,
+`langfuse.experiment.item.id`), and the four scores attach to that root observation. gd-eval speaks to Langfuse
+over four REST endpoints and uses no Langfuse SDK, so it runs on every Python version the package supports:
+
+| Endpoint | Used for |
+|---|---|
+| `POST /api/public/otel/v1/traces` | exporting the experiment root span as OTLP/HTTP JSON |
+| `POST /api/public/scores` | one score per write, on a trace or on a single observation inside it |
+| `GET /api/public/v2/observations` | finding the agent's gen-ai trace for a conversation |
+| `GET /api/public/dataset-items` | loading `--langfuse-dataset` items, and resolving an item's dataset id |
+
+Two consequences of v4's immutable observations. gd-eval sets `version` only on its own experiment span, never on
+the agent's gen-ai trace — filter on the gd-eval experiment's `langfuse.version` to compare models. And on the
+agentic kinds the latency in `value_score` is the gen-ai trace's root generation latency, read from the
+observations endpoint; the single-shot `--langfuse` sink keeps using the item's own measured average latency.
+
+Langfuse Cloud drops v3 on **2026-11-16**; a self-hosted Langfuse must be on v4 for any of this to work.
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to turn the whole **agentic** Langfuse write path off — no trace lookup, no
+span export and no scores for `agentic_*` items. The run says so once. It does not reach the `--langfuse` sink,
+which still writes a span and four scores for every single-shot item; drop `--langfuse` to silence that too.
+
+**A local `--dataset` cannot be attached to a Langfuse experiment.** `--langfuse` is refused alongside
+`--dataset` because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not
+depend on that flag — each `evaluate_agentic_*` builds its own client whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported — so a local run still finds its traces and writes its
+scores onto them, and only the per-run grouping fails: the dataset-item lookup 404s and the run reports the item
+as one that does not exist in Langfuse, once. The run also warns about this before it starts. Use
 `--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
 skip linking altogether.
 
@@ -207,10 +232,12 @@ Winner is selected by **pass rate → quality score → latency** (lower latency
 Each item reports **how many of its runs passed**, not only whether one did:
 
 ```json
-"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false, "gate_passed": true
 ```
 
-`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+`pass_at_k` is "did any run pass" — always literal, whatever the gate. `gate_passed` is the verdict the item
+was decided on and is what `passed` counts; the two differ only under `--gate power`, where an item that
+passed 4 of 5 runs is `"pass_at_k": true, "gate_passed": false`. `runs_passed` is the fact that separates a
 reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
 `quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
 run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
@@ -220,6 +247,14 @@ stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on ev
 `runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
 no K and drives its fixture exactly once.
 
+Which of the two decides pass/fail is `--gate`: `any` (default) gates on `pass_at_k`, `power` gates on
+`pass_power_k`. The run records it as a top-level `gate`, and a failure under `power` says so —
+`Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure` — because the message body describes
+the best run, which under pass^K can be a run that passed, and the console repeats the count in `Notes` for
+the same reason. When some runs were ungraded the note says so instead of calling the remainder unstable.
+`agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than
+labelling a report `power` that only part of the dataset was decided under.
+
 Each item additionally carries a per-phase breakdown:
 
 ```json
@@ -380,10 +415,19 @@ Without `[llm-judge]`, those items are **skipped**.
 
 ## Scores (in JSON report and Langfuse)
 
+In Langfuse every score is written to the experiment run's root observation — `traceId` plus `observationId` of
+the item's own root span. On the agentic path each score is mirrored onto the agent's gen-ai trace as well
+(`traceId` only), so a score survives even when one of the two traces is missing.
+
 | Score | Description |
 |---|---|
-| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |
+| `pass_at_k` | 1 if **any** of the K runs passed strict checks, else 0. |
+| `pass_power_k` | 1 only if **every** one of the K runs passed. Agentic kinds only. |
+| `gate_passed` | The verdict that decided the item: `pass_at_k` under `--gate any`, `pass_power_k` under `--gate power`. Agentic kinds only. |
 | `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. |
 | `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (speed = max(0, 1 − latency/60s)). |
 | `latency_s` | Average per-run latency in seconds. |
 | `provider_type` | Model vendor + gateway label (e.g. `ANTHROPIC`, `BEDROCK/ANTHROPIC`, `AZURE/OPENAI`). Stored in Langfuse trace metadata and tags. |
+
+Score names carry no K; K and the gate are on the dataset-run metadata as `eval_k` and `eval_gate`.
+`agentic_visualization` also still writes its historic `pass_at_{K}` / `pass_power_{K}` pair.

{gooddata_eval-1.74.1.dev2 → gooddata_eval-1.74.1.dev3}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.74.1.dev2"
+version = "1.74.1.dev3"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.74.1.dev2",
+    "gooddata-sdk~=1.74.1.dev3",
     "httpx>=0.27,<1.0",
     "orjson>=3.9.15,<4.0.0",
     "pydantic>=2.6,<3.0",

{gooddata_eval-1.74.1.dev2 → gooddata_eval-1.74.1.dev3}/src/gooddata_eval/cli/agentic_runner.py

@@ -8,6 +8,7 @@ import time
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, TypedDict
 
+from gooddata_eval.core.agentic._gate import DEFAULT_GATE, EvalGate, normalize_gate
 from gooddata_eval.core.agentic._langfuse import make_langfuse_client
 from gooddata_eval.core.agentic._trace_linker import BackgroundTraceLinker, SubmitTraceLink, run_trace_link_inline
 from gooddata_eval.core.agentic.alert_skill import evaluate_agentic_alert_skill
@@ -48,6 +49,13 @@ AGENTIC_TEST_KINDS = frozenset(
 )
 
 
+# Agentic kinds that no gate applies to: they drive their fixture exactly once, so there is
+# no K to take pass@K or pass^K over. Named here rather than inline in _dispatch_agentic so
+# the CLI can refuse --gate power for a dataset containing one instead of labelling the whole
+# report `power` when part of it was never gated.
+UNGATED_AGENTIC_TEST_KINDS = frozenset({"agentic_conversation"})
+
+
 # Kinds cleared to run several at a time. An EXPLICIT allowlist, not a subtraction: nothing
 # in this package can prove a kind is read-only, because the mutation happens server-side in
 # whichever tools the agent decides to call. So each entry here is a reviewed judgement, and
@@ -128,9 +136,13 @@ def _dispatch_agentic(
     reasoning_effort: ReasoningEffort | None = None,
     agent_id: str | None = None,
     submit_trace_link: SubmitTraceLink = run_trace_link_inline,
+    gate: EvalGate = DEFAULT_GATE,
 ) -> AgenticEvalOutcome:
     """Call the appropriate evaluate_agentic_* function for the item's test_kind.
 
+    `gate` reaches every kind except those in UNGATED_AGENTIC_TEST_KINDS, which have no K
+    to gate over; the CLI refuses --gate power for a dataset containing one.
+
     Every evaluate_agentic_* function returns an AgenticEvalOutcome (reasoning_steps,
     conversation_id, response_id, detail) on success and attaches the same four attributes
     to its raised *AssertionError on failure -- no kind is exempt.
@@ -155,6 +167,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_outputs=_parse_visualization_expected(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -166,6 +179,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, (dict, list)) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -177,6 +191,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, dict) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -191,6 +206,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_tool_call=expected_args,
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -202,6 +218,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             user_context=item.user_context,
             **lf_kw,
@@ -214,6 +231,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -225,6 +243,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, dict) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -291,6 +310,7 @@ def run_agentic_items(
     on_item_done: Any = None,
     agent_id: str | None = None,
     concurrency: int = 1,
+    gate: EvalGate = DEFAULT_GATE,
 ) -> EvalReport:
     """Run agentic items through evaluate_agentic_* and return an EvalReport.
 
@@ -303,7 +323,7 @@ def run_agentic_items(
     """
     langfuse = make_langfuse_client() if use_langfuse else None
 
-    report = EvalReport(model=model_version)
+    report = EvalReport(model=model_version, gate=normalize_gate(gate))
     total = len(items)
     # Trace linking runs here rather than inside each evaluate_agentic_*, so an item's
     # Langfuse poll overlaps the NEXT item's agent call instead of extending its own
@@ -325,6 +345,9 @@ def run_agentic_items(
             test_kind=item.test_kind,
             question=item.question,
         )
+        # None, not False, for the kinds _dispatch_agentic passes no gate to: ItemReport.passed
+        # then falls back to pass_at_k and gate_passed keeps meaning "a gate ran".
+        gated = item.test_kind not in UNGATED_AGENTIC_TEST_KINDS
         t0 = time.perf_counter()
         try:
             outcome = _dispatch_agentic(
@@ -339,6 +362,7 @@ def run_agentic_items(
                 reasoning_effort,
                 agent_id,
                 submit_trace_link=linker.submit,
+                gate=gate,
             )
             if isinstance(outcome, AgenticEvalOutcome):
                 reasoning_steps = outcome.reasoning_steps
@@ -347,6 +371,8 @@ def run_agentic_items(
                 detail = outcome.detail
             else:
                 reasoning_steps, conversation_id, response_id, detail = outcome, None, None, {}
+            item_report.gate_passed = True if gated else None
+            # Whichever gate decided the item, clearing it means at least one run passed.
             item_report.pass_at_k = True
             item_report.runs = k
             item_report.reasoning_steps = reasoning_steps or []
@@ -356,7 +382,7 @@ def run_agentic_items(
             _apply_timings(item_report, getattr(outcome, "timings", None))
             _apply_run_counts(item_report, outcome)
         except AssertionError as exc:
-            item_report.pass_at_k = False
+            item_report.gate_passed = False if gated else None
             item_report.runs = k
             item_report.reasoning_steps = getattr(exc, "reasoning_steps", None) or []
             item_report.conversation_id = getattr(exc, "conversation_id", None)
@@ -364,6 +390,10 @@ def run_agentic_items(
             item_report.best_detail = getattr(exc, "detail", None) or {}
             _apply_timings(item_report, getattr(exc, "timings", None))
             _apply_run_counts(item_report, exc)
+            # Read off the counts, not off the gate: pass^K fails items where runs did pass,
+            # and reporting those as pass_at_k False would contradict the Langfuse score of
+            # the same name. Kinds that report no count read as 0, i.e. a clean failure.
+            item_report.pass_at_k = item_report.runs_passed > 0
             print(f"[agentic] {item.id} FAIL: {exc}", flush=True)
         except Exception as exc:
             item_report.error = f"{type(exc).__name__}: {exc}"

{gooddata_eval-1.74.1.dev2 → gooddata_eval-1.74.1.dev3}/src/gooddata_eval/cli/main.py

@@ -14,11 +14,20 @@ from gooddata_api_client.exceptions import ApiException
 from rich.console import Console
 from rich.table import Table
 
-from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, run_agentic_items
+from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, UNGATED_AGENTIC_TEST_KINDS, run_agentic_items
 from gooddata_eval.core.chat.sse_client import ChatClient
-from gooddata_eval.core.config import DEFAULT_JUDGE_MODEL, JUDGE_MODEL_ENV_VAR, ReasoningEffort, RunConfig
+from gooddata_eval.core.config import (
+    DEFAULT_GATE,
+    DEFAULT_JUDGE_MODEL,
+    JUDGE_MODEL_ENV_VAR,
+    EvalGate,
+    ReasoningEffort,
+    RunConfig,
+    normalize_gate,
+)
 from gooddata_eval.core.connection import ConnectionError_, resolve_connection
 from gooddata_eval.core.dataset.local import load_local_dataset
+from gooddata_eval.core.evaluators import supported_test_kinds
 from gooddata_eval.core.langfuse.sink import LangfuseSink
 from gooddata_eval.core.models import ChatResult, DatasetItem
 from gooddata_eval.core.reporting.console import render_comparison, render_console
@@ -91,7 +100,15 @@ def _build_parser() -> argparse.ArgumentParser:
             "Default: workspace's current active model."
         ),
     )
-    run.add_argument("--runs", type=int, default=2, help="Independent runs per item (pass@K). Default 2.")
+    run.add_argument("--runs", type=int, default=2, help="Independent runs per item. Default 2.")
+    run.add_argument(
+        "--gate",
+        choices=get_args(EvalGate),
+        default=DEFAULT_GATE,
+        help="Which verdict decides an item: 'any' = pass@K (a run passing is enough, the "
+        "default and historic behaviour), 'power' = pass^K (every run must pass, so the verdict "
+        "measures stability). Identical at --runs 1. Agentic kinds only.",
+    )
     run.add_argument(
         "--concurrency",
         type=int,
@@ -180,16 +197,46 @@ def _apply_timer_flag(enabled: bool) -> None:
         os.environ[TIMERS_ENV_VAR] = "1"
 
 
+def _reject_power_gate_on_ungated_items(config: RunConfig, items: list) -> None:
+    """Refuse a pass^K request the run cannot honour for every item.
+
+    Two kinds of item are never gated: everything on the non-agentic path, because
+    `run_items` has no gate and always decides on pass@K, and agentic_conversation, which
+    drives its fixture once whatever --runs says and so has no K to gate over. Running a
+    mixed dataset anyway would decide part of it under each rule and label the whole report
+    `power`. test_kind is resolved per item, so a dataset does not have to be homogeneous.
+
+    Kinds no evaluator supports are not counted: those items are skipped rather than
+    decided, so refusing on them would make --gate power fail where --gate any runs.
+    """
+    if normalize_gate(config.gate) != "power":
+        return
+    supported = supported_test_kinds()
+    ungated = [
+        i
+        for i in items
+        if i.test_kind in UNGATED_AGENTIC_TEST_KINDS
+        or (i.test_kind not in AGENTIC_TEST_KINDS and i.test_kind in supported)
+    ]
+    if not ungated:
+        return
+    kinds = sorted({i.test_kind for i in ungated})
+    raise ValueError(
+        f"--gate power applies to kinds that repeat K runs, but this dataset has {len(ungated)} "
+        f"item(s) of kind {kinds}, which are always decided on pass@K. Run them separately, or "
+        f"use --gate any."
+    )
+
+
 def _warn_if_local_dataset_cannot_link(config: RunConfig, agentic_items: list) -> None:
-    """Say up front that dataset-run assembly will fail, rather than after the run.
+    """Say up front that experiment assembly will fail, rather than after the run.
 
     --langfuse is refused outright with a local dataset because local item ids cannot be
     linked. But every evaluate_agentic_* falls back to try_make_langfuse_client() when the
-    caller passes none, so with LANGFUSE_* exported the linking runs anyway and each
-    conversation earns a 404 from dataset-run-items -- arriving in a block at the very end
-    of the run, long after the flag that would have prevented it could be changed. The
-    fallback is deliberate (direct library and tavern callers rely on it), so this warns
-    instead of disabling it.
+    caller passes none, so with LANGFUSE_* exported the linking runs anyway and every
+    dataset-item lookup 404s -- arriving in a block at the very end of the run, long after
+    the flag that would have prevented it could be changed. The fallback is deliberate
+    (direct library and tavern callers rely on it), so this warns instead of disabling it.
     """
     from gooddata_eval.core.agentic._langfuse import SKIP_ENV_VAR, langfuse_credentials_present  # noqa: PLC0415
     from gooddata_eval.core.config import env_flag  # noqa: PLC0415
@@ -201,7 +248,7 @@ def _warn_if_local_dataset_cannot_link(config: RunConfig, agentic_items: list) -
     print(
         f"warning: --dataset is a local folder, so its item ids are not Langfuse dataset item ids. "
         f"Traces will be found and scored, but the per-run grouping that makes models comparable "
-        f"cannot be created and each conversation will report a 404 from dataset-run-items. "
+        f"cannot be created and each conversation will report that its item does not exist in Langfuse. "
         f"Use --langfuse-dataset for comparable runs, or set {SKIP_ENV_VAR}=1 to skip trace linking.",
         file=sys.stderr,
     )
@@ -253,7 +300,7 @@ def _make_progress_callbacks(console: Console):
             tag = "[yellow]SKIP[/yellow]"
         elif report.error:
             tag = "[red]ERR [/red]"
-        elif report.pass_at_k:
+        elif report.passed:
             tag = "[green]PASS[/green]"
         else:
             tag = "[red]FAIL[/red]"
@@ -343,6 +390,7 @@ def _run(config: RunConfig) -> int:
     items = _load_dataset(config)
     agentic_items = [i for i in items if i.test_kind in AGENTIC_TEST_KINDS]
     non_agentic_items = [i for i in items if i.test_kind not in AGENTIC_TEST_KINDS]
+    _reject_power_gate_on_ungated_items(config, items)
     _warn_if_local_dataset_cannot_link(config, agentic_items)
     models = config.models or []
     run_ts = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H-%M")
@@ -417,6 +465,7 @@ def _run(config: RunConfig) -> int:
                     token=config.token,
                     workspace_id=config.workspace_id,
                     k=config.runs,
+                    gate=config.gate,
                     model_version=resolved.model_id,
                     reasoning_effort=config.reasoning_effort,
                     use_langfuse=config.log_to_langfuse,
@@ -466,6 +515,7 @@ def _run(config: RunConfig) -> int:
                 provider_name=resolved.provider_name or resolved.provider_id,
                 provider_type=resolved.provider_type,
                 workspace_id=config.workspace_id,
+                gate=config.gate,
             )
             if agentic_report is not None:
                 report.items.extend(agentic_report.items)
@@ -530,6 +580,7 @@ def main(argv: list[str] | None = None) -> int:
             kind=args.kind,
             preserve_failed=args.preserve_failed,
             reasoning_effort=args.reasoning_effort,
+            gate=normalize_gate(args.gate),
             agent_id=args.agent_id or os.environ.get("GD_EVAL_AGENT_ID"),
         )
         return _run(config)