gooddata-eval 1.78.1.dev1__tar.gz → 1.78.1.dev2__tar.gz

gooddata_eval-1.78.1.dev1/README.md → gooddata_eval-1.78.1.dev2/PKG-INFO

@@ -1,3 +1,32 @@
+Metadata-Version: 2.5
+Name: gooddata-eval
+Version: 1.78.1.dev2
+Summary: Evaluate the GoodData AI agent against your own questions and models.
+Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
+Author-email: GoodData <support@gooddata.com>
+License-Expression: MIT
+License-File: LICENSE.txt
+Keywords: ai,analytics,cli,evaluation,gooddata,llm
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: gooddata-sdk~=1.78.1.dev2
+Requires-Dist: httpx<1.0,>=0.27
+Requires-Dist: jsonpatch<2.0,>=1.33
+Requires-Dist: orjson<4.0.0,>=3.9.15
+Requires-Dist: pydantic<3.0,>=2.6
+Requires-Dist: rich<15.0,>=13.0
+Provides-Extra: llm-judge
+Requires-Dist: openai<2.0,>=1.45; extra == 'llm-judge'
+Description-Content-Type: text/markdown
+
 # gooddata-eval
 
 CLI to evaluate the GoodData AI agent against a dataset of natural-language
@@ -709,6 +738,63 @@ A dataset is a folder of `.json` files, one per question:
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
 `search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
+### `visualization` items
+
+The `expected_output` lists the candidate charts in the AAC format, and the item passes when
+the agent's chart matches one of them. Keys beside the candidates add item-level checks:
+
+```json
+{
+  "id": "sankey-001",
+  "dataset_name": "agent_visualization_extended_charts",
+  "test_kind": "visualization",
+  "question": "Show how net sales flow from product category to customer country.",
+  "expected_output": {
+    "visualization": {
+      "type": "sankey_chart",
+      "query": {"fields": {"m": {"using": "metric/net_sales"}, "c": {"using": "label/product_category"},
+                           "d": {"using": "label/customer_country"}}},
+      "metrics": ["m"],
+      "from": "c",
+      "to": "d",
+      "buckets_strict": true
+    },
+    "requires_visualization_part": true
+  }
+}
+```
+
+Item-level keys, all optional except the candidates:
+
+- `visualization` — one candidate chart, or a list of them. `expected_outputs`
+  (`[{"visualization": {...}}, ...]`) is the same list in the agentic fixture form.
+- `requires_execution` — the agent must also run the chart it built.
+- `expects_visualization: false` — the agent must answer without a chart, for example by asking
+  for a part the request leaves out. The item then lists no candidates.
+- `visualization_optional: true` — either an answer without a chart, or a chart that matches a
+  candidate, is right.
+- `answer_must_include` / `answer_must_not_match` — regular expressions, matched without regard
+  to case against the final answer. Every `answer_must_include` pattern must match, and no
+  `answer_must_not_match` pattern may.
+- `format_intents` — per metric uri, how the metric's numbers must read, for example
+  `{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`.
+- `requires_visualization_part` — the chart must come back as a part of the answer, not only as
+  the arguments of the chart tool call.
+
+Switches on a candidate chart:
+
+- `buckets_strict` — compare bucket by bucket, not only the union of the buckets, so swapped
+  heatmap rows and columns or a reversed sankey fail.
+- `metrics_ordered` — compare the metrics by position, for bullet roles and bubble axes.
+
+Any other key, at item level or on a candidate chart, makes the item error before any request
+is sent, because a misspelled key would otherwise be dropped and its check would never run:
+
+```
+ValueError: the expectation states ['answer_must_inlcude'], which is not one of ['answer_must_include', ...]
+ValueError: a candidate chart states ['bucket_strict'], which the chart model does not read
+```
+
 ### `dashboard_summary` items
 
 Summary items call the dedicated summary endpoint

gooddata_eval-1.78.1.dev1/PKG-INFO → gooddata_eval-1.78.1.dev2/README.md

@@ -1,32 +1,3 @@
-Metadata-Version: 2.5
-Name: gooddata-eval
-Version: 1.78.1.dev1
-Summary: Evaluate the GoodData AI agent against your own questions and models.
-Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
-Author-email: GoodData <support@gooddata.com>
-License-Expression: MIT
-License-File: LICENSE.txt
-Keywords: ai,analytics,cli,evaluation,gooddata,llm
-Classifier: Development Status :: 4 - Beta
-Classifier: Environment :: Console
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Scientific/Engineering
-Classifier: Topic :: Software Development
-Classifier: Typing :: Typed
-Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.78.1.dev1
-Requires-Dist: httpx<1.0,>=0.27
-Requires-Dist: jsonpatch<2.0,>=1.33
-Requires-Dist: orjson<4.0.0,>=3.9.15
-Requires-Dist: pydantic<3.0,>=2.6
-Requires-Dist: rich<15.0,>=13.0
-Provides-Extra: llm-judge
-Requires-Dist: openai<2.0,>=1.45; extra == 'llm-judge'
-Description-Content-Type: text/markdown
-
 # gooddata-eval
 
 CLI to evaluate the GoodData AI agent against a dataset of natural-language
@@ -738,6 +709,63 @@ A dataset is a folder of `.json` files, one per question:
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
 `search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
+### `visualization` items
+
+The `expected_output` lists the candidate charts in the AAC format, and the item passes when
+the agent's chart matches one of them. Keys beside the candidates add item-level checks:
+
+```json
+{
+  "id": "sankey-001",
+  "dataset_name": "agent_visualization_extended_charts",
+  "test_kind": "visualization",
+  "question": "Show how net sales flow from product category to customer country.",
+  "expected_output": {
+    "visualization": {
+      "type": "sankey_chart",
+      "query": {"fields": {"m": {"using": "metric/net_sales"}, "c": {"using": "label/product_category"},
+                           "d": {"using": "label/customer_country"}}},
+      "metrics": ["m"],
+      "from": "c",
+      "to": "d",
+      "buckets_strict": true
+    },
+    "requires_visualization_part": true
+  }
+}
+```
+
+Item-level keys, all optional except the candidates:
+
+- `visualization` — one candidate chart, or a list of them. `expected_outputs`
+  (`[{"visualization": {...}}, ...]`) is the same list in the agentic fixture form.
+- `requires_execution` — the agent must also run the chart it built.
+- `expects_visualization: false` — the agent must answer without a chart, for example by asking
+  for a part the request leaves out. The item then lists no candidates.
+- `visualization_optional: true` — either an answer without a chart, or a chart that matches a
+  candidate, is right.
+- `answer_must_include` / `answer_must_not_match` — regular expressions, matched without regard
+  to case against the final answer. Every `answer_must_include` pattern must match, and no
+  `answer_must_not_match` pattern may.
+- `format_intents` — per metric uri, how the metric's numbers must read, for example
+  `{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`.
+- `requires_visualization_part` — the chart must come back as a part of the answer, not only as
+  the arguments of the chart tool call.
+
+Switches on a candidate chart:
+
+- `buckets_strict` — compare bucket by bucket, not only the union of the buckets, so swapped
+  heatmap rows and columns or a reversed sankey fail.
+- `metrics_ordered` — compare the metrics by position, for bullet roles and bubble axes.
+
+Any other key, at item level or on a candidate chart, makes the item error before any request
+is sent, because a misspelled key would otherwise be dropped and its check would never run:
+
+```
+ValueError: the expectation states ['answer_must_inlcude'], which is not one of ['answer_must_include', ...]
+ValueError: a candidate chart states ['bucket_strict'], which the chart model does not read
+```
+
 ### `dashboard_summary` items
 
 Summary items call the dedicated summary endpoint

{gooddata_eval-1.78.1.dev1 → gooddata_eval-1.78.1.dev2}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.78.1.dev1"
+version = "1.78.1.dev2"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.78.1.dev1",
+    "gooddata-sdk~=1.78.1.dev2",
     "httpx>=0.27,<1.0",
     "jsonpatch>=1.33,<2.0",
     "orjson>=3.9.15,<4.0.0",

{gooddata_eval-1.78.1.dev1 → gooddata_eval-1.78.1.dev2}/src/gooddata_eval/cli/agentic_runner.py

@@ -26,7 +26,7 @@ from gooddata_eval.core.agentic.visualization import evaluate_agentic_visualizat
 from gooddata_eval.core.agentic.what_if import evaluate_agentic_what_if
 from gooddata_eval.core.config import ReasoningEffort
 from gooddata_eval.core.evaluators._llm_judge import JudgeResponseError
-from gooddata_eval.core.evaluators.visualization import requires_execution_of
+from gooddata_eval.core.evaluators.visualization import expectation_options
 from gooddata_eval.core.models import AgenticEvalOutcome, CreatedVisualization, DatasetItem
 from gooddata_eval.core.runner import EvalReport, ItemReport
 from gooddata_eval.core.timing import TraceRecorder
@@ -119,6 +119,8 @@ def _parse_visualization_expected(expected_output: Any) -> list[CreatedVisualiza
       {"visualization": {...}} or {"visualization": [{...}]}  <- single/multi candidate
       [{"visualization": {...}}, ...]                          <- bare list
     """
+    if isinstance(expected_output, dict) and expected_output.get("expects_visualization") is False:
+        return []
     if isinstance(expected_output, dict):
         raw_list = expected_output.get("expected_outputs")
         if raw_list is not None:
@@ -187,7 +189,7 @@ def _dispatch_agentic(
             workspace_id=workspace_id,
             question=item.question,
             expected_outputs=_parse_visualization_expected(eo),
-            requires_execution=requires_execution_of(eo),
+            options=expectation_options(eo),
             k=k,
             gate=gate,
             agent_id=agent_id,

{gooddata_eval-1.78.1.dev1 → gooddata_eval-1.78.1.dev2}/src/gooddata_eval/core/agentic/dashboard_skill.py

@@ -40,6 +40,7 @@ from gooddata_eval.core.models import (
     build_latency_breakdown,
     shift_and_index_events,
 )
+from gooddata_eval.core.scoring import AAC_VISUALIZATION_TYPES, same_viz_type
 from gooddata_eval.core.timing import PhaseTimings, log_timer, sum_timings
 
 _DEFAULT_K = 1
@@ -582,6 +583,64 @@ def _check_date_bindings(widgets: list[dict], new_ids: set[str], expected: list[
     return failures
 
 
+def _has_chart_types(expected_output: dict) -> bool:
+    """Whether any expected chart states the type it must be placed as."""
+    return any(isinstance(v, dict) and "chart_type" in v for v in expected_output.get("visualizations") or [])
+
+
+def _bodies_by_id(*references: dict) -> dict[str, dict]:
+    """Every chart body the response's references carry, existing and authored, by chart id."""
+    bodies: dict[str, dict] = {}
+    for refs in references:
+        for key in ("visualizations", "new_visualizations"):
+            for body in refs.get(key) or []:
+                if isinstance(body, dict) and body.get("id"):
+                    bodies[body["id"]] = body
+    return bodies
+
+
+def _check_chart_types(
+    widgets: list[dict], new_ids: set[str], expected: list[dict], bodies: dict[str, dict]
+) -> tuple[list[str], list[str]]:
+    """Every widget of a chart that states a ``chart_type`` must show a chart of exactly that type.
+
+    A widget names only the chart id, so the type is read from the chart body the references
+    carry. The title alone cannot catch a chart authored as another type than the one asked
+    for, which is the silent swap GDAI-2054 rules out.
+
+    Returns the failures and the notes. A widget whose chart has no body in the references
+    cannot be checked: gen-ai carries a body only when it could resolve one, so a missing
+    body is a platform fault, noted as ``_check_references`` notes a missing id, and not
+    scored as the agent's.
+    """
+    failures: list[str] = []
+    notes: list[str] = []
+    for exp in expected:
+        if not isinstance(exp, dict) or "chart_type" not in exp:
+            continue
+        wanted = exp["chart_type"]
+        label = exp.get("title") if exp.get("id") is None else exp.get("id")
+        candidates = _matching_widgets(widgets, new_ids, exp)
+        if not candidates:
+            failures.append(f"cannot check the type of chart {label!r}: it is not on the dashboard")
+            continue
+        types: list[str] = []
+        for widget in candidates:
+            chart_type = (bodies.get(widget.get("visualization")) or {}).get("type")
+            if isinstance(chart_type, str):
+                types.append(chart_type)
+            else:
+                notes.append(
+                    f"cannot check the type of chart {label!r}: the response's references carry no body "
+                    f"for {widget.get('visualization')!r}"
+                )
+        # Every placement, as for date bindings: one right placement must not hide a wrong one.
+        if not all(same_viz_type(wanted, t) for t in types):
+            found = ", ".join(repr(t) for t in types)
+            failures.append(f"chart {label!r} is {found}, expected {wanted!r}")
+    return failures, notes
+
+
 def _tabs_of(dashboard: dict) -> list[dict]:
     """The dashboard's tabs, reading a document with root sections as its single tab."""
     tabs = dashboard.get("tabs")
@@ -936,6 +995,12 @@ def _validate_layout_expectations(expected_output: dict) -> None:
     for chart in expected_output.get("visualizations") or []:
         if isinstance(chart, dict) and "date_dataset" in chart and not _is_dataset_or_null(chart["date_dataset"]):
             raise ValueError(f"date_dataset must name a date dataset or be null, got {chart!r}")
+        if (
+            isinstance(chart, dict)
+            and "chart_type" in chart
+            and (not isinstance(chart["chart_type"], str) or chart["chart_type"] not in AAC_VISUALIZATION_TYPES)
+        ):
+            raise ValueError(f"chart_type must be an AAC visualization type, got {chart!r}")
 
     if "tabs" in expected_output:
         tabs = expected_output["tabs"]
@@ -1037,6 +1102,7 @@ class _Applies:
     tab_filters: bool = False
     date_filters: bool = False
     preserved_widgets: bool = False
+    chart_types: bool = False
 
     @classmethod
     def of(cls, expected_output: dict) -> _Applies:
@@ -1049,6 +1115,7 @@ class _Applies:
             tab_filters="tab_filters" in expected_output,
             date_filters="date_filters" in expected_output,
             preserved_widgets="preserved_widgets" in expected_output,
+            chart_types=_has_chart_types(expected_output),
         )
 
 
@@ -1089,6 +1156,7 @@ class DashboardEvaluation:
     tab_filters_correct: bool = False
     date_filters_correct: bool = False
     widgets_preserved: bool = False
+    chart_types_correct: bool = False
     failures: list[str] = field(default_factory=list)
     notes: list[str] = field(default_factory=list)
 
@@ -1129,6 +1197,8 @@ class DashboardEvaluation:
             checks["dashboard_date_filters_correct"] = self.date_filters_correct
         if self.applies.preserved_widgets:
             checks["dashboard_widgets_preserved"] = self.widgets_preserved
+        if self.applies.chart_types:
+            checks["dashboard_chart_types_correct"] = self.chart_types_correct
         return checks
 
     @property
@@ -1280,12 +1350,14 @@ def evaluate_dashboard_response(
             | {v.get("id") for v in patch_references.get("visualizations") or [] if v.get("id")}
             | new_ids
         )
+        bodies = _bodies_by_id(base_references, patch_references)
     else:
         if actual_saved_id is not None:
             saved_failures.append(f"a new draft must not be saved yet, but it reports id {actual_saved_id!r}")
         document = base
         new_ids = {v.get("id") for v in base_references.get("new_visualizations") or [] if v.get("id")}
         known_ids = {v.get("id") for v in base_references.get("visualizations") or [] if v.get("id")} | new_ids
+        bodies = _bodies_by_id(base_references)
 
     if document is None:
         return DashboardEvaluation(
@@ -1337,6 +1409,11 @@ def evaluate_dashboard_response(
     preserved_failures = (
         _check_preserved_widgets(widgets, expected_output["preserved_widgets"]) if applies.preserved_widgets else []
     )
+    chart_type_failures, chart_type_notes = (
+        _check_chart_types(widgets, new_ids, expected_output.get("visualizations") or [], bodies)
+        if applies.chart_types
+        else ([], [])
+    )
 
     return DashboardEvaluation(
         drafted=True,
@@ -1356,6 +1433,7 @@ def evaluate_dashboard_response(
         tab_filters_correct=not tab_filter_failures,
         date_filters_correct=not date_filter_failures,
         widgets_preserved=not preserved_failures,
+        chart_types_correct=not chart_type_failures,
         failures=[
             *chart_failures,
             *saved_failures,
@@ -1366,9 +1444,10 @@ def evaluate_dashboard_response(
             *tab_filter_failures,
             *date_filter_failures,
             *preserved_failures,
+            *chart_type_failures,
             *new_failures,
         ],
-        notes=[*title_notes, *reference_notes],
+        notes=[*title_notes, *reference_notes, *chart_type_notes],
     )