gooddata-eval 1.78.1.dev1__tar.gz → 1.78.1.dev3__tar.gz

gooddata_eval-1.78.1.dev1/README.md → gooddata_eval-1.78.1.dev3/PKG-INFO

@@ -1,3 +1,32 @@
+Metadata-Version: 2.5
+Name: gooddata-eval
+Version: 1.78.1.dev3
+Summary: Evaluate the GoodData AI agent against your own questions and models.
+Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
+Author-email: GoodData <support@gooddata.com>
+License-Expression: MIT
+License-File: LICENSE.txt
+Keywords: ai,analytics,cli,evaluation,gooddata,llm
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: gooddata-sdk~=1.78.1.dev3
+Requires-Dist: httpx<1.0,>=0.27
+Requires-Dist: jsonpatch<2.0,>=1.33
+Requires-Dist: orjson<4.0.0,>=3.9.15
+Requires-Dist: pydantic<3.0,>=2.6
+Requires-Dist: rich<15.0,>=13.0
+Provides-Extra: llm-judge
+Requires-Dist: openai<2.0,>=1.45; extra == 'llm-judge'
+Description-Content-Type: text/markdown
+
 # gooddata-eval
 
 CLI to evaluate the GoodData AI agent against a dataset of natural-language
@@ -709,6 +738,63 @@ A dataset is a folder of `.json` files, one per question:
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
 `search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
+### `visualization` items
+
+The `expected_output` lists the candidate charts in the AAC format, and the item passes when
+the agent's chart matches one of them. Keys beside the candidates add item-level checks:
+
+```json
+{
+  "id": "sankey-001",
+  "dataset_name": "agent_visualization_extended_charts",
+  "test_kind": "visualization",
+  "question": "Show how net sales flow from product category to customer country.",
+  "expected_output": {
+    "visualization": {
+      "type": "sankey_chart",
+      "query": {"fields": {"m": {"using": "metric/net_sales"}, "c": {"using": "label/product_category"},
+                           "d": {"using": "label/customer_country"}}},
+      "metrics": ["m"],
+      "from": "c",
+      "to": "d",
+      "buckets_strict": true
+    },
+    "requires_visualization_part": true
+  }
+}
+```
+
+Item-level keys, all optional except the candidates:
+
+- `visualization` — one candidate chart, or a list of them. `expected_outputs`
+  (`[{"visualization": {...}}, ...]`) is the same list in the agentic fixture form.
+- `requires_execution` — the agent must also run the chart it built.
+- `expects_visualization: false` — the agent must answer without a chart, for example by asking
+  for a part the request leaves out. The item then lists no candidates.
+- `visualization_optional: true` — either an answer without a chart, or a chart that matches a
+  candidate, is right.
+- `answer_must_include` / `answer_must_not_match` — regular expressions, matched without regard
+  to case against the final answer. Every `answer_must_include` pattern must match, and no
+  `answer_must_not_match` pattern may.
+- `format_intents` — per metric uri, how the metric's numbers must read, for example
+  `{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`.
+- `requires_visualization_part` — the chart must come back as a part of the answer, not only as
+  the arguments of the chart tool call.
+
+Switches on a candidate chart:
+
+- `buckets_strict` — compare bucket by bucket, not only the union of the buckets, so swapped
+  heatmap rows and columns or a reversed sankey fail.
+- `metrics_ordered` — compare the metrics by position, for bullet roles and bubble axes.
+
+Any other key, at item level or on a candidate chart, makes the item error before any request
+is sent, because a misspelled key would otherwise be dropped and its check would never run:
+
+```
+ValueError: the expectation states ['answer_must_inlcude'], which is not one of ['answer_must_include', ...]
+ValueError: a candidate chart states ['bucket_strict'], which the chart model does not read
+```
+
 ### `dashboard_summary` items
 
 Summary items call the dedicated summary endpoint

gooddata_eval-1.78.1.dev1/PKG-INFO → gooddata_eval-1.78.1.dev3/README.md

@@ -1,32 +1,3 @@
-Metadata-Version: 2.5
-Name: gooddata-eval
-Version: 1.78.1.dev1
-Summary: Evaluate the GoodData AI agent against your own questions and models.
-Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
-Author-email: GoodData <support@gooddata.com>
-License-Expression: MIT
-License-File: LICENSE.txt
-Keywords: ai,analytics,cli,evaluation,gooddata,llm
-Classifier: Development Status :: 4 - Beta
-Classifier: Environment :: Console
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Scientific/Engineering
-Classifier: Topic :: Software Development
-Classifier: Typing :: Typed
-Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.78.1.dev1
-Requires-Dist: httpx<1.0,>=0.27
-Requires-Dist: jsonpatch<2.0,>=1.33
-Requires-Dist: orjson<4.0.0,>=3.9.15
-Requires-Dist: pydantic<3.0,>=2.6
-Requires-Dist: rich<15.0,>=13.0
-Provides-Extra: llm-judge
-Requires-Dist: openai<2.0,>=1.45; extra == 'llm-judge'
-Description-Content-Type: text/markdown
-
 # gooddata-eval
 
 CLI to evaluate the GoodData AI agent against a dataset of natural-language
@@ -738,6 +709,63 @@ A dataset is a folder of `.json` files, one per question:
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
 `search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
+### `visualization` items
+
+The `expected_output` lists the candidate charts in the AAC format, and the item passes when
+the agent's chart matches one of them. Keys beside the candidates add item-level checks:
+
+```json
+{
+  "id": "sankey-001",
+  "dataset_name": "agent_visualization_extended_charts",
+  "test_kind": "visualization",
+  "question": "Show how net sales flow from product category to customer country.",
+  "expected_output": {
+    "visualization": {
+      "type": "sankey_chart",
+      "query": {"fields": {"m": {"using": "metric/net_sales"}, "c": {"using": "label/product_category"},
+                           "d": {"using": "label/customer_country"}}},
+      "metrics": ["m"],
+      "from": "c",
+      "to": "d",
+      "buckets_strict": true
+    },
+    "requires_visualization_part": true
+  }
+}
+```
+
+Item-level keys, all optional except the candidates:
+
+- `visualization` — one candidate chart, or a list of them. `expected_outputs`
+  (`[{"visualization": {...}}, ...]`) is the same list in the agentic fixture form.
+- `requires_execution` — the agent must also run the chart it built.
+- `expects_visualization: false` — the agent must answer without a chart, for example by asking
+  for a part the request leaves out. The item then lists no candidates.
+- `visualization_optional: true` — either an answer without a chart, or a chart that matches a
+  candidate, is right.
+- `answer_must_include` / `answer_must_not_match` — regular expressions, matched without regard
+  to case against the final answer. Every `answer_must_include` pattern must match, and no
+  `answer_must_not_match` pattern may.
+- `format_intents` — per metric uri, how the metric's numbers must read, for example
+  `{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`.
+- `requires_visualization_part` — the chart must come back as a part of the answer, not only as
+  the arguments of the chart tool call.
+
+Switches on a candidate chart:
+
+- `buckets_strict` — compare bucket by bucket, not only the union of the buckets, so swapped
+  heatmap rows and columns or a reversed sankey fail.
+- `metrics_ordered` — compare the metrics by position, for bullet roles and bubble axes.
+
+Any other key, at item level or on a candidate chart, makes the item error before any request
+is sent, because a misspelled key would otherwise be dropped and its check would never run:
+
+```
+ValueError: the expectation states ['answer_must_inlcude'], which is not one of ['answer_must_include', ...]
+ValueError: a candidate chart states ['bucket_strict'], which the chart model does not read
+```
+
 ### `dashboard_summary` items
 
 Summary items call the dedicated summary endpoint

{gooddata_eval-1.78.1.dev1 → gooddata_eval-1.78.1.dev3}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.78.1.dev1"
+version = "1.78.1.dev3"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.78.1.dev1",
+    "gooddata-sdk~=1.78.1.dev3",
     "httpx>=0.27,<1.0",
     "jsonpatch>=1.33,<2.0",
     "orjson>=3.9.15,<4.0.0",

{gooddata_eval-1.78.1.dev1 → gooddata_eval-1.78.1.dev3}/src/gooddata_eval/cli/agentic_runner.py

@@ -26,7 +26,7 @@ from gooddata_eval.core.agentic.visualization import evaluate_agentic_visualizat
 from gooddata_eval.core.agentic.what_if import evaluate_agentic_what_if
 from gooddata_eval.core.config import ReasoningEffort
 from gooddata_eval.core.evaluators._llm_judge import JudgeResponseError
-from gooddata_eval.core.evaluators.visualization import requires_execution_of
+from gooddata_eval.core.evaluators.visualization import expectation_options
 from gooddata_eval.core.models import AgenticEvalOutcome, CreatedVisualization, DatasetItem
 from gooddata_eval.core.runner import EvalReport, ItemReport
 from gooddata_eval.core.timing import TraceRecorder
@@ -119,6 +119,8 @@ def _parse_visualization_expected(expected_output: Any) -> list[CreatedVisualiza
       {"visualization": {...}} or {"visualization": [{...}]}  <- single/multi candidate
       [{"visualization": {...}}, ...]                          <- bare list
     """
+    if isinstance(expected_output, dict) and expected_output.get("expects_visualization") is False:
+        return []
     if isinstance(expected_output, dict):
         raw_list = expected_output.get("expected_outputs")
         if raw_list is not None:
@@ -187,7 +189,7 @@ def _dispatch_agentic(
             workspace_id=workspace_id,
             question=item.question,
             expected_outputs=_parse_visualization_expected(eo),
-            requires_execution=requires_execution_of(eo),
+            options=expectation_options(eo),
             k=k,
             gate=gate,
             agent_id=agent_id,

{gooddata_eval-1.78.1.dev1 → gooddata_eval-1.78.1.dev3}/src/gooddata_eval/core/agentic/_catalog.py

@@ -67,6 +67,12 @@ class CatalogMetricAlert:
     """Expected group-by attributes; ``None`` means the fixture states no expectation."""
     granularity: AnomalyDetectionGranularity | None = None
     """Detection interval for an ANOMALY alert (DAY/WEEK/MONTH/...). Not a date filter."""
+    comparison: dict[str, str | int | float | None] | None = None
+    """Expected period comparison as ``create_metric_alert`` argument names; ``None`` means unasserted."""
+    title: str | None = None
+    """Alert name the user asked for; ``None`` means unasserted."""
+    dashboard_id: str | None = None
+    """Dashboard the alert must be bound to; ``None`` means unasserted."""
 
     @classmethod
     def from_dict(cls, d: dict) -> CatalogMetricAlert:
@@ -87,4 +93,7 @@ class CatalogMetricAlert:
             filters=d.get("filters"),
             attributes=d.get("attributes"),
             granularity=AnomalyDetectionGranularity.parse(d.get("granularity")),
+            comparison=d.get("comparison"),
+            title=d.get("title"),
+            dashboard_id=d.get("dashboard_id"),
         )