gooddata-eval 1.78.0__tar.gz → 1.78.1.dev1__tar.gz

{gooddata_eval-1.78.0 → gooddata_eval-1.78.1.dev1}/AGENTS.md

@@ -27,6 +27,7 @@ experiment. The newest and most actively developed package in the repo.
 | `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
 | `core/chat/` | SSE client for the agent's streaming chat endpoint |
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
+| `core/takeaway/` | HTTP client for the AI Publisher takeaway endpoint — a single-shot backend like `core/summary/`, keeping the whole response on `ChatResult.raw_response` — plus `evaluate_document_takeaway`, the K-run, gate and Langfuse entry point the Tavern shim calls |
 | `core/dataset/` | dataset format, loading, and `from_insights.py` — dataset generation from a workspace's real insights |
 | `core/evaluators/` | single-shot evaluators and their registry |
 | `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `item_scope` (an item's experiment scope sent to gen-ai as baggage, and which conversations joined), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
@@ -59,7 +60,8 @@ implementation, sometimes under different `test_kind` strings (`search_tool` vs
 `expected_output: Any`. `expected_output` is deliberately untyped — each evaluator parses
 its own shape. `test_kind` on the item is what labels the result, not the evaluator class,
 which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
-`dashboard_summary` items additionally need `summary_input`.
+`dashboard_summary` items additionally need `summary_input`, `document_takeaway` items
+`takeaway_input`.
 
 ## Running the pipeline
 
@@ -155,8 +157,8 @@ Run this before guessing a `--model` string.
 | `GD_EVAL_JUDGE_MODEL` | judge model, same as `--judge-model` |
 | `GD_EVAL_AGENT_ID` | which agent to drive, same as `--agent-id` |
 | `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` | `--langfuse`, `--langfuse-dataset` |
-| `GOODDATA_EVAL_CHAT_*` | SSE retry, backoff and timeout knobs |
-| `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary runs are not labelled |
+| `GOODDATA_EVAL_CHAT_*` | retry and backoff knobs shared by the chat and takeaway clients (`core/_retry.py`), and the SSE turn and item timeouts |
+| `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary and document-takeaway runs are not labelled |
 | `GOODDATA_EVAL_JOIN_GENAI_TRACE` | on unless set to `0`, `false`, `no`, `off` or empty. An agentic item's chat turns ask gen-ai to open them under gd-eval's experiment root, so each item is one Langfuse trace with its scores written once. A gen-ai that drops caller baggage keeps today's separate traces; see the gotcha below |
 | `GD_EVAL_TIMERS` | same as `--timers` |
 
@@ -181,12 +183,15 @@ That default is correct: agent tool calls create real server-side objects (metri
 Adding a kind to `PARALLEL_SAFE_TEST_KINDS` is a deliberate assertion that it is read-only,
 which nothing in the package can prove for you.
 
-**The SSE client's retry predicate is load-bearing.** `core/chat/sse_client.py` retries
-429/502/503/504 and `httpx.RemoteProtocolError` (a mid-stream disconnect) with exponential
-backoff, and treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The
-`RemoteProtocolError` case was added after it was confirmed live to contaminate a small
-percentage of visualization runs with a hard fail and no retry. Narrowing that predicate
-reintroduces the problem.
+**The SSE client's retry predicate is load-bearing.** The backoff loop and the retryable
+status set (429/500/502/503/504) live in `core/_retry.py`, shared by the chat and takeaway
+clients and tuned by the same `GOODDATA_EVAL_CHAT_*` knobs. `core/chat/sse_client.py`
+retries that status set and `httpx.RemoteProtocolError` (a mid-stream disconnect), and
+treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The `RemoteProtocolError` case
+was added after it was confirmed live to contaminate a small percentage of visualization
+runs with a hard fail and no retry. Narrowing that predicate reintroduces the problem. The
+takeaway client retries the same status set plus connection failures, but not a read
+timeout: a slow page would spend the whole timeout again on every attempt.
 
 **Langfuse trace linking is deliberately off the item critical path.** Polling for trace
 ingestion has no pass/fail signal and inflates or misattributes per-item latency, so

gooddata_eval-1.78.1.dev1/Makefile

@@ -0,0 +1,6 @@
+# (C) 2026 GoodData Corporation
+include ../../project_common.mk
+
+.PHONY: calibrate-takeaway
+calibrate-takeaway:
+	uv run --package gooddata-eval --extra llm-judge python scripts/calibrate_takeaway_judge.py

{gooddata_eval-1.78.0 → gooddata_eval-1.78.1.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.78.0
+Version: 1.78.1.dev1
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,7 +17,7 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.78.0
+Requires-Dist: gooddata-sdk~=1.78.1.dev1
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: jsonpatch<2.0,>=1.33
 Requires-Dist: orjson<4.0.0,>=3.9.15
@@ -161,8 +161,8 @@ gd-eval run \
 | `--runs K` | `2` | Independent runs per item. |
 | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
-| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
-| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
+| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. |
 
 **Concurrency and workspace safety.** Agentic kinds that create workspace objects
 (`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
@@ -736,7 +736,7 @@ A dataset is a folder of `.json` files, one per question:
 ```
 
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
-`search_tool`, `general_question`, `guardrail`, `dashboard_summary`.
+`search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
 ### `dashboard_summary` items
 
@@ -775,6 +775,64 @@ The `expected_output` rubric:
 Each criterion is scored independently by the LLM judge, so `quality_score`
 is the fraction of satisfied criteria.
 
+### `document_takeaway` items
+
+Takeaway items call the AI Publisher takeaway endpoint
+(`POST /api/v1/ai/workspaces/{ws}/takeaway`) with a `takeaway_input` block, authored in
+snake_case and sent as the endpoint's camelCase body. In a Langfuse dataset the block goes
+in the item's `metadata` (or its `input` object), and `input` can stay a plain case label;
+an item carrying `takeaway_input` is a `document_takeaway` item unless it declares another
+`test_kind`. The endpoint needs the `enableGenAiReportTakeaways` feature flag.
+
+```json
+{
+  "id": "takeaway-001",
+  "dataset_name": "agent_document_takeaway",
+  "test_kind": "document_takeaway",
+  "question": "P01 default format, Sales page, no instructions",
+  "takeaway_input": {
+    "visualizations": ["<visualization id>", {"visualization_id": "<id>", "filters": []}],
+    "filter_context": [],
+    "instructions": {"document": null, "slot": null, "format_hint": null},
+    "max_characters": 300
+  },
+  "expected_output": {
+    "no_data": false,
+    "format": {"no_lists": false},
+    "must_include": ["States one finding that connects at least two of the charts"],
+    "must_not_include": [],
+    "rubric": []
+  }
+}
+```
+
+`takeaway_input` requires `visualizations`; unknown keys are rejected. Every
+`expected_output` key is optional:
+
+- `no_data` — whether the endpoint must answer `noData`; when either side says no data,
+  nothing else is graded.
+- `format` — overrides of the deterministic checks, which gate pass/fail: `bold_headline`,
+  `max_headline_words` (10), `max_sentences` (2), `max_words` (50), `single_paragraph`,
+  `no_lists`, `no_headings`. `false` or `null` switches a check off; unknown keys are
+  rejected. `length` (`max_characters`), `no_ids` and `no_meta` always apply.
+- `default_rules` — `false` drops the default judge criteria (one finding; no executive
+  summary, chart walk-through, background opener, figures, dates, causes the data cannot
+  show, chart description).
+- `must_include`, `must_not_include`, `rubric` — the item's own judge criteria, graded like
+  `dashboard_summary`.
+
+`word_count` and `headline_word_count` are reported in the item detail. The default judge
+criteria are calibrated with `make calibrate-takeaway` (needs `OPENAI_API_KEY`); re-run it
+after changing their wording.
+
+The client retries 429/5xx answers and connection failures. A request error that survives
+the retries is handled differently by the two entry points: `gd-eval run` aborts the item
+with that error, whatever `--runs` is, while the CI entry point
+(`evaluate_document_takeaway`) records it as an errored run, keeps the item's other runs,
+and raises only when no run completed. A read timeout is not retried, and the CI entry
+point records it as an errored run too. Any other 4xx and an unsupported URL scheme are
+never retried and fail the item in both.
+
 ### Items asked in a dashboard context (`user_context`)
 
 An item can be asked the way a user asks from an open dashboard or an attached widget.
@@ -782,7 +840,8 @@ Its `user_context` is sent verbatim as `userContext` on every chat message of th
 follow-up and clarification messages included — the server treats a message without one
 as a cleared context. Requires the `enableAiContextSetup` feature flag on the target
 organization — without it the server ignores the dashboard view. Applies to chat items
-only; `dashboard_summary` items carry their scope in `summary_input` instead. No
+only; `dashboard_summary` and `document_takeaway` items carry their scope in
+`summary_input` / `takeaway_input` instead. No
 `userContext` key is sent at all when neither the item nor, for `agentic_conversation`,
 the current turn sets one.
 
@@ -835,6 +894,7 @@ from that turn on until another turn changes it, as the attached context does in
 | `general_question` | Text answer judged by LLM | `[llm-judge]` |
 | `guardrail` | Refusal/redirect (visualization response auto-fails) | `[llm-judge]` |
 | `dashboard_summary` | Dashboard summary (via `/summary` endpoint) scored against a rubric by LLM | `[llm-judge]` |
+| `document_takeaway` | One short takeaway for a page (via `/takeaway` endpoint): deterministic format checks plus a rubric scored by LLM | `[llm-judge]` |
 
 ## Optional extras
 

{gooddata_eval-1.78.0 → gooddata_eval-1.78.1.dev1}/README.md

@@ -132,8 +132,8 @@ gd-eval run \
 | `--runs K` | `2` | Independent runs per item. |
 | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
-| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
-| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
+| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. |
 
 **Concurrency and workspace safety.** Agentic kinds that create workspace objects
 (`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
@@ -707,7 +707,7 @@ A dataset is a folder of `.json` files, one per question:
 ```
 
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
-`search_tool`, `general_question`, `guardrail`, `dashboard_summary`.
+`search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
 ### `dashboard_summary` items
 
@@ -746,6 +746,64 @@ The `expected_output` rubric:
 Each criterion is scored independently by the LLM judge, so `quality_score`
 is the fraction of satisfied criteria.
 
+### `document_takeaway` items
+
+Takeaway items call the AI Publisher takeaway endpoint
+(`POST /api/v1/ai/workspaces/{ws}/takeaway`) with a `takeaway_input` block, authored in
+snake_case and sent as the endpoint's camelCase body. In a Langfuse dataset the block goes
+in the item's `metadata` (or its `input` object), and `input` can stay a plain case label;
+an item carrying `takeaway_input` is a `document_takeaway` item unless it declares another
+`test_kind`. The endpoint needs the `enableGenAiReportTakeaways` feature flag.
+
+```json
+{
+  "id": "takeaway-001",
+  "dataset_name": "agent_document_takeaway",
+  "test_kind": "document_takeaway",
+  "question": "P01 default format, Sales page, no instructions",
+  "takeaway_input": {
+    "visualizations": ["<visualization id>", {"visualization_id": "<id>", "filters": []}],
+    "filter_context": [],
+    "instructions": {"document": null, "slot": null, "format_hint": null},
+    "max_characters": 300
+  },
+  "expected_output": {
+    "no_data": false,
+    "format": {"no_lists": false},
+    "must_include": ["States one finding that connects at least two of the charts"],
+    "must_not_include": [],
+    "rubric": []
+  }
+}
+```
+
+`takeaway_input` requires `visualizations`; unknown keys are rejected. Every
+`expected_output` key is optional:
+
+- `no_data` — whether the endpoint must answer `noData`; when either side says no data,
+  nothing else is graded.
+- `format` — overrides of the deterministic checks, which gate pass/fail: `bold_headline`,
+  `max_headline_words` (10), `max_sentences` (2), `max_words` (50), `single_paragraph`,
+  `no_lists`, `no_headings`. `false` or `null` switches a check off; unknown keys are
+  rejected. `length` (`max_characters`), `no_ids` and `no_meta` always apply.
+- `default_rules` — `false` drops the default judge criteria (one finding; no executive
+  summary, chart walk-through, background opener, figures, dates, causes the data cannot
+  show, chart description).
+- `must_include`, `must_not_include`, `rubric` — the item's own judge criteria, graded like
+  `dashboard_summary`.
+
+`word_count` and `headline_word_count` are reported in the item detail. The default judge
+criteria are calibrated with `make calibrate-takeaway` (needs `OPENAI_API_KEY`); re-run it
+after changing their wording.
+
+The client retries 429/5xx answers and connection failures. A request error that survives
+the retries is handled differently by the two entry points: `gd-eval run` aborts the item
+with that error, whatever `--runs` is, while the CI entry point
+(`evaluate_document_takeaway`) records it as an errored run, keeps the item's other runs,
+and raises only when no run completed. A read timeout is not retried, and the CI entry
+point records it as an errored run too. Any other 4xx and an unsupported URL scheme are
+never retried and fail the item in both.
+
 ### Items asked in a dashboard context (`user_context`)
 
 An item can be asked the way a user asks from an open dashboard or an attached widget.
@@ -753,7 +811,8 @@ Its `user_context` is sent verbatim as `userContext` on every chat message of th
 follow-up and clarification messages included — the server treats a message without one
 as a cleared context. Requires the `enableAiContextSetup` feature flag on the target
 organization — without it the server ignores the dashboard view. Applies to chat items
-only; `dashboard_summary` items carry their scope in `summary_input` instead. No
+only; `dashboard_summary` and `document_takeaway` items carry their scope in
+`summary_input` / `takeaway_input` instead. No
 `userContext` key is sent at all when neither the item nor, for `agentic_conversation`,
 the current turn sets one.
 
@@ -806,6 +865,7 @@ from that turn on until another turn changes it, as the attached context does in
 | `general_question` | Text answer judged by LLM | `[llm-judge]` |
 | `guardrail` | Refusal/redirect (visualization response auto-fails) | `[llm-judge]` |
 | `dashboard_summary` | Dashboard summary (via `/summary` endpoint) scored against a rubric by LLM | `[llm-judge]` |
+| `document_takeaway` | One short takeaway for a page (via `/takeaway` endpoint): deterministic format checks plus a rubric scored by LLM | `[llm-judge]` |
 
 ## Optional extras
 

{gooddata_eval-1.78.0 → gooddata_eval-1.78.1.dev1}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.78.0"
+version = "1.78.1.dev1"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.78.0",
+    "gooddata-sdk~=1.78.1.dev1",
     "httpx>=0.27,<1.0",
     "jsonpatch>=1.33,<2.0",
     "orjson>=3.9.15,<4.0.0",

gooddata_eval-1.78.1.dev1/scripts/calibrate_takeaway_judge.py

@@ -0,0 +1,75 @@
+# (C) 2026 GoodData Corporation
+"""Does the real judge put each calibration text on the right side of every default criterion?
+
+tests/test_takeaway_evaluator.py pins the wiring and the criterion wording; it cannot pin
+the verdict, because the verdict is an LLM's. This does, against the texts in
+tests/fixtures/takeaway_calibration.json:
+
+- a "pass" text must satisfy every default judge criterion on every run;
+- a "fail" text must violate each criterion it names on every run.
+
+Run it after changing any DEFAULT_MUST_INCLUDE / DEFAULT_MUST_NOT_INCLUDE wording:
+
+    OPENAI_API_KEY=... make calibrate-takeaway
+
+It is a script rather than a test because CI has no OPENAI_API_KEY. Exits 1 on any mismatch.
+"""
+
+import json
+import os
+import sys
+from pathlib import Path
+
+from gooddata_eval.core.evaluators.takeaway import (
+    DEFAULT_MUST_INCLUDE,
+    DEFAULT_MUST_NOT_INCLUDE,
+    DocumentTakeawayEvaluator,
+)
+from gooddata_eval.core.models import ChatResult, DatasetItem, TakeawayInput
+
+RUNS = 3
+FIXTURE = Path(__file__).resolve().parent.parent / "tests" / "fixtures" / "takeaway_calibration.json"
+
+CRITERIA = {f"default_include_{k}": k for k in DEFAULT_MUST_INCLUDE} | {
+    f"default_exclude_{k}": k for k in DEFAULT_MUST_NOT_INCLUDE
+}
+
+
+def main() -> int:
+    if not os.environ.get("OPENAI_API_KEY"):
+        raise SystemExit("OPENAI_API_KEY required")
+    fixture = json.loads(FIXTURE.read_text())
+    evaluator = DocumentTakeawayEvaluator()
+    print(f"judge model: {evaluator._positive_judge.model}, runs: {RUNS}")
+    mismatches = 0
+    for case in fixture["texts"]:
+        item = DatasetItem(
+            id=case["label"],
+            dataset_name="takeaway_calibration",
+            test_kind="document_takeaway",
+            question=fixture["question"],
+            expected_output={},
+            takeaway_input=TakeawayInput(visualizations=["calibration"]),
+        )
+        chat = ChatResult(text_response=case["text"], raw_response={"takeaway": case["text"], "noData": False})
+        satisfied = dict.fromkeys(CRITERIA.values(), 0)
+        for _ in range(RUNS):
+            detail = evaluator.evaluate(item, chat).detail
+            for key, name in CRITERIA.items():
+                satisfied[name] += detail.get(key) is True
+        want_pass = case["want"] == "pass"
+        must = dict.fromkeys(satisfied, RUNS) if want_pass else dict.fromkeys(case["criteria"], 0)
+        print(f"\n{case['label']} (want {case['want']})")
+        for name, count in satisfied.items():
+            flag = ""
+            if name in must:
+                ok = count == must[name]
+                mismatches += not ok
+                flag = "OK" if ok else "MISMATCH"
+            print(f"  {name:20} satisfied {count}/{RUNS}  {flag}")
+    print(f"\n{'all calibration texts on the right side' if not mismatches else f'{mismatches} mismatch(es)'}")
+    return 1 if mismatches else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{gooddata_eval-1.78.0 → gooddata_eval-1.78.1.dev1}/src/gooddata_eval/cli/main.py

@@ -7,7 +7,7 @@ import sys
 import threading
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import get_args
+from typing import Any, get_args
 
 import httpx
 from gooddata_api_client.exceptions import ApiException, ApiTypeError
@@ -36,32 +36,31 @@ from gooddata_eval.core.reporting.html_report import load_report_files, write_ht
 from gooddata_eval.core.reporting.json_report import build_multi_model_report, write_multi_model_report
 from gooddata_eval.core.runner import ItemReport, run_items
 from gooddata_eval.core.summary.http_client import SummaryClient
+from gooddata_eval.core.takeaway.http_client import TakeawayClient
 from gooddata_eval.core.timing import TIMERS_ENV_VAR
 from gooddata_eval.core.workspace import ModelResolutionError, WorkspaceModelController
 
 _EXIT_OK = 0
 _EXIT_OPERATIONAL_ERROR = 2
-_SUMMARY_TEST_KIND = "dashboard_summary"
 
 
 class _RoutingBackend:
     """Dispatch each item to the right backend by test_kind.
 
-    `dashboard_summary` items go to the dedicated summary endpoint; everything
-    else uses the conversational chat endpoint.
+    Kinds in `by_kind` go to their dedicated endpoint client (`dashboard_summary` to the
+    summary endpoint, `document_takeaway` to the takeaway endpoint); everything else uses
+    the conversational chat endpoint.
     """
 
-    def __init__(self, chat: ChatClient, summary: SummaryClient):
+    def __init__(self, chat: ChatClient, by_kind: dict[str, Any]):
         self._chat = chat
-        self._summary = summary
+        self._by_kind = by_kind
 
     def ask(self, item: DatasetItem) -> ChatResult:
-        if item.test_kind == _SUMMARY_TEST_KIND:
-            return self._summary.ask(item)
-        return self._chat.ask(item)
+        return self._by_kind.get(item.test_kind, self._chat).ask(item)
 
     def close(self) -> None:
-        for backend in (self._chat, self._summary):
+        for backend in (self._chat, *self._by_kind.values()):
             if hasattr(backend, "close"):
                 backend.close()
 
@@ -614,7 +613,14 @@ def _run(config: RunConfig) -> int:
                     turn_timeout_s=config.turn_timeout_s,
                     item_timeout_s=config.item_timeout_s,
                 ),
-                SummaryClient(host=config.host, token=config.token, workspace_id=config.workspace_id),
+                {
+                    "dashboard_summary": SummaryClient(
+                        host=config.host, token=config.token, workspace_id=config.workspace_id
+                    ),
+                    "document_takeaway": TakeawayClient(
+                        host=config.host, token=config.token, workspace_id=config.workspace_id
+                    ),
+                },
             )
             try:
                 single_report = run_items(

gooddata_eval-1.78.1.dev1/src/gooddata_eval/core/_retry.py

@@ -0,0 +1,71 @@
+# (C) 2026 GoodData Corporation
+"""Bounded exponential-backoff retry shared by the gen-ai HTTP clients.
+
+The budget is read from the ``GOODDATA_EVAL_CHAT_*`` environment on every call, so the chat
+and takeaway clients retune together.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import time
+from collections.abc import Callable
+from typing import TypeVar
+
+_log = logging.getLogger(__name__)
+
+T = TypeVar("T")
+
+# 500 is here on evidence, not on principle: in one visualization eval batch it hard-failed
+# 10 of 56 runs with zero retry attempts, and every affected question scored normally when the
+# same question/model ran again the next day -- i.e. transient gen-ai faults, not deterministic
+# server bugs. A genuinely deterministic 500 still terminates, just after the bounded backoff.
+RETRYABLE_STATUS_CODES: frozenset[int] = frozenset({429, 500, 502, 503, 504})
+
+
+def int_env(name: str, default: int) -> int:
+    """Read an int from the environment, falling back to ``default`` when unset or blank."""
+    raw = os.getenv(name)
+    return int(raw) if raw else default
+
+
+def float_env(name: str, default: float) -> float:
+    """Read a float from the environment, falling back to ``default`` when unset or blank."""
+    raw = os.getenv(name)
+    return float(raw) if raw else default
+
+
+# Retry budget defaults, giving a ~2 min worst-case cap per send (5/10/20/40/60s).
+# Each is overridable via env so CI can retune without cutting a new release --
+# read per call rather than at import, so an exported value cannot silently
+# rewrite what a test that patches these attributes expects.
+_MAX_RETRIES_DEFAULT = 5
+_INITIAL_BACKOFF_S_DEFAULT = 5.0
+_BACKOFF_FACTOR_DEFAULT = 2.0
+_MAX_BACKOFF_S_DEFAULT = 60.0
+
+
+def retry_transient(operation: Callable[[], T], *, is_retryable: Callable[[Exception], bool]) -> T:
+    """Run ``operation``; retry retryable failures with bounded exponential backoff."""
+    max_retries = int_env("GOODDATA_EVAL_CHAT_MAX_RETRIES", _MAX_RETRIES_DEFAULT)
+    delay = float_env("GOODDATA_EVAL_CHAT_INITIAL_BACKOFF_S", _INITIAL_BACKOFF_S_DEFAULT)
+    factor = float_env("GOODDATA_EVAL_CHAT_BACKOFF_FACTOR", _BACKOFF_FACTOR_DEFAULT)
+    max_backoff = float_env("GOODDATA_EVAL_CHAT_MAX_BACKOFF_S", _MAX_BACKOFF_S_DEFAULT)
+    for attempt in range(max_retries + 1):  # 0..N => N retries + 1 initial attempt
+        try:
+            return operation()
+        except Exception as exc:  # noqa: PERF203 — retry loop: per-attempt try/except is intentional
+            if attempt == max_retries or not is_retryable(exc):
+                raise
+            sleep_s = min(delay, max_backoff)
+            _log.warning(
+                "Transient gen-ai error (attempt %d/%d): %s; retrying in %.0fs",
+                attempt + 1,
+                max_retries + 1,
+                exc,
+                sleep_s,
+            )
+            time.sleep(sleep_s)
+            delay *= factor
+    raise AssertionError("unreachable")  # loop either returns or raises

{gooddata_eval-1.78.0 → gooddata_eval-1.78.1.dev1}/src/gooddata_eval/core/agentic/_conversation_context.py

@@ -37,14 +37,36 @@ CONFIRMATION_REPLY = "Yes, go ahead."
 # Asked to pick from a list (which dashboard to bind an alert to), a real user picks one.
 NO_ANSWER_REPLY = "Use your best judgement and go ahead. If you need me to pick from a list, take the first option."
 
-# Part of the metric and alert skills' designed flow, not a clarification.
+# Part of the metric and alert skills' designed flow, not a clarification; so is a retry of a
+# corrected proposal. Keep `use` and `run` out: "Which one should I use?" is a choice question.
+_RETRY_VERBS = r"retry|try\s+(?:(?:it|this|that)\s+)?(?:again|once\s+more)|resubmit"
+_APPROVAL_VERBS = rf"go\s+ahead|proceed|create|save|update|apply|{_RETRY_VERBS}"
+_SHOULD_I = r"\b(?:should|shall|may|can)\s+i\s+"
+_WANT_ME_TO = r"\b(?:do\s+you\s+want|would\s+you\s+like)\s+me\s+to\s+"
 _CONFIRMATION_RE = re.compile(
-    r"\b(?:should|shall|may|can)\s+i\s+(?:go\s+ahead|proceed|create|save|update|apply|add|set\s+(?:it|this)\s+up)"
-    r"|\bdo\s+you\s+want\s+me\s+to\s+(?:go\s+ahead|proceed|create|save|update|apply)"
-    r"|\bwould\s+you\s+like\s+me\s+to\s+(?:go\s+ahead|proceed|create|save|update|apply)"
+    rf"{_SHOULD_I}(?:{_APPROVAL_VERBS}|add|set\s+(?:it|this)\s+up)\b"
+    rf"|{_WANT_ME_TO}(?:{_APPROVAL_VERBS})\b"
     r"|\bplease\s+confirm\b|\bconfirm\s+(?:if|whether|that)\b",
     re.IGNORECASE,
 )
+# "Should I retry with gross or net revenue?" offers a choice, so it stays a question, as does
+# a second option asked right after ("Should I retry? Or adjust the formula first?"). The
+# clause runs to the sentence end: a dot inside a token ("2.5") does not end it. Only a
+# prose "or"/"Or" counts: MAQL writes its OR in uppercase. An "or" that hands the turn back
+# to the user ("..., or would you like to adjust it?", "or not?") is not a second option.
+_NOT_AN_OPTION = (
+    r"not\b|(?:would|do)\s+you\s+(?:like|want|prefer)\s+to\b|(?:is|are)\s+there\b"
+    r"|anything\b|something\b|if\s+you\b|let\s+me\s+know\b"
+)
+_RETRY_CHOICE_RE = re.compile(
+    rf"(?:{_SHOULD_I}|{_WANT_ME_TO})(?:{_RETRY_VERBS})\b(?:[^?.!]|\.(?=\S))*"
+    rf"(?:[?.!]\s+)?(?:(?-i:\b[Oo]r\b)|\balternatively\b)(?!\s+(?:{_NOT_AN_OPTION}))",
+    re.IGNORECASE,
+)
+# Inline code, {type/id} references and quoted literals are the proposal itself; an OR
+# inside a formula is not a choice offered to the user. A single quote counts only outside a
+# word, so the apostrophes in "don't ... it's" do not mask the text between them.
+_FORMULA_SPAN_RE = re.compile(r"`+[^`]*`+|\{[^{}]*\}|\"[^\"\n]*\"|(?<!\w)'[^'\n]*'(?!\w)")
 
 # Checked after _CONFIRMATION_RE, so "please confirm" stays a confirmation.
 _REQUEST_RE = re.compile(
@@ -77,7 +99,7 @@ def classify_reply(chat_result: ChatResult, rendered_text: str) -> ReplyKind:
     text = (rendered_text or "").strip()
     if not text:
         return "no_action"
-    if _CONFIRMATION_RE.search(text):
+    if _CONFIRMATION_RE.search(text) and not _RETRY_CHOICE_RE.search(_FORMULA_SPAN_RE.sub(" ", text)):
         return "confirmation"
     if "?" in text or _REQUEST_RE.search(text):
         return "question"
@@ -238,6 +260,33 @@ class ClarificationJudge:
 
 _SIM_USER_MODEL = "gpt-4o"
 
+# The simulated user plays a business user: choosing the formula is the assistant's job.
+_NO_DEFINITIONS = (
+    "Never write MAQL, SQL, formulas or object definitions. If asked to approve one, reply that it is fine to go ahead."
+)
+# An uppercase SELECT counts once a reference, `(`, `*`, DISTINCT or a call follows it, so
+# "SELECT the second option" and "Select the first one" are left alone; in any case it counts
+# once an aggregation call follows, as in "select sum(revenue)". An
+# aggregation call is a formula even without SELECT: in uppercase ("SUM (units)") or in any
+# case with the parenthesis attached ("sum(units) / count(orders)"), so prose such as
+# "Count (all) is fine" is left alone. Lowercase SQL counts by its shape (`*`, a column list,
+# or a table followed by a clause), so "Select the first option from the list" is left alone.
+_MAQL_RE = re.compile(
+    r"\bSELECT\s+(?:[{<(*]|DISTINCT\b|\w+\s*\()|(?i:\bselect\s+(?:sum|count|avg|min|max|median|distinct)\s*\()"
+    r"|(?i:\bselect\s+(?:\*|[\w.]+(?:\s*,\s*[\w.]+)+)\s+from\b)"
+    r"|(?i:\bselect\b[^?!]*?\bfrom\s+[\w.]+\s+(?:where|group\s+by|order\s+by|join)\b)"
+    r"|\b(?:SUM|COUNT|AVG|MIN|MAX|MEDIAN)\s*\(|(?i:\b(?:sum|count|avg|min|max|median)\()"
+    r"|[{<](?:metric|fact|label|attribute|dataset)/"
+)
+
+
+def _without_maql(reply: str | None) -> str | None:
+    """A reply that slipped MAQL in past the prompt leaves the choice to the assistant.
+
+    Both callers answer a question, so the fallback must answer one too, not just approve.
+    """
+    return NO_ANSWER_REPLY if reply and _MAQL_RE.search(reply) else reply
+
 
 def _chat(prompt_system: str, prompt_user: str) -> str | None:
     try:
@@ -271,12 +320,12 @@ def reply_from_facts(transcript: str, question: str, facts: list[str]) -> str:
         return NO_ANSWER_REPLY
     reply = _chat(
         "You play a business user talking to an analytics assistant. Answer its question briefly, "
-        "using ONLY the facts listed below. Never invent requirements. If the facts do not answer "
-        f"the question, reply exactly: {NO_ANSWER_REPLY}",
+        f"using ONLY the facts listed below. Never invent requirements. {_NO_DEFINITIONS} If the facts "
+        f"do not answer the question, reply exactly: {NO_ANSWER_REPLY}",
         f"Conversation so far:\n{transcript}\n\nThe assistant now asks:\n{question}\n\n"
         "Facts you know:\n" + "\n".join(f"- {f}" for f in facts),
     )
-    return reply or NO_ANSWER_REPLY
+    return _without_maql(reply) or NO_ANSWER_REPLY
 
 
 def reply_restating(transcript: str, question: str) -> str:
@@ -289,7 +338,7 @@ def reply_restating(transcript: str, question: str) -> str:
         "You play a business user talking to an analytics assistant. The assistant just asked for "
         "something that was already said or found earlier in the conversation. Reply briefly, "
         "starting with 'As I said earlier,' and repeat only that information, taken from the "
-        "conversation. Do not add anything new.",
+        f"conversation. Do not add anything new. {_NO_DEFINITIONS}",
         f"Conversation so far:\n{transcript}\n\nThe assistant now asks:\n{question}",
     )
-    return reply or "As I said earlier in this conversation — please use what we already established."
+    return _without_maql(reply) or "As I said earlier in this conversation — please use what we already established."