gooddata-eval 1.74.1.dev5__tar.gz → 1.75.1.dev1__tar.gz

{gooddata_eval-1.74.1.dev5 → gooddata_eval-1.75.1.dev1}/AGENTS.md

@@ -9,7 +9,7 @@ experiment. The newest and most actively developed package in the repo.
 
 ## Owns
 
-- The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
+- The `gd-eval` CLI (`generate`, `run`, `report`, `models`)
 - Dataset loading and the evaluation run loop
 - Per-capability evaluators and their scoring
 - Result reporting, and pushing experiments, scores and trace links to Langfuse
@@ -27,7 +27,7 @@ experiment. The newest and most actively developed package in the repo.
 | `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
 | `core/chat/` | SSE client for the agent's streaming chat endpoint |
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
-| `core/dataset/` | dataset format and loading |
+| `core/dataset/` | dataset format, loading, and `from_insights.py` — dataset generation from a workspace's real insights |
 | `core/evaluators/` | single-shot evaluators and their registry |
 | `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
 | `core/reporting/` | console and JSON output rendering |
@@ -61,6 +61,106 @@ its own shape. `test_kind` on the item is what labels the result, not the evalua
 which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
 `dashboard_summary` items additionally need `summary_input`.
 
+## Running the pipeline
+
+Four subcommands, in the order you use them. Everything runs through `uv`; never a bare
+`python`. There is no build step -- `uv run` syncs the environment from `uv.lock` on first
+use, so a fresh clone needs nothing but:
+
+```bash
+uv run --package gooddata-eval gd-eval <subcommand> --help
+```
+
+`openai` is an optional extra (`llm-judge`) so the published package stays installable
+without it, but the `dev` dependency group pulls it in, which is why a plain `uv run` here
+has the phrasing step and the LLM judge. Installing `gooddata-eval` from PyPI does not --
+there the extra is explicit, and every `openai` import site is guarded or deferred.
+
+Connection is the same for every subcommand that talks to the platform: `--host` +
+`--token`, or `GOODDATA_TOKEN` in the environment, or `--profile <name>` reading
+`~/.gooddata/profiles.yaml`. Precedence is flags > env > profile.
+
+### 1. `generate` — build a dataset from a workspace
+
+Reverse-engineers `visualization` items out of the charts a workspace already has, so the
+expected output is copied from a live object rather than invented. Needs `OPENAI_API_KEY`
+for the phrasing step, or `--no-phrase` to emit mechanical `Show <title>` questions.
+
+```bash
+uv run --package gooddata-eval gd-eval generate \
+  --host "$GOODDATA_HOST" --workspace "$WORKSPACE_ID" \
+  --dataset-name ecommerce --out ./datasets/ecommerce \
+  --snapshot-out /tmp/ws.json \
+  --phrase-model gpt-4o --skip-ambiguous
+```
+
+Iterate offline instead of re-fetching: `--snapshot-out` writes everything the generator
+read as one JSON file, and `--snapshot-in` replays it with no host, token or network. Add
+`--dry-run` to print the shape counts and each spec's brief without writing anything —
+the fastest way to see what a workspace yields.
+
+Quality gates fail the command (exit 1) below `--min-questions` (15) or `--min-filtered` (1). Lower them for a smoke test; do not lower them to ship a dataset.
+`--langfuse-out` additionally writes a Langfuse-importable file, and `--id-prefix` rewrites
+ids on that export only, because Langfuse item ids are unique per project.
+
+Insights the AAC spec cannot express without guessing are skipped with a printed reason
+(`SKIP <id>: derived measure (previousPeriodMeasure)`). Read those — they are the
+generator telling you what it refused to invent, not noise.
+
+### 2. `run` — evaluate
+
+```bash
+uv run --package gooddata-eval gd-eval run \
+  --host "$GOODDATA_HOST" --workspace "$WORKSPACE_ID" \
+  --dataset ./datasets/ecommerce --kind visualization \
+  --model gpt-5.2 --model ProviderName/gpt-4o \
+  --runs 3 --gate power --concurrency 4 \
+  --json ./results/run.json --html ./results/run.html
+```
+
+`--dataset` reads a local folder; `--langfuse-dataset` pulls one by name instead. `--kind`
+only supplies a default for items that do not carry their own `test_kind`. Repeat
+`--model` to compare models in one run. `--runs` with `--gate power` measures stability
+(every run must pass) rather than pass@K. `--langfuse` pushes the run as a scored
+experiment, needing `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY`.
+
+`--concurrency` is capped for you where it matters: kinds that create workspace objects
+run one at a time regardless, see the parallel-safety gotcha below.
+
+### 3. `report` — compare runs
+
+```bash
+uv run --package gooddata-eval gd-eval report \
+  ./results/*.json -o ./results/comparison.html --title "luna vs 4o" --redact
+```
+
+Several JSON reports become side-by-side columns keyed by file name. `--redact` is the
+customer-safe form: conversation ids, response ids and raw reasoning dropped, model names
+replaced with "Model A", "Model B".
+
+### 4. `models` — what the org has configured
+
+```bash
+uv run --package gooddata-eval gd-eval models --host "$GOODDATA_HOST"
+```
+
+Run this before guessing a `--model` string.
+
+### Environment
+
+| Variable | Used by |
+|---|---|
+| `GOODDATA_TOKEN` | every platform-facing subcommand |
+| `OPENAI_API_KEY` | `generate` phrasing, and the LLM-as-judge evaluators |
+| `GD_EVAL_JUDGE_MODEL` | judge model, same as `--judge-model` |
+| `GD_EVAL_AGENT_ID` | which agent to drive, same as `--agent-id` |
+| `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` | `--langfuse`, `--langfuse-dataset` |
+| `GOODDATA_EVAL_CHAT_*` | SSE retry, backoff and timeout knobs |
+| `GD_EVAL_TIMERS` | same as `--timers` |
+
+A gitignored `.env` at the repo root is the normal place for these; load it with
+`set -a && . ./.env && set +a` before the command.
+
 ## Gotchas
 
 **Adding an evaluator is a registry change, not a naming convention.** Single-shot kinds go
@@ -89,6 +189,23 @@ ingestion has no pass/fail signal and inflates or misattributes per-item latency
 (`run_trace_link_inline` is the synchronous alternative). Do not "fix" a slow item by
 making trace scoring synchronous again.
 
+**A generated item's `expected_output` is copied, never invented — keep it that way.**
+`core/dataset/from_insights.py` converts each insight with the platform's own
+`declarative_visualization_to_aac()` (from `gooddata-code-convertors`, via `gooddata-sdk`),
+so the mapping is not ours to get wrong. What is ours is deciding what the evaluator cannot
+yet score — derived measures and measure-level filters convert fine and then compare wrong,
+so they are skipped with a printed reason — and stripping the no-op filters AD saves for an
+"All" selection, which would otherwise let a question claim a filter its chart lacks. Teach
+the comparator about a construct and the matching skip can go; do not make one convert by
+hand. Chart type names are the convertor's, which are also the agent's — do not rename them. One
+granularity is patched in `CONVERTOR_GRANULARITY_FIXES`: `week_us` → `WEEK_US` is a convertor
+bug, the platform enum is `WEEK`.
+
+**The snapshot is a plain-JSON contract.** `--snapshot-in`/`--snapshot-out` is what makes
+the generator testable offline and iterable without re-fetching, and it is why the
+generator reads the declarative analytics model rather than `sdk.visualizations`. Anything
+that changes the fetch shape invalidates every saved snapshot.
+
 **Scoring weights do not sum to 1.** `quality_score` is the fraction of boolean-valued keys
 in `best_detail` that are true, falling back to `pass_at_k` when there are none (text
 evaluators). `value_score` is `0.6 * quality + 0.2 * speed` — the 0.8 total is what the

{gooddata_eval-1.74.1.dev5 → gooddata_eval-1.75.1.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.74.1.dev5
+Version: 1.75.1.dev1
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,7 +17,7 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.74.1.dev5
+Requires-Dist: gooddata-sdk~=1.75.1.dev1
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: orjson<4.0.0,>=3.9.15
 Requires-Dist: pydantic<3.0,>=2.6
@@ -44,7 +44,10 @@ Or install `gd-eval` as a standalone tool:
 | Command | Description |
 |---|---|
 | `gd-eval run` | Run an evaluation dataset against one or more models. |
+| `gd-eval report` | Render JSON report(s) as one self-contained HTML file. |
 | `gd-eval models` | List LLM providers and models configured in the org. |
+| `gd-eval generate` | Generate a `visualization` dataset from a workspace's existing insights. |
+
 
 ---
 
@@ -173,6 +176,8 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
+| `--html PATH` | Write a self-contained HTML report to this path (same output as `gd-eval report`). |
+| `--redact` | Make the HTML customer-safe. See `gd-eval report`. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
 | `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
 | `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
@@ -333,6 +338,59 @@ linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogethe
 
 ---
 
+## `gd-eval report`
+
+Turns JSON report(s) into one HTML file you can actually navigate. No server, no
+credentials, no external assets — it opens over `file://`, attaches to a Jira issue and
+survives a Slack thread.
+
+```bash
+# one run
+gd-eval report results.json -o report.html
+
+# several runs side by side -- each file becomes its own column
+gd-eval report aug-21.json sep-07.json -o comparison.html --title "H200 regression check"
+
+# customer-safe
+gd-eval report results.json -o customer.html --redact
+```
+
+| Flag | Description |
+|---|---|
+| `-o, --out PATH` | Where to write the HTML. Required. |
+| `--title TEXT` | Title shown in the report header. |
+| `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … Pass rate, per-item results, questions and latency survive. |
+
+The report is a *view* over the JSON — it computes no numbers of its own. It gives you:
+
+- **Run cards and a comparison table** — pass rate, quality, latency per run.
+- **An item table** with a pass/fail column per run, so a model or run-over-run
+  regression is one glance rather than a hand-assembled spreadsheet.
+- **An expression filter** for cross-cutting questions the fixed filters can't
+  anticipate, e.g. `d.filter_ranking_score === false` or
+  `d.expected_metric_uris.length > 1 && !d.metrics_correct`. Available variables:
+  `d` (the focused run's `detail`), `it` (its item), `i` (the row, `i.per[label]` for any
+  run), `q` (question), `kind`.
+- **The conversation**, when the item ran the agentic multi-turn path — every turn in
+  order, with the simulated user marked apart from a real question, so you can see
+  whether the agent got there or was handed the answer.
+- **A per-item drawer** — checks as pass/fail chips, expected vs actual side by side,
+  full reasoning, conversation/response ids.
+- **A latency timeline** from `detail.latency_breakdown`, in execution order, one bar per
+  step. Clicking a step expands its full record, joined by `index`: the paragraph a
+  reasoning step was summarised from, or a tool call's arguments and result from
+  `detail.tool_calls`.
+
+`--redact` additionally drops `transcript` and `tool_calls`: the exchange shows that the
+simulated user is primed with the expected output, and a tool result carries
+semantic-layer internals and real query rows. The turn count and the timeline shape
+survive.
+
+Passing several files keyed by file name is the whole run-over-run mechanism: no
+database, no run registry, just the JSON files you already have on disk.
+
+---
+
 ## `gd-eval models`
 
 List all LLM providers and their models in the org. Marks the active model
@@ -353,6 +411,143 @@ gd-eval models \
 
 ---
 
+## `gd-eval generate`
+
+Reverse-engineers a `visualization` dataset out of the charts a customer has already
+built, so you get eval questions without hand-authoring any. Reads the workspace's
+declarative analytics model (read-only), translates each visible insight's buckets,
+sorts and filters into an `expected_output.visualization` spec, then asks an LLM to
+write the analyst question that chart answers. Because `expected_output` is copied from
+a live object rather than authored, every question is grounded in the real LDM by
+construction — the LLM only writes English.
+
+**Setup:** host + token (read access to the workspace), and `OPENAI_API_KEY` plus the
+`llm-judge` extra for the phrasing step (`uv add 'gooddata-eval[llm-judge]'`; skip both
+with `--no-phrase`).
+
+```bash
+export GOODDATA_TOKEN='your-api-token'
+
+# 1. see what a workspace yields before writing anything
+gd-eval generate \
+  --host  https://your.gooddata.cloud \
+  --workspace  ecommerce_demo \
+  --dataset-name  ecommerce \
+  --dry-run
+
+# 2. generate, phrase, validate, and export
+gd-eval generate \
+  --host  https://your.gooddata.cloud \
+  --workspace  ecommerce_demo \
+  --dataset-name  ecommerce \
+  --dashboard  dash_1_returns \
+  --out  ./my-dataset \
+  --langfuse-out  out/langfuse-dataset.json
+
+# 3. run it
+gd-eval run --host … --workspace ecommerce_demo --dataset ./my-dataset --model gpt-5.2
+```
+
+`--workspace` is where insights are read from; `--dataset-name` is the `dataset_name`
+written into every item (and the default output folder).
+
+| Flag | Effect |
+|---|---|
+| `--dashboard <id>` | restrict to insights on that dashboard (repeatable); default is the whole workspace |
+| `--out <dir>` | output folder (default `./<dataset-name>`); this is what `gd-eval run --dataset` reads |
+| `--snapshot-out` / `--snapshot-in` | save/replay the fetched model — replay needs no host, token, or network |
+| `--langfuse-out <file>` | also write a Langfuse-importable dataset JSON |
+| `--id-prefix` | prefix exported Langfuse item ids (they're unique per *project*, so re-importing an item under its original id is a 409) |
+| `--no-phrase` | skip the LLM; emit mechanical `Show <title>` questions |
+| `--phrase-model` | OpenAI model for phrasing (default `gpt-4o`) |
+| `--no-viz-type` | always blank the expected chart type |
+| `--enrich-ranked <N>` | additionally derive up to N ranked questions (see below); default 0 (off) |
+| `--skip-ambiguous` | drop items naming something the model carries more than once; reported either way |
+| `--min-questions` / `--min-shapes` / `--min-filtered` | quality gate, default 15, 3 and 1 |
+
+### Ranked questions (`--enrich-ranked`)
+
+Analysts sort in Analytical Designer and save the chart without persisting the sort, so
+`sort_by`/`ranking_filter` coverage is near zero on most real models — the eval can
+punish a spurious ranking but never confirm the agent builds a required one.
+`--enrich-ranked N` fills that gap by *deriving* ranked items from the specs already
+extracted. Adding a limit or a sort to a definition that executes cannot make it
+unanswerable, and "the top 3 X by Y" has exactly one correct spec, so a derived item is
+less ambiguous to grade than the insight it came from.
+
+The budget is spent best-grounded first:
+
+1. **Insights whose own title promised a ranking their definition never implemented** —
+   "Top Returned Reasons" saved with `sorts: []`. The direction comes from the title
+   (`highest`/`most`/`largest` vs `lowest`/`least`/`worst`) and the N too when it states
+   one; a title naming both ends names neither and is still skipped.
+2. **Ranking filters added to a plain breakdown** — one metric, one non-date dimension,
+   no existing sort. N follows the dimension's element count, so a top-5 over six values
+   is never emitted.
+3. **Sort-only variants**, which order without limiting.
+
+Eligibility is deliberately narrow: two metrics leave "top 3 by what?" unanswered, a
+second dimension leaves the N ambiguous between the pair and within a group, and a date
+dimension turns the result into "top 3 months", which nobody asks. Variants are
+deduplicated by resolved definition — differently-titled insights over one metric and
+dimension would otherwise produce the same question twice — and bases are taken
+round-robin by metric so one popular metric cannot become a third of the corpus.
+
+Derived items carry `derived_from` (the insight id) and `derived_basis` (`title` when a
+human's chart title asked for the ranking, `shape` when this generator chose to add
+one), so a pass rate over each can be computed separately.
+
+### Items that cannot say what they mean
+
+Two classes of question are unwinnable however well the agent behaves, and both are
+reported:
+
+- **A name the model carries more than once.** One workspace has six labels all titled
+  "Product Title"; a question naming one cannot say which is meant, and a perfect chart
+  over the wrong one scores zero. `--skip-ambiguous` drops them; the count and the
+  offending names are printed either way.
+- **A date granularity's cyclical twin.** `MONTH` walks consecutive calendar months,
+  `MONTH_OF_YEAR` stacks every January together. Date dimensions are therefore briefed
+  by what they do ("one point per calendar month over time, not month-of-year") and the
+  writer is told to say it in natural words while keeping the date dataset's name —
+  never as a label id in prose ("Order Created At - Month").
+
+**The question must never contradict its own expected output.** Four rules enforce that:
+
+- The writer is briefed on buckets, sorts and filters only — never the insight title,
+  and never the chart type. Titles routinely describe intent the definition doesn't
+  implement ("Products by Most Items Sold" over `sorts: []`).
+- Every generated question is checked against its spec, and any hit is a hard error:
+  ranking words (`top`, `most`, `highest`, …) require a real sort or ranking filter;
+  filter words (`only`, `last quarter`, `in 2025`, …) require a real date or attribute
+  filter; a breakdown clause requires a non-empty `view_by`/`segment_by` and vice versa;
+  a metric may never be broken down by itself; and no template residue (`breakdown
+  dimension`, `{…}`) may survive. A violation is fed back once for a rewrite, then
+  dropped — and a drop fails the run.
+- The writer's rules are built per insight, so an insight with no `view_by` is never
+  asked to name a breakdown at all.
+- `type` is set only when the question actually names a chart form. An insight's
+  `visualizationUrl` records what a human clicked, not what the question constrains —
+  with one exception: a chart with no breakdown *must* name its form ("as a KPI", "as a
+  single number"). Without it the agent reads a bare "Show me Gross Revenue" as a metric
+  lookup, activates only its search skill and builds nothing.
+
+Everything the writer sees is a display name (`Spend Amount`, `Merchant Name`), never a
+raw URI, so questions read like a person wrote them.
+
+**What it won't do.** Insights it can't express without guessing are skipped with a
+printed reason, never approximated: derived (arithmetic/PoP) measures, measure-level
+filters, `uris`-form attribute filters, unmapped chart types, hidden objects, and
+insights whose title promises behaviour their definition lacks (though `--enrich-ranked`
+implements a promised *ranking* rather than discarding it). If too few survive, the
+quality gate fails the run rather than fabricating items to hit the minimum — point at
+more dashboards, or lower `--min-questions`.
+
+Every written item is validated as a `DatasetItem` with a scorable AAC visualization
+before the command reports success.
+
+---
+
 ## Dataset format
 
 A dataset is a folder of `.json` files, one per question:
@@ -423,7 +618,8 @@ is the fraction of satisfied criteria.
 
 ### `[llm-judge]` — LLM-as-judge evaluators
 
-`general_question` and `guardrail` items are scored by a GPT-4o judge.
+`general_question` and `guardrail` items are scored by a GPT-4o judge, and
+`gd-eval generate` uses the same package to write question text.
 Requires the OpenAI package and `OPENAI_API_KEY`:
 
 ```bash
@@ -432,13 +628,15 @@ uv add 'gooddata-eval[llm-judge]'
 uv tool install 'gooddata-eval[llm-judge]'
 ```
 
-Without `[llm-judge]`, those items are **skipped**.
+Without `[llm-judge]`, those items are **skipped** and `gd-eval generate` needs
+`--no-phrase`.
 
 ## Exit codes
 
 | Code | Meaning |
 |---|---|
 | `0` | Run completed. Evaluation failures do **not** cause a non-zero exit. |
+| `1` | `gd-eval generate` only: a quality gate failed, an item was dropped, or a written item failed validation. |
 | `2` | Operational error: bad connection, missing model, unreadable dataset, missing credentials. |
 
 ## Scores (in JSON report and Langfuse)

{gooddata_eval-1.74.1.dev5 → gooddata_eval-1.75.1.dev1}/README.md

@@ -16,7 +16,10 @@ Or install `gd-eval` as a standalone tool:
 | Command | Description |
 |---|---|
 | `gd-eval run` | Run an evaluation dataset against one or more models. |
+| `gd-eval report` | Render JSON report(s) as one self-contained HTML file. |
 | `gd-eval models` | List LLM providers and models configured in the org. |
+| `gd-eval generate` | Generate a `visualization` dataset from a workspace's existing insights. |
+
 
 ---
 
@@ -145,6 +148,8 @@ interleaves when K > 1, and per-item latencies rise, so they stop being clean si
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
+| `--html PATH` | Write a self-contained HTML report to this path (same output as `gd-eval report`). |
+| `--redact` | Make the HTML customer-safe. See `gd-eval report`. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
 | `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
 | `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
@@ -305,6 +310,59 @@ linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogethe
 
 ---
 
+## `gd-eval report`
+
+Turns JSON report(s) into one HTML file you can actually navigate. No server, no
+credentials, no external assets — it opens over `file://`, attaches to a Jira issue and
+survives a Slack thread.
+
+```bash
+# one run
+gd-eval report results.json -o report.html
+
+# several runs side by side -- each file becomes its own column
+gd-eval report aug-21.json sep-07.json -o comparison.html --title "H200 regression check"
+
+# customer-safe
+gd-eval report results.json -o customer.html --redact
+```
+
+| Flag | Description |
+|---|---|
+| `-o, --out PATH` | Where to write the HTML. Required. |
+| `--title TEXT` | Title shown in the report header. |
+| `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … Pass rate, per-item results, questions and latency survive. |
+
+The report is a *view* over the JSON — it computes no numbers of its own. It gives you:
+
+- **Run cards and a comparison table** — pass rate, quality, latency per run.
+- **An item table** with a pass/fail column per run, so a model or run-over-run
+  regression is one glance rather than a hand-assembled spreadsheet.
+- **An expression filter** for cross-cutting questions the fixed filters can't
+  anticipate, e.g. `d.filter_ranking_score === false` or
+  `d.expected_metric_uris.length > 1 && !d.metrics_correct`. Available variables:
+  `d` (the focused run's `detail`), `it` (its item), `i` (the row, `i.per[label]` for any
+  run), `q` (question), `kind`.
+- **The conversation**, when the item ran the agentic multi-turn path — every turn in
+  order, with the simulated user marked apart from a real question, so you can see
+  whether the agent got there or was handed the answer.
+- **A per-item drawer** — checks as pass/fail chips, expected vs actual side by side,
+  full reasoning, conversation/response ids.
+- **A latency timeline** from `detail.latency_breakdown`, in execution order, one bar per
+  step. Clicking a step expands its full record, joined by `index`: the paragraph a
+  reasoning step was summarised from, or a tool call's arguments and result from
+  `detail.tool_calls`.
+
+`--redact` additionally drops `transcript` and `tool_calls`: the exchange shows that the
+simulated user is primed with the expected output, and a tool result carries
+semantic-layer internals and real query rows. The turn count and the timeline shape
+survive.
+
+Passing several files keyed by file name is the whole run-over-run mechanism: no
+database, no run registry, just the JSON files you already have on disk.
+
+---
+
 ## `gd-eval models`
 
 List all LLM providers and their models in the org. Marks the active model
@@ -325,6 +383,143 @@ gd-eval models \
 
 ---
 
+## `gd-eval generate`
+
+Reverse-engineers a `visualization` dataset out of the charts a customer has already
+built, so you get eval questions without hand-authoring any. Reads the workspace's
+declarative analytics model (read-only), translates each visible insight's buckets,
+sorts and filters into an `expected_output.visualization` spec, then asks an LLM to
+write the analyst question that chart answers. Because `expected_output` is copied from
+a live object rather than authored, every question is grounded in the real LDM by
+construction — the LLM only writes English.
+
+**Setup:** host + token (read access to the workspace), and `OPENAI_API_KEY` plus the
+`llm-judge` extra for the phrasing step (`uv add 'gooddata-eval[llm-judge]'`; skip both
+with `--no-phrase`).
+
+```bash
+export GOODDATA_TOKEN='your-api-token'
+
+# 1. see what a workspace yields before writing anything
+gd-eval generate \
+  --host  https://your.gooddata.cloud \
+  --workspace  ecommerce_demo \
+  --dataset-name  ecommerce \
+  --dry-run
+
+# 2. generate, phrase, validate, and export
+gd-eval generate \
+  --host  https://your.gooddata.cloud \
+  --workspace  ecommerce_demo \
+  --dataset-name  ecommerce \
+  --dashboard  dash_1_returns \
+  --out  ./my-dataset \
+  --langfuse-out  out/langfuse-dataset.json
+
+# 3. run it
+gd-eval run --host … --workspace ecommerce_demo --dataset ./my-dataset --model gpt-5.2
+```
+
+`--workspace` is where insights are read from; `--dataset-name` is the `dataset_name`
+written into every item (and the default output folder).
+
+| Flag | Effect |
+|---|---|
+| `--dashboard <id>` | restrict to insights on that dashboard (repeatable); default is the whole workspace |
+| `--out <dir>` | output folder (default `./<dataset-name>`); this is what `gd-eval run --dataset` reads |
+| `--snapshot-out` / `--snapshot-in` | save/replay the fetched model — replay needs no host, token, or network |
+| `--langfuse-out <file>` | also write a Langfuse-importable dataset JSON |
+| `--id-prefix` | prefix exported Langfuse item ids (they're unique per *project*, so re-importing an item under its original id is a 409) |
+| `--no-phrase` | skip the LLM; emit mechanical `Show <title>` questions |
+| `--phrase-model` | OpenAI model for phrasing (default `gpt-4o`) |
+| `--no-viz-type` | always blank the expected chart type |
+| `--enrich-ranked <N>` | additionally derive up to N ranked questions (see below); default 0 (off) |
+| `--skip-ambiguous` | drop items naming something the model carries more than once; reported either way |
+| `--min-questions` / `--min-shapes` / `--min-filtered` | quality gate, default 15, 3 and 1 |
+
+### Ranked questions (`--enrich-ranked`)
+
+Analysts sort in Analytical Designer and save the chart without persisting the sort, so
+`sort_by`/`ranking_filter` coverage is near zero on most real models — the eval can
+punish a spurious ranking but never confirm the agent builds a required one.
+`--enrich-ranked N` fills that gap by *deriving* ranked items from the specs already
+extracted. Adding a limit or a sort to a definition that executes cannot make it
+unanswerable, and "the top 3 X by Y" has exactly one correct spec, so a derived item is
+less ambiguous to grade than the insight it came from.
+
+The budget is spent best-grounded first:
+
+1. **Insights whose own title promised a ranking their definition never implemented** —
+   "Top Returned Reasons" saved with `sorts: []`. The direction comes from the title
+   (`highest`/`most`/`largest` vs `lowest`/`least`/`worst`) and the N too when it states
+   one; a title naming both ends names neither and is still skipped.
+2. **Ranking filters added to a plain breakdown** — one metric, one non-date dimension,
+   no existing sort. N follows the dimension's element count, so a top-5 over six values
+   is never emitted.
+3. **Sort-only variants**, which order without limiting.
+
+Eligibility is deliberately narrow: two metrics leave "top 3 by what?" unanswered, a
+second dimension leaves the N ambiguous between the pair and within a group, and a date
+dimension turns the result into "top 3 months", which nobody asks. Variants are
+deduplicated by resolved definition — differently-titled insights over one metric and
+dimension would otherwise produce the same question twice — and bases are taken
+round-robin by metric so one popular metric cannot become a third of the corpus.
+
+Derived items carry `derived_from` (the insight id) and `derived_basis` (`title` when a
+human's chart title asked for the ranking, `shape` when this generator chose to add
+one), so a pass rate over each can be computed separately.
+
+### Items that cannot say what they mean
+
+Two classes of question are unwinnable however well the agent behaves, and both are
+reported:
+
+- **A name the model carries more than once.** One workspace has six labels all titled
+  "Product Title"; a question naming one cannot say which is meant, and a perfect chart
+  over the wrong one scores zero. `--skip-ambiguous` drops them; the count and the
+  offending names are printed either way.
+- **A date granularity's cyclical twin.** `MONTH` walks consecutive calendar months,
+  `MONTH_OF_YEAR` stacks every January together. Date dimensions are therefore briefed
+  by what they do ("one point per calendar month over time, not month-of-year") and the
+  writer is told to say it in natural words while keeping the date dataset's name —
+  never as a label id in prose ("Order Created At - Month").
+
+**The question must never contradict its own expected output.** Four rules enforce that:
+
+- The writer is briefed on buckets, sorts and filters only — never the insight title,
+  and never the chart type. Titles routinely describe intent the definition doesn't
+  implement ("Products by Most Items Sold" over `sorts: []`).
+- Every generated question is checked against its spec, and any hit is a hard error:
+  ranking words (`top`, `most`, `highest`, …) require a real sort or ranking filter;
+  filter words (`only`, `last quarter`, `in 2025`, …) require a real date or attribute
+  filter; a breakdown clause requires a non-empty `view_by`/`segment_by` and vice versa;
+  a metric may never be broken down by itself; and no template residue (`breakdown
+  dimension`, `{…}`) may survive. A violation is fed back once for a rewrite, then
+  dropped — and a drop fails the run.
+- The writer's rules are built per insight, so an insight with no `view_by` is never
+  asked to name a breakdown at all.
+- `type` is set only when the question actually names a chart form. An insight's
+  `visualizationUrl` records what a human clicked, not what the question constrains —
+  with one exception: a chart with no breakdown *must* name its form ("as a KPI", "as a
+  single number"). Without it the agent reads a bare "Show me Gross Revenue" as a metric
+  lookup, activates only its search skill and builds nothing.
+
+Everything the writer sees is a display name (`Spend Amount`, `Merchant Name`), never a
+raw URI, so questions read like a person wrote them.
+
+**What it won't do.** Insights it can't express without guessing are skipped with a
+printed reason, never approximated: derived (arithmetic/PoP) measures, measure-level
+filters, `uris`-form attribute filters, unmapped chart types, hidden objects, and
+insights whose title promises behaviour their definition lacks (though `--enrich-ranked`
+implements a promised *ranking* rather than discarding it). If too few survive, the
+quality gate fails the run rather than fabricating items to hit the minimum — point at
+more dashboards, or lower `--min-questions`.
+
+Every written item is validated as a `DatasetItem` with a scorable AAC visualization
+before the command reports success.
+
+---
+
 ## Dataset format
 
 A dataset is a folder of `.json` files, one per question:
@@ -395,7 +590,8 @@ is the fraction of satisfied criteria.
 
 ### `[llm-judge]` — LLM-as-judge evaluators
 
-`general_question` and `guardrail` items are scored by a GPT-4o judge.
+`general_question` and `guardrail` items are scored by a GPT-4o judge, and
+`gd-eval generate` uses the same package to write question text.
 Requires the OpenAI package and `OPENAI_API_KEY`:
 
 ```bash
@@ -404,13 +600,15 @@ uv add 'gooddata-eval[llm-judge]'
 uv tool install 'gooddata-eval[llm-judge]'
 ```
 
-Without `[llm-judge]`, those items are **skipped**.
+Without `[llm-judge]`, those items are **skipped** and `gd-eval generate` needs
+`--no-phrase`.
 
 ## Exit codes
 
 | Code | Meaning |
 |---|---|
 | `0` | Run completed. Evaluation failures do **not** cause a non-zero exit. |
+| `1` | `gd-eval generate` only: a quality gate failed, an item was dropped, or a written item failed validation. |
 | `2` | Operational error: bad connection, missing model, unreadable dataset, missing credentials. |
 
 ## Scores (in JSON report and Langfuse)