gooddata-eval 1.77.1.dev2__tar.gz → 1.78.1.dev1__tar.gz

{gooddata_eval-1.77.1.dev2 → gooddata_eval-1.78.1.dev1}/AGENTS.md

@@ -27,6 +27,7 @@ experiment. The newest and most actively developed package in the repo.
 | `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
 | `core/chat/` | SSE client for the agent's streaming chat endpoint |
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
+| `core/takeaway/` | HTTP client for the AI Publisher takeaway endpoint — a single-shot backend like `core/summary/`, keeping the whole response on `ChatResult.raw_response` — plus `evaluate_document_takeaway`, the K-run, gate and Langfuse entry point the Tavern shim calls |
 | `core/dataset/` | dataset format, loading, and `from_insights.py` — dataset generation from a workspace's real insights |
 | `core/evaluators/` | single-shot evaluators and their registry |
 | `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `item_scope` (an item's experiment scope sent to gen-ai as baggage, and which conversations joined), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
@@ -59,7 +60,8 @@ implementation, sometimes under different `test_kind` strings (`search_tool` vs
 `expected_output: Any`. `expected_output` is deliberately untyped — each evaluator parses
 its own shape. `test_kind` on the item is what labels the result, not the evaluator class,
 which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
-`dashboard_summary` items additionally need `summary_input`.
+`dashboard_summary` items additionally need `summary_input`, `document_takeaway` items
+`takeaway_input`.
 
 ## Running the pipeline
 
@@ -155,8 +157,8 @@ Run this before guessing a `--model` string.
 | `GD_EVAL_JUDGE_MODEL` | judge model, same as `--judge-model` |
 | `GD_EVAL_AGENT_ID` | which agent to drive, same as `--agent-id` |
 | `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` | `--langfuse`, `--langfuse-dataset` |
-| `GOODDATA_EVAL_CHAT_*` | SSE retry, backoff and timeout knobs |
-| `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary runs are not labelled |
+| `GOODDATA_EVAL_CHAT_*` | retry and backoff knobs shared by the chat and takeaway clients (`core/_retry.py`), and the SSE turn and item timeouts |
+| `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary and document-takeaway runs are not labelled |
 | `GOODDATA_EVAL_JOIN_GENAI_TRACE` | on unless set to `0`, `false`, `no`, `off` or empty. An agentic item's chat turns ask gen-ai to open them under gd-eval's experiment root, so each item is one Langfuse trace with its scores written once. A gen-ai that drops caller baggage keeps today's separate traces; see the gotcha below |
 | `GD_EVAL_TIMERS` | same as `--timers` |
 
@@ -168,8 +170,11 @@ A gitignored `.env` at the repo root is the normal place for these; load it with
 **Adding an evaluator is a registry change, not a naming convention.** Single-shot kinds go
 into `_EAGER_EVALUATORS`, or `_LAZY_EVALUATOR_MODULES` plus `_LAZY_EVALUATOR_CLASSES`, in
 `core/evaluators/__init__.py`. Agentic kinds need the string added to `AGENTIC_TEST_KINDS`
-and a new branch in `_dispatch_agentic`. Test file naming follows the capability, but
-naming a test file correctly registers nothing.
+and a new branch in `_dispatch_agentic`, and must also be classified in `core/interaction.py`:
+add it to `SINGLE_TURN_AGENTIC_KINDS` (one chat message per run) or `MULTI_TURN_KINDS` (a
+simulated user or several turns). That decides which latency section its items land in;
+`test_every_agentic_kind_is_classified_on_purpose` fails until it is done. Test file naming
+follows the capability, but naming a test file correctly registers nothing.
 
 **Parallel-safety is a reviewed allowlist, and getting it wrong corrupts results.**
 `WORKSPACE_MUTATING_TEST_KINDS` is computed as `AGENTIC_TEST_KINDS - PARALLEL_SAFE_TEST_KINDS`,
@@ -178,12 +183,15 @@ That default is correct: agent tool calls create real server-side objects (metri
 Adding a kind to `PARALLEL_SAFE_TEST_KINDS` is a deliberate assertion that it is read-only,
 which nothing in the package can prove for you.
 
-**The SSE client's retry predicate is load-bearing.** `core/chat/sse_client.py` retries
-429/502/503/504 and `httpx.RemoteProtocolError` (a mid-stream disconnect) with exponential
-backoff, and treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The
-`RemoteProtocolError` case was added after it was confirmed live to contaminate a small
-percentage of visualization runs with a hard fail and no retry. Narrowing that predicate
-reintroduces the problem.
+**The SSE client's retry predicate is load-bearing.** The backoff loop and the retryable
+status set (429/500/502/503/504) live in `core/_retry.py`, shared by the chat and takeaway
+clients and tuned by the same `GOODDATA_EVAL_CHAT_*` knobs. `core/chat/sse_client.py`
+retries that status set and `httpx.RemoteProtocolError` (a mid-stream disconnect), and
+treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The `RemoteProtocolError` case
+was added after it was confirmed live to contaminate a small percentage of visualization
+runs with a hard fail and no retry. Narrowing that predicate reintroduces the problem. The
+takeaway client retries the same status set plus connection failures, but not a read
+timeout: a slow page would spend the whole timeout again on every attempt.
 
 **Langfuse trace linking is deliberately off the item critical path.** Polling for trace
 ingestion has no pass/fail signal and inflates or misattributes per-item latency, so

gooddata_eval-1.78.1.dev1/Makefile

@@ -0,0 +1,6 @@
+# (C) 2026 GoodData Corporation
+include ../../project_common.mk
+
+.PHONY: calibrate-takeaway
+calibrate-takeaway:
+	uv run --package gooddata-eval --extra llm-judge python scripts/calibrate_takeaway_judge.py

{gooddata_eval-1.77.1.dev2 → gooddata_eval-1.78.1.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.77.1.dev2
+Version: 1.78.1.dev1
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,7 +17,7 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.77.1.dev2
+Requires-Dist: gooddata-sdk~=1.78.1.dev1
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: jsonpatch<2.0,>=1.33
 Requires-Dist: orjson<4.0.0,>=3.9.15
@@ -161,8 +161,8 @@ gd-eval run \
 | `--runs K` | `2` | Independent runs per item. |
 | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
-| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
-| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
+| `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. |
 
 **Concurrency and workspace safety.** Agentic kinds that create workspace objects
 (`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
@@ -263,6 +263,11 @@ The JSON report always uses the nested multi-model shape:
 
 Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
 
+The HTML report orders runs by when they started (`summary.started_at`, an ISO 8601 UTC timestamp that `gd-eval run` writes
+into each run's `summary`, `null` when a run was never timed), so the order the files are passed to
+`gd-eval report` does not decide which run is the latest. A report without it falls back to the date a run's file is
+named for (`2026-09-28.json`), and if any run has neither it keeps the order the files were given in.
+
 Each item reports **how many of its runs passed**, not only whether one did:
 
 ```json
@@ -337,13 +342,73 @@ The other six agentic kinds report `0.0` for the first three. Trace linking itse
 runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
 linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
 
+### Latency: single-shot and multi-turn
+
+`avg_latency_s` is one number over a mixed population. The report instead keeps two kinds of item apart, by
+their eval kind, and states latency for each:
+
+- **Single-shot** — kinds that send exactly one chat message per run: every non-agentic kind, plus
+  `agentic_general_question`, `agentic_guardrail` and `agentic_search`.
+- **Multi-turn** — kinds that drive a simulated user or fixture turns: `agentic_visualization` (and
+  `vis_agentic`), `agentic_metric_skill`, `agentic_alert_skill`, `agentic_dashboard_skill`,
+  `agentic_document_skill`, `agentic_kda_skill`, `agentic_anomaly_detection`, `agentic_what_if` and
+  `agentic_conversation`.
+
+```json
+"summary": {
+  "by_mode": {
+    "single_shot": {"total": 12, "passed": 10,
+                    "traces_s": {"n": 24, "mean": 5.39, "trimmed_mean": 5.36, "min": 3.88, "p50": 5.33, "p70": 6.23, "p90": 6.82, "p99": 7.18, "max": 7.22}},
+    "multi_turn":  {"total": 6, "passed": 4,
+                    "traces_s": {"n": 45, "mean": 6.91, "trimmed_mean": 6.84, "min": 2.78, "p50": 6.52, "p70": 8.03, "p90": 10.79, "p99": 13.57, "max": 14.25},
+                    "runs_s":   {"n": 15, "mean": 23.55, "trimmed_mean": 23.31, "min": 12.64, "p50": 21.23, "p70": 28.52, "p90": 33.02, "p99": 36.94, "max": 37.51}}
+  }
+}
+```
+
+`by_mode` is in each run's `summary` and in its `comparison` entry. `total` and `passed` count the evaluated
+items of that mode (skipped ones belong to neither). Every distribution is `null` when nothing was timed.
+Each item also states its `mode`, its `trace_latencies_s` and its `run_latencies_s`.
+
+- **Per trace is the main figure.** gen-ai starts one trace per chat message, so a trace is one chat turn.
+  Its latency is that turn's response time, from sending the message to the end of its stream
+  (`turn_wall_clock_sec`), recorded where the chat client completes the turn. It holds none of the LLM judge,
+  the simulated user or conversation create/delete. `dashboard_summary` is timed the same way around its one
+  request. A turn cut off by its turn or item deadline, or one that ended without a final answer (the agent's
+  iteration or token limit, a content filter), still counts, with the time it ran for: those are the slowest
+  turns, and the run that scored them is in the per-run figures too. A turn that ended in any other error (an
+  error event, or a 503 that outlasted its retries) does not: the time is that of a failed request, not a
+  response, and would only add a fast outlier. On the single-shot path such a turn ends its item as an error, so
+  it has no per-run figure either. An item that errored outside its evaluator's own failure reporting
+  contributes neither.
+- **Single-shot** runs are exactly one trace, so `traces_s` is also the latency of a run.
+- **Multi-turn** runs span several traces, so they get both views: `traces_s` over every turn of every
+  conversation (comparable with single-shot), and `runs_s`, the wall clock of each whole conversation from just
+  before its first message to the end of its scoring — every turn, simulated-user reply and judge call, but not
+  conversation create/delete. `agentic_conversation` drives its fixture once and contributes one run per item.
+  The two single-shot and multi-turn populations are never mixed.
+- **Percentiles** use linear interpolation between closest ranks (NumPy's default), so `p50` is the median and
+  the report carries no separate median field. On a small sample the high percentiles sit at or near the
+  maximum: P99 over fewer than 100 samples is effectively the slowest one, which is why `n` always travels with it.
+- **Trimmed mean** (`trimmed_mean`) is the average of the middle 80%: the fastest 10% and the slowest 10% of the
+  samples are dropped from each end (`n * 10 // 100` of them, so 25 samples drop two each) before averaging. It is
+  not the median, which is one middle value (`p50`). It is `null` below 10 samples, where there is nothing to
+  trim and the figure would only repeat `mean`.
+- **Older reports** have no `by_mode`. The HTML report says so and nothing is backfilled.
+
+`avg_latency_s` and the other existing fields are unchanged. The console prints each run's pass counts per
+mode and one `Latency, <population> (n=...)` line per distribution beneath its summary line, and Trace
+P50/P90/P99 columns (single-shot) in the multi-model comparison.
+
 ---
 
 ## `gd-eval report`
 
 Turns JSON report(s) into one HTML file you can actually navigate. No server, no
 credentials, no external assets — it opens over `file://`, attaches to a Jira issue and
-survives a Slack thread.
+survives a Slack thread. It follows GoodData's product look (colours, type, status chips). The faces are Inter, with
+Funnel Display for headings, named but never loaded, so a viewer without them gets the system sans and the file stays
+self-contained.
 
 ```bash
 # one run
@@ -360,27 +425,134 @@ gd-eval report results.json -o customer.html --redact
 |---|---|
 | `-o, --out PATH` | Where to write the HTML. Required. |
 | `--title TEXT` | Title shown in the report header. |
-| `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … Pass rate, per-item results, questions and latency survive. |
-
-The report is a *view* over the JSON — it computes no numbers of its own. It gives you:
-
-- **Run cards and a comparison table** — pass rate, quality, latency per run.
-- **An item table** with a pass/fail column per run, so a model or run-over-run
-  regression is one glance rather than a hand-assembled spreadsheet.
-- **An expression filter** for cross-cutting questions the fixed filters can't
-  anticipate, e.g. `d.filter_ranking_score === false` or
+| `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … A model keeps its letter across all its runs, which are then numbered (`Run 1 · Model A`, `Run 2 · Model A`). Pass rate, per-item results, questions and latency survive. |
+
+The report is a *view* over the JSON. Unfiltered, every figure on it is read from the report, so it cannot
+disagree with the console or the JSON. Narrow the selection with a filter and the figures are recomputed in the
+page from the selected items' own records, by the same rules gd-eval uses. The page reads top to bottom as a
+document, in plain words: a *test case* is one prompt given to the assistant, an *attempt* is one go at it, and the
+two formats are *single-question* tests (one answer, no follow-up) and *conversation* tests (a back-and-forth with a
+simulated user), which are `single_shot` and `multi_turn` in the JSON.
+
+- **A masthead** with the report's title (`--title`, or "Run history", "Run comparison", "Model comparison" or
+  "Run results" plus the model and workspace), when it was generated in the reader's own time zone, and a **Run setup** tooltip listing what
+  is the same in every run (model, provider, workspace, dataset, test cases, attempts per test case, pass rule) and
+  what differs. Results from different setups are not directly comparable, so a caution icon on a latency row and the
+  *New* / *Removed* marks in the test case table say where two runs stopped being alike.
+- **View by** — a dropdown above the filters that lays the same report out two ways. **Date** is the layout the
+  rest of this list describes: a column per run, oldest to latest by start time, with change lines, sparklines and the latency line
+  chart. **Model** compares language models instead: a column per model, ranked best first by the console's winner
+  rule (pass rate, then quality, then the quicker average), with no time axis. Where a model has several runs its
+  latest stands for it, so each model is read at its most recent state. The Run filter becomes a Model filter, the hint
+  beside the dropdown says what is being compared (and names the dates when the compared runs differ in date), and the
+  Run setup tooltip does not flag the model itself as a difference. Every report opens on Date, including one that
+  compares several models (`gd-eval run --model a --model b --html`); choose Model to compare them. A single run has
+  nothing to switch between, so the dropdown is hidden. In the Model view the page reads, top to bottom:
+  - **Results** — one row per model, best pass rate first, with a bar of how its evaluated test cases ended (passed
+    green, failed red, could not be graded orange), so the green part is the pass rate beside it, and the **Median
+    latency** of one attempt (half the attempts were faster, half slower). A list under the table says what each term means. The counts and the
+    caution icon (a model evaluated on different test cases than the one with the most) are the date view's Outcomes
+    table. There are no headline cards by model: the table is the first thing on the page.
+  - **Quality against speed** — each model as a point: the share of test cases it passed against the median time of
+    one attempt, the same pass rate as the Results table and the same median as the latency tables. Up and to the left is better. Time runs from zero;
+    the pass rate uses the range the models span. It needs two models with timings.
+  - **By category** — a heat map of the pass rate per category and model, the best model in each row marked, the
+    categories where the models differ most first. Every cell prints its share and its counts, so the shade is never the
+    only signal. This is where a model that wins overall but loses on one category shows.
+  - **Consistency across attempts** — per model, the share of test cases that passed on at least one attempt (pass@K,
+    the light bar) beside the share that passed on every attempt (pass^K, the dark bar), and how many passed only some.
+    A long gap is a model that can do the task but not reliably. It appears only for a report that recorded
+    `pass_power_k` and attempted a test case more than once.
+  - **Latency sections** — the same four percentiles, one row per model, quickest first, with a bar per figure drawn
+    from zero beside the numbers. The quickest and slowest model in each column are marked as in the date view.
+  - **Test cases** — one result column per model in ranked order, a response-time column per model, and an **Across
+    models** column in place of *Since previous run*: *Mixed* (some models passed and some did not; hover to see who),
+    *All pass* or *All fail*. A skipped or ungraded test case is left out of the verdict. The table opens on the
+    *Mixed* ones, the test cases where the models differ.
+- **Filters in two tiers** — the **Filter the report** bar at the top (category, format, result and run) decides what
+  the whole page covers: the Results cards, the outcomes table, the latency tables and the test case table all follow it, so the runs stay
+  comparable. The Test cases section repeats those four and adds the search box and the expression; they narrow the
+  table only, inside whatever the top bar allows, and only offer what the top bar leaves. Category, result and run are
+  attribute filters like the product's: a search box, an All row, the values with an **Only** shortcut and a count,
+  then Cancel and Apply. Changes are staged in the popup and take effect on Apply; closing it any other way discards
+  them. Nothing ticked means all. The run filter chooses which runs are shown; a result matches a test case when any of
+  the runs in play has it (so "Failed" over previous and latest finds every test case that broke in either), and the
+  expression is evaluated once per run in play with the same any-run rule. A filtered page states how many test cases
+  it covers, and each tier has its own Reset.
+- **Results** — three headline cards for the latest run: test cases passed (of those evaluated: skipped ones are not
+  counted, as in the console's percentage), the average quality score and the median time per attempt. That last one
+  pools every format on the page and counts whole attempts, grading included, so it matches neither per-response latency table
+  below it; its tooltip says which formats it covers. With an earlier run each card carries one change line in words, "down from 86% on Sep 28", rather
+  than a signed number. The line is green when that direction is good and red when it is not, with a chevron and
+  screen-reader text (*Better* / *Worse*) so colour is not the only signal. From three runs each card also has a
+  sparkline of every run: hover a point for its value, click it to compare the latest run with that one instead of the
+  previous. The dot being compared with is drawn white, and a key under the cards says so (latest run, compared with,
+  other runs) and that a click on any other dot moves the comparison; it shows only when a sparkline does. A pick that
+  a filter has removed, or that has become the latest run, falls back to the previous run. The
+  pass-rate card links to the test case table to see *which* ones changed; the split by format lives in that table's
+  filters.
+- **Outcomes** — how every test case ended, as a table with one row per run, newest first, in the same layout as the
+  latency tables. A run is identified by three columns: the **Run date**, the day and time it started (`summary.started_at`,
+  shown in the reader's own time zone, the UTC stamp on hover; a report written before it existed shows the date its file is
+  named for, without a time), and the **Run name**, the name the run was given, as given, so a run named `2026-09-28` reads
+  `2026-09-28` here although the charts style it as "Sep 28", and the **Model** it evaluated, its provider on hover. Then **Passed**, **Failed**, **Could not be graded** (an error left it without a verdict) and
+  **Skipped** (not run), then the **Pass rate**. The four are the values of the Result filter, and the pass rate is
+  the card's: passed out of the evaluated test cases, skipped ones left out, as in the console. The highest pass rate
+  is marked green with an up chevron and the lowest red with a down chevron, and the latest run is marked in lime. A
+  caution icon flags a run evaluated over a different number of test cases than the one before, because its pass
+  rate is then not a like-for-like change. The counts are read from the run's `summary` (`passed`, `failed`,
+  `errored`, `skipped`); a filtered view recounts them from the selected test cases, and a report written before
+  `summary.failed` existed shows what the other three leave.
+- **Latency sections** — **Single-question tests · response time** first, then **Conversation tests**, which shows
+  both per response (each assistant reply, one gen-ai trace) and per conversation (start to finish, with the
+  simulated user and the judge). Each opens with a **line chart** of the median (P50), P70, P90 and P99 across the
+  runs, oldest on the left and the latest on the right, drawn from zero so line heights compare honestly; the lines
+  share one blue that darkens towards the tail. Hover or focus a run for its crosshair and every figure, in a tooltip
+  that says how many responses it rests on. A line needs two runs with timings, so with one run the chart is left
+  out. End labels name each line and give its latest value, and are dropped when two would touch. Under the chart a
+  table has one row per run, newest first, identified by the same **Run date**, **Run name** and **Model** columns as the outcomes table, with the same four figures. The quickest run in each column is marked
+  green and the slowest red, each with a chevron (down for least time, up for most) so colour is not the only
+  signal, and the latest run is marked in lime. A caution icon flags a run timed on a different number of samples
+  than the one before. Unfiltered they are read from `by_mode`; a filtered view recomputes them in the page by the
+  same rules. The fastest and slowest single response, the average and the trimmed average stay in the JSON (P99
+  over a few dozen samples sits close to the slowest one, which is why `n` travels with it in the tooltip and
+  the cell hint). Seconds read with one decimal and no redundant ".0" (`3s`, `4.6s`).
+- **A test case table**, newest first: the latest run's result, a **Since previous run** column, then the earlier runs
+  going back in time (click any result to open the drawer on that run). Each run's column is headed by its run date over
+  its run name, one to a line (hover the heading for the full run label); by model the columns are headed by the model. The change column reads *Regressed* (passed in
+  the previous run, not in the latest), *Fixed*, *Flaky* (flipped between pass and fail more than once), *New*
+  (in no earlier run) or *Removed* (in an earlier run, gone from the latest). A test case skipped in the latest run
+  has no label, since the change would describe earlier runs. With several runs the table opens on the regressions. Two time columns
+  keep the populations apart: **Response time** is the median of the test case's assistant responses, and **Time per
+  attempt** is the mean wall-clock of a whole attempt (the responses plus the judge and, for conversations, the
+  simulated user). With more than two runs shown they follow the latest run only. Hover a value to see every sample
+  behind it; the drawer labels its two latency chips the same way. A report written before per-trace timing has no
+  response figure and shows `avg_latency_s` as the attempt figure.
+- **An expression filter**, folded under *Advanced: filter by a check expression* in the Test cases section, for
+  cross-cutting questions the fixed filters can't anticipate, e.g. `d.filter_ranking_score === false` or
   `d.expected_metric_uris.length > 1 && !d.metrics_correct`. Available variables:
-  `d` (the focused run's `detail`), `it` (its item), `i` (the row, `i.per[label]` for any
-  run), `q` (question), `kind`.
-- **The conversation**, when the item ran the agentic multi-turn path — every turn in
+  `d` (the checks of the run being read, its `detail`), `it` (that run's result), `i` (the row, `i.per[label]` for any
+  run), `q` (question), `kind` (category id).
+- **The conversation**, when the test case ran the agentic multi-turn path — every turn in
   order, with the simulated user marked apart from a real question, so you can see
   whether the agent got there or was handed the answer.
-- **A per-item drawer** — checks as pass/fail chips, expected vs actual side by side,
-  full reasoning, conversation/response ids.
+- **A per-test-case drawer** — checks as pass/fail chips, expected vs actual side by side,
+  the latency timeline, conversation/response ids.
 - **A latency timeline** from `detail.latency_breakdown`, in execution order, one bar per
-  step. Clicking a step expands its full record, joined by `index`: the paragraph a
-  reasoning step was summarised from, or a tool call's arguments and result from
-  `detail.tool_calls`.
+  step. Every step has a chevron that opens its full record in place, joined by `index`: the
+  paragraph a reasoning step was summarised from, or a tool call's arguments and result from
+  `detail.tool_calls`. A reasoning paragraph's markdown `**title**` is shown as bold text,
+  not as asterisks. Any number of steps can be open at once, and the chevron above the list
+  opens or closes all of them. The reasoning therefore lives on the timeline; a separate
+  Reasoning section appears only for paragraphs no step points at (or when a run recorded no
+  timeline).
+- **A print stylesheet** that drops the filters and the drawer, so "Save as PDF" gives a clean page.
+
+A report that merges several files names each run after its file, and a file named for its date (`2026-09-28.json`)
+reads as "Sep 28". A model is read from the run itself (`model` and the provider), never from the file name, so files
+named for their model work as well as files named for their date. In the Date view the page treats the runs as one
+model followed over time; when a report holds several models it adds the model to each run's name and points at
+**Model** (see above) for the comparison.
 
 `--redact` additionally drops `transcript` and `tool_calls`: the exchange shows that the
 simulated user is primed with the expected output, and a tool result carries
@@ -564,7 +736,7 @@ A dataset is a folder of `.json` files, one per question:
 ```
 
 Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
-`search_tool`, `general_question`, `guardrail`, `dashboard_summary`.
+`search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`.
 
 ### `dashboard_summary` items
 
@@ -603,6 +775,64 @@ The `expected_output` rubric:
 Each criterion is scored independently by the LLM judge, so `quality_score`
 is the fraction of satisfied criteria.
 
+### `document_takeaway` items
+
+Takeaway items call the AI Publisher takeaway endpoint
+(`POST /api/v1/ai/workspaces/{ws}/takeaway`) with a `takeaway_input` block, authored in
+snake_case and sent as the endpoint's camelCase body. In a Langfuse dataset the block goes
+in the item's `metadata` (or its `input` object), and `input` can stay a plain case label;
+an item carrying `takeaway_input` is a `document_takeaway` item unless it declares another
+`test_kind`. The endpoint needs the `enableGenAiReportTakeaways` feature flag.
+
+```json
+{
+  "id": "takeaway-001",
+  "dataset_name": "agent_document_takeaway",
+  "test_kind": "document_takeaway",
+  "question": "P01 default format, Sales page, no instructions",
+  "takeaway_input": {
+    "visualizations": ["<visualization id>", {"visualization_id": "<id>", "filters": []}],
+    "filter_context": [],
+    "instructions": {"document": null, "slot": null, "format_hint": null},
+    "max_characters": 300
+  },
+  "expected_output": {
+    "no_data": false,
+    "format": {"no_lists": false},
+    "must_include": ["States one finding that connects at least two of the charts"],
+    "must_not_include": [],
+    "rubric": []
+  }
+}
+```
+
+`takeaway_input` requires `visualizations`; unknown keys are rejected. Every
+`expected_output` key is optional:
+
+- `no_data` — whether the endpoint must answer `noData`; when either side says no data,
+  nothing else is graded.
+- `format` — overrides of the deterministic checks, which gate pass/fail: `bold_headline`,
+  `max_headline_words` (10), `max_sentences` (2), `max_words` (50), `single_paragraph`,
+  `no_lists`, `no_headings`. `false` or `null` switches a check off; unknown keys are
+  rejected. `length` (`max_characters`), `no_ids` and `no_meta` always apply.
+- `default_rules` — `false` drops the default judge criteria (one finding; no executive
+  summary, chart walk-through, background opener, figures, dates, causes the data cannot
+  show, chart description).
+- `must_include`, `must_not_include`, `rubric` — the item's own judge criteria, graded like
+  `dashboard_summary`.
+
+`word_count` and `headline_word_count` are reported in the item detail. The default judge
+criteria are calibrated with `make calibrate-takeaway` (needs `OPENAI_API_KEY`); re-run it
+after changing their wording.
+
+The client retries 429/5xx answers and connection failures. A request error that survives
+the retries is handled differently by the two entry points: `gd-eval run` aborts the item
+with that error, whatever `--runs` is, while the CI entry point
+(`evaluate_document_takeaway`) records it as an errored run, keeps the item's other runs,
+and raises only when no run completed. A read timeout is not retried, and the CI entry
+point records it as an errored run too. Any other 4xx and an unsupported URL scheme are
+never retried and fail the item in both.
+
 ### Items asked in a dashboard context (`user_context`)
 
 An item can be asked the way a user asks from an open dashboard or an attached widget.
@@ -610,7 +840,8 @@ Its `user_context` is sent verbatim as `userContext` on every chat message of th
 follow-up and clarification messages included — the server treats a message without one
 as a cleared context. Requires the `enableAiContextSetup` feature flag on the target
 organization — without it the server ignores the dashboard view. Applies to chat items
-only; `dashboard_summary` items carry their scope in `summary_input` instead. No
+only; `dashboard_summary` and `document_takeaway` items carry their scope in
+`summary_input` / `takeaway_input` instead. No
 `userContext` key is sent at all when neither the item nor, for `agentic_conversation`,
 the current turn sets one.
 
@@ -663,6 +894,7 @@ from that turn on until another turn changes it, as the attached context does in
 | `general_question` | Text answer judged by LLM | `[llm-judge]` |
 | `guardrail` | Refusal/redirect (visualization response auto-fails) | `[llm-judge]` |
 | `dashboard_summary` | Dashboard summary (via `/summary` endpoint) scored against a rubric by LLM | `[llm-judge]` |
+| `document_takeaway` | One short takeaway for a page (via `/takeaway` endpoint): deterministic format checks plus a rubric scored by LLM | `[llm-judge]` |
 
 ## Optional extras