gooddata-eval 1.74.1.dev1__tar.gz → 1.74.1.dev2__tar.gz

gooddata_eval-1.74.1.dev2/AGENTS.md

@@ -0,0 +1,114 @@
+# gooddata-eval
+
+`gd-eval` — a CLI and library that drives the GoodData AI agent (a separate service, in
+`gdc-nas`) through a dataset of natural-language questions and scores what comes back,
+including side-by-side comparison across models. Each dataset item is a JSON envelope
+loaded from a local folder or pulled from a Langfuse dataset. Results are aggregated into
+pass@K / pass^K reports and optionally pushed to Langfuse as scored traces tied to a
+dataset run. The newest and most actively developed package in the repo.
+
+## Owns
+
+- The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
+- Dataset loading and the evaluation run loop
+- Per-capability evaluators and their scoring
+- Result reporting, and pushing runs, scores and trace links to Langfuse
+
+## Does NOT Own
+
+- The agent under evaluation — that lives in `gdc-nas` (gen-ai)
+- Platform access → `gooddata-sdk`
+
+## Architecture
+
+| Path | Role |
+|---|---|
+| `cli/` | argument parsing, and `agentic_runner` — the agentic dispatch and concurrency phases |
+| `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
+| `core/chat/` | SSE client for the agent's streaming chat endpoint |
+| `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
+| `core/dataset/` | dataset format and loading |
+| `core/evaluators/` | single-shot evaluators and their registry |
+| `core/langfuse/` | `sink.py` only — pushes single-turn scores and dataset-run items |
+| `core/reporting/` | console and JSON output rendering |
+| `core/scoring.py`, `core/runner.py` | scoring and orchestration |
+| `core/models.py` | `DatasetItem`, `ChatResult`, `ItemReport` and friends |
+
+**Depends on**: `gooddata-sdk`, `httpx`, `pydantic`, `orjson`, `rich`. The LLM-judge
+evaluator is an optional extra (`llm-judge`, pulling `openai>=1.45,<2.0`); every `openai`
+import site is guarded or deferred so the base install stays usable without it — keep it
+that way.
+
+### Two evaluation paths that share almost nothing
+
+- **Single-shot** kinds send one chat turn and are scored by an `Evaluator` (a Protocol:
+  a `test_kind` attribute plus `evaluate(item, chat_result) -> ItemEvaluation`) looked up
+  from a registry in `core/evaluators/__init__.py`.
+- **Agentic** kinds (`agentic_*`, `vis_agentic`) drive a full multi-turn conversation over
+  the SSE endpoint and are dispatched by an explicit `if`/`elif` chain in
+  `cli/agentic_runner.py`.
+
+Many capabilities exist in **both** forms — visualization, metric skill, alert skill,
+search, general question and guardrail each have a single-turn and a multi-turn
+implementation, sometimes under different `test_kind` strings (`search_tool` vs
+`agentic_search`). These are parallel implementations, not layers.
+
+### Dataset items
+
+`DatasetItem` is the envelope: `id`, `dataset_name`, `test_kind`, `question`, and
+`expected_output: Any`. `expected_output` is deliberately untyped — each evaluator parses
+its own shape. `test_kind` on the item is what labels the result, not the evaluator class,
+which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
+`dashboard_summary` items additionally need `summary_input`.
+
+## Gotchas
+
+**Adding an evaluator is a registry change, not a naming convention.** Single-shot kinds go
+into `_EAGER_EVALUATORS`, or `_LAZY_EVALUATOR_MODULES` plus `_LAZY_EVALUATOR_CLASSES`, in
+`core/evaluators/__init__.py`. Agentic kinds need the string added to `AGENTIC_TEST_KINDS`
+and a new branch in `_dispatch_agentic`. Test file naming follows the capability, but
+naming a test file correctly registers nothing.
+
+**Parallel-safety is a reviewed allowlist, and getting it wrong corrupts results.**
+`WORKSPACE_MUTATING_TEST_KINDS` is computed as `AGENTIC_TEST_KINDS - PARALLEL_SAFE_TEST_KINDS`,
+so a newly added kind defaults to workspace-mutating and runs serially in its own phase.
+That default is correct: agent tool calls create real server-side objects (metrics, alerts).
+Adding a kind to `PARALLEL_SAFE_TEST_KINDS` is a deliberate assertion that it is read-only,
+which nothing in the package can prove for you.
+
+**The SSE client's retry predicate is load-bearing.** `core/chat/sse_client.py` retries
+429/502/503/504 and `httpx.RemoteProtocolError` (a mid-stream disconnect) with exponential
+backoff, and treats a `METADATA_SYNC_IN_PROGRESS` payload as transient. The
+`RemoteProtocolError` case was added after it was confirmed live to contaminate a small
+percentage of visualization runs with a hard fail and no retry. Narrowing that predicate
+reintroduces the problem.
+
+**Langfuse trace linking is deliberately off the item critical path.** Polling for trace
+ingestion has no pass/fail signal and inflates or misattributes per-item latency, so
+`BackgroundTraceLinker` defers it and is drained before the report renders
+(`run_trace_link_inline` is the synchronous alternative). Do not "fix" a slow item by
+making trace scoring synchronous again.
+
+**Scoring weights do not sum to 1.** `quality_score` is the fraction of boolean-valued keys
+in `best_detail` that are true, falling back to `pass_at_k` when there are none (text
+evaluators). `value_score` is `0.6 * quality + 0.2 * speed` — the 0.8 total is what the
+code does; treat it as intentional unless you have checked with the owner.
+
+### Fixture shapes
+
+Group-by / attribute expectations in the alert-skill fixtures are written in AAC shape,
+while the tool arguments the agent emits are AFM-shaped. Never deep-compare those two
+directly — convert, or compare field by field. This applies specifically to the
+attribute/group-by fields: `Filters` in the same fixtures is AFM-shaped on both sides and
+is correctly deep-compared as-is. The attribute comparison itself lands with the
+alert group-by work currently on `jt/gdai-2175-eval-alert-attributes`, so on `master` this
+is guidance for the incoming code rather than a description of what is already there.
+
+## Testing
+
+Plain pytest under `tests/`, no cassettes — the agent is stubbed with
+`unittest.mock`. Tests are named per capability (`test_agentic_*.py`), which is the
+convention to follow when adding one.
+
+`ty` is configured here with `allowed-unresolved-imports` for `openai.**` and
+`gooddata_api_client.**`; do not widen that list to paper over a real typing problem.

gooddata_eval-1.74.1.dev2/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-eval
-Version: 1.74.1.dev1
+Version: 1.74.1.dev2
 Summary: Evaluate the GoodData AI agent against your own questions and models.
 Project-URL: Source, https://github.com/gooddata/gooddata-python-sdk
 Author-email: GoodData <support@gooddata.com>
@@ -17,7 +17,7 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development
 Classifier: Typing :: Typed
 Requires-Python: >=3.10
-Requires-Dist: gooddata-sdk~=1.74.1.dev1
+Requires-Dist: gooddata-sdk~=1.74.1.dev2
 Requires-Dist: httpx<1.0,>=0.27
 Requires-Dist: orjson<4.0.0,>=3.9.15
 Requires-Dist: pydantic<3.0,>=2.6

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev2}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2026 GoodData Corporation
 [project]
 name = "gooddata-eval"
-version = "1.74.1.dev1"
+version = "1.74.1.dev2"
 description = "Evaluate the GoodData AI agent against your own questions and models."
 readme = "README.md"
 license = "MIT"
@@ -11,7 +11,7 @@ authors = [
 keywords = ["gooddata", "ai", "evaluation", "llm", "analytics", "cli"]
 requires-python = ">=3.10"
 dependencies = [
-    "gooddata-sdk~=1.74.1.dev1",
+    "gooddata-sdk~=1.74.1.dev2",
     "httpx>=0.27,<1.0",
     "orjson>=3.9.15,<4.0.0",
     "pydantic>=2.6,<3.0",
@@ -43,8 +43,8 @@ dev = [
     "pytest>=8.3.5",
 ]
 test = [
-    "pytest~=8.3.4",
-    "pytest-cov~=6.0.0",
+    "pytest~=9.1.1",
+    "pytest-cov~=7.1.0",
     "pytest-json-report==1.5.0",
     "pytest-mock>=3.14.0",
 ]

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev2}/src/gooddata_eval/core/agentic/_catalog.py

@@ -2,6 +2,41 @@
 from __future__ import annotations
 
 from dataclasses import dataclass, field
+from enum import Enum
+
+
+class AnomalyDetectionGranularity(str, Enum):
+    """Detection intervals an anomaly alert accepts.
+
+    Mirrors gen-ai's enum of the same name; `StrEnum` is unavailable on the 3.10 floor, so
+    the `str` mixin carries the comparison against the raw tool argument.
+    """
+
+    HOUR = "HOUR"
+    DAY = "DAY"
+    WEEK = "WEEK"
+    MONTH = "MONTH"
+    QUARTER = "QUARTER"
+    YEAR = "YEAR"
+
+    @classmethod
+    def parse(cls, value: object) -> AnomalyDetectionGranularity | None:
+        """Coerce a fixture value, or None when the fixture states none.
+
+        Raises ValueError on an unknown interval: fixtures are hand-written, and a typo has
+        to fail before the run spends an API call rather than score the item against an
+        interval the product cannot produce.
+        """
+        if value is None:
+            return None
+        candidate = str(value).strip().upper()
+        if not candidate:
+            return None
+        try:
+            return cls(candidate)
+        except ValueError:
+            expected = ", ".join(member.value for member in cls)
+            raise ValueError(f"Invalid granularity {value!r}; expected one of {expected}.") from None
 
 
 @dataclass
@@ -28,6 +63,10 @@ class CatalogMetricAlert:
     """List of recipient email addresses."""
     filters: list | str | None = None
     """Attribute filters applied to the alert condition."""
+    attributes: list | None = None
+    """Expected group-by attributes; ``None`` means the fixture states no expectation."""
+    granularity: AnomalyDetectionGranularity | None = None
+    """Detection interval for an ANOMALY alert (DAY/WEEK/MONTH/...). Not a date filter."""
 
     @classmethod
     def from_dict(cls, d: dict) -> CatalogMetricAlert:
@@ -46,4 +85,6 @@ class CatalogMetricAlert:
             metric_id=d.get("metric_id"),
             recipients=recipients,
             filters=d.get("filters"),
+            attributes=d.get("attributes"),
+            granularity=AnomalyDetectionGranularity.parse(d.get("granularity")),
         )

{gooddata_eval-1.74.1.dev1 → gooddata_eval-1.74.1.dev2}/src/gooddata_eval/core/agentic/alert_skill.py

@@ -11,7 +11,7 @@ from typing import Any
 
 from gooddata_sdk import GoodDataSdk
 
-from gooddata_eval.core.agentic._catalog import CatalogMetricAlert
+from gooddata_eval.core.agentic._catalog import AnomalyDetectionGranularity, CatalogMetricAlert
 from gooddata_eval.core.agentic._trace_linker import (
     RunIdentity,
     RunTraceContext,
@@ -21,6 +21,7 @@ from gooddata_eval.core.agentic._trace_linker import (
     submit_trace_scoring,
     utc_now,
 )
+from gooddata_eval.core.chat.render import render_answer_text
 from gooddata_eval.core.chat.sse_client import ChatClient
 from gooddata_eval.core.config import ReasoningEffort
 from gooddata_eval.core.models import (
@@ -29,6 +30,7 @@ from gooddata_eval.core.models import (
     ReasoningStepEvent,
     ToolCallEvent,
     build_latency_breakdown,
+    shift_and_index_events,
 )
 
 try:
@@ -126,6 +128,94 @@ def _check_filters(expected: CatalogMetricAlert, actual_args: dict) -> bool:
     return _deep_subset(exp_filters, act_filters)
 
 
+def _attribute_label_ids(items: list, *, side: str) -> list[str]:
+    """Canonicalise group-by entries to bare label ids, whatever spelling they arrive in.
+
+    The two sides of the comparison speak different vocabularies for the same grouping.
+    Fixtures author the AAC tool-input form, ``{"using": "label/x"}``; ``create_metric_alert``
+    receives the resolved AFM form, ``{"localIdentifier": "a0", "label": {"identifier":
+    {"id": "x", "type": "label"}}}``, forwarded verbatim from ``prepare_metric_alert_proposal``.
+    Identity is therefore the only thing they can be compared on.
+
+    A shape not listed here, or a URI prefix other than ``label/``, raises: ``label/x`` and
+    ``attribute/x`` are different objects, and an unknown spelling must fail loudly rather
+    than quietly compare unequal.
+    """
+    if not isinstance(items, list):
+        raise ValueError(f"Unrecognised {side} group-by attributes, expected a list: {items!r}")
+    ids: list[str] = []
+    for item in items:
+        raw: object = None
+        if isinstance(item, str):
+            raw = item
+        elif isinstance(item, dict):
+            label = item.get("label")
+            identifier = item.get("identifier")
+            if isinstance(item.get("using"), str):
+                raw = item["using"]
+            elif isinstance(label, dict) and isinstance(label.get("identifier"), dict):
+                raw = label["identifier"].get("id")
+            elif isinstance(identifier, dict):
+                raw = identifier.get("id")
+        if not isinstance(raw, str) or not raw:
+            raise ValueError(f"Unrecognised {side} group-by attribute entry: {item!r}")
+        prefix, slash, rest = raw.partition("/")
+        if not slash:
+            ids.append(raw)
+        elif prefix == "label" and rest:
+            ids.append(rest)
+        else:
+            raise ValueError(f"Unrecognised {side} group-by attribute reference: {raw!r}")
+    return ids
+
+
+def _check_attributes(expected: CatalogMetricAlert, actual_args: dict) -> bool:
+    """Compare group-by identity only.
+
+    Per-entry properties — ``showAllValues``, the converter-assigned ``localIdentifier`` —
+    are deliberately not asserted, and the comparison is a multiset so entry order does not
+    matter.
+    """
+    exp_attributes = expected.attributes
+    if exp_attributes is None:
+        return True
+    act_attributes = actual_args.get("attributes")
+    if act_attributes is None:
+        # Arguments are raw `json.loads` output, where an unset nullable argument arrives as
+        # null rather than absent. Both spellings of "no grouping" have to land on [], which
+        # is why this is not `actual_args.get("attributes", [])`.
+        act_attributes = []
+    elif not isinstance(act_attributes, list):
+        # An argument that is not a list of groupings is the agent answering wrongly, so it
+        # scores False. Raising instead would make the runner record an ERROR, and errored
+        # items are excluded from the failure count — a malformed answer must not rank above
+        # a merely wrong one. An unreadable *entry* still raises, in `_attribute_label_ids`:
+        # entries are typed at the tool boundary, so the plausible cause there is the wire
+        # format moving, which has to be unmissable.
+        return False
+    if not exp_attributes:
+        return not act_attributes
+    exp_ids = sorted(_attribute_label_ids(exp_attributes, side="expected"))
+    act_ids = sorted(_attribute_label_ids(act_attributes, side="actual"))
+    return exp_ids == act_ids
+
+
+def _check_granularity(expected: CatalogMetricAlert, actual_args: dict) -> bool:
+    """Compare the ANOMALY detection interval when the fixture states one.
+
+    ``None`` means unasserted, mirroring ``attributes``: only the ANOMALY items carry a
+    ``Granularity``, and every other item must stay unaffected. The expectation is already
+    canonical by the time it lands here; the tool argument is a raw string, so only that
+    side needs folding.
+    """
+    if expected.granularity is None:
+        return True
+    actual = actual_args.get("granularity")
+    if not actual:
+        return False
+    return str(actual).strip().upper() == expected.granularity.value
+
+
 def _check_metric(expected: CatalogMetricAlert, actual_args: dict) -> bool:
     if not expected.metric_id:
         return True
@@ -257,20 +347,38 @@ def generate_simulated_alert_response(
         )
     elif filters == []:
         filters_rule = (
-            "5. Your alert must have NO filters and NO date/time window — it evaluates over all time. "
-            "If the agent asks which time period each check should cover, or offers a choice such as "
-            "'last Day / Week / Month', do NOT pick one: reply that you want no date filter at all, "
-            "all time. Never invent a period, a granularity or an 'evaluate each run on a X basis' "
-            "instruction the goal did not ask for.\n"
+            "5. Your alert must have NO filters and NO date/time window on the metric — it evaluates "
+            "over all time. If the agent asks which time period each check should cover, or offers a "
+            "choice such as 'last Day / Week / Month', do NOT pick one: reply that you want no date "
+            "filter at all, all time.\n"
         )
     else:
         filters_rule = (
             "5. Ask only for the filters your original request implies — do not invent an evaluation "
-            "period, granularity or date window that was not requested. If the agent offers a choice "
+            "period or date window that was not requested. If the agent offers a choice "
             "such as 'last Day / Week / Month' that your request never mentioned, say you do not want "
             "a date window.\n"
         )
 
+    if operator == "ANOMALY":
+        # The fallback keeps the conversation alive when the fixture names no interval -- an
+        # anomaly alert cannot be created without one. It is deliberately NOT mirrored into
+        # `expected.granularity`: `_check_granularity` asserts only what the fixture stated,
+        # and scoring an item against an interval it never asked for is the defect this rule
+        # exists to undo.
+        granularity = (expected.granularity or AnomalyDetectionGranularity.DAY).value
+        anomaly_rule = (
+            "7. This is an ANOMALY alert. Anomaly detection REQUIRES a time granularity, and that "
+            f"granularity is NOT a date filter. State it in your first reply and repeat it whenever "
+            f"asked: use {granularity} granularity. Rule 5 constrains filters on the metric only — it "
+            "never applies to this detection interval, so never refuse to give one.\n"
+        )
+    else:
+        anomaly_rule = (
+            "7. Do not invent an evaluation period, a granularity or an 'evaluate each run on a X "
+            "basis' instruction your goal never asked for.\n"
+        )
+
     original_request = f'Your original request to the agent was: "{question}"\n' if question else ""
 
     system_prompt = (
@@ -294,8 +402,7 @@ def generate_simulated_alert_response(
         "   Do not wait for the agent to ask — state it alongside the metric and condition answers.\n"
         + filters_rule
         + f"6. Proactively state how often you want to be alerted in your first reply: {trigger_request}. "
-        "   Repeat it if the agent proposes a different cadence.\n"
-        "Reply concisely and directly."
+        "   Repeat it if the agent proposes a different cadence.\n" + anomaly_rule + "Reply concisely and directly."
     )
 
     messages: list = [{"role": "system", "content": system_prompt}]
@@ -334,6 +441,8 @@ class AlertEvaluation:
     filters_correct: bool
     metric_correct: bool
     recipients_correct: bool
+    attributes_correct: bool = True
+    granularity_correct: bool = True
 
     @property
     def strict_pass(self) -> bool:
@@ -346,6 +455,8 @@ class AlertEvaluation:
                 self.filters_correct,
                 self.metric_correct,
                 self.recipients_correct,
+                self.attributes_correct,
+                self.granularity_correct,
             ]
         )
 
@@ -409,6 +520,35 @@ def _normalize_expected_filters(expected: dict) -> list | str | None:
     return None
 
 
+_NO_GROUPING_MARKERS = ("none", "no grouping")
+
+
+def _normalize_expected_attributes(expected: dict) -> list | None:
+    """
+    * ``Attributes`` list        -> that list (exact expectation)
+    * "None" / "no grouping"     -> ``[]``   (stated: no group-by; extras fail)
+    * absent, or other prose     -> ``None`` (unstated; grouping not asserted)
+
+    A date narrows an alert as a group-by as well as a filter, and a group-by makes it fire
+    per period value instead of on the latest one — so ``[]`` has to be expressible separately
+    from "absent", exactly as it is for ``filters``.
+
+    The simulated user is told nothing about groupings, so a non-empty expectation requires the
+    item's own question to request that grouping; ``[]`` needs no such support, because the
+    simulated user does not invent a grouping and the check verifies it did not.
+    """
+    attributes = _case_insensitive_get(expected, "attributes")
+    if isinstance(attributes, list):
+        # Validated here so a malformed fixture fails before the run spends an API call.
+        _attribute_label_ids(attributes, side="expected")
+        return attributes
+    if attributes is None:
+        return None
+    if isinstance(attributes, str):
+        return [] if any(kw in attributes.lower() for kw in _NO_GROUPING_MARKERS) else None
+    raise ValueError(f"Attributes expectation must be a list or a display string, got {type(attributes).__name__}")
+
+
 def _normalize_expected_output(expected: dict) -> CatalogMetricAlert:
     """Parse expected_output dict into CatalogMetricAlert, accepting display-format or internal-format keys."""
     operator = _case_insensitive_get(expected, "operator") or "GREATER_THAN"
@@ -433,6 +573,11 @@ def _normalize_expected_output(expected: dict) -> CatalogMetricAlert:
         recipients = list(raw_recip)
 
     filters = _normalize_expected_filters(expected)
+    attributes = _normalize_expected_attributes(expected)
+
+    granularity = AnomalyDetectionGranularity.parse(
+        _case_insensitive_get(expected, "granularity", "detection granularity")
+    )
 
     return CatalogMetricAlert(
         operator=operator,
@@ -443,6 +588,8 @@ def _normalize_expected_output(expected: dict) -> CatalogMetricAlert:
         metric_id=metric_id,
         recipients=recipients,
         filters=filters,
+        attributes=attributes,
+        granularity=granularity,
     )
 
 
@@ -526,21 +673,14 @@ def run_agentic_alert_skill(
                 chat_result = client.send_message(conv_id, current_question)
                 reasoning_steps.extend(chat_result.reasoning_steps or [])
                 response_id = chat_result.response_id or response_id
-                for tc in chat_result.tool_call_events or []:
-                    if tc.call_ts is not None:
-                        tc.call_ts += turn_offset
-                    if tc.result_ts is not None:
-                        tc.result_ts += turn_offset
-                    if tc.index is not None:
-                        tc.index += tool_index_offset
-                for rs in chat_result.reasoning_step_events or []:
-                    rs.ts += turn_offset
-                    rs.index += reasoning_index_offset
+                turn_offset, tool_index_offset, reasoning_index_offset = shift_and_index_events(
+                    chat_result,
+                    turn_offset=turn_offset,
+                    tool_index_offset=tool_index_offset,
+                    reasoning_index_offset=reasoning_index_offset,
+                )
                 all_tool_call_events.extend(chat_result.tool_call_events or [])
                 all_reasoning_step_events.extend(chat_result.reasoning_step_events or [])
-                tool_index_offset += len(chat_result.tool_call_events or [])
-                reasoning_index_offset += len(chat_result.reasoning_step_events or [])
-                turn_offset += chat_result.turn_wall_clock_sec or 0.0
                 alert_id, actual_args, tool_called = _extract_alert_call(chat_result.tool_call_events or [])
                 if tool_called:
                     alert_id_to_delete = alert_id
@@ -548,6 +688,8 @@ def run_agentic_alert_skill(
                 response_text = (chat_result.text_response or "").strip()
                 if not response_text and chat_result.alert_proposals:
                     response_text = render_alert_proposal(chat_result.alert_proposals[-1])
+                if not response_text:
+                    response_text = render_answer_text(chat_result)
                 # Stop if agent gave a completely empty response (stuck)
                 if not response_text and not chat_result.tool_call_events:
                     break
@@ -570,6 +712,8 @@ def run_agentic_alert_skill(
                 filters_correct=tool_called and _check_filters(expected, actual_args),
                 metric_correct=tool_called and _check_metric(expected, actual_args),
                 recipients_correct=tool_called and _check_recipients(expected, actual_args, sdk=sdk),
+                attributes_correct=tool_called and _check_attributes(expected, actual_args),
+                granularity_correct=tool_called and _check_granularity(expected, actual_args),
             )
             return AlertRunResult(
                 conversation_id=conv_id,
@@ -615,6 +759,8 @@ def run_agentic_alert_skill(
                 r.eval.filters_correct,
                 r.eval.metric_correct,
                 r.eval.recipients_correct,
+                r.eval.attributes_correct,
+                r.eval.granularity_correct,
             ]
         ),
     )
@@ -689,6 +835,8 @@ def evaluate_agentic_alert_skill(
                     "filters_correct": ev.filters_correct,
                     "metric_correct": ev.metric_correct,
                     "recipients_correct": ev.recipients_correct,
+                    "attributes_correct": ev.attributes_correct,
+                    "granularity_correct": ev.granularity_correct,
                 }
                 with ctx.observe(pt, run_idx) as tid:
                     for score_name, value in strict_checks.items():
@@ -735,6 +883,8 @@ def evaluate_agentic_alert_skill(
         "filters_correct": ev.filters_correct,
         "metric_correct": ev.metric_correct,
         "recipients_correct": ev.recipients_correct,
+        "attributes_correct": ev.attributes_correct,
+        "granularity_correct": ev.granularity_correct,
         "actual_alert_arguments": best.actual_alert_arguments,
         "latency_breakdown": build_latency_breakdown(best.tool_call_events, best.reasoning_step_events),
     }
@@ -745,7 +895,9 @@ def evaluate_agentic_alert_skill(
             f"alert_created={ev.alert_created}, operator_correct={ev.operator_correct}, "
             f"threshold_correct={ev.threshold_correct}, trigger_correct={ev.trigger_correct}, "
             f"filters_correct={ev.filters_correct}, metric_correct={ev.metric_correct}, "
-            f"recipients_correct={ev.recipients_correct}. "
+            f"recipients_correct={ev.recipients_correct}, "
+            f"attributes_correct={ev.attributes_correct}, "
+            f"granularity_correct={ev.granularity_correct}. "
             f"Actual args: {best.actual_alert_arguments}"
         )
         exc.reasoning_steps = best.reasoning_steps