opencode-skills-collection 4.1.15 → 4.1.16

package/bundled-skills/monte-carlo-monitoring-advisor/references/agent-monitor-creation.md

@@ -0,0 +1,471 @@
+# Agent Monitor Creation Procedure
+
+This is the agent monitor creation procedure for AI agent observability. Follow
+these steps in order when a user asks to monitor their AI agents — setting up
+alerts on agent behavior or creating agent monitors.
+
+All tools are available via the `monte-carlo-mcp` MCP server.
+
+---
+
+## Step 1: Discover agents
+
+Call `get_agent_metadata` to list all AI agents in the account. Present agent
+names to the user (never expose MCONs or internal IDs). Ask which agent(s)
+they want to monitor.
+
+Key fields in the response:
+
+| Field | Description |
+|-------|-------------|
+| `agentName` | Human-readable agent name |
+| `agentReference` | The value to pass as the `agent` arg when creating monitors — a platform `{database}:{schema}.{name}` reference (Snowflake Cortex / Databricks) or an OpenTelemetry `service_name`. May be null for agents that cannot be referenced. |
+| `traceTableMcon` | Trace table MCON — used as the `trace_table_mcon` input for the read tools (`get_agent_conversations`, `get_agent_conversation`, `get_agent_traces`, `get_agent_segments`; the parameter is named `mcon` on `get_agent_trace`) |
+| `sourceType` | `TRACE_TABLE` (custom) or `PLATFORM_AGENT` (Monte Carlo native) |
+| `backend_class` | Which backend the agent's traces live in — `ao_clickhouse_otel`, `platform_agent`, `customer_otel_trace_table`, `databricks_genie`, `databricks_mlflow_sdk`, or `databricks_mlflow_ka`. Null when the server could not classify the agent (or predates the field). |
+| `warehouse_uuid` | Warehouse holding the agent's trace data — the value to pass as the `warehouse` arg when creating monitors. Null when the warehouse was deleted or cannot be resolved; fall back to `get_warehouses` (see Warehouse below). |
+| `warehouse_name` | Display name of that warehouse — what you show the user. Null alongside `warehouse_uuid`; fall back to `get_warehouses`. |
+
+**What `backend_class` tells you about capabilities:** conversation-grain
+evaluation monitors (`is_agent_conversation_aggregation=True`) are supported for
+`ao_clickhouse_otel`, `platform_agent` (Snowflake Cortex), and `databricks_genie`;
+the MLflow classes are span-only (the backend rejects conversation aggregation for
+them). At conversation grain, set `include_tool_calls: true` on every eval transform
+by default — it adds the agent's tool calls (name, inputs, outputs, errors) to the
+judged conversation as clearly identifiable TOOL entries, in call order, so evals
+score what the agent did, not just what it said. Omit it only for pure style/tone
+judges; the field is rejected at span grain. `databricks_genie` agents emit no
+token or model data — skip token-usage
+metrics for them (see `agent-metric-monitor.md`). A null `backend_class` means the
+server couldn't classify the agent — default to span-grain proposals.
+
+**Duplicate agent names:** The same agent name may appear more than once (e.g.,
+deployed in both prod and staging). Each entry is distinguished by its own
+`agentReference`, `traceTableMcon`, and warehouse — ask the user which one they
+want to monitor and pass that entry's `agentReference` verbatim. When you ask the
+user to choose, present each entry's `warehouse_name`, never a UUID.
+
+---
+
+## Step 2: Investigate agent behavior
+
+Use the read tools to understand the agent's behavior before suggesting monitors.
+These read tools are your sampling surface — do **not** query the trace store with
+SQL. Agents are tracked as agents, not tables: platform agents (Snowflake Cortex /
+Databricks) have no queryable trace table, and for custom agents the raw table's
+columns are not the fields monitors use.
+
+### 2a. Review recent conversations
+
+Call `get_agent_conversations` with the agent's `agent_name` and `trace_table_mcon`
+to list recent conversations (newest first). Filter to surface interesting ones —
+`has_errors`, `status`, or turn/token/duration bounds — and set
+`include_transcript=True` to read the prompt/completion transcripts inline. Drill
+into one conversation you already have the id for with `get_agent_conversation`.
+Look for:
+
+- **Error patterns** — spans with error status or failure indicators
+- **Latency outliers** — unusually long durations
+- **Token usage** — high token counts that may indicate inefficiency
+- **Conversation quality** — check prompt/completion text for relevance
+
+### 2b. Inspect execution shape and traces
+
+Call `get_agent_traces` to list traces with per-trace `workflows`, `tasks`,
+`models`, `count_llm_calls`, `total_tokens`, `duration_seconds`, and error counts —
+sort by the field you plan to monitor to see typical values and outliers. Call
+`get_agent_segments` to enumerate the distinct `workflow` / `task` / `model` values
+so you can scope a monitor to a real segment. Then pick a trace id and call
+`get_agent_trace` to see the full span tree. Look for:
+
+- **Excessive tool calls** — an agent calling the same tool many times
+- **Missing steps** — expected spans that don't appear
+- **Error cascades** — a failed span causing downstream failures
+- **Unusual paths** — the agent taking an unexpected execution route
+
+You are identifying **what** to monitor — you don't need exact percentiles up
+front; anomaly-detection operators (`AUTO`) learn the baseline themselves.
+
+### 2c. Summarize the agent before proposing
+
+Condense the investigation into a short agent understanding and show it to the
+user — every monitor you propose should trace back to an item in it:
+
+- **Purpose** — 1–2 sentences on what this agent does, grounded in the sampled
+  transcripts (e.g. "a revenue-analytics assistant that answers questions about
+  bookings").
+- **Conversational?** — multi-turn user conversations (eval-worthy for
+  satisfaction / task completion) vs. a batch / single-shot pipeline where
+  structural and span checks fit better.
+- **Tools and the dominant span** — which tool spans the agent runs, and which
+  one does its core work (the SQL execution tool for an analytics agent,
+  retrieval for a RAG agent).
+- **Healthy trajectory shape** — how many times the dominant span runs per answer
+  in healthy traces (the per-trace distribution and its max), typical turn counts,
+  and latency/token magnitudes. This is the basis for every derived threshold.
+- **Recurring intents** — what users repeatedly ask (from the transcripts) —
+  seeds for custom conversation evals.
+- **Observed failure modes** — what actually went wrong in the sample — seeds for
+  evals and structural monitors.
+- **Existing monitors** — from `get_monitors`, so proposals don't duplicate
+  coverage.
+
+---
+
+## Propose with the POBC framing (walk the user through all four pillars)
+
+When the user is setting up monitoring for an agent (rather than asking for one
+specific monitor), structure the proposal around the four pillars of agent
+observability — **Performance, Output, Behavior, Context (POBC)** — and walk
+through them one at a time. Do NOT dump every proposed monitor in one
+monolithic list.
+
+### 1. Open with the framing
+
+Before presenting any monitors, briefly explain the framework. Use this copy,
+adapting the agent's actual name into the prose where it reads naturally:
+
+> A quick word on how we think about agent observability. Agents fail in four
+> distinct ways, so we monitor four distinct things — **Performance, Output,
+> Behavior, and Context (POBC)**: how efficiently the agent answers, what it
+> says, how it gets there, and the data it stands on.
+>
+> **Performance** — is it fast and affordable? Latency, token cost, and error
+> monitoring catch drift in both the typical experience and the worst one.
+>
+> **Output** — is the agent giving good answers? Evals score response quality
+> (helpfulness, non-answers, user corrections) so quality regressions surface
+> as alerts, not user complaints.
+>
+> **Behavior** — is it working sensibly under the hood? Trajectory monitoring
+> flags runs that loop or take paths a healthy run never takes — including
+> failures the agent recovers from and hides.
+>
+> **Context** — is the data it relies on healthy? The agent's answers are only
+> as good as its upstream tables; we monitor those for freshness, schema
+> changes, and anomalies.
+>
+> Everything below maps to one of these four. Here's the plan:
+
+### Monitor conventions — every monitor in this playbook
+
+Four conventions apply to EVERY monitor created in this playbook — the agent
+monitors (metric, evaluation, trajectory, validation) AND any warehouse
+data-quality monitor created for the Context pillar (table or field monitors
+on the agent's upstream tables; see `data-monitor-creation.md`):
+
+- **Agent tag on every create.** Pass
+  `tags=[{"name": "agent", "value": "<AGENT_NAME>"}]`, where `<AGENT_NAME>` is
+  the agent's display name exactly as returned by `get_agent_metadata`
+  (trimmed, case preserved — do not slugify or rename). This single tag is the
+  footprint contract: one filter retrieves everything the playbook created for
+  this agent.
+- **Audit / teardown contract.**
+  `get_monitors(monitor_tags=["agent:<AGENT_NAME>"])` (or the UI monitors tag
+  filter) returns the agent's full monitoring footprint — use it to audit
+  what exists, tune, or tear down everything for an agent. This is why a
+  create call without the tag is a defect: it silently drops the monitor out
+  of the footprint.
+- **Audience — ask once, apply everywhere.** Before the FIRST create call of
+  the playbook (not per monitor), ask the user once which audiences should be
+  notified when these monitors fire — call `get_audiences` to list the
+  options; the user can pick one, several, or none. Pass the chosen audience
+  **names** (labels, never UUIDs) as `audiences` on EVERY monitor created in
+  the playbook, and set `failure_audiences` to the same selection unless the
+  user asks for a different failure-notification audience. Do not re-ask per
+  pillar or per monitor; if the user declines, omit `audiences`.
+- **Domain — same for every monitor.** If the account uses domains, resolve
+  one domain for the agent's footprint and pass the same `domain_uuids` on
+  every monitor — including the Context-pillar DQ monitors (resolve via the
+  domain-assignment steps in `data-monitor-creation.md`) — so the whole
+  footprint lives in one domain.
+
+### 2. Walk through the plan pillar by pillar
+
+Present the pillars in order — Performance, Output, Behavior, Context — one
+short block each:
+
+1. **Evidence** — one or two sentences of what you observed in Step 2 that
+   motivates this pillar's monitors ("p95 latency is 40s with outliers over
+   three minutes", "several conversations show repeated user corrections").
+   If you found nothing notable for a pillar, say so and propose baseline
+   coverage anyway — monitoring exists to catch what hasn't happened yet.
+2. **Proposed monitors** — the specific monitors for this pillar, each with
+   its monitor type, field or judge, and alert condition.
+3. **Confirm** — ask whether to keep, adjust, or drop this pillar's monitors,
+   and fold the answer in before moving to the next pillar.
+
+A healthy agent usually warrants coverage in every pillar you can serve —
+keep proposals broad across pillars, not deep in one.
+
+**Global defaults for proposed monitors** (apply unless the user asks
+otherwise):
+
+- **Daily schedule** — pass `interval_minutes=1440` explicitly; the tools'
+  built-in default is hourly (see Schedule configuration below).
+- **Eval sampling** — `sampling_config={"count": 100}` (a fixed 100-sample
+  budget per run), not a percentage, so evaluation cost stays predictable as
+  traffic grows.
+- **Audience on every create** — apply the playbook-level audience selection
+  (see "Monitor conventions — every monitor in this playbook" above); never
+  create a monitor without that once-asked selection applied (Step 4).
+
+What each pillar maps to:
+
+| Pillar | Monitor types | Reference |
+|---|---|---|
+| **Performance** | Metric (validation for hard limits) — latency (`duration_sec`), token cost (`total_tokens`), error rate (`status_code`), volume (`ROW_COUNT_CHANGE`) | `agent-metric-monitor.md` |
+| **Output** | Evaluation — lead with the Output-pillar starting packs (see Step 3): baseline pack for every agent, analytics pack for Cortex/Genie. Add predefined judges (`answer_relevance`, `task_completion`, `clarity`, `prompt_adherence`), rule checks (`output_length`, `json_validity`), and one custom eval per recurring user intent or failure mode you observed | `agent-evaluation-monitor.md` |
+| **Behavior** | Trajectory (validation for aggregate assertions) — runaway loops (`SPAN_OCCURRENCE`), missing or mis-ordered steps (`SPAN_RELATION`), token budgets | `agent-trajectory-monitor.md`, `agent-validation-monitor.md` |
+| **Context** | Table monitors (freshness / schema changes / volume) on the agent's upstream tables, plus an optional `Context for {AGENT_NAME}` data product wrapper — see below | `data-table-monitor.md` |
+
+Mind the backend caveats from Step 1 (`backend_class`): no token or model
+metrics for Genie / Knowledge Assistant agents, and conversation-grain evals
+only on OTel/ClickHouse, Snowflake Cortex, and Genie — with
+`include_tool_calls: true` on every conversation-grain eval transform by default
+(rejected at span grain; see Step 1). Aggregate (per-trace)
+validation assertions require `is_agent_trace_aggregation=True`, supported
+only on `ao_clickhouse_otel` / `customer_otel_trace_table` agents — on other
+backends, use per-span assertions instead.
+
+**Context — monitor the tables the agent reads.** Unlike the other pillars,
+Context coverage lives on warehouse tables, not spans. Automatic
+lineage-derived table discovery is not available on this surface, so ask the
+user which upstream tables the agent depends on (the tables its SQL tools
+query, its knowledge bases are built from, or its features are loaded from) —
+never guess table names, and never drop the pillar silently. On the named
+tables:
+
+1. **Create the table monitors** with `create_or_update_table_monitor`,
+   following `data-table-monitor.md` for warehouse resolution and asset
+   selection (scope as narrowly as that reference allows). The tool's
+   default alert conditions are exactly the Context coverage — freshness,
+   schema changes, and volume — so omit `alert_conditions` unless the user
+   asks for more. Apply the Monitor conventions above on every create: the
+   `agent` tag, the playbook `audiences`, and `domain_uuids`. Dry-run
+   preview first, deploy on explicit confirmation, like every other create
+   in this playbook.
+2. **Optionally wrap the tables in a data product** named
+   `Context for {AGENT_NAME}` via `create_or_update_data_product`: pass the
+   tables' `mcons` (from `search` / `get_table` — do not guess them) and a
+   `description` naming the agent. Keep the default `dry_run=True` to show
+   the asset-footprint preview; set `dry_run=False` only on explicit
+   confirmation. The preview's asset count can exceed the tables you
+   named — the tool's backend automatically expands the footprint to
+   include their upstream dependencies. When the count is notably larger
+   than the named set, explain to the user what is being added before
+   asking them to confirm the live create. Two caveats: data products
+   take `audience_ids` — UUIDs from `get_audiences` — unlike monitors,
+   which take audience names; and the data product itself carries no
+   `agent` tag (the footprint contract rides on the monitors). If the
+   live create is rejected because the account lacks the Data Mesh
+   module, that is terminal for the wrapper: say so in one line (their
+   Monte Carlo representative can enable it) and keep the table monitors
+   — the data product is packaging; the monitors are the pillar.
+3. **Add field-level depth** where the user wants specific field checks
+   (null rates, distributions, custom rules) on an upstream table: use the
+   data-monitor workflow's field-level references (`data-metric-monitor.md`,
+   `data-validation-monitor.md`, `data-custom-sql-monitor.md`), carrying
+   the same tag, audiences, and domain on every create.
+
+If the user cannot name any upstream tables, present the pillar as a
+recommendation — name what you would monitor and why — rather than failing
+or silently dropping it.
+
+### 3. Create the confirmed monitors
+
+Once the user has confirmed the pillars, continue to Step 3 (pick each
+monitor's reference doc) and Step 4 (dry-run preview, applying the
+already-collected audience selection, creation on explicit confirmation) for
+each approved monitor.
+
+---
+
+## The `agent` reference
+
+All four `create_or_update_agent_*_monitor` tools author the monitor's source from
+a single top-level **`agent`** argument (there is no `dw_id` and no `data_source`
+argument — the `agent` reference is the whole source). Two accepted forms:
+
+- **Platform agent reference** — `{database}:{schema}.{name}` (Snowflake Cortex /
+  Databricks agents), e.g. `analytics:agents.support_bot`.
+- **OpenTelemetry `service_name`** — for OTel-instrumented agents, e.g. `checkout-agent`.
+
+Get the exact value from `get_agent_metadata`'s **`agentReference`** field and pass
+it verbatim — never construct, modify, or truncate it, and never pass an MCON. Two
+optional companions:
+
+- `trace_table` — only for non-ClickHouse OTel agents whose trace storage cannot be
+  inferred from the agent reference.
+- The per-type reference tells you whether `warehouse` is required (see below).
+
+---
+
+## Warehouse
+
+`warehouse` names the warehouse the agent's trace data lives in — pass it as a name
+or UUID. Use the agent entry's `warehouse_uuid` from `get_agent_metadata` (and its
+`warehouse_name` when talking to the user); when both are null, use `get_warehouses`.
+Whether `warehouse` is required or optional depends on the monitor type — see the
+per-type reference.
+
+---
+
+## Agent span filters
+
+The optional `agent_span_filters` parameter refines which spans are monitored. It
+accepts **at most one** filter object. Each field of the object holds a
+`{"value": "..."}` sub-object.
+
+| Filter field | Description | Example |
+|-------------|-------------|---------|
+| `agent` | Filter by agent name | `{"agent": {"value": "My Agent"}}` |
+| `workflow` | Filter by workflow name | `{"workflow": {"value": "Chat Agent"}}` |
+| `task` | Filter by task name | `{"task": {"value": "call_model"}}` |
+| `span_name` | Filter by span name | `{"span_name": {"value": "ChatBedrockConverse.chat"}}` |
+
+Multiple fields can be combined in the single filter object:
+
+```json
+[{"workflow": {"value": "Chat Agent"}, "task": {"value": "call_model"}}]
+```
+
+`agent_span_filters` is a refinement and is optional — the `agent` reference already
+scopes the monitor. Some monitor types restrict which fields are allowed here (e.g.
+trajectory monitors, and trace-aggregated metric / validation monitors) — see the
+per-type reference for the exact rule.
+
+---
+
+## Schedule configuration
+
+**Propose daily schedules by default** — pass `interval_minutes=1440` explicitly.
+Schedule is set via two top-level args, not a nested object:
+
+- `schedule_type` — defaults to `fixed`. Valid values: `fixed`, `manual`.
+- `interval_minutes` — defaults to `60` (hourly), so omitting it creates an hourly
+  monitor. The floor and alignment differ per monitor type — see each reference.
+
+No agent monitor accepts a dynamic schedule — use `fixed` or `manual`. Daily is
+the right cadence for most agents. If you judge an agent critical enough that a
+same-hour alert would matter, suggest hourly to the user and let them decide —
+the default stays daily unless they opt in.
+
+---
+
+## Time filter configuration
+
+Used by trajectory and validation monitors. The `time_field` is an object with
+a `field` property — always use `ingest_ts`:
+
+```json
+{"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24}
+```
+
+---
+
+## Step 3: Choose the right monitor type
+
+Based on your investigation, recommend one or more monitor types. Read the
+corresponding reference doc for the detailed creation guide.
+
+| I want to... | Monitor type | Reference file |
+|-------------|-------------|----------------|
+| Track a numeric metric trend (latency, tokens) | Agent Metric | `agent-metric-monitor.md` |
+| Set up performance coverage (latency, token cost, errors, SLO) | Agent Metric — Performance pillar | `agent-metric-monitor.md` |
+| Score output quality with LLM evaluation | Agent Evaluation | `agent-evaluation-monitor.md` |
+| Alert on execution patterns or span sequences | Agent Trajectory | `agent-trajectory-monitor.md` |
+| Assert a logical rule on span data | Agent Validation | `agent-validation-monitor.md` |
+| Monitor span volume over time | Agent Metric | `agent-metric-monitor.md` |
+| Detect answer relevance drops | Agent Evaluation | `agent-evaluation-monitor.md` |
+| Catch runaway tool call loops | Agent Trajectory | `agent-trajectory-monitor.md` |
+| Ensure token count stays below threshold | Agent Validation | `agent-validation-monitor.md` |
+
+**Output-pillar starting packs** — for a newly onboarded agent (or one with no eval coverage
+yet), lead with the named packs from `agent-evaluation-monitor.md` rather than inventing a
+one-off list:
+
+- **Baseline pack — every agent:** the predefined `helpfulness_conversation` judge (plain
+  `helpfulness` on span-grain-only backends) plus the `frustration_free_score` template.
+  Defaults: daily schedule (`interval_minutes=1440`), `{"count": 100}` sampling, an `agent`
+  tag (`{"name": "agent", "value": "<AGENT_NAME>"}`) on every monitor, and
+  `include_tool_calls: true` on every conversation-grain transform (see the `backend_class`
+  capabilities in Step 1).
+- **Analytics pack — only when `backend_class` is `platform_agent` (Snowflake Cortex) or
+  `databricks_genie`:** the `answer_attempt_score` and `user_correction` templates — the
+  dominant NL2SQL/analytics failure modes are deflected answers and user-corrected answers.
+  Do not propose this pack for other agents.
+
+Render each template with the agent's actual name and observed intents (never boilerplate) and
+show the full prompt text for approval — see "Custom-prompt template library" and
+"Output-pillar eval packs" in `agent-evaluation-monitor.md`.
+
+After selecting the monitor type, **read the reference doc** for that type to
+get the detailed parameter guide, examples, constraints, and creation workflow.
+For a blanket "performance monitoring" ask, follow the **Performance pillar**
+baseline set in `agent-metric-monitor.md` rather than assembling one-offs.
+
+### Behavior monitors — two trajectory proposals for (almost) every agent
+
+Grounded in the Step 2c summary, propose these two patterns whenever they apply
+(`agent-trajectory-monitor.md` has the full playbooks and payload shapes):
+
+1. **Runaway loop — create live.** SPAN_OCCURRENCE on the agent's dominant tool
+   span, threshold derived from the observed per-trace occurrence distribution:
+   max observed + headroom, never a stock number. The proposal's evidence must
+   show the dominant span, the distribution, and the derived threshold with its
+   headroom rationale — plus a pre-create breach `preview` (dry run) proving zero
+   historical matches; if the preview breaches, the sample missed the heavy tail
+   (e.g. multi-turn accumulation in one trace) — re-derive from a wider window.
+   Zero historical matches is the point — it is a regression guardrail that stays
+   silent until the agent's behavior regresses.
+2. **Ungrounded-in-data — create as a DRAFT** (only for agents that answer
+   questions from data). Negated `occurs_with` SPAN_RELATION: an answer was
+   produced without the agent's data-access tool span. Show a breach `preview`
+   (dry run) as evidence, then create with `is_draft=True` — generic questions
+   legitimately skip the data tool, and the LLM-judge filter needed to separate
+   them from real data questions cannot be combined with a trajectory condition
+   yet.
+
+Tag both with the agent's name (`tags=[{"name": "agent", "value": "<AGENT_NAME>"}]`)
+and schedule them daily (`interval_minutes=1440`).
+
+Beyond these two, propose **agent-tailored behavioral custom prompts** for
+behaviors a span pattern can't see — e.g. "did the agent claim it ran a query it
+never executed?", "did the agent re-ask for information the user already gave?".
+One boolean `custom_prompt` per behavior, alerting on `TRUE_RATE` / `FALSE_RATE`,
+at conversation grain where the backend supports it (see `backend_class` in
+Step 1 and `agent-evaluation-monitor.md`) — keep `include_tool_calls: true` on
+these so the judge can see the tool calls it is judging.
+
+---
+
+## Step 4: Create the monitor
+
+All four tools follow the same **two-call preview-then-confirm pattern** as the data
+monitor tools: the first call (`dry_run=True`, the default) returns the rendered MaC
+YAML for review; the second call (`dry_run=False`) deploys the monitor live and
+returns its UUID. Pass `monitor_uuid` on either call to update an existing agent
+monitor in place instead of creating a new one (PUT semantics — re-pass every field
+you want to keep, since omitted fields revert to defaults).
+
+1. **Always start with `dry_run=True`** (the default). Show the user the
+   configuration preview (the rendered YAML).
+2. **Apply the playbook-level audience selection** (see "Monitor conventions —
+   every monitor in this playbook"): the audience question was already asked
+   once before the playbook's first create — pass that same selection of
+   audience **names** (not UUIDs) as the `audiences` list, and default
+   `failure_audiences` to the same selection. Fall back to asking here (one
+   question, `get_audiences` for options) only when the playbook-level ask has
+   not happened (e.g. the user jumped straight to a single monitor outside the
+   walkthrough).
+3. After showing the preview, offer to create or adjust settings.
+4. Only set `dry_run=False` when the user explicitly confirms creation.
+
+---
+
+## Field name reference
+
+See `agent-span-fields.md` for the complete list of known span field names
+available in agent monitors. Do not guess field names — use only the ones
+documented there.