@ivannxbt/ai-config 0.1.0

package/.agents/skills/microsoft-foundry/foundry-agent/observe/observe.md

@@ -0,0 +1,51 @@
+# Agent Observability Loop
+
+Orchestrate the full eval-driven optimization cycle for a Foundry agent. This skill manages the **multi-step workflow** — auto-creating evaluators, generating test datasets, running batch evals, clustering failures, optimizing prompts, redeploying, and comparing versions. Use this skill instead of calling individual foundry-mcp evaluation tools manually.
+
+## When to Use This Skill
+
+USE FOR: evaluate my agent, run an eval, test my agent, check agent quality, run batch evaluation, analyze eval results, why did my eval fail, cluster failures, improve agent quality, optimize agent prompt, compare agent versions, re-evaluate after changes, set up CI/CD evals, agent monitoring, eval-driven optimization.
+
+> ⚠️ **DO NOT manually call** `evaluation_agent_batch_eval_create`, `evaluator_catalog_create`, `evaluation_comparison_create`, or `prompt_optimize` **without reading this skill first.** This skill defines required pre-checks, artifact persistence, and multi-step orchestration that the raw tools do not enforce.
+
+## Quick Reference
+
+| Property | Value |
+|----------|-------|
+| MCP server | `foundry-mcp` |
+| Key MCP tools | `evaluation_agent_batch_eval_create`, `evaluator_catalog_create`, `evaluation_comparison_create`, `prompt_optimize`, `agent_update` |
+| Prerequisite | Agent deployed and running (use [deploy skill](../deploy/deploy.md)) |
+
+## Entry Points
+
+| User Intent | Start At |
+|-------------|----------|
+| "Deploy and evaluate my agent" | [Step 1: Deploy & Setup](references/deploy-and-setup.md) |
+| "Evaluate my agent" / "Run an eval" | [Step 2: Evaluate](references/evaluate-step.md) |
+| "Why did my eval fail?" / "Analyze results" | [Step 3: Analyze](references/analyze-results.md) |
+| "Improve my agent" / "Optimize prompt" | [Step 4: Optimize](references/optimize-deploy.md) |
+| "Compare agent versions" | [Step 5: Compare](references/compare-iterate.md) |
+| "Set up CI/CD evals" | [Step 6: CI/CD](references/cicd-monitoring.md) |
+
+## Before Starting — Detect Current State
+
+1. Check `.env` for `AZURE_AI_PROJECT_ENDPOINT` and `AZURE_AI_AGENT_NAME`
+2. Use `agent_get` and `agent_container_status_get` to verify the agent exists and is running
+3. Use `evaluation_get` to check for existing eval runs
+4. Jump to the appropriate entry point
+
+## Behavioral Rules
+
+1. **Auto-poll in background.** After creating eval runs or starting containers, poll in a background terminal. Only surface the final result.
+2. **Confirm before changes.** Show diff/summary before modifying agent code or deploying. Wait for sign-off.
+3. **Prompt for next steps.** After each step, present options. Never assume the path forward.
+4. **Write scripts to files.** Python scripts go in `scripts/` — no inline code blocks.
+5. **Persist eval artifacts.** Save to `evaluators/`, `datasets/`, and `results/` for version tracking (see [deploy-and-setup](references/deploy-and-setup.md) for structure).
+
+## Related Skills
+
+| User Intent | Skill |
+|-------------|-------|
+| "Analyze production traces" / "Search conversations" / "Find errors in App Insights" | [trace skill](../trace/trace.md) |
+| "Debug container issues" / "Container logs" | [troubleshoot skill](../troubleshoot/troubleshoot.md) |
+| "Deploy or redeploy agent" | [deploy skill](../deploy/deploy.md) |

package/.agents/skills/microsoft-foundry/foundry-agent/observe/references/analyze-results.md

@@ -0,0 +1,47 @@
+# Steps 3–5 — Download Results, Cluster Failures, Dive Into Category
+
+## Step 3 — Download Results
+
+`evaluation_get` returns run metadata but **not** full per-row output. Write a Python script (save to `scripts/`) to download detailed results:
+
+1. Initialize `AIProjectClient` with project endpoint and `DefaultAzureCredential`
+2. Get OpenAI client via `project_client.get_openai_client()`
+3. Call `openai_client.evals.runs.output_items.list(eval_id=..., run_id=...)`
+4. Serialize each item with `item.model_dump()` and save to `results/<eval-id>/<run-id>.json` (use `default=str` for non-serializable fields)
+5. Print summary: total items, passed, failed, errored counts
+
+> SDK setup: `pip install azure-ai-projects azure-identity openai`
+
+## Step 4 — Cluster Failures by Root Cause
+
+Analyze every row in the results. Group failures into clusters:
+
+| Cluster | Description |
+|---------|-------------|
+| Incorrect / hallucinated answer | Agent gave a wrong or fabricated response |
+| Incomplete answer | Agent missed key parts |
+| Tool call failure | Agent failed to invoke or misused a tool |
+| Safety / content violation | Flagged by safety evaluators |
+| Runtime error | Agent crashed or returned an error |
+| Off-topic / refusal | Agent refused or went off-topic |
+
+Produce a **prioritized action table**:
+
+| Priority | Cluster | Suggested Action |
+|----------|---------|------------------|
+| P0 | Runtime errors | Check container logs |
+| P1 | Incorrect answers | Optimize prompt ([Step 6](optimize-deploy.md)) |
+| P2 | Incomplete answers | Optimize prompt ([Step 6](optimize-deploy.md)) |
+| P3 | Tool call failures | Fix tool definitions or instructions |
+| P4 | Safety violations | Add guardrails to instructions |
+| P5 | Off-topic / refusal | Clarify scope in instructions |
+
+**Rule:** Runtime errors first (P0), then by count × severity.
+
+## Step 5 — Dive Into Category
+
+When the user wants to inspect a specific cluster, display the individual rows: input query, the agent's original response, evaluator scores, and failure reason. Let the user confirm which category to optimize.
+
+## Next Steps
+
+After clustering → proceed to [Step 6: Optimize Prompt](optimize-deploy.md).

package/.agents/skills/microsoft-foundry/foundry-agent/observe/references/cicd-monitoring.md

@@ -0,0 +1,35 @@
+# Step 11 — Enable CI/CD Evals & Continuous Monitoring
+
+After confirming the final agent version, prompt with two options:
+
+## Option 1 — CI/CD Evaluations
+
+*"Would you like to add automated evaluations to your CI/CD pipeline so every deployment is evaluated before going live?"*
+
+If yes, generate a GitHub Actions workflow (e.g., `.github/workflows/agent-eval.yml`) that:
+
+1. Triggers on push to `main` or on pull request
+2. Reads evaluator definitions from `evaluators/` and test datasets from `datasets/`
+3. Runs `evaluation_agent_batch_eval_create` against the newly deployed agent version
+4. Fails the workflow if any evaluator score falls below configured thresholds
+5. Posts a summary as a PR comment or workflow annotation
+
+Use repository secrets for `AZURE_AI_PROJECT_ENDPOINT` and Azure credentials. Confirm the workflow file with the user before committing.
+
+## Option 2 — Continuous Production Monitoring
+
+*"Would you like to set up continuous evaluations to monitor your agent's quality in production?"*
+
+If yes, generate a scheduled GitHub Actions workflow (e.g., `.github/workflows/agent-eval-scheduled.yml`) that:
+
+1. Runs on a cron schedule (ask user preference: daily, weekly, etc.)
+2. Evaluates the current production agent version using stored evaluators and datasets
+3. Saves results to `results/`
+4. Opens a GitHub issue or sends a notification if any score degrades below thresholds
+
+The user may choose one, both, or neither.
+
+## Reference
+
+- [Azure AI Foundry Cloud Evaluation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation)
+- [Hosted Agents](https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/hosted-agents)

package/.agents/skills/microsoft-foundry/foundry-agent/observe/references/compare-iterate.md

@@ -0,0 +1,48 @@
+# Steps 8–10 — Re-Evaluate, Compare Versions, Iterate
+
+## Step 8 — Re-Evaluate
+
+Use **`evaluation_agent_batch_eval_create`** with the **same `evaluationId`** as the baseline run. This places both runs in the same eval group for comparison. Use the same local test dataset (from `datasets/`) and evaluators. Update `agentVersion` to the new version.
+
+Auto-poll for completion in a background terminal (same as [Step 2](evaluate-step.md)).
+
+## Step 9 — Compare Versions
+
+> **Critical:** `displayName` is **required** in the `insightRequest`. Omitting it causes a BadRequest error. `state` must be `"NotStarted"`.
+
+### Required Parameters for `evaluation_comparison_create`
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `insightRequest.displayName` | ✅ | Human-readable name. **Omitting causes BadRequest.** |
+| `insightRequest.state` | ✅ | Must be `"NotStarted"` |
+| `insightRequest.request.evalId` | ✅ | Eval group ID containing both runs |
+| `insightRequest.request.baselineRunId` | ✅ | Run ID of the baseline |
+| `insightRequest.request.treatmentRunIds` | ✅ | Array of treatment run IDs |
+
+Use **`evaluation_comparison_create`** with a nested `insightRequest`:
+
+```json
+{
+  "insightRequest": {
+    "displayName": "V1 vs V2 Comparison",
+    "state": "NotStarted",
+    "request": {
+      "type": "EvaluationComparison",
+      "evalId": "<eval-group-id>",
+      "baselineRunId": "<baseline-run-id>",
+      "treatmentRunIds": ["<new-run-id>"]
+    }
+  }
+}
+```
+
+> **Important:** Both runs must be in the **same eval group** (same `evaluationId` in Steps 2 and 8).
+
+Then use **`evaluation_comparison_get`** (with the returned `insightId`) to retrieve comparison results. Present a summary showing which version performed better per evaluator, and recommend which version to keep.
+
+## Step 10 — Iterate or Finish
+
+If more categories remain in the prioritized action table (from [Step 4](analyze-results.md)), loop back to **Step 5** (dive into next category) → **Step 6** (optimize) → **Step 7** (deploy) → **Step 8** (re-evaluate) → **Step 9** (compare).
+
+Otherwise, confirm the final agent version with the user, then prompt for [CI/CD evals & monitoring](cicd-monitoring.md).

package/.agents/skills/microsoft-foundry/foundry-agent/observe/references/deploy-and-setup.md

@@ -0,0 +1,76 @@
+# Step 1 — Deploy Agent & Setup Evaluators
+
+## Deploy
+
+For deployment, follow the [deploy skill](../../deploy/deploy.md). It handles project detection, Dockerfile generation, ACR build, agent creation, and container startup.
+
+After deployment completes, persist these variables to `.env` for future conversations:
+
+| Variable | Purpose |
+|----------|---------|
+| `AZURE_AI_PROJECT_ENDPOINT` | Foundry project endpoint |
+| `AZURE_AI_AGENT_NAME` | Deployed agent name |
+| `AZURE_AI_AGENT_VERSION` | Current agent version |
+| `AZURE_CONTAINER_REGISTRY` | ACR resource (hosted agents) |
+
+## Auto-Create Evaluators & Dataset
+
+> **This step is fully automatic.** After deployment, immediately prepare evaluators and a local test dataset without waiting for the user to request it.
+
+### 1. Read Agent Instructions
+
+Use **`agent_get`** (or local `agent.yaml`) to understand the agent's purpose and capabilities.
+
+### 2. Select Evaluators
+
+Combine **built-in, custom, and safety evaluators**:
+
+| Category | Evaluators |
+|----------|-----------|
+| **Quality (built-in)** | intent_resolution, task_adherence, coherence, fluency, relevance |
+| **Safety (include ≥2)** | violence, self_harm, hate_unfairness, sexual, indirect_attack |
+| **Custom (create 1–2)** | Domain-specific via `evaluator_catalog_create` (see below) |
+
+### 3. Create Custom Evaluators
+
+Use **`evaluator_catalog_create`** with:
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `projectEndpoint` | ✅ | Azure AI Project endpoint |
+| `name` | ✅ | e.g., `domain_accuracy`, `citation_quality` |
+| `category` | ✅ | `quality`, `safety`, or `agents` |
+| `scoringType` | ✅ | `ordinal`, `continuous`, or `boolean` |
+| `promptText` | ✅* | Template with `{{query}}`, `{{response}}` placeholders |
+| `minScore` / `maxScore` | | Default: 1 / 5 |
+| `passThreshold` | | Scores ≥ this value pass |
+
+> **LLM-judge tip:** Include in the evaluator prompt: *"Do NOT penalize the response for mentioning dates or events beyond your training cutoff. The agent has real-time access."*
+
+### 4. Identify LLM-Judge Deployment
+
+Use **`model_deployment_get`** to find a suitable model (e.g., `gpt-4o`) for quality evaluators.
+
+### 5. Generate Local Test Dataset
+
+Use the identified LLM deployment to generate realistic test queries based on the agent's instructions and tool capabilities. Save to `datasets/<agent-name>-test.jsonl` with each line containing at minimum a `query` field (optionally `context`, `ground_truth`).
+
+### 6. Persist Artifacts
+
+```
+evaluators/        # custom evaluator definitions
+  <name>.yaml      # prompt text, scoring type, thresholds
+datasets/          # locally generated input datasets
+  *.jsonl          # test queries
+results/           # evaluation run outputs (populated later)
+  <eval-id>/
+    <run-id>.json
+```
+
+Save evaluator definitions to `evaluators/<name>.yaml` and test data to `datasets/*.jsonl`.
+
+### 7. Prompt User
+
+*"Your agent is deployed and running. Evaluators and a local test dataset have been auto-configured. Would you like to run an evaluation to identify optimization opportunities?"*
+
+If yes → proceed to [Step 2: Evaluate](evaluate-step.md). If no → stop.

package/.agents/skills/microsoft-foundry/foundry-agent/observe/references/evaluate-step.md

@@ -0,0 +1,51 @@
+# Step 2 — Create Batch Evaluation
+
+## Prerequisites
+
+- Agent deployed and running
+- Evaluators configured (from [Step 1](deploy-and-setup.md) or `evaluators/` folder)
+- Local test dataset available (from `datasets/` or will use synthetic data)
+
+## Run Evaluation
+
+Use **`evaluation_agent_batch_eval_create`** to run evaluators against the agent.
+
+### Required Parameters
+
+| Parameter | Description |
+|-----------|-------------|
+| `projectEndpoint` | Azure AI Project endpoint |
+| `agentName` | Agent name |
+| `agentVersion` | Agent version (string, e.g. `"1"`) |
+| `evaluatorNames` | Array of evaluator names (NOT `evaluators`) |
+
+### Test Data Options
+
+**Preferred — local dataset:** Read JSONL from `datasets/` and pass via `inputData` (array of objects with `query` and optionally `context`, `ground_truth`). Provides reproducibility and version control.
+
+**Fallback — server-side synthetic data:** Set `generateSyntheticData=true` AND provide `generationModelDeploymentName`. Optionally set `samplesCount` (default 50) and `generationPrompt` with the agent's instructions.
+
+### Additional Parameters
+
+| Parameter | When Needed |
+|-----------|-------------|
+| `deploymentName` | Required for quality evaluators (the LLM-judge model) |
+| `evaluationId` | Pass existing eval group ID to group runs for comparison |
+| `evaluationName` | Name for a new evaluation group |
+
+> **Important:** Use `evaluationId` (NOT `evalId`) to group runs.
+
+## Auto-Poll for Completion
+
+Immediately after creating the run, poll **`evaluation_get`** in a **background terminal** until completion. Use `evalId` + `isRequestForRuns=true`. The run ID parameter is `evalRunId` (NOT `runId`).
+
+Only surface the final result when status reaches `completed`, `failed`, or `cancelled`.
+
+## Next Steps
+
+When evaluation completes → proceed to [Step 3: Analyze Results](analyze-results.md).
+
+## Reference
+
+- [Azure AI Foundry Cloud Evaluation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation)
+- [Built-in Evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/evaluate-sdk#built-in-evaluators)

package/.agents/skills/microsoft-foundry/foundry-agent/observe/references/optimize-deploy.md

@@ -0,0 +1,30 @@
+# Steps 6–7 — Optimize Prompt & Deploy New Version
+
+## Step 6 — Optimize Prompt
+
+Use **`prompt_optimize`** with:
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `developerMessage` | ✅ | Agent's current system prompt / instructions |
+| `deploymentName` | ✅ | Model for optimization (e.g., `gpt-4o-mini`) |
+| `projectEndpoint` or `foundryAccountResourceId` | ✅ | At least one required |
+| `requestedChanges` | | Concise improvement suggestions from cluster analysis |
+
+**Example `requestedChanges`:** *"Be more specific when answering geography questions"*, *"Always cite sources when providing factual claims"*
+
+> Use the optimized prompt returned by the tool. Do NOT manually rewrite.
+
+## Step 7 — Deploy New Version
+
+> **Always confirm before deploying.** Show the user a diff or summary of prompt changes and wait for explicit sign-off.
+
+After approval:
+
+1. Use **`agent_update`** to create a new agent version with the optimized prompt
+2. Start the container with **`agent_container_control`** (action: `start`)
+3. Poll **`agent_container_status_get`** in a **background terminal** until status is `Running`
+
+## Next Steps
+
+When the new version is running → proceed to [Step 8: Re-Evaluate](compare-iterate.md).

package/.agents/skills/microsoft-foundry/foundry-agent/trace/references/analyze-failures.md

@@ -0,0 +1,83 @@
+# Analyze Failures — Find and Cluster Failing Traces
+
+Identify failing agent traces, group them by root cause, and produce a prioritized action table.
+
+## Step 1 — Find Failing Traces
+
+```kql
+dependencies
+| where timestamp > ago(24h)
+| where success == false or toint(resultCode) >= 400
+| extend
+    operation = tostring(customDimensions["gen_ai.operation.name"]),
+    errorType = tostring(customDimensions["error.type"]),
+    model = tostring(customDimensions["gen_ai.request.model"]),
+    agentName = tostring(customDimensions["gen_ai.agent.name"]),
+    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
+| project timestamp, name, duration, resultCode, errorType, operation, model,
+    agentName, conversationId, operation_Id, id
+| order by timestamp desc
+| take 100
+```
+
+## Step 2 — Cluster by Error Type
+
+```kql
+dependencies
+| where timestamp > ago(24h)
+| where success == false or toint(resultCode) >= 400
+| extend
+    errorType = tostring(customDimensions["error.type"]),
+    operation = tostring(customDimensions["gen_ai.operation.name"])
+| summarize
+    count = count(),
+    firstSeen = min(timestamp),
+    lastSeen = max(timestamp),
+    avgDuration = avg(duration),
+    sampleOperationId = take_any(operation_Id)
+  by errorType, operation, resultCode
+| order by count desc
+```
+
+## Step 3 — Prioritized Action Table
+
+Present results as:
+
+| Priority | Error Type | Operation | Count | Result Code | Suggested Action |
+|----------|-----------|-----------|-------|-------------|-----------------|
+| P0 | timeout | invoke_agent | 15 | 504 | Check agent container health, increase timeout |
+| P1 | rate_limited | chat | 8 | 429 | Check quota, add retry logic |
+| P2 | content_filter | chat | 5 | 400 | Review prompt for policy violations |
+| P3 | tool_error | execute_tool | 3 | 500 | Check tool implementation and permissions |
+
+**Prioritization:** P0 = highest count or most severe (5xx), then by count × recency.
+
+## Step 4 — Drill Into Specific Failure
+
+When the user selects a cluster, show individual failing traces:
+
+```kql
+dependencies
+| where timestamp > ago(24h)
+| where success == false
+| where customDimensions["error.type"] == "<selected_error_type>"
+| where customDimensions["gen_ai.operation.name"] == "<selected_operation>"
+| project timestamp, name, duration, resultCode,
+    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
+    responseId = tostring(customDimensions["gen_ai.response.id"]),
+    operation_Id
+| order by timestamp desc
+| take 20
+```
+
+Also check `exceptions` table for stack traces:
+
+```kql
+exceptions
+| where timestamp > ago(24h)
+| where operation_Id in ("<operation_id_1>", "<operation_id_2>")
+| project timestamp, type, message, outerMessage, details, operation_Id
+| order by timestamp desc
+```
+
+Offer to view the full conversation for any trace via [Conversation Detail](conversation-detail.md).

package/.agents/skills/microsoft-foundry/foundry-agent/trace/references/analyze-latency.md

@@ -0,0 +1,90 @@
+# Analyze Latency — Find and Diagnose Slow Traces
+
+Identify slow agent traces, find bottleneck spans, and correlate with token usage.
+
+## Step 1 — Find Slow Conversations
+
+```kql
+dependencies
+| where timestamp > ago(24h)
+| where customDimensions["gen_ai.operation.name"] == "invoke_agent"
+| project timestamp, duration, success,
+    agentName = tostring(customDimensions["gen_ai.agent.name"]),
+    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
+    operation_Id
+| summarize
+    totalDuration = sum(duration),
+    spanCount = count(),
+    hasErrors = countif(success == false) > 0
+  by conversationId, operation_Id
+| where totalDuration > 5000
+| order by totalDuration desc
+| take 50
+```
+
+> **Default threshold:** 5 seconds. Ask the user for their latency threshold if not specified.
+
+## Step 2 — Latency Distribution (P50/P95/P99)
+
+```kql
+dependencies
+| where timestamp > ago(24h)
+| where customDimensions["gen_ai.operation.name"] in ("chat", "invoke_agent")
+| summarize
+    p50 = percentile(duration, 50),
+    p95 = percentile(duration, 95),
+    p99 = percentile(duration, 99),
+    avg = avg(duration),
+    count = count()
+  by operation = tostring(customDimensions["gen_ai.operation.name"]),
+     model = tostring(customDimensions["gen_ai.request.model"])
+| order by p95 desc
+```
+
+Present as:
+
+| Operation | Model | P50 (ms) | P95 (ms) | P99 (ms) | Avg (ms) | Count |
+|-----------|-------|---------|---------|---------|---------|-------|
+
+## Step 3 — Bottleneck Breakdown
+
+For a specific slow conversation, break down time spent per span type:
+
+```kql
+dependencies
+| where operation_Id == "<operation_id>"
+| extend operation = tostring(customDimensions["gen_ai.operation.name"])
+| summarize
+    totalDuration = sum(duration),
+    spanCount = count(),
+    avgDuration = avg(duration)
+  by operation, name
+| order by totalDuration desc
+```
+
+Common bottleneck patterns:
+- **`chat` spans dominate** → LLM inference is slow (consider smaller model or caching)
+- **`execute_tool` spans dominate** → Tool execution is slow (optimize tool implementation)
+- **`invoke_agent` has long gaps** → Orchestration overhead (check agent framework)
+
+## Step 4 — Token Usage vs Latency Correlation
+
+```kql
+dependencies
+| where timestamp > ago(24h)
+| where customDimensions["gen_ai.operation.name"] == "chat"
+| extend
+    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
+    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
+| where isnotempty(inputTokens)
+| project duration, inputTokens, outputTokens,
+    model = tostring(customDimensions["gen_ai.request.model"]),
+    operation_Id
+| order by duration desc
+| take 100
+```
+
+High token counts often correlate with high latency. If confirmed, suggest:
+- Reduce system prompt length
+- Limit conversation history window
+- Use a faster model for simpler queries

package/.agents/skills/microsoft-foundry/foundry-agent/trace/references/conversation-detail.md

@@ -0,0 +1,84 @@
+# Conversation Detail — Reconstruct Full Span Tree
+
+Reconstruct the complete span tree for a single conversation to see exactly what happened: every LLM call, tool execution, and agent invocation with timing, tokens, and errors.
+
+## Step 1 — Fetch All Spans for a Conversation
+
+Use `operation_Id` (trace ID) to get all spans in a single request:
+
+```kql
+dependencies
+| where operation_Id == "<operation_id>"
+| project timestamp, name, duration, resultCode, success,
+    spanId = id,
+    parentSpanId = operation_ParentId,
+    operation = tostring(customDimensions["gen_ai.operation.name"]),
+    model = tostring(customDimensions["gen_ai.request.model"]),
+    responseModel = tostring(customDimensions["gen_ai.response.model"]),
+    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
+    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"]),
+    responseId = tostring(customDimensions["gen_ai.response.id"]),
+    finishReason = tostring(customDimensions["gen_ai.response.finish_reasons"]),
+    errorType = tostring(customDimensions["error.type"]),
+    toolName = tostring(customDimensions["gen_ai.tool.name"]),
+    toolCallId = tostring(customDimensions["gen_ai.tool.call.id"])
+| order by timestamp asc
+```
+
+Also fetch the parent request:
+
+```kql
+requests
+| where operation_Id == "<operation_id>"
+| project timestamp, name, duration, resultCode, success, id, operation_ParentId
+```
+
+## Step 2 — Build Span Tree
+
+Use `spanId` and `parentSpanId` to reconstruct the hierarchy:
+
+```
+invoke_agent (root) ─── 4200ms
+├── chat (LLM call #1) ─── 1800ms, gpt-4o, 450→120 tokens
+│   └── [output: "Let me check the weather..."]
+├── execute_tool (get_weather) [tool: remote_functions.weather_api] ─── 200ms
+│   └── [result: "rainy, 57°F"]
+├── chat (LLM call #2) ─── 1500ms, gpt-4o, 620→85 tokens
+│   └── [output: "The weather in Paris is rainy, 57°F"]
+└── [total: 450+620=1070 input, 120+85=205 output tokens]
+```
+
+Present as an indented tree with:
+- **Operation type** and name
+- **Duration** (highlight if > P95 for that operation type)
+- **Model** and token counts (for chat operations)
+- **Error type** and result code (if failed, highlight in red)
+- **Finish reason** (stop, length, content_filter, tool_calls)
+
+## Step 3 — Check for Associated Events
+
+GenAI events may contain input/output message content:
+
+```kql
+traces
+| where operation_Id == "<operation_id>"
+| where message contains "gen_ai"
+| project timestamp, message, customDimensions
+| order by timestamp asc
+```
+
+## Step 4 — Check for Exceptions
+
+```kql
+exceptions
+| where operation_Id == "<operation_id>"
+| project timestamp, type, message, outerMessage,
+    details = parse_json(details)
+| order by timestamp asc
+```
+
+Present exceptions inline in the span tree at their position in the timeline.
+
+## Step 5 — Fetch Evaluation Results
+
+See [Eval Correlation](eval-correlation.md) for the full workflow to look up evaluation scores by response ID or conversation ID. Use `gen_ai.response.id` values from Step 1 spans to correlate.

package/.agents/skills/microsoft-foundry/foundry-agent/trace/references/eval-correlation.md

@@ -0,0 +1,57 @@
+# Eval Correlation — Find Evaluation Results by Response or Conversation ID
+
+Look up evaluation scores for a specific agent response using App Insights.
+
+> **IMPORTANT:** The Foundry evaluation API does NOT support querying by response ID or conversation ID. App Insights `customEvents` is the ONLY way to correlate eval scores to specific responses. Always use this KQL approach when the user asks for eval results for a specific response or conversation.
+
+## Prerequisites
+
+- App Insights resource resolved (see [trace.md](../trace.md) Before Starting)
+- A response ID (`gen_ai.response.id`) or conversation ID (`gen_ai.conversation.id`) from a previous trace query
+
+## Search by Response ID
+
+```kql
+customEvents
+| where timestamp > ago(30d)
+| where name == "gen_ai.evaluation.result"
+| where customDimensions["gen_ai.response.id"] == "<response_id>"
+| extend
+    evalName = tostring(customDimensions["gen_ai.evaluation.name"]),
+    score = todouble(customDimensions["gen_ai.evaluation.score.value"]),
+    label = tostring(customDimensions["gen_ai.evaluation.score.label"]),
+    explanation = tostring(customDimensions["gen_ai.evaluation.explanation"]),
+    responseId = tostring(customDimensions["gen_ai.response.id"]),
+    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
+| project timestamp, evalName, score, label, explanation, responseId, conversationId
+| order by evalName asc
+```
+
+## Search by Conversation ID
+
+```kql
+customEvents
+| where timestamp > ago(30d)
+| where name == "gen_ai.evaluation.result"
+| where customDimensions["gen_ai.conversation.id"] == "<conversation_id>"
+| extend
+    evalName = tostring(customDimensions["gen_ai.evaluation.name"]),
+    score = todouble(customDimensions["gen_ai.evaluation.score.value"]),
+    label = tostring(customDimensions["gen_ai.evaluation.score.label"]),
+    explanation = tostring(customDimensions["gen_ai.evaluation.explanation"]),
+    responseId = tostring(customDimensions["gen_ai.response.id"])
+| project timestamp, evalName, score, label, explanation, responseId
+| order by responseId asc, evalName asc
+```
+
+## Present Results
+
+Show eval scores as a table:
+
+| Evaluator | Score | Label | Explanation |
+|-----------|-------|-------|-------------|
+| coherence | 5.0 | pass | Response is well-structured... |
+| fluency | 4.0 | pass | Natural language flow... |
+| relevance | 2.0 | fail | Response doesn't address... |
+
+When showing alongside a span tree (see [Conversation Detail](conversation-detail.md)), attach eval scores to the span whose `gen_ai.response.id` matches.