@botpress/adk-cli 1.19.0-beta.3 → 1.19.0-beta.5

package/dist/conversation-analysis-8ka4nrm4.md

@@ -0,0 +1,420 @@
+# Conversation Analysis
+
+How to summarize and explain full conversations using the `adk conversations` command. A conversation spans multiple turns (traces), so this is a higher-level view than single-trace analysis.
+
+For single-trace summaries, see `trace-summarization.md`. This document covers multi-turn conversation-level analysis.
+
+---
+
+## CLI Commands
+
+### List recent conversations
+
+```bash
+adk conversations --format json
+adk conversations list limit=5 --format json
+```
+
+**Output** (JSON): Array of objects, each with:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `conversationId` | string | Unique conversation identifier |
+| `firstSeen` | string | ISO timestamp of the first recorded turn |
+| `lastSeen` | string | ISO timestamp of the most recent turn |
+| `traceCount` | number | Total number of turns in the conversation |
+| `integration` | string | Which integration the conversation came from (e.g., `slack`, `webchat`) |
+| `channel` | string | Which channel within the integration (e.g., `dm`, `thread`) |
+| `hasErrors` | boolean | Whether any turn in the conversation had errors |
+
+### Show a conversation's timeline
+
+```bash
+# Standard view
+adk conversations show <conversation-id> --format json
+
+# With LLM reasoning included
+adk conversations show <conversation-id> --include-llm --format json
+```
+
+**Output** (JSON): Object with:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `conversationId` | string | The conversation ID |
+| `turnCount` | number | Total number of turns |
+| `turns` | array | Ordered list of turn objects |
+
+Each turn object contains:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `traceId` | string | Trace ID for this turn (use for drill-down) |
+| `startedAt` | string | ISO timestamp when the turn started |
+| `duration` | string | How long the turn took (e.g., `"2.3s"`) |
+| `status` | string | `"ok"` or `"error"` |
+| `trigger` | string | What started this turn (e.g., user message, event, workflow) |
+| `toolCalls` | array | Tools invoked during this turn |
+| `errors` | array | Error strings, if any |
+| `llmContent` | array | LLM reasoning data (only with `--include-llm`) |
+
+Each `toolCalls` entry has: `name`, `status`, `duration`, `error?`
+
+Each `llmContent` entry has: `name`, `data`
+
+---
+
+## Two Modes of Analysis
+
+### 1. Summarize a Conversation
+
+**Purpose:** Give a concise overview of what happened -- a paragraph or two covering the full conversation arc.
+
+**When to use:** The developer asks "What happened in this conversation?", "Summarize conversation X", or wants a quick picture before diving deeper.
+
+**Data source:**
+
+```bash
+adk conversations show <conversation-id> --format json
+```
+
+No need for `--include-llm` -- the standard view has enough for a summary.
+
+**What to cover:**
+
+1. **Who and where** -- integration, channel, how many turns, time span
+2. **The flow** -- what the user asked for, what the agent did, how the conversation progressed
+3. **Tools used** -- which tools were called and how often
+4. **Outcome** -- did the conversation succeed? Were there errors? How did it end?
+5. **Red flags** -- only if present: errors, long durations, many turns for a simple task
+
+**How to write it:**
+
+Write a natural-language paragraph, not a list or template. Synthesize the turn data into a narrative.
+
+#### Example
+
+Given this conversation data:
+
+```json
+{
+  "conversationId": "conv_abc123",
+  "turnCount": 4,
+  "turns": [
+    {
+      "traceId": "tr_001",
+      "startedAt": "2026-04-21T14:00:00.000Z",
+      "duration": "1.8s",
+      "status": "ok",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [],
+      "errors": []
+    },
+    {
+      "traceId": "tr_002",
+      "startedAt": "2026-04-21T14:00:05.000Z",
+      "duration": "3.2s",
+      "status": "ok",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [
+        { "name": "lookupOrder", "status": "ok", "duration": "0.9s" }
+      ],
+      "errors": []
+    },
+    {
+      "traceId": "tr_003",
+      "startedAt": "2026-04-21T14:00:15.000Z",
+      "duration": "4.1s",
+      "status": "error",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [
+        { "name": "cancelOrder", "status": "error", "duration": "1.2s", "error": "Cannot cancel shipped order" }
+      ],
+      "errors": ["Cannot cancel shipped order"]
+    },
+    {
+      "traceId": "tr_004",
+      "startedAt": "2026-04-21T14:00:25.000Z",
+      "duration": "2.0s",
+      "status": "ok",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [],
+      "errors": []
+    }
+  ]
+}
+```
+
+**Good summary:**
+
+> A 4-turn Slack DM conversation over about 30 seconds. The user started with a greeting, then asked about an order -- the agent looked it up with `lookupOrder`. The user then asked to cancel the order, but `cancelOrder` failed because the order was already shipped. The agent recovered in the final turn without needing more tools, likely explaining the situation to the user. One error occurred (`cancelOrder` failure) but it was handled gracefully.
+
+**Bad summary:**
+
+> Turn 1: traceId tr_001, status ok, no tools. Turn 2: traceId tr_002, called lookupOrder, status ok. Turn 3: traceId tr_003, called cancelOrder, status error, error "Cannot cancel shipped order". Turn 4: traceId tr_004, status ok, no tools.
+
+---
+
+### 2. Explain a Conversation
+
+**Purpose:** Provide a detailed, step-by-step breakdown where the developer understands every decision the agent made.
+
+**When to use:** The developer asks "Walk me through this conversation", "Explain what happened", "Why did the bot respond that way?", or is investigating a specific behavior.
+
+**Data source:**
+
+```bash
+adk conversations show <conversation-id> --include-llm --format json
+```
+
+**Always use `--include-llm`** for explanations. Without it, you lose the LLM's reasoning, which is essential for explaining *why* the agent did what it did.
+
+**What to cover for each turn:**
+
+1. **The trigger** -- what started this turn (user message, event, workflow continuation)
+2. **LLM reasoning** -- what the agent decided to do and why (from `llmContent`)
+3. **Tool calls** -- what was called, with what inputs, what happened
+4. **Errors** -- what went wrong and how the agent responded
+5. **The response** -- what the agent said back (infer from context if not explicit)
+6. **Transitions** -- how one turn's outcome led to the next
+
+**How to write it:**
+
+Walk through turns chronologically. For each turn, narrate what happened as a coherent paragraph. Use tool names in backticks, quote key values, and explain the agent's reasoning when `llmContent` provides it.
+
+#### Example
+
+Given the same conversation with `--include-llm`:
+
+```json
+{
+  "conversationId": "conv_abc123",
+  "turnCount": 4,
+  "turns": [
+    {
+      "traceId": "tr_001",
+      "startedAt": "2026-04-21T14:00:00.000Z",
+      "duration": "1.8s",
+      "status": "ok",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [],
+      "errors": [],
+      "llmContent": [
+        { "name": "cognitive.iteration", "data": "The user said 'Hi, I need help with an order'. This is a greeting with intent to discuss an order. I'll respond and ask for the order number." }
+      ]
+    },
+    {
+      "traceId": "tr_002",
+      "startedAt": "2026-04-21T14:00:05.000Z",
+      "duration": "3.2s",
+      "status": "ok",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [
+        { "name": "lookupOrder", "status": "ok", "duration": "0.9s" }
+      ],
+      "errors": [],
+      "llmContent": [
+        { "name": "cognitive.iteration", "data": "The user provided order number #4521. I should look this up to get the details before responding." }
+      ]
+    },
+    {
+      "traceId": "tr_003",
+      "startedAt": "2026-04-21T14:00:15.000Z",
+      "duration": "4.1s",
+      "status": "error",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [
+        { "name": "cancelOrder", "status": "error", "duration": "1.2s", "error": "Cannot cancel shipped order" }
+      ],
+      "errors": ["Cannot cancel shipped order"],
+      "llmContent": [
+        { "name": "cognitive.iteration", "data": "The user wants to cancel order #4521. I'll use cancelOrder." },
+        { "name": "cognitive.iteration", "data": "cancelOrder failed -- the order is already shipped. I need to tell the user and suggest a return instead." }
+      ]
+    },
+    {
+      "traceId": "tr_004",
+      "startedAt": "2026-04-21T14:00:25.000Z",
+      "duration": "2.0s",
+      "status": "ok",
+      "trigger": "User message via slack/dm",
+      "toolCalls": [],
+      "errors": [],
+      "llmContent": [
+        { "name": "cognitive.iteration", "data": "The user asked how to start a return. I know the process from the instructions -- they need to go to the returns portal. No tool call needed." }
+      ]
+    }
+  ]
+}
+```
+
+**Good explanation:**
+
+> **Turn 1** (tr_001, 1.8s): The user opened the conversation in Slack DM with "Hi, I need help with an order." The agent recognized this as a greeting with order-related intent and responded by asking for the order number. No tools were needed.
+>
+> **Turn 2** (tr_002, 3.2s): The user provided order #4521. The agent called `lookupOrder` to retrieve the order details (completed in 0.9s). The lookup succeeded and the agent relayed the order information to the user.
+>
+> **Turn 3** (tr_003, 4.1s): The user asked to cancel the order. The agent called `cancelOrder`, which failed with "Cannot cancel shipped order" -- the order had already shipped. The agent's reasoning shows it recognized the failure and decided to suggest a return as an alternative. The turn is marked as `error` because of the tool failure, but the agent handled it gracefully.
+>
+> **Turn 4** (tr_004, 2.0s): The user asked how to start a return. The agent answered from its instructions without needing any tool calls, directing the user to the returns portal. The conversation ended successfully.
+>
+> **Overall:** The conversation followed a natural support flow -- greeting, order lookup, failed cancellation handled with a fallback suggestion, and resolution. The `cancelOrder` error is expected behavior for a shipped order, not a bug.
+
+---
+
+## Picking Which Conversation to Analyze
+
+When the developer doesn't specify a conversation ID, help them find the right one.
+
+### Step 1: List conversations
+
+```bash
+adk conversations --format json
+```
+
+### Step 2: Narrow down
+
+Use the list output to identify the conversation. Key signals:
+
+| Signal | How to Use |
+|--------|-----------|
+| `lastSeen` | Most recent conversation is usually the one they're asking about |
+| `hasErrors` | If they're debugging a failure, look for `hasErrors: true` |
+| `integration` / `channel` | If they mention "the Slack conversation" or "the webchat issue", match on these |
+| `traceCount` | High turn counts may indicate stuck loops or long interactions |
+
+### Step 3: Show the conversation
+
+```bash
+adk conversations show <conversation-id> --format json
+```
+
+If the developer needs more detail, add `--include-llm`.
+
+---
+
+## Correlating with Trace Data
+
+Each turn in the conversation has a `traceId`. This is the bridge between conversation-level and trace-level analysis.
+
+**When to drill into a trace:**
+
+- A turn has `status: "error"` and you need the full span tree to understand why
+- A tool call failed and you need the stack trace or detailed error
+- The LLM reasoning (from `llmContent`) is surprising and you want to see the full `think` spans
+- You need to see the exact inputs/outputs of a tool call
+
+**How to drill down:**
+
+```bash
+# Get the full trace for a specific turn
+adk traces trace=<trace-id> --include-llm --format json
+```
+
+Then use the trace-reading techniques from `trace-summarization.md` to analyze the individual trace.
+
+**Flow:**
+
+```
+adk conversations list          →  find the conversation
+adk conversations show <id>     →  see the timeline
+adk traces trace=<trace-id>     →  drill into a specific turn
+```
+
+---
+
+## Common Patterns to Highlight
+
+When analyzing conversations, watch for these patterns and call them out.
+
+### Long conversations
+
+A conversation with many turns (10+) for what should be a simple task.
+
+**What to say:** Flag the turn count, identify where the conversation got stuck or went off track, and suggest whether the issue is in the agent's instructions, tool design, or workflow logic.
+
+### Error recovery
+
+The agent encounters an error but recovers in a subsequent turn.
+
+**What to say:** Note that the agent handled the error gracefully, explain the recovery path, and confirm whether the behavior is correct or accidental.
+
+### Tool call chains
+
+Multiple tools called in sequence within a single turn, or across turns, to accomplish a multi-step task.
+
+**What to say:** Describe the chain, note whether each step depended on the previous one's output, and flag any unnecessary calls.
+
+### Stuck loops
+
+The agent calls the same tool repeatedly across multiple turns or within a single turn.
+
+**What to say:** Identify the repeating pattern, explain why the agent is stuck (from `llmContent` if available), and suggest fixes (improve tool output, add `onBeforeTool` guard, set `maxIterations`). See `llm-debugging.md` for more on looping.
+
+### Silent failures
+
+A turn completes with `status: "ok"` but the agent didn't actually accomplish the task (e.g., returned a generic response without calling a tool).
+
+**What to say:** Flag the gap between expected and actual behavior, and suggest investigating the LLM reasoning with `--include-llm`.
+
+### Integration-specific patterns
+
+Different integrations have different conversation patterns -- Slack threads vs webchat sessions vs API calls.
+
+**What to say:** Note the integration and channel, and flag anything unusual for that context (e.g., a webchat conversation with 50 turns might indicate the user is stuck in a loop, while a Slack thread with 50 turns might be a busy channel).
+
+---
+
+## Patterns
+
+✅ **Summarize first, explain on request**
+
+```
+Developer: "What happened in conv_abc123?"
+→ Start with a summary (no --include-llm needed)
+→ Only drill into explanation if they ask for more detail
+```
+
+✅ **Use the conversation timeline as a map**
+
+```
+Good: "The conversation had 4 turns. Turn 3 is where the error occurred. Let me drill into that trace."
+Bad: Immediately fetching all traces for all turns and dumping everything.
+```
+
+✅ **Name the integration and channel**
+
+```
+Good: "A 4-turn Slack DM conversation..."
+Bad: "A conversation with 4 turns..."
+```
+
+✅ **Connect turns to each other**
+
+```
+Good: "The user asked about the order in turn 2, then tried to cancel it in turn 3 -- the cancellation failed because the lookup in turn 2 showed the order was already shipped."
+Bad: "Turn 2: lookupOrder called. Turn 3: cancelOrder failed."
+```
+
+✅ **Suggest drill-down when appropriate**
+
+```
+Good: "Turn 3 failed with a cancelOrder error. To see the full stack trace, run: adk traces trace=tr_003 --include-llm --format json"
+Bad: Automatically fetching and dumping the full trace without being asked.
+```
+
+❌ **Don't dump raw JSON in the analysis**
+
+The developer already has the JSON. Your job is to synthesize it into understanding.
+
+❌ **Don't speculate about what the user said**
+
+The conversation timeline shows what the agent did, not the exact user messages. Use `trigger` and `llmContent` to infer context, but be clear when you're inferring vs. when you have direct evidence.
+
+❌ **Don't re-explain tool behavior the developer already knows**
+
+If the developer built the tools, they know what `lookupOrder` does. Focus on *what happened* and *why*, not what the tool is for.
+
+❌ **Don't ignore the overall arc**
+
+Even in a detailed explanation, start and end with the big picture. A list of turns without a conclusion is incomplete.