@botpress/adk-cli 1.19.0-beta.3 → 1.19.0-beta.5

package/dist/trace-summarization-hdyx1c2e.md

@@ -0,0 +1,313 @@
+# Trace Summarization
+
+How to fetch a trace, walk its span tree, and produce a free-form natural-language summary of what happened. The goal is a synthesized narrative, not a raw dump of span data.
+
+---
+
+## Fetching Trace Data
+
+Use the `adk traces` CLI with filters to get the raw data.
+
+```bash
+# Full trace with LLM reasoning included
+adk traces trace=<trace-id> --include-llm --format json
+
+# All traces for a conversation
+adk traces conversation=<id> --include-llm --format json
+
+# Recent traces (start here if you don't have an ID)
+adk traces --include-llm --format json
+```
+
+**Always pass `--include-llm`** when summarizing. Without it, `think` spans are omitted and you lose the model's reasoning, which is essential for explaining *why* the agent did what it did.
+
+**Always pass `--format json`** for structured, parseable output.
+
+---
+
+## Reading the Span Tree
+
+Traces contain a flat list of spans, but spans form a parent-child hierarchy via `parentSpanId`. To summarize effectively, reconstruct the tree.
+
+### Span hierarchy (typical)
+
+```
+handler.conversation (or handler.workflow, handler.action)
+  └─ autonomous.execution
+       └─ autonomous.iteration (one per LLM turn)
+            ├─ cognitive.request (the LLM call)
+            ├─ think (model reasoning)
+            ├─ tool_call (tool invocation)
+            │    └─ autonomous.tool (tool execution detail)
+            ├─ tool_call (another tool)
+            ├─ code_execution_exception (if something broke)
+            └─ end (turn complete)
+```
+
+### How to walk it
+
+1. **Find the root span** -- the span with no `parentSpanId`, or the handler span (`handler.conversation`, `handler.workflow`, `handler.action`). This tells you what triggered the trace.
+2. **Walk children in chronological order** by `startTime`. Each child span is a step in the agent's behavior.
+3. **Group by iteration** -- `autonomous.iteration` spans each represent one LLM turn. Within each iteration, look for the `think` -> `tool_call` -> `end` sequence.
+4. **Follow errors upward** -- if a `code_execution_exception` or failed `tool_call` appears, check its parent iteration and the subsequent iteration to see how the agent recovered (or didn't).
+
+### Key fields per span type
+
+| Span Type | Fields to Read | What They Tell You |
+|-----------|---------------|-------------------|
+| `handler.conversation` | `context.conversationId`, `context.userId` | What triggered the trace (incoming message) |
+| `handler.workflow` | `name`, `context` | Which workflow was invoked |
+| `handler.action` | `name`, `context` | Which action was called |
+| `autonomous.iteration` | `data.iterationIndex` | Which LLM turn this is (0-indexed) |
+| `cognitive.request` | `data.model`, `data.tokens` | Which model was used, token counts |
+| `think` | `data.content` | The LLM's reasoning -- why it chose an action |
+| `tool_call` | `data.tool_name`, `data.input`, `data.output`, `status`, `data.error` | What tool was called, with what, and what happened |
+| `code_execution_exception` | `data.message`, `data.stackTrace` | Runtime error in the LLMz sandbox |
+| `end` | `data.response` | What the agent said back to the user |
+
+---
+
+## Producing a Summary
+
+Do **not** use a fixed template. Write a natural-language narrative that reads like a concise incident report or activity log. Cover these aspects in whatever order makes the most sense for the specific trace:
+
+### What to cover
+
+1. **The trigger** -- What started this trace? An incoming user message? A workflow invocation? A scheduled trigger? Extract this from the root span type and context.
+
+2. **The LLM's plan** -- What did the model decide to do and why? Synthesize from `think` spans. Don't quote them verbatim unless a specific phrase is important -- paraphrase the reasoning.
+
+3. **Tool calls and their outcomes** -- Which tools were called, with what key inputs, and what happened? For successful calls, note what was returned if relevant. For failures, state the error clearly.
+
+4. **Errors and recovery** -- Were there exceptions? Did the agent retry or give up? What was the error message? If the agent recovered, explain how.
+
+5. **The outcome** -- What was the final result? What did the agent respond with? Did the conversation turn succeed or fail?
+
+6. **Performance** -- Only mention if relevant (e.g., unusually slow, many iterations, high token usage). Don't include timing details for normal traces.
+
+### Adapting depth to context
+
+The summary depth should match what the developer is asking:
+
+| Developer Question | Summary Depth |
+|-------------------|---------------|
+| "What happened?" / "Summarize this trace" | **Brief** -- 3-5 sentences covering trigger, actions taken, outcome |
+| "Why did it do X?" / "Why did the bot call tool Y?" | **Focused** -- Zoom into the relevant `think` span and the tool call, explain the reasoning chain |
+| "What went wrong?" / "Why did this fail?" | **Diagnostic** -- Focus on the error path: what led to the failure, what the error was, whether recovery was attempted |
+| "Walk me through the whole trace" | **Detailed** -- Full chronological narrative covering every iteration and tool call |
+
+---
+
+## Examples
+
+### Sample trace data (abbreviated)
+
+```json
+{
+  "traceId": "tr_01J5EXAMPLE",
+  "spans": [
+    {
+      "spanId": "sp_001",
+      "name": "handler.conversation",
+      "startTime": "2026-04-21T14:00:00.000Z",
+      "endTime": "2026-04-21T14:00:03.200Z",
+      "status": "ok",
+      "context": {
+        "conversationId": "conv_abc123",
+        "userId": "user_xyz"
+      }
+    },
+    {
+      "spanId": "sp_002",
+      "parentSpanId": "sp_001",
+      "name": "autonomous.execution",
+      "startTime": "2026-04-21T14:00:00.050Z",
+      "endTime": "2026-04-21T14:00:03.100Z",
+      "status": "ok"
+    },
+    {
+      "spanId": "sp_003",
+      "parentSpanId": "sp_002",
+      "name": "autonomous.iteration",
+      "startTime": "2026-04-21T14:00:00.100Z",
+      "endTime": "2026-04-21T14:00:02.800Z",
+      "status": "ok",
+      "data": { "iterationIndex": 0 }
+    },
+    {
+      "spanId": "sp_004",
+      "parentSpanId": "sp_003",
+      "name": "think",
+      "startTime": "2026-04-21T14:00:00.200Z",
+      "data": {
+        "content": "The user is asking for the status of order #4521. I should look this up using the lookupOrder tool with the order ID."
+      }
+    },
+    {
+      "spanId": "sp_005",
+      "parentSpanId": "sp_003",
+      "name": "tool_call",
+      "startTime": "2026-04-21T14:00:00.500Z",
+      "endTime": "2026-04-21T14:00:01.200Z",
+      "status": "ok",
+      "data": {
+        "tool_name": "lookupOrder",
+        "input": { "orderId": "4521" },
+        "output": { "status": "shipped", "trackingNumber": "1Z999AA10123456784", "estimatedDelivery": "2026-04-23" }
+      }
+    },
+    {
+      "spanId": "sp_006",
+      "parentSpanId": "sp_003",
+      "name": "end",
+      "startTime": "2026-04-21T14:00:02.800Z",
+      "data": {
+        "response": "Your order #4521 has been shipped! Tracking number: 1Z999AA10123456784. Estimated delivery: April 23rd."
+      }
+    }
+  ]
+}
+```
+
+### Brief summary ("What happened?")
+
+> The user asked about order #4521. The agent looked it up with `lookupOrder`, found it was shipped with tracking number 1Z999AA10123456784 and an estimated delivery of April 23rd, and relayed that information to the user. Completed in one iteration, ~3 seconds.
+
+### Sample trace with an error (abbreviated)
+
+```json
+{
+  "traceId": "tr_01J5FAILURE",
+  "spans": [
+    {
+      "spanId": "sp_010",
+      "name": "handler.conversation",
+      "startTime": "2026-04-21T15:00:00.000Z",
+      "endTime": "2026-04-21T15:00:08.500Z",
+      "status": "error",
+      "context": {
+        "conversationId": "conv_def456",
+        "userId": "user_xyz"
+      }
+    },
+    {
+      "spanId": "sp_011",
+      "parentSpanId": "sp_010",
+      "name": "autonomous.execution",
+      "startTime": "2026-04-21T15:00:00.050Z",
+      "endTime": "2026-04-21T15:00:08.400Z",
+      "status": "error"
+    },
+    {
+      "spanId": "sp_012",
+      "parentSpanId": "sp_011",
+      "name": "autonomous.iteration",
+      "startTime": "2026-04-21T15:00:00.100Z",
+      "endTime": "2026-04-21T15:00:04.000Z",
+      "status": "error",
+      "data": { "iterationIndex": 0 }
+    },
+    {
+      "spanId": "sp_013",
+      "parentSpanId": "sp_012",
+      "name": "think",
+      "startTime": "2026-04-21T15:00:00.200Z",
+      "data": {
+        "content": "The user wants to cancel order #7890. I'll use the cancelOrder tool."
+      }
+    },
+    {
+      "spanId": "sp_014",
+      "parentSpanId": "sp_012",
+      "name": "tool_call",
+      "startTime": "2026-04-21T15:00:00.500Z",
+      "endTime": "2026-04-21T15:00:02.000Z",
+      "status": "error",
+      "data": {
+        "tool_name": "cancelOrder",
+        "input": { "orderId": "7890" },
+        "error": "OrderService: Cannot cancel order in 'shipped' status"
+      }
+    },
+    {
+      "spanId": "sp_015",
+      "parentSpanId": "sp_011",
+      "name": "autonomous.iteration",
+      "startTime": "2026-04-21T15:00:04.100Z",
+      "endTime": "2026-04-21T15:00:08.000Z",
+      "status": "ok",
+      "data": { "iterationIndex": 1 }
+    },
+    {
+      "spanId": "sp_016",
+      "parentSpanId": "sp_015",
+      "name": "think",
+      "startTime": "2026-04-21T15:00:04.200Z",
+      "data": {
+        "content": "The cancelOrder tool failed because the order is already shipped. I should inform the user that shipped orders cannot be cancelled and suggest they initiate a return instead."
+      }
+    },
+    {
+      "spanId": "sp_017",
+      "parentSpanId": "sp_015",
+      "name": "end",
+      "startTime": "2026-04-21T15:00:07.900Z",
+      "data": {
+        "response": "I wasn't able to cancel order #7890 because it has already shipped. You can initiate a return once it arrives — would you like me to help with that?"
+      }
+    }
+  ]
+}
+```
+
+### Diagnostic summary ("What went wrong?")
+
+> The user asked to cancel order #7890. The agent called `cancelOrder` but it failed with "Cannot cancel order in 'shipped' status" -- the order was already shipped. In the second iteration, the agent recognized the failure, informed the user that shipped orders can't be cancelled, and suggested initiating a return instead. The agent handled the error gracefully; this is expected behavior for an already-shipped order, not a bug.
+
+### Focused summary ("Why did it suggest a return?")
+
+> After `cancelOrder` failed with a "shipped" status error, the agent's reasoning (iteration 1 `think` span) explicitly noted that shipped orders can't be cancelled and decided to suggest a return as an alternative. This came from the model's own reasoning, not from the tool output or instructions -- it inferred the next-best action based on the error context.
+
+---
+
+## Patterns
+
+✅ **Synthesize, don't transcribe**
+
+```
+Good: "The agent looked up the order, found it was shipped, and told the user."
+Bad:  "Span sp_004 is a think span with content 'The user is asking...' then span sp_005 is a tool_call to lookupOrder with input orderId=4521..."
+```
+
+✅ **Name tools and key values**
+
+```
+Good: "Called `lookupOrder` with orderId '4521', got status 'shipped'."
+Bad:  "A tool was called and it returned some data."
+```
+
+✅ **Explain causation, not just sequence**
+
+```
+Good: "The cancelOrder call failed because the order was already shipped, so the agent suggested a return instead."
+Bad:  "cancelOrder failed. Then the agent said to try a return."
+```
+
+✅ **Flag what matters for the developer's question**
+
+```
+Good (for "why did it fail?"): "The root cause is that cancelOrder doesn't handle shipped orders -- it throws instead of returning an error object."
+Bad:  Giving a full trace walkthrough when the developer only asked about the failure.
+```
+
+❌ **Don't dump raw JSON or span IDs in the summary**
+
+Span IDs and raw field values are debugging artifacts. The summary is for understanding, not for grep. Reference tool names, error messages, and key data values instead.
+
+❌ **Don't speculate beyond the trace data**
+
+If the trace doesn't show why something happened, say so. Don't invent explanations.
+
+❌ **Don't include timing unless it's notable**
+
+A 3-second conversation turn is normal. A 45-second turn with 12 iterations is worth calling out.

package/dist/traces-and-logs-f2kjfh19.md

@@ -0,0 +1,203 @@
+# Traces & Logs
+
+Traces and logs are the primary debugging interface for ADK agents. Every conversation turn, tool call, LLM reasoning step, and error is recorded as a trace with spans. The CLI provides structured access to all of it.
+
+## CLI Debugging Tools
+
+Most commands support `--format json` for structured output — **always use it** when consuming output programmatically. The exception is `adk dev`, which has no `--format` flag: use `adk dev --non-interactive` to get a structured NDJSON event stream instead.
+
+| Command | Purpose | Options / tokens |
+|---------|---------|-----------------|
+| `adk check --format json` | Offline project validation — catches config/schema issues before runtime | — |
+| `adk logs --format json` | Browse or stream log entries | `--follow`, `--summary`, level filters (`error`, `warning`) |
+| `adk traces --format json` | Query traces with span details — filter by workflow, action, trigger, conversation, error, time | `workflow=`, `action=`, `trigger=`, `conversation=`, `trace=`, `error`, `since=`, `until=`, `limit=`, `--follow`, `--include-llm` |
+| `adk workflows runs --format json` | List or inspect durable workflow runs by id, name, or status | `name=`, `status=`, `limit=`, `nextToken=`, `<wrkflow_id>` |
+| `adk chat --single "msg" --format json` | Send a test message and get structured response | `--conversation-id` |
+| `adk dev --non-interactive` | Start dev server with structured NDJSON event stream (no TUI) | — |
+
+## Querying Logs
+
+Logs give a high-level view of what happened. Start here before diving into traces.
+
+```bash
+adk logs --format json                          # last 50 entries
+adk logs error --format json                    # errors only
+adk logs warning --format json                  # warnings and above
+adk logs warning since=1h --format json         # warnings from the last hour
+adk logs error limit=10 --format json           # last 10 errors
+adk logs --follow --format json                 # stream live as NDJSON
+adk logs --summary --format json                # single snapshot summary
+```
+
+### When to use logs vs traces
+
+- **Logs** — quick error scanning, startup issues, integration failures, deployment problems
+- **Traces** — detailed per-conversation analysis, tool call inspection, LLM reasoning review
+
+## Querying Traces
+
+Traces provide the full picture of a conversation turn: every span in order. The command takes positional filter tokens (not flags) for slicing the trace store.
+
+```bash
+adk traces --format json                                    # recent traces (default limit 20)
+adk traces error --format json                              # error traces only
+adk traces workflow=<name> --format json                    # traces for a workflow
+adk traces conversation=<id> --format json                  # traces for a conversation
+adk traces trace=<id> --include-llm --format json           # drill into one trace with LLM content
+adk traces since=1h limit=50 --format json                  # last hour, up to 50
+adk traces error --follow --format json                     # stream errors live (NDJSON)
+```
+
+Combine tokens freely (`adk traces error workflow=onboarding since=1h`). `--format` only accepts `json`; omit it for the default text output.
+
+### Trace Structure
+
+A trace represents one unit of agent activity. Each trace contains:
+
+- **Trace ID** — unique identifier
+- **Timestamp** — when the trace was recorded
+- **Conversation ID** — which conversation it belongs to
+- **Type** — the kind of activity
+- **Spans** — ordered list of sub-events within the trace
+
+### Span Types
+
+Spans are the atomic units inside a trace. Each span has a `type` field:
+
+| Span Type | Key Fields | What It Means |
+|-----------|-----------|---------------|
+| `tool_call` | `tool_name`, `input`, `output`, `success`, `error` | A tool was invoked. Check `success` to see if it worked. |
+| `code_execution_exception` | `message`, `stackTrace` | Code error in the LLMz sandbox. Read `stackTrace` to find the source. |
+| `think` | `content` | LLM reasoning step — shows why the model made a decision. |
+| `end` | — | Conversation turn completed. |
+
+### Reading a Trace
+
+When analyzing a trace:
+
+1. Look at spans in chronological order
+2. Find where behavior diverges from expected
+3. For `tool_call` spans: check `success`, read `error` if false, verify `input` matches expectations
+4. For `think` spans: understand model reasoning before a tool call
+5. For `code_execution_exception`: read the stack trace, correlate with source code
+
+## Reproducing Issues with `adk chat`
+
+Use `adk chat` to send test messages and generate fresh traces:
+
+```bash
+# Single message with structured output
+adk chat --single "Hello" --format json
+
+# Continue an existing conversation
+adk chat --single "Follow-up message" --conversation-id <id> --format json
+```
+
+The JSON output includes the bot's response and conversation ID, which you can use to query traces for that specific conversation and send subsequent messages.
+
+## Programmatic Trace Access: `onTrace` Hook
+
+For monitoring traces in code, use the `onTrace` hook in conversation handlers. This is useful for custom logging, performance monitoring, and automated error detection. The Botpress ADK also has an OTLP endpoint that can be enabled with the `--otlp` flag on `adk dev`, if the current environment is setup for OTLP traces.
+
+### Basic Trace Logging
+
+```typescript
+// src/conversations/extensions/logging.ts
+import { Autonomous } from "@botpress/runtime";
+
+export const onTraceLogging: Autonomous.Hooks["onTrace"] = ({ trace, iteration }) => {
+  if (trace.type === "code_execution_exception") {
+    console.error(`Code Execution Error: ${trace.message}`, trace.stackTrace);
+  }
+
+  if (trace.type === "tool_call" && !trace.success) {
+    console.error(
+      `Error during tool call to "${trace.tool_name}" with input "${JSON.stringify(trace.input)}":`,
+      trace.error
+    );
+  }
+};
+```
+
+### Comprehensive Trace Logging
+
+```typescript
+export const onTraceLogging: Autonomous.Hooks["onTrace"] = ({ trace, iteration }) => {
+  switch (trace.type) {
+    case "code_execution_exception":
+      console.error(`[CODE ERROR] ${trace.message}`, trace.stackTrace);
+      break;
+    case "tool_call":
+      if (trace.success) {
+        console.log(`[TOOL SUCCESS] ${trace.tool_name}`, trace.output);
+      } else {
+        console.error(`[TOOL ERROR] ${trace.tool_name}`, trace.error);
+      }
+      break;
+    case "think":
+      console.debug(`[THINK] ${trace.content}`);
+      break;
+    default:
+      console.log(`[TRACE] ${trace.type}`, trace);
+  }
+};
+```
+
+### Using in a Conversation
+
+```typescript
+import { Conversation } from "@botpress/runtime";
+import { onTraceLogging } from "./extensions/logging";
+
+export default new Conversation({
+  channel: "*",
+  handler: async ({ execute }) => {
+    await execute({
+      instructions: `You are a helpful assistant...`,
+      hooks: {
+        onTrace: (props) => onTraceLogging!(props),
+      },
+    });
+  },
+});
+```
+
+### Performance Monitoring
+
+```typescript
+export const makePerformanceMonitor = () => {
+  const startTime = Date.now();
+  const toolStartTimes = new Map<string, number>();
+  const toolMetrics = new Map<string, number[]>();
+
+  const onBeforeTool: Autonomous.Hooks["onBeforeTool"] = async ({ tool }) => {
+    toolStartTimes.set(tool.name, Date.now());
+  };
+
+  const onAfterTool: Autonomous.Hooks["onAfterTool"] = async ({ tool }) => {
+    const start = toolStartTimes.get(tool.name);
+    const duration = start ? Date.now() - start : 0;
+    const metrics = toolMetrics.get(tool.name) || [];
+    metrics.push(duration);
+    toolMetrics.set(tool.name, metrics);
+    console.log(`[PERF] ${tool.name}: ${duration}ms`);
+  };
+
+  const onTrace: Autonomous.Hooks["onTrace"] = ({ trace }) => {
+    if (trace.type === "end") {
+      console.log(`[PERF] Total conversation: ${Date.now() - startTime}ms`);
+      console.log(`[PERF] Tool metrics:`, Object.fromEntries(toolMetrics));
+    }
+  };
+
+  return { onBeforeTool, onAfterTool, onTrace };
+};
+```
+
+### Hook Reference
+
+| Hook | Parameters | When It Fires |
+|------|-----------|--------------|
+| `onBeforeTool` | `{ iteration, tool, input, controller }` | Before each tool call |
+| `onAfterTool` | `{ tool, output }` | After each tool call |
+| `onTrace` | `{ trace, iteration }` | On every trace event |