@botpress/adk-cli 1.19.0-beta.4 → 1.19.0-beta.5

package/dist/test-patterns-hj47w4nc.md

@@ -0,0 +1,355 @@
+# Per-Primitive Testing Patterns
+
+How to write evals for each ADK primitive type. Each section shows the recommended pattern, what to assert, and common pitfalls.
+
+---
+
+## Actions
+
+Actions are strongly-typed functions callable from conversations, workflows, other actions, or exposed as AI-callable tools. Test them by triggering a conversation that invokes the action and asserting on the response.
+
+```typescript
+import { Eval } from '@botpress/evals'
+
+export default new Eval({
+  name: 'lookup-order-action',
+  tags: ['actions'],
+
+  conversation: [
+    {
+      user: 'What is the status of order ORD-123?',
+      assert: {
+        response: [
+          { contains: 'ORD-123' },
+          { llm_judge: 'Response includes order status information' },
+        ],
+        tools: [
+          { called: 'lookupOrder', params: { orderId: { equals: 'ORD-123' } } },
+        ],
+      },
+    },
+  ],
+})
+```
+
+**What to assert:**
+- `tools.called` with expected params — verify the action was invoked correctly
+- `response.contains` — verify the bot surfaced the action's result
+- `state` — if the action writes to state, verify the write
+
+---
+
+## Tools
+
+Tools are LLM-callable functions. Focus on asserting which tools were called, with what params, and in what order.
+
+```typescript
+export default new Eval({
+  name: 'ticket-creation-tool',
+  tags: ['tools'],
+
+  conversation: [
+    {
+      user: 'I need help with a billing issue',
+      assert: {
+        tools: [{ not_called: 'createTicket' }], // not yet
+      },
+    },
+    {
+      user: 'Yes, please create a ticket for this',
+      assert: {
+        tools: [
+          { called: 'createTicket', params: {
+              category: { equals: 'billing' },
+              priority: { in: ['normal', 'high'] },
+          }},
+        ],
+        response: [{ contains: 'ticket' }],
+      },
+    },
+  ],
+})
+```
+
+**What to assert:**
+- `tools.called` with `params` — verify inputs are correctly extracted from the conversation
+- `tools.not_called` — verify tools are NOT called prematurely or inappropriately
+- `tools.call_order` — verify the correct sequence when multiple tools are involved
+
+### Tool call order pattern
+
+```typescript
+assert: {
+  tools: [
+    { call_order: ['lookupUser', 'fetchAccountBalance', 'formatResponse'] },
+  ],
+}
+```
+
+---
+
+## Workflows
+
+Workflows are long-running resumable processes. Use `setup.workflow` to trigger one before the conversation and assert on its execution.
+
+```typescript
+export default new Eval({
+  name: 'onboarding-workflow',
+  tags: ['workflows'],
+
+  setup: {
+    workflow: {
+      trigger: 'onboarding',
+      input: { userId: 'test-user-1', plan: 'pro' },
+    },
+  },
+
+  conversation: [
+    {
+      user: 'What happens next?',
+      assert: {
+        workflow: [
+          { name: 'onboarding', entered: true },
+        ],
+        response: [
+          { llm_judge: 'Response explains the onboarding next steps' },
+        ],
+      },
+    },
+  ],
+
+  outcome: {
+    workflow: [
+      { name: 'onboarding', completed: true },
+    ],
+    state: [
+      { path: 'user.onboardingComplete', equals: true },
+    ],
+  },
+})
+```
+
+### Testing time-based workflows (`step.sleep()`)
+
+Server-side scheduling cannot be fast-forwarded. Workaround: make the sleep duration configurable in the workflow by reading from bot state.
+
+**In your workflow:**
+
+```typescript
+const state = await client.getState({ type: 'bot', id: botId, name: 'botState' })
+const sleepMs = (state.state.payload?.value as any)?.sleepDurationMs ?? 10 * 60 * 1000
+await step.sleep('wait', sleepMs)
+```
+
+**In your eval:**
+
+```typescript
+setup: {
+  state: {
+    bot: { sleepDurationMs: 3000 }, // 3 seconds instead of 10 minutes
+  },
+  workflow: {
+    trigger: 'reminderFlow',
+    input: { userId: 'test-user-1' },
+  },
+}
+```
+
+**What to assert:**
+- `workflow.entered` — workflow was triggered
+- `workflow.completed` — workflow ran to completion (use in `outcome`)
+- `state` — final state after the workflow completes
+- `tools` — any tools the workflow invoked
+
+---
+
+## Conversations
+
+Conversations are channel-specific message handlers. Test multi-turn flows and context retention across turns.
+
+```typescript
+export default new Eval({
+  name: 'multi-turn-context',
+  tags: ['conversations'],
+
+  conversation: [
+    {
+      user: 'My name is Alice',
+      assert: {
+        response: [{ llm_judge: 'Bot acknowledges the name' }],
+      },
+    },
+    {
+      user: 'What is my name?',
+      assert: {
+        response: [{ contains: 'Alice' }], // bot retained context
+      },
+    },
+  ],
+})
+```
+
+### Testing event-driven conversations
+
+```typescript
+export default new Eval({
+  name: 'payment-failed-event',
+  tags: ['conversations', 'events'],
+
+  conversation: [
+    {
+      event: {
+        type: 'payment:failed',
+        payload: { amount: 99.99, currency: 'USD', customerId: 'cust-001' },
+      },
+      assert: {
+        response: [
+          { llm_judge: 'Bot notifies about the failed payment and offers help' },
+        ],
+        tools: [{ called: 'lookupCustomer' }],
+      },
+    },
+  ],
+})
+```
+
+### Testing that events produce no response
+
+```typescript
+export default new Eval({
+  name: 'internal-event-silence',
+  tags: ['conversations', 'events'],
+
+  conversation: [
+    {
+      event: { type: 'internal:heartbeat' },
+      expectSilence: true, // bot should not respond to internal events
+    },
+  ],
+})
+```
+
+---
+
+## State
+
+State is bot/user/conversation-scoped storage. Use `setup.state` to seed values and assert on state changes.
+
+### Asserting state was set
+
+```typescript
+export default new Eval({
+  name: 'state-write',
+  tags: ['state'],
+
+  conversation: [
+    {
+      user: 'I prefer to be contacted by email',
+      assert: {
+        state: [
+          { path: 'user.contactPreference', equals: 'email' },
+        ],
+      },
+    },
+  ],
+})
+```
+
+### Asserting state changed from seeded value
+
+```typescript
+export default new Eval({
+  name: 'state-transition',
+  tags: ['state'],
+
+  setup: {
+    state: {
+      conversation: { phase: 'greeting' },
+    },
+  },
+
+  conversation: [
+    {
+      user: 'I need help with billing',
+      assert: {
+        state: [
+          { path: 'conversation.phase', equals: 'support' },
+          { path: 'conversation.phase', changed: true }, // changed from 'greeting'
+        ],
+      },
+    },
+  ],
+})
+```
+
+### Asserting state did NOT change
+
+```typescript
+outcome: {
+  state: [
+    { path: 'bot.version', changed: false }, // seeded value, should be unchanged
+  ],
+}
+```
+
+---
+
+## Quick Reference: What to Assert Per Primitive
+
+| Primitive | Primary assertions | Secondary assertions |
+|-----------|-------------------|---------------------|
+| Actions | `tools.called` + params | `response`, `state` |
+| Tools | `tools.called/not_called/call_order` | `response` |
+| Workflows | `workflow.entered/completed` | `state`, `outcome` |
+| Conversations | `response` (multi-turn) | `tools`, `state` |
+| State | `state.equals/changed` | `outcome.state` |
+
+---
+
+## Pattern: Negative Testing
+
+Always test what the bot should NOT do, not just what it should do.
+
+```typescript
+conversation: [
+  {
+    user: 'Show me all users in the system',  // unauthorized request
+    assert: {
+      tools: [{ not_called: 'listAllUsers' }], // should not call admin tool
+      response: [
+        { not_contains: 'user@example.com' },  // should not leak data
+        { llm_judge: 'Response politely declines the request' },
+      ],
+    },
+  },
+]
+```
+
+## Pattern: Happy Path + Edge Case
+
+```typescript
+export const happyPath = new Eval({
+  name: 'create-ticket-success',
+  type: 'regression',
+  conversation: [{ user: 'Create a ticket', assert: { tools: [{ called: 'createTicket' }] } }],
+})
+
+export const missingInfo = new Eval({
+  name: 'create-ticket-missing-info',
+  type: 'capability',
+  conversation: [
+    {
+      user: 'Create a ticket', // no details provided
+      assert: {
+        tools: [{ not_called: 'createTicket' }], // should ask for more info first
+        response: [{ llm_judge: 'Bot asks for more information before creating a ticket' }],
+      },
+    },
+  ],
+})
+```
+
+## See Also
+
+- [eval-format.md](./eval-format.md) — Full eval file format and all assertion types
+- [testing-workflow.md](./testing-workflow.md) — Running evals and the write → test → iterate loop

package/dist/testing-workflow-2vkp932f.md

@@ -0,0 +1,225 @@
+# Testing Workflow
+
+The ADK testing loop: **write → run → inspect → iterate**. This document covers how to drive that loop using evals and the CLI.
+
+## Prerequisites
+
+`adk dev` must be running before you run evals. It provides the dev bot that evals talk to.
+
+```bash
+cd my-agent
+adk dev
+```
+
+If `adk dev` isn't running, `adk evals` auto-starts a lightweight server automatically — fine for one-off runs, but for repeated eval invocations always use `adk dev` to avoid the startup latency on every run.
+
+---
+
+## Running Evals
+
+### Run all evals
+
+```bash
+adk evals
+```
+
+### Run a single eval by name
+
+```bash
+adk evals greeting
+```
+
+### Filter by tag or type
+
+```bash
+adk evals --tag tools          # all evals tagged 'tools'
+adk evals --tag billing        # all evals tagged 'billing'
+adk evals --type regression    # all regression evals
+adk evals --type capability    # all capability evals
+```
+
+### Combine filters
+
+```bash
+adk evals --tag tools --type regression
+```
+
+### Output modes
+
+```bash
+adk evals --verbose    # show all assertions including passing ones
+adk evals --format json # machine-readable JSON output (for CI)
+```
+
+### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | All evals passed |
+| `1` | One or more evals failed |
+
+Use this in CI to gate merges on eval results.
+
+---
+
+## Interpreting Output
+
+> Always run with `--format json` when consuming output programmatically — structured output is more reliable than parsing text.
+
+### Passing eval
+
+```
+✔ greeting  (1.2s)
+  Turn 1: Hi there!
+    ✔ response.not_contains: error
+    ✔ response.llm_judge: Response is friendly and offers to help
+    ✔ tools.not_called: createTicket
+```
+
+### Failing eval
+
+```
+✖ ticket-creation  (3.4s)
+  Turn 2: Create a high priority ticket
+    ✔ response.contains: ticket
+    ✖ tools.called: createTicket
+        expected: called
+        actual:   not called
+    ✖ state.equals: conversation.topic = 'support'
+        expected: 'support'
+        actual:   null
+```
+
+Read the `expected` / `actual` diff to understand exactly what the bot did vs what you asserted.
+
+### LLM judge scores
+
+`llm_judge` assertions score 1–5. The default pass threshold is 3. A score of 1–2 = fail, 3–5 = pass (unless overridden via `judgePassThreshold`).
+
+---
+
+## Viewing Run History
+
+```bash
+adk evals runs              # list all recent runs
+adk evals runs --latest     # show the most recent run
+adk evals runs --latest -v  # latest run with full assertion details
+adk evals runs <id>         # show a specific run by ID
+```
+
+Run history is also visible in the Dev Console at `http://localhost:3001/evals` while `adk dev` is running. Direct humans to this URL for a clear visual overview of eval runs.
+
+---
+
+## Testing via Chat
+
+For exploratory testing before writing a formal eval:
+
+```bash
+adk chat
+```
+
+This opens an interactive chat session with your dev bot. Use it to:
+- Manually test a flow before formalizing it as an eval
+- Quickly verify a fix after a failing eval
+- Explore edge cases interactively
+
+For a one-shot message without opening an interactive session:
+
+```bash
+adk chat --single "What is my account balance?"
+```
+
+For more information on chat subcommands, read the `adk` skill or run `adk chat -h`.
+
+---
+
+## Inspecting Traces After a Turn
+
+After a conversation, traces show exactly what the bot did internally — tool calls, workflow steps, LLM calls, state reads/writes.
+
+Use the CLI to inspect traces:
+
+```bash
+adk traces                  # list recent traces
+adk logs                    # browse recent logs
+```
+
+Use traces to verify:
+- Which tools were called (and with what inputs/outputs)
+- Whether a workflow was entered or completed
+- Where the bot spent its time (latency analysis)
+- Any errors or unexpected state transitions
+
+If the human needs a visual overview, direct them to `http://localhost:3001/traces` in the Dev Console while `adk dev` is running.
+
+---
+
+## The Write → Test → Inspect → Iterate Loop
+
+### Step 1: Write the eval
+
+Create `evals/my-feature.eval.ts` with the scenario you want to test. Start with one turn and simple assertions (`response.not_contains: 'error'`).
+
+### Step 2: Run it
+
+```bash
+adk evals my-feature
+```
+
+### Step 3: Read the output
+
+- ✔ All pass → add more assertions or more turns
+- ✖ Something fails → read the `expected` / `actual` diff
+
+### Step 4: Inspect traces if needed
+
+If the failure isn't obvious from the eval output, use `adk traces` or `adk logs` to see what the bot actually did internally.
+
+### Step 5: Fix the code
+
+Update your action, tool, workflow, or conversation handler to fix the failing behavior.
+
+### Step 6: Re-run
+
+```bash
+adk evals my-feature
+```
+
+Repeat until green, then run the full suite to check for regressions:
+
+```bash
+adk evals
+```
+
+---
+
+## CI Integration
+
+Add eval runs to your CI pipeline using `--format json` and the exit code:
+
+```yaml
+# GitHub Actions example
+- name: Run evals
+  run: adk evals --format json > eval-results.json
+
+- name: Check results
+  run: cat eval-results.json | jq '.failed == 0'
+```
+
+Tag regression evals and run them on every PR:
+
+```bash
+adk evals --type regression --format json
+```
+
+Run capability evals only on feature branches or release cuts:
+
+```bash
+adk evals --type capability --format json
+```
+
+## See Also
+
+- [eval-format.md](./eval-format.md) — Full eval file format and all assertion types
+- [test-patterns.md](./test-patterns.md) — Per-primitive testing patterns