@botpress/adk-cli 1.19.0-beta.3 → 1.19.0-beta.5

package/dist/doc-standards-ck6ax2jd.md

@@ -0,0 +1,124 @@
+# ADK Documentation Standards
+
+Standards, templates, and quality metrics for ADK documentation.
+
+## Document Types
+
+Choose the appropriate style based on what you're documenting:
+
+### Reference Documentation (CLI, API endpoints)
+
+- **Length:** 400-500 lines
+- **Style:** Concise, one clear example per command/endpoint from actual project code
+- **Structure:** Quick Start at top, command reference, quick reference at bottom
+- **Focus:** Syntax, options, and practical workflows from real code
+
+```markdown
+# [Topic Name]
+
+Brief introduction (1-2 sentences)
+
+## Quick Start
+5-10 line essential usage example from actual project
+
+## Commands / Endpoints
+### command-name / GET /endpoint
+Brief description
+**Options:** (if applicable)
+**Example:** (one example from project code)
+
+## Common Workflows
+Only 3-4 workflows verified from actual project code
+
+## Quick Reference
+Command summary table or list
+
+## See Also
+Links to related docs
+```
+
+### Conceptual Guides (workflows, state management, patterns)
+
+- **Length:** 500-700 lines
+- **Style:** Examples from actual project implementations
+- **Structure:** Intro, Core Concepts, Features, Quick Reference
+- **Focus:** Understanding through real code patterns
+
+### Comprehensive Guides (messages, conversations, tables)
+
+- **Length:** 600-800 lines
+- **Style:** Complete API coverage with verified examples
+- **Structure:** Full TOC, Features, Patterns, Reference
+- **Focus:** Complete feature coverage with real implementations
+
+```markdown
+# [Topic Name]
+
+Brief introduction (2-3 sentences)
+
+## Table of Contents
+- Clear links to major sections
+- Include subsection highlights
+
+## Core Concepts
+Basic understanding needed
+
+## [Main Feature 1]
+### Basic Usage (example from actual project code)
+### Edge Cases (only if critical and seen in project)
+
+## [Main Feature 2]
+...
+
+## Quick Reference
+Essential patterns at a glance (if helpful)
+```
+
+## Quality Checklists
+
+### All Documents
+
+- [ ] Clear `##` and `###` headers for ripgrep
+- [ ] Every concept has code example from actual project
+- [ ] All examples verified in actual source locations
+- [ ] Searchable section names (no vague "Advanced Topics")
+- [ ] Critical distinctions highlighted (ADK vs SDK)
+- [ ] File paths included for code verification
+- [ ] No Common Mistakes or Best Practices sections unless user-provided
+
+### Reference Docs (CLI, API)
+
+- [ ] Quick Start at top with example from project
+- [ ] One clear example per command/endpoint
+- [ ] 3-4 workflows verified from project code
+- [ ] 400-500 lines total
+
+### Conceptual/Comprehensive Guides
+
+- [ ] Table of contents with descriptive links
+- [ ] Quick Reference section (if helpful)
+- [ ] 500-800 lines depending on complexity
+
+## Document Health Metrics
+
+**Reference Docs (CLI, API):**
+- Good: 400-500 lines, Quick Start at top, one example per command, 3-4 workflows
+- Warning: >600 lines, multiple examples per command, separate best practices section
+
+**Conceptual Guides:**
+- Good: 500-700 lines, progressive examples, verified workflows
+- Warning: >800 lines, vague sections, missing patterns
+
+**All Documents:**
+- Good: Every concept has code example, clear `##` headers, searchable section names
+- Warning: Vague section names ("Advanced", "Other"), no table of contents
+
+## Review Report Format
+
+When reviewing documentation, report findings in this structure:
+
+1. **Critical Issues** — Wrong code, broken examples, invented examples
+2. **Searchability Issues** — Missing TOC, unclear headers
+3. **Speculative Content** — Common mistakes without user feedback, invented best practices
+4. **Missing Verification** — Examples without file path references
+5. **Recommendations** — Priority improvements

package/dist/eval-format-7n9brja4.md

@@ -0,0 +1,354 @@
+# Eval File Format
+
+Evals are TypeScript files in the `evals/` directory. Each file exports one or more eval definitions using `new Eval`.
+
+## File Location
+
+Evals live in `evals/` at the **project root** (not inside `src/`). Create it if it doesn't exist:
+
+```
+my-agent/
+├── agent.config.ts
+├── src/
+│   ├── actions/
+│   └── workflows/
+└── evals/              ← create this
+    ├── greeting.eval.ts
+    └── billing.eval.ts
+```
+
+- **Naming**: `*.eval.ts` convention (recommended)
+- **Auto-discovery**: All files in `evals/` are picked up by `adk evals`
+
+## Full Structure
+
+```typescript
+import { Eval } from '@botpress/evals'
+
+export default new Eval({
+  name: 'my-eval',              // unique identifier (required)
+  description: 'What this tests', // optional
+  type: 'regression',                // 'capability' or 'regression' — optional, for filtering
+  tags: ['tools', 'multi-turn'],     // optional, for filtering
+
+  setup: {
+    // Seed state or trigger a workflow before the conversation (optional)
+  },
+
+  conversation: [
+    {
+      user: 'message from user', // or: event, expectSilence
+      assert: { /* per-turn assertions */ },
+    },
+  ],
+
+  outcome: { /* post-conversation assertions (optional) */ },
+
+  options: { /* per-eval overrides (optional) */ },
+})
+```
+
+## Types
+
+| Type | Purpose |
+|------|---------|
+| `capability` | Verify the bot can do something new |
+| `regression` | Verify the bot still does it correctly |
+
+## Multiple Evals Per File
+
+```typescript
+export const greeting = new Eval({ name: 'greeting', ... })
+export const farewell = new Eval({ name: 'farewell', ... })
+```
+
+---
+
+## Conversation Turns
+
+Each entry in `conversation` is one turn. A turn must have either `user` or `event`.
+
+### User Message
+
+```typescript
+{
+  user: 'What is my account balance?',
+  assert: { /* assertions on the bot's response */ },
+}
+```
+
+### Event Trigger
+
+Fire a non-message event (webhook, integration event) instead of a user message.
+
+```typescript
+{
+  event: {
+    type: 'checkout:order_placed',
+    payload: { orderId: 'ORD-001', total: 49.99 },
+  },
+  assert: {
+    workflow: [{ name: 'orderConfirmation', entered: true }],
+  },
+}
+```
+
+### Expect Silence
+
+Assert the bot does **not** respond. Add `expectSilence: true` to any turn.
+
+```typescript
+// Silence after a user message
+{ user: 'Please ignore this.', expectSilence: true }
+
+// Silence after an event
+{ event: { type: 'internal:ping' }, expectSilence: true }
+```
+
+> **Note:** `expectSilence` is mutually exclusive with `assert.response`. Every turn must have `user` or `event` — `expectSilence` is a flag on top of that, not a standalone turn type.
+
+---
+
+## Assertion Categories
+
+### Response
+
+What the bot said back.
+
+```typescript
+assert: {
+  response: [
+    { contains: 'ticket' },                    // substring present
+    { not_contains: 'error' },                 // substring absent
+    { matches: 'TKT-\\d{3}' },               // regex match
+    { llm_judge: 'Response confirms the ticket was created' }, // AI judge, scores 1–5
+  ],
+}
+```
+
+### Tools
+
+Which tools the bot called and with what parameters.
+
+```typescript
+assert: {
+  tools: [
+    { called: 'createTicket' },                          // tool was invoked
+    { called: 'createTicket', params: {                  // with specific params
+        priority: { equals: 'high' },
+        department: { contains: 'Engineering' },
+    }},
+    { not_called: 'deleteTicket' },                      // tool was NOT invoked
+    { call_order: ['lookupUser', 'createTicket'] },      // ordered calls
+  ],
+}
+```
+
+### State
+
+Bot, user, or conversation state values after the turn.
+
+```typescript
+assert: {
+  state: [
+    { path: 'conversation.topic', equals: 'support' },  // exact value
+    { path: 'conversation.topic', changed: true },       // value changed from before
+    { path: 'bot.ticketCount', equals: 3 },
+  ],
+}
+```
+
+### Workflow
+
+Workflow execution (verified via trace spans).
+
+```typescript
+assert: {
+  workflow: [
+    { name: 'onboarding', entered: true },    // workflow was started
+    { name: 'onboarding', completed: true },  // workflow finished
+  ],
+}
+```
+
+### Timing
+
+How long the bot took to respond (milliseconds).
+
+```typescript
+assert: {
+  timing: [
+    { response_time: { lte: 5000 } },  // must respond within 5s
+    { response_time: { gte: 100 } },   // sanity-check: not suspiciously fast
+  ],
+}
+```
+
+---
+
+## Match Operators
+
+Used in tool params, state values, and workflow params:
+
+| Operator | Example | Description |
+|----------|---------|-------------|
+| `equals` | `{ equals: 'urgent' }` | Exact match |
+| `contains` | `{ contains: 'HR' }` | Substring |
+| `not_contains` | `{ not_contains: 'test' }` | Excludes substring |
+| `matches` | `{ matches: '\\d+' }` | Regex |
+| `in` | `{ in: ['high', 'urgent'] }` | One of |
+| `exists` | `{ exists: true }` | Property exists |
+
+**Numeric-only operators** (for `timing` and numeric state values):
+
+| Operator | Example | Description |
+|----------|---------|-------------|
+| `gte` | `{ gte: 100 }` | Greater than or equal |
+| `lte` | `{ lte: 5000 }` | Less than or equal |
+
+---
+
+## Eval Setup
+
+Use `setup` to put the bot in a known state before the conversation starts.
+
+### Seed State
+
+Pre-populate bot, user, or conversation state.
+
+```typescript
+setup: {
+  state: {
+    bot: { welcomeMessageSent: true },
+    user: { plan: 'pro' },
+    conversation: { topic: 'billing' },
+  },
+}
+```
+
+The seeded state becomes the baseline for `changed` assertions — `{ changed: false }` passes if the value matches the seeded value at the end.
+
+### Trigger a Workflow
+
+Start a workflow before the conversation begins.
+
+```typescript
+setup: {
+  workflow: {
+    trigger: 'onboarding',
+    input: { userId: 'test-user-1' },
+  },
+}
+```
+
+Both can be combined:
+
+```typescript
+setup: {
+  state: {
+    bot: { sleepDurationMs: 3000 }, // 3 seconds instead of default 10 minutes
+  },
+  workflow: {
+    trigger: 'reminderFlow',
+    input: { userId: 'test-user-1' },
+  },
+}
+```
+
+> **Testing `step.sleep()`:** Server-side scheduling can't be fast-forwarded. Workaround: make the sleep duration configurable by reading it from bot state in the workflow, then seed a short value in ms (e.g. `3000`) in `setup.state.bot`.
+
+---
+
+## Outcome Assertions
+
+Run once after all conversation turns complete. Supports `state` and `workflow` (not `response` or `tools` — those are per-turn only).
+
+```typescript
+outcome: {
+  state: [
+    { path: 'conversation.resolved', equals: true },
+  ],
+  workflow: [
+    { name: 'ticketFlow', completed: true },
+  ],
+}
+```
+
+---
+
+## Options
+
+Override defaults for a specific eval. Cascades: **eval options → agent config → default**.
+
+```typescript
+options: {
+  idleTimeout: 60000,       // ms to wait for bot response (default: 30000)
+  judgePassThreshold: 4,    // llm_judge score required to pass, 1–5 (default: 3)
+}
+```
+
+Agent-level defaults in `agent.config.ts`:
+
+```typescript
+export default defineConfig({
+  evals: {
+    idleTimeout: 20000,
+    judgePassThreshold: 3,
+    judgeModel: 'fast',     // 'fast', 'best', or a model ref like 'openai:gpt-4o'
+  },
+})
+```
+
+---
+
+## Common Mistakes
+
+❌ **Turn with neither `user` nor `event`**
+
+```typescript
+// WRONG — every turn needs a trigger
+{ expectSilence: true }
+```
+
+✅ **Correct**
+
+```typescript
+{ user: 'hello', expectSilence: true }
+```
+
+---
+
+❌ **`expectSilence` with `assert.response`**
+
+```typescript
+// WRONG — mutually exclusive
+{ user: 'hello', expectSilence: true, assert: { response: [{ contains: 'hi' }] } }
+```
+
+✅ **Correct — pick one**
+
+```typescript
+{ user: 'hello', expectSilence: true }
+// or
+{ user: 'hello', assert: { response: [{ contains: 'hi' }] } }
+```
+
+---
+
+❌ **Both `user` and `event` on the same turn**
+
+```typescript
+// WRONG — mutually exclusive
+{ user: 'hello', event: { type: 'payment.failed' } }
+```
+
+✅ **Correct — use separate turns**
+
+```typescript
+{ event: { type: 'payment.failed', payload: { amount: 50 } } }
+```
+
+## See Also
+
+- [testing-workflow.md](./testing-workflow.md) — Running evals, interpreting output, the write → test → iterate loop
+- [test-patterns.md](./test-patterns.md) — Per-primitive testing patterns

package/dist/evals-y1zcr1mc.md

@@ -0,0 +1,11 @@
+# Evals
+
+Evals let you test your agent with automated scenarios and assertions.
+
+- File location: `evals/`
+- Run with: `adk evals`
+- View history: `adk evals runs`
+
+For detailed guidance — writing evals, assertion types, testing workflows, per-primitive patterns — use the `adk-evals` skill.
+
+Install it with `npx skills add botpress/skills --skill adk-evals`.