@botpress/adk-cli 1.19.0-beta.3 → 1.19.0-beta.5

package/dist/SKILL-d4bs53y3.md

@@ -0,0 +1,244 @@
+---
+name: adk-evals
+description: Complete reference for writing, running, and iterating on evals (automated conversation tests) for ADK agents. Covers eval file format, all assertion types, CLI usage, and per-primitive testing patterns.
+license: MIT
+---
+
+# ADK Evals Skill
+
+## What are Evals?
+
+Evals are automated conversation tests for ADK agents. Each eval defines a scenario — a sequence of user messages or events — and asserts on what the bot should do: what it says, which tools it calls, how state changes, which workflows run, and more.
+
+Evals run against a live dev bot (`adk dev`), so they test the full stack — not mocks.
+
+## When to Use This Skill
+
+Use this skill when the developer asks about:
+
+- **Writing evals** — file format, assertions, turn types, setup
+- **Running evals** — CLI commands, filtering, output interpretation
+- **Testing specific primitives** — how to test actions, tools, workflows, conversations, state
+- **The testing loop** — write → run → inspect traces → iterate
+- **CI integration** — exit codes, `--format json` flag, tagging strategies
+- **Eval configuration** — idleTimeout, judgePassThreshold, judgeModel
+
+Or when you are developing an ADK bot and need to write the equivalent of unit/end-to-end tests.
+
+**Trigger questions:**
+- "How do I write an eval?"
+- "How do I test my workflow?"
+- "How do I assert that a tool was called with specific params?"
+- "My eval is failing, how do I debug it?"
+- "How do I test that the bot stays silent?"
+- "How do I run evals in CI?"
+- "How do I seed state before an eval?"
+- "How do I trigger a workflow in an eval?"
+
+## Available Documentation
+
+| File | Contents |
+|------|----------|
+| `references/eval-format.md` | Complete file format — all fields, turn types, assertion categories, match operators, setup, outcome, options |
+| `references/testing-workflow.md` | Running evals, interpreting output, using traces, the write → test → iterate loop, CI integration |
+| `references/test-patterns.md` | Per-primitive patterns for actions, tools, workflows, conversations, and state |
+
+## How to Answer
+
+1. **Writing an eval** → Read `eval-format.md` for structure and assertions
+2. **Running evals** → Read `testing-workflow.md` for CLI commands and output
+3. **Testing a specific primitive** → Read `test-patterns.md` for the relevant section
+4. **Debugging a failure** → Combine `testing-workflow.md` (inspect traces) + `eval-format.md` (check assertion syntax)
+
+---
+
+## Quick Reference
+
+### Eval file structure
+
+```typescript
+import { Eval } from '@botpress/evals'
+
+export default new Eval({
+  name: 'greeting',
+  type: 'regression',
+  tags: ['basic'],
+
+  setup: {
+    state: { bot: { welcomeSent: false } },
+    workflow: { trigger: 'onboarding', input: { userId: 'test-1' } },
+  },
+
+  conversation: [
+    {
+      user: 'Hi!',
+      assert: {
+        response: [
+          { not_contains: 'error' },
+          { llm_judge: 'Response is friendly and offers to help' },
+        ],
+        tools: [{ not_called: 'createTicket' }],
+        state: [{ path: 'conversation.greeted', equals: true }],
+      },
+    },
+  ],
+
+  outcome: {
+    state: [{ path: 'conversation.greeted', equals: true }],
+  },
+
+  options: {
+    idleTimeout: 60000,
+    judgePassThreshold: 4,
+  },
+})
+```
+
+### Turn types
+
+| Turn | When to use |
+|------|------------|
+| `user: 'message'` | Standard user message |
+| `event: { type, payload }` | Non-message trigger (webhook, integration event) |
+| `expectSilence: true` | Assert bot does NOT respond |
+
+### Assertion categories
+
+| Category | What it checks |
+|----------|---------------|
+| `response` | Bot reply text (contains, not_contains, matches, llm_judge) |
+| `tools` | Tool calls (called, not_called, call_order, params) |
+| `state` | Bot/user/conversation state (equals, changed) |
+| `workflow` | Workflow execution (entered, completed) |
+| `timing` | Response time in ms (lte, gte) |
+
+### CLI commands
+
+```bash
+adk evals                        # run all evals
+adk evals <name>                 # run one eval
+adk evals --tag <tag>            # filter by tag
+adk evals --type regression      # filter by type
+adk evals --verbose              # show all assertions
+adk evals --format json          # JSON output for CI
+
+adk evals runs                   # list recent runs
+adk evals runs --latest          # most recent run
+adk evals runs --latest -v       # with full details
+```
+
+---
+
+## Critical Patterns
+
+✅ **Every turn needs `user` or `event`**
+
+```typescript
+// CORRECT
+{ user: 'hello', expectSilence: true }
+{ event: { type: 'payment.failed' }, expectSilence: true }
+```
+
+❌ **`expectSilence` alone is not a valid turn**
+
+```typescript
+// WRONG — missing user or event
+{ expectSilence: true }
+```
+
+---
+
+✅ **Assert tool params to verify correct extraction**
+
+```typescript
+// CORRECT — verifies the LLM extracted the right values
+{ called: 'createTicket', params: { priority: { equals: 'high' } } }
+```
+
+❌ **Only asserting the tool was called**
+
+```typescript
+// INCOMPLETE — doesn't verify params were correct
+{ called: 'createTicket' }
+```
+
+---
+
+✅ **Use `outcome` for post-conversation state and workflow assertions**
+
+```typescript
+// CORRECT — final state checked once after all turns
+outcome: {
+  state: [{ path: 'conversation.resolved', equals: true }],
+  workflow: [{ name: 'ticketFlow', completed: true }],
+}
+```
+
+---
+
+✅ **Seed state to test conditional behavior without running setup turns**
+
+```typescript
+// CORRECT — start in a known state
+setup: {
+  state: {
+    user: { plan: 'pro' },
+    conversation: { phase: 'support' },
+  },
+}
+```
+
+❌ **Using conversation turns to set up state (slow and fragile)**
+
+```typescript
+// WRONG — depends on the bot correctly processing setup turns
+conversation: [
+  { user: 'I am on the pro plan' },      // hoping bot sets user.plan
+  { user: 'I need help with billing' },   // actual test turn
+]
+```
+
+---
+
+## Example Questions
+
+**Writing evals:**
+- "Write an eval that tests my createTicket tool is called with the right priority"
+- "How do I assert that the bot stays silent after an internal event?"
+- "How do I test a multi-turn conversation where context is retained?"
+
+**Running evals:**
+- "How do I run only regression evals?"
+- "How do I see which assertions failed and why?"
+- "How do I integrate evals into GitHub Actions?"
+
+**Debugging:**
+- "My eval says the tool wasn't called but I think it was — how do I check?"
+- "How do I inspect what the bot actually did during an eval?"
+
+**Per-primitive:**
+- "How do I test a workflow that uses step.sleep()?"
+- "How do I test that state changed from the seeded value?"
+
+---
+
+## Response Format
+
+**Match depth to the question.**
+
+### Simple questions ("what assertions are available?", "how do I run evals?")
+
+Answer directly — show the relevant table or CLI command. Don't generate a full eval file for an informational question.
+
+### Writing an eval
+
+1. Show the complete `new Eval({})` call with realistic field values
+2. Include imports (`import { Eval } from '@botpress/evals'`)
+3. Briefly explain non-obvious assertions — skip if the assertion is self-explanatory
+4. Suggest the CLI command to run it: `adk evals <name>`
+
+### Debugging a failing eval
+
+1. Ask for or show the failing assertion (`expected` / `actual` diff)
+2. Suggest opening traces in the Dev Console to see what the bot did
+3. Identify whether the issue is in the eval assertion or the bot's behavior

package/dist/SKILL-h4b2f1n9.md

@@ -0,0 +1,114 @@
+---
+name: adk-docs
+description: guidelines for creating, reviewing, updating, and searching ADK documentation - use when users ask about writing, maintaining, or auditing ADK bot docs
+license: MIT
+---
+
+# ADK Documentation Management
+
+Use this skill when helping users create, review, update, or maintain documentation for their ADK projects. Users write guides and docs for their own bots and features — code examples should come from their project and the official ADK repo.
+
+## When to Use This Skill
+
+Activate this skill when users:
+
+- Want to create documentation for their bot ("Document my order workflow", "Write a guide for my custom actions")
+- Need to review existing project docs for accuracy ("Check if my docs are still correct")
+- Want to update docs after changing their bot ("I refactored the checkout flow, update the docs")
+- Need to check if docs are in sync with their code ("Are my docs current?")
+- Want to search their project documentation ("Find docs about my payment handler")
+- Ask about documentation standards or templates
+- Mention `/adk-document`
+
+## Available Documentation
+
+| File | Description |
+|------|-------------|
+| [references/doc-standards.md](./references/doc-standards.md) | Document types, templates, quality checklists, and health metrics |
+
+## Available Commands
+
+| Command (and suggested subcommand) | Description |
+|---------|-------------|
+| `/adk-document create [topic]` | Create documentation for a feature in the user's bot |
+| `/adk-document review [doc-path]` | Review project docs for accuracy and completeness |
+| `/adk-document update [doc-path] [what-changed]` | Update project docs after code changes |
+| `/adk-document sync [optional-doc-path]` | Check if project docs are in sync with the bot's code |
+| `/adk-document search [search-term]` | Search project documentation for specific topics |
+
+## Shared Principles
+
+### 1. AI-Optimized Structure
+
+- Clear section headers (`##`, `###`, `####`) so ripgrep can find sections
+- Table of contents at top with anchor links
+- Right-sized for document type (see doc-standards reference)
+- Keyword-rich section names — no vague "Advanced Topics" or "Other"
+
+### 2. Code-First Approach
+
+- Every concept needs a working code example from actual project code
+- Include file paths with line numbers for verification
+- Primary sources for examples (in priority order):
+  1. The user's own ADK project (look for `agent.config.ts` in the workspace) — BEST
+  2. The official ADK repo examples (clone or find locally)
+  3. ADK runtime usage (`@botpress/runtime` packages)
+- Never invent or speculate examples — all code must be verifiable
+
+### 3. Critical Distinctions
+
+- ADK primitives (from `@botpress/runtime`) vs Botpress SDK primitives
+- `this.send()` in conversations vs `client.createMessage()` in workflows
+- Messages (persistent, stored) vs Events (ephemeral, not stored)
+- Agnostic APIs vs channel-specific features
+
+### 4. No Speculation
+
+- Do NOT add Common Mistakes or Best Practices sections unless the user explicitly provides them
+- Use `❌ WRONG` / `✅ CORRECT` only when documenting actual reported errors
+- Workflows and examples must be verified from actual code, not imagined
+
+### 5. Writing Style
+
+- **Direct and actionable** — "Use `this.send()`" not "You might want to consider..."
+- **Technically accurate** — test examples against actual ADK code
+- **Assumes intelligence** — provide context and guidance, not hand-holding
+- **No marketing fluff** — straight to the technical substance
+
+## How to Answer
+
+When a user asks about documentation without invoking a specific command:
+
+1. **Load the `adk` skill (& more, if needed)** for ADK context and knowledge
+2. **Identify what they need** — creating, reviewing, updating, syncing, or searching
+3. **Point them to the right command** or help directly using the principles above
+4. **Reference doc-standards** for template and quality guidance
+
+## Documentation Location
+
+Documentation is written in the user's own project. Ask the user where they want docs saved if not obvious (common locations: `./docs/`, `./guides/`, or project root).
+
+## Discovering Code Sources
+
+Discover the user's project and the official ADK repo for examples:
+
+```javascript
+// Find ADK projects in the workspace
+Glob({ pattern: "**/agent.config.ts" })
+
+// Find ADK runtime usage in the user's project
+Grep({ pattern: "from ['\"]@botpress/runtime", output_mode: "files_with_matches" })
+
+// Look for existing project documentation
+Glob({ pattern: "./{docs,guides}/**/*.md" })
+
+// Find official ADK examples if available locally
+Glob({ pattern: "**/adk/examples/**/*.ts" })
+```
+
+## Cross-References
+
+- **`adk` skill** — Core ADK knowledge (actions, workflows, conversations, etc.)
+- **`adk-evals` skill** — Testing and eval documentation
+- **`adk-frontend` skill** — Frontend integration documentation
+- **`adk` skill `integrations.md` reference** — Integration lifecycle documentation

package/dist/SKILL-hbw5nt95.md

@@ -0,0 +1,111 @@
+---
+name: adk-dev-console
+description: Explains the ADK Dev Console — what each tab shows, how to read Agent Steps, traces, multi-agent dashboard, agent switching, console modes, and other UI features visible at localhost:3001 during adk dev
+license: MIT
+---
+
+# ADK Dev Console
+
+The Dev Console is a local app served at port `3001` (by default, but can be customized) during `adk dev`. It gives developers real-time visibility into their agent's behavior — conversations, execution traces, data, integrations, and configuration.
+
+The Dev Console is a **singleton shared dashboard**: one UI server per user, with multiple `adk dev` agents registering and deregistering dynamically. Developers can switch between running agents in the sidebar, toggle between dev and prod targets, and connect to deployed Cloud bots — all from the same browser tab.
+
+## When to Use This Skill
+
+Activate when users ask about:
+
+- **UI concepts** — "What are Agent Steps?", "What does the Observe tab show?", "What is a turn?", "What is the component registry?", "What is the Agent Map?"
+- **Dev Console navigation** — "What tabs are available?", "Where do I find traces?", "How do I test RAG?"
+- **Reading execution data** — "What do the steps mean?", "Why is my step red?", "What's the cost shown?"
+- **Component registry** — "Where do I find custom components?", "How do I install a component?", "What components are available?"
+- **Multi-agent dashboard** — "How do I switch between agents?", "How do I run multiple agents?", "What does the agent selector show?", "How do I see which agents are running?"
+- **Console modes and targets** — "What's the difference between dev and prod?", "How do I connect to a Cloud bot?", "What is Cloud Dev Console?"
+- **Agent management CLI** — "How do I list running agents?", "How do I stop an agent?", "What does `adk ps` show?", "How do I open the dashboard?"
+- **Specific pages** — "How do I use the evals page?", "Where do I configure integrations?", "How do I search knowledge?"
+- **Comparing UI vs CLI** — "Should I use the Dev Console or CLI for debugging?"
+- Mentions of `localhost:3001`, "dev console", "Dev Console", "component registry", "agent selector", "agent picker", "agent map", "Cloud Dev Console", or specific tab names (Chat, Build, Components, Data, Test, Observe, Config)
+
+## Available Documentation
+
+| File | Contents |
+|------|----------|
+| `references/agent-steps.md` | Agent Steps visualization — turns, iterations, tools, messages, state mutations, cost tracking, status indicators |
+| `references/pages.md` | Every page/tab in the Dev Console — what it shows, key features, layout |
+| `references/component-registry.md` | Component Registry — installed vs registry tabs, component lifecycle, runtime registry internals, UI features |
+| `references/multi-agent-dashboard.md` | Multi-agent architecture, agent selector, console modes (local/cloud), dev/prod targets, CLI commands (`adk ps`, `adk kill`, `adk dashboard`, `adk status`) |
+
+## How to Answer
+
+Match depth to the question:
+
+- **"What is X?"** (e.g., "What are Agent Steps?") → One sentence definition + what the user sees. Don't dump the full data model.
+- **"Where do I find X?"** → Name the tab group and page, give the URL path.
+- **"How do I read X?"** → Explain the visual hierarchy and what each element means.
+- **"What does this mean?"** (pointing at something in the UI) → Identify the component, explain its meaning.
+
+## Quick Reference
+
+### Tab Groups
+
+| Group | Pages | Purpose |
+|-------|-------|---------|
+| **Chat** | Chat | Test the agent via webchat + see Agent Steps |
+| **Build** | Agent Map | Interactive agent architecture graph (experimental, feature-flagged behind `enable_agent_forge`) |
+| **Components** | Webchat Components, Actions, Workflows, Triggers | Browse component registry, test bot primitives |
+| **Test** | RAG Search, Evals | Test knowledge search and run automated evals |
+| **Data** | Knowledge, Tables, Files | Manage knowledge bases, tables, and files |
+| **Observe** | Conversations, Traces, Logs | View conversation history, execution traces, runtime logs |
+| **Config** | Settings, Integrations | Agent config, secrets, LLM settings, integration management |
+
+### Multi-Agent Sidebar
+
+The sidebar header shows the currently selected agent with a status indicator and mode pill (dev/prod/cloud). Clicking it opens the **Agent Selector** dropdown:
+
+- **Active agents** — all running local agents with status dot, name, project path, and a close button
+- **Cloud Dev Console** — switch to a deployed Botpress Cloud bot (workspace + bot picker)
+- **Recent projects** — previously opened projects not currently running
+- **Footer actions** — Create new project, Open existing, Switch environment (dev↔prod), About
+
+### Agent Steps (Chat Page)
+
+The right panel of the Chat page shows **Agent Steps** — a real-time visualization of what the agent did to process each message.
+
+**Hierarchy:** Turn → Iterations → Tools / Messages / State Mutations
+
+- **Turn** = one conversation exchange (user message → agent processing → response)
+- **Iteration** = one loop of the autonomous agent (think → decide → act)
+- **Tool** = a tool call within an iteration (violet card)
+- **Message** = a bot message sent (blue card)
+- **State Mutation** = a state change (teal card, shows before/after)
+
+**Status indicators:** ✓ green = ok, ✗ red = error, ⟳ blue spinning = running
+
+**AI metrics per iteration:** model name, input/output tokens, cost (USD)
+
+### Key URLs
+
+| Path | Page |
+|------|------|
+| `/chat` | Chat + Agent Steps |
+| `/agent-map` | Agent Map — interactive architecture graph (experimental) |
+| `/components` | Component registry + installed components |
+| `/actions` | Actions browser |
+| `/workflows` | Workflows + run history |
+| `/search` | RAG search testing |
+| `/evals` | Eval definitions + runs |
+| `/knowledge` | Knowledge base management |
+| `/tables` | Table data management |
+| `/traces` | Full trace viewer |
+| `/conversations` | Conversation history |
+| `/logs` | Runtime logs |
+| `/settings` | Agent configuration |
+| `/integrations` | Integration management |
+
+### Multi-Agent CLI Commands
+
+| Command | Purpose |
+|---------|---------|
+| `adk ps` | List running agents and processes with PIDs, ports, uptime (supports `--watch`) |
+| `adk dashboard` | Open the DevConsole in standalone mode (no agent required) |
+| `adk kill` | Stop agents or the entire DevConsole (`--all`, `--current`, `--pid`, or by name) |
+| `adk status` | Show project health and status info |