@indemn/cli 1.2.1 → 1.2.2

package/skills/agent-setup/SKILL.md

@@ -1,131 +0,0 @@
----
-name: agent-setup
-description: End-to-end agent creation workflow using the indemn CLI — create, configure, and test an Indemn AI agent. Use when the user wants to set up a new agent from scratch.
----
-
-# Agent Setup
-
-Guide the user through creating a fully configured Indemn AI agent using the `indemn` CLI. This skill coordinates across all agent management commands to go from zero to a working, testable agent.
-
-**Prerequisite:** The `indemn` CLI must be installed and authenticated. Run `indemn whoami` to verify.
-
----
-
-## Workflow
-
-### Step 1: Gather Requirements
-
-Ask the user:
-1. **Agent name** — What should this agent be called?
-2. **Purpose** — What does the agent do? (insurance quotes, customer service, FAQ, lead gen, etc.)
-3. **Channels** — Where will it be deployed? (WEB, VOICE, or both)
-4. **Key behaviors** — Any specific rules, tone, or constraints?
-
-### Step 2: Create the Agent
-
-```bash
-indemn agents create --name "<agent name>"
-```
-
-Capture the agent ID from the output. Report it to the user.
-
-### Step 3: Set the System Prompt
-
-Based on the gathered requirements, draft a system prompt. Include:
-- Agent persona (name, role, company)
-- Core instructions (what the agent does, how it handles conversations)
-- Key constraints (what it should NEVER do)
-- Tone guidance
-
-Present the draft prompt to the user for review before setting it.
-
-```bash
-indemn config set <agent-id> --prompt "You are..."
-```
-
-Verify it was set correctly:
-```bash
-indemn config get <agent-id> --field prompt
-```
-
-### Step 4: Create Functions (if needed)
-
-If the agent needs to call external APIs or perform actions:
-
-```bash
-indemn functions create <agent-id> --type rest_api --name "function_name" --description "What this function does"
-```
-
-### Step 5: Create Knowledge Base (if needed)
-
-If the agent needs reference information:
-
-1. Create the KB:
-```bash
-indemn kb create --name "<KB Name>" --type qna
-```
-
-2. Add QA data sources:
-```bash
-indemn kb data add <kb-id> --question "What is your coverage?" --answer "We offer auto, home, and life insurance."
-```
-
-3. Link the KB to the agent:
-```bash
-indemn kb link <agent-id> <kb-id>
-```
-
-### Step 6: Test the Agent
-
-Send test messages to verify the agent responds correctly:
-```bash
-indemn chat <agent-id> --message "Hello, I need help with insurance"
-```
-
-The `--message` flag sends a single message, prints the agent's response, and exits. Run multiple messages to test different scenarios:
-```bash
-indemn chat <agent-id> --message "What types of coverage do you offer?"
-indemn chat <agent-id> --message "Can you help me file a claim?"
-```
-
-If the response looks wrong:
-- Check the prompt: `indemn config get <agent-id> --field prompt`
-- Check agent details: `indemn agents get <agent-id> --json`
-- Adjust the prompt: `indemn config set <agent-id> --prompt "..."`
-- Re-test with `indemn chat <agent-id> --message "..."`
-
-### Step 7: Summary
-
-Report to the user:
-- Agent ID and name
-- Configured prompt (brief summary)
-- Functions created (if any)
-- Knowledge bases linked (if any)
-- Test chat result
-- Next steps: run evaluations with `/eval-orchestration`, iterate on prompt
-
----
-
-## CLI Commands Used
-
-| Command | Step | Purpose |
-|---------|------|---------|
-| `indemn agents create` | 2 | Create the agent |
-| `indemn config set --prompt` | 3 | Configure system prompt |
-| `indemn config get --field prompt` | 3 | Verify prompt |
-| `indemn functions create` | 4 | Add API functions |
-| `indemn kb create` | 5 | Create knowledge base |
-| `indemn kb data add` | 5 | Add QA pairs |
-| `indemn kb link` | 5 | Connect KB to agent |
-| `indemn chat --message` | 6 | Send test message and get response |
-| `indemn agents get` | 6 | Check full agent config |
-
----
-
-## Tips
-
-- Start simple — a good prompt is more important than functions or KBs
-- Test after each major change to catch issues early
-- Use `/eval-orchestration` after setup to create formal test coverage
-- Keep the initial prompt focused — you can iterate after seeing test results
-- All list commands support `--json` for machine-readable output

package/skills/eval-analysis/SKILL.md

@@ -1,178 +0,0 @@
----
-name: eval-analysis
-description: Analyze evaluation results to classify failures as agent issues vs evaluation issues. Also export conversation traces for sharing. Use when the user asks to analyze eval results, understand why tests failed, triage failures, improve evaluation scores, pull conversation traces, or share eval results.
-argument-hint: [run-id or agent-id]
-allowed-tools: Bash(indemn *)
----
-
-# Eval Analysis
-
-Analyze evaluation run results, classify each failure, and recommend fixes. Also supports exporting conversation traces and generating branded PDF reports.
-
-**Input:** `$ARGUMENTS` — a run_id (UUID) or agent_id (MongoDB ObjectId)
-
-## Step 1: Resolve the Run
-
-Determine what `$ARGUMENTS` is:
-- **UUID format** (contains hyphens, 36 chars) → it's a `run_id`. Use directly.
-- **ObjectId format** (24 hex chars) → it's an `agent_id`. Fetch the latest completed run:
-  ```bash
-  indemn eval list --agent-id $ARGUMENTS --json --limit 5
-  ```
-  Extract the latest `run_id` with `status: completed`.
-
-If no completed runs exist, tell the user and stop.
-
-## Step 2: Pull All Data
-
-Fetch all data using the CLI (parallelize where possible):
-
-```bash
-# Run summary
-indemn eval status <run-id> --json
-
-# Per-item results
-indemn eval results <run-id> --json
-
-# Rubric (get rubric_id from run summary)
-indemn rubric get <rubric-id> --json
-
-# Test set (get test_set_id from run summary)
-indemn testset get <test-set-id> --json
-
-# Bot context (get agent_id from run summary)
-indemn eval bot-context <agent-id> --json
-```
-
-For exact response shapes, see `references/data-shapes.md`.
-
-## Step 3: Present Run Overview
-
-Show the user a summary table:
-
-```
-Run: {run_id}
-Agent: {bot_name} ({agent_id})
-LLM: {bot_llm_provider}/{bot_llm_model}
-Status: {passed}/{total} items passed ({pass_rate}%)
-
-Criteria: {criteria_passed}/{criteria_total} checks passed
-Rubric: {rubric_rules_passed}/{rubric_rules_total} rules passed
-
-Component Scores:
-  prompt:         {score}% ({passed}/{total})
-  knowledge_base: {score}% ({passed}/{total})
-  function:       {score}% ({passed}/{total})
-  general:        {score}% ({passed}/{total})
-```
-
-## Step 4: Analyze Each Failed Item
-
-For every result where `passed == false`:
-
-1. **Get the test item** — match `test_case_id` to the test set's `items[].item_id` to get the item name, type, inputs, and success_criteria
-2. **Check criteria scores** — from `criteria_scores[]`, list which criteria passed and which failed with their reasoning
-3. **Check rubric scores** — from `rubric_scores[]`, list which rules passed and which failed with their reasoning
-4. **Check for divergence** — flag when criteria say FAIL but rubric says PASS (or vice versa). Divergence is a strong signal of an evaluation problem.
-5. **Read the conversation** — look at `output` field for the full agent response or conversation transcript
-6. **Cross-reference system prompt** — check if the agent's behavior aligns with what the system prompt instructs
-7. **Classify** — assign one of four categories (see below)
-
-### Failure Categories
-
-| Category | Signal | Action |
-|----------|--------|--------|
-| **Real Agent Problem** | Both criteria and rubric agree agent misbehaved, or output contradicts system prompt | Investigate agent code/prompt |
-| **Evaluation Problem** | Criteria ambiguous/too strict, evaluator reasoning shows misunderstanding, agent response was reasonable | Update test set criteria |
-| **Test Design Problem** | Test assumptions wrong (e.g., assumed KB lacked info it has), criteria contradict agent's actual capabilities | Redesign test item |
-| **KB/Retrieval Problem** | Agent behavior correct but retrieval returned wrong content, or KB is missing needed info | Fix KB content or retrieval config |
-
-### Divergence Analysis
-
-When `criteria_passed != rubric_passed` for an item, this is a first-class finding:
-- **Criteria FAIL + Rubric PASS** → criteria are likely too strict or testing the wrong thing
-- **Criteria PASS + Rubric FAIL** → rubric may have rules that don't apply to this scenario, or agent violated a behavioral rule despite answering correctly
-
-Always highlight divergences prominently in the report.
-
-## Step 5: Aggregate Findings
-
-Group all failed items by category:
-
-```
-## Findings
-
-### Real Agent Problems (N items)
-- Item X: [description of what went wrong]
-
-### Evaluation Problems (N items)
-- Item Y: [why the criteria/rubric is wrong]
-
-### Test Design Problems (N items)
-- Item Z: [what assumption was incorrect]
-
-### KB/Retrieval Problems (N items)
-- Item W: [what retrieval issue occurred]
-```
-
-## Step 6: Recommendations
-
-Prioritize fixes:
-
-- **P1 — Evaluation fixes** (false failures inflate failure count, masking real issues): specific criteria rewrites or rubric rule adjustments
-- **P2 — Agent investigation** (real bugs): which code paths or prompt sections to examine
-- **P3 — KB/Retrieval fixes**: which knowledge bases need content updates or which retrieval configs need tuning
-
-For each recommendation, be specific: quote the current criterion/rule, explain what's wrong, and propose the exact replacement text.
-
-## Step 7: Offer to Execute Fixes
-
-Ask the user if they want to apply the evaluation fixes. If yes:
-
-**Update test set criteria** (creates a new version, non-destructive):
-```bash
-# Save the updated test set to a JSON file, then:
-indemn testset update <test-set-id> --file updated-testset.json
-```
-
-**Update rubric rules** (creates a new version, non-destructive):
-```bash
-# Save the updated rubric to a JSON file, then:
-indemn rubric update <rubric-id> --file updated-rubric.json
-```
-
-Both update commands create new versions — the previous version is preserved and accessible via `indemn testset versions <id>` / `indemn rubric versions <id>`.
-
-After applying fixes, suggest re-running the evaluation:
-```bash
-indemn eval run <agent-id> --test-set <test-set-id> --rubric <rubric-id> --wait
-```
-
-## Export: Conversation Traces
-
-When the user wants to **share results** or **export conversation traces** (rather than triage failures):
-
-### Markdown Export
-```bash
-indemn eval export <run-id>
-# Optional: specify output path
-indemn eval export <run-id> --output results.md
-```
-
-Produces a structured markdown file with all conversations, criteria scores, and rubric scores.
-
-### PDF Report
-```bash
-indemn eval report <run-id>
-# Optional: specify output path
-indemn eval report <run-id> --output report.pdf
-```
-
-Produces a branded Indemn PDF with cover page, dual scores (criteria + rubric), and per-item detail pages.
-
-### Agent Card
-```bash
-indemn agents card <agent-id>
-```
-
-Produces a multi-page branded PDF with system prompt, tools, KBs, LLM config, evaluation history, and metadata.

package/skills/eval-orchestration/SKILL.md

@@ -1,257 +0,0 @@
----
-name: eval-orchestration
-description: Evaluation workflow using the indemn CLI — create rubrics, test sets, run evaluations, and analyze results. Use for any evaluation task on Indemn agents.
----
-
-# Evaluation Orchestration
-
-You guide evaluation workflows for AI agents using the `indemn` CLI. This skill covers the full lifecycle: creating evaluation materials, running evaluations, and analyzing results.
-
-**Prerequisite:** The `indemn` CLI must be installed and authenticated. Run `indemn whoami` to verify.
-
----
-
-## Mode Detection
-
-Check the FIRST LINE of the user's message:
-
-**HEADLESS** — Execute autonomously without asking questions. Optimize for completion.
-
-**INTERACTIVE** — Get explicit approval before every action that creates, modifies, or triggers anything.
-
-### Interactive Mode Approval Gates
-
-1. Before creating anything → describe what you plan to do, ask "Should I proceed?"
-2. After creation → summarize what was created, ask "Want to review or should I continue?"
-3. Before triggering an evaluation → confirm test set + rubric selection, ask "Ready to run?"
-4. Before modifying anything → explain the change, ask "OK to update?"
-
----
-
-## Evaluation Harness Overview
-
-The harness tests conversational AI agents against quality standards:
-
-1. **Sends test items** to the agent (single-turn messages or multi-turn scenarios)
-2. **Collects responses** (single response or full conversation transcript)
-3. **Scores responses** against per-item success criteria and universal rubric rules
-4. **Reports results** with pass/fail rates and detailed feedback
-
-### Two Evaluation Layers
-
-| Layer | Source | Applies To | Purpose |
-|-------|--------|------------|---------|
-| **Success Criteria** | Embedded in each test item | That specific item | Did the agent handle THIS situation correctly? |
-| **Rubric Rules** | Separate rubric document | ALL items universally | Does the agent meet baseline behavioral standards? |
-
-### Three Test Types
-
-| Type | Input | What It Tests |
-|------|-------|---------------|
-| **single_turn** | A single message | Specific directive compliance, edge cases |
-| **scenario** | Persona + initial message + max turns | End-to-end workflow completion (chat agents) |
-| **voice_simulation** | Persona + initial message + max turns (via LiveKit) | End-to-end voice workflow |
-
----
-
-## The Directive Allocation Workflow
-
-**This is the core innovation. Follow this BEFORE creating rubric or test set.**
-
-### Step 1: Fetch Bot Context
-
-```bash
-indemn eval bot-context <agent-id> --json
-```
-
-Extract the system prompt, tools, and knowledge bases.
-
-### Step 2: Parse All Directives
-
-Identify every ALWAYS/NEVER/MUST/CRITICAL directive from the system prompt. List them all.
-
-### Step 3: Apply the 3-Message Universal Applicability Test
-
-For EACH directive, mentally test it against these three messages:
-
-1. **Greeting:** "Hello, I'm interested in wedding insurance"
-2. **Off-topic:** "Can you help me with my taxes?"
-3. **Utility:** "I need to correct my phone number"
-
-Ask: "Would I check this directive against ALL THREE of these messages?"
-
-### Step 4: Classify Each Directive
-
-| Classification | Rule | Where It Goes |
-|---------------|------|---------------|
-| **UNIVERSAL** | Passes the 3-message test | Rubric rule |
-| **WORKFLOW-SPECIFIC** | Only applies in certain situations | Test set scenario success criteria |
-| **AMBIGUOUS** | Unclear whether universal | Default to scenario criteria |
-
-### Step 5: Produce Allocation Table
-
-```
-| Directive | Classification | Destination |
-|-----------|---------------|-------------|
-| "ALWAYS maintain professional tone" | UNIVERSAL | Rubric rule |
-| "MUST collect DOB before quoting" | WORKFLOW-SPECIFIC | Scenario criteria |
-| "NEVER provide medical advice" | UNIVERSAL | Rubric rule |
-```
-
-### Step 6: Create Artifacts Using the Allocation
-
-- Read the `rubric-creation` skill and create the rubric using ONLY universal directives
-- Read the `test-set-creation` skill and create the test set, mapping workflow-specific directives as scenario success criteria
-
----
-
-## Workflows
-
-### Workflow A: New Evaluation
-
-1. **Gather intent** — Which agent? Any focus areas?
-2. **Fetch bot context:**
-   ```bash
-   indemn eval bot-context <agent-id> --json
-   ```
-3. **Run directive allocation** — Follow the workflow above
-4. **Create rubric** — Write rubric JSON to a temp file, then:
-   ```bash
-   indemn rubric create --name "<name>" --agent-id <agent-id> --file /tmp/rubric.json
-   ```
-5. **Create test set** — Write test set JSON to a temp file, then:
-   ```bash
-   indemn testset create --name "<name>" --agent-id <agent-id> --file /tmp/testset.json
-   ```
-6. **Review** — Summarize what was created (interactive: wait for approval)
-7. **Trigger evaluation:**
-   ```bash
-   indemn eval run <agent-id> --test-set <test-set-id> --rubric <rubric-id> --wait
-   ```
-   The `--wait` flag polls until completion and shows results.
-   **CRITICAL:** For voice_simulation items, add `--concurrency 1`.
-8. **Analyze results:**
-   ```bash
-   indemn eval results <run-id> --json
-   ```
-
-### Workflow B: Browse Existing Resources
-
-```bash
-indemn rubric list [--agent-id <id>]
-indemn testset list [--agent-id <id>]
-indemn eval status <run-id>
-```
-
-### Workflow C: Update Existing Resources
-
-1. Get the existing resource:
-   ```bash
-   indemn rubric get <id> --json
-   indemn testset get <id> --json
-   ```
-2. Write updated JSON to a file
-3. Send the update:
-   ```bash
-   indemn rubric update <id> --file /tmp/rubric-updated.json
-   indemn testset update <id> --file /tmp/testset-updated.json
-   ```
-
-### Workflow D: Re-run with Existing Resources
-
-1. List existing resources: `indemn rubric list`, `indemn testset list`
-2. Trigger evaluation:
-   ```bash
-   indemn eval run <agent-id> --test-set <id> --rubric <id> --wait
-   ```
-
-### Workflow E: Analyze Results
-
-```bash
-indemn eval results <run-id> --json
-```
-
-Analyze patterns:
-- **By test item**: Which items failed? Check criteria_scores and rubric_scores
-- **By criteria**: Patterns in which criteria fail across items
-- **By rubric rule**: Which universal rules have lowest pass rates?
-- **By test type**: Do scenarios fail more than single-turn?
-
-Suggest improvements: system prompt changes, tool configuration, KB updates.
-
----
-
-## CLI Commands Used
-
-| Command | Purpose |
-|---------|---------|
-| `indemn eval bot-context <id>` | Get bot configuration (prompt, tools, KBs) |
-| `indemn config get <id> --field prompt` | Get just the system prompt |
-| `indemn functions list <id>` | List agent's functions |
-| `indemn rubric create --file` | Create a rubric from JSON |
-| `indemn rubric get <id>` | Get rubric details |
-| `indemn rubric list` | List rubrics |
-| `indemn rubric update <id> --file` | Update a rubric |
-| `indemn rubric delete <id>` | Delete a rubric |
-| `indemn testset create --file` | Create a test set from JSON |
-| `indemn testset get <id>` | Get test set details |
-| `indemn testset list` | List test sets |
-| `indemn testset update <id> --file` | Update a test set |
-| `indemn testset delete <id>` | Delete a test set |
-| `indemn eval run <id> --test-set --rubric` | Trigger an evaluation |
-| `indemn eval status <run-id>` | Get run status |
-| `indemn eval results <run-id>` | Get detailed results |
-
----
-
-## Rubric JSON Format
-
-```json
-{
-  "rules": [
-    {
-      "id": "snake_case_id",
-      "name": "Human Readable Name",
-      "severity": "high|medium|low",
-      "category": "accuracy|persona|instruction_compliance|safety|response_quality",
-      "description": "What this rule checks and why",
-      "evaluation_criteria": {
-        "pass_conditions": ["Observable behavior that passes"],
-        "fail_conditions": ["Observable behavior that fails"]
-      }
-    }
-  ]
-}
-```
-
-## Test Set JSON Format
-
-```json
-{
-  "items": [
-    {
-      "type": "single_turn",
-      "name": "Test name",
-      "inputs": { "message": "User message" },
-      "expected": { "success_criteria": ["Criterion 1", "Criterion 2"] },
-      "tags": ["category"],
-      "priority": "high|medium|low"
-    },
-    {
-      "type": "scenario",
-      "name": "Scenario name",
-      "inputs": {
-        "persona": "You are a [type] who [traits].",
-        "initial_message": "Natural first message",
-        "max_turns": 5
-      },
-      "expected": {
-        "success_criteria": ["Criterion 1"],
-        "expected_outcome": "What the conversation should achieve"
-      },
-      "tags": ["scenario"],
-      "priority": "high"
-    }
-  ]
-}
-```