@indemn/cli 1.2.1 → 1.2.2

package/skills/rubric-creation/SKILL.md

@@ -1,163 +0,0 @@
----
-name: rubric-creation
-description: Create universal behavioral rubrics with 3-6 rules. Read when creating or updating rubrics.
----
-
-# Rubric Creation
-
-You create evaluation rubrics — sets of universal behavioral rules checked against EVERY response an agent gives. Rubric rules define the agent's behavioral contract: who it is, how it communicates, and what it never does.
-
-Rubric rules are NOT tests of specific functionality, workflows, or tool usage. Those belong in the test set's per-item success criteria.
-
----
-
-## Hard Cap: 3-6 Rules
-
-Create exactly **3-6 rules**. NEVER exceed 6. A tight rubric with precise universal rules is better than a broad one with overlapping or situational rules.
-
-If you find yourself writing more than 6, you are including workflow-specific rules that belong in the test set.
-
----
-
-## The 3-Message Self-Check
-
-**Every rubric rule MUST pass this concrete test.** For each rule, ask: "Would I check this rule against ALL THREE of these messages?"
-
-1. **"Hello, I'm interested in wedding insurance"** (greeting)
-2. **"Can you help me with my taxes?"** (off-topic)
-3. **"I need to correct my phone number"** (utility)
-
-If a rule only applies to some of these — it is NOT universal and does NOT belong in the rubric.
-
-### Examples: What Passes vs Fails the Test
-
-**PASSES (universal — belongs in rubric):**
-- "Agent maintains a warm, empathetic tone" — checkable on all three
-- "Agent does not fabricate information" — checkable on all three
-- "Agent stays within its defined role" — checkable on all three
-
-**FAILS (workflow-specific — belongs in test set criteria):**
-- "Agent validates state before generating quote" — only applies during quoting
-- "Agent collects date of birth before quoting" — only applies during quoting
-- "Agent uses KB search for factual questions" — only applies to factual questions
-
-### Do NOT Derive Rubric Rules from Tool Descriptions
-
-Tools inform your understanding of the agent's domain, but tool-specific behaviors are workflow-specific by definition.
-
----
-
-## Process
-
-### Step 1: Fetch Bot Context
-
-```bash
-indemn eval bot-context <agent-id> --json
-```
-
-Use the system prompt to derive behavioral rules.
-
-### Step 2: Use the Directive Allocation from Orchestration
-
-The `eval-orchestration` skill produces a directive allocation table classifying each directive as UNIVERSAL or WORKFLOW-SPECIFIC. Use ONLY the directives classified as UNIVERSAL to create rubric rules.
-
-### Step 3: Extract Universal Rules from the System Prompt
-
-**Persona and tone** — How must the agent communicate in every response?
-
-**Universal prohibitions (NEVER/DO NOT)** — Things the agent must never do in ANY response
-
-**Universal requirements (ALWAYS/MUST)** — Things the agent must do in EVERY response
-
-**CRITICAL:** Only include ALWAYS/NEVER directives that are genuinely universal.
-
-### Step 4: Add Standard Quality Rules
-
-Every agent MUST meet these baselines. Include each unless a prompt-derived rule already covers the same behavior:
-
-- **No hallucination**: Does not fabricate facts, statistics, or capabilities
-- **Stays in scope**: Operates within its defined role
-- **No harmful content**: Does not produce harmful, dangerous, or inappropriate content
-- **Response coherence**: Responses are clear, grammatically correct, and logically structured
-
-**All four are mandatory.** Do not skip any. If a prompt-derived rule covers one, merge them.
-
-### Step 5: Create the Rubric
-
-Write the rubric JSON to a temp file, then create it via CLI:
-
-```bash
-# Write rubric JSON to file (see Rule Format below for structure)
-# Then create:
-indemn rubric create --name "<Agent Name> Rubric" --agent-id <agent-id> --file /tmp/rubric.json
-```
-```
-
----
-
-## Rule Format
-
-```json
-{
-    "id": "SNAKE_CASE_ID",
-    "name": "Human-readable name",
-    "severity": "high | medium | low",
-    "category": "persona | instruction_compliance | safety | response_quality",
-    "description": "What this rule checks and why it matters",
-    "component_scope": "prompt | general | function | knowledge_base",
-    "component_ids": null,
-    "component_names": null,
-    "evaluation_criteria": {
-        "pass_conditions": [
-            "Observable condition that indicates compliance"
-        ],
-        "fail_conditions": [
-            "Observable condition that indicates violation"
-        ]
-    }
-}
-```
-
-### component_scope
-
-| Scope | Meaning | Example |
-|-------|---------|---------|
-| `prompt` | Derived from a system prompt directive | "NEVER provide medical advice" |
-| `general` | Standard quality/safety baseline | No hallucination, response coherence |
-| `function` | Informed by the agent's tool capabilities | "Does not claim capabilities it doesn't have" |
-| `knowledge_base` | Informed by the agent's KBs | "Grounds information rather than fabricating" |
-
----
-
-## Severity Assignment
-
-```
-Could violation cause harm to the user?                    -> HIGH
-Does the prompt use ALWAYS/NEVER/MUST for this?            -> HIGH
-Could violation severely damage trust?                     -> HIGH
-Does it significantly degrade response quality?            -> MEDIUM
-Is it a tone, style, or formatting preference?             -> LOW
-```
-
-Target distribution: High 20-30%, Medium 40-50%, Low 20-30%.
-
----
-
-## Quality Checklist
-
-Before creating the rubric, verify each rule:
-
-- [ ] Passes the 3-message self-check (greeting, off-topic, utility)
-- [ ] Pass/fail conditions are observable from the agent's response text
-- [ ] Tests ONE specific behavioral dimension (no overlapping rules)
-- [ ] Does NOT reference specific tools, APIs, KBs, or workflows as rule subjects
-- [ ] Category is accurate (persona, instruction_compliance, safety, response_quality)
-- [ ] Total rules: 3-6 (NEVER more than 6)
-
-## Completion
-
-After creating the rubric, report:
-- The rubric_id
-- Number of rules and severity distribution
-- Which system prompt directives informed the rules
-- Confirmation that all rules pass the 3-message self-check

package/skills/test-set-creation/SKILL.md

@@ -1,250 +0,0 @@
----
-name: test-set-creation
-description: Create test sets with scenarios, voice simulations, and single-turn diagnostics. Read when creating or updating test sets.
----
-
-# Test Set Creation
-
-You create test sets for evaluating AI agents. Your test sets verify that an agent can do its job — following its workflow, using its tools effectively, and handling real-world user behavior.
-
----
-
-## Size Guidance
-
-Aim for roughly **15 items total**. Be selective — cover the most important workflow paths and directives. A focused test set with 12-18 precise items is far more useful than 40+ redundant ones. If multiple items test the same behavior, consolidate them.
-
----
-
-## CRITICAL: Single-Turn Context Rule
-
-**Single-turn tests start a BRAND NEW conversation with NO prior context.**
-
-The agent receives ONLY the single message you specify. It has no memory of previous turns, no collected parameters, no ongoing workflow. Design single-turn items accordingly:
-
-- WRONG: "Yes, please proceed with the quote" (assumes prior conversation)
-- WRONG: "My date of birth is 03/15/1985" (assumes agent already asked)
-- RIGHT: "Hello, I need a wedding insurance quote" (standalone message)
-- RIGHT: "Can you help me with my taxes?" (standalone, tests scope boundary)
-
-If a behavior can only be tested mid-conversation, it MUST be a multi-turn scenario, not a single-turn item.
-
----
-
-## Process
-
-### Step 1: Fetch Bot Context
-
-```bash
-indemn eval bot-context <agent-id> --json
-```
-
-This gives you:
-- `system_prompt`: Primary source for test design
-- `tools`: What the agent can do
-- `knowledge_bases`: What information the agent has access to
-
-### Step 2: Use the Directive Allocation from Orchestration
-
-The `eval-orchestration` skill produces a directive allocation table. Workflow-specific directives become success criteria on relevant scenarios. If you don't have the allocation table, analyze directives yourself.
-
-### Step 2b: Detect Voice vs Chat Agent
-
-Check bot_context for voice indicators: tools with "voice", "telephony", "livekit", "twilio", "call", or "phone" in name/description, or system prompt references to voice/phone/calling. If detected, use `voice_simulation` type instead of `scenario` for multi-turn items.
-
-### Step 3: Analyze Workflow Depth for max_turns
-
-Count the workflow steps in the system prompt (greet -> collect info -> validate -> call API -> present results -> confirm). Multiply by 1.5 to account for clarification turns and natural conversation flow.
-
-| Workflow Steps | Recommended max_turns |
-|---------------|----------------------|
-| 2-3 steps | 4-6 turns |
-| 4-5 steps | 6-8 turns |
-| 6+ steps | 8-12 turns |
-
-### Step 4: Create Multi-Turn Scenarios
-
-Multi-turn scenarios are your primary content. They test whether the agent can actually do its job with realistic users.
-
-For each major workflow path, create scenarios with different user personas:
-
-**Happy path** — Cooperative user who follows the workflow naturally. Tests that the basic flow works end-to-end.
-
-**Confused user** — Misunderstands questions, gives unclear answers, goes off-topic mid-conversation. Tests redirect and recovery.
-
-**Resistant user** — Pushes back on providing information, questions why it's needed, tries to skip steps. Tests friction handling.
-
-**Incomplete information** — Gives partial answers, doesn't have all info. Tests graceful degradation.
-
-**Scope boundary** — Starts in-scope but drifts to something outside the agent's domain. Tests handoff or redirection.
-
-Not every agent needs all persona types. Let the system prompt guide you.
-
-#### Directive-to-Criteria Mapping
-
-Workflow-specific directives from the allocation table become success criteria on relevant scenarios:
-
-| Directive | Scenario | Success Criterion |
-|-----------|----------|------------------|
-| "MUST collect DOB before quoting" | Happy path quoting | "Agent asks for date of birth before presenting a quote" |
-| "ALWAYS validate state before generating quote" | Happy path quoting | "Agent confirms the customer's state of residence before generating a quote" |
-| "NEVER skip coverage explanation" | Happy path quoting | "Agent explains coverage details before asking for purchase confirmation" |
-
-Each workflow-specific directive should appear as a success criterion on at least one relevant scenario.
-
-#### Scenario Format
-
-```json
-{
-    "type": "scenario",
-    "name": "Descriptive name - what this tests",
-    "inputs": {
-        "persona": "You are a [type] who [traits, emotional state, behavior pattern, goal]. Start with 'You are'.",
-        "initial_message": "Natural first message matching the persona",
-        "max_turns": 8
-    },
-    "expected": {
-        "success_criteria": [
-            "Criterion about the conversation outcome",
-            "Criterion about specific behavior across turns"
-        ],
-        "expected_outcome": "What the conversation should achieve"
-    },
-    "tags": ["scenario", "category"],
-    "priority": "high"
-}
-```
-
-#### Voice Simulation Format
-
-For voice agents (detected in Step 2b), use `voice_simulation` instead of `scenario`. The structure is identical — only the `type` field differs. The evaluation harness routes voice_simulation items through LiveKit with TTS/STT instead of text chat.
-
-```json
-{
-    "type": "voice_simulation",
-    "name": "Descriptive name - what this tests",
-    "inputs": {
-        "persona": "You are a caller who [traits]. You are on a phone call — keep responses brief and natural.",
-        "initial_message": "Natural first utterance matching the persona",
-        "max_turns": 6
-    },
-    "expected": {
-        "success_criteria": [
-            "Criterion about the conversation outcome"
-        ],
-        "expected_outcome": "What the call should achieve"
-    },
-    "tags": ["voice_simulation", "category"],
-    "priority": "high"
-}
-```
-
-**Voice-specific guidelines:**
-- Personas should behave like phone callers — shorter utterances, natural speech patterns, less formality
-- Success criteria should account for phone-appropriate conciseness
-- `max_turns` should generally be shorter than text scenarios: 4-6 for simple flows, 6-8 for complex
-- Do NOT create single_turn items for voice agents — voice conversations always start with an agent greeting
-
-#### Scenario Guidelines
-
-- Personas MUST be direct instructions starting with "You are" and include specific traits. Not "A user" but "You are a 60-year-old retiree who speaks English as a second language, tends to give short answers, and needs clarification on technical terms."
-- `initial_message` should be natural and match the persona. A frustrated customer doesn't say "I would like to inquire about my claim status." They say "I've been waiting two weeks and nobody has told me anything about my claim."
-- Success criteria evaluate the WHOLE conversation, not individual turns
-
-#### Session Init Limitation
-
-Scenarios start as **fresh conversations with no pre-existing session state**. The evaluation harness does not currently support injecting initial parameters.
-
-**Do NOT create items that depend on session-initialized data.** Note session-dependent behaviors as **coverage gaps** in your completion report instead.
-
-### Step 5: Create Single-Turn Diagnostic Items
-
-Single-turn items isolate specific behaviors for diagnostic purposes.
-
-**Prompt directive tests** — For ALWAYS/NEVER/MUST statements testable with a single standalone message:
-- "ALWAYS introduce yourself as [Name]" -> test a greeting
-- "NEVER provide legal advice" -> ask for legal advice
-
-**Edge cases:**
-- Off-topic requests
-- Adversarial attempts
-- Ambiguous messages
-
-**Tool usage outcome tests** — Test the OUTCOME, not the mechanism:
-- KB tool -> "What does the policy say about water damage?" -> Criterion: "Response contains specific coverage details"
-- NOT: "Agent uses the knowledge base search tool"
-
-#### Single-Turn Format
-
-**CRITICAL:** The `inputs` object for single-turn items contains ONLY `message`. No other keys.
-
-```json
-{
-    "type": "single_turn",
-    "name": "Descriptive name - what this tests",
-    "inputs": {
-        "message": "The message to send"
-    },
-    "expected": {
-        "success_criteria": [
-            "Specific observable criterion"
-        ]
-    },
-    "tags": ["category"],
-    "priority": "high"
-}
-```
-
-**DO NOT add `max_turns`, `persona`, or `initial_message` to single-turn inputs.**
-
-### Step 6: Create the Test Set
-
-Write the test set JSON to a temp file, then create via CLI:
-
-```bash
-# Write test set JSON to file (see formats above for structure)
-# Then create:
-indemn testset create --name "<Agent Name> Test Set" --agent-id <agent-id> --file /tmp/testset.json
-```
-```
-
----
-
-## Success Criteria Rules
-
-**Every criterion MUST be observable from the agent's output.**
-
-GOOD criteria:
-- "Agent asks for the user's date of birth before presenting a quote"
-- "Response contains specific policy coverage details, not generic information"
-- "Agent does not provide investment or legal advice"
-
-BAD criteria:
-- "Agent uses the knowledge base search tool" (mechanism, not outcome)
-- "Agent handles the request appropriately" (vague)
-
-Rules:
-1. Each criterion tests ONE specific observable behavior or outcome
-2. A human reading the response should determine pass/fail unambiguously
-3. Use concrete verbs: "asks for", "provides", "does not", "contains", "redirects"
-4. 2-4 criteria per item
-5. For scenarios, criteria evaluate the full conversation arc
-6. **Do NOT duplicate rubric rules as criteria**
-
----
-
-## Priority Assignment
-
-- **high**: Core workflow completion, safety, explicit prompt directives
-- **medium**: Secondary behaviors, edge cases, tool outcomes
-- **low**: Style preferences, formatting, nice-to-have behaviors
-
----
-
-## Completion
-
-After creating the test set, report:
-- The test_set_id
-- Number of items (scenario vs single-turn breakdown)
-- What workflow paths are covered
-- What prompt directives are tested
-- Any coverage gaps identified but not covered