@indemn/cli 1.6.0 → 1.7.1

package/skills/evaluations/SKILL.md

@@ -9,6 +9,8 @@ Complete evaluation workflow for Indemn AI agents. Covers rubric creation, test
 
 **Prerequisite:** `indemn whoami` must succeed. The `indemn` CLI must be installed and authenticated.
 
+**Talking to people:** call rubrics, test sets and runs by their names; show ids only when asked. Give the verdict and the main reason in plain words first, details on request.
+
 ---
 
 ## Mode Detection
@@ -121,35 +123,7 @@ NEVER exceed 6 rules. A tight rubric with precise universal rules is better than
 
 ### Rubric JSON Format
 
-```json
-{
-  "rules": [
-    {
-      "id": "SNAKE_CASE_ID",
-      "name": "Human-readable name",
-      "severity": "high | medium | low",
-      "category": "persona | instruction_compliance | safety | response_quality",
-      "description": "What this rule checks and why it matters",
-      "component_scope": "prompt | general | function | knowledge_base",
-      "component_ids": null,
-      "component_names": null,
-      "evaluation_criteria": {
-        "pass_conditions": ["Observable condition that indicates compliance"],
-        "fail_conditions": ["Observable condition that indicates violation"]
-      }
-    }
-  ]
-}
-```
-
-**component_scope values:**
-
-| Scope | Meaning | Example |
-|-------|---------|---------|
-| `prompt` | Derived from a system prompt directive | "NEVER provide medical advice" |
-| `general` | Standard quality/safety baseline | No hallucination, response coherence |
-| `function` | Informed by the agent's tool capabilities | "Does not claim capabilities it doesn't have" |
-| `knowledge_base` | Informed by the agent's KBs | "Grounds information rather than fabricating" |
+See `references/json-formats.md` (rubric and test set bodies).
 
 ### Severity Assignment
 
@@ -204,70 +178,9 @@ Aim for roughly **15 items total**. A focused test set with 12-18 precise items
    indemn testset create --name "<Agent Name> Test Set" --agent-id <agent-id> --file /tmp/testset.json
    ```
 
-### Test Set JSON: Scenario
-
-```json
-{
-  "type": "scenario",
-  "name": "Descriptive name - what this tests",
-  "inputs": {
-    "persona": "You are a [type] who [traits, emotional state, behavior pattern, goal].",
-    "initial_message": "Natural first message matching the persona",
-    "max_turns": 8
-  },
-  "expected": {
-    "success_criteria": ["Criterion about the conversation outcome"],
-    "expected_outcome": "What the conversation should achieve"
-  },
-  "tags": ["scenario", "category"],
-  "priority": "high"
-}
-```
-
-### Test Set JSON: Voice Simulation
-
-Identical to scenario except `"type": "voice_simulation"`. The harness routes these through LiveKit with TTS/STT.
-
-```json
-{
-  "type": "voice_simulation",
-  "name": "Descriptive name - what this tests",
-  "inputs": {
-    "persona": "You are a caller who [traits]. You are on a phone call -- keep responses brief and natural.",
-    "initial_message": "Natural first utterance",
-    "max_turns": 6
-  },
-  "expected": {
-    "success_criteria": ["Criterion about the conversation outcome"],
-    "expected_outcome": "What the call should achieve"
-  },
-  "tags": ["voice_simulation", "category"],
-  "priority": "high"
-}
-```
-
-Voice-specific: personas should behave like phone callers (shorter utterances, less formality). max_turns generally shorter (4-6 simple, 6-8 complex). Do NOT create single_turn items for voice agents.
-
-### Test Set JSON: Single Turn
-
-**Single-turn tests start a BRAND NEW conversation with NO prior context.** Design messages that are standalone.
-
-```json
-{
-  "type": "single_turn",
-  "name": "Descriptive name - what this tests",
-  "inputs": {
-    "message": "The standalone message to send"
-  },
-  "expected": {
-    "success_criteria": ["Specific observable criterion"]
-  },
-  "tags": ["category"],
-  "priority": "high"
-}
-```
+### Test Set JSON
 
-**CRITICAL:** The `inputs` object for single-turn items contains ONLY `message`. No `max_turns`, `persona`, or `initial_message`.
+Scenario, voice simulation and single-turn item bodies are in `references/json-formats.md`.
 
 ### Success Criteria Rules
 
@@ -294,139 +207,37 @@ Scenarios start as fresh conversations with no pre-existing session state. Do NO
 
 ## Running Evaluations
 
-### Trigger a Run
+Runs cost model calls and cannot be cancelled, so `indemn eval run` asks first; scripts and agents pass `--yes` only after the user agreed. The agent must be in the org you act on, and the test set must belong to that agent (`--allow-other-agent` overrides).
 
 ```bash
-indemn eval run <agent-id> --test-set <id> --rubric <id> --wait
+indemn eval run <agent-id> --test-set <id> --rubric <id> --wait --yes
 ```
 
-The `--wait` flag polls until completion and shows a pass/fail summary.
-
-**Two models are involved in evaluations:**
-- **Agent model** — set via `indemn config set <id> --model`. This is the LLM the agent uses to generate responses during the eval.
-- **Judge model** — set via `--eval-model` below. This is the LLM that scores the agent's responses against rubric rules and success criteria.
-
-**Key flags:**
-
-| Flag | Purpose | Notes |
-|------|---------|-------|
-| `--wait` | Poll until complete, show summary | Recommended for all runs |
-| `--concurrency <n>` | Parallel items | Use `1` for voice_simulation |
-| `--eval-model <model>` | Judge LLM (e.g., "gpt-4o") | Scores responses; does NOT change the agent's model |
-| `--keep` | Preserve intermediate artifacts | Useful for debugging |
-| `--limit <n>` | Run only first N items | Good for quick smoke tests |
-| `--component-scope <scope>` | Filter by component scope | prompt, function, knowledge_base, general |
-| `--component-ids <ids>` | Filter by component IDs | Comma-separated |
-| `--conversation-ids <ids>` | Filter by conversation IDs | Comma-separated |
-| `--mode <mode>` | Evaluation mode | Use "transcript" for transcript-based evaluation |
+**Two models are involved:** the agent model (`indemn config set <id> --model provider/model`) answers; the judge model (`--eval-model`) scores.
 
-### Check Status
+| Flag | Purpose |
+|------|---------|
+| `--wait` | Poll until complete and show pass rate, errors and p95 latency (`--timeout` seconds, default 1800) |
+| `--concurrency <n>` | Parallel items; use `1` for voice_simulation |
+| `--eval-model <model>` | Judge LLM; does not change the agent |
+| `--limit <n>` | Only the first N items (smoke test) |
+| `--component-scope`, `--component-ids` | Only rules and items for those components |
+| `--mode transcript` with `--conversation-ids` or `--date-from/--date-to` | Grade real conversations instead of the test set (a date range turns transcript mode on) |
+| `--min-pass-rate <0..1>`, `--max-p95-ms <ms>`, `--fail-on-error`, `--error-policy block\|warn` | CI gate: exit 1 when the run misses the bar (same rules as the evaluations repo's gate script) |
+| `--json` | Final run (and gate verdict) as JSON |
 
 ```bash
 indemn eval status <run-id> [--json]
-```
-
-### List Runs
-
-```bash
-indemn eval list [--agent-id <id>] [--json] [--limit <n>] [--page <n>]
+indemn eval list --agent-id <agent-id> [--json] [--limit <n>]
+indemn eval trends --agent <agent-id> [--test-set <id>]      # pass rate over time, drift warning
+indemn eval compare <run-a> <run-b>
 ```
 
 ---
 
 ## Analyzing Results
 
-### Step 1: Resolve the Run
-
-- **UUID format** (contains hyphens, 36 chars) -- it is a `run_id`. Use directly.
-- **ObjectId format** (24 hex chars) -- it is an `agent_id`. Fetch the latest completed run:
-  ```bash
-  indemn eval list --agent-id <agent-id> --json --limit 5
-  ```
-
-### Step 2: Pull All Data
-
-```bash
-indemn eval status <run-id> --json
-indemn eval results <run-id> --json
-indemn rubric get <rubric-id> --json
-indemn testset get <test-set-id> --json
-indemn eval bot-context <agent-id> --json
-```
-
-For exact response shapes, see `references/data-shapes.md`.
-
-### Step 3: Present Run Overview
-
-```
-Run: {run_id}
-Agent: {bot_name} ({agent_id})
-LLM: {bot_llm_provider}/{bot_llm_model}
-Status: {passed}/{total} items passed ({pass_rate}%)
-
-Criteria: {criteria_passed}/{criteria_total} checks passed
-Rubric: {rubric_rules_passed}/{rubric_rules_total} rules passed
-
-Component Scores:
-  prompt:         {score}% ({passed}/{total})
-  knowledge_base: {score}% ({passed}/{total})
-  function:       {score}% ({passed}/{total})
-  general:        {score}% ({passed}/{total})
-```
-
-### Step 4: Analyze Each Failed Item
-
-For every result where `passed == false`:
-
-1. **Get the test item** -- match `test_case_id` to the test set's `items[].item_id` for name, type, inputs, success_criteria
-2. **Check criteria scores** -- from `criteria_scores[]`, list which passed/failed with reasoning
-3. **Check rubric scores** -- from `rubric_scores[]`, list which passed/failed with reasoning
-4. **Check for divergence** -- flag when `criteria_passed != rubric_passed`. Divergence is a strong signal of an evaluation problem.
-5. **Read the conversation** -- look at `output` field for the full response or transcript
-6. **Cross-reference system prompt** -- check if the agent's behavior aligns with prompt instructions
-7. **Classify** into one of four categories:
-
-| Category | Signal | Action |
-|----------|--------|--------|
-| **Real Agent Problem** | Both layers agree agent misbehaved, or output contradicts prompt | Investigate agent code/prompt |
-| **Evaluation Problem** | Criteria ambiguous/too strict, evaluator reasoning shows misunderstanding | Update test set criteria |
-| **Test Design Problem** | Test assumptions wrong (e.g., assumed KB lacked info it has) | Redesign test item |
-| **KB/Retrieval Problem** | Agent behavior correct but retrieval returned wrong content | Fix KB content or retrieval config |
-
-### Divergence Analysis
-
-When `criteria_passed != rubric_passed` for an item, this is a first-class finding:
-
-- **Criteria FAIL + Rubric PASS** -- criteria are likely too strict or testing the wrong thing
-- **Criteria PASS + Rubric FAIL** -- rubric may have rules that don't apply to this scenario, or agent violated a behavioral rule despite answering correctly
-
-Always highlight divergences prominently.
-
-### Step 5: Aggregate and Recommend
-
-Group failures by category. Prioritize fixes:
-
-- **P1 -- Evaluation fixes** (false failures mask real issues): specific criteria rewrites or rubric rule adjustments
-- **P2 -- Agent investigation** (real bugs): which code paths or prompt sections to examine
-- **P3 -- KB/Retrieval fixes**: which KBs need content updates or retrieval tuning
-
-For each recommendation, quote the current criterion/rule, explain what is wrong, and propose the exact replacement text.
-
-### Step 6: Apply Fixes
-
-Updates create new versions (non-destructive, previous versions preserved):
-
-```bash
-indemn testset update <test-set-id> --file updated-testset.json
-indemn rubric update <rubric-id> --file updated-rubric.json
-```
-
-Check version history: `indemn testset versions <id>` / `indemn rubric versions <id>`
-
-After applying fixes, re-run:
-```bash
-indemn eval run <agent-id> --test-set <id> --rubric <id> --wait
-```
+Follow `references/analysis.md`: resolve the run, pull results (`get_eval_results` is concise by default; `failed_only` narrows it), present the overview, analyse each failed item, then recommend and apply fixes.
 
 ---
 
@@ -488,26 +299,4 @@ Produces a branded Indemn PDF with cover page, dual scores (criteria + rubric),
 
 ## CLI Reference
 
-| Command | Purpose |
-|---------|---------|
-| `indemn eval bot-context <id> [--json]` | Get agent prompt, tools, KBs, LLM config |
-| `indemn eval run <id> --test-set <id> [--rubric <id>] [--wait]` | Trigger an evaluation run |
-| `indemn eval list [--agent-id <id>] [--json] [--limit] [--page]` | List evaluation runs |
-| `indemn eval status <run-id> [--json]` | Get run status and summary |
-| `indemn eval results <run-id> [--json]` | Get per-item detailed results |
-| `indemn eval export <run-id> [--output <path>]` | Export results as markdown |
-| `indemn eval report <run-id> [--output <path>]` | Export results as branded PDF |
-| `indemn rubric create [--file <path>] [--name "<name>"] [--agent-id <id>]` | Create rubric (--file or --name required) |
-| `indemn rubric get <id> [--json] [--revision <n>]` | Get rubric details |
-| `indemn rubric list [--agent-id <id>] [--json] [--limit] [--page]` | List rubrics |
-| `indemn rubric update <id> [--file <path>] [--name "<name>"]` | Update rubric (creates new version) |
-| `indemn rubric delete <id>` | Delete rubric (requires confirmation) |
-| `indemn rubric versions <id> [--json]` | List rubric version history |
-| `indemn testset create [--file <path>] [--name "<name>"] [--agent-id <id>]` | Create test set (--file or --name required) |
-| `indemn testset get <id> [--json] [--revision <n>]` | Get test set details |
-| `indemn testset list [--agent-id <id>] [--json] [--limit] [--page]` | List test sets (always use --agent-id) |
-| `indemn testset update <id> [--file <path>] [--name "<name>"]` | Update test set (creates new version) |
-| `indemn testset delete <id>` | Delete test set (requires confirmation) |
-| `indemn testset versions <id> [--json]` | List test set version history |
-| `indemn config get <id> --field prompt` | Get just the system prompt |
-| `indemn functions list <id>` | List agent's functions |
+Every evaluation command is listed in `references/cli-reference.md`.

package/skills/evaluations/references/analysis.md

@@ -0,0 +1,95 @@
+## Analyzing Results
+
+### Step 1: Resolve the Run
+
+- **UUID format** (contains hyphens, 36 chars) -- it is a `run_id`. Use directly.
+- **ObjectId format** (24 hex chars) -- it is an `agent_id`. Fetch the latest completed run:
+  ```bash
+  indemn eval list --agent-id <agent-id> --json --limit 5
+  ```
+
+### Step 2: Pull All Data
+
+```bash
+indemn eval status <run-id> --json
+indemn eval results <run-id> --json
+indemn rubric get <rubric-id> --json
+indemn testset get <test-set-id> --json
+indemn eval bot-context <agent-id> --json
+```
+
+For exact response shapes, see `references/data-shapes.md`.
+
+### Step 3: Present Run Overview
+
+```
+Run: {run_id}
+Agent: {bot_name} ({agent_id})
+LLM: {bot_llm_provider}/{bot_llm_model}
+Status: {passed}/{total} items passed ({pass_rate}%)
+
+Criteria: {criteria_passed}/{criteria_total} checks passed
+Rubric: {rubric_rules_passed}/{rubric_rules_total} rules passed
+
+Component Scores:
+  prompt:         {score}% ({passed}/{total})
+  knowledge_base: {score}% ({passed}/{total})
+  function:       {score}% ({passed}/{total})
+  general:        {score}% ({passed}/{total})
+```
+
+### Step 4: Analyze Each Failed Item
+
+For every result where `passed == false`:
+
+1. **Get the test item** -- match `test_case_id` to the test set's `items[].item_id` for name, type, inputs, success_criteria
+2. **Check criteria scores** -- from `criteria_scores[]`, list which passed/failed with reasoning
+3. **Check rubric scores** -- from `rubric_scores[]`, list which passed/failed with reasoning
+4. **Check for divergence** -- flag when `criteria_passed != rubric_passed`. Divergence is a strong signal of an evaluation problem.
+5. **Read the conversation** -- look at `output` field for the full response or transcript
+6. **Cross-reference system prompt** -- check if the agent's behavior aligns with prompt instructions
+7. **Classify** into one of four categories:
+
+| Category | Signal | Action |
+|----------|--------|--------|
+| **Real Agent Problem** | Both layers agree agent misbehaved, or output contradicts prompt | Investigate agent code/prompt |
+| **Evaluation Problem** | Criteria ambiguous/too strict, evaluator reasoning shows misunderstanding | Update test set criteria |
+| **Test Design Problem** | Test assumptions wrong (e.g., assumed KB lacked info it has) | Redesign test item |
+| **KB/Retrieval Problem** | Agent behavior correct but retrieval returned wrong content | Fix KB content or retrieval config |
+
+### Divergence Analysis
+
+When `criteria_passed != rubric_passed` for an item, this is a first-class finding:
+
+- **Criteria FAIL + Rubric PASS** -- criteria are likely too strict or testing the wrong thing
+- **Criteria PASS + Rubric FAIL** -- rubric may have rules that don't apply to this scenario, or agent violated a behavioral rule despite answering correctly
+
+Always highlight divergences prominently.
+
+### Step 5: Aggregate and Recommend
+
+Group failures by category. Prioritize fixes:
+
+- **P1 -- Evaluation fixes** (false failures mask real issues): specific criteria rewrites or rubric rule adjustments
+- **P2 -- Agent investigation** (real bugs): which code paths or prompt sections to examine
+- **P3 -- KB/Retrieval fixes**: which KBs need content updates or retrieval tuning
+
+For each recommendation, quote the current criterion/rule, explain what is wrong, and propose the exact replacement text.
+
+### Step 6: Apply Fixes
+
+Updates create new versions (non-destructive, previous versions preserved):
+
+```bash
+indemn testset update <test-set-id> --file updated-testset.json
+indemn rubric update <rubric-id> --file updated-rubric.json
+```
+
+Check version history: `indemn testset versions <id>` / `indemn rubric versions <id>`
+
+After applying fixes, re-run:
+```bash
+indemn eval run <agent-id> --test-set <id> --rubric <id> --wait
+```
+
+---

package/skills/evaluations/references/cli-reference.md

@@ -0,0 +1,28 @@
+# Evaluations CLI reference
+
+All commands act on the org chosen by `--org`, the linked folder, or your only org. Agents of other orgs are refused.
+
+| Command | What it does |
+|---|---|
+| `indemn eval run <agent-id> --test-set <id> [--rubric <id>] [--wait] [--yes]` | Start a run (asks first: paid, cannot be cancelled). CI gate flags: `--min-pass-rate`, `--max-p95-ms`, `--fail-on-error`, `--error-policy` |
+| `indemn eval status <run-id> [--json]` | Status and counts |
+| `indemn eval results <run-id> [--json]` | Per-item results |
+| `indemn eval list --agent-id <id> [--json] [--limit n]` | Runs of one agent |
+| `indemn eval trends --agent <id> [--test-set <id>]` | Pass rate and p95 over runs; drift when the pass rate fell by 0.1 or more |
+| `indemn eval compare <run-a> <run-b>` | Pass rate and p95 differences |
+| `indemn eval bot-context <agent-id>` | What the evaluator sees: prompt, skills, knowledge bases, model |
+| `indemn eval export <run-id> [--output file.md]` | Markdown export |
+| `indemn eval report <run-id> [--output file.pdf] [--server [--force]]` | PDF report, made locally or by the evaluations service |
+| `indemn eval report-link <report-id>` | Fresh download link for a server report |
+| `indemn eval metrics` | Voice quality metric names for `expected.voice.required_metrics` |
+| `indemn eval transcript --agent <id> --baseline a.json --replay b.json` | Grade a replayed conversation against a baseline |
+| `indemn eval prod-calls list --agent <id> [--search text]` | Ingested production voice calls |
+| `indemn eval prod-calls promote <session-id>` | Add a call to the agent's "Regressions (review)" test set |
+| `indemn eval datasets list --agent <id>` / `datasets cases <dataset-id>` | Low-level datasets the trigger creates |
+| `indemn eval evaluators types` / `evaluators list --agent <id>` / `evaluators delete <config-id>` | Judge types and stored evaluator configs |
+| `indemn rubric list\|get\|create\|update\|delete\|versions` | Rubrics (each update is a new version) |
+| `indemn testset list\|get\|create\|update\|delete\|versions` | Test sets (each update is a new version) |
+| `indemn conversations export <id> --agent <agent-id> --as-testset -o set.json` | Turn a real conversation into a scenario test item, then `indemn testset create --file set.json` |
+| `indemn jarvis jobs create --bots <ids> --wait` | Ask Jarvis to generate baseline rubrics and test sets |
+
+MCP equivalents: `run_evaluation` (asks first), `get_eval_status`, `get_eval_results` (`response_format`, `failed_only`), `eval_gate_verdict`, `eval_trends`, `compare_eval_runs`, `eval_report_link`, `eval_quality_metrics`, `list_prod_calls`, `promote_prod_call`, `evaluate_transcript`, `list_evaluator_configs`, the rubric and test-set tools, and `jarvis_create_job`.

package/skills/evaluations/references/json-formats.md

@@ -0,0 +1,100 @@
+# Rubric and test set JSON formats
+
+Used by `indemn rubric create --file` and `indemn testset create --file`. Field-level shapes are in `data-shapes.md`.
+
+#### Rubric JSON Format
+
+```json
+{
+  "rules": [
+    {
+      "id": "SNAKE_CASE_ID",
+      "name": "Human-readable name",
+      "severity": "high | medium | low",
+      "category": "persona | instruction_compliance | safety | response_quality",
+      "description": "What this rule checks and why it matters",
+      "component_scope": "prompt | general | function | knowledge_base",
+      "component_ids": null,
+      "component_names": null,
+      "evaluation_criteria": {
+        "pass_conditions": ["Observable condition that indicates compliance"],
+        "fail_conditions": ["Observable condition that indicates violation"]
+      }
+    }
+  ]
+}
+```
+
+**component_scope values:**
+
+| Scope | Meaning | Example |
+|-------|---------|---------|
+| `prompt` | Derived from a system prompt directive | "NEVER provide medical advice" |
+| `general` | Standard quality/safety baseline | No hallucination, response coherence |
+| `function` | Informed by the agent's tool capabilities | "Does not claim capabilities it doesn't have" |
+| `knowledge_base` | Informed by the agent's KBs | "Grounds information rather than fabricating" |
+
+#### Test Set JSON: Scenario
+
+```json
+{
+  "type": "scenario",
+  "name": "Descriptive name - what this tests",
+  "inputs": {
+    "persona": "You are a [type] who [traits, emotional state, behavior pattern, goal].",
+    "initial_message": "Natural first message matching the persona",
+    "max_turns": 8
+  },
+  "expected": {
+    "success_criteria": ["Criterion about the conversation outcome"],
+    "expected_outcome": "What the conversation should achieve"
+  },
+  "tags": ["scenario", "category"],
+  "priority": "high"
+}
+```
+
+#### Test Set JSON: Voice Simulation
+
+Identical to scenario except `"type": "voice_simulation"`. The harness routes these through LiveKit with TTS/STT.
+
+```json
+{
+  "type": "voice_simulation",
+  "name": "Descriptive name - what this tests",
+  "inputs": {
+    "persona": "You are a caller who [traits]. You are on a phone call -- keep responses brief and natural.",
+    "initial_message": "Natural first utterance",
+    "max_turns": 6
+  },
+  "expected": {
+    "success_criteria": ["Criterion about the conversation outcome"],
+    "expected_outcome": "What the call should achieve"
+  },
+  "tags": ["voice_simulation", "category"],
+  "priority": "high"
+}
+```
+
+Voice-specific: personas should behave like phone callers (shorter utterances, less formality). max_turns generally shorter (4-6 simple, 6-8 complex). Do NOT create single_turn items for voice agents.
+
+#### Test Set JSON: Single Turn
+
+**Single-turn tests start a BRAND NEW conversation with NO prior context.** Design messages that are standalone.
+
+```json
+{
+  "type": "single_turn",
+  "name": "Descriptive name - what this tests",
+  "inputs": {
+    "message": "The standalone message to send"
+  },
+  "expected": {
+    "success_criteria": ["Specific observable criterion"]
+  },
+  "tags": ["category"],
+  "priority": "high"
+}
+```
+
+**CRITICAL:** The `inputs` object for single-turn items contains ONLY `message`. No `max_turns`, `persona`, or `initial_message`.