@opensearch-project/agent-health 0.3.0 → 0.5.2

package/docs/skills/AGENT_HEALTH.md

@@ -0,0 +1,598 @@
+# Agent Health - AI Assistant Instructions
+
+Use these instructions to evaluate and improve your agent using the agent-health CLI and server APIs.
+
+---
+
+## Setup By AI Assistant
+
+Copy this file into your project so your AI coding assistant auto-loads it —
+either from a local checkout or straight from GitHub.
+
+### Claude Code
+```bash
+mkdir -p .claude/skills
+curl -o .claude/skills/agent-health.md \
+  https://raw.githubusercontent.com/opensearch-project/agent-health/main/docs/skills/AGENT_HEALTH.md
+# or, from a checkout: cp docs/skills/AGENT_HEALTH.md .claude/skills/
+```
+Auto-discovered as a skill.
+
+### Kiro
+```bash
+mkdir -p .kiro/steering
+curl -o .kiro/steering/agent-health.md \
+  https://raw.githubusercontent.com/opensearch-project/agent-health/main/docs/skills/AGENT_HEALTH.md
+```
+Auto-loaded as a steering file.
+
+### Cursor / Windsurf / Others
+Copy to project root, then tell your assistant: "Read AGENT_HEALTH.md and follow it"
+
+---
+
+## Quick start — what to ask your assistant
+
+Once this file is loaded, just ask:
+
+```
+"Evaluate my agent using agent-health and fix any failures"
+```
+
+Or step by step:
+
+```
+"Run an agent-health benchmark against my agent"
+"What failed and why?"
+"Fix the high-priority issues"
+"Re-run to verify"
+```
+
+What happens under the hood:
+
+```
+1. Assistant runs: npx agent-health benchmark ...
+        ↓
+2. Gets JSON with failures + judge feedback (matcherResults + improvementStrategies)
+        ↓
+3. Reads your agent code
+        ↓
+4. Makes fixes based on the improvement strategies
+        ↓
+5. Re-runs the benchmark to verify
+```
+
+---
+
+## Prerequisites
+
+### Storage (file-based by default; OpenSearch optional)
+
+Agent Health uses **file-based storage by default** — results are written to a
+local `.agent-health/data/` directory and `run` / `benchmark` work with **no
+OpenSearch cluster required**. Point at OpenSearch only when you want shared,
+production-grade persistence (multiple machines / a team dashboard).
+
+To use OpenSearch instead of file storage, set:
+
+| Variable | Description |
+|---|---|
+| `OPENSEARCH_STORAGE_ENDPOINT` | OpenSearch cluster URL (e.g. `https://search-my-cluster.us-west-2.es.amazonaws.com`) |
+| `OPENSEARCH_STORAGE_USERNAME` | OpenSearch username (basic auth) |
+| `OPENSEARCH_STORAGE_PASSWORD` | OpenSearch password (basic auth) |
+
+SigV4 auth (AWS profile **or** the default credential chain; `es` domains or
+`aoss` Serverless collections), `none` (no auth), and basic auth are all
+supported, configurable via env vars, `agent-health.config.json`, or the
+Settings UI — see [docs/CONFIGURATION.md](https://github.com/opensearch-project/agent-health/blob/main/docs/CONFIGURATION.md).
+
+### AWS Credentials for LLM Judge (Required)
+
+The Bedrock LLM judge scores evaluation results and needs AWS credentials to call Bedrock.
+
+| Variable | Description |
+|---|---|
+| `AWS_PROFILE` | AWS profile with Bedrock access |
+| `AWS_REGION` | AWS region for Bedrock (e.g. `us-west-2`) |
+| `BEDROCK_MODEL_ID` | *(Optional)* Default judge model ID. Default: `us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
+
+### `.env` File
+
+The CLI automatically loads a `.env` file from the current working directory (the folder where commands are run). Create a `.env` file with all required variables:
+
+```bash
+# OpenSearch Storage
+OPENSEARCH_STORAGE_ENDPOINT=https://search-my-cluster.us-west-2.es.amazonaws.com
+OPENSEARCH_STORAGE_USERNAME=admin
+OPENSEARCH_STORAGE_PASSWORD=your-password
+
+# AWS / Bedrock
+AWS_PROFILE=your-aws-profile
+AWS_REGION=us-west-2
+# BEDROCK_MODEL_ID=us.anthropic.claude-sonnet-4-5-20250929-v1:0  # optional, this is the default
+```
+
+Alternatively, pass `--env-file <path>` to load a `.env` file from a different location.
+
+### Doctor Check
+
+After configuring your environment, verify everything is set up correctly:
+
+```bash
+npx @opensearch-project/agent-health doctor
+```
+
+This checks OpenSearch connectivity, AWS credentials, and Bedrock access. Fix any reported issues before proceeding.
+
+---
+
+## Commands
+
+```bash
+# Verify setup
+npx @opensearch-project/agent-health doctor --output json
+
+# Generate config files
+npx @opensearch-project/agent-health init
+
+# List available resources
+npx @opensearch-project/agent-health list agents --output json
+npx @opensearch-project/agent-health list test-cases --output json
+npx @opensearch-project/agent-health list benchmarks --output json
+
+# Run single test case
+npx @opensearch-project/agent-health run -t <test-case-id> -a <agent-key> --output json
+
+# Run full benchmark with export
+npx @opensearch-project/agent-health benchmark -n <benchmark-name> -a <agent-key> --export results.json
+
+# Import test cases from JSON file and run benchmark
+npx @opensearch-project/agent-health benchmark -f ./test-cases.json -a <agent-key>
+
+# Import with a custom benchmark name
+npx @opensearch-project/agent-health benchmark -f ./test-cases.json -n "My Benchmark" -a <agent-key>
+
+# Export test cases from a benchmark (produces import-compatible JSON)
+npx @opensearch-project/agent-health export -b <benchmark-name> -o test-cases.json
+```
+
+---
+
+## Code-based test SDK (write tests as files)
+
+Besides JSON test cases, Agent Health has an **experimental code-based SDK** for
+writing tests as `.eval.js` / `.eval.ts` files (Playwright-style `test()` /
+`expect()` / fixtures / `judge()` / lifecycle hooks). Run them the same way you
+run a JSON file:
+
+```bash
+# `-f` accepts BOTH JSON test-case files and code SDK (.eval.js / .eval.ts) files
+npx @opensearch-project/agent-health benchmark -f ./evals/demo.eval.js -a my-agent
+```
+
+They produce **per-matcher results** (`matcherResults[]`) instead of a single
+pass/fail. Full guide: [docs/SDK.md](https://github.com/opensearch-project/agent-health/blob/main/docs/SDK.md).
+
+---
+
+## Improvement Workflow
+
+### Step 0: Verify Setup
+```bash
+npx @opensearch-project/agent-health doctor --output json
+```
+Confirm all checks pass before proceeding. Fix any issues reported.
+
+### Step 1: Baseline Evaluation
+```bash
+npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export baseline.json
+```
+
+### Step 2: Analyze Failures
+Read `baseline.json` and find entries where `passFailStatus: "failed"`.
+
+Key fields to examine:
+- `matcherResults[*].reasoning` - the canonical judge verdict + reasoning per matcher (`llmJudgeReasoning` is a legacy, deprecated shim carrying the same text)
+- `improvementStrategies` - Specific recommendations with priority
+- `trajectory` - Step-by-step agent execution
+
+> Distinguish **`failed`** (the agent answered, judge scored it below threshold)
+> from **errored** (`metricsStatus: "error"` — the *evaluator itself* could not
+> run, e.g. a judge validation error). Errored runs are **excluded** from
+> pass-rate aggregation; don't treat them as agent misses.
+
+### Step 3: Fix Based on Strategies
+Focus on `priority: "high"` issues first:
+```json
+{
+  "category": "Tool Usage",
+  "issue": "Agent called search without time filter",
+  "recommendation": "Always include start_time and end_time parameters",
+  "priority": "high"
+}
+```
+
+Read the agent's code and implement the recommendation.
+
+### Step 4: Verify Fix
+```bash
+npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export after-fix.json
+```
+
+Compare `passRate` between baseline and after-fix.
+
+### Step 5: Iterate
+Repeat until all high-priority issues are resolved.
+
+---
+
+## Output Reference
+
+### Benchmark Export Structure
+```json
+{
+  "benchmark": { "id": "...", "name": "...", "testCaseCount": 10 },
+  "runs": [{
+    "agent": { "key": "my-agent", "name": "My Agent" },
+    "passed": 7,
+    "failed": 3,
+    "passRate": 70,
+    "reports": [{
+      "testCaseId": "tc-001",
+      "passFailStatus": "failed",
+      "metricsStatus": "ready",
+      "metrics": { "accuracy": 45 },
+      "matcherResults": [
+        { "method": "llm-judge", "description": "identifies the root cause", "pass": false, "score": 0.45, "reasoning": "The agent failed because..." }
+      ],
+      "llmJudgeReasoning": "The agent failed because...",
+      "improvementStrategies": [{
+        "category": "Tool Usage | Reasoning | Completeness",
+        "issue": "What went wrong",
+        "recommendation": "How to fix it",
+        "priority": "high | medium | low"
+      }],
+      "trajectory": [
+        { "type": "thinking", "content": "Agent's reasoning..." },
+        { "type": "action", "toolName": "search", "toolArgs": {} },
+        { "type": "tool_result", "content": "...", "status": "SUCCESS" },
+        { "type": "response", "content": "Final answer..." }
+      ]
+    }]
+  }]
+}
+```
+
+### Metrics, scoring, and run status
+
+- **Metrics are evaluator-defined and heterogeneous.** Only the *RCA Default*
+  evaluator emits a metric named `accuracy`. Other evaluators emit their own
+  metric names (`tool_selection_accuracy`, `reasoning_coherence`,
+  `bias_detection`, …) and custom evaluators emit whatever their
+  `scoringConfig` defines. Don't assume `metrics.accuracy` exists.
+- **A run's overall score** is the rounded mean of whatever numeric metrics the
+  run's evaluator emitted (`null`/`—` when none), not a single "accuracy".
+- **`matcherResults[]` is the canonical judge surface.** Read the judge verdict
+  + reasoning from `matcherResults` entries (`method: 'llm-judge'`). Each entry
+  is a `MatcherResult`: `{ description, pass (boolean), score? (0–1), method,
+  reasoning?, durationMs? }`. `llmJudgeReasoning` is still populated as a
+  backward-compatible shim but is deprecated.
+- **Run status:** `passed` / `failed` come from `passFailStatus`; a separate
+  **`errored`** state (`metricsStatus: 'error'`) means the evaluator could not
+  run and is excluded from pass-rate denominators.
+
+---
+
+## Tips
+
+1. **Always use `--output json`** for reliable parsing
+2. **Use `--export`** to get full reports with improvement strategies
+3. **Fix high-priority issues first** - they cause actual failures
+4. **Compare trajectories** between passing and failing cases
+5. **Make incremental changes** - one fix, then re-test
+6. **Don't over-engineer** - fix the specific issue identified
+
+---
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `npx agent-health` not found | `npm install -g @opensearch-project/agent-health` |
+| "No agents configured" | `npx agent-health init` (creates config with sample agents) |
+| Server / config issues | `npx agent-health doctor` (checks config + connectivity) |
+| "OpenSearch storage not configured" | Fine for local use — file-based storage is the default. Set `OPENSEARCH_STORAGE_*` only for shared / production persistence. |
+
+---
+
+## Server API Reference
+
+The Agent Health server runs on port 4001 and exposes the following REST APIs. All endpoints return JSON unless noted (SSE endpoints return `text/event-stream`).
+
+### Health & Configuration
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/health` | Health check → `{ status: 'ok', version, service: 'agent-health' }` |
+| GET | `/api/agents` | List all agents → `{ agents: AgentConfig[], total }` |
+| POST | `/api/agents/custom` | Add custom agent → `{ name, endpoint, connectorType?, useTraces? }` |
+| DELETE | `/api/agents/custom/:id` | Remove custom agent |
+| GET | `/api/models` | List all models → `{ models: ModelConfig[], total }` |
+| GET | `/api/debug` | Debug status → `{ enabled: boolean }` |
+| POST | `/api/debug` | Toggle debug → `{ enabled: boolean }` |
+
+### Agent Execution & Evaluation
+
+| Method | Path | Description |
+|---|---|---|
+| POST | `/api/agent` | Proxy agent request (SSE) → `{ endpoint, payload, headers?, agentKey? }` |
+| POST | `/api/evaluate` | Run evaluation (SSE) → `{ testCaseId?, testCase?, agentKey, modelId }` |
+
+### Judge
+
+| Method | Path | Description |
+|---|---|---|
+| POST | `/api/judge` | Evaluate a captured trajectory → `{ trajectory, expectedOutcomes? \| expectedTrajectory?, modelId?, evaluatorId?, runId?, logs?, agents? }` → `JudgeResponse` `{ passFailStatus, metrics, llmJudgeReasoning, improvementStrategies }` |
+| GET | `/api/judge/bedrock-models` | Discover Bedrock judge models (`ListInferenceProfiles`) |
+| GET | `/api/judge/openai-compatible-models` | List OpenAI-compatible models → `{ models, endpoint, configured }` (renamed from `/api/judge/litellm-models`) |
+| GET | `/api/judge/anthropic-models` | Discover Anthropic-direct models (needs `ANTHROPIC_API_KEY`) |
+| GET | `/api/judge/github-models` | Discover GitHub Models / Copilot models (needs `GITHUB_TOKEN`) |
+
+**On `/api/judge`, `modelId` is the *judge* model** — the trajectory is already
+captured, so there is no agent invocation here, and `/api/judge` does **not**
+take a `judgeModelId`. The agent-vs-judge model distinction (where `modelId`
+is the *agent's* LLM and `judgeModelId` is the *judge's*) applies to the run
+endpoints — `POST /api/evaluate` and `POST /api/storage/evaluation-runs` — and
+the CLI (`-m` / `--judge-model`). Agentic-provider judges (`pi` / `agent` /
+`agentic` / `claude-code`) pick their own model and ignore the judge model id.
+The run-level `matcherResults[]` are assembled by the runner from the judge's
+response (see [Output Reference](#output-reference)); the raw `/api/judge`
+primitive returns `llmJudgeReasoning`.
+
+### Traces & Metrics
+
+| Method | Path | Description |
+|---|---|---|
+| POST | `/api/traces` | Fetch traces → `{ traceId?, runIds?, startTime?, endTime?, size? }` → `{ spans, total, hasMore }` |
+| GET | `/api/traces/health` | Traces health → `{ status: 'ok' \| 'error' }` |
+| GET | `/api/metrics/:runId` | Run metrics → `{ totalTokens, costUsd, durationMs, llmCalls, toolCalls }` |
+| POST | `/api/metrics/batch` | Batch metrics → `{ runIds: string[] }` → `{ metrics[], aggregate }` |
+
+### Logs
+
+| Method | Path | Description |
+|---|---|---|
+| POST | `/api/logs` | Fetch logs → `{ runId?, query?, startTime?, endTime?, size? }` → `{ logs[], total }` |
+
+### Observability
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/observability/health` | Check observability data source health |
+| POST | `/api/observability/test-connection` | Test connection to observability cluster |
+| GET | `/api/observability/defaults` | Get default OTEL index patterns |
+
+### Storage: Test Cases
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/test-cases` | List test cases (latest versions). Query: `ids?`, `fields?`, `size?`, `after?` |
+| GET | `/api/storage/test-cases/:id` | Get latest version of test case |
+| GET | `/api/storage/test-cases/:id/versions` | Get all versions |
+| GET | `/api/storage/test-cases/:id/versions/:version` | Get specific version |
+| POST | `/api/storage/test-cases` | Create test case (v1) |
+| PUT | `/api/storage/test-cases/:id` | Update (creates new version) |
+| DELETE | `/api/storage/test-cases/:id` | Delete all versions |
+| POST | `/api/storage/test-cases/bulk` | Bulk create |
+
+### Storage: Benchmarks
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/benchmarks` | List benchmarks. Query: `fields?`, `size?` |
+| GET | `/api/storage/benchmarks/:id` | Get by ID. Query: `fields?`, `runsSize?`, `runsOffset?` |
+| GET | `/api/storage/benchmarks/:id/export` | Export test cases as JSON |
+| POST | `/api/storage/benchmarks` | Create → `{ name, description?, testCaseIds }` |
+| PUT | `/api/storage/benchmarks/:id` | Update |
+| PATCH | `/api/storage/benchmarks/:id/metadata` | Update metadata |
+| DELETE | `/api/storage/benchmarks/:id` | Delete |
+| POST | `/api/storage/benchmarks/:id/execute` | Execute benchmark (SSE) → `{ runConfig: RunConfigInput }` |
+| DELETE | `/api/storage/benchmarks/:id/runs/:runId` | Delete specific run |
+| POST | `/api/storage/benchmarks/:id/cancel` | Cancel execution |
+| POST | `/api/storage/benchmarks/:id/refresh-all-stats` | Recompute all run stats |
+
+### Storage: Runs (TestCaseRun)
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/runs` | List runs. Query: `size?`, `from?`, `fields?` |
+| GET | `/api/storage/runs/:id` | Get run by ID |
+| POST | `/api/storage/runs` | Create run |
+| PATCH | `/api/storage/runs/:id` | Update run |
+| DELETE | `/api/storage/runs/:id` | Delete run |
+| POST | `/api/storage/runs/search` | Search with filters |
+| GET | `/api/storage/runs/by-test-case/:testCaseId` | Runs for test case |
+| GET | `/api/storage/runs/by-benchmark/:benchmarkId` | Runs for benchmark |
+| GET | `/api/storage/runs/by-benchmark-run/:benchmarkId/:runId` | Results for benchmark run |
+| GET | `/api/storage/runs/iterations/:benchmarkId/:testCaseId` | Iterations for test case in benchmark |
+| POST | `/api/storage/runs/:id/annotations` | Add annotation |
+
+### Storage: Analytics
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/analytics` | Query analytics. Filters: `experimentId?`, `testCaseId?`, `agentId?`, `modelId?` |
+| GET | `/api/storage/analytics/aggregations` | Aggregated metrics |
+| POST | `/api/storage/analytics/search` | Complex search with aggregations |
+
+### Storage: Reports
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/benchmarks/:id/report` | Download report. Query: `format?` ('json'\|'html'\|'pdf') |
+
+### Storage: Evaluators
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/evaluators` | List evaluators (5 built-in system + custom) |
+| GET | `/api/storage/evaluators/:id` | Get evaluator (latest version) |
+| GET | `/api/storage/evaluators/:id/versions` | List all versions (newest first) |
+| GET | `/api/storage/evaluators/:id/versions/:version` | Get a specific version snapshot |
+| POST | `/api/storage/evaluators` | Create custom evaluator |
+| PUT | `/api/storage/evaluators/:id` | Update (creates a new immutable version) |
+| DELETE | `/api/storage/evaluators/:id` | Delete custom evaluator (system evaluators are protected) |
+
+### Storage: Evaluation Runs (code-based SDK)
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/evaluation-runs` | List evaluation runs |
+| GET | `/api/storage/evaluation-runs/:id` | Get evaluation run |
+| POST | `/api/storage/evaluation-runs` | Start a run (SSE). Accepts `sources: [{ type: 'code-import', filenames }]` for `.eval.js` / `.eval.ts` SDK files, plus `agentKey`, `modelId`, `judgeModelId?`, `evaluatorId?` |
+| POST | `/api/storage/evaluation-runs/:id/cancel` | Cancel a run |
+| POST | `/api/storage/evaluation-runs/:id/promote` | Promote a run |
+
+### Comparison
+
+| Method | Path | Description |
+|---|---|---|
+| POST | `/api/comparison/cluster-failures` | Cluster regressed cases into named failure patterns (`knowledge` / `tool_gap` / `reasoning` / `other`) |
+| GET | `/api/comparison/clusters/:clusterId` | Fetch a previously computed cluster by id |
+
+### Storage: Admin
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/api/storage/health` | Storage backend health |
+| POST | `/api/storage/test-connection` | Test storage connection |
+| POST | `/api/storage/init` | Initialize indexes |
+| GET | `/api/storage/config/status` | Config status |
+| POST | `/api/storage/config/storage` | Update storage config |
+| POST | `/api/storage/config/observability` | Update observability config |
+
+### Assistant (NEW)
+
+| Method | Path | Description |
+|---|---|---|
+| POST | `/api/assistant/chat` | Chat with AI assistant (SSE) → `{ sessionId, message, context }` → `{ type: 'delta'\|'done', content }` |
+| DELETE | `/api/assistant/session/:sessionId` | Clear session |
+| GET | `/api/assistant/health` | Check assistant availability |
+
+---
+
+## UI Pages
+
+The evaluation surface lives under `/evaluations/*`. The app uses
+`BrowserRouter` — plain paths, no `#` prefix.
+
+| Page | Route | Description |
+|---|---|---|
+| Dashboard | `/` | Overview with agent stats, recent runs, system health |
+| Benchmarks | `/evaluations/benchmarks` | List benchmarks with pass rates, run counts, version badges |
+| Benchmark Runs | `/evaluations/benchmarks/:id/runs` | Runs for a benchmark (split / tabs layout) |
+| Run Inspector | `/evaluations/benchmarks/:id/runs/:runId/inspect` | Per-test-case results, trajectory, judge, traces |
+| Test Cases | `/evaluations/test-cases` | List / create / edit / version test cases |
+| Test Case Detail | `/evaluations/test-cases/:id` | Definition + runs with an inline live-run panel |
+| Eval Runs | `/evaluations/runs` · `/evaluations/runs/:id` | Code-import (SDK) and ad-hoc run results |
+| Evaluators | `/evaluators` · `/evaluators/:id` | Manage evaluators; version history + Git-style diff |
+| Compare | `/compare/:benchmarkId?runs=a,b` | Diagnosis surface: verdict strip, first-divergence, failure clusters |
+| Agent Traces | `/agent-traces` | OpenTelemetry trace explorer (timeline / flow) |
+| Coding Agents | `/coding-agents` | Coding Agent Analytics (Claude Code / Kiro / Codex) |
+| Settings | `/settings` | Configure agents, models, storage, observability connections |
+| Assistant | `/assistant` | Full-page AI chat interface for help and analysis |
+
+---
+
+## Common Tasks
+
+### Ask about a benchmark's results
+"What's the pass rate for benchmark bench-xxx? Which test cases are failing and why?"
+→ The assistant will query `/api/storage/benchmarks/:id` and `/api/storage/runs/by-benchmark/:id`
+
+### Interpret judge reasoning
+"Why did test case tc-xxx fail in run run-xxx? What should I fix?"
+→ The assistant reads `llmJudgeReasoning` and `improvementStrategies` from the run
+
+### Write a test case
+"Help me write a test case for testing log search with time filters"
+→ The assistant creates a test case with prompt, context, expectedOutcomes, and labels
+
+### Analyze traces
+"What are the most expensive LLM calls in run run-xxx?"
+→ The assistant queries `/api/traces` and `/api/metrics/:runId` to find token-heavy spans
+
+### Compare runs
+"Compare the results of run A vs run B in benchmark bench-xxx"
+→ The assistant fetches both runs and diffs pass/fail status, accuracy, and strategies
+
+### Diagnose eval-vs-reality gaps
+"My agent passes 8/9 in evals but fails in real life — why?"
+→ The assistant investigates: fetches the runs and test cases, examines judge reasoning and expectedOutcomes, checks whether the evaluator is appropriate for the domain, and recommends tightening outcomes or creating a custom evaluator
+
+---
+
+## Evaluator System
+
+### How Evaluation Works
+
+1. Agent runs against a test case → produces a **trajectory** (thinking, action, tool_result, response steps)
+2. The **judge** (LLM) receives the trajectory + expectedOutcomes and scores each outcome
+3. Each outcome: Fully achieved = 1.0, Partially = 0.5, Not achieved = 0.0
+4. `accuracy = (sum of scores / total outcomes) × 100`
+5. Pass/Fail determined by evaluator's `passThreshold` + critical failure checks
+
+### Built-in Evaluators
+
+| ID | Name | Threshold | Key Metrics |
+|---|---|---|---|
+| `system-rca-default` | RCA Default | 70% | accuracy |
+| `system-factuality` | Factuality | 80% | factual_accuracy, hallucination_rate, source_grounding |
+| `system-tool-usage` | Tool Usage | 80% | tool_selection_accuracy, redundant_calls, tool_ordering |
+| `system-reasoning-depth` | Reasoning Depth | 75% | reasoning_coherence, step_completeness, logical_validity |
+| `system-safety` | Safety | 90% | safety_score, bias_detection, guardrail_adherence |
+
+### Custom Evaluators
+
+Create custom evaluators via the UI or API for domain-specific scoring:
+
+```json
+{
+  "name": "My Domain Evaluator",
+  "systemPrompt": "You are evaluating an agent for [domain]. CRITICAL CRITERIA: ...",
+  "scoringConfig": {
+    "metrics": [
+      { "name": "routing_accuracy", "weight": 0.4, "scale": 100 },
+      { "name": "tool_correctness", "weight": 0.4, "scale": 100 },
+      { "name": "diagnostic_completeness", "weight": 0.2, "scale": 100 }
+    ],
+    "passThreshold": 85,
+    "scale": 100
+  }
+}
+```
+
+Assign to a benchmark run via `evaluatorId` in the run config.
+
+### When to Recommend Custom Evaluators
+
+- **Domain has hard correctness rules**: specific CLI commands, valid syntax, routing tables, API formats
+- **Default judge is too charitable**: agents pass evals but fail in real-world usage (eval-vs-reality gap)
+- **Multiple independent dimensions**: routing correctness AND tool syntax AND diagnostic completeness all matter
+- **70% threshold is too low**: critical domains (oncall, security, compliance) need 85%+
+
+### Common Eval-vs-Reality Gap Causes
+
+1. **expectedOutcomes describe results, not procedures**: Judge rewards "got the right answer" without verifying "used the right method"
+2. **70% threshold too permissive**: Missing 30% of steps may be catastrophic in practice
+3. **No tool-correctness validation**: Generic judge can't verify domain-specific tool syntax
+4. **Trajectory compaction hides details**: toolOutput truncated to 1000 chars, content to 500 chars — wrong commands may be invisible to judge
+5. **Test cases cleaner than reality**: Hand-authored prompts include hints that real incidents don't
+6. **Single-turn evaluation**: Real work is iterative; test cases may compress multi-step workflows
+
+### Fix Strategies (in order)
+
+1. **Tighten expectedOutcomes**: Add negative constraints ("do NOT use X"), require specific tool names, specify correct syntax
+2. **Raise passThreshold**: Set to 85%+ for critical domains
+3. **Use `system-tool-usage` as secondary evaluator**: Catches redundant/wrong tool calls
+4. **Create domain-specific custom evaluator**: Include explicit rubric criteria the generic judge can't verify