@opensearch-project/agent-health 0.5.2 → 0.7.0

package/docs/CLI.md

@@ -61,11 +61,13 @@ agent-health list <resource> [-o table|json]
 | `models` | | Available models |
 | `test-cases` | `testcases`, `tc` | Stored test cases |
 | `benchmarks` | `bench` | Stored benchmarks |
+| `images` | `img` | Benchmark images (content-addressed evaluation-condition snapshots; runs sharing a digest are directly comparable) |
 
 ```bash
 agent-health list agents
 agent-health list tc -o json
 agent-health list bench
+agent-health list images
 ```
 
 ---
@@ -122,20 +124,61 @@ agent-health benchmark [options]
 | `--stop-server` | Stop the server after benchmark completes | Keep running |
 
 **Modes:**
-- **Quick mode** (no `-n`, no `-f`): Auto-creates a benchmark from all stored test cases
+- **Quick mode** (no `-n`, no `-f`): Runs all stored test cases as an **ad-hoc evaluation run** (no benchmark entity is created)
 - **Named mode** (`-n <name>`): Runs a specific existing benchmark
-- **File mode** (`-f <path>`): Imports test cases from a JSON file **or runs a code SDK file** (`.eval.js` / `.eval.ts` — see [SDK.md](./SDK.md)), creates a benchmark, and runs it
+- **File mode** (`-f <path>`): Imports test cases from a JSON file **or runs a code SDK file** (`.eval.js` / `.eval.ts` — see [SDK.md](./SDK.md)), creates a benchmark, and runs it. `.eval.ts` is executed as synthetic CJS (like `.eval.js`) and works from anywhere on disk; only `.eval.mjs` resolves `@opensearch-project/agent-health` through normal Node module resolution, so an `.eval.mjs` file needs the package reachable as a real dependency from its location (see the note in [SDK.md](./SDK.md#migrating-v1--v2))
+
+Every evaluation run is stamped with an **image digest** — a content hash of
+its test-case contents + eval conditions (evaluator, judge model). Runs with
+the same digest ran under identical conditions and are directly comparable;
+re-running the same command converges on the same image instead of creating
+new entities. See `agent-health list images`.
 
 ```bash
 agent-health benchmark                                           # quick mode
 agent-health benchmark -n "Baseline" -a ml-commons               # named mode
 agent-health benchmark -f ./test-cases.json -a pulsar -v         # file mode (JSON)
-agent-health benchmark -f ./evals/demo.eval.js -a observio       # file mode (code SDK)
+agent-health benchmark -f ./examples/eval-files/demo.eval.js -a observio       # file mode (code SDK)
 agent-health benchmark -f ./test-cases.json -n "My Run" -a pulsar --export results.json
 agent-health benchmark -n "Baseline" -e system-tool-usage -c 4   # custom evaluator, 4 in parallel
 agent-health benchmark -n "Baseline" --export report.html --format html
 ```
 
+#### benchmark doctor
+
+Detect and clean up duplicated / debris benchmarks (dry-run by default).
+
+```
+agent-health benchmark doctor [--dry-run] [--apply] [--migrate-images] [--json]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--dry-run` | Preview only — this is already the default; use --apply to execute |
+| `--apply` | Execute the plan (default: dry-run report only) |
+| `--migrate-images` | Convert remaining benchmarks into tagged benchmark images |
+| `--json` | Output as JSON instead of the human-readable report |
+
+What it detects:
+- **Timestamped debris** — `quick-<ts>` / `*-<epoch-ms>` benchmarks with no
+  runs anywhere and older than 24h (deleted).
+- **Content duplicates** — benchmarks with identical test-case sets. Embedded
+  runs are merged into the canonical (most runs › most references › oldest),
+  evaluation runs are re-pointed, duplicate shells deleted.
+
+Runs and reports are **never** deleted. Sample data (`demo-*`) is never touched.
+
+**Read-only mode**: When running without `--apply` or `--migrate-images` (pure dry-run),
+the command safely reuses existing foreign servers in read-only mode. This allows
+the diagnostic to run even when another agent-health instance is operating on a
+different worktree/port, with a clear notice that no writes will be issued.
+
+```bash
+agent-health benchmark doctor                    # report what would change (read-only)
+agent-health benchmark doctor --apply            # clean up (strict server guard)
+agent-health benchmark doctor --apply --migrate-images
+```
+
 ---
 
 ### export
@@ -429,3 +472,140 @@ Most commands support `-o, --output`:
 |--------|----------|
 | `table` | Human-readable (default) |
 | `json` | Machine-readable, scripting |
+
+---
+
+## Running from Outside the Repo
+
+When running `npx @opensearch-project/agent-health` from a directory *outside* the agent-health repository (e.g., from a customer or partner project), you may encounter configuration and environment issues. This section covers the most common friction points and their solutions.
+
+### Issue 1: Port Already In Use
+
+**Error message:**
+```
+Port 4001 is in use, trying 4002...
+```
+
+**Root cause:** The default ports (4001 for server, 4000 for frontend) are busy, and all fallback ports (4001–4010) are occupied.
+
+**Solution:** Set `AH_PORT` to use a different port:
+
+```bash
+AH_PORT=8001 npx @opensearch-project/agent-health
+```
+
+The server will listen on port 8001. If your frontend dev port (4000) is also in use, set `AH_DEV_PORT` as well:
+
+```bash
+AH_PORT=8001 AH_DEV_PORT=8000 npx @opensearch-project/agent-health
+```
+
+### Issue 2: Config File Loading Fails (`package.json` Type)
+
+**Error message:**
+```
+Failed to load config file agent-health.config.ts: ERR_MODULE_NOT_FOUND
+```
+
+**Root cause:** Your cwd's `package.json` doesn't declare `"type": "module"`, and Node.js cannot load the TypeScript config as an ES module.
+
+**Solution:** Add `"type": "module"` to your local `package.json`:
+
+```json
+{
+  "type": "module",
+  "name": "my-project",
+  "version": "1.0.0"
+}
+```
+
+Alternatively, create a minimal `package.json` in the directory where you're running the command:
+
+```bash
+echo '{"type":"module"}' > package.json
+npx @opensearch-project/agent-health
+```
+
+### Issue 3: TypeScript Config Parsing Fails (`TSX_TSCONFIG_PATH`)
+
+**Error message:**
+```
+Failed to load config file agent-health.config.ts: [TypeScript parsing error]
+```
+
+**Root cause:** The TypeScript loader cannot find your `tsconfig.json` (e.g., if you placed your config in a subdirectory).
+
+**Solution:** Set `TSX_TSCONFIG_PATH` to point to the agent-health repo's `tsconfig.json`:
+
+```bash
+TSX_TSCONFIG_PATH=/path/to/agent-health/tsconfig.json npx @opensearch-project/agent-health
+```
+
+Or, if you have your own `tsconfig.json` that extends agent-health's:
+
+```bash
+TSX_TSCONFIG_PATH=./tsconfig.json npx @opensearch-project/agent-health
+```
+
+### Issue 4: Claude Code Binary Not Found
+
+**Error message:**
+```
+Subprocess timed out or failed: ENOENT: no such file or directory, spawn 'claude'
+```
+
+**Root cause:** The `claude` CLI is not installed or not in your `PATH`.
+
+**Solution:** Ensure Claude Code CLI is installed and accessible:
+
+```bash
+which claude                              # Verify CLI is in PATH
+claude --version                          # Check installation
+```
+
+If the CLI is installed but not in `PATH`, set `CLAUDE_CODE_BIN` to the full path:
+
+```bash
+CLAUDE_CODE_BIN=/usr/local/bin/claude npx @opensearch-project/agent-health
+```
+
+Or install Claude Code globally:
+
+```bash
+npm install -g @anthropic-ai/claude-code
+```
+
+### Complete External Invocation Example
+
+Here's a full example running from outside the repo with all common overrides:
+
+```bash
+# From ~/my-project/ directory (outside agent-health repo):
+export AH_PORT=8001
+export AH_DEV_PORT=8000
+export TSX_TSCONFIG_PATH=/opt/agent-health/tsconfig.json
+export CLAUDE_CODE_BIN=/usr/local/bin/claude
+export AWS_PROFILE=my-profile
+
+# Create minimal package.json if needed
+echo '{"type":"module"}' > package.json
+
+# Run the CLI
+npx @opensearch-project/agent-health --no-browser
+```
+
+Then navigate to http://localhost:8001 in your browser.
+
+### Checklist
+
+- [ ] `AH_PORT` set if default ports are busy
+- [ ] `package.json` has `"type": "module"`
+- [ ] `TSX_TSCONFIG_PATH` set if config parsing fails
+- [ ] Claude Code CLI installed (if using claude-code connector)
+- [ ] AWS credentials configured (if using Bedrock judge)
+
+If issues persist, run with `DEBUG=true` for verbose output:
+
+```bash
+DEBUG=true npx @opensearch-project/agent-health
+```

package/docs/CONFIGURATION.md

@@ -113,7 +113,7 @@ Rules of thumb:
   `OPENSEARCH_STORAGE_*` or `storage` in your config to use a cluster instead).
 - **`evals/`** is **your test source** — `.eval.js` / `.eval.ts` files you write
   with the [code SDK](./SDK.md) and run with
-  `agent-health benchmark -f ./evals/demo.eval.js`. They are **not** stored under
+  `agent-health benchmark -f ./examples/eval-files/demo.eval.js`. They are **not** stored under
   `.agent-health/data/`; *running* them produces run records that land there (or
   in OpenSearch).
 

package/docs/CONNECTORS.md

@@ -529,7 +529,7 @@ See `services/connectors/claude-code/ClaudeCodeConnector.ts` for a complete exam
 ### Kiro Connector
 
 See `services/connectors/kiro/KiroConnector.ts` for a `SubprocessConnector`
-subclass that overrides `parseStderrChunk()` to convert Kiro's stderr-borne
+subclass that overrides `parseStderrChunk(chunk, trajectory, onProgress, state)` to convert Kiro's stderr-borne
 `[tool] Running:` / `[tool] status:` markers into structured `action` +
 `tool_result` steps. The base `SubprocessConnector` also persists `stderr` to
 `rawOutput` and honors per-request `connectorConfig` overrides (`args` /

package/docs/INSTRUMENT_WITH_OTEL.md

@@ -385,6 +385,13 @@ If you run an MCP server, expose this document as a resource at `agent-health://
 
 ## Reference
 
+### New to OpenTelemetry?
+
+- [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/) — start here for the concepts (traces, spans, exporters, collectors).
+- [OpenTelemetry GenAI semantic conventions (repo)](https://github.com/open-telemetry/semantic-conventions-genai) — the source-of-truth for the `gen_ai.*` attributes this guide uses.
+
+### Specs used by this guide
+
 - [OTel GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
 - [OTel GenAI Agent Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
 - [Agent Health Telemetry Setup](./CLAUDE_CODE_TELEMETRY.md)

package/docs/SDK.md

@@ -64,7 +64,7 @@ async function ({ agent, judge, evaluate, expect, testInfo, provisioned }) { ...
 | `agent`  | `AgentFixture`                                   | `await agent.run(prompt?, options?)` — invoke the agent **once** (control inversion). Returns an `EvalResult`. |
 | `judge`  | `JudgeFn`                                         | Non-throwing LLM judge. `judge(...)` gates; `judge.observe(...)` is observational. Returns a `Verdict`. |
 | `evaluate` | `EvaluateFn`                                     | Run a custom programmatic evaluator registered with `defineEvaluator()`. |
-| `expect` | chai's `expect` with our recording plugin        | Synchronous matcher entry-point |
+| `expect` | chai's `expect` with our recording plugin        | Synchronous matcher entry-point. `expect.soft(...)` records a failing assertion instead of throwing — see [Always-record guarantee](#5-matchers-record-structured-verdicts). |
 | `testInfo` | `TestInfo` (read-only)                          | `{ name, benchmarkPath, sourceFile, testCaseId }` |
 | `provisioned` | `Readonly<Record<string, unknown>>`          | Values set by `beforeEach` via `provide()` |
 | `result` | `EvalResult` *(legacy eager path)*               | Pre-populated agent result — present when the runner invokes before the body. Prefer `await agent.run()`. |
@@ -166,7 +166,7 @@ Key points:
 Hooks are a no-op when no test in the run uses them — the orchestrator
 is short-circuited to a noop variant and existing tests pay zero cost.
 
-See the demo at [`evals/sdk-hooks-demo.eval.js`](../evals/sdk-hooks-demo.eval.js).
+See the demo at [`examples/eval-files/sdk-hooks-demo.eval.js`](../examples/eval-files/sdk-hooks-demo.eval.js).
 
 ### 5. Matchers record structured verdicts
 
@@ -187,6 +187,79 @@ Matchers (4/5 passed)
 This is the major upgrade over throw-and-fail: every assertion gets its own
 row, status, and detail block.
 
+#### Always-record guarantee
+
+chai's `expect()` is fail-fast — the first failing assertion throws, and the
+rest of the test body (any later `expect()`/`judge()`/`evaluate()` calls)
+never executes. For a pass/fail verdict that's the intended Playwright-style
+contract, but a report consumed by an optimizer (or by you, comparing runs)
+needs the objective actuals — duration, token usage, USD cost — from every
+axis, even when one gate fails partway through. A token-budget gate failing
+shouldn't erase the cost figure that would otherwise have been on the report.
+
+So regardless of where (or whether) the body throws, the runner **always**
+stamps `performanceMetrics.durationMs` / `.agentDurationMs` /
+`.totalTokens` / `.totalCostUsd` onto the report the moment `agent.run()`
+resolves — reading them straight from the same OTel-derived data
+`result.traces`/the `traces` fixture exposes, independent of whether the
+body's own code ever reached a matcher that asserted on them. (`totalTokens`/
+`totalCostUsd` are `undefined`, not `0`, when `useTraces: true` but spans
+never arrived — see the loud-failure accessor in "Traces fixture" below;
+they're real `0`s when `useTraces: false`.)
+
+This guarantee has one hard limit: a `judge()`/`evaluate()` call itself is
+already non-throwing and records its score/verdict the instant it's called —
+so a REACHED judge call is always on the report today, throw-or-not. But a
+judge call placed textually *after* a failing `expect()` never executes at
+all — that's a pure source-order problem the runner cannot retroactively fix
+(it can't record a call that never ran). Use `expect.soft(...)` (below) when
+you need later judge/evaluate calls to run even after an earlier assertion
+fails.
+
+The matcher panel also gets a distinct **"not reached"** row appended
+whenever the body threw — so "this axis never ran" (grey, not-reached) reads
+differently from "this axis ran and failed" (red). It's excluded from the
+passed/failed counts and gets its own tally: `(3/4 passed, 1 not reached)`.
+
+#### `expect.soft(...)` — keep going instead of bailing on the first failure
+
+`expect.soft(value)` is the same chai assertion surface as `expect(value)`
+— every built-in BDD matcher plus the custom ones below — except a failing
+assertion **records** the MatcherResult and returns instead of throwing:
+
+```javascript
+test('budget-aware RCA', { prompt: '...' }, async ({ agent, expect, judge }) => {
+  const result = await agent.run();
+  expect.soft(result.traces.totalTokens).to.be.lessThan(10_000);  // fails — recorded, body continues
+  expect.soft(result.traces.totalCost).to.be.lessThan(0.05);      // still runs
+  await judge(result, 'identifies the root cause');                // still runs — the whole point
+  expect(result.agentOutput).to.contain('root cause');             // a HARD expect still bails on ITS OWN failure
+});
+```
+
+The runner's overall verdict is unaffected by soft vs. hard: a test already
+fails when ANY recorded gate matcher has `pass: false` (that's exactly how
+non-throwing `judge()`/`evaluate()` gates have always worked) — `expect.soft`
+just extends that same non-throwing contract to chai assertions. Mix soft and
+hard freely in one body; a hard `expect()` after a soft failure still bails
+at that point (source order still matters for the calls *after* a HARD
+failure). Reach for `.soft` on measurement/budget-style axis checks where you
+want every number on the report regardless of which gates failed; keep hard
+`expect()` for a precondition that makes the rest of the body meaningless if
+it's false (e.g. "the agent produced any output at all").
+
+> **Known limitation: multi-step property chains can record a derivative
+> second failure.** `expect.soft(obj).to.have.property('x').that.equals(5)`
+> is really two chai assertions run back-to-back on the same chain (`property`
+> checks existence, `equals` checks the value) — with a HARD `expect()`, a
+> missing property throws immediately and `.that.equals(5)` never runs. In
+> soft mode nothing throws, so `.that.equals(5)` *does* run against
+> `undefined` and records its own "expected undefined to equal 5" failure —
+> real, but a symptom of the first failure, not independent information.
+> Prefer single-step matchers on primitive/simple values with `.soft` (as in
+> every example above); for a value that might not exist, use a hard
+> `expect()` to check existence first, then `.soft` on the value itself.
+
 ---
 
 ## Test options
@@ -483,8 +556,57 @@ enforced regardless of configuration.
 
 ---
 
+## Weighted scoring across criteria
+
+Each `expect(...)`, `judge(...)`, and `evaluate(...)` call records **one
+per-matcher pass/fail** — they are not weighted against each other. When you
+need a **single aggregate score** that weights criteria differently (e.g. root
+cause 60%, SOP selection 20%, relevant-metrics check 10%, latency 10%), the
+weighting lives in a **custom evaluator's `scoringConfig`**, not in the test
+body. Define the weighted metrics once and attach the evaluator to the run via
+`evaluatorId`:
+
+```json
+{
+  "name": "Ops RCA (weighted)",
+  "systemPrompt": "You evaluate an ops agent that triages a ticket. Score each metric 0-100. CRITICAL: root_cause_accuracy is the primary metric.",
+  "scoringConfig": {
+    "metrics": [
+      { "name": "root_cause_accuracy",  "weight": 0.6, "scale": 100 },
+      { "name": "sop_selection",        "weight": 0.2, "scale": 100 },
+      { "name": "relevant_metrics",     "weight": 0.1, "scale": 100 },
+      { "name": "latency",              "weight": 0.1, "scale": 100 }
+    ],
+    "passThreshold": 80,
+    "scale": 100
+  }
+}
+```
+
+The run's overall score is the weighted mean of the metrics the evaluator
+emits, so you get one comparable number per run for progress tracking. Use the
+deterministic `expect(...)` matchers for the hard checks (exact classification,
+tool was called, budget cap) and the judge/evaluator + weighted `scoringConfig`
+for the aggregate score. See the ops-RCA worked example at
+[`examples/eval-files/ops-rca-classification.eval.js`](../examples/eval-files/ops-rca-classification.eval.js)
+and its companion [`examples/eval-files/ops-rca-evaluator.json`](../examples/eval-files/ops-rca-evaluator.json),
+and the evaluator reference in [docs/skills/AGENT_HEALTH.md](./skills/AGENT_HEALTH.md#custom-evaluators).
+
+---
+
 ## Running the tests
 
+> **Security note:** the ENTIRE contents of an imported `.eval.js`/`.eval.ts`
+> file are persisted verbatim on the resulting test case (`sourceCode`) and
+> rendered on the Test Case detail page as an IDE-style code view, so anyone
+> who can view test cases in this deployment can read the full file --
+> including any comments, hardcoded values, or internal URLs it contains.
+> Treat eval files like any other source file that lands in your repo and
+> gets deployed alongside the app: don't hardcode secrets, tokens, or
+> customer data in them -- pull those from environment variables /
+> `agent-health.config.ts` instead, same as you already would for the app's
+> own credentials.
+
 ### Via the UI
 
 `/evaluations/runs/new` → pick "Code import" → select your `.eval.js` files.
@@ -492,7 +614,7 @@ enforced regardless of configuration.
 ### Via the CLI
 
 ```bash
-npx @opensearch-project/agent-health benchmark -f ./evals/demo.eval.js -a observio
+npx @opensearch-project/agent-health benchmark -f ./examples/eval-files/demo.eval.js -a observio
 ```
 
 ### Via the HTTP API
@@ -504,7 +626,7 @@ curl -sN -X POST http://localhost:4001/api/storage/evaluation-runs \
     "name": "Demo",
     "sources": [{
       "type": "code-import",
-      "filenames": ["evals/demo.eval.js"],
+      "filenames": ["examples/eval-files/demo.eval.js"],
       "testCaseIds": []
     }],
     "agentKey": "observio",
@@ -526,6 +648,57 @@ The `.eval.js`, `.eval.ts`, and `.eval.mjs` loaders all run through a single
 code-import execution path, so `benchmark -f <file>` executes the SDK body
 directly.
 
+> **`.eval.js` and `.eval.ts` are both executed as synthetic CJS — only
+> `.eval.mjs` needs the package to be really resolvable.** `.eval.js` and
+> `.eval.ts` files are both executed in a synthetic context where the
+> `require(...)` call for `@opensearch-project/agent-health` is intercepted
+> directly, so they work from anywhere on disk regardless of whether the
+> package is actually installed at that location. (`.eval.ts` is
+> transpiled to CommonJS with `esbuild` — a required runtime dependency of
+> this package — before running through that exact same synthetic-CJS
+> path; it deliberately does NOT use a native `import()`, both to avoid two
+> different Node versions giving the same file two different
+> module-execution semantics, and because `import()` caches by URL, which
+> would make a `.eval.ts` file unloadable a second time in one process
+> without a restart. One consequence: an `.eval.ts` fixture can't use
+> `import.meta` or a top-level `await` — real-ESM-only features with no
+> CommonJS equivalent.) `.eval.mjs` files use a plain native `import()`, so
+> `import { test } from '@opensearch-project/agent-health'` in an `.mjs`
+> file must resolve through Node's normal module resolution — the package
+> needs to be a real dependency reachable from the file's location
+> (installed in `node_modules`, a workspace link, or a `node_modules`
+> symlink to a local checkout). Test registrations are shared process-wide
+> (keyed off `globalThis`) regardless of loader, so it doesn't matter
+> *which* physical copy of the package a given import resolves to — dist
+> vs source, symlinked vs installed all register into the same registry
+> the loader reads from.
+
+> **`.eval.ts` can `import` sibling `.ts` helper files, but not ESM-only
+> packages.** A relative/absolute `import` inside an `.eval.ts` file that
+> resolves to another `.ts` file (e.g. `import { helper } from
+> './helper.ts'`) is transpiled and executed through the same synthetic-CJS
+> mechanism recursively — multi-file `.eval.ts` fixtures work, and a helper
+> required more than once within one load executes exactly once (cached
+> for the duration of that load only, never across separate loads/reloads).
+> An `import` of a package that ships **no CommonJS entry point** (ESM-only,
+> e.g. modern `chalk`) fails with an actionable error — `.eval.ts` executes
+> as synthetic CJS, so under require()'s own rules that package can't load
+> there — unless the running Node version has native `require(esm)`
+> interop (stable since Node 22.12; check `process.features.require_module`),
+> in which case it works transparently via Node's own mechanism. If you hit
+> the error on an older Node, switch the fixture to `.eval.mjs` (real ESM,
+> can import ESM-only packages directly) or pre-compile to `.eval.js`.
+
+> **`.eval.mjs` reloads are cache-busted, not cached.** Every `import()` of
+> an `.eval.mjs` file is given a unique query string
+> (`?ah-reload=<timestamp>-<random>`) so re-loading the same file a second
+> time in one process (e.g. re-running the CLI against an edited fixture)
+> re-executes its top-level code instead of silently returning Node's
+> already-cached module for that URL. Accepted cost: each reload leaves one
+> module instance cached under a never-reused query string for the
+> lifetime of the process — negligible for CLI/server process lifetimes
+> (a handful to a few hundred loads, not an unbounded long-running loop).
+
 ---
 
 ## Dev tips
@@ -574,4 +747,4 @@ keeping it user-supplied lets you opt in without breaking anyone else.
 - [x] Non-throwing run-scoped `judge` with `gate` / `observe` roles + `skip` + `orThrow()`
 - [x] `defineEvaluator()` / `evaluate()` for mechanical / external verification ([#244](https://github.com/opensearch-project/agent-health/issues/244))
 - [x] Single code-import execution path (`benchmark -f *.eval.js` runs the SDK body) + unified `.js` / `.ts` / `.mjs` loaders + `agent-health migrate sdk-v2` codemod
-- [ ] `expect.soft()` to collect-all-failures instead of bail-on-first
+- [x] `expect.soft()` to collect-all-failures instead of bail-on-first