@opensearch-project/agent-health 0.3.0 → 0.5.2

package/docs/plans/field-redesign.md

@@ -0,0 +1,468 @@
+# Plan: Code-Based Test Case SDK — Field Redesign (v2)
+
+## Status (as of 2026-05-16)
+
+### Completed (in working tree, not yet committed)
+- [x] `lib/testCases/types.ts` — Rewritten with `TestOptions` (prompt required), `CodeTestCase`, `EvalResult`
+- [x] `lib/testCases/define.ts` — Rewritten with `test()` API, file-scoped registries, `defineTestCases()` compat wrapper
+- [x] `lib/testCases/judge.ts` — Rewritten with runtime `wasJudgeCalled()`/`resetJudgeFlag()` flags
+- [x] `lib/testCases/loader.ts` — Rewritten with per-test-case `computeTestCaseHash()`, file-scoped loading via registry
+- [x] `lib/testCases/index.ts` — Updated barrel exports for new API
+- [x] `lib/index.ts` — Added SDK exports (`test`, `defineTestCases`, `judge`, `expect`)
+- [x] `services/sourceResolver.ts` — Added `EvaluateFn` type, `evaluateFnMap` to `ResolvedSources`, `code-import` case in switch (partial — needs unification to `import` type)
+
+### Not Started
+- [ ] Unified `import` source type (replace `file-import` + `code-import` + `directory-import`)
+- [ ] `services/evaluationRunner.ts` — Deterministic eval path with `resetJudgeFlag`/`wasJudgeCalled` inference
+- [ ] `types/index.ts` — `sourceFile` replaces `sourceUri`, `evaluationType` on TestCaseVersion, unified source type
+- [ ] `server/constants/indexMappings.ts` — `sourceFile` field
+- [ ] `server/adapters/file/StorageModule.ts` — `bulkUpsert` match on `sourceFile`
+- [ ] `server/adapters/opensearch/StorageModule.ts` — same
+- [ ] `cli/commands/benchmark.ts` — `-f` accepts dirs, exit codes, `--reporter`
+- [ ] `cli/reporters/` — `summary.ts`, `junit.ts`, `json.ts`
+- [ ] `components/RunDetailsContent.tsx` — Read evaluationType from version
+- [ ] `components/evals3/TestCasesPage.tsx` — Badge on `sourceFile`
+- [ ] All test updates (see step 11)
+
+### Key Context for Handoff
+- Branch: `feat/test-case-sdk` (local only, working tree has uncommitted changes)
+- The v1 implementation (committed on remote `feat/test-case-sdk`) uses `sourceUri`, `defineTestCases()`, per-file hash, and `evaluationType` on TestCaseRun. All of that is being replaced.
+- `evaluationRunner.ts` already has `runWithConcurrencyLimit()` — no need for `p-limit` dep.
+- The plan file was pushed to `fork/docs/sdk-field-redesign-plan` branch.
+- Existing tests reference old field names (`sourceUri`, old `defineTestCases` API) and will need updating.
+
+---
+
+## Context
+
+The initial SDK implementation (steps 1-11) is complete and passing. This redesign incorporates principal engineer review feedback to make the SDK actually production-ready — not just a working API, but something teams adopt over alternatives (Promptfoo, Braintrust, Langsmith).
+
+**Core insight:** The previous plan focused on internal field placement. This revision prioritizes the **customer-facing contract** — the API signature, CI exit codes, parallelism, and cost visibility — and defers internal storage plumbing where it doesn't block adoption.
+
+**Key changes from v1:**
+1. **Kill `agentic: boolean`** — infer evaluation type at runtime from whether `judge()` was called. No customer-facing field.
+2. **Add explicit `prompt` field** — test name is a description, prompt is what gets sent to the agent.
+3. **Add `--concurrency N`** — parallel test execution with semaphore.
+4. **Define CI contract** — exit codes, `--reporter junit|json`, pass/fail gate.
+5. **File-scoped isolation** — each file gets its own registry (no global mutable state).
+6. **Unify source types** — merge `file-import`, `code-import`, `directory-import` into a single `import` type that auto-detects by extension. `-f` accepts files and dirs.
+
+---
+
+## Implementation Steps
+
+### 1. Redesign SDK API: `test()` with explicit prompt
+
+**Files:** `lib/testCases/types.ts`, `lib/testCases/define.ts`
+
+```typescript
+// lib/testCases/types.ts
+export interface TestOptions {
+  prompt: string;              // REQUIRED — what gets sent to the agent
+  category: string;
+  difficulty: 'Easy' | 'Medium' | 'Hard';
+  description?: string;
+  context?: { description: string; value: string }[];
+  labels?: string[];
+  timeout?: number;            // ms, default 120000 (2 min)
+}
+
+export interface CodeTestCase {
+  name: string;
+  options: TestOptions;
+  evaluate: (result: EvalResult) => Promise<void> | void;
+}
+
+export interface EvalResult {
+  trajectory: TrajectoryStep[];
+  agentOutput: string;
+  rawEvents: any[];
+  runId?: string;
+  durationMs: number;
+  tokenUsage?: { prompt: number; completion: number; total: number };
+}
+```
+
+```typescript
+// lib/testCases/define.ts
+export interface FileRegistry {
+  tests: CodeTestCase[];
+  filePath?: string;
+}
+
+// Per-file registries (keyed by absolute file path)
+const registries = new Map<string, FileRegistry>();
+let activeFile: string | null = null;
+
+export function setActiveFile(filePath: string): void {
+  activeFile = filePath;
+  if (!registries.has(filePath)) {
+    registries.set(filePath, { tests: [], filePath });
+  }
+}
+
+export function test(
+  name: string,
+  options: TestOptions,
+  evaluate: (result: EvalResult) => Promise<void> | void
+): void {
+  if (!name || typeof name !== 'string') throw new Error('test() requires a name');
+  if (!options.prompt) throw new Error('test() requires options.prompt');
+  if (!options.category) throw new Error('test() requires options.category');
+  if (!options.difficulty) throw new Error('test() requires options.difficulty');
+  if (typeof evaluate !== 'function') throw new Error('test() requires an evaluate function');
+
+  const registry = activeFile ? registries.get(activeFile)! : getDefaultRegistry();
+  registry.tests.push({ name, options, evaluate });
+}
+
+export function getRegisteredTests(filePath?: string): CodeTestCase[] {
+  if (filePath) return [...(registries.get(filePath)?.tests ?? [])];
+  // Return all tests across all files
+  return [...registries.values()].flatMap(r => [...r.tests]);
+}
+
+export function clearRegistry(filePath?: string): void {
+  if (filePath) registries.delete(filePath);
+  else registries.clear();
+}
+```
+
+**Customer usage:**
+```typescript
+// evals/cybergym.eval.ts
+import { test, expect } from '@opensearch-project/agent-health';
+
+test('CyberGym Task 42', {
+  prompt: 'Analyze the vulnerability in the authentication module and provide a proof of concept exploit',
+  category: 'Security',
+  difficulty: 'Hard',
+}, async (result) => {
+  const poc = extractPoc(result.agentOutput);
+  const res = await submitToCyberGym(poc, 'task-42');
+  expect(res.exit_code).to.not.equal(0);
+});
+
+test('RCA Log Analysis', {
+  prompt: 'Investigate the spike in 5xx errors starting at 14:30 UTC. Identify the root cause.',
+  category: 'RCA',
+  difficulty: 'Medium',
+}, async (result) => {
+  // Use LLM judge for semantic evaluation
+  await judge(result.trajectory, ['Identifies root cause', 'Suggests remediation']);
+  // Can also add deterministic checks
+  expect(result.durationMs).to.be.lessThan(60000);
+});
+```
+
+### 2. Infer `evaluationType` at runtime (kill `agentic` field)
+
+**No customer-facing field.** Instead, detect at runtime:
+
+**File:** `lib/testCases/judge.ts`
+
+```typescript
+// Thread-local flag set when judge() is called during evaluate()
+let judgeCalledInCurrentEval = false;
+
+export function wasJudgeCalled(): boolean {
+  return judgeCalledInCurrentEval;
+}
+
+export function resetJudgeFlag(): void {
+  judgeCalledInCurrentEval = false;
+}
+
+export async function judge(trajectory: TrajectoryStep[], criteria: string[]): Promise<JudgeVerdict> {
+  judgeCalledInCurrentEval = true;
+  // Call LLM judge, return verdict
+  // ...
+}
+```
+
+**File:** `services/evaluationRunner.ts` — after calling evaluate():
+
+```typescript
+resetJudgeFlag();
+try {
+  await evalFn({ trajectory, agentOutput, rawEvents, durationMs });
+  const usedJudge = wasJudgeCalled();
+  report.passFailStatus = 'passed';
+  // evaluationType is inferred, stored on version after first run
+} catch (evalError: any) {
+  const usedJudge = wasJudgeCalled();
+  report.passFailStatus = 'failed';
+  report.assertionError = evalError.message;
+}
+```
+
+**On TestCaseVersion:** `evaluationType` is set after the first execution:
+- If `judge()` was called → `'agentic'`
+- If only assertions → `'deterministic'`
+- If both → `'hybrid'` (new value)
+
+This means version gets its `evaluationType` lazily on first run, not at import time.
+
+### 3. Replace `sourceUri` with `sourceFile` + per-test-case hash
+
+**File:** `types/index.ts` — on TestCase:
+```typescript
+sourceFile?: string;   // Relative path: "evals/cybergym.eval.ts"
+sourceHash?: string;   // SHA-256 of per-test-case content (not the whole file)
+```
+
+**Per-test-case hash** (`lib/testCases/loader.ts`):
+```typescript
+export function computeTestCaseHash(tc: CodeTestCase): string {
+  const content = JSON.stringify({
+    name: tc.name,
+    prompt: tc.options.prompt,
+    category: tc.options.category,
+    difficulty: tc.options.difficulty,
+    context: tc.options.context,
+    labels: tc.options.labels,
+    description: tc.options.description,
+  });
+  return crypto.createHash('sha256').update(content).digest('hex');
+}
+```
+
+**Match key for upsert:** `name + sourceFile`
+
+### 4. Add `--concurrency N` to benchmark runner
+
+**File:** `cli/commands/benchmark.ts` — add option:
+```typescript
+.option('-c, --concurrency <number>', 'Number of parallel test executions', '3')
+```
+
+**File:** `services/evaluationRunner.ts` — parallel execution with semaphore:
+
+```typescript
+import pLimit from 'p-limit';
+
+export async function executeEvaluationRun(run, testCases, options) {
+  const concurrency = options.concurrency ?? 3;
+  const limit = pLimit(concurrency);
+
+  const tasks = testCases.map(tc =>
+    limit(() => executeSingleTestCase(run, tc, options))
+  );
+
+  await Promise.allSettled(tasks);
+}
+```
+
+**New dependency:** `p-limit` (tiny, zero-dep semaphore).
+
+### 5. Define CI contract: exit codes + reporters
+
+**File:** `cli/commands/benchmark.ts`:
+
+```typescript
+.option('--reporter <type>', 'Output format: summary (default), junit, json, tap', 'summary')
+.option('--fail-fast', 'Stop on first failure')
+```
+
+**Exit codes:**
+- `0` — all tests passed
+- `1` — one or more tests failed
+- `2` — configuration error (bad file, missing agent, etc.)
+
+**File:** `cli/reporters/` (new directory):
+
+| File | Purpose |
+|------|---------|
+| `summary.ts` | Default terminal output (pass/fail counts, duration, cost) |
+| `junit.ts` | JUnit XML for CI integration (GitHub Actions, Jenkins) |
+| `json.ts` | Machine-readable JSON (for programmatic consumption) |
+
+**JUnit output** enables:
+```yaml
+# .github/workflows/eval.yml
+- run: npx agent-health benchmark -f evals/ -a my-agent --reporter junit > results.xml
+- uses: dorny/test-reporter@v1
+  with:
+    reporter: java-junit
+    path: results.xml
+```
+
+### 6. Add timeout + cost visibility
+
+**Timeout per test case** (from `TestOptions.timeout`, default 120s):
+```typescript
+const result = await Promise.race([
+  runAgentAndEvaluate(tc),
+  timeout(tc.options.timeout ?? 120000),
+]);
+```
+
+**Cost summary** in reporter output:
+```
+Results: 48 passed, 2 failed (50 total)
+Duration: 4m 32s
+Tokens: 1.2M prompt / 340K completion
+Est. cost: $4.82 (agent) + $1.20 (judge) = $6.02
+```
+
+**File:** `cli/reporters/summary.ts` — aggregate token usage from all runs and compute cost based on model pricing from config.
+
+### 7. Move `evaluationType` to TestCaseVersion
+
+**File:** `types/index.ts`
+
+```typescript
+export interface TestCaseVersion {
+  // ... existing fields ...
+  evaluationType?: 'agentic' | 'deterministic' | 'hybrid';
+}
+```
+
+Remove from `TestCaseRun`. Keep `assertionError` on run (execution-time data).
+
+**Lazy population:** On first execution of a code-imported test case, update the version's `evaluationType` based on runtime inference (step 2). Subsequent runs don't change it (immutable once set).
+
+### 8. Update storage: `sourceFile` + match logic
+
+**File:** `server/constants/indexMappings.ts`:
+```diff
+- sourceUri: { type: 'keyword' },
++ sourceFile: { type: 'keyword' },
+  sourceHash: { type: 'keyword' },
+```
+
+**File:** `server/adapters/file/StorageModule.ts` + `opensearch/StorageModule.ts`:
+- `bulkUpsert` matches on `name + sourceFile`
+- On hash change → new version (with `evaluationType` if known)
+
+### 9. Unified `import` source type in source resolver
+
+**File:** `services/sourceResolver.ts`
+
+Replace the three separate source types (`file-import`, `code-import`, `directory-import`) with a single `import` type:
+
+```typescript
+case 'import': {
+  const { testCases, fnMap } = await resolveImport(source.paths, storage);
+  // ...
+}
+```
+
+**`resolveImport(paths, storage)`** handles the unified logic:
+1. For each path, check if it's a file or directory
+2. If directory → scan for `*.eval.ts`, `*.eval.js`, `*.json` recursively
+3. For each file, route by extension:
+   - `.json` → existing JSON import path (parse + `bulkCreate`)
+   - `.ts` / `.js` / `.mjs` → code import path (load module + `bulkUpsert`)
+4. Code files produce `evaluateFnMap` entries; JSON files don't
+5. Return combined test cases + fnMap
+
+**CLI `-f` flag** accepts both files and directories:
+```bash
+npx agent-health benchmark -f evals/              # scans dir
+npx agent-health benchmark -f evals/rca.eval.ts   # single code file  
+npx agent-health benchmark -f test-cases.json     # single JSON file
+npx agent-health benchmark -f evals/ -f extra.json # mixed
+```
+
+**`TestCaseSource` type** simplification:
+```typescript
+// Before: 3 source types
+| { type: 'file-import'; filenames: string[]; testCaseIds: string[] }
+| { type: 'code-import'; filenames: string[]; testCaseIds: string[] }
+| { type: 'directory-import'; dirPaths: string[]; testCaseIds: string[] }
+
+// After: 1 unified type
+| { type: 'import'; paths: string[]; testCaseIds: string[] }
+```
+
+Keep old types as aliases for backward compat (existing benchmarks may have these in storage).
+
+### 10. Update UI
+
+**File:** `components/evals3/TestCasesPage.tsx`:
+- "Code" badge on `testCase.sourceFile` (not `sourceUri`)
+- Disable edit for code-sourced test cases
+
+**File:** `components/RunDetailsContent.tsx`:
+- Read `evaluationType` from test case version (not report)
+- Show "Hybrid" badge when both judge and assertions were used
+- Show assertion error for deterministic/hybrid failures
+
+### 11. Update existing tests
+
+| Test File | Change |
+|-----------|--------|
+| `tests/unit/lib/testCases/define.test.ts` | Rewrite for `test()` API with `prompt` field, file-scoped registry |
+| `tests/unit/services/evaluationRunner.deterministic.test.ts` | Remove `evaluationType` from report assertions |
+| `tests/unit/services/sourceResolver.codeImport.test.ts` | `sourceFile` instead of `sourceUri`, prompt field |
+| `tests/unit/server/adapters/bulkUpsert.test.ts` | Match on `sourceFile` |
+| `tests/integration/services/sourceResolver.codeImport.integration.test.ts` | Update for new API |
+| `tests/integration/services/codeSdkObservio.integration.test.ts` | Update field references |
+| `tests/unit/cli/reporters/` (new) | Test junit, json, summary output |
+
+### 12. Package exports
+
+**File:** `lib/testCases/index.ts`:
+```typescript
+export { test, getRegisteredTests, clearRegistry } from './define.js';
+export { judge } from './judge.js';
+export { defineTestCases } from './compat.js';  // Deprecated
+export type { TestOptions, CodeTestCase, EvalResult } from './types.js';
+```
+
+**File:** `lib/index.ts`:
+```typescript
+export { test, defineTestCases, judge } from './testCases/index.js';
+export { expect } from 'chai';
+```
+
+**New dependency:** `p-limit` (production — for concurrency control).
+
+---
+
+## Critical Files to Modify
+
+| File | Change |
+|------|--------|
+| `lib/testCases/types.ts` | `TestOptions` with `prompt` (required), `timeout`; drop `agentic` |
+| `lib/testCases/define.ts` | File-scoped registries, `test()` with prompt validation |
+| `lib/testCases/loader.ts` | Per-test-case hash, file-scoped loading |
+| `lib/testCases/judge.ts` | Runtime flag for `judge()` calls |
+| `services/evaluationRunner.ts` | Parallel execution (`p-limit`), timeout, infer evaluationType |
+| `services/sourceResolver.ts` | Unified `import` source type, `sourceFile` (relative), per-test-case hash |
+| `types/index.ts` | Replace 3 source types with unified `import` type |
+| `cli/commands/benchmark.ts` | `-f` accepts files+dirs, `--reporter`, `--fail-fast`, exit codes |
+| `cli/reporters/` (new) | `summary.ts`, `junit.ts`, `json.ts` |
+| `types/index.ts` | `evaluationType` on TestCaseVersion (lazy), `sourceFile` on TestCase |
+| `server/constants/indexMappings.ts` | `sourceFile` replaces `sourceUri` |
+| `server/adapters/file/StorageModule.ts` | Match on `sourceFile` in bulkUpsert |
+| `components/RunDetailsContent.tsx` | Read evaluationType from version |
+| `components/evals3/TestCasesPage.tsx` | Badge on `sourceFile` |
+
+---
+
+## Implementation Order (prioritized by customer impact)
+
+1. **SDK API** (steps 1-2) — `test()` with prompt, file-scoped registry, runtime inference
+2. **CI contract** (step 5) — exit codes, reporters
+3. **Parallelism + timeout** (steps 4, 6) — `--concurrency`, per-test timeout
+4. **Storage redesign** (steps 3, 7, 8, 9) — sourceFile, per-test hash, version-level evaluationType
+5. **UI** (step 10) — badges, version-aware rendering
+6. **Tests** (step 11) — update all affected tests
+7. **Exports** (step 12) — package API surface
+
+---
+
+## Verification Plan
+
+1. **Build:** `npm run build:all` passes
+2. **Unit tests:** `npm run test:unit` — all pass
+3. **CI contract test:** `npx agent-health benchmark -f tests/fixtures/sample.eval.ts -a observio --reporter json` returns exit 0/1 correctly
+4. **Parallel test:** 10 test cases with `--concurrency 5` complete in ~2x single-case time (not 10x)
+5. **JUnit output:** Valid XML parseable by `dorny/test-reporter`
+6. **Cost summary:** Token counts + estimated cost displayed in summary reporter
+7. **Per-test hash:** Modify one test in a multi-test file → only that test gets new version
+8. **Runtime inference:** Test with `judge()` → version gets `evaluationType: 'agentic'`; test without → `'deterministic'`; test with both → `'hybrid'`
+9. **Timeout:** Test case exceeding timeout fails with clear error message
+10. **Backward compat:** `defineTestCases([...])` still works, old test cases render correctly