@opensearch-project/agent-health 0.3.0 → 0.5.2

package/docs/PLAN-non-agui-agent-support.md

@@ -0,0 +1,525 @@
+<!--
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+-->
+
+# Plan: Non-AG-UI Agent Support in Agent Health
+
+> **Status: Implemented** via the connector system (not the separate adapter pattern originally proposed here).
+> See `services/connectors/strands/` and `services/connectors/langgraph/` for the Strands and LangGraph implementations.
+
+## Problem Statement
+
+Currently, Agent Health only supports agents that emit AG-UI protocol SSE events. This excludes:
+- **Claude Code** - CLI tool with its own execution model
+- **LangGraph non-AG-UI instances** - LangGraph without AG-UI endpoints
+- **Amazon Strands Agents** - AWS agent framework
+
+## Key Insight
+
+OTEL traces alone are insufficient for evaluation - they typically capture metadata (timing, spans) but not the **actual content** (full responses, tool outputs) needed for LLM judge evaluation.
+
+Reference implementations:
+- **Strands Evals**: Direct execution captures output, OTEL supplements with trajectory
+- **LangSmith**: Rich tracing with full content at every step
+
+## Solution Approach
+
+**Unified Agent Adapter Pattern** - One interface that works for API, CLI, and streaming agents:
+
+```typescript
+interface AgentAdapter {
+  execute(prompt: string, context?: any): Promise<AgentResult>;
+}
+
+interface AgentResult {
+  output: string;                    // The actual response (required)
+  trajectory?: TrajectoryStep[];     // Execution steps (optional)
+  metadata?: Record<string, any>;    // Agent-specific data
+}
+```
+
+Each agent type has an adapter that:
+1. **Executes the agent** (CLI, API, or SSE)
+2. **Captures the actual output** directly
+3. **Optionally captures trajectory** from output parsing or OTEL
+4. **Returns unified result** for evaluation
+
+---
+
+## Architecture Overview
+
+```
+                         ┌──────────────────────┐
+                         │      Agent Health       │
+                         │   (Experiment Runner)│
+                         └──────────┬───────────┘
+                                    │ calls adapter.execute()
+                         ┌──────────▼───────────┐
+                         │   Agent Adapter      │
+                         │   Registry           │
+                         └──────────┬───────────┘
+            ┌───────────────────────┼───────────────────────┐
+            │                       │                       │
+     ┌──────▼──────┐        ┌──────▼──────┐        ┌──────▼──────┐
+     │ Claude Code │        │   Strands   │        │  LangGraph  │
+     │   Adapter   │        │   Adapter   │        │   Adapter   │
+     │    (CLI)    │        │   (API)     │        │  (AG-UI)    │
+     └──────┬──────┘        └──────┬──────┘        └──────┬──────┘
+            │                      │                      │
+            │ exec CLI             │ AWS SDK              │ SSE stream
+            ▼                      ▼                      ▼
+     ┌─────────────┐        ┌─────────────┐        ┌─────────────┐
+     │ claude CLI  │        │ Strands API │        │ LangGraph   │
+     └─────────────┘        └─────────────┘        └─────────────┘
+            │                      │                      │
+            └──────────────────────┼──────────────────────┘
+                                   │
+                         ┌─────────▼─────────┐
+                         │   AgentResult     │
+                         │ {output, trajectory}│
+                         └─────────┬─────────┘
+                                   │
+                         ┌─────────▼─────────┐
+                         │    LLM Judge      │
+                         │   (unchanged)     │
+                         └───────────────────┘
+```
+
+---
+
+## Phase 1: Agent Adapter Framework (Foundation)
+
+### 1.1 Core Types
+
+**File: `types/index.ts`** - Add:
+```typescript
+// Adapter types
+export type AdapterType = 'ag-ui' | 'cli' | 'api' | 'custom';
+
+export interface AgentResult {
+  output: string;                    // Required: the actual response
+  trajectory?: TrajectoryStep[];     // Optional: execution steps
+  metadata?: {
+    executionTime?: number;
+    tokenCount?: number;
+    traceId?: string;
+    [key: string]: any;
+  };
+}
+
+export interface AgentAdapter {
+  type: AdapterType;
+  execute(
+    prompt: string,
+    context?: { tools?: any[]; history?: any[] }
+  ): Promise<AgentResult>;
+
+  // Optional streaming support
+  executeStream?(
+    prompt: string,
+    context?: any,
+    onChunk?: (chunk: string) => void
+  ): Promise<AgentResult>;
+}
+
+// Extended AgentConfig
+export interface AgentConfig {
+  // ... existing fields ...
+  adapterType: AdapterType;          // Required: which adapter to use
+  adapterConfig?: {
+    command?: string;                // For CLI adapters
+    apiEndpoint?: string;            // For API adapters
+    timeout?: number;
+    [key: string]: any;
+  };
+}
+```
+
+### 1.2 Adapter Registry
+
+**New file: `services/adapters/index.ts`**
+```typescript
+import { AgentAdapter, AgentConfig } from '@/types';
+
+// Registry of available adapters
+const adapters: Record<string, new (config: AgentConfig) => AgentAdapter> = {};
+
+export function registerAdapter(
+  type: string,
+  adapterClass: new (config: AgentConfig) => AgentAdapter
+): void;
+
+export function getAdapter(config: AgentConfig): AgentAdapter;
+
+export function listAdapters(): string[];
+```
+
+### 1.3 Base Adapter Class
+
+**New file: `services/adapters/baseAdapter.ts`**
+```typescript
+export abstract class BaseAdapter implements AgentAdapter {
+  protected config: AgentConfig;
+
+  constructor(config: AgentConfig) {
+    this.config = config;
+  }
+
+  abstract execute(prompt: string, context?: any): Promise<AgentResult>;
+
+  // Helper to build trajectory from raw output
+  protected parseTrajectory(output: string): TrajectoryStep[] {
+    // Default implementation - can be overridden
+  }
+}
+```
+
+---
+
+## Phase 2: Built-in Adapters
+
+### 2.1 AG-UI Adapter (Existing Flow)
+
+**New file: `services/adapters/aguiAdapter.ts`**
+```typescript
+/**
+ * Wraps existing AG-UI flow as an adapter
+ * Uses SSEClient + AGUIToTrajectoryConverter internally
+ */
+export class AGUIAdapter extends BaseAdapter {
+  type = 'ag-ui' as const;
+
+  async execute(prompt: string, context?: any): Promise<AgentResult> {
+    // Use existing AG-UI streaming flow
+    const trajectory = await streamAgentExecution(
+      this.config.endpoint,
+      buildPayload(prompt, context),
+      this.config.headers
+    );
+
+    return {
+      output: extractFinalResponse(trajectory),
+      trajectory,
+    };
+  }
+
+  async executeStream(prompt, context, onChunk): Promise<AgentResult> {
+    // Real-time streaming with callbacks
+  }
+}
+```
+
+### 2.2 Claude Code Adapter (CLI)
+
+**New file: `services/adapters/claudeCodeAdapter.ts`**
+```typescript
+import { spawn } from 'child_process';
+
+export class ClaudeCodeAdapter extends BaseAdapter {
+  type = 'cli' as const;
+
+  async execute(prompt: string, context?: any): Promise<AgentResult> {
+    const { command = 'claude', timeout = 300000 } = this.config.adapterConfig || {};
+
+    // Execute: claude --print "prompt" --output-format json
+    const result = await this.runCLI(command, [
+      '--print', prompt,
+      '--output-format', 'stream-json',  // Get structured output
+    ], timeout);
+
+    return {
+      output: result.finalResponse,
+      trajectory: this.parseClaudeCodeOutput(result.events),
+      metadata: { executionTime: result.duration },
+    };
+  }
+
+  private parseClaudeCodeOutput(events: any[]): TrajectoryStep[] {
+    // Parse Claude Code's JSON output format
+    // Maps: assistant → response, tool_use → action, tool_result → tool_result
+  }
+}
+```
+
+### 2.3 Strands Adapter (AWS API)
+
+**New file: `services/adapters/strandsAdapter.ts`**
+```typescript
+import { BedrockAgentRuntimeClient, InvokeAgentCommand } from '@aws-sdk/client-bedrock-agent-runtime';
+
+export class StrandsAdapter extends BaseAdapter {
+  type = 'api' as const;
+  private client: BedrockAgentRuntimeClient;
+
+  async execute(prompt: string, context?: any): Promise<AgentResult> {
+    const { agentId, agentAliasId, region } = this.config.adapterConfig || {};
+
+    const response = await this.client.send(new InvokeAgentCommand({
+      agentId,
+      agentAliasId,
+      sessionId: context?.sessionId || generateSessionId(),
+      inputText: prompt,
+    }));
+
+    return {
+      output: await this.extractCompletion(response),
+      trajectory: this.parseStrandsTrace(response.trace),
+      metadata: { sessionId: response.sessionId },
+    };
+  }
+
+  private parseStrandsTrace(trace: any): TrajectoryStep[] {
+    // Parse Strands trace format to TrajectoryStep[]
+  }
+}
+```
+
+### 2.4 LangGraph Adapter (Non-AG-UI)
+
+**New file: `services/adapters/langgraphAdapter.ts`**
+```typescript
+export class LangGraphAdapter extends BaseAdapter {
+  type = 'api' as const;
+
+  async execute(prompt: string, context?: any): Promise<AgentResult> {
+    const { apiEndpoint } = this.config.adapterConfig || {};
+
+    // Call LangGraph API directly (not AG-UI SSE)
+    const response = await fetch(apiEndpoint, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ input: prompt, ...context }),
+    });
+
+    const data = await response.json();
+
+    return {
+      output: data.output,
+      trajectory: this.parseLangGraphOutput(data),
+    };
+  }
+}
+```
+
+### 2.5 Custom Adapter (User-Provided)
+
+**New file: `services/adapters/customAdapter.ts`**
+```typescript
+/**
+ * Allows users to provide a task function (like Strands Evals pattern)
+ */
+export class CustomAdapter extends BaseAdapter {
+  type = 'custom' as const;
+  private taskFunction: (prompt: string, context?: any) => Promise<AgentResult>;
+
+  setTaskFunction(fn: typeof this.taskFunction): void {
+    this.taskFunction = fn;
+  }
+
+  async execute(prompt: string, context?: any): Promise<AgentResult> {
+    if (!this.taskFunction) {
+      throw new Error('Custom adapter requires a task function');
+    }
+    return this.taskFunction(prompt, context);
+  }
+}
+```
+
+---
+
+## Phase 3: Integration & UI
+
+### 3.1 Update Experiment Runner
+
+**File: `services/experimentRunner.ts`** - Refactor:
+```typescript
+import { getAdapter } from './adapters';
+
+async function executeTestCase(run, testCase) {
+  const agent = getAgentConfig(run.agentKey);
+  const adapter = getAdapter(agent);
+
+  // Unified execution through adapter
+  const result = await adapter.execute(
+    testCase.initialPrompt,
+    { tools: testCase.tools, history: [] }
+  );
+
+  // Evaluate with judge (same for all adapters)
+  return await evaluateResult(result, testCase);
+}
+```
+
+### 3.2 Update Agent Configuration UI
+
+**File: `components/ConfigEditor.tsx`** - Add:
+- Adapter type selector dropdown
+- Adapter-specific configuration fields
+- Test connection button
+
+### 3.3 Update QuickRunModal
+
+**File: `components/QuickRunModal.tsx`**
+- Show adapter type indicator
+- Streaming output for adapters that support it
+- Progress indication for non-streaming adapters
+
+### 3.4 Add Agent Configuration
+
+**File: `lib/constants.ts`** - Add new agents:
+```typescript
+{
+  key: "claude-code",
+  name: "Claude Code",
+  adapterType: "cli",
+  adapterConfig: {
+    command: "claude",
+    timeout: 300000,
+  },
+  models: ["claude-sonnet-4", "claude-opus-4"],
+  description: "Anthropic's Claude Code CLI agent",
+},
+{
+  key: "strands",
+  name: "Amazon Strands",
+  adapterType: "api",
+  adapterConfig: {
+    agentId: "${STRANDS_AGENT_ID}",
+    agentAliasId: "${STRANDS_ALIAS_ID}",
+    region: "us-east-1",
+  },
+  models: ["claude-sonnet-4", "amazon-nova"],
+  description: "Amazon Strands agent framework",
+},
+{
+  key: "langgraph-api",
+  name: "LangGraph (API)",
+  adapterType: "api",
+  adapterConfig: {
+    apiEndpoint: "${LANGGRAPH_API_ENDPOINT}",
+  },
+  models: ["claude-sonnet-4.5"],
+  description: "LangGraph agent via direct API",
+}
+```
+
+---
+
+## Phase 4: OTEL Trace Enrichment (Optional)
+
+For agents that emit OTEL traces, add optional trace fetching to enrich trajectory:
+
+### 4.1 Trace Enrichment Service
+
+**New file: `services/adapters/traceEnrichment.ts`**
+```typescript
+/**
+ * Optionally fetch OTEL traces to enrich trajectory with timing/metadata
+ */
+export async function enrichWithTraces(
+  result: AgentResult,
+  traceConfig: { serviceName: string; traceId?: string }
+): Promise<AgentResult> {
+  if (!result.metadata?.traceId) return result;
+
+  const spans = await fetchTraces(traceConfig);
+
+  return {
+    ...result,
+    trajectory: mergeTrajectoryWithSpans(result.trajectory, spans),
+  };
+}
+```
+
+---
+
+## Files Summary
+
+### New Files
+| File | Purpose |
+|------|---------|
+| `services/adapters/index.ts` | Adapter registry and factory |
+| `services/adapters/baseAdapter.ts` | Base adapter class |
+| `services/adapters/aguiAdapter.ts` | AG-UI adapter (wraps existing) |
+| `services/adapters/claudeCodeAdapter.ts` | Claude Code CLI adapter |
+| `services/adapters/strandsAdapter.ts` | Amazon Strands API adapter |
+| `services/adapters/langgraphAdapter.ts` | LangGraph API adapter |
+| `services/adapters/customAdapter.ts` | User-provided task function |
+| `services/adapters/traceEnrichment.ts` | Optional OTEL trace enrichment |
+
+### Modified Files
+| File | Changes |
+|------|---------|
+| `types/index.ts` | Add `AdapterType`, `AgentResult`, `AgentAdapter` |
+| `lib/constants.ts` | Add Claude Code, Strands, LangGraph configs |
+| `services/experimentRunner.ts` | Use adapter pattern for execution |
+| `components/QuickRunModal.tsx` | Adapter-aware UI |
+| `components/ConfigEditor.tsx` | Adapter configuration UI |
+
+---
+
+## Implementation Order
+
+1. **Phase 1** - Adapter Framework
+   - Core types (`AgentResult`, `AgentAdapter`)
+   - Adapter registry
+   - Base adapter class
+
+2. **Phase 2** - Built-in Adapters
+   - AG-UI adapter (wrap existing)
+   - Claude Code adapter (CLI)
+   - Strands adapter (AWS API)
+   - LangGraph adapter (API)
+
+3. **Phase 3** - Integration & UI
+   - Update experiment runner
+   - Configuration UI
+   - Quick run modal updates
+
+4. **Phase 4** - OTEL Enrichment (Optional)
+   - Trace enrichment service
+   - Merge spans with trajectory
+
+---
+
+## Environment Variables
+
+```bash
+# Claude Code
+CLAUDE_CODE_COMMAND=claude
+CLAUDE_CODE_TIMEOUT=300000
+
+# Amazon Strands
+STRANDS_AGENT_ID=your-agent-id
+STRANDS_ALIAS_ID=your-alias-id
+AWS_REGION=us-east-1
+
+# LangGraph (non-AG-UI)
+LANGGRAPH_API_ENDPOINT=http://localhost:8000/invoke
+
+# Optional: OTEL for trace enrichment
+OTEL_TRACE_ENDPOINT=http://localhost:16686
+```
+
+---
+
+## Verification Plan
+
+1. **Unit tests**:
+   - Each adapter's output parsing
+   - Adapter registry functionality
+   - AgentResult validation
+
+2. **Integration tests**:
+   - Claude Code adapter with mock CLI
+   - Strands adapter with mock AWS SDK
+   - AG-UI adapter (existing flow)
+
+3. **E2E tests**:
+   - Mixed-adapter experiment (AG-UI + CLI)
+   - Comparison view with different adapters
+
+4. **Manual testing**:
+   - Run Claude Code via adapter, evaluate
+   - Compare Claude Code vs AG-UI agent on same test case