@opensearch-project/agent-health 0.3.0 → 0.5.2

package/observio-sample-agent/pi-package/skills/implement-feature/SKILL.md

@@ -0,0 +1,156 @@
+---
+name: implement-feature
+description: >
+  Guided feature implementation for Agent Health. Use when a user wants to add a new
+  capability, endpoint, CLI command, or UI component to the Agent Health project.
+  Ensures correct architecture layer, coding conventions, test coverage, and PR readiness.
+---
+
+# Implement Feature in Agent Health
+
+You are helping the user implement a feature in the Agent Health project. Follow the project's architecture and conventions strictly.
+
+## Step 1: Understand the Request
+
+Clarify:
+1. What does the feature do? (user-facing behavior)
+2. Who triggers it? (UI, CLI, API, background process)
+3. What data does it read/write? (OpenSearch indexes, config, files)
+
+## Step 2: Identify the Architecture Layer
+
+Use the `find_architecture` tool to locate where this feature belongs. The layers are:
+
+| Trigger | Implementation Layer | Key Pattern |
+|---------|---------------------|-------------|
+| HTTP API | `server/routes/` | Route validation, early return 404, `{ error }` responses |
+| CLI command | `cli/commands/` | Uses `apiClient.ts` to call server, `ensureServer()` for lifecycle |
+| Background work | `services/` | SSE streaming, cancellation tokens, persist state immediately |
+| UI interaction | `components/` + `hooks/` | React, shadcn/ui, client service calls |
+| Data persistence | `services/storage/` | `asyncXStorage` with `toAppFormat()`/`toStorageFormat()` |
+| Agent protocol | `services/connectors/` | ConnectorType, streaming adapter |
+
+**Critical rule**: All clients access OpenSearch through the server HTTP API. Never bypass the server.
+
+## Step 3: Plan the Implementation
+
+Before writing code, identify:
+- **Files to create**: New routes, services, types, tests
+- **Files to modify**: Existing types, route registrations, CLI command list
+- **Types needed**: Add to `types/index.ts` — use string unions (not enums), discriminated unions with `type` field
+
+### Coding Conventions to Follow
+
+- **Route validation**: `validateX(): string | null`, return 400 early
+- **Storage wrappers**: `toAppFormat()`/`toStorageFormat()` conversion
+- **SSE streaming**: Set event-stream headers, `res.flushHeaders()`, structured events
+- **Cancellation**: `{ isCancelled, cancel() }` token in a Map, check before each iteration
+- **Run status**: `pending` → `running` → `completed` | `failed` | `cancelled`
+- **Exports**: Barrel exports via `index.ts` with named functions
+- **Path alias**: Use `@/` prefix for all imports (maps to project root)
+- **Types**: `RunConfigInput` (user fields) vs full entity. String unions not enums.
+
+### What NOT to Do
+
+- Don't add unnecessary error handling for internal code paths
+- Don't add docstrings/comments to code you didn't change
+- Don't create helpers/abstractions for one-time operations
+- Don't add feature flags or backwards-compatibility shims
+- Don't modify `package.json` name or version fields
+
+## Step 4: Implement
+
+Write the code following the plan. For each file:
+
+1. Add SPDX header: `/* Copyright OpenSearch Contributors\n SPDX-License-Identifier: Apache-2.0 */`
+2. Use `@/` path alias for imports
+3. Follow existing patterns in the same directory
+
+### If Adding a Route
+
+```typescript
+// server/routes/myFeature.ts
+import { Router } from 'express';
+
+const router = Router();
+
+function validateInput(body: any): string | null {
+  if (!body.requiredField) return 'requiredField is required';
+  return null;
+}
+
+router.post('/api/my-feature', async (req, res) => {
+  const error = validateInput(req.body);
+  if (error) return res.status(400).json({ error });
+  // ...implementation
+});
+
+export default router;
+```
+
+### If Adding a CLI Command
+
+```typescript
+// cli/commands/myCommand.ts
+// Uses apiClient — NEVER calls OpenSearch directly
+import { apiClient } from '../utils/apiClient';
+import { ensureServer } from '../utils/serverLifecycle';
+
+export async function myCommand(args: any) {
+  await ensureServer();
+  const result = await apiClient.get('/api/my-feature');
+  // ...
+}
+```
+
+### If Adding a Storage Entity
+
+```typescript
+// services/storage/asyncMyStorage.ts
+// Follow the asyncTestCaseStorage.ts pattern exactly
+export const asyncMyStorage = {
+  async create(entity: MyType): Promise<MyType> { /* ... */ },
+  async get(id: string): Promise<MyType | null> { /* ... */ },
+  async update(id: string, updates: Partial<MyType>): Promise<MyType> {
+    // MUST throw if entity doesn't exist (no upsert)
+  },
+  async delete(id: string): Promise<void> { /* ... */ },
+  async list(): Promise<MyType[]> { /* ... */ },
+};
+```
+
+## Step 5: Write Tests
+
+Use the `/write-test` skill or follow these rules:
+- Place tests in `tests/` mirroring source structure
+- Use `@/` path alias, never relative imports
+- Coverage thresholds: 90% lines/statements, 80% functions/branches
+- Mock external services (OpenSearch, Bedrock), not internal modules
+- Integration tests that create data MUST clean up in `afterEach`/`afterAll`
+
+## Step 6: Validate
+
+Run `run_validation` tool with scope "all" to verify:
+- TypeScript compiles cleanly (`npm run build:all`)
+- All tests pass (`npm run test:all`)
+- No coverage regression
+
+## Step 7: Prepare PR
+
+Use `check_compliance` tool to verify:
+- [ ] SPDX headers on all new files
+- [ ] CHANGELOG.md updated
+- [ ] All commits have DCO signoff (`git commit -s`)
+- [ ] Build passes, tests pass
+- [ ] No `--no-verify` bypasses
+
+Then create the PR following the project template:
+```
+## Summary
+- <1-3 bullet points of what changed>
+
+## Test plan
+- [ ] Unit tests added/updated
+- [ ] Integration tests pass
+- [ ] Manual verification steps
+```

package/observio-sample-agent/pi-package/skills/instrument-otel/SKILL.md

@@ -0,0 +1,208 @@
+---
+name: instrument-otel
+description: >
+  Add OpenTelemetry instrumentation to an AI agent for Agent Health observability.
+  Use when a user wants to make their agent's traces visible in Agent Health dashboards,
+  or when debugging why traces aren't appearing. Covers span structure, Gen AI semantic
+  conventions, required attributes, and OTLP exporter setup.
+---
+
+# Instrument Agent with OpenTelemetry
+
+You are helping the user add OpenTelemetry instrumentation to their AI agent so it produces traces that Agent Health can consume and visualize.
+
+## What Agent Health Expects
+
+Agent Health categorizes spans by reading `gen_ai.operation.name`:
+- `invoke_agent` → AGENT span (root)
+- `chat` → LLM span (model calls)
+- `execute_tool` → TOOL span (tool invocations)
+
+All other spans are categorized as FRAMEWORK.
+
+## Step 1: Determine the User's Stack
+
+Ask the user:
+1. What language/runtime? (Node.js, Python, Java, Go)
+2. What LLM provider? (Bedrock, OpenAI, Anthropic, etc.)
+3. What framework? (LangChain, LangGraph, Strands, custom)
+4. Where should traces export to? (OSIS pipeline, local collector, direct to OpenSearch)
+
+## Step 2: Set Up the Tracer Provider
+
+Generate a telemetry provider module. Requirements:
+- Use `BatchSpanProcessor` (not Simple) for production
+- Set `service.name` resource attribute to identify the agent
+- Export via OTLP/HTTP to the configured endpoint
+- Make it opt-in via environment variable
+
+### Reference Pattern (Node.js/TypeScript)
+
+```typescript
+import { NodeTracerProvider, BatchSpanProcessor } from '@opentelemetry/sdk-trace-node';
+import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
+import { resourceFromAttributes } from '@opentelemetry/resources';
+
+const resource = resourceFromAttributes({
+  'service.name': process.env.OTEL_SERVICE_NAME || 'my-agent',
+});
+
+const provider = new NodeTracerProvider({
+  resource,
+  spanProcessors: [
+    new BatchSpanProcessor(
+      new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT })
+    ),
+  ],
+});
+provider.register();
+```
+
+### Python Equivalent
+
+```python
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanExporter
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+
+resource = Resource.create({"service.name": "my-agent"})
+provider = TracerProvider(resource=resource)
+provider.add_span_processor(BatchSpanExporter(OTLPSpanExporter()))
+trace.set_tracer_provider(provider)
+```
+
+## Step 3: Create Span Helpers
+
+Generate helper functions for each span type. These MUST follow the naming and attribute conventions exactly.
+
+### Root Agent Span
+
+```typescript
+function startAgentSpan(runId: string) {
+  const span = tracer.startSpan('invoke_agent <agent-name>', {
+    kind: SpanKind.SERVER,
+    attributes: {
+      'gen_ai.operation.name': 'invoke_agent',
+      'gen_ai.system': '<provider>',        // e.g. 'aws.bedrock'
+      'gen_ai.agent.name': '<agent-name>',
+      'gen_ai.conversation.id': runId,   // links the trace to an Agent Health run (also accepted as agent_health.run.id)
+    },
+  });
+  return { span, ctx: trace.setSpan(context.active(), span) };
+}
+```
+
+### LLM Call Span (child of agent)
+
+```typescript
+function startLLMSpan(parentCtx: Context, opts: { modelId: string; temperature?: number; maxTokens?: number }) {
+  const span = tracer.startSpan('chat <system>', {
+    kind: SpanKind.CLIENT,
+    attributes: {
+      'gen_ai.operation.name': 'chat',
+      'gen_ai.system': '<provider>',
+      'gen_ai.request.model': opts.modelId,
+      'gen_ai.request.temperature': opts.temperature,
+      'gen_ai.request.max_tokens': opts.maxTokens,
+    },
+  }, parentCtx);
+  return { span, ctx: trace.setSpan(parentCtx, span) };
+}
+
+function endLLMSpan(span: Span, opts: { inputTokens?: number; outputTokens?: number; stopReason?: string }) {
+  if (opts.inputTokens) span.setAttribute('gen_ai.usage.input_tokens', opts.inputTokens);
+  if (opts.outputTokens) span.setAttribute('gen_ai.usage.output_tokens', opts.outputTokens);
+  if (opts.stopReason) span.setAttribute('gen_ai.response.finish_reason', opts.stopReason);
+
+  // Add content events
+  span.addEvent('gen_ai.content.prompt', { 'gen_ai.prompt': '...' });
+  span.addEvent('gen_ai.content.completion', { 'gen_ai.completion': '...' });
+  span.end();
+}
+```
+
+### Tool Execution Span (child of agent)
+
+```typescript
+function startToolSpan(parentCtx: Context, toolName: string, toolInput?: any) {
+  const span = tracer.startSpan(`execute_tool ${toolName}`, {
+    kind: SpanKind.CLIENT,
+    attributes: {
+      'gen_ai.operation.name': 'execute_tool',
+      'gen_ai.tool.name': toolName,
+    },
+  }, parentCtx);
+
+  if (toolInput) {
+    span.addEvent('gen_ai.tool.input', {
+      'gen_ai.tool.input': JSON.stringify(toolInput).slice(0, 3000),
+    });
+  }
+  return { span, ctx: trace.setSpan(parentCtx, span) };
+}
+
+function endToolSpan(span: Span, result?: any, error?: Error) {
+  if (result) {
+    span.addEvent('gen_ai.tool.output', {
+      'gen_ai.tool.output': JSON.stringify(result).slice(0, 3000),
+    });
+  }
+  if (error) span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
+  span.end();
+}
+```
+
+## Step 4: Wire Into the Agent Loop
+
+Show the user where to place span calls in their agent's execution flow:
+
+```
+┌─────────────────────────────────────────────┐
+│ startAgentSpan(runId)                        │  ← Root span
+│                                              │
+│   ┌─── Loop ───────────────────────────┐    │
+│   │ startLLMSpan(agentCtx, {...})       │    │  ← Each model call
+│   │   → call LLM                        │    │
+│   │ endLLMSpan(span, {tokens, reason})  │    │
+│   │                                      │    │
+│   │ if tool_calls:                       │    │
+│   │   startToolSpan(agentCtx, name)     │    │  ← Each tool execution
+│   │     → execute tool                   │    │
+│   │   endToolSpan(span, result)         │    │
+│   └─────────────────────────────────────┘    │
+│                                              │
+│ agentSpan.end()                              │  ← End root span
+└─────────────────────────────────────────────┘
+```
+
+## Step 5: Verify
+
+After instrumentation, help the user verify:
+
+1. **Exporter connectivity**: Check OTLP endpoint is reachable
+2. **Span hierarchy**: Root agent span has LLM + tool children
+3. **Required attributes**: All `gen_ai.*` attributes present
+4. **Events**: Prompt/completion events on LLM spans, input/output on tool spans
+5. **Agent Health visibility**: Traces appear in the Traces tab with correct categorization
+
+Use the `validate_spans` tool to check their code automatically.
+
+## Common Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| No traces in Agent Health | Exporter endpoint wrong or unreachable | Check `OTEL_EXPORTER_OTLP_ENDPOINT` |
+| Spans show as FRAMEWORK | Missing `gen_ai.operation.name` attribute | Add the attribute to every span |
+| No token metrics | Missing `gen_ai.usage.*` attributes | Set on LLM span end |
+| Flat span list (no tree) | Not passing parent context | Use `parentCtx` in startSpan |
+| Traces appear but no cost data | Missing model attribute | Set `gen_ai.request.model` |
+
+## Environment Variables
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-osis-pipeline/v1/traces
+OTEL_SERVICE_NAME=my-agent
+OTEL_ENABLED=true  # optional, default true
+```

package/observio-sample-agent/pi-package/skills/setup-collector/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: setup-collector
+description: >
+  Set up an OpenTelemetry collector or OSIS pipeline to route traces from an agent
+  to OpenSearch for Agent Health consumption. Covers OSIS, OTel Collector, and
+  direct export configurations.
+---
+
+# Set Up Trace Collection Pipeline
+
+You are helping the user configure a pipeline to get their agent's OTel traces into OpenSearch where Agent Health can read them.
+
+## Architecture Options
+
+```
+Agent → OTLP/HTTP → [Pipeline] → OpenSearch
+```
+
+Three pipeline options:
+
+| Option | Best For | Complexity |
+|--------|----------|------------|
+| OSIS (OpenSearch Ingestion) | AWS-managed, production | Low (managed) |
+| OTel Collector | Self-hosted, flexible routing | Medium |
+| Direct export | Dev/testing only | Lowest |
+
+## Option 1: OSIS Pipeline (Recommended for AWS)
+
+### Prerequisites
+- AWS account with OSIS access
+- OpenSearch domain (managed or serverless)
+
+### Pipeline Configuration
+
+```yaml
+version: "2"
+otel-trace-pipeline:
+  source:
+    otel_trace_source:
+      path: "/v1/traces"
+  processor:
+    - otel_trace_raw:
+  sink:
+    - opensearch:
+        hosts: ["https://your-opensearch-domain.us-west-2.es.amazonaws.com"]
+        index: "otel-v1-apm-span-*"
+        aws:
+          sts_role_arn: "arn:aws:iam::ACCOUNT:role/osis-role"
+          region: "us-west-2"
+```
+
+### Agent Configuration
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-osis-pipeline.us-west-2.osis.amazonaws.com/v1/traces
+OTEL_SERVICE_NAME=my-agent
+```
+
+## Option 2: OTel Collector (Self-hosted)
+
+### docker-compose.yml
+
+```yaml
+services:
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    ports:
+      - "4318:4318"  # OTLP HTTP
+    volumes:
+      - ./otel-config.yaml:/etc/otelcol-contrib/config.yaml
+```
+
+### otel-config.yaml
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  batch:
+    timeout: 5s
+    send_batch_size: 100
+
+exporters:
+  opensearch:
+    http:
+      endpoint: "https://your-opensearch:9200"
+      tls:
+        insecure: false
+    dataset: "traces"
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [opensearch]
+```
+
+### Agent Configuration
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces
+OTEL_SERVICE_NAME=my-agent
+```
+
+## Option 3: Direct Export (Dev Only)
+
+Export directly to OpenSearch Data Prepper format. Not recommended for production (no batching, no retry).
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-opensearch:9200/_plugins/_trace_analytics/v1/traces
+```
+
+## Agent Health Configuration
+
+Once traces are flowing to OpenSearch, configure Agent Health to read them:
+
+```bash
+# In your Agent Health .env
+OPENSEARCH_LOGS_ENDPOINT=https://your-opensearch:9200
+OPENSEARCH_LOGS_USERNAME=admin
+OPENSEARCH_LOGS_PASSWORD=admin
+# Or for AWS SigV4:
+# OPENSEARCH_LOGS_AUTH_TYPE=sigv4
+# OPENSEARCH_LOGS_AWS_REGION=us-west-2
+```
+
+## Verification Checklist
+
+1. **Agent sends spans**: Check agent logs for "telemetry enabled" message
+2. **Pipeline receives**: Check OSIS/Collector metrics for incoming spans
+3. **OpenSearch has data**: Query `otel-v1-apm-span-*` index
+4. **Agent Health sees traces**: Open Traces tab, verify spans appear with correct categorization
+
+## Troubleshooting
+
+| Issue | Check |
+|-------|-------|
+| No spans arriving | Is `OTEL_EXPORTER_OTLP_ENDPOINT` correct? Is the agent calling `provider.register()`? |
+| Spans in OS but not in Agent Health | Check index pattern matches what Agent Health queries (`otel-v1-apm-span-*`) |
+| Auth errors | Check IAM role for OSIS, or username/password for basic auth |
+| Spans arrive but incomplete | Ensure `provider.shutdown()` is called on agent exit (flushes batch) |

package/observio-sample-agent/pi-package/skills/write-test/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: write-test
+description: >
+  Write tests for Agent Health following project conventions. Use when adding tests
+  for new features, bug fixes, or improving coverage. Covers Jest setup, mocking
+  patterns, path aliases, and coverage thresholds.
+---
+
+# Write Tests for Agent Health
+
+You are helping the user write tests following Agent Health's conventions.
+
+## Test Structure
+
+- Tests live in `tests/` mirroring the source directory structure
+- Jest config: `jest.config.cjs`
+- **Always use `@/` path alias** — never relative imports
+
+```typescript
+// ✅ Correct
+import { myFunction } from '@/services/myService';
+
+// ❌ Wrong
+import { myFunction } from '../../services/myService';
+```
+
+## Test File Template
+
+```typescript
+/* Copyright OpenSearch Contributors
+ SPDX-License-Identifier: Apache-2.0 */
+
+import { functionUnderTest } from '@/path/to/module';
+
+describe('functionUnderTest', () => {
+  beforeEach(() => {
+    jest.clearAllMocks();
+  });
+
+  it('should handle the happy path', () => {
+    const result = functionUnderTest(validInput);
+    expect(result).toEqual(expectedOutput);
+  });
+
+  it('should handle edge case', () => {
+    // ...
+  });
+});
+```
+
+## Mocking Patterns
+
+### Mock External Services (OpenSearch, Bedrock)
+
+```typescript
+jest.mock('@/server/adapters/opensearch/StorageModule', () => ({
+  getStorageModule: () => ({
+    testCases: {
+      create: jest.fn().mockResolvedValue({ id: 'test-1', name: 'Test' }),
+      get: jest.fn().mockResolvedValue(null),
+    },
+  }),
+}));
+```
+
+### Mock Fetch/HTTP
+
+```typescript
+const mockFetch = jest.fn().mockResolvedValue({
+  ok: true,
+  json: () => Promise.resolve({ data: 'result' }),
+});
+global.fetch = mockFetch;
+```
+
+### Don't Mock Internal Modules
+
+Test through the public API. Only mock at system boundaries.
+
+## Integration Tests
+
+If the test creates data in OpenSearch or starts a server:
+
+```typescript
+afterEach(async () => {
+  // Clean up created resources
+  await storage.delete(createdId);
+});
+
+afterAll(async () => {
+  // Close connections
+  await server.close();
+});
+```
+
+## Coverage Thresholds
+
+| Metric | Minimum |
+|--------|---------|
+| Lines | 90% |
+| Statements | 90% |
+| Functions | 80% |
+| Branches | 80% |
+
+Run coverage: `npm test -- --coverage`
+
+## Running Tests
+
+```bash
+npm test                           # All tests
+npm run test:unit                  # Unit only
+npm run test:integration           # Integration only
+npm test -- path/to/file.test.ts   # Single file
+npm test -- --watch                # Watch mode
+```