@opensearch-project/agent-health 0.3.0 → 0.5.2

package/docs/INSTRUMENT_WITH_OTEL.md

@@ -0,0 +1,390 @@
+<!--
+  * Copyright OpenSearch Contributors
+  * SPDX-License-Identifier: Apache-2.0
+-->
+
+# Instrument Your Agent with OpenTelemetry
+
+This guide is a prompt you can give to a coding agent (Claude Code, Cursor, Copilot, etc.) to instrument your application with OpenTelemetry and route telemetry to the Agent Health observability stack.
+
+---
+
+## Quick Start: Give This Prompt to Your Coding Agent
+
+Copy the prompt below and paste it into your coding agent. It will instrument your application with the correct OTel attributes that Agent Health expects.
+
+---
+
+## The Prompt
+
+````markdown
+# Task: Instrument this application with OpenTelemetry
+
+Add OpenTelemetry instrumentation to this application following the GenAI semantic conventions so that traces are compatible with the Agent Health observability dashboard.
+
+## Requirements
+
+### 1. Install Dependencies
+
+For Python:
+```bash
+pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-semantic-conventions
+```
+
+For TypeScript/Node.js:
+```bash
+npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/semantic-conventions
+```
+
+### 2. Initialize the Tracer
+
+Configure an OTLP exporter that sends traces to the endpoint specified by `OTEL_EXPORTER_OTLP_ENDPOINT`. Use `http/protobuf` protocol.
+
+**Python:**
+```python
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+import os
+
+resource = Resource.create({
+    "service.name": os.getenv("OTEL_SERVICE_NAME", "my-agent"),
+})
+
+provider = TracerProvider(resource=resource)
+exporter = OTLPSpanExporter(
+    endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318") + "/v1/traces",
+)
+provider.add_span_processor(BatchSpanProcessor(exporter))
+trace.set_tracer_provider(provider)
+tracer = trace.get_tracer("my-agent")
+```
+
+**TypeScript:**
+```typescript
+import { NodeSDK } from '@opentelemetry/sdk-node';
+import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
+import { Resource } from '@opentelemetry/resources';
+
+const sdk = new NodeSDK({
+  resource: new Resource({
+    'service.name': process.env.OTEL_SERVICE_NAME || 'my-agent',
+  }),
+  traceExporter: new OTLPTraceExporter({
+    url: (process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318') + '/v1/traces',
+  }),
+});
+sdk.start();
+
+import { trace } from '@opentelemetry/api';
+const tracer = trace.getTracer('my-agent');
+```
+
+### 3. Span Structure (Required)
+
+Create spans in this hierarchy:
+
+```
+Root span: agent invocation (gen_ai.operation.name = "invoke_agent")
+├── LLM call (gen_ai.operation.name = "chat")
+├── Tool execution (gen_ai.operation.name = "execute_tool")
+├── LLM call (gen_ai.operation.name = "chat")
+└── ...
+```
+
+### 4. Required Span Attributes
+
+These attributes MUST be set on spans for Agent Health to correctly categorize and compute metrics.
+
+#### Agent Spans (root span)
+
+| Attribute | Value | Example |
+|-----------|-------|---------|
+| `gen_ai.operation.name` | `"invoke_agent"` or `"create_agent"` | `"invoke_agent"` |
+| `gen_ai.agent.name` | Your agent's name | `"rca-agent"` |
+| `gen_ai.system` | Provider identifier | `"aws.bedrock"`, `"openai"`, `"anthropic"` |
+| `gen_ai.conversation.id` | Unique run/session ID — links the trace to an Agent Health run | `"run-abc123"` |
+
+> **Run correlation.** Agent Health ties a trace to a run with, in order of
+> preference: **(A)** W3C trace context (if you propagate the `traceparent`
+> Agent Health injects, your spans share its `traceId` automatically); **(B)**
+> the OTEL-standard **`gen_ai.conversation.id`** set to the run id (recommended
+> for your own instrumentation — it's also accepted as `agent_health.run.id`);
+> **(C)** `service.name` + the run's time window; **(D)** the OTEL `session.id`
+> if your agent already emits one per run (e.g. Claude Code). You only need one;
+> `gen_ai.conversation.id` is the simplest to add. (`gen_ai.request.id` is **not**
+> a registered attribute — don't use it.)
+
+#### LLM Call Spans
+
+| Attribute | Value | Example |
+|-----------|-------|---------|
+| `gen_ai.operation.name` | `"chat"`, `"text_completion"`, or `"generate_content"` | `"chat"` |
+| `gen_ai.request.model` | Full model identifier | `"anthropic.claude-sonnet-4-20250514-v1:0"` |
+| `gen_ai.system` | Provider identifier | `"aws.bedrock"` |
+| `gen_ai.usage.input_tokens` | Integer token count | `1234` |
+| `gen_ai.usage.output_tokens` | Integer token count | `567` |
+| `gen_ai.request.temperature` | Float (optional) | `0.7` |
+
+#### Tool Execution Spans
+
+| Attribute | Value | Example |
+|-----------|-------|---------|
+| `gen_ai.operation.name` | `"execute_tool"` | `"execute_tool"` |
+| `gen_ai.tool.name` | Name of the tool invoked | `"search_logs"` |
+| `gen_ai.tool.call_id` | Unique tool call ID (optional) | `"call_abc123"` |
+
+### 5. Implementation Pattern
+
+**Python example — wrapping an agent loop:**
+
+```python
+from opentelemetry import trace
+from opentelemetry.trace import StatusCode
+import uuid
+
+tracer = trace.get_tracer("my-agent")
+
+def run_agent(prompt: str, run_id: str = None):
+    run_id = run_id or str(uuid.uuid4())
+
+    with tracer.start_as_current_span("invoke_agent") as agent_span:
+        agent_span.set_attribute("gen_ai.operation.name", "invoke_agent")
+        agent_span.set_attribute("gen_ai.agent.name", "my-rca-agent")
+        agent_span.set_attribute("gen_ai.system", "aws.bedrock")
+        agent_span.set_attribute("gen_ai.conversation.id", run_id)  # links the trace to an Agent Health run
+
+        # LLM call
+        with tracer.start_as_current_span("chat") as llm_span:
+            llm_span.set_attribute("gen_ai.operation.name", "chat")
+            llm_span.set_attribute("gen_ai.request.model", "anthropic.claude-sonnet-4-20250514-v1:0")
+            llm_span.set_attribute("gen_ai.system", "aws.bedrock")
+
+            response = call_llm(prompt)
+
+            llm_span.set_attribute("gen_ai.usage.input_tokens", response.input_tokens)
+            llm_span.set_attribute("gen_ai.usage.output_tokens", response.output_tokens)
+
+        # Tool execution (if the LLM requested a tool call)
+        if response.tool_calls:
+            for tool_call in response.tool_calls:
+                with tracer.start_as_current_span("execute_tool") as tool_span:
+                    tool_span.set_attribute("gen_ai.operation.name", "execute_tool")
+                    tool_span.set_attribute("gen_ai.tool.name", tool_call.name)
+                    tool_span.set_attribute("gen_ai.tool.call_id", tool_call.id)
+
+                    result = execute_tool(tool_call)
+
+                    if result.error:
+                        tool_span.set_status(StatusCode.ERROR, result.error)
+
+    return response
+```
+
+**TypeScript example:**
+
+```typescript
+import { trace, SpanStatusCode } from '@opentelemetry/api';
+import { v4 as uuid } from 'uuid';
+
+const tracer = trace.getTracer('my-agent');
+
+async function runAgent(prompt: string, runId?: string) {
+  runId = runId || uuid();
+
+  return tracer.startActiveSpan('invoke_agent', async (agentSpan) => {
+    agentSpan.setAttribute('gen_ai.operation.name', 'invoke_agent');
+    agentSpan.setAttribute('gen_ai.agent.name', 'my-rca-agent');
+    agentSpan.setAttribute('gen_ai.system', 'aws.bedrock');
+    agentSpan.setAttribute('gen_ai.conversation.id', runId); // links the trace to an Agent Health run
+
+    try {
+      // LLM call
+      const response = await tracer.startActiveSpan('chat', async (llmSpan) => {
+        llmSpan.setAttribute('gen_ai.operation.name', 'chat');
+        llmSpan.setAttribute('gen_ai.request.model', 'anthropic.claude-sonnet-4-20250514-v1:0');
+        llmSpan.setAttribute('gen_ai.system', 'aws.bedrock');
+
+        const resp = await callLLM(prompt);
+
+        llmSpan.setAttribute('gen_ai.usage.input_tokens', resp.inputTokens);
+        llmSpan.setAttribute('gen_ai.usage.output_tokens', resp.outputTokens);
+        llmSpan.end();
+        return resp;
+      });
+
+      // Tool execution
+      for (const toolCall of response.toolCalls || []) {
+        await tracer.startActiveSpan('execute_tool', async (toolSpan) => {
+          toolSpan.setAttribute('gen_ai.operation.name', 'execute_tool');
+          toolSpan.setAttribute('gen_ai.tool.name', toolCall.name);
+          toolSpan.setAttribute('gen_ai.tool.call_id', toolCall.id);
+
+          const result = await executeTool(toolCall);
+          if (result.error) {
+            toolSpan.setStatus({ code: SpanStatusCode.ERROR, message: result.error });
+          }
+          toolSpan.end();
+        });
+      }
+
+      agentSpan.end();
+      return response;
+    } catch (err) {
+      agentSpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
+      agentSpan.end();
+      throw err;
+    }
+  });
+}
+```
+
+### 6. Configure Telemetry Export and Agent Health
+
+#### A. Set environment variables to emit traces
+
+Set these in your shell or `.env` file to route telemetry from your agent to the OTLP endpoint:
+
+```bash
+# Required: enable OTLP export
+export OTEL_TRACES_EXPORTER=otlp
+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+export OTEL_EXPORTER_OTLP_ENDPOINT=<YOUR_OTLP_ENDPOINT>
+
+# Recommended: identify your service
+export OTEL_SERVICE_NAME=my-agent
+
+# Optional: also export logs and metrics
+export OTEL_METRICS_EXPORTER=otlp
+export OTEL_LOGS_EXPORTER=otlp
+```
+
+Replace `<YOUR_OTLP_ENDPOINT>` with one of:
+- **Agent Health API Gateway**: `https://<api-gateway-id>.execute-api.<region>.amazonaws.com/prod` (from CloudFormation stack)
+- **Local ADOT collector**: `http://localhost:4318`
+- **Direct OSIS pipeline**: `https://<osis-endpoint>.osis.<region>.amazonaws.com`
+
+#### B. Configure Agent Health to read traces from OpenSearch
+
+After traces are flowing into your OpenSearch domain, configure Agent Health to access them.
+
+**Option 1: CLI setup (recommended)**
+
+Run the setup command to configure everything interactively:
+
+```bash
+npx @opensearch-project/agent-health setup-telemetry
+```
+
+This deploys the telemetry stack and writes the observability config to `agent-health.config.json` automatically.
+
+**Option 2: JSON config file (manual)**
+
+In your `agent-health.config.json`:
+
+```json
+{
+  "observability": {
+    "endpoint": "https://search-my-domain.us-west-2.es.amazonaws.com",
+    "authType": "sigv4",
+    "awsRegion": "us-west-2",
+    "awsService": "es",
+    "tracesIndex": "otel-v1-apm-span-*"
+  }
+}
+```
+
+For basic auth:
+
+```json
+{
+  "observability": {
+    "endpoint": "https://search-my-domain.us-west-2.es.amazonaws.com",
+    "authType": "basic",
+    "username": "admin",
+    "password": "your-password",
+    "tracesIndex": "otel-v1-apm-span-*"
+  }
+}
+```
+
+#### C. Verify the connection
+
+```bash
+npx @opensearch-project/agent-health doctor
+```
+
+Confirm that the "Observability" check shows a connected OpenSearch domain with the correct traces index.
+
+### 7. Validation Checklist
+
+After instrumenting, verify your spans include:
+
+- [ ] Root span has `gen_ai.operation.name` = `"invoke_agent"` and `gen_ai.agent.name`
+- [ ] LLM spans have `gen_ai.operation.name` = `"chat"` and `gen_ai.request.model`
+- [ ] LLM spans report `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`
+- [ ] Tool spans have `gen_ai.operation.name` = `"execute_tool"` and `gen_ai.tool.name`
+- [ ] All spans share the same trace ID (automatic if using `start_as_current_span`)
+- [ ] `gen_ai.conversation.id` on the root span matches the run ID used in Agent Health (or you propagate W3C `traceparent`)
+- [ ] Failed operations set span status to ERROR
+
+### 8. What Agent Health Does With Your Traces
+
+Agent Health uses these attributes to:
+- **Categorize spans**: `gen_ai.operation.name` determines if a span is AGENT, LLM, TOOL, or OTHER
+- **Calculate cost**: `gen_ai.usage.input_tokens` + `gen_ai.usage.output_tokens` + `gen_ai.request.model` → USD cost
+- **Track tool usage**: `gen_ai.tool.name` populates tool call counts and tool lists
+- **Correlate runs**: `gen_ai.conversation.id` (or W3C trace context / `agent_health.run.id`, with `session.id` as a precise fallback) links traces to benchmark runs
+- **Measure duration**: Span start/end times compute total and per-step latency
+- **Detect errors**: Span status ERROR is surfaced in the trace viewer
+
+### 9. Common Mistakes
+
+- **Missing `gen_ai.operation.name`**: Spans without this attribute fall into "OTHER" category and won't be counted as LLM/Tool calls
+- **Token counts as strings**: Must be integers, not strings like `"1234"`
+- **Not ending spans**: Always call `span.end()` (or use context manager in Python) — unended spans are never exported
+- **Wrong model ID format**: Use the full model identifier (e.g., `anthropic.claude-sonnet-4-20250514-v1:0`), not just `claude-sonnet`
+````
+
+---
+
+## Deploying the Telemetry Backend
+
+If you haven't set up the observability stack yet, use the Agent Health CLI:
+
+```bash
+npx @opensearch-project/agent-health setup-telemetry --deploy
+```
+
+This deploys a CloudFormation stack with:
+- API Gateway (OTLP endpoint with Lambda SigV4 proxy)
+- OSIS pipelines (traces + logs)
+- OpenSearch domain (stores spans at `otel-v1-apm-span-*`)
+
+After deployment, the CLI prints the `OTEL_EXPORTER_OTLP_ENDPOINT` value to use.
+
+## Using as a Claude Code Skill
+
+To make this available as a slash command in Claude Code, add to your project's `.claude/skills/instrument-otel.md`:
+
+```markdown
+---
+description: Instrument the current application with OpenTelemetry for Agent Health
+---
+
+[paste the prompt section above]
+```
+
+Then invoke with `/instrument-otel` in any Claude Code session.
+
+## Using as an MCP Resource
+
+If you run an MCP server, expose this document as a resource at `agent-health://instrumentation-guide` so any connected coding agent can retrieve it on demand.
+
+## Reference
+
+- [OTel GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
+- [OTel GenAI Agent Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
+- [Agent Health Telemetry Setup](./CLAUDE_CODE_TELEMETRY.md)

package/docs/ML-COMMONS-SETUP.md

@@ -0,0 +1,289 @@
+# ML-Commons Agent Setup
+
+This guide covers setting up the ML-Commons AG-UI agent for use with Agent Health.
+
+## Architecture
+
+```
+Agent Health --> ML-Commons Agent (9200) --> MCP Server (3030) --> OpenSearch Data Cluster
+                    |
+              Bedrock LLM (Claude)
+```
+
+**Components:**
+- **ML-Commons**: OpenSearch plugin providing AG-UI streaming agent
+- **MCP Server**: Provides OpenSearch tools to the agent
+- **Bedrock**: LLM backend for agent reasoning
+
+## Prerequisites
+
+| Requirement | Version | Purpose |
+|-------------|---------|---------|
+| Java | 11+ | OpenSearch and ML-Commons |
+| Python | 3.9+ | MCP server (`pip install uvx`) |
+| AWS CLI | 2.x | Profile-based authentication |
+| AWS Profile | - | Configured profile with Bedrock access |
+
+---
+
+## Option 1: Automated Setup
+
+### Prerequisites
+1. ML-Commons must be running on port 9200
+2. `AWS_PROFILE` must be set (in `.env` or environment)
+
+### Quick Start
+```bash
+# Set your AWS profile
+export AWS_PROFILE=Bedrock
+
+# Run setup
+./scripts/setup.sh
+```
+
+### Script Options
+
+| Command | Description |
+|---------|-------------|
+| `./scripts/setup.sh` | Register agent, update .env, start servers |
+| `./scripts/setup.sh --stop` | Stop MCP server and Agent Health (not ML-Commons) |
+| `./scripts/setup.sh --status` | Check which services are running |
+
+### What the Script Does
+
+1. Validates AWS profile and fetches credentials
+2. Starts MCP server if not running (port 3030)
+3. Registers Bedrock model with credentials from profile
+4. Creates MCP connector pointing to localhost:3030
+5. Registers AG-UI agent with the model and MCP connector
+6. Tests agent execution to verify setup works
+7. Updates `.env` with new agent endpoint
+8. Starts Agent Health server (port 4001)
+
+---
+
+## Option 2: Manual Setup
+
+### Step 1: Clone and Build OpenSearch Core
+
+```bash
+mkdir ~/agenteval-workspace && cd ~/agenteval-workspace
+
+git clone https://github.com/opensearch-project/OpenSearch
+cd OpenSearch
+
+# Build streaming plugins
+./gradlew :plugins:transport-reactor-netty4:assemble
+./gradlew :plugins:arrow-flight-rpc:assemble
+
+export OPENSEARCH_CORE_PATH=$(pwd)
+```
+
+### Step 2: Clone and Start ML-Commons
+
+```bash
+cd ~/agenteval-workspace
+
+git clone https://github.com/jiapingzeng/ml-commons
+cd ml-commons
+git switch 3.4-jpz
+
+# Start with streaming enabled (keep terminal running)
+./gradlew run -Dstreaming=true
+```
+
+### Step 3: Start MCP Server
+
+```bash
+# Set credentials for the OpenSearch cluster the agent will query
+OPENSEARCH_URL=https://your-opensearch-cluster.aos.us-west-2.on.aws \
+OPENSEARCH_USERNAME=admin \
+OPENSEARCH_PASSWORD=your_password \
+uvx opensearch-mcp-server-py@0.5.2 --transport stream --port 3030
+```
+
+### Step 4: Configure ML-Commons Settings
+
+```bash
+curl -X PUT 'http://localhost:9200/_cluster/settings' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "persistent": {
+      "plugins.ml_commons.trusted_connector_endpoints_regex": [
+        "http://localhost:3030",
+        "^https://bedrock-runtime\\..*[a-z0-9-]\\.amazonaws\\.com/.*$"
+      ],
+      "plugins.ml_commons.stream_enabled": true,
+      "plugins.ml_commons.mcp_connector_enabled": true,
+      "plugins.ml_commons.ag_ui_enabled": true,
+      "plugins.ml_commons.mcp_header_passthrough_enabled": true
+    }
+  }'
+```
+
+### Step 5: Register Bedrock Model
+
+```bash
+# Ensure your AWS credentials are configured (choose one method):
+# Option 1: Use AWS CLI profile
+export AWS_PROFILE=your-profile-name
+
+# Option 2: Set credentials directly
+export AWS_ACCESS_KEY_ID=your_access_key
+export AWS_SECRET_ACCESS_KEY=your_secret_key
+export AWS_SESSION_TOKEN=your_session_token  # if using temporary credentials
+
+# Register model (save the model_id from response)
+curl 'http://localhost:9200/_plugins/_ml/models/_register?deploy=true' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "name": "Claude Sonnet",
+    "function_name": "remote",
+    "connector": {
+      "name": "Bedrock Converse Connector",
+      "protocol": "aws_sigv4",
+      "parameters": {
+        "region": "us-east-1",
+        "model": "us.anthropic.claude-sonnet-4-20250514-v1:0",
+        "service_name": "bedrock"
+      },
+      "credential": {
+        "access_key": "YOUR_AWS_ACCESS_KEY_ID",
+        "secret_key": "YOUR_AWS_SECRET_ACCESS_KEY",
+        "session_token": "YOUR_AWS_SESSION_TOKEN"
+      },
+      "actions": [{
+        "action_type": "predict",
+        "method": "POST",
+        "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/converse",
+        "request_body": "{\"messages\": [${parameters._chat_history:-}{\"role\":\"user\",\"content\":[{\"text\":\"${parameters.prompt}\"}]}${parameters._interactions:-}]${parameters.tool_configs:-}}"
+      }]
+    }
+  }'
+```
+
+### Step 6: Create MCP Connector
+
+```bash
+curl 'http://localhost:9200/_plugins/_ml/connectors/_create' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "name": "OpenSearch MCP Server",
+    "protocol": "mcp_streamable_http",
+    "url": "http://localhost:3030",
+    "parameters": { "endpoint": "/mcp/" }
+  }'
+```
+
+### Step 7: Register AG-UI Agent
+
+```bash
+# Replace MODEL_ID and MCP_CONNECTOR_ID from previous steps
+curl 'http://localhost:9200/_plugins/_ml/agents/_register' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "name": "AG-UI chat agent",
+    "type": "AG_UI",
+    "llm": {
+      "model_id": "MODEL_ID",
+      "parameters": {
+        "max_iteration": 50,
+        "system_prompt": "You are a helpful assistant.",
+        "prompt": "Context:${parameters.context}\nQuestion:${parameters.question}"
+      }
+    },
+    "parameters": {
+      "_llm_interface": "bedrock/converse/claude",
+      "mcp_connectors": [{ "mcp_connector_id": "MCP_CONNECTOR_ID" }]
+    },
+    "tools": []
+  }'
+```
+
+### Step 8: Configure Agent Health
+
+Update `.env` with your agent ID:
+```bash
+MLCOMMONS_ENDPOINT=http://localhost:9200/_plugins/_ml/agents/{agent_id}/_execute/stream
+```
+
+---
+
+## Environment Variables
+
+### ML-Commons Agent Headers
+
+These headers allow the ML-Commons agent to access the OpenSearch data cluster:
+
+| Variable | Description |
+|----------|-------------|
+| `MLCOMMONS_ENDPOINT` | Agent streaming endpoint (includes agent_id) |
+| `MLCOMMONS_HEADER_OPENSEARCH_URL` | Target OpenSearch cluster URL |
+| `MLCOMMONS_HEADER_AWS_REGION` | AWS region for signing |
+| `MLCOMMONS_HEADER_AWS_SERVICE_NAME` | Service name (`es` or `aoss`) |
+| `MLCOMMONS_HEADER_AWS_ACCESS_KEY_ID` | AWS access key |
+| `MLCOMMONS_HEADER_AWS_SECRET_ACCESS_KEY` | AWS secret key |
+| `MLCOMMONS_HEADER_AWS_SESSION_TOKEN` | AWS session token |
+
+### OpenSearch Storage & Observability (AWS SigV4)
+
+Agent Health can connect to OpenSearch clusters using AWS SigV4 instead of username/password. This is useful for AWS-managed OpenSearch domains and OpenSearch Serverless.
+
+**Storage cluster (for persisting test cases, benchmarks, and results):**
+
+| Variable | Description |
+|----------|-------------|
+| `OPENSEARCH_STORAGE_ENDPOINT` | Storage cluster URL |
+| `OPENSEARCH_STORAGE_AUTH_TYPE` | Set to `sigv4` for AWS SigV4 auth |
+| `OPENSEARCH_STORAGE_AWS_REGION` | AWS region (required for SigV4) |
+| `OPENSEARCH_STORAGE_AWS_PROFILE` | AWS profile name (uses default credential chain if omitted) |
+| `OPENSEARCH_STORAGE_AWS_SERVICE` | `es` for managed OpenSearch, `aoss` for Serverless |
+
+**Observability cluster (for traces and logs):**
+
+| Variable | Description |
+|----------|-------------|
+| `OPENSEARCH_LOGS_ENDPOINT` | Observability cluster URL |
+| `OPENSEARCH_LOGS_AUTH_TYPE` | Set to `sigv4` for AWS SigV4 auth |
+| `OPENSEARCH_LOGS_AWS_REGION` | AWS region (required for SigV4) |
+| `OPENSEARCH_LOGS_AWS_PROFILE` | AWS profile name (uses default credential chain if omitted) |
+| `OPENSEARCH_LOGS_AWS_SERVICE` | `es` for managed OpenSearch, `aoss` for Serverless |
+
+SigV4 uses the AWS credential provider chain (`AWS_PROFILE`, `~/.aws/credentials`, IAM role, etc.), so no explicit access keys are needed. You can also configure these settings via the Settings UI (select "AWS SigV4" from the Authentication Type dropdown).
+
+---
+
+## Services Summary
+
+| Service | Port | Command | Purpose |
+|---------|------|---------|---------|
+| ML-Commons | 9200 | `./gradlew run -Dstreaming=true` | AG-UI agent endpoint |
+| MCP Server | 3030 | `uvx opensearch-mcp-server-py@0.5.2` | OpenSearch tools for agent |
+| Agent Health | 4001 | `npm run server` | Evaluation UI and API |
+
+---
+
+## Troubleshooting
+
+| Issue | Solution |
+|-------|----------|
+| Agent not responding | Check ML-Commons: `curl http://localhost:9200/_cat/health` |
+| MCP server issues | Verify: `curl http://localhost:3030/health`, check MCP env vars |
+| AWS profile invalid | Ensure profile exists: `aws sts get-caller-identity --profile $AWS_PROFILE` |
+| Port conflicts | Use `./scripts/setup.sh --stop` to stop MCP and Agent Health services |
+
+### Check Service Status
+```bash
+./scripts/setup.sh --status
+```
+
+### Stop All Services
+```bash
+./scripts/setup.sh --stop
+```
+
+---
+
+## Related Documentation
+
+- [AG-UI Protocol](https://docs.ag-ui.com/sdk/js/core/types#runagentinput)