@opensearch-project/agent-health 0.1.0 → 0.2.0

package/README.md

@@ -1,27 +1,37 @@
 # Agent Health
 
-[![CI](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml/badge.svg)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE.txt)
 [![npm version](https://img.shields.io/npm/v/@opensearch-project/agent-health.svg)](https://www.npmjs.com/package/@opensearch-project/agent-health)
+[![Documentation](https://img.shields.io/badge/View_Documentation-blue?logo=readthedocs&logoColor=white)](https://observability.opensearch.org/docs/agent-health/)
 
-[![Unit Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/unit-tests.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
-[![Unit Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/unit-coverage.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
-[![Integration Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/integration-tests.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
-[![E2E Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/e2e-tests.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
+## What is Agent Health?
 
-An evaluation and observability framework for AI agents. Features real-time trace visualization, "Golden Path" trajectory comparison, and LLM-based evaluation scoring.
+Agent Health is an evaluation and observability framework for AI agents. It helps you measure agent performance through "Golden Path" trajectory comparison—where an LLM judge evaluates agent actions against expected outcomes.
 
-Try It by running:
+**Who uses Agent Health:**
+- AI teams building autonomous agents (RCA, customer support, data analysis)
+- QA engineers testing agent behavior across scenarios
+- Platform teams monitoring agent performance in production
+
+**Key capabilities:**
+- Real-time agent execution streaming and visualization
+- LLM-based evaluation with pass/fail scoring
+- Batch experiments comparing agents and models
+- OpenTelemetry trace integration for performance analysis
+- Pluggable connectors for different agent types (REST, SSE, CLI)
+
+## Quick Start
 
 ```bash
-npx @goyamegh/agent-health@latest
+# Start Agent Health with demo data (no configuration needed)
+npx @opensearch-project/agent-health
 ```
 
-Opens http://localhost:4001 for the web UI.
+Opens http://localhost:4001 with pre-loaded sample data for exploration.
 
-### Architecture
-
-![Agent Health Architecture](docs/diagrams/architecture.png)
+**Next steps:**
+- [Getting Started Guide](./GETTING_STARTED.md) - Step-by-step walkthrough
+- [Connect Your Agent](./docs/CONFIGURATION.md) - Configure your own agent
 
 ## Features
 
@@ -34,9 +44,6 @@ Opens http://localhost:4001 for the web UI.
 - **Reports**: Evaluation reports with LLM judge reasoning
 - **Connectors**: Pluggable protocol adapters for different agent types
 
-For a detailed walkthrough, see [Getting Started](./GETTING_STARTED.md).
-
-
 ### Supported Connectors
 
 | Connector | Protocol | Description |
@@ -51,97 +58,104 @@ For creating custom connectors, see [docs/CONNECTORS.md](./docs/CONNECTORS.md).
 
 ---
 
+## Architecture
+
+![Agent Health Architecture](docs/diagrams/architecture.png)
+
+Agent Health uses a client-server architecture where all clients (UI, CLI) access OpenSearch through a unified HTTP API. The server handles agent communication via pluggable connectors and proxies LLM judge calls to AWS Bedrock.
 
+For detailed architecture documentation, see [docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md).
 
 ---
 
-## Quick Start
+## CLI Commands
 
 ```bash
-# Start the web UI
+# Start server (default action)
 npx @opensearch-project/agent-health
 
-# Open http://localhost:4001
-```
-
-### CLI Commands
+# Initialize a new project (creates agent-health.config.ts and .env.example)
+npx @opensearch-project/agent-health init
 
-```bash
-# Check configuration
+# Check configuration and connectivity
 npx @opensearch-project/agent-health doctor
 
-# List available agents and connectors
+# List resources (agents, connectors, models, test-cases, benchmarks)
 npx @opensearch-project/agent-health list agents
 npx @opensearch-project/agent-health list connectors
 
-# Run a test case against an agent
+# Run a single test case against an agent
 npx @opensearch-project/agent-health run -t demo-otel-001 -a demo
 
-# Initialize a new project
-npx @opensearch-project/agent-health init
-```
+# Run a benchmark (batch of test cases)
+npx @opensearch-project/agent-health benchmark -f ./test-cases.json -a my-agent
+npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export results.json
 
-For full CLI documentation, see [docs/CLI.md](./docs/CLI.md).
+# Export benchmark test cases as JSON
+npx @opensearch-project/agent-health export -b "My Benchmark" -o test-cases.json
 
+# Generate reports (HTML, PDF, JSON)
+npx @opensearch-project/agent-health report -b "My Benchmark"
+npx @opensearch-project/agent-health report -b "My Benchmark" -f pdf -o report.pdf
 
+# One-time migration for existing benchmark runs
+npx @opensearch-project/agent-health migrate --dry-run
+```
 
+For full CLI documentation, see [docs/CLI.md](./docs/CLI.md).
 
-## Authentication (Required)
 
-AWS credentials are required for the Bedrock LLM Judge to score evaluations.
 
-Create a `.env` file:
-```bash
-cp .env.example .env
-```
 
-Add your AWS credentials:
-```bash
-AWS_REGION=us-east-1
-AWS_ACCESS_KEY_ID=your_access_key
-AWS_SECRET_ACCESS_KEY=your_secret_key
-AWS_SESSION_TOKEN=your_session_token  # if using temporary credentials
-```
-
----
+## Configuration
 
-## Configuration (Optional)
+Agent Health works out-of-the-box with demo data. Configure when you're ready to connect your own agent.
 
-All optional settings have sensible defaults. Configure only what you need.
+### Config File: `agent-health.config.ts`
 
-### Agent Endpoints
-
-Agent endpoints default to localhost. Override if your agent runs elsewhere:
+This is the primary way to configure custom agents, models, and hooks. Create it in your working directory (the directory you run `npx` or `agent-health` from):
 
 ```bash
-LANGGRAPH_ENDPOINT=http://localhost:3000
-HOLMESGPT_ENDPOINT=http://localhost:5050/api/agui/chat
-MLCOMMONS_ENDPOINT=http://localhost:9200/_plugins/_ml/agents/{agent_id}/_execute/stream
+# Generate a config file with examples
+npx @opensearch-project/agent-health init
 ```
 
-### Storage (Persistence)
-
-For persisting test cases, experiments, and runs. Features gracefully degrade if not configured.
-
-```bash
-OPENSEARCH_STORAGE_ENDPOINT=https://your-cluster.opensearch.amazonaws.com
-OPENSEARCH_STORAGE_USERNAME=admin
-OPENSEARCH_STORAGE_PASSWORD=your_password
-OPENSEARCH_STORAGE_TLS_SKIP_VERIFY=false  # Set to true for self-signed certificates
+Or create it manually:
+
+```typescript
+// agent-health.config.ts
+export default {
+  agents: [
+    {
+      key: "my-agent",
+      name: "My Agent",
+      endpoint: "http://localhost:8000/agent",
+      connectorType: "rest",  // or "agui-streaming", "subprocess"
+      models: ["claude-sonnet-4"],
+      useTraces: true,        // Enable OpenTelemetry trace collection
+    }
+  ],
+};
 ```
 
-### Traces (Observability)
+The config file is auto-detected from the current working directory. Supported file names (in priority order): `agent-health.config.ts`, `agent-health.config.js`, `agent-health.config.mjs`. See [`agent-health.config.example.ts`](./agent-health.config.example.ts) for all available options including authentication hooks.
+
+> **Tip:** Run `npx @opensearch-project/agent-health doctor` to verify your configuration is loaded correctly.
 
-For agent execution traces. Features gracefully degrade if not configured.
+### Environment Variables (Optional)
 
+**For LLM Judge evaluation** (uses AWS Bedrock):
 ```bash
-OPENSEARCH_LOGS_ENDPOINT=https://your-logs-cluster.opensearch.amazonaws.com
-OPENSEARCH_LOGS_USERNAME=admin
-OPENSEARCH_LOGS_PASSWORD=your_password
-OPENSEARCH_LOGS_TLS_SKIP_VERIFY=false  # Set to true for self-signed certificates
+# Create .env file
+cp .env.example .env
+
+# Add AWS credentials
+AWS_REGION=us-east-1
+AWS_ACCESS_KEY_ID=your_access_key
+AWS_SECRET_ACCESS_KEY=your_secret_key
 ```
 
-See `.env.example` for all available options.
+**Full configuration guide:** [CONFIGURATION.md](./docs/CONFIGURATION.md)
 
 ---
 
@@ -303,10 +317,25 @@ Agent Health supports multiple agent types:
 
 | Agent | Endpoint Variable | Setup |
 |-------|-------------------|-------|
+| **Observio** (sample) | `localhost:3001` | Included — see [observio-sample-agent/](./observio-sample-agent/) |
 | Langgraph | `LANGGRAPH_ENDPOINT` | Simple localhost agent |
 | HolmesGPT | `HOLMESGPT_ENDPOINT` | AG-UI compatible RCA agent |
 | ML-Commons | `MLCOMMONS_ENDPOINT` | See [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) |
 
+### Observio Sample Agent
+
+Agent Health includes **Observio**, a reference ReAct agent you can use as a practice target for evaluating and improving agent performance. It's a great starting point if you don't have your own agent yet.
+
+```bash
+# Start Observio
+cd observio-sample-agent && npm install && npm run start:ag-ui
+
+# Evaluate it with Agent Health
+npx @opensearch-project/agent-health run -t demo-otel-001 -a observio
+```
+
+See the [Observio README](./observio-sample-agent/README.md) for setup details and improvement areas.
+
 
 ---
 
@@ -357,7 +386,14 @@ All commits require DCO signoff and all PRs must pass CI checks (tests, coverage
 
 ## Documentation
 
-- [Getting Started](./GETTING_STARTED.md) - Installation, demo mode, and usage walkthrough
-- [ML-Commons Agent Setup](./docs/ML-COMMONS-SETUP.md) - Configure ML-Commons agent
-- [Development Guide](./CLAUDE.md) - Architecture and coding conventions
-- [AG-UI Protocol](https://docs.ag-ui.com/sdk/js/core/types#runagentinput)
+### User Guides
+- [Getting Started](./GETTING_STARTED.md) - Step-by-step walkthrough from install to first evaluation
+- [Configuration](./docs/CONFIGURATION.md) - Connect your agent and configure the environment
+- [CLI Reference](./docs/CLI.md) - Command-line interface documentation
+- [Observio Sample Agent](./observio-sample-agent/) - Reference agent for practicing agent health improvements
+
+### Developer Guides
+- [Development Guide](./CLAUDE.md) - Architecture, coding conventions, and contributing
+- [Connectors Guide](./docs/CONNECTORS.md) - Create custom connectors for your agent type
+- [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) - OpenSearch ML-Commons integration
+- [Architecture](./docs/ARCHITECTURE.md) - System design and patterns