@opensearch-project/agent-health 0.1.1 → 0.2.0

package/README.md

@@ -2,7 +2,7 @@
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE.txt)
 [![npm version](https://img.shields.io/npm/v/@opensearch-project/agent-health.svg)](https://www.npmjs.com/package/@opensearch-project/agent-health)
-[![Documentation](https://img.shields.io/badge/View_Documentation-blue?logo=readthedocs&logoColor=white)](https://goyamegh.github.io/opensearch-agentops-website/)
+[![Documentation](https://img.shields.io/badge/View_Documentation-blue?logo=readthedocs&logoColor=white)](https://observability.opensearch.org/docs/agent-health/)
 
 ## What is Agent Health?
 
@@ -71,18 +71,35 @@ For detailed architecture documentation, see [docs/ARCHITECTURE.md](./docs/ARCHI
 ## CLI Commands
 
 ```bash
-# Check configuration
+# Start server (default action)
+npx @opensearch-project/agent-health
+
+# Initialize a new project (creates agent-health.config.ts and .env.example)
+npx @opensearch-project/agent-health init
+
+# Check configuration and connectivity
 npx @opensearch-project/agent-health doctor
 
-# List available agents and connectors
+# List resources (agents, connectors, models, test-cases, benchmarks)
 npx @opensearch-project/agent-health list agents
 npx @opensearch-project/agent-health list connectors
 
-# Run a test case against an agent
+# Run a single test case against an agent
 npx @opensearch-project/agent-health run -t demo-otel-001 -a demo
 
-# Initialize a new project
-npx @opensearch-project/agent-health init
+# Run a benchmark (batch of test cases)
+npx @opensearch-project/agent-health benchmark -f ./test-cases.json -a my-agent
+npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export results.json
+
+# Export benchmark test cases as JSON
+npx @opensearch-project/agent-health export -b "My Benchmark" -o test-cases.json
+
+# Generate reports (HTML, PDF, JSON)
+npx @opensearch-project/agent-health report -b "My Benchmark"
+npx @opensearch-project/agent-health report -b "My Benchmark" -f pdf -o report.pdf
+
+# One-time migration for existing benchmark runs
+npx @opensearch-project/agent-health migrate --dry-run
 ```
 
 For full CLI documentation, see [docs/CLI.md](./docs/CLI.md).
@@ -94,21 +111,19 @@ For full CLI documentation, see [docs/CLI.md](./docs/CLI.md).
 
 Agent Health works out-of-the-box with demo data. Configure when you're ready to connect your own agent.
 
-### Minimal Setup (Optional)
+### Config File: `agent-health.config.ts`
 
-**For LLM Judge evaluation** (uses AWS Bedrock):
-```bash
-# Create .env file
-cp .env.example .env
+This is the primary way to configure custom agents, models, and hooks. Create it in your working directory (the directory you run `npx` or `agent-health` from):
 
-# Add AWS credentials
-AWS_REGION=us-east-1
-AWS_ACCESS_KEY_ID=your_access_key
-AWS_SECRET_ACCESS_KEY=your_secret_key
+```bash
+# Generate a config file with examples
+npx @opensearch-project/agent-health init
 ```
 
-**For your own agent** - create `agent-health.config.ts`:
+Or create it manually:
+
 ```typescript
+// agent-health.config.ts
 export default {
   agents: [
     {
@@ -117,11 +132,29 @@ export default {
       endpoint: "http://localhost:8000/agent",
       connectorType: "rest",  // or "agui-streaming", "subprocess"
       models: ["claude-sonnet-4"],
+      useTraces: true,        // Enable OpenTelemetry trace collection
     }
   ],
 };
 ```
 
+The config file is auto-detected from the current working directory. Supported file names (in priority order): `agent-health.config.ts`, `agent-health.config.js`, `agent-health.config.mjs`. See [`agent-health.config.example.ts`](./agent-health.config.example.ts) for all available options including authentication hooks.
+
+> **Tip:** Run `npx @opensearch-project/agent-health doctor` to verify your configuration is loaded correctly.
+
+### Environment Variables (Optional)
+
+**For LLM Judge evaluation** (uses AWS Bedrock):
+```bash
+# Create .env file
+cp .env.example .env
+
+# Add AWS credentials
+AWS_REGION=us-east-1
+AWS_ACCESS_KEY_ID=your_access_key
+AWS_SECRET_ACCESS_KEY=your_secret_key
+```
+
 **Full configuration guide:** [CONFIGURATION.md](./docs/CONFIGURATION.md)
 
 ---
@@ -284,10 +317,25 @@ Agent Health supports multiple agent types:
 
 | Agent | Endpoint Variable | Setup |
 |-------|-------------------|-------|
+| **Observio** (sample) | `localhost:3001` | Included — see [observio-sample-agent/](./observio-sample-agent/) |
 | Langgraph | `LANGGRAPH_ENDPOINT` | Simple localhost agent |
 | HolmesGPT | `HOLMESGPT_ENDPOINT` | AG-UI compatible RCA agent |
 | ML-Commons | `MLCOMMONS_ENDPOINT` | See [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) |
 
+### Observio Sample Agent
+
+Agent Health includes **Observio**, a reference ReAct agent you can use as a practice target for evaluating and improving agent performance. It's a great starting point if you don't have your own agent yet.
+
+```bash
+# Start Observio
+cd observio-sample-agent && npm install && npm run start:ag-ui
+
+# Evaluate it with Agent Health
+npx @opensearch-project/agent-health run -t demo-otel-001 -a observio
+```
+
+See the [Observio README](./observio-sample-agent/README.md) for setup details and improvement areas.
+
 
 ---
 
@@ -342,6 +390,7 @@ All commits require DCO signoff and all PRs must pass CI checks (tests, coverage
 - [Getting Started](./GETTING_STARTED.md) - Step-by-step walkthrough from install to first evaluation
 - [Configuration](./docs/CONFIGURATION.md) - Connect your agent and configure the environment
 - [CLI Reference](./docs/CLI.md) - Command-line interface documentation
+- [Observio Sample Agent](./observio-sample-agent/) - Reference agent for practicing agent health improvements
 
 ### Developer Guides
 - [Development Guide](./CLAUDE.md) - Architecture, coding conventions, and contributing