@opensearch-project/agent-health 0.2.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. package/README.md +210 -298
  2. package/cli/dist/index.js +1438 -110
  3. package/deployment/cloudformation/agent-health-observability.yaml +571 -0
  4. package/dist/assets/index-BOIP5L7h.js +246 -0
  5. package/dist/assets/index-CU9YKpAL.css +1 -0
  6. package/dist/index.html +2 -2
  7. package/lib/dist/config/index.js +50 -2
  8. package/lib/dist/index.js +66 -6
  9. package/package.json +25 -8
  10. package/server/dist/app.js +7495 -2524
  11. package/server/dist/index.js +5531 -328
  12. package/dist/assets/index-4BAkkFzo.js +0 -267
  13. package/dist/assets/index-C3K5cBQr.css +0 -1
package/README.md CHANGED
@@ -1,399 +1,311 @@
1
- # Agent Health
1
+ <h1 align="center" style="border-bottom: none">
2
+ <div>
3
+ <a href="https://opensearch.org">
4
+ <img alt="OpenSearch Agent Health" src="assets/opensearch-logo.svg" width="200" />
5
+ </a>
6
+ <br>
7
+ Agent Health
8
+ </div>
9
+ </h1>
10
+
11
+ <h2 align="center" style="border-bottom: none">Open-source AI Agent Evaluation & Observability</h2>
12
+
13
+ <p align="center">
14
+ Agent Health helps you evaluate, monitor, and optimize AI agents. From autonomous RCA agents to coding assistants, it provides real-time execution streaming, LLM-based evaluation with trajectory comparison, batch experiments, and deep observability through OpenTelemetry traces — all backed by OpenSearch.
15
+ </p>
16
+
17
+ <div align="center">
2
18
 
3
19
  [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE.txt)
4
20
  [![npm version](https://img.shields.io/npm/v/@opensearch-project/agent-health.svg)](https://www.npmjs.com/package/@opensearch-project/agent-health)
5
- [![Documentation](https://img.shields.io/badge/View_Documentation-blue?logo=readthedocs&logoColor=white)](https://observability.opensearch.org/docs/agent-health/)
21
+ [![Build](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml/badge.svg)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
22
+
23
+ </div>
24
+
25
+ <p align="center">
26
+ <a href="https://opensearch.org"><b>Website</b></a> &bull;
27
+ <a href="https://opensearch.org/slack.html"><b>Slack</b></a> &bull;
28
+ <a href="https://x.com/OpenSearchProj"><b>Twitter/X</b></a> &bull;
29
+ <a href="https://www.youtube.com/watch?v=MU3tTv4lKtc"><b>Demo Video</b></a> &bull;
30
+ <a href="https://observability.opensearch.org/docs/agent-health/"><b>Documentation</b></a> &bull;
31
+ <a href="CHANGELOG.md"><b>Changelog</b></a>
32
+ </p>
33
+
34
+ <div align="center" style="margin-top: 1em; margin-bottom: 1em;">
35
+ <a href="#what-is-agent-health">What is Agent Health?</a> &bull;
36
+ <a href="#installation">Installation</a> &bull;
37
+ <a href="#features">Features</a> &bull;
38
+ <a href="#quick-configuration">Configuration</a> &bull;
39
+ <a href="#contributing">Contributing</a>
40
+ </div>
41
+
42
+ <br>
43
+
44
+ <p align="center">
45
+ <a href="screenshots/Comparison.png">
46
+ <img alt="Agent Health — Run Comparison Dashboard" src="screenshots/Comparison.png" />
47
+ </a>
48
+ </p>
49
+
50
+ <p align="center">
51
+ <i>Side-by-side comparison of agent evaluation runs with pass rate, accuracy, cost, and performance metrics over time.</i>
52
+ </p>
53
+
54
+ ---
6
55
 
56
+ <a id="what-is-agent-health"></a>
7
57
  ## What is Agent Health?
8
58
 
9
- Agent Health is an evaluation and observability framework for AI agents. It helps you measure agent performance through "Golden Path" trajectory comparison—where an LLM judge evaluates agent actions against expected outcomes.
59
+ Agent Health is an evaluation and observability framework for AI agents, built on [OpenSearch](https://opensearch.org). It helps you measure agent performance through **"Golden Path" trajectory comparison** — where an LLM judge evaluates agent actions against expected outcomes — and provides deep observability into agent execution via OpenTelemetry traces.
10
60
 
11
61
  **Who uses Agent Health:**
12
62
  - AI teams building autonomous agents (RCA, customer support, data analysis)
13
63
  - QA engineers testing agent behavior across scenarios
14
64
  - Platform teams monitoring agent performance in production
65
+ - Developers using AI coding agents who want visibility into usage, costs, and productivity
15
66
 
16
- **Key capabilities:**
17
- - Real-time agent execution streaming and visualization
18
- - LLM-based evaluation with pass/fail scoring
19
- - Batch experiments comparing agents and models
20
- - OpenTelemetry trace integration for performance analysis
21
- - Pluggable connectors for different agent types (REST, SSE, CLI)
22
-
23
- ## Quick Start
24
-
25
- ```bash
26
- # Start Agent Health with demo data (no configuration needed)
27
- npx @opensearch-project/agent-health
28
- ```
29
-
30
- Opens http://localhost:4001 with pre-loaded sample data for exploration.
31
-
32
- **Next steps:**
33
- - [Getting Started Guide](./GETTING_STARTED.md) - Step-by-step walkthrough
34
- - [Connect Your Agent](./docs/CONFIGURATION.md) - Configure your own agent
35
-
36
- ## Features
37
-
38
- - **Evals**: Real-time agent evaluation with trajectory streaming
39
- - **Experiments**: Batch evaluation runs with configurable parameters
40
- - **Compare**: Side-by-side trace comparison with aligned and merged views
41
- - **Agent Traces**: Table-based trace view with latency histogram, filtering, and detailed flyout with input/output display
42
- - **Live Traces**: Real-time trace monitoring with auto-refresh and filtering
43
- - **Trace Views**: Timeline and Flow visualizations for debugging
44
- - **Reports**: Evaluation reports with LLM judge reasoning
45
- - **Connectors**: Pluggable protocol adapters for different agent types
46
-
47
- ### Supported Connectors
48
-
49
- | Connector | Protocol | Description |
50
- |-----------|----------|-------------|
51
- | `agui-streaming` | AG-UI SSE | ML-Commons agents (default) |
52
- | `rest` | HTTP POST | Non-streaming REST APIs |
53
- | `subprocess` | CLI | Command-line tools |
54
- | `claude-code` | Claude CLI | Claude Code agent comparison |
55
- | `mock` | In-memory | Demo and testing |
56
-
57
- For creating custom connectors, see [docs/CONNECTORS.md](./docs/CONNECTORS.md).
67
+ > **See it in action:** Watch the [demo video on YouTube](https://www.youtube.com/watch?v=MU3tTv4lKtc)
58
68
 
59
69
  ---
60
70
 
61
- ## Architecture
62
-
63
- ![Agent Health Architecture](docs/diagrams/architecture.png)
64
-
65
- Agent Health uses a client-server architecture where all clients (UI, CLI) access OpenSearch through a unified HTTP API. The server handles agent communication via pluggable connectors and proxies LLM judge calls to AWS Bedrock.
66
-
67
- For detailed architecture documentation, see [docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md).
71
+ <a id="installation"></a>
72
+ ## Installation
68
73
 
69
- ---
74
+ Get Agent Health running in minutes. Choose the option that best suits your needs:
70
75
 
71
- ## CLI Commands
76
+ ### Option 1: NPX (Fastest — No Setup)
72
77
 
73
78
  ```bash
74
- # Start server (default action)
79
+ # Start Agent Health with demo data (no configuration needed)
75
80
  npx @opensearch-project/agent-health
76
-
77
- # Initialize a new project (creates agent-health.config.ts and .env.example)
78
- npx @opensearch-project/agent-health init
79
-
80
- # Check configuration and connectivity
81
- npx @opensearch-project/agent-health doctor
82
-
83
- # List resources (agents, connectors, models, test-cases, benchmarks)
84
- npx @opensearch-project/agent-health list agents
85
- npx @opensearch-project/agent-health list connectors
86
-
87
- # Run a single test case against an agent
88
- npx @opensearch-project/agent-health run -t demo-otel-001 -a demo
89
-
90
- # Run a benchmark (batch of test cases)
91
- npx @opensearch-project/agent-health benchmark -f ./test-cases.json -a my-agent
92
- npx @opensearch-project/agent-health benchmark -n "My Benchmark" -a my-agent --export results.json
93
-
94
- # Export benchmark test cases as JSON
95
- npx @opensearch-project/agent-health export -b "My Benchmark" -o test-cases.json
96
-
97
- # Generate reports (HTML, PDF, JSON)
98
- npx @opensearch-project/agent-health report -b "My Benchmark"
99
- npx @opensearch-project/agent-health report -b "My Benchmark" -f pdf -o report.pdf
100
-
101
- # One-time migration for existing benchmark runs
102
- npx @opensearch-project/agent-health migrate --dry-run
103
81
  ```
104
82
 
105
- For full CLI documentation, see [docs/CLI.md](./docs/CLI.md).
106
-
107
-
108
-
83
+ Opens http://localhost:4001 with pre-loaded sample data for exploration. If port 4001 is already in use, the server automatically tries the next available port (4002, 4003, etc., up to 10 attempts).
109
84
 
110
- ## Configuration
85
+ ### Option 2: Docker Compose (with OpenSearch Observability Stack)
111
86
 
112
- Agent Health works out-of-the-box with demo data. Configure when you're ready to connect your own agent.
87
+ For the full observability stack with OpenSearch, OpenTelemetry Collector, and Data Prepper for trace ingestion:
113
88
 
114
- ### Config File: `agent-health.config.ts`
115
-
116
- This is the primary way to configure custom agents, models, and hooks. Create it in your working directory (the directory you run `npx` or `agent-health` from):
89
+ **Quick start (one command):**
117
90
 
118
91
  ```bash
119
- # Generate a config file with examples
120
- npx @opensearch-project/agent-health init
92
+ curl -fsSL https://raw.githubusercontent.com/opensearch-project/agent-health/main/scripts/install.sh | bash
121
93
  ```
122
94
 
123
- Or create it manually:
124
-
125
- ```typescript
126
- // agent-health.config.ts
127
- export default {
128
- agents: [
129
- {
130
- key: "my-agent",
131
- name: "My Agent",
132
- endpoint: "http://localhost:8000/agent",
133
- connectorType: "rest", // or "agui-streaming", "subprocess"
134
- models: ["claude-sonnet-4"],
135
- useTraces: true, // Enable OpenTelemetry trace collection
136
- }
137
- ],
138
- };
139
- ```
95
+ This clones the repo, starts the Docker stack, waits for OpenSearch, auto-configures `agent-health.config.json`, and launches Agent Health.
140
96
 
141
- The config file is auto-detected from the current working directory. Supported file names (in priority order): `agent-health.config.ts`, `agent-health.config.js`, `agent-health.config.mjs`. See [`agent-health.config.example.ts`](./agent-health.config.example.ts) for all available options including authentication hooks.
97
+ <details>
98
+ <summary><b>Or step-by-step:</b></summary>
142
99
 
143
- > **Tip:** Run `npx @opensearch-project/agent-health doctor` to verify your configuration is loaded correctly.
100
+ ```bash
101
+ # Clone the repository
102
+ git clone https://github.com/opensearch-project/agent-health.git
103
+ cd agent-health
144
104
 
145
- ### Environment Variables (Optional)
105
+ # Start the OpenSearch observability stack
106
+ docker compose up -d
146
107
 
147
- **For LLM Judge evaluation** (uses AWS Bedrock):
148
- ```bash
149
- # Create .env file
150
- cp .env.example .env
108
+ # Copy Docker environment configuration
109
+ cp .env.docker .env
151
110
 
152
- # Add AWS credentials
153
- AWS_REGION=us-east-1
154
- AWS_ACCESS_KEY_ID=your_access_key
155
- AWS_SECRET_ACCESS_KEY=your_secret_key
111
+ # Start Agent Health (connects to local OpenSearch automatically)
112
+ npx @opensearch-project/agent-health
156
113
  ```
114
+ </details>
157
115
 
158
- **Full configuration guide:** [CONFIGURATION.md](./docs/CONFIGURATION.md)
116
+ This brings up:
117
+ - **OpenSearch** — Stores traces, test cases, benchmarks, and evaluation results
118
+ - **OpenTelemetry Collector** — Receives telemetry data via OTLP (ports 4317/4318)
119
+ - **Data Prepper** — Transforms and enriches traces before OpenSearch ingestion
159
120
 
160
- ---
121
+ > **Prerequisites:** Docker Desktop with 4GB+ memory allocated. See [docker-compose.yml](./docker-compose.yml) for configuration options.
161
122
 
162
- ## Development Commands
123
+ ### Option 3: AWS CloudFormation (Managed OpenSearch)
163
124
 
164
- | Command | Description |
165
- |---------|-------------|
166
- | `npm install` | Install dependencies |
167
- | `npm run dev` | Start frontend dev server (port 4000) |
168
- | `npm run dev:server` | Start backend server (port 4001) |
169
- | `npm run build` | TypeScript compile + Vite production build |
170
- | `npm test` | Run all tests |
171
- | `npm run test:unit` | Run unit tests only |
172
- | `npm run test:integration` | Run integration tests only |
173
- | `npm run test:e2e` | Run E2E tests with Playwright |
174
- | `npm run test:e2e:ui` | Run E2E tests with Playwright UI |
175
- | `npm run test:all` | Run all tests (unit + integration + e2e) |
176
- | `npm test -- --coverage` | Run tests with coverage report |
177
- | `npm run build:all` | Build UI + server + CLI |
178
- | `npm run build:cli` | Build CLI only |
179
-
180
- ### Production Mode
125
+ Deploy a fully managed observability backend using the included CloudFormation template:
181
126
 
182
127
  ```bash
183
- npm run server # Build UI + start single server on port 4001
128
+ aws cloudformation create-stack \
129
+ --stack-name AgentHealthObservability \
130
+ --template-body file://deployment/cloudformation/agent-health-observability.yaml \
131
+ --capabilities CAPABILITY_NAMED_IAM
184
132
  ```
185
133
 
186
- Open http://localhost:4001
134
+ This deploys:
135
+ - **Amazon OpenSearch Service** domain for trace storage
136
+ - **OpenSearch Ingestion (OSIS)** pipeline for OTLP data collection
137
+ - **IAM roles** for pipeline execution and agent telemetry ingestion
187
138
 
188
- ### NPX Usage
189
-
190
- After publishing, run directly with npx:
139
+ After deployment, connect it to Agent Health:
191
140
 
192
141
  ```bash
193
- npx @opensearch-project/agent-health # Start server on port 4001
194
- npx @opensearch-project/agent-health --port 8080
195
- npx @opensearch-project/agent-health --env-file .env
142
+ npx @opensearch-project/agent-health configure --from-stack AgentHealthObservability
196
143
  ```
197
144
 
198
- ### Ports Summary
145
+ Or manually copy the `AgentHealthConfigJSON` stack output into your `agent-health.config.json`. See [deployment/cloudformation/](./deployment/cloudformation/) for details and regional Launch Stack URLs.
199
146
 
200
- | Mode | Command | Port(s) |
201
- |------|---------|---------|
202
- | **Dev (frontend)** | `npm run dev` | 4000 |
203
- | **Dev (backend)** | `npm run dev:server` | 4001 |
204
- | **Production** | `npm run server` | 4001 |
205
- | **NPX** | `npx @opensearch-project/agent-health` | 4001 (default) |
147
+ ### Next Steps
206
148
 
207
- In development, the Vite dev server (4000) proxies `/api` requests to the backend (4001).
149
+ - [Getting Started Guide](./GETTING_STARTED.md) — Step-by-step walkthrough from install to first evaluation
150
+ - [Configuration Guide](./docs/CONFIGURATION.md) — Connect your own agent and configure the environment
151
+ - [CLI Reference](./docs/CLI.md) — Full command-line documentation
208
152
 
209
153
  ---
210
154
 
211
- ## Testing
212
-
213
- AgentEval uses a comprehensive test suite with three layers:
214
-
215
- ### Test Types
216
-
217
- | Type | Location | Command | Description |
218
- |------|----------|---------|-------------|
219
- | **Unit** | `tests/unit/` | `npm run test:unit` | Fast, isolated function tests |
220
- | **Integration** | `tests/integration/` | `npm run test:integration` | Tests with real backend server |
221
- | **E2E** | `tests/e2e/` | `npm run test:e2e` | Browser-based UI tests with Playwright |
222
-
223
- ### Running Tests
224
-
225
- ```bash
226
- # All tests
227
- npm test # Unit + integration
228
- npm run test:all # Unit + integration + E2E
229
-
230
- # By type
231
- npm run test:unit # Unit tests only
232
- npm run test:integration # Integration tests (starts server)
233
- npm run test:e2e # E2E tests (starts servers)
234
- npm run test:e2e:ui # E2E with Playwright UI for debugging
235
-
236
- # With coverage
237
- npm run test:unit -- --coverage
238
-
239
- # Specific file
240
- npm test -- path/to/file.test.ts
241
- npx playwright test tests/e2e/dashboard.spec.ts
242
- ```
155
+ <a id="features"></a>
156
+ ## Features
243
157
 
244
- ### E2E Testing with Playwright
158
+ ### Agent Evaluation & Observability
245
159
 
246
- E2E tests use [Playwright](https://playwright.dev/) to test the UI in a real browser.
160
+ | Feature | Description |
161
+ |---------|-------------|
162
+ | **Evals** | Real-time agent evaluation with trajectory streaming |
163
+ | **Experiments** | Batch evaluation runs with configurable parameters |
164
+ | **Compare** | Side-by-side trace comparison with aligned and merged views |
165
+ | **Agent Traces** | Table-based trace view with latency histogram, filtering, and detailed flyout |
166
+ | **Live Traces** | Real-time trace monitoring with auto-refresh and filtering |
167
+ | **Trace Views** | Timeline and Flow visualizations for debugging |
168
+ | **Reports** | Evaluation reports with LLM judge reasoning |
169
+ | **Connectors** | Pluggable protocol adapters (AG-UI SSE, REST, CLI, Claude Code) |
247
170
 
248
- ```bash
249
- # First time: install browsers
250
- npx playwright install
171
+ ### Coding Agent Analytics
251
172
 
252
- # Run all E2E tests
253
- npm run test:e2e
173
+ A unified dashboard for monitoring AI coding agent usage across **Claude Code**, **Kiro**, and **Codex CLI**. Zero configuration — just run `agent-health` and it auto-detects installed agents.
254
174
 
255
- # Interactive UI mode (recommended for debugging)
256
- npm run test:e2e:ui
175
+ - **Multi-agent dashboard**: Session history, cost estimation, tool usage, activity patterns, and efficiency metrics
176
+ - **9 analytics tabs**: Overview, Sessions, Projects, Costs, Activity, Efficiency, Tools, Advanced, and Workspace management
177
+ - **Interactive drill-downs**: Click any chart, card, or metric to drill into filtered session views
178
+ - **Workspace management**: View and edit Claude Code memory files, plans, tasks; browse Kiro MCP servers, agents, and extensions
179
+ - **Privacy-first**: All data stays local — reads directly from `~/.claude/`, `~/.kiro/`, `~/.codex/`
257
180
 
258
- # View test report
259
- npm run test:e2e:report
260
- ```
181
+ [Full Coding Agent Analytics documentation](./docs/CODING_AGENT_ANALYTICS.md)
261
182
 
262
- **Writing E2E Tests:**
263
- - Place tests in `tests/e2e/*.spec.ts`
264
- - Use `data-testid` attributes for reliable selectors
265
- - Handle empty states gracefully (check if data exists before asserting)
266
- - See existing tests for patterns
183
+ ### Supported Connectors
267
184
 
268
- ### CI Pipeline
185
+ | Connector | Protocol | Description |
186
+ |-----------|----------|-------------|
187
+ | `agui-streaming` | AG-UI SSE | ML-Commons agents (default) |
188
+ | `rest` | HTTP POST | Non-streaming REST APIs |
189
+ | `openai-compatible` | OpenAI Chat | LiteLLM, Ollama, vLLM |
190
+ | `strands` | Bedrock Agent Runtime | Amazon Strands agents (server-only) |
191
+ | `langgraph` | LangGraph REST | Non-AG-UI LangGraph instances |
192
+ | `subprocess` | CLI | Command-line tools |
193
+ | `claude-code` | Claude CLI | Claude Code agent comparison |
194
+ | `mock` | In-memory | Demo and testing |
269
195
 
270
- All PRs must pass these CI checks:
196
+ For creating custom connectors, see [docs/CONNECTORS.md](./docs/CONNECTORS.md).
271
197
 
272
- | Job | What it checks |
273
- |-----|----------------|
274
- | `build-and-test` | Build + unit tests + 90% coverage |
275
- | `lint-and-typecheck` | TypeScript compilation |
276
- | `license-check` | SPDX headers on all source files |
277
- | `integration-tests` | Backend integration tests with coverage |
278
- | `e2e-tests` | Playwright browser tests with pass/fail tracking |
279
- | `security-scan` | npm audit for vulnerabilities |
280
- | `test-summary` | Consolidated test results summary |
198
+ ### Observio Sample Agent
281
199
 
282
- ### Coverage Thresholds
200
+ Agent Health includes **Observio**, a reference ReAct agent you can use as a practice target for evaluating and improving agent performance:
283
201
 
284
- | Test Type | Metric | Threshold |
285
- |-----------|--------|-----------|
286
- | Unit | Lines | ≥ 90% |
287
- | Unit | Branches | ≥ 80% |
288
- | Unit | Functions | ≥ 80% |
289
- | Unit | Statements | ≥ 90% |
290
- | Integration | Lines | Informational (no threshold) |
291
- | E2E | Pass Rate | 100% |
202
+ ```bash
203
+ cd observio-sample-agent && npm install && npm run start:ag-ui
204
+ npx @opensearch-project/agent-health run -t demo-otel-001 -a observio
205
+ ```
292
206
 
293
- ### CI Artifacts
207
+ See the [Observio README](./observio-sample-agent/README.md) for details.
294
208
 
295
- Each CI run produces these artifacts (downloadable from Actions tab):
209
+ ---
296
210
 
297
- | Artifact | Contents |
298
- |----------|----------|
299
- | `coverage-report` | Unit test coverage (HTML, LCOV) |
300
- | `integration-coverage-report` | Integration test coverage |
301
- | `playwright-report` | E2E test report with screenshots/traces |
302
- | `test-badges` | Badge data JSON for coverage visualization |
211
+ <a id="architecture"></a>
212
+ ## Architecture
303
213
 
304
- ### Full Evaluation Flow E2E Tests
214
+ <p align="center">
215
+ <img alt="Agent Health Architecture" src="docs/diagrams/architecture.png" />
216
+ </p>
305
217
 
306
- The E2E test suite includes tests for the complete evaluation flow using mock modes:
307
- - **Demo Agent** (`mock://demo`) - Simulated AG-UI streaming responses
308
- - **Demo Model** (`provider: "demo"`) - Simulated LLM judge evaluation
218
+ Agent Health uses a client-server architecture where all clients (UI, CLI) access OpenSearch through a unified HTTP API. The server handles agent communication via pluggable connectors and proxies LLM judge calls to AWS Bedrock.
309
219
 
310
- This allows testing the full Create Test Case → Create Benchmark → Run Evaluation → View Results flow without requiring AWS credentials or a live agent in CI.
220
+ For detailed architecture documentation, see [docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md).
311
221
 
312
222
  ---
313
223
 
314
- ## Agent Setup
315
-
316
- Agent Health supports multiple agent types:
224
+ <a id="quick-configuration"></a>
225
+ ## Quick Configuration
317
226
 
318
- | Agent | Endpoint Variable | Setup |
319
- |-------|-------------------|-------|
320
- | **Observio** (sample) | `localhost:3001` | Included — see [observio-sample-agent/](./observio-sample-agent/) |
321
- | Langgraph | `LANGGRAPH_ENDPOINT` | Simple localhost agent |
322
- | HolmesGPT | `HOLMESGPT_ENDPOINT` | AG-UI compatible RCA agent |
323
- | ML-Commons | `MLCOMMONS_ENDPOINT` | See [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) |
324
-
325
- ### Observio Sample Agent
326
-
327
- Agent Health includes **Observio**, a reference ReAct agent you can use as a practice target for evaluating and improving agent performance. It's a great starting point if you don't have your own agent yet.
227
+ Agent Health works out-of-the-box with demo data. Configure when you're ready to connect your own agent:
328
228
 
329
229
  ```bash
330
- # Start Observio
331
- cd observio-sample-agent && npm install && npm run start:ag-ui
230
+ # Generate a config file with examples
231
+ npx @opensearch-project/agent-health init
232
+ ```
332
233
 
333
- # Evaluate it with Agent Health
334
- npx @opensearch-project/agent-health run -t demo-otel-001 -a observio
234
+ ```typescript
235
+ // agent-health.config.ts
236
+ export default {
237
+ agents: [
238
+ {
239
+ key: "my-agent",
240
+ name: "My Agent",
241
+ endpoint: "http://localhost:8000/agent",
242
+ connectorType: "rest", // or "agui-streaming", "langgraph", "strands", "subprocess"
243
+ models: ["claude-sonnet-4"],
244
+ useTraces: true, // Enable OpenTelemetry trace collection
245
+ }
246
+ ],
247
+ };
335
248
  ```
336
249
 
337
- See the [Observio README](./observio-sample-agent/README.md) for setup details and improvement areas.
250
+ > **Tip:** Run `npx @opensearch-project/agent-health doctor` to verify your configuration is loaded correctly.
338
251
 
252
+ For full configuration options including authentication hooks and environment variables, see [CONFIGURATION.md](./docs/CONFIGURATION.md).
339
253
 
340
254
  ---
341
255
 
342
- ## Debugging
256
+ <a id="star-history"></a>
257
+ ## Star History
343
258
 
344
- Enable verbose debug logging to diagnose issues:
345
-
346
- ```bash
347
- # Via environment variable
348
- DEBUG=true npx @opensearch-project/agent-health
349
-
350
- # Or toggle at runtime via API
351
- curl -X POST http://localhost:4001/api/debug -H 'Content-Type: application/json' -d '{"enabled":true}'
352
- ```
259
+ If you find Agent Health useful, please consider giving us a star! Your support helps us grow our community and continue improving the project.
353
260
 
354
- Debug logging can also be toggled from the **Settings** page using the "Verbose Logging" switch, which syncs to both the browser console and server terminal.
261
+ [![Star History Chart](https://api.star-history.com/svg?repos=opensearch-project/agent-health&type=Date)](https://github.com/opensearch-project/agent-health)
355
262
 
356
263
  ---
357
264
 
358
- ## Troubleshooting
265
+ <a id="contributing"></a>
266
+ ## Contributing
359
267
 
360
- | Issue | Solution |
361
- |-------|----------|
362
- | Cannot connect to backend | Run `npm run dev:server`, check `curl http://localhost:4001/health` |
363
- | AWS credentials expired | Refresh credentials in `.env` |
364
- | Storage/Traces not working | Check OpenSearch endpoint and credentials in `.env` |
365
- | Need verbose logs | Set `DEBUG=true` in `.env` or toggle in Settings page |
268
+ We welcome contributions! There are many ways to get involved:
366
269
 
367
- ---
270
+ - [Report a Bug](https://github.com/opensearch-project/agent-health/issues/new/choose) — Found something broken? Let us know
271
+ - [Request a Feature](https://github.com/opensearch-project/agent-health/issues/new/choose) — Have an idea? We'd love to hear it
272
+ - [Submit a Pull Request](https://github.com/opensearch-project/agent-health/pulls) — Code contributions are always welcome
273
+ - [Join the Discussion](https://opensearch.org/slack.html) — Chat with us on the OpenSearch Slack
368
274
 
369
- ## Contributing
275
+ ### Development Quick Start
370
276
 
371
- We welcome contributions! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.
277
+ ```bash
278
+ git clone https://github.com/opensearch-project/agent-health.git
279
+ cd agent-health
280
+ npm install
281
+ npm run dev # Frontend on port 4000
282
+ npm run dev:server # Backend on port 4001
283
+ ```
372
284
 
373
- ### Development Workflow
285
+ > **Port conflicts:** If port 4001 is already in use, the backend server automatically tries 4002, 4003, etc. (up to 10 attempts). The actual port is displayed in the console output.
374
286
 
375
- 1. Fork and clone the repository
376
- 2. Install dependencies: `npm install`
377
- 3. Create a feature branch: `git checkout -b feature/your-feature`
378
- 4. Make changes and add tests
379
- 5. Run tests: `npm test`
380
- 6. Commit with DCO signoff: `git commit -s -m "feat: your message"`
381
- 7. Push and create a Pull Request
287
+ All commits require DCO signoff (`git commit -s`) and all PRs must pass CI checks.
382
288
 
383
- All commits require DCO signoff and all PRs must pass CI checks (tests, coverage, linting).
289
+ For detailed development setup, testing, CI pipeline, debugging, and troubleshooting, see the [Developer Guide](./DEVELOPER_GUIDE.md). For full contribution guidelines, see [CONTRIBUTING.md](./CONTRIBUTING.md).
384
290
 
385
291
  ---
386
292
 
387
293
  ## Documentation
388
294
 
389
- ### User Guides
390
- - [Getting Started](./GETTING_STARTED.md) - Step-by-step walkthrough from install to first evaluation
391
- - [Configuration](./docs/CONFIGURATION.md) - Connect your agent and configure the environment
392
- - [CLI Reference](./docs/CLI.md) - Command-line interface documentation
393
- - [Observio Sample Agent](./observio-sample-agent/) - Reference agent for practicing agent health improvements
394
-
395
- ### Developer Guides
396
- - [Development Guide](./CLAUDE.md) - Architecture, coding conventions, and contributing
397
- - [Connectors Guide](./docs/CONNECTORS.md) - Create custom connectors for your agent type
398
- - [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) - OpenSearch ML-Commons integration
399
- - [Architecture](./docs/ARCHITECTURE.md) - System design and patterns
295
+ | Guide | Description |
296
+ |-------|-------------|
297
+ | [Getting Started](./GETTING_STARTED.md) | Step-by-step walkthrough from install to first evaluation |
298
+ | [Configuration](./docs/CONFIGURATION.md) | Connect your agent and configure the environment |
299
+ | [CLI Reference](./docs/CLI.md) | Command-line interface documentation |
300
+ | [Coding Agent Analytics](./docs/CODING_AGENT_ANALYTICS.md) | Multi-agent dashboard and remote server monitoring |
301
+ | [Observio Sample Agent](./observio-sample-agent/) | Reference agent for practicing evaluations |
302
+ | [Developer Guide](./DEVELOPER_GUIDE.md) | Development setup, testing, CI, debugging |
303
+ | [Connectors Guide](./docs/CONNECTORS.md) | Create custom connectors for your agent type |
304
+ | [Architecture](./docs/ARCHITECTURE.md) | System design and patterns |
305
+ | [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) | OpenSearch ML-Commons integration |
306
+
307
+ ---
308
+
309
+ <p align="center">
310
+ Made with care by the <a href="https://opensearch.org">OpenSearch</a> community
311
+ </p>