@opensearch-project/agent-health 0.0.1 → 0.1.0

package/LICENSE.txt

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -0,0 +1,363 @@
+# Agent Health
+
+[![CI](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml/badge.svg)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE.txt)
+[![npm version](https://img.shields.io/npm/v/@opensearch-project/agent-health.svg)](https://www.npmjs.com/package/@opensearch-project/agent-health)
+
+[![Unit Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/unit-tests.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
+[![Unit Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/unit-coverage.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
+[![Integration Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/integration-tests.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
+[![E2E Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/opensearch-project/agent-health/badges/e2e-tests.json)](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
+
+An evaluation and observability framework for AI agents. Features real-time trace visualization, "Golden Path" trajectory comparison, and LLM-based evaluation scoring.
+
+Try It by running:
+
+```bash
+npx @goyamegh/agent-health@latest
+```
+
+Opens http://localhost:4001 for the web UI.
+
+### Architecture
+
+![Agent Health Architecture](docs/diagrams/architecture.png)
+
+## Features
+
+- **Evals**: Real-time agent evaluation with trajectory streaming
+- **Experiments**: Batch evaluation runs with configurable parameters
+- **Compare**: Side-by-side trace comparison with aligned and merged views
+- **Agent Traces**: Table-based trace view with latency histogram, filtering, and detailed flyout with input/output display
+- **Live Traces**: Real-time trace monitoring with auto-refresh and filtering
+- **Trace Views**: Timeline and Flow visualizations for debugging
+- **Reports**: Evaluation reports with LLM judge reasoning
+- **Connectors**: Pluggable protocol adapters for different agent types
+
+For a detailed walkthrough, see [Getting Started](./GETTING_STARTED.md).
+
+
+### Supported Connectors
+
+| Connector | Protocol | Description |
+|-----------|----------|-------------|
+| `agui-streaming` | AG-UI SSE | ML-Commons agents (default) |
+| `rest` | HTTP POST | Non-streaming REST APIs |
+| `subprocess` | CLI | Command-line tools |
+| `claude-code` | Claude CLI | Claude Code agent comparison |
+| `mock` | In-memory | Demo and testing |
+
+For creating custom connectors, see [docs/CONNECTORS.md](./docs/CONNECTORS.md).
+
+---
+
+
+
+---
+
+## Quick Start
+
+```bash
+# Start the web UI
+npx @opensearch-project/agent-health
+
+# Open http://localhost:4001
+```
+
+### CLI Commands
+
+```bash
+# Check configuration
+npx @opensearch-project/agent-health doctor
+
+# List available agents and connectors
+npx @opensearch-project/agent-health list agents
+npx @opensearch-project/agent-health list connectors
+
+# Run a test case against an agent
+npx @opensearch-project/agent-health run -t demo-otel-001 -a demo
+
+# Initialize a new project
+npx @opensearch-project/agent-health init
+```
+
+For full CLI documentation, see [docs/CLI.md](./docs/CLI.md).
+
+
+
+
+## Authentication (Required)
+
+AWS credentials are required for the Bedrock LLM Judge to score evaluations.
+
+Create a `.env` file:
+```bash
+cp .env.example .env
+```
+
+Add your AWS credentials:
+```bash
+AWS_REGION=us-east-1
+AWS_ACCESS_KEY_ID=your_access_key
+AWS_SECRET_ACCESS_KEY=your_secret_key
+AWS_SESSION_TOKEN=your_session_token  # if using temporary credentials
+```
+
+---
+
+## Configuration (Optional)
+
+All optional settings have sensible defaults. Configure only what you need.
+
+### Agent Endpoints
+
+Agent endpoints default to localhost. Override if your agent runs elsewhere:
+
+```bash
+LANGGRAPH_ENDPOINT=http://localhost:3000
+HOLMESGPT_ENDPOINT=http://localhost:5050/api/agui/chat
+MLCOMMONS_ENDPOINT=http://localhost:9200/_plugins/_ml/agents/{agent_id}/_execute/stream
+```
+
+### Storage (Persistence)
+
+For persisting test cases, experiments, and runs. Features gracefully degrade if not configured.
+
+```bash
+OPENSEARCH_STORAGE_ENDPOINT=https://your-cluster.opensearch.amazonaws.com
+OPENSEARCH_STORAGE_USERNAME=admin
+OPENSEARCH_STORAGE_PASSWORD=your_password
+OPENSEARCH_STORAGE_TLS_SKIP_VERIFY=false  # Set to true for self-signed certificates
+```
+
+### Traces (Observability)
+
+For agent execution traces. Features gracefully degrade if not configured.
+
+```bash
+OPENSEARCH_LOGS_ENDPOINT=https://your-logs-cluster.opensearch.amazonaws.com
+OPENSEARCH_LOGS_USERNAME=admin
+OPENSEARCH_LOGS_PASSWORD=your_password
+OPENSEARCH_LOGS_TLS_SKIP_VERIFY=false  # Set to true for self-signed certificates
+```
+
+See `.env.example` for all available options.
+
+---
+
+## Development Commands
+
+| Command | Description |
+|---------|-------------|
+| `npm install` | Install dependencies |
+| `npm run dev` | Start frontend dev server (port 4000) |
+| `npm run dev:server` | Start backend server (port 4001) |
+| `npm run build` | TypeScript compile + Vite production build |
+| `npm test` | Run all tests |
+| `npm run test:unit` | Run unit tests only |
+| `npm run test:integration` | Run integration tests only |
+| `npm run test:e2e` | Run E2E tests with Playwright |
+| `npm run test:e2e:ui` | Run E2E tests with Playwright UI |
+| `npm run test:all` | Run all tests (unit + integration + e2e) |
+| `npm test -- --coverage` | Run tests with coverage report |
+| `npm run build:all` | Build UI + server + CLI |
+| `npm run build:cli` | Build CLI only |
+
+### Production Mode
+
+```bash
+npm run server  # Build UI + start single server on port 4001
+```
+
+Open http://localhost:4001
+
+### NPX Usage
+
+After publishing, run directly with npx:
+
+```bash
+npx @opensearch-project/agent-health           # Start server on port 4001
+npx @opensearch-project/agent-health --port 8080
+npx @opensearch-project/agent-health --env-file .env
+```
+
+### Ports Summary
+
+| Mode | Command | Port(s) |
+|------|---------|---------|
+| **Dev (frontend)** | `npm run dev` | 4000 |
+| **Dev (backend)** | `npm run dev:server` | 4001 |
+| **Production** | `npm run server` | 4001 |
+| **NPX** | `npx @opensearch-project/agent-health` | 4001 (default) |
+
+In development, the Vite dev server (4000) proxies `/api` requests to the backend (4001).
+
+---
+
+## Testing
+
+AgentEval uses a comprehensive test suite with three layers:
+
+### Test Types
+
+| Type | Location | Command | Description |
+|------|----------|---------|-------------|
+| **Unit** | `tests/unit/` | `npm run test:unit` | Fast, isolated function tests |
+| **Integration** | `tests/integration/` | `npm run test:integration` | Tests with real backend server |
+| **E2E** | `tests/e2e/` | `npm run test:e2e` | Browser-based UI tests with Playwright |
+
+### Running Tests
+
+```bash
+# All tests
+npm test                        # Unit + integration
+npm run test:all                # Unit + integration + E2E
+
+# By type
+npm run test:unit               # Unit tests only
+npm run test:integration        # Integration tests (starts server)
+npm run test:e2e                # E2E tests (starts servers)
+npm run test:e2e:ui             # E2E with Playwright UI for debugging
+
+# With coverage
+npm run test:unit -- --coverage
+
+# Specific file
+npm test -- path/to/file.test.ts
+npx playwright test tests/e2e/dashboard.spec.ts
+```
+
+### E2E Testing with Playwright
+
+E2E tests use [Playwright](https://playwright.dev/) to test the UI in a real browser.
+
+```bash
+# First time: install browsers
+npx playwright install
+
+# Run all E2E tests
+npm run test:e2e
+
+# Interactive UI mode (recommended for debugging)
+npm run test:e2e:ui
+
+# View test report
+npm run test:e2e:report
+```
+
+**Writing E2E Tests:**
+- Place tests in `tests/e2e/*.spec.ts`
+- Use `data-testid` attributes for reliable selectors
+- Handle empty states gracefully (check if data exists before asserting)
+- See existing tests for patterns
+
+### CI Pipeline
+
+All PRs must pass these CI checks:
+
+| Job | What it checks |
+|-----|----------------|
+| `build-and-test` | Build + unit tests + 90% coverage |
+| `lint-and-typecheck` | TypeScript compilation |
+| `license-check` | SPDX headers on all source files |
+| `integration-tests` | Backend integration tests with coverage |
+| `e2e-tests` | Playwright browser tests with pass/fail tracking |
+| `security-scan` | npm audit for vulnerabilities |
+| `test-summary` | Consolidated test results summary |
+
+### Coverage Thresholds
+
+| Test Type | Metric | Threshold |
+|-----------|--------|-----------|
+| Unit | Lines | ≥ 90% |
+| Unit | Branches | ≥ 80% |
+| Unit | Functions | ≥ 80% |
+| Unit | Statements | ≥ 90% |
+| Integration | Lines | Informational (no threshold) |
+| E2E | Pass Rate | 100% |
+
+### CI Artifacts
+
+Each CI run produces these artifacts (downloadable from Actions tab):
+
+| Artifact | Contents |
+|----------|----------|
+| `coverage-report` | Unit test coverage (HTML, LCOV) |
+| `integration-coverage-report` | Integration test coverage |
+| `playwright-report` | E2E test report with screenshots/traces |
+| `test-badges` | Badge data JSON for coverage visualization |
+
+### Full Evaluation Flow E2E Tests
+
+The E2E test suite includes tests for the complete evaluation flow using mock modes:
+- **Demo Agent** (`mock://demo`) - Simulated AG-UI streaming responses
+- **Demo Model** (`provider: "demo"`) - Simulated LLM judge evaluation
+
+This allows testing the full Create Test Case → Create Benchmark → Run Evaluation → View Results flow without requiring AWS credentials or a live agent in CI.
+
+---
+
+## Agent Setup
+
+Agent Health supports multiple agent types:
+
+| Agent | Endpoint Variable | Setup |
+|-------|-------------------|-------|
+| Langgraph | `LANGGRAPH_ENDPOINT` | Simple localhost agent |
+| HolmesGPT | `HOLMESGPT_ENDPOINT` | AG-UI compatible RCA agent |
+| ML-Commons | `MLCOMMONS_ENDPOINT` | See [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) |
+
+
+---
+
+## Debugging
+
+Enable verbose debug logging to diagnose issues:
+
+```bash
+# Via environment variable
+DEBUG=true npx @opensearch-project/agent-health
+
+# Or toggle at runtime via API
+curl -X POST http://localhost:4001/api/debug -H 'Content-Type: application/json' -d '{"enabled":true}'
+```
+
+Debug logging can also be toggled from the **Settings** page using the "Verbose Logging" switch, which syncs to both the browser console and server terminal.
+
+---
+
+## Troubleshooting
+
+| Issue | Solution |
+|-------|----------|
+| Cannot connect to backend | Run `npm run dev:server`, check `curl http://localhost:4001/health` |
+| AWS credentials expired | Refresh credentials in `.env` |
+| Storage/Traces not working | Check OpenSearch endpoint and credentials in `.env` |
+| Need verbose logs | Set `DEBUG=true` in `.env` or toggle in Settings page |
+
+---
+
+## Contributing
+
+We welcome contributions! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.
+
+### Development Workflow
+
+1. Fork and clone the repository
+2. Install dependencies: `npm install`
+3. Create a feature branch: `git checkout -b feature/your-feature`
+4. Make changes and add tests
+5. Run tests: `npm test`
+6. Commit with DCO signoff: `git commit -s -m "feat: your message"`
+7. Push and create a Pull Request
+
+All commits require DCO signoff and all PRs must pass CI checks (tests, coverage, linting).
+
+---
+
+## Documentation
+
+- [Getting Started](./GETTING_STARTED.md) - Installation, demo mode, and usage walkthrough
+- [ML-Commons Agent Setup](./docs/ML-COMMONS-SETUP.md) - Configure ML-Commons agent
+- [Development Guide](./CLAUDE.md) - Architecture and coding conventions
+- [AG-UI Protocol](https://docs.ag-ui.com/sdk/js/core/types#runagentinput)

package/bin/cli.js

@@ -0,0 +1,32 @@
+#!/usr/bin/env node
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+/**
+ * CLI Entry Point
+ *
+ * In development: Runs TypeScript source directly via tsx (instant changes)
+ * In production: Runs pre-built JavaScript from cli/dist/
+ */
+
+import { existsSync } from 'fs';
+import { fileURLToPath } from 'url';
+import { dirname, join } from 'path';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const cliSourcePath = join(__dirname, '..', 'cli', 'index.ts');
+const isInNodeModules = import.meta.url.split('/').includes('node_modules');
+
+// Development mode: TypeScript source exists and not running from node_modules
+const isDev = existsSync(cliSourcePath) && !isInNodeModules;
+
+if (isDev) {
+  // Import tsx to enable TypeScript execution, then run source
+  await import('tsx/esm');
+  await import('../cli/index.ts');
+} else {
+  // Production: use pre-built JavaScript
+  await import('../cli/dist/index.js');
+}