@opensearch-project/agent-health 0.3.0 → 0.5.2

package/docs/SKILLS.md

@@ -0,0 +1,264 @@
+<!--
+  * Copyright OpenSearch Contributors
+  * SPDX-License-Identifier: Apache-2.0
+-->
+
+# Skills Evaluator — Authoring Guide
+
+The Skills Evaluator runs an A/B benchmark on an [AgentSkills](https://agentskills.io/) /
+[Claude Code skill](https://docs.claude.com/en/docs/claude-code/skills): one set of
+agent runs **with** the skill injected, one set **without**, then grades the same
+assertions on both trajectories and reports the delta in pass-rate, latency, and
+tokens. Optionally, an LLM proposes an improved `SKILL.md` body from the failure
+evidence and (with `auto`) applies it.
+
+This guide tells you **how to write a skill that the evaluator can score
+fairly** — and what every warning / status the UI shows actually means.
+
+```
+┌──────────┐  ┌──────────────┐   ┌──────────────┐   ┌────────────┐
+│ SKILL.md │→ │ parseSkill   │→  │ runSkillEval │→  │ benchmark  │
+└──────────┘  │  validate +  │   │  with vs     │   │  + diff +  │
+              │  warn        │   │  without     │   │  improve   │
+              └──────────────┘   └──────────────┘   └────────────┘
+```
+
+---
+
+## 1. Anatomy of a skill folder
+
+```
+my-skill/
+├── SKILL.md            # required — frontmatter + body
+└── evals/
+    └── evals.json      # optional — auto-generated if absent
+```
+
+`SKILL.md` is a markdown file with a YAML frontmatter block delimited by `---`
+on lines by themselves:
+
+```markdown
+---
+name: rca-log-analysis
+description: Use when the user asks why a service is failing or wants to find
+  the root cause from log lines. Triggers on terms like "why is X failing",
+  "RCA", "root cause", "error spike".
+allowed-tools:
+  - Read
+  - Bash(git:*)
+license: Apache-2.0
+---
+
+# Root-cause analysis from logs
+
+When the user asks why a service is failing:
+
+1. Pull the last 5 minutes of logs from the suspect service.
+2. Group by error code; the dominant code is your starting hypothesis.
+3. ...
+```
+
+### Frontmatter rules (enforced by `parseSkill`)
+
+| Field           | Required | Constraint                                                              |
+| --------------- | -------- | ----------------------------------------------------------------------- |
+| `name`          | yes      | lowercase kebab-case, ≤ 64 chars                                        |
+| `description`   | yes      | ≤ 1024 chars; **should describe trigger conditions** (see §2)           |
+| `allowed-tools` | no       | YAML list **or** whitespace-separated string (`Bash(git:*)` patterns ok) |
+| `license`       | no       | free-form string                                                        |
+| `compatibility` | no       | free-form string                                                        |
+| `metadata`      | no       | arbitrary YAML mapping                                                  |
+
+The closing `---` must be on its own line. CRLF line endings are normalised
+before matching.
+
+---
+
+## 2. Why your `description` is the most important field
+
+In Claude Code's skill loader the `description` is what the matcher
+**actually reads** when deciding whether to inject the skill into a given
+turn. The body is only loaded after the description has matched. A
+description that reads like a tagline ("A helper for log analysis") will
+silently fail to fire.
+
+The parser surfaces two warnings for this:
+
+| Warning text                                                                                                                 | What to do                                                                                                                                       |
+| ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `description is only N characters … aim for ≥ 30 chars and explicitly state trigger conditions` | The description is too short to discriminate from peers. Expand it.                                                                              |
+| `description does not appear to state when the skill should be used`                                                         | Add a phrase like **`Use when …`**, **`For requests about …`**, **`Trigger on …`**, **`Invoke when the user asks …`**. The matcher looks for these. |
+
+Good descriptions:
+
+> ✅ "Use when the user asks why a service is failing or wants RCA from logs.
+> Triggers on phrases like 'why is X erroring', 'root cause', 'error spike'."
+
+> ✅ "For requests about diffing two trajectories — invoke when the user
+> mentions 'compare runs', 'why did the agent behave differently', or pastes
+> two run IDs."
+
+Bad descriptions (parser will warn):
+
+> ❌ "Helper for logs."
+> ❌ "Skill for trace analysis."
+> ❌ "Improves debugging."
+
+---
+
+## 3. Authoring `evals/evals.json`
+
+Each eval case is a triple **prompt → expected_output → assertions**. The
+evaluator runs the prompt twice (with and without the skill), then asks the
+LLM judge whether **each** assertion holds against the resulting trajectory.
+
+```json
+{
+  "skill_name": "rca-log-analysis",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "The checkout service started returning 503s 10 minutes ago. Why?",
+      "expected_output": "Identifies a downstream dependency or resource exhaustion as the cause.",
+      "assertions": [
+        "The agent searches logs for the affected service before guessing",
+        "The agent groups errors by error code or status",
+        "The agent's final answer names a specific suspect (service / resource / config)"
+      ]
+    }
+  ]
+}
+```
+
+Tips:
+
+- **Make assertions discriminating.** A good assertion is one a *baseline*
+  agent (without the skill) would plausibly fail — that's the only way the
+  A/B delta surfaces real value. Trivial assertions like "the response is in
+  English" don't move the needle.
+- **2–3 assertions per eval is plenty.** Each assertion costs one judge call.
+- **`expected_output` is free-form** — it's used as judge context, not as a
+  string-match target.
+- **Auto-generation** runs only if `evals/evals.json` is missing. If it
+  fails to parse a JSON eval set from the model response, the evaluator
+  **throws with the exact JSON shape** rather than substituting a generic
+  placeholder. Hand-author the file in that case.
+
+---
+
+## 4. Reading the validation panel
+
+After you select a skill in the UI, the validation panel shows:
+
+| Symbol | Meaning                                                                                              |
+| ------ | ---------------------------------------------------------------------------------------------------- |
+| ✅     | `valid: true` — frontmatter parsed, required fields present                                          |
+| ❌     | `valid: false` — at least one hard error; the **Run Evaluation** button stays disabled               |
+| ⚠      | Warning, listed below the description. Skill still runs, but you should fix before publishing.       |
+
+Common warnings (and what they mean):
+
+| Warning                                                              | Cause / Fix                                                                                              |
+| -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| `description does not appear to state when the skill should be used` | Add a "Use when …" trigger phrase (§2)                                                                   |
+| `Instructions are ~N tokens (recommended <5000)`                     | Skill body too long. Split into multiple skills or move examples to bundled reference files.            |
+| `evals/evals.json present but invalid: …`                            | The file exists but parsing failed. The reason follows the colon (e.g. `invalid JSON`, `missing skill_name`). |
+| `No evals/evals.json found — skill cannot be evaluated …`            | Auto-generation will run on Evaluate. Provide your own to control what's tested.                         |
+
+---
+
+## 5. Reading the run progress
+
+While a run is in flight, the progress bar shows three counters:
+
+```
+N passed   N failed   ⚠ N errored
+```
+
+The distinction is intentional and matters:
+
+| Status     | Meaning                                                                                                                                                                 | Improver behaviour                                                                                                       |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `passed`   | Agent ran cleanly **and** every assertion graded true.                                                                                                                  | Treated as a positive example.                                                                                          |
+| `failed`   | Agent ran cleanly but **at least one assertion graded false**.                                                                                                          | This is the signal the improver learns from — the skill is the suspect.                                                 |
+| `errored`  | The **agent itself** crashed / endpoint unreachable / timed out — assertions were skipped because grading them would (a) waste a judge call and (b) wrongly blame the skill. | **Excluded** from improvement-proposal evidence. Fix the agent (endpoint, auth, network) before drawing skill conclusions. |
+
+If you see all `errored`, the skill is **unknowable** — the evaluator is
+telling you to debug the agent, not the skill.
+
+---
+
+## 6. Auto-applying improvement proposals (`auto: true`)
+
+When the LLM proposes an improved `SKILL.md` body, you can ship it
+automatically with the **Apply & Re-run** button (UI) or `--auto` (CLI).
+
+The server applies the change **non-destructively**:
+
+1. Snapshots the current `SKILL.md` to `SKILL.md.bak` next to it.
+2. Refuses to apply (with an explanatory error pointing to
+   `iteration-N/improvement-proposal.json`) if the original instructions
+   snapshot doesn't appear verbatim in the file — this prevents the silent
+   no-op case where `String.replace` matched nothing but the API still
+   claimed `applied: true`.
+
+To roll back: `mv SKILL.md.bak SKILL.md`.
+
+---
+
+## 7. Where skills are discovered
+
+The discover endpoint scans both **user scope** and **project scope**, in
+this order:
+
+| Path                       | Source label             | Notes                                  |
+| -------------------------- | ------------------------ | -------------------------------------- |
+| `~/.claude/skills/`        | `Claude Code (user)`     | Per Claude Code spec — user-global.    |
+| `<cwd>/.claude/skills/`    | `Claude Code`            | Project scope.                         |
+| `<cwd>/.kiro/skills/`      | `Kiro`                   |                                        |
+| `<cwd>/.kiro/steering/`    | `Kiro`                   |                                        |
+| `<cwd>/.codex/`            | `Codex`                  |                                        |
+| `<cwd>/.cursor/rules/`     | `Cursor`                 |                                        |
+| `<cwd>/.github/copilot/`   | `Copilot`                |                                        |
+| `<cwd>/.continue/skills/`  | `Continue`               |                                        |
+| `<cwd>/skills/`            | `Project`                |                                        |
+
+Duplicate skill folders (same absolute path) appear once. Home-relative
+paths render as `~/…` in the dropdown. You can also paste a path manually
+or upload a single `SKILL.md` from the UI.
+
+---
+
+## 8. CLI
+
+```bash
+# Run on a specific skill folder
+agent-health skill ./my-skill --agent claude-code --model claude-sonnet
+
+# Auto-apply the proposed improvement and re-run
+agent-health skill ./my-skill --auto
+```
+
+The CLI emits the same SSE event stream the UI consumes; `evalStatus` per
+case is logged with the same tri-state semantics described in §5.
+
+---
+
+## 9. Troubleshooting
+
+| Symptom                                                                   | Likely cause                                                                                                                                  |
+| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| Skill validates but A/B delta is ~0%                                      | Assertions aren't discriminating — a baseline agent passes them too. Rewrite assertions to require knowledge the skill uniquely provides.     |
+| All evals graded `errored`                                                | Agent endpoint unreachable, auth failed, or model timed out. Check the agent's settings page first; the skill is unknowable until then.       |
+| `Cannot auto-apply: original instructions snapshot does not match …`      | The `SKILL.md` was edited between proposal generation and apply. Open `iteration-N/improvement-proposal.json`, review, and merge by hand.     |
+| `description does not appear to state when the skill should be used`     | Add a "Use when …" trigger phrase. The matcher needs it (§2).                                                                                 |
+| `allowed-tools` parsed wrong (e.g. `["Read,Write"]`)                       | Pre-PR-206 behaviour — upgrade. The current parser accepts both YAML lists and whitespace-separated strings.                                  |
+
+---
+
+## See also
+
+- [AgentSkills open standard](https://agentskills.io/)
+- [Claude Code skills documentation](https://docs.claude.com/en/docs/claude-code/skills)
+- [`docs/SDK.md`](./SDK.md) — code-based test SDK (different surface; complementary)
+- Source: [`services/skills/`](../services/skills/), [`server/routes/skills.ts`](../server/routes/skills.ts), [`components/skills/SkillsPage.tsx`](../components/skills/SkillsPage.tsx)

package/docs/blogs/2026-02-28-opensearch-agent-health.md

@@ -0,0 +1,200 @@
+---
+layout: post
+title: "OpenSearch Agent Health: Open-Source Observability and Evaluation for AI Agents"
+authors:
+  - goyamegh
+  - thottan
+date: 2026-02-28
+categories:
+  - technical-posts
+  - community
+meta_keywords:
+meta_description:
+---
+
+*How Observability-Driven Development Lets You Ship Agents in Weeks, Not Months*
+
+You've built an Agentic AI application. It's sophisticated, using recursive loops and autonomous tool-calling to navigate data. It passed your local testing with flying colors, so you deploy.
+
+Then the logs hit.
+
+A user asks for an order update. Your agent retrieves the wrong context, decides it needs to "reason" through a product manual, and triggers a sequence of expensive, irrelevant tool calls. After several seconds and a surprising inference cost, it returns a confident but unhelpful response. Your dashboard shows a healthy cluster and a successful 200 OK, but your agent just failed in silence.
+
+### Why do agents fail in silence?
+
+As AI agents move from prototype to production, organizations face a critical challenge that's becoming impossible to ignore. Agents are autonomously making decisions, calling tools, and delivering results, yet the teams building them are often operating without visibility. This creates three critical challenges:
+
+**The Reasoning Gap**: You see the final answer your agent delivers, but you don't see the hallucinated steps it took to get there. Did it call the wrong API? Did it misinterpret the user's intent? Did it retry a failed operation three times before succeeding? Without visibility into the reasoning chain, debugging becomes guesswork.
+
+**The Cost-Latency Spiral**: Agentic workflows are inherently recursive. A single user query can trigger a cascade of expensive and time-consuming sub-tasks. Without real-time operational observability, you don't know which specific tool or sub-agent is burning your budget. Performance issues are only discovered when the bill arrives.
+
+**The Evaluation Paradox**: Without a systematic way to grade agent performance at scale, teams fall back on manual spot-checking. How do you know if your latest prompt change improved accuracy by 5% or silently degraded it by 10%? Without structured evaluation, teams can't confidently deploy to production.
+
+### How will Agent Health help?
+
+The transition from a multi-month deployment cycle to weekly deployments comes down to eliminating the "vibes-based" QA cycle. Every time a prompt is tweaked, or a new tool/skill is added to an agent, testing every possible trajectory manually is not feasible at scale. Agent Health accelerates development by converting this testing process into an automated, AI judge based workflow. By catching regressions early during the development phase (and in CI/CD pipelines) and providing real-time visual traces of exactly where the agent failed/diverged, teams reclaim engineering hours previously lost to staring at logs or waiting for customer incident reports. You debug in minutes, test systematically, and deploy with greater confidence.
+
+![Before and after Agent Health](/assets/media/blog-images/2026-02-28-opensearch-agent-health/before-after-agent-health.png){:class="img-centered"}
+
+### Introducing OpenSearch Agent Health
+
+```
+npx @opensearch-project/agent-health
+# ✓ Server running at http://localhost:4001
+# ✓ Demo data loaded
+```
+
+That's it. No installation required. Run one command and the full Agent Health interface is ready with traces, benchmarks, evaluations, and comparisons.
+
+OpenSearch Agent Health is an open-source observability and evaluation solution for AI agents. Available as a zero-installation NPX tool that delivers three core capabilities in a lightweight (~4 MB) package.
+
+**1. Solving the Reasoning Gap: OpenTelemetry-Native Trace Observability**
+
+Agent Health provides timeline and flow visualizations that show exactly what your agent is doing at every step such as which tools it's calling, the sequence of decisions, the data flowing between components, and where things go wrong. All tracing follows OpenTelemetry standards, so your instrumentation remains portable and works alongside your existing observability tooling.
+
+**2. Solving the Cost-Latency Spiral: Structured Benchmarking**
+
+Agent Health enables systematic A/B comparison of agent configurations across test suites. Test different prompts, models, or configurations side-by-side, and track results over time. By storing all benchmarks and results in OpenSearch, Agent Health gives you a persistent, queryable history of every evaluation run, so you catch regressions before they reach production, not after.
+
+**3. Solving the Evaluation Paradox: Real-Time Agent Evaluation**
+
+Agent Health uses "Golden Path" trajectory comparison, where an LLM judge scores agent actions against expected outcomes. You define what "good" looks like for your agent (the expected steps, tool calls, and outcomes) and Agent Health measures how well your agent performs against these criteria. Use your preferred LLM provider as your judge, giving you flexibility to choose the evaluation model that fits your needs and budget.
+
+### Built for Developer Workflows
+
+Agent Health includes both a web UI (`localhost:4001`) and a headless CLI to support different workflows throughout the development lifecycle.
+
+- **AI Agent Developers** can write test cases, run evaluations locally, and integrate agent evaluation directly into CI/CD pipelines. Benchmarks defined in JSON can be executed from the command line, exporting both human and machine-readable reports for automated quality gating.
+- **ML Scientists and Evaluators** can use the web UI (`localhost:4001`) to design golden datasets, define quality criteria, run batch benchmarks across configurations, perform A/B testing to track accuracy trends over time, export results for reports or further analysis, and even compare their benchmarks against different comparable agents.
+
+### Try it yourself
+
+Ready to see Agent Health in action? Choose your path: explore with pre-loaded sample data, or connect your own agent and start evaluating.
+
+**Quick Start: Explore with Sample Data (No Agent Required)**
+
+The fastest way to experience Agent Health is with the built-in sample data. Run a single command and you'll have real traces, benchmarks, and evaluation results to explore immediately.
+
+```
+npx @opensearch-project/agent-health
+# ✓ Server running at http://localhost:4001
+# ✓ Demo data loaded
+# ✓ Ready to trace, evaluate, and ship
+```
+
+Navigate to Traces to explore pre-loaded agent execution data, go to Benchmarks to run the "Travel Planning Accuracy - Demo" and see LLM judge evaluations in action, and visit Compare to see side-by-side run comparisons.
+
+**Evaluate Your Own Agent**
+
+This walkthrough shows you how to instrument your agent, create benchmarks, and iterate based on evaluation results.
+
+**Step 1: Start Agent Health**
+
+Start the tool and configure your connections:
+
+```
+npx @opensearch-project/agent-health
+# ✓ Server running at http://localhost:4001
+```
+
+Agent Health needs an OpenSearch cluster for evaluation data (test cases, benchmarks, and results) and for observability data (agent traces). You can use any existing hosted OpenSearch cluster, or spin one up locally with the [OpenSearch Observability Stack](https://github.com/opensearch-project/observability-stack?tab=readme-ov-file#-quickstart).
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/opensearch-project/observability-stack/main/install.sh | bash
+```
+
+The local stack launches two components:
+
+- **OpenSearch cluster on port 9200** — Use this as both your evaluation storage and observability storage endpoints in Agent Health Settings.
+- **OTEL Collector on port 4317** — Point your instrumented agent's traces here. Agent Health uses OpenTelemetry for trace collection. Follow the [OpenTelemetry instrumentation guide](https://opentelemetry.io/docs/instrumentation/) for your language to add tracing to your agent.
+
+If you're using an existing cluster, add your cluster endpoint directly as both storage endpoints in Settings.
+
+In Settings, configure your agent endpoint (name, URL, and connector type), and you can set your own LLM Judge via env variables.
+
+**Step 2: Create a Benchmark**
+
+Define test cases in JSON format. Example for a travel planner agent (travel-agent-benchmark.json):
+
+```json
+[
+  {
+    "name": "Check hotel availability",
+    "description": "User wants to find hotels in New York for specific dates",
+    "labels": ["category:Travel", "difficulty:Easy"],
+    "initialPrompt": "Are there any hotels available in Manhattan for next weekend?",
+    "expectedOutcomes": [
+      "Agent should call search_hotels tool with location and date range",
+      "Agent should present available hotels with prices and amenities"
+    ]
+  },
+  {
+    "name": "Complete booking with confirmation",
+    "description": "User wants to book a specific flight and hotel",
+    "labels": ["category:Travel", "difficulty:Medium"],
+    "initialPrompt": "Book the morning flight to New York and the Hilton hotel for 2 nights",
+    "context": [
+      {
+        "description": "Previous search results showing available flights and hotels",
+        "value": "{\"flights\":[{\"id\":\"AA123\",\"time\":\"8:00 AM\",\"price\":\"$350\"}],\"hotels\":[{\"id\":\"hilton-manhattan\",\"name\":\"Hilton Manhattan\",\"price\":\"$200/night\"}]}"
+      }
+    ],
+    "expectedOutcomes": [
+      "Agent should call book_flight tool with the correct flight ID",
+      "Agent should call book_hotel tool with hotel ID and number of nights",
+      "Agent should confirm both bookings with confirmation numbers and total cost"
+    ]
+  }
+]
+```
+
+**Step 3: Run and Analyze**
+
+Choose your workflow:
+
+**Option A: CLI Mode**
+
+```bash
+npx @opensearch-project/agent-health benchmark \
+  -f travel-agent-benchmark.json \
+  -a travel-agent \
+  --export baseline-results.json \
+  -v
+```
+
+This imports the benchmark, creates it, runs it against your agent, and exports results in one command.
+
+**Option B: Via UI**
+
+Navigate to Benchmarks, then Import JSON and select your file. Choose "Run" and configure the agent and Judge Model. Watch real-time progress as each test case executes.
+
+**Step 4: View Results**
+
+Access results through the UI (Benchmarks, then Latest Run) to see overall pass rates, per-test-case results with LLM judge reasoning, and improvement strategies ranked by priority. Navigate to Traces for detailed execution visualization showing timeline views, flow views, and span details. Alternatively, analyze the JSON export:
+
+```bash
+cat baseline-results.json | jq '.runs[0].reports[] | select(.passFailStatus == "failed") | {
+  testCase: .testCaseId,
+  reasoning: .llmJudgeReasoning,
+  strategies: .improvementStrategies
+}'
+```
+
+**Step 5: Iterate and Improve**
+
+Update your agent based on high-priority recommendations, then re-run the benchmark and compare results. Navigate to Compare from Benchmarks and select two runs to visualize improvements across metrics like pass rate, latency, cost, and failed tests.
+
+![Agent Health benchmark comparison view](/assets/media/blog-images/2026-02-28-opensearch-agent-health/benchmark-comparison.png){:class="img-centered"}
+
+### What's Next
+
+This is an experimental release, and we're actively shaping what Agent Health becomes next. As the AI agent ecosystem matures and standards solidify, we'll continue evolving the tool based on real-world usage.
+
+Agent Health is designed as a development-phase companion that helps you build, test, and iterate on your agents before they reach production. As the ecosystem matures, we're exploring how Agent Health's evaluation and observability capabilities can extend into production-scale deployments.
+
+Install it, run your benchmarks, and share what you find. Open an issue on GitHub, join the discussion in the RFC, or share your experience with the community. Your feedback will directly influence how we prioritize features and refine the developer experience.
+
+### Learn More
+
+- **GitHub Repository**: [opensearch-project/agent-health](https://github.com/opensearch-project/agent-health)
+- **RFC**: [Agent Health RFC](https://github.com/opensearch-project/agent-health/issues/42)