@opensearch-project/agent-health 0.3.0 → 0.5.2

package/docs/skills/AGENT_PROFILE.md

@@ -0,0 +1,191 @@
+<!--
+  * Copyright OpenSearch Contributors
+  * SPDX-License-Identifier: Apache-2.0
+-->
+
+# Agent Profiling — improve an agent from its own sessions
+
+**Agent profiling** is to an AI agent what a CPU/JVM profiler is to a running
+program: you attach to a *real session*, sample its execution (OpenTelemetry
+traces), and get back a report of where the agent went wrong and what to fix —
+in the agent's **own codebase**. Instead of profiling for performance hotspots,
+it profiles for *behavioral* problems, scored against an evaluator you choose as
+the rubric.
+
+No synthetic test cases. The session you just ran is the input, and the
+corrections you made while steering it are part of the signal. Because it reads
+**traces**, not chat logs, it works for non-chat agents too.
+
+It reuses the same pluggable `Evaluator` definitions and the same
+`improvementStrategies[]` shape as the batch eval pipeline — it just points them
+at a live session's traces.
+
+---
+
+## Setup (once per project)
+
+```bash
+# 1. Install the agent-profiling hook + skill into Claude Code.
+#    - a PreToolUse hook that records the current session id to
+#      .claude/agent-health/current-session (deterministic — no mtime guessing)
+#    - the /agent-health:profile slash command
+#    - the curated customer skills (agent-health-profile, agent-health-assistant,
+#      instrument-otel)
+npx @opensearch-project/agent-health setup
+
+# 2. Stream Claude Code telemetry into Agent Health's OpenSearch.
+npx @opensearch-project/agent-health setup-telemetry
+```
+
+`setup` is idempotent — re-running won't duplicate the hook, and won't overwrite
+skill/command files unless you pass `--force`. For **Kiro**, drop
+`docs/skills/AGENT_PROFILE.md` under `.kiro/steering/`; for other tools, tell the
+assistant "read AGENT_PROFILE.md and follow it".
+
+### pi (pi.dev)
+
+pi has no native telemetry, so a **zero-dependency, single-file extension**
+([examples/pi-profiling/agent-health-profile.ts](../../examples/pi-profiling/agent-health-profile.ts))
+both *instruments* the live pi session (emitting `invoke_agent` / `chat` /
+`execute_tool` spans stamped with `session.id`, service `pi-agent`) and registers
+the profiling command. It speaks OTLP over `fetch` — nothing to `npm install`.
+Full guide: [docs/PI_PROFILING.md](../PI_PROFILING.md).
+
+```bash
+# 1. Drop the extension in (global = every pi session).
+cp examples/pi-profiling/agent-health-profile.ts ~/.pi/agent/extensions/
+
+# 2. Point telemetry at a trace store BEFORE starting pi. Simplest is
+#    Agent Health in FILE mode (no OpenSearch) — its embedded receiver persists
+#    spans and matches sessions out of the box:
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4001   # /v1/traces is appended
+```
+
+The extension records the session id to `.pi/agent-health/current-session` on
+`session_start`, so `agent-health profile` resolves it with no guessing and
+defaults `--service` to `pi-agent`. When done, run `/agent-health-profile -e
+<evaluator-id>` from inside pi (or `agent-health profile --service pi-agent
+--output json` from a shell). See [examples/pi-profiling/README.md](../../examples/pi-profiling/README.md)
+for OpenSearch-backed routing notes (export to the cluster's own OTLP ingest;
+the embedded receiver drops payloads when a cluster is configured).
+
+---
+
+## Usage
+
+Use your agent normally and steer it as you like. When the session is done:
+
+```bash
+# From inside the coding session (slash command):
+/agent-health:profile -e <evaluator-id>
+
+# …or directly, optionally with upfront feedback to steer the analysis:
+npx @opensearch-project/agent-health profile -e <evaluator-id> --feedback "focus on routing; it ignored the SOP"
+```
+
+If `-e` is omitted it defaults to `system-rca-default`. `--feedback "<text>"` is
+optional upfront human steering — context the traces can't capture; the reasoner
+weights it above the deterministic signals. List evaluators with
+`agent-health list` (or the `/api/storage/evaluators` API).
+
+---
+
+## How it works
+
+```
+your session  ──OTel spans──▶ Agent Health (OpenSearch)
+                                   │
+agent-health profile -e <id>       │  resolves: evaluator rubric (by id)
+                                   ▼  samples:  this session's spans → trajectory
+                          { evaluator, trajectory, signals }   + a deterministic signal scan
+                                   │
+        the coding agent adds: the live chat + the codebase it's sitting in
+                                   ▼
+                 prioritized edits → applied on a branch → you review
+```
+
+`profile` supplies only the half the agent can't see — the **traces** and the
+**rubric**. The agent already has the conversation and the code, so it does the
+reasoning and the edits. (For a headless context the JSON profile can be fed to
+any reasoner.)
+
+### What the profile contains (`--output json`)
+
+```jsonc
+{
+  "session":   { "sessionId", "serviceName", "traceIds", "spanCount", "trajectorySteps", "durationMs", "tokens" },
+  "evaluator": { "id", "name", "systemPrompt", "metrics", "passThreshold" },
+  "signals":   [ { "id", "title", "severity", "count", "evidence" } ],
+  "userFeedback": "<your --feedback text, if given>",
+  "trajectory":[ /* TrajectoryStep[] reconstructed from spans */ ],
+  "instructions": "Using the rubric + feedback, review trajectory + signals + chat + codebase; propose edits…"
+}
+```
+
+It is also written to `.agent-health/data/profiles/<sessionId>/profile.json`.
+
+### Signals scanned (deterministic, pre-LLM)
+
+| id | meaning |
+|---|---|
+| `user_rejection` | you rejected/aborted a tool the agent proposed (Claude Code permission denial — always available, no prompt logging needed) |
+| `user_redirect` | you corrected/redirected the agent mid-session (requires `OTEL_LOG_USER_PROMPTS=1`) |
+| `tool_error_retry` | a tool failed, then was retried — tool-usage / description gap |
+| `repeated_tool_calls` | identical tool+args invoked more than once — loop / distrust |
+| `long_session` | unusually many interactions — confusion or scope creep |
+| `write_before_read` | mutated state before reading — safety / grounding gap |
+
+Signals are evidence handed to the rubric, not the verdict — the evaluator's
+`systemPrompt` decides what to improve. The adapter reads real Claude Code
+telemetry (attribute-based `claude_code.*` spans) and falls back to generic OTel
+GenAI event-based spans for other agents.
+
+---
+
+## Session identification
+
+`profile` resolves the session id in priority order:
+1. `--session <id>` if you pass it,
+2. `.pi/agent-health/current-session` (written by the pi `agent-health-profile`
+   extension on `session_start` — exact),
+3. `.claude/agent-health/current-session` (written by the Claude setup hook — exact),
+4. newest Claude Code transcript for this cwd (heuristic fallback).
+
+Every Claude Code span carries `session.id`, and the pi extension stamps the
+same `session.id` on the spans it emits, so once resolved the fetch is exact
+(`fetchTracesBySessionId`). When the session is resolved from the pi marker,
+`--service` defaults to `pi-agent`.
+
+---
+
+## The loop
+
+```
+profile → read findings → fix on a branch → merge → next session → re-profile
+```
+
+Run it after any session, or on a schedule (cron) over production sessions to
+get a continuous improvement signal at production distribution.
+
+---
+
+## Guardrails
+
+- `profile` is read-only — it never edits code; it produces the profile + a plan.
+- Apply edits on a **branch**, never the working tree, and keep them minimal and
+  generalizable.
+- The rubric is the customer's evaluator, so improvements track *their*
+  definition of good.
+
+---
+
+## Prerequisites
+
+`profile` reads traces from, and the judge scores against, the same backend the
+rest of Agent Health uses:
+- **OpenSearch** for traces (`OPENSEARCH_LOGS_*` or `agent-health.config.json`
+  → `observability`) — the *same* cluster your telemetry is exported to.
+- **AWS Bedrock** credentials for the LLM judge (`AWS_PROFILE` / `AWS_REGION`).
+
+Verify with `agent-health doctor`. Litmus test: if your session shows up in the
+**Traces** tab, `profile` will find it.

package/docs/skills/add-connector/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: add-connector
+description: Use when adding, modifying, or debugging agent connectors. Provides connector architecture, built-in types, and custom connector template.
+---
+
+## Connector System (`services/connectors/`)
+
+Pluggable abstraction for agent communication protocols.
+
+### Key Files
+- `types.ts`: `AgentConnector`, `ConnectorRequest`, `ConnectorResponse` interfaces
+- `registry.ts`: Singleton `connectorRegistry` — `get()`, `getForAgent()`
+- `base/BaseConnector.ts`: Abstract base with auth header building
+- `index.ts`: Browser-safe exports (no Node.js deps)
+- `server.ts`: All connectors including Node.js-only (subprocess, claude-code)
+
+### Built-in Connectors
+
+| Connector | Protocol | Use Case |
+|-----------|----------|----------|
+| `agui-streaming` | AG-UI SSE | ML-Commons agents (default) |
+| `rest` | HTTP POST | Non-streaming REST APIs |
+| `openai-compatible` | OpenAI Chat Completions | LiteLLM, Ollama, vLLM |
+| `langgraph` | LangGraph REST `/invoke` | Non-AG-UI LangGraph instances |
+| `strands` | Bedrock Agent Runtime | Amazon Strands agents (server-only) |
+| `subprocess` | CLI stdin/stdout | Generic command-line tools |
+| `claude-code` | Claude CLI | Claude Code agent (extends `subprocess`) |
+| `kiro` | Kiro CLI | Kiro coding agent (extends `subprocess`; parses `[tool]` stderr markers) |
+| `pi` | Pi CLI | Pi coding agent (extends `subprocess`) |
+| `mock` | In-memory | Demo and testing |
+
+### Usage
+```typescript
+import { connectorRegistry } from '@/services/connectors';
+const connector = connectorRegistry.getForAgent(agentConfig);
+const response = await connector.execute(endpoint, request, auth, onProgress);
+```
+
+### Creating a Custom Connector
+```typescript
+import { BaseConnector } from '@/services/connectors';
+
+class CustomConnector extends BaseConnector {
+  readonly type = 'custom' as const;
+  readonly name = 'My Custom Agent';
+  readonly supportsStreaming = true;
+
+  async execute(endpoint, request, auth, onProgress) {
+    // Your protocol implementation
+  }
+}
+
+connectorRegistry.register(new CustomConnector());
+```
+
+### Trace correlation & subprocess hooks
+
+- **Trace context propagation** is handled by `BaseConnector` — set
+  `connectorConfig.traceContext` so the agent's spans join the eval `test_case`
+  trace tree:
+  - `propagateEnv: true` → inject `TRACEPARENT` env into subprocess agents (`buildTraceparentEnv()`).
+  - `propagateHeader: true` → inject a `traceparent` HTTP header into HTTP/SSE agents (`injectTraceparentHeaders()`).
+  - `serviceName: '<otel-service-name>'` → service-name + time-window fallback. Defaults: `claude-code-agent`, `kiro-agent`, `pi-agent`, `observio-sample-agent`. See the "Trace correlation conventions" section in `AGENTS.md`.
+- **Subprocess connectors** (`SubprocessConnector` subclasses) can override
+  `parseStderrChunk(chunk)` to turn stderr markers into trajectory steps (how
+  `kiro` surfaces `[tool] Running:` / `[tool] status:` as `action` +
+  `tool_result` steps). The base class persists `stderr` to `rawOutput` and
+  honors per-request `connectorConfig` overrides (`args` / `inputMode` / `timeout`).

package/docs/skills/agent-health-profile/SKILL.md

@@ -0,0 +1,40 @@
+---
+name: agent-health-profile
+description: Use when the user wants to profile or improve the agent/codebase based on the current session. Profiles this session's traces against a chosen evaluator and proposes concrete edits.
+---
+
+# Agent Health — Profile this session
+
+When the user asks to profile/improve the agent (or runs `/agent-health:profile`):
+
+1. Determine the evaluator id (the rubric). If the user gave one (e.g. `-e my-eval`),
+   use it; otherwise default to `system-rca-default`. You can list options with
+   `agent-health list` if needed.
+
+1b. **Ask the user for any upfront feedback** on the session before profiling
+   (e.g. "anything you want me to focus on — a wrong decision, a skipped step?").
+   If they give some, pass it via `--feedback "..."`; it becomes the primary lens
+   over the deterministic signals. Skip if they have nothing to add.
+
+2. Run:
+   ```bash
+   __AH_CLI__ profile -e <evaluator-id> [--feedback "<their feedback>"] --output json
+   ```
+   It auto-detects the current session id (from `.claude/agent-health/current-session`,
+   written by the setup hook) and prints a JSON profile: the evaluator rubric,
+   the session trajectory (reconstructed from traces), deterministic signals, and
+   your `userFeedback`.
+
+3. Using `evaluator.systemPrompt` as the rubric, review together:
+   - the `userFeedback` (weight it highest, if present),
+   - the `trajectory` and `signals` from the JSON profile,
+   - THIS conversation (including any corrections the user made while steering you),
+   - the codebase in the current working directory.
+
+4. Produce a prioritized list of concrete edits. For each: the file, the change,
+   why (tie it to a signal or a rubric criterion), a priority, and **cite the
+   evidence** — the signal that triggered it and the session's `traceIds`
+   (open them in the Traces tab to verify).
+
+5. Apply the edits on a new branch (never the working tree directly), then
+   summarize the diff for the user to review.

package/docs/skills/config-auth/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: config-auth
+description: Use when working on config loading, authentication, AWS profiles, or data source resolution. Explains the two config systems (TS vs JSON), auth priority, and multi-profile setup.
+---
+
+## Config & Auth Architecture
+
+Agent Health uses **two separate config systems** for different purposes. Understanding when each applies prevents auth confusion when users have multiple AWS profiles.
+
+### Two Config Systems
+
+| System | File | Format | Purpose | Managed By |
+|--------|------|--------|---------|------------|
+| **Agent/Model Config** | `agent-health.config.ts` | TS/JS/MJS | Agents, models, connectors, hooks, judge | User edits manually |
+| **Data Source Config** | `agent-health.config.json` | JSON only | OpenSearch credentials (storage + observability) | CLI `setup-telemetry` / Settings UI |
+
+**Priority for TS/JS config loading** (`lib/config/loader.ts`):
+1. `agent-health.config.ts` (highest)
+2. `agent-health.config.js`
+3. `agent-health.config.mjs`
+4. Built-in defaults — **only merged when `extends: false` is NOT set**; setting `extends: false` in your config disables default merging entirely
+
+**Why two systems?**
+- TS config can contain executable code (hooks, dynamic values, custom connectors)
+- JSON config stores credentials that the server persists and the Settings UI manages
+- **Browser never sees JSON credentials** — the server only exposes metadata (endpoint, authType, hasPassword) to the browser; all authenticated requests are proxied server-side
+
+---
+
+### Auth Resolution Per Service
+
+#### OpenSearch (Storage & Observability)
+
+**Resolution order** (`server/middleware/dataSourceConfig.ts`):
+1. **`agent-health.config.json`** (file) — highest priority
+2. **Environment variables** (`OPENSEARCH_STORAGE_*` / `OPENSEARCH_LOGS_*`) — fallback
+3. **null** — features degrade gracefully
+
+**JSON config structure:**
+```json
+{
+  "storage": {
+    "endpoint": "https://search-....es.amazonaws.com",
+    "authType": "sigv4",
+    "awsProfile": "default",
+    "awsRegion": "us-west-2",
+    "awsService": "es"
+  },
+  "observability": {
+    "endpoint": "https://search-....es.amazonaws.com",
+    "authType": "sigv4",
+    "awsProfile": "default",
+    "awsRegion": "us-west-2",
+    "awsService": "es",
+    "tracesIndex": "otel-v1-apm-span-*"
+  }
+}
+```
+
+**Observability config** is where Agent Health reads OTel traces from. This is configured by:
+1. `npx @opensearch-project/agent-health setup-telemetry` (writes to JSON automatically)
+2. Manual edit of `agent-health.config.json` (add `observability` block with endpoint + auth)
+
+The `tracesIndex` field defaults to `otel-v1-apm-span-*` if omitted. See `/instrument-otel` skill for the full instrumentation flow.
+
+**Auth types** (`server/services/opensearchClientFactory.ts`):
+
+| authType | Fields needed | How it works |
+|----------|--------------|--------------|
+| `none` | endpoint only | No auth headers |
+| `basic` | endpoint, username, password | HTTP Basic Auth |
+| `sigv4` | endpoint, awsRegion, awsService, awsProfile (optional) | AWS SigV4 credential chain |
+
+**SigV4 credential resolution:**
+```typescript
+// If awsProfile specified → use that profile
+// If omitted → use default AWS credential chain
+const provider = fromNodeProviderChain({
+  ...(config.awsProfile && { profile: config.awsProfile })
+});
+```
+
+#### Bedrock Judge (LLM Evaluation)
+
+**Resolution** (`server/services/bedrockService.ts`):
+- Uses AWS SDK **default credential chain** — picks up whatever `AWS_PROFILE` is set in the server's environment
+- Region: `process.env.AWS_REGION || 'us-west-2'`
+- Model: `process.env.BEDROCK_MODEL_ID || 'us.anthropic.claude-sonnet-4-5-20250929-v1:0'`
+- No explicit `awsProfile` field — relies on environment
+
+#### Claude Code Agent (Subprocess)
+
+**Resolution** (`agent-health.config.ts` → agent `connectorConfig.env`):
+```typescript
+{
+  key: "claude-code",
+  connectorConfig: {
+    env: {
+      CLAUDE_CODE_USE_BEDROCK: "1",
+      AWS_PROFILE: "Bedrock",     // ← explicit profile for Claude subprocess
+      AWS_REGION: "us-west-2",
+    },
+  },
+}
+```
+These env vars are set **inside the spawned Claude Code process** — isolated from the server's own credentials.
+
+---
+
+### Multi-Profile Setup Pattern
+
+A user with separate profiles for separate concerns:
+
+| Purpose | AWS Profile | Account | Used By |
+|---------|-------------|---------|---------|
+| OpenSearch reads | `default` | 651304888251 | Server → `agent-health.config.json` (awsProfile: "default") |
+| Bedrock judge | `Bedrock` | 070785328308 | Server env: `AWS_PROFILE=Bedrock` |
+| Claude Code agent | `Bedrock` | 070785328308 | TS config: `connectorConfig.env.AWS_PROFILE` |
+| Telemetry sending | (none needed) | — | API Gateway is public (no SigV4 on client) |
+
+**Problem:** The server process can only have ONE `AWS_PROFILE` env var. If Bedrock judge needs profile "Bedrock" but OpenSearch needs profile "default", you must:
+
+1. Set `AWS_PROFILE=Bedrock` for the server (judge uses default chain)
+2. Set `"awsProfile": "default"` in `agent-health.config.json` for OpenSearch (explicit override via `fromNodeProviderChain({ profile })`)
+
+This works because OpenSearch SigV4 uses **explicit profile from config**, while Bedrock uses **ambient credentials from environment**.
+
+---
+
+### Key Files
+
+| File | Purpose |
+|------|---------|
+| `lib/config/loader.ts` | Load + merge TS/JS config with defaults |
+| `lib/config/types.ts` | `AgentHealthConfig`, `UserAgentConfig`, `UserModelConfig` |
+| `server/middleware/dataSourceConfig.ts` | Resolve OpenSearch config (file → env → null) |
+| `server/services/configService.ts` | Read/write `agent-health.config.json` |
+| `server/services/opensearchClientFactory.ts` | Build OpenSearch client with correct auth |
+| `server/services/bedrockService.ts` | Bedrock client init (uses ambient AWS creds) |
+| `server/config/index.ts` | Server environment variable defaults |
+| `agent-health.config.example.ts` | Reference TS config |
+
+---
+
+### Environment Variables Reference
+
+**OpenSearch Storage (fallback when not in JSON):**
+```bash
+OPENSEARCH_STORAGE_ENDPOINT=https://...
+OPENSEARCH_STORAGE_AUTH_TYPE=sigv4|basic|none
+OPENSEARCH_STORAGE_USERNAME=admin
+OPENSEARCH_STORAGE_PASSWORD=secret
+OPENSEARCH_STORAGE_AWS_REGION=us-west-2
+OPENSEARCH_STORAGE_AWS_PROFILE=default
+OPENSEARCH_STORAGE_AWS_SERVICE=es|aoss
+```
+
+**OpenSearch Observability (fallback when not in JSON):**
+```bash
+OPENSEARCH_LOGS_ENDPOINT=https://...
+OPENSEARCH_LOGS_AUTH_TYPE=sigv4|basic|none
+OPENSEARCH_LOGS_AWS_REGION=us-west-2
+OPENSEARCH_LOGS_AWS_PROFILE=default
+OPENSEARCH_LOGS_AWS_SERVICE=es|aoss
+```
+
+**Bedrock Judge:**
+```bash
+AWS_PROFILE=Bedrock          # Profile with Bedrock access
+AWS_REGION=us-west-2         # Bedrock region
+BEDROCK_MODEL_ID=us.anthropic.claude-sonnet-4-5-20250929-v1:0
+```
+
+**Telemetry (Client-side, in shell RC):**
+```bash
+CLAUDE_CODE_ENABLE_TELEMETRY=1
+OTEL_EXPORTER_OTLP_ENDPOINT=https://<api-gw>.execute-api.<region>.amazonaws.com
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+OTEL_TRACES_EXPORTER=otlp
+OTEL_LOGS_EXPORTER=otlp
+OTEL_METRICS_EXPORTER=otlp
+```
+
+---
+
+### Design Principles
+
+1. **JSON for credentials, TS for behavior** — never put secrets in TS config (it's committed to git)
+2. **Explicit profile wins** — `awsProfile` in JSON config overrides ambient `AWS_PROFILE`
+3. **Each service resolves independently** — OpenSearch can use profile A while Bedrock uses profile B
+4. **Degrade gracefully** — missing config means features are unavailable, not errors
+5. **Browser never sees secrets** — the server proxies all authenticated requests; the browser only receives safe metadata (endpoint URL, authType, hasPassword boolean), never raw credentials
+6. **`setup-telemetry` writes JSON** — it's the only automated way to populate credentials
+7. **`extends: false` opts out of defaults** — without this flag, built-in defaults are always merged into your config; with it, you take full control of every value

package/docs/skills/config-auth/evals/evals.json

@@ -0,0 +1,35 @@
+{
+  "skill_name": "config-auth",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Where does Agent Health store OpenSearch credentials and what file format does it use?",
+      "expected_output": "A clear explanation that OpenSearch credentials are stored in agent-health.config.json (JSON format, not TypeScript), managed by the CLI setup-telemetry command or Settings UI, and distinct from the agent-health.config.ts file which handles agents/models/hooks.",
+      "assertions": [
+        "Response identifies agent-health.config.json as the file storing OpenSearch credentials, not agent-health.config.ts",
+        "Response explains that agent-health.config.json is managed by the CLI `setup-telemetry` command or the Settings UI, not manually edited like the TS config",
+        "Response mentions that the browser never sees the JSON credentials — only metadata like endpoint, authType, and hasPassword"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "I have two AWS profiles: 'default' for my OpenSearch cluster and 'Bedrock' for the LLM judge. How do I configure Agent Health so each service uses the right profile? The server can only have one AWS_PROFILE set.",
+      "expected_output": "A concrete multi-profile setup: set AWS_PROFILE=Bedrock in the server environment (so the Bedrock judge uses its default credential chain), and set awsProfile: 'default' inside the storage block of agent-health.config.json (so OpenSearch SigV4 explicitly overrides the ambient profile). Explains why this works — OpenSearch uses fromNodeProviderChain with an explicit profile override while Bedrock uses ambient credentials.",
+      "assertions": [
+        "Response instructs setting AWS_PROFILE=Bedrock as the server environment variable so the Bedrock judge picks it up via default credential chain",
+        "Response instructs setting `awsProfile: 'default'` inside agent-health.config.json under the storage block to override the ambient AWS_PROFILE for OpenSearch SigV4 auth",
+        "Response explains that OpenSearch uses fromNodeProviderChain with an explicit profile parameter (from the JSON config) while Bedrock relies on the ambient environment — this is why the two services can use different profiles simultaneously"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "My agent config TypeScript file isn't being picked up. I have agent-health.config.mjs but Agent Health seems to be using defaults. What's the config loading priority and which file takes precedence?",
+      "expected_output": "Explanation of the four-level priority order for TS/JS config loading from lib/config/loader.ts: agent-health.config.ts (highest) → agent-health.config.js → agent-health.config.mjs → built-in defaults. Since .mjs is third priority, if a .ts or .js file exists it would shadow the .mjs file. Also notes that built-in defaults are only applied if `extends: false` is NOT set.",
+      "assertions": [
+        "Response states the exact priority order: agent-health.config.ts > agent-health.config.js > agent-health.config.mjs > built-in defaults, identifying .mjs as third priority",
+        "Response identifies lib/config/loader.ts as the file responsible for this loading priority logic",
+        "Response mentions that built-in defaults are only merged in when `extends: false` is NOT present in the config, meaning setting extends: false disables default merging"
+      ]
+    }
+  ]
+}

package/docs/skills/create-pr/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: create-pr
+description: Prepare and create a pull request against the upstream repository with all OpenSearch project compliance checks.
+disable-model-invocation: true
+argument-hint: "[branch-name]"
+---
+
+## PR Workflow for opensearch-project/agent-health
+
+**IMPORTANT:** PRs are created from your personal fork, NOT from branches on `opensearch-project/agent-health` directly. Push to your fork remote (e.g., `fork` or your GitHub username), then open a PR targeting `opensearch-project/agent-health:main`.
+
+**MANDATORY:** Every commit MUST have DCO (Developer Certificate of Origin) signoff. Always use `git commit -s`. CI will reject commits without signoff. Never use `--no-verify` to bypass this.
+
+### 1. Fetch latest and create branch
+```bash
+git fetch origin main
+git checkout -b <branch-name> origin/main
+```
+
+### 2. Cherry-pick commits (if from a dev branch)
+```bash
+git cherry-pick <oldest-commit>^..<newest-commit>
+```
+
+### 3. Verify DCO sign-off on ALL commits
+```bash
+git log --format="%h %s" origin/main..HEAD
+git log origin/main..HEAD | grep -c "Signed-off-by"
+# If missing: git rebase origin/main --signoff
+```
+
+### 4. Add changelog entry (REQUIRED — CI will fail without this)
+Update `CHANGELOG.md` under `## [Unreleased]` with appropriate category (`Added`, `Changed`, `Fixed`, `Removed`, `Security`). Include PR link:
+```markdown
+### Added
+- Description of change ([#PR](https://github.com/opensearch-project/agent-health/pull/PR))
+```
+
+### 5. Run pre-PR checks (ALL must pass)
+```bash
+npm run build:all && npm run test:all
+npm audit --audit-level=high
+```
+
+### 6. Pre-PR Checklist
+- [ ] All commits have DCO signoff
+- [ ] `CHANGELOG.md` updated under `## [Unreleased]` with PR link
+- [ ] `npm run build:all` succeeds
+- [ ] `npm run test:all` passes (unit + integration + e2e)
+- [ ] `npm audit --audit-level=high` reports no vulnerabilities
+- [ ] New source files have SPDX license headers
+- [ ] No secrets committed (`.env`, credentials, tokens)
+
+### 7. Push to your fork and create PR
+```bash
+# Push to your personal fork (NOT opensearch-project directly)
+git push -u fork <branch-name>
+# Create PR targeting the upstream repo
+gh pr create --repo opensearch-project/agent-health --base main
+```
+
+### License Headers (for new files)
+```typescript
+// .ts, .tsx, .js, .jsx, .cjs, .mjs, .css:
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+```
+
+### DCO Signoff
+All commits need `-s` flag: `git commit -s -m "message"`
+Signoff: `Signed-off-by: Megha Goyal <goyamegh@amazon.com>`