@opensearch-project/agent-health 0.4.0 → 0.6.0

package/docs/SKILLS.md

@@ -0,0 +1,264 @@
+<!--
+  * Copyright OpenSearch Contributors
+  * SPDX-License-Identifier: Apache-2.0
+-->
+
+# Skills Evaluator — Authoring Guide
+
+The Skills Evaluator runs an A/B benchmark on an [AgentSkills](https://agentskills.io/) /
+[Claude Code skill](https://docs.claude.com/en/docs/claude-code/skills): one set of
+agent runs **with** the skill injected, one set **without**, then grades the same
+assertions on both trajectories and reports the delta in pass-rate, latency, and
+tokens. Optionally, an LLM proposes an improved `SKILL.md` body from the failure
+evidence and (with `auto`) applies it.
+
+This guide tells you **how to write a skill that the evaluator can score
+fairly** — and what every warning / status the UI shows actually means.
+
+```
+┌──────────┐  ┌──────────────┐   ┌──────────────┐   ┌────────────┐
+│ SKILL.md │→ │ parseSkill   │→  │ runSkillEval │→  │ benchmark  │
+└──────────┘  │  validate +  │   │  with vs     │   │  + diff +  │
+              │  warn        │   │  without     │   │  improve   │
+              └──────────────┘   └──────────────┘   └────────────┘
+```
+
+---
+
+## 1. Anatomy of a skill folder
+
+```
+my-skill/
+├── SKILL.md            # required — frontmatter + body
+└── evals/
+    └── evals.json      # optional — auto-generated if absent
+```
+
+`SKILL.md` is a markdown file with a YAML frontmatter block delimited by `---`
+on lines by themselves:
+
+```markdown
+---
+name: rca-log-analysis
+description: Use when the user asks why a service is failing or wants to find
+  the root cause from log lines. Triggers on terms like "why is X failing",
+  "RCA", "root cause", "error spike".
+allowed-tools:
+  - Read
+  - Bash(git:*)
+license: Apache-2.0
+---
+
+# Root-cause analysis from logs
+
+When the user asks why a service is failing:
+
+1. Pull the last 5 minutes of logs from the suspect service.
+2. Group by error code; the dominant code is your starting hypothesis.
+3. ...
+```
+
+### Frontmatter rules (enforced by `parseSkill`)
+
+| Field           | Required | Constraint                                                              |
+| --------------- | -------- | ----------------------------------------------------------------------- |
+| `name`          | yes      | lowercase kebab-case, ≤ 64 chars                                        |
+| `description`   | yes      | ≤ 1024 chars; **should describe trigger conditions** (see §2)           |
+| `allowed-tools` | no       | YAML list **or** whitespace-separated string (`Bash(git:*)` patterns ok) |
+| `license`       | no       | free-form string                                                        |
+| `compatibility` | no       | free-form string                                                        |
+| `metadata`      | no       | arbitrary YAML mapping                                                  |
+
+The closing `---` must be on its own line. CRLF line endings are normalised
+before matching.
+
+---
+
+## 2. Why your `description` is the most important field
+
+In Claude Code's skill loader the `description` is what the matcher
+**actually reads** when deciding whether to inject the skill into a given
+turn. The body is only loaded after the description has matched. A
+description that reads like a tagline ("A helper for log analysis") will
+silently fail to fire.
+
+The parser surfaces two warnings for this:
+
+| Warning text                                                                                                                 | What to do                                                                                                                                       |
+| ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `description is only N characters … aim for ≥ 30 chars and explicitly state trigger conditions` | The description is too short to discriminate from peers. Expand it.                                                                              |
+| `description does not appear to state when the skill should be used`                                                         | Add a phrase like **`Use when …`**, **`For requests about …`**, **`Trigger on …`**, **`Invoke when the user asks …`**. The matcher looks for these. |
+
+Good descriptions:
+
+> ✅ "Use when the user asks why a service is failing or wants RCA from logs.
+> Triggers on phrases like 'why is X erroring', 'root cause', 'error spike'."
+
+> ✅ "For requests about diffing two trajectories — invoke when the user
+> mentions 'compare runs', 'why did the agent behave differently', or pastes
+> two run IDs."
+
+Bad descriptions (parser will warn):
+
+> ❌ "Helper for logs."
+> ❌ "Skill for trace analysis."
+> ❌ "Improves debugging."
+
+---
+
+## 3. Authoring `evals/evals.json`
+
+Each eval case is a triple **prompt → expected_output → assertions**. The
+evaluator runs the prompt twice (with and without the skill), then asks the
+LLM judge whether **each** assertion holds against the resulting trajectory.
+
+```json
+{
+  "skill_name": "rca-log-analysis",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "The checkout service started returning 503s 10 minutes ago. Why?",
+      "expected_output": "Identifies a downstream dependency or resource exhaustion as the cause.",
+      "assertions": [
+        "The agent searches logs for the affected service before guessing",
+        "The agent groups errors by error code or status",
+        "The agent's final answer names a specific suspect (service / resource / config)"
+      ]
+    }
+  ]
+}
+```
+
+Tips:
+
+- **Make assertions discriminating.** A good assertion is one a *baseline*
+  agent (without the skill) would plausibly fail — that's the only way the
+  A/B delta surfaces real value. Trivial assertions like "the response is in
+  English" don't move the needle.
+- **2–3 assertions per eval is plenty.** Each assertion costs one judge call.
+- **`expected_output` is free-form** — it's used as judge context, not as a
+  string-match target.
+- **Auto-generation** runs only if `evals/evals.json` is missing. If it
+  fails to parse a JSON eval set from the model response, the evaluator
+  **throws with the exact JSON shape** rather than substituting a generic
+  placeholder. Hand-author the file in that case.
+
+---
+
+## 4. Reading the validation panel
+
+After you select a skill in the UI, the validation panel shows:
+
+| Symbol | Meaning                                                                                              |
+| ------ | ---------------------------------------------------------------------------------------------------- |
+| ✅     | `valid: true` — frontmatter parsed, required fields present                                          |
+| ❌     | `valid: false` — at least one hard error; the **Run Evaluation** button stays disabled               |
+| ⚠      | Warning, listed below the description. Skill still runs, but you should fix before publishing.       |
+
+Common warnings (and what they mean):
+
+| Warning                                                              | Cause / Fix                                                                                              |
+| -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| `description does not appear to state when the skill should be used` | Add a "Use when …" trigger phrase (§2)                                                                   |
+| `Instructions are ~N tokens (recommended <5000)`                     | Skill body too long. Split into multiple skills or move examples to bundled reference files.            |
+| `evals/evals.json present but invalid: …`                            | The file exists but parsing failed. The reason follows the colon (e.g. `invalid JSON`, `missing skill_name`). |
+| `No evals/evals.json found — skill cannot be evaluated …`            | Auto-generation will run on Evaluate. Provide your own to control what's tested.                         |
+
+---
+
+## 5. Reading the run progress
+
+While a run is in flight, the progress bar shows three counters:
+
+```
+N passed   N failed   ⚠ N errored
+```
+
+The distinction is intentional and matters:
+
+| Status     | Meaning                                                                                                                                                                 | Improver behaviour                                                                                                       |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `passed`   | Agent ran cleanly **and** every assertion graded true.                                                                                                                  | Treated as a positive example.                                                                                          |
+| `failed`   | Agent ran cleanly but **at least one assertion graded false**.                                                                                                          | This is the signal the improver learns from — the skill is the suspect.                                                 |
+| `errored`  | The **agent itself** crashed / endpoint unreachable / timed out — assertions were skipped because grading them would (a) waste a judge call and (b) wrongly blame the skill. | **Excluded** from improvement-proposal evidence. Fix the agent (endpoint, auth, network) before drawing skill conclusions. |
+
+If you see all `errored`, the skill is **unknowable** — the evaluator is
+telling you to debug the agent, not the skill.
+
+---
+
+## 6. Auto-applying improvement proposals (`auto: true`)
+
+When the LLM proposes an improved `SKILL.md` body, you can ship it
+automatically with the **Apply & Re-run** button (UI) or `--auto` (CLI).
+
+The server applies the change **non-destructively**:
+
+1. Snapshots the current `SKILL.md` to `SKILL.md.bak` next to it.
+2. Refuses to apply (with an explanatory error pointing to
+   `iteration-N/improvement-proposal.json`) if the original instructions
+   snapshot doesn't appear verbatim in the file — this prevents the silent
+   no-op case where `String.replace` matched nothing but the API still
+   claimed `applied: true`.
+
+To roll back: `mv SKILL.md.bak SKILL.md`.
+
+---
+
+## 7. Where skills are discovered
+
+The discover endpoint scans both **user scope** and **project scope**, in
+this order:
+
+| Path                       | Source label             | Notes                                  |
+| -------------------------- | ------------------------ | -------------------------------------- |
+| `~/.claude/skills/`        | `Claude Code (user)`     | Per Claude Code spec — user-global.    |
+| `<cwd>/.claude/skills/`    | `Claude Code`            | Project scope.                         |
+| `<cwd>/.kiro/skills/`      | `Kiro`                   |                                        |
+| `<cwd>/.kiro/steering/`    | `Kiro`                   |                                        |
+| `<cwd>/.codex/`            | `Codex`                  |                                        |
+| `<cwd>/.cursor/rules/`     | `Cursor`                 |                                        |
+| `<cwd>/.github/copilot/`   | `Copilot`                |                                        |
+| `<cwd>/.continue/skills/`  | `Continue`               |                                        |
+| `<cwd>/skills/`            | `Project`                |                                        |
+
+Duplicate skill folders (same absolute path) appear once. Home-relative
+paths render as `~/…` in the dropdown. You can also paste a path manually
+or upload a single `SKILL.md` from the UI.
+
+---
+
+## 8. CLI
+
+```bash
+# Run on a specific skill folder
+agent-health skill ./my-skill --agent claude-code --model claude-sonnet
+
+# Auto-apply the proposed improvement and re-run
+agent-health skill ./my-skill --auto
+```
+
+The CLI emits the same SSE event stream the UI consumes; `evalStatus` per
+case is logged with the same tri-state semantics described in §5.
+
+---
+
+## 9. Troubleshooting
+
+| Symptom                                                                   | Likely cause                                                                                                                                  |
+| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| Skill validates but A/B delta is ~0%                                      | Assertions aren't discriminating — a baseline agent passes them too. Rewrite assertions to require knowledge the skill uniquely provides.     |
+| All evals graded `errored`                                                | Agent endpoint unreachable, auth failed, or model timed out. Check the agent's settings page first; the skill is unknowable until then.       |
+| `Cannot auto-apply: original instructions snapshot does not match …`      | The `SKILL.md` was edited between proposal generation and apply. Open `iteration-N/improvement-proposal.json`, review, and merge by hand.     |
+| `description does not appear to state when the skill should be used`     | Add a "Use when …" trigger phrase. The matcher needs it (§2).                                                                                 |
+| `allowed-tools` parsed wrong (e.g. `["Read,Write"]`)                       | Pre-PR-206 behaviour — upgrade. The current parser accepts both YAML lists and whitespace-separated strings.                                  |
+
+---
+
+## See also
+
+- [AgentSkills open standard](https://agentskills.io/)
+- [Claude Code skills documentation](https://docs.claude.com/en/docs/claude-code/skills)
+- [`docs/SDK.md`](./SDK.md) — code-based test SDK (different surface; complementary)
+- Source: [`services/skills/`](../services/skills/), [`server/routes/skills.ts`](../server/routes/skills.ts), [`components/skills/SkillsPage.tsx`](../components/skills/SkillsPage.tsx)

package/docs/STORAGE_INDEX_FIELD_LIMITS.md

@@ -0,0 +1,216 @@
+# Storage: OpenSearch index field-limit growth (`evals_runs`)
+
+## Incident
+
+Owner-hit while running code-QA benchmarks: report/run persistence on the
+shared cluster's `evals_runs` index failed with
+
+```
+illegal_argument_exception: Limit of total fields [5000] has been exceeded
+```
+
+Run **execution** succeeded — only the **write** errored out, i.e. data loss
+(the completed report was never persisted).
+
+This is the same *class* of bug PR #418 fixed for `evals_experiments`
+(`EvaluationRun.results` / `testCaseSnapshots`): OpenSearch's default dynamic
+mapping mints a new mapped field for every previously-unseen key under a
+free-form object, and the field-count budget (`index.mapping.total_fields.limit`)
+is **shared across every document in the index**, not per-document. #418 did
+not cover `evals_runs` (the report/`TestCaseRun` index used by the code-SDK
+path) — this fix does.
+
+## Root cause: two unprotected growth vectors in `evals_runs`
+
+Both are driven by the same source: `EvaluationMetrics`
+(`types/index.ts`) is an open index signature (`[key: string]: number |
+undefined`) by design — custom/system evaluators declare arbitrary metric
+dimension names via `evaluator.scoringConfig.metrics`
+(`server/services/judgeResponseParser.ts`'s `extractMetrics()`,
+`services/storage/asyncRunStorage.ts`'s `storedMetricsToApp()` /
+`toStorageFormat()` — see the comments in both, which explicitly call out
+"preserve every metric the judge emitted, not just the four legacy keys").
+Every *distinct* custom metric name, across every run/matcher ever written,
+used to mint a brand-new mapped field, shared index-wide, forever.
+
+| Field (in `evals_runs`) | Shape | Growth vector |
+|---|---|---|
+| `metrics` (report-level) | `Record<string, number>` | One set of dynamic names per run — one custom evaluator with N metric names adds ≤N new fields **the first time it's seen**, but a code-QA benchmark suite iterating on many custom evaluators over time accumulates without bound. |
+| `matcherResults[].judgeMetrics` | `Record<string, number>`, nested inside a `nested`-typed array | Same growth, but **per SDK `judge()` call** — a single code-QA test case with many `expect`/`judge()` claims × many custom judge dimensions multiplies fast. This is the "code-SDK path" referenced in the incident — `matcherResults` is populated exclusively by the code-based test SDK (`docs/SDK.md`), not the legacy UI-driven runner. |
+
+Everything else already flagged in the original bug report — matcher
+`actual`/`expected`, `trajectory`, `logs`, `rawEvents`, `improvementStrategies`,
+`spans` (span attributes) — was **already** `{ type: 'object', enabled: false
+}` in `server/constants/indexMappings.ts` before this change (audited, not
+touched). `llmJudgeResponse` (which itself has an open `extraFields`/
+`parsedMetrics` shape) is **never persisted** to `evals_runs` at all
+(`toStorageFormat()` doesn't include it) — confirmed via `git grep
+llmJudgeResponse services/storage server/adapters`, no hits — so it isn't a
+growth vector for this index either.
+
+## Fix (mirrors #418's pattern)
+
+`server/constants/indexMappings.ts`, `evals_runs` index:
+
+```diff
+   metrics: {
++    dynamic: false,
+     properties: {
+       accuracy: { type: 'float' },
+       faithfulness: { type: 'float' },
+       latency_score: { type: 'float' },
+       trajectory_alignment_score: { type: 'float' },
+     },
+   },
+   ...
+   matcherResults: {
+     type: 'nested',
+     properties: {
+       ...
+       judgeMetrics: {
++        dynamic: false,
+         properties: {
+           accuracy: { type: 'float' },
+           faithfulness: { type: 'float' },
+           latency_score: { type: 'float' },
+           trajectory_alignment_score: { type: 'float' },
+         },
+       },
+     },
+   },
+```
+
+Unlike #418's `results`/`testCaseSnapshots` (`enabled: false`, fully opaque),
+this uses `dynamic: false` **with explicit typed sub-properties** for the
+four legacy metric names — they stay real, typed, queryable fields (nothing
+queries them today — see the audit below — but it's free to keep them typed),
+while every *other* metric/dimension name is stored in `_source` (readable,
+unaffected) but never added to the mapping. `_source` is unaffected either
+way — the choice between `enabled:false` and `dynamic:false` only changes
+what OpenSearch can filter/sort/aggregate on, never what's persisted or
+returned.
+
+## Query audit — nothing queried becomes unsearchable
+
+Every OpenSearch-level query/filter/sort/aggregation against `evals_runs`
+(`server/adapters/opensearch/StorageModule.ts`'s `OpenSearchRunOperations`)
+was enumerated. None touch `metrics.*` or `matcherResults[].judgeMetrics.*`
+beyond the four legacy names, which stay mapped:
+
+| Consumer | Query | Fields used | Affected by this fix? |
+|---|---|---|---|
+| `OpenSearchRunOperations.search()` | `term` filters | `experimentId`, `experimentRunId`, `testCaseId`, `agentId`, `modelId`, `status`, `passFailStatus` | No — untouched, still explicit `keyword` fields |
+| `OpenSearchRunOperations.search()` | `range` filter | `createdAt` | No — untouched, still `date` |
+| `OpenSearchRunOperations.getAll()` / `.search()` | `sort` | `createdAt` | No |
+| `OpenSearchRunOperations.countsByTestCase()` | `terms` agg | `testCaseId` | No |
+| `asyncRunStorage.ts` `SearchQuery.minAccuracy` | **application-level** `Array.filter()`, not an OpenSearch query (`reports.filter(r => r.metrics.accuracy >= ...)`) | `metrics.accuracy` (read from `_source` in JS) | No — reads the value out of `_source`, which is unaffected by `dynamic: false`. If this were ever converted to a server-side `range` query, it would still work: `accuracy` stays an explicitly mapped, queryable field. |
+| UI (`MatcherResultsPanel.tsx`, `JudgeSection.tsx`, `RunDetailsContent.tsx`) | none — reads `matcherResults`/`judgeMetrics` out of the fetched JSON document, never issues its own OpenSearch query | n/a | No |
+| `services/evaluation/index.ts`, `services/benchmarkRunner.ts`, `services/hookOrchestrator.ts` | none — same, in-process consumption of the already-fetched report | n/a | No |
+
+Conclusion: **no consumer anywhere issues an OpenSearch-side query against a
+non-legacy `metrics.*` or `judgeMetrics.*` name.** Both are read back via
+`_source` wherever consumed (search, list, comparison, UI). This mirrors
+exactly the trade-off #418 already made and documented for
+`EvaluationRun.results`.
+
+## Migration story — what to run, exactly
+
+**Nothing runs automatically against the live cluster from this PR.**
+
+### New / fresh indexes
+
+No action needed. `ensureIndexes()` (`server/services/indexInitializer.ts`,
+called on every server boot and on "attach new cluster") creates any missing
+index straight from the updated `INDEX_MAPPINGS` — new deployments and any
+environment that doesn't have `evals_runs` yet get the fix immediately.
+
+### Existing, NOT-YET-poisoned `evals_runs` (most environments)
+
+Also no action needed, but not immediate — `ensureIndexes()` also calls
+`client.indices.putMapping()` on every boot for existing indexes, which is
+how the `dynamic: false` fix reaches an already-existing-but-clean index: it
+succeeds silently and the index is protected from the next write onward.
+
+### The shared cluster's `evals_runs`, if already poisoned
+
+If any code-QA benchmark run already wrote a custom evaluator metric name to
+the shared cluster before this fix ships, `evals_runs.metrics` (and/or
+`matcherResults.judgeMetrics`) already has real, dynamically-inferred
+sub-properties. OpenSearch's `putMapping` **rejects** an `enabled`/`dynamic`
+change on a field that already has sub-properties
+(`mapper_exception: the [dynamic] parameter can't be updated for the object
+mapping [metrics]`) — `ensureIndexes()` catches this, logs a warning, and
+otherwise no-ops (no crash, no data loss, same as #418's documented
+`mapper_exception` handling for `results`). **The index keeps growing** until
+an explicit reindex is run.
+
+**This is not new migration code** — the existing generic reindex mechanism
+(`reindexSingleIndex()`, `server/services/mappingFixer.ts`, already shipped
+and already exposed at `POST /api/storage/reindex`, `server/routes/storage/admin.ts`)
+already recreates any `INDEX_MAPPINGS`-registered index from scratch and
+copies every document across, which sheds a poisoned mapping's dynamically-
+inferred sub-fields while preserving 100% of the underlying `_source` data
+(proven in `tests/integration/services/storage/evalsRunsMappingMigrationRecipe.integration.test.ts`,
+run against a real OpenSearch container with a deliberately-poisoned index).
+
+**The owner's exact recipe, when ready to run it against the shared cluster:**
+
+```bash
+# 1. Confirm the target index actually needs it (optional sanity check):
+curl -s -X GET "$OPENSEARCH_STORAGE_ENDPOINT/evals_runs/_mapping" \
+  -u "$OPENSEARCH_STORAGE_USERNAME:$OPENSEARCH_STORAGE_PASSWORD" \
+  | jq '.evals_runs.mappings.properties.metrics'
+# If this prints a `dynamic` key, it's already fixed. If it prints only
+# `properties` with more than the 4 legacy metric names, it's poisoned.
+
+# 2. Run the reindex via the running server's admin API (recreates the
+#    index from the current INDEX_MAPPINGS and copies every document
+#    across; the same doc-count-validated, recovery-safe path
+#    reindexSingleIndex() has always used for keyword-type mismatch fixes):
+curl -s -X POST "http://localhost:4001/api/storage/reindex" \
+  -H 'Content-Type: application/json' \
+  -d '{"index": "evals_runs"}'
+```
+
+Caveats to read before running this against the shared cluster:
+
+- **No write lock during a manual `/api/storage/reindex` call.** The
+  auto-fix boot path (`fixIndexMappings()`) acquires a process-local
+  migration lock around the reindex; the manual admin route calls
+  `reindexSingleIndex()` directly and does **not** (pre-existing gap in
+  `server/routes/storage/admin.ts`, not introduced by this PR — flagged here,
+  not fixed, since it's out of scope for this change). Run it during a quiet
+  window (no in-flight evaluation runs writing reports) to avoid a write
+  racing the index delete/recreate step.
+- It touches only `evals_runs`. The already-known-poisoned `evals_experiments`
+  (800+ stale `results.*` fields, per the incident notes) uses the identical
+  recipe (`{"index": "evals_experiments"}`) — that cleanup is separately
+  planned by ops; this PR does not touch or schedule it.
+- Document count is validated before the temporary index is deleted; if the
+  copy-back count doesn't match, the error message names the surviving temp
+  index (`evals_runs_reindex_temp`) for manual recovery — nothing is deleted
+  until the counts are confirmed equal.
+
+## Tests
+
+- Unit (`tests/unit/server/constants/indexMappings.test.ts`): mapping-shape
+  assertions — `dynamic: false` + typed legacy properties on both `metrics`
+  and `matcherResults.judgeMetrics`; pre-existing `enabled:false` fields stay
+  disabled; every field `OpenSearchRunOperations.search()` queries stays
+  explicitly mapped.
+- Integration, real OpenSearch
+  (`tests/integration/services/storage/testCaseRunMetricsMappingGrowth.integration.test.ts`):
+  writes one report with 1000+ distinct custom `metrics`/`judgeMetrics` names
+  (500 report-level + 125 `judge()` calls × 4 dimensions), asserts it
+  round-trips correctly and the index's total mapped-field count does not
+  grow; asserts the query-audit fields stay queryable.
+- Integration, real OpenSearch, migration recipe
+  (`tests/integration/services/storage/evalsRunsMappingMigrationRecipe.integration.test.ts`):
+  deliberately poisons a throwaway index the old way, runs the *existing*
+  `reindexSingleIndex()`, asserts the mapping resets to `dynamic: false` and
+  all document data survives byte-for-byte.
+
+Both integration suites skip gracefully (with a console warning) if no
+OpenSearch cluster is reachable at `TEST_OPENSEARCH_ENDPOINT` (default
+`http://localhost:9200`) — the unit suite covers the mapping-shape assertions
+unconditionally.