@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/accessibility/references/aria-patterns.md

@@ -0,0 +1,113 @@
+# Accessible patterns (copy-ready)
+
+The minimal correct ARIA + keyboard behavior for the widgets you can't build from a single native element. Written fresh; verify intent against the W3C ARIA Authoring Practices Guide. Reach for these **only** when no native element fits (Rule 0).
+
+Recurring obligation for everything custom: **Name, Role, Value** — a name, the right role, and *state kept in sync* with the handler that changes the visuals.
+
+---
+
+## Modal dialog
+
+Prefer native `<dialog>` + `el.showModal()` — it gives you the role, the backdrop, focus trap, and Escape-to-close for free. Build by hand only if you can't.
+
+If hand-rolling: `role="dialog"` (or `alertdialog`), `aria-modal="true"`, and `aria-labelledby` pointing at the title.
+
+| Key            | Behavior                                          |
+| -------------- | ------------------------------------------------- |
+| Open           | Move focus into the dialog (first control/title). |
+| `Tab`          | Cycle within; from last element wrap to first.    |
+| `Shift+Tab`    | From first element wrap to last.                  |
+| `Esc`          | Close; **return focus to the trigger**.           |
+
+Trap focus while open; mark the rest of the page inert (`inert` attribute or `aria-hidden` on the background). Never leave focus loose behind the overlay.
+
+## Disclosure / accordion
+
+A button that shows/hides a region. Native `<details><summary>` covers most cases — use it first.
+
+Hand-rolled: a `<button>` with `aria-expanded` and `aria-controls` pointing at the panel id.
+
+| Key             | Behavior                          |
+| --------------- | --------------------------------- |
+| `Enter`/`Space` | Toggle the panel; flip `aria-expanded`. |
+
+```html
+<button aria-expanded="false" aria-controls="sect1">Details</button>
+<div id="sect1" hidden>…</div>
+```
+
+Toggle `aria-expanded` **and** the `hidden` attribute in the same handler.
+
+## Tabs
+
+`role="tablist"` wraps `role="tab"` buttons; each tab `aria-controls` its `role="tabpanel"`. The active tab has `aria-selected="true"`; the panel has `tabindex="0"`.
+
+Use **roving tabindex**: active tab `tabindex="0"`, others `-1`.
+
+| Key                 | Behavior                                  |
+| ------------------- | ----------------------------------------- |
+| `Tab`               | Into the tablist (one stop), then panel.  |
+| `←` / `→`           | Move between tabs (horizontal tablist).   |
+| `Home` / `End`      | First / last tab.                         |
+| `Enter`/`Space`     | Activate (if not auto-activating).        |
+
+## Combobox (autocomplete)
+
+`role="combobox"` on the input with `aria-expanded`, `aria-controls` → the listbox, and `aria-activedescendant` → the highlighted option id. The popup is `role="listbox"` of `role="option"`.
+
+| Key             | Behavior                                            |
+| --------------- | --------------------------------------------------- |
+| `↓` / `↑`       | Open list / move highlight; update `aria-activedescendant`. |
+| `Enter`         | Select the highlighted option, collapse.            |
+| `Esc`           | Close the list, keep or clear the value.            |
+| typing          | Filter; keep focus in the input the whole time.     |
+
+Focus stays in the input — you move a virtual highlight via `aria-activedescendant`, not real focus.
+
+## Menu button
+
+A `<button aria-haspopup="menu" aria-expanded>` opens a `role="menu"` of `role="menuitem"`. Roving tabindex inside.
+
+| Key                 | Behavior                                  |
+| ------------------- | ----------------------------------------- |
+| `Enter`/`Space`/`↓` | Open menu, focus first item.              |
+| `↑` / `↓`           | Move between items.                       |
+| `Home` / `End`      | First / last item.                        |
+| `Esc`               | Close, return focus to the button.        |
+| `Enter`             | Activate item, close.                     |
+
+Note: a simple navigation dropdown of links is often better as a plain disclosure of `<a>` elements — don't reach for `role="menu"` unless it's an application action menu.
+
+## Toast / live region
+
+Transient status messages. Don't move focus to them — announce via a live region that already exists in the DOM.
+
+- `role="status"` / `aria-live="polite"` — non-urgent ("Saved", "3 results"). Waits for a pause.
+- `role="alert"` / `aria-live="assertive"` — urgent ("Submit failed", "Session expiring"). Interrupts. Use sparingly.
+
+```html
+<!-- present in the DOM before the message; inject text into it -->
+<div role="status" aria-live="polite" class="sr-only"></div>
+```
+
+The region must exist *before* you write into it — injecting both the container and the text at once often isn't announced.
+
+---
+
+## The `.sr-only` utility
+
+Visually hide while keeping it in the accessibility tree (skip links, live regions, icon-button names):
+
+```css
+.sr-only {
+  position: absolute;
+  width: 1px; height: 1px;
+  padding: 0; margin: -1px;
+  overflow: hidden;
+  clip: rect(0 0 0 0);
+  white-space: nowrap;
+  border: 0;
+}
+```
+
+Do **not** use `display:none` or `visibility:hidden` for this — both remove the text from screen readers too.

package/skills/shared/accessibility/references/wcag22-checklist.md

@@ -0,0 +1,83 @@
+# WCAG 2.2 Level AA checklist (grouped by POUR)
+
+The conformance bar. Each item is tagged how it's best verified:
+
+- **[auto]** — an engine (axe-core) catches it reliably.
+- **[manual]** — needs a human (keyboard, screen reader, judgement).
+- **[both]** — automation flags candidates, a human confirms intent.
+
+**NEW** marks a criterion introduced in WCAG 2.2 (W3C Recommendation 2023-10-05). WCAG 2.2 added 9 criteria and **removed 4.1.1 Parsing** — don't chase it anymore.
+
+This is the AA conformance set (A + AA). Numbers (`x.y.z`) are the official SC ids so you can cross-reference the spec.
+
+---
+
+## Perceivable
+
+- **1.1.1 Non-text Content (A)** — every image/icon/control has a text alternative; decorative images use `alt=""`. **[both]** (axe flags missing alt; a human judges whether it's *meaningful*).
+- **1.3.1 Info and Relationships (A)** — structure is in the markup: headings, lists, `<label for>`, `<th scope>`, fieldset/legend. **[both]**
+- **1.3.2 Meaningful Sequence (A)** — DOM order matches reading order; don't reorder with CSS in a way that breaks it. **[manual]**
+- **1.3.4 Orientation (AA)** — don't lock to portrait/landscape unless essential. **[manual]**
+- **1.3.5 Identify Input Purpose (AA)** — use `autocomplete` tokens on personal-data fields (`name`, `email`, `tel`). **[auto]**
+- **1.4.1 Use of Color (A)** — color is never the only way to convey info; pair with text/icon/shape. **[manual]**
+- **1.4.3 Contrast (Minimum) (AA)** — text **4.5:1**; large text (≥24px or ≥18.66px bold) **3:1**. **[auto in a real browser; jsdom can't]**
+- **1.4.4 Resize Text (AA)** — text scales to 200% with no loss of content/function. **[manual]**
+- **1.4.5 Images of Text (AA)** — use real text, not pictures of text, except logos. **[manual]**
+- **1.4.10 Reflow (AA)** — content reflows to a 320px-wide viewport with no 2-D scrolling. **[manual]**
+- **1.4.11 Non-text Contrast (AA)** — UI components and meaningful graphics hit **3:1** (input borders, focus indicators, icon glyphs, chart segments). **[both]**
+- **1.4.12 Text Spacing (AA)** — no content clipped when users override line/letter/word spacing. **[manual]**
+- **1.4.13 Content on Hover or Focus (AA)** — hover/focus popups are dismissable, hoverable, and persistent. **[manual]**
+
+## Operable
+
+- **2.1.1 Keyboard (A)** — all functionality available from the keyboard. **[manual]**
+- **2.1.2 No Keyboard Trap (A)** — focus can always move away with the keyboard. **[manual]**
+- **2.1.4 Character Key Shortcuts (A)** — single-key shortcuts can be turned off/remapped or only fire on focus. **[manual]**
+- **2.4.1 Bypass Blocks (A)** — a skip link or landmarks let users bypass repeated content. **[both]**
+- **2.4.2 Page Titled (A)** — every page has a unique, descriptive `<title>`. **[auto]**
+- **2.4.3 Focus Order (A)** — focus order preserves meaning and operability. **[manual]**
+- **2.4.4 Link Purpose (In Context) (A)** — link text says where it goes; no bare "click here". **[both]**
+- **2.4.5 Multiple Ways (AA)** — more than one way to find a page (nav, search, sitemap). **[manual]**
+- **2.4.6 Headings and Labels (AA)** — headings and labels describe topic/purpose. **[manual]**
+- **2.4.7 Focus Visible (AA)** — keyboard focus is always visibly indicated. **[both]**
+- **2.4.11 Focus Not Obscured (Minimum) (AA)** — **NEW** — a focused element isn't entirely hidden by sticky headers/footers/cookie bars. **[manual]**
+- **2.5.1 Pointer Gestures (A)** — multipoint/path gestures have a single-pointer alternative. **[manual]**
+- **2.5.2 Pointer Cancellation (A)** — actions fire on up-event, allow abort. **[manual]**
+- **2.5.3 Label in Name (A)** — the accessible name contains the visible label text. **[both]**
+- **2.5.4 Motion Actuation (A)** — device-motion features have a UI alternative and can be disabled. **[manual]**
+- **2.5.7 Dragging Movements (AA)** — **NEW** — anything by dragging has a single-pointer (tap/button) alternative. **[manual]**
+- **2.5.8 Target Size (Minimum) (AA)** — **NEW** — interactive targets ≥ **24×24 CSS px**, or spaced so a 24px circle doesn't overlap neighbors (spacing exception). **[both]**
+- **2.3.1 Three Flashes or Below (A)** — nothing flashes more than 3×/second. **[manual]**
+
+## Understandable
+
+- **3.1.1 Language of Page (A)** — `<html lang="…">` set. **[auto]**
+- **3.1.2 Language of Parts (AA)** — inline language changes marked with `lang`. **[manual]**
+- **3.2.1 On Focus (A)** — focus alone doesn't trigger a context change. **[manual]**
+- **3.2.2 On Input (A)** — changing a setting doesn't auto-change context without warning. **[manual]**
+- **3.2.3 Consistent Navigation (AA)** — repeated nav stays in the same relative order. **[manual]**
+- **3.2.4 Consistent Identification (AA)** — same-function components are labeled consistently. **[manual]**
+- **3.2.6 Consistent Help (A)** — **NEW** — help (contact, FAQ link, chat) appears in the same relative order across pages. **[manual]**
+- **3.3.1 Error Identification (A)** — errors are identified in text and described. **[both]**
+- **3.3.2 Labels or Instructions (A)** — inputs have labels/instructions. **[both]**
+- **3.3.3 Error Suggestion (AA)** — when known, suggest a correction. **[manual]**
+- **3.3.4 Error Prevention (Legal/Financial/Data) (AA)** — reversible/checked/confirmed submissions. **[manual]**
+- **3.3.7 Redundant Entry (A)** — **NEW** — don't ask for info already provided in the same process; auto-populate or let them select it. **[manual]**
+- **3.3.8 Accessible Authentication (Minimum) (AA)** — **NEW** — no cognitive-function test to authenticate (no puzzles, no transcription); allow paste and password managers. **[manual]**
+
+## Robust
+
+- **4.1.2 Name, Role, Value (A)** — every UI component exposes a correct name, role, and current state/value to assistive tech. **[both]**
+- **4.1.3 Status Messages (AA)** — status updates are announced without moving focus (live regions / roles). **[both]**
+
+> 4.1.1 Parsing was **removed** in WCAG 2.2. Modern browsers recover from minor markup errors; the criterion no longer applies.
+
+---
+
+## How to work this list
+
+1. Run axe (browser, `wcag22aa` tag) — clears most **[auto]** rows.
+2. Do a keyboard-only pass — clears the operable **[manual]** rows.
+3. One screen-reader spot check — names/roles/state, live-region announcements (4.1.2, 4.1.3).
+4. Zoom 200% + 320px reflow — 1.4.4 / 1.4.10.
+5. Manually verify the 2.2 NEW rows; they're almost all **[manual]** and the ones most teams miss.

package/skills/shared/accessibility/scripts/verify.sh

@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+# verify.sh — accessibility conformance check (read-only by default).
+#
+# Detects the project's a11y tooling and runs the cheapest verdict it can:
+#   1. an explicit a11y test script in package.json (e.g. "test:a11y")
+#   2. @axe-core/playwright  -> needs a URL; runs an inline scan only if A11Y_URL is set
+#   3. jest-axe              -> runs the project's jest tests
+#   4. eslint-plugin-jsx-a11y -> lints src and counts jsx-a11y violations
+#   5. nothing installed     -> prints install hints and EXITS 0 (skip, never a false failure)
+#
+# Exits non-zero ONLY when serious/critical violations are found.
+# Usage: verify.sh [target-dir]   (default: current directory)
+set -euo pipefail
+
+TARGET="${1:-.}"
+cd "$TARGET"
+
+say() { printf '%s\n' "$*"; }
+
+if [ ! -f package.json ]; then
+  say "a11y: no package.json in '$TARGET' — nothing to verify. (skip)"
+  exit 0
+fi
+
+# read a top-level field out of package.json without extra tooling
+pkg_has() { node -e "const p=require('./package.json');const d={...p.dependencies,...p.devDependencies};process.exit(d['$1']?0:1)" 2>/dev/null; }
+script_has() { node -e "const p=require('./package.json');process.exit((p.scripts||{})['$1']?0:1)" 2>/dev/null; }
+
+PM="npm"
+if [ -f pnpm-lock.yaml ]; then PM="pnpm"; elif [ -f yarn.lock ]; then PM="yarn"; fi
+run_script() { case "$PM" in pnpm) pnpm run "$1";; yarn) yarn "$1";; *) npm run "$1";; esac; }
+
+# 1) explicit a11y script wins
+for s in test:a11y a11y test:accessibility accessibility; do
+  if script_has "$s"; then
+    say "a11y: running '$s' via $PM"
+    run_script "$s"
+    exit $?
+  fi
+done
+
+# 2) @axe-core/playwright — needs a live URL
+if pkg_has "@axe-core/playwright"; then
+  if [ -z "${A11Y_URL:-}" ]; then
+    say "a11y: @axe-core/playwright present but A11Y_URL not set."
+    say "      Set A11Y_URL=http://localhost:3000 (with the app running) to scan. (skip)"
+    exit 0
+  fi
+  say "a11y: scanning $A11Y_URL with @axe-core/playwright (wcag2a,wcag2aa,wcag22aa)"
+  node - "$A11Y_URL" <<'NODE'
+const { chromium } = require('playwright');
+const AxeBuilder = require('@axe-core/playwright').default;
+(async () => {
+  const url = process.argv[2];
+  const browser = await chromium.launch();
+  const page = await browser.newPage();
+  await page.goto(url, { waitUntil: 'load' });
+  const r = await new AxeBuilder({ page }).withTags(['wcag2a','wcag2aa','wcag22aa']).analyze();
+  await browser.close();
+  const by = {};
+  for (const v of r.violations) by[v.impact || 'unknown'] = (by[v.impact || 'unknown'] || 0) + 1;
+  console.log('a11y: violations by impact ' + (JSON.stringify(by) || '{}'));
+  const blocking = (by.serious || 0) + (by.critical || 0);
+  if (blocking > 0) { console.error(`a11y: ${blocking} serious/critical violation(s) — FAIL`); process.exit(1); }
+  console.log('a11y: no serious/critical violations');
+})().catch(e => { console.error('a11y: scan error — ' + e.message + ' (skip)'); process.exit(0); });
+NODE
+  exit $?
+fi
+
+# 3) jest-axe — run jest if a test script exists
+if pkg_has "jest-axe"; then
+  if script_has "test"; then
+    say "a11y: jest-axe present — running '$PM test' (note: contrast disabled in jsdom)"
+    run_script test
+    exit $?
+  fi
+  say "a11y: jest-axe present but no 'test' script. (skip)"
+  exit 0
+fi
+
+# 4) eslint-plugin-jsx-a11y — lint and count
+if pkg_has "eslint-plugin-jsx-a11y"; then
+  SRC="src"; [ -d "$SRC" ] || SRC="."
+  say "a11y: linting $SRC with eslint (jsx-a11y rules)"
+  OUT="$(npx --no-install eslint "$SRC" --ext .js,.jsx,.ts,.tsx -f json 2>/dev/null || true)"
+  if [ -z "$OUT" ]; then
+    say "a11y: eslint produced no output (config or install issue). (skip)"
+    exit 0
+  fi
+  COUNT="$(printf '%s' "$OUT" | node -e "let s='';process.stdin.on('data',d=>s+=d).on('end',()=>{try{const r=JSON.parse(s);let n=0;for(const f of r)for(const m of f.messages)if((m.ruleId||'').startsWith('jsx-a11y/'))n++;console.log(n)}catch{console.log(0)}})")"
+  say "a11y: jsx-a11y violations: $COUNT"
+  [ "$COUNT" -gt 0 ] && exit 1
+  exit 0
+fi
+
+# 5) nothing detected — guide, don't fail
+say "a11y: no a11y tooling detected. (skip)"
+say "      Install one of:"
+say "        npm i -D eslint-plugin-jsx-a11y   # static lint of JSX"
+say "        npm i -D jest-axe                 # unit-level axe (contrast off in jsdom)"
+say "        npm i -D @axe-core/playwright     # real-browser scan incl. contrast"
+exit 0

package/skills/shared/agent-eval/SKILL.md

@@ -0,0 +1,187 @@
+---
+name: agent-eval
+description: "Use when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual recall) or agent trajectories (tool correctness, completion), or picking an eval framework. NOT building the agent loop, tools or RAG plumbing (that is `building-agents`)."
+tags: [evals, llm, agents, llm-as-judge, regression-gate, ai]
+recommends: [building-agents, prompt-engineering, observability]
+origin: ai-hub
+---
+
+# Measure agent quality you can defend and gate on
+
+Turn "the agent feels better" into a number you can put in a PR check. You own the eval dataset, the scorer mix, the LLM-as-judge calibration, and the block-on-regression CI gate — framework-neutral, provider-neutral.
+
+## Do NOT use — route instead
+
+| The ask | Route to | Why it is not this skill |
+| --- | --- | --- |
+| Build the agent loop, tools, RAG plumbing | `building-agents` | It builds the system; you score it. They cross-link. |
+| "Make the answers shorter / rewrite the prompt" | `prompt-engineering` | Evals say it is worse; that skill changes the words. You never edit the prompt. |
+| pytest/jest on deterministic functions | `testing-py` / `testing-web` | Assert-equals on pure code, not stochastic outputs scored by a judge. |
+| Dashboards / tracing of live production traffic | `observability` | Online monitoring; you are offline + pre-merge. |
+| Red-team, jailbreak, prompt injection | `agent-safety` | Adversarial coverage, not quality measurement. |
+| Per-token cost budgets and accounting | `cost-tracking` | You report cost-per-task as one metric; the discipline lives there. |
+| A/B stats on product/funnel metrics | `ab-testing` | Web experiments, not offline model comparison on a fixed set. |
+
+## The eval anatomy
+
+Every framework instantiates the same five-stage pipeline. Learn it once; the tool is a detail.
+
+```text
+dataset ──▶ runner ──▶ scorers ──▶ metrics ──▶ gate
+(JSONL    (calls the   (det / judge  (aggregate +  (pass/fail
+golden    system per   / human)      bootstrap CI) exit code)
+set)      case)
+```
+
+DeepEval, Inspect AI, and promptfoo are all just opinionated wrappers around this. If you understand the stages you can switch tools without relearning the craft.
+
+## Build the dataset first
+
+Build your own golden set — a public leaderboard number is not your number, because identical model weights swing SWE-Bench Verified by 10–20 points just by changing the harness. Measure *your* task on *your* data. The dataset is the asset; everything else is replaceable. Rules:
+
+- **50–200 hand-labeled cases per failure mode**, not per total. Coverage of how the system fails beats raw volume. 80 real failure cases > 1000 generic ones.
+- **Never synthetic-only.** A set the model wrote will not surface the model's blind spots. Mine real traffic / tickets / transcripts and hand-label.
+- **Version it in git as JSONL**, a first-class reviewed asset — same as code. Diffs are reviewable; relabels are auditable.
+- **Decontaminate.** The eval set must not appear in training data or few-shot examples, or the score is a memorization artifact, not a capability.
+
+Case schema — one JSON object per line:
+
+```jsonl
+{"id":"refund-001","input":"Where is my refund for order 4821?","expected":"States refunds take 5-7 business days and asks for nothing already on file","context":["policy: refunds 5-7 business days"],"meta":{"failure_mode":"hallucinated_policy","source":"ticket#4821"}}
+{"id":"refund-002","input":"Cancel my subscription and refund this month","expected":"Cancels, refunds prorated amount, confirms no future charge","context":["policy: prorated refund on cancel"],"meta":{"failure_mode":"missed_tool_call","source":"ticket#5190"}}
+```
+
+`failure_mode` in `meta` is what lets you slice metrics by mode and find *which* kind of bug regressed — not just that the aggregate dropped.
+
+> Bad: "generate 1000 test questions with GPT and use those." Good: "80 real failure-mode cases pulled from support tickets, hand-labeled, tagged by failure mode."
+
+## Choose the scorer — the 60/30/10 mix
+
+Reach for the cheapest scorer that correlates with human judgment. Default mix:
+
+| Share | Scorer kind | Use for | Why |
+| --- | --- | --- | --- |
+| ~60% | Deterministic — exact match, regex, JSON-schema validation, latency threshold | Anything with a checkable shape: format, required fields, a known string, a budget | Free, instant, zero drift. Never spend a judge call on something a regex settles. |
+| ~30% | LLM-as-judge — G-Eval, DAG, custom Python scorer | Meaning: is this answer faithful, relevant, helpful | Only where correctness is semantic. Costs money and can drift — so calibrate it. |
+| ~10% | Human-in-the-loop | Genuinely ambiguous cases the judge disagrees on | The ground truth you calibrate the judge against. |
+
+One `Scorer` protocol, two implementations behind it — deterministic and judge are interchangeable to the runner:
+
+```python
+from typing import Protocol
+
+class Scorer(Protocol):
+    name: str
+    def score(self, case: dict, output: str) -> float: ...  # 0.0–1.0
+
+class JsonSchemaScorer:
+    name = "schema_valid"
+    def score(self, case, output):  # deterministic, free, no drift
+        import json
+        try:
+            json.loads(output)
+            return 1.0
+        except ValueError:
+            return 0.0
+
+class FaithfulnessJudge:
+    name = "faithfulness"
+    def __init__(self, judge_model): self.judge = judge_model
+    def score(self, case, output):  # judge only where meaning matters
+        return self.judge.rate(case["context"], output)  # see judge-design.md
+```
+
+## LLM-as-judge you can trust
+
+A score you do not trust is worse than no score: an uncalibrated judge gives false confidence, which is more dangerous than admitted ignorance. Each rule, with its why:
+
+- **Judge model ≥ system under test.** A weaker judge cannot reliably rank a stronger system — it scores noise.
+- **The rubric must force a written rationale before the score.** Rationale-first judging is what pushes judge–human agreement to ~85% — higher than two humans agree with each other. A bare number is a vibe with a decimal point.
+- **Pairwise beats pointwise for stability.** "Is A or B better?" is more reproducible than "rate A from 1–10," which inflates and clusters at 8–9.
+- **Swap positions and average.** Judges favor whichever answer came first; run A-then-B and B-then-A to cancel position bias.
+- **Calibrate against human gold and report the agreement before you gate anything on the judge.** Not a formality — this is the step that makes every number downstream defensible.
+
+> Bad judge prompt: "Rate this answer 1–10." → everything lands 8–9, useless.
+> Good: "Compare answer A and answer B against the reference. First write one sentence on each per the rubric, then output the better label." → forces reasoning, gives a stable signal.
+
+Full rubric templates (pointwise + pairwise), the position-swap harness, the calibration script (agreement / Cohen's kappa vs human gold), G-Eval vs DAG, and the judge bias catalog (length, position, self-preference) with mitigations live in **[references/judge-design.md](references/judge-design.md)**.
+
+## Agent and RAG scorers
+
+Score the path, not only the destination. Beyond exact/judge:
+
+**RAG** (DeepEval / RAGAS names):
+- **Faithfulness** — does the answer only claim what the retrieved context supports? Catches hallucination.
+- **Answer relevancy** — does it actually address the question, or drift?
+- **Contextual recall / precision** — did retrieval fetch the right chunks, and not bury them in noise? Separates a retrieval bug from a generation bug.
+
+**Agent:**
+- **Tool correctness** — right tool, right arguments, right order.
+- **Task completion / goal accuracy** — did it finish the job, not just produce plausible text.
+- **Trajectory scoring** — grade the sequence of steps. A correct final answer from a wrong path will fail differently next time; only trajectory scoring catches it.
+
+The system side of these (how the loop and tools are built) is `../building-agents/SKILL.md`; a common system-under-test is `../chatbot/SKILL.md`.
+
+## The regression gate
+
+> Gate policy: **block on regression vs a committed baseline, not on an absolute threshold.** An absolute threshold flaps CI on judge noise and gives no signal on drift; "did this PR make a tracked metric worse than `main`?" is the question that matters.
+
+- Compute a **bootstrap confidence interval** on each metric so judge noise alone does not fail the build — only a drop beyond the CI counts.
+- The runner writes `eval-report.json` (metrics, per-failure-mode slices, baseline, pass/fail) and **exits non-zero** on a real regression so the merge is blocked.
+
+```python
+import json, sys
+
+def gate(current: dict, baseline: dict, margin: float = 0.0) -> int:
+    regressed = []
+    for metric, score in current.items():
+        if metric in baseline and score < baseline[metric] - margin:
+            regressed.append((metric, baseline[metric], score))
+    report = {"metrics": current, "baseline": baseline, "regressed": regressed,
+              "passed": not regressed}
+    with open("eval-report.json", "w") as f:
+        json.dump(report, f, indent=2)
+    if regressed:
+        for m, b, c in regressed:
+            print(f"REGRESSION {m}: {b:.3f} -> {c:.3f}", file=sys.stderr)
+        return 1
+    return 0
+
+sys.exit(gate(run_eval(), json.load(open("eval-baseline.json"))))
+```
+
+The complete provider-neutral runner (JSONL loader, scorer registry, bootstrap-CI metrics), the GitHub Actions workflow, and side-by-side DeepEval-pytest + Inspect-AI Task/Solver/Scorer versions of the same eval live in **[references/runner-and-gate.md](references/runner-and-gate.md)**.
+
+## Framework cheat-sheet
+
+Pick by where the eval runs and what it must do. Versions as of 2026-06 — re-verify, they rot.
+
+| Tool | What it is | Reach for it when |
+| --- | --- | --- |
+| **DeepEval** v4.0.3 | pytest-native, 50+ metrics, Decision-Graph (DAG) logic | Your CI is Python/pytest and you want metrics that read like tests. |
+| **Inspect AI** v0.3.225 (UK AISI) | dataset→Task→Solver→Scorer, bootstrap CIs, first-class tool-use & trajectory logging, 200+ pre-built evals | Multi-provider, safety-adjacent, or you need real trajectory scoring. |
+| **promptfoo** (acquired by OpenAI 2026-03) | CLI + YAML, strong pre-deploy + red-team across 50+ vuln types | Config-driven pre-deploy checks; route the red-team half to `agent-safety`. |
+| **Braintrust / LangSmith / Phoenix** v16.0.0 | platforms: annotation, regression tracking, dashboards | You need human annotation queues and historical regression tracking. |
+
+> The two-tool pattern is normal, not over-engineering: a light CI gate (DeepEval / RAGAS / promptfoo) **plus** a platform (Braintrust / LangSmith / Arize) for annotation and history. They share data; different jobs.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+| --- | --- | --- |
+| Vibes-gating ("feels better, merge it") | No artifact to defend or reproduce | Gate on a number from a committed dataset |
+| Synthetic-only dataset | Model-written cases miss the model's blind spots | Hand-label real traffic by failure mode |
+| Uncalibrated judge | Confident wrong scores; worse than none | Report agreement vs human gold first |
+| Judge weaker than system | Cannot rank a stronger system; scores noise | Judge model ≥ system under test |
+| Absolute-threshold gate | Flaps CI on judge noise, blind to drift | Block on regression vs baseline + bootstrap CI |
+| Shipping on a leaderboard number | Harness effect = 10–20pt swing | Build your own golden set |
+| Scoring only the final answer | A right answer from a wrong path regresses later | Score the trajectory too |
+| Never relabeling drifted gold | Stale "truth" silently rots the gate | Review and relabel the golden set on a schedule |
+
+## Project grounding
+
+If the workspace has a `02-DOCS/` harness, record the eval policy in `02-DOCS/wiki/stack/evals.md`: dataset location, scorer mix, gate baseline file, judge model, and the failure modes covered. Follow the harness [`wiki-article-template.md`](../harness/references/wiki-article-template.md) (`type: stack`) and index it in `02-DOCS/wiki/index.md`. This is **recorded, not gated** — skip silently if there is no harness.
+
+## verify.sh
+
+`scripts/verify.sh` is read-only and tool-detecting. It validates that every `*.jsonl` golden set in the project parses and that each line carries the required `id`, `input`, `expected` keys; checks the shape of any `eval-report.json`; and runs `ruff` / `mypy` on example Python and `markdownlint` on docs when those tools are installed. Every missing tool prints a yellow WARN and is skipped — never a failure. An empty or clean target exits 0.

package/skills/shared/agent-eval/evals/README.md

@@ -0,0 +1,12 @@
+# Evals for agent-eval
+
+`cases.yaml` is the trigger and capability spec for this skill. It is read by the catalog's
+skill-eval harness, not by a standalone runner here. `should_trigger` lists prompts the skill
+must claim (including non-obvious and Spanish phrasings); `should_not_trigger` lists prompts
+that must route to a named sibling instead; `capability` is a rubric a graded run must satisfy.
+
+To check by hand: read each `should_trigger` prompt and confirm the description in `SKILL.md`
+would plausibly fire on it; read each `should_not_trigger` prompt and confirm the `route_to`
+sibling is the better home and is a real catalog id. For the `capability` case, draft the
+skill's answer and confirm every `must_include` bullet is covered. If your harness scores these
+automatically, point it at this file with `skill: agent-eval` as the key.

package/skills/shared/agent-eval/evals/cases.yaml

@@ -0,0 +1,45 @@
+skill: agent-eval
+
+should_trigger:
+  - prompt: "Is the new system prompt actually better than the old one, or am I just fooling myself?"
+    why: "Before/after on a fixed dataset is the core job. Non-obvious: uses no eval vocabulary at all."
+  - prompt: "Build me a golden set and a CI check that fails the PR if our support bot's answer quality drops."
+    why: "Dataset construction plus a block-on-regression gate, stated explicitly."
+  - prompt: "Our LLM judge gives every answer a 9 out of 10 — the scores are useless."
+    why: "Judge calibration / inflation; the fix is rationale-forcing rubric and pairwise over pointwise. Non-obvious symptom phrasing, no mention of 'eval'."
+  - prompt: "Score our RAG pipeline for faithfulness and whether it actually uses the retrieved context."
+    why: "RAG metrics — faithfulness and contextual recall — are squarely this skill."
+  - prompt: "I need to know if the agent took the right tool path, not just whether the final answer looks ok."
+    why: "Trajectory and tool-correctness scoring. Non-obvious: framed as behavior, not as 'eval'."
+  - prompt: "Necesito medir si el agente mejoró con el modelo nuevo antes de hacer merge."
+    why: "Spanish — before/after measurement plus a pre-merge gate."
+  - prompt: "Should we use DeepEval or Inspect AI for our eval suite, and how do we wire the gate?"
+    why: "Framework choice plus gate wiring; the cheat-sheet and runner are the answer."
+
+should_not_trigger:
+  - prompt: "Rewrite this system prompt so the answers come out shorter and punchier."
+    route_to: prompt-engineering
+    why: "Changing the words, not measuring quality. Evals say it is worse; that skill changes it."
+  - prompt: "Build the agent loop and tool registry for our new assistant."
+    route_to: building-agents
+    why: "Building the system under test, not scoring it."
+  - prompt: "Add a dashboard showing token cost and latency of our live LLM traffic."
+    route_to: observability
+    why: "Online monitoring of production, not offline pre-merge evaluation."
+  - prompt: "Write pytest unit tests for our data-cleaning functions."
+    route_to: testing-py
+    why: "Deterministic code with assert-equals, not stochastic outputs scored by a judge."
+  - prompt: "Red-team our chatbot for jailbreaks and prompt injection."
+    route_to: agent-safety
+    why: "Adversarial safety coverage, not quality measurement on a golden set."
+
+capability:
+  - scenario: "Set up an eval plus a regression gate for a RAG support bot that is regressing silently between releases."
+    must_include:
+      - "Builds a versioned JSONL golden set of 50-200 hand-labeled cases per failure mode, decontaminated, not synthetic-only"
+      - "Picks a scorer mix: deterministic first (schema/regex/latency), faithfulness and answer-relevancy judge where meaning matters"
+      - "Calibrates the LLM-as-judge against human labels and reports agreement (raw and/or Cohen's kappa) before trusting it"
+      - "Gate policy is block-on-regression vs a committed baseline with a bootstrap CI, not an absolute threshold"
+      - "Runner emits eval-report.json and exits non-zero to block the merge"
+      - "Names at least one concrete framework (DeepEval, Inspect AI, or promptfoo) and fits it to the case"
+      - "Records the eval policy in docs/stack/evals.md if a harness exists — recorded, not gated"