hatch3r 1.7.0 → 1.7.5

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hatch3r",
-  "version": "1.7.0",
+  "version": "1.7.5",
   "description": "Battle-tested agentic coding setup framework. One command to hatch your agent stack -- agents, skills, rules, commands, and MCP for every major AI coding tool.",
   "type": "module",
   "exports": {
@@ -24,7 +24,9 @@
     "inventory:check-docs": "tsx scripts/inventory.ts --check-docs",
     "validate:rule-parity": "tsx scripts/validate-rule-parity.ts",
     "validate:efficiency": "tsx scripts/validate-efficiency-invariants.ts",
-    "validate": "npm run validate:rule-parity && npm run validate:efficiency",
+    "validate:cli-skills": "tsx scripts/validate-cli-skills.ts",
+    "generate:cli-skills": "tsx scripts/generate-cli-skills.ts",
+    "validate": "npm run validate:rule-parity && npm run validate:efficiency && npm run validate:cli-skills",
     "audit:validate-registry": "tsx scripts/validate-finding-registry.ts",
     "audit:migrate": "tsx scripts/migrate-finding-registry.ts",
     "audit:archive": "tsx scripts/audit-archive.ts",

package/rules/hatch3r-accessibility-standards.md

@@ -78,3 +78,24 @@ All user-facing features must meet WCAG 2.2 Level AA conformance. This is the ba
 - Run automated accessibility checks (axe-core, Lighthouse) in CI.
 - Automated tools catch ~30% of accessibility issues. Manual testing is required for keyboard flows, screen reader experience, and cognitive accessibility.
 - Maintain an accessibility test checklist per component type (form, modal, navigation, data table).
+
+## WCAG 2.2 New Success Criteria (Mandatory Audit Items)
+
+WCAG 2.2 (W3C Recommendation, October 2023) added nine Success Criteria; three apply to most UI surfaces and are listed here as required audit items.
+
+- **SC 2.5.8 Target Size (Minimum) — AA:** every pointer-target's hit area is at least 24 by 24 CSS pixels. Smaller targets are permitted only when surrounded by ≥24px of spacing (the spacing exception) or when the larger inline target is also reachable (the inline exception). Test densest UI elements (sidebar collapse, table-row checkboxes, icon-only toolbar buttons).
+- **SC 2.4.11 Focus Not Obscured (Minimum) — AA:** when a control receives keyboard focus, the focused element must not be entirely hidden by author-created content. Common violations: sticky headers, persistent cookie banners, chatbot launcher widgets, fixed action bars. Mitigate with `scroll-margin-top` on focus targets equal to the sticky header height, or by collapsing sticky chrome on focus.
+- **SC 2.5.7 Dragging Movements — AA:** any drag operation (reorder, slider thumb drag, map pan, kanban card move) must offer a single-pointer non-drag alternative. Examples: list reorder with up/down arrow buttons; slider with numeric input or +/- buttons; map pan with arrow-key navigation.
+
+Reference: https://www.w3.org/TR/WCAG22/
+
+## Mobile and Touch Accessibility
+
+Touch surfaces have stricter target and spacing requirements than pointer-only surfaces; native platform guidance supersedes WCAG 2.5.8 on touch.
+
+- Touch targets: 44 by 44 points on iOS (Apple Human Interface Guidelines), 48 by 48 dp on Android (Material Design 3). These supersede WCAG 2.5.8's 24-pixel minimum on touch-only surfaces.
+- Spacing between interactive elements: ≥8 pixels of separation to prevent mis-taps; ≥4 mm on physical-density screens.
+- Avoid tap-to-reveal patterns (hover tooltips, pure-hover dropdowns) — touch has no hover state. Replace with permanent visible labels or with long-press that has a visible affordance and announces itself to screen readers.
+- Apply `env(safe-area-inset-*)` padding on full-bleed surfaces so content clears notches, home indicators, and rounded corners on iOS and Android edge devices.
+- Support Dynamic Type (iOS) and rem-based font scaling — declare body text in `rem` or `em` units, never `px`, so OS-level font size settings cascade.
+- Zoom to 200% and 400% (per WCAG 1.4.4 and 1.4.10 Reflow) must remain functional with no horizontal scroll trap. Audit for `width: 100vw` and fixed pixel widths that break reflow.

package/rules/hatch3r-accessibility-standards.mdc

@@ -74,3 +74,24 @@ All user-facing features must meet WCAG 2.2 Level AA conformance. This is the ba
 - Run automated accessibility checks (axe-core, Lighthouse) in CI.
 - Automated tools catch ~30% of accessibility issues. Manual testing is required for keyboard flows, screen reader experience, and cognitive accessibility.
 - Maintain an accessibility test checklist per component type (form, modal, navigation, data table).
+
+## WCAG 2.2 New Success Criteria (Mandatory Audit Items)
+
+WCAG 2.2 (W3C Recommendation, October 2023) added nine Success Criteria; three apply to most UI surfaces and are listed here as required audit items.
+
+- **SC 2.5.8 Target Size (Minimum) — AA:** every pointer-target's hit area is at least 24 by 24 CSS pixels. Smaller targets are permitted only when surrounded by ≥24px of spacing (the spacing exception) or when the larger inline target is also reachable (the inline exception). Test densest UI elements (sidebar collapse, table-row checkboxes, icon-only toolbar buttons).
+- **SC 2.4.11 Focus Not Obscured (Minimum) — AA:** when a control receives keyboard focus, the focused element must not be entirely hidden by author-created content. Common violations: sticky headers, persistent cookie banners, chatbot launcher widgets, fixed action bars. Mitigate with `scroll-margin-top` on focus targets equal to the sticky header height, or by collapsing sticky chrome on focus.
+- **SC 2.5.7 Dragging Movements — AA:** any drag operation (reorder, slider thumb drag, map pan, kanban card move) must offer a single-pointer non-drag alternative. Examples: list reorder with up/down arrow buttons; slider with numeric input or +/- buttons; map pan with arrow-key navigation.
+
+Reference: https://www.w3.org/TR/WCAG22/
+
+## Mobile and Touch Accessibility
+
+Touch surfaces have stricter target and spacing requirements than pointer-only surfaces; native platform guidance supersedes WCAG 2.5.8 on touch.
+
+- Touch targets: 44 by 44 points on iOS (Apple Human Interface Guidelines), 48 by 48 dp on Android (Material Design 3). These supersede WCAG 2.5.8's 24-pixel minimum on touch-only surfaces.
+- Spacing between interactive elements: ≥8 pixels of separation to prevent mis-taps; ≥4 mm on physical-density screens.
+- Avoid tap-to-reveal patterns (hover tooltips, pure-hover dropdowns) — touch has no hover state. Replace with permanent visible labels or with long-press that has a visible affordance and announces itself to screen readers.
+- Apply `env(safe-area-inset-*)` padding on full-bleed surfaces so content clears notches, home indicators, and rounded corners on iOS and Android edge devices.
+- Support Dynamic Type (iOS) and rem-based font scaling — declare body text in `rem` or `em` units, never `px`, so OS-level font size settings cascade.
+- Zoom to 200% and 400% (per WCAG 1.4.4 and 1.4.10 Reflow) must remain functional with no horizontal scroll trap. Audit for `width: 100vw` and fixed pixel widths that break reflow.

package/rules/hatch3r-agent-orchestration.md

@@ -19,6 +19,8 @@ Hatch3r's orchestration uses a **phase-gated pipeline** (Research, Implement, Re
 
 This rule applies to EVERY context without exception: board-pickup (epic, sub-issue, standalone, batch), workflow command (full/quick), plain chat, issue references, and natural language requests. The full sub-agent pipeline is mandatory — never implement code inline without sub-agents.
 
+**"Inline implementation" defined.** Inline implementation means calling any code-writing tool — `Edit`, `Write`, `MultiEdit`, `NotebookEdit`, `replace_string_in_file`, `multi_replace_string_in_file`, `create_file`, `str_replace_based_edit_tool`, `apply_patch`, or any platform equivalent — from the orchestrator turn itself, rather than from inside a spawned `hatch3r-implementer` (Phase 2) or `hatch3r-fixer` (Phase 3) sub-agent. The only carve-out is `hatch3r-quick-change` for Tier 1 single-line trivial edits per its declared scope.
+
 ## Universal Sub-Agent Pipeline
 
 Every task MUST follow this four-phase pipeline: **Phase 1 — Research** (`hatch3r-researcher`), **Phase 2 — Implement** (`hatch3r-implementer`), **Phase 3 — Review Loop** (`hatch3r-reviewer` + `hatch3r-fixer`), **Phase 4 — Final Quality** (parallel specialists). See Mandatory Delegation Directives below.
@@ -46,7 +48,7 @@ Every task MUST follow this four-phase pipeline: **Phase 1 — Research** (`hatc
 
 Score task complexity per the `hatch3r-deep-context` rule before Phase 1. Apply the resulting tier:
 
-- **Tier 2 (Standard):** Present elicitation questions inline. Await answers before Phase 2.
+- **Tier 2 hard gate (B1).** Before Phase 2, run `hatch3r-researcher` with `requirements-elicitation:quick` mode to detect ambiguity and ask the user via the platform-native question tool (per `agents/shared/user-question-protocol.md`). Orchestrator awaits answers and integrates them into the Phase 1 brief; do not begin Phase 2 with unresolved questions. Tier 1 is exempt only when scope is single-file, single-concern, and acceptance is testable from the user message alone.
 - **Tier 3 (Deep):** Present Pre-Implementation Summary and ASK for confirmation. Do NOT proceed until all unresolved questions are answered.
 
 ## Mandatory Delegation Directives
@@ -84,6 +86,27 @@ Spawn `hatch3r-implementer` via Task tool for ALL code changes. Never implement
 
 **Implementer prompt enrichment (Tier 2+):** Include `similar-implementation` findings as "Reference Conventions", resolved `requirements-elicitation` answers as "Resolved Requirements", and blast radius data (Tier 3 only).
 
+### Per-Turn Pipeline-State Header
+
+Whenever a tracked task is active at Tier 2 or Tier 3 (deep-context score >= 3), the orchestrator MUST emit a single-line pipeline-state header at the very start of every assistant turn that touches the task. Format:
+
+```
+[hatch3r-pipeline: phase {1|2|3|4} | last: {agent} → {SUCCESS|PARTIAL|FAILED|BLOCKED|n/a} | next: {agent or "user-confirmation" or "complete"}]
+```
+
+Examples:
+
+- `[hatch3r-pipeline: phase 1 | last: n/a | next: hatch3r-researcher]`
+- `[hatch3r-pipeline: phase 2 | last: hatch3r-researcher → SUCCESS | next: hatch3r-implementer]`
+- `[hatch3r-pipeline: phase 3 | last: hatch3r-reviewer → PARTIAL | next: hatch3r-fixer]`
+- `[hatch3r-pipeline: phase 3 | last: hatch3r-implementer → SUCCESS | next: user-confirmation]`
+
+A missing header on a tracked Tier >= 2 task is a self-detectable drift signal — the user may halt the turn and request re-grounding. The header also functions as a per-reply cache prime: rendering it forces the orchestrator to re-resolve which phase it is in before choosing tools. Tier 1 tasks, read-only answers, and chat-only iterations do NOT require the header.
+
+### Mandatory Delegation Directive (No Inline Implementation)
+
+Restating with maximum clarity for sub-agent prompt inclusion: the orchestrator MUST NOT call `Edit`, `Write`, `MultiEdit`, `NotebookEdit`, `replace_string_in_file`, `multi_replace_string_in_file`, `create_file`, `str_replace_based_edit_tool`, `apply_patch`, or any platform-equivalent code-writing tool from its own turn. The only path for code mutation is the Task tool spawning `hatch3r-implementer` (Phase 2) or `hatch3r-fixer` (Phase 3). Carve-out: `hatch3r-quick-change` Tier 1 trivial items per its declared scope. No other carve-out exists. Violations are bypass mode (see issue #73) — surface them by halting the turn and re-delegating.
+
 ### Mid-Implementation Research Gap Checkpoint
 
 At the midpoint of Phase 2 (after initial files are modified but before completion), the implementer MUST evaluate whether research gaps exist. This prevents discovering missing context too late in the pipeline.
@@ -230,6 +253,14 @@ ALL three must hold:
 
 **Default:** When in doubt, serialize. For typical hatch3r tasks (1–5 sub-tasks) the DAG-scheduling overhead often outweighs concurrency gain.
 
+### Cost-Dominance Principle
+
+Token cost of sub-agent invocation never justifies serialization of independent work. The three safety conditions (read-only or disjoint writes, deterministic aggregation, no shared mutable state) govern WHEN parallelism is safe; cost does not govern WHETHER to parallelize. When in doubt, fan out. Serialization is only valid on true dependency edges.
+
+### Scaling Heuristic
+
+Sub-agent count tracks task decomposition: N independent modules → N parallel Phase-2 implementers; M specialist gates → M parallel Phase-4 specialists; K independent research questions → K parallel researcher modes. Orchestrators emit `sub_agents_spawned: {count, rationale}` in their structured output.
+
 ## Cross-Phase Error Propagation
 
 When a phase produces a non-SUCCESS status, the orchestrator must propagate error context to downstream phases rather than silently dropping it:

package/rules/hatch3r-agent-orchestration.mdc

@@ -14,6 +14,8 @@ Hatch3r's orchestration uses a **phase-gated pipeline** (Research, Implement, Re
 
 This rule applies to EVERY context without exception: board-pickup (epic, sub-issue, standalone, batch), workflow command (full/quick), plain chat, issue references, and natural language requests. The full sub-agent pipeline is mandatory — never implement code inline without sub-agents.
 
+**"Inline implementation" defined.** Inline implementation means calling any code-writing tool — `Edit`, `Write`, `MultiEdit`, `NotebookEdit`, `replace_string_in_file`, `multi_replace_string_in_file`, `create_file`, `str_replace_based_edit_tool`, `apply_patch`, or any platform equivalent — from the orchestrator turn itself, rather than from inside a spawned `hatch3r-implementer` (Phase 2) or `hatch3r-fixer` (Phase 3) sub-agent. The only carve-out is `hatch3r-quick-change` for Tier 1 single-line trivial edits per its declared scope.
+
 ## Universal Sub-Agent Pipeline
 
 Every task MUST follow this four-phase pipeline: **Phase 1 — Research** (`hatch3r-researcher`), **Phase 2 — Implement** (`hatch3r-implementer`), **Phase 3 — Review Loop** (`hatch3r-reviewer` + `hatch3r-fixer`), **Phase 4 — Final Quality** (parallel specialists). See Mandatory Delegation Directives below.
@@ -41,7 +43,7 @@ Every task MUST follow this four-phase pipeline: **Phase 1 — Research** (`hatc
 
 Score task complexity per the `hatch3r-deep-context` rule before Phase 1. Apply the resulting tier:
 
-- **Tier 2 (Standard):** Present elicitation questions inline. Await answers before Phase 2.
+- **Tier 2 hard gate (B1).** Before Phase 2, run `hatch3r-researcher` with `requirements-elicitation:quick` mode to detect ambiguity and ask the user via the platform-native question tool (per `agents/shared/user-question-protocol.md`). Orchestrator awaits answers and integrates them into the Phase 1 brief; do not begin Phase 2 with unresolved questions. Tier 1 is exempt only when scope is single-file, single-concern, and acceptance is testable from the user message alone.
 - **Tier 3 (Deep):** Present Pre-Implementation Summary and ASK for confirmation. Do NOT proceed until all unresolved questions are answered.
 
 ## Mandatory Delegation Directives
@@ -79,6 +81,27 @@ Spawn `hatch3r-implementer` via Task tool for ALL code changes. Never implement
 
 **Implementer prompt enrichment (Tier 2+):** Include `similar-implementation` findings as "Reference Conventions", resolved `requirements-elicitation` answers as "Resolved Requirements", and blast radius data (Tier 3 only).
 
+### Per-Turn Pipeline-State Header
+
+Whenever a tracked task is active at Tier 2 or Tier 3 (deep-context score >= 3), the orchestrator MUST emit a single-line pipeline-state header at the very start of every assistant turn that touches the task. Format:
+
+```
+[hatch3r-pipeline: phase {1|2|3|4} | last: {agent} → {SUCCESS|PARTIAL|FAILED|BLOCKED|n/a} | next: {agent or "user-confirmation" or "complete"}]
+```
+
+Examples:
+
+- `[hatch3r-pipeline: phase 1 | last: n/a | next: hatch3r-researcher]`
+- `[hatch3r-pipeline: phase 2 | last: hatch3r-researcher → SUCCESS | next: hatch3r-implementer]`
+- `[hatch3r-pipeline: phase 3 | last: hatch3r-reviewer → PARTIAL | next: hatch3r-fixer]`
+- `[hatch3r-pipeline: phase 3 | last: hatch3r-implementer → SUCCESS | next: user-confirmation]`
+
+A missing header on a tracked Tier >= 2 task is a self-detectable drift signal — the user may halt the turn and request re-grounding. The header also functions as a per-reply cache prime: rendering it forces the orchestrator to re-resolve which phase it is in before choosing tools. Tier 1 tasks, read-only answers, and chat-only iterations do NOT require the header.
+
+### Mandatory Delegation Directive (No Inline Implementation)
+
+Restating with maximum clarity for sub-agent prompt inclusion: the orchestrator MUST NOT call `Edit`, `Write`, `MultiEdit`, `NotebookEdit`, `replace_string_in_file`, `multi_replace_string_in_file`, `create_file`, `str_replace_based_edit_tool`, `apply_patch`, or any platform-equivalent code-writing tool from its own turn. The only path for code mutation is the Task tool spawning `hatch3r-implementer` (Phase 2) or `hatch3r-fixer` (Phase 3). Carve-out: `hatch3r-quick-change` Tier 1 trivial items per its declared scope. No other carve-out exists. Violations are bypass mode (see issue #73) — surface them by halting the turn and re-delegating.
+
 ### Mid-Implementation Research Gap Checkpoint
 
 At the midpoint of Phase 2 (after initial files are modified but before completion), the implementer MUST evaluate whether research gaps exist. This prevents discovering missing context too late in the pipeline.
@@ -225,6 +248,14 @@ ALL three must hold:
 
 **Default:** When in doubt, serialize. For typical hatch3r tasks (1–5 sub-tasks) the DAG-scheduling overhead often outweighs concurrency gain.
 
+### Cost-Dominance Principle
+
+Token cost of sub-agent invocation never justifies serialization of independent work. The three safety conditions (read-only or disjoint writes, deterministic aggregation, no shared mutable state) govern WHEN parallelism is safe; cost does not govern WHETHER to parallelize. When in doubt, fan out. Serialization is only valid on true dependency edges.
+
+### Scaling Heuristic
+
+Sub-agent count tracks task decomposition: N independent modules → N parallel Phase-2 implementers; M specialist gates → M parallel Phase-4 specialists; K independent research questions → K parallel researcher modes. Orchestrators emit `sub_agents_spawned: {count, rationale}` in their structured output.
+
 ## Cross-Phase Error Propagation
 
 When a phase produces a non-SUCCESS status, the orchestrator must propagate error context to downstream phases rather than silently dropping it:

package/rules/hatch3r-ai-evals.md

@@ -0,0 +1,158 @@
+---
+id: hatch3r-ai-evals
+type: rule
+description: AI feature evaluation, prompt versioning, cost telemetry, prompt caching, model fallback, and hallucination-as-SLI for end-user projects shipping LLM features
+scope: "**/ai/**,**/llm/**,**/chat/**,**/assistant/**,**/agents/**,**/copilot/**,**/evals/**,**/prompts/**,**/rag/**"
+tags: [review, implementation, ai]
+quality_charter: agents/shared/quality-charter.md
+cache_friendly: true
+---
+# AI Feature Evaluation and Cost Governance (2026)
+
+## Scope
+
+This rule governs the BACKEND half of an LLM-driven feature: eval harness, prompt versioning, cost telemetry, prompt caching, model fallback, hallucination-as-SLI, tool-use evals, safety/red-team, and audit logging. The FRONTEND half (streaming UI, tool-call cards, human-approval gates, cancel/abort/undo, citations) is the paired companion rule `rules/hatch3r-ai-ux-patterns.md`. Apply both rules for any LLM-driven feature — UI-only or backend-only coverage is a regression.
+
+Detection mirrors the companion rule: this rule activates when the project imports an LLM SDK (`openai`, `@anthropic-ai/sdk`, `@google/generative-ai`, `cohere-ai`, `ai`, `@ai-sdk/*`, `langchain`, `llama-index`) or contains files under `ai/`, `llm/`, `chat/`, `assistant/`, `agents/`, `copilot/`, `evals/`, `prompts/`, `rag/`.
+
+## Eval Harness Mandate
+
+Every AI feature has an automated eval harness committed to the repo before the feature ships. Hand-rolled "ask the model and eyeball the answer" is a regression in 2026.
+
+Pick one tool by task class:
+
+- **promptfoo** — broad coverage, declarative YAML, model-comparison defaults
+- **DeepEval** — pytest-style assertions for CI gate integration
+- **RAGAS** — retrieval-augmented generation metrics (context_precision, context_recall, faithfulness, answer_relevance)
+- **Inspect** — UK AISI framework for safety and agentic evals
+- **braintrust** — SaaS + OSS hybrid, run history retained per prompt version
+- **TruLens** — observability-coupled, runs evals against live traces
+- **Arize Phoenix** — open-source observability with eval modules
+
+Document the chosen tool in `evals/README.md` so the agent picks the same tool on every future change.
+
+## Golden Dataset Versioning
+
+- Eval cases live in repo at `evals/<feature>/golden.{jsonl,csv}`.
+- Minimum 20 cases per feature; one input plus one expected output (or graded rubric) per row.
+- Dataset version in the filename (`golden.v3.jsonl`) — never overwrite v2 in place.
+- Ground truth refreshed quarterly; refresh PR includes a diff of changed expected outputs and the rationale.
+- Hand-curated edge cases (failure modes the model has historically tripped on) live in `evals/<feature>/edge.jsonl` alongside the golden set.
+
+## Eval Metrics
+
+Match the metric to the task class:
+
+- Classification → exact-match accuracy + per-class precision/recall.
+- Open-ended generation → rubric-scored LLM-as-judge with a 50-example calibration set sanity-checking judge agreement against human labels; recompute calibration every model change.
+- Retrieval/RAG → RAGAS metrics: context_precision, context_recall, faithfulness, answer_relevance.
+- Pairwise comparison → win-rate against the prior prompt version (>=55% required to adopt).
+- Refusal calibration → refusal-rate as an explicit SLI (refusals on prohibited inputs vs false-positive refusals on benign inputs).
+
+## Regression Gating
+
+- Every PR that touches `**/prompts/**`, `**/rag/**`, `**/ai/**`, `**/llm/**` runs the eval set in CI.
+- PR blocked when any metric drops below the per-feature threshold defined in `evals/<feature>/thresholds.json`.
+- Model-version upgrade (Sonnet to Opus, 4.6 to 4.7) runs the full eval with a 5% accuracy budget; cross over 5% requires a sign-off comment from a named reviewer and a 24-hour canary at 5% traffic.
+- Eval failure is treated as a test failure — never an advisory warning.
+
+## Prompt Versioning
+
+- Prompts are first-class artifacts at `prompts/<feature>/v<N>.{md,txt}` with an SHA-256 hash recorded in the eval thresholds file.
+- Runtime emits `prompt_version` + `prompt_hash` per request as log + span attribute.
+- A/B framework supports concurrent versions behind a flag — traffic split recorded per request so eval delta is computable.
+- Hash drift between repo prompt and deployed prompt is a P0 incident.
+
+## Cost Telemetry per Request
+
+Every LLM call logs: `tokens_in`, `tokens_out`, `cache_hit` (boolean + cached_tokens count), `model`, `cost_usd`, `latency_ms`, `cost_center` (feature ID), `prompt_version`, `prompt_hash`, `user_id_hash`.
+
+Aggregate dashboards in the observability stack — cross-reference `rules/hatch3r-observability-metrics.md` and `rules/hatch3r-observability-tracing-detail.md` for the SLI/SLO vocabulary, and `skills/hatch3r-observability-verify` for the wiring checklist. Per-feature budget alerts fire at 50%, 75%, and 90% of monthly budget; abuse-detection alert at 10x user p99 cost over a 1-hour window.
+
+## Prompt Caching (Anthropic)
+
+- Apply `cache_control` breakpoints to long system prompts, tool definitions, and large RAG context blocks above 1024 tokens (the Claude Opus/Sonnet 4.x minimum cache size; 2048 tokens for Haiku 4.x).
+- Up to 4 breakpoints per request, longest-TTL-first.
+- 5-minute TTL costs 1.25x the standard write rate; 1-hour TTL costs 2x the standard write rate. Reads are 0.1x base.
+- Track cache_hit ratio per feature; <30% hit ratio on a stable prompt is a sign the prefix is changing — investigate before next deploy.
+
+## OpenAI Prompt Caching
+
+- Automatic for requests over 1024 tokens with a stable deterministic prefix; ~50% discount on cached input tokens.
+- Cache duration documented by OpenAI as 5-10 minutes idle, up to 1 hour during off-peak.
+- Order the request prefix deterministically — same system prompt, same tool definitions, same retrieved-doc order — or the cache misses silently.
+
+## Model Router and Fallback
+
+Every LLM call wraps in a retry-with-decorrelated-jitter plus model-fallback chain:
+
+1. **Primary** — production-quality model (e.g. Sonnet 4.7).
+2. **Secondary** — faster/cheaper model (e.g. Haiku 4.5, GPT-5-mini, or a local Llama variant).
+3. **Static fallback** — cached response from a similar prior request or a canned templated reply that names the failure ("Search temporarily unavailable — retry in a minute").
+
+Cross-reference `rules/hatch3r-resilience-patterns.md` (Slice 8) for the circuit-breaker + retry-with-decorrelated-jitter primitives the chain reuses. Each fallback path has its own eval suite — a silent quality cliff between primary and secondary is a regression.
+
+## Hallucination and Groundedness as SLI
+
+Track per feature and treat as service-level indicators with explicit SLO targets:
+
+- `ai.hallucination_rate` — % of responses producing claims not present in retrieved sources. SLO: <5% on the golden set.
+- `ai.citation_precision` — % of citations whose source span verifiably contains the cited claim. SLO: >95% on retrieved claims.
+- `ai.refusal_rate` — overall refusal-rate; track false-positive refusals separately.
+- `ai.groundedness_score` — average RAGAS faithfulness across the golden set. SLO: >0.85.
+
+Cross-reference `rules/hatch3r-observability-metrics.md` for the SLI/SLO authoring template.
+
+## Safety and Red-Team
+
+Every feature exercising user-controlled prompts runs adversarial eval on a schedule (weekly minimum, on every prompt-version bump):
+
+- **Garak** — open-source jailbreak / prompt-injection / PII-leakage probe suite.
+- **PyRIT** — Microsoft red-team framework, scenario-driven.
+- **Inspect-redteam** — UK AISI safety eval modules.
+
+Block release on regression. PII-leakage tests use a synthetic-PII corpus (never production PII). Prompt-injection tests cover OWASP Top 10 for Agentic Applications 2026.
+
+## Tool-Use Evals
+
+When the AI feature uses tools (per the companion UX rule), the eval suite covers:
+
+- Tool-selection accuracy — correct tool chosen for each input class.
+- Argument validity — emitted args satisfy the tool schema (Zod, JSON Schema, Pydantic).
+- Chain correctness — multi-tool plans reach the goal with the minimum required step count plus a 20% tolerance.
+- Latency budget — p99 tool-execution time within the budget from `rules/hatch3r-performance-budgets.md`.
+
+Methodology aligned with **BFCL v4** (Berkeley Function Calling Leaderboard) and **tau-bench** (multi-turn tool-use benchmark).
+
+## OpenTelemetry GenAI Semantic Conventions
+
+Every LLM call emits an OpenTelemetry span named `gen_ai.<operation>` with the attributes prescribed by the OpenTelemetry GenAI semantic conventions: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cached_tokens`, `gen_ai.request.temperature`, `gen_ai.tool.name` (when tools used). Cross-reference Slice 2 observability rules for the broader span taxonomy.
+
+## User-Feedback Loop
+
+- Every AI response surface emits a thumbs-up/down control wired to a feedback queue.
+- A monthly triage job promotes thumbs-down examples into regression-test fixtures in `evals/<feature>/edge.jsonl`.
+- Promotion is a manual review step — never auto-promote raw user feedback (it contains noise and adversarial labels).
+
+## Audit Logging
+
+- LLM inputs + outputs logged to a compliance store separate from APM with a 30-90 day retention window (configurable per data-classification policy).
+- PII redaction before persistence — same redaction primitive as the framework's data-classification pipeline.
+- Reproducibility key: `model` + `prompt_hash` + `seed` (when the model exposes a seed parameter) + `temperature`. Without all four, the response is non-reproducible.
+
+## Eval-Driven Development Workflow
+
+Write eval before prompt, measure baseline, write prompt, measure delta, iterate until eval threshold passes. Cross-reference `skills/hatch3r-ai-feature/SKILL.md` for the step-by-step workflow.
+
+## References
+
+- promptfoo — `promptfoo.dev`
+- DeepEval — `github.com/confident-ai/deepeval`
+- RAGAS — `docs.ragas.io`
+- Inspect (UK AISI) — `github.com/UKGovernmentBEIS/inspect_ai`
+- Anthropic prompt caching guide — `docs.anthropic.com/en/docs/build-with-claude/prompt-caching`
+- OpenAI prompt caching guide — `platform.openai.com/docs/guides/prompt-caching`
+- OpenTelemetry GenAI semantic conventions — `opentelemetry.io/docs/specs/semconv/gen-ai/`
+- Berkeley Function Calling Leaderboard (BFCL v4) — `gorilla.cs.berkeley.edu/leaderboard.html`
+- tau-bench — `github.com/sierra-research/tau-bench`
+- OWASP Top 10 for LLM Applications (Agentic 2026) — `genai.owasp.org`

package/rules/hatch3r-ai-evals.mdc

@@ -0,0 +1,154 @@
+---
+description: AI feature evaluation, prompt versioning, cost telemetry, prompt caching, model fallback, and hallucination-as-SLI for end-user projects shipping LLM features
+globs: ["**/ai/**", "**/llm/**", "**/chat/**", "**/assistant/**", "**/agents/**", "**/copilot/**", "**/evals/**", "**/prompts/**", "**/rag/**"]
+alwaysApply: false
+---
+# AI Feature Evaluation and Cost Governance (2026)
+
+## Scope
+
+This rule governs the BACKEND half of an LLM-driven feature: eval harness, prompt versioning, cost telemetry, prompt caching, model fallback, hallucination-as-SLI, tool-use evals, safety/red-team, and audit logging. The FRONTEND half (streaming UI, tool-call cards, human-approval gates, cancel/abort/undo, citations) is the paired companion rule `rules/hatch3r-ai-ux-patterns.md`. Apply both rules for any LLM-driven feature — UI-only or backend-only coverage is a regression.
+
+Detection mirrors the companion rule: this rule activates when the project imports an LLM SDK (`openai`, `@anthropic-ai/sdk`, `@google/generative-ai`, `cohere-ai`, `ai`, `@ai-sdk/*`, `langchain`, `llama-index`) or contains files under `ai/`, `llm/`, `chat/`, `assistant/`, `agents/`, `copilot/`, `evals/`, `prompts/`, `rag/`.
+
+## Eval Harness Mandate
+
+Every AI feature has an automated eval harness committed to the repo before the feature ships. Hand-rolled "ask the model and eyeball the answer" is a regression in 2026.
+
+Pick one tool by task class:
+
+- **promptfoo** — broad coverage, declarative YAML, model-comparison defaults
+- **DeepEval** — pytest-style assertions for CI gate integration
+- **RAGAS** — retrieval-augmented generation metrics (context_precision, context_recall, faithfulness, answer_relevance)
+- **Inspect** — UK AISI framework for safety and agentic evals
+- **braintrust** — SaaS + OSS hybrid, run history retained per prompt version
+- **TruLens** — observability-coupled, runs evals against live traces
+- **Arize Phoenix** — open-source observability with eval modules
+
+Document the chosen tool in `evals/README.md` so the agent picks the same tool on every future change.
+
+## Golden Dataset Versioning
+
+- Eval cases live in repo at `evals/<feature>/golden.{jsonl,csv}`.
+- Minimum 20 cases per feature; one input plus one expected output (or graded rubric) per row.
+- Dataset version in the filename (`golden.v3.jsonl`) — never overwrite v2 in place.
+- Ground truth refreshed quarterly; refresh PR includes a diff of changed expected outputs and the rationale.
+- Hand-curated edge cases (failure modes the model has historically tripped on) live in `evals/<feature>/edge.jsonl` alongside the golden set.
+
+## Eval Metrics
+
+Match the metric to the task class:
+
+- Classification → exact-match accuracy + per-class precision/recall.
+- Open-ended generation → rubric-scored LLM-as-judge with a 50-example calibration set sanity-checking judge agreement against human labels; recompute calibration every model change.
+- Retrieval/RAG → RAGAS metrics: context_precision, context_recall, faithfulness, answer_relevance.
+- Pairwise comparison → win-rate against the prior prompt version (>=55% required to adopt).
+- Refusal calibration → refusal-rate as an explicit SLI (refusals on prohibited inputs vs false-positive refusals on benign inputs).
+
+## Regression Gating
+
+- Every PR that touches `**/prompts/**`, `**/rag/**`, `**/ai/**`, `**/llm/**` runs the eval set in CI.
+- PR blocked when any metric drops below the per-feature threshold defined in `evals/<feature>/thresholds.json`.
+- Model-version upgrade (Sonnet to Opus, 4.6 to 4.7) runs the full eval with a 5% accuracy budget; cross over 5% requires a sign-off comment from a named reviewer and a 24-hour canary at 5% traffic.
+- Eval failure is treated as a test failure — never an advisory warning.
+
+## Prompt Versioning
+
+- Prompts are first-class artifacts at `prompts/<feature>/v<N>.{md,txt}` with an SHA-256 hash recorded in the eval thresholds file.
+- Runtime emits `prompt_version` + `prompt_hash` per request as log + span attribute.
+- A/B framework supports concurrent versions behind a flag — traffic split recorded per request so eval delta is computable.
+- Hash drift between repo prompt and deployed prompt is a P0 incident.
+
+## Cost Telemetry per Request
+
+Every LLM call logs: `tokens_in`, `tokens_out`, `cache_hit` (boolean + cached_tokens count), `model`, `cost_usd`, `latency_ms`, `cost_center` (feature ID), `prompt_version`, `prompt_hash`, `user_id_hash`.
+
+Aggregate dashboards in the observability stack — cross-reference `rules/hatch3r-observability-metrics.md` and `rules/hatch3r-observability-tracing-detail.md` for the SLI/SLO vocabulary, and `skills/hatch3r-observability-verify` for the wiring checklist. Per-feature budget alerts fire at 50%, 75%, and 90% of monthly budget; abuse-detection alert at 10x user p99 cost over a 1-hour window.
+
+## Prompt Caching (Anthropic)
+
+- Apply `cache_control` breakpoints to long system prompts, tool definitions, and large RAG context blocks above 1024 tokens (the Claude Opus/Sonnet 4.x minimum cache size; 2048 tokens for Haiku 4.x).
+- Up to 4 breakpoints per request, longest-TTL-first.
+- 5-minute TTL costs 1.25x the standard write rate; 1-hour TTL costs 2x the standard write rate. Reads are 0.1x base.
+- Track cache_hit ratio per feature; <30% hit ratio on a stable prompt is a sign the prefix is changing — investigate before next deploy.
+
+## OpenAI Prompt Caching
+
+- Automatic for requests over 1024 tokens with a stable deterministic prefix; ~50% discount on cached input tokens.
+- Cache duration documented by OpenAI as 5-10 minutes idle, up to 1 hour during off-peak.
+- Order the request prefix deterministically — same system prompt, same tool definitions, same retrieved-doc order — or the cache misses silently.
+
+## Model Router and Fallback
+
+Every LLM call wraps in a retry-with-decorrelated-jitter plus model-fallback chain:
+
+1. **Primary** — production-quality model (e.g. Sonnet 4.7).
+2. **Secondary** — faster/cheaper model (e.g. Haiku 4.5, GPT-5-mini, or a local Llama variant).
+3. **Static fallback** — cached response from a similar prior request or a canned templated reply that names the failure ("Search temporarily unavailable — retry in a minute").
+
+Cross-reference `rules/hatch3r-resilience-patterns.md` (Slice 8) for the circuit-breaker + retry-with-decorrelated-jitter primitives the chain reuses. Each fallback path has its own eval suite — a silent quality cliff between primary and secondary is a regression.
+
+## Hallucination and Groundedness as SLI
+
+Track per feature and treat as service-level indicators with explicit SLO targets:
+
+- `ai.hallucination_rate` — % of responses producing claims not present in retrieved sources. SLO: <5% on the golden set.
+- `ai.citation_precision` — % of citations whose source span verifiably contains the cited claim. SLO: >95% on retrieved claims.
+- `ai.refusal_rate` — overall refusal-rate; track false-positive refusals separately.
+- `ai.groundedness_score` — average RAGAS faithfulness across the golden set. SLO: >0.85.
+
+Cross-reference `rules/hatch3r-observability-metrics.md` for the SLI/SLO authoring template.
+
+## Safety and Red-Team
+
+Every feature exercising user-controlled prompts runs adversarial eval on a schedule (weekly minimum, on every prompt-version bump):
+
+- **Garak** — open-source jailbreak / prompt-injection / PII-leakage probe suite.
+- **PyRIT** — Microsoft red-team framework, scenario-driven.
+- **Inspect-redteam** — UK AISI safety eval modules.
+
+Block release on regression. PII-leakage tests use a synthetic-PII corpus (never production PII). Prompt-injection tests cover OWASP Top 10 for Agentic Applications 2026.
+
+## Tool-Use Evals
+
+When the AI feature uses tools (per the companion UX rule), the eval suite covers:
+
+- Tool-selection accuracy — correct tool chosen for each input class.
+- Argument validity — emitted args satisfy the tool schema (Zod, JSON Schema, Pydantic).
+- Chain correctness — multi-tool plans reach the goal with the minimum required step count plus a 20% tolerance.
+- Latency budget — p99 tool-execution time within the budget from `rules/hatch3r-performance-budgets.md`.
+
+Methodology aligned with **BFCL v4** (Berkeley Function Calling Leaderboard) and **tau-bench** (multi-turn tool-use benchmark).
+
+## OpenTelemetry GenAI Semantic Conventions
+
+Every LLM call emits an OpenTelemetry span named `gen_ai.<operation>` with the attributes prescribed by the OpenTelemetry GenAI semantic conventions: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cached_tokens`, `gen_ai.request.temperature`, `gen_ai.tool.name` (when tools used). Cross-reference Slice 2 observability rules for the broader span taxonomy.
+
+## User-Feedback Loop
+
+- Every AI response surface emits a thumbs-up/down control wired to a feedback queue.
+- A monthly triage job promotes thumbs-down examples into regression-test fixtures in `evals/<feature>/edge.jsonl`.
+- Promotion is a manual review step — never auto-promote raw user feedback (it contains noise and adversarial labels).
+
+## Audit Logging
+
+- LLM inputs + outputs logged to a compliance store separate from APM with a 30-90 day retention window (configurable per data-classification policy).
+- PII redaction before persistence — same redaction primitive as the framework's data-classification pipeline.
+- Reproducibility key: `model` + `prompt_hash` + `seed` (when the model exposes a seed parameter) + `temperature`. Without all four, the response is non-reproducible.
+
+## Eval-Driven Development Workflow
+
+Write eval before prompt, measure baseline, write prompt, measure delta, iterate until eval threshold passes. Cross-reference `skills/hatch3r-ai-feature/SKILL.md` for the step-by-step workflow.
+
+## References
+
+- promptfoo — `promptfoo.dev`
+- DeepEval — `github.com/confident-ai/deepeval`
+- RAGAS — `docs.ragas.io`
+- Inspect (UK AISI) — `github.com/UKGovernmentBEIS/inspect_ai`
+- Anthropic prompt caching guide — `docs.anthropic.com/en/docs/build-with-claude/prompt-caching`
+- OpenAI prompt caching guide — `platform.openai.com/docs/guides/prompt-caching`
+- OpenTelemetry GenAI semantic conventions — `opentelemetry.io/docs/specs/semconv/gen-ai/`
+- Berkeley Function Calling Leaderboard (BFCL v4) — `gorilla.cs.berkeley.edu/leaderboard.html`
+- tau-bench — `github.com/sierra-research/tau-bench`
+- OWASP Top 10 for LLM Applications (Agentic 2026) — `genai.owasp.org`