hatch3r 1.7.1 → 1.7.5

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hatch3r",
-  "version": "1.7.1",
+  "version": "1.7.5",
   "description": "Battle-tested agentic coding setup framework. One command to hatch your agent stack -- agents, skills, rules, commands, and MCP for every major AI coding tool.",
   "type": "module",
   "exports": {
@@ -24,7 +24,9 @@
     "inventory:check-docs": "tsx scripts/inventory.ts --check-docs",
     "validate:rule-parity": "tsx scripts/validate-rule-parity.ts",
     "validate:efficiency": "tsx scripts/validate-efficiency-invariants.ts",
-    "validate": "npm run validate:rule-parity && npm run validate:efficiency",
+    "validate:cli-skills": "tsx scripts/validate-cli-skills.ts",
+    "generate:cli-skills": "tsx scripts/generate-cli-skills.ts",
+    "validate": "npm run validate:rule-parity && npm run validate:efficiency && npm run validate:cli-skills",
     "audit:validate-registry": "tsx scripts/validate-finding-registry.ts",
     "audit:migrate": "tsx scripts/migrate-finding-registry.ts",
     "audit:archive": "tsx scripts/audit-archive.ts",

package/rules/hatch3r-accessibility-standards.md

@@ -78,3 +78,24 @@ All user-facing features must meet WCAG 2.2 Level AA conformance. This is the ba
 - Run automated accessibility checks (axe-core, Lighthouse) in CI.
 - Automated tools catch ~30% of accessibility issues. Manual testing is required for keyboard flows, screen reader experience, and cognitive accessibility.
 - Maintain an accessibility test checklist per component type (form, modal, navigation, data table).
+
+## WCAG 2.2 New Success Criteria (Mandatory Audit Items)
+
+WCAG 2.2 (W3C Recommendation, October 2023) added nine Success Criteria; three apply to most UI surfaces and are listed here as required audit items.
+
+- **SC 2.5.8 Target Size (Minimum) — AA:** every pointer-target's hit area is at least 24 by 24 CSS pixels. Smaller targets are permitted only when surrounded by ≥24px of spacing (the spacing exception) or when the larger inline target is also reachable (the inline exception). Test densest UI elements (sidebar collapse, table-row checkboxes, icon-only toolbar buttons).
+- **SC 2.4.11 Focus Not Obscured (Minimum) — AA:** when a control receives keyboard focus, the focused element must not be entirely hidden by author-created content. Common violations: sticky headers, persistent cookie banners, chatbot launcher widgets, fixed action bars. Mitigate with `scroll-margin-top` on focus targets equal to the sticky header height, or by collapsing sticky chrome on focus.
+- **SC 2.5.7 Dragging Movements — AA:** any drag operation (reorder, slider thumb drag, map pan, kanban card move) must offer a single-pointer non-drag alternative. Examples: list reorder with up/down arrow buttons; slider with numeric input or +/- buttons; map pan with arrow-key navigation.
+
+Reference: https://www.w3.org/TR/WCAG22/
+
+## Mobile and Touch Accessibility
+
+Touch surfaces have stricter target and spacing requirements than pointer-only surfaces; native platform guidance supersedes WCAG 2.5.8 on touch.
+
+- Touch targets: 44 by 44 points on iOS (Apple Human Interface Guidelines), 48 by 48 dp on Android (Material Design 3). These supersede WCAG 2.5.8's 24-pixel minimum on touch-only surfaces.
+- Spacing between interactive elements: ≥8 pixels of separation to prevent mis-taps; ≥4 mm on physical-density screens.
+- Avoid tap-to-reveal patterns (hover tooltips, pure-hover dropdowns) — touch has no hover state. Replace with permanent visible labels or with long-press that has a visible affordance and announces itself to screen readers.
+- Apply `env(safe-area-inset-*)` padding on full-bleed surfaces so content clears notches, home indicators, and rounded corners on iOS and Android edge devices.
+- Support Dynamic Type (iOS) and rem-based font scaling — declare body text in `rem` or `em` units, never `px`, so OS-level font size settings cascade.
+- Zoom to 200% and 400% (per WCAG 1.4.4 and 1.4.10 Reflow) must remain functional with no horizontal scroll trap. Audit for `width: 100vw` and fixed pixel widths that break reflow.

package/rules/hatch3r-accessibility-standards.mdc

@@ -74,3 +74,24 @@ All user-facing features must meet WCAG 2.2 Level AA conformance. This is the ba
 - Run automated accessibility checks (axe-core, Lighthouse) in CI.
 - Automated tools catch ~30% of accessibility issues. Manual testing is required for keyboard flows, screen reader experience, and cognitive accessibility.
 - Maintain an accessibility test checklist per component type (form, modal, navigation, data table).
+
+## WCAG 2.2 New Success Criteria (Mandatory Audit Items)
+
+WCAG 2.2 (W3C Recommendation, October 2023) added nine Success Criteria; three apply to most UI surfaces and are listed here as required audit items.
+
+- **SC 2.5.8 Target Size (Minimum) — AA:** every pointer-target's hit area is at least 24 by 24 CSS pixels. Smaller targets are permitted only when surrounded by ≥24px of spacing (the spacing exception) or when the larger inline target is also reachable (the inline exception). Test densest UI elements (sidebar collapse, table-row checkboxes, icon-only toolbar buttons).
+- **SC 2.4.11 Focus Not Obscured (Minimum) — AA:** when a control receives keyboard focus, the focused element must not be entirely hidden by author-created content. Common violations: sticky headers, persistent cookie banners, chatbot launcher widgets, fixed action bars. Mitigate with `scroll-margin-top` on focus targets equal to the sticky header height, or by collapsing sticky chrome on focus.
+- **SC 2.5.7 Dragging Movements — AA:** any drag operation (reorder, slider thumb drag, map pan, kanban card move) must offer a single-pointer non-drag alternative. Examples: list reorder with up/down arrow buttons; slider with numeric input or +/- buttons; map pan with arrow-key navigation.
+
+Reference: https://www.w3.org/TR/WCAG22/
+
+## Mobile and Touch Accessibility
+
+Touch surfaces have stricter target and spacing requirements than pointer-only surfaces; native platform guidance supersedes WCAG 2.5.8 on touch.
+
+- Touch targets: 44 by 44 points on iOS (Apple Human Interface Guidelines), 48 by 48 dp on Android (Material Design 3). These supersede WCAG 2.5.8's 24-pixel minimum on touch-only surfaces.
+- Spacing between interactive elements: ≥8 pixels of separation to prevent mis-taps; ≥4 mm on physical-density screens.
+- Avoid tap-to-reveal patterns (hover tooltips, pure-hover dropdowns) — touch has no hover state. Replace with permanent visible labels or with long-press that has a visible affordance and announces itself to screen readers.
+- Apply `env(safe-area-inset-*)` padding on full-bleed surfaces so content clears notches, home indicators, and rounded corners on iOS and Android edge devices.
+- Support Dynamic Type (iOS) and rem-based font scaling — declare body text in `rem` or `em` units, never `px`, so OS-level font size settings cascade.
+- Zoom to 200% and 400% (per WCAG 1.4.4 and 1.4.10 Reflow) must remain functional with no horizontal scroll trap. Audit for `width: 100vw` and fixed pixel widths that break reflow.

package/rules/hatch3r-agent-orchestration.md

@@ -48,7 +48,7 @@ Every task MUST follow this four-phase pipeline: **Phase 1 — Research** (`hatc
 
 Score task complexity per the `hatch3r-deep-context` rule before Phase 1. Apply the resulting tier:
 
-- **Tier 2 (Standard):** Present elicitation questions inline. Await answers before Phase 2.
+- **Tier 2 hard gate (B1).** Before Phase 2, run `hatch3r-researcher` with `requirements-elicitation:quick` mode to detect ambiguity and ask the user via the platform-native question tool (per `agents/shared/user-question-protocol.md`). Orchestrator awaits answers and integrates them into the Phase 1 brief; do not begin Phase 2 with unresolved questions. Tier 1 is exempt only when scope is single-file, single-concern, and acceptance is testable from the user message alone.
 - **Tier 3 (Deep):** Present Pre-Implementation Summary and ASK for confirmation. Do NOT proceed until all unresolved questions are answered.
 
 ## Mandatory Delegation Directives
@@ -253,6 +253,14 @@ ALL three must hold:
 
 **Default:** When in doubt, serialize. For typical hatch3r tasks (1–5 sub-tasks) the DAG-scheduling overhead often outweighs concurrency gain.
 
+### Cost-Dominance Principle
+
+Token cost of sub-agent invocation never justifies serialization of independent work. The three safety conditions (read-only or disjoint writes, deterministic aggregation, no shared mutable state) govern WHEN parallelism is safe; cost does not govern WHETHER to parallelize. When in doubt, fan out. Serialization is only valid on true dependency edges.
+
+### Scaling Heuristic
+
+Sub-agent count tracks task decomposition: N independent modules → N parallel Phase-2 implementers; M specialist gates → M parallel Phase-4 specialists; K independent research questions → K parallel researcher modes. Orchestrators emit `sub_agents_spawned: {count, rationale}` in their structured output.
+
 ## Cross-Phase Error Propagation
 
 When a phase produces a non-SUCCESS status, the orchestrator must propagate error context to downstream phases rather than silently dropping it:

package/rules/hatch3r-agent-orchestration.mdc

@@ -43,7 +43,7 @@ Every task MUST follow this four-phase pipeline: **Phase 1 — Research** (`hatc
 
 Score task complexity per the `hatch3r-deep-context` rule before Phase 1. Apply the resulting tier:
 
-- **Tier 2 (Standard):** Present elicitation questions inline. Await answers before Phase 2.
+- **Tier 2 hard gate (B1).** Before Phase 2, run `hatch3r-researcher` with `requirements-elicitation:quick` mode to detect ambiguity and ask the user via the platform-native question tool (per `agents/shared/user-question-protocol.md`). Orchestrator awaits answers and integrates them into the Phase 1 brief; do not begin Phase 2 with unresolved questions. Tier 1 is exempt only when scope is single-file, single-concern, and acceptance is testable from the user message alone.
 - **Tier 3 (Deep):** Present Pre-Implementation Summary and ASK for confirmation. Do NOT proceed until all unresolved questions are answered.
 
 ## Mandatory Delegation Directives
@@ -248,6 +248,14 @@ ALL three must hold:
 
 **Default:** When in doubt, serialize. For typical hatch3r tasks (1–5 sub-tasks) the DAG-scheduling overhead often outweighs concurrency gain.
 
+### Cost-Dominance Principle
+
+Token cost of sub-agent invocation never justifies serialization of independent work. The three safety conditions (read-only or disjoint writes, deterministic aggregation, no shared mutable state) govern WHEN parallelism is safe; cost does not govern WHETHER to parallelize. When in doubt, fan out. Serialization is only valid on true dependency edges.
+
+### Scaling Heuristic
+
+Sub-agent count tracks task decomposition: N independent modules → N parallel Phase-2 implementers; M specialist gates → M parallel Phase-4 specialists; K independent research questions → K parallel researcher modes. Orchestrators emit `sub_agents_spawned: {count, rationale}` in their structured output.
+
 ## Cross-Phase Error Propagation
 
 When a phase produces a non-SUCCESS status, the orchestrator must propagate error context to downstream phases rather than silently dropping it:

package/rules/hatch3r-ai-evals.md

@@ -0,0 +1,158 @@
+---
+id: hatch3r-ai-evals
+type: rule
+description: AI feature evaluation, prompt versioning, cost telemetry, prompt caching, model fallback, and hallucination-as-SLI for end-user projects shipping LLM features
+scope: "**/ai/**,**/llm/**,**/chat/**,**/assistant/**,**/agents/**,**/copilot/**,**/evals/**,**/prompts/**,**/rag/**"
+tags: [review, implementation, ai]
+quality_charter: agents/shared/quality-charter.md
+cache_friendly: true
+---
+# AI Feature Evaluation and Cost Governance (2026)
+
+## Scope
+
+This rule governs the BACKEND half of an LLM-driven feature: eval harness, prompt versioning, cost telemetry, prompt caching, model fallback, hallucination-as-SLI, tool-use evals, safety/red-team, and audit logging. The FRONTEND half (streaming UI, tool-call cards, human-approval gates, cancel/abort/undo, citations) is the paired companion rule `rules/hatch3r-ai-ux-patterns.md`. Apply both rules for any LLM-driven feature — UI-only or backend-only coverage is a regression.
+
+Detection mirrors the companion rule: this rule activates when the project imports an LLM SDK (`openai`, `@anthropic-ai/sdk`, `@google/generative-ai`, `cohere-ai`, `ai`, `@ai-sdk/*`, `langchain`, `llama-index`) or contains files under `ai/`, `llm/`, `chat/`, `assistant/`, `agents/`, `copilot/`, `evals/`, `prompts/`, `rag/`.
+
+## Eval Harness Mandate
+
+Every AI feature has an automated eval harness committed to the repo before the feature ships. Hand-rolled "ask the model and eyeball the answer" is a regression in 2026.
+
+Pick one tool by task class:
+
+- **promptfoo** — broad coverage, declarative YAML, model-comparison defaults
+- **DeepEval** — pytest-style assertions for CI gate integration
+- **RAGAS** — retrieval-augmented generation metrics (context_precision, context_recall, faithfulness, answer_relevance)
+- **Inspect** — UK AISI framework for safety and agentic evals
+- **braintrust** — SaaS + OSS hybrid, run history retained per prompt version
+- **TruLens** — observability-coupled, runs evals against live traces
+- **Arize Phoenix** — open-source observability with eval modules
+
+Document the chosen tool in `evals/README.md` so the agent picks the same tool on every future change.
+
+## Golden Dataset Versioning
+
+- Eval cases live in repo at `evals/<feature>/golden.{jsonl,csv}`.
+- Minimum 20 cases per feature; one input plus one expected output (or graded rubric) per row.
+- Dataset version in the filename (`golden.v3.jsonl`) — never overwrite v2 in place.
+- Ground truth refreshed quarterly; refresh PR includes a diff of changed expected outputs and the rationale.
+- Hand-curated edge cases (failure modes the model has historically tripped on) live in `evals/<feature>/edge.jsonl` alongside the golden set.
+
+## Eval Metrics
+
+Match the metric to the task class:
+
+- Classification → exact-match accuracy + per-class precision/recall.
+- Open-ended generation → rubric-scored LLM-as-judge with a 50-example calibration set sanity-checking judge agreement against human labels; recompute calibration every model change.
+- Retrieval/RAG → RAGAS metrics: context_precision, context_recall, faithfulness, answer_relevance.
+- Pairwise comparison → win-rate against the prior prompt version (>=55% required to adopt).
+- Refusal calibration → refusal-rate as an explicit SLI (refusals on prohibited inputs vs false-positive refusals on benign inputs).
+
+## Regression Gating
+
+- Every PR that touches `**/prompts/**`, `**/rag/**`, `**/ai/**`, `**/llm/**` runs the eval set in CI.
+- PR blocked when any metric drops below the per-feature threshold defined in `evals/<feature>/thresholds.json`.
+- Model-version upgrade (Sonnet to Opus, 4.6 to 4.7) runs the full eval with a 5% accuracy budget; cross over 5% requires a sign-off comment from a named reviewer and a 24-hour canary at 5% traffic.
+- Eval failure is treated as a test failure — never an advisory warning.
+
+## Prompt Versioning
+
+- Prompts are first-class artifacts at `prompts/<feature>/v<N>.{md,txt}` with an SHA-256 hash recorded in the eval thresholds file.
+- Runtime emits `prompt_version` + `prompt_hash` per request as log + span attribute.
+- A/B framework supports concurrent versions behind a flag — traffic split recorded per request so eval delta is computable.
+- Hash drift between repo prompt and deployed prompt is a P0 incident.
+
+## Cost Telemetry per Request
+
+Every LLM call logs: `tokens_in`, `tokens_out`, `cache_hit` (boolean + cached_tokens count), `model`, `cost_usd`, `latency_ms`, `cost_center` (feature ID), `prompt_version`, `prompt_hash`, `user_id_hash`.
+
+Aggregate dashboards in the observability stack — cross-reference `rules/hatch3r-observability-metrics.md` and `rules/hatch3r-observability-tracing-detail.md` for the SLI/SLO vocabulary, and `skills/hatch3r-observability-verify` for the wiring checklist. Per-feature budget alerts fire at 50%, 75%, and 90% of monthly budget; abuse-detection alert at 10x user p99 cost over a 1-hour window.
+
+## Prompt Caching (Anthropic)
+
+- Apply `cache_control` breakpoints to long system prompts, tool definitions, and large RAG context blocks above 1024 tokens (the Claude Opus/Sonnet 4.x minimum cache size; 2048 tokens for Haiku 4.x).
+- Up to 4 breakpoints per request, longest-TTL-first.
+- 5-minute TTL costs 1.25x the standard write rate; 1-hour TTL costs 2x the standard write rate. Reads are 0.1x base.
+- Track cache_hit ratio per feature; <30% hit ratio on a stable prompt is a sign the prefix is changing — investigate before next deploy.
+
+## OpenAI Prompt Caching
+
+- Automatic for requests over 1024 tokens with a stable deterministic prefix; ~50% discount on cached input tokens.
+- Cache duration documented by OpenAI as 5-10 minutes idle, up to 1 hour during off-peak.
+- Order the request prefix deterministically — same system prompt, same tool definitions, same retrieved-doc order — or the cache misses silently.
+
+## Model Router and Fallback
+
+Every LLM call wraps in a retry-with-decorrelated-jitter plus model-fallback chain:
+
+1. **Primary** — production-quality model (e.g. Sonnet 4.7).
+2. **Secondary** — faster/cheaper model (e.g. Haiku 4.5, GPT-5-mini, or a local Llama variant).
+3. **Static fallback** — cached response from a similar prior request or a canned templated reply that names the failure ("Search temporarily unavailable — retry in a minute").
+
+Cross-reference `rules/hatch3r-resilience-patterns.md` (Slice 8) for the circuit-breaker + retry-with-decorrelated-jitter primitives the chain reuses. Each fallback path has its own eval suite — a silent quality cliff between primary and secondary is a regression.
+
+## Hallucination and Groundedness as SLI
+
+Track per feature and treat as service-level indicators with explicit SLO targets:
+
+- `ai.hallucination_rate` — % of responses producing claims not present in retrieved sources. SLO: <5% on the golden set.
+- `ai.citation_precision` — % of citations whose source span verifiably contains the cited claim. SLO: >95% on retrieved claims.
+- `ai.refusal_rate` — overall refusal-rate; track false-positive refusals separately.
+- `ai.groundedness_score` — average RAGAS faithfulness across the golden set. SLO: >0.85.
+
+Cross-reference `rules/hatch3r-observability-metrics.md` for the SLI/SLO authoring template.
+
+## Safety and Red-Team
+
+Every feature exercising user-controlled prompts runs adversarial eval on a schedule (weekly minimum, on every prompt-version bump):
+
+- **Garak** — open-source jailbreak / prompt-injection / PII-leakage probe suite.
+- **PyRIT** — Microsoft red-team framework, scenario-driven.
+- **Inspect-redteam** — UK AISI safety eval modules.
+
+Block release on regression. PII-leakage tests use a synthetic-PII corpus (never production PII). Prompt-injection tests cover OWASP Top 10 for Agentic Applications 2026.
+
+## Tool-Use Evals
+
+When the AI feature uses tools (per the companion UX rule), the eval suite covers:
+
+- Tool-selection accuracy — correct tool chosen for each input class.
+- Argument validity — emitted args satisfy the tool schema (Zod, JSON Schema, Pydantic).
+- Chain correctness — multi-tool plans reach the goal with the minimum required step count plus a 20% tolerance.
+- Latency budget — p99 tool-execution time within the budget from `rules/hatch3r-performance-budgets.md`.
+
+Methodology aligned with **BFCL v4** (Berkeley Function Calling Leaderboard) and **tau-bench** (multi-turn tool-use benchmark).
+
+## OpenTelemetry GenAI Semantic Conventions
+
+Every LLM call emits an OpenTelemetry span named `gen_ai.<operation>` with the attributes prescribed by the OpenTelemetry GenAI semantic conventions: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cached_tokens`, `gen_ai.request.temperature`, `gen_ai.tool.name` (when tools used). Cross-reference Slice 2 observability rules for the broader span taxonomy.
+
+## User-Feedback Loop
+
+- Every AI response surface emits a thumbs-up/down control wired to a feedback queue.
+- A monthly triage job promotes thumbs-down examples into regression-test fixtures in `evals/<feature>/edge.jsonl`.
+- Promotion is a manual review step — never auto-promote raw user feedback (it contains noise and adversarial labels).
+
+## Audit Logging
+
+- LLM inputs + outputs logged to a compliance store separate from APM with a 30-90 day retention window (configurable per data-classification policy).
+- PII redaction before persistence — same redaction primitive as the framework's data-classification pipeline.
+- Reproducibility key: `model` + `prompt_hash` + `seed` (when the model exposes a seed parameter) + `temperature`. Without all four, the response is non-reproducible.
+
+## Eval-Driven Development Workflow
+
+Write eval before prompt, measure baseline, write prompt, measure delta, iterate until eval threshold passes. Cross-reference `skills/hatch3r-ai-feature/SKILL.md` for the step-by-step workflow.
+
+## References
+
+- promptfoo — `promptfoo.dev`
+- DeepEval — `github.com/confident-ai/deepeval`
+- RAGAS — `docs.ragas.io`
+- Inspect (UK AISI) — `github.com/UKGovernmentBEIS/inspect_ai`
+- Anthropic prompt caching guide — `docs.anthropic.com/en/docs/build-with-claude/prompt-caching`
+- OpenAI prompt caching guide — `platform.openai.com/docs/guides/prompt-caching`
+- OpenTelemetry GenAI semantic conventions — `opentelemetry.io/docs/specs/semconv/gen-ai/`
+- Berkeley Function Calling Leaderboard (BFCL v4) — `gorilla.cs.berkeley.edu/leaderboard.html`
+- tau-bench — `github.com/sierra-research/tau-bench`
+- OWASP Top 10 for LLM Applications (Agentic 2026) — `genai.owasp.org`

package/rules/hatch3r-ai-evals.mdc

@@ -0,0 +1,154 @@
+---
+description: AI feature evaluation, prompt versioning, cost telemetry, prompt caching, model fallback, and hallucination-as-SLI for end-user projects shipping LLM features
+globs: ["**/ai/**", "**/llm/**", "**/chat/**", "**/assistant/**", "**/agents/**", "**/copilot/**", "**/evals/**", "**/prompts/**", "**/rag/**"]
+alwaysApply: false
+---
+# AI Feature Evaluation and Cost Governance (2026)
+
+## Scope
+
+This rule governs the BACKEND half of an LLM-driven feature: eval harness, prompt versioning, cost telemetry, prompt caching, model fallback, hallucination-as-SLI, tool-use evals, safety/red-team, and audit logging. The FRONTEND half (streaming UI, tool-call cards, human-approval gates, cancel/abort/undo, citations) is the paired companion rule `rules/hatch3r-ai-ux-patterns.md`. Apply both rules for any LLM-driven feature — UI-only or backend-only coverage is a regression.
+
+Detection mirrors the companion rule: this rule activates when the project imports an LLM SDK (`openai`, `@anthropic-ai/sdk`, `@google/generative-ai`, `cohere-ai`, `ai`, `@ai-sdk/*`, `langchain`, `llama-index`) or contains files under `ai/`, `llm/`, `chat/`, `assistant/`, `agents/`, `copilot/`, `evals/`, `prompts/`, `rag/`.
+
+## Eval Harness Mandate
+
+Every AI feature has an automated eval harness committed to the repo before the feature ships. Hand-rolled "ask the model and eyeball the answer" is a regression in 2026.
+
+Pick one tool by task class:
+
+- **promptfoo** — broad coverage, declarative YAML, model-comparison defaults
+- **DeepEval** — pytest-style assertions for CI gate integration
+- **RAGAS** — retrieval-augmented generation metrics (context_precision, context_recall, faithfulness, answer_relevance)
+- **Inspect** — UK AISI framework for safety and agentic evals
+- **braintrust** — SaaS + OSS hybrid, run history retained per prompt version
+- **TruLens** — observability-coupled, runs evals against live traces
+- **Arize Phoenix** — open-source observability with eval modules
+
+Document the chosen tool in `evals/README.md` so the agent picks the same tool on every future change.
+
+## Golden Dataset Versioning
+
+- Eval cases live in repo at `evals/<feature>/golden.{jsonl,csv}`.
+- Minimum 20 cases per feature; one input plus one expected output (or graded rubric) per row.
+- Dataset version in the filename (`golden.v3.jsonl`) — never overwrite v2 in place.
+- Ground truth refreshed quarterly; refresh PR includes a diff of changed expected outputs and the rationale.
+- Hand-curated edge cases (failure modes the model has historically tripped on) live in `evals/<feature>/edge.jsonl` alongside the golden set.
+
+## Eval Metrics
+
+Match the metric to the task class:
+
+- Classification → exact-match accuracy + per-class precision/recall.
+- Open-ended generation → rubric-scored LLM-as-judge with a 50-example calibration set sanity-checking judge agreement against human labels; recompute calibration every model change.
+- Retrieval/RAG → RAGAS metrics: context_precision, context_recall, faithfulness, answer_relevance.
+- Pairwise comparison → win-rate against the prior prompt version (>=55% required to adopt).
+- Refusal calibration → refusal-rate as an explicit SLI (refusals on prohibited inputs vs false-positive refusals on benign inputs).
+
+## Regression Gating
+
+- Every PR that touches `**/prompts/**`, `**/rag/**`, `**/ai/**`, `**/llm/**` runs the eval set in CI.
+- PR blocked when any metric drops below the per-feature threshold defined in `evals/<feature>/thresholds.json`.
+- Model-version upgrade (Sonnet to Opus, 4.6 to 4.7) runs the full eval with a 5% accuracy budget; cross over 5% requires a sign-off comment from a named reviewer and a 24-hour canary at 5% traffic.
+- Eval failure is treated as a test failure — never an advisory warning.
+
+## Prompt Versioning
+
+- Prompts are first-class artifacts at `prompts/<feature>/v<N>.{md,txt}` with an SHA-256 hash recorded in the eval thresholds file.
+- Runtime emits `prompt_version` + `prompt_hash` per request as log + span attribute.
+- A/B framework supports concurrent versions behind a flag — traffic split recorded per request so eval delta is computable.
+- Hash drift between repo prompt and deployed prompt is a P0 incident.
+
+## Cost Telemetry per Request
+
+Every LLM call logs: `tokens_in`, `tokens_out`, `cache_hit` (boolean + cached_tokens count), `model`, `cost_usd`, `latency_ms`, `cost_center` (feature ID), `prompt_version`, `prompt_hash`, `user_id_hash`.
+
+Aggregate dashboards in the observability stack — cross-reference `rules/hatch3r-observability-metrics.md` and `rules/hatch3r-observability-tracing-detail.md` for the SLI/SLO vocabulary, and `skills/hatch3r-observability-verify` for the wiring checklist. Per-feature budget alerts fire at 50%, 75%, and 90% of monthly budget; abuse-detection alert at 10x user p99 cost over a 1-hour window.
+
+## Prompt Caching (Anthropic)
+
+- Apply `cache_control` breakpoints to long system prompts, tool definitions, and large RAG context blocks above 1024 tokens (the Claude Opus/Sonnet 4.x minimum cache size; 2048 tokens for Haiku 4.x).
+- Up to 4 breakpoints per request, longest-TTL-first.
+- 5-minute TTL costs 1.25x the standard write rate; 1-hour TTL costs 2x the standard write rate. Reads are 0.1x base.
+- Track cache_hit ratio per feature; <30% hit ratio on a stable prompt is a sign the prefix is changing — investigate before next deploy.
+
+## OpenAI Prompt Caching
+
+- Automatic for requests over 1024 tokens with a stable deterministic prefix; ~50% discount on cached input tokens.
+- Cache duration documented by OpenAI as 5-10 minutes idle, up to 1 hour during off-peak.
+- Order the request prefix deterministically — same system prompt, same tool definitions, same retrieved-doc order — or the cache misses silently.
+
+## Model Router and Fallback
+
+Every LLM call wraps in a retry-with-decorrelated-jitter plus model-fallback chain:
+
+1. **Primary** — production-quality model (e.g. Sonnet 4.7).
+2. **Secondary** — faster/cheaper model (e.g. Haiku 4.5, GPT-5-mini, or a local Llama variant).
+3. **Static fallback** — cached response from a similar prior request or a canned templated reply that names the failure ("Search temporarily unavailable — retry in a minute").
+
+Cross-reference `rules/hatch3r-resilience-patterns.md` (Slice 8) for the circuit-breaker + retry-with-decorrelated-jitter primitives the chain reuses. Each fallback path has its own eval suite — a silent quality cliff between primary and secondary is a regression.
+
+## Hallucination and Groundedness as SLI
+
+Track per feature and treat as service-level indicators with explicit SLO targets:
+
+- `ai.hallucination_rate` — % of responses producing claims not present in retrieved sources. SLO: <5% on the golden set.
+- `ai.citation_precision` — % of citations whose source span verifiably contains the cited claim. SLO: >95% on retrieved claims.
+- `ai.refusal_rate` — overall refusal-rate; track false-positive refusals separately.
+- `ai.groundedness_score` — average RAGAS faithfulness across the golden set. SLO: >0.85.
+
+Cross-reference `rules/hatch3r-observability-metrics.md` for the SLI/SLO authoring template.
+
+## Safety and Red-Team
+
+Every feature exercising user-controlled prompts runs adversarial eval on a schedule (weekly minimum, on every prompt-version bump):
+
+- **Garak** — open-source jailbreak / prompt-injection / PII-leakage probe suite.
+- **PyRIT** — Microsoft red-team framework, scenario-driven.
+- **Inspect-redteam** — UK AISI safety eval modules.
+
+Block release on regression. PII-leakage tests use a synthetic-PII corpus (never production PII). Prompt-injection tests cover OWASP Top 10 for Agentic Applications 2026.
+
+## Tool-Use Evals
+
+When the AI feature uses tools (per the companion UX rule), the eval suite covers:
+
+- Tool-selection accuracy — correct tool chosen for each input class.
+- Argument validity — emitted args satisfy the tool schema (Zod, JSON Schema, Pydantic).
+- Chain correctness — multi-tool plans reach the goal with the minimum required step count plus a 20% tolerance.
+- Latency budget — p99 tool-execution time within the budget from `rules/hatch3r-performance-budgets.md`.
+
+Methodology aligned with **BFCL v4** (Berkeley Function Calling Leaderboard) and **tau-bench** (multi-turn tool-use benchmark).
+
+## OpenTelemetry GenAI Semantic Conventions
+
+Every LLM call emits an OpenTelemetry span named `gen_ai.<operation>` with the attributes prescribed by the OpenTelemetry GenAI semantic conventions: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cached_tokens`, `gen_ai.request.temperature`, `gen_ai.tool.name` (when tools used). Cross-reference Slice 2 observability rules for the broader span taxonomy.
+
+## User-Feedback Loop
+
+- Every AI response surface emits a thumbs-up/down control wired to a feedback queue.
+- A monthly triage job promotes thumbs-down examples into regression-test fixtures in `evals/<feature>/edge.jsonl`.
+- Promotion is a manual review step — never auto-promote raw user feedback (it contains noise and adversarial labels).
+
+## Audit Logging
+
+- LLM inputs + outputs logged to a compliance store separate from APM with a 30-90 day retention window (configurable per data-classification policy).
+- PII redaction before persistence — same redaction primitive as the framework's data-classification pipeline.
+- Reproducibility key: `model` + `prompt_hash` + `seed` (when the model exposes a seed parameter) + `temperature`. Without all four, the response is non-reproducible.
+
+## Eval-Driven Development Workflow
+
+Write eval before prompt, measure baseline, write prompt, measure delta, iterate until eval threshold passes. Cross-reference `skills/hatch3r-ai-feature/SKILL.md` for the step-by-step workflow.
+
+## References
+
+- promptfoo — `promptfoo.dev`
+- DeepEval — `github.com/confident-ai/deepeval`
+- RAGAS — `docs.ragas.io`
+- Inspect (UK AISI) — `github.com/UKGovernmentBEIS/inspect_ai`
+- Anthropic prompt caching guide — `docs.anthropic.com/en/docs/build-with-claude/prompt-caching`
+- OpenAI prompt caching guide — `platform.openai.com/docs/guides/prompt-caching`
+- OpenTelemetry GenAI semantic conventions — `opentelemetry.io/docs/specs/semconv/gen-ai/`
+- Berkeley Function Calling Leaderboard (BFCL v4) — `gorilla.cs.berkeley.edu/leaderboard.html`
+- tau-bench — `github.com/sierra-research/tau-bench`
+- OWASP Top 10 for LLM Applications (Agentic 2026) — `genai.owasp.org`

package/rules/hatch3r-ai-ux-patterns.md

@@ -0,0 +1,131 @@
+---
+id: hatch3r-ai-ux-patterns
+type: rule
+description: 2026 AI/agentic UX patterns for end-user projects shipping AI features — streaming, tool-call UI, human-approval gates, cancel/abort/undo, citations
+scope: "**/*.vue,**/*.jsx,**/*.tsx,**/*.svelte,**/ai/**,**/chat/**,**/assistant/**,**/agents/**,**/llm/**,**/copilot/**"
+tags: [ux, ai, frontend]
+quality_charter: agents/shared/quality-charter.md
+cache_friendly: true
+---
+# AI/Agentic UX Patterns (2026)
+
+## Scope
+
+This rule applies when the end-user project ships LLM-driven UI — chat, assistant, copilot, agent dashboards, generative UI surfaces. It does NOT govern the LLM backend itself (model selection, prompt engineering, retrieval pipeline). For non-AI UX rules (loading, empty, error, partial states; form patterns; microcopy), cross-reference `rules/hatch3r-ux-states-and-flows.md`. When both rules apply to the same surface, the non-AI rule sets the baseline and this rule layers AI-specific behavior on top.
+
+Backend companion: `rules/hatch3r-ai-evals.md` defines eval harness, prompt versioning, cost telemetry, prompt caching, model fallback, and hallucination-as-SLI. Apply both rules for any LLM-driven feature.
+
+Detection: this rule activates when the project imports an AI SDK (`ai`, `@ai-sdk/*`, `openai`, `@anthropic-ai/sdk`, `@google/generative-ai`), or contains files under `ai/`, `chat/`, `assistant/`, `agents/`, `llm/`, `copilot/`. Adding any of these to a project that previously had no AI surface triggers the full ruleset on the next agent run.
+
+## Streaming-First Defaults
+
+- Every LLM-driven surface uses framework-agnostic streaming hooks: `useChat`, `useCompletion`, or `useObject` from Vercel AI SDK UI (or equivalent for non-React stacks). Hand-rolled SSE or `fetch` against the model endpoint is a regression in 2026.
+- Render progressive tokens from the moment the request starts. Pair with a skeleton state during the pre-token window (request-sent, first-token-pending). A blank surface during model latency is a regression.
+- Render markdown incrementally without re-parsing the whole buffer on every chunk (Vercel AI Elements `MessageResponse` pattern). Use a streaming-safe markdown renderer that tracks last-rendered offset and appends from there.
+- Emit chunk-level error boundaries. If a stream fails mid-flight, retain prior tokens, render an inline retry control adjacent to the truncated response, and do not blank the surface or reset the message body.
+- Indicate completion explicitly via a final-token marker or `onFinish` callback transition. Stale "thinking" indicators after stream-end erode trust and waste user attention.
+- Non-streaming responses on LLM-driven surfaces are a regression. The narrow exception is structured-output endpoints that must return a single validated JSON payload (`useObject` already streams partial objects — prefer that).
+- Typing indicator policy: show only while no token has arrived; switch to streamed content the moment the first delta lands. Two-second indicator timeouts that linger past first-token are a regression.
+- Backpressure: when the model emits faster than the renderer can paint, batch deltas to one paint per animation frame (`requestAnimationFrame`) rather than dropping tokens or queuing into an unbounded buffer.
+
+## Generative UI (RSC Streaming)
+
+- For dynamic dashboards, cards, and surfaces driven by tool calls, use the `streamUI()` RSC pattern (or framework equivalent). The model returns a tool invocation, the server renders an RSC fragment per tool, the client merges fragments into the thread without re-hydrating.
+- Build on shadcn primitives via Vercel AI Elements — 20+ shadcn-based React components for AI surfaces: messages, input, reasoning panel, tool-call display, response actions. Reuse before authoring; bespoke equivalents are a duplication regression.
+- Keep AI-generated UI off the client JS bundle. RSC streaming keeps the payload server-side; do not ship a client-side renderer that duplicates server-rendered output.
+- Each RSC fragment is independently hydratable — never wrap two tool outputs in a single fragment that requires both to complete before render.
+- For non-React stacks, the equivalent is server-rendered HTML fragments streamed via Server-Sent Events and merged into the DOM with explicit anchor elements per tool call. Do not bypass the server-render boundary with raw `innerHTML` of model output.
+
+## Tool-Call UI Cards
+
+Every tool invocation is user-visible by default. Hidden tool calls are a regression — transparency is the trust contract with the user.
+
+- Render each tool call as a structured card with: tool name, args (collapsed by default, expandable on click), status (`pending` → `in-progress` → `complete` / `failed`), elapsed duration, result preview (truncated, expand to full).
+- Indicate state through both color and icon — a single channel (color only) fails for color-vision-deficient users.
+- Args display redacts secrets (API keys, tokens, bearer credentials) and PII per project data-classification rules. Show a `[redacted]` placeholder so the user knows a value was scrubbed, not omitted.
+- Async sub-flows — a tool that triggers more tools — render as nested cards. Do not collapse a multi-tool sub-flow to a single line; the user must see every external action.
+- Failed tool calls show the error message in plain language plus a single-click retry control. Stack traces and raw API responses belong behind an "Details" toggle, not in the user-visible card body.
+- Result preview length: cap at 200 characters or 5 lines (whichever first); always paired with an "Expand" affordance. Long binary or base64 payloads display size + content-type, not the raw bytes.
+- Tool-call cards are keyboard-focusable and announce status transitions via `aria-live="polite"` regions so screen-reader users perceive state changes without polling the visual surface.
+
+## Human-Approval Gates for Side-Effects
+
+Required for any tool that:
+
+- Mutates external state (database write, email send, post to Slack/GitHub/external service)
+- Spends money (paid-API call, transaction, billable inference run on a third-party provider)
+- Modifies the user's environment (file write, shell command, git push, package install)
+
+Gate pattern:
+
+1. Pause execution before invoking the tool.
+2. Render an approval card showing tool name, full args (no collapse), diff preview if the tool mutates a file, named target ("Send email to alice@example.com", not "Send email").
+3. Require explicit user confirm (`Approve` button, not Enter-to-dismiss).
+4. On approve, invoke the tool and transition the card to `in-progress`. On deny, surface a cancel-reason field and persist the rejection in the run transcript.
+
+Read-only tools (search, fetch, list, read) do not require approval — gating them adds friction without trust value.
+
+Destructive irreversible actions (drop table, delete repo, permanent send) disclose irreversibility on the approval card with a typed-confirmation pattern (user types the named target to enable the confirm button).
+
+Auto-approve scopes (the "trusted tool" pattern) live behind a per-tool setting with a named scope ("Auto-approve git status, never git push"). Default to opt-in, never opt-out. Scopes are revocable from a single settings surface and revocation takes effect for in-flight runs, not just future runs.
+
+## Cancel / Abort / Undo
+
+Three distinct affordances — do not collapse them into a single control:
+
+- **Cancel** — every long-running tool call (>2s expected duration) and every streaming response renders a visible cancel control adjacent to the surface. Wire to an `AbortController` covering both the model stream and the tool worker. Cancellation produces a clean stop with a "Cancelled by user" marker in the transcript — not a hang, not a silent terminate.
+- **Abort** — distinct from cancel. Aborts the entire agent task (a multi-tool plan). Persists prior tool results so the user can resume from any completed step. Renders as a separate control on the agent-task header, not the in-flight tool card.
+- **Undo** — every reversible action (file write, message send, database row insert, configuration change) renders an undo affordance for 5-30s after completion (named target shown: "Undo: deleted file foo.txt"). Maps to a compensating tool call (delete-after-write, recall-after-send, soft-delete-restore). Destructive irreversible actions disclose this on the approval card per the previous section.
+- Keyboard shortcuts: cancel = `Esc` while the surface is focused; abort = `Esc` then confirm (double-tap pattern) to prevent accidental task termination; undo = `Cmd/Ctrl+Z` while the undo affordance is visible.
+- All three controls remain operable after the user navigates away from the surface — running streams and undo windows persist in a thread-level state, not component-local state. A user closing the panel does not cancel the in-flight task.
+
+## Citations and Grounding
+
+- Factual claims from retrieval show inline citations with span-level linkage. Hovering or clicking the citation highlights the source passage in the retrieved document.
+- Cite the retrieval source URL or document anchor, not just the document name — "Knowledge Base / Onboarding" without an anchor is unverifiable.
+- Ungroundable claims are flagged in the response surface ("I'm not certain — this isn't in your retrieved sources") rather than silently emitted. Span-level verification is preferred when the model exposes it.
+- Do not show uncalibrated confidence badges. A "90% confident" indicator without empirical calibration data actively misleads — drop it or replace with grounding framing ("Based on [doc]" or "Not found in your data").
+- When retrieval returns zero results, say so explicitly. Silent fall-through to model parametric memory is a regression — the user cannot distinguish retrieved from confabulated content.
+- Citation rendering: superscript number link in the response body that opens a side panel (or popover on hover) with source title, URL, and highlighted span. Avoid footnote-only patterns that force the user to scroll.
+- Source freshness indicator: when retrieval metadata includes document timestamps, surface the source date on the citation popover. Stale sources flagged at >12 months by default; threshold configurable per project domain.
+
+## Multi-Step Agent UX
+
+- Render the agent's plan as an ordered checklist before execution. Allow the user to edit step text, reorder, or skip individual steps before approving the plan. A pre-execution plan view is required for any agent run with >3 steps.
+- As each step completes, mark with status (success/failure/skipped) plus elapsed duration. Failed steps show the failure cause and a retry control; retry restarts from the failed step, not from the beginning.
+- Streaming reasoning (when the model exposes it via a reasoning channel) renders in a collapsed "Reasoning" panel that the user opens on click. Never inline reasoning with the user-facing message body — it dilutes the response and conflates the user-facing answer with internal model state.
+- Maintain a thread or run ID per conversation for resumption, audit trail, and observability backlink. The run ID is visible in the UI (footer or thread metadata) so users and support can reference a specific run.
+- Plan revision: when the model revises its plan mid-run (adds, removes, or reorders remaining steps), diff the previous plan against the new plan and surface the diff to the user before continuing. Silent plan mutation defeats the pre-execution review.
+- Parallel steps: when the plan contains independent steps that the agent runs in parallel, render them as a horizontal group with shared progress, not a flat vertical list — the user must see the parallelism to interpret elapsed time.
+
+## Failure Modes and Degradation
+
+- **Model timeout** — render a fallback card with a single-click recovery ("Model is slow — retry?" or "Switching to faster model"). Do not silently retry; the user must know the model failed.
+- **Rate limit** — show a countdown timer and queue position when available. Surface vendor rate-limit headers, not raw API error JSON. "Try again in 47s" beats "429 Too Many Requests".
+- **Token budget exceeded** — indicate context truncation in-line ("Conversation truncated — earlier turns dropped") and offer a summary/restart control. Do not silently truncate; the user must know which turns are still in context.
+- **Tool unavailability** — degrade gracefully. Disable the affected feature with a tooltip explaining why ("Slack integration offline — reconnect in Settings"), not a crash or an unhandled exception bubbled to the user.
+- **Stream interruption mid-response** — retain the partial response, render a "Stream interrupted" marker, offer continue-from-here (resume the stream from the last received token) and restart-from-scratch (drop partial, restart) as two separate controls.
+- **Content-policy refusal** — when the model refuses to answer (safety filter, policy violation), render the refusal verbatim in a distinct style (not a normal assistant message), and surface a single-click "Report this" affordance so users can flag false-positive refusals.
+- **Stale session** — if the user returns to a tab where the stream has died (network sleep, tab discard), detect on focus and offer a "Reconnect" control rather than silently re-issuing the request and double-billing the user.
+
+## Verification Gate
+
+Before declaring an AI surface "done":
+
+- Streaming verified end-to-end: scripted Playwright test that asserts progressive token render (first token <1s after request, last token marked complete).
+- Tool-call card snapshot per state (`pending`, `in-progress`, `complete`, `failed`) — missing any state is a blocker.
+- Approval-gate test: simulate a side-effecting tool, assert execution pauses, approval card renders required fields, deny path persists rejection in transcript.
+- Cancel/abort/undo each have at least one end-to-end test exercising the AbortController wire-up; cancellation produces a transcript marker, not a silent terminate.
+- Citation rendering: snapshot of a grounded response shows inline citation with source URL or anchor; an ungroundable claim renders the flag string, not silent emission.
+- Failure-mode coverage: at least one test per failure mode in this rule (timeout, rate limit, token budget, tool unavailability, stream interruption, content-policy refusal, stale session). Missing coverage is a blocker.
+- Accessibility: axe-core scan against every AI surface state (idle, streaming, tool-call pending, approval card open, error) returns 0 serious or critical violations.
+
+## References
+
+- Vercel — Introducing AI Elements (`vercel.com/changelog/introducing-ai-elements`)
+- Vercel AI SDK UI docs (`ai-sdk.dev/docs/ai-sdk-ui`) — `useChat`, `useCompletion`, `useObject`
+- Vercel — AI SDK 3 Generative UI / `streamUI()` (`vercel.com/blog/ai-sdk-3-generative-ui`)
+- Anthropic — Building agents with the Claude Agent SDK (`anthropic.com/engineering/building-agents-with-the-claude-agent-sdk`)
+- OpenAI Apps SDK — UI guidelines (`developers.openai.com/apps-sdk/concepts/ui-guidelines`)
+- ClarityArc — Hallucination, grounding, and citation in enterprise systems
+- Lakera — LLM hallucinations in 2026