universal-dev-standards 5.4.0 → 5.5.0

package/bundled/ai/standards/incident-response.ai.yaml

@@ -0,0 +1,107 @@
+# Incident Response Standards - AI Optimized
+# Source: XSPEC-063 Wave 3 SRE Pack
+
+id: incident-response
+title: Incident Response Standards
+version: "1.0.0"
+status: Active
+tags: [sre, incident, oncall, postmortem, severity]
+summary: |
+  Defines the end-to-end incident response lifecycle: severity classification,
+  response time SLAs, roles and responsibilities, communication protocols,
+  escalation paths, and blameless postmortem requirements. Designed to reduce
+  MTTR, ensure consistent stakeholder communication during incidents, and
+  drive systemic reliability improvements through structured postmortems.
+
+requirements:
+  - id: REQ-001
+    title: Severity Classification
+    description: |
+      Every incident MUST be classified at declaration using a 4-level
+      severity scale. SEV-1 (Critical): complete service outage or data
+      breach affecting all users, response within 15 minutes, C-suite
+      notification. SEV-2 (High): major feature unavailable or significant
+      performance degradation >25% users, response within 30 minutes.
+      SEV-3 (Medium): minor feature unavailable or degradation <25% users,
+      response within 4 hours. SEV-4 (Low): cosmetic issue or very minor
+      impact, response within 24 hours.
+    level: MUST
+    examples:
+      - "SEV-1: Payment processing completely down → IC declared, bridge opened in <5min"
+      - "SEV-2: Checkout latency >10s for 30% of users → on-call paged, IC assigned"
+      - "SEV-3: Search autocomplete broken → ticket created, fix in next sprint"
+
+  - id: REQ-002
+    title: Incident Commander Role
+    description: |
+      Every SEV-1 and SEV-2 incident MUST have a designated Incident
+      Commander (IC) assigned within 10 minutes of declaration. The IC is
+      responsible for: coordinating the response bridge, assigning roles
+      (scribe, comms lead, technical leads), driving timeline, making
+      go/no-go decisions on fixes, and initiating postmortem. The IC does
+      NOT directly troubleshoot — their sole focus is coordination.
+    level: MUST
+    examples:
+      - "IC assignment: 'I'm taking IC for this SEV-2, @alice please scribe, @bob comms'"
+      - "IC checklist: bridge open, roles assigned, status page updated, stakeholders notified"
+      - "IC does not debug code; delegates to technical leads who report findings"
+
+  - id: REQ-003
+    title: Stakeholder Communication Protocol
+    description: |
+      During SEV-1/SEV-2 incidents, stakeholder updates MUST be sent on a
+      defined cadence: initial notification within 15 minutes of declaration,
+      updates every 30 minutes until resolution, immediate update on any
+      severity change or major development. Updates MUST include: current
+      status, known impact, what is being done, next update time. Status
+      page MUST be updated simultaneously with internal communications.
+    level: MUST
+    examples:
+      - "T+15min: 'We are investigating elevated error rates on checkout. ~500 users affected. Next update in 30min.'"
+      - "T+45min: 'Root cause identified as database connection pool exhaustion. Fix in progress. Next update in 30min.'"
+      - "Resolution: 'Service restored at 14:32 UTC. Postmortem scheduled for 2026-05-02.'"
+
+  - id: REQ-004
+    title: Blameless Postmortem Requirements
+    description: |
+      Every SEV-1 incident MUST have a blameless postmortem completed within
+      5 business days. SEV-2 incidents MUST have a postmortem within 10
+      business days. Postmortems MUST be blameless — focusing on systemic
+      causes, not individual mistakes. Required sections: timeline, impact,
+      root cause (5 Whys), contributing factors, action items with owners
+      and due dates, lessons learned. Action items MUST be tracked to
+      completion.
+    level: MUST
+    examples:
+      - "Timeline: 14:00 Alert fired → 14:05 IC assigned → 14:23 RCA found → 14:32 resolved"
+      - "Root cause: 5 Whys tracing alert → connection pool exhaustion → missing timeout config → no review gate"
+      - "Action item: INFRA-2341 Add connection pool monitoring, owner: @dave, due: 2026-05-15"
+
+  - id: REQ-005
+    title: On-Call Rotation and Handoff
+    description: |
+      Every production service MUST have a documented on-call rotation with
+      at least 2 engineers. Rotation schedules MUST be published at least
+      2 weeks in advance. On-call handoff MUST include a written summary of:
+      active incidents, recent incidents still under investigation,
+      known flaky alerts, upcoming planned maintenance, and any service
+      health concerns. Handoff MUST be acknowledged by incoming on-call.
+    level: MUST
+    examples:
+      - "Handoff doc: 3 active flaky alerts (JIRA links), 1 ongoing SEV-3, maintenance window Friday 02:00 UTC"
+      - "Rotation: 1-week shifts, 2 primary + 1 secondary, published monthly in PagerDuty"
+      - "Handoff acknowledgment: incoming engineer replies 'Handoff received, I have the pager'"
+
+  - id: REQ-006
+    title: Incident Retrospective Metrics
+    description: |
+      Teams SHOULD track and review incident metrics monthly: MTTD (Mean
+      Time To Detect), MTTR (Mean Time To Resolve), incident frequency by
+      severity, repeat incidents (same root cause within 90 days), and
+      postmortem action item completion rate. These metrics SHOULD be
+      reviewed in monthly reliability reviews with engineering leadership.
+    level: SHOULD
+    examples:
+      - "Monthly metrics: MTTD 8min, MTTR 42min, 3 SEV-2s, 0 SEV-1s, 0 repeat incidents"
+      - "Repeat incident flag: same DB timeout root cause in March and April → escalate to reliability sprint"
+      - "Action item completion rate: 85% of previous month's items closed on time"

package/bundled/ai/standards/license-compliance.ai.yaml

@@ -0,0 +1,106 @@
+# License Compliance Standards - AI Optimized
+# Source: XSPEC-066 Wave 3 Compliance Pack
+
+id: license-compliance
+title: License Compliance Standards
+version: "1.0.0"
+status: Active
+tags: [compliance, licensing, open-source, legal, supply-chain]
+summary: |
+  Defines how teams identify, track, and manage open-source and third-party
+  software licenses throughout the software development lifecycle. Covers
+  license classification (permissive vs. copyleft), prohibited licenses,
+  SBOM generation, license scanning in CI/CD, and remediation processes
+  for license violations. Designed to prevent legal exposure from
+  incompatible license combinations and ensure supply-chain transparency.
+
+requirements:
+  - id: REQ-001
+    title: License Classification and Allowlist
+    description: |
+      Every project MUST maintain a documented license policy classifying
+      licenses into three tiers. APPROVED: MIT, Apache 2.0, BSD-2-Clause,
+      BSD-3-Clause, ISC, CC0 — can be used without review. REVIEW-REQUIRED:
+      LGPL-2.1, LGPL-3.0, MPL-2.0, CDDL — requires legal/architecture
+      review before adoption. PROHIBITED: GPL-2.0 (without exception),
+      GPL-3.0, AGPL-3.0, SSPL, Commons Clause additions — MUST NOT be
+      included in proprietary or commercial products.
+    level: MUST
+    examples:
+      - "MIT dependency lodash → auto-approved, no review needed"
+      - "LGPL-3.0 dependency → open GitHub issue tagged 'license-review' before merging"
+      - "GPL-3.0 dependency found in PR → PR blocked by CI until dependency replaced"
+
+  - id: REQ-002
+    title: Automated License Scanning in CI
+    description: |
+      Every repository MUST run automated license scanning on every pull
+      request and on a daily scheduled basis. The scan MUST fail the build
+      if any PROHIBITED license is detected in direct or transitive
+      dependencies. REVIEW-REQUIRED licenses MUST generate a warning that
+      blocks merge until manually approved and documented. Scan results
+      MUST be stored as build artifacts for 1 year.
+    level: MUST
+    examples:
+      - "CI step: `npx license-checker --onlyAllow 'MIT;Apache-2.0;BSD-2-Clause;BSD-3-Clause;ISC'`"
+      - "Daily scheduled scan via GitHub Actions: runs license-checker across all repos"
+      - "Build artifact: license-report-2026-04-30.json retained for 1 year"
+
+  - id: REQ-003
+    title: Software Bill of Materials (SBOM) Generation
+    description: |
+      Every production release MUST include a Software Bill of Materials
+      (SBOM) in SPDX 2.3 or CycloneDX 1.5 format. The SBOM MUST list all
+      direct and transitive dependencies with: package name, version,
+      license identifier (SPDX), and supplier. SBOMs MUST be generated
+      automatically as part of the release pipeline and stored alongside
+      release artifacts.
+    level: MUST
+    examples:
+      - "SBOM generated via: `syft . -o spdx-json > sbom-v1.2.0.spdx.json`"
+      - "SBOM uploaded to release: GitHub Release v1.2.0 includes sbom-v1.2.0.spdx.json"
+      - "SBOM SPDX entry: {name: 'lodash', version: '4.17.21', licenseConcluded: 'MIT'}"
+
+  - id: REQ-004
+    title: License Attribution and Notices
+    description: |
+      Products distributed to external users MUST include a NOTICES or
+      LICENSE-THIRD-PARTY file listing all open-source components and their
+      license texts. This file MUST be auto-generated from the SBOM data.
+      For web applications, attribution MUST be accessible via a dedicated
+      page or endpoint (e.g., /licenses or /legal/notices).
+    level: MUST
+    examples:
+      - "NOTICES file auto-generated: `npx generate-license-file --input package.json --output NOTICES`"
+      - "Web app: GET /legal/notices returns HTML page with all dependency licenses"
+      - "Mobile app: 'Open Source Licenses' screen in Settings menu"
+
+  - id: REQ-005
+    title: License Violation Remediation Process
+    description: |
+      When a license violation is detected, teams MUST follow the remediation
+      process: (1) immediately block merge/release if PROHIBITED license;
+      (2) create a tracking issue within 1 business day; (3) remediate
+      within 5 business days for PROHIBITED, 30 days for REVIEW-REQUIRED;
+      (4) document decision and alternative chosen; (5) update SBOM and
+      re-scan to verify clean. Unresolved violations MUST be reported to
+      the engineering lead monthly.
+    level: MUST
+    examples:
+      - "Violation: transitive dep uses GPL-3.0 → PR blocked, issue COMP-112 created same day"
+      - "Remediation: replaced dep with MIT-licensed alternative within 3 days"
+      - "Monthly report: 0 open violations as of 2026-04-30"
+
+  - id: REQ-006
+    title: License Review for New Technology Adoption
+    description: |
+      Before adopting any new framework, library, or tool, engineers SHOULD
+      verify its license tier as part of the technology evaluation. License
+      status SHOULD be documented in the relevant ADR or technical spike
+      document. For REVIEW-REQUIRED licenses, architecture review board
+      approval MUST be obtained before adoption.
+    level: SHOULD
+    examples:
+      - "ADR-042 notes: 'Library X uses Apache 2.0 — approved tier, no legal review needed'"
+      - "ADR-043 notes: 'Library Y uses LGPL-3.0 — review-required, legal approved 2026-03-10'"
+      - "Technology radar entry includes license classification for each evaluated tool"

package/bundled/ai/standards/llm-output-validation.ai.yaml

@@ -0,0 +1,269 @@
+# LLM Output Validation Standards - AI Optimized
+# Source: core/llm-output-validation.md
+
+id: llm-output-validation
+meta:
+  version: "1.0.0"
+  updated: "2026-05-05"
+  source: core/llm-output-validation.md
+  description: >
+    Standards for validating LLM and AI agent outputs to ensure schema
+    conformance, detect hallucinations, and assess response groundedness.
+    Covers structural validation, semantic validation, and test strategies.
+
+# ─────────────────────────────────────────────────────────
+# Core Concepts
+# ─────────────────────────────────────────────────────────
+core_concepts:
+  definition: >
+    LLM outputs are non-deterministic and may violate expected schema,
+    introduce hallucinated facts, or fail to stay grounded in provided context.
+    LLM output validation is the practice of systematically testing that
+    AI agent outputs meet structural and semantic quality standards.
+
+  validation_dimensions:
+    - dimension: Schema Conformance
+      definition: >
+        The output matches the declared JSON / type schema exactly —
+        required fields present, types correct, enums respected.
+      test_type: Contract Test
+      automated: true
+
+    - dimension: Hallucination Detection
+      definition: >
+        Factual claims in the output can be traced back to source material
+        (grounded) or are fabricated (hallucinated).
+      test_type: Semantic / NLI probe
+      automated: partial
+
+    - dimension: Grounded Answer Rate
+      definition: >
+        Ratio of responses that cite or derive from provided context
+        vs. responses that introduce unsupported facts.
+      metric: "grounded_answers / total_answers"
+      target: "≥ 0.95 for RAG-based agents"
+      automated: partial
+
+    - dimension: Refusal Evaluation
+      definition: >
+        Agent correctly refuses (or escalates) out-of-scope, harmful,
+        or ambiguous requests rather than hallucinating a plausible answer.
+      test_type: Adversarial probe
+      automated: true (via test corpus)
+
+    - dimension: Determinism / Stability
+      definition: >
+        With temperature=0, same prompt yields same output schema structure
+        (if not identical content).
+      test_type: Replay / Golden file
+      automated: true
+
+# ─────────────────────────────────────────────────────────
+# Contract Testing (Schema Conformance)
+# ─────────────────────────────────────────────────────────
+contract_testing:
+  definition: >
+    Contract tests verify that LLM/agent outputs satisfy a declared schema
+    (JSON Schema, Zod, Pydantic, etc.) before being consumed downstream.
+    They are the cheapest and most automated form of LLM output validation.
+
+  patterns:
+    - name: Schema-driven contract test
+      description: >
+        Maintain an output-schema.json per agent. Run fixture outputs through
+        a schema validator (Ajv / Pydantic / jsonschema) in unit tests.
+      example: |
+        // TypeScript with ajv
+        import Ajv from "ajv"
+        import schema from "./output-schema.json"
+        const ajv = new Ajv()
+        const validate = ajv.compile(schema)
+        const result = validate(agentOutput)
+        expect(result).toBe(true)
+
+    - name: Golden file test
+      description: >
+        Capture a real LLM response as a golden fixture (JSON).
+        Re-validate against the schema on every schema change.
+        Diff the fixture if schema is unchanged (regression).
+
+    - name: Negative contract test
+      description: >
+        Provide outputs that violate the schema (missing required fields,
+        wrong types). Assert the validator correctly rejects them.
+        Proves the schema is strict enough to catch real failures.
+
+  per_agent_structure:
+    - "agents/<agent-name>/output-schema.json — JSON Schema definition"
+    - "agents/<agent-name>/__tests__/contract.test.ts — contract test suite"
+    - "agents/<agent-name>/__fixtures__/valid.json — valid output fixture"
+    - "agents/<agent-name>/__fixtures__/invalid-*.json — invalid fixtures"
+
+# ─────────────────────────────────────────────────────────
+# Hallucination Detection
+# ─────────────────────────────────────────────────────────
+hallucination_detection:
+  definition: >
+    Hallucination occurs when an LLM generates plausible-sounding but
+    factually incorrect or unsupported content.
+
+  detection_strategies:
+    - strategy: Grounding check
+      description: >
+        For RAG-based agents: compare named entities / key facts in response
+        against retrieved documents using NLI or embedding similarity.
+      tools: [DeepEval, Ragas, custom NLI probe]
+      complexity: high
+
+    - strategy: Schema field validation
+      description: >
+        Structured outputs (JSON) partially prevent hallucination by requiring
+        specific enumerations and reference IDs. If the LLM hallucinates a
+        field value, it either violates the schema (caught by contract test)
+        or produces a real-looking but wrong value (requires semantic check).
+      complexity: low
+
+    - strategy: Confidence calibration
+      description: >
+        Agent includes a confidence score or explicitly marks uncertain claims.
+        Downstream consumers reject or escalate low-confidence outputs.
+      requires: Prompt design to elicit uncertainty markers
+      complexity: medium
+
+  rate_targets:
+    structured_output_agents:
+      schema_conformance: "≥ 99%"
+      factual_hallucination_rate: "≤ 5%"
+    rag_agents:
+      grounded_answer_rate: "≥ 95%"
+    chat_agents:
+      refusal_on_out_of_scope: "≥ 90%"
+
+# ─────────────────────────────────────────────────────────
+# Prompt Regression & Stability
+# ─────────────────────────────────────────────────────────
+prompt_regression:
+  definition: >
+    When a prompt is changed, verify that the output schema structure
+    is preserved (even if content differs). A prompt change that silently
+    breaks downstream field expectations is a regression.
+
+  test_pattern: |
+    # 1. Before prompt change: capture golden output structure
+    # 2. Change prompt
+    # 3. Re-run with temperature=0 on same input fixture
+    # 4. Validate output against schema (contract test)
+    # 5. Diff field presence / enum values against golden
+
+  required_on:
+    - Any change to agents/*/prompt.md
+    - Model version upgrade (same prompt, new model)
+    - New required output field added to schema
+
+# ─────────────────────────────────────────────────────────
+# Tools & Libraries
+# ─────────────────────────────────────────────────────────
+tools:
+  schema_validation:
+    - name: Ajv (TypeScript/JavaScript)
+      package: "ajv"
+      strengths: [JSON Schema draft-07 support, fast, minimal]
+      use_case: Contract tests for agents with JSON output-schema.json
+
+    - name: Zod (TypeScript)
+      package: "zod"
+      strengths: [Type-safe, composable, coercion support]
+      use_case: Runtime validation when TypeScript types are available
+
+    - name: Pydantic (Python)
+      package: "pydantic"
+      strengths: [Automatic validation from type annotations]
+      use_case: Python-based agent contract tests
+
+  hallucination_evaluation:
+    - name: DeepEval
+      package: "deepeval"
+      features: [Hallucination metric, Contextual Precision, Faithfulness]
+      note: Requires LLM judge (adds cost)
+
+    - name: Ragas
+      package: "ragas"
+      features: [Answer Grounding, Context Recall, Faithfulness]
+      note: Designed for RAG pipelines
+
+# ─────────────────────────────────────────────────────────
+# Quality Gates
+# ─────────────────────────────────────────────────────────
+quality_gates:
+  - gate: Schema conformance (CI)
+    threshold: "100% of valid fixtures pass schema validation"
+    enforcement: Block merge
+    automated: true
+    note: Run in every PR via contract.test.ts
+
+  - gate: Negative schema rejection (CI)
+    threshold: "100% of invalid fixtures are rejected by schema validator"
+    enforcement: Block merge
+    automated: true
+    note: Proves schema is strict enough
+
+  - gate: Hallucination rate (pre-release)
+    threshold: "≤ 5% hallucinated facts on benchmark dataset"
+    enforcement: Advisory (escalate to human review if exceeded)
+    automated: partial
+    note: Run monthly or on model upgrade
+
+  - gate: Prompt regression (on prompt change)
+    threshold: "Schema conformance maintained after prompt change"
+    enforcement: Block merge
+    automated: true (via contract test re-run)
+
+# ─────────────────────────────────────────────────────────
+# Rules
+# ─────────────────────────────────────────────────────────
+rules:
+  - id: agent-must-have-output-schema
+    trigger: defining or modifying an AI agent
+    instruction: >
+      Every agent MUST have a declared output-schema.json.
+      Agents without a schema cannot be safely composed into pipelines.
+    priority: required
+
+  - id: contract-test-per-agent
+    trigger: defining or modifying an AI agent
+    instruction: >
+      Every agent MUST have a contract.test.ts with at least:
+      (1) one valid fixture that passes schema validation
+      (2) one invalid fixture that fails schema validation
+    priority: required
+
+  - id: prompt-change-triggers-contract-rerun
+    trigger: modifying agents/*/prompt.md
+    instruction: >
+      Re-run the agent's contract tests with temperature=0 after any
+      prompt change. A schema violation in the golden fixture signals regression.
+    priority: required
+
+  - id: model-upgrade-triggers-contract-rerun
+    trigger: upgrading the LLM model version used by an agent
+    instruction: >
+      Run all agent contract tests against the new model.
+      Compare output structure, field presence, and enum values against golden.
+    priority: required
+
+anti_patterns:
+  - Defining agents without output-schema.json
+  - Only validating JSON.parse (syntax) without schema (semantic) validation
+  - Writing contract tests with only valid fixtures (no negative cases)
+  - Accepting model upgrades without re-running contract tests
+  - Using high temperature (> 0.3) in contract test fixtures (non-deterministic)
+
+quick_reference:
+  contract_test_checklist: |
+    □ agents/<name>/output-schema.json exists
+    □ agents/<name>/__tests__/contract.test.ts exists
+    □ Valid fixture: all required fields, correct types, enum values
+    □ Invalid fixtures: missing required field, wrong type, invalid enum
+    □ Schema validator: Ajv (JSON Schema) or Zod (TypeScript)
+    □ CI: contract tests run on every PR
+    □ Prompt change: re-run contract test, compare against golden