universal-dev-standards 5.4.0 → 5.5.0

package/bundled/ai/standards/mock-boundary.ai.yaml

@@ -0,0 +1,250 @@
+# Mock Boundary Standards - AI Optimized
+# Source: core/mock-boundary.md
+
+id: mock-boundary
+meta:
+  version: "1.0.0"
+  updated: "2026-05-04"
+  source: core/mock-boundary.md
+  description: >
+    Rules defining what can and cannot be mocked to prevent hollow tests —
+    tests that pass while the real system is broken.
+
+# ─────────────────────────────────────────────────────────
+# Core Problem
+# ─────────────────────────────────────────────────────────
+core_problem:
+  name: Hollow Test Anti-Pattern
+  description: >
+    Over-mocking replaces real business logic with test doubles, making the test suite
+    a specification of mock behavior rather than system behavior.
+    The tests pass in CI while the real system silently fails.
+  real_world_example: |
+    // SPEC-002.test.ts (VibeOps) — hollow test example
+    vi.mock('../../src/runner/agent-runner.js')      // Core dependency mocked
+    vi.mock('../../src/runner/guardian-hooks.js')     // Core dependency mocked
+    vi.mock('../../src/runner/prototyper.js')         // Core dependency mocked
+    vi.mock('../../src/runner/iteration-report.js')   // Core dependency mocked
+    vi.mock('../../src/memory/memory-store.js')       // Core dependency mocked
+    vi.mock('node:fs/promises', ...)                  // I/O mocked
+
+    // Result: runPipeline() runs but touches ZERO real code.
+    // All 8 agent calls are faked. The test proves nothing about pipeline logic.
+
+# ─────────────────────────────────────────────────────────
+# Allowed Mocks
+# ─────────────────────────────────────────────────────────
+allowed:
+  - category: External HTTP Services
+    description: Third-party APIs, LLM providers, payment gateways, email services
+    reason: Prevents flaky tests from external dependencies; enables response scenario control
+    examples:
+      - OpenAI / Anthropic / Grok API
+      - Stripe / payment processors
+      - SendGrid / email providers
+      - External OAuth providers
+    implementation: Mock the HTTP client or provider factory; never mock the internal caller
+
+  - category: Time Functions
+    description: Date.now(), new Date(), setTimeout, setInterval
+    reason: Makes tests deterministic and enables time-travel scenarios
+    examples:
+      - "vi.useFakeTimers()"
+      - "vi.setSystemTime(new Date('2026-01-01'))"
+    note: Always restore real timers after test with vi.useRealTimers()
+
+  - category: Environment Variables
+    description: process.env values
+    reason: Tests need different configurations without changing system state
+    implementation: Use vi.stubEnv() or process.env assignment in beforeEach/afterEach
+
+  - category: File System (unit tests only)
+    description: fs.readFile, fs.writeFile, fs.stat in UNIT tests
+    reason: Avoids slow I/O in fast unit tests
+    constraint: >
+      Integration tests, flow tests, and E2E tests MUST use real filesystem
+      or in-memory FS (memfs) — never vi.mock('node:fs/promises') at those levels
+
+  - category: Cross-Module Boundaries (with counterpart)
+    description: Calls to OTHER modules when testing THIS module's own logic
+    reason: Isolates the unit under test from its collaborators
+    constraint: >
+      A corresponding integration test MUST exist that exercises the real interaction.
+      Mock only when the real collaborator has its own test coverage.
+    example: |
+      // Mocking the DB layer in a service unit test is OK IF:
+      // 1. The DB layer has its own integration tests
+      // 2. The service test focuses on service logic (not DB behavior)
+
+# ─────────────────────────────────────────────────────────
+# Forbidden Mocks
+# ─────────────────────────────────────────────────────────
+forbidden:
+  - category: Own Module Core Logic
+    description: Mocking the module's OWN functions in the file that tests it
+    example: |
+      // ❌ Testing pipeline-runner.ts but mocking pipeline-runner itself
+      vi.mock('../../src/runner/pipeline-runner.js')
+      import { runPipeline } from '../../src/runner/pipeline-runner.js'
+      // runPipeline is now a no-op stub — the test proves nothing
+    violation_indicator: >
+      The mock import path resolves to the same directory as the test file,
+      or the mock replaces the primary export being tested.
+    fix: Remove the mock and let the real code run; mock only its external dependencies
+
+  - category: Database Layer in Integration/Flow Tests
+    description: Replacing DB calls with in-memory return values in integration or flow tests
+    reason: Masks query bugs, schema constraint violations, index issues, and migration errors
+    alternative: >
+      Use in-memory SQLite (better-sqlite3 / sql.js), test containers,
+      or a dedicated test schema — a real database with controlled data
+    example: |
+      // ❌ Forbidden in integration/flow tests
+      vi.mock('../../src/db/client.js', () => ({ query: vi.fn().mockResolvedValue([]) }))
+
+      // ✅ Correct: use real in-memory DB
+      import Database from 'better-sqlite3'
+      const testDb = new Database(':memory:')
+
+  - category: Core Framework Internals
+    description: Express/Fastify routing, ORM core (Drizzle/Prisma internals), auth middleware core
+    reason: Tests pass while real routing, query building, or auth enforcement is broken
+    example: |
+      // ❌ Forbidden
+      vi.mock('express', () => ({ Router: () => ({ get: vi.fn(), post: vi.fn() }) }))
+
+  - category: Security Controls
+    description: Auth token validators, permission checks, rate limiters, input sanitizers
+    reason: Mocking security controls makes tests useless for security validation
+    example: |
+      // ❌ Forbidden — this test proves nothing about auth
+      vi.mock('../../src/auth/middleware.js', () => ({
+        requireAuth: (req, res, next) => next()  // always passes
+      }))
+    fix: Use a real test user with a real valid token; test with real auth logic
+
+# ─────────────────────────────────────────────────────────
+# Hollow Test Detection Patterns
+# ─────────────────────────────────────────────────────────
+detection_patterns:
+  hollow_test_indicators:
+    - name: Mock Count Exceeds Import Count
+      check: "vi.mock() call count >= number of non-type imports in the test file"
+      severity: high
+      action: Review all assertions; verify at least one assertion is on actual output
+
+    - name: Assertions Only on Mock Calls
+      check: "All expect() statements use .toHaveBeenCalled() or .toHaveBeenCalledWith()"
+      severity: high
+      action: Add assertions on actual return values and system state changes
+
+    - name: More Mock Setup Than Assertions
+      check: "Lines of mock setup > lines of expect() assertions"
+      severity: medium
+      action: Consider if the test is testing behavior or just mock wiring
+
+    - name: Self-Referential Mock
+      check: "A vi.mock() path resolves to the same module being imported as the subject under test"
+      severity: critical
+      action: Remove the self-mock immediately; it makes the test a no-op
+
+  ai_generation_warning: >
+    AI tools (including this assistant) tend to generate hollow tests because:
+    1. Mocking makes tests compile and pass without requiring real infrastructure
+    2. AI cannot know the full dependency graph at generation time
+    When reviewing AI-generated tests, always apply the hollow test indicators above.
+
+# ─────────────────────────────────────────────────────────
+# Anti-Patterns
+# ─────────────────────────────────────────────────────────
+anti_patterns:
+  - name: Total Mock Isolation
+    description: Every import is mocked; test verifies only mock interaction counts
+    problem: Tests pass regardless of actual logic correctness
+    symptom: Deleting the implementation file doesn't break the test
+
+  - name: Mock the World
+    description: External + internal + database + filesystem all mocked in one test
+    problem: Test becomes a specification of mock behavior, not system behavior
+
+  - name: Mock Without Integration Counterpart
+    description: Cross-module mock with no corresponding integration test
+    problem: The interaction between modules is never actually exercised
+
+  - name: Security Mock Bypass
+    description: Auth/permission middleware replaced with always-pass stub
+    problem: Security regression cannot be detected
+
+  - name: Database Mock Cascade
+    description: DB mock returns hardcoded data, hiding query logic errors
+    problem: Schema migrations, wrong predicates, missing joins — all invisible
+
+# ─────────────────────────────────────────────────────────
+# Rules
+# ─────────────────────────────────────────────────────────
+rules:
+  - id: no-own-module-mock
+    trigger: writing any test for a module
+    instruction: Never vi.mock() a path that resolves to the module being tested
+    priority: required
+
+  - id: real-db-in-flow-tests
+    trigger: writing flow test, integration test, or E2E test
+    instruction: >
+      Use a real database (in-memory SQLite, test container, or test schema).
+      Never mock the DB layer in these test levels.
+    priority: required
+
+  - id: mock-needs-integration-counterpart
+    trigger: adding a vi.mock() for a cross-module dependency in a unit test
+    instruction: >
+      Ensure a corresponding integration test exercises the real interaction.
+      Note the counterpart test in a comment: "// integration: see src/__tests__/integration/..."
+    priority: required
+
+  - id: security-no-mock
+    trigger: test involves authentication, authorization, or rate limiting
+    instruction: >
+      Never mock security controls.
+      Create a real test user with a real token; exercise the real auth logic.
+    priority: required
+
+  - id: hollow-test-review
+    trigger: mock count in test file equals or exceeds non-type import count
+    instruction: >
+      Apply hollow test indicators checklist before submitting.
+      At least one assertion must verify an actual output value (not a mock call).
+    priority: required
+
+  - id: ai-generated-test-mock-review
+    trigger: tests are AI-generated
+    instruction: >
+      AI-generated tests frequently over-mock. Apply all detection_patterns checks.
+      If any hollow test indicator triggers, rewrite the mocking strategy.
+    priority: required
+
+# ─────────────────────────────────────────────────────────
+# Quick Reference
+# ─────────────────────────────────────────────────────────
+quick_reference:
+  allowed_mock_summary:
+    - "✅ External HTTP/LLM/payment APIs"
+    - "✅ Time functions (Date.now, setTimeout)"
+    - "✅ Environment variables"
+    - "✅ File system — in unit tests only"
+    - "✅ Cross-module boundaries — with integration test counterpart"
+
+  forbidden_mock_summary:
+    - "❌ Own module's core logic (self-referential mock)"
+    - "❌ Database layer in integration/flow/E2E tests"
+    - "❌ HTTP framework internals (Express router, etc.)"
+    - "❌ Security controls (auth middleware, permission checks)"
+
+  checklist: |
+    Before submitting test with mocks:
+    □ No vi.mock() path matches the module under test
+    □ DB layer not mocked in integration/flow tests
+    □ Security controls not mocked
+    □ Mock count < import count (or justified with comment)
+    □ At least one assertion on actual output value (not mock call)
+    □ Integration counterpart exists for each cross-module mock

package/bundled/ai/standards/mutation-testing.ai.yaml

@@ -0,0 +1,192 @@
+# Mutation Testing Standards - AI Optimized
+# Source: core/mutation-testing.md
+
+id: mutation-testing
+meta:
+  version: "1.0.0"
+  updated: "2026-05-04"
+  source: core/mutation-testing.md
+  description: >
+    Mutation testing methodology to evaluate test suite effectiveness.
+    Answers "do my tests actually catch bugs?" beyond line coverage.
+
+# ─────────────────────────────────────────────────────────
+# Core Concepts
+# ─────────────────────────────────────────────────────────
+core_concepts:
+  definition: >
+    Mutation testing automatically injects small bugs (mutations) into source code,
+    then runs the test suite to see if tests detect (kill) the bug.
+    A test suite that kills most mutations is effective; one that misses them is hollow.
+
+  key_terms:
+    - term: Mutant
+      definition: A copy of source code with one small artificial bug injected
+    - term: Killed mutant
+      definition: Test suite detected the bug (test failed)
+    - term: Survived mutant
+      definition: Test suite missed the bug (all tests still pass) — indicates weak tests
+    - term: Mutation Score
+      formula: "Killed / (Killed + Survived) × 100%"
+      interpretation: Higher is better; 0% = tests prove nothing; 100% = very thorough
+
+  common_mutation_operators:
+    - category: Arithmetic operators
+      examples: ["+ → -", "* → /", "++ → --"]
+    - category: Conditional boundaries
+      examples: ["> → >=", "< → <=", "=== → !=="]
+    - category: Statement deletion
+      examples: ["Remove return statement", "Remove function call"]
+    - category: Boolean literal
+      examples: ["true → false", "false → true"]
+
+# ─────────────────────────────────────────────────────────
+# Tools
+# ─────────────────────────────────────────────────────────
+tools:
+  typescript_javascript:
+    - name: Stryker Mutator
+      packages: ["@stryker-mutator/core", "@stryker-mutator/vitest-runner"]
+      config_file: stryker.config.json
+      command: "npx stryker run"
+      strengths: [Deep vitest/jest integration, incremental mode, HTML reports]
+      note: Use incremental mode to speed up re-runs (--incremental flag)
+
+  python:
+    - name: mutmut
+      command: "mutmut run"
+      config: setup.cfg or pyproject.toml
+    - name: Cosmic Ray
+      command: "cosmic-ray init config.toml && cosmic-ray exec config.toml"
+
+  java:
+    - name: PIT (Pitest)
+      command: "mvn org.pitest:pitest-maven:mutationCoverage"
+      strengths: [Industry standard for Java, excellent IDE integration]
+
+# ─────────────────────────────────────────────────────────
+# Thresholds
+# ─────────────────────────────────────────────────────────
+thresholds:
+  description: Minimum acceptable mutation scores by module criticality
+
+  critical_modules:
+    description: Auth, payment, license validation, security controls
+    minimum_score: 80
+    enforcement: Block release if below threshold
+    examples: [auth/*, license/*, payment/*, security/*]
+
+  standard_modules:
+    description: Core business logic
+    minimum_score: 70
+    enforcement: Warning in CI; must be resolved before next release
+
+  ai_generated_tests:
+    description: Tests produced by AI tools (including this assistant)
+    minimum_score: 50
+    enforcement: Required review before accepting AI-generated test files
+    rationale: AI tends to generate hollow tests; mutation score reveals this
+
+  overall_project:
+    minimum_score: 60
+    enforcement: Advisory (track trend; alert on regression > 5%)
+
+# ─────────────────────────────────────────────────────────
+# Stryker Quick Start (TypeScript/Vitest)
+# ─────────────────────────────────────────────────────────
+stryker_quickstart:
+  install: "npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner"
+
+  minimal_config: |
+    {
+      "testRunner": "vitest",
+      "coverageAnalysis": "perTest",
+      "mutate": [
+        "src/license/**/*.ts",
+        "src/enterprise/quota/**/*.ts",
+        "src/runner/pipeline-runner.ts",
+        "!src/**/*.test.ts"
+      ],
+      "vitest": {
+        "configFile": "vitest.config.ts"
+      },
+      "thresholds": {
+        "high": 80,
+        "low": 60,
+        "break": 50
+      },
+      "reporters": ["progress", "html", "json"],
+      "htmlReporter": {
+        "fileName": "reports/mutation/index.html"
+      }
+    }
+
+  incremental_mode: "npx stryker run --incremental"
+  full_run: "npx stryker run"
+
+# ─────────────────────────────────────────────────────────
+# When to Run
+# ─────────────────────────────────────────────────────────
+execution_timing:
+  - trigger: On-demand local run
+    command: "npm run test:mutation"
+    frequency: Before committing changes to critical modules
+    note: Mutation testing is slow (minutes to hours); do NOT run in every commit hook
+
+  - trigger: Pre-release quality gate
+    command: "npm run test:mutation -- --breakAt 60"
+    frequency: Before every release
+    enforcement: Break if overall score < 60%
+
+  - trigger: Critical module change
+    command: "npx stryker run --mutate 'src/license/**'"
+    frequency: Any change to auth/license/payment/security code
+    enforcement: Must maintain ≥ 80% on changed module
+
+  - trigger: AI-generated tests acceptance
+    command: "npx stryker run --mutate [module under test]"
+    frequency: Before accepting AI-generated test PRs
+    enforcement: Score < 50% → reject; require human-written tests
+
+# ─────────────────────────────────────────────────────────
+# Rules
+# ─────────────────────────────────────────────────────────
+rules:
+  - id: mutation-pre-release
+    trigger: preparing a release
+    instruction: Run mutation testing; overall score must be ≥ 60% to proceed
+    priority: required
+
+  - id: mutation-critical-modules
+    trigger: modifying auth, license, payment, or security code
+    instruction: Run module-scoped mutation testing; maintain ≥ 80% mutation score
+    priority: required
+
+  - id: mutation-ai-generated
+    trigger: accepting AI-generated test files
+    instruction: >
+      Run mutation testing on the module under test.
+      Score < 50% → reject tests; require human-authored replacements.
+    priority: required
+
+  - id: do-not-run-in-every-commit
+    trigger: planning CI pipeline
+    instruction: Do NOT add mutation testing to commit hooks or every-PR CI; it is too slow
+    priority: required
+    note: Reserve for pre-release gate and on-demand runs
+
+anti_patterns:
+  - Treating 100% line coverage as sufficient (lines covered ≠ mutations killed)
+  - Adding mutation testing to pre-commit hooks (makes commits 10-60 minutes long)
+  - Accepting AI-generated tests without mutation score validation
+  - Killing mutations by adding trivial assertions (expect(x).toBeDefined())
+  - Targeting only happy paths in mutation testing (branches and boundaries are key)
+
+quick_reference:
+  mutation_testing_checklist: |
+    □ Stryker configured for critical modules (license/*, auth/*, quota/*)
+    □ test:mutation script in package.json
+    □ Thresholds set: critical ≥ 80%, overall ≥ 60%, break at 50%
+    □ Pre-release: run full mutation suite before tagging version
+    □ AI-generated tests: validate with mutation score before accepting
+    □ NOT in commit hooks (too slow)

package/bundled/ai/standards/pii-classification.ai.yaml

@@ -0,0 +1,109 @@
+# PII Classification and Handling Standards - AI Optimized
+# Source: XSPEC-066 Wave 3 Compliance Pack
+
+id: pii-classification
+title: PII Classification and Handling Standards
+version: "1.0.0"
+status: Active
+tags: [compliance, privacy, pii, gdpr, data-protection, security]
+summary: |
+  Defines how Personally Identifiable Information (PII) and sensitive personal
+  data is classified, labeled, stored, transmitted, and disposed of. Covers
+  a three-tier data sensitivity classification, mandatory handling controls
+  per tier, data minimization principles, consent management requirements,
+  retention and deletion schedules, and cross-border transfer restrictions.
+  Aligned with GDPR Article 9, CCPA, and general privacy-by-design principles.
+
+requirements:
+  - id: REQ-001
+    title: PII Data Sensitivity Classification
+    description: |
+      All data fields containing personal information MUST be classified into
+      one of three tiers before storage or processing. TIER-1 (Highly
+      Sensitive): health data, financial account numbers, government IDs,
+      biometrics, passwords, SSNs — requires encryption at rest and in
+      transit, access logging, no caching. TIER-2 (Sensitive): full name +
+      contact info combination, location history, behavioral profiles,
+      IP addresses — requires encryption in transit, access controls.
+      TIER-3 (General PII): first name only, country-level location, general
+      demographics — standard access controls sufficient.
+    level: MUST
+    examples:
+      - "Field: credit_card_number → TIER-1, encrypted AES-256-GCM, no logging of value"
+      - "Field: user_email + user_name together → TIER-2, TLS required, RBAC enforced"
+      - "Field: country_code → TIER-3, standard DB access controls"
+
+  - id: REQ-002
+    title: Data Minimization and Purpose Limitation
+    description: |
+      Systems MUST collect only the minimum PII necessary for the explicitly
+      stated purpose. Each PII field in the data model MUST have a documented
+      business purpose and legal basis (consent, contract, legitimate
+      interest, legal obligation). Collection of PII without documented
+      purpose is PROHIBITED. Purpose limitation MUST be enforced: data
+      collected for purpose A MUST NOT be used for unrelated purpose B
+      without separate consent.
+    level: MUST
+    examples:
+      - "Data dictionary entry: email_address, purpose: account authentication, legal_basis: contract"
+      - "Phone number collected for 2FA cannot be reused for marketing without new consent"
+      - "PR review checklist: 'Does this new field have a documented purpose in the data dictionary?'"
+
+  - id: REQ-003
+    title: PII Masking and Anonymization in Non-Production
+    description: |
+      PII MUST NOT exist in non-production environments (development, staging,
+      test) unless explicitly required and approved. Test and staging databases
+      MUST use anonymized or synthetic data. Any approved exception MUST be
+      time-limited, access-controlled, and documented. PII MUST be masked
+      in application logs: email addresses shown as u***@domain.com, phone
+      numbers as +1-XXX-XXX-1234, card numbers as ****-****-****-1234.
+    level: MUST
+    examples:
+      - "Staging DB: email stored as 'user_12345@test.invalid', not real email"
+      - "Log output: 'User u***@example.com logged in' not 'User alice@example.com logged in'"
+      - "Exception process: production data copy to staging requires security team approval + 7-day TTL"
+
+  - id: REQ-004
+    title: Data Retention and Deletion Schedule
+    description: |
+      Every data category containing PII MUST have a documented retention
+      schedule with maximum retention period aligned to legal requirements
+      and business need. Automated deletion MUST be implemented for data
+      past its retention period. Deletion MUST be verifiable (deletion
+      receipts or audit logs). Users exercising right-to-erasure MUST
+      receive deletion confirmation within 30 days (GDPR) or 45 days (CCPA).
+    level: MUST
+    examples:
+      - "Customer account data: retained 7 years after account closure (tax requirements)"
+      - "Session tokens: deleted after 24 hours of inactivity via automated cron job"
+      - "Right-to-erasure request: user data purged from all systems within 25 days, confirmation email sent"
+
+  - id: REQ-005
+    title: Cross-Border Data Transfer Controls
+    description: |
+      Transfers of TIER-1 or TIER-2 PII across national borders MUST comply
+      with applicable transfer mechanisms. EU → non-adequate country transfers
+      MUST use Standard Contractual Clauses (SCCs) or Binding Corporate Rules.
+      Data residency requirements MUST be documented in the system design.
+      Cross-border transfers MUST be logged with destination country and
+      legal basis.
+    level: MUST
+    examples:
+      - "EU user data stored in AWS eu-west-1, not replicated to us-east-1 without SCC"
+      - "Transfer log: destination=US, mechanism=SCC-2021, purpose=customer-support, timestamp=..."
+      - "Architecture doc notes: 'All PII stored in EU region per GDPR Article 46'"
+
+  - id: REQ-006
+    title: PII Impact Assessment for New Features
+    description: |
+      Any new feature or system change that introduces new PII collection or
+      processing SHOULD undergo a Privacy Impact Assessment (PIA) before
+      implementation. The PIA MUST document: what PII is collected, purpose,
+      legal basis, retention period, third-party sharing, and risk mitigations.
+      Features with TIER-1 PII require mandatory PIA; TIER-2 is recommended.
+    level: SHOULD
+    examples:
+      - "New feature: 'Save payment method' → PIA required (TIER-1 card data)"
+      - "PIA template: docs/templates/privacy-impact-assessment.md"
+      - "PIA outcome: fingerprint auth approved with biometric data stored only on-device"