ai-core-framework 0.1.0

package/core/rules/05-testing-mandatory.md

@@ -0,0 +1,374 @@
+# 🔒 RULE 05: Testing Mandatory (Strict)
+
+> **No code without tests**. Non-negotiable.
+> Enforced by: CI coverage gates + tech-lead review + pre-commit hooks.
+
+---
+
+## 🎯 Core Principle
+
+**Untested code = technical debt shipped to production.**
+
+Tests are:
+- Documentation for behavior
+- Regression safety net
+- Refactoring enabler
+- Design feedback (hard to test = bad design)
+
+---
+
+## 🔒 Rules
+
+### RULE TEST-001: TDD mandatory
+
+Mandatory workflow for every new feature/fix:
+
+```
+1. RED: Write failing test
+   - Test reflects AC scenario
+   - Run test → MUST fail
+   - Commit: test(TICKET-XXX): add failing test for <scenario>
+
+2. GREEN: Minimum code to pass
+   - Write simplest code to make test pass
+   - No extra features
+   - Run test → MUST pass
+   - Commit: feat(TICKET-XXX): implement <scenario>
+
+3. REFACTOR: Improve code
+   - Extract, rename, dedupe
+   - Tests must still pass
+   - Commit: refactor(TICKET-XXX): <what improved>
+```
+
+**Evidence of TDD** in commit history:
+- `test(...)` commit before `feat(...)` commit
+- CI can check via commit order analysis
+
+**Exception**: Exploratory spike tickets (explicit type: `spike`). Must convert to TDD when real implementation starts.
+
+### RULE TEST-002: Coverage thresholds
+
+Configured in `config/project-config.yaml`:
+
+| Metric | Threshold |
+|--------|-----------|
+| **Diff coverage** (new/changed lines in PR) | ≥ 80% |
+| **Overall coverage** (whole codebase) | ≥ 70% |
+| **Critical paths** (auth, payment, data) | ≥ 95% |
+
+**Enforcement**:
+- CI fails PR if diff coverage < 80%
+- CI fails if overall drops > 1% from main
+- Tech Lead review: critical path coverage ≥ 95%
+
+**What counts as "covered"**:
+- ✅ Line executed by test
+- ✅ Branch taken (both true/false)
+- ✅ Meaningful assertion on outcome
+
+**What doesn't count**:
+- ❌ Line executed but no assertion (hitting code without verifying)
+- ❌ Mocked behavior only (must test real logic)
+- ❌ `console.log` statements
+
+### RULE TEST-003: Test pyramid
+
+Proportions:
+```
+        /\
+       /E2E\          10%  (critical user journeys only)
+      /------\
+     /Integra-\       20%  (API + DB + external mocks)
+    /tion     \
+   /----------\
+  /   Unit     \      70%  (pure logic, fast, isolated)
+ /--------------\
+```
+
+**Why**: Speed, reliability, cost.
+- Unit: ms to run, 100% reliable
+- Integration: sec, 99% reliable
+- E2E: min, 95% reliable (flakiness risk)
+
+### RULE TEST-004: Test quality
+
+Every test **MUST**:
+
+- [ ] **Descriptive name**: `should_<expected>_when_<condition>`
+  - ✅ `should_return_401_when_token_expired`
+  - ❌ `testAuth`, `test1`
+
+- [ ] **AAA structure**: Arrange, Act, Assert
+  ```javascript
+  it('should return 401 when token expired', async () => {
+    // Arrange
+    const expiredToken = createToken({ exp: Date.now() - 1000 });
+    
+    // Act
+    const response = await request.get('/api/me')
+      .set('Authorization', `Bearer ${expiredToken}`);
+    
+    // Assert
+    expect(response.status).toBe(401);
+    expect(response.body.error).toBe('TOKEN_EXPIRED');
+  });
+  ```
+
+- [ ] **1 concept per test**: Multiple `expect` OK when they verify related state
+- [ ] **No shared mutable state**: Each test independent, parallel-safe
+- [ ] **Minimal mocks**: Test real code when possible. Mock only:
+  - External services (HTTP, DB in unit tests)
+  - Time (`Date.now()`)
+  - Randomness
+- [ ] **Deterministic**: No flakiness. No `setTimeout` for "waiting".
+
+### RULE TEST-005: Coverage for every AC
+
+Every AC scenario in the ticket **MUST** have corresponding test(s):
+
+```
+Ticket AC:
+  Scenario 1: Happy path     → Test 1
+  Scenario 2: Edge case      → Test 2
+  Scenario 3: Error case     → Test 3
+```
+
+PR review checks: "AC coverage: N/N scenarios" MUST be 100%.
+
+### RULE TEST-006: No disabled tests
+
+**MUST NOT** commit:
+- `.skip`, `xit`, `describe.skip`, `it.skip`
+- `@Disabled` (Java)
+- `@pytest.mark.skip` (Python)
+- Commented-out test files
+
+**Exception**: If temporary skipping is required (known broken, fix in next ticket):
+- Add `TICKET-XXX` in comment:
+  ```javascript
+  it.skip('TICKET-099: should handle concurrent requests', () => {
+    // Skipped pending rate limiter refactor, see TICKET-099
+  });
+  ```
+- Create the follow-up ticket
+- Max 1 sprint for skip duration
+
+### RULE TEST-007: Failing tests block merge
+
+**MUST NOT** merge PR with failing tests. Even "unrelated flaky tests".
+
+If test is flaky → fix flakiness first (add to tech-debt backlog if can't fix immediately).
+
+### RULE TEST-008: Bug fix requires regression test
+
+Every bug fix **MUST** include test that:
+1. Fails before fix
+2. Passes after fix
+3. Named clearly references the bug: `should_<correct_behavior>_regression_TICKET-XXX`
+
+Example:
+```javascript
+describe('login endpoint - regression tests', () => {
+  it('should handle email with + character - regression TICKET-089', async () => {
+    // This test prevents regression of bug where + in email caused 500
+    const response = await login('user+tag@example.com', 'password');
+    expect(response.status).toBe(200);
+  });
+});
+```
+
+### RULE TEST-009: Critical paths require extra rigor
+
+**Critical paths** (authentication, authorization, payments, PII handling):
+
+- Coverage ≥ 95%
+- Include security tests:
+  - Privilege escalation attempts
+  - Injection attempts
+  - Rate limit bypass attempts
+- Include concurrency tests (race conditions)
+- Include failure mode tests:
+  - What if DB is down?
+  - What if external service is slow?
+  - What if token is malformed?
+
+### RULE TEST-010: Performance tests for hot paths
+
+Endpoints on critical user flow **MUST** have performance tests:
+
+- **p50** (median): < 200ms
+- **p95**: < 500ms
+- **p99**: < 1000ms
+
+Thresholds are customizable in `config/project-config.yaml`.
+
+CI runs perf tests on main branch, alerts if regression > 20%.
+
+### RULE TEST-011: Test data management
+
+- **MUST NOT** use production data in tests
+- **MUST NOT** commit real PII (even test accounts)
+- Use factories/builders for test data:
+  ```javascript
+  const user = userFactory.build({ email: 'test@example.com' });
+  ```
+- Seed data deterministic, idempotent
+
+### RULE TEST-012: Test types matrix
+
+| Ticket Type | Unit | Integration | E2E | Security | Performance |
+|-------------|------|-------------|-----|----------|-------------|
+| Feature (UI) | ✓ | ✓ | ✓ (if critical) | - | - |
+| Feature (API) | ✓ | ✓ | - | ✓ (if sensitive) | ✓ (if hot path) |
+| Bug fix | ✓ (regression) | ✓ (if integration bug) | - | - | - |
+| Refactor | ✓ (existing preserved) | - | - | - | ✓ (if hot path) |
+| Security fix | ✓ | ✓ | - | ✓ (mandatory) | - |
+| Performance fix | ✓ | - | - | - | ✓ (mandatory) |
+
+---
+
+## 🛠️ Required Test Infrastructure
+
+### Unit tests
+- Framework: Project-specific (Vitest/Jest/Pytest/JUnit)
+- Run: `pnpm test:unit` (or equivalent)
+- Fast: < 10 sec for whole suite
+- Parallel-safe
+
+### Integration tests
+- Framework: Same as unit, with extended setup
+- DB: Test container (testcontainers, Docker) or in-memory (SQLite)
+- External services: Mocked (MSW, nock) or test containers
+- Run: `pnpm test:integration`
+
+### E2E tests
+- Framework: Playwright (recommended) or Cypress
+- Environment: Staging-like
+- Run: `pnpm test:e2e`
+- Only run on PR + nightly
+
+### Coverage tool
+- Collect coverage: during unit + integration tests
+- Report format: `lcov` + HTML
+- Upload to Codecov/Coveralls (optional)
+
+---
+
+## 🔧 CI Configuration (example GitHub Actions)
+
+```yaml
+name: Tests
+on: [pull_request]
+
+jobs:
+  unit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: pnpm/action-setup@v2
+      - run: pnpm install
+      - run: pnpm test:unit --coverage
+      - name: Check coverage threshold
+        run: |
+          coverage=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
+          if (( $(echo "$coverage < 70" | bc -l) )); then
+            echo "❌ Coverage $coverage% below 70% threshold"
+            exit 1
+          fi
+      - name: Check diff coverage
+        run: pnpm run coverage:diff
+      
+  integration:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:15
+    steps:
+      - uses: actions/checkout@v4
+      - run: pnpm install
+      - run: pnpm test:integration
+      
+  e2e:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: pnpm install
+      - run: pnpm test:e2e
+```
+
+---
+
+## 💡 Testing Anti-patterns
+
+❌ **Tautological tests** (test mocks its own mock):
+```javascript
+it('should call service', () => {
+  const spy = vi.spyOn(service, 'doThing');
+  controller.handle();
+  expect(spy).toHaveBeenCalled();  // Useless - just verifies wiring
+});
+```
+✅ Test real behavior and outcome.
+
+❌ **Over-mocking**:
+```javascript
+// Don't mock everything
+vi.mock('../user-service');
+vi.mock('../email-service');
+vi.mock('../token-service');
+// Now you're testing mocks, not code
+```
+✅ Mock boundaries (external services only).
+
+❌ **Brittle selectors** in E2E:
+```javascript
+await page.click('.btn-primary.mt-4.px-6');  // ❌ Breaks on CSS change
+```
+✅ Use data-testid: `await page.click('[data-testid="submit-btn"]')`
+
+❌ **Copy-paste tests**:
+```javascript
+it('test 1', () => { /* nearly identical */ });
+it('test 2', () => { /* nearly identical */ });
+```
+✅ Use `it.each`:
+```javascript
+it.each([
+  { input: 'a', expected: 1 },
+  { input: 'b', expected: 2 },
+])('should return $expected for $input', ({ input, expected }) => {
+  expect(fn(input)).toBe(expected);
+});
+```
+
+---
+
+## 🚨 Violation Consequences
+
+| Violation | Consequence |
+|-----------|-------------|
+| Committed code without test | CI blocks PR (coverage < 80% diff) |
+| Disabled test without ticket | PR review rejects |
+| Flaky test ignored | Tech-debt ticket mandatory |
+| Production data in test | IMMEDIATE: remove, rotate if PII, incident report |
+| Mock-heavy tests | Tech-lead flags in review, refactor required |
+
+---
+
+## 📊 Metrics to Track
+
+- Coverage trend (weekly)
+- Test suite duration trend (weekly)
+- Flaky test count (weekly)
+- Bugs found in production that had no regression test (post-mortem)
+- TDD adherence (commit order analysis)
+
+Dashboard: `/sprint-report --tests`
+
+---
+
+**Version**: 1.0.0
+**Last updated**: 2026-04-18
+**Maintainer**: Tech Lead + QA Lead
+**Next review**: End of each sprint