@qball-inc/the-bulwark 1.0.0

package/skills/mock-detection/references/false-positive-prevention.md

@@ -0,0 +1,402 @@
+# False Positive Prevention Reference
+
+Two-tier allowlist and decision tree for mock-detection Stage 2 analysis. Prevents flagging legitimate test patterns as T1-T4 violations.
+
+---
+
+## Tier 1: Universal Safe (Never Flag)
+
+These patterns are safe regardless of test type. Do not flag as violations.
+
+### Test Framework Primitives
+
+| Pattern | Why Safe |
+|---------|----------|
+| `describe()`, `it()`, `test()` | Test structure, not test doubles |
+| `beforeEach()`, `afterEach()` | Lifecycle hooks |
+| `beforeAll()`, `afterAll()` | Suite-level setup/teardown |
+| `expect()`, `assert()` | Assertion primitives |
+| `jest.setTimeout()`, `vi.setConfig()` | Framework configuration |
+| `jest.useFakeTimers()` | Timer control (not mocking SUT) |
+| `jest.clearAllMocks()` | Cleanup utility |
+
+### Test Data Construction
+
+| Pattern | Why Safe |
+|---------|----------|
+| `*Builder` classes | Construct input data, not replacement behavior |
+| `*Factory` functions | `UserFactory.create()`, `OrderFactory.build()` |
+| `create*()` data factories | `createTestUser()`, `createSampleOrder()` |
+| `build*()` data factories | `buildConfig()`, `buildPayload()` |
+| `make*()` data factories | `makeUser()`, `makeRequest()` |
+| `generate*()` data factories | `generateToken()`, `generateId()` |
+| Faker/Chance libraries | `faker.person.fullName()`, `chance.email()` |
+| Literal test data | `const input = { name: 'Alice', age: 30 }` in unit tests |
+
+### Assertion Utilities
+
+| Pattern | Why Safe |
+|---------|----------|
+| Custom matchers | `expect.extend({ toBeValidEmail })` |
+| Snapshot utilities | `expect(result).toMatchSnapshot()` |
+| `supertest` / `pactum` | HTTP assertion libraries (test real endpoints) |
+| `testing-library` queries | `screen.getByText()`, `render()` |
+
+### Logging and Instrumentation
+
+| Pattern | Why Safe |
+|---------|----------|
+| Test loggers | `const logger = createTestLogger()` — observability, not behavior replacement |
+| Console suppression | `jest.spyOn(console, 'error').mockImplementation(() => {})` — noise reduction |
+| Performance timers | `performance.mark()`, `performance.measure()` |
+
+**Exception for console suppression**: If a test asserts on console output (e.g., `expect(console.error).toHaveBeenCalledWith('specific message')`), this becomes a T2 pattern. Flag only if there is no accompanying result assertion.
+
+---
+
+## Tier 2: Context-Dependent (Check Test Type)
+
+These patterns require evaluating the test type before deciding.
+
+### HTTP Interception (nock / MSW)
+
+```typescript
+// MSW setup
+const server = setupServer(
+  rest.get('https://api.external.com/users', (req, res, ctx) => {
+    return res(ctx.json([{ id: 1, name: 'Alice' }]));
+  })
+);
+```
+
+| Test Type | Verdict | Rationale |
+|-----------|---------|-----------|
+| Unit test | Safe | Isolating external HTTP is expected |
+| Integration test (external API) | Safe | Mocking third-party APIs you don't control is acceptable |
+| Integration test (own services) | **Flag T3** | Mocking your own service boundaries defeats integration testing |
+| E2E test | **Flag** | E2E should hit real endpoints |
+
+**Decision rule**: Check the intercepted URL. If it points to an external third-party service, safe. If it points to the application's own API or internal services, flag.
+
+### InMemory* Fakes
+
+```typescript
+class InMemoryUserRepository implements UserRepository {
+  private users = new Map<string, User>();
+  async findById(id: string) { return this.users.get(id); }
+  async save(user: User) { this.users.set(user.id, user); }
+}
+```
+
+| Test Type | Verdict | Rationale |
+|-----------|---------|-----------|
+| Unit test | Safe | Isolating persistence for unit logic is expected |
+| Integration test (repo layer) | **Flag T3** | If testing repository integration, use real database |
+| Integration test (service layer) | Context-dependent | OK if repo isn't the boundary under test |
+
+**Decision rule**: Is the InMemory replacement sitting at the integration boundary the test claims to verify? If yes, flag. If it's a supporting dependency not under test, safe.
+
+### Dependency Injection Test Doubles
+
+```typescript
+// Test provides a mock via DI
+const service = new OrderService(mockPaymentGateway, mockInventory);
+```
+
+| Test Type | Verdict | Rationale |
+|-----------|---------|-----------|
+| Unit test | Safe | DI-based isolation is textbook unit testing |
+| Integration test | **Evaluate** | Which dependencies are mocked? Are they the integration boundaries? |
+| E2E test | **Flag** | DI should use real implementations |
+
+**Decision rule**: In integration tests, flag DI mocks only for the specific boundaries the test claims to integrate. Supporting dependencies that aren't part of the integration scope are acceptable.
+
+### Test Containers
+
+```typescript
+const pgContainer = await new PostgreSqlContainer().start();
+const redisContainer = await new GenericContainer('redis:7').start();
+```
+
+| Test Type | Verdict | Rationale |
+|-----------|---------|-----------|
+| All types | **Always safe** | Real infrastructure, not a test double |
+
+### Environment/Config Overrides
+
+```typescript
+process.env.DATABASE_URL = 'postgres://localhost:5432/test';
+process.env.API_KEY = 'test-key-not-real';
+```
+
+| Test Type | Verdict | Rationale |
+|-----------|---------|-----------|
+| All types | Safe | Configuration, not behavior replacement |
+
+---
+
+## Decision Tree
+
+Use this tree to evaluate whether a detected pattern is a violation.
+
+```
+Is it a test framework primitive or assertion utility?
+  YES -> SAFE (Tier 1)
+  NO  -> continue
+
+Is it test data construction (factory, builder, literal)?
+  YES -> SAFE (Tier 1)
+  NO  -> continue
+
+Is it a test container or real infrastructure?
+  YES -> SAFE (Tier 1)
+  NO  -> continue
+
+Is it console/logging suppression without assertion on output?
+  YES -> SAFE (Tier 1)
+  NO  -> continue
+
+--- Beyond this point: context-dependent (Tier 2) ---
+
+Identify the TEST TYPE (unit / integration / E2E):
+  - File name contains `.unit.` or is in `__tests__/unit/` -> UNIT
+  - File name contains `.integration.` or `.int.` -> INTEGRATION
+  - File name contains `.e2e.` or `.spec.` with E2E markers -> E2E
+  - Otherwise -> infer from test content (imports, setup patterns)
+
+Is it HTTP interception (nock, MSW, fetch mock)?
+  UNIT test        -> SAFE
+  INTEGRATION test -> Is the intercepted URL your own service?
+    YES -> FLAG T3 (mocking own integration boundary)
+    NO  -> SAFE (external API mock is acceptable)
+  E2E test         -> FLAG
+
+Is it an InMemory*/Fake* class?
+  UNIT test        -> SAFE
+  INTEGRATION test -> Does it replace the boundary under test?
+    YES -> FLAG T3
+    NO  -> SAFE (supporting dependency)
+  E2E test         -> FLAG
+
+Is it a DI-injected mock?
+  UNIT test        -> SAFE
+  INTEGRATION test -> Is the mocked dependency an integration boundary?
+    YES -> FLAG T3
+    NO  -> SAFE
+  E2E test         -> FLAG
+
+Is it jest.mock()/vi.mock() on a module?
+  Does it mock the module under test?
+    YES -> FLAG T1 (mocking SUT)
+    NO  -> Is this an integration test mocking an integration boundary?
+      YES -> FLAG T3
+      NO  -> SAFE (isolating irrelevant dependency)
+
+None of the above matched?
+  -> FLAG as confidence: medium, recommend manual review
+```
+
+---
+
+## Decision Tree (YAML Encoding)
+
+For structured consumption by the detection agent:
+
+```yaml
+decision_tree:
+  - check: "test_framework_primitive"
+    description: "describe/it/test/expect/beforeEach/afterEach/beforeAll/afterAll"
+    verdict: safe
+    tier: 1
+
+  - check: "test_data_construction"
+    description: "*Builder, *Factory, create*/build*/make*/generate*, faker, literal input data"
+    verdict: safe
+    tier: 1
+
+  - check: "test_container_or_real_infra"
+    description: "testcontainers, real database connections, real servers"
+    verdict: safe
+    tier: 1
+
+  - check: "console_logging_suppression"
+    description: "jest.spyOn(console, ...) without assertion on output"
+    verdict: safe
+    tier: 1
+
+  - check: "http_interception"
+    description: "nock, MSW, jest.mock('node-fetch')"
+    conditions:
+      - test_type: unit
+        verdict: safe
+      - test_type: integration
+        sub_check: "intercepted URL is own service?"
+        if_yes: "flag_T3"
+        if_no: "safe"
+      - test_type: e2e
+        verdict: flag
+
+  - check: "inmemory_fake_class"
+    description: "InMemory*, Fake* implements Interface"
+    conditions:
+      - test_type: unit
+        verdict: safe
+      - test_type: integration
+        sub_check: "replaces boundary under test?"
+        if_yes: "flag_T3"
+        if_no: "safe"
+      - test_type: e2e
+        verdict: flag
+
+  - check: "di_injected_mock"
+    description: "new Service(mockDep, fakeDep)"
+    conditions:
+      - test_type: unit
+        verdict: safe
+      - test_type: integration
+        sub_check: "mocked dep is integration boundary?"
+        if_yes: "flag_T3"
+        if_no: "safe"
+      - test_type: e2e
+        verdict: flag
+
+  - check: "module_mock"
+    description: "jest.mock('./module'), vi.mock('./module')"
+    conditions:
+      - sub_check: "mocks module under test?"
+        if_yes: "flag_T1"
+        if_no:
+          sub_check: "integration test mocking boundary?"
+          if_yes: "flag_T3"
+          if_no: "safe"
+
+  - check: "unmatched_pattern"
+    description: "Pattern not covered by above rules"
+    verdict: "flag_medium_confidence"
+    note: "Recommend manual review"
+```
+
+---
+
+## Common False Positive Scenarios
+
+These patterns have been observed to trigger false positives. The detection agent should recognize and skip them.
+
+### 1. Supertest / Pactum Request Chains
+
+```typescript
+// NOT a violation - supertest creates real HTTP requests to real server
+const response = await request(app)
+  .get('/api/users')
+  .expect(200)
+  .expect('Content-Type', /json/);
+```
+
+**Why flagged incorrectly**: `request(app)` looks like it might be wrapping the app in a test double. It's actually creating a real HTTP connection.
+
+### 2. React Testing Library Render
+
+```typescript
+// NOT a violation - renders real component in real DOM
+const { getByText, getByRole } = render(<LoginForm onSubmit={mockSubmit} />);
+```
+
+**Why flagged incorrectly**: `render()` might look like it creates a fake DOM. It uses jsdom, which is real DOM behavior. The `mockSubmit` callback is an acceptable mock for a unit test.
+
+### 3. Timer Mocking
+
+```typescript
+jest.useFakeTimers();
+// ... test code ...
+jest.advanceTimersByTime(1000);
+```
+
+**Why flagged incorrectly**: "Fake" in the method name. Timer mocking is framework-supported test control, not SUT mocking.
+
+### 4. Module Re-exports in Test Utilities
+
+```typescript
+// test-utils.ts
+export { render, screen } from '@testing-library/react';
+export { userEvent } from '@testing-library/user-event';
+export const testDb = createTestDatabase();
+```
+
+**Why flagged incorrectly**: Test utility file might look like a mock factory. It's re-exporting real libraries.
+
+### 5. Snapshot Testing
+
+```typescript
+expect(component).toMatchInlineSnapshot(`
+  <div class="user-card">
+    <span>Alice</span>
+  </div>
+`);
+```
+
+**Why flagged incorrectly**: Inline snapshots contain hardcoded HTML that might look like manually constructed test data. Snapshots verify real output.
+
+---
+
+## Worked Example: Mixed-Type File
+
+A single test file may contain both unit and integration sections. The **same mock pattern** can be safe or a violation depending on which section it appears in.
+
+### Scenario
+
+`error-handler.test.ts` contains:
+- **Lines 1-100**: Unit tests for `categorizeError`, `calculateBackoff`, `withRetry`, `ErrorHandlerService`
+- **Lines 105-150**: Integration tests under `describe('Error Handling Integration', ...)`
+
+### The Pattern: `jest.fn().mockResolvedValue()`
+
+**In the unit section (SAFE — no violation):**
+```typescript
+describe('Retry Mechanism', () => {
+  it('should succeed on first attempt', async () => {
+    const operation = jest.fn().mockResolvedValue('success');
+    const result = await withRetry(operation, { maxAttempts: 3 });
+    expect(result).toBe('success');
+  });
+});
+```
+- `operation` is an **injected dependency** (callback parameter to `withRetry`)
+- The SUT is `withRetry`, not `operation`
+- Mocking injected deps in unit tests is appropriate
+- **Verdict: SAFE** (Tier 1 Universal Safe — injected dependency in unit test)
+
+**In the integration section (T3 VIOLATION):**
+```typescript
+describe('Error Handling Integration', () => {
+  it('should handle transient errors with retry and recovery', async () => {
+    const operation = jest.fn()
+      .mockRejectedValueOnce(new Error('ECONNRESET'))
+      .mockResolvedValue({ data: 'recovered' });
+    const result = await service.executeWithRetry(operation, { maxAttempts: 3 });
+    expect(result).toEqual({ data: 'recovered' });
+  });
+});
+```
+- `operation` is the **integration boundary** — the external system call
+- Integration tests exist to verify real system interactions
+- Mocking the boundary defeats the purpose of the integration test
+- **Verdict: T3 VIOLATION** (Mock at integration boundary)
+
+### Key Takeaway
+
+Never classify an entire file as one test type. Evaluate each describe block independently against the rubric for **its** test type.
+
+If AST integration-mock metadata is available (from `just integration-mocks`), it provides ground truth for section boundaries and mock locations within integration/e2e blocks.
+
+---
+
+## Summary for LLM Agent
+
+When evaluating a potential violation:
+
+1. **Check Tier 1 first** — if the pattern matches Universal Safe, skip immediately
+2. **Identify test type** — unit, integration, or E2E (file name, directory, or content)
+3. **Walk the decision tree** for Tier 2 patterns — context determines the verdict
+4. **When uncertain**, flag as `confidence: medium` with a note explaining the ambiguity
+5. **Never flag** test data factories, test containers, or framework primitives
+6. **Always flag** mocking the system under test (T1) regardless of test type

package/skills/mock-detection/references/stub-patterns.md

@@ -0,0 +1,236 @@
+# Stub Patterns Reference
+
+Extended test double detection patterns for mock-detection Stage 2 analysis. Covers the Meszaros test double taxonomy beyond basic `jest.mock()` / `vi.mock()` patterns.
+
+---
+
+## Meszaros Test Double Taxonomy
+
+| Type | Purpose | Detection Priority |
+|------|---------|-------------------|
+| **Dummy** | Fills parameter lists, never used | Low (rarely harmful) |
+| **Stub** | Returns canned answers | Medium (may hide real behavior) |
+| **Spy** | Records calls for later verification | Medium (T2 risk if no result check) |
+| **Mock** | Pre-programmed expectations | High (T1/T3 risk) |
+| **Fake** | Working implementation with shortcuts | Context-dependent |
+
+---
+
+## Naming Convention Patterns
+
+### Direct Name Matches
+
+These naming patterns indicate test doubles. When found in integration tests, evaluate whether they break the integration chain.
+
+| Pattern | Regex | Examples |
+|---------|-------|----------|
+| `Mock*` | `/^[Mm]ock[A-Z]/` | `MockDatabase`, `mockUserService`, `MockHttpClient` |
+| `Stub*` | `/^[Ss]tub[A-Z]/` | `StubRepository`, `stubAuthProvider`, `StubCache` |
+| `Fake*` | `/^[Ff]ake[A-Z]/` | `FakeFileSystem`, `fakeEmailSender`, `FakeQueue` |
+| `InMemory*` | `/^[Ii]n[Mm]emory[A-Z]/` | `InMemoryDatabase`, `InMemoryCache`, `InMemoryEventBus` |
+| `Dummy*` | `/^[Dd]ummy[A-Z]/` | `DummyLogger`, `dummyConfig`, `DummyTransport` |
+| `Spy*` | `/^[Ss]py[A-Z]/` | `SpyLogger`, `spyNotifier` |
+| `*Builder` | `/[A-Z]\w+Builder$/` | `UserBuilder`, `OrderBuilder`, `ConfigBuilder` |
+| `Test*` | `/^[Tt]est[A-Z]/` | `TestServer`, `TestDatabase`, `TestHelper` |
+
+### Counter-Examples (NOT test doubles)
+
+These match naming patterns but are NOT test doubles:
+
+| Name | Why It's Not a Double |
+|------|----------------------|
+| `mockImplementation()` | Jest API method, not a variable |
+| `InMemoryCache` in production code | Real implementation choice, not a test shortcut |
+| `TestUtils.formatDate()` | Test utility, not a replacement for production code |
+| `buildUser()` in a factory module | Production factory, not test-only builder |
+| `stubborn`, `mockingbird` | English words, not test double prefixes |
+
+**Rule**: Match on PascalCase/camelCase boundaries only. `mockUser` matches; `mockingbird` does not.
+
+---
+
+## Class Hierarchy Detection
+
+### Implements/Extends Patterns
+
+Classes that implement interfaces or extend base classes as test replacements:
+
+```typescript
+// DETECT: Class implements interface with test-double name
+class FakeDatabase implements DatabasePort {
+  private data = new Map<string, unknown>();
+  async get(key: string) { return this.data.get(key); }
+  async set(key: string, value: unknown) { this.data.set(key, value); }
+}
+
+// DETECT: Class extends base with override
+class StubEmailService extends EmailService {
+  override async send() { return { sent: true }; }
+}
+
+// DETECT: Abstract test base
+class TestRepository<T> implements Repository<T> {
+  protected items: T[] = [];
+  async findAll() { return this.items; }
+}
+```
+
+**Detection signals** (AST-level):
+1. Class name matches a test double naming pattern (Fake*, Stub*, Mock*, Test*, InMemory*)
+2. Class implements an interface OR extends a base class
+3. Class defined in a test file or `__tests__/` directory
+
+**Violation evaluation**:
+- In unit tests: Generally acceptable (isolating external dependencies)
+- In integration tests: Flag if the class replaces an integration boundary the test claims to verify
+
+### Manual Stub Without Naming Convention
+
+Some developers create test doubles without naming conventions:
+
+```typescript
+// No Fake/Stub/Mock prefix, but still a test double
+const database = {
+  query: async () => [{ id: 1, name: 'test' }],
+  insert: async () => ({ id: 1 }),
+  delete: async () => true,
+};
+
+// Replacement object matching an interface shape
+const logger = {
+  info: () => {},
+  warn: () => {},
+  error: () => {},
+};
+```
+
+**Detection signals**:
+1. Object literal assigned to variable with interface-matching shape
+2. All methods are no-ops (`() => {}`) or return hardcoded values
+3. Variable used where a dependency injection parameter is expected
+
+**Note**: These are harder to detect deterministically. Flag as `confidence: medium` for LLM review.
+
+---
+
+## Factory Function Patterns
+
+### Test Data Factories (Usually Safe)
+
+Factory functions that create test input data are generally NOT violations:
+
+```typescript
+// SAFE: Creates input data for the system under test
+function createTestUser(overrides = {}) {
+  return { id: 'user-1', name: 'Test User', email: 'test@example.com', ...overrides };
+}
+
+// SAFE: Builder pattern for test data
+const user = new UserBuilder().withName('Alice').withRole('admin').build();
+
+// SAFE: Faker/factory-based generation
+const order = OrderFactory.create({ status: 'pending' });
+```
+
+**Why safe**: These create *input data*, not *replacement behavior*. The system under test still processes this data through real code paths.
+
+### Test Infrastructure Factories (Context-Dependent)
+
+Factory functions that create test infrastructure need evaluation:
+
+```typescript
+// CONTEXT-DEPENDENT: Creates a test server
+function createTestServer(routes) {
+  const app = express();
+  routes.forEach(r => app.get(r.path, r.handler));
+  return app.listen(0); // Random port
+}
+
+// CONTEXT-DEPENDENT: Creates an in-memory database
+function createTestDatabase(seed = []) {
+  const db = new Map();
+  seed.forEach(item => db.set(item.id, item));
+  return { get: (id) => db.get(id), set: (id, v) => db.set(id, v) };
+}
+```
+
+**Evaluation rule**: If the factory replaces a real dependency that the test claims to verify integration with, it's a violation. If it provides supporting infrastructure (e.g., a test HTTP server to receive real requests), it's acceptable.
+
+### Factory Naming Conventions
+
+| Pattern | Regex | Typically Safe? |
+|---------|-------|-----------------|
+| `create*` | `/^create[A-Z]/` | Yes (data factory) |
+| `build*` | `/^build[A-Z]/` | Yes (builder pattern) |
+| `make*` | `/^make[A-Z]/` | Yes (data factory) |
+| `generate*` | `/^generate[A-Z]/` | Yes (data factory) |
+| `setup*` | `/^setup[A-Z]/` | Evaluate (infrastructure) |
+| `createTest*` | `/^createTest[A-Z]/` | Evaluate (test infrastructure) |
+| `buildMock*` | `/^buildMock[A-Z]/` | Flag (explicit mock factory) |
+| `createFake*` | `/^createFake[A-Z]/` | Flag (explicit fake factory) |
+
+---
+
+## Framework-Specific Patterns
+
+### Jest / Vitest
+
+```typescript
+// Already detected by mock-detection core patterns:
+jest.mock('./module');
+jest.spyOn(object, 'method');
+vi.mock('./module');
+vi.spyOn(object, 'method');
+
+// Extended patterns (detect via this reference):
+jest.fn().mockReturnValue(value);     // Inline mock function
+jest.fn().mockResolvedValue(value);   // Async mock
+jest.fn().mockImplementation(fn);     // Custom implementation mock
+```
+
+### Sinon
+
+```typescript
+sinon.stub(object, 'method').returns(value);
+sinon.mock(object).expects('method');
+sinon.fake.returns(value);
+sinon.spy(object, 'method');
+```
+
+### Test Containers / Real Infrastructure
+
+```typescript
+// NOT a test double - real infrastructure for integration tests
+const container = await new GenericContainer('postgres:15').start();
+const redis = await new GenericContainer('redis:7').start();
+
+// NOT a test double - real HTTP interception for external APIs
+const server = setupServer(
+  rest.get('/api/users', (req, res, ctx) => res(ctx.json(users)))
+);
+```
+
+**Rule**: Test containers and MSW/nock interceptors for *external* APIs are legitimate test infrastructure, not violations. See `false-positive-prevention.md` for the full allowlist.
+
+---
+
+## Detection Priority by Test Type
+
+| Test Type | What to Flag | What to Allow |
+|-----------|-------------|---------------|
+| **Unit** | T1 (mock SUT), T2 (call-only assert) | External dep mocks, data factories |
+| **Integration** | T3 (mock boundary), T3+ (broken chain), manual stubs replacing integration points | Data factories, test containers, external API mocks |
+| **E2E** | Any mock/stub/fake that shortcuts the flow | Seed data, test user factories |
+
+---
+
+## Summary for LLM Agent
+
+When analyzing a test file:
+
+1. **Scan for naming patterns** from the tables above
+2. **Check class hierarchies** for implements/extends with test-double names
+3. **Identify factory functions** and classify as data factory (safe) vs infrastructure factory (evaluate)
+4. **Cross-reference with test type** (unit vs integration vs E2E)
+5. **Apply the mock appropriateness rubric** from mock-detection SKILL.md
+6. **Consult false-positive-prevention.md** before flagging borderline patterns