mdcontext 0.1.0 → 0.2.0

package/review/ALP-76/4-test-coverage.md

@@ -0,0 +1,625 @@
+# Test Coverage Review: ALP-76 Consolidated Error Handling
+
+**Reviewer**: Claude Sonnet 4.5
+**Date**: 2026-01-24
+**Branch**: nancy-ALP-76 worktree
+**Issue**: ALP-76 - Consolidated Error Handling
+
+---
+
+## Executive Summary
+
+**Overall Assessment**: ⚠️ **PARTIAL PASS** - Core error types have excellent unit test coverage, but integration testing and production error flow verification are insufficient.
+
+### Acceptance Criteria Status
+
+| Criterion | Status | Notes |
+|-----------|--------|-------|
+| All domain errors use Data.TaggedError | ✅ PASS | All 14 error types properly defined |
+| No silent error swallowing | ⚠️ PARTIAL | One intentional case documented, but no tests verify no regressions |
+| Error presentation only at CLI boundary | ✅ PASS | Clean separation verified |
+| catchTag pattern used exhaustively | ⚠️ PARTIAL | Used correctly, but not all error paths tested |
+| Tests verify error type discrimination | ⚠️ PARTIAL | Unit tests good, integration tests missing |
+
+### Key Findings
+
+**Strengths:**
+- Comprehensive unit tests for error types (50 tests, 610 lines)
+- Every error type has construction, _tag, code, and catchTag tests
+- Error formatter has complete Match.exhaustive coverage
+- Error codes tested for uniqueness and format
+
+**Critical Gaps:**
+1. No integration tests for error flows through commands
+2. Silent error swallowing not regression-tested
+3. Error recovery scenarios untested
+4. Type compiler enforcement not verified in tests
+
+---
+
+## 1. Error Type Discrimination Tests
+
+### 1.1 Unit Test Coverage (GOOD)
+
+**Location**: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-76/src/errors/errors.test.ts`
+
+All 14 error types have comprehensive unit tests:
+
+```typescript
+// File System Errors (4 types)
+✅ FileReadError - lines 41-89
+✅ FileWriteError - lines 91-113
+✅ DirectoryCreateError - lines 115-125
+✅ DirectoryWalkError - lines 127-137
+
+// Parse Errors (1 type)
+✅ ParseError - lines 143-169
+
+// API Key Errors (2 types)
+✅ ApiKeyMissingError - lines 175-210
+✅ ApiKeyInvalidError - lines 212-232
+
+// Embedding Errors (1 type with 5 sub-reasons)
+✅ EmbeddingError - lines 238-312
+  - RateLimit, QuotaExceeded, Network, ModelError, Unknown
+
+// Index Errors (3 types)
+✅ IndexNotFoundError - lines 318-330
+✅ IndexCorruptedError - lines 332-361
+✅ IndexBuildError - lines 363-373
+
+// Search Errors (2 types)
+✅ DocumentNotFoundError - lines 379-399
+✅ EmbeddingsNotFoundError - lines 401-417
+
+// Vector Store Errors (1 type)
+✅ VectorStoreError - lines 423-441
+
+// Config Errors (1 type)
+✅ ConfigError - lines 447-467
+
+// Watch Errors (1 type)
+✅ WatchError - lines 473-483
+
+// CLI Errors (1 type)
+✅ CliValidationError - lines 489-515
+```
+
+**Test Quality**:
+- Each error tests `_tag` correctness for catchTag discrimination
+- Error code getters verified (both constant and dynamic)
+- Data field preservation tested
+- Cause chain preservation tested (for errors with cause field)
+- catchTag integration tested with Effect.runPromise
+- Dynamic message generation tested (for errors with computed messages)
+
+### 1.2 Integration Test Coverage (WEAK)
+
+**catchTags Integration Test** (lines 521-554):
+```typescript
+✅ Tests multiple error types with catchTags (FileReadError, ApiKeyMissingError, IndexNotFoundError)
+```
+
+However, this is the **only** integration test for catchTags. Missing:
+- No tests for commands actually using catchTags
+- No tests for error handler (formatError/createErrorHandler)
+- No tests verifying exhaustive matching works
+
+### 1.3 Type Compiler Tests (MISSING)
+
+**Gap**: No tests verify that TypeScript compiler catches:
+- Missing error cases in catchTags
+- Invalid error tags
+- Non-exhaustive error handling
+
+**Recommendation**: Add type-level tests or document compiler verification process.
+
+---
+
+## 2. Coverage Analysis
+
+### 2.1 Error Path Coverage
+
+**Good Coverage**:
+- ✅ Error construction and data preservation (100%)
+- ✅ Error code mapping (100%)
+- ✅ catchTag discrimination (tested for 3 error types)
+- ✅ Dynamic message generation (for 4 error types)
+
+**Weak Coverage**:
+- ⚠️ Error formatter (formatError) - **0 dedicated tests**
+- ⚠️ Error display (displayError) - **0 tests**
+- ⚠️ createErrorHandler - **0 tests**
+- ⚠️ Real error flows through commands - **2 basic tests only**
+
+### 2.2 Production Error Usage
+
+**Analysis of catchTags Usage in Production Code**:
+
+Found 6 uses of catchTags/catchTag in production:
+
+1. **search.ts:350-361** - Handles IndexNotFoundError gracefully (with logging)
+2. **search.ts:392-404** - Handles ApiKeyMissingError, ApiKeyInvalidError
+3. **search.ts:473-485** - Duplicate of #2 (different code path)
+4. **index-cmd.ts:250-260** - Handles IndexNotFoundError (with logging)
+5. **index-cmd.ts:303-313** - Handles ApiKeyMissingError, ApiKeyInvalidError
+6. **vector-store.ts** - Uses catchTag for VectorStoreError
+
+**Test Coverage for These Paths**: ❌ **NONE**
+
+No integration tests verify these error handling paths work correctly.
+
+### 2.3 Silent Error Swallowing
+
+**Found 1 Instance** (properly documented):
+
+```typescript
+// src/summarize/summarizer.ts:537-543
+// Note: catchAll is intentional - measureReduction is a utility function
+// where failures should return default values (no reduction) rather than throw
+const result = await Effect.runPromise(
+  summarizeFile(filePath, { level }).pipe(
+    Effect.catchAll(() => Effect.succeed(null)),
+  ),
+)
+```
+
+**Status**: ✅ This is properly documented and intentional (utility function that should degrade gracefully).
+
+**Gap**: ⚠️ No test verifies this behavior or prevents regression to silent swallowing elsewhere.
+
+### 2.4 CLI Error Handling
+
+**E2E Tests** (`src/cli/cli.test.ts`):
+- ✅ 222 tests total, 50 related to errors
+- ⚠️ Only 2 explicit error handling tests:
+  - "handles non-existent file gracefully" (line 356)
+  - "handles non-existent directory gracefully" (line 363)
+- ✅ Tests only verify error output contains "error|not found|no such"
+- ❌ No tests for specific error types or error codes
+
+**Missing**:
+- No tests for API key errors
+- No tests for index corruption scenarios
+- No tests for embedding errors
+- No tests verify error codes are displayed
+- No tests verify suggestions are shown
+
+---
+
+## 3. Test Quality Assessment
+
+### 3.1 Meaningful Tests (GOOD)
+
+Tests go beyond type checking and verify actual behavior:
+
+**Example - ApiKeyMissingError** (lines 186-192):
+```typescript
+it('generates dynamic message', () => {
+  const error = new ApiKeyMissingError({
+    provider: 'openai',
+    envVar: 'OPENAI_API_KEY',
+  })
+  expect(error.message).toBe('OPENAI_API_KEY not set')
+})
+```
+
+**Example - EmbeddingError reason mapping** (lines 247-290):
+Tests each reason maps to correct error code dynamically.
+
+**Example - catchTag integration** (lines 78-88):
+```typescript
+it('can be caught with catchTag', async () => {
+  const effect = Effect.fail(
+    new FileReadError({ path: '/test.md', message: 'error' }),
+  )
+  const result = await Effect.runPromise(
+    effect.pipe(
+      Effect.catchTag('FileReadError', (e) => Effect.succeed(e.path)),
+    ),
+  )
+  expect(result).toBe('/test.md')
+})
+```
+
+### 3.2 Error Recovery Scenarios (WEAK)
+
+**Tested Recovery Scenarios**:
+- ✅ catchTag returns success with data (3 error types tested)
+- ✅ catchTags handles multiple error types (1 integration test)
+
+**Missing Recovery Scenarios**:
+- ❌ Retrying after rate limit error
+- ❌ Falling back to keyword search after embedding error
+- ❌ Creating index when IndexNotFoundError occurs
+- ❌ Error recovery in watch mode
+- ❌ Partial success scenarios (e.g., some files fail, others succeed)
+
+### 3.3 Edge Cases (PARTIAL)
+
+**Good Coverage**:
+- ✅ Optional fields (ParseError path/line/column, ConfigError field)
+- ✅ Cause chain preservation
+- ✅ Unknown embedding error reasons (catch-all)
+- ✅ Error code uniqueness (lines 561-565)
+
+**Missing Edge Cases**:
+- ❌ Very long error messages
+- ❌ Nested cause chains
+- ❌ Concurrent errors in parallel operations
+- ❌ Error during error handling (meta-errors)
+- ❌ Unicode/special characters in paths/messages
+
+---
+
+## 4. Regression Prevention
+
+### 4.1 Anti-Pattern Detection (WEAK)
+
+**Tests That Would Catch Regressions**:
+- ✅ Error code uniqueness test (prevents duplicate codes)
+- ✅ Error code format test (prevents invalid formats)
+- ✅ Match.exhaustive in formatter (compiler catches missing cases)
+
+**Missing Regression Tests**:
+- ❌ No test prevents adding catchAll(() => succeed(null)) without logging
+- ❌ No test verifies all error constructors use TaggedError
+- ❌ No test prevents throwing Error instead of failing with typed error
+- ❌ No test prevents error formatting in business logic
+
+### 4.2 Constructor Pattern (VERIFIED)
+
+**OpenAI Provider Constructor Fix** (GOOD):
+
+```typescript
+// src/embeddings/openai-provider.ts:43-68
+private constructor(apiKey: string, options: OpenAIProviderOptions = {}) {
+  // Private constructor cannot throw
+}
+
+static create(options: OpenAIProviderOptions = {}): Effect.Effect<OpenAIProvider, ApiKeyMissingError> {
+  const apiKey = options.apiKey ?? process.env.OPENAI_API_KEY
+  if (!apiKey) {
+    return Effect.fail(new ApiKeyMissingError({...}))
+  }
+  return Effect.succeed(new OpenAIProvider(apiKey, options))
+}
+```
+
+✅ Private constructor + static factory pattern correctly implemented.
+
+**Test Coverage**: ❌ No test verifies this pattern works or prevents regression.
+
+### 4.3 Silent Failure Tests (MISSING)
+
+**Current State**: Found 1 intentional silent failure (documented).
+
+**Gap**: No tests verify:
+- No new silent failures introduced
+- Existing silent failure has correct behavior
+- Errors are logged before being swallowed
+
+---
+
+## 5. Identified Gaps
+
+### 5.1 Critical Gaps
+
+1. **Error Handler Not Tested**
+   - File: `src/cli/error-handler.ts` (444 lines)
+   - Functions: `formatError`, `displayError`, `displayErrorDebug`, `createErrorHandler`
+   - Coverage: **0 dedicated tests**
+   - Risk: High - this is the CLI boundary where all errors are formatted
+
+2. **Command Error Flows Not Tested**
+   - 6 production uses of catchTags found
+   - 0 integration tests verify these work
+   - Risk: High - errors may not be caught correctly in production
+
+3. **OpenAI Provider Factory Not Tested**
+   - Static factory pattern implemented to fix constructor issue
+   - No test verifies ApiKeyMissingError is returned correctly
+   - Risk: Medium - could regress to throwing constructor
+
+4. **Error Recovery Not Tested**
+   - Fallback logic in search command (semantic → keyword)
+   - Auto-indexing logic
+   - Risk: Medium - recovery may silently fail
+
+### 5.2 Important Gaps
+
+5. **Partial Success Scenarios**
+   - Indexing multiple files where some fail
+   - Batch embedding operations
+   - Risk: Medium - may lose data without user awareness
+
+6. **Error Code Display**
+   - E2E tests don't verify error codes shown
+   - No test for debug mode error display
+   - Risk: Low - but affects user experience
+
+7. **Silent Failure Regression Prevention**
+   - No automated check for new catchAll(() => succeed(null))
+   - Risk: Low - but was a major issue to fix
+
+### 5.3 Minor Gaps
+
+8. **Edge Cases**
+   - Unicode in paths
+   - Very long messages
+   - Nested cause chains
+   - Risk: Low - rare scenarios
+
+9. **Type-Level Verification**
+   - No test that compiler catches invalid error handling
+   - Risk: Low - TypeScript will catch at build time
+
+---
+
+## 6. Recommendations for Additional Tests
+
+### 6.1 High Priority (Must Have)
+
+#### Test 1: Error Handler Formatting
+```typescript
+// test/cli/error-handler.test.ts
+describe('formatError', () => {
+  it('formats FileReadError correctly', () => {
+    const error = new FileReadError({
+      path: '/test.md',
+      message: 'ENOENT: no such file',
+    })
+    const formatted = formatError(error)
+    expect(formatted.code).toBe('E100')
+    expect(formatted.message).toContain('/test.md')
+    expect(formatted.exitCode).toBe(EXIT_CODE.SYSTEM_ERROR)
+    expect(formatted.suggestions).toBeDefined()
+  })
+
+  // Test all 14 error types...
+})
+```
+
+**Coverage**: Test formatError for all error types
+**Estimate**: ~15 test cases
+
+#### Test 2: Command Error Flow Integration
+```typescript
+// test/cli/commands/search.test.ts
+describe('search command error handling', () => {
+  it('handles IndexNotFoundError with helpful message', async () => {
+    // Mock index loading to fail
+    const result = await runCommand('search "test" /nonexistent')
+    expect(result.exitCode).toBe(1)
+    expect(result.stderr).toContain('E400')
+    expect(result.stderr).toContain('mdcontext index')
+  })
+
+  it('handles ApiKeyMissingError in semantic search', async () => {
+    // Test error when embeddings requested but no API key
+  })
+
+  it('falls back to keyword search when embeddings missing', async () => {
+    // Test graceful degradation
+  })
+})
+```
+
+**Coverage**: Test error paths in index, search, context commands
+**Estimate**: ~20 test cases
+
+#### Test 3: OpenAI Provider Factory
+```typescript
+// test/embeddings/openai-provider.test.ts
+describe('OpenAIProvider.create', () => {
+  it('returns ApiKeyMissingError when no key provided', async () => {
+    delete process.env.OPENAI_API_KEY
+    const result = await Effect.runPromise(
+      OpenAIProvider.create().pipe(
+        Effect.either
+      )
+    )
+    expect(result._tag).toBe('Left')
+    expect(result.left._tag).toBe('ApiKeyMissingError')
+  })
+
+  it('succeeds when API key provided', async () => {
+    const result = await Effect.runPromise(
+      OpenAIProvider.create({ apiKey: 'test-key' })
+    )
+    expect(result.name).toContain('openai')
+  })
+})
+```
+
+**Coverage**: Test factory pattern works correctly
+**Estimate**: 3 test cases
+
+### 6.2 Medium Priority (Should Have)
+
+#### Test 4: Error Recovery Scenarios
+```typescript
+describe('error recovery', () => {
+  it('retries after transient network error', async () => {
+    // Mock network failure then success
+  })
+
+  it('falls back to keyword search after embedding error', async () => {
+    // Test fallback logic
+  })
+
+  it('continues indexing after single file failure', async () => {
+    // Test partial success
+  })
+})
+```
+
+**Coverage**: Test recovery and fallback logic
+**Estimate**: ~10 test cases
+
+#### Test 5: Error Code Display in E2E
+```typescript
+describe('error display', () => {
+  it('shows error code in output', async () => {
+    const result = await runCommand('tree /nonexistent', { expectError: true })
+    expect(result.stderr).toMatch(/\[E\d{3}\]/)
+  })
+
+  it('shows suggestions for common errors', async () => {
+    const result = await runCommand('search "test"', { expectError: true })
+    expect(result.stderr).toContain('mdcontext index')
+  })
+})
+```
+
+**Coverage**: Verify error formatting in actual CLI output
+**Estimate**: ~8 test cases
+
+#### Test 6: Silent Failure Verification
+```typescript
+describe('error logging', () => {
+  it('logs warning when catchAll used in estimateEmbeddingCost', async () => {
+    // Verify logged warning appears
+  })
+
+  it('utility function returns null on failure with documented reason', async () => {
+    // Test measureReduction fallback behavior
+  })
+})
+```
+
+**Coverage**: Test intentional silent failures are logged
+**Estimate**: 2 test cases
+
+### 6.3 Low Priority (Nice to Have)
+
+#### Test 7: Edge Cases
+```typescript
+describe('error edge cases', () => {
+  it('handles very long error messages', () => {
+    const longMessage = 'x'.repeat(10000)
+    const error = new FileReadError({ path: '/test', message: longMessage })
+    const formatted = formatError(error)
+    expect(formatted.message).toBeDefined()
+  })
+
+  it('handles unicode in file paths', () => {
+    const error = new FileReadError({ path: '/path/with/émojis/🎉.md', message: 'error' })
+    expect(formatError(error).message).toContain('🎉')
+  })
+
+  it('preserves nested cause chains', () => {
+    const root = new Error('root cause')
+    const middle = new Error('middle', { cause: root })
+    const error = new FileReadError({ path: '/test', message: 'top', cause: middle })
+    expect(error.cause).toBe(middle)
+  })
+})
+```
+
+**Coverage**: Unusual but possible scenarios
+**Estimate**: ~5 test cases
+
+---
+
+## 7. Summary Statistics
+
+### Current Test Coverage
+
+| Category | Tests | Lines | Quality |
+|----------|-------|-------|---------|
+| Error Type Unit Tests | 50 | 610 | Excellent |
+| Integration Tests | 1 | ~15 | Weak |
+| E2E Error Tests | 2 | ~20 | Basic |
+| **Total Error Tests** | **53** | **~645** | **Good** |
+
+### Recommended Additional Tests
+
+| Priority | Category | Est. Tests | Est. Lines |
+|----------|----------|------------|------------|
+| High | Error Handler | 15 | ~300 |
+| High | Command Integration | 20 | ~500 |
+| High | Provider Factory | 3 | ~50 |
+| Medium | Error Recovery | 10 | ~250 |
+| Medium | Error Display E2E | 8 | ~150 |
+| Medium | Silent Failure | 2 | ~40 |
+| Low | Edge Cases | 5 | ~100 |
+| **Total** | **63** | **~1,390** |
+
+**Combined Total**: 116 error-related tests (~2,035 lines)
+
+---
+
+## 8. Final Assessment
+
+### Strengths
+1. ✅ Comprehensive unit test coverage for all error types
+2. ✅ Every error type has construction, _tag, code, and catchTag tests
+3. ✅ Error formatter uses Match.exhaustive for compiler safety
+4. ✅ OpenAI provider constructor issue properly fixed with factory pattern
+5. ✅ Silent error swallowing minimized and documented
+
+### Critical Issues
+1. ❌ Error handler has 0 dedicated tests (444 lines untested)
+2. ❌ Production catchTags usage not integration tested (6 instances)
+3. ❌ OpenAI factory pattern not tested (regression risk)
+4. ❌ Error recovery scenarios not tested
+5. ❌ No regression prevention for silent failures
+
+### Recommendation
+
+**PASS with Major Conditions**:
+
+The error handling **implementation** is excellent. The error handling **test coverage** is incomplete.
+
+**Required before merge to main**:
+1. Add error handler tests (formatError, displayError) - **HIGH PRIORITY**
+2. Add command integration tests for error flows - **HIGH PRIORITY**
+3. Add OpenAI factory tests - **HIGH PRIORITY**
+
+**Required before declaring complete**:
+4. Add error recovery tests - **MEDIUM PRIORITY**
+5. Add error display E2E tests - **MEDIUM PRIORITY**
+
+**Recommended**:
+6. Add automated check for catchAll without logging (linter rule or test)
+7. Document test strategy for error handling in TESTING.md
+
+### Test Coverage Grade: **B-** (73/100)
+
+- Unit tests: A+ (95/100)
+- Integration tests: C (40/100)
+- E2E tests: D+ (45/100)
+- Regression prevention: C- (50/100)
+
+**With recommended tests implemented: A- (88/100)**
+
+---
+
+## Appendix: Code References
+
+### Error Definitions
+- Error types: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-76/src/errors/index.ts` (lines 1-484)
+- Error codes: Lines 90-129
+
+### Error Handling
+- Error handler: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-76/src/cli/error-handler.ts` (lines 1-444)
+- Error formatter: Lines 74-287
+- createErrorHandler: Lines 376-395
+
+### Production Usage
+- search.ts catchTags: Lines 350-361, 392-404, 473-485
+- index-cmd.ts catchTags: Lines 250-260, 303-313
+- vector-store.ts catchTag: (not read in detail)
+
+### Tests
+- Error unit tests: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-76/src/errors/errors.test.ts` (610 lines)
+- CLI E2E tests: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-76/src/cli/cli.test.ts` (error tests at lines 355-369)
+
+### Silent Error Handling
+- Documented catchAll: `/Users/alphab/Dev/LLM/DEV/mdcontext/worktrees/nancy-ALP-76/src/summarize/summarizer.ts` (lines 537-543)
+
+---
+
+**Review Complete**
+Next recommended action: Implement high-priority tests (Error Handler, Command Integration, Provider Factory)