sqlew 4.0.4 → 4.1.0

package/docs/BATCH_VALIDATION.md

@@ -1,617 +0,0 @@
-# Batch Validation Utilities
-
-**Version**: 4.0.0
-**Status**: ✅ Implemented and Tested
-**Impact**: 70-85% reduction in batch validation failures
-
-## Overview
-
-Shared batch validation utilities that provide comprehensive pre-transaction validation for all batch operations (tasks, decisions, file changes, constraints). Validates ALL items at once and reports ALL errors with AI-friendly, actionable fix instructions.
-
-## Key Benefits
-
-1. **Pre-Transaction Validation**: Validate ALL items BEFORE starting database transaction
-2. **Comprehensive Error Reporting**: Report ALL errors at once (not just first error)
-3. **AI-Friendly Messages**: Structured errors with actionable fix instructions, typo suggestions, and valid options
-4. **Reusable Validators**: Shared validation building blocks for all batch operations
-5. **70-85% Failure Reduction**: Expected reduction in batch validation failures based on design analysis
-
-## Architecture
-
-### Core Components
-
-```
-src/utils/
-├── batch-validation.ts          # Main validation utilities (530 lines)
-│   ├── Core Validators          # validateRequiredField, validateEnum, validateType, etc.
-│   ├── Batch Orchestrator       # validateBatch (runs all validators)
-│   └── Error Formatter          # formatBatchValidationError (AI-readable output)
-│
-├── levenshtein.ts              # Shared Levenshtein distance (45 lines)
-│   └── Used by parameter-validator.ts and batch-validation.ts
-│
-└── batch-validation.example.ts # Usage examples and documentation
-```
-
-### Test Coverage
-
-```
-src/tests/batch-validation.test.ts
-├── 7 test suites
-├── 33 test cases
-└── ✅ 100% pass rate
-```
-
-## API Reference
-
-### Type Definitions
-
-```typescript
-/**
- * Structured validation error for single field in single batch item
- */
-interface BatchValidationError {
-  itemIndex: number;         // 0-based index in batch array
-  itemIdentifier: string;    // Item identifier (e.g., task.title, decision.key)
-  field: string;             // Field name that failed
-  issue: string;             // Human-readable issue description
-  fix: string;               // Actionable fix instruction
-  current?: any;             // Current invalid value
-  validOptions?: string[];   // Valid enum values (if applicable)
-}
-
-/**
- * Aggregate validation result for entire batch
- */
-interface BatchValidationResult {
-  valid: boolean;            // Overall validation status
-  errors: BatchValidationError[];
-  validCount: number;        // Count of valid items
-  invalidCount: number;      // Count of invalid items
-  summary: string;           // Summary message for AI
-}
-
-/**
- * Batch item validator function signature
- */
-type BatchItemValidator<T> = (
-  item: T,
-  index: number,
-  adapter: DatabaseAdapter,
-  errors: BatchValidationError[]
-) => Promise<void>;
-```
-
-### Core Validators
-
-#### validateRequiredField
-```typescript
-function validateRequiredField(
-  value: any,
-  fieldName: string,
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): void
-```
-Validates field exists and is non-empty (undefined, null, or empty string).
-
-**Example**:
-```typescript
-validateRequiredField(task.title, 'title', 0, 'Task 1', errors);
-// If missing: adds error with fix "Provide a non-empty value for 'title'"
-```
-
-#### validateEnum
-```typescript
-function validateEnum(
-  value: any,
-  fieldName: string,
-  validOptions: readonly string[],
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): void
-```
-Validates enum value with typo suggestions using Levenshtein distance ≤ 2.
-
-**Example**:
-```typescript
-validateEnum(task.layer, 'layer', STANDARD_LAYERS, 0, 'Task 1', errors);
-// If typo "busines": suggests "business (closest match)"
-```
-
-#### validateType
-```typescript
-function validateType(
-  value: any,
-  fieldName: string,
-  expectedType: 'array' | 'string' | 'number' | 'object',
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): void
-```
-Validates type (array, string, number, object).
-
-**Example**:
-```typescript
-validateType(task.tags, 'tags', 'array', 0, 'Task 1', errors);
-// If string "api,security": fix "Change to array format: [\"item1\", \"item2\"]"
-```
-
-#### validateRange
-```typescript
-function validateRange(
-  value: any,
-  fieldName: string,
-  min: number,
-  max: number,
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): void
-```
-Validates numeric range (inclusive).
-
-**Example**:
-```typescript
-validateRange(task.priority, 'priority', 1, 4, 0, 'Task 1', errors);
-// If 5: fix "Provide a number between 1 and 4"
-```
-
-#### validateLength
-```typescript
-function validateLength(
-  value: any,
-  fieldName: string,
-  maxLength: number,
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): void
-```
-Validates string length.
-
-**Example**:
-```typescript
-validateLength(task.title, 'title', 200, 0, 'Task 1', errors);
-// If 201 chars: fix "Shorten to 200 characters or less (currently 201 chars)"
-```
-
-#### validateForeignKey
-```typescript
-async function validateForeignKey(
-  value: any,
-  fieldName: string,
-  tableName: string,
-  columnName: string,
-  adapter: DatabaseAdapter,
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): Promise<void>
-```
-Validates foreign key reference exists in database.
-
-**Example**:
-```typescript
-await validateForeignKey(
-  task.layer, 'layer', 'v4_layers', 'name',
-  adapter, 0, 'Task 1', errors
-);
-// If invalid: provides list of valid layers from database
-```
-
-#### validateLayerFileRequirement
-```typescript
-function validateLayerFileRequirement(
-  layer: string | undefined,
-  fileActions: any,
-  itemIndex: number,
-  itemIdentifier: string,
-  errors: BatchValidationError[]
-): void
-```
-Validates FILE_REQUIRED layers have file_actions parameter (v3.8.0).
-
-**Example**:
-```typescript
-validateLayerFileRequirement('business', undefined, 0, 'Task 1', errors);
-// Error: "Layer 'business' requires file_actions parameter"
-// Fix: "Add file_actions: [{ action: 'edit', path: 'src/...' }] or use [] for planning tasks"
-```
-
-### Batch Orchestrator
-
-#### validateBatch
-```typescript
-async function validateBatch<T>(
-  items: T[],
-  itemValidator: BatchItemValidator<T>,
-  adapter: DatabaseAdapter
-): Promise<BatchValidationResult>
-```
-Validates entire batch before transaction. Returns structured result with ALL validation errors.
-
-**Example**:
-```typescript
-const result = await validateBatch(
-  params.tasks,
-  validateTaskItem,  // Your item validator function
-  adapter
-);
-
-if (!result.valid) {
-  throw new Error(formatBatchValidationError(result));
-}
-```
-
-### Error Formatter
-
-#### formatBatchValidationError
-```typescript
-function formatBatchValidationError(
-  result: BatchValidationResult
-): string
-```
-Formats batch validation errors for AI comprehension with actionable fix instructions.
-
-**Example Output**:
-```
-Batch validation failed. Found 4 validation error(s) in 2 item(s). 1 items are valid.
-
-Item 0 (Task 1):
-  ❌ layer: Invalid layer: "busines"
-     💡 Fix: Change to "business" (closest match)
-     Current: "busines"
-     Valid: presentation, business, data, infrastructure, cross-cutting, documentation, planning, coordination, review
-
-  ❌ priority: Field "priority" must be between 1 and 4
-     💡 Fix: Provide a number between 1 and 4
-     Current: 5
-
-Item 1 (Task 2):
-  ❌ title: Field "title" is required but missing or empty
-     💡 Fix: Provide a non-empty value for "title"
-     Current: undefined
-
-  ❌ file_actions: Layer "presentation" requires file_actions parameter
-     💡 Fix: Add file_actions: [{ action: "edit", path: "src/..." }] or use [] for planning tasks
-     Current: undefined
-     Valid: Add file_actions array, Change to FILE_OPTIONAL layer (planning/coordination/review)
-
-💡 Result: 1 valid, 2 invalid
-💡 Action: Fix the 2 invalid item(s) and retry batch operation
-```
-
-## Integration Pattern
-
-### Before (Without Batch Validation)
-
-```typescript
-// ❌ PROBLEMATIC - Validation happens INSIDE transaction
-export async function batchCreateTasks(params: { tasks: any[] }) {
-  if (!params.tasks || !Array.isArray(params.tasks)) {
-    throw new Error('Parameter "tasks" is required and must be an array');
-  }
-
-  // Start transaction immediately
-  return await adapter.transaction(async (trx) => {
-    for (const task of params.tasks) {
-      // Validation happens INSIDE transaction
-      // Fails on FIRST error, wastes resources on partial processing
-      const result = await createTaskInternal(task, adapter, trx);
-      // ...
-    }
-  });
-}
-```
-
-**Problems**:
-- Validates inside transaction (wasted resources on partial processing)
-- Fails on first error (AI must fix one error at a time, retry, discover next error)
-- Generic error messages (no context, no guidance)
-
-### After (With Batch Validation)
-
-```typescript
-// ✅ IMPROVED - Pre-validation BEFORE transaction
-export async function batchCreateTasks(params: { tasks: any[] }, adapter?: DatabaseAdapter) {
-  const actualAdapter = adapter ?? getAdapter();
-
-  // Basic validation
-  if (!params.tasks || !Array.isArray(params.tasks)) {
-    throw new Error('Parameter "tasks" is required and must be an array');
-  }
-
-  if (params.tasks.length > 50) {
-    throw new Error('Parameter "tasks" must contain at most 50 items');
-  }
-
-  // ✅ PRE-VALIDATION: Check ALL items BEFORE transaction
-  const validationResult = await validateBatch(
-    params.tasks,
-    validateTaskItem,
-    actualAdapter
-  );
-
-  // ✅ FAIL FAST: If validation fails, throw formatted error with ALL issues
-  if (!validationResult.valid) {
-    const errorMessage = formatBatchValidationError(validationResult);
-    throw new Error(errorMessage);
-  }
-
-  // Validation passed - proceed with transaction
-  const atomic = params.atomic !== undefined ? params.atomic : true;
-
-  if (atomic) {
-    // All items are valid - safe to process in single transaction
-    return await actualAdapter.transaction(async (trx) => {
-      const results = [];
-      for (const task of params.tasks) {
-        // No validation needed here - already validated
-        const result = await createTaskInternal(task, actualAdapter, trx);
-        results.push({
-          title: task.title,
-          task_id: result.task_id,
-          success: true
-        });
-      }
-      return { success: true, created: results.length, failed: 0, results };
-    });
-  } else {
-    // Non-atomic mode: still validate upfront, but process independently
-    // ... existing non-atomic logic
-  }
-}
-```
-
-**Benefits**:
-- Validates ALL items BEFORE transaction (no wasted resources)
-- Reports ALL errors at once (AI fixes all issues in one retry)
-- AI-friendly error messages (actionable fixes, typo suggestions, valid options)
-
-## Usage Examples
-
-### Example 1: Task Batch Validator
-
-```typescript
-import {
-  validateRequiredField,
-  validateEnum,
-  validateType,
-  validateRange,
-  validateLength,
-  validateLayerFileRequirement,
-  type BatchValidationError
-} from '../utils/batch-validation.js';
-
-async function validateTaskItem(
-  task: any,
-  index: number,
-  adapter: DatabaseAdapter,
-  errors: BatchValidationError[]
-): Promise<void> {
-  const identifier = task.title || `Item ${index}`;
-
-  // Required fields
-  validateRequiredField(task.title, 'title', index, identifier, errors);
-
-  // String length
-  if (task.title) {
-    validateLength(task.title, 'title', 200, index, identifier, errors);
-  }
-
-  // Enums
-  if (task.status) {
-    validateEnum(
-      task.status,
-      'status',
-      ['todo', 'in_progress', 'waiting_review', 'blocked', 'done', 'archived', 'rejected'],
-      index,
-      identifier,
-      errors
-    );
-  }
-
-  if (task.layer) {
-    validateEnum(
-      task.layer,
-      'layer',
-      ['presentation', 'business', 'data', 'infrastructure', 'cross-cutting',
-       'documentation', 'planning', 'coordination', 'review'],
-      index,
-      identifier,
-      errors
-    );
-  }
-
-  // Range validation
-  if (task.priority !== undefined) {
-    validateRange(task.priority, 'priority', 1, 4, index, identifier, errors);
-  }
-
-  // Type validation
-  if (task.tags !== undefined) {
-    validateType(task.tags, 'tags', 'array', index, identifier, errors);
-  }
-
-  // Complex field validation: file_actions
-  if (task.file_actions !== undefined) {
-    validateType(task.file_actions, 'file_actions', 'array', index, identifier, errors);
-
-    if (Array.isArray(task.file_actions)) {
-      task.file_actions.forEach((fa: any, faIndex: number) => {
-        if (!fa.action) {
-          errors.push({
-            itemIndex: index,
-            itemIdentifier: identifier,
-            field: `file_actions[${faIndex}].action`,
-            issue: 'Missing action field in file_actions',
-            fix: 'Add action field: { action: "create"|"edit"|"delete", path: "..." }',
-            current: fa
-          });
-        } else {
-          validateEnum(
-            fa.action,
-            `file_actions[${faIndex}].action`,
-            ['create', 'edit', 'delete'],
-            index,
-            identifier,
-            errors
-          );
-        }
-
-        if (!fa.path || typeof fa.path !== 'string') {
-          errors.push({
-            itemIndex: index,
-            itemIdentifier: identifier,
-            field: `file_actions[${faIndex}].path`,
-            issue: 'Missing or invalid path field in file_actions',
-            fix: 'Add path field as string: { action: "...", path: "src/file.ts" }',
-            current: fa
-          });
-        }
-      });
-    }
-  }
-
-  // Layer-specific validation (FILE_REQUIRED vs FILE_OPTIONAL)
-  validateLayerFileRequirement(task.layer, task.file_actions, index, identifier, errors);
-}
-```
-
-### Example 2: Decision Batch Validator
-
-```typescript
-async function validateDecisionItem(
-  decision: any,
-  index: number,
-  adapter: DatabaseAdapter,
-  errors: BatchValidationError[]
-): Promise<void> {
-  const identifier = decision.key || `Item ${index}`;
-
-  // Required fields
-  validateRequiredField(decision.key, 'key', index, identifier, errors);
-  validateRequiredField(decision.value, 'value', index, identifier, errors);
-
-  // Enums
-  if (decision.status) {
-    validateEnum(
-      decision.status,
-      'status',
-      ['active', 'deprecated', 'draft'],
-      index,
-      identifier,
-      errors
-    );
-  }
-
-  if (decision.layer) {
-    validateEnum(
-      decision.layer,
-      'layer',
-      STANDARD_LAYERS,
-      index,
-      identifier,
-      errors
-    );
-  }
-
-  // Type validation
-  if (decision.tags !== undefined) {
-    validateType(decision.tags, 'tags', 'array', index, identifier, errors);
-  }
-}
-```
-
-## Success Metrics
-
-### Before (Current State)
-- **AI retry rate**: 5-10 retries per batch with multiple issues
-- **Error comprehension**: Low (generic messages, single error at a time)
-- **Fix accuracy**: 60-70% (AI guesses at fixes due to incomplete error info)
-
-### After (With Batch Validation)
-- **AI retry rate**: 1-2 retries per batch (all errors reported at once)
-- **Error comprehension**: High (structured errors with fix instructions)
-- **Fix accuracy**: 90-95% (AI has complete error context and actionable fixes)
-- **Reduction in batch failures**: 70-85% (pre-validation catches issues before transaction)
-
-## AI Interaction Flow Comparison
-
-### Before (Iterative Debugging Loop - 4 attempts)
-1. AI submits batch with 3 invalid items
-2. System processes item 0, item 1, fails at item 2 with "Invalid layer: busines"
-3. AI fixes item 2 layer typo, retries
-4. System processes item 0, fails with "Priority must be between 1 and 4"
-5. AI fixes item 0 priority, retries
-6. System processes item 0, item 1, fails with "Missing required field: title"
-7. AI fixes item 1 title, retries
-8. Success
-
-**Total iterations**: 4
-**Time wasted**: 3 failed transactions
-**AI frustration**: High (whack-a-mole debugging)
-
-### After (Single Fix Cycle - 2 attempts)
-1. AI submits batch with 3 invalid items
-2. System validates all items BEFORE transaction
-3. System returns formatted error with ALL 5 validation issues across 3 items
-4. AI reads error, sees all issues at once:
-   - Item 0: layer typo "busines" → "business"
-   - Item 0: priority 5 → must be 1-4
-   - Item 1: missing title
-   - Item 2: file_actions wrong type (string → array)
-   - Item 2: layer requires file_actions
-5. AI fixes all 5 issues, retries
-6. Success
-
-**Total iterations**: 2 (1 failure + 1 success)
-**Time wasted**: 0 transactions (validation before transaction)
-**AI frustration**: Low (clear fix instructions)
-
-## Files
-
-```
-src/utils/
-├── batch-validation.ts         # Main implementation (530 lines)
-├── levenshtein.ts              # Shared Levenshtein distance (45 lines)
-└── batch-validation.example.ts # Usage examples (230 lines)
-
-src/tests/
-└── batch-validation.test.ts    # Unit tests (33 tests, 7 suites)
-
-docs/
-└── BATCH_VALIDATION.md         # This file
-```
-
-## Related Documentation
-
-- **Design Document**: `batch-validation-architecture-design` (Serena memory)
-- **Parameter Validation**: `src/utils/parameter-validator.ts` (uses shared Levenshtein)
-- **Constants**: `src/constants.ts` (STANDARD_LAYERS, FILE_REQUIRED_LAYERS, etc.)
-- **Types**: `src/adapters/types.ts` (DatabaseAdapter interface)
-
-## Version History
-
-- **v3.8.0**: Initial implementation (2025-11-09)
-  - Core validators: validateRequiredField, validateEnum, validateType, validateRange, validateLength, validateForeignKey, validateLayerFileRequirement
-  - Batch orchestrator: validateBatch
-  - Error formatter: formatBatchValidationError
-  - Shared Levenshtein distance extraction
-  - 33 unit tests (100% pass rate)
-  - Integration examples and documentation
-
-## Future Enhancements
-
-1. **Phase 2**: Tool-specific validators (validateTaskItem, validateDecisionItem, validateFileChangeItem)
-2. **Phase 3**: Integration into batch actions (batch-create.ts, batch-set.ts, record-batch.ts)
-3. **Phase 4**: Integration tests with real batch operations
-4. **Metrics**: Track actual AI retry rate reduction and fix accuracy
-
-## License
-
-MIT License - Part of MCP Shared Context Server (mcp-sqlew)