@cakemail-org/cakemail-cli 1.7.0 → 2.0.0

package/docs/testing/AI_USER_SIMULATION_DESIGN.md

@@ -0,0 +1,1342 @@
+# AI-Based User Simulation Testing Framework
+
+## Executive Summary
+
+This document outlines a comprehensive AI-based user simulation testing framework designed to validate software systems (CLI, web, mobile) against their documentation. The framework uses AI agents to act as users, execute commands/actions based on documentation, and validate outcomes against expected behavior.
+
+**Core Principle:** The AI reads documentation like a human user would, attempts to perform tasks, and reports discrepancies between documented behavior and actual behavior.
+
+---
+
+## Table of Contents
+
+1. [Architecture Overview](#architecture-overview)
+2. [Core Components](#core-components)
+3. [Testing Workflow](#testing-workflow)
+4. [Platform Adaptations](#platform-adaptations)
+5. [Implementation Design](#implementation-design)
+6. [Quality Metrics](#quality-metrics)
+7. [Example Scenarios](#example-scenarios)
+8. [Future Enhancements](#future-enhancements)
+
+---
+
+## Architecture Overview
+
+### High-Level Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     AI User Simulation System                    │
+└─────────────────────────────────────────────────────────────────┘
+                                 │
+        ┌────────────────────────┼────────────────────────┐
+        │                        │                        │
+        ▼                        ▼                        ▼
+┌───────────────┐        ┌──────────────┐        ┌──────────────┐
+│  Knowledge    │        │  Execution   │        │  Validation  │
+│  Ingestion    │───────▶│   Engine     │───────▶│   Engine     │
+│  Layer        │        │              │        │              │
+└───────────────┘        └──────────────┘        └──────────────┘
+        │                        │                        │
+        │                        │                        │
+        ▼                        ▼                        ▼
+┌───────────────┐        ┌──────────────┐        ┌──────────────┐
+│ Documentation │        │ Target       │        │ Test Results │
+│ Repository    │        │ System       │        │ Database     │
+│ (Markdown,    │        │ (CLI, Web,   │        │ (Pass/Fail,  │
+│  API specs)   │        │  Mobile)     │        │  Evidence)   │
+└───────────────┘        └──────────────┘        └──────────────┘
+```
+
+### Key Principles
+
+1. **Documentation-Driven:** AI agents derive test cases exclusively from documentation
+2. **Platform-Agnostic:** Core framework adapts to CLI, web, and mobile interfaces
+3. **Autonomous Learning:** AI learns expected behaviors from examples in documentation
+4. **Self-Healing:** AI can adapt to minor UI/output changes without manual updates
+5. **Evidence-Based:** Every assertion is backed by screenshots, logs, or output captures
+
+---
+
+## Core Components
+
+### 1. Knowledge Ingestion Layer
+
+**Purpose:** Parse and understand documentation to build a knowledge graph of system capabilities.
+
+#### Components:
+
+**1.1 Documentation Parser**
+- **Input:** Markdown files, API specifications, command references, tutorials
+- **Output:** Structured knowledge graph of commands, parameters, expected outputs, examples
+- **Technology:** LLM with retrieval-augmented generation (RAG)
+
+**Example Structure:**
+```json
+{
+  "command": "cakemail campaigns list",
+  "category": "campaigns",
+  "documentation_source": "docs/user-manual/09-command-reference/02-campaigns.md",
+  "description": "List all campaigns with optional filtering",
+  "parameters": {
+    "status": {
+      "type": "option",
+      "flag": "-s, --status",
+      "description": "Filter by status",
+      "values": ["draft", "scheduled", "sent", "failed"],
+      "required": false
+    },
+    "format": {
+      "type": "global_option",
+      "flag": "-f, --format",
+      "values": ["json", "table", "compact"],
+      "default_behavior": "profile-dependent"
+    }
+  },
+  "expected_outputs": {
+    "json": {
+      "structure": "array of campaign objects",
+      "sample": "{\"data\":[{\"id\":123,\"name\":\"Newsletter\",...}]}"
+    },
+    "table": {
+      "description": "Formatted table with key fields",
+      "sample": "ASCII table with columns: ID, Name, Status, Created"
+    }
+  },
+  "examples": [
+    {
+      "command": "cakemail campaigns list --status sent",
+      "expected_behavior": "Shows only sent campaigns",
+      "success_criteria": ["All returned campaigns have status=sent", "Exit code 0"]
+    }
+  ],
+  "edge_cases": [
+    "Empty list returns empty array/table",
+    "Invalid status returns error with suggestion"
+  ]
+}
+```
+
+**1.2 Example Extractor**
+- Parses code blocks in documentation
+- Identifies input commands and expected outputs
+- Builds test case templates from examples
+
+**1.3 Context Builder**
+- Understands relationships between commands (e.g., "create campaign" requires "list" first)
+- Builds dependency graphs (must create list before adding contacts)
+- Identifies authentication requirements
+
+### 2. Execution Engine
+
+**Purpose:** Execute commands/actions against the target system and capture results.
+
+#### Platform-Specific Adapters:
+
+**2.1 CLI Adapter**
+```typescript
+interface CLIAdapter {
+  executeCommand(command: string, env: Environment): ExecutionResult;
+  captureOutput(): { stdout: string, stderr: string, exitCode: number };
+  captureScreenshot(): string; // For terminal output capture
+  getEnvironmentState(): EnvironmentState; // Auth tokens, config files
+}
+```
+
+**2.2 Web Adapter**
+```typescript
+interface WebAdapter {
+  navigate(url: string): void;
+  findElement(selector: string): Element;
+  click(element: Element): void;
+  type(element: Element, text: string): void;
+  captureScreenshot(): Buffer;
+  getPageSource(): string;
+  waitForElement(selector: string, timeout: number): Element;
+  evaluateJavaScript(script: string): any;
+}
+```
+
+**2.3 Mobile Adapter**
+```typescript
+interface MobileAdapter {
+  tap(x: number, y: number): void;
+  swipe(direction: 'up' | 'down' | 'left' | 'right'): void;
+  enterText(text: string): void;
+  captureScreenshot(): Buffer;
+  getViewHierarchy(): ViewTree;
+  waitForElement(accessibilityId: string): Element;
+}
+```
+
+#### Execution Context Manager
+- Manages test data isolation (separate test accounts, sandboxed environments)
+- Handles authentication and session management
+- Cleanup after test runs (delete test data)
+- State verification between test cases
+
+### 3. Validation Engine
+
+**Purpose:** Compare actual results against documented expected behavior.
+
+#### Validation Strategies:
+
+**3.1 Output Structure Validation**
+```typescript
+interface OutputValidator {
+  validateJSON(actual: object, expected: JSONSchema): ValidationResult;
+  validateTable(actual: string, expectedColumns: string[]): ValidationResult;
+  validateText(actual: string, expectedPatterns: RegExp[]): ValidationResult;
+  validateStatusCode(actual: number, expected: number): ValidationResult;
+}
+```
+
+**3.2 Semantic Validation**
+- Uses LLM to understand if output semantically matches documentation
+- Example: "Shows only sent campaigns" → Validates all items have `status: "sent"`
+- Handles variations in formatting (dates, numbers, etc.)
+
+**3.3 Visual Validation** (Web/Mobile)
+- Screenshot comparison using image diffing
+- AI-based visual assertion ("button should be green", "table has 5 rows")
+- Accessibility validation (screen reader compatibility)
+
+**3.4 Behavioral Validation**
+- Validates workflows: "Create → List → Delete" sequence
+- Validates error handling: "Invalid input shows helpful error message"
+- Validates state changes: "After delete, item no longer appears in list"
+
+---
+
+## Testing Workflow
+
+### Phase 1: Test Plan Generation
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│ 1. AI reads documentation                                     │
+│    - Command reference pages                                  │
+│    - User guides and tutorials                                │
+│    - API specifications                                       │
+└──────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌──────────────────────────────────────────────────────────────┐
+│ 2. AI generates test plan                                     │
+│    - Identifies all testable commands/features                │
+│    - Extracts examples from documentation                     │
+│    - Builds dependency graph (order of operations)            │
+│    - Identifies edge cases and error scenarios                │
+└──────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌──────────────────────────────────────────────────────────────┐
+│ 3. Test plan review (optional human-in-the-loop)             │
+│    - Human reviews generated test cases                       │
+│    - Adds missing scenarios                                  │
+│    - Approves or refines plan                                │
+└──────────────────────────────────────────────────────────────┘
+```
+
+**Example Test Plan:**
+```yaml
+test_suite: "Campaigns Management"
+prerequisites:
+  - authenticated: true
+  - minimum_lists: 1
+  - minimum_senders: 1
+
+test_cases:
+  - id: "CAMP-001"
+    name: "List all campaigns"
+    command: "cakemail campaigns list"
+    expected_behavior:
+      - exit_code: 0
+      - output_format: "json by default (developer profile)"
+      - contains_fields: ["id", "name", "status", "created_on"]
+      - validates_against_schema: true
+
+  - id: "CAMP-002"
+    name: "List campaigns with status filter"
+    command: "cakemail campaigns list --status sent"
+    expected_behavior:
+      - exit_code: 0
+      - all_items_match: "status == 'sent'"
+      - error_if_no_matches: false (returns empty array)
+
+  - id: "CAMP-003"
+    name: "Create campaign interactively"
+    profile: "marketer"
+    command: "cakemail campaigns create"
+    interactions:
+      - prompt: "Campaign name:"
+        input: "Test Campaign {{timestamp}}"
+      - prompt: "Select a list:"
+        action: "select_first"
+      - prompt: "Select a sender:"
+        action: "select_first"
+    expected_behavior:
+      - exit_code: 0
+      - output_contains: "Campaign created successfully"
+      - new_campaign_appears_in_list: true
+```
+
+### Phase 2: Test Execution
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│ 1. Environment setup                                          │
+│    - Create isolated test account/environment                │
+│    - Configure authentication                                │
+│    - Seed test data (lists, contacts, senders)               │
+└──────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌──────────────────────────────────────────────────────────────┐
+│ 2. Execute test cases                                         │
+│    For each test case:                                        │
+│      a) Set up prerequisites                                  │
+│      b) Execute command/action                                │
+│      c) Capture output/screenshots                            │
+│      d) Validate against expected behavior                    │
+│      e) Record results with evidence                          │
+└──────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌──────────────────────────────────────────────────────────────┐
+│ 3. Cleanup                                                    │
+│    - Delete test data                                         │
+│    - Reset environment state                                  │
+│    - Archive test artifacts (logs, screenshots)               │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Phase 3: Results Analysis & Reporting
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│ 1. AI analyzes failures                                       │
+│    - Categorizes failures (bug, documentation issue, flaky)   │
+│    - Identifies root causes                                   │
+│    - Suggests fixes                                           │
+└──────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌──────────────────────────────────────────────────────────────┐
+│ 2. Generate comprehensive report                              │
+│    - Test coverage metrics                                    │
+│    - Pass/fail breakdown                                      │
+│    - Failed test details with evidence                        │
+│    - Recommendations for documentation/code fixes             │
+└──────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌──────────────────────────────────────────────────────────────┐
+│ 3. Create GitHub issues (optional automation)                │
+│    - Bug reports with reproduction steps                      │
+│    - Documentation improvement suggestions                    │
+│    - Links to test run artifacts                              │
+└──────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Platform Adaptations
+
+### CLI Testing (Current: Cakemail CLI)
+
+**Execution Strategy:**
+```typescript
+class CLIUserSimulation {
+  async testCommand(testCase: TestCase): Promise<TestResult> {
+    // 1. Set up environment
+    const env = await this.setupEnvironment(testCase.prerequisites);
+
+    // 2. Execute command
+    const result = await this.executeCommand(testCase.command, env);
+
+    // 3. Capture output
+    const evidence = {
+      stdout: result.stdout,
+      stderr: result.stderr,
+      exitCode: result.exitCode,
+      terminalScreenshot: await this.captureTerminal()
+    };
+
+    // 4. Validate
+    const validations = await this.validateOutput(
+      result,
+      testCase.expectedBehavior
+    );
+
+    // 5. Return result
+    return {
+      testCaseId: testCase.id,
+      passed: validations.every(v => v.passed),
+      evidence,
+      validations
+    };
+  }
+}
+```
+
+**CLI-Specific Validations:**
+- Exit code validation (0 = success, non-zero = error)
+- stdout/stderr content validation
+- JSON structure validation
+- Table format validation (column headers, alignment)
+- Color output validation (ANSI codes)
+- Interactive prompt handling (PTY simulation)
+- Configuration file state changes
+
+### Web Testing (Adaptable to Cakemail Web App)
+
+**Execution Strategy:**
+```typescript
+class WebUserSimulation {
+  async testUserFlow(testCase: TestCase): Promise<TestResult> {
+    // 1. Navigate to page
+    await this.browser.navigate(testCase.url);
+
+    // 2. Execute user actions
+    for (const step of testCase.steps) {
+      await this.executeStep(step);
+      await this.captureScreenshot(step.name);
+    }
+
+    // 3. Validate final state
+    const validations = await this.validatePageState(
+      testCase.expectedState
+    );
+
+    // 4. Return result
+    return {
+      testCaseId: testCase.id,
+      passed: validations.every(v => v.passed),
+      screenshots: this.screenshots,
+      validations
+    };
+  }
+
+  async executeStep(step: UIStep): Promise<void> {
+    switch (step.type) {
+      case 'click':
+        const element = await this.findElement(step.selector);
+        await element.click();
+        break;
+      case 'type':
+        const input = await this.findElement(step.selector);
+        await input.type(step.text);
+        break;
+      case 'verify':
+        await this.verifyElementText(step.selector, step.expectedText);
+        break;
+    }
+  }
+}
+```
+
+**Web-Specific Validations:**
+- Page title and URL validation
+- Element presence/absence validation
+- Text content validation
+- Form submission validation
+- JavaScript state validation
+- Network request validation (API calls)
+- Visual regression testing (screenshot comparison)
+- Accessibility validation (WCAG compliance)
+
+**Example Test Case (Web):**
+```yaml
+test_case:
+  id: "WEB-CAMP-001"
+  name: "Create campaign via web UI"
+  url: "https://app.cakemail.com/campaigns"
+  steps:
+    - type: "click"
+      selector: "button[data-testid='create-campaign']"
+      description: "Click Create Campaign button"
+
+    - type: "wait"
+      selector: "input[name='campaign-name']"
+      description: "Wait for modal to appear"
+
+    - type: "type"
+      selector: "input[name='campaign-name']"
+      text: "Test Campaign {{timestamp}}"
+      description: "Enter campaign name"
+
+    - type: "click"
+      selector: "select[name='list-id']"
+      description: "Open list dropdown"
+
+    - type: "click"
+      selector: "select[name='list-id'] option:first-child"
+      description: "Select first list"
+
+    - type: "click"
+      selector: "button[type='submit']"
+      description: "Submit form"
+
+    - type: "verify"
+      selector: ".success-message"
+      expectedText: "Campaign created successfully"
+      description: "Verify success message"
+
+  expected_state:
+    - url_contains: "/campaigns/"
+    - element_exists: ".campaign-details"
+    - api_called: "POST /campaigns"
+    - api_response_status: 201
+```
+
+### Mobile Testing (Adaptable to Cakemail Mobile App)
+
+**Execution Strategy:**
+```typescript
+class MobileUserSimulation {
+  async testMobileFlow(testCase: TestCase): Promise<TestResult> {
+    // 1. Launch app
+    await this.app.launch();
+
+    // 2. Navigate to screen
+    await this.navigateToScreen(testCase.screen);
+
+    // 3. Execute gestures
+    for (const gesture of testCase.gestures) {
+      await this.executeGesture(gesture);
+      await this.captureScreenshot(gesture.name);
+    }
+
+    // 4. Validate screen state
+    const validations = await this.validateScreenState(
+      testCase.expectedState
+    );
+
+    return {
+      testCaseId: testCase.id,
+      passed: validations.every(v => v.passed),
+      screenshots: this.screenshots,
+      validations
+    };
+  }
+}
+```
+
+**Mobile-Specific Validations:**
+- Screen title validation
+- Element hierarchy validation
+- Gesture response validation (swipe, tap, long-press)
+- Orientation changes
+- Push notification handling
+- Offline mode behavior
+- Native API integration (camera, contacts, etc.)
+
+---
+
+## Implementation Design
+
+### Technology Stack
+
+**AI/LLM Layer:**
+- **LLM Provider:** OpenAI GPT-4 or Anthropic Claude (for reasoning and validation)
+- **RAG System:** LangChain or LlamaIndex for documentation ingestion
+- **Vector Store:** Pinecone or Weaviate for semantic search
+
+**Execution Layer:**
+- **CLI:** Node.js with `execa` for process execution, `node-pty` for interactive terminals
+- **Web:** Playwright or Selenium for browser automation
+- **Mobile:** Appium for iOS/Android testing
+
+**Validation Layer:**
+- **JSON Schema:** Ajv for JSON validation
+- **Visual Diff:** Pixelmatch or Percy for screenshot comparison
+- **Semantic Analysis:** LLM-based natural language validation
+
+**Reporting:**
+- **Test Results:** PostgreSQL or MongoDB
+- **Artifacts:** S3 or local filesystem
+- **Dashboard:** Custom React app or Allure Reports
+
+### Core Modules
+
+#### Module 1: Documentation Analyzer
+
+```typescript
+class DocumentationAnalyzer {
+  constructor(
+    private llm: LLMProvider,
+    private vectorStore: VectorStore
+  ) {}
+
+  async ingestDocumentation(docsPath: string): Promise<KnowledgeGraph> {
+    // 1. Read all markdown files
+    const files = await this.readDocumentationFiles(docsPath);
+
+    // 2. Parse and chunk
+    const chunks = await this.chunkDocuments(files);
+
+    // 3. Generate embeddings
+    await this.vectorStore.addDocuments(chunks);
+
+    // 4. Extract structured knowledge
+    const knowledge = await this.extractKnowledge(chunks);
+
+    return knowledge;
+  }
+
+  async extractKnowledge(chunks: Document[]): Promise<KnowledgeGraph> {
+    const prompt = `
+      Extract structured information from this documentation:
+
+      For each command or feature, extract:
+      1. Name and description
+      2. Parameters (required, optional, types, defaults)
+      3. Expected outputs (format, structure, examples)
+      4. Example commands from code blocks
+      5. Error scenarios and expected error messages
+      6. Dependencies (prerequisites, related commands)
+
+      ${chunks.map(c => c.content).join('\n\n')}
+    `;
+
+    const response = await this.llm.complete(prompt);
+    return this.parseKnowledgeGraph(response);
+  }
+
+  async searchDocumentation(query: string): Promise<Document[]> {
+    return this.vectorStore.similaritySearch(query, 5);
+  }
+}
+```
+
+#### Module 2: Test Case Generator
+
+```typescript
+class TestCaseGenerator {
+  constructor(
+    private analyzer: DocumentationAnalyzer,
+    private llm: LLMProvider
+  ) {}
+
+  async generateTestPlan(knowledge: KnowledgeGraph): Promise<TestPlan> {
+    const testCases: TestCase[] = [];
+
+    // Generate test cases for each command
+    for (const command of knowledge.commands) {
+      // 1. Basic happy path test
+      testCases.push(await this.generateHappyPathTest(command));
+
+      // 2. Parameter variation tests
+      testCases.push(...await this.generateParameterTests(command));
+
+      // 3. Error scenario tests
+      testCases.push(...await this.generateErrorTests(command));
+
+      // 4. Example-based tests
+      testCases.push(...await this.generateExampleTests(command));
+    }
+
+    // Generate workflow tests (multi-step)
+    testCases.push(...await this.generateWorkflowTests(knowledge));
+
+    return {
+      testCases,
+      totalCommands: knowledge.commands.length,
+      coverage: this.calculateCoverage(testCases, knowledge)
+    };
+  }
+
+  async generateHappyPathTest(command: Command): Promise<TestCase> {
+    const prompt = `
+      Generate a test case for the command: ${command.name}
+
+      Documentation says: ${command.description}
+      Parameters: ${JSON.stringify(command.parameters)}
+      Expected output: ${JSON.stringify(command.expectedOutputs)}
+
+      Create a test case that validates the basic happy path:
+      - Command with required parameters only
+      - Expected successful execution
+      - Output validation criteria
+    `;
+
+    const response = await this.llm.complete(prompt);
+    return this.parseTestCase(response);
+  }
+}
+```
+
+#### Module 3: Execution Orchestrator
+
+```typescript
+class ExecutionOrchestrator {
+  constructor(
+    private adapter: PlatformAdapter,
+    private validator: ValidationEngine
+  ) {}
+
+  async runTestPlan(plan: TestPlan): Promise<TestResults> {
+    const results: TestResult[] = [];
+
+    // Set up test environment
+    await this.setupEnvironment();
+
+    try {
+      // Execute test cases in dependency order
+      for (const testCase of this.sortByDependencies(plan.testCases)) {
+        const result = await this.runTestCase(testCase);
+        results.push(result);
+
+        // Stop on critical failure
+        if (testCase.critical && !result.passed) {
+          break;
+        }
+      }
+    } finally {
+      // Clean up
+      await this.cleanupEnvironment();
+    }
+
+    return {
+      summary: this.calculateSummary(results),
+      results,
+      coverage: this.calculateCoverage(results, plan)
+    };
+  }
+
+  async runTestCase(testCase: TestCase): Promise<TestResult> {
+    const startTime = Date.now();
+
+    try {
+      // 1. Execute command/action
+      const executionResult = await this.adapter.execute(testCase);
+
+      // 2. Validate result
+      const validations = await this.validator.validate(
+        executionResult,
+        testCase.expectedBehavior
+      );
+
+      // 3. Collect evidence
+      const evidence = await this.collectEvidence(executionResult);
+
+      return {
+        testCaseId: testCase.id,
+        passed: validations.every(v => v.passed),
+        duration: Date.now() - startTime,
+        validations,
+        evidence
+      };
+    } catch (error) {
+      return {
+        testCaseId: testCase.id,
+        passed: false,
+        duration: Date.now() - startTime,
+        error: error.message,
+        evidence: { error: error.stack }
+      };
+    }
+  }
+}
+```
+
+#### Module 4: AI Validation Engine
+
+```typescript
+class AIValidationEngine {
+  constructor(private llm: LLMProvider) {}
+
+  async validateSemantics(
+    actual: any,
+    expected: ExpectedBehavior
+  ): Promise<ValidationResult> {
+    const prompt = `
+      You are validating software output against documented behavior.
+
+      Expected behavior from documentation:
+      ${JSON.stringify(expected, null, 2)}
+
+      Actual output received:
+      ${JSON.stringify(actual, null, 2)}
+
+      Analyze whether the actual output matches the expected behavior.
+      Consider:
+      1. Does the structure match?
+      2. Do the values make sense?
+      3. Are all required fields present?
+      4. Do error messages match expected patterns?
+      5. Are there any discrepancies?
+
+      Respond with:
+      {
+        "passed": true/false,
+        "confidence": 0.0-1.0,
+        "reasoning": "explanation",
+        "discrepancies": ["list of issues found"]
+      }
+    `;
+
+    const response = await this.llm.complete(prompt, {
+      responseFormat: 'json'
+    });
+
+    return JSON.parse(response);
+  }
+
+  async validateVisual(
+    screenshot: Buffer,
+    expectedDescription: string
+  ): Promise<ValidationResult> {
+    const prompt = `
+      You are viewing a screenshot of a user interface.
+
+      The documentation states: "${expectedDescription}"
+
+      Based on the screenshot, does it match the documentation?
+      Consider layout, content, colors, and overall appearance.
+
+      Respond with validation result.
+    `;
+
+    // Use vision model (GPT-4V or Claude with vision)
+    const response = await this.llm.completeWithImage(prompt, screenshot);
+
+    return this.parseValidationResult(response);
+  }
+}
+```
+
+### Configuration Example
+
+```yaml
+# ai-test-config.yaml
+framework:
+  name: "Cakemail CLI AI Testing"
+  version: "1.0.0"
+
+documentation:
+  sources:
+    - path: "docs/user-manual"
+      type: "markdown"
+      recursive: true
+    - path: "README.md"
+      type: "markdown"
+    - path: "dist/cli.js --help"
+      type: "command_help"
+
+environment:
+  platform: "cli"
+  test_account:
+    email: "${CAKEMAIL_TEST_EMAIL}"
+    password: "${CAKEMAIL_TEST_PASSWORD}"
+  api_base: "https://api.cakemail.dev"
+  cleanup_after_tests: true
+
+ai_config:
+  llm_provider: "anthropic"
+  model: "claude-sonnet-4"
+  temperature: 0.1  # Low temperature for consistent validation
+  max_tokens: 4000
+
+test_generation:
+  auto_generate: true
+  include_edge_cases: true
+  include_error_scenarios: true
+  max_parameter_combinations: 5
+
+execution:
+  parallel: false  # Run sequentially for CLI
+  timeout_per_test: 60000  # 60 seconds
+  retry_on_failure: 1
+  capture_screenshots: true
+  capture_logs: true
+
+validation:
+  strict_mode: false  # Allow minor formatting differences
+  semantic_validation: true  # Use AI for semantic matching
+  visual_validation: false  # Not applicable for CLI
+
+reporting:
+  output_format: "html"
+  output_path: "test-results/ai-simulation"
+  create_github_issues: false
+  slack_notifications: false
+```
+
+---
+
+## Quality Metrics
+
+### Coverage Metrics
+
+```typescript
+interface CoverageMetrics {
+  // Documentation coverage
+  totalCommands: number;
+  testedCommands: number;
+  untested Commands: string[];
+
+  // Parameter coverage
+  totalParameters: number;
+  testedParameters: number;
+  parameterCombinationsCovered: number;
+
+  // Scenario coverage
+  happyPathsCovered: number;
+  errorScenariosCovered: number;
+  edgeCasesCovered: number;
+
+  // Example coverage
+  examplesInDocumentation: number;
+  examplesTested: number;
+}
+```
+
+### Quality Metrics
+
+```typescript
+interface QualityMetrics {
+  // Test execution
+  totalTests: number;
+  passed: number;
+  failed: number;
+  skipped: number;
+  flaky: number;
+
+  // Failure analysis
+  bugCount: number;  // Actual bugs in code
+  documentationIssues: number;  // Docs don't match reality
+  testIssues: number;  // Problems with test itself
+
+  // Confidence
+  averageConfidence: number;  // AI confidence in validations
+  manualReviewRequired: number;  // Low confidence cases
+}
+```
+
+### Reporting Dashboard
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│              AI User Simulation Test Results                 │
+│                   Cakemail CLI v1.7.0                        │
+└─────────────────────────────────────────────────────────────┘
+
+📊 Coverage Summary
+├── Commands: 136/136 (100%)
+├── Parameters: 342/450 (76%)
+├── Examples: 45/45 (100%)
+└── Workflows: 12/15 (80%)
+
+✅ Test Results
+├── Total: 234 tests
+├── Passed: 228 (97.4%)
+├── Failed: 6 (2.6%)
+└── Duration: 12m 34s
+
+❌ Failed Tests (6)
+┌────────────────────────────────────────────────────────────┐
+│ CAMP-042: Create campaign with invalid list ID             │
+│ Status: FAILED                                             │
+│ Category: Bug                                              │
+│                                                            │
+│ Expected: "Error: List not found" message                 │
+│ Actual: Application crashed with unhandled exception      │
+│                                                            │
+│ Evidence: test-results/CAMP-042/stderr.txt                │
+│ Recommendation: Add error handling in campaigns.ts:145    │
+└────────────────────────────────────────────────────────────┘
+
+📝 Documentation Issues (2)
+┌────────────────────────────────────────────────────────────┐
+│ DOC-001: campaigns list output format                      │
+│                                                            │
+│ Documentation states: "Returns JSON by default"           │
+│ Reality: Returns profile-based format (table/json/compact)│
+│                                                            │
+│ Location: docs/user-manual/09-command-reference/02-*.md   │
+│ Suggestion: Update to mention profile-based defaults      │
+└────────────────────────────────────────────────────────────┘
+
+🎯 Recommendations
+1. Fix error handling in campaigns create (HIGH priority)
+2. Update documentation for output formats (MEDIUM priority)
+3. Add integration test for campaign deletion (LOW priority)
+```
+
+---
+
+## Example Scenarios
+
+### Scenario 1: CLI Command Validation
+
+**Documentation Extract:**
+```markdown
+# Create Campaign
+
+Creates a new email campaign.
+
+## Usage
+```bash
+cakemail campaigns create -n "Newsletter" -l 123 -s 456
+```
+
+## Parameters
+- `-n, --name <name>` - Campaign name (required)
+- `-l, --list-id <id>` - List ID (required)
+- `-s, --sender-id <id>` - Sender ID (required)
+
+## Output
+```json
+{
+  "id": 789,
+  "name": "Newsletter",
+  "status": "draft",
+  "list_id": 123,
+  "sender_id": 456
+}
+```
+```
+
+**Generated Test Case:**
+```typescript
+const testCase = {
+  id: "CAMP-CREATE-001",
+  command: "cakemail campaigns create -n 'AI Test Campaign' -l 123 -s 456",
+  expectedBehavior: {
+    exitCode: 0,
+    outputFormat: "json",
+    schema: {
+      type: "object",
+      required: ["id", "name", "status", "list_id", "sender_id"],
+      properties: {
+        id: { type: "number" },
+        name: { type: "string", enum: ["AI Test Campaign"] },
+        status: { type: "string", enum: ["draft"] },
+        list_id: { type: "number", enum: [123] },
+        sender_id: { type: "number", enum: [456] }
+      }
+    }
+  }
+};
+```
+
+**Execution & Validation:**
+```typescript
+// Execute
+const result = await executor.run(testCase.command);
+
+// Validate structure
+const structureValid = validateJSONSchema(
+  JSON.parse(result.stdout),
+  testCase.expectedBehavior.schema
+);
+
+// AI semantic validation
+const semanticValid = await aiValidator.validate({
+  actual: result.stdout,
+  expected: "Should create a draft campaign with the specified name, list, and sender",
+  confidence: 0.95
+});
+
+// Final result
+return {
+  passed: structureValid && semanticValid.passed,
+  confidence: semanticValid.confidence,
+  evidence: {
+    stdout: result.stdout,
+    exitCode: result.exitCode,
+    validations: [structureValid, semanticValid]
+  }
+};
+```
+
+### Scenario 2: Error Handling Validation
+
+**Documentation:**
+```markdown
+## Errors
+
+If the list ID doesn't exist, returns:
+```
+Error: List not found
+💡 Tip: To see all lists, use: cakemail lists list
+```
+Exit code: 1
+```
+
+**Test Case:**
+```typescript
+const testCase = {
+  id: "CAMP-CREATE-ERR-001",
+  command: "cakemail campaigns create -n 'Test' -l 999999 -s 456",
+  expectedBehavior: {
+    exitCode: 1,
+    stderrContains: "Error: List not found",
+    stderrContains: "cakemail lists list",
+    semanticExpectation: "Should show helpful error message with suggestion"
+  }
+};
+```
+
+### Scenario 3: Workflow Validation
+
+**Documentation:**
+```markdown
+# Campaign Lifecycle
+
+1. Create campaign: `cakemail campaigns create`
+2. Schedule campaign: `cakemail campaigns schedule <id> -d "2025-12-01"`
+3. Verify status: `cakemail campaigns get <id>` (should show status: "scheduled")
+```
+
+**Test Case:**
+```typescript
+const workflowTest = {
+  id: "WORKFLOW-CAMP-001",
+  steps: [
+    {
+      action: "create",
+      command: "cakemail campaigns create -n 'Workflow Test' -l 123 -s 456",
+      capture: "campaign_id",
+      expectedStatus: "draft"
+    },
+    {
+      action: "schedule",
+      command: "cakemail campaigns schedule {{campaign_id}} -d '2025-12-01T10:00:00Z'",
+      expectedOutput: "Campaign scheduled successfully"
+    },
+    {
+      action: "verify",
+      command: "cakemail campaigns get {{campaign_id}}",
+      expectedField: { status: "scheduled", scheduled_for: "2025-12-01T10:00:00Z" }
+    }
+  ],
+  cleanup: [
+    "cakemail campaigns delete {{campaign_id}} --force"
+  ]
+};
+```
+
+---
+
+## Future Enhancements
+
+### Phase 2: Self-Healing Tests
+
+**Adaptive Test Cases:**
+- AI detects when UI elements change (selectors, IDs)
+- Automatically updates test cases to match new implementation
+- Learns from manual corrections to improve future adaptations
+
+**Example:**
+```typescript
+// Button selector changed from #create-btn to #new-campaign-btn
+// AI detects failure, searches for similar elements, updates selector
+const healedTest = await aiHealer.attemptHeal(failedTest, {
+  oldSelector: "#create-btn",
+  context: "Create campaign button on campaigns list page"
+});
+// → Suggests: "#new-campaign-btn" based on text content and position
+```
+
+### Phase 3: Exploratory Testing
+
+**AI Explorer:**
+- AI actively explores the application beyond documented features
+- Discovers undocumented features or edge cases
+- Tests combinations of actions not explicitly documented
+- Reports unexpected behavior or potential bugs
+
+**Example:**
+```typescript
+const explorerAgent = new AIExplorer({
+  objective: "Explore campaign management features",
+  constraints: ["Don't delete production data", "Stay in test account"],
+  creativity: 0.7  // Balance between following patterns and trying new things
+});
+
+const findings = await explorerAgent.explore();
+// → Discovers: "Can create campaign without sender if template has default sender"
+//   (Not documented, but works in practice)
+```
+
+### Phase 4: Multi-Platform Consistency Testing
+
+**Cross-Platform Validator:**
+- Tests same feature across CLI, web, and mobile
+- Validates consistent behavior and data
+- Identifies platform-specific bugs or discrepancies
+
+**Example:**
+```yaml
+consistency_test:
+  feature: "Create Campaign"
+  platforms:
+    - cli: "cakemail campaigns create -n 'Test' -l 123 -s 456"
+    - web: "Navigate to /campaigns → Click Create → Fill form → Submit"
+    - mobile: "Tap Campaigns → Tap + → Enter details → Tap Save"
+
+  expected_consistency:
+    - all_create_same_campaign: true
+    - all_show_same_success_message: true
+    - campaign_appears_in_all_platforms: true
+```
+
+### Phase 5: Continuous Documentation Validation
+
+**CI/CD Integration:**
+```yaml
+# .github/workflows/ai-doc-validation.yml
+name: AI Documentation Validation
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'docs/**'
+      - 'src/**'
+  pull_request:
+
+jobs:
+  validate-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Run AI User Simulation
+        run: npm run ai-test
+        env:
+          CAKEMAIL_TEST_EMAIL: ${{ secrets.TEST_EMAIL }}
+          CAKEMAIL_TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+
+      - name: Generate Report
+        run: npm run ai-test:report
+
+      - name: Create GitHub Issues for Failures
+        if: failure()
+        run: npm run ai-test:create-issues
+
+      - name: Comment on PR
+        if: github.event_name == 'pull_request'
+        uses: actions/github-script@v6
+        with:
+          script: |
+            const report = require('./test-results/summary.json');
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              body: `## AI Documentation Validation Results\n\n${report.summary}`
+            });
+```
+
+---
+
+## Cost Considerations
+
+### LLM API Costs (Estimated)
+
+**Assumptions:**
+- 136 commands × 3 test cases each = 408 test cases
+- Average 2,000 tokens per test generation
+- Average 1,000 tokens per validation
+
+**Test Generation:**
+- 408 tests × 2,000 tokens = 816,000 tokens
+- Cost (Claude Sonnet): ~$2.45 per test run
+
+**Test Validation:**
+- 408 tests × 1,000 tokens = 408,000 tokens
+- Cost (Claude Sonnet): ~$1.22 per test run
+
+**Total per full test run:** ~$3.67
+
+**Monthly cost (daily runs):** ~$110/month
+
+### Optimization Strategies
+
+1. **Cache generated test cases** (regenerate only when docs change)
+2. **Use cheaper models for simple validations** (GPT-3.5 for schema validation)
+3. **Batch API calls** to reduce overhead
+4. **Run full suite weekly**, quick validations daily
+5. **Use local models** (Llama 3) for non-critical validations
+
+---
+
+## Conclusion
+
+This AI-based user simulation framework provides:
+
+✅ **Automated documentation validation** - Ensures docs match reality
+✅ **Platform-agnostic design** - Works for CLI, web, mobile
+✅ **Continuous quality assurance** - Runs in CI/CD
+✅ **Self-healing capabilities** - Adapts to minor changes
+✅ **Comprehensive coverage** - Tests happy paths, errors, edge cases
+✅ **Evidence-based reporting** - Screenshots, logs, detailed analysis
+
+The framework shifts testing from "what developers think works" to "what users experience based on documentation," ensuring a seamless user experience across all platforms.
+
+---
+
+## Appendix: Sample Test Output
+
+```json
+{
+  "testRunId": "run-2025-10-26-143022",
+  "framework": "ai-user-simulation",
+  "version": "1.0.0",
+  "platform": "cli",
+  "application": {
+    "name": "Cakemail CLI",
+    "version": "1.7.0"
+  },
+  "summary": {
+    "totalTests": 408,
+    "passed": 396,
+    "failed": 10,
+    "skipped": 2,
+    "duration": 1847000,
+    "passRate": 97.06
+  },
+  "coverage": {
+    "commands": { "total": 136, "tested": 136, "percentage": 100 },
+    "parameters": { "total": 450, "tested": 342, "percentage": 76 },
+    "examples": { "total": 45, "tested": 45, "percentage": 100 }
+  },
+  "failures": [
+    {
+      "testId": "CAMP-042",
+      "category": "bug",
+      "severity": "high",
+      "title": "Create campaign with invalid list ID crashes",
+      "expected": "Error message with helpful suggestion",
+      "actual": "Unhandled exception",
+      "evidence": {
+        "stderr": "TypeError: Cannot read property 'id' of undefined\n    at createCampaign (src/commands/campaigns.ts:145)",
+        "exitCode": 1
+      },
+      "recommendation": "Add null check before accessing list properties",
+      "githubIssue": null
+    }
+  ],
+  "documentationIssues": [
+    {
+      "issueId": "DOC-001",
+      "severity": "medium",
+      "title": "Default output format documentation incorrect",
+      "location": "README.md:104",
+      "expected": "Returns JSON by default",
+      "actual": "Returns profile-based format (developer=json, marketer=compact, balanced=table)",
+      "suggestion": "Update documentation to mention profile-based defaults",
+      "fixedIn": "PR #123"
+    }
+  ],
+  "aiMetrics": {
+    "averageConfidence": 0.94,
+    "lowConfidenceTests": 3,
+    "manualReviewRequired": 2
+  }
+}
+```