@democratize-quality/mcp-server 1.2.0 → 1.2.1

package/src/skills/test-execution/SKILL.md

@@ -0,0 +1,777 @@
+---
+name: test-execution
+description: Execute API tests directly from test plans using actual API requests. Run tests section-by-section or entirely, validate responses, chain requests, and generate comprehensive HTML reports. Use when user wants to run tests, execute API calls from test plan, validate endpoints, or create test execution reports.
+---
+
+# API Test Execution Skill
+
+Use this skill to execute API tests directly from test plans without writing code. Run comprehensive test sequences, validate responses, and generate visual HTML reports.
+
+## When to Use This Skill
+
+- User has a test plan and wants to execute tests immediately
+- User mentions running tests without generating code
+- User needs to validate API endpoints from test plan
+- User wants to execute specific sections of a test plan
+- User requests test execution reports or validation results
+- User needs quick API testing without setup
+
+## Core Workflow
+
+### Step 1: Load Test Plan
+
+**Read the test plan file** that was generated by `api_planner`:
+
+```javascript
+// User provides test plan path
+const testPlanPath = "./api-test-reports/api-test-plan.md";
+
+// Read the test plan content
+const testPlanContent = await readFile(testPlanPath);
+
+// Parse sections using markdown headers (##)
+// Each section represents a testable API endpoint or feature
+```
+
+### Step 2: Select Section or Full Execution
+
+**Ask user what to execute** (if not specified):
+
+```javascript
+// Parse test plan sections
+const sections = parseTestPlanSections(testPlanContent);
+
+// User can specify:
+// - "Execute section 1" or "Run tests for GET /api/v1/Books"
+// - "Execute all sections" or "Run entire test plan"
+// - "Run sections 1, 3, and 5"
+
+// Section structure example:
+// ## 1. GET /api/v1/Books - Retrieve all books
+// - Endpoint: GET /api/v1/Books
+// - Expected Status: 200
+// - Sample Request: [details]
+// - Expected Response: [structure]
+```
+
+**Section Selection Examples:**
+- Section number: "Execute section 1"
+- Endpoint match: "Run tests for Books API"
+- Multiple sections: "Execute sections 1, 2, and 4"
+- Full plan: "Run all tests" or "Execute entire test plan"
+
+### Step 3: Parse Test Data from Section
+
+**Extract test information** from the selected section(s):
+
+```javascript
+// For each test section, extract:
+const testData = {
+  method: "GET",              // HTTP method
+  endpoint: "/api/v1/Books",  // API endpoint
+  baseUrl: "https://api.example.com",  // From test plan overview
+  expectedStatus: 200,        // Expected HTTP status
+  headers: {                  // Authentication and headers
+    "Content-Type": "application/json",
+    "Authorization": "Bearer token-if-needed"
+  },
+  data: {                     // Request body (for POST/PUT/PATCH)
+    // Parsed from sample request in test plan
+  },
+  expectedResponse: {         // Expected response structure
+    // Parsed from expected response section
+  }
+}
+```
+
+**Parsing Rules:**
+- Extract HTTP method from section title or endpoint line
+- Get endpoint URL from "Endpoint:" line
+- Parse expected status from "Expected Status:" or "Status Code:" line
+- Extract request body from "Sample Request:" or "Request Body:" section
+- Get expected response from "Expected Response:" or "Response Structure:" section
+- Identify authentication requirements from test plan
+
+### Step 3a: Smart Variable Extraction
+
+When the user specifies a variable to extract (e.g., "extract token and userId"), **automatically resolve the correct path from the actual API response structure** — never rely on hardcoded field names. Use structural and semantic reasoning against the real response.
+
+#### Rule 1: Semantic Field Matching
+
+For the requested variable name, find the best matching field in the response using semantic proximity — not exact name matching. Apply these principles:
+
+- **Partial name match**: A requested variable `userId` should match any field whose name contains `id` within an object whose name contains `user` — e.g., `user.id`, `account.id`, `customer.identifier`, `user.uuid` all qualify
+- **Token/auth semantics**: A requested variable `token` should match any field whose name contains `token`, `jwt`, `key`, `secret`, or `credential` at any nesting level
+- **ID semantics**: A requested variable `orderId` should match any field that looks like the primary identifier of the returned resource — typically a field named `id`, `uuid`, `reference`, or `<resourceName>Id` at the top level of the created object
+- **When multiple candidates exist**, prefer the one closest to the top level and with the shortest path
+
+#### Rule 2: Collection First-Item Extraction
+
+When the user says "extract X from first result / first item / first product", do not assume the array key name. Instead:
+
+1. Inspect the response for **any key whose value is an array** with at least one element
+2. Take the first element (`[0]`)
+3. Within that element, find the field semantically matching the requested variable (apply Rule 1)
+
+This works regardless of whether the collection is called `products`, `records`, `results`, `items`, `data`, `entries`, or anything else.
+
+#### Rule 3: Response Envelope Unwrapping
+
+Many APIs wrap the actual payload in an envelope. Before applying extraction rules, detect and unwrap one level automatically:
+
+- **Single-key object**: `{ data: { ... } }` → unwrap `data`
+- **Resource + metadata**: `{ items: [...], total: N, page: N }` → the array key is the payload
+- **Auth envelope**: `{ user: { ... }, token: "..." }` → both `user` and `token` are top-level
+- **Flat response**: `{ id: ..., name: ... }` → no unwrapping needed
+
+If extraction still fails after unwrapping one level, try two levels deep before giving up.
+
+#### Rule 4: Extraction Verification Guard
+
+Before injecting any `{{variable}}` into a subsequent request, verify it resolved to a non-empty, non-null, non-undefined value. If verification fails:
+
+1. **Stop execution** — do not proceed to the next step
+2. **Report clearly**: state which variable failed, which step it came from, and show the actual response structure that was searched
+3. **Never produce broken URLs** — a URL like `/api/cart//items` (empty segment) is a silent failure and must be caught here
+
+#### Rule 5: Type-Safe Substitution
+
+When injecting extracted values into request bodies, preserve the original type from the response. Do not coerce types:
+
+- A number in the response (`"price": 3.99`) must stay a number in the request body — not become a string `"3.99"`
+- A boolean stays a boolean
+- A string stays a string
+- Arrays and objects are passed by reference as-is
+
+### Step 4: Execute Tests with Unique Session ID
+
+**Create a unique session** and execute tests:
+
+```javascript
+// Generate unique session ID
+const sessionId = `test-execution-${Date.now()}`;
+// Or use meaningful name: `test-execution-books-api-${Date.now()}`
+
+// Execute each test in the section
+for (const test of testsInSection) {
+  await tools.api_request({
+    sessionId: sessionId,
+    method: test.method,
+    url: `${test.baseUrl}${test.endpoint}`,
+    headers: test.headers,
+    data: test.data,  // Only for POST/PUT/PATCH/DELETE
+    expect: {
+      status: test.expectedStatus,
+      // Optional: Add response validation
+      body: {
+        // Expected fields from test plan
+      }
+    },
+    extract: {
+      // Extract data for chained requests
+      // Example: { "bookId": "id" } to use in next request
+    }
+  })
+  
+  console.log(`✅ Test passed: ${test.method} ${test.endpoint}`);
+}
+```
+
+**Request Chaining Example:**
+```javascript
+// Step 1: Create a book and extract its ID
+await tools.api_request({
+  sessionId: sessionId,
+  method: "POST",
+  url: "https://api.example.com/api/v1/Books",
+  data: {
+    title: "Test Book",
+    author: "Test Author"
+  },
+  expect: { status: 201 },
+  extract: { "bookId": "id" }  // Save book ID for next request
+})
+
+// Step 2: Use extracted bookId in next request
+await tools.api_request({
+  sessionId: sessionId,
+  method: "GET",
+  url: "https://api.example.com/api/v1/Books/{{bookId}}",  // Uses extracted ID
+  expect: { status: 200 }
+})
+
+// Step 3: Update the book
+await tools.api_request({
+  sessionId: sessionId,
+  method: "PUT",
+  url: "https://api.example.com/api/v1/Books/{{bookId}}",
+  data: {
+    title: "Updated Test Book",
+    author: "Test Author"
+  },
+  expect: { status: 200 }
+})
+
+// Step 4: Delete the book
+await tools.api_request({
+  sessionId: sessionId,
+  method: "DELETE",
+  url: "https://api.example.com/api/v1/Books/{{bookId}}",
+  expect: { status: 204 }
+})
+```
+
+### Step 5: Generate HTML Test Report
+
+**After all tests execute**, generate comprehensive report:
+
+```javascript
+await tools.api_session_report({
+  sessionId: sessionId,
+  outputPath: `./api-test-reports/${sessionId}-report.html`
+})
+
+// Optionally check final status
+await tools.api_session_status({
+  sessionId: sessionId
+})
+```
+
+## Execution Modes
+
+### Mode 1: Single Section Execution
+```javascript
+// User: "Execute section 1 from the test plan"
+
+// 1. Parse test plan
+// 2. Extract section 1
+// 3. Execute all tests in section 1
+// 4. Generate report
+```
+
+### Mode 2: Multiple Sections Execution
+```javascript
+// User: "Run tests for sections 1, 3, and 5"
+
+// 1. Parse test plan
+// 2. Extract sections 1, 3, and 5
+// 3. Execute tests sequentially
+// 4. Generate single comprehensive report
+```
+
+### Mode 3: Full Plan Execution
+```javascript
+// User: "Execute all tests from the test plan"
+
+// 1. Parse entire test plan
+// 2. Execute all sections in order
+// 3. Handle dependencies between tests
+// 4. Generate comprehensive report with all results
+```
+
+### Mode 4: Filtered Execution
+```javascript
+// User: "Run only GET and POST tests"
+
+// 1. Parse test plan
+// 2. Filter tests by HTTP method
+// 3. Execute filtered tests
+// 4. Generate filtered report
+```
+
+## Test Plan Parsing
+
+### Expected Test Plan Structure
+
+```markdown
+# API Test Plan
+
+## Overview
+- Base URL: https://api.example.com
+- Authentication: Bearer Token
+- API Version: v1
+
+## 1. GET /api/v1/Books - Retrieve all books
+
+### Endpoint
+GET /api/v1/Books
+
+### Expected Status
+200 OK
+
+### Request Headers
+- Content-Type: application/json
+- Authorization: Bearer {{authToken}}
+
+### Sample Request
+No request body required
+
+### Expected Response
+```json
+[
+  {
+    "id": 1,
+    "title": "Book Title",
+    "author": "Author Name",
+    "isbn": "978-3-16-148410-0"
+  }
+]
+```
+
+### Validation Criteria
+- Response is an array
+- Each book has id, title, author fields
+- Status code is 200
+
+## 2. POST /api/v1/Books - Create new book
+
+### Endpoint
+POST /api/v1/Books
+
+### Expected Status
+201 Created
+
+### Request Headers
+- Content-Type: application/json
+- Authorization: Bearer {{authToken}}
+
+### Sample Request
+```json
+{
+  "title": "New Book Title",
+  "author": "Author Name",
+  "isbn": "978-3-16-148410-0",
+  "publishedDate": "2026-02-15"
+}
+```
+
+### Expected Response
+```json
+{
+  "id": 42,
+  "title": "New Book Title",
+  "author": "Author Name",
+  "isbn": "978-3-16-148410-0",
+  "publishedDate": "2026-02-15",
+  "createdAt": "2026-02-15T10:30:00Z"
+}
+```
+```
+
+### Parsing Strategy
+
+**1. Extract Base Configuration:**
+- Find "Base URL" or "API Base URL" in overview section
+- Extract authentication method and headers
+- Identify API version
+
+**2. Identify Test Sections:**
+- Look for markdown headers (##) that contain HTTP methods
+- Pattern: `## N. METHOD /endpoint - Description`
+- Examples: `## 1. GET /api/v1/Books`, `## 2. POST /api/v1/Users`
+
+**3. Parse Section Content:**
+- Extract endpoint URL
+- Parse expected status code
+- Find request headers
+- Extract request body from code blocks
+- Parse expected response from code blocks
+- Identify validation criteria
+
+**4. Handle Code Blocks:**
+```javascript
+// JSON code blocks contain sample data
+// Look for: ```json [content] ```
+// Parse as JSON for request/response data
+```
+
+## Error Handling and Validation
+
+### Test Failure Handling
+
+```javascript
+try {
+  await tools.api_request({
+    sessionId: sessionId,
+    method: "GET",
+    url: "https://api.example.com/api/v1/Books",
+    expect: { status: 200 }
+  })
+  console.log(`✅ Test passed: GET /api/v1/Books`);
+} catch (error) {
+  console.error(`❌ Test failed: GET /api/v1/Books`);
+  console.error(`   Error: ${error.message}`);
+  // Continue with remaining tests
+}
+```
+
+### Validation Options
+
+**Basic Status Validation:**
+```javascript
+expect: { status: 200 }
+```
+
+**Expected vs Actual Validation:**
+```javascript
+expect: {
+  status: 200,
+  body: {
+    "items": expectedResponseFromTestPlan,
+    "count": "{{any}}"  // Allow any value
+  }
+}
+```
+
+**Flexible Validation:**
+```javascript
+expect: {
+  status: [200, 201],  // Accept either status
+  body: {
+    "id": "{{number}}",      // Must be a number
+    "title": "{{string}}",   // Must be a string
+    "items": "{{array}}"     // Must be an array
+  }
+}
+```
+
+## Session Management
+
+### Tracking Test Execution
+
+```javascript
+// Check test progress
+await tools.api_session_status({
+  sessionId: sessionId
+})
+
+// Output includes:
+// - Total requests made
+// - Successful requests
+// - Failed requests
+// - Extracted variables
+// - Request history
+```
+
+### Session Naming Conventions
+
+**Descriptive Session IDs:**
+```javascript
+// Format: test-execution-<feature>-<timestamp>
+const sessionId = `test-execution-books-api-${Date.now()}`;
+const sessionId = `test-execution-full-plan-${Date.now()}`;
+const sessionId = `test-execution-section-1-${Date.now()}`;
+
+// Benefits:
+// - Easy to identify in reports
+// - Helps organize multiple test runs
+// - Clear audit trail
+```
+
+## Report Generation
+
+### HTML Report Features
+
+The `api_session_report` tool generates:
+
+- **Test Summary**
+  - Total tests executed
+  - Pass/fail statistics
+  - Execution duration
+  - Success rate percentage
+
+- **Request Details**
+  - Each API call made
+  - Request method, URL, headers, body
+  - Response status, headers, body
+  - Timing information (response time)
+
+- **Validation Results**
+  - Expected vs Actual comparisons
+  - Detailed failure messages
+  - Status code validation
+  - Response body validation
+
+- **Visual Elements**
+  - Color-coded pass/fail indicators (✅/❌)
+  - Expandable request/response sections
+  - Timing charts
+  - Interactive HTML with filtering
+
+### Report Example
+
+```javascript
+// Generate comprehensive report
+await tools.api_session_report({
+  sessionId: "test-execution-books-api-1739597400000",
+  outputPath: "./api-test-reports/books-api-execution-report.html"
+})
+
+// Report file created at:
+// ./api-test-reports/books-api-execution-report.html
+
+// Open in browser to view:
+// - 15 total requests
+// - 13 passed (86.7%)
+// - 2 failed (13.3%)
+// - Average response time: 145ms
+// - Total execution time: 2.3s
+```
+
+## Advanced Scenarios
+
+### Scenario 1: Testing with Authentication
+
+```javascript
+// Step 1: Authenticate first
+await tools.api_request({
+  sessionId: sessionId,
+  method: "POST",
+  url: "https://api.example.com/auth/login",
+  data: {
+    email: "test@example.com",
+    password: "password123"
+  },
+  expect: { status: 200 },
+  extract: { "authToken": "access_token" }
+})
+
+// Step 2: Use token in subsequent requests
+await tools.api_request({
+  sessionId: sessionId,
+  method: "GET",
+  url: "https://api.example.com/api/v1/protected-resource",
+  headers: {
+    "Authorization": "Bearer {{authToken}}"  // Uses extracted token
+  },
+  expect: { status: 200 }
+})
+```
+
+### Scenario 2: Testing CRUD Operations
+
+```javascript
+const sessionId = `test-execution-crud-${Date.now()}`;
+
+// CREATE
+await tools.api_request({
+  sessionId: sessionId,
+  method: "POST",
+  url: "https://api.example.com/api/v1/Books",
+  data: { title: "Test Book", author: "Test Author" },
+  expect: { status: 201 },
+  extract: { "newBookId": "id" }
+})
+
+// READ
+await tools.api_request({
+  sessionId: sessionId,
+  method: "GET",
+  url: "https://api.example.com/api/v1/Books/{{newBookId}}",
+  expect: { status: 200 }
+})
+
+// UPDATE
+await tools.api_request({
+  sessionId: sessionId,
+  method: "PUT",
+  url: "https://api.example.com/api/v1/Books/{{newBookId}}",
+  data: { title: "Updated Book", author: "Test Author" },
+  expect: { status: 200 }
+})
+
+// DELETE
+await tools.api_request({
+  sessionId: sessionId,
+  method: "DELETE",
+  url: "https://api.example.com/api/v1/Books/{{newBookId}}",
+  expect: { status: 204 }
+})
+
+// Generate report showing complete CRUD workflow
+await tools.api_session_report({
+  sessionId: sessionId,
+  outputPath: "./api-test-reports/crud-test-report.html"
+})
+```
+
+### Scenario 3: Parallel Execution (Multiple Sessions)
+
+```javascript
+// Execute different sections in parallel
+const session1 = `test-execution-books-${Date.now()}`;
+const session2 = `test-execution-users-${Date.now()}`;
+const session3 = `test-execution-orders-${Date.now()}`;
+
+// Run tests for different sections independently
+// Note: These should be awaited sequentially or use Promise.all if truly parallel
+
+// Generate individual reports
+await tools.api_session_report({
+  sessionId: session1,
+  outputPath: "./api-test-reports/books-report.html"
+})
+
+await tools.api_session_report({
+  sessionId: session2,
+  outputPath: "./api-test-reports/users-report.html"
+})
+
+await tools.api_session_report({
+  sessionId: session3,
+  outputPath: "./api-test-reports/orders-report.html"
+})
+```
+
+## Best Practices
+
+### DO:
+- Use unique, descriptive session IDs for each test run
+- Parse test plan systematically (base URL, auth, sections)
+- Execute tests in logical order (auth first, dependencies second)
+- Use request chaining for dependent tests (extract/reuse IDs)
+- Generate comprehensive HTML reports after execution
+- Handle test failures gracefully (continue with remaining tests)
+- Provide clear progress updates to user during execution
+- Validate responses using expect parameters
+- Use consistent session naming conventions
+
+### DON'T:
+- Don't reuse session IDs across different test runs
+- Don't skip authentication if test plan requires it
+- Don't execute dependent tests out of order
+- Don't forget to generate the final HTML report
+- Don't stop execution on first failure (unless critical)
+- Don't ignore validation criteria from test plan
+- Don't execute tests without parsing test plan structure first
+- Don't forget to extract base URL from test plan
+
+## Expected Output
+
+After successful test execution, provide user with:
+
+1. **Execution Summary:**
+```
+Test Execution Summary:
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Session ID: test-execution-books-api-1739597400000
+Total Tests: 15
+Passed: 13 (86.7%)
+Failed: 2 (13.3%)
+Execution Time: 2.3s
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+Test Results:
+✅ GET /api/v1/Books - List all books
+✅ GET /api/v1/Books/{id} - Get book by ID
+✅ POST /api/v1/Books - Create new book
+❌ PUT /api/v1/Books/{id} - Update book (404 Not Found)
+✅ DELETE /api/v1/Books/{id} - Delete book
+...
+
+📊 HTML Report Generated:
+./api-test-reports/books-api-execution-report.html
+
+Open this file in a browser for detailed analysis!
+```
+
+2. **Failed Test Details:**
+```
+Failed Tests:
+
+❌ PUT /api/v1/Books/{id}
+   Expected: 200 OK
+   Actual: 404 Not Found
+   Response: {"error": "Book not found"}
+   
+   Possible causes:
+   - Book ID may have been deleted
+   - Endpoint URL may have changed
+   - Authentication may have expired
+```
+
+3. **Next Steps:**
+```
+Next Steps:
+1. Open HTML report for detailed analysis
+2. Review failed tests and fix issues
+3. Re-run failed tests or use /test-healing skill
+4. Update test plan if API has changed
+```
+
+## Integration with Other Skills
+
+### After API Planning → Execute Tests
+```
+1. User: /api-planning create test plan for API
+   → Test plan generated
+
+2. User: /test-execution run tests from section 1
+   → Tests executed, report generated
+```
+
+### Execute → Heal → Re-execute
+```
+1. User: /test-execution run all tests
+   → Some tests fail
+
+2. User: /test-healing fix failed tests  
+   → Tests updated with fixes
+
+3. User: /test-execution re-run failed tests
+   → All tests pass now
+```
+
+### Execute → Generate Code → Execute Again
+```
+1. User: /test-execution run quick validation
+   → Quick API validation completed
+
+2. User: /test-generation create Playwright tests
+   → Executable test code generated
+
+3. User: Run generated Playwright tests
+   → Automated test suite running
+```
+
+## Troubleshooting
+
+### Issue: Base URL not found in test plan
+- Check test plan overview section
+- Ask user to provide base URL
+- Look for API URL in first few sections
+
+### Issue: Cannot parse test sections
+- Verify test plan uses markdown headers (##)
+- Check section titles include HTTP method
+- Ensure sections have required fields (endpoint, status)
+
+### Issue: Authentication failures (401/403)
+- Extract auth requirements from test plan
+- Execute auth endpoint first
+- Use extracted tokens in subsequent requests
+
+### Issue: Request chaining not working
+- The skill will automatically try common extraction paths — check the session log to see what path resolved and what value was found
+- If all common paths fail, inspect the raw response body in the session log and look for the field manually
+- Ensure the variable name used in `{{variableName}}` matches exactly what was extracted
+- Never pass an empty variable into a URL — the skill should halt and report before this happens
+
+### Issue: Tests fail with timeouts
+- Check API availability
+- Verify network connectivity
+- Consider increasing timeout in api_request
+
+### Issue: Report not generated
+- Ensure session ID is correct
+- Verify api_session_report is called after all tests
+- Check output path is valid
+
+## Additional Tools Available
+
+Use these tools as needed:
+- `api_request` - Execute individual API requests
+- `api_session_status` - Check test execution progress
+- `api_session_report` - Generate HTML test reports
+- `search/readFile` - Read test plan files
+- `search/listDirectory` - Find test plan files