agentic-qe 2.5.6 → 2.5.8

package/.claude/skills/n8n-workflow-testing-fundamentals/SKILL.md

@@ -0,0 +1,447 @@
+---
+name: n8n-workflow-testing-fundamentals
+description: "Comprehensive n8n workflow testing including execution lifecycle, node connection patterns, data flow validation, and error handling strategies. Use when testing n8n workflow automation applications."
+category: n8n-testing
+priority: high
+tokenEstimate: 1200
+agents: [n8n-workflow-executor, n8n-node-validator, n8n-trigger-test]
+implementation_status: production
+optimization_version: 1.0
+last_optimized: 2025-12-15
+dependencies: []
+quick_reference_card: true
+tags: [n8n, workflow, automation, testing, data-flow, nodes, triggers]
+---
+
+# n8n Workflow Testing Fundamentals
+
+<default_to_action>
+When testing n8n workflows:
+1. VALIDATE workflow structure before execution
+2. TEST with realistic test data
+3. VERIFY node-to-node data flow
+4. CHECK error handling paths
+5. MEASURE execution performance
+
+**Quick n8n Testing Checklist:**
+- All nodes properly connected (no orphans)
+- Trigger node correctly configured
+- Data mappings between nodes valid
+- Error workflows defined
+- Credentials properly referenced
+
+**Critical Success Factors:**
+- Test each execution path separately
+- Validate data transformations at each node
+- Check retry and error handling behavior
+- Verify integrations with external services
+</default_to_action>
+
+## Quick Reference Card
+
+### When to Use
+- Testing new n8n workflows
+- Validating workflow changes
+- Debugging failed executions
+- Performance optimization
+- Pre-deployment validation
+
+### n8n Workflow Components
+
+| Component | Purpose | Testing Focus |
+|-----------|---------|---------------|
+| **Trigger** | Starts workflow | Reliable activation, payload handling |
+| **Action Nodes** | Process data | Configuration, data mapping |
+| **Logic Nodes** | Control flow | Conditional routing, branches |
+| **Integration Nodes** | External APIs | Auth, rate limits, errors |
+| **Error Workflow** | Handle failures | Recovery, notifications |
+
+### Workflow Execution States
+
+| State | Meaning | Test Action |
+|-------|---------|-------------|
+| `running` | Currently executing | Monitor progress |
+| `success` | Completed successfully | Validate outputs |
+| `failed` | Execution failed | Analyze error |
+| `waiting` | Waiting for trigger | Test trigger mechanism |
+
+---
+
+## Workflow Structure Validation
+
+```typescript
+// Validate workflow structure before execution
+async function validateWorkflowStructure(workflowId: string) {
+  const workflow = await getWorkflow(workflowId);
+
+  // Check for trigger node
+  const triggerNode = workflow.nodes.find(n =>
+    n.type.includes('trigger') || n.type.includes('webhook')
+  );
+  if (!triggerNode) {
+    throw new Error('Workflow must have a trigger node');
+  }
+
+  // Check for orphan nodes (no connections)
+  const connectedNodes = new Set();
+  for (const [source, targets] of Object.entries(workflow.connections)) {
+    connectedNodes.add(source);
+    for (const outputs of Object.values(targets)) {
+      for (const connections of outputs) {
+        for (const conn of connections) {
+          connectedNodes.add(conn.node);
+        }
+      }
+    }
+  }
+
+  const orphans = workflow.nodes.filter(n => !connectedNodes.has(n.name));
+  if (orphans.length > 0) {
+    console.warn('Orphan nodes detected:', orphans.map(n => n.name));
+  }
+
+  // Validate credentials
+  for (const node of workflow.nodes) {
+    if (node.credentials) {
+      for (const [type, ref] of Object.entries(node.credentials)) {
+        if (!await credentialExists(ref.id)) {
+          throw new Error(`Missing credential: ${type} for node ${node.name}`);
+        }
+      }
+    }
+  }
+
+  return { valid: true, orphans, triggerNode };
+}
+```
+
+---
+
+## Execution Testing
+
+```typescript
+// Test workflow execution with various inputs
+async function testWorkflowExecution(workflowId: string, testCases: TestCase[]) {
+  const results: TestResult[] = [];
+
+  for (const testCase of testCases) {
+    const startTime = Date.now();
+
+    // Execute workflow
+    const execution = await executeWorkflow(workflowId, testCase.input);
+
+    // Wait for completion
+    const result = await waitForCompletion(execution.id, testCase.timeout || 30000);
+
+    // Validate output
+    const outputValid = validateOutput(result.data, testCase.expected);
+
+    results.push({
+      testCase: testCase.name,
+      success: result.status === 'success' && outputValid,
+      duration: Date.now() - startTime,
+      actualOutput: result.data,
+      expectedOutput: testCase.expected
+    });
+  }
+
+  return results;
+}
+
+// Example test cases
+const testCases = [
+  {
+    name: 'Valid customer data',
+    input: { name: 'John Doe', email: 'john@example.com' },
+    expected: { processed: true, customerId: /^cust_/ },
+    timeout: 10000
+  },
+  {
+    name: 'Missing email',
+    input: { name: 'Jane Doe' },
+    expected: { error: 'Email required' },
+    timeout: 5000
+  },
+  {
+    name: 'Invalid email format',
+    input: { name: 'Bob', email: 'not-an-email' },
+    expected: { error: 'Invalid email' },
+    timeout: 5000
+  }
+];
+```
+
+---
+
+## Data Flow Validation
+
+```typescript
+// Trace data through workflow nodes
+async function validateDataFlow(executionId: string) {
+  const execution = await getExecution(executionId);
+  const nodeResults = execution.data.resultData.runData;
+
+  const dataFlow: DataFlowStep[] = [];
+
+  for (const [nodeName, runs] of Object.entries(nodeResults)) {
+    for (const run of runs) {
+      dataFlow.push({
+        node: nodeName,
+        input: run.data?.main?.[0]?.[0]?.json || {},
+        output: run.data?.main?.[0]?.[0]?.json || {},
+        executionTime: run.executionTime,
+        status: run.executionStatus
+      });
+    }
+  }
+
+  // Validate data transformations
+  for (let i = 1; i < dataFlow.length; i++) {
+    const prev = dataFlow[i - 1];
+    const curr = dataFlow[i];
+
+    // Check if expected data passed through
+    validateDataMapping(prev.output, curr.input);
+  }
+
+  return dataFlow;
+}
+
+// Validate data mapping between nodes
+function validateDataMapping(sourceOutput: any, targetInput: any) {
+  // Check all required fields are present
+  const missingFields: string[] = [];
+
+  for (const [key, value] of Object.entries(targetInput)) {
+    if (value === undefined && sourceOutput[key] === undefined) {
+      missingFields.push(key);
+    }
+  }
+
+  if (missingFields.length > 0) {
+    console.warn('Missing fields in data mapping:', missingFields);
+  }
+
+  return missingFields.length === 0;
+}
+```
+
+---
+
+## Error Handling Testing
+
+```typescript
+// Test error handling paths
+async function testErrorHandling(workflowId: string) {
+  const errorScenarios = [
+    {
+      name: 'API timeout',
+      inject: { delay: 35000 }, // Trigger timeout
+      expectedError: 'timeout'
+    },
+    {
+      name: 'Invalid data',
+      inject: { invalidField: true },
+      expectedError: 'validation'
+    },
+    {
+      name: 'Missing credentials',
+      inject: { removeCredentials: true },
+      expectedError: 'authentication'
+    }
+  ];
+
+  const results: ErrorTestResult[] = [];
+
+  for (const scenario of errorScenarios) {
+    // Execute with error injection
+    const execution = await executeWithErrorInjection(workflowId, scenario.inject);
+
+    // Check error was caught
+    const result = await waitForCompletion(execution.id);
+
+    // Validate error handling
+    results.push({
+      scenario: scenario.name,
+      errorCaught: result.status === 'failed',
+      errorType: result.data?.resultData?.error?.type,
+      expectedError: scenario.expectedError,
+      errorWorkflowTriggered: await checkErrorWorkflowTriggered(execution.id),
+      alertSent: await checkAlertSent(execution.id)
+    });
+  }
+
+  return results;
+}
+
+// Verify error workflow was triggered
+async function checkErrorWorkflowTriggered(executionId: string): Promise<boolean> {
+  const errorExecutions = await getExecutions({
+    filter: {
+      metadata: { errorTriggeredBy: executionId }
+    }
+  });
+
+  return errorExecutions.length > 0;
+}
+```
+
+---
+
+## Node Connection Patterns
+
+### Linear Flow
+```
+Trigger → Process → Transform → Output
+```
+**Testing:** Execute once, validate each node output
+
+### Branching Flow
+```
+Trigger → IF → [Branch A] → Merge → Output
+              → [Branch B] →
+```
+**Testing:** Test both branches separately, verify merge behavior
+
+### Parallel Flow
+```
+Trigger → Split → [Process A] → Merge → Output
+                → [Process B] →
+```
+**Testing:** Validate parallel execution, check merge timing
+
+### Loop Flow
+```
+Trigger → SplitInBatches → Process → [Loop back until done] → Output
+```
+**Testing:** Test with varying batch sizes, verify all items processed
+
+---
+
+## Common Testing Patterns
+
+### Test Data Generation
+
+```typescript
+// Generate test data for common n8n patterns
+const testDataGenerators = {
+  webhook: () => ({
+    body: { event: 'test', timestamp: new Date().toISOString() },
+    headers: { 'Content-Type': 'application/json' },
+    query: { source: 'test' }
+  }),
+
+  slack: () => ({
+    type: 'message',
+    channel: 'C123456',
+    user: 'U789012',
+    text: 'Test message'
+  }),
+
+  github: () => ({
+    action: 'opened',
+    issue: {
+      number: 1,
+      title: 'Test Issue',
+      body: 'Test body'
+    },
+    repository: {
+      full_name: 'test/repo'
+    }
+  }),
+
+  stripe: () => ({
+    type: 'payment_intent.succeeded',
+    data: {
+      object: {
+        id: 'pi_test123',
+        amount: 1000,
+        currency: 'usd'
+      }
+    }
+  })
+};
+```
+
+### Execution Assertions
+
+```typescript
+// Common assertions for workflow execution
+const workflowAssertions = {
+  // Assert workflow completed
+  assertCompleted: (execution) => {
+    expect(execution.finished).toBe(true);
+    expect(execution.status).toBe('success');
+  },
+
+  // Assert specific node executed
+  assertNodeExecuted: (execution, nodeName) => {
+    const nodeData = execution.data.resultData.runData[nodeName];
+    expect(nodeData).toBeDefined();
+    expect(nodeData[0].executionStatus).toBe('success');
+  },
+
+  // Assert data transformation
+  assertDataTransformed: (execution, nodeName, expectedData) => {
+    const nodeOutput = execution.data.resultData.runData[nodeName][0].data.main[0][0].json;
+    expect(nodeOutput).toMatchObject(expectedData);
+  },
+
+  // Assert execution time
+  assertExecutionTime: (execution, maxMs) => {
+    const duration = new Date(execution.stoppedAt) - new Date(execution.startedAt);
+    expect(duration).toBeLessThan(maxMs);
+  }
+};
+```
+
+---
+
+## Agent Coordination Hints
+
+### Memory Namespace
+```
+aqe/n8n/
+├── workflows/*          - Cached workflow definitions
+├── test-results/*       - Test execution results
+├── validations/*        - Validation reports
+├── patterns/*           - Discovered testing patterns
+└── executions/*         - Execution tracking
+```
+
+### Fleet Coordination
+```typescript
+// Comprehensive n8n testing with fleet
+const n8nFleet = await FleetManager.coordinate({
+  strategy: 'n8n-testing',
+  agents: [
+    'n8n-workflow-executor',  // Execute and validate
+    'n8n-node-validator',     // Validate configurations
+    'n8n-trigger-test',       // Test triggers
+    'n8n-expression-validator', // Validate expressions
+    'n8n-integration-test'    // Test integrations
+  ],
+  topology: 'parallel'
+});
+```
+
+---
+
+## Related Skills
+- [n8n-expression-testing](../n8n-expression-testing/) - Expression validation
+- [n8n-trigger-testing-strategies](../n8n-trigger-testing-strategies/) - Trigger testing
+- [n8n-integration-testing-patterns](../n8n-integration-testing-patterns/) - Integration testing
+- [n8n-security-testing](../n8n-security-testing/) - Security validation
+
+---
+
+## Remember
+
+**n8n workflows are JSON-based execution flows** that connect 400+ services. Testing requires validating:
+- Workflow structure (nodes, connections)
+- Trigger reliability (webhooks, schedules)
+- Data flow (transformations between nodes)
+- Error handling (retry, fallback, notifications)
+- Performance (execution time, resource usage)
+
+**With Agents:** Use n8n-workflow-executor for execution testing, n8n-node-validator for configuration validation, and coordinate multiple agents for comprehensive workflow testing.

package/CHANGELOG.md

@@ -7,6 +7,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.5.8] - 2025-12-18
+
+### Added
+
+#### Phase 0: LLM Independence Foundation
+
+Major milestone implementing the foundation for reduced LLM dependency through pattern learning and vector similarity search.
+
+**M0.3: HNSW Pattern Store Integration**
+- **HNSWPatternAdapter** (`src/learning/HNSWPatternAdapter.ts`) - Bridge between LearningEngine and HNSWPatternStore
+  - O(log n) similarity search with <1ms p95 latency
+  - Converts LearnedPattern ↔ QEPattern formats
+  - Fallback hash-based embeddings when RuvLLM unavailable
+  - 768-dimension vector embeddings
+- **LearningEngine HNSW Integration** - Added `enableHNSW` config option
+  - `searchSimilarPatterns()` - Vector similarity search across learned patterns
+  - `getHNSWStats()` - Pattern count, embedding dimension, RuvLLM status
+  - `isHNSWEnabled()` - Check HNSW availability
+  - Dual storage: SQLite (primary) + HNSW (vector search)
+
+**M0.5: Federated Learning**
+- **FederatedManager** (`src/learning/FederatedManager.ts`) - Cross-agent pattern sharing
+  - Register agents with team for collective learning
+  - Share learned patterns across agent instances
+  - Sync with team knowledge on initialization
+
+**M0.6: Pattern Curation**
+- **PatternCurator** (`src/learning/PatternCurator.ts`) - Manual curation workflow
+  - `findLowConfidencePatterns()` - Identify patterns needing review
+  - `reviewPattern()` - Approve/reject patterns with feedback
+  - `autoCurate()` - Automatic curation based on confidence thresholds
+  - `forceLearning()` - Trigger learning consolidation
+  - Interactive curation generator for batch review
+- **RuvllmPatternCurator** (`src/providers/RuvllmPatternCurator.ts`) - RuvLLM integration
+  - Implements IPatternSource using HNSWPatternAdapter
+  - Implements ILearningTrigger using RuvllmProvider
+  - Enables 20% better routing through curated patterns
+
+**RuvllmProvider Enhancements**
+- **Session Management** - Multi-turn context preservation (50% latency reduction)
+  - `createSession()`, `getSession()`, `endSession()`
+  - Session timeout: 30 minutes, max 100 concurrent sessions
+- **Batch API** - Parallel request processing (4x throughput)
+  - `batchComplete()` for multiple prompts
+  - Rate limiting and queue management
+- **TRM (Test-time Reasoning & Metacognition)** - Iterative refinement
+  - Up to 7 iterations with 95% convergence threshold
+- **SONA (Self-Organizing Neural Architecture)** - Continuous adaptation
+  - LoRA rank: 8, alpha: 16, EWC lambda: 2000
+- **Learning Methods** - Pattern feedback and consolidation
+  - `searchMemory()`, `provideFeedback()`, `forceLearn()`, `getMetrics()`
+
+**HybridRouter Enhancements**
+- **RuVector Cache Integration** - Semantic caching with vector similarity
+- **Cost Optimization Routing** - Smart provider selection based on task complexity
+
+**New Components**
+- **RuVectorClient** (`src/providers/RuVectorClient.ts`) - Vector database client
+- **LLMBaselineTracker** (`src/providers/LLMBaselineTracker.ts`) - Performance baseline tracking
+
+#### Integration Tests
+- **phase0-integration.test.ts** - 18 comprehensive tests covering:
+  - HNSWPatternStore direct usage (4 tests)
+  - HNSWPatternAdapter with LearningEngine (3 tests)
+  - LearningEngine + HNSW integration (3 tests)
+  - PatternCurator session/curation workflow (7 tests)
+  - End-to-end: execute → learn → store → retrieve (1 test)
+
+#### Documentation
+- **agent-learning-system.md** - Complete architecture documentation
+  - Agent lifecycle with all integration points
+  - LLM provider selection matrix
+  - Learning from execution flow diagrams
+  - Pattern retrieval and acceleration explanation
+  - Ruv solutions integration summary
+
+### Changed
+- Updated `LearnedPattern` type with optional `agentId` and `averageReward` fields
+- Extended `src/learning/index.ts` with HNSWPatternAdapter exports
+- Extended `src/providers/index.ts` with RuvllmPatternCurator and RuVectorClient exports
+- Extended `src/memory/index.ts` with HNSWPatternStore exports
+
+### Fixed
+- Test isolation in HNSWPatternAdapter tests (unique temp directories per test)
+- TypeScript compilation errors in pattern conversion methods
+
+## [2.5.7] - 2025-12-17
+
+### Added
+
+#### n8n Workflow Testing Agents (PR #151)
+*Contributed by [@fndlalit](https://github.com/fndlalit)*
+
+Comprehensive suite of **15 n8n workflow testing agents** for production-ready workflow automation testing:
+
+- **N8nWorkflowExecutorAgent** - Execute workflows with data flow validation and assertions
+- **N8nPerformanceTesterAgent** - Load/stress testing with timing metrics and percentiles
+- **N8nChaosTesterAgent** - Fault injection using N8nTestHarness for real failure simulation
+- **N8nBDDScenarioTesterAgent** - Cucumber-style BDD testing with real execution
+- **N8nSecurityAuditorAgent** - 40+ secret patterns, runtime leak detection
+- **N8nExpressionValidatorAgent** - Safe expression validation using pattern matching
+- **N8nIntegrationTestAgent** - Real API connectivity testing via workflow execution
+- **N8nTriggerTestAgent** - Webhook testing with correct n8n URL patterns
+- **N8nComplianceValidatorAgent** - GDPR/HIPAA/SOC2/PCI-DSS compliance with runtime PII tracing
+- **N8nMonitoringValidatorAgent** - SLA compliance checking with runtime metrics
+- Plus 5 additional n8n agents (node-validator, unit-tester, version-comparator, ci-orchestrator, base-agent)
+
+**5 new n8n testing skills:**
+- `n8n-workflow-testing-fundamentals` - Core workflow testing concepts
+- `n8n-security-testing` - Credential and secret management testing
+- `n8n-integration-testing-patterns` - API and webhook testing strategies
+- `n8n-expression-testing` - Safe expression validation
+- `n8n-trigger-testing-strategies` - Trigger testing patterns
+
+**Key Design Decisions:**
+- Runtime execution is DEFAULT (not opt-in)
+- Safe expression evaluation using pattern matching (no unsafe eval)
+- Correct n8n webhook URL patterns (production + test mode)
+- Dual authentication support (API key + session cookie fallback)
+
+### Changed
+
+- Updated skill count from 41 to 46 (added 5 n8n skills)
+- Updated agent documentation with n8n workflow testing section
+- Updated `aqe init` to copy n8n agent definitions to user projects
+- Added Smithery badge to README (PR #152 by @gurdasnijor)
+
 ## [2.5.6] - 2025-12-16
 
 ### Changed

package/README.md

@@ -1,5 +1,6 @@
 # Agentic Quality Engineering Fleet
 
+
 <div align="center">
 
 [![npm version](https://img.shields.io/npm/v/agentic-qe.svg)](https://www.npmjs.com/package/agentic-qe)
@@ -7,13 +8,14 @@
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
 [![Node.js](https://img.shields.io/badge/Node.js-20+-green.svg)](https://nodejs.org/)
 <img alt="NPM Downloads" src="https://img.shields.io/npm/dw/agentic-qe">
+[![Run in Smithery](https://smithery.ai/badge/skills/proffesor-for-testing)](https://smithery.ai/skills?ns=proffesor-for-testing&utm_source=github&utm_medium=badge)
 
 
-**Version 2.5.6** | [Changelog](CHANGELOG.md) | [Contributors](CONTRIBUTORS.md) | [Issues](https://github.com/proffesor-for-testing/agentic-qe/issues) | [Discussions](https://github.com/proffesor-for-testing/agentic-qe/discussions)
+**Version 2.5.8** | [Changelog](CHANGELOG.md) | [Contributors](CONTRIBUTORS.md) | [Issues](https://github.com/proffesor-for-testing/agentic-qe/issues) | [Discussions](https://github.com/proffesor-for-testing/agentic-qe/discussions)
 
 > AI-powered test automation that learns from every task, switches between 300+ AI models on-the-fly, scores code testability, visualizes agent activity in real-time, and improves autonomously overnight — with built-in safety guardrails and full observability.
 
-🎨 **Real-Time Visualization** | 📊 **Testability Scoring** | 🧠 **QE Agent Learning** | 🚀 **QUIC Transport** | 📋 **Constitution System** | 📚 **41 QE Skills** | 🎯 **Flaky Detection** | 💰 **Multi-Model Router**
+🎨 **Real-Time Visualization** | 📊 **Testability Scoring** | 🧠 **QE Agent Learning** | 🚀 **QUIC Transport** | 📋 **Constitution System** | 📚 **46 QE Skills** | 🎯 **Flaky Detection** | 💰 **Multi-Model Router** | 🔄 **n8n Workflow Testing**
 
 </div>
 
@@ -60,9 +62,10 @@ claude "Use qe-flaky-test-hunter to analyze the last 100 test runs and identify
 - ✅ Learning System (20% improvement target)
 - ✅ Pattern Bank (cross-project reuse)
 - ✅ ML Flaky Detection (90%+ accuracy with root cause analysis)
-- ✅ 19 Specialized agent definitions (including qe-code-complexity)
+- ✅ 19 Specialized QE agent definitions (including qe-code-complexity)
+- ✅ 15 n8n workflow testing agents (workflow execution, chaos, compliance, security, performance by [@fndlalit](https://github.com/fndlalit))
 - ✅ 11 TDD subagent definitions (RED/GREEN/REFACTOR phases + specialized)
-- ✅ 41 World-class QE skills library (accessibility, shift-left/right, verification, visual testing, XP practices, **testability-scoring** by [@fndlalit](https://github.com/fndlalit))
+- ✅ 46 World-class QE skills library (accessibility, shift-left/right, verification, visual testing, XP practices, n8n testing, **testability-scoring** by [@fndlalit](https://github.com/fndlalit))
 - ✅ 8 AQE slash commands
 - ✅ Modular init system with comprehensive project setup