@agentic-qe/v3 3.0.0-alpha.6 → 3.0.0-alpha.8

package/assets/skills/performance-testing/SKILL.md

@@ -0,0 +1,310 @@
+---
+name: performance-testing
+description: "Test application performance, scalability, and resilience. Use when planning load testing, stress testing, or optimizing system performance."
+category: specialized-testing
+priority: high
+tokenEstimate: 1100
+agents: [qe-performance-tester, qe-quality-analyzer, qe-production-intelligence]
+implementation_status: optimized
+optimization_version: 1.0
+last_optimized: 2025-12-02
+dependencies: []
+quick_reference_card: true
+tags: [performance, load-testing, stress-testing, scalability, k6, bottlenecks]
+---
+
+# Performance Testing
+
+<default_to_action>
+When testing performance or planning load tests:
+1. DEFINE SLOs: p95 response time, throughput, error rate targets
+2. IDENTIFY critical paths: revenue flows, high-traffic pages, key APIs
+3. CREATE realistic scenarios: user journeys, think time, varied data
+4. EXECUTE with monitoring: CPU, memory, DB queries, network
+5. ANALYZE bottlenecks and fix before production
+
+**Quick Test Type Selection:**
+- Expected load validation → Load testing
+- Find breaking point → Stress testing
+- Sudden traffic spike → Spike testing
+- Memory leaks, resource exhaustion → Endurance/soak testing
+- Horizontal/vertical scaling → Scalability testing
+
+**Critical Success Factors:**
+- Performance is a feature, not an afterthought
+- Test early and often, not just before release
+- Focus on user-impacting bottlenecks
+</default_to_action>
+
+## Quick Reference Card
+
+### When to Use
+- Before major releases
+- After infrastructure changes
+- Before scaling events (Black Friday)
+- When setting SLAs/SLOs
+
+### Test Types
+| Type | Purpose | When |
+|------|---------|------|
+| **Load** | Expected traffic | Every release |
+| **Stress** | Beyond capacity | Quarterly |
+| **Spike** | Sudden surge | Before events |
+| **Endurance** | Memory leaks | After code changes |
+| **Scalability** | Scaling validation | Infrastructure changes |
+
+### Key Metrics
+| Metric | Target | Why |
+|--------|--------|-----|
+| p95 response | < 200ms | User experience |
+| Throughput | 10k req/min | Capacity |
+| Error rate | < 0.1% | Reliability |
+| CPU | < 70% | Headroom |
+| Memory | < 80% | Stability |
+
+### Tools
+- **k6**: Modern, JS-based, CI/CD friendly
+- **JMeter**: Enterprise, feature-rich
+- **Artillery**: Simple YAML configs
+- **Gatling**: Scala, great reporting
+
+### Agent Coordination
+- `qe-performance-tester`: Load test orchestration
+- `qe-quality-analyzer`: Results analysis
+- `qe-production-intelligence`: Production comparison
+
+---
+
+## Defining SLOs
+
+**Bad:** "The system should be fast"
+**Good:** "p95 response time < 200ms under 1,000 concurrent users"
+
+```javascript
+export const options = {
+  thresholds: {
+    http_req_duration: ['p(95)<200'],  // 95% < 200ms
+    http_req_failed: ['rate<0.01'],     // < 1% failures
+  },
+};
+```
+
+---
+
+## Realistic Scenarios
+
+**Bad:** Every user hits homepage repeatedly
+**Good:** Model actual user behavior
+
+```javascript
+// Realistic distribution
+// 40% browse, 30% search, 20% details, 10% checkout
+export default function () {
+  const action = Math.random();
+  if (action < 0.4) browse();
+  else if (action < 0.7) search();
+  else if (action < 0.9) viewProduct();
+  else checkout();
+
+  sleep(randomInt(1, 5)); // Think time
+}
+```
+
+---
+
+## Common Bottlenecks
+
+### Database
+**Symptoms:** Slow queries under load, connection pool exhaustion
+**Fixes:** Add indexes, optimize N+1 queries, increase pool size, read replicas
+
+### N+1 Queries
+```javascript
+// BAD: 100 orders = 101 queries
+const orders = await Order.findAll();
+for (const order of orders) {
+  const customer = await Customer.findById(order.customerId);
+}
+
+// GOOD: 1 query
+const orders = await Order.findAll({ include: [Customer] });
+```
+
+### Synchronous Processing
+**Problem:** Blocking operations in request path (sending email during checkout)
+**Fix:** Use message queues, process async, return immediately
+
+### Memory Leaks
+**Detection:** Endurance testing, memory profiling
+**Common causes:** Event listeners not cleaned, caches without eviction
+
+### External Dependencies
+**Solutions:** Aggressive timeouts, circuit breakers, caching, graceful degradation
+
+---
+
+## k6 CI/CD Example
+
+```javascript
+// performance-test.js
+import http from 'k6/http';
+import { check, sleep } from 'k6';
+
+export const options = {
+  stages: [
+    { duration: '1m', target: 50 },   // Ramp up
+    { duration: '3m', target: 50 },   // Steady
+    { duration: '1m', target: 0 },    // Ramp down
+  ],
+  thresholds: {
+    http_req_duration: ['p(95)<200'],
+    http_req_failed: ['rate<0.01'],
+  },
+};
+
+export default function () {
+  const res = http.get('https://api.example.com/products');
+  check(res, {
+    'status is 200': (r) => r.status === 200,
+    'response time < 200ms': (r) => r.timings.duration < 200,
+  });
+  sleep(1);
+}
+```
+
+```yaml
+# GitHub Actions
+- name: Run k6 test
+  uses: grafana/k6-action@v0.3.0
+  with:
+    filename: performance-test.js
+```
+
+---
+
+## Analyzing Results
+
+### Good Results
+```
+Load: 1,000 users | p95: 180ms | Throughput: 5,000 req/s
+Error rate: 0.05% | CPU: 65% | Memory: 70%
+```
+
+### Problems
+```
+Load: 1,000 users | p95: 3,500ms ❌ | Throughput: 500 req/s ❌
+Error rate: 5% ❌ | CPU: 95% ❌ | Memory: 90% ❌
+```
+
+### Root Cause Analysis
+1. Correlate metrics: When response time spikes, what changes?
+2. Check logs: Errors, warnings, slow queries
+3. Profile code: Where is time spent?
+4. Monitor resources: CPU, memory, disk
+5. Trace requests: End-to-end flow
+
+---
+
+## Anti-Patterns
+
+| ❌ Anti-Pattern | ✅ Better |
+|----------------|-----------|
+| Testing too late | Test early and often |
+| Unrealistic scenarios | Model real user behavior |
+| 0 to 1000 users instantly | Ramp up gradually |
+| No monitoring during tests | Monitor everything |
+| No baseline | Establish and track trends |
+| One-time testing | Continuous performance testing |
+
+---
+
+## Agent-Assisted Performance Testing
+
+```typescript
+// Comprehensive load test
+await Task("Load Test", {
+  target: 'https://api.example.com',
+  scenarios: {
+    checkout: { vus: 100, duration: '5m' },
+    search: { vus: 200, duration: '5m' },
+    browse: { vus: 500, duration: '5m' }
+  },
+  thresholds: {
+    'http_req_duration': ['p(95)<200'],
+    'http_req_failed': ['rate<0.01']
+  }
+}, "qe-performance-tester");
+
+// Bottleneck analysis
+await Task("Analyze Bottlenecks", {
+  testResults: perfTest,
+  metrics: ['cpu', 'memory', 'db_queries', 'network']
+}, "qe-performance-tester");
+
+// CI integration
+await Task("CI Performance Gate", {
+  mode: 'smoke',
+  duration: '1m',
+  vus: 10,
+  failOn: { 'p95_response_time': 300, 'error_rate': 0.01 }
+}, "qe-performance-tester");
+```
+
+---
+
+## Agent Coordination Hints
+
+### Memory Namespace
+```
+aqe/performance/
+├── results/*       - Test execution results
+├── baselines/*     - Performance baselines
+├── bottlenecks/*   - Identified bottlenecks
+└── trends/*        - Historical trends
+```
+
+### Fleet Coordination
+```typescript
+const perfFleet = await FleetManager.coordinate({
+  strategy: 'performance-testing',
+  agents: [
+    'qe-performance-tester',
+    'qe-quality-analyzer',
+    'qe-production-intelligence',
+    'qe-deployment-readiness'
+  ],
+  topology: 'sequential'
+});
+```
+
+---
+
+## Pre-Production Checklist
+
+- [ ] Load test passed (expected traffic)
+- [ ] Stress test passed (2-3x expected)
+- [ ] Spike test passed (sudden surge)
+- [ ] Endurance test passed (24+ hours)
+- [ ] Database indexes in place
+- [ ] Caching configured
+- [ ] Monitoring and alerting set up
+- [ ] Performance baseline established
+
+---
+
+## Related Skills
+- [agentic-quality-engineering](../agentic-quality-engineering/) - Agent coordination
+- [api-testing-patterns](../api-testing-patterns/) - API performance
+- [chaos-engineering-resilience](../chaos-engineering-resilience/) - Resilience testing
+
+---
+
+## Remember
+
+**Performance is a feature:** Test it like functionality
+**Test continuously:** Not just before launch
+**Monitor production:** Synthetic + real user monitoring
+**Fix what matters:** Focus on user-impacting bottlenecks
+**Trend over time:** Catch degradation early
+
+**With Agents:** Agents automate load testing, analyze bottlenecks, and compare with production. Use agents to maintain performance at scale.

package/assets/skills/quality-metrics/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: quality-metrics
+description: "Measure quality effectively with actionable metrics. Use when establishing quality dashboards, defining KPIs, or evaluating test effectiveness."
+category: testing-methodologies
+priority: high
+tokenEstimate: 900
+agents: [qe-quality-analyzer, qe-test-executor, qe-coverage-analyzer, qe-production-intelligence, qe-quality-gate]
+implementation_status: optimized
+optimization_version: 1.0
+last_optimized: 2025-12-02
+dependencies: []
+quick_reference_card: true
+tags: [metrics, dora, quality-gates, dashboards, kpis, measurement]
+---
+
+# Quality Metrics
+
+<default_to_action>
+When measuring quality or building dashboards:
+1. MEASURE outcomes (bug escape rate, MTTD) not activities (test count)
+2. FOCUS on DORA metrics: Deployment frequency, Lead time, MTTD, MTTR, Change failure rate
+3. AVOID vanity metrics: 100% coverage means nothing if tests don't catch bugs
+4. SET thresholds that drive behavior (quality gates block bad code)
+5. TREND over time: Direction matters more than absolute numbers
+
+**Quick Metric Selection:**
+- Speed: Deployment frequency, lead time for changes
+- Stability: Change failure rate, MTTR
+- Quality: Bug escape rate, defect density, test effectiveness
+- Process: Code review time, flaky test rate
+
+**Critical Success Factors:**
+- Metrics without action are theater
+- What you measure is what you optimize
+- Trends matter more than snapshots
+</default_to_action>
+
+## Quick Reference Card
+
+### When to Use
+- Building quality dashboards
+- Defining quality gates
+- Evaluating testing effectiveness
+- Justifying quality investments
+
+### Meaningful vs Vanity Metrics
+| ✅ Meaningful | ❌ Vanity |
+|--------------|-----------|
+| Bug escape rate | Test case count |
+| MTTD (detection) | Lines of test code |
+| MTTR (recovery) | Test executions |
+| Change failure rate | Coverage % (alone) |
+| Lead time for changes | Requirements traced |
+
+### DORA Metrics
+| Metric | Elite | High | Medium | Low |
+|--------|-------|------|--------|-----|
+| Deploy Frequency | On-demand | Weekly | Monthly | Yearly |
+| Lead Time | < 1 hour | < 1 week | < 1 month | > 6 months |
+| Change Failure Rate | < 5% | < 15% | < 30% | > 45% |
+| MTTR | < 1 hour | < 1 day | < 1 week | > 1 month |
+
+### Quality Gate Thresholds
+| Metric | Blocking Threshold | Warning |
+|--------|-------------------|---------|
+| Test pass rate | 100% | - |
+| Critical coverage | > 80% | > 70% |
+| Security critical | 0 | - |
+| Performance p95 | < 200ms | < 500ms |
+| Flaky tests | < 2% | < 5% |
+
+---
+
+## Core Metrics
+
+### Bug Escape Rate
+```
+Bug Escape Rate = (Production Bugs / Total Bugs Found) × 100
+
+Target: < 10% (90% caught before production)
+```
+
+### Test Effectiveness
+```
+Test Effectiveness = (Bugs Found by Tests / Total Bugs) × 100
+
+Target: > 70%
+```
+
+### Defect Density
+```
+Defect Density = Defects / KLOC
+
+Good: < 1 defect per KLOC
+```
+
+### Mean Time to Detect (MTTD)
+```
+MTTD = Time(Bug Reported) - Time(Bug Introduced)
+
+Target: < 1 day for critical, < 1 week for others
+```
+
+---
+
+## Dashboard Design
+
+```typescript
+// Agent generates quality dashboard
+await Task("Generate Dashboard", {
+  metrics: {
+    delivery: ['deployment-frequency', 'lead-time', 'change-failure-rate'],
+    quality: ['bug-escape-rate', 'test-effectiveness', 'defect-density'],
+    stability: ['mttd', 'mttr', 'availability'],
+    process: ['code-review-time', 'flaky-test-rate', 'coverage-trend']
+  },
+  visualization: 'grafana',
+  alerts: {
+    critical: { bug_escape_rate: '>20%', mttr: '>24h' },
+    warning: { coverage: '<70%', flaky_rate: '>5%' }
+  }
+}, "qe-quality-analyzer");
+```
+
+---
+
+## Quality Gate Configuration
+
+```json
+{
+  "qualityGates": {
+    "commit": {
+      "coverage": { "min": 80, "blocking": true },
+      "lint": { "errors": 0, "blocking": true }
+    },
+    "pr": {
+      "tests": { "pass": "100%", "blocking": true },
+      "security": { "critical": 0, "blocking": true },
+      "coverage_delta": { "min": 0, "blocking": false }
+    },
+    "release": {
+      "e2e": { "pass": "100%", "blocking": true },
+      "performance_p95": { "max_ms": 200, "blocking": true },
+      "bug_escape_rate": { "max": "10%", "blocking": false }
+    }
+  }
+}
+```
+
+---
+
+## Agent-Assisted Metrics
+
+```typescript
+// Calculate quality trends
+await Task("Quality Trend Analysis", {
+  timeframe: '90d',
+  metrics: ['bug-escape-rate', 'mttd', 'test-effectiveness'],
+  compare: 'previous-90d',
+  predictNext: '30d'
+}, "qe-quality-analyzer");
+
+// Evaluate quality gate
+await Task("Quality Gate Evaluation", {
+  buildId: 'build-123',
+  environment: 'staging',
+  metrics: currentMetrics,
+  policy: qualityPolicy
+}, "qe-quality-gate");
+```
+
+---
+
+## Agent Coordination Hints
+
+### Memory Namespace
+```
+aqe/quality-metrics/
+├── dashboards/*         - Dashboard configurations
+├── trends/*             - Historical metric data
+├── gates/*              - Gate evaluation results
+└── alerts/*             - Triggered alerts
+```
+
+### Fleet Coordination
+```typescript
+const metricsFleet = await FleetManager.coordinate({
+  strategy: 'quality-metrics',
+  agents: [
+    'qe-quality-analyzer',         // Trend analysis
+    'qe-test-executor',            // Test metrics
+    'qe-coverage-analyzer',        // Coverage data
+    'qe-production-intelligence',  // Production metrics
+    'qe-quality-gate'              // Gate decisions
+  ],
+  topology: 'mesh'
+});
+```
+
+---
+
+## Common Traps
+
+| Trap | Problem | Solution |
+|------|---------|----------|
+| Coverage worship | 100% coverage, bugs still escape | Measure bug escape rate instead |
+| Test count focus | Many tests, slow feedback | Measure execution time |
+| Activity metrics | Busy work, no outcomes | Measure outcomes (MTTD, MTTR) |
+| Point-in-time | Snapshot without context | Track trends over time |
+
+---
+
+## Related Skills
+- [agentic-quality-engineering](../agentic-quality-engineering/) - Agent coordination
+- [cicd-pipeline-qe-orchestrator](../cicd-pipeline-qe-orchestrator/) - Quality gates
+- [risk-based-testing](../risk-based-testing/) - Risk-informed metrics
+- [shift-right-testing](../shift-right-testing/) - Production metrics
+
+---
+
+## Remember
+
+**Measure outcomes, not activities.** Bug escape rate > test count. MTTD/MTTR > coverage %. Trends > snapshots. Set gates that block bad code. What you measure is what you optimize.
+
+**With Agents:** Agents track metrics automatically, analyze trends, trigger alerts, and make gate decisions. Use agents to maintain continuous quality visibility.