agentic-qe 2.0.0 → 2.1.0

package/.claude/skills/shift-right-testing/SKILL.md

@@ -1,585 +1,227 @@
 ---
 name: shift-right-testing
-description: Testing in production with feature flags, canary deployments, synthetic monitoring, and chaos engineering. Use when validating real-world behavior, implementing safe deployments, or ensuring production resilience.
+description: "Testing in production with feature flags, canary deployments, synthetic monitoring, and chaos engineering. Use when implementing production observability or progressive delivery."
+category: testing-methodologies
+priority: high
+tokenEstimate: 1000
+agents: [qe-production-intelligence, qe-chaos-engineer, qe-performance-tester, qe-quality-analyzer]
+implementation_status: optimized
+optimization_version: 1.0
+last_optimized: 2025-12-02
+dependencies: []
+quick_reference_card: true
+tags: [shift-right, production-testing, canary, feature-flags, chaos-engineering, monitoring]
 ---
 
 # Shift-Right Testing
 
-## Core Principle
-
-**Production is different. Test where it matters most.**
-
-Shift-right testing moves testing activities into production environments to validate real-world behavior, user experience, and system resilience under actual conditions.
-
-## What is Shift-Right Testing?
-
-**Shift-Right:** Moving testing activities later (right on timeline) into production environments.
-
-**Why Test in Production?**
-
-Pre-production testing can't replicate:
-- Real user traffic patterns
-- Actual data volumes and variety
-- Production dependencies and integrations
-- Real network conditions and latency
-- Unpredictable load and edge cases
-- Geographic distribution
-- Third-party service behavior
-
-**Shift-Right solves this by:**
-- Validating deployments safely
-- Detecting regressions immediately
-- Monitoring real user experience
-- Testing with production data (safely)
-- Validating system resilience
-
-**Timeline:**
-```
-Requirements → Design → Code → Deploy → Monitor
-                                   ↓       ↓
-                                 Test    Test (production)
-```
+<default_to_action>
+When testing in production or implementing progressive delivery:
+1. IMPLEMENT feature flags for progressive rollout (1% → 10% → 50% → 100%)
+2. DEPLOY with canary releases (compare metrics before full rollout)
+3. MONITOR with synthetic tests (proactive) + RUM (reactive)
+4. INJECT failures with chaos engineering (build resilience)
+5. ANALYZE production data to improve pre-production testing
+
+**Quick Shift-Right Techniques:**
+- Feature flags → Control who sees what, instant rollback
+- Canary deployment → 5% traffic, compare error rates
+- Synthetic monitoring → Simulate users 24/7, catch issues before users
+- Chaos engineering → Netflix-style failure injection
+- RUM (Real User Monitoring) → Actual user experience data
+
+**Critical Success Factors:**
+- Production is the ultimate test environment
+- Ship fast with safety nets, not slow with certainty
+- Use production data to improve shift-left testing
+</default_to_action>
+
+## Quick Reference Card
+
+### When to Use
+- Progressive feature rollouts
+- Production reliability validation
+- Performance monitoring at scale
+- Learning from real user behavior
+
+### Shift-Right Techniques
+| Technique | Purpose | When |
+|-----------|---------|------|
+| Feature Flags | Controlled rollout | Every feature |
+| Canary | Compare new vs old | Every deployment |
+| Synthetic Monitoring | Proactive detection | 24/7 |
+| RUM | Real user metrics | Always on |
+| Chaos Engineering | Resilience validation | Regularly |
+| A/B Testing | User behavior validation | Feature decisions |
+
+### Progressive Rollout Pattern
+```
+1% → 10% → 25% → 50% → 100%
+↓      ↓      ↓      ↓
+Check  Check  Check  Monitor
+```
+
+### Key Metrics to Monitor
+| Metric | SLO Target | Alert Threshold |
+|--------|------------|-----------------|
+| Error rate | < 0.1% | > 1% |
+| p95 latency | < 200ms | > 500ms |
+| Availability | 99.9% | < 99.5% |
+| Apdex | > 0.95 | < 0.8 |
 
 ---
 
-## Shift-Right Techniques
-
-### 1. Feature Flags (Progressive Rollout)
-
-**Concept:** Deploy code to production but control who sees it.
+## Feature Flags
 
 ```javascript
-import { FeatureFlags } from './feature-flags';
-
-// New feature behind flag
-if (FeatureFlags.isEnabled('new-checkout-flow', user)) {
-  return <NewCheckout />;  // New code (for selected users)
-} else {
-  return <OldCheckout />;  // Existing code (fallback)
-}
-```
-
-**Rollout Strategy:**
-```
-1% → Monitor metrics for 1 hour
-     ↓ (if healthy)
-10% → A/B test performance vs old version
-     ↓ (if successful)
-50% → Validate at scale, monitor errors
-     ↓ (if stable)
-100% → Full rollout complete
-```
-
-**Benefits:**
-- Test in production safely
-- Instant rollback (disable flag)
-- A/B testing built-in
-- Gradual risk exposure
-- Dark launches (test without users seeing)
-
-**Implementation with LaunchDarkly:**
-```javascript
-import * as ld from 'launchdarkly-node-server-sdk';
-
-const client = ld.init(process.env.LD_SDK_KEY);
-
-// Check if feature enabled for user
-const showNewFeature = await client.variation(
-  'new-checkout-flow',
-  { key: user.id, email: user.email },
-  false  // default value
-);
+// Progressive rollout with LaunchDarkly/Unleash pattern
+const newCheckout = featureFlags.isEnabled('new-checkout', {
+  userId: user.id,
+  percentage: 10,  // 10% of users
+  allowlist: ['beta-testers']
+});
 
-if (showNewFeature) {
-  // New code path
+if (newCheckout) {
+  return <NewCheckoutFlow />;
 } else {
-  // Old code path
+  return <LegacyCheckoutFlow />;
 }
-```
 
-**Targeting Rules:**
-```yaml
-feature: new-checkout-flow
-variations:
-  - on: true
-  - off: false
-targeting:
-  - rule: Internal employees
-    serve: on
-    match: email ends with "@company.com"
-
-  - rule: Beta testers
-    serve: on
-    match: user in segment "beta-users"
-
-  - rule: Percentage rollout
-    serve: on
-    match: 10% of users (by user ID hash)
-
-  default: off
+// Instant rollback on issues
+await featureFlags.disable('new-checkout');
 ```
 
 ---
 
-### 2. Canary Deployments
-
-**Concept:** Deploy new version to small percentage of infrastructure, monitor, then gradually increase.
-
-**Manual Canary with Kubernetes:**
-```bash
-# Deploy new version to 5% of pods
-kubectl set image deployment/api api=v2.0 --record
+## Canary Deployment
 
-# Monitor for 10 minutes
-./monitor-metrics.sh --deployment=api --duration=10m \
-  --metrics="error_rate,latency_p95,cpu_usage"
-
-# If healthy, scale up gradually
-kubectl scale deployment/api-v2 --replicas=20   # 10%
-./monitor-metrics.sh --duration=10m
-
-kubectl scale deployment/api-v2 --replicas=100  # 50%
-./monitor-metrics.sh --duration=10m
-
-kubectl scale deployment/api-v2 --replicas=200  # 100%
-kubectl scale deployment/api-v1 --replicas=0    # Remove old
-```
-
-**Automated Canary with Flagger:**
 ```yaml
+# Flagger canary config
 apiVersion: flagger.app/v1beta1
 kind: Canary
-metadata:
-  name: api-canary
 spec:
   targetRef:
     apiVersion: apps/v1
     kind: Deployment
-    name: api
-
-  # Canary analysis configuration
+    name: checkout-service
+  progressDeadlineSeconds: 60
   analysis:
-    interval: 1m           # Check every minute
-    threshold: 10          # Fail after 10 failed checks
-    maxWeight: 50          # Max 50% canary traffic
-    stepWeight: 10         # Increase by 10% each step
-
-    # Success metrics (must pass)
+    interval: 1m
+    threshold: 5      # Max failed checks
+    maxWeight: 50     # Max traffic to canary
+    stepWeight: 10    # Increment per interval
     metrics:
       - name: request-success-rate
-        thresholdRange:
-          min: 99          # 99%+ success rate required
-
-      - name: request-duration-p95
-        thresholdRange:
-          max: 500         # p95 latency < 500ms
-
-      - name: error-rate
-        thresholdRange:
-          max: 1           # < 1% errors
-
-  # Webhook notifications
-  webhooks:
-    - name: slack-notification
-      url: https://hooks.slack.com/services/YOUR/WEBHOOK
-      type: post-rollout
+        threshold: 99
+      - name: request-duration
+        threshold: 500
 ```
 
-**Automated Process:**
-1. Deploy v2 to 10% of traffic
-2. Monitor success rate, latency, errors
-3. If metrics healthy → increase to 20%
-4. Continue until 100% or failure detected
-5. On failure → automatic rollback to v1
-
-**Benefits:**
-- Real production validation
-- Gradual risk mitigation
-- Automatic rollback on failures
-- Minimal blast radius (5-10% impact)
-
 ---
 
-### 3. Synthetic Monitoring (Active Testing)
-
-**Concept:** Continuously run automated tests against production to detect issues before users do.
+## Synthetic Monitoring
 
-**Playwright Synthetic Monitor:**
 ```javascript
-// synthetic-monitor.js
-import { chromium } from 'playwright';
-
-async function runCheckoutFlowMonitor() {
-  const browser = await chromium.launch();
-  const page = await browser.newPage();
-  const start = Date.now();
-
-  try {
-    // Critical user journey: Add to cart → Checkout
-    await page.goto('https://example.com');
-    await page.click('[data-test=add-to-cart]');
-    await page.click('[data-test=checkout]');
-    await page.fill('[data-test=email]', 'synthetic@monitor.test');
-    await page.fill('[data-test=card]', '4242424242424242'); // Test mode
-
-    // Don't actually complete purchase (test mode stops here)
-
-    const duration = Date.now() - start;
-
-    // Report success metric
-    await reportMetric('checkout-flow', {
-      success: true,
-      duration,
-      timestamp: new Date()
-    });
-
-    console.log(`✅ Checkout flow healthy (${duration}ms)`);
-
-  } catch (error) {
-    // Alert on failure
-    await reportMetric('checkout-flow', {
-      success: false,
-      error: error.message,
-      timestamp: new Date()
-    });
-
-    await alertOncall({
-      severity: 'critical',
-      message: 'Checkout flow failed in production',
-      error: error.message
-    });
-
-    console.error(`❌ Checkout flow failed: ${error.message}`);
-  } finally {
-    await browser.close();
+// Continuous production validation
+await Task("Synthetic Tests", {
+  endpoints: [
+    { path: '/health', expected: 200, interval: '30s' },
+    { path: '/api/products', expected: 200, interval: '1m' },
+    { path: '/checkout', flow: 'full-purchase', interval: '5m' }
+  ],
+  locations: ['us-east', 'eu-west', 'ap-south'],
+  alertOn: {
+    statusCode: '!= 200',
+    latency: '> 500ms',
+    contentMismatch: true
   }
-}
-
-// Run every 5 minutes
-setInterval(runCheckoutFlowMonitor, 5 * 60 * 1000);
+}, "qe-production-intelligence");
 ```
 
-**Datadog Synthetic Monitoring:**
-```yaml
-# synthetics.yaml
-tests:
-  - name: "API Health Check"
-    type: api
-    request:
-      url: "https://api.example.com/health"
-      method: GET
-    assertions:
-      - type: statusCode
-        operator: is
-        target: 200
-      - type: responseTime
-        operator: lessThan
-        target: 500
-    locations: ["us-east-1", "eu-west-1", "ap-southeast-1"]
-    frequency: 300  # 5 minutes
-
-  - name: "Checkout Flow E2E"
-    type: browser
-    steps:
-      - type: navigateTo
-        url: "https://example.com"
-      - type: click
-        selector: "[data-test=add-to-cart]"
-      - type: click
-        selector: "[data-test=checkout]"
-    assertions:
-      - type: element
-        selector: "[data-test=checkout-success]"
-        operator: isVisible
-    frequency: 600  # 10 minutes
-```
-
-**Benefits:**
-- Proactive issue detection
-- User experience validation
-- SLA monitoring
-- Geographic validation (test from multiple regions)
-
 ---
 
-### 4. Chaos Engineering (Resilience Testing)
-
-**Concept:** Intentionally introduce failures in production to validate system resilience.
-
-**Principles (Netflix Chaos Monkey):**
-1. Define steady state (normal system behavior)
-2. Hypothesize steady state continues during chaos
-3. Introduce real-world failures
-4. Try to disprove hypothesis
-5. Minimize blast radius
-
-**Example: Instance Failure Test**
-```javascript
-import { ChaosMonkey } from './chaos';
-
-async function testInstanceResilience() {
-  // 1. Baseline: Record normal behavior
-  const baseline = await collectMetrics('api', '5m');
-  console.log(`Baseline: ${baseline.successRate}% success, ${baseline.latencyP95}ms p95`);
-
-  // 2. Hypothesis: System handles 1 instance failure gracefully
-  console.log('Hypothesis: Killing 1 instance won\'t impact users');
-
-  // 3. Introduce chaos (kill random instance)
-  await ChaosMonkey.killRandomInstance({
-    service: 'api',
-    count: 1,  // Kill 1 instance
-    duration: '5m'
-  });
-
-  // 4. Measure impact
-  const chaosMetrics = await collectMetrics('api', '5m');
-  console.log(`During chaos: ${chaosMetrics.successRate}% success, ${chaosMetrics.latencyP95}ms p95`);
-
-  // 5. Verify hypothesis
-  const successRateDrop = baseline.successRate - chaosMetrics.successRate;
-  const latencyIncrease = chaosMetrics.latencyP95 - baseline.latencyP95;
-
-  if (successRateDrop < 0.1 && latencyIncrease < 50) {
-    console.log('✅ System is resilient to instance failures');
-  } else {
-    console.log('❌ System not resilient. Add redundancy!');
-  }
-}
-
-// Run weekly during low traffic
-schedule.weekly('Sunday 3am', testInstanceResilience);
-```
-
-**Common Chaos Experiments:**
-
-**a) Instance Failures**
-```javascript
-// Kill random instances (10% of fleet)
-await ChaosMonkey.killRandomInstance({
-  service: 'api',
-  percentage: 10,
-  duration: '10m'
-});
-```
+## Chaos Engineering
 
-**b) Network Latency**
-```javascript
-// Inject 500ms latency to database calls
-await ChaosMonkey.injectLatency({
-  service: 'database',
-  latency: '500ms',
-  percentage: 20  // 20% of requests
-});
-```
-
-**c) Dependency Failures**
-```javascript
-// Simulate payment gateway outage
-await ChaosMonkey.blockService({
-  service: 'payment-gateway',
-  duration: '5m'
-});
-
-// Verify: Graceful degradation? Retry logic working?
-```
-
-**d) Resource Exhaustion**
-```javascript
-// Stress test: High CPU load
-await ChaosMonkey.stressCPU({
-  service: 'api',
-  percentage: 80,  // 80% CPU usage
-  duration: '10m'
-});
-```
-
-**Chaos Testing Tools:**
-- Chaos Monkey (Netflix) - Random instance termination
-- Chaos Toolkit - Programmable chaos experiments
-- Gremlin - Chaos engineering platform
-- Litmus Chaos - Kubernetes chaos engineering
-
----
-
-### 5. A/B Testing (Hypothesis Validation)
-
-**Concept:** Test two versions in production to determine which performs better.
-
-```javascript
-import { ABTest } from './ab-testing';
-
-// Define A/B test
-const checkoutTest = ABTest.create({
-  name: 'checkout-redesign',
-  hypothesis: 'New checkout flow increases conversion by 10%',
-
-  variants: {
-    control: {
-      weight: 50,  // 50% of traffic
-      implementation: () => <OldCheckout />
-    },
-    treatment: {
-      weight: 50,  // 50% of traffic
-      implementation: () => <NewCheckout />
-    }
+```typescript
+// Controlled failure injection
+await Task("Chaos Experiment", {
+  hypothesis: 'System handles database latency gracefully',
+  steadyState: {
+    metric: 'error_rate',
+    expected: '< 0.1%'
   },
-
-  metrics: {
-    primary: 'conversion_rate',      // Primary success metric
-    secondary: ['cart_abandonment', 'time_to_purchase']
+  experiment: {
+    type: 'network-latency',
+    target: 'database',
+    delay: '500ms',
+    duration: '5m'
   },
-
-  sample_size: 10000,  // Users needed for statistical significance
-  confidence: 0.95      // 95% confidence level
-});
-
-// Render based on variant
-function CheckoutPage({ user }) {
-  const variant = checkoutTest.getVariant(user.id);
-  const Checkout = variant.implementation;
-
-  // Track metrics
-  useEffect(() => {
-    checkoutTest.trackImpression(user.id, variant.name);
-  }, []);
-
-  return <Checkout onComplete={() => {
-    checkoutTest.trackConversion(user.id, variant.name);
-  }} />;
-}
-
-// Analyze results after sufficient data
-async function analyzeTest() {
-  const results = await checkoutTest.analyze();
-
-  console.log(`Control conversion: ${results.control.conversionRate}%`);
-  console.log(`Treatment conversion: ${results.treatment.conversionRate}%`);
-  console.log(`Lift: ${results.lift}%`);
-  console.log(`P-value: ${results.pValue}`);
-  console.log(`Statistical significance: ${results.significant ? 'YES' : 'NO'}`);
-
-  if (results.significant && results.lift > 0) {
-    console.log('✅ Treatment wins! Rolling out to 100%');
-    await rolloutToProduction('treatment');
-  } else {
-    console.log('❌ No significant improvement. Keeping control.');
+  rollback: {
+    automatic: true,
+    trigger: 'error_rate > 5%'
   }
-}
+}, "qe-chaos-engineer");
 ```
 
 ---
 
-## Shift-Right Best Practices
-
-### 1. Minimize Blast Radius
-
-**Always limit exposure:**
-- Feature flags: 1% → 10% → 50% → 100%
-- Canary: 5% → 10% → 25% → 50% → 100%
-- Geographic: 1 region → 2 regions → All regions
-
-### 2. Automate Rollback
+## Production → Pre-Production Feedback Loop
 
-**Never rely on manual rollback:**
-```javascript
-// Automatic rollback on error rate spike
-if (errorRate > 1% || latencyP95 > 500) {
-  await rollback();
-  await alert('Automatic rollback triggered');
-}
-```
-
-### 3. Monitor Everything
-
-**Key Metrics:**
-- Success/error rates
-- Latency (p50, p95, p99)
-- CPU/memory usage
-- User-facing metrics (conversion, engagement)
-
-### 4. Test During Low Traffic
-
-**Chaos engineering schedule:**
-- Weekday mornings: Low traffic
-- Sunday 3am: Minimal users
-- Avoid holidays, sales events
-
-### 5. Have a Kill Switch
+```typescript
+// Convert production incidents to regression tests
+await Task("Incident Replay", {
+  incident: {
+    id: 'INC-2024-001',
+    type: 'performance-degradation',
+    conditions: { concurrent_users: 500, cart_items: 10 }
+  },
+  generateTests: true,
+  addToRegression: true
+}, "qe-production-intelligence");
 
-**Emergency stop for everything:**
-```javascript
-// Global kill switch (stops all experiments)
-if (FeatureFlags.isEnabled('global-kill-switch')) {
-  return <SafeMode />;  // Fallback to known-good state
-}
+// Output: New test added to prevent recurrence
 ```
 
 ---
 
-## Shift-Right Metrics
-
-**1. Mean Time to Detect (MTTD)**
-```
-Time from issue occurrence to detection
-
-Target: < 5 minutes (synthetic monitoring)
-```
-
-**2. Mean Time to Recover (MTTR)**
-```
-Time from detection to resolution
-
-Target: < 15 minutes (with automatic rollback)
-```
-
-**3. Blast Radius**
-```
-Percentage of users impacted by failure
-
-Target: < 10% (canary deployment)
-```
-
-**4. False Positive Rate**
-```
-Alerts that weren't real issues
-
-Target: < 5%
+## Agent Coordination Hints
+
+### Memory Namespace
+```
+aqe/shift-right/
+├── canary-results/*      - Canary deployment metrics
+├── synthetic-tests/*     - Monitoring configurations
+├── chaos-experiments/*   - Experiment results
+├── production-insights/* - Issues → test conversions
+└── rum-analysis/*        - Real user data patterns
+```
+
+### Fleet Coordination
+```typescript
+const shiftRightFleet = await FleetManager.coordinate({
+  strategy: 'shift-right-testing',
+  agents: [
+    'qe-production-intelligence',  // RUM, incident replay
+    'qe-chaos-engineer',           // Resilience testing
+    'qe-performance-tester',       // Synthetic monitoring
+    'qe-quality-analyzer'          // Metrics analysis
+  ],
+  topology: 'mesh'
+});
 ```
 
 ---
 
 ## Related Skills
-
-**Testing Methodologies:**
-- [shift-left-testing](../shift-left-testing/) - Testing BEFORE production (complement)
-- [chaos-engineering-resilience](../chaos-engineering-resilience/) - Detailed chaos testing
-- [regression-testing](../regression-testing/)
-
-**Infrastructure:**
-- [test-environment-management](../test-environment-management/)
-- [performance-testing](../performance-testing/)
-
-**Monitoring:**
-- [test-reporting-analytics](../test-reporting-analytics/)
-- [production-intelligence](../production-intelligence/) (agent)
+- [shift-left-testing](../shift-left-testing/) - Pre-production testing
+- [chaos-engineering-resilience](../chaos-engineering-resilience/) - Failure injection deep dive
+- [performance-testing](../performance-testing/) - Load testing
+- [agentic-quality-engineering](../agentic-quality-engineering/) - Agent coordination
 
 ---
 
 ## Remember
 
-**Production is the ultimate test environment.**
-
-**Shift-Right complements Shift-Left:**
-- **Shift-Left**: Catch bugs early (cheap)
-- **Shift-Right**: Validate real-world behavior (accurate)
-
-**Best Practices:**
-1. Use feature flags for safe deployments
-2. Canary deploy with automatic rollback
-3. Synthetic monitoring for proactive detection
-4. Chaos engineering for resilience
-5. Always minimize blast radius
-6. Monitor everything, alert intelligently
+**Production is the ultimate test environment.** Feature flags enable instant rollback. Canary catches issues before 100% rollout. Synthetic monitoring detects problems before users. Chaos engineering builds resilience. RUM shows real user experience.
 
-**With Agents:** `qe-production-intelligence` monitors production metrics and converts real usage patterns into tests. `qe-chaos-engineer` orchestrates safe chaos experiments with automatic rollback. Together, they enable comprehensive shift-right testing with minimal risk.
+**With Agents:** Agents monitor production, replay incidents as tests, run chaos experiments, and convert production insights to pre-production tests. Use agents to maintain continuous production quality.