agentic-qe 2.0.0 → 2.1.1

package/.claude/skills/quality-metrics/SKILL.md

@@ -1,580 +1,225 @@
 ---
 name: quality-metrics
-description: Measure quality effectively with actionable metrics. Use when establishing quality dashboards, defining KPIs, or evaluating test effectiveness.
+description: "Measure quality effectively with actionable metrics. Use when establishing quality dashboards, defining KPIs, or evaluating test effectiveness."
+category: testing-methodologies
+priority: high
+tokenEstimate: 900
+agents: [qe-quality-analyzer, qe-test-executor, qe-coverage-analyzer, qe-production-intelligence, qe-quality-gate]
+implementation_status: optimized
+optimization_version: 1.0
+last_optimized: 2025-12-02
+dependencies: []
+quick_reference_card: true
+tags: [metrics, dora, quality-gates, dashboards, kpis, measurement]
 ---
 
 # Quality Metrics
 
-## Core Principle
+<default_to_action>
+When measuring quality or building dashboards:
+1. MEASURE outcomes (bug escape rate, MTTD) not activities (test count)
+2. FOCUS on DORA metrics: Deployment frequency, Lead time, MTTD, MTTR, Change failure rate
+3. AVOID vanity metrics: 100% coverage means nothing if tests don't catch bugs
+4. SET thresholds that drive behavior (quality gates block bad code)
+5. TREND over time: Direction matters more than absolute numbers
+
+**Quick Metric Selection:**
+- Speed: Deployment frequency, lead time for changes
+- Stability: Change failure rate, MTTR
+- Quality: Bug escape rate, defect density, test effectiveness
+- Process: Code review time, flaky test rate
+
+**Critical Success Factors:**
+- Metrics without action are theater
+- What you measure is what you optimize
+- Trends matter more than snapshots
+</default_to_action>
+
+## Quick Reference Card
+
+### When to Use
+- Building quality dashboards
+- Defining quality gates
+- Evaluating testing effectiveness
+- Justifying quality investments
+
+### Meaningful vs Vanity Metrics
+| ✅ Meaningful | ❌ Vanity |
+|--------------|-----------|
+| Bug escape rate | Test case count |
+| MTTD (detection) | Lines of test code |
+| MTTR (recovery) | Test executions |
+| Change failure rate | Coverage % (alone) |
+| Lead time for changes | Requirements traced |
+
+### DORA Metrics
+| Metric | Elite | High | Medium | Low |
+|--------|-------|------|--------|-----|
+| Deploy Frequency | On-demand | Weekly | Monthly | Yearly |
+| Lead Time | < 1 hour | < 1 week | < 1 month | > 6 months |
+| Change Failure Rate | < 5% | < 15% | < 30% | > 45% |
+| MTTR | < 1 hour | < 1 day | < 1 week | > 1 month |
+
+### Quality Gate Thresholds
+| Metric | Blocking Threshold | Warning |
+|--------|-------------------|---------|
+| Test pass rate | 100% | - |
+| Critical coverage | > 80% | > 70% |
+| Security critical | 0 | - |
+| Performance p95 | < 200ms | < 500ms |
+| Flaky tests | < 2% | < 5% |
 
-**Measure what matters, not what's easy to measure.**
-
-Metrics should drive better decisions, not just prettier dashboards. If a metric doesn't change behavior or inform action, stop tracking it.
-
-## The Vanity Metrics Problem
-
-### Vanity Metrics (Stop Measuring These)
-
-**Test Count**
-- "We have 5,000 tests!"
-- So what? Are they finding bugs? Are they maintainable? Do they give confidence?
-
-**Code Coverage Percentage**
-- "We achieved 85% coverage!"
-- Useless without context. 85% of what? Critical paths? Or just getters/setters?
-
-**Test Cases Executed**
-- "Ran 10,000 test cases today!"
-- How many found problems? How many are redundant?
-
-**Bugs Found**
-- "QA found 200 bugs this sprint!"
-- Is that good or bad? Are they trivial or critical? Should they have been found earlier?
-
-**Story Points Completed**
-- "We completed 50 points of testing work!"
-- Points are relative and gameable. What actually got better?
-
-### Why Vanity Metrics Fail
-
-1. **Easily gamed**: People optimize for the metric, not the goal
-2. **No context**: Numbers without meaning
-3. **No action**: What do you do differently based on this number?
-4. **False confidence**: High numbers that mean nothing
-
-## Meaningful Metrics
-
-### 1. Defect Escape Rate
-
-**What**: Percentage of bugs that reach production vs. caught before release
+---
 
-**Why it matters**: Measures effectiveness of your quality process
+## Core Metrics
 
-**How to measure**:
-```
-Defect Escape Rate = (Production Bugs / Total Bugs Found) × 100
+### Bug Escape Rate
 ```
+Bug Escape Rate = (Production Bugs / Total Bugs Found) × 100
 
-**Good**: < 5% escape rate
-**Needs work**: > 15% escape rate
-
-**Actions**:
-- High escape rate → Shift testing left, improve risk assessment
-- Low escape rate but slow releases → Maybe over-testing, reduce friction
-
-### 2. Mean Time to Detect (MTTD)
-
-**What**: How long from bug introduction to discovery
-
-**Why it matters**: Faster detection = cheaper fixes
-
-**How to measure**:
+Target: < 10% (90% caught before production)
 ```
-MTTD = Time bug found - Time bug introduced
-```
-
-**Good**: < 1 day for critical paths
-**Needs work**: > 1 week
-
-**Actions**:
-- High MTTD → Add monitoring, improve test coverage on critical paths
-- Very low MTTD → Your fast feedback loops are working
-
-### 3. Mean Time to Resolution (MTTR)
-
-**What**: Time from bug discovery to fix deployed
 
-**Why it matters**: Indicates team efficiency and process friction
-
-**How to measure**:
-```
-MTTR = Time fix deployed - Time bug discovered
+### Test Effectiveness
 ```
+Test Effectiveness = (Bugs Found by Tests / Total Bugs) × 100
 
-**Good**: < 24 hours for critical bugs, < 1 week for minor
-**Needs work**: > 1 week for critical bugs
-
-**Actions**:
-- High MTTR → Investigate bottlenecks (test env access? deployment pipeline? handoffs?)
-- Very low MTTR but high escape rate → Rushing fixes, need better verification
-
-### 4. Deployment Frequency
-
-**What**: How often you deploy to production
-
-**Why it matters**: Proxy for team confidence and process maturity
-
-**How to measure**:
-```
-Deployments per week (or day)
+Target: > 70%
 ```
 
-**Good**: Multiple per day
-**Decent**: Multiple per week
-**Needs work**: Less than weekly
-
-**Actions**:
-- Low frequency → Reduce batch size, improve automation, build confidence
-- High frequency with high defect rate → Need better automated checks
-
-### 5. Change Failure Rate
-
-**What**: Percentage of deployments that cause production issues
-
-**Why it matters**: Measures release quality
-
-**How to measure**:
+### Defect Density
 ```
-Change Failure Rate = (Failed Deployments / Total Deployments) × 100
-```
-
-**Good**: < 5%
-**Needs work**: > 15%
+Defect Density = Defects / KLOC
 
-**Actions**:
-- High failure rate → Improve pre-production validation, add canary deployments
-- Very low but slow releases → Maybe you can deploy more frequently
-
-### 6. Test Execution Time
-
-**What**: How long your test suite takes to run
-
-**Why it matters**: Slow tests = slow feedback = less frequent testing
-
-**How to measure**:
-```
-Time from commit to test completion
+Good: < 1 defect per KLOC
 ```
 
-**Good**: < 10 minutes for unit tests, < 30 minutes for full suite
-**Needs work**: > 1 hour
-
-**Actions**:
-- Slow tests → Parallelize, remove redundant tests, optimize slow tests
-- Fast tests but bugs escaping → Coverage gaps, need better tests
-
-### 7. Flaky Test Rate
-
-**What**: Percentage of tests that fail intermittently
-
-**Why it matters**: Flaky tests destroy confidence
-
-**How to measure**:
+### Mean Time to Detect (MTTD)
 ```
-Flaky Test Rate = (Flaky Tests / Total Tests) × 100
-```
-
-**Good**: < 1%
-**Needs work**: > 5%
-
-**Actions**:
-- High flakiness → Fix or delete flaky tests immediately (quarantine pattern)
-- Low flakiness → Maintain vigilance, don't let it creep up
-
-## Context-Specific Metrics
-
-### For Startups
-
-**Focus on**:
-- Deployment frequency (speed to market)
-- Critical path coverage (protect revenue)
-- MTTR (move fast, fix fast)
-
-**Skip**:
-- Comprehensive coverage metrics
-- Detailed test documentation
-- Complex traceability
-
-### For Regulated Industries
-
-**Focus on**:
-- Traceability (requirement → test → result)
-- Test documentation completeness
-- Audit trail integrity
-
-**Don't skip**:
-- Deployment frequency still matters
-- But compliance isn't optional
-
-### For Established Products
-
-**Focus on**:
-- Defect escape rate (protect reputation)
-- Regression detection (maintain stability)
-- Test maintenance cost
-
-**Balance**:
-- Innovation vs. stability
-- New features vs. technical debt
-
-## Leading vs. Lagging Indicators
-
-### Lagging Indicators (Rearview Mirror)
-- Defect escape rate
-- Production incidents
-- Customer complaints
-- MTTR
-
-**Use for**: Understanding what happened, trending over time
-
-### Leading Indicators (Windshield)
-- Code review quality
-- Test coverage on new code
-- Deployment frequency trend
-- Team confidence surveys
-
-**Use for**: Predicting problems, early intervention
-
-## Metrics for Different Audiences
-
-### For Developers
-- Test execution time
-- Flaky test rate
-- Code review turnaround
-- Build failure frequency
-
-**Language**: Technical, actionable
-
-### For Product/Management
-- Deployment frequency
-- Change failure rate
-- Feature lead time
-- Customer-impacting incidents
-
-**Language**: Business outcomes, not technical details
-
-### For Executive Leadership
-- Defect escape rate trend
-- Mean time to resolution
-- Release velocity
-- Customer satisfaction (related to quality)
-
-**Language**: Business impact, strategic
-
-## Building a Metrics Dashboard
-
-### Essential Dashboard (Start Here)
-
-**Top Row (Health)**
-- Defect escape rate (last 30 days)
-- Deployment frequency (last 7 days)
-- Change failure rate (last 30 days)
-
-**Middle Row (Speed)**
-- MTTD (average, last 30 days)
-- MTTR (average, last 30 days)
-- Test execution time (current)
-
-**Bottom Row (Trends)**
-- All of the above as sparklines (3-6 months)
+MTTD = Time(Bug Reported) - Time(Bug Introduced)
 
-### Advanced Dashboard (If Needed)
-
-Add:
-- Flaky test rate
-- Test coverage on critical paths (not overall %)
-- Production error rate
-- Customer-reported bugs vs. internally found
-
-## Anti-Patterns
-
-### ❌ Metric-Driven Development
-**Problem**: Optimizing for metrics instead of quality
-
-**Example**: Writing useless tests to hit coverage targets
-
-**Fix**: Focus on outcomes (can we deploy confidently?) not numbers
-
-### ❌ Too Many Metrics
-**Problem**: Dashboard overload, no clear priorities
-
-**Example**: Tracking 30+ metrics that no one understands
-
-**Fix**: Start with 5-7 core metrics, add only if they drive decisions
-
-### ❌ Metrics Without Action
-**Problem**: Tracking numbers but not changing behavior
-
-**Example**: Watching MTTR climb for months without investigating
-
-**Fix**: For every metric, define thresholds and actions
-
-### ❌ Gaming the System
-**Problem**: People optimize for metrics, not quality
-
-**Example**: Marking bugs as "won't fix" to improve resolution time
-
-**Fix**: Multiple complementary metrics, qualitative reviews
-
-### ❌ One-Size-Fits-All
-**Problem**: Using same metrics for all teams/contexts
-
-**Example**: Measuring startup team same as regulated medical device team
-
-**Fix**: Context-driven metric selection
-
-## Metric Hygiene
-
-### Review Quarterly
-- Are we still using this metric to make decisions?
-- Is it being gamed?
-- Does it reflect current priorities?
-
-### Adjust Thresholds
-- What's "good" changes as you improve
-- Don't keep celebrating the same baseline
-- Raise the bar when appropriate
-
-### Kill Zombie Metrics
-- If no one looks at it → Delete it
-- If no one can explain what action to take → Delete it
-- If it's always green or always red → Delete it
-
-## Real-World Examples
-
-### Example 1: E-Commerce Company
-
-**Before**:
-- Measured: Test count (5,000 tests)
-- Result: Slow CI, frequent production bugs
-
-**After**:
-- Measured: Defect escape rate (8%), MTTD (3 days), deployment frequency (2/week)
-- Actions: 
-  - Removed 2,000 redundant tests
-  - Added monitoring for critical paths
-  - Improved deployment pipeline
-- Result: Escape rate to 3%, MTTD to 6 hours, deploy 5x/day
-
-### Example 2: SaaS Platform
-
-**Before**:
-- Measured: Code coverage (85%)
-- Result: False confidence, bugs in uncovered critical paths
-
-**After**:
-- Measured: Critical path coverage (60%), deployment frequency, change failure rate
-- Actions:
-  - Focused testing on payment, auth, data integrity
-  - Removed tests on deprecated features
-  - Added production monitoring
-- Result: Fewer production incidents, faster releases
-
-## Questions to Ask About Any Metric
-
-1. **What decision does this inform?**
-   - If none → Don't track it
-
-2. **What action do we take if it's red?**
-   - If you don't know → Define thresholds and actions
-
-3. **Can this be gamed?**
-   - If yes → Add complementary metrics
-
-4. **Does this reflect actual quality?**
-   - If no → Replace it with something that does
-
-5. **Who needs to see this?**
-   - If no one → Stop tracking it
-
-## Remember
-
-**Good metrics**:
-- Drive better decisions
-- Are actionable
-- Reflect actual outcomes
-- Change as you mature
-
-**Bad metrics**:
-- Make dashboards pretty
-- Are easily gamed
-- Provide false confidence
-- Persist long after they're useful
-
-**Start small**: 5-7 metrics that matter
-**Review often**: Quarterly at minimum
-**Kill ruthlessly**: Remove metrics that don't drive action
-**Stay contextual**: What matters changes with your situation
-
-## Using with QE Agents
-
-### Automated Metrics Collection
-
-**qe-quality-analyzer** collects and analyzes quality metrics:
-```typescript
-// Agent collects comprehensive metrics automatically
-await agent.collectMetrics({
-  scope: 'all',
-  timeframe: '30d',
-  categories: [
-    'deployment-frequency',
-    'defect-escape-rate',
-    'test-execution-time',
-    'flaky-test-rate',
-    'coverage-trends'
-  ]
-});
-
-// Returns real-time dashboard data
-// No manual tracking required
+Target: < 1 day for critical, < 1 week for others
 ```
 
-### Intelligent Metric Analysis
+---
 
-**qe-quality-analyzer** identifies trends and anomalies:
-```typescript
-// Agent detects metric anomalies
-const analysis = await agent.analyzeTrends({
-  metric: 'defect-escape-rate',
-  timeframe: '90d',
-  alertThreshold: 0.15
-});
+## Dashboard Design
 
-// Returns:
-// {
-//   trend: 'increasing',
-//   currentValue: 0.18,
-//   avgValue: 0.08,
-//   anomaly: true,
-//   recommendation: 'Increase pre-release testing focus',
-//   relatedMetrics: ['test-coverage: decreasing', 'MTTR: increasing']
-// }
+```typescript
+// Agent generates quality dashboard
+await Task("Generate Dashboard", {
+  metrics: {
+    delivery: ['deployment-frequency', 'lead-time', 'change-failure-rate'],
+    quality: ['bug-escape-rate', 'test-effectiveness', 'defect-density'],
+    stability: ['mttd', 'mttr', 'availability'],
+    process: ['code-review-time', 'flaky-test-rate', 'coverage-trend']
+  },
+  visualization: 'grafana',
+  alerts: {
+    critical: { bug_escape_rate: '>20%', mttr: '>24h' },
+    warning: { coverage: '<70%', flaky_rate: '>5%' }
+  }
+}, "qe-quality-analyzer");
 ```
 
-### Actionable Insights from Metrics
+---
 
-**qe-quality-gate** uses metrics for decision-making:
-```typescript
-// Agent makes GO/NO-GO decisions based on metrics
-const decision = await agent.evaluateMetrics({
-  release: 'v3.2',
-  thresholds: {
-    defectEscapeRate: '<5%',
-    changeFailureRate: '<10%',
-    testExecutionTime: '<15min',
-    flakyTestRate: '<2%'
+## Quality Gate Configuration
+
+```json
+{
+  "qualityGates": {
+    "commit": {
+      "coverage": { "min": 80, "blocking": true },
+      "lint": { "errors": 0, "blocking": true }
+    },
+    "pr": {
+      "tests": { "pass": "100%", "blocking": true },
+      "security": { "critical": 0, "blocking": true },
+      "coverage_delta": { "min": 0, "blocking": false }
+    },
+    "release": {
+      "e2e": { "pass": "100%", "blocking": true },
+      "performance_p95": { "max_ms": 200, "blocking": true },
+      "bug_escape_rate": { "max": "10%", "blocking": false }
+    }
   }
-});
-
-// Returns:
-// {
-//   decision: 'NO-GO',
-//   blockers: [
-//     'Flaky test rate: 4.2% (threshold: 2%)'
-//   ],
-//   recommendations: [
-//     'Run qe-flaky-test-hunter to stabilize tests'
-//   ]
-// }
+}
 ```
 
-### Real-Time Metrics Dashboard
+---
+
+## Agent-Assisted Metrics
 
-**qe-quality-analyzer** generates live dashboards:
 ```typescript
-// Agent creates context-specific dashboards
-await agent.createDashboard({
-  audience: 'executive',  // or 'developer', 'product'
-  focus: 'release-readiness',
-  updateFrequency: 'real-time'
-});
+// Calculate quality trends
+await Task("Quality Trend Analysis", {
+  timeframe: '90d',
+  metrics: ['bug-escape-rate', 'mttd', 'test-effectiveness'],
+  compare: 'previous-90d',
+  predictNext: '30d'
+}, "qe-quality-analyzer");
 
-// Executive Dashboard:
-// - Defect escape rate: 3.2% ✅
-// - Deployment frequency: 5/day ✅
-// - Change failure rate: 7% ✅
-// - Customer-impacting incidents: 1 (down from 3)
+// Evaluate quality gate
+await Task("Quality Gate Evaluation", {
+  buildId: 'build-123',
+  environment: 'staging',
+  metrics: currentMetrics,
+  policy: qualityPolicy
+}, "qe-quality-gate");
 ```
 
-### Metric-Driven Test Optimization
+---
 
-**qe-regression-risk-analyzer** uses metrics to optimize testing:
-```typescript
-// Agent identifies which tests provide most value
-const optimization = await agent.optimizeTestSuite({
-  metrics: {
-    executionTime: 'per-test',
-    defectDetectionRate: 'per-test',
-    maintenanceCost: 'per-test'
-  },
-  goal: 'maximize-value-per-minute'
-});
+## Agent Coordination Hints
 
-// Recommends:
-// - Remove 50 tests with 0% defect detection (save 15 min)
-// - Keep top 200 tests (95% defect detection)
-// - Result: 40% faster suite, 5% defect detection loss
+### Memory Namespace
+```
+aqe/quality-metrics/
+├── dashboards/*         - Dashboard configurations
+├── trends/*             - Historical metric data
+├── gates/*              - Gate evaluation results
+└── alerts/*             - Triggered alerts
 ```
 
-### Fleet Coordination for Metrics
-
+### Fleet Coordination
 ```typescript
-// Multiple agents collaborate on metrics collection and analysis
 const metricsFleet = await FleetManager.coordinate({
   strategy: 'quality-metrics',
   agents: [
-    'qe-test-executor',         // Collect execution metrics
-    'qe-coverage-analyzer',     // Collect coverage metrics
-    'qe-production-intelligence', // Collect production metrics
-    'qe-quality-analyzer',      // Analyze and visualize
-    'qe-quality-gate'           // Make decisions
+    'qe-quality-analyzer',         // Trend analysis
+    'qe-test-executor',            // Test metrics
+    'qe-coverage-analyzer',        // Coverage data
+    'qe-production-intelligence',  // Production metrics
+    'qe-quality-gate'              // Gate decisions
   ],
-  topology: 'hierarchical'
-});
-
-// Continuous metrics pipeline
-await metricsFleet.execute({
-  schedule: 'continuous',
-  aggregationInterval: '5min'
+  topology: 'mesh'
 });
 ```
 
-### Context-Aware Metric Selection
+---
 
-```typescript
-// Agent recommends metrics based on context
-const recommendation = await qe-quality-analyzer.recommendMetrics({
-  context: 'startup',
-  stage: 'early',
-  team: 'small',
-  compliance: 'none'
-});
+## Common Traps
 
-// Recommends:
-// - deployment-frequency (speed to market)
-// - critical-path-coverage (protect revenue)
-// - MTTR (move fast, fix fast)
-//
-// Skip:
-// - comprehensive coverage %
-// - detailed traceability
-// - process compliance metrics
-```
+| Trap | Problem | Solution |
+|------|---------|----------|
+| Coverage worship | 100% coverage, bugs still escape | Measure bug escape rate instead |
+| Test count focus | Many tests, slow feedback | Measure execution time |
+| Activity metrics | Busy work, no outcomes | Measure outcomes (MTTD, MTTR) |
+| Point-in-time | Snapshot without context | Track trends over time |
 
 ---
 
 ## Related Skills
-
-**Core Quality Practices:**
-- [agentic-quality-engineering](../agentic-quality-engineering/) - Metrics-driven agent coordination
-- [holistic-testing-pact](../holistic-testing-pact/) - Metrics across test quadrants
-
-**Testing Approaches:**
-- [risk-based-testing](../risk-based-testing/) - Risk-based metric selection
-- [test-automation-strategy](../test-automation-strategy/) - Automation effectiveness metrics
-- [exploratory-testing-advanced](../exploratory-testing-advanced/) - Exploratory session metrics
-
-**Development Practices:**
-- [xp-practices](../xp-practices/) - XP success metrics (velocity, lead time)
+- [agentic-quality-engineering](../agentic-quality-engineering/) - Agent coordination
+- [cicd-pipeline-qe-orchestrator](../cicd-pipeline-qe-orchestrator/) - Quality gates
+- [risk-based-testing](../risk-based-testing/) - Risk-informed metrics
+- [shift-right-testing](../shift-right-testing/) - Production metrics
 
 ---
 
-## Resources
-
-- **Accelerate** by Forsgren, Humble, Kim (DORA metrics)
-- **How to Measure Anything** by Douglas Hubbard (measuring intangibles)
-- Your own retrospectives (which metrics helped? Which didn't?)
+## Remember
 
-Metrics are tools for better decisions, not scorecards for performance reviews. Use them wisely.
+**Measure outcomes, not activities.** Bug escape rate > test count. MTTD/MTTR > coverage %. Trends > snapshots. Set gates that block bad code. What you measure is what you optimize.
 
-**With Agents**: Agents automate metrics collection, detect trends and anomalies, and provide context-aware recommendations. Use agents to make metrics actionable and avoid vanity metrics. Agents continuously analyze what drives quality outcomes in your specific context.
+**With Agents:** Agents track metrics automatically, analyze trends, trigger alerts, and make gate decisions. Use agents to maintain continuous quality visibility.