agentic-qe 1.9.4 → 2.1.0

package/.claude/skills/test-reporting-analytics/SKILL.md

@@ -1,137 +1,214 @@
 ---
 name: test-reporting-analytics
-description: Advanced test reporting, quality dashboards, predictive analytics, trend analysis, and executive reporting for QE metrics. Use when communicating quality status, tracking trends, or making data-driven decisions.
+description: "Advanced test reporting, quality dashboards, predictive analytics, trend analysis, and executive reporting for QE metrics. Use when communicating quality status, tracking trends, or making data-driven decisions."
+category: analytics
+priority: high
+tokenEstimate: 850
+agents: [qe-quality-analyzer, qe-quality-gate, qe-deployment-readiness]
+implementation_status: optimized
+optimization_version: 1.0
+last_optimized: 2025-12-03
+dependencies: []
+quick_reference_card: true
+tags: [reporting, analytics, dashboards, metrics, trends, predictive]
 ---
 
 # Test Reporting & Analytics
 
-## Core Principle
+<default_to_action>
+When building test reports:
+1. DEFINE audience (dev team vs executives)
+2. CHOOSE key metrics (max 5-7)
+3. SHOW trends (not just snapshots)
+4. HIGHLIGHT actions (what to do about it)
+5. AUTOMATE generation
 
-**Measure to improve. Report to communicate.**
+**Dashboard Quick Setup:**
+```
++------------------+------------------+------------------+
+| Tests Passed     | Code Coverage    | Flaky Tests      |
+| 1,247/1,250 ✅   | 82.3% ⬆️ +2.1%  | 1.2% ⬇️ -0.3%   |
++------------------+------------------+------------------+
+| Critical Bugs    | Deploy Freq      | MTTR             |
+| 0 open ✅        | 12x/day ⬆️       | 2.3h ⬇️          |
++------------------+------------------+------------------+
+```
 
-Test reporting transforms raw test data into actionable insights. Analytics enable data-driven quality decisions.
+**Key Metrics by Audience:**
+- **Dev Team**: Pass rate, flaky %, execution time, coverage gaps
+- **QE Team**: Defect detection rate, test velocity, automation ROI
+- **Leadership**: Escaped defects, deployment frequency, quality cost
+</default_to_action>
 
-## Key Metrics
+## Quick Reference Card
 
-### Test Execution Metrics
-- Pass/Fail rate
-- Flaky test percentage
-- Execution time (total, per test)
-- Test coverage (code, requirements)
+### Essential Metrics
 
-### Quality Metrics
-- Defect density
-- Defect detection rate
-- Escaped defects
-- Mean time to detect (MTTD)
-- Mean time to resolve (MTTR)
+| Category | Metric | Target |
+|----------|--------|--------|
+| **Execution** | Pass Rate | >98% |
+| **Execution** | Flaky Test % | <2% |
+| **Execution** | Suite Duration | <10 min |
+| **Coverage** | Line Coverage | >80% |
+| **Coverage** | Branch Coverage | >70% |
+| **Quality** | Escaped Defects | <5/release |
+| **Quality** | MTTR | <4 hours |
+| **Efficiency** | Automation Rate | >90% |
 
-### Efficiency Metrics
-- Automation rate
-- Test maintenance cost
-- ROI of automation
-- Velocity (features tested/sprint)
+### Trend Indicators
 
-## Dashboards
+| Symbol | Meaning | Action |
+|--------|---------|--------|
+| ⬆️ | Improving | Continue current approach |
+| ⬇️ | Declining | Investigate root cause |
+| ➡️ | Stable | Maintain or improve |
+| ⚠️ | Threshold breach | Immediate attention |
 
-**Real-Time Quality Dashboard:**
-```
-+------------------+------------------+------------------+
-| Tests Passed     | Code Coverage    | Flaky Tests      |
-| 1,247 / 1,250    | 82.3%           | 1.2%            |
-| 99.76% ✅       | ⬆️ +2.1%        | ⬇️ -0.3%        |
-+------------------+------------------+------------------+
+---
 
-+------------------+------------------+------------------+
-| Critical Bugs    | Test Velocity    | Deploy Freq     |
-| 0 open           | 47 tests/sprint  | 12x/day         |
-| ✅              | ⬆️ +5           | ⬆️ +2x          |
-+------------------+------------------+------------------+
+## Report Types
 
-Recent Trends (30 days):
-[Graph showing pass rate, coverage, flaky tests over time]
+### Real-Time Dashboard
+```
+Live quality status for CI/CD
+- Build status (green/red)
+- Test results (pass/fail counts)
+- Coverage delta
+- Flaky test alerts
 ```
 
-## Trend Analysis
-
-```javascript
-// Identify trends
-const testResults = await fetchTestResults(30); // 30 days
+### Sprint Summary
+```markdown
+## Sprint 47 Quality Summary
+
+### Metrics
+| Metric | Value | Trend |
+|--------|-------|-------|
+| Tests Added | +47 | ⬆️ |
+| Coverage | 82.3% | ⬆️ +2.1% |
+| Bugs Found | 12 | ➡️ |
+| Escaped | 0 | ✅ |
+
+### Highlights
+- ✅ Zero escaped defects
+- ⚠️ E2E suite now 45min (target: 30min)
+
+### Actions
+1. Optimize slow E2E tests
+2. Add coverage for payment module
+```
 
-const trend = analyzeTrend(testResults, 'passRate');
-if (trend === 'declining') {
-  alert('⚠️ Test pass rate declining for 7 days');
-}
+### Executive Report
+```markdown
+## Monthly Quality Report - Oct 2025
 
-const coverage = analyzeTrend(testResults, 'coverage');
-if (coverage === 'stagnant') {
-  alert('ℹ️ Code coverage unchanged. Add tests for new code.');
-}
+### Executive Summary
+✅ Production uptime: 99.97% (target: 99.95%)
+✅ Deploy frequency: 12x/day (up from 8x)
+⚠️ Coverage: 82.3% (target: 85%)
+
+### Business Impact
+- Automation saves 120 hrs/month
+- Bug cost: $150/bug found vs $5,000 escaped
+- Estimated annual savings: $450K
+
+### Recommendations
+1. Invest in performance testing tooling
+2. Hire senior QE for mobile coverage
 ```
 
+---
+
 ## Predictive Analytics
 
 ```typescript
 // Predict test failures
-const prediction = await agent.predictTestFailures({
-  historicalData: testResults,
+const prediction = await Task("Predict Failures", {
   codeChanges: prDiff,
-  teamMetrics: velocityData
-});
+  historicalData: last90Days,
+  model: 'gradient-boosting'
+}, "qe-quality-analyzer");
 
 // Returns:
 // {
-//   probabilityOfFailure: 0.73,
-//   likelyFailingTests: ['payment.test.ts', 'checkout.test.ts'],
-//   suggestedAction: 'Review payment module changes carefully',
+//   failureProbability: 0.73,
+//   likelyFailingTests: ['payment.test.ts'],
+//   suggestedAction: 'Review payment module carefully',
 //   confidence: 0.89
 // }
+
+// Trend analysis with anomaly detection
+const trends = await Task("Analyze Trends", {
+  metrics: ['passRate', 'coverage', 'flakyRate'],
+  period: '30d',
+  detectAnomalies: true
+}, "qe-quality-analyzer");
 ```
 
-## Executive Reporting
+---
 
-**Monthly Quality Report:**
-```markdown
-## Quality Report - October 2025
+## Agent Integration
 
-### Executive Summary
-✅ Production: 99.97% uptime (target: 99.95%)
-✅ Deployment: 12x/day (up from 8x/day)
-⚠️ Test Coverage: 82.3% (target: 85%)
-
-### Key Achievements
-- Reduced flaky tests from 3.2% to 1.2%
-- Automated 47 new tests (95% automation rate)
-- 0 critical bugs escaped to production
-
-### Action Items
-- Increase coverage for new payment module
-- Address 3 long-running flaky tests
-- Train team on performance testing
-
-### ROI
-- Automation saves 120 hours/month
-- Bug detection cost: $150/bug vs $5,000 in production
-- Estimated annual savings: $450k
+```typescript
+// Generate comprehensive quality report
+const report = await Task("Generate Quality Report", {
+  period: 'sprint',
+  audience: 'executive',
+  includeROI: true,
+  includeTrends: true
+}, "qe-quality-analyzer");
+
+// Real-time quality gate check
+const gateResult = await Task("Quality Gate Check", {
+  metrics: currentMetrics,
+  thresholds: qualityPolicy,
+  environment: 'production'
+}, "qe-quality-gate");
 ```
 
+---
+
+## Agent Coordination Hints
+
+### Memory Namespace
+```
+aqe/reporting/
+├── dashboards/*      - Dashboard configurations
+├── reports/*         - Generated reports
+├── trends/*          - Trend analysis data
+└── predictions/*     - Predictive model outputs
+```
+
+### Fleet Coordination
+```typescript
+const reportingFleet = await FleetManager.coordinate({
+  strategy: 'quality-reporting',
+  agents: [
+    'qe-quality-analyzer',      // Metrics aggregation
+    'qe-quality-gate',          // Threshold validation
+    'qe-deployment-readiness'   // Release readiness
+  ],
+  topology: 'parallel'
+});
+```
+
+---
+
 ## Related Skills
+- [quality-metrics](../quality-metrics/) - Metric definitions
+- [shift-right-testing](../shift-right-testing/) - Production metrics
+- [consultancy-practices](../consultancy-practices/) - Client reporting
 
-- [quality-metrics](../quality-metrics/)
-- [agentic-quality-engineering](../agentic-quality-engineering/)
-- [continuous-testing-shift-left](../continuous-testing-shift-left/)
+---
 
 ## Remember
 
-**Track metrics to improve quality.**
-
-Report:
-- Test results (pass/fail trends)
-- Code coverage (gaps and trends)
-- Flaky test rate (reliability)
-- Defect metrics (escaped bugs)
-- ROI of testing (business value)
+**Measure to improve. Report to communicate.**
 
-**Make data actionable, not just visible.**
+Good reports:
+- Answer "so what?" (actionable insights)
+- Show trends (not just snapshots)
+- Match audience needs
+- Automate where possible
 
-**With Agents:** `qe-quality-analyzer` aggregates metrics, generates insights, predicts trends, and creates executive reports automatically.
+**Data without action is noise. Action without data is guessing.**

package/.claude/skills/testability-scoring/README.md

@@ -0,0 +1,71 @@
+# Testability Scoring Skill
+
+Quick reference for AI-powered testability assessment using 10 principles of intrinsic testability.
+
+## Quick Start
+
+```bash
+# Run assessment
+.claude/skills/testability-scoring/scripts/run-assessment.sh https://example.com/
+
+# Generate HTML report from JSON
+AUTO_OPEN=false node .claude/skills/testability-scoring/scripts/generate-html-report.js tests/reports/testability-results-*.json
+```
+
+## 10 Principles
+
+1. **Observability** (15%) - Can we see what's happening?
+2. **Controllability** (15%) - Can we control the application?
+3. **Algorithmic Simplicity** (10%) - Are behaviors simple?
+4. **Algorithmic Transparency** (10%) - Can we understand it?
+5. **Algorithmic Stability** (10%) - Does it stay consistent?
+6. **Explainability** (10%) - Is the interface clear?
+7. **Unbugginess** (10%) - How error-free is it?
+8. **Smallness** (10%) - Are components appropriate size?
+9. **Decomposability** (5%) - Can we test parts separately?
+10. **Similarity** (5%) - How familiar is the tech?
+
+## Scoring
+
+- **90-100 (A)**: Excellent
+- **80-89 (B)**: Good
+- **70-79 (C)**: Adequate
+- **60-69 (D)**: Below average
+- **0-59 (F)**: Poor
+
+## Files
+
+```
+.claude/skills/testability-scoring/
+├── SKILL.md                    # Complete documentation
+├── scripts/
+│   ├── run-assessment.sh       # Main runner
+│   └── generate-html-report.js # Report generator
+└── resources/templates/        # Templates
+
+tests/testability-scoring/
+├── testability-scoring.spec.js # Test implementation
+└── config.js                   # Configuration
+
+tests/reports/
+├── testability-results-*.json  # Results
+└── testability-report-*.html   # Reports
+```
+
+## Common Commands
+
+```bash
+# Disable auto-open
+AUTO_OPEN=false ./scripts/run-assessment.sh https://example.com/
+
+# With timeout
+timeout 180 ./scripts/run-assessment.sh https://example.com/
+
+# Specific browser
+./scripts/run-assessment.sh https://example.com/ firefox
+```
+
+## Credits
+
+Based on James Bach and Michael Bolton's *Heuristics for Software Testability*  
+Implementation: https://github.com/fndlalit/testability-scorer

package/.claude/skills/testability-scoring/SKILL.md

@@ -0,0 +1,245 @@
+---
+name: testability-scoring
+description: "AI-powered testability assessment using 10 principles of intrinsic testability with Playwright. Evaluates web applications against Observability, Controllability, Algorithmic Simplicity, Transparency, Stability, Explainability, Unbugginess, Smallness, Decomposability, and Similarity. Use when assessing software testability, evaluating test readiness, identifying testability improvements, or generating testability reports."
+category: testing-methodologies
+priority: high
+tokenEstimate: 1100
+agents: [qe-quality-analyzer, qx-partner, qe-visual-tester]
+implementation_status: optimized
+optimization_version: 2.1
+last_optimized: 2025-12-03
+dependencies: []
+quick_reference_card: true
+tags: [testability, scoring, playwright, assessment, 10-principles, intrinsic-testability, james-bach, michael-bolton]
+contributor: "@fndlalit"
+---
+
+# Testability Scoring
+
+<default_to_action>
+When assessing testability:
+1. RUN assessment against target URL
+2. ANALYZE all 10 principles automatically
+3. GENERATE HTML report with radar chart
+4. PRIORITIZE improvements by impact/effort
+5. INTEGRATE with QX Partner for holistic view
+
+**Quick Assessment:**
+```bash
+# Run assessment on any URL
+TEST_URL='https://example.com/' npx playwright test tests/testability-scoring/testability-scoring.spec.js --project=chromium --workers=1
+
+# Or use shell script wrapper
+.claude/skills/testability-scoring/scripts/run-assessment.sh https://example.com/
+```
+
+**The 10 Principles at a Glance:**
+| Principle | Weight | Key Question |
+|-----------|--------|--------------|
+| **Observability** | 15% | Can we see what's happening? |
+| **Controllability** | 15% | Can we control the application? |
+| **Algorithmic Simplicity** | 10% | Are behaviors predictable? |
+| **Algorithmic Transparency** | 10% | Can we understand what it does? |
+| **Algorithmic Stability** | 10% | Does behavior remain consistent? |
+| **Explainability** | 10% | Is the interface understandable? |
+| **Unbugginess** | 10% | How error-free is it? |
+| **Smallness** | 10% | Are components appropriately sized? |
+| **Decomposability** | 5% | Can we test parts in isolation? |
+| **Similarity** | 5% | Is the tech stack familiar? |
+
+**Grade Scale:**
+- **A (90-100)**: Excellent testability
+- **B (80-89)**: Good testability
+- **C (70-79)**: Adequate testability
+- **D (60-69)**: Below average
+- **F (0-59)**: Poor testability
+</default_to_action>
+
+## Quick Reference Card
+
+### Running Assessments
+
+| Method | Command | When to Use |
+|--------|---------|-------------|
+| Shell Script | `./scripts/run-assessment.sh URL` | One-time assessment |
+| ENV Override | `TEST_URL='URL' npx playwright test...` | CI/CD integration |
+| Config File | Update `tests/testability-scoring/config.js` | Repeated runs |
+
+### Principle Details
+
+#### High Weight (15% each)
+| Principle | Measures | Indicators |
+|-----------|----------|------------|
+| **Observability** | State visibility, logging, monitoring | Console output, network tracking, error visibility |
+| **Controllability** | Input control, state manipulation | API access, test data injection, determinism |
+
+#### Medium Weight (10% each)
+| Principle | Measures | Indicators |
+|-----------|----------|------------|
+| **Simplicity** | Predictable behavior | Clear I/O relationships, low complexity |
+| **Transparency** | Understanding what system does | Visible processes, readable code |
+| **Stability** | Consistent behavior | Change resilience, maintainability |
+| **Explainability** | Interface understanding | Good docs, semantic structure, help text |
+| **Unbugginess** | Error-free operation | Console errors, warnings, runtime issues |
+| **Smallness** | Component size | Element count, script bloat, page complexity |
+
+#### Low Weight (5% each)
+| Principle | Measures | Indicators |
+|-----------|----------|------------|
+| **Decomposability** | Isolation testing | Component separation, modular design |
+| **Similarity** | Technology familiarity | Standard frameworks, known patterns |
+
+---
+
+## Assessment Workflow
+
+```
+1. Navigate to URL → 2. Collect Metrics → 3. Score Principles
+                                              ↓
+4. Generate JSON ← 5. Calculate Grades ← 6. Apply Weights
+         ↓
+7. Generate HTML Report with Radar Chart
+         ↓
+8. Open in Browser (auto-opens)
+```
+
+### Output Files
+
+```
+tests/reports/
+├── testability-results-<timestamp>.json  # Raw data
+├── testability-report-<timestamp>.html   # Visual report
+└── latest.json                           # Symlink
+```
+
+---
+
+## Integration Examples
+
+### CI/CD Integration
+```yaml
+# GitHub Actions
+- name: Testability Assessment
+  run: |
+    timeout 180 .claude/skills/testability-scoring/scripts/run-assessment.sh ${{ env.APP_URL }}
+
+- name: Upload Reports
+  uses: actions/upload-artifact@v3
+  with:
+    name: testability-reports
+    path: tests/reports/testability-*.html
+```
+
+### QX Partner Integration
+```typescript
+// Combine testability with QX analysis
+const qxAnalysis = await Task("QX Analysis", {
+  target: 'https://example.com',
+  integrateTestability: true
+}, "qx-partner");
+
+// Returns combined insights:
+// - QX Score: 78/100
+// - Testability Integration: Observability 72/100
+// - Combined Insight: Low observability may mask UX issues
+```
+
+### Programmatic Usage
+```typescript
+import { runTestabilityAssessment } from './testability';
+
+const results = await runTestabilityAssessment('https://example.com');
+console.log(`Overall: ${results.overallScore}/100 (${results.grade})`);
+console.log('Recommendations:', results.recommendations);
+```
+
+---
+
+## Agent Integration
+
+```typescript
+// Run testability assessment
+const assessment = await Task("Testability Assessment", {
+  url: 'https://example.com',
+  generateReport: true,
+  openBrowser: true
+}, "qe-quality-analyzer");
+
+// Use with QX Partner for holistic analysis
+const qxReport = await Task("Full QX Analysis", {
+  target: 'https://example.com',
+  integrateTestability: true,
+  detectOracleProblems: true
+}, "qx-partner");
+```
+
+---
+
+## Agent Coordination Hints
+
+### Memory Namespace
+```
+aqe/testability/
+├── assessments/*       - Assessment results by URL
+├── historical/*        - Historical scores for trend analysis
+├── recommendations/*   - Improvement recommendations
+└── integration/*       - QX integration data
+```
+
+### Fleet Coordination
+```typescript
+const testabilityFleet = await FleetManager.coordinate({
+  strategy: 'testability-assessment',
+  agents: [
+    'qe-quality-analyzer',  // Primary assessment
+    'qx-partner',           // UX integration
+    'qe-visual-tester'      // Visual validation
+  ],
+  topology: 'sequential'
+});
+```
+
+---
+
+## Common Issues & Solutions
+
+| Issue | Solution |
+|-------|----------|
+| Tests timing out | Increase timeout: `timeout 300 ./scripts/run-assessment.sh URL` |
+| Partial results | Check console errors, increase network timeout |
+| Report not opening | Use `AUTO_OPEN=false`, open manually |
+| Config not updating | Use `TEST_URL` env var instead |
+
+---
+
+## Related Skills
+- [accessibility-testing](../accessibility-testing/) - WCAG compliance (overlaps with Explainability)
+- [visual-testing-advanced](../visual-testing-advanced/) - UI consistency
+- [performance-testing](../performance-testing/) - Load time metrics
+
+---
+
+## Credits & References
+
+### Framework Origin
+- **Heuristics for Software Testability** by James Bach and Michael Bolton
+- Available at: https://www.satisfice.com/download/heuristics-of-software-testability
+
+### Implementation
+- Based on https://github.com/fndlalit/testability-scorer (contributed by [@fndlalit](https://github.com/fndlalit))
+- Playwright v1.49.0+ with AI capabilities
+- Chart.js for radar visualizations
+
+---
+
+## Remember
+
+**Testability is an investment, not an afterthought.**
+
+Good testability:
+- Reduces debugging time
+- Enables faster feedback loops
+- Makes defects easier to find
+- Supports continuous testing
+
+**Low scores = High risk. Prioritize improvements by weight × impact.**