qa-skills 3.0.0

package/skills/qa-test-reporter/references/best-practices.md

@@ -0,0 +1,162 @@
+# Test Reporting Best Practices
+
+*Guidance for meaningful metrics, trend analysis, and stakeholder communication.*
+
+---
+
+## Meaningful Metrics
+
+### Primary Metrics
+
+| Metric | Use | Avoid |
+| ------ | --- | ----- |
+| **Pass rate** | % passed / total executed | Excluding skipped inflates rate; document skip reasons |
+| **Failure count** | Absolute and trend | Ignoring flaky tests masks real regressions |
+| **Duration** | Trend over time; SLA compliance | Single-run duration without context |
+| **Coverage** | Risk areas; uncovered code | Coverage as sole quality gate |
+
+### Secondary Metrics
+
+- **Flaky rate** — Tests that pass/fail inconsistently; prioritize stabilization
+- **Failure density** — Failures per module/suite; identify hotspots
+- **Blocked count** — Tests blocked by environment or defects
+- **Defect escape rate** — Production defects / total defects found (post-release)
+
+### Anti-Patterns
+
+1. **Vanity metrics** — High pass rate with many skipped tests
+2. **Single-number focus** — Pass rate alone without failure analysis
+3. **Ignoring trends** — One bad run vs. sustained regression
+4. **Coverage as proxy for quality** — 100% coverage with weak assertions
+
+---
+
+## Trend Analysis
+
+### What to Track
+
+| Data Point | Purpose |
+| ---------- | ------- |
+| Pass rate over last N runs | Detect regression |
+| Failure count trend | New vs. recurring failures |
+| Duration trend | Performance degradation |
+| Flaky test list | Prioritize stabilization |
+| Coverage trend | Coverage growth or regression |
+
+### Storage (Memory MCP)
+
+Store per run:
+
+```json
+{
+  "runId": "build-123",
+  "timestamp": "2025-03-07T10:00:00Z",
+  "total": 100,
+  "passed": 92,
+  "failed": 5,
+  "skipped": 3,
+  "duration": 120000,
+  "passRate": 92
+}
+```
+
+Query for:
+
+- Last 7 days
+- Last 10 builds
+- Same branch/commit family
+
+### Reporting Trends
+
+- **Improving** — Pass rate up, failures down
+- **Stable** — No significant change
+- **Regressing** — Pass rate down, failures up; flag for investigation
+- **New baseline** — First run or major scope change; avoid over-interpretation
+
+---
+
+## Stakeholder Communication
+
+### Audience Tiers
+
+| Audience | Focus | Detail Level |
+| -------- | ----- | ------------- |
+| **Executive** | Go/no-go, risk, timeline | 1-page summary |
+| **Project manager** | Progress, blockers, next steps | Status report |
+| **Developers** | Failures, flaky tests, top issues | Failure analysis, links to failures |
+| **QA** | Full metrics, trends, coverage | Full dashboard |
+
+### Report Cadence
+
+- **Daily/Sprint** — Status report (progress, blockers)
+- **Release** — Completion report (go/no-go, recommendations)
+- **Ad-hoc** — HTML dashboard, Markdown summary for PRs
+
+### Clarity Principles
+
+1. **Lead with the answer** — Go/no-go and key metrics first
+2. **Use plain language** — Avoid jargon for non-technical readers
+3. **Provide context** — "Pass rate 92%, up from 88% last week"
+4. **Actionable** — "5 flaky tests in checkout flow — stabilize before release"
+5. **Traceability** — Link failures to test IDs, defects, requirements
+
+---
+
+## Failure Analysis
+
+### Top Failures
+
+Rank failures by:
+
+1. **Frequency** — Same test failing across runs
+2. **Impact** — Critical path, high-priority area
+3. **Recency** — New failures vs. long-standing
+
+### Flaky Detection
+
+- Same test: pass and fail across runs without code change
+- Require minimum runs (e.g., 5+) before flagging
+- Report: "Flaky (3/10 runs failed)"
+
+### Failure Grouping
+
+Group by:
+
+- **Module/suite** — Identify weak areas
+- **Error type** — Timeout, assertion, environment
+- **Root cause** — When known, tag for reporting
+
+---
+
+## Risk Areas
+
+Flag when:
+
+- **Low coverage** — Module/suite below threshold (e.g., < 70%)
+- **High failure density** — Many failures in one area
+- **Blocked tests** — Cannot execute; blocks exit criteria
+- **Open Critical/High** — Unresolved defects in scope
+- **Duration spike** — Significant slowdown vs. baseline
+
+Present in report: "Risk areas: Checkout module (65% coverage, 3 open High defects)"
+
+---
+
+## ISO 29119-3 Alignment
+
+- Use **Test Status Report** for periodic updates; **Test Completion Report** for release
+- Include all mandatory sections per `references/iso-29119-reports.md`
+- Tailor for Agile: shorter cycles, lighter documentation, focus on working software
+- Traceability: link tests to requirements, defects to test cases
+
+---
+
+## Tool Integration
+
+| Tool | Best Practice |
+| ---- | ------------- |
+| **CI/CD** | Publish reports as artifacts; parse in post-job step |
+| **GitHub Actions** | Use `actions/upload-artifact` for JUnit/JSON; fetch via GitHub MCP |
+| **Allure** | Rich metadata; use labels for suite, priority, epic |
+| **Coverage** | Merge with test results; include in same report |
+| **Memory MCP** | Store trends; query before generating report |

package/skills/qa-test-reporter/references/iso-29119-reports.md

@@ -0,0 +1,236 @@
+# ISO 29119-3 Test Status and Completion Report Templates
+
+*Per ISO/IEC/IEEE 29119-3:2021 — Software and systems engineering — Software testing — Part 3: Test documentation*
+
+---
+
+## Test Status Report
+
+**Purpose:** Periodic progress report during test execution. Use for sprint reviews, standups, or milestone checkpoints.
+
+### Template
+
+```markdown
+# Test Status Report
+
+**Identifier:** TSR-{YYYYMMDD}-{sequence}
+**Reporting Period:** {start date} to {end date}
+**Report Date:** {date}
+**Project/Release:** {name}
+
+---
+
+## 1. Summary
+
+{Brief executive summary: overall progress, key achievements, blockers.}
+
+---
+
+## 2. Test Execution Summary
+
+| Metric | Planned | Executed | Passed | Failed | Blocked | Skipped | Pass Rate |
+| ------ | ------- | -------- | ------ | ------ | ------- | ------- | --------- |
+| Total  | {n}     | {n}      | {n}    | {n}    | {n}     | {n}     | {pct}%    |
+
+*Breakdown by level/type if applicable:*
+
+| Category     | Passed | Failed | Skipped | Pass Rate |
+| ------------ | ------ | ------ | ------- | --------- |
+| Unit         | {n}    | {n}    | {n}     | {pct}%    |
+| Integration  | {n}    | {n}    | {n}     | {pct}%    |
+| E2E / System | {n}    | {n}    | {n}     | {pct}%    |
+
+---
+
+## 3. Incidents
+
+| Status | Count | Notes |
+| ------ | ----- | ----- |
+| Open   | {n}   | {High/Critical: n} |
+| Closed | {n}   | —     |
+| Deferred | {n} | —     |
+
+*Top open incidents (ID, summary, severity):*
+
+- TIR-001: {summary} — {severity}
+- TIR-002: {summary} — {severity}
+
+---
+
+## 4. Risks and Issues
+
+- {Risk/blocker 1}
+- {Risk/blocker 2}
+
+---
+
+## 5. Next Period Plan
+
+- {Planned activity 1}
+- {Planned activity 2}
+
+---
+
+## 6. Metrics (Optional)
+
+- **Coverage:** {line}% lines, {branch}% branches
+- **Progress:** {pct}% of planned tests executed
+- **Trend:** Pass rate vs. previous period: {up/down/stable}
+```
+
+### Mandatory Sections (ISO 29119-3)
+
+| Section | Mandatory | Description |
+| ------- | --------- | ----------- |
+| Identifier | ✓ | Unique report ID |
+| Reporting period | ✓ | Date range covered |
+| Summary | ✓ | Executive summary |
+| Test execution summary | ✓ | Pass/fail/blocked counts |
+| Incidents | ✓ | Open/closed incident summary |
+| Risks and issues | | Blockers, risks |
+| Next period plan | | Planned activities |
+| Metrics | | Coverage, progress % |
+
+---
+
+## Test Completion Report
+
+**Purpose:** Final report when testing is complete. Summarizes outcomes, exit criteria, and release readiness.
+
+### Template
+
+```markdown
+# Test Completion Report
+
+**Identifier:** TCR-{YYYYMMDD}-{sequence}
+**Report Date:** {date}
+**Project/Release:** {name}
+**Testing Period:** {start date} to {end date}
+
+---
+
+## 1. Summary
+
+{Executive summary: testing complete, overall outcome, go/no-go recommendation.}
+
+---
+
+## 2. Test Execution Summary
+
+| Metric | Total | Passed | Failed | Blocked | Skipped | Pass Rate |
+| ------ | ----- | ------ | ------ | ------- | ------- | --------- |
+| Count  | {n}   | {n}    | {n}    | {n}     | {n}     | {pct}%    |
+
+**Duration:** {total hours} hours
+
+*By category:*
+
+| Category     | Passed | Failed | Skipped | Pass Rate |
+| ------------ | ------ | ------ | ------- | --------- |
+| Unit         | {n}    | {n}    | {n}     | {pct}%    |
+| Integration  | {n}    | {n}    | {n}     | {pct}%    |
+| E2E / System | {n}    | {n}    | {n}     | {pct}%    |
+
+---
+
+## 3. Exit Criteria
+
+| Criterion | Target | Actual | Met? |
+| --------- | ------ | ------ | ---- |
+| Pass rate ≥ X% | {target}% | {actual}% | Yes/No |
+| No Critical/High open | 0 | {n} | Yes/No |
+| Coverage ≥ Y% | {target}% | {actual}% | Yes/No |
+| {Custom criterion} | — | — | Yes/No |
+
+**Overall:** {All criteria met / Criteria not met}
+
+---
+
+## 4. Incidents Summary
+
+| Status | Count |
+| ------ | ----- |
+| Open   | {n}   |
+| Closed | {n}   |
+| Deferred | {n} |
+
+*Open defects by severity:*
+
+- Critical: {n}
+- High: {n}
+- Medium: {n}
+- Low: {n}
+
+---
+
+## 5. Recommendations
+
+**Release Readiness:** {Go / No-Go / Conditional}
+
+**Rationale:**
+- {Point 1}
+- {Point 2}
+
+**Follow-up:**
+- {Action 1}
+- {Action 2}
+
+---
+
+## 6. Lessons Learned (Optional)
+
+- {Improvement 1}
+- {Improvement 2}
+
+---
+
+## 7. Metrics (Optional)
+
+- **Coverage:** {line}% lines, {branch}% branches
+- **Defect density:** {defects per KLOC or per test}
+- **Flaky tests:** {count} identified
+```
+
+### Mandatory Sections (ISO 29119-3)
+
+| Section | Mandatory | Description |
+| ------- | --------- | ----------- |
+| Identifier | ✓ | Unique report ID |
+| Summary | ✓ | Executive summary |
+| Test execution summary | ✓ | Final pass/fail/blocked counts |
+| Exit criteria met | ✓ | Whether criteria were satisfied |
+| Incidents summary | ✓ | Open, closed, deferred |
+| Recommendations | ✓ | Release readiness, follow-up |
+| Lessons learned | | Process improvements |
+| Metrics | | Coverage, defect density |
+
+---
+
+## Go/No-Go Recommendation Format
+
+Use in both Status and Completion reports, and in executive summaries:
+
+```markdown
+## Go/No-Go Recommendation
+
+**Recommendation:** {Go | No-Go | Conditional}
+
+**Summary:**
+- Pass rate: {pct}%
+- Open Critical/High: {n}
+- Exit criteria: {met / not met}
+- Risk areas: {list}
+
+**Rationale:** {2-3 sentence justification}
+```
+
+### Decision Matrix (Example)
+
+| Condition | Go | Conditional | No-Go |
+| --------- | --- | ----------- | ----- |
+| Pass rate | ≥ 95% | 90–95% | < 90% |
+| Critical open | 0 | 0 | > 0 |
+| High open | 0 | 1–2 (documented) | > 2 |
+| Exit criteria | All met | Most met, minor gaps | Not met |
+
+*Tailor thresholds per project.*

package/skills/qa-test-reporter/references/report-formats.md

@@ -0,0 +1,287 @@
+# Report Format Reference
+
+*Reference for parsing test results from common frameworks and CI systems.*
+
+---
+
+## JUnit XML
+
+Widely supported by Jest, pytest, Maven, Gradle, PHPUnit, and most CI runners.
+
+### Schema Overview
+
+```xml
+<testsuites>
+  <testsuite name="Suite Name" tests="10" failures="1" skipped="2" errors="0" time="5.23">
+    <testcase name="test_name" classname="module.Class" time="0.5">
+      <!-- Pass: no child elements -->
+    </testcase>
+    <testcase name="failing_test" classname="module.Class" time="0.1">
+      <failure message="AssertionError">Stack trace...</failure>
+    </testcase>
+    <testcase name="skipped_test" classname="module.Class">
+      <skipped message="Not implemented"/>
+    </testcase>
+    <testcase name="error_test" classname="module.Class">
+      <error message="RuntimeError">Error details...</error>
+    </testcase>
+  </testsuite>
+</testsuites>
+```
+
+### Key Attributes
+
+| Element | Attribute | Description |
+| ------- | --------- | ----------- |
+| `testsuites` | — | Root; may have `name`, `tests`, `failures`, `errors`, `time` |
+| `testsuite` | `name` | Suite/module name |
+| `testsuite` | `tests` | Total test count |
+| `testsuite` | `failures` | Failed count |
+| `testsuite` | `skipped` | Skipped count |
+| `testsuite` | `errors` | Error count (uncaught exceptions) |
+| `testsuite` | `time` | Total duration (seconds) |
+| `testcase` | `name` | Test name |
+| `testcase` | `classname` | Class/module (often `file.Class` or `path`) |
+| `testcase` | `time` | Duration (seconds) |
+| `failure` / `error` | `message` | Short error message |
+| `failure` / `error` | (text) | Full stack trace |
+| `skipped` | `message` | Skip reason |
+
+### Framework Variations
+
+| Framework | Notes |
+| --------- | ----- |
+| **Jest** | `testsuites` → `testsuite` per file; `classname` = file path |
+| **pytest** | `classname` = `ClassName` or module; `file` attribute sometimes present |
+| **Vitest** | JUnit-compatible; similar to Jest |
+| **Playwright** | JUnit reporter outputs per project/shard |
+| **Maven/Gradle** | Standard JUnit 4/5 schema |
+
+### Parsing Rules
+
+1. Treat `failure` and `error` as **failed**.
+2. `skipped` = **skipped**; absence of failure/error/skipped = **passed**.
+3. Use `classname` + `name` for unique ID; `classname` often maps to suite/module.
+4. Aggregate `testsuite`-level counts if `testcase` children are present; prefer sum of children for accuracy.
+
+---
+
+## JSON Reports
+
+Format varies by framework. Common patterns:
+
+### Vitest JSON
+
+```json
+{
+  "result": {
+    "startTime": 1234567890,
+    "totalDuration": 5000,
+    "numTotalTestSuites": 5,
+    "numTotalTests": 50,
+    "numPassedTests": 45,
+    "numFailedTests": 3,
+    "numSkippedTests": 2,
+    "testResults": [
+      {
+        "name": "suite.spec.ts",
+        "assertionResults": [
+          {
+            "ancestorTitles": ["Suite"],
+            "title": "test name",
+            "status": "passed",
+            "duration": 100
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+### Playwright JSON
+
+```json
+{
+  "config": { "projects": [...] },
+  "suites": [
+    {
+      "title": "Suite",
+      "specs": [
+        {
+          "title": "test name",
+          "tests": [
+            {
+              "results": [
+                {
+                  "status": "passed",
+                  "duration": 100
+                }
+              ]
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}
+```
+
+### Generic Aggregated JSON (Suggested)
+
+When generating or normalizing:
+
+```json
+{
+  "summary": {
+    "total": 50,
+    "passed": 45,
+    "failed": 3,
+    "skipped": 2,
+    "duration": 5000
+  },
+  "suites": [
+    {
+      "name": "Suite",
+      "tests": [
+        {
+          "name": "test name",
+          "result": "passed",
+          "duration": 100,
+          "failureMessage": null
+        }
+      ]
+    }
+  ]
+}
+```
+
+---
+
+## Allure Results
+
+Allure uses a directory of JSON files.
+
+### result.json (per test)
+
+```json
+{
+  "uuid": "abc-123",
+  "name": "Test name",
+  "fullName": "Suite.Test name",
+  "status": "passed",
+  "stage": "finished",
+  "start": 1234567890000,
+  "stop": 1234567890100,
+  "labels": [
+    { "name": "suite", "value": "Suite" },
+    { "name": "priority", "value": "critical" }
+  ],
+  "steps": [...],
+  "attachments": [...]
+}
+```
+
+**Status values:** `passed`, `failed`, `broken`, `skipped`, `unknown`
+
+### container.json (per suite/container)
+
+```json
+{
+  "uuid": "container-uuid",
+  "children": ["test-uuid-1", "test-uuid-2"],
+  "befores": [...],
+  "afters": [...]
+}
+```
+
+### Parsing Rules
+
+1. Read all `*-result.json` and `*-container.json` in the results directory.
+2. Build hierarchy from `container.json` → `children` → `result.json`.
+3. Map `status`: `passed`→pass, `failed`/`broken`→fail, `skipped`→skip.
+4. Use `labels` for suite, priority, epic, feature when present.
+
+---
+
+## Coverage JSON
+
+### Istanbul / c8 (Node.js)
+
+```json
+{
+  "total": {
+    "lines": { "total": 100, "covered": 80, "pct": 80 },
+    "statements": { "total": 120, "covered": 95, "pct": 79.17 },
+    "functions": { "total": 20, "covered": 18, "pct": 90 },
+    "branches": { "total": 40, "covered": 30, "pct": 75 }
+  }
+}
+```
+
+### pytest-cov JSON
+
+```json
+{
+  "totals": {
+    "num_statements": 100,
+    "covered_lines": 80,
+    "percent_covered": 80.0,
+    "missing_lines": 20
+  }
+}
+```
+
+### JaCoCo XML (common in Java)
+
+```xml
+<report>
+  <counter type="LINE" missed="20" covered="80"/>
+  <counter type="BRANCH" missed="10" covered="30"/>
+</report>
+```
+
+### Parsing Rules
+
+1. Normalize to: `lines`, `branches`, `functions` (or `statements`) with `covered` and `total` or `pct`.
+2. If only one metric (e.g., `lines`), use that for coverage summary.
+3. Flag files or packages below threshold (e.g., &lt; 70%) as risk areas.
+
+---
+
+## CI/CD Output
+
+### GitHub Actions
+
+- **Workflow runs:** `GET /repos/{owner}/{repo}/actions/runs`
+- **Job logs:** `GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs`
+- **Artifacts:** `GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts`
+
+Artifacts often contain JUnit XML or JSON. Download artifact ZIP, extract, then parse.
+
+### GitLab CI
+
+- JUnit report artifact path configurable (e.g., `junit: report.xml`)
+- Coverage from `coverage` keyword or regex in job log
+
+### Jenkins
+
+- JUnit plugin publishes from `**/target/surefire-reports/*.xml` or similar
+- Coverage from JaCoCo, Cobertura plugins
+
+---
+
+## Normalized Internal Model
+
+Use a common structure for aggregation:
+
+| Field | Type | Source |
+| ----- | ---- | ------ |
+| `id` | string | `classname` + `name` or `fullName` |
+| `name` | string | Test name |
+| `suite` | string | Suite/module/class |
+| `result` | enum | `passed`, `failed`, `skipped`, `blocked` |
+| `duration` | number | ms or seconds (normalize to ms) |
+| `failureMessage` | string? | From `failure`/`error` message |
+| `stackTrace` | string? | Full trace if available |
+| `labels` | object? | `priority`, `type`, `epic`, etc. |