qa-skills 3.0.0 → 3.1.0

package/skills/qa-test-reporter/references/best-practices.md

@@ -1,162 +0,0 @@
-# Test Reporting Best Practices
-
-*Guidance for meaningful metrics, trend analysis, and stakeholder communication.*
-
----
-
-## Meaningful Metrics
-
-### Primary Metrics
-
-| Metric | Use | Avoid |
-| ------ | --- | ----- |
-| **Pass rate** | % passed / total executed | Excluding skipped inflates rate; document skip reasons |
-| **Failure count** | Absolute and trend | Ignoring flaky tests masks real regressions |
-| **Duration** | Trend over time; SLA compliance | Single-run duration without context |
-| **Coverage** | Risk areas; uncovered code | Coverage as sole quality gate |
-
-### Secondary Metrics
-
-- **Flaky rate** — Tests that pass/fail inconsistently; prioritize stabilization
-- **Failure density** — Failures per module/suite; identify hotspots
-- **Blocked count** — Tests blocked by environment or defects
-- **Defect escape rate** — Production defects / total defects found (post-release)
-
-### Anti-Patterns
-
-1. **Vanity metrics** — High pass rate with many skipped tests
-2. **Single-number focus** — Pass rate alone without failure analysis
-3. **Ignoring trends** — One bad run vs. sustained regression
-4. **Coverage as proxy for quality** — 100% coverage with weak assertions
-
----
-
-## Trend Analysis
-
-### What to Track
-
-| Data Point | Purpose |
-| ---------- | ------- |
-| Pass rate over last N runs | Detect regression |
-| Failure count trend | New vs. recurring failures |
-| Duration trend | Performance degradation |
-| Flaky test list | Prioritize stabilization |
-| Coverage trend | Coverage growth or regression |
-
-### Storage (Memory MCP)
-
-Store per run:
-
-```json
-{
-  "runId": "build-123",
-  "timestamp": "2025-03-07T10:00:00Z",
-  "total": 100,
-  "passed": 92,
-  "failed": 5,
-  "skipped": 3,
-  "duration": 120000,
-  "passRate": 92
-}
-```
-
-Query for:
-
-- Last 7 days
-- Last 10 builds
-- Same branch/commit family
-
-### Reporting Trends
-
-- **Improving** — Pass rate up, failures down
-- **Stable** — No significant change
-- **Regressing** — Pass rate down, failures up; flag for investigation
-- **New baseline** — First run or major scope change; avoid over-interpretation
-
----
-
-## Stakeholder Communication
-
-### Audience Tiers
-
-| Audience | Focus | Detail Level |
-| -------- | ----- | ------------- |
-| **Executive** | Go/no-go, risk, timeline | 1-page summary |
-| **Project manager** | Progress, blockers, next steps | Status report |
-| **Developers** | Failures, flaky tests, top issues | Failure analysis, links to failures |
-| **QA** | Full metrics, trends, coverage | Full dashboard |
-
-### Report Cadence
-
-- **Daily/Sprint** — Status report (progress, blockers)
-- **Release** — Completion report (go/no-go, recommendations)
-- **Ad-hoc** — HTML dashboard, Markdown summary for PRs
-
-### Clarity Principles
-
-1. **Lead with the answer** — Go/no-go and key metrics first
-2. **Use plain language** — Avoid jargon for non-technical readers
-3. **Provide context** — "Pass rate 92%, up from 88% last week"
-4. **Actionable** — "5 flaky tests in checkout flow — stabilize before release"
-5. **Traceability** — Link failures to test IDs, defects, requirements
-
----
-
-## Failure Analysis
-
-### Top Failures
-
-Rank failures by:
-
-1. **Frequency** — Same test failing across runs
-2. **Impact** — Critical path, high-priority area
-3. **Recency** — New failures vs. long-standing
-
-### Flaky Detection
-
-- Same test: pass and fail across runs without code change
-- Require minimum runs (e.g., 5+) before flagging
-- Report: "Flaky (3/10 runs failed)"
-
-### Failure Grouping
-
-Group by:
-
-- **Module/suite** — Identify weak areas
-- **Error type** — Timeout, assertion, environment
-- **Root cause** — When known, tag for reporting
-
----
-
-## Risk Areas
-
-Flag when:
-
-- **Low coverage** — Module/suite below threshold (e.g., < 70%)
-- **High failure density** — Many failures in one area
-- **Blocked tests** — Cannot execute; blocks exit criteria
-- **Open Critical/High** — Unresolved defects in scope
-- **Duration spike** — Significant slowdown vs. baseline
-
-Present in report: "Risk areas: Checkout module (65% coverage, 3 open High defects)"
-
----
-
-## ISO 29119-3 Alignment
-
-- Use **Test Status Report** for periodic updates; **Test Completion Report** for release
-- Include all mandatory sections per `references/iso-29119-reports.md`
-- Tailor for Agile: shorter cycles, lighter documentation, focus on working software
-- Traceability: link tests to requirements, defects to test cases
-
----
-
-## Tool Integration
-
-| Tool | Best Practice |
-| ---- | ------------- |
-| **CI/CD** | Publish reports as artifacts; parse in post-job step |
-| **GitHub Actions** | Use `actions/upload-artifact` for JUnit/JSON; fetch via GitHub MCP |
-| **Allure** | Rich metadata; use labels for suite, priority, epic |
-| **Coverage** | Merge with test results; include in same report |
-| **Memory MCP** | Store trends; query before generating report |

package/skills/qa-test-reporter/references/iso-29119-reports.md

@@ -1,236 +0,0 @@
-# ISO 29119-3 Test Status and Completion Report Templates
-
-*Per ISO/IEC/IEEE 29119-3:2021 — Software and systems engineering — Software testing — Part 3: Test documentation*
-
----
-
-## Test Status Report
-
-**Purpose:** Periodic progress report during test execution. Use for sprint reviews, standups, or milestone checkpoints.
-
-### Template
-
-```markdown
-# Test Status Report
-
-**Identifier:** TSR-{YYYYMMDD}-{sequence}
-**Reporting Period:** {start date} to {end date}
-**Report Date:** {date}
-**Project/Release:** {name}
-
----
-
-## 1. Summary
-
-{Brief executive summary: overall progress, key achievements, blockers.}
-
----
-
-## 2. Test Execution Summary
-
-| Metric | Planned | Executed | Passed | Failed | Blocked | Skipped | Pass Rate |
-| ------ | ------- | -------- | ------ | ------ | ------- | ------- | --------- |
-| Total  | {n}     | {n}      | {n}    | {n}    | {n}     | {n}     | {pct}%    |
-
-*Breakdown by level/type if applicable:*
-
-| Category     | Passed | Failed | Skipped | Pass Rate |
-| ------------ | ------ | ------ | ------- | --------- |
-| Unit         | {n}    | {n}    | {n}     | {pct}%    |
-| Integration  | {n}    | {n}    | {n}     | {pct}%    |
-| E2E / System | {n}    | {n}    | {n}     | {pct}%    |
-
----
-
-## 3. Incidents
-
-| Status | Count | Notes |
-| ------ | ----- | ----- |
-| Open   | {n}   | {High/Critical: n} |
-| Closed | {n}   | —     |
-| Deferred | {n} | —     |
-
-*Top open incidents (ID, summary, severity):*
-
-- TIR-001: {summary} — {severity}
-- TIR-002: {summary} — {severity}
-
----
-
-## 4. Risks and Issues
-
-- {Risk/blocker 1}
-- {Risk/blocker 2}
-
----
-
-## 5. Next Period Plan
-
-- {Planned activity 1}
-- {Planned activity 2}
-
----
-
-## 6. Metrics (Optional)
-
-- **Coverage:** {line}% lines, {branch}% branches
-- **Progress:** {pct}% of planned tests executed
-- **Trend:** Pass rate vs. previous period: {up/down/stable}
-```
-
-### Mandatory Sections (ISO 29119-3)
-
-| Section | Mandatory | Description |
-| ------- | --------- | ----------- |
-| Identifier | ✓ | Unique report ID |
-| Reporting period | ✓ | Date range covered |
-| Summary | ✓ | Executive summary |
-| Test execution summary | ✓ | Pass/fail/blocked counts |
-| Incidents | ✓ | Open/closed incident summary |
-| Risks and issues | | Blockers, risks |
-| Next period plan | | Planned activities |
-| Metrics | | Coverage, progress % |
-
----
-
-## Test Completion Report
-
-**Purpose:** Final report when testing is complete. Summarizes outcomes, exit criteria, and release readiness.
-
-### Template
-
-```markdown
-# Test Completion Report
-
-**Identifier:** TCR-{YYYYMMDD}-{sequence}
-**Report Date:** {date}
-**Project/Release:** {name}
-**Testing Period:** {start date} to {end date}
-
----
-
-## 1. Summary
-
-{Executive summary: testing complete, overall outcome, go/no-go recommendation.}
-
----
-
-## 2. Test Execution Summary
-
-| Metric | Total | Passed | Failed | Blocked | Skipped | Pass Rate |
-| ------ | ----- | ------ | ------ | ------- | ------- | --------- |
-| Count  | {n}   | {n}    | {n}    | {n}     | {n}     | {pct}%    |
-
-**Duration:** {total hours} hours
-
-*By category:*
-
-| Category     | Passed | Failed | Skipped | Pass Rate |
-| ------------ | ------ | ------ | ------- | --------- |
-| Unit         | {n}    | {n}    | {n}     | {pct}%    |
-| Integration  | {n}    | {n}    | {n}     | {pct}%    |
-| E2E / System | {n}    | {n}    | {n}     | {pct}%    |
-
----
-
-## 3. Exit Criteria
-
-| Criterion | Target | Actual | Met? |
-| --------- | ------ | ------ | ---- |
-| Pass rate ≥ X% | {target}% | {actual}% | Yes/No |
-| No Critical/High open | 0 | {n} | Yes/No |
-| Coverage ≥ Y% | {target}% | {actual}% | Yes/No |
-| {Custom criterion} | — | — | Yes/No |
-
-**Overall:** {All criteria met / Criteria not met}
-
----
-
-## 4. Incidents Summary
-
-| Status | Count |
-| ------ | ----- |
-| Open   | {n}   |
-| Closed | {n}   |
-| Deferred | {n} |
-
-*Open defects by severity:*
-
-- Critical: {n}
-- High: {n}
-- Medium: {n}
-- Low: {n}
-
----
-
-## 5. Recommendations
-
-**Release Readiness:** {Go / No-Go / Conditional}
-
-**Rationale:**
-- {Point 1}
-- {Point 2}
-
-**Follow-up:**
-- {Action 1}
-- {Action 2}
-
----
-
-## 6. Lessons Learned (Optional)
-
-- {Improvement 1}
-- {Improvement 2}
-
----
-
-## 7. Metrics (Optional)
-
-- **Coverage:** {line}% lines, {branch}% branches
-- **Defect density:** {defects per KLOC or per test}
-- **Flaky tests:** {count} identified
-```
-
-### Mandatory Sections (ISO 29119-3)
-
-| Section | Mandatory | Description |
-| ------- | --------- | ----------- |
-| Identifier | ✓ | Unique report ID |
-| Summary | ✓ | Executive summary |
-| Test execution summary | ✓ | Final pass/fail/blocked counts |
-| Exit criteria met | ✓ | Whether criteria were satisfied |
-| Incidents summary | ✓ | Open, closed, deferred |
-| Recommendations | ✓ | Release readiness, follow-up |
-| Lessons learned | | Process improvements |
-| Metrics | | Coverage, defect density |
-
----
-
-## Go/No-Go Recommendation Format
-
-Use in both Status and Completion reports, and in executive summaries:
-
-```markdown
-## Go/No-Go Recommendation
-
-**Recommendation:** {Go | No-Go | Conditional}
-
-**Summary:**
-- Pass rate: {pct}%
-- Open Critical/High: {n}
-- Exit criteria: {met / not met}
-- Risk areas: {list}
-
-**Rationale:** {2-3 sentence justification}
-```
-
-### Decision Matrix (Example)
-
-| Condition | Go | Conditional | No-Go |
-| --------- | --- | ----------- | ----- |
-| Pass rate | ≥ 95% | 90–95% | < 90% |
-| Critical open | 0 | 0 | > 0 |
-| High open | 0 | 1–2 (documented) | > 2 |
-| Exit criteria | All met | Most met, minor gaps | Not met |
-
-*Tailor thresholds per project.*

package/skills/qa-test-reporter/references/report-formats.md

@@ -1,287 +0,0 @@
-# Report Format Reference
-
-*Reference for parsing test results from common frameworks and CI systems.*
-
----
-
-## JUnit XML
-
-Widely supported by Jest, pytest, Maven, Gradle, PHPUnit, and most CI runners.
-
-### Schema Overview
-
-```xml
-<testsuites>
-  <testsuite name="Suite Name" tests="10" failures="1" skipped="2" errors="0" time="5.23">
-    <testcase name="test_name" classname="module.Class" time="0.5">
-      <!-- Pass: no child elements -->
-    </testcase>
-    <testcase name="failing_test" classname="module.Class" time="0.1">
-      <failure message="AssertionError">Stack trace...</failure>
-    </testcase>
-    <testcase name="skipped_test" classname="module.Class">
-      <skipped message="Not implemented"/>
-    </testcase>
-    <testcase name="error_test" classname="module.Class">
-      <error message="RuntimeError">Error details...</error>
-    </testcase>
-  </testsuite>
-</testsuites>
-```
-
-### Key Attributes
-
-| Element | Attribute | Description |
-| ------- | --------- | ----------- |
-| `testsuites` | — | Root; may have `name`, `tests`, `failures`, `errors`, `time` |
-| `testsuite` | `name` | Suite/module name |
-| `testsuite` | `tests` | Total test count |
-| `testsuite` | `failures` | Failed count |
-| `testsuite` | `skipped` | Skipped count |
-| `testsuite` | `errors` | Error count (uncaught exceptions) |
-| `testsuite` | `time` | Total duration (seconds) |
-| `testcase` | `name` | Test name |
-| `testcase` | `classname` | Class/module (often `file.Class` or `path`) |
-| `testcase` | `time` | Duration (seconds) |
-| `failure` / `error` | `message` | Short error message |
-| `failure` / `error` | (text) | Full stack trace |
-| `skipped` | `message` | Skip reason |
-
-### Framework Variations
-
-| Framework | Notes |
-| --------- | ----- |
-| **Jest** | `testsuites` → `testsuite` per file; `classname` = file path |
-| **pytest** | `classname` = `ClassName` or module; `file` attribute sometimes present |
-| **Vitest** | JUnit-compatible; similar to Jest |
-| **Playwright** | JUnit reporter outputs per project/shard |
-| **Maven/Gradle** | Standard JUnit 4/5 schema |
-
-### Parsing Rules
-
-1. Treat `failure` and `error` as **failed**.
-2. `skipped` = **skipped**; absence of failure/error/skipped = **passed**.
-3. Use `classname` + `name` for unique ID; `classname` often maps to suite/module.
-4. Aggregate `testsuite`-level counts if `testcase` children are present; prefer sum of children for accuracy.
-
----
-
-## JSON Reports
-
-Format varies by framework. Common patterns:
-
-### Vitest JSON
-
-```json
-{
-  "result": {
-    "startTime": 1234567890,
-    "totalDuration": 5000,
-    "numTotalTestSuites": 5,
-    "numTotalTests": 50,
-    "numPassedTests": 45,
-    "numFailedTests": 3,
-    "numSkippedTests": 2,
-    "testResults": [
-      {
-        "name": "suite.spec.ts",
-        "assertionResults": [
-          {
-            "ancestorTitles": ["Suite"],
-            "title": "test name",
-            "status": "passed",
-            "duration": 100
-          }
-        ]
-      }
-    ]
-  }
-}
-```
-
-### Playwright JSON
-
-```json
-{
-  "config": { "projects": [...] },
-  "suites": [
-    {
-      "title": "Suite",
-      "specs": [
-        {
-          "title": "test name",
-          "tests": [
-            {
-              "results": [
-                {
-                  "status": "passed",
-                  "duration": 100
-                }
-              ]
-            }
-          ]
-        }
-      ]
-    }
-  ]
-}
-```
-
-### Generic Aggregated JSON (Suggested)
-
-When generating or normalizing:
-
-```json
-{
-  "summary": {
-    "total": 50,
-    "passed": 45,
-    "failed": 3,
-    "skipped": 2,
-    "duration": 5000
-  },
-  "suites": [
-    {
-      "name": "Suite",
-      "tests": [
-        {
-          "name": "test name",
-          "result": "passed",
-          "duration": 100,
-          "failureMessage": null
-        }
-      ]
-    }
-  ]
-}
-```
-
----
-
-## Allure Results
-
-Allure uses a directory of JSON files.
-
-### result.json (per test)
-
-```json
-{
-  "uuid": "abc-123",
-  "name": "Test name",
-  "fullName": "Suite.Test name",
-  "status": "passed",
-  "stage": "finished",
-  "start": 1234567890000,
-  "stop": 1234567890100,
-  "labels": [
-    { "name": "suite", "value": "Suite" },
-    { "name": "priority", "value": "critical" }
-  ],
-  "steps": [...],
-  "attachments": [...]
-}
-```
-
-**Status values:** `passed`, `failed`, `broken`, `skipped`, `unknown`
-
-### container.json (per suite/container)
-
-```json
-{
-  "uuid": "container-uuid",
-  "children": ["test-uuid-1", "test-uuid-2"],
-  "befores": [...],
-  "afters": [...]
-}
-```
-
-### Parsing Rules
-
-1. Read all `*-result.json` and `*-container.json` in the results directory.
-2. Build hierarchy from `container.json` → `children` → `result.json`.
-3. Map `status`: `passed`→pass, `failed`/`broken`→fail, `skipped`→skip.
-4. Use `labels` for suite, priority, epic, feature when present.
-
----
-
-## Coverage JSON
-
-### Istanbul / c8 (Node.js)
-
-```json
-{
-  "total": {
-    "lines": { "total": 100, "covered": 80, "pct": 80 },
-    "statements": { "total": 120, "covered": 95, "pct": 79.17 },
-    "functions": { "total": 20, "covered": 18, "pct": 90 },
-    "branches": { "total": 40, "covered": 30, "pct": 75 }
-  }
-}
-```
-
-### pytest-cov JSON
-
-```json
-{
-  "totals": {
-    "num_statements": 100,
-    "covered_lines": 80,
-    "percent_covered": 80.0,
-    "missing_lines": 20
-  }
-}
-```
-
-### JaCoCo XML (common in Java)
-
-```xml
-<report>
-  <counter type="LINE" missed="20" covered="80"/>
-  <counter type="BRANCH" missed="10" covered="30"/>
-</report>
-```
-
-### Parsing Rules
-
-1. Normalize to: `lines`, `branches`, `functions` (or `statements`) with `covered` and `total` or `pct`.
-2. If only one metric (e.g., `lines`), use that for coverage summary.
-3. Flag files or packages below threshold (e.g., &lt; 70%) as risk areas.
-
----
-
-## CI/CD Output
-
-### GitHub Actions
-
-- **Workflow runs:** `GET /repos/{owner}/{repo}/actions/runs`
-- **Job logs:** `GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs`
-- **Artifacts:** `GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts`
-
-Artifacts often contain JUnit XML or JSON. Download artifact ZIP, extract, then parse.
-
-### GitLab CI
-
-- JUnit report artifact path configurable (e.g., `junit: report.xml`)
-- Coverage from `coverage` keyword or regex in job log
-
-### Jenkins
-
-- JUnit plugin publishes from `**/target/surefire-reports/*.xml` or similar
-- Coverage from JaCoCo, Cobertura plugins
-
----
-
-## Normalized Internal Model
-
-Use a common structure for aggregation:
-
-| Field | Type | Source |
-| ----- | ---- | ------ |
-| `id` | string | `classname` + `name` or `fullName` |
-| `name` | string | Test name |
-| `suite` | string | Suite/module/class |
-| `result` | enum | `passed`, `failed`, `skipped`, `blocked` |
-| `duration` | number | ms or seconds (normalize to ms) |
-| `failureMessage` | string? | From `failure`/`error` message |
-| `stackTrace` | string? | Full trace if available |
-| `labels` | object? | `priority`, `type`, `epic`, etc. |