@keber/qa-framework 1.3.0-0 → 1.4.0

package/skills/qa-spec-generation/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: qa-spec-generation
+description: >
+  Stage 2 of the QA pipeline. Generates or updates the 6-file specification set
+  (00-inventory, 01-business-rules, 02-workflows, 03-roles-permissions, 04-test-data,
+  05-test-scenarios) for a module submodule. Use when asked to write specs, generate
+  test scenarios, document business rules, create a test case table, or produce the
+  QA specification for a feature. Requires completed module analysis (Stage 1).
+---
+
+# QA Skill: Specification Generation (Stage 2 of 6)
+
+**Stage**: 2 — Specification Generation  
+**Prerequisite**: `00-inventory.md` must exist (Stage 1 complete)  
+**Output**: Complete 6-file spec set under `qa/01-specifications/module-{name}/submodule-{name}/`  
+**Next stage**: Stage 3 — Test Plan (`qa-test-plan`)
+
+> **Pipeline rule**: Do not generate specs for features marked `PENDING-CODE` in the inventory.
+> Do not create TCs without having observed or been given the behavior being tested.
+
+---
+
+## Inputs Required
+
+1. Module exploration notes or existing `00-inventory.md`
+2. Module code from `qa/00-standards/naming-conventions.md`
+3. Submodule name and kebab-case directory name
+4. QA environment observations (what was actually seen in the UI)
+
+---
+
+## File Generation Rules
+
+Generate or update each file following the formats in `references/spec-file-formats.md`.
+
+### `00-inventory.md`
+- Header block: module/submodule codes, primary URL, status, last updated date
+- UI element table: name, type, required, notes (validation rule)
+- API endpoint table: method, path, purpose
+- **Never include**: credentials, personal data, real user IDs
+
+### `01-business-rules.md`
+- Format: `RN-{MODULE}-{NNN}` with type, trigger, behavior, error message (exact UI text)
+- **Rule sources**: every required field → 1 RN; every unique constraint → 1 RN; every state transition → 1 RN; every role restriction → 1 RN
+- Minimum 5 rules per submodule; typical 8–15
+
+### `02-workflows.md`
+- Format: `FL-{MODULE}-{NNN}` with actor, trigger, precondition, ASCII/Mermaid flow diagram, postcondition
+- Must cover: primary happy path + rejection/cancellation + error path
+
+### `03-roles-permissions.md`
+- Role × feature access matrix: ✅ / ❌ / ⚠️ (limited)
+- Test user reference table: role name + env var names only (never actual credentials)
+
+### `04-test-data.md`
+- Prerequisites table (data that must exist before tests run)
+- Data shapes per key scenario (reference env vars for credentials)
+- EXEC_IDX pattern for unique dynamic values
+- Data isolation rules (each test operates on its own data)
+
+### `05-test-scenarios.md`
+- Summary table at top (total TCs by priority and type)
+- TC format: `TC-{MODULE}-{SUBMODULE}-{NNN}` with priority, type, origin, automation feasibility, preconditions, steps, expected result
+- **Target**: 50–85 TCs per submodule
+- **Mandatory coverage categories** (≥1 TC each where applicable):
+  - Access: unauthenticated user; role without permission
+  - Happy path: primary workflow end-to-end
+  - Negative: required field missing; duplicate; invalid format
+  - State transitions
+  - Export/download (if feature exists)
+  - Pagination/search (if feature exists)
+
+Full format templates: `references/spec-file-formats.md`
+
+---
+
+## Anti-Patterns to Avoid
+
+| Anti-pattern | Correct approach |
+|---|---|
+| TC for a feature not in QA env | Mark as `PENDING-CODE`; do not write test steps |
+| Real credentials in any TC step | Use env var references only |
+| "Verify system works correctly" as expected result | Write observable, measurable expected result |
+| 100+ TCs by copy-pasting near-identical variations | Merge cases; use parameterization |
+| TC written without observing the actual behavior | Mark as `PENDING-CODE` until verified |
+
+---
+
+## Outputs
+
+- 6 spec files per submodule in `qa/01-specifications/module-{name}/submodule-{name}/`
+- Module `README.md` updated with submodule TC count
+- `qa/README.md` module status updated

package/skills/qa-test-cases/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: qa-test-cases
+description: >
+  Stage 4 of the QA pipeline. Generates individual test case documents from
+  approved specifications and the test plan. Produces TC-{ID}.md files with
+  preconditions, steps, and expected results.
+  Use when asked to write test cases, create TCs, expand scenarios into steps,
+  document test procedures, or generate TC-{ID} files.
+  Requires existing specs (Stage 2) and, ideally, a test plan (Stage 3).
+---
+
+# QA Skill: Test Case Generation (Stage 4 of 6)
+
+**Stage**: 4 — Test Cases  
+**Prerequisite**: Submodule specs exist (Stage 2). Test plan is optional but strongly recommended.  
+**Output**: `qa/03-test-cases/automated/TC-{ID}.md` (one file per TC)  
+**Next stage**: Stage 5 — Automation (`qa-automation`)
+
+> **Pipeline rule**: TC IDs must be unique across the module. Never reuse a TC ID.
+
+---
+
+## Inputs Required
+
+1. `qa/01-specifications/{module}/05-test-scenarios.md` — source scenarios to expand
+2. Test plan (if available) — defines priority and which scenarios are in scope
+3. Module TC-ID range prefix (e.g., `PROV` → `TC-PROV-001`)
+
+---
+
+## Process
+
+### Step 1 — Audit existing TCs
+
+Check `qa/03-test-cases/automated/` for existing TC files in this module. Note:
+- Highest existing TC number (to continue sequence from next number)
+- Any `PENDING-CODE` TCs (skip automation loop; flag for manual follow-up)
+- Any TCs for scenarios already documented (do not duplicate)
+
+### Step 2 — Map scenarios to TC IDs
+
+For each scenario in `05-test-scenarios.md`:
+- Assign sequential TC ID: `TC-{PREFIX}-{NNN}` (zero-padded to 3 digits)
+- Annotate with priority and automation feasibility from test plan (if available)
+- Skip scenarios marked with `BLOCKED-PERMISSIONS` or `NOT-AUTOMATABLE` — generate TC shells only
+
+### Step 3 — Write TC documents
+
+Full template: `references/test-case-template.md`
+
+Each TC file must include:
+- **ID** and **Title**
+- **Priority** (P0/P1/P2/P3)
+- **Automation feasibility** (fully / partially / not)
+- **Preconditions** (setup state + test data reference)
+- **Steps** (numbered, action + input + expected result per step)
+- **Postconditions** (cleanup or state left after TC)
+- **Tags** (e.g., `smoke`, `regression`, `happy-path`)
+
+### Step 4 — Quality gates
+
+Before saving each TC:
+- Every step has an expected result — no steps with "verify it works"
+- Test data references a named fixture or `qa/01-specifications/{module}/04-test-data.md`
+- For CRUD TCs: steps cover create → verify → update → verify → delete → verify
+- Preconditions do not duplicate setup that belongs in a shared fixture
+
+### Step 5 — Update index
+
+Create or update `qa/03-test-cases/automated/{module}-index.md`:
+- Table: TC ID | Title | Priority | Feasibility | Spec reference
+
+---
+
+## Key Rules
+
+| Rule | Correct | Wrong |
+|---|---|---|
+| TC isolation | Each TC is independent | TCs depend on prior TC's data |
+| Step granularity | One action + one assertion per step | Multi-action steps |
+| Data hygiene | Named fixtures or spec data | Hardcoded values in steps |
+| ID stability | Never re-number existing IDs | Renaming breaks ADO links |
+
+---
+
+## Outputs
+
+- `qa/03-test-cases/automated/TC-{ID}.md` (one per scenario)
+- `qa/03-test-cases/automated/{module}-index.md`

package/skills/qa-test-cases/references/test-case-template.md

@@ -0,0 +1,142 @@
+# Reference: Test Case Template
+
+> Loaded by `skills/qa-test-cases/` when generating TC documents.
+
+---
+
+## Full TC Document Template
+
+```markdown
+# TC-{MODULE}-{SUBMODULE}-{NNN}: {TC Title}
+
+| Field | Value |
+|-------|-------|
+| TC ID | TC-{MODULE}-{SUBMODULE}-{NNN} |
+| Module | {Module display name} |
+| Submodule | {Submodule display name} |
+| Priority | P0 / P1 / P2 / P3 |
+| Type | Functional / Negative / Regression / Security / Integration |
+| Origin | UI-OBSERVED / PENDING-CODE / BLOCKED-PERMISSIONS |
+| Observation date | YYYY-MM-DD |
+| Automatable | Yes / Partial / No |
+| Playwright file | (fill when automation is written) |
+| Status | Draft / Approved / Automated / Deprecated |
+
+---
+
+## Objective
+
+{1-2 sentences: what this TC verifies and why it matters}
+
+---
+
+## Preconditions
+
+1. User is logged in as **{role}**
+2. The following data exists in the QA environment:
+   - {item 1}: {description}
+3. {any other system state requirement}
+
+---
+
+## Test Data
+
+| Field | Value |
+|-------|-------|
+| {field name} | {value or generation rule} |
+| {field name} | Use `EXEC_IDX`-generated value for uniqueness |
+| User | Credentials from `QA_USER_{ROLE}_EMAIL` env var |
+
+---
+
+## Steps
+
+| Step | Action | Expected result |
+|------|--------|----------------|
+| 1 | Navigate to {URL} | Page loads, {element} visible |
+| 2 | Click {element description} | {modal opens / form appears} |
+| 3 | Fill {field} with {value} | Field accepts input |
+| 4 | Click {submit button} | {success message / state change} |
+| 5 | Verify {observable outcome} | {specific text or element state} |
+
+---
+
+## Expected Results
+
+**Final state**: {what the system should look like after all steps complete}
+
+**Verification points**:
+- [ ] {element} should {state}
+- [ ] {element} should contain {text}
+- [ ] Page/modal should transition to {expected state}
+
+---
+
+## Known Issues / Related Defects
+
+| Issue | DEF-ID / ADO WI | Status | Impact on this TC |
+|-------|----------------|--------|------------------|
+| {description} | DEF-NNN | Open | TC skipped until resolved |
+
+---
+
+## Automation Notes
+
+```typescript
+// Key Playwright selector
+// page.locator('{selector}')
+
+// Key assertion
+// await expect(page.locator('{selector}')).toHaveText('{expected}');
+```
+
+---
+
+## Changelog
+
+| Version | Date | Author | Description |
+|---------|------|--------|-------------|
+| 1.0 | YYYY-MM-DD | {author} | Initial creation |
+```
+
+---
+
+## Module Index Template
+
+Create at: `qa/03-test-cases/automated/{module}-index.md`
+
+```markdown
+# TC Index — {Module Display Name}
+
+| TC ID | Title | Priority | Feasibility | Spec reference |
+|-------|-------|----------|-------------|---------------|
+| TC-{M}-{S}-001 | {title} | P0 | Yes | 05-test-scenarios.md#{anchor} |
+```
+
+---
+
+## Content Quality Rules
+
+### Measurable expected results
+
+❌ `The system should work correctly`  
+✅ `A toast message 'Record saved' appears and the datatable row count increases by 1`
+
+❌ `The form validates the input`  
+✅ `A red validation message 'This field is required' appears below the Name field`
+
+### Observable steps
+
+❌ `Add a new record with valid data`  
+✅ `Click 'New'. Fill 'Name' with 'Test-{EXEC_IDX}'. Click 'Save'.`
+
+### Defect links
+
+When skipping a step due to a bug:  
+✅ `Step 4 — SKIPPED: DEF-001 (ADO #21944). Switch defaults to OFF. Re-enable after fix.`
+
+### TC isolation
+
+- Each TC is fully independent — no TC depends on state from a prior TC
+- All required data is provisioned in preconditions or `beforeAll`
+- CRUD TCs cover: create → verify → update → verify → delete → verify

package/skills/qa-test-plan/SKILL.md

@@ -0,0 +1,81 @@
+---
+name: qa-test-plan
+description: >
+  Stage 3 of the QA pipeline. Generates a test plan document from completed
+  submodule specifications. Defines scope, priority distribution, automation
+  feasibility, ADO mapping, and TC selection for a sprint or release.
+  Use when asked to create a test plan, define test scope, plan a sprint,
+  prioritize test cases, or decide what to automate vs test manually.
+  Requires completed specs (Stage 2).
+---
+
+# QA Skill: Test Plan Generation (Stage 3 of 6)
+
+**Stage**: 3 — Test Plan  
+**Prerequisite**: `05-test-scenarios.md` must exist for all in-scope submodules (Stage 2 complete)  
+**Output**: `qa/02-test-plans/automated/{module}-test-plan.md`  
+**Next stage**: Stage 4 — Test Cases (`qa-test-cases`) or Stage 5 — Automation (`qa-automation`)
+
+> **Pipeline rule**: Do not automate TCs that are not in an approved or active test plan.
+
+---
+
+## Inputs Required
+
+1. `qa/01-specifications/module-{name}/` — all submodule specs
+2. Scope definition — which submodules are in this test plan (ask the user if not given)
+3. Priority guidance (from team or implied by sprint scope)
+
+---
+
+## Process
+
+### Step 1 — Collect TC inventory
+
+For each in-scope submodule, read `05-test-scenarios.md` and aggregate:
+- Total TC count by priority (P0/P1/P2/P3)
+- TC count by automation feasibility (fully / partially / not automatable)
+- Any TCs marked `BLOCKED-PERMISSIONS` or `PENDING-CODE` (these go in the exclusion list)
+
+### Step 2 — Apply priority rules
+
+Use the priority decision tree (see `references/priority-and-feasibility.md`) to assign or validate priorities. Key rules:
+
+- **P0**: primary happy path; show-stopper risk; critical data corruption
+- **P1**: common negative scenarios; cross-module dependencies; state machine transitions
+- **P2**: secondary features (export, pagination, search)
+- **P3**: edge cases, cosmetic, rare scenarios
+
+### Step 3 — Apply automation feasibility rules
+
+- **Fully automatable**: deterministic, observable via browser, no external system dependency
+- **Partially automatable**: some result requires human inspection OR external system can be mocked
+- **Not automatable**: physical access required; inherently non-deterministic; irreversible QA side effects; BLOCKED-PERMISSIONS
+
+### Step 4 — Write the test plan document
+
+Full document template: `references/test-plan-template.md`
+
+Required sections:
+1. Objective
+2. Scope (in/out)
+3. Automation feasibility analysis table (per submodule)
+4. Priority distribution table
+5. TC selection suites (P0 suite, P1 suite)
+6. Azure DevOps mapping (if ADO enabled)
+7. Test environment
+8. Risks and assumptions
+9. Excluded TCs (with reason)
+10. Execution schedule
+
+### Step 5 — Update status
+
+- Update `qa/README.md` module row (plan status → ✅)
+- If ADO enabled: stub the ADO Plan ID field for Step 6 (ADO integration)
+
+---
+
+## Outputs
+
+- `qa/02-test-plans/automated/{module}-test-plan.md`
+- `qa/README.md` plan status updated

package/skills/qa-test-plan/references/priority-and-feasibility.md

@@ -0,0 +1,159 @@
+# Reference: Test Plan Template
+
+> Loaded by `skills/qa-test-plan/` when generating a test plan document.
+
+---
+
+## Complete Test Plan Template
+
+Create at: `qa/02-test-plans/automated/{module}-test-plan.md`
+
+```markdown
+# Test Plan: {Module Display Name}
+
+| Field | Value |
+|-------|-------|
+| Version | 1.0 |
+| Date | YYYY-MM-DD |
+| Module | {module-code} |
+| Test plan type | Automated / Manual / Combined |
+| Author | {agent or human name} |
+| Status | Draft / Approved / Active |
+
+---
+
+## 1. Objective
+
+{1-2 sentences: what this test plan covers and why}
+
+---
+
+## 2. Scope
+
+### In scope
+- {submodule 1}: {what is being tested}
+
+### Out of scope
+- {what is explicitly excluded and why}
+
+---
+
+## 3. Automation Feasibility Analysis
+
+| Submodule | Total TCs | Fully automatable | Partially | Not automatable | Reason for exclusions |
+|-----------|-----------|------------------|-----------|-----------------|----------------------|
+| {name} | {N} | {N} ({%}) | {N} ({%}) | {N} ({%}) | {brief reason} |
+
+---
+
+## 4. Priority Distribution
+
+| Priority | TCs selected | Automated | Manual |
+|----------|-------------|-----------|--------|
+| P0 (Critical — every build) | {N} | {N} | {N} |
+| P1 (High — every release) | {N} | {N} | {N} |
+| P2 (Medium — periodic regression) | {N} | {N} | {N} |
+| P3 (Low — manual only) | {N} | 0 | {N} |
+
+---
+
+## 5. TC Selection and Automation Plan
+
+### P0 Suite
+
+| TC ID | Title | Type | Automation status |
+|-------|-------|------|------------------|
+| TC-{M}-{S}-001 | {title} | Functional | Queued |
+
+### P1 Suite
+
+| TC ID | Title | Type | Automation status |
+|-------|-------|------|------------------|
+| TC-{M}-{S}-010 | {title} | Functional | Queued |
+
+---
+
+## 6. Azure DevOps Test Plan (if enabled)
+
+| Item | Value |
+|------|-------|
+| ADO Plan ID | (populate after creation) |
+| ADO Suite IDs | (populate after creation) |
+| Reporter | @alex_neo/playwright-azure-reporter |
+
+---
+
+## 7. Test Environment
+
+| Item | Value |
+|------|-------|
+| QA URL | {{QA_BASE_URL}} |
+| Browser | Chromium (Playwright) |
+| Workers (local) | {N} |
+| Workers (CI) | {N} |
+
+---
+
+## 8. Risks and Assumptions
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|-----------|--------|------------|
+| Unstable QA data | Medium | High | Use EXEC_IDX-generated data |
+| External service unavailable | High | Medium | Use intercept / mock strategy |
+
+---
+
+## 9. Excluded TCs
+
+| TC ID | Reason for exclusion |
+|-------|---------------------|
+| TC-{M}-{S}-NNN | `BLOCKED-PERMISSIONS` |
+| TC-{M}-{S}-NNN | `PENDING-CODE` |
+
+---
+
+## 10. Execution Schedule
+
+| Suite | Trigger | Frequency |
+|-------|---------|-----------|
+| P0 smoke | Every CI build | On push to main / PR |
+| P1 regression | Scheduled | Nightly |
+| P2 full regression | Manual trigger | Before each release |
+
+---
+
+## Changelog
+
+| Version | Date | Description |
+|---------|------|-------------|
+| 1.0 | YYYY-MM-DD | Initial creation |
+```
+
+---
+
+## Priority Decision Tree
+
+```
+Primary happy path of the feature?
+  YES → P0
+Critical data corruption or show-stopper risk?
+  YES → P0
+Common negative scenario (empty required field, duplicate)?
+  YES → P1
+Cross-module dependency or state machine transition?
+  YES → P1
+Secondary feature (export, pagination, search)?
+  YES → P2
+Edge case, cosmetic, or low-frequency scenario?
+  YES → P3
+```
+
+---
+
+## Automation Feasibility Criteria
+
+| Feasibility | Criteria |
+|---|---|
+| Fully automatable | Deterministic, observable via browser, no external system dependency |
+| Partially automatable | External system can be mocked, or some step requires human inspection |
+| Not automatable | Physical access required; irreversible side effects; non-deterministic; BLOCKED-PERMISSIONS |

package/skills/qa-test-stabilization/SKILL.md

@@ -0,0 +1,123 @@
+---
+name: qa-test-stabilization
+description: >
+  Stage 5b of the QA pipeline. Diagnoses and fixes failing or flaky Playwright
+  tests. Classifies root causes, applies targeted fixes, and produces a
+  stabilization report.
+  Use when asked to fix failing tests, stabilize flaky tests, diagnose test
+  failures, debug e2e tests, reduce test flakiness, or improve test reliability.
+  Can be used any time after Stage 5 (Automation).
+---
+
+# QA Skill: Test Stabilization (Stage 5b)
+
+**Stage**: 5b — Stabilization (parallel to / after Stage 5)  
+**Prerequisite**: Failing or flaky tests exist in `qa/07-automation/e2e/`  
+**Output**: Stable passing tests + `STABILIZATION-REPORT.md`  
+**Next stage**: Return to Stage 5 to continue automation, or Stage 6 if all tests pass
+
+> **Scope boundary**: Only fix tests inside `qa/07-automation/`. Do not modify application source code. Never introduce `waitForTimeout` as a stabilization fix.
+
+---
+
+## Inputs Required
+
+1. Failure output — CI log, local run output, or list of failing test names
+2. `qa/07-automation/e2e/{file}.spec.ts` — failing spec files
+3. `qa/01-specifications/{module}/` — ground truth for expected behavior
+4. `qa/07-automation/playwright.config.ts` — current runner config
+
+---
+
+## Process
+
+### Step 1 — Reproduce locally
+
+Run failing tests in isolation:
+```
+npx playwright test {file}.spec.ts --project={project} --reporter=list
+```
+Do not trust CI-only failures until reproduced, or classify immediately as Category H (CI environment).
+
+### Step 2 — Classify each failure
+
+Use the classification protocol: `references/classification-protocol.md`
+
+Category summary:
+- **A — Selector broken**: locator no longer matches; fix selector
+- **B — Timing/flaky**: async operation not awaited correctly; replace with proper wait
+- **C — Data conflict**: parallel test data collision; apply `EXEC_IDX`
+- **D — Auth/session**: session expired or role mismatch; repair fixture
+- **E — Spec mismatch**: behavior changed; update spec first, then test
+- **F — Network dependency**: external call not mocked; add mock or skip
+- **G — Environment**: missing env var or wrong base URL; fix config
+- **H — CI environment**: OS/browser difference; add retry or tag as `@ci-skip`
+- **I — Logic error**: test assertion is wrong; fix assertion to match spec
+
+### Step 3 — Apply fixes in priority order
+
+Fix category priorities: A → C → D → B → I → E → F → G → H
+
+For each fix:
+1. Apply minimal change — do not refactor surrounding code
+2. Re-run the specific TC to verify fix
+3. Re-run the full describe block to verify no regression
+4. Note confidence score (0-100) for each fix
+
+### Step 4 — Confidence scoring
+
+Calculate composite confidence:
+- Consistent pass ×3 locally: +40
+- Root cause identified and understood: +30
+- No `waitForTimeout` introduced: +20
+- Spec confirms expected behavior: +10
+
+Target: ≥ 90 before marking TC as stable.
+
+### Step 5 — Handle unresolvable failures
+
+If a failure cannot be fixed without application code change:
+- Add `test.skip(true, 'PENDING-CODE: {ADO item or description}')` annotation
+- Update `qa/01-specifications/{module}/05-test-scenarios.md` with `PENDING-CODE` note
+
+### Step 6 — Update spec if behavior changed (Category E)
+
+Follow `qa-maintenance` skill for mid-sprint spec updates. Do not change spec to match wrong behavior.
+
+### Step 7 — Re-run full suite
+
+After all fixes applied, run the full module suite:
+```
+npx playwright test --project={module} --reporter=list
+```
+All previously failing TCs should pass or be explicitly skipped.
+
+### Step 8 — Produce stabilization report
+
+Template: `references/classification-protocol.md` (STABILIZATION-REPORT section)
+
+Required sections:
+- Run date + environment
+- TC table: ID | Category | Root cause | Fix applied | Confidence | Status
+- Summary: pass/fail/skip counts, blockers needing application fix
+- Recommendations (if any systemic issues found)
+
+---
+
+## Key Rules
+
+| Rule | Required |
+|---|---|
+| No `waitForTimeout` | Use `waitForSelector` or `waitForResponse` |
+| No app code changes | Fix tests only, not application |
+| Spec is ground truth | Never change spec to match wrong behavior |
+| Unresolvable → skip | With PENDING-CODE annotation |
+| Report required | Every stabilization session produces a report |
+
+---
+
+## Outputs
+
+- Fixed spec files in `qa/07-automation/e2e/`
+- `qa/07-automation/STABILIZATION-REPORT-{date}.md`
+- Updated `qa/01-specifications/` (if Category E found)