@keber/qa-framework 1.5.2 → 1.6.1

package/agent-instructions/02-test-plan-generation.md

@@ -1,205 +0,0 @@
-# Agent Instructions: Test Plan Generation
-
-> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-plan/SKILL.md`.
-> Kept as reference during the transition period. Do not update this file.
-
-**File**: `agent-instructions/02-test-plan-generation.md`  
-**Purpose**: Instructions for generating a test plan document for a module, after its spec set is complete.
-
----
-
-## When to use
-
-- After all submodule specs (`05-test-scenarios.md`) are written and reviewed
-- When preparing for a sprint or release
-- When deciding which TCs to automate vs test manually
-
----
-
-## Inputs Required
-
-1. `qa/01-specifications/module-{name}/` — all submodule specs
-2. Scope definition: which submodules are in this test plan
-3. Priority guidance (from the team or implied by sprint scope)
-
----
-
-## Test Plan Document Structure
-
-Create: `qa/02-test-plans/automated/{module}-test-plan.md`
-
-```markdown
-# Test Plan: {Module Display Name}
-
-| Field | Value |
-|-------|-------|
-| Version | 1.0 |
-| Date | YYYY-MM-DD |
-| Module | {module-code} |
-| Test plan type | Automated / Manual / Combined |
-| Author | {agent or human name} |
-| Status | Draft / Approved / Active |
-
----
-
-## 1. Objective
-
-{1-2 sentences: what this test plan covers and why}
-
----
-
-## 2. Scope
-
-### In scope
-- {submodule 1}: {what is being tested}
-- {submodule 2}: {what is being tested}
-
-### Out of scope
-- {what is explicitly excluded and why}
-
----
-
-## 3. Automation Feasibility Analysis
-
-| Submodule | Total TCs | Fully automatable | Partially | Not automatable | Reason for exclusions |
-|-----------|-----------|------------------|-----------|-----------------|----------------------|
-| {name} | {N} | {N} ({%}) | {N} ({%}) | {N} ({%}) | {brief reason} |
-
-**Definition**:
-- **Fully automatable**: deterministic, observable, no external dependencies
-- **Partially automatable**: some steps can be automated, others require manual verification
-- **Not automatable**: requires human judgment, external system access, or is too risky to automate in QA
-
----
-
-## 4. Priority Distribution
-
-| Priority | TCs selected | Automated | Manual |
-|----------|-------------|-----------|--------|
-| P0 (Critical — must run every build) | {N} | {N} | {N} |
-| P1 (High — must run before release) | {N} | {N} | {N} |
-| P2 (Medium — periodic regression) | {N} | {N} | {N} |
-| P3 (Low — manual only) | {N} | 0 | {N} |
-
----
-
-## 5. TC Selection and Automation Plan
-
-### P0 Suite — {suite name}
-
-| TC ID | Title | Type | Automation status |
-|-------|-------|------|------------------|
-| TC-{M}-{S}-001 | {title} | Functional | Queued |
-| TC-{M}-{S}-002 | {title} | Negative | Queued |
-
-### P1 Suite — {suite name}
-
-| TC ID | Title | Type | Automation status |
-|-------|-------|------|------------------|
-| TC-{M}-{S}-010 | {title} | Functional | Queued |
-
----
-
-## 6. Azure DevOps Test Plan (if enabled)
-
-| Item | Value |
-|------|-------|
-| ADO Plan ID | (populate after creation) |
-| ADO Suite IDs | (populate after creation) |
-| Reporter | @alex_neo/playwright-azure-reporter |
-| Sync script | qa/08-azure-integration/scripts/inject-ado-ids.ps1 |
-
----
-
-## 7. Test Environment
-
-| Item | Value |
-|------|-------|
-| QA URL | {{QA_BASE_URL}} |
-| Browser | Chromium (Playwright) |
-| Workers (local) | {N from config} |
-| Workers (CI) | {N from config} |
-| Parallel sessions supported | Yes / No |
-
----
-
-## 8. Risks and Assumptions
-
-| Risk | Likelihood | Impact | Mitigation |
-|------|-----------|--------|------------|
-| Unstable QA data | Medium | High | Use EXEC_IDX-generated data |
-| External service (email, etc.) not available | High | Medium | Use 3-layer validation strategy |
-| QA env down-time | Low | High | Retry with 2h buffer in CI |
-
----
-
-## 9. Excluded TCs
-
-| TC ID | Reason for exclusion |
-|-------|---------------------|
-| TC-{M}-{S}-NNN | `BLOCKED-PERMISSIONS`: QA user cannot access |
-| TC-{M}-{S}-NNN | `PENDING-CODE`: feature not yet deployed |
-| TC-{M}-{S}-NNN | Not automatable: requires visual inspection / external access |
-
----
-
-## 10. Execution Schedule
-
-| Suite | Trigger | Frequency |
-|-------|---------|-----------|
-| P0 smoke | Every CI build | On push to main / PR |
-| P1 regression | Scheduled | Nightly |
-| P2 full regression | Manual trigger | Before each release |
-
----
-
-## Changelog
-
-| Version | Date | Description |
-|---------|------|-------------|
-| 1.0 | YYYY-MM-DD | Initial creation |
-```
-
----
-
-## Rules for Priority Assignment
-
-Use this decision tree for each TC:
-
-```
-Is this the primary happy path of the feature?
-├── YES → P0
-
-Is a bug here a show-stopper or critical data corruption risk?
-├── YES → P0
-├── NO →
-   Is this a common negative scenario (empty required field, duplicate)?
-   ├── YES → P1
-   Is this a cross-module dependency or state machine transition?
-   ├── YES → P1
-   Is this a secondary feature (export, pagination, search)?
-   ├── YES → P2
-   Is this an edge case, cosmetic, or low-frequency scenario?
-   ├── YES → P3
-```
-
----
-
-## Automation Feasibility Decision Criteria
-
-**Mark as fully automatable** when:
-- All interactions are observable via browser automation
-- The result is deterministic (same input → same output)
-- No external systems are involved (live email server, payment gateway)
-- Data can be set up programmatically
-
-**Mark as partially automatable** when:
-- Some result requires human visual inspection
-- An external system is involved but can be mocked (email intercept, API stub)
-- Data setup requires a manual step first
-
-**Mark as not automatable** when:
-- Requires physical access or real hardware
-- Result is inherently non-deterministic (performance, visual design)
-- Test would create irreversible side effects in QA environment (live invoice, real payment)
-- The feature is blocked by permissions that cannot be granted to QA users

package/agent-instructions/03-test-case-generation.md

@@ -1,150 +0,0 @@
-# Agent Instructions: Test Case Generation
-
-> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-cases/SKILL.md`.
-> Kept as reference during the transition period. Do not update this file.
-
-**File**: `agent-instructions/03-test-case-generation.md`  
-**Purpose**: Instructions for writing detailed, standalone test case documents from approved spec scenarios.
-
----
-
-## When to use
-
-- When a TC from `05-test-scenarios.md` needs more detail than the table row provides
-- When writing complex multi-step TCs that require dedicated documentation
-- When preparing TCs for manual testers who need step-by-step instructions
-
----
-
-## Test Case Template
-
-```markdown
-# TC-{MODULE}-{SUBMODULE}-{NNN}: {TC Title}
-
-| Field | Value |
-|-------|-------|
-| TC ID | TC-{MODULE}-{SUBMODULE}-{NNN} |
-| Module | {Module display name} |
-| Submodule | {Submodule display name} |
-| Priority | P0 / P1 / P2 / P3 |
-| Type | Functional / Negative / Regression / Security / Integration |
-| Origin | UI-OBSERVED / PENDING-CODE / BLOCKED-PERMISSIONS |
-| Observation date | YYYY-MM-DD |
-| Automatable | Yes / Partial / No |
-| Playwright file | (fill when automation is written) |
-| Status | Draft / Approved / Automated / Deprecated |
-
----
-
-## Objective
-
-{1-2 sentences: what this test case verifies and why it matters}
-
----
-
-## Preconditions
-
-1. User is logged in as **{role}**
-2. The following data exists in the QA environment:
-   - {item 1}: {description}
-   - {item 2}: {description}
-3. {any other system state requirement}
-
----
-
-## Test Data
-
-| Field | Value |
-|-------|-------|
-| {field name} | {value or generation rule} |
-| {field name} | Use `EXEC_IDX`-generated value for uniqueness |
-| User | Credentials from `QA_USER_{ROLE}_EMAIL` env var |
-
----
-
-## Steps
-
-| Step | Action | Expected result |
-|------|--------|----------------|
-| 1 | Navigate to {URL} | {page loads, specific element visible} |
-| 2 | Click {element description} | {modal opens / form appears / etc.} |
-| 3 | Fill {field} with {value} | {field accepts input} |
-| 4 | Click {submit/save/confirm button} | {success message / redirect / state change} |
-| 5 | Verify {observable outcome} | {specific text, element, state} |
-
----
-
-## Expected Results
-
-**Final state**: {what the system should look like after all steps are complete}
-
-**Verification points**:
-- [ ] {verification 1}: {element} should {state}
-- [ ] {verification 2}: {element} should contain {text}
-- [ ] {verification 3}: page/modal should transition to {expected state}
-
----
-
-## Known Issues / Related Defects
-
-| Issue | DEF-ID / ADO WI | Status | Impact on this TC |
-|-------|----------------|--------|------------------|
-| {description} | DEF-NNN | Open | TC skipped until resolved |
-
----
-
-## Evidence
-
-{Link to screenshot or video from execution, if available}
-
----
-
-## Automation Notes
-
-```typescript
-// Playwright selector for key element (discovered during implementation)
-// page.locator('{selector}')
-
-// Key assertion
-// await expect(page.locator('{selector}')).toHaveText('{expected text}');
-```
-
----
-
-## Changelog
-
-| Version | Date | Author | Description |
-|---------|------|--------|-------------|
-| 1.0 | YYYY-MM-DD | {author} | Initial creation |
-```
-
----
-
-## Rules for TC Content Quality
-
-### Write measurable expected results
-
-❌ BAD: "The system should work correctly"  
-✅ GOOD: "A success toast message 'Record saved' appears and the datatable row count increases by 1"
-
-❌ BAD: "The form validates the input"  
-✅ GOOD: "A red validation message 'This field is required' appears below the Name field"
-
-### Write observable steps
-
-❌ BAD: "Add a new record with valid data"  
-✅ GOOD: "Click the 'New' button. Fill 'Name' with 'Test-{EXEC_IDX}'. Click 'Save'."
-
-### Link to defects explicitly
-
-If a TC is skipped due to a bug:  
-✅ GOOD: "Step 4 — SKIPPED: DEF-001 (ADO #21944). The switch defaults to OFF when it should be ON. Reactivate after fix."
-
-### Priority guidelines recap
-
-| Priority | Use when |
-|---|---|
-| P0 | Core feature is unusable without this working |
-| P1 | Major functionality gap if this fails |
-| P2 | Minor functionality gap, workaround exists |
-| P3 | Edge case, cosmetic, or rare scenario |

package/agent-instructions/04-automation-generation.md

@@ -1,343 +0,0 @@
-# Agent Instructions: Automation Generation
-
-> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-automation/SKILL.md`.
-> Kept as reference during the transition period. Do not update this file.
-
-**File**: `agent-instructions/04-automation-generation.md`  
-**Purpose**: Instructions for writing Playwright E2E automation tests that trace back to approved test cases in `01-specifications/`.
-
----
-
-> **Prerequisite**: Specifications must be reviewed and approved before writing automation. Do not automate TCs that are `PENDING-CODE` or `BLOCKED-PERMISSIONS`.
-
----
-
-## Structure: How to Organize Spec Files
-
-Each module gets a directory under `qa/07-automation/e2e/tests/`:
-
-```
-qa/07-automation/e2e/tests/
-└── {module-kebab}/
-    ├── COVERAGE-MAPPING.md          ← TC-ID → spec location map
-    ├── {suite-name}.spec.ts         ← P0 suite
-    ├── {suite-name}-p1.spec.ts      ← P1 suite
-    └── integration/
-        └── cross-module.spec.ts    ← Cross-module dependencies
-```
-
-Spec file naming: `{description}.spec.ts` in kebab-case.
-
----
-
-## Spec File Template
-
-```typescript
-/**
- * @module {MODULE_CODE}
- * @submodule {SUBMODULE_CODE}
- * @spec qa/01-specifications/module-{name}/submodule-{name}/05-test-scenarios.md
- * @priority P0
- */
-
-import { test, expect } from '@playwright/test';
-
-// EXEC_IDX: changes every 60 seconds, provides collision-resistant unique values
-const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
-
-test.describe('{Feature Suite Name}', () => {
-
-  test.beforeAll(async ({ browser }) => {
-    // Provision data needed by tests in this suite
-    // Rule: provision AT LEAST as many records as tests that consume them
-  });
-
-  test('[TC-{MODULE}-{SUB}-001] {TC title} @P0', async ({ page }) => {
-    // Arrange
-    // ...
-
-    // Act
-    // ...
-
-    // Assert
-    // ...
-  });
-
-  test('[TC-{MODULE}-{SUB}-002] {TC title} @P1', async ({ page }) => {
-    // ...
-  });
-
-  test.skip('[TC-{MODULE}-{SUB}-003] {TC title} @P1 — DEF-{NNN}: {description}. Reactivate when {ADO WI or DEF ID} is resolved.', async ({ page }) => {
-    // Skip body: write the test as if the bug were fixed
-    // This documents the expected behavior, not the buggy state
-  });
-
-});
-```
-
----
-
-## Required Patterns
-
-### Pattern 1: EXEC_IDX — Unique Test Data
-
-Use EXEC_IDX everywhere you create records to avoid collisions between consecutive test runs:
-
-```typescript
-const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
-
-// For titles/names
-const entityName = `Test-Record-${EXEC_IDX}`;
-
-// For dates (future-dated to avoid prod data collisions)
-const year = 2120 + (EXEC_IDX % 10);
-const month = String((EXEC_IDX % 12) + 1).padStart(2, '0');
-const day = String((EXEC_IDX % 28) + 1).padStart(2, '0');
-const testDate = `${year}-${month}-${day}`;
-```
-
-Why year 2120+: avoids any overlap with real production data.  
-Why modular arithmetic: the date always represents a valid date, never day 32 or month 13.
-
-### Pattern 2: Multi-Role Auth with storageState
-
-Use `global-setup.ts` to log in once per role and save storageState:
-
-```typescript
-// playwright.config.ts
-use: {
-  storageState: '.auth/session.json', // single role
-  // or: storageState: (workerInfo) => `.auth/role-${workerInfo.workerIndex}.json`
-}
-
-// global-setup.ts — runs once before all tests
-async function globalSetup(config) {
-  const browser = await chromium.launch();
-  const page = await browser.newPage();
-  
-  // Log in and save session
-  await page.goto(process.env.QA_BASE_URL + process.env.QA_LOGIN_PATH);
-  await page.fill(process.env.QA_LOGIN_EMAIL_SELECTOR, process.env.QA_USER_EMAIL);
-  // Use evaluate() for password — excluded from traces
-  await page.evaluate(
-    ([sel, pwd]) => { (document.querySelector(sel) as HTMLInputElement).value = pwd; },
-    [process.env.QA_LOGIN_PASSWORD_SELECTOR, process.env.QA_USER_PASSWORD]
-  );
-  await page.click(process.env.QA_LOGIN_SUBMIT_SELECTOR);
-  await page.waitForURL(url => !url.includes(process.env.QA_LOGIN_PATH));
-  
-  await page.context().storageState({ path: '.auth/session.json' });
-  await browser.close();
-}
-```
-
-For multi-role: call login sequence sequentially for each role (apps often reject concurrent sessions).
-
-### Pattern 3: Fast-Fail for Status Checks
-
-When checking the text content of status indicators, use a short timeout to prevent hanging:
-
-```typescript
-// BAD: inherits the full test timeout (30s or more)
-const status = await page.locator('.status-badge').textContent();
-
-// GOOD: fails fast if not available in 3 seconds
-const status = await page.locator('.status-badge').textContent({ timeout: 3_000 });
-```
-
-### Pattern 4: Email Validation — 3-Layer Strategy
-
-When testing email notification behavior without a real inbox:
-
-```typescript
-// Layer 1: HTTP route interception (most reliable)
-const emailRequests: string[] = [];
-await page.route('**/*mail*', (route) => {
-  emailRequests.push(route.request().url());
-  route.continue();
-});
-await page.route('**/*correo*', (route) => {
-  emailRequests.push(route.request().url());
-  route.continue();
-});
-// ... perform the action ...
-expect(emailRequests.length).toBeGreaterThan(0);
-
-// Layer 2: UI confirmation (toast, banner, badge)
-await expect(page.locator('.email-sent-indicator')).toBeVisible();
-
-// Layer 3: test.fixme() — if inbox access is needed and not available
-test.fixme(process.env.QA_MAILBOX_ACCESS !== 'true', 'Requires mailbox access');
-```
-
-### Pattern 5: Skip Pattern for Known Bugs
-
-```typescript
-test.skip(true, 
-  'DEF-001: {Description of bug}. ' +
-  'Expected: {expected behavior}. ' +
-  'Actual: {buggy behavior}. ' +
-  'Reactivate when ADO #{WI_ID} / DEF-001 is resolved.'
-);
-```
-
-**Never** make a test artificially "pass" around a bug. Document with exact expected behavior.
-
-### Pattern 6: beforeAll Provisioning Ratio
-
-In `beforeAll`, create **at least as many records as the number of tests that consume them**:
-
-```typescript
-test.beforeAll(async ({ browser }) => {
-  const page = await browser.newPage();
-  // If 3 tests each need 1 different pending record:
-  const records: string[] = [];
-  for (let i = 0; i < 3; i++) {
-    const id = await createRecord(page, `Record-${EXEC_IDX}-${i}`);
-    records.push(id);
-  }
-  // Store in closure or page object for use in tests
-});
-```
-
-### Pattern 7: Password Injection (Trace Safety)
-
-```typescript
-// Passwords must be set via evaluate() to exclude from Playwright traces
-await page.evaluate(
-  ([selector, pwd]) => {
-    (document.querySelector(selector) as HTMLInputElement).value = pwd;
-  },
-  [process.env.QA_LOGIN_PASSWORD_SELECTOR, process.env.QA_USER_PASSWORD]
-);
-```
-
----
-
-## `playwright.config.ts` Checklist
-
-Every project Playwright config should configure:
-
-```typescript
-export default defineConfig({
-  testDir: './tests',
-  timeout: 30_000,           // Per-test timeout
-  actionTimeout: 10_000,     // Per-action timeout (REQUIRED — prevents silent hangs)
-  navigationTimeout: 60_000, // Per-navigation timeout
-  retries: process.env.CI ? 2 : 0,
-  workers: process.env.CI ? 1 : 2,
-  
-  // Exclude seeds and debug scripts from normal runs
-  testIgnore: ['**/seeds/**', '**/debug-*.spec.ts', '**/seed.spec.ts'],
-  
-  use: {
-    baseURL: process.env.QA_BASE_URL,
-    storageState: '.auth/session.json',
-    locale: 'en-US', // or 'es-CL' for Chilean apps
-    screenshot: 'only-on-failure',
-    video: 'retain-on-failure',
-    trace: 'on-first-retry',
-  },
-  
-  reporter: [
-    ['html', { outputFolder: 'playwright-report' }],
-    // ADO reporter: only when integration enabled
-    // ['@alex_neo/playwright-azure-reporter', { ... }],
-  ],
-  
-  projects: [
-    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
-    // Add firefox, webkit when cross-browser coverage is needed
-  ],
-  
-  globalSetup: './global-setup.ts',
-  outputDir: '../../05-test-execution/automated/test-results',
-});
-```
-
-`actionTimeout` is **not optional**. Without it, `textContent()`, `getAttribute()`, and similar calls inherit the full test timeout and can silently hang if the element is slow.
-
----
-
-## Stability Criteria
-
-A test is only considered stable when:
-
-1. It passes ≥ 2 consecutive runs with **different** `EXEC_IDX` values (i.e., at least 60 seconds apart)
-2. It does not rely on another test's side effects
-3. It produces the same result across 3+ independent runs
-4. No `waitForTimeout(N)` longer than 2000ms is used (use `waitForSelector` or `waitForLoadState` instead)
-
----
-
-## Debugging Failed Tests
-
-Use this 5-layer methodology:
-
-1. **Environment layer**: Is QA_BASE_URL reachable? Are credentials valid? Run login test first.
-2. **Timeout layer**: Did the test fail with "timeout" or "Timeout exceeded"? Check `actionTimeout`, increase if justified.
-3. **Data layer**: Did `beforeAll` create the expected records? Add `console.log` per step in `beforeAll`.
-4. **Code layer**: Is the selector correct? Use page.pause() or Playwright UI mode to inspect.
-5. **System defect layer**: Is the failure due to a real bug? File a defect and add `test.skip()`.
-
-**Diagnostic script pattern**: For complex data discovery, create a diagnostic spec in `helpers/debug/`:
-
-```typescript
-// helpers/debug/inspect-{feature}.spec.ts
-test('diagnostic: dump form options', async ({ page }) => {
-  await page.goto(`${process.env.QA_BASE_URL}/my-form`);
-  const options = await page.$$eval('select option', els => 
-    els.map(el => ({ value: el.value, text: el.textContent }))
-  );
-  console.log(JSON.stringify(options, null, 2));
-});
-```
-
-This is not part of the test suite (excluded via `testIgnore`). Use it to discover valid option values.
-
----
-
-## Coverage Mapping Document
-
-After writing automation for a module, create `COVERAGE-MAPPING.md` in the test directory:
-
-```markdown
-# Coverage Mapping — {Module}
-
-| TC ID | Title | Spec file | Playwright spec | Status |
-|-------|-------|-----------|----------------|--------|
-| TC-OPER-CAT-001 | Access catalog list | 01-specifications/.../05-test-scenarios.md | tests/operacion/catalogos/critical.spec.ts | ✅ Automated |
-| TC-OPER-CAT-002 | Create labor type | ... | tests/operacion/... | ✅ Automated |
-| TC-OPER-CAT-003 | Duplicate rejection | ... | tests/operacion/... | ⚠️ Partially (UI only) |
-| TC-OPER-ASI-001 | Access assistance list | ... | — | ⛔ Blocked (BLOCKED-PERMISSIONS) |
-```
-
----
-
-## Module Completion Checklist
-
-Run this checklist when all P0 tests for a module are passing.
-
-### Stability gate
-- [ ] All P0 tests pass on **2 consecutive runs** with different `EXEC_IDX` values (≥ 60 seconds apart)
-- [ ] No `waitForTimeout(N)` longer than 2000ms left in the suite
-- [ ] All skipped tests have a `DEF-` or ADO WI reference in their skip message
-
-### Artifacts
-- [ ] `COVERAGE-MAPPING.md` created or updated in the test directory
-- [ ] Execution report written to `qa/05-test-execution/automated/`
-
-### Living Index update — `qa/README.md`
-- [ ] Update the module's row in the status table (TCs automated, date, status → ✅)
-- [ ] Add a row to the `## Sprint History` section with: sprint name, date, submódulos covered, TC count, link to execution report
-
-### AGENT-NEXT-STEPS.md — trim
-- [ ] Move the completed sprint checklist block to `qa/README.md` under `## Sprint History`
-- [ ] Delete the completed sprint section from `AGENT-NEXT-STEPS.md`
-- [ ] Verify `AGENT-NEXT-STEPS.md` now contains **only one active sprint** (the next one)
-
-### Memory
-- [ ] If new patterns or environment quirks were discovered: add them to `qa/memory/` with a descriptive filename
-- [ ] Update `qa/memory/INDEX.md` to reflect any new or updated files
-
-> **Rule**: `AGENT-NEXT-STEPS.md` is a task queue, not a log. Completed sprints live in `qa/README.md`. The file must never accumulate more than one active sprint.