@keber/qa-framework 1.2.0-0 → 1.4.0-0

package/skills/qa-ado-integration/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: qa-ado-integration
+description: >
+  Optional stage of the QA pipeline. Syncs test plans, test cases, and
+  automation results with Azure DevOps (ADO) Test Plans. Injects ADO IDs
+  back into local spec files and configures the Playwright ADO reporter.
+  Use when asked to sync with Azure DevOps, push test cases to ADO, create
+  a test plan in ADO, configure ADO reporting, inject ADO IDs, or link
+  Playwright results to ADO.
+  Requires ADO enabled in qa-framework.config.json with valid PAT in env vars.
+---
+
+# QA Skill: ADO Integration (Optional Stage)
+
+**Stage**: Optional — ADO Integration (no pipeline order dependency)  
+**Prerequisite**: ADO configuration in `qa/qa-framework.config.json`; valid `ADO_PAT` env var  
+**Output**: ADO Test Plan populated + `qa/08-azure-integration/module-registry.json` updated  
+**Use any time** after Stage 4 (Test Cases) or Stage 5 (Automation) are ready to sync
+
+> **Credential rule**: NEVER hardcode PAT, username, or organization URL in any file. Always read from env vars or `<PLACEHOLDER>` in documentation.
+
+---
+
+## Inputs Required
+
+1. `qa/qa-framework.config.json` — ADO org URL, project name, test plan ID (or "create new")
+2. `qa/03-test-cases/automated/` — TC files with IDs to sync
+3. `ADO_PAT` environment variable — personal access token with Test Plans read/write
+4. Optionally: `qa/08-azure-integration/module-registry.json` for existing ID mapping
+
+---
+
+## Process
+
+### Step 1 — Verify ADO config
+
+Check `qa/qa-framework.config.json` for:
+```json
+"ado": {
+  "org": "<ADO_ORG_URL>",
+  "project": "<PROJECT_NAME>",
+  "planId": <PLAN_ID_OR_NULL>
+}
+```
+If `planId` is null, a new plan will be created in Step 2.
+
+### Step 2 — Create or verify Test Plan
+
+Use the PowerShell script: `integrations/ado-powershell/scripts/create-testplan-from-mapping.ps1`
+
+See `references/scripts-and-config.md` for usage, parameters, and required env vars.
+
+Required env vars:
+- `$env:ADO_ORG` — organization URL (e.g., `https://dev.azure.com/myorg`)
+- `$env:ADO_PROJECT` — project name
+- `$env:ADO_PAT` — PAT token
+
+### Step 3 — Sync test cases
+
+Use `inject-ado-ids.ps1` to push TCs to ADO and write the returned ADO test case IDs back into local TC files.
+
+After sync, each TC file must have `adoId: <NNN>` in its YAML frontmatter block.
+
+### Step 4 — Configure Playwright ADO reporter
+
+See `references/scripts-and-config.md` for the reporter config block to add to `qa/07-automation/playwright.config.ts`.
+
+Required fields:
+- `orgUrl`, `projectName`, `planId`, `runName`
+- All values must come from environment variables — no hardcoded IDs
+
+### Step 5 — Run sync and validate
+
+Run `sync-ado-titles.ps1` after automation runs to update ADO test outcomes from Playwright report JSON.
+
+### Step 6 — Update registry
+
+Update `qa/08-azure-integration/module-registry.json` with:
+- Module name
+- ADO Plan ID
+- ADO Suite IDs per submodule
+- Sync date
+
+---
+
+## Troubleshooting Quick Reference
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| 401 Unauthorized | Expired or wrong PAT | Regenerate PAT; check `$env:ADO_PAT` |
+| Plan not found | Wrong planId in config | Verify planId or set to null to create new |
+| TCs not appearing | Wrong suite parent ID | Check `module-registry.json` suite mapping |
+| Reporter not posting | Missing env var at runtime | Pass vars via CI pipeline or `.env` |
+| Duplicate TCs | Re-ran create script | Use `inject-ado-ids.ps1` idempotently |
+
+---
+
+## Outputs
+
+- ADO Test Plan populated with test cases
+- `qa/03-test-cases/automated/TC-*.md` — ADO IDs injected
+- `qa/08-azure-integration/module-registry.json` updated
+- `qa/07-automation/playwright.config.ts` updated with reporter config

package/skills/qa-ado-integration/references/scripts-and-config.md

@@ -0,0 +1,201 @@
+# Reference: ADO Scripts and Configuration
+
+> Loaded by `skills/qa-ado-integration/` for PowerShell scripts, reporter config, and CI pipeline.
+
+---
+
+## PowerShell Scripts Usage
+
+### create-testplan-from-mapping.ps1
+
+Creates a new ADO Test Plan with suites and test cases from the mapping JSON file.
+
+```powershell
+.\integrations\ado-powershell\scripts\create-testplan-from-mapping.ps1 `
+  -Organization $env:ADO_ORG `
+  -Project $env:ADO_PROJECT `
+  -Token $env:ADO_PAT `
+  -MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json
+```
+
+Output: `TestPlanId` and `SuiteIds` — record in `module-registry.json`.
+
+---
+
+### inject-ado-ids.ps1
+
+Injects ADO Work Item IDs into Playwright test titles and adds ADO annotations.
+
+```powershell
+# Dry run first
+.\integrations\ado-powershell\scripts\inject-ado-ids.ps1 `
+  -MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
+  -TestDir .\qa\07-automation\e2e\tests `
+  -DryRun
+
+# Apply
+.\integrations\ado-powershell\scripts\inject-ado-ids.ps1 `
+  -MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
+  -TestDir .\qa\07-automation\e2e\tests
+```
+
+Transforms test titles:
+```
+# Before
+'[TC-OPER-CAT-001] Access catalog list @P0'
+
+# After
+'[22957] Access catalog list @P0'
+```
+
+Adds annotation:
+```typescript
+test.info().annotations.push({ type: 'TestCase', description: '22957' });
+```
+
+---
+
+### sync-ado-titles.ps1
+
+Syncs ADO TC titles with local spec file titles (detects drift):
+
+```powershell
+.\integrations\ado-powershell\scripts\sync-ado-titles.ps1 `
+  -Organization $env:ADO_ORG `
+  -Project $env:ADO_PROJECT `
+  -Token $env:ADO_PAT `
+  -MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
+  -DryRun
+```
+
+Apply with `-DryRun:$false`.
+
+---
+
+## ADO IDs Mapping File
+
+```json
+{
+  "planId": 22304,
+  "module": "operacion",
+  "specsPath": "qa/07-automation/e2e/tests/operacion",
+  "suites": [
+    {
+      "suiteId": 22956,
+      "suiteName": "Suite 1.1 - {Submodule Name}",
+      "testCases": [
+        { "id": 22957, "title": "Access catalog list @P0", "tags": ["automatable", "playwright"] },
+        { "id": 22958, "title": "Create labor type @P1", "tags": ["automatable", "playwright"] }
+      ]
+    }
+  ]
+}
+```
+
+---
+
+## Module Registry
+
+```json
+{
+  "modules": [
+    {
+      "name": "{module-kebab}",
+      "specsPath": "qa/07-automation/e2e/tests/{module-kebab}",
+      "planId": 0,
+      "suiteId": 0,
+      "description": "{Module display name} — {N} submodules"
+    }
+  ]
+}
+```
+
+---
+
+## playwright-azure-reporter Config
+
+Add to `qa/07-automation/playwright.config.ts` reporter array:
+
+```typescript
+['@alex_neo/playwright-azure-reporter', {
+  orgUrl: `https://dev.azure.com/${process.env.ADO_ORG}`,
+  token: process.env.ADO_PAT,
+  planId: Number(process.env.ADO_PLAN_ID),
+  projectName: process.env.ADO_PROJECT,
+  isDisabled: !process.env.CI,          // only publish in CI
+  publishTestResultsMode: 'testRun',
+  testRunTitle: `QA Run - ${new Date().toISOString()}`,
+  uploadAttachments: true,
+  attachmentsType: ['screenshot', 'video', 'trace'],
+}]
+```
+
+**`isDisabled: !process.env.CI`** — prevents publishing from local developer runs.
+
+---
+
+## CI Pipeline Template
+
+```yaml
+trigger:
+  branches:
+    include:
+      - {{CI_TRIGGER_BRANCH}}
+
+pool:
+  vmImage: 'ubuntu-latest'
+
+variables:
+  - group: qa-secrets   # Must contain: QA_USER_EMAIL, QA_USER_PASSWORD, QA_BASE_URL, ADO_PAT, ADO_PLAN_ID
+
+steps:
+  - checkout: self
+    fetchDepth: 2
+
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '20.x'
+    displayName: 'Install Node.js 20'
+
+  - script: |
+      cd qa/07-automation/e2e
+      npm install
+      npx playwright install chromium --with-deps
+    displayName: 'Install Playwright'
+
+  - script: |
+      cd qa/07-automation/e2e
+      npx playwright test
+    displayName: 'Run Playwright tests'
+    env:
+      QA_BASE_URL: $(QA_BASE_URL)
+      QA_USER_EMAIL: $(QA_USER_EMAIL)
+      QA_USER_PASSWORD: $(QA_USER_PASSWORD)
+      ADO_PAT: $(ADO_PAT)
+      ADO_PLAN_ID: $(ADO_PLAN_ID)
+      CI: 'true'
+
+  - task: PublishTestResults@2
+    inputs:
+      testResultsFormat: 'JUnit'
+      testResultsFiles: '**/test-results/*.xml'
+    displayName: 'Publish JUnit results'
+    condition: always()
+
+  - task: PublishPipelineArtifact@1
+    inputs:
+      targetPath: 'qa/07-automation/e2e/playwright-report'
+      artifact: 'playwright-html-report'
+    displayName: 'Publish HTML report'
+    condition: always()
+```
+
+---
+
+## Credential Security Rules
+
+1. `ADO_PAT` is **never** committed to any file — always from env var or CI variable group
+2. `ADO_PAT` is **never** logged, printed, or written to any markdown file
+3. If a PAT appears in any log or output: revoke it immediately in ADO Portal, generate new one
+4. Variable group `qa-secrets` must be scoped to the pipeline only — no project-wide access
+5. PAT minimum scopes: `Work Items (Read, Write)`, `Test Management (Read, Write)` — never use full-access PATs

package/skills/qa-automation/SKILL.md

@@ -0,0 +1,101 @@
+---
+name: qa-automation
+description: >
+  Stage 5 of the QA pipeline. Implements Playwright test automation from
+  approved specifications. Generates spec files, page objects, fixtures, and
+  configures the test runner.
+  Use when asked to automate tests, write Playwright specs, implement automation,
+  implement test scripts, code the e2e suite, or generate .spec.ts files.
+  Requires completed specifications with no PENDING-CODE sections (Stage 2 complete,
+  Stage 4 recommended).
+---
+
+# QA Skill: Automation Generation (Stage 5 of 6)
+
+**Stage**: 5 — Automation  
+**Prerequisite**: All specs for the target submodule are complete and contain no `PENDING-CODE` sections  
+**Output**: `qa/07-automation/e2e/{module}/{submodule}.spec.ts` + supporting files  
+**Next stage**: Stage 6 — Maintenance (`qa-maintenance`) or Stage 5b — Stabilization (`qa-test-stabilization`) if tests fail
+
+> **Pipeline rule**: Never automate a TC whose spec contains `PENDING-CODE`. Resolve spec gaps first.
+
+---
+
+## Inputs Required
+
+1. `qa/01-specifications/{module}/` — full spec set for target submodule
+2. `qa/07-automation/playwright.config.ts` — confirm project name maps to submodule tag
+3. `qa/07-automation/fixtures/auth.ts` — identify available auth fixtures
+4. TC list (from test plan or Stage 4) — defines which TCs to automate in this session
+
+---
+
+## Process
+
+### Step 1 — Scan for blockers
+
+Before writing a line of code:
+- Search all spec files for `PENDING-CODE` — stop and flag if found
+- Confirm `playwright.config.ts` exists; scaffold it if missing using `references/config-checklist.md`
+- Verify `fixtures/auth.ts` has the roles required by this submodule
+
+### Step 2 — Scaffold spec file
+
+Full spec file template and 7 implementation patterns: `references/patterns.md`
+
+Required scaffold elements:
+- `import { test, expect } from '@playwright/test'`
+- `const EXEC_IDX = process.env.EXEC_IDX ?? '0'` (prevents DB conflicts in parallel runs)
+- `test.describe('{Submodule Name}', () => { ... })`
+- Use fixture-based auth (never hardcode credentials)
+
+### Step 3 — Implement tests in priority order
+
+Implement P0 TCs first, then P1. Within each priority, follow scenario order from `05-test-scenarios.md`.
+
+For each TC:
+1. Map TC preconditions to `beforeAll`/`beforeEach` setup
+2. Write navigation to the starting URL (use relative paths, not hardcoded base URL)
+3. Use `test.step()` to group logical sub-actions — improves traceability
+4. Assert observable outcomes — avoid asserting internal implementation details
+
+### Step 4 — Apply stability rules
+
+- Never use `waitForTimeout` — use `waitForSelector`, `waitForResponse`, or role-based locators
+- Prefer `getByRole`, `getByLabel`, `getByTestId` over CSS selectors
+- Locators attached to dynamic data must use `EXEC_IDX` suffix
+- All test data cleared in `afterAll` — never leave residue
+
+### Step 5 — Module Completion Checklist
+
+Before marking the submodule as ✅ Automation Complete:
+- [ ] All P0 TCs passing in CI
+- [ ] All P1 TCs passing or explicitly deferred (with reason in spec)
+- [ ] No `test.skip` without a linked issue or `PENDING-CODE` tag
+- [ ] `playwright.config.ts` project includes submodule tag
+- [ ] `qa/README.md` automation status updated
+- [ ] AGENT-NEXT-STEPS.md active sprint updated (remove completed items)
+
+---
+
+## Key Patterns (reference names for Layer 3)
+
+| Pattern | When to use |
+|---|---|
+| `EXEC_IDX` | Any TC that creates persistent data |
+| `auth-fixture` | All authenticated TCs |
+| `fast-fail` | P0 TCs — skip P1+ if P0 fails |
+| `email-intercept` | Email verification flows |
+| `skip-conditional` | Scenarios blocked by PENDING-CODE |
+| `beforeAll-setup` | Shared precondition across TC group |
+| `password-env` | Any credential usage |
+
+See `references/patterns.md` for full TypeScript implementations.
+
+---
+
+## Outputs
+
+- `qa/07-automation/e2e/{module}/{submodule}.spec.ts`
+- `qa/07-automation/playwright.config.ts` (updated or confirmed)
+- `qa/README.md` automation status updated

package/skills/qa-automation/references/config-checklist.md

@@ -0,0 +1,79 @@
+# Reference: Playwright Config Checklist
+
+> Loaded by `skills/qa-automation/` when scaffolding or reviewing `playwright.config.ts`.
+
+---
+
+## Required Configuration Template
+
+```typescript
+import { defineConfig, devices } from '@playwright/test';
+
+export default defineConfig({
+  testDir: './tests',
+  timeout: 30_000,            // Per-test timeout
+  actionTimeout: 10_000,      // Per-action timeout — REQUIRED, prevents silent hangs
+  navigationTimeout: 60_000,  // Per-navigation timeout (SPAs may need longer)
+  retries: process.env.CI ? 2 : 0,
+  workers: process.env.CI ? 1 : 2,
+
+  // Exclude seeds and debug scripts from normal test runs
+  testIgnore: ['**/seeds/**', '**/debug-*.spec.ts', '**/seed.spec.ts'],
+
+  use: {
+    baseURL: process.env.QA_BASE_URL,
+    storageState: '.auth/session.json',
+    locale: 'es-CL',           // Adjust to app locale
+    screenshot: 'only-on-failure',
+    video: 'retain-on-failure',
+    trace: 'on-first-retry',
+  },
+
+  reporter: [
+    ['html', { outputFolder: 'playwright-report' }],
+    // ADO reporter — add only when integration is enabled:
+    // ['@alex_neo/playwright-azure-reporter', {
+    //   orgUrl: process.env.ADO_ORG,
+    //   projectName: process.env.ADO_PROJECT,
+    //   planId: Number(process.env.ADO_PLAN_ID),
+    //   token: process.env.ADO_PAT,
+    //   runName: `Automated run ${new Date().toISOString()}`,
+    // }],
+  ],
+
+  projects: [
+    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
+    // Add when cross-browser coverage required:
+    // { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
+    // { name: 'webkit', use: { ...devices['Desktop Safari'] } },
+  ],
+
+  globalSetup: './global-setup.ts',
+  outputDir: '../../05-test-execution/automated/test-results',
+});
+```
+
+---
+
+## Checklist for Review
+
+Before marking automation as complete, confirm:
+
+- [ ] `actionTimeout` is set — **not optional**; without it, `textContent()` and similar calls silently hang
+- [ ] `testIgnore` excludes seed scripts and debug files
+- [ ] `baseURL` reads from `process.env.QA_BASE_URL` (never hardcoded)
+- [ ] `storageState` path exists (created by `globalSetup`)
+- [ ] `outputDir` points to `qa/05-test-execution/automated/test-results`
+- [ ] `retries: 2` in CI for flake tolerance
+- [ ] `workers: 1` in CI (or EXEC_IDX used to handle parallel conflicts)
+- [ ] ADO reporter commented out unless ADO integration is active
+
+---
+
+## global-setup.ts Checklist
+
+- [ ] Reads every credential from `process.env` — no hardcoded values
+- [ ] Uses `evaluate()` for password input (trace safety)
+- [ ] Saves storageState to `.auth/session.json` (or per-role paths)
+- [ ] Handles login failure explicitly (does not silently save an unauthenticated state)
+- [ ] Runs sequentially for multiple roles (no parallel logins)

package/skills/qa-automation/references/patterns.md

@@ -0,0 +1,180 @@
+# Reference: Automation Patterns
+
+> Loaded by `skills/qa-automation/` when implementing Playwright tests.
+
+---
+
+## Spec File Template
+
+```typescript
+/**
+ * @module {MODULE_CODE}
+ * @submodule {SUBMODULE_CODE}
+ * @spec qa/01-specifications/module-{name}/submodule-{name}/05-test-scenarios.md
+ * @priority P0
+ */
+
+import { test, expect } from '@playwright/test';
+
+const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
+
+test.describe('{Feature Suite Name}', () => {
+
+  test.beforeAll(async ({ browser }) => {
+    // Provision data needed by all tests in this suite
+    // Rule: provision AT LEAST as many records as tests that consume them
+  });
+
+  test('[TC-{MODULE}-{SUB}-001] {TC title} @P0', async ({ page }) => {
+    // Arrange ... Act ... Assert
+  });
+
+  test.skip(
+    '[TC-{MODULE}-{SUB}-003] {TC title} @P1 — DEF-{NNN}: {description}. Reactivate when {ADO WI or DEF ID} is resolved.',
+    async ({ page }) => {
+      // Write test as if bug is fixed — documents expected behavior
+    }
+  );
+
+});
+```
+
+---
+
+## Pattern 1: EXEC_IDX — Unique Test Data
+
+```typescript
+const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
+
+// Titles / names
+const entityName = `Test-Record-${EXEC_IDX}`;
+
+// Dates (year 2120+ avoids production data overlap; modular math always valid)
+const year = 2120 + (EXEC_IDX % 10);
+const month = String((EXEC_IDX % 12) + 1).padStart(2, '0');
+const day = String((EXEC_IDX % 28) + 1).padStart(2, '0');
+const testDate = `${year}-${month}-${day}`;
+```
+
+---
+
+## Pattern 2: Multi-Role Auth with storageState
+
+```typescript
+// global-setup.ts
+import { chromium } from '@playwright/test';
+
+export default async function globalSetup() {
+  const browser = await chromium.launch();
+  const page = await browser.newPage();
+
+  await page.goto(process.env.QA_BASE_URL + process.env.QA_LOGIN_PATH);
+  await page.fill(process.env.QA_LOGIN_EMAIL_SELECTOR, process.env.QA_USER_EMAIL);
+  // Use evaluate() for password — excluded from traces
+  await page.evaluate(
+    ([sel, pwd]) => { (document.querySelector(sel) as HTMLInputElement).value = pwd; },
+    [process.env.QA_LOGIN_PASSWORD_SELECTOR, process.env.QA_USER_PASSWORD]
+  );
+  await page.click(process.env.QA_LOGIN_SUBMIT_SELECTOR);
+  await page.waitForURL(url => !url.includes(process.env.QA_LOGIN_PATH));
+
+  await page.context().storageState({ path: '.auth/session.json' });
+  await browser.close();
+}
+```
+
+For multiple roles: run login sequence **sequentially** (not in parallel — apps often reject concurrent sessions).
+
+---
+
+## Pattern 3: Fast-Fail for Status Checks
+
+```typescript
+// BAD — hangs silently for full test timeout
+const status = await page.locator('.status-badge').textContent();
+
+// GOOD — fails fast if element not available
+const status = await page.locator('.status-badge').textContent({ timeout: 3_000 });
+```
+
+---
+
+## Pattern 4: Email Validation — 3-Layer Strategy
+
+```typescript
+// Layer 1: HTTP route interception
+const emailRequests: string[] = [];
+await page.route('**/*mail*', (route) => {
+  emailRequests.push(route.request().url());
+  route.continue();
+});
+// Perform the action...
+expect(emailRequests.length).toBeGreaterThan(0);
+
+// Layer 2: UI confirmation (toast / banner / badge)
+await expect(page.locator('.email-sent-indicator')).toBeVisible();
+
+// Layer 3: skip if inbox access not available
+test.fixme(process.env.QA_MAILBOX_ACCESS !== 'true', 'Requires mailbox access');
+```
+
+---
+
+## Pattern 5: Skip for Known Bugs
+
+```typescript
+test.skip(true,
+  'DEF-001: {Description}. ' +
+  'Expected: {expected behavior}. ' +
+  'Actual: {buggy behavior}. ' +
+  'Reactivate when ADO #{WI_ID} / DEF-001 is resolved.'
+);
+```
+
+Never make a test pass around a bug. Document the expected behavior in the skip body.
+
+---
+
+## Pattern 6: beforeAll Provisioning Ratio
+
+```typescript
+test.beforeAll(async ({ browser }) => {
+  const page = await browser.newPage();
+  // If 3 tests each consume 1 record, create at least 3
+  const records: string[] = [];
+  for (let i = 0; i < 3; i++) {
+    const id = await createRecord(page, `Record-${EXEC_IDX}-${i}`);
+    records.push(id);
+  }
+});
+```
+
+---
+
+## Pattern 7: Password Injection (Trace Safety)
+
+```typescript
+// Passwords must be set via evaluate() to keep them out of Playwright traces
+await page.evaluate(
+  ([selector, pwd]) => {
+    (document.querySelector(selector) as HTMLInputElement).value = pwd;
+  },
+  [process.env.QA_LOGIN_PASSWORD_SELECTOR, process.env.QA_USER_PASSWORD]
+);
+```
+
+---
+
+## Coverage Mapping Template
+
+Create `COVERAGE-MAPPING.md` in the test directory after automation is complete:
+
+```markdown
+# Coverage Mapping — {Module}
+
+| TC ID | Title | Spec file | Playwright spec | Status |
+|-------|-------|-----------|----------------|--------|
+| TC-MOD-SUB-001 | {title} | 05-test-scenarios.md | tests/{module}/{file}.spec.ts | ✅ Automated |
+| TC-MOD-SUB-002 | {title} | 05-test-scenarios.md | — | ⛔ BLOCKED-PERMISSIONS |
+| TC-MOD-SUB-003 | {title} | 05-test-scenarios.md | — | 🔲 PENDING-CODE |
+```