@keber/qa-framework 1.0.4

package/docs/comparison-matrix.md

@@ -0,0 +1,131 @@
+# docs/comparison-matrix.md
+
+**Phase 1 Artifact** — Source: analysis of two embedded QA implementations.
+
+> **Note**: This document covers the comparative inventory of Repo A (`redacted-repo-web`) and Repo B (`redacted-repo`). It was the primary input for Phase 2 (classification) and Phase 3 (framework design).
+
+---
+
+## Repository Profiles
+
+| Attribute | Repo A (`redacted-repo-web`) | Repo B (`redacted-repo`) |
+|---|---|---|
+| **Application stack** | ASP.NET MVC 5 + Web API 2 + SQL Server | Blazor WebAssembly + .NET + Radzen |
+| **UI component library** | jQuery, Select2, toastr, SweetAlert2, Metronic | Radzen Blazor components |
+| **Authentication** | Form login: RUT + password, `/Seguridad/Login` | Form login: email + password, Blazor identity |
+| **QA environment URL** | `https://redactedURL.com` | `https://redactedURL.com` |
+| **Modules documented** | 1 module (RCL Colación Fría), 7 submodules | 4 modules (Personas, Operación, Sistema, Reportes), 17+ submodules |
+| **Test cases documented** | ~160 TCs across 7 submodules | ~894 TCs across 17+ submodules |
+| **ADO integration level** | Partial (Wiki tools, Plan IDs referenced) | Full (pipeline, reporter, inject-ado-ids.ps1, module-registry.json) |
+| **Playwright suite** | Yes (Sprint 40: 26 P0 tests, 24 pass/2 skip) | Yes (67 automated, 53 pass/2 fail/12 skip) |
+| **Standards folder** | No (`00-standards/` not present) | Yes (naming-conventions, bug-template, TC-template, test-data-guidelines) |
+| **Session retrospectives** | Yes (06-defects used, 2 defects filed) | Extensive (SESSION-SUMMARY-* files at qa root) |
+| **08-azure-integration** | Not present | Fully implemented |
+
+---
+
+## Item-by-Item Comparison Matrix
+
+| Item | Repo A | Repo B | Common? | Different? | Generalizable? | Needs Param? | Out of Framework? |
+|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| **8-folder qa/ structure (00–08)** | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
+| **6-file submodule template** | ✅ | ✅ | ✅ | Minor | ✅ | ❌ | ❌ |
+| `00-inventory.md` | ✅ | ✅ | ✅ | ❌ | ✅ | Module code | ❌ |
+| `01-business-rules.md` | ✅ (RN-* IDs) | ✅ (RN-* IDs) | ✅ | ❌ | ✅ | Module code | ❌ |
+| `02-workflows.md` (FL-* IDs) | ✅ | ✅ | ✅ | ❌ | ✅ | Module code | ❌ |
+| `03-roles-permissions.md` | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
+| `04-test-data.md` | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
+| `05-test-scenarios.md` (TC-* IDs) | ✅ | ✅ | ✅ | ❌ | ✅ | Module code | ❌ |
+| **Agent instructions: module analysis** | ✅ (766 lines) | ✅ (834 lines) | ✅ | Minor | ✅ | Project URL | ❌ |
+| **Agent instructions: E2E generation** | ✅ (Sprint-specific) | ❌ | ❌ | Major | ✅ (extract pattern) | Module scope | ❌ |
+| **ADO integration instructions** | ❌ | ✅ (AGENT-ADO-INTEGRATION.md) | ❌ | — | ✅ | Org/Project/IDs | ❌ |
+| **QA Structure Guide** | ✅ (994 lines) | ✅ (1018 lines) | ✅ | Minor wording | ✅ | ❌ | ❌ |
+| **00-standards/ folder** | ❌ | ✅ | ❌ | — | ✅ (adopt from B) | ❌ | ❌ |
+| **Naming conventions doc** | Embedded in structure guide | Standalone doc | Partial | Yes | ✅ | Module codes | ❌ |
+| **Bug report template** | ❌ | ✅ | ❌ | — | ✅ | ❌ | ❌ |
+| **Test case template** | ❌ | ✅ | ❌ | — | ✅ | ❌ | ❌ |
+| **Test data guidelines** | ❌ | ✅ | ❌ | — | ✅ | Env var names | ❌ |
+| **session-summary.md pattern** | ❌ | ✅ (2 examples) | ❌ | — | ✅ | ❌ | ❌ |
+| **Multi-role globalSetup (Playwright)** | ✅ (3 roles, sequential) | ✅ (1 role, single) | ✅ (pattern) | Parameters | ✅ | Role count, selectors | ❌ |
+| **storageState auth reuse** | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
+| **EXEC_IDX / RUN_SLOT date pattern** | ✅ | ❌ (uses Date.now() only) | ❌ | Major | ✅ (adopt from A) | ❌ | ❌ |
+| **3-layer email validation strategy** | ✅ | ❌ | ❌ | — | ✅ (domain-agnostic) | ❌ | Optional |
+| **playwright-azure-reporter** | ❌ | ✅ | ❌ | — | ✅ | planId, suiteId | Optional |
+| **inject-ado-ids.ps1** | ❌ | ✅ | ❌ | — | ✅ | ❌ | Optional |
+| **ADO pipeline (azure-pipeline-qa.yml)** | ❌ | ✅ (367 lines) | ❌ | — | ✅ | Org, project, branch | Optional |
+| **module-registry.json** | ❌ | ✅ | ❌ | — | ✅ | Module paths/IDs | Optional |
+| **ado-ids-mapping.json** | ❌ | ✅ | ❌ | — | ✅ | All IDs | Optional |
+| **Debug artifact policy** | ❌ | ✅ (DEBUG-ARTIFACTS-POLICY.md) | ❌ | — | ✅ | ❌ | ❌ |
+| **UI as source of truth principle** | Implied | ✅ (UI_COMO_FUENTE_DE_VERDAD.md) | ✅ (implicit) | Documentation | ✅ | ❌ | ❌ |
+| **TC origin classification** | ❌ | ✅ (UI-OBSERVADO/PENDIENTE-CODE/BLOQUEADO-PERMISOS) | ❌ | — | ✅ | ❌ | ❌ |
+| **BLAZOR_WASM_AUTOMATION.md** | ❌ | ✅ | ❌ | — | Partial | UI library | Optional |
+| **RADZEN_COMPONENTS_PLAYWRIGHT.md** | ❌ | ✅ | ❌ | — | Partial | UI library | Optional (Blazor adapter) |
+| **DB schema documentation pattern** | ✅ (Mermaid ER) | ❌ | ❌ | — | ✅ (pattern only) | DB naming | Optional |
+| **Correctness criteria** | ✅ (≥2 passes with different EXEC_IDX) | ❌ | ❌ | — | ✅ | ❌ | ❌ |
+| **5-layer debugging methodology** | ✅ | ❌ | ❌ | — | ✅ | ❌ | ❌ |
+| **Page Object Model (POM) pattern** | ✅ (4 POMs) | Inline (no POMs) | Partial | Approach | ✅ (recommend POMs) | ❌ | ❌ |
+| **06-defects/ directory** | ✅ (2 active DEFs used) | ✅ (folder only, no active DEFs) | ✅ | Usage | Conditional | ❌ | Optional |
+| **Seed specs pattern** | ❌ | ✅ (seeds/*.spec.ts excluded from runs) | ❌ | — | ✅ | ❌ | Optional |
+| **automation-feasibility-analysis.md** | ❌ | ✅ | ❌ | — | ✅ | ❌ | ❌ |
+| **Coverage mapping (COVERAGE-MAPPING.md)** | ❌ | ✅ | ❌ | — | ✅ | ADO IDs | Optional |
+| **Project-specific URLs** | ✅ (redacted) | ✅ (redacted) | Both | Different | ❌ (parameterize) | QA_BASE_URL | ❌ |
+| **ADO org / project names** | redacted-organization / redacted-project | redacted-organization / redacted-project | Org same | Project different | ❌ (parameterize) | ADO_ORG, ADO_PROJECT | ❌ |
+| **Login selectors** | `#m_login_signin_submit` (Metronic) | `#email-input`, `#password-input` (Blazor) | Different | Major | ❌ (parameterize) | LOGIN_EMAIL_SELECTOR, etc. | ❌ |
+| **Application-specific form selectors** | `#tbl_redacted`, `#redacted` | `.rz-data-row`, `input.rz-textbox` | None | Major | ❌ | ❌ | Yes (project adapters) |
+| **Test user DNIs / real personal data** | DNIs in test-data.ts, personas.md | None (uses env vars) | None | Major | ❌ | ❌ | Yes (project data) |
+| **Module-specific business data** | redacted | redacted | None | Major | ❌ | ❌ | Yes (project data) |
+| **Sprint references** | redacted sprint | No sprints (session-based) | None | Different | ❌ | ❌ | Yes (project context) |
+| **Developer names** | redacted names | None | None | — | ❌ | ❌ | Yes (project context) |
+| **Internal SOAP email service** | EnvioMailSoapClient | None | None | — | ❌ | ❌ | Yes |
+
+---
+
+## Summary Counts
+
+| Category | Count |
+|---|---|
+| Items common to both repos | 14 |
+| Items only in Repo A, generalizable | 8 |
+| Items only in Repo B, generalizable | 16 |
+| Items that need parameterization | 12 |
+| Items that are purely project-specific (out of framework) | 10+ |
+| Optional/plugin items | 9 |
+
+---
+
+## Key Differences That Create Design Tensions
+
+### 1. Login credential type: RUT vs email
+
+- Repo A uses RUT (Chilean national ID) as username
+- Repo B uses email as username
+- **Resolution**: Parameterize via `QA_USER_EMAIL` and `QA_USER_PASSWORD` env vars; the login selector set is a project config parameter in `qa-framework.config.json`
+
+### 2. Single-role vs multi-role globalSetup
+
+- Repo A: 3 roles (solicitante, encargado, admin) saved sequentially to `.auth/{role}.json`
+- Repo B: 1 role saved to `.auth/session.json`
+- **Resolution**: Scaffold supports both. Config parameter `testUsers[]` drives how many roles are set up. Template defaults to single-role; multi-role is documented as an extension.
+
+### 3. ADO integration depth
+
+- Repo A: ADO IDs referenced in comments; no pipeline or reporter
+- Repo B: Full pipeline, playwright-azure-reporter, inject-ado-ids.ps1, module-registry.json
+- **Resolution**: ADO integration is a fully optional plugin (`integrations/ado-powershell/`). The framework core does not require it.
+
+### 4. Standards folder
+
+- Repo B has a mature `00-standards/` folder absent from Repo A
+- **Resolution**: Adopt the standards from Repo B as the framework baseline, since they are the more complete and generalized set.
+
+### 5. `06-defects/` directory
+
+- Repo A: Used actively (2 bugs filed, test skips reference DEF-IDs)
+- Repo B: Folder exists but no active defect files
+- **Resolution**: See `docs/generalization-decisions.md` §7 for the full evaluation. Decision: **keep as optional, not required by framework core**.
+
+### 6. Test naming convention
+
+- Repo A: `[Nx] Title @Pp` (sprint-relative sequential numbering)
+- Repo B: `[ADO_ID] Title` (ADO Work Item ID prefix)
+- **Resolution**: Framework uses `[{ID}] {title} @{priority}` where `{ID}` defaults to a local TC-ID (e.g., TC-MOD-SUB-001) and is replaced with ADO WI ID when ADO integration is enabled.

package/docs/final-report.md

@@ -0,0 +1,279 @@
+# Phase 6 — Final Report: keber/qa-framework
+
+**Date**: 2025-01-15
+**Version analyzed**: 1.0.0
+**Prepared by**: GitHub Copilot (agent output)
+
+---
+
+## 1. Executive Summary
+
+Two production QA repositories were analyzed, compared, and generalized into a single,
+decoupled, npm-installable QA framework: `keber/qa-framework`.
+
+The framework:
+- Encodes all conventions shared between both source repositories
+- Removes all project-specific hardcoding
+- Is designed for consumption by an IDE agent (GitHub Copilot / similar)
+- Provides a complete opinionated structure: docs, templates, agent instructions, integrations, CLI, and examples
+- Supports both ADO-integrated and ADO-free project configurations
+- Is ready for `npm publish` or local `npm link` installation
+
+Both source repositories can migrate to this framework with minimal disruption using the
+procedures documented in `MIGRATION-NOTES.md`.
+
+---
+
+## 2. Source Repository Findings
+
+### Repo A — redacted-repo-web
+
+| Category | Finding |
+|----------|---------|
+| Stack | ASP.NET MVC 5 + Web API 2 + SQL Server |
+| Frontend | jQuery + Select2 + toastr + SweetAlert2 + Metronic |
+| Auth | RUT-based login (`#m_login_signin_submit`) |
+| QA maturity | Medium — spec files present, standards missing |
+| Automation | Sprint 40: 26 P0 tests (24 pass, 2 skip via DEF-001/DEF-002) |
+| ADO | Referenced (Plan 21992/Suite 21993) but not reporter-integrated |
+| Modules | 7 submodules documented (~160 TCs across 82 QA files) |
+| Strengths | EXEC_IDX pattern, POM structure, multi-role auth, skip+DEF |
+| Gaps | No `00-standards/`, no `08-azure-integration/`, no session summaries |
+
+### Repo B — redacted-repo
+
+| Category | Finding |
+|----------|---------|
+| Stack | Blazor WebAssembly + .NET + Radzen components |
+| Auth | Email-based (`#email-input` / `#password-input`) |
+| QA maturity | High — complete standards folder, ADO fully integrated |
+| Automation | 67 automated TCs (53 pass / 2 fail / 12 skip) |
+| ADO | Full: Plans 22304/22794/22875, WI IDs 22957–23034, bi-directional sync |
+| Modules | 4 modules × 17+ submodules (~894 TCs) |
+| Strengths | `00-standards/`, `08-azure-integration/`, module-registry, ado reporter |
+| Gaps | No EXEC_IDX pattern, no POM convention, no session summaries |
+
+---
+
+## 3. Common Elements (Adopted by Framework)
+
+| Element | Source | Decision |
+|---------|--------|----------|
+| 6-file submodule spec pattern | Both | Core framework pattern |
+| Playwright `@playwright/test` | Both | Standard automation library |
+| Priority levels P0–P3 | Both | Universal tagging via `@PX` |
+| `test.skip()` + DEF reference | Repo A | Adopted as standard skip convention |
+| `06-defects/` folder | Repo A | Optional — recommended without ADO |
+| `00-standards/` folder | Repo B | Adopted; templates included |
+| `08-azure-integration/` folder | Repo B | Optional; fully documented |
+| `module-registry.json` | Repo B | Included in ADO integration |
+| ADO reporter integration | Repo B | Optional plugin (peer dep) |
+| Multi-role globalSetup | Repo A | Default single-role; multi-role documented |
+| EXEC_IDX uniqueness pattern | Repo A | Adopted; added to test-helpers.ts |
+| POM page object pattern | Repo A | Recommended; not enforced |
+| ADO inject-ado-ids script | Repo B | Generalized; included in integrations/ |
+| Azure Pipeline QA template | Repo B | Generalized; included in integrations/ |
+
+---
+
+## 4. Project-Specific Elements (Excluded from Framework)
+
+| Element | Source | Disposition |
+|---------|--------|-------------|
+| `/Seguridad/Login` route | Repo A | Parameterized via `QA_LOGIN_PATH` env var |
+| `#m_login_signin_submit` selector | Repo A | Parameterized via `QA_LOGIN_SUBMIT_SELECTOR` |
+| RUT-based username field | Repo A | Parameterized via `QA_LOGIN_EMAIL_SELECTOR` |
+| `#email-input` / `#password-input` | Repo B | Parameterized via env vars |
+| ADO Plan IDs (21992, 22304, etc.) | Both | Parameterized via `QA_ADO_PLAN_ID` |
+| ADO WI IDs (22957–23034) | Repo B | Injected per-project via inject-ado-ids.ps1 |
+| Specific module names/selectors | Both | Remain in project-specific spec files |
+| Sprint numbers | Both | Remain in project-specific test plans |
+| Metronic theme selectors | Repo A | Project-specific; documented in Repo A specs |
+| Radzen component patterns | Repo B | Documented in Repo B's BLAZOR guides |
+| `redacted-package` package name | Repo A | Replaced by project-configurable name |
+
+---
+
+## 5. Generalization Decisions Summary
+
+Fifteen formal decisions were made and documented in `docs/generalization-decisions.md`:
+
+| # | Topic | Decision |
+|---|-------|----------|
+| §1 | Login selectors | Fully parameterized via env vars |
+| §2 | TC naming | Unified `[TC-M-S-NNN]` + inject-ado for ADO |
+| §3 | Spec file structure | 6-file pattern adopted from both repos |
+| §4 | Auth storageState | Single-role default; multi-role documented |
+| §5 | EXEC_IDX | Adopted from Repo A; added to test-helpers.ts |
+| §6 | ADO integration | Optional plugin; fully documented |
+| §7 | `06-defects/` | Optional; ADO-conditional usage documented |
+| §8 | POM pattern | Recommended (Repo A); inline acceptable |
+| §9 | Standards folder | Adopted from Repo B (more complete) |
+| §10 | Session summaries | New addition to both repos' patterns |
+| §11 | Priority levels | P0–P3 universal; `@P0` grep-tag convention |
+| §12 | `08-azure-integration/` | Optional; fully documented scripts |
+| §13 | CI pipeline | Parameterized `azure-pipeline-qa.yml` |
+| §14 | Framework distribution | npm package (`keber/qa-framework`) |
+| §15 | ADO vs local defects | Decision tree documented; both patterns valid |
+
+---
+
+## 6. Final Framework Structure
+
+```
+qa-framework/
+├── package.json                          ← npm package, CLI entry
+├── qa-framework.config.json             ← example project config
+├── README.md
+├── CHANGELOG.md
+├── MIGRATION-NOTES.md
+│
+├── docs/
+│   ├── architecture.md                  ← 4-layer component map
+│   ├── comparison-matrix.md             ← Phase 1 artifact
+│   ├── generalization-decisions.md      ← Phase 2 artifact (15 decisions)
+│   ├── installation.md                  ← 3 install options
+│   ├── spec-driven-philosophy.md        ← Core methodology
+│   ├── folder-structure-guide.md        ← Full folder reference
+│   ├── usage-with-agent.md              ← Agent prompt patterns
+│   └── final-report.md                  ← This file
+│
+├── agent-instructions/                  ← Consumed by Copilot agent
+│   ├── 00-module-analysis.md
+│   ├── 01-spec-generation.md
+│   ├── 02-test-plan-generation.md
+│   ├── 03-test-case-generation.md
+│   ├── 04-automation-generation.md
+│   ├── 05-ado-integration.md
+│   └── 06-maintenance.md
+│
+├── templates/
+│   ├── specification/                   ← 6-file submodule template set
+│   │   ├── 00-inventory.md
+│   │   ├── 01-business-rules.md
+│   │   ├── 02-workflows.md
+│   │   ├── 03-roles-permissions.md
+│   │   ├── 04-test-data.md
+│   │   └── 05-test-scenarios.md
+│   ├── test-plan.md
+│   ├── test-case.md
+│   ├── execution-report.md
+│   ├── defect-report.md
+│   ├── session-summary.md
+│   └── automation-scaffold/
+│       ├── package.json
+│       ├── playwright.config.ts
+│       ├── global-setup.ts
+│       ├── .env.example
+│       └── fixtures/
+│           ├── auth.ts
+│           └── test-helpers.ts
+│
+├── integrations/
+│   ├── playwright/README.md
+│   ├── playwright-azure-reporter/README.md
+│   └── ado-powershell/
+│       ├── README.md
+│       ├── scripts/
+│       │   ├── inject-ado-ids.ps1
+│       │   ├── create-testplan-from-mapping.ps1
+│       │   └── sync-ado-titles.ps1
+│       └── pipelines/
+│           └── azure-pipeline-qa.yml
+│
+├── examples/
+│   └── module-example/
+│       ├── README.md
+│       └── suppliers/
+│           ├── 00-inventory.md
+│           └── suppliers-create.spec.ts
+│
+└── scripts/                             ← CLI implementation
+    ├── cli.js
+    ├── init.js
+    ├── generate.js
+    └── validate.js
+```
+
+**File count**: 48 files across 20 directories
+
+---
+
+## 7. Risks
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|-----------|--------|------------|
+| New project uses different login mechanism not covered by parameterization | Medium | Low | Add custom `QA_LOGIN_PATH`, selectors in `.env`; override `global-setup.ts` |
+| ADO API version changes break ps1 scripts | Low | Medium | Scripts use `api-version=7.1`; update when needed |
+| npm registry not available (intranet) | Medium | Medium | Use Option B (clone) or Option C (manual copy) from installation.md |
+| Agent hallucinates selectors not in inventory | Medium | High | Always run agents with live browser access; validate every selector before commit |
+| Framework version conflicts between projects | Low | Low | Each project pins its own version in `package.json` |
+
+---
+
+## 8. Assumptions
+
+1. Projects use Playwright `>=1.40.0` for E2E automation.
+2. CI environment supports Node.js `>=18`.
+3. ADO integration requires a PAT with `Work Items R/W` + `Test Management R/W`.
+4. QA engineers have read access to the application under test.
+5. Spec files are written by agents but reviewed and committed by humans.
+6. Login forms are HTTP/HTTPS web forms accessible from the Playwright browser context.
+
+---
+
+## 9. Open Items
+
+| Item | Priority | Owner | Notes |
+|------|----------|-------|-------|
+| Add `examples/module-example/suppliers/01-business-rules.md` through `05-test-scenarios.md` | Low | QA team | Example shows inventory + spec only; add full 6-file set |
+| Publish to npm registry | Medium | DevOps | Requires `npm publish` with org scope `keber` |
+| Add `--watch` mode to `validate.js` | Low | Framework maintainer | Useful in local dev |
+| Radzen/Blazor guide | Medium | QA team | Extract from Repo B's `BLAZOR_WASM_AUTOMATION.md` into `docs/` |
+| Metronic/jQuery guide | Low | QA team | Extract from Repo A patterns into `docs/` |
+| Jest unit tests for CLI scripts | Low | Framework maintainer | Add `scripts/__tests__/` |
+
+---
+
+## 10. Recommended Next Steps
+
+### Immediate (Week 1)
+
+1. **Install the package** in both source projects using `npm link` for local testing:
+   ```bash
+   cd c:\Users\keber.flores\source\repos\qa-framework
+   npm link
+   cd c:\Users\keber.flores\source\repos\redacted-repo-web\qa\07-automation
+   npm link keber/qa-framework
+   ```
+
+2. **Run `qa-framework validate`** in Repo A:
+   ```bash
+   npx keber/qa-framework validate
+   ```
+   Address any reported gaps (missing `00-standards/`, etc.)
+
+3. **Add `copilot-instructions.md`** to both repos pointing to agent instructions:
+   ```markdown
+   When generating QA artifacts, follow:
+   node_modules/keber/qa-framework/agent-instructions/00-module-analysis.md
+   ```
+
+### Short-term (Month 1)
+
+4. **Migrate Repo A** to the full framework structure using `MIGRATION-NOTES.md §Pattern A`
+5. **Migrate Repo B** using `MIGRATION-NOTES.md §Pattern B`
+6. **Add the 5 remaining example spec files** to `examples/module-example/suppliers/`
+7. **Publish to internal npm registry** (or GitHub Packages)
+
+### Medium-term (Quarter 1)
+
+8. **Add Blazor/Radzen guide** to `docs/` (extracted from Repo B)
+9. **Add Metronic/MVC guide** to `docs/` (extracted from Repo A)
+10. **Set up CI** using `integrations/ado-powershell/pipelines/azure-pipeline-qa.yml`
+11. **Version 1.1.0**: Add `--watch` validation mode and `generate spec` interactive prompts
+
+---
+
+*This report was generated by GitHub Copilot as the Phase 6 deliverable of the
+`keber/qa-framework` analysis and implementation project.*