@keber/qa-framework 1.0.4

package/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to `qa-framework` will be documented in this file.
+
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+Versioning follows [Semantic Versioning](https://semver.org/).
+
+---
+
+## [1.0.0] - 2026-03-04
+
+### Added
+
+- **Framework scaffold** — initial `qa/` directory tree with 9 numbered folders (00–08)
+- **6-file submodule template set** — `00-inventory` through `05-test-scenarios`
+- **Agent instructions** — 7 purpose-specific Markdown instruction files for IDE agents
+- **Standards** — naming conventions, bug report template, test case template, test data guidelines
+- **Automation scaffold** — generalized Playwright `playwright.config.ts`, `global-setup.ts`, `.env.example`, `fixtures/auth.ts`
+- **Optional integrations** — stubs and READMEs for Playwright, `playwright-azure-reporter`, and ADO PowerShell
+- **CLI entry point** — `qa-framework init`, `generate`, `validate` commands
+- **Documentation** — architecture, comparison matrix, generalization decisions, installation guide, usage-with-agent guide, spec-driven philosophy, folder structure guide
+- **Migration notes** — `MIGRATION-NOTES.md` for projects moving from embedded to package-based framework
+
+### Source repositories
+
+This version was bootstrapped from analysis of two existing embedded QA implementations:
+
+- `redacted-repo` (Repo A) — ASP.NET MVC 5 + jQuery + toastr + Sprint-based E2E
+- `redacted-repo` (Repo B) — Blazor WebAssembly + Radzen + full ADO pipeline integration
+
+All project-specific content (URLs, ADO IDs, user credentials, module names, form selectors)
+has been removed or parameterized. See `docs/comparison-matrix.md` and `docs/generalization-decisions.md`.
+
+---
+
+## [Unreleased]
+
+### Added
+
+- **Stage 3.5 — Test Stabilization** (`agent-instructions/04b-test-stabilization.md`) — new agent
+  instruction file for exhaustively reviewing and stabilizing generated Playwright tests before
+  ADO integration. Covers baseline collection, failure classification (9 categories), false-positive
+  validation, confidence scoring (≥90% exit threshold), `STABILIZATION-REPORT.md` schema, and
+  upstream artifact update rules. Sits between Stage 4 (Automation Generation) and Stage 5
+  (ADO Integration) and is mandatory before any CI pipeline registration.
+
+### Planned
+
+- `qa-framework generate spec <module-name>` — scaffold a full spec set from CLI
+- `qa-framework validate` — check `qa/` structure for compliance with conventions
+- `qa-framework upgrade` — migrate old embedded structure to package-based layout
+- Integration: `jest` support alongside Playwright
+- Integration: GitHub Actions pipeline template (alongside existing Azure Pipelines template)

package/README.md

@@ -0,0 +1,233 @@
+# qa-framework
+
+> Reusable, installable, agent-oriented QA framework for spec-driven automated testing.
+
+---
+
+## What is this?
+
+`qa-framework` is a methodology package that provides:
+
+- **A standardized `qa/` directory structure** for any project
+- **Reusable templates** for specifications, test plans, test cases, defect reports, and execution reports
+- **Agent instructions** (for GitHub Copilot and equivalent IDE agents) that enable AI-assisted QA work
+- **Optional integrations** for Playwright and Azure DevOps
+- **Bootstrap commands** to scaffold the structure into any project in seconds
+
+This framework was designed to be:
+
+| Property | Description |
+|---|---|
+| **Reusable** | Not tied to any specific project, domain, or tech stack |
+| **Decoupled** | Core is independent from Azure DevOps, Playwright, or any other tool |
+| **Agent-oriented** | Instructions are written for IDE agents (Copilot, etc.) to consume |
+| **Spec-driven** | Every automation artifact traces back to a specification |
+| **Extensible** | Adapters/plugins for optional integrations are well-separated |
+
+---
+
+## Quick Start
+
+### Option A: Install from npm (recommended)
+
+```bash
+npm install --save-dev github:keber/qa-framework
+npx qa-framework init
+```
+
+### Option B: Clone directly (during early adoption)
+
+```bash
+# Clone into your project's tools directory or as a submodule
+git clone <this-repo> tools/qa-framework
+node tools/qa-framework/scripts/cli.js init
+```
+
+The `init` command will:
+
+1. Create the `qa/` directory tree in your project root
+2. Copy all base templates into place
+3. Create `qa/qa-framework.config.json` for project-specific settings
+4. Optionally set up Playwright and Azure DevOps integrations
+
+---
+
+## Repository Structure (this package)
+
+```
+qa-framework/
+├── README.md                         ← This file
+├── CHANGELOG.md                      ← Version history
+├── MIGRATION-NOTES.md                ← How to migrate from embedded to package
+├── package.json                      ← npm descriptor
+├── qa-framework.config.json          ← Example config (copy to project)
+│
+├── docs/                             ← Framework documentation
+│   ├── architecture.md               ← Design decisions and component map
+│   ├── comparison-matrix.md          ← Source-repo analysis (Phase 1 artifact)
+│   ├── generalization-decisions.md   ← What was abstracted and why (Phase 2 artifact)
+│   ├── installation.md               ← Detailed installation guide
+│   ├── usage-with-agent.md           ← How to use the framework with an IDE agent
+│   ├── spec-driven-philosophy.md     ← Core QA methodology
+│   └── folder-structure-guide.md     ← Explanation of every qa/ folder
+│
+├── agent-instructions/               ← Instructions for AI agents, per task
+│   ├── 00-module-analysis.md
+│   ├── 01-spec-generation.md
+│   ├── 02-test-plan-generation.md
+│   ├── 03-test-case-generation.md
+│   ├── 04-automation-generation.md
+│   ├── 04b-test-stabilization.md
+│   ├── 05-ado-integration.md
+│   └── 06-maintenance.md
+│
+├── templates/                        ← Reusable file templates
+│   ├── specification/                ← 6-file submodule set
+│   │   ├── 00-inventory.md
+│   │   ├── 01-business-rules.md
+│   │   ├── 02-workflows.md
+│   │   ├── 03-roles-permissions.md
+│   │   ├── 04-test-data.md
+│   │   └── 05-test-scenarios.md
+│   ├── test-plan.md
+│   ├── test-case.md
+│   ├── execution-report.md
+│   ├── defect-report.md
+│   ├── session-summary.md
+│   └── automation-scaffold/          ← Files to bootstrap a Playwright E2E project
+│       ├── package.json
+│       ├── playwright.config.ts
+│       ├── global-setup.ts
+│       ├── .env.example
+│       └── fixtures/
+│           └── auth.ts
+│
+├── integrations/                     ← Optional integration adapters
+│   ├── playwright/
+│   │   └── README.md
+│   ├── playwright-azure-reporter/
+│   │   └── README.md
+│   └── ado-powershell/
+│       ├── README.md
+│       └── scripts/
+│           ├── inject-ado-ids.ps1
+│           ├── create-testplan-from-mapping.ps1
+│           └── sync-ado-titles.ps1
+│
+├── examples/                         ← Reference examples (non-project-specific)
+│   └── module-example/
+│       ├── README.md
+│       └── submodule-example/        ← Full 6-file spec example
+│
+└── scripts/                          ← CLI commands
+    ├── cli.js                        ← Entry point (qa-framework <command>)
+    ├── init.js                       ← Scaffold qa/ tree
+    ├── generate.js                   ← Generate individual artifact
+    └── validate.js                   ← Check structure compliance
+```
+
+---
+
+## The `qa/` Directory Structure
+
+When you run `qa-framework init`, this structure is created inside your project:
+
+```
+qa/
+├── README.md                    ← Living index of all QA artifacts
+├── QA-STRUCTURE-GUIDE.md        ← Local copy of the structure guide
+│
+├── 00-guides/                   ← Process guides and agent instructions
+├── 00-standards/                ← Naming conventions, templates, data policies
+├── 01-specifications/           ← Functional specs by module/submodule
+├── 02-test-plans/               ← Test plans (automated + manual)
+├── 03-test-cases/               ← Explicit test case documents
+├── 04-test-data/                ← Test data, fixtures, factories
+├── 05-test-execution/           ← Execution reports and results
+├── 06-defects/                  ← Defect tracking (optional)
+├── 07-automation/               ← Automation code and config
+└── 08-azure-integration/        ← Azure DevOps integration (optional)
+```
+
+See [docs/folder-structure-guide.md](docs/folder-structure-guide.md) for a full explanation of every folder.
+
+---
+
+## Agent Instructions
+
+The `agent-instructions/` folder contains Markdown documents designed for IDE agents
+(such as GitHub Copilot in VS Code). Each file covers one specific task:
+
+| File | Agent Task |
+|---|---|
+| `00-module-analysis.md` | Analyze a module and produce the 6-file spec set |
+| `01-spec-generation.md` | Generate functional specifications |
+| `02-test-plan-generation.md` | Create a test plan for a module |
+| `03-test-case-generation.md` | Generate detailed test cases |
+| `04-automation-generation.md` | Write Playwright E2E tests |
+| `04b-test-stabilization.md` | Exhaustively review and stabilize generated tests (Stage 3.5) |
+| `05-ado-integration.md` | Sync with Azure DevOps Test Plans |
+| `06-maintenance.md` | Update QA artifacts after code changes |
+
+To use: copy the relevant instruction into a Copilot/agent chat, or reference it from a
+`.github/copilot-instructions.md` or equivalent workspace instruction file.
+
+---
+
+## Configuration
+
+`qa-framework init` now bootstraps a default config automatically at `qa/qa-framework.config.json` if none exists.
+
+You can then edit that file, or place a config in your project root (`qa-framework.config.json`) and run `init` again. You can also pass a specific file with `--config <path>`.
+
+Example config:
+
+```json
+{
+  "project": {
+    "name": "my-project",
+    "qaBaseUrl": "https://your-qa-env.example.com"
+  },
+  "modules": [],
+  "integrations": {
+    "playwright": { "enabled": false },
+    "azureDevOps": { "enabled": false }
+  }
+}
+```
+
+See [docs/installation.md](docs/installation.md) for full config documentation.
+
+---
+
+## Design Principles
+
+1. **UI is the source of truth.** Test cases are defined from what is observable in the running QA environment, not from source code or requirements docs alone.
+2. **Spec before automation.** Every automated test traces back to a written specification (TC-ID → spec file).
+3. **Credentials never in documentation.** All credentials are handled via environment variables and sanitized from markdown with `<PLACEHOLDER>`.
+4. **Agent-readable conventions.** File names, folder names, and IDs follow strict, predictable patterns so agents can navigate without guessing.
+5. **Optional integrations don't pollute the core.** Azure DevOps and Playwright configuration live in separate, opt-in sections.
+
+---
+
+## Versioning
+
+This framework follows [Semantic Versioning](https://semver.org/):
+
+- **MAJOR**: breaking changes to the `qa/` directory structure or agent instruction APIs
+- **MINOR**: new templates, new optional integrations, new agent instruction files
+- **PATCH**: bug fixes in CLI scripts, documentation corrections
+
+See [CHANGELOG.md](CHANGELOG.md) for detailed history.
+
+---
+
+## Contributing / Adapting
+
+This framework is designed to be forked and adapted. If you need a project-specific adapter:
+
+1. Create a new `integrations/<adapter-name>/` folder
+2. Place your adapter config and scripts there
+3. Reference it from `qa-framework.config.json` under `integrations`
+
+Do **not** put project-specific configuration inside `agent-instructions/` or `templates/`. Those must remain generic.

package/agent-instructions/00-module-analysis.md

@@ -0,0 +1,263 @@
+# Agent Instructions: Module Analysis
+
+**File**: `agent-instructions/00-module-analysis.md`  
+**Purpose**: Instructions for an AI agent to perform complete functional analysis of an application module and produce the 6-file specification set.
+
+---
+
+## When to use these instructions
+
+Use this instruction set when:
+- Analyzing a new module for the first time
+- A module has been updated and its spec needs to be refreshed
+- Building the initial `01-specifications/` tree for a project
+
+---
+
+## Prerequisites (Agent must verify before starting)
+
+1. Confirm that `qa-framework.config.json` exists and contains `project.qaBaseUrl`
+2. Confirm that `qa/00-standards/naming-conventions.md` is available
+3. Confirm that the Playwright CLI (playwright-mcp or equivalent) is available for browser interaction
+4. Ask the user for:
+   - The module name and URL (or route prefix) to analyze
+   - The submodules to analyze (if known; otherwise discover from the UI)
+   - QA user credentials (handle via env vars — never store in markdown)
+
+**⚠️ SECURITY**: Never write real credentials to any file. If credentials appear in any output, replace immediately with `<PLACEHOLDER>`.
+
+---
+
+## Phase 1 — Preparation
+
+### 1.1 Review existing specs
+
+Check if a `qa/01-specifications/module-{name}/` directory already exists.
+
+- If it does: read all existing files to understand what is already documented. Only re-analyze what has changed.
+- If it does not: create the directory structure.
+
+### 1.2 Generate module code
+
+If no module code exists yet, allocate a unique 2–6 uppercase letter code for the module.  
+Check `qa/00-standards/naming-conventions.md` for existing codes to avoid conflicts.
+
+Example: module "Operación" → code `OPER`; submodule "Catálogos" → code `CAT`
+
+### 1.3 Plan the session
+
+Identify all submodules in scope. A submodule typically corresponds to:
+- A distinct menu item or section in the application
+- A distinct CRUD entity (users, products, orders, etc.)
+- A distinct workflow (approval, registration, reporting)
+
+Document the planned submodules in `qa/01-specifications/module-{name}/README.md` before analysis begins.
+
+---
+
+## Phase 2 — Exploration
+
+For each submodule, perform the following exploration using browser automation (Playwright CLI):
+
+### 2.1 Navigation and route mapping
+
+```
+Navigate to {{QA_BASE_URL}} and log in with the appropriate role
+Navigate to the submodule URL
+Record:
+  - Full URL of each page/view
+  - Page title
+  - Navigation breadcrumb
+```
+
+### 2.2 UI inventory
+
+For each screen in the submodule:
+
+```
+Record all:
+  - Input fields (name, type, required/optional, validation rules visible in UI)
+  - Dropdowns (static or dynamic/AJAX, options if static)
+  - Buttons (action, what they trigger)
+  - Tables/grids (columns, default sorting, pagination)
+  - Modals (trigger condition, fields, confirmation behavior)
+  - Status indicators / badges / alerts
+  - File upload / download controls
+```
+
+### 2.3 API endpoint discovery
+
+```
+Observe network requests during key interactions (create, read, update, delete)
+Record:
+  - Endpoint path (e.g., /api/GetUsers, api/CreateUser)
+  - HTTP method
+  - Request payload shape (field names)
+  - Response shape (key fields)
+```
+
+### 2.4 Role-based access testing
+
+```
+For each configured test user role:
+  Log in as that role
+  Navigate to the submodule
+  Record: can access? (YES / NO / PERMISSION_ERROR)
+  Record: which actions are available vs hidden
+```
+
+### 2.5 Workflow discovery
+
+```
+Execute the primary happy path with appropriate test data
+Record each step
+Identify branch points (e.g., approve vs reject)
+Record state transitions
+```
+
+### 2.6 Business rule discovery
+
+```
+Attempt invalid inputs for each validation
+Record: validation message text, field that shows the error
+Attempt boundary conditions (empty, max length, duplicate, etc.)
+Record: system behavior
+```
+
+---
+
+## Phase 3 — Documentation (6-file output)
+
+Produce the following files for each submodule. Use the templates in `qa/00-standards/` and `templates/specification/`.
+
+### File 00: `00-inventory.md`
+
+Contents:
+- Module header (name, code, URL, deployment date if known)
+- Submodule header (name, code, routes)
+- UI element inventory table
+  - Element name | Type | Required | Notes
+- API endpoint table
+  - Method | Path | Purpose
+- Key identifiers and selectors observed in the UI
+
+### File 01: `01-business-rules.md`
+
+Contents:
+- Each rule as `RN-{MODULE}-{NNN}`: Description
+- Rule type: Validation / Access Control / State Machine / Calculation / Integration
+- Trigger: what user action triggers this rule
+- Error message: exact text shown when rule is violated (if applicable)
+
+Aim for 5–15 business rules per submodule.
+
+### File 02: `02-workflows.md`
+
+Contents:
+- Each major workflow as `FL-{MODULE}-{NNN}`: Title
+- Use ASCII flowchart or Mermaid block diagram
+- Cover: happy path + main alternative paths + error paths
+
+### File 03: `03-roles-permissions.md`
+
+Contents:
+- Role × Feature matrix table
+  - Row: test user role
+  - Column: feature/action (view, create, edit, delete, approve, export, etc.)
+  - Cell: ✅ / ❌ / ⚠️ (limited)
+- Notes on role inheritance or conditional access
+
+### File 04: `04-test-data.md`
+
+Contents:
+- Prerequisites: what data must exist before tests run
+- Data shapes: for each key test scenario, what data is needed
+  - DO NOT include real user credentials
+  - Use env var references for any credential: `process.env.QA_USER_EMAIL`
+- Data dependencies between submodules (if any)
+
+### File 05: `05-test-scenarios.md`
+
+Contents:
+- Summary table at top: total TCs, by priority, by type
+- For each TC:
+  - ID: `TC-{MODULE}-{SUBMODULE}-{NNN}`
+  - Title: brief action description
+  - Priority: P0 / P1 / P2 / P3
+  - Type: Functional / Regression / Security / Negative / Integration
+  - Origin: `UI-OBSERVED` / `PENDING-CODE` / `BLOCKED-PERMISSIONS`
+  - Preconditions
+  - Steps (numbered)
+  - Expected result
+  - Automation feasibility: Yes / Partial / No
+
+Target: **50–85 TCs per submodule** is a well-analyzed submodule.
+
+#### TC coverage must include:
+- ✅ Access control (each role: can access / cannot access)
+- ✅ Happy path (primary workflow end-to-end)
+- ✅ Negative scenarios (required field empty, invalid format, duplicate, etc.)
+- ✅ State transitions (create → edit → delete; pending → approved → rejected)
+- ✅ Cross-module dependencies (if any)
+- ✅ Export/download (if feature exists)
+- ✅ Pagination/search (if feature exists)
+
+TC origin rules:
+- Only write `UI-OBSERVED` if you directly observed the behavior in the QA environment
+- If a feature is documented but not visible in the QA environment: `PENDING-CODE`
+- If you couldn't test it due to permission restrictions: `BLOCKED-PERMISSIONS`
+
+---
+
+## Phase 4 — Review and Session Summary
+
+### 4.1 Completeness checklist
+
+After completing all submodule files, verify:
+
+- [ ] All menu items in the module have a corresponding submodule folder
+- [ ] All TCs marked `UI-OBSERVED` were actually observed in this session
+- [ ] `01-business-rules.md` covers all visible validation behaviors
+- [ ] `03-roles-permissions.md` reflects the access observed for each role
+- [ ] No credentials appear in any file
+- [ ] All TC IDs are unique across the module
+- [ ] Module README lists all submodules with their codes and TC counts
+
+### 4.2 Write session summary
+
+Create `qa/SESSION-SUMMARY-{YYYY-MM-DD}.md` with:
+
+```markdown
+# Session Summary — {YYYY-MM-DD}
+
+## Module analyzed
+{module-name} ({module-code})
+
+## Submodules documented
+| Submodule | Code | TC count | Notes |
+|-----------|------|----------|-------|
+| {name}    | {code} | {N}    | {any notable blocking issues} |
+
+## Files created
+- {list of all files created with their paths}
+
+## Blockers
+- {any features that couldn't be analyzed and why}
+
+## Observations for next session
+- {what to tackle next}
+```
+
+---
+
+## Quality Standards
+
+A well-executed module analysis session:
+
+1. Takes between 2-6 hours depending on module complexity
+2. Produces 6 files per submodule (not more, not less)
+3. Has no hardcoded credentials in any file
+4. Has TC IDs that follow the `TC-{MODULE}-{SUBMODULE}-{NNN}` pattern
+5. Has business rules that are verifiable in automated tests
+6. Has at least 1 P0 TC per submodule (the primary happy path)
+7. Does not invent TCs for behavior that wasn't observed in the QA environment