@keber/qa-framework 1.5.2 → 1.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +67 -8
- package/README.md +150 -76
- package/docs/architecture.md +49 -44
- package/docs/folder-structure-guide.md +80 -23
- package/docs/installation.md +53 -43
- package/docs/usage-with-agent.md +52 -64
- package/integrations/ado-powershell/pipelines/azure-pipeline-qa.yml +1 -1
- package/integrations/playwright/README.md +1 -1
- package/package.json +1 -1
- package/scripts/init.js +22 -12
- package/scripts/upgrade.js +139 -11
- package/skills/qa-ado-integration/SKILL.md +2 -2
- package/skills/qa-automation/SKILL.md +6 -6
- package/skills/qa-maintenance/SKILL.md +2 -2
- package/templates/automation-scaffold/playwright.config.ts +3 -1
- package/agent-instructions/00-module-analysis.md +0 -266
- package/agent-instructions/01-spec-generation.md +0 -281
- package/agent-instructions/02-test-plan-generation.md +0 -205
- package/agent-instructions/03-test-case-generation.md +0 -150
- package/agent-instructions/04-automation-generation.md +0 -343
- package/agent-instructions/04b-test-stabilization.md +0 -309
- package/agent-instructions/05-ado-integration.md +0 -247
- package/agent-instructions/06-maintenance.md +0 -135
- package/docs/comparison-matrix.md +0 -131
- package/docs/generalization-decisions.md +0 -203
package/CHANGELOG.md
CHANGED
|
@@ -33,21 +33,80 @@ has been removed or parameterized. See `docs/comparison-matrix.md` and `docs/gen
|
|
|
33
33
|
|
|
34
34
|
---
|
|
35
35
|
|
|
36
|
-
## [
|
|
36
|
+
## [1.6.2] - 2026-05-05
|
|
37
37
|
|
|
38
38
|
### Added
|
|
39
39
|
|
|
40
|
-
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
40
|
+
- **`tests/helpers/debug/`** — diagnostic scripts directory created by `init` and `upgrade` inside
|
|
41
|
+
`07-automation/e2e/tests/`. Scripts here are excluded from test runs via `testIgnore`.
|
|
42
|
+
- **`tests/seeds/`** — data seeding specs directory created by `init` and `upgrade` inside
|
|
43
|
+
`07-automation/e2e/tests/`. Excluded from test runs via `testIgnore`.
|
|
44
|
+
- **`testIgnore`** in `playwright.config.ts` template — excludes `**/helpers/debug/**` and
|
|
45
|
+
`**/seeds/**` from Playwright test discovery.
|
|
46
|
+
- **`upgrade` step 5e** — patches existing `playwright.config.ts` in `e2e/`: replaces
|
|
47
|
+
`testDir: '.'` with `'./tests'`, injects `testIgnore` if missing, creates `helpers/debug/`
|
|
48
|
+
and `seeds/` if absent.
|
|
49
|
+
- **Dual-Track Pipeline** section in `docs/folder-structure-guide.md` and `qa/QA-STRUCTURE-GUIDE.md`
|
|
50
|
+
— documents the automation track and manual track as parallel paths sharing `01-specifications/`
|
|
51
|
+
as source of truth. Includes entry point table (3 scenarios) and track selection guidance.
|
|
52
|
+
Clarifies that `03-test-cases/` is the manual track artifact and `04-test-data/` is optional.
|
|
53
|
+
|
|
54
|
+
### Fixed
|
|
55
|
+
|
|
56
|
+
- `qa/QA-STRUCTURE-GUIDE.md` — removed stale reference to deleted `docs/generalization-decisions.md`
|
|
57
|
+
from the `06-defects/` section.
|
|
58
|
+
- `README.md` — pipeline table updated with Track column; `03-test-cases/` and `04-test-data/`
|
|
59
|
+
annotated as manual-track / optional in the directory tree.
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## [1.6.0] - 2026-04-29
|
|
64
|
+
|
|
65
|
+
### Changed (breaking - automated by `upgrade`)
|
|
66
|
+
|
|
67
|
+
- **`07-automation/` restructured**: the Playwright project root moves from `07-automation/` to
|
|
68
|
+
`07-automation/e2e/`. Affected files: `playwright.config.ts`, `global-setup.ts`, `package.json`,
|
|
69
|
+
`.env.example`, `fixtures/`. Spec stubs move from `07-automation/e2e/{module}/` to
|
|
70
|
+
`07-automation/e2e/tests/{module}/`. Run `npx qa-framework upgrade` to migrate automatically.
|
|
71
|
+
- `testDir` in `playwright.config.ts` template is now `'./tests'` (was `'.'`).
|
|
72
|
+
- `QA_DIR` in the Azure Pipeline template now points to `qa/07-automation/e2e` (was `qa/07-automation`).
|
|
73
|
+
- CI commands: `cd qa/07-automation/e2e && npm install && npx playwright install chromium`.
|
|
74
|
+
|
|
75
|
+
### Removed
|
|
76
|
+
|
|
77
|
+
- `agent-instructions/` directory (legacy monolithic per-task instruction files) — superseded by `skills/`
|
|
78
|
+
architecture since v1.5.0. All behavior is now in `skills/*/SKILL.md` + `references/`.
|
|
79
|
+
- `docs/comparison-matrix.md` — superseded by `docs/skills-architecture.md`.
|
|
80
|
+
- `docs/generalization-decisions.md` — internal design notes; no longer relevant.
|
|
81
|
+
|
|
82
|
+
### Documentation
|
|
83
|
+
|
|
84
|
+
- `docs/architecture.md` — updated for v1.6.0: correct 07-automation/e2e/ tree, removed blazor-radzen
|
|
85
|
+
stub, added `.github/skills/` and `memory/` to target project box.
|
|
86
|
+
- `docs/installation.md` — rewritten: correct `npm install` command, no interactive prompts, correct
|
|
87
|
+
directory tree with e2e/integration/load, added `upgrade` command section.
|
|
88
|
+
- `docs/usage-with-agent.md` — rewritten: all `00-guides/` references replaced with `.github/skills/`,
|
|
89
|
+
session workflows updated to reflect skills model, DO/DON'T list corrected (e2e/ is now correct run location).
|
|
90
|
+
- `README.md` — version bumped, `agent-instructions/` entry removed from package tree.
|
|
91
|
+
|
|
92
|
+
### Added
|
|
93
|
+
|
|
94
|
+
- `07-automation/integration/` — placeholder for API/integration tests (k6, JMeter, Azure Load Testing);
|
|
95
|
+
created on `init` and `upgrade`.
|
|
96
|
+
- `07-automation/load/` — placeholder for load and performance tests; created on `init` and `upgrade`.
|
|
97
|
+
- **`scripts/upgrade.js` migration section 5** — detects old v1.5.x structure and relocates all
|
|
98
|
+
affected files non-destructively (moves only when destination does not exist; warns on conflict).
|
|
99
|
+
- **Stage 5b - Test Stabilization** (`skills/qa-test-stabilization/`) — skill for exhaustively
|
|
100
|
+
reviewing and stabilizing generated Playwright tests before ADO integration. Covers baseline
|
|
101
|
+
collection, failure classification (9 categories), false-positive validation, confidence scoring
|
|
102
|
+
(>=90% exit threshold), `STABILIZATION-REPORT.md` schema, and upstream artifact update rules.
|
|
46
103
|
|
|
47
104
|
### Planned
|
|
48
105
|
|
|
49
106
|
- `qa-framework generate spec <module-name>` — scaffold a full spec set from CLI
|
|
50
107
|
- `qa-framework validate` — check `qa/` structure for compliance with conventions
|
|
51
|
-
- `qa-framework upgrade` — migrate old embedded structure to package-based layout
|
|
52
108
|
- Integration: `jest` support alongside Playwright
|
|
53
109
|
- Integration: GitHub Actions pipeline template (alongside existing Azure Pipelines template)
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
package/README.md
CHANGED
|
@@ -1,16 +1,17 @@
|
|
|
1
1
|
# qa-framework
|
|
2
2
|
|
|
3
3
|
> Reusable, installable, agent-oriented QA framework for spec-driven automated testing.
|
|
4
|
+
> Current version: **v1.6.2**
|
|
4
5
|
|
|
5
6
|
---
|
|
6
7
|
|
|
7
8
|
## What is this?
|
|
8
9
|
|
|
9
|
-
|
|
10
|
+
`@keber/qa-framework` is a methodology package that provides:
|
|
10
11
|
|
|
11
12
|
- **A standardized `qa/` directory structure** for any project
|
|
12
13
|
- **Reusable templates** for specifications, test plans, test cases, defect reports, and execution reports
|
|
13
|
-
- **Agent
|
|
14
|
+
- **Agent skills** (for GitHub Copilot and equivalent IDE agents) that enable AI-assisted QA work via a 3-layer skills architecture
|
|
14
15
|
- **Optional integrations** for Playwright and Azure DevOps
|
|
15
16
|
- **Bootstrap commands** to scaffold the structure into any project in seconds
|
|
16
17
|
|
|
@@ -20,7 +21,7 @@ This framework was designed to be:
|
|
|
20
21
|
|---|---|
|
|
21
22
|
| **Reusable** | Not tied to any specific project, domain, or tech stack |
|
|
22
23
|
| **Decoupled** | Core is independent from Azure DevOps, Playwright, or any other tool |
|
|
23
|
-
| **Agent-oriented** |
|
|
24
|
+
| **Agent-oriented** | Skills are written for IDE agents (Copilot, etc.) to consume with minimal token overhead |
|
|
24
25
|
| **Spec-driven** | Every automation artifact traces back to a specification |
|
|
25
26
|
| **Extensible** | Adapters/plugins for optional integrations are well-separated |
|
|
26
27
|
|
|
@@ -34,17 +35,19 @@ This framework was designed to be:
|
|
|
34
35
|
npm install @keber/qa-framework
|
|
35
36
|
```
|
|
36
37
|
|
|
37
|
-
The postinstall script
|
|
38
|
+
The `postinstall` script runs `init --skip-if-exists` automatically, scaffolding the `qa/`
|
|
39
|
+
structure on first install without overwriting existing files.
|
|
38
40
|
|
|
39
41
|
### Using with an AI agent (GitHub Copilot, Claude, etc.)
|
|
40
42
|
|
|
41
43
|
Use this prompt in your IDE agent chat:
|
|
42
44
|
|
|
43
45
|
```
|
|
44
|
-
Install @keber/qa-framework, check
|
|
46
|
+
Install @keber/qa-framework, check its readme file and set up the project for integration with azure devops
|
|
45
47
|
```
|
|
46
48
|
|
|
47
|
-
The install creates `qa/AGENT-NEXT-STEPS.md` with the exact steps the agent needs to complete
|
|
49
|
+
The install creates `qa/AGENT-NEXT-STEPS.md` with the exact steps the agent needs to complete
|
|
50
|
+
the setup (optional integrations, module configuration, credentials, etc.).
|
|
48
51
|
|
|
49
52
|
### Using manually (without an agent)
|
|
50
53
|
|
|
@@ -53,13 +56,25 @@ npm install @keber/qa-framework
|
|
|
53
56
|
# then read qa/AGENT-NEXT-STEPS.md for next steps
|
|
54
57
|
```
|
|
55
58
|
|
|
56
|
-
|
|
59
|
+
`init` will:
|
|
57
60
|
|
|
58
61
|
1. Create the `qa/` directory tree in your project root
|
|
59
|
-
2. Copy all base templates
|
|
62
|
+
2. Copy all base templates into place
|
|
60
63
|
3. Create `qa/qa-framework.config.json` for project-specific settings
|
|
61
|
-
4.
|
|
62
|
-
5.
|
|
64
|
+
4. Generate `.github/copilot-instructions.md` with QA agent behavior rules (11 rules + ADO detection)
|
|
65
|
+
5. Copy the `skills/` set to `.github/skills/` so the agent can load them on demand
|
|
66
|
+
6. Create `qa/AGENT-NEXT-STEPS.md` with follow-up steps
|
|
67
|
+
|
|
68
|
+
### Upgrading
|
|
69
|
+
|
|
70
|
+
After `npm update @keber/qa-framework`, run:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
npx qa-framework upgrade
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
This overwrites **only** framework-owned files (skills, `copilot-instructions.md`,
|
|
77
|
+
`QA-STRUCTURE-GUIDE.md`) and never touches your specs, test plans, automation code, or memory.
|
|
63
78
|
|
|
64
79
|
---
|
|
65
80
|
|
|
@@ -67,33 +82,32 @@ The `init` command will:
|
|
|
67
82
|
|
|
68
83
|
```
|
|
69
84
|
qa-framework/
|
|
70
|
-
├── README.md
|
|
71
|
-
├── CHANGELOG.md
|
|
72
|
-
├──
|
|
73
|
-
├──
|
|
74
|
-
├── qa-framework.config.json ← Example config (copy to project)
|
|
85
|
+
├── README.md <- This file
|
|
86
|
+
├── CHANGELOG.md <- Version history
|
|
87
|
+
├── package.json <- npm descriptor
|
|
88
|
+
├── qa-framework.config.json <- Example config (copy to project)
|
|
75
89
|
│
|
|
76
|
-
├── docs/
|
|
77
|
-
│ ├── architecture.md
|
|
78
|
-
│ ├──
|
|
79
|
-
│ ├──
|
|
80
|
-
│ ├──
|
|
81
|
-
│ ├──
|
|
82
|
-
│
|
|
83
|
-
│ └── folder-structure-guide.md ← Explanation of every qa/ folder
|
|
90
|
+
├── docs/ <- Framework documentation
|
|
91
|
+
│ ├── architecture.md <- Design decisions and component map
|
|
92
|
+
│ ├── skills-architecture.md <- 3-layer skills model explanation
|
|
93
|
+
│ ├── installation.md <- Detailed installation guide
|
|
94
|
+
│ ├── usage-with-agent.md <- How to use the framework with an IDE agent
|
|
95
|
+
│ ├── spec-driven-philosophy.md <- Core QA methodology
|
|
96
|
+
│ └── folder-structure-guide.md <- Explanation of every qa/ folder
|
|
84
97
|
│
|
|
85
|
-
├──
|
|
86
|
-
│ ├──
|
|
87
|
-
│ ├──
|
|
88
|
-
│ ├──
|
|
89
|
-
│ ├──
|
|
90
|
-
│ ├──
|
|
91
|
-
│ ├──
|
|
92
|
-
│ ├──
|
|
93
|
-
│ └──
|
|
98
|
+
├── skills/ <- Agent skills (3-layer model; copied to .github/skills/ on init)
|
|
99
|
+
│ ├── qa-module-analysis/ <- Stage 1: analyze a module, produce 6-file spec set
|
|
100
|
+
│ ├── qa-spec-generation/ <- Stage 2: generate/update the 6-file spec set
|
|
101
|
+
│ ├── qa-test-plan/ <- Stage 3: generate a test plan
|
|
102
|
+
│ ├── qa-test-cases/ <- Stage 4: generate individual TC-{ID}.md files
|
|
103
|
+
│ ├── qa-automation/ <- Stage 5: implement Playwright automation
|
|
104
|
+
│ ├── qa-test-stabilization/ <- Stage 5b: stabilize failing tests
|
|
105
|
+
│ ├── qa-maintenance/ <- Stage 6: update artifacts after app changes
|
|
106
|
+
│ └── qa-ado-integration/ <- Optional: sync with Azure DevOps
|
|
107
|
+
│ (each skill contains SKILL.md + references/*.md)
|
|
94
108
|
│
|
|
95
|
-
├── templates/
|
|
96
|
-
│ ├── specification/
|
|
109
|
+
├── templates/ <- Reusable file templates
|
|
110
|
+
│ ├── specification/ <- 6-file submodule set
|
|
97
111
|
│ │ ├── 00-inventory.md
|
|
98
112
|
│ │ ├── 01-business-rules.md
|
|
99
113
|
│ │ ├── 02-workflows.md
|
|
@@ -105,15 +119,17 @@ qa-framework/
|
|
|
105
119
|
│ ├── execution-report.md
|
|
106
120
|
│ ├── defect-report.md
|
|
107
121
|
│ ├── session-summary.md
|
|
108
|
-
│
|
|
122
|
+
│ ├── qa-readme.md <- Template for qa/README.md
|
|
123
|
+
│ └── automation-scaffold/ <- Files to bootstrap a Playwright E2E project
|
|
109
124
|
│ ├── package.json
|
|
110
125
|
│ ├── playwright.config.ts
|
|
111
126
|
│ ├── global-setup.ts
|
|
112
127
|
│ ├── .env.example
|
|
113
128
|
│ └── fixtures/
|
|
114
|
-
│
|
|
129
|
+
│ ├── auth.ts
|
|
130
|
+
│ └── test-helpers.ts
|
|
115
131
|
│
|
|
116
|
-
├── integrations/
|
|
132
|
+
├── integrations/ <- Optional integration adapters
|
|
117
133
|
│ ├── playwright/
|
|
118
134
|
│ │ └── README.md
|
|
119
135
|
│ ├── playwright-azure-reporter/
|
|
@@ -125,16 +141,17 @@ qa-framework/
|
|
|
125
141
|
│ ├── create-testplan-from-mapping.ps1
|
|
126
142
|
│ └── sync-ado-titles.ps1
|
|
127
143
|
│
|
|
128
|
-
├── examples/
|
|
144
|
+
├── examples/ <- Reference examples (non-project-specific)
|
|
129
145
|
│ └── module-example/
|
|
130
146
|
│ ├── README.md
|
|
131
|
-
│ └── submodule-example/
|
|
147
|
+
│ └── submodule-example/ <- Full 6-file spec example
|
|
132
148
|
│
|
|
133
|
-
└── scripts/
|
|
134
|
-
├── cli.js
|
|
135
|
-
├── init.js
|
|
136
|
-
├──
|
|
137
|
-
|
|
149
|
+
└── scripts/ <- CLI commands
|
|
150
|
+
├── cli.js <- Entry point (qa-framework <command>)
|
|
151
|
+
├── init.js <- Scaffold qa/ tree + skills + copilot-instructions
|
|
152
|
+
├── upgrade.js <- Refresh framework-owned files after npm update
|
|
153
|
+
├── generate.js <- Generate individual artifact from template
|
|
154
|
+
└── validate.js <- Check structure compliance
|
|
138
155
|
```
|
|
139
156
|
|
|
140
157
|
---
|
|
@@ -145,46 +162,102 @@ When you run `qa-framework init`, this structure is created inside your project:
|
|
|
145
162
|
|
|
146
163
|
```
|
|
147
164
|
qa/
|
|
148
|
-
├── README.md
|
|
149
|
-
├── QA-STRUCTURE-GUIDE.md
|
|
165
|
+
├── README.md <- Living index of all QA artifacts
|
|
166
|
+
├── QA-STRUCTURE-GUIDE.md <- Local copy of the structure guide (framework-owned)
|
|
167
|
+
├── AGENT-NEXT-STEPS.md <- Active sprint checklist for the agent
|
|
150
168
|
│
|
|
151
|
-
├── 00-
|
|
152
|
-
├──
|
|
153
|
-
├──
|
|
154
|
-
├── 02-test-plans/
|
|
155
|
-
├── 03-test-cases/
|
|
156
|
-
├── 04-test-data/
|
|
157
|
-
├── 05-test-execution/
|
|
158
|
-
├── 06-defects/
|
|
159
|
-
├── 07-automation/
|
|
160
|
-
|
|
169
|
+
├── 00-standards/ <- Naming conventions, templates, data policies
|
|
170
|
+
├── 01-specifications/ <- Functional specs (used by generate command)
|
|
171
|
+
├── {module}/{submodule}/ <- 6-file spec sets created per module/submodule in config
|
|
172
|
+
├── 02-test-plans/ <- Test plans (automated + manual)
|
|
173
|
+
├── 03-test-cases/ <- TC-*.md step-by-step docs (manual track; optional if fully automated)
|
|
174
|
+
├── 04-test-data/ <- Shared test data factories/seeders (optional — per-module data lives in 01-specs)
|
|
175
|
+
├── 05-test-execution/ <- Execution reports and results
|
|
176
|
+
├── 06-defects/open|resolved/ <- Defect tracking
|
|
177
|
+
├── 07-automation/ <- Playwright automation code and config
|
|
178
|
+
│ ├── e2e/ <- Playwright E2E project (self-contained)
|
|
179
|
+
│ │ ├── playwright.config.ts
|
|
180
|
+
│ │ ├── package.json
|
|
181
|
+
│ │ ├── global-setup.ts
|
|
182
|
+
│ │ ├── .env.example
|
|
183
|
+
│ │ ├── fixtures/auth.ts
|
|
184
|
+
│ │ ├── fixtures/test-helpers.ts
|
|
185
|
+
│ │ └── tests/{module}/ <- Generated .spec.ts stubs per submodule
|
|
186
|
+
│ ├── integration/ <- API / integration tests (k6, JMeter...)
|
|
187
|
+
│ └── load/ <- Load / performance tests (future)
|
|
188
|
+
├── 08-azure-integration/ <- Azure DevOps integration (optional)
|
|
189
|
+
└── memory/ <- Agent learnings and session notes
|
|
190
|
+
└── INDEX.md
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
Additionally, `init` writes to the project root:
|
|
194
|
+
|
|
195
|
+
```
|
|
196
|
+
.github/
|
|
197
|
+
├── copilot-instructions.md <- Generated agent rules (11 rules + ADO detection)
|
|
198
|
+
└── skills/ <- Copied from package skills/ (one folder per pipeline stage)
|
|
161
199
|
```
|
|
162
200
|
|
|
163
201
|
See [docs/folder-structure-guide.md](docs/folder-structure-guide.md) for a full explanation of every folder.
|
|
164
202
|
|
|
165
203
|
---
|
|
166
204
|
|
|
167
|
-
##
|
|
205
|
+
## Skills Architecture
|
|
168
206
|
|
|
169
|
-
The `
|
|
170
|
-
|
|
207
|
+
The `skills/` folder contains agent skills organized in a **3-layer model** that minimizes
|
|
208
|
+
token overhead compared to the legacy monolithic per-task instruction file approach.
|
|
171
209
|
|
|
172
|
-
|
|
|
173
|
-
|
|
174
|
-
| `
|
|
175
|
-
| `
|
|
176
|
-
| `
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
210
|
+
| Layer | Location | Purpose |
|
|
211
|
+
|---|---|---|
|
|
212
|
+
| 1 - Descriptor | YAML frontmatter in `SKILL.md` | Name + description; used for relevance detection (~50 tokens) |
|
|
213
|
+
| 2 - Process outline | `SKILL.md` body | Step-by-step process; always loaded when the skill applies (~400-600 tokens) |
|
|
214
|
+
| 3 - Reference material | `references/*.md` | Detailed templates, code patterns, decision trees; loaded on demand (~500-1500 tokens each) |
|
|
215
|
+
|
|
216
|
+
On `init`, the full `skills/` tree is copied to `.github/skills/` so the agent can load
|
|
217
|
+
them via `read_file` without touching `node_modules`.
|
|
218
|
+
|
|
219
|
+
### QA Pipeline
|
|
220
|
+
|
|
221
|
+
Skills map to the pipeline stages enforced by `.github/copilot-instructions.md`:
|
|
222
|
+
|
|
223
|
+
| Stage | Skill | Task | Track | Prerequisite |
|
|
224
|
+
|---|---|---|---|---|
|
|
225
|
+
| 1 | `qa-module-analysis` | Analyze module, produce 6-file spec set | Both | None |
|
|
226
|
+
| 2 | `qa-spec-generation` | Generate/update 6-file spec set | Both | `00-inventory.md` exists |
|
|
227
|
+
| 3 | `qa-test-plan` | Generate test plan | Both | `05-test-scenarios.md` exists |
|
|
228
|
+
| 4 | `qa-test-cases` | Generate TC-{ID}.md files | Manual | Test plan exists |
|
|
229
|
+
| 5 | `qa-automation` | Implement Playwright tests | Automation | Specs approved, no PENDING-CODE |
|
|
230
|
+
| 5b | `qa-test-stabilization` | Stabilize failing tests | Automation | Failing tests exist |
|
|
231
|
+
| 6 | `qa-maintenance` | Update artifacts after app changes | Both | Delivered change |
|
|
232
|
+
| - | `qa-ado-integration` | Sync with Azure DevOps Test Plans | Both | ADO enabled in config |
|
|
233
|
+
|
|
234
|
+
See [docs/skills-architecture.md](docs/skills-architecture.md) for the full design rationale.
|
|
185
235
|
|
|
186
236
|
---
|
|
187
237
|
|
|
238
|
+
## CLI Reference
|
|
239
|
+
|
|
240
|
+
```
|
|
241
|
+
qa-framework <command> [options]
|
|
242
|
+
|
|
243
|
+
Commands:
|
|
244
|
+
init [--config <path>] Scaffold qa/ structure from config file
|
|
245
|
+
upgrade [--dry-run] Refresh framework-owned files after npm update
|
|
246
|
+
generate <artifact> Generate from template
|
|
247
|
+
artifact: spec | test-plan | test-case |
|
|
248
|
+
execution-report | defect-report |
|
|
249
|
+
session-summary
|
|
250
|
+
validate [--strict] Validate qa/ structure and naming conventions
|
|
251
|
+
|
|
252
|
+
Options:
|
|
253
|
+
--help, -h Show help
|
|
254
|
+
--version, -v Show version
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
**upgrade** overwrites only framework-owned files (skills, `copilot-instructions.md`,
|
|
258
|
+
`QA-STRUCTURE-GUIDE.md`) and never touches project-owned files. Run `--dry-run` to preview
|
|
259
|
+
without writing.
|
|
260
|
+
|
|
188
261
|
## Configuration
|
|
189
262
|
|
|
190
263
|
`qa-framework init` now bootstraps a default config automatically at `qa/qa-framework.config.json` if none exists.
|
|
@@ -214,10 +287,11 @@ See [docs/installation.md](docs/installation.md) for full config documentation.
|
|
|
214
287
|
## Design Principles
|
|
215
288
|
|
|
216
289
|
1. **UI is the source of truth.** Test cases are defined from what is observable in the running QA environment, not from source code or requirements docs alone.
|
|
217
|
-
2. **Spec before automation.** Every automated test traces back to a written specification (TC-ID
|
|
290
|
+
2. **Spec before automation.** Every automated test traces back to a written specification (TC-ID -> spec file).
|
|
218
291
|
3. **Credentials never in documentation.** All credentials are handled via environment variables and sanitized from markdown with `<PLACEHOLDER>`.
|
|
219
292
|
4. **Agent-readable conventions.** File names, folder names, and IDs follow strict, predictable patterns so agents can navigate without guessing.
|
|
220
293
|
5. **Optional integrations don't pollute the core.** Azure DevOps and Playwright configuration live in separate, opt-in sections.
|
|
294
|
+
6. **Assertion polarity.** A test must assert what the spec requires, not what the app currently does. A failing test that exposes a real defect is more valuable than a passing test that hides one.
|
|
221
295
|
|
|
222
296
|
---
|
|
223
297
|
|
|
@@ -225,8 +299,8 @@ See [docs/installation.md](docs/installation.md) for full config documentation.
|
|
|
225
299
|
|
|
226
300
|
This framework follows [Semantic Versioning](https://semver.org/):
|
|
227
301
|
|
|
228
|
-
- **MAJOR**: breaking changes to the `qa/` directory structure or
|
|
229
|
-
- **MINOR**: new templates, new optional integrations, new
|
|
302
|
+
- **MAJOR**: breaking changes to the `qa/` directory structure or skill APIs
|
|
303
|
+
- **MINOR**: new templates, new optional integrations, new skills
|
|
230
304
|
- **PATCH**: bug fixes in CLI scripts, documentation corrections
|
|
231
305
|
|
|
232
306
|
See [CHANGELOG.md](CHANGELOG.md) for detailed history.
|
|
@@ -241,4 +315,4 @@ This framework is designed to be forked and adapted. If you need a project-speci
|
|
|
241
315
|
2. Place your adapter config and scripts there
|
|
242
316
|
3. Reference it from `qa-framework.config.json` under `integrations`
|
|
243
317
|
|
|
244
|
-
Do **not** put project-specific configuration inside `
|
|
318
|
+
Do **not** put project-specific configuration inside `skills/` or `templates/`. Those must remain generic.
|
package/docs/architecture.md
CHANGED
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
## Framework Architecture
|
|
4
4
|
|
|
5
|
-
**Version**: 1.
|
|
6
|
-
**Date**: 2026-
|
|
5
|
+
**Version**: 1.6.0
|
|
6
|
+
**Date**: 2026-04-29
|
|
7
7
|
|
|
8
8
|
---
|
|
9
9
|
|
|
@@ -27,16 +27,17 @@
|
|
|
27
27
|
│ │ FRAMEWORK CORE │ │
|
|
28
28
|
│ │ │ │
|
|
29
29
|
│ │ ┌────────────────┐ ┌──────────────────┐ ┌─────────────────────┐ │ │
|
|
30
|
-
│ │ │ qa/ structure │ │ Agent
|
|
31
|
-
│ │ │ (
|
|
32
|
-
│ │ │ │ │
|
|
30
|
+
│ │ │ qa/ structure │ │ Agent skills │ │ Standards & │ │ │
|
|
31
|
+
│ │ │ (10 folders) │ │ (8 skill sets, │ │ Naming conventions │ │ │
|
|
32
|
+
│ │ │ │ │ 3-layer model) │ │ │ │ │
|
|
33
33
|
│ │ └────────────────┘ └──────────────────┘ └─────────────────────┘ │ │
|
|
34
34
|
│ │ │ │
|
|
35
35
|
│ │ ┌────────────────┐ ┌──────────────────┐ ┌─────────────────────┐ │ │
|
|
36
36
|
│ │ │ 6-file │ │ CLI commands │ │ Config schema │ │ │
|
|
37
|
-
│ │ │ submodule │ │ (init/
|
|
38
|
-
│ │ │ templates │ │
|
|
39
|
-
│ │ └────────────────┘
|
|
37
|
+
│ │ │ submodule │ │ (init/upgrade/ │ │ (qa-framework. │ │ │
|
|
38
|
+
│ │ │ templates │ │ generate/ │ │ config.json) │ │ │
|
|
39
|
+
│ │ └────────────────┘ │ validate) │ └─────────────────────┘ │ │
|
|
40
|
+
│ │ └──────────────────┘ │ │
|
|
40
41
|
│ └──────────────────────────────────────────────────────────────────────┘ │
|
|
41
42
|
│ │
|
|
42
43
|
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
|
@@ -44,18 +45,18 @@
|
|
|
44
45
|
│ │ │ │
|
|
45
46
|
│ │ ┌─────────────────────────┐ ┌──────────────────────────────┐ │ │
|
|
46
47
|
│ │ │ playwright/ │ │ playwright-azure-reporter/ │ │ │
|
|
47
|
-
│ │ │ - playwright.
|
|
48
|
-
│ │ │ - global-setup.ts
|
|
49
|
-
│ │ │ - fixtures/
|
|
50
|
-
│ │ │ -
|
|
48
|
+
│ │ │ - e2e/playwright.cfg │ │ - reporter config template │ │ │
|
|
49
|
+
│ │ │ - e2e/global-setup.ts │ │ - ado-ids-mapping.json │ │ │
|
|
50
|
+
│ │ │ - e2e/fixtures/ │ │ - inject-ado-ids.ps1 │ │ │
|
|
51
|
+
│ │ │ - e2e/.env.example │ │ - module-registry.json │ │ │
|
|
51
52
|
│ │ └─────────────────────────┘ └──────────────────────────────┘ │ │
|
|
52
53
|
│ │ │ │
|
|
53
|
-
│ │ ┌─────────────────────────┐
|
|
54
|
-
│ │ │ ado-powershell/ │
|
|
55
|
-
│ │ │ - create-testplan.ps1 │
|
|
56
|
-
│ │ │ - sync-ado-titles.ps1 │
|
|
57
|
-
│ │ │ - azure-pipeline.yml │
|
|
58
|
-
│ │ └─────────────────────────┘
|
|
54
|
+
│ │ ┌─────────────────────────┐ │ │
|
|
55
|
+
│ │ │ ado-powershell/ │ │ │
|
|
56
|
+
│ │ │ - create-testplan.ps1 │ │ │
|
|
57
|
+
│ │ │ - sync-ado-titles.ps1 │ │ │
|
|
58
|
+
│ │ │ - azure-pipeline.yml │ │ │
|
|
59
|
+
│ │ └─────────────────────────┘ │ │
|
|
59
60
|
│ └──────────────────────────────────────────────────────────────────────┘ │
|
|
60
61
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
61
62
|
|
|
@@ -65,20 +66,26 @@
|
|
|
65
66
|
│ TARGET PROJECT │
|
|
66
67
|
│ │
|
|
67
68
|
│ qa/ │
|
|
68
|
-
│ ├── README.md
|
|
69
|
-
│ ├──
|
|
70
|
-
│ ├──
|
|
69
|
+
│ ├── README.md <- living index (project-specific) │
|
|
70
|
+
│ ├── AGENT-NEXT-STEPS.md <- active sprint queue (project-specific) │
|
|
71
|
+
│ ├── QA-STRUCTURE-GUIDE.md <- copy from framework (framework-owned) │
|
|
72
|
+
│ ├── qa-framework.config.json <- project config (parameterized) │
|
|
71
73
|
│ │ │
|
|
72
|
-
│ ├── 00-
|
|
73
|
-
│ ├──
|
|
74
|
-
│ ├──
|
|
75
|
-
│ ├──
|
|
76
|
-
│ ├──
|
|
77
|
-
│ ├──
|
|
78
|
-
│ ├──
|
|
79
|
-
│ ├──
|
|
80
|
-
│ ├── 07-automation/
|
|
81
|
-
│
|
|
74
|
+
│ ├── 00-standards/ <- naming + templates (copied from pkg) │
|
|
75
|
+
│ ├── 01-specifications/ <- generated per module (project work) │
|
|
76
|
+
│ ├── 02-test-plans/ <- generated (project work) │
|
|
77
|
+
│ ├── 03-test-cases/ <- generated (project work) │
|
|
78
|
+
│ ├── 04-test-data/ <- project data (project work) │
|
|
79
|
+
│ ├── 05-test-execution/ <- execution reports (project work) │
|
|
80
|
+
│ ├── 06-defects/open|resolved/ <- optional defect cache │
|
|
81
|
+
│ ├── 07-automation/e2e/ <- Playwright project (self-contained) │
|
|
82
|
+
│ ├── 07-automation/integration/ <- API/integration tests │
|
|
83
|
+
│ ├── 07-automation/load/ <- load/performance tests │
|
|
84
|
+
│ ├── 08-azure-integration/ <- optional ADO integration │
|
|
85
|
+
│ └── memory/ <- agent learnings (project-maintained) │
|
|
86
|
+
│ │
|
|
87
|
+
│ .github/copilot-instructions.md <- generated (framework-owned) │
|
|
88
|
+
│ .github/skills/ <- copied from package (framework-owned) │
|
|
82
89
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
83
90
|
```
|
|
84
91
|
|
|
@@ -107,7 +114,7 @@ Module analysis (agent)
|
|
|
107
114
|
│
|
|
108
115
|
▼
|
|
109
116
|
07-automation/e2e/tests/{module}/
|
|
110
|
-
*.spec.ts
|
|
117
|
+
*.spec.ts <--- test title contains TC-ID
|
|
111
118
|
│
|
|
112
119
|
┌────────────────────┘
|
|
113
120
|
│
|
|
@@ -129,21 +136,25 @@ Module analysis (agent)
|
|
|
129
136
|
|
|
130
137
|
Installed always. Contains:
|
|
131
138
|
|
|
132
|
-
- `qa/` directory skeleton (
|
|
133
|
-
-
|
|
139
|
+
- `qa/` directory skeleton (10 folders)
|
|
140
|
+
- `.github/skills/` — 8 agent skill sets (3-layer model: SKILL.md + references/)
|
|
134
141
|
- `00-standards/` — naming conventions, templates
|
|
135
142
|
- `QA-STRUCTURE-GUIDE.md`
|
|
143
|
+
- `.github/copilot-instructions.md` — generated pipeline sequencer
|
|
136
144
|
- `qa-framework.config.json` schema
|
|
137
145
|
|
|
138
146
|
### Layer 2 — Playwright Integration (opt-in)
|
|
139
147
|
|
|
140
|
-
Installed
|
|
148
|
+
Installed always (scaffold is always created by `init`):
|
|
141
149
|
|
|
142
150
|
- `qa/07-automation/e2e/playwright.config.ts`
|
|
143
151
|
- `qa/07-automation/e2e/global-setup.ts`
|
|
144
152
|
- `qa/07-automation/e2e/.env.example`
|
|
145
153
|
- `qa/07-automation/e2e/fixtures/auth.ts`
|
|
154
|
+
- `qa/07-automation/e2e/fixtures/test-helpers.ts`
|
|
146
155
|
- `qa/07-automation/e2e/package.json`
|
|
156
|
+
- `qa/07-automation/integration/README.md` (placeholder)
|
|
157
|
+
- `qa/07-automation/load/README.md` (placeholder)
|
|
147
158
|
|
|
148
159
|
### Layer 3 — Azure DevOps Integration (opt-in)
|
|
149
160
|
|
|
@@ -155,13 +166,6 @@ Installed when `integrations.azureDevOps.enabled = true`:
|
|
|
155
166
|
- PowerShell scripts (inject, create-plan, sync)
|
|
156
167
|
- Azure Pipeline YAML template
|
|
157
168
|
|
|
158
|
-
### Layer 4 — Framework Adapters (opt-in per adapter)
|
|
159
|
-
|
|
160
|
-
Technology-specific extensions. Currently available:
|
|
161
|
-
|
|
162
|
-
- `blazor-radzen` — Playwright helpers for Radzen Blazor apps (STUB - v1.1 planned)
|
|
163
|
-
- `aspnet-mvc` — Select2/jQuery datatable helpers (STUB - v1.1 planned)
|
|
164
|
-
|
|
165
169
|
---
|
|
166
170
|
|
|
167
171
|
## Configuration Architecture
|
|
@@ -188,10 +192,11 @@ The framework uses **Semantic Versioning**:
|
|
|
188
192
|
|
|
189
193
|
| Change type | Version bump |
|
|
190
194
|
|---|---|
|
|
191
|
-
| New folder added to `qa/` tree | MAJOR
|
|
195
|
+
| New folder added to `qa/` tree or restructured | MAJOR |
|
|
192
196
|
| New template file | MINOR |
|
|
193
197
|
| New integration adapter | MINOR |
|
|
194
|
-
|
|
|
198
|
+
| New skill or skill update | MINOR |
|
|
199
|
+
| CLI command update | PATCH |
|
|
195
200
|
| Documentation correction | PATCH |
|
|
196
201
|
|
|
197
202
|
**Upgrade path**: When a new MAJOR version changes the `qa/` structure, the `MIGRATION-NOTES.md` in the package provides a step-by-step migration script. Projects should pin their framework version in `package.json` and upgrade deliberately.
|