@keber/qa-framework 1.5.2 → 1.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +39 -6
- package/README.md +150 -76
- package/docs/architecture.md +49 -44
- package/docs/folder-structure-guide.md +27 -23
- package/docs/installation.md +53 -43
- package/docs/usage-with-agent.md +52 -64
- package/integrations/ado-powershell/pipelines/azure-pipeline-qa.yml +1 -1
- package/integrations/playwright/README.md +1 -1
- package/package.json +1 -1
- package/scripts/init.js +20 -12
- package/scripts/upgrade.js +100 -11
- package/skills/qa-ado-integration/SKILL.md +2 -2
- package/skills/qa-automation/SKILL.md +6 -6
- package/skills/qa-maintenance/SKILL.md +2 -2
- package/templates/automation-scaffold/playwright.config.ts +1 -1
- package/agent-instructions/00-module-analysis.md +0 -266
- package/agent-instructions/01-spec-generation.md +0 -281
- package/agent-instructions/02-test-plan-generation.md +0 -205
- package/agent-instructions/03-test-case-generation.md +0 -150
- package/agent-instructions/04-automation-generation.md +0 -343
- package/agent-instructions/04b-test-stabilization.md +0 -309
- package/agent-instructions/05-ado-integration.md +0 -247
- package/agent-instructions/06-maintenance.md +0 -135
- package/docs/comparison-matrix.md +0 -131
- package/docs/generalization-decisions.md +0 -203
|
@@ -1,309 +0,0 @@
|
|
|
1
|
-
# Agent Instructions: Test Stabilization
|
|
2
|
-
|
|
3
|
-
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-stabilization/SKILL.md`.
|
|
4
|
-
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
-
|
|
6
|
-
**File**: `agent-instructions/04b-test-stabilization.md`
|
|
7
|
-
**Purpose**: Instructions for exhaustively reviewing and stabilizing generated Playwright tests to ensure they are logically correct, technically sound, and free of false positives and false negatives before ADO integration.
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## Stage Context
|
|
12
|
-
|
|
13
|
-
| Field | Value |
|
|
14
|
-
|-------|-------|
|
|
15
|
-
| Stage number | 3.5 |
|
|
16
|
-
| Stage name | Test Stabilization |
|
|
17
|
-
| Preceding stage | Stage 4 — Automation Generation (`04-automation-generation.md`) |
|
|
18
|
-
| Following stage | Stage 5 — ADO Integration (`05-ado-integration.md`) |
|
|
19
|
-
| Can be skipped? | No — ADO integration with unreliable tests produces misleading test run history |
|
|
20
|
-
| Required inputs | Generated `.spec.ts` files in `qa/07-automation/e2e/`; approved `05-test-scenarios.md` for the module |
|
|
21
|
-
| Produced outputs | Stabilized spec files; `STABILIZATION-REPORT.md` per submodule |
|
|
22
|
-
| Exit criterion | Confidence ≥ 90% that every test produces a true positive on failure and a true negative on pass |
|
|
23
|
-
|
|
24
|
-
---
|
|
25
|
-
|
|
26
|
-
## When to use
|
|
27
|
-
|
|
28
|
-
Run this stage immediately after **Stage 4 — Automation Generation** and before any ADO
|
|
29
|
-
integration or CI pipeline registration. This stage is mandatory regardless of whether tests
|
|
30
|
-
are passing or failing after generation.
|
|
31
|
-
|
|
32
|
-
Also re-run this stage when:
|
|
33
|
-
- Sprint changes introduce significant new test code (more than 3 new spec files)
|
|
34
|
-
- A test suite starts flaking in CI after a previously stable period
|
|
35
|
-
- A test is suspected of being a false positive (passing despite the feature being broken)
|
|
36
|
-
- A test is suspected of being a false negative (failing despite the feature being correct)
|
|
37
|
-
|
|
38
|
-
---
|
|
39
|
-
|
|
40
|
-
> **Scope boundary**: This stage operates exclusively on test code (`.spec.ts`, fixtures,
|
|
41
|
-
> helpers, `global-setup.ts`, `playwright.config.ts`). The application source code is
|
|
42
|
-
> strictly out of scope. Do not modify, patch, or work around application behavior to make
|
|
43
|
-
> tests pass — document the mismatch as a defect instead.
|
|
44
|
-
|
|
45
|
-
---
|
|
46
|
-
|
|
47
|
-
> **Coverage boundary**: Do not add new tests or increase coverage during this stage. If a
|
|
48
|
-
> gap in coverage is identified, note it in the STABILIZATION-REPORT.md for Stage 6
|
|
49
|
-
> (Maintenance) to address. The goal is correctness of existing tests, not completeness.
|
|
50
|
-
|
|
51
|
-
---
|
|
52
|
-
|
|
53
|
-
## Step 1 — Establish the Reference Baseline
|
|
54
|
-
|
|
55
|
-
Before running any tests, collect the ground truth against which correctness will be judged.
|
|
56
|
-
|
|
57
|
-
1. Open `05-test-scenarios.md` for the target module/submodule. For each TC row, note:
|
|
58
|
-
- TC-ID and title
|
|
59
|
-
- Priority (P0/P1/P2/P3)
|
|
60
|
-
- Expected result (acceptance criterion)
|
|
61
|
-
- Automatable status (Yes / Partial / No)
|
|
62
|
-
|
|
63
|
-
2. Open `COVERAGE-MAPPING.md` for the module. For each row:
|
|
64
|
-
- Confirm the `.spec.ts` file and test function name exist
|
|
65
|
-
- Flag any rows where the spec or function is missing (these are generation gaps, not
|
|
66
|
-
stabilization issues — document in the report but do not create new tests)
|
|
67
|
-
|
|
68
|
-
3. If `05-test-scenarios.md` or `COVERAGE-MAPPING.md` are absent or incomplete, stop and
|
|
69
|
-
complete Stage 4 before proceeding.
|
|
70
|
-
|
|
71
|
-
---
|
|
72
|
-
|
|
73
|
-
## Step 2 — First Run: Collect Baseline Results
|
|
74
|
-
|
|
75
|
-
Run the full suite for the target module without any modifications:
|
|
76
|
-
|
|
77
|
-
```bash
|
|
78
|
-
npx playwright test qa/07-automation/e2e/tests/{module-kebab}/ --reporter=list
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
Record the outcome of every test:
|
|
82
|
-
|
|
83
|
-
| TC-ID | Test title | Result | Notes |
|
|
84
|
-
|-------|-----------|--------|-------|
|
|
85
|
-
| TC-MOD-SUB-001 | ... | ✅ pass / ❌ fail / ⚠️ flaky / ⏭ skip | Error message if fail |
|
|
86
|
-
|
|
87
|
-
Collect trace artifacts for every failing test:
|
|
88
|
-
|
|
89
|
-
```bash
|
|
90
|
-
npx playwright test ... --trace=on
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
Do NOT make any changes yet. The first run result is the diagnostic baseline.
|
|
94
|
-
|
|
95
|
-
---
|
|
96
|
-
|
|
97
|
-
## Step 3 — Classify Every Failing or Skipped Test
|
|
98
|
-
|
|
99
|
-
For each non-passing test, determine its failure category before writing any code.
|
|
100
|
-
|
|
101
|
-
### Failure Categories
|
|
102
|
-
|
|
103
|
-
| Category | Definition | Correct action |
|
|
104
|
-
|----------|-----------|----------------|
|
|
105
|
-
| **A — Test Bug: Wrong selector** | The selector no longer matches the UI element | Fix selector in `.spec.ts` |
|
|
106
|
-
| **B — Test Bug: Wrong assertion** | The assertion does not match the specified acceptance criterion | Fix assertion to match spec |
|
|
107
|
-
| **C — Test Bug: Wrong flow** | The test steps don't reflect the actual user flow described in the TC | Rewrite steps to match TC |
|
|
108
|
-
| **D — Test Bug: Fragile timing** | Test fails intermittently due to missing `await`, race conditions, or missing `waitFor` | Add proper async handling |
|
|
109
|
-
| **E — Test Bug: Incorrect data** | Test data is malformed, not unique, or conflicts with QA environment state | Fix data generation or provisioning |
|
|
110
|
-
| **F — App Bug** | The test is correct and the application is not behaving as the TC specifies | Do not fix the test; file a defect |
|
|
111
|
-
| **G — TC Mismatch** | The spec TC is ambiguous or incorrect vs the actual implemented behavior | Do not fix the test or the app; update `05-test-scenarios.md` and note as a spec defect |
|
|
112
|
-
| **H — Infra / Environment** | Test fails due to QA environment unavailability, network, or credentials | Retry in a clean environment; do not classify as a test bug until confirmed |
|
|
113
|
-
| **I — Intentional skip** | Test is marked `test.skip()` due to a known defect or pending feature | Verify skip reason is still valid; update `06-defects/open/` reference |
|
|
114
|
-
|
|
115
|
-
For each failing test, write your classification in the working STABILIZATION-REPORT.md
|
|
116
|
-
before making any change.
|
|
117
|
-
|
|
118
|
-
### Decision Protocol
|
|
119
|
-
|
|
120
|
-
```
|
|
121
|
-
Is the test step sequence and assertion derived correctly from 05-test-scenarios.md?
|
|
122
|
-
No → Category G (TC Mismatch — fix the spec, not the test)
|
|
123
|
-
Yes ↓
|
|
124
|
-
Does the application behave as the TC acceptance criterion describes?
|
|
125
|
-
No → Category F (App Bug — file a defect, skip the test with a DEF reference)
|
|
126
|
-
Yes ↓
|
|
127
|
-
Does the failure appear in every run (deterministic)?
|
|
128
|
-
No → Category D or H (timing or environment — diagnose further)
|
|
129
|
-
Yes ↓
|
|
130
|
-
Is the failure caused by a selector, assertion, data, or flow issue in the test code?
|
|
131
|
-
→ Category A, B, C, or E (Test Bug — fix the test code)
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
---
|
|
135
|
-
|
|
136
|
-
## Step 4 — Fix Iterations
|
|
137
|
-
|
|
138
|
-
Apply fixes one category at a time, in this order: E → A → C → B → D.
|
|
139
|
-
(Data problems mask selector problems; flow problems mask assertion problems.)
|
|
140
|
-
|
|
141
|
-
After each batch of fixes, re-run the affected tests:
|
|
142
|
-
|
|
143
|
-
```bash
|
|
144
|
-
npx playwright test qa/07-automation/e2e/tests/{module-kebab}/{file}.spec.ts --reporter=list
|
|
145
|
-
```
|
|
146
|
-
|
|
147
|
-
Do not move to the next category until the current category's tests are stable across
|
|
148
|
-
**at least two consecutive runs**.
|
|
149
|
-
|
|
150
|
-
### Rules for fixing test code
|
|
151
|
-
|
|
152
|
-
- **Selectors**: prefer `getByRole`, `getByLabel`, `getByTestId` — avoid CSS/XPath selectors
|
|
153
|
-
that depend on position or class names that aren't semantic
|
|
154
|
-
- **Assertions**: every assertion must trace back to a specific acceptance criterion in the TC;
|
|
155
|
-
remove or replace any `expect().toBeVisible()` that does not correspond to a TC requirement
|
|
156
|
-
- **Waits**: prefer `await expect(locator).toBeVisible()` over `page.waitForTimeout()`;
|
|
157
|
-
never use `waitForTimeout` as a fix for timing — diagnose the actual cause
|
|
158
|
-
- **Data**: use `EXEC_IDX + RUN_SALT` for unique identifiers (see ISSUE-02 note below);
|
|
159
|
-
provision data in `beforeAll`, not inside individual tests
|
|
160
|
-
- **Skips**: when skipping for a Category F (App Bug), always include the defect reference:
|
|
161
|
-
```typescript
|
|
162
|
-
test.skip(true, 'DEF-{NNN}: {one-line bug description}. Filed {YYYY-MM-DD}.');
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
> ⚠️ **EXEC_IDX collision**: Two runs within the same 60-second window produce identical
|
|
166
|
-
> `EXEC_IDX` values. For fields requiring guaranteed uniqueness, add a salt:
|
|
167
|
-
> ```typescript
|
|
168
|
-
> const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
|
|
169
|
-
> const RUN_SALT = Math.floor(Math.random() * 9999).toString().padStart(4, '0');
|
|
170
|
-
> ```
|
|
171
|
-
|
|
172
|
-
---
|
|
173
|
-
|
|
174
|
-
## Step 5 — Validate Passing Tests for False Positives
|
|
175
|
-
|
|
176
|
-
A passing test is not necessarily a correct test. After fixing failing tests, audit every
|
|
177
|
-
**passing** test for false positives.
|
|
178
|
-
|
|
179
|
-
For each passing test, perform one of the following checks:
|
|
180
|
-
|
|
181
|
-
### Negation check (preferred)
|
|
182
|
-
Temporarily modify the test to assert the opposite of the expected result, then run it.
|
|
183
|
-
If it still passes — the test is a false positive (asserting nothing meaningful).
|
|
184
|
-
|
|
185
|
-
```typescript
|
|
186
|
-
// Original: expect(toast).toHaveText('Guardado exitosamente');
|
|
187
|
-
// Negation check: replace temporarily with:
|
|
188
|
-
expect(toast).toHaveText('TEXTO_QUE_NO_EXISTE_JAMAS');
|
|
189
|
-
// If this passes → the original assertion is not actually evaluating the toast content
|
|
190
|
-
```
|
|
191
|
-
|
|
192
|
-
Restore the original assertion after the check.
|
|
193
|
-
|
|
194
|
-
### Trace review (secondary)
|
|
195
|
-
Open the Playwright trace for the passing test and step through it. Verify:
|
|
196
|
-
- Every action in the `test()` body corresponds to a step in the TC
|
|
197
|
-
- Every `expect()` is actually being evaluated (no skipped assertions due to early returns
|
|
198
|
-
or unreachable code)
|
|
199
|
-
- The test does not pass because an early `expect()` short-circuits before the meaningful one
|
|
200
|
-
|
|
201
|
-
### API/network verification (for tests touching data persistence)
|
|
202
|
-
For tests that create, update, or delete records: after the UI action, verify the result
|
|
203
|
-
is persisted, not just that a success toast appeared. A test that only checks the toast
|
|
204
|
-
is a false positive if the actual save fails silently.
|
|
205
|
-
|
|
206
|
-
---
|
|
207
|
-
|
|
208
|
-
## Step 6 — Iteration and Convergence
|
|
209
|
-
|
|
210
|
-
Repeat Steps 4 and 5 until:
|
|
211
|
-
|
|
212
|
-
1. All deterministic failures are resolved (either fixed or legitimately skipped with a
|
|
213
|
-
defect reference)
|
|
214
|
-
2. All passing tests have passed the negation check or trace review
|
|
215
|
-
3. No test produces intermittent results across 3 consecutive runs
|
|
216
|
-
|
|
217
|
-
**Confidence scoring**: after each iteration, estimate confidence per test:
|
|
218
|
-
|
|
219
|
-
| Confidence | Criteria |
|
|
220
|
-
|-----------|---------|
|
|
221
|
-
| ✅ High (≥90%) | Passed negation check AND matches TC acceptance criterion exactly |
|
|
222
|
-
| ⚠️ Medium (70–89%) | Passes consistently; trace reviewed; no negation check yet |
|
|
223
|
-
| ❌ Low (<70%) | Intermittent, unclassified failure, or negation check not attempted |
|
|
224
|
-
|
|
225
|
-
The stage is complete when **all tests are ✅ High or legitimately skipped** (Category F
|
|
226
|
-
with a defect reference or Category I with a valid skip reason).
|
|
227
|
-
|
|
228
|
-
---
|
|
229
|
-
|
|
230
|
-
## Step 7 — Document Findings
|
|
231
|
-
|
|
232
|
-
For each submodule, create or update `STABILIZATION-REPORT.md` in the module's automation
|
|
233
|
-
directory:
|
|
234
|
-
|
|
235
|
-
```
|
|
236
|
-
qa/07-automation/e2e/tests/{module-kebab}/STABILIZATION-REPORT.md
|
|
237
|
-
```
|
|
238
|
-
|
|
239
|
-
### STABILIZATION-REPORT.md structure
|
|
240
|
-
|
|
241
|
-
```markdown
|
|
242
|
-
# Stabilization Report — {MODULE} > {SUBMODULE}
|
|
243
|
-
|
|
244
|
-
**Date**: YYYY-MM-DD
|
|
245
|
-
**Sprint**: {sprint number or label}
|
|
246
|
-
**Analyst**: {agent session name or human name}
|
|
247
|
-
**Spec reference**: `qa/{module-key}/{submodule-key}/05-test-scenarios.md`
|
|
248
|
-
|
|
249
|
-
## Summary
|
|
250
|
-
|
|
251
|
-
| Metric | Value |
|
|
252
|
-
|--------|-------|
|
|
253
|
-
| Tests evaluated | N |
|
|
254
|
-
| Tests stabilized (now ✅ High confidence) | N |
|
|
255
|
-
| Tests skipped — App Bug (Category F) | N |
|
|
256
|
-
| Tests skipped — Intentional (Category I) | N |
|
|
257
|
-
| Remaining low-confidence tests | N |
|
|
258
|
-
| Overall confidence | NN% |
|
|
259
|
-
|
|
260
|
-
## Test-by-Test Record
|
|
261
|
-
|
|
262
|
-
| TC-ID | Test title | Baseline result | Final result | Category | Changes made | Confidence |
|
|
263
|
-
|-------|-----------|-----------------|-------------|----------|-------------|-----------|
|
|
264
|
-
| TC-... | ... | ❌ fail | ✅ pass | A | Fixed selector `#old` → `getByRole('button', { name: '...' })` | ✅ High |
|
|
265
|
-
|
|
266
|
-
## App Bugs Filed
|
|
267
|
-
|
|
268
|
-
| DEF-ID | TC-ID | Description | Filed date | ADO WI (if created) |
|
|
269
|
-
|--------|-------|-------------|-----------|---------------------|
|
|
270
|
-
|
|
271
|
-
## Spec Defects (Category G)
|
|
272
|
-
|
|
273
|
-
| TC-ID | Issue | Recommended `05-test-scenarios.md` update |
|
|
274
|
-
|-------|-------|------------------------------------------|
|
|
275
|
-
|
|
276
|
-
## Coverage Gaps Identified (do not fix in this stage)
|
|
277
|
-
|
|
278
|
-
- {TC-ID}: test mapped in COVERAGE-MAPPING.md but `.spec.ts` function not found
|
|
279
|
-
- ...
|
|
280
|
-
|
|
281
|
-
## Decisions Log
|
|
282
|
-
|
|
283
|
-
Chronological record of non-trivial decisions made during stabilization:
|
|
284
|
-
|
|
285
|
-
- YYYY-MM-DD: Classified TC-XXX failure as Category F after verifying the save endpoint
|
|
286
|
-
returns 500 on duplicate RUTs. App bug, not test bug. DEF-001 filed.
|
|
287
|
-
- ...
|
|
288
|
-
```
|
|
289
|
-
|
|
290
|
-
---
|
|
291
|
-
|
|
292
|
-
## Step 8 — Update Upstream Artifacts
|
|
293
|
-
|
|
294
|
-
After completing STABILIZATION-REPORT.md:
|
|
295
|
-
|
|
296
|
-
1. **If any Category G issues were found** (TC Mismatch): update `05-test-scenarios.md`
|
|
297
|
-
with the corrected acceptance criteria. Note the change with `[REVISED - {YYYY-MM-DD}]`.
|
|
298
|
-
|
|
299
|
-
2. **If any Category F issues were found** (App Bug): create defect files in
|
|
300
|
-
`qa/{module-key}/{submodule-key}/06-defects/open/DEF-{NNN}.md` for each bug filed.
|
|
301
|
-
|
|
302
|
-
3. **Update `COVERAGE-MAPPING.md`**: set the `Status` column for each TC:
|
|
303
|
-
- `Automated` — test is ✅ High confidence
|
|
304
|
-
- `Skipped-Defect` — test is skipped via `test.skip()` with a DEF reference
|
|
305
|
-
- `Skipped-Infra` — test is skipped due to environment limitation
|
|
306
|
-
- `Manual` — TC is intentionally not automated
|
|
307
|
-
|
|
308
|
-
4. **Update session-summary.md** for the submodule: mark Stage 3.5 as ✅ Complete and
|
|
309
|
-
record the overall confidence percentage.
|
|
@@ -1,247 +0,0 @@
|
|
|
1
|
-
# Agent Instructions: Azure DevOps Integration
|
|
2
|
-
|
|
3
|
-
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-ado-integration/SKILL.md`.
|
|
4
|
-
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
-
|
|
6
|
-
**File**: `agent-instructions/05-ado-integration.md`
|
|
7
|
-
**Purpose**: Instructions for an agent to create, configure, and maintain Azure DevOps Test Plans integration with Playwright automation results.
|
|
8
|
-
|
|
9
|
-
> **This section is OPTIONAL**. Only use it when `integrations.azureDevOps.enabled = true` in `qa-framework.config.json`.
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## Prerequisites
|
|
14
|
-
|
|
15
|
-
1. Azure DevOps PAT stored in `ADO_PAT` environment variable
|
|
16
|
-
Required scopes: `Work Items (Read, Write)`, `Test Management (Read, Write)`
|
|
17
|
-
2. `ADO_ORG` and `ADO_PROJECT` environment variables set
|
|
18
|
-
3. PowerShell 5.1+ available
|
|
19
|
-
4. `@alex_neo/playwright-azure-reporter` installed in the E2E package
|
|
20
|
-
5. `qa/08-azure-integration/module-registry.json` exists with module paths and plan IDs
|
|
21
|
-
|
|
22
|
-
---
|
|
23
|
-
|
|
24
|
-
## Step 1: Create ADO Test Plan
|
|
25
|
-
|
|
26
|
-
If no Test Plan exists yet:
|
|
27
|
-
|
|
28
|
-
```powershell
|
|
29
|
-
# qa/08-azure-integration/scripts/create-testplan-from-mapping.ps1
|
|
30
|
-
# Reads ado-ids-mapping-{project}.json and creates the plan + suites + TCs in ADO
|
|
31
|
-
# Parameters: -Organization $env:ADO_ORG -Project $env:ADO_PROJECT -Token $env:ADO_PAT
|
|
32
|
-
.\qa\08-azure-integration\scripts\create-testplan-from-mapping.ps1 `
|
|
33
|
-
-Organization $env:ADO_ORG `
|
|
34
|
-
-Project $env:ADO_PROJECT `
|
|
35
|
-
-Token $env:ADO_PAT `
|
|
36
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
After running, the script outputs a `TestPlanId` and `SuiteIds`.
|
|
40
|
-
Record these in `qa/08-azure-integration/module-registry.json`.
|
|
41
|
-
|
|
42
|
-
---
|
|
43
|
-
|
|
44
|
-
## Step 2: Inject ADO IDs into Spec Files
|
|
45
|
-
|
|
46
|
-
Once Test Cases are created in ADO and have Work Item IDs:
|
|
47
|
-
|
|
48
|
-
```powershell
|
|
49
|
-
# Dry run first to preview changes
|
|
50
|
-
.\qa\08-azure-integration\scripts\inject-ado-ids.ps1 `
|
|
51
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
|
|
52
|
-
-TestDir .\qa\07-automation\e2e\tests `
|
|
53
|
-
-DryRun
|
|
54
|
-
|
|
55
|
-
# Apply changes
|
|
56
|
-
.\qa\08-azure-integration\scripts\inject-ado-ids.ps1 `
|
|
57
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
|
|
58
|
-
-TestDir .\qa\07-automation\e2e\tests
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
This converts test titles from:
|
|
62
|
-
```
|
|
63
|
-
'[TC-OPER-CAT-001] Access catalog list @P0'
|
|
64
|
-
```
|
|
65
|
-
to:
|
|
66
|
-
```
|
|
67
|
-
'[22957] Access catalog list @P0'
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
Plus adds annotation:
|
|
71
|
-
```typescript
|
|
72
|
-
test.info().annotations.push({ type: 'TestCase', description: '22957' });
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
---
|
|
76
|
-
|
|
77
|
-
## Step 3: Configure playwright-azure-reporter
|
|
78
|
-
|
|
79
|
-
In `playwright.config.ts`:
|
|
80
|
-
|
|
81
|
-
```typescript
|
|
82
|
-
import type { PlaywrightTestConfig } from '@playwright/test';
|
|
83
|
-
|
|
84
|
-
const config: PlaywrightTestConfig = {
|
|
85
|
-
reporter: [
|
|
86
|
-
['html'],
|
|
87
|
-
['@alex_neo/playwright-azure-reporter', {
|
|
88
|
-
orgUrl: `https://dev.azure.com/${process.env.ADO_ORG}`,
|
|
89
|
-
token: process.env.ADO_PAT,
|
|
90
|
-
planId: Number(process.env.ADO_PLAN_ID),
|
|
91
|
-
projectName: process.env.ADO_PROJECT,
|
|
92
|
-
isDisabled: !process.env.CI, // only publish in CI
|
|
93
|
-
publishTestResultsMode: 'testRun',
|
|
94
|
-
testRunTitle: `QA Run - ${new Date().toISOString()}`,
|
|
95
|
-
uploadAttachments: true,
|
|
96
|
-
attachmentsType: ['screenshot', 'video', 'trace'],
|
|
97
|
-
}],
|
|
98
|
-
],
|
|
99
|
-
};
|
|
100
|
-
export default config;
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
**Note**: Set `isDisabled: !process.env.CI` so the reporter only publishes from CI, not from local runs.
|
|
104
|
-
|
|
105
|
-
---
|
|
106
|
-
|
|
107
|
-
## Step 4: Module Registry
|
|
108
|
-
|
|
109
|
-
Maintain `qa/08-azure-integration/module-registry.json`:
|
|
110
|
-
|
|
111
|
-
```json
|
|
112
|
-
{
|
|
113
|
-
"modules": [
|
|
114
|
-
{
|
|
115
|
-
"name": "operacion",
|
|
116
|
-
"specsPath": "qa/07-automation/e2e/tests/operacion",
|
|
117
|
-
"planId": 0,
|
|
118
|
-
"suiteId": 0,
|
|
119
|
-
"description": "Módulo Operación — 7 submodules"
|
|
120
|
-
}
|
|
121
|
-
]
|
|
122
|
-
}
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
The CI pipeline uses this file to detect which module changed (via `git diff`) and run only the relevant spec.
|
|
126
|
-
|
|
127
|
-
---
|
|
128
|
-
|
|
129
|
-
## Step 5: ADO IDs Mapping File
|
|
130
|
-
|
|
131
|
-
Maintain `qa/08-azure-integration/ado-ids-mapping-{project}.json`:
|
|
132
|
-
|
|
133
|
-
```json
|
|
134
|
-
{
|
|
135
|
-
"planId": 22304,
|
|
136
|
-
"module": "operacion",
|
|
137
|
-
"specsPath": "qa/07-automation/e2e/tests/operacion",
|
|
138
|
-
"suites": [
|
|
139
|
-
{
|
|
140
|
-
"suiteId": 22956,
|
|
141
|
-
"suiteName": "Suite 1.1 - Catálogos de Trabajo",
|
|
142
|
-
"testCases": [
|
|
143
|
-
{ "id": 22957, "title": "Access catalog list @P0", "tags": ["automatable", "playwright"] },
|
|
144
|
-
{ "id": 22958, "title": "Create labor type @P1", "tags": ["automatable", "playwright"] }
|
|
145
|
-
]
|
|
146
|
-
}
|
|
147
|
-
]
|
|
148
|
-
}
|
|
149
|
-
```
|
|
150
|
-
|
|
151
|
-
---
|
|
152
|
-
|
|
153
|
-
## Step 6: Configure CI Pipeline
|
|
154
|
-
|
|
155
|
-
Template in `qa/08-azure-integration/pipelines/azure-pipeline-qa.yml`:
|
|
156
|
-
|
|
157
|
-
```yaml
|
|
158
|
-
trigger:
|
|
159
|
-
branches:
|
|
160
|
-
include:
|
|
161
|
-
- {{CI_TRIGGER_BRANCH}}
|
|
162
|
-
|
|
163
|
-
pool:
|
|
164
|
-
vmImage: 'ubuntu-latest'
|
|
165
|
-
|
|
166
|
-
variables:
|
|
167
|
-
- group: qa-secrets # Must contain: QA_USER_EMAIL, QA_USER_PASSWORD, QA_BASE_URL, ADO_PAT
|
|
168
|
-
|
|
169
|
-
steps:
|
|
170
|
-
- checkout: self
|
|
171
|
-
fetchDepth: 2
|
|
172
|
-
|
|
173
|
-
- task: NodeTool@0
|
|
174
|
-
inputs:
|
|
175
|
-
versionSpec: '20.x'
|
|
176
|
-
displayName: 'Install Node.js 20'
|
|
177
|
-
|
|
178
|
-
- script: |
|
|
179
|
-
cd qa/07-automation/e2e
|
|
180
|
-
npm install
|
|
181
|
-
npx playwright install chromium --with-deps
|
|
182
|
-
displayName: 'Install Playwright'
|
|
183
|
-
|
|
184
|
-
- script: |
|
|
185
|
-
cd qa/07-automation/e2e
|
|
186
|
-
npx playwright test
|
|
187
|
-
displayName: 'Run Playwright tests'
|
|
188
|
-
env:
|
|
189
|
-
QA_BASE_URL: $(QA_BASE_URL)
|
|
190
|
-
QA_USER_EMAIL: $(QA_USER_EMAIL)
|
|
191
|
-
QA_USER_PASSWORD: $(QA_USER_PASSWORD)
|
|
192
|
-
ADO_PAT: $(ADO_PAT)
|
|
193
|
-
ADO_PLAN_ID: $(ADO_PLAN_ID)
|
|
194
|
-
CI: 'true'
|
|
195
|
-
|
|
196
|
-
- task: PublishTestResults@2
|
|
197
|
-
inputs:
|
|
198
|
-
testResultsFormat: 'JUnit'
|
|
199
|
-
testResultsFiles: '**/test-results/*.xml'
|
|
200
|
-
displayName: 'Publish JUnit results'
|
|
201
|
-
condition: always()
|
|
202
|
-
|
|
203
|
-
- task: PublishPipelineArtifact@1
|
|
204
|
-
inputs:
|
|
205
|
-
targetPath: 'qa/07-automation/e2e/playwright-report'
|
|
206
|
-
artifact: 'playwright-html-report'
|
|
207
|
-
displayName: 'Publish HTML report'
|
|
208
|
-
condition: always()
|
|
209
|
-
```
|
|
210
|
-
|
|
211
|
-
---
|
|
212
|
-
|
|
213
|
-
## Syncing ADO TC Titles with Spec Titles
|
|
214
|
-
|
|
215
|
-
If TC titles drift between ADO and the spec files:
|
|
216
|
-
|
|
217
|
-
```powershell
|
|
218
|
-
.\qa\08-azure-integration\scripts\sync-ado-titles.ps1 `
|
|
219
|
-
-Organization $env:ADO_ORG `
|
|
220
|
-
-Project $env:ADO_PROJECT `
|
|
221
|
-
-Token $env:ADO_PAT `
|
|
222
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
|
|
223
|
-
-DryRun
|
|
224
|
-
```
|
|
225
|
-
|
|
226
|
-
This reads ADO TC titles and compares them against the spec. Outputs a diff report. Apply with `-DryRun:$false`.
|
|
227
|
-
|
|
228
|
-
---
|
|
229
|
-
|
|
230
|
-
## Credential Security Rules for ADO
|
|
231
|
-
|
|
232
|
-
1. `ADO_PAT` is **never** committed to any file — always comes from env var or CI variable group
|
|
233
|
-
2. `ADO_PAT` is **never** logged, printed, or written to any markdown file
|
|
234
|
-
3. If a PAT appears in any log or output, revoke it immediately in ADO Portal and generate a new one
|
|
235
|
-
4. Variable group `qa-secrets` in Azure DevOps must be scoped to the pipeline only
|
|
236
|
-
5. PAT minimum scopes: `Work Items (Read, Write)`, `Test Management (Read, Write)` — do not use full access PATs
|
|
237
|
-
|
|
238
|
-
---
|
|
239
|
-
|
|
240
|
-
## Troubleshooting
|
|
241
|
-
|
|
242
|
-
| Symptom | Likely Cause | Fix |
|
|
243
|
-
|---------|-------------|-----|
|
|
244
|
-
| Reporter silently does nothing in CI | `isDisabled: true` or `CI` env not set | Verify `process.env.CI` is `'true'` in pipeline |
|
|
245
|
-
| Tests report as "Not Run" in ADO | ADO WI ID not in test title or annotation | Re-run `inject-ado-ids.ps1` |
|
|
246
|
-
| TestRun created but no results attached | `publishTestResultsMode` wrong | Use `'testRun'` not `'testCase'` |
|
|
247
|
-
| PAT error 401 | PAT expired or wrong scopes | Generate new PAT with correct scopes |
|
|
@@ -1,135 +0,0 @@
|
|
|
1
|
-
# Agent Instructions: Maintenance and Updates
|
|
2
|
-
|
|
3
|
-
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-maintenance/SKILL.md`.
|
|
4
|
-
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
-
|
|
6
|
-
**File**: `agent-instructions/06-maintenance.md`
|
|
7
|
-
**Purpose**: Instructions for updating QA artifacts after application code changes.
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## When to use
|
|
12
|
-
|
|
13
|
-
- A developer reports a feature change or new feature in a module
|
|
14
|
-
- A sprint delivers changes to an already-analyzed module
|
|
15
|
-
- A defect is resolved and related `test.skip()` calls need to be reactivated
|
|
16
|
-
- The QA environment is updated and existing specs need to be refreshed
|
|
17
|
-
|
|
18
|
-
---
|
|
19
|
-
|
|
20
|
-
## Step 1: Identify What Changed
|
|
21
|
-
|
|
22
|
-
Gather from the developer or ticket:
|
|
23
|
-
- Which module/submodule was changed?
|
|
24
|
-
- Was it a new feature, a modification, or a bug fix?
|
|
25
|
-
- Are there new routes, fields, or API endpoints?
|
|
26
|
-
- Were any existing behaviors removed or changed?
|
|
27
|
-
|
|
28
|
-
If the change scope is unclear, navigate to the affected module in the QA environment and compare against the existing `00-inventory.md`.
|
|
29
|
-
|
|
30
|
-
---
|
|
31
|
-
|
|
32
|
-
## Step 2: Spec Update Rules
|
|
33
|
-
|
|
34
|
-
### A new field was added to a form
|
|
35
|
-
|
|
36
|
-
1. Add the field to `00-inventory.md` under UI Elements table
|
|
37
|
-
2. If the field has validation: add a business rule to `01-business-rules.md` (`RN-{MODULE}-{next-number}`)
|
|
38
|
-
3. Add TCs to `05-test-scenarios.md`:
|
|
39
|
-
- TC for happy path with the new field
|
|
40
|
-
- TC for required validation (if applicable)
|
|
41
|
-
- TC for any new business rule
|
|
42
|
-
|
|
43
|
-
### A field was removed from the UI
|
|
44
|
-
|
|
45
|
-
1. Mark the field in `00-inventory.md` as `[REMOVED - {YYYY-MM-DD}]` — do not delete the row
|
|
46
|
-
2. If automation tests reference the field selector: update or skip them
|
|
47
|
-
3. If a business rule only applied to that field: mark as `[DEPRECATED - {YYYY-MM-DD}]`
|
|
48
|
-
4. Leave TCs in `05-test-scenarios.md` marked as `[DEPRECATED]` for traceability
|
|
49
|
-
|
|
50
|
-
### A workflow was changed
|
|
51
|
-
|
|
52
|
-
1. Update `02-workflows.md` with the new flow
|
|
53
|
-
2. Mark the old version as `[PREVIOUS - {YYYY-MM-DD}]`
|
|
54
|
-
3. Review all TCs that test the affected steps
|
|
55
|
-
4. Update expected results in `05-test-scenarios.md`
|
|
56
|
-
|
|
57
|
-
### A new submodule was added
|
|
58
|
-
|
|
59
|
-
Follow the full module analysis process in `00-module-analysis.md`.
|
|
60
|
-
|
|
61
|
-
### A defect was fixed
|
|
62
|
-
|
|
63
|
-
1. Find all `test.skip()` calls that reference the DEF-ID or ADO WI
|
|
64
|
-
2. Remove the `test.skip()` wrapper
|
|
65
|
-
3. Run the test at least 2 times with different EXEC_IDX values to confirm stability
|
|
66
|
-
4. Update `06-defects/DEF-{NNN}.md` (or ADO WI) status to Resolved
|
|
67
|
-
5. If the spec assertion was adapted to document the bug (e.g., `toBeFalsy()` for a wrong default), restore the correct assertion
|
|
68
|
-
|
|
69
|
-
---
|
|
70
|
-
|
|
71
|
-
## Step 3: Automation Update Rules
|
|
72
|
-
|
|
73
|
-
### Selectors changed (redesign, framework update)
|
|
74
|
-
|
|
75
|
-
1. Locate the POM file or spec file that contains the old selector
|
|
76
|
-
2. Check `CORRECTIONS` block at top of spec file (convention for documenting selector audits)
|
|
77
|
-
3. Update the selector
|
|
78
|
-
4. Run the affected test 2+ times to confirm the new selector is stable
|
|
79
|
-
5. Add an entry to the `CORRECTIONS` block noting the change and date
|
|
80
|
-
|
|
81
|
-
### New test suite needed
|
|
82
|
-
|
|
83
|
-
Follow `04-automation-generation.md` for the new suite.
|
|
84
|
-
Update `COVERAGE-MAPPING.md` with the new TCs.
|
|
85
|
-
|
|
86
|
-
### Test became flaky after app update
|
|
87
|
-
|
|
88
|
-
Use the 5-layer debugging methodology:
|
|
89
|
-
1. Environment: is QA env stable?
|
|
90
|
-
2. Timeout: did an operation become slower?
|
|
91
|
-
3. Data: did a seed or precondition change?
|
|
92
|
-
4. Selector: did the element structure change?
|
|
93
|
-
5. System: is this a real defect?
|
|
94
|
-
|
|
95
|
-
---
|
|
96
|
-
|
|
97
|
-
## Step 4: Version the Spec
|
|
98
|
-
|
|
99
|
-
At the bottom of each updated spec file, add to the changelog:
|
|
100
|
-
|
|
101
|
-
```markdown
|
|
102
|
-
## Changelog
|
|
103
|
-
|
|
104
|
-
| Version | Date | Description |
|
|
105
|
-
|---------|------|-------------|
|
|
106
|
-
| 1.0 | YYYY-MM-DD | Initial creation |
|
|
107
|
-
| 1.1 | YYYY-MM-DD | Added RN-{MODULE}-{NNN}: {rule title}. TC count: N → M |
|
|
108
|
-
| 1.2 | YYYY-MM-DD | Removed field {name} from UI inventory (deprecated) |
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
---
|
|
112
|
-
|
|
113
|
-
## Step 5: Update qa/README.md
|
|
114
|
-
|
|
115
|
-
After every maintenance session, update the master index:
|
|
116
|
-
- TC count (if changed)
|
|
117
|
-
- Automation status (if changed)
|
|
118
|
-
- Last updated date for the module
|
|
119
|
-
|
|
120
|
-
If a **sprint was fully completed** as part of this maintenance pass (all TCs passing after the changes):
|
|
121
|
-
1. Move the completed sprint checklist from `AGENT-NEXT-STEPS.md` to the `## Sprint History` section of `qa/README.md`
|
|
122
|
-
2. Delete the completed sprint block from `AGENT-NEXT-STEPS.md`
|
|
123
|
-
3. Confirm `AGENT-NEXT-STEPS.md` contains **only one active sprint** — never let it accumulate history
|
|
124
|
-
|
|
125
|
-
Also update `qa/memory/INDEX.md` if any memory files were created or modified.
|
|
126
|
-
|
|
127
|
-
---
|
|
128
|
-
|
|
129
|
-
## Step 6: Write a Session Summary
|
|
130
|
-
|
|
131
|
-
Create `qa/SESSION-SUMMARY-{YYYY-MM-DD}-maintenance.md` documenting:
|
|
132
|
-
- What changed
|
|
133
|
-
- Which files were updated
|
|
134
|
-
- Any new TCs or deprecated TCs
|
|
135
|
-
- Any tests that were reactivated or newly skipped
|