@keber/qa-framework 1.5.2 → 1.6.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,343 +0,0 @@
1
- # Agent Instructions: Automation Generation
2
-
3
- > ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-automation/SKILL.md`.
4
- > Kept as reference during the transition period. Do not update this file.
5
-
6
- **File**: `agent-instructions/04-automation-generation.md`
7
- **Purpose**: Instructions for writing Playwright E2E automation tests that trace back to approved test cases in `01-specifications/`.
8
-
9
- ---
10
-
11
- > **Prerequisite**: Specifications must be reviewed and approved before writing automation. Do not automate TCs that are `PENDING-CODE` or `BLOCKED-PERMISSIONS`.
12
-
13
- ---
14
-
15
- ## Structure: How to Organize Spec Files
16
-
17
- Each module gets a directory under `qa/07-automation/e2e/tests/`:
18
-
19
- ```
20
- qa/07-automation/e2e/tests/
21
- └── {module-kebab}/
22
- ├── COVERAGE-MAPPING.md ← TC-ID → spec location map
23
- ├── {suite-name}.spec.ts ← P0 suite
24
- ├── {suite-name}-p1.spec.ts ← P1 suite
25
- └── integration/
26
- └── cross-module.spec.ts ← Cross-module dependencies
27
- ```
28
-
29
- Spec file naming: `{description}.spec.ts` in kebab-case.
30
-
31
- ---
32
-
33
- ## Spec File Template
34
-
35
- ```typescript
36
- /**
37
- * @module {MODULE_CODE}
38
- * @submodule {SUBMODULE_CODE}
39
- * @spec qa/01-specifications/module-{name}/submodule-{name}/05-test-scenarios.md
40
- * @priority P0
41
- */
42
-
43
- import { test, expect } from '@playwright/test';
44
-
45
- // EXEC_IDX: changes every 60 seconds, provides collision-resistant unique values
46
- const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
47
-
48
- test.describe('{Feature Suite Name}', () => {
49
-
50
- test.beforeAll(async ({ browser }) => {
51
- // Provision data needed by tests in this suite
52
- // Rule: provision AT LEAST as many records as tests that consume them
53
- });
54
-
55
- test('[TC-{MODULE}-{SUB}-001] {TC title} @P0', async ({ page }) => {
56
- // Arrange
57
- // ...
58
-
59
- // Act
60
- // ...
61
-
62
- // Assert
63
- // ...
64
- });
65
-
66
- test('[TC-{MODULE}-{SUB}-002] {TC title} @P1', async ({ page }) => {
67
- // ...
68
- });
69
-
70
- test.skip('[TC-{MODULE}-{SUB}-003] {TC title} @P1 — DEF-{NNN}: {description}. Reactivate when {ADO WI or DEF ID} is resolved.', async ({ page }) => {
71
- // Skip body: write the test as if the bug were fixed
72
- // This documents the expected behavior, not the buggy state
73
- });
74
-
75
- });
76
- ```
77
-
78
- ---
79
-
80
- ## Required Patterns
81
-
82
- ### Pattern 1: EXEC_IDX — Unique Test Data
83
-
84
- Use EXEC_IDX everywhere you create records to avoid collisions between consecutive test runs:
85
-
86
- ```typescript
87
- const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
88
-
89
- // For titles/names
90
- const entityName = `Test-Record-${EXEC_IDX}`;
91
-
92
- // For dates (future-dated to avoid prod data collisions)
93
- const year = 2120 + (EXEC_IDX % 10);
94
- const month = String((EXEC_IDX % 12) + 1).padStart(2, '0');
95
- const day = String((EXEC_IDX % 28) + 1).padStart(2, '0');
96
- const testDate = `${year}-${month}-${day}`;
97
- ```
98
-
99
- Why year 2120+: avoids any overlap with real production data.
100
- Why modular arithmetic: the date always represents a valid date, never day 32 or month 13.
101
-
102
- ### Pattern 2: Multi-Role Auth with storageState
103
-
104
- Use `global-setup.ts` to log in once per role and save storageState:
105
-
106
- ```typescript
107
- // playwright.config.ts
108
- use: {
109
- storageState: '.auth/session.json', // single role
110
- // or: storageState: (workerInfo) => `.auth/role-${workerInfo.workerIndex}.json`
111
- }
112
-
113
- // global-setup.ts — runs once before all tests
114
- async function globalSetup(config) {
115
- const browser = await chromium.launch();
116
- const page = await browser.newPage();
117
-
118
- // Log in and save session
119
- await page.goto(process.env.QA_BASE_URL + process.env.QA_LOGIN_PATH);
120
- await page.fill(process.env.QA_LOGIN_EMAIL_SELECTOR, process.env.QA_USER_EMAIL);
121
- // Use evaluate() for password — excluded from traces
122
- await page.evaluate(
123
- ([sel, pwd]) => { (document.querySelector(sel) as HTMLInputElement).value = pwd; },
124
- [process.env.QA_LOGIN_PASSWORD_SELECTOR, process.env.QA_USER_PASSWORD]
125
- );
126
- await page.click(process.env.QA_LOGIN_SUBMIT_SELECTOR);
127
- await page.waitForURL(url => !url.includes(process.env.QA_LOGIN_PATH));
128
-
129
- await page.context().storageState({ path: '.auth/session.json' });
130
- await browser.close();
131
- }
132
- ```
133
-
134
- For multi-role: call login sequence sequentially for each role (apps often reject concurrent sessions).
135
-
136
- ### Pattern 3: Fast-Fail for Status Checks
137
-
138
- When checking the text content of status indicators, use a short timeout to prevent hanging:
139
-
140
- ```typescript
141
- // BAD: inherits the full test timeout (30s or more)
142
- const status = await page.locator('.status-badge').textContent();
143
-
144
- // GOOD: fails fast if not available in 3 seconds
145
- const status = await page.locator('.status-badge').textContent({ timeout: 3_000 });
146
- ```
147
-
148
- ### Pattern 4: Email Validation — 3-Layer Strategy
149
-
150
- When testing email notification behavior without a real inbox:
151
-
152
- ```typescript
153
- // Layer 1: HTTP route interception (most reliable)
154
- const emailRequests: string[] = [];
155
- await page.route('**/*mail*', (route) => {
156
- emailRequests.push(route.request().url());
157
- route.continue();
158
- });
159
- await page.route('**/*correo*', (route) => {
160
- emailRequests.push(route.request().url());
161
- route.continue();
162
- });
163
- // ... perform the action ...
164
- expect(emailRequests.length).toBeGreaterThan(0);
165
-
166
- // Layer 2: UI confirmation (toast, banner, badge)
167
- await expect(page.locator('.email-sent-indicator')).toBeVisible();
168
-
169
- // Layer 3: test.fixme() — if inbox access is needed and not available
170
- test.fixme(process.env.QA_MAILBOX_ACCESS !== 'true', 'Requires mailbox access');
171
- ```
172
-
173
- ### Pattern 5: Skip Pattern for Known Bugs
174
-
175
- ```typescript
176
- test.skip(true,
177
- 'DEF-001: {Description of bug}. ' +
178
- 'Expected: {expected behavior}. ' +
179
- 'Actual: {buggy behavior}. ' +
180
- 'Reactivate when ADO #{WI_ID} / DEF-001 is resolved.'
181
- );
182
- ```
183
-
184
- **Never** make a test artificially "pass" around a bug. Document with exact expected behavior.
185
-
186
- ### Pattern 6: beforeAll Provisioning Ratio
187
-
188
- In `beforeAll`, create **at least as many records as the number of tests that consume them**:
189
-
190
- ```typescript
191
- test.beforeAll(async ({ browser }) => {
192
- const page = await browser.newPage();
193
- // If 3 tests each need 1 different pending record:
194
- const records: string[] = [];
195
- for (let i = 0; i < 3; i++) {
196
- const id = await createRecord(page, `Record-${EXEC_IDX}-${i}`);
197
- records.push(id);
198
- }
199
- // Store in closure or page object for use in tests
200
- });
201
- ```
202
-
203
- ### Pattern 7: Password Injection (Trace Safety)
204
-
205
- ```typescript
206
- // Passwords must be set via evaluate() to exclude from Playwright traces
207
- await page.evaluate(
208
- ([selector, pwd]) => {
209
- (document.querySelector(selector) as HTMLInputElement).value = pwd;
210
- },
211
- [process.env.QA_LOGIN_PASSWORD_SELECTOR, process.env.QA_USER_PASSWORD]
212
- );
213
- ```
214
-
215
- ---
216
-
217
- ## `playwright.config.ts` Checklist
218
-
219
- Every project Playwright config should configure:
220
-
221
- ```typescript
222
- export default defineConfig({
223
- testDir: './tests',
224
- timeout: 30_000, // Per-test timeout
225
- actionTimeout: 10_000, // Per-action timeout (REQUIRED — prevents silent hangs)
226
- navigationTimeout: 60_000, // Per-navigation timeout
227
- retries: process.env.CI ? 2 : 0,
228
- workers: process.env.CI ? 1 : 2,
229
-
230
- // Exclude seeds and debug scripts from normal runs
231
- testIgnore: ['**/seeds/**', '**/debug-*.spec.ts', '**/seed.spec.ts'],
232
-
233
- use: {
234
- baseURL: process.env.QA_BASE_URL,
235
- storageState: '.auth/session.json',
236
- locale: 'en-US', // or 'es-CL' for Chilean apps
237
- screenshot: 'only-on-failure',
238
- video: 'retain-on-failure',
239
- trace: 'on-first-retry',
240
- },
241
-
242
- reporter: [
243
- ['html', { outputFolder: 'playwright-report' }],
244
- // ADO reporter: only when integration enabled
245
- // ['@alex_neo/playwright-azure-reporter', { ... }],
246
- ],
247
-
248
- projects: [
249
- { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
250
- // Add firefox, webkit when cross-browser coverage is needed
251
- ],
252
-
253
- globalSetup: './global-setup.ts',
254
- outputDir: '../../05-test-execution/automated/test-results',
255
- });
256
- ```
257
-
258
- `actionTimeout` is **not optional**. Without it, `textContent()`, `getAttribute()`, and similar calls inherit the full test timeout and can silently hang if the element is slow.
259
-
260
- ---
261
-
262
- ## Stability Criteria
263
-
264
- A test is only considered stable when:
265
-
266
- 1. It passes ≥ 2 consecutive runs with **different** `EXEC_IDX` values (i.e., at least 60 seconds apart)
267
- 2. It does not rely on another test's side effects
268
- 3. It produces the same result across 3+ independent runs
269
- 4. No `waitForTimeout(N)` longer than 2000ms is used (use `waitForSelector` or `waitForLoadState` instead)
270
-
271
- ---
272
-
273
- ## Debugging Failed Tests
274
-
275
- Use this 5-layer methodology:
276
-
277
- 1. **Environment layer**: Is QA_BASE_URL reachable? Are credentials valid? Run login test first.
278
- 2. **Timeout layer**: Did the test fail with "timeout" or "Timeout exceeded"? Check `actionTimeout`, increase if justified.
279
- 3. **Data layer**: Did `beforeAll` create the expected records? Add `console.log` per step in `beforeAll`.
280
- 4. **Code layer**: Is the selector correct? Use page.pause() or Playwright UI mode to inspect.
281
- 5. **System defect layer**: Is the failure due to a real bug? File a defect and add `test.skip()`.
282
-
283
- **Diagnostic script pattern**: For complex data discovery, create a diagnostic spec in `helpers/debug/`:
284
-
285
- ```typescript
286
- // helpers/debug/inspect-{feature}.spec.ts
287
- test('diagnostic: dump form options', async ({ page }) => {
288
- await page.goto(`${process.env.QA_BASE_URL}/my-form`);
289
- const options = await page.$$eval('select option', els =>
290
- els.map(el => ({ value: el.value, text: el.textContent }))
291
- );
292
- console.log(JSON.stringify(options, null, 2));
293
- });
294
- ```
295
-
296
- This is not part of the test suite (excluded via `testIgnore`). Use it to discover valid option values.
297
-
298
- ---
299
-
300
- ## Coverage Mapping Document
301
-
302
- After writing automation for a module, create `COVERAGE-MAPPING.md` in the test directory:
303
-
304
- ```markdown
305
- # Coverage Mapping — {Module}
306
-
307
- | TC ID | Title | Spec file | Playwright spec | Status |
308
- |-------|-------|-----------|----------------|--------|
309
- | TC-OPER-CAT-001 | Access catalog list | 01-specifications/.../05-test-scenarios.md | tests/operacion/catalogos/critical.spec.ts | ✅ Automated |
310
- | TC-OPER-CAT-002 | Create labor type | ... | tests/operacion/... | ✅ Automated |
311
- | TC-OPER-CAT-003 | Duplicate rejection | ... | tests/operacion/... | ⚠️ Partially (UI only) |
312
- | TC-OPER-ASI-001 | Access assistance list | ... | — | ⛔ Blocked (BLOCKED-PERMISSIONS) |
313
- ```
314
-
315
- ---
316
-
317
- ## Module Completion Checklist
318
-
319
- Run this checklist when all P0 tests for a module are passing.
320
-
321
- ### Stability gate
322
- - [ ] All P0 tests pass on **2 consecutive runs** with different `EXEC_IDX` values (≥ 60 seconds apart)
323
- - [ ] No `waitForTimeout(N)` longer than 2000ms left in the suite
324
- - [ ] All skipped tests have a `DEF-` or ADO WI reference in their skip message
325
-
326
- ### Artifacts
327
- - [ ] `COVERAGE-MAPPING.md` created or updated in the test directory
328
- - [ ] Execution report written to `qa/05-test-execution/automated/`
329
-
330
- ### Living Index update — `qa/README.md`
331
- - [ ] Update the module's row in the status table (TCs automated, date, status → ✅)
332
- - [ ] Add a row to the `## Sprint History` section with: sprint name, date, submódulos covered, TC count, link to execution report
333
-
334
- ### AGENT-NEXT-STEPS.md — trim
335
- - [ ] Move the completed sprint checklist block to `qa/README.md` under `## Sprint History`
336
- - [ ] Delete the completed sprint section from `AGENT-NEXT-STEPS.md`
337
- - [ ] Verify `AGENT-NEXT-STEPS.md` now contains **only one active sprint** (the next one)
338
-
339
- ### Memory
340
- - [ ] If new patterns or environment quirks were discovered: add them to `qa/memory/` with a descriptive filename
341
- - [ ] Update `qa/memory/INDEX.md` to reflect any new or updated files
342
-
343
- > **Rule**: `AGENT-NEXT-STEPS.md` is a task queue, not a log. Completed sprints live in `qa/README.md`. The file must never accumulate more than one active sprint.
@@ -1,309 +0,0 @@
1
- # Agent Instructions: Test Stabilization
2
-
3
- > ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-stabilization/SKILL.md`.
4
- > Kept as reference during the transition period. Do not update this file.
5
-
6
- **File**: `agent-instructions/04b-test-stabilization.md`
7
- **Purpose**: Instructions for exhaustively reviewing and stabilizing generated Playwright tests to ensure they are logically correct, technically sound, and free of false positives and false negatives before ADO integration.
8
-
9
- ---
10
-
11
- ## Stage Context
12
-
13
- | Field | Value |
14
- |-------|-------|
15
- | Stage number | 3.5 |
16
- | Stage name | Test Stabilization |
17
- | Preceding stage | Stage 4 — Automation Generation (`04-automation-generation.md`) |
18
- | Following stage | Stage 5 — ADO Integration (`05-ado-integration.md`) |
19
- | Can be skipped? | No — ADO integration with unreliable tests produces misleading test run history |
20
- | Required inputs | Generated `.spec.ts` files in `qa/07-automation/e2e/`; approved `05-test-scenarios.md` for the module |
21
- | Produced outputs | Stabilized spec files; `STABILIZATION-REPORT.md` per submodule |
22
- | Exit criterion | Confidence ≥ 90% that every test produces a true positive on failure and a true negative on pass |
23
-
24
- ---
25
-
26
- ## When to use
27
-
28
- Run this stage immediately after **Stage 4 — Automation Generation** and before any ADO
29
- integration or CI pipeline registration. This stage is mandatory regardless of whether tests
30
- are passing or failing after generation.
31
-
32
- Also re-run this stage when:
33
- - Sprint changes introduce significant new test code (more than 3 new spec files)
34
- - A test suite starts flaking in CI after a previously stable period
35
- - A test is suspected of being a false positive (passing despite the feature being broken)
36
- - A test is suspected of being a false negative (failing despite the feature being correct)
37
-
38
- ---
39
-
40
- > **Scope boundary**: This stage operates exclusively on test code (`.spec.ts`, fixtures,
41
- > helpers, `global-setup.ts`, `playwright.config.ts`). The application source code is
42
- > strictly out of scope. Do not modify, patch, or work around application behavior to make
43
- > tests pass — document the mismatch as a defect instead.
44
-
45
- ---
46
-
47
- > **Coverage boundary**: Do not add new tests or increase coverage during this stage. If a
48
- > gap in coverage is identified, note it in the STABILIZATION-REPORT.md for Stage 6
49
- > (Maintenance) to address. The goal is correctness of existing tests, not completeness.
50
-
51
- ---
52
-
53
- ## Step 1 — Establish the Reference Baseline
54
-
55
- Before running any tests, collect the ground truth against which correctness will be judged.
56
-
57
- 1. Open `05-test-scenarios.md` for the target module/submodule. For each TC row, note:
58
- - TC-ID and title
59
- - Priority (P0/P1/P2/P3)
60
- - Expected result (acceptance criterion)
61
- - Automatable status (Yes / Partial / No)
62
-
63
- 2. Open `COVERAGE-MAPPING.md` for the module. For each row:
64
- - Confirm the `.spec.ts` file and test function name exist
65
- - Flag any rows where the spec or function is missing (these are generation gaps, not
66
- stabilization issues — document in the report but do not create new tests)
67
-
68
- 3. If `05-test-scenarios.md` or `COVERAGE-MAPPING.md` are absent or incomplete, stop and
69
- complete Stage 4 before proceeding.
70
-
71
- ---
72
-
73
- ## Step 2 — First Run: Collect Baseline Results
74
-
75
- Run the full suite for the target module without any modifications:
76
-
77
- ```bash
78
- npx playwright test qa/07-automation/e2e/tests/{module-kebab}/ --reporter=list
79
- ```
80
-
81
- Record the outcome of every test:
82
-
83
- | TC-ID | Test title | Result | Notes |
84
- |-------|-----------|--------|-------|
85
- | TC-MOD-SUB-001 | ... | ✅ pass / ❌ fail / ⚠️ flaky / ⏭ skip | Error message if fail |
86
-
87
- Collect trace artifacts for every failing test:
88
-
89
- ```bash
90
- npx playwright test ... --trace=on
91
- ```
92
-
93
- Do NOT make any changes yet. The first run result is the diagnostic baseline.
94
-
95
- ---
96
-
97
- ## Step 3 — Classify Every Failing or Skipped Test
98
-
99
- For each non-passing test, determine its failure category before writing any code.
100
-
101
- ### Failure Categories
102
-
103
- | Category | Definition | Correct action |
104
- |----------|-----------|----------------|
105
- | **A — Test Bug: Wrong selector** | The selector no longer matches the UI element | Fix selector in `.spec.ts` |
106
- | **B — Test Bug: Wrong assertion** | The assertion does not match the specified acceptance criterion | Fix assertion to match spec |
107
- | **C — Test Bug: Wrong flow** | The test steps don't reflect the actual user flow described in the TC | Rewrite steps to match TC |
108
- | **D — Test Bug: Fragile timing** | Test fails intermittently due to missing `await`, race conditions, or missing `waitFor` | Add proper async handling |
109
- | **E — Test Bug: Incorrect data** | Test data is malformed, not unique, or conflicts with QA environment state | Fix data generation or provisioning |
110
- | **F — App Bug** | The test is correct and the application is not behaving as the TC specifies | Do not fix the test; file a defect |
111
- | **G — TC Mismatch** | The spec TC is ambiguous or incorrect vs the actual implemented behavior | Do not fix the test or the app; update `05-test-scenarios.md` and note as a spec defect |
112
- | **H — Infra / Environment** | Test fails due to QA environment unavailability, network, or credentials | Retry in a clean environment; do not classify as a test bug until confirmed |
113
- | **I — Intentional skip** | Test is marked `test.skip()` due to a known defect or pending feature | Verify skip reason is still valid; update `06-defects/open/` reference |
114
-
115
- For each failing test, write your classification in the working STABILIZATION-REPORT.md
116
- before making any change.
117
-
118
- ### Decision Protocol
119
-
120
- ```
121
- Is the test step sequence and assertion derived correctly from 05-test-scenarios.md?
122
- No → Category G (TC Mismatch — fix the spec, not the test)
123
- Yes ↓
124
- Does the application behave as the TC acceptance criterion describes?
125
- No → Category F (App Bug — file a defect, skip the test with a DEF reference)
126
- Yes ↓
127
- Does the failure appear in every run (deterministic)?
128
- No → Category D or H (timing or environment — diagnose further)
129
- Yes ↓
130
- Is the failure caused by a selector, assertion, data, or flow issue in the test code?
131
- → Category A, B, C, or E (Test Bug — fix the test code)
132
- ```
133
-
134
- ---
135
-
136
- ## Step 4 — Fix Iterations
137
-
138
- Apply fixes one category at a time, in this order: E → A → C → B → D.
139
- (Data problems mask selector problems; flow problems mask assertion problems.)
140
-
141
- After each batch of fixes, re-run the affected tests:
142
-
143
- ```bash
144
- npx playwright test qa/07-automation/e2e/tests/{module-kebab}/{file}.spec.ts --reporter=list
145
- ```
146
-
147
- Do not move to the next category until the current category's tests are stable across
148
- **at least two consecutive runs**.
149
-
150
- ### Rules for fixing test code
151
-
152
- - **Selectors**: prefer `getByRole`, `getByLabel`, `getByTestId` — avoid CSS/XPath selectors
153
- that depend on position or class names that aren't semantic
154
- - **Assertions**: every assertion must trace back to a specific acceptance criterion in the TC;
155
- remove or replace any `expect().toBeVisible()` that does not correspond to a TC requirement
156
- - **Waits**: prefer `await expect(locator).toBeVisible()` over `page.waitForTimeout()`;
157
- never use `waitForTimeout` as a fix for timing — diagnose the actual cause
158
- - **Data**: use `EXEC_IDX + RUN_SALT` for unique identifiers (see ISSUE-02 note below);
159
- provision data in `beforeAll`, not inside individual tests
160
- - **Skips**: when skipping for a Category F (App Bug), always include the defect reference:
161
- ```typescript
162
- test.skip(true, 'DEF-{NNN}: {one-line bug description}. Filed {YYYY-MM-DD}.');
163
- ```
164
-
165
- > ⚠️ **EXEC_IDX collision**: Two runs within the same 60-second window produce identical
166
- > `EXEC_IDX` values. For fields requiring guaranteed uniqueness, add a salt:
167
- > ```typescript
168
- > const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
169
- > const RUN_SALT = Math.floor(Math.random() * 9999).toString().padStart(4, '0');
170
- > ```
171
-
172
- ---
173
-
174
- ## Step 5 — Validate Passing Tests for False Positives
175
-
176
- A passing test is not necessarily a correct test. After fixing failing tests, audit every
177
- **passing** test for false positives.
178
-
179
- For each passing test, perform one of the following checks:
180
-
181
- ### Negation check (preferred)
182
- Temporarily modify the test to assert the opposite of the expected result, then run it.
183
- If it still passes — the test is a false positive (asserting nothing meaningful).
184
-
185
- ```typescript
186
- // Original: expect(toast).toHaveText('Guardado exitosamente');
187
- // Negation check: replace temporarily with:
188
- expect(toast).toHaveText('TEXTO_QUE_NO_EXISTE_JAMAS');
189
- // If this passes → the original assertion is not actually evaluating the toast content
190
- ```
191
-
192
- Restore the original assertion after the check.
193
-
194
- ### Trace review (secondary)
195
- Open the Playwright trace for the passing test and step through it. Verify:
196
- - Every action in the `test()` body corresponds to a step in the TC
197
- - Every `expect()` is actually being evaluated (no skipped assertions due to early returns
198
- or unreachable code)
199
- - The test does not pass because an early `expect()` short-circuits before the meaningful one
200
-
201
- ### API/network verification (for tests touching data persistence)
202
- For tests that create, update, or delete records: after the UI action, verify the result
203
- is persisted, not just that a success toast appeared. A test that only checks the toast
204
- is a false positive if the actual save fails silently.
205
-
206
- ---
207
-
208
- ## Step 6 — Iteration and Convergence
209
-
210
- Repeat Steps 4 and 5 until:
211
-
212
- 1. All deterministic failures are resolved (either fixed or legitimately skipped with a
213
- defect reference)
214
- 2. All passing tests have passed the negation check or trace review
215
- 3. No test produces intermittent results across 3 consecutive runs
216
-
217
- **Confidence scoring**: after each iteration, estimate confidence per test:
218
-
219
- | Confidence | Criteria |
220
- |-----------|---------|
221
- | ✅ High (≥90%) | Passed negation check AND matches TC acceptance criterion exactly |
222
- | ⚠️ Medium (70–89%) | Passes consistently; trace reviewed; no negation check yet |
223
- | ❌ Low (<70%) | Intermittent, unclassified failure, or negation check not attempted |
224
-
225
- The stage is complete when **all tests are ✅ High or legitimately skipped** (Category F
226
- with a defect reference or Category I with a valid skip reason).
227
-
228
- ---
229
-
230
- ## Step 7 — Document Findings
231
-
232
- For each submodule, create or update `STABILIZATION-REPORT.md` in the module's automation
233
- directory:
234
-
235
- ```
236
- qa/07-automation/e2e/tests/{module-kebab}/STABILIZATION-REPORT.md
237
- ```
238
-
239
- ### STABILIZATION-REPORT.md structure
240
-
241
- ```markdown
242
- # Stabilization Report — {MODULE} > {SUBMODULE}
243
-
244
- **Date**: YYYY-MM-DD
245
- **Sprint**: {sprint number or label}
246
- **Analyst**: {agent session name or human name}
247
- **Spec reference**: `qa/{module-key}/{submodule-key}/05-test-scenarios.md`
248
-
249
- ## Summary
250
-
251
- | Metric | Value |
252
- |--------|-------|
253
- | Tests evaluated | N |
254
- | Tests stabilized (now ✅ High confidence) | N |
255
- | Tests skipped — App Bug (Category F) | N |
256
- | Tests skipped — Intentional (Category I) | N |
257
- | Remaining low-confidence tests | N |
258
- | Overall confidence | NN% |
259
-
260
- ## Test-by-Test Record
261
-
262
- | TC-ID | Test title | Baseline result | Final result | Category | Changes made | Confidence |
263
- |-------|-----------|-----------------|-------------|----------|-------------|-----------|
264
- | TC-... | ... | ❌ fail | ✅ pass | A | Fixed selector `#old` → `getByRole('button', { name: '...' })` | ✅ High |
265
-
266
- ## App Bugs Filed
267
-
268
- | DEF-ID | TC-ID | Description | Filed date | ADO WI (if created) |
269
- |--------|-------|-------------|-----------|---------------------|
270
-
271
- ## Spec Defects (Category G)
272
-
273
- | TC-ID | Issue | Recommended `05-test-scenarios.md` update |
274
- |-------|-------|------------------------------------------|
275
-
276
- ## Coverage Gaps Identified (do not fix in this stage)
277
-
278
- - {TC-ID}: test mapped in COVERAGE-MAPPING.md but `.spec.ts` function not found
279
- - ...
280
-
281
- ## Decisions Log
282
-
283
- Chronological record of non-trivial decisions made during stabilization:
284
-
285
- - YYYY-MM-DD: Classified TC-XXX failure as Category F after verifying the save endpoint
286
- returns 500 on duplicate RUTs. App bug, not test bug. DEF-001 filed.
287
- - ...
288
- ```
289
-
290
- ---
291
-
292
- ## Step 8 — Update Upstream Artifacts
293
-
294
- After completing STABILIZATION-REPORT.md:
295
-
296
- 1. **If any Category G issues were found** (TC Mismatch): update `05-test-scenarios.md`
297
- with the corrected acceptance criteria. Note the change with `[REVISED - {YYYY-MM-DD}]`.
298
-
299
- 2. **If any Category F issues were found** (App Bug): create defect files in
300
- `qa/{module-key}/{submodule-key}/06-defects/open/DEF-{NNN}.md` for each bug filed.
301
-
302
- 3. **Update `COVERAGE-MAPPING.md`**: set the `Status` column for each TC:
303
- - `Automated` — test is ✅ High confidence
304
- - `Skipped-Defect` — test is skipped via `test.skip()` with a DEF reference
305
- - `Skipped-Infra` — test is skipped due to environment limitation
306
- - `Manual` — TC is intentionally not automated
307
-
308
- 4. **Update session-summary.md** for the submodule: mark Stage 3.5 as ✅ Complete and
309
- record the overall confidence percentage.