@keber/qa-framework 1.9.1 → 1.11.3

package/templates/memory/ci-pipeline-findings.md

@@ -0,0 +1,36 @@
+# CI Pipeline Findings
+
+> Document CI pipeline issues, root causes, and applied fixes as they are discovered.
+> This file is loaded by the agent when diagnosing CI failures or reviewing pipeline changes.
+
+## Applied Fixes
+
+Document each fix that was applied to the pipeline or test setup. Use a table or prose - whatever fits the change.
+
+| Problem | Root cause | Fix applied |
+|---|---|---|
+| _Add pipeline issue here_ | _Root cause_ | _What was changed and where_ |
+
+> Examples: auth setup defaults, missing reporter packages, artifact name collisions,
+> browser binary not installed in a CI stage, grep flag escaping issues.
+
+## Known Limitations
+
+Document constraints that are accepted but not fixed (timeout budgets, worker counts, retry behavior).
+
+> Examples:
+> - Full suite exceeds the CI timeout with 1 worker. Workaround: run only P0/P1 tags by default.
+> - `retries: 2` counts each attempt separately, so reported execution count exceeds test count.
+> - Attachment upload warnings on `PublishTestResults` when multiple failures share filenames.
+
+## Failure Triage Categories
+
+Use these categories when classifying CI failures. Add project-specific categories as needed.
+
+| Category | Description | Action |
+|---|---|---|
+| TIMEOUT | Test exceeded its time budget | Adjust `timeout` in the spec or in `playwright.config.ts` |
+| SESSION | Auth session expired mid-run | Implement session-refresh fixture or reduce suite duration |
+| FLAKY | Intermittent failure, passes on retry | Acceptable if retries cover it; investigate if failure rate > 20% |
+| APP-DEFECT | Application does not meet the spec | Open defect file; mark test with `test.fail()` |
+| TEST-CODE | Selector or assertion is wrong in CI | Fix the spec file |

package/templates/memory/data-volatility-strategies.md

@@ -0,0 +1,70 @@
+# Data Volatility Strategies
+
+> The QA environment is typically shared between automated tests and manual testers.
+> Data (record counts, available filter values, document states) changes constantly,
+> which invalidates hardcoded values in specs. Document the chosen strategy here.
+
+## Problem
+
+List the types of data that are volatile in this project's QA environment.
+
+| Data type | Example of fragile assertion | Tests affected |
+|---|---|---|
+| Filter option values | `hasText: 'specific-value'` | _Add count here_ |
+| Exact row counts | `toBe(33)` | _Add count here_ |
+| Record names used as search inputs | `fill('specific-record-name')` | _Add count here_ |
+| Process or transaction IDs | `TEST_ID = '608'` | _Add count here_ |
+
+## Strategy 1: Shared Constants with Dynamic Discovery
+
+Maintain a `fixtures/test-data-constants.ts` file with well-named constants. Populate them
+from a discovery step (global-setup or a `beforeAll` fixture) that reads the actual environment.
+
+```typescript
+// fixtures/test-data-constants.ts
+export const QA_ITEM_CURRENT = 'current-item-name';  // most recent available
+export const QA_ITEM_OLDEST  = 'oldest-item-name';   // oldest available
+export const QA_ITEM_INVALID = 'nonexistent-item';   // guaranteed absent
+```
+
+**Tradeoff:** eliminates hardcoded filter values centrally, but discovery adds startup time
+and blocks the suite if it fails.
+
+## Strategy 2: Resilient Assertions (Patterns and Ranges)
+
+Replace exact-value assertions with pattern or range checks.
+
+```typescript
+// Instead of:
+expect(count).toBe(33);
+await expect(counter).toHaveText('1-5 of 5');
+
+// Use:
+expect(count).toBeGreaterThan(0);
+await expect(counter).toHaveText(/\d+-\d+ of \d+/);
+```
+
+**Tradeoff:** no extra infrastructure, each test is self-contained, but precision is lower.
+Useful for count assertions; less useful for filter-value selection.
+
+## Strategy 3: Snapshot Delta Validation
+
+Capture the initial state before applying a filter, then assert that the filtered state
+differs from the initial state in the expected direction (fewer rows, etc.).
+
+**Tradeoff:** maximum resilience, but tests become longer and harder to trace. Use only for
+critical filter tests where behavior (not data) must be validated.
+
+## Chosen Approach
+
+> Document which strategy (or combination) was chosen for this project, and which test
+> categories each strategy applies to.
+
+| Test category | Strategy | Status |
+|---|---|---|
+| Filter option values | Strategy 1 or 2 | _Pending / Implemented_ |
+| Row count assertions | Strategy 2 | _Pending / Implemented_ |
+| Search input values | Strategy 2 or 3 | _Pending / Implemented_ |
+
+> Tests that are blocked on a data strategy can be marked `test.skip()` with a
+> `PENDING-DATA-STRATEGY` annotation until the strategy is implemented.

package/templates/memory/e2e-stabilization-patterns.md

@@ -0,0 +1,54 @@
+# E2E Stabilization Patterns
+
+> A running log of flakiness patterns found in this project and the techniques used to fix them.
+> Load this file when investigating intermittent test failures or refactoring fragile specs.
+
+## Crash-Recovery Pattern
+
+When the application server crashes intermittently (e.g. unhandled error page with a Reload link),
+a bare `waitFor` will timeout. Use `Promise.race` to detect the error state and recover:
+
+```typescript
+const outcome = await Promise.race([
+  targetElement.waitFor({ state: 'visible', timeout: 60_000 }).then(() => 'ready' as const),
+  page.getByText('An unhandled error has occurred.').waitFor({ state: 'visible', timeout: 60_000 }).then(() => 'crash' as const),
+]).catch(() => 'timeout' as const);
+
+if (outcome === 'crash') {
+  await page.getByRole('link', { name: 'Reload' }).click();
+  await targetElement.waitFor({ state: 'visible', timeout: 60_000 });
+}
+```
+
+> Document which spec files use this pattern and under what conditions it fires.
+
+## Strict-Mode Selector Conflicts
+
+`getByText()` and similar matchers can match multiple elements, triggering strict-mode violations.
+Scope selectors to the most specific container available.
+
+> Document conflicts found in this project here.
+> Example: a pagination counter regex that also matches a card element.
+> Fix: use a scoped `locator('.specific-class')` instead of a broad text match.
+
+## Known Flaky Failure Types
+
+Document categories of instability specific to this project. A row per pattern is enough.
+
+| ID | Root cause | Fix | Affected tests |
+|---|---|---|---|
+| _Add ID_ | _Describe the underlying cause_ | _Technique used to fix_ | _Spec files or TC IDs_ |
+
+> Examples: framework hydration delays, input binding issues requiring `pressSequentially`,
+> elements that are in the DOM but not yet rendered, session expiry during long runs.
+
+## Reference Timeouts
+
+Fill in the timeout values used in this project once they are stabilized.
+
+| Setting | Value | Location |
+|---|---|---|
+| Default test timeout | - | `playwright.config.ts` |
+| `actionTimeout` | - | `playwright.config.ts` |
+| `navigationTimeout` | - | `playwright.config.ts` |
+| Long-running spec timeout | - | Per-spec `describe.configure` |

package/templates/qa-framework.instructions.md

@@ -24,7 +24,15 @@ This project uses `@keber/qa-framework` v{{VERSION}} for spec-driven automated t
 1. Generate the output in `UTF-8` encoding without BOM (`UTF-8`, no signature).
 2. Ensure the raw output is `UTF-8` encoded with no `EF BB BF` bytes at the beginning.
 
-{{ADO_SECTION}}
+## Azure DevOps integration
+
+Before any ADO operation, check if `.github/skills/ado-qa/` exists or if
+`integrations.ado.enabled` is `true` in `qa/qa-framework.config.json`. If either
+condition is met, load `.github/skills/qa-ado-integration/SKILL.md` and use it for
+all ADO interactions (work items, test plans, test cases, bugs, etc.).
+
+> To enable Azure DevOps integration run: `npm install github:keber/ado-qa`
+
 ## QA Pipeline
 
 Stages must run in order. Never start a stage unless its prerequisite is met.

package/templates/qa-readme.md

@@ -2,6 +2,7 @@
 
 > Framework: `@keber/qa-framework` v{VERSION}
 > Language: {LANGUAGE} | Base URL: `{QA_BASE_URL}`
+> ADO Project: `{ADO_ORG} / {ADO_PROJECT}` _(optional — remove line if not using Azure DevOps)_
 
 ---
 
@@ -47,12 +48,32 @@ npx playwright test --last-failed
 
 ---
 
+## Last Run
+
+| Suite | Date | Pass | Fail | Skip | CI Link |
+|---|---|---|---|---|---|
+| E2E | — | — | — | — | — |
+| API | — | — | — | — | — |
+
+> Update this table after each significant run. The CI Link can be an ADO TestRun URL, a
+> GitHub Actions run URL, or a path to a local report file.
+
 ## Last Execution
 
 | Date | Suite | Pass | Skip | Fail | Duration | Report |
 |---|---|---|---|---|---|---|
 | — | — | — | — | — | — | — |
 
+## Flaky Tests
+
+> Mark known flaky tests with this inline annotation directly in the table or spec file.
+> Format: `[flaky] <test-id-or-name> - <root cause> - last seen <YYYY-MM-DD>`
+>
+> Example:
+> `[flaky] TC-MOD-012 - timing issue on slow CI agents - last seen 2026-01-15`
+>
+> Remove the annotation once the test has been stable for 2+ consecutive CI runs.
+
 ---
 
 ## Sprint History