@keber/qa-framework 1.10.0 → 1.11.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +63 -18
- package/README.md +1 -1
- package/docs/folder-structure-guide.md +6 -0
- package/integrations/ado-powershell/pipelines/azure-pipeline-qa.yml +92 -25
- package/integrations/playwright/README.md +124 -0
- package/package.json +3 -4
- package/qa-framework.config.json +1 -1
- package/scripts/init.js +53 -38
- package/scripts/upgrade.js +105 -10
- package/templates/automation-scaffold/.env.example +20 -1
- package/templates/automation-scaffold/fixtures/base.ts +108 -0
- package/templates/automation-scaffold/global-setup.ts +119 -29
- package/templates/automation-scaffold/playwright.config.ts +24 -0
- package/templates/defect-report.md +1 -1
- package/templates/integration-scaffold/.env.example +40 -0
- package/templates/integration-scaffold/README.md +77 -0
- package/templates/integration-scaffold/global-setup.ts +99 -0
- package/templates/integration-scaffold/package.json +15 -0
- package/templates/integration-scaffold/playwright.config.ts +78 -0
- package/templates/integration-scaffold/tests/example.spec.ts +43 -0
- package/templates/memory/ci-pipeline-findings.md +36 -0
- package/templates/memory/data-volatility-strategies.md +70 -0
- package/templates/memory/e2e-stabilization-patterns.md +54 -0
- package/templates/qa-framework.instructions.md +9 -1
- package/templates/qa-readme.md +21 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,29 +7,59 @@ Versioning follows [Semantic Versioning](https://semver.org/).
|
|
|
7
7
|
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
## [1.
|
|
10
|
+
## [1.11.0] - 2026-06-24
|
|
11
11
|
|
|
12
12
|
### Added
|
|
13
13
|
|
|
14
|
-
-
|
|
15
|
-
|
|
16
|
-
-
|
|
17
|
-
-
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
-
|
|
23
|
-
|
|
24
|
-
|
|
14
|
+
- **`templates/automation-scaffold/fixtures/base.ts`** — new fixture with TTL-aware session
|
|
15
|
+
refresh. Replaces the plain `@playwright/test` import in long-running suites where the
|
|
16
|
+
server-side session may expire before the suite finishes.
|
|
17
|
+
- **`templates/automation-scaffold/global-setup.ts`** — hardened login flow:
|
|
18
|
+
- `resolveVar()` with source logging (`param` / `env:KEY` / `default`)
|
|
19
|
+
- Login retry loop (up to 3 attempts) with crash-page detection and recovery
|
|
20
|
+
- `resolveAuthUsers()` — derives which users to authenticate from `QA_AUTH_USER` env var
|
|
21
|
+
or from configured `storageState` paths in `playwright.config.ts`
|
|
22
|
+
- `dismissOnboardingFlow()` — configurable via `QA_DISMISS_ONBOARDING_LABELS`
|
|
23
|
+
(comma-separated button labels); no-op by default
|
|
24
|
+
- Support for a second test user (`.auth/user-2.json`) via `QA_USER2_EMAIL` /
|
|
25
|
+
`QA_USER2_PASSWORD`
|
|
26
|
+
- **`templates/automation-scaffold/playwright.config.ts`** — exports `SESSION_TTL_MS`
|
|
27
|
+
(default 2 h, overridable via `QA_SESSION_TTL_MS`); commented-out parallel 2-user variant
|
|
28
|
+
(`chromium-user1` / `chromium-user2`).
|
|
29
|
+
- **`templates/automation-scaffold/.env.example`** — new variables: `QA_SESSION_TTL_MS`,
|
|
30
|
+
`QA_AUTH_USER`, `QA_USER2_EMAIL`, `QA_USER2_PASSWORD`, `QA_DISMISS_ONBOARDING_LABELS`,
|
|
31
|
+
`PLAYWRIGHT_SERVICE_URL`.
|
|
32
|
+
- **`templates/integration-scaffold/`** — new scaffold for API testing with Playwright
|
|
33
|
+
`APIRequestContext`. Includes `package.json`, `playwright.config.ts`, `global-setup.ts`
|
|
34
|
+
(token-based or browser-login auth), `tests/example.spec.ts`, `README.md`, `.env.example`.
|
|
35
|
+
Deployed to `qa/07-automation/integration/` on `init` and `upgrade`.
|
|
36
|
+
- **`templates/memory/`** — three semi-guided operational memory templates:
|
|
37
|
+
`ci-pipeline-findings.md`, `e2e-stabilization-patterns.md`, `data-volatility-strategies.md`.
|
|
38
|
+
Seeded into `qa/memory/` on `init` and `upgrade`.
|
|
39
|
+
- **`qa/06-defects/disputed/`** — third defect state for cases where QA has evidence but the
|
|
40
|
+
business or dev team disputes the expected behavior. Created on `init` and `upgrade`.
|
|
41
|
+
- **`integrations/ado-powershell/pipelines/azure-pipeline-qa.yml`** — operational hardening:
|
|
42
|
+
- `suiteFilter` pipeline parameter (default `all`); auto-overridden to `all` on scheduled runs
|
|
43
|
+
- Bash array (`GREP_ARGS`) for safe grep pattern expansion with bracket characters
|
|
44
|
+
- `PublishTestResults@2` scoped to exact XML path instead of `**/*.xml`
|
|
45
|
+
- `publishRunAttachments: false` on both E2E and API stages
|
|
46
|
+
- Artifact names suffixed with `$(System.JobAttempt)` to prevent collision on retries
|
|
47
|
+
- `QA_AUTH_USER` documented as commented example for the 2-job parallel model
|
|
48
|
+
- **`integrations/playwright/README.md`** — new section: Visual AI helper optional pattern.
|
|
49
|
+
- **`templates/qa-readme.md`** — new fields: `ADO Project`, `Last Run` summary table,
|
|
50
|
+
`Flaky Tests` inline annotation convention.
|
|
25
51
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
- `redacted-repo` (Repo A) — ASP.NET MVC 5 + jQuery + toastr + Sprint-based E2E
|
|
29
|
-
- `redacted-repo` (Repo B) — Blazor WebAssembly + Radzen + full ADO pipeline integration
|
|
52
|
+
### Changed
|
|
30
53
|
|
|
31
|
-
|
|
32
|
-
|
|
54
|
+
- **`scripts/init.js`** — `fixtures/base.ts` added to e2e scaffold copy list; integration
|
|
55
|
+
scaffold replaces the `integration/README.md` placeholder; memory template files seeded
|
|
56
|
+
from `templates/memory/` via `fs.readFileSync`.
|
|
57
|
+
- **`scripts/upgrade.js`** — integration scaffold, `disputed/` defect folder, and memory
|
|
58
|
+
templates are all seeded idempotently on upgrade for existing projects.
|
|
59
|
+
- **`docs/folder-structure-guide.md`** — `06-defects/` updated to document `disputed/`.
|
|
60
|
+
- **`templates/defect-report.md`** — `Status` field extended with `Disputed` value.
|
|
61
|
+
- **`qa/memory/INDEX.md`** template updated to pre-populate rows for the three new memory
|
|
62
|
+
template files.
|
|
33
63
|
|
|
34
64
|
---
|
|
35
65
|
|
|
@@ -300,3 +330,18 @@ See `MIGRATION-NOTES.md` — section "Upgrading from a version that used `.githu
|
|
|
300
330
|
|
|
301
331
|
---
|
|
302
332
|
|
|
333
|
+
## [1.0.0] - 2026-03-04
|
|
334
|
+
|
|
335
|
+
### Added
|
|
336
|
+
|
|
337
|
+
- **Framework scaffold** — initial `qa/` directory tree with 9 numbered folders (00–08)
|
|
338
|
+
- **6-file submodule template set** — `00-inventory` through `05-test-scenarios`
|
|
339
|
+
- **Agent instructions** — 7 purpose-specific Markdown instruction files for IDE agents
|
|
340
|
+
- **Standards** — naming conventions, bug report template, test case template, test data guidelines
|
|
341
|
+
- **Automation scaffold** — generalized Playwright `playwright.config.ts`, `global-setup.ts`, `.env.example`, `fixtures/auth.ts`
|
|
342
|
+
- **Optional integrations** — stubs and READMEs for Playwright, `playwright-azure-reporter`, and ADO PowerShell
|
|
343
|
+
- **CLI entry point** — `qa-framework init`, `generate`, `validate` commands
|
|
344
|
+
- **Documentation** — architecture, comparison matrix, generalization decisions, installation guide, usage-with-agent guide, spec-driven philosophy, folder structure guide
|
|
345
|
+
- **Migration notes** — `MIGRATION-NOTES.md` for projects moving from embedded to package-based framework
|
|
346
|
+
|
|
347
|
+
---
|
package/README.md
CHANGED
|
@@ -260,10 +260,16 @@ Each skill folder contains:
|
|
|
260
260
|
├── README.md
|
|
261
261
|
├── open/
|
|
262
262
|
│ └── DEF-{NNN}-{slug}.md
|
|
263
|
+
├── disputed/
|
|
264
|
+
│ └── DEF-{NNN}-{slug}.md
|
|
263
265
|
└── resolved/
|
|
264
266
|
└── DEF-{NNN}-{slug}.md
|
|
265
267
|
```
|
|
266
268
|
|
|
269
|
+
- `open/` - confirmed defect, fix pending
|
|
270
|
+
- `disputed/` - QA has evidence but business or dev team disputes the expected behavior; pending functional decision
|
|
271
|
+
- `resolved/` - closed by fix or formal decision
|
|
272
|
+
|
|
267
273
|
**Decision guide**:
|
|
268
274
|
- ADO enabled → use ADO Work Items. `06-defects/` contains lightweight references only
|
|
269
275
|
- ADO disabled → use this folder as primary defect tracker
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# -----------------------------------------------------------------------
|
|
2
|
-
# keber/qa-framework
|
|
2
|
+
# keber/qa-framework - Azure Pipeline QA Template
|
|
3
3
|
# -----------------------------------------------------------------------
|
|
4
4
|
# Parameterized CI pipeline for running Playwright E2E and API tests and
|
|
5
5
|
# syncing results to Azure DevOps Test Plans via SYSTEM_ACCESSTOKEN.
|
|
@@ -8,28 +8,28 @@
|
|
|
8
8
|
# Update 'resources.repositories.name' to match your org/project/repo.
|
|
9
9
|
#
|
|
10
10
|
# Required Pipeline Library Variable Group: "QA-Variables"
|
|
11
|
-
# Create this group in ADO
|
|
11
|
+
# Create this group in ADO -> Pipelines -> Library and add the following:
|
|
12
12
|
#
|
|
13
|
-
# QA_BASE_URL
|
|
14
|
-
# QA_USER_EMAIL
|
|
15
|
-
# QA_USER_PASSWORD
|
|
16
|
-
# ADO_ORG
|
|
17
|
-
# ADO_PROJECT
|
|
18
|
-
# ADO_E2E_PLAN_ID
|
|
19
|
-
# ADO_API_PLAN_ID
|
|
13
|
+
# QA_BASE_URL - Target application URL (e.g. https://app-qa.example.com)
|
|
14
|
+
# QA_USER_EMAIL - Test user email / RUT
|
|
15
|
+
# QA_USER_PASSWORD - Test user password [secret]
|
|
16
|
+
# ADO_ORG - ADO organization slug (e.g. my-org)
|
|
17
|
+
# ADO_PROJECT - ADO project name (e.g. my-project)
|
|
18
|
+
# ADO_E2E_PLAN_ID - Test Plan ID for the E2E suite
|
|
19
|
+
# ADO_API_PLAN_ID - Test Plan ID for the API/integration suite
|
|
20
20
|
#
|
|
21
|
-
# NOTE on multiple Plan IDs: ADO variable groups store plain strings
|
|
21
|
+
# NOTE on multiple Plan IDs: ADO variable groups store plain strings - there
|
|
22
22
|
# is no array type. Use one variable per suite (ADO_E2E_PLAN_ID / ADO_API_PLAN_ID).
|
|
23
23
|
# If the project has only one suite, a single ADO_PLAN_ID variable is enough.
|
|
24
24
|
# Results are reported to ADO via System.AccessToken (OAuth); no PAT required.
|
|
25
25
|
# -----------------------------------------------------------------------
|
|
26
26
|
|
|
27
|
-
# Cross-project repository
|
|
27
|
+
# Cross-project repository - update <ADO_SOURCE_PROJECT>/<REPO_NAME> as needed
|
|
28
28
|
resources:
|
|
29
29
|
repositories:
|
|
30
30
|
- repository: source
|
|
31
31
|
type: git
|
|
32
|
-
name: <ADO_SOURCE_PROJECT>/<REPO_NAME> # e.g.
|
|
32
|
+
name: <ADO_SOURCE_PROJECT>/<REPO_NAME> # e.g. my-qa-project/my-repo
|
|
33
33
|
ref: refs/heads/main
|
|
34
34
|
|
|
35
35
|
# Trigger: push and PR to main.
|
|
@@ -44,15 +44,34 @@ pr:
|
|
|
44
44
|
include:
|
|
45
45
|
- main
|
|
46
46
|
|
|
47
|
-
# Optional scheduled run
|
|
47
|
+
# Optional scheduled run - uncomment and adjust if a nightly regression is needed:
|
|
48
48
|
# schedules:
|
|
49
|
-
# - cron: '0 6 * * 1-5' # 06:00 UTC Mon
|
|
49
|
+
# - cron: '0 6 * * 1-5' # 06:00 UTC Mon-Fri (03:00 CLT / 02:00 CLST)
|
|
50
50
|
# displayName: Nightly regression
|
|
51
51
|
# branches:
|
|
52
52
|
# include:
|
|
53
53
|
# - main
|
|
54
54
|
# always: true
|
|
55
55
|
|
|
56
|
+
# -----------------------------------------------------------------------
|
|
57
|
+
# Pipeline parameters
|
|
58
|
+
# -----------------------------------------------------------------------
|
|
59
|
+
# suiteFilter controls which tests run when the pipeline is triggered manually
|
|
60
|
+
# or from a PR. Scheduled runs always override this to 'all' automatically
|
|
61
|
+
# (see the "Resolve suite filter" step in each test job).
|
|
62
|
+
#
|
|
63
|
+
# Default is 'all' so a plain manual run executes the full suite.
|
|
64
|
+
# Override at queue time to run a subset (e.g. 'smoke', '[P0]', 'login').
|
|
65
|
+
#
|
|
66
|
+
# The value is passed to Playwright via --grep. Special regex characters such
|
|
67
|
+
# as '[' and ']' are escaped by the bash array pattern in the run step, so
|
|
68
|
+
# values like '[P0]' are safe.
|
|
69
|
+
parameters:
|
|
70
|
+
- name: suiteFilter
|
|
71
|
+
displayName: 'Test suite filter (grep pattern, or "all" to skip filtering)'
|
|
72
|
+
type: string
|
|
73
|
+
default: 'all'
|
|
74
|
+
|
|
56
75
|
pool:
|
|
57
76
|
vmImage: 'ubuntu-latest'
|
|
58
77
|
|
|
@@ -111,6 +130,21 @@ stages:
|
|
|
111
130
|
# -----------------------------------------------------------------------
|
|
112
131
|
# Stage 2: E2E Tests (Playwright/Chromium)
|
|
113
132
|
# -----------------------------------------------------------------------
|
|
133
|
+
# Single-job configuration. For parallel execution with two test users,
|
|
134
|
+
# split this into two jobs and set QA_AUTH_USER per job:
|
|
135
|
+
#
|
|
136
|
+
# - job: E2E_User1
|
|
137
|
+
# variables:
|
|
138
|
+
# QA_AUTH_USER: '1'
|
|
139
|
+
# ...
|
|
140
|
+
# - job: E2E_User2
|
|
141
|
+
# variables:
|
|
142
|
+
# QA_AUTH_USER: '2'
|
|
143
|
+
# ...
|
|
144
|
+
#
|
|
145
|
+
# Each job must use a distinct Playwright project (--project=chromium-user1 /
|
|
146
|
+
# --project=chromium-user2) to avoid session conflicts between parallel users.
|
|
147
|
+
# -----------------------------------------------------------------------
|
|
114
148
|
- stage: E2E
|
|
115
149
|
displayName: 'E2E Tests (Playwright/Chromium)'
|
|
116
150
|
dependsOn: Setup
|
|
@@ -138,11 +172,11 @@ stages:
|
|
|
138
172
|
HTTP_CODE=$(curl -o /dev/null -s -w "%{http_code}" --max-time 30 "$(QA_BASE_URL)")
|
|
139
173
|
echo "HTTP response: $HTTP_CODE"
|
|
140
174
|
if [ "$HTTP_CODE" -lt 200 ] || [ "$HTTP_CODE" -ge 400 ]; then
|
|
141
|
-
echo "##[error]Smoke test failed
|
|
175
|
+
echo "##[error]Smoke test failed - $(QA_BASE_URL) returned HTTP $HTTP_CODE"
|
|
142
176
|
exit 1
|
|
143
177
|
fi
|
|
144
178
|
echo "Smoke test passed (HTTP $HTTP_CODE)"
|
|
145
|
-
displayName: 'Smoke test
|
|
179
|
+
displayName: 'Smoke test - verify QA_BASE_URL is reachable'
|
|
146
180
|
|
|
147
181
|
# ---------------------------------------------------------------
|
|
148
182
|
# Playwright Workspaces (Azure App Testing)
|
|
@@ -159,9 +193,28 @@ stages:
|
|
|
159
193
|
# inlineScript: echo "Authenticated via Entra ID"
|
|
160
194
|
|
|
161
195
|
- script: |
|
|
196
|
+
# Scheduled runs always execute the full suite, regardless of the
|
|
197
|
+
# suiteFilter parameter. This prevents accidental partial runs in
|
|
198
|
+
# nightly regressions when the parameter was changed by a prior
|
|
199
|
+
# manual queue.
|
|
200
|
+
SUITE_FILTER="${{ parameters.suiteFilter }}"
|
|
201
|
+
if [ "$(Build.Reason)" = "Schedule" ]; then
|
|
202
|
+
SUITE_FILTER="all"
|
|
203
|
+
echo "Scheduled run detected - overriding suiteFilter to 'all'"
|
|
204
|
+
fi
|
|
205
|
+
echo "Effective suiteFilter: $SUITE_FILTER"
|
|
206
|
+
|
|
207
|
+
# Build --grep args as a bash array to safely handle patterns that
|
|
208
|
+
# contain special regex characters such as '[' and ']'. Inline
|
|
209
|
+
# string interpolation would break those patterns at the shell level.
|
|
210
|
+
GREP_ARGS=()
|
|
211
|
+
if [ "$SUITE_FILTER" != "all" ]; then
|
|
212
|
+
GREP_ARGS=(--grep "$SUITE_FILTER")
|
|
213
|
+
fi
|
|
214
|
+
|
|
162
215
|
cd $(E2E_DIR)
|
|
163
216
|
npx playwright test --project=chromium --ignore-snapshots \
|
|
164
|
-
--grep-invert "_inspect|_diag"
|
|
217
|
+
--grep-invert "_inspect|_diag" "${GREP_ARGS[@]}"
|
|
165
218
|
displayName: 'Run E2E tests'
|
|
166
219
|
continueOnError: true
|
|
167
220
|
env:
|
|
@@ -173,31 +226,40 @@ stages:
|
|
|
173
226
|
ADO_PLAN_ID: $(ADO_E2E_PLAN_ID)
|
|
174
227
|
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
|
|
175
228
|
CI: 'true'
|
|
176
|
-
#
|
|
229
|
+
# QA_AUTH_USER: '1' # Set explicitly when running parallel jobs (user-1 / user-2 model)
|
|
230
|
+
# PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint - enable together with the login task
|
|
177
231
|
|
|
232
|
+
# publishRunAttachments is false because screenshots and traces are
|
|
233
|
+
# already uploaded as pipeline artifacts below. Setting it to true
|
|
234
|
+
# would double-publish the same files.
|
|
178
235
|
- task: PublishTestResults@2
|
|
179
236
|
condition: always()
|
|
180
237
|
inputs:
|
|
181
238
|
testResultsFormat: 'JUnit'
|
|
182
|
-
testResultsFiles: '
|
|
239
|
+
testResultsFiles: 'test-results/junit-results.xml'
|
|
183
240
|
searchFolder: $(E2E_DIR)
|
|
184
241
|
mergeTestResults: true
|
|
242
|
+
publishRunAttachments: false
|
|
185
243
|
failTaskOnFailedTests: true
|
|
186
244
|
testRunTitle: '[Auto E2E] $(Build.BuildNumber)'
|
|
187
245
|
displayName: 'Publish E2E test results (JUnit)'
|
|
188
246
|
|
|
247
|
+
# Artifact names include $(System.JobAttempt) to avoid name collisions
|
|
248
|
+
# when the job is retried. ADO rejects duplicate artifact names within
|
|
249
|
+
# the same pipeline run, so without the suffix a retry always fails at
|
|
250
|
+
# the publish step.
|
|
189
251
|
- task: PublishPipelineArtifact@1
|
|
190
252
|
condition: always()
|
|
191
253
|
inputs:
|
|
192
254
|
targetPath: '$(E2E_DIR)/playwright-report'
|
|
193
|
-
artifact: 'e2e-playwright-report'
|
|
255
|
+
artifact: 'e2e-playwright-report-attempt-$(System.JobAttempt)'
|
|
194
256
|
displayName: 'Publish E2E HTML report'
|
|
195
257
|
|
|
196
258
|
- task: PublishPipelineArtifact@1
|
|
197
259
|
condition: always()
|
|
198
260
|
inputs:
|
|
199
261
|
targetPath: '$(E2E_DIR)/test-results'
|
|
200
|
-
artifact: 'e2e-test-results'
|
|
262
|
+
artifact: 'e2e-test-results-attempt-$(System.JobAttempt)'
|
|
201
263
|
displayName: 'Publish E2E trace artifacts'
|
|
202
264
|
|
|
203
265
|
# -----------------------------------------------------------------------
|
|
@@ -237,6 +299,7 @@ stages:
|
|
|
237
299
|
# inlineScript: echo "Authenticated via Entra ID"
|
|
238
300
|
|
|
239
301
|
- script: |
|
|
302
|
+
# API integration tests do not use suite tags - always run all.
|
|
240
303
|
cd $(API_DIR)
|
|
241
304
|
npx playwright test --grep-invert "_inspect|_diag"
|
|
242
305
|
displayName: 'Run API integration tests'
|
|
@@ -250,29 +313,33 @@ stages:
|
|
|
250
313
|
ADO_PLAN_ID: $(ADO_API_PLAN_ID)
|
|
251
314
|
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
|
|
252
315
|
CI: 'true'
|
|
253
|
-
# PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint
|
|
316
|
+
# PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint - enable together with the login task
|
|
254
317
|
|
|
318
|
+
# publishRunAttachments is false because trace artifacts are already
|
|
319
|
+
# uploaded as pipeline artifacts below.
|
|
255
320
|
- task: PublishTestResults@2
|
|
256
321
|
condition: always()
|
|
257
322
|
inputs:
|
|
258
323
|
testResultsFormat: 'JUnit'
|
|
259
|
-
testResultsFiles: '
|
|
324
|
+
testResultsFiles: 'test-results/junit-results.xml'
|
|
260
325
|
searchFolder: $(API_DIR)
|
|
261
326
|
mergeTestResults: true
|
|
327
|
+
publishRunAttachments: false
|
|
262
328
|
failTaskOnFailedTests: true
|
|
263
329
|
testRunTitle: '[Auto API] $(Build.BuildNumber)'
|
|
264
330
|
displayName: 'Publish API test results (JUnit)'
|
|
265
331
|
|
|
332
|
+
# Artifact names include $(System.JobAttempt) to prevent collision on retries.
|
|
266
333
|
- task: PublishPipelineArtifact@1
|
|
267
334
|
condition: always()
|
|
268
335
|
inputs:
|
|
269
336
|
targetPath: '$(API_DIR)/playwright-report'
|
|
270
|
-
artifact: 'api-playwright-report'
|
|
337
|
+
artifact: 'api-playwright-report-attempt-$(System.JobAttempt)'
|
|
271
338
|
displayName: 'Publish API HTML report'
|
|
272
339
|
|
|
273
340
|
- task: PublishPipelineArtifact@1
|
|
274
341
|
condition: always()
|
|
275
342
|
inputs:
|
|
276
343
|
targetPath: '$(API_DIR)/test-results'
|
|
277
|
-
artifact: 'api-test-results'
|
|
344
|
+
artifact: 'api-test-results-attempt-$(System.JobAttempt)'
|
|
278
345
|
displayName: 'Publish API trace artifacts'
|
|
@@ -66,3 +66,127 @@ test.describe('[TC-M-S-NNN] Module > Submodule @P0', () => {
|
|
|
66
66
|
});
|
|
67
67
|
});
|
|
68
68
|
```
|
|
69
|
+
|
|
70
|
+
## Visual AI helper (optional advanced pattern)
|
|
71
|
+
|
|
72
|
+
> **Advanced / optional.** This pattern is not part of the default scaffold and requires
|
|
73
|
+
> an AI vision API key. Use it when a selector-based assertion is too brittle or the
|
|
74
|
+
> element structure changes frequently (e.g. PDF previews, dashboard charts, export outputs).
|
|
75
|
+
|
|
76
|
+
Instead of asserting specific selectors, take a screenshot and ask an AI vision model to
|
|
77
|
+
verify that the expected content is visually present. This decouples the assertion from DOM
|
|
78
|
+
structure and is useful for generated documents or complex layouts.
|
|
79
|
+
|
|
80
|
+
```typescript
|
|
81
|
+
// fixtures/vision-helper.ts
|
|
82
|
+
import * as https from 'https';
|
|
83
|
+
|
|
84
|
+
export interface VisionResult {
|
|
85
|
+
passes: boolean;
|
|
86
|
+
reason: string;
|
|
87
|
+
raw?: string;
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
/**
|
|
91
|
+
* Sends a screenshot to an AI vision model and asks whether the described
|
|
92
|
+
* content is visually present. Degrades gracefully when the API key is absent.
|
|
93
|
+
*
|
|
94
|
+
* Requires an API key in the environment (e.g. OPENAI_API_KEY for GPT-4o,
|
|
95
|
+
* or adapt the request body for any other vision-capable model).
|
|
96
|
+
*/
|
|
97
|
+
export async function assertScreenshotShowsContent(
|
|
98
|
+
screenshotBuffer: Buffer,
|
|
99
|
+
contentDescription: string,
|
|
100
|
+
options: { model?: string; degradeOnMissingKey?: boolean } = {}
|
|
101
|
+
): Promise<VisionResult> {
|
|
102
|
+
const apiKey = process.env.AI_VISION_API_KEY;
|
|
103
|
+
const degrade = options.degradeOnMissingKey ?? true;
|
|
104
|
+
|
|
105
|
+
if (!apiKey) {
|
|
106
|
+
if (degrade) {
|
|
107
|
+
console.warn('[vision-helper] AI_VISION_API_KEY not set. Skipping visual assertion.');
|
|
108
|
+
return { passes: true, reason: 'SKIPPED: AI_VISION_API_KEY not configured' };
|
|
109
|
+
}
|
|
110
|
+
return { passes: false, reason: 'AI_VISION_API_KEY not set in environment' };
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
const base64Image = screenshotBuffer.toString('base64');
|
|
114
|
+
const model = options.model ?? 'gpt-4o';
|
|
115
|
+
|
|
116
|
+
const prompt = `You are a QA assistant verifying screenshots of a web application.
|
|
117
|
+
Answer YES or NO: Does the screenshot clearly show ${contentDescription}?
|
|
118
|
+
Reply in exactly this format:
|
|
119
|
+
VERDICT: YES
|
|
120
|
+
REASON: <one short sentence>`;
|
|
121
|
+
|
|
122
|
+
const body = JSON.stringify({
|
|
123
|
+
model,
|
|
124
|
+
max_tokens: 150,
|
|
125
|
+
messages: [{
|
|
126
|
+
role: 'user',
|
|
127
|
+
content: [
|
|
128
|
+
{ type: 'text', text: prompt },
|
|
129
|
+
{ type: 'image_url', image_url: { url: `data:image/png;base64,${base64Image}`, detail: 'low' } },
|
|
130
|
+
],
|
|
131
|
+
}],
|
|
132
|
+
});
|
|
133
|
+
|
|
134
|
+
const raw = await new Promise<string>((resolve, reject) => {
|
|
135
|
+
const req = https.request({
|
|
136
|
+
hostname: 'api.openai.com',
|
|
137
|
+
path: '/v1/chat/completions',
|
|
138
|
+
method: 'POST',
|
|
139
|
+
headers: {
|
|
140
|
+
'Content-Type': 'application/json',
|
|
141
|
+
'Authorization': `Bearer ${apiKey}`,
|
|
142
|
+
'Content-Length': Buffer.byteLength(body),
|
|
143
|
+
},
|
|
144
|
+
}, (res) => {
|
|
145
|
+
let data = '';
|
|
146
|
+
res.on('data', (chunk) => { data += chunk; });
|
|
147
|
+
res.on('end', () => resolve(data));
|
|
148
|
+
});
|
|
149
|
+
req.on('error', reject);
|
|
150
|
+
req.write(body);
|
|
151
|
+
req.end();
|
|
152
|
+
});
|
|
153
|
+
|
|
154
|
+
const parsed = JSON.parse(raw);
|
|
155
|
+
const text: string = parsed.choices?.[0]?.message?.content ?? '';
|
|
156
|
+
const verdict = text.match(/VERDICT:\s*(YES|NO)/i);
|
|
157
|
+
const reason = text.match(/REASON:\s*(.+)/i);
|
|
158
|
+
|
|
159
|
+
if (!verdict) return { passes: false, reason: `Unexpected response: ${text}`, raw: text };
|
|
160
|
+
return {
|
|
161
|
+
passes: verdict[1].toUpperCase() === 'YES',
|
|
162
|
+
reason: reason?.[1]?.trim() ?? text,
|
|
163
|
+
raw: text,
|
|
164
|
+
};
|
|
165
|
+
}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
**Usage in a test:**
|
|
169
|
+
|
|
170
|
+
```typescript
|
|
171
|
+
import { assertScreenshotShowsContent } from '../fixtures/vision-helper';
|
|
172
|
+
|
|
173
|
+
test('[TC-MOD-001] Export PDF contains expected content', async ({ page }) => {
|
|
174
|
+
// ... trigger export ...
|
|
175
|
+
const screenshot = await page.screenshot();
|
|
176
|
+
const result = await assertScreenshotShowsContent(
|
|
177
|
+
screenshot,
|
|
178
|
+
'a completed export confirmation message with a download link'
|
|
179
|
+
);
|
|
180
|
+
expect(result.passes, result.reason).toBe(true);
|
|
181
|
+
});
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
**Setup notes:**
|
|
185
|
+
|
|
186
|
+
- Copy `vision-helper.ts` into your `fixtures/` directory.
|
|
187
|
+
- Set `AI_VISION_API_KEY` in your `.env` file (not committed). The example uses the OpenAI
|
|
188
|
+
API; adapt the request body for any other vision model.
|
|
189
|
+
- By default, the helper degrades to a skip (passes: true) when the key is absent, so CI
|
|
190
|
+
runs without the key do not fail on this assertion.
|
|
191
|
+
- `pdf-parse` (v2+) exports a `PDFParse` class rather than a function. Use
|
|
192
|
+
`new PDFParse({ data: buffer })` and call `.getText()` on the instance.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@keber/qa-framework",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.11.3",
|
|
4
4
|
"description": "Reusable spec-driven QA framework for IDE-agent-assisted automated testing. Installable as an npm package. Provides structure, templates, agent instructions, and optional integrations for Playwright and Azure DevOps.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"qa",
|
|
@@ -38,10 +38,9 @@
|
|
|
38
38
|
"engines": {
|
|
39
39
|
"node": ">=18"
|
|
40
40
|
},
|
|
41
|
-
"devDependencies": {},
|
|
42
41
|
"peerDependencies": {
|
|
43
|
-
"@
|
|
44
|
-
"@
|
|
42
|
+
"@keber/ado-qa": ">=1.0.0",
|
|
43
|
+
"@playwright/test": ">=1.40.0"
|
|
45
44
|
},
|
|
46
45
|
"peerDependenciesMeta": {
|
|
47
46
|
"@playwright/test": {
|