@keber/qa-framework 1.10.0 → 1.11.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,29 +7,59 @@ Versioning follows [Semantic Versioning](https://semver.org/).
7
7
 
8
8
  ---
9
9
 
10
- ## [1.0.0] - 2026-03-04
10
+ ## [1.11.0] - 2026-06-24
11
11
 
12
12
  ### Added
13
13
 
14
- - **Framework scaffold** — initial `qa/` directory tree with 9 numbered folders (00–08)
15
- - **6-file submodule template set** — `00-inventory` through `05-test-scenarios`
16
- - **Agent instructions** — 7 purpose-specific Markdown instruction files for IDE agents
17
- - **Standards** — naming conventions, bug report template, test case template, test data guidelines
18
- - **Automation scaffold** — generalized Playwright `playwright.config.ts`, `global-setup.ts`, `.env.example`, `fixtures/auth.ts`
19
- - **Optional integrations** — stubs and READMEs for Playwright, `playwright-azure-reporter`, and ADO PowerShell
20
- - **CLI entry point** — `qa-framework init`, `generate`, `validate` commands
21
- - **Documentation** — architecture, comparison matrix, generalization decisions, installation guide, usage-with-agent guide, spec-driven philosophy, folder structure guide
22
- - **Migration notes** — `MIGRATION-NOTES.md` for projects moving from embedded to package-based framework
23
-
24
- ### Source repositories
14
+ - **`templates/automation-scaffold/fixtures/base.ts`** — new fixture with TTL-aware session
15
+ refresh. Replaces the plain `@playwright/test` import in long-running suites where the
16
+ server-side session may expire before the suite finishes.
17
+ - **`templates/automation-scaffold/global-setup.ts`** — hardened login flow:
18
+ - `resolveVar()` with source logging (`param` / `env:KEY` / `default`)
19
+ - Login retry loop (up to 3 attempts) with crash-page detection and recovery
20
+ - `resolveAuthUsers()` — derives which users to authenticate from `QA_AUTH_USER` env var
21
+ or from configured `storageState` paths in `playwright.config.ts`
22
+ - `dismissOnboardingFlow()` — configurable via `QA_DISMISS_ONBOARDING_LABELS`
23
+ (comma-separated button labels); no-op by default
24
+ - Support for a second test user (`.auth/user-2.json`) via `QA_USER2_EMAIL` /
25
+ `QA_USER2_PASSWORD`
26
+ - **`templates/automation-scaffold/playwright.config.ts`** — exports `SESSION_TTL_MS`
27
+ (default 2 h, overridable via `QA_SESSION_TTL_MS`); commented-out parallel 2-user variant
28
+ (`chromium-user1` / `chromium-user2`).
29
+ - **`templates/automation-scaffold/.env.example`** — new variables: `QA_SESSION_TTL_MS`,
30
+ `QA_AUTH_USER`, `QA_USER2_EMAIL`, `QA_USER2_PASSWORD`, `QA_DISMISS_ONBOARDING_LABELS`,
31
+ `PLAYWRIGHT_SERVICE_URL`.
32
+ - **`templates/integration-scaffold/`** — new scaffold for API testing with Playwright
33
+ `APIRequestContext`. Includes `package.json`, `playwright.config.ts`, `global-setup.ts`
34
+ (token-based or browser-login auth), `tests/example.spec.ts`, `README.md`, `.env.example`.
35
+ Deployed to `qa/07-automation/integration/` on `init` and `upgrade`.
36
+ - **`templates/memory/`** — three semi-guided operational memory templates:
37
+ `ci-pipeline-findings.md`, `e2e-stabilization-patterns.md`, `data-volatility-strategies.md`.
38
+ Seeded into `qa/memory/` on `init` and `upgrade`.
39
+ - **`qa/06-defects/disputed/`** — third defect state for cases where QA has evidence but the
40
+ business or dev team disputes the expected behavior. Created on `init` and `upgrade`.
41
+ - **`integrations/ado-powershell/pipelines/azure-pipeline-qa.yml`** — operational hardening:
42
+ - `suiteFilter` pipeline parameter (default `all`); auto-overridden to `all` on scheduled runs
43
+ - Bash array (`GREP_ARGS`) for safe grep pattern expansion with bracket characters
44
+ - `PublishTestResults@2` scoped to exact XML path instead of `**/*.xml`
45
+ - `publishRunAttachments: false` on both E2E and API stages
46
+ - Artifact names suffixed with `$(System.JobAttempt)` to prevent collision on retries
47
+ - `QA_AUTH_USER` documented as commented example for the 2-job parallel model
48
+ - **`integrations/playwright/README.md`** — new section: Visual AI helper optional pattern.
49
+ - **`templates/qa-readme.md`** — new fields: `ADO Project`, `Last Run` summary table,
50
+ `Flaky Tests` inline annotation convention.
25
51
 
26
- This version was bootstrapped from analysis of two existing embedded QA implementations:
27
-
28
- - `redacted-repo` (Repo A) — ASP.NET MVC 5 + jQuery + toastr + Sprint-based E2E
29
- - `redacted-repo` (Repo B) — Blazor WebAssembly + Radzen + full ADO pipeline integration
52
+ ### Changed
30
53
 
31
- All project-specific content (URLs, ADO IDs, user credentials, module names, form selectors)
32
- has been removed or parameterized. See `docs/comparison-matrix.md` and `docs/generalization-decisions.md`.
54
+ - **`scripts/init.js`** — `fixtures/base.ts` added to e2e scaffold copy list; integration
55
+ scaffold replaces the `integration/README.md` placeholder; memory template files seeded
56
+ from `templates/memory/` via `fs.readFileSync`.
57
+ - **`scripts/upgrade.js`** — integration scaffold, `disputed/` defect folder, and memory
58
+ templates are all seeded idempotently on upgrade for existing projects.
59
+ - **`docs/folder-structure-guide.md`** — `06-defects/` updated to document `disputed/`.
60
+ - **`templates/defect-report.md`** — `Status` field extended with `Disputed` value.
61
+ - **`qa/memory/INDEX.md`** template updated to pre-populate rows for the three new memory
62
+ template files.
33
63
 
34
64
  ---
35
65
 
@@ -300,3 +330,18 @@ See `MIGRATION-NOTES.md` — section "Upgrading from a version that used `.githu
300
330
 
301
331
  ---
302
332
 
333
+ ## [1.0.0] - 2026-03-04
334
+
335
+ ### Added
336
+
337
+ - **Framework scaffold** — initial `qa/` directory tree with 9 numbered folders (00–08)
338
+ - **6-file submodule template set** — `00-inventory` through `05-test-scenarios`
339
+ - **Agent instructions** — 7 purpose-specific Markdown instruction files for IDE agents
340
+ - **Standards** — naming conventions, bug report template, test case template, test data guidelines
341
+ - **Automation scaffold** — generalized Playwright `playwright.config.ts`, `global-setup.ts`, `.env.example`, `fixtures/auth.ts`
342
+ - **Optional integrations** — stubs and READMEs for Playwright, `playwright-azure-reporter`, and ADO PowerShell
343
+ - **CLI entry point** — `qa-framework init`, `generate`, `validate` commands
344
+ - **Documentation** — architecture, comparison matrix, generalization decisions, installation guide, usage-with-agent guide, spec-driven philosophy, folder structure guide
345
+ - **Migration notes** — `MIGRATION-NOTES.md` for projects moving from embedded to package-based framework
346
+
347
+ ---
package/README.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # qa-framework
2
2
 
3
3
  > Reusable, installable, agent-oriented QA framework for spec-driven automated testing.
4
- > Current version: **v1.9.1**
4
+ > Current version: **v1.10.0**
5
5
 
6
6
  ---
7
7
 
@@ -260,10 +260,16 @@ Each skill folder contains:
260
260
  ├── README.md
261
261
  ├── open/
262
262
  │ └── DEF-{NNN}-{slug}.md
263
+ ├── disputed/
264
+ │ └── DEF-{NNN}-{slug}.md
263
265
  └── resolved/
264
266
  └── DEF-{NNN}-{slug}.md
265
267
  ```
266
268
 
269
+ - `open/` - confirmed defect, fix pending
270
+ - `disputed/` - QA has evidence but business or dev team disputes the expected behavior; pending functional decision
271
+ - `resolved/` - closed by fix or formal decision
272
+
267
273
  **Decision guide**:
268
274
  - ADO enabled → use ADO Work Items. `06-defects/` contains lightweight references only
269
275
  - ADO disabled → use this folder as primary defect tracker
@@ -1,5 +1,5 @@
1
1
  # -----------------------------------------------------------------------
2
- # keber/qa-framework — Azure Pipeline QA Template
2
+ # keber/qa-framework - Azure Pipeline QA Template
3
3
  # -----------------------------------------------------------------------
4
4
  # Parameterized CI pipeline for running Playwright E2E and API tests and
5
5
  # syncing results to Azure DevOps Test Plans via SYSTEM_ACCESSTOKEN.
@@ -8,28 +8,28 @@
8
8
  # Update 'resources.repositories.name' to match your org/project/repo.
9
9
  #
10
10
  # Required Pipeline Library Variable Group: "QA-Variables"
11
- # Create this group in ADO → Pipelines → Library and add the following:
11
+ # Create this group in ADO -> Pipelines -> Library and add the following:
12
12
  #
13
- # QA_BASE_URL — Target application URL (e.g. https://app-qa.example.com)
14
- # QA_USER_EMAIL — Test user email / RUT
15
- # QA_USER_PASSWORD — Test user password [secret]
16
- # ADO_ORG — ADO organization slug (e.g. garcesfruit-desarrollo)
17
- # ADO_PROJECT — ADO project name (e.g. gf-portalproveedores)
18
- # ADO_E2E_PLAN_ID — Test Plan ID for the E2E suite (e.g. 27044)
19
- # ADO_API_PLAN_ID — Test Plan ID for the API/integration suite (e.g. 27046)
13
+ # QA_BASE_URL - Target application URL (e.g. https://app-qa.example.com)
14
+ # QA_USER_EMAIL - Test user email / RUT
15
+ # QA_USER_PASSWORD - Test user password [secret]
16
+ # ADO_ORG - ADO organization slug (e.g. my-org)
17
+ # ADO_PROJECT - ADO project name (e.g. my-project)
18
+ # ADO_E2E_PLAN_ID - Test Plan ID for the E2E suite
19
+ # ADO_API_PLAN_ID - Test Plan ID for the API/integration suite
20
20
  #
21
- # NOTE on multiple Plan IDs: ADO variable groups store plain strings — there
21
+ # NOTE on multiple Plan IDs: ADO variable groups store plain strings - there
22
22
  # is no array type. Use one variable per suite (ADO_E2E_PLAN_ID / ADO_API_PLAN_ID).
23
23
  # If the project has only one suite, a single ADO_PLAN_ID variable is enough.
24
24
  # Results are reported to ADO via System.AccessToken (OAuth); no PAT required.
25
25
  # -----------------------------------------------------------------------
26
26
 
27
- # Cross-project repository — update <ADO_SOURCE_PROJECT>/<REPO_NAME> as needed
27
+ # Cross-project repository - update <ADO_SOURCE_PROJECT>/<REPO_NAME> as needed
28
28
  resources:
29
29
  repositories:
30
30
  - repository: source
31
31
  type: git
32
- name: <ADO_SOURCE_PROJECT>/<REPO_NAME> # e.g. gf-gestion-qa/gf-portalproveedores
32
+ name: <ADO_SOURCE_PROJECT>/<REPO_NAME> # e.g. my-qa-project/my-repo
33
33
  ref: refs/heads/main
34
34
 
35
35
  # Trigger: push and PR to main.
@@ -44,15 +44,34 @@ pr:
44
44
  include:
45
45
  - main
46
46
 
47
- # Optional scheduled run — uncomment and adjust if a nightly regression is needed:
47
+ # Optional scheduled run - uncomment and adjust if a nightly regression is needed:
48
48
  # schedules:
49
- # - cron: '0 6 * * 1-5' # 06:00 UTC Mon–Fri (03:00 CLT / 02:00 CLST)
49
+ # - cron: '0 6 * * 1-5' # 06:00 UTC Mon-Fri (03:00 CLT / 02:00 CLST)
50
50
  # displayName: Nightly regression
51
51
  # branches:
52
52
  # include:
53
53
  # - main
54
54
  # always: true
55
55
 
56
+ # -----------------------------------------------------------------------
57
+ # Pipeline parameters
58
+ # -----------------------------------------------------------------------
59
+ # suiteFilter controls which tests run when the pipeline is triggered manually
60
+ # or from a PR. Scheduled runs always override this to 'all' automatically
61
+ # (see the "Resolve suite filter" step in each test job).
62
+ #
63
+ # Default is 'all' so a plain manual run executes the full suite.
64
+ # Override at queue time to run a subset (e.g. 'smoke', '[P0]', 'login').
65
+ #
66
+ # The value is passed to Playwright via --grep. Special regex characters such
67
+ # as '[' and ']' are escaped by the bash array pattern in the run step, so
68
+ # values like '[P0]' are safe.
69
+ parameters:
70
+ - name: suiteFilter
71
+ displayName: 'Test suite filter (grep pattern, or "all" to skip filtering)'
72
+ type: string
73
+ default: 'all'
74
+
56
75
  pool:
57
76
  vmImage: 'ubuntu-latest'
58
77
 
@@ -111,6 +130,21 @@ stages:
111
130
  # -----------------------------------------------------------------------
112
131
  # Stage 2: E2E Tests (Playwright/Chromium)
113
132
  # -----------------------------------------------------------------------
133
+ # Single-job configuration. For parallel execution with two test users,
134
+ # split this into two jobs and set QA_AUTH_USER per job:
135
+ #
136
+ # - job: E2E_User1
137
+ # variables:
138
+ # QA_AUTH_USER: '1'
139
+ # ...
140
+ # - job: E2E_User2
141
+ # variables:
142
+ # QA_AUTH_USER: '2'
143
+ # ...
144
+ #
145
+ # Each job must use a distinct Playwright project (--project=chromium-user1 /
146
+ # --project=chromium-user2) to avoid session conflicts between parallel users.
147
+ # -----------------------------------------------------------------------
114
148
  - stage: E2E
115
149
  displayName: 'E2E Tests (Playwright/Chromium)'
116
150
  dependsOn: Setup
@@ -138,11 +172,11 @@ stages:
138
172
  HTTP_CODE=$(curl -o /dev/null -s -w "%{http_code}" --max-time 30 "$(QA_BASE_URL)")
139
173
  echo "HTTP response: $HTTP_CODE"
140
174
  if [ "$HTTP_CODE" -lt 200 ] || [ "$HTTP_CODE" -ge 400 ]; then
141
- echo "##[error]Smoke test failed — $(QA_BASE_URL) returned HTTP $HTTP_CODE"
175
+ echo "##[error]Smoke test failed - $(QA_BASE_URL) returned HTTP $HTTP_CODE"
142
176
  exit 1
143
177
  fi
144
178
  echo "Smoke test passed (HTTP $HTTP_CODE)"
145
- displayName: 'Smoke test — verify QA_BASE_URL is reachable'
179
+ displayName: 'Smoke test - verify QA_BASE_URL is reachable'
146
180
 
147
181
  # ---------------------------------------------------------------
148
182
  # Playwright Workspaces (Azure App Testing)
@@ -159,9 +193,28 @@ stages:
159
193
  # inlineScript: echo "Authenticated via Entra ID"
160
194
 
161
195
  - script: |
196
+ # Scheduled runs always execute the full suite, regardless of the
197
+ # suiteFilter parameter. This prevents accidental partial runs in
198
+ # nightly regressions when the parameter was changed by a prior
199
+ # manual queue.
200
+ SUITE_FILTER="${{ parameters.suiteFilter }}"
201
+ if [ "$(Build.Reason)" = "Schedule" ]; then
202
+ SUITE_FILTER="all"
203
+ echo "Scheduled run detected - overriding suiteFilter to 'all'"
204
+ fi
205
+ echo "Effective suiteFilter: $SUITE_FILTER"
206
+
207
+ # Build --grep args as a bash array to safely handle patterns that
208
+ # contain special regex characters such as '[' and ']'. Inline
209
+ # string interpolation would break those patterns at the shell level.
210
+ GREP_ARGS=()
211
+ if [ "$SUITE_FILTER" != "all" ]; then
212
+ GREP_ARGS=(--grep "$SUITE_FILTER")
213
+ fi
214
+
162
215
  cd $(E2E_DIR)
163
216
  npx playwright test --project=chromium --ignore-snapshots \
164
- --grep-invert "_inspect|_diag"
217
+ --grep-invert "_inspect|_diag" "${GREP_ARGS[@]}"
165
218
  displayName: 'Run E2E tests'
166
219
  continueOnError: true
167
220
  env:
@@ -173,31 +226,40 @@ stages:
173
226
  ADO_PLAN_ID: $(ADO_E2E_PLAN_ID)
174
227
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)
175
228
  CI: 'true'
176
- # PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint — enable together with the login task
229
+ # QA_AUTH_USER: '1' # Set explicitly when running parallel jobs (user-1 / user-2 model)
230
+ # PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint - enable together with the login task
177
231
 
232
+ # publishRunAttachments is false because screenshots and traces are
233
+ # already uploaded as pipeline artifacts below. Setting it to true
234
+ # would double-publish the same files.
178
235
  - task: PublishTestResults@2
179
236
  condition: always()
180
237
  inputs:
181
238
  testResultsFormat: 'JUnit'
182
- testResultsFiles: '**/test-results/**/*.xml'
239
+ testResultsFiles: 'test-results/junit-results.xml'
183
240
  searchFolder: $(E2E_DIR)
184
241
  mergeTestResults: true
242
+ publishRunAttachments: false
185
243
  failTaskOnFailedTests: true
186
244
  testRunTitle: '[Auto E2E] $(Build.BuildNumber)'
187
245
  displayName: 'Publish E2E test results (JUnit)'
188
246
 
247
+ # Artifact names include $(System.JobAttempt) to avoid name collisions
248
+ # when the job is retried. ADO rejects duplicate artifact names within
249
+ # the same pipeline run, so without the suffix a retry always fails at
250
+ # the publish step.
189
251
  - task: PublishPipelineArtifact@1
190
252
  condition: always()
191
253
  inputs:
192
254
  targetPath: '$(E2E_DIR)/playwright-report'
193
- artifact: 'e2e-playwright-report'
255
+ artifact: 'e2e-playwright-report-attempt-$(System.JobAttempt)'
194
256
  displayName: 'Publish E2E HTML report'
195
257
 
196
258
  - task: PublishPipelineArtifact@1
197
259
  condition: always()
198
260
  inputs:
199
261
  targetPath: '$(E2E_DIR)/test-results'
200
- artifact: 'e2e-test-results'
262
+ artifact: 'e2e-test-results-attempt-$(System.JobAttempt)'
201
263
  displayName: 'Publish E2E trace artifacts'
202
264
 
203
265
  # -----------------------------------------------------------------------
@@ -237,6 +299,7 @@ stages:
237
299
  # inlineScript: echo "Authenticated via Entra ID"
238
300
 
239
301
  - script: |
302
+ # API integration tests do not use suite tags - always run all.
240
303
  cd $(API_DIR)
241
304
  npx playwright test --grep-invert "_inspect|_diag"
242
305
  displayName: 'Run API integration tests'
@@ -250,29 +313,33 @@ stages:
250
313
  ADO_PLAN_ID: $(ADO_API_PLAN_ID)
251
314
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)
252
315
  CI: 'true'
253
- # PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint — enable together with the login task
316
+ # PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL) # Playwright Workspaces endpoint - enable together with the login task
254
317
 
318
+ # publishRunAttachments is false because trace artifacts are already
319
+ # uploaded as pipeline artifacts below.
255
320
  - task: PublishTestResults@2
256
321
  condition: always()
257
322
  inputs:
258
323
  testResultsFormat: 'JUnit'
259
- testResultsFiles: '**/test-results/**/*.xml'
324
+ testResultsFiles: 'test-results/junit-results.xml'
260
325
  searchFolder: $(API_DIR)
261
326
  mergeTestResults: true
327
+ publishRunAttachments: false
262
328
  failTaskOnFailedTests: true
263
329
  testRunTitle: '[Auto API] $(Build.BuildNumber)'
264
330
  displayName: 'Publish API test results (JUnit)'
265
331
 
332
+ # Artifact names include $(System.JobAttempt) to prevent collision on retries.
266
333
  - task: PublishPipelineArtifact@1
267
334
  condition: always()
268
335
  inputs:
269
336
  targetPath: '$(API_DIR)/playwright-report'
270
- artifact: 'api-playwright-report'
337
+ artifact: 'api-playwright-report-attempt-$(System.JobAttempt)'
271
338
  displayName: 'Publish API HTML report'
272
339
 
273
340
  - task: PublishPipelineArtifact@1
274
341
  condition: always()
275
342
  inputs:
276
343
  targetPath: '$(API_DIR)/test-results'
277
- artifact: 'api-test-results'
344
+ artifact: 'api-test-results-attempt-$(System.JobAttempt)'
278
345
  displayName: 'Publish API trace artifacts'
@@ -66,3 +66,127 @@ test.describe('[TC-M-S-NNN] Module > Submodule @P0', () => {
66
66
  });
67
67
  });
68
68
  ```
69
+
70
+ ## Visual AI helper (optional advanced pattern)
71
+
72
+ > **Advanced / optional.** This pattern is not part of the default scaffold and requires
73
+ > an AI vision API key. Use it when a selector-based assertion is too brittle or the
74
+ > element structure changes frequently (e.g. PDF previews, dashboard charts, export outputs).
75
+
76
+ Instead of asserting specific selectors, take a screenshot and ask an AI vision model to
77
+ verify that the expected content is visually present. This decouples the assertion from DOM
78
+ structure and is useful for generated documents or complex layouts.
79
+
80
+ ```typescript
81
+ // fixtures/vision-helper.ts
82
+ import * as https from 'https';
83
+
84
+ export interface VisionResult {
85
+ passes: boolean;
86
+ reason: string;
87
+ raw?: string;
88
+ }
89
+
90
+ /**
91
+ * Sends a screenshot to an AI vision model and asks whether the described
92
+ * content is visually present. Degrades gracefully when the API key is absent.
93
+ *
94
+ * Requires an API key in the environment (e.g. OPENAI_API_KEY for GPT-4o,
95
+ * or adapt the request body for any other vision-capable model).
96
+ */
97
+ export async function assertScreenshotShowsContent(
98
+ screenshotBuffer: Buffer,
99
+ contentDescription: string,
100
+ options: { model?: string; degradeOnMissingKey?: boolean } = {}
101
+ ): Promise<VisionResult> {
102
+ const apiKey = process.env.AI_VISION_API_KEY;
103
+ const degrade = options.degradeOnMissingKey ?? true;
104
+
105
+ if (!apiKey) {
106
+ if (degrade) {
107
+ console.warn('[vision-helper] AI_VISION_API_KEY not set. Skipping visual assertion.');
108
+ return { passes: true, reason: 'SKIPPED: AI_VISION_API_KEY not configured' };
109
+ }
110
+ return { passes: false, reason: 'AI_VISION_API_KEY not set in environment' };
111
+ }
112
+
113
+ const base64Image = screenshotBuffer.toString('base64');
114
+ const model = options.model ?? 'gpt-4o';
115
+
116
+ const prompt = `You are a QA assistant verifying screenshots of a web application.
117
+ Answer YES or NO: Does the screenshot clearly show ${contentDescription}?
118
+ Reply in exactly this format:
119
+ VERDICT: YES
120
+ REASON: <one short sentence>`;
121
+
122
+ const body = JSON.stringify({
123
+ model,
124
+ max_tokens: 150,
125
+ messages: [{
126
+ role: 'user',
127
+ content: [
128
+ { type: 'text', text: prompt },
129
+ { type: 'image_url', image_url: { url: `data:image/png;base64,${base64Image}`, detail: 'low' } },
130
+ ],
131
+ }],
132
+ });
133
+
134
+ const raw = await new Promise<string>((resolve, reject) => {
135
+ const req = https.request({
136
+ hostname: 'api.openai.com',
137
+ path: '/v1/chat/completions',
138
+ method: 'POST',
139
+ headers: {
140
+ 'Content-Type': 'application/json',
141
+ 'Authorization': `Bearer ${apiKey}`,
142
+ 'Content-Length': Buffer.byteLength(body),
143
+ },
144
+ }, (res) => {
145
+ let data = '';
146
+ res.on('data', (chunk) => { data += chunk; });
147
+ res.on('end', () => resolve(data));
148
+ });
149
+ req.on('error', reject);
150
+ req.write(body);
151
+ req.end();
152
+ });
153
+
154
+ const parsed = JSON.parse(raw);
155
+ const text: string = parsed.choices?.[0]?.message?.content ?? '';
156
+ const verdict = text.match(/VERDICT:\s*(YES|NO)/i);
157
+ const reason = text.match(/REASON:\s*(.+)/i);
158
+
159
+ if (!verdict) return { passes: false, reason: `Unexpected response: ${text}`, raw: text };
160
+ return {
161
+ passes: verdict[1].toUpperCase() === 'YES',
162
+ reason: reason?.[1]?.trim() ?? text,
163
+ raw: text,
164
+ };
165
+ }
166
+ ```
167
+
168
+ **Usage in a test:**
169
+
170
+ ```typescript
171
+ import { assertScreenshotShowsContent } from '../fixtures/vision-helper';
172
+
173
+ test('[TC-MOD-001] Export PDF contains expected content', async ({ page }) => {
174
+ // ... trigger export ...
175
+ const screenshot = await page.screenshot();
176
+ const result = await assertScreenshotShowsContent(
177
+ screenshot,
178
+ 'a completed export confirmation message with a download link'
179
+ );
180
+ expect(result.passes, result.reason).toBe(true);
181
+ });
182
+ ```
183
+
184
+ **Setup notes:**
185
+
186
+ - Copy `vision-helper.ts` into your `fixtures/` directory.
187
+ - Set `AI_VISION_API_KEY` in your `.env` file (not committed). The example uses the OpenAI
188
+ API; adapt the request body for any other vision model.
189
+ - By default, the helper degrades to a skip (passes: true) when the key is absent, so CI
190
+ runs without the key do not fail on this assertion.
191
+ - `pdf-parse` (v2+) exports a `PDFParse` class rather than a function. Use
192
+ `new PDFParse({ data: buffer })` and call `.getText()` on the instance.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@keber/qa-framework",
3
- "version": "1.10.0",
3
+ "version": "1.11.3",
4
4
  "description": "Reusable spec-driven QA framework for IDE-agent-assisted automated testing. Installable as an npm package. Provides structure, templates, agent instructions, and optional integrations for Playwright and Azure DevOps.",
5
5
  "keywords": [
6
6
  "qa",
@@ -38,10 +38,9 @@
38
38
  "engines": {
39
39
  "node": ">=18"
40
40
  },
41
- "devDependencies": {},
42
41
  "peerDependencies": {
43
- "@playwright/test": ">=1.40.0",
44
- "@keber/ado-qa": ">=1.0.0"
42
+ "@keber/ado-qa": ">=1.0.0",
43
+ "@playwright/test": ">=1.40.0"
45
44
  },
46
45
  "peerDependenciesMeta": {
47
46
  "@playwright/test": {
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "$schema": "./qa-framework.config.schema.json",
3
- "frameworkVersion": "1.1.3",
3
+ "frameworkVersion": "1.10.0",
4
4
 
5
5
  "project": {
6
6
  "name": "{{PROJECT_NAME}}",