@msalaam/xray-qe-toolkit 1.4.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,372 @@
1
+ # Spec-Driven API Test Automation — QE Process & Workflow
2
+
3
+ > **Scope:** API testing with Playwright (no UI/browser).
4
+ > **Source of truth:** `openapi.yaml` (API shape) + `business-rules.yaml` (domain logic & scenarios).
5
+ > **Per-service repos:** Each API gets its own `{ServiceName}_Automation` repo.
6
+ >
7
+ > This document defines the end-to-end workflow from specification to test execution.
8
+
9
+ ---
10
+
11
+ ## Overview
12
+
13
+ The spec-driven approach uses two knowledge artifacts — an **OpenAPI specification** and a **`business-rules.yaml`** file — to produce a complete, traceable API test suite.
14
+
15
+ The process follows a strict sequence:
16
+
17
+ ```
18
+ openapi.yaml + business-rules.yaml
19
+
20
+ Generate tests.json (test definitions)
21
+
22
+ xqt push-tests → creates Xray Test issues + Test Sets in Jira
23
+
24
+ xray-mapping.json populated with issue keys
25
+
26
+ Playwright API tests created (1:1 with tests.json entries)
27
+
28
+ Sprint starts → create Test Plan → link Test Sets
29
+
30
+ Run tests → xqt import-results → Test Execution in Xray
31
+ ```
32
+
33
+ ### Why Two Files?
34
+
35
+ | File | What It Provides | Who Maintains |
36
+ |------|-----------------|---------------|
37
+ | `openapi.yaml` | Schemas, paths, status codes, parameter types, auth schemes | Developers |
38
+ | `business-rules.yaml` | Domain rules, validation logic, edge-case scenarios, test data | Developers + QA |
39
+
40
+ The OpenAPI spec tells you **what the API looks like**.
41
+ The business rules tell you **what the API should do**.
42
+ Together they are sufficient to generate a complete test suite. Neither alone is enough.
43
+
44
+ ### 1:1 Traceability
45
+
46
+ Every `business-rules.yaml` rule maps to exactly one `tests.json` entry, one Xray Test issue, and one Playwright `test()` call. Nothing exists in one place that doesn't exist in all three.
47
+
48
+ ---
49
+
50
+ ## Test Organisation in Jira/Xray
51
+
52
+ ### Test Sets (persistent groupings)
53
+
54
+ Tests live in **Test Sets** in Jira. Test Sets group related tests by feature, endpoint, or category. They are the primary organisational unit.
55
+
56
+ - One Test Set per `business-rules.yaml` feature (e.g. "Health Check", "User Management", "Payments")
57
+ - Tests are added to Test Sets when pushed via `xqt push-tests`
58
+ - Test Sets are permanent — they persist across sprints
59
+ - The `testSet` field in `tests.json` determines which Test Set a test belongs to
60
+
61
+ ### Test Plans (per-sprint)
62
+
63
+ A **Test Plan** represents work for a specific sprint or release. When entering a sprint:
64
+
65
+ 1. Create a Test Plan for the sprint: `npx xqt create-plan --summary "Sprint 12 — Payments API"`
66
+ 2. Link the relevant Test Sets to the Test Plan
67
+ 3. Each CI run creates a new **Test Execution** linked to that Test Plan
68
+ 4. At sprint end, the Test Plan shows full pass/fail history for that sprint's scope
69
+
70
+ This means:
71
+ - **Test Sets** = what tests exist (permanent)
72
+ - **Test Plans** = what we're testing this sprint (per-sprint)
73
+ - **Test Executions** = individual CI runs (ephemeral)
74
+
75
+ ---
76
+
77
+ ## Deterministic Generation Rules
78
+
79
+ > **These rules are contracts, not guidelines.**
80
+ > The same inputs must always produce the same outputs. Any deviation breaks `xray-mapping.json` stability and orphans Xray test issues.
81
+
82
+ ### `testId` Derivation
83
+
84
+ ```
85
+ testId = "TC-{SERVICE_CODE}-{SCENARIO_ID}"
86
+ ```
87
+
88
+ - `SERVICE_CODE` — uppercase alphanumeric slug of the service name, max 8 chars. Example: `PAYMENTS`, `USERMGMT`, `AUTH`.
89
+ - `SCENARIO_ID` — the `id` field from `business-rules.yaml` verbatim. Example: `BR-001`, `BR-042`.
90
+ - Full example: `TC-PAYMENTS-BR-001`
91
+ - `testId` values are **stable and permanent**. Once pushed to Xray they never change.
92
+ - If a rule is removed from `business-rules.yaml`, its entry in `tests.json` gets `"skip": true` — it is never deleted.
93
+
94
+ ### `tests.json` Field Mapping
95
+
96
+ Every field is derived deterministically from the knowledge artifacts. No creative decisions.
97
+
98
+ | `tests.json` field | Source | Derivation rule |
99
+ |---|---|---|
100
+ | `test_id` | `rule.id` | `TC-{SERVICE_CODE}-{rule.id}` |
101
+ | `xray.summary` | `rule.description` | Copied verbatim |
102
+ | `xray.description` | `rule.given` + `rule.when` + `rule.then` | `"Given {given}, when {when}, then {then}"` |
103
+ | `tags` | `rule.tags` | Copied from rule tags array |
104
+ | `xray.priority` | `rule.priority` | Direct mapping (see table below) |
105
+ | `requirementKeys` | `rule.requirementKeys` | Copied verbatim. Empty array if absent |
106
+ | `folder` | Feature name + category | `"/{ServiceName}/{FeatureName}/{CategoryFolder}"` |
107
+ | `testSet` | Feature `name` | The feature name becomes the Test Set name |
108
+ | `skip` | `rule.skip` or absence flag | `false` by default. `true` if rule deprecated |
109
+ | `xray.testType` | Always `"Generic"` | All automated API tests are Generic type |
110
+ | `xray.definition` | rule description + endpoint | `"Automated Playwright API test: {endpoint} — {rule.description}"` |
111
+
112
+ #### Priority Mapping
113
+
114
+ | `business-rules.yaml` priority | `tests.json` / Xray priority |
115
+ |---|---|
116
+ | `Highest` | `Highest` |
117
+ | `High` | `High` |
118
+ | `Medium` | `Medium` |
119
+ | `Low` | `Low` |
120
+ | `Lowest` | `Lowest` |
121
+ | *(absent)* | `Medium` |
122
+
123
+ #### Folder Path Rules
124
+
125
+ ```
126
+ /{ServiceName}/{FeatureName}/{CategoryFolder}
127
+ ```
128
+
129
+ | `rule.category` | Folder segment |
130
+ |---|---|
131
+ | `validation` | `Validation` |
132
+ | `authorization` | `Authorization` |
133
+ | `business-logic` | `BusinessLogic` |
134
+ | `edge-case` | `Edge` |
135
+ | `integration` | `Integration` |
136
+ | `security` | `Security` |
137
+ | `performance` | `Performance` |
138
+ | *(absent)* | `General` |
139
+
140
+ Full example: `/Payments/Transactions/Validation`
141
+
142
+ ---
143
+
144
+ ## `business-rules.yaml` Contract
145
+
146
+ Every rule must contain these fields. Validate before generation — report violations before writing any files.
147
+
148
+ ```yaml
149
+ service: my-service
150
+ version: "1.0"
151
+
152
+ features:
153
+ - name: Health Check # Required. Becomes the Test Set name.
154
+ endpoints: # Required. Links rules to OpenAPI paths.
155
+ - GET /health
156
+ rules:
157
+ - id: BR-001 # Required. Stable. Never reuse a retired ID.
158
+ description: Service returns 200 OK # Required. Becomes xray.summary.
159
+ given: Service is running # Required. Precondition.
160
+ when: GET /health is called # Required. Action.
161
+ then: Response is 200 # Required. Expected outcome.
162
+ category: validation # Required. Controls folder placement.
163
+ priority: High # Required. Maps to Xray priority.
164
+ tags: [smoke, regression] # Optional. Used as labels.
165
+ requirementKeys: [] # Optional. Creates Xray coverage links.
166
+ skip: false # Optional. Defaults to false.
167
+ ```
168
+
169
+ If `business-rules.yaml` does not exist, generate a draft from the OpenAPI spec and flag it as `DRAFT — REQUIRES QA + DEV REVIEW`.
170
+
171
+ ---
172
+
173
+ ## Workflow
174
+
175
+ ### Step 0: Scaffold the Project
176
+
177
+ **Who:** QA Engineer (one-time)
178
+ **Command:** `npx xqt init`
179
+
180
+ 1. Create the repo: `{ServiceName}_Automation`
181
+ 2. Run `npx xqt init` — produces the full scaffold
182
+ 3. Copy `.env.example` to `.env` and fill in credentials
183
+ 4. Set up ADO variable groups:
184
+ - **`xray-qa-shared`** (shared across repos): `xray_client_id`, `xray_client_secret`, `JIRA_PROJECT_KEY`, `JIRA_URL`, `JIRA_API_TOKEN`, `JIRA_EMAIL`
185
+ - **`{ServiceName}-qa`** (per-repo): `base_url`, `gravitee_api_key`, service-specific secrets
186
+
187
+ `xqt init` is idempotent — running it again won't overwrite existing files.
188
+
189
+ ### Step 1: Prepare Knowledge Artifacts
190
+
191
+ **Who:** QA Engineer + Dev Team
192
+
193
+ 1. Place the OpenAPI spec at `resources/api-specs/openapi.yaml`
194
+ 2. Create `resources/business-rules.yaml` using the contract above
195
+ - One feature per logical API grouping
196
+ - One rule per testable behaviour
197
+ - Every rule has `id`, `description`, `given`, `when`, `then`, `category`, `priority`
198
+ 3. Validate: `npx xqt validate`
199
+
200
+ ### Step 2: Generate `tests.json`
201
+
202
+ **Who:** AI Agent (Copilot skill)
203
+ **Input:** `openapi.yaml` + `business-rules.yaml`
204
+ **Output:** Populated `tests.json`
205
+
206
+ The AI agent reads both artifacts and produces `tests.json` using the **Deterministic Generation Rules** above. Every `business-rules.yaml` rule becomes exactly one `tests.json` entry.
207
+
208
+ - Each feature's rules are grouped under a `testSet` matching the feature name
209
+ - `test_id` values follow the `TC-{SERVICE_CODE}-{SCENARIO_ID}` pattern
210
+ - All fields derived deterministically — no creative decisions
211
+
212
+ After generation, validate: `npx xqt validate`
213
+
214
+ ### Step 3: Push Tests to Xray
215
+
216
+ **Who:** QA Engineer or CI
217
+ **Command:** `npx xqt push-tests`
218
+
219
+ This creates/updates Xray Test issues and Test Sets in Jira:
220
+ - New tests → created in Xray, added to appropriate Test Set
221
+ - Existing tests (in `xray-mapping.json`) → updated in place
222
+ - Tests with `"skip": true` → excluded
223
+ - `xray-mapping.json` is auto-populated with the Jira issue keys
224
+
225
+ After push, `xray-mapping.json` contains the mapping:
226
+ ```json
227
+ {
228
+ "TC-PAYMENTS-BR-001": { "key": "APIEE-1234", "id": "8765432" },
229
+ "TC-PAYMENTS-BR-002": { "key": "APIEE-1235", "id": "8765433" }
230
+ }
231
+ ```
232
+
233
+ **Commit `tests.json` and `xray-mapping.json` together.**
234
+
235
+ ### Step 4: Create Playwright API Tests
236
+
237
+ **Who:** AI Agent (Copilot skill — separate from XQT)
238
+
239
+ Playwright tests are created **after** `xray-mapping.json` exists, so every test can reference its real Jira key from the start.
240
+
241
+ - One `test()` call per `tests.json` entry — 1:1 mapping
242
+ - Test titles include the Xray key: `[APIEE-1234] Service returns 200 OK when healthy`
243
+ - Tests use `test.info().annotations` to link to Xray
244
+ - The Copilot skill reads `tests.json` and `xray-mapping.json` to generate each spec
245
+
246
+ > **Note:** XQT does not generate Playwright tests. Test code is created by a separate Copilot skill
247
+ > that uses this spec-driven approach, `tests.json`, and `xray-mapping.json` as its inputs.
248
+
249
+ ### Step 5: Sprint Workflow — Test Plans
250
+
251
+ **Who:** QA Engineer
252
+
253
+ When entering a new sprint:
254
+
255
+ 1. **Create a Test Plan:**
256
+ ```bash
257
+ npx xqt create-plan --summary "Sprint 12 — Payments API Regression"
258
+ ```
259
+
260
+ 2. **Link relevant Test Sets** to the Test Plan in Jira
261
+ - Select the Test Sets containing tests relevant to this sprint's scope
262
+ - This makes the Test Plan a curated view of what matters for the sprint
263
+
264
+ 3. **Run tests and import results:**
265
+ ```bash
266
+ npx playwright test
267
+ npx xqt import-results --file test-results/results.json --env IOP-QA
268
+ ```
269
+ Each run creates a new Test Execution linked to the Test Plan.
270
+
271
+ 4. **Sprint complete:** The Test Plan in Jira shows the full execution history for the sprint.
272
+
273
+ ### Step 6: CI/CD — Ongoing
274
+
275
+ Every CI pipeline run:
276
+ ```bash
277
+ npx xqt validate # Schema check
278
+ npx playwright test # Run tests
279
+ npx xqt import-results --file test-results/results.json \
280
+ --env IOP-QA # Import to Xray
281
+ ```
282
+
283
+ Results appear as new Test Executions linked to the active Test Plan.
284
+
285
+ ---
286
+
287
+ ## Brownfield Workflow (Updating Existing Tests)
288
+
289
+ When the OpenAPI spec or business rules change:
290
+
291
+ 1. **Update `business-rules.yaml`** — add new rules, modify existing, mark removed rules with `skip: true`
292
+ 2. **Regenerate `tests.json`** — AI agent re-derives from updated artifacts
293
+ - New rules → new `tests.json` entries appended
294
+ - Modified rules → existing entries updated field-by-field (never change `test_id`)
295
+ - Removed rules → `"skip": true` (never delete)
296
+ 3. **Push changes:** `npx xqt push-tests` — creates new Xray issues, updates existing
297
+ 4. **Update Playwright tests** — Copilot skill creates tests for new entries, updates existing
298
+ 5. **Validate:** run the suite, import results
299
+
300
+ **Critical rule:** `xray-mapping.json` entries are never removed or overwritten for existing `test_id` values. The mapping only grows.
301
+
302
+ ---
303
+
304
+ ## Workflow Summary
305
+
306
+ | Step | Who | Action | Output |
307
+ |------|-----|--------|--------|
308
+ | 0 | QA Engineer | `xqt init` | Scaffolded project |
309
+ | 1 | QA + Dev | Prepare `openapi.yaml` + `business-rules.yaml` | Validated knowledge artifacts |
310
+ | 2 | AI Agent | Generate from specs | `tests.json` populated |
311
+ | 3 | QA / CI | `xqt push-tests` | Xray Tests + Test Sets created, `xray-mapping.json` populated |
312
+ | 4 | AI Agent | Create Playwright tests | `*.spec.ts` files (1:1 with `tests.json`) |
313
+ | 5 | QA Engineer | `xqt create-plan` + link Test Sets | Sprint Test Plan ready |
314
+ | 6 | CI | `playwright test` + `xqt import-results` | Test Execution in Xray |
315
+
316
+ ---
317
+
318
+ ## Key Principles
319
+
320
+ 1. **`tests.json` is the source of truth** for test metadata — never edit tests directly in Jira
321
+ 2. **`xray-mapping.json` is owned by XQT** — only `xqt push-tests` writes it, never edit manually
322
+ 3. **`testId` values are permanent** — once pushed, they never change; retired rules get `"skip": true`
323
+ 4. **Test Sets are the home for tests** — persistent groupings that live across sprints
324
+ 5. **Test Plans are per-sprint** — create one per sprint, link relevant Test Sets
325
+ 6. **Test Executions are ephemeral** — each CI run creates a fresh one
326
+ 7. **1:1 mapping everywhere** — one rule → one `tests.json` entry → one Xray issue → one `test()` call
327
+ 8. **Generation is deterministic** — same inputs always produce same outputs
328
+ 9. **Playwright tests come last** — only created after `xray-mapping.json` exists with real Jira keys
329
+ 10. **Never store secrets in code** — all credentials in `.env` or ADO variable groups
330
+
331
+ ---
332
+
333
+ ## QE Process — End-to-End
334
+
335
+ ```
336
+ Dev team maintains openapi.yaml + business-rules.yaml
337
+
338
+
339
+ AI Agent generates tests.json from both artifacts
340
+ (deterministic rules — no creative decisions)
341
+
342
+
343
+ npx xqt validate (hard stop on any error)
344
+
345
+
346
+ npx xqt push-tests
347
+ → Xray Test issues created/updated
348
+ → Test Sets created/updated
349
+ → xray-mapping.json auto-populated
350
+
351
+
352
+ AI Agent (Copilot skill) creates Playwright API tests
353
+ → reads tests.json + xray-mapping.json
354
+ → one spec per tests.json entry
355
+ → real Jira keys in test titles from day one
356
+
357
+
358
+ Sprint starts → QA creates Test Plan → links Test Sets
359
+
360
+
361
+ CI runs: playwright test → xqt import-results
362
+ → Test Execution created in Xray
363
+ → linked to Test Plan
364
+
365
+
366
+ Full traceability chain:
367
+ Business Rule → tests.json → Xray Test (in Test Set) → Playwright test → Test Execution (in Test Plan)
368
+ ```
369
+
370
+ ---
371
+
372
+ **The contract:** The OpenAPI spec defines what the API looks like. The business rules define what the API should do. `tests.json` captures both as structured test definitions. XQT syncs those definitions to Xray. Copilot writes the Playwright tests. Test Sets organise the tests permanently. Test Plans scope them to sprints. Nothing else is needed.
@@ -1,84 +1,136 @@
1
- # ──────────────────────────────────────────────────────────────
2
- # Azure DevOps Pipeline — Xray QE Regression Runner
3
- # Generated by @msalaam/xray-qe-toolkit
4
- # ──────────────────────────────────────────────────────────────
5
- #
6
- # This pipeline runs Playwright tests and imports the JSON results
7
- # into Xray Cloud via xqt import-results.
8
- #
9
- # IMPORTANT: edit-json and QE review gates are NOT part of CI.
10
- # Those steps must be run locally before committing.
1
+
2
+ # azure-pipelines.yml
3
+ # Purpose: Standalone QA Regression pipeline — Playwright API tests + Xray result import
11
4
  #
12
- # Required pipeline variables (set in Azure DevOps UI Variables):
13
- # XRAY_ID, XRAY_SECRET, JIRA_PROJECT_KEY, JIRA_URL,
14
- # JIRA_API_TOKEN, JIRA_EMAIL, TEST_EXEC_KEY
5
+ # Variable groups required (configure in ADO Library before first run):
6
+ # 1. xray-qa-shared — Shared across ALL repos (one-time setup):
7
+ # xray_client_id, xray_client_secret,
8
+ # JIRA_PROJECT_KEY, JIRA_URL,
9
+ # JIRA_API_TOKEN, JIRA_EMAIL,
10
+ # XRAY_GRAPHQL_URL_US, XRAY_REST_URL
11
+ # 2. {{SERVICE_NAME}}-qa — Per-repo (create in ADO Library for each service):
12
+ # base_url, gravitee_api_key,
13
+ # and any other service-specific secrets
15
14
  #
16
- # Optional pipeline variables:
17
- # API_BASE_URL — base URL passed to Playwright tests
18
- # ──────────────────────────────────────────────────────────────
15
+ # After `npx xqt init`, update {{SERVICE_NAME}}-qa above to match your ADO variable group name.
19
16
 
20
- trigger:
21
- branches:
22
- include:
23
- - main
24
- - release/*
17
+ trigger: none
18
+ pr: none
25
19
 
26
20
  pool:
27
- vmImage: "ubuntu-latest"
21
+ name: 'linux-docker-pool'
28
22
 
29
23
  variables:
30
- NODE_VERSION: "20.x"
31
-
32
- steps:
33
- - task: NodeTool@0
34
- displayName: "Install Node.js $(NODE_VERSION)"
35
- inputs:
36
- versionSpec: "$(NODE_VERSION)"
37
-
38
- - script: |
39
- npm ci
40
- displayName: "Install dependencies"
41
-
42
- - script: |
43
- npx playwright install --with-deps chromium
44
- displayName: "Install Playwright browsers"
45
-
46
- - script: |
47
- npx playwright test \
48
- --reporter=list,json \
49
- --output=playwright-results
50
- displayName: "Run Playwright tests"
51
- continueOnError: true
52
- env:
53
- PLAYWRIGHT_JSON_OUTPUT_NAME: playwright-results.json
54
- API_BASE_URL: $(API_BASE_URL)
55
-
56
- - script: |
57
- npx xqt import-results \
58
- --file playwright-results.json \
59
- --testExecKey $(TEST_EXEC_KEY)
60
- displayName: "Import Playwright results to Xray"
61
- env:
62
- XRAY_ID: $(XRAY_ID)
63
- XRAY_SECRET: $(XRAY_SECRET)
64
- JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
65
- JIRA_URL: $(JIRA_URL)
66
- JIRA_API_TOKEN: $(JIRA_API_TOKEN)
67
- JIRA_EMAIL: $(JIRA_EMAIL)
68
-
69
- - task: PublishTestResults@2
70
- displayName: "Publish Playwright JUnit results"
71
- condition: always()
72
- inputs:
73
- testResultsFormat: "JUnit"
74
- testResultsFiles: "playwright-results/results.xml"
75
- mergeTestResults: true
76
- failTaskOnFailedTests: true
77
-
78
- - task: PublishPipelineArtifact@1
79
- displayName: "Upload Playwright HTML report"
80
- condition: always()
81
- inputs:
82
- targetPath: "playwright-report"
83
- artifact: "playwright-report"
84
- publishLocation: "pipeline"
24
+ - group: xray-qa-shared # Shared Xray + Jira credentials (all repos)
25
+ - group: "{{SERVICE_NAME}}-qa" # Per-repo: base_url, gravitee_api_key, etc.
26
+
27
+ stages:
28
+
29
+ # ─────────────────────────────────────────────────────────────────────────────
30
+ # Stage 1 — Code Quality: ESLint + TypeScript type check
31
+ # ─────────────────────────────────────────────────────────────────────────────
32
+ - stage: CodeQuality
33
+ displayName: "Code Quality"
34
+ jobs:
35
+ - job: Lint
36
+ displayName: "ESLint and TypeScript Check"
37
+ steps:
38
+ - checkout: self
39
+
40
+ - task: NodeTool@0
41
+ inputs:
42
+ versionSpec: '18.x'
43
+ displayName: "Use Node 18"
44
+
45
+ - script: |
46
+ set -euo pipefail
47
+ npm ci
48
+ displayName: "Install npm dependencies"
49
+
50
+ - script: |
51
+ set -euo pipefail
52
+ npx eslint . --ext .ts --max-warnings 0
53
+ displayName: "Run ESLint"
54
+
55
+ - script: |
56
+ set -euo pipefail
57
+ npx tsc --noEmit
58
+ displayName: "TypeScript type check"
59
+
60
+ # ─────────────────────────────────────────────────────────────────────────────
61
+ # Stage 2 — QA Regression: Playwright API tests, Xray import, artifact publish
62
+ # ─────────────────────────────────────────────────────────────────────────────
63
+ - stage: QA_Regression
64
+ displayName: "Playwright QA Regression"
65
+ dependsOn: CodeQuality
66
+ jobs:
67
+ - job: RunPlaywright
68
+ displayName: "Run Playwright API Tests and Publish Results"
69
+ steps:
70
+ - checkout: self
71
+
72
+ - task: NodeTool@0
73
+ inputs:
74
+ versionSpec: '18.x'
75
+ displayName: "Use Node 18"
76
+
77
+ - script: |
78
+ set -euo pipefail
79
+ npm ci
80
+ displayName: "Install npm dependencies"
81
+
82
+ - script: |
83
+ set -euo pipefail
84
+ npx playwright test
85
+ displayName: 'Run Playwright API Tests'
86
+ env:
87
+ CI: 'true'
88
+ BASE_URL: $(base_url)
89
+ GRAVITEE_API_KEY: $(gravitee_api_key)
90
+ XRAY_CLIENT_ID: $(xray_client_id)
91
+ XRAY_CLIENT_SECRET: $(xray_client_secret)
92
+
93
+ # xqt import-results: creates a new Xray Test Execution per run.
94
+ # Uploads Playwright JSON results with per-step status and attaches
95
+ # response body evidence to each test via attachResponse() calls in specs.
96
+ - script: |
97
+ set -euo pipefail
98
+ REVISION=$(echo "$(Build.SourceVersion)" | cut -c1-8)
99
+ npx xqt import-results \
100
+ --file test-results/results.json \
101
+ --version "$(Build.BuildNumber)" \
102
+ --revision "$REVISION"
103
+ displayName: "Import Playwright Results → Xray"
104
+ condition: succeededOrFailed()
105
+ env:
106
+ XRAY_ID: $(xray_client_id)
107
+ XRAY_SECRET: $(xray_client_secret)
108
+ JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
109
+ JIRA_URL: $(JIRA_URL)
110
+ JIRA_API_TOKEN: $(JIRA_API_TOKEN)
111
+ JIRA_EMAIL: $(JIRA_EMAIL)
112
+ XRAY_GRAPHQL_URL: $(XRAY_GRAPHQL_URL_US)
113
+ XRAY_REST_URL: $(XRAY_REST_URL)
114
+
115
+ - task: PublishTestResults@2
116
+ inputs:
117
+ testResultsFormat: 'JUnit'
118
+ testResultsFiles: '**/results.xml'
119
+ failTaskOnFailedTests: true
120
+ testRunTitle: '{{SERVICE_NAME}} Playwright API Tests'
121
+ displayName: 'Publish JUnit test results'
122
+ condition: succeededOrFailed()
123
+
124
+ - task: PublishPipelineArtifact@1
125
+ inputs:
126
+ targetPath: 'playwright-report'
127
+ artifact: 'playwright-html-report'
128
+ displayName: 'Publish Playwright HTML Report'
129
+ condition: succeededOrFailed()
130
+
131
+ - task: PublishPipelineArtifact@1
132
+ inputs:
133
+ targetPath: 'test-results'
134
+ artifact: 'playwright-test-results'
135
+ displayName: 'Publish Playwright Test Results'
136
+ condition: succeededOrFailed()
@@ -0,0 +1,83 @@
1
+ # ─────────────────────────────────────────────────────────────────────────────
2
+ # business-rules.yaml — Service Business Rules
3
+ #
4
+ # Source of truth for test scenario generation alongside the OpenAPI spec.
5
+ # Each rule maps 1:1 to a tests.json entry, an Xray Test issue, and a
6
+ # Playwright test() call.
7
+ #
8
+ # Format guide:
9
+ # service — the service/microservice name
10
+ # version — rules document version (semantic versioning)
11
+ # features — list of features/modules in this service
12
+ # name — feature name (becomes the Test Set name in Jira)
13
+ # endpoints — API endpoints covered by this feature
14
+ # rules — individual business rules, each producing exactly 1 test
15
+ #
16
+ # Rule categories:
17
+ # validation — input validation, schema validation, required fields
18
+ # authorization — auth, RBAC, ownership checks
19
+ # business-logic — core domain rules, calculations, state transitions
20
+ # edge-case — boundary values, empty/null inputs, rate limits
21
+ # integration — downstream service behaviour, webhooks, events
22
+ # security — security-specific checks beyond auth
23
+ # performance — response time, throughput checks
24
+ # ─────────────────────────────────────────────────────────────────────────────
25
+
26
+ service: my-service
27
+ version: "1.0"
28
+
29
+ features:
30
+
31
+ - name: Health Check
32
+ endpoints:
33
+ - GET /health
34
+ rules:
35
+ - id: BR-001
36
+ description: Service returns 200 OK when healthy
37
+ given: Service is running with all dependencies reachable
38
+ when: GET /health is called
39
+ then: "Response is 200 with body { \"status\": \"ok\" }"
40
+ category: validation
41
+ priority: High
42
+ tags: [smoke, regression]
43
+
44
+ - name: Example Feature
45
+ endpoints:
46
+ - GET /api/v1/resource
47
+ - POST /api/v1/resource
48
+ rules:
49
+ - id: BR-002
50
+ description: Unauthenticated requests are rejected
51
+ given: No Authorization header is provided
52
+ when: Any protected endpoint is called
53
+ then: Response is 401 Unauthorized
54
+ category: authorization
55
+ priority: Highest
56
+ tags: [regression, security]
57
+
58
+ - id: BR-003
59
+ description: Valid resource creation returns 201 with resource id
60
+ given: Authenticated user with create permission
61
+ when: POST /api/v1/resource with valid body
62
+ then: "Response is 201 with { \"id\": \"<uuid>\" } and Location header"
63
+ category: business-logic
64
+ priority: High
65
+ tags: [regression, smoke]
66
+
67
+ - id: BR-004
68
+ description: Missing required fields return 400 with field-level errors
69
+ given: Authenticated user
70
+ when: POST /api/v1/resource with missing required fields
71
+ then: Response is 400 with errors array listing each missing field
72
+ category: validation
73
+ priority: High
74
+ tags: [regression, edge]
75
+
76
+ - id: BR-005
77
+ description: Resource id not found returns 404
78
+ given: Authenticated user
79
+ when: GET /api/v1/resource/{id} with non-existent id
80
+ then: Response is 404 with standard error body
81
+ category: edge-case
82
+ priority: Medium
83
+ tags: [regression, edge]