@msalaam/xray-qe-toolkit 1.4.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,1434 +1,768 @@
1
- # @msalaam/xray-qe-toolkit
2
-
3
- > Full QE workflow toolkit for Xray Cloud integration — test management, Postman generation, CI pipeline scaffolding, and browser-based review gates.
4
-
5
- ---
6
-
7
- ## Table of Contents
8
-
9
- - [Overview](#overview)
10
- - [Prerequisites](#prerequisites)
11
- - [Installation](#installation)
12
- - [AI Setup (Optional)](#ai-setup-optional)
13
- - [Quick Start](#quick-start)
14
- - [Playwright Quick Start](#playwright-quick-start)
15
- - [CLI Commands](#cli-commands)
16
- - [init](#xray-qe-init)
17
- - [gen-tests](#xray-qe-gen-tests)
18
- - [edit-json](#xray-qe-edit-json)
19
- - [push-tests](#xray-qe-push-tests)
20
- - [gen-postman](#xray-qe-gen-postman)
21
- - [create-execution](#xray-qe-create-execution)
22
- - [import-results](#xray-qe-import-results)
23
- - [gen-pipeline](#xray-qe-gen-pipeline)
24
- - [mcp-server](#xray-qe-mcp-server)
25
- - [compare-openapi](#xray-qe-compare-openapi)
26
- - [update-snapshot](#xray-qe-update-snapshot)
27
- - [Configuration](#configuration)
28
- - [Environment Variables (.env)](#environment-variables-env)
29
- - [Project Config (.xrayrc)](#project-config-xrayrc)
30
- - [File Reference](#file-reference)
31
- - [tests.json](#testsjson)
32
- - [xray-mapping.json](#xray-mappingjson)
33
- - [Multi-Company Test Format](#multi-company-test-format)
34
- - [Jira/Xray Project Setup](#jiraxray-project-setup)
35
- - [QE Workflow](#qe-workflow)
36
- - [Building Regression Packs](#building-regression-packs)
37
- - [Working with Existing Xray Tests](#working-with-existing-xray-tests)
38
- - [Idempotent Push](#idempotent-push)
39
- - [Programmatic API](#programmatic-api)
40
- - [Troubleshooting](#troubleshooting)
41
-
42
- ---
43
-
44
- ## Overview
45
-
46
- `@msalaam/xray-qe-toolkit` (XQT) is a modular, CLI-driven toolkit for Xray Cloud and JIRA-based QE workflows. It provides:
47
-
48
- 1. **`tests.json`** as the single source-of-truth for test logic
49
- 2. **Browser-based QE review gate** (`edit-json`) for human governance before any push
50
- 3. **Idempotent push** — updates existing tests, creates new ones, skips duplicates
51
- 4. **Postman collection generation** from test definitions
52
- 5. **CI pipeline template** for Azure DevOps (Newman + Xray import)
53
- 6. **Playwright integration** — import Playwright JSON results with automatic test mapping
54
- 7. **OpenAPI contract enforcement** — `compare-openapi` diffs a live spec against a QA snapshot and fails the pipeline on breaking changes; `update-snapshot` promotes a new baseline deliberately
55
- 8. **Modular architecture** every function is importable for programmatic use
56
- 9. **AI-ready scaffolds** optional AI assistance for test generation (manual workflow fully supported)
57
-
58
- ### QE Review Gate Philosophy
59
-
60
- **The knowledge/ folder is your single source of truth** — OpenAPI specs, requirements docs, JIRA tickets, and business logic live here. The toolkit uses these sources to generate test cases with AI assistance.
61
-
62
- **Nothing is pushed to Xray until a QE manually reviews and approves** via the `edit-json` command. This ensures quality governance and prevents untested automation from reaching Xray Cloud.
63
-
64
- ```
65
- init → populate knowledge/ → gen-tests (AI) → edit-json (QE gate) → push-tests → gen-postman (AI) → CI (run + import)
66
- ```
67
-
68
- **Generated tests are scaffolds** always review and enhance before pushing to Xray.
69
-
70
- ---
71
-
72
- ## Prerequisites
73
-
74
- - **Node.js** >= 18.0.0
75
- - **npm** >= 8.0.0
76
- - **Xray Cloud** API Key (from Xray > Settings > API Keys)
77
- - **JIRA Cloud** API Token (from https://id.atlassian.com/manage-profile/security/api-tokens)
78
- - Both credentials must belong to the **same JIRA user** (Xray impersonation requirement)
79
-
80
- ---
81
-
82
- ## Installation
83
-
84
- ### From npm (public)
85
-
86
- ```bash
87
- # Install as dev dependency
88
- npm install --save-dev @msalaam/xray-qe-toolkit
89
- ```
90
-
91
- ### Verify installation
92
-
93
- ```bash
94
- npx xqt --version
95
- npx xqt --help
96
- ```
97
-
98
- ---
99
-
100
- ## AI Setup (Optional)
101
-
102
- **AI is completely optional.** The toolkit works fully without AI — you can create and edit tests manually using the `edit-json` UI editor.
103
-
104
- ### Current AI Status
105
-
106
- 🚧 **All AI features are scaffolds** — ready for implementation but not yet connected to AI providers.
107
-
108
- **What's included:**
109
- - `--ai` flags in `gen-tests` and `gen-postman` commands
110
- - `knowledge/` folder scanner for API specs, requirements, tickets
111
- - ✅ MCP (Model Context Protocol) server scaffold in `mcp/server.js`
112
- - VS Code chat participant extension scaffold in `.vscode/xray-qe-participant/`
113
- - ❌ **Not included:** Actual AI provider connections (Azure OpenAI, GitHub Copilot, etc.)
114
-
115
- ### To Enable AI Features (Future)
116
-
117
- Choose one of three integration paths:
118
-
119
- #### Option 1: MCP Server (for GitHub Copilot CLI/Desktop)
120
-
121
- 1. Implement AI provider in `mcp/server.js` (Azure OpenAI, Anthropic, etc.)
122
- 2. Configure MCP client to connect to toolkit server
123
- 3. Start server: `npx xqt mcp-server`
124
- 4. Use `gen-tests --ai` with AI provider connected
125
-
126
- **Requires:** MCP client setup, AI provider API keys (Azure OpenAI, etc.)
127
-
128
- #### Option 2: VS Code Chat Participant (for GitHub Copilot in VS Code)
129
-
130
- 1. Implement AI provider in `.vscode/xray-qe-participant/extension.js`
131
- 2. Install extension: Copy folder to `~/.vscode/extensions/`
132
- 3. Reload VS Code
133
- 4. Use `@xray-qe` in chat with commands like `/generate`, `/postman`
134
-
135
- **Requires:** GitHub Copilot subscription, VS Code extension development
136
-
137
- #### Option 3: Direct CLI Integration
138
-
139
- 1. Modify `commands/genTests.js` to call your preferred AI service directly
140
- 2. Add provider credentials to `.env` (e.g., `AZURE_OPENAI_KEY`)
141
- 3. Use `gen-tests --ai` with integrated provider
142
-
143
- **Requires:** AI provider API keys, custom implementation
144
-
145
- ### Manual Workflow (No AI Required)
146
-
147
- **This workflow works TODAY without any AI setup:**
148
-
149
- ```bash
150
- # 1. Create tests manually using the UI editor
151
- npx xqt init
152
- npx xqt edit-json # Create tests from scratch or edit templates
153
-
154
- # 2. Push to Xray
155
- npx xqt push-tests
156
-
157
- # 3. Generate Postman collection (no AI)
158
- npx xqt gen-postman # Uses tests.json + xray-mapping.json directly
159
-
160
- # 4. Run in CI
161
- newman run collection.postman.json
162
- ```
163
-
164
- **All AI features gracefully degrade** without AI, commands provide helpful guidance for manual workflows.
165
-
166
- ---
167
-
168
- ## Quick Start
169
-
170
- ### With AI (Future — Requires Setup)
171
-
172
- ```bash
173
- # 1. Initialize project with knowledge/ folder and starter files
174
- npx xqt init
175
-
176
- # 2. Configure credentials
177
- cp .env.example .env
178
- # Edit .env with your Xray + JIRA credentials
179
-
180
- # 3. Add your API specs and requirements to knowledge/ folder
181
- # See knowledge/README.md for supported file types
182
-
183
- # 4. Set up AI provider (see "AI Setup" section above)
184
- # ... implement AI connection in MCP server or VS Code extension ...
185
-
186
- # 5. Generate test cases from knowledge sources (AI-assisted)
187
- npx xqt gen-tests --ai
188
-
189
- # 6. QE review gate — review and refine generated tests
190
- npx xqt edit-json
191
-
192
- # 7. Push approved tests to Xray Cloud
193
- npx xqt push-tests
194
-
195
- # 8. Generate Postman collection (AI-enhanced assertions)
196
- npx xqt gen-postman --ai
197
-
198
- # 9. Generate CI pipeline
199
- npx xqt gen-pipeline
200
-
201
- # 10. Run tests in CI
202
- newman run collection.postman.json --reporters cli,junit
203
- npx xqt import-results --file results.xml --testExecKey QE-123
204
- ```
205
-
206
- ### Without AI (Works Today)
207
-
208
- ```bash
209
- # 1. Initialize project
210
- npx xqt init
211
-
212
- # 2. Configure credentials
213
- cp .env.example .env
214
- # Edit .env with your Xray + JIRA credentials
215
-
216
- # 3. Create tests manually using UI editor
217
- npx xqt edit-json
218
- # - Click "Add Test" to create new tests
219
- # - Use knowledge/ docs as reference (manual review)
220
- # - Save when ready
221
-
222
- # 4. Push to Xray Cloud
223
- npx xqt push-tests
224
-
225
- # 5. Generate Postman collection (from tests.json)
226
- npx xqt gen-postman
227
-
228
- # 6. Generate CI pipeline
229
- npx xqt gen-pipeline
230
-
231
- # 7. Run tests in CI
232
- newman run collection.postman.json --reporters cli,junit
233
- npx xqt import-results --file results.xml --testExecKey QE-123
234
- ```
235
-
236
- ---
237
-
238
- ## CLI Commands
239
-
240
- All commands support these global options:
241
-
242
- | Option | Description |
243
- |---------------|------------------------------------|
244
- | `--verbose` | Enable debug output |
245
- | `--env <path>`| Custom path to .env file |
246
- | `--version` | Show version number |
247
- | `--help` | Show help for any command |
248
-
249
- ---
250
-
251
- ### `xqt init`
252
-
253
- Scaffold a new project with starter templates.
254
-
255
- ```bash
256
- npx xqt init
257
- ```
258
-
259
- **Creates:**
260
- - `knowledge/` folder with subdirectories (`api-specs/`, `requirements/`, `tickets/`)
261
- - `knowledge/README.md` — guide for organizing documentation
262
- - `XQT-GUIDE.md` — getting started guide for this toolkit (never named `README.md` to avoid overwriting yours)
263
- - `tests.json` — starter test definitions (marked as scaffolds)
264
- - `xray-mapping.json` — empty mapping file
265
- - `.env.example` — environment variable template
266
- - `.xrayrc` project-level config
267
-
268
- Existing files are **never overwritten** — the command skips them with a warning.
269
-
270
- ---
271
-
272
- ### `xqt gen-tests`
273
-
274
- Generate test cases from `knowledge/` folder documentation (AI-assisted or manual guidance).
275
-
276
- ```bash
277
- # AI-assisted generation (requires AI provider setup — see "AI Setup" section)
278
- npx xqt gen-tests --ai
279
-
280
- # Focus on a specific OpenAPI spec
281
- npx xqt gen-tests --ai --spec knowledge/api-specs/users-api.yaml
282
-
283
- # Use a custom knowledge folder
284
- npx xqt gen-tests --ai --knowledge ./docs
285
-
286
- # Fetch and analyze a JIRA ticket
287
- npx xqt gen-tests --ai --ticket APIEE-123
288
-
289
- # Generate from a prompt
290
- npx xqt gen-tests --ai --prompt "Generate tests for user authentication flows"
291
- ```
292
-
293
- | Option | Description | Required |
294
- |---------------------|--------------------------------------------|----------|
295
- | `--ai` | Enable AI-assisted generation | Yes |
296
- | `--spec <path>` | OpenAPI/Swagger spec file | No |
297
- | `--knowledge <path>`| Custom knowledge folder path | No |
298
- | `--ticket <key>` | JIRA ticket key to fetch and analyze | No |
299
- | `--prompt <text>` | Natural language prompt | No |
300
-
301
- **Output:** Appends generated tests to `tests.json` (or creates it if it doesn't exist)
302
-
303
- **🚧 AI Provider Required:** The `--ai` flag currently shows a scaffold message. To enable actual AI generation, implement an AI provider connection (see [AI Setup](#ai-setup-optional)).
304
-
305
- **Without AI:** Run `gen-tests` without `--ai` for manual creation guidance, or use `edit-json` to create tests directly in the UI editor.
306
-
307
- Existing files are **never overwritten** the command skips them with a warning.
308
-
309
- ---
310
-
311
- ### `xqt edit-json`
312
-
313
- Launch the browser-based QE review gate editor.
314
-
315
- ```bash
316
- npx xqt edit-json
317
- npx xqt edit-json --port 3000
318
- ```
319
-
320
- | Option | Description | Default |
321
- |-----------------|------------------------------------------|---------|
322
- | `--port <n>` | Port for local editor server | Random |
323
-
324
- **Features:**
325
- - Add, edit, delete tests in a visual editor
326
- - Tag tests: `regression`, `smoke`, `edge`, `critical`, `integration`, `e2e`, `security`, `performance`
327
- - Toggle skip/push per test
328
- - View Xray mapping status (which tests are already pushed)
329
- - Real-time validation
330
- - "Save & Exit" writes `tests.json` and shuts down the server
331
-
332
- **Important:** This is the QE governance gate. Nothing is pushed to Xray until the QE saves and explicitly runs `push-tests`.
333
-
334
- ---
335
-
336
- ### `xqt push-tests`
337
-
338
- Push or update tests in Xray Cloud from `tests.json`.
339
-
340
- ```bash
341
- # Create new Test Execution and link all tests
342
- npx xqt push-tests
343
-
344
- # Link tests to an existing Test Execution
345
- npx xqt push-tests --testExecKey QE-123
346
-
347
- # Push tests without creating/linking any execution
348
- npx xqt push-tests --skip-exec
349
- ```
350
-
351
- | Option | Description |
352
- |---------------------|----------------------------------------------------|
353
- | `--testExecKey <key>` | Link to an existing Test Execution instead of creating new |
354
- | `--skip-exec` | Don't create or link any Test Execution |
355
-
356
- **Behavior:**
357
- - Tests with `"skip": true` are excluded
358
- - Tests already in `xray-mapping.json` are **updated** (summary, description, labels, priority, steps)
359
- - New tests are **created** with type set to "Automated"
360
- - Mapping is saved incrementally (crash-safe)
361
- - 300ms rate-limit delay between API calls
362
-
363
- ---
364
-
365
- ### `xqt gen-postman`
366
-
367
- Generate a Postman Collection v2.1 JSON from `tests.json` (works with or without AI).
368
-
369
- ```bash
370
- # Generate collection with JIRA keys embedded (run after push-tests)
371
- npx xqt gen-postman
372
-
373
- # Use a custom base URL
374
- npx xqt gen-postman --base-url https://api.example.com
375
-
376
- # AI-enhanced generation with better assertions
377
- npx xqt gen-postman --ai
378
-
379
- # Schema-driven generation from OpenAPI spec (no AI needed)
380
- npx xqt gen-postman --spec knowledge/api-specs/api.yaml
381
-
382
- # Use custom knowledge folder
383
- npx xqt gen-postman --knowledge ./docs
384
- ```
385
-
386
- | Option | Description | Default |
387
- |---------------------|---------------------------------|-----------------|
388
- | `--base-url <url>` | Base URL for API requests | `{{baseUrl}}` |
389
- | `--ai` | Enable AI-enhanced assertions | No |
390
- | `--spec <path>` | OpenAPI spec for schema-driven generation | No |
391
- | `--knowledge <path>`| Custom knowledge folder path | `knowledge/` |
392
-
393
- **Output:** `collection.postman.json` in project root
394
-
395
- **Behavior:**
396
- - Only generates for tests with `type: "api"` (or unset, for backward compatibility)
397
- - Embeds JIRA keys from `xray-mapping.json` when available (e.g., `[APIEE-6933] Test Summary`)
398
- - Falls back to `test_id` if not yet pushed to Xray
399
- - Each test becomes a folder, each step becomes a request with:
400
- - Inferred HTTP method and endpoint from step data
401
- - Pre-request scripts with step context
402
- - Test scripts with assertions inferred from expected results
403
- - SCAFFOLD markers for manual enhancement
404
-
405
- **Important:** Generated collections are **starting points** — review and enhance assertions, environment variables, and edge cases before use.
406
-
407
- ---
408
-
409
- ### `xqt create-execution`
410
-
411
- Create a standalone Test Execution issue in JIRA.
412
-
413
- ```bash
414
- npx xqt create-execution --summary "Sprint 24 Regression" --description "Full API regression"
415
- npx xqt create-execution --summary "Feature XYZ" --issue QE-123
416
- ```
417
-
418
- | Option | Description | Required |
419
- |---------------------|--------------------------------------------|----------|
420
- | `--summary <text>` | Test Execution summary/title | Yes |
421
- | `--description <text>` | Description | No |
422
- | `--issue <key>` | Parent issue key to link the execution to | No |
423
-
424
- ---
425
-
426
- ### `xqt import-results`
427
-
428
- Import test results into Xray Cloud. Supports JUnit XML and Playwright JSON formats. Designed for CI — no human interaction.
429
-
430
- #### Format Comparison
431
-
432
- | Feature | JUnit XML | Playwright JSON |
433
- |---------|-----------|----------------|
434
- | **Update existing tests** | ❌ No - always creates new tests | ✅ Yes - via annotations |
435
- | **Test key mapping** | ❌ Not supported | ✅ `test.info().annotations` |
436
- | **Attachments/Evidence** | ❌ Not supported | ✅ Supported |
437
- | **Best for** | Newman, generic test runners | Playwright tests |
438
- | **Recommendation** | ⚠️ Use only for tools without test keys | **Recommended for Playwright** |
439
-
440
- #### JUnit XML (Newman, generic test runners)
441
-
442
- ⚠️ **Important:** JUnit XML **cannot update existing Xray tests** - it will always create new test cases. Only use this for Newman or test runners that don't support test keys.
443
-
444
- ```bash
445
- npx xqt import-results --file results.xml --testExecKey QE-123
446
- ```
447
-
448
- #### Playwright JSON (Recommended)
449
-
450
- ✅ **Updates existing tests** via annotations. Use this format to link results to your existing Xray test cases.
451
-
452
- ```bash
453
- # Generate JSON report in Playwright
454
- npx playwright test --reporter=json > playwright-results.json
455
-
456
- # Import to Xray (updates existing tests via annotations)
457
- npx xqt import-results --file playwright-results.json --testExecKey QE-123
458
-
459
- # Create new Test Execution automatically
460
- npx xqt import-results --file playwright-results.json --summary "Regression Suite"
461
- ```
462
-
463
- **Required test code:**
464
- ```typescript
465
- import { test, expect } from '@playwright/test';
466
-
467
- test('Verify API endpoint', async ({ request }) => {
468
- // THIS LINE links to existing Xray test
469
- test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
470
-
471
- // Your test code...
472
- });
473
- ```
474
-
475
- | Option | Description | Required |
476
- |-----------------------------|---------------------------------------------------|----------|
477
- | `--file <path>` | Path to results file (.xml or .json) | Yes |
478
- | `--testExecKey <key>` | Test Execution to link results to | No* |
479
- | `--summary <text>` | Summary for new Test Execution (if no testExecKey) | No |
480
- | `--description <text>` | Description for new Test Execution | No |
481
-
482
- \* If `--testExecKey` is omitted, a new Test Execution will be created automatically.
483
-
484
- **Mapping Playwright Tests to Xray:**
485
-
486
- To link Playwright test results to existing Xray test cases, **you must use annotations**:
487
-
488
- ```typescript
489
- // ✅ CORRECT - Updates existing test PROJ-123
490
- test('User login flow', async ({ page }) => {
491
- test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
492
- // ... test code
493
- });
494
-
495
- // ✅ Alternative - Include test key in title
496
- test('[PROJ-456] User registration validates email', async ({ page }) => {
497
- // ... test code
498
- });
499
-
500
- // ❌ WRONG - Creates new test every time
501
- test('User login flow', async ({ page }) => {
502
- // Missing annotation - will create duplicate test!
503
- // ... test code
504
- });
505
- ```
506
-
507
- **Without annotations:** A new test will be created in Xray with the test title as the summary (not recommended for existing tests).
508
-
509
- ---
510
-
511
- ### `xqt gen-pipeline`
512
-
513
- Generate an Azure Pipelines YAML template.
514
-
515
- ```bash
516
- npx xqt gen-pipeline
517
- npx xqt gen-pipeline --output ci/azure-pipelines.yml
518
- ```
519
-
520
- | Option | Description | Default |
521
- |---------------------|----------------------|------------------------|
522
- | `--output <path>` | Output file path | `azure-pipelines.yml` |
523
-
524
- The generated pipeline:
525
- - Installs Node.js and dependencies
526
- - Runs Newman against `collection.postman.json`
527
- - Calls `import-results` with environment variables
528
- - Publishes JUnit test results
529
-
530
- **Note:** `edit-json` and QE review logic are NOT in the CI pipeline. Those happen pre-commit.
531
-
532
- ---
533
-
534
- ### `xqt mcp-server`
535
-
536
- Start a Model Context Protocol server for GitHub Copilot agent-mode integration.
537
-
538
- ```bash
539
- # Start MCP server in stdio mode (for agent integration)
540
- npx xqt mcp-server
541
-
542
- # Start in HTTP mode for testing (optional)
543
- npx xqt mcp-server --port 3100
544
- ```
545
-
546
- | Option | Description | Default |
547
- |---------------------|----------------------|------------------------|
548
- | `--port <n>` | HTTP port (testing) | stdio mode |
549
-
550
- **MCP Tools Exposed:**
551
- - `generate_test_cases` — Generate tests from knowledge/ sources
552
- - `generate_postman` — Generate Postman collection with AI assertions
553
- - `analyze_knowledge` — List and summarize knowledge sources
554
- - `push_tests` Push test cases to Xray Cloud
555
- - `analyze_spec` — Parse OpenAPI spec and extract endpoints
556
-
557
- **Status:** 🚧 Scaffold ready AI provider connection not yet implemented. See [AI Setup](#ai-setup-optional) for implementation guidance.
558
-
559
- **Learn more:** [Model Context Protocol](https://modelcontextprotocol.io/)
560
-
561
- **Alternative:** `.vscode/xray-qe-participant/` contains a VS Code chat participant scaffold for GitHub Copilot integration.
562
-
563
- ---
564
-
565
- ### `xqt compare-openapi`
566
-
567
- Compare an OpenAPI spec against an approved QA snapshot and fail the pipeline if breaking changes are detected.
568
-
569
- ```bash
570
- # Basic comparison exits 1 on breaking changes
571
- npx xqt compare-openapi \
572
- --current ../api-repo/openapi.yaml \
573
- --snapshot ./openapi.snapshot.yaml
574
-
575
- # Save a JSON diff report as a pipeline artifact
576
- npx xqt compare-openapi \
577
- --current ../api-repo/openapi.yaml \
578
- --snapshot ./openapi.snapshot.yaml \
579
- --report openapi-diff-report.json
580
- ```
581
-
582
- | Option | Description | Required |
583
- |----------------------|------------------------------------------------------------------|----------|
584
- | `--current <path>` | Path to the current (live) OpenAPI spec | Yes |
585
- | `--snapshot <path>` | Path to the approved QA baseline snapshot | Yes |
586
- | `--report <path>` | Write full diff results to a JSON file (useful as CI artifact) | No |
587
-
588
- **Behaviour:**
589
- - Snapshot = approved QA contract baseline
590
- - Current = proposed/live spec
591
- - Exits `0` if no breaking changes; logs a count of non-breaking differences
592
- - Exits `1` on any breaking change and prints the full diff
593
- - If `--report` is specified and breaking changes are found, a JSON report is written
594
-
595
- **Example Azure Pipelines step (from your test repo pipeline):**
596
-
597
- ```yaml
598
- steps:
599
- - checkout: self
600
-
601
- - checkout: git://Project/portfolio-api
602
- path: api-repo
603
-
604
- - script: |
605
- npx xqt compare-openapi \
606
- --current api-repo/openapi.yaml \
607
- --snapshot openapi.snapshot.yaml \
608
- --report openapi-diff-report.json
609
- displayName: Compare OpenAPI Contracts
610
-
611
- - publish: openapi-diff-report.json
612
- artifact: openapi-diff
613
- condition: failed()
614
- ```
615
-
616
- **Governance model:**
617
- - Dev repo is never modified by QE
618
- - QE test repo pipeline checks out the API repo and enforces the contract
619
- - Breaking change → pipeline fails → dev must fix spec or raise a contract change review
620
- - QE then runs `update-snapshot` to promote the new baseline
621
-
622
- ---
623
-
624
- ### `xqt update-snapshot`
625
-
626
- Overwrite the QA snapshot baseline with the current spec. **Always a deliberate, manual action — never called automatically.**
627
-
628
- ```bash
629
- npx xqt update-snapshot \
630
- --current ../api-repo/openapi.yaml \
631
- --snapshot ./openapi.snapshot.yaml
632
- ```
633
-
634
- | Option | Description | Required |
635
- |----------------------|-----------------------------------------------------|----------|
636
- | `--current <path>` | Path to the current (live) OpenAPI spec to promote | Yes |
637
- | `--snapshot <path>` | Path to the snapshot file to overwrite | Yes |
638
-
639
- **Workflow:**
640
- 1. Dev raises a contract change review
641
- 2. QE approves the new contract
642
- 3. QE runs `update-snapshot` locally
643
- 4. QE raises a PR in the test repo with the updated snapshot
644
- 5. PR is reviewed and merged — new baseline is established
645
-
646
- ```bash
647
- # After merging the approved contract change:
648
- npx xqt update-snapshot \
649
- --current ../api-repo/openapi.yaml \
650
- --snapshot ./openapi.snapshot.yaml
651
-
652
- git add openapi.snapshot.yaml
653
- git commit -m "chore: promote OpenAPI snapshot to v2.5.0"
654
- git push
655
- ```
656
-
657
- ---
658
-
659
- ## Configuration
660
-
661
- ### Environment Variables (.env)
662
-
663
- Create a `.env` file in your project root (or copy `.env.example`):
664
-
665
- | Variable | Required | Description |
666
- |-------------------|----------|------------------------------------------------|
667
- | `XRAY_ID` | Yes | Xray Cloud API Client ID |
668
- | `XRAY_SECRET` | Yes | Xray Cloud API Client Secret |
669
- | `JIRA_PROJECT_KEY` | Yes | JIRA project key (e.g., `APIEE`, `QE`) |
670
- | `JIRA_URL` | Yes | JIRA instance URL (e.g., `https://your-domain.atlassian.net`) |
671
- | `JIRA_API_TOKEN` | Yes | JIRA API token |
672
- | `JIRA_EMAIL` | Yes | JIRA user email (must match Xray API Key owner) |
673
- | `XRAY_GRAPHQL_URL` | No | Region-specific GraphQL endpoint (default: US) |
674
-
675
- **Region-specific GraphQL endpoints:**
676
-
677
- | Region | URL |
678
- |--------|-----|
679
- | US (default) | `https://us.xray.cloud.getxray.app/api/v2/graphql` |
680
- | EU | `https://eu.xray.cloud.getxray.app/api/v2/graphql` |
681
- | AU | `https://au.xray.cloud.getxray.app/api/v2/graphql` |
682
-
683
- ### Project Config (.xrayrc)
684
-
685
- Optional JSON file for non-sensitive project settings:
686
-
687
- ```json
688
- {
689
- "testsPath": "tests.json",
690
- "mappingPath": "xray-mapping.json",
691
- "collectionPath": "collection.postman.json"
692
- }
693
- ```
694
-
695
- ---
696
-
697
- ## File Reference
698
-
699
- ### tests.json
700
-
701
- Source-of-truth for test definitions. Created by `init`, edited by `edit-json`, consumed by `push-tests` and `gen-postman`.
702
-
703
- ```json
704
- {
705
- "testExecution": {
706
- "summary": "Sprint 24 - Automated Regression Suite",
707
- "description": "API regression tests for Sprint 24 release"
708
- },
709
- "tests": [
710
- {
711
- "test_id": "TC-API-GET-001",
712
- "skip": false,
713
- "tags": ["regression", "smoke"],
714
- "xray": {
715
- "summary": "Verify GET /api/resource returns 200",
716
- "description": "Test that the API returns expected data.",
717
- "priority": "High",
718
- "labels": ["API", "GET", "Regression"],
719
- "steps": [
720
- {
721
- "action": "Send GET request to /api/resource/123",
722
- "data": "Method: GET, Headers: Authorization: Bearer {token}",
723
- "expected_result": "200 OK with resource object"
724
- }
725
- ]
726
- }
727
- }
728
- ]
729
- }
730
- ```
731
-
732
- **Fields:**
733
-
734
- | Field | Type | Required | Description |
735
- |-------|------|----------|-------------|
736
- | `test_id` | string | Yes | Unique identifier (alphanumeric, hyphens, underscores) |
737
- | `skip` | boolean | No | If `true`, excluded from `push-tests` |
738
- | `tags` | string[] | No | QE tags: `regression`, `smoke`, `edge`, `critical`, etc. |
739
- | `xray.summary` | string | Yes | JIRA issue title |
740
- | `xray.description` | string | No | JIRA issue description |
741
- | `xray.priority` | string | No | `Highest`, `High`, `Medium`, `Low`, `Lowest` |
742
- | `xray.labels` | string[] | No | JIRA labels |
743
- | `xray.steps[].action` | string | Yes | What action to perform |
744
- | `xray.steps[].data` | string | No | Input data / parameters |
745
- | `xray.steps[].expected_result` | string | Yes | Expected outcome |
746
-
747
- ### xray-mapping.json
748
-
749
- Maps `test_id` → JIRA issue `{ key, id }`. Generated by `push-tests`, used for idempotent updates.
750
-
751
- ```json
752
- {
753
- "TC-API-GET-001": { "key": "APIEE-6933", "id": "1865623" },
754
- "TC-API-POST-001": { "key": "APIEE-6934", "id": "1865627" },
755
- "_testexecution": { "key": "APIEE-6941", "id": "1865637" }
756
- }
757
- ```
758
-
759
- ---
760
-
761
- ## Multi-Company Test Format
762
-
763
- This toolkit pushes tests into **one JIRA project per run** (via `JIRA_PROJECT_KEY`). For multiple companies, use consistent naming and keep per-company configs and files.
764
-
765
- **Recommended conventions:**
766
- - `test_id`: Prefix with company + domain + sequence, e.g. `ACME-BILLING-PAYMENTS-001`
767
- - `xray.summary`: Start with company or product tag, e.g. `[ACME] Payments - create invoice`
768
- - `xray.labels`: Add `company:<slug>`, `system:<slug>`, `team:<slug>` for filtering
769
- - `testExecution.summary`: Include company + release/sprint, e.g. `ACME - Sprint 12 Regression`
770
-
771
- **Example snippet:**
772
-
773
- ```json
774
- {
775
- "testExecution": {
776
- "summary": "ACME - Sprint 12 Regression"
777
- },
778
- "tests": [
779
- {
780
- "test_id": "ACME-BILLING-PAYMENTS-001",
781
- "tags": ["regression", "smoke"],
782
- "xray": {
783
- "summary": "[ACME] Payments - create invoice",
784
- "labels": ["company:acme", "system:billing", "team:payments"],
785
- "steps": [
786
- {
787
- "action": "Send POST /payments/invoices",
788
- "expected_result": "201 Created"
789
- }
790
- ]
791
- }
792
- }
793
- ]
794
- }
795
- ```
796
-
797
- **Per-company setup options:**
798
- - **Separate folders (recommended):** run `xray-qe init` once per company and keep `tests.json`, `xray-mapping.json`, `.env`, and `.xrayrc` isolated.
799
- - **Shared repo:** use a company-specific `.env` and `.xrayrc` (swap before running). Example `.xrayrc`:
800
-
801
- ```json
802
- {
803
- "testsPath": "tests.acme.json",
804
- "mappingPath": "xray-mapping.acme.json",
805
- "collectionPath": "collection.acme.postman.json"
806
- }
807
- ```
808
-
809
- ---
810
-
811
- ## Jira/Xray Project Setup
812
-
813
- Use this checklist to align a new team's board and Xray configuration with the toolkit:
814
-
815
- 1. **Create a JIRA project** per company or business unit (Software or Service project).
816
- 2. **Enable Xray** for the project and confirm issue types: **Test** and **Test Execution** (optional: Test Plan).
817
- 3. **Configure screens/fields** to include Summary, Description, Priority, Labels, and Test Steps.
818
- 4. **Set permissions** so the API user can create/edit Test and Test Execution issues.
819
- 5. **Define components/labels** that match your naming conventions (company, system, team).
820
- 6. **Create a board** with a filter like `project = KEY AND issuetype in (Test, "Test Execution")` and use components/labels for swimlanes.
821
-
822
- ---
823
-
824
- ## QE Workflow
825
-
826
- ```
827
- ┌─────────────────────────────────────────────────────────────────────────┐
828
- │ QE WORKFLOW (LOCAL) │
829
- │ │
830
- │ 1. npx xqt init ← scaffold project + knowledge/ folder │
831
- │ 2. Configure .env ← credentials │
832
- │ 3. Populate knowledge/ ← add API specs, requirements, tickets │
833
- │ • knowledge/api-specs/ (OpenAPI, Swagger) │
834
- │ • knowledge/requirements/ (BRDs, logic docs) │
835
- │ • knowledge/tickets/ (JIRA exports, Confluence) │
836
- │ 4. npx xqt gen-tests --ai ← AI generates test cases from knowledge│
837
- │ 5. npx xqt edit-json ← QE REVIEW GATE (browser UI) │
838
- │ • Review AI-generated tests │
839
- │ • Add/edit/delete tests │
840
- │ • Tag tests (regression, critical, etc.) │
841
- │ • Mark skip/push per test │
842
- │ • Save & Exit │
843
- │ 6. npx xqt push-tests ← push to Xray Cloud │
844
- │ 7. npx xqt gen-postman --ai ← generate Postman collection │
845
- │ 8. git commit & push ← CI picks up from here │
846
- │ │
847
- ├─────────────────────────────────────────────────────────────────────────┤
848
- │ CI PIPELINE (AUTOMATED) │
849
- │ │
850
- │ 9. npm ci ← install deps │
851
- │ 10. newman run collection.postman.json --reporters junit │
852
- │ 11. npx xqt import-results --file results.xml --testExecKey QE-123│
853
- │ │
854
- ├─────────────────────────────────────────────────────────────────────────┤
855
- │ CONTRACT ENFORCEMENT (TEST REPO PIPELINE) │
856
- │ │
857
- │ 12. Checkout API repo │
858
- │ 13. npx xqt compare-openapi │
859
- │ --current api-repo/openapi.yaml │
860
- │ --snapshot openapi.snapshot.yaml │
861
- │ → Fails pipeline on breaking changes │
862
- │ │
863
- │ (When QE approves a contract change) │
864
- │ 14. npx xqt update-snapshot │
865
- │ --current api-repo/openapi.yaml │
866
- │ --snapshot openapi.snapshot.yaml │
867
- │ → Commit updated snapshot + raise PR │
868
- │ │
869
- └─────────────────────────────────────────────────────────────────────────┘
870
- ```
871
-
872
- ## Playwright Quick Start
873
-
874
- ### Complete Setup for Updating Existing Xray Tests
875
-
876
- This is the **recommended workflow** for teams using Playwright with existing Xray test cases.
877
-
878
- #### Step 1: Install Playwright (in your test repo)
879
-
880
- ```bash
881
- npm install --save-dev @playwright/test
882
- ```
883
-
884
- #### Step 2: Configure Playwright
885
-
886
- Create `playwright.config.ts` in your test repo:
887
-
888
- ```typescript
889
- import { defineConfig } from '@playwright/test';
890
-
891
- export default defineConfig({
892
- reporter: [
893
- ['html'], // For local viewing
894
- ['json', { outputFile: 'playwright-results.json' }], // For Xray import
895
- ],
896
-
897
- use: {
898
- baseURL: process.env.API_BASE_URL || 'https://your-api.com',
899
- extraHTTPHeaders: {
900
- 'Authorization': `Bearer ${process.env.API_TOKEN}`,
901
- 'X-API-Key': process.env.API_KEY || '',
902
- },
903
- },
904
- });
905
- ```
906
-
907
- #### Step 3: Write Tests with Xray Annotations
908
-
909
- ⚠️ **Critical:** Every test MUST have an annotation to update existing Xray tests.
910
-
911
- ```typescript
912
- import { test, expect } from '@playwright/test';
913
-
914
- test.describe('Regression Tests', () => {
915
-
916
- test('Verify API returns 200 for valid request', async ({ request }) => {
917
- // THIS LINE links to your existing Xray test
918
- test.info().annotations.push({ type: 'xray', description: 'APIEE-7131' });
919
-
920
- const response = await request.post('/api/verify', {
921
- data: { userId: '123', action: 'validate' }
922
- });
923
-
924
- // Attach response as evidence for Xray
925
- test.info().attach('response-evidence', {
926
- body: JSON.stringify({
927
- status: response.status(),
928
- headers: response.headers(),
929
- body: await response.json()
930
- }, null, 2),
931
- contentType: 'application/json'
932
- });
933
-
934
- expect(response.status()).toBe(200);
935
- });
936
-
937
- test('Verify API returns 400 for invalid data', async ({ request }) => {
938
- test.info().annotations.push({ type: 'xray', description: 'APIEE-7132' });
939
- // ... test implementation
940
- });
941
-
942
- });
943
- ```
944
-
945
- #### Step 4: Map All Your Tests
946
-
947
- Based on your `xray-mapping.json`, add annotations to each test:
948
-
949
- ```typescript
950
- // If your xray-mapping.json shows:
951
- // "TC-001": { "key": "APIEE-7131", "id": "1879092" }
952
- // "TC-002": { "key": "APIEE-7132", "id": "1879095" }
953
-
954
- test('Test Case 1', async ({ request }) => {
955
- test.info().annotations.push({ type: 'xray', description: 'APIEE-7131' });
956
- // ...
957
- });
958
-
959
- test('Test Case 2', async ({ request }) => {
960
- test.info().annotations.push({ type: 'xray', description: 'APIEE-7132' });
961
- // ...
962
- });
963
- ```
964
-
965
- #### Step 5: Run Locally
966
-
967
- ```bash
968
- # Run tests
969
- npx playwright test
970
-
971
- # View HTML report
972
- npx playwright show-report
973
-
974
- # Upload to Xray (updates existing tests)
975
- npx xqt import-results --file playwright-results.json --testExecKey APIEE-6811
976
- ```
977
-
978
- **What happens:**
979
- - ✅ Tests WITH annotations (`test.info().annotations.push(...)`) → Updates existing Xray tests
980
- - ⏭️ Tests WITHOUT annotations → **Automatically skipped** (won't create duplicates)
981
- - 📊 Summary shows: Passed, Failed, Skipped counts
982
- - 🔗 Direct link to view results in Xray
983
-
984
- **Verbose mode** (see exactly what's being uploaded):
985
- ```bash
986
- npx xqt import-results \
987
- --file playwright-results.json \
988
- --testExecKey APIEE-6811 \
989
- --verbose
990
- ```
991
-
992
- This saves `playwright-results-xray-debug.json` for inspection.
993
-
994
- #### Step 6: Configure CI/CD
995
-
996
- **Azure Pipelines:**
997
-
998
- ```yaml
999
- steps:
1000
- - task: NodeTool@0
1001
- inputs:
1002
- versionSpec: '18.x'
1003
-
1004
- - script: npm ci
1005
- displayName: 'Install dependencies'
1006
-
1007
- - script: npx playwright test
1008
- displayName: 'Run Playwright tests'
1009
- continueOnError: true
1010
-
1011
- - script: |
1012
- npx xqt import-results \
1013
- --file playwright-results.json \
1014
- --testExecKey APIEE-6811
1015
- displayName: 'Upload results to Xray'
1016
- env:
1017
- XRAY_ID: $(XRAY_ID)
1018
- XRAY_SECRET: $(XRAY_SECRET)
1019
- JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
1020
- JIRA_URL: $(JIRA_URL)
1021
- JIRA_API_TOKEN: $(JIRA_API_TOKEN)
1022
- JIRA_EMAIL: $(JIRA_EMAIL)
1023
- ```
1024
-
1025
- **GitHub Actions:**
1026
-
1027
- ```yaml
1028
- steps:
1029
- - uses: actions/setup-node@v3
1030
- with:
1031
- node-version: '18'
1032
-
1033
- - run: npm ci
1034
-
1035
- - run: npx playwright test
1036
-
1037
- - run: |
1038
- npx xqt import-results \
1039
- --file playwright-results.json \
1040
- --testExecKey APIEE-6811
1041
- env:
1042
- XRAY_ID: ${{ secrets.XRAY_ID }}
1043
- XRAY_SECRET: ${{ secrets.XRAY_SECRET }}
1044
- JIRA_PROJECT_KEY: ${{ secrets.JIRA_PROJECT_KEY }}
1045
- JIRA_URL: ${{ secrets.JIRA_URL }}
1046
- JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
1047
- JIRA_EMAIL: ${{ secrets.JIRA_EMAIL }}
1048
- ```
1049
-
1050
- ### Benefits
1051
-
1052
- ✅ **Updates existing tests** - No duplicate test creation
1053
- ✅ **Automatic mapping** - Via annotations in test code
1054
- ✅ **Evidence attachments** - Screenshots, responses, traces
1055
- ✅ **Detailed reporting** - Retries, worker info, error details
1056
- ✅ **CI/CD ready** - Standard workflow for automation
1057
- ✅ **Single source of truth** - Test code + Xray together
1058
-
1059
- ---
1060
-
1061
- ## Building Regression Packs
1062
-
1063
- Regression packs are curated test suites that verify your system still works after changes. The toolkit makes it easy to build and maintain regression packs using tags, AI-generated tests, and idempotent push.
1064
-
1065
- ### 1. Organize Knowledge Sources by Domain
1066
-
1067
- ```
1068
- knowledge/
1069
- ├── api-specs/
1070
- │ ├── auth-api.yaml ← Authentication domain
1071
- │ ├── users-api.yaml ← User management
1072
- │ └── payments-api.yaml ← Payment processing
1073
- ├── requirements/
1074
- │ ├── auth-flows.md
1075
- │ ├── user-roles.md
1076
- │ └── payment-validation.md
1077
- └── tickets/
1078
- ├── APIEE-123.json ← Login epic
1079
- ├── APIEE-456.json ← Payment refactor epic
1080
- └── confluence-sso.html
1081
- ```
1082
-
1083
- ### 2. Generate Tests by Domain
1084
-
1085
- ```bash
1086
- # Generate auth tests from auth spec
1087
- npx xqt gen-tests --ai --spec knowledge/api-specs/auth-api.yaml
1088
-
1089
- # Generate payment tests
1090
- npx xqt gen-tests --ai --spec knowledge/api-specs/payments-api.yaml
1091
-
1092
- # Generate from a specific ticket
1093
- npx xqt gen-tests --ai --ticket APIEE-123
1094
- ```
1095
-
1096
- ### 3. Tag Tests for Pack Categorization
1097
-
1098
- Use `edit-json` to assign tags:
1099
-
1100
- | Tag | Purpose |
1101
- |-----|---------|
1102
- | `regression` | Full regression pack — all core functionality |
1103
- | `smoke` | Smoke test pack — critical paths only |
1104
- | `critical` | Critical business flows (subset of regression) |
1105
- | `edge` | Edge case and error handling tests |
1106
- | `integration` | Multi-system integration tests |
1107
- | `security` | Security and auth tests |
1108
- | `performance` | Performance/load tests |
1109
-
1110
- **Example workflow:**
1111
- 1. Generate tests: `npx xqt gen-tests --ai`
1112
- 2. Open editor: `npx xqt edit-json`
1113
- 3. Add `regression` tag to all tests
1114
- 4. Add `smoke` tag to critical path tests
1115
- 5. Add `critical` tag to business-critical tests
1116
- 6. Save and push to Xray
1117
-
1118
- ### 4. Filter and Run Specific Packs
1119
-
1120
- **In tests.json:**
1121
- ```json
1122
- {
1123
- "tests": [
1124
- {
1125
- "test_id": "001",
1126
- "type": "api",
1127
- "tags": ["regression", "smoke", "critical"]
1128
- },
1129
- {
1130
- "test_id": "002",
1131
- "type": "api",
1132
- "tags": ["regression", "edge"]
1133
- },
1134
- {
1135
- "test_id": "003",
1136
- "type": "api",
1137
- "tags": ["regression"],
1138
- "skip": true
1139
- }
1140
- ]
1141
- }
1142
- ```
1143
-
1144
- **Filtering in edit-json:**
1145
- - Use the dropdown to filter by tag (e.g., show only `smoke` tests)
1146
- - Mark tests as `skip: true` to exclude from push/generation
1147
-
1148
- **CI pipeline filtering:**
1149
- - Generate smoke pack: filter `tests.json` to `tags.includes("smoke")` before `gen-postman`
1150
- - Generate regression pack: filter to `tags.includes("regression")` and `skip !== true`
1151
- - Schedule different packs on different cadences (smoke nightly, regression weekly)
1152
-
1153
- ### 5. Maintain Packs Over Sprints
1154
-
1155
- **Idempotent push** keeps Xray in sync as your pack evolves:
1156
-
1157
- | Sprint Change | Action | Result |
1158
- |---------------|--------|--------|
1159
- | API endpoint added | `gen-tests --ai --spec new-api.yaml` → `edit-json` → `push-tests` | New tests created in Xray |
1160
- | Test assertion updated | Edit in `edit-json` → `push-tests` | Existing test updated in Xray |
1161
- | Test deprecated | Mark `skip: true` in `edit-json` → `push-tests` | Test excluded from future runs |
1162
- | Requirements changed | Update `knowledge/requirements/` → `gen-tests --ai` | Regenerate affected tests |
1163
-
1164
- **Best practices:**
1165
- - ✅ Regenerate tests when specs change (toolkit updates existing tests)
1166
- - ✅ Use meaningful `test_id` values (`AUTH-LOGIN-001` instead of `001`)
1167
- - ✅ Commit `tests.json` and `xray-mapping.json` to source control
1168
- - ✅ Review AI-generated tests before pushing — they're scaffolds, not production-ready
1169
- - ✅ Keep `knowledge/` up to date with your latest specs and docs
1170
-
1171
- ### 6. Example: Sprint Regression Pack Workflow
1172
-
1173
- ```bash
1174
- # Sprint start: Generate tests from updated specs
1175
- npx xqt gen-tests --ai
1176
-
1177
- # QE reviews and tags tests
1178
- npx xqt edit-json
1179
- # → Tag new tests with "regression"
1180
- # → Mark experimental tests as "skip"
1181
- # → Verify all critical paths have "smoke" tag
1182
-
1183
- # Push to Xray (creates new, updates existing)
1184
- npx xqt push-tests
1185
-
1186
- # Generate Postman collection for CI
1187
- npx xqt gen-postman --ai
1188
-
1189
- # Commit regression pack to repo
1190
- git add tests.json xray-mapping.json collection.postman.json
1191
- git commit -m "Sprint 24 regression pack"
1192
-
1193
- # CI runs nightly
1194
- newman run collection.postman.json --folder "[smoke]"
1195
- newman run collection.postman.json # Full regression weekly
1196
- ```
1197
-
1198
- ---
1199
-
1200
- ## Working with Existing Xray Tests
1201
-
1202
- If your team already has test cases in Xray Cloud that were created manually or by another tool, you can set up this toolkit to manage and update those existing tests.
1203
-
1204
- ### Step 1: Query Existing Tests from Xray
1205
-
1206
- Use the Xray GraphQL API to fetch your existing tests. Here's a script to generate the mapping file:
1207
-
1208
- **fetch-existing-tests.js:**
1209
- ```javascript
1210
- import { authenticate, loadConfig } from "@msalaam/xray-qe-toolkit";
1211
- import fs from "fs";
1212
-
1213
- const cfg = loadConfig();
1214
- const token = await authenticate(cfg);
1215
-
1216
- // GraphQL query to fetch all tests in your project
1217
- const query = `
1218
- query {
1219
- getTests(jql: "project = ${cfg.jiraProjectKey} AND issuetype = Test", limit: 1000) {
1220
- total
1221
- results {
1222
- issueId
1223
- jira(fields: ["key", "summary", "description", "priority", "labels"])
1224
- }
1225
- }
1226
- }
1227
- `;
1228
-
1229
- const response = await fetch(cfg.xrayGraphQLUrl || "https://us.xray.cloud.getxray.app/api/v2/graphql", {
1230
- method: "POST",
1231
- headers: {
1232
- "Content-Type": "application/json",
1233
- Authorization: `Bearer ${token}`,
1234
- },
1235
- body: JSON.stringify({ query }),
1236
- });
1237
-
1238
- const data = await response.json();
1239
- const tests = data.data.getTests.results;
1240
-
1241
- // Generate xray-mapping.json
1242
- const mapping = {};
1243
- tests.forEach((test) => {
1244
- const testId = test.jira.key; // Use JIRA key as test_id initially
1245
- mapping[testId] = {
1246
- key: test.jira.key,
1247
- id: test.issueId,
1248
- };
1249
- });
1250
-
1251
- fs.writeFileSync("xray-mapping.json", JSON.stringify(mapping, null, 2));
1252
- console.log(`✓ Created xray-mapping.json with ${tests.length} tests`);
1253
-
1254
- // Generate tests.json scaffold
1255
- const testsJson = {
1256
- testExecution: {
1257
- summary: "Existing Tests - Managed by Toolkit",
1258
- description: "Imported from existing Xray tests",
1259
- },
1260
- tests: tests.map((test) => ({
1261
- test_id: test.jira.key,
1262
- skip: false,
1263
- tags: [],
1264
- xray: {
1265
- summary: test.jira.summary,
1266
- description: test.jira.description || "",
1267
- priority: test.jira.priority?.name || "Medium",
1268
- labels: test.jira.labels || [],
1269
- steps: [
1270
- // You'll need to fetch test steps separately via another API call
1271
- {
1272
- action: "PLACEHOLDER - Edit this in npx xqt edit-json",
1273
- data: "",
1274
- expected_result: "PLACEHOLDER",
1275
- },
1276
- ],
1277
- },
1278
- })),
1279
- };
1280
-
1281
- fs.writeFileSync("tests.json", JSON.stringify(testsJson, null, 2));
1282
- console.log(`✓ Created tests.json with ${tests.length} test scaffolds`);
1283
- console.log("\nNext steps:");
1284
- console.log("1. Run 'npx xqt edit-json' to review and complete test steps");
1285
- console.log("2. Run 'npx xqt push-tests' to update tests in Xray");
1286
- ```
1287
-
1288
- ### Step 2: Run the Script
1289
-
1290
- ```bash
1291
- # Save the script above as fetch-existing-tests.js
1292
- node fetch-existing-tests.js
1293
- ```
1294
-
1295
- This generates:
1296
- - `xray-mapping.json` — maps your existing JIRA test keys to their IDs
1297
- - `tests.json` — scaffold with summaries, descriptions, priorities, labels
1298
-
1299
- ### Step 3: Complete Test Steps
1300
-
1301
- The script can't fetch test steps automatically (requires additional API calls). Complete them in the editor:
1302
-
1303
- ```bash
1304
- npx xqt edit-json
1305
- # Edit each test to add proper steps
1306
- # Save when done
1307
- ```
1308
-
1309
- ### Step 4: Update Existing Tests
1310
-
1311
- Now you can update your existing Xray tests:
1312
-
1313
- ```bash
1314
- npx xqt push-tests
1315
- ```
1316
-
1317
- Because the tests are in `xray-mapping.json`, they'll be **updated** (not duplicated).
1318
-
1319
- ### Alternative: Manual Mapping Creation
1320
-
1321
- If you have a small number of tests, create the mapping manually:
1322
-
1323
- **xray-mapping.json:**
1324
- ```json
1325
- {
1326
- "APIEE-6933": { "key": "APIEE-6933", "id": "1865623" },
1327
- "APIEE-6934": { "key": "APIEE-6934", "id": "1865627" },
1328
- "APIEE-6935": { "key": "APIEE-6935", "id": "1865628" }
1329
- }
1330
- ```
1331
-
1332
- **tests.json:**
1333
- ```json
1334
- {
1335
- "tests": [
1336
- {
1337
- "test_id": "APIEE-6933",
1338
- "xray": {
1339
- "summary": "Test User Login",
1340
- "steps": [
1341
- { "action": "...", "expected_result": "..." }
1342
- ]
1343
- }
1344
- }
1345
- ]
1346
- }
1347
- ```
1348
-
1349
- Then run `npx xqt push-tests` to update.
1350
-
1351
- ### Future Enhancement: Pull Command
1352
-
1353
- A `pull-tests` command to automatically fetch and sync existing tests is planned:
1354
-
1355
- ```bash
1356
- # Future feature (not yet implemented)
1357
- npx xqt pull-tests --project APIEE
1358
- npx xqt pull-tests --jql "project = APIEE AND labels = regression"
1359
- ```
1360
-
1361
- This would automatically generate both `tests.json` and `xray-mapping.json` from Xray.
1362
-
1363
- **Want this feature?** [Open an issue on GitHub](https://github.com/Muhammad-Salaam_omit/xray-qe-toolkit/issues) or contribute via PR.
1364
-
1365
- ---
1366
-
1367
- ## Idempotent Push
1368
-
1369
- `push-tests` checks `xray-mapping.json` before each operation:
1370
-
1371
- | Scenario | Action |
1372
- |----------|--------|
1373
- | `test_id` not in mapping | **Create** new JIRA Test issue + steps |
1374
- | `test_id` in mapping | **Update** existing issue fields + replace steps |
1375
- | `skip: true` | **Skip** entirely |
1376
-
1377
- This means you can safely run `push-tests` multiple times without creating duplicates.
1378
-
1379
- ---
1380
-
1381
- ## Programmatic API
1382
-
1383
- All library functions are importable for custom scripts:
1384
-
1385
- ```javascript
1386
- import {
1387
- loadConfig,
1388
- validateConfig,
1389
- authenticate,
1390
- createIssue,
1391
- buildAndPush,
1392
- generatePostmanCollection,
1393
- logger,
1394
- } from "@msalaam/xray-qe-toolkit";
1395
-
1396
- const cfg = loadConfig();
1397
- validateConfig(cfg);
1398
-
1399
- const token = await authenticate(cfg);
1400
- // ... use any exported function
1401
- ```
1402
-
1403
- ---
1404
-
1405
- ## Troubleshooting
1406
-
1407
- ### "disallowed to impersonate" / "no valid active user exists"
1408
-
1409
- Your `JIRA_EMAIL` doesn't match the Xray API Key owner.
1410
-
1411
- **Fix:**
1412
- 1. Ensure `JIRA_EMAIL` matches the email of the user who created the Xray API Key
1413
- 2. Verify the user has an active Xray license
1414
- 3. Regenerate the Xray API Key with the same user as `JIRA_API_TOKEN`
1415
-
1416
- ### "issueId provided is not valid" (transient)
1417
-
1418
- Xray's GraphQL API needs time to index newly created JIRA issues. The toolkit automatically retries with exponential backoff (2s → 4s → 8s → 16s → 32s).
1419
-
1420
- If it still fails after 5 retries, wait a minute and try again.
1421
-
1422
- ### Rate limiting / 429 errors
1423
-
1424
- The toolkit includes a 300ms delay between API calls. If you still hit rate limits, wait and retry. For very large test suites (100+), consider splitting across multiple runs.
1425
-
1426
- ### Browser doesn't open for `edit-json`
1427
-
1428
- In headless environments, copy the URL printed in the terminal and open it manually.
1429
-
1430
- ---
1431
-
1432
- ## License
1433
-
1434
- See LICENSE.
1
+ # @msalaam/xray-qe-toolkit
2
+
3
+ > QE toolkit for Xray Cloud — manage test definitions, sync to Jira/Xray, import Playwright results.
4
+
5
+ [![npm version](https://badge.fury.io/js/%40msalaam%2Fxray-qe-toolkit.svg)](https://www.npmjs.com/package/@msalaam/xray-qe-toolkit)
6
+ [![Node.js](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)
7
+
8
+ ---
9
+
10
+ ## Table of Contents
11
+
12
+ - [Overview](#overview)
13
+ - [How It Works](#how-it-works)
14
+ - [Prerequisites](#prerequisites)
15
+ - [Installation](#installation)
16
+ - [Quick Start](#quick-start)
17
+ - [CLI Commands](#cli-commands)
18
+ - [Configuration](#configuration)
19
+ - [tests.json Reference](#testsjson-reference)
20
+ - [Test Sets, Plans & Executions](#test-sets-plans--executions)
21
+ - [Playwright Integration](#playwright-integration)
22
+ - [xray-mapping.json](#xray-mappingjson)
23
+ - [CI/CD Integration](#cicd-integration)
24
+ - [Programmatic API](#programmatic-api)
25
+ - [Troubleshooting](#troubleshooting)
26
+
27
+ ---
28
+
29
+ ## Overview
30
+
31
+ `@msalaam/xray-qe-toolkit` (CLI: `xqt`) manages the boundary between your local test definitions (`tests.json`) and Xray Cloud. It does not generate Playwright tests — those are created by a separate Copilot skill after Xray issue keys are known.
32
+
33
+ **What it does:**
34
+
35
+ - **`tests.json`** — single source of truth for test metadata (synced to Xray)
36
+ - **Idempotent push** — creates new tests and Test Sets, updates existing ones, never duplicates
37
+ - **Test Set management** groups tests by feature, persistent across sprints
38
+ - **Test Plan management** — create per-sprint plans, link Test Sets
39
+ - **Test Executions** — auto-created per CI run, tagged by environment, linked to plan
40
+ - **Playwright result import** — JSON reporter output → Xray with step-level results
41
+ - **Xray folder sync** — organise the Test Repository from `folder` fields in tests.json
42
+ - **Schema validation** — validate tests.json as a CI gate before pushing
43
+ - **Visual editor** — browser-based editor for reviewing/editing tests.json
44
+
45
+ > For the full spec-driven QE process — how `openapi.yaml` + `business-rules.yaml` become `tests.json`,
46
+ > how Playwright tests are created, and how sprints are managed see
47
+ > **[SPEC-DRIVEN-APPROACH.md](SPEC-DRIVEN-APPROACH.md)** (scaffolded into every project by `xqt init`).
48
+
49
+ ---
50
+
51
+ ## How It Works
52
+
53
+ ```
54
+ openapi.yaml + business-rules.yaml
55
+ (AI agent generates tests.json)
56
+ tests.jsonstructured test definitions
57
+
58
+ xqt validate ← schema check
59
+
60
+ xqt push ← creates/updates Tests + Test Sets in Xray
61
+
62
+ xray-mapping.json auto-populated with Jira issue keys
63
+
64
+ Playwright API tests ← created by Copilot (reads tests.json + xray-mapping.json)
65
+
66
+ Sprint starts ← xqt plan → link Test Sets to Test Plan
67
+
68
+ CI runs tests xqt import Test Execution in Xray
69
+ ```
70
+
71
+ **Key design decisions:**
72
+ - **Test Sets are persistent** — tests live in Test Sets grouped by feature
73
+ - **Test Plans are per-sprint** — create one per sprint, link relevant Test Sets
74
+ - **Test Executions are ephemeral** every CI run creates a fresh execution
75
+ - **Environments are labels** `IOP-DEV`, `IOP-QA`, `IOP-PROD` tag each execution
76
+ - **tests.json is the source of truth** never edit test metadata directly in JIRA
77
+ - **xray-mapping.json is auto-generated** maps local `testId` → JIRA key; never edit manually
78
+ - **Playwright tests come after push** created only once `xray-mapping.json` has real Jira keys
79
+
80
+ ---
81
+
82
+ ## Prerequisites
83
+
84
+ - **Node.js** >= 18.0.0
85
+ - **Xray Cloud** API credentials (Client ID + Client Secret) — from **Apps → Xray → API Keys**
86
+ - **JIRA Cloud** API Token — from [https://id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens)
87
+
88
+ ---
89
+
90
+ ## Installation
91
+
92
+ ```bash
93
+ # As a dev dependency (recommended)
94
+ npm install --save-dev @msalaam/xray-qe-toolkit
95
+
96
+ # Globally
97
+ npm install -g @msalaam/xray-qe-toolkit
98
+ ```
99
+
100
+ ---
101
+
102
+ ## Quick Start
103
+
104
+ ```bash
105
+ # 1. Scaffold a new QA project
106
+ npx xqt init
107
+
108
+ # 2. Fill in credentials
109
+ cp .env.example .env
110
+ # Edit .env with XRAY_ID, XRAY_SECRET, JIRA_* values
111
+
112
+ # 3. Place your API spec and business rules in resources/
113
+ # resources/api-specs/openapi.yaml
114
+ # resources/business-rules.yaml
115
+
116
+ # 4. Generate tests.json from openapi.yaml + business-rules.yaml
117
+ # (done by AI agent / Copilot skill — reads both files, produces tests.json)
118
+
119
+ # 5. Validate and push to Xray
120
+ npx xqt validate
121
+ npx xqt push
122
+ # → Tests + Test Sets created in Jira
123
+ # → xray-mapping.json populated with issue keys
124
+
125
+ # 6. Create Playwright tests (done by Copilot skill — reads tests.json + xray-mapping.json)
126
+
127
+ # 7. When entering a sprint, create a Test Plan and link Test Sets
128
+ npx xqt plan --summary "Sprint 12 My Service Regression"
129
+
130
+ # 8. Run tests and import results
131
+ npx playwright test
132
+ npx xqt import --file test-results/results.json --env IOP-QA
133
+ ```
134
+
135
+ ---
136
+
137
+ ## CLI Commands
138
+
139
+ All commands accept `--verbose` for debug output and `--env <path>` for a custom .env file path.
140
+
141
+ ---
142
+
143
+ ### `xqt init`
144
+
145
+ Scaffold a QA project in the current directory. Idempotent — never overwrites existing files.
146
+
147
+ Creates: `resources/` (API spec + business rules), `tests/` (models/services/resources/specs), `playwright.config.ts`, `tests.json`, `xray-mapping.json`, `.xrayrc`, `.env.example`, `XQT-GUIDE.md` (xqt command reference), `SPEC-DRIVEN-APPROACH.md` (QE process guide).
148
+
149
+ ```bash
150
+ npx xqt init
151
+ ```
152
+
153
+ ---
154
+
155
+ ### `xqt edit`
156
+
157
+ Launch a browser-based visual editor for `tests.json`. Use this to review, add, or update test definitions.
158
+
159
+ ```bash
160
+ npx xqt edit
161
+ npx xqt edit --port 3000
162
+ ```
163
+
164
+ | Flag | Description | Default |
165
+ |---|---|---|
166
+ | `--port <n>` | Port for local editor server | Random |
167
+
168
+ ---
169
+
170
+ ### `xqt plan`
171
+
172
+ Create a new Xray Test Plan in JIRA. Automatically saves the key to `.xrayrc` and `tests.json`.
173
+ `create-plan` is accepted as a legacy alias.
174
+
175
+ ```bash
176
+ npx xqt plan --summary "My Service v2 Regression"
177
+ npx xqt plan --summary "Sprint 12 Smoke Tests" --version "2024.12"
178
+ ```
179
+
180
+ | Flag | Description |
181
+ |---|---|
182
+ | `--summary <text>` | Test Plan title **(required)** |
183
+ | `--version <ver>` | Fix version to associate |
184
+ | `--label <labels>` | Comma-separated JIRA labels |
185
+
186
+ ---
187
+
188
+ ### `xqt push`
189
+
190
+ Create or update tests in Xray Cloud, then sync Test Plan membership and the Xray folder structure.
191
+ Optionally create a Test Execution linked to the plan in a single command.
192
+ `push-tests` is accepted as a legacy alias.
193
+
194
+ ```bash
195
+ npx xqt push
196
+ npx xqt push --plan APIEE-1234
197
+ npx xqt push --plan APIEE-1234 --create-execution --execution-env IOP-QA
198
+ ```
199
+
200
+ | Flag | Description |
201
+ |---|---|
202
+ | `--plan <key>` | Test Plan key (overrides `.xrayrc testPlanKey`) |
203
+ | `--create-execution` | Create a Test Execution linked to the Test Plan after pushing |
204
+ | `--execution-env <label>` | Environment label for the created execution (e.g. `IOP-QA`) |
205
+ | `--execution-summary <text>` | Custom summary for the created execution |
206
+
207
+ **Behaviour:**
208
+ - Tests in `xray-mapping.json` → **updated** in Xray
209
+ - Tests not in mapping → **created** in Xray, mapping entry added
210
+ - Tests with `"skip": true` → excluded entirely
211
+ - **Test Sets auto-created** from the `testSet` field and tests added to them (see below)
212
+ - Xray folder structure synced from `folder` fields
213
+ - With `--create-execution`, a new Test Execution is created and linked to the Test Plan
214
+
215
+ ---
216
+
217
+ ### `xqt pull`
218
+
219
+ Pull test definitions from Xray Cloud and merge them into `tests.json`. Useful for onboarding to an existing Xray project.
220
+ `pull-tests` is accepted as a legacy alias.
221
+
222
+ ```bash
223
+ npx xqt pull --plan APIEE-1234
224
+ npx xqt pull --project APIEE --limit 200
225
+ ```
226
+
227
+ | Flag | Description |
228
+ |---|---|
229
+ | `--plan <key>` | Fetch tests from a specific Test Plan |
230
+ | `--project <key>` | Fetch all tests in a JIRA project |
231
+ | `--limit <n>` | Max tests to fetch (default: 100) |
232
+
233
+ ---
234
+
235
+ ### `xqt exec`
236
+
237
+ Pre-create a Test Execution with a specific set of tests before running them.
238
+ Use `--quiet` to capture the key in CI for use with `import --exec`.
239
+ `create-execution` is accepted as a legacy alias.
240
+
241
+ ```bash
242
+ # Standard output
243
+ npx xqt exec --env IOP-QA
244
+
245
+ # CI mode prints ONLY the key
246
+ EXEC_KEY=$(npx xqt exec --env IOP-QA --quiet)
247
+
248
+ # Specific test selection
249
+ npx xqt exec --env IOP-QA --plan APIEE-1234 --tests TC-001,TC-002
250
+ ```
251
+
252
+ | Flag | Description |
253
+ |---|---|
254
+ | `--env <label>` | Environment label (e.g. `IOP-QA`) |
255
+ | `--plan <key>` | Test Plan to link to (overrides `.xrayrc`) |
256
+ | `--tests <ids>` | Comma-separated testIds or JIRA keys (default: all mapped) |
257
+ | `--summary <text>` | Custom execution title |
258
+ | `--quiet` | Print only the execution key |
259
+
260
+ > `xqt import` creates an execution automatically — this command is only needed when pre-selecting a specific subset of tests.
261
+
262
+ ---
263
+
264
+ ### `xqt import`
265
+
266
+ Import test execution results into Xray Cloud.
267
+ `import-results` is accepted as a legacy alias.
268
+
269
+ - **Without `--exec`** — creates a **new** Test Execution linked to the Test Plan
270
+ - **With `--exec`** — imports results INTO a pre-created execution
271
+
272
+ Supported: Playwright JSON (`.json`) and JUnit XML (`.xml`).
273
+
274
+ ```bash
275
+ # Auto-create execution (standard CI)
276
+ npx xqt import --file test-results/results.json --env IOP-QA
277
+
278
+ # Import into a pre-created execution
279
+ npx xqt import --file test-results/results.json --exec APIEE-9876
280
+
281
+ # JUnit XML
282
+ npx xqt import --file test-results/results.xml --env IOP-PROD
283
+
284
+ # Full options
285
+ npx xqt import \
286
+ --file test-results/results.json \
287
+ --env IOP-QA \
288
+ --plan APIEE-1234 \
289
+ --version "2024.12" \
290
+ --revision "a1b2c3d4"
291
+ ```
292
+
293
+ | Flag | Description |
294
+ |---|---|
295
+ | `--file <path>` | Path to results file **(required)** |
296
+ | `--env <label>` | Environment label (default: `defaultEnvironment` from `.xrayrc`) |
297
+ | `--plan <key>` | Test Plan key (overrides `.xrayrc`; used when no `--exec`) |
298
+ | `--exec <key>` | Import INTO an existing execution (from `xqt exec`) |
299
+ | `--version <ver>` | Fix version / release label |
300
+ | `--revision <rev>` | Build number or git SHA |
301
+ | `--summary <text>` | Custom execution summary |
302
+
303
+ ---
304
+
305
+ ### `xqt sync`
306
+
307
+ Sync the Xray Test Repository folder structure from `folder` fields in `tests.json` without touching test content.
308
+ `sync-folders` is accepted as a legacy alias.
309
+
310
+ ```bash
311
+ npx xqt sync
312
+ ```
313
+
314
+ ---
315
+
316
+ ### `xqt status`
317
+
318
+ Show local and remote project status: tests.json counts, mapping coverage, Test Plan info.
319
+
320
+ ```bash
321
+ npx xqt status
322
+ npx xqt status --plan APIEE-1234
323
+ ```
324
+
325
+ ---
326
+
327
+ ### `xqt validate`
328
+
329
+ Validate `tests.json` against its schema. Exits with code 1 on errors — use as a CI gate before `push-tests`.
330
+
331
+ ```bash
332
+ npx xqt validate
333
+ npx xqt validate --file path/to/tests.json
334
+ ```
335
+
336
+ | Flag | Description |
337
+ |---|---|
338
+ | `--file <path>` | Override path to tests.json |
339
+
340
+ ---
341
+
342
+ ### `xqt pipeline`
343
+
344
+ Generate an Azure Pipelines YAML template for the current project.
345
+ `gen-pipeline` is accepted as a legacy alias.
346
+
347
+ ```bash
348
+ npx xqt pipeline
349
+ npx xqt pipeline --output .azure/ci.yml
350
+ ```
351
+
352
+ ---
353
+
354
+ ## Configuration
355
+
356
+ ### Environment Variables (.env)
357
+
358
+ Copy `.env.example` to `.env` and fill in credentials. **Never commit `.env`.**
359
+
360
+ ```env
361
+ # ── Xray Cloud ─────────────────────────────────────────────────
362
+ XRAY_ID=your_xray_cloud_client_id
363
+ XRAY_SECRET=your_xray_cloud_client_secret
364
+
365
+ # ── JIRA ───────────────────────────────────────────────────────
366
+ JIRA_PROJECT_KEY=APIEE
367
+ JIRA_URL=https://your-org.atlassian.net
368
+ JIRA_API_TOKEN=your_jira_api_token
369
+ JIRA_EMAIL=your.email@company.com
370
+
371
+ # ── Optional ───────────────────────────────────────────────────
372
+ # XRAY_REGION=us # "us" (default) or "eu"
373
+ # XQT_TEST_PLAN_KEY=APIEE-1234 # default Test Plan
374
+ # XQT_DEFAULT_ENV=IOP-DEV # default environment label
375
+ ```
376
+
377
+ | Variable | Required | Description |
378
+ |---|---|---|
379
+ | `XRAY_ID` | Yes | Xray Cloud Client ID |
380
+ | `XRAY_SECRET` | Yes | Xray Cloud Client Secret |
381
+ | `JIRA_PROJECT_KEY` | Yes | JIRA project key |
382
+ | `JIRA_URL` | Yes | JIRA instance URL |
383
+ | `JIRA_API_TOKEN` | Yes | JIRA API token |
384
+ | `JIRA_EMAIL` | Yes | JIRA user email |
385
+ | `XRAY_REGION` | No | `us` (default) or `eu` |
386
+ | `XQT_TEST_PLAN_KEY` | No | Default Test Plan key |
387
+ | `XQT_DEFAULT_ENV` | No | Default environment label |
388
+
389
+ ---
390
+
391
+ ### Project Config (.xrayrc)
392
+
393
+ `.xrayrc` (JSON) at the project root. Created by `xqt init`, updated by `xqt create-plan`.
394
+
395
+ ```json
396
+ {
397
+ "testsPath": "tests.json",
398
+ "mappingPath": "xray-mapping.json",
399
+ "testPlanKey": "APIEE-1234",
400
+ "defaultEnvironment": "IOP-DEV",
401
+ "environments": ["IOP-DEV", "IOP-QA", "IOP-PROD"],
402
+ "folderRoot": "/MyService",
403
+ "xrayRegion": "us"
404
+ }
405
+ ```
406
+
407
+ | Field | Description |
408
+ |---|---|
409
+ | `testPlanKey` | Default Test Plan key for push-tests and import-results |
410
+ | `defaultEnvironment` | Fallback environment when `--env` is not provided |
411
+ | `environments` | Allowed environment labels |
412
+ | `folderRoot` | Base folder path in Xray Test Repository |
413
+ | `xrayRegion` | `us` (default) or `eu` |
414
+
415
+ ---
416
+
417
+ ## tests.json Reference
418
+
419
+ `tests.json` is the canonical list of test cases. It is generated from `openapi.yaml` + `business-rules.yaml` by an AI agent, then consumed by `xqt push-tests` to sync with Xray Cloud.
420
+
421
+ ```json
422
+ {
423
+ "testPlan": {
424
+ "key": "APIEE-1234",
425
+ "summary": "Sprint 12 — My Service API Regression"
426
+ },
427
+ "tests": [
428
+ {
429
+ "test_id": "TC-MYSVC-BR-001",
430
+ "type": "api",
431
+ "skip": false,
432
+ "tags": ["smoke", "regression"],
433
+ "folder": "/MyService/HealthCheck/Validation",
434
+ "testSet": "Health Check",
435
+ "requirementKeys": [],
436
+ "xray": {
437
+ "summary": "Service returns 200 OK when healthy",
438
+ "description": "Given Service is running, when GET /health is called, then Response is 200",
439
+ "testType": "Generic",
440
+ "definition": "Automated Playwright API test: GET /health — Service returns 200 OK when healthy",
441
+ "priority": "High",
442
+ "labels": ["smoke", "regression"]
443
+ }
444
+ }
445
+ ]
446
+ }
447
+ ```
448
+
449
+ ### Field Reference
450
+
451
+ | Field | Type | Required | Description |
452
+ |---|---|---|---|
453
+ | `test_id` | string | Yes | Stable local ID — maps to the Xray key in `xray-mapping.json` |
454
+ | `type` | string | — | `api` / `ui` / `e2e` (informational) |
455
+ | `skip` | boolean | — | `true` to exclude from `push-tests` |
456
+ | `tags` | string[] | | QE tags for categorisation and filtering. Allowed: `regression`, `smoke`, `edge`, `critical`, `integration`, `e2e`, `security`, `performance`, `contract`, `functional`, `negative`, `positive`, `boundary`, `acceptance`, `sanity`, `data-driven`, `exploratory`, `accessibility` |
457
+ | `folder` | string | | Xray repository folder path — must start with `/` |
458
+ | `testSet` | string | — | Test Set name in Jira — tests are grouped into Test Sets by this value |
459
+ | `requirementKeys` | string[] | — | JIRA keys this test covers (creates traceability links) |
460
+ | `xray.summary` | string | Yes | Test case title (JIRA issue summary) |
461
+ | `xray.description` | string | — | Detailed description |
462
+ | `xray.testType` | string | — | `Generic` (default for automated) / `Manual` / `Cucumber` |
463
+ | `xray.definition` | string | — | Generic test definition field in Xray |
464
+ | `xray.priority` | string | — | `Highest` / `High` / `Medium` / `Low` / `Lowest` |
465
+ | `xray.labels` | string[] | | JIRA labels on the test issue |
466
+
467
+ ---
468
+
469
+ ## Test Sets, Plans & Executions
470
+
471
+ ### Test Sets (persistent groupings)
472
+
473
+ Tests live in **Test Sets** in Jira. Each Test Set groups related tests by feature or area — they persist across every sprint.
474
+
475
+ Set the `testSet` field on each test in `tests.json`:
476
+
477
+ ```json
478
+ { "test_id": "TC-001", "testSet": "Client Lookup", ... }
479
+ { "test_id": "TC-002", "testSet": "Client Lookup", ... }
480
+ { "test_id": "TC-003", "testSet": "Health Check", ... }
481
+ ```
482
+
483
+ When you run `xqt push`:
484
+ 1. Tests are created/updated in Xray
485
+ 2. For each unique `testSet` value, the Test Set JIRA issue is **created automatically** if it doesn't exist yet
486
+ 3. Tests are added to their Test Set (idempotent — Xray ignores tests already in the set)
487
+ 4. Test Set → JIRA key mappings (e.g. `"Client Lookup" → APIEE-99`) are stored in `xray-mapping.json` under `_testSets`
488
+
489
+ ```json
490
+ // xray-mapping.json (auto-managed)
491
+ {
492
+ "_testSets": {
493
+ "Client Lookup": { "key": "APIEE-99", "id": "123456" },
494
+ "Health Check": { "key": "APIEE-100", "id": "123457" }
495
+ },
496
+ "TC-001": { "key": "APIEE-200", "id": "234001" },
497
+ ...
498
+ }
499
+ ```
500
+
501
+ Test Sets are **not deleted** when tests are removed from `tests.json`. They persist in Jira as a record of all tests that have ever existed for that feature.
502
+
503
+ ### Test Plans (per-sprint)
504
+
505
+ A Test Plan scopes testing work for a specific sprint or release.
506
+
507
+ - Create with `xqt plan --summary "Sprint 12 My Service"`
508
+ - Link relevant Test Sets to the Test Plan in Jira (manual step in Jira UI)
509
+ - Key stored in `.xrayrc` (`testPlanKey`)
510
+ - `xqt import` links executions to the active plan
511
+
512
+ ### Test Executions (ephemeral per run)
513
+
514
+ A Test Execution represents a single CI run.
515
+
516
+ - **Auto-created** by `xqt import` — one new execution per run
517
+ - **Pre-created** by `xqt exec` for controlled test selection
518
+ - Tagged with environment labels (`IOP-DEV`, `IOP-QA`, `IOP-PROD`)
519
+ - Linked to the Test Plan automatically
520
+
521
+ ### Summary
522
+
523
+ | Concept | Lifecycle | Purpose |
524
+ |---------|-----------|---------|
525
+ | **Test Set** | Permanent | Groups tests by feature — where tests live |
526
+ | **Test Plan** | Per-sprint | Scopes testing for a sprint — links Test Sets |
527
+ | **Test Execution** | Per-CI-run | Records pass/fail for one run |
528
+
529
+ ### Three execution modes
530
+
531
+ **Mode A — Auto (standard CI):**
532
+ ```bash
533
+ npx xqt import --file results.json --env IOP-QA
534
+ ```
535
+
536
+ **Mode B Pre-create (controlled test selection):**
537
+ ```bash
538
+ EXEC_KEY=$(npx xqt exec --env IOP-QA --plan APIEE-1234 --quiet)
539
+ npx playwright test
540
+ npx xqt import --file results.json --exec $EXEC_KEY
541
+ ```
542
+
543
+ **Mode C Push & execute (single command):**
544
+ ```bash
545
+ npx xqt push --plan APIEE-1234 --create-execution --execution-env IOP-QA
546
+ npx playwright test
547
+ npx xqt import --file results.json --env IOP-QA
548
+ ```
549
+
550
+ ---
551
+
552
+ ## Playwright Integration
553
+
554
+ ### Annotate tests with Xray keys
555
+
556
+ ```typescript
557
+ import { test, expect } from '@playwright/test';
558
+
559
+ test('GET /users returns 200', async ({ request }) => {
560
+ test.info().annotations.push({ type: 'xray', description: 'APIEE-1234' });
561
+
562
+ const response = await request.get('/users');
563
+ expect(response.status()).toBe(200);
564
+ });
565
+ ```
566
+
567
+ Or include the key in the test title:
568
+
569
+ ```typescript
570
+ test('[APIEE-1234] GET /users returns 200', async ({ request }) => { ... });
571
+ ```
572
+
573
+ ### playwright.config.ts reporters
574
+
575
+ ```typescript
576
+ reporter: [
577
+ ['json', { outputFile: 'test-results/results.json' }], // ← xqt import-results
578
+ ['junit', { outputFile: 'test-results/results.xml' }],
579
+ ['html', { open: 'never' }],
580
+ ],
581
+ ```
582
+
583
+ ### Test steps → Xray step results
584
+
585
+ Steps created with `test.step()` are mapped to Xray step results automatically:
586
+
587
+ ```typescript
588
+ test('Create and retrieve user', async ({ request }) => {
589
+ test.info().annotations.push({ type: 'xray', description: 'APIEE-2001' });
590
+
591
+ await test.step('POST /users expect 201', async () => {
592
+ const r = await request.post('/users', { data: { name: 'Alice' } });
593
+ expect(r.status()).toBe(201);
594
+ });
595
+
596
+ await test.step('GET /users/:id — expect 200', async () => {
597
+ const r = await request.get('/users/1');
598
+ expect(r.status()).toBe(200);
599
+ });
600
+ });
601
+ ```
602
+
603
+ ---
604
+
605
+ ## Folder Structure in Xray
606
+
607
+ Tests are organised in the Xray Test Repository using the `folder` field in `tests.json`:
608
+
609
+ ```json
610
+ {
611
+ "test_id": "TC-MYSVC-BR-001",
612
+ "folder": "/MyService/HealthCheck/Validation"
613
+ }
614
+ ```
615
+
616
+ - Paths must start with `/`
617
+ - `folderRoot` in `.xrayrc` sets the base path
618
+ - Folders are created automatically by `xqt push` and `xqt sync`
619
+
620
+ ---
621
+
622
+ ## CI/CD Integration
623
+
624
+ ### Azure DevOps pipeline
625
+
626
+ Generate a template: `npx xqt pipeline`
627
+
628
+ ```yaml
629
+ - script: npx xqt validate
630
+ displayName: Validate tests.json
631
+
632
+ - script: npx playwright test
633
+ displayName: Run tests
634
+ continueOnError: true
635
+
636
+ - script: |
637
+ npx xqt import \
638
+ --file test-results/results.json \
639
+ --env $(XQT_ENV)
640
+ displayName: Import results to Xray
641
+ env:
642
+ XRAY_ID: $(XRAY_ID)
643
+ XRAY_SECRET: $(XRAY_SECRET)
644
+ JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
645
+ JIRA_URL: $(JIRA_URL)
646
+ JIRA_API_TOKEN: $(JIRA_API_TOKEN)
647
+ JIRA_EMAIL: $(JIRA_EMAIL)
648
+ ```
649
+
650
+ ### Required pipeline variables
651
+
652
+ | Variable | Description |
653
+ |---|---|
654
+ | `XRAY_ID` | Xray Cloud Client ID |
655
+ | `XRAY_SECRET` | Xray Cloud Client Secret |
656
+ | `JIRA_PROJECT_KEY` | JIRA project key |
657
+ | `JIRA_URL` | JIRA instance URL |
658
+ | `JIRA_API_TOKEN` | JIRA API token |
659
+ | `JIRA_EMAIL` | JIRA user email |
660
+ | `XQT_ENV` | Environment label (`IOP-DEV` / `IOP-QA` / `IOP-PROD`) |
661
+
662
+ ### Pre-create execution in pipeline (Mode B)
663
+
664
+ ```yaml
665
+ - script: |
666
+ EXEC_KEY=$(npx xqt exec --env $(XQT_ENV) --quiet)
667
+ echo "##vso[task.setvariable variable=EXEC_KEY]$EXEC_KEY"
668
+ displayName: Pre-create Test Execution
669
+
670
+ - script: npx playwright test
671
+ continueOnError: true
672
+
673
+ - script: |
674
+ npx xqt import --file test-results/results.json --exec $(EXEC_KEY)
675
+ displayName: Import results into pre-created execution
676
+ ```
677
+
678
+ ---
679
+
680
+ ## Programmatic API
681
+
682
+ Every function is exported for custom scripts:
683
+
684
+ ```javascript
685
+ import {
686
+ loadConfig,
687
+ authenticate,
688
+ createTestExecution,
689
+ importResultsMultipart,
690
+ getTestPlan,
691
+ syncTestPlan,
692
+ } from '@msalaam/xray-qe-toolkit';
693
+
694
+ const cfg = loadConfig();
695
+ const token = await authenticate(cfg);
696
+
697
+ const { key } = await createTestExecution(cfg, token, {
698
+ summary: 'My custom execution',
699
+ environments: ['IOP-QA'],
700
+ testPlanKey: 'APIEE-1234',
701
+ });
702
+ ```
703
+
704
+ Key exports: `authenticate`, `createIssue`, `updateIssue`, `getIssue`, `getTest`, `getTests`, `getTestPlan`, `createTestPlan`, `addTestsToTestPlan`, `addTestsToTestExecution`, `createTestExecution`, `importResultsMultipart`, `importTestsBulk`, `syncTestPlan`, `syncFolders`, `buildAndPush`, `loadMapping`, `saveMapping`, `loadConfig`, `validateConfig`.
705
+
706
+ ---
707
+
708
+ ## xray-mapping.json
709
+
710
+ Auto-managed by `xqt push`. Maps local `test_id` → JIRA issue key and numeric Xray ID.
711
+
712
+ ```json
713
+ {
714
+ "TC-MYSVC-BR-001": {
715
+ "key": "APIEE-1234",
716
+ "id": "8765432"
717
+ },
718
+ "TC-MYSVC-BR-002": {
719
+ "key": "APIEE-1235",
720
+ "id": "8765433"
721
+ }
722
+ }
723
+ ```
724
+
725
+ - **Commit this file** — the whole team must share the same mapping
726
+ - Used by Copilot skill to generate Playwright tests with real Jira keys
727
+ - Do not edit manually
728
+ - Re-push to regenerate a missing entry
729
+
730
+ ---
731
+
732
+ ## Troubleshooting
733
+
734
+ **Missing credentials**
735
+ ```
736
+ Error: Missing required config: xrayId, xraySecret
737
+ ```
738
+ Ensure `.env` exists and is populated.
739
+
740
+ **Test not found in mapping after push**
741
+ ```bash
742
+ npx xqt status
743
+ npx xqt push --verbose
744
+ ```
745
+
746
+ **Import results — 0 tests matched**
747
+ Ensure Playwright tests have `xray` annotations or the key in the title:
748
+ ```typescript
749
+ test.info().annotations.push({ type: 'xray', description: 'APIEE-1234' });
750
+ ```
751
+
752
+ **Validate fails in CI**
753
+ Fix schema errors before pushing. Common causes: missing `testId`, invalid `priority`, malformed `folder` path (must start with `/`).
754
+
755
+ **Old command names still work**
756
+ All previous multi-word commands (`push-tests`, `pull-tests`, `create-plan`, `create-execution`, `import-results`, `sync-folders`, `gen-pipeline`, `mcp-server`) remain valid as aliases.
757
+
758
+ **EU region**
759
+ Set `XRAY_REGION=eu` in `.env` or `"xrayRegion": "eu"` in `.xrayrc`.
760
+
761
+ **"disallowed to impersonate" error**
762
+ `JIRA_EMAIL` must match the email of the user who created the Xray API Key.
763
+
764
+ ---
765
+
766
+ ## License
767
+
768
+ MIT see [LICENSE](LICENSE)