@msalaam/xray-qe-toolkit 1.4.0 → 1.4.1

package/templates/README.template.md

@@ -1,169 +1,570 @@
-# XQT-GUIDE — Xray QE Toolkit
-
-> Quality Engineering workflow automation for Xray Cloud integration
->
-> This file was created by `npx xqt init`. It is intentionally named `XQT-GUIDE.md`
-> so it never overwrites your project's own `README.md`.
-
-## Quick Start
-
-### 1. Configure Credentials
-
-```bash
-# Copy the template and fill in your credentials
-cp .env.example .env
-```
-
-Edit `.env` with your Xray Cloud and JIRA credentials:
-
-```env
-XRAY_ID=your_xray_client_id
-XRAY_SECRET=your_xray_client_secret
-JIRA_PROJECT_KEY=YOUR_PROJECT
-JIRA_URL=https://your-domain.atlassian.net
-JIRA_API_TOKEN=your_jira_api_token
-JIRA_EMAIL=your.email@company.com
-```
-
-**Important:** Both Xray and JIRA credentials must belong to the same user.
-
-### 2. Add Your Documentation
-
-Populate the `knowledge/` folder with your API specs and requirements:
-
-```
-knowledge/
-├── api-specs/       ← OpenAPI/Swagger specs (.yaml, .json)
-├── requirements/    ← Business requirements (.md, .pdf, .docx)
-└── tickets/         ← JIRA exports, Confluence pages
-```
-
-See [knowledge/README.md](knowledge/README.md) for details.
-
-### 3. Create Test Cases
-
-**Option A: Manual (No AI required)**
-```bash
-# Open the visual editor
-npx xqt edit-json
-```
-
-**Option B: AI-Assisted (Requires setup)**
-```bash
-# Generate from knowledge sources
-npx xqt gen-tests --ai
-
-# Then review in the editor
-npx xqt edit-json
-```
-
-### 4. Push to Xray Cloud
-
-```bash
-npx xqt push-tests
-```
-
-### 5. Set Up CI Pipeline
-
-```bash
-# Generate Azure Pipelines template
-npx xqt gen-pipeline
-```
-
-## Available Commands
-
-| Command | Description |
-|---------|-------------|
-| `npx xqt init` | Scaffold new project (already done) |
-| `npx xqt gen-tests` | Generate test cases from knowledge/ |
-| `npx xqt edit-json` | Review/edit tests in browser |
-| `npx xqt push-tests` | Push tests to Xray Cloud |
-| `npx xqt gen-pipeline` | Generate CI pipeline template |
-| `npx xqt create-execution` | Create Test Execution issue |
-| `npx xqt import-results` | Import Playwright JSON results (CI) |
-| `npx xqt compare-openapi` | Diff live spec vs QA snapshot — fails on breaking changes |
-| `npx xqt update-snapshot` | Promote current spec as new QA baseline (manual only) |
-
-Run any command with `--help` for details:
-```bash
-npx xqt gen-tests --help
-```
-
-## Workflow
-
-```
-1. Add docs to knowledge/
-2. Generate tests (AI or manual)
-3. Review in edit-json
-4. Push to Xray
-5. Generate CI pipeline: npx xqt gen-pipeline
-6. Run Playwright tests in CI
-7. Import results: npx xqt import-results --file playwright-results.json
-
-Contract Enforcement (from your test repo pipeline):
-8. Checkout API repo in pipeline
-9. npx xqt compare-openapi --current api-repo/openapi.yaml --snapshot openapi.snapshot.yaml
-   → Fail pipeline on breaking changes
-
-When a contract change is approved:
-10. npx xqt update-snapshot --current api-repo/openapi.yaml --snapshot openapi.snapshot.yaml
-    → Commit updated snapshot + raise PR to establish new baseline
-```
-
-## Files
-
-| File | Purpose |
-|------|---------|
-| `tests.json` | Test definitions (source of truth) |
-| `xray-mapping.json` | Maps test IDs to JIRA keys |
-| `playwright-results.json` | Playwright JSON report (CI output) |
-| `.env` | Credentials (DO NOT COMMIT) |
-| `.xrayrc` | Project config |
-| `knowledge/` | API specs and requirements |
-
-## Multi-Company Test Format
-
-If you work across multiple companies or business units, keep a consistent naming pattern:
-
-- `test_id`: `COMPANY-AREA-FLOW-###` (e.g., `ACME-BILLING-PAYMENTS-001`)
-- `xray.summary`: prefix with `[COMPANY]` for easy filtering
-- `xray.labels`: add `company:<slug>`, `system:<slug>`, `team:<slug>`
-- `testExecution.summary`: include company + release/sprint
-
-For multiple companies, keep separate configs and files (recommended), or swap `.env` and `.xrayrc` before running.
-
-## Jira/Xray Board Alignment
-
-1. Create a JIRA project per company/team and enable Xray.
-2. Ensure issue types include **Test** and **Test Execution**.
-3. Add fields to screens: Summary, Description, Priority, Labels, Test Steps.
-4. Grant API user permissions to create/edit Test and Test Execution issues.
-5. Create a board with a filter for your project and those issue types.
-6. Use components/labels that match your naming conventions.
-
-## Tags
-
-Use tags in `edit-json` to organize test suites:
-
-- `regression` — Full regression pack
-- `smoke` — Critical path tests only
-- `critical` — Business-critical flows
-- `edge` — Edge cases and error handling
-- `integration` — Multi-system tests
-
-## Need Help?
-
-```bash
-# Show all commands
-npx xqt --help
-
-# Show command-specific help
-npx xqt push-tests --help
-
-# Enable debug output
-npx xqt push-tests --verbose
-```
-
-## Package Documentation
-
-Full documentation: https://www.npmjs.com/package/@msalaam/xray-qe-toolkit
+# XQT-GUIDE — Xray QE Toolkit Reference
+
+> **@msalaam/xray-qe-toolkit** — Quality Engineering workflow automation for Xray Cloud.
+>
+> This file was created by `npx xqt init`. It is intentionally saved as `XQT-GUIDE.md`
+> so it never replaces your project's own `README.md`.
+
+---
+
+## Table of Contents
+
+1. [Architecture Overview](#1-architecture-overview)
+2. [Setup & Credentials](#2-setup--credentials)
+3. [Project Structure](#3-project-structure)
+4. [tests.json Reference](#4-testsjson-reference)
+5. [Command Reference](#5-command-reference)
+6. [Test Plans & Executions](#6-test-plans--executions)
+7. [Playwright Integration](#7-playwright-integration)
+8. [Folder Structure in Xray](#8-folder-structure-in-xray)
+9. [Business Rules](#9-business-rules)
+10. [CI/CD Integration](#10-cicd-integration)
+11. [xray-mapping.json](#11-xray-mappingjson)
+12. [Configuration (.xrayrc)](#12-configuration-xrayrc)
+
+> **This file covers the xqt toolkit commands and Xray configuration.**
+> For the full QE process — spec-driven workflow, greenfield/brownfield steps, AI agent
+> integration, and architecture patterns — see **[SPEC-DRIVEN-APPROACH.md](SPEC-DRIVEN-APPROACH.md)**.
+
+---
+
+## 1. Architecture Overview
+
+```
+tests.json              ← Source of truth for test case definitions
+xray-mapping.json       ← Maps local test_id → JIRA/Xray issue key + numeric ID
+resources/              ← OpenAPI spec + business rules (inputs for tests.json generation)
+tests/                  ← Playwright spec files + helpers
+SPEC-DRIVEN-APPROACH.md ← Full QE process and workflow
+```
+
+**End-to-end flow:**
+
+```
+openapi.yaml + business-rules.yaml
+      ↓ (AI agent generates tests.json)
+xqt validate → xqt push-tests (creates Tests + Test Sets in Xray)
+      ↓
+xray-mapping.json populated with Jira issue keys
+      ↓
+Copilot creates Playwright tests (reads tests.json + xray-mapping.json)
+      ↓
+Sprint starts → xqt create-plan → link Test Sets
+      ↓
+CI: playwright test → xqt import-results → Test Execution in Xray
+```
+
+**Key design decisions:**
+- **Test Sets are persistent** — tests live in Test Sets grouped by feature
+- **Test Plans are per-sprint** — create one per sprint, link relevant Test Sets
+- **Test Executions are ephemeral** — every CI run creates a fresh execution
+- **Environments are labels** — `IOP-DEV`, `IOP-QA`, `IOP-PROD` tag each execution
+- **tests.json is the source of truth** — never edit test metadata directly in JIRA
+- **Playwright tests come after push** — created only once xray-mapping.json has real Jira keys
+
+---
+
+## 2. Setup & Credentials
+
+### Install
+
+```bash
+npm install --save-dev @msalaam/xray-qe-toolkit
+# or globally
+npm install -g @msalaam/xray-qe-toolkit
+```
+
+### Initialize a new project
+
+```bash
+npx xqt init
+```
+
+### Configure `.env`
+
+```bash
+cp .env.example .env
+```
+
+```env
+# ── Xray Cloud ─────────────────────────────────────────────────
+XRAY_ID=your_xray_cloud_client_id
+XRAY_SECRET=your_xray_cloud_client_secret
+
+# ── JIRA ───────────────────────────────────────────────────────
+JIRA_PROJECT_KEY=APIEE
+JIRA_URL=https://your-org.atlassian.net
+JIRA_API_TOKEN=your_jira_api_token
+JIRA_EMAIL=your.email@company.com
+```
+
+> **Important:** The JIRA credentials must belong to a user with permission to
+> create and edit Xray tests in the target project.
+
+### Get Xray API credentials
+
+1. Go to [https://id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens)
+2. In your JIRA instance, go to **Apps → Xray → API Keys**
+3. Create a new Client ID / Client Secret pair
+
+---
+
+## 3. Project Structure
+
+After running `npx xqt init`:
+
+```
+{{SERVICE_NAME}}/
+├── resources/
+│   ├── openapi.yaml            ← OpenAPI spec (replace with your actual spec)
+│   └── business-rules.yaml     ← Business rules (scenarios → tests.json)
+│
+├── tests/
+│   ├── models/                 ← Data models, request/response fixtures
+│   ├── resources/              ← Shared helpers, utilities, constants
+│   ├── services/               ← Service-level setup/teardown
+│   └── specs/                  ← Playwright spec files (.spec.ts)
+│
+├── playwright.config.ts        ← Reporters: JSON + JUnit + HTML
+├── tests.json                  ← Test definitions (push to Xray with xqt push-tests)
+├── xray-mapping.json           ← Auto-managed: localId → Xray key
+├── .xrayrc                     ← Project config (testPlanKey, environments…)
+├── .env.example                ← Credential template (commit this)
+├── .env                        ← Your credentials (DO NOT commit)
+└── XQT-GUIDE.md                ← This file
+```
+
+---
+
+## 4. tests.json Reference
+
+```json
+{
+  "testPlan": {
+    "key": "APIEE-1234",
+    "summary": "Sprint 12 — {{SERVICE_NAME}} API Regression"
+  },
+  "tests": [
+    {
+      "test_id": "TC-MYSVC-BR-001",
+      "type": "api",
+      "skip": false,
+      "tags": ["smoke", "regression"],
+      "folder": "/{{SERVICE_NAME}}/HealthCheck/Validation",
+      "testSet": "Health Check",
+      "requirementKeys": [],
+      "xray": {
+        "summary": "Service returns 200 OK when healthy",
+        "description": "Given Service is running, when GET /health is called, then Response is 200",
+        "testType": "Generic",
+        "definition": "Automated Playwright API test: GET /health — Service returns 200 OK",
+        "priority": "High",
+        "labels": ["smoke", "regression"]
+      }
+    }
+  ]
+}
+```
+
+### Field Reference
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| `test_id` | string | Yes | Stable local identifier — maps to Xray key in xray-mapping.json |
+| `type` | string | — | `api` / `ui` / `e2e` (informational) |
+| `skip` | boolean | — | `true` to exclude from push-tests |
+| `tags` | string[] | — | QE tags for categorisation and filtering |
+| `folder` | string | — | Xray repository folder path — must start with `/` |
+| `testSet` | string | — | Test Set name in Jira — groups tests by feature |
+| `requirementKeys` | string[] | — | JIRA issue keys this test covers (creates coverage links) |
+| `xray.summary` | string | Yes | Test case title (JIRA issue summary) |
+| `xray.description` | string | — | Detailed description |
+| `xray.testType` | string | — | `Generic` (default for automated) / `Manual` / `Cucumber` |
+| `xray.definition` | string | — | Generic test definition field in Xray |
+| `xray.priority` | string | — | `Highest` / `High` / `Medium` / `Low` / `Lowest` |
+| `xray.labels` | string[] | — | JIRA labels on the test issue |
+
+### testPlan block
+
+| Field | Description |
+|---|---|
+| `testPlan.key` | Xray Test Plan issue key — populated by `xqt create-plan` |
+| `testPlan.summary` | Human-readable description (not synced to Xray, local only) |
+
+---
+
+## 5. Command Reference
+
+### `xqt init`
+
+Scaffold a new QA project in the current directory. Idempotent — never overwrites existing files.
+
+```bash
+npx xqt init
+npx xqt init --verbose
+```
+
+### `xqt create-plan`
+
+Create a new Xray Test Plan in JIRA. Automatically updates `.xrayrc` and `tests.json`.
+
+```bash
+npx xqt create-plan --summary "{{SERVICE_NAME}} v2 Regression"
+npx xqt create-plan --summary "Sprint 12 Smoke Tests" --version "2024.12"
+```
+
+### `xqt push-tests`
+
+Create or update tests in Xray Cloud, then sync the Test Plan membership and folder structure.
+
+```bash
+npx xqt push-tests
+npx xqt push-tests --plan APIEE-1234   # override plan key
+npx xqt push-tests --verbose
+```
+
+### `xqt pull-tests`
+
+Fetch test definitions from Xray and merge them into `tests.json`.
+
+```bash
+npx xqt pull-tests --plan APIEE-1234
+npx xqt pull-tests --project APIEE --limit 200
+```
+
+### `xqt import-results`
+
+Import test execution results into Xray. Creates a **new** Test Execution each time.
+
+```bash
+npx xqt import-results --file test-results/results.json --env IOP-QA
+npx xqt import-results --file test-results/results.xml   --env IOP-PROD
+npx xqt import-results --file test-results/results.json --env IOP-DEV \
+  --plan APIEE-1234 --version "2024.12" --revision "a1b2c3d"
+```
+
+**Options:**
+
+| Flag | Description |
+|---|---|
+| `--file <path>` | Path to results file (`.json` = Playwright, `.xml` = JUnit) |
+| `--env <label>` | Environment label: `IOP-DEV`, `IOP-QA`, `IOP-PROD` |
+| `--plan <key>` | Test Plan key (overrides `.xrayrc`) |
+| `--version <ver>` | Fix version / release label |
+| `--revision <sha>` | Build number or git SHA |
+| `--summary <text>` | Custom execution summary |
+
+### `xqt sync-folders`
+
+Sync the Xray repository folder structure from `folder` fields in tests.json.
+
+```bash
+npx xqt sync-folders
+```
+
+### `xqt status`
+
+Show local and remote project status.
+
+```bash
+npx xqt status
+npx xqt status --plan APIEE-1234
+```
+
+### `xqt validate`
+
+Validate `tests.json` against its schema.
+Exit code 1 on errors — suitable for CI quality gates.
+
+```bash
+npx xqt validate
+npx xqt validate --file path/to/tests.json
+```
+
+### `xqt edit`
+
+Launch a browser-based visual editor for `tests.json`.
+
+```bash
+npx xqt edit
+```
+
+### `xqt gen-pipeline`
+
+Generate an Azure Pipelines YAML template.
+
+```bash
+npx xqt gen-pipeline
+npx xqt gen-pipeline --output .azure/ci.yml
+```
+
+---
+
+## 6. Test Sets, Plans & Executions
+
+### Test Sets (persistent groupings)
+
+Tests live in **Test Sets** in Jira, grouped by feature (matching the feature name in `business-rules.yaml`).
+
+- Created automatically by `push-tests` from the `testSet` field in `tests.json`
+- Persistent across sprints — they represent what tests exist
+- Link Test Sets to Test Plans when entering a sprint
+
+### Test Plans (per-sprint)
+
+A Test Plan scopes testing for a specific sprint or release.
+
+- Create per sprint: `npx xqt create-plan --summary "Sprint 12 — {{SERVICE_NAME}}"`
+- Link relevant Test Sets to the Test Plan in Jira
+- The key is stored in `.xrayrc` (`testPlanKey`)
+- `import-results` links executions to the active plan
+
+### Test Executions (ephemeral)
+
+A Test Execution represents one CI run.
+- Created automatically by `xqt import-results`
+- Tagged with the environment (`IOP-DEV`, `IOP-QA`, `IOP-PROD`)
+- Linked to the Test Plan
+
+### Summary
+
+| Concept | Lifecycle | Purpose |
+|---------|-----------|--------|
+| **Test Set** | Permanent | Groups tests by feature |
+| **Test Plan** | Per-sprint | Scopes testing for a sprint |
+| **Test Execution** | Per-CI-run | Records pass/fail for one run |
+
+### Environment labels
+
+Set the default environment and allowed values in `.xrayrc`:
+
+```json
+{
+  "defaultEnvironment": "IOP-QA",
+  "environments": ["IOP-DEV", "IOP-QA", "IOP-PROD"]
+}
+```
+
+Override per run with `--env`:
+
+```bash
+npx xqt import-results --file results.json --env IOP-PROD
+```
+
+---
+
+## 7. Playwright Integration
+
+### Annotate tests with Xray keys
+
+```typescript
+import { test, expect } from '@playwright/test';
+
+test('GET /users returns 200', async ({ request }) => {
+  // Link this Playwright test to an Xray test case
+  test.info().annotations.push({ type: 'xray', description: 'APIEE-1234' });
+
+  const response = await request.get('/users');
+  expect(response.status()).toBe(200);
+});
+```
+
+Alternatively, include the key in the test title:
+
+```typescript
+test('[APIEE-1234] GET /users returns 200', async ({ request }) => { ... });
+```
+
+### playwright.config.ts reporters
+
+The generated `playwright.config.ts` configures three reporters:
+
+- **JSON** → `test-results/results.json` (consumed by `xqt import-results`)
+- **JUnit** → `test-results/results.xml` (also consumed by `xqt import-results --file *.xml`)
+- **HTML** → `playwright-report/` (CI artifact)
+
+### Import results
+
+```bash
+# After running: npx playwright test
+npx xqt import-results --file test-results/results.json --env IOP-QA
+```
+
+### Test steps in Xray
+
+Steps defined with `test.step()` are automatically mapped to Xray step results:
+
+```typescript
+test('Create and retrieve user', async ({ request }) => {
+  test.info().annotations.push({ type: 'xray', description: 'APIEE-2001' });
+
+  await test.step('Create user via POST /users', async () => {
+    const r = await request.post('/users', { data: { name: 'Alice' } });
+    expect(r.status()).toBe(201);
+  });
+
+  await test.step('Retrieve user via GET /users/:id', async () => {
+    const r = await request.get('/users/1');
+    expect(r.status()).toBe(200);
+  });
+});
+```
+
+---
+
+## 8. Folder Structure in Xray
+
+Organise tests in the Xray Test Repository using the `folder` field.
+
+```json
+{
+  "test_id": "TC-MYSVC-BR-001",
+  "folder": "/{{SERVICE_NAME}}/HealthCheck/Validation"
+}
+```
+
+- Paths must start with `/`
+- The `folderRoot` in `.xrayrc` is used as the base (default: `/{{SERVICE_NAME}}`)
+- Folders are created automatically by `push-tests` and `sync-folders`
+- Run `xqt sync-folders` to update folders without pushing test changes
+
+---
+
+## 9. Business Rules
+
+Business rules live in `resources/business-rules.yaml` and describe the
+service's expected behaviour. They are the input (alongside `openapi.yaml`) for generating `tests.json`.
+
+```yaml
+service: "{{SERVICE_NAME}}"
+version: "1.0"
+
+features:
+  - name: "User Management"
+    endpoints:
+      - GET /api/v1/users
+      - POST /api/v1/users
+    rules:
+      - id: "BR-001"
+        description: "Valid user creation returns 201"
+        given: "Authenticated user with create permission"
+        when: "POST /api/v1/users with valid body"
+        then: "Response is 201 with user id"
+        category: "business-logic"
+        priority: "High"
+        tags: [regression, smoke]
+```
+
+- Each feature name becomes a **Test Set** in Jira
+- Each rule maps 1:1 to a `tests.json` entry and a Playwright test
+- See **[SPEC-DRIVEN-APPROACH.md](SPEC-DRIVEN-APPROACH.md)** for full generation rules
+
+Validate the rules file with:
+
+```bash
+npx xqt validate --schema business-rules
+```
+
+---
+
+## 10. CI/CD Integration
+
+### Azure DevOps
+
+`npx xqt init` scaffolds a ready-to-use `azure-pipelines.yml` with two stages:
+
+- **Stage 1 `CodeQuality`** — ESLint + TypeScript type check
+- **Stage 2 `QA_Regression`** — Playwright tests → `xqt import-results` → artifact publish
+
+Key import step (runs even on test failure):
+
+```yaml
+- script: |
+    REVISION=$(echo "$(Build.SourceVersion)" | cut -c1-8)
+    npx xqt import-results \
+      --file test-results/results.json \
+      --version "$(Build.BuildNumber)" \
+      --revision "$REVISION"
+  condition: succeededOrFailed()
+  env:
+    XRAY_ID: $(xray_client_id)
+    XRAY_SECRET: $(xray_client_secret)
+    JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
+    JIRA_URL: $(JIRA_URL)
+    JIRA_API_TOKEN: $(JIRA_API_TOKEN)
+    JIRA_EMAIL: $(JIRA_EMAIL)
+    XRAY_GRAPHQL_URL: $(XRAY_GRAPHQL_URL_US)
+    XRAY_REST_URL: $(XRAY_REST_URL)
+```
+
+### Required ADO variable groups
+
+Create these in **ADO Library → Variable groups** before running the pipeline:
+
+| Group | Variable | Description |
+|---|---|---|
+| `xray-qa-shared` *(shared — one-time)* | `xray_client_id` | Xray Cloud Client ID |
+| | `xray_client_secret` | Xray Cloud Client Secret |
+| | `JIRA_PROJECT_KEY` | JIRA project key |
+| | `JIRA_URL` | JIRA instance URL |
+| | `JIRA_API_TOKEN` | JIRA API token |
+| | `JIRA_EMAIL` | JIRA user email |
+| | `XRAY_GRAPHQL_URL_US` | Xray GraphQL endpoint |
+| | `XRAY_REST_URL` | Xray REST endpoint |
+| `{{SERVICE_NAME}}-qa` *(per-repo)* | `base_url` | API base URL for this service |
+| | `gravitee_api_key` | Gateway API key |
+
+---
+
+## 11. xray-mapping.json
+
+Auto-managed by `push-tests` — do not edit manually.
+
+```json
+{
+  "TC-MYSVC-BR-001": {
+    "key": "APIEE-1234",
+    "id": "8765432"
+  }
+}
+```
+
+- `key` — JIRA issue key (e.g. `APIEE-1234`)
+- `id` — Xray numeric issue ID (used for GraphQL operations)
+- Entries are created by `push-tests` and updated on every push
+- **Commit this file** — the whole team must share the same mapping
+- Used by Copilot skill to generate Playwright tests with real Jira keys
+
+---
+
+## 12. Configuration (.xrayrc)
+
+`.xrayrc` (JSON) at the project root configures xqt behaviour per-project.
+
+```json
+{
+  "testsPath":          "tests.json",
+  "mappingPath":        "xray-mapping.json",
+  "resourcesPath":      "resources",
+  "playwrightDir":      "tests",
+  "testPlanKey":        "APIEE-1234",
+  "defaultEnvironment": "IOP-DEV",
+  "environments":       ["IOP-DEV", "IOP-QA", "IOP-PROD"],
+  "folderRoot":         "/{{SERVICE_NAME}}",
+  "xrayRegion":         "us"
+}
+```
+
+### xrayRegion values
+
+| Value | GraphQL endpoint |
+|---|---|
+| `us` (default) | `https://xray.cloud.getxray.app/` |
+| `eu` | `https://eu.xray.cloud.getxray.app/` |
+| `au` | `https://au.xray.cloud.getxray.app/` |
+
+You can also set `XRAY_REGION=eu` in `.env`.
+
+---
+
+*Generated by @msalaam/xray-qe-toolkit — See [npm package](https://www.npmjs.com/package/@msalaam/xray-qe-toolkit) for updates.*