@msalaam/xray-qe-toolkit 1.5.0 → 1.6.1

package/templates/SPEC-DRIVEN-APPROACH.md

@@ -1,372 +0,0 @@
-# Spec-Driven API Test Automation — QE Process & Workflow
-
-> **Scope:** API testing with Playwright (no UI/browser).  
-> **Source of truth:** `openapi.yaml` (API shape) + `business-rules.yaml` (domain logic & scenarios).  
-> **Per-service repos:** Each API gets its own `{ServiceName}_Automation` repo.
->
-> This document defines the end-to-end workflow from specification to test execution.
-
----
-
-## Overview
-
-The spec-driven approach uses two knowledge artifacts — an **OpenAPI specification** and a **`business-rules.yaml`** file — to produce a complete, traceable API test suite.
-
-The process follows a strict sequence:
-
-```
-openapi.yaml + business-rules.yaml
-      ↓
-  Generate tests.json (test definitions)
-      ↓
-  xqt push-tests → creates Xray Test issues + Test Sets in Jira
-      ↓
-  xray-mapping.json populated with issue keys
-      ↓
-  Playwright API tests created (1:1 with tests.json entries)
-      ↓
-  Sprint starts → create Test Plan → link Test Sets
-      ↓
-  Run tests → xqt import-results → Test Execution in Xray
-```
-
-### Why Two Files?
-
-| File | What It Provides | Who Maintains |
-|------|-----------------|---------------|
-| `openapi.yaml` | Schemas, paths, status codes, parameter types, auth schemes | Developers |
-| `business-rules.yaml` | Domain rules, validation logic, edge-case scenarios, test data | Developers + QA |
-
-The OpenAPI spec tells you **what the API looks like**.  
-The business rules tell you **what the API should do**.  
-Together they are sufficient to generate a complete test suite. Neither alone is enough.
-
-### 1:1 Traceability
-
-Every `business-rules.yaml` rule maps to exactly one `tests.json` entry, one Xray Test issue, and one Playwright `test()` call. Nothing exists in one place that doesn't exist in all three.
-
----
-
-## Test Organisation in Jira/Xray
-
-### Test Sets (persistent groupings)
-
-Tests live in **Test Sets** in Jira. Test Sets group related tests by feature, endpoint, or category. They are the primary organisational unit.
-
-- One Test Set per `business-rules.yaml` feature (e.g. "Health Check", "User Management", "Payments")
-- Tests are added to Test Sets when pushed via `xqt push-tests`
-- Test Sets are permanent — they persist across sprints
-- The `testSet` field in `tests.json` determines which Test Set a test belongs to
-
-### Test Plans (per-sprint)
-
-A **Test Plan** represents work for a specific sprint or release. When entering a sprint:
-
-1. Create a Test Plan for the sprint: `npx xqt create-plan --summary "Sprint 12 — Payments API"`
-2. Link the relevant Test Sets to the Test Plan
-3. Each CI run creates a new **Test Execution** linked to that Test Plan
-4. At sprint end, the Test Plan shows full pass/fail history for that sprint's scope
-
-This means:
-- **Test Sets** = what tests exist (permanent)
-- **Test Plans** = what we're testing this sprint (per-sprint)
-- **Test Executions** = individual CI runs (ephemeral)
-
----
-
-## Deterministic Generation Rules
-
-> **These rules are contracts, not guidelines.**  
-> The same inputs must always produce the same outputs. Any deviation breaks `xray-mapping.json` stability and orphans Xray test issues.
-
-### `testId` Derivation
-
-```
-testId = "TC-{SERVICE_CODE}-{SCENARIO_ID}"
-```
-
-- `SERVICE_CODE` — uppercase alphanumeric slug of the service name, max 8 chars. Example: `PAYMENTS`, `USERMGMT`, `AUTH`.
-- `SCENARIO_ID` — the `id` field from `business-rules.yaml` verbatim. Example: `BR-001`, `BR-042`.
-- Full example: `TC-PAYMENTS-BR-001`
-- `testId` values are **stable and permanent**. Once pushed to Xray they never change.
-- If a rule is removed from `business-rules.yaml`, its entry in `tests.json` gets `"skip": true` — it is never deleted.
-
-### `tests.json` Field Mapping
-
-Every field is derived deterministically from the knowledge artifacts. No creative decisions.
-
-| `tests.json` field | Source | Derivation rule |
-|---|---|---|
-| `test_id` | `rule.id` | `TC-{SERVICE_CODE}-{rule.id}` |
-| `xray.summary` | `rule.description` | Copied verbatim |
-| `xray.description` | `rule.given` + `rule.when` + `rule.then` | `"Given {given}, when {when}, then {then}"` |
-| `tags` | `rule.tags` | Copied from rule tags array |
-| `xray.priority` | `rule.priority` | Direct mapping (see table below) |
-| `requirementKeys` | `rule.requirementKeys` | Copied verbatim. Empty array if absent |
-| `folder` | Feature name + category | `"/{ServiceName}/{FeatureName}/{CategoryFolder}"` |
-| `testSet` | Feature `name` | The feature name becomes the Test Set name |
-| `skip` | `rule.skip` or absence flag | `false` by default. `true` if rule deprecated |
-| `xray.testType` | Always `"Generic"` | All automated API tests are Generic type |
-| `xray.definition` | rule description + endpoint | `"Automated Playwright API test: {endpoint} — {rule.description}"` |
-
-#### Priority Mapping
-
-| `business-rules.yaml` priority | `tests.json` / Xray priority |
-|---|---|
-| `Highest` | `Highest` |
-| `High` | `High` |
-| `Medium` | `Medium` |
-| `Low` | `Low` |
-| `Lowest` | `Lowest` |
-| *(absent)* | `Medium` |
-
-#### Folder Path Rules
-
-```
-/{ServiceName}/{FeatureName}/{CategoryFolder}
-```
-
-| `rule.category` | Folder segment |
-|---|---|
-| `validation` | `Validation` |
-| `authorization` | `Authorization` |
-| `business-logic` | `BusinessLogic` |
-| `edge-case` | `Edge` |
-| `integration` | `Integration` |
-| `security` | `Security` |
-| `performance` | `Performance` |
-| *(absent)* | `General` |
-
-Full example: `/Payments/Transactions/Validation`
-
----
-
-## `business-rules.yaml` Contract
-
-Every rule must contain these fields. Validate before generation — report violations before writing any files.
-
-```yaml
-service: my-service
-version: "1.0"
-
-features:
-  - name: Health Check                       # Required. Becomes the Test Set name.
-    endpoints:                               # Required. Links rules to OpenAPI paths.
-      - GET /health
-    rules:
-      - id: BR-001                           # Required. Stable. Never reuse a retired ID.
-        description: Service returns 200 OK  # Required. Becomes xray.summary.
-        given: Service is running            # Required. Precondition.
-        when: GET /health is called          # Required. Action.
-        then: Response is 200               # Required. Expected outcome.
-        category: validation                 # Required. Controls folder placement.
-        priority: High                       # Required. Maps to Xray priority.
-        tags: [smoke, regression]            # Optional. Used as labels.
-        requirementKeys: []                  # Optional. Creates Xray coverage links.
-        skip: false                          # Optional. Defaults to false.
-```
-
-If `business-rules.yaml` does not exist, generate a draft from the OpenAPI spec and flag it as `DRAFT — REQUIRES QA + DEV REVIEW`.
-
----
-
-## Workflow
-
-### Step 0: Scaffold the Project
-
-**Who:** QA Engineer (one-time)  
-**Command:** `npx xqt init`
-
-1. Create the repo: `{ServiceName}_Automation`
-2. Run `npx xqt init` — produces the full scaffold
-3. Copy `.env.example` to `.env` and fill in credentials
-4. Set up ADO variable groups:
-   - **`xray-qa-shared`** (shared across repos): `xray_client_id`, `xray_client_secret`, `JIRA_PROJECT_KEY`, `JIRA_URL`, `JIRA_API_TOKEN`, `JIRA_EMAIL`
-   - **`{ServiceName}-qa`** (per-repo): `base_url`, `gravitee_api_key`, service-specific secrets
-
-`xqt init` is idempotent — running it again won't overwrite existing files.
-
-### Step 1: Prepare Knowledge Artifacts
-
-**Who:** QA Engineer + Dev Team
-
-1. Place the OpenAPI spec at `resources/api-specs/openapi.yaml`
-2. Create `resources/business-rules.yaml` using the contract above
-   - One feature per logical API grouping
-   - One rule per testable behaviour
-   - Every rule has `id`, `description`, `given`, `when`, `then`, `category`, `priority`
-3. Validate: `npx xqt validate`
-
-### Step 2: Generate `tests.json`
-
-**Who:** AI Agent (Copilot skill)  
-**Input:** `openapi.yaml` + `business-rules.yaml`  
-**Output:** Populated `tests.json`
-
-The AI agent reads both artifacts and produces `tests.json` using the **Deterministic Generation Rules** above. Every `business-rules.yaml` rule becomes exactly one `tests.json` entry.
-
-- Each feature's rules are grouped under a `testSet` matching the feature name
-- `test_id` values follow the `TC-{SERVICE_CODE}-{SCENARIO_ID}` pattern
-- All fields derived deterministically — no creative decisions
-
-After generation, validate: `npx xqt validate`
-
-### Step 3: Push Tests to Xray
-
-**Who:** QA Engineer or CI  
-**Command:** `npx xqt push-tests`
-
-This creates/updates Xray Test issues and Test Sets in Jira:
-- New tests → created in Xray, added to appropriate Test Set
-- Existing tests (in `xray-mapping.json`) → updated in place
-- Tests with `"skip": true` → excluded
-- `xray-mapping.json` is auto-populated with the Jira issue keys
-
-After push, `xray-mapping.json` contains the mapping:
-```json
-{
-  "TC-PAYMENTS-BR-001": { "key": "APIEE-1234", "id": "8765432" },
-  "TC-PAYMENTS-BR-002": { "key": "APIEE-1235", "id": "8765433" }
-}
-```
-
-**Commit `tests.json` and `xray-mapping.json` together.**
-
-### Step 4: Create Playwright API Tests
-
-**Who:** AI Agent (Copilot skill — separate from XQT)
-
-Playwright tests are created **after** `xray-mapping.json` exists, so every test can reference its real Jira key from the start.
-
-- One `test()` call per `tests.json` entry — 1:1 mapping
-- Test titles include the Xray key: `[APIEE-1234] Service returns 200 OK when healthy`
-- Tests use `test.info().annotations` to link to Xray
-- The Copilot skill reads `tests.json` and `xray-mapping.json` to generate each spec
-
-> **Note:** XQT does not generate Playwright tests. Test code is created by a separate Copilot skill
-> that uses this spec-driven approach, `tests.json`, and `xray-mapping.json` as its inputs.
-
-### Step 5: Sprint Workflow — Test Plans
-
-**Who:** QA Engineer
-
-When entering a new sprint:
-
-1. **Create a Test Plan:**
-   ```bash
-   npx xqt create-plan --summary "Sprint 12 — Payments API Regression"
-   ```
-
-2. **Link relevant Test Sets** to the Test Plan in Jira
-   - Select the Test Sets containing tests relevant to this sprint's scope
-   - This makes the Test Plan a curated view of what matters for the sprint
-
-3. **Run tests and import results:**
-   ```bash
-   npx playwright test
-   npx xqt import-results --file test-results/results.json --env IOP-QA
-   ```
-   Each run creates a new Test Execution linked to the Test Plan.
-
-4. **Sprint complete:** The Test Plan in Jira shows the full execution history for the sprint.
-
-### Step 6: CI/CD — Ongoing
-
-Every CI pipeline run:
-```bash
-npx xqt validate                                          # Schema check
-npx playwright test                                        # Run tests
-npx xqt import-results --file test-results/results.json \
-  --env IOP-QA                                             # Import to Xray
-```
-
-Results appear as new Test Executions linked to the active Test Plan.
-
----
-
-## Brownfield Workflow (Updating Existing Tests)
-
-When the OpenAPI spec or business rules change:
-
-1. **Update `business-rules.yaml`** — add new rules, modify existing, mark removed rules with `skip: true`
-2. **Regenerate `tests.json`** — AI agent re-derives from updated artifacts
-   - New rules → new `tests.json` entries appended
-   - Modified rules → existing entries updated field-by-field (never change `test_id`)
-   - Removed rules → `"skip": true` (never delete)
-3. **Push changes:** `npx xqt push-tests` — creates new Xray issues, updates existing
-4. **Update Playwright tests** — Copilot skill creates tests for new entries, updates existing
-5. **Validate:** run the suite, import results
-
-**Critical rule:** `xray-mapping.json` entries are never removed or overwritten for existing `test_id` values. The mapping only grows.
-
----
-
-## Workflow Summary
-
-| Step | Who | Action | Output |
-|------|-----|--------|--------|
-| 0 | QA Engineer | `xqt init` | Scaffolded project |
-| 1 | QA + Dev | Prepare `openapi.yaml` + `business-rules.yaml` | Validated knowledge artifacts |
-| 2 | AI Agent | Generate from specs | `tests.json` populated |
-| 3 | QA / CI | `xqt push-tests` | Xray Tests + Test Sets created, `xray-mapping.json` populated |
-| 4 | AI Agent | Create Playwright tests | `*.spec.ts` files (1:1 with `tests.json`) |
-| 5 | QA Engineer | `xqt create-plan` + link Test Sets | Sprint Test Plan ready |
-| 6 | CI | `playwright test` + `xqt import-results` | Test Execution in Xray |
-
----
-
-## Key Principles
-
-1. **`tests.json` is the source of truth** for test metadata — never edit tests directly in Jira
-2. **`xray-mapping.json` is owned by XQT** — only `xqt push-tests` writes it, never edit manually
-3. **`testId` values are permanent** — once pushed, they never change; retired rules get `"skip": true`
-4. **Test Sets are the home for tests** — persistent groupings that live across sprints
-5. **Test Plans are per-sprint** — create one per sprint, link relevant Test Sets
-6. **Test Executions are ephemeral** — each CI run creates a fresh one
-7. **1:1 mapping everywhere** — one rule → one `tests.json` entry → one Xray issue → one `test()` call
-8. **Generation is deterministic** — same inputs always produce same outputs
-9. **Playwright tests come last** — only created after `xray-mapping.json` exists with real Jira keys
-10. **Never store secrets in code** — all credentials in `.env` or ADO variable groups
-
----
-
-## QE Process — End-to-End
-
-```
-Dev team maintains openapi.yaml + business-rules.yaml
-  │
-  ▼
-AI Agent generates tests.json from both artifacts
-(deterministic rules — no creative decisions)
-  │
-  ▼
-npx xqt validate (hard stop on any error)
-  │
-  ▼
-npx xqt push-tests
-  → Xray Test issues created/updated
-  → Test Sets created/updated
-  → xray-mapping.json auto-populated
-  │
-  ▼
-AI Agent (Copilot skill) creates Playwright API tests
-  → reads tests.json + xray-mapping.json
-  → one spec per tests.json entry
-  → real Jira keys in test titles from day one
-  │
-  ▼
-Sprint starts → QA creates Test Plan → links Test Sets
-  │
-  ▼
-CI runs: playwright test → xqt import-results
-  → Test Execution created in Xray
-  → linked to Test Plan
-  │
-  ▼
-Full traceability chain:
-Business Rule → tests.json → Xray Test (in Test Set) → Playwright test → Test Execution (in Test Plan)
-```
-
----
-
-**The contract:** The OpenAPI spec defines what the API looks like. The business rules define what the API should do. `tests.json` captures both as structured test definitions. XQT syncs those definitions to Xray. Copilot writes the Playwright tests. Test Sets organise the tests permanently. Test Plans scope them to sprints. Nothing else is needed.