@msalaam/xray-qe-toolkit 1.3.3 → 1.4.0

package/README.md

@@ -22,6 +22,8 @@
   - [import-results](#xray-qe-import-results)
   - [gen-pipeline](#xray-qe-gen-pipeline)
   - [mcp-server](#xray-qe-mcp-server)
+  - [compare-openapi](#xray-qe-compare-openapi)
+  - [update-snapshot](#xray-qe-update-snapshot)
 - [Configuration](#configuration)
   - [Environment Variables (.env)](#environment-variables-env)
   - [Project Config (.xrayrc)](#project-config-xrayrc)
@@ -49,8 +51,9 @@
 4. **Postman collection generation** from test definitions
 5. **CI pipeline template** for Azure DevOps (Newman + Xray import)
 6. **Playwright integration** — import Playwright JSON results with automatic test mapping
-7. **Modular architecture** — every function is importable for programmatic use
-8. **AI-ready scaffolds** — optional AI assistance for test generation (manual workflow fully supported)
+7. **OpenAPI contract enforcement** — `compare-openapi` diffs a live spec against a QA snapshot and fails the pipeline on breaking changes; `update-snapshot` promotes a new baseline deliberately
+8. **Modular architecture** — every function is importable for programmatic use
+9. **AI-ready scaffolds** — optional AI assistance for test generation (manual workflow fully supported)
 
 ### QE Review Gate Philosophy
 
@@ -559,6 +562,100 @@ npx xqt mcp-server --port 3100
 
 ---
 
+### `xqt compare-openapi`
+
+Compare an OpenAPI spec against an approved QA snapshot and fail the pipeline if breaking changes are detected.
+
+```bash
+# Basic comparison — exits 1 on breaking changes
+npx xqt compare-openapi \
+  --current ../api-repo/openapi.yaml \
+  --snapshot ./openapi.snapshot.yaml
+
+# Save a JSON diff report as a pipeline artifact
+npx xqt compare-openapi \
+  --current ../api-repo/openapi.yaml \
+  --snapshot ./openapi.snapshot.yaml \
+  --report openapi-diff-report.json
+```
+
+| Option               | Description                                                      | Required |
+|----------------------|------------------------------------------------------------------|----------|
+| `--current <path>`   | Path to the current (live) OpenAPI spec                          | Yes      |
+| `--snapshot <path>`  | Path to the approved QA baseline snapshot                        | Yes      |
+| `--report <path>`    | Write full diff results to a JSON file (useful as CI artifact)   | No       |
+
+**Behaviour:**
+- Snapshot = approved QA contract baseline
+- Current = proposed/live spec
+- Exits `0` if no breaking changes; logs a count of non-breaking differences
+- Exits `1` on any breaking change and prints the full diff
+- If `--report` is specified and breaking changes are found, a JSON report is written
+
+**Example Azure Pipelines step (from your test repo pipeline):**
+
+```yaml
+steps:
+  - checkout: self
+
+  - checkout: git://Project/portfolio-api
+    path: api-repo
+
+  - script: |
+      npx xqt compare-openapi \
+        --current api-repo/openapi.yaml \
+        --snapshot openapi.snapshot.yaml \
+        --report openapi-diff-report.json
+    displayName: Compare OpenAPI Contracts
+
+  - publish: openapi-diff-report.json
+    artifact: openapi-diff
+    condition: failed()
+```
+
+**Governance model:**
+- Dev repo is never modified by QE
+- QE test repo pipeline checks out the API repo and enforces the contract
+- Breaking change → pipeline fails → dev must fix spec or raise a contract change review
+- QE then runs `update-snapshot` to promote the new baseline
+
+---
+
+### `xqt update-snapshot`
+
+Overwrite the QA snapshot baseline with the current spec. **Always a deliberate, manual action — never called automatically.**
+
+```bash
+npx xqt update-snapshot \
+  --current ../api-repo/openapi.yaml \
+  --snapshot ./openapi.snapshot.yaml
+```
+
+| Option               | Description                                         | Required |
+|----------------------|-----------------------------------------------------|----------|
+| `--current <path>`   | Path to the current (live) OpenAPI spec to promote  | Yes      |
+| `--snapshot <path>`  | Path to the snapshot file to overwrite              | Yes      |
+
+**Workflow:**
+1. Dev raises a contract change review
+2. QE approves the new contract
+3. QE runs `update-snapshot` locally
+4. QE raises a PR in the test repo with the updated snapshot
+5. PR is reviewed and merged — new baseline is established
+
+```bash
+# After merging the approved contract change:
+npx xqt update-snapshot \
+  --current ../api-repo/openapi.yaml \
+  --snapshot ./openapi.snapshot.yaml
+
+git add openapi.snapshot.yaml
+git commit -m "chore: promote OpenAPI snapshot to v2.5.0"
+git push
+```
+
+---
+
 ## Configuration
 
 ### Environment Variables (.env)
@@ -754,6 +851,21 @@ Use this checklist to align a new team's board and Xray configuration with the t
 │  10. newman run collection.postman.json --reporters junit              │
 │  11. npx xqt import-results --file results.xml --testExecKey QE-123│
 │                                                                         │
+├─────────────────────────────────────────────────────────────────────────┤
+│                   CONTRACT ENFORCEMENT (TEST REPO PIPELINE)             │
+│                                                                         │
+│  12. Checkout API repo                                                  │
+│  13. npx xqt compare-openapi                                       │
+│        --current api-repo/openapi.yaml                                  │
+│        --snapshot openapi.snapshot.yaml                                 │
+│      → Fails pipeline on breaking changes                               │
+│                                                                         │
+│  (When QE approves a contract change)                                   │
+│  14. npx xqt update-snapshot                                       │
+│        --current api-repo/openapi.yaml                                  │
+│        --snapshot openapi.snapshot.yaml                                 │
+│      → Commit updated snapshot + raise PR                               │
+│                                                                         │
 └─────────────────────────────────────────────────────────────────────────┘
 ```
 

package/bin/cli.js

@@ -1,4 +1,4 @@
-#!/usr/bin/env node
+#!/usr/bin/env node
 
 /**
  * @msalaam/xray-qe-toolkit — CLI Entry Point
@@ -16,6 +16,8 @@
  *   import-results     Import test results (JUnit XML or Playwright JSON) into Xray
  *   gen-pipeline       Generate an Azure Pipelines YAML template
  *   mcp-server         Start Model Context Protocol server for agent integration
+ *   compare-openapi    Compare OpenAPI specs and fail if breaking changes are detected
+ *   update-snapshot    Overwrite the QA snapshot baseline with the current spec
  */
 
 import { Command } from "commander";
@@ -131,6 +133,27 @@ program
     await mod.default({ ...opts, ...cmd.optsWithGlobals() });
   });
 
+program
+  .command("compare-openapi")
+  .description("Compare OpenAPI specs against a QA snapshot and fail if breaking changes are detected")
+  .requiredOption("--current <path>", "Path to the current (live) OpenAPI spec")
+  .requiredOption("--snapshot <path>", "Path to the approved QA snapshot")
+  .option("--report <path>", "Write a JSON diff report to this path (useful as a pipeline artifact)")
+  .action(async (opts, cmd) => {
+    const mod = await import("../commands/compareOpenapi.js");
+    await mod.default({ ...opts, ...cmd.optsWithGlobals() });
+  });
+
+program
+  .command("update-snapshot")
+  .description("Overwrite the QA snapshot baseline with the current spec (always a deliberate, manual action)")
+  .requiredOption("--current <path>", "Path to the current (live) OpenAPI spec to promote")
+  .requiredOption("--snapshot <path>", "Path to the snapshot file to overwrite")
+  .action(async (opts, cmd) => {
+    const mod = await import("../commands/updateSnapshot.js");
+    await mod.default({ ...opts, ...cmd.optsWithGlobals() });
+  });
+
 // ─── Parse & run ───────────────────────────────────────────────────────────────
 
 program.parseAsync(process.argv).catch((err) => {

package/commands/compareOpenapi.js

@@ -0,0 +1,78 @@
+import { diff } from 'openapi-diff';
+import fs from 'fs';
+import path from 'path';
+
+/**
+ * Compare two OpenAPI specs and detect breaking changes.
+ * @param {string} currentPath  - Path to the current (live) OpenAPI spec.
+ * @param {string} snapshotPath - Path to the approved QA snapshot.
+ * @param {string|null} reportPath - Optional path to write a JSON diff report.
+ */
+export async function compareOpenApiSpecs(currentPath, snapshotPath, reportPath = null) {
+  const resolvedCurrent = path.resolve(currentPath);
+  const resolvedSnapshot = path.resolve(snapshotPath);
+
+  if (!fs.existsSync(resolvedCurrent)) {
+    throw new Error(`Current OpenAPI spec not found: ${resolvedCurrent}`);
+  }
+
+  if (!fs.existsSync(resolvedSnapshot)) {
+    throw new Error(`Snapshot OpenAPI spec not found: ${resolvedSnapshot}`);
+  }
+
+  const currentSpec = fs.readFileSync(resolvedCurrent, 'utf-8');
+  const snapshotSpec = fs.readFileSync(resolvedSnapshot, 'utf-8');
+
+  const result = await diff({
+    sourceSpec: { content: snapshotSpec },      // approved baseline
+    destinationSpec: { content: currentSpec }   // proposed change
+  });
+
+  const summary = {
+    breaking: result.breakingDifferencesFound,
+    breakingDifferences: result.breakingDifferences,
+    nonBreakingDifferences: result.nonBreakingDifferences
+  };
+
+  if (reportPath && result.breakingDifferencesFound) {
+    const resolvedReport = path.resolve(reportPath);
+    fs.writeFileSync(resolvedReport, JSON.stringify(summary, null, 2), 'utf-8');
+    console.log(`📄 Diff report written to: ${resolvedReport}`);
+  }
+
+  return summary;
+}
+
+export default async function compareOpenapi(opts) {
+  const { current, snapshot, report } = opts;
+
+  try {
+    const result = await compareOpenApiSpecs(current, snapshot, report || null);
+
+    if (result.breaking) {
+      console.error('\n❌ Breaking changes detected:\n');
+      console.error(JSON.stringify(result.breakingDifferences, null, 2));
+
+      const count = result.breakingDifferences?.length ?? 0;
+      console.error(`\n${count} breaking difference(s) found.`);
+
+      if (!report) {
+        console.error('\nTip: use --report <path> to save a full diff report as a pipeline artifact.');
+      }
+
+      process.exit(1);
+    }
+
+    const nonBreaking = result.nonBreakingDifferences?.length ?? 0;
+    console.log('\n✅ No breaking changes detected.');
+
+    if (nonBreaking > 0) {
+      console.log(`ℹ️  ${nonBreaking} non-breaking difference(s) found (safe to proceed).`);
+    }
+
+    process.exit(0);
+  } catch (err) {
+    console.error(`\n❌ ${err.message}`);
+    process.exit(1);
+  }
+}

package/commands/genPipeline.js

@@ -2,7 +2,7 @@
  * Command: xray-qe gen-pipeline [--output <path>]
  *
  * Copies the Azure Pipelines YAML template into the consuming project.
- * The template runs Newman and imports results — no QE logic in CI.
+ * The template runs Playwright and imports JSON results — no QE logic in CI.
  */
 
 import fs from "node:fs";
@@ -34,8 +34,11 @@ export default async function genPipeline(opts = {}) {
   console.log("   1. Set pipeline variables in Azure DevOps:");
   console.log("      XRAY_ID, XRAY_SECRET, JIRA_PROJECT_KEY, JIRA_URL,");
   console.log("      JIRA_API_TOKEN, JIRA_EMAIL, TEST_EXEC_KEY");
-  console.log("   2. Commit the generated file to your repo");
-  console.log("   3. Create a pipeline in Azure DevOps pointing to this YAML");
+  console.log("   2. Optionally set API_BASE_URL for Playwright tests");
+  console.log("   3. Commit the generated file to your repo");
+  console.log("   4. Create a pipeline in Azure DevOps pointing to this YAML");
+  console.log("   5. Playwright tests must have xray annotations to map results:");
+  console.log("      test.info().annotations.push({ type: 'xray', description: 'PROJ-123' })");
   console.log("");
   console.log("⚠️  Remember: edit-json and QE review gates are NOT part of CI.");
   console.log("   Run those steps locally before committing.\n");

package/commands/init.js

@@ -114,7 +114,7 @@ export default async function init(opts = {}) {
     const xrayrc = {
       testsPath: "tests.json",
       mappingPath: "xray-mapping.json",
-      collectionPath: "collection.postman.json",
+      playwrightResultsPath: "playwright-results.json",
       knowledgePath: "knowledge",
       playwrightDir: "playwright-tests",
     };
@@ -136,7 +136,7 @@ export default async function init(opts = {}) {
   console.log("   4. Generate test cases: npx xray-qe gen-tests --ai (or edit tests.json manually)");
   console.log("   5. Review tests: npx xray-qe edit-json");
   console.log("   6. Push tests to Xray: npx xray-qe push-tests");
-  console.log("   7. Generate Postman collection: npx xray-qe gen-postman --ai");
-  console.log("   8. Generate CI pipeline: npx xray-qe gen-pipeline");
+  console.log("   7. Generate CI pipeline: npx xqt gen-pipeline");
+  console.log("   8. Run Playwright tests and import results: npx xqt import-results --file playwright-results.json");
   console.log("");
 }

package/commands/updateSnapshot.js

@@ -0,0 +1,34 @@
+import fse from 'fs-extra';
+import path from 'path';
+
+/**
+ * Copy the current OpenAPI spec over the snapshot baseline.
+ * This is intentionally a manual, explicit action — never called automatically.
+ *
+ * @param {string} currentPath  - Path to the current (live) OpenAPI spec.
+ * @param {string} snapshotPath - Path to the snapshot file to overwrite.
+ */
+export async function updateSnapshot(currentPath, snapshotPath) {
+  const resolvedCurrent = path.resolve(currentPath);
+  const resolvedSnapshot = path.resolve(snapshotPath);
+
+  if (!(await fse.pathExists(resolvedCurrent))) {
+    throw new Error(`Current OpenAPI spec not found: ${resolvedCurrent}`);
+  }
+
+  await fse.copy(resolvedCurrent, resolvedSnapshot, { overwrite: true });
+  console.log(`✅ Snapshot updated: ${resolvedSnapshot}`);
+  console.log(`   Source: ${resolvedCurrent}`);
+}
+
+export default async function updateSnapshotCmd(opts) {
+  const { current, snapshot } = opts;
+
+  try {
+    await updateSnapshot(current, snapshot);
+    console.log('\nBaseline established. Commit the updated snapshot to your test repo to record the new contract.');
+  } catch (err) {
+    console.error(`\n❌ ${err.message}`);
+    process.exit(1);
+  }
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@msalaam/xray-qe-toolkit",
-  "version": "1.3.3",
-  "description": "Full QE workflow toolkit for Xray Cloud integration — test management, Postman generation, CI pipeline scaffolding, and browser-based review gates for API regression projects.",
+  "version": "1.4.0",
+  "description": "Full QE workflow toolkit for Xray Cloud integration — test management, Playwright integration, CI pipeline scaffolding, and browser-based review gates for API regression projects.",
   "type": "module",
   "bin": {
     "xqt": "./bin/cli.js"
@@ -35,7 +35,10 @@
     "commander": "^13.1.0",
     "dotenv": "^17.2.3",
     "express": "^5.1.0",
-    "open": "^10.1.2"
+    "fs-extra": "^11.3.3",
+    "open": "^10.1.2",
+    "openapi-diff": "^0.24.1",
+    "yaml": "^2.8.2"
   },
   "devDependencies": {
     "ajv": "^8.17.1"

package/templates/README.template.md

@@ -63,13 +63,7 @@ npx xqt edit-json
 npx xqt push-tests
 ```
 
-### 5. Generate Postman Collection
-
-```bash
-npx xqt gen-postman
-```
-
-### 6. Set Up CI Pipeline
+### 5. Set Up CI Pipeline
 
 ```bash
 # Generate Azure Pipelines template
@@ -84,10 +78,11 @@ npx xqt gen-pipeline
 | `npx xqt gen-tests` | Generate test cases from knowledge/ |
 | `npx xqt edit-json` | Review/edit tests in browser |
 | `npx xqt push-tests` | Push tests to Xray Cloud |
-| `npx xqt gen-postman` | Generate Postman collection |
 | `npx xqt gen-pipeline` | Generate CI pipeline template |
 | `npx xqt create-execution` | Create Test Execution issue |
-| `npx xqt import-results` | Import test results (CI) |
+| `npx xqt import-results` | Import Playwright JSON results (CI) |
+| `npx xqt compare-openapi` | Diff live spec vs QA snapshot — fails on breaking changes |
+| `npx xqt update-snapshot` | Promote current spec as new QA baseline (manual only) |
 
 Run any command with `--help` for details:
 ```bash
@@ -101,8 +96,18 @@ npx xqt gen-tests --help
 2. Generate tests (AI or manual)
 3. Review in edit-json
 4. Push to Xray
-5. Generate Postman collection
-6. Run in CI (newman + import-results)
+5. Generate CI pipeline: npx xqt gen-pipeline
+6. Run Playwright tests in CI
+7. Import results: npx xqt import-results --file playwright-results.json
+
+Contract Enforcement (from your test repo pipeline):
+8. Checkout API repo in pipeline
+9. npx xqt compare-openapi --current api-repo/openapi.yaml --snapshot openapi.snapshot.yaml
+   → Fail pipeline on breaking changes
+
+When a contract change is approved:
+10. npx xqt update-snapshot --current api-repo/openapi.yaml --snapshot openapi.snapshot.yaml
+    → Commit updated snapshot + raise PR to establish new baseline
 ```
 
 ## Files
@@ -111,7 +116,7 @@ npx xqt gen-tests --help
 |------|---------|
 | `tests.json` | Test definitions (source of truth) |
 | `xray-mapping.json` | Maps test IDs to JIRA keys |
-| `collection.postman.json` | Generated Postman collection |
+| `playwright-results.json` | Playwright JSON report (CI output) |
 | `.env` | Credentials (DO NOT COMMIT) |
 | `.xrayrc` | Project config |
 | `knowledge/` | API specs and requirements |

package/templates/azure-pipelines.yml

@@ -3,8 +3,8 @@
 # Generated by @msalaam/xray-qe-toolkit
 # ──────────────────────────────────────────────────────────────
 #
-# This pipeline runs Newman against the generated Postman collection
-# and imports results into Xray Cloud.
+# This pipeline runs Playwright tests and imports the JSON results
+# into Xray Cloud via xqt import-results.
 #
 # IMPORTANT: edit-json and QE review gates are NOT part of CI.
 # Those steps must be run locally before committing.
@@ -12,6 +12,9 @@
 # Required pipeline variables (set in Azure DevOps UI → Variables):
 #   XRAY_ID, XRAY_SECRET, JIRA_PROJECT_KEY, JIRA_URL,
 #   JIRA_API_TOKEN, JIRA_EMAIL, TEST_EXEC_KEY
+#
+# Optional pipeline variables:
+#   API_BASE_URL  — base URL passed to Playwright tests
 # ──────────────────────────────────────────────────────────────
 
 trigger:
@@ -24,7 +27,7 @@ pool:
   vmImage: "ubuntu-latest"
 
 variables:
-  NODE_VERSION: "18.x"
+  NODE_VERSION: "20.x"
 
 steps:
   - task: NodeTool@0
@@ -37,17 +40,24 @@ steps:
     displayName: "Install dependencies"
 
   - script: |
-      npx newman run collection.postman.json \
-        --reporters cli,junit \
-        --reporter-junit-export results.xml \
-        --suppress-exit-code
-    displayName: "Run Newman tests"
+      npx playwright install --with-deps chromium
+    displayName: "Install Playwright browsers"
+
+  - script: |
+      npx playwright test \
+        --reporter=list,json \
+        --output=playwright-results
+    displayName: "Run Playwright tests"
+    continueOnError: true
+    env:
+      PLAYWRIGHT_JSON_OUTPUT_NAME: playwright-results.json
+      API_BASE_URL: $(API_BASE_URL)
 
   - script: |
       npx xqt import-results \
-        --file results.xml \
+        --file playwright-results.json \
         --testExecKey $(TEST_EXEC_KEY)
-    displayName: "Import results to Xray"
+    displayName: "Import Playwright results to Xray"
     env:
       XRAY_ID: $(XRAY_ID)
       XRAY_SECRET: $(XRAY_SECRET)
@@ -57,9 +67,18 @@ steps:
       JIRA_EMAIL: $(JIRA_EMAIL)
 
   - task: PublishTestResults@2
-    displayName: "Publish JUnit results"
+    displayName: "Publish Playwright JUnit results"
+    condition: always()
     inputs:
       testResultsFormat: "JUnit"
-      testResultsFiles: "results.xml"
+      testResultsFiles: "playwright-results/results.xml"
       mergeTestResults: true
       failTaskOnFailedTests: true
+
+  - task: PublishPipelineArtifact@1
+    displayName: "Upload Playwright HTML report"
+    condition: always()
+    inputs:
+      targetPath: "playwright-report"
+      artifact: "playwright-report"
+      publishLocation: "pipeline"