@msalaam/xray-qe-toolkit 1.1.0 → 1.3.0

package/README.md

@@ -11,6 +11,7 @@
 - [Installation](#installation)
 - [AI Setup (Optional)](#ai-setup-optional)
 - [Quick Start](#quick-start)
+- [Playwright Quick Start](#playwright-quick-start)
 - [CLI Commands](#cli-commands)
   - [init](#xray-qe-init)
   - [gen-tests](#xray-qe-gen-tests)
@@ -31,6 +32,7 @@
 - [Jira/Xray Project Setup](#jiraxray-project-setup)
 - [QE Workflow](#qe-workflow)
 - [Building Regression Packs](#building-regression-packs)
+- [Working with Existing Xray Tests](#working-with-existing-xray-tests)
 - [Idempotent Push](#idempotent-push)
 - [Programmatic API](#programmatic-api)
 - [Troubleshooting](#troubleshooting)
@@ -46,8 +48,9 @@
 3. **Idempotent push** — updates existing tests, creates new ones, skips duplicates
 4. **Postman collection generation** from test definitions
 5. **CI pipeline template** for Azure DevOps (Newman + Xray import)
-6. **Modular architecture** — every function is importable for programmatic use
-7. **AI-ready scaffolds** — optional AI assistance for test generation (manual workflow fully supported)
+6. **Playwright integration** — import Playwright JSON results with automatic test mapping
+7. **Modular architecture** — every function is importable for programmatic use
+8. **AI-ready scaffolds** — optional AI assistance for test generation (manual workflow fully supported)
 
 ### QE Review Gate Philosophy
 
@@ -242,7 +245,7 @@ All commands support these global options:
 
 ---
 
-### `xray-qe init`
+### `xqt init`
 
 Scaffold a new project with starter templates.
 
@@ -261,7 +264,7 @@ Existing files are **never overwritten** — the command skips them with a warni
 
 ---
 
-### `xray-qe gen-tests`
+### `xqt gen-tests`
 
 Generate test cases from `knowledge/` folder documentation (AI-assisted or manual guidance).
 
@@ -300,7 +303,7 @@ Existing files are **never overwritten** — the command skips them with a warni
 
 ---
 
-### `xray-qe edit-json`
+### `xqt edit-json`
 
 Launch the browser-based QE review gate editor.
 
@@ -325,7 +328,7 @@ npx xqt edit-json --port 3000
 
 ---
 
-### `xray-qe push-tests`
+### `xqt push-tests`
 
 Push or update tests in Xray Cloud from `tests.json`.
 
@@ -354,7 +357,7 @@ npx xqt push-tests --skip-exec
 
 ---
 
-### `xray-qe gen-postman`
+### `xqt gen-postman`
 
 Generate a Postman Collection v2.1 JSON from `tests.json` (works with or without AI).
 
@@ -398,7 +401,7 @@ npx xqt gen-postman --knowledge ./docs
 
 ---
 
-### `xray-qe create-execution`
+### `xqt create-execution`
 
 Create a standalone Test Execution issue in JIRA.
 
@@ -415,22 +418,92 @@ npx xqt create-execution --summary "Feature XYZ" --issue QE-123
 
 ---
 
-### `xray-qe import-results`
+### `xqt import-results`
 
-Import JUnit/Newman XML results into Xray Cloud. Designed for CI — no human interaction.
+Import test results into Xray Cloud. Supports JUnit XML and Playwright JSON formats. Designed for CI — no human interaction.
+
+#### Format Comparison
+
+| Feature | JUnit XML | Playwright JSON |
+|---------|-----------|----------------|
+| **Update existing tests** | ❌ No - always creates new tests | ✅ Yes - via annotations |
+| **Test key mapping** | ❌ Not supported | ✅ `test.info().annotations` |
+| **Attachments/Evidence** | ❌ Not supported | ✅ Supported |
+| **Best for** | Newman, generic test runners | Playwright tests |
+| **Recommendation** | ⚠️ Use only for tools without test keys | ✅ **Recommended for Playwright** |
+
+#### JUnit XML (Newman, generic test runners)
+
+⚠️ **Important:** JUnit XML **cannot update existing Xray tests** - it will always create new test cases. Only use this for Newman or test runners that don't support test keys.
 
 ```bash
 npx xqt import-results --file results.xml --testExecKey QE-123
 ```
 
-| Option              | Description                                | Required |
-|---------------------|--------------------------------------------|----------|
-| `--file <path>`     | Path to the JUnit/Newman results XML file  | Yes      |
-| `--testExecKey <key>` | Test Execution to associate results with | Yes      |
+#### Playwright JSON (Recommended)
+
+✅ **Updates existing tests** via annotations. Use this format to link results to your existing Xray test cases.
+
+```bash
+# Generate JSON report in Playwright
+npx playwright test --reporter=json > playwright-results.json
+
+# Import to Xray (updates existing tests via annotations)
+npx xqt import-results --file playwright-results.json --testExecKey QE-123
+
+# Create new Test Execution automatically
+npx xqt import-results --file playwright-results.json --summary "Regression Suite"
+```
+
+**Required test code:**
+```typescript
+import { test, expect } from '@playwright/test';
+
+test('Verify API endpoint', async ({ request }) => {
+  // THIS LINE links to existing Xray test
+  test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
+  
+  // Your test code...
+});
+```
+
+| Option                      | Description                                       | Required |
+|-----------------------------|---------------------------------------------------|----------|
+| `--file <path>`             | Path to results file (.xml or .json)             | Yes      |
+| `--testExecKey <key>`       | Test Execution to link results to                | No*      |
+| `--summary <text>`          | Summary for new Test Execution (if no testExecKey) | No    |
+| `--description <text>`      | Description for new Test Execution               | No       |
+
+\* If `--testExecKey` is omitted, a new Test Execution will be created automatically.
+
+**Mapping Playwright Tests to Xray:**
+
+To link Playwright test results to existing Xray test cases, **you must use annotations**:
+
+```typescript
+// ✅ CORRECT - Updates existing test PROJ-123
+test('User login flow', async ({ page }) => {
+  test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
+  // ... test code
+});
+
+// ✅ Alternative - Include test key in title
+test('[PROJ-456] User registration validates email', async ({ page }) => {
+  // ... test code
+});
+
+// ❌ WRONG - Creates new test every time
+test('User login flow', async ({ page }) => {
+  // Missing annotation - will create duplicate test!
+  // ... test code
+});
+```
+
+**Without annotations:** A new test will be created in Xray with the test title as the summary (not recommended for existing tests).
 
 ---
 
-### `xray-qe gen-pipeline`
+### `xqt gen-pipeline`
 
 Generate an Azure Pipelines YAML template.
 
@@ -453,7 +526,7 @@ The generated pipeline:
 
 ---
 
-### `xray-qe mcp-server`
+### `xqt mcp-server`
 
 Start a Model Context Protocol server for GitHub Copilot agent-mode integration.
 
@@ -682,6 +755,177 @@ Use this checklist to align a new team's board and Xray configuration with the t
 └─────────────────────────────────────────────────────────────────────────┘
 ```
 
+## Playwright Quick Start
+
+### Complete Setup for Updating Existing Xray Tests
+
+This is the **recommended workflow** for teams using Playwright with existing Xray test cases.
+
+#### Step 1: Install Playwright (in your test repo)
+
+```bash
+npm install --save-dev @playwright/test
+```
+
+#### Step 2: Configure Playwright
+
+Create `playwright.config.ts` in your test repo:
+
+```typescript
+import { defineConfig } from '@playwright/test';
+
+export default defineConfig({
+  reporter: [
+    ['html'],  // For local viewing
+    ['json', { outputFile: 'playwright-results.json' }],  // For Xray import
+  ],
+  
+  use: {
+    baseURL: process.env.API_BASE_URL || 'https://your-api.com',
+    extraHTTPHeaders: {
+      'Authorization': `Bearer ${process.env.API_TOKEN}`,
+      'X-API-Key': process.env.API_KEY || '',
+    },
+  },
+});
+```
+
+#### Step 3: Write Tests with Xray Annotations
+
+⚠️ **Critical:** Every test MUST have an annotation to update existing Xray tests.
+
+```typescript
+import { test, expect } from '@playwright/test';
+
+test.describe('Regression Tests', () => {
+  
+  test('Verify API returns 200 for valid request', async ({ request }) => {
+    // THIS LINE links to your existing Xray test
+    test.info().annotations.push({ type: 'xray', description: 'APIEE-7131' });
+    
+    const response = await request.post('/api/verify', {
+      data: { userId: '123', action: 'validate' }
+    });
+    
+    // Attach response as evidence for Xray
+    test.info().attach('response-evidence', {
+      body: JSON.stringify({
+        status: response.status(),
+        headers: response.headers(),
+        body: await response.json()
+      }, null, 2),
+      contentType: 'application/json'
+    });
+    
+    expect(response.status()).toBe(200);
+  });
+  
+  test('Verify API returns 400 for invalid data', async ({ request }) => {
+    test.info().annotations.push({ type: 'xray', description: 'APIEE-7132' });
+    // ... test implementation
+  });
+  
+});
+```
+
+#### Step 4: Map All Your Tests
+
+Based on your `xray-mapping.json`, add annotations to each test:
+
+```typescript
+// If your xray-mapping.json shows:
+// "TC-001": { "key": "APIEE-7131", "id": "1879092" }
+// "TC-002": { "key": "APIEE-7132", "id": "1879095" }
+
+test('Test Case 1', async ({ request }) => {
+  test.info().annotations.push({ type: 'xray', description: 'APIEE-7131' });
+  // ...
+});
+
+test('Test Case 2', async ({ request }) => {
+  test.info().annotations.push({ type: 'xray', description: 'APIEE-7132' });
+  // ...
+});
+```
+
+#### Step 5: Run Locally
+
+```bash
+# Run tests
+npx playwright test
+
+# View HTML report
+npx playwright show-report
+
+# Upload to Xray (updates existing tests)
+npx xqt import-results --file playwright-results.json --testExecKey APIEE-6811
+```
+
+#### Step 6: Configure CI/CD
+
+**Azure Pipelines:**
+
+```yaml
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '18.x'
+
+  - script: npm ci
+    displayName: 'Install dependencies'
+
+  - script: npx playwright test
+    displayName: 'Run Playwright tests'
+    continueOnError: true
+
+  - script: |
+      npx xqt import-results \
+        --file playwright-results.json \
+        --testExecKey APIEE-6811
+    displayName: 'Upload results to Xray'
+    env:
+      XRAY_ID: $(XRAY_ID)
+      XRAY_SECRET: $(XRAY_SECRET)
+      JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
+      JIRA_URL: $(JIRA_URL)
+      JIRA_API_TOKEN: $(JIRA_API_TOKEN)
+      JIRA_EMAIL: $(JIRA_EMAIL)
+```
+
+**GitHub Actions:**
+
+```yaml
+steps:
+  - uses: actions/setup-node@v3
+    with:
+      node-version: '18'
+
+  - run: npm ci
+
+  - run: npx playwright test
+
+  - run: |
+      npx xqt import-results \
+        --file playwright-results.json \
+        --testExecKey APIEE-6811
+    env:
+      XRAY_ID: ${{ secrets.XRAY_ID }}
+      XRAY_SECRET: ${{ secrets.XRAY_SECRET }}
+      JIRA_PROJECT_KEY: ${{ secrets.JIRA_PROJECT_KEY }}
+      JIRA_URL: ${{ secrets.JIRA_URL }}
+      JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
+      JIRA_EMAIL: ${{ secrets.JIRA_EMAIL }}
+```
+
+### Benefits
+
+✅ **Updates existing tests** - No duplicate test creation  
+✅ **Automatic mapping** - Via annotations in test code  
+✅ **Evidence attachments** - Screenshots, responses, traces  
+✅ **Detailed reporting** - Retries, worker info, error details  
+✅ **CI/CD ready** - Standard workflow for automation  
+✅ **Single source of truth** - Test code + Xray together
+
 ---
 
 ## Building Regression Packs
@@ -823,6 +1067,173 @@ newman run collection.postman.json  # Full regression weekly
 
 ---
 
+## Working with Existing Xray Tests
+
+If your team already has test cases in Xray Cloud that were created manually or by another tool, you can set up this toolkit to manage and update those existing tests.
+
+### Step 1: Query Existing Tests from Xray
+
+Use the Xray GraphQL API to fetch your existing tests. Here's a script to generate the mapping file:
+
+**fetch-existing-tests.js:**
+```javascript
+import { authenticate, loadConfig } from "@msalaam/xray-qe-toolkit";
+import fs from "fs";
+
+const cfg = loadConfig();
+const token = await authenticate(cfg);
+
+// GraphQL query to fetch all tests in your project
+const query = `
+  query {
+    getTests(jql: "project = ${cfg.jiraProjectKey} AND issuetype = Test", limit: 1000) {
+      total
+      results {
+        issueId
+        jira(fields: ["key", "summary", "description", "priority", "labels"])
+      }
+    }
+  }
+`;
+
+const response = await fetch(cfg.xrayGraphQLUrl || "https://us.xray.cloud.getxray.app/api/v2/graphql", {
+  method: "POST",
+  headers: {
+    "Content-Type": "application/json",
+    Authorization: `Bearer ${token}`,
+  },
+  body: JSON.stringify({ query }),
+});
+
+const data = await response.json();
+const tests = data.data.getTests.results;
+
+// Generate xray-mapping.json
+const mapping = {};
+tests.forEach((test) => {
+  const testId = test.jira.key; // Use JIRA key as test_id initially
+  mapping[testId] = {
+    key: test.jira.key,
+    id: test.issueId,
+  };
+});
+
+fs.writeFileSync("xray-mapping.json", JSON.stringify(mapping, null, 2));
+console.log(`✓ Created xray-mapping.json with ${tests.length} tests`);
+
+// Generate tests.json scaffold
+const testsJson = {
+  testExecution: {
+    summary: "Existing Tests - Managed by Toolkit",
+    description: "Imported from existing Xray tests",
+  },
+  tests: tests.map((test) => ({
+    test_id: test.jira.key,
+    skip: false,
+    tags: [],
+    xray: {
+      summary: test.jira.summary,
+      description: test.jira.description || "",
+      priority: test.jira.priority?.name || "Medium",
+      labels: test.jira.labels || [],
+      steps: [
+        // You'll need to fetch test steps separately via another API call
+        {
+          action: "PLACEHOLDER - Edit this in npx xqt edit-json",
+          data: "",
+          expected_result: "PLACEHOLDER",
+        },
+      ],
+    },
+  })),
+};
+
+fs.writeFileSync("tests.json", JSON.stringify(testsJson, null, 2));
+console.log(`✓ Created tests.json with ${tests.length} test scaffolds`);
+console.log("\nNext steps:");
+console.log("1. Run 'npx xqt edit-json' to review and complete test steps");
+console.log("2. Run 'npx xqt push-tests' to update tests in Xray");
+```
+
+### Step 2: Run the Script
+
+```bash
+# Save the script above as fetch-existing-tests.js
+node fetch-existing-tests.js
+```
+
+This generates:
+- `xray-mapping.json` — maps your existing JIRA test keys to their IDs
+- `tests.json` — scaffold with summaries, descriptions, priorities, labels
+
+### Step 3: Complete Test Steps
+
+The script can't fetch test steps automatically (requires additional API calls). Complete them in the editor:
+
+```bash
+npx xqt edit-json
+# Edit each test to add proper steps
+# Save when done
+```
+
+### Step 4: Update Existing Tests
+
+Now you can update your existing Xray tests:
+
+```bash
+npx xqt push-tests
+```
+
+Because the tests are in `xray-mapping.json`, they'll be **updated** (not duplicated).
+
+### Alternative: Manual Mapping Creation
+
+If you have a small number of tests, create the mapping manually:
+
+**xray-mapping.json:**
+```json
+{
+  "APIEE-6933": { "key": "APIEE-6933", "id": "1865623" },
+  "APIEE-6934": { "key": "APIEE-6934", "id": "1865627" },
+  "APIEE-6935": { "key": "APIEE-6935", "id": "1865628" }
+}
+```
+
+**tests.json:**
+```json
+{
+  "tests": [
+    {
+      "test_id": "APIEE-6933",
+      "xray": {
+        "summary": "Test User Login",
+        "steps": [
+          { "action": "...", "expected_result": "..." }
+        ]
+      }
+    }
+  ]
+}
+```
+
+Then run `npx xqt push-tests` to update.
+
+### Future Enhancement: Pull Command
+
+A `pull-tests` command to automatically fetch and sync existing tests is planned:
+
+```bash
+# Future feature (not yet implemented)
+npx xqt pull-tests --project APIEE
+npx xqt pull-tests --jql "project = APIEE AND labels = regression"
+```
+
+This would automatically generate both `tests.json` and `xray-mapping.json` from Xray.
+
+**Want this feature?** [Open an issue on GitHub](https://github.com/Muhammad-Salaam_omit/xray-qe-toolkit/issues) or contribute via PR.
+
+---
+
 ## Idempotent Push
 
 `push-tests` checks `xray-mapping.json` before each operation:

package/bin/cli.js

@@ -13,7 +13,7 @@
  *   push-tests         Push / update tests in Xray Cloud
  *   gen-postman        Generate Postman collection from tests.json (with optional AI)
  *   create-execution   Create a new Test Execution in JIRA
- *   import-results     Import JUnit/Newman results into Xray
+ *   import-results     Import test results (JUnit XML or Playwright JSON) into Xray
  *   gen-pipeline       Generate an Azure Pipelines YAML template
  *   mcp-server         Start Model Context Protocol server for agent integration
  */
@@ -103,9 +103,11 @@ program
 
 program
   .command("import-results")
-  .description("Import JUnit/Newman XML results into Xray Cloud")
-  .requiredOption("--file <path>", "Path to the results XML file")
-  .requiredOption("--testExecKey <key>", "Test Execution key to associate results with")
+  .description("Import test results into Xray Cloud (JUnit XML or Playwright JSON)")
+  .requiredOption("--file <path>", "Path to the results file (.xml for JUnit, .json for Playwright)")
+  .option("--testExecKey <key>", "Test Execution key to associate results with (creates new if omitted)")
+  .option("--summary <text>", "Test Execution summary (for new executions)")
+  .option("--description <text>", "Test Execution description (for new executions)")
   .action(async (opts, cmd) => {
     const mod = await import("../commands/importResults.js");
     await mod.default({ ...opts, ...cmd.optsWithGlobals() });

package/commands/importResults.js

@@ -1,8 +1,11 @@
 /**
- * Command: xray-qe import-results --file <path> --testExecKey <key>
+ * Command: xqt import-results --file <path> [--testExecKey <key>]
  *
- * Imports JUnit/Newman XML results into Xray Cloud and associates
- * them with the specified Test Execution.
+ * Imports test results into Xray Cloud and associates them with a Test Execution.
+ *
+ * Supported formats:
+ * - JUnit XML (.xml) - Standard JUnit or XUnit format
+ * - Playwright JSON (.json) - Playwright JSON reporter output
  *
  * Designed to run in CI — no human interaction required.
  */
@@ -11,7 +14,8 @@ import fs from "node:fs";
 import path from "node:path";
 import logger, { setVerbose } from "../lib/logger.js";
 import { loadConfig, validateConfig } from "../lib/config.js";
-import { authenticate, importResults as importResultsApi } from "../lib/xrayClient.js";
+import { authenticate, importResults as importResultsXml, importResultsXrayJson } from "../lib/xrayClient.js";
+import { convertPlaywrightToXray } from "../lib/playwrightConverter.js";
 
 export default async function importResults(opts = {}) {
   if (opts.verbose) setVerbose(true);
@@ -27,23 +31,53 @@ export default async function importResults(opts = {}) {
     process.exit(1);
   }
 
-  logger.info(`File: ${path.basename(filePath)}`);
-  logger.info(`Test Execution: ${opts.testExecKey}`);
+  const fileName = path.basename(filePath);
+  const fileExt = path.extname(filePath).toLowerCase();
+
+  logger.info(`File: ${fileName}`);
+  if (opts.testExecKey) {
+    logger.info(`Test Execution: ${opts.testExecKey}`);
+  }
   logger.blank();
 
   // Authenticate
   const xrayToken = await authenticate(cfg);
 
-  // Read XML file
-  const xmlBuffer = fs.readFileSync(filePath);
-  logger.send(`Importing ${xmlBuffer.length} bytes to Xray...`);
+  // Detect format and import
+  let result;
+
+  if (fileExt === '.json') {
+    // Playwright JSON format
+    logger.send(`Importing Playwright JSON results...`);
 
-  const result = await importResultsApi(cfg, xrayToken, xmlBuffer, opts.testExecKey);
+    const playwrightJson = JSON.parse(fs.readFileSync(filePath, 'utf-8'));
+    
+    // Convert to Xray format
+    const xrayJson = convertPlaywrightToXray(playwrightJson, {
+      testExecutionKey: opts.testExecKey,
+      projectKey: cfg.jiraProjectKey,
+      summary: opts.summary || `Playwright Test Execution - ${new Date().toLocaleString()}`,
+      description: opts.description,
+    });
+
+    logger.step(`Converted ${xrayJson.tests.length} test results to Xray format`);
+
+    // Import to Xray
+    result = await importResultsXrayJson(cfg, xrayToken, xrayJson);
+  } else {
+    // JUnit XML format (default)
+    logger.send(`Importing JUnit XML results...`);
+
+    const xmlBuffer = fs.readFileSync(filePath);
+    result = await importResultsXml(cfg, xrayToken, xmlBuffer, opts.testExecKey);
+  }
 
   logger.success("Results imported successfully");
 
   if (result?.key) {
     logger.link(`View: ${cfg.jiraUrl}/browse/${result.key}`);
+  } else if (result?.testExecIssue?.key) {
+    logger.link(`View: ${cfg.jiraUrl}/browse/${result.testExecIssue.key}`);
   }
 
   logger.blank();