@msalaam/xray-qe-toolkit 1.2.0 → 1.3.1

package/README.md

@@ -11,6 +11,7 @@
 - [Installation](#installation)
 - [AI Setup (Optional)](#ai-setup-optional)
 - [Quick Start](#quick-start)
+- [Playwright Quick Start](#playwright-quick-start)
 - [CLI Commands](#cli-commands)
   - [init](#xray-qe-init)
   - [gen-tests](#xray-qe-gen-tests)
@@ -421,23 +422,51 @@ npx xqt create-execution --summary "Feature XYZ" --issue QE-123
 
 Import test results into Xray Cloud. Supports JUnit XML and Playwright JSON formats. Designed for CI — no human interaction.
 
-**JUnit XML (JUnit, Newman, etc.):**
+#### Format Comparison
+
+| Feature | JUnit XML | Playwright JSON |
+|---------|-----------|----------------|
+| **Update existing tests** | ❌ No - always creates new tests | ✅ Yes - via annotations |
+| **Test key mapping** | ❌ Not supported | ✅ `test.info().annotations` |
+| **Attachments/Evidence** | ❌ Not supported | ✅ Supported |
+| **Best for** | Newman, generic test runners | Playwright tests |
+| **Recommendation** | ⚠️ Use only for tools without test keys | ✅ **Recommended for Playwright** |
+
+#### JUnit XML (Newman, generic test runners)
+
+⚠️ **Important:** JUnit XML **cannot update existing Xray tests** - it will always create new test cases. Only use this for Newman or test runners that don't support test keys.
+
 ```bash
 npx xqt import-results --file results.xml --testExecKey QE-123
 ```
 
-**Playwright JSON:**
+#### Playwright JSON (Recommended)
+
+✅ **Updates existing tests** via annotations. Use this format to link results to your existing Xray test cases.
+
 ```bash
 # Generate JSON report in Playwright
 npx playwright test --reporter=json > playwright-results.json
 
-# Import to Xray (links to existing tests or creates new ones)
+# Import to Xray (updates existing tests via annotations)
 npx xqt import-results --file playwright-results.json --testExecKey QE-123
 
 # Create new Test Execution automatically
 npx xqt import-results --file playwright-results.json --summary "Regression Suite"
 ```
 
+**Required test code:**
+```typescript
+import { test, expect } from '@playwright/test';
+
+test('Verify API endpoint', async ({ request }) => {
+  // THIS LINE links to existing Xray test
+  test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
+  
+  // Your test code...
+});
+```
+
 | Option                      | Description                                       | Required |
 |-----------------------------|---------------------------------------------------|----------|
 | `--file <path>`             | Path to results file (.xml or .json)             | Yes      |
@@ -449,22 +478,28 @@ npx xqt import-results --file playwright-results.json --summary "Regression Suit
 
 **Mapping Playwright Tests to Xray:**
 
-To link Playwright test results to existing Xray test cases, use annotations or include test keys in titles:
+To link Playwright test results to existing Xray test cases, **you must use annotations**:
 
 ```typescript
-// Option 1: Using annotations (recommended)
+// ✅ CORRECT - Updates existing test PROJ-123
 test('User login flow', async ({ page }) => {
   test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
   // ... test code
 });
 
-// Option 2: Include test key in title
+// ✅ Alternative - Include test key in title
 test('[PROJ-456] User registration validates email', async ({ page }) => {
   // ... test code
 });
+
+// ❌ WRONG - Creates new test every time
+test('User login flow', async ({ page }) => {
+  // Missing annotation - will create duplicate test!
+  // ... test code
+});
 ```
 
-If no test key is found, a new test will be created in Xray with the test title as the summary.
+**Without annotations:** A new test will be created in Xray with the test title as the summary (not recommended for existing tests).
 
 ---
 
@@ -720,33 +755,192 @@ Use this checklist to align a new team's board and Xray configuration with the t
 └─────────────────────────────────────────────────────────────────────────┘
 ```
 
-### Playwright Integration Workflow
+## Playwright Quick Start
+
+### Complete Setup for Updating Existing Xray Tests
+
+This is the **recommended workflow** for teams using Playwright with existing Xray test cases.
 
-For teams using Playwright for API or E2E testing:
+#### Step 1: Install Playwright (in your test repo)
 
 ```bash
-# 1. Write Playwright tests with Xray annotations
-# tests/login.spec.ts
-test('User login with valid credentials', async ({ page }) => {
-  test.info().annotations.push({ type: 'xray', description: 'PROJ-123' });
-  // ... test implementation
+npm install --save-dev @playwright/test
+```
+
+#### Step 2: Configure Playwright
+
+Create `playwright.config.ts` in your test repo:
+
+```typescript
+import { defineConfig } from '@playwright/test';
+
+export default defineConfig({
+  reporter: [
+    ['html'],  // For local viewing
+    ['json', { outputFile: 'playwright-results.json' }],  // For Xray import
+  ],
+  
+  use: {
+    baseURL: process.env.API_BASE_URL || 'https://your-api.com',
+    extraHTTPHeaders: {
+      'Authorization': `Bearer ${process.env.API_TOKEN}`,
+      'X-API-Key': process.env.API_KEY || '',
+    },
+  },
+});
+```
+
+#### Step 3: Write Tests with Xray Annotations
+
+⚠️ **Critical:** Every test MUST have an annotation to update existing Xray tests.
+
+```typescript
+import { test, expect } from '@playwright/test';
+
+test.describe('Regression Tests', () => {
+  
+  test('Verify API returns 200 for valid request', async ({ request }) => {
+    // THIS LINE links to your existing Xray test
+    test.info().annotations.push({ type: 'xray', description: 'APIEE-7131' });
+    
+    const response = await request.post('/api/verify', {
+      data: { userId: '123', action: 'validate' }
+    });
+    
+    // Attach response as evidence for Xray
+    test.info().attach('response-evidence', {
+      body: JSON.stringify({
+        status: response.status(),
+        headers: response.headers(),
+        body: await response.json()
+      }, null, 2),
+      contentType: 'application/json'
+    });
+    
+    expect(response.status()).toBe(200);
+  });
+  
+  test('Verify API returns 400 for invalid data', async ({ request }) => {
+    test.info().annotations.push({ type: 'xray', description: 'APIEE-7132' });
+    // ... test implementation
+  });
+  
 });
+```
 
-# 2. Run tests and generate JSON report
-npx playwright test --reporter=json > playwright-results.json
+#### Step 4: Map All Your Tests
 
-# 3. Import results to Xray (links to existing tests or creates new)
-npx xqt import-results --file playwright-results.json --testExecKey PROJ-456
+Based on your `xray-mapping.json`, add annotations to each test:
 
-# Or create new execution automatically
-npx xqt import-results --file playwright-results.json --summary "Playwright Regression"
+```typescript
+// If your xray-mapping.json shows:
+// "TC-001": { "key": "APIEE-7131", "id": "1879092" }
+// "TC-002": { "key": "APIEE-7132", "id": "1879095" }
+
+test('Test Case 1', async ({ request }) => {
+  test.info().annotations.push({ type: 'xray', description: 'APIEE-7131' });
+  // ...
+});
+
+test('Test Case 2', async ({ request }) => {
+  test.info().annotations.push({ type: 'xray', description: 'APIEE-7132' });
+  // ...
+});
 ```
 
-**Benefits:**
-- Automatically maps Playwright tests to Xray test cases using annotations or `[TEST-123]` in titles
-- Creates new test cases in Xray if no mapping found
-- Supports Playwright retries, worker info, and detailed error reporting
-- Works in CI/CD pipelines for continuous test reporting
+#### Step 5: Run Locally
+
+```bash
+# Run tests
+npx playwright test
+
+# View HTML report
+npx playwright show-report
+
+# Upload to Xray (updates existing tests)
+npx xqt import-results --file playwright-results.json --testExecKey APIEE-6811
+```
+
+**What happens:**
+- ✅ Tests WITH annotations (`test.info().annotations.push(...)`) → Updates existing Xray tests
+- ⏭️ Tests WITHOUT annotations → **Automatically skipped** (won't create duplicates)
+- 📊 Summary shows: Passed, Failed, Skipped counts
+- 🔗 Direct link to view results in Xray
+
+**Verbose mode** (see exactly what's being uploaded):
+```bash
+npx xqt import-results \
+  --file playwright-results.json \
+  --testExecKey APIEE-6811 \
+  --verbose
+```
+
+This saves `playwright-results-xray-debug.json` for inspection.
+
+#### Step 6: Configure CI/CD
+
+**Azure Pipelines:**
+
+```yaml
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '18.x'
+
+  - script: npm ci
+    displayName: 'Install dependencies'
+
+  - script: npx playwright test
+    displayName: 'Run Playwright tests'
+    continueOnError: true
+
+  - script: |
+      npx xqt import-results \
+        --file playwright-results.json \
+        --testExecKey APIEE-6811
+    displayName: 'Upload results to Xray'
+    env:
+      XRAY_ID: $(XRAY_ID)
+      XRAY_SECRET: $(XRAY_SECRET)
+      JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
+      JIRA_URL: $(JIRA_URL)
+      JIRA_API_TOKEN: $(JIRA_API_TOKEN)
+      JIRA_EMAIL: $(JIRA_EMAIL)
+```
+
+**GitHub Actions:**
+
+```yaml
+steps:
+  - uses: actions/setup-node@v3
+    with:
+      node-version: '18'
+
+  - run: npm ci
+
+  - run: npx playwright test
+
+  - run: |
+      npx xqt import-results \
+        --file playwright-results.json \
+        --testExecKey APIEE-6811
+    env:
+      XRAY_ID: ${{ secrets.XRAY_ID }}
+      XRAY_SECRET: ${{ secrets.XRAY_SECRET }}
+      JIRA_PROJECT_KEY: ${{ secrets.JIRA_PROJECT_KEY }}
+      JIRA_URL: ${{ secrets.JIRA_URL }}
+      JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
+      JIRA_EMAIL: ${{ secrets.JIRA_EMAIL }}
+```
+
+### Benefits
+
+✅ **Updates existing tests** - No duplicate test creation  
+✅ **Automatic mapping** - Via annotations in test code  
+✅ **Evidence attachments** - Screenshots, responses, traces  
+✅ **Detailed reporting** - Retries, worker info, error details  
+✅ **CI/CD ready** - Standard workflow for automation  
+✅ **Single source of truth** - Test code + Xray together
 
 ---
 

package/bin/cli.js

@@ -1,4 +1,4 @@
-#!/usr/bin/env node
+#!/usr/bin/env node
 
 /**
  * @msalaam/xray-qe-toolkit — CLI Entry Point

package/commands/importResults.js

@@ -48,7 +48,7 @@ export default async function importResults(opts = {}) {
 
   if (fileExt === '.json') {
     // Playwright JSON format
-    logger.send(`Importing Playwright JSON results...`);
+    logger.send(`Converting Playwright results to Xray format...`);
 
     const playwrightJson = JSON.parse(fs.readFileSync(filePath, 'utf-8'));
     
@@ -58,11 +58,38 @@ export default async function importResults(opts = {}) {
       projectKey: cfg.jiraProjectKey,
       summary: opts.summary || `Playwright Test Execution - ${new Date().toLocaleString()}`,
       description: opts.description,
+      skipWithoutAnnotations: !!opts.testExecKey, // Skip tests without keys when updating existing execution
     });
 
-    logger.step(`Converted ${xrayJson.tests.length} test results to Xray format`);
+    // Check if we have tests to upload
+    if (xrayJson.tests.length === 0) {
+      logger.warn('⚠️  No tests with Xray annotations found.');
+      logger.info('\nAdd annotations to your tests:');
+      logger.info(`  test.info().annotations.push({ type: 'xray', description: 'APIEE-XXXX' });\n`);
+      process.exit(0);
+    }
+
+    const testsWithKeys = xrayJson.tests.filter(t => t.testKey).length;
+    const testsWithoutKeys = xrayJson.tests.filter(t => !t.testKey).length;
+
+    logger.step(`Found ${testsWithKeys} test(s) with Xray annotations`);
+    
+    if (testsWithoutKeys > 0 && !opts.testExecKey) {
+      logger.info(`  ${testsWithoutKeys} test(s) without annotations will create new tests`);
+    } else if (testsWithoutKeys > 0 && opts.testExecKey) {
+      logger.warn(`  Skipped ${testsWithoutKeys} test(s) without annotations`);
+    }
+
+    // Save debug output if verbose
+    if (opts.verbose) {
+      const debugPath = filePath.replace('.json', '-xray-debug.json');
+      fs.writeFileSync(debugPath, JSON.stringify(xrayJson, null, 2));
+      logger.info(`  Debug: Xray JSON saved to ${path.basename(debugPath)}`);
+    }
+    logger.blank();
 
     // Import to Xray
+    logger.send(`Uploading ${xrayJson.tests.length} test result(s) to Xray...`);
     result = await importResultsXrayJson(cfg, xrayToken, xrayJson);
   } else {
     // JUnit XML format (default)
@@ -73,12 +100,46 @@ export default async function importResults(opts = {}) {
   }
 
   logger.success("Results imported successfully");
+  logger.blank();
 
+  // Display summary for JSON results
+  if (fileExt === '.json' && xrayJson) {
+    const summaryStats = calculateTestSummary(xrayJson.tests);
+    
+    logger.info('Summary:');
+    logger.success(`  ✓ Passed:  ${summaryStats.passed}`)
+;
+    if (summaryStats.failed > 0) {
+      logger.error(`  ✗ Failed:  ${summaryStats.failed}`);
+    }
+    if (summaryStats.skipped > 0) {
+      logger.warn(`  ⊘ Skipped: ${summaryStats.skipped}`);
+    }
+    logger.blank();
+  }
+
+  // Show link to Test Execution
   if (result?.key) {
     logger.link(`View: ${cfg.jiraUrl}/browse/${result.key}`);
   } else if (result?.testExecIssue?.key) {
     logger.link(`View: ${cfg.jiraUrl}/browse/${result.testExecIssue.key}`);
+  } else if (opts.testExecKey) {
+    logger.link(`View: ${cfg.jiraUrl}/browse/${opts.testExecKey}`);
   }
 
   logger.blank();
 }
+
+/**
+ * Calculate test result summary
+ * @param {array} tests - Xray test results
+ * @returns {object} Summary stats
+ */
+function calculateTestSummary(tests) {
+  return {
+    passed: tests.filter(t => t.status === 'PASSED').length,
+    failed: tests.filter(t => t.status === 'FAILED').length,
+    skipped: tests.filter(t => t.status === 'SKIPPED').length,
+    total: tests.length,
+  };
+}

package/lib/playwrightConverter.js

@@ -19,6 +19,7 @@
  * @param {string} options.projectKey - JIRA project key for test creation
  * @param {string} options.summary - Test Execution summary (optional)
  * @param {string} options.description - Test Execution description (optional)
+ * @param {boolean} options.skipWithoutAnnotations - Skip tests without Xray annotations (default: false)
  * @returns {object} Xray JSON format
  */
 export function convertPlaywrightToXray(playwrightJson, options = {}) {
@@ -147,6 +148,11 @@ function convertTest(test, spec, suite, options) {
   // Extract Xray test key from annotations or title
   const testKey = extractTestKey(spec.title, test.annotations);
 
+  // Skip tests without annotations if requested
+  if (options.skipWithoutAnnotations && !testKey) {
+    return null;
+  }
+
   // Map Playwright status to Xray status
   const status = mapStatus(result.status);
 

package/lib/xrayClient.js

@@ -371,15 +371,51 @@ export async function importResults(cfg, xrayToken, xmlBuffer, testExecKey) {
 export async function importResultsXrayJson(cfg, xrayToken, xrayJson) {
   const url = "https://xray.cloud.getxray.app/api/v2/import/execution";
 
-  const response = await axios.post(url, xrayJson, {
-    httpsAgent,
-    headers: {
-      Authorization: `Bearer ${xrayToken}`,
-      "Content-Type": "application/json",
-    },
-  });
-
-  return response.data;
+  try {
+    const response = await axios.post(url, xrayJson, {
+      httpsAgent,
+      headers: {
+        Authorization: `Bearer ${xrayToken}`,
+        "Content-Type": "application/json",
+      },
+    });
+
+    return response.data;
+  } catch (error) {
+    // Enhanced error reporting for Xray API issues
+    if (error.response) {
+      const status = error.response.status;
+      const data = error.response.data;
+      
+      let errorMsg = `Xray API returned ${status} error`;
+      
+      if (typeof data === 'string') {
+        errorMsg += `: ${data}`;
+      } else if (data?.error) {
+        errorMsg += `: ${data.error}`;
+      } else if (data?.message) {
+        errorMsg += `: ${data.message}`;
+      } else if (data) {
+        errorMsg += `: ${JSON.stringify(data)}`;
+      }
+      
+      logger.error(errorMsg);
+      
+      // Common issues help
+      if (status === 400) {
+        logger.warn('\n💡 Common causes of 400 errors:');
+        logger.warn('   • Missing test annotations in Playwright tests');
+        logger.warn('   • Invalid test key format (should be PROJ-123)');
+        logger.warn('   • Test execution key not found');
+        logger.warn('   • Malformed JSON structure');
+        logger.warn('   • Test keys reference non-existent tests\n');
+      }
+      
+      throw new Error(errorMsg);
+    }
+    
+    throw error;
+  }
 }
 
 // ─── Retry wrapper ─────────────────────────────────────────────────────────────

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@msalaam/xray-qe-toolkit",
-  "version": "1.2.0",
+  "version": "1.3.1",
   "description": "Full QE workflow toolkit for Xray Cloud integration — test management, Postman generation, CI pipeline scaffolding, and browser-based review gates for API regression projects.",
   "type": "module",
   "bin": {