@appliqation/automation-sdk 2.1.11 → 2.2.0

package/README.md

@@ -41,6 +41,8 @@ Before you begin, ensure you have:
 
 Follow these steps in order to integrate the SDK with your Playwright tests.
 
+**📌 Important:** Appliqation reporting is **opt-in**. You must either set `APPQ_ENABLE=1` environment variable or add `-- --appq` flag to your test command to send results to Appliqation. Without enabling reporting, tests run normally but results are not reported.
+
 ### Step 1: Install the SDK
 
 Navigate to your Playwright project and install the SDK:
@@ -196,14 +198,46 @@ test('should display user profile', async ({ page }, testInfo) => {
 
 ### Step 5: Run Your Tests
 
-Run your Playwright tests as normal:
+**IMPORTANT:** To enable Appliqation reporting, you must either set `APPQ_ENABLE=1` environment variable or use the `-- --appq` flag.
+
+**Reporting DISABLED (Default):**
+```bash
+# Tests run normally, but results are NOT sent to Appliqation
+npx playwright test
+```
+
+**Reporting ENABLED (Method 1 - Environment Variable - Recommended):**
+```bash
+# Set APPQ_ENABLE to enable reporting (simpler approach)
+APPQ_ENABLE=1 APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx playwright test
+```
 
+**Reporting ENABLED (Method 2 - CLI Flag):**
 ```bash
-APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx playwright test
+# Use -- --appq flag to enable reporting (requires separator)
+APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx playwright test -- --appq
+```
 
+**Examples:**
+```bash
+# Method 1: Using environment variable (recommended)
+APPQ_ENABLE=1 npx playwright test
+APPQ_ENABLE=1 npx playwright test tests/login.spec.js
+APPQ_ENABLE=1 npx playwright test --project=chromium
+APPQ_ENABLE=1 npx playwright test --headed
+
+# Method 2: Using CLI flag (requires -- separator)
+npx playwright test -- --appq
+npx playwright test tests/login.spec.js -- --appq
+npx playwright test --project=chromium -- --appq
+npx playwright test --headed -- --appq
+
+# Bonus: Set run title via CLI flag
+npx playwright test -- --appq --appq_run_title="My Custom Run"
+npx playwright test tests/login.spec.js -- --appq --appq_run_title="Login Tests Sprint 23"
 ```
 
-**What happens:**
+**What happens when `--appq` flag is present:**
 
 1. **Global Setup Runs** (once before all tests):
    - Reads `.env` configuration
@@ -232,9 +266,208 @@ APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx pla
 - Navigate to the scenario you configured
 - See test results in the run matrix!
 
+**Remember:** Results only appear in Appliqation when you enable reporting with `APPQ_ENABLE=1` or `-- --appq` flag!
+
+---
+
+✅ **That's it!** Your tests are now integrated with Appliqation. Remember to enable reporting with `APPQ_ENABLE=1` or `-- --appq`!
+
+---
+
+## Enabling Appliqation Reporting (--appq Flag)
+
+By default, Appliqation reporting is **DISABLED**. You must explicitly enable it by adding the `--appq` flag to your test command.
+
+### Why Opt-In?
+
+This design allows you to:
+- ✅ Run tests locally without creating runs in Appliqation
+- ✅ Debug and develop tests without affecting production data
+- ✅ Control when results are sent to Appliqation portal
+- ✅ Avoid accidental test runs in your dashboard
+
+### How to Enable
+
+You can enable Appliqation reporting in two ways:
+
+#### Method 1: Environment Variable (Recommended - Simpler)
+
+Set the `APPQ_ENABLE` environment variable to `1` or `true`:
+
+```bash
+# Enable reporting with environment variable
+APPQ_ENABLE=1 npx playwright test
+
+# Works with any other flags
+APPQ_ENABLE=1 npx playwright test tests/login.spec.js --headed --project=chromium
+
+# Also works with APPLIQATION_ENABLE
+APPLIQATION_ENABLE=true npx playwright test
+```
+
+#### Method 2: CLI Flag (Requires `--` Separator)
+
+Add `--appq` flag after the `--` separator:
+
+```bash
+# Enable reporting with CLI flag (note the -- separator)
+npx playwright test -- --appq
+
+# Works with any other flags
+npx playwright test tests/login.spec.js --headed --project=chromium -- --appq
+```
+
+**Note:** The `--` separator is required because Playwright doesn't recognize custom flags directly. The environment variable method is simpler and doesn't require the separator.
+
+### Setting Run Title via CLI
+
+You can also specify the run title directly in the command line using the `--appq_run_title` flag:
+
+```bash
+# Set run title via CLI flag
+npx playwright test -- --appq --appq_run_title="My Custom Run Title"
+
+# Works with other flags
+npx playwright test tests/login.spec.js --headed --project=chromium -- --appq --appq_run_title="Login Tests"
+
+# With quotes for titles containing spaces
+npx playwright test -- --appq --appq_run_title="Regression Suite - Sprint 23"
+```
+
+**Priority Order for Run Title:**
+1. Config file `title` option (if specified)
+2. CLI flag `--appq_run_title` (new!)
+3. Environment variable `APPLIQATION_RUN_TITLE`
+4. Auto-generated timestamp (default)
+
+**Examples:**
+```bash
+# Method 1: Environment variable
+APPLIQATION_RUN_TITLE="My Run" npx playwright test -- --appq
+
+# Method 2: CLI flag (overrides environment variable)
+APPLIQATION_RUN_TITLE="Old Title" npx playwright test -- --appq --appq_run_title="New Title"
+# Result: Run title will be "New Title"
+
+# Method 3: Combined usage
+npx playwright test -- --appq --appq_run_title="Sprint 23 Regression"
+```
+
+### What Happens When Reporting is Disabled
+
+When you run tests **without** enabling reporting (no flag, no env var):
+- ✅ Tests execute normally
+- ✅ Execution summary file is still created
+- ❌ No run is created in Appliqation
+- ❌ No results are sent to Appliqation portal
+- ℹ️  Console shows: "Appliqation reporting disabled: Add --appq flag to send results"
+
+**Example:**
+```bash
+# Reporting disabled - no results sent to Appliqation
+npx playwright test
+```
+
+### What Happens When Reporting is Enabled
+
+When you run tests **with** reporting enabled (using flag or env var):
+- ✅ Tests execute normally
+- ✅ Run matrix created in Appliqation
+- ✅ Results sent to Appliqation portal
+- ✅ Execution summary file created
+- ✅ Console shows: "Appliqation reporting enabled: Results will be sent to Appliqation portal"
+
+**Examples:**
+```bash
+# Reporting enabled with environment variable
+APPQ_ENABLE=1 npx playwright test
+
+# Reporting enabled with CLI flag
+npx playwright test -- --appq
+```
+
 ---
 
-✅ **That's it!** Your tests are now integrated with Appliqation.
+## Execution Summary Files
+
+After each test run, the SDK automatically creates a timestamped summary file with comprehensive results.
+
+**Location:** `test-results/AppQ_Execution_Summary/`
+
+**Filename Format:** `{run_title}_2025-01-21_14-30-45.txt`
+
+**What's Included:**
+- ✅ Execution time (start, end, duration)
+- ✅ Test counts (submitted, accepted, rejected, passed, failed, skipped)
+- ✅ Run IDs for all created matrices
+- ✅ Detailed error information (duplicates, orphans, backend rejections)
+- ✅ Complete ASCII table (same as terminal output)
+
+**Example:**
+```
+test-results/
+└── AppQ_Execution_Summary/
+    ├── My_Test_Run_2025-01-21_09-30-15.txt
+    ├── My_Test_Run_2025-01-21_14-45-30.txt
+    └── Regression_Suite_2025-01-22_08-00-00.txt
+```
+
+**Features:**
+- Always enabled automatically (no configuration needed)
+- Each run creates a NEW file (never overwrites)
+- If file writing fails, tests continue normally (non-blocking)
+
+**Sample File Content:**
+```
+═══════════════════════════════════════════════════════════
+          APPLIQATION TEST EXECUTION SUMMARY
+═══════════════════════════════════════════════════════════
+
+EXECUTION METADATA:
+─────────────────────────────────────────────────────────
+Start Time:     2025-01-21T14:30:15.123Z
+End Time:       2025-01-21T14:35:45.456Z
+Duration:       5m 30s
+Run Title:      My_Test_Run
+
+╔═══════════════════════════════════════════════════════════╗
+║         Appliqation Test Results Summary                 ║
+╠═══════════════════════════════════════════════════════════╣
+║  Submitted to Backend:                                    ║
+║    Total Submitted:    10                                 ║
+║    ✅ Accepted:         8                                 ║
+║    ❌ Rejected:         2                                 ║
+║                                                           ║
+║  Test Execution Results (Playwright):                    ║
+║    Passed:             8                                  ║
+║    Failed:             0                                  ║
+║    Skipped:            0                                  ║
+║                                                           ║
+║  Not Submitted:                                           ║
+║    Orphan (No UUID):   3                                  ║
+║    Duplicates:         2                                  ║
+╠═══════════════════════════════════════════════════════════╣
+║  Run Matrices Created:     2                              ║
+║    Desktop-Windows            : run_abc123_1234567890     ║
+║    Desktop-Linux              : run_xyz789_0987654321     ║
+╚═══════════════════════════════════════════════════════════╝
+
+DETAILED ERRORS & WARNINGS:
+═══════════════════════════════════════════════════════════
+[Duplicate UUIDs, Orphan Tests, and Backend Rejections details...]
+```
+
+---
+
+## Auto-tag Accepted Tests
+
+After a test is automated, the SDK can tag its UUID (default tag: `Appq_automated`) for accepted results only. Tagging is fire-and-forget: failures are logged as warnings and never fail your run.
+
+- Enabled by default when Appq is enabled.
+- Overrides:
+  - Tag name: `APPLIQATION_TAG_NAME=MyTag`
+  - Endpoint (if customized): `APPLIQATION_TAG_ENDPOINT=/api/automation/testcase/tag`
+- Works for both batch and single submissions; backend-rejected results are skipped automatically.
 
 ---
 
@@ -649,14 +882,21 @@ const appliqationConfig = {
 
 **Possible causes:**
 
-1. **Orphan tests** - Tests don't have `mapAppqUuid()` calls
+1. **Reporting not enabled** - Missing `APPQ_ENABLE` env var or `--appq` flag (MOST COMMON!)
+   - **Fix (Method 1 - Recommended):** Set `APPQ_ENABLE=1` environment variable
+     - Example: `APPQ_ENABLE=1 npx playwright test`
+   - **Fix (Method 2):** Add `-- --appq` flag to your test command
+     - Example: `npx playwright test -- --appq`
+   - Check console output for: "Appliqation reporting disabled"
+
+2. **Orphan tests** - Tests don't have `mapAppqUuid()` calls
    - **Fix:** Add UUID mapping to all tests
    - Enable `logOrphans: true` to see which tests are orphans
 
-2. **API key invalid** - Authentication failed
+3. **API key invalid** - Authentication failed
    - **Fix:** Regenerate API key from portal
 
-3. **Network issues** - Portal unreachable
+4. **Network issues** - Portal unreachable
    - **Fix:** Check `TEST_APP_URL` is accessible
 
 ---

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@appliqation/automation-sdk",
-  "version": "2.1.11",
+  "version": "2.2.0",
   "description": "Appliqation Automation SDK with API key authentication, custom run titles, and framework-specific reporters",
   "main": "src/index.js",
   "types": "src/index.d.ts",

package/src/AppliqationClient.js

@@ -112,7 +112,7 @@ class AppliqationClient {
 
     // Initialize services
     this.runMatrix = new RunMatrixService(this.http, this.config);
-    this.results = new ResultService(this.http);
+    this.results = new ResultService(this.http, this.config);
     this.orphans = new OrphanTestService(this.http);
 
     // Track current run context

package/src/playwright/global-setup.js

@@ -29,6 +29,43 @@ const fs = require('fs');
 const path = require('path');
 const { DEFAULT_APPLIQATION_BASE_URL } = require('../constants');
 
+function toBoolean(value) {
+  if (value === undefined || value === null) return undefined;
+  if (typeof value === 'boolean') return value;
+  const normalized = String(value).trim().toLowerCase();
+  if (['1', 'true', 'yes', 'on'].includes(normalized)) return true;
+  if (['0', 'false', 'no', 'off'].includes(normalized)) return false;
+  return undefined;
+}
+
+function parseCliEnableFlag() {
+  const argv = process.argv || [];
+  for (let i = 0; i < argv.length; i++) {
+    const arg = argv[i];
+    if (arg.startsWith('--appq=')) {
+      return toBoolean(arg.split('=')[1]);
+    }
+    if (arg === '--appq') {
+      return true;
+    }
+    if (arg.startsWith('--enable-appq=')) {
+      return toBoolean(arg.split('=')[1]);
+    }
+    if (arg === '--enable-appq') {
+      return true;
+    }
+  }
+  return undefined;
+}
+
+function resolveEnableAppq(config) {
+  const cliValue = parseCliEnableFlag();
+  const envValue = process.env.APPLIQATION_ENABLE ?? process.env.APPQ ?? process.env.APPLIQATION_APPQ;
+  const configValue = config?.enableAppq ?? config?.options?.enableAppq;
+  const resolved = toBoolean(configValue) ?? toBoolean(cliValue) ?? toBoolean(envValue);
+  return resolved !== undefined ? resolved : true;
+}
+
 /**
  * Setup JWT-based browser authentication
  *
@@ -40,11 +77,17 @@ async function globalSetup(config) {
 
   // Extract SDK configuration from reporter config or use env
   const sdkConfig = extractSDKConfig(config);
+  sdkConfig.enableAppq = resolveEnableAppq(sdkConfig);
 
   // Display startup information
   console.log(`🌍 Environment: ${sdkConfig.environment || 'Not specified'}`);
   console.log(`🔗 Test App URL: ${sdkConfig.appUrl || 'Not specified'}`);
 
+  if (sdkConfig.enableAppq === false) {
+    console.log('⚠️  Appq disabled via flag. Skipping JWT setup and backend calls.\n');
+    return;
+  }
+
   // Show Run ID and Timestamp only in DEBUG mode
   if (process.env.LOG_LEVEL === 'DEBUG') {
     const crypto = require('crypto');

package/src/reporters/cypress/CypressReporter.js

@@ -2,6 +2,43 @@ const AppliqationClient = require('../../AppliqationClient');
 const UuidExtractor = require('./UuidExtractor');
 const logger = require('../../utils/logger');
 
+function toBoolean(value) {
+  if (value === undefined || value === null) return undefined;
+  if (typeof value === 'boolean') return value;
+  const normalized = String(value).trim().toLowerCase();
+  if (['1', 'true', 'yes', 'on'].includes(normalized)) return true;
+  if (['0', 'false', 'no', 'off'].includes(normalized)) return false;
+  return undefined;
+}
+
+function parseCliEnableFlag() {
+  const argv = process.argv || [];
+  for (let i = 0; i < argv.length; i++) {
+    const arg = argv[i];
+    if (arg.startsWith('--appq=')) {
+      return toBoolean(arg.split('=')[1]);
+    }
+    if (arg === '--appq') {
+      return true;
+    }
+    if (arg.startsWith('--enable-appq=')) {
+      return toBoolean(arg.split('=')[1]);
+    }
+    if (arg === '--enable-appq') {
+      return true;
+    }
+  }
+  return undefined;
+}
+
+function resolveEnableAppq(config) {
+  const cliValue = parseCliEnableFlag();
+  const envValue = process.env.APPLIQATION_ENABLE ?? process.env.APPQ ?? process.env.APPLIQATION_APPQ;
+  const configValue = config?.enableAppq ?? config?.options?.enableAppq;
+  const resolved = toBoolean(configValue) ?? toBoolean(cliValue) ?? toBoolean(envValue);
+  return resolved !== undefined ? resolved : true;
+}
+
 /**
  * Appliqation Reporter for Cypress
  *
@@ -39,7 +76,8 @@ class CypressReporter {
       title: config.title || process.env.APPLIQATION_RUN_TITLE,
       autoCreateRun: config.autoCreateRun !== false,
       logLevel: config.logLevel || 'info',
-      submitOrphans: config.submitOrphans !== false
+      submitOrphans: config.submitOrphans !== false,
+      enableAppq: resolveEnableAppq(config)
     };
 
     // Validate required config
@@ -55,10 +93,15 @@ class CypressReporter {
       projectKey: this.config.projectKey,
       title: this.config.title,
       options: {
-        logLevel: this.config.logLevel
+        logLevel: this.config.logLevel,
+        enableAppq: this.config.enableAppq
       }
     });
 
+    if (this.config.enableAppq === false) {
+      logger.warn('Appq disabled via flag. Runs/results will not be sent to Appliqation.');
+    }
+
     // Test result storage
     this.testResults = [];
     this.orphanTests = [];
@@ -360,6 +403,10 @@ class CypressReporter {
    * Validate configuration
    */
   validateConfig() {
+    if (this.config.enableAppq === false) {
+      return;
+    }
+
     const required = ['baseUrl', 'apiKey', 'projectKey'];
     const missing = required.filter(key => !this.config[key]);
 

package/src/reporters/jest/JestReporter.js

@@ -2,6 +2,43 @@ const AppliqationClient = require('../../AppliqationClient');
 const UuidExtractor = require('./UuidExtractor');
 const logger = require('../../utils/logger');
 
+function toBoolean(value) {
+  if (value === undefined || value === null) return undefined;
+  if (typeof value === 'boolean') return value;
+  const normalized = String(value).trim().toLowerCase();
+  if (['1', 'true', 'yes', 'on'].includes(normalized)) return true;
+  if (['0', 'false', 'no', 'off'].includes(normalized)) return false;
+  return undefined;
+}
+
+function parseCliEnableFlag() {
+  const argv = process.argv || [];
+  for (let i = 0; i < argv.length; i++) {
+    const arg = argv[i];
+    if (arg.startsWith('--appq=')) {
+      return toBoolean(arg.split('=')[1]);
+    }
+    if (arg === '--appq') {
+      return true;
+    }
+    if (arg.startsWith('--enable-appq=')) {
+      return toBoolean(arg.split('=')[1]);
+    }
+    if (arg === '--enable-appq') {
+      return true;
+    }
+  }
+  return undefined;
+}
+
+function resolveEnableAppq(config) {
+  const cliValue = parseCliEnableFlag();
+  const envValue = process.env.APPLIQATION_ENABLE ?? process.env.APPQ ?? process.env.APPLIQATION_APPQ;
+  const configValue = config?.enableAppq ?? config?.options?.enableAppq;
+  const resolved = toBoolean(configValue) ?? toBoolean(cliValue) ?? toBoolean(envValue);
+  return resolved !== undefined ? resolved : true;
+}
+
 /**
  * Appliqation Reporter for Jest
  *
@@ -39,7 +76,8 @@ class JestReporter {
       title: options.title || process.env.APPLIQATION_RUN_TITLE,
       autoCreateRun: options.autoCreateRun !== false,
       logLevel: options.logLevel || 'info',
-      submitOrphans: options.submitOrphans !== false
+      submitOrphans: options.submitOrphans !== false,
+      enableAppq: resolveEnableAppq(options)
     };
 
     // Validate required config
@@ -55,10 +93,15 @@ class JestReporter {
       projectKey: this.config.projectKey,
       title: this.config.title,
       options: {
-        logLevel: this.config.logLevel
+        logLevel: this.config.logLevel,
+        enableAppq: this.config.enableAppq
       }
     });
 
+    if (this.config.enableAppq === false) {
+      logger.warn('Appq disabled via flag. Runs/results will not be sent to Appliqation.');
+    }
+
     // Test result storage
     this.testResults = [];
     this.orphanTests = [];
@@ -339,6 +382,10 @@ class JestReporter {
    * Validate configuration
    */
   validateConfig() {
+    if (this.config.enableAppq === false) {
+      return;
+    }
+
     const required = ['baseUrl', 'apiKey', 'projectKey'];
     const missing = required.filter(key => !this.config[key]);