@appliqation/automation-sdk 2.1.11 → 2.3.0

package/README.md

@@ -41,6 +41,8 @@ Before you begin, ensure you have:
 
 Follow these steps in order to integrate the SDK with your Playwright tests.
 
+**📌 Important:** Appliqation reporting is **opt-in**. You must either set `APPQ_ENABLE=1` environment variable or add `-- --appq` flag to your test command to send results to Appliqation. Without enabling reporting, tests run normally but results are not reported.
+
 ### Step 1: Install the SDK
 
 Navigate to your Playwright project and install the SDK:
@@ -196,14 +198,46 @@ test('should display user profile', async ({ page }, testInfo) => {
 
 ### Step 5: Run Your Tests
 
-Run your Playwright tests as normal:
+**IMPORTANT:** To enable Appliqation reporting, you must either set `APPQ_ENABLE=1` environment variable or use the `-- --appq` flag.
+
+**Reporting DISABLED (Default):**
+```bash
+# Tests run normally, but results are NOT sent to Appliqation
+npx playwright test
+```
+
+**Reporting ENABLED (Method 1 - Environment Variable - Recommended):**
+```bash
+# Set APPQ_ENABLE to enable reporting (simpler approach)
+APPQ_ENABLE=1 APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx playwright test
+```
 
+**Reporting ENABLED (Method 2 - CLI Flag):**
 ```bash
-APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx playwright test
+# Use -- --appq flag to enable reporting (requires separator)
+APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx playwright test -- --appq
+```
 
+**Examples:**
+```bash
+# Method 1: Using environment variable (recommended)
+APPQ_ENABLE=1 npx playwright test
+APPQ_ENABLE=1 npx playwright test tests/login.spec.js
+APPQ_ENABLE=1 npx playwright test --project=chromium
+APPQ_ENABLE=1 npx playwright test --headed
+
+# Method 2: Using CLI flag (requires -- separator)
+npx playwright test -- --appq
+npx playwright test tests/login.spec.js -- --appq
+npx playwright test --project=chromium -- --appq
+npx playwright test --headed -- --appq
+
+# Bonus: Set run title via CLI flag
+npx playwright test -- --appq --appq_run_title="My Custom Run"
+npx playwright test tests/login.spec.js -- --appq --appq_run_title="Login Tests Sprint 23"
 ```
 
-**What happens:**
+**What happens when `--appq` flag is present:**
 
 1. **Global Setup Runs** (once before all tests):
    - Reads `.env` configuration
@@ -232,9 +266,470 @@ APPLIQATION_RUN_TITLE=yourruntitle APPLIQATION_ENVIRONMENT=test_env_name npx pla
 - Navigate to the scenario you configured
 - See test results in the run matrix!
 
+**Remember:** Results only appear in Appliqation when you enable reporting with `APPQ_ENABLE=1` or `-- --appq` flag!
+
+---
+
+✅ **That's it!** Your tests are now integrated with Appliqation. Remember to enable reporting with `APPQ_ENABLE=1` or `-- --appq`!
+
+---
+
+## Enabling Appliqation Reporting (--appq Flag)
+
+By default, Appliqation reporting is **DISABLED**. You must explicitly enable it by adding the `--appq` flag to your test command.
+
+### Why Opt-In?
+
+This design allows you to:
+- ✅ Run tests locally without creating runs in Appliqation
+- ✅ Debug and develop tests without affecting production data
+- ✅ Control when results are sent to Appliqation portal
+- ✅ Avoid accidental test runs in your dashboard
+
+### How to Enable
+
+You can enable Appliqation reporting in two ways:
+
+#### Method 1: Environment Variable (Recommended - Simpler)
+
+Set the `APPQ_ENABLE` environment variable to `1` or `true`:
+
+```bash
+# Enable reporting with environment variable
+APPQ_ENABLE=1 npx playwright test
+
+# Works with any other flags
+APPQ_ENABLE=1 npx playwright test tests/login.spec.js --headed --project=chromium
+
+# Also works with APPLIQATION_ENABLE
+APPLIQATION_ENABLE=true npx playwright test
+```
+
+#### Method 2: CLI Flag (Requires `--` Separator)
+
+Add `--appq` flag after the `--` separator:
+
+```bash
+# Enable reporting with CLI flag (note the -- separator)
+npx playwright test -- --appq
+
+# Works with any other flags
+npx playwright test tests/login.spec.js --headed --project=chromium -- --appq
+```
+
+**Note:** The `--` separator is required because Playwright doesn't recognize custom flags directly. The environment variable method is simpler and doesn't require the separator.
+
+### Setting Run Title via CLI
+
+You can also specify the run title directly in the command line using the `--appq_run_title` flag:
+
+```bash
+# Set run title via CLI flag
+npx playwright test -- --appq --appq_run_title="My Custom Run Title"
+
+# Works with other flags
+npx playwright test tests/login.spec.js --headed --project=chromium -- --appq --appq_run_title="Login Tests"
+
+# With quotes for titles containing spaces
+npx playwright test -- --appq --appq_run_title="Regression Suite - Sprint 23"
+```
+
+**Priority Order for Run Title:**
+1. Config file `title` option (if specified)
+2. CLI flag `--appq_run_title` (new!)
+3. Environment variable `APPLIQATION_RUN_TITLE`
+4. Auto-generated timestamp (default)
+
+**Examples:**
+```bash
+# Method 1: Environment variable
+APPLIQATION_RUN_TITLE="My Run" npx playwright test -- --appq
+
+# Method 2: CLI flag (overrides environment variable)
+APPLIQATION_RUN_TITLE="Old Title" npx playwright test -- --appq --appq_run_title="New Title"
+# Result: Run title will be "New Title"
+
+# Method 3: Combined usage
+npx playwright test -- --appq --appq_run_title="Sprint 23 Regression"
+```
+
+### What Happens When Reporting is Disabled
+
+When you run tests **without** enabling reporting (no flag, no env var):
+- ✅ Tests execute normally
+- ✅ Execution summary file is still created
+- ❌ No run is created in Appliqation
+- ❌ No results are sent to Appliqation portal
+- ℹ️  Console shows: "Appliqation reporting disabled: Add --appq flag to send results"
+
+**Example:**
+```bash
+# Reporting disabled - no results sent to Appliqation
+npx playwright test
+```
+
+### What Happens When Reporting is Enabled
+
+When you run tests **with** reporting enabled (using flag or env var):
+- ✅ Tests execute normally
+- ✅ Run matrix created in Appliqation
+- ✅ Results sent to Appliqation portal
+- ✅ Execution summary file created
+- ✅ Console shows: "Appliqation reporting enabled: Results will be sent to Appliqation portal"
+
+**Examples:**
+```bash
+# Reporting enabled with environment variable
+APPQ_ENABLE=1 npx playwright test
+
+# Reporting enabled with CLI flag
+npx playwright test -- --appq
+```
+
+---
+
+## Execution Summary Files
+
+After each test run, the SDK automatically creates a timestamped summary file with comprehensive results.
+
+**Location:** `test-results/AppQ_Execution_Summary/`
+
+**Filename Format:** `{run_title}_2025-01-21_14-30-45.txt`
+
+**What's Included:**
+- ✅ Execution time (start, end, duration)
+- ✅ Test counts (submitted, accepted, rejected, passed, failed, skipped)
+- ✅ Run IDs for all created matrices
+- ✅ Detailed error information (duplicates, orphans, backend rejections)
+- ✅ Complete ASCII table (same as terminal output)
+
+**Example:**
+```
+test-results/
+└── AppQ_Execution_Summary/
+    ├── My_Test_Run_2025-01-21_09-30-15.txt
+    ├── My_Test_Run_2025-01-21_14-45-30.txt
+    └── Regression_Suite_2025-01-22_08-00-00.txt
+```
+
+**Features:**
+- Always enabled automatically (no configuration needed)
+- Each run creates a NEW file (never overwrites)
+- If file writing fails, tests continue normally (non-blocking)
+
+**Sample File Content:**
+```
+═══════════════════════════════════════════════════════════
+          APPLIQATION TEST EXECUTION SUMMARY
+═══════════════════════════════════════════════════════════
+
+EXECUTION METADATA:
+─────────────────────────────────────────────────────────
+Start Time:     2025-01-21T14:30:15.123Z
+End Time:       2025-01-21T14:35:45.456Z
+Duration:       5m 30s
+Run Title:      My_Test_Run
+
+╔═══════════════════════════════════════════════════════════╗
+║         Appliqation Test Results Summary                 ║
+╠═══════════════════════════════════════════════════════════╣
+║  Submitted to Backend:                                    ║
+║    Total Submitted:    10                                 ║
+║    ✅ Accepted:         8                                 ║
+║    ❌ Rejected:         2                                 ║
+║                                                           ║
+║  Test Execution Results (Playwright):                    ║
+║    Passed:             8                                  ║
+║    Failed:             0                                  ║
+║    Skipped:            0                                  ║
+║                                                           ║
+║  Not Submitted:                                           ║
+║    Orphan (No UUID):   3                                  ║
+║    Duplicates:         2                                  ║
+╠═══════════════════════════════════════════════════════════╣
+║  Run Matrices Created:     2                              ║
+║    Desktop-Windows            : run_abc123_1234567890     ║
+║    Desktop-Linux              : run_xyz789_0987654321     ║
+╚═══════════════════════════════════════════════════════════╝
+
+DETAILED ERRORS & WARNINGS:
+═══════════════════════════════════════════════════════════
+[Duplicate UUIDs, Orphan Tests, and Backend Rejections details...]
+```
+
+---
+
+## Auto-Tagging Test Cases
+
+The SDK automatically tags test cases with "Appq_automated" (configurable) after their **first successful run**. This helps you track which test cases have been automated and are actively running in your test suite.
+
+### How It Works
+
+1. **Test runs and passes** → SDK submits result to Appliqation
+2. **Backend accepts result** → SDK triggers auto-tagging (fire-and-forget)
+3. **Check if already tagged** → Skip if test case already has the tag
+4. **Add tag** → Test case gets tagged in Appliqation UI
+
+**Key Features:**
+- ✅ **Enabled by default** when Appliqation reporting is enabled
+- ✅ **Fire-and-forget** - tagging failures never block your test runs
+- ✅ **Smart deduplication** - checks before tagging, won't create duplicate tags
+- ✅ **Only accepted results** - backend-rejected results are NOT tagged
+- ✅ **Works for both** single and batch result submissions
+- ✅ **Async execution** - zero impact on test execution performance
+
+### Configuration
+
+#### Environment Variables
+
+Add to your `.env` file:
+
+```bash
+# Auto-Tagging Configuration (optional - all have sensible defaults)
+APPLIQATION_AUTO_TAG_ENABLED=true              # Enable/disable (default: true)
+APPLIQATION_AUTO_TAG_NAME=Appq_automated       # Custom tag name (default: Appq_automated)
+```
+
+#### Playwright Reporter Config
+
+Configure in `playwright.config.js`:
+
+```javascript
+reporter: [
+  ['@appliqation/automation-sdk/playwright/reporter', {
+    apiKey: process.env.APPLIQATION_API_KEY,
+    projectKey: process.env.APPLIQATION_PROJECT_KEY,
+
+    // Auto-tagging options (optional)
+    autoTag: true,                    // Enable auto-tagging (default: true)
+    autoTagName: 'My_Custom_Tag'      // Custom tag name (default: 'Appq_automated')
+  }]
+]
+```
+
+#### Programmatic Configuration
+
+When using the SDK directly:
+
+```javascript
+const { AppliqationClient } = require('@appliqation/automation-sdk');
+
+const client = new AppliqationClient({
+  apiKey: 'your_api_key',
+  projectKey: 'your_project_key',
+
+  // Auto-tagging options
+  options: {
+    autoTag: true,                    // Enable auto-tagging (default: true)
+    autoTagName: 'Automated_Test'     // Custom tag name (default: 'Appq_automated')
+  }
+});
+```
+
+### Disabling Auto-Tagging
+
+If you want to disable auto-tagging:
+
+**Option 1: Environment Variable**
+```bash
+APPLIQATION_AUTO_TAG_ENABLED=false
+```
+
+**Option 2: Config**
+```javascript
+{
+  options: {
+    autoTag: false
+  }
+}
+```
+
+### What You'll See
+
+When auto-tagging is working:
+
+```
+✅ Auto-tagged 3 test case(s) with "Appq_automated"
+```
+
+When test cases are already tagged (second run):
+
+```
+DEBUG: Skipped 3 already-tagged test case(s)
+```
+
+If tagging fails (non-blocking):
+
+```
+⚠️  Auto-tagging failed (non-blocking): Connection timeout
+```
+
+### Troubleshooting
+
+**Q: I don't see the tag in Appliqation UI**
+
+Check:
+1. Is `APPQ_ENABLE=1` set? (Auto-tagging only works when reporting is enabled)
+2. Did the test result get accepted by backend? (Check for backend validation errors)
+3. Check SDK logs for "Auto-tagged X test case(s)" message
+
+**Q: Can I use a custom tag name?**
+
+Yes! Set `APPLIQATION_AUTO_TAG_NAME=Your_Tag_Name` in your `.env` file or use the config options shown above.
+
+**Q: Does tagging failure affect my test results?**
+
+No! Auto-tagging is fire-and-forget. If tagging fails, it's logged as a warning but your test run continues normally and results are still submitted successfully.
+
 ---
 
-✅ **That's it!** Your tests are now integrated with Appliqation.
+## Handling Orphan Tests
+
+### What are Orphan Tests?
+
+**Orphan tests** are tests that execute successfully but **cannot be mapped to Appliqation test cases** because they're missing UUID annotations. When a test runs without a UUID, the SDK cannot link it to a specific test case in your Appliqation project, making the result "orphaned."
+
+```javascript
+// ❌ This test will be orphaned (no UUID annotation)
+test('Login with valid credentials', async ({ page }) => {
+  await page.goto('/login');
+  await page.fill('#username', 'user@example.com');
+  // ... test code
+});
+
+// ✅ This test will be properly mapped (has UUID annotation)
+test('Login with valid credentials', { tag: '@uuid:1154-abc-def' }, async ({ page }) => {
+  await page.goto('/login');
+  await page.fill('#username', 'user@example.com');
+  // ... test code
+});
+```
+
+### Automatic Orphan Run Cleanup
+
+By default, the SDK **automatically prevents corrupted runs** from being created when ALL tests in a run are orphaned:
+
+**Default Behavior:**
+- ✅ If ALL tests lack UUIDs → Run is **deleted** and a clear error message is shown
+- ✅ If SOME tests have UUIDs → Run is **kept**, valid results are submitted, orphans are logged as warnings
+- ✅ CI/CD pipeline **fails with exit code 1** when orphan-only runs are detected
+- ✅ Clear, actionable error message guides users on how to fix the issue
+
+**Why?** Orphan-only runs create empty entries in Appliqation with "N/A" pass rates, which corrupts your analytics and dashboards.
+
+### Error Message Example
+
+When all tests are orphaned, you'll see:
+
+```
+╔════════════════════════════════════════════════════════════════════╗
+║  ❌ RUN CREATION FAILED - ALL TESTS MISSING UUID ANNOTATIONS      ║
+╠════════════════════════════════════════════════════════════════════╣
+║  Project: 1162-MyProject                                           ║
+║  Orphan Tests: 6                                                   ║
+║                                                                    ║
+║  ⚠️  NO RESULTS WERE SUBMITTED TO APPLIQATION                      ║
+║  The test run was automatically deleted to prevent analytics       ║
+║  corruption. All tests are missing UUID annotations.               ║
+╠════════════════════════════════════════════════════════════════════╣
+║  ✅ ACTION REQUIRED: Add UUID Annotations                          ║
+╠════════════════════════════════════════════════════════════════════╣
+║  Option 1: Using test tags (Recommended)                          ║
+║  test('My Test', { tag: '@uuid:123-xxx' }, async ({ page }) => {  ║
+║    // your test code                                               ║
+║  });                                                               ║
+╚════════════════════════════════════════════════════════════════════╝
+```
+
+### Configuration Options
+
+You can customize orphan handling behavior via environment variables:
+
+```env
+# .env file
+
+# Delete runs with only orphan tests (default: true)
+APPLIQATION_DELETE_ORPHAN_RUNS=true
+
+# Exit with error code 1 for orphan-only runs (default: true)
+APPLIQATION_FAIL_ON_ORPHAN_RUNS=true
+```
+
+**Configuration via playwright.config.js:**
+
+```javascript
+reporter: [
+  [
+    '@appliqation/automation-sdk-js/playwright',
+    {
+      deleteOrphanOnlyRuns: true,   // Delete orphan-only runs (default: true)
+      failOnOrphanOnlyRuns: true,   // Fail CI/CD for orphan-only runs (default: true)
+    }
+  ]
+]
+```
+
+### Mixed Scenarios (Some Tests Have UUIDs)
+
+When your test suite has **both valid and orphan tests**, the SDK handles it gracefully:
+
+```javascript
+// Project 1162: 3 tests total
+test('Valid Test 1', { tag: '@uuid:1154-abc' }, async ({ page }) => {
+  // ✅ Will be submitted to Appliqation
+});
+
+test('Orphan Test 1', async ({ page }) => {
+  // ⚠️ Logged as warning, not submitted
+});
+
+test('Valid Test 2', { tag: '@uuid:1155-def' }, async ({ page }) => {
+  // ✅ Will be submitted to Appliqation
+});
+```
+
+**Result:**
+- ✅ Run is kept (because 2 tests have UUIDs)
+- ✅ 2 valid results submitted to Appliqation
+- ⚠️ 1 orphan logged in console and summary file
+- ✅ CI/CD passes (because at least some tests were valid)
+- ✅ Analytics remain accurate (only valid tests counted)
+
+### Troubleshooting FAQ
+
+**Q: Why does my run get deleted?**
+
+Your run is deleted only when **100% of your tests lack UUID annotations**. This prevents corrupted analytics. Add UUIDs to at least one test to keep the run.
+
+**Q: How do I disable automatic deletion?**
+
+Set `APPLIQATION_DELETE_ORPHAN_RUNS=false` in your `.env` file. However, this is not recommended as it will corrupt your analytics with N/A pass rates.
+
+**Q: Can I keep orphan-only runs but still fail CI/CD?**
+
+Yes! Set:
+```env
+APPLIQATION_DELETE_ORPHAN_RUNS=false
+APPLIQATION_FAIL_ON_ORPHAN_RUNS=true
+```
+
+This will create the run in Appliqation but still fail your pipeline, forcing developers to fix UUIDs.
+
+**Q: Where do I find UUIDs for my tests?**
+
+1. Log into Appliqation portal
+2. Navigate to your project
+3. Go to "Test Cases" tab
+4. Find your test case
+5. The UUID is in the format: `{test_nid}-{uuid}` (e.g., `1154-c1f9559c-b978-43cc-9c76-fd539c717cb4`)
+
+**Q: Does orphan cleanup affect test execution?**
+
+No! Cleanup happens **after** all tests complete in the `onEnd()` hook. Test execution is never blocked or interrupted.
+
+**Q: What if deletion fails?**
+
+Deletion is fire-and-forget with error handling. If deletion fails (network issue, permission, etc.), it's logged as an error but doesn't crash your test run. The corrupted run may remain in Appliqation in this rare case.
 
 ---
 
@@ -649,14 +1144,21 @@ const appliqationConfig = {
 
 **Possible causes:**
 
-1. **Orphan tests** - Tests don't have `mapAppqUuid()` calls
+1. **Reporting not enabled** - Missing `APPQ_ENABLE` env var or `--appq` flag (MOST COMMON!)
+   - **Fix (Method 1 - Recommended):** Set `APPQ_ENABLE=1` environment variable
+     - Example: `APPQ_ENABLE=1 npx playwright test`
+   - **Fix (Method 2):** Add `-- --appq` flag to your test command
+     - Example: `npx playwright test -- --appq`
+   - Check console output for: "Appliqation reporting disabled"
+
+2. **Orphan tests** - Tests don't have `mapAppqUuid()` calls
    - **Fix:** Add UUID mapping to all tests
    - Enable `logOrphans: true` to see which tests are orphans
 
-2. **API key invalid** - Authentication failed
+3. **API key invalid** - Authentication failed
    - **Fix:** Regenerate API key from portal
 
-3. **Network issues** - Portal unreachable
+4. **Network issues** - Portal unreachable
    - **Fix:** Check `TEST_APP_URL` is accessible
 
 ---

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@appliqation/automation-sdk",
-  "version": "2.1.11",
+  "version": "2.3.0",
   "description": "Appliqation Automation SDK with API key authentication, custom run titles, and framework-specific reporters",
   "main": "src/index.js",
   "types": "src/index.d.ts",
@@ -114,7 +114,7 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/appliqation/automation-sdk-js"
+    "url": "git+https://github.com/appliqation/automation-sdk-js.git"
   },
   "bugs": {
     "url": "https://github.com/appliqation/automation-sdk-js/issues"

package/src/AppliqationClient.js

@@ -2,6 +2,7 @@ const HttpClient = require('./core/HttpClient');
 const AuthManager = require('./core/AuthManager');
 const RunMatrixService = require('./services/RunMatrixService');
 const ResultService = require('./services/ResultService');
+const TaggingService = require('./services/TaggingService');
 const OrphanTestService = require('./services/OrphanTestService');
 const UuidValidator = require('./utils/UuidValidator');
 const PayloadBuilder = require('./utils/PayloadBuilder');
@@ -99,7 +100,15 @@ class AppliqationClient {
         timeout: normalizedConfig.options?.timeout || 30000,
         retries: normalizedConfig.options?.retries || 3,
         logOrphans: normalizedConfig.options?.logOrphans !== false,
-        logLevel: normalizedConfig.options?.logLevel || 'info'
+        logLevel: normalizedConfig.options?.logLevel || 'info',
+        // Auto-tagging configuration
+        autoTag: normalizedConfig.options?.autoTag !== false,
+        autoTagName: normalizedConfig.options?.autoTagName ||
+                     normalizedConfig.autoTagName ||
+                     process.env.APPLIQATION_AUTO_TAG_NAME ||
+                     'Appq_automated',
+        autoTagBatchSize: normalizedConfig.options?.autoTagBatchSize || 50,
+        autoTagRetries: normalizedConfig.options?.autoTagRetries || 2
       }
     };
 
@@ -112,9 +121,16 @@ class AppliqationClient {
 
     // Initialize services
     this.runMatrix = new RunMatrixService(this.http, this.config);
-    this.results = new ResultService(this.http);
+    this.tagging = new TaggingService(this.http, this.config);
+    this.results = new ResultService(this.http, this.tagging, this.config);
     this.orphans = new OrphanTestService(this.http);
 
+    // Log tagging configuration
+    logger.debug('TaggingService initialized', {
+      enabled: this.tagging.isEnabled(),
+      tagName: this.tagging.tagName
+    });
+
     // Track current run context
     this.currentRun = null;
 
@@ -405,6 +421,35 @@ class AppliqationClient {
     this.currentRun = run;
   }
 
+  /**
+   * Delete a test run
+   * @param {string} runId - Run ID to delete
+   * @param {string} reason - Deletion reason (for audit logging)
+   * @returns {Promise<Object>} Deletion result
+   */
+  async deleteRun(runId, reason = 'orphan_cleanup') {
+    try {
+      logger.info('Deleting run...', { runId, reason });
+
+      const result = await this.runMatrix.delete(runId, reason);
+
+      // Clear from current run if it matches
+      if (this.currentRun && this.currentRun.runId === runId) {
+        this.currentRun = null;
+      }
+
+      logger.info('Run deleted successfully', { runId });
+
+      return result;
+    } catch (error) {
+      logger.error('Failed to delete run', {
+        error: error.message,
+        runId
+      });
+      throw error;
+    }
+  }
+
   /**
    * Validate UUID format
    * @param {string} uuid - UUID to validate