@keber/qa-framework 1.9.1 → 1.10.0

package/CHANGELOG.md

@@ -33,6 +33,48 @@ has been removed or parameterized. See `docs/comparison-matrix.md` and `docs/gen
 
 ---
 
+## [1.10.0] - 2026-05-19
+
+### Added
+
+- **`integrations/ado-powershell/pipelines/azure-pipeline-qa.yml`** — pipeline template
+  significantly expanded:
+  - New **Stage 3: API Integration Tests** — separate stage for `qa/07-automation/integration`,
+    with its own `PublishTestResults` and `PublishPipelineArtifact` tasks.
+  - **Cross-project repository checkout** via `resources.repositories` block; each stage
+    now performs an explicit `checkout: source`.
+  - **Smoke test step** in the E2E stage — validates `QA_BASE_URL` is reachable via `curl`
+    before running the suite.
+  - **Commented placeholder** for `AzureCLI@2` login task (Playwright Workspaces / Entra ID)
+    in both E2E and API stages, with `PLAYWRIGHT_SERVICE_URL` env var placeholder.
+  - Optional nightly schedule block (commented out).
+  - `PublishPipelineArtifact` tasks for HTML report and trace artifacts in both stages.
+- **`integrations/playwright-azure-reporter/README.md`** — fully rewritten:
+  - Variables split into local (PAT) vs CI/CD (`SYSTEM_ACCESSTOKEN`) sections.
+  - Config snippet updated to use `ADO_ORG`, `ADO_PROJECT`, `orgUrl` constructed from
+    `ADO_ORG`, and `SYSTEM_ACCESSTOKEN ?? AZURE_TOKEN` token fallback.
+  - New **Playwright Workspaces reporter** section (`@azure/playwright` +
+    `playwright.service.config.ts` + Entra ID authentication notes).
+  - Troubleshooting table expanded with pipeline-specific `401` scenario.
+
+### Changed
+
+- **`azure-pipeline-qa.yml`** — variable names aligned with current conventions:
+  `AZURE_TOKEN` / `ADO_ORG_URL` / `ADO_PROJECT_NAME` replaced by `ADO_ORG` / `ADO_PROJECT`;
+  `ADO_PLAN_ID` split into `ADO_E2E_PLAN_ID` / `ADO_API_PLAN_ID`; `SYSTEM_ACCESSTOKEN`
+  replaces PAT for ADO reporter authentication.
+- **`azure-pipeline-qa.yml`** — E2E job `timeoutInMinutes` increased from 60 to 90.
+- **`azure-pipeline-qa.yml`** — trigger reduced to `main` only (removed `qa` and
+  `release/*` branches); trigger rationale documented in comments.
+- **`azure-pipeline-qa.yml`** — E2E test run consolidated from two `@grep`-tag steps
+  (P0 + P1/P2) into a single `--grep-invert "_inspect|_diag"` run with `continueOnError: true`.
+- **`azure-pipeline-qa.yml`** — cache target changed from `~/.cache/ms-playwright` to
+  `node_modules` per workspace; `cacheHitVar` added to both cache tasks.
+- **`azure-pipeline-qa.yml`** — `pool` block moved above `variables` for readability;
+  `ADO_SYNC_DISABLED` variable removed.
+
+---
+
 ## [1.9.1] - 2026-05-15
 
 ### Added

package/README.md

@@ -1,7 +1,7 @@
 # qa-framework
 
 > Reusable, installable, agent-oriented QA framework for spec-driven automated testing.
-> Current version: **v1.9.0**
+> Current version: **v1.9.1**
 
 ---
 

package/integrations/ado-powershell/pipelines/azure-pipeline-qa.yml

@@ -1,133 +1,278 @@
 # -----------------------------------------------------------------------
 # keber/qa-framework — Azure Pipeline QA Template
 # -----------------------------------------------------------------------
-# Parameterized CI pipeline for running Playwright E2E tests and
-# optionally syncing results to Azure DevOps Test Plans.
+# Parameterized CI pipeline for running Playwright E2E and API tests and
+# syncing results to Azure DevOps Test Plans via SYSTEM_ACCESSTOKEN.
+#
+# Cross-project repository: the test repo lives in a different ADO project.
+# Update 'resources.repositories.name' to match your org/project/repo.
 #
 # Required Pipeline Library Variable Group: "QA-Variables"
-#   QA_BASE_URL          — Target application URL for QA environment
-#   QA_USER_EMAIL        — Default test user email
-#   QA_USER_PASSWORD     — Default test user password (secret)
-#   AZURE_TOKEN          — ADO Personal Access Token (secret)
-#   ADO_ORG_URL          — https://dev.azure.com/your-org
-#   ADO_PROJECT_NAME     — ADO project name
-#   ADO_PLAN_ID          — Test Plan ID
+# Create this group in ADO → Pipelines → Library and add the following:
+#
+#   QA_BASE_URL          — Target application URL  (e.g. https://app-qa.example.com)
+#   QA_USER_EMAIL        — Test user email / RUT
+#   QA_USER_PASSWORD     — Test user password                        [secret]
+#   ADO_ORG              — ADO organization slug  (e.g. garcesfruit-desarrollo)
+#   ADO_PROJECT          — ADO project name        (e.g. gf-portalproveedores)
+#   ADO_E2E_PLAN_ID      — Test Plan ID for the E2E suite            (e.g. 27044)
+#   ADO_API_PLAN_ID      — Test Plan ID for the API/integration suite (e.g. 27046)
+#
+#   NOTE on multiple Plan IDs: ADO variable groups store plain strings — there
+#   is no array type. Use one variable per suite (ADO_E2E_PLAN_ID / ADO_API_PLAN_ID).
+#   If the project has only one suite, a single ADO_PLAN_ID variable is enough.
+#   Results are reported to ADO via System.AccessToken (OAuth); no PAT required.
 # -----------------------------------------------------------------------
 
+# Cross-project repository — update <ADO_SOURCE_PROJECT>/<REPO_NAME> as needed
+resources:
+  repositories:
+    - repository: source
+      type: git
+      name: <ADO_SOURCE_PROJECT>/<REPO_NAME>   # e.g. gf-gestion-qa/gf-portalproveedores
+      ref: refs/heads/main
+
+# Trigger: push and PR to main.
+# The test repo is decoupled from the app repo, so branch triggers are minimal.
 trigger:
   branches:
     include:
       - main
-      - qa
-      - release/*
 
 pr:
   branches:
     include:
       - main
 
+# Optional scheduled run — uncomment and adjust if a nightly regression is needed:
+# schedules:
+#   - cron: '0 6 * * 1-5'   # 06:00 UTC Mon–Fri (03:00 CLT / 02:00 CLST)
+#     displayName: Nightly regression
+#     branches:
+#       include:
+#         - main
+#     always: true
+
+pool:
+  vmImage: 'ubuntu-latest'
+
 variables:
   - group: QA-Variables
   - name: NODE_VERSION
     value: '20.x'
-  - name: QA_DIR
+  - name: E2E_DIR
     value: 'qa/07-automation/e2e'
-  - name: ADO_SYNC_DISABLED
-    value: 'false'
-
-pool:
-  vmImage: 'ubuntu-latest'
+  - name: API_DIR
+    value: 'qa/07-automation/integration'
 
 stages:
   # -----------------------------------------------------------------------
-  # Stage 1: Install & Setup
+  # Stage 1: Install & cache dependencies
   # -----------------------------------------------------------------------
   - stage: Setup
     displayName: 'Install Dependencies'
     jobs:
       - job: Install
-        displayName: 'npm install + Playwright browsers'
+        displayName: 'npm ci + Playwright browsers'
         steps:
+          - checkout: source
+            displayName: 'Checkout source repo'
+
           - task: NodeTool@0
             inputs:
               versionSpec: $(NODE_VERSION)
             displayName: 'Use Node.js $(NODE_VERSION)'
 
           - script: |
-              cd $(QA_DIR)
+              cd $(E2E_DIR)
               npm ci
               npx playwright install chromium --with-deps
-            displayName: 'Install npm deps and Playwright Chromium'
+            displayName: 'Install E2E deps and Playwright Chromium'
 
           - task: CacheBeta@1
             inputs:
-              key: 'playwright | "$(Agent.OS)" | $(QA_DIR)/package-lock.json'
-              path: '~/.cache/ms-playwright'
-            displayName: 'Cache Playwright browsers'
+              key: 'e2e | "$(Agent.OS)" | $(E2E_DIR)/package-lock.json'
+              path: '$(E2E_DIR)/node_modules'
+              cacheHitVar: 'CACHE_RESTORED_E2E'
+            displayName: 'Cache E2E node_modules'
+
+          - script: |
+              cd $(API_DIR)
+              npm ci
+            displayName: 'Install API deps'
+
+          - task: CacheBeta@1
+            inputs:
+              key: 'api | "$(Agent.OS)" | $(API_DIR)/package-lock.json'
+              path: '$(API_DIR)/node_modules'
+              cacheHitVar: 'CACHE_RESTORED_API'
+            displayName: 'Cache API node_modules'
 
   # -----------------------------------------------------------------------
-  # Stage 2: E2E Tests
+  # Stage 2: E2E Tests (Playwright/Chromium)
   # -----------------------------------------------------------------------
   - stage: E2E
-    displayName: 'E2E Tests'
+    displayName: 'E2E Tests (Playwright/Chromium)'
     dependsOn: Setup
     jobs:
-      - job: PlaywrightTests
-        displayName: 'Run Playwright suite'
-        timeoutInMinutes: 60
+      - job: PlaywrightE2E
+        displayName: 'Run E2E suite'
+        timeoutInMinutes: 90
         steps:
+          - checkout: source
+            displayName: 'Checkout source repo'
+
           - task: NodeTool@0
             inputs:
               versionSpec: $(NODE_VERSION)
+            displayName: 'Use Node.js $(NODE_VERSION)'
 
           - script: |
-              cd $(QA_DIR)
+              cd $(E2E_DIR)
               npm ci
               npx playwright install chromium --with-deps
-            displayName: 'Restore dependencies'
+            displayName: 'Restore E2E dependencies'
+
+          - script: |
+              echo "QA_BASE_URL = $(QA_BASE_URL)"
+              HTTP_CODE=$(curl -o /dev/null -s -w "%{http_code}" --max-time 30 "$(QA_BASE_URL)")
+              echo "HTTP response: $HTTP_CODE"
+              if [ "$HTTP_CODE" -lt 200 ] || [ "$HTTP_CODE" -ge 400 ]; then
+                echo "##[error]Smoke test failed — $(QA_BASE_URL) returned HTTP $HTTP_CODE"
+                exit 1
+              fi
+              echo "Smoke test passed (HTTP $HTTP_CODE)"
+            displayName: 'Smoke test — verify QA_BASE_URL is reachable'
+
+          # ---------------------------------------------------------------
+          # Playwright Workspaces (Azure App Testing)
+          # Enable once the ADO service connection is configured.
+          # Prerequisite: the service connection must have the Contributor
+          # role on the workspace in Azure Portal (Access control IAM).
+          # ---------------------------------------------------------------
+          # - task: AzureCLI@2
+          #   displayName: 'Login to Azure (Playwright Workspaces)'
+          #   inputs:
+          #     azureSubscription: '<AZURE_SERVICE_CONNECTION_NAME>'
+          #     scriptType: bash
+          #     scriptLocation: inlineScript
+          #     inlineScript: echo "Authenticated via Entra ID"
 
           - script: |
-              cd $(QA_DIR)
-              npx playwright test --grep @P0
-            displayName: 'Run P0 smoke suite'
+              cd $(E2E_DIR)
+              npx playwright test --project=chromium --ignore-snapshots \
+                --grep-invert "_inspect|_diag"
+            displayName: 'Run E2E tests'
+            continueOnError: true
             env:
-              QA_BASE_URL:       $(QA_BASE_URL)
-              QA_USER_EMAIL:     $(QA_USER_EMAIL)
-              QA_USER_PASSWORD:  $(QA_USER_PASSWORD)
-              AZURE_TOKEN:       $(AZURE_TOKEN)
-              ADO_ORG_URL:       $(ADO_ORG_URL)
-              ADO_PROJECT_NAME:  $(ADO_PROJECT_NAME)
-              ADO_PLAN_ID:       $(ADO_PLAN_ID)
-              ADO_SYNC_DISABLED: $(ADO_SYNC_DISABLED)
-              CI:                'true'
+              QA_BASE_URL:        $(QA_BASE_URL)
+              QA_USER_EMAIL:      $(QA_USER_EMAIL)
+              QA_USER_PASSWORD:   $(QA_USER_PASSWORD)
+              ADO_ORG:            $(ADO_ORG)
+              ADO_PROJECT:        $(ADO_PROJECT)
+              ADO_PLAN_ID:        $(ADO_E2E_PLAN_ID)
+              SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+              CI:                 'true'
+              # PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL)  # Playwright Workspaces endpoint — enable together with the login task
+
+          - task: PublishTestResults@2
+            condition: always()
+            inputs:
+              testResultsFormat:     'JUnit'
+              testResultsFiles:      '**/test-results/**/*.xml'
+              searchFolder:          $(E2E_DIR)
+              mergeTestResults:      true
+              failTaskOnFailedTests: true
+              testRunTitle:          '[Auto E2E] $(Build.BuildNumber)'
+            displayName: 'Publish E2E test results (JUnit)'
+
+          - task: PublishPipelineArtifact@1
+            condition: always()
+            inputs:
+              targetPath: '$(E2E_DIR)/playwright-report'
+              artifact:   'e2e-playwright-report'
+            displayName: 'Publish E2E HTML report'
+
+          - task: PublishPipelineArtifact@1
+            condition: always()
+            inputs:
+              targetPath: '$(E2E_DIR)/test-results'
+              artifact:   'e2e-test-results'
+            displayName: 'Publish E2E trace artifacts'
+
+  # -----------------------------------------------------------------------
+  # Stage 3: API Integration Tests
+  # -----------------------------------------------------------------------
+  - stage: API
+    displayName: 'API Integration Tests'
+    dependsOn: E2E
+    jobs:
+      - job: PlaywrightAPI
+        displayName: 'Run API suite'
+        timeoutInMinutes: 30
+        steps:
+          - checkout: source
+            displayName: 'Checkout source repo'
+
+          - task: NodeTool@0
+            inputs:
+              versionSpec: $(NODE_VERSION)
+            displayName: 'Use Node.js $(NODE_VERSION)'
+
+          - script: |
+              cd $(API_DIR)
+              npm ci
+            displayName: 'Restore API dependencies'
+
+          # ---------------------------------------------------------------
+          # Playwright Workspaces (Azure App Testing)
+          # Enable once the ADO service connection is configured.
+          # ---------------------------------------------------------------
+          # - task: AzureCLI@2
+          #   displayName: 'Login to Azure (Playwright Workspaces)'
+          #   inputs:
+          #     azureSubscription: '<AZURE_SERVICE_CONNECTION_NAME>'
+          #     scriptType: bash
+          #     scriptLocation: inlineScript
+          #     inlineScript: echo "Authenticated via Entra ID"
 
           - script: |
-              cd $(QA_DIR)
-              npx playwright test --grep "@P1|@P2"
-            displayName: 'Run P1/P2 regression suite'
-            condition: succeededOrFailed()
+              cd $(API_DIR)
+              npx playwright test --grep-invert "_inspect|_diag"
+            displayName: 'Run API integration tests'
+            continueOnError: true
             env:
-              QA_BASE_URL:       $(QA_BASE_URL)
-              QA_USER_EMAIL:     $(QA_USER_EMAIL)
-              QA_USER_PASSWORD:  $(QA_USER_PASSWORD)
-              AZURE_TOKEN:       $(AZURE_TOKEN)
-              ADO_ORG_URL:       $(ADO_ORG_URL)
-              ADO_PROJECT_NAME:  $(ADO_PROJECT_NAME)
-              ADO_PLAN_ID:       $(ADO_PLAN_ID)
-              CI:                'true'
+              QA_BASE_URL:        $(QA_BASE_URL)
+              QA_USER_EMAIL:      $(QA_USER_EMAIL)
+              QA_USER_PASSWORD:   $(QA_USER_PASSWORD)
+              ADO_ORG:            $(ADO_ORG)
+              ADO_PROJECT:        $(ADO_PROJECT)
+              ADO_PLAN_ID:        $(ADO_API_PLAN_ID)
+              SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+              CI:                 'true'
+              # PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL)  # Playwright Workspaces endpoint — enable together with the login task
 
           - task: PublishTestResults@2
             condition: always()
             inputs:
-              testResultsFormat: 'JUnit'
-              testResultsFiles:  '$(QA_DIR)/test-results/junit*.xml'
-              mergeTestResults:  true
-              testRunTitle:      'Playwright E2E — $(Build.SourceBranchName)'
-            displayName: 'Publish test results'
+              testResultsFormat:     'JUnit'
+              testResultsFiles:      '**/test-results/**/*.xml'
+              searchFolder:          $(API_DIR)
+              mergeTestResults:      true
+              failTaskOnFailedTests: true
+              testRunTitle:          '[Auto API] $(Build.BuildNumber)'
+            displayName: 'Publish API test results (JUnit)'
+
+          - task: PublishPipelineArtifact@1
+            condition: always()
+            inputs:
+              targetPath: '$(API_DIR)/playwright-report'
+              artifact:   'api-playwright-report'
+            displayName: 'Publish API HTML report'
 
-          - task: PublishBuildArtifacts@1
+          - task: PublishPipelineArtifact@1
             condition: always()
             inputs:
-              pathToPublish: '$(QA_DIR)/playwright-report'
-              artifactName:  'playwright-report'
-            displayName: 'Publish Playwright HTML report'
+              targetPath: '$(API_DIR)/test-results'
+              artifact:   'api-test-results'
+            displayName: 'Publish API trace artifacts'

package/integrations/playwright-azure-reporter/README.md

@@ -1,4 +1,4 @@
-# Integration — playwright-azure-reporter
+# Integration - playwright-azure-reporter
 
 Connects Playwright test results to Azure DevOps Test Plans automatically.
 
@@ -8,42 +8,98 @@ Connects Playwright test results to Azure DevOps Test Plans automatically.
 npm install --save-dev @alex_neo/playwright-azure-reporter
 ```
 
-## Required environment variables
+## Environment variables
+
+### Local development (`.env`)
 
 ```
-ADO_ORG_URL=https://dev.azure.com/your-org
-ADO_PROJECT_NAME=YourProject
+ADO_ORG=<org-slug>
+ADO_PROJECT=<project-name>
 ADO_PLAN_ID=<plan-id>
-AZURE_TOKEN=<personal-access-token>
+AZURE_TOKEN=<personal-access-token> # PAT with Test Management: Read & Write
+```
+
+> NEVER commit `.env` files to source control.
+
+### CI/CD pipeline (`QA-Variables` Library Variable Group)
+
+In the pipeline, `SYSTEM_ACCESSTOKEN` (OAuth) is used instead of a PAT -
+no `AZURE_TOKEN` is needed. The reporter accepts both authentication methods.
+
+```
+ADO_ORG=<org-slug>
+ADO_PROJECT=<project-name>
+ADO_E2E_PLAN_ID=<plan-id>
+ADO_API_PLAN_ID=<plan-id>
 ```
 
-> These variables MUST be injected via CI secrets (Azure Pipeline Library Variable Group).
-> NEVER store tokens in `.env` files committed to source control.
+`SYSTEM_ACCESSTOKEN` is injected automatically by the pipeline via `$(System.AccessToken)`.
 
 ## playwright.config.ts snippet
 
 ```typescript
-import type { PlaywrightTestConfig } from '@playwright/test';
+import { defineConfig } from '@playwright/test';
 
-const config: PlaywrightTestConfig = {
+export default defineConfig({
   reporter: [
     ['list'],
     ['html', { open: 'never' }],
     ['@alex_neo/playwright-azure-reporter', {
-      token:           process.env.AZURE_TOKEN!,
-      planId:          Number(process.env.ADO_PLAN_ID),
-      projectName:     process.env.ADO_PROJECT_NAME!,
-      orgUrl:          process.env.ADO_ORG_URL!,
-      testRunTitle:    `[Automated] ${new Date().toISOString().slice(0, 10)}`,
+      token:                   process.env.SYSTEM_ACCESSTOKEN ?? process.env.AZURE_TOKEN!,
+      planId:                  Number(process.env.ADO_PLAN_ID),
+      projectName:             process.env.ADO_PROJECT!,
+      orgUrl:                  `https://dev.azure.com/${process.env.ADO_ORG}`,
+      testRunTitle:            `[Automated] ${new Date().toISOString().slice(0, 10)}`,
       publishResultsOnFailure: true,
-      isDisabled:      process.env.ADO_SYNC_DISABLED === 'true',
     }],
   ],
-};
+});
+```
+
+> `token` falls back to `AZURE_TOKEN` when `SYSTEM_ACCESSTOKEN` is not present (local runs).
+
+## Playwright Workspaces reporter (optional)
+
+To also publish results to the Azure Playwright Workspaces dashboard, add
+`@azure/playwright` alongside `@alex_neo/playwright-azure-reporter`.
 
-export default config;
+```bash
+npm install --save-dev @azure/playwright
 ```
 
+Create a `playwright.service.config.ts` file next to `playwright.config.ts`:
+
+```typescript
+import { defineConfig } from '@playwright/test';
+import { createAzurePlaywrightConfig } from '@azure/playwright';
+import { DefaultAzureCredential } from '@azure/identity';
+import config from './playwright.config';
+
+export default defineConfig(
+  config,
+  createAzurePlaywrightConfig(config, {
+    credential: new DefaultAzureCredential(),
+  })
+);
+```
+
+Run using the service config in CI:
+
+```bash
+npx playwright test -c playwright.service.config.ts
+```
+
+Required variable (add to `QA-Variables` Library Variable Group):
+
+```
+PLAYWRIGHT_SERVICE_URL=<workspace-endpoint-url>   # Azure Portal -> Playwright workspace -> Get Started
+```
+
+Authentication uses Microsoft Entra ID via the pipeline's Azure service connection
+(`AzureCLI@2` login task). No `PLAYWRIGHT_SERVICE_ACCESS_TOKEN` is needed when
+Entra ID is configured. See `pipelines/azure-pipeline-qa.yml` for the commented
+placeholder steps.
+
 ## Test case title format (required for ADO sync)
 
 The reporter matches test titles to ADO Work Items by the numeric prefix.
@@ -73,16 +129,11 @@ file/describe block to its ADO Test Suite. Example:
 }
 ```
 
-## Disabling ADO sync locally
-
-Set `ADO_SYNC_DISABLED=true` in your `.env` to skip reporting while running
-tests locally.
-
 ## Troubleshooting
 
 | Symptom | Cause | Fix |
 |---------|-------|-----|
-| `401 Unauthorized` | Expired PAT | Regenerate PAT in ADO → User Settings |
+| `401 Unauthorized` (local) | Expired PAT | Regenerate PAT in ADO -> User Settings |
+| `401 Unauthorized` (pipeline) | `SYSTEM_ACCESSTOKEN` not passed | Verify `env: SYSTEM_ACCESSTOKEN: $(System.AccessToken)` in the pipeline step |
 | `Test case not found` | WI ID mismatch | Re-run `inject-ado-ids.ps1` |
-| Results not published | `isDisabled: true` | Remove or set to `false` |
 | Duplicate test runs | Reporter called twice | Check for duplicate reporter arrays |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@keber/qa-framework",
-  "version": "1.9.1",
+  "version": "1.10.0",
   "description": "Reusable spec-driven QA framework for IDE-agent-assisted automated testing. Installable as an npm package. Provides structure, templates, agent instructions, and optional integrations for Playwright and Azure DevOps.",
   "keywords": [
     "qa",