@keber/qa-framework 1.5.2 → 1.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +67 -8
- package/README.md +150 -76
- package/docs/architecture.md +49 -44
- package/docs/folder-structure-guide.md +80 -23
- package/docs/installation.md +53 -43
- package/docs/usage-with-agent.md +52 -64
- package/integrations/ado-powershell/pipelines/azure-pipeline-qa.yml +1 -1
- package/integrations/playwright/README.md +1 -1
- package/package.json +1 -1
- package/scripts/init.js +22 -12
- package/scripts/upgrade.js +139 -11
- package/skills/qa-ado-integration/SKILL.md +2 -2
- package/skills/qa-automation/SKILL.md +6 -6
- package/skills/qa-maintenance/SKILL.md +2 -2
- package/templates/automation-scaffold/playwright.config.ts +3 -1
- package/agent-instructions/00-module-analysis.md +0 -266
- package/agent-instructions/01-spec-generation.md +0 -281
- package/agent-instructions/02-test-plan-generation.md +0 -205
- package/agent-instructions/03-test-case-generation.md +0 -150
- package/agent-instructions/04-automation-generation.md +0 -343
- package/agent-instructions/04b-test-stabilization.md +0 -309
- package/agent-instructions/05-ado-integration.md +0 -247
- package/agent-instructions/06-maintenance.md +0 -135
- package/docs/comparison-matrix.md +0 -131
- package/docs/generalization-decisions.md +0 -203
|
@@ -1,247 +0,0 @@
|
|
|
1
|
-
# Agent Instructions: Azure DevOps Integration
|
|
2
|
-
|
|
3
|
-
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-ado-integration/SKILL.md`.
|
|
4
|
-
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
-
|
|
6
|
-
**File**: `agent-instructions/05-ado-integration.md`
|
|
7
|
-
**Purpose**: Instructions for an agent to create, configure, and maintain Azure DevOps Test Plans integration with Playwright automation results.
|
|
8
|
-
|
|
9
|
-
> **This section is OPTIONAL**. Only use it when `integrations.azureDevOps.enabled = true` in `qa-framework.config.json`.
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## Prerequisites
|
|
14
|
-
|
|
15
|
-
1. Azure DevOps PAT stored in `ADO_PAT` environment variable
|
|
16
|
-
Required scopes: `Work Items (Read, Write)`, `Test Management (Read, Write)`
|
|
17
|
-
2. `ADO_ORG` and `ADO_PROJECT` environment variables set
|
|
18
|
-
3. PowerShell 5.1+ available
|
|
19
|
-
4. `@alex_neo/playwright-azure-reporter` installed in the E2E package
|
|
20
|
-
5. `qa/08-azure-integration/module-registry.json` exists with module paths and plan IDs
|
|
21
|
-
|
|
22
|
-
---
|
|
23
|
-
|
|
24
|
-
## Step 1: Create ADO Test Plan
|
|
25
|
-
|
|
26
|
-
If no Test Plan exists yet:
|
|
27
|
-
|
|
28
|
-
```powershell
|
|
29
|
-
# qa/08-azure-integration/scripts/create-testplan-from-mapping.ps1
|
|
30
|
-
# Reads ado-ids-mapping-{project}.json and creates the plan + suites + TCs in ADO
|
|
31
|
-
# Parameters: -Organization $env:ADO_ORG -Project $env:ADO_PROJECT -Token $env:ADO_PAT
|
|
32
|
-
.\qa\08-azure-integration\scripts\create-testplan-from-mapping.ps1 `
|
|
33
|
-
-Organization $env:ADO_ORG `
|
|
34
|
-
-Project $env:ADO_PROJECT `
|
|
35
|
-
-Token $env:ADO_PAT `
|
|
36
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
After running, the script outputs a `TestPlanId` and `SuiteIds`.
|
|
40
|
-
Record these in `qa/08-azure-integration/module-registry.json`.
|
|
41
|
-
|
|
42
|
-
---
|
|
43
|
-
|
|
44
|
-
## Step 2: Inject ADO IDs into Spec Files
|
|
45
|
-
|
|
46
|
-
Once Test Cases are created in ADO and have Work Item IDs:
|
|
47
|
-
|
|
48
|
-
```powershell
|
|
49
|
-
# Dry run first to preview changes
|
|
50
|
-
.\qa\08-azure-integration\scripts\inject-ado-ids.ps1 `
|
|
51
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
|
|
52
|
-
-TestDir .\qa\07-automation\e2e\tests `
|
|
53
|
-
-DryRun
|
|
54
|
-
|
|
55
|
-
# Apply changes
|
|
56
|
-
.\qa\08-azure-integration\scripts\inject-ado-ids.ps1 `
|
|
57
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
|
|
58
|
-
-TestDir .\qa\07-automation\e2e\tests
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
This converts test titles from:
|
|
62
|
-
```
|
|
63
|
-
'[TC-OPER-CAT-001] Access catalog list @P0'
|
|
64
|
-
```
|
|
65
|
-
to:
|
|
66
|
-
```
|
|
67
|
-
'[22957] Access catalog list @P0'
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
Plus adds annotation:
|
|
71
|
-
```typescript
|
|
72
|
-
test.info().annotations.push({ type: 'TestCase', description: '22957' });
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
---
|
|
76
|
-
|
|
77
|
-
## Step 3: Configure playwright-azure-reporter
|
|
78
|
-
|
|
79
|
-
In `playwright.config.ts`:
|
|
80
|
-
|
|
81
|
-
```typescript
|
|
82
|
-
import type { PlaywrightTestConfig } from '@playwright/test';
|
|
83
|
-
|
|
84
|
-
const config: PlaywrightTestConfig = {
|
|
85
|
-
reporter: [
|
|
86
|
-
['html'],
|
|
87
|
-
['@alex_neo/playwright-azure-reporter', {
|
|
88
|
-
orgUrl: `https://dev.azure.com/${process.env.ADO_ORG}`,
|
|
89
|
-
token: process.env.ADO_PAT,
|
|
90
|
-
planId: Number(process.env.ADO_PLAN_ID),
|
|
91
|
-
projectName: process.env.ADO_PROJECT,
|
|
92
|
-
isDisabled: !process.env.CI, // only publish in CI
|
|
93
|
-
publishTestResultsMode: 'testRun',
|
|
94
|
-
testRunTitle: `QA Run - ${new Date().toISOString()}`,
|
|
95
|
-
uploadAttachments: true,
|
|
96
|
-
attachmentsType: ['screenshot', 'video', 'trace'],
|
|
97
|
-
}],
|
|
98
|
-
],
|
|
99
|
-
};
|
|
100
|
-
export default config;
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
**Note**: Set `isDisabled: !process.env.CI` so the reporter only publishes from CI, not from local runs.
|
|
104
|
-
|
|
105
|
-
---
|
|
106
|
-
|
|
107
|
-
## Step 4: Module Registry
|
|
108
|
-
|
|
109
|
-
Maintain `qa/08-azure-integration/module-registry.json`:
|
|
110
|
-
|
|
111
|
-
```json
|
|
112
|
-
{
|
|
113
|
-
"modules": [
|
|
114
|
-
{
|
|
115
|
-
"name": "operacion",
|
|
116
|
-
"specsPath": "qa/07-automation/e2e/tests/operacion",
|
|
117
|
-
"planId": 0,
|
|
118
|
-
"suiteId": 0,
|
|
119
|
-
"description": "Módulo Operación — 7 submodules"
|
|
120
|
-
}
|
|
121
|
-
]
|
|
122
|
-
}
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
The CI pipeline uses this file to detect which module changed (via `git diff`) and run only the relevant spec.
|
|
126
|
-
|
|
127
|
-
---
|
|
128
|
-
|
|
129
|
-
## Step 5: ADO IDs Mapping File
|
|
130
|
-
|
|
131
|
-
Maintain `qa/08-azure-integration/ado-ids-mapping-{project}.json`:
|
|
132
|
-
|
|
133
|
-
```json
|
|
134
|
-
{
|
|
135
|
-
"planId": 22304,
|
|
136
|
-
"module": "operacion",
|
|
137
|
-
"specsPath": "qa/07-automation/e2e/tests/operacion",
|
|
138
|
-
"suites": [
|
|
139
|
-
{
|
|
140
|
-
"suiteId": 22956,
|
|
141
|
-
"suiteName": "Suite 1.1 - Catálogos de Trabajo",
|
|
142
|
-
"testCases": [
|
|
143
|
-
{ "id": 22957, "title": "Access catalog list @P0", "tags": ["automatable", "playwright"] },
|
|
144
|
-
{ "id": 22958, "title": "Create labor type @P1", "tags": ["automatable", "playwright"] }
|
|
145
|
-
]
|
|
146
|
-
}
|
|
147
|
-
]
|
|
148
|
-
}
|
|
149
|
-
```
|
|
150
|
-
|
|
151
|
-
---
|
|
152
|
-
|
|
153
|
-
## Step 6: Configure CI Pipeline
|
|
154
|
-
|
|
155
|
-
Template in `qa/08-azure-integration/pipelines/azure-pipeline-qa.yml`:
|
|
156
|
-
|
|
157
|
-
```yaml
|
|
158
|
-
trigger:
|
|
159
|
-
branches:
|
|
160
|
-
include:
|
|
161
|
-
- {{CI_TRIGGER_BRANCH}}
|
|
162
|
-
|
|
163
|
-
pool:
|
|
164
|
-
vmImage: 'ubuntu-latest'
|
|
165
|
-
|
|
166
|
-
variables:
|
|
167
|
-
- group: qa-secrets # Must contain: QA_USER_EMAIL, QA_USER_PASSWORD, QA_BASE_URL, ADO_PAT
|
|
168
|
-
|
|
169
|
-
steps:
|
|
170
|
-
- checkout: self
|
|
171
|
-
fetchDepth: 2
|
|
172
|
-
|
|
173
|
-
- task: NodeTool@0
|
|
174
|
-
inputs:
|
|
175
|
-
versionSpec: '20.x'
|
|
176
|
-
displayName: 'Install Node.js 20'
|
|
177
|
-
|
|
178
|
-
- script: |
|
|
179
|
-
cd qa/07-automation/e2e
|
|
180
|
-
npm install
|
|
181
|
-
npx playwright install chromium --with-deps
|
|
182
|
-
displayName: 'Install Playwright'
|
|
183
|
-
|
|
184
|
-
- script: |
|
|
185
|
-
cd qa/07-automation/e2e
|
|
186
|
-
npx playwright test
|
|
187
|
-
displayName: 'Run Playwright tests'
|
|
188
|
-
env:
|
|
189
|
-
QA_BASE_URL: $(QA_BASE_URL)
|
|
190
|
-
QA_USER_EMAIL: $(QA_USER_EMAIL)
|
|
191
|
-
QA_USER_PASSWORD: $(QA_USER_PASSWORD)
|
|
192
|
-
ADO_PAT: $(ADO_PAT)
|
|
193
|
-
ADO_PLAN_ID: $(ADO_PLAN_ID)
|
|
194
|
-
CI: 'true'
|
|
195
|
-
|
|
196
|
-
- task: PublishTestResults@2
|
|
197
|
-
inputs:
|
|
198
|
-
testResultsFormat: 'JUnit'
|
|
199
|
-
testResultsFiles: '**/test-results/*.xml'
|
|
200
|
-
displayName: 'Publish JUnit results'
|
|
201
|
-
condition: always()
|
|
202
|
-
|
|
203
|
-
- task: PublishPipelineArtifact@1
|
|
204
|
-
inputs:
|
|
205
|
-
targetPath: 'qa/07-automation/e2e/playwright-report'
|
|
206
|
-
artifact: 'playwright-html-report'
|
|
207
|
-
displayName: 'Publish HTML report'
|
|
208
|
-
condition: always()
|
|
209
|
-
```
|
|
210
|
-
|
|
211
|
-
---
|
|
212
|
-
|
|
213
|
-
## Syncing ADO TC Titles with Spec Titles
|
|
214
|
-
|
|
215
|
-
If TC titles drift between ADO and the spec files:
|
|
216
|
-
|
|
217
|
-
```powershell
|
|
218
|
-
.\qa\08-azure-integration\scripts\sync-ado-titles.ps1 `
|
|
219
|
-
-Organization $env:ADO_ORG `
|
|
220
|
-
-Project $env:ADO_PROJECT `
|
|
221
|
-
-Token $env:ADO_PAT `
|
|
222
|
-
-MappingFile .\qa\08-azure-integration\ado-ids-mapping-{project}.json `
|
|
223
|
-
-DryRun
|
|
224
|
-
```
|
|
225
|
-
|
|
226
|
-
This reads ADO TC titles and compares them against the spec. Outputs a diff report. Apply with `-DryRun:$false`.
|
|
227
|
-
|
|
228
|
-
---
|
|
229
|
-
|
|
230
|
-
## Credential Security Rules for ADO
|
|
231
|
-
|
|
232
|
-
1. `ADO_PAT` is **never** committed to any file — always comes from env var or CI variable group
|
|
233
|
-
2. `ADO_PAT` is **never** logged, printed, or written to any markdown file
|
|
234
|
-
3. If a PAT appears in any log or output, revoke it immediately in ADO Portal and generate a new one
|
|
235
|
-
4. Variable group `qa-secrets` in Azure DevOps must be scoped to the pipeline only
|
|
236
|
-
5. PAT minimum scopes: `Work Items (Read, Write)`, `Test Management (Read, Write)` — do not use full access PATs
|
|
237
|
-
|
|
238
|
-
---
|
|
239
|
-
|
|
240
|
-
## Troubleshooting
|
|
241
|
-
|
|
242
|
-
| Symptom | Likely Cause | Fix |
|
|
243
|
-
|---------|-------------|-----|
|
|
244
|
-
| Reporter silently does nothing in CI | `isDisabled: true` or `CI` env not set | Verify `process.env.CI` is `'true'` in pipeline |
|
|
245
|
-
| Tests report as "Not Run" in ADO | ADO WI ID not in test title or annotation | Re-run `inject-ado-ids.ps1` |
|
|
246
|
-
| TestRun created but no results attached | `publishTestResultsMode` wrong | Use `'testRun'` not `'testCase'` |
|
|
247
|
-
| PAT error 401 | PAT expired or wrong scopes | Generate new PAT with correct scopes |
|
|
@@ -1,135 +0,0 @@
|
|
|
1
|
-
# Agent Instructions: Maintenance and Updates
|
|
2
|
-
|
|
3
|
-
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-maintenance/SKILL.md`.
|
|
4
|
-
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
-
|
|
6
|
-
**File**: `agent-instructions/06-maintenance.md`
|
|
7
|
-
**Purpose**: Instructions for updating QA artifacts after application code changes.
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## When to use
|
|
12
|
-
|
|
13
|
-
- A developer reports a feature change or new feature in a module
|
|
14
|
-
- A sprint delivers changes to an already-analyzed module
|
|
15
|
-
- A defect is resolved and related `test.skip()` calls need to be reactivated
|
|
16
|
-
- The QA environment is updated and existing specs need to be refreshed
|
|
17
|
-
|
|
18
|
-
---
|
|
19
|
-
|
|
20
|
-
## Step 1: Identify What Changed
|
|
21
|
-
|
|
22
|
-
Gather from the developer or ticket:
|
|
23
|
-
- Which module/submodule was changed?
|
|
24
|
-
- Was it a new feature, a modification, or a bug fix?
|
|
25
|
-
- Are there new routes, fields, or API endpoints?
|
|
26
|
-
- Were any existing behaviors removed or changed?
|
|
27
|
-
|
|
28
|
-
If the change scope is unclear, navigate to the affected module in the QA environment and compare against the existing `00-inventory.md`.
|
|
29
|
-
|
|
30
|
-
---
|
|
31
|
-
|
|
32
|
-
## Step 2: Spec Update Rules
|
|
33
|
-
|
|
34
|
-
### A new field was added to a form
|
|
35
|
-
|
|
36
|
-
1. Add the field to `00-inventory.md` under UI Elements table
|
|
37
|
-
2. If the field has validation: add a business rule to `01-business-rules.md` (`RN-{MODULE}-{next-number}`)
|
|
38
|
-
3. Add TCs to `05-test-scenarios.md`:
|
|
39
|
-
- TC for happy path with the new field
|
|
40
|
-
- TC for required validation (if applicable)
|
|
41
|
-
- TC for any new business rule
|
|
42
|
-
|
|
43
|
-
### A field was removed from the UI
|
|
44
|
-
|
|
45
|
-
1. Mark the field in `00-inventory.md` as `[REMOVED - {YYYY-MM-DD}]` — do not delete the row
|
|
46
|
-
2. If automation tests reference the field selector: update or skip them
|
|
47
|
-
3. If a business rule only applied to that field: mark as `[DEPRECATED - {YYYY-MM-DD}]`
|
|
48
|
-
4. Leave TCs in `05-test-scenarios.md` marked as `[DEPRECATED]` for traceability
|
|
49
|
-
|
|
50
|
-
### A workflow was changed
|
|
51
|
-
|
|
52
|
-
1. Update `02-workflows.md` with the new flow
|
|
53
|
-
2. Mark the old version as `[PREVIOUS - {YYYY-MM-DD}]`
|
|
54
|
-
3. Review all TCs that test the affected steps
|
|
55
|
-
4. Update expected results in `05-test-scenarios.md`
|
|
56
|
-
|
|
57
|
-
### A new submodule was added
|
|
58
|
-
|
|
59
|
-
Follow the full module analysis process in `00-module-analysis.md`.
|
|
60
|
-
|
|
61
|
-
### A defect was fixed
|
|
62
|
-
|
|
63
|
-
1. Find all `test.skip()` calls that reference the DEF-ID or ADO WI
|
|
64
|
-
2. Remove the `test.skip()` wrapper
|
|
65
|
-
3. Run the test at least 2 times with different EXEC_IDX values to confirm stability
|
|
66
|
-
4. Update `06-defects/DEF-{NNN}.md` (or ADO WI) status to Resolved
|
|
67
|
-
5. If the spec assertion was adapted to document the bug (e.g., `toBeFalsy()` for a wrong default), restore the correct assertion
|
|
68
|
-
|
|
69
|
-
---
|
|
70
|
-
|
|
71
|
-
## Step 3: Automation Update Rules
|
|
72
|
-
|
|
73
|
-
### Selectors changed (redesign, framework update)
|
|
74
|
-
|
|
75
|
-
1. Locate the POM file or spec file that contains the old selector
|
|
76
|
-
2. Check `CORRECTIONS` block at top of spec file (convention for documenting selector audits)
|
|
77
|
-
3. Update the selector
|
|
78
|
-
4. Run the affected test 2+ times to confirm the new selector is stable
|
|
79
|
-
5. Add an entry to the `CORRECTIONS` block noting the change and date
|
|
80
|
-
|
|
81
|
-
### New test suite needed
|
|
82
|
-
|
|
83
|
-
Follow `04-automation-generation.md` for the new suite.
|
|
84
|
-
Update `COVERAGE-MAPPING.md` with the new TCs.
|
|
85
|
-
|
|
86
|
-
### Test became flaky after app update
|
|
87
|
-
|
|
88
|
-
Use the 5-layer debugging methodology:
|
|
89
|
-
1. Environment: is QA env stable?
|
|
90
|
-
2. Timeout: did an operation become slower?
|
|
91
|
-
3. Data: did a seed or precondition change?
|
|
92
|
-
4. Selector: did the element structure change?
|
|
93
|
-
5. System: is this a real defect?
|
|
94
|
-
|
|
95
|
-
---
|
|
96
|
-
|
|
97
|
-
## Step 4: Version the Spec
|
|
98
|
-
|
|
99
|
-
At the bottom of each updated spec file, add to the changelog:
|
|
100
|
-
|
|
101
|
-
```markdown
|
|
102
|
-
## Changelog
|
|
103
|
-
|
|
104
|
-
| Version | Date | Description |
|
|
105
|
-
|---------|------|-------------|
|
|
106
|
-
| 1.0 | YYYY-MM-DD | Initial creation |
|
|
107
|
-
| 1.1 | YYYY-MM-DD | Added RN-{MODULE}-{NNN}: {rule title}. TC count: N → M |
|
|
108
|
-
| 1.2 | YYYY-MM-DD | Removed field {name} from UI inventory (deprecated) |
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
---
|
|
112
|
-
|
|
113
|
-
## Step 5: Update qa/README.md
|
|
114
|
-
|
|
115
|
-
After every maintenance session, update the master index:
|
|
116
|
-
- TC count (if changed)
|
|
117
|
-
- Automation status (if changed)
|
|
118
|
-
- Last updated date for the module
|
|
119
|
-
|
|
120
|
-
If a **sprint was fully completed** as part of this maintenance pass (all TCs passing after the changes):
|
|
121
|
-
1. Move the completed sprint checklist from `AGENT-NEXT-STEPS.md` to the `## Sprint History` section of `qa/README.md`
|
|
122
|
-
2. Delete the completed sprint block from `AGENT-NEXT-STEPS.md`
|
|
123
|
-
3. Confirm `AGENT-NEXT-STEPS.md` contains **only one active sprint** — never let it accumulate history
|
|
124
|
-
|
|
125
|
-
Also update `qa/memory/INDEX.md` if any memory files were created or modified.
|
|
126
|
-
|
|
127
|
-
---
|
|
128
|
-
|
|
129
|
-
## Step 6: Write a Session Summary
|
|
130
|
-
|
|
131
|
-
Create `qa/SESSION-SUMMARY-{YYYY-MM-DD}-maintenance.md` documenting:
|
|
132
|
-
- What changed
|
|
133
|
-
- Which files were updated
|
|
134
|
-
- Any new TCs or deprecated TCs
|
|
135
|
-
- Any tests that were reactivated or newly skipped
|
|
@@ -1,131 +0,0 @@
|
|
|
1
|
-
# docs/comparison-matrix.md
|
|
2
|
-
|
|
3
|
-
**Phase 1 Artifact** — Source: analysis of two embedded QA implementations.
|
|
4
|
-
|
|
5
|
-
> **Note**: This document covers the comparative inventory of Repo A (`redacted-repo-web`) and Repo B (`redacted-repo`). It was the primary input for Phase 2 (classification) and Phase 3 (framework design).
|
|
6
|
-
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
## Repository Profiles
|
|
10
|
-
|
|
11
|
-
| Attribute | Repo A (`redacted-repo-web`) | Repo B (`redacted-repo`) |
|
|
12
|
-
|---|---|---|
|
|
13
|
-
| **Application stack** | ASP.NET MVC 5 + Web API 2 + SQL Server | Blazor WebAssembly + .NET + Radzen |
|
|
14
|
-
| **UI component library** | jQuery, Select2, toastr, SweetAlert2, Metronic | Radzen Blazor components |
|
|
15
|
-
| **Authentication** | Form login: RUT + password, `/Seguridad/Login` | Form login: email + password, Blazor identity |
|
|
16
|
-
| **QA environment URL** | `https://redactedURL.com` | `https://redactedURL.com` |
|
|
17
|
-
| **Modules documented** | 1 module (RCL Colación Fría), 7 submodules | 4 modules (Personas, Operación, Sistema, Reportes), 17+ submodules |
|
|
18
|
-
| **Test cases documented** | ~160 TCs across 7 submodules | ~894 TCs across 17+ submodules |
|
|
19
|
-
| **ADO integration level** | Partial (Wiki tools, Plan IDs referenced) | Full (pipeline, reporter, inject-ado-ids.ps1, module-registry.json) |
|
|
20
|
-
| **Playwright suite** | Yes (Sprint 40: 26 P0 tests, 24 pass/2 skip) | Yes (67 automated, 53 pass/2 fail/12 skip) |
|
|
21
|
-
| **Standards folder** | No (`00-standards/` not present) | Yes (naming-conventions, bug-template, TC-template, test-data-guidelines) |
|
|
22
|
-
| **Session retrospectives** | Yes (06-defects used, 2 defects filed) | Extensive (SESSION-SUMMARY-* files at qa root) |
|
|
23
|
-
| **08-azure-integration** | Not present | Fully implemented |
|
|
24
|
-
|
|
25
|
-
---
|
|
26
|
-
|
|
27
|
-
## Item-by-Item Comparison Matrix
|
|
28
|
-
|
|
29
|
-
| Item | Repo A | Repo B | Common? | Different? | Generalizable? | Needs Param? | Out of Framework? |
|
|
30
|
-
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
|
31
|
-
| **8-folder qa/ structure (00–08)** | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
|
|
32
|
-
| **6-file submodule template** | ✅ | ✅ | ✅ | Minor | ✅ | ❌ | ❌ |
|
|
33
|
-
| `00-inventory.md` | ✅ | ✅ | ✅ | ❌ | ✅ | Module code | ❌ |
|
|
34
|
-
| `01-business-rules.md` | ✅ (RN-* IDs) | ✅ (RN-* IDs) | ✅ | ❌ | ✅ | Module code | ❌ |
|
|
35
|
-
| `02-workflows.md` (FL-* IDs) | ✅ | ✅ | ✅ | ❌ | ✅ | Module code | ❌ |
|
|
36
|
-
| `03-roles-permissions.md` | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
|
|
37
|
-
| `04-test-data.md` | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
|
|
38
|
-
| `05-test-scenarios.md` (TC-* IDs) | ✅ | ✅ | ✅ | ❌ | ✅ | Module code | ❌ |
|
|
39
|
-
| **Agent instructions: module analysis** | ✅ (766 lines) | ✅ (834 lines) | ✅ | Minor | ✅ | Project URL | ❌ |
|
|
40
|
-
| **Agent instructions: E2E generation** | ✅ (Sprint-specific) | ❌ | ❌ | Major | ✅ (extract pattern) | Module scope | ❌ |
|
|
41
|
-
| **ADO integration instructions** | ❌ | ✅ (AGENT-ADO-INTEGRATION.md) | ❌ | — | ✅ | Org/Project/IDs | ❌ |
|
|
42
|
-
| **QA Structure Guide** | ✅ (994 lines) | ✅ (1018 lines) | ✅ | Minor wording | ✅ | ❌ | ❌ |
|
|
43
|
-
| **00-standards/ folder** | ❌ | ✅ | ❌ | — | ✅ (adopt from B) | ❌ | ❌ |
|
|
44
|
-
| **Naming conventions doc** | Embedded in structure guide | Standalone doc | Partial | Yes | ✅ | Module codes | ❌ |
|
|
45
|
-
| **Bug report template** | ❌ | ✅ | ❌ | — | ✅ | ❌ | ❌ |
|
|
46
|
-
| **Test case template** | ❌ | ✅ | ❌ | — | ✅ | ❌ | ❌ |
|
|
47
|
-
| **Test data guidelines** | ❌ | ✅ | ❌ | — | ✅ | Env var names | ❌ |
|
|
48
|
-
| **session-summary.md pattern** | ❌ | ✅ (2 examples) | ❌ | — | ✅ | ❌ | ❌ |
|
|
49
|
-
| **Multi-role globalSetup (Playwright)** | ✅ (3 roles, sequential) | ✅ (1 role, single) | ✅ (pattern) | Parameters | ✅ | Role count, selectors | ❌ |
|
|
50
|
-
| **storageState auth reuse** | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
|
|
51
|
-
| **EXEC_IDX / RUN_SLOT date pattern** | ✅ | ❌ (uses Date.now() only) | ❌ | Major | ✅ (adopt from A) | ❌ | ❌ |
|
|
52
|
-
| **3-layer email validation strategy** | ✅ | ❌ | ❌ | — | ✅ (domain-agnostic) | ❌ | Optional |
|
|
53
|
-
| **playwright-azure-reporter** | ❌ | ✅ | ❌ | — | ✅ | planId, suiteId | Optional |
|
|
54
|
-
| **inject-ado-ids.ps1** | ❌ | ✅ | ❌ | — | ✅ | ❌ | Optional |
|
|
55
|
-
| **ADO pipeline (azure-pipeline-qa.yml)** | ❌ | ✅ (367 lines) | ❌ | — | ✅ | Org, project, branch | Optional |
|
|
56
|
-
| **module-registry.json** | ❌ | ✅ | ❌ | — | ✅ | Module paths/IDs | Optional |
|
|
57
|
-
| **ado-ids-mapping.json** | ❌ | ✅ | ❌ | — | ✅ | All IDs | Optional |
|
|
58
|
-
| **Debug artifact policy** | ❌ | ✅ (DEBUG-ARTIFACTS-POLICY.md) | ❌ | — | ✅ | ❌ | ❌ |
|
|
59
|
-
| **UI as source of truth principle** | Implied | ✅ (UI_COMO_FUENTE_DE_VERDAD.md) | ✅ (implicit) | Documentation | ✅ | ❌ | ❌ |
|
|
60
|
-
| **TC origin classification** | ❌ | ✅ (UI-OBSERVADO/PENDIENTE-CODE/BLOQUEADO-PERMISOS) | ❌ | — | ✅ | ❌ | ❌ |
|
|
61
|
-
| **BLAZOR_WASM_AUTOMATION.md** | ❌ | ✅ | ❌ | — | Partial | UI library | Optional |
|
|
62
|
-
| **RADZEN_COMPONENTS_PLAYWRIGHT.md** | ❌ | ✅ | ❌ | — | Partial | UI library | Optional (Blazor adapter) |
|
|
63
|
-
| **DB schema documentation pattern** | ✅ (Mermaid ER) | ❌ | ❌ | — | ✅ (pattern only) | DB naming | Optional |
|
|
64
|
-
| **Correctness criteria** | ✅ (≥2 passes with different EXEC_IDX) | ❌ | ❌ | — | ✅ | ❌ | ❌ |
|
|
65
|
-
| **5-layer debugging methodology** | ✅ | ❌ | ❌ | — | ✅ | ❌ | ❌ |
|
|
66
|
-
| **Page Object Model (POM) pattern** | ✅ (4 POMs) | Inline (no POMs) | Partial | Approach | ✅ (recommend POMs) | ❌ | ❌ |
|
|
67
|
-
| **06-defects/ directory** | ✅ (2 active DEFs used) | ✅ (folder only, no active DEFs) | ✅ | Usage | Conditional | ❌ | Optional |
|
|
68
|
-
| **Seed specs pattern** | ❌ | ✅ (seeds/*.spec.ts excluded from runs) | ❌ | — | ✅ | ❌ | Optional |
|
|
69
|
-
| **automation-feasibility-analysis.md** | ❌ | ✅ | ❌ | — | ✅ | ❌ | ❌ |
|
|
70
|
-
| **Coverage mapping (COVERAGE-MAPPING.md)** | ❌ | ✅ | ❌ | — | ✅ | ADO IDs | Optional |
|
|
71
|
-
| **Project-specific URLs** | ✅ (redacted) | ✅ (redacted) | Both | Different | ❌ (parameterize) | QA_BASE_URL | ❌ |
|
|
72
|
-
| **ADO org / project names** | redacted-organization / redacted-project | redacted-organization / redacted-project | Org same | Project different | ❌ (parameterize) | ADO_ORG, ADO_PROJECT | ❌ |
|
|
73
|
-
| **Login selectors** | `#m_login_signin_submit` (Metronic) | `#email-input`, `#password-input` (Blazor) | Different | Major | ❌ (parameterize) | LOGIN_EMAIL_SELECTOR, etc. | ❌ |
|
|
74
|
-
| **Application-specific form selectors** | `#tbl_redacted`, `#redacted` | `.rz-data-row`, `input.rz-textbox` | None | Major | ❌ | ❌ | Yes (project adapters) |
|
|
75
|
-
| **Test user DNIs / real personal data** | DNIs in test-data.ts, personas.md | None (uses env vars) | None | Major | ❌ | ❌ | Yes (project data) |
|
|
76
|
-
| **Module-specific business data** | redacted | redacted | None | Major | ❌ | ❌ | Yes (project data) |
|
|
77
|
-
| **Sprint references** | redacted sprint | No sprints (session-based) | None | Different | ❌ | ❌ | Yes (project context) |
|
|
78
|
-
| **Developer names** | redacted names | None | None | — | ❌ | ❌ | Yes (project context) |
|
|
79
|
-
| **Internal SOAP email service** | EnvioMailSoapClient | None | None | — | ❌ | ❌ | Yes |
|
|
80
|
-
|
|
81
|
-
---
|
|
82
|
-
|
|
83
|
-
## Summary Counts
|
|
84
|
-
|
|
85
|
-
| Category | Count |
|
|
86
|
-
|---|---|
|
|
87
|
-
| Items common to both repos | 14 |
|
|
88
|
-
| Items only in Repo A, generalizable | 8 |
|
|
89
|
-
| Items only in Repo B, generalizable | 16 |
|
|
90
|
-
| Items that need parameterization | 12 |
|
|
91
|
-
| Items that are purely project-specific (out of framework) | 10+ |
|
|
92
|
-
| Optional/plugin items | 9 |
|
|
93
|
-
|
|
94
|
-
---
|
|
95
|
-
|
|
96
|
-
## Key Differences That Create Design Tensions
|
|
97
|
-
|
|
98
|
-
### 1. Login credential type: RUT vs email
|
|
99
|
-
|
|
100
|
-
- Repo A uses RUT (Chilean national ID) as username
|
|
101
|
-
- Repo B uses email as username
|
|
102
|
-
- **Resolution**: Parameterize via `QA_USER_EMAIL` and `QA_USER_PASSWORD` env vars; the login selector set is a project config parameter in `qa-framework.config.json`
|
|
103
|
-
|
|
104
|
-
### 2. Single-role vs multi-role globalSetup
|
|
105
|
-
|
|
106
|
-
- Repo A: 3 roles (solicitante, encargado, admin) saved sequentially to `.auth/{role}.json`
|
|
107
|
-
- Repo B: 1 role saved to `.auth/session.json`
|
|
108
|
-
- **Resolution**: Scaffold supports both. Config parameter `testUsers[]` drives how many roles are set up. Template defaults to single-role; multi-role is documented as an extension.
|
|
109
|
-
|
|
110
|
-
### 3. ADO integration depth
|
|
111
|
-
|
|
112
|
-
- Repo A: ADO IDs referenced in comments; no pipeline or reporter
|
|
113
|
-
- Repo B: Full pipeline, playwright-azure-reporter, inject-ado-ids.ps1, module-registry.json
|
|
114
|
-
- **Resolution**: ADO integration is a fully optional plugin (`integrations/ado-powershell/`). The framework core does not require it.
|
|
115
|
-
|
|
116
|
-
### 4. Standards folder
|
|
117
|
-
|
|
118
|
-
- Repo B has a mature `00-standards/` folder absent from Repo A
|
|
119
|
-
- **Resolution**: Adopt the standards from Repo B as the framework baseline, since they are the more complete and generalized set.
|
|
120
|
-
|
|
121
|
-
### 5. `06-defects/` directory
|
|
122
|
-
|
|
123
|
-
- Repo A: Used actively (2 bugs filed, test skips reference DEF-IDs)
|
|
124
|
-
- Repo B: Folder exists but no active defect files
|
|
125
|
-
- **Resolution**: See `docs/generalization-decisions.md` §7 for the full evaluation. Decision: **keep as optional, not required by framework core**.
|
|
126
|
-
|
|
127
|
-
### 6. Test naming convention
|
|
128
|
-
|
|
129
|
-
- Repo A: `[Nx] Title @Pp` (sprint-relative sequential numbering)
|
|
130
|
-
- Repo B: `[ADO_ID] Title` (ADO Work Item ID prefix)
|
|
131
|
-
- **Resolution**: Framework uses `[{ID}] {title} @{priority}` where `{ID}` defaults to a local TC-ID (e.g., TC-MOD-SUB-001) and is replaced with ADO WI ID when ADO integration is enabled.
|