qaa-agent 1.9.2 → 1.9.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +206 -179
- package/CLAUDE.md +718 -557
- package/README.md +384 -357
- package/VERSION +1 -1
- package/agents/qa-pipeline-orchestrator.md +1739 -1425
- package/agents/qaa-analyzer.md +0 -1
- package/agents/qaa-bug-detective.md +790 -655
- package/agents/qaa-codebase-mapper.md +50 -1
- package/agents/qaa-discovery.md +421 -384
- package/agents/qaa-e2e-runner.md +715 -577
- package/agents/qaa-executor.md +947 -830
- package/agents/qaa-planner.md +14 -1
- package/agents/qaa-project-researcher.md +533 -400
- package/agents/qaa-scanner.md +77 -1
- package/agents/qaa-testid-injector.md +0 -1
- package/agents/qaa-validator.md +86 -1
- package/bin/install.cjs +375 -253
- package/bin/lib/context7-cache.cjs +299 -0
- package/bin/lib/intent-detector.cjs +488 -0
- package/commands/qa-audit.md +255 -126
- package/commands/qa-create-test.md +666 -365
- package/commands/qa-fix.md +684 -513
- package/commands/qa-map.md +283 -139
- package/commands/qa-pr.md +63 -0
- package/commands/qa-research.md +181 -157
- package/commands/qa-start.md +62 -6
- package/commands/qa-test-report.md +219 -219
- package/frameworks/cypress.json +54 -0
- package/frameworks/jest.json +59 -0
- package/frameworks/playwright.json +58 -0
- package/frameworks/pytest.json +59 -0
- package/frameworks/robot-framework.json +54 -0
- package/frameworks/selenium.json +65 -0
- package/frameworks/vitest.json +57 -0
- package/package.json +5 -2
- package/workflows/qa-analyze.md +100 -4
- package/workflows/qa-from-ticket.md +50 -2
- package/workflows/qa-gap.md +50 -2
- package/workflows/qa-start.md +2048 -1405
- package/workflows/qa-testid.md +50 -2
- package/workflows/qa-validate.md +50 -2
package/CLAUDE.md
CHANGED
|
@@ -1,557 +1,718 @@
|
|
|
1
|
-
# QA Automation Standards
|
|
2
|
-
|
|
3
|
-
This project follows strict QA automation standards. Every test, page object, and analysis produced MUST follow these rules.
|
|
4
|
-
|
|
5
|
-
## Framework Detection
|
|
6
|
-
|
|
7
|
-
Before generating any code, **detect what the project already uses**:
|
|
8
|
-
|
|
9
|
-
1. Check for existing test config files: `cypress.config.ts`, `playwright.config.ts`, `jest.config.ts`, `vitest.config.ts`, `pytest.ini`, etc.
|
|
10
|
-
2. Check `package.json` or `requirements.txt` for test dependencies
|
|
11
|
-
3. Check existing test files for patterns and conventions
|
|
12
|
-
4. **Always match the project's existing framework, language, and conventions**
|
|
13
|
-
|
|
14
|
-
If no framework exists yet, ask the user which one to use. Never assume.
|
|
15
|
-
|
|
16
|
-
## Testing Pyramid
|
|
17
|
-
|
|
18
|
-
Target distribution for every project:
|
|
19
|
-
|
|
20
|
-
```
|
|
21
|
-
/ E2E \ 3-5% (critical path smoke only)
|
|
22
|
-
/ API \ 20-25% (endpoints + contracts)
|
|
23
|
-
/ Integration\ 10-15% (component interactions)
|
|
24
|
-
/ Unit \ 60-70% (business logic, pure functions)
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
Adjust percentages based on the actual app architecture.
|
|
28
|
-
|
|
29
|
-
## Locator Strategy
|
|
30
|
-
|
|
31
|
-
All UI test locators MUST follow this priority order. Never skip to a lower tier without written justification.
|
|
32
|
-
|
|
33
|
-
**Tier 1 -- BEST (always try these first):**
|
|
34
|
-
- Test IDs: `data-testid`, `data-cy`, `data-test` (adapt to framework)
|
|
35
|
-
- Semantic roles: ARIA roles + accessible name
|
|
36
|
-
|
|
37
|
-
**Tier 2 -- GOOD (when Tier 1 not available):**
|
|
38
|
-
- Form labels, placeholders, visible text content
|
|
39
|
-
|
|
40
|
-
**Tier 3 -- ACCEPTABLE (when Tier 1-2 not available):**
|
|
41
|
-
- Alt text, title attributes
|
|
42
|
-
|
|
43
|
-
**Tier 4 -- LAST RESORT (always add a TODO comment):**
|
|
44
|
-
- CSS selectors, XPath -- mark with `// TODO: Request test ID for this element`
|
|
45
|
-
|
|
46
|
-
### Framework-Specific Examples
|
|
47
|
-
|
|
48
|
-
**Playwright:**
|
|
49
|
-
```typescript
|
|
50
|
-
page.getByTestId('submit') // Tier 1
|
|
51
|
-
page.getByRole('button', {name: 'Log in'}) // Tier 1
|
|
52
|
-
page.getByLabel('Email') // Tier 2
|
|
53
|
-
page.locator('.btn') // Tier 4 -- add TODO
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
**Cypress:**
|
|
57
|
-
```typescript
|
|
58
|
-
cy.get('[data-cy="submit"]') // Tier 1
|
|
59
|
-
cy.findByRole('button', {name: 'Log in'}) // Tier 1 (with testing-library)
|
|
60
|
-
cy.get('[data-testid="submit"]') // Tier 1
|
|
61
|
-
cy.contains('Submit') // Tier 2
|
|
62
|
-
cy.get('.btn') // Tier 4 -- add TODO
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
**Selenium / other:**
|
|
66
|
-
```
|
|
67
|
-
driver.findElement(By.cssSelector('[data-testid="submit"]')) // Tier 1
|
|
68
|
-
driver.findElement(By.className('btn')) // Tier 4 -- add TODO
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
## Page Object Model Rules
|
|
72
|
-
|
|
73
|
-
These rules apply regardless of framework:
|
|
74
|
-
|
|
75
|
-
1. **One class/object per page or view** -- no god objects
|
|
76
|
-
2. **No assertions in page objects** -- assertions belong ONLY in test specs
|
|
77
|
-
3. **Locators are properties** -- defined in constructor or as class fields
|
|
78
|
-
4. **Actions return void or the next page** -- for fluent chaining
|
|
79
|
-
5. **State queries return data** -- let the test file decide what to assert
|
|
80
|
-
6. **Every POM extends a shared base** -- shared navigation, screenshots, waits
|
|
81
|
-
|
|
82
|
-
### POM File Structure
|
|
83
|
-
```
|
|
84
|
-
[pages or page-objects or support/page-objects]/
|
|
85
|
-
base/
|
|
86
|
-
BasePage.[ext] -- shared methods
|
|
87
|
-
[feature]/
|
|
88
|
-
[Feature]Page.[ext] -- one file per page
|
|
89
|
-
components/
|
|
90
|
-
[Component].[ext] -- reusable UI components
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
Adapt folder location to match the project's existing conventions.
|
|
94
|
-
|
|
95
|
-
## Test Spec Rules
|
|
96
|
-
|
|
97
|
-
### Every test case MUST have:
|
|
98
|
-
- **Unique ID**: `UT-MODULE-001`, `API-AUTH-001`, `E2E-FLOW-001`
|
|
99
|
-
- **Exact target**: file path + function name, or HTTP method + endpoint
|
|
100
|
-
- **Concrete inputs**: actual values, not "valid data"
|
|
101
|
-
- **Explicit expected outcome**: exact assertion, not "works correctly"
|
|
102
|
-
- **Priority**: P0 (blocks release), P1 (should fix), P2 (nice to have)
|
|
103
|
-
|
|
104
|
-
### BAD assertions (never do this):
|
|
105
|
-
```
|
|
106
|
-
expect(response.status).toBeTruthy()
|
|
107
|
-
expect(data).toBeDefined()
|
|
108
|
-
cy.get('.result').should('exist') // what should it contain?
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
### GOOD assertions (always do this):
|
|
112
|
-
```
|
|
113
|
-
expect(response.status).toBe(200)
|
|
114
|
-
expect(data.name).toBe('Test User')
|
|
115
|
-
cy.get('[data-cy="result"]').should('have.text', 'Todo created successfully')
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
## Naming Conventions
|
|
119
|
-
|
|
120
|
-
Adapt file extensions to match the project's language:
|
|
121
|
-
|
|
122
|
-
| Type | Pattern | Example (.ts) | Example (.cy.ts) |
|
|
123
|
-
|------|---------|---------------|-------------------|
|
|
124
|
-
| Page Object | `[PageName]Page.[ext]` | `LoginPage.ts` | `LoginPage.ts` |
|
|
125
|
-
| Component POM | `[ComponentName].[ext]` | `NavigationBar.ts` | `NavigationBar.ts` |
|
|
126
|
-
| E2E test | `[feature].e2e.[ext]` | `login.e2e.spec.ts` | `login.e2e.cy.ts` |
|
|
127
|
-
| API test | `[resource].api.[ext]` | `users.api.spec.ts` | `users.api.cy.ts` |
|
|
128
|
-
| Unit test | `[module].unit.[ext]` | `validate.unit.spec.ts` | `validate.test.ts` |
|
|
129
|
-
| Fixture | `[domain]-data.[ext]` | `auth-data.ts` | `auth-data.json` |
|
|
130
|
-
|
|
131
|
-
If the project already has naming conventions, **follow those instead**.
|
|
132
|
-
|
|
133
|
-
## Repo Structure
|
|
134
|
-
|
|
135
|
-
Recommended structure -- adapt to match what the project already has:
|
|
136
|
-
|
|
137
|
-
```
|
|
138
|
-
tests/ or cypress/ or __tests__/
|
|
139
|
-
e2e/
|
|
140
|
-
smoke/ # P0 critical path (every PR)
|
|
141
|
-
regression/ # Full suite (nightly)
|
|
142
|
-
api/ # API-level tests
|
|
143
|
-
unit/ # Unit tests
|
|
144
|
-
|
|
145
|
-
pages/ or page-objects/ or support/page-objects/
|
|
146
|
-
base/
|
|
147
|
-
[feature]/
|
|
148
|
-
components/
|
|
149
|
-
|
|
150
|
-
fixtures/ # Test data & factories
|
|
151
|
-
config/ # Test configs (if separate from root)
|
|
152
|
-
reports/ # Generated reports (gitignored)
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
## Test Data Rules
|
|
156
|
-
|
|
157
|
-
- **NEVER** hardcode real credentials
|
|
158
|
-
- Use environment variables with test fallbacks
|
|
159
|
-
- Fixtures go in dedicated folder
|
|
160
|
-
- Each domain gets its own fixture file
|
|
161
|
-
|
|
162
|
-
## Analysis Documents
|
|
163
|
-
|
|
164
|
-
When analyzing a repository, produce these documents:
|
|
165
|
-
|
|
166
|
-
### QA_ANALYSIS.md must include:
|
|
167
|
-
- Architecture overview (system type, language, runtime, entry points, dependencies)
|
|
168
|
-
- Risk assessment (HIGH / MEDIUM / LOW with justification)
|
|
169
|
-
- Top 10 unit test targets with rationale
|
|
170
|
-
- Recommended testing pyramid with percentages adjusted to this app
|
|
171
|
-
- External dependencies with risk levels
|
|
172
|
-
|
|
173
|
-
### TEST_INVENTORY.md must include:
|
|
174
|
-
- Every test case with ID, target, inputs, expected outcome, priority
|
|
175
|
-
- Organized by pyramid level (unit -> integration -> API -> E2E)
|
|
176
|
-
- No test case without an explicit expected outcome
|
|
177
|
-
|
|
178
|
-
## Quality Gates
|
|
179
|
-
|
|
180
|
-
Before delivering ANY QA artifact, verify:
|
|
181
|
-
- [ ] Framework matches what the project already uses
|
|
182
|
-
- [ ] Every test case has an explicit expected outcome with a concrete value
|
|
183
|
-
- [ ] No outcome says "correct", "proper", "appropriate", or "works" without defining what that means
|
|
184
|
-
- [ ] All locators follow the tier hierarchy
|
|
185
|
-
- [ ] No assertions inside page objects
|
|
186
|
-
- [ ] No hardcoded credentials
|
|
187
|
-
- [ ] File naming follows the project's existing conventions (or the standards above if none exist)
|
|
188
|
-
- [ ] Test IDs are unique and follow naming convention
|
|
189
|
-
- [ ] Priority assigned to every test case
|
|
190
|
-
|
|
191
|
-
---
|
|
192
|
-
|
|
193
|
-
## Agent Pipeline
|
|
194
|
-
|
|
195
|
-
The QA automation system runs agents in a defined pipeline. Each stage produces artifacts consumed by the next stage.
|
|
196
|
-
|
|
197
|
-
### Pipeline Stages
|
|
198
|
-
|
|
199
|
-
```
|
|
200
|
-
scan -> codebase-map -> research -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [e2e-runner if E2E tests] -> [bug-detective if failures] -> deliver
|
|
201
|
-
```
|
|
202
|
-
|
|
203
|
-
### Workflow Options
|
|
204
|
-
|
|
205
|
-
**Option 1: Dev-Only Repo (no existing QA repo)**
|
|
206
|
-
Full pipeline from scratch:
|
|
207
|
-
```
|
|
208
|
-
scan -> codebase-map -> research -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [e2e-runner if E2E tests] -> [bug-detective if failures] -> deliver
|
|
209
|
-
```
|
|
210
|
-
Produces: SCAN_MANIFEST.md -> 8 codebase map documents -> QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md -> [TESTID_AUDIT_REPORT.md] -> generation plan -> test files + POMs + fixtures + configs -> VALIDATION_REPORT.md -> [E2E_RUN_REPORT.md] -> branch + PR
|
|
211
|
-
|
|
212
|
-
**Option 2: Dev + Immature QA Repo (existing QA repo with low coverage or quality)**
|
|
213
|
-
Gap-fill and standardize:
|
|
214
|
-
```
|
|
215
|
-
scan both repos -> gap analysis -> fix broken tests -> add missing coverage -> standardize existing -> validate -> deliver
|
|
216
|
-
```
|
|
217
|
-
Produces: SCAN_MANIFEST.md (both repos) -> GAP_ANALYSIS.md -> fixed test files -> new test files -> standardized files -> VALIDATION_REPORT.md -> branch + PR
|
|
218
|
-
|
|
219
|
-
**Option 3: Dev + Mature QA Repo (existing QA repo with solid coverage)**
|
|
220
|
-
Surgical additions only:
|
|
221
|
-
```
|
|
222
|
-
scan both repos -> identify thin coverage -> add only missing tests -> validate -> deliver
|
|
223
|
-
```
|
|
224
|
-
Produces: SCAN_MANIFEST.md (both repos) -> GAP_ANALYSIS.md (thin areas only) -> new test files (targeted) -> VALIDATION_REPORT.md -> branch + PR
|
|
225
|
-
|
|
226
|
-
### Stage Transitions
|
|
227
|
-
|
|
228
|
-
| From | To | Condition |
|
|
229
|
-
|------|----|-----------|
|
|
230
|
-
| scan | research | SCAN_MANIFEST.md exists with > 0 testable surfaces |
|
|
231
|
-
| research | codebase-map | Research complete (non-blocking — continues even if no research output) |
|
|
232
|
-
| codebase-map | analyze | At least 4 of 8 codebase map documents exist |
|
|
233
|
-
| analyze | testid-inject | QA_ANALYSIS.md exists AND frontend components detected |
|
|
234
|
-
| analyze | plan | QA_ANALYSIS.md + TEST_INVENTORY.md exist (skip testid-inject if no frontend) |
|
|
235
|
-
| testid-inject | plan | TESTID_AUDIT_REPORT.md exists with coverage score calculated |
|
|
236
|
-
| plan | generate | Generation plan approved (or auto-approved in auto-advance mode) |
|
|
237
|
-
| generate | validate | All planned test files exist on disk |
|
|
238
|
-
| validate | e2e-runner | VALIDATION_REPORT.md shows PASS AND E2E test files were generated AND live app available |
|
|
239
|
-
| validate | deliver | VALIDATION_REPORT.md shows PASS and no E2E files or --skip-run |
|
|
240
|
-
| e2e-runner | bug-detective | E2E_RUN_REPORT.md exists AND failures need classification |
|
|
241
|
-
| e2e-runner | deliver | E2E_RUN_REPORT.md exists AND all tests pass or failures classified |
|
|
242
|
-
| bug-detective | deliver | FAILURE_CLASSIFICATION_REPORT.md exists |
|
|
243
|
-
|
|
244
|
-
---
|
|
245
|
-
|
|
246
|
-
## Module Boundaries
|
|
247
|
-
|
|
248
|
-
Each agent owns specific artifacts. No agent may produce artifacts assigned to another agent.
|
|
249
|
-
|
|
250
|
-
| Agent | Reads | Produces | Template |
|
|
251
|
-
|-------|-------|----------|----------|
|
|
252
|
-
| qa-scanner | repo source files, package.json, file tree | SCAN_MANIFEST.md | templates/scan-manifest.md |
|
|
253
|
-
| qa-project-researcher | SCAN_MANIFEST.md, repo source files, Context7 MCP, WebSearch, WebFetch | TESTING_STACK.md, FRAMEWORK_CAPABILITIES.md, API_TESTING_STRATEGY.md, E2E_STRATEGY.md | -- |
|
|
254
|
-
| qa-codebase-mapper | SCAN_MANIFEST.md, repo source files, CLAUDE.md | TESTABILITY.md, TEST_SURFACE.md, RISK_MAP.md, CRITICAL_PATHS.md, CODE_PATTERNS.md, API_CONTRACTS.md, TEST_ASSESSMENT.md, COVERAGE_GAPS.md | -- |
|
|
255
|
-
| qa-analyzer | SCAN_MANIFEST.md, codebase map documents, CLAUDE.md | QA_ANALYSIS.md, TEST_INVENTORY.md, QA_REPO_BLUEPRINT.md (Option 1) or GAP_ANALYSIS.md (Option 2/3) | templates/qa-analysis.md, templates/test-inventory.md, templates/qa-repo-blueprint.md, templates/gap-analysis.md |
|
|
256
|
-
| qa-planner | TEST_INVENTORY.md, QA_ANALYSIS.md, codebase map documents | Generation plan (internal) | -- |
|
|
257
|
-
| qa-executor | TEST_INVENTORY.md, codebase map documents, CLAUDE.md | test files, POMs, fixtures, configs | qa-template-engine patterns |
|
|
258
|
-
| qa-validator | generated test files, CLAUDE.md | VALIDATION_REPORT.md (validation mode) or QA_AUDIT_REPORT.md (audit mode) | templates/validation-report.md, templates/qa-audit-report.md |
|
|
259
|
-
| qa-e2e-runner | generated E2E test files, POM files, CLAUDE.md, live application | E2E_RUN_REPORT.md, fixed test/POM files with real locators | -- |
|
|
260
|
-
| qa-testid-injector | repo source files, SCAN_MANIFEST.md, CLAUDE.md | TESTID_AUDIT_REPORT.md, modified source files with data-testid attributes | templates/scan-manifest.md, templates/testid-audit-report.md |
|
|
261
|
-
| qa-bug-detective | test execution results, test source files, CLAUDE.md | FAILURE_CLASSIFICATION_REPORT.md | templates/failure-classification.md |
|
|
262
|
-
|
|
263
|
-
**Rule:** An agent MUST NOT produce artifacts assigned to another agent.
|
|
264
|
-
|
|
265
|
-
**Rule:** An agent MUST read all artifacts listed in its "Reads" column before producing output.
|
|
266
|
-
|
|
267
|
-
---
|
|
268
|
-
|
|
269
|
-
## Verification Commands
|
|
270
|
-
|
|
271
|
-
Every artifact must pass verification before the pipeline advances. Below are the validation rules per artifact type.
|
|
272
|
-
|
|
273
|
-
### SCAN_MANIFEST.md
|
|
274
|
-
- Has > 0 files in File List table
|
|
275
|
-
- Project Detection section is populated (framework, language, component patterns)
|
|
276
|
-
- Testable Surfaces has at least 1 category with entries
|
|
277
|
-
- File priority ordering is present (HIGH/MEDIUM/LOW)
|
|
278
|
-
|
|
279
|
-
### QA_ANALYSIS.md
|
|
280
|
-
- All 6 sections present: Architecture Overview, External Dependencies, Risk Assessment, Top 10 Unit Test Targets, API/Contract Test Targets, Recommended Testing Pyramid
|
|
281
|
-
- Top 10 has exactly 10 entries with module, rationale, and complexity
|
|
282
|
-
- Testing pyramid percentages sum to 100%
|
|
283
|
-
- Risk assessment uses only HIGH/MEDIUM/LOW with justification per item
|
|
284
|
-
|
|
285
|
-
### TEST_INVENTORY.md
|
|
286
|
-
- Every test case has all mandatory fields: ID, target, inputs, expected outcome, priority
|
|
287
|
-
- IDs are unique across the entire document (no duplicates)
|
|
288
|
-
- IDs follow naming convention: UT-MODULE-NNN, INT-MODULE-NNN, API-RESOURCE-NNN, E2E-FLOW-NNN
|
|
289
|
-
- Pyramid tier counts match the summary table
|
|
290
|
-
- No expected outcome says "correct", "proper", "appropriate", or "works" without concrete value
|
|
291
|
-
|
|
292
|
-
### QA_REPO_BLUEPRINT.md
|
|
293
|
-
- Folder structure tree is present with explanations per directory
|
|
294
|
-
- Config files section has actual content (not placeholders)
|
|
295
|
-
- npm scripts defined for smoke, regression, and API test runs
|
|
296
|
-
- CI/CD strategy section includes PR-gate and nightly run configurations
|
|
297
|
-
- Definition of Done checklist is present
|
|
298
|
-
|
|
299
|
-
### VALIDATION_REPORT.md
|
|
300
|
-
- All 4 layers reported per file: Syntax, Structure, Dependencies, Logic
|
|
301
|
-
- Each layer shows PASS or FAIL with details
|
|
302
|
-
- Confidence level assigned: HIGH (all layers pass), MEDIUM (1-2 minor issues), LOW (structural problems)
|
|
303
|
-
- Fix loop log shows iteration count and what was found/fixed per loop
|
|
304
|
-
- Unresolved issues section documents anything not auto-fixed
|
|
305
|
-
|
|
306
|
-
### FAILURE_CLASSIFICATION_REPORT.md
|
|
307
|
-
- Every failure has classification: APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, or INCONCLUSIVE
|
|
308
|
-
- Every failure has confidence level: HIGH, MEDIUM-HIGH, MEDIUM, or LOW
|
|
309
|
-
- Every failure has evidence: code snippet + reasoning explaining the classification
|
|
310
|
-
- No APPLICATION BUG is marked as auto-fixed (application bugs require developer action)
|
|
311
|
-
- Auto-fix log documents what was fixed and at what confidence level
|
|
312
|
-
|
|
313
|
-
### TESTID_AUDIT_REPORT.md
|
|
314
|
-
- Coverage score calculated: existing data-testid count / total interactive elements
|
|
315
|
-
- All proposed data-testid values follow `{context}-{description}-{element-type}` naming convention
|
|
316
|
-
- No duplicate data-testid values within the same page/route scope
|
|
317
|
-
- Elements classified by priority: P0 (form inputs, buttons), P1 (links, images), P2 (containers, decorative)
|
|
318
|
-
- Decision gate threshold applied: >90% SELECTIVE, 50-90% TARGETED, <50% FULL PASS, 0% P0 FIRST
|
|
319
|
-
|
|
320
|
-
### GAP_ANALYSIS.md
|
|
321
|
-
- Coverage map shows all modules from SCAN_MANIFEST.md
|
|
322
|
-
- Missing tests have IDs following naming convention and priorities assigned
|
|
323
|
-
- Broken tests have failure reasons documented with file path and error
|
|
324
|
-
- Quality assessment includes locator tier distribution and assertion quality rating
|
|
325
|
-
- Recommendations are prioritized: fix broken first, then add P0, then P1
|
|
326
|
-
|
|
327
|
-
### QA_AUDIT_REPORT.md
|
|
328
|
-
- All 6 dimensions scored: Locator Quality, Assertion Specificity, POM Compliance, Test Coverage, Naming Convention, Test Data Management
|
|
329
|
-
- Weights sum to 100%: Locator 20%, Assertion 20%, POM 15%, Coverage 20%, Naming 15%, Test Data 10%
|
|
330
|
-
- Overall score matches weighted calculation of dimension scores
|
|
331
|
-
- Critical issues listed with file path, line number, and description
|
|
332
|
-
- Each recommendation has effort estimate: S (small), M (medium), L (large)
|
|
333
|
-
|
|
334
|
-
---
|
|
335
|
-
|
|
336
|
-
## Git Workflow
|
|
337
|
-
|
|
338
|
-
All QA automation output follows these git conventions.
|
|
339
|
-
|
|
340
|
-
### Branch Naming
|
|
341
|
-
|
|
342
|
-
```
|
|
343
|
-
qa/auto-{project}-{date}
|
|
344
|
-
```
|
|
345
|
-
|
|
346
|
-
Examples:
|
|
347
|
-
- `qa/auto-shopflow-2026-03-18`
|
|
348
|
-
- `qa/auto-acme-api-2026-04-01`
|
|
349
|
-
|
|
350
|
-
### Commit Message Format
|
|
351
|
-
|
|
352
|
-
```
|
|
353
|
-
qa({agent}): {description}
|
|
354
|
-
```
|
|
355
|
-
|
|
356
|
-
Examples:
|
|
357
|
-
- `qa(scanner): produce SCAN_MANIFEST.md for shopflow`
|
|
358
|
-
- `qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md`
|
|
359
|
-
- `qa(executor): generate 24 test files with POMs and fixtures`
|
|
360
|
-
- `qa(validator): validate generated tests - PASS with HIGH confidence`
|
|
361
|
-
- `qa(testid-injector): inject 47 data-testid attributes across 12 components`
|
|
362
|
-
- `qa(bug-detective): classify 5 failures - 2 APP BUG, 2 TEST ERROR, 1 ENV ISSUE`
|
|
363
|
-
|
|
364
|
-
### Commit Conventions
|
|
365
|
-
|
|
366
|
-
- One commit per agent stage (scanner produces one commit, analyzer produces one commit, etc.)
|
|
367
|
-
- Descriptive messages that include artifact names and counts
|
|
368
|
-
- Never commit .env files, credentials, or secrets
|
|
369
|
-
- Include modified file count in commit body when relevant
|
|
370
|
-
|
|
371
|
-
### PR Template
|
|
372
|
-
|
|
373
|
-
PR description must include:
|
|
374
|
-
- Analysis summary (architecture type, framework, risk areas)
|
|
375
|
-
- Test counts by pyramid level (unit: N, integration: N, API: N, E2E: N)
|
|
376
|
-
- Coverage metrics (modules covered, estimated line coverage)
|
|
377
|
-
- Validation pass/fail status with confidence level
|
|
378
|
-
- Link to VALIDATION_REPORT.md in the PR files
|
|
379
|
-
|
|
380
|
-
---
|
|
381
|
-
|
|
382
|
-
## Team Settings
|
|
383
|
-
|
|
384
|
-
Configuration for multi-agent pipeline execution.
|
|
385
|
-
|
|
386
|
-
### Concurrent Execution
|
|
387
|
-
|
|
388
|
-
Agents in the same pipeline stage can run in parallel when their inputs are independent. Examples:
|
|
389
|
-
- qa-testid-injector and qa-analyzer can run simultaneously after scan completes (both read SCAN_MANIFEST.md)
|
|
390
|
-
- Multiple qa-executor instances can generate tests for different modules in parallel
|
|
391
|
-
|
|
392
|
-
Agents in different pipeline stages MUST respect stage ordering. A downstream agent cannot start until all its required inputs exist on disk.
|
|
393
|
-
|
|
394
|
-
### Worktree Isolation
|
|
395
|
-
|
|
396
|
-
Each agent operates on the same branch. No worktree splits are needed for this system. Agents coordinate through file-based artifacts -- each agent writes its own files and reads other agents' files.
|
|
397
|
-
|
|
398
|
-
### Dependency Ordering
|
|
399
|
-
|
|
400
|
-
Respect stage transitions from the Agent Pipeline section:
|
|
401
|
-
1. qa-scanner runs first (no dependencies)
|
|
402
|
-
2. qa-project-researcher runs after codebase-map (depends on SCAN_MANIFEST.md, uses Context7 MCP — non-blocking, pipeline continues if it fails)
|
|
403
|
-
3. qa-codebase-mapper runs after scanner (depends on SCAN_MANIFEST.md, spawns 4 parallel focus agents)
|
|
404
|
-
4. qa-analyzer and qa-testid-injector run after codebase-map (both depend on SCAN_MANIFEST.md + codebase map documents + research documents if available)
|
|
405
|
-
5. qa-planner runs after analyzer (depends on QA_ANALYSIS.md + TEST_INVENTORY.md + codebase map documents)
|
|
406
|
-
6. qa-executor runs after planner (depends on generation plan + codebase map documents + research documents if available + Context7 MCP)
|
|
407
|
-
7. qa-validator runs after executor (depends on generated test files)
|
|
408
|
-
8. qa-e2e-runner runs after validator (depends on E2E test files + live application + Context7 MCP)
|
|
409
|
-
9. qa-bug-detective runs after e2e-runner or validator if failures (depends on test results + Context7 MCP)
|
|
410
|
-
|
|
411
|
-
### Auto-Advance Mode
|
|
412
|
-
|
|
413
|
-
When auto-advance is enabled, pipeline stages advance automatically when:
|
|
414
|
-
1. The previous stage completes
|
|
415
|
-
2. All output artifacts from the previous stage exist on disk
|
|
416
|
-
3. All output artifacts pass their verification commands (from Verification Commands section)
|
|
417
|
-
|
|
418
|
-
No human confirmation is needed between stages in auto-advance mode. The pipeline pauses only at explicit checkpoint tasks or when verification fails.
|
|
419
|
-
|
|
420
|
-
---
|
|
421
|
-
|
|
422
|
-
## Agent Coordination
|
|
423
|
-
|
|
424
|
-
Rules governing how agents communicate and hand off work through artifacts.
|
|
425
|
-
|
|
426
|
-
### Read-Before-Write Rules
|
|
427
|
-
|
|
428
|
-
Every agent MUST read its required inputs before producing any output. Failure to read inputs produces low-quality, inconsistent artifacts.
|
|
429
|
-
|
|
430
|
-
| Agent | MUST Read Before Producing Output |
|
|
431
|
-
|-------|-----------------------------------|
|
|
432
|
-
| qa-scanner | package.json (or equivalent), folder tree structure, all source file extensions
|
|
433
|
-
| qa-project-researcher | SCAN_MANIFEST.md, repo source files, MY_PREFERENCES.md. Uses Context7 MCP as primary source. |
|
|
434
|
-
| qa-analyzer | SCAN_MANIFEST.md (complete, verified), CLAUDE.md (all QA standards sections), research documents (if available) |
|
|
435
|
-
| qa-planner | TEST_INVENTORY.md (all test cases), QA_ANALYSIS.md (architecture and risk context) |
|
|
436
|
-
| qa-executor | TEST_INVENTORY.md (test cases to implement), CLAUDE.md (POM rules, locator hierarchy, assertion rules, naming conventions, quality gates), research documents (if available). Must query Context7 MCP before generating code. |
|
|
437
|
-
| qa-validator | CLAUDE.md (quality gates, locator tiers, assertion rules), all generated test files to validate |
|
|
438
|
-
| qa-testid-injector | SCAN_MANIFEST.md (component file list), CLAUDE.md (data-testid Convention section for naming rules) |
|
|
439
|
-
| qa-bug-detective | Test execution output (stdout/stderr, exit codes), test source files (to read the failing code), CLAUDE.md (classification rules), research documents (if available). Must query Context7 MCP before auto-fixing. |
|
|
440
|
-
| qa-e2e-runner | Generated E2E test files, POM files, CLAUDE.md, live application, research documents (if available). Must query Context7 MCP before fixing locators. |
|
|
441
|
-
|
|
442
|
-
### Handoff Patterns
|
|
443
|
-
|
|
444
|
-
Agents communicate exclusively through file-based artifacts:
|
|
445
|
-
|
|
446
|
-
1. **Producer writes** -- Agent completes its task and writes output artifact(s) to disk
|
|
447
|
-
2. **Pipeline verifies** -- Output artifacts pass verification commands before advancing
|
|
448
|
-
3. **Consumer reads** -- Next agent reads the artifact(s) as its first action
|
|
449
|
-
4. **No direct communication** -- Agents never pass data in memory or through environment variables between stages
|
|
450
|
-
|
|
451
|
-
### Quality Gates Per Artifact
|
|
452
|
-
|
|
453
|
-
Before the pipeline advances past any stage, the produced artifact(s) must pass verification:
|
|
454
|
-
|
|
455
|
-
| Stage Complete | Artifact | Gate |
|
|
456
|
-
|----------------|----------|------|
|
|
457
|
-
| scan | SCAN_MANIFEST.md | > 0 files listed, project detection populated |
|
|
458
|
-
| analyze | QA_ANALYSIS.md + TEST_INVENTORY.md | All sections present, IDs unique, pyramid sums to 100% |
|
|
459
|
-
| testid-inject | TESTID_AUDIT_REPORT.md | Coverage score calculated, naming convention compliant |
|
|
460
|
-
| plan | Generation plan | Test cases mapped to output files, no unassigned cases |
|
|
461
|
-
| generate | Test files + POMs | All planned files exist, imports resolve, syntax valid |
|
|
462
|
-
| validate | VALIDATION_REPORT.md | All 4 layers checked per file, confidence level assigned |
|
|
463
|
-
| deliver | Branch + PR | Branch pushed, PR created with required description sections |
|
|
464
|
-
|
|
465
|
-
### Error Recovery
|
|
466
|
-
|
|
467
|
-
If an agent fails or produces an artifact that does not pass verification:
|
|
468
|
-
1. Log the failure with the specific verification check that failed
|
|
469
|
-
2. Retry the agent (max 3 attempts per stage)
|
|
470
|
-
3. If still failing after 3 attempts, pause the pipeline and report the blocked stage
|
|
471
|
-
4. Do not advance to the next stage with a failed artifact
|
|
472
|
-
|
|
473
|
-
---
|
|
474
|
-
|
|
475
|
-
## data-testid Convention
|
|
476
|
-
|
|
477
|
-
All `data-testid` attributes injected by qa-testid-injector and referenced by generated tests MUST follow this naming convention.
|
|
478
|
-
|
|
479
|
-
### Naming Pattern
|
|
480
|
-
|
|
481
|
-
```
|
|
482
|
-
{context}-{description}-{element-type}
|
|
483
|
-
```
|
|
484
|
-
|
|
485
|
-
All values are **kebab-case**. No camelCase, no underscores, no periods.
|
|
486
|
-
|
|
487
|
-
### Context Derivation
|
|
488
|
-
|
|
489
|
-
1. **Page-level context**: Derived from the component filename or route
|
|
490
|
-
- `LoginPage.tsx` -> context is `login`
|
|
491
|
-
- `ProductDetailPage.tsx` -> context is `product-detail`
|
|
492
|
-
- Route `/settings/profile` -> context is `settings-profile`
|
|
493
|
-
|
|
494
|
-
2. **Component-level context**: Derived from the component name
|
|
495
|
-
- `<NavBar>` -> context is `navbar`
|
|
496
|
-
- `<ShoppingCart>` -> context is `shopping-cart`
|
|
497
|
-
- `<UserAvatar>` -> context is `user-avatar`
|
|
498
|
-
|
|
499
|
-
3. **Nested context**: Parent-child hierarchy, max 3 levels deep
|
|
500
|
-
- `checkout-shipping-address-input` (page -> section -> field)
|
|
501
|
-
- `dashboard-sidebar-nav-link` (page -> component -> element)
|
|
502
|
-
- Never exceed 3 levels: `a-b-c-element` is the maximum depth
|
|
503
|
-
|
|
504
|
-
4. **Dynamic list items**: Use template literals with unique keys
|
|
505
|
-
```tsx
|
|
506
|
-
data-testid={`product-${product.id}-card`}
|
|
507
|
-
data-testid={`order-${order.id}-status-badge`}
|
|
508
|
-
```
|
|
509
|
-
|
|
510
|
-
### Element Type Suffix Table
|
|
511
|
-
|
|
512
|
-
Every `data-testid` value ends with a suffix indicating the element type:
|
|
513
|
-
|
|
514
|
-
| Element | Suffix | Example |
|
|
515
|
-
|---------|--------|---------|
|
|
516
|
-
| `<button>` | `-btn` | `login-submit-btn` |
|
|
517
|
-
| `<input>` | `-input` | `login-email-input` |
|
|
518
|
-
| `<select>` | `-select` | `settings-language-select` |
|
|
519
|
-
| `<textarea>` | `-textarea` | `feedback-comment-textarea` |
|
|
520
|
-
| `<a>` (link) | `-link` | `navbar-profile-link` |
|
|
521
|
-
| `<form>` | `-form` | `checkout-payment-form` |
|
|
522
|
-
| `<img>` | `-img` | `product-hero-img` |
|
|
523
|
-
| `<table>` | `-table` | `users-list-table` |
|
|
524
|
-
| `<tr>` (row) | `-row` | `users-item-row` |
|
|
525
|
-
| `<dialog>/<modal>` | `-modal` | `confirm-delete-modal` |
|
|
526
|
-
| `<div>` container | `-container` | `dashboard-stats-container` |
|
|
527
|
-
| `<ul>/<ol>` list | `-list` | `notifications-list` |
|
|
528
|
-
| `<li>` item | `-item` | `notifications-item` |
|
|
529
|
-
| dropdown menu | `-dropdown` | `navbar-user-dropdown` |
|
|
530
|
-
| tab | `-tab` | `settings-security-tab` |
|
|
531
|
-
| checkbox | `-checkbox` | `terms-accept-checkbox` |
|
|
532
|
-
| radio | `-radio` | `shipping-express-radio` |
|
|
533
|
-
| toggle/switch | `-toggle` | `notifications-enabled-toggle` |
|
|
534
|
-
| badge/chip | `-badge` | `cart-count-badge` |
|
|
535
|
-
| alert/toast | `-alert` | `error-validation-alert` |
|
|
536
|
-
|
|
537
|
-
### Third-Party Component Handling
|
|
538
|
-
|
|
539
|
-
When adding `data-testid` to third-party UI library components, use this priority order:
|
|
540
|
-
|
|
541
|
-
1. **Props passthrough** (preferred): If the library supports passing `data-testid` directly as a prop
|
|
542
|
-
```tsx
|
|
543
|
-
<MuiButton data-testid="checkout-pay-btn">Pay</MuiButton>
|
|
544
|
-
```
|
|
545
|
-
|
|
546
|
-
2. **Wrapper div**: If the library does not support prop passthrough, wrap with a `<div>` that has the `data-testid`
|
|
547
|
-
```tsx
|
|
548
|
-
<div data-testid="checkout-pay-container">
|
|
549
|
-
<ThirdPartyButton>Pay</ThirdPartyButton>
|
|
550
|
-
</div>
|
|
551
|
-
```
|
|
552
|
-
|
|
553
|
-
3. **inputProps / slotProps** (MUI-specific): Use component-specific prop APIs
|
|
554
|
-
```tsx
|
|
555
|
-
<TextField inputProps={{ 'data-testid': 'login-email-input' }} />
|
|
556
|
-
<Autocomplete slotProps={{ input: { 'data-testid': 'search-query-input' } }} />
|
|
557
|
-
```
|
|
1
|
+
# QA Automation Standards
|
|
2
|
+
|
|
3
|
+
This project follows strict QA automation standards. Every test, page object, and analysis produced MUST follow these rules.
|
|
4
|
+
|
|
5
|
+
## Framework Detection
|
|
6
|
+
|
|
7
|
+
Before generating any code, **detect what the project already uses**:
|
|
8
|
+
|
|
9
|
+
1. Check for existing test config files: `cypress.config.ts`, `playwright.config.ts`, `jest.config.ts`, `vitest.config.ts`, `pytest.ini`, etc.
|
|
10
|
+
2. Check `package.json` or `requirements.txt` for test dependencies
|
|
11
|
+
3. Check existing test files for patterns and conventions
|
|
12
|
+
4. **Always match the project's existing framework, language, and conventions**
|
|
13
|
+
|
|
14
|
+
If no framework exists yet, ask the user which one to use. Never assume.
|
|
15
|
+
|
|
16
|
+
## Testing Pyramid
|
|
17
|
+
|
|
18
|
+
Target distribution for every project:
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
/ E2E \ 3-5% (critical path smoke only)
|
|
22
|
+
/ API \ 20-25% (endpoints + contracts)
|
|
23
|
+
/ Integration\ 10-15% (component interactions)
|
|
24
|
+
/ Unit \ 60-70% (business logic, pure functions)
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Adjust percentages based on the actual app architecture.
|
|
28
|
+
|
|
29
|
+
## Locator Strategy
|
|
30
|
+
|
|
31
|
+
All UI test locators MUST follow this priority order. Never skip to a lower tier without written justification.
|
|
32
|
+
|
|
33
|
+
**Tier 1 -- BEST (always try these first):**
|
|
34
|
+
- Test IDs: `data-testid`, `data-cy`, `data-test` (adapt to framework)
|
|
35
|
+
- Semantic roles: ARIA roles + accessible name
|
|
36
|
+
|
|
37
|
+
**Tier 2 -- GOOD (when Tier 1 not available):**
|
|
38
|
+
- Form labels, placeholders, visible text content
|
|
39
|
+
|
|
40
|
+
**Tier 3 -- ACCEPTABLE (when Tier 1-2 not available):**
|
|
41
|
+
- Alt text, title attributes
|
|
42
|
+
|
|
43
|
+
**Tier 4 -- LAST RESORT (always add a TODO comment):**
|
|
44
|
+
- CSS selectors, XPath -- mark with `// TODO: Request test ID for this element`
|
|
45
|
+
|
|
46
|
+
### Framework-Specific Examples
|
|
47
|
+
|
|
48
|
+
**Playwright:**
|
|
49
|
+
```typescript
|
|
50
|
+
page.getByTestId('submit') // Tier 1
|
|
51
|
+
page.getByRole('button', {name: 'Log in'}) // Tier 1
|
|
52
|
+
page.getByLabel('Email') // Tier 2
|
|
53
|
+
page.locator('.btn') // Tier 4 -- add TODO
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
**Cypress:**
|
|
57
|
+
```typescript
|
|
58
|
+
cy.get('[data-cy="submit"]') // Tier 1
|
|
59
|
+
cy.findByRole('button', {name: 'Log in'}) // Tier 1 (with testing-library)
|
|
60
|
+
cy.get('[data-testid="submit"]') // Tier 1
|
|
61
|
+
cy.contains('Submit') // Tier 2
|
|
62
|
+
cy.get('.btn') // Tier 4 -- add TODO
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
**Selenium / other:**
|
|
66
|
+
```
|
|
67
|
+
driver.findElement(By.cssSelector('[data-testid="submit"]')) // Tier 1
|
|
68
|
+
driver.findElement(By.className('btn')) // Tier 4 -- add TODO
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Page Object Model Rules
|
|
72
|
+
|
|
73
|
+
These rules apply regardless of framework:
|
|
74
|
+
|
|
75
|
+
1. **One class/object per page or view** -- no god objects
|
|
76
|
+
2. **No assertions in page objects** -- assertions belong ONLY in test specs
|
|
77
|
+
3. **Locators are properties** -- defined in constructor or as class fields
|
|
78
|
+
4. **Actions return void or the next page** -- for fluent chaining
|
|
79
|
+
5. **State queries return data** -- let the test file decide what to assert
|
|
80
|
+
6. **Every POM extends a shared base** -- shared navigation, screenshots, waits
|
|
81
|
+
|
|
82
|
+
### POM File Structure
|
|
83
|
+
```
|
|
84
|
+
[pages or page-objects or support/page-objects]/
|
|
85
|
+
base/
|
|
86
|
+
BasePage.[ext] -- shared methods
|
|
87
|
+
[feature]/
|
|
88
|
+
[Feature]Page.[ext] -- one file per page
|
|
89
|
+
components/
|
|
90
|
+
[Component].[ext] -- reusable UI components
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Adapt folder location to match the project's existing conventions.
|
|
94
|
+
|
|
95
|
+
## Test Spec Rules
|
|
96
|
+
|
|
97
|
+
### Every test case MUST have:
|
|
98
|
+
- **Unique ID**: `UT-MODULE-001`, `API-AUTH-001`, `E2E-FLOW-001`
|
|
99
|
+
- **Exact target**: file path + function name, or HTTP method + endpoint
|
|
100
|
+
- **Concrete inputs**: actual values, not "valid data"
|
|
101
|
+
- **Explicit expected outcome**: exact assertion, not "works correctly"
|
|
102
|
+
- **Priority**: P0 (blocks release), P1 (should fix), P2 (nice to have)
|
|
103
|
+
|
|
104
|
+
### BAD assertions (never do this):
|
|
105
|
+
```
|
|
106
|
+
expect(response.status).toBeTruthy()
|
|
107
|
+
expect(data).toBeDefined()
|
|
108
|
+
cy.get('.result').should('exist') // what should it contain?
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
### GOOD assertions (always do this):
|
|
112
|
+
```
|
|
113
|
+
expect(response.status).toBe(200)
|
|
114
|
+
expect(data.name).toBe('Test User')
|
|
115
|
+
cy.get('[data-cy="result"]').should('have.text', 'Todo created successfully')
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
## Naming Conventions
|
|
119
|
+
|
|
120
|
+
Adapt file extensions to match the project's language:
|
|
121
|
+
|
|
122
|
+
| Type | Pattern | Example (.ts) | Example (.cy.ts) |
|
|
123
|
+
|------|---------|---------------|-------------------|
|
|
124
|
+
| Page Object | `[PageName]Page.[ext]` | `LoginPage.ts` | `LoginPage.ts` |
|
|
125
|
+
| Component POM | `[ComponentName].[ext]` | `NavigationBar.ts` | `NavigationBar.ts` |
|
|
126
|
+
| E2E test | `[feature].e2e.[ext]` | `login.e2e.spec.ts` | `login.e2e.cy.ts` |
|
|
127
|
+
| API test | `[resource].api.[ext]` | `users.api.spec.ts` | `users.api.cy.ts` |
|
|
128
|
+
| Unit test | `[module].unit.[ext]` | `validate.unit.spec.ts` | `validate.test.ts` |
|
|
129
|
+
| Fixture | `[domain]-data.[ext]` | `auth-data.ts` | `auth-data.json` |
|
|
130
|
+
|
|
131
|
+
If the project already has naming conventions, **follow those instead**.
|
|
132
|
+
|
|
133
|
+
## Repo Structure
|
|
134
|
+
|
|
135
|
+
Recommended structure -- adapt to match what the project already has:
|
|
136
|
+
|
|
137
|
+
```
|
|
138
|
+
tests/ or cypress/ or __tests__/
|
|
139
|
+
e2e/
|
|
140
|
+
smoke/ # P0 critical path (every PR)
|
|
141
|
+
regression/ # Full suite (nightly)
|
|
142
|
+
api/ # API-level tests
|
|
143
|
+
unit/ # Unit tests
|
|
144
|
+
|
|
145
|
+
pages/ or page-objects/ or support/page-objects/
|
|
146
|
+
base/
|
|
147
|
+
[feature]/
|
|
148
|
+
components/
|
|
149
|
+
|
|
150
|
+
fixtures/ # Test data & factories
|
|
151
|
+
config/ # Test configs (if separate from root)
|
|
152
|
+
reports/ # Generated reports (gitignored)
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
## Test Data Rules
|
|
156
|
+
|
|
157
|
+
- **NEVER** hardcode real credentials
|
|
158
|
+
- Use environment variables with test fallbacks
|
|
159
|
+
- Fixtures go in dedicated folder
|
|
160
|
+
- Each domain gets its own fixture file
|
|
161
|
+
|
|
162
|
+
## Analysis Documents
|
|
163
|
+
|
|
164
|
+
When analyzing a repository, produce these documents:
|
|
165
|
+
|
|
166
|
+
### QA_ANALYSIS.md must include:
|
|
167
|
+
- Architecture overview (system type, language, runtime, entry points, dependencies)
|
|
168
|
+
- Risk assessment (HIGH / MEDIUM / LOW with justification)
|
|
169
|
+
- Top 10 unit test targets with rationale
|
|
170
|
+
- Recommended testing pyramid with percentages adjusted to this app
|
|
171
|
+
- External dependencies with risk levels
|
|
172
|
+
|
|
173
|
+
### TEST_INVENTORY.md must include:
|
|
174
|
+
- Every test case with ID, target, inputs, expected outcome, priority
|
|
175
|
+
- Organized by pyramid level (unit -> integration -> API -> E2E)
|
|
176
|
+
- No test case without an explicit expected outcome
|
|
177
|
+
|
|
178
|
+
## Quality Gates
|
|
179
|
+
|
|
180
|
+
Before delivering ANY QA artifact, verify:
|
|
181
|
+
- [ ] Framework matches what the project already uses
|
|
182
|
+
- [ ] Every test case has an explicit expected outcome with a concrete value
|
|
183
|
+
- [ ] No outcome says "correct", "proper", "appropriate", or "works" without defining what that means
|
|
184
|
+
- [ ] All locators follow the tier hierarchy
|
|
185
|
+
- [ ] No assertions inside page objects
|
|
186
|
+
- [ ] No hardcoded credentials
|
|
187
|
+
- [ ] File naming follows the project's existing conventions (or the standards above if none exist)
|
|
188
|
+
- [ ] Test IDs are unique and follow naming convention
|
|
189
|
+
- [ ] Priority assigned to every test case
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
193
|
+
## Agent Pipeline
|
|
194
|
+
|
|
195
|
+
The QA automation system runs agents in a defined pipeline. Each stage produces artifacts consumed by the next stage.
|
|
196
|
+
|
|
197
|
+
### Pipeline Stages
|
|
198
|
+
|
|
199
|
+
```
|
|
200
|
+
scan -> codebase-map -> research -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [e2e-runner if E2E tests] -> [bug-detective if failures] -> deliver
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
### Workflow Options
|
|
204
|
+
|
|
205
|
+
**Option 1: Dev-Only Repo (no existing QA repo)**
|
|
206
|
+
Full pipeline from scratch:
|
|
207
|
+
```
|
|
208
|
+
scan -> codebase-map -> research -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [e2e-runner if E2E tests] -> [bug-detective if failures] -> deliver
|
|
209
|
+
```
|
|
210
|
+
Produces: SCAN_MANIFEST.md -> 8 codebase map documents -> QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md -> [TESTID_AUDIT_REPORT.md] -> generation plan -> test files + POMs + fixtures + configs -> VALIDATION_REPORT.md -> [E2E_RUN_REPORT.md] -> branch + PR
|
|
211
|
+
|
|
212
|
+
**Option 2: Dev + Immature QA Repo (existing QA repo with low coverage or quality)**
|
|
213
|
+
Gap-fill and standardize:
|
|
214
|
+
```
|
|
215
|
+
scan both repos -> gap analysis -> fix broken tests -> add missing coverage -> standardize existing -> validate -> deliver
|
|
216
|
+
```
|
|
217
|
+
Produces: SCAN_MANIFEST.md (both repos) -> GAP_ANALYSIS.md -> fixed test files -> new test files -> standardized files -> VALIDATION_REPORT.md -> branch + PR
|
|
218
|
+
|
|
219
|
+
**Option 3: Dev + Mature QA Repo (existing QA repo with solid coverage)**
|
|
220
|
+
Surgical additions only:
|
|
221
|
+
```
|
|
222
|
+
scan both repos -> identify thin coverage -> add only missing tests -> validate -> deliver
|
|
223
|
+
```
|
|
224
|
+
Produces: SCAN_MANIFEST.md (both repos) -> GAP_ANALYSIS.md (thin areas only) -> new test files (targeted) -> VALIDATION_REPORT.md -> branch + PR
|
|
225
|
+
|
|
226
|
+
### Stage Transitions
|
|
227
|
+
|
|
228
|
+
| From | To | Condition |
|
|
229
|
+
|------|----|-----------|
|
|
230
|
+
| scan | research | SCAN_MANIFEST.md exists with > 0 testable surfaces |
|
|
231
|
+
| research | codebase-map | Research complete (non-blocking — continues even if no research output) |
|
|
232
|
+
| codebase-map | analyze | At least 4 of 8 codebase map documents exist |
|
|
233
|
+
| analyze | testid-inject | QA_ANALYSIS.md exists AND frontend components detected |
|
|
234
|
+
| analyze | plan | QA_ANALYSIS.md + TEST_INVENTORY.md exist (skip testid-inject if no frontend) |
|
|
235
|
+
| testid-inject | plan | TESTID_AUDIT_REPORT.md exists with coverage score calculated |
|
|
236
|
+
| plan | generate | Generation plan approved (or auto-approved in auto-advance mode) |
|
|
237
|
+
| generate | validate | All planned test files exist on disk |
|
|
238
|
+
| validate | e2e-runner | VALIDATION_REPORT.md shows PASS AND E2E test files were generated AND live app available |
|
|
239
|
+
| validate | deliver | VALIDATION_REPORT.md shows PASS and no E2E files or --skip-run |
|
|
240
|
+
| e2e-runner | bug-detective | E2E_RUN_REPORT.md exists AND failures need classification |
|
|
241
|
+
| e2e-runner | deliver | E2E_RUN_REPORT.md exists AND all tests pass or failures classified |
|
|
242
|
+
| bug-detective | deliver | FAILURE_CLASSIFICATION_REPORT.md exists |
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
## Module Boundaries
|
|
247
|
+
|
|
248
|
+
Each agent owns specific artifacts. No agent may produce artifacts assigned to another agent.
|
|
249
|
+
|
|
250
|
+
| Agent | Reads | Produces | Template |
|
|
251
|
+
|-------|-------|----------|----------|
|
|
252
|
+
| qa-scanner | repo source files, package.json, file tree, frameworks/*.json (registry) | SCAN_MANIFEST.md | templates/scan-manifest.md |
|
|
253
|
+
| qa-project-researcher | SCAN_MANIFEST.md, repo source files, Context7 MCP, WebSearch, WebFetch | TESTING_STACK.md, FRAMEWORK_CAPABILITIES.md, API_TESTING_STRATEGY.md, E2E_STRATEGY.md | -- |
|
|
254
|
+
| qa-codebase-mapper | SCAN_MANIFEST.md, repo source files, CLAUDE.md | TESTABILITY.md, TEST_SURFACE.md, RISK_MAP.md, CRITICAL_PATHS.md, CODE_PATTERNS.md, API_CONTRACTS.md, TEST_ASSESSMENT.md, COVERAGE_GAPS.md | -- |
|
|
255
|
+
| qa-analyzer | SCAN_MANIFEST.md, codebase map documents, CLAUDE.md | QA_ANALYSIS.md, TEST_INVENTORY.md, QA_REPO_BLUEPRINT.md (Option 1) or GAP_ANALYSIS.md (Option 2/3) | templates/qa-analysis.md, templates/test-inventory.md, templates/qa-repo-blueprint.md, templates/gap-analysis.md |
|
|
256
|
+
| qa-planner | TEST_INVENTORY.md, QA_ANALYSIS.md, codebase map documents | Generation plan (internal) | -- |
|
|
257
|
+
| qa-executor | TEST_INVENTORY.md, codebase map documents, CLAUDE.md, frameworks/{detected_framework}.json (registry) | test files, POMs, fixtures, configs | qa-template-engine patterns |
|
|
258
|
+
| qa-validator | generated test files, CLAUDE.md | VALIDATION_REPORT.md (validation mode) or QA_AUDIT_REPORT.md (audit mode) | templates/validation-report.md, templates/qa-audit-report.md |
|
|
259
|
+
| qa-e2e-runner | generated E2E test files, POM files, CLAUDE.md, live application | E2E_RUN_REPORT.md, fixed test/POM files with real locators | -- |
|
|
260
|
+
| qa-testid-injector | repo source files, SCAN_MANIFEST.md, CLAUDE.md | TESTID_AUDIT_REPORT.md, modified source files with data-testid attributes | templates/scan-manifest.md, templates/testid-audit-report.md |
|
|
261
|
+
| qa-bug-detective | test execution results, test source files, CLAUDE.md | FAILURE_CLASSIFICATION_REPORT.md | templates/failure-classification.md |
|
|
262
|
+
|
|
263
|
+
**Rule:** An agent MUST NOT produce artifacts assigned to another agent.
|
|
264
|
+
|
|
265
|
+
**Rule:** An agent MUST read all artifacts listed in its "Reads" column before producing output.
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Verification Commands
|
|
270
|
+
|
|
271
|
+
Every artifact must pass verification before the pipeline advances. Below are the validation rules per artifact type.
|
|
272
|
+
|
|
273
|
+
### SCAN_MANIFEST.md
|
|
274
|
+
- Has > 0 files in File List table
|
|
275
|
+
- Project Detection section is populated (framework, language, component patterns)
|
|
276
|
+
- Testable Surfaces has at least 1 category with entries
|
|
277
|
+
- File priority ordering is present (HIGH/MEDIUM/LOW)
|
|
278
|
+
|
|
279
|
+
### QA_ANALYSIS.md
|
|
280
|
+
- All 6 sections present: Architecture Overview, External Dependencies, Risk Assessment, Top 10 Unit Test Targets, API/Contract Test Targets, Recommended Testing Pyramid
|
|
281
|
+
- Top 10 has exactly 10 entries with module, rationale, and complexity
|
|
282
|
+
- Testing pyramid percentages sum to 100%
|
|
283
|
+
- Risk assessment uses only HIGH/MEDIUM/LOW with justification per item
|
|
284
|
+
|
|
285
|
+
### TEST_INVENTORY.md
|
|
286
|
+
- Every test case has all mandatory fields: ID, target, inputs, expected outcome, priority
|
|
287
|
+
- IDs are unique across the entire document (no duplicates)
|
|
288
|
+
- IDs follow naming convention: UT-MODULE-NNN, INT-MODULE-NNN, API-RESOURCE-NNN, E2E-FLOW-NNN
|
|
289
|
+
- Pyramid tier counts match the summary table
|
|
290
|
+
- No expected outcome says "correct", "proper", "appropriate", or "works" without concrete value
|
|
291
|
+
|
|
292
|
+
### QA_REPO_BLUEPRINT.md
|
|
293
|
+
- Folder structure tree is present with explanations per directory
|
|
294
|
+
- Config files section has actual content (not placeholders)
|
|
295
|
+
- npm scripts defined for smoke, regression, and API test runs
|
|
296
|
+
- CI/CD strategy section includes PR-gate and nightly run configurations
|
|
297
|
+
- Definition of Done checklist is present
|
|
298
|
+
|
|
299
|
+
### VALIDATION_REPORT.md
|
|
300
|
+
- All 4 layers reported per file: Syntax, Structure, Dependencies, Logic
|
|
301
|
+
- Each layer shows PASS or FAIL with details
|
|
302
|
+
- Confidence level assigned: HIGH (all layers pass), MEDIUM (1-2 minor issues), LOW (structural problems)
|
|
303
|
+
- Fix loop log shows iteration count and what was found/fixed per loop
|
|
304
|
+
- Unresolved issues section documents anything not auto-fixed
|
|
305
|
+
|
|
306
|
+
### FAILURE_CLASSIFICATION_REPORT.md
|
|
307
|
+
- Every failure has classification: APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, or INCONCLUSIVE
|
|
308
|
+
- Every failure has confidence level: HIGH, MEDIUM-HIGH, MEDIUM, or LOW
|
|
309
|
+
- Every failure has evidence: code snippet + reasoning explaining the classification
|
|
310
|
+
- No APPLICATION BUG is marked as auto-fixed (application bugs require developer action)
|
|
311
|
+
- Auto-fix log documents what was fixed and at what confidence level
|
|
312
|
+
|
|
313
|
+
### TESTID_AUDIT_REPORT.md
|
|
314
|
+
- Coverage score calculated: existing data-testid count / total interactive elements
|
|
315
|
+
- All proposed data-testid values follow `{context}-{description}-{element-type}` naming convention
|
|
316
|
+
- No duplicate data-testid values within the same page/route scope
|
|
317
|
+
- Elements classified by priority: P0 (form inputs, buttons), P1 (links, images), P2 (containers, decorative)
|
|
318
|
+
- Decision gate threshold applied: >90% SELECTIVE, 50-90% TARGETED, <50% FULL PASS, 0% P0 FIRST
|
|
319
|
+
|
|
320
|
+
### GAP_ANALYSIS.md
|
|
321
|
+
- Coverage map shows all modules from SCAN_MANIFEST.md
|
|
322
|
+
- Missing tests have IDs following naming convention and priorities assigned
|
|
323
|
+
- Broken tests have failure reasons documented with file path and error
|
|
324
|
+
- Quality assessment includes locator tier distribution and assertion quality rating
|
|
325
|
+
- Recommendations are prioritized: fix broken first, then add P0, then P1
|
|
326
|
+
|
|
327
|
+
### QA_AUDIT_REPORT.md
|
|
328
|
+
- All 6 dimensions scored: Locator Quality, Assertion Specificity, POM Compliance, Test Coverage, Naming Convention, Test Data Management
|
|
329
|
+
- Weights sum to 100%: Locator 20%, Assertion 20%, POM 15%, Coverage 20%, Naming 15%, Test Data 10%
|
|
330
|
+
- Overall score matches weighted calculation of dimension scores
|
|
331
|
+
- Critical issues listed with file path, line number, and description
|
|
332
|
+
- Each recommendation has effort estimate: S (small), M (medium), L (large)
|
|
333
|
+
|
|
334
|
+
---
|
|
335
|
+
|
|
336
|
+
## Git Workflow
|
|
337
|
+
|
|
338
|
+
All QA automation output follows these git conventions.
|
|
339
|
+
|
|
340
|
+
### Branch Naming
|
|
341
|
+
|
|
342
|
+
```
|
|
343
|
+
qa/auto-{project}-{date}
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
Examples:
|
|
347
|
+
- `qa/auto-shopflow-2026-03-18`
|
|
348
|
+
- `qa/auto-acme-api-2026-04-01`
|
|
349
|
+
|
|
350
|
+
### Commit Message Format
|
|
351
|
+
|
|
352
|
+
```
|
|
353
|
+
qa({agent}): {description}
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Examples:
|
|
357
|
+
- `qa(scanner): produce SCAN_MANIFEST.md for shopflow`
|
|
358
|
+
- `qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md`
|
|
359
|
+
- `qa(executor): generate 24 test files with POMs and fixtures`
|
|
360
|
+
- `qa(validator): validate generated tests - PASS with HIGH confidence`
|
|
361
|
+
- `qa(testid-injector): inject 47 data-testid attributes across 12 components`
|
|
362
|
+
- `qa(bug-detective): classify 5 failures - 2 APP BUG, 2 TEST ERROR, 1 ENV ISSUE`
|
|
363
|
+
|
|
364
|
+
### Commit Conventions
|
|
365
|
+
|
|
366
|
+
- One commit per agent stage (scanner produces one commit, analyzer produces one commit, etc.)
|
|
367
|
+
- Descriptive messages that include artifact names and counts
|
|
368
|
+
- Never commit .env files, credentials, or secrets
|
|
369
|
+
- Include modified file count in commit body when relevant
|
|
370
|
+
|
|
371
|
+
### PR Template
|
|
372
|
+
|
|
373
|
+
PR description must include:
|
|
374
|
+
- Analysis summary (architecture type, framework, risk areas)
|
|
375
|
+
- Test counts by pyramid level (unit: N, integration: N, API: N, E2E: N)
|
|
376
|
+
- Coverage metrics (modules covered, estimated line coverage)
|
|
377
|
+
- Validation pass/fail status with confidence level
|
|
378
|
+
- Link to VALIDATION_REPORT.md in the PR files
|
|
379
|
+
|
|
380
|
+
---
|
|
381
|
+
|
|
382
|
+
## Team Settings
|
|
383
|
+
|
|
384
|
+
Configuration for multi-agent pipeline execution.
|
|
385
|
+
|
|
386
|
+
### Concurrent Execution
|
|
387
|
+
|
|
388
|
+
Agents in the same pipeline stage can run in parallel when their inputs are independent. Examples:
|
|
389
|
+
- qa-testid-injector and qa-analyzer can run simultaneously after scan completes (both read SCAN_MANIFEST.md)
|
|
390
|
+
- Multiple qa-executor instances can generate tests for different modules in parallel
|
|
391
|
+
|
|
392
|
+
Agents in different pipeline stages MUST respect stage ordering. A downstream agent cannot start until all its required inputs exist on disk.
|
|
393
|
+
|
|
394
|
+
### Worktree Isolation
|
|
395
|
+
|
|
396
|
+
Each agent operates on the same branch. No worktree splits are needed for this system. Agents coordinate through file-based artifacts -- each agent writes its own files and reads other agents' files.
|
|
397
|
+
|
|
398
|
+
### Dependency Ordering
|
|
399
|
+
|
|
400
|
+
Respect stage transitions from the Agent Pipeline section:
|
|
401
|
+
1. qa-scanner runs first (no dependencies)
|
|
402
|
+
2. qa-project-researcher runs after codebase-map (depends on SCAN_MANIFEST.md, uses Context7 MCP — non-blocking, pipeline continues if it fails)
|
|
403
|
+
3. qa-codebase-mapper runs after scanner (depends on SCAN_MANIFEST.md, spawns 4 parallel focus agents)
|
|
404
|
+
4. qa-analyzer and qa-testid-injector run after codebase-map (both depend on SCAN_MANIFEST.md + codebase map documents + research documents if available)
|
|
405
|
+
5. qa-planner runs after analyzer (depends on QA_ANALYSIS.md + TEST_INVENTORY.md + codebase map documents)
|
|
406
|
+
6. qa-executor runs after planner (depends on generation plan + codebase map documents + research documents if available + Context7 MCP)
|
|
407
|
+
7. qa-validator runs after executor (depends on generated test files)
|
|
408
|
+
8. qa-e2e-runner runs after validator (depends on E2E test files + live application + Context7 MCP)
|
|
409
|
+
9. qa-bug-detective runs after e2e-runner or validator if failures (depends on test results + Context7 MCP)
|
|
410
|
+
|
|
411
|
+
### Auto-Advance Mode
|
|
412
|
+
|
|
413
|
+
When auto-advance is enabled, pipeline stages advance automatically when:
|
|
414
|
+
1. The previous stage completes
|
|
415
|
+
2. All output artifacts from the previous stage exist on disk
|
|
416
|
+
3. All output artifacts pass their verification commands (from Verification Commands section)
|
|
417
|
+
|
|
418
|
+
No human confirmation is needed between stages in auto-advance mode. The pipeline pauses only at explicit checkpoint tasks or when verification fails.
|
|
419
|
+
|
|
420
|
+
---
|
|
421
|
+
|
|
422
|
+
## Agent Coordination
|
|
423
|
+
|
|
424
|
+
Rules governing how agents communicate and hand off work through artifacts.
|
|
425
|
+
|
|
426
|
+
### Read-Before-Write Rules
|
|
427
|
+
|
|
428
|
+
Every agent MUST read its required inputs before producing any output. Failure to read inputs produces low-quality, inconsistent artifacts.
|
|
429
|
+
|
|
430
|
+
| Agent | MUST Read Before Producing Output |
|
|
431
|
+
|-------|-----------------------------------|
|
|
432
|
+
| qa-scanner | package.json (or equivalent), folder tree structure, all source file extensions, frameworks/*.json (registry) for detection patterns |
|
|
433
|
+
| qa-project-researcher | SCAN_MANIFEST.md, repo source files, MY_PREFERENCES.md. Uses Context7 MCP as primary source. |
|
|
434
|
+
| qa-analyzer | SCAN_MANIFEST.md (complete, verified), CLAUDE.md (all QA standards sections), research documents (if available) |
|
|
435
|
+
| qa-planner | TEST_INVENTORY.md (all test cases), QA_ANALYSIS.md (architecture and risk context) |
|
|
436
|
+
| qa-executor | TEST_INVENTORY.md (test cases to implement), CLAUDE.md (POM rules, locator hierarchy, assertion rules, naming conventions, quality gates), research documents (if available), frameworks/{detected_framework}.json (registry: conventions, extensions, context7_keys). Must query Context7 MCP before generating code. |
|
|
437
|
+
| qa-validator | CLAUDE.md (quality gates, locator tiers, assertion rules), all generated test files to validate, frameworks/{detected_framework}.json (registry: validation_commands) |
|
|
438
|
+
| qa-testid-injector | SCAN_MANIFEST.md (component file list), CLAUDE.md (data-testid Convention section for naming rules) |
|
|
439
|
+
| qa-bug-detective | Test execution output (stdout/stderr, exit codes), test source files (to read the failing code), CLAUDE.md (classification rules), research documents (if available). Must query Context7 MCP before auto-fixing. |
|
|
440
|
+
| qa-e2e-runner | Generated E2E test files, POM files, CLAUDE.md, live application, research documents (if available). Must query Context7 MCP before fixing locators. |
|
|
441
|
+
|
|
442
|
+
### Handoff Patterns
|
|
443
|
+
|
|
444
|
+
Agents communicate exclusively through file-based artifacts:
|
|
445
|
+
|
|
446
|
+
1. **Producer writes** -- Agent completes its task and writes output artifact(s) to disk
|
|
447
|
+
2. **Pipeline verifies** -- Output artifacts pass verification commands before advancing
|
|
448
|
+
3. **Consumer reads** -- Next agent reads the artifact(s) as its first action
|
|
449
|
+
4. **No direct communication** -- Agents never pass data in memory or through environment variables between stages
|
|
450
|
+
|
|
451
|
+
### Quality Gates Per Artifact
|
|
452
|
+
|
|
453
|
+
Before the pipeline advances past any stage, the produced artifact(s) must pass verification:
|
|
454
|
+
|
|
455
|
+
| Stage Complete | Artifact | Gate |
|
|
456
|
+
|----------------|----------|------|
|
|
457
|
+
| scan | SCAN_MANIFEST.md | > 0 files listed, project detection populated |
|
|
458
|
+
| analyze | QA_ANALYSIS.md + TEST_INVENTORY.md | All sections present, IDs unique, pyramid sums to 100% |
|
|
459
|
+
| testid-inject | TESTID_AUDIT_REPORT.md | Coverage score calculated, naming convention compliant |
|
|
460
|
+
| plan | Generation plan | Test cases mapped to output files, no unassigned cases |
|
|
461
|
+
| generate | Test files + POMs | All planned files exist, imports resolve, syntax valid |
|
|
462
|
+
| validate | VALIDATION_REPORT.md | All 4 layers checked per file, confidence level assigned |
|
|
463
|
+
| deliver | Branch + PR | Branch pushed, PR created with required description sections |
|
|
464
|
+
|
|
465
|
+
### Error Recovery
|
|
466
|
+
|
|
467
|
+
If an agent fails or produces an artifact that does not pass verification:
|
|
468
|
+
1. Log the failure with the specific verification check that failed
|
|
469
|
+
2. Retry the agent (max 3 attempts per stage)
|
|
470
|
+
3. If still failing after 3 attempts, pause the pipeline and report the blocked stage
|
|
471
|
+
4. Do not advance to the next stage with a failed artifact
|
|
472
|
+
|
|
473
|
+
---
|
|
474
|
+
|
|
475
|
+
## data-testid Convention
|
|
476
|
+
|
|
477
|
+
All `data-testid` attributes injected by qa-testid-injector and referenced by generated tests MUST follow this naming convention.
|
|
478
|
+
|
|
479
|
+
### Naming Pattern
|
|
480
|
+
|
|
481
|
+
```
|
|
482
|
+
{context}-{description}-{element-type}
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
All values are **kebab-case**. No camelCase, no underscores, no periods.
|
|
486
|
+
|
|
487
|
+
### Context Derivation
|
|
488
|
+
|
|
489
|
+
1. **Page-level context**: Derived from the component filename or route
|
|
490
|
+
- `LoginPage.tsx` -> context is `login`
|
|
491
|
+
- `ProductDetailPage.tsx` -> context is `product-detail`
|
|
492
|
+
- Route `/settings/profile` -> context is `settings-profile`
|
|
493
|
+
|
|
494
|
+
2. **Component-level context**: Derived from the component name
|
|
495
|
+
- `<NavBar>` -> context is `navbar`
|
|
496
|
+
- `<ShoppingCart>` -> context is `shopping-cart`
|
|
497
|
+
- `<UserAvatar>` -> context is `user-avatar`
|
|
498
|
+
|
|
499
|
+
3. **Nested context**: Parent-child hierarchy, max 3 levels deep
|
|
500
|
+
- `checkout-shipping-address-input` (page -> section -> field)
|
|
501
|
+
- `dashboard-sidebar-nav-link` (page -> component -> element)
|
|
502
|
+
- Never exceed 3 levels: `a-b-c-element` is the maximum depth
|
|
503
|
+
|
|
504
|
+
4. **Dynamic list items**: Use template literals with unique keys
|
|
505
|
+
```tsx
|
|
506
|
+
data-testid={`product-${product.id}-card`}
|
|
507
|
+
data-testid={`order-${order.id}-status-badge`}
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
### Element Type Suffix Table
|
|
511
|
+
|
|
512
|
+
Every `data-testid` value ends with a suffix indicating the element type:
|
|
513
|
+
|
|
514
|
+
| Element | Suffix | Example |
|
|
515
|
+
|---------|--------|---------|
|
|
516
|
+
| `<button>` | `-btn` | `login-submit-btn` |
|
|
517
|
+
| `<input>` | `-input` | `login-email-input` |
|
|
518
|
+
| `<select>` | `-select` | `settings-language-select` |
|
|
519
|
+
| `<textarea>` | `-textarea` | `feedback-comment-textarea` |
|
|
520
|
+
| `<a>` (link) | `-link` | `navbar-profile-link` |
|
|
521
|
+
| `<form>` | `-form` | `checkout-payment-form` |
|
|
522
|
+
| `<img>` | `-img` | `product-hero-img` |
|
|
523
|
+
| `<table>` | `-table` | `users-list-table` |
|
|
524
|
+
| `<tr>` (row) | `-row` | `users-item-row` |
|
|
525
|
+
| `<dialog>/<modal>` | `-modal` | `confirm-delete-modal` |
|
|
526
|
+
| `<div>` container | `-container` | `dashboard-stats-container` |
|
|
527
|
+
| `<ul>/<ol>` list | `-list` | `notifications-list` |
|
|
528
|
+
| `<li>` item | `-item` | `notifications-item` |
|
|
529
|
+
| dropdown menu | `-dropdown` | `navbar-user-dropdown` |
|
|
530
|
+
| tab | `-tab` | `settings-security-tab` |
|
|
531
|
+
| checkbox | `-checkbox` | `terms-accept-checkbox` |
|
|
532
|
+
| radio | `-radio` | `shipping-express-radio` |
|
|
533
|
+
| toggle/switch | `-toggle` | `notifications-enabled-toggle` |
|
|
534
|
+
| badge/chip | `-badge` | `cart-count-badge` |
|
|
535
|
+
| alert/toast | `-alert` | `error-validation-alert` |
|
|
536
|
+
|
|
537
|
+
### Third-Party Component Handling
|
|
538
|
+
|
|
539
|
+
When adding `data-testid` to third-party UI library components, use this priority order:
|
|
540
|
+
|
|
541
|
+
1. **Props passthrough** (preferred): If the library supports passing `data-testid` directly as a prop
|
|
542
|
+
```tsx
|
|
543
|
+
<MuiButton data-testid="checkout-pay-btn">Pay</MuiButton>
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
2. **Wrapper div**: If the library does not support prop passthrough, wrap with a `<div>` that has the `data-testid`
|
|
547
|
+
```tsx
|
|
548
|
+
<div data-testid="checkout-pay-container">
|
|
549
|
+
<ThirdPartyButton>Pay</ThirdPartyButton>
|
|
550
|
+
</div>
|
|
551
|
+
```
|
|
552
|
+
|
|
553
|
+
3. **inputProps / slotProps** (MUI-specific): Use component-specific prop APIs
|
|
554
|
+
```tsx
|
|
555
|
+
<TextField inputProps={{ 'data-testid': 'login-email-input' }} />
|
|
556
|
+
<Autocomplete slotProps={{ input: { 'data-testid': 'search-query-input' } }} />
|
|
557
|
+
```
|
|
558
|
+
---
|
|
559
|
+
|
|
560
|
+
## Framework Registry
|
|
561
|
+
|
|
562
|
+
The Framework Registry is a folder-based metadata system at `frameworks/` containing one JSON file per supported framework. Each file declares stable metadata that the system needs to recognize and operate the framework — extensions, detection patterns, validation commands, conventions, Context7 keys.
|
|
563
|
+
|
|
564
|
+
### What lives in `frameworks/<name>.json`
|
|
565
|
+
|
|
566
|
+
```json
|
|
567
|
+
{
|
|
568
|
+
"name": "playwright",
|
|
569
|
+
"display_name": "Playwright",
|
|
570
|
+
"languages": ["typescript", "javascript"],
|
|
571
|
+
"extensions": [".spec.ts", ".spec.js"],
|
|
572
|
+
"detection": {
|
|
573
|
+
"manifest_files": ["package.json"],
|
|
574
|
+
"manifest_grep": "@playwright/test",
|
|
575
|
+
"config_files": ["playwright.config.ts"],
|
|
576
|
+
"file_patterns": ["tests/**/*.spec.ts"]
|
|
577
|
+
},
|
|
578
|
+
"validation_commands": {
|
|
579
|
+
"syntax_check_ts": "tsc --noEmit",
|
|
580
|
+
"dry_run": "npx playwright test --list",
|
|
581
|
+
"lint": "eslint {file}"
|
|
582
|
+
},
|
|
583
|
+
"conventions": {
|
|
584
|
+
"test_dirs": ["tests/", "e2e/"],
|
|
585
|
+
"pom_dirs": ["pages/"],
|
|
586
|
+
"fixture_dirs": ["fixtures/"],
|
|
587
|
+
"config_filename": "playwright.config.ts"
|
|
588
|
+
},
|
|
589
|
+
"context7_keys": ["playwright", "@playwright/test"]
|
|
590
|
+
}
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
### What does NOT live in the registry
|
|
594
|
+
|
|
595
|
+
The registry stores **stable metadata**, NOT **code idioms**. Code idioms (imports, syntax, API usage, pattern examples) come from **Context7** at runtime — they change with framework releases and Context7 always has the latest. The registry tells the system "what to query in Context7" via `context7_keys`; Context7 tells the system "how to write the code".
|
|
596
|
+
|
|
597
|
+
### Who consumes the registry
|
|
598
|
+
|
|
599
|
+
| Agent | What it reads | How |
|
|
600
|
+
|-------|---------------|-----|
|
|
601
|
+
| qa-scanner | All `frameworks/*.json` | Iterate for detection (manifest_grep, config_files, file_patterns) |
|
|
602
|
+
| qa-executor | `frameworks/<detected>.json` | Use `extensions`, `conventions`, `context7_keys` |
|
|
603
|
+
| qa-validator | `frameworks/<detected>.json` | Use `validation_commands` for Layer 1 (Syntax) and Layer 2 (Structure) |
|
|
604
|
+
|
|
605
|
+
Other agents (analyzer, planner, testid-injector, bug-detective, e2e-runner) get the framework name from the scanner's output (SCAN_MANIFEST.md) and don't need to read the registry directly.
|
|
606
|
+
|
|
607
|
+
### Adding a new framework
|
|
608
|
+
|
|
609
|
+
1. Create `frameworks/<new-framework>.json` with the schema above
|
|
610
|
+
2. Add `"VERSION"` and `"frameworks/"` are already in `package.json` `files` array (no change needed)
|
|
611
|
+
3. Test detection by running scanner against a project that uses the framework
|
|
612
|
+
4. Test code generation: executor will read `context7_keys` and query Context7 for syntax
|
|
613
|
+
|
|
614
|
+
### Fallback when framework not in registry
|
|
615
|
+
|
|
616
|
+
If a project uses a framework not in `frameworks/`, the system:
|
|
617
|
+
1. Logs a WARNING: "Framework `<name>` not in registry — relying on Context7 only"
|
|
618
|
+
2. Asks Context7 for everything (slower, more queries, but still works)
|
|
619
|
+
3. The pipeline does NOT halt unless the user requests `--strict-registry`
|
|
620
|
+
|
|
621
|
+
The registry is an **optimization**, not a gate. Common frameworks are pre-registered for speed; rare frameworks fall back to Context7.
|
|
622
|
+
|
|
623
|
+
### Context7 cache (project-local)
|
|
624
|
+
|
|
625
|
+
Each project, after the first pipeline run, has a `.qa-output/frameworks/<framework>-<version>.json` file that caches Context7 query responses. This:
|
|
626
|
+
- Reduces Context7 API calls (free tier: 60/hour)
|
|
627
|
+
- Auto-invalidates when project framework version changes (filename includes version)
|
|
628
|
+
- Default TTL: 30 days
|
|
629
|
+
|
|
630
|
+
The cache is consumed by 4 agents: project-researcher (writes), executor / e2e-runner / bug-detective (read first, fall back to Context7 if cache miss). See the `<context7_verification>` → "Verification Cascade" section in those agents.
|
|
631
|
+
|
|
632
|
+
---
|
|
633
|
+
|
|
634
|
+
## Intent Detection
|
|
635
|
+
|
|
636
|
+
The Intent Detector at `bin/lib/intent-detector.cjs` parses command arguments to extract structured intent (URLs, paths, framework hints, tickets, flags) and supports both explicit flags and natural language.
|
|
637
|
+
|
|
638
|
+
### Hierarchy
|
|
639
|
+
|
|
640
|
+
```
|
|
641
|
+
Flags > Natural Language > Defaults
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
- **Flags always win**: `--app-url https://example.com` is authoritative
|
|
645
|
+
- **NL fills gaps**: if `--app-url` is not passed but a URL is mentioned, the detector extracts it
|
|
646
|
+
- **Defaults**: if neither flag nor NL provides a value, fall back to defaults (cwd for dev-repo, etc.)
|
|
647
|
+
|
|
648
|
+
### INPUT DETECTION banner
|
|
649
|
+
|
|
650
|
+
Every command (qa-start, qa-create-test, qa-fix, qa-audit, qa-pr) shows an **INPUT DETECTION** banner before starting work. This banner is shown ALWAYS — with flags, with NL, or with defaults — and lists every resolved value with its source:
|
|
651
|
+
|
|
652
|
+
```
|
|
653
|
+
=== INPUT DETECTION ===
|
|
654
|
+
From explicit flags:
|
|
655
|
+
--dev-repo /home/user/proj (source: flag)
|
|
656
|
+
Detected from natural language:
|
|
657
|
+
--app-url https://staging.example.com (source: NL)
|
|
658
|
+
--framework playwright (source: NL)
|
|
659
|
+
Defaults applied:
|
|
660
|
+
--auto false (source: default)
|
|
661
|
+
=======================
|
|
662
|
+
```
|
|
663
|
+
|
|
664
|
+
In interactive mode, the user can press Ctrl-C if the detection looks wrong. In `--auto` mode, the banner serves as a structured log.
|
|
665
|
+
|
|
666
|
+
### Input validations (5 pre-pipeline checks)
|
|
667
|
+
|
|
668
|
+
After intent detection, qa-start runs 5 validation checks before invoking any sub-agent:
|
|
669
|
+
|
|
670
|
+
1. **URL reachable** — `curl` ping with 5s timeout (warning, not fatal)
|
|
671
|
+
2. **Path writable** — verify dev-repo path exists and is writable, or parent is writable
|
|
672
|
+
3. **URL not ambiguous with ticket platforms** — warn if AUT URL looks like a ticket URL
|
|
673
|
+
4. **No multiple AUT URLs** — fail if more than 1 AUT URL detected (excluding tickets)
|
|
674
|
+
5. **Framework in registry** — warn if `--framework <name>` not in `frameworks/`
|
|
675
|
+
|
|
676
|
+
These are **Capa 1** of a 3-layer validation system. Capa 2 is each agent's `<quality_gate>`. Capa 3 is each agent's mandatory bash checklist.
|
|
677
|
+
|
|
678
|
+
---
|
|
679
|
+
|
|
680
|
+
## Standard Task() Directive
|
|
681
|
+
|
|
682
|
+
Every `Task()` / `Agent()` spawn in this codebase (orchestrator, commands, workflows) MUST embed the following directive **verbatim** at the very top of the spawned prompt, substituting ONLY `<AGENT_FILE>` with the target agent's exact filename (e.g. `qaa-scanner`, `qa-pipeline-orchestrator`). Do NOT paraphrase it, and do NOT add `subagent_type: "qaa-X"` — native subagent routing for QAA agents is not enabled and would hard-error (see the architectural note below).
|
|
683
|
+
|
|
684
|
+
```
|
|
685
|
+
<critical_directive priority="MANDATORY">
|
|
686
|
+
You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
|
|
687
|
+
~/.claude/qaa/agents/<AGENT_FILE>.md and adopt it as your operating contract.
|
|
688
|
+
|
|
689
|
+
Even though you may be running as a general-purpose agent (native subagent type
|
|
690
|
+
routing for QAA agents is not yet enabled), you MUST behave exactly as the
|
|
691
|
+
dedicated agent defined in that file: read everything in its <required_reading>,
|
|
692
|
+
obey its <quality_gate>, and run its mandatory bash checklist before returning.
|
|
693
|
+
|
|
694
|
+
Never modify application source code (src/, app/, lib/, or equivalent
|
|
695
|
+
directories) unless the agent's <scope> section explicitly permits it. If the
|
|
696
|
+
agent's contract restricts what files it can touch, honor that restriction —
|
|
697
|
+
even if a fix seems obvious or trivial.
|
|
698
|
+
|
|
699
|
+
If you cannot read ~/.claude/qaa/agents/<AGENT_FILE>.md (file missing, corrupted, parse
|
|
700
|
+
error), STOP and report the failure. Do NOT proceed with default agent
|
|
701
|
+
behavior — that defeats the purpose of this directive.
|
|
702
|
+
|
|
703
|
+
Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
|
|
704
|
+
make, or skip any of the above. Skipping the agent's required_reading,
|
|
705
|
+
quality_gate, or bash checklist makes the run INVALID — its output cannot be
|
|
706
|
+
trusted.
|
|
707
|
+
</critical_directive>
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
### Architectural convention: wrap the flow in MANDATORY
|
|
711
|
+
|
|
712
|
+
QAA hardens execution at two layers, and new commands/agents MUST follow both:
|
|
713
|
+
|
|
714
|
+
1. **Command headers** — every NL command opens with a `## ⚠ MANDATORY: How to Execute This Command` block defining the full ordered flow + DO / DO NOT (not just the first tool call).
|
|
715
|
+
2. **Task() prompts** — every spawn embeds the Standard Task() Directive above.
|
|
716
|
+
|
|
717
|
+
This is the standard QAA enforcement pattern. Native `subagent_type` routing is deferred (the agents are not registered in `~/.claude/agents/`); until then the embedded directive is the enforcement mechanism.
|
|
718
|
+
|