qaa-agent 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/create-test.md +40 -0
- package/.claude/commands/qa-analyze.md +60 -0
- package/.claude/commands/qa-audit.md +37 -0
- package/.claude/commands/qa-blueprint.md +54 -0
- package/.claude/commands/qa-fix.md +36 -0
- package/.claude/commands/qa-from-ticket.md +88 -0
- package/.claude/commands/qa-gap.md +54 -0
- package/.claude/commands/qa-pom.md +36 -0
- package/.claude/commands/qa-pyramid.md +37 -0
- package/.claude/commands/qa-report.md +38 -0
- package/.claude/commands/qa-start.md +33 -0
- package/.claude/commands/qa-testid.md +54 -0
- package/.claude/commands/qa-validate.md +54 -0
- package/.claude/commands/update-test.md +58 -0
- package/.claude/settings.json +19 -0
- package/.claude/skills/qa-bug-detective/SKILL.md +122 -0
- package/.claude/skills/qa-repo-analyzer/SKILL.md +88 -0
- package/.claude/skills/qa-self-validator/SKILL.md +109 -0
- package/.claude/skills/qa-template-engine/SKILL.md +113 -0
- package/.claude/skills/qa-testid-injector/SKILL.md +93 -0
- package/.claude/skills/qa-workflow-documenter/SKILL.md +87 -0
- package/CLAUDE.md +543 -0
- package/README.md +418 -0
- package/agents/qa-pipeline-orchestrator.md +1217 -0
- package/agents/qaa-analyzer.md +508 -0
- package/agents/qaa-bug-detective.md +444 -0
- package/agents/qaa-executor.md +618 -0
- package/agents/qaa-planner.md +374 -0
- package/agents/qaa-scanner.md +422 -0
- package/agents/qaa-testid-injector.md +583 -0
- package/agents/qaa-validator.md +450 -0
- package/bin/install.cjs +176 -0
- package/bin/lib/commands.cjs +709 -0
- package/bin/lib/config.cjs +307 -0
- package/bin/lib/core.cjs +497 -0
- package/bin/lib/frontmatter.cjs +299 -0
- package/bin/lib/init.cjs +989 -0
- package/bin/lib/milestone.cjs +241 -0
- package/bin/lib/model-profiles.cjs +60 -0
- package/bin/lib/phase.cjs +911 -0
- package/bin/lib/roadmap.cjs +306 -0
- package/bin/lib/state.cjs +748 -0
- package/bin/lib/template.cjs +222 -0
- package/bin/lib/verify.cjs +842 -0
- package/bin/qaa-tools.cjs +607 -0
- package/package.json +34 -0
- package/templates/failure-classification.md +391 -0
- package/templates/gap-analysis.md +409 -0
- package/templates/pr-template.md +48 -0
- package/templates/qa-analysis.md +381 -0
- package/templates/qa-audit-report.md +465 -0
- package/templates/qa-repo-blueprint.md +636 -0
- package/templates/scan-manifest.md +312 -0
- package/templates/test-inventory.md +582 -0
- package/templates/testid-audit-report.md +354 -0
- package/templates/validation-report.md +243 -0
|
@@ -0,0 +1,618 @@
|
|
|
1
|
+
<purpose>
|
|
2
|
+
Read the generation plan (produced by qaa-planner), TEST_INVENTORY.md, and CLAUDE.md to produce actual test files, page object models, fixtures, and configuration files. This is the most complex agent in the pipeline -- it handles framework detection, BasePage scaffolding, POM generation following strict rules, test spec writing with concrete assertions, and per-file atomic commits for maximum traceability. The executor does not decide WHAT to test (that is the planner's job) -- it decides HOW to write each test file following CLAUDE.md standards and qa-template-engine patterns.
|
|
3
|
+
|
|
4
|
+
The executor is spawned by the orchestrator after the planner completes successfully via Task(subagent_type='qaa-executor'). It consumes the generation plan's task list in dependency order, writing one file at a time and committing each file individually. Upon completion, all planned test files exist on disk, imports resolve, and every file follows the project's QA standards.
|
|
5
|
+
</purpose>
|
|
6
|
+
|
|
7
|
+
<required_reading>
|
|
8
|
+
Read ALL of the following files BEFORE producing any output. The executor's code quality depends on reading CLAUDE.md POM rules and locator tiers. Skipping any of these files will produce non-compliant, low-quality test files.
|
|
9
|
+
|
|
10
|
+
- **Generation plan** -- Path provided by orchestrator in files_to_read. This is the planner's output containing the task list with file assignments, dependencies, test case IDs per task, and estimated complexity. Read the entire file. Extract: task execution order (respecting depends_on), file paths to create, test case IDs per task.
|
|
11
|
+
|
|
12
|
+
- **TEST_INVENTORY.md** -- Path provided by orchestrator in files_to_read. This is the analyzer's output containing every test case with full details: unique ID, target, what_to_validate, concrete_inputs, mocks_needed (for unit tests), expected_outcome, and priority. Read the entire file. For each task in the generation plan, look up the assigned test case IDs and extract their full details.
|
|
13
|
+
|
|
14
|
+
- **CLAUDE.md** -- QA automation standards. Read these sections:
|
|
15
|
+
- **Page Object Model Rules** -- 6 mandatory rules: (1) one class per page, (2) no assertions in page objects, (3) locators as properties (defined in constructor or as class fields), (4) actions return void or next page, (5) state queries return data, (6) every POM extends shared base
|
|
16
|
+
- **Locator Strategy** -- 4-tier hierarchy: Tier 1 (data-testid, ARIA roles) preferred, Tier 2 (labels, placeholders, text), Tier 3 (alt text, title), Tier 4 (CSS selectors, XPath -- add TODO comment)
|
|
17
|
+
- **Test Spec Rules** -- Every test must have: unique ID, exact target, concrete inputs, explicit expected outcome, priority
|
|
18
|
+
- **Naming Conventions** -- File naming table: POM `[PageName]Page.[ext]`, E2E `[feature].e2e.spec.[ext]`, API `[resource].api.spec.[ext]`, unit `[module].unit.spec.[ext]`, fixture `[domain]-data.[ext]`
|
|
19
|
+
- **Quality Gates** -- Assertion specificity: no "correct", "proper", "appropriate", "works" without concrete values. No `toBeTruthy()` or `toBeDefined()` alone.
|
|
20
|
+
- **Module Boundaries** -- qa-executor reads TEST_INVENTORY.md, CLAUDE.md; produces test files, POMs, fixtures, configs
|
|
21
|
+
- **Repo Structure** -- Directory layout for tests, pages, fixtures
|
|
22
|
+
- **data-testid Convention** -- Naming pattern `{context}-{description}-{element-type}`, all kebab-case, element type suffix table
|
|
23
|
+
- **Framework-Specific Examples** -- Playwright, Cypress, Selenium locator examples per tier
|
|
24
|
+
|
|
25
|
+
- **templates/qa-repo-blueprint.md** -- Reference for folder structure when QA_REPO_BLUEPRINT.md was produced by the analyzer. If the orchestrator indicates a blueprint exists, read it for exact directory layout and framework-specific configs.
|
|
26
|
+
|
|
27
|
+
- **.claude/skills/qa-template-engine/SKILL.md** -- Test generation patterns and rules:
|
|
28
|
+
- Unit test template (Arrange/Act/Assert with concrete values)
|
|
29
|
+
- API test template (payload, response status, response body assertions)
|
|
30
|
+
- E2E test template (POM navigation, action, assertion)
|
|
31
|
+
- POM generation rules (readonly locators, void/page returns, data queries)
|
|
32
|
+
- Locator priority (data-testid first, ARIA roles, labels, CSS last resort)
|
|
33
|
+
- Expected outcome rules (specific, measurable, negative cases, state transitions)
|
|
34
|
+
|
|
35
|
+
Note: The executor MUST read CLAUDE.md POM rules and locator tiers before writing any page object or test file. These rules are non-negotiable and must be applied to every generated file.
|
|
36
|
+
</required_reading>
|
|
37
|
+
|
|
38
|
+
<process>
|
|
39
|
+
|
|
40
|
+
<step name="read_inputs" priority="first">
|
|
41
|
+
Read all input artifacts and build the execution context.
|
|
42
|
+
|
|
43
|
+
1. **Read the generation plan** (path from orchestrator's files_to_read):
|
|
44
|
+
- Extract the task list with all fields: task_id, feature_group, files_to_create, test_case_ids, depends_on, estimated_complexity
|
|
45
|
+
- Extract the dependency graph to determine execution order
|
|
46
|
+
- Extract the framework and file extension from the Summary section
|
|
47
|
+
- Perform topological sort on task dependencies to get execution order
|
|
48
|
+
- Record total_tasks and total_files for progress tracking
|
|
49
|
+
|
|
50
|
+
2. **Read TEST_INVENTORY.md** (path from orchestrator's files_to_read):
|
|
51
|
+
- For each task in the generation plan, look up the assigned test case IDs
|
|
52
|
+
- Extract full test case details for each ID:
|
|
53
|
+
- Unit tests: test_id, target (file:function), what_to_validate, concrete_inputs, mocks_needed, expected_outcome, priority
|
|
54
|
+
- Integration tests: test_id, components_involved, what_to_validate, setup_required, expected_outcome, priority
|
|
55
|
+
- API tests: test_id, method_endpoint, request_body, headers, expected_status, expected_response, priority
|
|
56
|
+
- E2E tests: test_id, user_journey, pages_involved, expected_outcome, priority
|
|
57
|
+
- Store test case details indexed by test_id for quick lookup during generation
|
|
58
|
+
|
|
59
|
+
3. **Read CLAUDE.md** -- Extract and memorize:
|
|
60
|
+
- POM Rules (all 6 rules -- these are hard constraints on every POM file)
|
|
61
|
+
- Locator Strategy (4-tier hierarchy with framework-specific examples)
|
|
62
|
+
- Test Spec Rules (5 mandatory fields per test case)
|
|
63
|
+
- Naming Conventions (file naming table)
|
|
64
|
+
- Quality Gates (assertion specificity checklist)
|
|
65
|
+
- data-testid Convention (naming pattern, suffixes, context derivation)
|
|
66
|
+
|
|
67
|
+
4. **Read QA_REPO_BLUEPRINT.md** (if path provided by orchestrator in files_to_read):
|
|
68
|
+
- Extract exact folder structure
|
|
69
|
+
- Extract framework-specific config file contents
|
|
70
|
+
- Extract npm scripts (test:smoke, test:regression, test:api, test:unit)
|
|
71
|
+
- If no blueprint exists, use CLAUDE.md Repo Structure defaults
|
|
72
|
+
|
|
73
|
+
5. **Read .claude/skills/qa-template-engine/SKILL.md**:
|
|
74
|
+
- Extract test template patterns (unit, API, E2E)
|
|
75
|
+
- Extract POM generation rules
|
|
76
|
+
- Extract expected outcome rules
|
|
77
|
+
- These patterns guide the code generation in step 4
|
|
78
|
+
</step>
|
|
79
|
+
|
|
80
|
+
<step name="detect_existing_infrastructure">
|
|
81
|
+
Before creating any files, check what already exists to avoid overwriting or duplicating infrastructure.
|
|
82
|
+
|
|
83
|
+
**Check for existing BasePage:**
|
|
84
|
+
- Glob for `**/BasePage.*` and `**/base-page.*` across the target output directory
|
|
85
|
+
- If BasePage found: record its path, read its contents, note its class name and methods
|
|
86
|
+
- Per CONTEXT.md locked decision: "Creates BasePage.ts only if missing -- extends existing if found. Respects existing QA repo structure."
|
|
87
|
+
- If found: the executor will extend the existing BasePage, not replace it. Feature POMs will import from the existing path.
|
|
88
|
+
|
|
89
|
+
**Check for existing test config:**
|
|
90
|
+
- Glob for `playwright.config.*`, `cypress.config.*`, `jest.config.*`, `vitest.config.*`, `pytest.ini`, `pyproject.toml` (test section)
|
|
91
|
+
- If config found: record the framework and config path. Do NOT overwrite existing config.
|
|
92
|
+
- If no config found: the executor will create one in the scaffold_base step.
|
|
93
|
+
|
|
94
|
+
**Check for existing POM structure:**
|
|
95
|
+
- Glob for `pages/**/*`, `page-objects/**/*`, `support/page-objects/**/*`
|
|
96
|
+
- If existing POMs found: record the directory structure and import patterns. New POMs must follow the same conventions.
|
|
97
|
+
|
|
98
|
+
**Check for existing test files:**
|
|
99
|
+
- Glob for `tests/**/*`, `cypress/**/*`, `__tests__/**/*`
|
|
100
|
+
- If existing tests found: record the directory structure and naming conventions. New tests must follow the same patterns.
|
|
101
|
+
|
|
102
|
+
**Framework detection priority (when no config exists):**
|
|
103
|
+
1. Generation plan Summary section (framework field from planner)
|
|
104
|
+
2. QA_REPO_BLUEPRINT.md Recommended Stack
|
|
105
|
+
3. QA_ANALYSIS.md Architecture Overview (framework field)
|
|
106
|
+
|
|
107
|
+
**If no framework can be determined and no QA_REPO_BLUEPRINT.md exists:**
|
|
108
|
+
|
|
109
|
+
```
|
|
110
|
+
CHECKPOINT_RETURN:
|
|
111
|
+
completed: "Read generation plan, TEST_INVENTORY.md, checked for existing infrastructure"
|
|
112
|
+
blocking: "Cannot determine test framework -- no existing config, no blueprint, no framework in generation plan"
|
|
113
|
+
details: "Checked for: playwright.config.*, cypress.config.*, jest.config.*, vitest.config.*, pytest.ini. None found. QA_REPO_BLUEPRINT.md: not provided. Generation plan framework field: [value]. Need framework to generate correct import statements, config, and test syntax."
|
|
114
|
+
awaiting: "User specifies the test framework to use (Playwright, Cypress, Jest, Vitest, pytest)"
|
|
115
|
+
```
|
|
116
|
+
</step>
|
|
117
|
+
|
|
118
|
+
<step name="scaffold_base">
|
|
119
|
+
Create infrastructure files that other tasks depend on. This step runs before any feature-specific tasks.
|
|
120
|
+
|
|
121
|
+
**1. BasePage (if missing):**
|
|
122
|
+
|
|
123
|
+
Create `pages/base/BasePage.{ext}` following CLAUDE.md POM Rules:
|
|
124
|
+
- Shared base class that all feature POMs extend
|
|
125
|
+
- Include: constructor accepting page/browser context, navigation helper method, screenshot method, wait helper methods
|
|
126
|
+
- NO assertions -- BasePage provides utilities only
|
|
127
|
+
- Locators as readonly properties where applicable
|
|
128
|
+
- Framework-specific implementation:
|
|
129
|
+
- Playwright: `import { Page } from '@playwright/test'; constructor(protected readonly page: Page)`
|
|
130
|
+
- Cypress: class with `cy` commands, no Page parameter needed
|
|
131
|
+
- Other: adapt to framework conventions
|
|
132
|
+
|
|
133
|
+
If BasePage already exists (detected in step 2): skip creation. Record "BasePage found at {path}, extending existing."
|
|
134
|
+
|
|
135
|
+
**2. Test framework config (if missing):**
|
|
136
|
+
|
|
137
|
+
Create the appropriate config file based on the detected or chosen framework:
|
|
138
|
+
- Playwright: `playwright.config.ts` with baseURL, testDir, reporter, use settings
|
|
139
|
+
- Cypress: `cypress.config.ts` with baseUrl, specPattern, supportFile settings
|
|
140
|
+
- Jest: `jest.config.ts` with transform, testMatch, moduleNameMapper settings
|
|
141
|
+
- Vitest: `vitest.config.ts` with test.include, test.environment settings
|
|
142
|
+
- pytest: `pytest.ini` or `conftest.py` with markers and fixtures
|
|
143
|
+
|
|
144
|
+
If QA_REPO_BLUEPRINT.md exists and has Config Files section: use the blueprint's config content exactly.
|
|
145
|
+
|
|
146
|
+
If config already exists (detected in step 2): skip creation. Record "Config found at {path}, using existing."
|
|
147
|
+
|
|
148
|
+
**3. Fixture directory (if missing):**
|
|
149
|
+
|
|
150
|
+
Create `fixtures/` directory if it does not exist. The executor will populate it with fixture files during per-task generation.
|
|
151
|
+
|
|
152
|
+
**4. Directory structure:**
|
|
153
|
+
|
|
154
|
+
Create any missing directories from the generation plan's file paths:
|
|
155
|
+
- `tests/unit/`
|
|
156
|
+
- `tests/api/`
|
|
157
|
+
- `tests/integration/`
|
|
158
|
+
- `tests/e2e/smoke/`
|
|
159
|
+
- `pages/base/`
|
|
160
|
+
- `pages/{feature}/` (for each feature with POMs)
|
|
161
|
+
- `pages/components/` (if shared component POMs are needed)
|
|
162
|
+
- `fixtures/`
|
|
163
|
+
|
|
164
|
+
**Commit scaffold:**
|
|
165
|
+
```bash
|
|
166
|
+
node bin/qaa-tools.cjs commit "qa(executor): scaffold test infrastructure" --files {list of infrastructure file paths}
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Only commit if files were actually created. If all infrastructure already exists, skip the commit.
|
|
170
|
+
</step>
|
|
171
|
+
|
|
172
|
+
<step name="generate_per_task">
|
|
173
|
+
For each task in the generation plan (in dependency order from topological sort), generate the assigned files.
|
|
174
|
+
|
|
175
|
+
**Execution loop:**
|
|
176
|
+
|
|
177
|
+
For each task (ordered by dependencies):
|
|
178
|
+
|
|
179
|
+
1. **Read assigned test cases:** Look up each test_case_id in the TEST_INVENTORY.md data extracted in step 1. Collect all test case details needed for this file.
|
|
180
|
+
|
|
181
|
+
2. **Generate the file** based on file type:
|
|
182
|
+
|
|
183
|
+
**Unit test spec (`tests/unit/{feature}.unit.spec.ts`):**
|
|
184
|
+
- Import the module under test from its source path (use relative import from test file to source file)
|
|
185
|
+
- Group test cases by target function using nested `describe` blocks
|
|
186
|
+
- For each test case (UT-MODULE-NNN):
|
|
187
|
+
- Create a `describe` block for the target function
|
|
188
|
+
- Create an `it`/`test` block with the test_id as a comment: `// UT-AUTH-001`
|
|
189
|
+
- Arrange: set up concrete_inputs from TEST_INVENTORY using actual values
|
|
190
|
+
- Mock: set up mocks_needed using framework-appropriate mocking:
|
|
191
|
+
- Jest/Vitest: `vi.mock()` or `jest.mock()` for module mocks, `vi.fn()` for function mocks
|
|
192
|
+
- Playwright: mock via route interception or dependency injection
|
|
193
|
+
- Act: call the target function with the concrete input values
|
|
194
|
+
- Assert: verify expected_outcome with exact values from TEST_INVENTORY
|
|
195
|
+
- Priority: add P0/P1/P2 as a tag or comment above the test
|
|
196
|
+
- Use `expect(result).toBe(exactValue)` -- NEVER `toBeTruthy()` or `toBeDefined()` alone
|
|
197
|
+
- Use `expect(result).toEqual(expectedObject)` for object comparisons with exact field values
|
|
198
|
+
- Use `expect(() => fn()).toThrow(ExactError)` for error cases with specific error type and message
|
|
199
|
+
- Both happy-path and error cases for each function
|
|
200
|
+
- Example structure:
|
|
201
|
+
```typescript
|
|
202
|
+
import { validateToken } from '../../src/services/auth.service';
|
|
203
|
+
|
|
204
|
+
describe('validateToken', () => {
|
|
205
|
+
// UT-AUTH-001 [P0]
|
|
206
|
+
test('returns decoded payload for valid JWT token', () => {
|
|
207
|
+
// Arrange
|
|
208
|
+
const token = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...';
|
|
209
|
+
// Act
|
|
210
|
+
const result = validateToken(token);
|
|
211
|
+
// Assert
|
|
212
|
+
expect(result.userId).toBe('usr_123');
|
|
213
|
+
expect(result.role).toBe('customer');
|
|
214
|
+
});
|
|
215
|
+
|
|
216
|
+
// UT-AUTH-002 [P0]
|
|
217
|
+
test('throws TokenExpiredError for expired token', () => {
|
|
218
|
+
// Arrange
|
|
219
|
+
const expiredToken = 'eyJ...expired...';
|
|
220
|
+
// Act & Assert
|
|
221
|
+
expect(() => validateToken(expiredToken)).toThrow(TokenExpiredError);
|
|
222
|
+
expect(() => validateToken(expiredToken)).toThrow('Token has expired');
|
|
223
|
+
});
|
|
224
|
+
});
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
**API test spec (`tests/api/{resource}.api.spec.ts`):**
|
|
228
|
+
- Import the API client or use framework's request helper
|
|
229
|
+
- Set up base URL from environment variable: `const baseUrl = process.env.API_URL || 'http://localhost:3000'`
|
|
230
|
+
- Group test cases by endpoint using `describe` blocks
|
|
231
|
+
- For each test case (API-RESOURCE-NNN):
|
|
232
|
+
- Create a `describe` block for the endpoint (e.g., `POST /api/v1/users`)
|
|
233
|
+
- Create an `it`/`test` block with the test_id as a comment: `// API-USERS-001`
|
|
234
|
+
- Arrange: prepare request_body (exact JSON payload), headers from TEST_INVENTORY
|
|
235
|
+
- Act: make the HTTP request using the detected framework:
|
|
236
|
+
- Playwright: `request.post(url, { data: payload })`
|
|
237
|
+
- Supertest: `request(app).post(url).send(payload)`
|
|
238
|
+
- Axios/fetch: `axios.post(url, payload, { headers })`
|
|
239
|
+
- Assert: verify expected_status (exact HTTP code) and expected_response (exact response body fields)
|
|
240
|
+
- Include both success (200/201) and error (400/401/404) scenarios
|
|
241
|
+
- Use environment variables for base URL and auth tokens, never hardcode
|
|
242
|
+
- Example structure:
|
|
243
|
+
```typescript
|
|
244
|
+
describe('POST /api/v1/users', () => {
|
|
245
|
+
// API-USERS-001 [P0]
|
|
246
|
+
test('creates a new user with valid data', async () => {
|
|
247
|
+
const response = await request.post(`${baseUrl}/api/v1/users`, {
|
|
248
|
+
data: { email: 'newuser@example.com', password: 'SecureP@ss123!', name: 'Test User' }
|
|
249
|
+
});
|
|
250
|
+
expect(response.status()).toBe(201);
|
|
251
|
+
const body = await response.json();
|
|
252
|
+
expect(body.email).toBe('newuser@example.com');
|
|
253
|
+
expect(body.name).toBe('Test User');
|
|
254
|
+
expect(body).toHaveProperty('id');
|
|
255
|
+
});
|
|
256
|
+
|
|
257
|
+
// API-USERS-002 [P0]
|
|
258
|
+
test('returns 400 for missing email', async () => {
|
|
259
|
+
const response = await request.post(`${baseUrl}/api/v1/users`, {
|
|
260
|
+
data: { password: 'SecureP@ss123!', name: 'Test User' }
|
|
261
|
+
});
|
|
262
|
+
expect(response.status()).toBe(400);
|
|
263
|
+
const body = await response.json();
|
|
264
|
+
expect(body.error).toBe('Email is required');
|
|
265
|
+
});
|
|
266
|
+
});
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
**Integration test spec (`tests/integration/{feature}.integration.spec.ts`):**
|
|
270
|
+
- Set up the test environment with the components_involved (database, services, etc.)
|
|
271
|
+
- For each test case (INT-MODULE-NNN):
|
|
272
|
+
- Apply setup_required: seed database, start mock servers, initialize service instances
|
|
273
|
+
- Execute the integration flow -- call the primary service method that triggers cross-module interaction
|
|
274
|
+
- Assert expected_outcome with specific values that verify the interaction succeeded
|
|
275
|
+
- Clean up: reset database state, stop mock servers
|
|
276
|
+
- Use `beforeEach`/`afterEach` for test isolation
|
|
277
|
+
- Example structure:
|
|
278
|
+
```typescript
|
|
279
|
+
describe('OrderService + PaymentService integration', () => {
|
|
280
|
+
beforeEach(async () => {
|
|
281
|
+
await db.seed({ users: [testUser], products: [testProduct] });
|
|
282
|
+
});
|
|
283
|
+
|
|
284
|
+
afterEach(async () => {
|
|
285
|
+
await db.cleanup();
|
|
286
|
+
});
|
|
287
|
+
|
|
288
|
+
// INT-ORDER-001 [P0]
|
|
289
|
+
test('creates order and processes payment in single transaction', async () => {
|
|
290
|
+
const order = await orderService.create({
|
|
291
|
+
userId: 'usr_123', items: [{ productId: 'prod_456', quantity: 3 }]
|
|
292
|
+
});
|
|
293
|
+
expect(order.status).toBe('confirmed');
|
|
294
|
+
expect(order.total).toBe(89.97);
|
|
295
|
+
const payment = await paymentService.getByOrderId(order.id);
|
|
296
|
+
expect(payment.status).toBe('captured');
|
|
297
|
+
expect(payment.amount).toBe(89.97);
|
|
298
|
+
});
|
|
299
|
+
});
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
**E2E test spec (`tests/e2e/smoke/{feature}.e2e.spec.ts`):**
|
|
303
|
+
- Import the feature POM(s) from pages/{feature}/
|
|
304
|
+
- Import fixture data from fixtures/
|
|
305
|
+
- For each test case (E2E-FLOW-NNN):
|
|
306
|
+
- Create a `test` block with the test_id as a comment: `// E2E-LOGIN-001`
|
|
307
|
+
- Instantiate required POM(s) in the test or in `beforeEach`
|
|
308
|
+
- Follow user_journey steps using POM action methods (never direct page interactions)
|
|
309
|
+
- Assert expected_outcome using POM state queries combined with test assertions
|
|
310
|
+
- All page interactions go through the POM -- never call `page.click()` or `page.fill()` directly in the spec
|
|
311
|
+
- Use Tier 1 locators exclusively in the POM (data-testid, ARIA roles)
|
|
312
|
+
- NO assertions in the POM -- all assertions in the spec file using `expect()`
|
|
313
|
+
- Use fixture data for test inputs, not magic strings inline
|
|
314
|
+
- Example structure (Playwright):
|
|
315
|
+
```typescript
|
|
316
|
+
import { test, expect } from '@playwright/test';
|
|
317
|
+
import { LoginPage } from '../../pages/auth/LoginPage';
|
|
318
|
+
import { DashboardPage } from '../../pages/dashboard/DashboardPage';
|
|
319
|
+
import { testUser } from '../../fixtures/auth-data';
|
|
320
|
+
|
|
321
|
+
test.describe('Login Flow', () => {
|
|
322
|
+
// E2E-LOGIN-001 [P0]
|
|
323
|
+
test('user can log in with valid credentials and see dashboard', async ({ page }) => {
|
|
324
|
+
const loginPage = new LoginPage(page);
|
|
325
|
+
const dashboardPage = new DashboardPage(page);
|
|
326
|
+
|
|
327
|
+
await loginPage.navigateTo();
|
|
328
|
+
await loginPage.login(testUser.email, testUser.password);
|
|
329
|
+
|
|
330
|
+
await expect(dashboardPage.welcomeMessage).toHaveText('Welcome, Test User');
|
|
331
|
+
await expect(page).toHaveURL('/dashboard');
|
|
332
|
+
});
|
|
333
|
+
});
|
|
334
|
+
```
|
|
335
|
+
- Example structure (Cypress):
|
|
336
|
+
```typescript
|
|
337
|
+
import { LoginPage } from '../../pages/auth/LoginPage';
|
|
338
|
+
import { DashboardPage } from '../../pages/dashboard/DashboardPage';
|
|
339
|
+
import { testUser } from '../../fixtures/auth-data';
|
|
340
|
+
|
|
341
|
+
describe('Login Flow', () => {
|
|
342
|
+
const loginPage = new LoginPage();
|
|
343
|
+
const dashboardPage = new DashboardPage();
|
|
344
|
+
|
|
345
|
+
// E2E-LOGIN-001 [P0]
|
|
346
|
+
it('user can log in with valid credentials and see dashboard', () => {
|
|
347
|
+
loginPage.navigateTo();
|
|
348
|
+
loginPage.login(testUser.email, testUser.password);
|
|
349
|
+
|
|
350
|
+
dashboardPage.getWelcomeText().should('eq', 'Welcome, Test User');
|
|
351
|
+
cy.url().should('include', '/dashboard');
|
|
352
|
+
});
|
|
353
|
+
});
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
**Feature POM (`pages/{feature}/{Feature}Page.ts`):**
|
|
357
|
+
- Extend BasePage (import from the base directory)
|
|
358
|
+
- Constructor accepts the framework's page/browser context
|
|
359
|
+
- Define ALL locators as readonly properties at the class level (never inline in methods):
|
|
360
|
+
```typescript
|
|
361
|
+
// Playwright POM example
|
|
362
|
+
import { Page } from '@playwright/test';
|
|
363
|
+
import { BasePage } from '../base/BasePage';
|
|
364
|
+
|
|
365
|
+
export class LoginPage extends BasePage {
|
|
366
|
+
// Locators -- Tier 1 (data-testid and ARIA roles)
|
|
367
|
+
readonly emailInput = this.page.getByTestId('login-email-input');
|
|
368
|
+
readonly passwordInput = this.page.getByTestId('login-password-input');
|
|
369
|
+
readonly submitButton = this.page.getByRole('button', { name: 'Log in' });
|
|
370
|
+
readonly errorMessage = this.page.getByTestId('login-error-alert');
|
|
371
|
+
|
|
372
|
+
// Locators -- Tier 2 (label/placeholder, only when Tier 1 unavailable)
|
|
373
|
+
readonly rememberMeCheckbox = this.page.getByLabel('Remember me');
|
|
374
|
+
|
|
375
|
+
constructor(page: Page) {
|
|
376
|
+
super(page);
|
|
377
|
+
}
|
|
378
|
+
|
|
379
|
+
// Actions -- return void or next page
|
|
380
|
+
async navigateTo(): Promise<void> {
|
|
381
|
+
await this.page.goto('/login');
|
|
382
|
+
}
|
|
383
|
+
|
|
384
|
+
async login(email: string, password: string): Promise<void> {
|
|
385
|
+
await this.emailInput.fill(email);
|
|
386
|
+
await this.passwordInput.fill(password);
|
|
387
|
+
await this.submitButton.click();
|
|
388
|
+
}
|
|
389
|
+
|
|
390
|
+
// State queries -- return data, NO assertions
|
|
391
|
+
async getErrorText(): Promise<string> {
|
|
392
|
+
return await this.errorMessage.textContent() ?? '';
|
|
393
|
+
}
|
|
394
|
+
|
|
395
|
+
async isFormVisible(): Promise<boolean> {
|
|
396
|
+
return await this.emailInput.isVisible();
|
|
397
|
+
}
|
|
398
|
+
}
|
|
399
|
+
```
|
|
400
|
+
- Cypress POM example:
|
|
401
|
+
```typescript
|
|
402
|
+
import { BasePage } from '../base/BasePage';
|
|
403
|
+
|
|
404
|
+
export class LoginPage extends BasePage {
|
|
405
|
+
// Locators -- Tier 1
|
|
406
|
+
readonly emailInput = '[data-testid="login-email-input"]';
|
|
407
|
+
readonly passwordInput = '[data-testid="login-password-input"]';
|
|
408
|
+
readonly submitButton = '[data-testid="login-submit-btn"]';
|
|
409
|
+
readonly errorMessage = '[data-testid="login-error-alert"]';
|
|
410
|
+
|
|
411
|
+
navigateTo(): void {
|
|
412
|
+
cy.visit('/login');
|
|
413
|
+
}
|
|
414
|
+
|
|
415
|
+
login(email: string, password: string): void {
|
|
416
|
+
cy.get(this.emailInput).type(email);
|
|
417
|
+
cy.get(this.passwordInput).type(password);
|
|
418
|
+
cy.get(this.submitButton).click();
|
|
419
|
+
}
|
|
420
|
+
|
|
421
|
+
getErrorText(): Cypress.Chainable<string> {
|
|
422
|
+
return cy.get(this.errorMessage).invoke('text');
|
|
423
|
+
}
|
|
424
|
+
}
|
|
425
|
+
```
|
|
426
|
+
- If Tier 1 locators not available, fall back to Tier 2 (labels, text), then Tier 3 (alt, title)
|
|
427
|
+
- If forced to use Tier 4 (CSS/XPath): add `// TODO: Request test ID for this element` comment
|
|
428
|
+
- POM locators are readonly properties, NOT inline strings scattered in methods
|
|
429
|
+
- One POM class per page or view -- no god objects combining multiple pages
|
|
430
|
+
|
|
431
|
+
**Fixture data file (`fixtures/{domain}-data.ts`):**
|
|
432
|
+
- Export typed test data objects with realistic but fake values
|
|
433
|
+
- Reference concrete_inputs from TEST_INVENTORY test cases -- these are the values tests will use
|
|
434
|
+
- Use environment variables with fallbacks for any sensitive or environment-specific values
|
|
435
|
+
- Organize by domain: auth fixtures in auth-data, product fixtures in product-data
|
|
436
|
+
- Example structure:
|
|
437
|
+
```typescript
|
|
438
|
+
// fixtures/auth-data.ts
|
|
439
|
+
export const testUser = {
|
|
440
|
+
email: process.env.TEST_EMAIL || 'test@example.com',
|
|
441
|
+
password: process.env.TEST_PASSWORD || 'SecureP@ss123!',
|
|
442
|
+
name: 'Test User',
|
|
443
|
+
};
|
|
444
|
+
|
|
445
|
+
export const adminUser = {
|
|
446
|
+
email: process.env.ADMIN_EMAIL || 'admin@example.com',
|
|
447
|
+
password: process.env.ADMIN_PASSWORD || 'AdminP@ss456!',
|
|
448
|
+
name: 'Admin User',
|
|
449
|
+
role: 'admin',
|
|
450
|
+
};
|
|
451
|
+
|
|
452
|
+
export const invalidCredentials = {
|
|
453
|
+
email: 'nonexistent@example.com',
|
|
454
|
+
password: 'WrongPassword123!',
|
|
455
|
+
};
|
|
456
|
+
```
|
|
457
|
+
- NEVER hardcode real credentials, API keys, or secrets
|
|
458
|
+
- Each domain gets its own fixture file following `{domain}-data.{ext}` naming
|
|
459
|
+
|
|
460
|
+
3. **Apply CLAUDE.md standards** to every generated file:
|
|
461
|
+
- Tier 1 locators preferred (data-testid, ARIA roles) -- always try these first
|
|
462
|
+
- No assertions inside page objects -- page objects return data, tests make assertions
|
|
463
|
+
- Concrete assertion values -- exact status codes, exact text content, exact return values
|
|
464
|
+
- No vague words in assertions: "correct", "proper", "appropriate", "works" MUST have a concrete value
|
|
465
|
+
- Unique test IDs following naming convention (UT-MODULE-NNN, API-RESOURCE-NNN, etc.)
|
|
466
|
+
- Correct file naming convention from CLAUDE.md Naming Conventions table
|
|
467
|
+
- No hardcoded credentials -- use environment variables with test fallbacks
|
|
468
|
+
- Priority (P0/P1/P2) tagged on every test case as a comment
|
|
469
|
+
|
|
470
|
+
4. **Anti-pattern verification per file** (check BEFORE committing):
|
|
471
|
+
- Scan the generated file for BAD assertion patterns:
|
|
472
|
+
- `toBeTruthy()` without a preceding specific check -- REPLACE with `toBe(expectedValue)`
|
|
473
|
+
- `toBeDefined()` alone -- REPLACE with `toBe(expectedValue)` or `toEqual(expectedObject)`
|
|
474
|
+
- `.should('exist')` without content check -- ADD content assertion
|
|
475
|
+
- Scan for inline locators in POM action methods -- MOVE to class-level readonly properties
|
|
476
|
+
- Scan for assertions inside POM files -- MOVE to test spec files
|
|
477
|
+
- Scan for hardcoded URLs -- REPLACE with environment variables
|
|
478
|
+
- Scan for magic string test data -- REPLACE with fixture imports
|
|
479
|
+
|
|
480
|
+
5. **Commit one test file per commit** (per CONTEXT.md locked decision: "One test file per commit: 'test(auth): add login.e2e.spec.ts'. Maximum traceability."):
|
|
481
|
+
```bash
|
|
482
|
+
node bin/qaa-tools.cjs commit "test({feature}): add {filename}" --files {file_path}
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
Replace `{feature}` with the feature_group name (e.g., "auth", "product", "order").
|
|
486
|
+
Replace `{filename}` with the actual filename (e.g., "login.e2e.spec.ts", "auth.unit.spec.ts").
|
|
487
|
+
Replace `{file_path}` with the full path to the file.
|
|
488
|
+
|
|
489
|
+
**Important:** Commit one file at a time. Do NOT batch multiple files in a single commit. The one-file-per-commit pattern provides maximum traceability -- every file change can be traced to a specific commit, reviewed independently, and reverted without affecting other files.
|
|
490
|
+
|
|
491
|
+
6. **Track progress:** After each task, record: task_id, files_created (with paths), commit_hash, test_case_count.
|
|
492
|
+
</step>
|
|
493
|
+
|
|
494
|
+
<step name="verify_output">
|
|
495
|
+
After all tasks are complete, verify the output is correct and complete.
|
|
496
|
+
|
|
497
|
+
**1. File existence check:**
|
|
498
|
+
For every file path listed in the generation plan's files_to_create fields, verify the file exists on disk:
|
|
499
|
+
```
|
|
500
|
+
[ -f "{file_path}" ] && echo "FOUND: {file_path}" || echo "MISSING: {file_path}"
|
|
501
|
+
```
|
|
502
|
+
If any file is missing, generate it now and commit.
|
|
503
|
+
|
|
504
|
+
**2. Import resolution check:**
|
|
505
|
+
For each generated file, verify that its imports reference files that exist:
|
|
506
|
+
- POM imports of BasePage: verify BasePage file exists at the import path
|
|
507
|
+
- E2E spec imports of POMs: verify POM files exist at the import paths
|
|
508
|
+
- Test spec imports of fixtures: verify fixture files exist at the import paths
|
|
509
|
+
- Test spec imports of source modules: verify source modules exist (these are in the DEV repo, not generated)
|
|
510
|
+
|
|
511
|
+
If any import cannot resolve to an existing file (among generated files), fix the import path and re-commit.
|
|
512
|
+
|
|
513
|
+
**3. No skipped tasks:**
|
|
514
|
+
Compare the list of completed tasks against the generation plan's task list. Every task must be completed. If any task was skipped, execute it now.
|
|
515
|
+
|
|
516
|
+
**4. Commit count verification:**
|
|
517
|
+
Count the total commits made during generation. This should approximately match the total_files count from the generation plan (one commit per file, plus the scaffold commit).
|
|
518
|
+
</step>
|
|
519
|
+
|
|
520
|
+
</process>
|
|
521
|
+
|
|
522
|
+
<output>
|
|
523
|
+
The executor agent produces multiple artifacts:
|
|
524
|
+
|
|
525
|
+
**Infrastructure (if missing):**
|
|
526
|
+
- `pages/base/BasePage.{ext}` -- Shared base page object (only if not already present)
|
|
527
|
+
- Test framework config file (only if not already present)
|
|
528
|
+
- Directory structure for tests, pages, fixtures
|
|
529
|
+
|
|
530
|
+
**Per-feature test files:**
|
|
531
|
+
- Unit test specs: `tests/unit/{feature}.unit.spec.{ext}`
|
|
532
|
+
- API test specs: `tests/api/{resource}.api.spec.{ext}`
|
|
533
|
+
- Integration test specs: `tests/integration/{feature}.integration.spec.{ext}`
|
|
534
|
+
- E2E smoke test specs: `tests/e2e/smoke/{feature}.e2e.spec.{ext}`
|
|
535
|
+
- Feature POMs: `pages/{feature}/{Feature}Page.{ext}`
|
|
536
|
+
- Component POMs: `pages/components/{Component}.{ext}` (if needed)
|
|
537
|
+
- Fixture data files: `fixtures/{domain}-data.{ext}`
|
|
538
|
+
|
|
539
|
+
All files are written to paths defined in the generation plan and follow CLAUDE.md standards.
|
|
540
|
+
|
|
541
|
+
**Return to orchestrator:**
|
|
542
|
+
|
|
543
|
+
After all tasks complete and verification passes, return these values:
|
|
544
|
+
|
|
545
|
+
```
|
|
546
|
+
EXECUTOR_COMPLETE:
|
|
547
|
+
files_created:
|
|
548
|
+
- path: "{file_path_1}"
|
|
549
|
+
type: "{unit_spec|api_spec|e2e_spec|pom|fixture|config}"
|
|
550
|
+
- path: "{file_path_2}"
|
|
551
|
+
type: "{type}"
|
|
552
|
+
[... one entry per file created ...]
|
|
553
|
+
total_files: {N}
|
|
554
|
+
commit_count: {N}
|
|
555
|
+
features_covered:
|
|
556
|
+
- "{feature_1}"
|
|
557
|
+
- "{feature_2}"
|
|
558
|
+
[... one entry per feature group ...]
|
|
559
|
+
test_case_count: {N}
|
|
560
|
+
```
|
|
561
|
+
</output>
|
|
562
|
+
|
|
563
|
+
<quality_gate>
|
|
564
|
+
Before considering the executor's work complete, verify ALL of the following.
|
|
565
|
+
|
|
566
|
+
**From CLAUDE.md Quality Gates (verbatim):**
|
|
567
|
+
|
|
568
|
+
- [ ] Every test case has an explicit expected outcome with a concrete value
|
|
569
|
+
- [ ] No outcome says "correct", "proper", "appropriate", or "works" without defining what that means
|
|
570
|
+
- [ ] All locators follow the tier hierarchy (Tier 1 preferred: data-testid, ARIA roles)
|
|
571
|
+
- [ ] No assertions inside page objects (assertions belong ONLY in test specs)
|
|
572
|
+
- [ ] No hardcoded credentials (use environment variables with test fallbacks)
|
|
573
|
+
- [ ] File naming follows the project's existing conventions (or CLAUDE.md standards if none exist)
|
|
574
|
+
- [ ] Test IDs are unique and follow naming convention (UT-MODULE-NNN, API-RESOURCE-NNN, E2E-FLOW-NNN)
|
|
575
|
+
- [ ] Priority assigned to every test case (P0, P1, or P2)
|
|
576
|
+
- [ ] Framework matches what the project already uses
|
|
577
|
+
|
|
578
|
+
**Additional executor-specific checks:**
|
|
579
|
+
|
|
580
|
+
- [ ] All planned files exist on disk (every file_path from generation plan verified)
|
|
581
|
+
- [ ] Imports resolve (no broken references between generated files)
|
|
582
|
+
- [ ] BasePage check performed before creating one (only if missing -- extends existing if found)
|
|
583
|
+
- [ ] One commit per test file (not batch commits -- each file has its own commit)
|
|
584
|
+
- [ ] Framework config matches detected or user-specified framework
|
|
585
|
+
- [ ] POM locators are readonly properties, not inline strings in methods
|
|
586
|
+
- [ ] POM actions return void or next page (no other return types)
|
|
587
|
+
- [ ] POM state queries return data (no assertions inside queries)
|
|
588
|
+
- [ ] Every POM extends BasePage (or the project's existing shared base)
|
|
589
|
+
- [ ] Tier 1 locators used wherever possible (data-testid, getByRole)
|
|
590
|
+
- [ ] Tier 4 locators (CSS/XPath) have `// TODO: Request test ID for this element` comment
|
|
591
|
+
- [ ] Unit tests use Arrange/Act/Assert pattern
|
|
592
|
+
- [ ] API tests verify exact status code AND response body fields
|
|
593
|
+
- [ ] E2E tests follow user journey steps from TEST_INVENTORY
|
|
594
|
+
- [ ] Fixture data uses realistic fake data (no real credentials, no generic placeholders)
|
|
595
|
+
- [ ] Commit messages follow `test({feature}): add {filename}` format
|
|
596
|
+
- [ ] No generated file references a non-existent import
|
|
597
|
+
|
|
598
|
+
If any check fails, fix the issue before returning EXECUTOR_COMPLETE. Do not proceed with a failing quality gate.
|
|
599
|
+
</quality_gate>
|
|
600
|
+
|
|
601
|
+
<success_criteria>
|
|
602
|
+
The executor agent has completed successfully when:
|
|
603
|
+
|
|
604
|
+
1. All planned files from the generation plan exist on disk at their assigned paths
|
|
605
|
+
2. Every file was committed individually with message format `test({feature}): add {filename}` via `node bin/qaa-tools.cjs commit`
|
|
606
|
+
3. BasePage check was performed -- created only if missing, extended existing if found
|
|
607
|
+
4. All imports between generated files resolve correctly (POM -> BasePage, E2E spec -> POM, spec -> fixture)
|
|
608
|
+
5. Every generated test file follows CLAUDE.md standards:
|
|
609
|
+
- Tier 1 locators preferred (data-testid, ARIA roles)
|
|
610
|
+
- No assertions in page objects
|
|
611
|
+
- Concrete assertion values (exact status codes, exact response fields, exact text content)
|
|
612
|
+
- Unique test IDs following naming convention
|
|
613
|
+
- Priority tagged on every test case
|
|
614
|
+
6. Every POM follows all 6 POM rules from CLAUDE.md
|
|
615
|
+
7. No hardcoded credentials in any file (environment variables with fallbacks used instead)
|
|
616
|
+
8. All quality gate checks pass
|
|
617
|
+
9. Return values provided to orchestrator: files_created, total_files, commit_count, features_covered, test_case_count
|
|
618
|
+
</success_criteria>
|