@sun-asterisk/sungen 2.7.0-beta.0 → 2.7.0-beta.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/orchestrator/project-initializer.d.ts +5 -0
- package/dist/orchestrator/project-initializer.d.ts.map +1 -1
- package/dist/orchestrator/project-initializer.js +16 -0
- package/dist/orchestrator/project-initializer.js.map +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-cmd-create-test.md +9 -1
- package/dist/orchestrator/templates/ai-instructions/claude-cmd-run-test.md +4 -2
- package/dist/orchestrator/templates/ai-instructions/claude-skill-selector-fix.md +57 -11
- package/dist/orchestrator/templates/ai-instructions/claude-skill-selector-keys.md +41 -31
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +13 -0
- package/dist/orchestrator/templates/ai-instructions/copilot-cmd-create-test.md +9 -1
- package/dist/orchestrator/templates/ai-instructions/copilot-cmd-run-test.md +4 -2
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-selector-fix.md +61 -15
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-selector-keys.md +41 -31
- package/dist/orchestrator/templates/qa-context.md +90 -0
- package/dist/orchestrator/templates/readme.md +16 -13
- package/package.json +1 -1
- package/src/orchestrator/project-initializer.ts +20 -0
- package/src/orchestrator/templates/ai-instructions/claude-cmd-create-test.md +9 -1
- package/src/orchestrator/templates/ai-instructions/claude-cmd-run-test.md +4 -2
- package/src/orchestrator/templates/ai-instructions/claude-skill-selector-fix.md +57 -11
- package/src/orchestrator/templates/ai-instructions/claude-skill-selector-keys.md +41 -31
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +13 -0
- package/src/orchestrator/templates/ai-instructions/copilot-cmd-create-test.md +9 -1
- package/src/orchestrator/templates/ai-instructions/copilot-cmd-run-test.md +4 -2
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-selector-fix.md +61 -15
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-selector-keys.md +41 -31
- package/src/orchestrator/templates/qa-context.md +90 -0
- package/src/orchestrator/templates/readme.md +16 -13
|
@@ -166,37 +166,47 @@ Resolver searches in this order:
|
|
|
166
166
|
|
|
167
167
|
If no YAML key exists, the resolver infers from the Gherkin element type:
|
|
168
168
|
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
|
178
|
-
|
|
179
|
-
| `[X]
|
|
180
|
-
| `[X]
|
|
181
|
-
| `[X]
|
|
182
|
-
| `[X]
|
|
183
|
-
| `[X]
|
|
184
|
-
| `[X]
|
|
185
|
-
| `[X]
|
|
186
|
-
| `[X]
|
|
187
|
-
| `[X]
|
|
188
|
-
| `[X]
|
|
189
|
-
| `[X]
|
|
190
|
-
| `[X]
|
|
191
|
-
| `[X]
|
|
192
|
-
| `[X]
|
|
193
|
-
| `[X]
|
|
194
|
-
| `[X]
|
|
195
|
-
| `[X]
|
|
196
|
-
| `[X]
|
|
197
|
-
| `[X]
|
|
198
|
-
|
|
199
|
-
|
|
169
|
+
> ⚠️ **Auto-infer pitfall — the #1 cause of selector failures in production.**
|
|
170
|
+
>
|
|
171
|
+
> `[X] button` auto-infers as `getByRole('button', { name: 'X' })`. This **only works** when the button's accessible name in the DOM is **exactly `X`** — same language, same text, same casing.
|
|
172
|
+
>
|
|
173
|
+
> The Gherkin `[Reference]` is your human label for the element, **not** the DOM name. If the app is in Vietnamese (or any language where the Gherkin label differs from DOM text), auto-infer will produce `No element found` at runtime. **Write an explicit YAML entry** with the real DOM name instead.
|
|
174
|
+
>
|
|
175
|
+
> **Decision rule**: auto-infer is safe ONLY when you have confirmed in the snapshot that the DOM element's accessible name / placeholder text is literally `X`. When in doubt → write YAML.
|
|
176
|
+
|
|
177
|
+
| Gherkin | Inferred locator | Safe when… |
|
|
178
|
+
|---|---|---|
|
|
179
|
+
| `[X] button` | `getByRole('button', { name: 'X' })` | Button's accessible name = X |
|
|
180
|
+
| `[X] link` | `getByRole('link', { name: 'X' })` | Link text = X |
|
|
181
|
+
| `[X] heading` / `header` | `getByRole('heading', { name: 'X' })` | Heading text = X |
|
|
182
|
+
| `[X] checkbox` | `getByRole('checkbox', { name: 'X' })` | Checkbox label = X |
|
|
183
|
+
| `[X] radio` | `getByRole('radio', { name: 'X' })` | Radio label = X |
|
|
184
|
+
| `[X] field` | `getByPlaceholder('X')` | Placeholder text = X AND field has a placeholder |
|
|
185
|
+
| `[X] text` / `message` / `label` | `getByText('X')` | Visible text = X (partial match) |
|
|
186
|
+
| `[X] logo/image/icon` | `getByRole('img', { name: 'X' })` | Image alt = X |
|
|
187
|
+
| `[X] search` | `getByRole('searchbox', { name: 'X' })` | Searchbox label = X |
|
|
188
|
+
| `[X] option` | `getByRole('option', { name: 'X' })` | Option text = X |
|
|
189
|
+
| `[X] slider` | `getByRole('slider', { name: 'X' })` | Slider label = X |
|
|
190
|
+
| `[X] toggle` | `getByRole('switch', { name: 'X' })` | Toggle label = X |
|
|
191
|
+
| `[X] tab` | `getByRole('tab', { name: 'X' })` | Tab text = X |
|
|
192
|
+
| `[X] table` | `getByRole('table', { name: 'X' })` | Table aria-label = X |
|
|
193
|
+
| `[X] list` | `getByRole('list', { name: 'X' })` | List aria-label = X |
|
|
194
|
+
| `[X] column` | `getByRole('columnheader', { name: 'X' })` | Column header text = X |
|
|
195
|
+
| `[X] dialog` / `modal` / `drawer` | `getByRole('dialog', { name: 'X' })` | Dialog aria-label/heading = X |
|
|
196
|
+
| `[X] dropdown` / `select` | `getByRole('combobox', { name: 'X' })` | Combobox label = X |
|
|
197
|
+
| `[X] menuitem` | `getByRole('menuitem', { name: 'X' })` | Menu item text = X |
|
|
198
|
+
| `[X] progressbar` | `getByRole('progressbar', { name: 'X' })` | Progressbar label = X |
|
|
199
|
+
| `[X] section` | `getByRole('region', { name: 'X' })` | Section aria-label = X |
|
|
200
|
+
| `[X] card` | `getByRole('article', { name: 'X' })` | Card aria-label = X |
|
|
201
|
+
| `[X] item` | `getByRole('listitem', { name: 'X' })` | List item text = X |
|
|
202
|
+
| `[X] cell` | `getByRole('cell', { name: 'X' })` | Cell text = X |
|
|
203
|
+
| `[X] spinner` | `getByRole('status', { name: 'X' })` | Spinner aria-label = X |
|
|
204
|
+
| `[X] breadcrumb` | `getByRole('navigation', { name: 'X' })` | Navigation aria-label = X |
|
|
205
|
+
| `[X] badge` / `tooltip` / `tag` | `getByText('X')` | Visible text = X |
|
|
206
|
+
|
|
207
|
+
**Special note on `[X] field`**: `getByPlaceholder('X')` only works when (1) the field has a placeholder attribute AND (2) the placeholder text equals X. For fields without placeholders (floating labels, aria-label), write explicit YAML: `type: label, value: "Actual label text"`.
|
|
208
|
+
|
|
209
|
+
**Only add a YAML entry when** auto-infer cannot work: DOM name differs from Gherkin label, need `testid`, need `nth`, need `exact: true`, or the field type requires explicit config.
|
|
200
210
|
|
|
201
211
|
### Types requiring YAML entry (no auto-infer)
|
|
202
212
|
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# Project Context
|
|
2
|
+
|
|
3
|
+
> Read by the AI before generating test cases for any screen in this project.
|
|
4
|
+
> Fill in what applies — leave sections empty if not relevant.
|
|
5
|
+
> **The more specific you are, the more accurate the generated test cases.**
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Project Overview
|
|
10
|
+
|
|
11
|
+
**Application:**
|
|
12
|
+
<!-- One sentence: what does this app do? -->
|
|
13
|
+
<!-- Example: B2B award nomination platform for enterprise HR teams. -->
|
|
14
|
+
|
|
15
|
+
**Target users:**
|
|
16
|
+
<!-- Who uses this app and in what context? -->
|
|
17
|
+
<!-- Example: HR managers submit nominations; employees view results. -->
|
|
18
|
+
|
|
19
|
+
**Domain notes:**
|
|
20
|
+
<!-- Key terminology, conventions, or constraints the AI should know. -->
|
|
21
|
+
<!-- Example: "Nomination = an award record. Once submitted, status cannot revert to Draft." -->
|
|
22
|
+
<!-- Example: "All monetary values are in JPY. No decimal places." -->
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Auth Roles
|
|
27
|
+
|
|
28
|
+
> The AI maps these directly to `@auth:X` tags and generates permission-boundary test scenarios.
|
|
29
|
+
> Leave the table empty (or delete it) if the app has no auth system.
|
|
30
|
+
|
|
31
|
+
| Role | Can do | Cannot do |
|
|
32
|
+
|------|--------|-----------|
|
|
33
|
+
| | | |
|
|
34
|
+
|
|
35
|
+
<!--
|
|
36
|
+
Example:
|
|
37
|
+
| Role | Can do | Cannot do |
|
|
38
|
+
|---------|---------------------------------------------|--------------------------------------|
|
|
39
|
+
| admin | All CRUD, manage users, configure settings | Nothing blocked |
|
|
40
|
+
| manager | Create/edit records, view reports | Delete records, manage users |
|
|
41
|
+
| staff | View and submit own records only | Edit others' records, view reports |
|
|
42
|
+
-->
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## Testing Strategy
|
|
47
|
+
|
|
48
|
+
**Focus areas** — what to cover thoroughly:
|
|
49
|
+
<!-- List from: functional, security, ui, accessibility, performance -->
|
|
50
|
+
<!-- Example: functional, security -->
|
|
51
|
+
|
|
52
|
+
**Mandatory coverage:**
|
|
53
|
+
<!-- Rules that override the AI's default tier decisions for every screen. -->
|
|
54
|
+
<!-- Example: "Every screen with admin-only actions MUST have a non-admin blocked-access scenario." -->
|
|
55
|
+
<!-- Example: "All free-text inputs MUST have XSS + SQL injection scenarios regardless of screen risk level." -->
|
|
56
|
+
|
|
57
|
+
**Deprioritize / skip:**
|
|
58
|
+
<!-- What to move to @low or skip entirely for this project. -->
|
|
59
|
+
<!-- Example: "Skip VP-UI cosmetic checks (label/placeholder presence) — handled separately by design review." -->
|
|
60
|
+
<!-- Example: "Skip accessibility scenarios — separate audit planned." -->
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Global Business Rules
|
|
65
|
+
|
|
66
|
+
> Rules that apply across multiple screens.
|
|
67
|
+
> The AI adds these to the Coverage Map for every screen as `[G]`-tagged Business rules.
|
|
68
|
+
> Screen-specific rules belong in `requirements/spec.md`, not here.
|
|
69
|
+
|
|
70
|
+
<!-- - Soft-delete only: records are never hard-deleted, only marked inactive -->
|
|
71
|
+
<!-- - All timestamps stored in UTC, displayed in UTC+7 -->
|
|
72
|
+
<!-- - Pagination default: 20 items per page; max 100 -->
|
|
73
|
+
<!-- - File uploads: PNG/JPG/PDF only, max 5 MB -->
|
|
74
|
+
<!-- - After any write operation, the list view must refresh automatically -->
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## Error Message Patterns
|
|
79
|
+
|
|
80
|
+
> If your app follows consistent validation error formats, list them here.
|
|
81
|
+
> The AI uses these to fill `test-data.yaml` error keys when `spec.md` doesn't specify exact text.
|
|
82
|
+
> Leave empty to let the AI infer from spec.md.
|
|
83
|
+
|
|
84
|
+
- Required field: `<!-- "This field is required" -->`
|
|
85
|
+
- Max length: `<!-- "Must be X characters or less" -->`
|
|
86
|
+
- Min length: `<!-- "Must be at least X characters" -->`
|
|
87
|
+
- Invalid format: `<!-- "Invalid format" -->`
|
|
88
|
+
- Unique constraint: `<!-- "Already exists" -->`
|
|
89
|
+
- Not found: `<!-- "Not found" -->`
|
|
90
|
+
- Unauthorized: `<!-- "You do not have permission to perform this action" -->`
|
|
@@ -12,14 +12,16 @@ sungen generate → compiles Gherkin + selectors + data → Playwright .spec.ts
|
|
|
12
12
|
## Directory Structure
|
|
13
13
|
|
|
14
14
|
```
|
|
15
|
-
├── qa/
|
|
16
|
-
│ ├──
|
|
17
|
-
│ ├──
|
|
18
|
-
│ ├──
|
|
19
|
-
│
|
|
20
|
-
│
|
|
21
|
-
│
|
|
22
|
-
│
|
|
15
|
+
├── qa/
|
|
16
|
+
│ ├── context.md # Project-wide context: roles, testing strategy, global rules (fill once)
|
|
17
|
+
│ ├── screens/<name>/
|
|
18
|
+
│ │ ├── features/ # .feature files (Gherkin)
|
|
19
|
+
│ │ ├── selectors/ # Element locator YAML mappings
|
|
20
|
+
│ │ ├── test-data/ # Test data YAML values
|
|
21
|
+
│ │ └── requirements/ # Screen specs, UI designs, notes
|
|
22
|
+
│ │ ├── spec.md # Structured screen specification
|
|
23
|
+
│ │ ├── ui/ # Screenshots, mockups, design images
|
|
24
|
+
│ │ └── test-viewpoint.md # Edge cases, decisions (optional)
|
|
23
25
|
├── specs/
|
|
24
26
|
│ └── generated/ # Auto-generated Playwright tests
|
|
25
27
|
├── .claude/
|
|
@@ -66,11 +68,12 @@ Scaffolds `qa/screens/<name>/` with empty feature, selectors, test-data, and req
|
|
|
66
68
|
| `/sungen:create-test login` | `/sungen-create-test login` |
|
|
67
69
|
|
|
68
70
|
AI acts as a **Senior QA Engineer**:
|
|
69
|
-
1. Reads `
|
|
70
|
-
2.
|
|
71
|
-
3.
|
|
72
|
-
4.
|
|
73
|
-
5.
|
|
71
|
+
1. Reads `qa/context.md` for project-wide context (roles, testing strategy, global rules)
|
|
72
|
+
2. Reads `requirements/spec.md` for screen specs (fields, validation, business rules, states)
|
|
73
|
+
3. Optionally explores the live page via Playwright MCP to verify and supplement
|
|
74
|
+
4. Identifies screen sections → asks user which to focus on
|
|
75
|
+
5. Generates **20+ scenarios per viewpoint** (UI/UX, Validation, Logic, Security) for each section
|
|
76
|
+
6. Confirms test plan before generating `.feature` + `test-data.yaml`
|
|
74
77
|
|
|
75
78
|
### Step 3: Compile & run tests
|
|
76
79
|
|
package/package.json
CHANGED
|
@@ -39,6 +39,9 @@ export class ProjectInitializer {
|
|
|
39
39
|
// Create directories
|
|
40
40
|
this.createDirectories();
|
|
41
41
|
|
|
42
|
+
// Create qa/context.md for QA lead to fill project-wide context
|
|
43
|
+
this.createContext();
|
|
44
|
+
|
|
42
45
|
// Ensure package.json and install Playwright
|
|
43
46
|
await this.setupDependencies();
|
|
44
47
|
|
|
@@ -363,6 +366,23 @@ export class ProjectInitializer {
|
|
|
363
366
|
|
|
364
367
|
}
|
|
365
368
|
|
|
369
|
+
/**
|
|
370
|
+
* Create qa/context.md for the QA lead to fill project-wide context
|
|
371
|
+
* (roles, testing strategy, global rules, error patterns).
|
|
372
|
+
*/
|
|
373
|
+
private createContext(): void {
|
|
374
|
+
const contextPath = path.join(this.cwd, 'qa', 'context.md');
|
|
375
|
+
|
|
376
|
+
if (fs.existsSync(contextPath)) {
|
|
377
|
+
this.skippedItems.push('qa/context.md');
|
|
378
|
+
return;
|
|
379
|
+
}
|
|
380
|
+
|
|
381
|
+
const content = this.readTemplate('qa-context.md');
|
|
382
|
+
fs.writeFileSync(contextPath, content, 'utf-8');
|
|
383
|
+
this.createdItems.push('qa/context.md');
|
|
384
|
+
}
|
|
385
|
+
|
|
366
386
|
/**
|
|
367
387
|
* Create specs/base.ts for shared browser context
|
|
368
388
|
*/
|
|
@@ -32,7 +32,15 @@ Parse **name** from `$ARGUMENTS`. If missing, ask the user.
|
|
|
32
32
|
- If no → fresh creation. Use `AskUserQuestion` to ask generation scope:
|
|
33
33
|
- **Tier 1 — Critical & High priority** — ~10-15 scenarios/section covering happy paths, core validation, security basics **(Recommended)**
|
|
34
34
|
- **Full coverage — All tiers at once** — generates Tier 1 + 2 + 3 in one run. Large output (~40-60 scenarios/section), best for experienced users who want complete coverage immediately
|
|
35
|
-
3. **Read
|
|
35
|
+
3. **Read project context + screen requirements**
|
|
36
|
+
|
|
37
|
+
**Project context** — check `qa/context.md` (project root, not screen-specific):
|
|
38
|
+
- If exists → read it. Extract: roles, testing strategy directives, global business rules, error patterns.
|
|
39
|
+
- Summarize what you found in one line (e.g. `"Roles: admin/staff/user | Strategy: focus security, skip VP-UI T1 | 2 global rules"`).
|
|
40
|
+
- These are carried into the Coverage Map when invoking `sungen-tc-generation`.
|
|
41
|
+
- If absent → continue without it, no action needed.
|
|
42
|
+
|
|
43
|
+
**Screen requirements** — check `qa/<screens|flows>/<name>/requirements/`:
|
|
36
44
|
- If `spec.md` exists → read it as PRIMARY source (sections, fields, validation rules, business rules, states).
|
|
37
45
|
- If `test-viewpoint.md` exists → read it. If it only contains HTML comments (scaffold template), use `AskUserQuestion` to ask:
|
|
38
46
|
- **Fill test-viewpoint.md first** — I'll help you identify edge cases, known issues, and design decisions for this screen before generating tests
|
|
@@ -41,11 +41,13 @@ Skip this pre-flight when `--env` matches the base locale (no overlay needed in
|
|
|
41
41
|
Phase 0 — Selector Generation decision tree
|
|
42
42
|
|
|
43
43
|
Live page reachable? (URL provided and loads without error)
|
|
44
|
-
YES → existing flow: browser_navigate →
|
|
44
|
+
YES → existing flow: browser_navigate → wait for page to fully load (no spinner/skeleton/empty table) →
|
|
45
|
+
one browser_snapshot → cross-verify every [Reference] label vs snapshot name →
|
|
46
|
+
generate selectors.yaml (verified entries; explicit YAML for any label≠DOM-name mismatch)
|
|
45
47
|
NO → spec_figma.md exists in requirements/?
|
|
46
48
|
YES → provisional flow (sungen-figma-source + sungen-selector-fix skills):
|
|
47
49
|
1. Read filtered Figma node data from spec_figma.md (## Components + ## Text Inventory)
|
|
48
|
-
2. Apply selector
|
|
50
|
+
2. Apply selector priority from sungen-selector-fix § Step 3 (testid > role+name > label > placeholder > text > locator CSS last)
|
|
49
51
|
3. Write selectors.yaml — every provisional entry gets this comment on the line above:
|
|
50
52
|
# @needs-live-verify source=figma node_id=<id>
|
|
51
53
|
4. Compile: Screen: sungen generate --screen <name>. Flow: sungen generate --flow <name> — must succeed
|
|
@@ -55,13 +55,23 @@ When running Phase 0 for a **flow** (`qa/flows/<name>/`), check existing screen
|
|
|
55
55
|
- Read `baseURL` from `playwright.config.ts`.
|
|
56
56
|
- `browser_navigate` to the page URL.
|
|
57
57
|
- If redirected to login → run **Phase 0.5: Auth Persistence** first (see below), then re-navigate to the target page.
|
|
58
|
-
5. **Snapshot**:
|
|
58
|
+
5. **Snapshot**: Wait for the page to fully load before snapshotting.
|
|
59
|
+
- Check if the page is still loading (spinner visible, skeleton placeholders, empty table with 0 rows). If so, use `browser_wait_for` to wait until content is rendered.
|
|
60
|
+
- Then take **ONE** `browser_snapshot`. All Phase 0 selectors come from this single snapshot.
|
|
59
61
|
6. **Generate YAML entries**:
|
|
60
62
|
- Keys: follow `sungen-selector-keys` (lowercase, Unicode preserved, `--type` / `--N` suffixes).
|
|
61
|
-
- Selector priority: follow the table in **Diagnosis & Fix § Step 3** (`testid` > `role`+name > `
|
|
63
|
+
- Selector priority: follow the table in **Diagnosis & Fix § Step 3** (`testid` > `role`+name > `label` > `placeholder` > `text` > `locator` CSS last resort).
|
|
62
64
|
- Copy names **character-for-character** from the snapshot. Never infer from the Gherkin label.
|
|
63
65
|
- If an element is auto-inferable per `sungen-selector-keys` § Auto-Infer, **omit it** from YAML — keep the file minimal.
|
|
64
66
|
- **i18n sites**: if the site supports multiple languages, use `{{variable}}` in `name`/`value` fields instead of hardcoded text. Add corresponding `lbl_*` keys to `test-data.yaml` + locale overlay files (see `sungen-selector-keys` § i18n).
|
|
67
|
+
- **Selector quality rule**: the Playwright MCP accessibility tree snapshot gives you roles and accessible names directly — use them. Do NOT write XPath or class-based CSS selectors. Only write `type: locator` when no role/text/label/placeholder/testid is available, and restrict the CSS to `#id` or `[data-*]` / `[aria-*]` attribute selectors.
|
|
68
|
+
6b. **Cross-verify Gherkin labels vs snapshot** (prevents the #1 production failure):
|
|
69
|
+
- For **every** `[Reference]` in the `.feature` that will rely on auto-infer (not written to YAML), check the snapshot:
|
|
70
|
+
- `[X] button` — is there a button with accessible name **exactly** `X`?
|
|
71
|
+
- `[X] field` — does an input have placeholder **exactly** `X`? Does it even have a placeholder?
|
|
72
|
+
- `[X] heading` / `text` / `message` — is that text literally visible in the snapshot?
|
|
73
|
+
- If any mismatch → write an explicit YAML entry using the real DOM name. Do not leave a mismatch to be caught at runtime.
|
|
74
|
+
- **Typical mismatch cases**: Gherkin uses English label (`[Submit]`) but app displays Vietnamese (`"Gửi"`); placeholder is descriptive (`"Nhập email của bạn"`) not a bare field name (`"Email"`); button text includes an icon glyph before/after the word.
|
|
65
75
|
7. **Substring ambiguity check**: for each `role` + `name` selector, check if any other element in the snapshot has a name that **contains** this name as a substring (e.g., `"Đăng ký"` vs `"Đăng ký bằng Google"`). If yes → add `exact: true` to prevent strict mode violation at runtime.
|
|
66
76
|
8. **Merge, don't overwrite**: preserve the page selector and any user-authored entries in `selectors.yaml`. Only add missing keys.
|
|
67
77
|
9. **Show summary + confirm**: list the keys that will be added, ask the user to approve, then write the file.
|
|
@@ -69,9 +79,13 @@ When running Phase 0 for a **flow** (`qa/flows/<name>/`), check existing screen
|
|
|
69
79
|
|
|
70
80
|
### Common Phase 0 pitfalls
|
|
71
81
|
|
|
72
|
-
- Writing keys inferred from the Gherkin label instead of the snapshot name → Phase 1 will fail with
|
|
82
|
+
- Writing keys inferred from the Gherkin label instead of the snapshot name → Phase 1 will fail with `No element found`.
|
|
73
83
|
- Skipping Phase 0.5 when an auth redirect happened → snapshot captures the login page, all selectors wrong.
|
|
84
|
+
- Taking snapshot while page is still loading (spinner visible, table empty) → selectors for dynamic content will be missing or wrong.
|
|
85
|
+
- Skipping step 6b for "simple" elements like buttons → silent mismatch between Gherkin label and DOM name fails at runtime.
|
|
74
86
|
- Using `browser_evaluate` alone to scrape cookies → misses httpOnly session cookies. Always use `browser_storage_state` (or the `browser_run_code` fallback).
|
|
87
|
+
- Writing XPath or class-based CSS selectors → breaks on DOM/style refactoring. Use role/testid/text/label/placeholder from the accessibility tree.
|
|
88
|
+
- Falling back to `locator: 'div.some-class > span'` when the element IS visible in the accessibility snapshot with a role + name → the snapshot gives you `getByRole` for free; use it.
|
|
75
89
|
- Overwriting user-authored selectors → always merge.
|
|
76
90
|
|
|
77
91
|
---
|
|
@@ -210,12 +224,24 @@ Selector priority (use first applicable):
|
|
|
210
224
|
|
|
211
225
|
| Priority | type | When |
|
|
212
226
|
|---|---|---|
|
|
213
|
-
| 1 | `testid` | `data-testid` exists |
|
|
214
|
-
| 2 | `role` + exact name | Interactive elements |
|
|
215
|
-
| 3 | `
|
|
216
|
-
| 4 | `
|
|
217
|
-
| 5 | `
|
|
218
|
-
| 6 | `
|
|
227
|
+
| 1 | `testid` | `data-testid` or any stable test attribute exists |
|
|
228
|
+
| 2 | `role` + exact name | Interactive elements with an accessible name |
|
|
229
|
+
| 3 | `label` | Form field with a visible `<label>` |
|
|
230
|
+
| 4 | `placeholder` | Input/textarea with a placeholder attribute |
|
|
231
|
+
| 5 | `text` | Static visible text content |
|
|
232
|
+
| 6 | `locator` (CSS) | Last resort — `#id` or `[attr=value]` **only** (see restrictions below) |
|
|
233
|
+
|
|
234
|
+
> ⚠️ **Playwright best practice** ([source](https://playwright.dev/docs/best-practices#use-locators)): user-facing locators (`role`, `label`, `text`, `placeholder`, `testid`) are resilient to refactoring and far less likely to break. CSS class selectors and XPath break whenever a developer renames a class or restructures the DOM — even without changing the UI.
|
|
235
|
+
>
|
|
236
|
+
> **Never write these in `selectors.yaml`**:
|
|
237
|
+
> - XPath: `xpath=//div[@class='...']` or `//button[contains(@class,'btn')]`
|
|
238
|
+
> - Class-based CSS: `div.btn-primary`, `.modal-footer > .submit-btn`
|
|
239
|
+
> - Deep structural CSS: `div:nth-child(3) > ul > li > button`
|
|
240
|
+
>
|
|
241
|
+
> **Acceptable CSS (last resort only)**:
|
|
242
|
+
> - Stable `id`: `#submit-button` (only if the id is truly stable and not dynamic)
|
|
243
|
+
> - Data attributes: `[data-id="123"]`, `[aria-controls="menu"]`
|
|
244
|
+
> - Input type: `input[type="file"]` (when no testid/label exists)
|
|
219
245
|
|
|
220
246
|
**Exact name rule**: copy name character-for-character from snapshot. Never infer from Gherkin label.
|
|
221
247
|
|
|
@@ -229,9 +255,9 @@ Common fixes:
|
|
|
229
255
|
- Name mismatch → copy exact name from snapshot
|
|
230
256
|
- Multiple matches → add `nth` or `exact: true`
|
|
231
257
|
- Substring ambiguity (e.g., `"Submit"` matches `"Submit"` and `"Submit & Continue"`) → add `exact: true`
|
|
232
|
-
- No accessible name → use `testid
|
|
258
|
+
- No accessible name → use `testid`; only fall back to `locator` CSS as last resort
|
|
233
259
|
- Element in iframe → add `frame` field
|
|
234
|
-
- Dynamic content → use `testid` or
|
|
260
|
+
- Dynamic content → use `testid` or `role` + `nth`
|
|
235
261
|
|
|
236
262
|
### Step 4: Recompile After Fix
|
|
237
263
|
|
|
@@ -248,6 +274,26 @@ Then re-run only the current phase's failing tests, not all tests.
|
|
|
248
274
|
|
|
249
275
|
---
|
|
250
276
|
|
|
277
|
+
## Common Failure Patterns
|
|
278
|
+
|
|
279
|
+
Quick reference for the most frequent production failures:
|
|
280
|
+
|
|
281
|
+
| Symptom | Root cause | Fix |
|
|
282
|
+
|---------|-----------|-----|
|
|
283
|
+
| `No element found` on button/link/heading | Gherkin `[Reference]` label ≠ DOM accessible name (different language or text) | Write explicit YAML: `type: role, value: button, name: "<exact DOM name>"` |
|
|
284
|
+
| `No element found` on `[X] field` | Field has no placeholder, or placeholder ≠ X | Write explicit YAML: `type: label, value: "Actual label"` or `type: placeholder, value: "Actual placeholder"` |
|
|
285
|
+
| `No element found` on `[X] text` / `message` | Visible text differs from Gherkin label, or text is dynamic | Write explicit YAML or use `{{variable}}` for dynamic content |
|
|
286
|
+
| `strict mode violation` | Multiple elements match the same name/text | Add `exact: true` to YAML entry, or add `nth` |
|
|
287
|
+
| `toBeVisible` timeout on dynamic content | Snapshot was taken while page was still loading | Wait for spinner/skeleton to clear before snapshotting; add `browser_wait_for` |
|
|
288
|
+
| All tests fail with page navigate error | Page selector URL wrong or baseURL mismatch | Re-check `playwright.config.ts` `baseURL` and page selector `value` path |
|
|
289
|
+
| Auth redirect on every test | `specs/.auth/<role>.json` missing or expired | Run Phase 0.5 to capture fresh session |
|
|
290
|
+
| Table row assertions fail | `columns` config has wrong indices | Count column headers left-to-right (0-indexed) from snapshot |
|
|
291
|
+
| Wrong text assertions on locale page | Hardcoded Vietnamese/English text in YAML `name`/`value` | Use `{{lbl_*}}` variables with locale overlay files |
|
|
292
|
+
| Element inside iframe not found | `frame` field missing in YAML entry | Add `frame: "iframe[src*='...']"` to the selector entry |
|
|
293
|
+
| Selector breaks after UI redesign with no functional change | CSS class or XPath used — brittle to style refactoring | Rewrite with `role`/`testid`/`label`/`text` from accessibility snapshot |
|
|
294
|
+
|
|
295
|
+
---
|
|
296
|
+
|
|
251
297
|
## Table Selectors
|
|
252
298
|
|
|
253
299
|
For table patterns, add table selectors with `columns` config:
|
|
@@ -166,37 +166,47 @@ Resolver searches in this order:
|
|
|
166
166
|
|
|
167
167
|
If no YAML key exists, the resolver infers from the Gherkin element type:
|
|
168
168
|
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
|
178
|
-
|
|
179
|
-
| `[X]
|
|
180
|
-
| `[X]
|
|
181
|
-
| `[X]
|
|
182
|
-
| `[X]
|
|
183
|
-
| `[X]
|
|
184
|
-
| `[X]
|
|
185
|
-
| `[X]
|
|
186
|
-
| `[X]
|
|
187
|
-
| `[X]
|
|
188
|
-
| `[X]
|
|
189
|
-
| `[X]
|
|
190
|
-
| `[X]
|
|
191
|
-
| `[X]
|
|
192
|
-
| `[X]
|
|
193
|
-
| `[X]
|
|
194
|
-
| `[X]
|
|
195
|
-
| `[X]
|
|
196
|
-
| `[X]
|
|
197
|
-
| `[X]
|
|
198
|
-
|
|
199
|
-
|
|
169
|
+
> ⚠️ **Auto-infer pitfall — the #1 cause of selector failures in production.**
|
|
170
|
+
>
|
|
171
|
+
> `[X] button` auto-infers as `getByRole('button', { name: 'X' })`. This **only works** when the button's accessible name in the DOM is **exactly `X`** — same language, same text, same casing.
|
|
172
|
+
>
|
|
173
|
+
> The Gherkin `[Reference]` is your human label for the element, **not** the DOM name. If the app is in Vietnamese (or any language where the Gherkin label differs from DOM text), auto-infer will produce `No element found` at runtime. **Write an explicit YAML entry** with the real DOM name instead.
|
|
174
|
+
>
|
|
175
|
+
> **Decision rule**: auto-infer is safe ONLY when you have confirmed in the snapshot that the DOM element's accessible name / placeholder text is literally `X`. When in doubt → write YAML.
|
|
176
|
+
|
|
177
|
+
| Gherkin | Inferred locator | Safe when… |
|
|
178
|
+
|---|---|---|
|
|
179
|
+
| `[X] button` | `getByRole('button', { name: 'X' })` | Button's accessible name = X |
|
|
180
|
+
| `[X] link` | `getByRole('link', { name: 'X' })` | Link text = X |
|
|
181
|
+
| `[X] heading` / `header` | `getByRole('heading', { name: 'X' })` | Heading text = X |
|
|
182
|
+
| `[X] checkbox` | `getByRole('checkbox', { name: 'X' })` | Checkbox label = X |
|
|
183
|
+
| `[X] radio` | `getByRole('radio', { name: 'X' })` | Radio label = X |
|
|
184
|
+
| `[X] field` | `getByPlaceholder('X')` | Placeholder text = X AND field has a placeholder |
|
|
185
|
+
| `[X] text` / `message` / `label` | `getByText('X')` | Visible text = X (partial match) |
|
|
186
|
+
| `[X] logo/image/icon` | `getByRole('img', { name: 'X' })` | Image alt = X |
|
|
187
|
+
| `[X] search` | `getByRole('searchbox', { name: 'X' })` | Searchbox label = X |
|
|
188
|
+
| `[X] option` | `getByRole('option', { name: 'X' })` | Option text = X |
|
|
189
|
+
| `[X] slider` | `getByRole('slider', { name: 'X' })` | Slider label = X |
|
|
190
|
+
| `[X] toggle` | `getByRole('switch', { name: 'X' })` | Toggle label = X |
|
|
191
|
+
| `[X] tab` | `getByRole('tab', { name: 'X' })` | Tab text = X |
|
|
192
|
+
| `[X] table` | `getByRole('table', { name: 'X' })` | Table aria-label = X |
|
|
193
|
+
| `[X] list` | `getByRole('list', { name: 'X' })` | List aria-label = X |
|
|
194
|
+
| `[X] column` | `getByRole('columnheader', { name: 'X' })` | Column header text = X |
|
|
195
|
+
| `[X] dialog` / `modal` / `drawer` | `getByRole('dialog', { name: 'X' })` | Dialog aria-label/heading = X |
|
|
196
|
+
| `[X] dropdown` / `select` | `getByRole('combobox', { name: 'X' })` | Combobox label = X |
|
|
197
|
+
| `[X] menuitem` | `getByRole('menuitem', { name: 'X' })` | Menu item text = X |
|
|
198
|
+
| `[X] progressbar` | `getByRole('progressbar', { name: 'X' })` | Progressbar label = X |
|
|
199
|
+
| `[X] section` | `getByRole('region', { name: 'X' })` | Section aria-label = X |
|
|
200
|
+
| `[X] card` | `getByRole('article', { name: 'X' })` | Card aria-label = X |
|
|
201
|
+
| `[X] item` | `getByRole('listitem', { name: 'X' })` | List item text = X |
|
|
202
|
+
| `[X] cell` | `getByRole('cell', { name: 'X' })` | Cell text = X |
|
|
203
|
+
| `[X] spinner` | `getByRole('status', { name: 'X' })` | Spinner aria-label = X |
|
|
204
|
+
| `[X] breadcrumb` | `getByRole('navigation', { name: 'X' })` | Navigation aria-label = X |
|
|
205
|
+
| `[X] badge` / `tooltip` / `tag` | `getByText('X')` | Visible text = X |
|
|
206
|
+
|
|
207
|
+
**Special note on `[X] field`**: `getByPlaceholder('X')` only works when (1) the field has a placeholder attribute AND (2) the placeholder text equals X. For fields without placeholders (floating labels, aria-label), write explicit YAML: `type: label, value: "Actual label text"`.
|
|
208
|
+
|
|
209
|
+
**Only add a YAML entry when** auto-infer cannot work: DOM name differs from Gherkin label, need `testid`, need `nth`, need `exact: true`, or the field type requires explicit config.
|
|
200
210
|
|
|
201
211
|
### Types requiring YAML entry (no auto-infer)
|
|
202
212
|
|
|
@@ -105,6 +105,14 @@ Auto-detected by `create-test` before invoking this skill:
|
|
|
105
105
|
2. Each row / bullet / item = 1 viewpoint → add to `Viewpoint items` in Coverage Map.
|
|
106
106
|
3. Do NOT pre-classify into buckets before scanning — classify only when
|
|
107
107
|
writing the scenario.
|
|
108
|
+
- `qa/context.md` — project-wide context set by the QA lead. Read ONCE before building the Coverage Map; apply to every screen. Extraction rules:
|
|
109
|
+
- **Roles** → for each role in the table: add to the `@auth:X` tag pool; generate a VP-SEC blocked-access scenario for every role boundary relevant to this screen.
|
|
110
|
+
- **Testing strategy → Focus areas** → if `security` listed: VP-SEC is mandatory Tier 1 for every free-text input regardless of spec risk level; if `ui` not listed: all VP-UI scenarios move to Tier 2 minimum.
|
|
111
|
+
- **Testing strategy → Mandatory coverage** → each line is a hard override applied to this screen regardless of spec risk; document in `Context constraints` of the Coverage Map.
|
|
112
|
+
- **Testing strategy → Deprioritize/skip** → record in `Context constraints`; suppress those VP categories from Tier 1 generation.
|
|
113
|
+
- **Global business rules** → add each to the `Business rules` section tagged `[G]` (e.g. `[G1 – soft-delete only]`); treat as `HIGH` risk unless stated otherwise.
|
|
114
|
+
- **Error patterns** → use as fallback only when `spec.md` does not give exact error text; never override spec-specified messages.
|
|
115
|
+
- If `qa/context.md` is absent: proceed without it — no impact on the generation flow.
|
|
108
116
|
|
|
109
117
|
**Single screen focus**: one URL = one screen. Modals on same page = part of this screen.
|
|
110
118
|
This means: do not test other screens' UI layout or navigation. It does NOT mean skip documenting business outcomes that your screen's actions cause on other surfaces. Those cross-surface outcomes must appear in the Coverage Map and be covered by at least `@manual` scenarios.
|
|
@@ -129,6 +137,11 @@ Read `spec.md` fully, then extract into a Coverage Map **before writing any scen
|
|
|
129
137
|
**Risk tags:** HIGH = complex business rules, cascading fields, multi-step state changes, auth/integration. LOW = display-only, static labels, read-only fields.
|
|
130
138
|
|
|
131
139
|
```
|
|
140
|
+
Context constraints: [populated from qa/context.md before writing any scenario]
|
|
141
|
+
roles: [list roles, e.g. admin / manager / staff]
|
|
142
|
+
strategy: [active overrides, e.g. "VP-SEC mandatory T1", "VP-UI → T2 only"]
|
|
143
|
+
global rules: [G1 – ...] → also appear in Business rules below tagged [G]
|
|
144
|
+
→ leave empty if qa/context.md is absent or has no entries applicable to this screen
|
|
132
145
|
User journeys: [J1 – ...], [J2 – ...]
|
|
133
146
|
Validation rules: [V1 – field → "exact error text"], [V2 – ...]
|
|
134
147
|
Business rules: [B1 HIGH – ...], [B2 LOW – ...]
|
|
@@ -27,7 +27,15 @@ You are a **Senior QA Engineer**. You structure test cases by viewpoint categori
|
|
|
27
27
|
- If no → fresh creation. Ask generation scope:
|
|
28
28
|
- **1) Tier 1 — Critical & High priority** — ~10-15 scenarios/section covering happy paths, core validation, security basics **(Recommended)**
|
|
29
29
|
- **2) Full coverage — All tiers at once** — generates Tier 1 + 2 + 3 in one run. Large output (~40-60 scenarios/section), best for experienced users who want complete coverage immediately
|
|
30
|
-
3. **Read
|
|
30
|
+
3. **Read project context + screen requirements**
|
|
31
|
+
|
|
32
|
+
**Project context** — check `qa/context.md` (project root, not screen-specific):
|
|
33
|
+
- If exists → read it. Extract: roles, testing strategy directives, global business rules, error patterns.
|
|
34
|
+
- Summarize what you found in one line (e.g. `"Roles: admin/staff/user | Strategy: focus security, skip VP-UI T1 | 2 global rules"`).
|
|
35
|
+
- These are carried into the Coverage Map when invoking the `sungen-tc-generation` skill.
|
|
36
|
+
- If absent → continue without it, no action needed.
|
|
37
|
+
|
|
38
|
+
**Screen requirements** — check `<base>/${input:name}/requirements/`:
|
|
31
39
|
- If `spec.md` exists → read it as PRIMARY source (sections, fields, validation rules, business rules, states).
|
|
32
40
|
- If `test-viewpoint.md` exists → read it. If it only contains HTML comments (scaffold template), ask:
|
|
33
41
|
- **1) Fill test-viewpoint.md first** — identify edge cases, known issues, and design decisions before generating tests
|
|
@@ -41,11 +41,13 @@ Skip when `--env` matches the base locale.
|
|
|
41
41
|
Phase 0 — Selector Generation decision tree
|
|
42
42
|
|
|
43
43
|
Live page reachable? (URL provided and loads without error)
|
|
44
|
-
YES → existing flow: browser_navigate →
|
|
44
|
+
YES → existing flow: browser_navigate → wait for page to fully load (no spinner/skeleton/empty table) →
|
|
45
|
+
one browser_snapshot → cross-verify every [Reference] label vs snapshot name →
|
|
46
|
+
generate selectors.yaml (verified entries; explicit YAML for any label≠DOM-name mismatch)
|
|
45
47
|
NO → spec_figma.md exists in requirements/?
|
|
46
48
|
YES → provisional flow (sungen-figma-source + sungen-selector-fix skills):
|
|
47
49
|
1. Read filtered Figma node data from spec_figma.md (## Components + ## Text Inventory)
|
|
48
|
-
2. Apply selector
|
|
50
|
+
2. Apply selector priority from sungen-selector-fix § Step 3 (testid > role+name > label > placeholder > text > locator CSS last)
|
|
49
51
|
3. Write selectors.yaml — every provisional entry gets this comment on the line above:
|
|
50
52
|
# @needs-live-verify source=figma node_id=<id>
|
|
51
53
|
4. Compile: Screen: sungen generate --screen <name>. Flow: sungen generate --flow <name> — must succeed
|