@sun-asterisk/sungen 2.6.6 → 2.6.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/index.js +1 -1
- package/dist/orchestrator/templates/ai-instructions/claude-cmd-create-test.md +24 -8
- package/dist/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +170 -54
- package/dist/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +39 -0
- package/dist/orchestrator/templates/ai-instructions/copilot-cmd-create-test.md +23 -7
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +150 -48
- package/dist/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +39 -0
- package/package.json +1 -1
- package/src/cli/index.ts +1 -1
- package/src/orchestrator/templates/ai-instructions/claude-cmd-create-test.md +24 -8
- package/src/orchestrator/templates/ai-instructions/claude-skill-tc-generation.md +170 -54
- package/src/orchestrator/templates/ai-instructions/claude-skill-test-design-techniques.md +39 -0
- package/src/orchestrator/templates/ai-instructions/copilot-cmd-create-test.md +23 -7
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-tc-generation.md +150 -48
- package/src/orchestrator/templates/ai-instructions/github-skill-sungen-test-design-techniques.md +39 -0
package/dist/cli/index.js
CHANGED
|
@@ -20,7 +20,7 @@ async function main() {
|
|
|
20
20
|
program
|
|
21
21
|
.name('sungen')
|
|
22
22
|
.description('Deterministic E2E Test Compiler — Gherkin + Selectors → Playwright')
|
|
23
|
-
.version('2.6.
|
|
23
|
+
.version('2.6.8');
|
|
24
24
|
// Global options
|
|
25
25
|
program
|
|
26
26
|
.option('-v, --verbose', 'Enable verbose logging');
|
|
@@ -27,7 +27,11 @@ Parse **name** from `$ARGUMENTS`. If missing, ask the user.
|
|
|
27
27
|
|
|
28
28
|
1. **Flow**: Verify `qa/flows/<name>/` exists. If not → `/sungen:add-flow` first.
|
|
29
29
|
**Screen**: Verify `qa/screens/<name>/` exists. If not → `/sungen:add-screen` first.
|
|
30
|
-
2. Check if `.feature` file already has scenarios.
|
|
30
|
+
2. Check if `.feature` file already has scenarios.
|
|
31
|
+
- If yes → use `AskUserQuestion` to ask the update mode (see `sungen-tc-generation` skill — mode depends on which tiers already exist).
|
|
32
|
+
- If no → fresh creation. Use `AskUserQuestion` to ask generation scope:
|
|
33
|
+
- **Tier 1 — Critical & High priority** — ~10-15 scenarios/section covering happy paths, core validation, security basics **(Recommended)**
|
|
34
|
+
- **Full coverage — All tiers at once** — generates Tier 1 + 2 + 3 in one run. Large output (~40-60 scenarios/section), best for experienced users who want complete coverage immediately
|
|
31
35
|
3. **Read requirements & resolve visual source** — check `qa/<screens|flows>/<name>/requirements/`:
|
|
32
36
|
- If `spec.md` exists → read it as PRIMARY source (sections, fields, validation rules, business rules, states).
|
|
33
37
|
- If `test-viewpoint.md` exists → read it. If it only contains HTML comments (scaffold template), use `AskUserQuestion` to ask:
|
|
@@ -47,13 +51,25 @@ Parse **name** from `$ARGUMENTS`. If missing, ask the user.
|
|
|
47
51
|
|
|
48
52
|
Summarize what you found in requirements and present to the user.
|
|
49
53
|
|
|
50
|
-
4. Follow the `sungen-tc-generation` skill for section identification, viewpoint generation, and output format. **For flows**, use the "Flow Test Generation" section in the skill. When requirements exist, use the "Requirements-Driven Generation" strategy.
|
|
54
|
+
4. Follow the `sungen-tc-generation` skill for section identification, viewpoint generation, and output format. **For flows**, use the "Flow Test Generation" section in the skill. When requirements exist, use the "Requirements-Driven Generation" strategy. **For Tier 1**, apply the **Lightweight Guard** — verify required fields, validation rules, business rules, security checks, and key state transitions all have TCs after generation. **For Tier 2+**, **MUST** apply the full **Mapping Contract** — walk every `spec.md` section top-to-bottom and produce the indicated TCs per Table 1; handle `test-viewpoint.md` per Table 2. Do not silently skip sections.
|
|
51
55
|
5. Generate or update `.feature` + `test-data.yaml` following `sungen-gherkin-syntax` and `sungen-tc-generation` skills. **For flows**: use `[Screen:Element]` namespace format, namespace test-data by phase, add `@flow` tag.
|
|
52
|
-
6. Show summary, then use `AskUserQuestion` to offer next steps:
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
- **`/sungen:
|
|
56
|
-
- **`/sungen:
|
|
57
|
-
-
|
|
56
|
+
6. Show summary, then use `AskUserQuestion` to offer next steps based on which tier was just generated:
|
|
57
|
+
|
|
58
|
+
**After Tier 1 generation:**
|
|
59
|
+
- **`/sungen:review <name>`** — Review syntax, coverage, viewpoint quality (Recommended)
|
|
60
|
+
- **`/sungen:run-test <name>`** — Skip review, generate selectors and run tests now
|
|
61
|
+
- **`/sungen:create-test <name>`** — Expand coverage: add @normal + @low scenarios (Tier 2)
|
|
62
|
+
- **Done for now** — I'll come back later
|
|
63
|
+
|
|
64
|
+
**After Tier 2 generation:**
|
|
65
|
+
- **`/sungen:create-test <name>`** — Deep coverage: add BVA combos, cross-field validation, negative inputs, race conditions (Tier 3) (Recommended)
|
|
66
|
+
- **`/sungen:review <name>`** — Review syntax, coverage, viewpoint quality
|
|
67
|
+
- **`/sungen:run-test <name>`** — Generate selectors and run tests now
|
|
68
|
+
- **Done for now** — I'll come back later
|
|
69
|
+
|
|
70
|
+
**After Tier 3 or Full generation:**
|
|
71
|
+
- **`/sungen:review <name>`** — Review syntax, coverage, viewpoint quality (Recommended)
|
|
72
|
+
- **`/sungen:run-test <name>`** — Generate selectors and run tests now
|
|
73
|
+
- **Done for now** — I'll come back later
|
|
58
74
|
|
|
59
75
|
**No selectors.yaml** — selectors are generated during `/sungen:run-test`.
|
|
@@ -14,17 +14,34 @@ Generate **focused test cases per screen section** using a **tier-based approach
|
|
|
14
14
|
|---|---|---|---|
|
|
15
15
|
| **Tier 1** (default) | `@high` | Happy paths, required validation, core business rules, security basics | First run of `create-test` |
|
|
16
16
|
| **Tier 2** (expand) | `@normal` + `@low` | UI presence, optional validation, edge cases, cosmetic checks | User runs `create-test` again with "Add viewpoints" mode |
|
|
17
|
+
| **Tier 3** (deep) | `@high` + `@normal` + `@low` | Extra BVA combinations, cross-field validation, negative/destructive inputs, concurrent/race conditions, complex state transitions | Recommended after Tier 2 completes |
|
|
18
|
+
| **Full** (all-at-once) | All | Tier 1 + 2 + 3 combined in one run | Option at first run, **not recommended** — large output |
|
|
17
19
|
|
|
18
|
-
**Round 1 (Tier 1)** targets **~10-15 scenarios per section** — enough to cover critical flows and catch real bugs. This is the default behavior.
|
|
20
|
+
**Round 1 (Tier 1)** targets **~10-15 scenarios per section** — enough to cover critical flows and catch real bugs. This is the default behavior. A **Full** option is available but not recommended — it generates all tiers at once, producing very large output (~40-60 scenarios/section).
|
|
19
21
|
|
|
20
|
-
**Round 2 (Tier 2)** expands
|
|
22
|
+
**Round 2 (Tier 2)** expands coverage when the user explicitly chooses "Add viewpoints" or "Add new sections" update mode. Only then generate `@normal` + `@low` scenarios to fill coverage gaps.
|
|
21
23
|
|
|
22
24
|
## Update Mode
|
|
23
25
|
|
|
24
|
-
When `.feature` already has scenarios,
|
|
26
|
+
When `.feature` already has scenarios, detect which tiers exist by scanning section comments, then ask the appropriate update mode.
|
|
27
|
+
|
|
28
|
+
**Tier detection** — scan for `# --- Section:` comments:
|
|
29
|
+
- Only `(Tier 1: @high)` comments → **Tier 1 only**
|
|
30
|
+
- Has `(Tier 2: @normal + @low)` comments → **Tier 1 + 2**
|
|
31
|
+
- Has `(Tier 3: deep)` comments → **Tier 1 + 2 + 3**
|
|
32
|
+
|
|
33
|
+
**If Tier 1 only:**
|
|
25
34
|
1. **Add new sections** — append new sections with Tier 2 (`@normal` + `@low`) scenarios, continue numbering
|
|
26
35
|
2. **Add viewpoints** — expand existing sections with Tier 2 (`@normal` + `@low`) scenarios
|
|
27
|
-
|
|
36
|
+
|
|
37
|
+
**If Tier 1 + 2:**
|
|
38
|
+
1. **Deep coverage (Tier 3)** — add advanced scenarios: extra BVA, cross-field validation, negative inputs, race conditions **(Recommended)**
|
|
39
|
+
2. **Add new sections** — append new sections with Tier 2 scenarios, continue numbering
|
|
40
|
+
|
|
41
|
+
**If Tier 1 + 2 + 3** (full coverage):
|
|
42
|
+
1. **Add new sections** — append new sections if screen has changed
|
|
43
|
+
|
|
44
|
+
> **Replace all** (available in all modes) — overwrite with fresh Tier 1 (`@high`) generation. Use when spec has changed significantly.
|
|
28
45
|
|
|
29
46
|
For append: read highest `VP-<CAT>-<NNN>`, continue from next number. Never modify existing scenarios.
|
|
30
47
|
|
|
@@ -39,6 +56,71 @@ Requirements improve every viewpoint: exact error messages for VAL, business rul
|
|
|
39
56
|
|
|
40
57
|
If also exploring live page: verify spec vs actual, flag mismatches, capture exact text.
|
|
41
58
|
|
|
59
|
+
### Tier 1 — Lightweight Guard
|
|
60
|
+
|
|
61
|
+
> Tier 1 uses the quick 3-step strategy (Step 1–3) without full section walking. After generation, verify these checks against `spec.md` — if any are missing, add them before finalizing:
|
|
62
|
+
|
|
63
|
+
| Spec section | Minimum TC requirement | Tag |
|
|
64
|
+
|---|---|---|
|
|
65
|
+
| Fields row, `Required=yes` | At least 1 required-error TC per required field | `@high` |
|
|
66
|
+
| Validation Rules row | At least 1 exact-message TC per validation rule | `@high` |
|
|
67
|
+
| Business Rule bullet | At least 1 behavioral TC per business rule | `@high` |
|
|
68
|
+
| Security-sensitive actions (auth, permissions) | At least 1 security check (VP-SEC) | `@high` |
|
|
69
|
+
| Entity with lifecycle states | At least 1 key state transition (VP-LOGIC) | `@high` |
|
|
70
|
+
|
|
71
|
+
Other spec sections (Section header, Default, Accessibility, Notes) are deferred to Tier 2.
|
|
72
|
+
|
|
73
|
+
### Mapping Contract (Tier 2+ — MANDATORY)
|
|
74
|
+
|
|
75
|
+
> **Full contract applies from Tier 2 onwards.** Walk `spec.md` top-to-bottom. For every section listed below, you **MUST** produce the indicated TCs. If a section is missing or empty, note it explicitly — do **NOT** silently skip.
|
|
76
|
+
|
|
77
|
+
#### Table 1 — `spec.md` → TC Mapping Contract
|
|
78
|
+
|
|
79
|
+
| Spec section | TC requirement | Tier | Tag | Viewpoint |
|
|
80
|
+
|---|---|---|---|---|
|
|
81
|
+
| Section header | 1 presence TC | T2 | `@low` | VP-UI |
|
|
82
|
+
| Fields row, `Required=yes` | 1 per-field required-error TC (individual, not bulk) | T2 | `@normal` | VP-VAL |
|
|
83
|
+
| Fields `Constraints` cell (non-empty, non-`—`) | 4 BVA for numeric range; 1 EP for format | T2 | `@normal` | VP-VAL |
|
|
84
|
+
| Fields `Constraints` cell (numeric) | 2 extra BVA (`min+1`, `max-1`) + mid-range → total 7-point with T2 | T3 | `@normal` | VP-VAL |
|
|
85
|
+
| Fields `Default` cell (non-empty, non-`—`) | 1 default-state TC | T2 | `@low` | VP-UI |
|
|
86
|
+
| Fields with dependencies (conditional required, cascading) | 1 cross-field TC per dependency | T3 | `@normal` | VP-VAL |
|
|
87
|
+
| Fields accepting free text input | 1 negative input TC (XSS/SQL injection/special chars) | T3 | `@high` | VP-SEC |
|
|
88
|
+
| Actions row (primary: submit, create, delete) | 1 edge/alternate behavior TC | T2 | `@normal` | VP-LOGIC |
|
|
89
|
+
| Actions row (secondary: cancel, reset, export) | 1 behavior TC | T2 | `@normal` | VP-LOGIC |
|
|
90
|
+
| Actions with submit/save | 1 race condition TC (double-submit, back after POST) | T3 | `@normal` | VP-LOGIC |
|
|
91
|
+
| Validation Rules row | 1 exact-message TC | T2 | `@normal` | VP-VAL |
|
|
92
|
+
| States row (Default/Loading/Error/Success) | 1 visual state TC | T2 | `@normal` | VP-UI / VP-LOGIC |
|
|
93
|
+
| States row (other states) | 1 visual TC | T2 | `@low` | VP-UI |
|
|
94
|
+
| Business Rule bullet | 1 edge-case behavioral TC; `@manual` if unautomatable | T2 | `@normal` | VP-LOGIC |
|
|
95
|
+
| Accessibility tab order | 1 tab-walking TC | T2 | `@low` | VP-UI |
|
|
96
|
+
| Notes bullet | classify per content (page title→VP-UI, env→`@manual`, edge→relevant section) | T2 | `@low` | varies |
|
|
97
|
+
|
|
98
|
+
#### Table 2 — `test-viewpoint.md` → TC Mapping Contract
|
|
99
|
+
|
|
100
|
+
| Section / format | TC requirement |
|
|
101
|
+
|---|---|
|
|
102
|
+
| `## Edge Cases` (bulleted) | classify per spec Notes |
|
|
103
|
+
| `## Known Issues` (bulleted) | `@manual` TCs with bug ID comment |
|
|
104
|
+
| `## Design Decisions` | behavioral TCs |
|
|
105
|
+
| `## UI Patterns Identified` | confirm `sungen-viewpoint` pattern checklists |
|
|
106
|
+
| `## Priority Viewpoints` | adjust Tier-2 emphasis |
|
|
107
|
+
| `## Element Overview` (table) | 1 presence/state TC per row (T2, `@low`) |
|
|
108
|
+
| `## State Transitions` (table) | 1 TC per valid transition + key blocked |
|
|
109
|
+
| Free-form notes | re-read with heading as context |
|
|
110
|
+
|
|
111
|
+
### Figma supplement (`spec_figma.md`)
|
|
112
|
+
|
|
113
|
+
When `requirements/spec_figma.md` is present alongside `spec.md`, treat it as a **secondary input** with these rules:
|
|
114
|
+
|
|
115
|
+
- **Never override `spec.md`**: `spec.md` is authoritative for all business rules, field constraints, and behavior. `spec_figma.md` only supplements with visual/text data that `spec.md` may lack.
|
|
116
|
+
- **`## Text Inventory` → literal strings**: use text label values from this section verbatim in `test-data.yaml` (button labels, input placeholders, error messages shown in Figma). Do not paraphrase or invent alternatives.
|
|
117
|
+
- **`## Interaction States` → state coverage checkpoints**: use the listed variants (e.g., empty, loading, error, success) as a checklist for state-coverage scenarios. Only generate scenarios for states that are either (a) confirmed in both `spec.md` and `spec_figma.md`, or (b) explicitly documented in one source without contradiction from the other.
|
|
118
|
+
- **Flag disagreements**: if a field name, label, or behavior in `spec_figma.md` contradicts `spec.md`, insert an HTML comment at the top of the `.feature` file:
|
|
119
|
+
```gherkin
|
|
120
|
+
<!-- FIGMA-SPEC CONFLICT: <brief description> — using spec.md value -->
|
|
121
|
+
```
|
|
122
|
+
Then proceed using the `spec.md` value.
|
|
123
|
+
|
|
42
124
|
## Screen Input Sources
|
|
43
125
|
|
|
44
126
|
**Auto-detect** — the parent command (`create-test`) resolves visual sources before invoking this skill. By the time generation starts, the available sources are already determined:
|
|
@@ -89,8 +171,15 @@ Apply `sungen-test-design-techniques` to spec-extracted conditions:
|
|
|
89
171
|
Use `sungen-viewpoint` skill for per-pattern checklists across 4 viewpoints: UI/UX, Data & Validate, Logic, Security.
|
|
90
172
|
|
|
91
173
|
**Tier-aware gap filling:**
|
|
92
|
-
- **Tier 1 (first run)**: only add `@high` items from the checklists
|
|
174
|
+
- **Tier 1 (first run)**: only add `@high` items from the checklists. Skip `@normal`/`@low` items. After generation, apply the **Lightweight Guard** to verify coverage.
|
|
93
175
|
- **Tier 2 (expand run)**: add `@normal` + `@low` scenarios — UI presence, optional validation, edge cases, cosmetic checks, keyboard nav, hover effects.
|
|
176
|
+
- **Tier 3 (deep run)**: do NOT use viewpoint checklists. Instead, re-apply test design techniques with deeper analysis:
|
|
177
|
+
- **BVA**: expand from 4-point to 6-point (`min-1`, `min`, `min+1`, `max-1`, `max`, `max+1`) + typical mid-range value
|
|
178
|
+
- **Decision Table**: enumerate combinations previously capped — cover remaining outcome rows
|
|
179
|
+
- **Cross-field**: identify field dependencies from spec, generate one scenario per dependency
|
|
180
|
+
- **Negative inputs**: add security-oriented inputs (SQL `' OR 1=1`, XSS `<script>`, special chars `<>&"'`, max+100 length)
|
|
181
|
+
- **Race conditions**: double-click submit, browser back after POST, concurrent edit by two users
|
|
182
|
+
- **Full (all-at-once)**: apply Tier 1 + 2 + 3 rules in a single pass. Use all viewpoint checklists and all test design technique depths. Mark sections with all three tier comments.
|
|
94
183
|
|
|
95
184
|
**Validation rule**: capture actual error messages from live page or spec.md. Use `User see {{error_var}}` — never assert just "is visible".
|
|
96
185
|
|
|
@@ -100,29 +189,35 @@ Every scenario **MUST** have exactly one priority tag. Add it before the scenari
|
|
|
100
189
|
|
|
101
190
|
**Rule: look at what the WHEN step does and what THEN asserts — that determines the tag.**
|
|
102
191
|
|
|
103
|
-
| Tag |
|
|
104
|
-
|
|
105
|
-
| `@high` |
|
|
106
|
-
| `@
|
|
107
|
-
| `@
|
|
192
|
+
| Tag | Scenario type | Examples |
|
|
193
|
+
|---|---|---|
|
|
194
|
+
| `@high` | Auth actions | login, logout, protected page redirect |
|
|
195
|
+
| `@high` | CRUD happy path | create / update / delete → success message or redirect |
|
|
196
|
+
| `@high` | Security gate | unauthenticated access → redirected to login |
|
|
197
|
+
| `@high` | Primary business flow | core flow step completes successfully |
|
|
198
|
+
| `@normal` | Input validation | invalid / empty input → error message shown |
|
|
199
|
+
| `@normal` | Boundary / format | email format, length limit, numeric range |
|
|
200
|
+
| `@normal` | Secondary features | search, filter, sort, pagination, notification |
|
|
201
|
+
| `@normal` | Non-primary state transition | cancel, undo, back |
|
|
202
|
+
| `@low` | Element presence | component / section is visible |
|
|
203
|
+
| `@low` | Text content | label, placeholder, tooltip, warning text |
|
|
204
|
+
| `@low` | Visual / cosmetic | alignment, icon, empty state, hover style |
|
|
205
|
+
|
|
206
|
+
**`@steps:` scenarios** do NOT get a priority tag (they are setup blocks, not test cases).
|
|
108
207
|
|
|
109
|
-
###
|
|
208
|
+
### Priority assignment — disambiguation tie-breaker
|
|
110
209
|
|
|
111
|
-
|
|
210
|
+
When in doubt between `@low` and `@normal`, apply this rule:
|
|
211
|
+
|
|
212
|
+
| Pattern | Tag |
|
|
112
213
|
|---|---|
|
|
113
|
-
|
|
|
114
|
-
|
|
|
115
|
-
|
|
|
116
|
-
|
|
|
117
|
-
|
|
|
118
|
-
| Boundary / format validation (email, length, range) | `@normal` |
|
|
119
|
-
| Secondary features: search, filter, sort, pagination, notification | `@normal` |
|
|
120
|
-
| State transition not on primary success path (cancel, undo, back) | `@normal` |
|
|
121
|
-
| Element / component presence check (is it visible?) | `@low` |
|
|
122
|
-
| Text content: label, placeholder, tooltip, warning message text | `@low` |
|
|
123
|
-
| Visual / cosmetic: alignment, icon, empty state, hover style | `@low` |
|
|
214
|
+
| 0 When steps + Then asserts only `User see [T] type` (presence) | `@low` (always) |
|
|
215
|
+
| 0 When steps + Then asserts `User see [T] type with {{value}}` (content) | `@low` (label/text) |
|
|
216
|
+
| When steps but no state change (e.g., navigation only) | `@low` |
|
|
217
|
+
| When provides invalid input + Then asserts validation error | `@normal` |
|
|
218
|
+
| When triggers business logic + Then asserts state change | `@high` or `@normal` per heuristics |
|
|
124
219
|
|
|
125
|
-
|
|
220
|
+
**Example:** `Then User see [Email] field` (no When) → `@low`. `When User fill ... And User click ... Then User see [Error]` → `@normal`.
|
|
126
221
|
|
|
127
222
|
## SPA Wait-For Steps
|
|
128
223
|
|
|
@@ -240,7 +335,9 @@ Feature: <Screen> Screen
|
|
|
240
335
|
Then User is on [Login] page
|
|
241
336
|
```
|
|
242
337
|
|
|
243
|
-
**Tier 2 (expand run)**
|
|
338
|
+
**Tier 2 (expand run)** appends sections with `# --- Section: X (Tier 2: @normal + @low) ---` comments. Adds UI field presence, hover states, tooltips, empty states.
|
|
339
|
+
|
|
340
|
+
**Tier 3 (deep run)** appends sections with `# --- Section: X (Tier 3: deep) ---` comments. Adds extra BVA boundaries, cross-field validation, negative/destructive inputs, concurrent/race conditions.
|
|
244
341
|
|
|
245
342
|
### When to use DataTable vs Row Scope
|
|
246
343
|
|
|
@@ -253,17 +350,7 @@ Feature: <Screen> Screen
|
|
|
253
350
|
|
|
254
351
|
**Test data** — `qa/screens/<screen>/test-data/<screen>.yaml`, grouped by section. Data is loaded **at runtime** — keys become runtime lookups, not hardcoded strings. The same compiled test works across environments.
|
|
255
352
|
|
|
256
|
-
**Environment-specific data**:
|
|
257
|
-
|
|
258
|
-
```yaml
|
|
259
|
-
# login.yaml (base)
|
|
260
|
-
valid_email: admin@dev.example.com
|
|
261
|
-
valid_password: DevPass123
|
|
262
|
-
|
|
263
|
-
# login.staging.yaml (override for staging)
|
|
264
|
-
valid_email: admin@staging.example.com
|
|
265
|
-
valid_password: StagingPass456
|
|
266
|
-
```
|
|
353
|
+
**Environment-specific data**: create `<screen>.<env>.yaml` alongside the base file with only the keys that change. Users run `SUNGEN_ENV=staging npx playwright test` to merge overrides.
|
|
267
354
|
|
|
268
355
|
## Flow Test Generation
|
|
269
356
|
|
|
@@ -286,15 +373,41 @@ When generating tests for a **flow** (`qa/flows/<name>/`), adapt the strategy:
|
|
|
286
373
|
| Tier 1 focus | Happy path + required validation per section | Happy path through entire flow + auth + key error recovery |
|
|
287
374
|
| Background | Navigate to screen | Navigate to starting page |
|
|
288
375
|
|
|
289
|
-
### Flow-specific scenarios
|
|
376
|
+
### Flow-specific scenarios by tier
|
|
290
377
|
|
|
291
|
-
| Category | Examples |
|
|
292
|
-
|
|
293
|
-
| **Happy path** | Complete flow end-to-end with valid data |
|
|
294
|
-
| **Auth persistence** | Auth state maintained across screen transitions |
|
|
295
|
-
| **Error recovery** | Invalid input mid-flow → fix → continue |
|
|
296
|
-
| **Incomplete flow** | User abandons at each phase → state cleanup |
|
|
297
|
-
| **Cross-screen data** | Data entered on screen A visible on screen B |
|
|
378
|
+
| Category | Tier | Examples |
|
|
379
|
+
|---|---|---|
|
|
380
|
+
| **Happy path** | T1 | Complete flow end-to-end with valid data |
|
|
381
|
+
| **Auth persistence** | T1 | Auth state maintained across screen transitions |
|
|
382
|
+
| **Error recovery** | T1 | Invalid input mid-flow → fix → continue |
|
|
383
|
+
| **Incomplete flow** | T2 | User abandons at each phase → state cleanup |
|
|
384
|
+
| **Cross-screen data** | T2 | Data entered on screen A visible on screen B |
|
|
385
|
+
| **Screen transition edges** | T2 | Back button, refresh, deep link mid-flow |
|
|
386
|
+
| **UI state across screens** | T2 | Breadcrumb, progress indicator, nav state |
|
|
387
|
+
| **Negative inputs mid-flow** | T3 | XSS/SQL in cross-screen fields, special chars |
|
|
388
|
+
| **Concurrent flow** | T3 | Two tabs same flow, session expiry mid-flow |
|
|
389
|
+
| **Cross-screen validation** | T3 | Field dependency across screens, cascading errors |
|
|
390
|
+
|
|
391
|
+
### Flow Lightweight Guard (Tier 1)
|
|
392
|
+
|
|
393
|
+
> After generating Tier 1 flow scenarios, verify these checks — if any are missing, add them:
|
|
394
|
+
|
|
395
|
+
| Flow requirement | Minimum TC | Tag |
|
|
396
|
+
|---|---|---|
|
|
397
|
+
| Complete flow end-to-end with valid data | At least 1 happy path scenario | `@high` |
|
|
398
|
+
| Auth state across screen transitions | At least 1 auth persistence check (if flow crosses auth) | `@high` |
|
|
399
|
+
| Invalid input mid-flow → fix → continue | At least 1 error recovery scenario | `@high` |
|
|
400
|
+
|
|
401
|
+
### Flow Section Comments
|
|
402
|
+
|
|
403
|
+
Flow features use the same `# --- Section:` comment format, with phase names instead of UI sections:
|
|
404
|
+
|
|
405
|
+
```gherkin
|
|
406
|
+
# --- Section: Login Phase (Tier 1: @high) ---
|
|
407
|
+
# --- Section: Submission Phase (Tier 2: @normal + @low) ---
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
Update Mode tier detection works identically — scan for tier markers in these comments.
|
|
298
411
|
|
|
299
412
|
### Output Format (Flow)
|
|
300
413
|
|
|
@@ -305,6 +418,8 @@ Feature: Award Submission Flow
|
|
|
305
418
|
Background:
|
|
306
419
|
Given User is on [Login] page
|
|
307
420
|
|
|
421
|
+
# --- Section: Login Phase (Tier 1: @high) ---
|
|
422
|
+
|
|
308
423
|
@high
|
|
309
424
|
Scenario: User login successfully
|
|
310
425
|
When User fill [Login:Email] field with {{login.email}}
|
|
@@ -312,26 +427,27 @@ Feature: Award Submission Flow
|
|
|
312
427
|
And User click [Login:Submit] button
|
|
313
428
|
Then User see [Dashboard] page
|
|
314
429
|
|
|
430
|
+
# --- Section: Submission Phase (Tier 1: @high) ---
|
|
431
|
+
|
|
315
432
|
@high
|
|
316
433
|
Scenario: User navigates to awards and submits
|
|
317
|
-
|
|
318
|
-
Then User see [Awards] page
|
|
434
|
+
Given User is on [Awards] page
|
|
319
435
|
When User fill [Awards:Nominee] field with {{submission.nominee}}
|
|
436
|
+
And User fill [Awards:Reason] field with {{submission.reason}}
|
|
320
437
|
And User click [Awards:Submit] button
|
|
321
|
-
Then User see {{success_message}}
|
|
438
|
+
Then User see [Awards:Success Message] text with {{submission.success_message}}
|
|
322
439
|
```
|
|
323
440
|
|
|
324
|
-
**Test data** — `qa/flows/<name>/test-data/<name>.yaml`, namespaced by phase:
|
|
325
|
-
|
|
326
441
|
```yaml
|
|
442
|
+
# test-data — namespaced by phase
|
|
327
443
|
login:
|
|
328
|
-
email: "
|
|
329
|
-
password: "
|
|
444
|
+
email: "user@example.com"
|
|
445
|
+
password: "Password123"
|
|
446
|
+
|
|
330
447
|
submission:
|
|
331
448
|
nominee: "John Doe"
|
|
332
|
-
|
|
449
|
+
reason: "Outstanding contribution to the team"
|
|
450
|
+
success_message: "Award submitted successfully"
|
|
333
451
|
```
|
|
334
452
|
|
|
335
|
-
**Environment overrides** work the same: `<name>.<env>.yaml` merged at runtime via `SUNGEN_ENV`.
|
|
336
|
-
|
|
337
453
|
**Do NOT generate**: `selectors.yaml` (created during run-test), Playwright code (sungen compiles).
|
|
@@ -68,6 +68,45 @@ Scenario: VP-VAL-003 Above maximum is rejected # value = 101
|
|
|
68
68
|
- `@high VP-VAL-012 Maximum boundary (10) is accepted`
|
|
69
69
|
- `@high VP-VAL-013 Above maximum (11) is rejected`
|
|
70
70
|
|
|
71
|
+
### Expected outcome contract (MANDATORY)
|
|
72
|
+
|
|
73
|
+
When generating BVA scenarios, the **Then** assertion **MUST** match the scenario name's outcome verb:
|
|
74
|
+
|
|
75
|
+
| Name verb | Expected `Then` assertion |
|
|
76
|
+
|---|---|
|
|
77
|
+
| "is accepted" / "passes validation" | NO validation error visible. Form proceeds OR field is in normal state. Use `User see [Field Error] message is hidden` if uncertain. |
|
|
78
|
+
| "is rejected" / "fails validation" / "shows error" | Specific validation error visible with exact message text from spec. |
|
|
79
|
+
|
|
80
|
+
**Anti-pattern (do NOT generate):**
|
|
81
|
+
|
|
82
|
+
```gherkin
|
|
83
|
+
# BAD: name says "passes validation" but asserts auth error
|
|
84
|
+
Scenario: VP-VAL-005 Email at maximum boundary (254 chars) passes validation
|
|
85
|
+
When User fill [Email] field with {{email_max_254}}
|
|
86
|
+
And User fill [Password] field with {{valid_password}}
|
|
87
|
+
And User click [Login] button
|
|
88
|
+
Then User see [Auth Error] message with {{auth_error_message}}
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
**Correct pattern:**
|
|
92
|
+
|
|
93
|
+
```gherkin
|
|
94
|
+
# GOOD: name and assertion aligned
|
|
95
|
+
@normal
|
|
96
|
+
Scenario: VP-VAL-005 Email at maximum boundary (254 chars) passes validation
|
|
97
|
+
When User fill [Email] field with {{email_max_254}}
|
|
98
|
+
And User fill [Password] field with {{valid_password}}
|
|
99
|
+
And User click [Login] button
|
|
100
|
+
Then User see [Email Error] message is hidden
|
|
101
|
+
|
|
102
|
+
@normal
|
|
103
|
+
Scenario: VP-VAL-006 Email at maximum+1 (255 chars) fails validation
|
|
104
|
+
When User fill [Email] field with {{email_over_max_255}}
|
|
105
|
+
And User fill [Password] field with {{valid_password}}
|
|
106
|
+
And User click [Login] button
|
|
107
|
+
Then User see [Email Error] message with {{email_too_long_error}}
|
|
108
|
+
```
|
|
109
|
+
|
|
71
110
|
---
|
|
72
111
|
|
|
73
112
|
## 3. Decision Table
|
|
@@ -22,7 +22,11 @@ You are a **Senior QA Engineer**. You structure test cases by viewpoint categori
|
|
|
22
22
|
|
|
23
23
|
1. **Flow**: Verify `qa/flows/${input:name}/` exists. If not → `/sungen-add-flow` first.
|
|
24
24
|
**Screen**: Verify `qa/screens/${input:name}/` exists. If not → `/sungen-add-screen` first.
|
|
25
|
-
2. Check if `.feature` already has scenarios.
|
|
25
|
+
2. Check if `.feature` already has scenarios.
|
|
26
|
+
- If yes → summarize existing coverage and ask update mode (options depend on which tiers already exist — see `sungen-tc-generation` skill for details).
|
|
27
|
+
- If no → fresh creation. Ask generation scope:
|
|
28
|
+
- **1) Tier 1 — Critical & High priority** — ~10-15 scenarios/section covering happy paths, core validation, security basics **(Recommended)**
|
|
29
|
+
- **2) Full coverage — All tiers at once** — generates Tier 1 + 2 + 3 in one run. Large output (~40-60 scenarios/section), best for experienced users who want complete coverage immediately
|
|
26
30
|
3. **Read requirements & resolve visual source** — check `<base>/${input:name}/requirements/`:
|
|
27
31
|
- If `spec.md` exists → read it as PRIMARY source (sections, fields, validation rules, business rules, states).
|
|
28
32
|
- If `test-viewpoint.md` exists → read it. If it only contains HTML comments (scaffold template), ask:
|
|
@@ -42,13 +46,25 @@ You are a **Senior QA Engineer**. You structure test cases by viewpoint categori
|
|
|
42
46
|
|
|
43
47
|
Summarize what you found in requirements and present to the user.
|
|
44
48
|
|
|
45
|
-
4. Follow the `sungen-tc-generation` skill for section identification, viewpoint generation, and output format. **For flows**, use the "Flow Test Generation" section in the skill. When requirements exist, use the "Requirements-Driven Generation" strategy. Present sections as a numbered list and let user pick.
|
|
49
|
+
4. Follow the `sungen-tc-generation` skill for section identification, viewpoint generation, and output format. **For flows**, use the "Flow Test Generation" section in the skill. When requirements exist, use the "Requirements-Driven Generation" strategy. **For Tier 1**, apply the **Lightweight Guard** — verify required fields, validation rules, business rules, security checks, and key state transitions all have TCs after generation. **For Tier 2+**, **MUST** apply the full **Mapping Contract** — walk every `spec.md` section top-to-bottom and produce the indicated TCs per Table 1; handle `test-viewpoint.md` per Table 2. Do not silently skip sections. Present sections as a numbered list and let user pick.
|
|
46
50
|
5. Generate or update `.feature` + `test-data.yaml` following `sungen-gherkin-syntax` and `sungen-tc-generation` skills. **For flows**: use `[Screen:Element]` namespace format, namespace test-data by phase, add `@flow` tag.
|
|
47
|
-
6. Show summary and offer next steps:
|
|
51
|
+
6. Show summary and offer next steps based on which tier was just generated:
|
|
48
52
|
|
|
49
|
-
|
|
50
|
-
- **`/sungen-
|
|
51
|
-
- **`/sungen-
|
|
52
|
-
-
|
|
53
|
+
**After Tier 1 generation:**
|
|
54
|
+
- **`/sungen-review ${input:name}`** — Review syntax, coverage, viewpoint quality (Recommended)
|
|
55
|
+
- **`/sungen-run-test ${input:name}`** — Skip review, generate selectors and run tests now
|
|
56
|
+
- **`/sungen-create-test ${input:name}`** — Expand coverage: add @normal + @low scenarios (Tier 2)
|
|
57
|
+
- **Done for now** — I'll come back later
|
|
58
|
+
|
|
59
|
+
**After Tier 2 generation:**
|
|
60
|
+
- **`/sungen-create-test ${input:name}`** — Deep coverage: add BVA combos, cross-field validation, negative inputs, race conditions (Tier 3) (Recommended)
|
|
61
|
+
- **`/sungen-review ${input:name}`** — Review syntax, coverage, viewpoint quality
|
|
62
|
+
- **`/sungen-run-test ${input:name}`** — Generate selectors and run tests now
|
|
63
|
+
- **Done for now** — I'll come back later
|
|
64
|
+
|
|
65
|
+
**After Tier 3 or Full generation:**
|
|
66
|
+
- **`/sungen-review ${input:name}`** — Review syntax, coverage, viewpoint quality (Recommended)
|
|
67
|
+
- **`/sungen-run-test ${input:name}`** — Generate selectors and run tests now
|
|
68
|
+
- **Done for now** — I'll come back later
|
|
53
69
|
|
|
54
70
|
**No selectors.yaml** — selectors are generated during `/sungen-run-test`.
|